Usage with rsync

[JP-Ellis]

Usage with rsync

[astrada]

I started using your program and it really seems great. I was wondering though whether using rsync is of any
advantage for this in terms of bandwidth as I have noticed it to be quite slow.

You will get a little speedup only downloading from the server to your host. Drive API doesn't support partial uploads, so using rsync won't get you quicker uploads.

also, the mime type of the PDF according to Google is application/octet-stream.

Which version are you using?

[JP-Ellis]

Would the partial file transfers of rsync ever cause issues with the Drive API?

As for the version, I am using version 0.4.5 from Ubuntu's PPA.

[astrada]

The problem is that rsync copies the files using temporary random names (e.g. .filename.pdf.0MN9tN). But I choose the content type from the file extension, and when the file is uploaded, the extension is wrong. I can try to change the content type when the file is renamed, but I don't know if it's enough. I'll let you know if it works.

[astrada]

I made a test and it didn't work. It seems a problem on the Drive API side (see http://stackoverflow.com/questions/14629839/unable-to-update-mimetype-using-google-drive-api). The only workaround is to reupload the file, but this way you will upload all files 2 times.

[JP-Ellis]

Thanks for the quick reply.

Does that mean that using rsync (with its default settings) is incompatible with google-drive-ocamlfuse?

[astrada]

Not exactly, there are issues using rsync to upload files, but it works pretty well as a backup tool, to backup the content of your Drive on your host (I'm using it this way).

[astrada]

Looking at rsync manual, I found the --inplace option, that avoids creating temporary files. Maybe you can try it to see if using that option, the file MIME type is set to application/pdf.

[tobek]

For what it's worth, adding the --inplace option to rsync seems to have fixed this issue for me. I can't be totally certain, because sometimes it worked even before, so maybe I'm just getting lucky.

For other Googlers, here's the line in my crontab -e

0 * * * * date >> ~/drive-rsync-stats.log && rsync -trluhv --delete --stats --inplace /folder/to/upload/* /folder/mounted/with/google-drive-ocamlfuse/ >> ~/drive-rsync-stats.log

I'm using this so that I can access files from my main computer from all my other devices, so I don't care if this command clobbers stuff on Drive. Working great for me running every hour on my machine.

@astrada this thing is baller.

[aselvan]

I had the same issue with image files when I used rsync. After adding "--inplace" argument, backup to Google Drive via google-drive-ocamlfuse works perfect with rsync. Thanks @astrada

[brancomat]

Sorry to comment a closed issue, still this is the best thread I've found on using rsync on google-drive-ocamlfuse (imho some suggestions of this issue should be promoted to a wiki page, @astrada I'd gladly help)

I generally see everyone experiencing some vaguely defined "slow" rating but I can't find something more specific: I'm uploading files with rsync at something less than 1.4 Mbps, I'm wondering:

• is this considered "normal"?
• are there any fine tunings that can increase upload speed? (besides the --inplace option)
• (slightly OT) is this slowness due to google drive's api in general (meaning: there's no other way to significantly increase transfer speed to/from google drive)?

I'm aware that (quoting this interesting thread) there hasn't been a "clear statement from Google's side about what upload speed is expected" so I guess it's difficult to talk numbers.

[astrada]

imho some suggestions of this issue should be promoted to a wiki page, @astrada I'd gladly help

Sure! Feel free to create a new wiki page. Thanks!

is this considered "normal"?

I'm sorry, but I don't have a very stable connection (I'm on LTE that has very variable upload rate) so I can't measure a "normal" speed.

are there any fine tunings that can increase upload speed? (besides the --inplace option)

You can test these options:

• you can set a very high timeout in metadata_cache_time= so your cache doesn't risk to be invalidated
• you can try multi-threading with -m (I'm still testing this option so it's not yet super-stable) even if rsync is single-theaded (but you can test parallel)
• I'm still looking for some way to increase the maximum FUSE block size (that is 128KB)

(slightly OT) is this slowness due to google drive's api in general (meaning: there's no other way to significantly increase transfer speed to/from google drive)?

I'm experiencing very variable transfer rates too, but I'm not sure this is due to Google or to my 4G provider.

[brancomat]

Thank you,
wiki page created

I'm trying your suggestions, the rsync + parallel experiment went wrong: I was using this approach but after a while I got a lot of:

rsync error: errors selecting input/output files, dirs (code 3) at main.c(634) [Receiver=3.1.0]
rsync: ERROR: cannot stat destination "/my/ocamlfuse/mount/path": Transport endpoint is not connected (107)

Right now I've launched manually two separate instances of rsync on different subpath, so far so good but I'm not sure if this is affecting transfer speed

[astrada]

wiki page created

Nice, thanks.

rsync: ERROR: cannot stat destination "/my/ocamlfuse/mount/path": Transport endpoint is not connected (107)

OK, I will do a couple of tests.

Thanks for your feedback.

[brancomat]

It seems that the performance issues (and the "transport endpoint is not connected") might be related to the fact that I'm behind a ntlm authenticated proxy (I'm sorry I didn't mention that earlier, my bad). I repeated the test on the same network bypassing the proxy and speed improved greatly.
After around 100GB of data transfer I experienced a:

Device or resource busy (16)
rsync error: error in file IO (code 11) at receiver.c(856) [receiver=3.1.0]

But in that particular server I haven't still changed the metadata_cache_time, maybe it's because of that (?)

[astrada]

But in that particular server I haven't still changed the metadata_cache_time, maybe it's because of that (?)

No, I don't think so. It's probably a bug (if you are using multi-threading) or (less likely) you received too many errors server side (each request is retried at most 10 times).