Failed to upload file to serve webdav in rcloneExplorer

[xiaolei0125]

Failed to upload file to serve webdav in rcloneExplorer

[patrykcoding]

@xiaolei0125
Can you try this new release, it specifies a temporary path for rclone to use. It's based on the newest release (v1.7.1), so if you want to downgrade after testing you can only downgrade to the latest release.

RcloneExplorer.apk
RcloneExplorer-x86.apk
RcloneExplorer-ARM32.apk
RcloneExplorer-ARM64.apk

[patrykcoding]

I'll have a look where the problem lies. I'll try running rclone serve webdav in Termux to see if it's rclone related or not. I'll also have to look at the log files (currently rclone explorer doesn't generate any logs for serve commands unless it actually terminates with a non-zero exit value). I'll update once I'll know more.

[xiaolei0125]

@kaczmarkiewiczp Thanks for you reply.
rcloneExplorer is a very cool app on android. If you need some testing, I am happy to help.

[patrykcoding]

I compiled rclone using Termux. Then I run rclone serve webdav -vv on multiple remotes. I also set up FolderSync to sync a local folder to webdav. In all cases FolderSync shows that it failed. When syncing to a crpyt drive remote, FolderSync reported error: "Bad Gateway", and "Method Not Allowed" when using dropbox remote.

I also tried uploading using Solid Explorer and had better luck. While most files uploaded successfully, I would get an occasional error: "Solid Explorer attempted to perform an invalid operation" but it was very rare. Did all your uploads fail when trying Solid Explorer or ES Explorer?

This means that the problems you're experiencing are rclone related and have nothing to do with rcloneExplorer.

[xiaolei0125]

@kaczmarkiewiczp
I not compiled rclone using Termux, just copy rclone v1.42-066-g1f3778db arm64 to /data/data/com.termux/ on android.

Then run rclone serve webdav on termux, and the remote is sharepoint. uploading file using ES Explorer and Solid Explorer, or syncing directory using FolderSync, all are work fine.

Then closed termux and run serve webdav (remote is sharepoint) on rcloneExplorer(v1.6.0-20). uploading file using ES Explorer and Solid Explorer are work fine.

But syncing directory using FolderSync failed. FolderSync reported error: "Method Not Allowed" for second file, and the first file is sync success.

This is very strange, I am not sure what the problem is.

[xiaolei0125]

@kaczmarkiewiczp update for new testing.

I not compiled rclone using Termux, just copy rclone v1.42-066-g1f3778db arm64 to /data/data/com.termux/ on android.

Then run rclone serve webdav on termux, or on windows/linux, and the remote is sharepoint. uploading single or multiple files using ES Explorer and Solid Explorer, or syncing directory(single or multiple files) using FolderSync, all are work fine.

Then run serve webdav (remote is sharepoint) on rcloneExplorer(v1.6.0-20). uploading single file using ES Explorer and Solid Explorer, or syncing directory(contain only single file) using FolderSync, all are work fine.

But if uploading multiple files using ES Explorer and Solid Explorer, it's failed, or syncing directory(contain multiple files) using FolderSync, it's also failed, folderSync reported error: "Method Not Allowed" for second file, and the first file is sync success.

So the question seems to be why “rcloneExplore serve webdav” doesn't support handling multiple files uploading?

[patrykcoding]

Thanks for investigating. This is really strange. RcloneExplorer just executes rclone serve webdav and lets it run in the background. I really don't see how rcloneExplorer could introduce any bugs there.

What's also strange is that you say that you run rclone v1.42-066-g1f3778db downloaded from rclone.org (as opposed to compiled from source on Android). Whenever I try to run rclone that's compiled for Linux/ARM64, none of the commands that require internet connection (i.e. ls, copy, or serve) work.

I created a special APK that contains rclone v1.42-066-g1f3778db (you can confirm rclone version in rcloneExplorer by clicking on About in navigation drawer). Would you mind installing it, and letting me know if serve webdav works for you? The APK is based off of the master branch. The versionCode is the same as current beta and stable release, which means that you can easily update to current stable or beta after you're done testing. Thanks.

1.6.0-serve-webdev-bug-arm64-α.apk

[xiaolei0125]

@kaczmarkiewiczp Thanks.
I have tested 1.6.0-serve-webdev-bug-arm64-α.apk
Run serve webdav on RcloneExplorer:
Syncing directory(single or multiple files) using FolderSync are work fine.
But It's failed to uploading single or multiple files using ES Explorer and Solid Explorer.

It's seem very strange, I have no idea about this.

By the way:
copy a rclone that's compiled for Linux/ARM64 to /data/data/com.termux/files/usr/bin/ on android, need android is rooted, since linux-arm64 rclone on android need /etc/resolv.conf to work, so needing to copy resolv.conf to /system/etc/ on android before running rclone.

[xiaolei0125]

@kaczmarkiewiczp
I have tested 1.6.0-serve-webdev-bug-arm64-α.apk and old version with another remote AWS S3, it's seem all work fine with ES Explorer, Solid Explorer and FolderSync .

But it's still failed when remote is sharepoint(webdav) or JianGuoYun which is a cloud storage supported webdav.

Even if used sharepoint or JianGuoYun as a remote, ES Explorer, Solid Explorer and FolderSync are work fine when running rclone serve webdav on termux, windows or linux.

[xiaolei0125]

@kaczmarkiewiczp

I think the different between S3(S3 API compatible) and sharepoint(JianGuoYun) is the
file uploading method.

When using S3:
For small file:

DEBUG : S3 bucket xstor: File to upload is small (58977 bytes), uploading instead of streaming

For big file:

DEBUG : IMG_20160117_122457.jpg: Size and modification time the same (differ by 0s, within tolerance 1ns)
NFO  : /IMG_20160117_122457.jpg: PUT from 192.168.0.101:40921

When using sharepoint(JianGuoYun) webdav:
For small file:

DEBUG : webdav root '': File to upload is small (41264 bytes), uploading instead of streaming

For big file:

DEBUG : webdav root '': Target remote doesn't support streaming uploads, creating temporary local FS to spool file
DEBUG : test/IMG_20160117_122457.jpg: Sizes identical

I guess if rclone in rcloneExplorer supported to creating temporary local FS to spool file?

[xiaolei0125]

@kaczmarkiewiczp
I also test rcloneExplorer serve webdav using sharepoint(JianGuoYun) webdav as remote, and using ES Explorer to testing:
uploading small file(50k) will success, uploading big file(4M) will failed.

So this issue seems to be related to the file size which cause a different upload method for sharepoint(JianGuoYun) webdav remote.

I have not noticed this file size problem before, so the previous test results may be weird and wrong.

[patrykcoding]

I think the different between S3(S3 API compatible) and sharepoint(JianGuoYun) is the
file uploading method.

That could explain the bug you're experiencing. If rclone needs to create temporary files, it probably tries to create those in places it doesn't have access to on Android (that was the case with the cache remote). I tried to find a flag in rclone that would allow specifying where those temporary files are to be stored and found this:

--cache-tmp-upload-path string Directory to keep temporary files until they are uploaded to the cloud storage

I created a release that uses that flag in order to store temporary files in rcloneExplorer's cache directory. I also added logging to the serve webdav function. Can you enable logging in settings (under Miscellaneous > Use Logs). Run serve webdav, try to upload a file (either using a file explorer, or FolderSync), and then cancel serve webdav if it fails. After quitting serve webdav, the log will be located on the internal memory under Android/data/ca.pkay.rcexplorer/files/logs/log.txt. Hopefully it will reveal some extra information.

RcloneExplorer-serve-webdav-bug-alpha2.apk
RcloneExplorer-serve-webdav-bug-arm64-alpha2.apk
RcloneExplorer-serve-webdav-bug-arm32-alpha2.apk
RcloneExplorer-serve-webdav-bug-x86-alpha2.apk

[xiaolei0125]

@kaczmarkiewiczp
I have intalled and test for RcloneExplorer-serve-webdav-bug-arm64-alpha2.apk.
First I uploading a small file(mmexport1513758527117.jpg, 50k), it's success.
Second uploading a big file(IMG_20160117_122457.jpg, 3M), it's failed.

It seems to prove that my guess is correct.

The key log is:

2018/08/09 02:07:23 DEBUG : /test/IMG_20160117_122457.jpg: >OpenFile: err = <nil>
2018/08/09 02:07:23 DEBUG : webdav root 'Work-IT': Target remote doesn't support streaming uploads, creating temporary local FS to spool file
2018/08/09 02:07:23 ERROR : test/IMG_20160117_122457.jpg: WriteFileHandle.New Rcat failed: Failed to create temporary local FS to spool file: mkdir /data/data/com.termux/files/usr/tmp/rclone-spool516087129: permission denied
2018/08/09 02:07:23 ERROR : test/IMG_20160117_122457.jpg: WriteFileHandle.Write error: io: read/write on closed pipe

The detail log is: log.txt

[patrykcoding]

Sorry for the late response, I was away for a few days. And thanks for testing. So the error is caused by rclone when it tries to create temporary files in a directory it has no access to on Android. Since I'm not aware of any way to tell rclone to use a custom temporary path, I'll have to ask on the rclone forum page. I'll keep you updated.

[xiaolei0125]

@kaczmarkiewiczp Thank you for fix it.

I have test for the newest release (v1.7.1) arm64 apk, using ES Explorer or FolderSync, upload small or big file(>5M) to JianGuoYun(webdav server), all are work fine.

[patrykcoding]

Thanks for testing. I merged the fix into the master branch. It's available in the latest beta release and will be available in the future stable releases.