gilgalaad/duplicity-rclone

Duplicity backend using rclone

duplicity-rclone

Duplicity backend using rclone

Rclone is a powerful command line program to sync files and directories to and from various cloud storage providers.

At the time of deveopment, I was using duplicity v0.7.10 and Amazon Cloud Drive, but this backend should work with any storage provider supported by rclone, and with later version of duplicity.
I am currently running it with duplicity v0.8.12 and rclone v1.51.0, without issues.

Since I've never wrote Python before today, use it at your own risk. Every feedback, comment or suggestion is welcome.

Update 23/03/2020: This project has been merged into official duplicity codebase, since version 0.8.09, so basically this will remain only for people still running version 0.7.x on older distributions.

Update 12/05/2020: After rclone released some new dedicated commands to handle single files, I modified the backend currently shipped with duplicity 0.8.x, this code is a lot more efficient in dealing with the storage provider, and fixes a small bug. I hope this code will be merged into the official repository too. I will keep the 0.7.x branch for whoever should still need it.

Setup

Install rclonebackend.py into duplicity backend directory. This backend provides support for prefix rclone://.

Usage

Once you have configured rclone and successfully set up a remote (e.g. gdrive for Google Drive), assuming you can list your remote files with

rclone ls gdrive:mydocuments

you can start your backup with

duplicity /mydocuments rclone://gdrive:/mydocuments

Please note the slash after the second colon. Some storage provider will work with or without slash after colon, but some other will not. Since duplicity will complain about malformed URL if a slash is not present, always put it after the colon, and the backend will handle it for you.

duplicity-rclone's Issues

Integration into duplicity

Hi,

Kenneth Loafman here. I'm the maintainer of duplicity backup. I'm working on integrating your code into duplicity. Since duplicity is GPL v2, and rclonebackend.py is v3, I'd like your permission to release it as v2 instead of v3.

...Thanks,
...Ken

I've made a updated version for Python3

Thanks for your original work.
I've updated it for Python3 as now Duplicity is Python3
See:
https://github.com/jradxl/duplicity-rclone

please provide config for rclone.conf path

Hello,
Would it be easy to provide a way to pass the backend a non-default location for the rclone config file?
This backend is making duplicity work soooo much better!
Thanks!
C.

missing argument to handle 1Fichier hosting

Hi,

I've been using the latest duplicity version with rclone backend recently but I kept having troubles with 1Fichier API flagging my connection as flooding.

I solved it by adding arguments to rclone : --transfers=1 --multi-thread-streams=1 to line declaring self.rclone_cmd.

I think it should be considered to either add those arguments all the time or to add a new argument to duplicity to enable them.

Have a nice day and thanks a lot for this project!

Can't copy files on python3

duplicity 0.8.12
Args: /usr/local/bin/duplicity -v 5 --full-if-older-than 1M --allow-source-mismatch --no-encryption /app rclone://mageddohd03:/tmp2/
Linux 84313b1702fe 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:16:15 UTC 2018 x86_64 
/usr/local/bin/python3 3.7.7 (default, Mar 11 2020, 01:01:39) 
[GCC 9.2.0]
================================================================================
Using temporary directory /tmp/duplicity-gyybo6el-tempdir
Temp has 64052236288 available, backup will use approx 272629760.
Executing subprocess: 'rclone ls mageddohd03:tmp2/'
Attempt 1 failed. TypeError: a bytes-like object is required, not 'str'
^CINT intercepted...exiting.


/app # rclone --version
rclone v1.51.0
- os/arch: linux/amd64
- go version: go1.13.7

note about default python from EL7

Hi,
I've found that the urlparse.py from python-libs 2.7.5 (Redhat 7 / CentOS 7) gives the following error :
InvalidBackendURL: Syntax error (port) in: rclone://gdrive-backups4_o:/backups/beck.hep.wisc.edu_opt/duplicity AFalse BNone Cgdrive-backups4_o
when the URL looks like this 'rclone://gdrive-backups4_o:/backups/beck.hep.wisc.edu_opt/duplicity'.

Replacing urlparse.py from that included in python 2.7.13 is sufficient to work around the problem.

I find the rclone backend so much more reliable and easier to set up than those included with duplicity. Thanks!

Script does not work with hubic

Hi,
I am trying to use your script with hubic (OVH cloud storage), I think it almost work, the only issue I have is that I normally access the bucket like this

rclone ls hubic:default/backup/

and this does not work:

rclone ls hubic:/default/backup/

consequently, I need to do that on duplicity:

duplicity list-current-files rclone://hubic:default/backup/

which breaks because duplicity thinks that the bucket name (default) is the port number...

My workaround is to change the script like this:

if parsed_url.path.startswith('//'):
                    self.remote_path = self.remote_path[2:].replace('/',':',1)

which allows me do do this:

duplicity list-current-files rclone://hubic/default/backup/

and the first / is understood as :