wnyc/autosync

An extensible framework to synchronize a local directory to a remote location. Will support S3, Rackspace and generic http based workflows

autosync's Issues

Support moves

Inotify can detect when a file is renamed. Many backends support file rename or copy operations. Support file moving to prevent unnessesary uploads

Custom backend/server

Through inotify autosync has specific knowledge about which ranges have been changed within a file. Create an rsync or custom transfer service that can accept byte ranges.

FTP backend

Create an FTP backend

Retry on actor error

When the actor is unable to upload a file due to a recoverable error at its end, reschedule the operation.

Rsync backend

Create an rsync backend

Add ability to filter on regex

[2013-02-22 15:25:53,126 pyinotify ERROR] add_watch: cannot watch /var/www/ia-migration/js WD=-1, Errno=No space left on device (ENOSPC)

We need to have a limited capacity for things to watch. Eww.

Freenet backend

Create a backend that writes into freenet.

https://freenetproject.org/

Google drive backend

Create a google drive backend

Remove print statements

Replace print statements with logging

Start inotification before fs scan

inotify does not catch new directory creation

Creating and filling a new directory does not trigger an upload.

autosync --target_container=project2.wnyc.net --source_prefix=/var/www /var/www

mkdir /var/www/foobar2
echo hello world > /var/www/foobar2/index.html

index.html does not get uploaded.

Deduplication

Backends that support md5 are able to detect if a file already exists under another name.

• Support the detection of duplicate files in the target and the use of move instead.
• Provide a flag to disable this

add_watch leaks

When a new directory is created we add a watch, but watches are not later removed when directories are removed. Check if this is a problem and if it is, catch deletions to remove watches.

PEP-8

Allow actors to double check flags options

The actor interface should support a "validate_flags" option that will example the flag's it depends upon to make sure they are "sane." This is especially important for actors that use common flags such as --source-prefix and have requirements about path strings such as a mandatory leading /

Support both gevent and thread