kumina/awssyncer

Automatic syncer of local directories into Amazon AWS S3

awssyncer's Issues

Merge FILTER_REGEX and SYNC_EXCLUDES

Right now we both have a separate FILTER_REGEX parameter that is used by inotify and SYNC_EXCLUDES that is passed to the aws command line tool. It would be nice if both of these could be merged.

Maybe a solution is to never invoke aws s3 sync and only stick to syncing individual files.

Somehow preserve S3 metadata across calls

The awssyncer will inherently need to do a full synchronisation every couple of hours. The reason for this is that inotify may easily miss events in our case. For example, in case files are hardlinked, we'll only receive a single event for one of the paths.

Still, there is room for improving the number of requests we send over to S3. If we would cache the state of S3 across calls, we'd know exactly what to sync and what can be skipped.