dansan/zrsync

Continuously synchronize directories from a client to a server.

ZRsync

Continuously synchronize directories from a client to a server with emphasis on low latency (not on large files).

Permissions and timestamps are also copied, so rsync -avn --delete /source /target should always result in "nothing to do" (except during synchronization).

Uses inotify to monitor directories for changes, librsync to reduce bandwidth usage and zeromq (+ssh) for the transport.

Development is done using Python 2.7 and ZeroMQ 4.0. It may or may not run with other versions. Latest source code can be found on Github.

Installation

A shell script is provided for easy installation. Currently only Debian and UCS are supported. Please send me patches to include instructions for your distribution.

$ wget https://raw.githubusercontent.com/dansan/zrsync/master/zrsync_install
$ bash zrsync_install

This will

• install required packages supported by the Linux distribution
• download the zrsync code from github to ~/.zrsync/zrsync
• create a Python virtual environment in ~/.zrsync/virtenv
• install remaining requirements into the virtual environment
• create shortcuts for running the client and server in ~/bin

Running

(on target system) $ ~/bin/zrsyncd
(on source system) $ ~/bin/zrsync host /my/local/dir /remote/dir

The order in which client and server are started does not matter.

---

Continuously sync ~/Documents to /tmp/mydocs on the same system. Zrsyncd must already be running:

$ ~/bin/zrsync localhost ~/Documents /tmp/mydocs

Continuously sync ~/Documents to ~/Documents of user foo on my.ser.ver using a SSH tunnel:

$ ~/bin/zrsync [email protected] ~/Documents Documents

The same, but it will try to start zrsyncd on my.ser.ver, if it is not yet running and install it if is not already installed. When the client quits, it will shut down zrsyncd on the target system:

$ ~/bin/zrsync -a [email protected] ~/Documents Documents

Sync ~/Documents to ~/Documents of user foo on my.ser.ver using a SSH tunnel once and then stop zrsync client and server. This is what rsync does:

$ ~/bin/zrsync -it [email protected] ~/Documents Documents

Continuously sync three directories, logging in as user foo on my.ser.ver using a SSH: ~/Documents to ~/Documents, /etc to /tmp/etc and ~/git to /tmp/stuff.

$ ~/bin/zrsync [email protected] ~/Documents Documents /etc /tmp/etc ~/git /tmp/stuff

---

See --help for more options:

$ ~/bin/zrsync -h
zrsync client: continuously synchronize a local directory to a [remote] directory.

Usage:
  zrsync [-ahilnstqvp PORT] <server> <source-dir> <target-dir> [<source-dir> <target-dir>]...

Arguments:
  server      server to sync to: [user@]server[:port]
              ('user@' and ':port' automatically enable --ssh)
  source-dir  local directory to sync from
  target      directory on the server to sync to
              (not starting with '/' means relative to home of user running zrsyncd)

Options:
  -a --auto              enable -lrt (--install, --start and --shutdown)
  -h --help              show this help message and exit
  -i --initial-only      quit after the initial sync
  -l --install           try to install zrsync on target (implies --ssh)
  -n --no-delete         do not delete any files or diretories on the target
  -p PORT --port=PORT    zrsyncd (not ssh) port to connect to [default: 24240]
  -r --start             start server (zrsyncd) on target if not already running (implies --ssh)
  -q --quiet             print only errors
  -s --ssh               tunnel connection through SSH.
  -t --shutdown          shutdown server when finished
  -v --verbose           print debug statements
  --version              show version and exit

---

$ ~/bin/zrsyncd -h
zrsync server: receive continuous updates for local directories from zrsync.

Usage:
  zrsyncd.py [-hqv] [-i IP | --ip=IP] [-p PORT | --port=PORT]

Options:
  -h --help              show this help message and exit
  -i IP --ip=IP          IP to listen on [default: *]
  -p PORT --port=PORT    port to listen on [default: 24240]
  -q --quiet             print only errors
  -v --verbose           print debug statements
  --version              show version and exit

License

This software is licensed under the terms of the GNU General Public License v3, see LICENSE.

• ZeroMQ is licensed under the terms of the GNU Lesser General Public License v3.
• PyZMQ is licensed under the terms of the Modified BSD License as well as the GNU Lesser General Public License v3.
• librsync is licensed under the terms of the GNU Lesser General Public License v2.1.
• python-librsync is licensed under the terms of the MIT license.
• docopt is licensed under the terms of the MIT license.
• pyinotify is licensed under the terms of the GNU General Public License v2.
• pexpect is licensed under the terms of the ISC LICENSE.

zrsync's Issues

path mangling

• Create clear definition of what paths on source and target match (does / at the end matter?).
• Resolve consistently ~ and other relative forms.

support remote server installation

-l --install try to install zrsync on target. Automatically enables --ssh.

Python 3 compatibility

zrsync should run on both Python 2 and Python 3

chunk big transfers

Investigate if splitting big "full" (and "diff"?) transfers makes sense.

support SSH tunneling

-s --ssh tunnel connection through SSH. Assumes prior setup of password-less SSH login.

server terminates if user cannot chown

When zrsyncd is run as non-root user, and the UID of a source file/dir differs, it tries a chmod, fails, displays a traceback and quits:

2016-08-19 01:30:52 zrsyncd.cmp_node:504  DEBUG [24971] does not exist: u'/tmp/mytarget'
2016-08-19 01:30:52 zrsyncd.mkdir:627  DEBUG [24971] mkdir(u'/tmp/mytarget'); chown(); utime()
2016-08-19 01:30:52 zrsyncd.chown:623  DEBUG [24971] chown(u'/tmp/mytarget', 2311, 5001)
Process sync_server:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/daniel/.zrsync/zrsync/zrsync/zrsyncd.py", line 699, in run
    reply["tree_diff"] = self.sync_tree(src_tree, request["no_delete"])
  File "/home/daniel/.zrsync/zrsync/zrsync/zrsyncd.py", line 475, in sync_tree
    res = self.cmp_node(node)
  File "/home/daniel/.zrsync/zrsync/zrsync/zrsyncd.py", line 513, in cmp_node
    node["st_atime"], node["st_mtime"])
  File "/home/daniel/.zrsync/zrsync/zrsync/zrsyncd.py", line 629, in mkdir
    self.chown(dirname, uid, gid)
  File "/home/daniel/.zrsync/zrsync/zrsync/zrsyncd.py", line 624, in chown
    os.lchown(name, uid, gid)
OSError: [Errno 1] Operation not permitted: '/tmp/mytarget'

zrsyncd should instead send a note back to the client, which should display a warning.