seanh/snapshotter

Very easy incremental snapshot backups using rsync

Hi 👋, I'm Sean

I'm a developer and future Portuguese Water Dog owner currently working for Hypothesis and living in Edinburgh. You can find my blog posts and some of my projects on my site. You can also find me on GitHub Gist, Stack Overflow and Pinboard. If you want to get in touch you can send me an email.

Recent Posts

• How I Use Bitwarden
• How to Make Your Own Encrypted Backup of Your Bitwarden Vault
• How to authenticate a GitHub Actions workflow as a GitHub App so it can trigger other workflows
• My guide to todo.txt

snapshotter's Issues

Bug with destination paths with spaces in them

If the destination path has a space in it, the text before the space seems to get removed when the path is passed to the rsync command. It works if you wrap the path in '...' in zsh though.

Break it up into separate files

I'm thinking cli.py, unix.py and logic.py, each with its own test module.

Support an option for the maximum space to be used by all snapshots

Run du to figure out how much space all the snapshots use, if it's above a given value delete as many of the oldest snapshots as necessary.

Might also be nice to have a snapshotter du DEST command, print out the actual space used by each *.snapshot dir in DEST, taking hardlinks into account.

Add a unix manpage

Ideally generated from the same source as the github readme

Support Python 2.6

subprocess.check_output() doesn't exist on 2.6.

Source and destination args not treated the same

It seems to pass rsync a SRC arg like this: 'Mail/' (relative path wrapped in single quotes) but a DST arg like this: /home/seanh/Mail.snapshots (absolute path, not wrapped in quotes). Surely they should be treated the same

Use os to move and remove files and make symlinks

Use Python's standard os module's functions to move and remove files and make symlinks, instead of running mv, rm and ln as subprocesses.

When dealing with a remote destination I think we'll still have to run ssh [user@]host mv|rm|ln ... commands as subprocesses.

Use ln -f instead of rm

No need to do rm and then ln, can just do ln -f.

Ignore rsync's "some files vanished before they could be transferred (code 24)" error

rsync warning: some files vanished before they could be transferred (code 24) at main.c(1183) [sender=3.1.0]

Don't let this stop snapshotter from doing its mv and ln -s commands

Removing old snapshots fails if you don't have permission to write to the files

The permissions of files in the backups get copied from the permissions of the source files. It can try and chmod -R ug+w ... the snapshot dir before deleting it. If that fails I don't think there's anything that can be done.

Add a --max-snapshots option (deletes any snapshots older than --max-snapshots)

Hello.

I see option --min-snapshots, but how can I set max-snapshots or control how many snapshots can be on remote drive?

Add support for Python 2.6, 3.0, 3.1, 3.2 and 3.3

Update the README

Different port ?

Hi,

is there a chance I could make script connect to server on non-default SSH port ?

ty.

Add --version

Work with multiple source directories and one destination dir

rsync can do this, so backup should be able to as well

`AttributeError` from `snapshotter -h`

~ snapshotter -h
usage: snapshotter [-h] [-n] [--min-snapshots MIN_SNAPSHOTS] SRC DEST

positional arguments:
  SRC                   the path to be backed up
  DEST                  the directory to create snapshots in

optional arguments:
  -h, --help            show this help message and exit
  -n, --dry-run         Perform a trial-run with no changes made (pass the
                        --dry-run option to rsync)
  --min-snapshots MIN_SNAPSHOTS
                        The minimum number of snapshots to leave behind when
                        rolling out old snapshots to make space for new ones
                        (default: 3)
Traceback (most recent call last):
  File "/usr/local/bin/snapshotter", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/snapshotter/snapshotter.py", line 446, in main
    sys.exit(err.output)
AttributeError: 'CommandLineArgumentsError' object has no attribute 'output'

Don't get into a --max-snapshots infinite loop

When --max-snapshots is used and _remove_oldest_snapshot() doesn't actually reduce the number of snapshots reported to _ls_snapshots() (for example when --dry-run is used so that we don't actually run the rm command) then snapshotter gets into an infinite loop as it keeps calling _remove_oldest_snapshot() until _ls_snapshots() reports less than --max-snapshots snapshots, but _ls_snapshots() always reports the same number.

Email on successful/error backup

Hi Sean,

I know crontab can email me about tasks but something more "advanced" would be nice in example:

"Successfully backed up Webserver1" or "Backup of Database1 did not finish with error: not enough disk space on device"

Where Webserver1 and Database1 is actually root@Webserver1 of snapshotter line
A.

Non-standard SSH Port ?

Hi,
Is there any command line to sync to remote server that using a non-standard ssh port? Or is not possible for this version?

Thanks.

Edit:
I tried this command :
snapshotter --rsh='ssh -p 1002' user@host:/path/to/source /path/to/backup

but it keep making trailing / at the end, throwing error:
Bad port '1002/'

Add a how-to for creating an encrypted filesystem to backup to

Add prospector integration

Or some lint tool anyway

Use pytest and tox instead of nose

Don't ask for password multiple times when using remote locations

For now users can get around this by using ssh keys instead of passwords to access remote hosts.

Add tests for main()

Found some bugs in it.

Automatically delete old snapshots

If there isn't enough space on the destination volume, automatically delete old snapshots to make enough space.

This is quite complicated:

• Need to find out how much space the new snapshot requires
• Need to find out how much space is available on the destination volume (accounting for hard links)
• Need to find out how much space will be freed by deleting each existing snapshot (accounting for hard links)
• Need to keep at least one (or a configurable number) of existing snapshots, if enough space can't be freed without leaving at least N existing snapshots intact, then fail

Create target directories if they don't exist

Use codecov instead of coveralls

Pass arbitrary command-line options onto rsync

Any arbitrary command-line options not recognised by backup should just be passed into rsync unexamined.

Show progress as snapshots are made

Need to find a way to run a subprocess and gets its exit value, stdout and stderr, while also echoing the stdout and stderr to our stdout and stderr in real-time.

Get rid of print statements, use logging instead

Add configuration file

rsnapshot has a .conf file where one can define paths to backup, snapshotter seems to lack this.

While writing an ansible recipe I realized that I had to write paths in ansible's group variables, which does not allow me to decouple deployment from backup specs :/

@guillermo-carrasco

Publish to PyPI, add pypi.in badges

Attempting to back up to a remote destination generates exception

Trying to execute a snapshot to a remote destination fails with the following exception:

./.local/bin/snapshotter --dry-run /home/glagasse [email protected]:/tank/timemachine/praetorian Traceback (most recent call last): File "./.local/bin/snapshotter", line 9, in <module> load_entry_point('snapshotter==1.0.3', 'console_scripts', 'snapshotter')() File "/home/glagasse/.local/lib/python2.7/site-packages/snapshotter/snapshotter.py", line 480, in main snapshot(*_parse_cli()) File "/home/glagasse/.local/lib/python2.7/site-packages/snapshotter/snapshotter.py", line 400, in snapshot while len(_ls_snapshots(dest)) >= max_snapshots: File "/home/glagasse/.local/lib/python2.7/site-packages/snapshotter/snapshotter.py", line 303, in _ls_snapshots snapshots = [os.path.join(dest, d) for d in os.listdir(dest) OSError: [Errno 2] No such file or directory: '[email protected]:/tank/timemachine/praetorian'

This happens because snapshotter is trying to find the number of remote snapshots in the remote destination using the host machine.
Lines 400-402 is where _ls_snapshots(dest) is called and inside _ls_snapshots it is trying to do a listdir (on the host machine) of the destination which naturally won't exist.

Don't use shell=True

Convert all paths to absolute paths with ~ expanded etc. Run rsync and other executables directly, not via shell.

Fix handling of spaces in remote and local source and destination paths

I think source and destination paths may be treated differently, and remote and local paths are certainly treated differently. For example os.path.abspath(os.path.expanduser(path)) is called on local paths but not on remote ones, and this seems to change \ to \\.

This makes a difference when paths have spaces in them - whether you need to backslash-escape the spaces, quote the path, or both. Currently what you need to do depends on whether it's a local or remote path.

Multiple --link-dest arguments?

Currently files in a new snapshot will only be hard-linked to existing files in the latest snapshot. Find all existing snapshot directories in the target directory and pass rsync a --link-dest argument for each of them.

Add a github pages site

Ideally with a front page generated from the same source as the readme

Handle "no space left on device" by deleting old snapshots

It might be better to figure out before running rsync how much space the new snapshot will need. But failing that: detect "no space left on device" errors from rsync, delete the oldest snapshot, repeat. Fail if the number of remaining snapshots reaches a threshold.