cytopia/linux-timemachine

Rsync-based OSX-like time machine for Linux, MacOS and BSD for atomic and resumable local and remote backups

Linux Time Machine

Install | Uninstall | TL;DR | Features | How does it work | Restore | Retention | Usage | FAQ | Disclaimer | License

timemachine is a tiny and stable KISS driven and POSIX compliant script that mimics the behavior of OSX's timemachine. It uses rsync to incrementally back up your data to a different directory, hard disk or remote server via SSH. All operations are incremental, atomic and automatically resumable.

By default it uses the rsync options: --recursive, --perms, --owner, --group, --times and --links. In case your target filesystem does not support any of those options or you cannot use them due to missing permission, you can explicitly disable them via --no-perms, --no-owner, --no-group, --no-times, and --copy-links. See FAQ for examples.

Motivation

The goal of this project is to have a cross-operating system and minimal as possible backup script that can be easily reviewed by anyone without great effort. Additionally it should provide one task only and do it well without the need of external requirements and only rely on default installed tools.

🎉 Install

sudo make install

🚫 Uninstall

sudo make uninstall

☕ TL;DR

Using POSIX.1-2008 argument syntax:

# Recursive, incremental and atomic backup (locally)
$ timemachine /source/dir /target/dir

# Recursive, incremental and atomic backup (via ssh)
$ timemachine /source/dir user@host:target/dir

# Recursive, incremental and atomic backup (via ssh with non-standard port)
$ timemachine --port 10000 /source/dir user@host:target/dir

# Append rsync options
$ timemachine /source/dir /target/dir -- --specials --progress
$ timemachine /source/dir /target/dir -- --specials --no-perms
$ timemachine /source/dir /target/dir -- --archive --progress

# Make the timemachine script be more verbose
$ timemachine -v /source/dir /target/dir
$ timemachine --verbose /source/dir /target/dir

# Make the timemachine script be even more verbose
$ timemachine -d /source/dir /target/dir
$ timemachine --debug /source/dir /target/dir

# Make the timemachine script and rsync more verbose
$ timemachine -v /source/dir /target/dir -- --verbose
$ timemachine --verbose /source/dir /target/dir -- --verbose

⭐ Features

Feature	Description
SSH or local	Local backups as well as backups via SSH are supported.
Incremental	Backups are always done incrementally using rsync's ability to hardlink to previous backup directories. You can nevertheless always see the full backup on the file system of any incrementally made backup without having to generate it. This will also be true when deleting any of the previously created backup directories. See the Backups section for how this is achieved via rsync. Incremental Backups also mean that only the changes on your source, compared to what is already on the target, have to be backed up. This will save you time as well as disk space on the target disk.
Partial	When backing up, files are transmitted partially, so in case a 2GB movie file backup is interrupted the next run will pick up exactly where it left off at that file and will not start to copy it from scratch.
Resumable	Not only is this script keeping partial files, but also the whole backup run is also resumable. Whenever there is an unfinished backup and you start timemachine again, it will automatically resume it. It will resume any previously failed backup as long as it finally succeeds.
Atomic [1]	The whole backup procedure is atomic. Only if and when the backup procedure succeeds, it is then properly named and symlinked. Any non-successful backup directory is either waiting to be resumed or to be deleted.

• ^[1] The backup process is atomic, but not the backup itself. rsync copies files as it finds them and in the meantime there could already be changes on the source. To achieve an atomic backup, either back up from a read-only volume or from a snapshot.

ℹ️ How does it work?

Directory structure

The following directory structure will be created:

$ tree -L 1 /my/backup/folder
.
├── 2018-01-06__18-43-30/
├── 2018-01-06__18-44-23/
├── 2018-01-06__18-50-44/
├── 2018-01-06__18-50-52/
└── current -> 2018-01-06__18-50-52/

current will always link to the latest created backup. All backups are incremental except the first created one. You can nevertheless safely remove all previous folders and the remaining folders will still have all of their content.

Backup strategy

Except for the first one, backups are always and automatically done incrementally, so the least amount of space is consumed. Due to rsync's ability, every directory will still contain all files, even though they are just incremental backups. This is archived via hardlinks.

$ du -hd1 .
497M    ./2018-01-06__18-43-30
24K     ./2018-01-06__18-44-23
24K     ./2018-01-06__18-50-44
24K     ./2018-01-06__18-50-52
497M    .

You can also safely delete the initial full backup directory without having to worry about losing any of your full backup data:

$ rm -rf ./2018-01-06__18-43-30
$ du -hd1 .
497M    ./2018-01-06__18-44-23
24K     ./2018-01-06__18-50-44
24K     ./2018-01-06__18-50-52
497M    .

rsync and hardlinks are magic :-)

Failure handling and resume

In case the timemachine script aborts (self-triggered, disk unavailable or for any other reason) you can simply run it again to automatically resume the last failed run.

This is due to the fact that the backup process is atomic. During a non-complete run, all data will be stored in a directory named .inprogress/. This will hold all already transferred data and will be picked up during the next run. Once the backup is complete, it will be renamed and symlinked to current.

$ tree -a -L 1 /my/backup/folder
.
├── .inprogress/
├── 2018-01-06__18-43-30/
├── 2018-01-06__18-44-23/
├── 2018-01-06__18-50-44/
├── 2018-01-06__18-50-52/
└── current -> 2018-01-06__18-50-52/

♻️ Restore

No special software is required to restore your data. Backed up files can be easily browsed and thus copied back to where you need them. Recall the backup directory structure:

$ tree -L 1 /my/backup/folder
.
├── 2018-01-06__18-43-30/
├── 2018-01-06__18-44-23/
├── 2018-01-06__18-50-44/
├── 2018-01-06__18-50-52/
└── current -> 2018-01-06__18-50-52/

Chose a backup directory and simply copy them to where you need it:

# Test it out in dry run mode before applying
$ rsync --archive --progress --dry-run /my/backup/folder/2018-01-06__18-50-52/ /src/

# Apply restoration
$ rsync --archive --progress /my/backup/folder/2018-01-06__18-50-52/ /src/

🔁 Retention

As decribed above this project is KISS driven and only tries to do one job: back up your data.

Retention is a delicate topic as you want to be sure that data is removed as intended. For this there are already well-established tools that do an excellent job and have proven themselves over time: tmpreaper and tmpwatch.

🔒 Reliability

The script is written and maintained with maximum care. In order to retain a reliable and stable backup solution, a lot of effort goes into a vast amount of integration and regression tests. These tests not only give you measurable confidence, but also help new contributors to not accidentally introduce new or old bugs.

💻 Usage

Available options

$ timemachine -h

Usage: timemachine [-vd]   <source> <dest> -- [rsync opts]

       timemachine [-vdpi] <source> <host>:<dest>        -- [rsync opts]
       timemachine [-vdpi] <source> <user>@<host>:<dest> -- [rsync opts]
       timemachine [-vdpi] <source> <ssh-alias>:<dest>   -- [rsync opts]

       timemachine [-vdpi] <host>:<source>        <dest> -- [rsync opts]
       timemachine [-vdpi] <user>@<host>:<source> <dest> -- [rsync opts]
       timemachine [-vdpi] <ssh-alias>:<source>   <dest> -- [rsync opts]

       timemachine -V, --version
       timemachine -h, --help

This shell script mimics the behavior of OSX's timemachine.
It uses rsync to incrementally back up your data to a different directory or remote server via SSH.
All operations are incremental, atomic and automatically resumable.

By default it uses --recursive --perms --owner --group --times --links.
In case your target filesystem does not support any of those options, you can explicitly
disable those options via --no-perms --no-owner --no-group --no-times and --copy-links.

Required arguments:
  <source>              Local source directory
  <dest>                Local destination directory.
  <host>:<dest>         SSH host and source/destination directory on server
  <user>@<host>:<dest>  SSH user, SSH host and source/destination directory on server
  <ssh-alias>:<dest>    SSH alias (defined in ~/.ssh/config) and source/destination directory on server

Options:
  -p, --port            Specify alternative SSH port for remote backups if it is not 22.
  -i, --identity        Specify path to SSH key.
  -v, --verbose         Be verbose.
  -d, --debug           Be even more verbose.

Misc Options:
  -V, --version         Print version information and exit
  -h, --help            Show this help screen

Examples:
  Simply back up one directory recursively
      timemachine /home/user /data
  Do the same, but be verbose
      timemachine -v /home/user /data
  Append rsync options and be very verbose
      timemachine -d /home/user /data -- --progress --verbose
  Log to file
      timemachine -v /home/user /data > /var/log/timemachine.log 2> /var/log/timemachine.err

Documentation:
  View more examples at: https://github.com/cytopia/linux-timemachine

Use with cron

The following can be used as an example crontab entry. It assumes you have an external disk (NFS, USB, etc..) that mounts at /backup. Before adding the crontab entry, ensure the filesystem in /backup is mounted and use:

$ touch /backup/mounted

This guards against accidentally backing up to an unmounted directory

Next, add the following to crontab using crontab -e as whichever user you intend to run the backup script as. You may need to place this in the root crontab if you are backing up sensitive files that only root can read

0 2 * * * if [[ -e /backup/mounted ]]; then /usr/local/bin/timemachine /home/someuser /backup; fi

This will cause timemachine to run at 2AM once per day. Since timemachine keeps track of backups with granularity up to the hour, minute and second, you could have it run more than once per day if you want backups to run more often.

💡 FAQ

Should I add trailing directory slashes (/)?

Trailing directory slashes only matter for the source directory and will not make a difference if added to the destination directory.

# The following backs up the contents of the src directory
$ timemachine src/ dst/
$ tree -L 2 /dst
.
├── 2018-01-06__18-43-30/
│   └── file.txt
└── current -> 2018-01-06__18-43-30/

# The following backs up the the src directory itself
$ timemachine src dst/
$ tree -L 3 /dst
.
├── 2018-01-06__18-43-30/
│   └── src
│       └── file.txt
└── current -> 2018-01-06__18-43-30/

How to dry-run the backup?

$ timemachine src/ dst/ -- --dry-run

How to use a non-standard SSH port?

$ timemachine --port 1337 src/ user@host:dst/

How to use SSH aliases from ~/.ssh/config?

$ cat ~/.ssh/config
Host my-ssh-alias
  HostName     192.168.0.1
  Port         1234
  User         john
  IdentityFile ~/.ssh/id_rsa__alternative_key

$ timemachine src/ my-ssh-alias:dst/

How to speed up remote backups?

# With  this option, rsync compresses the file data as it is sent to the des‐
# tination machine, which reduces the amount of  data  being  transmitted
$ timemachine src/ user@host:dst/ --compress

How to preserve ACLs?

$ timemachine src/ dst/ -- --acls

How to preserve extended attributes?

$ timemachine src/ dst/ -- --xattrs

How to disable preserving file and directory permissions?

$ timemachine src/ dst/ -- --no-owner --no-perms

How to disable preserving modification time?

$ timemachine src/ dst/ -- --no-times

How to copy the content instead of a symlink?

# This is useful in case your file system does not support symlinks.
# It is recommended to read rsync man page about symlinks to be sure
# about what you are doing
$ timemachine src/ dst/ -- --copy-links --safe-links
$ timemachine src/ dst/ -- --copy-links --safe-links --keep-dirlinks

How to ensure all files in the back up have the ownership of current user?

# Regardless of who owns the files, ensure the backup has uid and gid of current user
# This will only work if you have read-permission on all files.
$ timemachine src/ dst/ -- --no-owner --no-group

# If you do not have permission to read all files, you require sudo or root permission.
# The following will instruct rsync to ensure the backed up data has the uid and gid
# of the desired user.
$ sudo timemachine src/ dst/ -- --chown=<user>:<group>

❗ Disclaimer

Backups are one of the most important things. We all care about our data and want it to be safe, so do not blindly trust scripts when it comes to backups! Do review the code, it is not too complex and kept as short as possible. Have a look at how much effort goes into the integration tests to provide measurable stability.

Learn about rsync it is a very powerful tool and you might even be able to just use this for backups.

There are many other backup solutions out there that might be a better fit for your needs. Do your own research, look at GitHub issues, source code, integration tests and try them out as well.

Contributing

See Contributing guidelines to help to improve this project.

📄 License

MIT License

Copyright (c) 2017 cytopia

linux-timemachine's Issues

Missing destination folder is resulting in script fail

When target folder does not exist on disk, script fails due to missing variable

linux-timemachine/timemachine

Line 184 in c4629a3

logerr "Local directory does not exist: ${dir_part}"

config with profiles

profiles with different rsync opts

Suggestion | Allow remote source

Currently, I'm trying to make my backup server retrieve the files from the actual source (remote server) and put it on a local destination directory. It seems that you're unable to use a remote source, this would be very handy.

How to use --files-from from rsync?

I don't want to backup the whole disk, but some dirs separated in many disks. So I have a file record all the path that I want to backup.
It seems that timemachine must have a specified <source>, right? Is there a workaround?

Pruning of old backups

There is a similar project here which uses a very convenient pruning logic to ensure short-term backups, long-term backups, and pruning of "old" backups using this logic:

Backup sets are automatically deleted following a simple expiration strategy > > defined with the --strategy flag. This strategy is a series of time intervals > with each item being defined as x:y, which means "after x days, keep one backup > every y days". The default strategy is 1:1 30:7 365:30, which means:

• After 1 day, keep one backup every 1 day (1:1).
• After 30 days, keep one backup every 7 days (30:7).
• After 365 days, keep one backup every 30 days (365:30).

Before the first interval (i.e. by default within the first 24h) it is implied > that all backup sets are kept. Additionally, if the backup destination > directory is full, the oldest backups are deleted until enough space is > available.*

This would be a great feature to have, but may be beyond the scope of the intent of this project.

Suggestion: multiple SOURCEs

I'm very glad I found your project! - I like it very much.

please, enable multiple sources like rsync does:
rsync SRC... [DEST]

reasons:
I have multiple SOURCE dirs which I want to backup at the same time into the same TARGET. Currently, I have to issue one commandline per SOURCE

• Multiple required commands glued together into one line are ugly, but this seems to be the only way to start a multi-source timemachine backup from cron or other system event.
• a multi-source multi-liner produces an ugly result, because only the latest SOURCE remains in target Current directory, while all other SOURCE dirs of my backub job are cluttered each into another timestamped/past directory in the target.

Is there a reason for not allowing a remote source?

What would break if I changed
! -d "${1}" ;
to
! dir_exists "${1}"
?

No preservation of symlinks

Not sure if this was intentional or not, but it seems symbolic links are not preserved. This is usually desirable since generally when symbolic links are present, they are there for a good reason. Restoring without the symbolic links would likely break things if a user is using them.

Missing variable when local directory is missing

The variable dir_part is not set when local directory is missing.

linux-timemachine/timemachine

Line 184 in 2fe23f3

logerr "Local directory does not exist: ${dir_part}"

It's only defined when using a remote directory:

linux-timemachine/timemachine

Lines 166 to 169 in 2fe23f3

	if is_remote "${directory}"; then
	ssh_part="$( echo "${directory}" | awk -F':' '{print $1}' )"
	dir_part="$( echo "${directory}" | awk -F':' '{print $2}' )"
	dir_part="$( escape_path "${dir_part}" )"

suggestion to add --without-overlaps option

I like this script and it very useful for me to backup my NAS storage. Thank you!

Here i would like to suggest the option which will helps to do backup process without overlaps.
In my situation, i have to backup my ssd storage per 10 minutes, and keep the backups in daily cycle.
So sometimes, the backup process getting overlapped, and rsync calculating diffs from "not really current" backup directory.

I made some changes for your script, so i would like you to check my pull request bellow.

Problem with some paths/volume names

When trying to do backup to a mounted "pendrive":

silverdr:~$ timemachine /home/silverdr /media/silverdr/Angela\'s\ Files/
/usr/local/bin/timemachine: 1: eval: Syntax error: Unterminated quoted string

I believe this is caused by the volume label including apostrophe character.

Warning, when destination directory is empty

timemachine 29c73ec, rsync v3.1.2

When the destination directory is empty, running timemachine gives the warning "--link-dest arg does not exist: ../current". The backup is fine and following runs don't show this warning.

$ mkdir dest
$ ./timemachine etc dest
--link-dest arg does not exist: ../current
$ ./timemachine etc dest
$ 

File permission issues cause backups to stop being created

I encountered a situation, where backups never finish, leaving the in_progress directory always there. After analysing, the culprit seemed to be a specific file, which had 0000 permissions set. Changing the permissions allowed backups to work normally. Such situations should probably be handled more gracefully. While zero permissions look fishy, I can imagine other permissions related situations that would also cause failures. Like files created when running processes with elevated privileges (sudo <something>).

Actually NOT incremental ?

When I execute the script it creates the backup directory with the date and time....
Now Executing the script again on the same SRC and DEST resulted in another directory with same data (nothing changed) but different time on the folder name....

I tried that about 10 times....same result. I ended up having a simple duplication (backup) of the SRC folder...

Am I doing something wrong here or should it not only create a new folder with just what was changed....and then adding it t the "current" folder?

Please help. It's frustrating because I love this script!
Thanks

Travis brew issue

Hi,

I was hoping to contribute to your repo and made some changes. However, while running travis even though my linux tests pass, the macOS keeps on running in a loop.

The error was:
Error: invalid option: --with-default-names (while installing make)

I experience this issue on my mac as well, which indicates that the "--with-default-names" option is invalid. I hope you can look into this and fix it soon if possible!

Also, the travis build for my PR (now closed) is continuously running since it is retrying to install make and failing everytime due to the error above, so please stop that build. It would be great if, in the new fixed travis file there be a timeout mechanism.

Thanks

no hard links to old backups under busybox

Hi,
set the script on my Synology NAS (DS215j) to shadow a backup folder to an external drive.

Source folder was a bit under 400GB, external had 2.8 of formatted capacity. Both disks are EXT4. After only 6-7 backups, the target disk is full. Going through and manually pruning backups via rmdir -rf date_time_directory/**/* freed up space in ~400GB chunks, which suggests every folder is a full backup, without linking to previously backed up files.

Looking at the directory structure, on disk, on the target, suggests it's OK.

admin_user@home-server:/volumeUSB1/usbshare/backups$ ls -la
total 40
drwxr-xr-x 9 admin_user users 4096 Jun  5 11:27 .
drwxrwxrwx 5 root       root  4096 Jun  2 18:19 ..
drwxr-xr-x 7 admin_user users 4096 May 21 20:44 2019-05-21__20-44-14
drwxr-xr-x 7 admin_user users 4096 Jun  5 11:25 2019-05-23__00-00-02
drwxr-xr-x 7 admin_user users 4096 May 25 00:00 2019-05-25__00-00-03
drwxr-xr-x 7 admin_user users 4096 May 27 00:00 2019-05-27__00-00-03
drwxr-xr-x 7 admin_user users 4096 May 28 00:00 2019-05-28__00-00-03
drwxr-xr-x 7 admin_user users 4096 May 29 00:00 2019-05-29__00-00-03
drwxr-xr-x 7 admin_user users 4096 May 31 00:00 2019-05-31__00-00-03
lrwxrwxrwx 1 admin_user users   20 Jun  2 23:03 current -> 2019-05-31__00-00-03

(this is missing several manually deleted backups)

Reading the source, the script should use the --link-dest option when the current dir exists, which it does.

rsync version information:

admin_user@home-server:/volume1/backups$ rsync --help
rsync  version 3.0.9  protocol version 30
<many lines of options>

Command executed (as a scheduled task via the Synology management UI:
bash /volume1/backups/timemachine.sh /volume1/backups/ /volumeUSB1/usbshare/backups/

Restricted ssh commands compatibility (rrsync)

I have restricted the commands SSH users can run to only rrsync using the command options in the SSH authorized_keys file. With rrsync I can allow access to only a certain directory (and optionally provide read-only access) and block shell access.

With this setup linux-timemachine fails:

timemachine user@server:/rsynctest ~/rsynctest/ -- --verbose
/usr/bin/rrsync: SSH_ORIGINAL_COMMAND='test -d /rsynctest' is not rsync

I have commented out these lines:

linux-timemachine/timemachine

Lines 371 to 375 in 2fe23f3

	if ! dir_exists "${1}"; then
	logerr "Source directory does not exist: ${1}"
	logerr "See -h for help."
	exit 1
	fi

Now the backup seems to complete successfully.

Would it be possible to not use test -d in this case? Perhaps the rsync command itself could be used to check if the source directory exists? Else if skipping this test doesn't cause issues, perhaps a new cli flag could be added?

Problem with moving .inprogress dir

Hi,
I'm just starting out using linux-timemachine. I wanted to save my entire home directory to an open veracrypt container on a mounted USB drive. Executing
timemachine -d /home/test /media/veracrypt1/test -- --dry-run --verbose
gives me the error message (after running through all the files in the directory)
2022-09-08 13:06:35 timemachine: [DEBUG] Renaming local dir: '/media/veracrypt1/test/.inprogress' -> '/media/veracrypt1/test/2022-09-08__13-06-27'
2022-09-08 13:06:35 timemachine: [DEBUG] $ mv '/media/veracrypt1/test/.inprogress' '/media/veracrypt1/test/2022-09-08__13-06-27'
mv: cannot stat '/media/veracrypt1/test/.inprogress': No such file or directory
2022-09-08 13:06:35 timemachine: [ERROR] Failed to rename local dir
I manually created a /media/veracrypt1/test/.inprogress file, so it can't be a permissions thing. The veracrypt container is also still open through all this. I feel like I'm missing something obvious; could someone please point me in the right direction?

Allow rsync options to be passed without requiring double quoting (--filter="'dir-merge /.rsync-filter'")

Considering that I can pass this option simply like this when using rsync directly...

# rsync -r <source> <dest> --filter='dir-merge /.rsync-filter'

, it seems like a surprising behavior that when you try to pass these exact same rsync args to timemachine, it results in an error:

# timemachine <source> <dest> -- --filter='dir-merge /.rsync-filter'

unexpected end of filter rule: dir-merge

It appears that timemachine is passing these args on to rsync without any quotes around it at all, the same as this:

# rsync -r <source> <dest> --filter=dir-merge /.rsync-filter

unexpected end of filter rule: dir-merge

I don't know off-hand what the right solution is, but IMHO this is a bug.

Attempts to solve...

I tried changing this line:

-               $* \
+               \"$*\" \

and in fact it does fix it for the simple example above. However, as soon as you add multiple pass-through args, it fails, because it seems to be sending them through as a single arg:

# timemachine <source> <dest> -- --progress --filter='dir-merge /.rsync-filter'

rsync: --progress --filter=dir-merge /.rsync-filter: unknown option

I suspect this behavior has something to do with the attempt to construct the command to run in a string and then eval it — which is very difficult to do properly and securely and handle all edge cases (see http://mywiki.wooledge.org/BashFAQ/048). Presumably word splitting is happening when we don't want it to be?

Why are we using eval anyway? I see that we were not using eval here in 4740817, for example. And of curiousity, I checked out that commit, and tried my example and it works!

I believe this could be solved fairly easily using an array of args rather than a string (see http://mywiki.wooledge.org/BashFAQ/050). However, it looks like you are using sh rather than bash, so I don't think we can use arrays, right? 😢

As the FAQ above says:

POSIX sh has no arrays, so the closest you can come is to build up a list of elements in the positional parameters.

I'll push up a MR with that approach and see if it works better... See #85.

See also

While you're at it, see if this is related to #82 and whether the same general solution could apply to both...

suggestion: my excludes, necessary / useful

To successfully complete a timemachine backup of my Ubuntu home directory, I had to exclude certain special files which can't be backupped. I maintain an --exclude-from file, to get my backup done:

necessary excludes to finish a backup of my Ubuntu desktop's home directory without error
[$USER]/.dbus
*.sock

other excludes, to make it nicer
/Trash/
/.Trash/
[$USER]/.cache/
**/*cache/
**/*Cache/
**/*CACHE/
lost+found

other excludes, especially because I have netatalk running:
[$USER]/.AppleDB/
[$USER]/.AppleDesktop/
**/.AppleDouble/
[$USER]/Network\ Trash\ Folder/
/Temporary\ Items/
.DS_Store

remark:
Not sure if a pattern like [$USER] will work. In my case I have simply written the literal name of my SOURCE directory.

remote backups do not work

Doing something like:

timemachine localdir server:/tmp/remotedir

fails.

1. There is code to check the destination folder exists (-d "${2}") which only works on local machine
2. Even without this test the tidy up code after the rsync 2/3 and 3/3 will fail since these also only work n local machine.

rsync perms, owner, group preservation options are not used

I think it is desirable to explicitly tell rsync to preserve file ownership, permissions and group. This would be an easy PR, I will add one and you can decide whether you would like to accept it.

Implement resume flag in case of interrupted backups

Implement -r (--resume) flag in case of interrupted backups

Errors from path arguments with spaces

In v1.3.2, I have found that errors are encountered from

• remote paths with spaces
• rsync arguments containing paths with spaces

Here is the behavior in Ubunty 20.04.6:
$ timemachine -d remotehost:/home/user/spaced\ dir /home/user/backup
2023-05-29 16:51:34 timemachine: [DEBUG] $ ssh -oStrictHostKeyChecking=no -oLogLevel=QUIET -q remotehost test -d '/home/user/spaced dir'
bash: line 0: test: /home/user/spaced: binary operator expected

$ timemachine -d /home/user/data /home/user/backup -- --exclude-from=/home/user/spaced\ dir/exclude.txt
2023-05-29 17:02:45 timemachine: [INFO] $ rsync --exclude-from=/home/user/spaced dir/exclude.txt '/home/user/data' '/home/user/backup/.inprogress'
rsync: failed to open exclude file /home/user/spaced: No such file or directory (2)
rsync error: error in file IO (code 11) at exclude.c(1496) [client=3.1.3]
2023-05-29 17:02:45 timemachine: [ERROR] timemachine Backup has failed

These changes (in two cases each) and one addition resolve these issues:
- ssh_cmd="test -d ${dir_part}"
+ ssh_cmd="test -d \"${dir_part}\""

- cmd="ssh ${SSH_ARGS} ${ssh_part} ${ssh_cmd}"
+ cmd="ssh ${SSH_ARGS} ${ssh_part} \"${ssh_cmd}\""

+# Path escape the rsync options in case there are any spaces in them
+RSYNC_OPTS=$( escape_path "${@}" )
+

- logmsg "\$ rsync $* $( escape_path "${SRC}" ) $( escape_path "${DEST}/${BACKUP_INPROGRESS}" )"
+ logmsg "\$ rsync ${RSYNC_OPTS} $( escape_path "${SRC}" ) $( escape_path "${DEST}/${BACKUP_INPROGRESS}" )"

- $* \
+ ${RSYNC_OPTS} \

Feature: a form of progress indication

I am unsure if it is feasible but some overview type of progress indication would be highly appreciated. rsync's --progress option is not really usable for this purpose. Other than that - thanks for the tool!

exclude file/dir options?

Thank you for timemachine! is there an option to 'exclude' directories/files like there is with rsync using syntax "--exclude" ?

Explicit instructions on how to restore from backups

Reading the documentation, it is not immediately clear how one would go about restoring from a backup. It would be nice to have this in the documentation.

Backup fails because of a file that is not even into the source directory to be backed up `sync warning: some files vanished before they could be transferred (code 24)`

I am trying to backup the /User/user01/Library using rsync (via the linux-timemachine wrapper https://github.com/cytopia/linux-timemachine) to an external hard drive Volumes/name_of_backup logged in as user01. During the backup I get the following error message

rsync warning: some files vanished before they could be transferred (code 24) at /System/Volumes/Data/SWE/macOS/BuildRoots/220e8a1b79/Library/Caches/com.apple.xbs/Sources/rsync/rsync-55/rsync/main.c(996) [sender=2.6.9]

What I don't understand is that the file /System/Volumes/Data/SWE/macOS/BuildRoots/220e8a1b79/Library/Caches/com.apple.xbs/Sources/rsync/rsync-55/rsync/main.c is not even located in the directory that I am trying to backup (namely /User/user01/Library). So why does this error appears? And most importantly how can I get rid of the problem?

Suggestion: improved doc for running tests / testing on MacOS 13

Thank you for linux-timemachine. Exactly what I was looking for. I like the focus on testing.

I want to use timemachine on MacOS13 and want to run all tests before really using it. I cannot find info on how to do this (I have written a runall.sh, but it fails with a docker error message - Unable to find image 'cytopia/ssh-server:latest' locally on test 10 Remote (absolute path)).

My suggestion: add documentation explaining how to run all tests and if necessary, add a bash file that runs all tests.

If I can figure it out myself, I will generate a pull request.

Note: I have not used github for a long time and am not familiar with how .sh automated tests can be run.

Make use of ionice/nice to allow less invasive backups

@cytopia It would be nice to have a default and/or an option to ionice/nice the backup operation to reduce lag on the system during a backup. It is a pretty simple thing to add, I can PR it if you want, maybe you prefer to put it in?

Incremental issue

The readme states as the very first option

# Recursive, incremental and atomic backup
$ timemachine /source/dir /target/dir

When testing this option, found that the only way to get the backup incremental is to add the archive option mentioned later

$ timemachine /source/dir /target/dir -- --archive --progress

Is there anything wrong with my test?
Or shouldn't the --archive option added to the script?

include file options

Thank you for timemachine. It look like a very clean, powerfull piece of software that I really would like to adopt!
Maybe similar to "exclude file/dir options? #57", is there an option to only include certain files into a directory (and exclude / ignore all the other without having to write them explicitly) for example for the dot files into the main directory (where many directory and other dot files should not be backedup)?

How to skip broken soft links?

When broken soft links exist, timemachine thinks it failed its job and doesn't proceed the final step.

automatic deletion when backup volume is getting full

There appears to be no feature to automatically delete backups when the backup volume gets full.
Is this planned, at some point?

Backup over SSH keeps asking for password

Trying to run a backup to a remote host (sudo ./timemachine / user@host:/path/) results in the script asking continuously for the password, even if it is indeed correct.