perfacilis/backup

RSYNC incremental backup script

backup's Issues

Only the folder for the first $PERIOD is created

Description

Using local directory as target, and configuring multiple periods to be backed up, only the first period is actually backed up.

Reproduction steps

Using:

readonly BACKUP_LOCAL_DIR="/root/backup/backup-dir"
readonly BACKUP_DIRS=($BACKUP_LOCAL_DIR /root/backup/src/dir1 /root/backup/src/dir2)

readonly RSYNC_TARGET="/root/backup/target"
readonly RSYNC_DEFAULTS="-trlqpz4 --delete --delete-excluded --prune-empty-dirs"
readonly RSYNC_EXCLUDE=(tmp/ temp/)
readonly RSYNC_SECRET=''

readonly MYSQL=""
readonly MYSQLDUMP=""

# Amount of increments per interval and duration per interval resp.
readonly -A INCREMENTS=([hourly]=5 [daily]=3 [weekly]=0 [monthly]=0 [yearly]=0)
readonly -A DURATIONS=([hourly]=10 [daily]=30 [weekly]=604800 [monthly]=2419200 [yearly]=31536000)

Running a few backups, with files changing in between:

root@matt-ThinkPad-X1-Yoga-4th:~/backup# bash backup-original.sh 
Back-up initiated at Mon May 9 17:51:17 CEST 2022
Back-up list of installed packages
MySQL not set up, skipping database backup.
Moving hourly back-up to target: 0
- /root/backup/backup-dir
- /root/backup/src/dir1
- /root/backup/src/dir2
Moving daily back-up to target: 0
- /root/backup/backup-dir
- /root/backup/src/dir1
- /root/backup/src/dir2
Back-up completed at Mon May 9 17:51:17 CEST 2022
root@matt-ThinkPad-X1-Yoga-4th:~/backup# echo toto >> src/dir1/toto 
root@matt-ThinkPad-X1-Yoga-4th:~/backup# bash backup-original.sh 
Back-up initiated at Mon May 9 17:51:30 CEST 2022
Back-up list of installed packages
MySQL not set up, skipping database backup.
Moving hourly back-up to target: 1
- /root/backup/backup-dir
- /root/backup/src/dir1
- /root/backup/src/dir2
Back-up completed at Mon May 9 17:51:30 CEST 2022
root@matt-ThinkPad-X1-Yoga-4th:~/backup# echo toto >> src/dir1/toto 
root@matt-ThinkPad-X1-Yoga-4th:~/backup# bash backup-original.sh 
Back-up initiated at Mon May 9 17:51:44 CEST 2022
Back-up list of installed packages
MySQL not set up, skipping database backup.
Moving hourly back-up to target: 2
- /root/backup/backup-dir
- /root/backup/src/dir1
- /root/backup/src/dir2
Back-up completed at Mon May 9 17:51:44 CEST 2022
root@matt-ThinkPad-X1-Yoga-4th:~/backup# echo toto >> src/dir1/toto 
root@matt-ThinkPad-X1-Yoga-4th:~/backup# bash backup-original.sh 
Back-up initiated at Mon May 9 17:51:57 CEST 2022
Back-up list of installed packages
MySQL not set up, skipping database backup.
Moving hourly back-up to target: 3
- /root/backup/backup-dir
- /root/backup/src/dir1
- /root/backup/src/dir2
Moving daily back-up to target: 1
- /root/backup/backup-dir
- /root/backup/src/dir1
- /root/backup/src/dir2
Back-up completed at Mon May 9 17:51:57 CEST 2022
root@matt-ThinkPad-X1-Yoga-4th:~/backup# echo toto >> src/dir1/toto 
root@matt-ThinkPad-X1-Yoga-4th:~/backup# bash backup-original.sh 
Back-up initiated at Mon May 9 17:52:11 CEST 2022
Back-up list of installed packages
MySQL not set up, skipping database backup.
Moving hourly back-up to target: 4
- /root/backup/backup-dir
- /root/backup/src/dir1
- /root/backup/src/dir2
Back-up completed at Mon May 9 17:52:11 CEST 2022

Checking our hourly and daily backups:

root@matt-ThinkPad-X1-Yoga-4th:~/backup# ll /hourly/
total 24
drwxr-xr-x  6 root root 4096 Mai  9 17:52 ./
drwxr-xr-x 22 root root 4096 Mai  9 17:51 ../
drwxr-xr-x  4 root root 4096 Mai  9 17:51 1/
drwxr-xr-x  4 root root 4096 Mai  9 17:51 2/
drwxr-xr-x  4 root root 4096 Mai  9 17:51 3/
drwxr-xr-x  4 root root 4096 Mai  9 17:52 4/
root@matt-ThinkPad-X1-Yoga-4th:~/backup# ll /daily/
ls: cannot access '/daily/': No such file or directory
root@matt-ThinkPad-X1-Yoga-4th:~/backup# ll target/current/
total 124
drwx------ 31 root root 4096 Mai  9 17:54 ./
drwxr-xr-x  3 root root 4096 Mai  9 17:51 ../
drwxr-xr-x  2 root root 4096 Mai  9 17:51 root_backup_backup-dir/
drwxr-xr-x  2 root root 4096 Mai  9 07:42 root_backup_src_dir1/
drwxr-xr-x  2 root root 4096 Mai  9 07:50 root_backup_src_dir2/
drwx------  2 root root 4096 Mai  9 17:51 tmp.3MXfouLPOi/
drwx------  2 root root 4096 Mai  9 17:54 tmp.6dY3Qz6Td3/
drwx------  2 root root 4096 Mai  9 17:51 tmp.71dMo4uWgx/
drwx------  2 root root 4096 Mai  9 17:51 tmp.840TVoqOg5/
drwx------  2 root root 4096 Mai  9 17:51 tmp.9vRK3qPjPA/
drwx------  2 root root 4096 Mai  9 17:54 tmp.cMNLT1ztfh/
drwx------  2 root root 4096 Mai  9 17:51 tmp.D5JFiDBiey/
drwx------  2 root root 4096 Mai  9 17:51 tmp.DLkYorw6DD/
drwx------  2 root root 4096 Mai  9 17:51 tmp.dwdz7S3vkO/
drwx------  2 root root 4096 Mai  9 17:51 tmp.FOhDENhZFX/
drwx------  2 root root 4096 Mai  9 17:51 tmp.jSj3WH1rTW/
drwx------  2 root root 4096 Mai  9 17:51 tmp.KgnYHdJ3T5/
drwx------  2 root root 4096 Mai  9 17:51 tmp.LEHjljTOL0/
drwx------  2 root root 4096 Mai  9 17:51 tmp.lfNnxZi7uC/
drwx------  2 root root 4096 Mai  9 17:51 tmp.m0rvlP7umj/
drwx------  2 root root 4096 Mai  9 17:51 tmp.n0vebuDx96/
drwx------  2 root root 4096 Mai  9 17:54 tmp.NLWbqms3lG/
drwx------  2 root root 4096 Mai  9 17:52 tmp.Sji7DNEYBI/
drwx------  2 root root 4096 Mai  9 17:51 tmp.THt6pKGwjS/
drwx------  2 root root 4096 Mai  9 17:54 tmp.tqOmejOqDO/
drwx------  2 root root 4096 Mai  9 17:54 tmp.tWzX2VfFb1/
drwx------  2 root root 4096 Mai  9 17:52 tmp.VeEKjM7Bqh/
drwx------  2 root root 4096 Mai  9 17:52 tmp.vGujxxQLBY/
drwx------  2 root root 4096 Mai  9 17:51 tmp.Z1AD6ySs69/
drwx------  2 root root 4096 Mai  9 17:51 tmp.z2qmCt2Uc3/
drwx------  2 root root 4096 Mai  9 17:54 tmp.zDpq7mbeBl/

Debugging

Folllowing rsync commands are being run by the script:

# Corresponds to the hourly backup
+ rsync -trlqpz4 --delete --delete-excluded --prune-empty-dirs --exclude-from=./rsync.exclude --backup --backup-dir=/hourly/5/root_backup_backup-dir /root/backup/backup-dir/ /root/backup/target/current/root_backup_backup-dir
+ rsync -trlqpz4 --delete --delete-excluded --prune-empty-dirs --exclude-from=./rsync.exclude --backup --backup-dir=/hourly/5/root_backup_src_dir1 /root/backup/src/dir1/ /root/backup/target/current/root_backup_src_dir1
+ rsync -trlqpz4 --delete --delete-excluded --prune-empty-dirs --exclude-from=./rsync.exclude --backup --backup-dir=/hourly/5/root_backup_src_dir2 /root/backup/src/dir2/ /root/backup/target/current/root_backup_src_dir2

# Corresponds to the daily backup
+ rsync -trlqpz4 --delete --delete-excluded --prune-empty-dirs --exclude-from=./rsync.exclude --backup --backup-dir=/daily/2/root_backup_backup-dir /root/backup/backup-dir/ /root/backup/target/current/root_backup_backup-dir
+ rsync -trlqpz4 --delete --delete-excluded --prune-empty-dirs --exclude-from=./rsync.exclude --backup --backup-dir=/daily/2/root_backup_src_dir1 /root/backup/src/dir1/ /root/backup/target/current/root_backup_src_dir1
+ rsync -trlqpz4 --delete --delete-excluded --prune-empty-dirs --exclude-from=./rsync.exclude --backup --backup-dir=/daily/2/root_backup_src_dir2 /root/backup/src/dir2/ /root/backup/target/current/root_backup_src_dir2

From my understanding:

• The three first rsync will work as expected: they check the differences between the src directory and the current, backup old versions to /hourly and copy the new version over to current.
• When we reach the daily backup, the current folder is already up-to-date, therefore the rsync command doesn't have any work to do.

prepare_remote_dir creates empty tmp directory in current

I couldn't grasp the need for calling prepare_remote_dir "current".

From my understanding, it simply creates an empty tmp directory within the current folder. It does so for each period and for each folder being backed-up. After a couple of run, my current folder is flooded with those tmp directory.

It looks like you purposefully removed the trailing / after $EMPTYDIR in v0.8.2, and that you want an empty directory to be created. Could you explain the reasoning here ?

In any case, I think those empty tmp directory should be cleaned up.

Date syntax (Debian vs BSD)

In the get_intervals_to_backup() function, there is a call to set

LAST=$(date +%s -r "$INCFILE"

This works on Debian, yet the BSD version wants the +%s portion at the end only, like so:

LAST=$(date -r "$INCFILE" +%s)

If the Debian version of this command is more permissive, then switching to the latter syntax would increase the portability.

Touch command variants (Debian vs BSD)

I think it would be ideal to note that this script is specialized for Debian. There are two touch commands out in the wild. The Debian version has a -d option. The BSD version does not. Therefore, the construction of the starttime variable is also specialized. I'm working on a version of this script for myself (macOS) and came up with this for the starttime binding:

starttime=$(date +%C%y%m%d%H%M.%S)

This gives an output for the BSD touch. And it would be called like this instead:

touch -t "$STARTTIME" "$INCFILE"

Create backup-dir on target

Hi, and first thanks a lot for sharing this script. I've been testing it over the week and I have a couple of remarks, for which I will open separate issues.

The path to the backup-dir in the script is --backup-dir=/$PERIOD/$INC/$TARGET. This will create for instance a /hourly folder on the host being backed up.

I prefer having those hourly, daily etc backups located on the target, next to the current folder. So I modified it as follow: --backup-dir=$RSYNC_TARGET/$PERIOD/$INC/$TARGET.

Personally that fits my use case better.

Git Log as History

I had a thought about using git on the names of the files as a sort of history of files over time, kept in the root of the backup directory. One might search this to see when a file has been removed permanently or other such things.

Variants of `pidof` command (Debian vs BSD)

There are multiple variants of the pidof command. The one with the options in the script is for Debian. I had attempted to recreate the functionality with ps, grep, etc. But upon thinking about it, the goal of the check_only_instance() function is ultimately not to check and filter the process ids, but to ensure that the script only runs one instance at a time.

Therefore, I suggest that that function is removed and instead the flock(1) command is used at the top of the file, below the shell shebang. While I did have to install it on macOS, there is a portable version maintained by discoteq. I used nix, but he also lists a Homebrew install method, so it should be generally accessible and portable.

Usage

Per the man page, the following command would be placed at the top of any script we only want a single instance of:

[ "${FLOCKER}" != "$0" ] && exec env FLOCKER="$0" flock -n "$0" "$0" "$@" || :

What this does is to use the script itself as a lockfile. While flock has the lock set, no cron jobs could start another instance of the backup.

Testing

I did a simple test involving trying to invoke a simple script twice. The script prints out a string and also a random number:

Test script:

echo "Check only instances."
echo "Random number is: $RANDOM."

Test:

The test involves invoking the following line with and without the above flock command at the top of the script.

(trap 'kill 0' SIGINT; myscript & myscript)

where myscript is the test script, and would eventually be the backup script from this repo.

With the flock line included, we get the following output:

Check only instances
Random number is 46523.

Without the flock line, we get:

Check only instances.
Check only instances.
Random number is 26128.
Random number is 18439.

Conclusion

I believe this is a portable method that works on Debian and BSD.