matteoguadrini/butterfly-backup

Butterfly Backup is a simple command line wrapper of rsync for complex task, written in python.

Butterfly Backup: presentation

The plan is great when the backup plan is excellent!

What is that?

Butterfly Backup is a simple command line wrapper of rsync for complex task, written in python.

Why butterfly?

Butterfly Backup exploits the potential of rsync with maximum simplicity, maximum flexibility and more. Moreover, its greatest strength is the organization of backups in a catalog, easy to consult.

What can I do?

With Butterfly Backup I can perform single or group backups (Full, Incremental, Differential and Mirror), restore, export, list and archive old backups.

How can you do it?

Naturally through the synergy of rsync and OpenSSH technology and the power of Python..

Which platforms support?

Butterfly Backup can backup Linux, BSD, MacOSX and Windows(with cygwin, see config docs)

Real uses

This list consists of only a few examples; applications can be endless:

• Backing up periodically (ex. once a month) a folder where I store my photos over the years;
• Log's backup of one or more servers;
• Backup of users of one or more machines;
• Backup system config of much servers;
• Create a backup snapshot of the my laptop;
• Create a central server than backupping client and server;
• Backup a entire file server, incrementally;

Real possibilities

• Configuration for silently or bulk backup
• All backup are organized into a catalog
• List single or all backup by the catalog
• List detail of a single backup
• Backup single PC, with Full,Incremental,Differential and Mirror mode;
• Backup more PCs, with Full,Incremental,Differential and Mirror mode (with parallelism algorithm);
• Backup custom folder or predefined data (User,Config,Application,System,Log): see backup docs
• Restore backup on the same PC
• Restore backup in other PC
• Restore backup in other operating system
• Apply retention policy to delete old backup
• Archive old backup in other file system or same (zip backup folder)
• Export one backup to another file system

Butterfly Backup: in action

Transform rsync in a powerfully backup/restore/archive tool

Operation

All operation of Butterfly Backup are server to client, agent-less. The server must be Unix machine with latest rsync installed. This means that all commands must be executed by the backup server. Of course, nothing prevents the backup server from being itself (localhost).

To see all the operations and more examples, see the docs.

Test

If you want to try or test Butterfly Backup before installing it, run the test:

$ git clone https://github.com/MatteoGuadrini/Butterfly-Backup.git
$ cd Butterfly-Backup
$ bash test_bb.py
...
[92512a6e-506e-11eb-b747-2ba55b805ea5]
type = Full
path = /tmp/bb_repo/localhost/2021_01_06__23_28
name = localhost
os = Unix
timestamp = 2021-01-06 23:28:59
start = 2021-01-06 23:28:59
end = 2021-01-06 23:29:04
status = 0

Installation

Install Butterfly Backup is very simple; run this:

git clone https://github.com/MatteoGuadrini/Butterfly-Backup.git
cd Butterfly-Backup
sudo python3 setup.py install -f # -f is for upgrade
# or
sudo pip install . --upgrade
bb --help
man bb

Backup machine

Backup a single PC or server is a everyday task. But most of the data may not change in the various backups made; then, in these cases, an incremental backup is needed. Butterfly Backup natively supports incremental and differential backups, starting from a full. In this case, the first backup to be performed on the machine will be as follows:

bb backup --computer pc1 --destination /nas/mybackup --data User Config --type MacOS --mode Full

or with short option:

bb backup -c pc1 -d /nas/mybackup -D User Config -t MacOS -m Full

So we created a first Full backup, on a MacOS machine, considering the folders User -> /Users and Config -> /private/etc in the destination /nas/mybackup

Note: if you do not specify the user, Butterfly Backup will assume that the source and the destination know the same user; for example, I start the backup with the above command and my user is calling arthur, he will use the latter to log in to pc1.

Now that we have our first Full backup, we can run incremental for the next few times.

bb backup --computer pc1 --destination /nas/mybackup --data User Config --type MacOS

or with short option:

bb backup -c pc1 -d /nas/mybackup -D User Config -t MacOS

Note: Incremental mode performs a Full backup when they have not been done before.

Restore machine

Before starting any restore, you need to understand what kind of data and in what time period you have to start. So, let's start checking our backups, with this command:

bb list --catalog /nas/mybackup

The result will be the following:

BUTTERFLY BACKUP CATALOG

Backup id: f65e5afe-9734-11e8-b0bb-005056a664e0
Hostname or ip: pc1
Timestamp: 2018-08-03 17:50:36

Backup id: 4f2b5f6e-9939-11e8-9ab6-005056a664e0
Hostname or ip: pc1
Timestamp: 2018-08-06 07:26:46

Backup id: cc6e2744-9944-11e8-b82a-005056a664e0
Hostname or ip: pc1
Timestamp: 2018-08-06 08:49:00

Select (copy) Backup id when you want restore a backup. For exit, press q Now, run this command for more detail (for example, try the first):

bb list --catalog /nas/mybackup --backup-id f65e5afe-9734-11e8-b0bb-005056a664e0

The result will be the following:

Backup id: f65e5afe-9734-11e8-b0bb-005056a664e0
Hostname or ip: pc1
Type: Full
Timestamp: 2018-08-03 17:50:36
Start: 2018-08-03 17:50:36
Finish: 2018-08-03 18:02:32
OS: MacOS
ExitCode: 0
Path: /nas/mybackup/pc1/2018_08_03__17_50
List: etc
Users

Now that we are sure that the selected backup is what we want (both in data and on date), run this command:

bb restore --computer pc1 --catalog /nas/mybackup --backup-id f65e5afe-9734-11e8-b0bb-005056a664e0

So we have restored the data saved on the date indicated in our pc1.

Other operation

With Butterfly Backup, you can perform Full, Incremental and Mirror backups, applying retention or archive rules; you can activate the log function, so as to track any operation over time and/or increase verbosity. Bulk backup operations can be performed using a simple text file, formatted in a list. Is possible create, by means of openssh operations, a configuration and copy them into the machines impacted by the backup without causing the machine to request the password (key exchange). For all this, Read the Docs or run help:

bb --help

Butterfly Backup: supports

One more thing

The name butterfly, is born precisely because agent-less; like a butterfly takes the pollen from a flower and brings it elsewhere. A backup or restore is performed without any iteration responsibility on the part of the final machine. The performances are not altered. While all the operations of Butterfly Backup are carried out, the impacted machine can continuously work with peace of mind.

Follow the project

See the new features in development through this link.

Open source

Butterfly Backup is a open source project. Any contribute, It's welcome.

A great thanks.

For donations, press this

For me

For Telethon

The Telethon Foundation is a non-profit organization recognized by the Ministry of University and Scientific and Technological Research. They were born in 1990 to respond to the appeal of patients suffering from rare diseases. Come today, we are organized to dare to listen to them and answers, every day of the year.

Adopt the future

butterfly-backup's Issues

Add tests

To instill confidence in the project.

Butterfly Backup enhancement : text file for the '--custom-data' command ?

Description

For the 'backup' mode there is an option [File_list.txt] that allow to use a text file in case of several hostnames, to be backed up.
It would be useful to have the same option for the '--custom-data' command. I'm intending to daily backup up to 5 directory trees.In that case the command line starts to be very long, with the full path of each directory, and so difficult to read and to maintained.

Impacted action

backup

Restore a single file...

Hello,

This looks really promising but I would like to be able to restore a single file (or multiple, but not all) from a backup. I've read the documentation and it's not obvious if this is possible. I see the export command could be a possibility but the only option without exporting the whole backup is to use excludes.

Thanks!

Butterfly Buckup bug

Description

Custom Port Argument does not seem to work, says "Error port 22 on yoga3 is closed"

Steps to Reproduce

bb backup --ssh-port 9992 --computer yoga3 --destination /mnt/Barracuda1/ --type Unix --mode Full --data System

Expected Behaviour

ssh's successfully

Your Environment

• Butterfly Backup version used: 1.8.0
• Operating System and version: Fedora 34

Additional context

yoga3 is configured in ~/.ssh/config as follows:

Host yoga3
    Hostname 192.168.1.8
    Port 9992
    User raghav
    IdentityFile ~/.ssh/yoga3
    ServerAliveInterval 120
    ForwardAgent yes

The port should automatically be detected from the ssh config but isn't. Supplying it myself does not seem to work either.

ssh yoga3 works as expected

Butterfly Buckup enhancement

Description

Use ssh-config to parse ssh hosts, edit documentation to differentiate between host aliases configured in ~/.ssh/config vs FQDN hostname

Impacted action

backup | restore | config

Proposed names of the parameters (short and long)

• name parameter - maybe add an argument for ssh-host vs FQDN hostname
• possible argument(s) - host

Additional context

Butterfly Backup : differential backup seems to behave like full backup

Description

I was successfully daily backing up a single directory tree using the 'differential' mode:
bb backup --computer Mint-Didier --destination /media/didier/MyDrive --mode differential --custom-data /media/didier/Disk_A/Data_A --user didier --type unix --retention 7 7 --skip-error --exclude EXCLUDED
As expected, after the first backup, the used space on 'MyDrive' is increasing very slowly, following the space took by the modified or new files.

I'm now intending to daily backup 3 directories located on 2 disks:
bb backup --computer Mint-Didier --destination /media/didier/MyDrive --mode differential --custom-data /media/didier/Disk_A/Data_A /media/didier/Disk_B/Data_B1 /media/didier/Disk_B/Data_B2 --user didier --type unix --retention 7 7 --skip-error --exclude EXCLUDED
In this configuration the used space is increasing daily of the full size of the directories backed up (ie. Data_A + Data_B1 + Data_B2). It is like if BB performed a full backup every day.

Steps to Reproduce

See the command above.

Expected Behaviour

For a differential backup the used space should increase only following the space took by the modified or new files.

Your Environment

• Butterfly Backup version used: 1.10.0
• Operating System and version: Linux Mint 21.2, kernel 5.15.0-91

Additional context

Thank you anyway for your efforts to propose this nice tool !