kraymer/cronicle

Use cron to rotate files and keep time-spaced copies using symlinks.

cronicle ⏳🔄💾

/ˈkɹɒnɪkəl/ :

1. n. a factual written account of important events in the order of their occurrence
2. n. software to archive the N most recent backups of a file in a folder named after the job frequency. Recommended use is to trigger it via a cron job.

Originally, cronicle has been conceived as a solution to this particular serverfault question : How to keep: daily backups for a week, weekly for a month, monthly for a year, and yearly after that.

Features

• simplicity: add one line to your crontab and you're done
• files rotation: keep the N most recent versions of a file
• space efficient: use symlinks in target directories to store a single occurence of each backup instead of performing copies. When removing a link, remove the underlying file if no other link point to it.

Usage

In order to manage a file backups with cronicle, you must have a section in the config.yaml that matches the backups names. Under it you can then define values (number of archives to keep) for the five kinds of periodic archives : hourly, daily, weekly, monthly, yearly.

Or define a custom periodicity using the pipe syntax eg bimonthly|60: 3 to keep archives every two months over the last six months.

Example

If you have dumps of a database in a ~/dumps directory named like mydb-20170101.dump, mydb-20170102.dump, and want to keep each dump for 7 days plus go back up to two months ; a working ${HOME}/.config/cronicle/config.yaml content would be :

/home/johndoe/dumps/mydb-*.dump:
    daily: 7
    monthly: 2

Next cronicle call will result in the creation of folders DAILY and MONTHLY in /home/johndoe/dumps/, each folder containing symlinks to the .dump files.

Installation

cronicle is written for Python 2.7 and Python 3, is tested on Linux and Mac OS X.

Install with pip via pip install cronicle command.

cron triggering

For a no-brainer use, I recommend to run cronicle via cron, just after the command in charge of performing the backup. A crontab example :

@daily pg_dump -Fc mydb > /home/johndoe/dumps/mydb-`date +%F`.dump
@daily cronicle -r /home/johndoe/dumps/mydb-`date +%F`.dump

If used with the config.yaml as defined in the previous section, this daily call to cronicle guarantees that you will keep at most 9 database dumps (7 latest daily + 2 monthly).

cronicle's Issues

--dry-run removes files

the --dry-run argument does not seem to work for me. I used your shell script to create those empty sample files and then ran cronicle --remove --dry-run /tmp/x/log/foobar*.txt.
I expected it to do nothing, but instead it already removed the files. Also the --verbose flag does not seem to do anything. There is no output at all...
I copied my terminal output below.

(venv) ubuntu@ubuntu:/tmp/x/log/DAILY
$ python --version
Python 3.6.9
(venv) ubuntu@ubuntu:/tmp/x/log/DAILY
$ pip freeze
click==7.1.2
confuse==1.4.0
cronicle==0.3.1
ordereddict==1.1
python-dateutil==2.8.1
PyYAML==5.3.1
six==1.15.0

(venv) ubuntu@ubuntu:/tmp/x/log
$ ls -lt
total 0
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 31  2007 foobar_2007-12-31.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 30  2007 foobar_2007-12-30.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 29  2007 foobar_2007-12-29.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 28  2007 foobar_2007-12-28.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 27  2007 foobar_2007-12-27.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 26  2007 foobar_2007-12-26.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 25  2007 foobar_2007-12-25.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 24  2007 foobar_2007-12-24.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 23  2007 foobar_2007-12-23.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 22  2007 foobar_2007-12-22.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 21  2007 foobar_2007-12-21.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 20  2007 foobar_2007-12-20.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 19  2007 foobar_2007-12-19.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 18  2007 foobar_2007-12-18.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 17  2007 foobar_2007-12-17.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 16  2007 foobar_2007-12-16.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 15  2007 foobar_2007-12-15.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 14  2007 foobar_2007-12-14.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 13  2007 foobar_2007-12-13.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 12  2007 foobar_2007-12-12.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 11  2007 foobar_2007-12-11.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez 10  2007 foobar_2007-12-10.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez   9  2007 foobar_2007-12-09.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez   8  2007 foobar_2007-12-08.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez   7  2007 foobar_2007-12-07.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez   6  2007 foobar_2007-12-06.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez   5  2007 foobar_2007-12-05.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez   4  2007 foobar_2007-12-04.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez   3  2007 foobar_2007-12-03.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez   2  2007 foobar_2007-12-02.txt
-rw-r--r-- 1 ubuntu Domain Users 0 Dez   1  2007 foobar_2007-12-01.txt
(venv) ubuntu@ubuntu:/tmp/x/log
$ cat ~/.config/cronicle/config.yaml
/tmp/x/log/foo*.txt:
    daily: 7
    monthly: 4
(venv) ubuntu@ubuntu:/tmp/x/log
$ cronicle --remove --dry-run /tmp/x/log/foobar*.txt
(venv) ubuntu@ubuntu:/tmp/x/log
$ ls -lt
total 8
drwxr-xr-x 2 ubuntu Domain Users 4096 Dez 20 16:45 DAILY
drwxr-xr-x 2 ubuntu Domain Users 4096 Dez 20 16:45 MONTHLY
-rw-r--r-- 1 ubuntu Domain Users    0 Dez 31  2007 foobar_2007-12-31.txt
-rw-r--r-- 1 ubuntu Domain Users    0 Dez 30  2007 foobar_2007-12-30.txt
-rw-r--r-- 1 ubuntu Domain Users    0 Dez 29  2007 foobar_2007-12-29.txt
-rw-r--r-- 1 ubuntu Domain Users    0 Dez 28  2007 foobar_2007-12-28.txt
-rw-r--r-- 1 ubuntu Domain Users    0 Dez 27  2007 foobar_2007-12-27.txt
-rw-r--r-- 1 ubuntu Domain Users    0 Dez 26  2007 foobar_2007-12-26.txt
-rw-r--r-- 1 ubuntu Domain Users    0 Dez 25  2007 foobar_2007-12-25.txt
-rw-r--r-- 1 ubuntu Domain Users    0 Dez   1  2007 foobar_2007-12-01.txt
(venv) ubuntu@ubuntu:/tmp/x/log

Use click-log for logging

https://click-log.readthedocs.io/en/stable/

Empty periodic dirs created

exemple: MONTHLY/ created whether a monthly attribute is present in the config or not

Why does `archives_create_days` care when a file was created?

I was trying to see if I could get this to work with some hourly backups that I have. (The TL;DR is: yes, hourly|0: 5 works and gives me 5 hourly backups, aside from the issue in this question.)

Where it breaks is in rotate, because archives_create_days returns a Dict split by day, so it only considers one file from any given day.

Is there a reason it needs to work this way? When it's used here:

cronicle/cronicle/__init__.py

Line 117 in 060cb58

for link in links[cfg[ffolder.lower()]:]: # skip the n most recents

it's just to skip the n most recent, so if there isn't more than one file per day, it acts the same as a plain directory listing. When it's used here:

cronicle/cronicle/__init__.py

Line 83 in 060cb58

last_archive_day = list(archives.keys())[-1]

it's just to get the last date a file was created, which could be done by calling file_create_day on the last filename.

Did I miss something else? Would you be open to a PR that changes this? I don't really mind if there's no built-in support for hourly rotations, as long as I can make the custom periodicity work.

---

Also, this is a cool project, thanks for building this! It's much nicer than the Bash script I was considering writing myself, haha.

Use external dep `confuse` instead of shipped-in confit.py

https://pypi.org/project/confuse/

Maintaining Quarterly Backup

Hi Kraymer,
Would you please like to enhance this tool for taking a quarterly backup of a file?
So, in a Quarterly folder/dir, only four files will be present. And at end of the year only latest (last quarter) backup file will be moved to Yearly dir.
Every quarter, latest backup file of the last month of each quarter will be moved to the Quarterly folder/dir.

Cleanup broken links instead of crashing

If a link is broken (underlying file has been deleted manually) in a periodic dir, program will crash at execution.
Would be better to dleete the link in such a case

Check of elapsed time between two archives is approximative

======================================================================
FAIL: test_number_of_archives (test_cronicle.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/travis/build/Kraymer/cronicle/test/test_cronicle.py", line 104, in test_number_of_archives
    {"foo_2019-12-01_09h", "foo_2020-01-01_09h", "foo_2020-02-01_09h",},
AssertionError: Items in the first set but not the second:
'foo_2019-12-31_09h'
'foo_2020-01-30_09h'
Items in the second set but not the first:
'foo_2020-02-01_09h'
'foo_2020-01-01_09h'

Allow to add custom periodicity

A custom periodicity can't just have the number of archives defined, we need to define the min number of days between two archives too.
This number will be appended as a suffix to the period name.
So if i want to have a folder last containing the last archive, the config would look like :

/home/johndoe/dumps/mydb-*.dump:
    daily: 7
    monthly: 2
    last|0: 1

last|0 meaning keep just 1 file, no days spacing requirement between archives

Create a docker image

create a docker image which allows to use this program on any file based backup program

Release v1.0.0 on pypi

Tasks :

• claim cronicle name on pypi index cf pypi/support#317
• add comprehensive unit tests suite
• drop python2 support
• fix issues reported by @501st-alpha1