bdrewery/zfstools

Various ZFS scripts. Most notably, zfs-auto-snapshot, a ruby clone of OpenSolaris auto snapshotting

ZFS Tools

Various scripts for administrating ZFS. Modeled after time-sliderd and ZFS Automatic Snapshots from OpenSolaris

Setup

Install the gem.

Production version

gem install zfstools

Development version

rake install

Setup crontab entries for scripts wanted. See below.

Scripts

zfs-auto-snapshot

This will handle automatically snapshotting datasets similar to time-sliderd from OpenSolaris. Setup allows you to define your own intervals, snapshot names, and how many to keep for each interval. Zero-sized snapshots will automatically be cleaned up.

Usage

/usr/local/bin/zfs-auto-snapshot INTERVAL KEEP

• INTERVAL - The interval for the snapshot. This is something such as frequent, hourly, daily, weekly, monthly, etc.
• KEEP - How many to keep for this INTERVAL. Older ones will be destroyed.

Crontab

15,30,45 * * * * root /usr/local/bin/zfs-auto-snapshot frequent  4
0        * * * * root /usr/local/bin/zfs-auto-snapshot hourly   24
7        0 * * * root /usr/local/bin/zfs-auto-snapshot daily     7
14       0 * * 7 root /usr/local/bin/zfs-auto-snapshot weekly    4
28       0 1 * * root /usr/local/bin/zfs-auto-snapshot monthly  12

Dataset setup

Only datasets with the com.sun:auto-snapshot property set to true will be snapshotted.

zfs set com.sun:auto-snapshot=true DATASET

MySQL Support

Setting a MySQL dataset's property to mysql will hook it into the zfs-snapshot-mysql script. See its section for setup instructions.

zfs set com.sun:auto-snapshot=mysql DATASET

PostgreSQL Support

Setting a PostgreSQL dataset's property to postgresql will cause zfs-auto-snapshot to put postgresql in online backup mode for the snapshot.

zfs set com.sun:auto-snapshot=postgresql DATASET

The user executing zfs-auto-snapshot will require passwordless login to the postgres database and will require either REPLICATION or SUPERUSER privileges. The easiest approach is to set up a trust or ident record in your pg_hba.conf. The zfs-auto-snapshot script will execute pg_start_backup() prior to saving the snapshot and execute pg_stop_backup() afterwards.

Overrides

You can override a child dataset to use, or not use auto snapshotting by settings its flag with the given interval.

zfs set com.sun:auto-snapshot:weekly=false DATASET

zfs-snapshot-mysql

Snapshots a mysql server's databases. This requires that mysql's datadir/innodb_data_home_dir/innodb_log_group_home_dir be a ZFS dataset.

Example MySQL+ZFS Setup

Datasets

tank/db/mysql
tank/db/mysql/bin-log
tank/db/mysql/data
tank/db/mysql/innodb
tank/db/mysql/innodb/data
tank/db/mysql/innodb/log

ZFS Settings

These settings should be set before importing any data.

zfs set primarycache=metadata tank/db/mysql/innodb
zfs set recordsize=16K tank/db/mysql/innodb/data
zfs set recordsize=8K tank/db/mysql/data
zfs set compression=lzjb tank/db/mysql/data

MySQL Settings

innodb_data_home_dir = /tank/db/mysql/innodb/data
innodb_log_group_home_dir = /tank/db/mysql/innodb/log
datadir = /tank/db/mysql/data
log-bin = /tank/db/mysql/bin-log/mysql-bin

Script Usage

Setup a /root/.my.cnf with the relevant information on where to connect to, with the proper username/password that has access to FLUSH LOGS and FLUSH TABLES WITH READ LOCK. The zfs-auto-snapshot script will automatically flush the tables before saving the snapshots.

zfs-cleanup-snapshots

Cleans up zero-sized snapshots. This ignores snapshots created by zfs-auto-snapshot as it handles zero-sized in its own special way.

Usage

Crontab

*/20 * * * * /usr/local/bin/zfs-cleanup-snapshots

zfstools's Issues

zfs-auto-snapshot does not destroy old snapshots if -P is used

Given the pool mypool and dataset mypool/myfs, with auto-snapshot enabled for myfs (implicitly disabled for mypool).

The command line zfs-auto-snapshot -P mypool daily 7 does not destroy snapshots older than 7 days.

Feature Request: Snapshot names compatible with Samba Windows Shadow Copies

See: https://github.com/zfsonlinux/zfs-auto-snapshot/wiki/Samba
Maybe not as a patch as they did, but with an option like --samba?

Misclassification of parent dataset as 'recursive'

$ zfs list -rt filesystem -o name,mounted,com.sun:auto-snapshot media
NAME                MOUNTED  COM.SUN:AUTO-SNAPSHOT
media               yes      true
media/movies        no       true
media/movies/anime  yes      true
media/movies/big    yes      true
media/movies/yt     yes      true

$ zfs-auto-snapshot -n -v hourly 24 -P media
zfs snapshot -r media@zfs-auto-snap_hourly-2021-04-27-10h53
zfs snapshot -r media/movies/anime@zfs-auto-snap_hourly-2021-04-27-10h53
zfs snapshot -r media/movies/big@zfs-auto-snap_hourly-2021-04-27-10h53
zfs snapshot -r media/movies/yt@zfs-auto-snap_hourly-2021-04-27-10h53

Because media/movies is not mounted, it decides to create the leaf children's snapshots recursively. (Which is an... interesting design decision, to not snapshot unmounted filesystems)

However, it fails to treat the parent (media) as non-recursive. This results in the creation (but not subsequent deletion) of snapshots in media/movies:

$ zfs list -t snapshot media/movies | wc -l
3989

If I may ask, why all this.. sophistication, especially given that it's (clearly) brittle and results in bugs? What's wrong with treating every dataset as independent and issuing a zfs snapshot per dataset?

Zero-sized snapshots - need to keep last one

Need to keep the last zero-sized snapshot. Otherwise no snapshot would ever get made between now and a week from now, if nothing changes until then.

Auto snapshot recursive destroy

Destroying snapshots should use recursive when possible.

Single

zfs snapshot  tank@zfstools-auto-snap_frequently-2012-02-13T13:27
zfs snapshot  tank/db@zfstools-auto-snap_frequently-2012-02-13T13:27
zfs snapshot -r tank/db/backup@zfstools-auto-snap_frequently-2012-02-13T13:27
zfs snapshot -r tank/mail@zfstools-auto-snap_frequently-2012-02-13T13:27
zfs snapshot -r tank/sweb@zfstools-auto-snap_frequently-2012-02-13T13:27
zfs snapshot -r tank/vm@zfstools-auto-snap_frequently-2012-02-13T13:27
zfs destroy -d tank/vm/xzib-test8@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -d tank/vm/xzib-test9@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -d tank@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -d tank/vm/xzib-test9/system@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -d tank/db/backup@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -d tank/db@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -d tank/vm/xzib-test9/boot@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -d tank/vm/xzib-test7/system@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -d tank/vm/xzib-test8/boot@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -d tank/vm/xzib-test8/system@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -d tank/mail@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -d tank/vm@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -d tank/sweb@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -d tank/vm/gentoo-dev@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -d tank/vm/xzib-test7/boot@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -d tank/vm/xzib-test7@zfstools-auto-snap_frequently-2012-02-13T13:21

Recursive

zfs snapshot  tank@zfstools-auto-snap_frequently-2012-02-13T13:27
zfs snapshot  tank/db@zfstools-auto-snap_frequently-2012-02-13T13:27
zfs snapshot -r tank/db/backup@zfstools-auto-snap_frequently-2012-02-13T13:27
zfs snapshot -r tank/mail@zfstools-auto-snap_frequently-2012-02-13T13:27
zfs snapshot -r tank/sweb@zfstools-auto-snap_frequently-2012-02-13T13:27
zfs snapshot -r tank/vm@zfstools-auto-snap_frequently-2012-02-13T13:27
zfs destroy -d tank@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -d tank/db@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -dr tank/db/backup@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -dr tank/mail@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -dr tank/sweb@zfstools-auto-snap_frequently-2012-02-13T13:21
zfs destroy -dr tank/vm@zfstools-auto-snap_frequently-2012-02-13T13:21

Create auto scrubber

Similar features as 800.scrub-zfs on FreeBSD

Feature: Postgres 15 backup support

Postgres 15 changed the backup function names which causes the snapshotting function to fail with:

ERROR:  function pg_start_backup(unknown) does not exist
LINE 1: SELECT PG_START_BACKUP('zfs-auto-snapshot');
              ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.
ERROR:  function pg_stop_backup() does not exist
LINE 1: SELECT PG_STOP_BACKUP();
              ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

in:

zfstools/lib/zfstools/snapshot.rb

Lines 70 to 72 in 0ac02ee

	when 'postgresql'
	sql_pre_query = "SELECT PG_START_BACKUP('zfs-auto-snapshot');"
	sql_post_query = "SELECT PG_STOP_BACKUP();"

In the docs for Postgres 15
E.4.2. Migration to Version 15
Remove long-deprecated exclusive backup mode (David Steele, Nathan Bossart)

If the database server stops abruptly while in this mode, the server could fail to start. The non-exclusive backup mode is considered superior for all purposes. Functions pg_start_backup()/pg_stop_backup() have been renamed to pg_backup_start()/pg_backup_stop(), and the functions pg_backup_start_time() and pg_is_in_backup() have been removed.

Auto snapshot catchup

If the server, or cron, is down during an infrequent snapshot interval, a snapshot may be missed. For example, if yearly is being used, there's a chance that this snapshot will be missed if the 1 time it tries to do it, the pool is down, or cron is down.

The script should run more often and be smarter about when to create new snapshots per interval.

Snapshots get destroyed for non-controlled datasets

Datasets that do not have the com.sun:auto-snapshot property set, still have their snapshots cleaned up at the end.

Snapshots are created for non-mounted filesystems, but not deleted

I have a filesystem that is not mounted in my pool, and zfstools keeps creating snapshots for that filesystem, but does not clean up any old snapshots for that filesystem since it skips filesystems that are not mounted.

I ended up with 12000 snapshots :)

I don't know if you would like to not make snapshots for unmounted filesystems, or delete expired snapshots for unmounted filesystems.

Need man pages

Changes for PostgreSQL in jail.

Hi!
I have PostgreSQL, installed in jail. But zfs is at host system. Because of this, cant working command from lib/zfstools/snapshot.rb
I need one parameter in two places at line 74:
cmd = %Q[(psql <<HERE!!!>> -c "#{sql_pre_query}" postgres ; #{zfs_cmd} ) ; psql <<AND_HERE!!!>> -c "#{sql_post_query}" postgres]
to put there something like
-U postgres -h jail.host.local
But I not understand, how most correctly do this. May be someone can tell me some about?

And after some thought ...
I think, that good idea create parameter in zfs properties, whose value would be the name of the variable containing the necessary launch flags... Because now I have one jail with postgres, but very soon I will have second.
And in this case also good idea to put variable with launch flags in file at /usr/local/etc.

Skipping unmounted datasets is confusing and not documented.

I see that a change was made in 3628f79 (and modified a few times since) to not snapshot unmounted systems, but this is confusing to me because I had explicitly set the com.sun:auto-snapshot=true flag as required by the documentation.

I have a problem now that I cannot apply recursive rules to snapshots because there are none of the relevant name on the root dataset.

Maybe if you want to preserve the existing behaviour can we have an additional flag for com.sun:auto-snapshot like force which does the snapshot even if unmounted?

Daylight savings duplicate snapshots

From @DimitryAndric

cannot create snapshot 'zcapsule/ca@zfs-auto-snap_frequent-2017-10-29-02h45': dataset already exists
cannot create snapshot 'zcapsule/git@zfs-auto-snap_frequent-2017-10-29-02h45': dataset already exists
cannot create snapshot 'zcapsule/springbank@zfs-auto-snap_frequent-2017-10-29-02h45': dataset already exists
cannot create snapshot 'zcapsule/tmb/coleburn@zfs-auto-snap_frequent-2017-10-29-02h45': dataset already exists
no snapshots were created
cannot create snapshot 'zroot/var/netatalk@zfs-auto-snap_frequent-2017-10-29-02h45': dataset already exists
no snapshots were created

Support calling zfs-auto-snapshot for a specific filesystem

I'd like to be able call "zfs-auto-snapshot" and specify a ZFS filesystem to work on.
The current "one call for all" architecture is restrictive in that it forces all filesystems to be treated in a row.

Readme links are dated

http://mail.opensolaris.org/pipermail/zfs-discuss/2009-November/033882.html and https://blogs.oracle.com/timf/entry/zfs_automatic_snapshots_0_12 are dead links

Usage banner of the scripts show "$0"

When running a script, instead of the script name, the help/usage banner displays "$0" on screen, e.g.:

    Usage: $0 [-dknpuv] <INTERVAL> <KEEP>

Parallel mode is unbounded

The number of threads need to be bounded by default and allow to be overridden.

Dangling snapshots

Emergency cleanup

Support clearing out older snapshots in emergency, warning and critical states.

Support multiple snapshot destroy

https://www.illumos.org/issues/1647

zfs destroy dataset@snap1,snap2,...,snapN

zvols no longer snapshotted?

After upgrading (on FreeBSD; moved from 0.3.2 to 0.3.5), the zvols on my system are no longer being auto-snapshotted. This is due to do with the mounted check.

ZVOLs (at least on FreeBSD 10.1) output '-' for 'mounted' during the main 'zfs list' call (exposed via -d run), so they are not being snapshotted now. Changing line 116 of zfstools.rb from:

    if dataset.properties['mounted'] == "yes" and ["true","mysql","postgresql"].include? value

to

    if dataset.properties['mounted'] != "no" and ["true","mysql","postgresql"].include? value

Fixes the issue, and retains the "don't snapshot unmounted" option.

Thanks for the excellent software!

Feature request: alternative snapshot prefix

Hi,

I have multiple servers returning me zfs error 63 on snapshot creation. The problem is the long snapshot name and path. So my solution for now is to use a shorter prefix like "auto" or even a single "a" instead of "zfs-auto-snap" and use different labels for the interval like "f" instead of "frequent". To do so I have to change zfstools.rb source. I would like to suggest a new command flag to overwrite the default snapshot_prefix if specified. Something like "-l auto" would be great. This might also be interesting for the "snapshot_format", the time format string appended.

Move code into a gem

Customize number of snapshots per filesystem

It would be nice if one could change the number of snapshot, doing something like:

$ zfs set com.sun:auto-snapshot:weekly=12 zroot/var/log

So the values would go from true/false to true/false/[0-9]+

tarsnap integration

Could be generalized with #15

Ruby 1.8 support

Error using select!

Destroying zero-sized snapshots is unexpected and undocumented

I am using this on FreeBSD (installed via ports) to replace the Solaris auto-snapshot functionality, and it seems like a nice drop-in replacement, except for one thing: some snapshots unexpectedly disappeared, breaking my backup/replication. Example:

$ zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
backup               949G   894G   256K  /export/backup
backup/mysql         422G   894G  75.4G  /export/backup/mysql
backup/www           112G   894G  48.9G  /export/backup/www
...

I replicate this offsite via an incremental zfs send/receive of the root FS (zfs send -R -I snap1 backup@snap2, for details see https://github.com/adaugherity/zfs-backup). The child FSes (mysql, www) change frequently but the root fs does not, and zfs-auto-snapshot was (un)helpfully removing these empty snapshots, preventing me from using them to easily replicate all children.

After looking at the source I discovered the -k option which does exactly what I need, but this is not mentioned in the README, nor is it said that removing empty snapshots is the default (which is a difference from Solaris, AFAICT). I can see why someone might want this (and I don't expect you to change the default just to match Solaris), but it it should be documented.

zfs-auto-snapshot not run.Shared object "libdl.so.1" not found

hi

I installed zfstools.

zfs-auto-snaphot can't run.

It prompts Shared object "libdl.so.1" not found, required by "ruby"

freebsd version is 11.1-RELEASE

Support parallel snapshot creation/deletion

Feature Request - Filter zero sized snapshots by written property.

OK, so it's taken me a while to break this down to actually figure out what I need, but I have no experience with Ruby so I'm hoping you'll consider adding this as an option for me.

Essentially I have discovered that the 'USED' space of a snapshot is only what is uniquely referenced by that snapshot. The setup I have is with two different overlapping automatic snapshots running, one is frequent but sparse (remove empties), and the other is infrequent but dense (keep empties). The aim is to have the last 500 changes in a data set at 5 minute intervals, and hourly intervals for the last 4 weeks. The problem is that where the two snapshot sequences overlap (i.e. on the hour) the sparse set always registers as zero-sized because it shares its dataset with the hourly snapshot and so is removed.

There is however a property called written@snapshot which describes how much data was written to the target since the requested snapshot. For example:

zfs get written@zfs-auto-snap_frequent-2017-01-30-06h00U tank/root@zfs-auto-snap_frequent-2017-01-30-06h05U

will say how much data was written to the 06h05U snapshot that didn't exist in the 06h00U snapshot. While this doesn't tell us the size of the 06h00U snapshot we know that it must also contain some data that the 06h05U snapshot doesn't (if only because the metadata addressing the written data must have been updated) and so it is the last snapshot on that 'interval' to reference that data. This means that if this snapshot has a zero size currently, we do know that at some point in the future if older snapshots are removed then this snapshot will become non-zero sized.

If we only used this metric then we may double our number of kept snapshots because the previous kept snapshot may already hold the same data as us. Effectively the snapshot we now keep may actually represents the last point that the kept snapshot before it remains unchanged. To differentiate this case we need to ensure that this snapshot also differs from the previous kept snapshot.

So, essentially what I am asking for is a filter for each zero sized snapshots that checks whether data was written between the previous snapshot and itself, and between itself and the next snapshot. If data exists in both of those cases I would like to preserve the snapshot because its 'USED' property would be non-zero if all snapshots not of its 'interval' were removed. The edge case of the most recent snapshot doesn't apply because it isn't subject to zero sized removal anyway, and the oldest snapshot should treat its written against previous as non-zero since it must be unique on this 'interval'.

Support libzfs

Auto Bookmark Creation option

hi there

Would it be possible to add an option to also create directly a bookmark for the snapshot that was created with the same name?

e.g.

zfs bookmark tank/DS@zfs-auto-snap_daily-2018-07-04-00h00 tank/DS#zfs-auto-snap_daily-2018-07-04-00h00

auto send snapshots elsewhere

Add ability to automatically send INTERVAL's snapshots elsewhere when snapshotting.

Hook mysql backups into auto-snapshots

Create user-defined dataset properties to define before/after hooks for the auto-snapshotting. This will allow flushing the tables, and then unlocking them, after creating a snapshot with the normal auto-snapshot script.

Zero-sized snapshot cleanup incorrectly deletes non-auto snapshots

root       39861    0.0  0.0   33420   2112 ??  DJ     9:47AM      0:00.01 zfs destroy -dr backup/db@Feb-01-2013

Support multiple snapshots at once

https://www.illumos.org/issues/2900 added support for specifying multiple snapshots at once. If this support can be detected it should speedup snapshotting significantly.

Support written property for avoiding needless snapshots

https://www.illumos.org/issues/1645

Add -n

Dry run.