Details of "Intelligent selection of full and incremental transfers"

[amescornish]

Details of "Intelligent selection of full and incremental transfers"

[AmesCornish]

You're correct that the backups are more resilient because buttersink does occasional full backups, and tries to reduce the length of the chain of differences required to restore any particular snapshot.

The algorithm for this is heuristic. Buttersink uses some factors for the cost of transferring the data to S3, of storing the data in S3, and of the likelihood of failure based on the amount of data and number of files that could possible be corrupted. It then attempts to find the "differences" that create the most resilient backups for the least cost. Because it is a heuristic algorithm, the exact scheme differs based on the data and the current set of differences in S3. It would be hard to predict the results in advance.

You have probably notices that the differences are labeled with a "to...from" scheme, so you can see what differences are dependent on what other differences.

Though I have not used Glacier personally, I did think about it during the design of buttersink. My vision of how to use it would be to set up an AWS policy to rotate your S3 differences into Glacier. As long as you leave the ".bs" alone, buttersink should be able to continue backing-up your snapshots. When you restore, you may need to request that AWS transfer a file from glacier back to S3.

Another possibility would be to backup to two separate buckets -- one for normal S3 and another where you have Glacier configured. Then any restores from the plain S3 bucket would never need a transfer from Glacier.

As you can see, buttersink hasn't been tested with Glacier much, so if you try it, I'd be interested in your experiences with either of the two methods above.

In answer to your question about btrfs receive, yes, the files are pure btrfs send data, and so should work normally with btrfs receive. FYI, buttersink does some extra things to enable things like restoring from a combination of local snapshots and remote "send" backup data, which btrfs does not support.

Thanks!