dop251/spgz

Sparse compressed files optimized for random reads and writes

spgz ('Sparse gzip')

Spgz is a Go library for handling compressed files optimized for random read and write access.

A file is split into blocks of equal size. Each block can be stored either uncompressed or compressed using gzip (it compresses the data and only stores it compressed if it saves at least 8KB). If the block is compressed a hole is punched at the reminder of the block and this is how disk space is saved. The position of each block remains fixed allowing efficient random access.

Because it uses Fallocate(FALLOC_FL_PUNCH_HOLE), it only works on filesystems that support it (see http://man7.org/linux/man-pages/man2/fallocate.2.html).

There is an overhead of one byte per block and a 4KB header.

This library was developed for the disk image backup tool.

spgz's Issues

Can we mark /dev/nbd* device as readonly when using -buse-readonly?

Currently, we need do this to mount an ext4 spgz image as readonly:

spgz -b imagefile -buse-readonly /dev/nbd0 -debug
mount -o ro,noload /dev/nbd0 /mnt

Reference: https://unix.stackexchange.com/questions/337638/mount-ext4-read-only

If you make the device readonly first:
blockdev --setro /dev/foo
then the filesystem will be unable to write to it, so you don't need to know any special type-dependent filesystem options.

If this blockdev is automatically called, mount would be easier for user.

Please support decompress to STDOUT; or maybe a general UI overhaul?

The usual convention is when specifying "-" as outfile, this would decompress to STDOUT.

Utils such as bzip2 and gzip also provide -c switch for STDOUT, but here it's already used for other purpose. On the other hand since this (amazingly ingenious) tool seems to be on the early stage of development and popularity, maybe it's still not too late to make the usage options as close as possible to standard gzip/bzip2? (see "gzip --help") Every new compression utility tends to go for that, such as xz/lzip/lzop/zstd/etc. Sure a few options from gzip are not possible or don't apply here, but the most basic ones still do.

how to build spgz?

Because of the help of a respondent to the "How to build this?" issue regarding diskrsync, I'm able to build diskrsync, but I have to use diskrsync in --no-compress mode because I have no idea how to build spgz. If spgz doesn't work or can't be built, maybe diskrsync's readme.md file shouldn't mention it. I'm attaching 2 files that show where I am with this. install_diskrsync.txt is a bash script, and terminal_output.txt shows what happens when I run that script. (I have almost no experience with Go. I'm running on an up-to-date Debian Buster/Sid system.)
install_diskrsync.txt
terminal_output.txt