yoryan/sia-slice

Maintain disk images or other large files indefinitely on the Sia network.

Sia Slice

...is a small program that maintains a mirror of a large file on the Sia decentralized storage network. The envisioned use case is storing a long-term, low-cost disk image, with periodic updates, for backups and archives on The Cloud™.

The basic idea behind Sia Slice is to chop up a single block device (or other large, monolithic file) into thousands of 80MiB chunks, LZMA-compress those chunks, and then upload them to Sia. That way, the next time you sync, you only have to upload the chunks that have changed since the last sync. Sync operations always construct a complete and identical mirror; there is no history, and there are no "full" or "incremental" snapshots. This minimizes both the complexity of the program and the storage requirements on Sia.

Besides, if you need those features, you can simply use any filesystem you choose on top of the original device or disk image. This ability is what makes Sia Slice unique: In contrast to other synchronization programs like Siasync and Repertory that operate at the file level, Sia Sice operates at the block level.

Sia Slice was written for GNU/Linux systems with Python 3.7 or later. Ports to other platforms should be possible with minimal effort.

The author uses Sia Slice weekly to mirror his Btrfs backup drive.

Installation

pip install https://github.com/YoRyan/sia-slice

Usage

You must provide the API password required to communicate with Sia, either through the command line or by setting $SIA_API_PASSWORD. I recommend using the environment variable:

export SIA_API_PASSWORD=xxxxxx

To copy the contents of /dev/sdb1 to a new Sia folder at /backupdrive:

siaslice --mirror /dev/sdb1 backupdrive

(To sync again, just run the same command. Sia Slice will locate the previous uploads and determine which blocks need to be re-uploaded.)

To download all Sia Slice data from /backupdrive and reassemble it on /dev/sdc1:

siaslice --download /dev/sdc1 backupdrive

Finally, Sia Slice writes a timestamped state file that can be used to resume any mirror or download operation in the event of a program crash, network interruption, or other catastrophic event.

To resume a stalled mirror operation:

siaslice --resume siaslice-mirror-20191024-1522.dat

Notes

Sia is an emerging technology, and I jumped through a lot of hoops to write this software. For the curious, I have written a companion blog post.

For contributors and forkers, some tests are located in tests/.

pip install -e .
pip install asynctest
cd tests/
python -m unittest ...

sia-slice's Issues

partition backup empty space

Hi,

Scenario: 100 mb used, partition size 1tb.

If I understand correctly, when you backup a partition it will backup the whole partition, even if empty space is not used at all, because sia-slice cannot detect that.
If the empty space is zero filled, the lz will minimize usage but still this remains an issue.

The only solution is using a loop device.
Any other solution?

Empty sia space causing issues?

Hello,

I try to sync a big file to Sia via siaslice. (Sia 1.4.2)
Sia space is empty (no files). I think this causes issues during first sync...
Folder to upload exists, but empty.

python3.7 ./siaslice.py -m  -b 40 -t --debug /pics_bak/enc_partition.img backupdrive

[204] POST /renter/validatesiapath/backupdrive {}
[200] GET /renter/dir/backupdrive {}
Traceback (most recent call last):
  File "./siaslice.py", line 638, in <module>
    main()
  File "./siaslice.py", line 275, in main
    start(None)
  File "./siaslice.py", line 273, in start
    asyncio.run(amain(args, stdscr=stdscr))
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "./siaslice.py", line 290, in amain
    block_size=args.block*1000*1000, stdscr=stdscr)
  File "./siaslice.py", line 312, in do_mirror
    await storage.update()
  File "./siaslice.py", line 169, in update
    for siafile in siafiles:
TypeError: 'NoneType' object is not iterable

100mb size decision is optimal?

Hi,

Can you please elaborate on 100MB size decision?
Sia is using 40MB as a block.

Does LZ always compress chunks of data to sizes near but not higher than 40 MB?
From your screenshot it seems that chunks can reach 50MB which is bad because it would consume sia storage and upload(?) of 80MB total.

Would it make sense to use a block size near 40MB?

In that case the blocks would be always close to the limit of 40MB without crossing it and you can estimate more accurately the space that sia-slice will consume. (at the risk of underusing that space)