Feature Request: Add support for sharing Snapshots with other ArchiveBox instances (to enable distributed / federated archiving)

[archivebox]

Feature Request: Add support for sharing Snapshots with other ArchiveBox instances (to enable distributed / federated archiving)

[pirate]

I like what ZeroNet is doing in this space: https://github.com/HelloZeroNet/ZeroNet

[pirate]

In theory, with enough people running BA we could archive a significant portion of Soundcloud before they go out of business. 😁

[pirate]

Step 1: merkel tree for identifying and querying archive blobs across a distributed system: https://gist.github.com/pirate/0a3545254615b985727b49bc5c3d99cf

[pirate]

Blocked by: #74

Once we have a good unique UUID/ULID ID scheme for Snapshots we can begin thinking about how to broadcast that with some metadata to other ArchiveBox instances / endpoints.

Planned baby steps towards this goal in the far-far-future:

1. Finalize ArchiveBox Add REST API endpoint to allow other services to POST new URLs and snapshots to ArchiveBox
2. Add functionality for ArchiveBox to announce new snapshots to the world via RSS/webhooks/realtime endpoint of some kind:
   • rest webhook support: i.e. add the ability to configure ArchiveBox to ping outside endpoints whenever a new Snapshot/ArchiveResult is created
   • RSS feed support, i.e. publish an RSS feed on the ArchiveBox server of all recent snapshots (like Pocket does for your pocket bookmarks)
3. Add native ArchiveBox UI support for searching some of these global federation mechanisms on your own instance so that you can browse snapshots from other instances and providers without leaving your one unified UI

External tools could then be developed that injest this feed to publish archivebox content on other platforms, e.g.:

• archivebox RSS -> proof-of-history blockchain e.g. Solana
• archivebox RSS -> bittorrent's magnet DHT and tracker sites
• archivebox RSS -> IFTTT/zapier/slack/zulip/etc. webhooks

Then later we can add functionality for ArchiveBox to publish snapshots/metadata to global lookup systems like proof-of-history blockchains (e.g. Solana), DHT's like bittorrent's magent system uses, distributed filesystems like IPFS, etc.