drozdziak1/nip

Yet another IPFS git remote helper

nip

nip is a git remote helper that'll put your repo's objects on IPFS - i.e. Nowhere In Particular.

Installation

Like with most Rust packages, the easiest way to install will be using Cargo:

$ cargo install nip

Usage

Important: Before you try to use nip please make sure that your local IPFS instance is running on its standard port.

Pushing an existing repo to a nip remote for the first time

$ git remote add nip nip::new-ipfs # Use a magic placeholder URL representing a new IPFS repo
$ git push --all nip # Push all refs to a brand new repo

Cloning a repo from nip

$ git clone nip::/ipfs/QmZq47khma5nP7DjHUPoERhKnfNUPqkr5pVwmS8A6TQSeN some_repo

Repo administration with nipctl (WIP)

nip comes with nipctl - a utility for nip repo administration. As for today Its functionality is very minimal (printing of objects and indices), but some of the planned features include:

• Garbage collection - for removing all objects not associated with any refs items
• Managing git push notification settings - Depends on #7

How does it all work?

See FAQ.md for a tour of underlying nip functionality.

Development

If you'd like to hack on nip, the dev_bootstrap.sh script is where you should start. It symlinks nipctl and git-remote-nip as nipdevctl and git-remote-nipdev in ~/.cargo/bin, respectively. As a result, git will pick git-remote-nipdev for every remote that has a nipdev::<hash_or_mode> address instead of git-remote-nip, which enables painless testing during developing.

Limitations

• Running times - nip will only work as fast as IPFS lets it.
• Repo pinning and git push notifications - people interested in keeping track of remote repo's progress have no way of knowing about pushes made to it. See this issue for progress on the solution.
• Disk space - by design local git objects need to have IPFS counterparts which are kept in your local IPFS node's data store. In practice this means that every local object pushed to a nip repo needs to be stored on your disk again in a form that IPFS understands. However, nip guarantees object deduplication for all repos you use with it, which means a given git object is stored on IPFS only once, no matter the repo it comes from.
• Object size - nip doesn't know yet how to stream objects into/out of the local repository and will attempt to load them into RAM, this increases the memory footprint substantially for repos that posess large objects. Tracked here.

nip's Issues

repo hash persistance, repo pinning and notification scheme

User story

After you push to nip you'd generally want all parties interested to be notified of your push and to pin your new index with underlying objects so that it can be pulled if e.g. you push just before closing your machine.

Requirements

• Index updates must be authenticated - Authenticity of latest repo index information should be cryptographically guaranteed
• Repo persistance must be ensured - After a push happens, the pusher must be able to ask nip to wait until someone else pins the index and the underlying objects, so that the pusher node's availability is no longer required - i.e. you need to be able to safely do:

$ git push nip_remote && systemctl poweroff # or any other thing that makes your IPFS node unavailable

Solutions

IPNS

The described issue can be pretty easily remedied by simply publishing an index to an IPNS record. The repo pinning daemon could then poll for underlying address change and ipfs pin the newest value. Cryptographically, everything would rely on IPNS PKI.

Pros

• Very easy to do
• Heavy lifting done by IPFS

Cons

• All parties allowed to push to a repo would need to share the IPNS key - If one of them leaks the key the repo can be abused by malicious actors; False updates could be mitigated by rightful owners if they choose to sign every branch tip, but differentiating between a false push and a situation where they just forget to sign is very hard.
• Publishing to IPNS can take tens of seconds - If you want to just quickly push and get out of wherever you are, you'll need to sit not only through the publishing, but also the time required for an update to reach any interested peers and however long them downloading the objects will take.

An In-house PKI-powered pubsub notification system

A more sophisticated approach would be to have every pusher send a GPG-signed push announcement to a IPFS pubsub topic (a common one for all users by default so that they can be more certain that it's carried along the P2P network since more nodes claim they're interested) and wait for a number of signed "I'm done pinning your thing" responses to come back.

Pros

• The single IPNS key problem disappears
• The pinners get to work as soon as they're notified

Cons

• Will take significant effort to design
• May turn out not to work that well, as pub and sub nodes often like to take their time before they find each other - An experimental implementation is required to really take a closer look at this behavior

Don't strive for 100% decentralization and offer multiple channels for push information, some decentralized, some not

Form a common message format that can be send over any communications channel. This could incorporate e.g. posting to a Matrix channel, Twitter account, i.e. anything persistent and publicly visible.

Pros

• Flexibility of any of the chosen channels
• Extensibility

Cons

• Labor intensity
• Many approaches will require representation in the nip link - Could be mitigated by extending nip index metadata further, which in turn could complicate the data format

Use recursive tree traversal only for gathering objects for a sequential "to-do" list

• - sequential to-do list

Also include the git hash in every NIPObject

After some consideration I elected it would be better for every NIPObject to mention the SHA hash of the git object it encapsulates, which should improve debugging and general nipctl user experience.

Add a 1-to-2 migration

The format has changed and once the v0.2 milestone is complete it should have migrations ready.

Simplify the README

I think I've put too much stuff in the README (or ordered said stuff poorly).

Support nip[dev] links

Support JSON for structure-listing

Is the IPFS crate's block API size-limited?

There doesn't seem to be an ipfs add equivalent in the bindings.

Design nipctl functionality

Draft feature list:

• Repo format migrations - not necessary yet
• Index reverts - let the user display an earlier version of a certain index
• Repo structure preview
  • Index
  • Objects

Implement a NIPObject enum type + serialization/deserialization methods

Garbage collection

TODO:

• Unpin all past indices except N last ones - refcounting might be tricky; research IPFS tree objects
• Dangling objects:
  • Detect
  • Unpin

It's impossible to list legacy objects

nipctl list will fast-forward an older index, but it can't do the same for individual objects.

Remote helper cannot understand submodules

Whenever git-remote-nip is asked to fully push a branch state introducing a submodule, the process crashes due to an object not being found.

Separate common functions to a nip_core crate

nip_core should also be suitable for people willing to interact with nip repos programmatically.

Render the object tree to a graphviz dot file

This should help the user view the structure of their nip repo and aid debugging.