rs-ipfs/rust-ipfs

The InterPlanetary File System (IPFS), implemented in Rust.

Rust IPFS

The Interplanetary File System (IPFS), implemented in Rust

Table of Contents

• Description
  • Project Status
• Install
  • Dependencies
  • Rust IPFS
• Getting Started
  • Running the tests
  • Contributing
• Roadmap
  • Completed
  • In progress
  • Still required
• Maintainers
• Alternatives
• Contributors
• License
• Trademarks

Description

This repository contains the crates for the IPFS core implementation which includes a blockstore, a libp2p integration which includes DHT content discovery and pubsub support, and HTTP API bindings. Our goal is to leverage both the unique properties of Rust to create powerful, performant software that works even in resource-constrained environments, while also maximizing interoperability with the other "flavors" of IPFS, namely JavaScript and Go.

Project Status - Alpha Not Maintained

You can see details about what's implemented, what's not, and also learn about other ecosystem projects, at Are We IPFS Yet?

For more information about IPFS see: https://docs.ipfs.io/introduction/overview/

Install

Rust IPFS depends on protoc and openssl.

Dependencies

First, install the dependencies.

With apt:

$ apt-get install protobuf-compiler libssl-dev zlib1g-dev

With yum:

$ yum install protobuf-compiler libssl-dev zlib1g-dev

Install rust-ipfs itself

The rust-ipfs binaries can be built from source. Our goal is to always be compatible with the stable release of Rust.

$ git clone https://github.com/rs-ipfs/rust-ipfs && cd rust-ipfs
$ cargo build --workspace

You will then find the binaries inside of the project root's /target/debug folder.

Getting started

We recommend browsing the examples, the http crate tutorial and tests in order to see how to use Rust-IPFS in different scenarios.

Running the tests

The project currently features unit, integration, conformance and interoperability tests. Unit and integation tests can be run with:

$ cargo test --workspace

The --workspace flag ensures the tests from the http and unixfs crates are also run.

Explanations on how to run the conformance tests can be found here. The Go and JS interoperability tests are behind a feature flag and can be run with:

$ cargo test --feature=test_go_interop
$ cargo test --feature=test_js_interop

These are mutually exclusive, i.e. --all-features won't work as expected.

Note: you will need to set the GO_IPFS_PATH and the JS_IPFS_PATH environment variables to point to the relevant IPFS binary.

Contributing

See the contributing docs for more info.

If you have any questions on the use of the library or other inquiries, you are welcome to submit an issue.

Roadmap

Special thanks to the Web3 Foundation and Protocol Labs for their devgrant support.

Completed Work

• Project Setup
• Testing Setup
  • Conformance testing
• HTTP API Scaffolding
• UnixFS Support
• /pubsub/{publish,subscribe,peers,ls}
• /swarm/{connect,peers,addrs,addrs/local,disconnect}
• /id
• /version
• /shutdown
• /block/{get,put,rm,stat}
• /dag/{put,resolve}
• /refs and /refs/local
• /bitswap/{stat,wantlist}
• /cat
• /get
• /resolve

Work in Progress

• /bootstrap
• /dht
• interop testing

Work still required

• /name
• /ping
• /key
• /config
• /stats
• /files (regular and mfs)
• a few other miscellaneous endpoints not enumerated here

Maintainers

Rust IPFS was originally authored by @dvc94ch and was maintained by @koivunej, and @aphelionz. Special thanks is given to Protocol Labs and Equilibrium.

Alternatives and other cool, related projects

It’s been noted that the Rust-IPFS name and popularity may serve its organization from a "first-mover" perspective. However, alternatives with different philosophies do exist, and we believe that supporting a diverse IPFS community is important and will ultimately help produce the best solution possible.

• rust-ipfs-api - A Rust client for an existing IPFS HTTP API. Supports both hyper and actix.
• ipfs-embed - An implementation based on sled
• rust-ipld - Basic rust ipld library supporting dag-cbor, dag-json and dag-pb formats.
• PolkaX's own rust-ipfs
• Parity's rust-libp2p, which does a lot the of heavy lifting here

If you know of another implementation or another cool project adjacent to these efforts, let us know!

Contributors

Code Contributors

This project exists thanks to all the people who contribute. [Contribute].

Financial Contributors

Become a financial contributor and help us sustain our community. [Contribute]

Individuals

Organizations

Support this project with your organization. Your logo will show up here with a link to your website. [Contribute]

License

Dual licensed under MIT or Apache License (Version 2.0). See LICENSE-MIT and LICENSE-APACHE for more details.

Trademarks

The Rust logo and wordmark are trademarks owned and protected by the Mozilla Foundation. The Rust and Cargo logos (bitmap and vector) are owned by Mozilla and distributed under the terms of the Creative Commons Attribution license (CC-BY).

rust-ipfs's Issues

Processes with the grant work

I propose we aim to work by issues which define the goals when starting and going uphill and be refined during discovery. Larger goals are tackled with "tracking issues" which simply link to multiple issues similar to #59 which links this issue.

For example: phase 1.0 will be a “tracking issue” with links to other issues regarding specific work we know we need to accomplish. We might find that we’d like to split tasks under the “tracking issues” and that should be done as needed. Assignment should help us not duplicate work. Code will be presented for review using pull-requests as have been so far. During the completion of a phase we can finally close the phase 1.0 issue as hopefully all of it’s subtasks have been completed.

bitswap should have a bounded blocksize

Support for /block endpoints

References:

• https://github.com/ipfs/js-ipfs/tree/master/packages/interface-ipfs-core/src/block
• https://docs-beta.ipfs.io/reference/http/api/
• https://github.com/ipfs/interop (no occurrences)

/block/put

• Unless specified, this command returns dag-pb CIDv0 CIDs.
• Arguments
  • arg [file]: The data to be stored as an IPFS block. Required: yes.
  • format [string]: cid format for blocks to be created with. Required: no. Note: raw is the only format used in tests
  • mhtype [string]: multihash hash function. Default: "sha2-256". Required: no.
    • Setting 'mhtype' to anything other than 'sha2-256' or format to anything other than 'v0' will result in CIDv1.
  • mhlen [int]: multihash hash length. Default: "-1". Required: no.
  • pin [bool]: pin added blocks recursively. Default: "false". Required: no.
• Request Body is Argument "data" is of file type. This endpoint expects a file in the body of the request as 'multipart/form-data'.
• On success, the call to this endpoint will return with 200 and the following body:

{
  "Key": "<string>",
  "Size": "<int>"
}

• curl -F file=@myfile "http://localhost:5001/api/v0/block/put?format=&mhtype=sha2-256&mhlen=-1&pin=false"
• should error with array of blocks

/block/rm

• requires dag.put (could refactor the tests to block.put)
• requires refs.local
• does NOT error with array of cids
• errors when removing nonexistant blocks, except with force
• Arguments
  • arg [string]: Bash58 encoded multihash of block(s) to remove. Required: yes.
  • force [bool]: Ignore nonexistent blocks. Required: no.
  • quiet [bool]: Write minimal output. Required: no.
    #Response
• will fail on this test

• On success, the call to this endpoint will return with 200 and the following body:

{
  "Error": "<string>",
  "Hash": "<string>"
}

• cURL Example curl "http://localhost:5001/api/v0/block/rm?arg=&force=&quiet="

/block/get

• requires block.put first
• block.get arguments: cid of block (/api/v0/block/?arg=[cid])
• As discussed in other issues, referencing block by cid requires overhead such as versioning e.g. https://github.com/ipfs/js-ipfs/blob/master/packages/interface-ipfs-core/src/block/get.js#L64
• Blocks added as CIDv1 can be retrieved as CIDv0. Question: does this generalize to any CID version? Probably.
• HTTP call responds with 200 and plain/text response

/block/stat

Print information of a raw IPFS block.

• Requires block.put
• Stats by CID

• Arguments
  • arg [string]: The base58 multihash of an existing block to stat. Required: yes.
• On success, the call to this endpoint will return with 200 and the following body:

{
  "Key": "<string>",
  "Size": "<int>"
}

• cURL Example: curl "http://localhost:5001/api/v0/block/stat?arg="

Fix dnslink lookup

Previously domain was used but that is tokio-0.1 library. Now domain seems to have upgraded to latest tokio in https://github.com/NLnetLabs/domain.

Our needs for the DNS are:

• stub resolving
• TXT records to get dnslink 1 2

rust-libp2p should already handle resolving names in /dns4 and /dns6 multiaddrs through threadpooled (stub?) resolver (haven't checked which library).

Being able to resolve dnslinks is the desired outcome. The async-std dependency seems a bit difficult. Alternatives on tokio would include trust-dns. Even if we decide to add back tokio support, we should be considering the what ever way libp2p is using as something we could leave for async-std.

Repo housekeeping

• Update README to "Standard Readme" format and include nodes about Rust Stable #72
• Create CONTRIBUTING.md
• Code of Conduct #68

refactor networking into it's own crate/repo

This would be the first step moving to rust-{ipld,collections,daemon}. The daemon doesn't do any networking yet. The way it currently works is the ipld daemon client directly reads blocks from the file system. In the future it will transparently ask the daemon to fetch the block if it's not cached in the repo. Only the daemon has write permission on the repo. When writing blocks the daemon needs to check that they are valid to uphold the guarantee that the block's data matches it's hash. Otherwise a malicious application could put garbage into the store. The design is inherently unix centered, it uses unix sockets for performance and assumes a posix file system, especially for the way pinning and unpinning blocks currently works.

#[async_trait]
impl Store for BlockStore {
    async fn read(&self, cid: &Cid) -> Result<Option<Box<[u8]>>> {
        let res = utils::read_file(&self.paths.block(cid)).await?;
        Ok(res)
    }

    async fn write(&self, cid: &Cid, data: Box<[u8]>) -> Result<()> {
        cid.write_cbor(&mut &self.socket).await?;
        data.write_cbor(&mut &self.socket).await?;
        Ok(())
    }

    ...
}

Add web API

We don't have to reimplement a cli since https://github.com/ferristseng/rust-ipfs-api looks nice.

HTTP v0 API for testing

Per the grant plan testing with interface-js-ipfs-core and interop requires an HTTP v0 API. The launch an ipfs "daemon" part of the two test resources is in js-ipfsd-ctl.

I understand the "daemon" to be just a process like go-ipfs daemon which will hang in the foreground until POST /shutdown has been initiated, but this should be verified with the tests of js-ipfsd-ctl.

The API endpoints we need in phase 1.0 can all return "501 Not Implemented" except for /shutdown which should allow us to stop the process gracefully. The test cases of js-ipfsd-ctl expect /id to return something.

Command line option init should do similar directory setup and keygen as go-ipfs init at the IPFS_PATH set by the environment. During normal startup there can be other arguments but after listening has been started, api listening address needs to be print API listening on: $multiaddr and finally a daemon is running line must be output.

init should produce an empty file called "api" inside the IPFS_PATH, which should be written to when the listening has started with the $multiaddr and on exit the file should be truncated.

In channel discussions a rust-ipfs/{http,cli} was suggested and adding such to the root Cargo.toml workspace and a relative dependency seems to work.

mutable file system (mfs)

Depends on #17, #18 and #6

Unixfs v2

Depends on #22

Support for /refs endpoints

References:

• https://github.com/ipfs/js-ipfs/blob/master/packages/interface-ipfs-core/src/refs.js
• https://docs-beta.ipfs.io/reference/http/api/#api-v0-refs
• https://github.com/ipfs/interop (no occurrences)

/refs

List links (references) from an object.
curl "http://localhost:5001/api/v0/refs?arg=<ipfs-path>&format=<dst>&edges=<value>&unique=<value>&recursive=<value>&max-depth=-1"

• Requires object.put Need to refactor tests.
• Arguments
  • arg [string]: Path to the object(s) to list refs from. Required: yes.
  • format [string]: Emit edges with given format. Available tokens: . Default: . Default: "". Required: no.
  • edges [bool]: Emit edge format: -> . Required: no.
  • unique [bool]: Omit duplicate refs from output. Required: no.
  • recursive [bool]: Recursively list links of child nodes. Required: no.
  • max-depth [int]: Only for recursive refs, limits fetch and listing to the given depth. Default: "-1". Required: no.
• On success, the call to this endpoint will return with 200 and the following body:

{
  "Err": "<string>",
  "Ref": "<string>"
}

/refs/local

List all local references.
curl "http://localhost:5001/api/v0/refs/local"

• Test requires IPFS add, perhaps we can refactor to ipfs.dag.put
• This endpoint takes no arguments.
• On success, the call to this endpoint will return with 200 and the following body:

{
  "Err": "<string>",
  "Ref": "<string>"
}

Support for /dag endpoints

References:

• https://github.com/ipfs/js-ipfs/tree/master/packages/interface-ipfs-core/src/dag
• https://docs-beta.ipfs.io/reference/http/api
• https://github.com/ipfs/interop (no occurrences)

/dag/get

Get a dag node from ipfs.
http://localhost:5001/api/v0/dag/get?arg=<ref>

• Requires dag.put
• Red Flag: ipfs.add required here, will likely need to refactor tests
• can get a dag-pb and dag-cbor node
  • with paths and local values
• Arguments
  • arg [string]: The object to get Required: yes.
• On success, the call to this endpoint will return with 200 and a plain/text body

/dag/put

Add a dag node to ipfs.
curl -F file=@myfile "http://localhost:5001/api/v0/dag/put?format=cbor&input-enc=json&pin=<value>&hash=<value>"

• 2 hash functions: sha2-256 and sha3-512
• can be called without options, no failure
• This test is curious, it seems like there may be more going on but perhaps not...
• Arguments
  • arg [file]: The object to put Required: yes.
  • format [string]: Format that the object will be added as. Default: "cbor". Required: no.
  • input-enc [string]: Format that the input object will be. Default: "json". Required: no.
  • pin [bool]: Pin this object when adding. Required: no.
  • hash [string]: Hash function to use. Default: . Required: no.
• Request Body Argument "object data" is of file type. This endpoint expects a file in the body of the request as 'multipart/form-data'.
• On success, the call to this endpoint will return with 200 and the following body:

{
  "Cid": {
    "/": "<cid-string>"
  }
}
```

Drop nightly requirement

Moving to stable would allow us to keep up to date easier, and allow tools like skeptic and compile the README.md example automatically. Currently there is only single feature being used: try_trait which could be replaced void which is already part of the dependency tree.

Phase 1.1 Project Milestone Report

• Issue updates
• Conformance testing report
• Interop testing report

Phase 1.0 Tracking Issue

For those who do not want to look at GitHub Projects 😆

These activities are taken directly from the approved grant proposal.

1. Project Setup
   1. Git Repositories
      • rs-ipfs/logo#2
      • #41
      • #54
      • #61
      • rs-ipfs/logo#2
   2. Code Organization
      • Done in the HTTP Scaffolding Issue
   3. CI/CD
      • #62
2. HTTP Scaffolding
   1. Simple Daemonization with 501 NOT IMPLEMENTED boilerplate
      • #60
3. Conformance Testing
   1. Update js-ipfsd-ctl
      • rs-ipfs/npm-rust-ipfs-dep#1 (potentially optional)
   2. Update ipfs/interface-js-ipfs-core (failing tests)
      • Potentially no updates required since they roll upward from `js-ipfsd-ctl
   3. Run conformance testing on implemented endpoints
4. Interoperability Testing
   1. Update ipfs/interop
      • rs-ipfs/interop#1
   2. Interop Testing
5. Project Milestone Report #66
   • #66
   1. Executive Summary
   2. Comformance Test Results
   3. Interop Report
6. Rust IPFS Blog Post
   • #67

mplex transport broken for rust-ipfs to rust-ipfs communication

IPLD support

• dag model
• dag path
• local dag resolution
• ipfs dag resolution

Format support:

• raw
• dag-pb
• dag-cbor
  -- [ ] hamt (hash-array mapped trie)
  --- [ ] unixfs v2
  ---- [ ] mfs (mutable file system)
• ipld-git

RocksDataStore is not actually async

ipld: dag-pb

Use released dependencies

• libp2p
• cid
• domain (waiting on decision async-std/tokio) #83 or remove while broken?
• ctr, possible workaround: #175
• ipld (not sure why we sticked with an unreleased but there have been heavy changes since)

Implement more sophisticated bitswap strategy

Try skeptic for keeping the README up to date

skeptic can compile the code inside markdown files (README.md). There was an issue when nightly rustc was used but now it might work with stable rustc.

Tagging as good first issue as this shouldn't be painful this time. If you are interested and run into issues or need help please ask on this issue.

Support for /bitswap endpoints

Finish work to wire up the /bitswap/* endpoints

Pinning support

Related issue is garbage collection of unpinned blocks

Use async/await and new futures API

The downside is that it requires a nightly rustc, but it looks like it's going to be more performant, require less allocations and boilerplate while being more flexible.

Fix ignored website.ipfs.io IPNS test case

The current implementation checks that it resolves to a specific CID but this is flaky if they happen to update their site. The test case includes a FIXME for ideas.

Save DHT nodes between sessions

Something for future consideration. See https://www.ctrl.blog/entry/dweb-readiness-runet

Locate providers via Kademlia DHT

Finding providers (peers) for multihashes (since go-ipfs 0.5) or the client DHT (at least as far as koivunej understands the feature), connecting to the peers and fulfilling the wants through the found peers. We could start with a (very) slow implementation which always finds providers for the blocks, while it's likely that the first provider(s) will have the all related successive blocks and later on add a way to "tag" or "correlate" nested Cids to the single root Cid to get away by finding the root providers, maybe refreshing them later on, then only go find providers for successive blocks if all of the root Cid providers fail to send us the successive blocks.

Support for /pubsub/ API endpoints

These are likely passthroughs to libp2p functionality. More detail TBD

Support for /version endpoint

This should just return a static string for the version. Bonus points if we can update this automatically when Cargo.toml is modified and/or when we do a new release

ipld: git-raw

rustfmt run and travis integration

Similar to how clippy was added, rustfmt could be integrated. I think the codebase is already quite rustfmt-ish so there shouldn't be huge changes.

Adding good first issue as there shouldn't be any issues here. Travis might not install the rustfmt component by default as it did not install clippy either. Before making the formatting changes make sure the cargo fmt -- --check or whatever CI command fails as there currently should be some formatting issues.

Phase 1.0 Project Milestone Report

1. Executive Summary
2. Comformance Test Results
3. Interop Report

Discover Go and JS nodes

I believe this requires adding an mDNS implementation that is compatible with the Go and JS nodes. Not sure if this should be addressed in an IPFS implementation or in rust-libp2p.

replace the ipld implementation with rust-ipld

depends on ipld/libipld#22

Persist bitswap ledger to file system

IPFS_PATH Folder Compatibility

Inside my go-ipfs generated ~/.ipfs folder, which is conected to a private IPFS network, I have the following item. I'd like to make sure we go through these one at a time with @vmx and/or @Stebalien's guidance and determine which files or folders need to be compatible with js <-> go <-> rust.

Should be compatible:

Note: asterisks signify "as compatible as possible, with different nomenclature denoting any departure from spec"

• api
• config*
• repo.lock*

Does not need to be compatible

• version
• datastore_spec
• blocks
• datastore

Out of grant scope

• keystore
• swarm.key

We should consider storing the repo in a different directory ~/.rs-ipfs or ~/.rust-ipfs to avoid confusion / mismanagement errors.

Rust IPFS Blog Posts

• PL Post
• EQ Post

yamux transport is broken for rust-ipfs to go-ipfs communication

Dependency Analysis

Based on #59 , I'd like to perform a pretty in-depth traversal of the dependency chain to see if anything stands out, at the very least.

IPNS support

Original from 2019-02:

- [x] resolve local records
- [x] resolve dnslink records
- [ ] resolve ipns records

Updated on 2020-09:

As the real ipns record resolving was left unimplemented and persistent "local record" support was never implemented (or maybe was, with rocksdb but it was removed in #283 for not being tested, long compilation times). In #356 we removed the in-memory only local records support but retained dnslink resolving, which is now recursive since #353.

Supporting IPNS would still be an interesting feature.

Some implementation ideas:

1. Start with reading and verifying ipns records off the DHT.
2. Re-providing the verified records (if not automatically done by libp2p-kad, which doubt is done, at least the verification part)
3. Currently not at all thought about interactions between persistence and kademlia need to be thought out
4. Once an attempt at persistence has been done the next step would easily be the implementation of locally created records and their publishing and cancellation

The first step is quite straightforward, and there is already example of using DHT to find blocks in #256, #260. The more in-depth DHT interactions become quickly difficult.

References:

dnslink: https://dnslink.io/
ipns: https://github.com/ipfs/specs/blob/master/IPNS.md, see also ipfs/specs#205
kademlia recordstore: https://docs.rs/libp2p-kad/0.23.0/libp2p_kad/record/store/trait.RecordStore.html

Repo Health Checks

CI in place? Testing? Linting? Clippy? etc etc.

This can be done manually for now but we'll likely need to automate this as we start adding more

Support for /id endpoint

Right now there is static placeholder JSON but this will need to be dynamic

Persist repo to file system

Support for /swarm/* endpoints

These are likely passthroughs to libp2p functionality. More details TBD

DataStore should be modular

DataStore is currently using RocksDB. This is probably fine for a lot of use cases, but for example on iOS it might be preferable to use Core Data, or elsewhere SQLite.

DataStore should be modular, so the user can choose between different underlying stores.

hash array mapped trie (hamt)

Create Builds for as many platforms and architectures as possible

Here is what go-ipfs has - we should try to match as closely as possible.

block store should notify when block arrives

Implemented something similar this weekend here https://github.com/dvc94ch/hashgraph/blob/master/src/state/queue.rs

BlockStore considerations

I wanted to reserve time in the grant plan to discuss/design/implement a spec compliant filesystem backed store which would also track the latest developments in go-ipfs and js-ipfs (sharding, store by multihash). At the grant writing time I was also thinking alternative designs to the Store traits and/or even non-async implementation might need to be explored but I no longer see any need to do such things.

The latest version of the go-ipfs filesystem repo is likely at ipfs/go-ds-flatfs but this is no longer spec compliant, as things seem to have evolved without the spec being kept up to date. The latest version of the js-ipfs repo implementation is at ipfs/js-ipfs-repo and the filesystem datastore at ipfs/js-datastore-fs.

Of the many topics mentioned in the first paragraph the "store by multihash" and it's linked issue ipfs/kubo#6815 (similar in ipfs/js-ipfs#2415, from 2019) is probably the most pressing to discuss. As the go-ipfs PR related to this are long running (started in 2018) and I no longer see this mentioned on go-ipfs 0.5 roadmap, I think this might have been postponed until after 0.5, as it's not present on ipfs/kubo#6776 or any of the existing milestones? Could @Stebalien comment on the plans related to this, or someone else who has been keeping eye on this? As the go and js impls are still in progress on this front, it might be better to aim to more traditional blockstore compatibiltity.

If storing by multihash is not a pressing concern, should we aim to be fs-blockstore compatible with js and go? This is tested in ipsf/interop/.../repo.js. This would imply supporting at least the sharding by /repo/flatfs/shard/v1/next-to-last/2 (this was the default with go-ipfs 0.4.22) but the tests might require in practice larger subset of the "repo": $IPFS_PATH/datastore_spec at least, possibly even the leveldb supported in both js and go. The interop tests are however only testing that a block stored in one implementation can be read by the other.