rs-ipfs / rust-ipfs Goto Github PK
View Code? Open in Web Editor NEWThe InterPlanetary File System (IPFS), implemented in Rust.
License: Apache License 2.0
The InterPlanetary File System (IPFS), implemented in Rust.
License: Apache License 2.0
References:
List links (references) from an object.
curl "http://localhost:5001/api/v0/refs?arg=<ipfs-path>&format=<dst>&edges=<value>&unique=<value>&recursive=<value>&max-depth=-1"
object.put
Need to refactor tests.{
"Err": "<string>",
"Ref": "<string>"
}
List all local references.
curl "http://localhost:5001/api/v0/refs/local"
ipfs.dag.put
{
"Err": "<string>",
"Ref": "<string>"
}
I wanted to reserve time in the grant plan to discuss/design/implement a spec compliant filesystem backed store which would also track the latest developments in go-ipfs and js-ipfs (sharding, store by multihash). At the grant writing time I was also thinking alternative designs to the Store traits and/or even non-async implementation might need to be explored but I no longer see any need to do such things.
The latest version of the go-ipfs filesystem repo is likely at ipfs/go-ds-flatfs but this is no longer spec compliant, as things seem to have evolved without the spec being kept up to date. The latest version of the js-ipfs repo implementation is at ipfs/js-ipfs-repo and the filesystem datastore at ipfs/js-datastore-fs.
Of the many topics mentioned in the first paragraph the "store by multihash" and it's linked issue ipfs/kubo#6815 (similar in ipfs/js-ipfs#2415, from 2019) is probably the most pressing to discuss. As the go-ipfs PR related to this are long running (started in 2018) and I no longer see this mentioned on go-ipfs
0.5 roadmap, I think this might have been postponed until after 0.5, as it's not present on ipfs/kubo#6776 or any of the existing milestones? Could @Stebalien comment on the plans related to this, or someone else who has been keeping eye on this? As the go and js impls are still in progress on this front, it might be better to aim to more traditional blockstore compatibiltity.
If storing by multihash is not a pressing concern, should we aim to be fs-blockstore compatible with js and go? This is tested in ipsf/interop/.../repo.js. This would imply supporting at least the sharding by /repo/flatfs/shard/v1/next-to-last/2
(this was the default with go-ipfs
0.4.22) but the tests might require in practice larger subset of the "repo": $IPFS_PATH/datastore_spec
at least, possibly even the leveldb supported in both js
and go
. The interop tests are however only testing that a block stored in one implementation can be read by the other.
skeptic
can compile the code inside markdown files (README.md). There was an issue when nightly rustc was used but now it might work with stable rustc.
Tagging as good first issue as this shouldn't be painful this time. If you are interested and run into issues or need help please ask on this issue.
The downside is that it requires a nightly rustc, but it looks like it's going to be more performant, require less allocations and boilerplate while being more flexible.
Per the grant plan testing with interface-js-ipfs-core
and interop
requires an HTTP v0 API. The launch an ipfs "daemon" part of the two test resources is in js-ipfsd-ctl
.
I understand the "daemon" to be just a process like go-ipfs daemon
which will hang in the foreground until POST /shutdown
has been initiated, but this should be verified with the tests of js-ipfsd-ctl
.
The API endpoints we need in phase 1.0 can all return "501 Not Implemented" except for /shutdown
which should allow us to stop the process gracefully. The test cases of js-ipfsd-ctl expect /id
to return something.
Command line option init
should do similar directory setup and keygen as go-ipfs init
at the IPFS_PATH
set by the environment. During normal startup there can be other arguments but after listening has been started, api listening address needs to be print API listening on: $multiaddr
and finally a daemon is running
line must be output.
init
should produce an empty file called "api" inside the IPFS_PATH
, which should be written to when the listening has started with the $multiaddr
and on exit the file should be truncated.
In channel discussions a rust-ipfs/{http,cli}
was suggested and adding such to the root Cargo.toml
workspace and a relative dependency seems to work.
Based on #59 , I'd like to perform a pretty in-depth traversal of the dependency chain to see if anything stands out, at the very least.
I believe this requires adding an mDNS implementation that is compatible with the Go and JS nodes. Not sure if this should be addressed in an IPFS implementation or in rust-libp2p.
DataStore is currently using RocksDB. This is probably fine for a lot of use cases, but for example on iOS it might be preferable to use Core Data, or elsewhere SQLite.
DataStore should be modular, so the user can choose between different underlying stores.
Implemented something similar this weekend here https://github.com/dvc94ch/hashgraph/blob/master/src/state/queue.rs
Previously domain
was used but that is tokio-0.1 library. Now domain
seems to have upgraded to latest tokio in https://github.com/NLnetLabs/domain.
Our needs for the DNS are:
rust-libp2p should already handle resolving names in /dns4
and /dns6
multiaddrs through threadpooled (stub?) resolver (haven't checked which library).
Being able to resolve dnslinks is the desired outcome. The async-std dependency seems a bit difficult. Alternatives on tokio would include trust-dns. Even if we decide to add back tokio support, we should be considering the what ever way libp2p is using as something we could leave for async-std.
We don't have to reimplement a cli since https://github.com/ferristseng/rust-ipfs-api looks nice.
This would be the first step moving to rust-{ipld,collections,daemon}
. The daemon doesn't do any networking yet. The way it currently works is the ipld daemon client directly reads blocks from the file system. In the future it will transparently ask the daemon to fetch the block if it's not cached in the repo. Only the daemon has write permission on the repo. When writing blocks the daemon needs to check that they are valid to uphold the guarantee that the block's data matches it's hash. Otherwise a malicious application could put garbage into the store. The design is inherently unix centered, it uses unix sockets for performance and assumes a posix file system, especially for the way pinning and unpinning blocks currently works.
#[async_trait]
impl Store for BlockStore {
async fn read(&self, cid: &Cid) -> Result<Option<Box<[u8]>>> {
let res = utils::read_file(&self.paths.block(cid)).await?;
Ok(res)
}
async fn write(&self, cid: &Cid, data: Box<[u8]>) -> Result<()> {
cid.write_cbor(&mut &self.socket).await?;
data.write_cbor(&mut &self.socket).await?;
Ok(())
}
...
}
These are likely passthroughs to libp2p functionality. More detail TBD
For those who do not want to look at GitHub Projects 😆
These activities are taken directly from the approved grant proposal.
501 NOT IMPLEMENTED
boilerplate
Inside my go-ipfs generated ~/.ipfs
folder, which is conected to a private IPFS network, I have the following item. I'd like to make sure we go through these one at a time with @vmx and/or @Stebalien's guidance and determine which files or folders need to be compatible with js <-> go <-> rust.
Note: asterisks signify "as compatible as possible, with different nomenclature denoting any departure from spec"
We should consider storing the repo in a different directory ~/.rs-ipfs
or ~/.rust-ipfs
to avoid confusion / mismanagement errors.
CI in place? Testing? Linting? Clippy? etc etc.
This can be done manually for now but we'll likely need to automate this as we start adding more
Something for future consideration. See https://www.ctrl.blog/entry/dweb-readiness-runet
This should just return a static string for the version. Bonus points if we can update this automatically when Cargo.toml is modified and/or when we do a new release
References:
Get a dag node from ipfs.
http://localhost:5001/api/v0/dag/get?arg=<ref>
dag.put
ipfs.add
required here, will likely need to refactor testsplain/text
bodyAdd a dag node to ipfs.
curl -F file=@myfile "http://localhost:5001/api/v0/dag/put?format=cbor&input-enc=json&pin=<value>&hash=<value>"
{
"Cid": {
"/": "<cid-string>"
}
}
```
Format support:
Depends on #22
Original from 2019-02:
- [x] resolve local records
- [x] resolve dnslink records
- [ ] resolve ipns records
Updated on 2020-09:
As the real ipns record resolving was left unimplemented and persistent "local record" support was never implemented (or maybe was, with rocksdb but it was removed in #283 for not being tested, long compilation times). In #356 we removed the in-memory only local records support but retained dnslink resolving, which is now recursive since #353.
Supporting IPNS would still be an interesting feature.
Some implementation ideas:
The first step is quite straightforward, and there is already example of using DHT to find blocks in #256, #260. The more in-depth DHT interactions become quickly difficult.
References:
dnslink: https://dnslink.io/
ipns: https://github.com/ipfs/specs/blob/master/IPNS.md, see also ipfs/specs#205
kademlia recordstore: https://docs.rs/libp2p-kad/0.23.0/libp2p_kad/record/store/trait.RecordStore.html
Right now there is static placeholder JSON but this will need to be dynamic
depends on ipld/libipld#22
I propose we aim to work by issues which define the goals when starting and going uphill and be refined during discovery. Larger goals are tackled with "tracking issues" which simply link to multiple issues similar to #59 which links this issue.
For example: phase 1.0 will be a “tracking issue” with links to other issues regarding specific work we know we need to accomplish. We might find that we’d like to split tasks under the “tracking issues” and that should be done as needed. Assignment should help us not duplicate work. Code will be presented for review using pull-requests as have been so far. During the completion of a phase we can finally close the phase 1.0 issue as hopefully all of it’s subtasks have been completed.
Similar to how clippy was added, rustfmt could be integrated. I think the codebase is already quite rustfmt-ish so there shouldn't be huge changes.
Adding good first issue as there shouldn't be any issues here. Travis might not install the rustfmt
component by default as it did not install clippy
either. Before making the formatting changes make sure the cargo fmt -- --check
or whatever CI command fails as there currently should be some formatting issues.
These are likely passthroughs to libp2p functionality. More details TBD
Finding providers (peers) for multihashes (since go-ipfs 0.5) or the client DHT (at least as far as koivunej understands the feature), connecting to the peers and fulfilling the wants through the found peers. We could start with a (very) slow implementation which always finds providers for the blocks, while it's likely that the first provider(s) will have the all related successive blocks and later on add a way to "tag" or "correlate" nested Cids to the single root Cid to get away by finding the root providers, maybe refreshing them later on, then only go find providers for successive blocks if all of the root Cid providers fail to send us the successive blocks.
Related issue is garbage collection of unpinned blocks
The current implementation checks that it resolves to a specific CID but this is flaky if they happen to update their site. The test case includes a FIXME for ideas.
Finish work to wire up the /bitswap/*
endpoints
References:
{
"Key": "<string>",
"Size": "<int>"
}
{
"Error": "<string>",
"Hash": "<string>"
}
plain/text
responsePrint information of a raw IPFS block.
block.put
{
"Key": "<string>",
"Size": "<int>"
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.