libp2p / rust-libp2p Goto Github PK
View Code? Open in Web Editor NEWThe Rust Implementation of the libp2p networking stack.
Home Page: https://libp2p.io
License: MIT License
The Rust Implementation of the libp2p networking stack.
Home Page: https://libp2p.io
License: MIT License
Since nodes self-report their identity, it cannot be trusted.
Create a QuicConfig<T>
struct.
Since we require the underlying transport to be UDP and UDP operates on individual packets of data and not a stream, T::Output
should implement Stream<Bytes> + Sink<Bytes>
(or something similar to Bytes
) instead of AsyncRead + AsyncWrite
.
QuicConfig
itself should implement Transport
and its Output
should implement StreamMuxer
.
These are all interesting because they have their own discovery mechanisms that don't require Kademlia and in fact discovered addresses should not be propagated over "globally addressable" transports like IP since they would be almost entirely false positives. We could propagate bluetooth addresses to other bluetooth devices (to prevent an identify
call) but I don't think that's necessary since we already narrow down the number of devices that we need to connect to by having the concept of "pairing", where the user selects which devices to trust.
I think that a good API would be for these to each have a Controller
struct that has a fn scan
taking a Fn(impl Stream<Item = Client>) -> impl Stream<Item = Client>
, where Client
is a struct containing information on a discovered client and the output is a list of clients to attempt connection to. For a chat app, this function could hand off control to a UI dialog and then return the clients that the user has chosen.
https://github.com/libp2p/specs/ doesn't mention rust libp2p much. Could you open up a project tracker for this repo, documenting what has been done, what is in progress and what needs to be done?
I am looking at using rust-libp2p for Ethereum sharding with Drops of Diamond.
The following features don't show up for rust-libp2p or in search results, so I suggest adding them to issues and the project tracker:
libnice
, libwebrtc
, or natty
. This actually [appears to be implemented](https://github.com/libp2p/rust-libp2p/search?utf8=%E2%9C%93&q=nat_traversal&type=0, it is just not too easy to find, so this is more of a docs issue.There are Rust implementations of all the major formats that IPFS is using:
— https://github.com/libp2p/libp2p/issues/33#issuecomment-392007707
What's already done in rust-libp2p:
This repository includes a facade crate named libp2p
, which reexports the rest of the repository.
Architecture of the other crates of this repository:
datastore
: Utility library whose API provides a key-value storage with multiple possible
backends. Used by peerstore
.example
: Example usages of this library.libp2p-identify
: Protocol implementation that allows a node A to query another node B what
information B knows about A. Implements the ConnectionUpgrade
trait of libp2p-core
.libp2p-peerstore
: Generic storage for information about remote peers (their multiaddresses and
their public key), with multiple possible backends. Each multiaddress also has a time-to-live.
Used by libp2p-core
.libp2p-ping
: Implementation of the ping
protocol (the exact protocol is specific to libp2p).
Implements the ConnectionUpgrade
trait of libp2p-core
.libp2p-secio
: Implementation of the secio
protocol. Encrypts communications. Implements the
ConnectionUpgrade
trait of libp2p-core
.libp2p-core
: Core library that contains all the traits of libp2p and plugs things together.libp2p-tcp-transport
: Implementation of the Transport
trait of libp2p-core
for TCP/IP.libp2p-websocket
: Implementation of the Transport
trait of libp2p-core
for Websockets.multistream-select
: Implementation of the multistream-select
protocol, which is used to
negotiate a protocol over a newly-established connection with a peer, or after a connection
upgrade.rw-stream-sink
: Utility library that makes it possible to wrap around a tokio Stream + Sink
of bytes and implements AsyncRead + AsyncWrite
.Right now the API doesn't easily allow for example putting multiplex on top of multiplex on top of multiplex. The number of layers is hard-coded at compile-time, and cannot be easily changed dynamically at runtime.
Another example problem is that it is not possible to negotiate directly multiplex without going through plaintext or secio.
Just to follow all the name convention used by all the other rust pieces we have :)
This test has been there for a long time and it has suddenly started failing, probably after a cargo update
.
Follow-up of #84
Right now pinging a websocket opened by this library will trigger an error and close the stream.
With asmjs and/or wasm.
Is that an objective as well? It should be easy to achieve.
When we connect to an IPFS Go node, the remote opens three substreams but doesn't use them at all (doesn't send any data on them).
PR #120 allows one to make Kademlia requests to other nodes. I would expect that if we connect to a node through Kademlia, this node would want to get information about us.
Note that if two Rust nodes connect to one another, they will successfully identify each other through a substream. It's only when connecting to a Go node that things don't work.
I am not sure whether this behaviour from Go nodes is expected, or if it's a bug in our code.
This is currently implemented with TODOs in the libp2p-kad
crate.
An AddrComponent
is not necessarily valid, while a Multiaddr
always is.
This should be fixed either by checking the validity of the AddrComponent
when building a Multiaddr
, or by changing the API of AddrComponent
.
multiformats/rust-multiaddr#28 has been merged!
Once the current spaghetti of pull requests has been merged, we should remove rust-multiaddr
from the local repo and switch to https://github.com/multiformats/rust-multiaddr/
We want some way to easily create a large and distributed (over IP-space) simulated network that allows for easy testing of peer discovery, routing and other aspects.
From initial research, following tools seem suitable:
This wasn't done before because this requires installing protoc
on the machine that compiles libp2p
.
However a new crate https://crates.io/crates/protobuf-codegen was released, and is pure Rust and can thus be used.
I don't know for sure if we need a timer, but if one is needed, then it should depend on tokio-timer 0.2
and not on tokio
.
Otherwise, tokio
depends on tokio-core
which doesn't compile for emscripten.
I find this module messy and I think we should split it in multiple modules for better readability.
Dialing hasn't really been sorted out yet. It's easy to do for simple multiaddrs, but for complex ones, particularly those with multiple hops, things get much fuzzier.
Imagine we have registered two forms of transport: TCP and p2p-circuit, which is used to relay connections.
One example which is difficult to make work is something like
ip4/1.2.3.4/tcp/8888/p2p-circuit/p2p/DestPeer
(circuit relay multiaddr format is <relay-peer>/p2p-circuit/<remote-peer>
)
This address, when used for dialing, says "Connect to the peer DestPeer on any available address, through a relay node we will connect to via tcp on port 8888 over the ipv4 address 1.2.3.4"
ip4/4.5.6.7/tcp/30042/p2p-circuit/ip4/1.2.3.4/tcp/8888/p2p-circuit/p2p/DestPeer
Here's a double-hop relay:
Connect to the peer DestPeer
through the relay on tcp://1.2.3.4:8888
, which we will connect to through the relay on tcp://4.5.6.7:30042
.
We'll need to require dialers to handle the whole address, and give them a closure or similar required to instantiate connections to different encapsulated multi-addresses.
So the logic for a circuit dial would (I think?) be
p2p-circuit
.relay
and remote
addresses.context.dial(relay).and_then(move |conn| RelayProtocol::dial_through(remote, conn))
This reads as: dial to the relay node, and then when we get a connection to it, attempt to dial the rest of the address through it.
One interesting thing to note is that our individual transport Conn
s to peers can actually be instances of a Socket
being multiplexed over a different Conn
to the same peer.
Even in the multi-hop case, we only open one logical connection, through TCP. this means that we're limited to basically one layer of virtual calls. I am not sure this holds true in general, but it depends how fine-grained we want to go.
As in, if we have the multi-addr ip4/1.1.1.1/tcp/20202/ws
, we can either have the ws
transport wrap a dialled tcp connection, or have the ws
transport handle the tcp connection internally. The first option is ideal for "composability" because it means that the ws
transport can handle any kind of wire beneath it, but then you can get into the situation of multiple levels of boxed trait objects, which are bad for performance in a multitude of ways.
It may not really be avoidable though, as long as we ensure the cost of connecting/communicating with a peer remains roughly proportional to how complicated the address for that peer is: simple peers will have fewer layers of indirection. More complex peer connections pay a cost for their complexity.
Would wrap around a Transport
and implement the Transport
trait itself.
For uploads, one possible way of implementing this is having a buffer whose content is dispatched over time to the underlying transport (using tokio-timer
for example). If the buffer is full when we want to send something, return WouldBlock
. Don't forget to later notify the tasks for which we returned WouldBlock
.
As for downloads, we can return WouldBlock
if we detect that the rate would be too fast. We would only read a maximum number of bytes from the underlying transport in order to not go over the rate. If we return WouldBlock
, then don't forget to notify the tasks later.
It is currently unimplemented.
This problem is easy to fix, however it becomes much harder to fix if we want to report failing clients to the user of the library so that they can for example perform logging.
So that the implementation can modify the address in place instead of building a new one.
However this requires adding some methods on the multiaddress struct for this to be feasible.
PR #163 replaced various uses of vectors of futures with FuturesUnordered
s and PR #169 wanted to revert those changes because the change resulted in problems, namely that SwarmFuture
could stall. One example is starting the echo-server
example, connecting to it and disconnecting again, without sending valid data:
TRACE 2018-05-09T15:46:10Z: mio::poll: registering with poller
TRACE 2018-05-09T15:46:10Z: tokio_threadpool::builder: build; num-workers=8
DEBUG 2018-05-09T15:46:10Z: tokio_reactor::background: starting background reactor
TRACE 2018-05-09T15:46:10Z: mio::poll: registering with poller
DEBUG 2018-05-09T15:46:10Z: libp2p_tcp_transport: Now listening on /ip4/0.0.0.0/tcp/10333
TRACE 2018-05-09T15:46:10Z: libp2p_swarm::swarm: Swarm listening on /ip4/0.0.0.0/tcp/10333
Now listening on "/ip4/0.0.0.0/tcp/10333"
DEBUG 2018-05-09T15:46:10Z: tokio_core::reactor: consuming notification queue
DEBUG 2018-05-09T15:46:10Z: tokio_core::reactor: loop poll - Duration { secs: 0, nanos: 46334 }
DEBUG 2018-05-09T15:46:10Z: tokio_core::reactor: loop time - Instant { tv_sec: 26326, tv_nsec: 680689193 }
DEBUG 2018-05-09T15:46:10Z: tokio_core::reactor: loop process, Duration { secs: 0, nanos: 69532 }
# first `echo "" | nc localhost 10333` from a separate console
TRACE 2018-05-09T15:46:31Z: tokio_reactor: event Readable Token(0)
DEBUG 2018-05-09T15:46:31Z: tokio_reactor: loop process - 1 events, 0.000s
DEBUG 2018-05-09T15:46:31Z: tokio_core::reactor: loop poll - Duration { secs: 20, nanos: 709840848 }
DEBUG 2018-05-09T15:46:31Z: tokio_core::reactor: loop time - Instant { tv_sec: 26347, tv_nsec: 390631598 }
DEBUG 2018-05-09T15:46:31Z: tokio_core::reactor: loop process, Duration { secs: 0, nanos: 92083 }
DEBUG 2018-05-09T15:46:31Z: libp2p_tcp_transport: Incoming connection from /ip4/127.0.0.1/tcp/54306
DEBUG 2018-05-09T15:46:31Z: libp2p_swarm::transport::upgrade: Starting protocol negotiation (listener)
TRACE 2018-05-09T15:46:31Z: mio::poll: registering with poller
TRACE 2018-05-09T15:46:31Z: tokio_reactor: event Readable | Writable Token(4194305)
DEBUG 2018-05-09T15:46:31Z: tokio_reactor: loop process - 1 events, 0.000s
DEBUG 2018-05-09T15:46:31Z: tokio_core::reactor: loop poll - Duration { secs: 0, nanos: 106871 }
DEBUG 2018-05-09T15:46:31Z: tokio_core::reactor: loop time - Instant { tv_sec: 26347, tv_nsec: 391297053 }
DEBUG 2018-05-09T15:46:31Z: tokio_core::reactor: loop process, Duration { secs: 0, nanos: 74241 }
# CTRL-C the connection
TRACE 2018-05-09T15:46:46Z: tokio_reactor: event Readable | Writable | Hup Token(4194305)
DEBUG 2018-05-09T15:46:46Z: tokio_reactor: loop process - 1 events, 0.000s
DEBUG 2018-05-09T15:46:46Z: tokio_core::reactor: loop poll - Duration { secs: 15, nanos: 152174476 }
DEBUG 2018-05-09T15:46:46Z: tokio_core::reactor: loop time - Instant { tv_sec: 26362, tv_nsec: 543692607 }
DEBUG 2018-05-09T15:46:46Z: tokio_core::reactor: loop process, Duration { secs: 0, nanos: 192847 }
TRACE 2018-05-09T15:46:46Z: mio::poll: deregistering handle with poller
DEBUG 2018-05-09T15:46:46Z: tokio_reactor: dropping I/O source: 1
DEBUG 2018-05-09T15:46:46Z: libp2p_swarm::transport::upgrade: Error while negotiated protocol upgrade: Custom { kind: Other, error: MultistreamSelectError(IoError(Custom { kind: BrokenPipe, error: StringError("unexpected eof") })) }
DEBUG 2018-05-09T15:46:46Z: libp2p_swarm::connection_reuse: error while upgrading listener connection: Custom { kind: Other, error: MultistreamSelectError(IoError(Custom { kind: BrokenPipe, error: StringError("unexpected eof") })) }
TRACE 2018-05-09T15:46:46Z: libp2p_swarm::swarm: Swarm received new connection on listener socket
WARN 2018-05-09T15:46:46Z: libp2p_swarm::swarm: Error in listener upgrade: Custom { kind: Other, error: MultistreamSelectError(IoError(Custom { kind: BrokenPipe, error: StringError("unexpected eof") })) }
# second `echo "" | nc localhost 10333`from a separate console
TRACE 2018-05-09T15:47:11Z: tokio_reactor: event Readable Token(0)
DEBUG 2018-05-09T15:47:11Z: tokio_reactor: loop process - 1 events, 0.000s
# second CTRL-C
# third `echo "" | nc localhost 10333` from a separate console
TRACE 2018-05-09T15:47:35Z: tokio_reactor: event Readable Token(0)
DEBUG 2018-05-09T15:47:35Z: tokio_reactor: loop process - 1 events, 0.000s
# third CTRL-C
# and so on ...
AFAICT this is because notifications of tasks of Future
s that have been added to FuturesUnordered
will not be propagated until FuturesUnordered::poll
is called. Presumably this problem does not always show up on the happy path because more notifications trigger SwarmFuture::poll
which in turn polls FuturesUnordered
.
Once we upgraded a connection through secio, the upper layer need to have somehow access to the public key that is used by the remote.
Required for a proper Kademlia implementation, otherwise we have no way to check the authenticity of nodes.
An easy solution would be to add a trait PublicKeyProvide
with a get_public_key(&self) -> ?
method, that would be implemented on the output of the secio middleware and on any stream that lies on top of secio. In my opinion it makes sense that this trait belongs to the core of the library (and not to libp2p-secio
), in order to avoid a spaghetti of dependencies and also to enable alternative implementations of secio/security.
The top-level README should cover what packages/crates are in this repository and a high-level overview of what they do and maybe something similar to https://github.com/libp2p/go-libp2p.
Every package should also contain its own README with a clearer explanation of what the library is, how it's intended to be used and what the API is. Tests, examples and obviously rustdocs would also be required here.
Check off a package if you've added README, tests and rustdocs:
Under the stream muxer interface.
See go implementation: https://github.com/whyrusleeping/go-smux-multiplex
Instead of using a Peerstore
trait, use a trait or an API more specific to identify or kademlia.
I wrote base58
, but the implementation is far from perfect. It's better to use bs58
This should be using https://github.com/multiformats/rust-multihash and if that is not working as expected please PR needed changes there.
I have try to compile the lates master
I get the following issue
error: paths in use
groups are experimental (see issue #44494)
Maybe it is a compiler version issue.
What is the compiler version used.
See the script dump belove including the compiler version and the OS version.
Compiling tokio-tcp v0.1.0
Compiling ring v0.12.1 (https://github.com/briansmith/ring?rev=3a14ef619559f7d4b69e2286d49c833409eef34a#3a14ef61)
Compiling cid v0.2.3
Compiling varint v0.1.0 (file:///home/cbr/work/hashwave/rust-libp2p/varint-rs)
warning: missing documentation for a variant
--> varint-rs/src/lib.rs:46:5
|
46 | / error_chain! {
47 | | errors {
48 | | ParseError {
49 | | description("error parsing varint")
... |
60 | | }
61 | | }
| |_____^
|
note: lint level defined here
--> varint-rs/src/lib.rs:21:9
|
21 | #![warn(missing_docs)]
| ^^^^^^^^^^^^
= note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)
warning: missing documentation for a variant
--> varint-rs/src/lib.rs:46:5
|
46 | / error_chain! {
47 | | errors {
48 | | ParseError {
49 | | description("error parsing varint")
... |
60 | | }
61 | | }
| |_____^
|
= note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)
warning: missing documentation for a method
--> varint-rs/src/lib.rs:285:5
|
285 | pub fn source(&self) -> &T {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^
warning: missing documentation for a struct
--> varint-rs/src/lib.rs:408:1
|
408 | pub struct VarintCodec<W> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^
Compiling tokio v0.1.6
Compiling multiaddr v0.3.0
Compiling multistream-select v0.1.0 (file:///home/cbr/work/hashwave/rust-libp2p/multistream-select)
Compiling libp2p-peerstore v0.1.0 (file:///home/cbr/work/hashwave/rust-libp2p/peerstore)
Compiling libp2p-swarm v0.1.0 (file:///home/cbr/work/hashwave/rust-libp2p/swarm)
Compiling aio-limited v0.1.0 (https://github.com/paritytech/aio-limited.git#c6f15ffb)
error: paths in `use` groups are experimental (see issue #44494)
--> swarm/src/upgrade/plaintext.rs:24:17
|
24 | use std::{iter, io::Error as IoError};
| ^^^^^^^^^^^^^^^^^^^^
error: paths in `use` groups are experimental (see issue #44494)
--> swarm/src/upgrade/simple.rs:25:17
|
25 | use std::{iter, io::Error as IoError, sync::Arc};
| ^^^^^^^^^^^^^^^^^^^^
error: paths in `use` groups are experimental (see issue #44494)
--> swarm/src/upgrade/simple.rs:25:39
|
25 | use std::{iter, io::Error as IoError, sync::Arc};
| ^^^^^^^^^
error: aborting due to 3 previous errors
error: Could not compile `libp2p-swarm`.
warning: build failed, waiting for other jobs to finish...
error: build failed
rumskib:~/work/hashwave/rust-libp2p>
rumskib:~/work/hashwave/rust-libp2p> git log | head -40
commit 37881d511ee731a3ebf532adfec90b45279b7952
Author: Pierre Krieger <[email protected]>
Date: Tue May 8 22:07:07 2018 +0200
Pass the host name with websockets (#177)
commit 64278244defc91b0470b582d402c3ec1b0a7e2be
Author: Toralf Wittner <[email protected]>
Date: Tue May 8 20:43:00 2018 +0200
Add log statements to multistream-select. (#180)
Addresses issue #126.
commit d9e1e437b8fa64aea23f7789ac6de7dd01594a10
Author: Pierre Krieger <[email protected]>
Date: Tue May 8 20:09:41 2018 +0200
Try fix circleci again (#181)
commit 773cf6c26db31c56828de4ba528730fa16890e68
Author: Toralf Wittner <[email protected]>
Date: Tue May 8 19:20:57 2018 +0200
Add log statements to libp2p-mplex. (#179)
* Add log statements to libp2p-mplex.
Addresses issue #126.
* Run `cargo fmt`.
* Wording.
* Adjust log levels.
commit fd0d75f61eb8dbd24de9de5dbc9220e7bb38e497
Author: Pierre Krieger <[email protected]>
Date: Sun May 6 17:41:06 2018 +0200
rumskib:~/work/hashwave/rust-libp2p>
rumskib:~/work/hashwave/rust-libp2p> rustc --version
rustc 1.24.1
rumskib:~/work/hashwave/rust-libp2p> cargo --version
cargo 0.25.0
rumskib:~/work/hashwave/rust-libp2p> lsbv���
lsb_release lsblk
rumskib:~/work/hashwave/rust-libp2p> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 17.10
Release: 17.10
Codename: artful
Made "difficult" because libp2p-relay
uses it.
Compilation of secio/src/lib.rs
fails with
error[E0308]: mismatched types
--> /home/my_user/.cargo/git/checkouts/rust-libp2p-98135dbcf5b63918/7d6ad23/secio/src/lib.rs:163:36
|
163 | RSAKeyPair::from_pkcs8(Input::from(&private[..])).map_err(|err| Box::new(err))?;
| ^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `untrusted::Input`, found a different struct `untrusted::Input`
|
= note: expected type `untrusted::Input<'_>` (struct `untrusted::Input`)
found type `untrusted::Input<'_>` (struct `untrusted::Input`)
note: Perhaps two different versions of crate `untrusted` are being used?
--> /home/my_user/.cargo/git/checkouts/rust-libp2p-98135dbcf5b63918/7d6ad23/secio/src/lib.rs:163:36
|
163 | RSAKeyPair::from_pkcs8(Input::from(&private[..])).map_err(|err| Box::new(err))?;
| ^^^^^^^^^^^^^^^^^^^^^^^^^
error[E0599]: no method named `digest_algorithm` found for type `ring::hmac::VerificationKey` in the current scope
and similar errors.
secio
seems to be using untrusted-0.6.0
, while ring-0.12.1
expects untrusted::Input
from untrusted-0.5.1
.
Likewise, secio
calls digest_algorithm
on ring::hmac::VerificationKey
, a method that exists now, but did not in release 0.12.1
.
This is what brings it all together into a nice usable format.
See: https://github.com/libp2p/go-libp2p/tree/master/examples/libp2p-host
Directories should simply be identify
, kad
, and so on, instead of libp2p-identify
, libp2p-kad
, etc.
The crate names, however, should have the libp2p-
prefix.
Although HTTP 2.0 is not a multiplexing protocol, we can use it in order to benefit from the existing ecosystem instead of having to create our own library.
A transport protocol like UDP doesn't guarantee reliability or ordering, but some upper layer protocols may require that the transport supports these characteristics.
Therefore it wouldn't be a bad idea to add methods to the Transport
trait so that upper layers can check for these capabilities.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.