Low-level transport protocol based on TLS+yamux

[ethereum]

Low-level transport protocol based on TLS+yamux

[vpulim]

Great discussion.

The current vision is to have clients with a libp2p and RLPx/devp2p stacks running side by side and bridging the two networks. Not at all ideal, but doable.

As @dryajov mentioned, https://github.com/ethereumjs/ethereumjs-client implements a working proof-of-concept that shows how a light client running in the browser is able to communicate with full nodes. Instructions are in the README, but essentially, we run two nodes:

1. A full node that runs on a server and provides libp2p and devp2p transports. It connects to an ethereum network over devp2p and syncs the chain. It also listens for new peer connections using both devp2p and libp2p and supports eth/62, eth/63 and les over both transports.
2. A light node that runs in the browser and connects to the full node over a web socket using the 'libp2p' transport.

The ethereumjs-client is only at the proof-of-concept stage but the hope is that the Go or Rust clients would eventually also support both transports and act as bridge nodes between devp2p and libp2p-based peers.

[ferranbt]

How is that different with TLS? I mean, TLS also encrypts and authenticates every message
sent. I agree that RLPx MAC is overly complicated and could be faster but it's
fundamentally the same thing that TLS also does.

From my point of view, RlPx is like UDP and TLS like TCP.

Quoting from here:

UDP

Message oriented, you have an API (send/recv and similar) that provide you with the ability to send one datagram, and receive one datagram. 1 send() call results in 1 datagram sent, and 1 recv() call will recieve exactly 1 datagram.

TCP

Stream oriented, you have an API (send/recv and similar) that gives you the ability to send or receive a byte stream.There is no preservation of message boundaries, TCP can bundle up data from many send() calls into one segment, or it could break down data from one send() call into many segments - but that's transparent to applications sitting on top of TCP, and recv() just gives you back data, with no relation to how many send() calls produced the data you get back.

Since TLS takes as input a byte stream with no specific segments, it splits the incoming stream in manageable blocks. perform the encryption on each independent block of a smaller size than the input and then sends the chunk. I think this is not only powerful in terms of reducing memory usage but also in the type of protocols that can be built on top.

In this repo there are some benchmarks comparing RLPx and TLS + Yamux.

I don't understand this part. The eth protocol uses numbered messages, but you'd need a
way to identify the specific message in any transport protocol. Using TLS as the transport
doesn't change this aspect.

I do not argue that protocols need a way to identify messages. My point is that it should be up to them how they want to do it. RLPx forces the protocols to use a specific format (code, message). However, if the transport takes a byte stream, the application protocols can build their own message formats specifics for their use case.

For example, the next list of protocols run on top of TCP and each one defines their own message format.

• TLS https://tools.ietf.org/html/rfc5246#section-6.2.1
• RPC: https://tools.ietf.org/html/rfc5531#section-9
• DNS: https://tools.ietf.org/html/rfc1035#section-4
• SSH: https://tools.ietf.org/html/rfc4253#section-6

This flexibility allows us to build many type of different applications for the clients. For example: use native RPC for light client calls, transfer blocks over FTP, notify new blocks with ZeroMQ...

Are you going to work on the handshake?

I have a couple of ideas about the handshake but nothing clear yet.

[ferranbt]

I am working on a modular Ethereum client here.

My objective with the proposal is twofold. On the one hand, lower the memory and CPU requirements to run a node. On the other hand, to have protocol backends (eth64, les, pip...) that can be added on runtime with ease. This could increase the number of use cases that can be applied on clients.

A couple of examples:

• Transfer data (headers, bodies, receipts...) at high speeds (maybe FTP) between two nodes that know each other.
• Private transactions.
• Support to create clusters of nodes.

These cases are more easily achievable if the transport uses a generic streaming interface.

A final note on the snappy issue. There are two types of snappy compression: normal and streaming. Each one uses his own format and they are not compatible. RLPx uses the normal mode so its not possible to do snappy streaming with RLPx.

[dryajov]

So I think this is a valid question to ask, is the current incarnation of devp2p/RLPx stack streamed or is it not? There are several places where streaming might break - transport level, application level, encryption, compression, etc...

In general, if at any point in the message/packet flow, the assumption that a message/packet can't be processed as it comes, without waiting for all of its parts to arrive breaks, the stack stops being streaming - i.e. if you have to wait for the full message before you can make sense of it, you're not streaming anymore. Ofcourse, each level will still need to be able to interpret the packet in some way, but only up to the point it cares about.

For example, a streaming stack can look something like this:

 +-----------------+                                                                                                            
 |    Application  |   Application wants to send a number of blocks                                                             
 +--------|--------+   The message can contain a hint to how many blocks there would be,                                         
          |            or use some termination tag to signal the end of the sequence (stream). 
          |             (Note how having a message size for all the blocks here breaks the streaming assumption, as we don't know what the final size is before fetching all of them)
   +------------+                                                                                                               
   | Blocks Msg |     Sends the first few in a message, while it fetches the next few                                           
   +------|-----+                                                                                                               
          |                                                                                                                     
 +--------|--------+                                                                                                            
 |     Muxing      | The muxer takes the message and negotiates a new virtual channel with the remote                           
 +-----------------+                                                                                                            
                                                                                                                                
     +-------+                                                                                                                  
     | chan1 |                                                                                                                  
     | Pak1  |                                                                                                                  
     +-------+     Takes the incoming sequence of bytes and splits into chunks of say 512kb, prepends the chunk with                     
                   the channel id and writes it over the virtual channel                                                         
     +-------+                                                                                                                  
     | chan1 |                                                                                                                  
     | Pak2  |                                                                                                                  
     +-------+                                                                                                                  
+-----------------+                                                                                                             
|     .......     |  Any number of additional processor can process the packets, e.g. compression, etc...                       
+--------|--------+                                                                                                             
         |                                                                                                                      
         |                                                                                                                      
+--------|--------+                                                                                                             
|    Encryption   |  Encryption takes the chunks from the muxer/prev processor and encrypts it with some scheme, adding its own meta info on top            
+--------|--------+                                                                                                             
         |                                                                                                                      
         |                                                                                                                      
+--------|--------+  Finally the transport picks up the packet and sends it over a socket, which could be reliable or unreliable. 
|                 |  It could choose to further break the packets into smaller chunks, or the other way around, buffer a few of them before writing to the socket 
|    Transport    |  
+-----------------+ 

• a reliable protocol/transport (e.g. tcp) will take care of packet drops, ordering, etc...
• an unreliable (e.g. udp) protocol/transport won't take care of packet drops, ordering, etc...

Furthermore, you can build a reliable protocol on top of udp (quick, utp, rudp)

In order to understand if RLPx/devp2p can support streaming flow, each layer needs to be revised.

So, let me rephrase the question as follows - do any of the layer of the RLPx/devp2p stack break the streaming assumptions or not?

• RLP encoding, is it streaming and if so does it propagate all the way down to the protocol/transport
• eth/X application level protocols, which messages are/should be streamable
• compression?
• encryption?
• muxing
• transport, etc...

[FrankSzendzielarz]

Also @dryajov Will you create a new issue for the requirement here #71 (comment) ? I thought it might be more appropriate if you formulate the driver/business goal/requirement how you want it. I don't mind doing it of course.

[FrankSzendzielarz]

You might all want to review the Handshake part of the new discv5 proposals here https://github.com/ethereum/devp2p/blob/22b3b00d70d1ef507cfaa2d2d68aa66288da3546/discv5/discv5-wire.md

My current thinking goes like this:

• We will have an encrypted channel established at the level of discovery
• I think the discovery protocol need not (should not) be tied to UDP
• The encrypted channel may be re-used for any message exchange, so long as the underlying transport is selected appropriately for the 'capability'. Eg: eth messages with 10MB payload can be sent over tcp using the same aes-gcm channel, while the handshaking can continue in parallel over UDP.

Right now there is no capability that really requires streaming. However, should it be required, aes-gcm is a streamed (but message oriented) cypher.

So, I do wonder, if this was the route to take, the ENR would specify what transport(s) was/were in use, on which ports, and for which capabilities. The choice of if to use libp2p or any other toolset would be an arbitrary one.

Thoughts? (#83)

[fjl]

I don't fully understand the limitations you listed:

Rlpx macs and encrypts all the message at the same time. This may be a burden for big messages.

How is that different with TLS? I mean, TLS also encrypts and authenticates every message
sent. I agree that RLPx MAC is overly complicated and could be faster but it's
fundamentally the same thing that TLS also does.

The upper protocols that use the transport (i.e. eth64) end up having to use a fixed
message format (code, message). This limits their flexibility and the type of
communication they can use.

I don't understand this part. The eth protocol uses numbered messages, but you'd need a
way to identify the specific message in any transport protocol. Using TLS as the transport
doesn't change this aspect.

Are you going to work on the handshake?

[FrankSzendzielarz]

Sorry for the delay in responding here, have been focused on other things. I will also spend some time digesting and considering this today. It's a good starting point for a conversation that needs to be had, as addressing RLPx weaknesses has been on the radar for some time.

[dryajov]

I have the same questions as @fjl - either TSL or RLPx would need "scramble" the data over the physical/virtual channel, weather that happens explicitly (RLPx) or hidden away (TLS), doesn't change that fact.

But I agree with most everything else proposed by @ferranbt, having a more modularized stack has many benefits in both evolving and extending the higher level protocols. Having real multiplexing such as yamux or mplex has the added benefit of abstracting streaming from the protocol developer - things like handling the max message/packet size, congestion control, etc, move out of the way of designing an application layer protocol like eth62/63/64/etc...

That said, everything @ferranbt outlined is already available as part of https://libp2p.io/ (there are Go, Rust and Js implementations, with python and others in active development). It might be worth exploring the possibility of using it instead of "rolling our own", specially when Eth2 has settled on using it already.

For an example of how devp2p (eth62/63/les) could be rolled on top of libp2p take a look at - https://github.com/ethereumjs/ethereumjs-client/blob/master/lib/net/peer.

[dryajov]

An added benefit of something like libp2p is transport abstraction, currently tcp, utp, websockets and webrtc are supported, with Quick and more exotic things like bluetooth also being a possibility. This is extremely important from the perspective of developing sustainable light client protocols for browsers, IoT, etc...

We're very interested in having this possibility @musteka-la and willing to lend a hand.

[FrankSzendzielarz]

Modularisation is certainly one argument. Some aspects of RLPx re-invent the wheel where it is now unnecessary. I am still uncertain about what the overall requirements should be for the p2p network protocol. Libp2p is definitely flexible, though it seems to me as though libp2p's design goals are to enable interoperability and extensibility. This is ideal for a platform like Polkadot, but I am not sure how that should overlap with specific implementations that just want to implement their own (eg:) optimised version of a streaming transport without the flexibility or extensibility that a toolset that libp2p provides.

The original issue though is , I think, mistaken somewhat:

1. The RLPx protocol is a streaming protocol
2. The message needs to be RLP encoded anyway....though in theory I think that can be made streaming (correct me if wrong)
3. The cypher is AES-CTR, which is streaming

So I think you could actually stream a message without the memory impact described in the OP.

It would be good to keep talking about this topic, I think a discussion on libp2p, p2p , rlpx etc is most welcome.

[FrankSzendzielarz]

Some other considerations and approaches.... please take a look here: #83

[dryajov]

Thanks for the answer @FrankSzendzielarz! Let me try to describe libp2p in a little more detail, which will hopefully clarify some of the benefits of using the stack, and also give us specific points for discussion. I haven't looked through the issues, so if this is a duplicate of previous discussions I apologize in advance.

Libp2p is definitely flexible, though it seems to me as though libp2p's design goals are to enable interoperability and extensibility.

Through interoperability is certainly something that libp2p implementations strive for, it isn't necessarily what drives the design of the stack. Just like devp2p implementations for different languages have to be "on the wire" compatible, to enable ethereum interop across different clients, libp2p implementations are expected to be "on the wire" interoperable as well, but implementations are not forced to any specific architecture other than the input/output bytes being the same.

Libp2p is build around a few concepts (this is strictly my own interpretation based on my experience as a maintainer and author of a few components):

• self describing formats and addresses
• swappable and modular networking components:
  • transports
  • protocol negotiation
  • encryption
  • stream multiplexing
  • discovery

Self describing formats and addresses

IMO this is easily one of the stronger points that makes libp2p appealing and different from previous attempts at creating a flexible networking stack. It solves the issues around transport selection and eliminates "port lock ins" completely. I won't cite the full spec here but a few examples are useful to illustrate this feature.

First briefly, multiaddrs are self describing addresses that carry enough information to be able to infer the transport, physical address and port, as well as the specific resource being requested. Think of it as Urls (tho probably a bad analogy) for network addresses. A few examples:

Let's say that our application is given an ip and port pair such as 127.0.0.1:80, there is no way to tell whether this is tcp or udp, nor what actually runs on port 80, sure port 80 is a well known port, so we can assume it's www, but what if the port was 64555, no way to tell what's going on there.

• connecting to 127.0.0.1 on port 80 using tcp, would look like this:
  • /ip4/127.0.0.1/tcp/80/
• we can also specify what's the protocol available on that port
  • /ip4/127.0.0.1/tcp/80/http and while we're at it we can also point to a specific resource /ip4/127.0.0.1/tcp/80/http/baz.jpg

A few more examples:

• what if we wanted to used udp - /ip4/127.0.0.1/udp/80/
• how about quick - /ip4/127.0.0.1/udp/9090/quic

This scheme allows the client to decide the best way to initiate a connection to the remote and it makes changing ports, transports and other aspects of the stack possible as opposed to "no way, it's going to break the world" type of thing. Moreover, it allows each individual node to decide which port and transports they wish to/can expose. I can't stress enough how important this is in practice, I'll try to give a bit more detail around this later in this post.

Say for example we want to change ethereum's port to something other than 30303, it would require a full redeploy of all the clients and possibly break the network in several ways. Moreover, not being able to quickly and reliably switch ports and/or transports for each individual node makes for an easy target for ISP providers (i.e. governments) to filter/throttle that traffic. It might not be a problem now, but there is nothing stopping a few major providers doing just that for things to get really hairy. I don't want to make this a post full of hypothetical scares, so I'll leave it at that :)

Swappable and modular networking components

First a quick overview of the full stack before I move on to the individual components. A typical libp2p implementation has at least a set of transports and the multistream negotiation protocol. When a libp2p connection is being established, the client dials the remote over one or more transports, if it succeeded, the connection is optionally encrypted, optionally muxed and then handed over to the high level caller (i.e. your app).

• Transports. Virtually anything can be used as a transport - tcp and udp (utp, quick), websockets, webrtc, etc... With multiaddrs, clients can pick and choose which transport to use to establish the connection.

• Multistream protocol negotiation. This is the mechanism used to negotiate most of the higher level interactions during the lifetime of the connection. For example, what sort of encryption and multiplexing mechanism is supported by both nodes is determined using multistream. This is also the mechanism used to add custom protocols. Currently, multistream uses simple strings as identifiers, so for example yamux is registered as /yamux/1.0.0 (or something similar). There is nothing particular about the string, it's just an identifier, and eth/62 could just the same be placed under say /devp2p/eth62.

• Encryption. Right now, the only encryption mechanisms available (AFAIK), is secio, but others can be plugged in. For example, the RLPx scheme could be potentially implemented, tho possibly not much benefit there since better mechanisms are available. I don't know secio well enough to point out its strengths and weaknesses, but the point is that there is nothing preventing from other schemes being plugged in instead of it.

• Muxing/multiplexing. There is a variety of muxers supported by each implementation, with mplex being the most widely supported. Yamux is supported by Go and Rust, but not by the Js implementation, and spdy (as a muxer) supported by js only (AFAIK). Again, pluging your own muxer is a matter of wrapping it with a libp2p compatible interface.

• Discovery. This is again optional and pluggable. Currently a modified S/Kademlia dht flavor is implemented by Go and Js, the spec is being finilized here - libp2p/specs#108. In general tho, discovery is not locked down to the DHT and other mechanisms can be used, i.e. gossip or epidemic broadcasts, rendezvous points (think bittorrent trackers), etc...

• Nat traversal, relaying connections and other fancy things.

  • Nat traversal is supported to varying degrees by the different implementations. I started a js one with support for UpnP and PMP, but it isn't fully backed into the libp2p stack yet. Go's Nat support is much better. I'm not sure about Rust at all - perhaps someone from the rust libp2p team can shed some light.
  • Connection relaying is implemented as well. This allows using a third node as a relay point. It's useful as an additional Nat traversal mechanism or to allow nodes that don't share the same transports to still communicate.

---

I hope this overview (however long), is enough to continue the conversation. But before I wrap it up I want to state the specific reasons why I feel strongly about libp2p support - browsers.

The browser has evolved into a full featured runtime. It is a ubiquitous and very capable platform that is a natural fit for light clients, however the lack of browser compatible transport supports in devp2p makes it quite hard to implement a light client on top of devp2p, moreover, it doesn't look like devp2p was designed with multi-transport support in mind at all, I wouldn't be at all opposed in extending devp2p to bring in browser support, but the time it would take us to "figure it out" (design, bikeshedding, implementation) feels prohibitive especially when a worthy alternative is already available.

That said, I rather have a working solution of any sort, rather than no solution at all, so let's keep at it and keep all options on the table ;-)

[FrankSzendzielarz]

Thanks for the lengthy write-up, it shows you are committed! :) @dryajov I do understand your points. When I first looked into developing my own light client, the first thought I had was that the protocol design was extremely unfriendly. I think historically the design goals were just to build a v1 network as fast as possible, without a proper set of requirements related to real world long term usage.

There is much to say on this topic, so I will delay a worthy response until I have a bit more time and have mulled things over a bit, but in the meantime a quick question:

Please could you elaborate on the kind of scenarios and protocols you envisage for the light client in the browser? In particular, over which protocols and transports would you see discovery working and then how the actual light clients communicate with the full node peers?

Oh and as an aside (kind of unrelated)...

Would it influence those scenarios and protocol choices in any way if the transmission of the Eth protocol messages was 'almost' transport agnostic (so long as the transport was reliable and streamed like tcp) but mandated aes-gcm encryption?

[dryajov]

Thanks for the answers @FrankSzendzielarz and looking forward to your write up.

TL;DR
Browsers don't have tcp/udp capabilities exposed. AFAIK, RLPx transport level is "hard coded" to tcp for data exchange and udp for discovery - correct me if I'm wrong. The protocols (eth/62/63/less) don't have any limitations and it should be possible to run them on top of any stream oriented channel.

Please could you elaborate on the kind of scenarios and protocols you envisage for the light client in the browser? In particular, over which protocols and transports would you see discovery working and then how the actual light clients communicate with the full node peers?

over which protocols and transports would you see discovery working

In the browser you're limited to HTTP, Websockets and WebRTC. In libp2p, we can use the DHT (https://github.com/libp2p/js-libp2p-kad-dht) which works on top of any channel available to it, so WS and/or WebRTC. But we can also use other means of discovery like a rendezvous point and/or more exotic/ad-hoc things like gossip discovery.

Specifically right now, WebRTC star server that doubles as a rendezvous discovery mechanism - https://github.com/libp2p/js-libp2p-webrtc-star, this will change as we evolve the implementation.

how the actual light clients communicate with the full node peers?

The current vision is to have clients with a libp2p and RLPx/devp2p stacks running side by side and bridging the two networks. Not at all ideal, but doable.

Would it influence those scenarios and protocol choices in any way if the transmission of the Eth protocol messages was 'almost' transport agnostic (so long as the transport was reliable and streamed like tcp) but mandated aes-gcm encryption?

Yes, most definitely and I believe this is already the case. The application level protocols should (I haven't tried it, but I don't see why not) run on top of any stream oriented channel (raw sockets, websockets/webrtc, multiplexed, etc...). What caveats are there with aes-gcm encryption, if any?

I think it @vpulim input on this would be very valuable, as it appears that he's already done the work in https://github.com/ethereumjs/ethereumjs-client/. Also pinging @holgerd77.

[FrankSzendzielarz]

@dryajov Thanks again for the response. Well it's exciting to have this discussion for a number of reasons.

On Discovery, when trying to collect requirements for Discoveryv5 (see the rationale document in discv5 in this repo) one thing I tried to achieve was separate the protocol from the underlying transport, making it as agnostic as possible about if the underlying transport was streamed/message based or even reliable.

It is not yet documented but Felix has taken the discovery protocol even further by solving some issues related to identities where public keys are not recoverable from signatures and the impact on obfuscation, by designing a low-cost handshake that establishes an encrypted channel at the discovery level.

One thing that does seem to be happening now though is that again the direction seems to be drifting towards tying discovery to UDP only. I emphasise seems because I am not clear on that 100% and if so, why. If it is the case, then usually Felix has very good reasons for making certain design decisions. @fjl

However, that's kind of the point: up until literally a few weeks ago there was no real agreed common hang-out for identifying and consolidating the requirements for the p2p stack. Things have tended to evolve in isolation, and the underlying rationale has been opaque. But now we have the devp2p repo and an effort underway to have everyone share their voice in the direction and real world requirements here.

Further, up until recently it has always seemed to those on the core team that a very small minority of people had any interest in the p2p layers. My intuition though suggests that this is not the case, and it is more about that voices like yours and others just have not had easily accessible and open channels to connect with those working on the specs. (I do think that it would be a really good idea if the startup community established a working group in participation with EF and Parity directly to prioritise and channel goals and requirements.)

Hopefully this devp2p repo will now serve those aims to some extent, and any help you can offer in bringing in other stakeholders here would be much appreciated.

So with that, and putting aside the libp2p stack topics for a while (I will get back to that in time), am I correct in saying that really what we are discussing now is specifically a requirement that the p2p protocols should be more permissive and support a more diverse set of transports to facilitate implementing a broader range of light clients ? (If so it's a personal ambition of mine to help make light protocol serving and consumption easier and more widespread. )

If so, can I suggest you (better than if I do it, but happy to do so if you prefer) create a new Issue to that effect, where we can actually involve everyone in the discussion, invite more stakeholders, and really get in depth with it? I do have many technical things to say on the topic and I am sure @fjl will have too, but it would be good to collate them there and decide what the direction should be.

In advance of the new issue, some topics I want to discuss on that:

1. Drawing the line between minimum protocol requirements on Ethereum implementations and the toolsets (eg libp2p)
2. ENRs at the discovery level and how they drive multiaddr
3. Discv5 aes-gcm channel re-use for the eth protocol
4. Micropayment incentivization and relay-only nodes or implementation wrappers to make standard implementations support multiple transports

[GregTheGreek]

Cc: @Mikerah @wemeetagain

[ferranbt]

Just to clarify my points and respond to some of the topics brought.

I would not consider RLPx a streaming protocol. As I pointed before, a streaming protocol usually takes a stream of bytes as input, but, RLPx has a fixed interface (code, message). This makes it a bit unefficient, to send 1MB of data requires to allocate 1MB in memory. You can build more memory/cpu efficient methods with a stream protocol (here). It is true that you could build a stream protocol on top of RLPx but I that does not make it a streaming protocol (otherwise, UDP would be a streaming protocol too). My point was not to use TLS per se but to use the method TLS uses to split the data.

A streaming protocol interface for the transport can increase the number of protocols and use cases we can implement in the upper layers. The more abstract and low uses we do it the better and more things we can build on top. It seems that we may agree on this point.

I have no experience with LibP2P whatsoever but my fear is that by having a wide abstraction interface to incorporate different protocols you may end up underperforming with respect to using a specific stack.

[FrankSzendzielarz]

@ferranbt "This makes it a bit unefficient, to send 1MB of data requires to allocate 1MB in memory." As far as I can tell this not actually true with RLPx. Yes, it does require that messages be transmitted, but it does not require that the message be streamed from memory.

[ferranbt]

Not sure what are you referring about "streamed from memory". I talk about total allocated memory by the CPU. For example this and this. These operations require to have memory allocated the size of the message.

[FrankSzendzielarz]

@ferranbt Ah snappy compression, yes. I had forgotten about that. Good catch. Snappy compression, internally, I think works on 32k blocks , so in theory again you could implement the protocol in a streaming manner. But that is academic. And I think this is getting besides the point.

With the above discussion regarding light clients, I think it would be a good idea to split this into a new 'requirement' on devp2p, where we clearly describe the design goal as a new GitHub issue, so we can discuss what impact that should have concretely on discoveryv5 and on the protocols for eth, etc. (@dryajov)

I think similarly, because we are pretty much exclusively dealing with messages anyway, could I please ask you @ferranbt to elaborate on some of the use cases you envisage supporting? Maybe I am not getting the motivation here. I do agree that things can and should be done to improve things.

[dryajov]

I'll create a separate issue for - #71 (comment)

[dryajov]

• We will have an encrypted channel established at the level of discovery

I'm not sure why encryption has to be tied to the underlying discovery mechanism? Isn't it possible to treat discovery as a totally separate element in the stack from the transport channel? Otherwise I agree, the channel should be reused when possible - I'll elaborate bellow.

• I think the discovery protocol need not (should not) be tied to UDP

This is another great topic for a discussion issue.

Totally agree. Although I believe the reasons behind restricting discovery to UDP is usually related to the chattiness of DHTs and kademlia in particular. By restricting traffic to a low overhead transport such as UDP, and furthermore decreasing the size of the payload to fit in an ethernet frame (mtu <=1500b), according to the discovery v5 spec, the max packet size is 1280b, this is somewhat mitigated. This are valid reasons, but they should not limit entry of platforms that don't have UDP capabilities. If anything this should be advisory?

This further reinforces my feeling that discovery should not leak its internals to other components and instead expose a high level interfaces to be consumed.

• The encrypted channel may be re-used for any message exchange, so long as the underlying transport is selected appropriately for the 'capability'. Eg: eth messages with 10MB payload can be sent over tcp using the same aes-gcm channel, while the handshaking can continue in parallel over UDP.

👍

So, I do wonder, if this was the route to take, the ENR would specify what transport(s) was/were in use, on which ports, and for which capabilities. The choice of if to use libp2p or any other toolset would be an arbitrary one.

Absolutely! I think we're thinking along the same lines, I'll go as far as suggesting that ENR records should be the lowest common denominator that all stacks agree to consume, i.e. how they are located, retrieved and stored should not be a concern of the transport layer nor the multiplexer, nor some other part of the stack.

[subtly]

Hi @ferranbt, there's a simple solution to what you've proposed. Just put whatever your protocol is inside the message and use a messagetype of 0. RLPx is a streaming protocol with dynamically sized messages and framing (as opposed to fixed length). This way, and because the wireline is binary RLP, you can use your own encapsulation – such as protocol buffers, thrift, json, or whatever you want. If the framing doesn't work for you, you can also make your own. Re: limits, you'll see the same with UDP and TCP via MTU and other mechanisms.

@dryajov and others re: #87 , please do reconsider the idea of running everything in a browser. The last thing we need is the decentralized Internet to only be available via Google Chrome or Microsoft Edge – their support for cryptography sucks, piping everything over HTTP and TLS is a regression, the CA system is broken, and they can drop support for crypto, wss, etc. whenever they feel like it.

#89 Discovery wasn't intended to be restricted to UDP nor the transport restricted to TCP, but, it was intentional that this is how it was first implemented. The most significant factor though was that there wasn't enough time to develop the two protocols in such a way that they could run over either TCP or UDP. There are peculiar challenges that pop-up, like, what happens if there aren't enough nodes running discovery on one protocol or the other, and what happens if the peer you connected to before can no longer use one or the other.

#90 RLPx can be used for streaming and that was the intention on day one. Should run voice or video without any problems.

@ferranbt The benchmark probably has TLS using hardware-accelerated AES-GCM.

A lot of whats being asked for here is orders of magnitude easier said than done. Creating, implementing and upgrading transport and discovery protocols is quite difficult and requires quite a bit of time. Have fun implementing an ethereum client and do remember to mac-then-encrypt!

[FrankSzendzielarz]

@subtly What a lovely response. This just made my Friday evening. :-)

[fjl]

Looking back at this issue, libp2p has now moved toward TLS 1.3 as the 'best' default, with a Noise-based protocol as an alternative. I would certainly be happy to remove RLPx in the long run, and since ENR support is now available in most clients, we have the means to experiment with other transports.

At the same time, it's important to understand that the numeric code + binary data message format has worked extremely well for us, and I would prefer we keep that system. It'd be nice to have a specification defining a modern transport protocol which can carry the capabilities we already have and use.

[subtly]

Perhaps open a new issue, linking this one, since this is for TLS?

I don't think we finished what we started with the network layer but maybe... just maybe, we can carry on from where we left off. I was hoping to have met again by now, since Osaka, but the world had other plans.