ethereum/devp2p

Ethereum peer-to-peer networking specifications

This repository contains specifications for the peer-to-peer networking protocols used by Ethereum. The issue tracker here is for discussions of protocol changes. It's also OK to open an issue if you just have a question.

Protocol level security issues are valuable! Please report serious issues responsibly through the Ethereum Foundation Bounty Program.

We have several specifications for low-level protocols:

• Ethereum Node Records
• DNS Node Lists
• Node Discovery Protocol v4
• Node Discovery Protocol v5
• RLPx protocol

The repository also contains specifications of many RLPx-based application-level protocols:

• Ethereum Wire Protocol (eth/68)
• Ethereum Snapshot Protocol (snap/1)
• Light Ethereum Subprotocol (les/4)
• Parity Light Protocol (pip/1)
• Ethereum Witness Protocol (wit/0)

The Mission

devp2p is a set of network protocols which form the Ethereum peer-to-peer network. 'Ethereum network' is meant in a broad sense, i.e. devp2p isn't specific to a particular blockchain, but should serve the needs of any networked application associated with the Ethereum umbrella.

We aim for an integrated system of orthogonal parts, implemented in multiple programming environments. The system provides discovery of other participants throughout the Internet as well as secure communication with those participants.

The network protocols in devp2p should be easy to implement from scratch given only the specification, and must work within the limits of a consumer-grade Internet connection. We usually design protocols in a 'specification first' approach, but any specification proposed must be accompanied by a working prototype or implementable within reasonable time.

Relationship with libp2p

The libp2p project was started at about the same time as devp2p and seeks to be a collection of modules for assembling a peer-to-peer network from modular components. Questions about the relationship between devp2p and libp2p come up rather often.

It's hard to compare the two projects because they have different scope and are designed with different goals in mind. devp2p is an integrated system definition that wants to serve Ethereum's needs well (although it may be a good fit for other applications, too) while libp2p is a collection of programming library parts serving no single application in particular.

That said, both projects are very similar in spirit and devp2p is slowly adopting parts of libp2p as they mature.

Implementations

devp2p is part of most Ethereum clients. Implementations include:

• C#: Nethermind https://github.com/NethermindEth/nethermind
• C++: Aleth https://github.com/ethereum/aleth
• C: Breadwallet https://github.com/breadwallet/breadwallet-core
• Elixir: Exthereum https://github.com/exthereum/ex_wire
• Go: go-ethereum/geth https://github.com/ethereum/go-ethereum
• Java: Tuweni RLPx library https://github.com/apache/incubator-tuweni/tree/master/rlpx
• Java: Besu https://github.com/hyperledger/besu
• JavaScript: EthereumJS https://github.com/ethereumjs/ethereumjs-devp2p
• Kotlin: Tuweni Discovery library https://github.com/apache/incubator-tuweni/tree/master/devp2p
• Nim: Nimbus nim-eth https://github.com/status-im/nim-eth
• Python: Trinity https://github.com/ethereum/trinity
• Ruby: Ciri https://github.com/ciri-ethereum/ciri
• Ruby: ruby-devp2p https://github.com/cryptape/ruby-devp2p
• Rust: rust-devp2p https://github.com/rust-ethereum/devp2p
• Rust: openethereum https://github.com/openethereum/openethereum
• Rust: reth https://github.com/paradigmxyz/reth

WireShark dissectors are available here: https://github.com/ConsenSys/ethereum-dissectors

devp2p's Issues

encrypted datagrams

update: bootstrap specs

specs: Use of ECC

risk profile

define fallback timeouts for peers

Why does Ethereum uses a structured overlay for broadcasting while Bitcoin doesn't?

Does anyone have insights into why Ethereum uses a structured overlay (Kademlia) for broadcasting while Bitcoin uses a randomized graph?

LES Server Capacity Management - Specific Proposal

Proposal Summary

Note: this document and idea by Zsolt Felfoldi was originally at https://gist.github.com/zsfelfoldi/8d1dfa05ce1e4b50ea5fe1573fb461d6

This proposal describes a server load management approach, where light clients may be categorised as free or premium and given percentages of a dynamically assessed server capacity.

Server capacity

In the LES service model capacity is defined as a scalar value which is assigned to each client while the total capacity of simultaneously connected clients is limited (totalConnCap <= totalCapacity). The exact meaning of capacity values may depend on the server implementation and may change in subsequent versions. The behavior expected from the server is that in any situation the served amount of similar types of requests should be proportional to the capacity value. (Note: this behavior is also tested by https://github.com/ethereum/go-ethereum/blob/master/les/api_test.go). The server also specifies specifies a minimum useful value for client capacity minCapacity which in practice means a large enough flow control buffer that allows the client to send all types of requests.

The total capacity of the server can change dynamically for various reasons. The actual request serving capacity can change due to external circumstances and it can only be properly measured while actually serving a high amount of requests, therefore dynamic corrections are necessary during operation. Also if most of the server capacity remains unused most of the time then a certain amount of dynamic "overbooking" is acceptable (see below). For these reasons the total capacity may be changed at any time which also means sometimes some clients might need to be kicked out.

Overbooking server capacity

LES flow control tries to more or less guarantee the possibility for the clients to send a certain amount of requests at any time and get a quick response. Most of the clients want this guarantee but don't actually need to send requests most of the time. Our goal is to serve as many clients as possible while the actually used server capacity does not exceed the limits. More exactly, the sum of assigned capacities of those clients nodes whose flow control buffer is currently being recharged (sumRecharge) should not exceed the actual request serving capacity which is controlled by limiting the total amount of flow control buffer recharge (totalRecharge). If totalCapacity == totalRecharge then there is no overbooking and sumRecharge can never exceed totalRecharge. If totalCapacity > totalRecharge then it is possible but the server may predict that this is unlikely to happen (especially if its client management/pricing strategy disincentivizes excessive request sending) and allow a somewhat higher amount of clients to connect. Should it happen anyway, the server can still disconnect the least important and/or most agressive clients and thereby instantly reduce sumRecharge under the desired limit.

Even though this is an undesirable event since it weakens the guarantee the server tries to provide, its average frequency of occurence can be limited to arbitrarily low levels if totalCapacity is also immediately reduced when clients are kicked out and it is only gradually increased at a limited rate when totalConnCap is close to totalCapacity and sumRecharge is still well below totalRecharge. totalCapacity should also be limited to maxCapacity in order to avoid mass disconnections after a low-usage period driving totalCapacity to extremely high levels.

Client management strategy

We identify two classes of clients: "free" and "priority" clients. Free clients are those who the server does not know anything about. Priority can be assigned to certain clients through the API (along with a capacity value). Priority clients can always connect as long as the total amount of connected priority client capacity (totalPriConnCap) does not exceed totalCapacity. They are only kicked out if totalCapacity is dynamically reduced under totalPriConnCap. In this case the API gives a short time window for the managing endpoint to reduce the assigned capacity of some priority clients (or at least decide which one to kick out) before the system disconnects one of them automatically.
Free clients get the minimum amount of capacity if there is enough free capacity for them or if there is another free client which can be kicked out. Although free clients don't pay for the service a virtual cost balance is kept for the recently seen ones which is used as a negative priority. Free clients from recently unseen IP addresses have a higher chance of connecting and they can even push out those which received service recently. Free clients with the highest negative priority are also kicked out first if a priority client wants to connect or if totalCapacity is dynamically reduced.

Time cost and request cost

Cost of service (either virtual or actual) consists of two components: the cost of having a live connection (proportional to connection time and capacity) and the cost of served requests (proportional to the sum of realCost of served requests). For actual micropayment incentivization different pricing strategies may be applied so the API provides information about both types of resource usage per client. A suggested strategy for choosing the relative weights of time and request costs:

• while totalCapacity is limited by disconnections and is therefore less than maxCapacity, request cost should dominate
• if totalCapacity == maxCapacity and sumRecharge is consistently well under totalRecharge then time cost should dominate

The API

Global parameter queries

The system provides the following global capacity values which could all be made accessible through the API:

• minCapacity
• freeCapacity (capacity assigned to free clients, currently equals minCapacity, should we expose it separately?)
• totalCapacity
• totalRecharge (sounds a bit technical, should we name it something else, like baseCapacity ?)
• maxCapacity
• totalConnCap
• totalPriConnCap
• totalPriCap (all assigned priority capacities whether connected or not)

Questions:

• should the API focus on priority clients entirely and make the free client pool completely transparent? (in this case that would mean omitting freeCapacity and totalConnCap)
• should I add separate queries for these or make a single capacity query that returns all of these in a map?

Individual priority client capacity assignment

• setClientCapacity(id, cap)
• getClientCapacity(id)

(zero means no priority status)

Client list queries

• listClients(filter) (returns a list of (id, capacity) pairs)

Filter can be either of the following:

• all connected (may be omitted if we care about priority clients only)
• all priority assigned
• all priority connected

Question: should we provide separate query functions instead?

Event subscription

The following types of events might require a subscription:

• reduced totalCapacity may require action from the endpoint if it goes under totalPriConnCap
• client connecting/disconnecting (only for priority clients? or use optional filter?)
• regular updates about priority client resource usage (send an update after a certain amount of usage)
  • alternatively just use polling? maybe add this info to the priority client list query?
  • or provide an alarm function that the endpoint can set up so that it gets an event when the client's balance is expired?

I propose a single subscription with some filter options and a general event format:

• totalCapacity, totalConnCap, totalPriConn
• client id (if it is a client event)
  • capacity
  • isPriority (only if we allow filtering for non-priority clients too)
  • event
    • "connented", "disconnected", "update"
  • timeCost, requestCost (total since connection)

The event filter options depend on the answers to the questions above so I will try to finalize them after figuring out those.

Low-level transport protocol based on TLS+yamux

The current devp2p stack provides an inefficient interface to protocol applications that end up having to use a fixed message format to communicate. This is due to the fact that the layers that compose the stack are not independent from each other. This document proposes a different modular stack based on the abstraction layers mechanism followed in the OSI network protocol stack.

In the OSI model [1], different protocols are built one under another, each layer designed with a single purpose in mind. This modularization makes design and evaluation easier. Upper layers are unaware about the logic of lower layers and communicate with a specific interface. This transparency makes it possible to build FTP, DNS or TLS under the same TCP interface.

The new proposed stack has two layers. First, the security layer establishes a secure connection with a remote peer, it is based on TLS. Second, once the connection is established the signal is multiplexed in different streams. Both of this layers take as input a batch of binary data without any specific format. Thus, it is possible to change any layer at any time. Application protocols can use this flexible input format to build any type of communication protocol: FTP, SSH, RPC or [2].

Objectives

• The transport protocol should care exclusively about securing a connection with the other peer and providing different streams for the protocols. It should not introduce any other logic like flow control, congestion avoidance... It should be up to the higher layers to introduce this logic if they require it on their application protocols.

• The transport protocol should be application protocol independent. Higher-level protocols can layer on top of it transparently.

DevP2P

A small recap on the workings of devp2p. First, both peers start a handshake process to agree on a set of mac and cipher stream keys. Second, a secure transport connection is established with the former keys. Messages are sent with the format (code int, data bytes), each message is compressed, hashed with a mac and encrypted before being written to the insecure channel (RLPX protocol). Third, the peers agree on a set of protocols (i.e. eth64) and for each one start an independent stream on the secure channel (multiplexing). The protocols send messages to the stream also with the format (code, data). The code in the message is used to send the data to the correct stream.

Limitations:

• Rlpx macs and encrypts all the message at the same time. This may be a burden for big messages.

• The upper protocols that use the transport (i.e. eth64) end up having to use a fixed message format (code, message). This limits their flexibility and the type of communication they can use.

Proposal

This new transport proposal follows the same methodology as devp2p but applies a different set of protocols for each layer.

Handshake is still TBD. It is assumed that the peers agree on a set of mac and cipher stream keys. Next, the TLS record protocol [3] secures the application data. It behaves similar to RLPX, it macs and encrypts the data and then writes it to the insecure connection. However, the main benefit is that it divides the outgoing message into manageable blocks and reassembles incoming messages. Thus, it consumes less memory. The only input the record protocol takes is a message (bytes) to be transmitted.

We then use the yamux protocol [4] to multiplex the secure signal into different streams. Those streams are the ones exposed to the upper layers (protocols). Again, the created streams only take bytes as input.

Benefits

• It provides a flexible interface for the protocol backends to implement any type of exchange mechanism (i.e. rpc, pub/sub).

• It uses well known and production hardened protocols (TLS record and yamux).

• Encrypting and sending data in blocks makes it more performant in terms of speed and memory. Benchmarks show that TLS is 10x faster than RLPX.

• Every major language has already optimal implementations of TLS and the record protocol.

• It would be faster and easier to build new protocols [5].

References

[1]. https://en.wikipedia.org/wiki/OSI_model
[2]. #70
[3]. https://docs.microsoft.com/en-us/windows/desktop/secauthn/tls-record-protocol
[4]. https://github.com/hashicorp/yamux/blob/master/spec.md
[5]. https://github.com/yperbasis/silkworm/blob/master/doc/sync_protocol.pdf

Specify Discovery v5

The existing discovery protocol has been around for a long time and has it's shortcomings. Some of those are listed in the specification.

There are a few things we want in the next version of the discovery protocol:

• Being independent of the clock
• Making traffic amplification prevention less weird
• Relaying more node metadata
• Indexing nodes by their capabilities
• Obfuscating discovery traffic

Some of these things, specifically the 'node metadata' parts are solved by ENR. We can even fit them into the existing protocol using the ENR extension.

We also have a prototype implementation of the indexing mechanism in go-ethereum, and a pretty clear idea about the wire protocol. But nothing is set in stone yet.

What remains to be done is:

• Writing the wire protocol spec as an EIP.
• Specifying the upgrade procedure. We will likely announce the version through ENR but need a document that says what the key/value pair for that is and how the protocols can coexist on the same UDP port.

If you want to start implementing discovery v5 now, it is best to start with ENR and the v4 extension.

---

Remaining tasks for this tracking issue:

• Commit initial wire protocol draft
• Commit initial requirements document draft
• Commit topic table spec draft
• Move ENR specification to this repo
• Write short explainer document that ties all the specs together

recommendation(s) for nonce values

finalize encapsulation of node endpoint details

The current specification only allots for a single node endpoint. It should be a list and/or concatenated in order to support 1 UDP and 1 TCP endpoint.

update: actions when manually connecting to node

Meta Capabilities

Abstract

This document proposes a standardized handshake and message/metadata format for peer-to-peer protocols. Implementing this standard in Ethereum-related protocols would make it easy to share mechanisms like flow control or micropayment incentivization. It could also replace devp2p protocol multiplexing which separates protocols completely and instead handle different protocols as not necessarily disjoint sets of supported messages or message/metadata combinations.

Motivation

The following ideas have led me to the current proposal:

• Abstract out the flow control mechanism from LES and add it as an optional and replaceable "hint layer" to messages. This would help experimenting with different versions of the mechanism and also help implementing simple clients by making the whole thing an optional performance enhancement and therefore not mandatory to implement.
• Add request IDs and (some version of) the LES flow control mechanism to ETH/64. This would make sense because syncing (especially fast sync) relies on heavily pulling data in one direction and therefore faces similar challenges. Adding the current strict and brittle version of the flow control to the main Ethereum wire protocol would probably be dangerous so going with an optional performance-optimizing hint mechanism would suit ETH/64 better too.
• Unify LES and ETH/64. Since these protocols share most of their messages and we are thinking about adding LES specific things to ETH anyway, it sounds sensible to have a single protocol. On the other hand these two modes of operation may require different prioritization and load balancing methods. Also ETH is more mission critical while LES is still being heavily developed. Having a common protocol framework would allow sharing some messages, some mechanisms and some code in the implementations but also allow different strategies (both in operation and in development).
• Serve Ethereum chain/state data from Swarm. Splitting chain data and state snapshots to smaller parts and storing them in the Swarm topology would theoretically make sense and could help with some use cases. Supporting the proposed protocol framework could also make it easier to realize interoperation between the two protocols.
• Micropayment incentivization. This feature will need some extra metainformation which could be added to LES in some specific form but it would be better to make it as flexible as possible. It should be easy to experiment with different payment methods and pricing policies because markets are not designed on a drawing board. Also, other protocols could share the same mechanisms.
• Moving towards Serenity and sharding. Serenity clients will also serve different roles and require either different protocols or (ideally) overlapping subsets of the same protocol which can still be developed more or less separately. These protocols could probably also benefit from the previously mentioned mechanisms. Sharing a common protocol framework (with a replaceable serialization format and underlying transport layer if necessary) could allow Serenity to benefit from existing infrastructure parts and make the transition easier.

Specification

Message format

Instead of the usual messageCode, messageData format we allow optional metadata in messageCode, messageData, [[metaCode, metaData], ...] format.

• messageCode: a non-negative integer indicating the message type and format. Mapping between message codes and protocol messages is decided during handshake.
• messageData: a single serialized object with a format defined in the relevant protocol specification.
• a list of metadata fields:
  • metaCode: a non-negative integer indicating the metadata type and format. Mapping between meta codes and types of metadata is decided during handshake.
  • metaData: a single serialized object with a format defined in the relevant metadata specification.

The only predefined message format is the announcement message with messageCode zero. This message contains a set of string to serialized object mappings and can be used for communicating available messages and metadata, their mapping, and any kind of protocol-specific parameters. Protocol handlers and other agents should "listen" on certain string prefixes and it is up to them to decide whether an announcement was meaningful and useful. Sending an excessive amount of meaningless or unnecessary announcements can result in disconnection.

The set of announcements is encoded in a tree format:

messageData = [name/prefix, [subTree1, ...], value]

where subtrees are encoded similarly.

All other messages are protocol specific. Metadata can be attached to any protocol message if it is understood and accepted by the recipient. Protocol message specification and metadata specification do not need to be aware of each other, it is up to the two actors on each end of the line to decide which combinations of message and metadata types are meaningful and useful for them.

Handshake process

Supported message types are identified by strings in the namespace under the "message/" prefix. Similarly, supported metadata fields are identified by strings with "meta/" prefix. During protocol negotiation messageCode and metaCode values are mapped to those message and metadata types one peer is willing to send and the other is willing to receive.

Instead of listing all capabilities of all supported protocols, the two parties first signal their intent to establish communication through a protocol, optionally also stating their intended role. For example:

• Alice to Bob: "connect/les/client"
• Bob to Alice: "connect/les/server"

After they agree about speaking les they list the message and metadata formats they are capable of sending, including the types of metadata they can attach to each of the messages (see the "Extension examples" section below). Additionally, protocol parameters can also be exchanged here. (Note that only a few messages and parameters of LES are shown here).

• Alice to Bob:
  • "meta/reqId"
  • "message/les/getHeaders" -> ["reqId"]
  • "message/les/getBlocks" -> ["reqId"]
  • "message/les/getProofs" -> ["reqId"]
  • "param/les/genesisHash" -> genesisHash
• Bob to Alice:
  • "meta/reqId"
  • "meta/flowcontrol"
  • "message/les/headers" -> ["reqId", "flowcontrol"]
  • "message/les/blocks" -> ["reqId", "flowcontrol"]
  • "message/les/proofs" -> ["reqId", "flowcontrol"]
  • "message/overload/stop"
  • "message/overload/resume" -> ["flowcontrol"]
  • "param/les/genesisHash" -> genesisHash
  • "param/flowcontrol/bufferLimit": bufferLimit
  • "param/flowcontrol/minRecharge": minRecharge

Finally, they list the message and metadata types they are willing to receive and their code mappings:

• Alice to Bob:

  • "map/meta/reqId" -> 0
  • "map/meta/flowcontrol" -> 1
  • "map/message/les/headers" -> [3, 0, 1]
  • "map/message/les/block"s -> [5, 0, 1]
  • "map/message/les/proofs" -> [16, 0, 1]
  • "map/message/overload/stop" -> [22]
  • "map/message/overload/resume" -> [23, 1]

• Bob to Alice:

  • "map/meta/reqId -> 0
  • "map/message/les/getHeaders" -> [2, 0]
  • "map/message/les/getBlocks" -> [4, 0]
  • "map/message/les/getProofs" -> [15, 0]

Note: the first integer in the message mapping list is the desired messageCode for the message itself, the subsequent integers are the metaCode identifiers of the metadata types to be attached to the given message type.

Mapping messages allows the other side to send those messages and therefore can be considered the end of the handshake process. Still, further announcements may be sent to update parameters (or even do further mappings) if the protocol specification and implementation allows that. In order to avoid excessive burden on all implementers of a protocol, such updates and late mappings should only be allowed by the protocol spec if they are needed for some use case. If handshake is restricted to the above format and there is a fixed set of supported and required messages and metadata then migrating an existing protocol to this standard should be easy.

Extension examples

An extension can include additional messages and metadata formats. Extensions may be applied to multiple protocols and they can be either optional or a requirement for connecting with certain clients. A few examples of LES mechanisms implemented as extensions:

Request ID

ReqID is a numeric field in request/reply type messages where the reply message simply mirrors the value received in the request. It was first added to LES as a fixed part of the message format which is an option for other protocols too but it can also be implemented as a metadata extension containing a single numeric value:

• "meta/reqId": reqId

LES-style client side flow control

LES flow control provides a feedback mechanism for clients to avoid server overload and ensure quick responses.

• "param/flowcontrol/messageCost/les/headers": [baseCost, perItemCost]
• "param/flowcontrol/bufferLimit": bufferLimit
• "param/flowcontrol/minRecharge": minRecharge
• "meta/flowcontrol": bufferValue

Overload protection

This is something soon to be implemented in LES. Currently the flow control system instantly drops a connection when the buffer is exhausted. This is very strict policy but it still cannot always avoid transient server faults due to external circumstances, in which case the situation can also be remedied only with dropping clients. Instead of instant disconnection LES will support "freezing" the connection for a few seconds and use the flow control feedback as a "hint layer" intended to avoid freezing or making its occurence sufficiently rare. It will also allow very simple client implementations to not implement flow control at all, making a step in the direction of the proposed modular protocol framework.

• "message/overload/stop": reqId (meaning: please stop sending messages now and do not expect a reply for your message reqId or any subsequent ones)
• "message/overload/resume" (meaning: now you can start sending messages again)

Add collaborators

LES: Server load management proposal - detail

Ref #66 - this adds detail to that general proposal. It is about defining common cross-platform data structures for both hints and configs

Load Management - Policies and protocol

Requirements

• General goal :

  • have the server side load management policy in a common format for all types of client implementation - make this an 'ethereum-scoped' format
  • this format or structure would be common to both server side config, rpc admin calls, and messages sent to clients as hints about the server policy
  • get initial general agreement through discussion here before moving it to EIP
  • apply this to various protocols or aspects of protocols: fast-sync, swarm, whisper
  • allow parts of the server side policy to be requested by clients as 'hints' , so clients can proactively manage the load they incur and avoid server policing
  • avoid unexpected terminations of connections, replace this with frozen connections that respond with 'hints' eg: overloaded, and/or warnings that eviction rules are being approached
  • allow for different types of client, such as paid, free, premium, etc..

• Throttling rules:

  • Rate-limit (token bucket rate limiter)
    • Allow for burst control eg: a rolling limit of 10 requests per second
  • Quota-limit (fixed time period limiter - eg: per hour, day, month)
    • Allow for subscription-like costs or server-wide limits
  • Multiple options for rule scope
    • Per IP:Port
    • Per Node ID
    • For server
    • Per….?
  • Multiple limit units
    • Request count
    • Accumulated message 'weight'
    • ..?
  • API request message types may have an absolute id and corresponding 'weight'. Eg: cap/version/message id : N. The 'weight' is determined by the server admin or indirectly via an automated process governed by the server admin. The 'weight' is akin to the SQL Azure concept DTU, which tries to be a general measure of load, taking into account memory usage, IO and compute.

• Hints and Warnings

  • The general protocol (devp2p) could allow clients to get parts of the throttling and eviction rules as hint messages. These could be supplied on request (initially, or polling for dynamic updates) and optionally on handshake. Which parts would be up to the server.
    • The message format would be common to all implementations because the payload would be parts of the common throttling/eviction rules spec.
    • Reply messages from the server can contain additional hint payload, such as :
      • Per client rate limiter status (token bucket depletion)
      • ..?
    • When a client exceeds the server policies that apply to it, devp2p response messages may include:
      • Rate limit exceeded (equivalent of http 429)
      • Quota exceeded
      • Server temporarily unavailable (equivalent of http 503) - server capacity exceeded due to temporary degradation of resources for the server

• Eviction rules

  • Allow admins to specify under what conditions a client can be evicted:
    • Ignoring too many rate or quota warnings
    • Blowing a rate limiter (0 warnings), as per current LES policy (discouraged)
      • ?
  • These are part of the policy common structure, used as both server config and message to client as a server policy hint

Data Model

On the basis of the above requirements, I would like to propose a data structure that would be common across Ethereum implementations, used to describe the capacity and eviction policies, both as part of server-side configurations, such as configuration files and administative APIs, and also as message body fragments sent to clients as hints.

Example Json

The following example expresses a number of capacity management rules. These would be given to servers to govern the devp2p capabilities, and passed to clients as hints.

To reiterate and clarify: a rate limiter is a token bucket (leaky bucket) limiter aimed at capping burst demand; a quota limiter is aimed at capping overall bandwidth over an absolute unit of time, eg: megabytes per calendar day.

For rate limiters the 'period' describes the refresh rate. So if a rate limiter is 10 calls per 60s, then every 60s the 10 calls will be refreshed (gradually or incrementally, depends on implementation). For quota limiters, every absolute period of time once and once only at the start of the period the limit will be refreshed.

All the following is at this stage fairly rough and intended as a starting point for further refinement.

This example below conveys the following information:

• There is one policy for LES
• The server treats clients as being one of 4 types: free, paid, greylist, banned. Info on the client status should be in some hint or metadata message (TBD)
• The server enforces a number of rate limiters:
  • 10 calls per 60s for free and paid nodes by Node ID
  • 624 total call weight for 60s for free and paid nodes by Node ID
  • 100 cals per 60s by IP
  • etc
• The server enforces a number of quota limiters:
  • 1000 calls or 9000 message-weight (whichever first) per calendar day in server timezone for free clients
  • ten times as much for paid
• There are some eviction rates and quotas, as numbers of violations per...
• The server specifies message weights. The default value is 0.

This would be provided to the server, and the server could also offer clients this in a hint message format.

{
    "capability-policies":[
        {
            "capability":"les",
            "clients": [
                "free",
                "paid",
                "greylist",
                "banned"
            ],
            "rate-limiters":[
                {
                    "calls": 10,
                    "period": 60,
                    "for" : ["free","paid"],
                    "per" : "NodeID"
                },
                {
                    "weight": 624,
                    "period": 60,
                    "for" : ["free","paid"],
                    "per" : "NodeID"
                },
                {
                    "calls": 100,
                    "period": 60,
                    "for" : ["free","paid"],
                    "per" : "IP"
                },
                {
                    "calls": 10000,
                    "period": 1,
                    "for" : ["server"]
                },
                {
                    "calls":1,
                    "period":60,
                    "for" : ["greylist"],
                    "per" : "NodeID"
                },
                {
                    "calls":0,
                    "period":1,
                    "for" : ["banned"],
                    "per" : "NodeID"
                }
            ],
            "quota-limiters" :[
                {
                    "calls": 1000,
                    "weight": 9000,
                    "periodunit": "day",
                    "period":1,
                    "for" : ["free"],
                    "per" : "NodeID"
                },
                {
                    "calls": 100000,
                    "weight": 900000,
                    "periodunit": "month",
                    "period":1,
                    "for" : ["paid"],
                    "per" : "NodeID"
                }
            ],
            "eviction-violation-rates":{
                "violations":10,
                "period":10
            },
            "eviction-violation-quotas":{
                "violation":100,
                "period":1,
                "periodunit":"day"
            },
            "message-weights": [
                {
                    "message": "les/1/getblockheaders",
                    "weight": 12
                },
                {
                    "message": "les/1/getblockbodies",
                    "weight": 62
                },
            ]
        }

    ]
}

Json Schema

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "capability-policies": {
      "type": "array",
      "items": [
        {
          "type": "object",
          "properties": {
            "capability": {
              "type": "string"
            },
            "clients": {
              "type": "array",
              "items": [
                {
                  "type": "string"
                },
                {
                  "type": "string"
                },
                {
                  "type": "string"
                },
                {
                  "type": "string"
                }
              ]
            },
            "rate-limiters": {
              "type": "array",
              "items": [
                {
                  "type": "object",
                  "properties": {
                    "calls": {
                      "type": "integer"
                    },
                    "period": {
                      "type": "integer"
                    },
                    "for": {
                      "type": "array",
                      "items": [
                        {
                          "type": "string"
                        },
                        {
                          "type": "string"
                        }
                      ]
                    },
                    "per": {
                      "type": "string"
                    }
                  },
                  "required": [
                    "calls",
                    "period",
                    "for",
                    "per"
                  ]
                },
                {
                  "type": "object",
                  "properties": {
                    "weight": {
                      "type": "integer"
                    },
                    "period": {
                      "type": "integer"
                    },
                    "for": {
                      "type": "array",
                      "items": [
                        {
                          "type": "string"
                        },
                        {
                          "type": "string"
                        }
                      ]
                    },
                    "per": {
                      "type": "string"
                    }
                  },
                  "required": [
                    "weight",
                    "period",
                    "for",
                    "per"
                  ]
                },
                {
                  "type": "object",
                  "properties": {
                    "calls": {
                      "type": "integer"
                    },
                    "period": {
                      "type": "integer"
                    },
                    "for": {
                      "type": "array",
                      "items": [
                        {
                          "type": "string"
                        },
                        {
                          "type": "string"
                        }
                      ]
                    },
                    "per": {
                      "type": "string"
                    }
                  },
                  "required": [
                    "calls",
                    "period",
                    "for",
                    "per"
                  ]
                },
                {
                  "type": "object",
                  "properties": {
                    "calls": {
                      "type": "integer"
                    },
                    "period": {
                      "type": "integer"
                    },
                    "for": {
                      "type": "array",
                      "items": [
                        {
                          "type": "string"
                        }
                      ]
                    }
                  },
                  "required": [
                    "calls",
                    "period",
                    "for"
                  ]
                },
                {
                  "type": "object",
                  "properties": {
                    "calls": {
                      "type": "integer"
                    },
                    "period": {
                      "type": "integer"
                    },
                    "for": {
                      "type": "array",
                      "items": [
                        {
                          "type": "string"
                        }
                      ]
                    },
                    "per": {
                      "type": "string"
                    }
                  },
                  "required": [
                    "calls",
                    "period",
                    "for",
                    "per"
                  ]
                },
                {
                  "type": "object",
                  "properties": {
                    "calls": {
                      "type": "integer"
                    },
                    "period": {
                      "type": "integer"
                    },
                    "for": {
                      "type": "array",
                      "items": [
                        {
                          "type": "string"
                        }
                      ]
                    },
                    "per": {
                      "type": "string"
                    }
                  },
                  "required": [
                    "calls",
                    "period",
                    "for",
                    "per"
                  ]
                }
              ]
            },
            "quota-limiters": {
              "type": "array",
              "items": [
                {
                  "type": "object",
                  "properties": {
                    "calls": {
                      "type": "integer"
                    },
                    "weight": {
                      "type": "integer"
                    },
                    "periodunit": {
                      "type": "string"
                    },
                    "period": {
                      "type": "integer"
                    },
                    "for": {
                      "type": "array",
                      "items": [
                        {
                          "type": "string"
                        }
                      ]
                    },
                    "per": {
                      "type": "string"
                    }
                  },
                  "required": [
                    "calls",
                    "weight",
                    "periodunit",
                    "period",
                    "for",
                    "per"
                  ]
                },
                {
                  "type": "object",
                  "properties": {
                    "calls": {
                      "type": "integer"
                    },
                    "weight": {
                      "type": "integer"
                    },
                    "periodunit": {
                      "type": "string"
                    },
                    "period": {
                      "type": "integer"
                    },
                    "for": {
                      "type": "array",
                      "items": [
                        {
                          "type": "string"
                        }
                      ]
                    },
                    "per": {
                      "type": "string"
                    }
                  },
                  "required": [
                    "calls",
                    "weight",
                    "periodunit",
                    "period",
                    "for",
                    "per"
                  ]
                }
              ]
            },
            "eviction-violation-rates": {
              "type": "object",
              "properties": {
                "violations": {
                  "type": "integer"
                },
                "period": {
                  "type": "integer"
                }
              },
              "required": [
                "violations",
                "period"
              ]
            },
            "eviction-violation-quotas": {
              "type": "object",
              "properties": {
                "violation": {
                  "type": "integer"
                },
                "period": {
                  "type": "integer"
                },
                "periodunit": {
                  "type": "string"
                }
              },
              "required": [
                "violation",
                "period",
                "periodunit"
              ]
            },
            "message-weights": {
              "type": "array",
              "items": [
                {
                  "type": "object",
                  "properties": {
                    "message": {
                      "type": "string"
                    },
                    "weight": {
                      "type": "integer"
                    }
                  },
                  "required": [
                    "message",
                    "weight"
                  ]
                },
                {
                  "type": "object",
                  "properties": {
                    "message": {
                      "type": "string"
                    },
                    "weight": {
                      "type": "integer"
                    }
                  },
                  "required": [
                    "message",
                    "weight"
                  ]
                }
              ]
            }
          },
          "required": [
            "capability",
            "clients",
            "rate-limiters",
            "quota-limiters",
            "eviction-violation-rates",
            "eviction-violation-quotas",
            "message-weights"
          ]
        }
      ]
    }
  },
  "required": [
    "capability-policies"
  ]
}

Node discovery and cademlia should use hash of public key as address

Since no independent bits assumption is guaranteed on EC public keys
peers' public key needs to be hashed to derive a node's address used in cademlia routing.
Also, in order to help storage persistence of address-driven dht nodes the address should be derived from a permanent public key not an autogenerated one.

nodes with multiple IP address

How would you deal with nodes that have multiple address?

EIP-868: Node Discovery v4 ENR Extension

https://eips.ethereum.org/EIPS/eip-868

This issue is for discussions of the discovery v4 ENR extension.

message type throttling

update: nat traversal

In it's current form the node discovery protocol works by ignoring endpoint information provided in ping packets. A feedback system should be designed so that nodes can be notified when nat is detected and keepalives are employed to keep the socket open (when appropriate); this can be added to the Pong packet. The functionality of utp/stun/ice/nat-pmp/upnp should be considered.

discv5 + encrypted channel : findnodes and topicquery response format

The current spec proposes that for a 'streamed' response , such as Nodes, the approach should be to just break up the overall message into MTU-sized datagrams and in the header include a {MessageNumber of N} message counter.

The rationale is twofold: 1) the recipient should know how many to expect without having to wait for what would be quite a long timeout 2) to support future message ordering

In the current prototype code, this field is missing. I suggest adding it back so that in smaller networks or when certain buckets are underpopulated , or when querying for very close buckets (bucket 30 in a 32 bit id for example) , the recipient does not have to stall for seconds.

Also there is the scenario that because of UDP lossiness a datagram might be lost. If the sender tries to send multiple datagrams as a batch (eg: bucket nodes parts 1, 2, and 3) and part 2 is lost, the higher level protocol or the specific implementation might want to know if the full set was received , if the peer is reliable, and in future may even wish to re-request the lost message.

Similarly for TopicQuery responses.

spec: response handling

discv5: choose findnode parameter

In discovery v4, FINDNODE queries for nodes close to a certain public key.
In the discovery v5 draft, we made it query for nodes in a given bucket instead.

While implementing discv5, I noticed that querying by bucket doesn't work at all for small networks (i.e. ones with 10 nodes or less) because FINDNODE will just receive an empty response for most queries.

Should we extend FINDNODE to also return nodes from neighboring buckets or switch back to querying by hash?

node IP updates

See also #16

Document EIP-8 and EIP-706

These EIPs amended the specifications and have been accepted long ago. We should work their content into the specs.

Possible use extendable-output functions SHAKE128 or SHAKE256 in the MAC

currently we are not and we have to aes to encrypt the data then hash it to produce the MAC. If we switched to SHAKE we could possible just hash the incoming data after initializing Keccak with some secure data derived by the key Exchange

handshake material

xor vs concatenate

add recipients to node discovery datagrams

discv5: define ENR exchange

The recent handshake proposal in #60 ensures that ENR of the initiator of a discovery request will be shared with the recipient. The spec also includes the PING and PONG messages which can exchange the node's respective ENR sequence numbers.

We still need to define how ENR 'sync' will work. When a node generates a new record, its neighbors should become aware of the new record quickly. Should nodes 'push' their new record or have it 'pulled' by neighbors as part of liveness checks? Should PING / PONG include the record or should we have an explicit REQUESTENR packet like in EIP-868?

Another question is how requesters get the record of the recipient during handshake. We assume that some version of the record is known to the requester before sending its first packet, but it might not necessarily be the latest version. Should information about the recipient's ENR sequence number be included in the handshake? If so, should the requester pull the new record after the handshake?

Consider alternatives to XOR distance metric

As far as I know, Ethereum uses a variant of Kademlia protocol. The distance of two node is based on the common prefixes of node ids (hashed actually).

In this way, the whole space of node ids are divided into two branches, one branch ids starting with 0 and another branch ids starting with 1. In theory, the nodes in branch 0 have near neighbors only from branch 0 and nodes in branch 1 have near neighbors from branch 1. Therefore, there is no neighbor connections between branch 0 and branch 1. In practice, bootstrap nodes have eased this problem probably.

I found this issue when I was designing my own blockchain algorithm/protocol. My fix is pretty simple as well, replacing xor metric by hamming distance, i.e. changing from distance = id1 xor id2 to distance = sum bits of(id1 xor id2). With hamming distance, the network has the same diameter as xor metric, but not partitioned.

P.S. I post this on ethresear.ch yesterday, but maybe here is a better place for it.

licensing and other legal concerns

This repo should have a clear license and clear message to contributors under what license they are contributing works.

For specifications, CC-0 is one option.

However, patent coverage is something else that needs to be considered. I have brought the patent issue up for EIPs as well, so consider this issue just to be licensing for now.

discv5: questions regarding wait period and topic search

(1) In topic-advertisement-storage section:

target-ad-lifetime = 600 # how long ads stay queued (10 min)
target-registration-interval = target-ad-lifetime / queue-length
min-wait-period = 60 # (1 min)
control-loop-constant = 600

period = time-of-registration - time-of-previous-registration
new-wait-period = wait-period * exp((target-registration-interval - period) / control-loop-constant)
wait-period = max(new-wait-period, min-wait-period)

In the second to last line wait-period was referenced before assignment.

(2) In the second paragraph of topic-search section:

To find nodes, the searcher generates random node IDs inside the topic radius and performs recursive Kademlia lookups on them. All (intermediate) nodes encountered during lookup are asked for topic queue enties using the TOPICQUERY packet.

Why are random generated node IDs used instead of just the hash of the topic(keccak256(T))?

determine initial peer connection strategy

With regard to TCP peer connectivity:

what occurs when a node is added or removed
what occurs when a peer disconnects
what occurs during discovery
what occurs during after temporary disconnect (disconnected < 60 seconds)
what occurs during after short disconnect (disconnected < 3600 seconds; based upon idle refresh interval)
what occurs during after long disconnect (disconnected >= 3600 seconds)

key rotation

spec / implementation discrepancy for ping and neighbours packets

spec of structure of ping & neighbors packets not according to implementations. or the other way round.

5275f56#diff-9999caa615e6404632c336aa8177c9edR108

rename sequence to strand

discv5 + encrypted channel: protocol documentation about cypher

The current prototype suggests that AES-GCM mode might be replaceable with some other cypher.

I was looking into AES-GCM and realized that it's a chained cypher for fixed-length messages, though it does support streaming of those messages. In other words, the counter-mode behaviour of the cypher is re-started for each message sent.

This is perfectly fine for transmission over UDP as long as the message is no more than one datagram in size.

Because UDP datagrams can be received out of order, and because they can be lost, if an implementation were to choose to support AES-CTR over a continuously streamed channel (like , with RLPx over TCP) then the datagram would need to include a plaintext message number, so that the correct keystream could be generated by the recipient, and would also need to implement a buffer window to re-order messages (like in DTLS). (Plaintext headers would re-introduce the need for obfuscation if we were looking to defeat certain DPI scenarios, though I don't know if that's a need).

While the above is theoretically possible, it would need modifications to the protocol.

I am not necessarily proposing that modifications to the protocol should be made, but I think if we are going with the current direction (aes-gcm with message-based cypher), then the documentation for the new version of the protocol should highlight the limitations of the transport and what types of cypher are possible.

(Thinking about streaming in general, and some of the other discussion around Eth 2.0 and also comments about RLPx, some of the proposed alternatives talk about TLS-like behavior, where the stream is broken in to TLS-Records and the cypher applied to those records. The disadvantage compared to the current discv5 prototype is that over a lossy transport, lost TLS-records would either mean some kind of complex re-requesting mechanism or because the TLS-record boundary would not fall on a higher-level message boundary complete messages would be impossible to decrypt. In the current prototype we can send multiple datagrams (eg node records in response to findnode) and if one of those datagrams is lost we can still continue to decrypt and process the other datagrams . With a 'message number X of Y' field in the message, higher level protocols can decide if the message was lost and to re-request or not.)

why to two MAC's ? one for header and one for the body

can't we just get away with one?

LES: client load management discussion

Proposal Summary

This proposal reviews the current load-management mechanisms for light client servers, namely LES flow control and the proposed server capacity management, to offer a more general and simplified direction. The proposal focuses on modifications to light protocols to simplify client implementation and moves towards defining a commonly-agreed server throttling, eviction and costing configuration format.

This general idea is extended with a detailed config/message format proposal #74

Overview

There are a couple of documents floating around that have a go at specifying how ethereum 'light-client servers' should handle 'light-client' capacity and load:

1. A flow control document

2. A server rate limiting document

Parity Tech seems to have its own LES message spec too, but it says little about client/server rate limiting.

The rest of this file contains:

1. The TL/DR of my opinions/recommendations on this topic of light server capacity regulation.

2. A summary of each of the above documents along with a collection of my opinions on the LES capacity management recommendations in the above two links.

The aim is to provide a review of the current proposals, get a bunch of people involved in discussions around this doc and adding issues on github, in the hope that all those people from different elements of the Ethereum community will quickly agree on a way forward for LES that is common to all implementations.

A note: 'client' can mean an Ethereum implementation or it can mean a light-client peer in a client-server role in the p2p network. Depends on context. We should come up with some better terminology.

Proposal (TL/DR)

1. Server operators would most likely prefer to see a setup where they can specify their throttling approach in some configurable and flexible set of 'policies'. Azure, for example, offers a number of types of policy configuration, where a config file is used to specify if rate limiting is per IP, per user, per client etc., what total quotas should be over longer periods, and so on. If a server wants to specify that certain groups of clients should be burst-limited to N requests per second on average but with total M megabytes bandwidth per day, that can't be achieved with the below approaches.

2. I propose concentrating on defining a commonly agreed-on policy configuration format that is general enough to encompass burst-control rate-limits (token buckets), periodic quotas (requests or bandwidth per day for example), eviction rules, response times, and client categories (free, premium, etc)

3. A common general format allows for flexibility in implementation, and allows consensus to be arrived at more easily between Geth, Parity and other client implementers, while also being potentially useful to other protocols, such as Swarm.

4. The throttling/eviction/etc policy (which could include per message costs for example) could be provided by the server as a hint to the client. I think this could be an optional request/reply call to the server and agreeing the message format across the implementers would be trivial as it would mirror the config file. It need not be part of handshakes.

5. Unlike the proposals in the documents below, I do not think there should be any obligation on clients to try and maintain a mirror of the state of each server's token bucket or other limiting algorithm to work out how to throttle calls to that server. It complicates client development. A far simpler approach is to have the server issue a warning that some eviction rule is about to be violated. The server can then make a dynamic assesment of how to handle that. The client could also receive messages that the server is overloaded, which is unrelated to the throttling policy. The main body of the work would be getting agreement on the elements to include in a policy configuration structure and their meanings (rate limit vs quota vs response time etc), and specifying a couple of control messages or response codes (server overloaded, etc) and a request/reply message exchange to obtain the policy from the server if desired.

6. I think we need an agreed set of higher level design goals that declare something like "implementing light clients should be really easy because we want the Ethereum ecosystem to expand" and that "server operators should be able to choose their own throttling approaches and not be limited to client implementation mandates."

Flow Control Review

Document Summary (Flow Control)

The document suggests that the design goals include the following:

1. Servers should be able to limit the 'amount of work' done for a light client.

2. Some sort of feedback should be given to the client about server load, to help them decide where to send requests when connected to multiple servers.

3. Clients should have some incentive to get their requests properly served.

With those design goals, the document proposes a model that can be summarised as:

1. Have servers assign different request types a 'cost'

2. Have the server implement a leaky-bucket rate limiter per-client

3. Incentivise the client by dropping the client connection (and forcing the client to reconnect) whenever the token-bucket is depleted. Incidentally, this forces the client to implement the token-bucket in a way that mirrors the server in order to avoid being evicted. Hints are provided as an initial request cost table message in the handshake, along with per-call updates in the reply about the rate-limiter status on the server.

Opinions (Flow Control)

1. One of the design goals should be to make it simple for light client implementers to join the network. Adoption of Ethereum will be enhanced when it is simple to discover a LES server and begin interacting.

2. Forcing light clients to mirror token-buckets is additional complexity. The client needs to mirror each connected server's token-bucket rules and parameters, depending on the connected server's implementation nuances and configuration. The model seems to make it hard to guess how to manage concurrent requests, especially when these are driven by UI events in a light client app.

3. Light clients already have sufficient incentive to find responsive servers. Disconnecting a client because they depleted the token-bucket is too draconian.

4. This additional complexity is not justified as a means of incentivising clients to get reliable responses.

5. The terms 'cost' (of a request) and 'the amount of work' (done by a server) attempt to express the same concept or metric. In reality each request will incur differing costs on each server or type of server depending on the relative importance and price of compute/memory/diskIO/networkIO and how each request type incurs those.

6. Given 5., it is potentially useful for clients to understand the relative cost of each request type and the status of the server rate-limiter. This extra information however certainly doesn't feel essential, more of a nice to have for client apps wishing to manage user interaction in a graceful way by spreading calls out across servers dynamically. I would prefer to see a response message along the lines of http 503 (temporarily unavailable) or 429 (too many requests) on the first call that is throttled. This brings client development more in line with usual API consumption.

7. Another minus to consider, though probably unimportant, is that expressing the cost table could identify the nature of the server, as some information about the relative balance of IO/compute/etc can be revealed.

8. Disconnecting a client can incite the client to attempt a reconnect. The net effect on the server might be counterproductive.

9. The handshake establishes the per-request type cost table, but real server cost can change dynamically.

Server Capacity Management Review

Document Summary (Server Capacity Management )

The document is quite hard to read if you are not already in context, so the following summary is just my understanding of it.

1. It proposes a model for regulating light client 'capacity' assigned as a proportion of overall server 'capacity' (note that 'capacity' is not defined) as part of a server-scoped throttling control.

2. It proposes that the server should be 'self aware' of both its actual maximum capacity and its average historical capacity usage by some kind of unspecified self-monitoring method.

3. When total server capacity drops enough that it exceeds the total minimum required capacity of light clients currently connected, it suggests that some of those clients should be evicted.

4. When the server capacity is underused most of the time it proposes that 'overbooking' of the capacity can be permitted to allow more clients to connect than normal.

5. It introduces a distinction between free and priority clients, and some rules about how to assign capacity and evict free clients.

6. It introduces a notion of 'cost of service,' being composed of the cost of reservation of capacity for the connection and the actual cost to the server of each request. It's not clear to me what this notion is then used for.

Opinions (Server Capacity Management)

1. I am not sure who the document is aimed at. While perhaps a nice guideline for a version of a specific implementation (eg: geth), I don't see why or how all implementers should feel compelled to implement their server capacity management in this way.

2. The real cost and real capacity of the server depends on how each type of request affects the different performance characteristics of the server. How does the server know that memory reduction is going to affect total capacity for example, and how does it calculate the costs?

3. Forced evictions made according to fluctuating server capacity or the joining of new free/priority light peers just makes light client development hard and usage unreliable. I think it is preferable if all stale connections get dropped after some period of inactivity, or whenever a client explicitly disconnects. If the server is unable to serve some of the connected light clients because of a transient fault, what rationale is there for converting that transient fault into a set of disconnections and terminating their service?

4. Clients could just be notified that the server is overloaded. It would be up to them to find new servers. Persistent server overload or denial (lack of payment for example) would eventually cause the client to look elsewhere. The client would abandon their connection, which would eventually be closed.

5. If the document implies that it is paving the way for micropayments, again I think we should look for inspiration from the cloud providers like Amazon and Azure. Clients could pay for burst capacity reservation (as suggested in the document) and/or per-request. This could be established in the handshake as a payment channel.

6. In general I feel again this is a detailed specification that prescribes a narrow policy. I would suggest focussing on a server-scoped rate-limiter/quota policy that follows the same kind of format as the per-client policy.

7. Administrators would just be able to specify types of client (eg: priority, free, etc) and their associated throttling policy. A global throttling policy would 'feed' the per-client policies.

remove padding

CTR mode doesn't require padding and the block-cipher and padding must be defined by the block-cipher/mode rather than RLPx.

libp2p integration

libp2p is a modular framework of peer-to-peer networking components,
implemented in several languages. Specs can be found here.

We want to have a shared transport protocol with libp2p to enable interconnectivity
between the IPFS and Ethereum networks. Transport protocols specified by libp2p
could be used as a replacement for RLPx.

See also libp2p/libp2p#33.

EIP-778: Ethereum Node Records

https://eips.ethereum.org/EIPS/eip-778

This issue is for discussions about ENR.

spec: ignore unsolicited messages

discv5: transport pre-negotiation packet (LETSTALK)

This is more of a discussion starter / idea than a concrete proposal:

Although it's not yet documented, the discv5 spec seems to be evolving towards encrypted aes-gcm based communication, rather than obfuscation and recoverable signatures....

Assuming we get there, the following idea:

0. The discv5 protocol now involves a quick handshake of secrets and keys from ENRs, before any kind of Kademlia communication can happen
1. Aes-gcm allows parallel communication
2. Can the same shared secrets and cypher ('channel') be re-used for TCP?
3. Because it is aes-gcm, large messages can be streamed.
4. This would allow ETH (etc) protocol to work without handshake.

Regarding libp2p - other implementations that want to use libp2p for flexibility and interoperability could do so (eg: polkadot) - and they could be able to communicate with nodes that use such a mechanism as proposed above (???). Eth 2.0 will use discovery v5 anyway in all likelihood.

However, for implementations that want to deliver a single performance optimized method of networking that employs a common udp+tcp handshake, the discovery 'channel' (pardon the pun) and this aes-gcm encrypted communication would be all that was needed, as long as the transport was reliable. Which particular transport outside of TCP could in future be determined by ENR (eg: ETH = tcp 30303, http 80 etc)

Thoughts?

update: nodeid is static between sessions

The local node alias (private key), nodes, and peers MUST be saved and restored between sessions (network state file/database). The identifier can be reset by deleting network state file/database. An option to reset identifier from UI or CLI is NOT recommended.

Dependencies:

• kademlia: availability is inverse to malice
• peer reputation
• peer connection strategy

Effects:

• reduces network churn
• reduces convergence overhead from major network events (splits)
• [future] reduced convergence overhead for DHT
• tofu: trust on first-use

DVE-2015-0099 AES CTR man-in-the-middle through keystream reusage

The two sides of a RLPx connection generate two CTR streams from the same key, nonce and IV.

If an attacker knows one plaintext, he can decrypt unknown plaintexts of the reused keystream.

Separate keys needs to be used for each stream. See for example the TLS 1.2 RFC 5246 section 6.3.

p2p effort consolidation

It is becoming difficult to track the different p2p efforts.

We now seem to have many different groups working on devp2p wire protocols in different repos.

Some people working on eth2 p2p:
https://github.com/ethresearch/p2p/issues

General repo intended to be everything ethereum p2p:
https://github.com/ethereum/devp2p/

Discovery protocol for all future versions:
https://github.com/ethereum/devp2p/tree/master/discv5

Another group of people working on eth2.0 p2p:
ethereum/consensus-specs#692

There are also now multiple proposed wire formats for different places. RLP, SSZ, SOS, SSS, each with differing types of design goals and differing clarity in design goals.

Recent convo with @felix Lange (fjl) we agreed that ethereum/devp2p should be the label for 'all things p2p' in Ethereum. Do we agree with that , and if so, I would like to kick off some effort to consolidate and focus these groups- eg: by moving the different draft specs under ethereum/devp2p and having all the stakeholders converse there.

EIP-1459: Node Discovery via DNS

https://eips.ethereum.org/EIPS/eip-1459

This issue is for discussion of the DNS-based discovery mechanism.

Move subprotocol specs into this repo

I would like to have all known subprotocol specs in one place. Protocols I know of are

• eth: everyone implements this one
• les: geth and a few others have it
• shh: many versions, it's unclear to me which one is canonical
• bzz, hive, discovery, stream: part of swarm

pong hashes should be validated

discv5 packet signatures / handshake

In discv5, we aim to support multiple cryptosystems for node identity. These are called 'identity schemes' in the ENR EIP. In the v4 wire protocol all packets are signed by the node's secp256k1 identity key. If we can't assume secp256k1 identity anymore, how do we create and verify packet signatures?

A couple options below, please submit more:

1. We can leave off the signature most of the time if the 'conversation nonce' is strong enough. The signature would only be needed on packets that mess with conversation nonce like ping, iAm, ...
2. Specify something like a 'signature type' and define the verification such that it can be traced back to the node id. For the "v4" identity scheme (which is secp256k1/keccak256 as used in v4) this would be verify(sig, id) -> keccak256(recover(sig)) == id. It's unclear to me how well this works with ed25519 identities because recovering the public from the signature isn't a commonly exposed operation for this curve. If signature recovery isn't possible (it likely isn't), we'd need to know the sender node id for every packet to even verify the signature, something we could avoid so far. Maybe whoareyou/iAm packets can help with this.
3. Maybe sender node id and id scheme should be part of the packet header.
4. Defer the whole thing for later and require secp256k1.