[Access] Relay messages to unstaked network directly

[synzhu]

[Access] Relay messages to unstaked network directly

[vishalchangrani]

I believe this is an improvement not only for the unstaked side of the network but also on the staked side of the network where multiple consensus nodes are publishing the same blocks.

[synzhu]

So I've done some more digging, and it turns out we won’t be able to access the GossipSubRouter since it is a private field. Thus, we can't directly relay messages by calling GossipSubRouter.Publish as hoped.

However, I realized that we may be approaching this completely wrong in the first place.

First of all, on the staked access node, we're creating two entirely separate instances of Network, which in turn creates two entirely distinct libp2p Nodes

• https://github.com/onflow/flow-go/blob/master/cmd/access/staked_access_node_builder.go#L77-L90
• https://github.com/onflow/flow-go/blob/master/network/p2p/middleware.go#L151

I'm guessing the reason it's implemented this way is because conceptually we are creating two "separate" networks, but in fact I'm not sure we have to explicitly separate them this way.

Here is a method that allows us to register a topic validator:

https://github.com/libp2p/go-libp2p-pubsub/blob/master/pubsub.go#L1297

The validators are run before any message is sent or gossiped-forward on the network

• https://github.com/libp2p/go-libp2p-pubsub/blob/master/pubsub.go#L1083-L1085
• https://github.com/libp2p/go-libp2p-pubsub/blob/master/validation.go#L228-L244

What we could to instead, is the following:

• For all staked AN's, we register a validator that checks if the message originated from a node who is in the protocol state (ie a staked node). If it is, then we can allow the validator to pass.
  • What this implies is, that messages from the staked network will be gossiped-forward to the unstaked nodes. This is handled automatically by libp2p, and we don't need to do any extra work.
• If the message did not original from a node that we recognize from the protocol state, we fail the validation. At the same time, we intercept the message (since it must be from an unstaked node) and process it as we should.
  • This implies that messages from the public / unstaked network won't be gossiped-forward to the staked network.
  • At the same time, the staked AN's themselves will receive the messages from the unstaked network, and can process them however it needs (respond to sync requests, etc.)

The reason this should work is because unstaked nodes can only bootstrap from staked AN's and other unstaked nodes. In other words, the staked AN's are the only things connecting the staked network with the public network, and the two would be completely disconnected otherwise. This is ensured by connection gating on all of the staked nodes. Therefore, all traffic between the public and private networks has to pass through them.

One thing I'm not sure about is how this would work with DHT. Ideally, we would want to ensure that it is possible for the rest of the staked nodes to not participate in the DHT, so that unstaked nodes can't find out their addresses via the DHT. However, i believe it should be okay even if they do find out about the addresses because connection gating will prevent any messages from unstaked nodes to be gossiped by staked nodes.

[vishalchangrani]

The separation of the networks is intentional. We do not want any traffic on the staked network from nodes which are not staked.

[huitseeker]

Yes, but we have a Connection gater, don't we @vishalchangrani ?

[synzhu]

Additional notes here: (Recording things here so I remember before putting everything into notion doc)

1. Wanted to verify that signature checking is done by libp2p before validators are called. In other words, by the time our validator is run, we know that the From field has been validated and we can take it at face value.
   • first, we check signing policy here. This ensures that if signing is required, that a signature must be present.
   • Then, signature validation is done before any of our validators are called.
2. Here is where the From field is populated. Here is the field definition.
3. Important: We must ensure that sign policy is never changed from the "strict" setting, for all of the nodes.
4. Here are the acceptable types we can use to implement a validator: https://github.com/libp2p/go-libp2p-pubsub/blob/master/validation.go#L162-L170

[synzhu]

Another note:

Since we would be intercepting messages from unstaked nodes at the gossip layer, they would never make it to the engines on their own. In other words, they wouldn't go through the normal processing flow of messages received on the staked network (from libp2p -> our network layer -> engine).

Instead, we will need some way of injecting that message back into the processing flow once it's been validated, whether by directly passing it to the network layer or directly passing it to an engine.

[synzhu]

This is done