Add documentation on how to run a resilient BP architecture

[matthewdarwin]

Add documentation on how to run a resilient BP architecture

[UrsaPolarisRecords]

Here's an updated draft for Nation's node architecture article. It may be beneficial to combine information from both articles, as the Sw/eden article is much more human-readable, while the Nation article outlines the specific recommendations and config options.

Antelope Node Architecture: Improving Reliability

September 1, 2022 by EOS Nation

Antelope blockchains share a powerful codebase that allows for high levels of throughput. However, large transaction loads can overwhelm nodes if the node architecture is improperly configured.

Our recommendations for this architecture coalesce around the idea of isolating public-facing nodes from peer nodes, and optimizing configuration parameters for each node's use case.

in short:

• The sentry node processes transactions to reduce the workload for the block producer node. It also increases reliability and speed.

• Block only nodes push blocks as fast as possible through the peering network and don't care about transactions.

• Transaction only nodes process transactions and sort them to reduce load on producers, to increase speed and reliability for users

• Push API nodes increase speed, capacity and reliability for trx to reach producers afap

• caching requests : adds a LOT of speed, reliability and decreases load on nodes, so less can do more, and users get their info faster.

We have explored this topic before in a blog post, and since this post, key block producers on Antelope chains have battle-tested and refined the outlined architecture. Today we want to update this information, with a technical look at the details of the architecture.

Troubleshooting

In late August 2022, the WAX blockchain began experiencing larger than normal transaction queues. Upon further inspection, it appeared that a large number of failed transactions would overwhelm nodes, causing nodes to generate blocks with very few or no transactions. This caused many transactions to expire and led to microforks. The block producers took many mitigation efforts to help diagnose the issue, experimenting with deferred transactions, subjective billing, bypassing sentry nodes, and various configuration options.

One effective step these block producers found related to the architecture of a block producer and the nodes connected to it.

With a large number of connections to the block producer node, the myriad of transactions hitting the signing node from multiple connections prevent the node from working efficiently. The solution requires a reduction in the number of connections to the signing node. Ultimately, that means having only 2 connections.

Many block producers have already adopted this node architecture. However, others might still have multiple connections to the public P2P cluster, multiple connections to API nodes, and other various connections. All of these connections need to be replaced by only 2 connections.

A few more adjustments can reduce the remaining dropped blocks further. To add context, the Sw/eden Block Producer published an article outlining the issues on WAX and how to configure settings to assure blocks and transactions arrive on time.

Block Production node:

The block production node, or block signing node, should be protected from public API and P2P connections, and should only have two node types connecting to the rest of the chain: the Sentry nodes (for transactions + blocks) and the Blocks Peer nodes (for blocks only).

The block producer node should have the following architecture:

• Have 2 block producer nodes for redundancy, one primary and one backup.
• Connect only to the two block relay nodes and two transaction sentry nodes, as defined above (4 connections total). Do not connect to any other nodes.
• A block producer node should implement the following configuration options:

wasm-runtime = eos-vm-jit

chain-state-db-size-mb = 32768

reversible-blocks-db-size-mb = 2048

disable\_subjective\_api\_billing = false

disable\_subjective\_p2p\_billing = false

database-map-mode = heap

http-max-response-time-ms = 300

http-validate-host = false

p2p-max-nodes-per-host = 2

agent-name = INSERT NAME OF BP HERE

max-clients = 0

net-threads = 2

verbose-http-errors = true

abi-serializer-max-time-ms = 2000

cpu-effort-percent = 40

last-block-cpu-effort-percent = 20

max-transaction-time = 35

subjective-cpu-leeway-us = 36000

producer-name = INSERT ACCOUNT OF PRODUCER

signature-provider = INSERT PRODUCER KEY

actor-blacklist = INSERT MULTIPLE LINES HERE

plugin = eosio::http\_plugin

plugin = eosio::chain\_api\_plugin

plugin = eosio::net\_api\_plugin

plugin = eosio::producer\_api\_plugin

plugin = eosio::db\_size\_api\_plugin

plugin = eosio::producer\_plugin

First Connection to the BP node: Blocks Only

The 'blocks only' P2P network quickly delivers blocks between block producers. Allowing block producers to pass blocks directly between each other while ignoring transactions means a reduced processing load – and when blocks arrive on time, microforks don't happen.

So a newly produced block has to go from producer 1 -> block peer node 1 -> block peer node 2 -> producer 2.

The block peer node should have the following architecture:

• Have 2 block peer nodes for redundancy.
• Connect block peer nodes to other block producers or other trusted entities.
• Additionally, connect block peer nodes to BP nodes.
• A block peer node should implement the following configuration options:

wasm-runtime = eos-vm-jit

eos-vm-oc-enable = true

chain-state-db-size-mb = 32768

reversible-blocks-db-size-mb = 2048

http-max-response-time-ms = 300

read-mode = head

database-map-mode = heap

p2p-accept-transactions = false

http-validate-host = false

p2p-max-nodes-per-host = 2

agent-name = INSERT NAME OF BP HERE

max-clients = 0

net-threads = 5

verbose-http-errors = true

abi-serializer-max-time-ms = 2000

plugin = eosio::http\_plugin

plugin = eosio::chain\_api\_plugin

plugin = eosio::net\_api\_plugin

plugin = eosio::producer\_api\_plugin

plugin = eosio::db\_size\_api\_plugin

Second Connection to the BP node: Transactions only

This new node architecture includes a transactions network. Note: the transactions network also includes blocks.

All incoming transactions, no matter where they are coming from, must consolidate into a single "transaction peer" node that connects to the signing node. This transaction peer node must handle as many transactions as possible, but also must not overwhelm the signing node with too many transactions so as to allow the signing node to continue producing blocks efficiently.

The transaction peer node should have the following architecture:

• Have 2 block peer nodes for redundancy.
• Connect transaction peer nodes to other block producers or other trusted entities.
• Additionally, connect transaction peer nodes to BP nodes.
• A transaction peer node should implement the following configuration options:

wasm-runtime = eos-vm-jit

chain-state-db-size-mb = 32768

reversible-blocks-db-size-mb = 2048

http-max-response-time-ms = 300

disable\_subjective\_api\_billing = false

disable\_subjective\_p2p\_billing = false

read-mode = head

database-map-mode = heap

http-validate-host = false

p2p-max-nodes-per-host = 2

agent-name = INSERT NAME OF BP HERE

max-clients = 0

net-threads = 5

verbose-http-errors = true

abi-serializer-max-time-ms = 2000

plugin = eosio::http\_plugin

plugin = eosio::chain\_api\_plugin

plugin = eosio::net\_api\_plugin

plugin = eosio::producer\_api\_plugin

plugin = eosio::db\_size\_api\_plugin

API Nodes:

API nodes serve responses when given a request.

You will notice that optimized compiling (OC) is not enabled, and p2p-accept-transactions = false. This avoids processing lots of transactions and the RPC requests for transactions are billed at the same value that will be used on the BP node (assuming hardware CPU is the same). enable-account-queries = true can be set to enable lookup of account information. This is important on public API nodes.

As a reminder, the Producer API and Chain API must not be exposed to public. Use a reverse proxy to expose the /v1/chain/... APIs, but keep the others private.

The API node should have the following architecture:

• Connect the API node to both the "block relay" and "transaction sentry" nodes.
• An API node should implement the following configuration options:

wasm-runtime = eos-vm-jit

chain-state-db-size-mb = 32768

reversible-blocks-db-size-mb = 2048

http-max-response-time-ms = 300

read-mode = head

database-map-mode = heap

p2p-accept-transactions = false

disable-api-persisted-trx = true

http-validate-host = false

p2p-max-nodes-per-host = 2

agent-name = INSERT NAME OF BP HERE

max-clients = 0

net-threads = 5

http-threads = 8

verbose-http-errors = true

abi-serializer-max-time-ms = 2000

http-server-address = 0.0.0.0:8888

enable-account-queries = true

plugin = eosio::http\_plugin

plugin = eosio::chain\_api\_plugin

plugin = eosio::net\_api\_plugin

plugin = eosio::producer\_api\_plugin

plugin = eosio::db\_size\_api\_plugin

Conclusion

Figuring out the problem and coming up with a solution was a collaborative effort between the Top 21 block producers. This took a tremendous amount of work and coordination on the part of many teams located all around the world. It's normal that some teams were more involved than others, but all the involved teams contributed what they could.

As a final note, it's important to note that not all block producers have the exact same configuration. Node topology, CPU speed, transaction load and EOSIO versions vary, as different block producers and node operators have different needs and requirements.

This summary should be helpful for other block producers on Wax chains, as well as other Antelope network block producers who might want to come up with similar designs to prevent overloading.