alchemyplatform/rundler

An ERC-4337 Bundler in Rust

Rundler

High-performance, modular implementation of an ERC-4337 bundler

Run | Developer Docs

🚧 Under active development, see status below. 🚧

Overview

Rundler (Rust Bundler) is an ERC-4337 bundler implementation written in Rust. Rundler is designed to achieve high-performance and high-reliability in cloud deployments via a modular architecture.

Built, maintained, and used by Alchemy to push the limits of user experience on EVM chains via Account Abstraction.

Goals

Rundler is meant to power the ERC-4337 ecosystem with an implementation that users can rely on to scale to their needs while maintaining high reliability. As ERC-4337 is a nascent technology, Rundler strives to be on the bleeding edge of feature support and the team will be contributing to push the protocol forward.

Our goals with Rundler:

1. ERC-4337 Specification Compliance: Rundler strives to implement the full ERC-4337 specification and to maintain support as the specification changes and new onchain components are released. This includes new Entry Point contract support, support for the upcoming P2P mempool specification, support for alternative mempools, and more.
2. High Performance and Reliability: Rundler strives to power the most demanding workloads in cloud environments. Rust was chosen for its high performance and memory safety. Rundler's modular architecture lets providers choose to run the stateless components (RPC, builder) in a fully horizontally scalable manner connecting to the stateful components (mempool, p2p, event cache) via the network. Rundler's bundle builder is designed to be able to support the full gas throughput of the network it's building for.
3. Extendability/Chain Support: ERC-4337 is designed to support any EVM chain. However, different EVM supporting networks have different rules around how they support things like gas usage, gas fees, precompiles, etc. Rundler is designed to be extendable and easily adapted to support any EVM chain.
4. Modularity: Rundler is written in a modular manner, allowing its components to be run as a single integrated binary, or as a distributed system. Rundler also strives for its individual crates to be used to support future ERC-4337 tooling.

Status

Rundler is under active development. It is used in Alchemy's cloud to power Alchemy's Account Abstraction APIs. However, Rundler is rapidly being upgraded, features are being added, interfaces will have breaking changes, and the ERC-4337 spec is evolving from onchain learnings.

The documentation is work in progress, and we are working to improve it. Please reach out with any questions.

Use in production at your own risk.

ERC-4337 Entry Point Version Support

Rundler currently supports the following Entry Point versions:

• v0.6.0
• v0.7.0

See more on Entry Point support here.

Chain Support

Rundler has been tested on the following networks and their testnets:

• Ethereum
• OP Stack
  • Generally any OP stack chain should work.
  • Rundler has been explicitly tested on Optimism, Base, Zora, and Frax.
• Arbitrum Orbit
  • Generally any Arbitrum Orbit chain should work.
  • Rundler has been explicitly tested on Arbitrum One.
• Polygon POS

Developers

Contributing

See CONTRIBUTING.md.

Running

The easiest way to run Rundler is via a docker container. There is currently no pre-built image. See our docker documentation for further instructions.

Developing

For detailed instructions see developing.

Install prerequisites.

Clone the repository and checkout submodules:

git clone https://github.com/alchemyplatform/rundler
cd rundler
git submodule update --init --recursive

Run unit tests:

make test-unit

Run ERC-4337 spec tests:

# Only need to run once to install testing frameworks
cd test/spec-tests/v0_6/bundler-spec-tests && pdm install && pdm run update-deps
cd test/spec-tests/v0_7/bundler-spec-tests && pdm install && pdm run update-deps

# Run the v0.6 and v0.7 tests
make test-spec-integrated

Help

If you have questions regarding the Rundler codebase, please first look through our documentation.

With further questions:

• Github discussions
• Telegram
• Github issues

If you have questions regarding Alchemy's APIs or service, kindly refrain from discussion here. Please join the Discord or reach out to [email protected].

Security

For security concerns do not file a public ticket, please reach out to [email protected].

See SECURITY.md for details.

Acknowledgements

The work on this project would not have been possible without the amazing contributions from:

• ERC-4337 team: The ERC-4337 team has pioneered the standard, and has answered countless questions during our development. They developed the bundler reference implementation and spec tests that were invaluable during our development process. We are excited to continue to work with this team to push ERC-4337 ahead.
• Reth: Shout-out to the Reth team, from which we've taken inspiration for many of our practices in this repo (including this README). They are pushing the Ethereum Rust ecosystem forward in an open way. We thank the Reth team for their continued contributions.

License

The Rundler library (i.e. all code outside of the bin directory) is licensed under the GNU Lesser General Public License v3.0, also included in our repository in the COPYING.LESSER file.

The Rundler binaries (i.e. all code inside of the bin directory) are licensed under the GNU General Public License v3.0, also included in our repository in the COPYING file.

Copyright 2023 Alchemy Insights, Inc.

Contact: Alchemy Insights, Inc., 548 Market St., PMB 49099, San Francisco, CA 94104; [email protected]

rundler's Issues

builder: add metrics to transaction submission

Describe the feature
Add metrics to the builder module to track performance of bundle submission in real time.

builder: Switch away from `eth_estimateGas` for gas limits

Describe the feature
Don't use eth_estimateGas for gas limits. We should sum up the verification/call limits of each UO in the bundle, add some overhead multiplier, and use eth_call to do the final check for reverts.

We need to ensure that the total amount of gas doesn't go over a max value.

metrics: Incorporate Reth metrics crate

Describe the feature
We currently have a hodgepodge approach to metrics with a bunch of hardcoded strings everywhere. We should clean this up and build an extendable system for metrics.

pool: WS event manager stops receiving blocks after disconnect

Describe the bug
The pool event mange stops receiving blocks and stays in that state until reset after a disconnection.

To reproduce
This happens pretty consistently as long as you leave the pool running long enough. It is always triggered by some sort of disconnection in the WS connection.

Expected behavior
The pool should automatically reconnect.

Logs
Trigger:

{
    "timestamp": "2023-04-06T18:30:58.999003Z",
    "level": "ERROR",
    "fields": {
        "message": "Close frame: CloudFlare WebSocket proxy restarting (1001)"
    },
    "target": "ethers_providers::rpc::transports::ws::backend"
}

"Sucessfull" reconnection:

{
    "timestamp": "2023-04-06T18:30:59.113123Z",
    "level": "INFO",
    "fields": {
        "message": "Re-connection complete",
        "subs": 1,
        "reqs": 0
    },
    "target": "ethers_providers::rpc::transports::ws::manager"
}

Additional context

op_pool: Add metrics

pool: add pack_len() method to UO for size tracking

for context: #71 (comment)

builder: Implement a KMS-based signer with key leasing

Describe the feature
The bundle builder needs access to a EOA signing key to send bundle transactions. When the Rundler is running in AWS we can use KMS to keep these keys secret and sign without the Rundler container seeing the private key. To ensure that no other bundler process is accessing the same key (during upgrades likely) we should implement a simple key leasing strategy based on distributed locks.

Ethers signer: https://github.com/gakonst/ethers-rs/blob/master/ethers-signers/src/aws/mod.rs
Rustoto KMS: https://docs.rs/rusoto_kms/latest/rusoto_kms/
Redis distributed locking: https://redis.io/docs/manual/patterns/distributed-locks/, https://redis.com/glossary/redlock/
Redlock in rust: https://github.com/hexcowboy/rslock

pool: Add waiting pool for UOs that are not ready for bundling

builder: Estimate a reasonable priority fee and filter out ops that don't pay

Describe the feature
The bundler currently loses money because it doesn't inspect the priority fee field of user ops, and eventually sends a transaction with a priority fee higher than many of the ops it bundles. The builder should ensure that it doesn't lose money on a transaction by ensuring that each op is at least breakeven in terms of fees.

op_pool: Allow event listener to listen to multiple entrypoints

builder: Determine if signature re-aggregation is required after removing op

Describe the feature
The builder currently re-computes an aggregated signature if any of the ops that contributed to that aggregated signature are removed from the bundle. For BLS signatures, it is unlikely that this re-computation is required as the aggregated signature is still valid for each of the other parts. However, its unclear if this is an invariant for all signature aggregation scheme. We should get clarity from the EIP authors and potentially add language to the EIP if this is in fact an invariant that should hold for all aggregators. If not, keep the code as it is.

pool: Add metric to track ops in pool

Describe the feature
Add a simple metric to track the number of ops in the mempool.

pool: Add entity blacklist

Describe the feature
Add the ability to pass in a list of addresses to blacklist in the pool. The pool will not validate or store UOs that refer to these entities.

pool: graceful shutdown of mempool runners

See: https://github.com/OMGWINNING/alchemy-bundler/pull/91/files#r1153136591

builder: Track paymaster balances in bundle builder

Describe the feature
While building a bundle, we should track paymaster balances to ensure that a paymaster does not run out of funds in the middle of a bundle.

rpc: Add tracing to methods where appropriate

Add configuration to export the traces to an OpenTelemetry collector.

oss: Repo docs

pool: bring simulation down to pool from RPC module

simulator: write op checker

op_pool: Modify to handle multiple mempools

pool: Handle reorgs

pool: Add endpoint to stream block updates to bundler

Describe the feature
To trigger a new bundle build, the builder should listen to the mempool for new blocks. This ensures that when the builder asks the pool for the next batch of transactions its had the chance to remove mined UOs from the pool

op_pool: Handle reconnects and errors in event listener

Support Entry Point 0.6.0 and remove 0.5.0

Describe the feature
Add support for entry point 0.6.0.

We can fully remove support for 0.5.0

rpc: `eth_getUserOperationReceipt` index out of bounds crash

Describe the bug
When retrieving a UO that had a revert, the RPC module crashes.

To reproduce
Call eth_getUserOperationReceipt on a UO hash that had a revert.

Expected behavior
It shouldn't crash, and should get the revert reason

Logs

thread 'tokio-runtime-worker' panicked at 'index out of bounds: the len is 1 but the index is 1', /Users/dcoombs/alchemy/rundler/src/rpc/eth/mod.rs:245:25
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
  2023-04-19T00:48:48.488222Z  INFO rundler::op_pool::mempool::uo_pool: New block: 3317009 with 4 entrypoint events
    at src/op_pool/mempool/uo_pool.rs:92

  2023-04-19T00:48:48.492134Z ERROR rundler::rpc::run: Server stopped unexpectedly
    at src/rpc/run.rs:142

  2023-04-19T00:48:48.500961Z ERROR rundler::cli::node: RPC server exited unexpectedly: Ok(Err(RPC server stopped unexpectedly))
    at src/cli/node.rs:73

pool: Backfill from event listener for persistent mempools.

Describe the feature
For persistent mempools (DB/Redis backed) the node will need to backfill from the event listener to remove any UOs from the persistent pool that have already been mined. This should be a simple extension to the event listener to provide a pull method from a range of blocks and return an aggregated set of block events (with UO events).

Additional context
This doesn't apply to ephemeral mempools, like the P2P mempool or the closed-beta mempool.

bundler: Implement simple bundling logic

pool: Build a persistent, multi-client, redis-backed mempool

Describe the feature
We need a persistent mempool to handle bundler crashes and upgrades. This can be done in 2 ways:

1. Local-file DB backed mempool. This will allow restarts, but requires attached storage to be managed. Attached storage isn't great for our cloud infrastructure. This may be a useful modality for if/when we integrate Rundler with block builders.
2. Redis backed mempool. Allows multiple clients to have a similar view over the mempool and allows us to manage the mempool without needing attached storage. Needs simple coordination between clients.

We've decided to build a Redis-backed mempool to support the private mempool use case.

Bad CLI error message on missing entrypoints

Describe the bug
Missing ENTRY_POINTS CLI arg leads to a misleading error message.

To reproduce
Start the Rundler without ENTRY_POINTS CLI arg.

Expected behavior
Rundler should emit a better error message indicating the missing CLI arg, instead of the current error message.

Logs

rundler-launcher-rundler-1  | 2023-04-01T18:21:04.205756Z ERROR rundler::cli::node: RPC server exited unexpectedly: Ok(Err(Only one entry point is supported at the moment))

simulation: add gas limit on tracing simulate handle ops

rpc: Use binary search to estimate gas

Describe the feature
Our gas estimation currently requires a 25% overhead multiplier to handle the 1/64th rule. This is due to our tracer taking the raw gas used value after the call is complete, not taking into account gas that needs to be set aside. Node clients handle this with a binary search that can estimate the gas limit within some amount of gas. We can do the same to calculate accurate limits for both validation/call gas limits without using a hardcoded overhead multiplier.

rpc: Make gas estimation more accurate

Describe the feature
We currently a few hard coded constants in order for gas estimation to provide results that are usable (by overestimating, or capping gas limits too low). We should figure out an approach that accurately estimates gas generically, without the use of arbitrary constants, and does so relatively performantly.

builder: Use private transaction RPCs on eth/polygon mainnets

Describe the feature
When transacting with real value, we need to use private transaction RPCs on Ethereum/Polygon mainnet.

On eth:
https://docs.flashbots.net/flashbots-protect/rpc/quick-start

On polygon:
https://docs.bloxroute.com/introduction/protect-rpcs/polygon-protect-rpc

Polygon's isn't exact, but its the best we have at the moment "Front-running prevention does not guarantee your transactions will not be attacked - your transaction info will eventually become public info."

oss: Contribution guide

pool: Event listener doesn’t shutdown when there are DNS issues

Describe the bug
The pool doesn't shutdown correctly when the event listener has DNS issues. It hangs and thus doesn't release resources.

To reproduce
Start up the Rundler pointing to an RPC endpoint that it cannot resolve. Then try to shut it down.

Expected behavior
Rundler should shut down cleanly.

Logs
This also happened:

thread 'tokio-runtime-worker' panicked at 'attempt to subtract with overflow', /Users/dcoombs/.cargo/registry/src/github.com-1ecc6299db9ec823/rslock-0.1.0/src/lock.rs:205:33

oss: Rename everything to Rundler

Local/remote client server abstraction

all: Add gas caps to all calls to nodes

Describe the feature
Every eth_call (and others like it) to a node should include a gas cap reasonable for that call (either during validation or execution) to protect long running calls from blocking the bundler and impacting nodes.

builder: use conditional RPC on arbitrum goerli

Describe the feature
Arbitrum now supports eth_sendRawTransactionConditional on Arbitrum Goerli. We should use that in the builder.

https://github.com/OffchainLabs/nitro/releases/tag/v2.0.14-alpha.2

simulation: Fails on Polygon

Describe the bug
Polygon Bor nodes use an old version of javascript, we need to compile our tracer into that version to run traces on Polygon.

To reproduce
Send any user op on polygon.

Expected behavior
It shouldn't fail.

rpc/simulation: replace all provider calls with retry providers

We need to add robustness to our RPC calls to nodes. We can use the ethers-rs retry provider to do this: https://github.com/gakonst/ethers-rs/blob/master/ethers-providers/src/rpc/transports/retry.rs

rpc: `eth_estimateUserOperationGas` requires all limits to be set in the call

Describe the bug
When using the eth_estimateUserOperationGas RPC endpoint, the verificationGasLimit field is optional, but if it is not set the call will fail with AA23 reverted (or OOG). This field should be filled in by the Rundler if unset to a value that allows estimation to occur.

To reproduce

1. Construct a UO without verificationGasLimit.
2. Call eth_estimateUserOperationGas

Expected behavior
Estimation should succeed.

pool: Add endpoint to remove all UOs by entity

Describe the feature
The spec states:

Remove the failed op that caused the revert from the batch and drop from the mempool. If the error is caused by a factory (error code is “AA1.“) or paymaster (error code is “AA3.“), then also drop from mempool all other UserOps of this entity. Repeat until eth_estimateGas succeeds.

To support this, the pool should have an endpoint that takes an entity address, and removes all operations associated with that entity.

contracts: Update to use IEntrypoint

Besides in scripts that need to deploy concrete entry point contracts.

rpc: Underestimating gas limits in `eth_estimateUserOperationGas`

Describe the bug
When using eth_estimateUserOperationGas to set the gas limits on a user operation, the resulting UO will revert with out of gas in each of the 3 fields: preVerificationGas, verificationGasLimit, and callGasLimit. The estimate should be accurate enough for the user to set these fields directly in their UO and not run out of gas.

To reproduce

1. Create a UO
2. Call eth_estimateUserOperationGas on that UO
3. Use the result to set the gas fields in the UO
4. Send that UO to a bundler

Expected behavior
The estimated gas limits should be high enough such that the UO doesn't run out of gas during verification.

rpc: Gas estimation error messages are not descriptive

Describe the bug
During estimation:

• If the call reverts with OOG and the account isn't yet deployed, we just return "Invalid data"
• If the call returns a revert reason we should prefix the reason

To reproduce

1. Estimate a UO that deploys an account, and set the gas limits too low. It will return with "Invalid data"
2. Estimate a UO that calls a method that doesn't exist on the target contract.

Expected behavior
Error messages should make it clear to the user how they need to modify their calls to succeed.

simulation: Handle banned L2 precompile opcodes

ci: incorporate bundler-spec-tests into CI

Describe the feature
Run the bundler-spec-tests in CI and fail PRs if we have regressions.

oss: setup commit linting for conventional commits

cli: Add health check endpoint