consensys/web3signer

Web3Signer is an open-source signing service capable of signing on multiple platforms (Ethereum1 and 2, Filecoin) using private keys stored in an external vault, or encrypted on a disk.

Web3signer

An application able to sign Eth2 artefacts (attestations and blocks) using BLS12-381 private key.

Issues

Web3Signer issues are tracked in GitHub.

See our contribution guidelines for more detail on searching and creating issues.

API

• API Documentation

Users

• User documentation

Chat

• Discord server

Developers

• Contribution Guidelines
• Coding Conventions

Release Notes

• Release Notes

web3signer's Issues

Add SignedAggregateAndProof signing endpoint

v0.11.1 of the beacon chain spec introduces SignedAggregateAndProof which is another type of thing to sign.

• Add an endpoint to the eth2signer API for it /signer/aggregate_and_proof this will just delegate to sign bytes like the other endpoints
• Update ATs for this new endpoint

Implement Multi-key toml loading

Copy multikey toml loading capability from EthSIgner
Initialise it with a commandline arg defining the path to the toml parent directory.
Remove Azure signing (only hashicorp and file-based).

SO need to pull in the hashicorp loading from EthSigner (bring the TLS too)

Cache validator keys

Cache validator keys so that the keys do not need to be fetched and decrypted each time a key is used to sign a block or attestation.

Acceptance Criteria

• Load validator keys referenced by config directory on startup
• Return validator key / artifact signer from Cache if it exists
• If key doesn't exist attempt to load it using the config directory
• Update the cache periodically when config is changed in the config directory. This could be a later feature.

Create endpoint to publish available public keys

Benchmark the the loading of keys from yaml files

Some benchmarking of what performance is like when reading from yaml files. Need to consider what the limits are (~10s, ~100s, ~1000s)

Can run this locally first and see if performance is sufficient, can then check on cloud providers.

Do a performance benchmark

• Build a framework
• Produce some data
• Measure the performance

Support a config file for CLI options

Support the CLI options through a config file

To support YAML

CLI overrides config file

Create repo and builds

Setup application repo and builds.

• Create a repo with license file, contributor and readme files
• Setup gradle build
• Setup CI builds

Upcheck endpoint

Unify key config files

Allow bls keys and secp config key files to use either YAML or TOML. BLS key files currently use YAML and secp256k1 key config files use TOML and have different file contents.

• Use either YAML or TOML
• Common file format
• Include curve type in file format

Refactor Creating Signers to produce more usable errors

At the moment parsing metadata files and creating files is lumped together, and thus their errors get conflated - it would produce better output if we separate these 2 concepts and handle errors separately.

OpenAPI support

Add openAPI support to Eth2Signer so that users can discover what APIs are available, provide documentation and allow for experimentation with the APIs.

This should include

• Exposing the openAPI spec
• Providing swagger ui as part of the application so the documentation can be viewed and API tried out

Add metrics for secp keys

Add metrics/ensure that the key signing metrics work for signing when using the secp keys

Create a BLS12 Keystore loading/generation into a standalone library

Load key from a BLS12 keystore formatted file

• Update toml/yaml file parsing to load password file, and also BLS12 keystore file
• Generate private key, and wrap in an ArtifactSigner

Note: This depends on the Bls12 Keystore library being ready for use.

Bad request error for request with invalid characters for the public key in url

Eth2Signer responds with a 400 bad request but expected a expecting a 404 not found http://localhost:9000/signer/attestation/!@$%^&*(

This similar request results in a 404 not found error
http://localhost:9000/signer/attestation/!@#$%^&*(

Sign voluntary exit endpoint

Add an endpoint that will accept a voluntary exit message and return a signature.

Include error messages in the http error responses

Eth2Signer will set the http status correctly if an error occurs e.g. 404, 400 etc. but doesn't include the error details in the response. We want to include the error details in the response so that user can fix the request without going to the eth2signer logs.

Update signing API to only provider R/S

Update signing API to only provider R/S not V - the V is to be recovered in "in app" e.g. EthSigner

Create KeyProvider with hard-coded key

Create signing endpoint/s using a key file from a known location.

• And wire up signing endpoint to use hard-coded key, but produces a signature.
• Copy signing functionality from Teku into here
• Create the Http endpoint which triggers a signing operation
• Add a metrics for count-times-called
• Add /signer/block, /signer/attestation endpoints, decode received Json body and sign supplied data with KeyProvider output

Create App, with basic commandline and http upcheck

Update Eth2Signer endpoints to BLS 0.10

Update endpoints to remove the domain and use use BLS 0.10 now that Teku has been updated.

Acceptance Criteria

• Replace the copied code in Eth2Signers with the BLS 0.10 implementation in Teku either depending on Teku utils jar or moving the BLS code into signers and depending on that
• Update the endpoints to no longer include the domain
• Ensure that the endpoints ignore unknown fields
• Add additional endpoint to sign the randao reveal

Additional Information

• There should now be 3 endpoints /signer/block, /signer/attestation & /signer/randao_reveal
• Each has the same payload { publicKey: <hex string>, signingRoot: <hex string> } (signingRoot is the equivalent of message in the previous endpoint)
• Unknown fields should be ignored. In the future other fields might be included as part of the request for the slashing protection e.g. the block data for the block endpoint

Create a threadrunner for acceptancetests

Genericse Hashicorp Key Vault wrappers

Pegasys has several versions of Hashicorp vault wrappers spread across our products - it would be useful to modify one of these code bases into a standalone library such that it can be reused (rather than copied) into dependent applications.

TLS support

Add TLS support to Eth2Signer. This will likely follow the approach taken by EthSigner and so we can copy the implementation from there.

Eth2Signer

This is required to sign blocks and attestations when using Teku in an Ethereum 2 environment.

Import keys endpoint

Add a new endpoint that will allow importing of private keys for BLS and secp256k1 curves into the existing backing services.

• Create new endpoint
• Determine how config for import will be provided
• Integrate with hashicorp vault to import keys
• Integrate with Azure to import keys
• Create local config files if using local disk for imported keys

Remove references from Mikuli, depend only on Teku's BLS package

Ethereum specific endpoint

Create an Ethereum specific endpoint so that the client doesn't need to deal with determining the recovery ID or dealing with the chain ID.

Suppress the Java reflective access errors in the log

Logging level is not set from CLI

Logging level command parameter has no affect on the logging level used in Eth2Signer. It looks like the --logging option wasn't implemented

Introduce Metrics

Eth2Signer needs to use a metrics framework similar/identical to Besu.

This will require some additions to the commandline, and maybe a few basic metrics (eg how many upchecks).

This will then set the scene for future metrics work.

Consolidate ArtifactSigner with Transaction signer

Consolidate ArtifactSigner with Transaction signer to allow reuse of data structures and signing and storage engines.

Upcheck endpoint

Create a basic app which responds to Upcheck

• Uses PicoCli to define a http end point
• Uses Vertx to create http endpoint using config from cmdline
• Success = able to ‘get’ http://a.b.c.d/upcheck (using curl), and displays a fixed string (“OK”).
• Requires AT

Use 32 byte private keys when stored externally

Change key loading so that private keys are stored as 32 byte values externally. Internally they need to become 48 byte keys with zeros appended for mikuli to work with them.

Add contributing guidelines

Can be copied from EthSigner

Create keys endpoint

Add a endpoint to create BLS and secp256k1 in the supported backing services including filesystem.

• Create config files for stored keys
• Integrate hashicorp to create keys in vault
• Integrate with Azure to create keys
• Determine ownership model
• How will failure be handled?

Signer API to have the public key as part of the URL

For load balancing it is much easier if the public key is part of the path rather than as part of the body.

This will update the API to be:

/signer/block/<publickey>
/signer/attestation/<publickey>
/signer/randao_reveal/<publickey>

{
	“signingRoot”: <hex encoded string>
}

Update the multi-file-loaded to use a yaml based format

Eth2Signer currently uses a toml based format for key loading with unencrypted key. Update this to use a yaml based format.

Ideally this will be a common yaml format will be common between Eth2Signer and Teku.

Remove the signer type from the signer endpoint

Design the HA behaviour

Determine how to provide high availability for Eth2Signer and what if any changes are needed for achieve this.

Metrics endpoint

Create metrics endpoint to provide metrics on block and attestations signings.

• Reuse Prometheus metrics framework from Besu

Create signing metrics

Leverage the metrics/prometheus framework to characterise the performance of Eth2Signer when performing signing operations, eg:

• Time taken to sign
• Total count of keys

Sign deposit message endpoint

Endpoint that will take a deposit message and return a signature.

Load keys from Hashicorp vault

The raw bytes of the key are to be stored in the vault (i.e. the vault will not hold password-encrypted BLS12 keystore formatted json).

• Reuse the standalone Hashicorp Vault wrapper library (Created in #17]
• Update toml/yaml parsing to connect to Vault and extract key
• Create AT to ensure keys are extracted correctly

Support Env Variables for CLI options

Enable the use of environment variables for all the CLI options used

Design a slashing protection service

High-level objective: A "rules engine" that determines if a signing request from a validator meets all the specified rule conditions, if yes, pass the request to the signer, if not, reject the request and do not pass on.

Refer to the original betting table pitch for high-level details: https://docs.google.com/document/d/1J0gXmt-7c9hA_M9ZOHN5lxqnmB2svJZNvzKGYhS82_Q/edit#heading=h.f2e2nkfwrhi3

Eth2Signer design if useful: https://docs.google.com/document/d/1slz9GnccCdQ9kQ70AoT4h6vNspT62_96KImvjm8e4GQ/edit#heading=h.6fee3oevsdz

Add secp256k1 signing

Add secp256k1 signing to Eth2Signer as new signing endpoint.

• Accepts Eth Address on the URL (yes, not ideal, but ok first place)
• Add signing public-key http endpoint, which converts to addr, then finds signer
• Can use separate config directory for secp256k1 keys (may dodge it due to BLS12=YAML, SECP=TOML)

Allow key config files to have any unique identifier

Allow the key config files to have a name other than the public key or address. This will allow key config files to be consistent for secp256k1 keys which use the address and bls12-381 keys which are named using the public keys.

• Allow any unique identifier for the key files
• At startup, need to load private-keys and associated identifier
• For Eth1 keys, these private-keys need to be mapped to public-key → address
• Do we want to cache the identifier → File mapping? Or re-parse on every restart

Create a basic graphana dashboard for metrics