consensys/ethsigner

DEPRECATED. A transaction signing application to be used with a web3 provider.

EthSigner [DEPRECATED]

A transaction signing application to be used with a web3 provider. All questions, queries and other discussion can be found on Discord.

⚠️ Project Deprecation ⚠️

EthSigner is no longer supported. Please migrate to Web3Signer which includes all EthSigner functionality.

---

As part of our ongoing commitment to deliver the best remote signing solutions, we are announcing a change in our product offerings.

We have decided to deprecate our EthSigner product to focus our efforts on enhancing Web3Signer, our newly comprehensive remote signing solution. This is rooted in our strategy to streamline our offerings and focus on a single, robust product that will provide functionality for both transaction and Ethereum validator signing. We hope this makes it applicable to all your use-cases like public Ethereum signing, staking infrastructure offerings, and in private network contexts.

Rest assured, we are not dropping existing EthSigner functionality. We are updating Web3Signer to incorporate the functionalities of EthSigner alongside everything else in Web3Signer. We will ensure a smooth transition by maintaining EthSigner with necessary patches until March 2024. We hope this provides ample time for any necessary migration to Web3Signer.

Issues

EthSigner issues are tracked in GitHub issues.

See our contribution guidelines for more detail on searching and creating issues.

Users

• User documentation

Chat

• Discord

Developers

• Contribution Guidelines
• Coding Conventions

Release Notes

• Release Notes

ethsigner's Issues

Upgrade docker-java library to 3.2.0

Issue: Current version of docker-java library used by EthSigner acceptance tests is not compatible with JDK 13+.

Solution: Update EthSigner to use docker-java version 3.2.0 when it is released (currently in RC).

Update picocli parseWithHandlers

Update deprecated parseWithHandlers to execute method and appropriate error handling.

Fix keystore password reading eol issue

Make the same fix in EthSigner as in Besu - hyperledger/besu#554

snyk security scanner github webhook

Install snyk security scanner so that we alerted to vulnerabilities in our code

auto nonce

so you're trying to replace transaction if(yours) nonce is the same. And this is about two transactions that did token transfers to two different addr'ses . But how could that be?

gradle build license dependencies broken

The license-dependency.html file produced is empty. This should contain the list of license dependencies

create threadrunner for AT DSL

Support Google Cloud KMS

It would be great if EthSigner could support Google KMS as a backend for keys. What do you think?

Note: I'm creating the issue here because of #218.

Create release of EthSigner

Probably 0.7.0
Should contain downstream_path, and also eth_sign changes.

Update changelog

Add EIP115 (chain Id) in transactions parameters

Hello,

Is possible to add EIP115 (https://github.com/ethereum/EIPs/blob/master/EIPS/eip-155.md) to add a "chainId" parameters when we sign transactions ?

      rawTransactionData = {
        gas: new Eth.BN(gas, 10),
        gasPrice: new Eth.BN(gasPrice, 10),
        nonce: new Eth.BN(nonce, 10),
        data: data,
        chainId: chainId,
        to: to,
        value:value,
      };
   
      const signedTxData = ethSigner.sign(Object.assign({}, rawTransactionData), privateKey);

Where to add chainId parameter ?

There is no options in transaction object :
https://docs.ethsigner.consensys.net/en/stable/Reference/API-Methods/#eea_sendtransaction

Requests not made to downstream web3 provider when host header required

The host header is being passed through from the original request to EthSigner this breaks requests to web3 providers that require virtual host support through the host header.

Fix is to

• Remove the original host header allow Vertx to set this to the correct value
• Set the original host value on a x-forwarded-host header

[ES-7] Build health/status/upcheck endpoint

Expected:

An endpoint that I can ping this service to check if it is up.

Why

When building a system with multiple services I would like the service providing transactions to ethsigner to be able to ping ethsigner on start-up to make sure it is running.

I.e.

1. I start ethsigner
2. When starting Pantheon it will ping ethsigner health/status/upcheck endpoint to check if it is running.

Similar to orion upcheck

Also for testing :)

ES-7

Publish to CloudSmith

CORS configuration invalid

I'm trying to setup CORS so I can access the signer in development from an application running at localhost:8000.
I tried "*", "all" and "http://localhost:8000", but they all seem to add an extra * to the header.

Access to fetch at 'http://localhost:8545/' from origin 'http://localhost:8000' has been blocked by CORS policy: The 'Access-Control-Allow-Origin' header contains multiple values '*, *', but only one is allowed. Have the server send the header with a valid value, or, if an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

eth_sendTransaction invalid params

I´ve been trying to send regular transactions to the node, but I keep getting

{"jsonrpc":"2.0","id":1,"error":{"code":-32602,"message":"Invalid params"}}

as a response.

I´m taking the default user in rinkeby (0x919C368FC9d4Ec8A48Fd413d966A1B21c275f594) and tried the following transaction as is on the documentation (Without the line jumps)

curl -X POST --data '{
"jsonrpc":"2.0",
"method":"eth_sendTransaction",
"params":[{
    "from": "0x919c368fc9d4ec8a48fd413d966a1b21c275f594",
    "to": "0xD641AEc56A709B1682cc6B1891495cC5aB6AD430",
    "gas": "0x7600",
    "gasPrice": "0x9184e72a000",
    "value": "0x9184e72a"}], 
"id":1}' 
http://127.0.0.1:8545

I got the same error.

{"jsonrpc":"2.0","id":1,"error":{"code":-32602,"message":"Invalid params"}}

So I tried with all the optional params in the official Ethereum JRPC docs

curl -X POST --data '{
"jsonrpc":"2.0",
"method":"eth_accounts",
"params": [{ 
    "from": "0x919c368fc9d4ec8a48fd413d966a1b21c275f594", 
    "to": "0xD641AEc56A709B1682cc6B1891495cC5aB6AD430", 
    "gas": "0x76c0", 
    "gasPrice": "0x9184e72a000", 
    "value": "0x9184e72a", 
    "data": "0xd46e8dd67c5d32be8d46e8dd67c5d32be8058bb8eb970870f072445675058bb8eb970870f072445675", 
    "nonce": "0x0" }],
"id":1}'
http://127.0.0.1:8545

And got the same response.

{"jsonrpc":"2.0","id":1,"error":{"code":-32602,"message":"Invalid params"}}

Am I missing something?

Failed to extract secret from Hashicorp vault.

Hello,
Working in hashicorp vault dev mode everything works fine but when I use a hashicor vault in production mode I am not able to connect ethsigner. However, i have been checkin if I can extract the secret from the vault using the api:
curl -H "X-Vault-Token: " -X GET http://127.0.0.1:8200/v1/secret/data/ethsignerSigningKey
{"request_id":"ba39f319-2f2c-0b46-61e4-7d6042334557","lease_id":"","renewable":false,"lease_duration":2764800,"data":{"value":"b0057716d5917badaf911b193b12b910811c1497b5bada8d7711f758981c3773"},"wrap_info":null,"warnings":null,"auth":null}
or using the kv instruction:
./vault kv get secret/ethsignerSigningKey
==== Data ====
Key Value

---

value b0057716d5917badaf911b193b12b910811c1497b5bada8d7711f758981c3773

I get the following error in ethsigner:
./ethsigner-0.6.1-SNAPSHOT/bin/ethsigner --chain-id=2018 --logging=ALL --downstream-http-host=localhost --downstream-http-port=8545 --http-listen-port=8549 --http-listen-host=0.0.0.0 hashicorp-signer --host=127.0.0.1 --port=8200 --auth-file=authFile --tls-enabled=false Setting logging level to ALL 2020-06-28 07:52:34.833+00:00 | main | DEBUG | SignerSubCommand | Configuration = HashicorpSubCommand{serverHost=127.0.0.1, serverPort=8200, authFilePath=authFile, timeout=10000, signingKeyPath=/secret/data/ethsignerSigningKey, tlsEnabled=false, tlsKnownServerFile=null} 2020-06-28 07:52:34.878+00:00 | main | INFO | SignerSubCommand | Version = ethsigner/v0.6.1-dev-1b6082f1/linux-x86_64/-ubuntu-openjdk64bitservervm-java-11 2020-06-28 07:52:34.936+00:00 | main | DEBUG | InternalLoggerFactory | Using SLF4J as the default logging framework 2020-06-28 07:52:34.940+00:00 | main | DEBUG | ResourceLeakDetector | -Dio.netty.leakDetection.level: simple 2020-06-28 07:52:34.940+00:00 | main | DEBUG | ResourceLeakDetector | -Dio.netty.leakDetection.targetRecords: 4 2020-06-28 07:52:34.951+00:00 | main | DEBUG | InternalThreadLocalMap | -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024 2020-06-28 07:52:34.951+00:00 | main | DEBUG | InternalThreadLocalMap | -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096 2020-06-28 07:52:34.962+00:00 | main | DEBUG | MultithreadEventLoopGroup | -Dio.netty.eventLoopThreads: 4 2020-06-28 07:52:34.986+00:00 | main | DEBUG | NioEventLoop | -Dio.netty.noKeySetOptimization: false 2020-06-28 07:52:34.986+00:00 | main | DEBUG | NioEventLoop | -Dio.netty.selectorAutoRebuildThreshold: 512 2020-06-28 07:52:35.001+00:00 | main | DEBUG | PlatformDependent0 | -Dio.netty.noUnsafe: false 2020-06-28 07:52:35.001+00:00 | main | DEBUG | PlatformDependent0 | Java version: 11 2020-06-28 07:52:35.003+00:00 | main | DEBUG | PlatformDependent0 | sun.misc.Unsafe.theUnsafe: available 2020-06-28 07:52:35.003+00:00 | main | DEBUG | PlatformDependent0 | sun.misc.Unsafe.copyMemory: available 2020-06-28 07:52:35.004+00:00 | main | DEBUG | PlatformDependent0 | java.nio.Buffer.address: available 2020-06-28 07:52:35.004+00:00 | main | DEBUG | PlatformDependent0 | direct buffer constructor: unavailable java.lang.UnsupportedOperationException: Reflective setAccessible(true) disabled at io.netty.util.internal.ReflectionUtil.trySetAccessible(ReflectionUtil.java:31) ~[netty-common-4.1.39.Final.jar:4.1.39.Final] at io.netty.util.internal.PlatformDependent0$4.run(PlatformDependent0.java:224) ~[netty-common-4.1.39.Final.jar:4.1.39.Final] at java.security.AccessController.doPrivileged(Native Method) ~[?:?] at io.netty.util.internal.PlatformDependent0.<clinit>(PlatformDependent0.java:218) [netty-common-4.1.39.Final.jar:4.1.39.Final] at io.netty.util.internal.PlatformDependent.isAndroid(PlatformDependent.java:272) [netty-common-4.1.39.Final.jar:4.1.39.Final] at io.netty.util.internal.PlatformDependent.<clinit>(PlatformDependent.java:92) [netty-common-4.1.39.Final.jar:4.1.39.Final] at io.netty.channel.nio.NioEventLoop.newTaskQueue0(NioEventLoop.java:284) [netty-transport-4.1.39.Final.jar:4.1.39.Final] at io.netty.channel.nio.NioEventLoop.newTaskQueue(NioEventLoop.java:155) [netty-transport-4.1.39.Final.jar:4.1.39.Final] at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:137) [netty-transport-4.1.39.Final.jar:4.1.39.Final] at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:138) [netty-transport-4.1.39.Final.jar:4.1.39.Final] at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37) [netty-transport-4.1.39.Final.jar:4.1.39.Final] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) [netty-common-4.1.39.Final.jar:4.1.39.Final] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) [netty-common-4.1.39.Final.jar:4.1.39.Final] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) [netty-common-4.1.39.Final.jar:4.1.39.Final] at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) [netty-transport-4.1.39.Final.jar:4.1.39.Final] at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:78) [netty-transport-4.1.39.Final.jar:4.1.39.Final] at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:73) [netty-transport-4.1.39.Final.jar:4.1.39.Final] at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:60) [netty-transport-4.1.39.Final.jar:4.1.39.Final] at io.vertx.core.net.impl.transport.Transport.eventLoopGroup(Transport.java:148) [vertx-core-3.8.2.jar:3.8.2] at io.vertx.core.impl.VertxImpl.<init>(VertxImpl.java:143) [vertx-core-3.8.2.jar:3.8.2] at io.vertx.core.impl.VertxImpl.vertx(VertxImpl.java:92) [vertx-core-3.8.2.jar:3.8.2] at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:40) [vertx-core-3.8.2.jar:3.8.2] at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:32) [vertx-core-3.8.2.jar:3.8.2] at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:27) [vertx-core-3.8.2.jar:3.8.2] at io.vertx.core.Vertx.vertx(Vertx.java:75) [vertx-core-3.8.2.jar:3.8.2] at tech.pegasys.ethsigner.signer.hashicorp.HashicorpSubCommand.createSigner(HashicorpSubCommand.java:126) [ethsigner-signer-hashicorp-0.6.1-SNAPSHOT.jar:0.6.1-dev-1b6082f1] at tech.pegasys.ethsigner.signer.hashicorp.HashicorpSubCommand.createSignerFactory(HashicorpSubCommand.java:150) [ethsigner-signer-hashicorp-0.6.1-SNAPSHOT.jar:0.6.1-dev-1b6082f1] at tech.pegasys.ethsigner.SignerSubCommand.run(SignerSubCommand.java:53) [ethsigner-commandline-0.6.1-SNAPSHOT.jar:0.6.1-dev-1b6082f1] at picocli.CommandLine.executeUserObject(CommandLine.java:1769) [picocli-4.2.0.jar:4.2.0] at picocli.CommandLine.access$900(CommandLine.java:145) [picocli-4.2.0.jar:4.2.0] at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2150) [picocli-4.2.0.jar:4.2.0] at picocli.CommandLine$RunLast.handle(CommandLine.java:2144) [picocli-4.2.0.jar:4.2.0] at picocli.CommandLine$RunLast.handle(CommandLine.java:2108) [picocli-4.2.0.jar:4.2.0] at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1975) [picocli-4.2.0.jar:4.2.0] at picocli.CommandLine.execute(CommandLine.java:1904) [picocli-4.2.0.jar:4.2.0] at tech.pegasys.ethsigner.CommandlineParser.parseCommandLine(CommandlineParser.java:69) [ethsigner-commandline-0.6.1-SNAPSHOT.jar:0.6.1-dev-1b6082f1] at tech.pegasys.ethsigner.EthSignerApp.main(EthSignerApp.java:40) [ethsigner-0.6.1-SNAPSHOT.jar:0.6.1-dev-1b6082f1] 2020-06-28 07:52:35.016+00:00 | main | DEBUG | PlatformDependent0 | java.nio.Bits.unaligned: available, true 2020-06-28 07:52:35.018+00:00 | main | DEBUG | PlatformDependent0 | jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable java.lang.IllegalAccessException: class io.netty.util.internal.PlatformDependent0$6 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @2bb7bd00 at jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:361) ~[?:?] at java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:591) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:558) ~[?:?] at io.netty.util.internal.PlatformDependent0$6.run(PlatformDependent0.java:334) ~[netty-common-4.1.39.Final.jar:4.1.39.Final] at java.security.AccessController.doPrivileged(Native Method) ~[?:?] at io.netty.util.internal.PlatformDependent0.<clinit>(PlatformDependent0.java:325) [netty-common-4.1.39.Final.jar:4.1.39.Final] at io.netty.util.internal.PlatformDependent.isAndroid(PlatformDependent.java:272) [netty-common-4.1.39.Final.jar:4.1.39.Final] at io.netty.util.internal.PlatformDependent.<clinit>(PlatformDependent.java:92) [netty-common-4.1.39.Final.jar:4.1.39.Final] at io.netty.channel.nio.NioEventLoop.newTaskQueue0(NioEventLoop.java:284) [netty-transport-4.1.39.Final.jar:4.1.39.Final] at io.netty.channel.nio.NioEventLoop.newTaskQueue(NioEventLoop.java:155) [netty-transport-4.1.39.Final.jar:4.1.39.Final] at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:137) [netty-transport-4.1.39.Final.jar:4.1.39.Final] at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:138) [netty-transport-4.1.39.Final.jar:4.1.39.Final] at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37) [netty-transport-4.1.39.Final.jar:4.1.39.Final] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) [netty-common-4.1.39.Final.jar:4.1.39.Final] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) [netty-common-4.1.39.Final.jar:4.1.39.Final] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) [netty-common-4.1.39.Final.jar:4.1.39.Final] at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) [netty-transport-4.1.39.Final.jar:4.1.39.Final] at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:78) [netty-transport-4.1.39.Final.jar:4.1.39.Final] at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:73) [netty-transport-4.1.39.Final.jar:4.1.39.Final] at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:60) [netty-transport-4.1.39.Final.jar:4.1.39.Final] at io.vertx.core.net.impl.transport.Transport.eventLoopGroup(Transport.java:148) [vertx-core-3.8.2.jar:3.8.2] at io.vertx.core.impl.VertxImpl.<init>(VertxImpl.java:143) [vertx-core-3.8.2.jar:3.8.2] at io.vertx.core.impl.VertxImpl.vertx(VertxImpl.java:92) [vertx-core-3.8.2.jar:3.8.2] at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:40) [vertx-core-3.8.2.jar:3.8.2] at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:32) [vertx-core-3.8.2.jar:3.8.2] at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:27) [vertx-core-3.8.2.jar:3.8.2] at io.vertx.core.Vertx.vertx(Vertx.java:75) [vertx-core-3.8.2.jar:3.8.2] at tech.pegasys.ethsigner.signer.hashicorp.HashicorpSubCommand.createSigner(HashicorpSubCommand.java:126) [ethsigner-signer-hashicorp-0.6.1-SNAPSHOT.jar:0.6.1-dev-1b6082f1] at tech.pegasys.ethsigner.signer.hashicorp.HashicorpSubCommand.createSignerFactory(HashicorpSubCommand.java:150) [ethsigner-signer-hashicorp-0.6.1-SNAPSHOT.jar:0.6.1-dev-1b6082f1] at tech.pegasys.ethsigner.SignerSubCommand.run(SignerSubCommand.java:53) [ethsigner-commandline-0.6.1-SNAPSHOT.jar:0.6.1-dev-1b6082f1] at picocli.CommandLine.executeUserObject(CommandLine.java:1769) [picocli-4.2.0.jar:4.2.0] at picocli.CommandLine.access$900(CommandLine.java:145) [picocli-4.2.0.jar:4.2.0] at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2150) [picocli-4.2.0.jar:4.2.0] at picocli.CommandLine$RunLast.handle(CommandLine.java:2144) [picocli-4.2.0.jar:4.2.0] at picocli.CommandLine$RunLast.handle(CommandLine.java:2108) [picocli-4.2.0.jar:4.2.0] at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1975) [picocli-4.2.0.jar:4.2.0] at picocli.CommandLine.execute(CommandLine.java:1904) [picocli-4.2.0.jar:4.2.0] at tech.pegasys.ethsigner.CommandlineParser.parseCommandLine(CommandlineParser.java:69) [ethsigner-commandline-0.6.1-SNAPSHOT.jar:0.6.1-dev-1b6082f1] at tech.pegasys.ethsigner.EthSignerApp.main(EthSignerApp.java:40) [ethsigner-0.6.1-SNAPSHOT.jar:0.6.1-dev-1b6082f1] 2020-06-28 07:52:35.020+00:00 | main | DEBUG | PlatformDependent0 | java.nio.DirectByteBuffer.<init>(long, int): unavailable 2020-06-28 07:52:35.020+00:00 | main | DEBUG | PlatformDependent | sun.misc.Unsafe: available 2020-06-28 07:52:35.021+00:00 | main | DEBUG | PlatformDependent | maxDirectMemory: 2090860544 bytes (maybe) 2020-06-28 07:52:35.021+00:00 | main | DEBUG | PlatformDependent | -Dio.netty.tmpdir: /tmp (java.io.tmpdir) 2020-06-28 07:52:35.022+00:00 | main | DEBUG | PlatformDependent | -Dio.netty.bitMode: 64 (sun.arch.data.model) 2020-06-28 07:52:35.023+00:00 | main | DEBUG | PlatformDependent | -Dio.netty.maxDirectMemory: -1 bytes 2020-06-28 07:52:35.023+00:00 | main | DEBUG | PlatformDependent | -Dio.netty.uninitializedArrayAllocationThreshold: -1 2020-06-28 07:52:35.025+00:00 | main | DEBUG | CleanerJava9 | java.nio.ByteBuffer.cleaner(): available 2020-06-28 07:52:35.025+00:00 | main | DEBUG | PlatformDependent | -Dio.netty.noPreferDirect: false 2020-06-28 07:52:35.032+00:00 | main | DEBUG | PlatformDependent | org.jctools-core.MpscChunkedArrayQueue: available 2020-06-28 07:52:35.036+00:00 | main | TRACE | NioEventLoop | instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@388ba540 2020-06-28 07:52:35.037+00:00 | main | TRACE | NioEventLoop | instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@3d484181 2020-06-28 07:52:35.037+00:00 | main | TRACE | NioEventLoop | instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@6111ba37 2020-06-28 07:52:35.037+00:00 | main | TRACE | NioEventLoop | instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@7be58f16 2020-06-28 07:52:35.038+00:00 | main | TRACE | NioEventLoop | instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@56c9bbd8 2020-06-28 07:52:35.069+00:00 | main | DEBUG | DefaultDnsServerAddressStreamProvider | Default DNS servers: [/127.0.0.53:53] (sun.net.dns.ResolverConfiguration) 2020-06-28 07:52:35.074+00:00 | main | DEBUG | NetUtil | -Djava.net.preferIPv4Stack: false 2020-06-28 07:52:35.074+00:00 | main | DEBUG | NetUtil | -Djava.net.preferIPv6Addresses: false 2020-06-28 07:52:35.075+00:00 | main | DEBUG | NetUtil | Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo) 2020-06-28 07:52:35.076+00:00 | main | DEBUG | NetUtil | /proc/sys/net/core/somaxconn: 128 2020-06-28 07:52:35.261+00:00 | vert.x-eventloop-thread-3 | DEBUG | PooledByteBufAllocator | -Dio.netty.allocator.numHeapArenas: 4 2020-06-28 07:52:35.261+00:00 | vert.x-eventloop-thread-3 | DEBUG | PooledByteBufAllocator | -Dio.netty.allocator.numDirectArenas: 4 2020-06-28 07:52:35.261+00:00 | vert.x-eventloop-thread-3 | DEBUG | PooledByteBufAllocator | -Dio.netty.allocator.pageSize: 8192 2020-06-28 07:52:35.262+00:00 | vert.x-eventloop-thread-3 | DEBUG | PooledByteBufAllocator | -Dio.netty.allocator.maxOrder: 11 2020-06-28 07:52:35.262+00:00 | vert.x-eventloop-thread-3 | DEBUG | PooledByteBufAllocator | -Dio.netty.allocator.chunkSize: 16777216 2020-06-28 07:52:35.262+00:00 | vert.x-eventloop-thread-3 | DEBUG | PooledByteBufAllocator | -Dio.netty.allocator.tinyCacheSize: 512 2020-06-28 07:52:35.262+00:00 | vert.x-eventloop-thread-3 | DEBUG | PooledByteBufAllocator | -Dio.netty.allocator.smallCacheSize: 256 2020-06-28 07:52:35.262+00:00 | vert.x-eventloop-thread-3 | DEBUG | PooledByteBufAllocator | -Dio.netty.allocator.normalCacheSize: 64 2020-06-28 07:52:35.263+00:00 | vert.x-eventloop-thread-3 | DEBUG | PooledByteBufAllocator | -Dio.netty.allocator.maxCachedBufferCapacity: 32768 2020-06-28 07:52:35.263+00:00 | vert.x-eventloop-thread-3 | DEBUG | PooledByteBufAllocator | -Dio.netty.allocator.cacheTrimInterval: 8192 2020-06-28 07:52:35.263+00:00 | vert.x-eventloop-thread-3 | DEBUG | PooledByteBufAllocator | -Dio.netty.allocator.cacheTrimIntervalMillis: 0 2020-06-28 07:52:35.263+00:00 | vert.x-eventloop-thread-3 | DEBUG | PooledByteBufAllocator | -Dio.netty.allocator.useCacheForAllThreads: true 2020-06-28 07:52:35.263+00:00 | vert.x-eventloop-thread-3 | DEBUG | PooledByteBufAllocator | -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023 2020-06-28 07:52:35.290+00:00 | vert.x-eventloop-thread-3 | DEBUG | DefaultChannelId | -Dio.netty.processId: 14340 (auto-detected) 2020-06-28 07:52:35.292+00:00 | vert.x-eventloop-thread-3 | DEBUG | DefaultChannelId | -Dio.netty.machineId: 7a:1d:18:ff:fe:59:0d:8b (auto-detected) 2020-06-28 07:52:35.324+00:00 | vert.x-eventloop-thread-3 | DEBUG | ByteBufUtil | -Dio.netty.allocator.type: pooled 2020-06-28 07:52:35.324+00:00 | vert.x-eventloop-thread-3 | DEBUG | ByteBufUtil | -Dio.netty.threadLocalDirectBufferSize: 0 2020-06-28 07:52:35.324+00:00 | vert.x-eventloop-thread-3 | DEBUG | ByteBufUtil | -Dio.netty.maxThreadLocalCharBufferSize: 16384 2020-06-28 07:52:35.416+00:00 | vert.x-eventloop-thread-3 | DEBUG | AbstractByteBuf | -Dio.netty.buffer.checkAccessible: true 2020-06-28 07:52:35.416+00:00 | vert.x-eventloop-thread-3 | DEBUG | AbstractByteBuf | -Dio.netty.buffer.checkBounds: true 2020-06-28 07:52:35.417+00:00 | vert.x-eventloop-thread-3 | DEBUG | ResourceLeakDetectorFactory | Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@3830b9fc 2020-06-28 07:52:35.457+00:00 | vert.x-eventloop-thread-3 | DEBUG | Recycler | -Dio.netty.recycler.maxCapacityPerThread: 4096 2020-06-28 07:52:35.457+00:00 | vert.x-eventloop-thread-3 | DEBUG | Recycler | -Dio.netty.recycler.maxSharedCapacityFactor: 2 2020-06-28 07:52:35.457+00:00 | vert.x-eventloop-thread-3 | DEBUG | Recycler | -Dio.netty.recycler.linkCapacity: 16 2020-06-28 07:52:35.457+00:00 | vert.x-eventloop-thread-3 | DEBUG | Recycler | -Dio.netty.recycler.ratio: 8 Failed to construct a signer from supplied arguments. Cause: Failed to extract secret from Hashicorp vault.
I have also tried with the options:
--signing-key-path=/v1/secret/data/ethsignerSigningKey
--signing-key-path=/v1/secret/ethsignerSigningKey
but i get the same error.

I am using v0.6.1 but i have also tried v0.5.1 (where the error was "Failed to construct a signer from supplied arguments.
Cause: Invalid response returned from Hashicorp Vault") and the latest docker build.

What could be the problem?

Thank you

No retry mechanism for private transactions

Ethsigner has a retry mechanism for nonce too low errors for public transactions. There's no equivalent retry mechanism for private transactions when the private transaction nonce is too low.

Run performance test at releases

Create a test script that runs 1000tx and captures performance. Run this regularly as part of the release process to determine any performance changes.

Move signing engine into signing library

Move the signing engine along with the signing implementations into the signer repo so that we can reuse the secp2561 signing as library in other projects.

• Signing engine will be in the signers library and can be used independently of EthSigner
• EthSigner updated to use the signer library for the secp256k1 signing

Investigate Performance Hit following 0.7.0 release

From Adhara:

I ran performance tests against EthSigner this morning and noticed a severe decline. Performance went down from 100 tx/s to 30 tx/s. When I roll back to a previous version, before commit 68f8676 (version 0.7.0), the performance is still as before. There has been a lot of changes and upgrades since then and I thought it worth asking if you maybe have an idea what could cause this before I try to find the problem myself?

Improving fault-tolerance by adding a failover node

As far as I could find, a single EthSigner instance always connects to exactly 1 Besu node.
When this Besu node is unavailable, the EthSigner instance will be unavailable as well, resulting in faults to cascade.
Ideally, we would like an EthSigner instance to connect to a different Besu node, if this node is offline.
For instance, we could allow multiple hosts and IPs to be configured in the downstream as a comma-separated list rather than a single value. Or, perhaps EthSigner could tap into the peer discovery to automatically populate a list of available nodes.

In any case, there is room for improvement on the fault-tolerance aspect. If we want high availability now without signing transactions locally, we need multiple Besu nodes and multiple EthSigner instance per node (in case EthSigner crashes but the node stays operational).
Ideally, if an EthSigner node is offline, we send a tx to another EthSigner node. Or, if a Besu node is offline, the EthSigner node should resolve this by sending the tx to a different node.

Refactor EthSigner cmdline flags to not use "arity=0" - and enforce "Enabled"style flags

Specifically the flags of note are:

• tls-allow-any-client
• tls-allow-ca-clients

Currently, TLS is implicitly enabled if a valid set of TLS flags are set on the commandline, however the "tls-enabled" flag is not specified - so it would need to be added.

Q: Do we want to support "allow any client"? I.e. do we ever want to disable client authentication? If so - what should we do if "tls-allow-ca-clients" is set (i.e. one supersedes the other, show a warning, terminate, or nothing?)
Q: Do we want to enforce taht if "allow-ca-clients" is false, then the whitelist must be specified (I believe we should, as without this check you can start EthSigner in a manner that it will not accept connections)

Update default metrics port number

The EthSigner default metrics port number is currently 8546 which clashes with Besu's default --rpc-ws-port port number.

Suggest doing something similar to Besu where its default metrics port number is 9545. So perhaps we could make EthSigner's metrics port number default to 9546 (or something else more suitable)?

[ES-8] Make --password-file flag optional

Actual

--password-file is mandatory

Expected:

By default use no password to decrypt the wallet file. This would make the --password-file flag optional.

Why

I don't want to create an empty file to pass into this flag when testing.

ES-8

error message "Invalid response returned from Hashicorp Vault"

I just trying to run ethSigner with Hashicorp Vault on docker.

Refer to tutorial, launched ethSigner's container(=2.), after running Vault(=1.) and unseal & login with root token in it.
But it outputs an error message and exit 0(=3.).

As a test, I changed "--host" option to "127.0.0.1" for the container not exit and execute curl command in it.
Then I can confirm the response it seems without problem(=4.).

So why does it output the error?
Please answer this question.

Thank you.

1. Vault
   docker run --rm --cap-add=IPC_LOCK --network tmp -v $PWD:/mnt -p 8200:8200 -e VAULT_ADDR='http://127.0.0.1:8200' --name vault vault server -config /mnt/vault/config/config.hcl

2. ethSigner
   docker run --rm -v $PWD:/mnt -p 8881:8881 --network tmp --name ethsigner1 pegasyseng/ethsigner:latest --chain-id=9999 --downstream-http-port=8541 --http-listen-port=8881 hashicorp-signer --host=***(=Vault's container address) --auth-file=/mnt/authfile --tls-enabled=false --signing-key-path=/secret/ethsignerSigningKey

(I'm going to run them with hyperledger besu(the port is "8541"), so I set the option its "--downstream-http-port" is "8541".)

3. log
   Setting logging level to INFO 2020-04-13 08:10:18.397+00:00 | main | INFO | SignerSubCommand | Version = ethsigner/v0.6.0/linux-x86_64/oracle_openjdk-java-11 Failed to construct a signer from supplied arguments. Cause: Invalid response returned from Hashicorp Vault

4. curl log
   curl -H "X-Vault-Token: ROOTTOKEN" -X GET http://***(=Vault's container address):8200/v1/secret/ethsignerSigningKey {"request_id":"2d51f63a-a267-9204-3111-6909d183cbb6","lease_id":"","renewable":false,"lease_duration":2764800,"data":{"value":"PrivateKey"},"wrap_info":null,"warnings":null,"auth":null}

"Parse Error" about 'send_transaction'

Hi,

I also issued about hashiorp vault(#259) and im trying to test file-based-signer too.
As a reference tutorial, made the environment attached file.
It launches 4 besu nodes & 4 ethsigner nodes on docker.

After docker-compose up -d, for example, execute the below curl.
The console returns "code": -32700, "message": "Parse error".

And according to log, it says Failed to decode:Invalid UTF-8 start byte 0x80.

Maybe you can get same error for using this attached file.
If you can, please let me know why does it return this error.

I'm sorry to bother you.

curl -X POST --data '{"jsonrpc":"2.0","method":"eth_sendTransaction","params":[{"from": “0x0cc565e500f6adf04b7f8b0d38ee843a8eb8d959","to": "0x21ccc45d19dada4460c6a807c11a18ff554b9fd6","gas": "0x7600","gasPrice": "0x9184e72a000","value": "0x00100}], "id":1}' http://127.0.0.1:8881 | jq .

besu-ethsigner-test.zip

Do a 0.1 release of eth2signer

Create metrics endpoint

Create metrics endpoint exposing basic metrics for EthSigner

#328

Setup CircleCI to track AT Artefacts

Currently when an AT fails on EthSIgner in CircleCI no artefacts are captured (unlike Besu).

This would be really useful - so copy out aspects of Besu's conf.yaml file and all the best.

Rework reporting, and creation of Error responses to failures

Generally EthSigner reports errors correctly - however there are a few places that the error is not ideal.

Eg. If an ethAccounts request is received, but contains (illegally) a param in the body - EthSigner will respond with:

• StatusCode = 500 (internal error)
• ResponseBody = Json Rpc Error (INVALID_PARAMS)

It probably should have a status code of BAD_REQUEST (i.e. there wasn't a failure internally, the request was invalid).

Nonce retry fails when EthSigner gets an error code that is doesn't recognise

EthSigner only needs to look for the nonce too low (code -3201) rpc error but if it receives another error which it doesn't know about it fails to deserialize the response. This is causing an issue for new error responses being added to the eea_sendRawTransaction rpc.

Update EthSigner so that it doesn't fail on unknown error codes otherwise releases would will have to be co-ordinated with Besu.

Support path for downstream web3 provider

Add support for using a path other than / for the web 3 provider. Currently only the hostname and port can be set using the --downstream-http-host and --downstream-http-port options.

Two possible solutions

1. Replace the --downstream-http-host and --downstream-http-port options with a --downstream-http-url option that has the scheme, host, port and optional path e.g. http://somehost:9000/a/path
2. Add a --downstream-http-path option that would default to / and could be used to define the path used

Update Vertx to 3.9.1

This will require some refactoring to correclty handle errors.

Use External Hashicorp Library

Deprecate and discontinue publishing to dockerhub pegasyseng

Not sure if this is still needed or not:

Should we just be using consensys/ethsigner and deprecate pegasyseng/ethsigner?

https://github.com/ConsenSys/ethsigner/blob/master/build.gradle#L509

Ethsigner fails to read empty password files and crashes

Hi,

This looks like a regression somewhere. When I attempt to pass in an empty password file with v 21.3.0 I get the following error:

jfernandes@falcon:~/workspace/quorum-performance/besu(main)$ docker logs 265aae1a50cc
Setting logging level to INFO
2021-05-18 02:13:39.304+00:00 | main | INFO  | SignerSubCommand | Version = ethsigner/v21.3.0/linux-x86_64/-na-openjdk64bitservervm-java-11
2021-05-18 02:13:39.464+00:00 | main | ERROR | PasswordFileUtil | Cannot read password from empty file: /opt/ethsigner/passwordfile
Failed to construct a signer from supplied arguments.
Cause: Cannot read password from empty file: /opt/ethsigner/passwordfile
Usage: ethsigner file-based-signer [-hV] [-k=<FILE>] [-p=<FILE>]
Sign transactions with a key stored in an encrypted V3 Keystore file.
  -h, --help              Show this help message and exit.
  -k, --key-file=<FILE>   The path to a file containing the key used to sign
                            transactions.
  -p, --password-file=<FILE>
                          The path to a file containing the password used to
                            decrypt the keyfile.
  -V, --version           Print version information and exit.

This works fine on v0.6.0 though, which just starts up fine and handles requests

For ref the password file is https://github.com/ConsenSys/quorum-performance/blob/main/besu/config/besu/passwords.txt

To reproduce:

1. Change this line https://github.com/ConsenSys/quorum-performance/blob/main/besu/docker-compose.yml#L318 to 21.3.0
2. Run ./run.sh
3. Check logs, all other ethsigners on 0.6.0 are fine, except this one which would have crashed

Refactor Runner to close Vertx on failure to start

At the moment if a failure occurs between starting Vertx and installing the Verticle, EthSigner may not terminate clealny (as seen in the ClientSideTlsAcceptanceTest).

Add environment variables support for Docker image

Hi all, great work with EthSigner.

It could be nice if EthSigner Docker image could support environment variables.

Currently, as I can see in EthSigner documentation, EthSigner can be used on docker, but it's necessary to use command arguments in order to define EthSigner options.

If this EthSigner options could be configured using environment variables, the docker container could be easier to manage with dockerfiles and docker-compose files.

Thank you.

Postman requests with Accept-encoding gzip don't work

Using EthSigner as pass-through to Besu, sending requests via Postman - if I have the header that says Accept-encoding gzip, deflate, br ticked, I get "Parse Error: The server returned a malformed response" and if I untick that, it works as normal. Maybe defaults have changed in Postman?

Sending the same request direct to besu works either way (with header or without)

Vertx transmitter refactoring

Feature Request: eea_sendTransaction

As per documentation of ethsigner: "https://docs.ethsigner.pegasys.tech/en/latest/Using-EthSigner/Using-EthSigner/"

EthSigner provides transaction signing and access to your keystore by implementing the following JSON-RPC methods:
eth_accounts
eth_sendTransaction

As eth_sendTransaction is used only for signing and sending public transactions.
But for signing and sending private transactions from the console we need an endpoint as like as eea_sendTransaction. So that we can add fields like privateFor, privateFrom, Restriction while sending the transactions with fields: from,to,value nonce,gas.

Is eea_sendTransaction in your development stage?

Can't create issues on Jira issue tracker

The README file says issues should be created on Jira, but it isn't possible for the general public to create issues there at the moment.

As a workaround, I think it's acceptable to create issues here until this is fixed.

Suppress the Java reflective access errors in the log

When EthSigner is run old versions of Java e.g. Java 8 then we get a lot of errors like "java.lang.UnsupportedOperationException: Reflective setAccessible(true) disabled" in the logs. These can be ignore and can be suppressed by tweaking the Java options in gradle.

This was fixed in Besu so we can copy what was done there.

CORS Issue when POST Method is done

Hi,
I've been facing a CORS issue when making post requests from my frontend.
Haven't found any documentation about it, and the only solution was installing Chrome's CORS extension. Since I want other people to make requests from my DAPP, this should be solved in the backend.
Any incoming update on this area?

Signing transactions with keys generated at runtime.

Hello,

I'm running ehtsigner in multikey mode and i'm generating keys at runtime (toml files and vault entries) but I cannot use them to sign transactions, while eth_accounts returns newly generated keys.

2020-11-06 09:44:44.407+01:00 | vert.x-worker-thread-3 | INFO  | SendTransactionHandler | From address (0x4a614e64192824b59a8cf227e2a7c8fe1438dcb5) does not match any available account

$ curl -X POST --data '{"jsonrpc":"2.0","method":"eth_accounts","params":[],"id":0}' http://127.0.0.1:8545
{"jsonrpc":"2.0","id":0,"result":["0x3bdaf240808f7bb7db9a0447ee6e24fdb26c346d","0x4a614e64192824b59a8cf227e2a7c8fe1438dcb5","0xa98f8ab0636b9b893bc88dd64727ced57f24a77d","0xbfb3c8333f44714af13f729e3f676b11acce4189","0xfe3b557e8fb62b89f4916b721be55ceb828dbd73"]}

After restarting ethsigner I can sign transactions with 0x4a614e64192824b59a8cf227e2a7c8fe1438dcb5 key successfully.

According to docs keys can be added without restarting ethsigner.
Files can be added or removed from the directory without needing to restart EthSigner.
https://docs.ethsigner.consensys.net/en/latest/HowTo/Use-Multiple-Signers/

It seems like AddressIndexedSignerProvider#addressToPublicKeyMap is never updated.
https://github.com/ConsenSys/ethsigner/blob/b55e5dc2ee1af4e91d44240a2d00200e131a2323/ethsigner/core/src/main/java/tech/pegasys/ethsigner/core/AddressIndexedSignerProvider.java#L28

I would expect ethsigner to sign transactions with newly generated keys without restart.

"invalid signature" error returned for nonce 70 (0x46) in Multi signers mode for multiple identical transacitons

Issue description

When running Hyperledger Besu + EthSigner with multiple signer, then sending multiple identical transactions (only the nonce incremented by one) an error is returned for nonce 70 (0x46):

{
  "jsonrpc" : "2.0",
  "id" : 1,
  "error" : {
    "code" : -32002,
    "message" : "Invalid signature"
  }
}

Environment

• Besu v1.4.0
• EthSigner v0.6.1

How to reproduce the issue

1. Spin up new Besu node, using the genesis as indicated in https://besu.hyperledger.org/en/stable/Tutorials/Private-Network/Create-IBFT-Network/#2-create-configuration-file
2. Spin up EthSigner in multi signer mode using files and create key & toml files for 2 accounts as per https://besu.hyperledger.org/en/stable/Reference/Accounts-for-Testing/
   The test accounts used in the config were f17f52151ebef6c7334fad080c5704d77216b732 and 627306090abab3a6e1400e9345bc60c78a8bef57
3. Submit 70 transactions like below, where the only difference is the nonce (it needs to go from 0x0 to 0x46):

   curl -X POST --data '{"jsonrpc":"2.0","method":"eth_sendTransaction","params":[{"from": 
   "0xf17f52151ebef6c7334fad080c5704d77216b732","to": 
   "0x627306090abaB3A6e1400e9345bC60c78a8BEf57","gas": "0x7600","gasPrice": 
   "0x00","value": "0x1", "nonce":"0x0", "data":"0x00"}], "id":1}' http://localhost:8545
   

4. Once I get to a nonce value of "0x46" (70) for account "0xf17f52151ebef6c7334fad080c5704d77216b732" I get the "Invalid signature" error message.

Additional info

I noticed that changing the tx a bit makes it pass, so for instance setting the value to "0x2" works even with 0x46 as nonce.

Setup nightly builds to determine when something breaks

Update EthSigner AT to use latest Besu version

Update EthSigner AT to use latest Besu version (21.1.1) and fix any issues which are identified.

Add option for configuring allowed origin domain

From Gitter:

Hi! I have a problem that I'm unable to resolve... I'm trying to connect Remix (Web3 provider) to EthSigner, because I'm working with a permissioned Besu network. In order to get that, I'm running EthSigner on a Docker container. I know that EthSigner - Besu node connection is working properly, because if I run this command, I can get this result:
curl -X POST --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":51}' http://192.168.43.125:9101
{
"jsonrpc" : "2.0",
"id" : 51,
"result" : "0xfe89"
}
But if I try to connect with Remix (Web3 provider), I get this error from Remix pop-up: Cannot get account list: Error: Invalid JSON RPC response: ""
And if I open the firefox console, i get this: Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://192.168.43.125:9101/. (Reason: CORS header 'Access-Control-Allow-Origin' missing)
I'm using http version of Remix, so I don't know what is the problem. What can I do? Maybe EthSigner isn't compatible with Remix Web3 Provider? Any ideas?
Update: I installed firefox add-on: "Allow CORS: Access-Control-Allow-Origin" and it's working now. But I think EthSigner should have any option for setting up this

We should have a CLI option for configuring the domain that we want to accept and add the Access-Control-Allow-Origin header to EthSigner response (when this config option is used).

The property might be --http-cors-origins and it should accept a list of domains that will be added to the Access-Control-Allow-Origin header in the response and enforced on EthSigner side.

Definition of done:

• EthSigner accepts the --http-cors-origins config option with a list of valid domains
• When the property is set, EthSigner will enforce that only connection from allowed origins will be accepted.
• When the property is set, EthSigner adds the Access-Control-Allow-Origin header with the list of accepted domain in every response to the client.
• Documentation should be updated

[ES-11] release ethsigner

Actual:

No jar hosted in maven/jcenter

Expected:

A release of ethsigner should be hosted in a public repo

Reason:

I would like to be able to write integration/acceptance tests for features in other projects that require ethsigner. Currently, I need to build a jar and copy the jar in the other projects. This is not ideal because I need to rebuild the jar every time ethsigner features are added and there needs to be this jar shipped with the source code to be able to compile. I should be able to handle this with gradle/maven...

ES-11