limechain/hashport-validator

Official repository containing the source code of the Hashport validators

Hashport Validator

Overview

This repository contains the Hedera <-> EVM Bridge Node. The bridge is operated by a set of validators who are running the Bridge Node software.

Technologies

The Validator node is using Hedera Consensus Service for aggregating authorisation signatures resolving the need for nodes to have p2p communication and providing traceability for the bridging operations. The node is a Go service with several watchers and handlers for Transfers, Message submission and EVM-based events. Postgres is used for persisting state. Prometheus and Grafana are used for monitoring the Hashport.

Prerequisite Tools

Necessary tools prior to running the validator:

• Docker

How to run?

To run the validator, execute the following commands in your terminal:

git clone https://github.com/LimeChain/hedera-eth-bridge-validator.git
cd hedera-eth-bridge-validator
docker-compose up

Documentation

• Overview
• Integration
• Installation
• Configuration
• Testing
• Workflows
• Release
• Mainnet Deployment
• Metrics
• API

Examples

• Three Validators Bridge Network

hashport-validator's Issues

Debug Mode

• Add debug mode that logs most of the changes that are occurring inside the application
• Add information in the docs on how to run it in debug mode
  *Note there are a lot of debug logs already so it might be only a matter of running it in "debug mode"

CryptoTransfer Event Handler

• Implement Watcher SDK event handler

Process for new TX:

0. Add new TX in the DB
1. Sanity check on the message (moved to Watcher)
2. State Proof Verification (moved to Watcher)
3. Fee Calculation (currently mocked)
4. Sign Authorisation Message
5. Submit to HCS
6. Update DB TX with the HCS TX ID (containing the Signature)
7. Wait for TX Success
8. Update DB TX with the new status

Test Environment

• Deploy a long-lived test environment with 3 validators on 3 container-optimised VMs.
• Add new three-validator-network.md doc file in the /docs folder that describes the process of deployment. Provide instructions on how to deploy new releases.
• The test-environment must be working on Hedera Testnet and Ethereum Testnet (to be defined later which test net will be used)

CryptoTransfer Event Handler flow

Process for new TX:
0. Add new TX in the DB
3. Fee Calculation
4. Sign Authorisation Message
5. Submit to HCS
6. Update DB TX with the HCS TX Hash (containing the Signature)

WHBAR Token

• Implement WHBAR mintable, burnable, pausable erc20 token
• Deploy WHBAR token on Georli to be used for the Test Environment
• Owner is the multisig wallet owned by the validaotrs
• Only bridgeContract can mint / burn
• bridgeContract can be changed only by the owner
• example

Watcher Application

CryptoTransfer Watcher

• Implement watcher SDK watcher interface
• Must be able to listen for transactions to a given account
• Must be able to recover on start in order to start watching from the previously processed timestamp

Gnosis Multisig

• Deploy Gnosis MultiSig managed by 3 custodians on Test Environment
• Use the Gnosis safe to deploy the bridge contract and the WHBAR token
• Test the use-case for adding new custodians

Implement CryptoTransfer Handler Unit tests

The crypto-transfer handler is crucial to the processing of Bridge TX. We need to make sure that it's working correctly.

Installation, Configuration & Deployment docs

• Add new deployment.md file that contains instructions on how to deploy the validator node into GCP.
• Add new config.md file that contains instructions on how to configure the env variables of a validator node
• Add new local-development.md file that contains instructions on how to run the bridge node locally and start developing

Automatically Update Contract-level Config

At the moment, there are several properties that are controlled on the contract level, however, they are set in the configuration application.yaml file (hardcoded). Those properties include:

• hedera.client.operator.service_fee_percent -> this can be read from the Smart Contract
• hedera.handler.consensusmessage.addresses -> the currently set validators in the contract. This can be read from the contract as-well

Issue:
If the validator set is changed in the contract, the node must be aware of that so that the number of signatures threshold is properly
calculated, the integrity of the auth signatures is validated correctly and the fee calculation is property estimated.

Tasks:

• add new function to the ethereum bridge service: loadConfig. This function must be called on node start and must:

1. Query the serviceFee from the contract; 2) Set the service fee in-memory in the EthereumBridgeService
2. Query the custodians from the contract; 2) Set the custodians in-memory in the EthereumBridgeService

• add new functions to the current ethereum bridge service: WatchFeeChangedEvent and WatchCustodianChanged
• When the ethereum watcher is started ("listenForEvents") we must subscribe to the WatchFeeChangedEvent and WatchCustodianChanged events as-well (not only for the burn event). Once triggered, the appropriate property must be updated (f.e service fee / custodian) in the EthereumBridgeService. Note: For the update of the properties, use Mutex and log the update
• Update the fee-calculator in such away that it calls the EthereumBridgeService to read the serviceFee property and the custodians count
• Update the CryptoTransferHandler to use the EthereumBridgeService for getting the custodians count
• Remove the hedera.client.operator.service_fee_percent and hedera.handler.consensusmessage.addresses from the application.yml

Implement State Proof Verification for V5 record files

At the moment, the validator node does not perform state proof verification.
The current state proof verification must be updated to support the new V5 Record file version.

This issue is blocked by the JS implementation of the record file verification.

Resolve Mirror node timeout

Watcher connections to the mirror node are dropped after several minutes.
Verify that connections are dropped only if there is more than 1 connection to the mirror node (in the 3 validator network). If the problem occurs only in the 3 node validator network we can dismiss the issue.

If the problem occurs in a standalone node as-well, the issue must be resolved.

How to reproduce:

• Start a signal node and wait for ~10 minutes. See in the logs whether there are any issues with a timed out connections.
• Start the three-node validator docker-compose. Wait a couple of minutes and notice the timeout errors.

E2E Tests

Add new e2e folder and implement the following end-to-end tests.

Config: Hedera Account with HBARs (stored as env variable or passed as argument)

1. Send HBARs to the configured custodial account.
2. Verify that auth signatures were published in HCS from all of the nodes
3. Verify that the first validator submitted HCS message with the Ethereum TX Hash
4. Verify that in Ethereum, the TX was mined, the appropriate service and TX reimbursement fees were charged, the specified Ethereum address received the required WHBARs.

Keep in mind that this e2e could be included as part of a CI/CD pipeline to be run as a smoke test. This means that the configuration of the variables such as sending address PK, bridge custodial account, HCS topic, ETH network and ETH smart contract must be configurable from env variables. Once this E2E test is integrated into the CI/CD pipeline we will be running the 3 validator network against Previewnet and Ethereum testnet OR Ganache

DockerHub Image

• Configure a release set up in GitHub Actions, that publishes the validator-node image into DockerHub.
• Details on how do it: https://docs.github.com/en/actions/guides/publishing-docker-images

Fee Calculator for CryptoTransfer Event Handler

1. Sanity Check
   Implement the fee-calculator.go to validate that the amount that is being transferred is transferAmount > txFee + serviceFee, where txFee is set in the crypto transfer memo and serviceFee is the fee set in the smart contract

2. Fee Estimator

• Add exchange rate service that updates the HBAR/ETH price every 1 hour from external API
• Store the exchange rate in a Postgres DB
• every time a new request is triggered, and ValidateExecutionFee is called:

1. perform a sanity check of the fee (1)
2. Get the gasPrice from the encoded gasPrice in the crypto transfer memo
3. using the estimated gas usage and gas price, calculate the required eth for the TX
4. Using the ETH/HBAR exchange rate, calculate the txFee in HBARS
5. Validate that the txFee >= estimated HBAR fee

3. Implement Unit tests for the fee calculator

Generalised Assets Bridge

At the moment the bridge is solely intended to operate with only HBARs. Ideally, we would like to extend it to any kind of assets, including bringing assets from Ethereum to Hedera.

Major Components to be changed:

• Supported tokens config (?retrieved from eth contract?)
• CryptoTransfer Watchers (watch for tokens not only for hbars)
• Signature message format (include the token address to be bridged)
• Smart Contract Change -> refactor the contracts so that an array of ERC20 tokens are supported; change burn event params to include the erc20 token to be burned
• Burn event watcher ->

Note:

• threshold account should be associated with the HTS token

Upload Logs from the nodes on e2e test run

Github actions support the "uploading of archive" information as part of every run.
It would be beneficial to upload the logs from the containers and the e2e tests as part of every run,

StateProof Verification

• Implement verification of the state proof (alpha)

Transaction Lifecycle (+persisting Ethereum TX statuses)

At the moment, when we receive a message with EthTX hash we are only cancelling the scheduled execution (consensus-message.go handler, handleEthTxMessage function.

Ideally, we would want to store the information whether ETH was completely mined at the end in order to do that we need to persist it in the DB.

1. Change the current statuses of the transaction.go entity to the following:

• INITIAL -> the first status once TX is created
• INSUFFICIENT_FEE -> after it is created, fees.ValidateExecutionFee is called and the result is validFee=false
• SIGNATURE_SUBMITTED -> after the auth signature was submitted (handleTopicSubmission), before the Transaction is mined.
• SIGNATURE_PROVIDED -> after the topic message was mined and accepted successfully (in checkForTransactionCompletion)
• SIGNATURE_FAILED -> if the message submission (for the auth signature) has failed (in checkForTransactionCompletion)
• ETH_TX_SUBMITTED -> if there is eth tx hash message published into the topic (in handleEthTxMessage)
• ETH_TX_REVERTED -> if the eth tx reverted after submission.
• COMPLETED -> if the eth tx was mined successfully

2. Once we receive new handleEthTxMessage we should update the status to ETH_TX_SUBMITTED initially and wait for the TX to be mined. After it is mined, we should check whether it reverted or not. If it reverts we must update the status to ETH_TX_REVERTED if is succeeds we must update the status to COMPLETED

Create project folder structure

Controller Contract

Implement & Test Controller contract such that:

• is able to mint / burn tokens

Explore solutions for returning User funds on error

At the moment, if a user submits an incorrectly encoded memo or insufficient fee, the TX will be dropped and won't be processed.
Validators must be able to refund lost user funds back

ConsensusMessage Event Handler

PR1 - Pending PR

• receive new message
• verification of auth signature (ethereum message) -> compare it with the set of addresses in the smart contract (use config of addresses).
• saving into db the auth message (link it to a transaction id)

PR2

• count signatures
• determine the soft elected leader
  (1)
  • if soft elected leader => submit TX to smart contract (mock it in the beginning since we do not have smart contract)
  • publish to HCS topic the TX hash

PR3

• If leader does not publish to HCS topic in a given minutes N => determine the second soft elected leader and execute (1)

Define Database Models

Policy Engine Application

ConsensusMessage Watcher

Ethereum Watcher

Watch for Contract Events

Establish DB connection

• Use gORM - go get -u gorm.io/gorm

Hedera Watcher Application

Add E2E tests to the CI workflow

At the moment we have E2E tests that validate the straightforward use-case for the Bridge.
Ideally we would want to have e2e tests as part of the CI workflow.

Implement a make file inside the e2e folder that:

1. runs the 3 node validator network
2. runs the e2e tests.

Once implemented, adding the e2e tests to run as part of the github workflow on every PR will be easy.

Note: The e2e tests will be time-consuming because we are waiting for the ETH TX to be mined. Thats why it will be better to run e2e tests on PR and not on every commit

Smart Contracts

Design and deploy an ERC20 Mintable/Burnable WHBAR token on Ethereum

Bridge Contract

Implement Bridge contact such that:

• Has a set of public keys -> custodians.

• Custodians can be changed only by the owner of the bridge.

• The owner of the bridge contract is a multisig wallet governed by the bridge operators (custodians).

• Only the bridge contract can mint / burn tokens.

• mint -> mints tokens to the provided address. Majority of operator signatures must be provided. fee parameter must be accredited to the tx sender. The fee is part of the signed message.

• burn -> anyone can call this function and provide WHBARs to be burned. Once burned, an event is emitted (picked up validators later on)

Create new repository with the smart contracts

• Create new repository containing the smart contracts of the Bridge -> WHBAR token + Bridge Contract
• Add deployment script that can be executed easily so that anyone can deploy the set Token/Bridge contracts that he needs for local development

Verify DB records after E2E execution

• Implement a DB verification service inside the e2e folder.
• The service must be able to easily assert that a given record exists in the DB.
• The service could be using the already implemented repositories for the validator node

We would like to ideally be able to:

• verify that a record is created by theTransactionRepo and it has the correct properties at the end of the execution
• verify that there are TransactionMessage records created by the MessageRepo and all of them have correct properties at the end of the execution.

Docker Compose

• Create a docker compose file that runs 3 watchers (bootstrapping a bridge network)

CryptoTransfer Event Handler Recovery

• Should be able to recover on start any transactions that are not verified and published into the bridge topic

Integrate hedera-watcher-sdk

Setup a AWS KMS infrastructure

Extend the eth signer to be able to use an external KMS in AWS.
Setup AWS KMS for with elliptic curve key (PK for the validator TX)
Extend the signer so that it uses the PK for signing mint transactions.

Example in TS:
https://github.com/lucashenning/aws-kms-ethereum-signing

AWS KMS docs:
https://aws.amazon.com/kms/resources/

It will be beneficial to describe the process in notion & have the AWS KMS eth signer in a separate repository inside LimeChain, so that we can reuse it in other projects.

Recovery Mechanism

Implementation of recovery mechanism.
On start, the go server should get all non-processed transactions and schedule them for processing

Expose REST endpoint MEMO info

Problem:
If the user does not provide correctly encoded memo the funds will be sent to the custodial account, but the transaction won't be processed. There are several ways a user can mess up the memo, so that the node rejects to process it. The most obvious one is providing insufficient fee or invalid gas price.

Solution
One solution for resolving the problem of sending funds and not get their transactions processed is by introducing an endpoint in the Validator Node.

Example:
Intorduce /api/v1/metadata endpoint. Once called it will return the following object:

{
    "txFee": "X",
    "txFeeCurrency": "HBAR",
    "gasPriceGwei": "Y"
}

where txFee is the "suggested" tx fee for the transaction at the moment and gasPrice is the suggested gas price at the moment (slow gas price from Etherscan f.e).
txFee can be calculated based on the newest FeeCalculator logic. At the moment, the fee calculator does the following:
By a given txFee, amountToTransfer and gasPriceGwei provided in the memo, verifies that:
txFee > estimatedFee, where estimatedFee=((hardcoded_gas_usage_in_config)*(specified_gas_price)) * HBAR/ETH rate.

for this API to work, we need another method to be implemented in the calculator that returns the txFee and gasPrice that should be used right now.
How to calculate:

• for gasPriceGwei get the current slow gas price from etherscan coingecko if they have it so that we have only one external 3rd party API integration
  based on that gasPriceGwei, multiply it to the hardcoded_gas_usage_in_config, this results in ETH(wei) required for the TX`.
• get the current rate between HBAR/ETH, convert it to tinyBar/wei
• convert the ETH Required for the TX to tinybars required by the TX using the tinyBar/wei currency rate
• as a result, return the calculated result (tinybars required by the TX) + the gasPriceGwei used for the calculation.

Infrastructure

All infrastructure tasks required for the initial MVP release.

Verify ETH TX Hash Authenticity

At the moment the consensus-message handler does not verify the TX Hash that is submitted to the topic. A validator can submit another TX and provide the Hash and this would cause the validators to unschedule all of their transactions.

Enhance the validator consensus-message handler by implementing a logic for verifying that the TX hash that is published in the HCS topic:

• Is targeting the right contract
• is calling the right function
• has the correct parameters

One way to do it might be to execute estimateGas function with the same parameters. Keep in mind that if we are going to do it this way, the estimateGas function might return revering TX because the original TX might get mined and the estimate gas would revert due to the checks in the mint function.

Double processing of Consensus Messages

Make sure that no consensus message is processed twice. The consensus message watcher starts listening for messages after timestamp X and checks whether there are any messages that were missed during the time that the node was offline. Make sure that once the node is started, the messages that were sent are correctly processed.

How to test:

• turn off the node, execute 1 crypto transfer that will initiate the process of bridging, turn on the node, verify that the auth signatures are going to be processed only if it is still in progress and the other nodes haven't completed the bridge operation

Database Migrations

Create Migration for:

• db creation
• tables creation

Explore KMS solution

Result of exploration - #89

Add licences

Double processing of account transfers

Make sure that no crypto transfer is processed twice. The crypto transfer watcher starts listening for account transfers after timestamp X and checks whether there are any TX that were missed during the time that the node was offline. Make sure that once the node is started, the transactions that were sent are correctly processed.

How to test:

• turn off the node, execute 5 transfers, turn on the node, verify that those 5 transactions are going to be processed only if it is still in progress and the other nodes haven't completed the bridge operation

Implementation details:

1. Recovery mechanism to be implemented separated from the watchers
2. Once node starts, the recovery is initiated and "recovers" the transactions up until the starting timestamp of the node.
3. Once the recovery finishes, the watchers are started with the same timestamp as the starting timestamp of the node.

Recovery:

1. The from = timestamp set in the config (higher priority) OR timestamp from DB
2. to = node starting timestamp
3. Get all messages from Pub/sub between from and to
4. Parse the messages and group them by the ID

Ignore for now:

1. Once the crypto transfer watcher starts, he must get all of the crypto-transfers that were not processed by him (based on the timestamp from the DB or the config). Once the crypto-transfers are queried from the mirror node, the watcher must figure-out the transactions that have already been processed. In order to do that, he must query the status of the transactions from the Ethereum contract. If the status is "executed", the tx has already been processed

Database persistence

Add Github Action for running Unit Tests