ava-labs/coreth

Code and wrapper to extract Ethereum blockchain functionalities without network/consensus, for building custom blockchain services.

Coreth and the C-Chain

Avalanche is a network composed of multiple blockchains. Each blockchain is an instance of a Virtual Machine (VM), much like an object in an object-oriented language is an instance of a class. That is, the VM defines the behavior of the blockchain. Coreth (from core Ethereum) is the Virtual Machine (VM) that defines the Contract Chain (C-Chain). This chain implements the Ethereum Virtual Machine and supports Solidity smart contracts as well as most other Ethereum client functionality.

Building

Coreth is a dependency of AvalancheGo which is used to implement the EVM based Virtual Machine for the Avalanche C-Chain. In order to run with a local version of Coreth, users must update their Coreth dependency within AvalancheGo to point to their local Coreth directory. If Coreth and AvalancheGo are at the standard location within your GOPATH, this will look like the following:

cd $GOPATH/src/github.com/ava-labs/avalanchego
go mod edit -replace github.com/ava-labs/coreth=../coreth

Now that AvalancheGo depends on the local version of Coreth, we can build with the normal build script:

./scripts/build.sh
./build/avalanchego

Note: the C-Chain originally ran in a separate process from the main AvalancheGo process and communicated with it over a local gRPC connection. When this was the case, AvalancheGo's build script would download Coreth, compile it, and place the binary into the avalanchego/build/plugins directory.

API

The C-Chain supports the following API namespaces:

• eth
• personal
• txpool
• debug

Only the eth namespace is enabled by default. To enable the other namespaces see the instructions for passing the C-Chain config to AvalancheGo here. Full documentation for the C-Chain's API can be found here.

Compatibility

The C-Chain is compatible with almost all Ethereum tooling, including Core, Metamask, Remix and Truffle.

Differences Between Avalanche C-Chain and Ethereum

Atomic Transactions

As a network composed of multiple blockchains, Avalanche uses atomic transactions to move assets between chains. Coreth modifies the Ethereum block format by adding an ExtraData field, which contains the atomic transactions.

Block Timing

Blocks are produced asynchronously in Snowman Consensus, so the timing assumptions that apply to Ethereum do not apply to Coreth. To support block production in an async environment, a block is permitted to have the same timestamp as its parent. Since there is no general assumption that a block will be produced every 10 seconds, smart contracts built on Avalanche should use the block timestamp instead of the block number for their timing assumptions.

A block with a timestamp more than 10 seconds in the future will not be considered valid. However, a block with a timestamp more than 10 seconds in the past will still be considered valid as long as its timestamp is greater than or equal to the timestamp of its parent block.

Difficulty and Random OpCode

Snowman consensus does not use difficulty in any way, so the difficulty of every block is required to be set to 1. This means that the DIFFICULTY opcode should not be used as a source of randomness.

Additionally, with the change from the DIFFICULTY OpCode to the RANDOM OpCode (RANDOM replaces DIFFICULTY directly), there is no planned change to provide a stronger source of randomness. The RANDOM OpCode relies on the Eth2.0 Randomness Beacon, which has no direct parallel within the context of either Coreth or Snowman consensus. Therefore, instead of providing a weaker source of randomness that may be manipulated, the RANDOM OpCode will not be supported. Instead, it will continue the behavior of the DIFFICULTY OpCode of returning the block's difficulty, such that it will always return 1.

Block Format

To support these changes, there have been a number of changes to the C-Chain block format compared to what exists on Ethereum.

Block Body

• Version: provides version of the ExtData in the block. Currently, this field is always 0.
• ExtData: extra data field within the block body to store atomic transaction bytes.

Block Header

• ExtDataHash: the hash of the bytes in the ExtDataHash field
• BaseFee: Added by EIP-1559 to represent the base fee of the block (present in Ethereum as of EIP-1559)
• ExtDataGasUsed: amount of gas consumed by the atomic transactions in the block
• BlockGasCost: surcharge for producing a block faster than the target rate

coreth's Issues

Unknown Codec Version during State Sync

Receiving the following errors while the C-Chain is state syncing indicating that peers are sending an invalid response during state sync:

INFO [07-08|15:45:26.834] <C Chain> github.com/ava-labs/coreth/sync/client/client.go:354: could not validate response, retrying nodeID=NodeID-8fq1H8oStMi9BZrEk8qiE8QAV7AtqGnXt attempt=0 request="LeafsRequest(Root=0x2420ccaf1b045b032a9643a835ad02e94b528099c402c4a060ba18a2faad8a6f, Account=0x4bf87171a2265f14de1abdf79c9d5d9c4f1d758eb98fd2e380d4a24ea3001fe1, Start=, End=, Limit=1024, NodeType=StateTrie)" err="couldn't unpack codec version"

INFO [07-08|16:08:58.449] <C Chain> github.com/ava-labs/coreth/sync/client/client.go:354: could not validate response, retrying nodeID=NodeID-AyByRGSrfbDaeW4njEYwCzhsnMyZ2KMcw attempt=0 request="LeafsRequest(Root=0x29b9b5c98f6f7f932bf533a452afb7e4e30417d4c102def28160b3df59112cc2, Account=0x784448fe2d7fb2d02af9b739ac0cf8ba294f89cd1830165ca452abb306439aab, Start=, End=, Limit=1024, NodeType=StateTrie)" err="couldn't unpack codec version"

INFO [07-08|17:26:20.239] <C Chain> github.com/ava-labs/coreth/sync/client/client.go:354: could not validate response, retrying nodeID=NodeID-6yS1J9HFdHgdX2ruW6XMouQvECDF2Fta4 attempt=0 request="LeafsRequest(Root=0x07b2ee27d2f5878fd52e35a4650fb54718b8af4123109ec36e5b2989c1176209, Account=0xba7a9ebf2263882130fcaba3d2ed4830d2eac01a11edc8fdac30bad6e97b0db2, Start=70319f3298d7f3568fc2806799da6cc9e165375f493ea769fbc1d94669c248a5, End=, Limit=1024, NodeType=StateTrie)" err="couldn't unpack codec version"

INFO [07-08|18:49:44.415] <C Chain> github.com/ava-labs/coreth/sync/client/client.go:354: could not validate response, retrying nodeID=NodeID-2KSietvSEq4mX2C8p1DAT83RTJPykxUqN attempt=0 request="LeafsRequest(Root=0x561bebb501f5988587fb53173711c9c93342be01c555282f2e42d20f4259ad00, Account=0x5a28e9e44ad9190861adab00f7b8fd0b3163bc1f45cfb74bf479b3716c5d5f40, Start=5b26cd7be727cd1e8b3b3409f9bae0a8642c593e3ee834d9871922da0f1756e1, End=, Limit=1024, NodeType=StateTrie)" err="couldn't unpack codec version"

INFO [07-08|19:36:23.473] <C Chain> github.com/ava-labs/coreth/sync/client/client.go:354: could not validate response, retrying nodeID=NodeID-6yS1J9HFdHgdX2ruW6XMouQvECDF2Fta4 attempt=0 request="LeafsRequest(Root=0x7dceafacc84480384741e247e822710404db229e508cc9980e9ea6be85a03777, Account=0x46d33986f2530692b444d5ff3e1da9646b7b393f2aad7fa5f88cbf30f50ae0c2, Start=, End=, Limit=1024, NodeType=StateTrie)" err="couldn't unpack codec version"

Cannot estimate gas on simple send transaction

Some Ethereum nodes are not able to calculate the gas correctly due to some external and arbitrary calls. However they give some estimation (even if it's not correct).

There is a very often error thrown by the RPC calls regarding the fee estimation:

{
   "reason":"cannot estimate gas; transaction may fail or may require manual gas limit",
   "code":"UNPREDICTABLE_GAS_LIMIT",
   "error":{
      "reason":"processing response error",
      "code":"SERVER_ERROR",
      "body":"{\"jsonrpc\":\"2.0\",\"id\":68,\"error\":{\"code\":-32000,\"message\":\"method handler crashed\"}}\n",
      "error":{
         "code":-32000
      },
      "requestBody":"{\"method\":\"eth_estimateGas\",\"params\":[{\"gasPrice\":\"0xaf16b1600\",\"value\":\"0x5af3107a4000\",\"from\":\"0xde174710543dced471a5747fe3060d11e881a356\",\"to\":\"0x9073086eab0357a6bbd2f3b3a941645ace9a40f6\"}],\"id\":68,\"jsonrpc\":\"2.0\"}",
      "requestMethod":"POST",
      "url":"http://127.0.0.1:9650/ext/bc/C/rpc"
   }
}

Broken functionality

Version: v0.4.2

NewLevelDBDatabaseWithFreezer

is broken. It initialises function

leveldb.New(file, cache, handles, namespace)

but it expects

leveldb.New(file, cache, handles, namespace, false) // 5th parameter is missing

and in NewDatabaseWithCache

return &cachingDB{
		db:            trie.NewDatabaseWithCache(db, cache, journal),
		codeSizeCache: csc,
		codeCache:     fastcache.New(codeCacheSize),
	}

trie.NewDatabaseWithCache is missing

Block.Hash produces a wrong hash

Running block.Hash() produces a wrong hash when comparing to the actual hash from the RPC response.
Filing the ticket per @aaronbuchwald request.

Code to reproduce the issue:

package main

import (
	"context"
	"log"

	"github.com/ethereum/go-ethereum/ethclient"
)

func main() {
	client, err := ethclient.Dial("https://api.avax-test.network/ext/bc/C/rpc")
	if err != nil {
		log.Fatal(err)
	}

	header, err := client.HeaderByNumber(context.Background(), nil)
	if err != nil {
		log.Fatal(err)
	}

	log.Println("block number:", header.Number)
	log.Println("block hash:", header.Hash().Hex())

        // This will fail as `header.Hash()` returns an invalid hash value
	header2, err := client.HeaderByHash(context.Background(), header.Hash())
	if err != nil {
		log.Fatal(err)
	}

	log.Println("fetched number:", header2.Number)
	log.Println("fetched hash:", header2.Hash().Hex())
}

Flaky Test: TestVMShutdownWhileSyncing

https://github.com/ava-labs/coreth/actions/runs/5246622497/jobs/9475682494?pr=256#step:8:1040

re-support debug_standardTraceBlockToFile

After burning two days trying to figure out how to get trace data out of avalanchego without having to painfully implement everything I want to do inside of coreth using Go (see ava-labs/avalanchego#984, where I finally won against the numerous arbitrary limits imposed on the GRPC ping-pong being played with my 3G JSON result), I'm realizing that, in a very real sense, the core underlying issue I'm running into here is that Avalanche removed support for debug_standardTraceBlockToFile--which the geth people explicitly introduced in order to solve this problem of allowing the state to be streamed to disk rather than re-architecting all of the JSON encoders to not buffer everything into RAM--in 439d173. I don't understand why this was removed instead of merely moved to something like internal-file-tracing-debug and I particularly don't understand why the code for it had to be entirely deleted instead of merely disconnected (maximizing differences between coreth and geth).

Regardless, I can confirm that the code for this method from geth, if placed back in eth/tracers/api.go and edited for some trivial geth/coreth differences (which are hopefully due to version skew and aren't egregious edits), works. (Note that I did remove the Overrides if-block, as I didn't want to figure out what was going on with the namespaces there.)

upgrading geth to support Istanbul

The Istanbul release of Ethereum from last December had a number of important changes, including a new opcode (chainid) that is used by our contracts for layer 2 fork protection (though the net gas metering change is also really important for a lot of contracts now, and the adjustments to the various other costs make some use cases much more viable, including zkp and rollups). Is there any hope of getting geth upgraded to a newer version? (I am not a Go developer myself, but it looks like you might not have made any real extensive changes to geth, other than to systematically change the all of import paths to your GitHub organization?)

https://medium.com/mycrypto/ethereums-istanbul-fork-technical-explanation-48a3aef718b0

How to use local build of coreth in avalanchego?

I've downloaded and built avalanchego and coreth. I'm having trouble figuring out how to get avalanchego to use my local build of coreth - I've searched the avalanche docs and the docs for the two repos but haven't found instructions.

How do I get a local build of avalanchego to use a local build of coreth? Do I have to place the coreth binary under avalanchego/build/plugins? If so, does the binary need to have a specific name?

Blockscout DDoS network for rejected blocks (failed to fetch internal transactions for blocks | cannot access atomic state for block)

Describe the bug
We have deployed private test network based on Avalanche.
Use Blockscout as Explorer for it.
After making several transactions in network (transfers from wallet to wallet), Blockscout continuously in loop requests network for rejected blocks and shows error in terminal (pointed in Logs section). This error log has led us to coreth.

To Reproduce
Nothing extra. Doing several regular transactions.

Expected behavior
No DDoS.

Logs
2023-05-09T14:04:36.831 application=indexer fetcher=internal_transaction count=10 error_count=10 [error] failed to fetch internal transactions for blocks: ** (ErlangError) Erlang error: [%{code: -32000, data: %{block_number: 3, transaction_hash: "0xc38b00b403bfe564d5b929a99f07271a1d53e1448252cf03c2a01ce1cb584200", transaction_index: 0}, message: "processing block 1 failed: engine finalization check failed: cannot access atomic state for block 0xe7b0052f525771fd8aa9b7f20fbf6f6221d9b89d23e4e1c0f7e2d07b9c68d5f4"}]

Operating System
Ubuntu

gas price should be set with some kind of market algorithm

(I was asked by @danlaine to bring this issue, which I noted a few days ago and which has been coming up occasionally both from myself and from someone else I don't know who was looking at the economics of running their DEX on Avalanche, from Discord to GitHub.)

The gas price on the various Avalanche EVM chains (C-chain, Athereum, etc.) is hardcoded; and I see #39, but in addition to only applying to other subnets, this is still just letting the subnet select a different genesis-hardcoded price, not resulting in a change of model. What makes this extremely problematic is that it means the transaction price of your network is thereby tied to your token price's volatility--which could fluctuate based on things like speculation and news/political cycles--as opposed to being any reasonable indicator of the cost of computation or bandwidth. In contrast, on Ethereum, if the price of ETH doubles, the cost of gas should quickly halve. With Avalanche, if the price of AVAX doubles, so does the cost of using the network, which has literally meant that I'm rooting for the value of the token to crash and stay depressed, so that the network is more usable... "awkward" ;P.

To provide what is hopefully a useless example of the kind of discussion surrounding this, I work on a project (Orchid) that sells bandwidth, and having this hardcoded gas price concept would be akin to the naive assumption many people seem to have about our product: that bandwidth costs are somehow fixed in OXT at like, 1 OXT / GB, and so people then ask us "how will the volatility of your token affect the viability of payments" and the answer is "it won't, as this is all based on market economics: if the token goes up the price of bandwidth as measured in that token will go down, as in some real sense the price of bandwidth is fundamentally measured in something like the price of bread and other staples; the amount of value from the user required to get bandwidth should only change if the price of networking in the world changes". Only, for Avalanche, this naive model is in fact how it works and is in fact going to be a problem.

As this value is currently hardcoded at 470 (a very recent change from 47 in dd6b031), the price of transactions on Avalanche are only 1/14.5th the cost of Ethereum right now; and it must be noted that the spread is actually only this high because the value of AVAX has been falling... if the value of AVAX were still closer to $6 (as it was for a while right after launch) the cost would be only around 1/9th. If the value of your token (awkwardly, "heaven forbid") goes up, that means that the usefulness of it over Ethereum will actually decline--and possibly quite quickly--if anyone deploys anything important that actually uses it (as opposed to merely encouraging people to start staking and running validators before you even have users, which is what you have been doing ;P).

I mean, already, the prices of Avalanche are kind of ridiculous for a system claiming to be so much fundamentally more economically viable than Ethereum, as while <10x is still better than nothing--and I appreciate that a key property of Avalanche is that it will (supposedly) scale to a lot more users and keep that same price--the price of Ethereum is and was already way too high for a lot of people, so merely being more scalable but not actually being drastically cheaper is, at least in the short term, kind of a massive let down: there are numerous other systems out there that have been in production longer with real users and which are also Ethereum compatible but which are offering prices that are already 10-100x lower than Avalanche, and that's with so-far zero usage of Avalanche (as the C-chain hasn't actually launched yet) and a falling token price.

FWIW, EIP-1559 has a mechanism proposed for Ethereum which is in many ways similar to the gas model used by Avalanche, which has people burning their fees rather than paying it to miners. In it, Vitalik et al discuss a model of shifting this fee based on the "percentage of the previous block" that is used. Now, I appreciate that Avalanche doesn't have blocks that work like this, but if you re-model gas target as "gas used per time" instead of "gas used per block" then you can probably use very a very similar algorithm to the one from EIP-1559 to auto-calibrate your gas price over time. (Not saying this is what you should do; I'm actually more a fan of the recently-released EIP-2593, which happens to be almost identical to the solution we had come up with for our layer 2 nanopayments--interestingly, ours is also in terms of time, while theirs is in terms of blocks ;P--but the current hardcoded concept and specific value are both unreasonable/concerning).

https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1559.md

C Chain Reflects block's state change before accepted

The C Chain sometimes reflects state changes of blocks before they've been accepted. Ie when a block is issued, the eth APIs report the block height increase and balance changes given in the block before the node has actually accepted the block.

Since adding the block to the chain does not mean that the block has been accepted by snowman consensus, this can cause UI issues.

Get Balance of an Asset on Xchain from Cchain

Hello there,

I've created an ANT asset on my node (X-chain) which is connected to Fuji network. Then I've deployed an ARC-20 contract on C-chain, which I've used some history contract . However I have an issue about depositing ANT tokens to ARC-20 contract. The transaction seems ok, but it's failing when tries to call NativeAssets.assetBalance() which sends a transaction to assetBalance contract.
Steps I've done:

• First I've created an ANT -> deployment
• Then I've minted some ANTs to my wallet -> transaction
• Next I've deployed contract with ANT's address here -> deployment
• Then I've called deposit() function of the contract -> transaction

The raw trace shows that, the last transaction fails, which is nativeAsset call.

[
  {
    "action": {
      "callType": "call",
      "from": "0xe6981a95885fdc9d2fcaa9a177648ecf7b124eda",
      "gas": "0x73A538",
      "input": "0xd0e30db0",
      "to": "0xffdafa401aea8041e1c77528e980e1b90d97ab1e",
      "value": "0x0"
    },
    "error": "execution reverted",
    "subtraces": 1,
    "traceAddress": [],
    "type": "call"
  },
  {
    "action": {
      "callType": "delegatecall",
      "from": "0xffdafa401aea8041e1c77528e980e1b90d97ab1e",
      "gas": "0x71C2D6",
      "input": "0xf271b3cc000000000000000000000000ffdafa401aea8041e1c77528e980e1b90d97ab1e198f6eea66310118477dbf29aff2eec83e2471fe08528fdec9642579713128e8",
      "to": "0xb7137ec15ff49b51071cc40fdfde863b011ac7b7",
      "value": "0x0"
    },
    "error": "execution reverted",
    "subtraces": 1,
    "traceAddress": [
      0
    ],
    "type": "call"
  },
  {
    "action": {
      "callType": "call",
      "from": "0xffdafa401aea8041e1c77528e980e1b90d97ab1e",
      "gas": "0x0",
      "input": "0xffdafa401aea8041e1c77528e980e1b90d97ab1e198f6eea66310118477dbf29aff2eec83e2471fe08528fdec9642579713128e8",
      "to": "0x0100000000000000000000000000000000000001",
      "value": "0x0"
    },
    "error": "execution reverted",
    "subtraces": 0,
    "traceAddress": [
      0,
      0
    ],
    "type": "call"
  }
]

I've looked into Coreth code here there should not be any problem with unpacking the transaction input 0xffdafa401aea8041e1c77528e980e1b90d97ab1e198f6eea66310118477dbf29aff2eec83e2471fe08528fdec9642579713128e8 .

Do you check the asset/contract names on cross chain transactions ? That might be my problem (one is HON, the other is ARC20), but I haven't see this kind of code in Coreth.
The other thing is there is no transactions on nativeAssetBalance contract .

Thank you

Flaky test (windows): TestChainIndexerWithChildren

Observed to fail on an unrelated pull request on the windows CI

--- FAIL: TestChainIndexerWithChildren (11.46s)
chain_indexer_test.go:166: Canonical section count mismatch: have 501, want 502
FAIL

debug_traceTransaction RPC command not available

Blockscout uses the debug_traceTransaction RPC call when indexing transactions. Without it the explorer is less useless, but it also seems to be causing Blockscout performance issues.

We need a way to enable this RPC, which belongs to the debug API group.

RPC documentation: https://github.com/ethereum/go-ethereum/wiki/Management-APIs#debug_tracetransaction

Set gasPrice in genesisData to allow separate subnets to configure gas price

Section processing failed

After broadcasting multiple “deploy” transactions, the chain got corrupted and the transactions were stuck in “Pending”.
Resetting the node did not help.

Log message:

ERROR[09-15|14:47:02.586] Section processing failed type=bloombits error="chain reorged during section processing"

where is the tx_fetcher.go and block_fetcher.go?

hi, I noticed that eth/tx_fetcher.go and eth/block_fetcher.go are removed from coreth, so where's are the hash -> transaction fetch logic now?

state sync failures observed with atomic root mismatch

Describe the bug
We have received 2 instances of reports with logs such as:

FATAL <C Chain> handler/handler.go:335 shutting down chain {"reason": "received an unexpected error", "error": "failed to notify VM that bootstrapping has started: synced root (0x35ae5c47aeab8d06a088b70034833af6414dfdb228f1f6e2a3cf65e94f4f04e5) does not match expected (0x48d9b80aa254dad0cbd11c36414eb36af46a595bbad975fefe5462a70eea9bd6) for atomic trie while processing async message:

To Reproduce
Unknown.

Expected behavior
No error during state sync.

`backends.SimulatedBackend` doesn't implement `ethclient.Client` interface

backends.SimulatedBackend doesn't implement ethclient.Client interface

To Reproduce:

var _ ethclient.Client = (*backends.SimulatedBackend)(nil)

Expected behavior
No compiler error.

Remove TriesInMemory constant

While researching how to increase number of snapshots in memory I found that TriesInMemory remained in code after refactoring the GETH logic. Therefore it would make sense to remove unused constant.

Additionally I am also using this issue to find some additional guidance how to increase cached snapshots size.

I found that snapshots are retrieved from stateLayers stored in Tree node. This layers are cleared while calling discard() function which is called from Discard() and Flatten() function. Discard()function is not triggered often, while Flatten() is called regularly when new block is accepted. Therefore I assume I should change some logic in Flatten() function to increase snapshots cached but I don't know what exactly. Here I would appreciate some help what should I do to increase cache size.

Flaky test: TestTransactionIndices

--- FAIL: TestTransactionIndices (1.05s)
blockchain_test.go:652:
Error Trace: /home/runner/work/coreth-internal/coreth-internal/core/blockchain_test.go:652
/home/runner/work/coreth-internal/coreth-internal/core/blockchain_test.go:731
Error: Not equal:
expected: (*uint64)(0xc0085ead88)
actual : (*uint64)(0xc000c49b58)
Test: TestTransactionIndices

observed to fail CI on unrelated PR

Import an asset by assetID

Hi there,
This is an extra feature request. Coreth allows exporting with assetID , however it doesn't allow import with assetID. Normally it imports all of the tokens with Avax exported from X-chain. It would be useful if we can specify which token to import and maybe even export/import functionality are done with a single transaction. But this should be another request.
Thanks !

Blocks not being produced for transactions

I have a repro here https://github.com/AaronKutch/avalanchego_block_bug . Run bash run.sh. What this does is start avalanchego with a genesis file that prefunds a 0xBf... account, and I want to make a transaction to send some of the funds fo a 0xb3 account. The log file has as its last line

INFO [06-29|03:59:07.594] <C Chain> github.com/ava-labs/coreth/internal/ethapi/api.go:1745: Submitted transaction hash=0xc0ea83c5c1b2ba2b2d1ae9d3748b87625044acd4f562ae79bf02f49f77e0a763 from=0xBf660843528035a5A4921534E156a27e64B231fE nonce=0 recipient=0xb3d82b1367D362DE99aB59A658165AFf520CBd4d value=10,000,000,000,000,000,000,000,000 type=0 gasFeeCap=30,000,000,000 gasTipCap=30,000,000,000 gasPrice=30,000,000,000

I expect that a block should be produced almost immediately to include the transaction, but the chain just sits there. After 20 seconds we check the account funds and block number and see that nothing has changed. Strangely, I do get blocks if contracts are submitted.

Deprecate push-based tx gossip

Once pull-based tx gossip is released, we should deprecate the old push-based gossip on the next network upgrade.

Blocked on #274

gas price change from 47 to 470 breaks key singletons

(I was asked by @danlaine to bring this issue, which I noted a few days ago and complained about again today, from Discord to GitHub.)

Apparently, 9 days ago--notably, a day after I was doing all of my Avalanche C-chain compatibility testing (which led to me filing the issue about Istanbul support)--a commit landed to change the gas price from 47 to 470. dd6b031

Without a market mechanism of some kind for pricing (which I think is extremely important for many reasons, per my issue #40), this means that transactions submitted below 470 Gwei are simply prohibited from being deployed. This might seem arbitrary, but it isn't: under what actually feels to me like a reasonable set of assumptions, there are some key ecosystem standards designed with cross-chain contract and tooling compatibility in mind, that assume that somehow, even if only someday, a transaction with a gasprice of 100 Gwei will be able to clear by the system.

These are so called "singleton contracts", and include notably the EIP-1820 Pseudo-Introspection Registry, which is has been in Final status for a long time and is depended on by various other standards, including the ERC-777 token standard some new assets are using--and the EIP-2470 Singleton Factory (which is still in Draft, but is usable).

https://eips.ethereum.org/EIPS/eip-1820

https://eips.ethereum.org/EIPS/eip-2470

They (and other contracts like them) are designed to live at the exact same address on any chain, and do so using what has come to be called "Nick's method" of creating a "forged" transaction (with a random signature) without an embedded EIP-155 chainid. As most of the time there will technically be a key pair that could have signed something with any given signature, this results in a transaction/account that can be used "once" (as there's no way to create a second transaction for the same account, as it would require the private key). You send the account enough ETH for the gas on this transaction, and it executes.

This does, of course, require hardcoding the gas price. It was determined that 100 Gwei was reasonable for this, and in fact kind of ridiculously high. Yes: the current situation on Ethereum has transaction fees occasionally in the range of your 470 Gwei, but not consistently and if you are willing to wait long enough and try hard enough you can get 100 Gwei through even in the worst of times (and, "worst case", you find a mining pool willing to help you by subsidizing the transaction to push the ecosystem forward). Regardless, right now EthGasStation is reporting a price of only 75 Gwei to get something through if you are willing to wait even five minutes.

The reason this is important is that tooling needs to be able to automatically discover functionality deployed on multiple chains, and contracts like the pseudo-introspection registry end up being overhead on other contracts, and so that makes them want their addresses to be hardcoded. The alternative is that every single contract that needs to provide services on multiple chains needs to have some registry somewhere (and like, where? centralized?) that tells you "where" to access it, to prevent issues with multiple people deploying it at different addresses.

I appreciate this one is a bit awkward, as it might at first just feel like an unreasonable assumption to make about gas pricing, but since the premise has always been that prices are set by some kind of market algorithm process (in addition to ways for miners to subsidize gas costs) the selection of 100 Gwei just doesn't seem unreasonable. It is just that without this, your chain becomes an awkward special case that everyone has to track. Software can no longer be written to say "if you trust a full node running the JSON/RPC protocol, point me at it and I will autoconfigure myself, even deploying the contracts I need if they aren't there", as the process will fail and require all related contracts to somehow be configurable for contract addresses (which isn't easy in solidity without resorting to expensive storage slots, and which will break the codehashes of existing trustable contracts, etc.).

Regardless, because I imagine going forward new singletons will be based on EIP-2470 (and we can push for this), another solution to this problem--with respect to the specs, but not necessarily anyone else's projects that were sitting on the side, which is admittedly unfortunate--than messing with your gas costs is to force into deployment by updated software consensus these two contracts. With the existence of EIP-2470 already in place, there is then a way for people to deploy new singletons without having to do the same transaction forgery trick, and so in some sense you only need to special case this "once" (twice, for EIP-1820, which is referenced by various other specifications and predates EIP-2470 and so is its own unrelated singleton).

502 Bad Gateway

Very often when the node is processing a lot of transactions (500+) in a short time (<3 seconds) the node becomes "non-responsible" until restarted.

{
   "reason":"bad response",
   "code":"SERVER_ERROR",
   "status":502,
   "headers":{
      "date":"Wed, 09 Sep 2020 07:46:48 GMT",
      "content-type":"text/html",
      "transfer-encoding":"chunked",
      "connection":"close",
      "cf-cache-status":"DYNAMIC",
      "cf-request-id":"05136ddcdf00001c3f7fb22200000001",
      "expect-ct":"max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"",
      "server":"cloudflare",
      "cf-ray":"5cff4c0e3bd91c3f-SOF"
   },
   "body":"<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n</body>\r\n</html>\r\n",
   "requestBody":"{\"method\":\"eth_getTransactionReceipt\",\"params\":[\"0x5964b37c0be07781731d575756198e139113c797d3ac862e790133c9e4d086e0\"],\"id\":7460,\"jsonrpc\":\"2.0\"}",
   "requestMethod":"POST",
   "url":"https://testapi.avax.network/ext/bc/C/rpc"
}

eth_gasPrice starts correctly but goes to 0

Our gas limit logic fails because eth_gasPrice goes from its initial 225000000000 down to 0 after a block is produced, but the pool gas tip cap stays at 225000000000. The local test logic multiplies the supposed gas price by 2 and gets 0, which in the following transaction results in transaction underpriced: address 0x3Cd0A705a2DC65e5b1E1205896BaA2be8A07c6e0 have gas tip cap (0) \u003c pool gas tip cap (225000000000). I added on a commit to https://github.com/AaronKutch/avalanchego_block_bug to exhibit this, run with bash run.sh. The logs have

INFO [07-07|01:12:34.021] <C Chain> github.com/ava-labs/coreth/internal/ethapi/api.go:1745: Submitted transaction hash=0x1f9e4098f9a4717a9352b401ed1c220d56aba742fba319feeeff5771f49d9504 from=0x3Cd0A705a2DC65e5b1E1205896BaA2be8A07c6e0 nonce=0 recipient=0x798d4Ba9baf0064Ec19eB4F0a1a45785ae9D6DFc value=1 type=0 gasFeeCap=321,000,000,000 gasTipCap=321,000,000,000 gasPrice=321,000,000,000
INFO [07-07|01:12:34.031] <C Chain> github.com/ava-labs/coreth/internal/ethapi/api.go:1745: Submitted transaction hash=0x33fc41add96dba33bfd645c8fb473a649a6767c01437ea48a87e9b2d29277053 from=0x3Cd0A705a2DC65e5b1E1205896BaA2be8A07c6e0 nonce=1 recipient=0x798d4Ba9baf0064Ec19eB4F0a1a45785ae9D6DFc value=1 type=0 gasFeeCap=321,000,000,000 gasTipCap=321,000,000,000 gasPrice=321,000,000,000
INFO [07-07|01:12:37.023] <C Chain> github.com/ava-labs/coreth/miner/worker.go:343: Commit new mining work number=1 hash=3d5189..ef3b83 uncles=0 txs=2 gas=42000 fees=0.013482 elapsed="362.313µs"
[07-07|01:12:37.023] INFO <C Chain> proposervm/pre_fork_block.go:163 built block U1JLTWLesCiRAKd9BL4waz2pmTSDK4eHXE4JtkiGxTBj5Mv2R - parent timestamp 1970-01-01 00:00:00 +0000 UTC
WARN [07-07|01:12:41.042] <C Chain> github.com/ava-labs/coreth/rpc/handler.go:404: Served eth_sendRawTransaction conn=127.0.0.1:33074 reqid=0 execTime="479.554µs" procTime="505.093µs" totalTime="505.433µs" err="transaction underpriced: address 0x3Cd0A705a2DC65e5b1E1205896BaA2be8A07c6e0 have gas tip cap (0) < pool gas tip cap (225000000000)"

and the curl responses are

{"jsonrpc":"2.0","id":0,"result":"0x34630b8a00"} // gas price
{"jsonrpc":"2.0","id":0,"result":"0x1f9e4098f9a4717a9352b401ed1c220d56aba742fba319feeeff5771f49d9504"}
{"jsonrpc":"2.0","id":0,"result":"0x34630b8a00"}
{"jsonrpc":"2.0","id":0,"result":"0x33fc41add96dba33bfd645c8fb473a649a6767c01437ea48a87e9b2d29277053"}
{"jsonrpc":"2.0","id":0,"result":"0x0"} // gas price is now zero after a block has been processed
{"jsonrpc":"2.0","id":0,"error":{"code":-32000,"message":"transaction underpriced: address 0x3Cd0A705a2DC65e5b1E1205896BaA2be8A07c6e0 have gas tip cap (0) \u003c pool gas tip cap (225000000000)"}}

Coreth depends on `go-ethereum` 1.9, which requires downgrading `solc` for Go ABI binding generation

This means that any project which a) wants to generate Go ABI bindings for a Solidity contract and b) imports avalanchego must intentionally downgrade a) their go-ethereum to 1.9 (latest is 1.10) and b) their solc to 0.7 (latest is 0.8).

This is because the Go ABI bindings generated by go-ethereum 1.10 aren't backwards compatible with 1.9, and go-ethereum 1.9 generation isn't forward-compatible with solc 0.8 (only 0.7).

Slow Tx Gossip

Describe the bug
A clear and concise description of what the bug is.

Coreth has as bug where transactions slow down as they are gossiped through the network. Nodes only gossip a tx if they haven't heard of it before, so once a majority of nodes hear of a transaction less and less nodes are willing to gossip the transaction. In the case that we gossip to a node that already knows of a transaction, it "aborbs" the gossip and doesn't forward it. This results in nodes potentially never even hearing about some transactions until a regossip cycle is hit.

We can use a pull-based strategy to make txs propagate exponentially fast instead of exponentially slowly by constantly polling peers for new txs instead of only pushing a tx when we haven't heard of it before.

The metrics of coreth txpool don't work

# TYPE avalanche_C_vm_eth_txpool_known gauge
avalanche_C_vm_eth_txpool_known 0
# TYPE avalanche_C_vm_eth_txpool_local gauge
avalanche_C_vm_eth_txpool_local 0
# TYPE avalanche_C_vm_eth_txpool_overflowed gauge
avalanche_C_vm_eth_txpool_overflowed 0
# TYPE avalanche_C_vm_eth_txpool_pending gauge
avalanche_C_vm_eth_txpool_pending 0
# TYPE avalanche_C_vm_eth_txpool_pending_discard gauge
avalanche_C_vm_eth_txpool_pending_discard 0
# TYPE avalanche_C_vm_eth_txpool_pending_nofunds gauge
avalanche_C_vm_eth_txpool_pending_nofunds 0
# TYPE avalanche_C_vm_eth_txpool_pending_ratelimit gauge
avalanche_C_vm_eth_txpool_pending_ratelimit 0
# TYPE avalanche_C_vm_eth_txpool_pending_replace gauge
avalanche_C_vm_eth_txpool_pending_replace 0

General tx metrics are always zero

RPC Segfault

Got a few sigfaults while on testnet. Running avalanchego 1.1.0 / coreth 0.3.18 (on client).

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x5d4ef0]

goroutine 747615 [running]:
bufio.(*Writer).Available(...)
	/usr/local/go/src/bufio/bufio.go:624
bufio.(*Writer).Write(0x0, 0x1b682b8, 0x2, 0x2, 0x4212, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:634 +0x70
net/http.(*chunkWriter).Write(0xc00400a660, 0xc000a34800, 0x4212, 0x4800, 0x7510c9, 0xc00339aee8, 0x0)
	/usr/local/go/src/net/http/server.go:385 +0x151
bufio.(*Writer).Write(0xc008480c00, 0xc000a34800, 0x4212, 0x4800, 0xc000a30000, 0x4248, 0x4248)
	/usr/local/go/src/bufio/bufio.go:639 +0x13d
net/http.(*response).write(0xc00400a620, 0x4212, 0xc000a34800, 0x4212, 0x4800, 0x0, 0x0, 0x0, 0xc0052e67e0, 0xc00930fa38)
	/usr/local/go/src/net/http/server.go:1607 +0x30a
net/http.(*response).Write(0xc00400a620, 0xc000a34800, 0x4212, 0x4800, 0x415f1b, 0xc00930fac8, 0xaa79ac1e)
	/usr/local/go/src/net/http/server.go:1565 +0x56
github.com/gorilla/handlers.(*responseLogger).Write(0xc008027d00, 0xc000a34800, 0x4212, 0x4800, 0x0, 0x0, 0x0)
	/go/pkg/mod/github.com/gorilla/[email protected]/handlers.go:59 +0x51
github.com/ava-labs/avalanchego/vms/rpcchainvm/ghttp/gresponsewriter.(*Server).Write(0xc00028e360, 0x15e08a0, 0xc00957a570, 0xc0047f29b0, 0xc00028e360, 0xc00957a570, 0xc002610ba0)
	/go/src/github.com/ava-labs/avalanchego/vms/rpcchainvm/ghttp/gresponsewriter/writer_server.go:49 +0x159
github.com/ava-labs/avalanchego/vms/rpcchainvm/ghttp/gresponsewriter/gresponsewriterproto._Writer_Write_Handler(0xedf3c0, 0xc00028e360, 0x15e08a0, 0xc00957a570, 0xc00500f9e0, 0x0, 0x15e08a0, 0xc00957a570, 0xc000a30000, 0x4248)
	/go/src/github.com/ava-labs/avalanchego/vms/rpcchainvm/ghttp/gresponsewriter/gresponsewriterproto/gresponsewriter.pb.go:561 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0018c1a00, 0x15ebac0, 0xc001345980, 0xc003ab0e00, 0xc0050cc960, 0x1ca1760, 0x0, 0x0, 0x0)
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1082 +0x522
google.golang.org/grpc.(*Server).handleStream(0xc0018c1a00, 0x15ebac0, 0xc001345980, 0xc003ab0e00, 0x0)
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1405 +0xcc5
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0016cd2f0, 0xc0018c1a00, 0x15ebac0, 0xc001345980, 0xc003ab0e00)
	/go/pkg/mod/google.golang.org/[email protected]/server.go:746 +0xa5
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/go/pkg/mod/google.golang.org/[email protected]/server.go:744 +0xa5

eth_gasPrice crash

Using the default settings, eth_gasPrice crashes with the following trace

ERROR[12-10|16:40:35.051] RPC method eth_gasPrice crashed: runtime error: invalid memory address or nil pointer dereference
goroutine 139 [running]:
github.com/ava-labs/coreth/rpc.(*callback).call.func1(0xc0015e0790, 0xc, 0xc001565cc0)
        /home/ubuntu/install/avalanche/go/pkg/mod/github.com/ava-labs/[email protected]/rpc/service.go:210 +0xbd
panic(0xf3c840, 0x18b2c00)
        /home/ubuntu/install/go1.16.8/src/runtime/panic.go:965 +0x1b9
math/big.(*Int).Cmp(0xc0015a2300, 0x0, 0xc001636440)
        /home/ubuntu/install/go1.16.8/src/math/big/int.go:328 +0x44
github.com/ethereum/go-ethereum/common/math.BigMin(...)
        /home/ubuntu/install/avalanche/go/pkg/mod/github.com/ethereum/[email protected]/common/math/big.go:148
github.com/ava-labs/coreth/eth/gasprice.(*Oracle).SuggestPrice(0xc00014c6c0, 0x1327c78, 0xc001636440, 0xc0012d58d0, 0x4c6e8b, 0xf6ba80)
        /home/ubuntu/install/avalanche/go/pkg/mod/github.com/ava-labs/[email protected]/eth/gasprice/gasprice.go:224 +0x1d1
github.com/ava-labs/coreth/eth.(*EthAPIBackend).SuggestPrice(0xc001480558, 0x1327c78, 0xc001636440, 0x30, 0x7f11b380d5b8, 0x30)
        /home/ubuntu/install/avalanche/go/pkg/mod/github.com/ava-labs/[email protected]/eth/api_backend.go:357 +0x45
github.com/ava-labs/coreth/internal/ethapi.(*PublicEthereumAPI).GasPrice(0xc0010fe910, 0x1327c78, 0xc001636440, 0x0, 0x0, 0x0)
        /home/ubuntu/install/avalanche/go/pkg/mod/github.com/ava-labs/[email protected]/internal/ethapi/api.go:71 +0x4e
reflect.Value.call(0xc0014fe2a0, 0xc00009f850, 0x13, 0x1082042, 0x4, 0xc001654b70, 0x2, 0x2, 0xc0012d5b58, 0x46a08c, ...)
        /home/ubuntu/install/go1.16.8/src/reflect/value.go:476 +0x8e7
reflect.Value.Call(0xc0014fe2a0, 0xc00009f850, 0x13, 0xc001654b70, 0x2, 0x2, 0x1312100, 0xc000032080, 0xc002420000)
        /home/ubuntu/install/go1.16.8/src/reflect/value.go:337 +0xb9
github.com/ava-labs/coreth/rpc.(*callback).call(0xc0014fe420, 0x1327c78, 0xc001636440, 0xc0015e0790, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/ubuntu/install/avalanche/go/pkg/mod/github.com/ava-labs/[email protected]/rpc/service.go:216 +0x2c5
github.com/ava-labs/coreth/rpc.(*handler).runMethod(0xc00055e640, 0x1327c78, 0xc001636440, 0xc0004f7260, 0xc0014fe420, 0x0, 0x0, 0x0, 0x0)
        /home/ubuntu/install/avalanche/go/pkg/mod/github.com/ava-labs/[email protected]/rpc/handler.go:486 +0x8a
github.com/ava-labs/coreth/rpc.(*handler).handleCall(0xc00055e640, 0xc0015eb680, 0xc0004f7260, 0x1000000426d9b)
        /home/ubuntu/install/avalanche/go/pkg/mod/github.com/ava-labs/[email protected]/rpc/handler.go:434 +0x265
github.com/ava-labs/coreth/rpc.(*handler).handleCallMsg(0xc00055e640, 0xc0015eb680, 0xc0004f7260, 0xc0015eb680)
        /home/ubuntu/install/avalanche/go/pkg/mod/github.com/ava-labs/[email protected]/rpc/handler.go:395 +0x212
github.com/ava-labs/coreth/rpc.(*handler).handleMsg.func1(0xc0015eb680)
        /home/ubuntu/install/avalanche/go/pkg/mod/github.com/ava-labs/[email protected]/rpc/handler.go:167 +0x46
github.com/ava-labs/coreth/rpc.(*handler).startCallProc.func1()
        /home/ubuntu/install/avalanche/go/pkg/mod/github.com/ava-labs/[email protected]/rpc/handler.go:307 +0x1e2
created by github.com/ava-labs/coreth/rpc.(*handler).startCallProc
        /home/ubuntu/install/avalanche/go/pkg/mod/github.com/ava-labs/[email protected]/rpc/handler.go:311 +0xa5

WARN [12-10|16:40:35.051] Served eth_gasPrice                      reqid=1 execTime="165.863µs" procTime="166.486µs" totalTime="166.77µs" err="method handler crashed"

keystore-directory cli arg unavailable

Docs: https://docs.avax.network/nodes/maintain/chain-config-flags#keystore-directory-string

The docs reference a --keystore-directory cli arg that we would be very useful to us, but is currently not part of the available args. This would be a very useful addition if it were added in.

The default dir (/tmp/coreth-keystore*) may not be a secure location to leave open and writeable for the app.

Add Comprehensive RPC and WebSocket Coverage for APIs

This ticket is to add comprehensive RPC and WebSocket API coverage via e2e testing.

We should add unit testing where possible, but expect that setting up a simple e2e test that we can use to test API calls work correctly will significantly improve our test coverage.

Flood may be a good tool to add comprehensive coverage and benchmark performance: https://github.com/paradigmxyz/flood/

How to test coreth locally

Hi, since avalanchego invokes coreth binary directly, so, when run avalanchego and set breakpoints in coreth it's useless. It won't stop.

May I know how you guys debug coreth?

Where exactly in the Avalanche codebase does the gRPC communication happen with EVM?

The C-Chain runs in a separate process from the main AvalancheGo process and communicates with it over a local gRPC connection.

Where exactly in the Avalanche codebase does the gRPC communication happen between Avalanche and EVM?

Issuing invalid transaction stops node from processing messages

Issuing an invalid transaction to a node currently causes that node to put it in a block and then stop being able to process additional messages.

This can be reproduced in avash by starting a local network with scripts/five_node_staking.lua.

1. Create a new account in metamask
2. Create a new keystore user and import the genesis private key on both the X and C chains
3. ExportAVAX from the X chain to the genesis private key address on the C chain
4. ImportAVAX on the C chain to the address created in metamask
5. Use advanced options to specify a transaction with gas below the minimum gas price of 47 gwei

The node that it's issued to should probably reject the transaction outright ideally in a way that allows apps such as metamask to display the error appropriately to users. Instead, the node it's issued to gets to a block height one above the rest of the nodes and then completely stops processing consensus messages.

Node stuck after upgrading to 1.8.0

Describe the bug
After upgrading the node to 1.8.0 from 1.7.16, X and P chains synced regularly, but C-Chain was stuck after fetching and executing missing blocks. Restart after ~10 minutes didn't help, stuck in the same place. Stopped the node, turned off the state sync and then it started recreating the snapshot. It's still going 6+ hrs in, with about 7 hrs to go...

To Reproduce
Not sure. Run 1.7.16 and upgrade to 1.8.0.?

Expected behavior
Expected regular, quick sync after upgrading.

Screenshots
If applicable, add screenshots to help explain your problem.

Logs
logs.tar.gz

Operating System
Ubuntu 20.04

C-Chain hangs forever on shutdown during state sync

Describe the bug

C-Chain never finished shutting down if the node is stopped with CTRL + C during state sync.

To Reproduce

• Run a node
• Wait until state sync starts
• CTRL + C

Expected behavior

Shutdown finishes in < 30 seconds.

Logs

[03-31|09:37:35.096] INFO <C Chain> syncer/state_syncer.go:411 starting state sync
[03-31|09:37:35.099] INFO <X Chain> avm/vm.go:629 fee asset is established {"alias": "AVAX", "assetID": "U8iRqJoiJm8xZHAacmvYyZVwqQx6uDNtQeP3CQ6fcgQk3JqnK"}
[03-31|09:37:35.100] INFO <X Chain> avm/vm.go:275 address transaction indexing is disabled
[03-31|09:37:35.101] INFO chains/manager.go:758 creating proposervm wrapper {"activationTime": "[09-16|21:00:00.000]", "minPChainHeight": 47437, "minBlockDelay": "1s"}
[03-31|09:37:35.102] INFO <X Chain> snowman/transitive.go:89 initializing consensus engine
[03-31|09:37:35.102] INFO <X Chain> avalanche/transitive.go:93 initializing consensus engine
[03-31|09:37:35.102] INFO server/server.go:269 adding route {"url": "/ext/bc/2JVSBoinj9C2J33VntvzYtVJNZdN2NKiwwKjcumHUWEb5DbBrm", "endpoint": ""}
[03-31|09:37:35.103] INFO server/server.go:269 adding route {"url": "/ext/bc/2JVSBoinj9C2J33VntvzYtVJNZdN2NKiwwKjcumHUWEb5DbBrm", "endpoint": "/wallet"}
[03-31|09:37:35.103] INFO server/server.go:269 adding route {"url": "/ext/bc/2JVSBoinj9C2J33VntvzYtVJNZdN2NKiwwKjcumHUWEb5DbBrm", "endpoint": "/events"}
[03-31|09:37:35.103] INFO <X Chain> bootstrap/bootstrapper.go:348 starting bootstrap
[03-31|09:37:50.729] INFO <X Chain> common/bootstrapper.go:244 bootstrapping started syncing {"numVerticesInFrontier": 1}
INFO [03-31|09:37:50.885] <C Chain> github.com/ava-labs/coreth/plugin/evm/syncervm_client.go:208: Starting state sync summary="SyncSummary(BlockHash=0x5377ae334a0938fb39c8eb93e416db9ed8e75c1fcc014ffbf69e13a233c514b8, BlockNumber=16433152, BlockRoot=0x225cc30aab5547e10546efc68457708e8422f8ef34bbcea0ae8d64b2cc6264e7, AtomicRoot=0x818b3c401b5011fac8f4209fb9881f8828c09d2fda9ad3ba6a494c7015a30cfc)"
[03-31|09:37:50.886] INFO <C Chain> syncer/state_syncer.go:312 accepted state summary {"summaryID": "gcH1hXojppjqtbvqZK76JJjEYxGDCaT6VGf7etiEUgGro75gg", "syncMode": "Static", "numTotalSummaries": 2}
INFO [03-31|09:37:50.911] <C Chain> github.com/ava-labs/coreth/plugin/evm/syncervm_client.go:279: fetched blocks from peer total=0
INFO [03-31|09:37:50.911] <C Chain> github.com/ava-labs/coreth/plugin/evm/syncervm_client.go:298: state sync: sync starting root=225cc3..6264e7
[03-31|09:37:51.163] INFO <X Chain> bootstrap/bootstrapper.go:578 executing transactions
[03-31|09:37:51.267] INFO <X Chain> queue/jobs.go:224 executed operations {"numExecuted": 236}
[03-31|09:37:51.268] INFO <X Chain> bootstrap/bootstrapper.go:595 executing vertices
[03-31|09:37:51.288] INFO <X Chain> queue/jobs.go:224 executed operations {"numExecuted": 236}
INFO [03-31|09:39:02.635] <C Chain> github.com/ava-labs/coreth/sync/statesync/trie_sync_stats.go:135: state sync: syncing storage tries triesRemaining=496,318 ETA=2h0m
INFO [03-31|09:40:03.087] <C Chain> github.com/ava-labs/coreth/sync/statesync/trie_sync_stats.go:135: state sync: syncing storage tries triesRemaining=490,382 ETA=1h24m
INFO [03-31|09:41:03.113] <C Chain> github.com/ava-labs/coreth/sync/statesync/trie_sync_stats.go:135: state sync: syncing storage tries triesRemaining=486,146 ETA=1h36m
INFO [03-31|09:42:05.021] <C Chain> github.com/ava-labs/coreth/sync/statesync/trie_sync_stats.go:135: state sync: syncing storage tries triesRemaining=481,779 ETA=1h41m
INFO [03-31|09:43:05.143] <C Chain> github.com/ava-labs/coreth/sync/statesync/trie_sync_stats.go:135: state sync: syncing storage tries triesRemaining=477,995 ETA=1h45m
INFO [03-31|09:44:05.147] <C Chain> github.com/ava-labs/coreth/sync/statesync/trie_sync_stats.go:135: state sync: syncing storage tries triesRemaining=474,565 ETA=1h55m
INFO [03-31|09:45:05.924] <C Chain> github.com/ava-labs/coreth/sync/statesync/trie_sync_stats.go:135: state sync: syncing storage tries triesRemaining=471,392 ETA=1h55m
INFO [03-31|09:46:10.980] <C Chain> github.com/ava-labs/coreth/sync/statesync/trie_sync_stats.go:135: state sync: syncing storage tries triesRemaining=467,376 ETA=1h55m
INFO [03-31|09:47:11.211] <C Chain> github.com/ava-labs/coreth/sync/statesync/trie_sync_stats.go:135: state sync: syncing storage tries triesRemaining=462,154 ETA=1h50m
INFO [03-31|09:48:11.960] <C Chain> github.com/ava-labs/coreth/sync/statesync/trie_sync_stats.go:135: state sync: syncing storage tries triesRemaining=458,063 ETA=2h1m
^C[03-31|09:49:10.804] INFO node/node.go:1385 shutting down node {"exitCode": 0}
[03-31|09:49:10.804] INFO ipcs/chainipc.go:123 shutting down chain IPCs
[03-31|09:49:10.804] INFO chains/manager.go:1398 shutting down chain manager
[03-31|09:49:10.804] INFO chains/manager.go:1391 stopping chain creator
[03-31|09:49:10.804] INFO router/chain_router.go:361 shutting down chain router
[03-31|09:49:10.805] INFO <P Chain> bootstrap/bootstrapper.go:295 shutting down bootstrapper
[03-31|09:49:10.805] INFO <C Chain> syncer/state_syncer.go:570 shutting down state syncer
[03-31|09:49:10.805] INFO <X Chain> bootstrap/bootstrapper.go:339 shutting down bootstrapper
[03-31|09:49:40.805] WARN router/chain_router.go:378 timed out while shutting down the chains
[03-31|09:49:40.805] INFO network/network.go:1280 shutting down the p2p networking
[03-31|09:49:40.809] INFO node/node.go:1438 cleaning up plugin runtimes
WARN [03-31|09:52:35.074] <C Chain> github.com/ava-labs/coreth/core/rawdb/accessors_metadata.go:157: Error reading unclean shutdown markers error=closed
WARN [03-31|09:52:35.074] <C Chain> github.com/ava-labs/coreth/core/rawdb/accessors_metadata.go:164: No unclean shutdown marker to update 
WARN [03-31|09:57:35.074] <C Chain> github.com/ava-labs/coreth/core/rawdb/accessors_metadata.go:157: Error reading unclean shutdown markers error=closed
WARN [03-31|09:57:35.074] <C Chain> github.com/ava-labs/coreth/core/rawdb/accessors_metadata.go:164: No unclean shutdown marker to update 
WARN [03-31|10:02:35.074] <C Chain> github.com/ava-labs/coreth/core/rawdb/accessors_metadata.go:157: Error reading unclean shutdown markers error=closed
WARN [03-31|10:02:35.074] <C Chain> github.com/ava-labs/coreth/core/rawdb/accessors_metadata.go:164: No unclean shutdown marker to update 
WARN [03-31|10:07:35.074] <C Chain> github.com/ava-labs/coreth/core/rawdb/accessors_metadata.go:157: Error reading unclean shutdown markers error=closed
WARN [03-31|10:07:35.074] <C Chain> github.com/ava-labs/coreth/core/rawdb/accessors_metadata.go:164: No unclean shutdown marker to update 
WARN [03-31|10:12:35.074] <C Chain> github.com/ava-labs/coreth/core/rawdb/accessors_metadata.go:157: Error reading unclean shutdown markers error=closed
WARN [03-31|10:12:35.074] <C Chain> github.com/ava-labs/coreth/core/rawdb/accessors_metadata.go:164: No unclean shutdown marker to update 
WARN [03-31|10:17:35.074] <C Chain> github.com/ava-labs/coreth/core/rawdb/accessors_metadata.go:157: Error reading unclean shutdown markers error=closed
WARN [03-31|10:17:35.074] <C Chain> github.com/ava-labs/coreth/core/rawdb/accessors_metadata.go:164: No unclean shutdown marker to update 

Operating System
Ubuntu 20.04

error="chain reorged during section processing"

Approximately 19 hours ago 3 of my testing nodes stopped working as expectedly and the public endpoint: https://testapi.avax.network/ext/bc/C/rpc which serves https://faucet.avax.network/ and is used to connect MetaMask to the C-chain showed similar behavior.

In the logs of my 3 nodes I found the following error:
Note: This appears to be a known go-ethereum issue which was fixed in geth 1.9.0 ethereum/go-ethereum#17227

ERROR[09-16|00:23:35.398] Section processing failed                type=bloombits error="chain reorged during section processing"
DEBUG[09-16|00:23:35.398] Chain index processing failed            type=bloombits section=25 err="chain reorged during section processing"
DEBUG[09-16|00:23:37.111] Persisted trie from memory database      nodes=23 size=8.19KiB  time=1.479891ms  gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
DEBUG[09-16|00:23:37.113] Inserted new block                       number=285481 hash=11a900…a99753 uncles=0 txs=3 gas=63000  elapsed=5.629ms   root=a31308…54e97d
INFO [09-16|00:23:37.113] Imported new chain segment               blocks=1 txs=3 mgas=0.063 elapsed=7.517ms   mgasps=8.380  number=285481 hash=11a900…a99753 dirty=0.00B
DEBUG[09-16|00:23:37.113] Reinjecting stale transactions           count=0
TRACE[09-16|00:23:37.114] Processing new chain section             type=bloombits section=25
DEBUG[09-16|00:23:37.123] Persisted trie from memory database      nodes=29 size=10.18KiB time=1.879889ms  gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
DEBUG[09-16|00:23:37.126] Inserted new block                       number=285482 hash=eb78ea…5f4142 uncles=0 txs=4 gas=84000  elapsed=8.520ms   root=8697e1…e49609
INFO [09-16|00:23:37.126] Imported new chain segment               blocks=1 txs=4 mgas=0.084 elapsed=10.446ms  mgasps=8.041  number=285482 hash=eb78ea…5f4142 dirty=0.00B
DEBUG[09-16|00:23:37.127] Reinjecting stale transactions           count=0
ERROR[09-16|00:23:37.800] Section processing failed                type=bloombits error="chain reorged during section processing"
DEBUG[09-16|00:23:37.800] Chain index processing failed            type=bloombits section=25 err="chain reorged during section processing"
DEBUG[09-16|00:23:38.206] Persisted trie from memory database      nodes=23 size=8.88KiB  time=2.39733ms   gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
DEBUG[09-16|00:23:38.208] Inserted new block                       number=285483 hash=e1a104…b61df9 uncles=0 txs=3 gas=63000  elapsed=7.920ms   root=1c8ba5…cdf04f
INFO [09-16|00:23:38.208] Imported new chain segment               blocks=1 txs=3 mgas=0.063 elapsed=9.771ms   mgasps=6.447  number=285483 hash=e1a104…b61df9 dirty=0.00B
DEBUG[09-16|00:23:38.208] Reinjecting stale transactions           count=0
TRACE[09-16|00:23:38.209] Processing new chain section             type=bloombits section=25
ERROR[09-16|00:23:38.892] Section processing failed                type=bloombits error="chain reorged during section processing"
DEBUG[09-16|00:23:38.892] Chain index processing failed            type=bloombits section=25 err="chain reorged during section processing"
DEBUG[09-16|00:23:39.801] Persisted trie from memory database      nodes=20 size=7.29KiB  time=1.340306ms  gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
DEBUG[09-16|00:23:39.803] Inserted new block                       number=285484 hash=a666f6…e8f6ac uncles=0 txs=2 gas=42000  elapsed=5.382ms   root=85aab9…b98c14
INFO [09-16|00:23:39.803] Imported new chain segment               blocks=1 txs=2 mgas=0.042 elapsed=6.332ms   mgasps=6.632  number=285484 hash=a666f6…e8f6ac dirty=0.00B
DEBUG[09-16|00:23:39.803] Reinjecting stale transactions           count=0
TRACE[09-16|00:23:39.805] Processing new chain section             type=bloombits section=25
DEBUG[09-16|00:23:40.064] Persisted trie from memory database      nodes=28 size=10.41KiB time=18.860793ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
DEBUG[09-16|00:23:40.068] Inserted new block                       number=285485 hash=725834…bb8165 uncles=0 txs=4 gas=84000  elapsed=28.153ms  root=863dff…e81cdd
INFO [09-16|00:23:40.068] Imported new chain segment               blocks=1 txs=4 mgas=0.084 elapsed=41.395ms  mgasps=2.029  number=285485 hash=725834…bb8165 dirty=0.00B
DEBUG[09-16|00:23:40.069] Reinjecting stale transactions           count=0
ERROR[09-16|00:23:40.623] Section processing failed                type=bloombits error="chain reorged during section processing"
DEBUG[09-16|00:23:40.623] Chain index processing failed            type=bloombits section=25 err="chain reorged during section processing"
DEBUG[09-16|00:23:43.434] Persisted trie from memory database      nodes=23 size=8.66KiB  time=1.981428ms  gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
DEBUG[09-16|00:23:43.437] Inserted new block                       number=285486 hash=12d80f…de81fb uncles=0 txs=3 gas=63000  elapsed=6.924ms   root=7867a5…de7601
INFO [09-16|00:23:43.437] Imported new chain segment               blocks=1 txs=3 mgas=0.063 elapsed=9.503ms   mgasps=6.629  number=285486 hash=12d80f…de81fb dirty=0.00B`

snow.Context logging does not show up from coreth

Using the Context logger from within the coreth plugin does not show up in the log files.

Expected behavior: the logger should print output to stdout as well as write it to log files.

We need either a fix for the context logger or to switch to exclusively using eth logging within the coreth plugin to ensure log messages appear as expected.

How Do I Interact with ERC20 Token Programmatically

I'd like to get token info based on the address just like what I do in the ethereum:

// NewIERC20 creates a new instance of IERC20, bound to a specific deployed contract.
func NewIERC20(address common.Address, backend bind.ContractBackend) (*IERC20, error) {
	contract, err := bindIERC20(address, backend, backend, backend)
	if err != nil {
		return nil, err
	}
	return &IERC20{IERC20Caller: IERC20Caller{contract: contract}, IERC20Transactor: IERC20Transactor{contract: contract}, IERC20Filterer: IERC20Filterer{contract: contract}}, nil
}

But the avalanche ethclient is not compatible with it and missing some methods such as PendingCodeAt(ctx context.Context, account common.Address) ([]byte, error) to be implemented.

License violation in LICENSE file

Since the project was forked from geth, it must preserve previous copyright holders. Otherwise, you're violating LGPL and can easily be served with a lawsuit.

Refactor Block Gen Timer

Refactor block gen timer to be more easily understandable

Transactions stuck in pool

Sometimes transactions get stuck in the transaction pool most likely due to them being placed into a block by the local node and then a block produced by a different node is selected at that height.

This seems to be a transaction pool management issue. Some potentially helpful leads on tx pool issues from another project are here: https://github.com/nextyio/gonex/issues/65

Unclean shutdown may lead to offline pruning to be incompatible with state sync setting.

Describe the bug
In case the node is shut down without writing the snapshot fully, offline pruning becomes incompatible with the delayed snapshot initialization. This theory should be confirmed and the initialization sequence should be fixed.

To Reproduce
Several users have reported this on discord with a workaround: https://discord.com/channels/578992315641626624/1113120800975888444/1113120800975888444
https://discord.com/channels/578992315641626624/757576823570825316/1115070900413419521

Expected behavior
Offline pruning should function without any workarounds or unrelated configuration changes.

Screenshots
If applicable, add screenshots to help explain your problem.

Logs
If applicable, please include the relevant logs that indicate a problem and/or the log directory of your node. By default, this can be found at ~/.avalanchego/logs/.

Metrics
N/A

Operating System
N/A

Additional context
Add any other context about the problem here.

Avalanche Bug Bounty program can be found here.

`no preimage found for hash` error when making `debug_getModifiedAccountsByNumber` call

Issue

I am unable to call debug_getModifiedAccountsByNumber. The response is always

{
    "jsonrpc": "2.0",
    "id": 1,
    "error": {
        "code": -32000,
        "message": "no preimage found for hash 5a9331cb1d584bc0703e5252f4fccf19b3887d5a12e7edad074d585640a7ff57"
    }
}

Likely related to commit 5095d5b

+       Preimages bool // Whether to store preimage of trie key to the disk

Version

# avalanchego --version
avalanche/1.4.12 [database=v1.4.5, commit=cae93d95c1bcdc02e1370d38ed1c9d87f1c8c814]

Steps to Reproduce

See: https://github.com/stevekunk/avalanche-testnet#issue-no-preimage-found-for-hash

You should be able to docker-compose up and then ./curls/debug_getModifiedAccountsByNumber.sh to reproduce the issue

Testing swap X chain to C chain not work always getting insufficient AVAX burned (0) to cover import tx fee (3016350)

Hi i'm trying to make avax work in local but i dont know how to transfer from X chain to C Chain

Thanks for your help

EADDRINUSE

Basically i was spamming the network with transactions and a few hundred of them were processed successfully but then suddenly that error start pop up and i had to restart the node. (edited)
It's a random error, I'm having 2 more machines next to me running without any issues for couple hours already (edited)

9/15/2020, 10:35:36 AM error: Send transaction error : {"reason":"missing response","code":"SERVER_ERROR","requestBody":"{\"method\":\"eth_getTransactionReceipt\",\"params\":[\"0x7fcef903203585d6e3386539ab083724b1177acc7bd77366d5aa36b9770f5a87\"],\"id\":34151,\"jsonrpc\":\"2.0\"}","requestMethod":"POST","serverError":{"errno":"EADDRINUSE","code":"EADDRINUSE","syscall":"connect","address":"127.0.0.1","port":9650},"url":"http://127.0.0.1:9650/ext/bc/C/rpc"}

coreth 0.11.3 uses avalanchego 1.9.3-rc.1

When building avalanchego 1.9.3 i noticed that it fetches coreth 0.11.3.rc-1, not 0.11.3.

Both coreth 0.11.3 and 0.11.3.rc-1 in turn use avalanchego 1.9.3.rc-1 to be built.

Isn't this a problem?