ethereum/tests

Common tests for all Ethereum implementations

Ethereum Execution Tests (EVM)

Common tests for all clients to test against. Test execution tool: https://github.com/ethereum/retesteth

Test Formats

Maintained tests:

/BasicTests
/BlockchainTests
/GeneralStateTests
/TransactionTests
/EIPTests
/EOFTests
/RLPTest
/src

See descriptions of the different test formats in the official documentation at http://ethereum-tests.readthedocs.io/.

Note:
The format of BlockchainTests recently changed with the introduction of a new field sealEngine (values: NoProof | Ethash), see related JSON Schema change or BlockchainTest format docs for reference.

This means that you can skip PoW validation for NoProof tests but also has the consequence that it is not possible to rely on/check PoW related block parameters for these tests any more.

Clients using the library

The following clients make use of the tests from this library. You can use these implementations for inspiration on how to integrate. If your client is missing, please submit a PR (requirement: at least some minimal test documentation)!

• Mana (Elixir): Docs, Test location: ethereum_common_tests
• go-ethereum (Go): Docs, Test location: tests/testdata
• Parity Ethereum (Rust): Docs, Test location: ethcore/res/ethereum/tests
• ethereumjs-vm (JavaScript): Docs, Test location: ethereumjs-testing dependency
• Trinity (Python): Docs, Test location: fixtures
• Hyperledger Besu (Java): Docs, Test Location: ethereum/referencetests/src/test/resources
• Nethermind (C#) Docs, Test Location: src/tests
• Nimbus-eth1 (Nim) Docs, Test location: tests/fixtures
• Mantis (Scala) Docs, Test Location: https://github.com/input-output-hk/mantis/tree/develop/ets

Using the Tests

We do versioned tag releases for tests and you should aim to run your client libraries against the latest repository snapshot tagged.

Generally the develop branch in ethereum/tests is always meant to be stable and you should be able to run your test against.

Contribute to the Test Suite

See the dedicated section in the docs on how to write new tests. Or read a shorter description here.

If you want to follow up with current tasks and what is currently in the works, have a look at the issues

Currently the geth evm t8ntool client is the reference client for generating tests. Besu client also has support for generating the tests using rpc test protocol. See at https://github.com/ethereum/retesteth/wiki

Testing stats

Testing results are available at http://retesteth.ethdevops.io/
There is a web tool for vmtracing the tests using supported clients and retesteth: http://retesteth.ethdevops.io/web/
All blockchain tests are being run by hive tool: https://hivetests.ethdevops.io/

Contents of this repository

Do not change test files in folders:

• StateTests
• BlockchainTests
• TransactionTests
• VMTests

It is being created by the testFillers which could be found at src folder. The filler specification and wiki are in development so please ask on gitter channel for more details.

If you want to modify a test filler or add a new test please contact @wdimitry at telegram Use the following guide: https://github.com/ethereum/retesteth/wiki/Creating-a-State-Test-with-retesteth

tests's Issues

Delete "Homestead" branch to avoid confusion

Callecrecover2 and C++ tracing

AZ version 6fa0cbf

http://pastebin.com/XwFbXpeP is the C++ trace

here is snippet
*** 095e7baea6a6c7c4c2dfeb977efac326af552d87: 20100000 (+100000)

• 0000000000000000000000000000000000000000000000000000000000000000: E5266519F86DBF1BAC6021C6BA9711B43AC8561C
  

#1: The test json for post of 095e7 should have a value stored at location 0 ?
#2: I am not sure if E5266519F86DBF1BAC6021C6BA9711B43AC8561C is the correct value that should be stored. In JS the value we have is zero, so that deletes what is stored in location 0 and our test passes.

Maybe the tracing is what's wrong?

mixed uppercase and lowercase hex strings

Some of the hex strings are uppercase and some are lowercase. It would be nice if it was consistent one way or the other.

invalid tx tests

some of the tx tests are invalid
Here the signatures are the wrong length, so there is no way for this to be a valid tx.

Also RLPElementsWithZeros isn't working out for me. Would love it if some one could double check that

consistent formating

https://github.com/ethereum/tests/blob/develop/BlockchainTests/bcForkBlockTest.json#L9-L24
i give you hex strings! decimal strings! hex strings prefixed with 0x! hex strings with out prefixes! welcome to the format zoo!

Add gas_used to to txtest.json

Invalid difficulty in blockchain tests

the difficulty looks wrong here https://github.com/ethereum/tests/blob/develop/BlockchainTests/bcBlockGasLimitTest.json#L8
since its the same as the genesis; here
https://github.com/ethereum/tests/blob/develop/BlockchainTests/bcBlockGasLimitTest.json#L110

Code is defined in both pre and exec

For many of the tests, I found code to be present in the pre-definition of the contract and in the exec-part of the test. This seems to be redundant. Is there a reason for this?

Incorrect expected gas remaining values

Some tests were made under the assumption that a stack-underflow still costs gas.
This results in incorrect post-values.

The following tests have been identified so far

vmIOandFlowOperationsTest.json

pop1         expected: [9999], actual: [10000]

vmPushDupSwapTest.json

swap2error   expected: [9997], actual: [9998]
dup2error    expected: [9998], actual: [9999]

201410211657.json

expected: [9974], actual: [9976]

201410211701.json

expected: [9999], actual: [10000]

201410211704.json

expected: [9999], actual: [10000]

201410211707.json

expected: [9999], actual: [10000]

201410211709.json

expected: [9999], actual: [10000]

201410211717.json

empty file

201410211719.json

expected: [9999], actual: [10000]

201410211722.json

expected: [9999], actual: [10000]

CODECOPY before, inside and after CALL/ *CALLCODE* /CREATE

need some tests for that

ADDMOD - another question

Still missing something:

https://github.com/ethereum/tests/blob/develop/VMTests/vmArithmeticTest.json#L269

testing for: addmod(-1, -2, 2)

according to this µ[0] mod µ[1] = sgn(µ[0]) |µ[0]| mod |µ[1]| the result should be: -1
but the expectation is for 1 😕

Please advice
Thanks in advance.

5 mod (-3) == 2 ?

in the test it check for 5 ?

https://github.com/ethereum/tests/blob/develop/VMTests/vmArithmeticTest.json#L479

last block hash is not present in any of blocks

https://github.com/ethereum/tests/blob/develop/BlockchainTests/basicBlockChain.json

"lastBlock": "9d0a03fd264306a8ccf624bbd52430c18016dd734f982f8e15a94e27c6a252d5",

that hash isn't any of existing blocks, am I missing anything ?

Eliptic curve param validations

Guys ,

Can you please add testing for that one:

ethereum/ethereumj#353

old rlp in genesishashes test

push32error isn't really having any error

https://github.com/ethereum/tests/blob/develop/VMTests/vmPushDupSwapTest.json#L1892

push32error is a good test, testing PUSH and SUICIDE but there is no error.

it seems 2 things may be needed:

1. rename this test, eg pushAndSuicide
2. add an error test, like what push32error may have originally intended

some SIGNEXTEND tests appear to need updating

Some SIGNEXTEND tests don't appear to have been updated: missing SIGNEXTEND
eg
signextendInvalidByteNumber
code is 0x62126af4605016600057 which AZ says is:
PUSH3 0x12 0x6a 0xf4 PUSH1 0x50 AND PUSH1 0x0 JUMPI

Found via ethereumjs/ethereumjs-lib#32

vmEnvironmentalInfoTest.ExtCodeSizeAddressInputTooBigRightMyAddress

https://github.com/ethereum/tests/blob/develop/VMTests/vmEnvironmentalInfoTest.json#L80

That test emitates code approach
by a wrong address, the result is
creation of that address, is it the
correct way?

572e5295c57f15886f9b263e2f6d2d6c7b5ec6aa

does cpp do it in the live state run ?

invalid JSON

this is invalid

tests/BasicTests/RandomRLPTests/example.json

Line 5 in 65b7106

},

and this is invalid

tests/BasicTests/invalidRLPtest.json

Line 10 in 65b7106

},

refund_CallA

In the trace for this test, it seems there is no output relating to how the ending balances ended up in that state:

Java

 06:02:27.126 [VM]  
 Spent Gas: [830]/[1500]
   Left Gas:  [670]

Cpp

EVM [ 07:03:08 | main | 0 | 095e7baea6a6c7c4c2dfeb977efac326af552d87 | #10 |
 0026 : STOP | 670 | -0 | 0x32 ]

And yet, even with the spent gas at 830, the testcase says:

Post:

  "2adc25665018aa1fe0e6bc666dac8fc2697ff9ba" : {
  "balance" : "730",
     "code" : "0x",
     "nonce" : "0",
     "storage" : {
    }

Is this right?

JSON can't have two keys with the same name in a object

This won't work
https://github.com/ethereum/tests/blob/develop/trietest.json#L28
https://github.com/ethereum/tests/blob/develop/trietest.json#L32

Maybe go back to using arrays?

can't parse odd number of characters as byte representation

[2 issues]
https://github.com/ethereum/tests/blob/develop/StateTests/stCallCreateCallCodeTest.json#L229
https://github.com/ethereum/tests/blob/develop/StateTests/stCallCreateCallCodeTest.json#L392

0xfffffffffffffffffffffffffffffff (31) => 0xffffffffffffffffffffffffffffffff (32)

Invalid Total difficultly test

https://github.com/ethereum/tests/blob/develop/BlockchainTests/bcTotalDifficultyTest.json#L2836-L2926

The last two block have the same difficulty but the first one according to the test should be head at the end. I believe the Yellow Paper doesn't specify which should head when blocks have the same TD.

OOG in create and the contract is still created

As one of our recent discussion I
understand that OOG in create
will rollback the contract creation,
however the https://github.com/ethereum/tests/blob/develop/StateTests/stInitCodeTest.json
OutOfGasContractCreation do exactly that.

trace: https://gist.github.com/romanman/8e42898703fa4274782b

@CJentzsch
Please advise.
Roman

modulus of negative

(-5) mod 3 = 1

need to fix that:

https://github.com/ethereum/tests/blob/develop/VMTests/vmArithmeticTest.json#L345

nonce not increased on contract halt

the context is NotEnoughCashContractCreation in this test suite:
https://github.com/ethereum/tests/blob/develop/StateTests/stInitCodeTest.json

a94f5374fce5edbc8e2a8697c15331677e6ebf0b - as I understand the nonce
should be increased, even if the execution of CREATE was halted.

@CJentzsch
Please advice.

stSystemOperationsTest.json createNameRegistratorValueTooHigh -- problem

I think that exception should not happen.

root@ip-10-167-187-237:~/cpp-poc-7/cpp-ethereum/build/test# ./testeth --log_level=test_suite --run_test=StateTests/userDefinedFileState --statetest /root/cpp-poc-7/tests/StateTests/stSystemOperationsTest.json createNameRegistratorValueTooHigh
Running 1 test case...
Entering test suite "EthereumTests"
Entering test suite "StateTests"
Entering test case "userDefinedFileState"
*** [ 08:18:49 | main ] Testing user defined test: /root/cpp-poc-7/tests/StateTests/stSystemOperationsTest.json
createNameRegistratorValueTooHigh
*** [ 08:18:50 | main ] state execution did throw an exception: /root/cpp-poc-7/cpp-ethereum/libethereum/Executive.cpp(77): Throw in function bool dev::eth::Executive::setup(dev::bytesConstRef)
Dynamic exception type: boost::exception_detail::clone_impl<dev::eth::NotEnoughCash>
std::exception::what: std::exception
[dev::tag_required*] = 110000
[dev::tag_got*] = 1000

Leaving test case "userDefinedFileState"; testing time: 292969mks
Leaving test suite "StateTests"
Leaving test suite "EthereumTests"

F2 - interpreted as RETURN

the context is TransactionContractCreation in this test suite:
https://github.com/ethereum/tests/blob/develop/StateTests/stInitCodeTest.json

here is the log of the execution:
https://gist.github.com/romanman/bdfb109eb56a2cb456d7

the last op in the run is PUSH1 and than to finish the run it looks like
RETURN been called, but if we look closely on the transaction
data 0x600a80600c6000396000 [f2] 00600160008035811a8100 the
F2 should be called and that F2 is not a RETURN anymore since
last change. Well still can be the case that I am missing
something, but will be happy for help anyway @CJentzsch

P.S. also TransactionCreateSuicideContract has F2 as expected RETURN

Thanks
Roman

Merge develop into master for PoC 6 conservation

The tests in develop are valid for PoC 6.
I want to commit new tests for PoC 7. My suggestion would be to merge the develop branch with master, and commit PoC 7 tests to develop. Alternatively we can have a PoC 6 and PoC 7 branch.

Gas for RETURN

In StateTests/stSystemOperationsTest.json : CallToReturn1
The gas charged for RETURN is 1
when should be 2*5 .

here is the trace:

https://gist.github.com/anonymous/4432cd54d90b3cc8f667#file-gistfile1-txt-L282

Am I missing something ?

poc-9 genesis

https://github.com/ethereum/tests/blob/develop/BasicTests/genesishashestest.json

is this genesis for this premine ?

https://github.com/ethereum/cpp-ethereum/blob/f8c06e92f7ea5d30074274580119d72ac35af5ee/libethereum/GenesisInfo.cpp

@CJentzsch

numbers should be strings

Incorrect JSON: BasicTests/difficulty*.json

Guys, could you please fix the JSON files so parsers don't fail on them:
BasicTests/difficultyXXX.json has an extra comma at the end of the testcases list.

Transaction; Can the Address field be less than 20?

How can this be valid? https://github.com/ethereum/tests/blob/develop/TransactionTests/ttTransactionTest.json#L12

I'm pretty sure that the yellow paper defines addresses as 160bits

bad trie test

this can't be right. The test before it has the same hash

invalid padding on transactions

In this test https://github.com/ethereum/tests/blob/develop/TransactionTests/ttTransactionTest.json#L422
There is padding on the end of R value. I believe that you are not suppose to produce padding on TX fields.

[vmArithmeticTest exp1] gas calculation problem

so im having a problem a with gas calculation on the exp1 test. From the yellow paper gas cost is
Gexp + Gexpbyte * (1 + floor(log256(exponent)));
The exponent in the exp1 test is fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffe. With Gexp and Gexpbyte set to 10 the cost works out to be 340. CPP is giving 330. So I wonder if there is a rounding error somewhere. log256(exponent) === 32 for me. Is this the same for CPP?

also related is my stack trace https://gist.github.com/wanderer/fd3a6922bf5588d5722e

Balance Overflows

should this be 115792089237316195423570985008687907853269984665640564039457584007913129640035 ?
I See that the balance is overflowing but is this correct behavior?

pre-compiled contract doesn't get endowment

0000..0004 is called with tx.value = 19 but no such account with +19 in post
https://github.com/ethereum/tests/blob/develop/StateTests/stPreCompiledContracts.json#L1083

@CJentzsch

statetest: nonce maybe wrong

this nonce maybe wrong https://github.com/ethereum/tests/blob/develop/StateTests/stSolidityTest.json#L226

I think it should be 2. The nonce gets incremented once for sending the transaction and once again for the creation of 2adc25665018aa1fe0e6bc666dac8fc2697ff9ba

`allotment` is not described on the wiki

https://github.com/ethereum/tests/wiki/Blockchain-Tests

allotment data is not described on the wiki ,
is it pre or post values ?

Some incorrect post-repo values

Issue with post repository on these tests:

bcInvalidHeaderTest.json
wrongNumber.
wrongDifficulty
wrongGasLimit
wrongParentHash
log1_wrongBlockNumber

bcUncleTest.json
threeUncle

logging from these is very similar:

        "postState" : {
            "095e7baea6a6c7c4c2dfeb977efac326af552d87" : {
                "balance" : "5000000100",
                "code" : "0x7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff600052600060206000a1",
                "nonce" : "0",
                "storage" : {
                }
            },
            "8888f1f195afa192cfee860698584c030f4c9db1" : {
                "balance" : "1500000000000220270",
                "code" : "0x",
                "nonce" : "0",
                "storage" : {
                }
            },
            "a94f5374fce5edbc8e2a8697c15331677e6ebf0b" : {
                "balance" : "4999779730",
                "code" : "0x",
                "nonce" : "1",
                "storage" : {
                }
            }
        },

versus what we're seeing in Java and Cpp

   current: 2
05:40:54.128 [DEBUG] [TestEventLogger]     05:40:54.081 [TCK-Test]  Account: 095e7baea6a6c7c4c2dfeb977efac326af552d87: has unexpected balance, expected balance: 5000000100 found balance: 100
05:40:54.129 [DEBUG] [TestEventLogger]     05:40:54.081 [TCK-Test]  Account: a94f5374fce5edbc8e2a8697c15331677e6ebf0b: has unexpected balance, expected balance: 4999779730 found balance: 10000000000
05:40:54.129 [DEBUG] [TestEventLogger]     05:40:54.083 [TCK-Test]  Account: a94f5374fce5edbc8e2a8697c15331677e6ebf0b: has unexpected nonce, expected nonce: 1 found nonce: 0
05:40:54.130 [DEBUG] [TestEventLogger]     05:40:54.083 [TCK-Test]  Account: 8888f1f195afa192cfee860698584c030f4c9db1: expected but doesn't exist
05:40:54.131 [DEBUG] [TestEventLogger]     05:40:54.083 [TCK-Test]  Root hash don't much: expected: f93c8db1e931daa2e22e39b5d2da6fb4074e3d544094857608536155e3521bc1 current: 925002c3260b44e44c3edebad1cc442142b03020209df1ab8bb86752edbd2cd7

Block numbers in the Homestead tests are less than 1150000

Block numbers in the Homestead difficulty tests are less than 1150000
https://github.com/ethereum/tests/blob/develop/BasicTests/difficultyHomestead.json

LICENSE?

MIT makes most sense

RPC Test Suite

@wanderer and I were thinking that there should be generic RPC tests for the ethereum clients.

have problem to run VMTests suite in cli

./testeth --log_level=test_suite --run_test=VMTests/userDefinedFileState --statetest /home/tests/VMTests/vmSystemOperationsTest.json CallToNameRegistratorNotMuchMemory0

also tried

./testeth --log_level=test_suite --run_test=VMTests/userDefinedFileState --vmtest /home/tests/VMTests/vmSystemOperationsTest.json  ABAcallsSuicide0

is it a valid syntax ?

VMTest => createNameRegistrator

2 questions:

• Why do you omit the address of a created contract:[945304eb96065b2a98b57a48a06ae28d285a71b5]
  https://github.com/ethereum/tests/blob/develop/VMTests/vmSystemOperationsTest.json#L1501
• Why the nonce of the creator doesn't increase:
  https://github.com/ethereum/tests/blob/develop/VMTests/vmSystemOperationsTest.json#L1532

Call precompiled contract by transaction

The call for 000...01 - 000...04 addresses strict from
regular transaction is not tested yet , I can contribute several
examples, but don't sure in which section include them. @CJentzsch

More intuitive interpretation of pre, exec and post requested

As I understood from @CJentzsch, the tests are created with the assumption that certain values used in the exec-part of the test, are assumed to be present, whereas it is more intuitive to define them in the pre-state of the test.

Example

balance0 in vmEnvironmentalInfoTest.json expects to have account 0xcd1722f3947def4cf144679da39c4c32bdc35681 in the post-state, but the pre-state doesn't define it.

Also, the balance of 0f572e5295c57f15886f9b263e2f6d2d6c7b5ec6 is defined both in the pre-state and post-state as 1000000000000000000 while the exec-part transfers a value of 1000000000000000000 to that account.

Could the tests be adapted to the interpretation that any pre-defined account, actually gets defined in the pre-part of the test and not by interpreting the exec-state, unless that's part of the protocol?

So in this case it would mean that pre would become

        "pre" : {
            "0f572e5295c57f15886f9b263e2f6d2d6c7b5ec6" : {
                "balance" : "0",
                "code" : "0x73cd1722f3947def4cf144679da39c4c32bdc3568131600057",
                "nonce" : "0",
                "storage" : {
                }
            }
            "cd1722f3947def4cf144679da39c4c32bdc35681" : {
                "balance" : "1000000000000000000",
                "code" : "0x",
                "nonce" : "0",
                "storage" : {
                }
            }
        }

trietest.json: jeff test permutations?

It looks as though the tests in tests/TrieTests/trietest.json are intended to be permuted, with deletions tacked onto the end to ensure that all bindings both added and deleted end up deleted. (Or maybe I'm wrong? I'm trying to follow along from e.g. here.)

That works fine for most of the tests there, but not, i think for test jeff in the current version , which has two items with identical keys but distinct values, and whose key is not eventually deleted.

      ["0x000000000000000000000000697c7b8c961b56f675d570498424ac8de1a918f6", "0x1234567890"],
      ["0x000000000000000000000000697c7b8c961b56f675d570498424ac8de1a918f6", "0x6f6f6f6820736f2067726561742c207265616c6c6c793f000000000000000000"],

So the hash of the trie after permutations of operations are legitimately dependent upon which binding is updated last, I think.

I apologize if I am (not unusually) missing something obvious!