The Trinity client for the Ethereum network
Home Page: https://trinity.ethereum.org
License: MIT License
continues from ethereum/py-evm#1131
The following LES command pairs do not have a round trip API available to them.
GetBlockBodies -> BlockBodiesGetReceipts -> ReceiptsGetProofs -> ProofsGetContractCodes -> ContractCodesGetProofsV1 -> ProofsV2Use and/or extend the existing Handler/Manager pattern to implement round trip APIs for these command pairs. At minimum each of these should be tested at the Request level and the Peer level. Feel free to borrow patterns and test cases from the existing tests.
We should update the fancy terminal code with output that matches the latest trinity console output. Looks to be a bit of a chore now that I see the HTML.
trinity --network-id 1 --sync-mode full
INFO 08-22 08:47:52 main
______ _ _ __
/_ __/____(_)___ (_) /___ __
/ / / ___/ / __ \/ / __/ / / /
/ / / / / / / / / / /_/ /_/ /
/_/ /_/ /_/_/ /_/_/\__/\__, /
/____/
INFO 08-22 08:47:52 main Trinity/0.1.0a14/linux/cpython3.6.3
State sync drops peers, does not finish and runs into the following issue:
INFO 08-22 08:34:50 peer Connected peers: 0 inbound, 1 outbound
INFO 08-22 08:34:50 peer Peer subscribers: 1, longest queue: StateDownloader(0)
...
ERROR 08-22 08:34:55 base_events Task exception was never retrieved
future: <Task finished coro=<BaseService.run_task.<locals>.f() done, defined at /root/git/trinity_venv/lib/python3.6/site-packages/p2p/service.py:129> exception=KeyError(b'6\x11\x08>A\x90\xc4\x9b\xe2\x1e\xf6^\x95Z\x11g\xcf\x15N-\xd3E_\xee;1-\x03\xd3e\x8a\xe2',)>
Traceback (most recent call last):
File "/root/git/trinity_venv/lib/python3.6/site-packages/p2p/service.py", line 131, in f
await awaitable
File "/root/git/trinity_venv/lib/python3.6/site-packages/trinity/sync/full/state.py", line 281, in _request_and_process_nodes
await self._process_nodes(node_data)
File "/root/git/trinity_venv/lib/python3.6/site-packages/trinity/sync/full/state.py", line 162, in _process_nodes
await self.scheduler.process([(node_key, node)])
File "/root/git/trinity_venv/lib/python3.6/site-packages/trinity/sync/full/hexary_trie.py", line 226, in process
await self.commit(request)
File "/root/git/trinity_venv/lib/python3.6/site-packages/trinity/sync/full/hexary_trie.py", line 241, in commit
await self.commit(ancestor)
File "/root/git/trinity_venv/lib/python3.6/site-packages/trinity/sync/full/hexary_trie.py", line 237, in commit
self.requests.pop(request.node_key)
KeyError: b'6\x11\x08>A\x90\xc4\x9b\xe2\x1e\xf6^\x95Z\x11g\xcf\x15N-\xd3E_\xee;1-\x03\xd3e\x8a\xe2'
INFO 08-22 08:34:57 state State-Sync: processed=2233917 tnps=13 committed=1874295 active_requests=8 queued=0 pending=1079163 missing=37 timeouts=384
Good question
More discussion at #868
Roughly:
Request blocks 1-5 from Peer A
Request blocks 6-10 from Peer B
Peer A responds with blocks 1-2
<-- at this point, we can request blocks 3-5 from Peer A, while waiting for Peer B to respond
At ECDC RocksDB was mentioned as a possibly faster alternative to leveldb.
Also comes with Python bindings
Another nice property of RocksDB is that it seems to allow multiple concurrent processes reading from the database which I think isn't possible with leveldb.
Trinity currently doesn't handle certain exceptional cases at startup very well. Here's a (probably incomplete) list of things that leave the user without great feedback:
Catch these things and print out user friendly error messages. Some of these things may blow up in separate processes that Trinity spawns so I'm unsure yet which would be the cleanest way to handle this.
The discovery v4 protocol is really bad at finding LES peers, so we had to resort to hard-coding known LES peers in the PeerPool we use with our LightChain. The latest release of geth (1.8.x) implements the discovery v5 protocol, though, which is much better at finding peers that support a specific sub-protocol (e.g. LES), so we should implement that (probably even dropping v4) and change our PeerPool to use p2p.DiscoveryProtocol instead of hard-coded nodes.
And then we need to write a canonicalizer which converts/removes trinity quirks so the traces can ve compared.
The complexity is in the details, getting the gas reporting and error handling to align across evms
About format: https://github.com/holiman/evmlab/blob/master/README.md#evm-format
(Writeup courtesy of @holiman)
Recently, someone on gitter asked if Trinity supports spinning up custom nets without touching the default code base. So, far that doesn't seem to be the case (maybe through subtile hacks idk). There seems to be a great opportunity to break up our existing node logic (full / light, mainnet / ropsten) into more fine grained building blocks to enable plugins to deliver such functionality. E.g. it would be nice if someone could just write a plugin to e.g. support the Kovan net (just using Kovan as an example, I have zero knowlege about any potential blockers with that).
Lots of exploration.
As suggested in ethereum/py-evm#841 (comment)
At present we have two peer classes, LESPeer and ETHPeer. When we start acting as a light server I believe we'll then have a 3rd peer class. Adding protocols like swarm and whisper will further expand this. Sharding and the beacon chain are also likely to add more peer classes.
As our codebase currently stands, we kind of act like our peer classes are interchangeable when in fact they are not. We've gotten this far by mirroring the API but I don't think this is sustainable.
Implement a framework for dealing with multiple different peer pools, each of which will only manage peers of a specific type.
ChainDBProxy provides both sync and async versions of the ChainDB methods, so it's up to callsites to decide which ones to use and whether or not to mix them. That feels icky and may lead to us accidentally making blocking calls in asyncio code, thus blocking the event loop and preventing other coroutines from running.
We could have separate ChainDB types for the sync and async versions, and ensure asyncio code only has access to the async ones
It's too slow to download all the headers from one peer.
Related to #1218 (because this issue depends the split of responsibility of choosing the peer and downloading the header)
Some code written, but paused on #1218.
Follow-up work: once all peers have throughput stats on headers, we can switch to a faster peer if our skeleton peer is too slow.
At the Ethereum Client Developer Conference @winsvega gave a quick overview about retesteth which aims to become the new standard for testing the different ethereum clients.
Here's a bit of background copied from the projects readme:
Currently ethereum-cpp has spearheaded the proof of concept phase for retesteth and @winsvega is looking for other client teams to adopt the testing APIs to be able to get tested through retesteth.
A good overview of the expected testing APIs can be found here: ethereum/retesteth#5
For anyone aiming to tackle this, it would probably be a good idea to connect with @winsvega and just quickly discuss how to get startet.
When we connect to a peer we extract some information from the Status message to store as indicators to the peer's current chain head.
https://github.com/ethereum/trinity/blob/master/trinity/protocol/eth/peer.py#L78-L79
Then, whenever a NewBlock message comes through, we update this information
https://github.com/ethereum/trinity/blob/master/trinity/protocol/eth/peer.py#L49-L53
This update assumes multiple things about the info transmitted via NewBlock which I do not believe we can reliably trust.
It appears that under these assumptions a malicious peer should be capable of luring trinity towards a header hash that is entirely bogus.
On connection, request the most recent N headers leading up to the peer's head_hash. This will be referred to as the "recent headers" chain.
When a NewBlock arrives validate it against the chain of N headers we have on-file. Using the block number from the new block, request the ancestor chain for the new block far enough back to reliably link up with the known recent headers. This should probably include a few extra headers to account for small chain re-orgs if the new block happens to be on a longer fork chain from the known recent header chain. If the new block can be validated against the peer's recent headers then extend the recent header chain up to the new block and discard the oldest headers to keep the recent headers list under a max size N
Validating the recent header chain should be as thorough as possible without incurring unreasonable overhead.
The recent header chain can also likely be a shared resource across multiple peers to avoid re-requesting the same headers from each peer.
Please open a pull request (or just edit in-place) with your preferred job title as well as correct URIs for twitter/github/whatever ASAP. Website launch will happen with tuesday's release.
make_trie_root_and_nodes() seems to be the most expensive part of importing blocks during a fast-sync. The stats below were obtained by running a fast-sync on a full ropsten DB (~34GB)
09:51:32 INFO: Starting sync with ETHPeer <Node([email protected])>
09:51:38 INFO: Imported 192 blocks (1529 txs) in 3.57 seconds, new head: #3713461
09:51:42 INFO: Imported 192 blocks (1437 txs) in 3.22 seconds, new head: #3713653
09:51:50 INFO: Imported 192 blocks (3345 txs) in 7.36 seconds, new head: #3713845
09:51:56 INFO: Imported 192 blocks (2486 txs) in 5.74 seconds, new head: #3714037
09:52:03 INFO: Imported 192 blocks (2814 txs) in 6.19 seconds, new head: #3714229
09:52:13 INFO: Imported 192 blocks (4042 txs) in 8.86 seconds, new head: #3714421
09:52:20 INFO: Imported 192 blocks (2921 txs) in 6.36 seconds, new head: #3714613
09:52:26 INFO: Imported 192 blocks (2282 txs) in 5.12 seconds, new head: #3714805
09:52:30 INFO: Imported 192 blocks (1574 txs) in 3.70 seconds, new head: #3714997
09:52:32 INFO: Connected peers: 0 inbound, 1 outbound
09:52:32 INFO: Peer subscribers: 1, longest queue: FastChainSyncer(0)
09:52:35 INFO: Imported 192 blocks (2128 txs) in 4.64 seconds, new head: #3715189
09:52:43 INFO: Imported 192 blocks (2888 txs) in 6.61 seconds, new head: #3715381
^C09:52:44 INFO: Stopping all peers ...
95825233 function calls (88180727 primitive calls) in 73.139 seconds
Ordered by: cumulative time
List reduced from 3383 to 50 due to restriction <50>
ncalls tottime percall cumtime percall filename:lineno(function)
460/1 0.008 0.000 73.160 73.160 {built-in method builtins.exec}
1 0.000 0.000 73.160 73.160 chain.py:821(_test)
1 0.028 0.028 72.616 72.616 base_events.py:405(run_forever)
33772 0.444 0.000 72.588 0.002 base_events.py:1336(_run_once)
120850 0.378 0.000 71.185 0.001 events.py:143(_run)
6179 0.004 0.000 45.780 0.007 chain.py:119(handle_msg)
6179 0.004 0.000 45.777 0.007 chain.py:538(_handle_msg)
34 0.006 0.000 45.239 1.331 chain.py:924(mock_run_in_executor)
4199 0.058 0.000 45.233 0.011 trie.py:16(make_trie_root_and_nodes)
4199 0.157 0.000 38.886 0.009 trie.py:25(_make_trie_root_and_nodes)
56102 0.053 0.000 37.186 0.001 hexary.py:417(__setitem__)
56102 0.169 0.000 37.133 0.001 hexary.py:86(set)
145122/56102 0.315 0.000 27.657 0.000 hexary.py:96(_set)
252448 0.504 0.000 23.323 0.000 hexary.py:200(_node_to_db_mapping)
4138 0.017 0.000 23.100 0.006 chain.py:593(_handle_block_bodies)
2029 0.009 0.000 22.671 0.011 chain.py:577(_handle_block_receipts)
89012/47996 0.206 0.000 19.907 0.000 hexary.py:321(_set_branch_node)
4302417/389130 5.010 0.000 19.277 0.000 codec.py:72(encode_raw)
385996/328777 1.233 0.000 15.411 0.000 {method 'join' of 'bytes' objects}
110997 0.129 0.000 15.260 0.000 codec.py:194(decode)
557707/240751 1.230 0.000 14.460 0.000 functional.py:20(inner)
4299282/4135450 1.595 0.000 14.245 0.000 codec.py:80(<genexpr>)
145122 0.224 0.000 12.798 0.000 hexary.py:218(_prune_node)
265824 0.381 0.000 11.722 0.000 crypto.py:8(keccak)
12563569/12563210 5.024 0.000 11.541 0.000 {built-in method builtins.isinstance}
110 0.000 0.000 11.540 0.105 service.py:46(run)
102 0.000 0.000 11.537 0.113 peer.py:418(_run)
158 0.002 0.000 11.534 0.073 peer.py:433(read_msg)
52 0.002 0.000 11.365 0.219 protocol.py:93(decode)
52 0.013 0.000 11.361 0.218 protocol.py:78(decode_payload)
107326 0.192 0.000 11.112 0.000 hexary.py:212(_persist_node)
324501 0.634 0.000 11.028 0.000 main.py:32(__call__)
77405/33976 0.299 0.000 10.362 0.000 serializable.py:254(deserialize)
163034/6554 0.193 0.000 10.035 0.002 lists.py:126(deserialize)
10615 0.019 0.000 9.946 0.001 integration_test_helpers.py:28(passthrough_method)
324501 0.509 0.000 9.869 0.000 pycryptodome.py:10(keccak256)
127534 0.266 0.000 7.833 0.000 codec.py:19(encode)
60301 0.265 0.000 7.673 0.000 hexary.py:171(_set_root_node)
324501 0.973 0.000 6.521 0.000 keccak.py:135(new)
5385592 3.499 0.000 6.518 0.000 abc.py:180(__instancecheck__)
60301 0.066 0.000 6.289 0.000 trie.py:18(<genexpr>)
25467 5.704 0.000 5.704 0.000 {method 'get' of 'plyvel._plyvel.DB' objects}
4427 0.018 0.000 5.558 0.001 header.py:143(header_exists)
4427 0.010 0.000 5.533 0.001 base.py:38(__contains__)
4427 0.009 0.000 5.522 0.001 level.py:31(_exists)
324501 1.985 0.000 5.325 0.000 keccak.py:63(__init__)
562090/213864 0.532 0.000 5.240 0.000 lists.py:78(deserialize)
110997 0.106 0.000 4.636 0.000 codec.py:182(consume_item)
2308885/110997 2.674 0.000 4.151 0.000 codec.py:153(consume_payload)
149321 0.197 0.000 3.775 0.000 hexary.py:186(get_node)
Need to investigate ways to optimize that function
When developing it's important to know the commit ID that was in use on a given trinity run
use the following algorithm to compute version string.
d = pkg_resources.get_distribution('trinity') to lookup the distribution information for trinityIf d.location is a git repository, then construct the version string as follows. The git commit portion can be retrieved using a subprocess call to git rev-parse HEAD
<d.version>-dev.<git-commit-hex-to-8-chars>
"1.2.3-dev-9e977472"
If d.location is not a git repository, then use d.version as the version string.
If the call to pkg_resources throws and exception then the setup_trinity.py file should be parsed (regex?) for the version string.
<version-string>-dev-<git-commit-hex-to-8-chars>
Right now we have the base logic for managing peer connections and the business logic for managing peer connections coupled.
Base logic is defined as things like the base p2p connection/handshake, is_full checks, starting and stopping peers.
Business logic is defined here as things like "preferred peers", dao fork checks, stats reporting, selection of best peers (not implemented).
I'm curious if these should be decoupled. The base logic pretty much never changes, but the business logic is more dynamic. We could potentially have a separate concept which I'll call peer pool manager. The manager is responsible for implementing these business logic components. This saves us from having to change our peer pool class so often, allowing us to just swap out the manager.
This will require some discovery work to figure out if it's a good idea, but dropping it here just in case.
Recently, research team is working on refining the new spec and the beacon chain PoC codebase (big shout out to V!). Basically, there’s a beacon chain that will produce random numbers with RANDAO-based scheme, and we can build Casper logic in the beacon chain to support both beacon chain and shard chains staking.
This issue is for discussing (1) what feature/components are required and (2) what the blueprint looks like if we want to implement the beacon chain in the Trinity pyeth world?
cc @djrtwo , please correct me if I'm wrong. :)
main_chain_ref in beacon block, the beacon chain validators have to follow the actual canonical main PoW chain. (see Beacon-chain-processing for more details)beacon module besides evm to put the chain logic?evm architecture.
├── chains
├── constants.py
├── db
├── exceptions.py
├── tools
├── utils
└── validation.py
Currently we first sync the chain data (blocks, receipts, etc) and then sync the state trie. This approach won't reliably work for doing a full sync of the chain because:
This means that unless trinity happens to be connected to a node running in archive mode (extremely rare) that by the time it completes the state sync there will be missing trie nodes which will not be available from most any peer it connects to.
And even if trinity happens to be connected to an archive node, in order to hit the necessary performance numbers for Trinity to be viable, we will need to process these two syncs concurrently.
An initial exploratory attempt can be found here: ethereum/py-evm#1231
Running the two sync processes concurrently is reasonably easy to do, however, the problems that need to be solved are related to how we orchestrate these two sync processes.
See: #54
We need a reference to the HEAD chain header that we are syncing towards.
state_root as our sync target.Currently the chain data sync updates the target but the state trie sync does not. We need to address the security issues laid out in #1233 as well as implement a mechanism for the state sync to update the state_root that it is targeting.
This isn't exactly required, however I believe it will make this easier to manage.
Currently chain data sync syncs headers and bodies together. This should be separated such that we sync the full header chain ahead of the block bodies, effectively making it into two sync processes.
By moving the header sync into a dedicated process we can institute the head tracking in that process and then cue off of the current target HEAD in the block body and state trie sync.
state_rootA naive approach for updating the state_root is to simply abandon the previous syncer and start a new one. This however is likely to incur potentially significant performance overhead since it will require re-walking the entire state trie.
Intelligent updating of the state root would ideally involve
SyncRequest objects that are still valid for the new state rootSyncRequest objects which are no longer valid and removing their entries from the database.Due to the overhead we also cannot migrate to a new state root on every block. Since go-ethereum garbage collects trie nodes after 128 blocks, it is likely appropriate to only migrate to a new target state root at around that interval.
Once we have a reliable way for both chain data and state trie sync to update the targets they are syncing towards, we need a mechanism to synchronize these two processes, i.e, ensure that both sync processes are syncing towards the same target.
Since updating state sync incurs overhead we likely need to only update the sync target every 128 blocks. This means that both state sync and chain data sync will need to be able to pause if they catch up to the sync target and to only resume if the other process does not also complete before the next update.
For the chain data at least, once it has caught up to the sync target we should be able to continue to download the chain data but to cache it in memory, only processing it once the sync target has been updated which would allow chain data sync to catch up faster than trie data sync.
Checklist of endpoints to support, and which are supported in master or pull requests
web_* methods
net_* methods
net_listeningnet_peerCountnet_version #840eth_* write methods
eth_* read-only methods
eth_protocolVersioneth_syncingeth_coinbaseeth_miningeth_hashrateeth_gasPriceeth_accountseth_blockNumbereth_getBalanceeth_getStorageAteth_getTransactionCounteth_getBlockTransactionCountByHasheth_getBlockTransactionCountByNumbereth_getUncleCountByBlockHasheth_getUncleCountByBlockNumber block inteth_getUncleCountByBlockNumber block stringeth_getCodeeth_calleth_estimateGaseth_getBlockByHasheth_getBlockByNumber block inteth_getBlockByNumber block stringeth_getTransactionByHasheth_getTransactionByBlockHashAndIndexeth_getTransactionByBlockNumberAndIndexeth_getTransactionReceipteth_getUncleByBlockHashAndIndexeth_getUncleByBlockNumberAndIndex block inteth_getUncleByBlockNumberAndIndex block stringeth_getCompilerseth_getLogseth_getWorkWe have some version of this pattern in place in multiple places across the codebase where we need to distribute requests across multiple peers while enforcing some set of requirements:
HEAD total difficulty is above a certain low water mark.ETHPeer peers for many types of requests.The exact way that this gets implemented is yet to be figured out, but I suspect the round trip request/response API that's been recently implemented will play a part.
Trio is a third party async library similar to asyncio. According to their documentation
Trio was built from the ground up to take advantage of the latest Python features, and draws
inspiration from many sources, in particular Dave Beazley’s Curio. The resulting design is
radically simpler than older competitors like asyncio and Twisted, yet just as capable. Trio is
the Python I/O library I always wanted; I find it makes building I/O-oriented programs easier,
less error-prone, and just plain more fun. Perhaps you’ll find the same.
The main reason why I prelimiarily prefer it over asyncio is that all concurrency is explicit: Parents can spawn child tasks, but only inside a "nursery" context manager. Whenever a nursery is being exited, all child tasks are awaited and all exceptions are collected. In particular, one cannot spawn tasks that just run somewhere in the void like with asyncio.ensure_future (whose exceptions may just get ignored if not handled explicitly).
As a result all tasks form an easy to overlook tree, enabling a second neat feature: Cancel scopes. Tasks can have (potentially nested) cancel scopes created with a context manager or inherited from their parent. This seems to fulfill the same functionality as our cancel tokens, but one doesn't have to pass them around explicitly (less clutter, less errors, and due to the task tree structure probably more clear).
The obvious concern is ecosystem adoption: The project is still young. It seems to be actively developed since 1 1/4 years, but mainly by a single developer (although in the last month there have been commits by 8 contributors). It doesn't seem like there are major projects already using it (I don't know of a good way to find them though). Requests is apparently experimenting with Trio for their version 3.0. There is a pytest plugin. A couple of projects are being ported here, including click and PyMySQL. There's also trio-asyncio which apparently allows using libraries written with asyncio support from trio.
On that topic, a relevant quote by the author:
If you need to ship something to production next week, then you should use asyncio (or Twisted or Tornado or gevent, which are even more mature). They have large ecosystems, other people have used them in production before you, and they're not going anywhere.
[...]
If you want to ship something to production a year from now... then I'm not sure what to tell you. Python concurrency is in flux. Trio has many advantages at the design level, but is that enough to overcome asyncio's head start? Will asyncio being in the standard library be an advantage, or a disadvantage? (Notice how these days everyone uses requests, even though the standard library has urllib.) How many of the new ideas in trio can be added to asyncio? No-one knows. I expect that there will be a lot of interesting discussions about this at PyCon this year :-).
References:
(Note: I'm not necessarily advocating to switch to Trio, I just want to make sure that an active and informed decision is made.)
We currently maintain this hack:
Lines 771 to 779 in f61a486
We can get rid of this as soon as a new lahja release lands that supports cancelling async generators properly
More than 90% of the hard drive footprint is chain data (headers/blocks/transactions/receipts).
We can both reduce the hard drive footprint as well as the time it takes to sync a node if we only store a subset of the chain data. This will be referred to as sharding the chain data.
The simple concept is for each node to only store 1/N blocks. We'll refer to N as the sharding_factor
Strategies:
f(block_hash, enode_public_key) which has an even distribution.In addition to only storing 1/N blocks we need to also be able to lazily retrieve blocks as needed for things like the JSON-RPC server.
In addition to only storing 1/N blocks we need to keep a cache of the most recent X blocks. We can start with something simple like X = 1024.
Experienced this today. Will have to investigate further, but these are the relevant log lines
INFO 08-06 12:54:09 state State-Sync: processed=1370048 tnps=426 committed=1365724 requested=198 scheduled=7631 timeouts=2940
[...]
DEBUG 08-06 12:54:15 state Got 60 NodeData entries from ETHPeer <Node([email protected])>
DEBUG 08-06 12:54:15 trie Node 0xd57d6e0df47c8c8a359b7af81d0dd76b80aa5a95aeb320d61b017df10ed029c7 already exists in db
DEBUG 08-06 12:54:15 trie Node 0x877bbce004d73032b7b05d968a259c8dfe39b10182ac5ced7d2478976ba5f5d6 already exists in db
DEBUG 08-06 12:54:15 trie Scheduling retrieval of 0x0a861d465d21c9527ee1cfaf8ca6f62b87055c803a89699e95781b903237571b
DEBUG 08-06 12:54:15 trie Scheduling retrieval of 0xd610162d87c8c87662627bcbff6916f447c6cebedb1b94a30130e5ef86edc2bd
DEBUG 08-06 12:54:15 trie Scheduling retrieval of 0x1243ccf7ee035b937b24b5f1867070e3ff0a14620b81854c28bdf7ecef789ec7
DEBUG 08-06 12:54:15 trie Scheduling retrieval of 0x4b7078982d505fc3d276ce3bbfee2f5c26f8222e119adcb887dea220e5ac9f59
DEBUG 08-06 12:54:15 trie Scheduling retrieval of 0x0939fb0d77a4a93698d6c1eefa273f9f38a8fbe1e41ff74c17e0e156434c120e
DEBUG 08-06 12:54:15 trie Scheduling retrieval of 0x1e73e41621afc7e7346b0ea80542f3d7f89d5578972e7e6a01dce9cf3cfccb05
DEBUG 08-06 12:54:15 trie Node 0x8e134fdcd8da2f69d596b1bab5b8caabb8f52fa3ac9a4513cc20595059cce251 already exists in db
DEBUG 08-06 12:54:15 trie Scheduling retrieval of 0x16c4626b8cbb1df7d6b36bc226d5307668cbf76e04f34e786a0c3728d4359150
DEBUG 08-06 12:54:15 trie Scheduling retrieval of 0x2d7afc9bfabf76eeb53b937a7ca0dc6b09f904e56ef5eee7cb886c6a45629a20
DEBUG 08-06 12:54:15 trie Scheduling retrieval of 0x6bbfa13e8d27d2ea2064a270b14c86d9e477cc405309dd204804189f104cb635
DEBUG 08-06 12:54:15 trie Node 0x42069d4881619fe803b5ecf58fa83a661ae459ff776aa3816bb13d2700c99b13 already exists in db
DEBUG 08-06 12:54:15 trie Scheduling retrieval of 0xefa0ff3d9336f417cf903c7a4098da5acb63a4bfbd3dc954b1bf910a5a66032d
DEBUG 08-06 12:54:15 trie Scheduling retrieval of 0x07f22aaeb3a0c0efe6518ec2a2e218aaa4910a731adbcc0d76dce33917567d5e
DEBUG 08-06 12:54:15 trie Scheduling retrieval of 0x378509d4ad2afe8ba44389aadf638738b1b195d7716f879f08e111905a4f316e
DEBUG 08-06 12:54:15 state Requesting 12 trie nodes to ETHPeer <Node([email protected])>
DEBUG 08-06 12:54:15 trie Scheduling retrieval of 0x3bc5bf4fcadc32dcab4bea7183ea0ad2f56cd5fd8f3bd969adac29a0d1d0eec8
DEBUG 08-06 12:54:15 state Requesting 1 trie nodes to ETHPeer <Node([email protected])>
DEBUG 08-06 12:54:15 state Scheduler queue is empty, sleeping a bit
DEBUG 08-06 12:54:15 state Got 1 NodeData entries from ETHPeer <Node([email protected])>
DEBUG 08-06 12:54:16 state Got 12 NodeData entries from ETHPeer <Node([email protected])>
DEBUG 08-06 12:54:16 state Scheduler queue is empty, sleeping a bit
[...]
INFO 08-06 12:54:19 state State-Sync: processed=1374321 tnps=426 committed=1373077 requested=0 scheduled=1821 timeouts=2940
[...]
INFO 08-06 12:54:29 state State-Sync: processed=1374321 tnps=425 committed=1373077 requested=0 scheduled=1821 timeouts=294
We need to keep track of peers that send us too many invalid transactions and eventually block those.
This depends on the block list mechanism as discussed in #893
We first need to tackle #893 to have a generic mechanism to block peers. Once we got that, the transaction pool should use that mechanism to eventually block peers that feed us too much junk.
Based on a note within the AllCoreDevs channel snappy compression may be manditory very soon.
We'll need to implement compression/decompression in our networking layer.
We currently are using dictionaries for the msg portion for commands like GetBlockHeaders. It's hard to specify types for these, as well as them being inherently mutable structures.
Commands should decode into a NamedTuple instead when they have key/value data.
When disconnecting from a peer, some kinds of disconnects suggest that we should not try to reconnect to it later (is peer myself? is it replying with protocol-breaking headers? etc). It would be best to not try to connect to it again later, but nothing is preventing us from retrying.
The blacklist can be in-memory, resetting on every trinity reboot.
But names are hard!
I suggest we continue from the origins of the name "Trinity" and name our releases after badass women. Maybe we can use this tread to collect some names?
Right now Trinity has a loose concept of a Handler found here: https://github.com/ethereum/py-evm/blob/683c53cced96a863f474ce4cf058aa2ff05e7665/trinity/p2p/handlers.py
These handlers are Services which run within the syncers (and thus are chained with the syncing Service classes). They handle p2p requests from peers, which normally involves looking some data up from our local chain database and then sending the response over the wire to the peer.
In the event that the peer in question disconnects, we end up doing un-necessary and potentially expensive data process as well as attempting to send to a disconnected peer.
The vast majority of the ways in which this can manifest are handled in ethereum/py-evm#1307 (review) . Since network latency is greater than local database latency, we can reasonably expect the majority of these errors to manifest in times when the various peer message processing queues are not empty and thus the co-routines to handle messages are experiencing some minor wait times before executing, thus allowing time for peer disconnects to happen prior to processing.
However, each of the Handler methods has multiple await calls to lookup data. The peer could disconnect during any of these await calls.
Not clear. The ideal solution has any call to the handler methods gracefully cancelling itself immediately upon either peer cancellation or cancellation of the handler service.
Ideas for how to do this:
Handler.wait and Peer.wait in a clean way such that either one causes cancellation but doesn't propagate cancellation to either parent service.We do a lot of caching across many different locations in the codebase. Our current approach of having many disparate caches is fine for running nodes on normal hardware, but when it comes to running trinity on a device with limited resources, this approach is likely to cause problems.
Ideally the user should be able to supply a single upper bound for cache size which would be applied to the total sizing for the majority of caches.
Trinity is not included on hive tests: http://hivetests.ethstats.net/
Information on where to get started:
One dockerfile, and one shellscript to init, load blocks, and start rpc services.
The shellscript translates env vars into client-specific configuration
(Info courtesy of @holiman and of course, karalabe)
Quick brain dump from ethereum/py-evm#744 (comment)
Trinity currently uses a hardcoded list of recommended peers + discovery protocol. It would be nice to maintain a list of recommended peers based on the following metrics that also survives restarts (persisted in db / config file).
Metrics:
Initial idea is to have some sort of PeerSubscriber API which would be given a view into peer requests and responses and then to aggregate that data into some accessible form. Then the a FancyPeerPool class could use this data to inform what nodes/peers it returns.
Upon stopping and resuming state sync, my sync has stalled. There appears to be a sincle trie node in the request queue that none of my peers have and it cannot move beyond it.
Haven't dug in yet.
During a brainstorming session, we picked out and prioritized some high-level tasks to work on for Trinity. They are below, and probably include issues that are already created.
The meta-task is to:
Do very soon
Do soon
Do
@pipermerriam 's comment:
As it stands I think that we may start syncing with peer A and then connect to peer B which has a higher TD. At this point we are locked into syncing with A even though B is the better peer.
(where better := higher total difficulty)
Each time, after adding headers to the task queue (in trinity.sync.common.chain), check if the current peer is the highest td peer. If so, continue as before. Otherwise, break out of _sync and start again with the better peer.
A randomly triggered failure in state sync tests would be difficult/impossible to reproduce. See discussion at ethereum/py-evm#676 (comment)
Some options:
An important component of this is to add a Local Broadcast Transaction Queue
This queue of transactions should exclusively contain transactions that were broadcasted locally (eg~ over json-rpc's eth_sendRawTransaction).
The queue should persist across Trinity sessions, broadcast to all peers, and be immune to local transaction pool eviction strategies.
The current state sync outputs a statistic of nodes processed per second.
trinity/trinity/sync/full/state.py
Lines 279 to 292 in 3b52bde
This number is the average since the start of syncing. We will run into situations where this number is not accurately representing the sync speed since we may lose or gain connections to fast peers, or experience changes in network speed.
Change this statistic to be reported using an exponential moving average, ideally weighting heavily on the performance over the most recent 3-5 minutes.
The official documentation recommends this pattern:
try:
loop.run_forever()
finally:
loop.run_until_complete(loop.shutdown_asyncgens())
loop.close()We need to double check if and where this would apply to our code base but probably worth to investigate.
Chain instances.Chain instances.Not sure, but it may look something like segregating this data into two separate databases.
Get this error during startup:
(trinity) bryant@camilla:~$ trinity
INFO 07-25 17:14:53 main
______ _ _ __
/_ __/____(_)___ (_) /___ __
/ / / ___/ / __ \/ / __/ / / /
/ / / / / / / / / / /_/ /_/ /
/_/ /_/ /_/_/ /_/_/\__/\__, /
/____/
INFO 07-25 17:14:53 main Trinity/0.1.0a13/linux/cpython3.6.5
INFO 07-25 17:14:53 main Trinity DEBUG log file is created at /home/bryant/.local/share/trinity/mainnet/logs/trinity.log
INFO 07-25 17:14:53 main Started DB server process (pid=16468)
INFO 07-25 17:14:54 main Started networking process (pid=16472)
INFO 07-25 17:14:54 server Running server...
INFO 07-25 17:14:54 nat Setting up NAT portmap...
INFO 07-25 17:14:54 ipc IPC started at: /home/bryant/.local/share/trinity/mainnet/jsonrpc.ipc
ERROR 07-25 17:14:59 ssdp Error '404 Client Error: Not Found for url: http://10.0.0.164:8008/ssdp/notfound' for http://10.0.0.164:8008/ssdp/device-desc.xml
ERROR 07-25 17:14:59 ssdp Error '404 Client Error: Not Found for url: http://10.0.0.164:8008/ssdp/notfound' for http://10.0.0.164:8008/ssdp/device-desc.xml
ERROR 07-25 17:14:59 ssdp Error '404 Client Error: Not Found for url: http://10.0.0.164:8008/ssdp/notfound' for http://10.0.0.164:8008/ssdp/device-desc.xml
ERROR 07-25 17:14:59 ssdp Error '404 Client Error: Not Found for url: http://10.0.0.164:8008/ssdp/notfound' for http://10.0.0.164:8008/ssdp/device-desc.xml
...
and on occasion (every half hour or so) during sync
...
INFO 07-25 17:45:05 chain Imported 192 blocks (135 txs) in 1.24 seconds, new head: #242880
ERROR 07-25 17:45:06 ssdp Error '404 Client Error: Not Found for url: http://10.0.0.164:8008/ssdp/notfound' for http://10.0.0.164:8008/ssdp/device-desc.xml
ERROR 07-25 17:45:06 ssdp Error '404 Client Error: Not Found for url: http://10.0.0.164:8008/ssdp/notfound' for http://10.0.0.164:8008/ssdp/device-desc.xml
ERROR 07-25 17:45:06 ssdp Error '404 Client Error: Not Found for url: http://10.0.0.164:8008/ssdp/notfound' for http://10.0.0.164:8008/ssdp/device-desc.xml
ERROR 07-25 17:45:06 ssdp Error '404 Client Error: Not Found for url: http://10.0.0.164:8008/ssdp/notfound' for http://10.0.0.164:8008/ssdp/device-desc.xml
INFO 07-25 17:45:07 nat NAT port forwarding successfully set up: PortMapping(internal='10.0.0.66:30303', external='24.63.123.192:30303')
INFO 07-25 17:45:08 chain Imported 192 blocks (169 txs) in 2.15 seconds, new head: #243072
...
link: ethereum/py-evm#712
Need to make $ trinity attach behave in the following way.
When $ trinity attach is run with no other arguments, the following logic should be done.
jsonrpc.ipc all chain folders found in the $TRINITY_HOME.
>1 are found, displays appropriate error.However, the path the socket should be able to be specified as a positional argument to the command as $ trinity attach /path/to/jsonrpc.ipc. When the path is specified this way trinity should log the path to the IPC socket it is connecting to and launch the console.
We have some services that do things like await other_service.method(), and when that happens inside a service's _run() method, we need to remember to handle OperationCancelled (e.g. https://github.com/ethereum/py-evm/blob/master/trinity/sync/common/chain.py#L178) or else it could bubble up and cause the wrong service to be cancelled
When the await other_service.method() is executed in the background with BaseService.run_task() & co, that is not a problem as run_task() will handle the OperationCancelled. For example, https://github.com/ethereum/py-evm/blob/master/trinity/sync/full/state.py#L231-L236 is fine
We need to decide if we want to try and completely forbid the await other_service.method() pattern to prevent issues like #1234, or if there's a way we can allow that but in a way that we don't have to always remember to handle OperationCancelled.
Maybe a new BaseService method that takes an awaitable, awaits for it and handles OperationCancelled would be good enough
eth_* read-only methods
eth_protocolVersioneth_syncingeth_gasPrice (should get its own issue to plan the algorithm)eth_blockNumbereth_getBalanceeth_getStorageAteth_getTransactionCounteth_getBlockTransactionCountByHasheth_getBlockTransactionCountByNumbereth_getUncleCountByBlockHasheth_getUncleCountByBlockNumber block inteth_getUncleCountByBlockNumber block stringeth_getCodeeth_calleth_estimateGaseth_getBlockByHasheth_getBlockByNumber block inteth_getBlockByNumber block stringeth_getTransactionByHasheth_getTransactionByBlockHashAndIndexeth_getTransactionByBlockNumberAndIndexeth_getTransactionReceipteth_getUncleByBlockHashAndIndexeth_getUncleByBlockNumberAndIndex block inteth_getUncleByBlockNumberAndIndex block stringeth_getCompilerseth_getLogseth_* read-only methods
eth_coinbase (returns error with: Trinity does not support mining)eth_hashrate (returns error with: Trinity does not support mining)eth_mining (returns False since Trinity does not support mining)eth_getWork (returns error with: Trinity does not support mining)eth_accounts (Trinity will always reply with [])chain reference ?Replaces ethereum/py-evm#381
The way things are currently structured there is a class which combines the logic of the Chain with the necessary networking logic for syncing with the ethereum network.
This duel purpose class is problematic for the following reasons.
Chain abstraction and whatever networking abstraction we are using.Lets split the networking specific code into a new class, NetworkClient. We can use a naming convention like LightClient and FullClient for various different syncing approaches, but the Client suffix feels appropriate.
The NetworkClient would be given a Chain object and probably the low level database connection or maybe both the low level database and the high level ChainDB. Interactions with the chain or chain databases can be done using Executors from asyncio.
To facilitate the JSON-RPC server we can also create a ChainOverNetwork class which implements the Chain API. This class would be given access to the NetworkClient instance and would implement the necessary logic to retrieve chain data like Blocks or Receipts over the network.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.