paritytech/substrate-archive

Blockchain Indexing Engine

Maintainance status: currently unmmaintained; please fork.

Have a look at #464 for more information.

Substrate Archive

Blockchain Indexing Engine

Install the CLI • Documentation • Contributing • FAQ

Run alongside a substrate-backed chain to index all Blocks, State, and Extrinsic data into PostgreSQL.

Usage

The schema for the PostgreSQL database is described in the PDF File at the root of this directory.

Examples for how to use substrate-archive are in the examples/ directory

Prerequisites

Extended requirements list found in the wiki

• depending on the chain you want to index, ~60GB free space
• PostgreSQL with a database ready for lots of new data
• Substrate-based Blockchain running with RocksDB as the backend
• Substrate-based Blockchain running under --pruning=archive

Install The CLI

The CLI

The CLI is an easier way to get started with substrate-archive. It provides a batteries-included binary, so that you don't have to write any rust code. All thats required is setting up a PostgreSQL DB, and modifying a config file. More information in the wiki

The Node-Template CLI

The node-template CLI (in /bin/node-template-archive) is provided as an example of implementing substrate-archive for your chain.

Quick Start

git clone https://github.com/paritytech/substrate-archive.git
# Set up the databases
source ./substrate-archive/scripts/up.sh # Run ./scripts/down.sh to drop the database
cd substrate-archive/bin/polkadot-archive/
# Start the normal polkadot node with `pruning` set to `archive`
polkadot --chain=polkadot --pruning=archive
# Start up the substrate-archive node. `chain` can be one of `polkadot`, `kusama`, or `westend`.
cargo run --release --  -c test_conf.toml --chain=polkadot

You can access the help dialog via cargo run --release -- --help. Note that up and down scripts are meant for convenience and are not meant to be complete. Look in the wiki for more information about the database setup.

FAQ

Contributing

Contributors are welcome!

Read the Doc

Documentation

You can build the documentation for this crate by running cargo doc. More Docs here

Troubleshooting

Archive fails to start with a too many open files error.

Because of the way a RocksDB Secondary Instance works, it requires that all the files of the primary instance remain open in the secondary instance. This could trigger the error on linux, but simply requires that you raise the max open files limit (ulimit):

• With Docker: $ docker run --ulimit nofile=90000:90000 <image-tag>
• For Current Shell Session: ulimit -a 90000
• Permanantly Per-User

  • Edit /etc/security/limits.conf

    # /etc/security/limits.conf
    *           hard    nofile      4096
    *           soft    nofile      1024
    some_usr    hard    nofile      90000
    some_usr    soft    nofile      90000
    insipx      hard    nofile      90000
    insipx      soft    nofile      90000
    root        hard    nofile      90000
    root        soft    nofile      90000
    

For macOS and Linux, a warning message will be raised on the startup when there is a low fd sources limit in the current system, but Windows won't have such a low fd limit warning.

Contact

You can contact us at:

• matrix: #substrate-archive:matrix.parity.io
• email: [email protected]

substrate-archive's Issues

The schema for the PostgreSQL database is described in the Pdf File at the root of this directory

I cannot find any pdf file in this repo.

Handle Postgres Disconnects in Listener

When running substrate-archive with a remote database, and there happens to be some network error or dropped packets that causes the PostgreSQL connection to drop and then re-connect, substrate-archive will stop running.

Rather than use next stream iterator we should use try_recv in Listener.

https://github.com/paritytech/substrate-archive/blob/master/src/database/listener.rs#L134

https://docs.rs/sqlx/0.4.0-beta.1/sqlx/postgres/struct.PgListener.html#method.try_recv

Introduce Concept of Storage Priority

Storage priority could be a configuration argument that specifies how storage should be indexed.

For example, a priority could be MostRecentlyFound which would index storage for the newest blocks first, even if older blocks were found.

This could be useful in systems where there exists a need for recent values first, and historical entries next.

verify data in database is correct with `ProofProvider`

It could be possible to verify data in PostgreSQL is correct by rebuilding the root hash using only the data in Postgres, at some chosen BlockId, and comparing with the data from ProofProvider trait methods.

Should eliminate RPC dependencies entirely

Since using a custom substrate client, an RPC is no longer needed and adds unnecessary bloat/complexity.

Should update the backend trait to use BlockchainEvents and RuntimeApi/CallApiAt providers. (BlockchainEvents for getting new finalized blocks, and RuntimeAPI/CallApiAt for getting the metadata.)

RuntimeApi may be tricky because of the associated types it requires

Consider Actix

Consider trade offs of Actix vs Bastion

Super Quick Overview:
Actix is trait based, Bastion is not. Bastion focuses on fault-tolerance/heavy Erlang inspiration, Actix focuses on being a more rusty actor model.

Allow a `max_block_load` parameter to limit the amount of blocks that can be loaded/inserted at any one time

Currently when substrate-archive starts syncing with a chain that is far ahead, it will load as many blocks as it is missing in it's Postgres Database. This could be anything from a few thousand to a couple million blocks. This quickly spikes memory usage if many hundreds of thousands of blocks are loaded at once. Allowing the archive to be configured with a maximum amount of blocks it can load at once would help mitigate this issue. Currently, this happens in the blocks actor:

Dataflow: https://camo.githubusercontent.com/48a4e5bca346a1487ea36cc0e72dca7a394417a7/68747470733a2f2f692e696d6775722e636f6d2f53304e727842332e706e67

Code:

substrate-archive/src/actors/workers/blocks.rs

Line 68 in 7d37573

async fn collect_blocks(

substrate-archive/src/actors/workers/blocks.rs

Line 91 in 7d37573

async fn re_index(&mut self) -> Result<Option<Vec<Block<B>>>> {

This may require some refactoring of how the blocks actor indexes blocks, but should be fairly straightforward

Make Client Generic over Type of Block

the substrate client is currently hardcoded for a blake2_256 and u32 as the header, with OpaqueExtrinsic for the Extrinsic Value. This makes it cumbersome in areas of the code that is generic over the hashtype (System::Hash) having to create an extra <T as System>::Hash: From<H256> bound in order to convert between the two types when using rpc/client in conjunction.

May need to create our own System sub-trait for the archive node and implement it on Kusama's polkadot runtime to pass to client, instead of relying on subxt. This will also mean we have less to impl on Kusama, since we only really need BlockNumber, Hash, and Header associated types of System. Desub will take care of the rest

Make `handle_shutdown` a util function that accepts a closure

used in pretty much every actor

this would make clean-up code cleaner

Speed up storage indexing

Current storage indexing stalls the archive db to such an extent it can no longer keep up with a polkadot node

Store deltas instead of all storage for each block

Consider Different method of Batch Inserts

There's a gist: https://gist.github.com/mehcode/c476922be0290a4f8502d18701cc8c74#file-example-rs
that defines a structures way to batch-insert with SQLx. Currently, however, batch inserts seem to be working fine with prepare_sql.rs and database.rs. The downside to the current way is that it may require more boilerplate code.

I think benchmarks would need to be created to see if there is any real difference in performance (don't think there is). The other option is to just wait until an official QueryBuilder is integrated into Sqlx.

Panic while shutting down

Just ran into this:

    Running `target/release/polkadot-archive -c my_conf.toml --chain=polkadot`
[12:44] INFO Running migrations for postgres://postgres:123@localhost:5432/polkadot-db
[12:44] INFO Open RocksDB at /home/dvdplm/.local/share/polkadot/chains/polkadot/db, state column budget: 3686 MiB, others(11) column cache: 41 MiB
[12:44] INFO Native Version: NativeVersion { runtime_version: RuntimeVersion { spec_name: RuntimeString::Borrowed("polkadot"), impl_name: RuntimeString::Borrowed("parity-polkadot"), authoring_version: 0, spec_version: 23, impl_version: 0, apis: [([223, 106, 203, 104, 153, 7, 96, 155], 3), ([55, 227, 151, 252, 124, 145, 245, 228], 1), ([64, 254, 58, 212, 1, 248, 149, 154], 4), ([210, 188, 152, 151, 238, 208, 143, 21], 2), ([247, 139, 39, 139, 229, 63, 69, 76], 2), ([175, 44, 2, 151, 162, 62, 109, 61], 3), ([237, 153, 197, 172, 178, 94, 237, 245], 2), ([203, 202, 37, 227, 159, 20, 35, 135], 2), ([104, 122, 212, 74, 211, 127, 3, 194], 1), ([171, 60, 5, 114, 41, 31, 235, 139], 1), ([188, 157, 137, 144, 79, 91, 146, 63], 1), ([55, 200, 187, 19, 80, 169, 162, 168], 1)], transaction_version: 5 }, can_author_with: {} }
[12:44] INFO Running archive for chain `polkadot` 🔗, implemention `parity-polkadot`. Latest known runtime version: 25. Latest finalized block 0xadb0…5248 🛡️
[12:44] INFO 🧘 Restoring missing storage entries... (This could take a few minutes)
[12:44] INFO 1 missing blocks
[12:44] INFO Took 810.840108ms to load 1 blocks
[12:44] INFO Inserted
[12:44] INFO Took 3.041877897s to load 283465 blocks
[12:44] INFO Getting metadata for hash 08aad1540d95e5aedf7dc65c627b4c7094626b78512a74ce482a82fd1beca5fc, version 25
[12:44] INFO NATIVE CALL true
[12:44] INFO Contextual CALL, executor
[12:44] INFO NATIVE CALL true
[12:44] INFO Contextual CALL, executor
^C[12:44] INFO Restoring 26 missing storage entries...
[12:44] INFO Storage restored
[12:44] INFO Traces dying
[12:44] INFO 🧘 waiting for last storage insert...
thread '<unnamed>' panicked at 'index out of bounds: the len is 0 but the index is 0', /home/dvdplm/work/substrate-archive/src/actors/workers/database.rs:104:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[12:44] ERROR One of the pooled db actors has disconnected. could not send message.
thread '<unnamed>' panicked at 'One Shot', /home/dvdplm/work/substrate-archive/src/actors/actor_pool.rs:118:65
thread '<unnamed>' panicked at 'task has failed', /home/dvdplm/.cargo/registry/src/github.com-1ecc6299db9ec823/multitask-0.2.0/src/lib.rs:166:55

This was on the state-tracing branch and I hit Ctrl-c to stop the daemon. Had to kill -9 it to get it to exit.

GraphQL support

Desub Decoding of State Tracing Values

State Tracing Values come out of the format described here: https://gist.github.com/insipx/8554729c330aa522c6bc54859ae57c21

The key and value keys are encoded, and should be decoded accordingly to extract useful information that can be used for querying

Cache Structure

Now that both actor generators (missing_blocks and missing_storage) are both hitting the rocksDB database by getting .block(), it is worth experimenting with a cache structure on the custom backend to see if there is any more speedup

Change to type-checked SQL queries with `query` and `query_as` macro

SQLx includes compile-time verified queries with query!() and query_as!(). It would be good to refactor any queries that currently use the sqlx::query methods to the compile-time-verified variants. This will give us assurance that the queries are semantically correct, as well as work correctly with the db schema at compile time.

query!()
query_as!()

Export to CSV

Create Example: CLI

Create a CLI from which to launch archive db from a supplied TOML configuration file for the Kusama chain.

Wasm Overwrite Block Execution

Right now Archive uses a 'custom' executor. We should investigate to instead to use the LocalCallExecutor defined in substrate. This allows local WASM overwrite and to run state-tracing binaries in archive, and reduces code-duplication.

This lets us remove src/backend/frontend/executor.rs (mostly code copied from substrate-client before client refactor) and replace with the LocalCallExecutor. However LocalCallExecutor is currently private, so we would need to see if that can be made into a public API.

Some Calls Fail to Decode

Registrar.0 fails to decode despite decoding directly into the enum Call from polkadot-runtime.

Here is the Call enum being decoded into:

pub enum Call {
	System(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<System, Runtime>)
	,
	Babe(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<Babe, Runtime>)
	,
	Timestamp(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<Timestamp, Runtime>)
	,
	Indices(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<Indices, Runtime>)
	,
	Balances(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<Balances, Runtime>)
	,
	Authorship(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<Authorship, Runtime>)
	,
	Staking(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<Staking, Runtime>)
	,
	Offences(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<Offences, Runtime>)
	,
	Session(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<Session, Runtime>)
	,
	FinalityTracker(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<FinalityTracker, Runtime>)
	,
	Grandpa(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<Grandpa, Runtime>)
	,
	ImOnline(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<ImOnline, Runtime>)
	,
	Democracy(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<Democracy, Runtime>)
	,
	Council(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<Council, Runtime>)
	,
	TechnicalCommittee(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<TechnicalCommittee, Runtime>)
	,
	ElectionsPhragmen(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<ElectionsPhragmen, Runtime>)
	,
	TechnicalMembership(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<TechnicalMembership, Runtime>)
	,
	Treasury(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<Treasury, Runtime>)
	,
	Claims(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<Claims, Runtime>)
	,
	Parachains(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<Parachains, Runtime>)
	,
	Attestations(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<Attestations, Runtime>)
	,
	Slots(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<Slots, Runtime>)
	,
	Registrar(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<Registrar, Runtime>)
	,
	Sudo(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<Sudo, Runtime>)
	,
	Nicks(IntellijRustDollarCrate_paint_support::dispatch::CallableCallFor<Nicks, Runtime>)
	,
}

And the Trace Output:

Error decoding signature ::src/extrinsics.rs;64
[2019-11-22][15:44:11] [polkadot_archive][WARN] Did not decode with current Signature, trying AnySignature Err(Error("No such variant in enum MultiSignature")) ::polkadot-archive/src/main.rs;65
[2019-11-22][15:44:11] [substrate_archive::extrinsics][TRACE] bytes read from decoding extrinsic: [16, 1, FF, 48, 6F, 27, 37, D9, 33, BC, 6D, B4, FE, CB, A5, 53, 77, D7, 89, DE, C3, F4, 8A, 96, 3B, 70, 11, 46, 55, C7, 90, E2, 56, E0, 6E] ::src/extrinsics.rs;73
[2019-11-22][15:44:11] [substrate_archive::extrinsics][ERROR] Codec(Error("Error decoding field Call :: Registrar.0")) ::src/extrinsics.rs;273
[2019-11-22][15:44:11] [substrate_archive::extrinsics][DEBUG] Decoding Extrinsic in block: 278 ::src/extrinsics.rs;232

Doesn't work when using the concrete Registrar type from polkadot-runtime. Possibility of error when deriving Decode in decl_module macro?

Reconsider frame-system

Could remove the frame-system dependency and rely on the traits in sp-runtime

Handle Postgres Database Restart

If the Postgres Database is restarted while substrate-archive is running, archive will panic because it can no longer insert/read the postgres database

Oct 23 16:14:43 erin-test-archive-10 substrate-archive[8788]: [16:14] INFO ping on idle connection returned error: error returned from database: terminating connection due to administrator command
Oct 23 16:14:43 erin-test-archive-10 substrate-archive[8788]: thread 'coil-3' panicked at 'Failed to delete job: Database(PgDatabaseError { severity: Fatal, code: "57P01", message: "terminating connection due to administrator command", detail: None, hint: None, position: None, where: None, schema: None, table: None, column: None, data_type: None, constraint: None, file: Some("postgres.c"), line: Some(2957), routine: Some("ProcessInterrupts") })', /home/insipx/.cargo/git/checkouts/coil-37907adb63dfed18/bb552d7/coil/src/runner.rs:386:34
Oct 23 16:14:43 erin-test-archive-10 substrate-archive[8788]: note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
Oct 23 16:14:43 erin-test-archive-10 substrate-archive[8788]: Rayon: detected unexpected panic; aborting
Oct 23 16:14:43 erin-test-archive-10 systemd[1]: substrate-archive-polkadot.service: Main process exited, code=killed, status=6/ABRT

We should develop some kind of strategy for recovering from faults that happen on the postgres side. ideally and from a resiliency/redundancy perspective I think we should keep running, but I'm not sure if there are other things we could integrate to mitigate such issues

Clean up clippy warnings and errors

Right now the CI check Rust / Lints always fails on historical code due to clippy complaining. In order to expunge those errors from the CI check and allow meaningful use of clippy in development, those warnings should be silenced or the code should be corrected.

State Tracing Actor

rel #92

Actor should use the tracing rust/tokio-team crate and tracing substrate libraries to consume traces, format them, and send them to the database actor for insertion.

Rename `Error` to `ArchiveError` to disambiguate

#108 (comment)

Don't need to import Kusama Runtime

Kusama Runtime is re-exported by Polkadot service

Encapsulate the substrate backend

src/backend/ should be moved into it's own crate. This crate should only expose the Client and ReadOnlyBackend that substrate-archive can use to work with.

ReadOnlyDatabase should be abstracted into an archive-specific trait that can be implemented for different database types. rel #76. This trait can then be included in ReadOnlyBackend instead of the currently-included struct. This would make the backend-database implementation opaque and generic to substrate-archive, allowing users to use whichever database they like.

The trait should:

• read key/value pairs of the database (get)
• iterating over all blocks in the database (iter)
• catch up with the latest information added to the database (catch_up_with_primary)

Look in

substrate-archive/src/backend/database.rs

Line 48 in 70c6d9d

impl ReadOnlyDatabase {

for the specific functionality required. This trait should replace the struct in ReadOnlyBackend defined here:

substrate-archive/src/backend/read_only_backend.rs

Line 49 in 70c6d9d

db: Arc<ReadOnlyDatabase>,

This should also encapsulate out many of the substrate dependencies when working on stuff related to the SQL database/actors. The 'backend' crate (or whatever it will be named) should also expose a way of getting the Archive Client implementing the ApiAccess trait and related types (This ApiAccess trait should also be moved into the new backend crate). This is the function that does this now:

substrate-archive/src/backend/frontend.rs

Line 44 in 2f3b4ec

pub fn runtime_api<Block, Runtime, Dispatch>(

This could be split into two PR's, one for ParityDB support and the abstract ReadOnlyDatabase trait, and one for creating a separate crate for backend.

Abstract Away Boilerplate with ProcMacros

As the boilerplate in polkadot-archive grows, it is making more sense to Abstract it away into procedural macros.

This way the user can just input/import current (and past! Backwards-compatible) runtime types and have all the implementors derived on the Runtime and Wrapper structs.

This also follows with substrate and ink! So it feels familiar to the user

Better Type Handling

The plague of this project so far was the way in which it decodes and serializes types received SCALE-encoded from the Substrate RPC API lib. This is what causes #21 and much of the boilerplate in main.rs.

Currently all types are statically assigned in polkadot-archive via the System trait. This works alright if no types have changed via Runtime upgrades, but becomes tricky if a type is changed in the runtime, and the System trait still refers to the older type. This causes the types not to decode correctly. Problems like this are can often be difficult to debug, requiring changes in projects other than substrate-archive.

A thorough interface which keeps track of runtime types, as well as which blocks the types are valid for based upon the metadata spec, and how to decode/serialize them would solve this issue. I have been doing some experiments here: https://github.com/insipx/desub/tree/type-metadata to create a shim between the RuntimeMetadataPrefixed and the type-metadata crate: https://github.com/type-metadata/type-metadata. It would be great if you could simply 'register' types in some sort of data structure (Eventually, when/if type-metadata is incorporated into substrate, this step will go away), and then this shim would be able to decode/serialize the types accordingly.

A similar solution is found in https://github.com/paritytech/substrate-subxt (events.rs) which uses the length of types as a hint as to what the real type is. However, this isn't totally generic once a type not accounted for comes along

Such an API may look like this:

// Polkadot Archive
// register_type::<T>(module: String, type: String, runtime_version: u32);
metadata.register_type::<RewardDestination>("SomeModule", "T::Reward", 1025);
metadata.register_type::<u64>("System", "T::BlockNumber", 1025);

These type assignments would have deltas, so e.g if you have a type T::BlockNumber that is the same and does not change for runtime versions 1000-1025, you would only need to declare it once in spec 1000. If it changes in 1026, then it would be declared in 1026 again.

// Substrate Archive
// Could Enable API's similar to:
let timestamp_call = twox("Timestamp Now")
let timestamp = rpc.typed_storage_call(storage_key).await;

// Or
let extrinsic = metadata.decode("T::Extrinsic");

This has the drawback of having to describe all the types from polkadot-runtime versions in polkadot-archive. However, it is the most robust and flexible way of decoding Storage/Call/Event types

Create a wiki

describe how to use an archive db with a substrate node effectively

Sync .rustfmt.toml with parity-common

Enforce a "tools team" rustfmt.

Perhaps the same as jsonrpsee (which comes from parity-common):

hard_tabs = true
max_width = 120
use_small_heuristics = "Max"
edition = "2018"

Originally posted by @dvdplm in #108 (comment)

Support ParityDB

Try using static generics instead of dynamic dispatch for ReadOnlyBackend

it might be worth using static generics here:

https://github.com/paritytech/substrate-archive/pull/108/files/1fc2c7c47b230b43510de8fef82967ae02bbaa5a#diff-26082dae258bbbb4883f22123d203832ebce121a5ab81a999f8288be84c323bbR48

Dynamic dispatch means jumping through vtables and it could be more performant to use static generics instead

Make sure spec + chain matches with running node

It's possible a user could run polkadot/westend/kusama archive in a mode for EX kusama when their full node is running westend. Substrate-archive will then try executing westend blocks in the context of Kusama. Should add a check in substrate-archive whether the passed-in spec matches with the spec fetched from RPC.

Implement Changes-Trie Layer

Related to #52 #31

use changes tries without importing entire client. This feature would be experimental. It is preferable to import sc-client-db or sp-state-machine where most of the changes-trie functionality is already maintained.

Utilize Multithreading for Key Prefixes

Multithread indexing of storage

Each prefix should get it's own futures context to wait for rocksdb IO and trie-fetches to finish.

Rel #52 #53

Database Migrations

Integrate database migrations into substrate-archive with refinery https://github.com/rust-db/refinery

Handle Signals

Add signal handling

this should send a broadcast message to all actors to finish up their work and shutdown

Memory Grows Without Limit when ran for a long time (rocksdb)

Especially as the RocksDB database is filled up when syncing with a node that is far ahead, substrate-archive will slowly increase in memory-usage until the primary instance (polkadot/kusama/etc) is restarted + substrate-archive is restarted.

I believe this to be related in some way to: facebook/rocksdb#4112

Since substrate-archive is a secondary instance, max_open_files is set to -1 (no limit). It will keep all files open and force allocations. We should maybe try playing with the LRUCache options, MALLOC_ARENA_MAXand other rocksdb settings to try and mitigate this.

Speed up Storage Indexing

There are some optimizations which can be done in order to speed up current storage indexing.

Ideas:

• Could subscribe to storage in conjunction with query_storage from the client.

  • Requires a new actor that would search for 'missing' storage entries. This is more challenging than 'missing blocks' since there's no easy way to determine what storage is missing without heavily querying the Substrate Chain, which defeats the purpose. It could be possible to figure out missing storage entries from the database, but this seems like a flawed implementation.

• Rework query_storage

  • not sure how I would go about this, don't have a solid plan

• [Currently working on] Don't use the Substrate Client to query for storage. Instead, create a thin wrapper around the State Backend, ReadOnlyDatabase, acting directly with the RocksDB Secondary Instance. This requires research around how substrate chooses keys/prefixes in RocksDB.

  • Using RocksDB directly has the advantage of reaping the benefits of the new Linux IO_URING interface with MultiGet. Async Rocksdb Gets have the potential to significantly decrease the latency of requests.

From RocksDB Wiki:

There is a lot of complexity in the underlying RocksDB implementation to lookup a key. The complexity results in a lot of computational overhead, mainly due to cache misses when probing bloom filters, virtual function call dispatches, key comparisons and IO. Users that need to lookup many keys in order to process an application level request end up calling Get() in a loop to read the required KVs. By providing a MultiGet() API that accepts a batch of keys, it is possible for RocksDB to make the lookup more CPU efficient by reducing the number of virtual function calls and pipelining cache misses. Furthermore, latency can be reduced by doing IO in parallel.

Let us consider the case of a workload with good locality of reference. Successive point lookups in such a workload are likely to repeatedly access the same SST files and index/data blocks. For such workloads, MultiGet provides the following optimizations -

• When options.cache_index_and_filter_block=true is set, filter and index blocks for an SST file are fetched from the block cache on each key lookup. On a system with many threads performing reads, this results in significant lock contention on the LRU mutex. MultiGet looks up the filter and index block in the block cache only once for a whole batch of keys overlapping an SST file key range, thus drastically reducing the LRU mutex contention.
• In steps 1, 2 and 3c, CPU cache misses occur due to bloom filter probes. Assuming a database with 6 levels and most keys being found in the bottommost level, with an average of 2 L0 files, we will have ~6 cache misses due to filter lookups in SST files. There may be an additional 1-2 cache misses if memtable bloom filters are configured. By batching the lookups at each stage, the filter cache line accesses can be pipelined, thus hiding the cache miss latency.
• In a large database, data block reads are highly likely to require IO. This introduces latency. MultiGet has the capability to issue IO requests for multiple data blocks in the same SST file in parallel, thus reducing latency. This depends on support for parallel reads in the same thread from the underlying Env implementation. On Linux, PosixEnv has the capability to do parallel IO for MultiGet() using the IO Uring interface. IO Uring is a new asynchronous IO implementation introduced in the Linux kernel starting from 5.1.

https://github.com/facebook/rocksdb/wiki/MultiGet-Performance

This seems like it fits the use-case of storage queries particularly well. We are both getting lots of keys, and then looking up the data for that key individually

Blocked on:

• rust-rocksdb doesn't yet wrap 'MultiGet'. So modifying rust-rocksdb wrapper and then kvdb-rocksdb parity wrapper over rust-rocksdb is required
• might be a simpler solution by just using get_iter instead, since Iterator->Next in RocksDB executes in constant time

On start, storage inserts violates block_hash foreign key

This does not occur often, and seems to be a race condition when indexing storage gets too far ahead of indexed blocks. Normally, these storage entries should be deferred until the blocks are inserted into the blocks table in order to satisfy the foreign key constraint. However, that does not seem to be happening here:

[2020-05-25][15:01:30] [substrate_archive::database][ERROR] Database(PgError(Response { severity: Error, code: "23503", message: "insert or update on table \"storage\" violates foreign key constraint \"storage_hash_fkey\"", detail: Some("Key (hash)=(\\x04ad5ddeadf4c80d39a658e150a05293cab628365efcb806a41b4975517f3146) is not present in table \"blocks\"."), hint: None, position: None, internal_position: None, internal_query: None, where_: None, schema: Some("public"), table: Some("storage"), column: None, data_type: None, constraint: Some("storage_hash_fkey"), file: Some("ri_triggers.c"), line: Some(2772), routine: Some("ri_Report

More Flexible/Generic Module System

Currently all srml modules are hard-coded as enums for querying into Storage or for decoding calls. This is to give these calls representations in the Postgres DB that may be useful for querying. However, this approach is fairly constrained since it limits the users ability to create custom runtime modules, as they would need to add implementations of SrmlExt on their own which is disallowed by the Rust Orphan Rule

Do not hardcode substrate-archive local directory path

Archive puts it's logs in ~/.local/share/substrate_archive by default, and doesn't offer any way to specify it's own data directory where it can store rocksdb_secondary/paritydb data or logs. It would be nice to have this as a configurable in the ArchiveBuilder.

Also there are some places where we use the same path but not the same method, EX:
https://github.com/paritytech/substrate-archive/blob/master/src/archive.rs#L165

https://github.com/paritytech/substrate-archive/blob/master/src/util.rs#L48

Don't pull in all of Client

The client interface (src/backend/client.rs) currently pulls many substrate dependencies that we don't need IE Transaction pooling, execution, rpc.
sc-client is in the process of being refactored and transitioned into sc-service
where a method 'new_client' will create a much 'slimmer' database-backed client
that won't require defining G and E, a chainspec, or pulling in a async-runtime (async-std in this case)

Switch to thiserror

Failure crate is deprecated, use https://github.com/dtolnay/thiserror for library-like error handling

Handle the case when substrate-archive is significantly ahead of a running chain

If a user wants to re-sync from genesis and runs a user_chain purge-chain but wants to keep indexed data in Postgres, substrate archive will complain about missing state for blocks it wants to execute. Instead, the archive shouldn't execute any blocks or index any state if it detects a chain that is far behind it.

Substrate archive could just exit, but it would be better if it could be left waiting until the running chain catches up.

Optimizations

• Don't starve RPC of connections -- spawn() less and move functions into one future via merge/join_all/join3/etc
  • related to "Too many open files" and "Connection closed before handshake could complete"

Raise the fdlimit if we detect `too many open files`

This crate: https://github.com/paritytech/fdlimit could help quell the common issue users run into when starting substrate archive for the first time, the too many open files error. This error that occurs as a result of the secondary rocksdb database requiring all files of the primary database to be kept open.

Async/Await

grep 'TODO'
there are quiet a few areas that deal with anticipated features for async/await

[TypeDecode] Extrinsic + Storage filtering based on user-inputs

Should allow users to optionally specify which storage items/extrinsics from specific pallets. This could be a precursor to a more advanced filtering strategy.