qdrvm / kagome Goto Github PK

Kagome - C++20 implementation of Polkadot Host

Home Page: https://kagome.readthedocs.io

License: Apache License 2.0

CMake 2.44% C++ 95.88% C 0.21% Shell 0.36% Dockerfile 0.20% Makefile 0.36% Python 0.22% JavaScript 0.19% TypeScript 0.02% WebAssembly 0.12%

polkadot cpp libp2p wasm cpp20

kagome's Introduction

Intro

KAGOME is a Polkadot Host (former Polkadot Runtime Environment) developed by Quadrivim and funded by a Web3 Foundation grant and Treasury proposals ( 1, 2).

Status

More details of KAGOME development can be found within the supported features section and in projects board

Getting Started

Build

Prerequisites

If you are using a Debian Linux system, the following command allows you to build KAGOME:

git clone https://github.com/qdrvm/kagome
cd kagome
sudo chmod +x scripts/init.sh scripts/build.sh

sudo ./scripts/init.sh
./scripts/build.sh

You will get KAGOME binary in the build/node/ folder

Other make commands are:

make docker
make command args="gcc --version"
make release
make release_docker
make debug_docker
make clear

Using KAGOME

Obtaining database snapshot (optional)

In order to avoid syncing from scratch we are maintaining the most recent snapshot of Polkadot network for KAGOME node available for anyone here: https://drive.google.com/drive/folders/1pAZ1ongWB3_zVPKXvgOo-4aBB7ybmKy5?usp=sharing

After downloading the snapshot you can extract it in the folder where the node will be running:

unzip polkadot-node-1.zip

Execute KAGOME Polkadot full syncing node

You can synchronize with Polkadot using KAGOME and obtain an archive node that can be used to query the Polkadot network at any state.

To launch KAGOME Polkadot syncing node execute:

cd examples/polkadot/
PATH=$PATH:../../build/node/
kagome --chain polkadot.json --base-path polkadot-node-1

Note: If you start KAGOME for the first time, you can add the --sync Fast flag to synchronize using Fast sync

After this command KAGOME will connect with other nodes in the network and start importing blocks. You may play with your local node using polkadot js apps: https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A9944#/explorer

You will also be able to see your node on https://telemetry.polkadot.io/. If you need to identify it more easily you can add --name <node-name> flag to node's execution command and find your node in telemetry by typing its name.

Run kagome --help to explore other CLI flags.

Execute KAGOME block_validator node in development mode

The easiest way to get started with KAGOME is to run it in development mode, which is a single node network:

kagome --dev

That executes node with default accounts Alice and Bob. You can read about these accounts here.

To launch with wiping existing data you can do:

kagome --dev-with-wipe

Run KAGOME node in validator mode

To start the KAGOME validator:

cd examples/first_kagome_chain
PATH=$PATH:../../build/node/
kagome --validator --chain localchain.json --base-path base_path

This command executes a KAGOME full node with an authority role.

Run KAGOME with collator

Read this tutorial

Configuration Details

To run a KAGOME node, you need to provide it with a genesis config, cryptographic keys, and a place to store db files.

Example of a genesis config file can be found in examples/first_kagome_chain/localchain.json
Example of a base path dir can be found in examples/first_kagome_chain/base_path
To create database files, just provide any base path into kagome executable (mind that start with authority role requires keys to start).

Contributing Guides

Please refer to the Contributor Documentation.

Modules

api
- JSON-RPC based on specification from PSP and Polkadot JS documentation. Uses HTTP and WebSockets for transport.
application
- Implements logic for running KAGOME node, such as processing CLI flags and defining an execution order of different modules
assets
- Artifacts needed to run in development mode
authority_discovery
- Logic for finding peer information by authority id provided
authoriship
- Mechanism for building blocks from extrinsics provided by the transaction pool
blockchain
- Implements blockchain logic such as fork handling and block digest information processing
clock
- Implements a clock interface that is used to access the system time
- Implements timer interface that is used to schedule events
common
- A set of miscellaneous primitives, data structures, and helpers that are widely used in the project to simplify development
consensus
- Implementation of BABE block production mechanism
- Implementation of Grandpa finality gadget
containers
- An implementation of a container that serves as an allocator for frequently created objects
crypto
- Crypto primitives implementation such as DSAs, Hashers, VRF, Random generators
filesystem
- A convenient interface for working with directories in the project
host_api
- Host APIs exposed by the Polkadot runtime as WASM import functions needed to handle storage, cryptography, and memory
injector
- A mechanism for matching class interfaces to their corresponding implementations using the Boost dependency injector
log
- A configuration of KAGOME logger
macro
- Convenience macros
metrics
- Prometheus metrics to retrieve KAGOME node execution statistics
network
- A set of networking substream protocols implementation on top of cpp-libp2p library
offchain
- Offchain worker implementation
outcome
- Convenient functions for boost outcome
parachain
- Parachains logic such as collation protocol, backing, availability and validity
runtime
- Integration of Binaryen and WAVM WebAssembly engines with Host APIs
scale
- Scale codec for some primitives
storage
- Storage and trie interfaces with RocksDB implementation
subscription
- Subscription engine
telemetry
- Connection to telemetry such as https://telemetry.polkadot.io/
transaction_pool
- Pool of transactions to be included into the block
utils
- Utils such as profiler, pruner, thread pool

You can find more information about the components by checking reference documentation. Check out tutorials and more examples in official documentation: https://kagome.readthedocs.io/

KAGOME in media

Press-release: Soramitsu to implement Polkadot Runtime Environment in C++
KAGOME: C++ implementation of PRE presentation at DOTCon (18.08.19)
KAGOME and consensus in Polkadot presentation (in Russian) during Innopolis blockchain meetup (28.10.19)
Web3 Builders: Soramitsu | C++ Implementation of Polkadot Host Polkadot's Web3 builders online presentation
Building alternative clients Polkadot Decoded 2023 presentation

kagome's People

Contributors

Stargazers

Watchers

kagome's Issues

Web sockets support in libp2p

Web sockets

We should be able to connect to the libp2p nodes that use websockets as transport. Their multiaddress can be defined as follows: /dns/kusama-bootnode-0.paritytech.net/tcp/30334/ws/p2p/QmTFUXWi98EADXdsUxvv7t9fhJG1XniRijahDXxdv1EbAW

Implement ws connection (implement RawConnection interface) ``` - ☐

Please note while implementing https://github.com/libp2p/go-ws-transport#security-and-multiplexing

Implement ws transport adaptor (Implement transport adaptor interface)

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-51-4

Network protocol

Kademlia peer discovery
#388
Web sockets support in libp2p (low priority)
#387
Ephemeral streams
#407

Aha! Link: https://soramitsucoltd.aha.io/features/PRE-51

New key storage

https://hackmd.io/EH0N9hB2SxCDDhBSVmAWxQ

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-43-10

Implement Polkadot Trie Cursor

Implement a face::MapCursor for Polkadot Trie.

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-43-9

Update runtime api tests by loading runtime

However, another thing that just came up on our side is that the current hostapi testsuite for kagome calls the hostapi directly and not from within wasm. That way we will also test the wasm interaction with external functions on your host. The runtime for this can be provided as a static C/C++ export symbol or as a binary file and can be currently found here.

How would you go about loading a runtime in kagome and then call "into" it? I moved the setup of the wasm environment into a helper, so that will be a part that needs updating. Secondly we will have to update all the calls extension->ext_... to actually call a runtime function (i.e. prefixed with rtm_).

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-52-5

Ephemeral streams

A Polkadot Host node should open several substreams. In particular, it should periodically open ephemeral substreams in order to:

ping the remote peer and check whether the connection is still alive. Failure for the remote peer to reply leads to a disconnection. This uses the libp2p ping protocol specified in [lab19].
ask information from the remote.
send Kademlia random walk queries. Each Kademlia query is done in a new separate substreams. This

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-51-5

Noise security protocol in libp2p

https://github.com/libp2p/specs/tree/master/noise

Add cmake to the c-noise library
Get familiar with spec
Go-libp2p compatible key generation
Go-libp2p compatible x25519 shared secret generation
cha-cha cipher
Handshake not done

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-51-1

Grandpa update

When a Polkadot node (re)joins the network it requests the history of state transition which it is missing in form of blocks. To achieve that catch up algorithm should be implemented

Make authority changes
#439
Add catch up
#398
Test authority changes
#468

Aha! Link: https://soramitsucoltd.aha.io/features/PRE-64

Polkadot JS compatibility

Support for Polkadot JS in Kagome will help community become familiar with the C++ Polkadot Host. It will also be easier to prepare and run tutorials similar to this

Aha! Link: https://soramitsucoltd.aha.io/features/PRE-58

system_health

https://github.com/w3f/PSPs/blob/psp-rpc-api/psp-002.md#system_health

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-44-9

SASSY/BADASS Babe

color: Color value is invalid

BADASS BABE is a constant-time block production protocol. It intends to ensure that there is exactly one block produced with constant-time intervals rather than multiple or none. It extends on BABE to address this shortcoming of BABE.

Unix slots
Slot 0 should correspond to the the beginning of Unix epoch (1970-1-1, 00-00-00). Therefore when Babe is executed genesis slot should be calculated as /

Aha! Link: https://soramitsucoltd.aha.io/features/PRE-47

Graceful shutdown handling for Grandpa

Grandpa Launcher

atLaunch: start

atStop: stop liveness_checker

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-52-12

state_subscribeRuntimeVersion

Not specified yet

Example request {"id":5,"jsonrpc":"2.0","method":"state_subscribeRuntimeVersion","params":[]}
Example response {"jsonrpc":"2.0","result":"dz0irSby2Y18ptbD","id":5}

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-44-6

Update Host API

Web 3 refactored the Runtime APIs therefore, the APIs need to be updated in Kagome implementation

Aha! Link: https://soramitsucoltd.aha.io/features/PRE-60

Bootstraping

w3f/polkadot-spec#135

Aha! Link: https://soramitsucoltd.aha.io/features/PRE-50

ext_storage_next_key

Gets the next key in storage after the given one in lexicographic order. Use cursor for implementation

E.1.9.1 Version 1 - Prototype

(func $ext_storage_next_key_version_1 (param $key i64) (return i64))

Arguments :

key: a pointer-size as defined in Definition E.2 indicating the key.
return: a pointer-size as defined in Definition E.2 indicating the SCALE encoded Option as defined in Definition B.4 containing the next key in lexicographic order.

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-43-2

Add catch up

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-49-1

Graceful shutdown handling for p2p networking

RouterLibp2p:

Prepair: RouterLibp2p::init (before host.start())

Launch: RouterLibp2p::init (host.start())

Stop: invoke stop all listeners on the host

PeerList:

Add constructor from ConfigurationManager where in constructor we add atPrepare which creates peer list from boot nodes

Gossiper:

At stop: close all streams

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-52-11

System apis

After we updated genesis format in #494 required for #484 new system RPCs should be implemented:
-[ ] https://github.com/w3f/PSPs/blob/psp-rpc-api/psp-002.md#system_properties
-[ ] https://github.com/w3f/PSPs/blob/psp-rpc-api/psp-002.md#system_chain

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-44-4

Add catch up

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-49-1

ext_crypto_sr25519* externals

ext_crypto_sr25519_public_keys Returns all sr25519 public keys for the given key id from the keystore.
ext_crypto_sr25519_generate Generates an sr25519 key for the given key type using an optional BIP-39 seed and stores it in the keystore.
ext_crypto_sr25519_sign Signs the given message with the sr25519 key that corresponds to the given public key and key type in the keystore.
ext_crypto_sr25519_verify Verifies an sr25519 signature. Only version 1 of this function supports deprecated Schnorr signatures introduced by the schnorrkel Rust library version 0.1.1 and should only be used for backward compatibility.

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-43-5

runtime_version

Extract the Runtime version of the given Wasm blob by calling Core_version as defined in Definition G.2.1. Returns the SCALE encoded runtime version or None as defined in Definition B.4 if the call fails. This function gets primarily used when upgrading Runtimes.

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-43-7

ext_crypto_ed25519* runtime externals

ext_crypto_ed25519_public_keys Returns all ed25519 public keys for the given key id from the keystore.
ext_crypto_ed25519_generate Generates an ed25519 key for the given key type using an optional BIP-39 seed and stores it in the keystore.
ext_crypto_ed25519_sign Signs the given message with the ed25519 key that corresponds to the given public key and key type in the keystore.
ext_crypto_ed25519_verify Verifies an ed25519 signature.

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-43-4

state subscribe storage

That RPC subscribes to the changes of the certain key. Example:

----> `

{"id":13,"jsonrpc":"2.0","method":"state_subscribeStorage","params":[["0xcc956bdb7605e3547539f321ac2bc95c"]]}

todo: add response example

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-44-2

ext_crypto_secp256k1_ecdsa* externals

ext_crypto_secp256k1_ecdsa_recover Verify and recover a SECP256k1 ECDSA signature.
ext_crypto_secp256k1_ecdsa_recover_compressed Verify and recover a SECP256k1 ECDSA signature

sec1-v2.pdf (https://soramitsucoltd.aha.io/attachments/6838832318672616694/token/3c430b20d2b9af743116b77c3290ef7cc9855303fe7b886c710b5970ece804d8.download?size=original)

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-43-6

Test feature 2

descr2

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-56-2

Bump boost version to fix mac build

Latest Mac's command line tools do not allow building boost library version 1.72.0-p0, whereas version 1.72.0-p0 fixes that. Therefore boost's version bump is required
cpp-pm/hunter#180

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-52-7

Sr25519 (Continued...)

Hello!

To continue discussion of Sr25519, what is your plan about this library? Will you keep using linked Rust library or have plans to port it to C++ for better portability?

Kademlia peer discovery

Current Kademlia implementation in cpp-libp2p is not compatible with go- or rust-libp2p. This should be fixed.
Libp2p Kademlia spec: libp2p/specs#108
Compatibility can be checked by implementing this example: libp2p/cpp-libp2p#82

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-51-2

Test feature 1

descr1

Task1
Task2

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-56-1

Graceful shutdown handling for Babe

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-52-8

ext_logging_log_version_1

Request to print a log message on the host. Note that this will be only displayed if the host is enabled to display log messages with given level and target.

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-43-8

state_getMetadata

Not specified yet

Example request:
{id: 7, jsonrpc: "2.0", method: "state_getMetadata", params: []} id: 7
Example response:

{"jsonrpc":"2.0","result":"0x6817ac..,","id":7}

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-44-7

Offchain worker

Offchain workers are needed to process off-chain data before including it into the on-chain state. Offchain workers require their own WASM execution runtime separate from Polkadot Runtime. To make it work the following APIs should be supported:

Local storage: get and set
HTTP request
Random seed generation
Timestamp
Sleep

Aha! Link: https://soramitsucoltd.aha.io/features/PRE-48

Stop kagome execution if some required port is not available

To launch kagome we need to open 3 ports: http_rpc, ws_rpc, p2p.
During the launch we should check if all of these ports are available, otherwise report error and stop execution

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-52-6

Fix find merge point

https://gitlab.w3f.tech/grants/soramitsu/-/issues/19

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-52-26

DNS support for addresses in libp2p

DNS for addresses of the form /dns4/example.com/tcp/5 or /dns4/example.com/tcp/5/ws. A node's address can contain a domain name.

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-51-3

rpc_methods

RPC Methods returns the list of existing RPC methods

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-44-5

Append existing runtime APIs with _version1

All legacy runtime APIs were updated to have _version1 postfix. Refactor correspondingly

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-43-1

Add parsing configs from file

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-52-27

Run kagome from existing database

Currenly Kagome cannot be executed from the existing database. We need to fix that.

Possible way to implement the fix is the following (it is okay to come up with different solution):

We should always store the hash of the latest finalized block

If provided leveldb exists then the following changes to trie db should be done (in application injector)

Get merkle root of the latest finalized block
Create trie db from obtained merkle root

BlockStorage initialization

Add static fabric method create(storage::BufferStorage storage), which does the same as createWithGenesis, but does not create genesis block
In injector check if storage contains merkle root of the latest finalized state (from the latest finalized block). If root does not exist, then create block storage using createWithGenesis method (as implemented now). Otherwise create block storage using create method

Babe changes:

if (first_peer) { NextEpochDescriptor epoch_0_and_1_digest; epoch_0_and_1_digest.randomness = genesis_configuration_->randomness; epoch_0_and_1_digest.authorities = genesis_configuration_->genesis_authorities; epoch_storage_->addEpochDescriptor(0, epoch_0_and_1_digest); epoch_storage_->addEpochDescriptor(1, epoch_0_and_1_digest); }

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-52-28

Graceful shutdown handling for leveldb

Prepair: check if database exist, if so use existing database. Otherwise create new database

Launch: empty

Stop: close leveldb

Aha! Link: https://soramitsucoltd.aha.io/requirements/PRE-52-9

[DOCS] Clarify Overall Architecture

It i not clear to me what kagome does.

My understanding so far is that:

Polkadot runtime, the core-blockchain-and-dapp-system
kagome, the C++ interfacing code?

Is there possibly some architectural overview re polkadot and the "language related interfacing code"?

It looks like its more than the usual language-related "wrappers".

EDIT:

Can it be that kagome is more like an C++ implementation of substrate, at least parts of it?

E.1.3.1 Version 1 - Prototype

(func $ext_storage_read_version_1 (param $key i64) (param $value_out i64) (param $offset i32) (result i64))