dbpunk-labs/db3

a Lightweight, Permanent JSON document database

db3's Issues

make a demo to show using db3 as a decentralized backup for mysql

Motivations

Current most of dapps use centralized database to store user data and use arweave as a decentralized backup, Now we want use
db3 to replace arweave.

Solution

We can split this issue into two steps

• step 1, use rust sdk to execute SQL. Resolved by dbpunk-labs/substrate-api-client#1

• step2 ，subscribe mysql binlog and write all data to db3. dbpunk-labs/substrate-api-client#2

Reference

1. https://github.com/rusuly/mysql_cdc， subscribe mysql binlog
2. https://github.com/scs/substrate-api-client ，substrate client sdk

WIP: propose a decentralized live backup tool for mysql

The Problems

Web3 native applications need a decentralized data architecture, but it's not easy to build a fully decentralized data architecture, for web3 startups MySQL is still the first option, but MySQL is a centralized data architecture and then they have to take a lot of time to develop tools to synchronize partial data of MySQL to arweave

Why DB3 CDC

db3_cdc can help web3 developers to setup synchronizing all data of MySQL to DB3 out of box

• no extra tools to be developed
• all data of MySQL will be synchronized to db3 in realtime

Who can use DB3 CDC

1. web3 applications which have used MySQL as the backend database already

if you are looking for a fully decentralized data architecture, you can use db3 directly

How to use DB3 CDC

1. Download the release of db3 cdc from github according to your operation system, if you want run db3 cdc in linux

wget https://github.com/dbpunk-labs/db3cdc/releases/download/v0.2.0/db3cdc-v0.2.0-linux-x86_64.tar.gz
tar -zxf db3cdc-v0.2.0-linux-x86_64.tar.gz
cd ./db3cdc-v0.2.0-linux-x86_64/
./bin/db3cdc --help
Usage: db3cdc <COMMAND>

Commands:
  sync     Start a interactive shell
  version  Get the version of DB3 CDC
  help     Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help information

2. use db3cdc to replicate mysql to db3

./bin/db3cdc sync --db3-node-grpc-url https://grpc.devnet.db3.network\
 --password mysql_password \
 --user mysql_root_user \
 --master-host mysql_host \
 --master-port mysql_port

the devnet is unstable, you can change the grpc url to the following nodes in our testnet

• ap-southeast node, grpc http://18.142.114.145:26659
• ap-east node, grpc http://18.162.230.6:26659
• us-east node, grpc http://18.117.125.43:26659
• eu-west node, grpc http://13.41.65.17:26659

3. You will see some information

2022-11-21T04:47:29.038953Z  INFO db3cdc: no gtid state in db3
WARNING, db3 will generate private key and save it to /home/jackwang/.db3/user.key
restore the key with addr 0x97b478ac2bef8762c2ecc08417f35838c4bf73fc
2022-11-21T04:47:29.040835Z  INFO db3cdc: binlog options BinlogOptions { filename: "", position: 4, gtid_set: None, gtid_list: None, starting_strategy: FromStart }
2022-11-21T04:47:29.349037Z  INFO db3cdc: mutation id "tWYRruF8dSwvOySMypYCD0pJTVZgquKZu116UddTAps="
2022-11-21T04:47:29.429778Z  INFO db3cdc: Your account 0x97b478ac2bef8762c2ecc08417f35838c4bf73fc status: total bills 0.000000 db3, total storage used 0.00 , total mutation 0, credits 10 db3

these information includes

• your key path
• last synchronization progress
• your account bills status and credit

The plans

• #155
• #160

Reference

• https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_replication_binlog_event.html

support parsing ddl in frontend node

send request to meta node to create a new table

How can I make contribution

?

Add query session into system

Motivations

• Would you like to implement this feature? [y/n]

Solution

Alternatives

Additional context

Make a research on OpenRPC and Get a conlusion on how many do chain node use OpenRPC

[CICD]Fail to run rustfmt

Bug description

Error: The process '/home/runner/.cargo/bin/cargo' failed with exit code 1

Run actions-rs/cargo@v1
/home/runner/.cargo/bin/cargo fmt --all -- --check
`cargo metadata` exited with an error: error: failed to load manifest for workspace member `/home/runner/work/db3/db3/rpc`

Caused by:
  failed to load manifest for dependency `db3-primitives`

Caused by:
  failed to load manifest for dependency `frame-system`

Caused by:
  failed to read `/home/runner/work/db3/db3/thirdparty/substrate/frame/system/Cargo.toml`

Caused by:
  No such file or directory (os error 2)
cargo-fmt 
This utility formats all bin and lib files of the current crate using rustfmt.

USAGE:
    cargo fmt [OPTIONS] [-- <rustfmt_options>...]

ARGS:
    <rustfmt_options>...    Options passed to rustfmt

OPTIONS:
        --all
            Format all packages, and also their local path-based dependencies

        --check
            Run rustfmt in check mode

    -h, --help
            Print help information

        --manifest-path <manifest-path>
            Specify path to Cargo.toml

        --message-format <message-format>
            Specify message-format: short|json|human

    -p, --package <package>...
            Specify package to format

    -q, --quiet
            No output printed to stdout

    -v, --verbose
            Use verbose output

        --version
            Print rustfmt version and exit


Error: The process '/home/runner/.cargo/bin/cargo' failed with exit code 1

• Would you like to work on a fix? [y/n]

To Reproduce

Steps to reproduce the behavior:

1. Pull Request trigger the Rusfmt

Expected behavior

Screenshots

Environment

• OS: [e.g. Ubuntu 20.04]
• rtstore-tpl version: [e.g. 0.1.0]

Additional context

POC of dvm

Motivations

• Would you like to implement this feature? [y/n]

Solution

Alternatives

Additional context

WIP: Propose the gas system for mutation and query session

Key points

• why we need gas fee
• how much gas the mutation will cost
• how much gas the query session will cost
• the max gas per block - BlockGasMeter

Solution

Alternatives

Additional context

• the mutation https://github.com/dbpunk-labs/db3/blob/main/docs/mutation.md
• the query https://github.com/dbpunk-labs/db3/blob/main/docs/query.md

How did the others do

• the graph, Queries are priced in USD and paid for with GRT, Query costs may vary; quoted cost is the average at time of publication
• https://eips.ethereum.org/EIPS/eip-1559

the first version for demo

use mysql cli to create a table

mysql -h 127.0.0.1 -p 3306 -u root -P 
>create database eth;
>use eth;
>create table price{
ts timestamp,
price double
};
>insert into price value('2022-12-10 01-01', 1000);
>select * from price limit 1;

data flow of executing ddl

data flow of syncing meta

subtasks

• #24
• #22
• #23
• #40
• #41
• #42
• #43
• #27
• #31
• #33

reference

• https://github.com/hyperium/tonic/blob/master/tonic/src/codec/mod.rs

2022-Q3 Millestone 🚀

Topics

1. #53
2. Cloud-Native timeseries data management
3. Server-Less sql query engine
4. multi-protocol frontend
5. Lanuch DataDao community
6. Integration Testing
7. Integrated ETH

Reference

• https://www.sqlite.org/sqllogictest/doc/trunk/about.wiki

Add introduction to Readme

Motivations

• Would you like to implement this feature? [y/n]

Solution

Alternatives

Additional context

Design the Accounting System

Context

1. use db3 token to record storage consumption

FAQ

1. how much one byte will cost in db3

Design how to convert chain node openrpc spec to streaming table definitions

Data Model

let‘s look at the method eth_getBlockByHash from etherum openrpc.json

name	type
parentHash	string
sha3Uncles	string
miner	string
stateRoot	string
transactionsRoot	string
receiptsRoot	string
receiptsRoot	string
transactions	array
number	string
...	...

Query Block

select timestamp, miner, number from block where number in (15044280);

this sql will call json rpc

call eth_getBlockByNumber(15044280)

do some process

select date(`timestamp`),  miner, number from block where number in (15044280);

Query Transaction in block

select * from Values (select block.transactions where number in (15044280));

use datafusion to query table against json

let ctx = SessionContext::new();
    let path = format!("{}/2.json", TEST_DATA_BASE);
    ctx.register_json("t1", &path, NdJsonReadOptions::default())
        .await
        .unwrap();

    let sql = "SELECT a, b FROM t1";
    let actual = execute_to_batches(&ctx, sql).await;
    let expected = vec![
        "+-----------------+------+",
        "| a               | b    |",
        "+-----------------+------+",
        "| 1               | 2    |",
        "| -10             | -3.5 |",
        "| 2               | 0.6  |",
        "| 1               | 2    |",
        "| 7               | -3.5 |",
        "| 1               | 0.6  |",
        "| 1               | 2    |",
        "| 5               | -3.5 |",
        "| 1               | 0.6  |",
        "| 1               | 2    |",
        "| 1               | -3.5 |",
        "| 100000000000000 | 0.6  |",
        "+-----------------+------+",
    ];

    assert_batches_eq!(expected, &actual);

An error is reported when executing gen_proto.sh

protoc-gen-js: program not found or is not executable
Please specify a program using absolute path or make sure the program is available in your PATH system variable
--js_out: protoc-gen-js: Plugin failed with status code 1.

protoc version: 3.21.9

WIP: Propose the new merkled index storage engine

Motivation

Currently, db3 uses merk as merkled index storage engine.
the advantage of merk

• an ordered kv store, db3 use merk to query a single key or query data with a range of keys
• support proof of query, merk can prove the seeking operation and range query
• memory management according to the tree level, when the level of children nodes is greater than the limit, merk will remove the nodes from memory

the shortcomings

• AVL tree that merk uses is not good for persistence storage
• inactive community

the unsupported feature that db3 needs

• namespace, in db3 every account is a namespace and every namespace has its root hash

the namespaced Merkle tree is proposed by Celestia in the LazyLedger academic paper.so I want to propose a new merkled index storage engine called merkdb based on merk and merkdb has the following improvements

• replace AVL tree with b+tree to improve the performance on persistence storage, the berkletree has made some good experiments
• support namespace level root hash

and you can consider merkdb as rocksdb in crypto

The Instruction set of the storage engine

The poof of query

• proof of membership
• proof of range
• proof of efficiency

Reference

• https://github.com/cberner/redb/blob/master/docs/design.md
• https://github.com/erthink/libmdbx
• https://github.com/vorot93/libmdbx-rs
• Verkle tree a bandwidth-efficient alternative to Merkle Tree

Refactor: move all libraries to src dir

Motivations

Move all libraries to src dir

• Would you like to implement this feature? [y/n]
  Y

Solution

Alternatives

Additional context

Support reqId for newly added pallet interface

Motivations

Support req_id in the newly added pallet calls API

• create_ns
• create_ns_and_add_delegate
• add_delegate
• delete_delegate

Solution

Alternatives

Additional context

Design the management of streaming task on metanode

sync onchain data node to timeseries store

Motivations

sync onchain data node to timeseries store

Solution

# start rtstore
docker run -p 9292:9292 -dt ghcr.io/rtstore/rtstore:0.1.0
#  use cli to connect to rtstore
mysql -h localhost -P 9292 -u root
# create tables
>create tables WITH ("blocknode_urls"="ws://xxxx","contract"="0x0a267cF51EF038fC00E71801F5a524aec06e4f07", "blockchain"="eth","auto_gen"="true");
# analysis the data 
> select * from xxx_event limit1;

Alternatives

Additional context

Sub Tasks

• streaming etl research

implement block streaming source

persistence table meta to etcd

Add roadmap to readme

Motivations

• Would you like to implement this feature? [y/n]

Solution

Alternatives

Additional context

[WIP] change type system to arrow schema to support json

Motivations

• Would you like to implement this feature? [y/n]

Solution

Alternatives

Additional context

support query in memory data

support simple query from compute node

[CICD]Code coverage checker fail

Bug description

• Would you like to work on a fix? [y/n]

Caused by:
  process didn't exit successfully: `/home/runner/work/db3/db3/target/llvm-cov-target/debug/build/substrate-test-runtime-3dc7e40855a606a9/build-script-build` (exit status: 1)
  --- stderr
  Rust WASM toolchain not installed, please install it!

  Further error information:
  ------------------------------------------------------------
     Compiling wasm-test v1.0.0 (/tmp/.tmpbRfrOO)
  error[E0463]: can't find crate for `std`
    |
    = note: the `wasm32-unknown-unknown` target may not be installed
    = help: consider downloading the target with `rustup target add wasm32-unknown-unknown`
    = help: consider building the standard library from source with `cargo build -Zbuild-std`

  error: cannot find macro `println` in this scope
   --> src/main.rs:3:5
    |
  3 |                 println!("{}", env!("RUSTC_VERSION"));
    |                 ^^^^^^^

  error: requires `sized` lang_item

  For more information about this error, try `rustc --explain E0463`.
  error: could not compile `wasm-test` due to 3 previous errors
  ------------------------------------------------------------

To Reproduce

Steps to reproduce the behavior:

1. ...
2. ...
3. ...
4. ...

Expected behavior

Screenshots

Environment

• OS: [e.g. Ubuntu 20.04]
• rtstore-tpl version: [e.g. 0.1.0]

Additional context

Initialize a development environment based on the kv storage example in tendermint-rs

Motivations

Initialize a development environment based on tendermint-rs

• add tendermint-rs to thirdparty
• add a test case showing how to test put a key to kvstore and get value from kvstore
• add a doc about how to initalize a local dev network

• Would you like to implement this feature? [y/n]

Solution

Alternatives

Additional context

wrap rust interface of dolt as db3 storage engine

implement a naive version of meta management system based on etcd

Etcd Path rules for rtstore

/rtstore
|
 \_tables/test.eth -> data // store meta for table 
|
 \_nodes_0/127.0.0.1_8989 -> data // store meta for frontend node
|
 \_nodes_1/127.0.0.1_9191 -> data // store meta for meta node
|
 \_nodes_2/127.0.0.1_9292 -> data // store meta for memory node
|
 \_nodes_3/127.0.0.1_7979 -> data // store meta for compute node

Unifiy the return data format

Motivations

let the frontend to handle response easily

Solution

GeneralResultEvent (Vec<u8>)
the data in GeneralResultEvent is a json string like

{
"status": 0,
"msg":"xxxx",
"schema":"xx",
"data":{},
"req_id":""
}

Alternatives

Additional context

support transform ddl to internal table descriputor

Use submodules to resolve error from checking dependencies when releasing rtstore

the list of dependencies to be updated

• arrow-datafusion
• sqlparser-rs
• pg_wire

Fail to build db3-runtime package

Bug description

• Would you like to work on a fix? [y/n]

To Reproduce

Steps to reproduce the behavior:

cargo build --package db3-runtime --release

Expected behavior

build package successfully

Screenshots

error[E0405]: cannot find trait `DBAccountApi` in crate `sqldb_rpc_runtime_api`
    --> runtime/src/lib.rs:1936:33
     |
1936 |     impl sqldb_rpc_runtime_api::DBAccountApi<Block, AccountId> for Runtime {
     |                                 ^^^^^^^^^^^^ not found in `sqldb_rpc_runtime_api`
     |
help: consider importing this trait
     |
25   | use sqldb_rpc_runtime_api::runtime_decl_for_DBAccountApi::DBAccountApi;
     |
help: if you import `DBAccountApi`, refer to it directly
     |
1936 -     impl sqldb_rpc_runtime_api::DBAccountApi<Block, AccountId> for Runtime {
1936 +     impl DBAccountApi<Block, AccountId> for Runtime {
     |

error[E0425]: cannot find function `is_ns_owner_call_api_at` in module `sqldb_rpc_runtime_api::runtime_decl_for_DBAccountApi`
    --> runtime/src/lib.rs:1774:1
     |
1774 | / impl_runtime_apis! {
1775 | |     impl sp_api::Core<Block> for Runtime {
1776 | |         fn version() -> RuntimeVersion {
1777 | |             VERSION
...    |
2169 | |     }
2170 | | }
     | |_^ not found in `sqldb_rpc_runtime_api::runtime_decl_for_DBAccountApi`
     |
     = note: this error originates in the macro `impl_runtime_apis` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0425]: cannot find function `is_ns_owner_native_call_generator` in module `sqldb_rpc_runtime_api::runtime_decl_for_DBAccountApi`
    --> runtime/src/lib.rs:1774:1
     |
1774 | / impl_runtime_apis! {
1775 | |     impl sp_api::Core<Block> for Runtime {
1776 | |         fn version() -> RuntimeVersion {
1777 | |             VERSION
...    |
2169 | |     }
2170 | | }
     | |_^ not found in `sqldb_rpc_runtime_api::runtime_decl_for_DBAccountApi`
     |
     = note: this error originates in the macro `impl_runtime_apis` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0425]: cannot find function `list_delegates_call_api_at` in module `sqldb_rpc_runtime_api::runtime_decl_for_DBAccountApi`
    --> runtime/src/lib.rs:1774:1
     |
1774 | / impl_runtime_apis! {
1775 | |     impl sp_api::Core<Block> for Runtime {
1776 | |         fn version() -> RuntimeVersion {
1777 | |             VERSION
...    |
2169 | |     }
2170 | | }
     | |_^ not found in `sqldb_rpc_runtime_api::runtime_decl_for_DBAccountApi`
     |
     = note: this error originates in the macro `impl_runtime_apis` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0425]: cannot find function `list_delegates_native_call_generator` in module `sqldb_rpc_runtime_api::runtime_decl_for_DBAccountApi`
    --> runtime/src/lib.rs:1774:1
     |
1774 | / impl_runtime_apis! {
1775 | |     impl sp_api::Core<Block> for Runtime {
1776 | |         fn version() -> RuntimeVersion {
1777 | |             VERSION
...    |
2169 | |     }
2170 | | }
     | |_^ not found in `sqldb_rpc_runtime_api::runtime_decl_for_DBAccountApi`
     |
     = note: this error originates in the macro `impl_runtime_apis` (in Nightly builds, run with -Z macro-backtrace for more info)

Some errors have detailed explanations: E0405, E0425.
For more information about an error, try `rustc --explain E0405`.
error: could not compile `db3-runtime` due to 5 previous errors

Environment

• OS: Mac OS 10.15.7

Additional context

support rolling the binlog when making a compaction

support database in frontend node

Design the streaming sql engine based on differential-dataflow

send request to memory node to create a new partition

let the frontend to get right response by adding reqid to pallet

Motivations

• Would you like to implement this feature? [y/n]

Solution

the method list

• create_ns
• create_ns_and_add_delegate
• add_delegate
• delete_delegate

how to return reqid?
add reqid to event that will be returned to frontent

Alternatives

Additional context

the implementation of query session

Overview

Details

Status

• Design the query session bill workflow and protocol
• Design SDK API #154
• Implement Session Manager #125
• Verify session info #162
• Implement Submit Session Query #231
• Remove session close/open/restart in cli. #163
• Implement new proposed proposal #175
• Support signature session info response in node. Blocked by move the path used to store user private key to ~/.db3/key #161

Alternatives

Additional context

the concept of query session

subscribe to the MySQL binlog and write it to db3

Support restarting abci and recover root hash and height

Motivations

• Would you like to implement this feature? [y/n]

Solution

Alternatives

Additional context

WIP: the design of javascript sdk

Motivations

• Would you like to implement this feature? [y/n]

Solution

Alternatives

Additional context

Support converting chain node json-rpc to table definitions

Move all functions for mysql to frontend node

Motivations

Move all functions for mysql to frontend node

• Would you like to implement this feature? [y/n]

Solution

Alternatives

Additional context

Make the compute node has not any logical for frontend node

Add current_user() and connection_id() to sql engine

Motivations

for mysql compatible

• Would you like to implement this feature? [y/n]

Solution

Alternatives

Additional context

Stop the process of db3 gracefully

Motivations

Avoid db3 to lost data in the system page cache

Solution

Alternatives

Additional context

Currently, only SIGKILL can be used to shutdown db3 and it's better to use SIGTERM to stop db3

Multi-chain Streaming ETL by Using sql to interact with json-rpc

Overview of ETL

Data Flow Of ETL

Design

• #57
• #61
• #62
• #63

Related Issues

• #67

Reference

• https://open-rpc.org/getting-started
• https://github.com/google/zetasql/blob/master/docs/data-types.md query on json

support submitting mutation in rpc

Motivations

the Javascript SDK needs to broadcast mutations to the DB3 network

• Would you like to implement this feature? [y/n]
  y

Desgin

How to sign the request

How to encode the request for json rpc

1. use protobuf to generate WriteRequest
2. encode WriteRequest to Uint8Array
3. encode Uint8Array as base64 String

sample rust code

        let json_rpc_url = "http://127.0.0.1:26670";
        let client = awc::Client::default();
        let kp = db3_cmd::get_key_pair(false).unwrap();
        let signer = Db3Signer::new(kp);
        let kv = KvPair {
            key: format!("kkkkk_tt{}", 1).as_bytes().to_vec(),
            value: format!("vkalue_tt{}", 1).as_bytes().to_vec(),
            action: MutationAction::InsertKv.into(),
        };  
        let mutation = Mutation {
            ns: "my_twitter".as_bytes().to_vec(),
            kv_pairs: vec![kv],
            nonce: 1110,
            chain_id: ChainId::MainNet.into(),
            chain_role: ChainRole::StorageShardChain.into(),
            gas_price: None,
            gas: 10, 
        };  
        let mut mbuf = BytesMut::with_capacity(1024 * 4); 
        mutation.encode(&mut mbuf).unwrap();
        let mbuf = mbuf.freeze();
        let signature = signer.sign(mbuf.as_ref()).unwrap();
        let request = WriteRequest {
            signature,
            mutation: mbuf.as_ref().to_vec().to_owned(),
        };  
        let mut buf = BytesMut::with_capacity(1024 * 4); 
        request.encode(&mut buf).unwrap();
        let buf = buf.freeze();
        // encode request to base64
        let data = base64::encode(buf.as_ref());
        let base64_str = String::from_utf8_lossy(data.as_ref()).to_string();
        let request = serde_json::json!(
            {"method": "broadcast",
            "params": vec![base64_str],
            "id": 1,
            "jsonrpc": "2.0"
            }   
        );  
        let mut response = client.post(json_rpc_url).send_json(&request).await.unwrap();
        if let serde_json::Value::Object(val) = response.json::<serde_json::Value>().await.unwrap() {
            if let Some(serde_json::Value::String(s)) = val.get("result") {
                assert!(s.len() > 0); 
            }else {
                assert!(false)
            }   
        }else {
            assert!(false)
        }   

How to encode the request for grpc

1. use protobuf to generate WriteRequest
2. encode WriteRequest to Uint8Array

How to use the rpc

sample request

curl --header "Content-Type: application/json" --request POST --data '{"method": "broadcast", "params": ["XADAFA="], "id": 1, "jsonrpc": "2.0"}' localhost:26670

Note: you must replace the params with valid one
sample response

{"id": 1, "jsonrpc": "2.0", "result": "bDJ9tsALNX42gR6UZ8+q/xh5hoacShtKcZj5mxeAmhU="}

How to start a localnet

this PR #151 has done some work

git clone [email protected]:dbpunk-labs/db3.git -b 149-support-submitting-mutation-in-json-rpc
cd db3 && cargo build && cd tools && bash start_localnet.sh

Alternatives

Additional context