casper-network / casper-db-utils Goto Github PK
View Code? Open in Web Editor NEWUtilities to perform operations on the lmdb databases for a casper-node.
License: Apache License 2.0
Utilities to perform operations on the lmdb databases for a casper-node.
License: Apache License 2.0
The tool needs to be able to download a snapshot from a configurable URL and unpack it to a configurable path
We should add a new subcommand which reads from a node's storage DB and prints data related to execution results per block.
The execution results which are stored in deploy_metadata_db
are indexed by deploy hash, but we want the results per block, so the tool will need to iterate each block in storage, and for every one which has deploys, build the collection of execution results.
The data to printed is:
BlockExecutionResultsOrChunk
Make a tool to call this migration and give to Joe to run on a realistic size. Add to db_utils.
See this ticket for context casper-network/casper-node#3849
Remove accumulated technical debt in casper-db-utils
, specifically:
--overwrite
parameterenv-logger
stdout
)Follow up on casper-network/casper-node#3921. The underlying problem is a dependency mismatch with casper-types and casper-node . More specifically, casper-types 1.6 (should be 3.0) had breaking changes compared to casper-types 1.5 . We should update the casper-db-utils dependencies to a stable state once the casper-types changes are sorted out.
This is needed in order to allow faster testing of historical sync to genesis in networks that have no strict finality. Currently, the alternative is to use either testnet or mainnet snapshot, but then the test takes weeks to execute. The new functionality should allow modification of the snapshot and removal of some or all finality signatures in order to simulate "weak" or "no" finality on a given block.
We need to tool that will take a file (format TBD )with finality signatures and an LMDB file and will write the missing finality signatures to the DB
I'm experimenting with the compact-trie
subcommand and I see lots of that messages:
2022-08-30T19:46:12.065183935Z [INFO] trie migration progress: bytes copied 3713498179, tries copied 6709106
Is it possible to add percentage please? Ideally also some rough ETA.
Add a command to purge a block from a storage database. Specifically, given a block hash, purge any associated block header, block body and execution results from storage. Context
We want to unify the various separate binaries related to working with node DBs to a single binary with subcommands for each current tool.
The existing --extract option should also unpack the downloaded file. The --output option should be interpreted as a folder and the file downloaded to there (filename derived from URL). If --extract is provided, the file (which will be the decompressed tarball) should be unpacked to that folder and removed. This end result should be equivalent to piping curl into zstd into tar.
remove anyhow and replace bool returns to a new custom Error enum, potentially comprised of nested custom errors specific to the subcommand
Cargo audit is failing on casper-db-utils
due to a vulnerability in tokio
which needs to be updated.
This is what i get once i run "cargo build --release"
error: failed to run custom build command for jemalloc-sys v0.3.2
Caused by:
process didn't exit successfully: /home/darren/casper-db-utils/target/release/build/jemalloc-sys-d96f0dc68fbad066/build-script-build
(exit status: 101)
thread 'main' panicked at 'failed to execute command: No such file or directory (os error 2)', /home/darren/.cargo/registry/src/github.com-1ecc6299db9ec823/jemalloc-sys-0.3.2/build.rs:389:19
Full output from "cargo build --release"
Compiling serde_derive v1.0.152
Compiling ctor v0.1.26
Compiling tokio-macros v1.8.2
Compiling futures-macro v0.3.26
Compiling tracing-attributes v0.1.23
Compiling synstructure v0.12.6
Compiling thiserror-impl v1.0.38
Compiling openssl-macros v0.1.0
Compiling pin-project-internal v1.0.12
Compiling serde_derive_internals v0.25.0
Compiling datasize_derive v0.2.13
Compiling openssl-sys v0.9.80
Compiling vergen v3.2.0
Compiling jemalloc-sys v0.3.2
Compiling num-derive v0.3.3
Compiling lmdb-sys v0.8.0
Compiling regex v1.7.1
Compiling num-rational v0.2.4
Compiling enum-ordinalize v3.1.12
Compiling proc-macro-error v1.0.4
Compiling zstd-sys v2.0.7+zstd.1.5.4
Compiling enum-iterator-derive v0.6.0
Compiling addr2line v0.19.0
Compiling zstd-safe v5.0.2+zstd.1.5.2
Compiling convert_case v0.4.0
Compiling time-core v0.1.0
Compiling termcolor v1.1.3
Compiling rustc-demangle v0.1.21
Compiling lazycell v1.3.0
Compiling signal-hook v0.3.15
Compiling derp v0.0.14
Compiling clap_lex v0.2.4
Compiling quanta v0.7.2
Compiling async-trait v0.1.64
Compiling pem v0.8.3
Compiling sys-info v0.8.0
Compiling serde_repr v0.1.10
Compiling filetime v0.2.20
Compiling xattr v0.2.3
Compiling fs2 v0.4.3
Compiling encoding_rs v0.8.32
Compiling lmdb v0.8.0
Compiling time v0.3.18
Compiling openssl v0.10.45
Compiling native-tls v0.2.11
Compiling tokio-openssl v0.6.3
Compiling wheelbuf v0.2.0
Compiling strsim v0.10.0
Compiling ipnet v2.7.1
Compiling humantime v2.1.0
Compiling backtrace v0.3.67
Compiling structopt-derive v0.4.18
Compiling wasmi v0.8.0
Compiling derive_more v0.99.17
Compiling base64 v0.21.0
Compiling textwrap v0.16.0
Compiling casper-node v1.4.8
Compiling cache-padded v1.2.0
Compiling zstd v0.11.2+zstd.1.5.2
Compiling tar v0.4.38
Compiling ringbuf v0.2.8
Compiling enum-iterator v0.6.0
Compiling zeroize_derive v1.3.3
Compiling value-bag v1.0.0-alpha.9
Compiling educe v0.4.20
Compiling Inflector v0.11.4
Compiling schemars_derive v0.8.5
Compiling futures-util v0.3.26
Compiling zeroize v1.3.0
Compiling thiserror v1.0.38
Compiling pin-project v1.0.12
Compiling casper-node-macros v1.4.3
Compiling elliptic-curve v0.8.5
Compiling curve25519-dalek v3.2.1
Compiling structopt v0.3.26
Compiling ecdsa v0.10.2
Compiling k256 v0.7.3
Compiling serde v1.0.152
Compiling futures-executor v0.3.26
Compiling futures v0.3.26
error: failed to run custom build command for jemalloc-sys v0.3.2
/home/darren/casper-db-utils/target/release/build/jemalloc-sys-d96f0dc68fbad066/build-script-build
(exit status: 101)jemalloc version : 0.0.0-0-g0000000000000000000000000000000000000000
library revision : 2
CONFIG : --disable-cxx --with-jemalloc-prefix=rjem --with-private-namespace=rjem --host=x86_64-unknown-linux-gnu --build=x86_64-unknown-linux-gnu --prefix=/home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out build_alias=x86_64-unknown-linux-gnu host_alias=x86_64-unknown-linux-gnu CC=cc 'CFLAGS=-O3 -ffunction-sections -fdata-sections -fPIC -m64 -Wall' 'LDFLAGS=-O3 -ffunction-sections -fdata-sections -fPIC -m64 -Wall' 'CPPFLAGS=-O3 -ffunction-sections -fdata-sections -fPIC -m64 -Wall'
CC : cc
CONFIGURE_CFLAGS : -std=gnu11 -Wall -Wsign-compare -Wundef -Wno-format-zero-length -pipe -g3 -fvisibility=hidden -O3 -funroll-loops
SPECIFIED_CFLAGS : -O3 -ffunction-sections -fdata-sections -fPIC -m64 -Wall
EXTRA_CFLAGS :
CPPFLAGS : -O3 -ffunction-sections -fdata-sections -fPIC -m64 -Wall -D_GNU_SOURCE -D_REENTRANT
CXX :
CONFIGURE_CXXFLAGS :
SPECIFIED_CXXFLAGS :
EXTRA_CXXFLAGS :
LDFLAGS : -O3 -ffunction-sections -fdata-sections -fPIC -m64 -Wall
EXTRA_LDFLAGS :
DSO_LDFLAGS : -shared -Wl,-soname,$(@f)
LIBS : -lm -lpthread -ldl
RPATH_EXTRA :
XSLTPROC : false
XSLROOT :
PREFIX : /home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out
BINDIR : /home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out/bin
DATADIR : /home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out/share
INCLUDEDIR : /home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out/include
LIBDIR : /home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out/lib
MANDIR : /home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out/share/man
srcroot : /home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out/jemalloc/
abs_srcroot : /home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out/jemalloc/
objroot :
abs_objroot : /home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out/build/
running: "make" "srcroot=../jemalloc/" "-j" "40"
--- stderr
thread 'main' panicked at 'failed to execute command: No such file or directory (os error 2)', /home/darren/.cargo/registry/src/github.com-1ecc6299db9ec823/jemalloc-sys-0.3.2/build.rs:389:19
note: run with RUST_BACKTRACE=1
environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
We need to move the code in casper-network/casper-node#2948 to a subcommand of the db-utils tool.
The review comments should be addressed as part of this work, ideally as a separate commit to simplify the review. Any further deviations/improvements other than changes strictly required to support the move should be kept to standalone commits too, for the same reason.
Start with version 0.1 as this is an internal tool.
We need a design for a subcommand of the db-utils tool which will download and validate a snapshot of global state + storage DBs.
We need a new subcommand extract-slice
which reads all data for a given block hash (block, deploys, execution results, global state) and stores these to two new DBs.
Tried to compile and it fails at casper-execution-engine-3.0.0/src/storage/trie_store/operations/mod.rs:1266:87 - the trait 'ToBytes' is not implemented for 'casper_hashing::Digest'
Condor has seen many changes related to storage. Now all the data we store is versioned. Also we have moved part of the storage functionality to a separate crate. We now pretty much need a refactor/rewrite of casper-db-utils to make it work with the data format of Condor.
The casper-db-utils should be able to parse the DB and return the size of execution results. We would ideally like to be able to see the minimum, maximum and average size of the execution results.
We want a subcommand which takes a path to a storage DB file, opens the DB and iterates every element to ensure they can all be parsed. It should provide feedback to the user about ongoing progress, e.g. via a log message every 1000th entry handled. For now, we don't need to handle global state store DBs, only storage ones.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.