casper-network/casper-db-utils

Utilities to perform operations on the lmdb databases for a casper-node.

casper-db-utils's Issues

Update db-utils to acquire and store all data under a given block hash

We need a new subcommand extract-slice which reads all data for a given block hash (block, deploys, execution results, global state) and stores these to two new DBs.

archive-get subcommand should unpack tarball

The existing --extract option should also unpack the downloaded file. The --output option should be interpreted as a folder and the file downloaded to there (filename derived from URL). If --extract is provided, the file (which will be the decompressed tarball) should be unpacked to that folder and removed. This end result should be equivalent to piping curl into zstd into tar.

Extend casper-db-utils - add ability to remove finality signatures from blocks

This is needed in order to allow faster testing of historical sync to genesis in networks that have no strict finality. Currently, the alternative is to use either testnet or mainnet snapshot, but then the test takes weeks to execute. The new functionality should allow modification of the snapshot and removal of some or all finality signatures in order to simulate "weak" or "no" finality on a given block.

Add a BackfillFinalitySignatures casper-db-util subcommand

We need to tool that will take a file (format TBD )with finality signatures and an LMDB file and will write the missing finality signatures to the DB

Feature request: Show progress as percentage + ETA?

I'm experimenting with the compact-trie subcommand and I see lots of that messages:

2022-08-30T19:46:12.065183935Z [INFO] trie migration progress: bytes copied 3713498179, tries copied 6709106

Is it possible to add percentage please? Ideally also some rough ETA.

release-1.4.6 trie-compact tool

We need to move the code in casper-network/casper-node#2948 to a subcommand of the db-utils tool.

The review comments should be addressed as part of this work, ideally as a separate commit to simplify the review. Any further deviations/improvements other than changes strictly required to support the move should be kept to standalone commits too, for the same reason.

Compile problem

Tried to compile and it fails at casper-execution-engine-3.0.0/src/storage/trie_store/operations/mod.rs:1266:87 - the trait 'ToBytes' is not implemented for 'casper_hashing::Digest'

Add command to purge a block from database

Add a command to purge a block from a storage database. Specifically, given a block hash, purge any associated block header, block body and execution results from storage. Context

Measure execution results for mainnet and testnet

We should add a new subcommand which reads from a node's storage DB and prints data related to execution results per block.

The execution results which are stored in deploy_metadata_db are indexed by deploy hash, but we want the results per block, so the tool will need to iterate each block in storage, and for every one which has deploys, build the collection of execution results.

The data to printed is:

• median and maximum size of execution results per block (size, being the bincode-encoded bytes count)
• median and maximum number of chunks each set of execution results will produce when converted to a BlockExecutionResultsOrChunk

Compiler issue / Ubuntu server 20.04

This is what i get once i run "cargo build --release"

error: failed to run custom build command for jemalloc-sys v0.3.2

Caused by:
process didn't exit successfully: /home/darren/casper-db-utils/target/release/build/jemalloc-sys-d96f0dc68fbad066/build-script-build (exit status: 101)

thread 'main' panicked at 'failed to execute command: No such file or directory (os error 2)', /home/darren/.cargo/registry/src/github.com-1ecc6299db9ec823/jemalloc-sys-0.3.2/build.rs:389:19

Full output from "cargo build --release"

Compiling serde_derive v1.0.152
Compiling ctor v0.1.26
Compiling tokio-macros v1.8.2
Compiling futures-macro v0.3.26
Compiling tracing-attributes v0.1.23
Compiling synstructure v0.12.6
Compiling thiserror-impl v1.0.38
Compiling openssl-macros v0.1.0
Compiling pin-project-internal v1.0.12
Compiling serde_derive_internals v0.25.0
Compiling datasize_derive v0.2.13
Compiling openssl-sys v0.9.80
Compiling vergen v3.2.0
Compiling jemalloc-sys v0.3.2
Compiling num-derive v0.3.3
Compiling lmdb-sys v0.8.0
Compiling regex v1.7.1
Compiling num-rational v0.2.4
Compiling enum-ordinalize v3.1.12
Compiling proc-macro-error v1.0.4
Compiling zstd-sys v2.0.7+zstd.1.5.4
Compiling enum-iterator-derive v0.6.0
Compiling addr2line v0.19.0
Compiling zstd-safe v5.0.2+zstd.1.5.2
Compiling convert_case v0.4.0
Compiling time-core v0.1.0
Compiling termcolor v1.1.3
Compiling rustc-demangle v0.1.21
Compiling lazycell v1.3.0
Compiling signal-hook v0.3.15
Compiling derp v0.0.14
Compiling clap_lex v0.2.4
Compiling quanta v0.7.2
Compiling async-trait v0.1.64
Compiling pem v0.8.3
Compiling sys-info v0.8.0
Compiling serde_repr v0.1.10
Compiling filetime v0.2.20
Compiling xattr v0.2.3
Compiling fs2 v0.4.3
Compiling encoding_rs v0.8.32
Compiling lmdb v0.8.0
Compiling time v0.3.18
Compiling openssl v0.10.45
Compiling native-tls v0.2.11
Compiling tokio-openssl v0.6.3
Compiling wheelbuf v0.2.0
Compiling strsim v0.10.0
Compiling ipnet v2.7.1
Compiling humantime v2.1.0
Compiling backtrace v0.3.67
Compiling structopt-derive v0.4.18
Compiling wasmi v0.8.0
Compiling derive_more v0.99.17
Compiling base64 v0.21.0
Compiling textwrap v0.16.0
Compiling casper-node v1.4.8
Compiling cache-padded v1.2.0
Compiling zstd v0.11.2+zstd.1.5.2
Compiling tar v0.4.38
Compiling ringbuf v0.2.8
Compiling enum-iterator v0.6.0
Compiling zeroize_derive v1.3.3
Compiling value-bag v1.0.0-alpha.9
Compiling educe v0.4.20
Compiling Inflector v0.11.4
Compiling schemars_derive v0.8.5
Compiling futures-util v0.3.26
Compiling zeroize v1.3.0
Compiling thiserror v1.0.38
Compiling pin-project v1.0.12
Compiling casper-node-macros v1.4.3
Compiling elliptic-curve v0.8.5
Compiling curve25519-dalek v3.2.1
Compiling structopt v0.3.26
Compiling ecdsa v0.10.2
Compiling k256 v0.7.3
Compiling serde v1.0.152
Compiling futures-executor v0.3.26
Compiling futures v0.3.26

error: failed to run custom build command for jemalloc-sys v0.3.2

Caused by:
process didn't exit successfully: /home/darren/casper-db-utils/target/release/build/jemalloc-sys-d96f0dc68fbad066/build-script-build (exit status: 101)
--- stdout
TARGET=x86_64-unknown-linux-gnu
HOST=x86_64-unknown-linux-gnu
NUM_JOBS=40
OUT_DIR="/home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out"
BUILD_DIR="/home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out/build"
SRC_DIR="/home/darren/.cargo/registry/src/github.com-1ecc6299db9ec823/jemalloc-sys-0.3.2"
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-gnu
CC_x86_64-unknown-linux-gnu = None
cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_gnu
CC_x86_64_unknown_linux_gnu = None
cargo:rerun-if-env-changed=HOST_CC
HOST_CC = None
cargo:rerun-if-env-changed=CC
CC = None
cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-gnu
CFLAGS_x86_64-unknown-linux-gnu = None
cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_gnu
CFLAGS_x86_64_unknown_linux_gnu = None
cargo:rerun-if-env-changed=HOST_CFLAGS
HOST_CFLAGS = None
cargo:rerun-if-env-changed=CFLAGS
CFLAGS = None
cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
CRATE_CC_NO_DEFAULTS = None
DEBUG = Some("false")
CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
CC="cc"
CFLAGS="-O3 -ffunction-sections -fdata-sections -fPIC -m64 -Wall"
JEMALLOC_REPO_DIR="jemalloc"
JEMALLOC_SRC_DIR="/home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out/jemalloc"
cargo:rustc-cfg=prefixed
--with-jemalloc-prefix=rjem
running: "sh" "/home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out/jemalloc/configure" "--disable-cxx" "--with-jemalloc-prefix=rjem" "--with-private-namespace=rjem" "--host=x86_64-unknown-linux-gnu" "--build=x86_64-unknown-linux-gnu" "--prefix=/home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out"
checking for xsltproc... false
checking for x86_64-unknown-linux-gnu-gcc... cc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether cc accepts -g... yes
checking for cc option to accept ISO C89... none needed
checking whether compiler is cray... no
checking whether compiler supports -std=gnu11... yes
checking whether compiler supports -Wall... yes
checking whether compiler supports -Wshorten-64-to-32... no
checking whether compiler supports -Wsign-compare... yes
checking whether compiler supports -Wundef... yes
checking whether compiler supports -Wno-format-zero-length... yes
checking whether compiler supports -pipe... yes
checking whether compiler supports -g3... yes
checking how to run the C preprocessor... cc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking whether byte ordering is bigendian... no
checking size of void ... 8
checking size of int... 4
checking size of long... 8
checking size of long long... 8
checking size of intmax_t... 8
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking whether pause instruction is compilable... yes
checking number of significant virtual address bits... 48
checking for x86_64-unknown-linux-gnu-ar... no
checking for ar... ar
checking for x86_64-unknown-linux-gnu-nm... no
checking for nm... nm
checking for gawk... gawk
checking malloc.h usability... yes
checking malloc.h presence... yes
checking for malloc.h... yes
checking whether malloc_usable_size definition can use const argument... no
checking for library containing log... -lm
checking whether attribute syntax is compilable... yes
checking whether compiler supports -fvisibility=hidden... yes
checking whether compiler supports -fvisibility=hidden... no
checking whether compiler supports -Werror... yes
checking whether compiler supports -herror_on_warning... no
checking whether tls_model attribute is compilable... yes
checking whether compiler supports -Werror... yes
checking whether compiler supports -herror_on_warning... no
checking whether alloc_size attribute is compilable... yes
checking whether compiler supports -Werror... yes
checking whether compiler supports -herror_on_warning... no
checking whether format(gnu_printf, ...) attribute is compilable... yes
checking whether compiler supports -Werror... yes
checking whether compiler supports -herror_on_warning... no
checking whether format(printf, ...) attribute is compilable... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking for x86_64-unknown-linux-gnu-ranlib... no
checking for ranlib... ranlib
checking for ld... /usr/bin/ld
checking for autoconf... false
checking for memalign... yes
checking for valloc... yes
checking whether compiler supports -O3... yes
checking whether compiler supports -O3... no
checking whether compiler supports -funroll-loops... yes
checking configured backtracing method... N/A
checking for sbrk... yes
checking whether utrace(2) is compilable... no
checking whether a program using __builtin_unreachable is compilable... yes
checking whether a program using __builtin_ffsl is compilable... yes
checking LG_PAGE... 12
checking pthread.h usability... yes
checking pthread.h presence... yes
checking for pthread.h... yes
checking for pthread_create in -lpthread... yes
checking dlfcn.h usability... yes
checking dlfcn.h presence... yes
checking for dlfcn.h... yes
checking for dlsym... no
checking for dlsym in -ldl... yes
checking whether pthread_atfork(3) is compilable... yes
checking whether pthread_setname_np(3) is compilable... yes
checking for library containing clock_gettime... none required
checking whether clock_gettime(CLOCK_MONOTONIC_COARSE, ...) is compilable... yes
checking whether clock_gettime(CLOCK_MONOTONIC, ...) is compilable... yes
checking whether mach_absolute_time() is compilable... no
checking whether compiler supports -Werror... yes
checking whether syscall(2) is compilable... yes
checking for secure_getenv... yes
checking for sched_getcpu... yes
checking for sched_setaffinity... yes
checking for issetugid... no
checking for _malloc_thread_cleanup... no
checking for _pthread_mutex_init_calloc_cb... no
checking for TLS... yes
checking whether C11 atomics is compilable... yes
checking whether GCC __atomic atomics is compilable... yes
checking whether GCC __sync atomics is compilable... yes
checking whether Darwin OSAtomic() is compilable... no
checking whether madvise(2) is compilable... yes
checking whether madvise(..., MADV_FREE) is compilable... yes
checking whether madvise(..., MADV_DONTNEED) is compilable... yes
checking whether madvise(..., MADV_DO[NT]DUMP) is compilable... yes
checking whether madvise(..., MADV_[NO]HUGEPAGE) is compilable... yes
checking whether to force 32-bit _sync{add,sub}_and_fetch()... no
checking whether to force 64-bit _sync{add,sub}_and_fetch()... no
checking for _builtin_clz... yes
checking whether Darwin os_unfair_lock() is compilable... no
checking whether Darwin OSSpin() is compilable... no
checking whether glibc malloc hook is compilable... yes
checking whether glibc memalign hook is compilable... yes
checking whether pthreads adaptive mutexes is compilable... yes
checking whether compiler supports -D_GNU_SOURCE... yes
checking whether compiler supports -Werror... yes
checking whether compiler supports -herror_on_warning... no
checking whether strerror_r returns char with gnu source is compilable... yes
checking for stdbool.h that conforms to C99... yes
checking for _Bool... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating jemalloc.pc
config.status: creating doc/html.xsl
config.status: creating doc/manpages.xsl
config.status: creating doc/jemalloc.xml
config.status: creating include/jemalloc/jemalloc_macros.h
config.status: creating include/jemalloc/jemalloc_protos.h
config.status: creating include/jemalloc/jemalloc_typedefs.h
config.status: creating include/jemalloc/internal/jemalloc_preamble.h
config.status: creating test/test.sh
config.status: creating test/include/test/jemalloc_test.h
config.status: creating config.stamp
config.status: creating bin/jemalloc-config
config.status: creating bin/jemalloc.sh
config.status: creating bin/jeprof
config.status: creating include/jemalloc/jemalloc_defs.h
config.status: include/jemalloc/jemalloc_defs.h is unchanged
config.status: creating include/jemalloc/internal/jemalloc_internal_defs.h
config.status: include/jemalloc/internal/jemalloc_internal_defs.h is unchanged
config.status: creating test/include/test/jemalloc_test_defs.h
config.status: test/include/test/jemalloc_test_defs.h is unchanged
config.status: executing include/jemalloc/internal/public_symbols.txt commands
config.status: executing include/jemalloc/internal/private_symbols.awk commands
config.status: executing include/jemalloc/internal/private_symbols_jet.awk commands
config.status: executing include/jemalloc/internal/public_namespace.h commands
config.status: executing include/jemalloc/internal/public_unnamespace.h commands
config.status: executing include/jemalloc/internal/size_classes.h commands
config.status: executing include/jemalloc/jemalloc_protos_jet.h commands
config.status: executing include/jemalloc/jemalloc_rename.h commands
config.status: executing include/jemalloc/jemalloc_mangle.h commands
config.status: executing include/jemalloc/jemalloc_mangle_jet.h commands
config.status: executing include/jemalloc/jemalloc.h commands

jemalloc version : 0.0.0-0-g0000000000000000000000000000000000000000
library revision : 2

CONFIG : --disable-cxx --with-jemalloc-prefix=rjem --with-private-namespace=rjem --host=x86_64-unknown-linux-gnu --build=x86_64-unknown-linux-gnu --prefix=/home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out build_alias=x86_64-unknown-linux-gnu host_alias=x86_64-unknown-linux-gnu CC=cc 'CFLAGS=-O3 -ffunction-sections -fdata-sections -fPIC -m64 -Wall' 'LDFLAGS=-O3 -ffunction-sections -fdata-sections -fPIC -m64 -Wall' 'CPPFLAGS=-O3 -ffunction-sections -fdata-sections -fPIC -m64 -Wall'
CC : cc
CONFIGURE_CFLAGS : -std=gnu11 -Wall -Wsign-compare -Wundef -Wno-format-zero-length -pipe -g3 -fvisibility=hidden -O3 -funroll-loops
SPECIFIED_CFLAGS : -O3 -ffunction-sections -fdata-sections -fPIC -m64 -Wall
EXTRA_CFLAGS :
CPPFLAGS : -O3 -ffunction-sections -fdata-sections -fPIC -m64 -Wall -D_GNU_SOURCE -D_REENTRANT
CXX :
CONFIGURE_CXXFLAGS :
SPECIFIED_CXXFLAGS :
EXTRA_CXXFLAGS :
LDFLAGS : -O3 -ffunction-sections -fdata-sections -fPIC -m64 -Wall
EXTRA_LDFLAGS :
DSO_LDFLAGS : -shared -Wl,-soname,$(@f)
LIBS : -lm -lpthread -ldl
RPATH_EXTRA :

XSLTPROC : false
XSLROOT :

PREFIX : /home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out
BINDIR : /home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out/bin
DATADIR : /home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out/share
INCLUDEDIR : /home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out/include
LIBDIR : /home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out/lib
MANDIR : /home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out/share/man

srcroot : /home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out/jemalloc/
abs_srcroot : /home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out/jemalloc/
objroot :
abs_objroot : /home/darren/casper-db-utils/target/release/build/jemalloc-sys-840f646a48e6a907/out/build/

JEMALLOC_PREFIX : rjem
JEMALLOC_PRIVATE_NAMESPACE
: rjem_je
install_suffix :
malloc_conf :
autogen : 0
debug : 0
stats : 1
prof : 0
prof-libunwind : 0
prof-libgcc : 0
prof-gcc : 0
fill : 1
utrace : 0
xmalloc : 0
log : 0
lazy_lock : 0
cache-oblivious : 1
cxx : 0

running: "make" "srcroot=../jemalloc/" "-j" "40"

--- stderr
thread 'main' panicked at 'failed to execute command: No such file or directory (os error 2)', /home/darren/.cargo/registry/src/github.com-1ecc6299db9ec823/jemalloc-sys-0.3.2/build.rs:389:19
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

Provide subcommand to scan a storage DB file

We want a subcommand which takes a path to a storage DB file, opens the DB and iterates every element to ensure they can all be parsed. It should provide feedback to the user about ongoing progress, e.g. via a log message every 1000th entry handled. For now, we don't need to handle global state store DBs, only storage ones.

Move DB utility tools from casper-node to casper-network/casper-db-utils

We want to unify the various separate binaries related to working with node DBs to a single binary with subcommands for each current tool.

[Design] Archiver tool

We need a design for a subcommand of the db-utils tool which will download and validate a snapshot of global state + storage DBs.

https://hackmd.io/@V-Me7vygSOqZwX7bhV1V-g/B1Uvw8YIc

Version and Release casper db utils

Start with version 0.1 as this is an internal tool.

Update db-utils dependencies

Cargo audit is failing on casper-db-utils due to a vulnerability in tokio which needs to be updated.

Use this error throughout db-utils

remove anyhow and replace bool returns to a new custom Error enum, potentially comprised of nested custom errors specific to the subcommand

Extend casper-db-utils - add ability to read the size of execution results

The casper-db-utils should be able to parse the DB and return the size of execution results. We would ideally like to be able to see the minimum, maximum and average size of the execution results.

Refactor casper-db-utils to work with Condor

Condor has seen many changes related to storage. Now all the data we store is versioned. Also we have moved part of the storage functionality to a separate crate. We now pretty much need a refactor/rewrite of casper-db-utils to make it work with the data format of Condor.

New tool to call migrations

Make a tool to call this migration and give to Joe to run on a realistic size. Add to db_utils.

See this ticket for context casper-network/casper-node#3849

Update dependencies to stable state

Follow up on casper-network/casper-node#3921. The underlying problem is a dependency mismatch with casper-types and casper-node . More specifically, casper-types 1.6 (should be 3.0) had breaking changes compared to casper-types 1.5 . We should update the casper-db-utils dependencies to a stable state once the casper-types changes are sorted out.

Create archive subcommand

The tool needs to be able to download a snapshot from a configurable URL and unpack it to a configurable path

Clean up `casper-db-utils`

Remove accumulated technical debt in casper-db-utils, specifically:

• revisit all file writes where overwrites can happen and allow them only through --overwrite parameter
• explore replacing the current logging implementation with env-logger
• ensure long-running commands give user feedback in some way (logfile or stdout)