linxgnu / grocksdb Goto Github PK
View Code? Open in Web Editor NEWRocksDB wrapper for Go. Support 9.x, 8.x, 7.x, 6.x, etc
License: MIT License
RocksDB wrapper for Go. Support 9.x, 8.x, 7.x, 6.x, etc
License: MIT License
Why do you need to open all column families when all i need to do / work on is the only 1 i want to use?
Any ways to work around?
enviroment:
Environment: MacOS Ventura with Apple Silicon
Both RocksDB(7.7.3) and its dependencies (including snappy) are installed via Homebrew.
using grocksdb 1.7.12
compile command:
CGO_CFLAGS="-I/opt/homebrew/Cellar/rocksdb/7.7.3/include" \
CGO_LDFLAGS="-L/opt/homebrew/Cellar/rocksdb/7.7.3 -lrocksdb -lstdc++ -lm -lz -lsnappy -llz4 -lzstd" \
go build
got this error
/usr/local/go/pkg/tool/darwin_arm64/link: running clang failed: exit status 1
ld: library not found for -lrocksdb
clang: error: linker command failed with exit code 1 (use -v to see invocation)
any suggesstions?
This is a bug that I reported in July 2022 originally at tecbot/gorocksdb#221 and it is still relevant to this repo:
Courtesy of the Cosmos Network Security, I discovered this code while auditing supply chain dependencies.
If we examine this code in
Lines 344 to 362 in 254f68b
and then extract it and run the following https://go.dev/play/p/MXpanwJKofY or inlined below:
package main
import (
"errors"
"fmt"
"io"
)
type wbi struct {
data []byte
err error
}
func (iter *wbi) decodeVarint() uint64 {
var n int
var x uint64
for shift := uint(0); shift < 64 && n < len(iter.data); shift += 7 {
b := uint64(iter.data[n])
n++
x |= (b & 0x7F) << shift
if (b & 0x80) == 0 {
iter.data = iter.data[n:]
return x
}
}
if n == len(iter.data) {
iter.err = io.ErrShortBuffer
} else {
iter.err = errors.New("malformed varint")
}
return 0
}
func main() {
// The 10th byte here is invalid and should be reported.
// Please see https://github.com/golang/go/issues/41185
b := []byte{0xd7, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x7f}
wb := &wbi{data: b}
v := wb.decodeVarint()
println(v)
fmt.Printf("%v\n", wb.err) // This should have reported an error.
}
which prints
18446744073709551575
<nil>
The fix entails just using the Go standard library's code in which I fixed this bug in March 2021 in the Go standard library per golang/go@aafad20
with a demo at https://go.dev/play/p/kYGBjzh24Qx which should properly print malformed varint
package main
import (
"encoding/binary"
"errors"
"fmt"
"io"
)
type wbi struct {
data []byte
err error
}
func (iter *wbi) decodeVarint() uint64 {
var n int
var x uint64
for shift := uint(0); shift < 64 && n < len(iter.data); shift += 7 {
b := uint64(iter.data[n])
n++
x |= (b & 0x7F) << shift
if (b & 0x80) == 0 {
iter.data = iter.data[n:]
return x
}
}
if n == len(iter.data) {
iter.err = io.ErrShortBuffer
} else {
iter.err = errors.New("malformed varint")
}
return 0
}
func (iter *wbi) decodeVarint_good() (x uint64) {
v, n := binary.Varint(iter.data)
if n == len(iter.data) {
iter.err = io.ErrShortBuffer
} else if n < 0 {
iter.err = errors.New("malformed varint")
} else {
x = uint64(v)
}
return x
}
func main() {
// The 10th byte here is invalid and should be reported.
// Please see https://github.com/golang/go/issues/41185
b := []byte{0xd7, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x7f}
wb := &wbi{data: b}
v := wb.decodeVarint_good()
println(v)
fmt.Printf("%v\n", wb.err) // This should have reported an error.
}
which correctly prints
0
malformed varint
Hi, in the Get() function, it uses refGoBytes() to convert []bytes to C.char, involving the use of unsafe.Pointer. Is it possible that a key has been garbage collected, posing a risk of cKey pointing to null?
As I research a lot on leveldb/rocksdb sst file, I recommend setting Block Max Size to at least 8MB and at most 32MB for Flash raft store.
I got these errors when running go build.
# github.com/linxGnu/grocksdb
../../go/pkg/mod/github.com/linx!gnu/[email protected]/backup.go:44:13: could not determine kind of name for C.rocksdb_backup_engine_open_opts
../../go/pkg/mod/github.com/linx!gnu/[email protected]/backup.go:136:2: could not determine kind of name for C.rocksdb_backup_engine_restore_db_from_backup
I am using Ubuntu 22.04 LTS, librocksdb-dev (6.11.4-3) from apt and [email protected].
Any help would be appreciated!
cfName is 12 items, if i created 11, it will work, but at 12, it will crash. can u try?
dbr, cfh, err = gorocksdb.OpenDbColumnFamilies(opts[i], "/d/a", cfNames, []*gorocksdb.Options{opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i]})
crashed at this last line
// OpenDbColumnFamilies opens a database with the specified column families.
func OpenDbColumnFamilies(
opts *Options,
name string,
cfNames []string,
cfOpts []*Options,
) (db *DB, cfHandles []*ColumnFamilyHandle, err error) {
numColumnFamilies := len(cfNames)
if numColumnFamilies != len(cfOpts) {
err = ErrColumnFamilyMustMatch
return
}
cName := C.CString(name)}
// OpenDbColumnFamilies opens a database with the specified column families.
func OpenDbColumnFamilies(
opts *Options,
name string,
cfNames []string,
cfOpts []*Options,
) (db *DB, cfHandles []*ColumnFamilyHandle, err error) {
numColumnFamilies := len(cfNames)
if numColumnFamilies != len(cfOpts) {
err = ErrColumnFamilyMustMatch
return
}
cName := C.CString(name)
cNames := make([]*C.char, numColumnFamilies)
for i, s := range cfNames {
cNames[i] = C.CString(s)
}
cOpts := make([]*C.rocksdb_options_t, numColumnFamilies)
for i, o := range cfOpts {
cOpts[i] = o.c
}
cHandles := make([]*C.rocksdb_column_family_handle_t, numColumnFamilies)
var cErr *C.char
_db := C.rocksdb_open_column_families(
opts.c,
cName,
C.int(numColumnFamilies),
&cNames[0],
&cOpts[0],
&cHandles[0],
&cErr,
cNames := make([]*C.char, numColumnFamilies)
for i, s := range cfNames {
cNames[i] = C.CString(s)
}
cOpts := make([]*C.rocksdb_options_t, numColumnFamilies)
for i, o := range cfOpts {
cOpts[i] = o.c
}
cHandles := make([]*C.rocksdb_column_family_handle_t, numColumnFamilies)
var cErr *C.char
_db := C.rocksdb_open_column_families(
opts.c,
cName,
C.int(numColumnFamilies),
&cNames[0],
&cOpts[0],
&cHandles[0],
&cErr,
for NewLRUCache, how to set strict_capacity_limit in grocksdb?
great piece of software. big fan of yours.
I have more situations that need to customize the link flags, for example recently I cross compiled the app to mingwW64 with the help of nixpkgs, but there I have to link with -lrocksdb-shared
.
currently there's a way to suppress the builtin link flags, which is the testing
build tag, it works perfectly right now, it's just the name feels hacky, shall we support sth like grocksdb_no_link
build tag, where there's no builtin link flags, user can supply with CGO_LDFLAGS
env var?
cfNames = []string{"default", "m", "v", "c", "e", "x", "k", "l", "s", "h", "z", "bxb"}
//cfNames = []string{"default", "m", "v", "c", "e", "x", "k", "l", "s", "h", "z", "b", "p"}
dbr[i], cfh, err = gorocksdb.OpenDbColumnFamilies(opts[i], defaultDBDir+"/d/"+ks, cfNames, []*gorocksdb.Options{opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i]})
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x478 pc=0xc41992]
runtime stack:
runtime.throw(0x123210e, 0x2a)
/usr/local/go/src/runtime/panic.go:1117 +0x72
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:718 +0x2e5
goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x9062fc, 0xc000207478, 0xc000010588)
/usr/local/go/src/runtime/cgocall.go:154 +0x5b fp=0xc000207448 sp=0xc000207410 pc=0x79123b
git.a.com/a/grocksdb._Cfunc_rocksdb_open_column_families(0x7f4bb919fa00, 0x7f4bb903f260, 0xc, 0xc00005a480, 0xc00001e120, 0xc00001e180, 0xc000010588, 0x0)
_cgo_gotypes.go:2705 +0x49 fp=0xc000207478 sp=0xc000207448 pc=0x874c29
git.a.com/a/grocksdb.OpenDbColumnFamilies.func1(0xc000059900, 0x7f4bb903f260, 0xc, 0xc0002075d0, 0xc0002075b8, 0xc0002075e8, 0xc000010588, 0x8)
/root/go/pkg/mod/git.a.com/a/[email protected]/db.go:158 +0x229 fp=0xc0002074e8 sp=0xc000207478 pc=0x878b49
git.a.com/a/grocksdb.OpenDbColumnFamilies(0xc000059900, 0xc000016c80, 0x8, 0xc000168540, 0xc, 0xc, 0xc000207878, 0xc, 0xc, 0x0, ...)
/root/go/pkg/mod/git.a.com/a/[email protected]/db.go:158 +0x285 fp=0xc000207610 sp=0xc0002074e8 pc=0x875f85
I have a recommendation on how we can improve the build and stability of this library. Our RocksDB library files (librocksdb.a
) depend on the operating system's RocksDB files, which can cause inconsistencies. For instance, if you have Ubuntu 23.04 on your local machine with version librocksdb7.8
and GitHub Action's ubuntu-latest
has version librocksdb6.11
, they will both be incompatible with specific versions of grocksdb.
We have two options: we can change the grocksdb
library to avoid giving errors based on the build and version particularities of the currently installed library, or we can have each version of the library directly usable and have the same library version. This method has been successfully used in other libraries like confluent-kafka-go. You can check out the libraries at https://github.com/confluentinc/confluent-kafka-go/tree/master/kafka/librdkafka_vendor.
On ubuntu22, I had to add -lbz2
to the CGO_LDFLAGS
to successfully run go build
I'd suggest to add it in the Readme build flags:
CGO_CFLAGS="-I/path/to/rocksdb/include" \
CGO_LDFLAGS="-L/path/to/rocksdb -lrocksdb -lstdc++ -lm -lz -lsnappy -llz4 -lzstd -lbz2" \
go build
Thanks fro your work
func() {
value:=db.Get()
// value.Free() <- is this necessary?
}
what if i dont free it?
how do you use getbatch or access wal logs?
sorry cant seem to find any examples online and no tests on this.
I´m trying to compile the project mentioned in #28 on a raspberry Pi4 running ubuntu 20.10 arm64.
The compiler interrupts with
/root/go/pkg/mod/github.com/linx!gnu/[email protected]/backup.go:44:13: could not determine kind of name for C.rocksdb_backup_engine_open_opts /root/go/pkg/mod/github.com/linx!gnu/[email protected]/backup.go:136:2: could not determine kind of name for C.rocksdb_backup_engine_restore_db_from_backup
Do you have any idea how to get around this?
Hi @linxGnu,
When I write into the same rocksdb file with 2 gorutines, i have this unexpected panic.
// configure
func (db *DB) init() error {
cache := grocksdb.NewLRUCache(8 * common.MiB)
blockBaseTableOptions := grocksdb.NewDefaultBlockBasedTableOptions()
blockBaseTableOptions.SetBlockCache(cache)
opts := grocksdb.NewDefaultOptions()
opts.SetCreateIfMissing(true)
opts.SetBlockBasedTableFactory(blockBaseTableOptions)
opts.SetWriteBufferSize(4 * common.MiB)
txDBOpts := grocksdb.NewDefaultTransactionDBOptions()
rdb, err := grocksdb.OpenTransactionDb(opts, txDBOpts, db.path)
if err != nil {
return errors.Wrap(err, "failed to open db")
}
db.db = rdb
db.ro = grocksdb.NewDefaultReadOptions()
db.wo = grocksdb.NewDefaultWriteOptions()
db.txOpts = grocksdb.NewDefaultTransactionOptions()
db.ro.SetFillCache(true)
db.wo.SetSync(db.fsync)
return nil
}
// test writes
func run(database *rocksdb.DB, chRenewTx chan struct{}) {
for i := 0; i < expectedTotalMessages; i++ {
for j := 0; j < 5; j++ {
key := []byte("key-" + strconv.Itoa(i) + "-" + strconv.Itoa(j))
value := []byte("value-" + strconv.Itoa(i) + "-" + strconv.Itoa(j))
if _, err := database.Get(key); err != nil {
panic(err)
}
if err := database.Set(key, value); err != nil {
panic(err)
}
if i > 0 && (i+j)%messagesToCommit == 0 {
chRenewTx <- struct{}{}
}
}
}
}
func main() {
database, err := rocksdbwrapper.New("./.db_data", false)
if err != nil {
panic(err)
}
if err := database.Begin(); err != nil {
panic(err)
}
chRenewTx := make(chan struct{}, 1000)
go func() {
for range chRenewTx {
// fmt.Println("Renewing tx...")
if err := database.RenewTransaction(); err != nil {
panic(err)
}
}
}()
now := time.Now()
go run(database, chRenewTx) // If i disable this gorutine, does not panic!
run(database, chRenewTx)
since := time.Since(now)
fmt.Println("Time:", since)
fmt.Println("Rate:", float64(expectedTotalMessages)/since.Seconds())
}
main(11175,0x700008e78000) malloc: *** error for object 0x5028c00: pointer being freed was not allocated
main(11175,0x700008e78000) malloc: *** set a breakpoint in malloc_error_break to debug
SIGABRT: abort
PC=0x7fff2061092e m=3 sigcode=0
goroutine 0 [idle]:
runtime: unknown pc 0x7fff2061092e
stack: frame={sp:0x700008e77ad8, fp:0x0} stack=[0x700008df8350,0x700008e77f50)
0000700008e779d8: 00007fff2048dba3 0000000000000000
0000700008e779e8: 0000700008e77a38 0000700008e77a30
0000700008e779f8: 0000000004ff8000 0000700008e78000
0000700008e77a08: 00007ffeefbff465 00007fff2048db2e
0000700008e77a18: 0000000000000050 0000700008e77b00
0000700008e77a28: 00007fff20683627 0000003000000018
0000700008e77a38: 0000700008e77b10 0000700008e77a50
0000700008e77a48: 00007fff2068376d 0000000000000000
0000700008e77a58: 0000000000000000 00007fff2048db2e
0000700008e77a68: 0000000004ff804b 0000000000000000
0000700008e77a78: 0000000000000000 0000700008e77c68
0000700008e77a88: 0000700008e77af8 0000000004ff7028
0000700008e77a98: 0000000000000050 00007fff809382a0
0000700008e77aa8: 00007fff8096bae8 00007fff2054d896
0000700008e77ab8: 0000000000000000 0000700008e77b00
0000700008e77ac8: 00007fff205502c6 0000000000000000
0000700008e77ad8: <00007fff2063f5bd 0000000000000003
0000700008e77ae8: 0000000004ff7028 0000700008e77b20
0000700008e77af8: 0000000000000000 0000700008e77b40
0000700008e77b08: 00007fff20594411 0000000004ff7028
0000700008e77b18: 00007fff2060b829 00007ffffffff9df
0000700008e77b28: ffffffff04ff7000 0000000004ff7000
0000700008e77b38: 0000000000000000 0000700008e77ba0
0000700008e77b48: 00007fff204741f5 000000002d004b10
0000700008e77b58: 0000700008e78000 0000700008e77c60
0000700008e77b68: 0000000000000000 00007fff2048b93d
0000700008e77b78: 00000000000017fd 00000000000017fd
0000700008e77b88: 0000000000003000 000000002c830040
0000700008e77b98: 000000002e00c200 0000700008e77c80
0000700008e77ba8: 00007fff2047734a 0000000400000000
0000700008e77bb8: 0000000004dc7380 0000000005028c00
0000700008e77bc8: 0000000000001400 0000000005000000
runtime: unknown pc 0x7fff2061092e
stack: frame={sp:0x700008e77ad8, fp:0x0} stack=[0x700008df8350,0x700008e77f50)
0000700008e779d8: 00007fff2048dba3 0000000000000000
0000700008e779e8: 0000700008e77a38 0000700008e77a30
0000700008e779f8: 0000000004ff8000 0000700008e78000
0000700008e77a08: 00007ffeefbff465 00007fff2048db2e
0000700008e77a18: 0000000000000050 0000700008e77b00
0000700008e77a28: 00007fff20683627 0000003000000018
0000700008e77a38: 0000700008e77b10 0000700008e77a50
0000700008e77a48: 00007fff2068376d 0000000000000000
0000700008e77a58: 0000000000000000 00007fff2048db2e
0000700008e77a68: 0000000004ff804b 0000000000000000
0000700008e77a78: 0000000000000000 0000700008e77c68
0000700008e77a88: 0000700008e77af8 0000000004ff7028
0000700008e77a98: 0000000000000050 00007fff809382a0
0000700008e77aa8: 00007fff8096bae8 00007fff2054d896
0000700008e77ab8: 0000000000000000 0000700008e77b00
0000700008e77ac8: 00007fff205502c6 0000000000000000
0000700008e77ad8: <00007fff2063f5bd 0000000000000003
0000700008e77ae8: 0000000004ff7028 0000700008e77b20
0000700008e77af8: 0000000000000000 0000700008e77b40
0000700008e77b08: 00007fff20594411 0000000004ff7028
0000700008e77b18: 00007fff2060b829 00007ffffffff9df
0000700008e77b28: ffffffff04ff7000 0000000004ff7000
0000700008e77b38: 0000000000000000 0000700008e77ba0
0000700008e77b48: 00007fff204741f5 000000002d004b10
0000700008e77b58: 0000700008e78000 0000700008e77c60
0000700008e77b68: 0000000000000000 00007fff2048b93d
0000700008e77b78: 00000000000017fd 00000000000017fd
0000700008e77b88: 0000000000003000 000000002c830040
0000700008e77b98: 000000002e00c200 0000700008e77c80
0000700008e77ba8: 00007fff2047734a 0000000400000000
0000700008e77bb8: 0000000004dc7380 0000000005028c00
0000700008e77bc8: 0000000000001400 0000000005000000
goroutine 7 [syscall]:
runtime.cgocall(0x4161b70, 0xc00005ed80, 0xc0000109d8)
/usr/local/go/src/runtime/cgocall.go:154 +0x5b fp=0xc00005ed50 sp=0xc00005ed18 pc=0x400445b
github.com/linxGnu/grocksdb._Cfunc_rocksdb_transaction_put(0xc50a970, 0xc000173528, 0x8, 0xc000173530, 0xa, 0xc0000109d8)
_cgo_gotypes.go:8175 +0x45 fp=0xc00005ed80 sp=0xc00005ed50 pc=0x4153625
github.com/linxGnu/grocksdb.(*Transaction).Put.func2(0xc0001820a0, 0xc000173528, 0xc000173528, 0x8, 0x8, 0xc000173530, 0xc000173530, 0xa, 0x10, 0xc0000109d8)
/usr/local/godeps/pkg/mod/github.com/linx!gnu/[email protected]/transaction.go:128 +0xcb fp=0xc00005edc8 sp=0xc00005ed80 pc=0x415674b
github.com/linxGnu/grocksdb.(*Transaction).Put(0xc0001820a0, 0xc000173528, 0x8, 0x8, 0xc000173530, 0xa, 0x10, 0x0, 0x0)
/usr/local/godeps/pkg/mod/github.com/linx!gnu/[email protected]/transaction.go:128 +0x105 fp=0xc00005ee40 sp=0xc00005edc8 pc=0x4155865
gitlab.com/etoshi/backend/normalizer-r2d2/pkg/db/providers/rocksdb.(*DB).Set(0xc00002c540, 0xc000173528, 0x8, 0x8, 0xc000173530, 0xa, 0x10, 0x0, 0x0)
/Users/sergio/Code/backend/normalizer-r2d2/pkg/db/providers/rocksdb/rocksdb.go:160 +0x1dd fp=0xc00005eec8 sp=0xc00005ee40 pc=0x4159d7d
main.run(0xc00002c540, 0xc00007c240)
/Users/sergio/Code/backend/normalizer-r2d2/_examples/foo/main.go:28 +0x2d5 fp=0xc00005efd0 sp=0xc00005eec8 pc=0x415a6b5
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00005efd8 sp=0xc00005efd0 pc=0x406a401
created by main.main
/Users/sergio/Code/backend/normalizer-r2d2/_examples/foo/main.go:62 +0x133
goroutine 1 [runnable]:
github.com/linxGnu/grocksdb._Cfunc_rocksdb_free(0xc510d30)
_cgo_gotypes.go:2661 +0x3c
github.com/linxGnu/grocksdb.(*Slice).Free.func1(0xc000600000)
/usr/local/godeps/pkg/mod/github.com/linx!gnu/[email protected]/slice.go:53 +0x55
github.com/linxGnu/grocksdb.(*Slice).Free(0xc000600000)
/usr/local/godeps/pkg/mod/github.com/linx!gnu/[email protected]/slice.go:53 +0x3b
gitlab.com/etoshi/backend/normalizer-r2d2/pkg/db/providers/rocksdb.(*DB).get(0xc00002c540, 0xc000580000, 0x8, 0x8, 0x4051925, 0xc000580010, 0xc00016fe58, 0xa, 0x10)
/Users/sergio/Code/backend/normalizer-r2d2/pkg/db/providers/rocksdb/rocksdb.go:205 +0x165
gitlab.com/etoshi/backend/normalizer-r2d2/pkg/db/providers/rocksdb.(*DB).Get(0xc00002c540, 0xc000580000, 0x8, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/sergio/Code/backend/normalizer-r2d2/pkg/db/providers/rocksdb/rocksdb.go:214 +0xaf
main.run(0xc00002c540, 0xc00007c240)
/Users/sergio/Code/backend/normalizer-r2d2/_examples/foo/main.go:24 +0x269
main.main()
/Users/sergio/Code/backend/normalizer-r2d2/_examples/foo/main.go:63 +0x14b
goroutine 6 [chan receive]:
main.main.func1(0xc00007c240, 0xc00002c540)
/Users/sergio/Code/backend/normalizer-r2d2/_examples/foo/main.go:52 +0x35
created by main.main
/Users/sergio/Code/backend/normalizer-r2d2/_examples/foo/main.go:51 +0xe5
rax 0x0
rbx 0x700008e78000
rcx 0x700008e77ad8
rdx 0x0
rdi 0x903
rsi 0x6
rbp 0x700008e77b00
rsp 0x700008e77ad8
r8 0x0
r9 0x0
r10 0x700008e78000
r11 0x246
r12 0x903
r13 0x50
r14 0x6
r15 0x16
rip 0x7fff2061092e
rflags 0x246
cs 0x7
fs 0x0
gs 0x0
exit status 2
CGO_ENABLED=1 CGO_CFLAGS="-I/usr/local/include" CGO_LDFLAGS="-L/usr/local/lib -lrocksdb -lz -lsnappy -llz4 -lzstd -lbz2" go build rocskdbmain.go
# command-line-arguments
/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: /usr/local/lib/librocksdb.a(fs_posix.o): in function `rocksdb::CreateIOUring()':
/usr/local/src/rocksdb/./env/io_posix.h:272: undefined reference to `io_uring_queue_init'
/usr/bin/ld: /usr/local/lib/librocksdb.a(fs_posix.o): in function `rocksdb::(anonymous namespace)::PosixFileSystem::Poll(std::vector<void*, std::allocator<void*> >&, unsigned long)':
/usr/include/liburing.h:745: undefined reference to `__io_uring_get_cqe'
/usr/bin/ld: /usr/local/lib/librocksdb.a(fs_posix.o): in function `rocksdb::(anonymous namespace)::PosixFileSystem::AbortIO(std::vector<void*, std::allocator<void*> >&)':
/usr/local/src/rocksdb/env/fs_posix.cc:1109: undefined reference to `io_uring_get_sqe'
/usr/bin/ld: /usr/local/src/rocksdb/env/fs_posix.cc:1118: undefined reference to `io_uring_submit'
/usr/bin/ld: /usr/local/lib/librocksdb.a(fs_posix.o): in function `rocksdb::(anonymous namespace)::PosixFileSystem::AbortIO(std::vector<void*, std::allocator<void*> >&)':
/usr/include/liburing.h:745: undefined reference to `__io_uring_get_cqe'
/usr/bin/ld: /usr/local/lib/librocksdb.a(io_posix.o): in function `rocksdb::PosixRandomAccessFile::ReadAsync(rocksdb::FSReadRequest&, rocksdb::IOOptions const&, std::function<void (rocksdb::FSReadRequest const&, void*)>, void*, void**, std::function<void (void*)>*, rocksdb::IODebugContext*)':
/usr/local/src/rocksdb/env/io_posix.cc:901: undefined reference to `io_uring_get_sqe'
/usr/bin/ld: /usr/local/src/rocksdb/env/io_posix.cc:910: undefined reference to `io_uring_submit'
/usr/bin/ld: /usr/local/lib/librocksdb.a(io_posix.o): in function `rocksdb::CreateIOUring()':
/usr/local/src/rocksdb/./env/io_posix.h:272: undefined reference to `io_uring_queue_init'
/usr/bin/ld: /usr/local/lib/librocksdb.a(io_posix.o): in function `rocksdb::PosixRandomAccessFile::MultiRead(rocksdb::FSReadRequest*, unsigned long, rocksdb::IOOptions const&, rocksdb::IODebugContext*)':
/usr/local/src/rocksdb/env/io_posix.cc:674: undefined reference to `io_uring_get_sqe'
/usr/bin/ld: /usr/local/src/rocksdb/env/io_posix.cc:684: undefined reference to `io_uring_submit_and_wait'
/usr/bin/ld: /usr/local/lib/librocksdb.a(io_posix.o): in function `rocksdb::PosixRandomAccessFile::MultiRead(rocksdb::FSReadRequest*, unsigned long, rocksdb::IOOptions const&, rocksdb::IODebugContext*)':
/usr/include/liburing.h:745: undefined reference to `__io_uring_get_cqe'
/usr/bin/ld: /usr/local/lib/librocksdb.a(io_posix.o): in function `rocksdb::CreateIOUring()':
/usr/local/src/rocksdb/./env/io_posix.h:272: undefined reference to `io_uring_queue_init'
/usr/bin/ld: /usr/local/lib/librocksdb.a(io_posix.o): in function `rocksdb::PosixRandomAccessFile::MultiRead(rocksdb::FSReadRequest*, unsigned long, rocksdb::IOOptions const&, rocksdb::IODebugContext*)':
/usr/include/liburing.h:745: undefined reference to `__io_uring_get_cqe'
collect2: error: ld returned 1 exit status
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
Linux host 5.15.0-73-generic #80-Ubuntu SMP
Hello!
First I really like to thank you for your awesome work.
This is outstanding!
We will switch our open source project after months of trying badger, boltdb, pebbledb to your lib I guess.
For our usecase rocksdb it much faster with less CPU and Memory usage.
The only thing I can't find is a way to register the EventListener:
https://github.com/facebook/rocksdb/blob/29e8f6a698acf0d00d88f21a2692fcdd974a74e5/include/rocksdb/listener.h#L358
Do I miss something, or is there no way yet?
Especially the compaction events would be nice for our usecase, to know when compaction is running in the background:
https://github.com/facebook/rocksdb/blob/29e8f6a698acf0d00d88f21a2692fcdd974a74e5/include/rocksdb/listener.h#L400
https://github.com/facebook/rocksdb/blob/29e8f6a698acf0d00d88f21a2692fcdd974a74e5/include/rocksdb/listener.h#L415
Can you give me a hint how to implement this if it is missing?
Thank you for your help!
i would like to manually get the lines of WAL records / content to use as a customized replication log for rocksdb replication.
do you know how i can access wal? i can provide the code of which to do replication once i'm done with it
package main
import (
"fmt"
"github.com/linxGnu/grocksdb"
)
const dbPath = "/tmp/rocksdb"
var allCFs = []string{"default", "cf1", "cf2"}
var keys = [][]byte{[]byte("k1"), []byte("k2")}
var values = [][]byte{[]byte("v1"), []byte("v2")}
func main() {
opts := grocksdb.NewDefaultOptions()
opts.SetCreateIfMissing(true)
opts.SetCreateIfMissingColumnFamilies(true)
dbOpts := grocksdb.NewDefaultTransactionDBOptions()
perTableOpts := make([]*grocksdb.Options, len(allCFs))
for i := range perTableOpts {
perTableOpts[i] = grocksdb.NewDefaultOptions()
perTableOpts[i].SetAtomicFlush(true)
}
db, cfs, err := grocksdb.OpenTransactionDbColumnFamilies(
opts,
dbOpts,
dbPath,
allCFs,
perTableOpts,
)
if err != nil {
panic(fmt.Sprintf("failed to open db : %s", err.Error()))
}
for i := range keys {
err = db.PutCF(grocksdb.NewDefaultWriteOptions(), cfs[1], keys[i], values[i])
if err != nil {
panic(fmt.Sprintf("failed to insert key : %s", err.Error()))
}
}
data, err := db.MultiGetWithCF(grocksdb.NewDefaultReadOptions(), cfs[1], keys...)
if err != nil {
panic(fmt.Sprintf("failed to run MultiGetWithCF : %s", err.Error()))
}
for i := range data {
fmt.Printf("fetched data[%d] is : %s\n", i, data[i].Data())
}
// Moreover if you specify only one non-default CF call to MultiGetWithCF will cause segfault
// This can be easily checked by removing `cf2` from `allCFs`
}
The reason of such behaviour is that here we have only one CF. However in rocksdb lib array of CFs is expected
cfs := make(ColumnFamilyHandles, len(keys))
for i := range keys {
cfs[i] = cf
}
C.rocksdb_transactiondb_multi_get_cf(
transaction.c,
opts.c,
cfs.toCSlice().c(),
C.size_t(len(keys)),
cKeys.c(),
cKeySizes.c(),
vals.c(),
valSizes.c(),
rocksErrs.c(),
)
Do you accept PRs? :)
Note: there are several such places in the lib
Build is failing and package ignores -tags builtin_shared and defaults to only ever using the prebuilt libs in ./dist
Why are binaries included in the SOURCE package?
The libs included are compiled for a system that has AVX2 support, meaning that when used on any system that doesn't have AVX2, we get "Illegal Instruction" errors.
how do i specify default column family name?
handleVer = value.Data()[6:14]
handleVerStr = strconv.FormatUint(my_func.BytesToUint64(handleVer),10)
var cfHandles []*gorocksdb.ColumnFamilyHandle
dbM, cfHandles, err = gorocksdb.OpenDbColumnFamilies(optsM, defaultDBDir+"/"+metaDB, []string{handleVerStr}, []*gorocksdb.Options{optsM})
if err != nil {
fmt.Printf("%v = %s\n",err,handleVerStr) // failed here
}
cfHandle = cfHandles[0]
no way to set dboptions delayed_write_rate
I am runnig this comand as per Documentation:
CGO_CFLAGS="-I/opt/homebrew/Cellar/rocksdb/8.5.4/include" \
CGO_LDFLAGS="-L/opt/homebrew/Cellar/rocksdb/8.5.4 -lrocksdb -lstdc++ -lm -lz -lsnappy -llz4 -lzstd" \
go build
But Geeting this error
/usr/local/go/pkg/tool/darwin_arm64/link: running clang failed: exit status 1
ld: library not found for -lsnappy
clang: error: linker command failed with exit code 1 (use -v to see invocation)
What is the alternate to this ?
Same code and dockerfile, github.com/tecbot/gorocksdb
works, grocksdb not. Does it support alpine?
Development environment works:
Ubuntu 20.04.1 Linux m-pc 5.4.0-58-generic #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
RocksDB 6.14.6 build from source use PORTABLE=1 make shared_lib
github.com/linxGnu/grocksdb v1.6.28(with mod)
Dockerfile not:
FROM registry.cn-shanghai.aliyuncs.com/xm69/alpine-rocksdb-apk AS app
WORKDIR /home
COPY code .
ENV GOPROXY=https://goproxy.cn
RUN set -eux && \
#设置源
echo "http://mirrors.ustc.edu.cn/alpine/edge/main/" > /etc/apk/repositories && \
echo "http://mirrors.ustc.edu.cn/alpine/edge/community/" >> /etc/apk/repositories && \
echo "http://mirrors.ustc.edu.cn/alpine/edge/testing/" >> /etc/apk/repositories && \
apk update && \
#安装RocksDB
apk add --allow-untrusted /rocksdb.apk /rocksdb-dev.apk && \
#打印编译环境
echo "编译环境:$(uname -a)" && \
#安装Golang环境
apk add build-base go && \
#下载并校验Golang模块
go mod download && go mod verify && \
#编译Golang应用
go build -o /home/main
If use Static lib, build failure. Without -tags
build ok, but can not run.
goroutine 0 [idle]:
runtime: unknown pc 0x7f4cf3f92a71
stack: frame={sp:0x7ffe49c331b8, fp:0x0} stack=[0x7ffe49c15480,0x7ffe49c344c0)
00007ffe49c330b8: 000000000000000a 0000000000000000
00007ffe49c330c8: 00007ffe49c33100 0000000000002710
00007ffe49c330d8: 0000000000000000 0000000000000000
00007ffe49c330e8: 00000000009a8000 0000000000000000
00007ffe49c330f8: 0000000000000000 0000000000000000
00007ffe49c33108: 00007f4cf3f70eac 0000000000800000 <crypto/tls.(*Config).Clone+32>
00007ffe49c33118: 0000000000000380 000000000000001b
00007ffe49c33128: 000000000000001b 0000001400000007
00007ffe49c33138: 00000000022f2c90 0000000000000004
00007ffe49c33148: 00007f4cf3f70eac 00000000f3fe0d88
00007ffe49c33158: 0000000000000020 0000000000000000
00007ffe49c33168: 00007f4c00000000 0000000000000026
00007ffe49c33178: 00000000022f46b0 0000000000000001
00007ffe49c33188: 000000000101dbcb 00007ffe49c332b0
00007ffe49c33198: 00000000022f4328 00000000013d59f0
00007ffe49c331a8: 00000000013d7c40 00000000022f4328
00007ffe49c331b8: <00007f4cf3f92bcc 0000000000000000
00007ffe49c331c8: 0000003000000010 00007ffe49c332a0
00007ffe49c331d8: 00007ffe49c331e0 00000000f3fe37c4
00007ffe49c331e8: 000000000000001f 000000000101dbcb
00007ffe49c331f8: 00007f4cf3fb1834 0000000000000016
00007ffe49c33208: 00000000022f4320 0000000000000001
00007ffe49c33218: 0000000000000040 0000000000000001
00007ffe49c33228: 0000000000000001 0000000000000040
00007ffe49c33238: 00000000022f4310 0000000000000008
00007ffe49c33248: 000000000101dbcb 00007ffe49c332b0
00007ffe49c33258: 00007f4cf3f6a713 00007f4cf3fe0a80
00007ffe49c33268: 00007f4cf3f71539 00007ffe49c33370
00007ffe49c33278: 0000000000b11713 00007ffe49c33310
00007ffe49c33288: 0000000000000120 000000000101dbcb
00007ffe49c33298: 0000000000517a38 0000000000000016
00007ffe49c332a8: 00007ffe49c332c0 00007ffe49c332e0
runtime: unknown pc 0x7f4cf3f92a71
stack: frame={sp:0x7ffe49c331b8, fp:0x0} stack=[0x7ffe49c15480,0x7ffe49c344c0)
00007ffe49c330b8: 000000000000000a 0000000000000000
00007ffe49c330c8: 00007ffe49c33100 0000000000002710
00007ffe49c330d8: 0000000000000000 0000000000000000
00007ffe49c330e8: 00000000009a8000 0000000000000000
00007ffe49c330f8: 0000000000000000 0000000000000000
00007ffe49c33108: 00007f4cf3f70eac 0000000000800000 <crypto/tls.(*Config).Clone+32>
00007ffe49c33118: 0000000000000380 000000000000001b
00007ffe49c33128: 000000000000001b 0000001400000007
00007ffe49c33138: 00000000022f2c90 0000000000000004
00007ffe49c33148: 00007f4cf3f70eac 00000000f3fe0d88
00007ffe49c33158: 0000000000000020 0000000000000000
00007ffe49c33168: 00007f4c00000000 0000000000000026
00007ffe49c33178: 00000000022f46b0 0000000000000001
00007ffe49c33188: 000000000101dbcb 00007ffe49c332b0
00007ffe49c33198: 00000000022f4328 00000000013d59f0
00007ffe49c331a8: 00000000013d7c40 00000000022f4328
00007ffe49c331b8: <00007f4cf3f92bcc 0000000000000000
00007ffe49c331c8: 0000003000000010 00007ffe49c332a0
00007ffe49c331d8: 00007ffe49c331e0 00000000f3fe37c4
00007ffe49c331e8: 000000000000001f 000000000101dbcb
00007ffe49c331f8: 00007f4cf3fb1834 0000000000000016
00007ffe49c33208: 00000000022f4320 0000000000000001
00007ffe49c33218: 0000000000000040 0000000000000001
00007ffe49c33228: 0000000000000001 0000000000000040
00007ffe49c33238: 00000000022f4310 0000000000000008
00007ffe49c33248: 000000000101dbcb 00007ffe49c332b0
00007ffe49c33258: 00007f4cf3f6a713 00007f4cf3fe0a80
00007ffe49c33268: 00007f4cf3f71539 00007ffe49c33370
00007ffe49c33278: 0000000000b11713 00007ffe49c33310
00007ffe49c33288: 0000000000000120 000000000101dbcb
00007ffe49c33298: 0000000000517a38 0000000000000016
00007ffe49c332a8: 00007ffe49c332c0 00007ffe49c332e0
goroutine 1 [syscall]:
runtime.cgocall(0x93e930, 0xc0000cfd88, 0xc0000b6580)
/usr/lib/go/src/runtime/cgocall.go:133 +0x5b fp=0xc0000cfd58 sp=0xc0000cfd20 pc=0x53d37b
github.com/linxGnu/grocksdb._Cfunc_rocksdb_open(0x22d06a0, 0x22d2120, 0xc0000b6580, 0x0)
_cgo_gotypes.go:3395 +0x4e fp=0xc0000cfd88 sp=0xc0000cfd58 pc=0x7528ae
github.com/linxGnu/grocksdb.OpenDb.func1(0xc0000afa80, 0x22d2120, 0xc0000b6580, 0xc0000ec000)
/root/go/pkg/mod/github.com/linx!gnu/[email protected]/db.go:37 +0xa7 fp=0xc0000cfdc0 sp=0xc0000cfd88 pc=0x775347
github.com/linxGnu/grocksdb.OpenDb(0xc0000afa80, 0xc0000b4d80, 0xc, 0xc0000b4d80, 0xc, 0x0)
/root/go/pkg/mod/github.com/linx!gnu/[email protected]/db.go:37 +0x7a fp=0xc0000cfe20 sp=0xc0000cfdc0 pc=0x762e3a
main.main()
/home/main.go:673 +0x870 fp=0xc0000cff88 sp=0xc0000cfe20 pc=0x93bff0
runtime.main()
/usr/lib/go/src/runtime/proc.go:204 +0x209 fp=0xc0000cffe0 sp=0xc0000cff88 pc=0x571a69
runtime.goexit()
/usr/lib/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0000cffe8 sp=0xc0000cffe0 pc=0x5a3461
rax 0x0
rbx 0x0
rcx 0x7f4cf3f92a71
rdx 0x0
rdi 0x2
rsi 0x7ffe49c331c0
rbp 0x7ffe49c331c0
rsp 0x7ffe49c331b8
r8 0x7f4cf3fe00c0
r9 0x7f4cf3fdd933
r10 0x8
r11 0x246
r12 0x22f4328
r13 0x13d59f0
r14 0x13d7c40
r15 0x22d26f8
rip 0x7f4cf3f92a71
rflags 0x246
cs 0x33
fs 0x0
gs 0x0
Hi y'all,
I am trying to use this package for my go project and I found its installation/setup procedure quite complicated. By default the package gives following compilation error:
➜ rocksdb go build
# github.com/linxGnu/grocksdb
../../go/pkg/mod/github.com/linx!gnu/[email protected]/array.go:4:11: fatal error: rocksdb/c.h: No such file or directory
4 | // #include "rocksdb/c.h"
| ^~~~~~~~~~~~~
compilation terminated.
To use grocksdb
bindings additional steps are required to compile rocksdb and its dependencies.
A better solution would be to let cgo compile the rocksdb sources. Similar to how sqlite3 go bindings do. cgo just picks *.c files stored together with go sources then compiles and automatically links it.
This requires an amalgamation file (i.e. a single file that contains all project sources) to be provided by rocksdb project (like sqlite does). It turns out rocksdb has a tool for that https://github.com/facebook/rocksdb/blob/main/build_tools/amalgamate.py that aim to achieve what we need.
Hi,
first I would like to say thank you for this package bridging rocksdb to Go.
per the recommendation in https://github.com/facebook/rocksdb/wiki/Track-WAL-in-MANIFEST
to set "track_and_verify_wals_in_manifest" to true (default is false),
I saw that there's no ability to set this option, is it possible to add a getter/setter for this option property?
Thanks!
Hi. Many thanks wholeheartedly for maintaining this repo, as the original Gorocksdb is no longer compatible with the newer version of RocksDB.
I've encountered the following issue during compiling.
/opt/homebrew/cellar/go/1.20.5/libexec/pkg/tool/darwin_arm64/link: running cc failed: exit status 1
ld: library not found for -lsnappy
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Environment: MacOS Ventura with Apple Silicon
Both RocksDB and its dependencies (including snappy) are installed via Homebrew.
Any assistance is greatly appreciated.
HI,
First thank you for maintaining this binding.
I have a small question regarding where one should call Destroy()
on option objects.
I have the following code
opts := grocksdb.NewDefaultOptions()
tableOpts := grocksdb.NewDefaultBlockBasedTableOptions()
cache := grocksdb.NewLRUCache(8 * 1024 * 1024)
tableOpts.SetBlockCache(cache)
opts.SetBlockBasedTableFactory(tableOpts)
db := grocksdb.OpenDb(opts ,"foo.rocksdb")
// ...
db.Close()
At which point in this code can I safely call Destroy()
on opts
, tableOpts
and cache
?
Should I wait after the db is closed or can I destroy them after the db is opened ?
sorry to ask something here. great work. using grocksdb now. will credit your work.
func something() {
var value *grocksdb.Slice
if abc == 1 {
value = dbr.GetCF(rodb,.......)
value.Free()
}
//value.Free() <-- is this needed here? coz it's not been used
}
possible to provide example on how to achieve transactions with column families? and/or optimistic transactions with cf too. thx
mentioned coz i didnt see any test with examples inside.
v1.7.7 rocksdb v7.5.3
can not build in windows
[email protected]\cache.go:79:74: cannot use _Ctype_ulong(s) (value of type _Ctype_ulong) as type _Ctype_ulonglong in variable declaration
[email protected]\options.go:1569:96: cannot use _Ctype_ulong(windowSize) (value of type _Ctype_ulong) as type _Ctype_ulonglong in variable declaration
[email protected]\options.go:1569:117: cannot use _Ctype_ulong(numDelsTrigger) (value of type _Ctype_ulong) as type _Ctype_ulonglong in variable declaration
is this used in production?
Hi. I have gorocksdb as a dependency for other project and experiencing compiling issues that are difficult to pinpoint.
System: Ubuntu 20
This folder has all the includes
echo $CGO_CFLAGS
-I/include/rocksdb/
This is the rocksdb repository with compiled lib
echo $CGO_LDFLAGS
-L/root/dev/rocksdb -lrocksdb -lstdc++ -lm -lz -lbz2 -lsnappy -llz4 -lzstd
Getting:
./cache.go:25:12: could not determine kind of name for C.rocksdb_cache_create_hyper_clock
./cache.go:31:12: could not determine kind of name for C.rocksdb_cache_create_hyper_clock_opts
./cache.go:148:6: could not determine kind of name for C.rocksdb_hyper_clock_cache_options_create
./cache.go:204:2: could not determine kind of name for C.rocksdb_hyper_clock_cache_options_destroy
./cache.go:154:2: could not determine kind of name for C.rocksdb_hyper_clock_cache_options_set_capacity
./cache.go:189:2: could not determine kind of name for C.rocksdb_hyper_clock_cache_options_set_estimated_entry_charge
./cache.go:199:2: could not determine kind of name for C.rocksdb_hyper_clock_cache_options_set_memory_allocator
./cache.go:194:2: could not determine kind of name for C.rocksdb_hyper_clock_cache_options_set_num_shard_bits
./cache.go:142:5: could not determine kind of name for C.rocksdb_hyper_clock_cache_options_t
I can't test iterator with 10.000 keys. I try to run iteration several times on them, using SetIterateUpperBound
in ReadOptions
.
/go/src/github.com/fubss/grocksdb-test-2$ GOFLAGS="-count=1" go test -timeout 30s -run ^TestDrop$ github.com/fubss/grocksdb-test-2 -v
=== RUN TestDrop
-------->NewProvider intialization...
-------->Opening DB and checking format...
-------->RocksDB constructing...
-------->Opening DB...
-------->ParanoidChecks is: true
-------->DB was successfully opened in path: [ /tmp/tests/grocksdbtest2 ]
-------->Checkin if DB is empty...
-------->Checking for emptiness has finished
-------->rocks db IsEmpty()=true
-------->Getting key [_f] from RocksDB...
-------->got data []
-------->Checking for db format at path [/tmp/tests/grocksdbtest2]
format is latest, nothing to do provider_test.go:69: expextedSetup_0
endKey is nil
-------->endKey is nil: eKey=[db1([]byte{0x64, 0x62, 0x31, 0x1})]
-------->Constructing iterator with sKey=[db1([]byte{0x64, 0x62, 0x31, 0x0})] and eKey=[db1([]byte{0x64, 0x62, 0x31, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db1 ([100 98 49 0]: [db1 ([100 98 49 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:69: expextedSetup_1
endKey is nil
-------->endKey is nil: eKey=[db2([]byte{0x64, 0x62, 0x32, 0x1})]
-------->Constructing iterator with sKey=[db2([]byte{0x64, 0x62, 0x32, 0x0})] and eKey=[db2([]byte{0x64, 0x62, 0x32, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db2 ([100 98 50 0]: [db2 ([100 98 50 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:69: expextedSetup_2
endKey is nil
-------->endKey is nil: eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Constructing iterator with sKey=[db3([]byte{0x64, 0x62, 0x33, 0x0})] and eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db3 ([100 98 51 0]: [db3 ([100 98 51 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:69: expextedSetup_3
endKey is nil
-------->endKey is nil: eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Constructing iterator with sKey=[db3([]byte{0x64, 0x62, 0x33, 0x0})] and eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db3 ([100 98 51 0]: [db3 ([100 98 51 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:69: expextedSetup_4
endKey is nil
-------->endKey is nil: eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Constructing iterator with sKey=[db3([]byte{0x64, 0x62, 0x33, 0x0})] and eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db3 ([100 98 51 0]: [db3 ([100 98 51 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
endKey is nil
-------->endKey is nil: eKey=[db1([]byte{0x64, 0x62, 0x31, 0x1})]
-------->Constructing iterator with sKey=[db1([]byte{0x64, 0x62, 0x31, 0x0})] and eKey=[db1([]byte{0x64, 0x62, 0x31, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db1 ([100 98 49 0]: [db1 ([100 98 49 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
-------->WritingBatch.Count()=[20]
endKey is nil
-------->endKey is nil: eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Constructing iterator with sKey=[db3([]byte{0x64, 0x62, 0x33, 0x0})] and eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db3 ([100 98 51 0]: [db3 ([100 98 51 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
-------->WritingBatch.Count()=[8404]
-------->Have removed 8404 entries for channel db3 in rocksdb /tmp/tests/grocksdbtest2
-------->WritingBatch.Count()=[1596]
provider_test.go:106: expextedResults_0
endKey is nil
-------->endKey is nil: eKey=[db1([]byte{0x64, 0x62, 0x31, 0x1})]
-------->Constructing iterator with sKey=[db1([]byte{0x64, 0x62, 0x31, 0x0})] and eKey=[db1([]byte{0x64, 0x62, 0x31, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db1 ([100 98 49 0]: [db1 ([100 98 49 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is not Valid
-------->iterator is not valid anymore
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:106: expextedResults_1
endKey is nil
-------->endKey is nil: eKey=[db2([]byte{0x64, 0x62, 0x32, 0x1})]
-------->Constructing iterator with sKey=[db2([]byte{0x64, 0x62, 0x32, 0x0})] and eKey=[db2([]byte{0x64, 0x62, 0x32, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db2 ([100 98 50 0]: [db2 ([100 98 50 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:106: expextedResults_2
endKey is nil
-------->endKey is nil: eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Constructing iterator with sKey=[db3([]byte{0x64, 0x62, 0x33, 0x0})] and eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db3 ([100 98 51 0]: [db3 ([100 98 51 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is not Valid
-------->iterator is not valid anymore
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
-------->Closing db...
endKey is nil
-------->endKey is nil: eKey=[db2([]byte{0x64, 0x62, 0x32, 0x1})]
-------->Constructing iterator with sKey=[db2([]byte{0x64, 0x62, 0x32, 0x0})] and eKey=[db2([]byte{0x64, 0x62, 0x32, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db2 ([100 98 50 0]: [db2 ([100 98 50 1])]
-------->itr.Err()=[error while obtaining db iterator: rocksdb: closed
github.com/fubss/grocksdb-test-2.(*DB).GetIterator
/home/ivanl/go/src/github.com/fubss/grocksdb-test-2/helper.go:151
github.com/fubss/grocksdb-test-2.(*DBHandle).GetIterator
/home/ivanl/go/src/github.com/fubss/grocksdb-test-2/provider.go:263
github.com/fubss/grocksdb-test-2.(*DBHandle).deleteAll
/home/ivanl/go/src/github.com/fubss/grocksdb-test-2/provider.go:168
github.com/fubss/grocksdb-test-2.TestDrop
/home/ivanl/go/src/github.com/fubss/grocksdb-test-2/provider_test.go:115
testing.tRunner
/usr/local/go/src/testing/testing.go:1259
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1581]. Impossible to create an iterator
-------->Error! Closing iterator...
--- PASS: TestDrop (0.39s)
PASS
ok github.com/fubss/grocksdb-test-2 0.406s
Sometimes test passes, sometimes not. When it doesn't, iterator becomes invalid in unexpected (every time new) place during iteration within 10.000 keys.
itr.Err()
returns nil
in this case.
/go/src/github.com/fubss/grocksdb-test-2$ GOFLAGS="-count=1" go test -timeout 30s -run ^TestDrop$ github.com/fubss/grocksdb-test-2 -v
=== RUN TestDrop
-------->NewProvider intialization...
-------->Opening DB and checking format...
-------->RocksDB constructing...
-------->Opening DB...
-------->ParanoidChecks is: true
-------->DB was successfully opened in path: [ /tmp/tests/grocksdbtest2 ]
-------->Checkin if DB is empty...
-------->Checking for emptiness has finished
-------->rocks db IsEmpty()=true
-------->Getting key [_f] from RocksDB...
-------->got data []
-------->Checking for db format at path [/tmp/tests/grocksdbtest2]
format is latest, nothing to do provider_test.go:69: expextedSetup_0
endKey is nil
-------->endKey is nil: eKey=[db1([]byte{0x64, 0x62, 0x31, 0x1})]
-------->Constructing iterator with sKey=[db1([]byte{0x64, 0x62, 0x31, 0x0})] and eKey=[db1([]byte{0x64, 0x62, 0x31, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db1 ([100 98 49 0]: [db1 ([100 98 49 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:69: expextedSetup_1
endKey is nil
-------->endKey is nil: eKey=[db2([]byte{0x64, 0x62, 0x32, 0x1})]
-------->Constructing iterator with sKey=[db2([]byte{0x64, 0x62, 0x32, 0x0})] and eKey=[db2([]byte{0x64, 0x62, 0x32, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db2 ([100 98 50 0]: [db2 ([100 98 50 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:69: expextedSetup_2
endKey is nil
-------->endKey is nil: eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Constructing iterator with sKey=[db3([]byte{0x64, 0x62, 0x33, 0x0})] and eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db3 ([100 98 51 0]: [db3 ([100 98 51 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:69: expextedSetup_3
endKey is nil
-------->endKey is nil: eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Constructing iterator with sKey=[db3([]byte{0x64, 0x62, 0x33, 0x0})] and eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db3 ([100 98 51 0]: [db3 ([100 98 51 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:69: expextedSetup_4
endKey is nil
-------->endKey is nil: eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Constructing iterator with sKey=[db3([]byte{0x64, 0x62, 0x33, 0x0})] and eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db3 ([100 98 51 0]: [db3 ([100 98 51 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
provider_test.go:129:
Error Trace: provider_test.go:129
provider_test.go:72
Error: Not equal:
expected: 9999
actual : 6356
Test: TestDrop
--- FAIL: TestDrop (0.27s)
FAIL
FAIL github.com/fubss/grocksdb-test-2 0.280s
FAIL
/go/src/github.com/fubss/grocksdb-test-2$
GOFLAGS="-count=1" go test -timeout 30s -run ^TestDrop$ github.com/fubss/grocksdb-test-2 -v
I'm using Ubuntu 20 LTS.
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ivanl/.cache/go-build"
GOENV="/home/ivanl/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ivanl/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ivanl/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17.3"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/ivanl/go/src/github.com/fubss/grocksdb-test-2/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2854272429=/tmp/go-build -gno-record-gcc-switches"
2021/12/15-16:23:43.954005 7f39f45b2ac0 [/db_impl/db_impl_open.cc:184] file size check will be skipped during open.
2021/12/15-16:23:43.954290 7f39f45b2ac0 RocksDB version: 6.22.1
2021/12/15-16:23:43.954304 7f39f45b2ac0 Git sha 51b540921dd7495c9cf2265eb58942dad1f2ef72
2021/12/15-16:23:43.954307 7f39f45b2ac0 Compile date 2021-12-04 14:52:07
2021/12/15-16:23:43.954316 7f39f45b2ac0 DB SUMMARY
2021/12/15-16:23:43.954317 7f39f45b2ac0 DB Session ID: 6449I48JEUTH264QRR4L
2021/12/15-16:23:43.954347 7f39f45b2ac0 SST files in /tmp/tests/grocksdbtest2 dir, Total Num: 0, files:
2021/12/15-16:23:43.954351 7f39f45b2ac0 Write Ahead Log file in /tmp/tests/grocksdbtest2:
2021/12/15-16:23:43.954353 7f39f45b2ac0 Options.error_if_exists: 0
2021/12/15-16:23:43.954355 7f39f45b2ac0 Options.create_if_missing: 1
2021/12/15-16:23:43.954357 7f39f45b2ac0 Options.paranoid_checks: 0
2021/12/15-16:23:43.954359 7f39f45b2ac0 Options.flush_verify_memtable_count: 1
2021/12/15-16:23:43.954360 7f39f45b2ac0 Options.track_and_verify_wals_in_manifest: 0
2021/12/15-16:23:43.954362 7f39f45b2ac0 Options.env: 0x7f39f55571c0
2021/12/15-16:23:43.954364 7f39f45b2ac0 Options.fs: Posix File System
2021/12/15-16:23:43.954366 7f39f45b2ac0 Options.info_log: 0x18aa860
2021/12/15-16:23:43.954368 7f39f45b2ac0 Options.max_file_opening_threads: 16
2021/12/15-16:23:43.954369 7f39f45b2ac0 Options.statistics: (nil)
2021/12/15-16:23:43.954371 7f39f45b2ac0 Options.use_fsync: 0
2021/12/15-16:23:43.954373 7f39f45b2ac0 Options.max_log_file_size: 0
2021/12/15-16:23:43.954374 7f39f45b2ac0 Options.max_manifest_file_size: 1073741824
2021/12/15-16:23:43.954376 7f39f45b2ac0 Options.log_file_time_to_roll: 0
2021/12/15-16:23:43.954378 7f39f45b2ac0 Options.keep_log_file_num: 1000
2021/12/15-16:23:43.954379 7f39f45b2ac0 Options.recycle_log_file_num: 0
2021/12/15-16:23:43.954381 7f39f45b2ac0 Options.allow_fallocate: 1
2021/12/15-16:23:43.954382 7f39f45b2ac0 Options.allow_mmap_reads: 0
2021/12/15-16:23:43.954384 7f39f45b2ac0 Options.allow_mmap_writes: 0
2021/12/15-16:23:43.954385 7f39f45b2ac0 Options.use_direct_reads: 0
2021/12/15-16:23:43.954387 7f39f45b2ac0 Options.use_direct_io_for_flush_and_compaction: 0
2021/12/15-16:23:43.954388 7f39f45b2ac0 Options.create_missing_column_families: 0
2021/12/15-16:23:43.954390 7f39f45b2ac0 Options.db_log_dir:
2021/12/15-16:23:43.954391 7f39f45b2ac0 Options.wal_dir: /tmp/tests/grocksdbtest2
2021/12/15-16:23:43.954393 7f39f45b2ac0 Options.table_cache_numshardbits: 6
2021/12/15-16:23:43.954394 7f39f45b2ac0 Options.WAL_ttl_seconds: 0
2021/12/15-16:23:43.954396 7f39f45b2ac0 Options.WAL_size_limit_MB: 0
2021/12/15-16:23:43.954397 7f39f45b2ac0 Options.max_write_batch_group_size_bytes: 1048576
2021/12/15-16:23:43.954399 7f39f45b2ac0 Options.manifest_preallocation_size: 4194304
2021/12/15-16:23:43.954401 7f39f45b2ac0 Options.is_fd_close_on_exec: 1
2021/12/15-16:23:43.954402 7f39f45b2ac0 Options.advise_random_on_open: 1
2021/12/15-16:23:43.954404 7f39f45b2ac0 Options.db_write_buffer_size: 0
2021/12/15-16:23:43.954406 7f39f45b2ac0 Options.write_buffer_manager: 0x1898340
2021/12/15-16:23:43.954407 7f39f45b2ac0 Options.access_hint_on_compaction_start: 1
2021/12/15-16:23:43.954408 7f39f45b2ac0 Options.new_table_reader_for_compaction_inputs: 0
2021/12/15-16:23:43.954410 7f39f45b2ac0 Options.random_access_max_buffer_size: 1048576
2021/12/15-16:23:43.954411 7f39f45b2ac0 Options.use_adaptive_mutex: 0
2021/12/15-16:23:43.954413 7f39f45b2ac0 Options.rate_limiter: (nil)
2021/12/15-16:23:43.954433 7f39f45b2ac0 Options.sst_file_manager.rate_bytes_per_sec: 0
2021/12/15-16:23:43.954435 7f39f45b2ac0 Options.wal_recovery_mode: 2
2021/12/15-16:23:43.954436 7f39f45b2ac0 Options.enable_thread_tracking: 0
2021/12/15-16:23:43.954438 7f39f45b2ac0 Options.enable_pipelined_write: 0
2021/12/15-16:23:43.954439 7f39f45b2ac0 Options.unordered_write: 0
2021/12/15-16:23:43.954441 7f39f45b2ac0 Options.allow_concurrent_memtable_write: 1
2021/12/15-16:23:43.954442 7f39f45b2ac0 Options.enable_write_thread_adaptive_yield: 1
2021/12/15-16:23:43.954444 7f39f45b2ac0 Options.write_thread_max_yield_usec: 100
2021/12/15-16:23:43.954446 7f39f45b2ac0 Options.write_thread_slow_yield_usec: 3
2021/12/15-16:23:43.954448 7f39f45b2ac0 Options.row_cache: None
2021/12/15-16:23:43.954449 7f39f45b2ac0 Options.wal_filter: None
2021/12/15-16:23:43.954451 7f39f45b2ac0 Options.avoid_flush_during_recovery: 0
2021/12/15-16:23:43.954452 7f39f45b2ac0 Options.allow_ingest_behind: 0
2021/12/15-16:23:43.954454 7f39f45b2ac0 Options.preserve_deletes: 0
2021/12/15-16:23:43.954456 7f39f45b2ac0 Options.two_write_queues: 0
2021/12/15-16:23:43.954457 7f39f45b2ac0 Options.manual_wal_flush: 0
2021/12/15-16:23:43.954459 7f39f45b2ac0 Options.atomic_flush: 0
2021/12/15-16:23:43.954460 7f39f45b2ac0 Options.avoid_unnecessary_blocking_io: 0
2021/12/15-16:23:43.954462 7f39f45b2ac0 Options.persist_stats_to_disk: 0
2021/12/15-16:23:43.954463 7f39f45b2ac0 Options.write_dbid_to_manifest: 0
2021/12/15-16:23:43.954465 7f39f45b2ac0 Options.log_readahead_size: 0
2021/12/15-16:23:43.954466 7f39f45b2ac0 Options.file_checksum_gen_factory: Unknown
2021/12/15-16:23:43.954468 7f39f45b2ac0 Options.best_efforts_recovery: 0
2021/12/15-16:23:43.954469 7f39f45b2ac0 Options.max_bgerror_resume_count: 2147483647
2021/12/15-16:23:43.954471 7f39f45b2ac0 Options.bgerror_resume_retry_interval: 1000000
2021/12/15-16:23:43.954473 7f39f45b2ac0 Options.allow_data_in_errors: 0
2021/12/15-16:23:43.954474 7f39f45b2ac0 Options.db_host_id: hostname
2021/12/15-16:23:43.954476 7f39f45b2ac0 Options.max_background_jobs: 2
2021/12/15-16:23:43.954478 7f39f45b2ac0 Options.max_background_compactions: -1
2021/12/15-16:23:43.954480 7f39f45b2ac0 Options.max_subcompactions: 1
2021/12/15-16:23:43.954481 7f39f45b2ac0 Options.avoid_flush_during_shutdown: 0
2021/12/15-16:23:43.954483 7f39f45b2ac0 Options.writable_file_max_buffer_size: 1048576
2021/12/15-16:23:43.954484 7f39f45b2ac0 Options.delayed_write_rate : 16777216
2021/12/15-16:23:43.954486 7f39f45b2ac0 Options.max_total_wal_size: 0
2021/12/15-16:23:43.954488 7f39f45b2ac0 Options.delete_obsolete_files_period_micros: 21600000000
2021/12/15-16:23:43.954489 7f39f45b2ac0 Options.stats_dump_period_sec: 600
2021/12/15-16:23:43.954491 7f39f45b2ac0 Options.stats_persist_period_sec: 600
2021/12/15-16:23:43.954492 7f39f45b2ac0 Options.stats_history_buffer_size: 1048576
2021/12/15-16:23:43.954493 7f39f45b2ac0 Options.max_open_files: -1
2021/12/15-16:23:43.954495 7f39f45b2ac0 Options.bytes_per_sync: 0
2021/12/15-16:23:43.954496 7f39f45b2ac0 Options.wal_bytes_per_sync: 0
2021/12/15-16:23:43.954498 7f39f45b2ac0 Options.strict_bytes_per_sync: 0
2021/12/15-16:23:43.954499 7f39f45b2ac0 Options.compaction_readahead_size: 0
2021/12/15-16:23:43.954501 7f39f45b2ac0 Options.max_background_flushes: -1
2021/12/15-16:23:43.954502 7f39f45b2ac0 Compression algorithms supported:
2021/12/15-16:23:43.954514 7f39f45b2ac0 kZSTDNotFinalCompression supported: 1
2021/12/15-16:23:43.954518 7f39f45b2ac0 kZSTD supported: 1
2021/12/15-16:23:43.954520 7f39f45b2ac0 kXpressCompression supported: 0
2021/12/15-16:23:43.954521 7f39f45b2ac0 kLZ4HCCompression supported: 1
2021/12/15-16:23:43.954523 7f39f45b2ac0 kLZ4Compression supported: 1
2021/12/15-16:23:43.954525 7f39f45b2ac0 kBZip2Compression supported: 1
2021/12/15-16:23:43.954526 7f39f45b2ac0 kZlibCompression supported: 1
2021/12/15-16:23:43.954528 7f39f45b2ac0 kSnappyCompression supported: 1
2021/12/15-16:23:43.954543 7f39f45b2ac0 Fast CRC32 supported: Supported on x86
2021/12/15-16:23:43.955343 7f39f45b2ac0 [/db_impl/db_impl_open.cc:285] Creating manifest 1
2021/12/15-16:23:43.957575 7f39f45b2ac0 [/version_set.cc:4577] Recovering from manifest file: /tmp/tests/grocksdbtest2/MANIFEST-000001
2021/12/15-16:23:43.957766 7f39f45b2ac0 [/column_family.cc:600] --------------- Options for column family [default]:
2021/12/15-16:23:43.957770 7f39f45b2ac0 Options.comparator: leveldb.BytewiseComparator
2021/12/15-16:23:43.957771 7f39f45b2ac0 Options.merge_operator: None
2021/12/15-16:23:43.957772 7f39f45b2ac0 Options.compaction_filter: None
2021/12/15-16:23:43.957772 7f39f45b2ac0 Options.compaction_filter_factory: None
2021/12/15-16:23:43.957773 7f39f45b2ac0 Options.sst_partitioner_factory: None
2021/12/15-16:23:43.957774 7f39f45b2ac0 Options.memtable_factory: SkipListFactory
2021/12/15-16:23:43.957775 7f39f45b2ac0 Options.table_factory: BlockBasedTable
2021/12/15-16:23:43.957802 7f39f45b2ac0 table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x18a6af0)
cache_index_and_filter_blocks: 0
cache_index_and_filter_blocks_with_high_priority: 1
pin_l0_filter_and_index_blocks_in_cache: 0
pin_top_level_index_and_filter: 1
index_type: 0
data_block_index_type: 0
index_shortening: 1
data_block_hash_table_util_ratio: 0.750000
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x18a6b40
block_cache_name: LRUCache
block_cache_options:
capacity : 8388608
num_shard_bits : 4
strict_capacity_limit : 0
memory_allocator : None
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
metadata_block_size: 4096
partition_filters: 0
use_delta_encoding: 1
filter_policy: nullptr
whole_key_filtering: 1
verify_compression: 0
read_amp_bytes_per_bit: 0
format_version: 5
enable_index_compression: 1
block_align: 0
max_auto_readahead_size: 262144
prepopulate_block_cache: 0
2021/12/15-16:23:43.957804 7f39f45b2ac0 Options.write_buffer_size: 67108864
2021/12/15-16:23:43.957804 7f39f45b2ac0 Options.max_write_buffer_number: 2
2021/12/15-16:23:43.957807 7f39f45b2ac0 Options.compression: Snappy
2021/12/15-16:23:43.957808 7f39f45b2ac0 Options.bottommost_compression: Disabled
2021/12/15-16:23:43.957809 7f39f45b2ac0 Options.prefix_extractor: nullptr
2021/12/15-16:23:43.957809 7f39f45b2ac0 Options.memtable_insert_with_hint_prefix_extractor: nullptr
2021/12/15-16:23:43.957810 7f39f45b2ac0 Options.num_levels: 7
2021/12/15-16:23:43.957811 7f39f45b2ac0 Options.min_write_buffer_number_to_merge: 1
2021/12/15-16:23:43.957812 7f39f45b2ac0 Options.max_write_buffer_number_to_maintain: 0
2021/12/15-16:23:43.957812 7f39f45b2ac0 Options.max_write_buffer_size_to_maintain: 0
2021/12/15-16:23:43.957813 7f39f45b2ac0 Options.bottommost_compression_opts.window_bits: -14
2021/12/15-16:23:43.957814 7f39f45b2ac0 Options.bottommost_compression_opts.level: 32767
2021/12/15-16:23:43.957815 7f39f45b2ac0 Options.bottommost_compression_opts.strategy: 0
2021/12/15-16:23:43.957816 7f39f45b2ac0 Options.bottommost_compression_opts.max_dict_bytes: 0
2021/12/15-16:23:43.957816 7f39f45b2ac0 Options.bottommost_compression_opts.zstd_max_train_bytes: 0
2021/12/15-16:23:43.957831 7f39f45b2ac0 Options.bottommost_compression_opts.parallel_threads: 1
2021/12/15-16:23:43.957832 7f39f45b2ac0 Options.bottommost_compression_opts.enabled: false
2021/12/15-16:23:43.957833 7f39f45b2ac0 Options.bottommost_compression_opts.max_dict_buffer_bytes: 0
2021/12/15-16:23:43.957834 7f39f45b2ac0 Options.compression_opts.window_bits: -14
2021/12/15-16:23:43.957835 7f39f45b2ac0 Options.compression_opts.level: 32767
2021/12/15-16:23:43.957835 7f39f45b2ac0 Options.compression_opts.strategy: 0
2021/12/15-16:23:43.957836 7f39f45b2ac0 Options.compression_opts.max_dict_bytes: 0
2021/12/15-16:23:43.957837 7f39f45b2ac0 Options.compression_opts.zstd_max_train_bytes: 0
2021/12/15-16:23:43.957838 7f39f45b2ac0 Options.compression_opts.parallel_threads: 1
2021/12/15-16:23:43.957838 7f39f45b2ac0 Options.compression_opts.enabled: false
2021/12/15-16:23:43.957839 7f39f45b2ac0 Options.compression_opts.max_dict_buffer_bytes: 0
2021/12/15-16:23:43.957840 7f39f45b2ac0 Options.level0_file_num_compaction_trigger: 4
2021/12/15-16:23:43.957840 7f39f45b2ac0 Options.level0_slowdown_writes_trigger: 20
2021/12/15-16:23:43.957841 7f39f45b2ac0 Options.level0_stop_writes_trigger: 36
2021/12/15-16:23:43.957842 7f39f45b2ac0 Options.target_file_size_base: 67108864
2021/12/15-16:23:43.957843 7f39f45b2ac0 Options.target_file_size_multiplier: 1
2021/12/15-16:23:43.957843 7f39f45b2ac0 Options.max_bytes_for_level_base: 268435456
2021/12/15-16:23:43.957844 7f39f45b2ac0 Options.level_compaction_dynamic_level_bytes: 0
2021/12/15-16:23:43.957845 7f39f45b2ac0 Options.max_bytes_for_level_multiplier: 10.000000
2021/12/15-16:23:43.957847 7f39f45b2ac0 Options.max_bytes_for_level_multiplier_addtl[0]: 1
2021/12/15-16:23:43.957848 7f39f45b2ac0 Options.max_bytes_for_level_multiplier_addtl[1]: 1
2021/12/15-16:23:43.957848 7f39f45b2ac0 Options.max_bytes_for_level_multiplier_addtl[2]: 1
2021/12/15-16:23:43.957849 7f39f45b2ac0 Options.max_bytes_for_level_multiplier_addtl[3]: 1
2021/12/15-16:23:43.957850 7f39f45b2ac0 Options.max_bytes_for_level_multiplier_addtl[4]: 1
2021/12/15-16:23:43.957850 7f39f45b2ac0 Options.max_bytes_for_level_multiplier_addtl[5]: 1
2021/12/15-16:23:43.957851 7f39f45b2ac0 Options.max_bytes_for_level_multiplier_addtl[6]: 1
2021/12/15-16:23:43.957852 7f39f45b2ac0 Options.max_sequential_skip_in_iterations: 8
2021/12/15-16:23:43.957852 7f39f45b2ac0 Options.max_compaction_bytes: 1677721600
2021/12/15-16:23:43.957853 7f39f45b2ac0 Options.arena_block_size: 1048576
2021/12/15-16:23:43.957854 7f39f45b2ac0 Options.soft_pending_compaction_bytes_limit: 68719476736
2021/12/15-16:23:43.957855 7f39f45b2ac0 Options.hard_pending_compaction_bytes_limit: 274877906944
2021/12/15-16:23:43.957855 7f39f45b2ac0 Options.rate_limit_delay_max_milliseconds: 100
2021/12/15-16:23:43.957856 7f39f45b2ac0 Options.disable_auto_compactions: 0
2021/12/15-16:23:43.957857 7f39f45b2ac0 Options.compaction_style: kCompactionStyleLevel
2021/12/15-16:23:43.957858 7f39f45b2ac0 Options.compaction_pri: kMinOverlappingRatio
2021/12/15-16:23:43.957859 7f39f45b2ac0 Options.compaction_options_universal.size_ratio: 1
2021/12/15-16:23:43.957860 7f39f45b2ac0 Options.compaction_options_universal.min_merge_width: 2
2021/12/15-16:23:43.957861 7f39f45b2ac0 Options.compaction_options_universal.max_merge_width: 4294967295
2021/12/15-16:23:43.957861 7f39f45b2ac0 Options.compaction_options_universal.max_size_amplification_percent: 200
2021/12/15-16:23:43.957862 7f39f45b2ac0 Options.compaction_options_universal.compression_size_percent: -1
2021/12/15-16:23:43.957863 7f39f45b2ac0 Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
2021/12/15-16:23:43.957863 7f39f45b2ac0 Options.compaction_options_fifo.max_table_files_size: 1073741824
2021/12/15-16:23:43.957872 7f39f45b2ac0 Options.compaction_options_fifo.allow_compaction: 0
2021/12/15-16:23:43.957875 7f39f45b2ac0 Options.table_properties_collectors:
2021/12/15-16:23:43.957876 7f39f45b2ac0 Options.inplace_update_support: 0
2021/12/15-16:23:43.957877 7f39f45b2ac0 Options.inplace_update_num_locks: 10000
2021/12/15-16:23:43.957877 7f39f45b2ac0 Options.memtable_prefix_bloom_size_ratio: 0.000000
2021/12/15-16:23:43.957879 7f39f45b2ac0 Options.memtable_whole_key_filtering: 0
2021/12/15-16:23:43.957879 7f39f45b2ac0 Options.memtable_huge_page_size: 0
2021/12/15-16:23:43.957880 7f39f45b2ac0 Options.bloom_locality: 0
2021/12/15-16:23:43.957881 7f39f45b2ac0 Options.max_successive_merges: 0
2021/12/15-16:23:43.957881 7f39f45b2ac0 Options.optimize_filters_for_hits: 0
2021/12/15-16:23:43.957882 7f39f45b2ac0 Options.paranoid_file_checks: 0
2021/12/15-16:23:43.957883 7f39f45b2ac0 Options.force_consistency_checks: 1
2021/12/15-16:23:43.957883 7f39f45b2ac0 Options.report_bg_io_stats: 0
2021/12/15-16:23:43.957884 7f39f45b2ac0 Options.ttl: 2592000
2021/12/15-16:23:43.957885 7f39f45b2ac0 Options.periodic_compaction_seconds: 0
2021/12/15-16:23:43.957886 7f39f45b2ac0 Options.enable_blob_files: false
2021/12/15-16:23:43.957886 7f39f45b2ac0 Options.min_blob_size: 0
2021/12/15-16:23:43.957887 7f39f45b2ac0 Options.blob_file_size: 268435456
2021/12/15-16:23:43.957888 7f39f45b2ac0 Options.blob_compression_type: NoCompression
2021/12/15-16:23:43.957889 7f39f45b2ac0 Options.enable_blob_garbage_collection: false
2021/12/15-16:23:43.957889 7f39f45b2ac0 Options.blob_garbage_collection_age_cutoff: 0.250000
2021/12/15-16:23:43.960088 7f39f45b2ac0 [/version_set.cc:4617] Recovered from manifest file:/tmp/tests/grocksdbtest2/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0
2021/12/15-16:23:43.960094 7f39f45b2ac0 [/version_set.cc:4632] Column family [default] (ID 0), log number is 0
2021/12/15-16:23:43.960191 7f39f45b2ac0 [/version_set.cc:4123] Creating manifest 4
2021/12/15-16:23:43.962209 7f39f45b2ac0 [DEBUG] [/db_impl/db_impl_files.cc:334] [JOB 1] Delete /tmp/tests/grocksdbtest2/MANIFEST-000001 type=3 #1 -- OK
2021/12/15-16:23:43.964586 7f39f45b2ac0 [/db_impl/db_impl_open.cc:1759] SstFileManager instance 0x18a9140
SetIterateUpperBound
. When I comment them, Iterator works stable. But I want to use all iterator features with upper bound setting.I'm new in DBs, golang, github, sorry if anything wrong.
I've saw that there's an option to manual compact specific files with the API of CompactFiles
https://github.com/facebook/rocksdb/wiki/Manual-Compaction#compactfiles
is it possible to implement it in this library?
Thanks!
possible to provide instructions on how to use terarkdb with grocksdb?
installation for ubuntu 20.04 environment
can you pls provide the terarkdb docker file for reference? thx in advance.
Possible to provide the code to add this function in?
code:
package main
import (
"os"
"github.com/linxGnu/grocksdb"
)
func main() {
dir, err := os.MkdirTemp(".", "t-")
CheckErr(err)
options := grocksdb.NewDefaultOptions()
options.SetCreateIfMissing(true)
options.SetPrefixExtractor(grocksdb.NewNoopPrefixTransform())
db, err := grocksdb.OpenDb(options, dir)
CheckErr(err)
db.Close()
options.Destroy()
}
func CheckErr(err error) {
if err != nil {
panic(err)
}
}
result:
i did the dbr.CompactRange(grocksdb.Range{nil, nil}) to trigger manual compaction but hv error below
dbr.CompactRange(grocksdb.Range{nil, nil}) (no value) used as value
i saw the code in _test so just copied and paste but doesnt work.
how to make it work? thx in advance
for batching writes (PUT), do I put the PutCF before the mutex lock? (faster) or after the mutex lock? can i do it before the mutex lock? the second version. is it ok?
first version
b0 := grocksdb.NewWriteBatch()
mutexCfhTK[((*i.(*[][]uint8))[2][0])+(((*i.(*[][]uint8))[4][0])%3)].Lock()
mutexCfhTK[((*i.(*[][]uint8))[3][0])+(((*i.(*[][]uint8))[4][0])%3)].Lock()
b0.PutCF(cfhTK[((*i.(*[][]uint8))[2][0])+((((*i.(*[][]uint8))[4][0]))%3)], (*i.(*[][]uint8))[5], (*i.(*[][]uint8))[6])
b0.PutCF(cfhTK[((*i.(*[][]uint8))[3][0])+((((*i.(*[][]uint8))[4][0]))%3)], (*i.(*[][]uint8))[6][5:], (*i.(*[][]uint8))[7])
dbr[((*i.(*[][]uint8))[1][0])].Write(wodb[((*i.(*[][]uint8))[1][0])], b0)
mutexCfhTK[((*i.(*[][]uint8))[3][0])+(((*i.(*[][]uint8))[4][0])%3)].Unlock()
mutexCfhTK[((*i.(*[][]uint8))[2][0])+(((*i.(*[][]uint8))[4][0])%3)].Unlock()
b0.Destroy()
second version
b0 := grocksdb.NewWriteBatch()
b0.PutCF(cfhTK[((*i.(*[][]uint8))[2][0])+((((*i.(*[][]uint8))[4][0]))%3)], (*i.(*[][]uint8))[5], (*i.(*[][]uint8))[6])
b0.PutCF(cfhTK[((*i.(*[][]uint8))[3][0])+((((*i.(*[][]uint8))[4][0]))%3)], (*i.(*[][]uint8))[6][5:], (*i.(*[][]uint8))[7])
mutexCfhTK[((*i.(*[][]uint8))[2][0])+(((*i.(*[][]uint8))[4][0])%3)].Lock()
mutexCfhTK[((*i.(*[][]uint8))[3][0])+(((*i.(*[][]uint8))[4][0])%3)].Lock()
dbr[((*i.(*[][]uint8))[1][0])].Write(wodb[((*i.(*[][]uint8))[1][0])], b0)
mutexCfhTK[((*i.(*[][]uint8))[3][0])+(((*i.(*[][]uint8))[4][0])%3)].Unlock()
mutexCfhTK[((*i.(*[][]uint8))[2][0])+(((*i.(*[][]uint8))[4][0])%3)].Unlock()
b0.Destroy()
how do u do
v:=value.Data()
value.Free()
fmt.Printf("v = %v\n",v) // <- from here onwards, "v" is lost. how to prevent v from being lost?
this shld be mentioned as part of the caveats of this module coz it took me some time to realise this was the case and it was not obvious
value, _ := db.Get(ro, k)
//v := value.Data()
b := make([]byte,value.Size())
copy(b,value.Data())
//defer value.Free()
//fmt.Printf("gv = %v = %v = %v\n",v,value.Data(),b)
value.Free()
//fmt.Printf("gv1 = %v = %v = %v\n",v,value.Data(),b)
return b
I met this bug when I start the go test to run the inner unit test I got the error told me that I have no linked with Lz4 or other Lib, Why cause this error ?
Comparator interface needs Native method
Native() *C.rocksdb_comparator_t
Cgo translates C types into equivalent unexported Go types. Because the translations are unexported, a Go package should not expose C types in its exported API: a C type used in one Go package is different from the same C type used in another.
Because of this, it is not possible to implement this interface.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.