wuxb45/remixdb

RemixDB: A read- and write-optimized concurrent KV store. Fast point and range queries. Extremely low write-amplification.

remixdb's Issues

Optional hash filters in REMIX

Using 1-byte-per-key hash tags in REMIX can reduce lookup cost to about 1.1 IO per operation. It's also cache friendly.

Aborts with wsl2

The current Makefile compiles with liburing enabled. But liburing is not implemented in wsl2. When running remixdb tests in wsl2 it will get error 38 ENOSYS (Function not implemented), which will crash with a stack trace printed out.

Possible workarounds include running in a different environment or manually disabling liburing in the build system or source code.

compile error

First of all, thank you for sharing your works : )

I tried to compile with clang-12 and kernel version 5.12.10 , however, I still see errors

[root@localhost remixdb]# make M=j xdbdemo.out
xdbdemo.out <= xdbdemo.c lib.c kv.c wh.c blkio.c sst.c xdb.c 
clang -std=gnu18   -march=native -mtune=native -pthread -Wall -Wextra -Wshadow  -fno-builtin-memcpy -fno-builtin-memmove -fno-builtin-memcmp -DNDEBUG -g3 -O3 -flto -fno-stack-protector -ferror-limit=3  -rdynamic -o xdbdemo.out xdbdemo.c lib.c kv.c wh.c blkio.c sst.c xdb.c  -lm -ljemalloc -lrt
lib.c:710:59: error: use of undeclared identifier 'MAP_HUGE_SHIFT'
  return pages_do_alloc(sz, MAP_PRIVATE | MAP_ANONYMOUS | PAGES_FLAGS_1G);
                                                          ^
lib.c:696:47: note: expanded from macro 'PAGES_FLAGS_1G'
#define PAGES_FLAGS_1G ((MAP_HUGETLB | (30 << MAP_HUGE_SHIFT)))
                                              ^
lib.c:724:59: error: use of undeclared identifier 'MAP_HUGE_SHIFT'
  return pages_do_alloc(sz, MAP_PRIVATE | MAP_ANONYMOUS | PAGES_FLAGS_2M);
                                                          ^
lib.c:697:47: note: expanded from macro 'PAGES_FLAGS_2M'
#define PAGES_FLAGS_2M ((MAP_HUGETLB | (21 << MAP_HUGE_SHIFT)))
                                              ^
2 errors generated.
make: *** [Makefile.common:180: xdbdemo.out] Error 1

And I add header in lib.h

#include <asm-generic/mman-common.h>

and new errors:

[root@localhost remixdb]# make M=j xdbdemo.out
xdbdemo.out <= xdbdemo.c lib.c kv.c wh.c blkio.c sst.c xdb.c 
clang -std=gnu18   -march=native -mtune=native -pthread -Wall -Wextra -Wshadow  -fno-builtin-memcpy -fno-builtin-memmove -fno-builtin-memcmp -DNDEBUG -g3 -O3 -flto -fno-stack-protector -ferror-limit=3  -rdynamic -o xdbdemo.out xdbdemo.c lib.c kv.c wh.c blkio.c sst.c xdb.c  -lm -ljemalloc -lrt
lib.c:2015:10: error: always_inline function '_pext_u32' requires target feature 'bmi2', but would be inlined into function 'm128_movemask_u16' that is compiled without support for 'bmi2'
  return _pext_u32((u32)_mm_movemask_epi8(v), 0xaaaau);
         ^
1 error generated.
make: *** [Makefile.common:180: xdbdemo.out] Error 1

Benchmark Results

YCSB benchmark results as of 7/14/2021. 16B keys, 120B values, 2.02 billion keys, no compression; 4GB block cache (user space), max 4GB memtables, max 8GB WALs. The loading phase only uses one user thread.

About the index file

Hello！ I am excited to read your paper.
But may i ask some questions?
Where are the remix index files stored?
I opened up wal1 but only found the key-value pairs.
what is 000.ssty used for?
I observed that the content in that file are constant.

Rust 实现Remix File

您好，不知道这样是否冒昧，我在计划使用rust实现一个数据库，看了您remixdb的论文觉得非常牛逼，我想把这个设计引入到我的设计中，您看如果方便的话，能否给一个您的邮箱或者微信，有一些问题想要请教，可以随时沟通。