theoanab/silk-usenixatc2019

Code for "SILK: Preventing Latency Spikes in Log-Structured Merge Key-Value Stores" published in USENIX ATC 19

silk-usenixatc2019's Issues

Exact Parameters of db_bench for ycsb{a-f} workloads

Hi,

I am trying to use your modified db_bench which has ycsb benchmarks settings unlike the main Rocksdb repo's db_bench. Can you please specify which parameters of db_bench I should use to reproduce the same experiment setting presented in your paper?

I tried the following parameter and found the following results:

>>> ./db_bench --benchmarks="fillseq,ycsbwklda"
RocksDB:    version 5.7
Date:       Wed Apr 29 03:43:50 2020
CPU:        80 * Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz
CPUCache:   28160 KB
... finished 1000000 ops
----->>>>> Set priority 0 for -1 threads
------> TOTAL THREADS LIMIT total_threads_limit_ 0

PRIORITY Pool 0 : 0
PRIORITY Pool 1 : 1
PRIORITY Pool 2 : 2
----->>>>> Set priority 2 for -1 threads
------> TOTAL THREADS LIMIT total_threads_limit_ 0

PRIORITY Pool 0 : 0
PRIORITY Pool 1 : 1
PRIORITY Pool 2 : 2
HLL-9900------> TOTAL THREADS LIMIT total_threads_limit_ 1

------> TOTAL THREADS LIMIT total_threads_limit_ 1

Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
>>>> WriteController rate limiter low_pri_rate_limiter_
>>>> DBImpl constructor low_pri_write_rate_limiter_
[COMPACTION-QUEUES] Compactions waiting: 0
[COMPACTION-QUEUES] Compactions waiting: 0
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    1000000
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    110.6 MB (estimated)
FileSize:   62.9 MB (estimated)
Write rate: 0 bytes/second
Read rate: 0 ops/second
Compression: Snappy
Memtablerep: skip_list
Perf Level: 1
WARNING: Assertions are enabled; benchmarks unnecessarily slow
------------------------------------------------
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
>>>> WriteController rate limiter low_pri_rate_limiter_
>>>> DBImpl constructor low_pri_write_rate_limiter_
[COMPACTION-QUEUES] Compactions waiting: 0
[COMPACTION-QUEUES] Compactions waiting: 0
DB path: [/tmp/rocksdbtest-99541262/dbbench]
[COMPACTION-QUEUES] Compactions waiting: 0
------> TOTAL THREADS LIMIT total_threads_limit_ 1

[COMPACTION-QUEUES] Compactions waiting: 0
[COMPACTION-QUEUES] Compactions waiting: 0
[COMPACTION-QUEUES] Compactions waiting: 0
------> TOTAL THREADS LIMIT total_threads_limit_ 1

fillseq      :       2.490 micros/op 401565 ops/sec;   44.4 MB/s
DB path: [/tmp/rocksdbtest-99541262/dbbench]
ycsbwklda    :   94619.000 micros/op 10 ops/sec; ( reads:0 writes:0 total:1000000 found:0)
[COMPACTION-QUEUES] Compactions waiting: 0

Thanks,
Tanvir.

Run db_bench with readrandom have error

I find the reason of it, but I don't understand why don't find key with going to disk.

Status DBImpl::GetImpl( ...) {
...
if (!done) {
    //s = Status::TryAgain(); // don't go to disk. See what happens with the latency
    //return s;
    PERF_TIMER_GUARD(get_from_output_files_time);
    sv->current->Get(read_options, lkey, pinnable_val, &s, &merge_context,
                     &range_del_agg, value_found);
    RecordTick(stats_, MEMTABLE_MISS);
    s = Status::TryAgain();
    //READ from DISK <-- shouldn't do this. Should return and re-schedule in a different queue. 
}
...
}

remove s = Status::TryAgain(); can run normally.

Want to read paper

Hi~
I am very interested in your work.
Can I read your paper in advance?

error: ld returned 1 exit status

hello! I have some problems. when I run make db_bench -j24 In the root directory, I get this error:

tools/db_bench_tool.o: In function `rocksdb::db_bench_tool(int, char**)':
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:6750: undefined reference to `google::SetUsageMessage(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:6754: undefined reference to `google::ParseCommandLineFlags(int*, char***, bool)'
tools/db_bench_tool.o: In function `__static_initialization_and_destruction_0(int, int) [clone .constprop.0]':
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:88: undefined reference to `google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*)'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:211: undefined reference to `google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*)'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:213: undefined reference to `google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*)'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:215: undefined reference to `google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*)'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:220: undefined reference to `google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*)'
tools/db_bench_tool.o:/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:225: more undefined references to `google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*)' follow
tools/db_bench_tool.o: In function `__static_initialization_and_destruction_0(int, int) [clone .constprop.0]':
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:365: undefined reference to `google::RegisterFlagValidator(unsigned long const*, bool (*)(char const*, unsigned long))'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:368: undefined reference to `google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*)'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:374: undefined reference to `google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*)'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:378: undefined reference to `google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*)'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:381: undefined reference to `google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*)'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:385: undefined reference to `google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*)'
tools/db_bench_tool.o:/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:388: more undefined references to `google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*)' follow
tools/db_bench_tool.o: In function `__static_initialization_and_destruction_0(int, int) [clone .constprop.0]':
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:748: undefined reference to `google::RegisterFlagValidator(int const*, bool (*)(char const*, int))'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:750: undefined reference to `google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*)'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:764: undefined reference to `google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*)'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:767: undefined reference to `google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*)'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:770: undefined reference to `google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*)'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:774: undefined reference to `google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*)'
tools/db_bench_tool.o:/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:777: more undefined references to `google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*)' follow
tools/db_bench_tool.o: In function `__static_initialization_and_destruction_0(int, int) [clone .constprop.0]':
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:1011: undefined reference to `google::RegisterFlagValidator(double const*, bool (*)(char const*, double))'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:1014: undefined reference to `google::RegisterFlagValidator(double const*, bool (*)(char const*, double))'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:1017: undefined reference to `google::RegisterFlagValidator(int const*, bool (*)(char const*, int))'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:1020: undefined reference to `google::RegisterFlagValidator(int const*, bool (*)(char const*, int))'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:1023: undefined reference to `google::RegisterFlagValidator(int const*, bool (*)(char const*, int))'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:1027: undefined reference to `google::RegisterFlagValidator(int const*, bool (*)(char const*, int))'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:1029: undefined reference to `google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*)'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:1033: undefined reference to `google::RegisterFlagValidator(int const*, bool (*)(char const*, int))'
/home/chunpu/silk/silk2/SILK-USENIXATC2019/tools/db_bench_tool.cc:1035: undefined reference to `google::RegisterFlagValidator(int const*, bool (*)(char const*, int))'
collect2: error: ld returned 1 exit status
make: *** [db_bench] Error 1

In the file Makefile, I add -lgflags in line 12,13:

CFLAGS += ${EXTRA_CFLAGS} -lgflags
CXXFLAGS += ${EXTRA_CXXFLAGS} -lgflags

but it does not help, This error still exists.
In addition, I am wondering how to conduct the experiment in your paper.
I would appreciate it if you give me some help.

failed to run on ycsb while using multithread workloads

Hello! I am trying to test SILK using ycsb benchmarks. But when I add parameters like "-thread=2" (Which means the number of client threads. This is often done to increase the amount of load offered against the database) , it will end like this:

delayed reclaim ratio=0.013858 size=2MiB/s,w=1366MiB/s][r=8381,w=8418 IOPS][eta 02m:30s]
2022-04-01 02:26:11:949 30 sec: 85489 operations; 0 current ops/sec; est completion in 58 minutes [READ: Count=0, Max=0, Min=9223372036854775807, Avg=�, 90=0, 99=0, 99.9=0, 99.99=0] [UPDATE: Count=0, Max=0, Min=9223372036854775807, Avg=�, 90=0, 99=0, 99.9=0, 99.99=0] 
2022-04-01 02:26:13:834 30 sec: 71531 operations; 0 current ops/sec; est completion in 1 hour 9 minutes [READ: Count=0, Max=0, Min=9223372036854775807, Avg=�, 90=0, 99=0, 99.9=0, 99.99=0] [UPDATE: Count=0, Max=0, Min=9223372036854775807, Avg=�, 90=0, 99=0, 99.9=0, 99.99=0] 
2022-04-01 02:26:15:852 30 sec: 561527 operations; 11536.2 current ops/sec; est completion in 8 minutes [READ: Count=57571, Max=26175, Min=3, Avg=64.89, 90=130, 99=240, 99.9=1294, 99.99=11175] [UPDATE: Count=57699, Max=16143, Min=11, Avg=88.3, 90=159, 99=271, 99.9=1727, 99.99=8151] 
2022-04-01 02:26:17:877 30 sec: 144130 operations; 0 current ops/sec; est completion in 34 minutes [READ: Count=0, Max=0, Min=9223372036854775807, Avg=�, 90=0, 99=0, 99.9=0, 99.99=0] [UPDATE: Count=0, Max=0, Min=9223372036854775807, Avg=�, 90=0, 99=0, 99.9=0, 99.99=0] 
2022-04-01 02:26:21:949 40 sec: 85489 operations; 0 current ops/sec; est completion in 1 hour 17 minutes [READ: Count=0, Max=0, Min=9223372036854775807, Avg=�, 90=0, 99=0, 99.9=0, 99.99=0] [UPDATE: Count=0, Max=0, Min=9223372036854775807, Avg=�, 90=0, 99=0, 99.9=0, 99.99=0] 
delayed reclaim ratio=0.010313 size=2MiB/s,w=1287MiB/s][r=7919,w=7879 IOPS][eta 02m:17s]
2022-04-01 02:26:23:834 40 sec: 71531 operations; 0 current ops/sec; est completion in 1 hour 32 minutes [READ: Count=0, Max=0, Min=9223372036854775807, Avg=�, 90=0, 99=0, 99.9=0, 99.99=0] [UPDATE: Count=0, Max=0, Min=9223372036854775807, Avg=�, 90=0, 99=0, 99.9=0, 99.99=0] 
2022-04-01 02:26:25:852 40 sec: 561527 operations; 0 current ops/sec; est completion in 11 minutes [READ: Count=0, Max=0, Min=9223372036854775807, Avg=�, 90=0, 99=0, 99.9=0, 99.99=0] [UPDATE: Count=0, Max=0, Min=9223372036854775807, Avg=�, 90=0, 99=0, 99.9=0, 99.99=0]
...

As you see, throughput will suddenly go to zero. I wonder why this happen. Can you give me some help? Thanks.

on which commit of rocksdb is based the fork?

I just watched the video of your presentation on Youtube . It can be interresting to port some of the changes on later versions of RocksDB. Can you let me know on which commit ID this fork is based? Or maybe just point which parts have been edited for it?