pmwkaa/ioarena

Embedded storage benchmarking tool

ioarena's Issues

pthread barriers are not implemented in OS X

@leo-yuriev
@pmwkaa

Ioarena is broken on OS X, errors are as follow:

src/ia.h:16:2: error: unknown type name 'pthread_barrier_t'
  pthread_barrier_t barrier_start;

Here is an implementation:
http://blog.albertarmea.com/post/47089939939/using-pthreadbarrier-on-mac-os-x

Only dummy that works, It looks like there is some tweeking needs to be done

vagrant@vagrant-ubuntu-trusty:~/projects/ioarena/build$ for i in sophia leveldb rocksdb wiredtiger forestdb lmdb mdbx dummy ; do echo "Storage engine:$i"; src/ioarena -D $i -B get -n 5000 ;done
Storage engine:sophia
IOARENA (embedded storage benchmarking)

error: unknown database driver 'sophia'
Storage engine:leveldb
IOARENA (embedded storage benchmarking)

error: unknown database driver 'leveldb'
Storage engine:rocksdb
IOARENA (embedded storage benchmarking)

error: unknown database driver 'rocksdb'
Storage engine:wiredtiger
IOARENA (embedded storage benchmarking)

error: unknown database driver 'wiredtiger'
Storage engine:forestdb
IOARENA (embedded storage benchmarking)

error: unknown database driver 'forestdb'
Storage engine:lmdb
IOARENA (embedded storage benchmarking)

error: unknown database driver 'lmdb'
Storage engine:mdbx
IOARENA (embedded storage benchmarking)

error: unknown database driver 'mdbx'
Storage engine:dummy
IOARENA (embedded storage benchmarking)

configuration:
engine = dummy
datadir = ./_ioarena
benchmark = get
durability = lazy
wal = indef
operations = 5000
key size = 16
value size = 32
binary = no
batch length = 500
continuous = no

nessDB driver is invalid

Briefly, it NOT implements the basic mandatory operations.
It just do nothing:

ioarena/src/ia_nessdb.c

Lines 135 to 138 in 892c50e

	case IA_DELETE:
	case IA_ITERATE:
	case IA_GET:
	rc = 0;

Moreover, the nessDB API don't provide/export such functions.
In other words, nessDB is NOT a ready engine, which could be used or tested here.

So, I suggest to drop the nessDB driver from the ioarena (with the hope that it will be added again when unacceptable flaws will be eliminated).

On the other hand, maybe @BohuTANG decide to fix this in a few days?

.travis.yml modern

https://travis-ci.org/pmwkaa/ioarena

• { os: osx, xcode(8.3-9.1) } LLVM/Clang, deps library via brew
• { dist: trusty, os: linux }
  • clang-4.0 & clang-5.0?
  • GCC 5 & GCC 6

sqlite3: `set` test with `lazy` durability produces an error

./src/ioarena -m lazy -D sqlite3 -v 2048 -B set -n 10000
IOARENA (embedded storage benchmarking)

configuration:
  engine       = sqlite3
  datadir      = ./_ioarena
  benchmark    = set
  durability   = lazy
  wal          = indef
  operations   = 10000
  key size     = 16
  value size   = 2048
  binary       = no
  batch length = 500
  continuous   = no

key-gen: using 16 bits, up to 10000 keys
doer.0: {set}, key-space 0, key-sequence 0
execution failed, error: UNIQUE constraint failed: benchmark_t.key
error: ia_sqlite3_next, set, (null)

Common api for integration into FastoNoSQL

Hi i found your solution will be interesting for end user, i think it will be great to add iorena as benchmark system into FastoNoSQL. What you think? Do you have some public api?

Unknown database driver

When I run ioarena command I get the error "unknown database driver 'sophia'" for all types of databases such as rocksdb, sophia, etc. What am I missing duing build?
In addition, when I try cmake .. -DENABLE_XXX=ON, this fails for some XXX database.
Could you please explain the steps of building in detail? What we do fisrt, second, etc.

Makefiles are Linux-specific, fail on other platforms

The cmake scripts in cmake/Build*.cmake are hardcoded to assume that the libraries end up with a filename suffix of .so. That's true for Linux, but not Mac OS (or iOS) where the default suffix is .dylib. (I don't develop for BSD or Windows, but those might use different suffixes too.)

For example, cmake/BuildLevelDB.cmake has 3 lines like:

OUTPUT ${PROJECT_BINARY_DIR}/db/leveldb/libleveldb.so

The result is link errors building the ioarena binary, since the *.so files can't be found.

I know nothing about Cmake, so I don't have a fix to submit. I was able to work around the problem by replacing .so with .dylib in my local copies of the Cmake files.

statistic-merge and histogram modules should be revised and redesigned

This is most moronic and unclear part of code (aka "говнокод"), which had created extremely fast for http://www.highload.ru/2015/abstracts/1831.html

Due multithreading we need a merge statistic of operations from the threads for a every period. This is not an obvious task, since the threads could perform a different operations, the length of which can vary greatly. Therefor we need a timeframe for such mergers, and check that statistic from each threads merged single time per frame, and this frame was the same for all threads. Also the implementation should be wait-free and extremely cheap for CPU, otherwise benchmark result would be distorted.

Current implementations seems to be works, but it is unclear, difficult to support and development.

Link errors building db_bench on OS X

Checked out current master (853a50d) and configured it to build all of the databases except leveldb. I don't have any of them installed in /usr, so it's building the sources in the submodules.

The build fails with a link error building db_bench as part of RocksDB. See below. Looks like it's not linking with libz, as well as whatever library bzip comes from.

(MacBook Pro running OS X 10.11.2. Dev tools from Xcode 7.1.1.)

$  cmake .. -DENABLE_ROCKSDB=ON -DENABLE_LMDB=ON -DENABLE_FORESTDB=ON -DENABLE_NESSDB=ON -DENABLE_WIREDTIGER=ON -DENABLE_SOPHIA=ON
...
...
$ make
...
...
  CCLD     db_bench
Undefined symbols for architecture x86_64:
  "_BZ2_bzCompress", referenced from:
      rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*) in block_based_table_builder.o
  "_BZ2_bzCompressEnd", referenced from:
      rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*) in block_based_table_builder.o
  "_BZ2_bzCompressInit", referenced from:
      rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*) in block_based_table_builder.o
  "_BZ2_bzDecompress", referenced from:
      rocksdb::BZip2_Uncompress(char const*, unsigned long, int*, unsigned int) in format.o
  "_BZ2_bzDecompressEnd", referenced from:
      rocksdb::BZip2_Uncompress(char const*, unsigned long, int*, unsigned int) in format.o
  "_BZ2_bzDecompressInit", referenced from:
      rocksdb::BZip2_Uncompress(char const*, unsigned long, int*, unsigned int) in format.o
  "_deflate", referenced from:
      rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*) in block_based_table_builder.o
  "_deflateEnd", referenced from:
      rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*) in block_based_table_builder.o
  "_deflateInit2_", referenced from:
      rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*) in block_based_table_builder.o
  "_inflate", referenced from:
      rocksdb::Zlib_Uncompress(char const*, unsigned long, int*, unsigned int, int) in format.o
  "_inflateEnd", referenced from:
      rocksdb::Zlib_Uncompress(char const*, unsigned long, int*, unsigned int, int) in format.o
  "_inflateInit2_", referenced from:
      rocksdb::Zlib_Uncompress(char const*, unsigned long, int*, unsigned int, int) in format.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [db_bench] Error 1
make[2]: *** [db/rocksdb/librocksdb.so] Error 2
make[1]: *** [CMakeFiles/librocksdb.dir/all] Error 2
make: *** [all] Error 2

Memory consumption is very high for RocksDB

Hi,
I've conduct a test by ioarena to benchmark rocksdb v4.2.0 & the newer version v4.8.0.
It's wired that tests on v4.8.0 will consume much more mem than v4.2. Do you guys have any ideas?

• The default version of rocksdb used by ioarena seems v4.2.0.
• Both tests run in default options.
• Use top to check the usage.
• Memory consumption: v4.8.0(~3G) , v4.2.0(~60M)

rocksdb 4.8.0:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
129456 root 20 0 3505.3m 3.354g 6.8m R 100.0 21.8 4:58.36 src/ioarena -D rocksdb -B set -n 100000000

rocksdb 4.2.0:
120386 root 20 0 182004 14632 5952 S 31.9 0.1 0:25.87 src/ioarena -D rocksdb -B set -n 100000000 (about 14m)

Only dummy that works, It looks like there is some tweeking needs to be done

vagrant@vagrant-ubuntu-trusty:~/projects/ioarena/build$ for i in sophia leveldb rocksdb wiredtiger forestdb lmdb mdbx dummy ; do echo "Storage engine:$i"; src/ioarena -D $i -B get -n 5000 ;done
Storage engine:sophia
IOARENA (embedded storage benchmarking)

error: unknown database driver 'sophia'
Storage engine:leveldb
IOARENA (embedded storage benchmarking)

error: unknown database driver 'leveldb'
Storage engine:rocksdb
IOARENA (embedded storage benchmarking)

error: unknown database driver 'rocksdb'
Storage engine:wiredtiger
IOARENA (embedded storage benchmarking)

error: unknown database driver 'wiredtiger'
Storage engine:forestdb
IOARENA (embedded storage benchmarking)

error: unknown database driver 'forestdb'
Storage engine:lmdb
IOARENA (embedded storage benchmarking)

error: unknown database driver 'lmdb'
Storage engine:mdbx
IOARENA (embedded storage benchmarking)

error: unknown database driver 'mdbx'
Storage engine:dummy
IOARENA (embedded storage benchmarking)

configuration:
engine = dummy
datadir = ./_ioarena
benchmark = get
durability = lazy
wal = indef
operations = 5000
key size = 16
value size = 32
binary = no
batch length = 500
continuous = no

formalize codestyle info .clang-format

nessdb driver should be updated (or removed)

nessdb driver now is just a stub with obsolete interface, so it should be updated or removed.

distribution test tasks to threads is unclear and should be refined

Current implementation is justified, but you cannot say that it is obvious and good. Until I don't come up with anything better. Therefore (as a minimum) it should be documented and explained why so.

Propose to remove ejdb driver

Hi guys! I propose to remove ejdb 1.x driver because at not maintained and replaced by ejdb2. On other side ejdb2 is based on iowow engine already listed here. So I don't see any reason to keep it.
What do you think? @pmwkaa

rework of random pool should be completed

The "random pool" used for fast creation of key-value pairs while benchmarking. In other words it is pre-generated key-value pairs, which have some predefined characteristics.

It must be:

• pseudorandom for reproducibility of benchmarks;
• not contain duplicate keys (taking into account the specified length) for workability of testcases;
• cheap by cpu and memory (include bandwidth) for correctness of results;
• distributed uniformly;
• organized into non-overlapping clusters for parallel multithreading testcases;

Current implementation:

• does not guarantee uniqueness, especially for short keys;
• uses a lot of memory, e.g. N * (key_size + value_size);

I have some idea how to do this. But it will require abandoning terminating zero at end of the keys and values.

have error when make unqlite 'EVP_CIPHER_CTX'

have error when make unqlite 'EVP_CIPHER_CTX'

does anyone have the benchmark chart or figures or stats on

1. speed
2. storage sizes

of all the dbs?

Crash when parsing malformed forestdb file

couchbase/forestdb#23

driver for BerkeleyDB

Seems be very useful have a such "legacy baseline" for the comparison.

error: Corruption: not an sstable (bad magic number)

i consistently get

error: Corruption: not an sstable (bad magic number)

with leveldb if -n is high enough.

# ./ioarena -n 4000000 -T set -D leveldb          

IOARENA (embedded storage benchmarking)

configuration:
  database:   leveldb
  output:     ./_ioarena
  benchmark:  set
  operations: 4000000
  key size:   16
  value size: 32

<<>> leveldb.set
0.1M   8.77 mb/sec
0.2M   9.85 mb/sec
0.3M   9.57 mb/sec
0.4M   9.22 mb/sec
0.5M   9.64 mb/sec
0.6M   9.26 mb/sec
0.7M   9.73 mb/sec
0.8M   9.30 mb/sec
0.9M   9.01 mb/sec
1.0M   9.17 mb/sec
error: Corruption: not an sstable (bad magic number)

☁  src [master] ⚡ ldd ioarena 
        linux-vdso.so.1 (0x00007fffe2d6b000)
        libleveldb.so.1 => /usr/lib/libleveldb.so.1 (0x00007f8725e8f000)
        librocksdb.so.3.13 => /usr/lib/librocksdb.so.3.13 (0x00007f8725989000)
        libsophia.so => /usr/lib/libsophia.so (0x00007f8725737000)
        libwiredtiger-2.7.0.so => /usr/lib/libwiredtiger-2.7.0.so (0x00007f8725414000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f8725116000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f8724ef9000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f8724b55000)
        libsnappy.so.1 => /usr/lib/libsnappy.so.1 (0x00007f872494d000)
        libtcmalloc.so.4 => /usr/lib/libtcmalloc.so.4 (0x00007f87246d8000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f8724356000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f8724140000)
        librt.so.1 => /usr/lib/librt.so.1 (0x00007f8723f38000)
        libz.so.1 => /usr/lib/libz.so.1 (0x00007f8723d22000)
        libbz2.so.1.0 => /usr/lib/libbz2.so.1.0 (0x00007f8723b12000)
        liblz4.so.1 => /usr/lib/liblz4.so.1 (0x00007f8723900000)
        libjemalloc.so.2 => /usr/lib/libjemalloc.so.2 (0x00007f87236c2000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f87260ed000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f87234be000)
☁  src [master] ⚡ pacman -Q leveldb
leveldb 1.18-2
☁  src [master] ⚡ 

Rocksdb build fails

I've managed to build Rocksdb locally in a different directory, but ioarena build script fails with

db/compaction_job.o: In function `rocksdb::CompactionJob::Run()':
/usr/include/c++/4.9/thread:136: undefined reference to `pthread_create'

Which is strange.

clock_gettime() not implemented in OS X

Rhetorical question - does OSX conform to SUSv2 and/or POSIX.1-2001?