yxwei-umn/dariadb

dariadb - is a numeric time-series database storage engine.

dariadb - numeric time-series database.

Continuous Integration

version	build & tests	test coverage
master
develop

Features

• True columnar storage
• Can be used as a server application or an embedded library.
• Full featured http api.
• Golang client (see folders "go" and "examples/go")
• Accept unordered data.
• Each measurement contains:
  • Id - x32 unsigned integer value.
  • Time - x64 timestamp.
  • Value - x64 float.
  • Flag - x32 unsigned integer.
• Write strategies:
  • wal - little cache and all values storing to disk in write ahead log. optimised for big write load(but slower than 'memory' strategy).
  • compressed - all values compressed for good disk usage without writing to sorted layer.
  • memory - all values stored in memory and dropped to disk when memory limit is ended.
  • cache - all values stored in memory with writes to disk.
  • memory-only - all valeus stored only in memory.
• LSM-like storage struct with three layers:
  • Memory cache or Append-only files layer, for fast write speed and crash-safety(if strategy is 'wal').
  • Old values stored in compressed block for better disk space usage.
• High write speed:
  • as embedded engine - to disk - 1.5 - 3.5 millions values per second to disk
  • as memory storage(when strategy is 'memory') - 7-9 millions.
  • across the network - 700k - 800k values per second
• Shard-engine: you can split values per shard in disk, for better compaction and read speed up.
• Crash recovery.
• CRC32 for all values.
• Two variants of API:
  • Functor API (async) - engine apply given function to each measurement in the incoming request.
  • Standard API - You can Query interval as list or values in time point as dictionary.
• Compaction old data with filtration support;
• Statistic:
  • time min/max
  • value min/max
  • measurement count
  • values sum
• Statistical functions:
  • minimum
  • maximum
  • count
  • average
  • median
  • sigma(standard deviation)
  • percentile90
  • percentile99
• Interval aggregation support. Available intervals: raw,minute, half hour, hour, day, week, month, year.

Usage example

• See folder "examples"
• How to use dariadb as a embedded storage engine: dariadb-example

Dependencies

• Boost 1.54.0 or higher: system, filesystem, date_time,regex, program_options, asio.
• cmake 3.1 or higher
• c++ 14/17 compiler (MSVC 2015, gcc 6.0, clang 3.8)

Build

---

Install dependencies

$ sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
$ sudo apt-get update
$ sudo apt-get install -y libboost-dev  libboost-filesystem-dev libboost-program-options-dev libasio-dev libboost-date-time-dev cmake  g++-6  gcc-6 cpp-6 
$ export CC="gcc-6"
$ export CXX="g++-6"

Jemalloc

Optionaly you can install jemalloc for better memory usage.

$ sudo apt-get install libjemalloc-dev

Or you may use builtin jemalloc source in dariadb - just add build option -DSYSTEM_JEMALLOC=OFF

Git submodules

$ cd dariadb
$ git submodules init 
$ git submodules update

Available build options

• DARIADB_ENABLE_TESTS - Enable testing of the dariadb. - ON
• DARIADB_ENABLE_INTEGRATION_TESTS - Enable integration test. - ON
• DARIADB_ENABLE_SERVER - Enable build dariadb server. - ON
• DARIADB_ENABLE_BENCHMARKS - Enable build dariadb benchmarks. - ON
• DARIADB_ENABLE_SAMPLES - Build dariadb sample programs. - ON
• DARIADB_ASAN_UBSAN - Enable address & undefined behavior sanitizer for binary. - OFF
• DARIADB_MSAN - Enable memory sanitizer for binary. - OFF
• DARIADB_SYSTEM_JEMALLOC - Use jemalloc installed in the system. - ON

Configure to build with all benchmarks, but without tests and server.

---

$ cmake  -DCMAKE_BUILD_TYPE=Release -DDARIADB_ENABLE_TESTS=OFF -DDARIADB_ENABLE_INTEGRATION_TESTS=OFF -DDARIADB_ENABLE_BENCHMARKS=ON -DDARIADB_ENABLE_SERVER=OFF . 

clang

---

Clang currently does not supported.

$ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS_RELEASE="${CMAKE_CXX_FLAGS_RELEASE} -stdlib=libc++" -DCMAKE_EXE_LINKER_FLAGS="${CMAKE_EXE_LINKER_FLAGS} -lstdc++" .
$ make

gcc

---

$ cmake -DCMAKE_BUILD_TYPE=Release .
$ make

Microsoft Visual Studio

---

$ cmake -G "Visual Studio 14 2015 Win64" .
$ cmake --build .

if you want to build benchmarks and tests

$ cmake -G "Visual Studio 14 2015 Win64" -DBUILD_SHARED_LIBS=FALSE  .
$ cmake --build .

build with non system installed boost

---

$ cmake  -DCMAKE_BUILD_TYPE=Release -DBOOST_ROOT="path/to/boost/" .
$ make