lysevi/dariadb

dariadb - is a numeric time-series database storage engine.

dariadb - numeric time-series database.

Continuous Integration

version	build & tests	test coverage
master
develop

Features

• True columnar storage
• Can be used as a server application or an embedded library.
• Full featured http api.
• Golang client (see folders "go" and "examples/go")
• Accept unordered data.
• Each measurement contains:
  • Id - x32 unsigned integer value.
  • Time - x64 timestamp.
  • Value - x64 float.
  • Flag - x32 unsigned integer.
• Write strategies:
  • wal - little cache and all values storing to disk in write ahead log. optimised for big write load(but slower than 'memory' strategy).
  • compressed - all values compressed for good disk usage without writing to sorted layer.
  • memory - all values stored in memory and dropped to disk when memory limit is ended.
  • cache - all values stored in memory with writes to disk.
  • memory-only - all valeus stored only in memory.
• LSM-like storage struct with three layers:
  • Memory cache or Append-only files layer, for fast write speed and crash-safety(if strategy is 'wal').
  • Old values stored in compressed block for better disk space usage.
• High write speed:
  • as embedded engine - to disk - 1.5 - 3.5 millions values per second to disk
  • as memory storage(when strategy is 'memory') - 7-9 millions.
  • across the network - 700k - 800k values per second
• Shard-engine: you can split values per shard in disk, for better compaction and read speed up.
• Crash recovery.
• CRC32 for all values.
• Two variants of API:
  • Functor API (async) - engine apply given function to each measurement in the incoming request.
  • Standard API - You can Query interval as list or values in time point as dictionary.
• Compaction old data with filtration support;
• Statistic:
  • time min/max
  • value min/max
  • measurement count
  • values sum
• Statistical functions:
  • minimum
  • maximum
  • count
  • average
  • median
  • sigma(standard deviation)
  • percentile90
  • percentile99
• Interval aggregation support. Available intervals: raw,minute, half hour, hour, day, week, month, year.

Usage example

• See folder "examples"
• How to use dariadb as a embedded storage engine: dariadb-example

Dependencies

• Boost 1.54.0 or higher: system, filesystem, date_time,regex, program_options, asio.
• cmake 3.1 or higher
• c++ 14/17 compiler (MSVC 2015, gcc 6.0, clang 3.8)

Build

---

Install dependencies

$ sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
$ sudo apt-get update
$ sudo apt-get install -y libboost-dev  libboost-filesystem-dev libboost-program-options-dev libasio-dev libboost-date-time-dev cmake  g++-6  gcc-6 cpp-6 
$ export CC="gcc-6"
$ export CXX="g++-6"

Jemalloc

Optionaly you can install jemalloc for better memory usage.

$ sudo apt-get install libjemalloc-dev

Or you may use builtin jemalloc source in dariadb - just add build option -DSYSTEM_JEMALLOC=OFF

Git submodules

$ cd dariadb
$ git submodules init 
$ git submodules update

Available build options

• DARIADB_ENABLE_TESTS - Enable testing of the dariadb. - ON
• DARIADB_ENABLE_INTEGRATION_TESTS - Enable integration test. - ON
• DARIADB_ENABLE_SERVER - Enable build dariadb server. - ON
• DARIADB_ENABLE_BENCHMARKS - Enable build dariadb benchmarks. - ON
• DARIADB_ENABLE_SAMPLES - Build dariadb sample programs. - ON
• DARIADB_ASAN_UBSAN - Enable address & undefined behavior sanitizer for binary. - OFF
• DARIADB_MSAN - Enable memory sanitizer for binary. - OFF
• DARIADB_SYSTEM_JEMALLOC - Use jemalloc installed in the system. - ON

Configure to build with all benchmarks, but without tests and server.

---

$ cmake  -DCMAKE_BUILD_TYPE=Release -DDARIADB_ENABLE_TESTS=OFF -DDARIADB_ENABLE_INTEGRATION_TESTS=OFF -DDARIADB_ENABLE_BENCHMARKS=ON -DDARIADB_ENABLE_SERVER=OFF . 

clang

---

Clang currently does not supported.

$ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS_RELEASE="${CMAKE_CXX_FLAGS_RELEASE} -stdlib=libc++" -DCMAKE_EXE_LINKER_FLAGS="${CMAKE_EXE_LINKER_FLAGS} -lstdc++" .
$ make

gcc

---

$ cmake -DCMAKE_BUILD_TYPE=Release .
$ make

Microsoft Visual Studio

---

$ cmake -G "Visual Studio 14 2015 Win64" .
$ cmake --build .

if you want to build benchmarks and tests

$ cmake -G "Visual Studio 14 2015 Win64" -DBUILD_SHARED_LIBS=FALSE  .
$ cmake --build .

build with non system installed boost

---

$ cmake  -DCMAKE_BUILD_TYPE=Release -DBOOST_ROOT="path/to/boost/" .
$ make

dariadb's Issues

Bucket

in memory storage can be write only sorted by time data.

1. in bucket measurements stored in sorted sets(by time).
2. when set in full, all meas in them writes to memstorage.
3. membucket check time of writed measurement. time should be great or equal some time. (write window).

• TimeOrderedSet
• Bucket
  • list of TimeOrderedSet
  • add meas with max time
  • insert in midle of set
  • insert in closest set
  • add meas with time less than min (equal max time of older set)
  • check time window
  • max count of appended meas
  • dump older set to memstorage.
  • split TimeOrderedSets by memseries id. (use inner meas representation)

refact

• rename storeage_test storage_bench. add "mem" prefix
• page_manager => storage dir

write to past

• - capacitor support unordered data.
• - Chunk::Info::is_sorted flag. if date ordered by time, flag is true.
• - chunk support unordered data. new class OverlappedChunks: set of chunks with overlapped times. return result of k-merge chunks data from interval.
• - engine support unordered data.
• - test with writing random data (engine and capacitor).

speed up drop chunks to disk

• async

continue writing

• Pagemanager load not closed chunks
• Memstorage: method, to add chunks (and FreeChunks filling)

Capacitor flush by timer

Capcitor should sync. with storage by timer (period eq. write window size+const)
add checks to common test after fillinf:

... sleep ...
dariadb::Meas::MeasList current_mlist;
as->currentValue(dariadb::IdArray{}, 0)->readAll(&current_mlist);

CurrentValueCache

subj

extended filter

• positive only
• negative only
• all by default
• by measurement source
  • - bloom filter

Page manager

• - create empty page with partition of index (allocate in memory. to test. )
  • - calc size of index
  • - readers count
  • - writers lock
• - append new chunk:
  • - add chunk buffer to chunks partition
  • - add index info.
• - read chunk to memory storage with read only mode (or copy on read).
• - rewrite old chunk on append
• - read cursor.
• - send all/old chunks from memstorage to page storage.
• - save in chunk BinaryBuffer params (position etc)
• - restore meas id.
• - page is a mapped file.
  • - create new
  • - reopen
• - read page info (without loading all page)
  • - free space
• - benchmark
  • - write single thread
  • - read single thread

dtw

Dynamic time wraping algorithm

optional unittest build

subj. unittest depends from boost.

Statistic

• average
• integral
• median
• benchmarks

time series analise

All analysis functions should be available for stored procedures.

capacitor struct

bucket sort.

storage procedure language

• - All statistic functions should be available for stored procedures.
• - scripts(lua):
  • - build options SYSTEM_LUA (ON by default). if off, use git subproject lua.
• - API:
  • - create database
  • - readInterval
  • - readInTimePoint
  • - join:
    • - table pretty print
  • - add continues query rule with map callback (for example: add to QueryTable sum of id=1 and id=2 for each time.)
  • - load raw to bystep with step.

median

subj

read optimisation

flag isOrderedStorage. stop reading in Reader when readed Time > to

subscribe

calback to new data. filter by id.

functional test

• capacitor + storage

bloom filter

• - memstorage

use storage::Cursor in chunk operations

• use storage::Cursor in chunk operations

memory allocation

Block must allocate one buffer.

query filtration by measurement source.

time series analise helpers

• flags
• steps
  • byChange - is default
  • byStep
• reader must be a callback
• clone readers
• reader reset pos. to read from begin.

steps

byChange - by defaul byTime - with step value in miliseconds

optimise page read

only in memory search algs.

replace size_t to uint32_t or uin64_t

replace size_t to uint32_t or uin64_t

Write-Ahead Log

subj

storage: reserve space to ids

subj.

page is chunk container

subj

one buf to all.

subj

speed up drop chunks from memory storage

speed up drop chunks from memory storage

flush data on write to page.

subj.

limit size of MemoryStorage.

limit size of MemoryStorage.

current values

delta encoding

store full time when delta>2048 (currently stored uint32 part)

reader must be a callback

subj

chunks on disk alg change

current not correct.

chunks pool

ChunksPoll class. create new if no free. Chunks not deleting, just get back to the pool. Chunks pool store buffers of uin8_t.

thread safety

• memory storage
• capacitor

Meas::Source

new field. works like Flag.

use stx-btree

in all memory caches.

https://github.com/lysevi/stx-btree

check Chunk.last filling

query filtration by measurement source.

• - bloom filter to measurement source
• - filtration in ReadInterval
• - filtration in RadInTimePoint

speed up test

storage_test to slow.

UnionStorage: read query to pagemanager

• Chunk readOnly param.
• methods readInterval, readInTimePoint implement in AbstractStorage.
• AbstractStorage is derived from ChunkStorage.
• AbstractStorage rename to BaseStorage.
• implement chunks operation in PageManager.
• implement chunks operation in UnionStorage.
• MemStorage dropAll

BinaryBuffer::write/read(ui64) limit

max limit 32 bit. not good.

storage test

fill file storage. read interval from minTime to maxTime.

base compressor

refact

bind many queryes to one reader

results of many queryes can be used in one operation (calculation).