bohutang/nessdb

A very fast transactional key-value, embedded database storage engine in Fractal-Tree. Teaching/Research purposes only.

nessDB v3.0 with Buffered-Tree.
(C) 2012-2020 nessDB Authors ________________
_____________________________  __ \__  __ )
__  __ \  _ \_  ___/_  ___/_  / / /_  __  |
_  / / /  __/(__  )_(__  )_  /_/ /_  /_/ /
/_/ /_/\___//____/ /____/ /_____/ /_____/
================================================================

nessDB is a fast Key-Value database(embedded), which is written in ANSI C with BSD LICENSE and works in most POSIX systems without external dependencies.

FEATURES

• Buffered-Tree index data structure
• Range-Query
• Transaction

How To Bench

Random write 10,000,000(10 Million) records with 512MB memory.

make db-bench
./db-bench --benchmarks=fillrandom --num=10000000 --cache_size=536870912
nessDB:     version 3.0.0
Date:       Sat Jun 20 19:06:42 2020
CPU:        4 *  Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
CPUCache:   3072 KB
Keys:       20 bytes each
Values:     100 bytes each
Entries:    10000000
RawSize:    1144.4 MB (estimated)
Compression:Snappy
WARNING: assertions are enabled; benchmarks unnecessarily slow
------------------------------------------------------------
random write finished 10000000 ops                              
[  t min,   t max]	  ops count	      %
--------------------------------------------------
[      0,       1]	    9998639	  99.99 
[      1,       2]	         92	   0.00 
[      2,       3]	         78	   0.00 
[      3,       4]	        100	   0.00 
[      4,       5]	         73	   0.00 
[      5,       6]	         60	   0.00 
[      6,       7]	         65	   0.00 
[      7,       8]	         79	   0.00 
[      8,       9]	         58	   0.00 
[      9,      10]	         58	   0.00 
[     10,      12]	         70	   0.00 
[     12,      14]	         72	   0.00 
[     14,      16]	         47	   0.00 
[     16,      18]	         35	   0.00 
[     18,      20]	         41	   0.00 
[     20,      25]	         40	   0.00 
[     25,      30]	         29	   0.00 
[     30,      35]	         24	   0.00 
[     35,      40]	         19	   0.00 
[     40,      45]	         31	   0.00 
[     45,      50]	         15	   0.00 
[     50,      60]	         42	   0.00 
[     60,      70]	         43	   0.00 
[     70,      80]	         47	   0.00 
[     80,      90]	         35	   0.00 
[     90,     100]	         39	   0.00 
[    100,     120]	         48	   0.00 
[    120,     140]	         19	   0.00 
[    140,     160]	          1	   0.00 
[    160,     180]	          1	   0.00 
--------------------------------------------------
total:       53514	   10000000	   100%
min latency       : 0.000000 * 1e-3 sec/op
avg latency       : 0.005351 * 1e-3 sec/op
max latency       : 171.927214 * 1e-3 sec/op
avg throughput    :  186866 ops/sec

V3.0 is still cooking. For more work-in-process, please see TODO.

nessdb's Issues

Java Bindings/Redis Interface

Are the Java bindings and Redis interface deprecated in the new version of nessDB? Looking at old issues and shuttler/nessDB it looks like there used to be Java bindings available.

about MVCC feature in your TODO

Hi,

I have heard that implement MVCC is hard and make your code bloated.
But I have read an implementation of MVCC that very simple and lightweight. Multi reader, One writer, reader not block writer.

You can read here: http://symas.com/mdb/
Read the PDF presentation, you can found there.

db-server CPU stall "forever"

This command :
./redis-benchmark -t set -d 1024 -r 10 -l
cause the CPU to stall "forever" after 1 or 2 minutes.

The db-server process stall to 100% with no IO, no swap. The most surprising is that the CPU stall even after the redis-benchmark command is stoped.

In this "state", db-server respond to nothing but do not crash and print "0 clients connected" every 3 seconds.

Bests regards, Olivier Goudron

install target in Makefile for libnessdb.so ?

I don't know if it's better to have an install target for libnessdb.so in Makefile or if the rule should be to manually copy libnessdb.so in the same directory that application requiring it.

What do you think ?

db-server sometime crash while concurent get and set of the same key

I start db-server with a fresh empty db.
I run this command in one shell : ./redis-cli -r 10000000 get test

Then i run this commands in another shell : ./redis-cli set test tata
followed by this command : ./redis-cli set test toto
And i repeat this patern until db-server crash.

The crash occur rapidly (between 4 to 10 set command).

Can you reproduce this and do you need more informations ?

Thanks in advance, Olivier Goudron

db-server crash under heavy load with redis-benchmark

I use the last version of redis-benchmark.

This is the command used to crash db-server :
./redis-benchmark -t set,get -n 1000000 -d 1024 -r 1000000 -l

After 2 or 3 loop, db-server crash.

Bests regards, Olivier Goudron

Add key traversal

Since the B+ tree's sorted nature of it is must have feature to traverse over the key values. This will be extremely useful and can be abstracted into a iterator structure.

benchmark feedback

Hi, I repost here what I post on googlecode

• I'm doing a bench and it become very slow after a while

Here the diff with my tunning:
-#define KEYSIZE 20
-#define VALSIZE 100
-#define NUM 1000000
+#define KEYSIZE 16
+#define VALSIZE 24
+#define NUM 100000000
#define R_NUM 10000
#define REMOVE_NUM 10000
-#define BUFFERPOOL (1024_1024_1024)
-#define BGSYNC (1)
+#define BUFFERPOOL (2_1024_1024*1024)
+#define BGSYNC (0)

@@ -88,10 +88,16 @@ void random_value()
}

void random_key(char key,int length) {
+/
char salt[36]= "abcdefghijklmnopqrstuvwxyz0123456789";
memset(key,0,length);
for (int i = 0; i < length; i++)
key[i] = salt[rand() % length];
+*/

•   _(int_)key = rand();
  

•   _(int_)(key+4) = rand();
  

•   _(int_)(key+8) = rand();
  

•   _(int_)(key+12) = rand();
  

What version of the product are you using? On what operating system?

• version 1.7
• Linux hyperstream 2.6.38-11-generic #50-Ubuntu SMP Mon Sep 12 21:17:25 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
• my config: core i7 2600, 16 GB of ram, SSD ocz vertex 3

Please provide any additional information below.

• it's not possible to have a buffer pool > to 2 GB since it's an int and not an int64 (I have 16 GB of ram)
• DB size is very huge with big overhead: I have a key of 16 bytes and a records of 24 bytes. So it's 40 bytes per records. But after inserting 100 millions records, the DB size (.db+.idx) is about 14 GB, so it's an average of 151 bytes per records !!! With Kyoto using hash DB it's rather ~ 55 bytes
• DB inserting is quite fast, but speed decrease quickly. I suppose that the bufferpool is too small.
• performance: it's an average of 29.000 insert/s (but it start at 140.000 and finish at a very slow rate) after 100 millions insert. Kyoto is at 90.000 insert/s after 250 millions insert !! But I set the cache to 12 GB of ram
• iostat show a very important write I/O. Often between 50 to 120 MB/s during the whole benchmark. It seems there is a lot of update (index balancing ??)

Some others information:

|Random-Write (done:99999999): 0.000035 sec/op; 28911.6 writes/sec(estimated); 2.6 MB/sec; cost:3458.821(sec)
lch@hyperstream:/DT/local/nessDB/src$ du -k .
14781888 ./ndbs
=> 151 bytes per records instead of 24 ...

-rw-r--r-- 1 lch users 245325984 2011-10-26 15:58 ness0.db
-rw-r--r-- 1 lch users 918359689 2011-10-26 15:58 ness0.idx
-rw-r--r-- 1 lch users 245150720 2011-10-26 15:58 ness10.db
-rw-r--r-- 1 lch users 918612711 2011-10-26 15:58 ness10.idx
-rw-r--r-- 1 lch users 245293760 2011-10-26 15:58 ness11.db
-rw-r--r-- 1 lch users 918339284 2011-10-26 15:58 ness11.idx
-rw-r--r-- 1 lch users 257679456 2011-10-26 15:58 ness12.db
-rw-r--r-- 1 lch users 917527165 2011-10-26 15:58 ness12.idx
[...]

Hope you can continue to enhance this promising DB :)

[txn]: transaction isolation under new msgbuf

Now, msgbuf is more flexible for txn isolation&gc.
We should re-format cursor.c.

Write very very slow when the data count more than 100millions

I tested write 200 million data last night.

This moring it only insert 140million data, after the data count is large(seems more than 100million) the speed of writing is very slow.

problem with GETing DELted key

This is the sequence used with redis-cli for this problem :

redis 127.0.0.1:6379> get test
(nil)
redis 127.0.0.1:6379> set test aaaa
OK
redis 127.0.0.1:6379> get test
"aaaa"
redis 127.0.0.1:6379> del test
OK
redis 127.0.0.1:6379> get test
"x\x1f\xf5\t\xe8\x1f\xf5\t"

When a GET again the deleted key i have a strange answer.

Bests regards, Olivier Goudron

DB corrupted after a crash

Hi,

I lightly modify the db-bench.c file to reflect my need, then do a test. It crash around 6 millions of inserted records. I redo the test, it crash again. I redo, but now the DB was broken.

I have enabled the log, so I'm really surprised that DB was dead.

Regards,

Laurent

note: I put the diff, then after I paste the console logs.

-#define TOLOG (0)
+#define TOLOG (1)
#define KSIZE 16
-#define VSIZE 80
-#define BUFFERPOOL (1024_1024_1024)
+#define VSIZE 24
+#define BUFFERPOOL ((size_t)3_1024_1024*1024)
@@ -113,13 +113,17 @@ void _write_test(long int count)

            sk.len = KSIZE;
            sk.data = key;

•           db_get(db, &sk, &sv);
  

  •       sv.len = VSIZE;
          sv.data = val;
    

          db_add(db, &sk, &sv);
  

•           if ((i % 10000) == 0) {
  

•                   fprintf(stderr,"random write finished %d ops%30s\r", i, "");
  

•                   fflush(stderr);
  

•           if ((i % 1000000) == 0) {
  

•                   time_t t2 = time(NULL);
  

•                   fprintf(stderr,"%d;%d \n", i, (int)(t2 - start));
          }
  

lch@hyperstream:/DT/local/nessDB$ ./db-bench write 10000000
Keys: 16 bytes each
Values: 24 bytes each
Entries: 10000000
IndexSize: 238.4 MB (estimated)

DataSize: 267.0 MB (estimated)

nessDB: version 1.8(LSM-Tree storage engine)
Date: Sat Apr 7 21:44:39 2012
CPU: 8 * Intel(R) Xeon(R) CPU E31245 @ 3.30GHz
CPUCache: 8192 KB

[18835] 07 Apr 09:44:39 . Load sst,all entries count:<0>
0;0
1000000;4
2000000;9
3000000;15
4000000;20
5000000;26
6000000;32
Segmentation fault
lch@hyperstream:/DT/local/nessDB$ ./db-bench write 10000000
Keys: 16 bytes each
Values: 24 bytes each
Entries: 10000000
IndexSize: 238.4 MB (estimated)

DataSize: 267.0 MB (estimated)

nessDB: version 1.8(LSM-Tree storage engine)
Date: Sat Apr 7 21:46:17 2012
CPU: 8 * Intel(R) Xeon(R) CPU E31245 @ 3.30GHz
CPUCache: 8192 KB

[18997] 07 Apr 09:46:20 . Load sst,all entries count:<6000000>
[18997] 07 Apr 09:46:20 . prepare to recovery from new log#/DT/local/nessDB/ndbs/1.log
[18997] 07 Apr 09:46:20 . recovery count ADD#173367, DEL#0
[18997] 07 Apr 09:46:20 . prepare to merge logs, merge count #173367....
[18997] 07 Apr 09:46:20 . adding log items to bloomfilter
0;0
1000000;4
2000000;9
3000000;15
4000000;20
5000000;26
6000000;33
Bus error
Keys: 16 bytes each
Values: 24 bytes each
Entries: 10000000
IndexSize: 238.4 MB (estimated)

DataSize: 267.0 MB (estimated)

nessDB: version 1.8(LSM-Tree storage engine)
Date: Sat Apr 7 21:48:17 2012
CPU: 8 * Intel(R) Xeon(R) CPU E31245 @ 3.30GHz
CPUCache: 8192 KB

[19196] 07 Apr 09:48:18 # read footer error, error:Invalid argument engine/sst.c:98
lch@hyperstream:/DT/local/nessDB$ rm ndbs/*
lch@hyperstream:/DT/local/nessDB$ gdb db-bench
(gdb) run write 10000000
Keys: 16 bytes each
Values: 24 bytes each
Entries: 10000000
IndexSize: 238.4 MB (estimated)

DataSize: 267.0 MB (estimated)

nessDB: version 1.8(LSM-Tree storage engine)
Date: Sat Apr 7 21:51:14 2012
CPU: 8 * Intel(R) Xeon(R) CPU E31245 @ 3.30GHz
CPUCache: 8192 KB

[19565] 07 Apr 09:51:14 . Load sst,all entries count:<0>
0;0
1000000;5
2000000;10
3000000;15
4000000;21
5000000;27
[New Thread 0x7fffec1a9700 (LWP 19670)]
6000000;33
[Thread 0x7fffec1a9700 (LWP 19670) exited]

Program received signal SIGSEGV, Segmentation fault.
0x00000000004081e3 in skiplist_lookup (list=0x60e460, data=0x7fffffffe940 "jdilkloimdkobnkn")
at engine/skiplist.c:162
162 while (x->forward[i] != NIL
(gdb) l
157 {
158 int i;
159 struct skipnode *x = list->hdr;
160
161 for (i = list->level; i >= 0; i--) {
162 while (x->forward[i] != NIL
163 && cmp_lt(x->forward[i]->key, data))
164 x = x->forward[i];
165 }
166 x = x->forward[0];
(gdb) p i
$1 = 758396511
(gdb) p x->foward[0]
There is no member named foward.
(gdb) p x->forward[0]
Cannot access memory at address 0x28
(gdb) p x
$2 = (struct skipnode *) 0x0
(gdb) info local
i = 758396511
x = 0x0
(gdb) where
#0 0x00000000004081e3 in skiplist_lookup (list=0x60e460, data=0x7fffffffe940 "jdilkloimdkobnkn")

at engine/skiplist.c:162

#1 0x0000000000406e92 in index_get (idx=0x60da30, sk=0x7fffffffe8d0, sv=0x7fffffffe8c0) at engine/index.c:260
#2 0x000000000040298f in db_get (db=0x60c110, sk=0x7fffffffe8d0, sv=0x7fffffffe8c0) at engine/db.c:65
#3 0x0000000000401f0f in _write_test (count=10000000) at bench/db-bench.c:117
#4 0x00000000004023c8 in main (argc=3, argv=0x7fffffffea88) at bench/db-bench.c:218

(gdb) up
#1 0x0000000000406e92 in index_get (idx=0x60da30, sk=0x7fffffffe8d0, sv=0x7fffffffe8c0) at engine/index.c:260

260 node = skiplist_lookup(merge_list, sk->data);
(gdb)
#2 0x000000000040298f in db_get (db=0x60c110, sk=0x7fffffffe8d0, sv=0x7fffffffe8c0) at engine/db.c:65

65 ret = index_get(db->idx, sk, sv);
(gdb) p *sk
$4 = {data = 0x7fffffffe940 "jdilkloimdkobnkn", len = 16}
(gdb) p *sv
$5 = {data = 0x7fffffffe920 "val:6176209", len = 24}
(gdb)

[Leaf] message apply to leaf node

apply ancestors message bound to leaf node

[pma] support minus/zero/plus finding

llru problem

I have a problem about llru part of nessDB.
I have seen the source ,the ht include all elements but any elements cannot expired
Because the implement have no delete operation about the expired elments.

Maybe is my wrong...
Can you explain the theory about the implement of llru part of nessDB.
Thank you!

[Feature Request]:User defined comparator

leveldb/lmdb both support user defined comparator, it's useful in many scenario.

[recovery]: redo log recovery

to add

[fifo]: _flush_some_child bug

very slow, maybe an length issue

[msgbuf]: add basement for leaf

basement for leaf, supports:
a) MVCC
b) transaction visibility checking
c) gc

Jedis client and dirty value

Hello,

When using with Jedis Java client, NessDB or Jedis gives me sometimes dirty values with GET.

Here is an example, where I try to simply write "helloword" in value (if a key already have a "helloworld" value, I replace):

In my java code:
In a SET:
jedis.set(key,"helloworld".getBytes());
In a GET:
byte[] value = jedis.get(key);
String printValue = value!=null ? new String(value) : null;
System.out.println("GET:"+new String(key) + " => " + printValue);

-------------------------------------------------- JEDIS CLIENT SIDE -------------------------------------
With this implementation, I have:
GET:1#14811020 => helloworld
GET:1#14c1102a => null
GET:1#17810021 => null
GET:1#110100a8 => helloworld
[...]
GET:1#171701ba => null
GET:1#171500aa => helloworldq[]
GET:1#4851122 => helloworld
GET:1#d5112a => null
GET:1#5501bb => null
[...]
GET:1#169501ab => helloworldu[]
GET:1#6d511a1 => null
GET:1#16d501bb => null
GET:1#10020 => helloworldu[]
[...]([] is a special rectangle char)

-------------------------------------------------- NESSDB SERVER SIDE -------------------------------------
I follow the key "1#4d501a2" in nessdb server.
There is only 1 op that write the key/value and 9 op to get the value.
Here is the operation list:

1. GET (value is null)
2. SET (value is OK)
3. GET working
4. GET working
5. GET working
6. GET not working (=> helloworldC)
7. GET working
8. GET working
9. GET working
10. GET not working (=> helloworldD)

I add 'request_dump' from db-server.c and a printf in GET case when (ret == 1) to see the return value after GET.

1. GET (value is null)
   request-dump--->{argc:<2>
   cmd:
   argv[0]:
   argv[1]:<1#4d501a2>
   }
   //no return value
2. SET (value is OK)
   request-dump--->{argc:<3>
   cmd:
   argv[0]:
   argv[1]:<1#4d501a2>
   argv[2]:
   }
3. GET working
   request-dump--->{argc:<2>
   cmd:
   argv[0]:
   argv[1]:<1#4d501a2>
   }
   return:helloworld
4. GET working
   request-dump--->{argc:<2>
   cmd:
   argv[0]:
   argv[1]:<1#4d501a2>
   }
   return:helloworld
5. GET working
   request-dump--->{argc:<2>
   cmd:
   argv[0]:
   argv[1]:<1#4d501a2>
   }
   return:helloworld
6. GET not working
   request-dump--->{argc:<2>
   cmd:
   argv[0]:
   argv[1]:<1#4d501a2>
   }
   return:helloworldC�
7. GET working
   request-dump--->{argc:<2>
   cmd:
   argv[0]:
   argv[1]:<1#4d501a2>
   }
   return:helloworld
8. GET working
   request-dump--->{argc:<2>
   cmd:
   argv[0]:
   argv[1]:<1#4d501a2>
   }
   return:helloworld
9. GET working
   request-dump--->{argc:<2>
   cmd:
   argv[0]:
   argv[1]:<1#4d501a2>
   }
   return:helloworld
10. GET not working
    request-dump--->{argc:<2>
    cmd:
    argv[0]:
    argv[1]:<1#4d501a2>
    }
    return:helloworldD�

Thanks in advance :)
Loïc

[search]: buffered-tree root-to-leaf message applied

The particular require of Buffered-Tree search.
We should check the txn visibility during applying.

[status] cache line friendly counter

Replace status with CPU cache line friendly counter structure

[txn]: gc on MVCC

Run leaf MVCC garbage collection when serializing node to disk.

segmentation fault...

./nessdb_bench add        
Keys:       20 bytes each
Values:     100 bytes each
Entries:    1000000
IndexSize:  64.8 MB (estimated)
DataSize:   103.0 MB (estimated)
BG SYNC:    on...
--------------------------------------------------------------------------------------------------------------
nessDB:     version 1.7(Multiple && Distributable B+Tree with Level-LRU,Background IO Sync)
Date:       Sun Oct  9 09:30:51 2011
CPU:        2 *  AMD Athlon(tm) II X2 250 Processor
CPUCache:   1024 KB

zsh: segmentation fault  ./nessdb_bench add  

BATCH and RANGE operations

Hi, I discovered your project via a link from the skiplist wikipedia page,

it seems quite similar to leveldb - except for lacking batch operations and range (iterator) queries.

The reason I am interested in having feature parity, I've been working a lot with
leveldb (among many collaborators, gathering around this repo: https:github.com/rvagg/node-levelup )

Basically, we have taken a binding to leveldb, and then in a dynamic language with a good package manager, experimented and build many high level features via a dynamic language (node.js)

So, essentially, it's possible to not only install the data base via the languages package manager, but also, you can install every feature you may want, and then combine it with your own customizations. node.js is make writing custom servers very easy, and now, it's making writing custom databases easy. This isn't something many people are running in production yet (although some are) but it's a very fertile ground for experimentation.

We started with just a binding to leveldb, but have since expanded this, and have a number of different bindings to various forks of leveldb, as well as completely different databases, provided they have at least get, put, del, batch, and range operations,
including a polyfil on top of indexeddb (in chrome, firefox & ie10) so our database also works in web browsers!

Anyway, I was just posting this incase you think that batch and range operations where worth having.

Cheers, Dominic

key length limit

Is key length limited to SHA1_LENGTH-1?

binlog will not be flushed to disk

running db-bench write, and press ctrl-c to exit it. 0.log will contains no data and all writes were lost. But the db file is still large....

[cursor] add search for brt

point queries/range queries based on MVCC

[fifo]: nonleaf size maybe > 4MB

[checkpoint]: sharp checkpoint

We need a non-block sharp-checkpoint.
During the checkpoint, we CAN NOT:
0x00) block any client write/read

java binding on Mac OS X

I make nessDB and java-binding on Mac OS . And then using it with JVM.
I configure jvm with arguments "-Djava.library.path=/Users/wangxin/Documents/workspace/LogServer/lib/native".
But it's alway failed.
at last, I drop all nessDB file into xcode. and build a libnessdb.dylib.
JVM runs fine with dylib.
So I think there are musth be something wrong in Makefile

EXISTS after REMOVE crash nessDB in Lua and Python binding

This Python code :

import nessdb

db=nessdb.NessDB("test.db")
db.db_add("key","value")
print db.db_get("key")
print db.db_exists("key")
db.db_remove("key")
print db.db_exists("key")
db.db_close()

Cause nessDB to crash at "print db.db_exists("key")" with this message :

*** glibc detected *** python: free(): invalid pointer: 0xb737d0b0 ***

I have the same problem with my Lua binding.

Olivier.

build on windows

I have successfully built nessDB on Windows machine using MinGW 64 with modifications. I suggest you change "ht.h" at line 6 became:

typedef enum {STRING=0,INTEGER} KEYTYPE;

because INT already defined on windef.h (windows.h) with different type.

[tests]: mempool unit test

to add

return value unintialize compile error

I am studying data structures in nessDB.
It's very useful.
Thank you for your efforts 😆

Today, I got macbook.
compile nessDB and got error message below

gcc -Iengine -Idb -c -std=c99 -pedantic -W -Wall -Werror -fPIC -DOS_MACOSX -g2 -DINFO -DASSERT    -c -o engine/hdrserialize.o engine/hdrserialize.c
engine/hdrserialize.c:61:6: error: variable 'r' is used uninitialized whenever
      'if' condition is true [-Werror,-Wsometimes-uninitialized]
        if (!buf_xsum(wbuf->buf, wbuf->NUL, &xsum)) goto ERR;
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
engine/hdrserialize.c:87:9: note: uninitialized use occurs here
        return r;
               ^
engine/hdrserialize.c:61:2: note: remove the 'if' if its condition is always
      false
        if (!buf_xsum(wbuf->buf, wbuf->NUL, &xsum)) goto ERR;
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
engine/hdrserialize.c:34:7: note: initialize the variable 'r' to silence this
      warning
        int r;
             ^
              = 0
engine/hdrserialize.c:206:6: error: variable 'r' is used uninitialized whenever
      'if' condition is true [-Werror,-Wsometimes-uninitialized]
        if (!buf_xsum(wbuf->buf, wbuf->NUL, &xsum)) goto ERR;
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
engine/hdrserialize.c:226:9: note: uninitialized use occurs here
        return r;
               ^
engine/hdrserialize.c:206:2: note: remove the 'if' if its condition is always
      false
        if (!buf_xsum(wbuf->buf, wbuf->NUL, &xsum)) goto ERR;
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
engine/hdrserialize.c:189:7: note: initialize the variable 'r' to silence this
      warning
        int r;
             ^
              = 0

AP910759:nessDB kgcrom$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)
Target: x86_64-apple-darwin12.5.0
Thread model: posix
AP910759:nessDB kgcrom$ uname -a
Darwin AP910759.local 12.5.0 Darwin Kernel Version 12.5.0: Sun Sep 29 13:33:47 PDT 2013; root:xnu-2050.48.12~1/RELEASE_X86_64 x86_64

It's necessary to initialize return variables.
modified: engine/hdrserialize.c
modified: engine/serialize.c

[txn]: transaction rollback

to add

Why did not the query function?

I saw the source code of nessDB. But I not find the implement of 'db_get', like this:
int db_get(struct nessdb *db, struct msg *k, struct msg *v)
{
(void)db;
(void)k;
(void)v;

        return -1; 
}

Now I want to query data in database, what should I do?
@BohuTANG

[flush]: add flush to background thread

when gorged node flush to child, we should put it in background thread

[txn]: basement MVCC succinct layout design

a succinct design of MVCC layout

db_get interface doesn't return data length

The current nessdb interface makes it problematic to retrieve binary data (e.g. containing NIL - or even not containing any NILs).

void *db_get(struct nessdb *db, struct slice *sk);

Other DBM/KV databases like BDB, Tokyo Cabinet and LevelDB output the length as part of their 'get' call.
e.g.

size_t data_sz = 0;
char* data = tcbdbget(db, key, key_sz, &data_sz);

and

char *dberr = NULL;
size_t data_sz = 0;
char* data = leveldb_get(db, db_roptions, key, key_sz, &data_sz, &dberr);   

One way to implement this is to provide an output slice as the third argument to db_get and return the number of bytes retrieved (with 0 being an error condition)
Example:

struct slice key = {"derp", 4};
struct slice val = {NULL, 0};
if( ! db_get(db, &key, &val) ) {
    ... db_error(db)

request : shared object library target in makefile

Hello,

I would like to use Lua for my tests and i will use luaJIT FFI for the binding.
I need the shared object version of the nessDB library.

Can you add this target to Makefile ?

Thanks in advance, Olivier Goudron

[cache]: cache limits amplification

it seems that's a memory usage amplification in cache limits.

[tests]: unit test TODO

need to re-add

need -fPIC

gcc -g -Wall -fPIC db.o storage.o bloom.o ht.o level.o llru.o -shared -fPIC -o libnessdb.so
/usr/bin/ld: db.o: relocation R_X86_64_32S against `.bss' can not be used when making a shared object; recompile with -fPIC
db.o: could not read symbols: Bad value
collect2: ld returned 1 exit status

Documentation request : nessDB limits

Can you document some nessDB limits ?
I think about this :

• Max key size
• Max value size
• Max keys number
• Max db file size

Thanks in advance, Olivier Goudron

db file grows forever?

according to nessDB.txt , I can't see when to resize ness.db? Does that mean the db file will grow forever? Is there any way to resize this file after a merge or deletion?

request : implementation of shutdown command in db-server

Can you implement the shutdown command in db-server ?

Thanks in advance, Olivier Goudron

make nessDB error in centos 6.4

make error message:
cc1: warnings being treated as errors
tree/layout.c: In function ‘_pack_base’:
tree/layout.c:127: error: implicit declaration of function ‘do_xsum’
tree/layout.c: In function ‘_deserialize_nonleaf_from_disk’:
tree/layout.c:405: error: ‘CRC_SIZE’ undeclared (first use in this function)
tree/layout.c:405: error: (Each undeclared identifier is reported only once
tree/layout.c:405: error: for each function it appears in.)
make: *** [tree/layout.o] Error 1

The reason is that there are lack of some header files.
Then make OK when I modified this as following:
$ git diff
diff --git a/tree/hdr.c b/tree/hdr.c
index b5e8eb0..262a239 100644
--- a/tree/hdr.c
+++ b/tree/hdr.c
@@ -9,6 +9,7 @@
#include "compress/compress.h"
#include "msgpack.h"
#include "hdr.h"
+#include "crc32.h"

/**********************************************************************
diff --git a/tree/layout.c b/tree/layout.c
index 9cec6ec..b8bb85f 100644
--- a/tree/layout.c
+++ b/tree/layout.c
@@ -11,6 +11,7 @@
#include "msgpack.h"
#include "compress/compress.h"
#include "layout.h"
+#include "crc32.h"

/**********************************************************************
*

FYI.

Thanks!

Quick patch to build nessdb-bench on windows, I include result too.

patch for src/db.c

41c41,43
< #include <pthread.h>

---
> #ifdef _WIN32
> #include <direct.h>
> #endif
77a80,82
>       #ifdef _WIN32
>       _mkdir(DB_DIR);
>       #else
78a84
>       #endif

add src/build.bat

@echo off
gcc -std=c99 -pedantic -Wall -Wextra -Os -c db.c
gcc -std=c99 -pedantic -Wall -Wextra -Os -c ht.c
gcc -std=c99 -pedantic -Wall -Wextra -Os -c level.c
gcc -std=c99 -pedantic -Wall -Wextra -Os -c llru.c
gcc -std=c99 -pedantic -Wall -Wextra -Os -c storage.c
ar -rcs -o libnessdb.a db.o ht.o level.o llru.o storage.o
gcc -std=c99 -pedantic -Wall -Wextra -Os -c nessdb-bench.c
gcc -s -o nessdb-bench nessdb-bench.o libnessdb.a -lws2_32
nessdb-bench add

rem test on Compaq Presario CQ42, CPU: Intel core i3 M 350 @ 2.27GHz, OS: Windows 7 64-bit
rem random write performance increase from 6.3 MB/sec (version 1.7) to 6.5 MB/sec

rem D:\local\nessDB-1.8\src>nessdb-bench add
rem Keys:           16 bytes each
rem Values:         80 bytes each
rem Entries:        2000000
rem IndexSize:      45.8 MB (estimated)
rem DataSize:       160.2 MB (estimated)
rem --------------------------------------------------------------------------------------------------------------
rem nessDB:         version 1.8(Multiple && Distributable B+Tree with Level-LRU,Page-Cache)
rem Date:           Mon Nov 14 10:00:32 2011
rem +-----------------------+---------------------------+----------------------------------+---------------------+
rem |Random-Write   (done:1999999): 0.000016 sec/op; 62971.3 writes/sec(estimated); 6.5 MB/sec; cost:31.760(sec)
rem +-----------------------+---------------------------+----------------------------------+---------------------+
rem |Random-Read    (found:20000): 0.000009 sec/op; 106837.0 reads /sec(estimated); 11.0 MB/sec; cost:0.187(sec)
rem +-----------------------+---------------------------+----------------------------------+---------------------+
rem |Seq-Read       (found:20000): 0.000009 sec/op; 106837.6 reads /sec(estimated); 11.0 MB/sec; cost:0.187(sec)
rem +-----------------------+---------------------------+----------------------------------+---------------------+

[status]: more details for leaf/nonleaf memory usage

add more status to track the memory usage.
such as:
0x00) every leaf memory usage
0x01) every nonleaf memory usage
0x02) every fifo memory usage