Giter Site home page Giter Site logo

yinqiwen / ardb Goto Github PK

View Code? Open in Web Editor NEW
1.8K 127.0 278.0 71.39 MB

A redis protocol compatible nosql, it support multiple storage engines as backend like Google's LevelDB, Facebook's RocksDB, OpenLDAP's LMDB, PerconaFT, WiredTiger, ForestDB.

License: BSD 3-Clause "New" or "Revised" License

C++ 86.37% C 11.91% Makefile 0.64% Lua 1.08%
redis key-value persistence database server c-plus-plus rocksdb lmdb leveldb wiredtiger

ardb's People

Contributors

aaronlidman avatar arjunmehta avatar cvan avatar gitter-badger avatar ipapapa avatar jianxinluo avatar olleolleolle avatar omegablitz avatar purplegrape avatar rafiot avatar samgaw avatar squalfof avatar temoto avatar ualtinok avatar wallin avatar yinqiwen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ardb's Issues

zrange limit ignored

Current version ignores zrange limit when over zset-max-ziplist-entries.
I checked zset type only, but other data types may have similar problems.

Memory use rise to maximum in 24 hours

Hi,
we have ardb as a hdd replacement for redis. System has about 1500 requests using ardb per second.
After ardb stop/start it uses few MB. Memory usage continuously rises to 75% of total. Then we restart the process, another 24 hours and same thing.

We use only commands: hget, hset, pipeline, hgetall, mset, hincrby.
We connect to ardb using socket from Credis PHP class with native php extension.

cat /proc/version

Linux version 3.2.0-4-amd64 ([email protected]) (gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian 3.2.54-2

ARDB conf

home ..
daemonize yes
pidfile ${ARDB_HOME}/ardb.pid
port 16379
thread-pool-size 2
unixsocket /tmp/ardb.sock
unixsocketperm 777
leveldb.block_cache_size 512m
leveldb.block_size 0
leveldb.write_buffer_size 128m
leveldb.block_restart_interval 0
leveldb.max_open_files 1024
leveldb.bloom_bits 10
leveldb.batch_commit_watermark 1024
timeout 0
tcp-keepalive 0
loglevel debug
logfile stdout
data-dir ${ARDB_HOME}/data
repl-dir ${ARDB_HOME}/repl
slave-serve-stale-data yes
slave-priority 100
slave-read-only yes
backup-dir ${ARDB_HOME}/backup
repl-disable-tcp-nodelay no
repl-backlog-size 100m
slave-cleardb-before-fullresync yes
repl-state-persist-period 5
slowlog-log-slower-than 10000
slowlog-max-len 128
lua-time-limit 5000
check_type_before_set_string 0
hash-max-ziplist-entries 128
hash-max-ziplist-value 64
list-max-ziplist-entries 128
list-max-ziplist-value 64
set-max-ziplist-entries 128
set-max-ziplist-value 64
zset-max-ziplist-entries 128
zset-max-ziplist-value 64

If You need more information, write me. We want to solve problem without changing ardb.

keys command is extremely slow

The result of the keys command is extremely slow,

redis 127.0.0.1:6200> keys P-road-DEU:784-*

  1. "P-road-DEU:784-10220333310101"
  2. "P-road-DEU:784-10220333310110"
  3. "P-road-DEU:784-10221222022231"
  4. "P-road-DEU:784-10221222022233"
    (62.50s)

redis 127.0.0.1:6200> INFO

Server

ardb_version:0.6.0
ardb_home:/map_int/home/larroy/devel
engine:LevelDB
gcc_version:4.6.4
server_key:10f5be480f2c0783ecd9678876f9cd756ed8395d
tcp_port:6200

Clients

connected_clients:1

Databases

data_dir:/map_int/home/larroy/devel/data
Compactions

Level Files Size(MB) Time(sec) Read(MB) Write(MB)

0 1 82 4 0 108
1 1 1 1 36 18
2 50 84 4 84 84
3 564 992 12 238 237
4 1144 2125 19 408 408

Disk

db_used_space:3493166346

Replication

role: master
repl_dir: /map_int/home/larroy/devel/repl
connected_slaves: 0
master_repl_offset: 29236438303
repl_backlog_size: 104857600
repl_backlog_first_byte_offset: 29131580704
repl_backlog_histlen: 104857600

Stats

total_commands_processed:811453
total_connections_received:3
period_commands_processed(1min):1
redis 127.0.0.1:6200> dbsize
(integer) 3493172996
redis 127.0.0.1:6200>

Memory leak in importing from rdb?

I've been trying to import my 17GB rdb dump in ARDB and I never managed to get to finish it. I'm always left with 13 GB in data folder and the ARDB server getting killed by the OS for running out of memory.

This is the graph of memory usage last night when I left the import process going
https://www.dropbox.com/s/6yy6wg87znm750p/Screenshot%202014-03-23%2011.11.48.png

As you can see, the active memory (green) raises steadily for one hour and then OS releases it after the ARDB is killed.

I'm running ardb in Ubuntu 12.04 in ec2

As usual, let me know if there's something I can do to help the investigation!

Thanks

Replication doesnt behave like Redis

I've just done some testing with blank databases (rm ./data and ./repl folders) and slaveof 10.0.1.120:6379 in ardb.conf. when I start ardb-server, i would expect to see a full sync with the master (in my case, the master is a read only slave to an upstream master) but ardb issues a psync and only gets a very small set of data from the master, which is 2gb

ardb console shows

[97718] 04-27 10:38:23,706 INFO Init storage engine success.
[97718] 04-27 10:38:23,707 WARN No zookeeper servers specified, zookeeper agent would not start.
[97718] 04-27 10:38:23,707 INFO Server started, Ardb version 0.7.1
[97718] 04-27 10:38:23,707 INFO The server is now ready to accept connections on port 2222
[97718] 04-27 10:38:23,709 INFO [Slave]Remote master is a Redis 2.8.7 instance, support partial sync:1
[97718] 04-27 10:38:23,710 INFO Send PSYNC 8e7595dbe06494799fe6b38fd875e718426bb9c6 1 \n
[97718] 04-27 10:38:23,710 INFO Recv psync reply:FULLRESYNC 57b1d385ee78fd25afa5eba471834d4e30c7f6f3 33246672 
[97718] 04-27 10:38:41,071 INFO [Slave]Remote master is a Redis 2.8.7 instance, support partial sync:1 
[97718] 04-27 10:38:41,071 INFO Send PSYNC 57b1d385ee78fd25afa5eba471834d4e30c7f6f3 33246672 
[97718] 04-27 10:38:41,072 INFO Recv psync reply:CONTINUE 

network listening shows the following (10.0.1.120 is the readonly master, 10.0.1.50 is the ardb server)

###
T 10.0.1.50:32067 -> 10.0.1.120:6379 [A]
  ......
#
T 10.0.1.50:32067 -> 10.0.1.120:6379 [AP]
  info Server..
##
T 10.0.1.120:6379 -> 10.0.1.50:32067 [AP]
  $412..# Server..redis_version:2.8.7..redis_git_sha1:00000000..redis_git_dirty:0..redis_build_id:4e504c85df90d66d
  ..redis_mode:standalone..os:Linux 3.14.0-031400-generic x86_64..arch_bits:64..multiplexing_api:epoll..gcc_versio
  n:4.8.1..process_id:1040..run_id:57b1d385ee78fd25afa5eba471834d4e30c7f6f3..tcp_port:6379..uptime_in_seconds:3906
  55..uptime_in_days:4..hz:10..lru_clock:1447474..config_file:/etc/redis/6379.conf....
#
T 10.0.1.50:32067 -> 10.0.1.120:6379 [A]
  ......
#
T 10.0.1.50:32067 -> 10.0.1.120:6379 [AP]
  replconf listening-port 2222..
#
T 10.0.1.120:6379 -> 10.0.1.50:32067 [AP]
  +OK..
#
T 10.0.1.50:32067 -> 10.0.1.120:6379 [AP]
  psync 8e7595dbe06494799fe6b38fd875e718426bb9c6 1..
#
T 10.0.1.120:6379 -> 10.0.1.50:32067 [AP]
  +FULLRESYNC 57b1d385ee78fd25afa5eba471834d4e30c7f6f3 33246672..
#
T 10.0.1.50:32067 -> 10.0.1.120:6379 [A]
  ......
##
T 10.0.1.50:32067 -> 10.0.1.120:6379 [AF]
  ......
####
T 10.0.1.50:32151 -> 10.0.1.120:6379 [A]
  ......
#
T 10.0.1.50:32151 -> 10.0.1.120:6379 [AP]
  info Server..
##
T 10.0.1.120:6379 -> 10.0.1.50:32151 [AP]
  $412..# Server..redis_version:2.8.7..redis_git_sha1:00000000..redis_git_dirty:0..redis_build_id:4e504c85df90d66d
  ..redis_mode:standalone..os:Linux 3.14.0-031400-generic x86_64..arch_bits:64..multiplexing_api:epoll..gcc_versio
  n:4.8.1..process_id:1040..run_id:57b1d385ee78fd25afa5eba471834d4e30c7f6f3..tcp_port:6379..uptime_in_seconds:3906
  72..uptime_in_days:4..hz:10..lru_clock:1447476..config_file:/etc/redis/6379.conf....
#
T 10.0.1.50:32151 -> 10.0.1.120:6379 [A]
  ......
#
T 10.0.1.50:32151 -> 10.0.1.120:6379 [AP]
  replconf listening-port 2222..
#
T 10.0.1.120:6379 -> 10.0.1.50:32151 [AP]
  +OK..
#
T 10.0.1.50:32151 -> 10.0.1.120:6379 [AP]
  psync 57b1d385ee78fd25afa5eba471834d4e30c7f6f3 33246672..
#
T 10.0.1.120:6379 -> 10.0.1.50:32151 [AP]
  +CONTINUE..
#
T 10.0.1.120:6379 -> 10.0.1.50:32151 [A]
  .*2..$6..SELECT..$1..0..*4..$4..HSET..$2......$14....{..R....J.....$6....\S.@..*4..$4..HSET..$2......$14..N....5
  oR..$i5...$6....\S....*4..$4..HSET..$2......$4..nY....$6....\SN...*4..$4..HSET..$2......$14..B.Wg..7.YI..$h..$6.
  ...\SG...*4..$4..HSET..$2..=...$14.............@x,..$6....\S_...*4..$4..HSET..$2..l...$14..I.l8.......).s..$6...
  .\S....*4..$4..HSET..$2......$14....VW/./..f......$6....\S.2..*4..$4..HSET..$2..L...$4....'...$6....\S....*4..$4
  ..HSET..$2...&..$4.....0..$6....\SN...*4..$4..HSET..$2..V^..$14..v...$;.*.u C....$6....\SC...*4..$4..HSET..$2...
  ...$14....?...H8.....7..$6....\SI...*4..$4..HSET..$2......$14..A..._.......l(..$6....\S....*4..$4..HSET..$2.._..
  .$4..x+....$6....\S....*4..$4..HSET..$2......$14..a5...T...l...a..$6....\S....*4..$4..HSET..$2...|..$14..|H...9]
  /;..^....$6....\S....*4..$4..HSET..$2..W\[email protected]..$6....\S....*4..$4..HSET..$2...7..$4..;:qz..$6..
  ..\S....*4..$4..HSET..$2......$14....GF.M..62......$6....\S....*4..$4..HSET..$2...=..$4........$6....\Ss...*4..$
  4..HSET..$2..;K..$14..<...s.N..4.. 7..$6....\S....*4..$4..HSET..$2......$4.....:..$6....\S....*4..$4..HSET..$2..
  ....$14...pcy^.6.........$6....\S3...*4..$4..HSET..$2......$14..x.s. ....C.R.i..$6....\S....*4..$4..HSET..$2..PO
  ..$4..._....$6....\SM...*4..$4..HSET..$2......$14......]6..*...D...$6....\S....*4..$4..HSET..$2..=...$14..S&....
  ..}r......$6....\S(...*4..$4..HSET..$2......$14..p.{.nnm....rH...$6....\S....*4..$4..HSET..$2.._...$4..nY5...$6.
  ...\S&...*4..$4..HSET..$2......$14..D_..3Q.....;....$6....\S....*4..$4..HSET..$2..Wq..$4..<..j..$6....\SY...*4..
  $4..HSET..$2..>t..$14.....T............$6....\S....*4..$4..HSET..$2...j..$14..P.n...Ms...Ji...$6....\S....*4..$4
  ..HSET..$2..X...$4........$6....\S....*4..$4..HSET..$2..*1..$14....'....\.q\..t..$6....\S?...*4..$4..HSET..$2...
  ...$14..(aT."TJ.....oz..$6....\S....*4..$4..HSET..$2...!..$14..fx].kP..?L.4I...$6....\S....*4..$4..HSET..$2.....
  .$14....'.G..\ .U..<..$6....\Sz...*4..$4..HSET..$2..:^..$14............L{.k..$6....\S....*4..$4..HSET..$2......$
  4....$]..$6....\S....*4..$4..HSET..$2......$4..nR....$6....\S....*4..$4..HSET..$2..<|..$14..$G..............$6..
  ..\S.f..*4..$4..HSET..$2..9...$14....B7U7.r5.<.....$6....\S....*4..$4..HSET..$2../...$4........$6....\S/...*4..$
  4..HSET..$2...S..$14...3...M..&..0....$6....\S....*4..$4..HSET..$2...x..$4..x+....$6....\SY/..*4..$4..HSET..$2..
  ....$14..1.l.0"..........$6....\S....*4..$4..HSET..$2......$14......).....j?....$6....\S....*4..$4..HSET..$2....
  ..$4...&>...$6....\S....*4..$4..HSET..$2..D...$14..C}t.F#..8."..c..$6....\S....*4..$4..HSET..$2..Q...$14......pD
  .jM2.;....$6....\S....*4..$4..HSET..$2......$14....Gs...y........$6....\S*...*4..$4..HSET..$2...K..$14..x.pzm.9r
  0.......$6....\S....*4..$4..HSET..$2..J_..$14...`aB....92.+....$6....\S....*4..$4..HSET..$2..!Y..$14..q..c.2....
  .J!...$6....\S....*4..$4..HSET..$2.../..$4..[..v..$6....\S....*4..$4..HSET..$2...4..$14....J....Sd....c..$6....\
  Sy...*4..$4..HSET..$2...c..$4..u.g...$6....\S....*4..$4..HSET..$2......$14..-..prk..3..x.y..$6....\S.7..*4..$4..
  HSET..$2..+...$4..x%.-..$6....\Sb...*4..$4..HSET..$2...$..$14....V...!Ij.......$6....\S....*4..$4..HSET..$2..;].
  .$14..?..J..2.... X...$6....\S....*4..$4..HSET..$2......$4...&>...$6....\S....*4..$4..HSET..$2..EC..$4...,.g..$6
  ....\S.i..*4..$4..HSET..$2..%...$14..A(.5Yq..^.}.....$6....\S....*4..$4..HSET..$2...'..$14.."Y.Q.Q....q.|b..$6..
  ..\S....*4..$4..HSET..$2..j...$14.....zn..;....}~..$6....\S....*4..$4..HSET..$2......$14......c..d........$6....
  \S....*4..$4..HSET..$2......$14...=..".?.2..C.H..$6....\S....*4..$4..HSET..$2......$4.....c..$6....\Sz...*4..$4.
  .HSET..$2..FJ..$14....'...."U!^nD...$6....\S....*4..$4..HSET..$2......$14...7......[.@]....$6....\S....*4..$4..H
  SET..$2......$4...M....$6....\S....*4..$4..HSET..$2../O..$4.....P..$6....\S....*4..$4..HSET..$2......$14....IL&.
  .',.2..=..$6....\S....*4..$4..HSET..$2...H..$14........k.........$6....\Sh...*4..$4..HSET..$2...Z..$14......&...
  ..|h....$6....\S....*4..$4..HSET..$2..\C..$4...)....$6....\S.C..*4..$4..HSET..$2...g..$14.......E...Iw.....$6...
  .\S.c..*4..$4..HSET..$2......$14......`.?.5..l.#..$6....\S....*4..$4..HSET..$2...r..$14...|.KTQ8.gH.>k;..$6....\
  S....*4..$4..HSET..$2......$14..<,I..z$I.....U..$6....\S....*4..$4..HSET..$2......$4...&>...$6....\S....*4..$4..
  HSET..$2...|..$4........$6....\S."..*4..$4..HSET..$2......$4...&>...$6....\S....*4..$4..HSET..$2..5...$14..{.^L.
  +;A0s.R.3..$6....\So...*4..$4..HSET..$2......$14...m.~...%.'......$6....\S!...*4..$4..HSET..$2......$4...,....$6
  ....\S....*4..$4..HSET..$2......$14..'...*.ais~H..r..$6....\S<...*4..$4..HSET..$2..{...$14..CnN....*D.......$6..
  ..\S....*4..$4..HSET..$2......$4..x+....$6....\S."..*4..$4..HSET..$2..FJ..$14....'...."U!^nD...$6....\S....*4..$
  4..HSET..$2..e...$14...7[K.|Z....{....$6....\S....*4..$4..HSET..$2../O..$4.....P..$6....\S....*4..$4..HSET..$2..
  .X..$14...(F....}...7.T..$6....\S. ..*4..$4..HSET..$2......$14..'...*.ais~H..r..$6....\S=...*4..$4..HSET..$2..#.
  ..$14.......E%D.....T..$6....\S....*4..$4..HSET..$2......$4..x+....$6....\S."..*4..$4..HSET..$2..@...$14...a.R#r
  ...n..Vg..$6....\S....*4..$4..HSET..$2...v..$4....%...$6....\Sq...*4..$4..HSET..$2......$14..a5...T...l...a..$6.
  ...\S....*4..$4..HSET..$2......$14..Kn.e4...F.".....$6....\Sw...*4..$4..HSET..$2......$14..9.6...s`...).}..$6...
  .\S....*4..$4..HSET..$2...{..$14...%.|.....'..L...$6....\S....*4..$4..HSET..$2......$4..po....$6....\S....*4..$4
  [email protected].......;L.#..$6....\S....*4..$4..HSET..$2......$4.....:..$6....\S....*4..$4..HSET..$2...
  H..$14.......".IM.i..B..$6....\S!...*4..$4..HSET..$2......$14..(V...n.....J.[..$6....\S....*4..$4..HSET..$2..V^.
  .$14..v...$;.*.u C....$6....\SD...*[email protected].....$6....\S"...*4..$4..HSET..$2...=..$
  4........$6....\St...*4..$4..HSET..$2...1..$14.....1..y..r7}....$6....\S*...*4..$4..HSET..$2......$14......%....
  .5:.)..$6....\S.`..*4..$4..HSET..$2..U...$14..fW.Mo.c.....!$..$6....\S ...*4..$4..HSET..$2..)}..$14..+....[.h_}.
  ..Z..$6....\S....*4..$4..HSET..$2..!...$4.....*..$6....\S....*4..$4..HSET..$2...h..$4.....V..$6....\S. ..*4..$4.
  .HSET..$2......$14...n..a@....}..N..$6....\S....*4..$4..HSET..$2...b..$14..j..u.vWs..[.e%..$6....\S....*4..$4..H
  SET..$2..D;..$14..A*zAu*.I'.......$6....\S....*4..$4..HSET..$2..a$..$14......O:z5.H5x90..$6....\S....*4..$4..HSE
  T..$2...@..$4........$6....\S....*4..$4..HSET..$2......$4........$6....\S....*4..$4..HSET..$2......$14........5.
  ...`....$6....\S....*4..$4..HSET..$2......$14..,u,[........,...$6....\S....*4..$4..HSET..$2..E/..$14..0...e....;
  X.....$6....\S....*4..$4..HSET..$2..<i..$4..e.(...$6....\S....*4..$4..HSET..$2..m...$4...,....$6....\S~Z..*4..$4
  ..HSET..$2..l...$4.....j..$6....\S.n..*4..$4..HSET..$2..s...$14...>,..B..:w9l....$6....\S....*4..$4..HSET..$2...
  !..$14....v.._..cro5....$6....\S....*4..$4..HSET..$2..O...$4..nR.G..$6....\S....*4..$4..HSET..$2...[..$14....1[.
  .|..O!Fo...$6....\SQ...*4..$4..HSET..$2..j...$14.....zn..;....}~..$6....\S....*4..$4..HSET..$2..j...$14.....zn..
  ;....}~..$6....\S....*4..$4..HSET..$2...L..$14...T...m.&.g..{...$6....\S....*4..$4..HSET..$2..<...$14..@.....}`T
  .ieA<..$6....\S....*4..$4..HSET..$2......$4...M....$6....\S....*4..$4..HSET..$2......$4...M....$6....\S....*4..$
  4..HSET..$2......$14....2a.....#......$6....\S....*4..$4..HSET..$2......$14.....7Fc.J%.F..K..$6....\S....*4..$4.
  .HSET..$2..,L..$14..Q.W~P}a..t.~.I..$6....\S....*4..$4..HSET..$2......$14........4o$...D...$6....\S....*4..$4..H
  SET..$2...c..$14......bei.........$6....\S....*4..$4..HSET..$2......$14...>..W....8/.....$6....\S ...*4..$4..HSE
  T..$2.._...$4....$I..$6....\Sz...*4..$4..HSET..$2......$4..jx....$6....\SW...*4..$4..HSET..$2...v..$4........$6.
  ...\S....*4..$4..HSET..$2..Z...$14...w...02../+../..$6....\Sl...*4..$4..HSET..$2......$4...X.e..$6....\Sh...*4..
  $4..HSET..$2..O...$14.. ...|.....f.....$6....\SU...*4..$4..HSET..$2...}..$14...E..............$6....\S,...*4..$4
  ..HSET..$2......$4..po....$6....\S....*4..$4..HSET..$2...>..$14..h..0...;........$6....\S....*4..$4..HSET..$2..Q
  _..$14......d".E...z....$6....\S....*4..$4..HSET..$2......$4.....:..$6....\S'...*4..$4..HSET..$2..@...$14...W.7.
  ...._*.`<..$6....\Sd...*4..$4..HSET..$2..VL..$14..`Y........j.....$6....\S....*4..$4..HSET..$2..UW..$4..x+.Q..$6
  ....\S....*4..$4..HSET..$2..f...$14......3.8p.J..t...$6....\Sy...*4..$4..HSET..$2..E...$14.............:_?..$6..
  ..\S....*4..$4..HSET..$2...'..$14.."Y.Q.Q....q.|b..$6....\S....*4..$4..HSET..$2......$14..(aT."TJ.....oz..$6....
  \S....*4..$4..HSET..$2......$4..po....$6....\S.,..*4..$4..HSET..$2...0..$14.......4.2...._4..$6....\S0`..*4..$4.
  .HSET..$2......$14.....dj.J.V..f4...$6....\S....*4..$4..HSET..$2..;...$14..#*..p.M..G .....$6....\S....*4..$4..H
  SET..$2......$4.....c..$6....\S{...*4..$4..HSET..$2..\...$4..u.....$6....\S....*4..$4..HSET..$2......$14..vd....
  ..Y...\...$6....\S....*4..$4..HSET..$2../...$4........$6....\S0...*4..$4..HSET..$2......$14...L.c.. ..._.mt..$6.
  ...\S....*4..$4..HSET..$2..v...$14....m.......H.....$6....\S....*4..$4..HSET..$2..]F..$4...X.b..$6....\S#...*4..
#
T 10.0.1.120:6379 -> 10.0.1.50:32151 [AP]
  $4..HSET..$2......$14.. ....p%4.....1..$6....\S....*4..$4..HSET..$2......$14..".....).v...p=..$6....\S....*4..$4
  ..HSET..$2...V..$14...I@`oOC.?....9..$6....\Sb...*4..$4..HSET..$2......$4...c w..$6....\S....*4..$4..HSET..$2...
  ~..$14..]...S.....M.!K..$6....\S....*4..$4..HSET..$2......$4........$6....\S....*4..$4..HSET..$2...X..$14...(F..
  ..}...7.T..$6....\S. ..*4..$4..HSET..$2......$14...%....^dHZk..(..$6....\S(...*4..$4..HSET..$2...v..$4....%...$6
  ....\Ss...*4..$4..HSET..$2......$14...MB6,V..V[Db.6..$6....\S....*4..$4..HSET..$2......$14...MPL..-...ab.s..$6..
  ..\S....*4..$4..HSET..$2...f..$4..x%....$6....\S....*4..$4..HSET..$2......$14......{G...A.Y.I..$6....\S1...*4..$
  4..HSET..$2......$14.....?....a.......$6....\S....*4..$4..HSET..$2...K..$4..nR.%..$6....\S....*4..$4..HSET..$2..
  .X..$14...(F....}...7.T..$6....\S. ..*4..$4..HSET..$2......$14...`..E.......|)..$6....\S)...*4..$4..HSET..$2...v
  ..$4....%...$6....\Ss...*4..$4..HSET..$2......$14..n.X...4....f:K..$6....\SM...*4..$4..HSET..$2..%...$14..+r....
  =.Z.......$6....\S<...*4..$4..HSET..$2..+...$14..00L.G....h......$6....\S....*4..$4..HSET..$2..b...$4..x!....$6.
  ...\S....*4..$4..HSET..$2..'...$14..........j....9..$6....\Sf...*4..$4..HSET..$2......$14..i.....&.q..`....$6...
  .\S!...*4..$4..HSET..$2......$4..nZ>S..$6....\Sq...*4..$4..HSET..$2...p..$14...I..$.......w...$6....\S....*4..$4
  ..HSET..$2......$4....}...$6....\S....*4..$4..HSET..$2...&..$14..D..V3pLR.z].@...$6....\S$...*4..$4..HSET..$2...
  ...$14..R....P..Z.=k.-..$6....\S....*4..$4..HSET..$2......$14..a5...T...l...a..$6....\S....*4..$4..HSET..$2.....
  .$14..cW....^~...E....$6....\So...*4..$4..HSET..$2......$14...A....%......H..$6....\S....*4..$4..HSET..$2..i...$
  14....:.fs.....<.S..$6....\S....*4..$4..HSET..$2......$4.....:..$6....\S....*4..$4..HSET..$2..1c..$4..x!.[..$6..
  ..\S....*4..$4..HSET..$2...t..$14..R..H}.C.J....%..$6....\S.c..*4..$4..HSET..$2......$14..'...*.ais~H..r..$6....
  \S>...*4..$4..HSET..$2......$14..s...x...8U.m....$6....\S....*4..$4..HSET..$2..S...$4........$6....\S....*4..$4.
  .HSET..$2..U...$14...+... [..w..6...$6....\S....*4..$4..HSET..$2..D...$4..x+.1..$6....\S. ..*4..$4..HSET..$2..}.
  ..$14...$.W..,"(gYzh]..$6....\S....*4..$4..HSET..$2......$14....B...E..[.k}e..$6....\S....*4..$4..HSET..$2..xG..
  $4...).z..$6....\S.y..*4..$4..HSET..$2..E`..$14...P@.'.A..U.t.Y..$6....\SH...*4..$4..HSET..$2...}..$4..po....$6.
  ...\S....*4..$4..HSET..$2......$14...)v........9....$6....\Sr...*4..$4..HSET..$2...~..$4..^..x..$6....\S....*4..
  $4..HSET..$2..<...$14.....h.%....-.....$6....\S....*4..$4..HSET..$2..b'..$14......5Lz\M...z...$6....\S....*4..$4
  ..HSET..$2......$14......%.....5:.)..$6....\S `..*4..$4..HSET..$2..S*..$4..q.....$6....\S....*4..$4..HSET..$2...
  ...$14...Q.l...9..[R....$6....\S....*4..$4..HSET..$2...h..$4.....V..$6....\S. ..*4..$4..HSET..$2...B..$14..{.N.P
  W....,.O...$6....\S....*4..$4..HSET..$2......$14..cW....^~...E....$6....\Sp...*4..$4..HSET..$2...X..$14...(F....
  }...7.T..$6....\S. ..*4..$4..HSET..$2...'..$14......b.W...k.2O..$6....\S....*4..$4..HSET..$2......$4..u.. ..$6..
  ..\S....*4..$4..HSET..$2..-...$14....#.L2...'......$6....\S ...*4..$4..HSET..$2..b ..$14...u..7..@...\....$6....
  \S....*4..$4..HSET..$2..1c..$4..x!.[..$6....\S....*4..$4..HSET..$2...v..$4....%...$6....\St...*4..$4..HSET..$2..
  ....$14...k~......~U..9..$6....\S....*4..$4..HSET..$2..\6..$14.......cTv...=rJ..$6....\S....*4..$4..HSET..$2....
  ..$4...&>...$6....\S....
#
T 10.0.1.50:32151 -> 10.0.1.120:6379 [A]
  ......
#
T 10.0.1.120:6379 -> 10.0.1.50:32151 [AP]
  *4..$4..HSET..$2...:..$14...#......9.g.....$6....\SS...
#
T 10.0.1.50:32151 -> 10.0.1.120:6379 [A]
  ......
#
T 10.0.1.120:6379 -> 10.0.1.50:32151 [AP]
  *4..$4..HSET..$2......$14....'.G..\ .U..<..$6....\S{...
#
T 10.0.1.120:6379 -> 10.0.1.50:32151 [AP]
  *4..$4..HSET..$2..C...$14..T.v..u.#.8...?..$6....\S....*4..$4..HSET..$2......$14..+.l...W....INV..$6....\S....*4
  ..$4..HSET..$2......$4..x%....$6....\SC...*4..$4..HSET..$2...x..$14....e.....e..h3z..$6....\Sf;..
#
T 10.0.1.50:32151 -> 10.0.1.120:6379 [A]
  ......
#
T 10.0.1.120:6379 -> 10.0.1.50:32151 [AP]
  *4..$4..HSET..$2..>...$4..po....$6....\S....

Large LevelDB cache

Hi,
I'd like to mimic as much as possible the in-memory performance of Redis by keeping a fixed size LRU cache of say, 8GB. The bulk (30+GB ) of the least used stuff can stay off-cache, no problem.
Is ARDB a good solution for this task?

As far as I understood from the configuration file, I could specify the LevelDB conf parameters:

leveldb.block_cache_size       8g
leveldb.block_size             0
leveldb.write_buffer_size      128m
leveldb.block_restart_interval 0
leveldb.max_open_files         10240
leveldb.bloom_bits             10
leveldb.batch_commit_watermark 1024

Is this correct? Can you advise on further configuration?

Thanks

Ardb does not compile in OSX (mavericks, Xcode 5.0.2)

Hi, I'm trying to compile master branch from source, but I cannot get this error passed:

$ make
cd src && /Applications/Xcode.app/Contents/Developer/usr/bin/make all
g++ -c -Wall -g  -O2  -fPIC -D__STDC_FORMAT_MACROS -DARDB_VERSION='"0.6.0"'  -I./ -I/tmp/ardb/src/../deps/cpp-btree -I/tmp/ardb/src/../deps/lua/src -I/tmp/ardb/src/../deps/snappy-1.1.1 -I/tmp/ardb/src/../deps/zookeeper-3.4.5/include -I/tmp/ardb/src/../deps/zookeeper-3.4.5/generated -I/tmp/ardb/src/../deps/leveldb-1.16.0/include ardb.cpp -o ardb.o
In file included from ardb.cpp:30:
In file included from ./ardb.hpp:38:
In file included from ./common.hpp:171:
In file included from /tmp/ardb/src/../deps/cpp-btree/btree_set.h:27:
/tmp/ardb/src/../deps/cpp-btree/btree.h:114:10: fatal error: 'tr1/type_traits' file not found
#include CPP_BTREE_TYPE_TRAITS_HEADER
         ^
/tmp/ardb/src/../deps/cpp-btree/btree_config.h:32:40: note: expanded from macro 'CPP_BTREE_TYPE_TRAITS_HEADER'
  #define CPP_BTREE_TYPE_TRAITS_HEADER <tr1/type_traits>
                                       ^
1 error generated.
make[1]: *** [ardb.o] Error 1
make: *** [all] Error 2

I got stuck, could you please help or provide documentation for the procedure under OSX?

info command

"info" command killed its process.

redis 127.0.0.1:16379> info
Could not connect to Redis at 127.0.0.1:16379: Connection refused

GET a certain key takes forever

I'm running in a large instance in AWS EC2.
I can get other existing and non-existing keys really fast. Unfortunately though, GET with a certain key, the system hangs forever.

I had to control-C because it was taking too long, as you can see it went beyond 4 minutes.

$ time redis-cli get mpme.cache.artist.1314
"\n\n\b\x01\x12\x041314\x18\x01\x12\x1eThomas Meinecke Framus Waikiki\x1a\x12\n\x0bfamiliarity\x12\x030.0\x1a\x0e\n\ahotness\x12\x030.0\x1a\t\n\x04year\x12\x010\x1a,\n\nsearchable\x12\x1ethomas meinecke framus waikiki2\x1ethomas_meinecke_framus_waikiki"

real    0m0.005s
user    0m0.001s
sys 0m0.004s
ubuntu@ip-10-73-130-120:~/ardb$ time redis-cli get mpme.cache.artist.1314xxxx
(nil)

real    0m0.005s
user    0m0.000s
sys 0m0.006s 
$ time redis-cli get mpme.cache.artist.1
    ^C
real    4m29.751s
user    0m0.000s
sys 0m0.006s

$ redis-cli info
# Server
ardb_version:0.6.0
ardb_home:/home/ubuntu
engine:LevelDB
gcc_version:4.8.1
server_key:3cceed9e5fc800774a0afcbf728665e53f57b5ac
tcp_port:6379
# Clients
connected_clients:42
# Databases
data_dir:/home/ubuntu/data
                               Compactions
Level  Files Size(MB) Time(sec) Read(MB) Write(MB)
--------------------------------------------------
  1        0        0        15        0       315
  2        0        0        59      315      1113
  3      356      716        50     1295       898

# Disk
db_used_space:799018418
# Replication
role: master
repl_dir: /home/ubuntu/repl
connected_slaves: 0
master_repl_offset: 2124903417
repl_backlog_size: 104857600
repl_backlog_first_byte_offset: 2020045818
repl_backlog_histlen: 104857600
# Stats
total_commands_processed:3394961
total_connections_received:1224
period_commands_processed(1min):3

$ redis-cli keyscount \*
(integer) 1167650

I suspect the leveldb files are getting corrupted. Can it be?
Please advise!

Redsmin - ardb GUI ?

Hi :)

Beside the "Unsupported Redis Commands:" all other Redis commands are supported (for instance DUMP/RESTORE) ?

Disclaimer:
I'm the founder of Redsmin an all-in-one Redis GUI and I wonder if ardb will work with it!

hyperloglog code aborts compilation

On Ubuntu (13.10+), the default compiler aborts with errors compiling the hyperloglog code (goto jumps variable initialisation)

hyperloglog.cpp: In function ‘int hllSparseAdd(char*, unsigned char, size_t, uint32_t)’:
hyperloglog.cpp:928:5: warning: jump to label ‘updated’ [-fpermissive]
updated:
^
hyperloglog.cpp:835:14: warning: from here [-fpermissive]
goto updated;
^
hyperloglog.cpp:918:9: error: crosses initialization of ‘int deltalen’
int deltalen = seqlen - oldlen;
^
hyperloglog.cpp:917:9: error: crosses initialization of ‘int oldlen’
int oldlen = is_xzero ? 2 : 1;
^
hyperloglog.cpp:916:9: error: crosses initialization of ‘int seqlen’
int seqlen = n - seq;
^
hyperloglog.cpp:854:9: error: crosses initialization of ‘int last’
int last = first + span - 1; /* Last register covered by the sequence. /
^
hyperloglog.cpp:853:22: error: crosses initialization of ‘uint8_t
n’
uint8_t seq[5], *n = seq;
^
hyperloglog.cpp:928:5: warning: jump to label ‘updated’ [-fpermissive]
updated:
^

Backup format and Redis compatibility

I would like to try the geospatial index of ardb but I have some concern about the format of the backup.

It seems that the format of ardb's backup file is in .ardb, I am wondering is it basically the same underlying format of redis's .rdb file. Assume that I only use those function that is redis compatible, can the .ardb format be directly imported in redis?

And if I use the function unique to ardb like GeoAdd, do I have to worry about error when syncing with redis? Suppose I have an ardb as master and redis as slave then would those incompatible data be ignored correctly?

ZREVRANGE not work with WITHSCORES

Hi, there.

Could you support WITHSCORES option in ZREVRANGE command?

redis 127.0.0.1:16379> ZADD ztest 1 test1
(integer) 1
redis 127.0.0.1:16379> ZADD ztest 100 test2
(integer) 1
redis 127.0.0.1:16379> ZADD ztest 10 test3
(integer) 1
redis 127.0.0.1:16379> ZRANGE ztest 0 -1 WITHSCORES
1) "test1"
2) "1"
3) "test3"
4) "10"
5) "test2"
6) "100"
redis 127.0.0.1:16379> ZREVRANGE ztest 0 -1 WITHSCORES
(error) ERR syntax error

Thanks

ardb-server with lmdb storage running at 100% CPU

Hello,

I'm currently evaluating ardb for a use-case we have. However I noticed that after pushing some data into the DB (~2GB), the server began to use up 100% of the CPU.
Even after a restart it directly goes up to 100% without any clients being connected.

When starting with an empty DB the server behaves normally.

I set the log level as high as I could but it didn't show anything suspicious. I'm happy to provide any extra information in case you need something specific.

Thanks,
Chris

Lua scripting (redis compatible) in Ardb ?

Hello!

The last thing that is blocking me from switching one of my project from Redis to Ardb (besides the "not production ready" state) is the loss of lua scripting. Is this on the roadmap?

Thanks :)

rocksdb engine not compiling anymore since version update

It looks like the commit 13921f0 has broken the rocksdb engine:

It has been working fine before, but now it doesn't compile properly and fails with the following error:

/home/server/ardb/src/../deps/rocksdb-rocksdb-3.1/librocksdb.a(env_posix.o): In function `rocksdb::(anonymous namespace)::PosixEnv::NowNanos()':
/home/server/ardb/deps/rocksdb-rocksdb-3.1/util/env_posix.cc:1327: undefined reference to `clock_gettime'
/home/server/ardb/src/../deps/rocksdb-rocksdb-3.1/librocksdb.a(format.o): In function `Zlib_Uncompress':
/home/server/ardb/deps/rocksdb-rocksdb-3.1/./port/port_posix.h:252: undefined reference to `inflateInit2_'
/home/server/ardb/deps/rocksdb-rocksdb-3.1/./port/port_posix.h:275: undefined reference to `inflate'
/home/server/ardb/deps/rocksdb-rocksdb-3.1/./port/port_posix.h:303: undefined reference to `inflateEnd'
/home/server/ardb/src/../deps/rocksdb-rocksdb-3.1/librocksdb.a(format.o): In function `BZip2_Uncompress':
/home/server/ardb/deps/rocksdb-rocksdb-3.1/./port/port_posix.h:372: undefined reference to `BZ2_bzDecompressInit'
/home/server/ardb/deps/rocksdb-rocksdb-3.1/./port/port_posix.h:391: undefined reference to `BZ2_bzDecompress'
/home/server/ardb/deps/rocksdb-rocksdb-3.1/./port/port_posix.h:410: undefined reference to `BZ2_bzDecompressEnd'
/home/server/ardb/src/../deps/rocksdb-rocksdb-3.1/librocksdb.a(format.o): In function `Zlib_Uncompress':
/home/server/ardb/deps/rocksdb-rocksdb-3.1/./port/port_posix.h:297: undefined reference to `inflateEnd'
/home/server/ardb/src/../deps/rocksdb-rocksdb-3.1/librocksdb.a(format.o): In function `BZip2_Uncompress':
/home/server/ardb/deps/rocksdb-rocksdb-3.1/./port/port_posix.h:416: undefined reference to `BZ2_bzDecompressEnd'
/home/server/ardb/src/../deps/rocksdb-rocksdb-3.1/librocksdb.a(block_based_table_builder.o): In function `Zlib_Compress':
/home/server/ardb/deps/rocksdb-rocksdb-3.1/./port/port_posix.h:192: undefined reference to `deflateInit2_'
/home/server/ardb/src/../deps/rocksdb-rocksdb-3.1/librocksdb.a(block_based_table_builder.o): In function `BZip2_Compress':
/home/server/ardb/deps/rocksdb-rocksdb-3.1/./port/port_posix.h:319: undefined reference to `BZ2_bzCompressInit'
/home/server/ardb/deps/rocksdb-rocksdb-3.1/./port/port_posix.h:338: undefined reference to `BZ2_bzCompress'
/home/server/ardb/deps/rocksdb-rocksdb-3.1/./port/port_posix.h:354: undefined reference to `BZ2_bzCompressEnd'
/home/server/ardb/src/../deps/rocksdb-rocksdb-3.1/librocksdb.a(block_based_table_builder.o): In function `Zlib_Compress':
/home/server/ardb/deps/rocksdb-rocksdb-3.1/./port/port_posix.h:213: undefined reference to `deflate'
/home/server/ardb/deps/rocksdb-rocksdb-3.1/./port/port_posix.h:237: undefined reference to `deflateEnd'
/home/server/ardb/deps/rocksdb-rocksdb-3.1/./port/port_posix.h:231: undefined reference to `deflateEnd'
/home/server/ardb/src/../deps/rocksdb-rocksdb-3.1/librocksdb.a(block_based_table_builder.o): In function `BZip2_Compress':
/home/server/ardb/deps/rocksdb-rocksdb-3.1/./port/port_posix.h:360: undefined reference to `BZ2_bzCompressEnd'
collect2: error: ld returned 1 exit status
make[1]: *** [test] Error 1
make[1]: Leaving directory `/home/server/ardb/src'
make: *** [all] Error 2

Still using gcc-4.8 and g++-4.8.

---Chris

expiration key bug with key longer than 23 symbols

Hi, first of all, thx for you work on ArDb, it is a very good solution.

I found a bug with expiration key, it won't work if you have a key longer than 23 symbols.

This test will fail on the last line:

        const string keyLength23 = "qwertyuiopqwertyuiopqwe";
        const string keyLength24 = "qwertyuiopqwertyuiopqwer";

        RedisClient.Set(keyLength23, "vlaue123", TimeSpan.FromSeconds(2));
        RedisClient.Set(keyLength24, "vlaue123", TimeSpan.FromSeconds(2));

        var valueKeyLength23 = RedisClient.Get(keyLength23);
        var valueKeyLength24 = RedisClient.Get(keyLength24);

        Assert.AreNotEqual(valueKeyLength23, null);
        Assert.AreNotEqual(valueKeyLength24, null);

        Thread.Sleep(TimeSpan.FromSeconds(2));

        valueKeyLength23 = RedisClient.Get(keyLength23);
        valueKeyLength24 = RedisClient.Get(keyLength24);

        Assert.AreEqual(valueKeyLength23, null);
        Assert.AreEqual(valueKeyLength24, null);

Unexpected $-1 reply on set

(gdb) p key
$9 = "U-02266a222036de6d8f00000000"
(gdb) p value
$10 = "402266a222036de6d8f00000000\000\000\000\000\000\000\000\000\004\301\377V\262c\023&@\277\016\234\063\242|I@l&\337ls\023&@\342\344~\207\242|I@\000\200@\020\b$01266a222
000000000$0136de6d8f00000000\301\377V\262c\023&@\277\016\234\063\242|I@l&\337ls\023&@\342\344~\207\242|I@"
(gdb) down
#2 0x0000000000cf19e6 in redis::base_clientwom::WomHasher::recv_ok_reply_ (this=0x7ffff2e53c80, socket=14) at /map_int/home/larroy/devel/mmcc/frontend/external-src/redis

/redisclient.h:2263
(gdb) down
#1 0x0000000000cf5302 in redis::base_clientwom::WomHasher::recv_single_line_reply_ (this=0x7ffff2e53c80, socket=14) at /map_int/home/larroy/devel/mmcc/frontend/external-

src/redis/redisclient.h:2256
(gdb) p line
$11 = "$-1"
(gdb)

I don't know why I got $-1

Failed ardb-test: test_zsets_zrange

I see this error when running ardb-test.

../test/zset_testcase.cpp:61: Assertion values.size() != 2 failed in test_zsets_zrange

I am using Ubuntu 14.04. Any idea where may have gone wrong?

keys pattern not work

Pattern matching not working in my case.
It returns all the keys stored in ardb.

redis 127.0.0.1:16379> set abc test1
OK
redis 127.0.0.1:16379> set def test2
OK
redis 127.0.0.1:16379> keys ab*
1) "abc"
2) "def"
redis 127.0.0.1:16379> keys unstored_key
1) "abc"
2) "def"

How to choose the storage engine

ARDB is a great software, but the fact that leaves three options to the user in term of storage engine always made me wonder, leaving me with some questions:

  1. From an ARDB user perspective, what are the major differences between the offered storage engines?
  2. For what task one or the other is more or less optimal?
  3. Facebook's RocksDB extends LevelDB and benchmarks show great benefits on all sides in favour of RocksDB.
    https://github.com/facebook/rocksdb/wiki/Performance-Benchmarks
    Then, why to maintain LevelDB storage engine at all?

All in all, from a product management point of view, one wants to minimise the number of knobs and buttons exposed to the user (enhance usability) , prevent configuration from allowing less than optimal performance, and lastly keep the architecture and dependencies to a minimum.

Also, it's perfectly healthy for a project to abandon parts of code. It means you're removing complexity (less code, less bugs principle).

Benchmarks need "units"

Hey there!

I figured out, after reading the documentation for redis-benchmark, that the graph on your README represents queries per second.

It would be nice if that was mentioned below the graph.

Thanks!

Segfault on atomic_get_and_set_vptr_t

I've got a new crash, the only one since yesterday around 5pm (around 22 hours ago).
I'm pretty sure we're going fast getting this system stable. Thanks so much for your effort.

Here's the gdb backtrace and debug log extract.

$ gdb ardb/src/ardb-server core.7583.ardb-server
GNU gdb (GDB) 7.6.1-ubuntu
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/ubuntu/ardb/src/ardb-server...done.
[New LWP 7616]
[New LWP 7615]
[New LWP 7617]
[New LWP 7583]
[New LWP 7614]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `src/ardb-server /etc/ardb.conf'.
Program terminated with signal 11, Segmentation fault.
#0  0x000000000052fa6e in atomic_get_and_set_vptr_t (var=0x303a7367616cb6, v=0x7f1af0dd07a0) at ./util/concurrent_queue.hpp:70
70  ATOMIC_FUNC_XCHG(get_and_set, "xchgq %1, %0", vptr_t)
(gdb) backtrace
#0  0x000000000052fa6e in atomic_get_and_set_vptr_t (var=0x303a7367616cb6, v=0x7f1af0dd07a0) at ./util/concurrent_queue.hpp:70
#1  0x0000000000532aa8 in ardb::MPSCQueue<ardb::ChannelAsyncIOContext>::Push (this=0x303a7367616cb6, value=...) at ./util/concurrent_queue.hpp:257
#2  0x00000000005314a1 in ardb::ChannelService::AsyncIO (this=0x303a7367616c66, ctx=...) at channel/channel_service.cpp:529
#3  0x00000000005319a4 in ardb::ChannelService::AsyncIO (this=0x303a7367616c66, id=308557619,
    cb=0x43e5fe <ardb::ArdbServer::AsyncWriteBlockListReply(ardb::Channel*, void*)>, data=0x7f1a705cb610) at channel/channel_service.cpp:629
#4  0x000000000043f2d4 in ardb::ArdbServer::CheckBlockingConnectionsByListKey (this=0x7fffab724a20, key=...) at lists.cpp:174
#5  0x000000000044017c in ardb::ArdbServer::RPush (this=0x7fffab724a20, ctx=..., cmd=...) at lists.cpp:352
#6  0x00000000004b7f2a in ardb::ArdbServer::DoRedisCommand (this=0x7fffab724a20, ctx=..., setting=0x7f1bfa0d0818, args=...) at ardb_server.cpp:581
#7  0x00000000004b7a83 in ardb::ArdbServer::ProcessRedisCommand (this=0x7fffab724a20, ctx=..., args=..., flags=0) at ardb_server.cpp:517
#8  0x00000000004b8445 in ardb::RedisRequestHandler::MessageReceived (this=0x7f1a704abac0, ctx=..., e=...) at ardb_server.cpp:624
#9  0x00000000004c826b in ardb::ChannelUpstreamHandler<ardb::codec::RedisCommandFrame>::HandleStreamEvent (this=0x7f1a704abac0, ctx=..., e=...)
    at ./channel/channel_upstream_handler.hpp:133
#10 0x00000000004c81c2 in ardb::ChannelPipeline::SendUpstream<ardb::codec::RedisCommandFrame> (this=0x7f1a704b04b8, ctx=0x7f1bf8b8caf0, e=...)
    at channel/all_includes.hpp:89
#11 0x00000000004c7ff9 in ardb::ChannelHandlerContext::SendUpstream<ardb::MessageEvent<ardb::codec::RedisCommandFrame> > (this=0x7f1bf8b8c370, e=...)
    at channel/all_includes.hpp:164
#12 0x00000000004c7b94 in ardb::fire_message_received<ardb::codec::RedisCommandFrame> (ctx=..., message=0x7f1bed962ac0, destructor=0x0)
    at ./channel/channel_helper.hpp:91
#13 0x00000000004c7879 in ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode (this=0x7f1bf8b8c0a0, context=..., channel=0x7f1a704b0480,
    cumulation=...) at ./channel/codec/stack_frame_decoder.hpp:102
#14 0x00000000004c6d52 in ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::MessageReceived (this=0x7f1bf8b8c0a0, ctx=..., e=...)
    at ./channel/codec/stack_frame_decoder.hpp:157
#15 0x0000000000528c1d in ardb::ChannelUpstreamHandler<ardb::Buffer>::HandleStreamEvent (this=0x7f1bf8b8c0a0, ctx=..., e=...)
    at ./channel/channel_upstream_handler.hpp:133
#16 0x0000000000528b8e in ardb::ChannelPipeline::SendUpstream<ardb::Buffer> (this=0x7f1a704b04b8, ctx=0x7f1bf8b8c370, e=...) at ./channel/all_includes.hpp:89
#17 0x0000000000528af8 in ardb::ChannelPipeline::SendUpstream<ardb::MessageEvent<ardb::Buffer> > (this=0x7f1a704b04b8, event=...) at ./channel/all_includes.hpp:128
#18 0x0000000000528a72 in ardb::fire_message_received<ardb::Buffer> (channel=0x7f1a704b0480, message=0x7f1a704b04e8, destructor=0x0) at ./channel/channel_helper.hpp:83
#19 0x0000000000527feb in ardb::Channel::OnRead (this=0x7f1a704b0480) at channel/channel.cpp:472
#20 0x0000000000526c95 in ardb::Channel::IOEventCallback (eventLoop=0x7f1bf8cf72e8, fd=3813, clientData=0x7f1a704b0480, mask=1) at channel/channel.cpp:50
#21 0x0000000000553e37 in aeProcessEvents (eventLoop=0x7f1bf8cf72e8, flags=3) at channel/redis/ae.c:429
#22 0x0000000000554196 in aeMain (eventLoop=0x7f1bf8cf72e8) at channel/redis/ae.c:485
#23 0x00000000005308a3 in ardb::ChannelService::Start (this=0x7f1bf926da00) at channel/channel_service.cpp:332
#24 0x00000000005306bc in ardb::ChannelService::LaunchThread::Run (this=0x7f1bf89444f0) at channel/channel_service.cpp:306
#25 0x000000000050b3e8 in ardb::Thread::ThreadFunc (data=0x7f1bf89444f0) at util/thread/thread.cpp:38
#26 0x00007f1bfb5c2f6e in start_thread (arg=0x7f1bed963700) at pthread_create.c:311
#27 0x00007f1bfaacf9cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

More info from debug logs

grep -B6 "Server started" log/*
log/ardb-server.log.2-[7583] 04-08 11:11:17,941 DEBUG Process recved cmd:rpush mpme.search.response.D2A8BB3B-BF45-4A33-86D8-037A31B41E12
log/ardb-server.log.2-
log/ardb-server.log.2-pink floyd
log/ardb-server.log.2��~j�t��? with flags:0
log/ardb-server.log.2-[2002] 04-08 11:14:19,065 WARN Replication backlog buffer is disabled, and current server can NOT accept slave connections.
log/ardb-server.log.2-[2002] 04-08 11:14:19,065 WARN No zookeeper servers specified, zookeeper agent would not start.
log/ardb-server.log.2:[2002] 04-08 11:14:19,065 INFO Server started, Ardb version 0.7.0

Description of master-master replication

Hi, it is a very interesting project!

Could you describe in a bit more detail (both high-level view and may be some implementation details) the master-master replication and main differences from a master-slave replication? Also, what are the main algorithmic deviations from original Redis replication design when it comes to implementation besides rewriting it in C++?

It would be also interesting to understand better if persistent backends affect the functionality of replication as compared to a vanilla Redis replication.

BTW, is it possible to use unmodified Redis slaves against ardb servers? Is it possible to use ardb slaves agains unmodified Redis servers? I.e. is it drop in replacement (assuming that only supported commands are used) ?

Thanks,
Leo

redis Compatibility issues

With redis-command, the error is as follows

darcy@darcy-sony:~/filmserv$ redis-commander 
path.existsSync is now called `fs.existsSync`.
listening on  8081

/usr/local/lib/node_modules/redis-commander/node_modules/redis/index.js:525
            throw err;
                  ^
TypeError: Cannot call method 'split' of undefined
  at RedisClient.on_info_cmd (/usr/local/lib/node_modules/redis-commander/node_modules/redis/index.js:332:23)
  at /usr/local/lib/node_modules/redis-commander/node_modules/redis/index.js:367:14
  at try_callback (/usr/local/lib/node_modules/redis-commander/node_modules/redis/index.js:522:9)
  at RedisClient.return_reply (/usr/local/lib/node_modules/redis-commander/node_modules/redis/index.js:592:13)
  at RedisReplyParser.<anonymous> (/usr/local/lib/node_modules/redis-commander/node_modules/redis/index.js:265:14)
  at RedisReplyParser.EventEmitter.emit (events.js:95:17)
  at RedisReplyParser.send_reply (/usr/local/lib/node_modules/redis-commander/node_modules/redis/lib/parser/javascript.js:279:18)
  at RedisReplyParser.execute (/usr/local/lib/node_modules/redis-commander/node_modules/redis/lib/parser/javascript.js:223:22)
  at RedisClient.on_data (/usr/local/lib/node_modules/redis-commander/node_modules/redis/index.js:478:27)
  at Socket.<anonymous> (/usr/local/lib/node_modules/redis-commander/node_modules/redis/index.js:79:14)
  at Socket.EventEmitter.emit (events.js:95:17)
  at Socket.<anonymous> (_stream_readable.js:746:14)
  at Socket.EventEmitter.emit (events.js:92:17)
  at emitReadable_ (_stream_readable.js:408:10)
  at emitReadable (_stream_readable.js:404:5)
  at readableAddChunk (_stream_readable.js:165:9)
  at Socket.Readable.push (_stream_readable.js:127:10)
  at TCP.onread (net.js:526:21)

Redis Desktop Manager 0.7.6 cannot use, too

General protection crash

ARDB does not segfault anymore and works well, but I still have some crashes. In syslog I keep reading this:

Apr  2 09:16:09 ip-10-49-11-32 kernel: [21072227.063770] traps: ardb-server[13046] general protection ip:427c10 sp:7f1e93bfc3a0 error:0 in ardb-server[400000+238000]

Probably the kernel is killing the process after it's attempting to read or write in forbidden memory addresses (as far as I understood from wikipedia).

Memory-wise we're never going OOM, I can see from the historical graph. Unfortunately this kind of error does not generate a core file, so, beside that log line, I'm not sure how to debug any further.

Any advice on how to investigate further?

Thank you!

About zset performance

Now I found zrange/zrank become slower for large offset.

So, it's difficult to use zset for timeline or realtime ranking like Redis. In Redis case, zset is sometimes used to avoid performance degradation of lrange for large offset, I think.
Do you have any idea to improve the performance?

And some issues found.

  1. Table data not support some SQL like commands
redis 127.0.0.1:16379> tcreate table dm
OK
redis 127.0.0.1:16379> tinsert table 3 dm google.com pv 10 uv 2
OK
redis 127.0.0.1:16379> tinsert table dm=1,pv=10,uv=2
(error) ERR wrong number of arguments for 'tinsert' command
redis 127.0.0.1:16379> treplace table 3 dm google.com pv 10 uv 3
OK
redis 127.0.0.1:16379> treplace table dm=google.com,pv=10,uv=4
(error) ERR wrong number of arguments for 'treplace' command
  1. Zookeeper error
    I got following error too many times after I start Ardb with "zookeeper-servers 127.0.0.1:2181" option.
2013-11-08 20:04:59,633:31643:ZOO_INFO@check_events@1703: initiated connection to server [127.0.0.1:2181]
2013-11-08 20:04:59,639:31643:ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:2181], sessionId=0x142375fd56a0003, negotiated timeout=10000
[31643] 11-08 20:04:59,645 ERROR Failed to create path:/ at state:1 for reason:bad arguments

And the zookeeper option in Ardb is not described clearly. Is this Redis Sentinel like solution?

  1. "loglevel warning" option are ignored
    Maybe "loglevel warn" work.

  2. pidfile option not supported
    I'm using daemonized Ardb. pidfile option is helpful.

Thanks

Problems about replication

master[redis 2.8.3]->slave[ardb] case.

[Master redis]
127.0.0.1:6379> zadd ztest 100 abc
(integer) 1
127.0.0.1:6379> rename ztest ztest2
OK
127.0.0.1:6379> zrange ztest 0 -1
(empty list or set)
127.0.0.1:6379> zrange ztest2 0 -1
1) "abc"
[Slave ardb]
redis 127.0.0.1:16379> zrange ztest 0 -1
1) "abc"
redis 127.0.0.1:16379> zrange ztest2 0 -1
(empty list or set)

redis 127.0.0.1:16379> keys *
  1. rename not replicated.
  2. keys * return nothing and CPU usage over 100%!

ERROR No handler found

Hi,
I've switched to info mode and I can now clearly see a lot of these error messages I can't explain:

[9376] 04-13 02:57:08,116 ERROR No handler found for:mpme.search.response.44228063-a93c-4830-a07b-4144e5a813dc
[9376] 04-13 02:57:08,116 ERROR No handler found for:$2
[9376] 04-13 02:57:08,116 ERROR No handler found for:15
[9376] 04-13 02:57:08,116 ERROR No handler found for:
[9376] 04-13 02:57:08,116 ERROR No handler found for:$5
[9376] 04-13 02:57:08,116 ERROR No handler found for:$57
[9376] 04-13 02:57:08,116 ERROR No handler found for:mpme.search.response.44228063-a93c-4830-a07b-4144e5a813dc
[9376] 04-13 02:57:08,116 ERROR No handler found for:$2
[9376] 04-13 02:57:08,116 ERROR No handler found for:15
[9376] 04-13 03:02:46,725 ERROR No handler found for:
[9376] 04-13 03:02:46,726 ERROR No handler found for:$5
[9376] 04-13 03:02:46,726 ERROR No handler found for:$57
[9376] 04-13 03:02:46,726 ERROR No handler found for:mpme.search.response.44228063-a93c-4830-a07b-4144e5a813dc
[9376] 04-13 03:02:46,726 ERROR No handler found for:$2
[9376] 04-13 03:02:46,726 ERROR No handler found for:15
[9376] 04-13 10:04:19,564 ERROR No handler found for:
[9376] 04-13 10:04:19,583 ERROR No handler found for:$3
[9376] 04-13 10:04:19,583 ERROR No handler found for:$23
[9376] 04-13 10:04:19,583 ERROR No handler found for:mpme.cache.artist.15932
[9376] 04-13 10:04:19,583 ERROR No handler found for:
[9376] 04-13 10:48:06,909 ERROR No handler found for:
[9376] 04-13 10:48:06,927 ERROR No handler found for:$5
[9376] 04-13 10:48:06,927 ERROR No handler found for:$57
[9376] 04-13 10:48:06,928 ERROR No handler found for:mpme.search.response.f1767dec-a743-4e75-8d0c-d219313e8bb0
[9376] 04-13 10:48:06,928 ERROR No handler found for:$2
[9376] 04-13 10:48:06,928 ERROR No handler found for:15
[9376] 04-13 13:03:07,878 ERROR No handler found for:
[9376] 04-13 13:03:07,902 ERROR No handler found for:$5
[9376] 04-13 13:03:07,902 ERROR No handler found for:$57
[9376] 04-13 13:03:07,902 ERROR No handler found for:mpme.search.response.acbaba48-5b21-450d-8434-3b67a7f592e4
[9376] 04-13 13:03:07,902 ERROR No handler found for:$2
[9376] 04-13 13:03:07,902 ERROR No handler found for:15
[9376] 04-13 13:28:47,931 ERROR No handler found for:
[9376] 04-13 13:28:48,019 ERROR No handler found for:$6
[9376] 04-13 13:28:48,025 ERROR No handler found for:$25
[9376] 04-13 13:28:48,025 ERROR No handler found for:mpme.fav.mpme.user.213388
[9376] 04-13 13:28:48,025 ERROR No handler found for:$1
[9376] 04-13 13:28:48,025 ERROR No handler found for:0
[9376] 04-13 13:28:48,025 ERROR No handler found for:$2
[9376] 04-13 13:28:48,025 ERROR No handler found for:-1
[9376] 04-13 13:28:48,025 ERROR No handler found for:
[9376] 04-13 13:28:48,025 ERROR No handler found for:$6
[9376] 04-13 13:28:48,025 ERROR No handler found for:$25
[9376] 04-13 13:28:48,025 ERROR No handler found for:mpme.fav.mpme.user.213388
[9376] 04-13 13:28:48,025 ERROR No handler found for:$1
[9376] 04-13 13:28:48,025 ERROR No handler found for:0
[9376] 04-13 13:28:48,025 ERROR No handler found for:$2
[9376] 04-13 13:28:48,025 ERROR No handler found for:-1

Although logs don't show what was the command for the corresponding keys, I know the traffic patterns of the clients and the keys shown are used by a variety of commands: GET, BLPOP, SMEMBERS.

Segfault under redis-benchmark

Hi,
Once I managed to import all my data, I decided to do some poor man's stress test. From localhost I invoked and completed redis-benchmark -q.

Then I tried to rise the number of connections to 500 and I had a segfault. From syslog:
Mar 24 19:29:56 1-ja-be-eu1 kernel: [29997069.525666] ardb-server[31167]: segfault at 7fe106800000 ip 00000000004ee6fa sp 00007fe170bfd990 error 4 in ardb-server[400000+22f000]

All I changed in the configuration is the listening port, daemonize=yes and LevelDB options as follows:

leveldb.block_cache_size       300m
leveldb.block_size             0
leveldb.write_buffer_size      128m
leveldb.block_restart_interval 0
leveldb.max_open_files         5000
leveldb.bloom_bits             10
leveldb.batch_commit_watermark 1024 

I restarted the server and tried to reproduce the segfault, but no luck. Can it have something to do with a post-rdb-import state of the memory?

I'm not sure how to collect more debug info than that segfault log line. Especially because I cannot reproduce it just yet.

I will try to re-run the import overnight and then redis-benchmark.

Segfault during low speed series of SET

I reproduced this segfault 3 or 4 times already. This is what I did:

  1. import 30GB worth of data from RDB dump
  2. redis-cli shutdown
  3. restart the ardb-server process
  4. refresh cache with my script that reads from MySQL, encodes entities to google protobuffer objects and writes them serialized into Redis keys (SET command).
  5. ardb-server segfaults after a few hundred SETs.

@yinqiwen you're gonna love this: I managed to obtain the core file! All I had to do was ulimit -c unlimited.

If you want to analyse the core dump, download it from here: http://www.uploadmb.com/dw.php?id=1396183938

Migration from Redis: IMPORT to support also RDB dump

I'm trying to migrate from Redis to ARDB my production database. I cannot use SLAVEOF because it's a hosted service and they don't not support it, since there's a lot of sharding behind the scenes.
What they do provide instead, is the .rdb dumps.

I tried to use the IMPORT command as explained here in ARDB wiki: https://github.com/yinqiwen/ardb/wiki/Backup-Commands

Unfortunately this only supports the .ardb format, which is apparently incompatible with Redis .rdb.

I already tried with external tools like rdb-tools but in vane, since my keys have binary values and they only support UTF-8 data for some reason.

I thought this can be a common migration pattern for people approaching ARDB. Any chances to get a .rdb imported?

Thanks

lmdb engine wont compile on gcc 4.8

storage_engine=lmdb make fails with the following

g++ -c -Wall -g -O2 -fPIC -fpermissive -D__STDC_FORMAT_MACROS -DARDB_VERSION='"0.7.1"' -D__USE_LMDB__ -I./ -I/home/gateway/dev/ardb/src/../deps/cpp-btree -I/home/gateway/dev/ardb/src/../deps/lua/src -I/home/gateway/dev/ardb/src/../deps/snappy-1.1.1 -I/home/gateway/dev/ardb/src/../deps/zookeeper-3.4.5/include -I/home/gateway/dev/ardb/src/../deps/zookeeper-3.4.5/generated -I/home/gateway/dev/ardb/src/../deps/mdb-mdb/libraries/liblmdb engine/lmdb_engine.cpp -o engine/lmdb_engine.o engine/lmdb_engine.cpp:321:45: warning: declaration of ‘virtual void ardb::LMDBIterator::Seek(const ardb::Slice&)’ outside of class is not definition [-fpermissive] void LMDBIterator::Seek(const Slice& target); ^ engine/lmdb_engine.cpp:322:2: error: expected unqualified-id before ‘{’ token { ^

"Killed: 9" under heavy load

I'm trying ardb on heavy load (lots of virtual clients and requests). The clients just do SET and GET and after more than an hour, it just dies and shows:

Killed: 9

What does it mean? And do you have any data on how much ardb can actually handle.

About snapshotting

Is snapshotting (save as in redis.conf) supported? Would there be a easy way to do backup automatically, like every hour?

command result shoud provide more information

erros from the storage engine should be propagated up or at least logged properly. I had leveldb returning corruption, this should be reported to the client or stated in the log.

replication logs

I've been testing ardb-slave:redis-2.8rc5 master (logs as 2.7.105) and having changed the slave to accept psync from this redis, the console now floods with

Can NOT happen since slave instance can NOT generate select command

It happens after replication each time to slave gets a new command. Im sending it HSET commands

I have noticed random core dumps when psync from redis 2.8rc5 is enabled that Im not seeing on a 2.6 master.

SREM behaviour differs from redis, SREM should return 1 when one key is deleted

ardb

127.0.0.1:6200> SADD aaas das
(integer) 1
127.0.0.1:6200> SMEMBERS aaas

  1. "das"
    127.0.0.1:6200> SREM aaas das
    (integer) 0
    127.0.0.1:6200>

REDIS

127.0.0.1:6379> SADD aas 1
(integer) 1
127.0.0.1:6379> SREM aas 1
(integer) 1
127.0.0.1:6379> SREM aas 1
(integer) 0
127.0.0.1:6379>

It should return the number of elements removed

127.0.0.1:6200> SADD aaas das
(integer) 1
127.0.0.1:6200> SADD aaas dasx
(integer) 1
127.0.0.1:6200> SREM aaas das dasx
(integer) 0
127.0.0.1:6200> SMEMBERS aaas
(empty list or set)
127.0.0.1:6200>

127.0.0.1:6379> SADD aas 1
(integer) 1
127.0.0.1:6379> SREM aas 1
(integer) 1
127.0.0.1:6379> SREM aas 1
(integer) 0
127.0.0.1:6379> SADD aas 1 2 3
(integer) 3
127.0.0.1:6379> SREM aas 1 2 3
(integer) 3
127.0.0.1:6379>

segfault report

I was playing around with the tables using the examples provided and I'm now getting a segfault

redis 10.0.0.90:6379> tgetall mytable

  1. "www.google.com"
  2. "page1"
  3. "10"
  4. "2"
    redis 10.0.0.90:6379> tget mytable 1 pv aggregate sum where domain=www.google.com
  5. (nil)
    redis 10.0.0.90:6379> tget mytable 1 pv aggregate sum where domain=www.google.com
  6. (nil)
    redis 10.0.0.90:6379> tget mytable 2 domain pv where domain=www.google.com and url=page1
    segfault

This is 0.5.1 compiled on a standard 64bit Ubuntu 13.04 running standard without specify a conf file. There is other existing data in the LevelDB as I was testing replication (which works well)

This Replicates the issue for me

redis 10.0.0.90:6379> tgetall mytable (empty list or set) redis 10.0.0.90:6379> tcreate mytable domain url OK redis 10.0.0.90:6379> tget mytable 2 domain pv where domain=www.google.com and url=page1 (empty list or set) redis 10.0.0.90:6379> tinsert mytable 4 domain www.google.com url page1 pv 10 uv 2 OK redis 10.0.0.90:6379> tget mytable 2 domain pv where domain=www.google.com and url=page1 Could not connect to Redis at 10.0.0.90:6379: Connection refused

gdb bt

#0 std::reverse_iterator >, btree::btree_map, std::allocator >, 256>, std::less > >, std::allocator > const, btree::btree_map, std::allocator >, 256> > >, 256> >, std::pair > const, btree::btree_map, std::allocator >, 256> >&, std::pair > const, btree::btree_map, std::allocator >, 256> >_> >::operator_ (
this=this@entry=0x7fffffffcdd0) at /usr/include/c++/4.7/bits/stl_iterator.h:165

#1 0x0000000000494220 in operator-> (this=0x7fffffffcdd0)

at /usr/include/c++/4.7/bits/stl_iterator.h:175

#2 ardb::Ardb::TInterIndexWalk::OnKeyValue (this=0x7fffffffcf10, k=,

v=<optimised out>, cursor=<optimised out>) at table.cpp:751

#3 0x0000000000458190 in ardb::Ardb::Walk (this=this@entry=0x7ffff64c9140, key=...,

reverse=reverse@entry=false, handler=handler@entry=0x7fffffffcf10) at ardb.cpp:428

#4 0x000000000048c83e in ardb::Ardb::TInterRowKeys (this=this@entry=0x7ffff64c9140,

db=@0x7ffff64c2b10: 0, tableName=..., schema=..., cond=..., prefetch_keyset=...,
interset=..., results=...) at table.cpp:764

#5 0x000000000048cc61 in ardb::Ardb::TGetIndexs (this=this@entry=0x7ffff64c9140,

db=@0x7ffff64c2b10: 0, tableName=..., schema=..., conds=..., prefetch_keyset=...,
indexs=@0x7fffffffd310: 0x7fffffffd2d0, temp=@0x7fffffffd318: 0x7fffffffd2e0)
at table.cpp:824

#6 0x000000000049169e in ardb::Ardb::TGet (this=0x7ffff64c9140, db=@0x7ffff64c2b10: 0,

tableName=..., options=..., values=..., err=...) at table.cpp:1160

#7 0x00000000004194ac in ardb::ArdbServer::TGet (this=0x7fffffffde90, ctx=..., cmd=...)

at ardb_server.cpp:2923

#8 0x000000000040d250 in ardb::ArdbServer::DoRedisCommand (this=this@entry=0x7fffffffde90,

ctx=..., setting=setting@entry=0x7ffff64c5218, args=...) at ardb_server.cpp:3142

#9 0x000000000041dbc1 in ardb::ArdbServer::ProcessRedisCommand (this=0x7fffffffde90,

ctx=..., args=..., flags=flags@entry=0) at ardb_server.cpp:3084

#10 0x000000000041ddff in ardb::RedisRequestHandler::MessageReceived (this=0x7ffff64c2b00,

ctx=..., e=...) at ardb_server.cpp:3182

#11 0x000000000040c173 in HandleStreamEvent (ctx=..., this=, e=...)

at ./channel/channel_upstream_handler.hpp:133

#12 ardb::ChannelPipeline::SendUpstreamardb::codec::RedisCommandFrame (

ctx=<optimised out>, e=..., this=<optimised out>) at channel/all_includes.hpp:89

#13 0x000000000042beb9 in SendUpstreamardb::MessageEvent<ardb::codec::RedisCommandFrame > (

e=..., this=<optimised out>) at channel/all_includes.hpp:164

#14 ardb::fire_message_receivedardb::codec::RedisCommandFrame (ctx=...,

message=message@entry=0x7fffffffda60, destructor=destructor@entry=0x0)
at ./channel/channel_helper.hpp:91

#15 0x000000000042c117 in ardb::codec::StackFrameDecoderardb::codec::RedisCommandFrame::CallDecode (this=0x7ffff656a520, context=..., channel=0x7ffff64c9780, cumulation=...)

at ./channel/codec/stack_frame_decoder.hpp:102

#16 0x000000000042c244 in ardb::codec::StackFrameDecoderardb::codec::RedisCommandFrame::MessageReceived (this=0x7ffff656a520, ctx=..., e=...)

at ./channel/codec/stack_frame_decoder.hpp:157

#17 0x00000000004baf93 in HandleStreamEvent (ctx=..., this=, e=...)

at ./channel/channel_upstream_handler.hpp:133

#18 ardb::ChannelPipeline::SendUpstreamardb::Buffer (ctx=, e=...,

this=<optimised out>) at ./channel/all_includes.hpp:89

#19 0x00000000004bb089 in SendUpstreamardb::MessageEvent<ardb::Buffer > (event=...,

this=0x7ffff64c97b8) at ./channel/all_includes.hpp:128

#20 fire_message_receivedardb::Buffer (destructor=0x0, message=0x7ffff64c9810,

channel=0x7ffff64c9780) at ./channel/channel_helper.hpp:83

#21 ardb::Channel::OnRead (this=0x7ffff64c9780) at channel/channel.cpp:449
#22 0x00000000004badad in ardb::Channel::IOEventCallback (eventLoop=,

fd=<optimised out>, clientData=0x7ffff64c9780, mask=<optimised out>)
at channel/channel.cpp:51

#23 0x00000000004cdffa in aeProcessEvents (eventLoop=eventLoop@entry=0x7ffff6430108,

flags=flags@entry=3) at channel/redis/ae.c:429

#24 0x00000000004ce30b in aeMain (eventLoop=0x7ffff6430108) at channel/redis/ae.c:485
#25 0x000000000041501a in ardb::ArdbServer::Start (this=this@entry=0x7fffffffde90, props=...)

at ardb_server.cpp:3340

#26 0x000000000040b56f in main (argc=, argv=) at main.cpp:108

Client connections hang forever without response

After some time of good service, sometimes ardb server stops responding to commands. This happened more than once in the past days.

The server gets in a state that connections are accepted (not sure if reads from the recv the buffer), but never responds with any data and never closes the connection.

That is, a simple redis-cli info is left hanging forever.

I used gcore to dump a core file of the running server (before killing and restarting it) and here is the backtrace:

$ gdb ardb/src/ardb-server core.23551
GNU gdb (GDB) 7.6.1-ubuntu
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/ubuntu/ardb/src/ardb-server...done.
[New LWP 23553]
[New LWP 23554]
[New LWP 23555]
[New LWP 23556]
[New LWP 23551]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `src/ardb-server'.
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.
(gdb) backtrace
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000000000053e483 in leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper(void*) ()
#2  0x00007f15b8a39f6e in start_thread (arg=0x7f15ac043700) at pthread_create.c:311
#3  0x00007f15b7f469cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

I hope this helps.

Ghost keys

Some keys are shown by the "keys" command but, when I try to get their values, they return nil

$ redis-cli keys mpme.fav.mpme.user.*
...
30704) "mpme.fav.mpme.user.91290"
30705) "mpme.fav.mpme.user.91302"
30706) "mpme.fav.mpme.user.91305"
...
$ redis-cli get mpme.fav.mpme.user.91290
(nil)
$ redis-cli get mpme.fav.mpme.user.91302
(nil)
$ redis-cli get mpme.fav.mpme.user.91305
(nil)

How is this even possible? This data was imported from rdb dump.

Hungs up on KEYS and KEYSCOUNT

I tried using the following commands on ardb:

KEYSCOUNT *

and

KEYS * from string test limit 1

but ardb hungs up and never responded to succeeding commands.

I also tried killing ardb-server (without and with SIGINT) but failed. I have to wait for around 30 minutes before it finally terminates. Upon checking the process, I was greeted with:

bitthegeek   21107   1.7 73.0 67250116 9163668 ??  TLs   1:45AM     1:15.90 ardb-server ardb-sadf.conf

Did this on FreeBSD 9.2-RELEASE 64-bit.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.