ideawu/ssdb

SSDB - A fast NoSQL database, an alternative to Redis

ssdb's Issues

Build a PHP demo: Notes

To demonstrate how SSDB works in real world.

del nonexistent key return ok

I think it‘s better to return not_found or change all del's returned values to deleted number.

send noop command with seq to slave when dump_end

Lua scripting

I started adding Lua (5.2) scripting to the latest master - https://github.com/dim/ssdb/commits/scripting. My C++ is terribly rusty, so I'd like you to take a look and comment before I proceed with https://github.com/dim/ssdb/blob/scripting/TODO#L6.

Flush DB via cli

Add a flushdb command so that the use can flush the db with ease

Slave's link.recv blocks on network failure

Slave's socket may block on read operation after a unknown network failure and blocks foever(STRANGE). So we need protocol level heartbeats to check this out. The BackendSync's send may encounter this situation(not sure).

So, I will change replication links on both master side and slave side to non-block mode.

clear slave's db before sync recover

need rscan, zrscan, hrscan

reverse scan

请问ssdb适合保存缩略图吗？

目前我们每张图片会生成20种左右的缩略图，容量共400G左右，前端采用Varnish顶着，一旦Varnish雪崩的话，会给后端造成很大的压力。

zkeys cmd comfused

zkeys key 0 9 5 does't work, but zkeys key any.char.here 0 9 5 works

诡异的内存占用，RES的值比FREE的buffer/cache的used大太多了

使用新版的SSDB出现了很诡异的问题，从top的内存使用情况如下：
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24235 admin 15 0 12.7g 2.0g 1.9g S 0.0 51.5 83:07.99 ssdb-server

但Free的结果如下：
total used free shared buffers cached
Mem: 4050044 3979464 70580 0 74872 2792404
-/+ buffers/cache: 1112188 2937856
Swap: 4194296 1430744 2763552

buffers/cache的used仅有1G的消耗，上一版的时候没有该问题，是引入jemalloc的自然现象吗，还是有其他问题。谢谢。

write replication logs into a different file

[Announce][warn!] SSDB 1.5.2 must be upgraded to 1.5.3

Users who use SSDB 1.5.2 MUST upgrade to 1.5.3, for version 1.5.2 has a servere bug, which may cause ssdb-server exit unexpectedly, when you backup data using ssdb-dump, or when a connected slave disconect.

This bug only affect SSDB 1.5.2.

Need exists/hexists/zexists commands

Need exists/hexists/zexists commands

and multi_*

平时ssdb的运行状态监控是怎么进行的？

有没有什么命令，啥的？

Java client api needed

replication hierarchy

• A node can be both master and slave at the same time
• A node can be set to ReadOnly mode
• A node can be set to WriteOnly mode
• So there would be multi write points of a SSDB network

ssdb-cli multi_get nonexistent keys error

error log:

Traceback (most recent call last):
File "/usr/local/ssdb/tools/../deps/cpy/cpy.py", line 65, in ?
execfile(dstfile)
File "/usr/local/ssdb/tools/cpy/ssdb-cli.py", line 281, in ?
sys.stdout.write('%d result(s) (%.3f sec)\n' % (len(resp.data['index']), time_consume))
KeyError: 'index'

Change hexists command to test if a member exists, and also zexists

Current, hexists command tests if a map exists, will change its meaning to test if a member exists.

And also zexists.

Replication mechanism redesign

Move replication out of leveldb, do not intercept leveldb's write operation. Log SSDB operations, and replay on slave side.

python client?

Hi,

老兄的SSDB好萌。不知道可否直接支持 https://github.com/andymccurdy/redis-py 这个算了？

It's good for adoption if users can switch to ssdb as drop-in replacement as redis.

LevelDB compaction IO speed limit

Must limit the speed of compaction IO, or the performance would be very low during compaction!

可以支持set和bitmap类型吗？

这是redis一个优秀的特性呀。用来记录每天登录的id数，然后通过交集、并集来计算次登之类的。

adding bloom filter support

need to list zsets and hashmaps

broken link

Hi,

this link from the README

https://github.com/ideawu/ssdb/blob/master/Commands

is broken.

Please fix, thanks!

请问可否指定内存占用上限？

例如配置最多占用512M内存，超出的就写硬盘。
这也是SSDB创建的初衷吧？

Need master-master architecture

hexists 和 hkeys 需要完善

1、hexists 函数只能判断hash表是否存在，不能判断表中是否存在某个key
2、hkeys 函数只有limit，不支持start，大数据量分页的效果无法实现

Need diff-patch mechanism on replication

how to control memory usage of ssdb

Hi, I use ssdb for about one month. Now the memory usage is too high. below is the dump of top:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4643 root 20 0 14.6g 7.8g 7.3g S 0.3 67.4 243:01.17 ssdb-server

all data file size is about 13.6G in data and 426M in meta:

du -sm *

13654 data
426 meta

So it seems ssdb put all data in memory. How could I control the total memory usage?
The current ssdb.conf is:
leveldb:
# in MB
cache_size: 1000
# in KB
block_size: 32
# in MB
write_buffer_size: 64

maybe don't need circular PersistentSynclogQueue

maybe don't need circular PersistentSynclogQueue, it makes things complicated, use a background thread to delete expired synclogs.

var/data & var/meta

这两个目录分别是什么作用？

我发现增加删除数据后(即ssdb没有任何数据)

即使重启ssdb后，这两个目录也会越来越大。

没有接口用于删除一个hash或者set，有计划添加吗

比如用：
hset hash_key h1 1
hset hash_key h2 2
...
这样添加几个k-v以后，没办法快速的将hask_key整个删除的指令。
只能用:
multi_hdel hask_key h1 h2 ...
来清空。

有没有一个hremove hask_key的方法呢?

TODO: Supports 'mode readonly/readwrite' commands

Purpose

• Gracefully shutdown master, let all data be syncchronized before is shutdown.
• Temperarily stop serving

如果hash类型的key是中文，如何获取全部的key

你好，如果hash的key部分是中文，用hkeys($name, '\001', 'XXXX', 1000)->data;
请问下要如何设置后面XXXX部分？谢谢。
还有几个问题请问下：
makefile需要自己加 -O2吗？没有见优化参数。

TODO: adding *push_front, *push_back, *pop_front, *pop_back commands

Adding persistent queue support for SSDB.

push_back/push
pop_front/pop
push_front/unshift
pop_back/shift

CPU 耗用问题

经过一个多月测试，虽然并发不大，内存占用也很少，但非常耗CPU。希望尽快解决下。

multi_zdel is not removing everything

When using multi_zdel it seems that not everything is correctly removed. Although the keys are gone, the sets themselves still remain. See example below:

> zset n a 1
  ok
  (0.003 sec)
> zset n b 2
  ok
  (0.000 sec)
> zsize n
  2
  (0.000 sec)
> zscan n a 0 100 -1
  key             score
  -------------------------
    a              : 1
    b              : 2
  2 result(s) (0.000 sec)
> multi_zdel n a b
  2
  (0.000 sec)
> zscan n a 0 100 -1        # <----- OK
  key             score
  -------------------------
  0 result(s) (0.000 sec)
> zsize n                   # <----- NOT OK, should return 0
  1
  (0.000 sec)
> zlist k m -1              # <----- NOT OK, should return no results
               name
  -----------------
                  n
  1 result(s) (0.000 sec)

Need info command to monitor SSDB's status

about C/C++ Client API

您好

看到介绍中说， 'Client API supports including C/C++, PHP, Python, Cpy, Java'. 但并没有找到相关的C/C++ client api的代码。

需要自己实现，对么？

Best wishes,

Core dump after update from 1.4.3 to 1.5.5

core dump in
Core was generated by `./ssdb-server ssdb.conf'.
Program terminated with signal 11, Segmentation fault.
#0 0x000000000044e97d in leveldb::ReadFileToString(leveldb::Env_, std::basic_string<char, std::char_traits, std::allocator > const&, std::basic_string<char, std::char_traits, std::allocator >_) ()

(gdb) bt
#0 0x000000000044e97d in leveldb::ReadFileToString(leveldb::Env_, std::basic_string<char, std::char_traits, std::allocator > const&, std::basic_string<char, std::char_traits, std::allocator >_) ()
#1 0x00000000004431b7 in leveldb::VersionSet::Recover() ()
#2 0x000000000043517c in leveldb::DBImpl::Recover(leveldb::VersionEdit*) ()
#3 0x00000000004358f5 in leveldb::DB::Open(leveldb::Options const&, std::basic_string<char, std::char_traits, std::allocator > const&, leveldb::DB**) ()
#4 0x0000000000404f40 in SSDB::open (conf=..., base_dir="./var_data") at ssdb.cpp:76
#5 0x0000000000481d93 in init (argc=2, argv=0x7fff69389038) at ssdb-server.cpp:339
#6 0x0000000000481ef4 in main (argc=2, argv=0x7fff69389038) at ssdb-server.cpp:36

need luajit_ffi binding and ngx_lua client asap

Related links:
http://luajit.org/ext_ffi_tutorial.html
https://github.com/shuttler/nessDB/tree/v2.0/bindings/luajit_ffi
https://github.com/agentzh/lua-resty-redis
https://github.com/agentzh/lua-resty-memcached

Feature required: PHP client batch commands

Purpose

• To avoid network round trip time affection

Usage

$ret = $ssdb->batch()
    ->set('k', $str)
    ->get('k')->zset('z', 'k', 2)
    ->zget('z', 'k')
    ->exec();
var_dump($ret);

terminate backend workers properly

Ruby client

Not an issue, just wanted to mention that I created a Ruby client: https://github.com/bsm/ssdb-rb

It currently lacks implementation for HASH commands, but I will add them within the next few days. It also comes with a full test suite. Not sure, but you may want to use them for/as your unit tests.

Naming conventions and command names

Currently, there is quite a bit of ambiguity around the word "keys". Keys can be top-level plain value holders, but also fields in hashes or members of zsets.

May I suggest the following changes to the API (to reduce confusion):

keys(key_start, key_end, limit)                       -> list(key_start, key_end, limit)
zkeys(name, key_start, score_start, score_end, limit) -> zmembers(key, member_start, score_start, score_end, limit)
hkeys(name, key_start, key_end, limit)                -> hfields(key, field_start, field_end, limit)

Documentation-wise, this would be the updates:

keys(key_start, key_end, limit) -> list(key_start, key_end, limit)

multi_hget(name, keys) -> multi_hget(key, fields)
multi_zget(name, keys) -> multi_zget(key, members)
multi_hdel(name, keys) -> multi_hdel(key, fields)
multi_zdel(name, keys) -> multi_zdel(key, members)
multi_hset(name, kvs)  -> multi_hset(key, fvs)
multi_zset(name, kvs)  -> multi_zset(key, mvs)

zset(name, key, score)  -> zset(key, member, score)
zadd(name, score, key)  -> zadd(key, score, member)
zget(name, key)         -> zget(key, member)
zexists(name)           -> zexists(key)
zdel(name, key)         -> zdel(key, member)
zincr(name, key, score) -> zincr(key, member, score)
zdecr(name, key, score) -> zdecr(key, member, score)
zsize(name)             -> zsize(key)

zscan(name, key_start, score_start, score_end, limit)  -> zscan(key, member_start, score_start, score_end, limit)
zrscan(name, key_start, score_start, score_end, limit) -> zrscan(key, member_start, score_start, score_end, limit)
zkeys(name, key_start, score_start, score_end, limit)  -> zmembers(key, member_start, score_start, score_end, limit)
zlist(name_start, name_end, limit)                     -> zlist(key_start, key_end, limit)

hset(name, key, val)                     -> hset(key, field, val)
hget(name, key)                          -> hget(key, field)
hexists(name)                            -> hexists(key)
hdel(name, key)                          -> hdel(key, field)
hscan(name, key_start, key_end, limit)   -> hscan(key, field_start, field_end, limit)
hrscan(name, key_start, key_end, limit)  -> hrscan(key, field_start, field_end, limit)
hkeys(name, key_start, key_end, limit)   -> hfields(key, field_start, field_end, limit)
hincr(name, key, val)                    -> hincr(key, field, val)
hdecr(name, key, val)                    -> hdecr(key, field, val)
hsize(name)                              -> hsize(key)
hlist(name_start, name_end, limit)       -> hlist(key_start, key_end, limit)

I understand that this is a pretty big API change, but we are still in "early stages", so I would implement this consistency update rather earlier than later.

Thanks,
dim

仅仅用ssdb做同步/备份，而用其它的操作接口来操作leveldb会不会有问题呢？

仅仅用ssdb做同步/备份，而用其它的操作接口来操作leveldb会不会有问题呢？
比如使用python的leveldb绑定，ssdb仅仅作为主从同步复制的工具。

另外ssdb可以支持redis那种事务么？

How to import/export data?

How to import/export data from ssdb? This feature is necessary for data backups.

Inconsistent zset behaviour

While scores of zsets can be floats, this is not consistently replicated. Example:

zset("myset", "k", 2.5)
zget("myset", "k") 
# => 2.5

zincr("myset", "k", 1) 
# => 3

zset("myset", "k", 2.5)
zscan("myset", "k", 0, 100, -1) 
# => [["k", 2]]

It would make sense to make zset scores consistently ints or floats.