reborndb/qdb

A fast, high availability, fully Redis compatible store.

qdb's Issues

service test check

If running "go test -v", service package test is much slower.
Sometimes, it may run fail.

use godep to build qdb

refactor command doc

We don't need to compare twemproxy command with qdb-server.

@qiuyesuifeng

support config set/get for some configurations

we may meet race conflict, e.g, we are using a configuration but it is changed in config set at the same time.

But I think it dose not matter, maybe now.

optimization: reuse last bgsave rdb for full sync

Now every full sync will do bgsave, which may be slow for large dataset. If we can reuse the last bgsave rdb, and the associated master sync offset is still in the backlog buffer, we may reuse this rdb for full sync and do incremental sync from last master sync offset.

This is only a optimization, and we may do it later, not now.

pass cmd flags to overwrite configurations in config file

Some configurations need to be set from command flags like Redis, so we can use agent to start different instants.

@qiuyesuifeng

remove reflect, add Register for qdb

1, Remove old reflect for registering commands
2, Supply a Register function like func Register(name string, f func(...)) to register commands and let other people register their self commands.

@qiuyesuifeng, we may define the Register function below.

use gocheck to improve whole test

fix and complete bit* command

安装qdb时遇到问题。帮忙指点一下

++ pwd

• BUILD_PATH=/home/qdb/extern
• INSTALL_PATH=/usr/local
• CXX=gcc
• test -d /usr/local/include
• test -d /usr/local/lib
• gcc -x c++ - -o /dev/null
• '[' 0 -ne 0 ']'
• gcc -x c++ - -o /dev/null -lleveldb
• '[' 0 -ne 0 ']'
• gcc -x c++ - -o /dev/null -lrocksdb
• '[' 1 -ne 0 ']'
• cd /home/qdb/extern/rocksdb
• make -j4 shared_lib
  g++ -shared -Wl,-soname -Wl,librocksdb.so -g -Wall -Werror -Wsign-compare -Wshadow -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DZLIB -DBZIP2 -DHAVE_JEMALLOC -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -DNDEBUG -Woverloaded-virtual -Wnon-virtual-dtor -fPIC db/builder.cc db/c.cc db/column_family.cc db/compaction.cc db/compaction_job.cc db/compaction_picker.cc db/db_filesnapshot.cc db/db_impl.cc db/db_impl_debug.cc db/db_impl_readonly.cc db/db_iter.cc db/dbformat.cc db/file_indexer.cc db/filename.cc db/flush_job.cc db/flush_scheduler.cc db/forward_iterator.cc db/internal_stats.cc db/log_reader.cc db/log_writer.cc db/memtable.cc db/memtable_list.cc db/merge_helper.cc db/merge_operator.cc db/repair.cc db/table_cache.cc db/table_properties_collector.cc db/transaction_log_impl.cc db/version_builder.cc db/version_edit.cc db/version_set.cc db/wal_manager.cc db/write_batch.cc db/write_controller.cc db/write_thread.cc table/adaptive_table_factory.cc table/block.cc table/block_based_filter_block.cc table/block_based_table_builder.cc table/block_based_table_factory.cc table/block_based_table_reader.cc table/block_builder.cc table/block_hash_index.cc table/block_prefix_index.cc table/bloom_block.cc table/cuckoo_table_builder.cc table/cuckoo_table_factory.cc table/cuckoo_table_reader.cc table/flush_block_policy.cc table/format.cc table/full_filter_block.cc table/get_context.cc table/iterator.cc table/merger.cc table/meta_blocks.cc table/plain_table_builder.cc table/plain_table_factory.cc table/plain_table_index.cc table/plain_table_key_coding.cc table/plain_table_reader.cc table/table_properties.cc table/two_level_iterator.cc util/arena.cc util/auto_roll_logger.cc util/blob_store.cc util/bloom.cc util/build_version.cc util/cache.cc util/coding.cc util/comparator.cc util/crc32c.cc util/db_info_dumper.cc util/dynamic_bloom.cc util/env.cc util/env_hdfs.cc util/env_posix.cc util/filter_policy.cc util/hash.cc util/hash_cuckoo_rep.cc util/hash_linklist_rep.cc util/hash_skiplist_rep.cc util/histogram.cc util/iostats_context.cc util/ldb_cmd.cc util/ldb_tool.cc util/log_buffer.cc util/logging.cc util/murmurhash.cc util/mutable_cf_options.cc util/options.cc util/options_builder.cc util/options_helper.cc util/perf_context.cc util/rate_limiter.cc util/skiplistrep.cc util/slice.cc util/sst_dump_tool.cc util/statistics.cc util/status.cc util/string_util.cc util/sync_point.cc util/thread_local.cc util/vectorrep.cc util/xxhash.cc utilities/backupable/backupable_db.cc utilities/compacted_db/compacted_db_impl.cc utilities/document/document_db.cc utilities/document/json_document.cc utilities/geodb/geodb_impl.cc utilities/merge_operators/put.cc utilities/merge_operators/string_append/stringappend.cc utilities/merge_operators/string_append/stringappend2.cc utilities/merge_operators/uint64add.cc utilities/redis/redis_lists.cc utilities/spatialdb/spatial_db.cc utilities/ttl/db_ttl_impl.cc utilities/write_batch_with_index/write_batch_with_index.cc port/stack_trace.cc port/port_posix.cc -lpthread -lrt -lsnappy -lz -lbz2 -o librocksdb.so
  util/options_builder.cc: In function 'void rocksdb::{anonymous}::OptimizeForLevel(int, int, uint64_t, rocksdb::Options_)':
  util/options_builder.cc:164:23: error: comparison between signed and unsigned integer expressions [-Werror=sign-compare]
  cc1plus: all warnings being treated as errors
  make: *_* [librocksdb.so] Error 1
• cp -rf /home/qdb/extern/rocksdb/include/rocksdb /usr/local/include
• cp -f '/home/qdb/extern/rocksdb/librocksdb.' /usr/local/lib
  cp: cannot stat `/home/qdb/extern/rocksdb/librocksdb.': No such file or directory

centos 6.5
gcc版本 4.7.2

setup.sh fails on linux platform

librocksdb.dylib is OSX specific, on linux platform, default to .so

$ git diff
diff --git a/extern/setup.sh b/extern/setup.sh
index 235063a..5ad8401 100755
--- a/extern/setup.sh
+++ b/extern/setup.sh
@@ -13,7 +13,7 @@ cd ./rocksdb; make clean
 make -j4 shared_lib && cd .. || exit 1

 cp -rf rocksdb/include/rocksdb ${BUILD}
-cp -f rocksdb/librocksdb.dylib ${BUILD}
+cp -f rocksdb/librocksdb.* ${BUILD}

 cd ./levigo
 CGO_CFLAGS="-I${BUILD}" CGO_LDFLAGS="-L${BUILD} -lleveldb -lsnappy -lstdc++" go install ./

is it deleteIfExists necessary in set command

I see that set* commands use deleteIfExists, but is it necessary?

We should check the source exactly and add more tests.

@qiuyesuifeng

remove replication backlog buffer if not used for a long time

If no slaves for a long time > repl_backlog_ttl, we may remove the backlog buffer.

Because every write operations will be written into the backlog buffer, if no replication, this is may be waste.

service supports sync and psync

@ngaut @c4pt0r

Now service doesn't support psync and sync.

slaveof only supports sync.

server close gracefully

Now server does not close all connections when closing, this is ok for an independent process but not ok for embedding qdb.

We need a graceful close.

@qiuyesuifeng

we may support double for zset score

ql stores double in a magic way in BTree, I think it can support double byte lexicographical comparison.

Maybe we can have a try.

@qiuyesuifeng

re-benchmark and update the doc.

integrate redis tests

@qiuyesuifeng

If we can pass Redis tests(not including lost feature like AOF, zipmap, sentinel, etc.), I think we can guarantee that qdb is robust and stable.

Redis tests uses TCL which is very easy and we may add our slots tests too.

But at first, I think we should refactor whole test framework using gocheck instead. I don't like old test, e.g, for service, olds tests uses a fake session but I prefer real tcp connection like in production environment.

how about lz4 instead of snappy ?

http://code.google.com/p/lz4/

support parallel for read operations

@ngaut @c4pt0r

Now all operations including read is serialization, this does not use the advantage of rocksdb.

If the data is expired in read, we will not delete it immediately, but put it in the expired list then delete it in another thread. So that we cau use read mutex for read and write mutex for write.

unify setex, psetex, setnx

if we implement #73 , we can let setex, psetex and setnx use set command inner.

complete set command

http://redis.io/commands/set

--create 已经不能用了吗？

如题

怎么让程序后台运行呢？

另外
如有QQ联系方式或者Q群，能方便告知一下吗？

register command should pass flags like ReadCommnad/WriteCommand

For write command, it is not allowed when readonly (slave mode)

rename store and binlog pkg name

store -> engine
binlog -> qdb

BinLog -> Store

@ngaut @c4pt0r

当key不存在时zset的部分接口会导致进程panic

问题涉及到的zset接口：ZCount, ZLexCount, ZRange, ZRevRange, ZRangeByLex, ZRevRangeByLex, ZRangeByScore, ZRevRangeByScore.

以ZCount为例：

o, err := s.loadZSetRow(db, key)
if err != nil {
    return 0, errors.Trace(err)
}

var count int64 = 0
f := func(o *zsetRow) error {
    count++
    return nil
}

if err = o.travelInRange(s, r, f); err != nil {
    return 0, errors.Trace(err)
}

s.loadZSetRow方法在zset集合key不存在的情况下，可能会err != nil 并且 o == nil，而代码只判断了err是否为nil，没有判断o为nil的情况。后面调用o的方法的时候就panic了。

if err != nil || o == nil {
    return 0, errors.Trace(err)
}

改成上面这样可以解决问题，其他的接口也同样修改，麻烦看下:)

support readonly for slave, refactor expired key when reading/writing

For redis, slave should be readonly and doesn't remove expired key, but qdb slave now doesn't support this.

• add a readonly flag, that every write operation will be denied.
• don't remove expired key when reading/writing.

@qiuyesuifeng

求助：如何让gorocks支持lz4

我阅读了qdb的代码后发现，qdb强制使用 Lz4 作为rocksdb的压缩策略。但我实际测试中总是遇到“Invalid argument: Compression type LZ4 is not linked with the binary.” 错误。

在各种文档中包括rocksdb官方文档中，也未描述，如何在编译中加入lz4的支持，已向rocksdb提issue尚未得到解答。且与gorocks相关，可能问题会有所不同，所以只能来此求助，还望大神解答。

optimizes full replication

Now generating rdb and then full sync is slow for huge data, may cost a long time. So we may think a better way to do this.

Maybe we can use our own file format, only saving the raw backend leveldb/rocksdb data, like below:

• generates a snapshot
• scans the snapshot from first to last, gets all k-v and saves to a file
• transfers this file to slave
• slave reads the file and reset

add build tags for leveldb and rocksdb

@ngaut

Building leveldb and rocksdb is not a easy thing, and many users just want to try qdb at first, so #1 may be a good choice.

If users want to use qdb in their real production environment, they can use make build_leveldb, make build_rocksdb or make build_all to use leveldb or rocksdb engine.

qdb主从出现不能同步的问题

master log：

2015/11/11 17:30:43 conn.go:126: [warning] handle commands failed, conn = <local> 10.121.85.35:6380 -- 10.121.116.235:51646 <remote>, request = 'KjMNCiQ1DQpQU1lOQw0KJDQwDQplOTQzZWY3MTRjMzVkMTg1ZDQ5NWMzYzg3MDkyY2ZkY2E5ZDFiZGU3DQokMQ0KMg0K', err = wait others do bgsave timeout

slave log:

2015/11/11 17:29:06 sync.go:276: [warning] slave <local> 10.121.116.235:51612 -- 10.121.85.35:6380 <remote> do psync err - read tcp 10.121.85.35:6380: i/o timeout

怀疑跟master一直在dump rdb导致slave连接不上，请问是否有问题？

refactor zset to support range* support

@c4pt0r @ngaut

Zset now only supports little commands which are not enough for real production using.

config sync replication mode

• Asynchronous replication
• Semi-sync replication
• Synchronous replication

use goleveldb for test

@qiuyesuifeng

support agent for better ops

agent can start server, monitor it and restart it when down.
agent can do failover to support HA.

rewrite README.md

support password

use goleveldb as the default storage engine

Make qdb embed and extent friendly

I understand qdb was tailored for reborndb, but it would be nice if qdb more friendly to embed and more extendable.

For example:

func Serve(config *Config, bl *store.Store) error {
    h, err := newHandler(config, bl)
    if err != nil {
        return errors.Trace(err)
    }

    defer h.close()

    err = h.run()
    return errors.Trace(err)
}

May be change to a Server struct, and serve from Server instance.

type Server struct {
    *Handler
}

func NewServer() *Server {}
func (srv *server) Serve() error {}

Or make newHandler a public method.

For the extend part, I think https://github.com/bsm/redeo was a good example, https://github.com/alicebob/miniredis using it implemented a redis for unittests.

redeo example of echo:

  srv := redeo.NewServer(&redeo.Config{Addr: "localhost:9736"})
  srv.HandleFunc("ping", func(out *redeo.Responder, _ *redeo.Request) error {
    out.WriteInlineString("PONG")
    return nil
  })

It let user define handler in a idiomatic way, I really wish qdb could expose a fully compatible redis and also let user extend(define) more commands(Handle Func).

What's diff between qdb and ledisdb?

hi 各位， 我又来了

几个问题啊：

1. qdb和ledisdb的差异？
2. codis proxy和reborn db的proxy的差异? 我看reborndb的proxy 是codis proxy的一个早期版本？
3. 今天偶尔看了下siddongtang's 《我推广ledisDB的惨痛里程》，不晓得你们现在是否还有这样的问题， 有些废话也想说下， 其实用户只通过git和开发者进行联系，这种联系方式太弱， 支持不够给力也不够快，具体来说， 有三个问题， 第一， 我这边是想用reborndb到线上环境，但是呢， 我要自己读代码要自己维护（因为你们的支持跟不上嘛）， 那能有多少人有自己维护的能力？ 第二， 对于功能的开发排期、db的特性取舍、未来发展的方向， 我也希望能够进行讨论， 至少你们如果能看到大量潜在客户的实际需求，再去做事情，会更有方向性么。 第三点， 你们潜在客户的一部分是具备开发能力的， 如果能够引导、合作去做事情， 会比你们自己去做好很多， 建议参照下linux内核的开源开发模式。

我这边， 我们会尝试引入reborndb， 在此基础上进行迭代开发，我只是希望， 第一，知道你们的近期排期（这样我们某些功能可以等你们去开发而不用自己开发，你看这样的需求都没法满足），第二， 有些功能的设计能够进行探讨(比如主从复制功能，复制redis的方式真的合适吗？）

from 猎豹某同学，哈哈（抱歉，之前一个版本是英文的， 我英文不太好，刚才写出来的issue我都不忍直视，所以又改成中文重新写了一遍）