reborndb/reborn

Distributed database fully compatible with redis protocol

#Reborn - yet another fast distributed solution for Redis

Reborn is a proxy based high performance Redis cluster solution written in Go/C, an alternative to Redis.

Reborn supports multiple stateless proxy with multiple redis instances.

Reborn is engineered to elastically scale, Easily add or remove redis or proxy instances on-demand/dynamicly.

Features

• Auto rebalance
• Extremely simple to use
• Support both redis or rocksdb transparently
• Support cold/hot data hybrid management.
• GUI dashboard & admin tools
• Supports most of Redis commands, Fully compatible with twemproxy(https://github.com/twitter/twemproxy)
• Native Redis clients are supported
• Safe and transparent data migration, Easily add or remove nodes on-demand.
• Command-line interface is also provided
• RESTful APIs
• Monitor and Failover

Design

RebornDB: the Next Generation Distributed Key-Value Store

Build and Install

• Install go(we recommend go1.3.3, go1.4 has some performance issue) & ZooKeeper
• Install godep, go get github.com/tools/godep
• go get github.com/reborndb/reborn
• cd reborn
• make
• make gotest
• cd sample
• follow instructions in usage.md

Tutorial

English

简体中文

FAQ

English

简体中文

Performance (Benchmark)

OS:   Ubuntu SMP x86_64 GNU/Linux
CPU:  Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz(8 cores)
Mem:  16G
Disk: 256G SSD

• twemproxy

version 0.4.1

alpha:
  listen: 127.0.0.1:22121
  hash: crc32a
  hash_tag: "{}"
  distribution: ketama
  auto_eject_hosts: false
  timeout: 400
  redis: true
  servers:
   - 127.0.0.1:6381:1
   - 127.0.0.1:6382:1

• qdb-server (https://github.com/reborndb/qdb)

####1 twemproxy + 2 reborn-server
redis-benchmark -p 22121 -c 500 -n 5000000 -P 100 -r 10000 -t get,set -q

SET: 209100.03 requests per second
GET: 212404.41 requests per second

####1 reborn-proxy + 2 reborn-server
redis-benchmark -p 19000 -c 500 -n 5000000 -P 100 -r 10000 -t get,set -q

SET: 410273.22 requests per second
GET: 455913.19 requests per second

####1 twemproxy + 2 qdb-server
redis-benchmark -p 22121 -c 500 -n 5000000 -P 100 -r 10000 -t get,set -q

SET: 133212.55 requests per second
GET: 165584.84 requests per second

####1 reborn-proxy + 2 qdb-server
redis-benchmark -p 19000 -c 500 -n 5000000 -P 100 -r 10000 -t get,set -q

SET: 45909.04 requests per second
GET: 77690.41 requests per second

Result:

For Java users who want to support HA

Reborn-java (HA Reborn Connection Pool based on Jedis)

Architecture

Snapshots

Dashboard

Migrate

Slots

Authors

• @goroutine
• @c4pt0r
• @spinlock9
• @siddontang
• @qiuyesuifeng

Thanks:

• @ivanzhaowy
• @Apache9

License

Reborn is licensed under MIT， see MIT-LICENSE.txt

---

You are welcome to use Reborn in your product, and feel free to let us know~ :)

reborn's Issues

About Performance Benchmark

性能测试数据比较了Twemproxy+Redis还有Reborn-proxy+Reborn-server的性能，在只有一个Proxy的情况下性能就限制于Proxy的性能了，而Twemproxy是单进程的，想问下Reborn-proxy测试的时候go的schedule的线程数目也是设置为1的吗？如果是设置为cpu的core数目的话，那这样测试出来的结果是不是没有那么合理？毕竟没有完全发挥Twemproxy的性能，比如这个时候如果在Twemproxy的机器上再跑一个Twemproxy那性能可能就一致了？

Faster mset and del like mget

use agent to support better ops

agent can start server, monitor it and restart it when down.
agent can do failover to support HA.

Add api for changing log level dynamically

unify juju/errors and reborndb/go/errors

use redis-2.8.21 in reborn

Better ha tool

问一个主备切换问题

若只安装一主一备，若主挂掉,reborn能自动将备转为主吗?

questions about desing and performance for reborndb

hi，你好

我司目前正在对redis集群方案选型， 之前看到了codis，但是比较遗憾的是codis的mget、zset之类的性能较差，今天看到你们新开发了reborndb， 很好奇，有几个疑问请教下

1. 在性能方面，你们有比较全面的测评数据吗？ 直接和redis比较的就好， 这块因为线上业务压力大，因而是主要考虑因素
2. 使用leveldb或者rocksdb，做引擎， 测试过compaction时的读写性能吗？ 如果业务偶发大量写，可能引起大量compaction，reborndb这块的考虑是？（比如自己会限写入速度或者分步compaction吗）？ reborndb和tair的设计理念我感觉有点类似，也是抽象集群管理层，存储引擎可以更换，但是tair的线上业务大部分用的还是mdb和rdb，leveldb更多的应用于线下方案。 因而对于线上业务的实时kv存储（我想reborndb应该不是冲着离线、半离线存储做的吧？），使用lsm树+磁盘的存储模型，还是ssd+hash（rb tree）的存储模型会更好一些？ 这块你们的考虑是什么样的？
3. reborndb的一致性模型是什么样的， 和codis一样的主写从同步？ 还是写完n/2+1副本？还是写完全副本？ 这块儿你们未来的考虑是如何的？ 对cap，你们的取舍是？

之前ngaut哥哥应毛老师之约来过我司，有一面之缘，盼qq联络方式~

猎豹 房晓 2829794533

Add test coverage

Simple password support for proxy

check https://github.com/wandoulabs/codis/issues/297

do we have the same problem?

ZREVRANGE key值长度超过2字符时会crash

# proxy
redis-cli -h 10.8.0.8 -p 19000
10.8.0.8:19000> ZREVRANGE t 0 99
(empty list or set)
10.8.0.8:19000> ZREVRANGE tt 0 99
(empty list or set)
10.8.0.8:19000> ZREVRANGE ttt 0 99
Error: Server closed the connection
(11.28s)
10.8.0.8:19000>

# reborn-server
redis-cli -h 10.8.0.6 -p 9002
10.8.0.6:9002> ZREVRANGE t 0 99
(empty list or set)
10.8.0.6:9002> ZREVRANGE tt 0 99
(empty list or set)
10.8.0.6:9002> ZREVRANGE ttt 0 99
(empty list or set)
10.8.0.6:9002> ZREVRANGE tttt 0 99
(empty list or set)

support failover

We must check master is down and do failover immediately to guarantee high availability.

Using zk to select a master proxy to do this, maybe.

Using goleveldb for test, not rocksdb

Add api for Removing instance

更新代码都不试着编辑一下吗？这种错误不应该啊

github.com/reborndb/reborn/cmd/agent

cmd/agent/proc_proxy.go:30: undefined: erros
cmd/agent/proc_qdb.go:26: undefined: erros
cmd/agent/proc_redis.go:7: imported and not used: "fmt"
godep: go exit status 2

Need restart server tests when we are migrating

reborn 支持原生redis

特别希望reborn能够支持原生redis，siddontang 大神，有木有这个计划啊

reborndb-agent goroutine泄露的问题

当coordinator为Etcd（{"etcdserver":"2.2.3","etcdcluster":"2.2.0"}）时，启动两个reborndb-agent --ha后，startHA时elector.RunTask产生的leader正常，其他的agent则会goroutine泄露，从而吃掉很多CPU和RAM。

11032 @ 0x445d13 0x445dd4 0x41bfaf 0x41bafb 0x6f121d 0x6f0cb2 0x574cdc 0x4768b1
# 0x6f121d    github.com/ngaut/pools.(*ResourcePool).get+0x4ad    /opt/app/src/github.com/reborndb/reborn/Godeps/_workspace/src/github.com/ngaut/pools/resource_pool.go:105
# 0x6f0cb2    github.com/ngaut/pools.(*ResourcePool).Get+0x42     /opt/app/src/github.com/reborndb/reborn/Godeps/_workspace/src/github.com/ngaut/pools/resource_pool.go:84
# 0x574cdc    github.com/ngaut/zkhelper.(*etcdImpl).watch.func1+0x8c  /opt/app/src/github.com/reborndb/reborn/Godeps/_workspace/src/github.com/ngaut/zkhelper/etcd.go:192

问一个agent ha检测的问题。

这里为什么不对offline状态的backend进行检测？
感觉加上重新上线的检测，运维会省很多事啊。

unify ngaut log and reborndb/go/log

Use which one?

Maybe create a new repository forked from ngaut log?
@qiuyesuifeng @ngaut

refactor redis pool connection

now we use garyburd redis for multi operations, use a simple redis connection for pipeline and use a redis pool for migration.

These burn me out, maybe we can simplify and unify redis connection use.

Handle zookeeper "session expired"

Fix panic when handling invalid mset command.

Test case:
mset x y z

Is there a benchmark for auto-rebalance?

Here's my step:

1. deploy all instances in single machine
2. init two group, each group with one master reborn-server
3. add 1024 keys(simple key-value, e.g. key_123: 123), which are allocated into two group
4. add one group with one master reborn-server
5. click Auto Rebalance on dashboard
   The process of rebalance takes 15 minutes, is this a normal time cost ?

tutorial_zh.md build部分少了一个步骤

https://github.com/reborndb/reborn/blob/master/doc/tutorial_zh.md
Build部分

go get github.com/reborndb/reborn
cd $GOPATH/src/github.com/reborndb/reborn
./bootstrap.sh
make gotest
make agent_test (可选, reborn-agent 的集成测试)

./bootstrap.sh下一步少了make

proxy use a unique random id when starts

Now proxy uses pre-defined id like proxy_1, proxy_2 in config file.
Proxy is stateless, so I think using random id may be more convenient.

More storage engine support

咨询一个codis-server挂掉的情况

遇到一个问题，一个proxy下挂了4个codis-server，没有从节点，然其中一个挂掉之后，整个proxy就不工作了，codis是这么设计的吗？
Error: Ser
10.168.10.123:19000> get abc
Error: Server closed the connection
(1.90s)
重新添加进去一个实例，升为master就好了。

backup old logs when agent restarts dead process

@qiuyesuifeng

When you use etcd >= 2.2.0, you should know this issue.

Duplicated issue codis issue#488

Make mget and del faster

Migrate multi-key in a single call

Do more check when add server

Some one may add redis server to group, we should check it.

Add CI support

performance for reborn-proxy

阿里云三台机器
reborn-server-1: 10.8.0.60 8核8G
reborn-server-2: 10.8.0.61 8核16G
reborn-proxy: 10.8.0.240 8核16G

$ uname -a
Linux VM-0-208-ubuntu 3.13.0-62-generic #102-Ubuntu SMP Tue Aug 11 14:29:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

在server 1,2上各运行了2个实例, 互为主备:

./bin/reborn-config server add 1 10.8.0.60:6001 master
./bin/reborn-config server add 1 10.8.0.61:6002 slave

./bin/reborn-config server add 2 10.8.0.61:7001 master
./bin/reborn-config server add 2 10.8.0.60:7002 slave

测试结果如下:

$ redis-benchmark  -c 500 -n 5000000 -P 100 -r 10000 -t get,set -q -p 6001 -h 10.8.0.60
SET: 609904.88 requests per second
GET: 648004.12 requests per second

$ redis-benchmark  -c 500 -n 5000000 -P 100 -r 10000 -t get,set -q -p 7001 -h 10.8.0.61
SET: 541301.25 requests per second
GET: 654450.25 requests per second

$ redis-benchmark  -c 500 -n 5000000 -P 100 -r 10000 -t get,set -q -p 19000 -h 10.8.0.240
SET: 120368.80 requests per second
GET: 120880.98 requests per second

请问为什么性能相差这么大, 如何改进?

OPS is show as 0 in the dashboard

I am working on complete latest git commit dbad7f1, I have one proxy two redis server, reborn-agent, reborn-config, zookeeper. All are on one physical host but on separate VMs. When i run redis-benchmark as follows
redis-benchmark -h 192.168.1.109 -p 19000 -c 500 -n 100000 -P 100 -r 10000 -t get,set -q, I get
SET: 17746.23 requests per second
GET: 17815.79 requests per second

but OPS on dashbaord is not changed also key number is not reflected correctly
Product Name: test
Keys: 10005
Mem Used: 21.82 MB
Performace: 0 OP/s

refactor config for all reborn services

We will use a base config for reborn proxy, config, agent, the base config is named env in the old source but it looks not intuitional, we need a better name.

Btw, later, all services will use this base config and use its own configs by passing flags in command.

agent redis/qdb data should not be stored in proc base data path

We may restart qdb/redis gracefully, at that time, the proc id will be changed, but the new server must use the old data.

maybe the data path like agent_data/qdb_addr may be better, because we will use a fix address for qdb/redis.

@qiuyesuifeng

set_proxy_online.sh run Error

[root@adapter-yz-017158-rtb sample]# sh set_proxy_online.sh
set proxy_1 online
2015/09/01 14:52:53 main.go:159: [fatal] /home/chenhuaying/go/src/github.com/reborndb/reborn/cmd/cconfig/utils.go:64: http status code 500, zk: node does not exist
/home/chenhuaying/go/src/github.com/reborndb/reborn/cmd/cconfig/main.go:87:
done

Try send error before close connection.

Probably there is a bug on zookeeper library.

Two machine. A, B
zookeeper on B
proxy on A

ifconfig network down on B, wait for 1 minutes, then ifconfig network up.
proxy on A did not get notified.

Make migration faster

Use gocheck for test framework

The origin testing is not powerful, especially leaking "assert", writing below code is disgusting:

err := doXXX()
if err != nil {
    t.Fatal(err)
}

I think https://labix.org/gocheck is better.

Add CI support

use a good way for waiting sometime test

Now we will wait some time for asynchronous test, like wait proxy to be online, wait ha doing failover over.

If the wait time is short, the test may not complete and we will get an invalid result, but if the wait time is long, the whole test will cost longer, which is unacceptable too.

add test for reborn-agent

Now I only test reborn-agent manually, using some tests may be better.

Refacter tests by using qdb to replace miniredis

So we can do migrate tests.

AddServer处缺少参数校验

由于命令reborn-config server add <group_id> <redis_addr> 处的对出入并没有进行校验，对应的此处也没有进行校验，会造成AddServer此处以错误的server type执行后续逻辑，并给出正确返回，给人错觉。

PS：之前操作role一直输入的是slaver，也没觉察到任何异常，操作也返回200 OK，但是并没有执行slave的逻辑，发现原来是输入错误了....

validate server validity before setting migrate status for a slot

See CodisLabs/codis#173

@qiuyesuifeng