reborndb / reborn Goto Github PK
View Code? Open in Web Editor NEWDistributed database fully compatible with redis protocol
License: MIT License
Distributed database fully compatible with redis protocol
License: MIT License
# proxy
redis-cli -h 10.8.0.8 -p 19000
10.8.0.8:19000> ZREVRANGE t 0 99
(empty list or set)
10.8.0.8:19000> ZREVRANGE tt 0 99
(empty list or set)
10.8.0.8:19000> ZREVRANGE ttt 0 99
Error: Server closed the connection
(11.28s)
10.8.0.8:19000>
# reborn-server
redis-cli -h 10.8.0.6 -p 9002
10.8.0.6:9002> ZREVRANGE t 0 99
(empty list or set)
10.8.0.6:9002> ZREVRANGE tt 0 99
(empty list or set)
10.8.0.6:9002> ZREVRANGE ttt 0 99
(empty list or set)
10.8.0.6:9002> ZREVRANGE tttt 0 99
(empty list or set)
Migrate multi-key in a single call
We may restart qdb/redis gracefully, at that time, the proc id will be changed, but the new server must use the old data.
maybe the data path like agent_data/qdb_addr may be better, because we will use a fix address for qdb/redis.
do we have the same problem?
The origin testing is not powerful, especially leaking "assert", writing below code is disgusting:
err := doXXX()
if err != nil {
t.Fatal(err)
}
I think https://labix.org/gocheck is better.
阿里云三台机器
reborn-server-1: 10.8.0.60 8核8G
reborn-server-2: 10.8.0.61 8核16G
reborn-proxy: 10.8.0.240 8核16G
$ uname -a
Linux VM-0-208-ubuntu 3.13.0-62-generic #102-Ubuntu SMP Tue Aug 11 14:29:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
在server 1,2上各运行了2个实例, 互为主备:
./bin/reborn-config server add 1 10.8.0.60:6001 master
./bin/reborn-config server add 1 10.8.0.61:6002 slave
./bin/reborn-config server add 2 10.8.0.61:7001 master
./bin/reborn-config server add 2 10.8.0.60:7002 slave
测试结果如下:
$ redis-benchmark -c 500 -n 5000000 -P 100 -r 10000 -t get,set -q -p 6001 -h 10.8.0.60
SET: 609904.88 requests per second
GET: 648004.12 requests per second
$ redis-benchmark -c 500 -n 5000000 -P 100 -r 10000 -t get,set -q -p 7001 -h 10.8.0.61
SET: 541301.25 requests per second
GET: 654450.25 requests per second
$ redis-benchmark -c 500 -n 5000000 -P 100 -r 10000 -t get,set -q -p 19000 -h 10.8.0.240
SET: 120368.80 requests per second
GET: 120880.98 requests per second
请问为什么性能相差这么大, 如何改进?
now we use garyburd redis for multi operations, use a simple redis connection for pipeline and use a redis pool for migration.
These burn me out, maybe we can simplify and unify redis connection use.
[root@adapter-yz-017158-rtb sample]# sh set_proxy_online.sh
set proxy_1 online
2015/09/01 14:52:53 main.go:159: [fatal] /home/chenhuaying/go/src/github.com/reborndb/reborn/cmd/cconfig/utils.go:64: http status code 500, zk: node does not exist
/home/chenhuaying/go/src/github.com/reborndb/reborn/cmd/cconfig/main.go:87:
done
Use which one?
Maybe create a new repository forked from ngaut log?
@qiuyesuifeng @ngaut
I am working on complete latest git commit dbad7f1, I have one proxy two redis server, reborn-agent, reborn-config, zookeeper. All are on one physical host but on separate VMs. When i run redis-benchmark as follows
redis-benchmark -h 192.168.1.109 -p 19000 -c 500 -n 100000 -P 100 -r 10000 -t get,set -q, I get
SET: 17746.23 requests per second
GET: 17815.79 requests per second
but OPS on dashbaord is not changed also key number is not reflected correctly
Product Name: test
Keys: 10005
Mem Used: 21.82 MB
Performace: 0 OP/s
cmd/agent/proc_proxy.go:30: undefined: erros
cmd/agent/proc_qdb.go:26: undefined: erros
cmd/agent/proc_redis.go:7: imported and not used: "fmt"
godep: go exit status 2
这里为什么不对offline状态的backend进行检测?
感觉加上重新上线的检测,运维会省很多事啊。
当coordinator为Etcd({"etcdserver":"2.2.3","etcdcluster":"2.2.0"})时,启动两个reborndb-agent --ha后,startHA时elector.RunTask产生的leader正常,其他的agent则会goroutine泄露,从而吃掉很多CPU和RAM。
11032 @ 0x445d13 0x445dd4 0x41bfaf 0x41bafb 0x6f121d 0x6f0cb2 0x574cdc 0x4768b1
# 0x6f121d github.com/ngaut/pools.(*ResourcePool).get+0x4ad /opt/app/src/github.com/reborndb/reborn/Godeps/_workspace/src/github.com/ngaut/pools/resource_pool.go:105
# 0x6f0cb2 github.com/ngaut/pools.(*ResourcePool).Get+0x42 /opt/app/src/github.com/reborndb/reborn/Godeps/_workspace/src/github.com/ngaut/pools/resource_pool.go:84
# 0x574cdc github.com/ngaut/zkhelper.(*etcdImpl).watch.func1+0x8c /opt/app/src/github.com/reborndb/reborn/Godeps/_workspace/src/github.com/ngaut/zkhelper/etcd.go:192
性能测试数据比较了Twemproxy+Redis还有Reborn-proxy+Reborn-server的性能,在只有一个Proxy的情况下性能就限制于Proxy的性能了,而Twemproxy是单进程的,想问下Reborn-proxy测试的时候go的schedule的线程数目也是设置为1的吗?如果是设置为cpu的core数目的话,那这样测试出来的结果是不是没有那么合理?毕竟没有完全发挥Twemproxy的性能,比如这个时候如果在Twemproxy的机器上再跑一个Twemproxy那性能可能就一致了?
hi,你好
我司目前正在对redis集群方案选型, 之前看到了codis,但是比较遗憾的是codis的mget、zset之类的性能较差,今天看到你们新开发了reborndb, 很好奇,有几个疑问请教下
之前ngaut哥哥应毛老师之约来过我司,有一面之缘,盼qq联络方式~
猎豹 房晓 2829794533
https://github.com/reborndb/reborn/blob/master/doc/tutorial_zh.md
Build部分
go get github.com/reborndb/reborn
cd $GOPATH/src/github.com/reborndb/reborn
./bootstrap.sh
make gotest
make agent_test (可选, reborn-agent 的集成测试)
./bootstrap.sh下一步少了make
遇到一个问题,一个proxy下挂了4个codis-server,没有从节点,然其中一个挂掉之后,整个proxy就不工作了,codis是这么设计的吗?
Error: Ser
10.168.10.123:19000> get abc
Error: Server closed the connection
(1.90s)
重新添加进去一个实例,升为master就好了。
Test case:
mset x y z
Now I only test reborn-agent manually, using some tests may be better.
Now we will wait some time for asynchronous test, like wait proxy to be online, wait ha doing failover over.
If the wait time is short, the test may not complete and we will get an invalid result, but if the wait time is long, the whole test will cost longer, which is unacceptable too.
Now proxy uses pre-defined id like proxy_1, proxy_2 in config file.
Proxy is stateless, so I think using random id may be more convenient.
Some one may add redis server to group, we should check it.
Duplicated issue codis issue#488
特别希望reborn能够支持原生redis,siddontang 大神,有木有这个计划啊
若只安装一主一备,若主挂掉,reborn能自动将备转为主吗?
We must check master is down and do failover immediately to guarantee high availability.
Using zk to select a master proxy to do this, maybe.
agent can start server, monitor it and restart it when down.
agent can do failover to support HA.
So we can do migrate tests.
Here's my step:
Two machine. A, B
zookeeper on B
proxy on A
ifconfig network down on B, wait for 1 minutes, then ifconfig network up.
proxy on A did not get notified.
We will use a base config for reborn proxy, config, agent, the base config is named env in the old source but it looks not intuitional, we need a better name.
Btw, later, all services will use this base config and use its own configs by passing flags in command.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.