reborndb / qdb Goto Github PK
View Code? Open in Web Editor NEWA fast, high availability, fully Redis compatible store.
A fast, high availability, fully Redis compatible store.
If running "go test -v", service package test is much slower.
Sometimes, it may run fail.
Now generating rdb and then full sync is slow for huge data, may cost a long time. So we may think a better way to do this.
Maybe we can use our own file format, only saving the raw backend leveldb/rocksdb data, like below:
Now all operations including read is serialization, this does not use the advantage of rocksdb.
If the data is expired in read, we will not delete it immediately, but put it in the expired list then delete it in another thread. So that we cau use read mutex for read and write mutex for write.
我阅读了qdb的代码后发现,qdb强制使用 Lz4 作为rocksdb的压缩策略。但我实际测试中总是遇到“Invalid argument: Compression type LZ4 is not linked with the binary.” 错误。
在各种文档中包括rocksdb官方文档中,也未描述,如何在编译中加入lz4的支持,已向rocksdb提issue尚未得到解答。且与gorocks相关,可能问题会有所不同,所以只能来此求助,还望大神解答。
ql stores double in a magic way in BTree, I think it can support double byte lexicographical comparison.
Maybe we can have a try.
For write command, it is not allowed when readonly (slave mode)
如题
怎么让程序后台运行呢?
另外
如有QQ联系方式或者Q群,能方便告知一下吗?
问题涉及到的zset接口:ZCount, ZLexCount, ZRange, ZRevRange, ZRangeByLex, ZRevRangeByLex, ZRangeByScore, ZRevRangeByScore.
以ZCount为例:
o, err := s.loadZSetRow(db, key)
if err != nil {
return 0, errors.Trace(err)
}
var count int64 = 0
f := func(o *zsetRow) error {
count++
return nil
}
if err = o.travelInRange(s, r, f); err != nil {
return 0, errors.Trace(err)
}
s.loadZSetRow方法在zset集合key不存在的情况下,可能会err != nil 并且 o == nil,而代码只判断了err是否为nil,没有判断o为nil的情况。后面调用o的方法的时候就panic了。
if err != nil || o == nil {
return 0, errors.Trace(err)
}
改成上面这样可以解决问题,其他的接口也同样修改,麻烦看下:)
if we implement #73 , we can let setex, psetex and setnx use set command inner.
I see that set* commands use deleteIfExists, but is it necessary?
We should check the source exactly and add more tests.
For redis, slave should be readonly and doesn't remove expired key, but qdb slave now doesn't support this.
agent can start server, monitor it and restart it when down.
agent can do failover to support HA.
Now every full sync will do bgsave, which may be slow for large dataset. If we can reuse the last bgsave rdb, and the associated master sync offset is still in the backlog buffer, we may reuse this rdb for full sync and do incremental sync from last master sync offset.
This is only a optimization, and we may do it later, not now.
librocksdb.dylib is OSX specific, on linux platform, default to .so
$ git diff
diff --git a/extern/setup.sh b/extern/setup.sh
index 235063a..5ad8401 100755
--- a/extern/setup.sh
+++ b/extern/setup.sh
@@ -13,7 +13,7 @@ cd ./rocksdb; make clean
make -j4 shared_lib && cd .. || exit 1
cp -rf rocksdb/include/rocksdb ${BUILD}
-cp -f rocksdb/librocksdb.dylib ${BUILD}
+cp -f rocksdb/librocksdb.* ${BUILD}
cd ./levigo
CGO_CFLAGS="-I${BUILD}" CGO_LDFLAGS="-L${BUILD} -lleveldb -lsnappy -lstdc++" go install ./
hi 各位, 我又来了
几个问题啊:
我这边, 我们会尝试引入reborndb, 在此基础上进行迭代开发,我只是希望, 第一,知道你们的近期排期(这样我们某些功能可以等你们去开发而不用自己开发,你看这样的需求都没法满足),第二, 有些功能的设计能够进行探讨(比如主从复制功能,复制redis的方式真的合适吗?)
from 猎豹某同学,哈哈(抱歉,之前一个版本是英文的, 我英文不太好,刚才写出来的issue我都不忍直视,所以又改成中文重新写了一遍)
Now server does not close all connections when closing, this is ok for an independent process but not ok for embedding qdb.
We need a graceful close.
we may meet race conflict, e.g, we are using a configuration but it is changed in config set
at the same time.
But I think it dose not matter, maybe now.
1, Remove old reflect for registering commands
2, Supply a Register function like func Register(name string, f func(...))
to register commands and let other people register their self commands.
@qiuyesuifeng, we may define the Register function below.
If we can pass Redis tests(not including lost feature like AOF, zipmap, sentinel, etc.), I think we can guarantee that qdb is robust and stable.
Redis tests uses TCL which is very easy and we may add our slots tests too.
But at first, I think we should refactor whole test framework using gocheck instead. I don't like old test, e.g, for service, olds tests uses a fake session but I prefer real tcp connection like in production environment.
Some configurations need to be set from command flags like Redis, so we can use agent to start different instants.
We don't need to compare twemproxy command with qdb-server.
If no slaves for a long time > repl_backlog_ttl
, we may remove the backlog buffer.
Because every write operations will be written into the backlog buffer, if no replication, this is may be waste.
I understand qdb was tailored for reborndb, but it would be nice if qdb more friendly to embed and more extendable.
For example:
func Serve(config *Config, bl *store.Store) error {
h, err := newHandler(config, bl)
if err != nil {
return errors.Trace(err)
}
defer h.close()
err = h.run()
return errors.Trace(err)
}
May be change to a Server struct, and serve from Server instance.
type Server struct {
*Handler
}
func NewServer() *Server {}
func (srv *server) Serve() error {}
Or make newHandler a public method.
For the extend part, I think https://github.com/bsm/redeo was a good example, https://github.com/alicebob/miniredis using it implemented a redis for unittests.
redeo example of echo:
srv := redeo.NewServer(&redeo.Config{Addr: "localhost:9736"})
srv.HandleFunc("ping", func(out *redeo.Responder, _ *redeo.Request) error {
out.WriteInlineString("PONG")
return nil
})
It let user define handler in a idiomatic way, I really wish qdb could expose a fully compatible redis and also let user extend(define) more commands(Handle Func).
++ pwd
centos 6.5
gcc版本 4.7.2
2015/11/11 17:30:43 conn.go:126: [warning] handle commands failed, conn = <local> 10.121.85.35:6380 -- 10.121.116.235:51646 <remote>, request = 'KjMNCiQ1DQpQU1lOQw0KJDQwDQplOTQzZWY3MTRjMzVkMTg1ZDQ5NWMzYzg3MDkyY2ZkY2E5ZDFiZGU3DQokMQ0KMg0K', err = wait others do bgsave timeout
2015/11/11 17:29:06 sync.go:276: [warning] slave <local> 10.121.116.235:51612 -- 10.121.85.35:6380 <remote> do psync err - read tcp 10.121.85.35:6380: i/o timeout
怀疑跟master一直在dump rdb导致slave连接不上,请问是否有问题?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.