snapchat / keydb Goto Github PK
View Code? Open in Web Editor NEWA Multithreaded Fork of Redis
Home Page: https://keydb.dev
License: BSD 3-Clause "New" or "Revised" License
A Multithreaded Fork of Redis
Home Page: https://keydb.dev
License: BSD 3-Clause "New" or "Revised" License
Lots of build errors on specific platforms. This tracks CI on Debian, Ubuntu, and Fedora.
Does KeyDB require Redis Sentinel? While I can see how they exist here for being compatible with Redis failover, do I need Sentinel when setting up active-active replication?
That is, after the master fails, I can continue writing to the replica with just switching the IP address. Do I need to do anything more?
just use make
make[3]: Leaving directory '/home/atn4517/KeyDB-0.9.2/deps/jemalloc'
make[2]: Leaving directory '/home/atn4517/KeyDB-0.9.2/deps'
CC adlist.o
CC quicklist.o
CC ae.o
CC anet.o
CC dict.o
CC server.o
In file included from server.cpp:31:0:
server.h:2331:68: error: declaration of ‘void* calloc(size_t, size_t)’ has a different exception specifier
void *calloc(size_t count, size_t size) __attribute__ ((deprecated));
^
In file included from server.h:42:0,
from server.cpp:31:
/usr/include/stdlib.h:468:14: error: from previous declaration ‘void* calloc(size_t, size_t) throw ()’
extern void *calloc (size_t __nmemb, size_t __size)
^
In file included from server.cpp:31:0:
server.h:2332:49: error: declaration of ‘void free(void*)’ has a different exception specifier
void free(void *ptr) __attribute__ ((deprecated));
^
In file included from server.h:42:0,
from server.cpp:31:
/usr/include/stdlib.h:483:13: error: from previous declaration ‘void free(void*) throw ()’
extern void free (void *__ptr) __THROW;
^
In file included from server.cpp:31:0:
server.h:2333:54: error: declaration of ‘void* malloc(size_t)’ has a different exception specifier
void *malloc(size_t size) __attribute__ ((deprecated));
^
In file included from server.h:42:0,
from server.cpp:31:
/usr/include/stdlib.h:466:14: error: from previous declaration ‘void* malloc(size_t) throw ()’
extern void *malloc (size_t __size) __THROW __attribute_malloc__ __wur;
^
In file included from server.cpp:31:0:
server.h:2334:66: error: declaration of ‘void* realloc(void*, size_t)’ has a different exception specifier
void *realloc(void *ptr, size_t size) __attribute__ ((deprecated));
^
In file included from server.h:42:0,
from server.cpp:31:
/usr/include/stdlib.h:480:14: error: from previous declaration ‘void* realloc(void*, size_t) throw ()’
extern void *realloc (void *__ptr, size_t __size)
^
Makefile:278: recipe for target 'server.o' failed
make[1]: *** [server.o] Error 1
make[1]: Leaving directory '/home/user/KeyDB-0.9.2/src'
Makefile:6: recipe for target 'all' failed
make: *** [all] Error 2
Today docker is vastly used so I really recommend updating project README with link to the docker image at https://hub.docker.com/r/eqalpha/keydb
A significant reason is great is that scripts and MULTI-EXEC commands are isolated from other clients making it easy to reason about transactions. This comes from the single threaded nature of redis. I didn't find a section in your README addressing the concern that KeyDB would allow 2 transactions to run concurrently.
I am trying to add a replica slave from host2:6380 to my master on host1:6379 and added
masterauth into the appropriate keydb-slave.conf.
The master is running with an identical requirepass definition.
However as the slave tries to connect to master it fails with:
Unexpected reply to PSYNC from master: -NOAUTH Authentication required.
Would someone know what I am doing wrong? Thx
Ubuntu 18.04 LTS - Kernel v5.1.9
Providing only the relevant part:
[34/52 done]: unit/type/list-3 (48 seconds)
Testing unit/wait
[err]: FLUSHDB ASYNC can reclaim memory in background in tests/unit/lazyfree.tcl
Expected condition '$peak_mem > $orig_mem+1000000' to be true (13838096 > 13734512+1000000)
I would like to point out that identifiers like “__AE_H__
” and “__RDB_H
” do not fit to the expected naming convention of the C++ language standard.
Would you like to adjust your selection for unique names?
I am using Ubuntu 18.04 LTS and running make tests throws the followings:
!!! WARNING The following tests failed:
*** [err]: Server is able to generate a stack trace on selected systems in tests/integration/logging.tcl
expected stack trace not found into log file
*** [err]: PSYNC2: cluster is consistent after load (x = 52143) in tests/integration/psync2.tcl
Instance #1 x variable is inconsistent
Should I be concerned?
As I understand it, in mainline redis the difference between KEYS
and ARGV
is that Redis will make sure that the KEYS
values exist on the current slot in a clustered environment and will otherwise fail with CROSSSLOT
if one of them does not.
This limits the ability to write scripts that perform dynamic key composition (where a key is built up in lua-land on the instance) in a clustered environment, making the two features mutually exclusive - which is unfortunate.
However, with KeyDB the clustering/replication properties have changed a bit. Is the KEYS
/dynamic key generation limitation still applicable in KeyDB? Or can I safely use dynamic keys in a clustered environment?
I understand that KeyDB provides both KEYS
and ARGV
for compatibility but I'm interested in whether or not dynamic key referencing in a clustered environment will work as intended or if it still follows the mainline Redis limitations.
Thanks John! :)
EDIT: I just read through most of the HN thread and realize now that I wouldn't even want a cluster mode with KeyDB (thanks!). I'm still interested in the answer, purely for academic purposes :)
We saw a KeyDB server built from the RELEASE_5
branch become completely unresponsive yesterday. Our systems responded by promoting a replica which crashed immediately when receiving SLAVEOF NO ONE
Unfortunately I don't have many details about the master to share, other than noting that the overall system was showing nominal behavior and load, with log output stopping around 12:19 UTC. The replica logs below start at about that time when the master connection is dropped, and our system attempts a promotion at 12:24 UTC.
Jul 14 12:19:55 keydb-server[2905]: 2905:S 14 Jul 2019 12:19:55.834 # MASTER timeout: no data nor PING received...
Jul 14 12:19:55 keydb-server[2905]: 2905:S 14 Jul 2019 12:19:55.834 # Connection with master lost.
Jul 14 12:19:55 keydb-server[2905]: 2905:S 14 Jul 2019 12:19:55.834 * Caching the disconnected master state.
Jul 14 12:19:55 keydb-server[2905]: 2905:S 14 Jul 2019 12:19:55.834 * Connecting to MASTER xxxxx
Jul 14 12:19:55 keydb-server[2905]: 2905:S 14 Jul 2019 12:19:55.836 * MASTER <-> REPLICA sync started
Jul 14 12:22:05 keydb-server[2905]: 2905:S 14 Jul 2019 12:22:05.349 # Error condition on socket for SYNC: Connection timed out
Jul 14 12:22:06 keydb-server[2905]: 2905:S 14 Jul 2019 12:22:06.073 * Connecting to MASTER xxxxx
Jul 14 12:22:06 keydb-server[2905]: 2905:S 14 Jul 2019 12:22:06.077 * MASTER <-> REPLICA sync started
Jul 14 12:24:16 keydb-server[2905]: 2905:S 14 Jul 2019 12:24:16.421 # Error condition on socket for SYNC: Connection timed out
Jul 14 12:24:17 keydb-server[2905]: 2905:S 14 Jul 2019 12:24:17.316 * Connecting to MASTER xxxxx
Jul 14 12:24:17 keydb-server[2905]: 2905:S 14 Jul 2019 12:24:17.320 * MASTER <-> REPLICA sync started
Jul 14 12:24:48 keydb-server[2905]: 2905:S 14 Jul 2019 12:24:48.290 # Setting secondary replication ID to 9b8fd931f161c8be497aa79b23823c188394905d, valid up to offset: 4659745187. New replication ID is 78b24b65287543c6c19c9e3ba964996d2881558c
Jul 14 12:24:48 keydb-server[2905]: 2905:S 14 Jul 2019 12:24:48.290 * Discarding previously cached master state.
Jul 14 12:24:48 keydb-server[2905]: AE_ASSER FAILURE
Jul 14 12:24:48 keydb-server[2905]: === KEYDB BUG REPORT START: Cut & paste starting from here ===
Jul 14 12:24:48 keydb-server[2905]: 2905:S 14 Jul 2019 12:24:48.290 # KeyDB 5.0.0 crashed by signal: 11
Jul 14 12:24:48 keydb-server[2905]: 2905:S 14 Jul 2019 12:24:48.290 # Crashed running the instruction at: 0x432eee
Jul 14 12:24:48 keydb-server[2905]: 2905:S 14 Jul 2019 12:24:48.290 # Accessing address: (nil)
Jul 14 12:24:48 keydb-server[2905]: 2905:S 14 Jul 2019 12:24:48.290 # Failed assertion: <no assertion failed> (<no file>:0)
Jul 14 12:24:48 keydb-server[2905]: ------ STACK TRACE ------
Jul 14 12:24:48 keydb-server[2905]: EIP:
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(aeDeleteFileEvent+0x3e) [0x432eee]
Jul 14 12:24:48 keydb-server[2905]: Backtrace:
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(logStackTrace(ucontext_t*)+0x39) [0x4851a9]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(sigsegvHandler(int, siginfo_t*, void*)+0xa4) [0x485824]
Jul 14 12:24:48 keydb-server[2905]: /lib64/libpthread.so.0(+0x121f0) [0x7f964976c1f0]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(aeDeleteFileEvent+0x3e) [0x432eee]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(undoConnectWithMaster(redisMaster*)+0x26) [0x457596]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(cancelReplicationHandshake(redisMaster*)+0x25) [0x457635]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(replicationUnsetMaster(redisMaster*)+0x97) [0x457ef7]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(replicaofCommand(client*)+0x17c) [0x45886c]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(call(client*, int)+0xbd) [0x43a6dd]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(execCommand(client*)+0xc7) [0x481ff7]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(call(client*, int)+0xbd) [0x43a6dd]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(processCommand(client*, int)+0x789) [0x43ebc9]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(processCommandAndResetClient(client*, int)+0x21) [0x447391]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(processInputBuffer(client*, int)+0x7a) [0x44bbba]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(readQueryFromClient(aeEventLoop*, int, void*, int)+0x338) [0x44c138]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(ProcessEventCore+0xe9) [0x433409]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(aeProcessEvents+0x14d) [0x43369d]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(aeMain+0x2d) [0x433a0d]
Jul 14 12:24:48 keydb-server[2905]: /usr/local/bin/keydb-server *:11042(workerThreadMain(void*)+0x90) [0x437aa0]
Jul 14 12:24:48 keydb-server[2905]: /lib64/libpthread.so.0(+0x754b) [0x7f964976154b]
Jul 14 12:24:48 keydb-server[2905]: /lib64/libc.so.6(clone+0x3f) [0x7f96494992ff]
Client details for the replica:
Jul 14 12:24:48 keydb-server[2905]: ------ CLIENT LIST OUTPUT ------
Jul 14 12:24:48 keydb-server[2905]: id=13 addr=172.16.101.10:44516 fd=98 name=redisgreen-monitoring age=197244 idle=24 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=slowlog
Jul 14 12:24:48 keydb-server[2905]: id=16 addr=172.16.101.11:57666 fd=96 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
Jul 14 12:24:48 keydb-server[2905]: id=17 addr=172.16.101.11:57668 fd=97 name=redisgreen age=0 idle=0 flags=x db=0 sub=0 psub=0 multi=2 qbuf=123 qbuf-free=32645 obl=27 oll=0 omem=0 events=r cmd=exec
Jul 14 12:24:48 keydb-server[2905]: ------ CURRENT CLIENT INFO ------
Jul 14 12:24:48 keydb-server[2905]: id=17 addr=172.16.101.11:57668 fd=97 name=redisgreen age=0 idle=0 flags=x db=0 sub=0 psub=0 multi=2 qbuf=123 qbuf-free=32645 obl=27 oll=0 omem=0 events=r cmd=exec
Jul 14 12:24:48 keydb-server[2905]: argv[0]: 'slaveof'
Jul 14 12:24:48 keydb-server[2905]: argv[1]: 'no'
Jul 14 12:24:48 keydb-server[2905]: argv[2]: 'one'
Hello,
We are trying out the evaluation for multi master setup where keydb would be installed in 2 different zones having multi master mode enabled.
I went through this link provided in the wiki https://github.com/JohnSully/KeyDB/wiki/Multimaster-Support but i was unable to figure out the complete configurations required.
Only parameter which i found is multi-master yes in the configuration, i am not sure how would i configure other master to which the sync would happen.
Could you please help me on this.
Thanks.
Shiva
And if it is, perhaps a note mentioning that in the readme would serve beneficial.
The root cause is the fake user we create to execute commands in RREPLAY doesn't have the authentication information of the client executing the RREPLAY command. This may cause commands to fail due to permission issues.
Hi there, in case it's useful here's a crash from a KeyDB instance built off the RELEASE_5
branch last week, using 6 threads, virtual network interface has 8 queues. I can provide more details and/or sanitize the INFO/CLIENT payloads if it'd be helpful.
Jul 14 03:20:04 keydb-server[2881]: === KEYDB BUG REPORT START: Cut & paste starting from here ===
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # === ASSERTION FAILED CLIENT CONTEXT ===
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # client->flags = 2097152
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # client->fd = 425
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # client->argc = 2
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # client->argv[0] = ">" (refcount: 2)
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # client->argv[1] = "queue:keydb2" (refcount: 1)
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # === ASSERTION FAILED ===
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # ==> networking.cpp:2633 'c->cmd != NULL' is not true
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # (forcing SIGSEGV to print the bug report.)
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # KeyDB 5.0.0 crashed by signal: 11
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # Crashed running the instruction at: 0x482738
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # Accessing address: 0xffffffffffffffff
Jul 14 03:20:04 keydb-server[2881]: 2881:M 14 Jul 2019 03:20:04.799 # Failed assertion: c->cmd != NULL (networking.cpp:2633)
Jul 14 03:20:04 keydb-server[2881]: ------ STACK TRACE ------
Jul 14 03:20:04 keydb-server[2881]: EIP:
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(_serverAssert+0x78) [0x482738]
Jul 14 03:20:04 keydb-server[2881]: Backtrace:
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(logStackTrace(ucontext_t*)+0x39) [0x4851a9]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(sigsegvHandler(int, siginfo_t*, void*)+0xa4) [0x485824]
Jul 14 03:20:04 keydb-server[2881]: /lib64/libpthread.so.0(+0x121f0) [0x7fc293da61f0]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(_serverAssert+0x78) [0x482738]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(rewriteClientCommandVector(client*, int, ...)+0x120) [0x447df0]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(blockingPopGenericCommand(client*, int)+0x1db) [0x4666bb]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(call(client*, int)+0xbd) [0x43a6dd]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(processCommand(client*, int)+0x789) [0x43ebc9]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(processCommandAndResetClient(client*, int)+0x21) [0x447391]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(processInputBuffer(client*, int)+0x7a) [0x44bbba]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(readQueryFromClient(aeEventLoop*, int, void*, int)+0x338) [0x44c138]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(ProcessEventCore+0xe9) [0x433409]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(aeProcessEvents+0x14d) [0x43369d]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(aeMain+0x2d) [0x433a0d]
Jul 14 03:20:04 keydb-server[2881]: /usr/local/bin/keydb-server *:11042(workerThreadMain(void*)+0x90) [0x437aa0]
Jul 14 03:20:04 keydb-server[2881]: /lib64/libpthread.so.0(+0x754b) [0x7fc293d9b54b]
Jul 14 03:20:04 keydb-server[2881]: /lib64/libc.so.6(clone+0x3f) [0x7fc293ad32ff]
The 0.9.6
tag dropped the 'v' prefix (and is thus harder to notice in the list) than v0.9.5
, v0.9.4
, etc. Looks accidental?
Hi,
I'm trying to use redisearch with keydb. Afaik, keydb supports modules. I can create indices but I get timeout when adding document. You can reproduce this issue with these codes:
root@keydb-0:/data# keydb-cli
127.0.0.1:6379> FT.CREATE idx SCHEMA id NUMERIC SORTABLE name TEXT SORTABLE
OK
127.0.0.1:6379> FT.ADD idx 1 1.0 FIELDS id 1 name 'NameTest'
I suggest to reuse a higher level build system than your current make script so that powerful checks for software features will become easier.
Hello,
I need few clarifications regarding the active-active replication and the performance impact.
For example I have 2 datacenters, in each datacenter for each shard I'm planning to add 2 nodes in active- active replication.
Each shard in DC1 will have an associated shard in DC2 as well with the same number of nodes.
So in the end each redis node from a shard will have 3 replicas (1 in the same dc and 2 in the other dc).
In case the replication is sync the performance impact will be high so this is my biggest concern in this moment comparing with
the https://github.com/Netflix/dynomite where the replication is async.
Let's suppose one node goes down. While is down the clients will send the read/writes to the other replicas.
After some time the node is fixed and started back. The node will accept connections from clients only after it's in sync with the other replicas ?
Because otherwise the clients can start connecting to it and send requests to keys that were not already propagated from replicas
Kind regards,
Silviu
When you run HGETALL against a key that doesnt exist, KeyDB returns a NIL response. Redis returns (empty list or set). This breaks compatibility with some libraries (StackExchange.Redis)
SETUP
docker run -it -p 6391:6379 eqalpha/keydb:latest
docker run -it -p 6390:6379 redis:latest
Test
will@45K9VT2:/mnt/c/Users/wdavis$ redis-cli -p 6390 HGETALL test
(empty list or set)
will@45K9VT2:/mnt/c/Users/wdavis$ redis-cli -p 6391 HGETALL test
(nil)
will@45K9VT2:/mnt/c/Users/wdavis$
REDIS warning..
..........................................
When the number of keys in redis (use DBSIZE to know the count) is less, this method can be used but when redis size grows, using keys cmd is extremely dangerous for performance and highly discouraged in production environment since redis is single threaded, keys cmd will block all other operations on redis and all the clients will have to wait for its completion.
..........................................
Is KeyDB subject to the same constraints or not ?
Thank you for your work on KeyDB!
This problem is reproducible on OSX and Linux - configure with databases 17
(or any number higher than 16) and keydb-server
will segfault on startup with (trace from Amazon Linux 2):
=== KEYDB BUG REPORT START: Cut & paste starting from here ===
2841:M 12 Jul 2019 05:51:37.727 # KeyDB 5.0.0 crashed by signal: 11
2841:M 12 Jul 2019 05:51:37.727 # Crashed running the instruction at: 0x43b081
2841:M 12 Jul 2019 05:51:37.727 # Accessing address: 0x48
2841:M 12 Jul 2019 05:51:37.727 # Failed assertion: <no assertion failed> (<no file>:0)
------ STACK TRACE ------
EIP:
./keydb-server *:6379(serverCron(aeEventLoop*, long long, void*)+0x1a1) [0x43b081]
Backtrace:
./keydb-server *:6379(logStackTrace(ucontext_t*)+0x39) [0x4851a9]
./keydb-server *:6379(sigsegvHandler(int, siginfo_t*, void*)+0xa4) [0x485824]
/lib64/libpthread.so.0(+0x121f0) [0x7fe1f24981f0]
./keydb-server *:6379(serverCron(aeEventLoop*, long long, void*)+0x1a1) [0x43b081]
./keydb-server *:6379(aeProcessEvents+0x223) [0x433773]
./keydb-server *:6379(aeMain+0x2d) [0x433a0d]
./keydb-server *:6379(workerThreadMain(void*)+0x90) [0x437aa0]
/lib64/libpthread.so.0(+0x754b) [0x7fe1f248d54b]
/lib64/libc.so.6(clone+0x3f) [0x7fe1f21c52ff]
------ INFO OUTPUT ------
Segmentation fault
refers to redis/redis#6125
can file a PR here too if desired.
ENV:
CentOS 6.5 x86_64 with devtoolset-3-gcc (4.9.2)
I use
make CFLAGS="-march=haswell -m128bit-long-double -m64 " MALLOC=jemalloc
I get this info:
server.cpp: In function 'char* genRedisInfoString(const char*)':
server.cpp:4453:29: error: expected ')' before 'PRIu64'
"long_lock_waits:%" PRIu64 "\r\n",
^
server.cpp:4459:36: warning: spurious trailing '%' in format [-Wformat=]
fastlock_getlongwaitcount());
^
server.cpp:4459:36: warning: too many arguments for format [-Wformat-extra-args]
make[1]: *** [server.o] Error 1
make[1]: Leaving directory `/opt/opstoolset/source/redis/src/KeyDB-0.9.4-r2/src'
How to fix this ? very very thanks :-)
Hi,
One of the much requested features, to expire individual members of a set, is unlikely to get implemented in Redis, hoping KeyDb can take to improve it
I created successfully a rpm with MALLOC=jemalloc
and jemalloc-devel
system libraries, on CentOS 7.6 and GCC 5.3.1. Everything fails when I try to use either deps/memkind
, or memkind-devel-1.7.0
system libraries.
%global modules_abi 1
%global source_name KeyDB
Name: keydb
Version: 0.9.4
Release: 1%{?dist}
Summary: Multithreaded fork of Redis
License: BSD
URL: https://github.com/JohnSully/KeyDB
Source0: %{url}/archive/v%{version}.tar.gz#/%{source_name}-%{version}.tar.gz
Source1: %{name}.logrotate
Source2: %{name}.service
Source3: %{name}-sentinel.service
Source4: %{name}.limit
Source5: %{name}.tmpfiles
Source6: %{name}.shutdown
Patch: 0002-install-redis-check-rdb-as-a-symlink-instead-of-dupl.patch
ExclusiveArch: x86_64
BuildRequires: gcc >= 5.3.0
BuildRequires: jemalloc-devel
BuildRequires: libuuid-devel
BuildRequires: memkind-devel
BuildRequires: nasm
BuildRequires: redis-rpm-macros
BuildRequires: systemd
BuildRequires: tcl-devel
Requires: gawk
Requires: logrotate
Provides: bundled(hiredis)
Provides: bundled(linenoise)
Provides: bundled(lua-libs)
Provides: bundled(lzf)
Provides: %{name}(modules_abi)%{?_isa} = %{modules_abi}
Conflicts: redis
%{?%systemd_requires}
%prep
%autosetup -n %{source_name}-%{version} -p1
%{__rm} -rf deps/jemalloc
%{__rm} -rf deps/memkind
%{__sed} -i -e '/cd jemalloc && /d' deps/Makefile
%{__sed} -i -e '/cd memkind && /d' deps/Makefile
%{__sed} -i -e 's|../deps/jemalloc/lib/libjemalloc.a|-ljemalloc -ldl|g' src/Makefile
%{__sed} -i -e 's|-I../deps/jemalloc.*|-DJEMALLOC_NO_DEMANGLE -I%{_includedir}/jemalloc|g' src/Makefile
%{__sed} -i -e 's|../deps/memkind/src/.libs/libmemkind.a|-lmemkind|g' src/Makefile
%{__sed} -i -e 's|-I../deps/memkind.*|-I%{_includedir}/memkind|g' src/Makefile
%{__sed} -i -e 's|redis|%{name}|g' redis.conf
%{__sed} -i -e 's|%{name}\.io|redis.io|g' redis.conf
%{__sed} -i -e 's|^logfile .*$|logfile %{_localstatedir}/log/%{name}/%{name}.log|g' redis.conf
%{__sed} -i -e '$ alogfile %{_localstatedir}/log/%{name}/sentinel.log' sentinel.conf
%{__sed} -i -e 's|^dir .*$|dir %{_sharedstatedir}/%{name}|g' redis.conf
api=$(sed -n -e 's/#define REDISMODULE_APIVER_[0-9][0-9]* //p' src/redismodule.h)
if [ "$api" != "%{modules_abi}" ]; then
: Error: Upstream API version is now ${api}, expecting %%{modules_abi}.
: Update the modules_abi macro, the rpmmacros file, and rebuild.
exit 1
fi
%build
%make_build CFLAGS+="%{optflags} -fPIC" LDFLAGS="%{__global_ldflags}" \
MALLOC=memkind PREFIX=%{buildroot}%{_prefix} all
CC storage.o
storage.cpp: In function 'int forkFile()':
storage.cpp:124:35: error: 'memkind_tmpfile' was not declared in this scope
memkind_tmpfile(PMEM_DIR, &fdT);
^
storage.cpp:125:46: error: 'memkind_fd' was not declared in this scope
if (ioctl(fdT, FICLONE, memkind_fd(mkdisk)) == -1)
^
storage.cpp: In function 'void storage_init(const char*, size_t)':
storage.cpp:177:44: error: 'memkind_fd' was not declared in this scope
posix_fallocate64(memkind_fd(mkdisk), 0, cbFileReserve);
^
storage.cpp: In function 'size_t salloc_usable_size(void*)':
storage.cpp:214:60: error: 'memkind_malloc_usable_size' was not declared in this scope
return memkind_malloc_usable_size(kindFromPtr(ptr), ptr);
^
storage.cpp: In function 'void handle_postfork_child()':
storage.cpp:269:39: error: 'memkind_fd' was not declared in this scope
int fdOriginal = memkind_fd(mkdisk);
^
storage.cpp:270:39: error: 'memkind_pmem_remapfd' was not declared in this scope
memkind_pmem_remapfd(mkdisk, fdNew);
^
make[1]: *** [storage.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory `/builddir/build/BUILD/KeyDB-0.9.4/src'
make: *** [all] Error 2
I'm only interested to build using system libraries. I tried building with the /deps but it failed also.
The README.md says,
KeyDB currently relies on SO_REUSEADDR's load balancing behavior which is available only in Linux
I think it should be SO_REUSEPORT here.
simple redis (redis:alpine)
redis-cli
127.0.0.1:6379> ZADD myzset 1 "one"
(integer) 1
127.0.0.1:6379> ZADD myzset 2 "two"
(integer) 1
127.0.0.1:6379> ZADD myzset 3 "three"
(integer) 1
127.0.0.1:6379> ZRANGEBYSCORE myzset -inf +inf
1) "one"
2) "two"
3) "three"
127.0.0.1:6379> ZRANGEBYSCORE myzset (1 (2
(empty list or set)
simple keydb (eqalpha/keydb:x86_64_v0.9.5)
keydb-cli -a password
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> ZADD myzset 1 "one"
(integer) 1
127.0.0.1:6379> ZADD myzset 2 "two"
(integer) 1
127.0.0.1:6379> ZADD myzset 3 "three"
(integer) 1
127.0.0.1:6379> ZRANGEBYSCORE myzset -inf +inf
1) "one"
2) "two"
3) "three"
127.0.0.1:6379> ZRANGEBYSCORE myzset (1 (2
(nil)
I hope I am not wasting your time, because I intend not to.
This is the results of the Redis tests on my Ubuntu 16.04 LTS:
!!! WARNING The following tests failed:
*** [err]: Command Filter REDISMODULE_CMDFILTER_NOSELF works as expected in tests/modules/commandfilter.tcl
Expected '' to be equal to '{set mykey @log}'
*** [err]: SLAVE can reload "lua" AUX RDB fields of duplicated scripts in tests/integration/replication-3.tcl
Replication not started.
If this is the expected result, please ignore this and close the thread.
The following script repeatedly sets a key to different values. Memory grows quickly and excessively as the script is run (over many GB). If the script is able to run to completion the memory is freed.
Script:
for i = 1, 10000000000, 1 do
redis.call("SET", "ZzZ_MYKEYZzZ"..i.."key", i)
end
Repro:
./keydb-cli --eval /tmp/bench.lua
This behavior is particularly bad when FLASH backing is used as it leads to write amplification.
I'm trying to understand the logic behind current versioning you use (e.g. 0.9.4 to 0.9.4-r2, instead of 0.9.5). I'm an Linux rpm packager and this type of versioning creates issues on the long term.
Regards,
Floren
Upon configuring KeyDB to use a non-default port the following error occurs:
$ keydb-cli
Could not connect to Redis at 127.0.0.1:6379: Connection refused
not connected>
In this instance I am using 6366
but it tries to connect to 6379
as shown above.
I've set the port to 6366
when running install_sever.sh
and the config file contains the following:
port 6366
KeyDB is running in the background just fine and my website happily uses it through port 6366
.
OS: Ubuntu 16.04
This runs the createMaster function before databases are allocated and has been broken more than once. Current tests only create replicas via command after boot has completed and so do not validate this scenario.
The test infra needs to be modified slightly to support this.
The latest example of this is issue #58
Hi, do you need gui for KeyDB?
Arm currently compiles, but crashes with multiple threads. Our fastlock implementation assumes X86's very strong memory model. This needs to be updated for ARM.
I'm running a very basic Docker setup with 2 KeyDB instances, both are configured via their respective config before start and have no previous data.
I've also attempted enabling active replication and multi-master via config and then using replicaof commands on the running instances resulting in the same crash.
https://gist.github.com/PurpleIsEverything/7fb1c90654ad3e07f0b537ec9e4f4331
Thanks.
I'm using active-active replication in docker environments
docker -v
Docker version 18.09.6, build 481bc77156
uname -a (CENTOS 7)
Linux ocean-02 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 21:24:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
...........
what(): Operation not permitted
Aborted
terminate called after throwing an instance of 'std::system_error'
what(): Operation not permitted
Aborted
terminate called after throwing an instance of 'std::system_error'
what(): Operation not permitted
Aborted
terminate called after throwing an instance of 'std::system_error'
what(): Operation not permitted
Aborted
terminate called after throwing an instance of 'std::system_error'
what(): Operation not permitted
Aborted
...........
Some users are reporting that these deprecated prototypes fail to compile because the exception specifier doesn't match. The hack removes the deprecated flags for C++ builds (C does not have exception specifiers).
These should still flag an error in C++ so this issue tracks reenabled the deprecated flag for C++.
Hack checked in here: a9e525c
We are running a two master setup in active-active mode, both instances crashed a few hours ago after running fine for about a week and now I am trying to get them back in sync. I started one of the master's and then proceeded to start the second master after the AOF file from the first master was loaded (I did remove the AOF file from the second master in order to get a clean state). After several attempts with the second master to sync with the first master I am now getting this error: (which loops indefinitely)
1:S 14 Jul 2019 19:52:41.104 * MASTER <-> REPLICA sync started
1:S 14 Jul 2019 19:52:41.104 * Non blocking connect for SYNC fired the event.
1:S 14 Jul 2019 19:52:41.105 * Master replied to PING, replication can continue...
1:S 14 Jul 2019 19:52:41.106 * Partial resynchronization not possible (no cached master)
1:S 14 Jul 2019 19:52:41.106 * Master does not support PSYNC or is in error state (reply: -ERR BGSAVE failed, replication can't continue)
1:S 14 Jul 2019 19:52:41.106 * Retrying with SYNC...
1:S 14 Jul 2019 19:52:41.106 # I/O error reading bulk count from MASTER: Resource temporarily unavailable
1:S 14 Jul 2019 19:52:41.662 * Replica XX.XX.XX.XXX:6380 asks for synchronization
1:S 14 Jul 2019 19:52:41.662 * Full resync requested by replica XX.XX.XX.XXX:6380
1:S 14 Jul 2019 19:52:41.662 * Starting BGSAVE for SYNC with target: disk
1:S 14 Jul 2019 19:52:41.662 # BGSAVE for replication: replication information not available, can't generate the RDB file right now. Try later.
1:S 14 Jul 2019 19:52:41.662 # BGSAVE for replication failed
...
BGSAVE for replication: replication information not available, can't generate the RDB file right now. Try later.
What is the preferred approach to recovering from a crash in an active-active replication setup?
This issue tracks the following two issues:
Reconnection loops
Command loops
Its caused by the feedback loop enabled by Active Replication. These issues are resolved for single masters but reappear with multiple masters and more complex topologies.
keydb_v0.9.5, i am getting error with centos 7
cc: error: unrecognized command line option ‘-std=c++14’
make[1]: *** [ae.o] Error 1
make[1]: Leaving directory `/home/hadn/dynomite/KeyDB-0.9.5/src'
make: *** [all] Error 2
[hadn@vsftpd src]$ uname -a
Linux vsftpd 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
This is just an informational bug in the output of the command.
Would you like to add more error handling for return values from functions like the following?
I can't find documentation about use KeyDB as a cluster
There is a "new" approach to very fast replication called CURP (Consistent Unordered Replication Protocol).
It shall not have the caveats the current replication scheme in Redis has and shall not be difficult to implement (actually it was already partially implemented for Redis to test it - don't know though where the source lies and what is the license).
Would you consider implementing such low hanging fruit?
I've made a similar ticket directly for Redis, but as Redis is under the hood of a company with specific opinions, I'm opening it here as well as I'd like Redis-like DBs thrive.
Hello!
First I'd like to thank you for all your hard work. I've noticed that via the INFO
command, KeyDB is reporting a very low redis_version
. This unfortunately breaks every piece of software which tests against this value.
Is it possible that instead of yielding the version of KeyDB in that field, it'd be better for compatibility purposes, to yield a version of Redis that the build is compatible with and maintain version information for KeyDB in a separate key like keydb_version
?
This ticket is to track the feature request.
x.y.z.d:6379> wait 1 1
(error) ERR WAIT cannot be used with replica instances. Please also note that since Redis 4.0 if a replica is configured to be writable (which is not the default) writes to replicas are just local and are not propagated.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.