yipit / dredis Goto Github PK
View Code? Open in Web Editor NEWDisk-based Redis implementation
License: MIT License
Disk-based Redis implementation
License: MIT License
This is because of their binary representation.
https://github.com/siddontang/ledisdb is the only project I've found that addresses this properly. They use a prefix indicating if the score is positive or negative, and store scores as uint64
(negative scores overflow and start from 0). This way you have -3
coming before -2
because of the size overflow, but still need the prefix to not mix them with the positive numbers.
DRedis uses double precision and can't use uint64, but I think it's doable to use the same logic of maximum value and overflow.
This is dredis 1.1.0:
127.0.0.1:6377> zadd z -2 first -3 second 0 fourth 1 fifth 2 sixth
(integer) 5
127.0.0.1:6377> zrange z 0 -1 withscores
1) "fourth"
2) "0"
3) "fifth"
4) "1"
5) "sixth"
6) "2"
7) "first"
8) "-2"
9) "second"
10) "-3"
This is Redis 3.2.6:
127.0.0.1:6379> zadd z -2 first -3 second 0 fourth 1 fifth 2 sixth
127.0.0.1:6379> zrange z 0 -1 withscores
1) "second"
2) "-3"
3) "first"
4) "-2"
5) "fourth"
6) "0"
7) "fifth"
8) "1"
9) "sixth"
10) "2"
We use zscan_iter
in Pipeapp to iteratively delete keys.
When I tried to remove a small key (<100 items) from a DRedis instance, it hung for a while. When I connected to the instance and ran the code line by line, I found that running redis-py's zscan_iter
took a surprisingly long time.
The line in Pipeapp (sheqel) looks something like:
items = redis_connection.zscan_iter(key, count=50000)
And when I reproduced it, I cut the count
down a lot (we default to 50K) and it took a long time but ended up returning thousands of results (even though I expected <100).
zscan
didn't seem to have the same problem, though I didn't test it thoroughly yet.
Here's the zscan_iter
code in redis-py (it's just using ZSCAN
and keeping track of the cursor): https://redis-py.readthedocs.io/en/latest/_modules/redis/client.html#Redis.zscan_iter
I ran a few experiments with Cython and saw a significant performance increase by mostly annotating a few files with static types.
I thought the asyncore code was the problem, but glued the Redis ae.c/anet.c libraries with Python but didn't see the performance improvement I was expecting. There's room for improvement with the way I am creating the parsers and buffers, but I think Cython would be a lot simpler. The C code can be found at https://gist.github.com/hltbra/1f2266b96c50700cc17a10115634e55f
It would be nice to be able to load Redis RDB files into dredis.
Related to #7. This PR from @rafaelcaricio is outdated but we can learn from it.
Related to #9
Some tools like Redis Desktop Manager (and I guess KeyLord) rely on the INFO command in order to connect to Redis.
RDM goes crazy when the INFO command doesn't return what it expects.
There's no check at the moment of what type is the key before trying write operations such as SET/HSET/ZADD/SADD.
$ redis-cli
127.0.0.1:6379> set foo bar
OK
127.0.0.1:6379> type foo
"string"
127.0.0.1:6379> sadd foo 1
(error) ERR "Traceback (most recent call last):\n File \"/Users/hugo/projects/dredis/dredis/server.py\", line 34, in execute_cmd\n result = run_command(keyspace, cmd, args)\n File \"dredis/commands.py\", line 384, in run_command\n return REDIS_COMMANDS[cmd.upper()](keyspace, *str_args)\n File \"dredis/commands.py\", line 34, in newfn\n return fn(keyspace, *args, **kwargs)\n File \"dredis/commands.py\", line 170, in cmd_sadd\n count += keyspace.sadd(key, value)\n File \"dredis/keyspace.py\", line 90, in sadd\n value_path.write(value)\n File \"dredis/path.py\", line 29, in write\n with open(self._path, 'w') as f:\nIOError: [Errno 2] No such file or directory: 'dredis-data/0/foo/values/c4ca4238a0b923820dcc509a6f75849b'\n"
https://redis.io/commands/slowlog
It'd be nice to know what commands are slow. We can implement it in-memory like Redis too.
With docker it can be easier to get dredis up and running without relying on system-wide packages (dredis will require leveldb after #12 is merged).
What license this project is using? With a quick glimpse I could not figure out.
I don't have any plans to implement native TLS to dredis, but I think it's a good idea to suggest stunnel
in front of dredis if you want to in-transit encryption.
At the end of chapter 7 (Security Techniques) of Redis Essentials, there's a short stunnel + Redis HOWTO.
More details can be found at http://bencane.com/2014/02/18/sending-redis-traffic-through-an-ssl-tunnel-with-stunnel/
These will be different than the Redis ones because dredis has different parameters, but the goal is to be able to alter parameters in run-time (e.g., password, log level).
CONFIG GET
and CONFIG SET
should be symmetric. CONFIG GET
should support glob patterns.
Can I install Dredis on Windows?
This should be similar to #43
It would be awesome to be able to expire a key, as well as fetch the TTL
I think we should enable SO_REUSEPORT to avoid issues when restarting the TCP server.
Related to #25
Redis's clientObject
struct has the field authenticated
indicating if the client is authenticated or not.
Before each command is processed, Redis checks if the user is authenticated. Relevant snippets:
/* Check if the user is authenticated */
if (!(DefaultUser->flags & USER_FLAG_NOPASS) &&
!c->authenticated &&
(c->cmd->proc != authCommand || c->cmd->proc == helloCommand))
{
flagTransaction(c);
addReply(c,shared.noautherr);
return C_OK;
}
/* AUTH <passowrd>
* AUTH <username> <password> (Redis >= 6.0 form)
*
* When the user is omitted it means that we are trying to authenticate
* against the default user. */
void authCommand(client *c) {
/* Only two or three argument forms are allowed. */
if (c->argc > 3) {
addReply(c,shared.syntaxerr);
return;
}
/* Handle the two different forms here. The form with two arguments
* will just use "default" as username. */
robj *username, *password;
if (c->argc == 2) {
/* Mimic the old behavior of giving an error for the two commands
* from if no password is configured. */
if (DefaultUser->flags & USER_FLAG_NOPASS) {
addReplyError(c,"AUTH <password> called without any password "
"configured for the default user. Are you sure "
"your configuration is correct?");
return;
}
username = createStringObject("default",7);
password = c->argv[1];
} else {
username = c->argv[1];
password = c->argv[2];
}
if (ACLCheckUserCredentials(username,password) == C_OK) {
c->authenticated = 1;
c->user = ACLGetUserByName(username->ptr,sdslen(username->ptr));
addReply(c,shared.ok);
} else {
addReplyError(c,"-WRONGPASS invalid username-password pair");
}
/* Free the "default" string object we created for the two
* arguments form. */
if (c->argc == 2) decrRefCount(username);
}
shared.noautherr = createObject(OBJ_STRING,sdsnew(
"-NOAUTH Authentication required.\r\n"));
Authentication was much simpler in the 2.8 branch:
void authCommand(redisClient *c) {
if (!server.requirepass) {
addReplyError(c,"Client sent AUTH, but no password is set");
} else if (!time_independent_strcmp(c->argv[1]->ptr, server.requirepass)) {
c->authenticated = 1;
addReply(c,shared.ok);
} else {
c->authenticated = 0;
addReplyError(c,"invalid password");
}
}
This will enforce a code-style and ensure uniformity in the codebase: https://github.com/python/black
It's annoying to add coverage to this project because the tests and the server are separate processes. It's probably doable with some digging and it'd be nice to know what parts aren't well-covered.
Related to #8
AOF should be easier than RDB support.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.