yipit/dredis

Disk-based Redis implementation

dredis's Issues

Add support to load AOF

AOF should be easier than RDB support.

Create Dockerfile

With docker it can be easier to get dredis up and running without relying on system-wide packages (dredis will require leveldb after #12 is merged).

Convert code to Cython

I ran a few experiments with Cython and saw a significant performance increase by mostly annotating a few files with static types.

I thought the asyncore code was the problem, but glued the Redis ae.c/anet.c libraries with Python but didn't see the performance improvement I was expecting. There's room for improvement with the way I am creating the parsers and buffers, but I think Cython would be a lot simpler. The C code can be found at https://gist.github.com/hltbra/1f2266b96c50700cc17a10115634e55f

Add CONFIG GET/SET commands

These will be different than the Redis ones because dredis has different parameters, but the goal is to be able to alter parameters in run-time (e.g., password, log level).

CONFIG GET and CONFIG SET should be symmetric. CONFIG GET should support glob patterns.

Add ZSCAN support

Think about SO_REUSEPORT

I think we should enable SO_REUSEPORT to avoid issues when restarting the TCP server.

Add support to load RDB files

It would be nice to be able to load Redis RDB files into dredis.

Add slowlog support

https://redis.io/commands/slowlog

It'd be nice to know what commands are slow. We can implement it in-memory like Redis too.

Redis-py's zscan_iter has unexpected behavior

We use zscan_iter in Pipeapp to iteratively delete keys.

When I tried to remove a small key (<100 items) from a DRedis instance, it hung for a while. When I connected to the instance and ran the code line by line, I found that running redis-py's zscan_iter took a surprisingly long time.

The line in Pipeapp (sheqel) looks something like:

items = redis_connection.zscan_iter(key, count=50000)

And when I reproduced it, I cut the count down a lot (we default to 50K) and it took a long time but ended up returning thousands of results (even though I expected <100).

zscan didn't seem to have the same problem, though I didn't test it thoroughly yet.

Here's the zscan_iter code in redis-py (it's just using ZSCAN and keeping track of the cursor): https://redis-py.readthedocs.io/en/latest/_modules/redis/client.html#Redis.zscan_iter

Implement INFO command

Some tools like Redis Desktop Manager (and I guess KeyLord) rely on the INFO command in order to connect to Redis.

RDM goes crazy when the INFO command doesn't return what it expects.

Add TLS suggestions to documentation

I don't have any plans to implement native TLS to dredis, but I think it's a good idea to suggest stunnel in front of dredis if you want to in-transit encryption.

At the end of chapter 7 (Security Techniques) of Redis Essentials, there's a short stunnel + Redis HOWTO.
More details can be found at http://bencane.com/2014/02/18/sending-redis-traffic-through-an-ssl-tunnel-with-stunnel/

Windows Support

Can I install Dredis on Windows?

Add RENAME command

Error when trying to change type of key

There's no check at the moment of what type is the key before trying write operations such as SET/HSET/ZADD/SADD.

$ redis-cli
127.0.0.1:6379> set foo bar
OK
127.0.0.1:6379> type foo
"string"
127.0.0.1:6379> sadd foo 1
(error) ERR "Traceback (most recent call last):\n  File \"/Users/hugo/projects/dredis/dredis/server.py\", line 34, in execute_cmd\n    result = run_command(keyspace, cmd, args)\n  File \"dredis/commands.py\", line 384, in run_command\n    return REDIS_COMMANDS[cmd.upper()](keyspace, *str_args)\n  File \"dredis/commands.py\", line 34, in newfn\n    return fn(keyspace, *args, **kwargs)\n  File \"dredis/commands.py\", line 170, in cmd_sadd\n    count += keyspace.sadd(key, value)\n  File \"dredis/keyspace.py\", line 90, in sadd\n    value_path.write(value)\n  File \"dredis/path.py\", line 29, in write\n    with open(self._path, 'w') as f:\nIOError: [Errno 2] No such file or directory: 'dredis-data/0/foo/values/c4ca4238a0b923820dcc509a6f75849b'\n"

Add codecov

It's annoying to add coverage to this project because the tests and the server are separate processes. It's probably doable with some digging and it'd be nice to know what parts aren't well-covered.

Port to Python 3

Related to #7. This PR from @rafaelcaricio is outdated but we can learn from it.

Add NX parameter support to the ZADD command

Add support to output RDB

Related to #8

Negative scores are stored out of order (after positive scores)

This is because of their binary representation.

https://github.com/siddontang/ledisdb is the only project I've found that addresses this properly. They use a prefix indicating if the score is positive or negative, and store scores as uint64 (negative scores overflow and start from 0). This way you have -3 coming before -2 because of the size overflow, but still need the prefix to not mix them with the positive numbers.

DRedis uses double precision and can't use uint64, but I think it's doable to use the same logic of maximum value and overflow.

This is dredis 1.1.0:

127.0.0.1:6377> zadd z -2 first -3 second 0 fourth 1 fifth 2 sixth
(integer) 5
127.0.0.1:6377> zrange z 0 -1 withscores
 1) "fourth"
 2) "0"
 3) "fifth"
 4) "1"
 5) "sixth"
 6) "2"
 7) "first"
 8) "-2"
 9) "second"
10) "-3"

This is Redis 3.2.6:

127.0.0.1:6379> zadd z -2 first -3 second 0 fourth 1 fifth 2 sixth
127.0.0.1:6379> zrange z 0 -1 withscores
 1) "second"
 2) "-3"
 3) "first"
 4) "-2"
 5) "fourth"
 6) "0"
 7) "fifth"
 8) "1"
 9) "sixth"
10) "2"

Implement RESTORE

Related to #25

Implement DUMP

Related to #8 and #11

We should start with the Redis 3.2.6 serialization format.

Add support to output AOF

Related to #9

License?

What license this project is using? With a quick glimpse I could not figure out.

Add support to HSCAN

This should be similar to #43

Run `black` as pre-commit hook

This will enforce a code-style and ensure uniformity in the codebase: https://github.com/python/black

Expire/TTL support

It would be awesome to be able to expire a key, as well as fetch the TTL

https://redis.io/commands/expire

https://redis.io/commands/ttl

Implement authentication (AUTH)

Redis's clientObject struct has the field authenticated indicating if the client is authenticated or not.

Before each command is processed, Redis checks if the user is authenticated. Relevant snippets:

    /* Check if the user is authenticated */
    if (!(DefaultUser->flags & USER_FLAG_NOPASS) &&
        !c->authenticated &&
        (c->cmd->proc != authCommand || c->cmd->proc == helloCommand))
    {
        flagTransaction(c);
        addReply(c,shared.noautherr);
        return C_OK;
    }

/* AUTH <passowrd>
 * AUTH <username> <password> (Redis >= 6.0 form)
 *
 * When the user is omitted it means that we are trying to authenticate
 * against the default user. */
void authCommand(client *c) {
    /* Only two or three argument forms are allowed. */
    if (c->argc > 3) {
        addReply(c,shared.syntaxerr);
        return;
    }

    /* Handle the two different forms here. The form with two arguments
     * will just use "default" as username. */
    robj *username, *password;
    if (c->argc == 2) {
        /* Mimic the old behavior of giving an error for the two commands
         * from if no password is configured. */
        if (DefaultUser->flags & USER_FLAG_NOPASS) {
            addReplyError(c,"AUTH <password> called without any password "
                            "configured for the default user. Are you sure "
                            "your configuration is correct?");
            return;
        }

        username = createStringObject("default",7);
        password = c->argv[1];
    } else {
        username = c->argv[1];
        password = c->argv[2];
    }

    if (ACLCheckUserCredentials(username,password) == C_OK) {
        c->authenticated = 1;
        c->user = ACLGetUserByName(username->ptr,sdslen(username->ptr));
        addReply(c,shared.ok);
    } else {
        addReplyError(c,"-WRONGPASS invalid username-password pair");
    }

    /* Free the "default" string object we created for the two
     * arguments form. */
    if (c->argc == 2) decrRefCount(username);
}

    shared.noautherr = createObject(OBJ_STRING,sdsnew(
        "-NOAUTH Authentication required.\r\n"));

Authentication was much simpler in the 2.8 branch:

void authCommand(redisClient *c) {
    if (!server.requirepass) {
        addReplyError(c,"Client sent AUTH, but no password is set");
    } else if (!time_independent_strcmp(c->argv[1]->ptr, server.requirepass)) {
      c->authenticated = 1;
      addReply(c,shared.ok);
    } else {
      c->authenticated = 0;
      addReplyError(c,"invalid password");
    }
}