eleme/corvus

A fast and lightweight Redis Cluster Proxy for Redis 3.0

Corvus

Corvus is a fast and lightweight redis cluster proxy for redis 3.0 with cluster mode enabled.

Why

Most redis client implementations don't support redis cluster. We have a lot of services relying on redis, which are written in Python, Java, Go, Nodejs etc. It's hard to provide redis client libraries for multiple languages without breaking compatibilities. We used twemproxy before, but it relies on sentinel for high availabity, it also requires restarting to add or remove backend redis instances, which causes service interruption. And twemproxy is single threaded, we have to deploy multiple twemproxy instances for large number of clients, which causes the sa headaches.

Therefore, we made corvus.

Features

• Fast.
• Lightweight.
• Painless upgrade to official redis cluster from twemproxy.
• Multiple threading.
• Reuseport support.
• Pipeline support.
• Statsd integration.
• Syslog integration.
• Can direct read operations to slaves.

Performance

See details in our test results.

Requirements

• Linux >= 3.9
• Redis <= 3.0.7 (For version up to/including 0.2.3)

Build

If use the releases downloaded from releases page, just make:

$ make

With debug mode enabled:

make debug

If build from latest source:

git clone https://github.com/eleme/corvus.git
cd corvus
git submodule update --init
make deps # need autoconf
make

Binary can be found at ./src/corvus.

Configuration

Example configuration file is at corvus.conf.

Usage

$ ./src/corvus path/to/corvus.conf

Commands

• All single-key commands (like SET, GET, INCR..) are supported.
• Batch commands are split into multiple single-key commands.
• Commands performing complex multi-key operations like unions or intersections are available as well as long as the keys all belong to the same node.

Modified commands

• MGET: split to multiple GET.
• MSET: split to multiple SET.
• DEL: split to multiple single key DEL.
• EXISTS: split to multiple single key EXISTS.
• PING: ignored and won't be forwarded.
• INFO, TIME: won't be forwarded to backend redis, information collected in proxy will be returned.
• SLOWLOG: return the slowlogs saved by corvus itself. Note that unlike redis, in the slowlog entry there's an additional remote latency field before the total latency field. The slowlog will also log the slowest sub command for multiple-key commands: MGET, MSET, DEL, EXISTS. The total latency of sub cmd entry will just be the same as its parent cmd.
• AUTH: do authentication in proxy.
• CONFIG: support get, set, and rewrite sub-command to retrieve and manipulate corvus config.
• SELECT: ignored if index is 0, won't be forwarded.

Restricted commands

• EVAL: at least one key should be given. If there are multiple keys, all of them should belong to the same node.

The following commands require all argument keys to belong to the same redis node:

• SORT.
• RPOP, LPUSH.
• SDIFF, SDIFFSTORE, SINTER, SINTERSTORE, SMOVE, SUNION, SUNIONSTORE.
• ZINTERSTORE, ZUNIONSTORE.
• PFCOUNTE, PFMERGE.

Unsupported commands

The following commands are not available, such as KEYS, we can't search keys across all backend redis instances.

• KEYS, MIGRATE, MOVE, OBJECT, RANDOMKEY, RENAME, RENAMENX, SCAN, WAIT.
• BITOP, MSETNX
• BLPOP, BRPOP, BRPOPLPUSH.
• PSUBSCRIBE, PUBLISH, PUBSUB, PUNSUBSCRIBE, SUBSCRIBE, UNSUBSCRIBE.
• EVALSHA, SCRIPT.
• DISCARD, EXEC, MULTI, UNWATCH, WATCH.
• CLUSTER.
• ECHO, QUIT.
• BGREWRITEAOF, BGSAVE, CLIENT, COMMAND, CONFIG, DBSIZE, DEBUG, FLUSHALL, FLUSHDB, LASTSAVE, MONITOR, ROLE, SAVE, SHUTDOWN, SLAVEOF, SYNC.

License

MIT. Copyright (c) 2016 Eleme Inc.

See LICENSE for details.

corvus's Issues

Awesome stuff!

Awesome stuff @maralla @xinming90 !

Corvus will be shared in our next RedisWeekly! Once published (in a matter of minutes) you will be able to see your project here :)

hot key and hot instance

1. record the hot access key by a counter
2. record the hot access instance by a counter

---

edit:

Add an command to retrieve top n accessed keys in corvus.

Bug: corvus stuck in a loop requesting cluster slots.

Version: 0.2.3

Requests are slow during this period, because the redis instance processing cluster slots is busy sending result back to corvus.

startup log:

2016-06-30 16:49:20.284 info corvus: [1757] starting slot manager thread
2016-06-30 16:49:20.288 info corvus: [1760] updating slot map using 10.0.15.60:7113
2016-06-30 16:49:20.300 info corvus: [1757] starting stats thread
2016-06-30 16:49:20.301 info corvus: [1757] serve at 0.0.0.0:9301

slot update logs:

https://gist.githubusercontent.com/wooparadog/ed40c5e284b95605e57851d789dbcb4a/raw/-

corvus是否支持ssdb或者pika？

corvus是否支持ssdb或者pika？

在redis_cluster集群基础上安装了corvus后为啥tps降低了

我创建了3个主节点 3个从节点 ，然后配置如下：
bind 12345
node 127.0.0.1:6379,127.0.0.1:6380,127.0.0.1:6381
thread 4
启动后用redis-benchmark压测发现tps降了
不用corvus是这样 GET: 33046.93 requests per second
用了corvus后 GET: 5262.88 requests per second

Cpu(all) metrics should be incremental.

So that we can put alerts on them.

Slowlog monitoring

We should count how many requests are slow in corvus, and send it to statsd.

Thoughts:

For a specific $cluster, if one $cmd (like GET, SET...) has cost more than 10ms(can be configured),
increase the inmemory counter for that slow log category by 1. And send the combined result to statsd
every 10s

1. statsd.corvus.$cluster.slow_query.$cmd
2. statsd.corvus.$cluster.nodes.$node.slow_query.$cmd

unknown command type 'Q' (parser.c:155)

phpredis

监控需求

1. corvus 本身的 流量监控

2. slow query 的监控.
   指标思路:

   1. statsd.corvus.$cluster.slow_query.$cmd
   2. statsd.corvus.$cluster.nodes.$node.slow_query.$cmd
   

Wrong stats data

Sometimes connected_clients and remote_latency get negative value.

pub/sub cmd

Is there any plan on this kind of commands supporting?
They are available when a redis node is in a cluster.

Redis 4 Support

Whats the status of using latest redis 4 with this proxy?

Cheers Max

Functional enhancement

Do you have any plan do below function?

1. Traffic restrictions,like ops ,connection
2. Blacklist,White-list
3. Copy and distribute traffic

在线压缩、backup-read、大key检查

增加了一些特性, 可以提交到主干. 现在这里的corvus还有人维护吗?

Exception when `mget` followed by no keys.

Allow listening on non-0.0.0.0 interface

Currently corvus always listens on 0.0.0.0, making it globally reachable over the machine's interfaces.
Additionally, the configuration parameter is called bind but sets the port corvus listens on.

It would be desirable to be able to configure the interface to listen to for incoming connection (e.g. 127.0.0.1 aka localhost only).
Redis also uses bind for the interfaces and port for the actual port it is listening to.

In addition, this would make it possible to add IPv6 support as well.

是否在检测到redis连接不上的情况下 关闭server的端口不接收请求？

问题：
1 qps非常高
2 corvus检测到redis连接不上时，会触发refresh
3 refresh的时候应该停掉server，因为如果不停掉server的端口，在qps非常高的情况下,请求堆积，corvus内存瞬间就打爆了
4如果在corvus和redis连接不可用的情况下，关闭server端口，客户端能做到识别从而不往他发请求

Will corvus handle cluster state on slot movements

I am using this from a php client.

If some slots move to another node... redis-cluster usually tells me with MOVED ip:port response.

Does corvus maintain these informations so requests for this slot will then be redirected by corvus to the correct new node?

Idea is to prevent the redirect/connect overhead if slots are moved...

thanks max

document seems needs to more explain

Corvus seems a good job, but I cannot understand what corvus' config file, and what corvus' cluster exactly means, and I also can't understand the logic of corvus.
there are some questions,

1. Dose it support sharding with hashtag like twemproxy?
2. How to config redis cluster?
   for example, i have 2 master redis(:6000 :7000) with some slaves(use the cmd of slaveof :7000). then should I config it as below(I tried and seems wrong...)?
   bind 1000 node localhost:6301,localhost:7000 thread 4

Could you kindly explain to us. Many Thanks!

Record slowest sub command

When a multiple key command is considered slow, besides adding itself to slowlog command queue, it should also add its slowest sub command so that we can determine whether there is a key or redis node slow down the whole command.

Remove unnecessary null pointer checks

An extra null pointer check is not needed in functions like the following.

• config_add
• pos_array_free

Support auth

For better management, we need to have auth. We can put password/auth info in config file, instead of using the native auth in redis itself.

support 2.x ?

Missing SELECT Command

We should implement SELECT command for compatibility. #113

corvus metrics信息, 有没有命令直接取出来？

corvus metrics信息, 有没有命令直接取出来？

类似redis的info命令，看配置里面是要配合statsd才能取到数据，但是我们的监控环境没有statsd。能否提供命令或者api？

Allow increasing maxclients

I see that the max clients I was able to reach is 468. Is it something hardcoded or I can change it via settings?

[Question] - Command split

Hi,

First, thanks for you great work! 🎉

Why do you do command splitting ?
Is it to reduce redis lock time of large MGET/MSET/DEL/... ?
Did you make some measurements ?

Completion of error handling

Would you like to add more error handling for return values from functions like the following?

• malloc ⇒ conn_create
• pthread_cond_init ⇒ slot_init_updater

Can I use Lettuce with corvus? Maybe there are some bugs!

client : Lettuce，a aync client base netty nio

corvus version : a Secondary development version base eleme/corvus master(update to date)
redis: redis cluster

problem:
when run my code some time, i got some exceptions like this:

tcpdump:

I find some things wrong by tcpdump.

1. the corvus response do not follow the RESP. like *2 $-1 should be *2 $-1 $-1
2. the response order is incorrect , so cannot be matched (request) one by one

just like this photos:

reproducer code:

public class RedisOptionV2 {
    private static RedisAsyncCommands<String, String> ASYNC;
    private static boolean flag = false;
    private static AtomicLong index = new AtomicLong(0);

    static {
        RedisURI redisURI = RedisURI.create("corvus ip",  6666);
        RedisClient redisClient = RedisClient.create(redisURI);
        StatefulRedisConnection<String, String> connection = redisClient.connect();
        ASYNC = connection.async();
    }

    @PostConstruct
    void init() {
        MyExecutorService.schedule(this::lettuceTestTask, 30, TimeUnit.SECONDS);
    }

    private void lettuceTestTask() {
        for (long i = 0; i < Long.MAX_VALUE; i++) {
            try {
                MyExecutorService.submit(this::lettuceDemo);
            } catch (Throwable t) {
                t.printStackTrace();
            }
        }
    }

    public void lettuceDemo() {
        try {
            List<String> keys = new ArrayList<>();
            keys.add("debugLettuceKey" + index.getAndIncrement());
            if (flag) {
                keys.add("debugLettuceKey2");
                flag = false;
            } else {
                flag = true;
            }

            //core code
            RedisFuture<List<KeyValue<String, String>>> future = ASYNC.mget(keys.toArray(new String[0]));
            future.whenCompleteAsync((e, t) -> {
                if (t == null) {
                    log.info("{} res: {}", keys, e);
                } else {
                    log.error("调用redis失败, {} res : {}", keys, e, t);
                }
            }, MyExecutorService.getExecutorService());

        } catch (Throwable t) {
            log.error("fail to execute task", t);
        }
    }
}

Looking forward to reply！Thanks

We should lock version of jemalloc

Metrics is not accurate in some siturations during high traffic.

Situation:

1. High traffic: over 1 million ops
2. Large a mount of pipelines.

Metrics recorded by corvus is higher than it actually is. We tried to reproduce it in offline environment, but failed.

Do you have a plan for supporting backend redis cluster configed with requirepass?

Hello,

With v0.2.7 tag and redis-cluster which is configured with requirepass.
Found that the corvus doesn't send AUTH command to the backend redis cluster instance.
So I got "(error) NOAUTH Authentication required."

Twemproxy has redis_auth configuration which implements "Authenticate to the Redis server on connect."

Thanks!

monitoring slowlog in Corvus

Hello,

When I set the slowlog-log-slower-than with big values like e.g. "24000", the slowlogs I get are also big e.g.

localhost:6380> config set slowlog-log-slower-than 24000
OK
localhost:6380> slowlog reset
OK
localhost:6380> slowlog get 2

1. 1. (integer) 12417746
   2. (integer) 1555674724
   3. (integer) 48043
   4. (integer) 50242
      ...
2. 1. (integer) 12417750
   2. (integer) 1555674724
   3. (integer) 47861
   4. (integer) 48369

In the same environment, when I set the slowlog-log-slower-than with small values "24", the slowlogs I get are also small and the big delays being displayed previously are disappeared:

localhost:6380> config set slowlog-log-slower-than 24
OK
localhost:6380> slowlog reset
OK
localhost:6380> slowlog get 2

1. 1. (integer) 12429661
   2. (integer) 1555675090
   3. (integer) 775
   4. (integer) 807
      ...
2. 1. (integer) 12429671
   2. (integer) 1555675090
   3. (integer) 282
   4. (integer) 318

So does Corvus understands the desired time unit (micro, mill, etc) from the config setting it so does display the result accordingly?

By the way, in my environment, the latencies are below 24ms so I can't interpret correctly the numbers like 282 and 318 ...

Thank you

support for redis 3.2?

I wonder if supporting higher versions of redis is on the roadmap? it'd be awesome to see the new features introduced by redis 3.2.

Better configuration manner for read/write seperation.

from two options read-slave and read-master-slave to one option like:

read-strategy with values being:

1. master default
2. read-slave-only send all read ops to slaves.
3. both send read ops to both master and slaves.

ERROR [30942]: cmd_parse_token: fail to parse command QUIT (command.c:716)

ERROR [30942]: cmd_parse_token: fail to parse command QUIT (command.c:716)

How to configuration node password in corvus

My node by cluster machine configurationed password ,but I don't know how to configuration password by corvus.

Issue installing v0.2.7 from the release page

Hi,

I'm trying to install using the package from the release page, but I cannot see the package there for v0.2.7, the last one is v0.2.4.

I'm trying to compile from the source code(downloaded from the release page), but it is failing:

$ make
/Library/Developer/CommandLineTools/usr/bin/make jemalloc -C ../deps
cd jemalloc && ./configure --with-jemalloc-prefix=je_ --enable-cc-silence
/bin/sh: ./configure: No such file or directory
make[2]: *** [jemalloc] Error 127
make[1]: *** [../deps/jemalloc/lib/libjemalloc.a] Error 2
make: *** [corvus] Error 2

And I cannot run git submodule update --init because this is not a git repository.

Can you provide corvus-0.2.7.tar.bz2. So we can install just running make?

Corvus key distribution algorithm

Hi.
We are currently working with nutcracker (twemproxy) in order to create our own redis cluster. However we are experiencing some scaling problems due to twemproxy is not multithreaded. In order to start testing with Corvus using the same existing cluster, we would like to know which key distribution algorithm is being used currently in order to know if we can use the same existing cluster to test in parallel both solutions.
If corvus use ketama as algorithm we would use is without any complex migration procedure for our persistent keys.

Thanks in advance.

Fix signal handler

The function "exit" does not belong to the list of async-signal-safe functions.
I guess that a different program design will be needed for your function "sig_handler".

Support of Redis 5+

Hello,

are there any plans to support Redis 5.0 or higher?

Thank you,
Apostolos

how to use corvus ?

Just came across corvus and was wondering if you could clarify how to use it ?

I use my redis-generator.sh to create 3 stand alone redis servers locally. But is there anything else I need to do before starting up corvus ? Do I need to setup an actual redis cluster first or just standalone redis server instances ?

On CentOS 7.3, created 3 redis 3.2.8 based servers on TCP ports 6479, 6480, and 6481

./redis-generator.sh 3

Creating redis servers starting at TCP = 6479...
-------------------------------------------------------
creating redis server: redis6479.service [increment value: 0]
redis TCP port: 6479
create systemd redis6479.service
cp -a /usr/lib/systemd/system/redis.service /usr/lib/systemd/system/redis6479.service
create /etc/redis6479/redis6479.conf config file
mkdir -p /etc/redis6479
cp -a /etc/redis.conf /etc/redis6479/redis6479.conf
-rw-r----- 1 redis root 46K Mar 13 21:22 /etc/redis6479/redis6479.conf
-rw-r--r-- 1 root  root 249 Sep 14 08:43 /usr/lib/systemd/system/redis6479.service
Created symlink from /etc/systemd/system/multi-user.target.wants/redis6479.service to /usr/lib/systemd/system/redis6479.service.
## Redis TCP 6479 Info ##
# Server
redis_version:3.2.8
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:dd923e72e9efa6d8
redis_mode:standalone
os:Linux 3.10.0-514.10.2.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.5
process_id:28820
run_id:75012c9827a5090346297b039e405fbe8e20263e
tcp_port:6479
uptime_in_seconds:0
uptime_in_days:0
hz:10
lru_clock:13154149
executable:/etc/redis6479/redis-server
config_file:/etc/redis6479/redis6479.conf
-------------------------------------------------------
creating redis server: redis6480.service [increment value: 1]
redis TCP port: 6480
create systemd redis6480.service
cp -a /usr/lib/systemd/system/redis.service /usr/lib/systemd/system/redis6480.service
create /etc/redis6480/redis6480.conf config file
mkdir -p /etc/redis6480
cp -a /etc/redis.conf /etc/redis6480/redis6480.conf
-rw-r----- 1 redis root 46K Mar 13 21:22 /etc/redis6480/redis6480.conf
-rw-r--r-- 1 root  root 249 Sep 14 08:43 /usr/lib/systemd/system/redis6480.service
Created symlink from /etc/systemd/system/multi-user.target.wants/redis6480.service to /usr/lib/systemd/system/redis6480.service.
## Redis TCP 6480 Info ##
# Server
redis_version:3.2.8
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:dd923e72e9efa6d8
redis_mode:standalone
os:Linux 3.10.0-514.10.2.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.5
process_id:28887
run_id:05e6b7f63f98d9bb1e6b80c072b78aa3126dfdd4
tcp_port:6480
uptime_in_seconds:0
uptime_in_days:0
hz:10
lru_clock:13154149
executable:/etc/redis6480/redis-server
config_file:/etc/redis6480/redis6480.conf
-------------------------------------------------------
creating redis server: redis6481.service [increment value: 2]
redis TCP port: 6481
create systemd redis6481.service
cp -a /usr/lib/systemd/system/redis.service /usr/lib/systemd/system/redis6481.service
create /etc/redis6481/redis6481.conf config file
mkdir -p /etc/redis6481
cp -a /etc/redis.conf /etc/redis6481/redis6481.conf
-rw-r----- 1 redis root 46K Mar 13 21:22 /etc/redis6481/redis6481.conf
-rw-r--r-- 1 root  root 249 Sep 14 08:43 /usr/lib/systemd/system/redis6481.service
Created symlink from /etc/systemd/system/multi-user.target.wants/redis6481.service to /usr/lib/systemd/system/redis6481.service.
## Redis TCP 6481 Info ##
# Server
redis_version:3.2.8
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:dd923e72e9efa6d8
redis_mode:standalone
os:Linux 3.10.0-514.10.2.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.5
process_id:28954
run_id:2019276e1ec4c6a359f9b6e952c31b467e7a04fb
tcp_port:6481
uptime_in_seconds:0
uptime_in_days:0
hz:10
lru_clock:13154150
executable:/etc/redis6481/redis-server
config_file:/etc/redis6481/redis6481.conf

then setup corvus.conf file

bind 12345
node 127.0.0.1:6479,127.0.0.1:6480,127.0.0.1:6481
thread 4

# debug, info, warn, error
loglevel debug
syslog 0

# Close the connection after client idle for `client_timeout` seconds.
# No response after `server_timeout` seconds waiting for redis-server
# connection to redis-server will be closed.
#
# Value 0 means never timeout.
#
# Default 0
#
# client_timeout 30
# server_timeout 5

# Statsd config
# metrics:
#   corvus.<cluster>.<host-port>.<value label>
#
#   corvus.default.localhost-12345.connected_clients
#   corvus.default.localhost-12345.completed_commands
#   corvus.default.localhost-12345.used_cpu_sys
#   corvus.default.localhost-12345.used_cpu_user
#   corvus.default.localhost-12345.latency
#   corvus.default.localhost-12345.redis-node.127-0-0-1-8000.bytes.{send,recv}
#
# Cluster annotation. Using `cluster` to add a cluster name to metrics.
#
# cluster default
#
# Metrices are sent using udp socket. Using `statsd` config to
# set statsd server address. Every `metric_interval` senconds
# will produce a set of metrices.
#
# statsd localhost:8125
# metric_interval 10

# Buffer size allocated each time avoiding fregments
# Buffer used in processing data recieving or sending
# Min buffer size is limited to 64 Bytes
# Default value is 16KBytes (16384)
#
# bufsize 16384
#
# Client should send AUTH <PASSWORD> if `requirepass` setted.
# Corvus will not forward this command, and do authentication just by itself.
# If it is given empty, it will be no effect and you can access the proxy with no password check.
#
# requirepass password
#
# Use `read-strategy` to config how to read from the cluster. It has three valid
# values:
#
#   * `master`, forward all reading commands to master, the default
#   * `read-slave-only`, forward all reading commands to slaves
#   * `both`, forward reading commands to both master and slaves
#
# If new slaves are added to the cluster, `PROXY UPDATESLOTMAP` should be emmited
# to tell corvus to use the newly added slaves.
#
# read-strategy master

# Slowlog
# The following two configs are almost the same with redis.
# Every command whose lantency exceeds `slowlog-log-slower-than` will be considered a slow command,
# which will be sent to statsd and can also be retrieved using `slowlog get`.
# Note that the lantency here is the time spent in proxy,
# including redirection caused by MOVED and ASK.
# Both slowlog command and sending slowlog to statsd are disabled by default.
# You can enable both or either of them.
#
# A zero value will log every command.
# slowlog-log-slower-than 10000
#
# Set this to positive value to enable slowlog command.
# slowlog-max-len 1024
#
# Set this to 1 if you want to send slowlog to statsd.
# slowlog-statsd-enabled 0

then start corvus

src/corvus /etc/corvus/corvus.conf 
2017-03-15 03:41:10,813 DEBUG [default 12345 28999 28999]: Registering signal[Interrupt] (corvus.c:62)
2017-03-15 03:41:10,814 DEBUG [default 12345 28999 28999]: Registering signal[Terminated] (corvus.c:62)
2017-03-15 03:41:10,814 DEBUG [default 12345 28999 28999]: Registering signal[Segmentation fault] (corvus.c:62)
2017-03-15 03:41:10,814 INFO [default 12345 28999 28999]: starting slot manager thread (slot.c:533)
2017-03-15 03:41:10,814 DEBUG [default 12345 28999 29001]: slowlog enabled (corvus.c:256)
2017-03-15 03:41:10,814 DEBUG [default 12345 28999 29002]: slowlog enabled (corvus.c:256)
2017-03-15 03:41:10,816 DEBUG [default 12345 28999 29003]: slowlog enabled (corvus.c:256)
2017-03-15 03:41:10,816 INFO [default 12345 28999 28999]: serve at 0.0.0.0:12345 (corvus.c:481)
2017-03-15 03:41:10,816 DEBUG [default 12345 28999 29004]: slowlog enabled (corvus.c:256)
2017-03-15 03:41:10,817 INFO [default 12345 28999 29000]: updating slot map using 127.0.0.1:6481 (slot.c:290)
2017-03-15 03:41:10,819 ERROR [default 12345 28999 29000]: parse_cluster_nodes: expect data type 2 got 5 (slot.c:189)
2017-03-15 03:41:10,819 INFO [default 12345 28999 29000]: updating slot map using 127.0.0.1:6480 (slot.c:290)
2017-03-15 03:41:10,820 ERROR [default 12345 28999 29000]: parse_cluster_nodes: expect data type 2 got 5 (slot.c:189)
2017-03-15 03:41:10,820 INFO [default 12345 28999 29000]: updating slot map using 127.0.0.1:6479 (slot.c:290)
2017-03-15 03:41:10,820 ERROR [default 12345 28999 29000]: parse_cluster_nodes: expect data type 2 got 5 (slot.c:189)
2017-03-15 03:41:10,820 WARN [default 12345 28999 29000]: can not update slot map (slot.c:351)

DEBUG output suggests not working well ?

Auth command for redis instances?

Great software, Does corvus support redis AUTH for instances or does it just accept a master like twemporxy? i.e.

myredis.com:6379 - AUTH mypass
myredis2.com:6379 - AUTH mypass2

reserved identifier violation

I would like to point out that identifiers like "__CORVUS_H" and "__PARSER_H" do not fit to the expected naming convention of the C language standard.
Would you like to adjust your selection for unique names?

Add metrics for `slot_create_job(SLOT_UPDATE)`

The number of slot_create_job(SLOT_UPDATE) can tell some very fundamental information about the health of redis cluster (if the number is too high, there might be a partition or resharding).

So, giving out this information is helpful for monitoring

corvus health check

Any suggestions on how to implement a simple prometheus health check for corvus?

I tried a simple TCP 'PING' request and expected 'PONG', but that didn't work.

Please share a simple 'telnet' or 'netcat' example command line for the suggestion. I can use that to automate the check on my end.

Support hashtag key ?

support redis 4.x later ?

thank you.

KEYS should be possible

From my perspective, the limitation of not being able to get all KEYS is a total dealbreaker.

Being sure, that this is only a design limitation made willingly, I want to ask politely if you really don't see any chance of implementing this anyhow.

Technically, it's just that all cluster nodes have to be SCAN from corvus and responded to the client just as if it was one single instance. This may be slower, but better than not available.

Since SCAN is also not supported, there seems to be no alternative for getting all keys in a cluster.

Is there any technical reason why this can't be done?

Metrics should be alert friendly.

Should send calculated metrics per 10s instead of sending a total number and let graphite calculate the increment.

So we can use banshee for alerting.