redis support

[driskell]

redis support

[driskell]

My plan would be to make redis completely redundant and just use ZMQ as it allows spreading the loads and offers simple proxy and load balancing, all with no need for a broker or single point of failure.

Redis as a broker seems to offer little benefit, I've only used it with LSF to mitigate the network issues (elastic/logstash-forwarder#180) during spurts when logstash is really busy. With Log Courier it will just slow down, wait, and use the local log files as the buffer.

[igkins]

so in our use case, we had the courier shipper's talk to zmq/courier over logstash. However, when there were many clients, it was several times slower than having the shipper talk to redis directly, and have logstash pop stuff off a redis list. Problem was, we had to use beaver for that setup and that took up more resources on the shipper than what we would have liked.

[driskell]

Ah I see. Exactly what do you mean by slow though? So I can look where we need to improve things.

Thanks.

[driskell]

Hi @igkins

Can you provide information on what was going slow, and how you measured it?

I cannot think of a reason that shipping to redis instead of logstash directly is any quicker overall. There may be an appearance of quicker on the shipper side, because the redis acts as a buffer behind the (possibly) slow logstash indexing. In the case of shipping directly to logstash, the shipper acts as the buffer and automatically drops its speed to the same as the speed of indexing at logstash, so that no message is ever lost.

Jason

[igkins]

So just to clarify, it wasn't exactly slow, just slower than redis, still lightyears above straight logstash or logstash-forwarder. Our two setups were:

beaver=>redis=>logstash (with redis input, 1 list)
courier=>logstash (with 0mq+courier)

both setups used the metric output at first, and the did a dots | pv -War (based on recs from irc, both basically outputed the same thing). After that test, we then did the same output to ES to see how thoroughput declined. This was all done on ec2 with the clients as 8 m3.mediums and 1 logstash server which was a r3.2xlarge with 16 logstash workers. At no point did the logstash server cpu or memory reach its limits, nor network usage.

with redis we maxed out ~100k/sec
with courier we maxed out at ~52k/sec

[driskell]

@igkins Thanks that's great figures.

When using courier ZMQ, were you using curve (transport: "zmq") or were you using plain (transport: "plainzmq"). If you were using transport "zmq" with curve keys, are you able to try "plainzmq"?

Curve is the encryption and authentication layer in 0mq, so will add overhead and be slower compared the unencrypted and unauthenticated beaver=>redis connections. Maybe this was the slowness. If not let me know and I can look elsewhere!

Jason

[igkins]

we were using plainzmq

[driskell]

Thanks. I'll do some testing here. Do you know when you did the tests? I did a few enhancements to the plugins that increased speed a small amount on Aug 22, 2014. I will do some testing and compare redis input to courier input.

[igkins]

we did our tests mostly this week, but we were using the stable branch of log-courier. Think it was from a couple weeks back.

[driskell]

OK. The develop branch has some revamps in the plugins to reduce the CPU usage of the plugins when receiving the payloads, decompressing and decrypting etc, so could increase throughput if you were hitting high CPU in logstash.

I will go ahead and do some test anyway just to make sure, and do further streamlining if need be.

One last question - was the redis on the same server as the logstash?

[igkins]

no, it was an elasticcache instance. so totally managed by aws.

[driskell]

Hi igkins,

I'll close this now as there's not much I can do. The develop branch changes though do increase the performance quite substantially in CPU contained environments like my own.

If you ever do get to test the changes, and want to help isolate the slowest code areas we need to improve that will be great, and we can open a new issue for them.

Thanks

Jason