backwards-compatible mode?

[driskell]

backwards-compatible mode?

[driskell]

The lumberjack import has several issues, a couple I consider major (blocking, partial ack, see elastic/logstash-forwarder#180). And the project intention is to have all these issues fixed and to allow us to add the new features the community needs.

For that reason I created a new input for log-courier with those fixed and other improvements. It also allows me to add further improvements over time such as the zmq transport and correct tags fields functionality.

This way one can also run side by side with logstash-forwarder with no change to its endpoint - the lumberjack input. So there's no chance of breaking an existing web of LSF when gradually adding new nodes with log-courier.

I tried to make it as easy to install into Logstash as possible. So it's just a gem build, gem install, and file copy. I'll soon write an installer too to locate logstash and automate this. Any suggestions or concerns just let me know and I'll always look for improvements.

Long term, I'm planning to look at their contrib plugin install process and ask if they'd be willing to automate 3rd party plugin installs from github etc to make this easier.

Jason

[driskell]

I'll get this documented more clearly, maybe with a quick mention in the overview. Or I'll set up an FAQ.

[matthughes]

@driskell So is the logcourier server plugin a drop-in replacement for the lumberjack plugin? I have existing clients using lsf and would love to take advantage of the improvements logcourier has.

[tedder]

@matthughes depends what you mean by "drop-in". It can't read lumberjack inputs, nor can lumberjack read logcourier inputs. You can run them side-by-side (on different ports) or you can remove lumberjack and put logcourier on the same port. I chose the former option.

[matthughes]

Ok thanks. I was hoping to replace lumberjack on the server as I migrate my clients, but looks like I'll have to do that in tandem.

[tedder]

Yeah, that's why I brought up both. I have Ansible bits if you want 'em.

[driskell]

@matthughes Yes @tedder is correct the plugins are incompatible.

Main reason is the timeout fix. With log courier plugin Logstash sends periodic updates to Log Courier to tell it which events it has completed. This communication serves two purposes: prevent timeouts, and prevent full duplication on subsequent timeout or general failure (it knows how many events were completed)

This is possible using the lumberjack protocol, however the forwarder never implemented the full protocol, it treats any and all return messages as a full acknowledgement of all events.

So rather than cause issues and further problems I created a new plugin and also a new protocol. It opened up ability to extend and improve it further too. It's no longer key value too and so the fields in the Log Courier config file can be arrays and dicts too (main use is multiple tags as an array - something forwarder cannot do)

Jason

[matthughes]

Main reason is the timeout fix. With log courier plugin Logstash sends periodic updates to Log Courier to tell it which events it has completed. This communication serves two purposes: prevent timeouts, and prevent full duplication on subsequent timeout or general failure (it knows how many events were completed)

Could you elaborate the communication paths described here? Specifically the "periodic updates...to tell it which events it has completed." Does that preclude using a stateless load balancer? If you had three nodes running Logstash/Logcourier plugin, what happens if the lc client points to different ones each time it sends logs? Where is the state for "which events it has completed" stored?

[driskell]

@matthughes

Specifically the "periodic updates...to tell it which events it has completed." Does that preclude using a stateless load balancer? If you had three nodes running Logstash/Logcourier plugin, what happens if the lc client points to different ones each time it sends logs?

A load balancer is fine but it must maintain TCP state. That is, once connection is established the remote server must remain the same. Load balancers that work with TCP already do this and only balance on connection establish. They'd need to understand the courier protocol to balance within a connection.

Where is the state for "which events it has completed" stored?

In the .log-courier file in the working directory, much like forwarder itself does.

Jason

[matthughes]

Right, but if the connection to the server got interrupted and the client tried to reconnect, it wouldn't be a problem if it connected to a different node in the LB pool. Is that correct?

[driskell]

Yes correct. When it reconnects it'll resume where it left off with the previous server.