Giter Site home page Giter Site logo

Comments (11)

driskell avatar driskell commented on July 23, 2024

The lumberjack import has several issues, a couple I consider major (blocking, partial ack, see elastic/logstash-forwarder#180). And the project intention is to have all these issues fixed and to allow us to add the new features the community needs.

For that reason I created a new input for log-courier with those fixed and other improvements. It also allows me to add further improvements over time such as the zmq transport and correct tags fields functionality.

This way one can also run side by side with logstash-forwarder with no change to its endpoint - the lumberjack input. So there's no chance of breaking an existing web of LSF when gradually adding new nodes with log-courier.

I tried to make it as easy to install into Logstash as possible. So it's just a gem build, gem install, and file copy. I'll soon write an installer too to locate logstash and automate this. Any suggestions or concerns just let me know and I'll always look for improvements.

Long term, I'm planning to look at their contrib plugin install process and ask if they'd be willing to automate 3rd party plugin installs from github etc to make this easier.

Jason

from log-courier.

driskell avatar driskell commented on July 23, 2024

I'll get this documented more clearly, maybe with a quick mention in the overview. Or I'll set up an FAQ.

from log-courier.

matthughes avatar matthughes commented on July 23, 2024

@driskell So is the logcourier server plugin a drop-in replacement for the lumberjack plugin? I have existing clients using lsf and would love to take advantage of the improvements logcourier has.

from log-courier.

tedder avatar tedder commented on July 23, 2024

@matthughes depends what you mean by "drop-in". It can't read lumberjack inputs, nor can lumberjack read logcourier inputs. You can run them side-by-side (on different ports) or you can remove lumberjack and put logcourier on the same port. I chose the former option.

from log-courier.

matthughes avatar matthughes commented on July 23, 2024

Ok thanks. I was hoping to replace lumberjack on the server as I migrate my clients, but looks like I'll have to do that in tandem.

from log-courier.

tedder avatar tedder commented on July 23, 2024

Yeah, that's why I brought up both. I have Ansible bits if you want 'em.

from log-courier.

driskell avatar driskell commented on July 23, 2024

@matthughes Yes @tedder is correct the plugins are incompatible.

Main reason is the timeout fix. With log courier plugin Logstash sends periodic updates to Log Courier to tell it which events it has completed. This communication serves two purposes: prevent timeouts, and prevent full duplication on subsequent timeout or general failure (it knows how many events were completed)

This is possible using the lumberjack protocol, however the forwarder never implemented the full protocol, it treats any and all return messages as a full acknowledgement of all events.

So rather than cause issues and further problems I created a new plugin and also a new protocol. It opened up ability to extend and improve it further too. It's no longer key value too and so the fields in the Log Courier config file can be arrays and dicts too (main use is multiple tags as an array - something forwarder cannot do)

Jason

from log-courier.

matthughes avatar matthughes commented on July 23, 2024

Main reason is the timeout fix. With log courier plugin Logstash sends periodic updates to Log Courier to tell it which events it has completed. This communication serves two purposes: prevent timeouts, and prevent full duplication on subsequent timeout or general failure (it knows how many events were completed)

Could you elaborate the communication paths described here? Specifically the "periodic updates...to tell it which events it has completed." Does that preclude using a stateless load balancer? If you had three nodes running Logstash/Logcourier plugin, what happens if the lc client points to different ones each time it sends logs? Where is the state for "which events it has completed" stored?

from log-courier.

driskell avatar driskell commented on July 23, 2024

@matthughes

Specifically the "periodic updates...to tell it which events it has completed." Does that preclude using a stateless load balancer? If you had three nodes running Logstash/Logcourier plugin, what happens if the lc client points to different ones each time it sends logs?

A load balancer is fine but it must maintain TCP state. That is, once connection is established the remote server must remain the same. Load balancers that work with TCP already do this and only balance on connection establish. They'd need to understand the courier protocol to balance within a connection.

Where is the state for "which events it has completed" stored?

In the .log-courier file in the working directory, much like forwarder itself does.

Jason

from log-courier.

matthughes avatar matthughes commented on July 23, 2024

Right, but if the connection to the server got interrupted and the client tried to reconnect, it wouldn't be a problem if it connected to a different node in the LB pool. Is that correct?

from log-courier.

driskell avatar driskell commented on July 23, 2024

Yes correct. When it reconnects it'll resume where it left off with the previous server.

from log-courier.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.