Comments (11)
The lumberjack import has several issues, a couple I consider major (blocking, partial ack, see elastic/logstash-forwarder#180). And the project intention is to have all these issues fixed and to allow us to add the new features the community needs.
For that reason I created a new input for log-courier with those fixed and other improvements. It also allows me to add further improvements over time such as the zmq transport and correct tags fields functionality.
This way one can also run side by side with logstash-forwarder with no change to its endpoint - the lumberjack input. So there's no chance of breaking an existing web of LSF when gradually adding new nodes with log-courier.
I tried to make it as easy to install into Logstash as possible. So it's just a gem build, gem install, and file copy. I'll soon write an installer too to locate logstash and automate this. Any suggestions or concerns just let me know and I'll always look for improvements.
Long term, I'm planning to look at their contrib plugin install process and ask if they'd be willing to automate 3rd party plugin installs from github etc to make this easier.
Jason
from log-courier.
I'll get this documented more clearly, maybe with a quick mention in the overview. Or I'll set up an FAQ.
from log-courier.
@driskell So is the logcourier server plugin a drop-in replacement for the lumberjack plugin? I have existing clients using lsf and would love to take advantage of the improvements logcourier has.
from log-courier.
@matthughes depends what you mean by "drop-in". It can't read lumberjack inputs, nor can lumberjack read logcourier inputs. You can run them side-by-side (on different ports) or you can remove lumberjack and put logcourier on the same port. I chose the former option.
from log-courier.
Ok thanks. I was hoping to replace lumberjack on the server as I migrate my clients, but looks like I'll have to do that in tandem.
from log-courier.
Yeah, that's why I brought up both. I have Ansible bits if you want 'em.
from log-courier.
@matthughes Yes @tedder is correct the plugins are incompatible.
Main reason is the timeout fix. With log courier plugin Logstash sends periodic updates to Log Courier to tell it which events it has completed. This communication serves two purposes: prevent timeouts, and prevent full duplication on subsequent timeout or general failure (it knows how many events were completed)
This is possible using the lumberjack protocol, however the forwarder never implemented the full protocol, it treats any and all return messages as a full acknowledgement of all events.
So rather than cause issues and further problems I created a new plugin and also a new protocol. It opened up ability to extend and improve it further too. It's no longer key value too and so the fields in the Log Courier config file can be arrays and dicts too (main use is multiple tags as an array - something forwarder cannot do)
Jason
from log-courier.
Main reason is the timeout fix. With log courier plugin Logstash sends periodic updates to Log Courier to tell it which events it has completed. This communication serves two purposes: prevent timeouts, and prevent full duplication on subsequent timeout or general failure (it knows how many events were completed)
Could you elaborate the communication paths described here? Specifically the "periodic updates...to tell it which events it has completed." Does that preclude using a stateless load balancer? If you had three nodes running Logstash/Logcourier plugin, what happens if the lc client points to different ones each time it sends logs? Where is the state for "which events it has completed" stored?
from log-courier.
Specifically the "periodic updates...to tell it which events it has completed." Does that preclude using a stateless load balancer? If you had three nodes running Logstash/Logcourier plugin, what happens if the lc client points to different ones each time it sends logs?
A load balancer is fine but it must maintain TCP state. That is, once connection is established the remote server must remain the same. Load balancers that work with TCP already do this and only balance on connection establish. They'd need to understand the courier protocol to balance within a connection.
Where is the state for "which events it has completed" stored?
In the .log-courier file in the working directory, much like forwarder itself does.
Jason
from log-courier.
Right, but if the connection to the server got interrupted and the client tried to reconnect, it wouldn't be a problem if it connected to a different node in the LB pool. Is that correct?
from log-courier.
Yes correct. When it reconnects it'll resume where it left off with the previous server.
from log-courier.
Related Issues (20)
- Duplicates observed when log-courier configuration file is overwritten HOT 3
- Undocumented behavior for paths matched by multiple fileglobs HOT 1
- log-courier can't seem to handle `~` for home in certain contexts. HOT 2
- error in logstash 7 HOT 4
- log-courier and logstash > 7.4.0? HOT 10
- Throughput question HOT 15
- does the includes configuration still work correctly? HOT 3
- Hold time setting not closing files properly
- syslog "progname" uses a full path rather than a basename HOT 1
- Information about payloads / different debug level ? HOT 3
- error while compiling code in the command line using MK in windows for Z wave HOT 1
- version.rb missing from the plugin package
- log-courier admin socket stuck HOT 2
- lc-admin does not default to tcp transport HOT 2
- order of files in lc-admin UI
- Logstash configuration auto reload is blocked by log-courier input HOT 2
- PPA packaging is using deprecated compatibility level 9 HOT 1
- 2.10.0 packages missing from PPA (Ubuntu)
- Receiver reload can cause panic
- TCP streaming receiver aborts if too many events received
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from log-courier.