Comments (4)
It's something I will be working on next. It was originally found in #28
There is a hard-coded limit in the logstash plugin that does this as a sanity check that needs adjusting, and some limits need placing in the courier side too at both multiline collection, line length and spool size. I'm just working out the best way to handle them.
The only question is what is better: skip the long line or split it. My initial instinct is to skip - if it's so big I would generally expect it to be a corruption or mistake anyway.
from log-courier.
Currently i'm in the position i'd happilly skip them, but don't know how ...
If log-courier would skip it, it'd be nice if logged something about it (log line in file XX, LINE YY too big - skipping)
The best solution, of course, would be an option in configuration file to skip or split the log. Some hosts which would want to be PCI compliant would want the log to be split, rather than skipped. Other hosts that are not PCI compliant don't care if one or two log lines would be skipped...
from log-courier.
If you look at the offset in the ".log-courier" file it tells you last offset. From there you may be able to look in your log files and see a really big line. In #28 a developer had written a PDF to the log file! That should get things moving forward hopefully. The other option is to modify the log-courier gem to remove or increase the limit. It's set to 1_048_576 (1MB) - which is the compressed size, so it must be quite large - chances are binary too as that's less compressible.
from log-courier.
An option to split/skip makes perfect sense. Sorry about the pain - hopefully I can get something implemented sooner rather than later.
from log-courier.
Related Issues (20)
- Duplicates observed when log-courier configuration file is overwritten HOT 3
- Undocumented behavior for paths matched by multiple fileglobs HOT 1
- log-courier can't seem to handle `~` for home in certain contexts. HOT 2
- error in logstash 7 HOT 4
- log-courier and logstash > 7.4.0? HOT 10
- Throughput question HOT 15
- does the includes configuration still work correctly? HOT 3
- Hold time setting not closing files properly
- syslog "progname" uses a full path rather than a basename HOT 1
- Information about payloads / different debug level ? HOT 3
- error while compiling code in the command line using MK in windows for Z wave HOT 1
- version.rb missing from the plugin package
- log-courier admin socket stuck HOT 2
- lc-admin does not default to tcp transport HOT 2
- order of files in lc-admin UI
- Logstash configuration auto reload is blocked by log-courier input HOT 2
- PPA packaging is using deprecated compatibility level 9 HOT 1
- 2.10.0 packages missing from PPA (Ubuntu)
- Receiver reload can cause panic
- TCP streaming receiver aborts if too many events received
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from log-courier.