Protocol error - packet too large

[driskell]

Protocol error - packet too large

[driskell]

It's something I will be working on next. It was originally found in #28

There is a hard-coded limit in the logstash plugin that does this as a sanity check that needs adjusting, and some limits need placing in the courier side too at both multiline collection, line length and spool size. I'm just working out the best way to handle them.

The only question is what is better: skip the long line or split it. My initial instinct is to skip - if it's so big I would generally expect it to be a corruption or mistake anyway.

[sysmonk]

Currently i'm in the position i'd happilly skip them, but don't know how ...

If log-courier would skip it, it'd be nice if logged something about it (log line in file XX, LINE YY too big - skipping)

The best solution, of course, would be an option in configuration file to skip or split the log. Some hosts which would want to be PCI compliant would want the log to be split, rather than skipped. Other hosts that are not PCI compliant don't care if one or two log lines would be skipped...

[driskell]

If you look at the offset in the ".log-courier" file it tells you last offset. From there you may be able to look in your log files and see a really big line. In #28 a developer had written a PDF to the log file! That should get things moving forward hopefully. The other option is to modify the log-courier gem to remove or increase the limit. It's set to 1_048_576 (1MB) - which is the compressed size, so it must be quite large - chances are binary too as that's less compressible.

[driskell]

An option to split/skip makes perfect sense. Sorry about the pain - hopefully I can get something implemented sooner rather than later.