Multiline filter and generation of an event when no line-ending at EOF

[driskell]

Multiline filter and generation of an event when no line-ending at EOF

[driskell]

I think current behaviour is that the last line will be buffered as never 'complete',

Files are currently treated as streams of lines that are constantly being written. So if we hit a line without EOL (end of line, \n) we wait. The main problem is non-line buffering in a lot of logging systems. They tend to write out in blocks of fixed size, and almost always end up with a half-written line with no EOL. Thus we have to wait until the next block is written in order to process the data and this could be a long time.

I think the best solution is to try ensure the application logs EOL at the end.

I'm not sure there's a way courier could handle this situation. It would need to understand the XML so when it opened a file it could ensure all data was logged, otherwise if it opened it a moment too soon it may catch it mid-write.

[java2kus]

I absolutely agree with you that log-courier cannot send an event when the file is being written and its a bit difficult to track this scenario within the code. The use-case I am talking about in the above example is searching for a pattern i.e. end of a soap envelope </soap:Envelope> . As soon as I hit the pattern, all the lines that were buffered into my multiline buffer need to be flushed as an event. Though I mentioned this as a EOL problem, its completely not related to EOL since, there is already a pattern match on the above.

Since there is no way for log-courier to know or understand the xml in a generic manner, is there anyway to provide a configuration parameter that will alter the behavior of log-courier multiline codec to emit the event on end of file on a pattern match. Let me know what you think.

Overall, log-courier is a fantastic piece of software! I have been using log-forwarder in my environment and have started experimenting with log-courier in the past couple of weeks with some awesome results.

[driskell]

The issue is not that the multiline buffer is not flushing, but that it isn't even receiving the </soap:Envelope> bit. The codecs only receive complete lines, and emit events (which are created from one or more lines). Until a line is complete it sits in the file read buffer which is deeper in the code. So it's definitely an EOL issue since the pattern matching in the codec will only be run against complete lines.

The best and (in my opinion) correct fix here would be to get the application to log EOL at the end. Text files like logs should always end like this, and most editors will correct it. Though I fear that's out of the question for you by the sounds of it. :-(

Failing fixing at source, it could be worth trying to match the previous line (which is complete) instead of </soap:Envelope>. This end tag could then be added back in by Logstash if its absolutely required. This way it's a temporary fix for a specific problem.

I'm just really reluctant to add an option to log-courier as it won't be straight forward and it does seem a very specific problem. Hopefully this is understandable :-)

[java2kus]

Sure Jason. I do understand. I have implemented a very ugly hack in log-courier (or else I would have submitted a pull request) to solve that problem. The only reason why I was checking on this is because its a valid use case. The multi-line filter in logstash had a similar behavior. In order to prevent the premature flushing issue, the `maxmillis elapsed`` feature was implemented to flush the buffer in logstash (currently the default is set to 5 seconds). Please find below the logstash jira issue below:

https://logstash.jira.com/browse/LOGSTASH-271

[java2kus]

Also, I forgot to mention in the above, that its not related to EOF but EOL. By mistake I wrote EOL in the previous comment. Also, the logic is very much required in the what => "previous" case also.

[driskell]

Hi @java2kus

Log Courier multiline codec has "previous timeout" which is the equivalent of the max millis in Logstash.

However, it won't flush the closing tag you have because it won't be in the codec buffer - it's not a finished line.

I'm pretty sure the lack of new line would have the same behaviour in logstash. It wouldn't reach the codec and would never be flushed.

[driskell]

I can see it happens in logstash too I think though it looks old:
https://logstash.jira.com/plugins/servlet/mobile#issue/LOGSTASH-2124

[driskell]

I will close this as I believe the issue lies with the logger, not the shipper, and working around it inside the shipper would be complex and extra code I need to maintain for little benefit globally.