solarwinds/fluent-plugin-loggly-syslog

Fluentd syslog plugin for Loggly

fluent-plugin-loggly-syslog's Issues

UndefinedConversionError error="\"\\xE2\" from ASCII-8BIT to UTF-8"

We're experiencing the following error:

2019-03-22 20:51:49 +0000 [warn]: #0 failed to flush the buffer. retry_time=1 next_retry_seconds=2019-03-22 20:51:49 +0000 chunk="5836b7b0060fa85ddf8aaa89ed735d79" error_class=Encoding::UndefinedConversionError error="\"\\xE2\" from ASCII-8BIT to UTF-8"
2019-03-22 20:51:49 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-loggly-syslog-0.0.4/lib/fluent/plugin/out_loggly_syslog.rb:118:in `encode'
2019-03-22 20:51:49 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-loggly-syslog-0.0.4/lib/fluent/plugin/out_loggly_syslog.rb:118:in `to_json'
2019-03-22 20:51:49 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-loggly-syslog-0.0.4/lib/fluent/plugin/out_loggly_syslog.rb:118:in `create_packet'
2019-03-22 20:51:49 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-loggly-syslog-0.0.4/lib/fluent/plugin/out_loggly_syslog.rb:50:in `block in write'
2019-03-22 20:51:49 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/event.rb:323:in `each'
2019-03-22 20:51:49 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/event.rb:323:in `block in each'
2019-03-22 20:51:49 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/plugin/buffer/file_chunk.rb:172:in `open'
2019-03-22 20:51:49 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/event.rb:322:in `each'
2019-03-22 20:51:49 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-loggly-syslog-0.0.4/lib/fluent/plugin/out_loggly_syslog.rb:47:in `write'
2019-03-22 20:51:49 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/compat/output.rb:131:in `write'
2019-03-22 20:51:49 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/plugin/output.rb:1111:in `try_flush'
2019-03-22 20:51:49 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/plugin/output.rb:1394:in `flush_thread_run'
2019-03-22 20:51:49 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/plugin/output.rb:445:in `block (2 levels) in start'
2019-03-22 20:51:49 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

While Fluentd treats everything as binary, our use of the Ruby built-in .to_json tries to treat the string data as UTF8:
https://github.com/solarwinds/fluent-plugin-loggly-syslog/blob/master/lib/fluent/plugin/out_loggly_syslog.rb#L118

We should avoid doing this conversion process in our plugin. Other Fluentd users are using Yajl.dump as a replacement mechanism, see the thread here:
tagomoris/fluent-mixin-plaintextformatter#10

Support applications logging directly as JSON

Check if msg.message is JSON here:
https://github.com/solarwinds/fluent-plugin-loggly-syslog/blob/master/lib/fluent/plugin/out_loggly_syslog.rb#L108

If so, make it part of the object.

Remove 2nd payload

Hello,

I am using fluent-plugin-loggly-syslog on a AWS EKS cluster in fluentd daemonset in order to forward the logs of my workload to Loggly. But apart from json: { ... } containing the information I want there is a 2nd payload which I do not need.
I'd like to remove this 2nd payload containing syslog: { severity: "...", appName: "...", priority: "...", facility: "...", timestamp: "..." }

Is there a way to filter it out?

Regards,
Sener