Unextracted Message Field in Graylog

[emsearcy]

Unextracted Message Field in Graylog

[emsearcy]

Hi! Can you see if the patch at emsearcy/gelf-rb@81656c2 makes a difference? I forget whether I made a pull request for that, I will go double-check.

[jonathancrosmer]

I ran into this same issue. It seems like a bug in out_gelf, or else I don't understand how it works. The fluentd output is double-encoded so that the log message in Graylog appears as some kind of JSON. When outputting the GELF JSON, shouldn't the fields from the log message be included separately, not as one giant "message" field?
Also @emsearcy how would the logger levels make a difference? It seems like the plugin is outputting JSON where one of the values is also the JSON input string, instead of extracting values from the JSON input and adding them to the output.

[jonathancrosmer]

After inspecting the source I see that the input must have a short_message field, or else the short_message in the GELF JSON output is set to a dump of the whole JSON input, which was confusing.
In my case, I had a simple "tail" input using format none, so I just added message_key short_message on the next line, then Graylog showed the correct value.
I suppose the example linked above had the same problem, which is that there was no short_message, so the plugin dumped the whole JSON string as the short_message. That confused me.

[emsearcy]

Oh, yes you're right that has nothing to do with the logger levels.

Yes, GELF documentation states that short_message is a required field and "MUST" be included, so if it is not included I dumped the JSON to populate this field. The purpose was not double-encoding, but to put something useful into short_message.

Are you using in_gelftail? If you notice my included formats in in_gelftail have a named regex capture for short_message (and full_message) which span several other fields, so that the original line is preserved for the Graylog web interface.

Aside from documenting this better, I'd welcome suggestions about how to build a default short_message.

• Do you like the JSON representation as-is now that you understand why it's there? Maybe include it but without the extra { }?
• Use a cleaner-looking "field1=value field2=value"? (I'd lean towards this)
• Set short_message to "WARNING: required short_message field not present"?
• Include a warning and a representation of the json?

[jonathancrosmer]

I'm totally new to FluentD and Graylog. I haven't used in_gelftail yet, but it sounds useful. At the moment I'm just trying to set up our system to get our existing web application log into Graylog.

Now that I understand it, it seems like this is a warning condition to me. It's friendly to not fail completely, but it is a kind of misconfiguration without an obvious fix. My tendency would be to include the warning (so you know your setup is broken) plus the JSON dump (just to be friendly, so you can see something). Also, I might try to make the warning string somewhat unique, so if someone googles it he'll quickly figure out what's going on.