StructuredSyslogMessage fromString() can not parse rsyslog data

[graylog-labs]

StructuredSyslogMessage fromString() can not parse rsyslog data

[joschi]

@xiaoleihuang Please post the complete syslog message. The example you've posted is missing the syslog priority and version (https://tools.ietf.org/html/rfc5424#section-6).

[xiaoleihuang]

I typed sudo nc -l 10514 to receive packets from rsyslog. Yet, it does not contain any version information like below. Shall I configure my rsyslog service to include those information? I did not find it.

<4>Jun 20 11:59:57 myname kernel: [ 357.266774] [UFW BLOCK] IN=enp9s0 OUT=enp9s0 MAC=01:2e:12:49:87:2b:01:36:1b:38:ad:80:08:50 SRC=289.15.121.109 DST=110.67.112.10 LEN=52 TOS=0x00 PREC=0x00 TTL=47 ID=43803 DF PROTO=TCP SPT=39693 DPT=23

More examples:

<30>Jun 21 00:35:33 xiaoleidouglas dhclient[7445]: bound to 289.15.121.109 -- renewal in 1189 seconds.

<29>Jun 21 00:35:33 xiaoleidouglas dbus[775]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'

<86>Jun 21 00:35:39 xiaoleidouglas compiz: gkr-pam: unlocked login keyring

[xiaoleihuang]

Hi, I tried methods from http://docs.graylog.org/en/1.0/pages/sending_data.html#rsyslog
But it does not work. My rsyslog version is 8.16.0

Here are the sample messages using the rsyslog built-in template SYSLOG_SyslogProtocol23Format:

<4>1 2016-06-21T15:27:15.771223+08:00 xiaoleidouglas kernel - - - [ 3132.531409] [UFW BLOCK] IN=enp9s0 OUT= MAC=01:2e:12:49:87:2b:01:36:1b:38:ad:80:08:50 SRC=219.15.121.109 DST=219.15.121.109 LEN=32 TOS=0x00 PREC=0x00 TTL=1 ID=0 PROTO=2 

<30>1 2016-06-21T15:27:27.627057+08:00 xiaoleidouglas dhclient 1641 - -  DHCPREQUEST of 219.15.121.109 on wlp3s0 to 219.15.121.109 port 67 (xid=0x327a7f27)

<6>1 2016-06-21T15:27:27.672963+08:00 xiaoleidouglas NetworkManager 810 - -  <info>  [1466494047.6728]   address 219.15.121.109

[joschi]

@xiaoleihuang What exactly doesn't work with these messages? Are you using a Syslog input in Graylog?

[xiaoleihuang]

I used class StructuredSyslogMessage with its static method fromString(). It seems that it could only parse its own POJO strings. Which is right way that I could the rsyslog's log data?

[joschi]

@xiaoleihuang StructuredSyslogMessage expects the string to be in a very specific format. You probably want to use StructuredSyslogServerEvent instead.

Example JUnit test:

    @Test
    public void testMessagesIssue13() throws Exception
    {
        final List<String> rfc3164Events = new ArrayList<String>();
        rfc3164Events.add("<4>Jun 20 11:59:57 xiaoleidouglas kernel: [ 357.266774] [UFW BLOCK] IN=enp9s0 OUT=enp9s0 MAC=01:2e:12:49:87:2b:01:36:1b:38:ad:80:08:50 SRC=289.15.121.109 DST=110.67.112.10 LEN=52 TOS=0x00 PREC=0x00 TTL=47 ID=43803 DF PROTO=TCP SPT=39693 DPT=23");
        rfc3164Events.add("<30>Jun 21 00:35:33 xiaoleidouglas dhclient[7445]: bound to 289.15.121.109 -- renewal in 1189 seconds.");
        rfc3164Events.add("<29>Jun 21 00:35:33 xiaoleidouglas dbus[775]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'");
        rfc3164Events.add("<86>Jun 21 00:35:39 xiaoleidouglas compiz: gkr-pam: unlocked login keyring");

        final List<String> rfc5424Events = new ArrayList<String>();
        rfc5424Events.add("<4>1 2016-06-21T15:27:15.771223+08:00 xiaoleidouglas kernel - - - [ 3132.531409] [UFW BLOCK] IN=enp9s0 OUT= MAC=01:2e:12:49:87:2b:01:36:1b:38:ad:80:08:50 SRC=219.15.121.109 DST=219.15.121.109 LEN=32 TOS=0x00 PREC=0x00 TTL=1 ID=0 PROTO=2");
        rfc5424Events.add("<30>1 2016-06-21T15:27:27.627057+08:00 xiaoleidouglas dhclient 1641 - -  DHCPREQUEST of 219.15.121.109 on wlp3s0 to 219.15.121.109 port 67 (xid=0x327a7f27)");
        rfc5424Events.add("<6>1 2016-06-21T15:27:27.672963+08:00 xiaoleidouglas NetworkManager 810 - -  <info>  [1466494047.6728]   address 219.15.121.109");

        for (String message : rfc3164Events) {
            final SyslogServerEvent event = new SyslogServerEvent(message, InetAddress.getLocalHost());
            assertEquals("xiaoleidouglas", event.getHost());
        }

        for (String message : rfc5424Events) {
            final StructuredSyslogServerEvent event = new StructuredSyslogServerEvent(message, InetAddress.getLocalHost());
            assertEquals("xiaoleidouglas", event.getHost());
            final StructuredSyslogMessage msg = event.getStructuredMessage();
            assertNull(msg.getStructuredData());
        }
    }

[xiaoleihuang]

Hi, @joschi
Thank you. But here as your official website claims using "GRAYLOGRFC5424" in http://docs.graylog.org/en/1.0/pages/sending_data.html#rsyslog

Actually, this did not work well. Maybe here is good one : http://www.rsyslog.com/doc/v8-stable/configuration/templates.html#reserved-template-names

The official version provides several default versions: RSYSLOG_SyslogProtocol23Format, RSYSLOG_TraditionalFileFormat, etc.

Am I wrong? What would be a good format template for the StructuredSyslogMessage?

[joschi]

@xiaoleihuang This library should work with the RSYSLOG_SyslogProtocol23Format template.

We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like one. Please post this issue to our public mailing list or join the #graylog channel on freenode IRC.

Thank you!