graylog-labs/syslog4j-graylog2

The syslog4j fork from Graylog2. This is used since 0.10.0. The original code was very "special" and no actions were taken to improve it. This is planned to be replaced completely in future versions.

syslog4j

Syslog4j provides client and server implementations of the BSD Syslog protocol (RFC 3164) and the draft "structured syslog" protocol (RFC Draft).

Warning

This is a repackaged fork used in Graylog, as the original package has no recent versions published to Maven Central.

Some modifications have been applied to fix bugs we ran into.

Original source code from http://syslog4j.org/

License

This project is licensed under the LGPL.

syslog4j-graylog2's Issues

Priority

Hi,

It seems that priority computed in appendPriority (AbstractSyslogMessageProcessor.java line 68:int priority = facility | level;) is not compliant with RFC 3164 (https://tools.ietf.org/html/rfc3164):
"The Priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the Severity"
He is intended? Or maybe i'm missing something...

I didn't find a bug tracker to the original syslog4j project and i don't know if the original project is still active, so I decide to post here.

AbstractSyslogServer needlessly casts to SyslogServerConfigIF to AbstractSyslogServerConfig

I'm working with my own config objects that do not inherit from AbstractSyslogServerConfig. This caused an exception during AbstractSyslogServer.initialize() because my class could not be cast to AbstractSyslogServerConfig even though I implement all of the methods requested.

Its API javadoc?

Is the api doc the same as syslog4j's java doc? Where is its api do Thank you.

Syslog Server fails to decode message facilities

When receiving syslog messages, the message facility is reported wrongly.

RFC 5424

The Priority value is calculated by first multiplying the Facility
number by 8 and then adding the numerical value of the Severity. For
example, a kernel message (Facility=0) with a Severity of Emergency
(Severity=0) would have a Priority value of 0. Also, a "local use 4"
message (Facility=20) with a Severity of Notice (Severity=5) would
have a Priority value of 165.

While the sending part of syslog4j is doing things right, the receiving part is not, and
SyslogUtility fails to properly decode the facilities due to wrong constants in SyslogConstants.

syslog4j-0.9.55, compatibility issue with JNA-4.1

I ran into this issue when I attempted to move from JNA 3.4.0 to 4.1.0.
I suspect we'd have the same issue with the latest release (0.9.60?) as well given that SockAddr imp under org.graylog2.syslog4j.impl.unix.socket.UnixSocketSyslog hasn't changed.

Caused by: java.lang.Error: Structure.getFieldOrder() on class org.graylog2.syslog4j.impl.unix.socket.UnixSocketSyslog$SockAddr does not provide enough names ([]) to match declared fields ([sun_family, sun_path])
at com.sun.jna.Structure.getFields(Structure.java:913)
at com.sun.jna.Structure.deriveLayout(Structure.java:1058)
at com.sun.jna.Structure.calculateSize(Structure.java:982)
at com.sun.jna.Structure.allocateMemory(Structure.java:363)
at com.sun.jna.Structure.ensureAllocated(Structure.java:339)
at com.sun.jna.Structure.ensureAllocated(Structure.java:329)
at com.sun.jna.Structure.size(Structure.java:393)
at org.graylog2.syslog4j.impl.unix.socket.UnixSocketSyslog.connect(UnixSocketSyslog.java:109)
at org.graylog2.syslog4j.impl.unix.socket.UnixSocketSyslog.write(UnixSocketSyslog.java:119)
at org.graylog2.syslog4j.impl.AbstractSyslog.write(AbstractSyslog.java:317)
at org.graylog2.syslog4j.impl.AbstractSyslog.log(AbstractSyslog.java:253)
at org.graylog2.syslog4j.impl.AbstractSyslog.log(AbstractSyslog.java:124)

Support jna 4

Hi,

I tried out the syslog4j with unix-sockets on an arm-linux. Unfortunately the jna 3 impl doesn't support arm. And it was not possible to just change the dependency to jna 4 because the syslog4j class SocketAddr needed a change (only one line - and I didn't really know what I was doing ;)).

@Override
protected List getFieldOrder() {
    return Arrays.asList("sun_family", "sun_path");
}

Without the change there was the following error:

java.lang.Error: Structure.getFieldOrder() on class org.graylog2.syslog4j.impl.unix.socket.UnixSocketSyslog$SockAddr does not provide enough names [0] ([]) to match declared fields [2] ([sun_family, sun_path])
	at com.sun.jna.Structure.getFields(Structure.java:1015)
	at com.sun.jna.Structure.deriveLayout(Structure.java:1172)
	at com.sun.jna.Structure.calculateSize(Structure.java:1097)
	at com.sun.jna.Structure.allocateMemory(Structure.java:390)
	at com.sun.jna.Structure.ensureAllocated(Structure.java:366)
	at com.sun.jna.Structure.ensureAllocated(Structure.java:356)
	at com.sun.jna.Structure.size(Structure.java:423)
	at org.graylog2.syslog4j.impl.unix.socket.UnixSocketSyslog.connect(UnixSocketSyslog.java:109)

Are there any plans to support jna 4?

Best regards
Lars

spring ： no cipher suites in common

the StringIndexOutOfBoundsException of SyslogServerEvent caused by empty byte[] can cause DatagramSocket does not work any more

When client send empty byte[] to UDPNetSyslogServer, create a SyslogServerEvent, now parse error,generate a StringIndexOutOfBoundsException,but not captured,so the run method of UDPNetSyslogServer over,does not work any more.

This bug also exists the original source code of Syslog4j,only exists in UDPNetSyslogServer,not in TCPNetSyslogServer.

This bug can be very simple resolved,in UDPNetSyslogServer just judge that the length of receiveData is greater than 0.if Less than or equal to 0,just ignore.

Update Maven Central with latest code base / artifact

Maven central is out of sync with the GitHub repo.

Here we can see that changes have been made in 2019 but Maven central is still on a release that is 2 years old. Can a new SNAPSHOT, or release be deployed on maven ?

Thanks

TCP mode support probe and interval parameters configuration.

Currently, only the keepalive function is enabled or disabled, but the probe and interval parameters are not supported.
In many Internet scenarios, the default interval of keepalive in the Linux kernel is too large, which will cause firewalls and other devices to disconnect.

#define TCP_KEEPALIVE_PROBES 9 /* Max of 9 keepalive probes /
#define TCP_KEEPALIVE_INTVL (75HZ)

Not all Syslog FACILITY levels can be specified

SyslogConstants.java:

The following Facility level constants are missing and therefore cannot be specified:

      12             NTP subsystem
      13             log audit (note 1)
      14             log alert (note 1)
      15             clock daemon (note 2)

See RFC3164, 4.1.1 PRI Part. This means that currently not all facility levels can be specified for sending a message.

Problems parsing RFC3164 messages with ISO8601 timestamps

Summary

Syslog4j's SyslogServerEvent mishandles older RFC3164-style syslog messages that contain an ISO8601 timestamp. Example message that causes unwanted behavior:

<6>2016-10-12T14:10:18Z hostname testmsg[20]: {"cf_origin":"firehose","deployment":"cf","event_type":"ValueMetric","ip":"127.0.0.1","job_index":"0","level_mod":"6","msg":"","name":"RequestLatency","origin":"bbs","time":"2016-10-12T14:10:18Z","unit":"nanos","value":2.6071897e+07}"

Expected Behavior

Section 5.1 of RFC3164 says that implementers may wish to use an ISO8601 timestamp, so the expected behavior is that this is parsed out properly.

Current Behavior

Creating a SyslogServerEvent() with the above message will cause the timestamp to get replaced with Date(), and the rest of the message to be improperly parsed.

Possible Solution

In SyslogServerEvent, import org.joda.time.DateTime and change the parseDate() function to the following:

protected void parseDate() {
        int datelength = 16;
        String dateFormatS = DATE_FORMAT;
        boolean isDate8601 = false;

        if (this.message.length() > datelength) {

            // http://jira.graylog2.org/browse/SERVER-287
            if (this.message.charAt(5) == ' ') {
                datelength = 15;
                dateFormatS = DATE_FORMAT_S;
            }

            if (Character.isDigit(this.message.charAt(0))) {
                datelength = this.message.indexOf(' ') + 1;
                isDate8601 = true;
            }

            String year = Integer.toString(Calendar.getInstance().get(Calendar.YEAR));
            String originalDate = this.message.substring(0, datelength - 1);
            String modifiedDate = originalDate + " " + year;

            DateFormat dateFormat = new SimpleDateFormat(dateFormatS, Locale.ENGLISH);
            try {
                if (!isDate8601) {
                    this.date = dateFormat.parse(modifiedDate);
                } else {
                    this.date = DateTime.parse(originalDate).toDate();
                }

                this.message = this.message.substring(datelength);

            } catch (ParseException pe) {
                this.date = new Date();
            }
        }

        parseHost();
    }

This will ensure that the timestamp is parsed properly and the parsing continues on to parse the host properly.

Steps to Reproduce (for bugs)

1. Run Graylog 2.1.1 and set up a Syslog TCP stream
2. Save the above message to a file, let's call it testmsg
3. cat testmsg|nc grayloghost port
4. Check Graylog, find mangled fields in the message.

Context

This was discovered when trying to use Graylog to collect stats and other messages from Pivotal Cloud Foundry using the firehose-to-syslog nozzle provided to funnel stats into syslog messages. Due to the timestamp being ISO8601, syslog4j replaces it with the time the message was parsed with SyslogServerEvent.parseDate()

Your Environment

• Graylog Version: 2.1.1
• Elasticsearch Version: 2.3.3
• MongoDB Version: 3.2.6
• Operating System: RHEL 7.2
• Browser version: Chrome 52

Layout can be configure？

org.apache.log4j.PatternLayout

message ident not showing up in syslogd

I was wondering what exactly needs to be configured to identify the application to syslogd? I tried below information:

syslog = Syslog.getInstance("unix_syslog");
config = syslog.getConfig();
config.setFacility(SyslogConstants.FACILITY_USER);
config.setThrowExceptionOnWrite(false);
config.setIdent("myapp");
config.setIncludeIdentInMessageModifier(true);

But a syslog.info("test me") produces in /var/log/messages:

Jun  6 14:50:06 api02 -Xmx512m: test me

where I would expect:

Jun  6 14:50:06 api02 myapp: test me

Does anyone know how to configure this? Could not find anything in the official documentation.

Caused by: java.security.AccessControlException: access denied ("java.lang.reflect.ReflectPermission" "newProxyInPackage.org.productivity.java.syslog4j.impl.unix")

i use
grant {
permission java.net.SocketPermission "*", "listen,connect,accept,resolve";

permission java.lang.reflect.ReflectPermission "newProxyInPackage.org.graylog2.syslog4j.impl.unix";
};
but not work

Why is it that the log synchronization IP is cached so that the last IP will be used again

I want to know how to turn off this cache, I don't need it, and I don't think most people need it, either.

log4j layouts should be supported

according to log4j's docs if an appender returns falls for an appenders #requiresLayout method layouts are always ignore. log4j's native syslog appenders supports layouts but this one does not. Was there a reason for this?

If the UDP message sent exceeds 65507, it will be sent repeatedly in a life and death cycle

code

syslog4j-graylog2/src/main/java/org/graylog2/syslog4j/impl/net/udp/UDPNetSyslog.java

Line 77 in 374bc20

if (attempts == (this.netSyslogConfig.getWriteRetries() + 1)) {

end

An exception occurs when the UDP message is greater than 65507.
The metering parameter "attempts" will not accumulate, resulting in failure to jump out of the cycle and finally produce the life and death cycle.

UDP Server Impl fails silently

If for some reason the Syslog Server is not able to start, for example by missing permissions on address binding, there's no way to detect it, since the UDP implementation "swallows" the exception.

See https://github.com/graylog-labs/syslog4j-graylog2/blob/master/src/main/java/org/graylog2/syslog4j/server/impl/net/udp/UDPNetSyslogServer.java#L66.

Is there a reason behind this design choice?

TCP syslog server thread exited.

Hi,

I observed in one of my setup that syslog tcp server thread exited.

I see the following output for "ss -t -a" command,
State Recv-Q Send-Q Local Address:Port Peer Address:Port
SYN-RECV 0 0 169.254.14.135%if372383026:514 169.254.14.129:43650
SYN-RECV 0 0 169.254.14.135%if372383037:514 10.142.194.8:54013
SYN-RECV 0 0 169.254.14.135%if372383037:1514 10.142.194.33:17801
SYN-RECV 0 0 169.254.14.135%if372383037:1514 10.142.194.46:22192
SYN-RECV 0 0 169.254.14.135%if372383037:1514 10.142.194.45:41966

Local Address - 169.254.14.135%if372383037 is in unusual format, not sure whether this a standard format, can this cause tcp server thread to exit?

Do we need to handle Exception in -

syslog4j-graylog2/src/main/java/org/graylog2/syslog4j/server/impl/net/tcp/TCPNetSyslogServer.java

Line 189 in 374bc20

public void run() {

Thank you.

StructuredSyslogMessage fromString() can not parse rsyslog data

Syslog message from rsyslog can not be parsed. The sample is below:

Jun 20 11:59:57 myname kernel: [  357.266774] [UFW BLOCK] IN=enp9s0 OUT=enp9s0 MAC=01:2e:12:49:87:2b:01:36:1b:38:ad:80:08:50 SRC=289.15.121.109 DST=110.67.112.10 LEN=52 TOS=0x00 PREC=0x00 TTL=47 ID=43803 DF PROTO=TCP SPT=39693 DPT=23

It complains by emitting the SyslogRuntimeException: "UnixSyslog not supported on non-Unix platforms" on OpenBSD.

Preamble:

1. The target OS to build and run is OpenBSD/amd64 6.3.
2. The application utilizing the syslog4j package is written in Clojure and the affected module is accessible here.
3. This application has been tested on Ubuntu Server (Ubuntu 16.04.6 LTS x86-64) as well as on Arch Linux (kernel 5.0.2-arch1-1-ARCH x86-64), and I claim – regarding using the UnixSyslog facility, – it is working well: i.e. it is opening, writing to, and closing the system logger as expected, without any exceptions or errors when running on the aforementioned OSes.

Actual behaviour when using the UnixSyslog facility on OpenBSD:

When calling its constructor – UnixSyslog – it produces the org.graylog2.syslog4j.SyslogRuntimeException:

Exception in thread "main" org.graylog2.syslog4j.SyslogRuntimeException: UnixSyslog not supported on non-Unix platforms, compiling:(/home/<username>/dnsresolvd-multilang/src/clojure/./dnsresolvd:171:3)
        at clojure.lang.Compiler.load(Compiler.java:7391)
        at clojure.lang.Compiler.loadFile(Compiler.java:7317)
        at clojure.main$load_script.invokeStatic(main.clj:275)
        at clojure.main$script_opt.invokeStatic(main.clj:335)
        at clojure.main$script_opt.invoke(main.clj:330)
        at clojure

Expected behaviour when using the UnixSyslog facility on OpenBSD:

Do not emit the SyslogRuntimeException exception, but simply open the system logger and write to it wherever indicated in the application code (just like it is tested on Ubuntu Server and Arch Linux, stated above).

Quick investigation for the root cause:

It seems that this exception is caused by the following piece of code:

        if (!OSDetectUtility.isUnix()) {
            throw new SyslogRuntimeException("UnixSyslog not supported on non-Unix platforms");
        }

OpenBSD is in fact a Unix-like operating system. But the OSDetectUtility.isUnix() method doesn't know anything about that. 👈

Messed up message ident while using UnixSyslog

While using UnixSyslog implementation of SyslogIF, after some time, syslog message ident will become corrupted. Fortunately, I was able to reproduce the problem on my system (ubuntu 16.04, default-jdk). The message ident usually become corrupted after some memory allocation:

syslog = Syslog.getInstance(SyslogConstants.UNIX_SYSLOG);
SyslogConfigIF config = (UnixSyslogConfig) syslog.getConfig();
config.setIdent("prog_name");        
syslog.error("here, syslog message ident is correct");
// some random memory alloction
String a = new String();
for (int i = 0; i < 100000; i++) {
    a += "a";
}
syslog.error("here, syslog message ident is corrupted");