ilanddev/syslogng_kafka

syslogng_kafka provides a Python module for syslog-ng 3.7 allowing one to filter and forward syslog messages to Apache Kafka brokers.

syslog-ng-mod-python Apache Kafka destination

syslogng_kafka provides a Python module for syslog-ng 3.7 allowing one to filter and forward syslog messages to Apache Kafka brokers.

The implementation leverages confluent-kafka which uses the awesome librdkafka library providing reliability and high performance.

Please read the doc as in most cases a `pip install` won't work as they are particular requirements that are currently not met by mainstream Linux distributions.

syslogng_kafka's Issues

Memory Leak

syslog-ng Kafka driver = latest.
Python version = 2.7.12
Operating System = Ubuntu 16.10

I am just sending simple messages to the syslog.

Huge memory leak. About 1MB every second

wrong json format produced message with single quote

Description

When i am using syslog-ng and syslogng-kafka to push system logs to kafka Server. I notice that the messages are not json format, messages use single quote (which should be double-quotes ) and can not be recognized and parsed by logstash

reproduce

I use syslog-ng -F to debug and the logs be produced like this

1 2018-05-30T13:57:14.721616+08:00 www.test.com syslogng_kafka 32288 - - Message produced: {'FACILITY': 'cron', 'PRIORITY': 'info', 'HOST': 'www.test.com', 'PROGRAM': 'CROND', 'DATE': '1527659821', 'MESSAGE': '(root) CMD (/usr/lib64/sa/sa1 1 1)'}

On logstash server i received the _jsonparsefailure error

syslog-ng conf

source s_sys {
    file ("/proc/kmsg" program_override("kernel: "));
    unix-stream ("/dev/log" max-connections(512));
    internal();
};
destination syslog_to_kafka {
    python(
        class("syslogng_kafka.kafkadriver.KafkaDestination")
            on-error("fallback-to-string")
            options(
                hosts("xxx:9092")
                topic("system-log")
                verbose("True")
                display_stats("True")
                )
    );
};
log {
    source(s_sys);
    destination(syslog_to_kafka);
};

logstash conf

input {
  kafka {
    bootstrap_servers => "xxxx:9092"
    topics => ["system-log"]
    codec => json
  }
}
......

Error parsing python, syntax error, unexpected '(' in /etc/syslog-ng/conf.d/kafka.conf

Ubuntu 18.04
Syslog-ng 3.29 installed via pip3

I am using the kafka.conf destination configuration in these instructions: https://syslogng-kafka.readthedocs.io/en/latest/installation.html and getting the following error when running sudo syslog-ng -F:

Error parsing python, syntax error, unexpected '(' in /etc/syslog-ng/conf.d/kafka.conf:6:22-6:23:
1       destination syslog_to_kafka {
2           python(
3               class("syslogng_kafka.kafkadriver.KafkaDestination")
4                   on-error("fallback-to-string")
5                   options(
6----->                 hosts("localhost:9092,localhost:9182")
6----->                      ^
7                       topic("syslog")
8                       partition("10")
9                       msg_key("src_ip")
10                      programs("firewall,nat")

destination syslog_to_kafka {
11                      broker_version("0.8.2.1")

Included from /etc/syslog-ng/syslog-ng.conf:163:1-163:1:

I have tried cutting and pasting and also entering everything by hand to no avail.