castaglia/proftpd-mod_kafka

ProFTPD module for logging via Kafka

proftpd-mod_kafka

Status

Synopsis

The mod_kafka module for ProFTPD allows for sending log messages, as JSON, to Kafka brokers using the librdkafka client library.

See the mod_kafka.html documentation for more details.

proftpd-mod_kafka's Issues

logformat failed

KafkaEngine on KafkaLog /var/log/ftpd/kafka.log KafkaBroker 172.0.0.1:9092

LogFormat kafka "%h %l %u %t "%r" %s %b"
KafkaLogOnEvent ALL kafka mytopic

when i start proftpd,there is an error: fatal : kafkaLogOnEvent : no LogFormat 'kafka' configured on line 37 of '/usr/local/proftpd/etc/proftpd.conf' ,is there anything wrong?

Send log messages to Kafka brokers using the librdkafka client library failed!

HI, I configured it according to the mod_kafka documentation mod_kafka.html , but I still cannot send logs to kafka broker. no error message! transfer log is ok!

What is the reason?

Here are the steps:
First, I installed version 1.9.2 of librdkafka

Second, copy the mod_kafka files into: proftpd-dir/contrib/

Then，I did the following
./configure \ --prefix=/usr/local/proftpd \ --sysconfdir=/etc/ --enable-nls --enable-openssl --enable-shadow \ --with-modules=mod_kafka \ --enable-dso --with-shared=mod_kafka \ --with-includes=/usr/local/include/librdkafka \ --with-libraries=/usr/local/lib

make 
make install

The proftpd.conf :

ServerName                     "ProFTPD Default Installation"
ServerType                       standalone
DefaultServer                   on
Port                                   21
UseIPv6                            off
Umask                              022
MaxInstances                  30
User                                  proftp
Group                               proftp
DefaultRoot                     /var/ftp
SystemLog                      /var/log/ftpd/proftpd.log
TransferLog                    /var/log/ftpd/transfer.log
AllowOverwrite              on
RequireValidShell          off
AuthUserFile                  /usr/local/proftpd/ftpd.passwd

<Directory "/var/ftp/*" >
<Limit ALL>
AllowAll
</Limit>
</Directory>

<IfModule mod_kafka.c>
  KafkaEngine on
  KafkaLog /var/log/ftpd/kafka.log
  KafkaBroker 192.168.118.71:9092

  LogFormat kafka "%h %l %u %t \"%r\" %s %b"
  KafkaLogOnEvent ALL kafka first-topic
</IfModule>

Add Kafka broker interactions to unit tests

The scope of this task is to refactor the mod_kafka.c file into a kafka.c file, such that its functions can be unit tested, using a live running Kafka broker in the e.g. CI environment.

Run integrations tests as part of CI workflow

The scope of this task is to run the mod_kafka integration tests as part of the CI workflow. This will require running a Kafka containers within the CI workflow.

This StackOverflow post might be a good starting point for how to get Kafka running in a GitHub workflow.