brg-liuwei/ngx_kafka_module

nginx kafka module, send post log data to kafka cluster

ngx_kafka_module's Issues

data is lost

with fast rate the data sended to kakfa will be lost

I need send http head params in kafak option message

I need send http head params in kafak option message
same time nedd some head parms

Nginx to kafka over socket

Hi,
I am trying to get data into kafka via nginx ,the producer plugin to kafka uses librdkafka which writes to kafka over socket.How doI redirect it to my local kafka via the nginx proxy when I don't have location directive ??

Unable to bring up nginx after adding the kafka module

Hi
I tried bringing up nginx after adding the kafka module but am getting the below error when I tried testing the config file:
[root@smsc ngx_kafka_module]# /usr/sbin/nginx -t
nginx: [emerg] unknown directive "kafka" in /etc/nginx/nginx.conf:37
nginx: configuration file /etc/nginx/nginx.conf test failed.
my Nginx version is 1.2.4
RHEL 6.4
attaching my config file as well.
nginx.txt

nginx 启动报错

hi~

我的nginx版本: 1.18.0

依次下载并编译 librdkafka、ngx_kafka_module、nginx，配置 nginx.conf 后，启动 nginx 报错如下：

%5|1634039250.497|CONFWARN|rdkafka#producer-1| [thrd:app]: No bootstrap.servers configured: client will not be able to connect to Kafka cluster

网上找了一段时间，除了有提到版本不匹配的，没有找到其他原因，因为该 nginx 版本是公司内通用的，所以没办法降版本号

Add support for using SSL with librdkafka

Please add configuration options to use SSL encrypted connections to backend Kafka as described in https://github.com/edenhill/librdkafka/wiki/Using-SSL-with-librdkafka.

rd_kafka_poll before rd_kafka_destroy

Hi,

Thank you so much for your cool module!

I have one suggestion about ngx_http_kafka_exit_worker.
https://github.com/brg-liuwei/ngx_kafka_module/blob/master/ngx_http_kafka_module.c#L386

In my opinion, rd_kafka_poll should be done until rd_kafka_outq_len(main_conf->rk) > 0, in order to ensure kafka messages to be delivered before exit.

rd_kafka_poll(main_conf->rk, 0);

while (rd_kafka_outq_len(main_conf->rk) > 0) {
    rd_kafka_poll(main_conf->rk, 100);
}

// TODO: rd_kafka_topic_destroy(each loc conf rkt );
rd_kafka_destroy(main_conf->rk);

• C example code in librdkafka repository
  https://github.com/edenhill/librdkafka/blob/master/examples/rdkafka_example.c#L619

No errors when kafka_broker_list is unavailable

I've been testing with this module and noticed that when no broker(s) in the kafka_broker_list are available, there is no indication/error in nginx. I just receive a 204 as if sending to a broker/topic was successful.

Make librdkafka message buffer size configurable to handle large messages

DESCRIPTION

ngx_kafka_module currently cannot handle large messages.

REPRODUCE PROCEDURE

Anyway, POST a large message using ngx_kafka_module.
In case of me, I used a 6MB bynary message that contains a movie file.

ANALYSYS and SUGGESTED RESOLUTION

At the moment, there is a consideration to handle a large message storing as files at:

https://github.com/brg-liuwei/ngx_kafka_module/blob/master/ngx_http_kafka_module.c#L413

But this is not implemented yet.

Here, it's also possible to workaround the large message handling
by configuring message buffer size by specifying 'client_body_buffer_size' like below:

        location = /api/test-topic {
             kafka_topic test-topic;
             client_max_body_size 1g;
             client_body_buffer_size 64m;
        }

Here, the problem is that we need to increase librdkafka layer buffer size
before initializing librdkafka by calling 'rd_kafka_new()' at:

https://github.com/brg-liuwei/ngx_kafka_module/blob/master/ngx_http_kafka_module.c#L500

Koutaro (@tsukada-ascade) has already written a patch for this enhancement.

https://github.com/tsukada-ascade/ngx_kafka_module/tree/feature/configurable-message-max-bytes

His patch introduces a new configuration parameter 'kafka_message_max_bytes' and
enables adjusting the desired buffer size so that we can reduce memory consumption.

I asked him to create a PR, and I hope this is useful for the community.

NGINX common variables not passed to kafka topic

Hi,

Would it be possible to pass some extra NGINX common variables to the data sent to the Kafka topic?

Currently, only the data of theNGINX variable $request_body is passed but , for instance, data from variables like $remote_addr would be great .

The goal is to have the IP of the client that is sending the data.

希望提供docker 镜像

When making librdkafka, there is a warning

WARNING: librdkafka-static.a: Not creating self-contained static library librdkafka-static.a: no static libraries available/enabled

Please tag the release

Would you be so kind to tag the release version of this project?

I'm starting to use this in our organization (https://hub.docker.com/r/salekseev/nginx/) and it would be much easier to refer to version nymbers instead of git commit hashes when building this.

Thanks.

/ngx_kafka_module/config: line 13: syntax error: unexpected end of file

Hi. When I run the following cmd
./configure --add-module=/ngx_kafka_module
the result told me the error like this
checking for getaddrinfo() ... found configuring additional modules adding module in /ngx_kafka_module /ngx_kafka_module/config: line 13: syntax error: unexpected end of file

I was following the README to install the modules, how can I make it right?

cannot handle in-file-post-buf

2022/11/03 16:03:28 [error] 4213#0: *285880 ngx_http_kafka_handler cannot handle in-file-post-buf, client: 192.168.1.244, server: localhost, request: "POST /reportlog HTTP/1.0", host: "log.XXXX.com.cn:1000"
2022/11/03 16:03:59 [error] 4217#0: *291578 ngx_http_kafka_handler cannot handle in-file-post-buf, client: 192.168.1.244, server: localhost, request: "POST /reportlog HTTP/1.0", host: "log.XXXX.com.cn:1000"
2022/11/03 16:04:25 [error] 4213#0: *291598 ngx_http_kafka_handler cannot handle in-file-post-buf, client: 192.168.1.244, server: localhost, request: "POST /reportlog HTTP/1.0", host: "log.XXXX.com.cn:1000"
这是个什么错，什么情况出现这个错误

Unable to execute ./configure --add-module=/path/to/ngx_kafka_module

Hi,

I am unable to execute following command in nginx
./configure --add-module=/Users/kunalgupta/Desktop/Backup/Kunal/Projects/DataScience/ngx_kafka_module

I am using mac OS X yosemite. I have installed nginx from following url http://www.robertmulley.com/tutorial/nginx-install-and-setup-mac-os-x-yosemite/

Please help me how will i compile your module in nginx directory.

Missing check for NULL

Hello,
I noticed there's no check for ngx_pcalloc possibly returning NULL at line 466.
Hope this helps...

nginx integrates Kafka Kafka consumers don't get the data

In the test,crul /localhsot/kafka/ bike-d "this is a test message" and no data was received from kafka consumers

Kafka Partition Key Routing

I've got basic kafka message routing working but I have a requirement to route messages to a partition based on a routing key. Is there a way to achieve this currently?

Options to Set Kafka Producer Configs

Is there any support to tweak the producer configurations. Kafka Producers has many tweakable parameters like
compression, max_retries, retry_backoff_ms, required_acks, ack_timeout_ms, max_queued_messages, min_queued_messages, linger_ms block_on_queue_full, max_request_size, sync, delivery_reports

352a507c78da1dea6dd42a5867d3c2cc

Return Statements not supported

When I add a return clause inside the location below the topic_name. I am not able to push the message into the kafka broker.

请完善从nginx临时文件读取数据的TODO部分代码

Add license information

Please add a license file to the project so users of the code can know if it is legal to use in their software, projects, etc. (and protect yourself from liability).

GitHub has a guide to help choose and add a license: https://help.github.com/articles/open-source-licensing/

If you would like more information about choosing a license: http://choosealicense.com/

If it is open source some examples might be:
Apache http://choosealicense.com/licenses/apache-2.0/
swift uses apache 2.0

MIT http://choosealicense.com/licenses/mit/
dotnet core uses MIT license