Comments (18)
@adridi http headers are actually the ngx variables (ngx.req.get_headers()) I'm injecting into the $request_body. Since they are just another ngx variable, I don't see why they couldn't technically be included. The bigger issue with including ngx variables into a kafka payload in some standard way is that kafka messages have no standard structure. It's just a text payload.
I prefer JSON so I make sure the incoming $request_body is JSON, I inject the ngx variables into the appropriate JSON key in $request_body, pass the resulting text payload of $request_body to the appropriate kafka topic using this module, and then parse that JSON structure on the other end with my kafka consumer. This works great for me but what if someone doesn't want to, or can't, use JSON... now the complexity of implementing this support in this module explodes.
from ngx_kafka_module.
I have add this issue into TODO LIST.
from ngx_kafka_module.
nginx common var is very usefull
from ngx_kafka_module.
@lifehacking yep, I cannot agree more. PR is welcomed :)
from ngx_kafka_module.
If anyone here is looking for a workaround in the interim, I am successfully injecting nginx variables into the $request_body using the lua nginx module.
from ngx_kafka_module.
@dkinon Yes, using lua_nginx_module injecting ngx variables into $request_body is easy to work, and I am an openresty user. But I still want to do some works to pass variables to kafka topic without lua module, cause not everyone use lua_nginx_module. :)
from ngx_kafka_module.
what about http headers - would it be possible to add them to the message body?
from ngx_kafka_module.
@dkinon Totally Agree with your statement! I was referring to the existing module, I believe it should have set kafka message as = request header + request body. Both are equally important.
from ngx_kafka_module.
@adridi I was also referring to the existing module, I merely used my lua_ngx module pseudocode example to illustrate one of the many formatted ways this could be done within the confines of the module. The larger point of my previous statement was discussing what format the headers in $request_header + $request_body would look like. Whatever the format, we wouldn't want this to be the default as it would definitely break downstream consumers as they exist today. I'm curious if anyone has suggestions on what those headers would look like?
from ngx_kafka_module.
When I was testing,crul localhost/ kafka-d "this is a test data",kafka consumers could not accept the data
from ngx_kafka_module.
当我测试时,crul localhost / kafka-d“这是一个测试数据”,kafka消费者无法接受数据
kafka and nginx No errors were reported
from ngx_kafka_module.
from ngx_kafka_module.
当我测试时,crul localhost / kafka-d“这是一个测试数据”,kafka消费者无法接受数据
kafka and nginx No errors were reported
哦,麻烦贴一下你的nginx配置文件
from ngx_kafka_module.
当我测试时,crul localhost / kafka-d“这是一个测试数据”,kafka消费者无法接受数据
kafka和nginx没有报错哦,麻烦贴一下你的nginx的配置文件
稍等
from ngx_kafka_module.
当我测试时,crul localhost / kafka-d“这是一个测试数据”,kafka消费者无法接受数据
kafka和nginx没有报错哦,麻烦贴一下你的nginx的配置文件
#user nobody;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
kafka;
kafka_broker_list jaydon-hadoop2:9092jaydon-hadoop3:9092jaydon-hadoop4:9092;
server {
listen 80;
server_name node-6.xiaoniu.com;
#charset koi8-r;
#access_log logs/host.access.log main;
location = /kafka/bike {
kafka_topic bike;
}
location = /kafka/user {
kafka_topic user;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
from ngx_kafka_module.
broker_list每个节点之间要以逗号分隔:
kafka_broker_list jaydon-hadoop2:9092,jaydon-hadoop3:9092,jaydon-hadoop4:9092;
请确保主机名jaydon-hadoopX
在你的nginx上可以被访问,如果你不确定的话,可以换成IPv4地址试试。
另外以后有问题请新建一个issue进行提问,本issue的主题和你提的问题无关,在这个issue下沟通会打扰到别人,谢谢。
from ngx_kafka_module.
from ngx_kafka_module.
broker_list每个节点之间要以逗号分隔:
kafka_broker_list jaydon-hadoop2:9092,jaydon-hadoop3:9092,jaydon-hadoop4:9092;
请中主机确保名
jaydon-hadoopX
在你的nginx的上可以被访问,如果你不确定的话,可以换成IPv4的地址试试。另外以后有问题请新建一个问题进行提问,本课题的主题和你提的问题无关,在这个问题下沟通会打扰到别人,谢谢。
Okay, I'm sorry
from ngx_kafka_module.
Related Issues (20)
- Unable to bring up nginx after adding the kafka module HOT 18
- Return Statements not supported HOT 1
- Nginx to kafka over socket HOT 1
- Add support for using SSL with librdkafka HOT 1
- Options to Set Kafka Producer Configs HOT 1
- No errors when kafka_broker_list is unavailable HOT 1
- Kafka Partition Key Routing HOT 6
- 352a507c78da1dea6dd42a5867d3c2cc
- Missing check for NULL HOT 1
- data is lost HOT 3
- nginx integrates Kafka Kafka consumers don't get the data HOT 1
- I need send http head params in kafak option message
- When making librdkafka, there is a warning
- /ngx_kafka_module/config: line 13: syntax error: unexpected end of file
- nginx 启动报错 HOT 3
- 请完善从nginx临时文件读取数据的TODO部分代码 HOT 1
- Make librdkafka message buffer size configurable to handle large messages HOT 1
- 希望提供docker 镜像
- cannot handle in-file-post-buf
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ngx_kafka_module.