A Kong plugin for propogating zipkin spans and reporting spans to a zipkin server - this plugin has been moved into https://github.com/Kong/kong, please open issues and PRs in that repo
License: Apache License 2.0
The plugin will remove existing x-B3 headers if they exist on the incoming request: https://github.com/Kong/kong-plugin-zipkin/blob/master/kong/plugins/zipkin/codec.lua#L88-L93
Users might want end to end traceability from the browser to the API. Therefore we would like to allow existing headers to be honored and any that are not present to be created by the plugin.
Maybe any x-b3 hearers that are included will be overwritten or removed unless the parentspanid header is present?
When the client cancels a pending request. I got error in kong log:
*418 failed to run log_by_lua*: .../local/share/lua/5.1/kong/plugins/zipkin/opentracing.lua:189: attempt to index field 'header_filter_span' (a nil value)
We are using Jaeger for collecting trace-information from our services and have just installed the zipkin-plugin for Kong (v1.0.3). However, the serviceName for kong is missing, and instead unknown-service-name is presented, as seen in the screenshot below.
As of my understanding, Jeager takes the serviceName from the localEndpoint, which works fine for the kong.proxy span that can be seen in the screenshot above and in reporter.lua#L42-L54.
#12 resolved the above problem in cases where the the peer.service exists, but I expected that a default value for the serviceName would be set to kong in other cases.
Maybe this is a duplicate of #23 and if that is the case we could continue the discussion in that issue?
Update
This is how Jaeger determines the serviceName from a zipkin span:
to_domain.go#L206-L233
I have a lot of kong clusters. The plugin (version 1.2.0) has the localEndpoint.serviceName hard-coded as "kong", making it difficult to understand which "kong cluster" handled the request.
So far, I forked and coded the plugin to let me customize the tag lc (component). I think the serviceName would be better tho.
My apologies if this is expected behaviour 🎩
My setup:
gateway-service-ahttp://service-a:3000,kong-gatewayzipkin-instrumentation-koa (traces incoming HTTP requests)service-aDependency, which would be created on http request, looks like:
gateway-service-a ----- service-a
So we can add more services, and we would get:
gateway-service-a ----- service-a
gateway-service-b ----- service-b
Goes to this pattern:
{service-name-on-kong} ----- {localServiceName}
What I would expect in graph (both services are connect to Kong, not services on Kong):
kong-gateway ------ service-a
|
service-b
Or even better:
kong-gateway ------ gateway-service-a ------ service-a
|
gateway-service-b
|
service-b
Even trace looks like this in UI:
kong-gateway directly to service-a. You have gateway-service-a when you would open service-a span
So the questing is should it be
{service-name-on-kong} ----- {localServiceName}
or
{kong-gateway} ----- {localServiceName}
or
{kong-gateway} ----- {service-name-on-kong} ----- {localServiceName}
?
Thank you!
Lua stacktrace:
2018/12/12 16:04:04 [error] 238#0: *2313102 failed to run log_by_lua*: .../local/share/lua/5.1/kong/plugins/zipkin/opentracing.lua:174: attempt to perform arithmetic on field 'KONG_ACCESS_ENDED_AT' (a nil value)
stack traceback:
.../local/share/lua/5.1/kong/plugins/zipkin/opentracing.lua:174: in function 'log'
/usr/local/share/lua/5.1/kong/plugins/zipkin/handler.lua:34: in function 'log'
/usr/local/share/lua/5.1/kong/init.lua:529: in function 'log'
log_by_lua(nginx.conf:150):2: in function <log_by_lua(nginx.conf:150):1> while logging request, client: 10.119.116.11, server: kong, request: "GET /F5/status HTTP/1.1", host: "edge-kong-gateway-stage.****-***-***.company.com"
Just need to anticipate that field being null occasionally I suppose and maybe set that value to -1 or 0 of some sort(HTTP Log plugin takes a similar approach on endpoints that don't actually route or fail to proxy).
I get Error when use plugin:
2018/07/05 04:27:55 [error] 42#0: *378 failed to run log_by_lua*: /usr/local/share/lua/5.1/opentracing/span.lua:46: span already finished
stack traceback:
[C]: in function 'assert'
/usr/local/share/lua/5.1/opentracing/span.lua:46: in function 'finish'
.../local/share/lua/5.1/kong/plugins/zipkin/opentracing.lua:189: in function 'log'
/usr/local/share/lua/5.1/kong/plugins/zipkin/handler.lua:34: in function 'log'
/usr/local/share/lua/5.1/kong/init.lua:528: in function 'log'
commit: ccb004a
As discussed on openzipkin/b3-propagation#21 and first implemented here: https://github.com/openzipkin/brave/blob/master/brave/src/main/java/brave/propagation/B3SingleFormat.java https://github.com/openzipkin/brave/blob/master/brave/src/test/java/brave/propagation/B3SingleFormatTest.java
Let's support at least reading "b3" header from a single string, most commonly traceid-spanid-1
It would also be nice to support optionally writing this, especially in message providers or others with constrained environments.
Brave currently has a property like this, but its name could change with feedback:
/**
* When true, only writes a single {@link B3SingleFormat b3 header} for outbound propagation.
*
* <p>Use this to reduce overhead. Note: normal {@link Tracing#propagation()} is used to parse
* incoming headers. The implementation must be able to read "b3" headers.
*/
public Builder b3SingleFormat(boolean b3SingleFormat) {
this.b3SingleFormat = b3SingleFormat;
return this;
}No service name in UI Jaeger with spans from kong-plugin-zipkin.
Screenshot:
Hello, guys,
Title says it all pretty much. Not sure if this issue is with the plugin (since the config is calculated by kong), but other plugins seem to prioritize configuration values correctly.
So, having config like this
_format_version: "2.1"
_transform: true
services:
- name: echo
url: 'https://httpbin.org'
routes:
- name: echo
paths:
- /echo
plugins:
- name: zipkin
config:
"http_endpoint": "http://jaeger-collector:9411/api/v2/spans"
"sample_ratio": 1
plugins:
- name: zipkin
config:
"http_endpoint": "http://jaeger-collector:9411/api/v2/spans"
"sample_ratio": 0
will leave without any traces from requests to /echo.
... and with which versions of Kong is it compatible?
Thx for you work!
We would like to be able to inject dynamically tags based on the request headers, i.e. be able to configure the plugin to select pre-defined request headers and inject them as tags inside spans.
Does this sound to you as a neat feature?
NOTE It will probably look like the following PR: feat(handler) static tags #84
I'm using Zipkin plugin connect to Zipkin Server but I didn't receive any metrics if I apply on Services or Routes but It works as a Global plugin. My configs are shown as bellow:
on Services - Do not work
[root@localhost]# http GET http://127.0.0.1:8001/services/abc-staging/plugins/188035bc-8e2e-4117-a4da-22e697403f8b
{
"config": {
"default_service_name": "local-zipkin",
"http_endpoint": "http://127.0.0.1:9411/api/v2/spans",
"include_credential": true,
"sample_ratio": 0.001
},
"consumer": null,
"created_at": 1570413367,
"enabled": true,
"id": "188035bc-8e2e-4117-a4da-22e697403f8b",
"name": "zipkin",
"protocols": [
"grpc",
"grpcs",
"http",
"https"
],
"route": null,
"run_on": "all",
"service": {
"id": "163041c1-ee14-4f21-aceb-dccf6c8391e9"
},
"tags": null
}on Routes - Do not work
[root@localhost]# http GET http://127.0.0.1:8001/routes/969d31ae-92cd-489c-a14f-d27e5a230b84/plugins/9fc6e9d3-5014-40d2-a76a-7af1e8c54273
{
"config": {
"default_service_name": "local-zipkin",
"http_endpoint": "http://127.0.0.1:9411/api/v2/spans",
"include_credential": false,
"sample_ratio": 1
},
"consumer": null,
"created_at": 1570417684,
"enabled": true,
"id": "9fc6e9d3-5014-40d2-a76a-7af1e8c54273",
"name": "zipkin",
"protocols": [
"grpc",
"grpcs",
"http",
"https"
],
"route": {
"id": "969d31ae-92cd-489c-a14f-d27e5a230b84"
},
"run_on": "all",
"service": null,
"tags": null
}
By dumping traffic using tcpdump , I can't see anything showing on port 9411
tcpdump -i any -A -s 0 'tcp port 9411 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'as a Global plugin, It works
[root@localhost]# http GET http://127.0.0.1:8001/plugins/7ef31e3d-d983-438c-afd6-32f90ac3c2cf
{
"config": {
"default_service_name": "local-zipkin",
"http_endpoint": "http://127.0.0.1:9411/api/v2/spans",
"include_credential": true,
"sample_ratio": 0.001
},
"consumer": null,
"created_at": 1570177165,
"enabled": false,
"id": "7ef31e3d-d983-438c-afd6-32f90ac3c2cf",
"name": "zipkin",
"protocols": [
"grpc",
"grpcs",
"http",
"https"
],
"route": null,
"run_on": "all",
"service": null,
"tags": null
}tcpdump of Global config
[root@localhost]# tcpdump -n -i any -A -s 0 'tcp port 9411 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
10:55:11.021575 IP 127.0.0.1.35146 > 127.0.0.1.9411: Flags [P.], seq 2898733062:2898733226, ack 1160890218, win 115, length 164
E.....@[email protected]!*w.\*v.*.J$...(.E1.jP..s.2..POST /api/v2/spans HTTP/1.1
User-Agent: lua-resty-http/0.14 (Lua) ngx_lua/10015
content-type: application/json
Content-Length: 1465
Host: 127.0.0.1:9411
................
10:55:11.021582 IP 127.0.0.1.35146 > 127.0.0.1.9411: Flags [P.], seq 0:164, ack 1, win 115, length 164
E.....@[email protected]!*w.\*v.*.J$...(.E1.jP..s.2..POST /api/v2/spans HTTP/1.1
User-Agent: lua-resty-http/0.14 (Lua) ngx_lua/10015
content-type: application/json
Content-Length: 1465
Host: 127.0.0.1:9411I can't find anything related to this problem in /usr/local/kong/logs/, how can I resolve this issue, thanks.
The following commit adds a run_on property inside the config which should really be at the fields level (one level up).
7d6da5f#diff-2ba4b18ab47bc44de697a2d3649d3971R9
This is also an issue with Azure functions plugin (Kong/kong-plugin-azure-functions@6675eed)
I took a look at the tags and i miss the error tag.
The zipkin-ui gitlab page says (source):
How do I make errors visible in yellow or red?
The UI interprets an "error" tag as a failed span, coloring it red. It interprets an annotation containing the substring "error" as a transient failure. To ensure the UI displays errors, please use the error key appropriately.
Hence it would be great if the kong-plugin-zipkin also writes the error tag. Perhaps every response with a http code >= 500 writes an error tag. But what is a good value? The code again?
The image below shows the behaviour of the zipkin-ui:
I was wondering if it would be possible to add a config toggle for including the kong.credential tag in the Zipkin tags?
I think this would be pretty easy to accomplish by adding a includeCredential = bool item in the config then evaluating if that bool was true or not in 219 in the file opentracing.lua.
With high concurrency, zipkin loses data
I specifically see this error printing over and over:
2019/05/29 16:29:56 [error] 36#0: *2801 lua coroutine: runtime error: .../local/share/lua/5.1/kong/plugins/zipkin/opentracing.lua:26: attempt to index field 'conf_to_tracer' (a nil value)
--
| stack traceback:
| coroutine 0:
| .../local/share/lua/5.1/kong/plugins/zipkin/opentracing.lua: in function 'get_tracer'
| .../local/share/lua/5.1/kong/plugins/zipkin/opentracing.lua:48: in function 'initialise_request'
| .../local/share/lua/5.1/kong/plugins/zipkin/opentracing.lua:87: in function 'get_context'
| .../local/share/lua/5.1/kong/plugins/zipkin/opentracing.lua:97: in function <.../local/share/lua/5.1/kong/plugins/zipkin/opentracing.lua:93>
| coroutine 1:
| [C]: in function 'resume'
| coroutine.wrap:21: in function <coroutine.wrap:21>
| /usr/local/share/lua/5.1/kong/init.lua:665: in function 'access'
| access_by_lua(nginx.conf:139):2: in function <access_by_lua(nginx.conf:139):1>, client: 10.xxx.xx.x, server: kong, request: "GET /ping HTTP/1.1", host: "gateway-dev-core.company.com"
Code snippit the nil call fails on:
local tracer = self.conf_to_tracer[conf]
Note the /ping endpoint has the request termination plugin on it and makes it throws a 200 success with http body of Success in kongs standard { message : Successs } json.
Due to the error this endpoint actually stops functioning:
With zipkin errors a flying:
/ $ curl localhost:8000/ping
curl: (52) Empty reply from server
No Zipkin:
/ $ curl localhost:8000/ping
{"message":"Success"}
Was on Kong 1.0.3 prior and it was working fine there.
OS: Docker Kong Alpine 3.6
Kong version: 1.2.0rc1
Zipkin was enabled globally fyi.
Thanks,
Jeremy
Just enabled this bad boy back on our Kong clusters and noticed a heckin bunch of stdout of kong.log warnings:
[warn] 88#0: *465011 [kong] tracing_headers.lua:408 [zipkin] Mismatched header types. conf: w3c. found: b3,
Derives from here:
https://github.com/Kong/kong-plugin-zipkin/blob/master/kong/plugins/zipkin/tracing_headers.lua#L408
Be that we don't really control what clients pass to a gateway, and we wanted to simply enable this plugin globally maybe it makes sense to use this more as a debug statement as you can standardize around how you want your headers and what to expect but clients will do what clients do. Thoughts on changing the log level away from warn, I usually think a warning as an actionable item that requires attention but it really can't be in this case.
I would think:
kong.log.info()
kong.log.debug()
Would be sufficient to help with the discovery of client passing other headers differently than the Zipkin plugin was configured to handle. Otherwise I may just write a patch to comment this line out at live deployments as its just filling our stdout with a log thats more or less undesired as we run kong in notice mode.
Other ideas(just as a side tangent) is this plugin could parse the headers it sees from clients and figure out what trace values the client personally wants to use when logging to the zipkin endpoint as well based on what the clients give it vs setting a "hard" use XYZ value. Not that we wanna use that because we intend to standardize with w3c BUT it would be a neat way to handle it anyways as a setting, maybe a "derive" header from client toggle on/off that will render the other conf hard settings unused and the plugin just goes with the flow of what the client gave it if detected otherwise no headers detected leads to just defaulting to the configured ones. Just an idea.
In order to be future proof, since uber-trace-id is not coded yet, I think issue #22 can be abandoned, and replaced with W3C TraceContext traceparent header implementation.
I am thinking of adding support for extraction of the traceparent header, which doesn't need to be a different plugin:
I would simply code it as "if traceparent found, then use it, if not, revert to B3 headers".
As for the injection propagation, I haven't looked at the code of this plugin yet, but I imagine it is better to be able to configure the output injection format, or worst case "use in output the format you received in input, with a default for traceparent".
My timeline for this is not necessarily very short though, somewhere between end march and june...
Hi ,
2018/10/12 13:15:31 [error] 1000#0: *676002 failed to run log_by_lua*: .../local/share/lua/5.1/kong/plugins/zipkin/opentracing.lua:174: attempt to perform arithmetic on field 'KONG_ACCESS_ENDED_AT' (a nil value)
stack traceback:
.../local/share/lua/5.1/kong/plugins/zipkin/opentracing.lua:174: in function 'log'
/usr/local/share/lua/5.1/kong/plugins/zipkin/handler.lua:34: in function 'log'
/usr/local/share/lua/5.1/kong/init.lua:529: in function 'log'
log_by_lua(nginx-kong.conf:124):2: in function <log_by_lua(nginx-kong.conf:124):1> while logging request, client: 101.81.182.185, server: kong, request:
kong version 0.14.1
I want to trace my request from httpclient, what i do is
but I miss kong span in zipkin dashboard
In our project we use some env variables to inject static tags, e.g. the environment.
Does this sound to you as a neat feature?
Hi. We had to make some changes in our UI to accommodate numerous data problems from Kong. If looking at this pull request openzipkin/zipkin#2834 or our api docs https://zipkin.io/zipkin-api/#/default/post_spans isn't clear what's wrong, please reach out to gitter so we can help increase the usability of this plugin.
When I use Kong Zipkin plugin with Jaeger backend, I got an error with the response body:
Unable to process request body: json: cannot unmarshal number into Go struct field Span.tags of type string
Kong version: 0.14.0rc1 on docker.
Request body of Reporter:
[
{
"traceId": "9aa46f78b4afbd37e6845ba62313d51d",
"localEndpoint": null,
"id": "d4097ab8438f67b0",
"name": "kong.rewrite",
"parentId": "ec270a5446cd8a68",
"remoteEndpoint": null,
"duration": 0,
"timestamp": 1.530259357671e+15
},
{
"traceId": "9aa46f78b4afbd37e6845ba62313d51d",
"localEndpoint": null,
"id": "8a63117b77519823",
"name": "kong.header_filter",
"parentId": "842b5d62df16ba9d",
"remoteEndpoint": null,
"duration": 0,
"timestamp": 1.530259357673e+15
},
{
"traceId": "9aa46f78b4afbd37e6845ba62313d51d",
"localEndpoint": null,
"id": "6188f6111cb62a67",
"name": "kong.access",
"parentId": "842b5d62df16ba9d",
"remoteEndpoint": null,
"duration": 999,
"timestamp": 1.530259357671e+15
},
{
"traceId": "9aa46f78b4afbd37e6845ba62313d51d",
"localEndpoint": null,
"id": "16517979531c5062",
"tags": {
"kong.balancer.try": 1,
"peer.port": 9500,
"peer.ipv4": "10.148.7.194"
},
"name": "kong.balancer",
"parentId": "842b5d62df16ba9d",
"remoteEndpoint": {
"ipv4": "10.148.7.194",
"port": 9500
},
"duration": 0,
"timestamp": 1.530259357672e+15
},
{
"traceId": "9aa46f78b4afbd37e6845ba62313d51d",
"localEndpoint": null,
"id": "cc23407a6eb33e4f",
"name": "kong.body_filter",
"parentId": "842b5d62df16ba9d",
"remoteEndpoint": null,
"duration": 0,
"timestamp": 1.530259357673e+15
},
{
"traceId": "9aa46f78b4afbd37e6845ba62313d51d",
"localEndpoint": null,
"id": "842b5d62df16ba9d",
"tags": {
"peer.service": "test",
"peer.port": 9500,
"peer.ipv4": "10.148.7.194",
"kong.service": "b1a71321-3b6d-46ef-b376-11e7275c3f36",
"kong.route": "107bb250-a65f-4915-8731-1d7b860bdba7",
"peer.hostname": "10.148.7.194",
"span.kind": "client"
},
"name": "kong.proxy",
"parentId": "ec270a5446cd8a68",
"kind": "CLIENT",
"remoteEndpoint": {
"ipv4": "10.148.7.194",
"port": 9500
},
"duration": 2000,
"timestamp": 1.530259357671e+15
},
{
"traceId": "9aa46f78b4afbd37e6845ba62313d51d",
"localEndpoint": null,
"id": "ec270a5446cd8a68",
"tags": {
"http.status_code": 400,
"span.kind": "server",
"peer.ipv4": "172.17.0.1",
"http.method": "GET",
"peer.port": 47580,
"http.url": "http:\/\/localhost:8000\/test",
"component": "kong"
},
"name": "kong.request",
"kind": "SERVER",
"remoteEndpoint": {
"ipv4": "172.17.0.1",
"port": 47580
},
"duration": 2000,
"timestamp": 1.530259357671e+15
}
]Hi,
zipkin no data.
i found ther error
2018/11/22 14:18:48 [error] 2016#0: *29644324 [lua] handler.lua:28: [opentracing] reporter flush failed: 405 Method Not Allowed, context: ngx.timer, client: 218.82.139.28, server: 0.0.0.0:8000
kong version 0.14.1
thanks
What do krs, krf, kaf, kas, khs, kbf, kbs, khf mean?
When I use Kong Zipkin plugins, the list of services in tracer dashboard is polluted with many services.
I think, we should to get service name from env variable (ex: KONG_ZIPKIN_PLUGIN_SERVICE_NAME), or add to config or just "kong".
Hi,
i am having a weird issue in my kong production instance. A small percentage of requests fail with status code 400 (upstream server returned 200) and in those cases i see in the kong log an exception coming from here:
failed to run log_by_lua*: /usr/local/share/lua/5.1/kong/plugins/zipkin/span.lua:124: invalid annotation timestamp
`stack traceback:
[C]: in function 'assert'
/usr/local/share/lua/5.1/kong/plugins/zipkin/span.lua:124: in function 'annotate'
/usr/local/share/lua/5.1/kong/plugins/zipkin/handler.lua:254: in function </usr/local/share/lua/5.1/kong/plugins/zipkin/handler.lua:228>
/usr/local/share/lua/5.1/kong/init.lua:238: in function 'execute_plugins_iterator'
/usr/local/share/lua/5.1/kong/init.lua:1153: in function 'log'
log_by_lua(nginx-kong.conf:105):2: in main chunk while logging request, client: XXX, server: kong, request:
Any idea, why the annotation timestamp could be wrong or in which direction i should look?
Thanks in advance
Nils
Concurrent 400,Loop 10 times
Jmeter:
ES:
Hit a tx that was blocked by the Mod Security NGINX WAF returning a 400 to the client. The Log phase Zipkin execution assumed that KONG_ACCESS_START would be set in such a case when it must not get set in these cases. Other NGINX modules may do so as well due to short circuit. I supposed when Kong short circuits via access blocking etc auth plugins/acl plugins. the KONG_ACCESS_START is set.
2020/02/13 09:19:20 [error] 72#0: *37331 failed to run log_by_lua*: ...arocks/share/lua/5.1/kong/plugins/zipkin/opentracing.lua:208: attempt to perform arithmetic on field 'KONG_ACCESS_START' (a nil value)
--
| stack traceback:
| ...arocks/share/lua/5.1/kong/plugins/zipkin/opentracing.lua:208: in function 'log'
| ...g/luarocks/share/lua/5.1/kong/plugins/zipkin/handler.lua:34: in function <...g/luarocks/share/lua/5.1/kong/plugins/zipkin/handler.lua:33>
| /usr/local/kong/luarocks/share/lua/5.1/kong/init.lua:209: in function 'execute_plugins_iterator'
| /usr/local/kong/luarocks/share/lua/5.1/kong/init.lua:1068: in function 'log'
| log_by_lua(nginx.conf:167):2: in main chunk while logging request, client: 10.96.29.29, server: kong, request: "GET /api/pdr/demo/test/v1.0?appealid=13414&trackingid=518sourcesystem=ETS HTTP/1.1", host: "gateway.company.com"
Might be worth considering the case that it's not set though to prevent NPE's. Generally preventing nil pointers in app execution is good anyways with proper checking in place due to 3rd party modules and integrations.
@james-callahan , We are using config.sample_ratio=0.2. Below is steps we have followed to created zipkin plugin.
ZIPKIN running in docker container.
docker run -d -p 9411:9411 openzipkin/zipkin
ZIPKIN logs
ava.util.Map<java.lang.String, java.lang.String>)
2018-07-26 12:17:21.984 INFO 1 --- [ main] s.b.a.e.w.s.WebMvcEndpointHandlerMapping : Mapped "{[/actuator],methods=[GET],produces=[application/vnd.spring-boot.actuator.v2+json || application/json]}" onto protected java.util.Map<java.lang.String, java.util.Map<java.lang.String, org.springframework.boot.actuate.endpoint.web.Link>> org.springframework.boot.actuate.endpoint.web.servlet.WebMvcEndpointHandlerMapping.links(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2018-07-26 12:17:22.113 INFO 1 --- [ main] o.xnio : XNIO version 3.3.8.Final
2018-07-26 12:17:22.138 INFO 1 --- [ main] o.x.nio : XNIO NIO Implementation Version 3.3.8.Final
2018-07-26 12:17:22.264 INFO 1 --- [ main] o.s.b.w.e.u.UndertowServletWebServer : Undertow started on port(s) 9411 (http) with context path ''
2018-07-26 12:17:22.270 INFO 1 --- [ main] z.s.ZipkinServer : Started ZipkinServer in 6.391 seconds (JVM running for 7.631)
2018-07-26 12:17:24.722 INFO 1 --- [ XNIO-1 task-1] i.u.servlet : Initializing Spring FrameworkServlet 'dispatcherServlet'
2018-07-26 12:17:24.722 INFO 1 --- [ XNIO-1 task-1] o.s.w.s.DispatcherServlet : FrameworkServlet 'dispatcherServlet': initialization started
2018-07-26 12:17:24.745 INFO 1 --- [ XNIO-1 task-1] o.s.w.s.DispatcherServlet : FrameworkServlet 'dispatcherServlet': initialization completed in 23 ms
Setup for kong- zipkin plugin we followed
creating API
/ # curl -i -X POST \
> --url http://localhost:8001/apis/ \
> --data 'uris=/Stub' \
> --data 'name=example-api' \
> --data 'upstream_url=http://backend-application-server:9999/Stub/Stub'
HTTP/1.1 201 Created
Date: Mon, 30 Jul 2018 08:09:41 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Access-Control-Allow-Origin: *
Server: kong/0.14.0
Content-Length: 360
{"created_at":1532938181592,"strip_uri":true,"id":"93b6e693-ab1a-481a-bfc1-bd0206f26fbf","name":"example-api","http_if_terminated":false,"preserve_host":false,"upstream_url":"http:\/\/backend-application-server:9999\/Stub\/Stub","uris":["\/Stub"],"upstream_connect_timeout":60000,"upstream_send_timeout":60000,"upstream_read_timeout":60000,"retries":5,"https_only":false}
enable zipkin plugin for api
/ # curl -X POST http://localhost:8001/apis/example-api/plugins \
> --data "name=zipkin" \
> --data "config.http_endpoint=http://zipkin-server:9411/api/v2/spans" \
> --data "config.sample_ratio=0.2"
{"created_at":1532938213000,"config":{"sample_ratio":0.2,"http_endpoint":"http:\/\/zipkin-server:9411\/api\/v2\/spans"},"id":"444441f9-cc57-4728-af2f-ae85081951ce","name":"zipkin","api_id":"93b6e693-ab1a-481a-bfc1-bd0206f26fbf","enabled":true}
Please suggest what could be the main culprit here. Thanks in advance!!
Regards,
Hi,
I am using openresty as API gateway because I need to write a custom lua script to integrate with our authorization backend. I couldn't find any lua plugin supporting zipkin except kong-plugin-zipkin. I installed it in openresty but I don't know how to configure it with openresty.
If you know how to do it, please give me the instruction.
Thanks,
cua dong
Issuing the command:
curl -X PUT --url http://localhost:8001/plugins/ -d name=zipkin -d config.http_endpoint=http://127.0.0.1:9411/api/v2/spans
Results in:
'{"config":"plugin 'zipkin' not enabled; add it to the 'custom_plugins' configuration property"}'
I think this means I have to clone this repo, then follow instructions as specified here(since I am running Kong in Docker):
Kong/docker-kong#24
A little frustrated that this plugin is shipped with Kong as there isn't a sliver of mention otherwise found, here:
And here:
2. https://getkong.org/plugins/zipkin/
Seems like if Kong is going to showcase it on their site, it should be part of the core and if not there should be instructions on how to add it to Kong in any of these 3 places, but particularly on the README.MD of this repo?
Am I missing something here? Were these instructions omitted for a purpose?
Instead of complaining, I am happy to help out and create them. I suppose I am just confused rather than frustrated.. Again, feels like I am missing something.
I currently have Kong and some backend services shipping spans in Zipkin format to the Zipkin-compatible endpoint on a Jaeger all-in-one instance, all running in Kubernetes and using the Kong Kubernetes Ingress Controller. For the most part everything is working, except the offset timestamps for the phase start/end logs viewed in the UI are all set to huge negative numbers that indicate the beginning of the epoch. See image below:
Zipkin-formatted spans received by the same endpoint from other sources correctly indicate the log time relative to the start of the trace:
Any assistance with this matter would be greatly appreciated.
Environment:
Kong Image: kong:2.0
Ingress Controller Image: kong-docker-kubernetes-ingress-controller.bintray.io/kong-ingress-controller:0.8.0
Jaeger Image: jaegertracing/all-in-one:1.18.0
Kong/kong#3502 renamed balancer_address to balancer_data
Hello everybody,
Im trying to trace latency of my service.
In image above, I saw kaf is 2ms, khs is 124ms
So, what does Kong do in time from 2ms ==> 124ms?
Thanks!
Hi, I can't make work Kong to sends traces to zipkin.
Kong version: 2.0.4
Zipkin version: 2.21
Enabled plugins:
{
"enabled_plugins": [
"correlation-id",
"pre-function",
"cors",
"ldap-auth",
"loggly",
"hmac-auth",
"zipkin",
"request-size-limiting",
"azure-functions",
"request-transformer",
"oauth2",
"response-transformer",
"ip-restriction",
"statsd",
"jwt",
"proxy-cache",
"basic-auth",
"key-auth",
"http-log",
"datadog",
"tcp-log",
"rate-limiting",
"post-function",
"prometheus",
"acl",
"syslog",
"file-log",
"acme",
"udp-log",
"response-ratelimiting",
"aws-lambda",
"session",
"bot-detection",
"request-termination"
]
}
Configure plugin:
curl --url http://kong:8001/plugins/ -d name=zipkin -d config.http_endpoint=http://zipkin:9411/api/v2/spans
{"created_at":1589105820,"config":{"sample_ratio":0.001,"http_endpoint":"http:\/\/zipkin:9411\/api\/v2\/spans","include_credential":true,"default_service_name":null},"id":"0641365b-4d1f-411c-a0b4-9df6be845a96","service":null,"enabled":true,"protocols":["grpc","grpcs","http","https"],"name":"zipkin","consumer":null,"route":null,"tags":null}
Plugin conf:
{
"next": null,
"data": [
{
"created_at": 1589105820,
"config": {
"sample_ratio": 0.001,
"http_endpoint": "http://zipkin:9411/api/v2/spans",
"include_credential": true,
"default_service_name": null
},
"id": "0641365b-4d1f-411c-a0b4-9df6be845a96",
"service": null,
"enabled": true,
"protocols": [
"grpc",
"grpcs",
"http",
"https"
],
"name": "zipkin",
"consumer": null,
"route": null,
"tags": null
},
Kong Logs:
2020/05/10 10:17:01 [debug] 23#0: *3917125 [lua] cache.lua:295: invalidate_local(): [DB cache] invalidating (local): 'plugins:zipkin::::'
2020/05/10 10:17:01 [debug] 23#0: *3917125 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=23, data=plugins:zipkin::::
2020/05/10 10:17:01 [debug] 23#0: *3917125 [lua] cache.lua:311: invalidate(): [DB cache] broadcasting (cluster) invalidation for key: 'plugins:zipkin::::'
2020/05/10 10:17:01 [debug] 23#0: *3917125 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=plugins, pid=nil, data=table: 0x7ff200039c18
2020/05/10 10:17:01 [debug] 23#0: *3917125 [lua] handler.lua:254: [events] Plugin updated, invalidating plugins iterator
2020/05/10 10:17:01 [debug] 23#0: *3917125 [lua] cache.lua:295: invalidate_local(): [DB cache] invalidating (local): 'plugins_iterator:version'
2020/05/10 10:17:01 [debug] 23#0: *3917125 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=23, data=plugins_iterator:version
2020/05/10 10:17:01 [debug] 23#0: *3917125 [lua] cache.lua:311: invalidate(): [DB cache] broadcasting (cluster) invalidation for key: 'plugins_iterator:version'
2020/05/10 10:17:01 [debug] 23#0: *3917125 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=plugins:create, pid=nil, data=table: 0x7ff200039c18
192.168.64.6 - - [10/May/2020:10:17:01 +0000] "POST /plugins/ HTTP/1.1" 201 353 "-" "curl/7.67.0"
2020/05/10 10:17:01 [info] 23#0: *3917125 client 192.168.64.6 closed keepalive connection
2020/05/10 10:17:01 [debug] 26#0: *3917134 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=23, data=plugins:zipkin::::
2020/05/10 10:17:01 [debug] 26#0: *3917134 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=23, data=plugins_iterator:version
2020/05/10 10:17:01 [debug] 24#0: *3917138 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=23, data=plugins:zipkin::::
2020/05/10 10:17:01 [debug] 24#0: *3917138 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=23, data=plugins_iterator:version
2020/05/10 10:17:01 [debug] 25#0: *3917146 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=23, data=plugins:zipkin::::
2020/05/10 10:17:01 [debug] 25#0: *3917146 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=23, data=plugins_iterator:version
zipkin configuration:
zipkin:
image: openzipkin/zipkin
container_name: zipkin
environment:
- STORAGE_TYPE=elasticsearch
- ES_HOSTS=http://elasticsearch:9200
- ES_USERNAME=elastic
- ES_PASSWORD=${ELASTIC_PASSWORD:-elastic}
- SELF_TRACING_ENABLED=true
- TZ=${TZ:-Europe/Madrid}
networks:
- net
ports:
- 9411:9411
depends_on:
- elasticsearch
Thanks!
When using the zipkin plugin, I am seeing traces like:
I am not seeing any latency value for the access, balancer, etc spans - is this intended?
Error is like so:
2018/12/25 04:45:56 [error] 38#0: *5243172 failed to run log_by_lua*: .../local/share/lua/5.1/kong/plugins/zipkin/opentracing.lua:40: attempt to index local 'addr' (a nil value)
stack traceback:
.../local/share/lua/5.1/kong/plugins/zipkin/opentracing.lua:40: in function 'ip_tag'
.../local/share/lua/5.1/kong/plugins/zipkin/opentracing.lua:202: in function 'log'
/usr/local/share/lua/5.1/kong/plugins/zipkin/handler.lua:31: in function 'log'
/usr/local/share/lua/5.1/kong/init.lua:751: in function 'log'
log_by_lua(nginx.conf:149):2: in function <log_by_lua(nginx.conf:149):1> while logging request, client: 10.129.52.1, server: kong, request: "GET /F5/status HTTP/1.1", host: "gateway-dev-core-datacenter.company.com"
A false assumption has been made that addr will always be non-nil for this code:
https://github.com/Kong/kong-plugin-zipkin/blob/master/kong/plugins/zipkin/opentracing.lua#L40
I've been asked to back-port Victor Soares' code to Kong 1.1.2, would that correspond to kong-plugin-zipkin version 0.1.2?
When using Kong Zipkin plugin with Jaeger client without B3TextMapCodec, Jaeger can't aggregate spans and services span:
To flexibility support integrating multiple tracing system (jaeger, zipkin) with Kong.
Let's support extracting span from Tracing Context header like uber-trace-id header when unable extracting from X-B3-... headers in the request, and also injecting Kong's span context to uber-trace-id as well as current X-B3-... headers to upstream request headers.
Currently the serviceName is hard-coded to "kong". Could this be made a plugin configuration?
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.