keyval-dev / opentelemetry-go-instrumentation Goto Github PK
View Code? Open in Web Editor NEWOpenTelemetry auto-instrumentation for Go applications
License: Apache License 2.0
OpenTelemetry auto-instrumentation for Go applications
License: Apache License 2.0
The example shown in the project documentation focuses on gRPC, it would be awesome if the example could demonstrate integration with AMQP. We already instrument our code using
https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrumentation
but then have to manually instrument our code for AMQP
We already instrument our code using
https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrumentation
but then have to manually instrument our code for AMQP . I'd hope that the auto-instrumentation
offered by this project could remove the need for the manual instrumentation required by AMQP.
use daemonset way not sidecar way
May be consider daemonset that it will deploy agent on nodes,which it will collect data by pid, and the use pid association with pod metadata.
what do you think? @edeNFed
I'm using Honeycomb as my otel endpoint directly, without going to a local collector first. It looks like I'm unable to connect to their endpoint: api.honeycomb.io:443
because the gRPC client created by this project is requiring an insecure connection:
I would like to be able to configure that the gRPC client is configured expecting a secure connection so that I can connect directly to honeycomb.
When a go binary file doesn't contain ".gosymtab" section, it should be able to instrument it.
Analyze process stops when it can't find ".gosymtab" section in target go binary file.
OTEL_SERVICE_NAME=goapp OTEL_TARGET_EXE=/goapp OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317 /kv-go-instrumentation
{"level":"info","ts":1667322994.3850787,"caller":"cli/main.go:22","msg":"starting Go OpenTelemetry Agent ..."}
{"level":"info","ts":1667322994.385147,"caller":"opentelemetry/controller.go:92","msg":"Establishing connection to OpenTelemetry collector ..."}
{"level":"info","ts":1667322996.387433,"caller":"process/discover.go:42","msg":"found process","pid":19}
{"level":"info","ts":1667322996.3889744,"caller":"process/analyze.go:73","msg":"found addr of keyval map","addr":140331000791040}
{"level":"error","ts":1667322996.3974175,"caller":"cli/main.go:61","msg":"error while analyzing target process","error":".gosymtab section not found in target binary, make sure this is a Go application","stacktrace":"main.main\n\t/app/cli/main.go:61\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release
:uname -r
:
"CentOS Linux 7 (Core)"
5.16.11-1.el7.elrepo.x86_64
The corresponding source code is from:
https://github.com/keyval-dev/opentelemetry-go-instrumentation/blob/master/pkg/process/analyze.go
line 117 to 120:
sec := elfF.Section(".gosymtab")
if sec == nil {
return nil, fmt.Errorf("%s section not found in target binary, make sure this is a Go application", ".gosymtab")
}
Read memory maps.
Can't find memory mapping as as all mapping has been filtered out.
The maps of the go lang program as below
00400000-00623000 r-xp 00000000 fd:00 1048990 /home/vagrant/go-http-demo/main
00623000-00822000 r--p 00223000 fd:00 1048990 /home/vagrant/go-http-demo/main
00822000-00860000 rw-p 00422000 fd:00 1048990 /home/vagrant/go-http-demo/main
00860000-00899000 rw-p 00000000 00:00 0
02061000-02082000 rw-p 00000000 00:00 0 [heap]
c000000000-c000400000 rw-p 00000000 00:00 0
...
7f32e948f000-7f32e94b7000 r--p 00000000 fd:00 525141 /usr/lib/x86_64-linux-gnu/libc.so.6
...
7f32e96aa000-7f32e96b7000 rw-p 00000000 00:00 0
7f32e96be000-7f32e96c0000 rw-p 00000000 00:00 0
7f32e96c0000-7f32e96c2000 r--p 00000000 fd:00 524900 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
...
7ffe501e6000-7ffe50207000 rw-p 00000000 00:00 0 [stack]
7ffe50311000-7ffe50315000 r--p 00000000 00:00 0 [vvar]
7ffe50315000-7ffe50317000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall]
so no map will be fond as m.Perms != nil && m.Perms.Read && m.Perms.Write && m.Perms.Execute
required in findKeyvalMmap
.
https://app.vagrantup.com/bento/boxes/ubuntu-22.04
operation-system/kernel version:
Output of awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release
:
Output of uname -r
:
Ubuntu 22.04.1 LTS
5.15.0-30-generic
As of now, go-agent is generating id in format which don't conforms with AWS X-Ray
I looked at code and didn't understand how to replace &eBPFSourceIDGenerator
with xray.NewIDGenerator()
without crushing the tracing.
In order to send traces to X-ray, go-agent needs generate ID in proper format.
After read the whole codes, I don't find any codes to send the trace ID in the different services. May be I don't find it. Could you share the parts for me? Thanks a lot ~
The exporter endpoint is not parsed directly in the code https://github.com/keyval-dev/opentelemetry-go-instrumentation/blob/master/pkg/opentelemetry/controller.go#L76 from which grpc connection is created.
This is not ideal because it limits how the exporter is initialized and it breaks semantics of OTEL_EXPORTER_OTLP_ENDPOINT
which should have form of http(s)://host:port
. See https://github.com/open-telemetry/opentelemetry-go/tree/main/exporters/otlp/otlptrace and OTEL spec.
I would propose to change the code to consume only env vars in OTEL defined format and leave parsing of the endpoint to OTEL SDK/exporter implementation.
Hiya!
I've caught this repo from Observability News by Michael Hasuenblas - great stuff:)
I wonder, since this requires 0 code changes, whether it will be a good fit to bundle into our OSS project KoolKits.
We have docker debug images specifically geared towards debugging applications written in a specific language - you can check out the project here.
This might fit into our Go KoolKit.
Your comments much appreciated @edeNFed :)
Hey, this is fabulous! :)
I'm wondering what (if any) fundamental limitations exist with this approach? For example:
Again, really awesome project, I'm excited to see where this goes!
Allow opentelemetry-go-instrumentation trace http requests.
A Golang project may send http request to other servers through Golang net/request, for doing distributed tracing need to add the trace id in http headers.
The header is a map https://cs.opensource.google/go/go/+/refs/tags/go1.19.2:src/net/http/request.go;l=148.
If we do this through uprobe
, we need edit a Golang map.
So how could we do this? Or do we have other options?
I guess eBPF program can't call a user's function easily, so we have to mange the memory, but map is complex, not sure how to do this.
Maybe we can calculate where we can put the key/value pair(we know the key and can know the length of value), and use the result to do memory management?
Does this package add instrumentation for all functions or only net/http
and grpc
?
eBPF allows to add instrumentation to every function call, would be good to have it here.
Automatically add instrumentation to all functions in Go pkg.
This is primary why I found this project, but surprised it is not supported.
Auto Instrumentation works without issues.
Sidecar container halts with error.
{"level":"info","ts":1666009431.26208,"caller":"cli/main.go:22","msg":"starting Go OpenTelemetry Agent ..."}
{"level":"info","ts":1666009431.2621613,"caller":"opentelemetry/controller.go:92","msg":"Establishing connection to OpenTelemetry collector ..."}
{"level":"info","ts":1666009433.2638783,"caller":"process/discover.go:42","msg":"found process","pid":13}
{"level":"info","ts":1666009433.2643228,"caller":"process/analyze.go:73","msg":"found addr of keyval map","addr":139872935333888}
{"level":"info","ts":1666009433.281399,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"net/http.(*ServeMux).ServeHTTP","returns":2}
{"level":"info","ts":1666009433.2829392,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc/internal/transport.(*http2Client).createHeaderFields","returns":3}
{"level":"info","ts":1666009433.2831843,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc.(*ClientConn).Invoke","returns":2}
{"level":"info","ts":1666009433.2836869,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc.(*Server).handleStream","returns":5}
{"level":"info","ts":1666009433.2840135,"caller":"cli/main.go:64","msg":"target process analysis completed","pid":13,"go_version":"1.17.11","dependencies":{"github.com/FZambia/eagle":"v0.0.2","github.com/FZambia/sentinel":"v1.1.0","github.com/FZambia/statik":"v0.1.2-0.20180217151304-b9f012bb2a1b","github.com/FZambia/tarantool":"v0.3.1","github.com/FZambia/viper-lite":"v0.0.0-20220110144934-1899f66c7d0e","github.com/beorn7/perks":"v1.0.1","github.com/centrifugal/centrifuge":"v0.23.1","github.com/centrifugal/protocol":"v0.8.8","github.com/cespare/xxhash/v2":"v2.1.2","github.com/cristalhq/jwt/v4":"v4.0.0","github.com/gobwas/glob":"v0.2.3","github.com/golang/protobuf":"v1.5.2","github.com/gomodule/redigo":"v1.8.8","github.com/google/uuid":"v1.3.0","github.com/gorilla/securecookie":"v1.1.1","github.com/gorilla/websocket":"v1.5.0","github.com/igm/sockjs-go/v3":"v3.0.2","github.com/josharian/intern":"v1.0.0","github.com/mailru/easyjson":"v0.7.7","github.com/mattn/go-isatty":"v0.0.12","github.com/matttproud/golang_protobuf_extensions":"v1.0.1","github.com/mitchellh/mapstructure":"v1.4.3","github.com/mna/redisc":"v1.3.2","github.com/nats-io/nats.go":"v1.14.0","github.com/nats-io/nkeys":"v0.3.0","github.com/nats-io/nuid":"v1.0.1","github.com/pelletier/go-toml":"v1.9.4","github.com/prometheus/client_golang":"v1.12.1","github.com/prometheus/client_model":"v0.2.0","github.com/prometheus/common":"v0.34.0","github.com/prometheus/procfs":"v0.7.3","github.com/rakutentech/jwk-go":"v1.0.1","github.com/rs/zerolog":"v1.21.0","github.com/segmentio/asm":"v1.1.4","github.com/segmentio/encoding":"v0.3.5","github.com/spf13/cast":"v1.4.1","github.com/spf13/cobra":"v0.0.7","github.com/spf13/jwalterweatherman":"v1.1.0","github.com/spf13/pflag":"v1.0.5","github.com/valyala/bytebufferpool":"v1.0.0","github.com/vmihailenco/msgpack/v5":"v5.3.5","github.com/vmihailenco/tagparser/v2":"v2.0.0","golang.org/x/crypto":"v0.0.0-20220411220226-7b82a4e95df4","golang.org/x/net":"v0.0.0-20220421235706-1d1ef9303861","golang.org/x/sync":"v0.0.0-20210220032951-036812b2e83c","golang.org/x/sys":"v0.0.0-20220422013727-9388b58f7150","golang.org/x/text":"v0.3.7","google.golang.org/genproto":"v0.0.0-20220422154200-b37d22cd5731","google.golang.org/grpc":"v1.46.0","google.golang.org/protobuf":"v1.28.0","gopkg.in/yaml.v2":"v2.4.0"},"total_functions_found":4}
{"level":"info","ts":1666009433.2841408,"caller":"cli/main.go:70","msg":"invoking instrumentors"}
{"level":"info","ts":1666009433.312439,"logger":"allocator","caller":"allocator/allocator_linux.go:19","msg":"Loading allocator","start_addr":139872935333888,"end_addr":139872947916800}
{"level":"info","ts":1666009433.3126094,"caller":"instrumentors/runner.go:68","msg":"loading instrumentor","name":"google.golang.org/grpc"}
{"level":"info","ts":1666009433.3144336,"caller":"inject/injector.go:67","msg":"Injecting variables","vars":{"clientconn_target_ptr_pos":24,"end_addr":139872947916800,"is_registers_abi":true,"start_addr":139872935333888,"total_cpus":4}}
{"level":"error","ts":1666009433.315479,"caller":"instrumentors/runner.go:71","msg":"error while loading instrumentors, cleaning up","name":"google.golang.org/grpc","error":"field UprobeClientConnInvoke: program uprobe_ClientConn_Invoke: load program: invalid argument: ; int uprobe_ClientConn_Invoke(struct pt_regs *ctx) {\n0: (bf) r6 = r1\n1: (b7) r7 = 0\n; struct grpc_request_t grpcReq = {};\n2: (7b) *(u64 *)(r10 -16) = r7\nlast_idx 2 first_idx 0\nregs=80 stack=0 before 1: (b7) r7 = 0\n3: (7b) *(u64 *)(r10 -24) = r7\n4: (7b) *(u64 *)(r10 -32) = r7\n5: (7b) *(u64 *)(r10 -40) = r7\n6: (7b) *(u64 *)(r10 -48) = r7\n7: (7b) *(u64 *)(r10 -56) = r7\n8: (7b) *(u64 *)(r10 -64) = r7\n9: (7b) *(u64 *)(r10 -72) = r7\n10: (7b) *(u64 *)(r10 -80) = r7\n11: (7b) *(u64 *)(r10 -88) = r7\n12: (7b) *(u64 *)(r10 -96) = r7\n13: (7b) *(u64 *)(r10 -104) = r7\n14: (7b) *(u64 *)(r10 -112) = r7\n15: (7b) *(u64 *)(r10 -120) = r7\n16: (7b) *(u64 *)(r10 -128) = r7\n17: (7b) *(u64 *)(r10 -136) = r7\n18: (7b) *(u64 *)(r10 -144) = r7\n19: (7b) *(u64 *)(r10 -152) = r7\n20: (7b) *(u64 *)(r10 -160) = r7\n21: (7b) *(u64 *)(r10 -168) = r7\n; grpcReq.start_time = bpf_ktime_get_boot_ns();\n22: (85) call unknown#125\ninvalid func unknown#125\nprocessed 23 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0","stacktrace":"github.com/keyval-dev/opentelemetry-go-instrumentation/pkg/instrumentors.(*instrumentorsManager).load\n\t/app/pkg/instrumentors/runner.go:71\ngithub.com/keyval-dev/opentelemetry-go-instrumentation/pkg/instrumentors.(*instrumentorsManager).Run\n\t/app/pkg/instrumentors/runner.go:19\nmain.main\n\t/app/cli/main.go:71\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
{"level":"info","ts":1666009433.3155577,"caller":"grpc/probe.go:213","msg":"closing gRPC instrumentor"}
{"level":"info","ts":1666009433.3155837,"caller":"server/probe.go:179","msg":"closing net/http instrumentor"}
{"level":"error","ts":1666009433.315593,"caller":"cli/main.go:73","msg":"error while running instrumentors","error":"field UprobeClientConnInvoke: program uprobe_ClientConn_Invoke: load program: invalid argument: ; int uprobe_ClientConn_Invoke(struct pt_regs *ctx) {\n0: (bf) r6 = r1\n1: (b7) r7 = 0\n; struct grpc_request_t grpcReq = {};\n2: (7b) *(u64 *)(r10 -16) = r7\nlast_idx 2 first_idx 0\nregs=80 stack=0 before 1: (b7) r7 = 0\n3: (7b) *(u64 *)(r10 -24) = r7\n4: (7b) *(u64 *)(r10 -32) = r7\n5: (7b) *(u64 *)(r10 -40) = r7\n6: (7b) *(u64 *)(r10 -48) = r7\n7: (7b) *(u64 *)(r10 -56) = r7\n8: (7b) *(u64 *)(r10 -64) = r7\n9: (7b) *(u64 *)(r10 -72) = r7\n10: (7b) *(u64 *)(r10 -80) = r7\n11: (7b) *(u64 *)(r10 -88) = r7\n12: (7b) *(u64 *)(r10 -96) = r7\n13: (7b) *(u64 *)(r10 -104) = r7\n14: (7b) *(u64 *)(r10 -112) = r7\n15: (7b) *(u64 *)(r10 -120) = r7\n16: (7b) *(u64 *)(r10 -128) = r7\n17: (7b) *(u64 *)(r10 -136) = r7\n18: (7b) *(u64 *)(r10 -144) = r7\n19: (7b) *(u64 *)(r10 -152) = r7\n20: (7b) *(u64 *)(r10 -160) = r7\n21: (7b) *(u64 *)(r10 -168) = r7\n; grpcReq.start_time = bpf_ktime_get_boot_ns();\n22: (85) call unknown#125\ninvalid func unknown#125\nprocessed 23 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0","stacktrace":"main.main\n\t/app/cli/main.go:73\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
I've followed the tutorial and tried to add instrumentation to an existing deployment in k8s.
Here's a part of deployment.yaml, it is basically a copy of diff between emojivoto and emojivoto-instrumented
...
spec:
shareProcessNamespace: true
initContainers:
- name: copy-launcher
image: keyval/launcher:v0.1
command:
- cp
- -a
- /kv-launcher/.
- /odigos-launcher/
volumeMounts:
- name: launcherdir
mountPath: /odigos-launcher
volumes:
- name: launcherdir
emptyDir: { }
- name: kernel-debug
hostPath:
path: /sys/kernel/debug
containers:
- name: ...
command:
- /odigos-launcher/launch
- /usr/local/bin/centrifugo
- --health
- --prometheus
...
- name: instrumentation
image: keyval/otel-go-agent:v0.6.0
env:
- name: OTEL_TARGET_EXE
value: /usr/local/bin/centrifugo
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: "otel-collector:4317"
- name: OTEL_SERVICE_NAME
value: "test"
securityContext:
runAsUser: 0
capabilities:
add:
- SYS_PTRACE
privileged: true
volumeMounts:
- mountPath: /sys/kernel/debug
name: kernel-debug
Kubernetes version:
Output of kubectl version
:
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-14T02:36:39Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.10-eks-15b7512", GitCommit:"cd6399691d9b1fed9ec20c9c5e82f5993c3f42cb", GitTreeState:"clean", BuildDate:"2022-08-31T19:17:01Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.25) and server (1.23) exceeds the supported minor version skew of +/-1
operation-system/kernel version:
Assuming we're talking about container OS:
Output of awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release
:
Output of uname -r
:
Alpine Linux v3.13
5.4.209-116.367.amzn2.x86_64
A stable connection to the collector.
Program terminates because the go version is not identified.
podman run --network=host --rm -it jaegertracing/opentelemetry-all-in-one:latest
make build && OTEL_SERVICE_NAME=test OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317 OTEL_TARGET_EXE=${PWD}/unstripped ./kv-go-instrumentation
./unstripped
(gorilla mux http server)operation-system/kernel version:
Output of awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release
:
Output of uname -r
:
"Fedora Linux 35 (Workstation Edition)"
5.17.4-200.fc35.x86_64
binary strings strings unstripped | grep "Go buildinf" -A 1
:
Go buildinf:
go1.18
go version go1.18 linux/amd64
{"level":"info","ts":1651701304.5921707,"caller":"cli/main.go:22","msg":"starting Go OpenTelemetry Agent ..."}
{"level":"info","ts":1651701304.5922222,"caller":"opentelemetry/controller.go:97","msg":"Establishing connection to OpenTelemetry collector ..."}
{"level":"info","ts":1651701306.6052349,"caller":"process/discover.go:37","msg":"process not found yet, trying again soon","exe_path":"/home/frzifus/git/go/observability/opentelemetry-go-instrumentation/unstripped"}
{"level":"info","ts":1651701308.5969474,"caller":"process/discover.go:37","msg":"process not found yet, trying again soon","exe_path":"/home/frzifus/git/go/observability/opentelemetry-go-instrumentation/unstripped"}
{"level":"info","ts":1651701310.5970006,"caller":"process/discover.go:42","msg":"found process","pid":1131585}
{"level":"error","ts":1651701310.5971448,"caller":"cli/main.go:61","msg":"error while analyzing target process","error":"could not detect go version","stacktrace":"main.main\n\t/home/frzifus/git/go/observability/opentelemetry-go-instrumentation/cli/main.go:61\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
Hi keyval-dev team,
Thanks for making this library for auto instrumentation otel in go.
Before read the report below, i would like to ask how do you group spans into one tracing id?
is it coming from same goroutine id?
Thanks,
Different request different tracing id
Different request same tracing id, (idk, maybe because golang/mux reused goroutine for another requests)
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.12-gke.500", GitCommit:"26604476cf3bcb64bec486e4b5880a5773a52e75", GitTreeState:"clean", BuildDate:"2022-07-20T09:22:48Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}
operation-system/kernel version:
Darwin 21.1.0
go version:
1.17
This repo is the best idea I've seen recently !!!
Will consider supporting the scenario of starting a new goroutine for asynchronous client requests ?
In this scenario, is it impossible to connect the trace context through the goroutine context map ?
Trace the data
Not tracing the data
Kubernetes version:
Output of 1.23
:
go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).startProcessingCycle
go.opentelemetry.io/[email protected]/processor/batchprocessor/batch_processor.go:144
2022-10-11T18:50:12.446Z WARN batchprocessor/batch_processor.go:187 Sender failed {"component_kind": "processor", "component_type": "batch", "component_name": "batch", "error": "Permanent error: OC span has an all zeros trace ID"}
go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).sendItems
go.opentelemetry.io/[email protected]/processor/batchprocessor/batch_processor.go:187
go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).startProcessingCycle
go.opentelemetry.io/[email protected]/processor/batchprocessor/batch_processor.go:144
2022-10-11T19:05:23.386Z ERROR exporterhelper/queued_retry.go:204 Exporting failed. Try enabling retry_on_failure config option. {"component_kind": "exporter", "component_type": "jaeger_memory", "component_name": "jaeger_memory", "error": "Permanent error: OC span has an all zeros trace ID"}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:204
go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
go.opentelemetry.io/[email protected]/exporter/exporterhelper/tracehelper.go:114 ```
operation-system/kernel version:
Output of awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release
:
Output of uname -r
:
(paste your output here)
It's not very clear what kind of OTEL integration this approach provide. Does it produce metrics? Distributed traces? Logs?
I'm largely deploying Go apps to Cloud Run and automatic instrumentation would be awesome. The docs only refer to k8s setups.
k8s is pretty complicated and for my apps, Cloud Run is a good match. I imagine other folks are deploying Go apps using other serverless platforms, too, and it would be helpful to be able to use this here vs. manual instrumentation.
I guess that "requires >= v5.2" may prompt me with the kernel version, and after upgrading the kernel to >=5.2, it was successfully confirmed to be true.
Many instances of golang processes are still running in lower version kernels.So do we have any idea or plan to make it compatible with lower version kernels?
[root@centos7test ~]# OTEL_TARGET_EXE=/root/http ./kv-go-instrumentation -stdout
{"level":"info","ts":1675775385.3834639,"caller":"cli/main.go:27","msg":"starting Go OpenTelemetry Agent ..."}
{"level":"info","ts":1675775387.3908978,"caller":"process/discover.go:46","msg":"found process","pid":4644}
{"level":"info","ts":1675775387.3916051,"caller":"process/analyze.go:75","msg":"found addr of keyval map","addr":139854386225152}
{"level":"info","ts":1675775387.398304,"caller":"process/analyze.go:138","msg":"found relevant function for instrumentation","function":"net/http.(*ServeMux).ServeHTTP","returns":2}
{"level":"info","ts":1675775387.3984025,"caller":"cli/main.go:70","msg":"target process analysis completed","pid":4644,"go_version":"1.13.10","dependencies":{},"total_functions_found":1}
{"level":"info","ts":1675775387.3984728,"caller":"cli/main.go:76","msg":"invoking instrumentors"}
{"level":"info","ts":1675775387.4092982,"logger":"allocator","caller":"allocator/allocator_linux.go:19","msg":"Loading allocator","start_addr":139854386225152,"end_addr":139854398808064}
{"level":"info","ts":1675775387.4094503,"caller":"instrumentors/runner.go:68","msg":"loading instrumentor","name":"net/http"}
{"level":"info","ts":1675775387.414667,"caller":"inject/injector.go:67","msg":"Injecting variables","vars":{"ctx_ptr_pos":232,"is_registers_abi":false,"method_ptr_pos":0,"path_ptr_pos":56,"url_ptr_pos":16}}
{"level":"error","ts":1675775387.4149885,"caller":"instrumentors/runner.go:71","msg":"error while loading instrumentors, cleaning up","name":"net/http","error":"field UprobeServerMuxServeHTTP: program uprobe_ServerMux_ServeHTTP: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)","stacktrace":"github.com/keyval-dev/opentelemetry-go-instrumentation/pkg/instrumentors.(*instrumentorsManager).load\n\t/opt/github/opentelemetry-go-instrumentation/pkg/instrumentors/runner.go:71\ngithub.com/keyval-dev/opentelemetry-go-instrumentation/pkg/instrumentors.(*instrumentorsManager).Run\n\t/opt/github/opentelemetry-go-instrumentation/pkg/instrumentors/runner.go:19\nmain.main\n\t/opt/github/opentelemetry-go-instrumentation/cli/main.go:77\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
{"level":"info","ts":1675775387.4150653,"caller":"server/probe.go:179","msg":"closing net/http instrumentor"}
{"level":"error","ts":1675775387.4150894,"caller":"cli/main.go:79","msg":"error while running instrumentors","error":"field UprobeServerMuxServeHTTP: program uprobe_ServerMux_ServeHTTP: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)","stacktrace":"main.main\n\t/opt/github/opentelemetry-go-instrumentation/cli/main.go:79\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
Sounds like a really interesting feature @withlin
Regarding implementation, I think we should consider changing the current logic of specifying the target binary via OTEL_TARGET_EXE
to logic like:
I think changing the behavior to this will allow us to both remove the OTEL_TARGET_EXE
config and support daemon set deployment at the same time. Basically, when deployed as a daemon set with hostPID: true
/proc will simply contain more processes and the instrumentation agent does not even aware of the fact that it is currently deployed as a daemon set.
What do you think?
Originally posted by @edeNFed in #9 (comment)
I follow the steps on the tutorial Getting Started on Kubernetes, but I can't select the emojivoto-voting service on Jaeger UI page, what happend?
root@localhost:~ # kubectl get no
NAME STATUS ROLES AGE VERSION
kind-control-plane Ready control-plane,master 2d14h v1.23.4
root@localhost:~ # kubectl get svc -n emojivoto
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
emoji-svc ClusterIP 10.96.195.205 <none> 8080/TCP,8801/TCP 2d13h
jaeger ClusterIP 10.96.8.171 <none> 4317/TCP,16686/TCP 2d13h
voting-svc ClusterIP 10.96.175.64 <none> 8080/TCP,8801/TCP 40h
web-svc ClusterIP 10.96.209.28 <none> 80/TCP 2d13h
root@localhost:~ # kubectl get po -n emojivoto
NAME READY STATUS RESTARTS AGE
emoji-5dbdd567bd-kcmxk 1/1 Running 1 (42h ago) 2d13h
jaeger-54f8b6d89d-mpqtj 1/1 Running 1 (42h ago) 2d13h
vote-bot-58b4f5fdb7-f578j 1/1 Running 1 (42h ago) 2d13h
voting-5fdcddcfc-hdcxh 1/1 Running 0 40h
web-67c857998c-d9sjj 1/1 Running 1 (42h ago) 2d13h
root@localhost:~ # kubectl patch deployment voting --patch-file voting-patch.yaml -n emojivoto
deployment.apps/voting patched (no change)
root@localhost:~ # kubectl port-forward svc/jaeger --address 0.0.0.0 16686:16686 -n emojivoto
Forwarding from 0.0.0.0:16686 -> 16686
Handling connection for 16686
Handling connection for 16686
kubectl get pods -n atlas
pod/go-service-8c86cf596-5rbpc 2/2 Running
kubectl get pods -n atlas
pod/go-service-8c86cf596-5rbpc 1/2 CrashLoopBackOff
Here's my manifest file relevant for go:
apiVersion: v1
kind: Service
metadata:
labels:
app: go-service
name: go-service
namespace: atlas
spec:
ports:
- port: 30002
protocol: TCP
targetPort: 30002
# If you set the `spec.type` field to `NodePort` and you want a specific port number,
# you can specify a value in the `spec.ports[*].nodePort` field.
nodePort: 30002
selector:
app: go-service
type: NodePort
status:
loadBalancer: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: go-service
name: go-service
namespace: atlas
spec:
replicas: 1
selector:
matchLabels:
app: go-service
strategy: {}
template:
metadata:
labels:
app: go-service
spec:
shareProcessNamespace: true
initContainers:
- name: copy-launcher
image: keyval/launcher:v0.1
command:
- cp
- -a
- /kv-launcher/.
- /odigos-launcher/
volumeMounts:
- name: launcherdir
mountPath: /odigos-launcher
containers:
- args:
- go-service
image: dev0zklabs/atlas-demo-microservice:go-service
name: go-service
command:
- /odigos-launcher/launch
- /root/go-service
volumeMounts:
- mountPath: /odigos-launcher
name: launcherdir
ports:
- containerPort: 30002
imagePullPolicy: Always
envFrom:
- configMapRef:
name: envs-config
resources: {}
- name: emojivoto-emoji-instrumentation
image: keyval/otel-go-agent:v0.6.4
env:
- name: OTEL_TARGET_EXE
value: /root/go-service
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: "jaeger:4317"
- name: OTEL_SERVICE_NAME
value: "go-service"
securityContext:
runAsUser: 0
capabilities:
add:
- SYS_PTRACE
privileged: true
volumeMounts:
- mountPath: /sys/kernel/debug
name: kernel-debug
volumes:
- name: launcherdir
emptyDir: {}
- name: kernel-debug
hostPath:
path: /sys/kernel/debug
status: {}
Kubernetes version:
Output of kubectl version
:
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:40:17Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25+", GitVersion:"v1.25.8-eks-ec5523e", GitCommit:"83fe90de881ffe7450876d204ccdc37a0f95bda3", GitTreeState:"clean", BuildDate:"2023-03-20T21:32:06Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
- operation-system/kernel version:
**Output of `awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release`:**
**Output of `uname -r`:**
Ubuntu 22.04.2 LTS
5.19.0-1022-aws
<!-- Any other additional information -->
Hi, thanks for such a nice project!
I tried using this with emojivoto, but I'm having trouble with context propagation.
Tracing a single service works fine but the traffic across some services is traced as different trace id.
Trace ID is propagated across multiple services
A series of events across multiple services are traced as different traces
curl "http://your-node-ip:30084/api/list"
http://your-node-ip:30081
and check the traced dataKubernetes version:
Output of kubectl version
:
clientVersion:
buildDate: "2022-07-13T14:21:56Z"
compiler: gc
gitCommit: aef86a93758dc3cb2c658dd9657ab4ad4afc21cb
gitTreeState: clean
gitVersion: v1.24.3
goVersion: go1.18.4
major: "1"
minor: "24"
platform: linux/amd64
kustomizeVersion: v4.5.4
serverVersion:
buildDate: "2022-05-24T12:18:48Z"
compiler: gc
gitCommit: 3ddd0f45aa91e2f30c70734b175631bec5b5825a
gitTreeState: clean
gitVersion: v1.24.1
goVersion: go1.18.2
major: "1"
minor: "24"
platform: linux/amd64
operation-system/kernel version:
Output of awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release
:
Output of uname -r
:
$ awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release
"Ubuntu 22.04 LTS"
$ uname -r
5.15.0-43-generic
I am trying to run on Mac M1 processor and it doesn't seem to work.
This issue is for removing the OTEL_TARGET_EXE
env variable and implementing process discovery instead.
Implementing this will make integration with OpenTelemetry Operator easier and overall provide a better experience to users (one less argument to specify).
I suggest splitting this into two phases:
phase 1: Apply automatic instrumentation to the first Go process found (there is already logic to detect Go processes at analyze.go
phase 2: Apply automatic instrumentation to any Go process found (multi-process support)
Working go-lang auto instrumentation
In sidecar I got this:
{"level":"info","ts":1666358871.3476882,"caller":"cli/main.go:22","msg":"starting Go OpenTelemetry Agent ..."}
{"level":"info","ts":1666358871.3477695,"caller":"opentelemetry/controller.go:92","msg":"Establishing connection to OpenTelemetry collector ..."} โ
{"level":"info","ts":1666358873.3498256,"caller":"process/discover.go:42","msg":"found process","pid":7}
**panic: cant find keyval map**
goroutine 1 [running]:
github.com/keyval-dev/opentelemetry-go-instrumentation/pkg/process.(*processAnalyzer).findKeyvalMmap(0xb51e40?, 0xc0000c0058?) โ
/app/pkg/process/analyze.go:77 +0x14f
github.com/keyval-dev/opentelemetry-go-instrumentation/pkg/process.(*processAnalyzer).Analyze(0xc00009e2a0?, 0x7, 0x2?) /app/pkg/process/analyze.go:103 +0x1a5 main.main() /app/cli/main.go:59 +0x37f
UPDATE:
Fixed panic by adding init container with odigos launcher - I don't understand why do I need add this launcher.
After that error I have another error:
{"level":"info","ts":1666363476.3613198,"caller":"process/analyze.go:73","msg":"found addr of keyval map","addr":140516020199424} โ
{"level":"info","ts":1666363476.3766043,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"net/http.(*ServeMux).ServeHTTP","returns":2} โ
{"level":"info","ts":1666363476.378454,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc/internal/transport.(*http2Client).createHeaderFields","returns":3}
{"level":"info","ts":1666363476.3786323,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc/internal/transport.(*decodeState).decodeHeader","returns":6}
{"level":"info","ts":1666363476.3787887,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc.(*ClientConn).Invoke","returns":2} โ
{"level":"info","ts":1666363476.3791306,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc.(*Server).handleStream","returns":5} โ
{"level":"info","ts":1666363476.379253,"caller":"cli/main.go:64","msg":"target process analysis completed","pid":13,"go_version":"1.15.0","dependencies":{"contrib.go.opencensus.io/exporter/ocagent":"v0.6.0","github.com/beorn7/perks":"v1.0. โ
1","github.com/census-instrumentation/opencensus-proto":"v0.2.1","github.com/cespare/xxhash/v2":"v2.1.1","github.com/golang/groupcache":"v0.0.0-20200121045136-8c9f03a8e57e","github.com/golang/protobuf":"v1.4.0","github.com/grpc-ecosystem/g โ
o-grpc-prometheus":"v1.2.0","github.com/grpc-ecosystem/grpc-gateway":"v1.14.4","github.com/matttproud/golang_protobuf_extensions":"v1.0.1","github.com/prometheus/client_golang":"v1.6.0","github.com/prometheus/client_model":"v0.2.0","github โ
.com/prometheus/common":"v0.9.1","github.com/prometheus/procfs":"v0.0.11","go.opencensus.io":"v0.22.3","golang.org/x/net":"v0.0.0-20200425230154-ff2c4b7c35a0","golang.org/x/sync":"v0.0.0-20200317015054-43a5402ce75a","golang.org/x/sys":"v0. 0.0202004300824071f5687305801","golang.org/x/text":"v0.3.2","google.golang.org/api":"v0.22.0","google.golang.org/genproto":"v0.0.020200430143042b979b6f78d84","google.golang.org/grpc":"v1.29.1","google.golang.org/protobuf":"v1.21.0"},"total_functions_found":5} {"level":"info","ts":1666363476.3793523,"caller":"cli/main.go:70","msg":"invoking instrumentors"}
{"level":"info","ts":1666363476.3992898,"logger":"allocator","caller":"allocator/allocator_linux.go:19","msg":"Loading allocator","start_addr":140516020199424,"end_addr":140516032782336} โ
{"level":"info","ts":1666363476.4011118,"caller":"instrumentors/runner.go:68","msg":"loading instrumentor","name":"google.golang.org/grpc"} โ
{"level":"info","ts":1666363476.4045897,"caller":"inject/injector.go:67","msg":"Injecting variables","vars":{"clientconn_target_ptr_pos":24,"end_addr":140516032782336,"is_registers_abi":false,"start_addr":140516020199424,"total_cpus":4}} โ
{"level":"error","ts":1666363476.4090192,"caller":"instrumentors/runner.go:71","msg":"error while loading instrumentors, cleaning up","name":"google.golang.org/grpc","error":"field UprobeHttp2ClientCreateHeaderFields: program uprobe_Http2C โ
โ lient_CreateHeaderFields: load program: invalid argument: Unrecognized arg#0 type PTR\n; int uprobe_Http2Client_CreateHeaderFields(struct pt_regs *ctx)
go-agent otel version 0.6.0
Kubernetes version:
Output of kubectl version
: Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:25:45Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.14-eks-6d3986b", GitCommit:"8877a3e28d597e1184c15e4b5d543d5dc36b083b", GitTreeState:"clean", BuildDate:"2022-07-20T22:05:32Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"} .......
Maybe the project can come with its own include
directory containing all standard bpf
header files required to build the project, which would make it much easier for everybody to just build the binary locally without bothering about bpf
dependencies.
After cloning the project, the make build
command failed because I did not have my bpf include files in /usr/include/bpf
.
Luckily, I had a personal project on my machine using libbpf
, and so I just copied some of my include files to /usr/include/bpf
and it worked. But for some other people less familiar with bpf
who just wants to try this amazing tool (still in early stage, but looks quite promising) it might be a blocker.
Is this feature request gets accepted, I can submit a PR.
I deployed the emojivoto
example in my local Kubernetes cluster. The emojivoto services, which uses gRPC, are reporting traces in Jaeger backend. This indicates to me, at least at first view, that the gRPC instrumentation in opentelemetry-go-instrumentation
is working correctly
But when I deploy my own demo web server, running with default net/http
package, I expect opentelemetry-go-instrumentation
to detect the request's path and method and to send a trace to the Otel collector.
But none of that happens. I do not see any traces in Jaeger.
I can confirm that my demo webserver, called hello
, is working properly since I am able to communicate with it with curl
commands. Also, the logs are showing that the hello
server is indeed working.
Here is a copy of my own demo server's Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello
namespace: emojivoto
labels:
app.kubernetes.io/name: hello
app.kubernetes.io/part-of: emojivoto
app.kubernetes.io/version: v11
spec:
replicas: 1
selector:
matchLabels:
app: hello-svc
version: v11
template:
metadata:
labels:
app: hello-svc
version: v11
spec:
shareProcessNamespace: true
terminationGracePeriodSeconds: 0
initContainers:
- name: copy-launcher
image: keyval/launcher:v0.1
command:
- cp
- -a
- /kv-launcher/.
- /odigos-launcher/
volumeMounts:
- name: launcherdir
mountPath: /odigos-launcher
containers:
- image: MY_DEMO_IMAGE
name: hello-svc
command:
- /odigos-launcher/launch
- /hello
volumeMounts:
- mountPath: /odigos-launcher
name: launcherdir
ports:
- containerPort: 4444
name: http
resources:
requests:
cpu: 100m
- name: emojivoto-web-instrumentation
image: keyval/otel-go-agent:v0.6.0
env:
- name: OTEL_TARGET_EXE
value: /hello
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: "jaeger:4317"
- name: OTEL_SERVICE_NAME
value: "emojivoto-web"
securityContext:
runAsUser: 0
capabilities:
add:
- SYS_PTRACE
privileged: true
volumeMounts:
- mountPath: /sys/kernel/debug
name: kernel-debug
volumes:
- name: launcherdir
emptyDir: {}
- name: kernel-debug
hostPath:
path: /sys/kernel/debug
---
apiVersion: v1
kind: Service
metadata:
name: hello-svc
namespace: emojivoto
spec:
type: NodePort
ports:
- name: http
port: 8080
targetPort: 8080
selector:
app: hello-svc
I can also confirm that the instrumentation sidecar is able to discover my hello server's process because the logs says so:
{"level":"info","ts":1669306652.8174577,"caller":"cli/main.go:22","msg":"starting Go OpenTelemetry Agent ..."}
{"level":"info","ts":1669306652.817512,"caller":"opentelemetry/controller.go:92","msg":"Establishing connection to OpenTelemetry collector ..."}
{"level":"info","ts":1669306654.818036,"caller":"process/discover.go:42","msg":"found process","pid":13}
{"level":"info","ts":1669306654.818234,"caller":"process/analyze.go:73","msg":"found addr of keyval map","addr":140388705247232}
{"level":"info","ts":1669306654.8208957,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"net/http.(*ServeMux).ServeHTTP","returns":2}
{"level":"info","ts":1669306654.8209321,"caller":"cli/main.go:64","msg":"target process analysis completed","pid":13,"go_version":"1.18.1","dependencies":{},"total_functions_found":1}
{"level":"info","ts":1669306654.8209662,"caller":"cli/main.go:70","msg":"invoking instrumentors"}
{"level":"info","ts":1669306654.8252027,"logger":"allocator","caller":"allocator/allocator_linux.go:19","msg":"Loading allocator","start_addr":140388705247232,"end_addr":140388717830144}
{"level":"info","ts":1669306654.826382,"caller":"instrumentors/runner.go:68","msg":"loading instrumentor","name":"net/http"}
{"level":"info","ts":1669306654.826874,"caller":"inject/injector.go:67","msg":"Injecting variables","vars":{"ctx_ptr_pos":232,"is_registers_abi":true,"method_ptr_pos":0,"path_ptr_pos":56,"url_ptr_pos":16}}
{"level":"info","ts":1669306654.8707218,"caller":"instrumentors/runner.go:77","msg":"loaded instrumentors to memory","total_instrumentors":1}
Further more, every time I send a request to my hello
server, the the tool opentelemetry-go-instrumentation
is detecting the request and writing a log to the console, but in the logs the request's path and method are empty, as we can see in the logs here:
{"level":"info","ts":1669306673.0694988,"caller":"opentelemetry/controller.go:44","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":""}},{"Key":"http.target","Value":{"Type":"STRING","Value":""}}]}
{"level":"info","ts":1669306673.9885561,"caller":"opentelemetry/controller.go:44","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":""}},{"Key":"http.target","Value":{"Type":"STRING","Value":""}}]}
{"level":"info","ts":1669306674.4386296,"caller":"opentelemetry/controller.go:44","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":""}},{"Key":"http.target","Value":{"Type":"STRING","Value":""}}]}
{"level":"info","ts":1669306674.734222,"caller":"opentelemetry/controller.go:44","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":""}},{"Key":"http.target","Value":{"Type":"STRING","Value":""}}]}
{"level":"info","ts":1669306674.9497232,"caller":"opentelemetry/controller.go:44","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":""}},{"Key":"http.target","Value":{"Type":"STRING","Value":""}}]}
{"level":"info","ts":1669306796.7762065,"caller":"opentelemetry/controller.go:44","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":""}},{"Key":"http.target","Value":{"Type":"STRING","Value":""}}]}
After some digging, I have found that the eBPF probe for net/http
seems broken. When debugging with the eBPF trace pipe, I can see that in the uprobe uprobe_ServerMux_ServeHTTP_Returns
, the *httpRequest
object, which is fetched from CPU register rdi
at the end of the function, does not yield any valid *httpRequest
pointer.
In other words, when using bpf_printk()
, I can print the request's method in uprobe_ServerMux_ServeHTTP
and it works.
u64 request_pos = 4;
struct http_request_t httpReq = {};
httpReq.start_time = bpf_ktime_get_boot_ns();
// Get request struct
void* req_ptr = get_argument(ctx, request_pos);
// Get method from request
void* method_ptr = 0;
bpf_probe_read(&method_ptr, sizeof(method_ptr), (void *)(req_ptr+method_ptr_pos));
u64 method_len = 0;
bpf_probe_read(&method_len, sizeof(method_len), (void *)(req_ptr+(method_ptr_pos+8)));
u64 method_size = sizeof(httpReq.method);
method_size = method_size < method_len ? method_size : method_len;
bpf_probe_read(&httpReq.method, method_size, method_ptr);
bpf_printk("method is %s\n", httpReq.method); // this prints the correct method to /sys/kernel/debug/tracing/trace_pipe
But when adding some code in the return probe (uprobe_ServerMux_ServeHTTP_Returns
) to test the same thing, the method is an empty string. Same for the request's path.
Also, there seems to be an issue with the switch to context's address as eBPF map keys instead of goroutine IDs. In both entry and return probe, the context address is different. This might be because the idiomatic way to modify the context of a request in Go is to create a new context instead. This changes the location of the context in memory, and so its own memory address. It seems that the current implementation does not take that into account.
I would gladly work on that and submit a PR if I can have more details as to why the net/http
instrumentation is failing. Is it suppose to fail? Is it not fully implemented yet (but it works a few months ago when the goroutine
instrumentation was in place if I remember correctly)?
Let me know and I can lend a hand.
net/http
libraryopen telemetry-go-instrumentation
as explained in the docs/getting-started
guideKubernetes version: v1.24.0
Output of kubectl version
:
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:46:05Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:38:19Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
operation-system/kernel version:
Output of awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release
: Ubuntu 22.04 LTS
Output of uname -r
: 5.15.0-53-generic
I have a Go process running on my machine, which is a web server using net/http
. The executable path is /home/ocampeau/hello
When I run the following command, I expect the tool to emit traces when an HTTP request is received by my process
sudo OTEL_TARGET_EXE=/home/olivier/hello ./kv-go-instrumentation
Instead, I get a panic
panic: cant find keyval map
sudo OTEL_TARGET_EXE=PATH_TO_EXECUTABLE ./kv-go-instrumentation
My guess is that there is a bug with the permission check of the memory mapping of the process.
When I inspect /proc/PID/maps, I see this:
00400000-00612000 r-xp 00000000 103:02 17174730 /home/olivier/hello
00612000-0080d000 r--p 00212000 103:02 17174730 /home/olivier/hello
0080d000-0084a000 rw-p 0040d000 103:02 17174730 /home/olivier/hello
0084a000-00883000 rw-p 00000000 00:00 0
0248f000-024b0000 rw-p 00000000 00:00 0 [heap]
c000000000-c000400000 rw-p 00000000 00:00 0
c000400000-c004000000 ---p 00000000 00:00 0
7f45f4000000-7f45f4021000 rw-p 00000000 00:00 0
7f45f4021000-7f45f8000000 ---p 00000000 00:00 0
7f45f8000000-7f45f8021000 rw-p 00000000 00:00 0
7f45f8021000-7f45fc000000 ---p 00000000 00:00 0
7f45fc000000-7f45fc021000 rw-p 00000000 00:00 0
7f45fc021000-7f4600000000 ---p 00000000 00:00 0
7f4600000000-7f4600021000 rw-p 00000000 00:00 0
...
It seems that the code segment referenced in the memory mapping of the process has READ and EXECUTABLE permissions, but not WRITE. In this case, the start address of the code segment would be 00400000
and the end address would be 00612000
.
When I update the following code to remove the check on the write permissions, the tool is indeed working as expected:
func (a *processAnalyzer) findKeyvalMmap(pid int) (uintptr, uintptr) {
fs, err := procfs.NewProc(pid)
if err != nil {
panic(err)
}
maps, err := fs.ProcMaps()
if err != nil {
panic(err)
}
for _, m := range maps {
if m.Perms != nil && m.Perms.Read && m.Perms.Execute { // I removed the check on write permission here
log.Logger.Info("found addr of keyval map", "addr", m.StartAddr)
return m.StartAddr, m.EndAddr
}
}
panic(errors.New("cant find keyval map"))
Hello, Id like to clarify your design decisions (for learning purposes and to share knowledge, hopefully, both ways), if you allow me. Sorry to post you an issue, but you didn't have "discussions" option set.
Based on your documentation, you say:
Instrumentation Stability
eBPF programs access user code and variables by analyzing the stack and the CPU registers. For example, to read the value of the target field in the google.golang.org/grpc.ClientConn struct (see gRPC instrumentor for an example), the eBPF program needs to know the offset of the field inside the struct. The offset is determined by the field location inside the struct definition.
This sounds to me like .BTF
and .BTF.ext
sections in eBPF objects, placed by LLVM, so libbpf can read the RELO
information and calculate relocations based on some speculations.
Hard coding this offset information into the eBPF programs creates a very unstable instrumentation. Fields locations inside structs are subject to change and the eBPF program needs to be recompiled every time the struct definition changes.
Wouldn't it be possible to make golang compiler to place .BTF/.BTF.ext like information into the generated objects ? Just like LLVM does for eBPF objects into specific ELF sections ?
Luckily for us, there is a way to analyze the target binary and extract the required offsets, by using DWARF. The DWARF debug information is generated by the compiler and is stored inside the binary.
This is similar of creating an external BTF files w/ pahole, representing kernels (which is analog to the userland binary you will place uprobes).
Notice that one of our design goals is to support stripped Go binaries - meaning binaries that do not contain debug information. In order to support stripped binaries and to create a stable instrumentation, we created a library called [offsets-tracker](https://github.com/keyval-dev/offsets-tracker). This library tracks the offset of different fields across versions.
Couldn't you generate RAW BTF files from all existing DWARF symbols ? And then use the BTF info (in a more generic way, instead of only picking up offsets for particular structs, you could get for all types used).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.