keyval-dev/opentelemetry-go-instrumentation

OpenTelemetry auto-instrumentation for Go applications

OpenTelemetry Auto-Instrumentation for Go

This project adds OpenTelemetry instrumentation to Go applications without having to modify their source code. We support a wide range of Go versions (1.12+) and even work on stripped binaries.

Our goal is to provide the same level of automatic instrumentation for Go as exists for languages such as Java and Python.

This automatic instrumentation is based on eBPF uprobes.

For more information, see our How it works document.

Getting Started

Check out our Getting Started on Kubernetes guide for easily instrumenting your first Go applications.

Current Instrumentations

Library/Framework
net/http - Server
gRPC - Client
gRPC - Server
Gorilla Mux

Project Status

This project is actively maintained by keyval and is currently in its initial days. We would love to receive your ideas, feedback & contributions.

Contributing

Please refer to the contributing.md file for information about how to get involved. We welcome issues, questions, and pull requests.

License

This project is licensed under the terms of the Apache 2.0 open source license. Please refer to LICENSE for the full terms.

opentelemetry-go-instrumentation's Issues

Can't select the emojivoto-voting service on Jaeger UI page

I follow the steps on the tutorial Getting Started on Kubernetes, but I can't select the emojivoto-voting service on Jaeger UI page, what happend?

root@localhost:~ # kubectl get no
NAME                 STATUS   ROLES                  AGE     VERSION
kind-control-plane   Ready    control-plane,master   2d14h   v1.23.4

root@localhost:~ # kubectl get svc -n emojivoto
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGE
emoji-svc    ClusterIP   10.96.195.205   <none>        8080/TCP,8801/TCP    2d13h
jaeger       ClusterIP   10.96.8.171     <none>        4317/TCP,16686/TCP   2d13h
voting-svc   ClusterIP   10.96.175.64    <none>        8080/TCP,8801/TCP    40h
web-svc      ClusterIP   10.96.209.28    <none>        80/TCP               2d13h

root@localhost:~ # kubectl get po -n emojivoto
NAME                        READY   STATUS    RESTARTS      AGE
emoji-5dbdd567bd-kcmxk      1/1     Running   1 (42h ago)   2d13h
jaeger-54f8b6d89d-mpqtj     1/1     Running   1 (42h ago)   2d13h
vote-bot-58b4f5fdb7-f578j   1/1     Running   1 (42h ago)   2d13h
voting-5fdcddcfc-hdcxh      1/1     Running   0             40h
web-67c857998c-d9sjj        1/1     Running   1 (42h ago)   2d13h

root@localhost:~ # kubectl patch deployment voting --patch-file voting-patch.yaml -n emojivoto
deployment.apps/voting patched (no change)

root@localhost:~ # kubectl port-forward svc/jaeger --address 0.0.0.0 16686:16686 -n emojivoto
Forwarding from 0.0.0.0:16686 -> 16686
Handling connection for 16686
Handling connection for 16686

Code segment in process memory mapping not always writable

Expected Behavior

I have a Go process running on my machine, which is a web server using net/http. The executable path is /home/ocampeau/hello

When I run the following command, I expect the tool to emit traces when an HTTP request is received by my process

sudo OTEL_TARGET_EXE=/home/olivier/hello ./kv-go-instrumentation

Actual Behavior

Instead, I get a panic

panic: cant find keyval map

Steps to Reproduce the Problem

1. Build an http server in Go
2. Compile the current version of opentelemetry-go-instrumentation
3. Run sudo OTEL_TARGET_EXE=PATH_TO_EXECUTABLE ./kv-go-instrumentation

Additional Info

My guess is that there is a bug with the permission check of the memory mapping of the process.

When I inspect /proc/PID/maps, I see this:

00400000-00612000 r-xp 00000000 103:02 17174730                          /home/olivier/hello
00612000-0080d000 r--p 00212000 103:02 17174730                            /home/olivier/hello
0080d000-0084a000 rw-p 0040d000 103:02 17174730                          /home/olivier/hello
0084a000-00883000 rw-p 00000000 00:00 0 
0248f000-024b0000 rw-p 00000000 00:00 0                                           [heap]
c000000000-c000400000 rw-p 00000000 00:00 0 
c000400000-c004000000 ---p 00000000 00:00 0 
7f45f4000000-7f45f4021000 rw-p 00000000 00:00 0 
7f45f4021000-7f45f8000000 ---p 00000000 00:00 0 
7f45f8000000-7f45f8021000 rw-p 00000000 00:00 0 
7f45f8021000-7f45fc000000 ---p 00000000 00:00 0 
7f45fc000000-7f45fc021000 rw-p 00000000 00:00 0 
7f45fc021000-7f4600000000 ---p 00000000 00:00 0 
7f4600000000-7f4600021000 rw-p 00000000 00:00 0 
...

It seems that the code segment referenced in the memory mapping of the process has READ and EXECUTABLE permissions, but not WRITE. In this case, the start address of the code segment would be 00400000 and the end address would be 00612000.

When I update the following code to remove the check on the write permissions, the tool is indeed working as expected:

func (a *processAnalyzer) findKeyvalMmap(pid int) (uintptr, uintptr) {
	fs, err := procfs.NewProc(pid)
	if err != nil {
		panic(err)
	}

	maps, err := fs.ProcMaps()
	if err != nil {
		panic(err)
	}

	for _, m := range maps {
		if m.Perms != nil && m.Perms.Read && m.Perms.Execute { // I removed the check on write permission here
			log.Logger.Info("found addr of keyval map", "addr", m.StartAddr)
			return m.StartAddr, m.EndAddr
		}
	}
	panic(errors.New("cant find keyval map"))

Adding this to KoolKits?

Hiya!

I've caught this repo from Observability News by Michael Hasuenblas - great stuff:)

I wonder, since this requires 0 code changes, whether it will be a good fit to bundle into our OSS project KoolKits.

We have docker debug images specifically geared towards debugging applications written in a specific language - you can check out the project here.

This might fit into our Go KoolKit.

Your comments much appreciated @edeNFed :)

[Bug/Question] Same Trace ID for different requests

Hi keyval-dev team,

Thanks for making this library for auto instrumentation otel in go.
Before read the report below, i would like to ask how do you group spans into one tracing id?
is it coming from same goroutine id?

Thanks,

Expected Behavior

Different request different tracing id

Actual Behavior

Different request same tracing id, (idk, maybe because golang/mux reused goroutine for another requests)

Steps to Reproduce the Problem

1. Add patch yaml for running k8s golang service
2. Add jaeger
3. Wait 5 minutes, and will be like this.

Additional Info

• Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.12-gke.500", GitCommit:"26604476cf3bcb64bec486e4b5880a5773a52e75", GitTreeState:"clean", BuildDate:"2022-07-20T09:22:48Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}

• operation-system/kernel version:
  Darwin 21.1.0

• go version:
  1.17

Doubt about trace ID

After read the whole codes, I don't find any codes to send the trace ID in the different services. May be I don't find it. Could you share the parts for me? Thanks a lot ~

use daemonset way not sidecar way

Feature request

use daemonset way not sidecar way

Use case

May be consider daemonset that it will deploy agent on nodes,which it will collect data by pid, and the use pid association with pod metadata.

what do you think? @edeNFed

Support a secure grpc endpoint?

Feature request

I'm using Honeycomb as my otel endpoint directly, without going to a local collector first. It looks like I'm unable to connect to their endpoint: api.honeycomb.io:443 because the gRPC client created by this project is requiring an insecure connection:

https://github.com/keyval-dev/opentelemetry-go-instrumentation/blob/a3b51e5242ff7026120530fa8aa3b0bbd4d5e170/pkg/opentelemetry/controller.go#L99

I would like to be able to configure that the gRPC client is configured expecting a secure connection so that I can connect directly to honeycomb.

Any limitations to this approach?

Hey, this is fabulous! :)

I'm wondering what (if any) fundamental limitations exist with this approach? For example:

• Could auto-instrumentation be mixed with regular instrumentation code? Can auto-instrumentation register Providers that instrumentation code can make use of? Is the context object populated with the usual Otel data?
• Going the other direction, could auto-instrumentation make use of Providers that are written in code and registered in the normal manner? For example, to allow custom SpanProcessors and other plugins to be installed.

Again, really awesome project, I'm excited to see where this goes!

How to add request-id/trace-id to http request header?

Feature request

Allow opentelemetry-go-instrumentation trace http requests.

How to implement?

A Golang project may send http request to other servers through Golang net/request, for doing distributed tracing need to add the trace id in http headers.

The header is a map https://cs.opensource.google/go/go/+/refs/tags/go1.19.2:src/net/http/request.go;l=148.

If we do this through uprobe, we need edit a Golang map.

So how could we do this? Or do we have other options?

I guess eBPF program can't call a user's function easily, so we have to mange the memory, but map is complex, not sure how to do this.

Maybe we can calculate where we can put the key/value pair(we know the key and can know the length of value), and use the result to do memory management?

Remove OTEL_TARGET_EXE and auto discover processes

This issue is for removing the OTEL_TARGET_EXE env variable and implementing process discovery instead.
Implementing this will make integration with OpenTelemetry Operator easier and overall provide a better experience to users (one less argument to specify).

I suggest splitting this into two phases:
phase 1: Apply automatic instrumentation to the first Go process found (there is already logic to detect Go processes at analyze.go
phase 2: Apply automatic instrumentation to any Go process found (multi-process support)

Go service is stuck in crashloopbackoff after instrumentation. Am I doing something wrong?

Expected Behavior

kubectl get pods -n atlas
pod/go-service-8c86cf596-5rbpc                 2/2     Running

Actual Behavior

kubectl get pods -n atlas
pod/go-service-8c86cf596-5rbpc                 1/2     CrashLoopBackOff

Steps to Reproduce the Problem

Here's my manifest file relevant for go:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: go-service
  name: go-service
  namespace: atlas
spec:
  ports:
  - port: 30002
    protocol: TCP
    targetPort: 30002
    # If you set the `spec.type` field to `NodePort` and you want a specific port number,
    # you can specify a value in the `spec.ports[*].nodePort` field.
    nodePort: 30002
  selector:
    app: go-service
  type: NodePort
status:
  loadBalancer: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: go-service
  name: go-service
  namespace: atlas
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-service
  strategy: {}
  template:
    metadata:
      labels:
        app: go-service
    spec:
      shareProcessNamespace: true
      initContainers:
        - name: copy-launcher
          image: keyval/launcher:v0.1
          command:
            - cp
            - -a
            - /kv-launcher/.
            - /odigos-launcher/
          volumeMounts:
            - name: launcherdir
              mountPath: /odigos-launcher
      containers:
        - args:
          - go-service
          image: dev0zklabs/atlas-demo-microservice:go-service
          name: go-service
          command:
            - /odigos-launcher/launch
            - /root/go-service
          volumeMounts:
            - mountPath: /odigos-launcher
              name: launcherdir
          ports:
          - containerPort: 30002
          imagePullPolicy: Always
          envFrom:
            - configMapRef:
                name: envs-config
          resources: {}
        - name: emojivoto-emoji-instrumentation
          image: keyval/otel-go-agent:v0.6.4
          env:
            - name: OTEL_TARGET_EXE
              value: /root/go-service
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "jaeger:4317"
            - name: OTEL_SERVICE_NAME
              value: "go-service"
          securityContext:
            runAsUser: 0
            capabilities:
              add:
                - SYS_PTRACE
            privileged: true
          volumeMounts:
            - mountPath: /sys/kernel/debug
              name: kernel-debug
      volumes:
        - name: launcherdir
          emptyDir: {}
        - name: kernel-debug
          hostPath:
            path: /sys/kernel/debug
status: {}

Additional Info

• Kubernetes version:

  Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:40:17Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25+", GitVersion:"v1.25.8-eks-ec5523e", GitCommit:"83fe90de881ffe7450876d204ccdc37a0f95bda3", GitTreeState:"clean", BuildDate:"2023-03-20T21:32:06Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}

- operation-system/kernel version:

**Output of `awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release`:**
**Output of `uname -r`:**

Ubuntu 22.04.2 LTS
5.19.0-1022-aws

<!-- Any other additional information -->

Doubt about design decision

Hello, Id like to clarify your design decisions (for learning purposes and to share knowledge, hopefully, both ways), if you allow me. Sorry to post you an issue, but you didn't have "discussions" option set.

Based on your documentation, you say:

Instrumentation Stability
eBPF programs access user code and variables by analyzing the stack and the CPU registers. For example, to read the value of the target field in the google.golang.org/grpc.ClientConn struct (see gRPC instrumentor for an example), the eBPF program needs to know the offset of the field inside the struct. The offset is determined by the field location inside the struct definition.

This sounds to me like .BTF and .BTF.ext sections in eBPF objects, placed by LLVM, so libbpf can read the RELO information and calculate relocations based on some speculations.

Hard coding this offset information into the eBPF programs creates a very unstable instrumentation. Fields locations inside structs are subject to change and the eBPF program needs to be recompiled every time the struct definition changes.

Wouldn't it be possible to make golang compiler to place .BTF/.BTF.ext like information into the generated objects ? Just like LLVM does for eBPF objects into specific ELF sections ?

Luckily for us, there is a way to analyze the target binary and extract the required offsets, by using DWARF. The DWARF debug information is generated by the compiler and is stored inside the binary.

This is similar of creating an external BTF files w/ pahole, representing kernels (which is analog to the userland binary you will place uprobes).

Notice that one of our design goals is to support stripped Go binaries - meaning binaries that do not contain debug information. In order to support stripped binaries and to create a stable instrumentation, we created a library called [offsets-tracker](https://github.com/keyval-dev/offsets-tracker). This library tracks the offset of different fields across versions.

Couldn't you generate RAW BTF files from all existing DWARF symbols ? And then use the BTF info (in a more generic way, instead of only picking up offsets for particular structs, you could get for all types used).

can this work with serverless platforms like Cloud Run?

Feature request

I'm largely deploying Go apps to Cloud Run and automatic instrumentation would be awesome. The docs only refer to k8s setups.

Use case

k8s is pretty complicated and for my apps, Cloud Run is a good match. I imagine other folks are deploying Go apps using other serverless platforms, too, and it would be helpful to be able to use this here vs. manual instrumentation.

Error to trace

Expected Behavior

Trace the data

Actual Behavior

Not tracing the data

Steps to Reproduce the Problem

1. Installed and configured my deployment with the launcher
2. the instrument connect to the jaeger service and start to try to send logs
3. show some errors on the jaeger logs

Additional Info

• Kubernetes version:

  Output of 1.23:

go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).startProcessingCycle
go.opentelemetry.io/[email protected]/processor/batchprocessor/batch_processor.go:144
2022-10-11T18:50:12.446Z WARN batchprocessor/batch_processor.go:187 Sender failed {"component_kind": "processor", "component_type": "batch", "component_name": "batch", "error": "Permanent error: OC span has an all zeros trace ID"}
go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).sendItems
go.opentelemetry.io/[email protected]/processor/batchprocessor/batch_processor.go:187
go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).startProcessingCycle
go.opentelemetry.io/[email protected]/processor/batchprocessor/batch_processor.go:144
2022-10-11T19:05:23.386Z ERROR exporterhelper/queued_retry.go:204 Exporting failed. Try enabling retry_on_failure config option. {"component_kind": "exporter", "component_type": "jaeger_memory", "component_name": "jaeger_memory", "error": "Permanent error: OC span has an all zeros trace ID"}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:204
go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
go.opentelemetry.io/[email protected]/exporter/exporterhelper/tracehelper.go:114 ```

• operation-system/kernel version:

  Output of awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release:
  Output of uname -r:

  (paste your output here)
  

`net/http` instrumentation (eBPF probe) is not working

Expected Behavior

I deployed the emojivoto example in my local Kubernetes cluster. The emojivoto services, which uses gRPC, are reporting traces in Jaeger backend. This indicates to me, at least at first view, that the gRPC instrumentation in opentelemetry-go-instrumentation is working correctly

But when I deploy my own demo web server, running with default net/http package, I expect opentelemetry-go-instrumentation to detect the request's path and method and to send a trace to the Otel collector.

Actual Behavior

But none of that happens. I do not see any traces in Jaeger.

I can confirm that my demo webserver, called hello, is working properly since I am able to communicate with it with curl commands. Also, the logs are showing that the hello server is indeed working.

Here is a copy of my own demo server's Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  namespace: emojivoto
  labels:
    app.kubernetes.io/name: hello
    app.kubernetes.io/part-of: emojivoto
    app.kubernetes.io/version: v11
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-svc
      version: v11
  template:
    metadata:
      labels:
        app: hello-svc
        version: v11
    spec:
      shareProcessNamespace: true
      terminationGracePeriodSeconds: 0
      initContainers:
        - name: copy-launcher
          image: keyval/launcher:v0.1
          command:
            - cp
            - -a
            - /kv-launcher/.
            - /odigos-launcher/
          volumeMounts:
            - name: launcherdir
              mountPath: /odigos-launcher
      containers:
        - image: MY_DEMO_IMAGE
          name: hello-svc
          command:
            - /odigos-launcher/launch
            - /hello
          volumeMounts:
            - mountPath: /odigos-launcher
              name: launcherdir
          ports:
            - containerPort: 4444
              name: http
          resources:
            requests:
              cpu: 100m
        - name: emojivoto-web-instrumentation
          image: keyval/otel-go-agent:v0.6.0
          env:
            - name: OTEL_TARGET_EXE
              value: /hello
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "jaeger:4317"
            - name: OTEL_SERVICE_NAME
              value: "emojivoto-web"
          securityContext:
            runAsUser: 0
            capabilities:
              add:
                - SYS_PTRACE
            privileged: true
          volumeMounts:
            - mountPath: /sys/kernel/debug
              name: kernel-debug
      volumes:
        - name: launcherdir
          emptyDir: {}
        - name: kernel-debug
          hostPath:
            path: /sys/kernel/debug
---
apiVersion: v1
kind: Service
metadata:
  name: hello-svc
  namespace: emojivoto
spec:
  type: NodePort
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: hello-svc

I can also confirm that the instrumentation sidecar is able to discover my hello server's process because the logs says so:

{"level":"info","ts":1669306652.8174577,"caller":"cli/main.go:22","msg":"starting Go OpenTelemetry Agent ..."}
{"level":"info","ts":1669306652.817512,"caller":"opentelemetry/controller.go:92","msg":"Establishing connection to OpenTelemetry collector ..."}
{"level":"info","ts":1669306654.818036,"caller":"process/discover.go:42","msg":"found process","pid":13}
{"level":"info","ts":1669306654.818234,"caller":"process/analyze.go:73","msg":"found addr of keyval map","addr":140388705247232}
{"level":"info","ts":1669306654.8208957,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"net/http.(*ServeMux).ServeHTTP","returns":2}
{"level":"info","ts":1669306654.8209321,"caller":"cli/main.go:64","msg":"target process analysis completed","pid":13,"go_version":"1.18.1","dependencies":{},"total_functions_found":1}
{"level":"info","ts":1669306654.8209662,"caller":"cli/main.go:70","msg":"invoking instrumentors"}
{"level":"info","ts":1669306654.8252027,"logger":"allocator","caller":"allocator/allocator_linux.go:19","msg":"Loading allocator","start_addr":140388705247232,"end_addr":140388717830144}
{"level":"info","ts":1669306654.826382,"caller":"instrumentors/runner.go:68","msg":"loading instrumentor","name":"net/http"}
{"level":"info","ts":1669306654.826874,"caller":"inject/injector.go:67","msg":"Injecting variables","vars":{"ctx_ptr_pos":232,"is_registers_abi":true,"method_ptr_pos":0,"path_ptr_pos":56,"url_ptr_pos":16}}
{"level":"info","ts":1669306654.8707218,"caller":"instrumentors/runner.go:77","msg":"loaded instrumentors to memory","total_instrumentors":1}

Further more, every time I send a request to my hello server, the the tool opentelemetry-go-instrumentation is detecting the request and writing a log to the console, but in the logs the request's path and method are empty, as we can see in the logs here:

{"level":"info","ts":1669306673.0694988,"caller":"opentelemetry/controller.go:44","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":""}},{"Key":"http.target","Value":{"Type":"STRING","Value":""}}]}
{"level":"info","ts":1669306673.9885561,"caller":"opentelemetry/controller.go:44","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":""}},{"Key":"http.target","Value":{"Type":"STRING","Value":""}}]}
{"level":"info","ts":1669306674.4386296,"caller":"opentelemetry/controller.go:44","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":""}},{"Key":"http.target","Value":{"Type":"STRING","Value":""}}]}
{"level":"info","ts":1669306674.734222,"caller":"opentelemetry/controller.go:44","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":""}},{"Key":"http.target","Value":{"Type":"STRING","Value":""}}]}
{"level":"info","ts":1669306674.9497232,"caller":"opentelemetry/controller.go:44","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":""}},{"Key":"http.target","Value":{"Type":"STRING","Value":""}}]}
{"level":"info","ts":1669306796.7762065,"caller":"opentelemetry/controller.go:44","msg":"got event","attrs":[{"Key":"http.method","Value":{"Type":"STRING","Value":""}},{"Key":"http.target","Value":{"Type":"STRING","Value":""}}]}

After some digging, I have found that the eBPF probe for net/http seems broken. When debugging with the eBPF trace pipe, I can see that in the uprobe uprobe_ServerMux_ServeHTTP_Returns, the *httpRequest object, which is fetched from CPU register rdi at the end of the function, does not yield any valid *httpRequest pointer.

In other words, when using bpf_printk(), I can print the request's method in uprobe_ServerMux_ServeHTTP and it works.

u64 request_pos = 4;
struct http_request_t httpReq = {};
httpReq.start_time = bpf_ktime_get_boot_ns();

// Get request struct
void* req_ptr = get_argument(ctx, request_pos);

// Get method from request
void* method_ptr = 0;
bpf_probe_read(&method_ptr, sizeof(method_ptr), (void *)(req_ptr+method_ptr_pos));
u64 method_len = 0;
bpf_probe_read(&method_len, sizeof(method_len), (void *)(req_ptr+(method_ptr_pos+8)));
u64 method_size = sizeof(httpReq.method);
method_size = method_size < method_len ? method_size : method_len;
bpf_probe_read(&httpReq.method, method_size, method_ptr);
bpf_printk("method is %s\n", httpReq.method); // this prints the correct method to /sys/kernel/debug/tracing/trace_pipe

But when adding some code in the return probe (uprobe_ServerMux_ServeHTTP_Returns) to test the same thing, the method is an empty string. Same for the request's path.

Also, there seems to be an issue with the switch to context's address as eBPF map keys instead of goroutine IDs. In both entry and return probe, the context address is different. This might be because the idiomatic way to modify the context of a request in Go is to create a new context instead. This changes the location of the context in memory, and so its own memory address. It seems that the current implementation does not take that into account.

I would gladly work on that and submit a PR if I can have more details as to why the net/http instrumentation is failing. Is it suppose to fail? Is it not fully implemented yet (but it works a few months ago when the goroutine instrumentation was in place if I remember correctly)?

Let me know and I can lend a hand.

Steps to Reproduce the Problem

1. Create a simple web server using net/http library
2. Deploy the server with the open telemetry-go-instrumentation as explained in the docs/getting-started guide
3. Look in Jaeger UI to see there is no traces

Additional Info

• Kubernetes version: v1.24.0

  Output of kubectl version:

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:46:05Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:38:19Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}

• operation-system/kernel version:

  Output of awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release: Ubuntu 22.04 LTS

  Output of uname -r: 5.15.0-53-generic

Error during instrumentation: `invalid argument: ;`

Expected Behavior

Auto Instrumentation works without issues.

Actual Behavior

Sidecar container halts with error.

{"level":"info","ts":1666009431.26208,"caller":"cli/main.go:22","msg":"starting Go OpenTelemetry Agent ..."}
{"level":"info","ts":1666009431.2621613,"caller":"opentelemetry/controller.go:92","msg":"Establishing connection to OpenTelemetry collector ..."}
{"level":"info","ts":1666009433.2638783,"caller":"process/discover.go:42","msg":"found process","pid":13}
{"level":"info","ts":1666009433.2643228,"caller":"process/analyze.go:73","msg":"found addr of keyval map","addr":139872935333888}
{"level":"info","ts":1666009433.281399,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"net/http.(*ServeMux).ServeHTTP","returns":2}
{"level":"info","ts":1666009433.2829392,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc/internal/transport.(*http2Client).createHeaderFields","returns":3}
{"level":"info","ts":1666009433.2831843,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc.(*ClientConn).Invoke","returns":2}
{"level":"info","ts":1666009433.2836869,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc.(*Server).handleStream","returns":5}
{"level":"info","ts":1666009433.2840135,"caller":"cli/main.go:64","msg":"target process analysis completed","pid":13,"go_version":"1.17.11","dependencies":{"github.com/FZambia/eagle":"v0.0.2","github.com/FZambia/sentinel":"v1.1.0","github.com/FZambia/statik":"v0.1.2-0.20180217151304-b9f012bb2a1b","github.com/FZambia/tarantool":"v0.3.1","github.com/FZambia/viper-lite":"v0.0.0-20220110144934-1899f66c7d0e","github.com/beorn7/perks":"v1.0.1","github.com/centrifugal/centrifuge":"v0.23.1","github.com/centrifugal/protocol":"v0.8.8","github.com/cespare/xxhash/v2":"v2.1.2","github.com/cristalhq/jwt/v4":"v4.0.0","github.com/gobwas/glob":"v0.2.3","github.com/golang/protobuf":"v1.5.2","github.com/gomodule/redigo":"v1.8.8","github.com/google/uuid":"v1.3.0","github.com/gorilla/securecookie":"v1.1.1","github.com/gorilla/websocket":"v1.5.0","github.com/igm/sockjs-go/v3":"v3.0.2","github.com/josharian/intern":"v1.0.0","github.com/mailru/easyjson":"v0.7.7","github.com/mattn/go-isatty":"v0.0.12","github.com/matttproud/golang_protobuf_extensions":"v1.0.1","github.com/mitchellh/mapstructure":"v1.4.3","github.com/mna/redisc":"v1.3.2","github.com/nats-io/nats.go":"v1.14.0","github.com/nats-io/nkeys":"v0.3.0","github.com/nats-io/nuid":"v1.0.1","github.com/pelletier/go-toml":"v1.9.4","github.com/prometheus/client_golang":"v1.12.1","github.com/prometheus/client_model":"v0.2.0","github.com/prometheus/common":"v0.34.0","github.com/prometheus/procfs":"v0.7.3","github.com/rakutentech/jwk-go":"v1.0.1","github.com/rs/zerolog":"v1.21.0","github.com/segmentio/asm":"v1.1.4","github.com/segmentio/encoding":"v0.3.5","github.com/spf13/cast":"v1.4.1","github.com/spf13/cobra":"v0.0.7","github.com/spf13/jwalterweatherman":"v1.1.0","github.com/spf13/pflag":"v1.0.5","github.com/valyala/bytebufferpool":"v1.0.0","github.com/vmihailenco/msgpack/v5":"v5.3.5","github.com/vmihailenco/tagparser/v2":"v2.0.0","golang.org/x/crypto":"v0.0.0-20220411220226-7b82a4e95df4","golang.org/x/net":"v0.0.0-20220421235706-1d1ef9303861","golang.org/x/sync":"v0.0.0-20210220032951-036812b2e83c","golang.org/x/sys":"v0.0.0-20220422013727-9388b58f7150","golang.org/x/text":"v0.3.7","google.golang.org/genproto":"v0.0.0-20220422154200-b37d22cd5731","google.golang.org/grpc":"v1.46.0","google.golang.org/protobuf":"v1.28.0","gopkg.in/yaml.v2":"v2.4.0"},"total_functions_found":4}
{"level":"info","ts":1666009433.2841408,"caller":"cli/main.go:70","msg":"invoking instrumentors"}
{"level":"info","ts":1666009433.312439,"logger":"allocator","caller":"allocator/allocator_linux.go:19","msg":"Loading allocator","start_addr":139872935333888,"end_addr":139872947916800}
{"level":"info","ts":1666009433.3126094,"caller":"instrumentors/runner.go:68","msg":"loading instrumentor","name":"google.golang.org/grpc"}
{"level":"info","ts":1666009433.3144336,"caller":"inject/injector.go:67","msg":"Injecting variables","vars":{"clientconn_target_ptr_pos":24,"end_addr":139872947916800,"is_registers_abi":true,"start_addr":139872935333888,"total_cpus":4}}
{"level":"error","ts":1666009433.315479,"caller":"instrumentors/runner.go:71","msg":"error while loading instrumentors, cleaning up","name":"google.golang.org/grpc","error":"field UprobeClientConnInvoke: program uprobe_ClientConn_Invoke: load program: invalid argument: ; int uprobe_ClientConn_Invoke(struct pt_regs *ctx) {\n0: (bf) r6 = r1\n1: (b7) r7 = 0\n; struct grpc_request_t grpcReq = {};\n2: (7b) *(u64 *)(r10 -16) = r7\nlast_idx 2 first_idx 0\nregs=80 stack=0 before 1: (b7) r7 = 0\n3: (7b) *(u64 *)(r10 -24) = r7\n4: (7b) *(u64 *)(r10 -32) = r7\n5: (7b) *(u64 *)(r10 -40) = r7\n6: (7b) *(u64 *)(r10 -48) = r7\n7: (7b) *(u64 *)(r10 -56) = r7\n8: (7b) *(u64 *)(r10 -64) = r7\n9: (7b) *(u64 *)(r10 -72) = r7\n10: (7b) *(u64 *)(r10 -80) = r7\n11: (7b) *(u64 *)(r10 -88) = r7\n12: (7b) *(u64 *)(r10 -96) = r7\n13: (7b) *(u64 *)(r10 -104) = r7\n14: (7b) *(u64 *)(r10 -112) = r7\n15: (7b) *(u64 *)(r10 -120) = r7\n16: (7b) *(u64 *)(r10 -128) = r7\n17: (7b) *(u64 *)(r10 -136) = r7\n18: (7b) *(u64 *)(r10 -144) = r7\n19: (7b) *(u64 *)(r10 -152) = r7\n20: (7b) *(u64 *)(r10 -160) = r7\n21: (7b) *(u64 *)(r10 -168) = r7\n; grpcReq.start_time = bpf_ktime_get_boot_ns();\n22: (85) call unknown#125\ninvalid func unknown#125\nprocessed 23 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0","stacktrace":"github.com/keyval-dev/opentelemetry-go-instrumentation/pkg/instrumentors.(*instrumentorsManager).load\n\t/app/pkg/instrumentors/runner.go:71\ngithub.com/keyval-dev/opentelemetry-go-instrumentation/pkg/instrumentors.(*instrumentorsManager).Run\n\t/app/pkg/instrumentors/runner.go:19\nmain.main\n\t/app/cli/main.go:71\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
{"level":"info","ts":1666009433.3155577,"caller":"grpc/probe.go:213","msg":"closing gRPC instrumentor"}
{"level":"info","ts":1666009433.3155837,"caller":"server/probe.go:179","msg":"closing net/http instrumentor"}
{"level":"error","ts":1666009433.315593,"caller":"cli/main.go:73","msg":"error while running instrumentors","error":"field UprobeClientConnInvoke: program uprobe_ClientConn_Invoke: load program: invalid argument: ; int uprobe_ClientConn_Invoke(struct pt_regs *ctx) {\n0: (bf) r6 = r1\n1: (b7) r7 = 0\n; struct grpc_request_t grpcReq = {};\n2: (7b) *(u64 *)(r10 -16) = r7\nlast_idx 2 first_idx 0\nregs=80 stack=0 before 1: (b7) r7 = 0\n3: (7b) *(u64 *)(r10 -24) = r7\n4: (7b) *(u64 *)(r10 -32) = r7\n5: (7b) *(u64 *)(r10 -40) = r7\n6: (7b) *(u64 *)(r10 -48) = r7\n7: (7b) *(u64 *)(r10 -56) = r7\n8: (7b) *(u64 *)(r10 -64) = r7\n9: (7b) *(u64 *)(r10 -72) = r7\n10: (7b) *(u64 *)(r10 -80) = r7\n11: (7b) *(u64 *)(r10 -88) = r7\n12: (7b) *(u64 *)(r10 -96) = r7\n13: (7b) *(u64 *)(r10 -104) = r7\n14: (7b) *(u64 *)(r10 -112) = r7\n15: (7b) *(u64 *)(r10 -120) = r7\n16: (7b) *(u64 *)(r10 -128) = r7\n17: (7b) *(u64 *)(r10 -136) = r7\n18: (7b) *(u64 *)(r10 -144) = r7\n19: (7b) *(u64 *)(r10 -152) = r7\n20: (7b) *(u64 *)(r10 -160) = r7\n21: (7b) *(u64 *)(r10 -168) = r7\n; grpcReq.start_time = bpf_ktime_get_boot_ns();\n22: (85) call unknown#125\ninvalid func unknown#125\nprocessed 23 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0","stacktrace":"main.main\n\t/app/cli/main.go:73\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}

Steps to Reproduce the Problem

I've followed the tutorial and tried to add instrumentation to an existing deployment in k8s.
Here's a part of deployment.yaml, it is basically a copy of diff between emojivoto and emojivoto-instrumented

...
    spec:
      shareProcessNamespace: true
      initContainers:
        - name: copy-launcher
          image: keyval/launcher:v0.1
          command:
            - cp
            - -a
            - /kv-launcher/.
            - /odigos-launcher/
          volumeMounts:
            - name: launcherdir
              mountPath: /odigos-launcher
      volumes:
        - name: launcherdir
          emptyDir: { }
        - name: kernel-debug
          hostPath:
            path: /sys/kernel/debug
      containers:
        - name: ...
          command:
            - /odigos-launcher/launch
            - /usr/local/bin/centrifugo
            - --health
            - --prometheus
          ...
        - name: instrumentation
          image: keyval/otel-go-agent:v0.6.0
          env:
            - name: OTEL_TARGET_EXE
              value: /usr/local/bin/centrifugo
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "otel-collector:4317"
            - name: OTEL_SERVICE_NAME
              value: "test"
          securityContext:
            runAsUser: 0
            capabilities:
              add:
                - SYS_PTRACE
            privileged: true
          volumeMounts:
            - mountPath: /sys/kernel/debug
              name: kernel-debug     

Additional Info

• Kubernetes version:

  Output of kubectl version:

  Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506",   GitTreeState:"clean", BuildDate:"2022-10-14T02:36:39Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
  Kustomize Version: v4.5.7
  Server Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.10-eks-15b7512", GitCommit:"cd6399691d9b1fed9ec20c9c5e82f5993c3f42cb", GitTreeState:"clean", BuildDate:"2022-08-31T19:17:01Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
  WARNING: version difference between client (1.25) and server (1.23) exceeds the supported minor version skew of +/-1
  

• operation-system/kernel version:
  Assuming we're talking about container OS:
  Output of awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release:
  Output of uname -r:

  Alpine Linux v3.13
  5.4.209-116.367.amzn2.x86_64
  

could not detect go version

Expected Behavior

A stable connection to the collector.

Actual Behavior

Program terminates because the go version is not identified.

Steps to Reproduce the Problem

1. podman run --network=host --rm -it jaegertracing/opentelemetry-all-in-one:latest
2. make build && OTEL_SERVICE_NAME=test OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317 OTEL_TARGET_EXE=${PWD}/unstripped ./kv-go-instrumentation
3. ./unstripped (gorilla mux http server)

Additional Info

• operation-system/kernel version:

  Output of awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release:
  Output of uname -r:

  "Fedora Linux 35 (Workstation Edition)"
  5.17.4-200.fc35.x86_64
  

• binary strings strings unstripped | grep "Go buildinf" -A 1:

Go buildinf:
go1.18

• go version:

go version go1.18 linux/amd64

• log:

{"level":"info","ts":1651701304.5921707,"caller":"cli/main.go:22","msg":"starting Go OpenTelemetry Agent ..."}
{"level":"info","ts":1651701304.5922222,"caller":"opentelemetry/controller.go:97","msg":"Establishing connection to OpenTelemetry collector ..."}
{"level":"info","ts":1651701306.6052349,"caller":"process/discover.go:37","msg":"process not found yet, trying again soon","exe_path":"/home/frzifus/git/go/observability/opentelemetry-go-instrumentation/unstripped"}
{"level":"info","ts":1651701308.5969474,"caller":"process/discover.go:37","msg":"process not found yet, trying again soon","exe_path":"/home/frzifus/git/go/observability/opentelemetry-go-instrumentation/unstripped"}
{"level":"info","ts":1651701310.5970006,"caller":"process/discover.go:42","msg":"found process","pid":1131585}
{"level":"error","ts":1651701310.5971448,"caller":"cli/main.go:61","msg":"error while analyzing target process","error":"could not detect go version","stacktrace":"main.main\n\t/home/frzifus/git/go/observability/opentelemetry-go-instrumentation/cli/main.go:61\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}

Generate ID for x-Ray

Feature request

As of now, go-agent is generating id in format which don't conforms with AWS X-Ray

I looked at code and didn't understand how to replace &eBPFSourceIDGenerator with xray.NewIDGenerator() without crushing the tracing.

Use case

In order to send traces to X-ray, go-agent needs generate ID in proper format.

Sounds like a really interesting feature @withlin

Sounds like a really interesting feature @withlin
Regarding implementation, I think we should consider changing the current logic of specifying the target binary via OTEL_TARGET_EXE to logic like:

1. Scan all processes in /proc
2. Detect all processes that are written in Go. we already have this code at https://github.com/keyval-dev/opentelemetry-go-instrumentation/blob/af32862e111b42be049956705d0dd27e9639fc15/pkg/process/module.go#L40
3. Apply instrumentation for every found Go process.

I think changing the behavior to this will allow us to both remove the OTEL_TARGET_EXE config and support daemon set deployment at the same time. Basically, when deployed as a daemon set with hostPID: true /proc will simply contain more processes and the instrumentation agent does not even aware of the fact that it is currently deployed as a daemon set.
What do you think?

Originally posted by @edeNFed in #9 (comment)

how to read request header map ?

Feature request

Use case

[Question] How to configure emojivoto to propagate context?

Hi, thanks for such a nice project!

I tried using this with emojivoto, but I'm having trouble with context propagation.
Tracing a single service works fine but the traffic across some services is traced as different trace id.

Expected Behavior

Trace ID is propagated across multiple services

Actual Behavior

A series of events across multiple services are traced as different traces

Steps to Reproduce the Problem

1. checkout https://github.com/ymtdzzz/emojivoto-opentelemetry-sample/tree/feature/ebpf
2. start k8s cluster (such as minikube)
3. deploy as described in the README
4. curl "http://your-node-ip:30084/api/list"
5. access to http://your-node-ip:30081 and check the traced data

Additional Info

• Kubernetes version:

  Output of kubectl version:

clientVersion:
  buildDate: "2022-07-13T14:21:56Z"
  compiler: gc
  gitCommit: aef86a93758dc3cb2c658dd9657ab4ad4afc21cb
  gitTreeState: clean
  gitVersion: v1.24.3
  goVersion: go1.18.4
  major: "1"
  minor: "24"
  platform: linux/amd64
kustomizeVersion: v4.5.4
serverVersion:
  buildDate: "2022-05-24T12:18:48Z"
  compiler: gc
  gitCommit: 3ddd0f45aa91e2f30c70734b175631bec5b5825a
  gitTreeState: clean
  gitVersion: v1.24.1
  goVersion: go1.18.2
  major: "1"
  minor: "24"
  platform: linux/amd64

• operation-system/kernel version:

  Output of awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release:
  Output of uname -r:

$ awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release
"Ubuntu 22.04 LTS"
$ uname -r
5.15.0-43-generic

Concatenation of trace context for asynchronous requests in different goroutine

Feature request

This repo is the best idea I've seen recently !!!

Will consider supporting the scenario of starting a new goroutine for asynchronous client requests ?
In this scenario, is it impossible to connect the trace context through the goroutine context map ?

Can't find find keyval map in analyze findKeyvalMmap

Expected Behavior

Read memory maps.

Actual Behavior

Can't find memory mapping as as all mapping has been filtered out.

Steps to Reproduce the Problem

The maps of the go lang program as below

00400000-00623000 r-xp 00000000 fd:00 1048990                            /home/vagrant/go-http-demo/main
00623000-00822000 r--p 00223000 fd:00 1048990                            /home/vagrant/go-http-demo/main
00822000-00860000 rw-p 00422000 fd:00 1048990                            /home/vagrant/go-http-demo/main
00860000-00899000 rw-p 00000000 00:00 0
02061000-02082000 rw-p 00000000 00:00 0                                  [heap]
c000000000-c000400000 rw-p 00000000 00:00 0
...
7f32e948f000-7f32e94b7000 r--p 00000000 fd:00 525141                     /usr/lib/x86_64-linux-gnu/libc.so.6
...
7f32e96aa000-7f32e96b7000 rw-p 00000000 00:00 0
7f32e96be000-7f32e96c0000 rw-p 00000000 00:00 0
7f32e96c0000-7f32e96c2000 r--p 00000000 fd:00 524900                     /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
...
7ffe501e6000-7ffe50207000 rw-p 00000000 00:00 0                          [stack]
7ffe50311000-7ffe50315000 r--p 00000000 00:00 0                          [vvar]
7ffe50315000-7ffe50317000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

so no map will be fond as m.Perms != nil && m.Perms.Read && m.Perms.Write && m.Perms.Execute required in findKeyvalMmap.

Additional Info

• vagrant box:

https://app.vagrantup.com/bento/boxes/ubuntu-22.04

• operation-system/kernel version:

  Output of awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release:
  Output of uname -r:

  Ubuntu 22.04.1 LTS
  5.15.0-30-generic
  

Support for aarch64

I am trying to run on Mac M1 processor and it doesn't seem to work.

All functions instrumentation

Feature request

Does this package add instrumentation for all functions or only net/http and grpc?

eBPF allows to add instrumentation to every function call, would be good to have it here.

Use case

Automatically add instrumentation to all functions in Go pkg.

This is primary why I found this project, but surprised it is not supported.

Compatibility with lower version kernels.

Feature request

I guess that "requires >= v5.2" may prompt me with the kernel version, and after upgrading the kernel to >=5.2, it was successfully confirmed to be true.
Many instances of golang processes are still running in lower version kernels.So do we have any idea or plan to make it compatible with lower version kernels?

[root@centos7test ~]# OTEL_TARGET_EXE=/root/http ./kv-go-instrumentation -stdout
{"level":"info","ts":1675775385.3834639,"caller":"cli/main.go:27","msg":"starting Go OpenTelemetry Agent ..."}
{"level":"info","ts":1675775387.3908978,"caller":"process/discover.go:46","msg":"found process","pid":4644}
{"level":"info","ts":1675775387.3916051,"caller":"process/analyze.go:75","msg":"found addr of keyval map","addr":139854386225152}
{"level":"info","ts":1675775387.398304,"caller":"process/analyze.go:138","msg":"found relevant function for instrumentation","function":"net/http.(*ServeMux).ServeHTTP","returns":2}
{"level":"info","ts":1675775387.3984025,"caller":"cli/main.go:70","msg":"target process analysis completed","pid":4644,"go_version":"1.13.10","dependencies":{},"total_functions_found":1}
{"level":"info","ts":1675775387.3984728,"caller":"cli/main.go:76","msg":"invoking instrumentors"}
{"level":"info","ts":1675775387.4092982,"logger":"allocator","caller":"allocator/allocator_linux.go:19","msg":"Loading allocator","start_addr":139854386225152,"end_addr":139854398808064}
{"level":"info","ts":1675775387.4094503,"caller":"instrumentors/runner.go:68","msg":"loading instrumentor","name":"net/http"}
{"level":"info","ts":1675775387.414667,"caller":"inject/injector.go:67","msg":"Injecting variables","vars":{"ctx_ptr_pos":232,"is_registers_abi":false,"method_ptr_pos":0,"path_ptr_pos":56,"url_ptr_pos":16}}
{"level":"error","ts":1675775387.4149885,"caller":"instrumentors/runner.go:71","msg":"error while loading instrumentors, cleaning up","name":"net/http","error":"field UprobeServerMuxServeHTTP: program uprobe_ServerMux_ServeHTTP: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)","stacktrace":"github.com/keyval-dev/opentelemetry-go-instrumentation/pkg/instrumentors.(*instrumentorsManager).load\n\t/opt/github/opentelemetry-go-instrumentation/pkg/instrumentors/runner.go:71\ngithub.com/keyval-dev/opentelemetry-go-instrumentation/pkg/instrumentors.(*instrumentorsManager).Run\n\t/opt/github/opentelemetry-go-instrumentation/pkg/instrumentors/runner.go:19\nmain.main\n\t/opt/github/opentelemetry-go-instrumentation/cli/main.go:77\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
{"level":"info","ts":1675775387.4150653,"caller":"server/probe.go:179","msg":"closing net/http instrumentor"}
{"level":"error","ts":1675775387.4150894,"caller":"cli/main.go:79","msg":"error while running instrumentors","error":"field UprobeServerMuxServeHTTP: program uprobe_ServerMux_ServeHTTP: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)","stacktrace":"main.main\n\t/opt/github/opentelemetry-go-instrumentation/cli/main.go:79\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}

Use OTEL SDK to parse exporter endpoint

Feature request

The exporter endpoint is not parsed directly in the code https://github.com/keyval-dev/opentelemetry-go-instrumentation/blob/master/pkg/opentelemetry/controller.go#L76 from which grpc connection is created.

This is not ideal because it limits how the exporter is initialized and it breaks semantics of OTEL_EXPORTER_OTLP_ENDPOINT which should have form of http(s)://host:port. See https://github.com/open-telemetry/opentelemetry-go/tree/main/exporters/otlp/otlptrace and OTEL spec.

I would propose to change the code to consume only env vars in OTEL defined format and leave parsing of the endpoint to OTEL SDK/exporter implementation.

Unable to instrument go binary file which doesn't contain ".gosymtab" section

Expected Behavior

When a go binary file doesn't contain ".gosymtab" section, it should be able to instrument it.

Actual Behavior

Analyze process stops when it can't find ".gosymtab" section in target go binary file.

Steps to Reproduce the Problem

1. ./launch goapp (launch binary copied from image keyval/launcher:v0.1)
2. start instrumentation:

OTEL_SERVICE_NAME=goapp  OTEL_TARGET_EXE=/goapp OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317 /kv-go-instrumentation

3. kv-go-instrumentation logs error and exits, error message are as below:

{"level":"info","ts":1667322994.3850787,"caller":"cli/main.go:22","msg":"starting Go OpenTelemetry Agent ..."}
{"level":"info","ts":1667322994.385147,"caller":"opentelemetry/controller.go:92","msg":"Establishing connection to OpenTelemetry collector ..."}
{"level":"info","ts":1667322996.387433,"caller":"process/discover.go:42","msg":"found process","pid":19}
{"level":"info","ts":1667322996.3889744,"caller":"process/analyze.go:73","msg":"found addr of keyval map","addr":140331000791040}
{"level":"error","ts":1667322996.3974175,"caller":"cli/main.go:61","msg":"error while analyzing target process","error":".gosymtab section not found in target binary, make sure this is a Go application","stacktrace":"main.main\n\t/app/cli/main.go:61\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}

• operation-system/kernel version:
  Output of awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release:
  Output of uname -r:

  "CentOS Linux 7 (Core)"
  5.16.11-1.el7.elrepo.x86_64
  

The corresponding source code is from:
https://github.com/keyval-dev/opentelemetry-go-instrumentation/blob/master/pkg/process/analyze.go
line 117 to 120:

	sec := elfF.Section(".gosymtab")
	if sec == nil {
		return nil, fmt.Errorf("%s section not found in target binary, make sure this is a Go application", ".gosymtab")
	}

AMQP example

Feature request

The example shown in the project documentation focuses on gRPC, it would be awesome if the example could demonstrate integration with AMQP. We already instrument our code using

https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrumentation

but then have to manually instrument our code for AMQP

Use case

We already instrument our code using

https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrumentation

but then have to manually instrument our code for AMQP . I'd hope that the auto-instrumentation
offered by this project could remove the need for the manual instrumentation required by AMQP.

[suggestion] Better scope definition in the Readme

It's not very clear what kind of OTEL integration this approach provide. Does it produce metrics? Distributed traces? Logs?

Got panic "cant find keyval map"

Expected Behavior

Working go-lang auto instrumentation

Actual Behavior

In sidecar I got this:

{"level":"info","ts":1666358871.3476882,"caller":"cli/main.go:22","msg":"starting Go OpenTelemetry Agent ..."}                                                                                                                                  
{"level":"info","ts":1666358871.3477695,"caller":"opentelemetry/controller.go:92","msg":"Establishing connection to OpenTelemetry collector ..."}                                                                                               │
{"level":"info","ts":1666358873.3498256,"caller":"process/discover.go:42","msg":"found process","pid":7}                                                                                                                                  
 **panic: cant find keyval map**                                                                                                                                                                                                                                                                                                                                                                                                                               
 goroutine 1 [running]:                                                                                                                                                                                                                          
 github.com/keyval-dev/opentelemetry-go-instrumentation/pkg/process.(*processAnalyzer).findKeyvalMmap(0xb51e40?, 0xc0000c0058?)                                                                                                                  │
     /app/pkg/process/analyze.go:77 +0x14f                                                                                                                                                                                                       
 github.com/keyval-dev/opentelemetry-go-instrumentation/pkg/process.(*processAnalyzer).Analyze(0xc00009e2a0?, 0x7, 0x2?)           /app/pkg/process/analyze.go:103 +0x1a5                                                                                                                                                                                   main.main()                                                                                                                                                                                                                                     /app/cli/main.go:59 +0x37f

UPDATE:
Fixed panic by adding init container with odigos launcher - I don't understand why do I need add this launcher.
After that error I have another error:

{"level":"info","ts":1666363476.3613198,"caller":"process/analyze.go:73","msg":"found addr of keyval map","addr":140516020199424}                                                                                                               │
{"level":"info","ts":1666363476.3766043,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"net/http.(*ServeMux).ServeHTTP","returns":2}                                                          │
{"level":"info","ts":1666363476.378454,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc/internal/transport.(*http2Client).createHeaderFields","returns":3}              
{"level":"info","ts":1666363476.3786323,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc/internal/transport.(*decodeState).decodeHeader","returns":6}                   
{"level":"info","ts":1666363476.3787887,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc.(*ClientConn).Invoke","returns":2}                                             │
{"level":"info","ts":1666363476.3791306,"caller":"process/analyze.go:136","msg":"found relevant function for instrumentation","function":"google.golang.org/grpc.(*Server).handleStream","returns":5}                                           │
{"level":"info","ts":1666363476.379253,"caller":"cli/main.go:64","msg":"target process analysis completed","pid":13,"go_version":"1.15.0","dependencies":{"contrib.go.opencensus.io/exporter/ocagent":"v0.6.0","github.com/beorn7/perks":"v1.0. │
1","github.com/census-instrumentation/opencensus-proto":"v0.2.1","github.com/cespare/xxhash/v2":"v2.1.1","github.com/golang/groupcache":"v0.0.0-20200121045136-8c9f03a8e57e","github.com/golang/protobuf":"v1.4.0","github.com/grpc-ecosystem/g │
o-grpc-prometheus":"v1.2.0","github.com/grpc-ecosystem/grpc-gateway":"v1.14.4","github.com/matttproud/golang_protobuf_extensions":"v1.0.1","github.com/prometheus/client_golang":"v1.6.0","github.com/prometheus/client_model":"v0.2.0","github │
.com/prometheus/common":"v0.9.1","github.com/prometheus/procfs":"v0.0.11","go.opencensus.io":"v0.22.3","golang.org/x/net":"v0.0.0-20200425230154-ff2c4b7c35a0","golang.org/x/sync":"v0.0.0-20200317015054-43a5402ce75a","golang.org/x/sys":"v0. 0.0202004300824071f5687305801","golang.org/x/text":"v0.3.2","google.golang.org/api":"v0.22.0","google.golang.org/genproto":"v0.0.020200430143042b979b6f78d84","google.golang.org/grpc":"v1.29.1","google.golang.org/protobuf":"v1.21.0"},"total_functions_found":5}                                                                                                                                                                                                                        {"level":"info","ts":1666363476.3793523,"caller":"cli/main.go:70","msg":"invoking instrumentors"}                                                                                                                                               
{"level":"info","ts":1666363476.3992898,"logger":"allocator","caller":"allocator/allocator_linux.go:19","msg":"Loading allocator","start_addr":140516020199424,"end_addr":140516032782336}                                                      │
{"level":"info","ts":1666363476.4011118,"caller":"instrumentors/runner.go:68","msg":"loading instrumentor","name":"google.golang.org/grpc"}                                                                                                     │
{"level":"info","ts":1666363476.4045897,"caller":"inject/injector.go:67","msg":"Injecting variables","vars":{"clientconn_target_ptr_pos":24,"end_addr":140516032782336,"is_registers_abi":false,"start_addr":140516020199424,"total_cpus":4}}   │
{"level":"error","ts":1666363476.4090192,"caller":"instrumentors/runner.go:71","msg":"error while loading instrumentors, cleaning up","name":"google.golang.org/grpc","error":"field UprobeHttp2ClientCreateHeaderFields: program uprobe_Http2C │
│ lient_CreateHeaderFields: load program: invalid argument: Unrecognized arg#0 type PTR\n; int uprobe_Http2Client_CreateHeaderFields(struct pt_regs *ctx)

Steps to Reproduce the Problem

1. Using otel operator and otel collector v0.55.0 and emojivoto app for testing
2. Add annotations
   instrumentation.opentelemetry.io/golang-target-exec: /usr/local/bin/emojivoto-emoji-svc
   instrumentation.opentelemetry.io/inject-golang: "true"

Additional Info

• go-agent otel version 0.6.0

• Kubernetes version:

  Output of kubectl version: Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:25:45Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"darwin/arm64"}
  Kustomize Version: v4.5.7
  Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.14-eks-6d3986b", GitCommit:"8877a3e28d597e1184c15e4b5d543d5dc36b083b", GitTreeState:"clean", BuildDate:"2022-07-20T22:05:32Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"} .......

Include bpf includes within the project

Feature request

Maybe the project can come with its own include directory containing all standard bpf header files required to build the project, which would make it much easier for everybody to just build the binary locally without bothering about bpf dependencies.

Use case

After cloning the project, the make build command failed because I did not have my bpf include files in /usr/include/bpf.

Luckily, I had a personal project on my machine using libbpf, and so I just copied some of my include files to /usr/include/bpf and it worked. But for some other people less familiar with bpf who just wants to try this amazing tool (still in early stage, but looks quite promising) it might be a blocker.

Is this feature request gets accepted, I can submit a PR.