Significant Memory Increase with OpenTelemetry Leading to OoMKilled Issues on Kubernetes

[phthaocse]

Significant Memory Increase with OpenTelemetry Leading to OoMKilled Issues on Kubernetes

[dmathieu]

This ask is rather vague.
OpenTelemetry does not "manage memory" per-se. Go manages memory.

We do have benchmarks that track allocations though.
They run on new releases, and manually on an as-needed basis in PRs.

Investigating this would require looking into what exactly is using memory within your application.
That may be due to otel (like anything, it does have a memory and cpu footprint). It could also be that you were stretched too thin in term of resources.
Without more information, I'm afraid there isn't much more we can do here.

[pellared]

Could you please provide us with any documentation or knowledge regarding how OpenTelemetry manages memory?

I think it would be an overkill. You can always read the codebase.

After implementing OpenTelemetry to record metrics, we noticed a significant increase in memory usage in our pods deployed on Kubernetes, leading to OoMKilled issues.

We cannot do anything without repro steps or profiling data.

[yaniv-s]

There's definitely a problem with memory allocations/usage in 1.27
Since I upgraded from 1.24 to 1.27 my service uses more memory, this is from pprof, I hope it can help

[MrAlias]

Please provide the example code that you used to generate that graphic. I mean not aware of a function in this project called AddTagToContext. It looks like an inlining to grow is happening there. Understanding of that call sight is needed to begin addressing this.