16bitt/kubemem

A small binary to probe memory usage in a kubernetes pod

kubemem's Issues

Document how to monitor warning logs

Hi there 👋
Nice tool you have here!

I found this feature interesting:

a warning percentage: When your RAM usage hits this threshold, kubemem will log the warning

I thought the program might be doing this by creating an Event in the k8s API.
However, I found it logs the warning and returns 0. This is quick and simple:
https://github.com/16Bitt/kubemem/blob/14a1c13/main.c

LivenessProbe logs get recorded by the Kubelet in an Event.
however, this is only on failure, not success.

There aren't many useful places probe logs actually end up.
They don't show up in container logs.
This stackoverflow response is still effectively correct today:
https://stackoverflow.com/a/34599554

Modern k8s now supports creating a Warning Event for probes:
https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/kubelet/prober/prober.go#L123-L130
This wasn't true in 2016 when that SO answer was written.

However, the Exec Prober doesn't support returning probe.Warning, so a Warning Event can't be created:
https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/probe/exec/exec.go#L41-L55

The kubelet will start logging all Exec Probe command output at '-v=4` and that's the only way you could monitor those messages.

It might be worth documenting this?
Perhaps there is another logging/event mechanism I missed.

How are you using these warning messages at $work?
Are you collecting your kubelet logs in something like ElasticSearch/Datadog/Loki and then monitoring for them?

Cheers :)

---

Unrelated:
Starting in v1.16.1, probe output is limited to 10kb:
kubernetes/kubernetes#82514
https://github.com/kubernetes/kubernetes/blob/v1.16.1/pkg/probe/exec/exec.go#L48-L72
(just a neat thing I learned)

License?

In which license this project is shared right now?
Could you add the proper entry for a license? Perhaps MIT or Apache?

Track RSS memory instead of total

Currently the total used memory is pulling from /sys/fs/cgroup/memory/memory.usage_in_bytes which provides the total used memory, including reclaimable caches. K8s uses the RSS memory (or something approximate to it) when calculating memory use for evictions. This is throwing the % calculation out by a significant factor, causing pods to be flagged as failed before k8s would consider them to be in any OOMKiller danger.

The RSS for the cgroup can be pulled from /sys/fs/cgroup/memory/memory.stat but the parsing is a bit more challenging. The last time I wrote any C was about 15 years ago but I managed to cobble something together which uses a regex to match the appropriate line and adjust the calculation accordingly.

Fancy a PR?