Document how to monitor warning logs

[stealthybox]

Document how to monitor warning logs

[16Bitt]

Great suggestions! Thank you so much for your input!

Here's a PR that allows setting the logfile along with updated documentation: #2

It also fixes a really dumb bug on my part, as I was assuming that sysinfo accounted for cgroups

[16Bitt]

Hi there! Thanks for pointing this out! I’ve primarily been using the logs for debugging the tool, but I had assumed the probe logs would show within the pod logs.

I’ll put together an option to manually create the event through the REST API, which may add a bit more complexity but would certain be worth the investment.

As for the logging infrastructure we us at $work, we use Sumologic-fluentd to aggregate pod and API server logs. A significant drawback to this is bumping up verbosity even one level in the cluster can rack up your bills super quickly, so I’d rather not burden users with a verbosity increase. Adding audit logging to our clusters added 2Gi a day in sumo logic.

I haven’t rolled this out in prod yet (I wrote this on my vacation) but I’m very certain a lot of changes will happen as I start to use this in production worker pods. Unfortunately my company doesn’t allow open source contributions, so this will have to be done after hours once I’m back in the office.

[stealthybox]

Appending to PID 1 stdout might work:

echo warning >> /proc/1/fd/1

It's worth testing in a Pod to see.

That might be a sensible default behavior in a Pod, or it could be opt-in and showed in the example.
It should be possible to disable as it relies on the probe having write access.

A more generic flag might work too:

--log-file=/proc/1/fd/1  # append to PID 1 stdout

[stealthybox]

I tested the idea with a basic shell script and an exec, and it seems to work:

terminal 1

# Start a pod that prints some numbers
kubectl run testpid1 --image busybox -- sh -c 'for i in $(seq 1 3600); do sleep 1; echo $i; done'
kubectl logs -f deploy/testpid1

terminal 2

# Run a separate process in that pod that appends to the pod log once
k exec -it deploy/testpid1 -- sh -c 'echo helloworld >> /proc/1/fd/1'

This seems simpler than creating Events.
The program can keep logging to both places so that the Failure Events still have logs.

[16Bitt]

Forgot to close this!

[darioleanbit]

Hi @16Bitt , if I try to do

# echo warning >> /proc/1/fd/1
bash: /proc/1/fd/1: Permission denied

I get this error, is there any workaround? Thanks!