Investigate on error with node-directory-size

[giantswarm]

Investigate on error with node-directory-size

[MarcelMue]

I think I fixed this in 633bca6 by simply touching the file before the while loop. It appears to be working in my minikube and scraping succeeds.

Another Issue is that if there is no folder >100mb then the file is just empty and not served. Prometheus gets a 404and believes the instance is down.

[webwurst]

Cool, thank you! This node-directory-size tool is more or less a quick hack to see if there are some surprisingly large directories around. And on the other side maybe a nice showcase how you can provide metrics for Prometheus with just some lines of scripting. In case there is a more clever tool around we might swap this out. But as long as it works "good enough" we should maintain and improve on this. 👍

[MarcelMue]

Another small Issue I had was that prometheus tried to scrape port 80 in addition to port 9102. Maybe port 80 is opened by the caddy container?

I resolved this by adding

- source_labels: [__meta_kubernetes_pod_container_port_number]
  action: keep
  regex: 9\d{3}

to the kubernetes-pods job. This allows prometheus to only parse ports in the 9000 range, but I am unsure if that causes trouble somewhere down the line.

[webwurst]

Hm, I also saw that for some time, but it not anymore in later installations. Maybe I updated the version of Prometheus.. I did updates of all the components and started testing locally today. No problems so far.

[pipo02mix]

I think it is solved, at least I can not reproduce feel free to reopen