Describe the bug
The k8s-views-nodes.json
dashboard will have many broken panels in specific Kubernetes setups.
This is currently the case on OKE.
Apparently, this happens when the node
label from kube_node_info
doesn't match the nodename
label from node_uname_info
.
Here's some extracted metrics from a broken setup where the labels differ.
TL;DR: node="k8s-wrk-002"
and nodename="kind-kube-prometheus-stack-worker2"
.
kube_node_info:
{
__name__="kube_node_info",
container="kube-state-metrics",
container_runtime_version="containerd://1.6.19-46-g941215f49",
endpoint="http",
instance="10.27.3.148:8080",
internal_ip="172.18.0.2",
job="kube-state-metrics",
kernel_version="6.2.12-arch1-1",
kubelet_version="v1.26.3",
kubeproxy_version="v1.26.3",
namespace="monitoring",
node="k8s-wrk-002",
os_image="Ubuntu 22.04.2 LTS",
pod="kube-prometheus-stack-kube-state-metrics-6df68756d8-zvd58",
pod_cidr="10.27.1.0/24",
provider_id="kind://docker/kind-kube-prometheus-stack/kind-kube-prometheus-stack-worker2",
service="kube-prometheus-stack-kube-state-metrics",
system_uuid="8422f117-6154-45bd-97c0-e3dec80a3f60"
}
node_uname_info:
{
__name__="node_uname_info",
container="node-exporter",
domainname="(none)",
endpoint="http-metrics",
instance="172.18.0.2:9100",
job="node-exporter",
machine="x86_64",
namespace="monitoring",
nodename="kind-kube-prometheus-stack-worker2",
pod="kube-prometheus-stack-prometheus-node-exporter-qvn22",
release="6.2.12-arch1-1",
service="kube-prometheus-stack-prometheus-node-exporter",
sysname="Linux",
version="#1 SMP PREEMPT_DYNAMIC Thu, 20 Apr 2023 16:11:55 +0000"
}
This issue will continue the discussion started in #41
@fcecagno @Chewie
How to reproduce?
You can use https://github.com/dotdc/kind-lab, that will create a kind cluster with renamed nodes.
# Create the kind cluster
./start.sh
# Export configuration
export KUBECONFIG="$(pwd)/kind-kubeconfig.yml"
# Expose Grafana
kubectl port-forward svc/kube-prometheus-stack-grafana -n monitoring 3000:80
Open http://localhost:3000
login: admin
password: prom-operator
Open broken dashboard:
http://localhost:3000/d/k8s_views_nodes/kubernetes-views-nodes?orgId=1&refresh=30s
Expected behavior
Dashboard should work with a relabel_configs
like suggested @Chewie.
The solution should be described in https://github.com/dotdc/grafana-dashboards-kubernetes#known-issues
Additional context
No response