gardener-attic/connectivity-exporter

A tool based on eBPF, prometheus and grafana to monitor network connectivity.

connectivity-exporter's Issues

False positive failed seconds

What happened:

When testing in a dev gardener seed cluster, for some shoots, the connectivity exporter reports “active_failed” seconds. We checked the network traffic with tcp dump on the same node, and couldn't find aborted TCP connections (i.e. couldn't find RST packets sent by envoy to the client). So it seems to be a "false" signal.

Query:

sum(increase(connectivity_exporter_seconds_total{kind="active_failed", sni="..."}[2m])) by (sni, pod, kind)

We couldn't reproduce it locally.

It seems to happen for all the shoot clusters, on all the nodes.

max(rate(connectivity_exporter_seconds_total{kind="active"}[100m])) by (sni, pod)
unless
max(rate(connectivity_exporter_seconds_total{kind="active_failed"}[100m])) by (sni, pod)

It could be related to this line

diff --git a/connectivity-exporter/packet/c/cap.c b/connectivity-exporter/packet/c/cap.c
index dd49eb1..807d511 100644
--- a/connectivity-exporter/packet/c/cap.c
+++ b/connectivity-exporter/packet/c/cap.c
@@ -399 +399 @@ int capture_packets_internal(struct __sk_buff *skb)
-      conn->state = RST_RECEIVED;
+      // conn->state = RST_RECEIVED;

With that line commented out, we don't see that background noise any more:

count(max(rate(connectivity_exporter_connections_total{kind="rejected"}[5m])) by (sni) > 0)
max(connectivity_exporter_seconds_total{kind="active_failed"}) by(sni) > 0

What you expected to happen:

No false positive failed seconds.

with @wyb1

Test connectivity-exporter on AKS with Azure CNI & Kubenet

I'd like to test connectivity-exporter on AKS with different network configurations:

• Azure CNI
• Kubenet

I will report here my findings.