traefik_open_connections metric drifts down until negative

[navaati]

traefik_open_connections metric drifts down until negative

[navaati]

I just realized there was a major Træfik version release during my development cycle ! That’ll teach me not pinning containers…

Anyway, just for the record I tested with container tag v2.11.3 and the result is much more stable (although I understand there were other problems with the former traefik_entrypoint_open_connections metric versus the new traefik_open_connections one):

Now I’ll test a modified v3.0 with the +1/-1 in the mutex critical section.

[navaati]

Digging a bit more… I don’t really speak Go but I’ll try… Clearly this function RemoveConnection is being called too much, or AddConnection not enough.

[navaati]

Errr, so something looks off to me, but probably is not. Reproducing the code of RemoveConnection here:

func (c *connectionTracker) RemoveConnection(conn net.Conn) {
	c.connsMu.Lock()
	delete(c.conns, conn)
	c.connsMu.Unlock()

	if c.openConnectionsGauge != nil {
		c.openConnectionsGauge.Add(-1)
	}
}

Is it normal that the Gauge decrementation is outside of the mutex section ? I guess the instrumentation library is threadsafe already, is it ?

[navaati]

@rtribotte I see that you authored this highly relevant commit 7c2af10 from PR #9656, might you be able to chip in ?

[thanhphuchuynh]

I have an issue too

my metrics:
HELP traefik_open_connections How many open connections exist, by entryPoint and protocol
TYPE traefik_open_connections gauge
traefik_open_connections{entrypoint="metrics",protocol="TCP"} 2
traefik_open_connections{entrypoint="traefik",protocol="TCP"} -26
traefik_open_connections{entrypoint="web",protocol="TCP"} -26
traefik_open_connections{entrypoint="websecure",protocol="TCP"} 8

[rtribotte]

Hello @navaati, thanks for opening this!

Is it normal that the Gauge decrementation is outside of the mutex section ? I guess the instrumentation library is threadsafe already, is it ?

It should be threadsafe indeed (https://github.com/prometheus/client_golang/blob/release-1.17/prometheus/gauge.go#L122C1-L130C2), Have you had a chance to try the alternative (protect with the mutex)? Did it fix the bug?

[navaati]

Hi.

No, I haven’t tested anything yet: I first need to figure out how to build the binary then the container, for which I haven’t found instructions yet (at least nothing in the contributing documentation). EDIT: found the doc https://doc.traefik.io/traefik/contributing/building-testing/ !

I’ll find and try to test, although with that link you dug I don’t have much hope. Is it actually the official prom client which is being used here though ? I see the type of the gauge is gokitmetrics.Gauge.

[navaati]

Yeah no, it’s just an abstraction layer over the official prom client: https://github.com/go-kit/kit/blob/dfe43fa6a8d72c23e2205d0b80e762346e203f78/metrics/prometheus/prometheus.go#L84.

[dennorske]

I don't have much useful information other than to say that I am running 3.0.2 and I have had lots of negative indicated open connections. My prometheus data confirms that this has not been an issue before.

I have added two extra queries to demonstrate:

You can see min-2.0 never goes below 0, while min-3.0 maxed out at almost 500.

The two fields used are:
traefik_open_connections (3.0 >)
traefik_entrypoint_open_connections (< 2.x)