Coordinator node regularly restarts in 3 node cluster

[apache]

Coordinator node regularly restarts in 3 node cluster

[colearendt]

FWIW liveness probes can be tricky and have much discussion in the community (i.e. here and here), but the one in this chart is on by default.

You can disable it by setting livenessProbe.enabled = false:

couchdb-helm/couchdb/values.yaml

Lines 191 to 197 in 78eff8c

	livenessProbe:
	enabled: true
	failureThreshold: 3
	initialDelaySeconds: 0
	periodSeconds: 10
	successThreshold: 1
	timeoutSeconds: 1

I know little about the underlying implementation here, but it's possible that the default should be changed and livenessProbe should be turned off 🤷 Or its configuration should be improved so flapping like this is less likely. At a minimum, failureThreshold should probably be increased to 10 or so (as the latter article recommends).

[cg1972]

From the logs it looks like the readiness probe is failing due to the increased time it is taking to make the check. There are times in there of 29613 and 14872 where are normal response would be < 10 ms. There appears to be something happening in couchdb that is causing the responses to slow down. When this is occurring there are no obvious cpu spikes or memory usage either.

Is there any additional debugging that can be enabled to try and capture what couchdb might be doing when it crashes? Given that this deployed service is hardly used at the moment it is concerning that this type of thing would occur when it is under no load.

[colearendt]

Woops - apologies for never responding here. I must admit that I am not super familiar with the internals of couchdb - I would definitely recommend turning off livenessProbe though. If the readinessProbe fails, that is fine - traffic may not get routed to the pod briefly, but the pod stays alive. Moreover, you can increase failureThreshold or increase periodSeconds or increase timeoutSeconds to make it check more often / allow for more variance.

livenessProbe should only be used when "the best thing to do for the service is to kill it." I highly doubt that that is the case if it responds a smidge slowly on occasion (during a sync or some such).