[feature-request] Add metrics for Services

[kubernetes]

[feature-request] Add metrics for Services

[brancz]

So essentially a metric for each Endpoints object containing a count of all addresses of all EndpointSubsets. Probably split in ready and non-ready. I think that would be a great candidate for a metric.

[brancz]

Can you retrieve that information using the kubernetes API? kube-state-metrics is mainly about converting information that can be requested from the apiserver to the prometheus format. Could you point at where we can find that information in the apiserver?

[dkapanidis]

The use case I'm looking for is the following:

I want to make sure that all defined Services have at least one Endpoint that redirects to a running Pod (meaning that they are functional and not broken)

Being broken can be either by misconfiguration, or a failing Pod. The failing pod can be caught by monitoring the Pod, but the misconfiguration is harder to catch.

When profiling the system basically I do kubectl describe service and see the endpoints it redirects to. Though the information is disperse on Kubernetes API, the above command does GET request to /services and /endpoints to collect the necessary info.

You can see that by adding debugging flag (--v=8) on CLI

kubectl describe svc myservice --v=8

If the information was concentrated as in Deployment resources on status section it would be a lot easier to expose that (I'm not sure if there is any pending work on something like that though):

kubectl get deployment mydeployment -o yaml
...
status:
  observedGeneration: 4
  replicas: 1
  unavailableReplicas: 1
  updatedReplicas: 1

Though the information for that can be retrieved from /endpoints directly.

kubectl get endpoints -o yaml

There the name of the endpoint is unique and maps directly to the name of the service. A quick info to extract is the number of addresses attached on the endpoint (each address is one pod). If no addresses exist the service is out-of-order.

Though, I would call those metrics endpoints metrics for clarity.

[nleib]

+1
I think its important to know the number of endpoints, so we can alert in case of 0 endpoints on a service.
so metric of number of endpoints on a service is must.

thanks

[brancz]

I'm happy to review if anyone wants to work on this.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close