cert-manager and metrics-server broken in 1.16 upgrade?

[digitalocean]

cert-manager and metrics-server broken in 1.16 upgrade?

[timoreimann]

I'd try some/all of the following for one of the applications:

1. Check the logs of the application pod(s)
2. Check which endpoint(s) the service maps to
3. Check if you can reach the endpoint(s) via kubectl port-forward
4. Check if you can reach the endpoint(s) from a running pod

Feel also free to submit a support ticket so that we can take a closer look.

[jmreicha]

1). Nothing in the cert-manager logs, looks like timout/connection errors in the cert manager logs.

E1118 16:33:21.799398       1 controller.go:131] cert-manager/controller/ingress-shim "msg"="re-queuing item  due to error processing" "error"="Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": the server is currently unable to handle the request" "key"="default/test"

2).

metrics-server

Name:              metrics-server
Namespace:         kube-system
Labels:            kubernetes.io/cluster-service=true
                   kubernetes.io/name=Metrics-server
Annotations:       kubectl.kubernetes.io/last-applied-configuration:
                     {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"kubernetes.io/cluster-service":"true","kubernetes.io/name":"Me...
Selector:          k8s-app=metrics-server
Type:              ClusterIP
IP:                10.245.47.213
Port:              <unset>  443/TCP
TargetPort:        main-port/TCP
Endpoints:         10.244.2.116:4443
Session Affinity:  None
Events:            <none>

cert-manager

Name:              cert-manager-webhook
Namespace:         cert-manager
Labels:            app=webhook
                   app.kubernetes.io/instance=cert-manager
                   app.kubernetes.io/managed-by=Tiller
                   app.kubernetes.io/name=webhook
                   helm.sh/chart=cert-manager-v0.11.0
Annotations:       kubectl.kubernetes.io/last-applied-configuration:
                     {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"webhook","app.kubernetes.io/instance":"cert-manager","ap...
Selector:          app.kubernetes.io/instance=cert-manager,app.kubernetes.io/managed-by=Tiller,app.kubernetes.io/name=webhook,app=webhook
Type:              ClusterIP
IP:                10.245.173.138
Port:              https  443/TCP
TargetPort:        6443/TCP
Endpoints:         10.244.3.27:6443
Session Affinity:  None
Events:            <none>

3 and 4). Can't seem to connect to these endpoints, they do respond, but I am getting a 403. Below is the response inside the cluster.

curl -k https://cert-manager-webhook.cert-manager
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403
}

What port is the API server configured to listen on? I wonder if the port mappings are incorrect or if there is something blocking requests to the API server?

[jmreicha]

Adding hostNetwork: true to the deployment spec "fixes" the issue but I'm not really sure why this would be needed.

The firewall rules in the DO console seem to indicate that the 10.0.0.0/8 network should be allowed, so I would have guessed that would include things in the Kubernetes cluster?

[timoreimann]

Hey @jmreicha. Sorry, this one fell off my radar.

Is this still an issue for you? If so, then I'd suggest to file a DO support ticket. That should kick off a process which is better suited to address customer support request in a reliable, short-term manner.

[jmreicha]

@timoreimann 👋

Still an issue, I already have a support ticket open. They said was better to use this issue 😄

[timoreimann]

@jmreicha did you manage to resolve the issue with our support, or am I misreading our internal communication?

[jmreicha]

@timoreimann Yep we got it sorted.

Just a note if anybody else comes across this issue, changing the cert-manager validating and mutating webhooks to failurePolicy: Ignore as well as restarting the control plane seems to fix the issue.

[timoreimann]

Thanks for the note explaining how you got this fixed, appreciated.