Web Hook in cert-manager is not working properly. Can anyone please me out.

[minigamkreddy]

Web Hook in cert-manager is not working properly. Can anyone please me out.

[hawksight]

Hey @minigamkreddy thanks for raising. This is a fairly common error to see and usually it is networking, DNS or cloud provider specific as the issue. We have a guide to help debug if you could try that first?

https://cert-manager.io/docs/troubleshooting/webhook/

Failing that, can you please share your k8s environment details.

In general every cert-manager resource if sent to the cert-manager-webhook deployment to validate the resource before it is saved to k8s to be actioned. It appears k8s cannot find that service, so please check that component is running.

[minigamkreddy]

Thanks For replying back

Yes I will Follow the link which you have provided me.

CERT MANAGER DETAILS
root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl get pods -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-7ddd8cdb9f-c7kwx 1/1 Running 1 38h
cert-manager-cainjector-57cd76c845-fk77m 1/1 Running 1 38h
cert-manager-webhook-cf8f9f895-n6n6q 1/1 Running 1 38h

Environment Details
Kubernetes version:
kubectl version
Client Version: v1.28.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.1

OS Details
Description: Ubuntu 20.04.6 LTS

cert-manager version: ert-manager-controller:
Image: quay.io/jetstack/cert-manager-controller:v1.14.5

Note : For more Details related to Environment please reply back.

MORE DEATAILS
root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl get endpoints -n cert-manager cert-manager-webhook
NAME ENDPOINTS AGE
cert-manager-webhook 192.168.224.104:10250 38h

kubectl get pod -n cert-manager -l app.kubernetes.io/name=webhook
NAME READY STATUS RESTARTS AGE
cert-manager-webhook-cf8f9f895-n6n6q 1/1 Running 1 39h

root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl get pod -n cert-manager -l app.kubernetes.io/name=webhook
NAME READY STATUS RESTARTS AGE
cert-manager-webhook-cf8f9f895-n6n6q 1/1 Running 1 39h

root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl logs -n cert-manager -l app.kubernetes.io/name=webhook | head -10
W0508 09:55:18.187126 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0508 09:55:18.273616 1 webhook.go:129] "using dynamic certificate generating using CA stored in Secret resource" logger="cert-manager.webhook" secret_namespace="cert-manager" secret_name="cert-manager-webhook-ca"
I0508 09:55:18.273781 1 server.go:146] "listening for insecure healthz connections" logger="cert-manager" address=":6080"
I0508 09:55:18.274173 1 server.go:206] "listening for secure connections" logger="cert-manager" address=":10250"
I0508 09:55:18.303989 1 reflector.go:351] Caches populated for *v1.Secret from k8s.io/[email protected]/tools/cache/reflector.go:229
I0508 09:55:19.280437 1 dynamic_source.go:255] "Updated cert-manager TLS certificate" logger="cert-manager" DNSNames=["cert-manager-webhook","cert-manager-webhook.cert-manager","cert-manager-webhook.cert-manager.svc"]
root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms#

root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl get deploy -n cert-manager cert-manager-webhook -oyaml | grep -A3 ports:
ports:
- containerPort: 10250
name: https
protocol: TCP

[minigamkreddy]

resource mapping not found for name: "cert-manager" namespace: "" from "test-resources.yaml": no matches for kind "Issuer" in version "v1"
ensure CRDs are installed first
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": proxyconnect tcp: dial tcp 10.10.224.60:3128: connect: connection refused
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": proxyconnect tcp: dial tcp 10.10.224.60:3128: connect: connection
refused
When I do the kubectl apply -f test-resources.yaml cert-manager is excepted the response from the proxy server.
10.10.224.60:3128: these should not happen cert-manager should except the response from the current cluster.

Below commands are not working
kubectl -n cert-manager port-forward deploy/cert-manager-webhook 6080

curl -sS --dump-header - 127.0.0.1:6080/healthz => These command except the response from there 10.10.224.60:3128. These is not communicated with interval cluster.

How to reslove these issue.

[hawksight]

If you run kubectl get crd do you see the cert-manager CRD resources in the cluster?
Just looking at this error:

resource mapping not found for name: "cert-manager" namespace: "" from "test-resources.yaml": no matches for kind "Issuer" in version "v1"
ensure CRDs are installed first

Perhaps the CRDs were not installed, and that could explain it. But i expect they are there otherwise there should be more failing components.

[minigamkreddy]

root@KmasterVM:/home/manoj/VM1_E810/cert-manager# kubectl get crd
NAME CREATED AT
bgpconfigurations.crd.projectcalico.org 2023-09-05T04:25:42Z
bgppeers.crd.projectcalico.org 2023-09-05T04:25:42Z
blockaffinities.crd.projectcalico.org 2023-09-05T04:25:42Z
caliconodestatuses.crd.projectcalico.org 2023-09-05T04:25:42Z
certificaterequests.cert-manager.io 2024-05-07T14:58:36Z
certificates.cert-manager.io 2024-05-07T14:58:36Z
challenges.acme.cert-manager.io 2024-05-07T14:58:36Z
clusterinformations.crd.projectcalico.org 2023-09-05T04:25:42Z
clusterissuers.cert-manager.io 2024-05-07T14:58:36Z
felixconfigurations.crd.projectcalico.org 2023-09-05T04:25:42Z
globalnetworkpolicies.crd.projectcalico.org 2023-09-05T04:25:42Z
globalnetworksets.crd.projectcalico.org 2023-09-05T04:25:42Z
hostendpoints.crd.projectcalico.org 2023-09-05T04:25:42Z
ipamblocks.crd.projectcalico.org 2023-09-05T04:25:42Z
ipamconfigs.crd.projectcalico.org 2023-09-05T04:25:42Z
ipamhandles.crd.projectcalico.org 2023-09-05T04:25:42Z
ippools.crd.projectcalico.org 2023-09-05T04:25:42Z
ipreservations.crd.projectcalico.org 2023-09-05T04:25:42Z
issuers.cert-manager.io 2024-05-07T14:58:36Z
kubecontrollersconfigurations.crd.projectcalico.org 2023-09-05T04:25:42Z
networkpolicies.crd.projectcalico.org 2023-09-05T04:25:42Z
networksets.crd.projectcalico.org 2023-09-05T04:25:42Z
orders.acme.cert-manager.io 2024-05-07T14:58:36Z
root@KmasterVM:/home/manoj/VM1_E810/cert-manager#

root@KmasterVM:/home/manoj/VM1_E810/cert-manager# kubectl apply -f test-resources.yaml
root@KmasterVM:/home/manoj/VM1_E810/cert-manager# kubectl apply -f test-resources.yaml
namespace/cert-manager unchanged
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": Service Unavailable
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": Service Unavailable