Error presenting challenge: the server could not find the requested resource

[snowdrop]

Error presenting challenge: the server could not find the requested resource

[morzan1001]

I use Raspberry Pi 4 with 8GB ram, but if I look at the workload, Pis with 4 GB ram would probably be enough. So far everything is pretty stable and the installation was easy with K3s.

[cmoulliard]

Did you see more information from the order resource (https://cert-manager.io/docs/troubleshooting/acme/) ?
Which version do you use of:

• Certificate Manager
• Kubernetes

[morzan1001]

I am using Kubernetes: v1.27.7 +k3s2 and Cert-Manager: v1.13.2

I took a closer look at the order and challenges and found the following output. I didn't see anything exciting here at first, except of course what was already in the first error message, but maybe I missed something?

kubectl -n cert-manager get issuer

NAME                       READY   AGE
godaddy-webhook-selfsign   True    31h
godaddy-webhook-ca         True    31h

kubectl get clusterissuer

NAME                     READY   AGE
letsencrypt-production   True    31h

kubectl describe clusterissuer letsencrypt-production

Name:         letsencrypt-production
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         ClusterIssuer
Metadata:
  Creation Timestamp:  2023-12-02T12:57:17Z
  Generation:          3
  Resource Version:    9711456
  UID:                <UUID>
Spec:
  Acme:
    Email:  <EMAIL>
    Private Key Secret Ref:
      Name:  letsencrypt-production
    Server:  https://acme-v02.api.letsencrypt.org/directory
    Solvers:
      dns01:
        Webhook:
          Config:
            API Key Secret Ref:
              Key:       token
              Name:      godaddy-api-key
            Production:  true
            Ttl:         600
          Group Name:    acme.mycompany.com
          Solver Name:   godaddy
      Selector:
        Dns Zones:
          <NAME>
Status:
  Acme:
    Last Private Key Hash:  <HASH>
    Last Registered Email:  <EMAIL>
    Uri:                    https://acme-v02.api.letsencrypt.org/acme/acct/1444635146
  Conditions:
    Last Transition Time:  2023-12-02T12:57:19Z
    Message:               The ACME account was registered with the ACME server
    Observed Generation:   3
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>

kubectl -n cert-manager describe order <NAME>

Name:         <NAME>
Namespace:    cert-manager
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: <NAME>
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: <NAME>
API Version:  acme.cert-manager.io/v1
Kind:         Order
Metadata:
  Creation Timestamp:  2023-12-02T12:57:36Z
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CertificateRequest
    Name:                  <NAME>
    UID:                   <UID>
  Resource Version:        9700802
  UID:                     <UID>
Spec:
  Common Name:  <NAME>
  Dns Names:
    <NAME>
    <NAME>
  Issuer Ref:
    Kind:   ClusterIssuer
    Name:   letsencrypt-production
  Request:  <REQUESTSTRING>
Status:
  Authorizations:
    Challenges:
      Token:        <TOKEN>
      Type:         dns-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/289667201276/3Ofw7Q
    Identifier:     <NAME>
    Initial State:  pending
    URL:            https://acme-v02.api.letsencrypt.org/acme/authz-v3/289667201276
    Wildcard:       true
    Challenges:
      Token:        <TOKEN>
      Type:         http-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/289667201286/RimpnQ
      Token:        <TOKEN>
      Type:         dns-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/289667201286/qrKBbA
      Token:        <TOKEN>
      Type:         tls-alpn-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/289667201286/sf9IGA
    Identifier:     <NAME>
    Initial State:  pending
    URL:            https://acme-v02.api.letsencrypt.org/acme/authz-v3/289667201286
    Wildcard:       false
  Finalize URL:     https://acme-v02.api.letsencrypt.org/acme/finalize/1444635146/226351673966
  State:            pending
  URL:              https://acme-v02.api.letsencrypt.org/acme/order/1444635146/226351673966
Events:             <none>

kubectl -n cert-manager describe challenge <NAME>

Name:         <NAME>
Namespace:    cert-manager
Labels:       <none>
Annotations:  <none>
API Version:  acme.cert-manager.io/v1
Kind:         Challenge
Metadata:
  Creation Timestamp:  2023-12-02T13:31:42Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  1
  Owner References:
    API Version:           acme.cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  <NAME>
    UID:                   <UID>
  Resource Version:        9711490
  UID:                     <UID>
Spec:
  Authorization URL:  https://acme-v02.api.letsencrypt.org/acme/authz-v3/289667201276
  Dns Name:           <NAME>
  Issuer Ref:
    Kind:  ClusterIssuer
    Name:  letsencrypt-production
  Key:     <KEY>
  Solver:
    dns01:
      Webhook:
        Config:
          API Key Secret Ref:
            Key:       token
            Name:      godaddy-api-key
          Production:  true
          Ttl:         600
        Group Name:    acme.mycompany.com
        Solver Name:   godaddy
    Selector:
      Dns Zones:
        <NAME>
  Token:     <TOKEN>
  Type:      DNS-01
  URL:       https://acme-v02.api.letsencrypt.org/acme/chall-v3/289667201276/3Ofw7Q
  Wildcard:  true
Status:
  Presented:   false
  Processing:  true
  Reason:      the server is currently unable to handle the request (post godaddy.acme.mycompany.com)
  State:       pending
Events:
  Type     Reason        Age                 From                     Message
  ----     ------        ----                ----                     -------
  Warning  PresentError  14m (x70 over 30h)  cert-manager-challenges  Error presenting challenge: the server is currently unable to handle the request (post godaddy.acme.mycompany.com)

[cmoulliard]

kubectl -n cert-manager describe challenge

Can you paste the content of YAML status please reported by the following command ?

kubectl -n cert-manager get challenge <NAME> -oyaml

[morzan1001]

status:
  presented: false
  processing: true
  reason: the server is currently unable to handle the request (post godaddy.acme.mycompany.com)
  state: pending

I think this is the same output as in the last output I provided.

[cmoulliard]

Do you see a more interesting message within the log of the Godaddy webhook pod or cert manager pod?

[morzan1001]

Yes, I think the API itself is not started correctly. The API service has the following status:

status:
  conditions:
    - lastTransitionTime: '2023-12-02T12:50:20Z'
      message: >-
        endpoints for service/godaddy-webhook in "cert-manager" have no
        addresses with port name "https"
      reason: MissingEndpoints
      status: 'False'
      type: Available

The pod itself does not seem to start correctly (?) In any case, I cannot view any logs from the pod:

kubectl -n cert-manager get pods

NAME                                     READY   STATUS             RESTARTS         AGE
cert-manager-cainjector-c86f8699-2f2sv   1/1     Running            0                44h
cert-manager-7d9dc8684c-m5kjh            1/1     Running            0                44h
cert-manager-7d9dc8684c-zcwkw            1/1     Running            0                44h
cert-manager-7d9dc8684c-p9f7q            1/1     Running            0                44h
cert-manager-webhook-f8f64cb85-hsjkb     1/1     Running            0                44h
godaddy-webhook-576c45df5f-cwcrl         0/1     CrashLoopBackOff   528 (103s ago)   44h

[cmoulliard]

So the webhhok's pod cannot start as apparently there is no service or service badly configured to expose the HTTPS port 443.
Here is what you should see:

k get pods -A
NAMESPACE              NAME                                        READY   STATUS    RESTARTS       AGE
cert-manager           cert-manager-8484c66d67-b7fhj               1/1     Running   1 (42d ago)    76d
cert-manager           cert-manager-cainjector-54c9d9b775-9zhrm    1/1     Running   1 (42d ago)    87d
cert-manager           cert-manager-webhook-7f7469bdb7-m5b5z       1/1     Running   1 (42d ago)    87d
cert-manager           godaddy-webhook-c6b5f74fd-5w7q6             1/1     Running   1 (42d ago)    87d

k get endpoints -n cert-manager
NAME                   ENDPOINTS             AGE
cert-manager           10.244.61.149:9402    87d
cert-manager-webhook   10.244.61.150:10250   87d
godaddy-webhook        10.244.61.147:443     87d

k get secrets -n cert-manager
NAME                          TYPE                DATA   AGE
cert-manager-webhook-ca       Opaque              3      87d
godaddy-webhook-ca            kubernetes.io/tls   3      87d
godaddy-webhook-webhook-tls   kubernetes.io/tls   3      87d

k get svc -n cert-manager
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
cert-manager           ClusterIP   10.103.153.41   <none>        9402/TCP   87d
cert-manager-webhook   ClusterIP   10.111.63.241   <none>        443/TCP    87d
godaddy-webhook        ClusterIP   10.107.66.226   <none>        443/TCP    87d

Remark: I also suspect that your service cert-manager-webhook is not exposed correctly too as it uses also the port HTTPS = 443.

[morzan1001]

You are right, it looks like the godaddy-webhook is not accessible via 443. But how do I solve this now?

kubectl get endpoints -n cert-manager

NAME                   ENDPOINTS                                         AGE
cert-manager           10.42.5.25:9402,10.42.8.18:9402,10.42.9.14:9402   45h
cert-manager-webhook   10.42.3.19:10250                                  45h
godaddy-webhook                                                          45h

kubectl get svc -n cert-manager

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
cert-manager-webhook   ClusterIP   10.43.226.12    <none>        443/TCP    45h
cert-manager           ClusterIP   10.43.71.104    <none>        9402/TCP   45h
godaddy-webhook        ClusterIP   10.43.205.193   <none>        443/TCP    45h

[cmoulliard]

Which error do you see (= its log or pod describe) around your godaddy-webhook-576c45df5f-cwcrl pod ?

[morzan1001]

The log for the pod only says:

exec /usr/local/bin/webhook: exec format error

And with your description I get the following:

Name:             godaddy-webhook-576c45df5f-cwcrl
Namespace:        cert-manager
Priority:         0
Service Account:  godaddy-webhook
Node:             cluster-agent-6/<IP>
Start Time:       Sat, 02 Dec 2023 13:50:21 +0100
Labels:           app.kubernetes.io/instance=godaddy-webhook
                  app.kubernetes.io/name=godaddy-webhook
                  pod-template-hash=<HASH>
Annotations:      <none>
Status:           Running
IP:               10.42.10.24
IPs:
  IP:           10.42.10.24
Controlled By:  ReplicaSet/godaddy-webhook-576c45df5f
Containers:
  godaddy-webhook:
    Container ID:  containerd://b7b2dde9107c1cb99021df50a719681c8d6c5c7f6faa58dda7c5a8fd419c7712
    Image:         quay.io/snowdrop/cert-manager-webhook-godaddy:0.2.0
    Image ID:      quay.io/snowdrop/cert-manager-webhook-godaddy@sha256:bea93fd77c4c5507bdcbf3a60e9ec424b532d00846669f07dec53bbe8fd31b1e
    Port:          443/TCP
    Host Port:     0/TCP
    Args:
      --tls-cert-file=/tls/tls.crt
      --tls-private-key-file=/tls/tls.key
      --secure-port=443
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 04 Dec 2023 11:05:56 +0100
      Finished:     Mon, 04 Dec 2023 11:05:56 +0100
    Ready:          False
    Restart Count:  535
    Liveness:       http-get https://:https/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get https://:https/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      GROUP_NAME:         acme.mycompany.com
      LOGGING_LEVEL:      info
      LOGGING_FORMAT:     color
      LOGGING_TIMESTAMP:  false
    Mounts:
      /tls from certs (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2bc8f (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  godaddy-webhook-webhook-tls
    Optional:    false
  kube-api-access-2bc8f:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                      From     Message
  ----     ------   ----                     ----     -------
  Normal   Pulled   14m (x533 over 45h)      kubelet  Container image "quay.io/snowdrop/cert-manager-webhook-godaddy:0.2.0" already present on machine
  Warning  BackOff  4m20s (x13045 over 45h)  kubelet  Back-off restarting failed container godaddy-webhook in pod godaddy-webhook-576c45df5f-cwcrl_cert-manager(df545dbb-cbe4-4380-bb79-19695d3ce4e5)

[cmoulliard]

exec /usr/local/bin/webhook: exec format error

Such error is typical about a difference between the compiled go application and platform where it runs - https://stackoverflow.com/questions/50785784/standard-init-linux-go178-exec-user-process-caused-exec-format-error-kuberne. What is the architecture of your k8s cluster (X86, ARM, etc) ?

[morzan1001]

The cluster is running on Raspberry PIs -> Architecture is ARM

[cmoulliard]

The cluster is running on Raspberry PIs -> Architecture is ARM

So we know why godaddy webhook fails to start. I will check if I can change the existing Dockerfile to release it for ARM platform ;-)

Remark: Such a post should help us: https://stackoverflow.com/questions/71372947/is-it-possible-to-manually-build-multi-arch-docker-image-without-docker-buildx

[morzan1001]

Thank you very much 👍 🥇

[cmoulliard]

Raspberry PIs

Something else: Which Pi version do you use and ram capacity you set to run your k8s cluster ?

[cmoulliard]

I created a PR to build a multi-arch image - #37 @morzan1001

[cmoulliard]

Can you make a test using this tagged image please quay.io/snowdrop/cert-manager-webhook-godaddy:pr-37-4b7cbaa@sha256:4bcdaa0f184e904e4c9431bc4b9e70b587e9bc49919ab6f8a2c6211f5d46a703 ? @morzan1001

[morzan1001]

Things are getting better :D

But unfortunately it doesn't work yet. The container starts now and can also register an API, but unfortunately it is still unhealthy. I can now see the following logs:

 kubectl -n cert-manager logs  godaddy-webhook-64cfdcd4f4-d4p4z

I1204 15:28:40.852556       1 handler.go:232] Adding GroupVersion acme.mycompany.com v1alpha1 to ResourceManager
I1204 15:28:40.864801       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I1204 15:28:40.864847       1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController
I1204 15:28:40.864976       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I1204 15:28:40.865016       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1204 15:28:40.865073       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I1204 15:28:40.865092       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1204 15:28:40.866590       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tls/tls.crt::/tls/tls.key"
I1204 15:28:40.867887       1 secure_serving.go:210] Serving securely on [::]:443
I1204 15:28:40.867985       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I1204 15:28:40.965192       1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController
I1204 15:28:40.965243       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1204 15:28:40.965190       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file

kubectl -n cert-manager describe pods

[...]
Events:
  Type     Reason     Age    From               Message
  ----     ------     ----   ----               -------
  Normal   Scheduled  8m6s   default-scheduler  Successfully assigned cert-manager/godaddy-webhook-64cfdcd4f4-d4p4z to cluster-agent-6
  Normal   Pulling    8m4s   kubelet            Pulling image "quay.io/snowdrop/cert-manager-webhook-godaddy:0.3.0"
  Normal   Pulled     7m57s  kubelet            Successfully pulled image "quay.io/snowdrop/cert-manager-webhook-godaddy:0.3.0" in 7.446531316s (7.446565483s including waiting)
  Normal   Created    7m57s  kubelet            Created container godaddy-webhook
  Normal   Started    7m56s  kubelet            Started container godaddy-webhook
  Warning  Unhealthy  7m55s  kubelet            Readiness probe failed: Get "https://10.42.10.28:443/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy  7m54s  kubelet            Readiness probe failed: Get "https://10.42.10.28:443/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

kubectl get endpoints -n cert-manager      

NAME                   ENDPOINTS                                         AGE
cert-manager           10.42.5.25:9402,10.42.8.18:9402,10.42.9.14:9402   2d3h
cert-manager-webhook   10.42.3.19:10250                                  2d3h
godaddy-webhook        10.42.10.28:443                                   16m

kubectl -n cert-manager get challenge <NAME> -oyaml

[...]
status:
  presented: false
  processing: true
  reason: the server is currently unable to handle the request (post godaddy.acme.mycompany.com)
  state: pending

[morzan1001]

Ok I don't think it's an architecture problem anymore, I just looked in my DNS records and a _acme-challenge TXT record was created.

[morzan1001]

Its working now for me, it just needed a bit more patience :)

[cmoulliard]

Can we then close this ticket as resolve ? @morzan1001

[morzan1001]

Yes, I thought you wanted to close the issue together with the pull request. Feel free to close the issue 👍

[cmoulliard]

Fixed with release: 0.3.0
See PR: #37