cert-manager/webhook-example

A cert-manager sample repository for creating an ACME DNS01 solver webhook

ACME webhook example

The ACME issuer type supports an optional 'webhook' solver, which can be used to implement custom DNS01 challenge solving logic.

This is useful if you need to use cert-manager with a DNS provider that is not officially supported in cert-manager core.

Why not in core?

As the project & adoption has grown, there has been an influx of DNS provider pull requests to our core codebase. As this number has grown, the test matrix has become un-maintainable and so, it's not possible for us to certify that providers work to a sufficient level.

By creating this 'interface' between cert-manager and DNS providers, we allow users to quickly iterate and test out new integrations, and then packaging those up themselves as 'extensions' to cert-manager.

We can also then provide a standardised 'testing framework', or set of conformance tests, which allow us to validate the a DNS provider works as expected.

Creating your own webhook

Webhook's themselves are deployed as Kubernetes API services, in order to allow administrators to restrict access to webhooks with Kubernetes RBAC.

This is important, as otherwise it'd be possible for anyone with access to your webhook to complete ACME challenge validations and obtain certificates.

To make the set up of these webhook's easier, we provide a template repository that can be used to get started quickly.

Creating your own repository

Running the test suite

All DNS providers must run the DNS01 provider conformance testing suite, else they will have undetermined behaviour when used with cert-manager.

It is essential that you configure and run the test suite when creating a DNS01 webhook.

An example Go test file has been provided in main_test.go.

You can run the test suite with:

$ TEST_ZONE_NAME=example.com. make test

The example file has a number of areas you must fill in and replace with your own options in order for tests to pass.

webhook-example's Issues

failed with: OpenAPI spec does not exist

Hi, I'm trying to implement Hetzner DNS API based on this sample, Everything is working as expected and the webhook issue the certificates normally, however, I'm getting this log error constantly:

controller.go:129] OpenAPI AggregationController: action for item v1alpha1.acme.mycompany.com: Rate Limited Requeue.
controller.go:116] loading OpenAPI spec for "v1alpha1.acme.mycompany.com" failed with: OpenAPI spec does not exist

any idea how to fix it?

Add logging example

See this Slack conversation for context- it would be great if we could show users how to get log output with the example webhook (i.e replace fmt.Printf here with a log line

Question: namespace to install the resources

Question

As the helm chart dont contain a namespace for several resources to be deployed excepted by example here - https://gist.github.com/cmoulliard/0aaa99004f930516c57902cefae2f89a#file-gadaddy-webhook-yaml-L1-L36, where should the resources be deployed (using default namespace, ...) ?

Examples:

What should be the namespace of the this value namespace: {{ .Release.Namespace | quote }} which is not defined within the helm values file - values.yaml for issuer, certificate

apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
  name: {{ include "example-webhook.selfSignedIssuer" . }}
  namespace: {{ .Release.Namespace | quote }}

Why this RoleBinding is installed under kube-system ?

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: {{ include "example-webhook.fullname" . }}:webhook-authentication-reader
  namespace: kube-system

Another example

apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1alpha1.{{ .Values.groupName }}
  labels:
    app: {{ include "example-webhook.name" . }}
    chart: {{ include "example-webhook.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
  annotations:
    certmanager.k8s.io/inject-ca-from: "{{ .Release.Namespace }}/{{ include "example-webhook.servingCertificate" . }}"
spec:
  group: {{ .Values.groupName }}
  groupPriorityMinimum: 1000
  versionPriority: 15
  service:
    name: {{ include "example-webhook.fullname" . }}
    namespace: {{ .Release.Namespace }}

Is there a reason you use sometimes | quote and sometimes not to generate the namespace name ?
Can this be better documented please and helm templates updated ?

Question: When the Present function is called

My webhook doesn't work if there are no _acme-challenge TXT records in the DNS record initially. Cert-manager constantly tries to get this entry with this message until I manually add the _acme-challenge entry with any value.

When querying the SOA record for the domain '_acme-challenge.kube.example.com.' using nameservers [10.96.0.10: ││ 53], rcode was expected to be 'NOERROR' or 'NXDOMAIN', but got 'SERVFAIL'" logger="cert-manager.challenges" key="cert-manager/wildcard-kube-example-com-2-3963891295-1304023134

The solution to my problem, as I see it, is to call the Present function when starting the certificate сhallenges, but the call only happens if I add the entry _acme-challenge manually with any value

This example really not work for cert-manger 0.12.0

Any plan to update it?

• apiVersion not match
• when upgrade dev dependency (for testing) cert-manger to 0.12.0, it will be errors when executing go test
• go mod update

integration test example does not work with certmanager v1.5.0

With v1.5.0 the test function dns.SetBinariesPath() has been removed and therefore the test example does not work anymore.

I have tried to adapt the test, but I have failed so far.

Set up basic e2e test that deploys the webhook and ensures we can POST a challenge

This test probably doesn't need cert-manager itself running for the time being.

I think the test should cover:

• Building this project
• Deploying the Helm chart
• Ensuring that the pod starts
• Ensuring that the APIService is marked Ready
• Ensuring that we can POST a ChallengePayload resource and get a 200 response

In future we may want to explore having some form of extended e2e that actually deploys and runs a version of cert-manager too, but let's keep the scope small for now 😄

apiVersion version is ourfdated since release 0.12.0

Issue

The apiversion is outdated since the release 0.12.0 as the apiversion to be used is

apiVersion: cert-manager.io/v1alpha2

and not anymore

apiVersion: certmanager.k8s.io/v1alpha1

To be fixed within the helm template resource and documented
: https://github.com/jetstack/cert-manager-webhook-example/blob/master/deploy/example-webhook/templates/pki.yaml#L4

Remark: I propose to create a branch in order to handle the apiversion change

Webhook CPU usage is rather high

I occasionally look onto my k8s metrics (CPU/Mem Usage) and i found that the webhook server uses rather much CPU (up to 20 percent) over the time.

From my understanding, a webhook server is a simple k8s OpenAPI server, an HTTP server, which should be actually sleeping most of the time, is this correct? So the CPU load in my expectation should be at most 5%.

in this case, i'm wondering what is causing that CPU load... 🤔

Set repository to be a GitHub template repository

This is an incredibly minor thing but GitHub's template repository functionality would be perfectly suited for this repo. It'll allow an easy alternative to forking or setting up a new remote: https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-repository-from-a-template

How to add api key secret during testing

I want to have my API credentials for my provider in a secret, as the comment above the structure suggests. However, the test will then fail, because the secret is not present.

Other examples use two config keys, one SecretKeySelector and one plain string, one is used during testing and one on an actual cluster. I don't like this solution as there is the possibility of a less than optimal configuration, and it also doesn't test the secret handling, as it will be executed in production.

Is there a place, where I can hook my own function in and submit my secret to the apiserver before testing?

failed to list *v1beta3.FlowSchema

I have upgraded my webhook to cert-manager v1.11 and have the following errors. I had also to change RBAC to support flowcontrol.apiserver.k8s.io group

W0307 09:07:34.843358       1 reflector.go:424] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1beta3.FlowSchema: the server could not find the requested resource
E0307 09:07:34.843372       1 reflector.go:140] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1beta3.FlowSchema: failed to list *v1beta3.FlowSchema: the server could not find the requested resource
W0307 09:07:46.213292       1 reflector.go:424] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1beta3.PriorityLevelConfiguration: the server could not find the requested resource
E0307 09:07:46.213356       1 reflector.go:140] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1beta3.PriorityLevelConfiguration: failed to list *v1beta3.PriorityLevelConfiguration: the server could not find the requested resource

Guidance on how to deploy webhook solver into a kubernetes cluster

I've been following along with #3 and trying my best to build a solver using this example as a starting point. I think I've done ok and based on what the tests are doing it seems to be working perfectly. I'm now just a little stuck on how exactly to make use of this solver in my actual kubernetes cluster 🤔

I figured because the solvers are Go libraries we will need to install them somehow, but I don't really have a clue how to proceed here. I'm going to go and look at some examples to try and divine how to get this in production but I thought I would open this issue to discuss adding some guidance to the readme or a partial example to the repo.

let me know what you think 😄

security description

The readme mentions its plugged into the apiserver for security reasons. I can see that resource, but I don't see anything on how trust is established/enforced between the apiserver and plugin. Is there a document somewhere that describes this?

cannot create resource "godaddy" in API group

Error presenting challenge: godaddy.acme.amprajin.in is forbidden: User "system:serviceaccount:cert-manager:cert-manager" cannot create resource "godaddy" in API group "acme.amprajin.in" at the cluster scope

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # ACME Server
    # prod : https://acme-v02.api.letsencrypt.org/directory
    # staging : https://acme-staging-v02.api.letsencrypt.org/directory
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # ACME Email address
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-staging # staging or production
    solvers:
    - selector:
        dnsNames:
        - '*.amprajin.in'
      dns01:
        webhook:
          config:
            apiKeySecretRef:
              name: godaddy-api-key
              key: token
            production: true
            ttl: 600
          groupName: acme.amprajin.in
          solverName: godaddy

Doesn't work with go versions >1.19

I have forked this repo to implement a custom solver for desec, but found myself unable to deploy a working instance on my up-to-date Debian system. The issue has been pinpointed as the go version used for implementing the hook, as an updated version will necessitate an update in the Dockerfile as well. The same problem was encountered by another user in #27

Steps to reproduce:

1. Install a version of go > 1.19 on a development machine
2. Implement a hook
3. Run go module init
4. Run go mod tidy
5. Extract go version from go.mod file and update the Dockerfile's FROM reference with it, i.e. set FROM golang:1.21-alpine3.18 AS build_dep
6. Build docker image for hook
7. Deploy hook

Expected Result:

The hook is deployed and the relevant cert-manager resources are up and running.

Actual Result:

The webhook pod does not start and ends in a CrashLoopBackOff state. Reviewing its logs, the following error shows up:
cert-manager: error executing command" err="error installing APIGroup for solvers: unable to get openapi models: OpenAPIV3 config must not be nil

Outcome:

Developers are forced to use either one of two ugly workarounds:

1. Install an old and outdated version of go (i.e. v1.19) on your system before you start implementing a hook
2. Diff your go.mod and go.sum files to those of another hook implemented in go v1.19, merging the newly added lines of your project into the existing lines of another working project, leaving the FROM reference in the Dockerfile at go v1.19

make test prints the apiKey

I have migrated my webhook to cert-manager v1.13.1.
Now I get the provided apiKey printed, which I do not want to see in automatic test protocols...

$ TEST_ZONE_NAME=my.domain. make test
TEST_ASSET_ETCD=_test/kubebuilder-1.28.0-linux-amd64/etcd \
TEST_ASSET_KUBE_APISERVER=_test/kubebuilder-1.28.0-linux-amd64/kube-apiserver \
TEST_ASSET_KUBECTL=_test/kubebuilder-1.28.0-linux-amd64/kubectl \
/usr/bin/go test -v .
=== RUN   TestRunsSuite
=== RUN   TestRunsSuite/Basic
=== RUN   TestRunsSuite/Basic/PresentRecord
    util.go:70: created fixture "basic-present-record"
    suite.go:38: Calling Present with ChallengeRequest: &v1alpha1.ChallengeRequest{UID:"", Action:"", Type:"", DNSName:"example.com", Key:"123d==", ResourceNamespace:"basic-present-record", ResolvedFQDN:"cert-manager-dns01-tests.my.domain.", ResolvedZone:"my.domain.", AllowAmbientCredentials:false, Config:(*v1.JSON)(0xc00070e270)}
I0315 12:42:10.574692  285584 main.go:142] call function Present: ResourceNamespace=basic-present-record, ResolvedZone=my.domain., ResolvedFQDN=cert-manager-dns01-tests.my.domain. DNSName=example.com
I0315 12:42:10.575049  285584 main.go:148] **Decoded configuration {dynu-secret}**
I0315 12:42:10.576462  285584 main.go:277] call function getDomainIdFromFQDN: **apiKey=abcABC123456**, ResolvedFQDN=cert-manager-dns01-tests.my.domain.
I0315 12:42:11.415191  285584 main.go:306] Domain Detail: id=9209690, node=cert-manager-dns01-tests
I0315 12:42:11.415221  285584 main.go:179] call function determineBaseRecordName: recordName=cert-manager-dns01-tests
I0315 12:42:12.234369  285584 main.go:375] Added TXT record result: {"statusCode":200,"id":10447515,"domainId":9209690,"domainName":"my.domain","nodeName":"cert-manager-dns01-tests","hostname":"cert-manager-dns01-tests.my.domain","recordType":"TXT","ttl":60,"state":true,"content":"cert-manager-dns01-tests.my.domain. 60 IN TXT \"123d==\"","updatedOn":"2024-03-15T11:42:12.117","textData":"123d=="}
I0315 12:42:12.994260  285584 main.go:375] Added TXT record result: {"statusCode":200,"id":10447516,"domainId":9209690,"domainName":"my.domain","nodeName":"","hostname":"my.domain","recordType":"TXT","ttl":60,"state":true,"content":"my.domain. 60 IN TXT \"123d==\"","updatedOn":"2024-03-15T11:42:12.89","textData":"123d=="}
I0315 12:42:12.994302  285584 main.go:173] Presented txt record cert-manager-dns01-tests.my.domain.
I0315 12:42:45.211687  285584 main.go:277] call function getDomainIdFromFQDN: **apiKey=abcABC123456**, ResolvedFQDN=cert-manager-dns01-tests.my.domain.
I0315 12:42:46.027191  285584 main.go:306] Domain Detail: id=9209690, node=cert-manager-dns01-tests
I0315 12:42:46.724078  285584 main.go:227] TXT entry with content my.domain. 60 IN TXT "123d==" (key value 123d==)
I0315 12:42:47.441375  285584 main.go:233] Deleted TXT record result: {"statusCode":200}
I0315 12:42:47.441414  285584 main.go:227] TXT entry with content cert-manager-dns01-tests.my.domain. 60 IN TXT "123d==" (key value 123d==)
I0315 12:42:48.391805  285584 main.go:233] Deleted TXT record result: {"statusCode":200}
I0315 12:42:48.391836  285584 main.go:227] TXT entry with content  (key value 123d==)
I0315 12:42:48.391846  285584 main.go:227] TXT entry with content apps.ocp4.my.domain. 60 IN TXT "07QkfbaBeGOsoMdL3NLxvN7yk4NMzUjhhAa7AhCVONQ" (key value 123d==)
I0315 12:42:48.391856  285584 main.go:227] TXT entry with content my.domain. 120 IN SOA ns1.dynu.com. administrator.dynu.com. 0 3600 900 604800 300 (key value 123d==)
I0315 12:42:48.425179  285584 main.go:277] call function getDomainIdFromFQDN: **apiKey=abcABC123456**, ResolvedFQDN=cert-manager-dns01-tests.my.domain.
I0315 12:42:49.153359  285584 main.go:306] Domain Detail: id=9209690, node=cert-manager-dns01-tests
I0315 12:42:49.901336  285584 main.go:227] TXT entry with content  (key value 123d==)
I0315 12:42:49.901363  285584 main.go:227] TXT entry with content apps.ocp4.my.domain. 60 IN TXT "07QkfbaBeGOsoMdL3NLxvN7yk4NMzUjhhAa7AhCVONQ" (key value 123d==)
I0315 12:42:49.901373  285584 main.go:227] TXT entry with content my.domain. 120 IN SOA ns1.dynu.com. administrator.dynu.com. 0 3600 900 604800 300 (key value 123d==)
--- PASS: TestRunsSuite (44.52s)
    --- PASS: TestRunsSuite/Basic (40.64s)
        --- PASS: TestRunsSuite/Basic/PresentRecord (40.64s)
PASS
ok  	github.com/Dopingus/cert-manager-webhook-dynu	44.544s

tests not working on macos monterey

Problem

macos monterey, go 17.2

got an error when running tests:

go test -v .
=== RUN   TestRunsSuite
    fixture.go:115: error starting apiserver: timeout waiting for process etcd to start

To problem cause is etcd - it is not starting:

V:cert-manager-webhook-vkcloud v$ _test/kubebuilder/bin/etcd 
fatal error: runtime: bsdthread_register error

runtime stack:
runtime.throw(0x1bed3fa, 0x21)
	/usr/local/go/src/runtime/panic.go:616 +0x81 fp=0x7ff7bfeff718 sp=0x7ff7bfeff6f8 pc=0x102a871
runtime.goenvs()
	/usr/local/go/src/runtime/os_darwin.go:129 +0x83 fp=0x7ff7bfeff748 sp=0x7ff7bfeff718 pc=0x10283f3
runtime.schedinit()
	/usr/local/go/src/runtime/proc.go:501 +0xd6 fp=0x7ff7bfeff7b0 sp=0x7ff7bfeff748 pc=0x102d166
runtime.rt0_go(0x7ff7bfeff7e8, 0x1, 0x7ff7bfeff7e8, 0x0, 0x1000000, 0x1, 0x7ff7bfeff970, 0x0, 0x7ff7bfeff98b, 0x7ff7bfeff9a7, ...)
	/usr/local/go/src/runtime/asm_amd64.s:252 +0x1f4 fp=0x7ff7bfeff7b8 sp=0x7ff7bfeff7b0 pc=0x1056474

After some investigation I've found this issue https://github.com/golang/go/wiki/MacOS12BSDThreadRegisterIssue, installed etcd manually, etcd starts but now I got kube-apiserver start timeout error.

Make unit testing easier/make examples work

Right now if you run go test within this repository, the tests will fail.

They fail for a few reasons:

• We don't provide any way to fetch the test binaries (i.e. kube-apiserver & etcd)
• The actual sample will always panic on Present/CleanUp
• We don't actually set any TXT records anywhere when Present is called, meaning the test suite will always fail

(1) can be fixed by adding a fetch-binaries.sh script (probably taken straight from sigs.k8s.io/controller-tools).

(2) and (3) can be most easily fixed by embedding a 'fake dns server' into the webhook, which will listen for DNS queries and serve TXT records accordingly. This is how cert-manager itself handles tests for the RFC2136 DNS01 provider, and there is a reusable library that we can use to build it easily: https://github.com/jetstack/cert-manager/tree/master/test/acme/dns/server

Panic: slice bounds out of range under arm64 because of old dependency pkg

Describe the bug:
I started to create a new webhook for another DNS provider. You can find it here: github.com/4nx/cert-manager-webook-joker. While building and testing the package under arm64 I found a bug within an old dependency package which is used.

TEST_ZONE_NAME=xxx go test .
panic: runtime error: slice bounds out of range [:9] with length 8

goroutine 1 [running]:
github.com/miekg/dns.ClientConfigFromFile(0x1346942, 0x10, 0x0, 0x0, 0x0)
	/home/ubuntu/go/pkg/mod/github.com/miekg/[email protected]/clientconfig.go:86 +0x8b4
github.com/jetstack/cert-manager/pkg/issuer/acme/dns/util.getNameservers(0x1346942, 0x10, 0x229a540, 0x2, 0x2, 0x0, 0x25, 0x21ba808)
	/home/ubuntu/go/pkg/mod/github.com/jetstack/[email protected]/pkg/issuer/acme/dns/util/wait.go:51 +0x30
github.com/jetstack/cert-manager/pkg/issuer/acme/dns/util.init()
	/home/ubuntu/go/pkg/mod/github.com/jetstack/[email protected]/pkg/issuer/acme/dns/util/wait.go:44 +0x88
FAIL	github.com/4nx/cert-manager-webhook-joker	0.122s
FAIL

The problem is that a very old version of github.com/miekg/dns package is still used:

https://github.com/jetstack/cert-manager-webhook-example/blob/7a722fd8517afee606f0c13673880f6eedee5d87/go.sum#L294

The project had a bug until version 1.0.4:

https://github.com/miekg/dns/blob/5364553f1ee9cddc7ac8b62dce148309c386695b/clientconfig.go#L94

which had been fixed in version 1.0.5 after this pull request miekg/dns#642 through:

miekg/dns@0079071

The most recent version of the project is 1.1.35.

Expected behaviour:
A working webhook under ARM64.

Steps to reproduce the bug:
Simply following: https://github.com/4nx/cert-manager-webhook-joker#development under ARM64

Anything else we need to know?:
The webhook is working under amd64.

Environment details::

• Kubernetes version: v1.20.0+k3s2
• Cloud-provider/provisioner: K3s under raspberry pi 4
• cert-manager version: v.1.1.0
• Install method: helm

/kind bug

Code reference a pull request to be merged, but the pull request was closed by a robot

webhook-example/main_test.go

Lines 36 to 37 in 9ea71ae

	//need to uncomment and RunConformance delete runBasic and runExtended once https://github.com/cert-manager/cert-manager/pull/4835 is merged
	//fixture.RunConformance(t)

It looks like the pull request cert-manager/cert-manager#4835 was closed automatically. Is this comment to be removed and the code to be uncommented, or should the pull-request be re-opened ?