Integrate Gatekeeper deny results with Kubernetes --dry-run

[open-policy-agent]

Integrate Gatekeeper deny results with Kubernetes --dry-run

[ritazh]

To clarify, do you mean to support kubectl --dry-run for validation not mutation?

[maxsmythe]

+1 to this

[ritazh]

I can repro this:

kubectl apply -f bad.yaml --server-dry-run
Error from server (BadRequest): error when creating "bad.yaml": admission webhook "validation.gatekeeper.sh" does not support dry run

I think we need to implement this:
https://kubernetes.io/docs/reference/using-api/api-concepts/#make-a-dry-run-request

[maxsmythe]

+1

[ritazh]

If we update sideEffects: None under webhooks in the validation webhook config per https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.12/#webhook-v1beta1-admissionregistration-k8s-io, then --server-dry-run works as expected.

To prevent updating sideEffects manually, there are two options:

1. add sideEffectsto kubebuilder as webhook builder does not expose this in kubebuilder v0.1.9
   https://github.com/open-policy-agent/gatekeeper/blob/master///vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/builder/builder.go#L33-L69 This would mean we have to vendor in the latest v0.1.x to support this. WDYT? @maxsmythe
2. deploy the validation webhook config manually instead of having kubebuilder to create it. Use the enable-manual-deploy flag.

[cypherfox]

Does the webhook actually have any side effects? Or can I simply tweak this in our cluster as a workaround?
The lack of support for dry-run is currently the only thing that blocks us from using gatekeeper, as our deployment solution depends on this check.

[maxsmythe]

Currently no side effects. I'm not aware of any plans to add any in the near term.

The most likely candidate for side effects in the future would be enforcementAction supporting actions like sendalert

[ritazh]

@cypherfox does option 2 work for you?

2. deploy the validation webhook config manually instead of having kubebuilder to create it. Use the enable-manual-deploy flag.

When you deploy the validating webhook config, you can add sideEffects: None under webhooks https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.12/#webhook-v1beta1-admissionregistration-k8s-io, then --server-dry-run works as expected.

[cypherfox]

Yes, manual webhook deployment is ok as a workaround. I will check this, but am pretty sure. I will just throw the additional spec into our helm chart.

But it would create an additional redundancy, which we would need to track. So, over the mid-term I would prefer a kubebuilder based solution.

I have currently just edited the deployed config by hand, after starting the controller manager. Having this pet is ok for our initial setup phase.

[cypherfox]

Ah, I just noticed: getting the CA bundle might get a bit fiddly.
Where do I get that from (in an automated manner)?

[maxsmythe]

secret is stored in the gatekeeper-system namespace:

kubectl get secrets -n gatekeeper-system gatekeeper-webhook-server-secret

Note that how keys are generated and integrated will change when we move to kubebuilder v2

[cypherfox]

Thank you, found it (and should have found myself. /me is a bad elf! )

Unfortunately it will require a manual update, each time I restart the operator, because of kubernetes/kubernetes#72944.

This is currently a blocker for me to roll this out in production, as nobody can read secrets in our production cluster (they are all generated/populated automatically)

[maxsmythe]

With the newest release, this should work out-of-the-box