Require Pod Probes

[rubroboletus]

Require Pod Probes

[chipzoller]

@rubroboletus Can you please test with Kyverno 1.4.3 and see if you have the same issue? If so, please respond here with a test manifest for a CronJob or Job that can be used to verify.

[rubroboletus]

@chipzoller tested, still in trouble. Pasting also my Pod Probes Policy, added some exclude logic. But fails even with plain policy from here (also tested).

kind: Policy
metadata:
 name: require-pod-probes
 namespace: rh-test
 annotations:
   pod-policies.kyverno.io/autogen-controllers: DaemonSet,Deployment,StatefulSet
   policies.kyverno.io/title: Require Pod Probes
   policies.kyverno.io/category: Best Practices
   policies.kyverno.io/severity: medium
   policies.kyverno.io/subject: Pod
   policies.kyverno.io/description: >-
     Liveness and readiness probes need to be configured to correctly manage a Pod's
     lifecycle during deployments, restarts, and upgrades. For each Pod, a periodic
     `livenessProbe` is performed by the kubelet to determine if the Pod's containers
     are running or need to be restarted. A `readinessProbe` is used by Services
     and Deployments to determine if the Pod is ready to receive network traffic.
     This policy validates that all containers have liveness and readiness probes by
     ensuring the `periodSeconds` field is greater than zero.
spec:
 validationFailureAction: audit
 rules:
 - name: validate-livenessProbe-readinessProbe
   exclude:
     any:
     - resources:
            kinds:
            - Pod
            namespaces:
            - tigera-operator
            - calico-system
     - resources:
            kinds:
            - Pod
       clusterRoles:
       - cluster-admin
       subjects:
       - kind: Group
         name: system:masters
   match:
     any:
     - resources:
         kinds:
         - Pod
   validate:
     message: "Liveness and readiness probes are required."
     pattern:
       spec:
         containers:
         - livenessProbe:
             periodSeconds: ">0"
           readinessProbe:
             periodSeconds: ">0"

kind: CronJob
metadata:
  name: hello
  namespace: rh-test
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            imagePullPolicy: IfNotPresent
            command:
            - /bin/sh
            - -c
            - date; echo Hello from the Kubernetes cluster
          restartPolicy: OnFailure

job was created from CronJob using simple kubectl create job --from=cronjob/hello hello-2 -n rh-test --token $RH_TOKEN command.
That token has RoleBinding "admin" in mentioned namespace, so it will not match the excludes.
Attaching also log from Kyverno pod:

[chipzoller]

@rubroboletus A few things I'm seeing

1. This is a Policy and not a ClusterPolicy which means it's namespaced. The exclude clauses which name other Namespaces aren't valid.
2. You've set pod-policies.kyverno.io/autogen-controllers and haven't named Job or CronJob so the auto-gen controller will not generate a complimentary policy to block Pods emanating from those controllers.
3. validationFailureAction is set to audit which means it will not block anything even if it were valid, merely report on it.

[rubroboletus]

@chipzoller hello, a few things I'm seeing in Kyverno 1.4.3 log, when I try to apply this policy, just changed to ClusterPolicy and action to enforce:

you are right, that I was trying to test it fast for you, so used just Policy, to not break all the Kubernetes, used also by other people. I have not full access to fresh, unused EKS all the day. For testing purpose is "audit" better, so it will not break anything. But we need to enforce it on our production environments and currently it's impossible with 1.4.3 because of the parsing problem, already known, with autogen, (spec.rules[1].match ....). and with 1.4.2 mainly because of autogen is generating nothing when used with any/all rules.
But ok, lets test it without any additional controls ...

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-pod-probes
  annotations:
    pod-policies.kyverno.io/autogen-controllers: DaemonSet,Deployment,StatefulSet
    policies.kyverno.io/title: Require Pod Probes
    policies.kyverno.io/category: Best Practices
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      Liveness and readiness probes need to be configured to correctly manage a Pod's
      lifecycle during deployments, restarts, and upgrades. For each Pod, a periodic
      `livenessProbe` is performed by the kubelet to determine if the Pod's containers
      are running or need to be restarted. A `readinessProbe` is used by Services
      and Deployments to determine if the Pod is ready to receive network traffic.
      This policy validates that all containers have liveness and readiness probes by
      ensuring the `periodSeconds` field is greater than zero.
spec:
  validationFailureAction: enforce
  background: false
  rules:
  - name: validate-livenessProbe-readinessProbe
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Liveness and readiness probes are required."
      pattern:
        spec:
          containers:
          - livenessProbe:
              periodSeconds: ">0"
            readinessProbe:
              periodSeconds: ">0"
              

so as you can see, pod for this cronjob / job, definition in earlier comment, is not going to be created. CronJob / Job are created without problem. The ClusterPolicy I have used is from sample policies - https://kyverno.io/policies/best-practices/require_probes/require_probes/. The test above was done on 1.4.2. Lets test it also on 1.4.3.
And nothing changed so far, CronJob is created, Job is created, Pod for CronJob / Job is blocked:

[chipzoller]

Can you test this now that 1.5.0 is available and report whether we should keep the issue open?

[rubroboletus]

Sorry, tested it on 1.5.1, nothing changed. Used plain policy from sample policies, without any modifications. Then, like previously, created cronjob and using kubectl created job from cronjob. Pod not created, is blocked.

[chipzoller]

Going forward, would you mind not posting screenshots of text output but just copying-and-pasting into a code block?

Are you still using the ClusterPolicy and CronJob you posted above? If so, I'll give them another shot.

[chipzoller]

@rubroboletus the reason why it's not enforcing on Jobs and CronJobs is because the policy explicitly specifies the Kyverno auto-gen controller is ONLY to create corresponding policies for Deployment, DaemonSet, and StatefulSet resources. This is seen in your posts here and here via this line (L6 in the latter response):

pod-policies.kyverno.io/autogen-controllers: DaemonSet,Deployment,StatefulSet

You can just remove this annotation and re-create the policy. Kyverno will then create the corresponding rules for ALL Pod controllers. You can verify this by doing a kubectl get [cpol | pol] <policy_name> -o yaml and inspect the auto-generated rules.

I'll go ahead and close for now.