Kyverno Policies for Kubernetes Security: Validate, Mutate, Generate

What admission control adds to a checklist#

A Kubernetes security checklist tells you what a good pod spec looks like. It does not stop the next kubectl apply from ignoring it. Admission control does: every create and update passes through the API server’s admission chain, and a policy engine there can refuse the object, fix it, or record that it would have refused it.

Kyverno is a policy engine written for Kubernetes rather than adapted to it. Since version 1.19 its stable policy types (ValidatingPolicy, MutatingPolicy, GeneratingPolicy, ImageValidatingPolicy in the policies.kyverno.io/v1 API group) are supersets of the Kubernetes ValidatingAdmissionPolicy and MutatingAdmissionPolicy types, written in CEL. The older kyverno.io/v1 ClusterPolicy still works but the API server now prints a deprecation warning when you apply one, so this article uses the current types.

Everything below was run against Kubernetes 1.35 on kind with Kyverno 1.19.1. Where output is shown, it is the real output, trimmed only for width.

Architecture#

A request from kubectl or a GitOps controller reaches the API server, which authenticates and authorises it. Mutating admission runs first: Kyverno mutate rules can add defaults such as a seccomp profile. After schema validation, validating admission runs: Pod Security Admission and Kyverno validate rules can reject the object with a message. Only accepted objects are persisted to etcd. Separately, the Kyverno background controller scans existing resources and writes PolicyReports, and generate rules create resources such as a default-deny NetworkPolicy when a namespace appears.

1. API server. A request from kubectl, a CI job or a GitOps controller is authenticated and authorised (RBAC) before admission runs. Admission never replaces RBAC; a policy cannot grant permission, only restrict or shape what an already-permitted request produces.
2. Mutating admission. Kyverno’s admission controller is registered as a mutating webhook. MutatingPolicy resources run here and can add fields the manifest left out, such as a default seccomp profile.
3. Validating admission. After the object’s schema is checked, validating webhooks run. Pod Security Admission (built in) and Kyverno ValidatingPolicy resources both live here. A policy whose action is Deny rejects the request with the policy’s message; one whose action is Audit records a violation and lets the request through.
4. Rejected. The client receives the message from the policy. This is the entire user experience of admission control, which is why the message field matters more than it looks.
5. etcd. Only admitted objects are stored. Nothing Kyverno does happens after this point for the request path.
6. Background scan. A separate controller evaluates policies against existing resources and writes PolicyReport objects. This is how you learn what a Deny policy would break before you switch it on.
7. Generate. When a trigger resource appears (a new Namespace, say), a GeneratingPolicy creates another resource from a template, such as a default-deny NetworkPolicy, and keeps it there.

The trust boundary is the API server. Kyverno’s own service account is highly privileged (it must read most resources and create some), so its namespace is treated like kube-system.

Prerequisites

• A disposable cluster: kind 0.31 was used here (kind create cluster --name policy-lab)
• kubectl and helm 3.x or 4.x
• Optional: the kyverno CLI for testing policies against manifests without a cluster

Implementation#

Install with Helm#

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno --namespace kyverno --create-namespace --version 3.9.1
kubectl -n kyverno rollout status deploy --timeout=600s

Pin the chart version. The default values are right for a lab; for production the chart supports three replicas of the admission controller and a PodDisruptionBudget, and you want both, because an unavailable admission webhook with a Fail failure policy blocks every matching request in the cluster. Kyverno registers its webhooks with failurePolicy: Fail for exactly the reason you would want: a missing policy engine should not mean an open door. The trade-off is availability, and the mitigation is replicas plus resource requests that keep the controller scheduled.

On a slow network the image pulls can take longer than Helm’s default five-minute wait, which marks the release failed even though the deployments finish rolling out a few minutes later; the rollout status line above is the check that matters. After it returns:

NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
kyverno-admission-controller    1/1     1            1           21m
kyverno-background-controller   1/1     1            1           21m
kyverno-cleanup-controller      1/1     1            1           21m
kyverno-reports-controller      1/1     1            1           21m

The admission controller is the only one on the request path. The others can be down without blocking the API.

Policy 1: containers must run as non-root#

apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: require-run-as-nonroot
spec:
  validationActions: [Audit] # start here; switch to [Deny] after reading the reports
  evaluation:
    background:
      enabled: true
  matchConstraints:
    resourceRules:
      - apiGroups: ['']
        apiVersions: ['v1']
        operations: [CREATE, UPDATE]
        resources: [pods]
  matchConditions:
    - name: exclude-system-namespaces
      expression: "!(request.namespace in ['kube-system', 'kyverno', 'local-path-storage'])"
  validations:
    - expression: >-
        object.spec.?securityContext.?runAsNonRoot.orValue(false) == true ||
        object.spec.containers.all(c, c.?securityContext.?runAsNonRoot.orValue(false) == true)
      message: 'Containers must run as non-root: set securityContext.runAsNonRoot=true at pod level or on every container.'

Three things to read in that policy:

• matchConstraints is the Kubernetes ValidatingAdmissionPolicy syntax: which API group, kinds and operations the policy sees. matchConditions narrows it further with CEL; here it keeps system namespaces out, using request.namespace so the expression works for both create and update.
• The validation expression uses CEL’s optional syntax. object.spec.?securityContext.?runAsNonRoot.orValue(false) reads “the value of runAsNonRoot, or false if any part of the path is missing”. Without the ?. operators a pod that omits securityContext would error rather than fail cleanly.
• evaluation.background.enabled: true makes the background controller evaluate existing pods too, which feeds the reports used for rollout.

Kyverno also auto-generates variants of this policy for pod controllers (Deployments, StatefulSets, Jobs, CronJobs and the rest) by rewriting the expression against spec.template. A Deployment with a bad pod template is therefore rejected at the Deployment, with the message shown to the person who ran kubectl apply, instead of silently producing a ReplicaSet that can never create pods.

Policy 2: reject latest and untagged images#

apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: disallow-latest-tag
spec:
  validationActions: [Audit]
  evaluation:
    background:
      enabled: true
  matchConstraints:
    resourceRules:
      - apiGroups: ['']
        apiVersions: ['v1']
        operations: [CREATE, UPDATE]
        resources: [pods]
  matchConditions:
    - name: exclude-system-namespaces
      expression: "!(request.namespace in ['kube-system', 'kyverno', 'local-path-storage'])"
  validations:
    - expression: >-
        object.spec.containers.all(c,
          c.image.contains('@sha256:') || (c.image.contains(':') && !c.image.endsWith(':latest')))
      messageExpression: >-
        "Every container image needs an explicit tag or digest; " +
        object.spec.containers
          .filter(c, !(c.image.contains('@sha256:') || (c.image.contains(':') && !c.image.endsWith(':latest'))))
          .map(c, c.image).join(', ') +
        " is mutable."
      message: 'Every container image needs an explicit tag or digest; :latest and untagged images are mutable.'

nginx (no tag) and nginx:latest are different strings that mean the same mutable thing, and a digest reference (image@sha256:…) is the only truly immutable form; the expression accepts a digest, or a tag that is not latest. messageExpression builds the rejection text from the offending images so the developer does not have to guess which of six containers failed; message is the fallback if the expression itself errors. One limitation to know: a registry with a port (registry.internal:5000/app) contains a colon and satisfies the tag check even without a tag. If you run such a registry, check the part after the last / instead.

Policy 3: allow only approved registries#

apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: restrict-image-registries
spec:
  validationActions: [Audit]
  evaluation:
    background:
      enabled: true
  matchConstraints:
    resourceRules:
      - apiGroups: ['']
        apiVersions: ['v1']
        operations: [CREATE, UPDATE]
        resources: [pods]
  matchConditions:
    - name: exclude-system-namespaces
      expression: "!(request.namespace in ['kube-system', 'kyverno', 'local-path-storage'])"
  variables:
    - name: allowedPrefixes
      expression: "['registry.example.com/', 'docker.io/nginxinc/', 'nginxinc/']"
  validations:
    - expression: >-
        object.spec.containers.all(c, variables.allowedPrefixes.exists(p, c.image.startsWith(p)))
      message: 'Images must come from registry.example.com or the approved nginxinc namespace on Docker Hub.'

variables keeps the allow-list in one place. Note the two spellings for Docker Hub: Kubernetes normalises nginxinc/x to docker.io/nginxinc/x when it pulls, but the pod spec you are validating contains whatever the author wrote, so either match both or reject the short form deliberately.

Policy 4: mutate a missing seccomp profile into place#

Validation says no. Mutation says “here is what you meant”. For fields with one correct value and no reason to vary, mutation is kinder to developers and produces the same security outcome.

apiVersion: policies.kyverno.io/v1
kind: MutatingPolicy
metadata:
  name: add-default-seccomp
spec:
  autogen:
    podControllers:
      controllers: [] # pods only; see the note below
  matchConstraints:
    resourceRules:
      - apiGroups: ['']
        apiVersions: ['v1']
        operations: [CREATE]
        resources: [pods]
  matchConditions:
    - name: exclude-system-namespaces
      expression: "!(request.namespace in ['kube-system', 'kyverno', 'local-path-storage'])"
  mutations:
    - patchType: ApplyConfiguration
      applyConfiguration:
        expression: >-
          object.spec.?securityContext.?seccompProfile.hasValue() ? Object{} :
          Object{
            spec: Object.spec{
              securityContext: Object.spec.securityContext{
                seccompProfile: Object.spec.securityContext.seccompProfile{ type: "RuntimeDefault" }
              }
            }
          }

ApplyConfiguration patches are merged server-side-apply style, so the expression returns only the fields to add. The conditional keeps the policy from overwriting an explicit profile, including Unconfined for a workload that genuinely needs it; whether that should be allowed is a separate validating policy.

The autogen block is not optional here. Mutating expressions are typed against the object they patch (Object.spec.securityContext…), and the auto-generated Deployment variant would need Object.spec.template.spec… instead. Left at the default, this policy rejects every Deployment with a type error. Restricting it to pods is correct anyway: pods created by a Deployment’s ReplicaSet still pass through admission and still get mutated, which the verification below shows.

Policy 5: generate a default-deny NetworkPolicy for every new namespace#

apiVersion: policies.kyverno.io/v1
kind: GeneratingPolicy
metadata:
  name: generate-default-deny
spec:
  evaluation:
    synchronize:
      enabled: true # recreate the policy if someone deletes it
  matchConstraints:
    resourceRules:
      - apiGroups: ['']
        apiVersions: ['v1']
        operations: [CREATE]
        resources: [namespaces]
  matchConditions:
    - name: exclude-system-namespaces
      expression: "!(object.metadata.name in ['kube-system', 'kube-public', 'kube-node-lease', 'kyverno', 'local-path-storage'])"
  variables:
    - name: nsName
      expression: 'object.metadata.name'
    - name: downstream
      expression: >-
        [
          {
            "kind": dyn("NetworkPolicy"),
            "apiVersion": dyn("networking.k8s.io/v1"),
            "metadata": dyn({ "name": "default-deny-all", "namespace": string(variables.nsName) }),
            "spec": dyn({ "podSelector": dyn({}), "policyTypes": dyn(["Ingress", "Egress"]) })
          }
        ]
  generate:
    - expression: generator.Apply(variables.nsName, variables.downstream)

With synchronize on, the generated policy is owned by the trigger: delete it and Kyverno puts it back, modify it and Kyverno reverts it. Teams add explicit allow policies next to it; the deny stays. To back-fill namespaces that existed before the policy, add evaluation.generateExisting.enabled: true.

Apply and inspect#

kubectl apply -f require-run-as-nonroot.yaml -f disallow-latest-tag.yaml \
  -f restrict-image-registries.yaml -f add-default-seccomp.yaml -f generate-default-deny.yaml
kubectl get vpol,mpol,gpol

NAME                                                             AGE   READY
validatingpolicy.policies.kyverno.io/disallow-latest-tag         10s   true
validatingpolicy.policies.kyverno.io/require-run-as-nonroot      10s   true
validatingpolicy.policies.kyverno.io/restrict-image-registries   10s   true

NAME                                                     AGE   READY
mutatingpolicy.policies.kyverno.io/add-default-seccomp   10s   true

NAME                                                         AGE
generatingpolicy.policies.kyverno.io/generate-default-deny   10s

READY: true means Kyverno has registered a webhook for the policy. Until then, matching requests are admitted as if the policy did not exist, which is a few seconds on a healthy cluster and worth knowing when you script policy deployment followed immediately by a test.

Rollout: audit first, then deny#

Every validating policy above starts with validationActions: [Audit]. Leave them there long enough for the background controller to see your workloads, then read the reports. Each PolicyReport is scoped to one resource, so the useful query joins the namespace and the scope name to the failing results:

kubectl get policyreport -A -o json | jq -r '
  .items[] | .metadata.namespace as $ns | .scope.name as $name
  | .results[]? | select(.result == "fail")
  | "\($ns)/\($name)\t\(.policy)\t\(.message)"'

payments/audit-test	disallow-latest-tag	Every container image needs an explicit tag or digest; nginxinc/nginx-unprivileged is mutable.

That row is a workload enforcement would block. Fix the manifests (or add a namespace to matchConditions with a written reason), watch the failing count reach zero, then change [Audit] to [Deny] for that one policy. One policy at a time; a cluster where five policies flip to Deny in one apply is a cluster where nobody knows which one broke the deployment.

Verification#

With the policies in Deny mode, a non-compliant pod is rejected with the message from the policy:

kubectl run bad --image=nginx:latest --restart=Never

Error from server: admission webhook "vpol.validate.kyverno.svc-fail-finegrained-require-run-as-nonroot"
denied the request: Policy require-run-as-nonroot failed: Containers must run as non-root: set
securityContext.runAsNonRoot=true at pod level or on every container.

That pod violates three policies, and the error names one. Kyverno 1.19 registers a webhook per policy, and the API server stops at the first webhook that denies, so the developer fixes runAsNonRoot, resubmits, and is told about the latest tag next. The audit reports are where you see all three at once.

A compliant pod goes through and comes out mutated:

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: good
spec:
  securityContext:
    runAsNonRoot: true
  containers:
    - name: app
      image: nginxinc/nginx-unprivileged:1.27
EOF
kubectl get pod good -o jsonpath='{.spec.securityContext}'

pod/good created
{"runAsNonRoot":true,"seccompProfile":{"type":"RuntimeDefault"}}

The seccompProfile was not in the manifest. The same happens for a pod created by a Deployment, because the ReplicaSet’s pod creation is an admission request like any other, while a Deployment whose template fails validation is rejected up front:

kubectl create deployment bad-deploy --image=nginx:1.27

error: failed to create deployment: admission webhook "vpol.validate.kyverno.svc-fail-finegrained-require-run-as-nonroot"
denied the request: Policy require-run-as-nonroot failed: Containers must run as non-root: ...

And a new namespace arrives with its network policy already in place, and keeps it:

kubectl create namespace payments
kubectl -n payments get networkpolicy
kubectl -n payments delete networkpolicy default-deny-all
sleep 5 && kubectl -n payments get networkpolicy

namespace/payments created
NAME               POD-SELECTOR   AGE
default-deny-all   <none>         4s
networkpolicy.networking.k8s.io "default-deny-all" deleted
NAME               POD-SELECTOR   AGE
default-deny-all   <none>         3s

Security implications#

• Admission is not a scanner. Kyverno evaluates the object at admission time. A base image that was fine on Monday and vulnerable on Friday passes every one of these policies. Image scanning (Trivy) and signature verification (ImageValidatingPolicy with Cosign or Notary) cover that gap.
• Webhook availability is a security property. With failurePolicy: Fail, an unavailable Kyverno rejects every matching request, including the ones needed to bring Kyverno back if its own namespace were not excluded. Run more than one replica and set resource requests. Ignore fails open and admits everything silently. Choose deliberately.
• Kyverno sees every object it validates. Its service account has broad read access, and the webhook receives full object bodies, including Secrets if you match them. Do not write policies that log or copy secret data, and restrict who can create cluster-scoped policies: a policy author is close to a cluster admin. The namespaced variants (NamespacedValidatingPolicy and friends) exist so teams can add rules for their own namespace without that power.
• Pod Security Admission still belongs on. Kyverno does not replace the built-in restricted profile; it extends it with rules PSA cannot express (registries, tags, generated resources, mutation). Run both.

Troubleshooting#

Running this in production#

• Store policies in Git and deploy them with the same GitOps controller as everything else, in their own project with cluster-scoped resources allowed only for the policies.kyverno.io group.
• Test policies in CI with the Kyverno CLI against your manifests (kyverno apply policies/ --resource manifests/) so a change to a policy shows its blast radius in the merge request.
• Export report counts to Prometheus (the chart exposes metrics) and alert on a rising fail count in Audit policies; that is drift you would otherwise discover during an incident.
• Upgrade Kyverno on a schedule and read the release notes. If you still have ClusterPolicy objects, the migration guide covers the CEL equivalents.

References#

• Kyverno: ValidatingPolicy, MutatingPolicy, GeneratingPolicy
• Kyverno: migration to CEL policies and policy library
• Kyverno Helm chart
• Kubernetes: Validating Admission Policy and Dynamic Admission Control
• Kubernetes: Pod Security Admission
• CEL language definition