Enforce Pod Security with Pod Security Admission and Kyverno

Goal#

Two admission controls, layered. First the built-in one: Pod Security Admission rejects pods that do not meet the restricted profile, and tells you every field that is missing. Then Kyverno, for a rule Pod Security cannot express (no :latest tags), rolled out the way you would in production: audit first, read the report, then deny. All outputs below are from a kind cluster running Kubernetes 1.35 and Kyverno 1.19.1.

Environment#

kind create cluster --name policy-lab
kubectl create namespace payments

Step 1: label the namespace, reject a default pod#

kubectl label namespace payments \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=latest \
  pod-security.kubernetes.io/warn=restricted
kubectl -n payments run bad --image=nginx:1.27 --restart=Never

Error from server (Forbidden): pods "bad" is forbidden: violates PodSecurity "restricted:latest":
allowPrivilegeEscalation != false (container "bad" must set securityContext.allowPrivilegeEscalation=false),
unrestricted capabilities (container "bad" must set securityContext.capabilities.drop=["ALL"]),
runAsNonRoot != true (pod or container "bad" must set securityContext.runAsNonRoot=true),
seccompProfile (pod or container "bad" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

Nothing was installed for this. The message lists all four missing fields at once, which is the best admission error message in the ecosystem and the reason to turn on Pod Security before anything else.

Step 2: a pod that satisfies restricted#

apiVersion: v1
kind: Pod
metadata:
  name: good
  namespace: payments
spec:
  automountServiceAccountToken: false
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile: { type: RuntimeDefault }
  containers:
    - name: app
      image: nginxinc/nginx-unprivileged:1.27
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities: { drop: ['ALL'] }
      resources:
        requests: { cpu: '100m', memory: '64Mi' }
        limits: { cpu: '500m', memory: '128Mi' }
      volumeMounts:
        - { name: tmp, mountPath: /tmp }
        - { name: cache, mountPath: /var/cache/nginx }
        - { name: run, mountPath: /var/run }
  volumes:
    - { name: tmp, emptyDir: {} }
    - { name: cache, emptyDir: {} }
    - { name: run, emptyDir: {} }

kubectl apply -f good.yaml
kubectl -n payments wait --for=condition=Ready pod/good --timeout=300s
kubectl -n payments exec good -- id

pod/good created
pod/good condition met
uid=10001 gid=0(root) groups=0(root)

The three emptyDir mounts exist because readOnlyRootFilesystem: true means nginx cannot write its cache or PID file anywhere else. That is the usual reason a restricted rollout breaks an application: not the security context, but the writable paths nobody documented.

Step 3: install Kyverno#

helm repo add kyverno https://kyverno.github.io/kyverno/ && helm repo update
helm install kyverno kyverno/kyverno --namespace kyverno --create-namespace --version 3.9.1
kubectl -n kyverno rollout status deploy --timeout=900s

deployment "kyverno-admission-controller" successfully rolled out
deployment "kyverno-background-controller" successfully rolled out
deployment "kyverno-cleanup-controller" successfully rolled out
deployment "kyverno-reports-controller" successfully rolled out

If Helm reports the release as failed because its five-minute wait expired during image pulls, ignore it; the rollout status above is the check that matters.

Step 4: a rule Pod Security cannot express, in Audit mode#

apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: disallow-latest-tag
spec:
  validationActions: [Audit]
  evaluation:
    background:
      enabled: true
  matchConstraints:
    resourceRules:
      - apiGroups: ['']
        apiVersions: ['v1']
        operations: [CREATE, UPDATE]
        resources: [pods]
  matchConditions:
    - name: exclude-system-namespaces
      expression: "!(request.namespace in ['kube-system', 'kyverno', 'local-path-storage'])"
  validations:
    - expression: >-
        object.spec.containers.all(c,
          c.image.contains('@sha256:') || (c.image.contains(':') && !c.image.endsWith(':latest')))
      messageExpression: >-
        "Every container image needs an explicit tag or digest; " +
        object.spec.containers
          .filter(c, !(c.image.contains('@sha256:') || (c.image.contains(':') && !c.image.endsWith(':latest'))))
          .map(c, c.image).join(', ') +
        " is mutable."
      message: 'Every container image needs an explicit tag or digest.'

kubectl apply -f disallow-latest-tag.yaml
kubectl get vpol

validatingpolicy.policies.kyverno.io/disallow-latest-tag created
NAME                  AGE   READY
disallow-latest-tag   10s   true

Now create a pod that passes Pod Security but uses an untagged image. In Audit mode it is admitted:

sed 's/name: good/name: audit-test/; s/nginx-unprivileged:1.27/nginx-unprivileged/' good.yaml | kubectl apply -f -
sleep 30
kubectl get policyreport -A -o json | jq -r '
  .items[] | .metadata.namespace as $ns | .scope.name as $name
  | .results[]? | select(.result == "fail")
  | "\($ns)/\($name)\t\(.policy)\t\(.message)"'

pod/audit-test created
payments/audit-test	disallow-latest-tag	Every container image needs an explicit tag or digest; nginxinc/nginx-unprivileged is mutable.

The report is the list of what enforcement would break. In a real cluster you leave the policy here until that list is empty, or every remaining line has an owner.

Step 5: switch to Deny#

kubectl patch vpol disallow-latest-tag --type=merge -p '{"spec":{"validationActions":["Deny"]}}'
sleep 5
sed 's/name: good/name: deny-test/; s/nginx-unprivileged:1.27/nginx-unprivileged/' good.yaml | kubectl apply -f -

validatingpolicy.policies.kyverno.io/disallow-latest-tag patched
Error from server: error when creating "STDIN": admission webhook "vpol.validate.kyverno.svc-fail-finegrained-disallow-latest-tag"
denied the request: Policy disallow-latest-tag failed: Every container image needs an explicit tag or digest; nginxinc/nginx-unprivileged is mutable.

The same expression, the same message, now a rejection. Pod Security still runs first: a pod that fails both gets the Pod Security message, because the API server stops at the first admission plugin that denies.

Verification#

• kubectl -n payments run bad --image=nginx:1.27 is rejected by PodSecurity with four listed fields
• good.yaml is admitted and exec … id prints uid=10001
• kubectl get vpol shows READY: true
• In Audit mode the untagged pod is created and appears in the PolicyReport as fail
• In Deny mode the same manifest is rejected with the policy’s message

Cleanup#

kind delete cluster --name policy-lab

Troubleshooting#