Platform Guardrails with Kyverno and GitOps

Overview#

A platform team that owns fifty namespaces cannot review every Deployment, and should not have to. What it can do is state the handful of things every workload must get right, make the cluster refuse anything that does not, and deliver that rule set the same way it delivers everything else: as files in Git that Argo CD applies. The application teams then work against a contract they can read, with error messages that name what to fix, and a written way to ask for an exception.

Kyverno Policies for Kubernetes Security covers what Kyverno is, how CEL policies are written, and the security policies a cluster should carry (non-root, pinned images, approved registries, seccomp, default-deny networking). This article does not repeat them. It treats Kyverno as a delivery system for platform rules: which rules belong in a baseline layer and which only in production, how the layers are keyed to environments, how they are rolled out without breaking anyone, how exceptions are granted without switching a policy off, and how all of it moves through the ApplicationSet model instead of a kubectl apply from a laptop.

Everything was run on a kind 0.31 cluster with Argo CD 3.5.3 and Kyverno 1.19.1; a Gitea container served as the Git remote. The outputs below are from that run, including the one where the rollout was done in the wrong order.

The platform-config repository holds three kinds of content: policies (baseline and production layers), exceptions, and the application environments. Argo CD reconciles all three: two platform-project Applications deliver the policies and the exceptions, and an ApplicationSet generates one Application per environment with the environment label stamped on its namespace. Kyverno evaluates every admission request in a labelled namespace against the policies, honouring only the exceptions that live in the platform-exceptions namespace, and writes a PolicyReport per resource. A non-compliant change is refused at admission, which Argo CD reports as a failed sync on the Application that carried it.

1. One repository, platform-config, holds the policies, the exceptions and the application environments. Different directories have different owners (CODEOWNERS), but the delivery path is the same.
2. platform-guardrails is an Argo CD Application in the platform project that syncs policies/ automatically with prune and self-heal. A policy change is a merge; a deleted file is a pruned policy.
3. The ci-demo ApplicationSet generates one Application per environment and stamps each namespace with platform.example.com/env and platform.example.com/team labels through managedNamespaceMetadata. The labels are the platform’s, not the team’s, and the policies select on them.
4. Kyverno evaluates admission requests in labelled namespaces against the policies, honours exceptions only from the platform-exceptions namespace, and writes a PolicyReport per resource in background mode.
5. Each environment namespace carries its labels from creation, so a policy scoped to env: prod applies to ci-demo-prod and to nothing else.
6. A refused change surfaces where the change came from: as a failed sync on the Application that carried it, with Kyverno’s message in the operation state.

Repository layout#

platform-config/
├── policies/                       # owned by the platform team
│   ├── kustomization.yaml          #   resources: [baseline, production]
│   ├── baseline/
│   │   ├── require-team-label.yaml
│   │   └── require-resources.yaml
│   └── production/
│       ├── require-min-replicas.yaml
│       └── require-readiness-probe.yaml
├── exceptions/                     # platform team merges; application teams open the MR
│   └── report-scheduler-single-replica.yaml
├── apps/ci-demo/                   # owned by the application team
│   ├── base/
│   └── envs/{dev,stage,prod}/      # kustomization.yaml + config.json
├── appsets/ci-demo.yaml            # platform team
└── platform/                       # projects, the two platform Applications

The three Argo CD projects draw the boundary. platform may create the cluster-scoped ValidatingPolicy kind and deploy into kyverno and platform-exceptions; non-prod and prod may create namespaces and deploy into their own. An application team’s Application cannot ship a policy, and the platform team’s Application cannot touch a workload namespace.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata: { name: platform, namespace: argocd }
spec:
  description: Platform-owned resources. Cluster-scoped policies are allowed here and nowhere else.
  sourceRepos: ['http://gitea:3000/platform/platform-config.git']
  destinations:
    - { server: https://kubernetes.default.svc, namespace: 'platform-exceptions' }
    - { server: https://kubernetes.default.svc, namespace: 'kyverno' }
  clusterResourceWhitelist:
    - { group: 'policies.kyverno.io', kind: ValidatingPolicy }

The ApplicationSet is the one from the multi-environment article with one addition. Namespace labels are set by the platform when the namespace is created and reasserted on every sync, so a team cannot relabel its namespace out of the production layer:

syncPolicy:
  syncOptions: [CreateNamespace=true]
  managedNamespaceMetadata:
    labels:
      platform.example.com/env: '{{ .env }}'
      platform.example.com/team: payments

NAME            ENV     TEAM
ci-demo-dev     dev     payments
ci-demo-stage   stage   payments

Two layers, four policies#

The set is deliberately small. Every policy here exists because a real operational question depends on it: who owns this, will it schedule, will it survive a node loss, will a rollout wait for it to be ready.

Baseline: applied wherever the platform runs a workload#

apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: baseline-require-team-label
  annotations:
    platform.example.com/layer: baseline
    platform.example.com/owner: platform-team
spec:
  # Audit first: report, do not block. The switch to Deny is a Git commit.
  validationActions: [Audit]
  evaluation:
    background: { enabled: true }
  matchConstraints:
    resourceRules:
      - apiGroups: ['apps']
        apiVersions: ['v1']
        operations: [CREATE, UPDATE]
        resources: [deployments, statefulsets, daemonsets]
      - apiGroups: ['batch']
        apiVersions: ['v1']
        operations: [CREATE, UPDATE]
        resources: [jobs, cronjobs]
  matchConditions:
    - name: platform-managed-namespaces-only
      expression: "has(namespaceObject.metadata.labels) && 'platform.example.com/env' in namespaceObject.metadata.labels"
  validations:
    - expression: "has(object.metadata.labels) && 'platform.example.com/team' in object.metadata.labels && object.metadata.labels['platform.example.com/team'] != ''"
      message: 'Workloads need a platform.example.com/team label naming the owning team.'

The matchConditions line is the scoping mechanism for the whole system. The policy does not list namespaces to include or exclude; it applies to any namespace the platform labelled, which is exactly the set the ApplicationSet created. kube-system, kyverno, argocd and a namespace someone created by hand are outside it, by construction rather than by list maintenance.

The second baseline policy requires CPU and memory requests and a memory limit on every container. It matches Pods but uses autogen.podControllers so the check runs when the Deployment is applied, which is where the error is useful; a policy that only matched Pods would let the Deployment in and fail the ReplicaSet’s pod creation quietly.

Production: additional rules keyed on the environment label#

apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: production-require-min-replicas
  annotations:
    platform.example.com/layer: production
    platform.example.com/owner: platform-team
spec:
  validationActions: [Deny]
  evaluation:
    background: { enabled: true }
  matchConstraints:
    resourceRules:
      - apiGroups: ['apps']
        apiVersions: ['v1']
        operations: [CREATE, UPDATE]
        resources: [deployments]
  matchConditions:
    - name: production-namespaces-only
      expression: "has(namespaceObject.metadata.labels) && namespaceObject.metadata.labels[?'platform.example.com/env'].orValue('') == 'prod'"
  validations:
    - expression: 'object.spec.?replicas.orValue(1) >= 2'
      messageExpression: "'Production Deployments run at least 2 replicas; ' + object.metadata.name + ' asks for ' + string(object.spec.?replicas.orValue(1)) + '.'"

The second production policy requires a readinessProbe on every container, again with autogen so the Deployment is the thing refused. Both start in Deny, because they were written for a production namespace that did not exist yet: there was nothing to disrupt. That is the general rule for the production layer. Add the control before the first production workload arrives, and it is a requirement; add it afterwards, and it is a migration.

Why two layers rather than one strict set everywhere: a single-replica Deployment in dev is correct and cheap, and a readiness probe on a throwaway branch environment is a nicety. Enforcing production rules in development teaches teams to work around the platform. The baseline is what makes a workload operable at all; the production layer is what makes it safe to depend on.

Delivery#

kubectl apply -f platform/projects.yaml
kubectl apply -f platform/guardrails-app.yaml      # platform-guardrails and platform-exceptions Applications
kubectl apply -f appsets/ci-demo.yaml

Those three commands are the last kubectl apply in the setup. Forty-five seconds later:

NAME                  PROJECT    PATH                      SYNC        HEALTH
ci-demo-dev           non-prod   apps/ci-demo/envs/dev     Synced      Healthy
ci-demo-prod          prod       apps/ci-demo/envs/prod    OutOfSync   Missing
ci-demo-stage         non-prod   apps/ci-demo/envs/stage   Synced      Healthy
platform-exceptions   platform   exceptions                Synced      Healthy
platform-guardrails   platform   policies                  Synced      Healthy

NAME                                 ACTIONS   READY
baseline-require-resources           [Audit]   true
baseline-require-team-label          [Audit]   true
production-require-min-replicas      [Deny]    true
production-require-readiness-probe   [Deny]    true

Policies are cluster resources with a status.conditionStatus.ready field; true means Kyverno compiled the CEL and registered the webhook. A policy with a CEL error shows false with the compiler’s message, and Argo CD shows the Application as Synced regardless, because the object was created. Health for policies is the ready field, not the sync status, and it belongs on a dashboard.

One migration detail was learned the hard way. The first commit used the older kyverno.io/v1 ClusterPolicy kind; Kyverno 1.19 accepted it with a deprecation warning, and the second commit replaced the files with ValidatingPolicy. The project whitelist was changed to the new kind in the same commit, and the sync failed:

one or more synchronization tasks are not valid: resource kyverno.io:ClusterPolicy is not permitted in project platform

Argo CD needed to prune the old kind and the project no longer allowed it to touch that kind. Allowing both kinds until the prune completed, then tightening, fixed it. The order for any resource-kind migration under a project whitelist is: widen, migrate, narrow.

Audit first, and what happens if you do not#

With the baseline in Audit, the reports controller scanned the running workloads:

kubectl get policyreport -A

ci-demo-dev    Deployment  ci-demo   baseline-require-resources    pass
ci-demo-dev    Deployment  ci-demo   baseline-require-team-label   fail  Workloads need a platform.example.com/team label naming the owning team.
ci-demo-stage  Deployment  ci-demo   baseline-require-resources    pass
ci-demo-stage  Deployment  ci-demo   baseline-require-team-label   fail  Workloads need a platform.example.com/team label naming the owning team.

Two findings, one cause: the ci-demo base has no team label. Nothing is blocked; the report says what would be. This is the state to stay in until it is clean, and to prove why, the rollout was then done in the wrong order on purpose. The team-label policy was switched to Deny by a commit, Argo CD synced it, and a routine change to the development overlay (replicas 2 → 3) was merged:

ci-demo-dev   phase=Running   sync=OutOfSync
one or more synchronization tasks completed unsuccessfully, reason: error when patching "…":
admission webhook "vpol.validate.kyverno.svc-fail-finegrained-baseline-require-team-label" denied the request:
Policy baseline-require-team-label failed: Workloads need a platform.example.com/team label naming the owning team.
Retrying attempt #4 at 8:06AM.
still 2 replicas

The running pods were fine; admission does not touch what already exists. The next change was refused, and every environment with the same base was one merge away from the same failure. In a cluster with fifty namespaces that is fifty teams whose next deploy fails at once, for a label they were never told about.

Rollback was a git revert of the enforce commit. Argo CD returned the policy to Audit within one poll, the retrying sync of ci-demo-dev went through on its next attempt (3 replicas), and nothing else had to be touched. The policy’s history, including the mistake, is in the repository.

The right order#

The application team fixes the workload. One line in the base, so every environment inherits it:

metadata:
  name: ci-demo
  labels:
    app: ci-demo
    platform.example.com/team: payments

Development and staging synced automatically; production waited for its manual sync. The reports followed the sync, not the commit:

ci-demo-dev    Deployment  ci-demo   baseline-require-team-label   pass
ci-demo-prod   Deployment  ci-demo   baseline-require-team-label   fail   ← until release-manager synced prod
ci-demo-stage  Deployment  ci-demo   baseline-require-team-label   pass

After the production sync, zero failures across every namespace. Only then did the platform team merge the enforce commit for both baseline policies:

kubectl get validatingpolicies.policies.kyverno.io

NAME                                 ACTIONS
baseline-require-resources           [Deny]
baseline-require-team-label          [Deny]
production-require-min-replicas      [Deny]
production-require-readiness-probe   [Deny]

And the guardrails do what the name says. A Deployment with no label and no resources, applied by hand to ci-demo-dev:

Error from server: admission webhook "vpol.validate.kyverno.svc-fail-finegrained-baseline-require-team-label"
denied the request: Policy baseline-require-team-label failed: Workloads need a platform.example.com/team label
naming the owning team.

The same Deployment with the label and resources: deployment.apps/quick-test created. The same compliant Deployment applied to ci-demo-prod, with two replicas but still no readiness probe:

Error from server: admission webhook "vpol.validate.kyverno.svc-fail-finegrained-production-require-readiness-probe"
denied the request: Policy production-require-readiness-probe failed: Production containers need a
readinessProbe; missing on: app

Compliant in development, refused in production, with the message naming the container. That asymmetry is the two-layer model working, and the message is the contract: a team that has never read a policy file knows what to add.

Exceptions#

Some workloads are legitimately outside a rule. The example here is report-scheduler, a leader-only process in production: two replicas would double-send every report until the team ships leader election. It has the label, the resources and the probe, and replicas: 1:

Error from server: admission webhook "vpol.validate.kyverno.svc-fail-finegrained-production-require-min-replicas"
denied the request: Policy production-require-min-replicas failed: Production Deployments run at least 2
replicas; report-scheduler asks for 1.

The wrong answers are to set the policy to Audit, to exclude the namespace, or to add a bypass annotation the policy checks for. All three widen the hole to every workload that discovers the trick. The right answer is an exception object that names the policy, the resource and the end date, in a directory the platform team merges:

apiVersion: policies.kyverno.io/v1
kind: PolicyException
metadata:
  name: report-scheduler-single-replica
  namespace: platform-exceptions
  annotations:
    platform.example.com/reason: 'report-scheduler is a leader-only process; a second replica double-sends every report until leader election ships'
    platform.example.com/owner: payments-team
    platform.example.com/ticket: PLAT-142
    platform.example.com/review: '2026-12-01: leader election is on the payments roadmap for Q4; re-evaluate then'
spec:
  policyRefs:
    - name: production-require-min-replicas
      kind: ValidatingPolicy
  matchConditions:
    - name: only-this-deployment
      expression: "request.namespace == 'ci-demo-prod' && object.metadata.name == 'report-scheduler'"
  # Kyverno stops honouring the exception after this instant, whatever the review annotation says.
  expiresAt: '2026-12-31T00:00:00Z'

Kyverno was installed with exceptions enabled and restricted to one namespace:

features:
  policyExceptions:
    enabled: true
    namespace: platform-exceptions # exceptions are only honoured from here

An exception object in any other namespace is ignored, so an application team cannot grant itself one; it opens a merge request against exceptions/, and the platform team’s review is the approval. Twenty seconds after the merge, the exception existed in the cluster and the same manifest was accepted: deployment.apps/report-scheduler created. A second single-replica Deployment named other-singleton in the same namespace was refused with the same message as before, because the exception’s matchConditions name one Deployment.

expiresAt was tested by moving it into the past through another commit. Once synced, report-scheduler was refused again on re-creation. The annotations carry the reason, owner, ticket and review date for humans; the field carries the deadline for the controller, and the two should agree. An exception without an expiresAt is a permanent hole with a comment on it.

Policy lifecycle through Git#

Everything above was a commit, and the remaining lifecycle events are too.

Removal. Deleting require-readiness-probe.yaml and its line in the kustomization pruned the policy on the next sync (pruned in the operation’s resource list, twenty seconds after the merge); reverting the commit brought it back. There is no separate “uninstall a policy” procedure and no way to remove one without a commit someone can see.

Rollback is the same revert shown above for the premature enforcement.

Versioning. The repository’s history is the policy’s version history, and the platform.example.com/layer and owner annotations travel with each object into the cluster, where kubectl get validatingpolicy -o yaml shows who owns what. For a platform serving many clusters the natural next step is to tag the repository and point each cluster’s platform-guardrails Application at a tag, promoting a policy release from a development cluster to production clusters the way the GitLab CI template article promotes a pipeline template; that was not exercised here, because there was one cluster.

Self-heal covers the other direction. A kubectl edit that sets a policy back to Audit is reverted by Argo CD within the reconciliation interval, and the edit appears as an out-of-sync event. The break-glass path for a platform engineer is a commit, not a cluster credential.

Who owns what#

The table is enforced, not aspirational: Argo CD projects decide which directory may create which kind in which namespace, Kyverno’s exception namespace decides whose exceptions count, and CODEOWNERS on the repository decides who approves which directory. An application team that reads only its own directory and the error messages still gets everything it needs.

Security implications#

• Kyverno’s webhooks are registered with failurePolicy: Fail. If the admission controller is down, every CREATE and UPDATE of a matched resource in a labelled namespace is refused until it is back. That is the correct setting for guardrails and the reason the admission controller runs with several replicas and a PodDisruptionBudget in production; the lab ran one replica.
• background: enabled is what produces the reports for existing resources. Without it, a policy switched straight to Deny has no findings to read first, and the audit step is skipped by accident.
• The exception namespace is the most sensitive namespace the platform owns. Argo CD’s platform project is the only thing that writes to it, and the repository’s exceptions/ directory should have the strictest CODEOWNERS rule in the repository.
• A matchConditions expression that references namespaceObject depends on the namespace carrying the right labels. If a namespace is created outside the ApplicationSet, it has no labels and no guardrails; a policy on Namespace creation itself (require the platform labels, or refuse namespaces not created by Argo CD’s service account) closes that gap and was not part of this run.

Troubleshooting#

Running this in production#

• Keep the baseline under ten policies and the production layer under ten more; every policy is a rule fifty teams must learn, and the error message is its documentation.
• Publish the enforce date for each new baseline policy at least one sprint ahead, with the report query teams can run (kubectl get policyreport -n <ns>); the reports are the migration tracker.
• Alert on ready: false policies and on the count of PolicyReport failures per policy; the observability article shows where those metrics land. Kyverno exposes kyverno_policy_results_total for exactly this.
• Review exceptions/ on a schedule: everything past its review date gets a decision, not a renewal by inertia.
• Add a guardrail on namespaces themselves so that the label scheme cannot be bypassed by creating a namespace by hand, and pair the platform layer with the security layer from the Secure Kubernetes path: this article’s policies are about operability, that path’s are about the attacker.
• The same audit-then-enforce sequence applies to infrastructure: the Checkov baseline in IaC Security in CI/CD is the Terraform equivalent of a policy in Audit.

References#

• Kyverno: ValidatingPolicy (CEL)
• Kyverno: PolicyException (policies.kyverno.io/v1)
• Kyverno: policy reports
• Kyverno: Helm chart values (features.policyExceptions)
• Argo CD: projects and cluster resource whitelists
• Argo CD: managed namespace metadata
• Kubernetes: CEL in admission (namespaceObject, optional fields)