Kubernetes Security Checklist for Production Clusters

Overview#

Most Kubernetes checklists are lists of nouns: “RBAC”, “network policies”, “audit logs”. This one is written as verifiable controls: each item states what good looks like and how to check it from a terminal. Work through the layers in order — a hardened pod spec does not help much if system:anonymous can list secrets.

The commands were run against a fresh Kubernetes 1.35 cluster created with kind, so the outputs shown are what a default cluster produces; where a managed provider differs, the text says so.

The layers, from outside in:

1. Cluster and control plane
2. Identity and RBAC
3. Workload configuration (Pod Security)
4. Network
5. Secrets
6. Supply chain and admission
7. Runtime and audit

Outside in: the control plane endpoint and its authentication, then RBAC, then admission control (Pod Security Admission and a policy engine), then the workload configuration itself, then the network policy around the pod, and finally runtime detection and audit logs. Each layer assumes the previous one can be bypassed. Secrets and supply chain controls cut across the layers: secrets arrive from an external manager, and images are scanned and signed before admission verifies them.

1. Control plane. A private API endpoint and no anonymous permissions beyond the defaults. Everything else assumes an attacker who can reach the API.
2. Identity and RBAC. Humans arrive through SSO groups mapped to namespaced roles; service accounts get tokens only when they need the API. A compromised identity is limited to its namespace.
3. Admission. Pod Security Admission and a policy engine reject or fix manifests before they are stored, so a permitted identity still cannot create a privileged pod.
4. Workload spec. Non-root, read-only filesystem, dropped capabilities, seccomp, digests. If admission is bypassed, the pod itself gives an attacker little to work with.
5. Network policy. Default deny, explicit allow. A compromised pod cannot reach what it was not meant to reach.
6. Supply chain. Images are scanned and signed before they exist in a manifest, and admission verifies the signature, so a compromised registry account cannot run arbitrary code in the cluster.
7. Secrets. Injected from an external manager at runtime rather than stored in Git or in environment variables that end up in logs.
8. Runtime and audit. Falco and API audit logs are the layer that tells you the others were bypassed.

Prerequisites#

Prerequisites

• kubectl with cluster-admin on a non-production cluster to test against
• A managed or self-hosted cluster on a supported Kubernetes minor version
• Optional: Kyverno and Falco installed for the admission and runtime sections

Implementation#

1. Cluster and control plane#

• Supported version, patched regularly. Only the three most recent minors receive fixes.
• API server not reachable from the internet without authentication and, ideally, not reachable at all (private endpoint + bastion/VPN).
• Anonymous auth disabled (--anonymous-auth=false) or confirmed to have no permissions.
• etcd encryption at rest enabled for secrets (managed clusters: enable KMS encryption).
• Audit logging enabled and shipped off-cluster.

# Which ClusterRoleBindings include unauthenticated callers?
kubectl get clusterrolebindings -o json | jq -r '
  .items[] | select(.subjects[]? | .name == "system:unauthenticated" or .name == "system:anonymous")
  | "\(.metadata.name) → \(.roleRef.name)"'

# What can an anonymous caller actually do? (impersonation needs cluster-admin)
kubectl get pods -A --as=system:anonymous
kubectl get --raw /version --as=system:anonymous | jq -r .gitVersion

system:public-info-viewer → system:public-info-viewer
Error from server (Forbidden): pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" at the cluster scope
v1.35.0

That is the expected state: the only binding for unauthenticated users is system:public-info-viewer, which exposes /version, /healthz, /readyz and /livez and nothing else. Any other binding in that list is a finding. Note that kubectl auth can-i --list --as=system:anonymous does not work for this check: anonymous users cannot create the SelfSubjectRulesReview the command relies on, so it returns Forbidden regardless of what they are allowed to do.

For encryption at rest on a self-managed control plane, confirm the API server is started with --encryption-provider-config (kind and kubeadm do not set it by default) and that the config lists secrets with a KMS or aescbc provider before identity. On managed clusters this is a provider setting: enable envelope encryption with a customer-managed key.

2. Identity and RBAC#

• No human uses cluster-admin day to day; access goes through SSO groups mapped to namespaced roles.
• No ClusterRoleBinding to system:unauthenticated or system:authenticated beyond the defaults.
• Service accounts do not mount tokens unless the workload needs the API (automountServiceAccountToken: false).
• Wildcards (*) in verbs or resources are treated as findings.

# Who is bound to cluster-admin?
kubectl get clusterrolebindings -o json | jq -r '
  .items[] | select(.roleRef.name=="cluster-admin") |
  "\(.metadata.name): \(.subjects // [] | map(.kind + "/" + .name) | join(", "))"'

# Roles that grant everything
kubectl get clusterroles -o json | jq -r '
  .items[] | select(.rules[]? | (.verbs|index("*")) and (.resources|index("*"))) | .metadata.name'

On a fresh cluster the answers are short, and that is the baseline to compare production against:

cluster-admin: Group/system:masters
kubeadm:cluster-admins: Group/kubeadm:cluster-admins
cluster-admin

Two bindings to cluster-admin, both to groups that only certificates issued by the control plane belong to, and one role with full wildcards. On a managed cluster expect a provider-specific binding as well. Anything beyond that (a user, a service account, a SSO group) is where the review starts.

# Can the default service account in a namespace read secrets? (expect: no)
kubectl auth can-i get secrets --all-namespaces --as=system:serviceaccount:default:default

no

3. Workload configuration#

Enforce the restricted Pod Security Standard per namespace, then write manifests that satisfy it.

apiVersion: v1
kind: Namespace
metadata:
  name: payments
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/warn: restricted

spec:
  template:
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        seccompProfile: { type: RuntimeDefault }
      containers:
        - name: app
          image: registry.example.com/payments/api@sha256:3f1b… # digest, not tag
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ['ALL'] }
          resources:
            requests: { cpu: '250m', memory: '256Mi' }
            limits: { cpu: '1', memory: '512Mi' }
          volumeMounts:
            - { name: tmp, mountPath: /tmp }
      volumes:
        - { name: tmp, emptyDir: {} }

With the namespace labelled restricted, a pod that leaves those fields out is rejected at admission, and the message lists every missing field at once:

kubectl -n payments run bad --image=nginx:1.27 --restart=Never

Error from server (Forbidden): pods "bad" is forbidden: violates PodSecurity "restricted:latest":
allowPrivilegeEscalation != false (container "bad" must set securityContext.allowPrivilegeEscalation=false),
unrestricted capabilities (container "bad" must set securityContext.capabilities.drop=["ALL"]),
runAsNonRoot != true (pod or container "bad" must set securityContext.runAsNonRoot=true),
seccompProfile (pod or container "bad" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

The deployment manifest above passes. If your image writes to paths other than /tmp, add an emptyDir for each (nginx, for example, needs /var/cache/nginx and /var/run), or use an image built to run unprivileged.

Checklist:

• runAsNonRoot, allowPrivilegeEscalation: false, capabilities.drop: [ALL], seccompProfile: RuntimeDefault
• readOnlyRootFilesystem: true with explicit writable emptyDir mounts
• Resource requests and limits on every container
• No hostPID, hostNetwork, hostPath or privileged outside a dedicated, reviewed namespace
• Images referenced by digest

# Find pods that violate the basics
kubectl get pods -A -o json | jq -r '
  .items[] | select(
    (.spec.containers[].securityContext.privileged == true) or
    (.spec.hostNetwork == true) or (.spec.hostPID == true)
  ) | "\(.metadata.namespace)/\(.metadata.name)"'

kube-system/etcd-scc-lab-control-plane
kube-system/kindnet-fnflk
kube-system/kube-apiserver-scc-lab-control-plane
kube-system/kube-controller-manager-scc-lab-control-plane
kube-system/kube-proxy-knww5
kube-system/kube-scheduler-scc-lab-control-plane

Every hit on a fresh cluster is in kube-system and is a component that needs host access (the CNI, kube-proxy, the static control-plane pods). That list is your allow-list; a hit in any other namespace needs a written reason.

4. Network#

• Every application namespace has a default-deny ingress and egress policy; allowed flows are explicit.
• Ingress terminates TLS with certificates from an automated issuer (cert-manager).
• The CNI actually enforces NetworkPolicy (some default CNIs do not).

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: payments
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: payments
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } }
          podSelector: { matchLabels: { k8s-app: kube-dns } }
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }

5. Secrets#

• Secrets are not committed to Git — not even “encrypted for now” unless you use a reviewed scheme (SOPS, Sealed Secrets).
• Secrets are sourced from an external manager (Vault, cloud secret store) via External Secrets Operator or the CSI driver.
• RBAC on secrets is namespaced and minimal; list/watch on secrets cluster-wide is a finding.
• Application logs never print environment variables.

# Who can read secrets cluster-wide?
kubectl auth can-i get secrets --all-namespaces --as=system:serviceaccount:default:default

6. Supply chain and admission#

• Images come only from allowed registries.
• Images are scanned (Trivy) and signed (Cosign); admission verifies signatures.
• latest tags are rejected.

The registry and tag rules are admission policies, not pod fields, so they need a policy engine. The Kyverno article in this series builds the ValidatingPolicy resources for non-root containers, explicit tags and approved registries, plus a generated default-deny NetworkPolicy, and shows the audit-first rollout. For signature verification, Kyverno’s ImageValidatingPolicy checks Cosign or Notary signatures at admission, so an image that was never signed by your pipeline cannot run even if someone can push to your registry.

7. Runtime and audit#

• Falco (or an equivalent) alerts on shells in containers, unexpected outbound connections and privilege escalation.
• API audit policy logs RequestResponse for secrets and RBAC changes at minimum.
• Alerts route to a channel someone actually watches.

Verification#

Run the whole checklist as a script and keep the output with your change records. Two open-source tools automate much of it:

# CIS benchmark checks against nodes and control plane
kube-bench run --targets node,policies

# Cluster-wide misconfiguration scan
trivy k8s --report summary cluster

Security Considerations#

Troubleshooting#

Production Recommendations#

• Adopt controls in audit → warn → enforce order and measure the number of violations dropping before flipping to enforce.
• Keep one “break-glass” cluster-admin identity with MFA and alerting on its use.
• Rebuild and re-scan base images monthly; most cluster CVEs are actually image CVEs.

Conclusion#

Security in Kubernetes is configuration, and configuration drifts. Verifiable controls, enforced at admission and checked on a schedule, are what turn a checklist into a security posture.

References#

• Kubernetes Pod Security Standards
• Kubernetes RBAC good practices
• Kubernetes Pod Security Admission and Network Policies
• Kubernetes: Encrypting Confidential Data at Rest
• Kyverno policies library · Falco rules · kube-bench
• CIS Kubernetes Benchmark