Secrets and Configuration Security: Images, Kubernetes, Vault and Rotation

Overview#

A secret has a lifecycle: it is created, stored, delivered to a process, used, rotated and eventually revoked. Detection tools such as Gitleaks cover one step, the moment a secret is written into source, and Secrets Detection with Gitleaks covers that step thoroughly. This part is about the other steps, where most real leaks happen: the build that bakes a token into an image, the cluster that stores the secret in plain text, the environment variable that never notices a rotation, and the access that nobody logged.

Everything below was run: the images were built and pulled apart, the cluster is the kind cluster from Part 1, and Vault ran in its development mode in Docker. Every credential shown is a random value generated for the run or an obvious placeholder; the shapes are real so that the scanners react to them.

Where a build leaks#

Three common Dockerfile patterns, all of which “work”:

FROM node:22.23.2-alpine3.24
WORKDIR /app
ARG NPM_TOKEN
ENV API_KEY=sk_live_X1Ot…                        # a Stripe-shaped key, random for this run
COPY .env /app/.env
RUN echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > /root/.npmrc
RUN echo "(npm ci would run here)"
RUN rm /root/.npmrc                              # "cleaned up"
CMD ["node", "-e", "console.log('up')"]

docker build -f Dockerfile.leaky --build-arg NPM_TOKEN="npm_$(openssl rand -hex 18)" -t leaky:1 .
docker history --no-trunc --format '{{.CreatedBy}}' leaky:1 | grep -E 'ARG|ENV API|COPY \.env|npmrc'

RUN |1 NPM_TOKEN=npm_fb45aa… /bin/sh -c rm /root/.npmrc
RUN |1 NPM_TOKEN=npm_fb45aa… /bin/sh -c echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > /root/.npmrc
COPY .env /app/.env
ENV API_KEY=sk_live_X1Ot…
ARG NPM_TOKEN=npm_fb45aa…

The build argument is recorded in the history of every layer that used it, so --build-arg is not a way to pass a secret; it is a way to publish one. The ENV is in the image configuration, where docker inspect and every registry UI shows it. The .env file is a layer. And the rm:

docker run --rm leaky:1 sh -c 'ls /root/.npmrc'
mkdir extract && docker save leaky:1 | tar -x -C extract
for b in extract/blobs/sha256/*; do
  tar -tf "$b" 2>/dev/null | grep -q 'root/.npmrc$' && tar -xOf "$b" root/.npmrc
done

ls: /root/.npmrc: No such file or directory
//registry.npmjs.org/:_authToken=npm_fb45aa…

The running container cannot see the file; anyone who pulls the image can. A layer is immutable, and a later rm adds a whiteout entry on top rather than editing the layer below. Squashing layers or multi-stage builds hide this particular file, but only a build that never writes the secret to a layer is safe.

What the scanners see:

trivy image --scanners secret --quiet leaky:1

/app/.env  | gitlab-pat           | CRITICAL | GitLab Personal Access Token | line 2
leaky:1    | stripe-secret-token  | CRITICAL | Stripe Secret Key            | (image config)

Trivy found the GitLab token in the copied .env and the Stripe key in the image configuration. It did not find the npm token in the deleted layer or the DATABASE_URL password on line 1 of .env; Gitleaks on the build context found the same two and missed the same password. Both tools match known credential formats and keyword-plus-entropy patterns; a random password after postgres://app: matches neither. Scanners are a backstop for the formats they know, not an inventory of what is in the image.

A build that leaves nothing behind#

# syntax=docker/dockerfile:1
FROM node:22.23.2-alpine3.24
WORKDIR /app
# The token is mounted for one RUN step only; it is never a layer, an ENV or an ARG.
RUN --mount=type=secret,id=npm_token,env=NPM_TOKEN \
    echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > /root/.npmrc \
 && echo "(npm ci would run here)" \
 && rm /root/.npmrc
# Runtime configuration arrives at runtime: no .env in the image, no ENV with a value.
CMD ["node", "-e", "console.log('up')"]

docker build -f Dockerfile.fixed --secret id=npm_token,src=npm_token.txt -t fixed:1 .
docker save fixed:1 | tar -x -C extract-fixed && grep -rl 'npm_' extract-fixed || echo "not present in any layer or config"

not present in any layer or config

--mount=type=secret makes the value available to that one RUN as a file or (with env=) a variable, and nothing about it is recorded. The .npmrc is created and removed in the same step, so no layer ever contains it. Runtime configuration is deliberately absent from the image: it arrives from the platform when the container starts, which is the next section.

What a Kubernetes Secret actually is#

kubectl create secret generic db-password --namespace payments --from-literal=password='not-a-real-password'
kubectl get secret db-password --namespace payments -o jsonpath='{.data.password}'

bm90LWEtcmVhbC1wYXNzd29yZA==

That is base64, an encoding chosen so that binary values survive JSON. It is not protection: base64 -d is the entire attack. The protection a Secret has is RBAC (Part 1’s service account was refused list secrets) and, once configured, encryption of what the API server writes to etcd. On a default cluster it is not configured:

kubectl -n kube-system exec etcd-cloud-lab-control-plane -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/payments/db-password | strings | grep -E 'password'

{"apiVersion":"v1","data":{"password":"cm90YXRlZC1wYXNzd29yZA=="},"kind":"Secret", ...   ← last-applied annotation
password
rotated-password

Two copies of the value in plain text: the stored object and, because the Secret was updated with kubectl apply, the last-applied-configuration annotation with the base64 of the value. Anyone with read access to etcd, to an etcd backup or to the control plane’s disk has every secret in the cluster. Part 5 shows that an etcd snapshot taken for backup contains the same string.

Encryption at rest#

The API server encrypts resources before writing them when given an EncryptionConfiguration. On the kind cluster this meant one file on the control plane node and one flag on the static pod:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ['secrets']
    providers:
      - secretbox:
          keys:
            - name: key1
              secret: <32 random bytes, base64>
      - identity: {} # still able to read what was written before encryption was enabled

- --encryption-provider-config=/etc/kubernetes/enc/enc.yaml

After the API server restarted, the existing Secret was still stored as written (the identity provider is listed second, so it is read but not used for writes). Rewriting every Secret pushes them through the new provider:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -
kubectl -n kube-system exec etcd-cloud-lab-control-plane -- etcdctl ... get /registry/secrets/payments/db-password | head -c 120 | strings
kubectl get secret db-password --namespace payments -o jsonpath='{.data.password}' | base64 -d

/registry/secrets/payments/db-password
k8s:enc:secretbox:v1:key1:
rotated-password

etcd now holds a k8s:enc:secretbox:v1:key1: prefix followed by ciphertext; grep rotated-password on the raw value returns nothing. The previous revision of the key, with the plaintext, stays in etcd’s history until the store is compacted; Part 5 finds it in a backup taken after this point and shows the compaction that removes it. The API still returns the plaintext to callers RBAC allows, which is the point: the control protects the storage layer and backups, not the API. On a managed cluster this is a checkbox with a provider-managed key (KMS); the property to verify is the same, by reading the raw storage if you can, or the provider’s attestation if you cannot.

Delivering the secret to the process#

A pod can receive a Secret as an environment variable or as a file. They behave differently in three ways that matter.

spec:
  serviceAccountName: api
  automountServiceAccountToken: false # this process never calls the API; give it no credential for it
  securityContext:
    fsGroup: 1000 # the node user in the image; the secret volume is chgrp'd to it
  containers:
    - name: app
      image: ci-demo:1.4.2
      env:
        - name: DB_PASSWORD_ENV
          valueFrom: { secretKeyRef: { name: db-password, key: password } }
      volumeMounts:
        - { name: db, mountPath: /run/secrets/db, readOnly: true }
  volumes:
    - name: db
      secret: { secretName: db-password, defaultMode: 0400 }

Visibility. kubectl describe pod shows the reference (<set to the key 'password' in secret 'db-password'>) and not the value. Inside the container the variable is visible to every process, every child process, every crash dump and every kubectl exec; a file is visible to the process that reads it.

Permissions. The first version of this pod had defaultMode: 0400 and no fsGroup, and the application, which runs as node (uid 1000), got Permission denied on the file: Secret volumes are owned by root. With fsGroup: 1000 the file is -r--r----- root node and readable. This is the most common reason “mount it as a file” gets reverted to “just use an env var”.

Rotation. Update the Secret and watch both:

kubectl create secret generic db-password --namespace payments --from-literal=password='rotated-password' \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl exec consumer --namespace payments -- cat /run/secrets/db/password   # after ~10 s
kubectl exec consumer --namespace payments -- sh -c 'echo $DB_PASSWORD_ENV'

rotated-password
not-a-real-password

The mounted file was updated by the kubelet within ten seconds; the environment variable will hold the old value until the pod is replaced. An application that reads the file on each connection can survive a rotation with no restart. One that reads an environment variable at start-up needs a rollout, and the old credential has to stay valid until that rollout completes, which is the window rotation was supposed to close.

A secret manager: TTL, use limits and an audit trail#

A Kubernetes Secret is a static value with access control. A secret manager adds what static values cannot have: a lifetime, a use count, dynamic generation and a log of every read. Vault 1.21 in development mode (in-memory, unsealed, single node; the version to learn on and never to deploy):

docker run -d --name vault-lab --cap-add=IPC_LOCK -e VAULT_DEV_ROOT_TOKEN_ID=root-dev-token -p 127.0.0.1:8200:8200 hashicorp/vault:1.21
vault audit enable file file_path=/tmp/vault-audit.log
vault kv put -mount=secret payments/db password=not-a-real-password
vault policy write payments-read payments-read.hcl   # path "secret/data/payments/*" { capabilities = ["read"] }
vault token create -policy=payments-read -ttl=2m -use-limit=3

creation_ttl        2m
num_uses            3
policies            [default payments-read]
ttl                 1m59s

The token can read one path prefix, three times, for two minutes. Reading works, writing does not, and the fourth call and any call after the TTL return permission denied (observed: the fourth use was refused; after 125 seconds a fresh token was refused with invalid token). That bound is the property Part 1 asked for and a static Secret cannot provide.

The audit device records every request with the identity, the policies, the source address and the response, and it HMACs every secret value so the log itself is not a secret store:

{
  "time": "2026-09-16T04:23:06.702277525Z",
  "operation": "read",
  "path": "secret/data/payments/db",
  "policies": ["default", "payments-read"],
  "token_type": "service",
  "remote_address": "127.0.0.1",
  "response_data": {
    "data": {
      "password": "hmac-sha256:a7348c520dabee6bef8ec635d74d8fc9e536c495ddff21f5ce06300d761c3969"
    }
  }
}

The refused write appears in the same log with "error": "permission denied". That is secret access logging: who read which secret, when, from where, and which attempts failed. It is the input Part 4 needs.

The pattern that makes a secret manager pay for itself is dynamic credentials, where Vault creates a database user with a TTL on each request and drops it on expiry; nothing static exists to leak or rotate. The Vault tool profile sketches it; it was not exercised in this run.

CI variables and configuration files#

CI platforms hold secrets as variables, and three properties decide whether that is acceptable. A masked variable is redacted from job logs; a protected variable is only exposed to jobs on protected branches, which is what stops a merge request from a fork from printing it; and neither stops a job from sending the value anywhere it likes. The DevSecOps pipeline article covers the settings; Part 1 of this path covered the better answer, which is a job that has no static variable to leak because it federates for a short-lived credential instead.

For configuration in general: a .env file is fine on a developer machine and wrong everywhere else, because it is a plain-text secret store with no access log that gets copied into images (above), commit history and backups. In a cluster, non-secret configuration belongs in a ConfigMap and secret configuration in a Secret or a manager, and the difference is only enforced if the audit policy (Part 4) treats them differently.

Security implications#

• Anything in docker history, image config or a layer is public to anyone who can pull the image. Registry read access is the effective permission on every secret ever built in.
• Base64 is not encryption, rm is not deletion, and a scanner that reports zero findings has reported the formats it knows about. Design so that the value is never there, then scan.
• Encryption at rest changes what a backup or disk leaks, not what the API returns. RBAC on secrets is still the control at the API, and get on secrets should be the rarest verb in the cluster.
• Environment variables leak to child processes and diagnostics and do not rotate; files rotate and can be permission-scoped. Choose files unless the application cannot read them.
• A secret manager’s audit log is only useful if it is shipped somewhere the token that reads secrets cannot also delete it.

Troubleshooting#

Running this in production#

• Fail the build on any ARG or ENV whose name looks like a credential, and run the Trivy secret scanner on every image before it is pushed; both are cheap and both catch the common case.
• Enable encryption at rest with a KMS provider before the first real Secret is created, so there is nothing to rewrite later.
• Prefer files to environment variables, fsGroup to 0444, and a manager with dynamic credentials to either.
• Ship the secret manager’s audit log and the API server’s audit log (Part 4) to a store the workload identities cannot write to, and alert on reads of high-value paths from unexpected identities.

References#

• Docker: build secrets (--mount=type=secret)
• Docker: docker history and OCI image layer format (whiteouts)
• Trivy: secret scanning
• Kubernetes: encrypting confidential data at rest
• Kubernetes: Secrets (good practices, mounted files, fsGroup)
• Vault: tokens (TTL, use limits) and audit devices
• GitLab: CI/CD variable security (masked, protected)