Logging, Audit and Security Visibility: Who Did What, From Where

Overview#

Parts 1 to 3 put controls in place: scoped identities, network boundaries, secrets that expire. Each control produced a record when it acted. Part 1’s forbidden request, Part 2’s dropped connection and hidden path, Part 3’s refused write to Vault all exist as log lines somewhere, or they do not exist at all, and the difference is whether anyone configured the log. A control you cannot observe is difficult to operate safely, because you cannot tell whether it is working, whether it is being tested by someone, or whether it was quietly removed.

This part sorts the logs a cloud system produces by the question each one answers, then does the work on the one most teams never turn on: the Kubernetes API audit log. The cluster is the kind cluster from Parts 1 and 3, created with audit logging enabled; every event quoted below was produced by the commands in those parts.

Five kinds of record#

They are not interchangeable. An access log shows a request reached the proxy but not what identity it ran as inside the cluster. An audit log shows a role binding was created but not whether the application then behaved differently. Security questions usually need two of them joined on time and identity, which is the argument for sending all of them to one place.

Kubernetes audit logging#

The API server writes an audit event for every request when started with an audit policy and a log path. On kind that is a kubeadm patch and two mounts:

kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
      extraArgs:
        audit-log-path: /var/log/kubernetes/audit/audit.log
        audit-policy-file: /etc/kubernetes/audit/policy.yaml
        audit-log-maxage: '7'
        audit-log-maxbackup: '3'
        audit-log-maxsize: '100'
extraMounts:
  - hostPath: ./audit-policy.yaml
    containerPath: /etc/kubernetes/audit/policy.yaml
    readOnly: true
  - hostPath: ./audit-log
    containerPath: /var/log/kubernetes/audit

The policy decides how much of each request is recorded. Rules are evaluated top to bottom and the first match wins, so the order is the policy:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Do not log the request/response bodies of secrets: the body would contain the secret value.
  - level: Metadata
    resources:
      - group: ''
        resources: ['secrets', 'configmaps', 'serviceaccounts/token']
  # Leases are heartbeats (kubelet, leader election): thousands of updates an hour, no security value.
  - level: None
    resources:
      - group: 'coordination.k8s.io'
        resources: ['leases']
      - group: ''
        resources: ['events']
  # Read-only, high-volume traffic from the control plane itself.
  - level: None
    verbs: ['get', 'list', 'watch']
    users: ['system:kube-controller-manager', 'system:kube-scheduler', 'system:apiserver']
  - level: None
    userGroups: ['system:nodes']
    verbs: ['get', 'list', 'watch']
  - level: None
    nonResourceURLs: ['/healthz*', '/readyz*', '/livez*', '/version', '/openapi*', '/api', '/apis']
  # Everything that changes state is recorded with the request body.
  - level: RequestResponse
    verbs: ['create', 'update', 'patch', 'delete', 'deletecollection']
  - level: Metadata

Four levels exist: None, Metadata (who, what, verb, result), Request (plus the request body) and RequestResponse (plus the response). The first rule is the one to get right: Secrets are logged at Metadata so that the audit log does not become a second copy of every secret in the cluster.

Reading it#

Part 1’s service account tried to list Secrets twice, once with the bound token through kubectl and once with the legacy token through curl. Both are in the log; the fields that matter, side by side:

{
  "verb": "list",
  "requestURI": "/api/v1/namespaces/payments/secrets?limit=500",
  "user": {
    "username": "system:serviceaccount:payments:api",
    "groups": ["system:serviceaccounts", "system:serviceaccounts:payments", "system:authenticated"],
    "extra": {
      "authentication.kubernetes.io/credential-id": ["JTI=d2dd3e2b-0e2a-4815-801d-ba7952f70e19"]
    }
  },
  "sourceIPs": ["192.168.117.1"],
  "userAgent": "kubectl/v1.32.7 (darwin/arm64) kubernetes/158eee9",
  "responseStatus": { "code": 403, "reason": "Forbidden" },
  "requestReceivedTimestamp": "2026-09-16T04:07:54.932321Z"
}

{
  "verb": "list",
  "requestURI": "/api/v1/namespaces/payments/secrets",
  "user": { "username": "system:serviceaccount:payments:api", "groups": ["…"] },
  "sourceIPs": ["192.168.117.1"],
  "userAgent": "curl/8.7.1",
  "responseStatus": { "code": 403, "reason": "Forbidden" },
  "requestReceivedTimestamp": "2026-09-16T04:07:54.955276Z"
}

Same identity, same refusal, twenty milliseconds apart. The first carries a credential-id that is the token’s jti: if that token leaks, every request ever made with that copy of it can be found by one string. The second carries nothing, because a legacy token has no identity of its own. That is what “short-lived, bound credentials” buys at the log level, and it is why Part 1 insisted on them.

The userAgent is worth a filter of its own. A service account whose normal client is the application’s SDK suddenly appearing as curl or kubectl is either an engineer debugging with a stolen token or an attacker doing the same thing; both deserve a question.

Who changed what#

The policy records state changes with their body. Part 3 created a Secret, a ConfigMap and a RoleBinding; this is how they appear (trimmed):

Metadata          create  secrets/db-password              kubernetes-admin  X509SHA256=4dadbd4f…  201
Metadata          create  configmaps/app-config            kubernetes-admin  X509SHA256=4dadbd4f…  201
RequestResponse   create  rolebindings/api-can-read-config kubernetes-admin  X509SHA256=4dadbd4f…  201
Metadata          delete  configmaps/app-config            kubernetes-admin  X509SHA256=4dadbd4f…  200

The Secret and ConfigMap are Metadata (first rule), so the log says they were created and by whom, not what they contained. The RoleBinding is RequestResponse, so its body is in the event:

"requestObject": {
  "kind": "RoleBinding",
  "subjects": [{ "kind": "ServiceAccount", "name": "api", "namespace": "payments" }],
  "roleRef": { "kind": "Role", "name": "nonexistent" }
}

A permission change with the full grant, the granting identity and its credential fingerprint. This is the event class to alert on: RBAC bindings, ClusterRoles, admission webhooks, the audit policy itself. The credential-id for the admin here is the SHA-256 of a client certificate, which is what a kubeconfig on a laptop looks like in the log; a person acting through SSO would show the identity provider’s username and groups instead.

Tuning the volume#

The first version of the policy did not exclude leases. After thirty minutes of an idle two-node cluster:

 1460  update   leases
   72  create   clusterroles
   72  create   serviceaccounts
   70  create   events
bytes by resource: leases 4,032,973 · pods 700,211 · clusterroles 243,811

Four of the five megabytes were kubelet heartbeats and leader-election renewals recorded at RequestResponse, because the catch-all rule for update matched them. Adding the None rule for leases and events and restarting the API server (the policy file is read at start-up; on kind, move the static pod manifest out of /etc/kubernetes/manifests/ and back) brought the following minute down to 54 events, led by ConfigMaps and Pods. An audit log that is 80 % heartbeat is one that gets sampled, truncated or switched off; tune it before shipping it.

The other logs, and where they come from#

Access logs were configured in Part 2’s proxy as JSON with the source address, request, status and upstream. The GET /healthz → 404 line is the only evidence the hidden path was probed; the application never saw the request. At an edge network the same record is richer (client country, TLS version, WAF action) and the WAF’s own decisions appear as security events. For this site those live in the Cloudflare dashboard’s security events view and the Workers observability logs; no export of them is shown here because none was collected for this article, and a dashboard screenshot would prove nothing.

Flow logs are the infrastructure log Part 2’s Checkov scan asked for (CKV2_AWS_11): a record per connection tuple with accept or reject, which is the only place the dropped connection in Part 2’s diagram appears. Configuration on AWS is an aws_flow_log resource pointed at a log group or bucket; it was not applied here. On Kubernetes the equivalent is the CNI’s flow log (Cilium Hubble, Calico flow logs), not the API server.

Secret access logs came from Vault’s audit device in Part 3: operation, path, policies, source address, HMAC of the value. The same record from a cloud provider is the KMS or secret-manager API call in the audit trail, keyed by the identity that made it.

Node and platform logs were read with journalctl -u containerd on the worker in this run; on a managed cluster the equivalent is whatever the provider forwards. They answer “why did the pod not start” rather than “who told it to”, and Debugging Kubernetes Workloads is built on them.

Alerting on the signals#

Logs answer questions after the fact; alerts ask a few of them continuously. Three rules for the signals this path produced, validated with promtool (the metrics they reference come from kube-state-metrics, the API server and the blackbox exporter; the rules were checked for syntax and semantics, not evaluated against a live Prometheus here):

groups:
  - name: platform-security
    rules:
      - alert: APIServerForbiddenSpike
        expr: sum by (namespace) (rate(apiserver_request_total{code="403"}[5m])) > 1
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: 'More than one 403/s from the API server: a workload or person is probing permissions'
      - alert: ContainerRestartingRepeatedly
        expr: increase(kube_pod_container_status_restarts_total[15m]) > 3
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: '{{ $labels.namespace }}/{{ $labels.pod }} restarted {{ $value }} times in 15m'
      - alert: TLSCertificateExpiringSoon
        expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 14
        for: 1h
        labels: { severity: critical }
        annotations:
          summary: 'Certificate for {{ $labels.instance }} expires in {{ $value | printf "%.0f" }} days'

docker run --rm -v "$PWD:/w" -w /w --entrypoint promtool prom/prometheus:v3.14.0 check rules alerts.yaml

Checking alerts.yaml
  SUCCESS: 3 rules found

The first rule is the one that belongs to this part: a steady rate of forbidden requests is what permission probing looks like from the outside, and a single one (Part 1’s) is normal. The for clauses are the difference between an alert and a notification feed; ten minutes of sustained 403s is a pattern, one burst is a deploy.

What is not a metric and needs a log-based alert instead: any create or update on clusterrolebindings, mutatingwebhookconfigurations or the audit policy; any use of the break-glass identity from Part 1; any read of a Vault path by an identity that has never read it before. Those are rare by design, which makes them cheap to alert on and expensive to miss.

Retention and integrity#

• Retention is set by the question you expect to ask. “Was this credential used before we rotated it” needs the credential’s whole lifetime plus the time it took to notice; ninety days is a common floor for audit logs, seven for access logs at full volume with aggregates kept longer.
• Integrity is set where the log lands. Append-only or object-lock storage, a separate account or project, and no delete permission for anything that also runs workloads. The audit-log-maxage: 7 in the kind config is rotation on the node, not retention; it exists so the node does not fill up.
• Time must agree. Every event above carries a UTC timestamp from the API server; the access log’s time_iso8601 and the Vault audit time are from other clocks. Joining them is only possible if the nodes run NTP, which is worth an alert of its own.

Security implications#

• An audit policy that logs Secret bodies at RequestResponse turns the log into the highest-value target in the environment. Metadata for Secrets, bodies for RBAC, is the safe default.
• Logs that stay on the node they were produced on are evidence the attacker can edit. Shipping is part of enabling.
• The credential-id field is only present for bound tokens and certificates. Legacy tokens and static keys produce events that cannot be tied to one copy of the credential, which is one more reason to remove them.
• Alert rules that are not validated silently fail to load. promtool check rules is the smallest possible test and should run on every change to them.

Troubleshooting#

Running this in production#

• Enable API server audit logging on day one with the policy above, ship it off the node, and set retention before the first workload.
• Send access, audit, flow and secret-manager logs to one store with one clock, and keep the identity field names consistent enough to join on.
• Alert on rare privileged events by log and on rates by metric; keep the rule set small enough that every alert has an owner.
• Review the log for the events you expect after every change in Parts 1 to 3: the forbidden request, the dropped connection, the refused write. If the change did not produce the event, one of the two is not working.

References#

• Kubernetes: auditing (policy, levels, backends)
• Kubernetes: audit policy reference (audit.k8s.io/v1)
• kind: configuration (kubeadmConfigPatches, extraMounts)
• Prometheus: promtool check rules and alerting rules
• kube-state-metrics: pod metrics
• AWS: VPC flow logs
• Cloudflare: security events and Workers logs
• Vault: audit devices