Debugging Kubernetes Workloads: A Repeatable Process on Real Failures

Overview#

A broken workload on Kubernetes almost always tells you what is wrong. The information is spread across the pod’s status, its events, its logs and the objects around it, and the skill is knowing which to read in which order. This part fixes the order, then applies it to four workloads that were broken on purpose on a kind cluster. The outputs are the real ones.

The sequence, for any pod that is not Running and READY:

1. kubectl get pods for the status word: Pending, ContainerCreating, ErrImagePull, CrashLoopBackOff, CreateContainerConfigError, Running with 0/1. Each word points to a different layer.
2. kubectl describe pod <name> for the events at the bottom and the container’s Last State.
3. kubectl logs <name> (and --previous for the container before the current one) for what the application said.
4. kubectl get events --sort-by=.lastTimestamp when the problem is not inside one pod.
5. kubectl exec and kubectl port-forward to test from inside and from your machine.
6. kubectl get deploy,rs,endpointslices for the objects around the pod when the pod itself looks fine.

Prerequisites

• The cluster and api Deployment from part 1
• kubectl 1.32 or newer

The broken workloads#

Four failures in one manifest: a container that exits because a variable is missing, an image tag that does not exist, a pod that references a ConfigMap that was never created, and a Service whose selector matches no pod.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: crasher
spec:
  replicas: 1
  selector:
    matchLabels: { app: crasher }
  template:
    metadata:
      labels: { app: crasher }
    spec:
      containers:
        - name: app
          image: ci-demo:1.4.2
          imagePullPolicy: Never
          command:
            [
              'sh',
              '-c',
              'test -n "$DATABASE_URL" || { echo "fatal: DATABASE_URL is not set" >&2; exit 78; }; node src/server.js',
            ]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wrong-image
spec:
  replicas: 1
  selector:
    matchLabels: { app: wrong-image }
  template:
    metadata:
      labels: { app: wrong-image }
    spec:
      containers:
        - name: app
          image: ci-demo:1.5.0 # does not exist anywhere
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: missing-config
spec:
  replicas: 1
  selector:
    matchLabels: { app: missing-config }
  template:
    metadata:
      labels: { app: missing-config }
    spec:
      containers:
        - name: app
          image: ci-demo:1.4.2
          imagePullPolicy: Never
          envFrom:
            - configMapRef: { name: api-settings } # never created
---
apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  selector: { app: web } # the pods are labelled app=api
  ports: [{ port: 80, targetPort: 3000 }]

kubectl apply -f broken.yaml; sleep 40
kubectl get pods -l 'app in (crasher,wrong-image,missing-config)'

NAME                              READY   STATUS                       RESTARTS      AGE
crasher-559b46ffdd-jpt4l          0/1     Error                        3 (27s ago)   40s
missing-config-67c8dbbb9d-w88zd   0/1     CreateContainerConfigError   0             40s
wrong-image-9b499c5f7-8tp6x       0/1     ErrImagePull                 0             40s

Three different status words, three different layers. Take them in turn.

Error / CrashLoopBackOff: the container starts and exits#

Error with a climbing restart count, alternating with CrashLoopBackOff as the kubelet backs off between attempts, means the image is fine and the container ran. The answer is in the logs:

P=$(kubectl get pod -l app=crasher -o jsonpath='{.items[0].metadata.name}')
kubectl logs $P
kubectl describe pod $P | grep -E 'Exit Code|Reason:|Restart Count|Back-off'

fatal: DATABASE_URL is not set
    Exit Code:    78
    Reason:       Error
    Restart Count:  3
  Warning  BackOff  4s (x4 over 38s)  kubelet  Back-off restarting failed container app in pod crasher-559b46ffdd-jpt4l_default(…)

One line of log and one exit code, and the cause is known. Exit code 78 is EX_CONFIG from sysexits.h; the sample script chose it deliberately, and an application that uses distinct exit codes for distinct failures makes this step faster every time. kubectl logs --previous reads the container before the current one, which matters when the current one has already been replaced and its log is short; in this run, during the back-off window, logs returned the message above and --previous reported the earlier container already gone, so read the current log first and reach for --previous when it is empty.

The fix is configuration, applied to the Deployment rather than the pod:

kubectl set env deployment/crasher DATABASE_URL=postgres://db-0.db:5432/app
kubectl rollout status deployment/crasher
kubectl get pods -l app=crasher

deployment "crasher" successfully rolled out
NAME                       READY   STATUS        RESTARTS   AGE
crasher-559b46ffdd-jpt4l   0/1     Terminating   4          98s
crasher-5c7f7c8f85-kxn9l   1/1     Running       0          2s

A new ReplicaSet (5c7f7c8f85), a new pod, 1/1. Editing the pod would have worked for seconds; the Deployment would have replaced it with the old template.

ErrImagePull / ImagePullBackOff: the image never arrives#

ErrImagePull, then ImagePullBackOff as the kubelet slows its retries. There are no logs because no container ever ran; the events carry the registry’s answer:

P=$(kubectl get pod -l app=wrong-image -o jsonpath='{.items[0].metadata.name}')
kubectl describe pod $P | grep Warning
kubectl get events --field-selector reason=Failed,involvedObject.name=$P -o jsonpath='{.items[0].message}{"\n"}'

Warning  Failed     20s (x2 over 37s)  kubelet  Error: ErrImagePull
Warning  Failed     5s (x2 over 37s)   kubelet  Error: ImagePullBackOff
Failed to pull image "ci-demo:1.5.0": failed to pull and unpack image "docker.io/library/ci-demo:1.5.0": failed to resolve reference "docker.io/library/ci-demo:1.5.0": pull access denied, repository does not exist or may require authorization

Two facts in that message decide the fix. The kubelet expanded ci-demo:1.5.0 to docker.io/library/ci-demo:1.5.0: an unqualified image name means Docker Hub, which is rarely where a private image lives. And “pull access denied, repository does not exist or may require authorization” is the registry’s deliberately ambiguous answer to both a typo and a missing pull secret. Check the tag exists, check the registry name is fully qualified, and check the namespace has an imagePullSecrets entry for private registries. Here the tag simply does not exist; the fix is kubectl set image deployment/wrong-image app=ci-demo:1.4.2.

CreateContainerConfigError: the pod cannot be assembled#

The image is present, nothing has crashed, and the pod still will not start. The kubelet could not build the container’s configuration, and the event says which piece was missing:

P=$(kubectl get pod -l app=missing-config -o jsonpath='{.items[0].metadata.name}')
kubectl describe pod $P | grep Warning | tail -1

Warning  Failed  2s (x5 over 41s)  kubelet  Error: configmap "api-settings" not found

The same status appears for a missing Secret, a Secret key that does not exist (couldn't find key … in Secret), or a volume that references a missing ConfigMap. The fix is to create the object; the pod recovers on the kubelet’s next attempt without a restart of anything:

kubectl create configmap api-settings --from-literal=LOG_LEVEL=info

Everything is Running and nothing works: the Service#

The fourth failure has no broken pod. The api Deployment is healthy, the api Service exists, and a request to it fails:

kubectl run curl --rm -i --restart=Never --image=curlimages/curl:8.14.1 -- curl -sS -m 3 http://api/version

curl: (7) Failed to connect to api port 80 after 10 ms: Could not connect to server

When pods are fine and the Service is not, look at what the Service selected:

kubectl get svc api -o jsonpath='selector={.spec.selector}{"\n"}'
kubectl get endpointslices -l kubernetes.io/service-name=api -o jsonpath='endpoints={.items[0].endpoints}{"\n"}'
kubectl get pods -l app=api --show-labels | head -2

selector={"app":"web"}
endpoints=null
NAME                   READY   STATUS    RESTARTS   AGE     LABELS
api-6c5dc7ffff-d65d7   1/1     Running   0          6m51s   app=api,pod-template-hash=6c5dc7ffff

The selector says app=web; the pods say app=api; the EndpointSlice is empty. With no endpoints, kube-proxy has no destination for the ClusterIP and the connection is refused in milliseconds (on some network setups it times out instead; either way, DNS resolved and nothing answered). Fix the selector and the slice fills immediately:

kubectl patch svc api -p '{"spec":{"selector":{"app":"api"}}}'
kubectl get endpointslices -l kubernetes.io/service-name=api -o jsonpath='{range .items[0].endpoints[*]}{.addresses[0]} {end}{"\n"}'
kubectl run curl --rm -i --restart=Never --image=curlimages/curl:8.14.1 -- curl -s http://api/version

10.244.1.2 10.244.2.3 10.244.1.8
{"version":"1.4.0","commit":"4e205ab"}

Part 4 goes through the rest of the Service path, including the other classic mistake, a targetPort that does not match the container.

Getting inside: exec and port-forward#

Two commands close the gap between “the cluster says it is fine” and “I can see it working”.

exec runs a command in the container, with the container’s view of the network. The sample image has no shell tools beyond BusyBox’s, and that is enough:

kubectl exec deploy/api -- sh -c 'hostname; wget -qO- http://localhost:3000/healthz'

api-6c5dc7ffff-d65d7
ok

port-forward opens a local port to a pod or Service without exposing anything in the cluster, which is the safest way to test a workload that has no Ingress yet:

kubectl port-forward svc/api 18080:80 &
curl -s localhost:18080/version
kill %1

{"version":"1.4.0","commit":"4e205ab"}

If exec works and port-forward works but requests from other pods do not, the problem is between pods: a NetworkPolicy (covered in the Secure Kubernetes path), a Service, or DNS.

Events across the cluster#

When more than one thing is wrong, or the pod you are looking at is a symptom, the event stream sorted by time is the overview:

kubectl get events --sort-by=.lastTimestamp --field-selector type=Warning | tail -5

21s   Warning   Failed    pod/wrong-image-9b499c5f7-8tp6x       Failed to pull image "ci-demo:1.5.0": …
21s   Warning   Failed    pod/wrong-image-9b499c5f7-8tp6x       Error: ErrImagePull
6s    Warning   Failed    pod/wrong-image-9b499c5f7-8tp6x       Error: ImagePullBackOff
5s    Warning   BackOff   pod/crasher-559b46ffdd-jpt4l          Back-off restarting failed container app in pod …
2s    Warning   Failed    pod/missing-config-67c8dbbb9d-w88zd   Error: configmap "api-settings" not found

Events expire (one hour by default), so this is a view of now, not history. For anything older, the answer is whatever observability stack collects them.

The decision table#

Security Considerations#

• kubectl exec is a shell on a production container. RBAC should limit pods/exec to the people who debug, and audit logs should record it; the least-privilege RBAC lab shows how narrow a role can be.
• Logs and events contain whatever the application printed, which can include secrets and personal data. The crasher script prints the name of the missing variable, never its value; hold applications to the same rule.
• An image that fails to pull because of a missing pull secret is a symptom worth chasing: a pod that suddenly cannot pull its image may be pulling from the wrong registry.

Cleanup#

kubectl delete -f broken.yaml --ignore-not-found
kubectl delete configmap api-settings

broken.yaml also contains the (now fixed) api Service, so recreate it for the next parts:

kubectl expose deployment api --port=80 --target-port=3000 --name=api

Conclusion#

Four failures, four layers, one sequence: the status word says where to look, describe says what the kubelet saw, logs says what the application said, and the objects around the pod explain the cases where the pod is fine. Part 4 follows the request path that the last failure broke, from a Service name to a pod.

References#

• Kubernetes: debug pods
• Kubernetes: debug Services
• Kubernetes: container lifecycle states and restart policy
• kubectl reference: logs, exec, port-forward
• sysexits.h exit codes