Kubernetes Health, Resources and Reliability

Overview#

Kubernetes cannot know whether an application is healthy; it can only ask, and it only asks the questions you configure. A pod with no readiness probe receives traffic the moment its container starts. A pod with no memory limit can take the node down with it. A pod with a liveness probe that is too eager gets restarted for being slow. Every setting in this part is a question the kubelet asks on your behalf, and every one was made to fail on purpose on a kind cluster so the answer could be read from the events.

Prerequisites

• The cluster and api Deployment from part 1
• kubectl 1.32 or newer

Three probes, three questions#

Readiness: may this pod receive traffic? Failing it removes the pod from Service endpoints; the container keeps running. Use it for “I am up but not yet able to serve” and for “my dependency is down, stop sending me requests”.

Liveness: is this container stuck? Failing it restarts the container. Use it for deadlocks and hung event loops, the states a process cannot recover from by itself. Do not point it at a dependency: a database outage would then restart every pod, which helps nothing.

Startup: has this container finished starting? While it fails, liveness and readiness are not evaluated. Use it for slow starters so the liveness probe can stay strict without killing the container mid-boot.

A readiness probe that fails#

The api image serves /healthz on port 3000. Probe the wrong port and the pod runs, but is never ready:

apiVersion: v1
kind: Pod
metadata:
  name: not-ready
  labels: { app: not-ready }
spec:
  containers:
    - name: web
      image: ci-demo:1.4.2
      imagePullPolicy: Never
      readinessProbe:
        httpGet: { path: /healthz, port: 8080 } # the app listens on 3000, not 8080
        periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: not-ready
spec:
  selector: { app: not-ready }
  ports: [{ port: 80, targetPort: 3000 }]

kubectl apply -f not-ready.yaml; sleep 30
kubectl get pod not-ready
kubectl get endpointslices -l kubernetes.io/service-name=not-ready \
  -o jsonpath='{range .items[0].endpoints[*]}{.addresses} ready={.conditions.ready}{"\n"}{end}'
kubectl describe pod not-ready | grep 'Readiness probe failed' | tail -1

NAME        READY   STATUS    RESTARTS   AGE
not-ready   0/1     Running   0          31s
["10.244.2.12"] ready=false
Warning  Unhealthy  4s (x7 over 30s)  kubelet  Readiness probe failed: Get "http://10.244.2.12:8080/healthz": dial tcp 10.244.2.12:8080: connect: connection refused

READY 0/1, RESTARTS 0: the container is fine, the probe is not, and Kubernetes did the right thing with that information. The EndpointSlice still lists the address but with ready=false, so kube-proxy sends it nothing. The event says exactly what the kubelet tried and what came back, which is the first place to look when a Service has pods but no traffic reaches them.

A liveness probe that fails#

This container is healthy for twenty seconds, then removes the file the probe checks:

apiVersion: v1
kind: Pod
metadata:
  name: flaky
spec:
  containers:
    - name: app
      image: busybox:1.37
      command: ['sh', '-c', 'touch /tmp/healthy; sleep 20; rm /tmp/healthy; sleep 600']
      livenessProbe:
        exec: { command: ['cat', '/tmp/healthy'] }
        periodSeconds: 5
        failureThreshold: 3

kubectl apply -f flaky.yaml; sleep 45
kubectl describe pod flaky | grep -E 'Liveness probe failed|Killing' | tail -2

Warning  Unhealthy  10s (x3 over 20s)  kubelet  Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
Normal   Killing    10s                kubelet  Container app failed liveness probe, will be restarted

Three failures five seconds apart (failureThreshold: 3 × periodSeconds: 5), then a kill. A minute later:

kubectl get pod flaky
kubectl get pod flaky -o jsonpath='restarts={.status.containerStatuses[0].restartCount} reason={.status.containerStatuses[0].lastState.terminated.reason} exit={.status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}'

NAME    READY   STATUS    RESTARTS      AGE
flaky   1/1     Running   1 (35s ago)   101s
restarts=1 reason=Error exit=137

Exit code 137 is 128 + 9: the container was killed with SIGKILL after the kubelet’s SIGTERM went unanswered by a sleep. The new container is healthy for another twenty seconds, and the cycle repeats. A liveness probe cannot fix an application that keeps breaking; it can only keep restarting it, which is why the restart count is worth alerting on.

A startup probe that protects a slow starter#

This container takes fifteen seconds to become healthy. A liveness probe with failureThreshold: 1 and periodSeconds: 5 would kill it before it finished; the startup probe holds the liveness probe off until the file appears:

apiVersion: v1
kind: Pod
metadata:
  name: slow-start
spec:
  containers:
    - name: app
      image: busybox:1.37
      command: ['sh', '-c', 'sleep 15; touch /tmp/started; sleep 600']
      startupProbe:
        exec: { command: ['cat', '/tmp/started'] }
        periodSeconds: 2
        failureThreshold: 30 # up to 60 s to start
      livenessProbe:
        exec: { command: ['cat', '/tmp/started'] }
        periodSeconds: 5
        failureThreshold: 1 # strict, because startup is handled separately

kubectl apply -f slow-start.yaml; sleep 45
kubectl get pod slow-start; kubectl describe pod slow-start | grep -c 'Startup probe failed'

NAME         READY   STATUS    RESTARTS   AGE
slow-start   1/1     Running   0          46s
1

One startup-probe failure logged, zero restarts. Without the startup probe the same liveness settings restart the container every twenty seconds forever and it never finishes starting. Set the startup budget from the application’s real cold-start time, measured, plus margin.

Requests, limits and what they decide#

requests are what the scheduler reserves: a pod is placed only on a node with that much unallocated CPU and memory. limits are what the kernel enforces at runtime: a container that exceeds its memory limit is killed; one that exceeds its CPU limit is throttled. The two numbers answer different questions and are set for different reasons.

Pending: nothing can hold this pod#

apiVersion: v1
kind: Pod
metadata:
  name: too-big
spec:
  containers:
    - name: app
      image: busybox:1.37
      command: ['sleep', '600']
      resources:
        requests: { cpu: '16' }

kubectl apply -f too-big.yaml; sleep 10
kubectl get pod too-big; kubectl describe pod too-big | grep FailedScheduling | tail -1

NAME      READY   STATUS    RESTARTS   AGE
too-big   0/1     Pending   0          46s
Warning  FailedScheduling  39s (x3 over 46s)  default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 Insufficient cpu. ...

The scheduler explains itself per node: one node is tainted, two do not have 16 CPUs free. Pending with a FailedScheduling event is a capacity or constraint problem, never an application problem; the fix is a smaller request, a bigger node, or a toleration.

OOMKilled: the memory limit is a hard wall#

apiVersion: v1
kind: Pod
metadata:
  name: memory-hog
spec:
  restartPolicy: Never
  containers:
    - name: app
      image: python:3.13-slim
      command: ['python', '-c', 'b = bytearray(300 * 1024 * 1024); import time; time.sleep(60)']
      resources:
        requests: { memory: 64Mi }
        limits: { memory: 64Mi }

kubectl apply -f memory-hog.yaml; sleep 10
kubectl get pod memory-hog
kubectl get pod memory-hog -o jsonpath='{.status.containerStatuses[0].state}{"\n"}'

NAME         READY   STATUS      RESTARTS   AGE
memory-hog   0/1     OOMKilled   0          46s
{"terminated":{"containerID":"containerd://c33b007e…","exitCode":137,"finishedAt":"2026-09-15T21:23:56Z","reason":"OOMKilled","startedAt":"2026-09-15T21:23:56Z"}}

Allocating 300 MiB inside a 64 MiB limit: the kernel’s OOM killer ends the process, the kubelet reports OOMKilled, exit code 137 again. With restartPolicy: Always (the Deployment default) this becomes a restart loop that looks like a crash. The tell is reason: OOMKilled in lastState, and kubectl describe shows it under Last State. The fix is either a higher limit or a smaller working set; a limit set below the real peak usage guarantees this outcome under load.

CPU throttling: the limit is a quota, not a kill#

A CPU limit does not terminate anything. It caps the container’s share of CPU time per scheduling period, and a container that wants more waits. The cgroup counters show it happening:

apiVersion: v1
kind: Pod
metadata:
  name: throttled
spec:
  containers:
    - name: app
      image: busybox:1.37
      command: ['sh', '-c', 'yes > /dev/null'] # one core's worth of work, forever
      resources:
        requests: { cpu: 100m }
        limits: { cpu: 100m }

kubectl apply -f throttled.yaml; sleep 45
kubectl exec throttled -- cat /sys/fs/cgroup/cpu.max
kubectl exec throttled -- grep -E 'nr_periods|nr_throttled|throttled_usec' /sys/fs/cgroup/cpu.stat

10000 100000
nr_periods 451
nr_throttled 450
throttled_usec 18080164

cpu.max is the limit as the kernel sees it: 10 000 µs of CPU per 100 000 µs period, which is 100m. In 451 periods the container was throttled in 450 of them, for 18 seconds in total out of 45. A process that wants a whole core and is allowed a tenth spends 90 % of its time waiting, and a latency-sensitive service in that state looks slow for no visible reason: no errors, no restarts, no events. nr_throttled climbing is the diagnostic; kubectl top shows usage pinned at the limit. Whether to set CPU limits at all is a live debate; the practical guidance is to always set requests (they drive scheduling and fair share) and to set CPU limits only where a runaway process would otherwise starve neighbours.

Termination#

Deleting a pod sends SIGTERM to the container’s main process, waits terminationGracePeriodSeconds (default 30), then sends SIGKILL. A process that handles SIGTERM exits fast; one that ignores it is killed at the deadline. Both were timed:

apiVersion: v1
kind: Pod
metadata:
  name: ignores-term
spec:
  terminationGracePeriodSeconds: 10
  containers:
    - name: app
      image: busybox:1.37
      command: ['sh', '-c', 'trap "" TERM; while true; do sleep 1; done']

kubectl apply -f ignores-term.yaml && kubectl wait --for=condition=ready pod/ignores-term
time kubectl delete pod ignores-term
kubectl run term-ok --image=ci-demo:1.4.2 --image-pull-policy=Never --restart=Never && kubectl wait --for=condition=ready pod/term-ok
time kubectl delete pod term-ok

delete took 12s   (ignores SIGTERM: grace period 10 s, then SIGKILL)
delete took 2s    (server.js closes its listener on SIGTERM and exits)

The sample application has three lines for this (process.on('SIGTERM', () => server.close(() => process.exit(0)))), and they are the difference between a rolling update that drains connections and one that drops them. Two related settings: a preStop hook runs before SIGTERM (a sleep 5 there gives endpoint removal time to propagate before the process stops accepting), and a shell as PID 1 (sh -c '… && node …') may not forward SIGTERM to the real process at all, which is why the sample image uses CMD ["node", "src/server.js"] in exec form.

Security Considerations#

• An exec liveness probe runs a command inside the container every period, with the container’s privileges. Keep it trivial; an HTTP probe is cheaper and leaves less surface.
• Probe endpoints (/healthz) are unauthenticated by design. Do not return internal detail from them, and do not expose them through the ingress.

Troubleshooting#

Cleanup#

kubectl delete pod not-ready flaky slow-start memory-hog too-big throttled --ignore-not-found
kubectl delete svc not-ready

Conclusion#

Every state in this part (0/1, Restarts 1, Pending, OOMKilled, a throttled counter, a 12-second delete) was Kubernetes answering a question exactly as configured. Ask the right questions and the events tell you what is wrong; ask none and the failures are silent. Part 3 uses those events to work through pods that are broken in less obvious ways.

References#

• Kubernetes: liveness, readiness and startup probes
• Kubernetes: resource management for pods and containers
• Kubernetes: pod lifecycle and termination
• Linux cgroup v2: CPU controller (cpu.max, cpu.stat)
• Kubernetes: LimitRange