Toolbox · Kubernetes
Kubernetes Troubleshooting
Symptom-driven diagnosis for pods that will not start, services that will not route and nodes that misbehave.
On this page
Pod will not start
Symptom (kubectl get pods) | First command | Usual cause |
|---|---|---|
Pending | kubectl describe pod <p> | tail -20 | No node with enough resources; unschedulable taints; PVC not bound |
ImagePullBackOff | kubectl describe pod <p> | grep -A3 Failed | Wrong tag/digest, missing imagePullSecrets, registry auth |
CrashLoopBackOff | kubectl logs <p> --previous | App exits on startup: config, secrets, permissions |
CreateContainerConfigError | kubectl describe pod <p> | Referenced ConfigMap/Secret key missing |
OOMKilled (in describe) | kubectl top pod <p> | Memory limit too low or a leak |
Init:Error | kubectl logs <p> -c <init-container> | Init container failing (migrations, waits) |
| Rejected at admission | kubectl apply … 2>&1 message | Pod Security or Kyverno policy violation |
Service not reachable
kubectl get endpoints <svc> # empty? selector does not match pod labels
kubectl get svc <svc> -o yaml | grep -A3 selector
kubectl run -it --rm net --image=nicolaka/netshoot -- bash # then: curl -v http://<svc>.<ns>.svc:80
kubectl get networkpolicy -n <ns> # default-deny without allow rules?
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50 # DNS issuesNode problems
| Command | Purpose |
|---|---|
kubectl get nodes -o wide | Ready state, versions, IPs |
kubectl describe node <n> | grep -A10 Conditions | MemoryPressure, DiskPressure, PIDPressure |
kubectl describe node <n> | grep -A20 "Allocated resources" | Requests vs capacity |
kubectl cordon <n> / kubectl drain <n> --ignore-daemonsets --delete-emptydir-data | Caution cordon the node and evict eligible workloads; emptyDir data is deleted |
kubectl get pods -A --field-selector spec.nodeName=<n> | Pods on a node |
Control plane and API
kubectl get --raw='/readyz?verbose' # component health
kubectl get componentstatuses 2>/dev/null # deprecated but still informative on some clusters
kubectl get apiservices | grep -v True # broken aggregated APIs (metrics-server)
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations # webhooks that can block everythingStorage
| Command | Purpose |
|---|---|
kubectl get pvc -A | Bound / Pending claims |
kubectl describe pvc <c> | Provisioning errors |
kubectl get storageclass | Default class present? |
See also
- Debugging Kubernetes Workloads, part 3 of the Kubernetes Operations path: the same failures reproduced end to end with real events.