Skip to content
KubeAtlas
Kubernetes DevOps SRE Debug Cheat Sheet kubectl

Kubernetes Pod Debug Cheat Sheet

Onur Ömer Tunç 7 min read

When a pod won’t come up in production, every second counts. This cheat sheet turns pod autopsies into a repeatable system: 3 starting commands, 6 failure modes with their exact fixes, and a 6-step debug order that cuts incident time in half.

1 Start Here
bash — pod triage
kubectl get pod <pod> -n <ns>
kubectl describe pod <pod> -n <ns>
kubectl logs <pod> -n <ns> -c <container> --previous
Read In This Order
  1. 1Status
  2. 2Events
  3. 3Last State
  4. 4Exit Code
  5. 5Restarts
  6. 6Logs
Use -c <container> for multi-container pods — otherwise you may be reading the wrong container's logs.
2 Pick Your Failure Mode
Pending
Kubernetes accepted the pod, but the scheduler can't place it.
Check This
  • kubectl describe pod <pod> -n <ns>
  • kubectl get nodes
  • kubectl get pvc -n <ns>
Look For
  • FailedScheduling
  • Insufficient cpu / memory
  • taints / tolerations mismatch
  • nodeSelector / affinity mismatch
  • quota exceeded
  • PVC Pending or not bound
  • storage class mismatch
Most Likely Fix
  • Lower requests
  • Add capacity
  • Fix taints / tolerations
  • Update quota
  • Bind storage
CrashLoopBackOff
Container starts, exits, and Kubernetes keeps restarting it.
Check This
  • kubectl logs <pod> -n <ns> -c <container> --previous
  • kubectl describe pod <pod> -n <ns>
Look For
  • Exit Code 1 → app / config failure
  • Exit Code 137 → OOMKilled
  • Exit Code 126 → not executable
  • Exit Code 127 → command not found
  • missing env var
  • bad startup command
  • failed dependency
Most Likely Fix
  • Read previous logs
  • Confirm Last State / Reason
  • Map the exit code
  • Fix app, config, command, or dependency
OOMKilled
The container exceeded its memory limit and was killed.
Check This
  • kubectl describe pod <pod> -n <ns>
  • kubectl top pod <pod> -n <ns>
  • kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.containers[*].resources}'
Look For
  • Reason: OOMKilled
  • Exit Code 137
  • memory limit too low
  • sudden memory spike
  • restarts after traffic bursts
Most Likely Fix
  • Increase limits
  • Set realistic requests
  • Profile memory usage
  • Fix the memory leak
kubectl top requires Metrics Server to be installed.
Init Container Failing
Main container never starts because an init container failed first.
Check This
  • kubectl get pod <pod> -n <ns> -o jsonpath='{.status.initContainerStatuses}'
  • kubectl logs <pod> -n <ns> -c <init-container>
Look For
  • DB migration failure
  • dependency unavailable
  • missing secret
  • permission issue
  • DNS failure
  • bad wait logic
Most Likely Fix
  • Fix dependency
  • Verify secret and DB connectivity
  • Fix migration script
  • Fix DNS or wait condition
ImagePullBackOff
Kubernetes cannot pull the container image.
Check This
  • kubectl describe pod <pod> -n <ns>
  • kubectl describe serviceaccount <sa> -n <ns>
Look For
  • manifest unknown
  • repository does not exist
  • pull access denied
  • authentication required
  • wrong image tag
  • wrong registry URL
  • missing imagePullSecret
Most Likely Fix
  • Fix image name and tag
  • Verify registry auth
  • Add imagePullSecret
  • Fix registry permissions
  • Verify serviceAccount reference
CreateContainerError / ConfigError
Kubernetes can pull the image but cannot create or configure the container.
Check This
  • kubectl describe pod <pod> -n <ns>
  • kubectl get cm -n <ns>
  • kubectl get secret -n <ns>
  • kubectl get pvc -n <ns>
Look For
  • missing ConfigMap
  • missing Secret
  • invalid env reference
  • invalid volume mount
  • permission denied
  • PVC Pending or unbound
  • read-only filesystem
  • mount path conflict
Most Likely Fix
  • Fix config references
  • Verify secrets and PVCs
  • Fix mounts and permissions
  • Investigate container runtime error
3 The Debug Order That Saves 45 Minutes
1
kubectl get pod
identify status
2
kubectl describe pod
read events
3
kubectl logs --previous
inspect last crash
4
kubectl get pod -o yaml
inspect env, args, volumes, resources
5
kubectl top pod / node
confirm CPU/memory pressure
6
kubectl debug / netshoot
test DNS, network, dependencies
Network Debug Options
bash — network debug
# Attach debug container to existing pod
kubectl debug -it <pod> -n <ns> \
  --image=nicolaka/netshoot --target=<container>
# Spin up a one-shot network debug pod
kubectl run netshoot -it --rm \
  --image=nicolaka/netshoot \
  --restart=Never -- sh

Common Mistakes

Misreading exit codes: Exit Code 137 is not always OOMKilled — it can also mean SIGKILL (signal 9). Always check the Reason field in kubectl describe for the authoritative answer.

Skipping --previous on logs: After a pod restarts, the current log is empty. For crash-time logs you must always add --previous.

Fixating on one container: When an init container fails, the main container never starts. Check initContainerStatuses in kubectl get pod -o yaml before assuming the main container is at fault.

Confusing requests and limits: The scheduler uses requests to place pods; the OOM killer uses limits. They are independent — inspect both separately when debugging resource-related failures.

Tags Kubernetes DevOps SRE Debug Cheat Sheet kubectl
All posts