Kubernetes cluster troubleshooting across pods, services, networking, and ingressContents
Kubernetes Troubleshooting Guide
Kubernetes Troubleshooting Guide
Kubernetes troubleshooting becomes easier when you follow a clear process. This guide explains how to debug Pods, Deployments, Services, Ingress, storage, RBAC, image pull errors, CrashLoopBackOff, Pending Pods, events, logs and rollout failures using practical kubectl commands.
Animated mental model: Kubernetes troubleshooting flow
Table of contents
• Quick answer — #quick-answer
• Golden troubleshooting flow — #golden-flow
• Check Pod status — #pod-status
• Pending Pods — #pending
• ImagePullBackOff — #imagepull
• CrashLoopBackOff — #crashloop
• Service issues — #services
• Ingress issues — #ingress
• PVC and volume issues — #storage
• RBAC and permission issues — #rbac
• Deployment rollout issues — #rollouts
• Troubleshooting cheat sheet — #cheatsheet
• Practice lab — #lab
Quick answer
Start with status
Use kubectl get pods -A and kubectl get events before changing configuration.
Use describe
kubectl describe usually shows scheduling, image, volume, probe and event details.
Read logs
kubectl logs helps when the container starts but the application fails.
Check selectors
Service issues are often caused by label and selector mismatches.
1. Golden Kubernetes troubleshooting flow
When something fails in Kubernetes, avoid guessing. Follow this repeatable flow:
1. Check cluster and context.
2. Check resource status.
3. Describe the resource.
4. Read container logs.
5. Check namespace events.
6. Inspect related objects such as Service, Endpoint, PVC, Secret, ConfigMap and Ingress.
7. Fix YAML in Git and apply the change.
8. Verify rollout status.
kubectl config current-context
kubectl get nodes
kubectl get pods -A
kubectl get events -A --sort-by=.metadata.creationTimestamp
2. Check Pod status first
Pod status gives the first clue. Start with a wide output view:
kubectl get pods -n app
kubectl get pods -n app -o wide
kubectl describe pod -n app
Status | Meaning | First command to run
Pending | Pod is accepted but not scheduled or not ready to start. | kubectl describe pod
ImagePullBackOff | Kubernetes cannot pull the container image. | kubectl describe pod
CrashLoopBackOff | Container starts, crashes, and restarts repeatedly. | kubectl logs --previous
Running | Pod is running, but app may still be unhealthy. | kubectl logs
Completed | Job-style Pod finished successfully. | kubectl logs
OOMKilled | Container was killed because it exceeded memory. | kubectl describe pod
3. Troubleshoot Pending Pods
3. Troubleshoot Pending Pods
A Pending Pod usually means the scheduler cannot place it on a node, or Kubernetes is waiting for a dependency such as storage.
Commands
kubectl describe pod -n app
kubectl get nodes
kubectl describe nodes
kubectl get pvc -n app
kubectl get events -n app --sort-by=.metadata.creationTimestamp
Common causes
• Not enough CPU or memory on available nodes.
• Node selector does not match any node.
• Taints exist but the Pod has no toleration.
• PVC is not bound.
• Scheduling constraints are too strict.
Example problem: resource request too high
resources:
requests:
cpu: "8"
memory: "32Gi"
If your cluster nodes are smaller than this request, the Pod can remain Pending. Reduce the request or add suitable nodes.
4. Troubleshoot ImagePullBackOff
ImagePullBackOff means Kubernetes tried to pull the image and failed.
kubectl describe pod -n app
kubectl get secret -n app
kubectl get events -n app
Cause | Fix
Wrong image name | Check repository and image spelling.
Wrong tag | Verify tag exists in the registry.
Private registry authentication missing | Create and attach an imagePullSecret.
Registry unreachable | Check DNS, network and firewall.
imagePullSecret example
kubectl create secret docker-registry regcred \
--docker-server=registry.example.com \
--docker-username=myuser \
--docker-password=mypassword \
--docker-email=myemail@example.com \
-n app
spec:
imagePullSecrets:
- name: regcred
5. Troubleshoot CrashLoopBackOff
CrashLoopBackOff means the container starts and then exits repeatedly. This is usually an application, command, configuration or dependency problem.
kubectl logs -n app
kubectl logs -n app --previous
kubectl describe pod -n app
kubectl exec -it -n app -- sh
Common causes
• Wrong command or entrypoint.
• Missing environment variable.
• Application cannot connect to database.
• ConfigMap or Secret key missing.
• Liveness probe is too strict.
• Application exits immediately.
6. Troubleshoot Kubernetes Service issues
When a Pod is running but traffic does not reach it, check Service selectors and endpoints.
kubectl get svc -n app
kubectl describe svc my-service -n app
kubectl get endpoints -n app
kubectl get pods -n app --show-labels
Common Service issue: selector mismatch
# Service selector
selector:
app: backend
# Pod label
labels:
app: api
In this case, the Service will not find the Pod. Either update the Service selector or Pod labels.
Test service DNS inside the cluster
kubectl run test-shell --rm -it --image=busybox:1.36 -- sh
nslookup my-service.app.svc.cluster.local
wget -qO- my-service:8080
7. Troubleshoot Ingress issues
7. Troubleshoot Ingress issues
Ingress problems usually involve the Ingress controller, host/path rules, TLS, DNS or Service backend.
kubectl get ingress -A
kubectl describe ingress my-ingress -n app
kubectl get svc -n app
kubectl get pods -n ingress-nginx
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller
Issue | Check
DNS not resolving | Check DNS record points to load balancer.
404 from ingress | Check host/path rule and ingress class.
503 from ingress | Check Service endpoints and Pod readiness.
TLS error | Check certificate secret and host name.
8. Troubleshoot PVC and volume issues
Storage issues can prevent Pods from starting or cause applications to fail after startup.
kubectl get pvc -n app
kubectl describe pvc my-pvc -n app
kubectl get pv
kubectl get storageclass
kubectl describe pod -n app
Common storage causes
• No default StorageClass exists.
• Requested storage size is too large.
• Access mode does not match the volume.
• Cloud volume provisioner is failing.
• Pod mounts the wrong path.
9. Troubleshoot RBAC and permission issues
If an app or user gets Forbidden, RBAC is likely blocking the action.
kubectl auth can-i get pods -n app
kubectl auth can-i create deployments -n app --as system:serviceaccount:app:my-sa
kubectl get role,rolebinding -n app
kubectl get clusterrole,clusterrolebinding
Example Forbidden error
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:app:my-sa" cannot list resource "pods" in API group "" in the namespace "app"
The fix is not to make the ServiceAccount cluster-admin. Create a focused Role with only the required verbs and resources.
10. Troubleshoot Deployment rollout issues
Deployment failures often appear during upgrades or CI/CD deployments.
kubectl rollout status deployment/myapp -n app
kubectl rollout history deployment/myapp -n app
kubectl describe deployment myapp -n app
kubectl get rs -n app
kubectl rollout undo deployment/myapp -n app
Common rollout causes
• New image has an application bug.
• Readiness probe never passes.
• Service points to old labels.
• Insufficient resources for new replicas.
• ConfigMap or Secret changed incorrectly.
11. Kubernetes troubleshooting cheat sheet
Task | Command
List all Pods | kubectl get pods -A
Check Pod details | kubectl describe pod POD -n NS
Read logs | kubectl logs POD -n NS
Read previous logs | kubectl logs POD -n NS --previous
Check events | kubectl get events -n NS --sort-by=.metadata.creationTimestamp
Check service endpoints | kubectl get endpoints -n NS
Check rollout | kubectl rollout status deployment/APP -n NS
Check permissions | kubectl auth can-i VERB RESOURCE -n NS
Open shell | kubectl exec -it POD -n NS -- sh
Common beginner mistakes
• Not checking the current context before running commands.
• Troubleshooting in the wrong namespace.
• Looking only at logs and ignoring events.
• Deleting Pods repeatedly without fixing the Deployment.
• Forgetting that Services route by labels.
• Using cluster-admin permissions as a quick fix.
• Ignoring readiness and liveness probe failures.
Best practices
Best practices
• Start with status, describe, logs and events.
• Use namespaces clearly and always specify -n.
• Keep Kubernetes YAML in Git.
• Use labels consistently.
• Use readiness probes before routing traffic.
• Use resource requests and limits.
• Use least-privilege RBAC.
• Record incident fixes in a team runbook.
Mini practice lab
1. Create a namespace named troubleshoot-lab.
2. Deploy an Nginx app with two replicas.
3. Expose it using a ClusterIP Service.
4. Break the Service selector intentionally.
5. Use kubectl get endpoints and kubectl get pods --show-labels to find the issue.
6. Fix the selector and confirm endpoints appear.
kubectl create ns troubleshoot-lab
kubectl create deployment web --image=nginx:1.25 --replicas=2 -n troubleshoot-lab
kubectl expose deployment web --port=80 -n troubleshoot-lab
kubectl get svc,endpoints,pods -n troubleshoot-lab
Conclusion
Kubernetes troubleshooting is not about memorizing one magic command. It is about following a repeatable investigation flow: status, describe, logs, events, related objects and rollout verification. Once you understand how Pods, Services, Ingress, PVCs and RBAC connect, Kubernetes issues become much easier to solve.
No comments yet. Start the discussion.