A Troubleshooting Methodology

The Exam Blueprint's Troubleshooting page teaches the sequence CKA rewards under a two-hour clock — wide view, describe, logs, exec — and that sequence is genuinely correct as far as it goes. It just stops at the container boundary, because exam tasks are pre-scoped to one broken object and real incidents usually aren't. This page generalizes that sequence into a method built to scale: work outward in six rings, starting at the cheapest possible signal — the object's own declared and observed state — and only escalate to a more invasive, more expensive ring once the cheaper one has actually told you something. It also spends the space an exam page can't afford: decoding the five failure signatures every Kubernetes operator eventually recognizes on sight — CrashLoopBackOff, ImagePullBackOff, Pending, Evicted, OOMKilled — precisely enough that seeing the label tells you exactly which hypothesis to test first, not just that something, somewhere, is wrong.

Work outward, ring by ring: the general method

☺ Like you're 10: Start exactly where the thing lives, and only widen the search once that spot comes up empty.

Every Kubernetes incident, from a single stuck Pod to a cluster-wide outage, can be attacked with the same six-ring model, each ring strictly more expensive and more invasive than the one before it. Ring 0 is the object's own declared and observed state — spec versus status — read with a single API call, nothing executed, nothing touched. Ring 1 is events: what the control plane itself already recorded while trying to reconcile the object, also free to read. Ring 2 is logs: what the workload itself said, which requires a container to have actually run at some point. Ring 3 is live process state — exec into the container, or attach a debug container alongside it — genuinely invasive, and the first ring that can itself perturb what you're investigating. Ring 4 widens past the single object entirely: is this Pod's failure actually every Pod on this node failing. Ring 5 widens further still: is a controller, an admission webhook, or the API server itself the real root cause, visible only once you correlate failures across nodes and namespaces.

Reading status precisely: conditions before logs

☺ Like you're 10: The Pod is already telling you exactly where to look — you just have to read the right field before the wrong one.

status.phase — Pending, Running, Succeeded, Failed, Unknown — is the coarsest signal a Pod exposes, and on its own it's nearly useless: a Running Pod can still be completely unable to serve traffic. The real information lives in status.conditions[], four of them, evaluated in a fixed order, each independently True, False, or Unknown with its own reason. The first condition reporting False, reading top to bottom, is the only one worth acting on.

kubectl get pod <pod> -n <ns> -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\n"}{end}'

kubectl get pod <pod> -n <ns> -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.state}{"\t"}{.restartCount}{"\n"}{end}'

Events: the cluster's own incident log — read it before it's gone

☺ Like you're 10: The cluster wrote down what it tried to do the moment it tried it — read that note before you go looking for clues yourself.

Ring 1 is the control plane's own contemporaneous account of what it attempted against this object, and it's worth reading before logs specifically because it's cheaper: no container has to be running, no shell has to work, for an Event to exist. Every Event carries a controlled-vocabulary reason, a type of Normal or Warning, and a human-readable message naming the actual blocker — which is exactly what turns a vague "it's Pending" into an actionable "Insufficient cpu" or "0/3 nodes match node affinity."

kubectl get events -n <ns> --sort-by='.lastTimestamp'
kubectl get events -n <ns> --field-selector involvedObject.name=<pod>,involvedObject.kind=Pod
kubectl get events -n <ns> -o json | jq -r '.items[] | select(.type=="Warning") | "\(.lastTimestamp)\t\(.reason)\t\(.message)"'

kube-apiserver garbage-collects Event objects after --event-ttl, one hour by default — on a real incident, read them promptly, because unlike logs shipped to an aggregator they don't survive to a postmortem on their own. And the row count you see isn't a literal event count: an identical, repeated event mutates a single object's count field and lastTimestamp instead of creating a new object each time, so a Pod restarting every 40 seconds for two hours shows up as one row with count: 180, not 180 rows.

The five failure signatures, decoded

☺ Like you're 10: Five scary-looking words, and every one of them has one exact, boring mechanical meaning once you know where to look.

These five status strings account for the overwhelming majority of "why is my Pod broken" questions, exam or otherwise, and they're routinely used loosely — as if Pending always means one thing, or CrashLoopBackOff is itself a cause rather than a symptom. It isn't. Each row below names what's literally happening at the kubelet or kernel level, and the one check that confirms you have the right one.

Past exec: ephemeral debug containers when there's no shell

☺ Like you're 10: When there's no shell to get into, bring your own toolbox and set it down right next to the thing you're fixing — don't touch the thing itself.

Ring 3 assumes kubectl exec works, and on a minimal or distroless image it often doesn't — there's no shell, no coreutils, sometimes not even a static binary to exec into, and the error is a flat OCI runtime exec failed: exec: "sh": executable file not found in $PATH. Rebuilding the image with debug tools just to diagnose it is exactly backwards — it changes the very thing you're trying to observe. Ephemeral containers solve this without touching the Pod's own spec: kubectl debug attaches a brand-new container into the already-running Pod, and with --target it shares the named container's process namespace, so a full-featured debug image can ps, walk /proc/<pid>/root, and inspect the target's filesystem and network namespace from the outside — with nothing already running there restarted or rebuilt.

kubectl debug -it reporting-worker -n payments \
  --image=busybox:1.36 --target=app -- sh
# inside the ephemeral container: ps shows the target container's real processes too
ls -la /proc/1/root/                          # the target's actual filesystem, read live, no rebuild
cat /proc/1/root/etc/resolv.conf              # e.g. confirm DNS config with no shell in that container at all

The same tool reaches Ring 4 without SSH: kubectl debug node/<node> schedules a privileged Pod onto that exact node with the host filesystem mounted at /host, which is the fastest way to run journalctl or inspect kubelet when you don't have (or don't want) direct node access.

kubectl debug node/<node> -it --image=busybox:1.36
# drops into a privileged pod scheduled on that node, host filesystem mounted at /host
chroot /host systemctl status kubelet
chroot /host journalctl -u kubelet -n 60 --no-pager

# older clusters without ephemeral-container support: copy the Pod instead, swapping one image
kubectl debug reporting-worker -n payments \
  --copy-to=reporting-worker-debug --container=app --image=busybox:1.36 -it -- sh

The two techniques trade off differently: --target attaches alongside a live container without disturbing it, ideal for read-only inspection; --copy-to creates a genuinely new Pod, heavier but necessary when you need to actually change a container's command or arguments rather than just look at it from next door. See kubectl for the rest of the fluency baseline this builds on.

When the ring widens: correlated failures, the node, and the control plane

☺ Like you're 10: If the same symptom shows up on five unrelated things at once, stop blaming any one of them individually.

The hardest instinct to build is recognizing early that a Pod-shaped symptom isn't a Pod-shaped cause. One failing Pod deserves the rings above, in order. A dozen Pods failing identically, spanning unrelated namespaces and owned by unrelated teams, deserves a completely different first move: group by node before you investigate a single one of them.

# Pending or crashing Pods clustering on one node = a node problem, not a dozen app problems
kubectl get pods -A -o wide --field-selector=status.phase!=Running,status.phase!=Succeeded \
  | awk '{print $8}' | sort | uniq -c | sort -rn

# the SAME reason across DIFFERENT nodes and namespaces = ring 5 — a controller, webhook, or the API
kubectl get events -A --field-selector type=Warning -o json \
  | jq -r '.items[].reason' | sort | uniq -c | sort -rn

A node-shaped cluster of failures routes back into the node's own Conditions and its kubelet's journal — the exact mechanics are covered in the blueprint's Troubleshooting page and, for what's actually running there, Control Plane Internals. A cluster-wide cause that isn't the node at all — an admission webhook rejecting broadly, a CNI regression, CoreDNS itself failing — is exactly what Networking & the CNI and Services & Networking cover from the networking angle, and RBAC/admission failures in bulk are RBAC & Admission Control's territory specifically.

This page is deliberately not scoped to the CKA clock — for the exam-paced version of rings 0 through 3, go back to the Exam Blueprint's Troubleshooting domain. For the same discipline generalized past Kubernetes entirely — the identical "cheapest signal first" instinct applied across workloads, networking, and delivery pipelines — Platform Engineering's Triage Playbook is worth reading as a second, complementary framing. And once a ring-5 incident is confirmed and real people need to be paged and coordinated, not just diagnosed, that's SRE's incident management & on-call territory — the technical method on this page and the human process on that one are two different, both-necessary halves of the same incident.