Troubleshooting

Troubleshooting is the single largest domain on the Certified Kubernetes Administrator exam — 30% of everything tested, bigger than Storage and Workloads and Scheduling combined. It's also the domain with the fewest new nouns to memorize, because it isn't really about facts at all: it's about a method. The CNCF's curriculum names five competencies here — troubleshooting clusters and nodes, troubleshooting cluster components, monitoring resource usage, reading container output streams, and troubleshooting services and networking — and every one of them rewards the same instinct: look at the cheapest, least invasive signal first, and only escalate to something more expensive once the cheap signal has told you where to look. This page teaches that sequence explicitly, then walks each of the five competencies with the real commands behind it, because on exam day the clock punishes anyone who guesses before they check.

The domain, and where it sits in the blueprint

☺ Like you're 10: The whole exam is cut into five slices, and this is by far the biggest one — bigger than the other two smallest slices put together.

The CNCF's CKA curriculum publishes five domains that sum to exactly 100%. This page covers the last and largest, Domain 5. The other four are covered on their own pages in this blueprint, and it's worth reading all five before you sit the exam — a stuck Pod is rarely only a troubleshooting question; it's usually a scheduling constraint, a storage mismatch, or a networking gap wearing a troubleshooting costume.

(The curriculum PDF prints Domain 3 as "Servicing and Networking" in some published versions; this course and its blueprint page use the more common phrasing, "Services & Networking" — same domain, same 20%.) The five published competencies under Troubleshooting, verbatim from the curriculum, are: troubleshoot clusters and nodes, troubleshoot cluster components, monitor cluster and application resource usage, manage and evaluate container output streams, and troubleshoot services and networking. That's 27 competencies published across all five domains — this course's CKA study plan maps every one of them onto a page. For a companion treatment of this exact exam from adjacent angles, see Platform Engineering's CKA page and SRE's CKA page — both good second opinions on the same 30%.

The method before the facts: describe, then logs, then exec

☺ Like you're 10: Always check the cheap, fast thing before the slow, invasive thing — most of the time the cheap thing already tells you exactly what's wrong.

Every troubleshooting question on the exam — and in production — can be attacked with the same four-stage sequence, each stage strictly cheaper and less invasive than the one after it. Skip a stage and you either waste time re-deriving what the cheap stage would have told you for free, or you jump straight to guessing.

# the standard opening sequence — memorize this exact order
kubectl get pods -A -o wide | grep -Ev 'Running|Completed'   # stage 1: who's unhappy?
kubectl describe pod <pod> -n <ns>                            # stage 2: Events, at the bottom
kubectl logs <pod> -n <ns> -c <container>                    # stage 3: what did it say?
kubectl logs <pod> -n <ns> -c <container> --previous          # stage 3b: what did the LAST crash say?
kubectl exec -it <pod> -n <ns> -c <container> -- sh          # stage 4: only once 1-3 pointed here

Troubleshooting clusters and nodes

☺ Like you're 10: Before you blame anything running on a machine, check whether the machine itself is healthy — cold, out of breath, or full up.

A node reports its health as a set of Conditions — Ready, MemoryPressure, DiskPressure, PIDPressure, and NetworkUnavailable — each independently True, False, or Unknown. A node stuck at Ready: Unknown almost always means the control plane has stopped hearing from that node's kubelet — check the node's own kubelet service before assuming the node is dead. NotReady with a known reason (disk full, memory pressure) is more useful: the kubelet is talking, and it's telling you exactly what's wrong.

kubectl get nodes -o wide
kubectl describe node <node> | grep -A6 Conditions           # Ready / MemoryPressure / DiskPressure / ...
kubectl describe node <node> | grep -A10 "Allocated resources"

# on the node itself — SSH in when kubectl's view of it stops making sense
systemctl status kubelet
journalctl -u kubelet -n 100 --no-pager -p err                # errors only, last 100 lines
crictl ps -a                                                   # containers the runtime actually has
df -h /var/lib/kubelet   /var/lib/containerd                  # DiskPressure's usual root cause

# taking a node out of rotation safely, and putting it back
kubectl cordon <node>                                          # stop new scheduling, don't evict anything yet
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
kubectl uncordon <node>                                        # forgetting this is the #1 self-inflicted outage

crictl — the CRI-level tool, not docker — is worth being fluent with specifically because it talks to the container runtime directly, one layer below the kubelet, which matters enormously in the next section. See control-plane internals for what the kubelet actually is on that node, and the Fix a Broken Cluster drill for hands-on practice with exactly this category of failure.

Troubleshooting cluster components: when the control plane itself is sick

☺ Like you're 10: If the thing that's broken IS the thing you'd normally ask for help — the API server — you can't ask it what's wrong with itself. You have to go around it.

The API server, etcd, the scheduler, and the controller-manager on a kubeadm-built control plane all run as static Pods — Pod manifests sitting in /etc/kubernetes/manifests/ on the control-plane node, which the kubelet watches directly and runs through the container runtime, with no API server involved in that specific loop at all. That's precisely what makes them recoverable: the mechanism that starts the API server can't itself depend on the API server being up.

# the API server itself won't come up — kubectl is useless here, so don't reach for it
ssh <control-plane-node>
journalctl -u kubelet -n 60 --no-pager                         # kubelet logs WHY it won't run the manifest
crictl ps -a --name kube-apiserver                              # exited? crash-looping? check the exit reason
crictl logs <container-id>                                      # the apiserver's own stderr, one layer down
ls /etc/kubernetes/manifests/                                   # kube-apiserver.yaml, etcd.yaml, kube-scheduler.yaml, ...
cat /etc/kubernetes/manifests/kube-apiserver.yaml                # look for a bad flag, a typo, a wrong cert path

# fix it by editing the file in place — no "kubectl apply" exists for a static pod
vi /etc/kubernetes/manifests/kube-apiserver.yaml
# the kubelet notices the file change on its own and restarts the pod within seconds

Monitoring resource usage

☺ Like you're 10: "It's slow" isn't a diagnosis — check whether it's actually short on CPU or memory before you guess at anything fancier.

kubectl top shows live CPU and memory usage for nodes or Pods, but it depends on the metrics-server add-on being installed — a fresh cluster has no built-in usage metrics, and kubectl top failing with "metrics not available" is itself a diagnosis, not a bug in the command. Once it's running, the real skill is comparing three different numbers that are easy to confuse: what the Pod requested, what it's limited to, and what it's actually using right now.

kubectl top nodes
kubectl top pods -n <ns> --sort-by=memory
kubectl describe node <node> | sed -n '/Allocated resources/,/Events/p'   # requests vs. capacity, per node
kubectl describe pod <pod> -n <ns> | grep -A4 -i "Limits:\|Requests:"

# a container that keeps disappearing is usually one of these two, and describe tells you which
kubectl get pod <pod> -n <ns> -o jsonpath='{.status.containerStatuses[*].lastState}'

Requests drive scheduling — the scheduler only checks requests, never limits, when deciding whether a node has room. Limits drive eviction — a container that exceeds its memory limit gets OOMKilled by the kernel, not gently throttled. That asymmetry between the two numbers is a favorite exam trap, and it's covered in depth from the resource-management angle in scheduling & resource management and from the platform-observability angle in observability on Kubernetes.

Container output streams

☺ Like you're 10: A well-behaved container writes what happened straight to the screen, not to a hidden file — that's the whole reason kubectl logs works at all.

Kubernetes captures whatever a container writes to stdout and stderr and nothing else — a container that logs to an internal file instead is invisible to kubectl logs no matter how much it writes, which is exactly why the twelve-factor convention of logging to standard streams matters operationally, not just stylistically. Reading those streams correctly, especially for a Pod that already crashed, is its own small skill:

kubectl logs <pod> -n <ns>                          # the CURRENT container's stdout/stderr
kubectl logs <pod> -n <ns> --previous                 # the LAST container before it restarted — read this for CrashLoopBackOff
kubectl logs <pod> -n <ns> -c <container>             # multi-container pod: name it, or you get an error
kubectl logs <pod> -n <ns> -c <init-container>         # init containers have logs too — check these before the app container
kubectl logs <pod> -n <ns> --since=10m -f              # follow, last 10 minutes only
kubectl logs -n <ns> -l app=web --all-containers --prefix=true   # every pod matching a label, one merged stream

The single most common miss under exam pressure: reading kubectl logs with no flags on a Pod that's already restarted, seeing nothing useful, and concluding there's nothing to see — when --previous was the entire answer, because you're looking at the brand-new container that hasn't failed yet instead of the one that already did.

Troubleshooting services and networking

☺ Like you're 10: A Service is just a label with a list of addresses attached to it — if the list is empty, nothing you do to the Service itself will fix it, because the problem is upstream, at the labels.

A Service doesn't route traffic itself; it's a stable name and IP that kube-proxy programs into rules pointing at whatever Pods match the Service's selector, tracked live in an EndpointSlice. The single most common networking failure on the exam is a Service with a perfectly correct spec and an empty EndpointSlice — meaning no running Pod's labels actually match the selector, usually from a typo or a label that got removed in a later edit.

kubectl get svc <svc> -n <ns> -o wide
kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<svc>   # empty = selector matches nothing
kubectl get pods -n <ns> --show-labels | grep <expected-label>           # do any pods actually carry it?
kubectl describe svc <svc> -n <ns>                                        # confirm the selector itself

# is it DNS, or is it the Service? a throwaway debug pod answers both in under a minute
kubectl run tmp-shell --rm -it --restart=Never --image=busybox:1.36 -- sh
#   inside:
#   nslookup <svc>.<ns>.svc.cluster.local     # DNS resolves? if not, check CoreDNS next
#   wget -qO- http://<svc>.<ns>:<port>         # resolves but hangs/refuses? it's the endpoints or a NetworkPolicy

kubectl get pods -n kube-system -l k8s-app=kube-dns   # is CoreDNS itself even running?
kubectl get networkpolicy -n <ns>                      # a default-deny policy with no matching allow rule looks identical to a routing bug

Once DNS resolves and endpoints exist but traffic still doesn't arrive, a NetworkPolicy is the next suspect: the moment any policy selects a Pod, that Pod flips from allow-everything to deny-by-default in whichever direction the policy names, and a policy with an Egress section but no explicit rule allowing UDP/TCP port 53 silently breaks every DNS lookup that Pod makes — a failure that looks exactly like a broken application from the outside. This whole layer — Pod-to-Pod connectivity, EndpointSlices, CoreDNS, NetworkPolicy, Ingress, and the newer Gateway API — is the subject of the Services & Networking blueprint page and goes deeper still in networking & the CNI; the Diagnose a Networking Failure drill drills exactly this failure mode hands-on.

Common exam traps in this domain

☺ Like you're 10: The exam likes to hand you a symptom that looks like one thing and is actually caused by something else nearby.

• Reading the current logs, not --previous. A crash-looping container's freshest restart has produced nothing useful yet — the failure is in the logs of the container instance that already died.
• Debugging the app when the node is the problem. NotReady, MemoryPressure, or DiskPressure on the node explains every Pod failure on it at once — check the node's Conditions before you chase an individual Pod.
• Chasing the API server with kubectl when the API server is what's down. If the symptom is "the control plane won't respond at all," the fix is on the node's filesystem and in its journal, not in a kubectl command.
• Confusing requests and limits. Scheduling failures ("Insufficient cpu") are about requests; OOMKills are about limits. A question that names one and expects an answer about the other is testing exactly this distinction.
• Assuming an empty EndpointSlice is a Service bug. The Service's own spec is almost always fine — the mismatch is between the selector and the Pods' actual labels.
• Forgetting egress DNS in a NetworkPolicy. Any policy naming Egress needs an explicit allow rule for UDP and TCP port 53, or every name lookup from that Pod fails.