Drill — Fix a Broken Cluster
One skill, one throwaway cluster, about half an hour: read a static pod manifest closely enough to catch a mistake that's small, plausible, and easy to make once yourself — a single wrong port in the flag that tells kube-apiserver where etcd lives. This drill is completely self-contained; it doesn't touch anything from the six-part capstone and nothing here needs to survive past today. You'll stand up a disposable kind cluster, make one small edit to a manifest you have every reason to trust, watch kubectl go completely silent, and then do the one thing that actually recovers a broken control plane: diagnose it without the tool that's broken. Give yourself 25 to 40 minutes before you open the walkthrough below — this exact skill, reading /etc/kubernetes/manifests/ and the kubelet's own logs instead of reaching for kubectl, is a real, live CKA task type, not an invented exercise.
Picture a night-shift building manager who never once calls the front desk — he just walks the halls with a clipboard, and taped inside one utility closet is a single index card listing exactly which machines should be humming right now and exactly how to start each one. Sneak in and change one number on that card — swap the boiler's pipe number for the one right next to it, close enough that it looks almost right — and the manager doesn't ask questions. He follows the card exactly as written, tries to hook the boiler up to the wrong pipe, watches it fail immediately, and quietly tries again a minute later, and again, forever, never once picking up a phone to explain why. Today you're the one who has to find the wrong number without calling the front desk either — because in this story, the front desk is the boiler.
kubectl goes quiet, and doesn't stop until the proof matches the hunch.You need Docker (or Podman) and kind and kubectl installed locally — nothing else. crictl and a real systemd already live inside every kindest/node container, so journalctl -u kubelet works exactly the way it would on a real kubeadm-built node; see the kind tool guide if any of that is new. Everything else is one disposable cluster you delete when you're done (kind delete cluster --name drill). kind and crictl's flags move between versions — if a command below errors, run it with --help and adapt; diagnosing a CLI that changed under you is its own small rep of the same skill this drill teaches.
How this drill works
☺ Like you're 10: You break one specific thing on purpose, then find it the hard way — the way you'd actually have to on a real node.
Real production clusters mostly built with kubeadm run four control-plane components — kube-apiserver, etcd, kube-scheduler, kube-controller-manager — as static pods: plain Pod manifests sitting in /etc/kubernetes/manifests/ on the node, which the kubelet watches directly and runs through the container runtime, with no API server involved in that loop at all. The Exam Blueprint's Troubleshooting page already covers why that matters and how to read it under exam pressure; this drill exists to make you actually do it once, on a cluster you can break as many times as you want. You'll self-inflict one specific, realistic mistake, watch the exact failure it causes, and fix it using only tools that don't depend on the thing that's down.
Stand up the throwaway cluster
☺ Like you're 10: One tiny pretend computer, just big enough to actually break and rebuild.
A single all-in-one node is all this drill needs — no workers, no CNI, no ingress. Create it, confirm it's healthy, and note the exact name kind gave the container, because you'll be talking to it directly for the rest of this drill:
kind create cluster --name drill
kubectl get nodes
# NAME STATUS ROLES AGE VERSION
# drill-control-plane Ready control-plane 38s v1.31.0
kubectl get pods -n kube-system -l tier=control-plane
# NAME READY STATUS RESTARTS AGE
# etcd-drill-control-plane 1/1 Running 0 52s
# kube-apiserver-drill-control-plane 1/1 Running 0 52s
# kube-controller-manager-drill-control-plane 1/1 Running 0 52s
# kube-scheduler-drill-control-plane 1/1 Running 0 52sEvery row you just saw is a mirror pod — the API-visible reflection of a static pod the kubelet is running from a local file, not from anything in etcd. You can kubectl describe a mirror pod, but you can't fix a broken one by editing it through the API; the source of truth is the file on disk, and that's exactly what you're about to touch.
Open the manifest — and make the one-character mistake
☺ Like you're 10: You glance at one number on a card in a different room and misremember it by the time you write the real one down.
docker exec straight into the control-plane container the same way you would SSH onto a real node, and look at what the kubelet is actually watching:
docker exec drill-control-plane ls /etc/kubernetes/manifests/
# etcd.yaml kube-apiserver.yaml kube-controller-manager.yaml kube-scheduler.yaml
docker exec drill-control-plane grep -- '--listen-client-urls\|--listen-peer-urls' /etc/kubernetes/manifests/etcd.yaml
# - --listen-client-urls=https://127.0.0.1:2379,https://172.18.0.2:2379
# - --listen-peer-urls=https://172.18.0.2:2380
docker exec drill-control-plane grep -- '--etcd-servers' /etc/kubernetes/manifests/kube-apiserver.yaml
# - --etcd-servers=https://127.0.0.1:2379etcd listens on two ports and they are not interchangeable: 2379 is the client port — the etcdserverpb key-value API that kube-apiserver actually talks to — and 2380 is the peer port, used only for Raft traffic between etcd members and speaking a completely different protocol. You just read both numbers thirty seconds apart. Now "tidy up" the apiserver line the way someone genuinely would after just having 2380 fresh in their head:
docker exec drill-control-plane sed -i \
's#--etcd-servers=https://127.0.0.1:2379#--etcd-servers=https://127.0.0.1:2380#' \
/etc/kubernetes/manifests/kube-apiserver.yaml
docker exec drill-control-plane grep -- '--etcd-servers' /etc/kubernetes/manifests/kube-apiserver.yaml
# - --etcd-servers=https://127.0.0.1:2380 # nine became zero. that's the entire bug.That's the whole self-inflicted mistake: one digit, in a line that still parses as perfectly valid YAML, referencing a port that is genuinely open and genuinely serving TLS — just not the service kube-apiserver needs.
The cluster goes quiet
☺ Like you're 10: The front desk you'd normally call for help is the exact thing that just stopped answering the phone.
The kubelet notices the file change on its own within seconds — no kubectl apply, nothing pushed, nothing you did except save a file on the node. Go back to your own terminal, outside the container, and try the thing you'd normally try first:
kubectl get nodes
# E0827 09:14:41.203817 41822 memcache.go:265] couldn't get current server API group list:
# Get "https://127.0.0.1:52341/api?timeout=32s": dial tcp 127.0.0.1:52341: connect: connection refused
# The connection to the server 127.0.0.1:52341 was refused - did you specify the right host or port?That's not a flaky network blip. Try it again, five more times if you want — it won't come back on its own, because there is no process listening on the other end anymore. kubectl only knows how to talk to kube-apiserver, and kube-apiserver is the exact thing you just broke.
Diagnose without kubectl
☺ Like you're 10: The thing that broke is the thing you'd normally ask for help — so you go around it, straight to the machine.
crictl talks directly to the container runtime, not through the API server, which is exactly why it still works. Use it first to confirm what's actually happening to the container:
docker exec drill-control-plane crictl ps -a --name kube-apiserver
# CONTAINER IMAGE STATE NAME ATTEMPT POD ID
# 8f2a1c9e4b03 registry.k8s.io/kube-apiserver... Exited kube-apiserver 14 6e9d0a...
docker exec drill-control-plane sh -c \
'crictl logs $(crictl ps -a --name kube-apiserver -q | head -1) 2>&1 | tail -12'
# W0827 09:14:02.118 ... grpc: addrConn.createTransport failed to connect to {127.0.0.1:2380 ...}
# E0827 09:14:02.204 ... rpc error: code = Unavailable desc = etcdserver: request timed out
# F0827 09:14:32.441 storage_decorator.go:110] Unable to create storage backend, error
# config (&{ /registry [https://127.0.0.1:2380] ...}), err (context deadline exceeded)ATTEMPT: 14 and climbing is your first real clue — this isn't a one-off, it's a loop. The F-prefixed log line is a fatal, not a warning: kube-apiserver cannot construct its storage backend without reaching etcd's key-value API within its dial timeout, so it exits on purpose rather than run half-configured, and the kubelet — faithfully following the manifest — starts it right back up to fail the exact same way. Cross-check with the kubelet's own view of events, which is a second, independent source that should tell the same story:
docker exec drill-control-plane journalctl -u kubelet -n 40 --no-pager | grep -i apiserver
# ... SyncLoop (PLEG): container "kube-apiserver" exited unexpectedly, restarting
# ... probe failed: readiness probe for "kube-apiserver" ... connection refusedNow go back to the two grep commands from the setup step and put them side by side: etcd.yaml serves its client API on 2379; kube-apiserver.yaml is dialing 2380. The container is healthy, the port is real and TLS-serving, and it is still completely the wrong port — which is precisely why the first error you saw (a generic connection/timeout message, not "wrong port") didn't hand you the answer for free.
A static pod with a bad value — like this drill's wrong port — is a manifest the kubelet can parse and run, so the container actually starts, fails, and shows up in crictl ps -a with a climbing restart count you can investigate. A static pod with invalid YAML is a manifest the kubelet can't even parse, so nothing is ever attempted — no container, no entry in crictl ps -a at all, nothing but a parse error buried in journalctl -u kubelet. Same symptom from kubectl's side (silence), two completely different places to look for evidence.
Fix the manifest, and let the kubelet do the rest
☺ Like you're 10: There's no "apply" button for this card — you just cross out the wrong number and put the card back exactly where it was.
There is no kubectl apply here, because there's no API server to apply anything to. You edit the file on disk, directly, the same way you'd fix it over an SSH session on a real node:
docker exec drill-control-plane sed -i \
's#--etcd-servers=https://127.0.0.1:2380#--etcd-servers=https://127.0.0.1:2379#' \
/etc/kubernetes/manifests/kube-apiserver.yaml
docker exec drill-control-plane grep -- '--etcd-servers' /etc/kubernetes/manifests/kube-apiserver.yaml
# - --etcd-servers=https://127.0.0.1:2379Give it a few seconds — the kubelet's own file-watch loop picks up the change on its own, kills the crash-looping container, and starts a fresh one against the corrected manifest. If it ever feels slower than it should on a real node, moving the manifest out of /etc/kubernetes/manifests/ and back in forces the same recreation immediately, which is a genuinely useful trick to know beyond this drill.
docker exec drill-control-plane crictl ps --name kube-apiserver
# CONTAINER IMAGE STATE NAME ATTEMPT POD ID
# a41c7d02f9e8 registry.k8s.io/kube-apiserver... Running kube-apiserver 0 6e9d0a...
kubectl get nodes
# NAME STATUS ROLES AGE VERSION
# drill-control-plane Ready control-plane 6m v1.31.0Done when: kubectl get nodes answers cleanly from your own terminal — not eventually, on the very first try — and ATTEMPT in crictl ps reads 0 against the brand-new container, not a number you watched keep climbing five minutes ago.
The other way a static pod dies: silently
☺ Like you're 10: A card the manager can't even read gets no attempt at all — not even a failed one.
This drill's bug was loud in one specific sense: the container actually started and actually failed, which is exactly what made crictl ps -a useful. The other common way to break a static pod manifest — genuinely invalid YAML, a bad indent, a missing - on a list item — never gets that far at all. The Exam Blueprint's Troubleshooting page covers that failure mode in depth; the short version is that the kubelet can't run what it can't parse, so there's no container, no restart count, nothing in crictl ps -a to even look at — only a parse error sitting in journalctl -u kubelet that nobody goes looking for unless they already suspect the manifest itself.
"The first time I hit the silent version I spent twenty minutes running crictl ps -a over and over, convinced I was fat-fingering the container name, because there was genuinely nothing there to find — not exited, not crash-looping, just absent, as if I'd never edited the file at all. It wasn't until I stopped trusting crictl to have the whole picture and read the kubelet's own journal that I found the parse error, sitting there the whole time. crictl only knows about containers that were actually attempted. A manifest the kubelet rejected before it ever got that far won't show up there, and I don't think I really believed that until I'd watched it happen to me."
Break it the other way. On a fresh line inside kube-scheduler.yaml's command: list, add --v=2 as bare text with no leading -, at the same indentation as its sibling list items, and save. Watch kube-scheduler-drill-control-plane quietly vanish from kubectl get pods -n kube-system with zero events, zero restarts, and zero entries in crictl ps -a — then find the exact line journalctl -u kubelet is complaining about, fix the dash, and confirm the mirror pod comes back. Same platform, same failure family, a completely different set of tools tells you the story.
Where this maps to the CKA
☺ Like you're 10: This isn't a made-up exercise — a real exam hands you almost exactly this scenario, on the clock.
Control-plane component troubleshooting — reading a static pod manifest, correlating it against crictl and the kubelet's journal without a working kubectl — sits inside Troubleshooting, the single largest domain on the CKA curriculum. The Exam Blueprint's Troubleshooting page covers the domain's full scope and the systematic triage order it rewards; the CKA study plan and CKA practice tasks build the rest of the exam around it. This is an independent, unofficial study resource, not affiliated with the CNCF or the Linux Foundation — confirm exam format, pricing, and every logistic detail on the Linux Foundation's own CKA page before you book anything.
Foxy: kubectl just went completely dead — not slow, not a timeout, refused-connection dead. That's new.
Professor Owl: Then don't reach for kubectl again, Foxy — it depends on the exact thing that's down. Go to the node directly.
Gizmo the Gremlin: Or — hear me out — kind delete cluster and stand it right back up. Ninety seconds, done, why read a manifest at all? 🤑
Timmy the Turtle: That trick only exists because this is a throwaway kind cluster, Gizmo. Try deleting a control-plane node in production and see how that conversation with your team goes.
Professor Owl: crictl shows attempt fourteen and climbing on kube-apiserver. Something in its own manifest is wrong, not the node underneath it.
Foxy: Found it. --etcd-servers is pointed at 2380. That's etcd's peer port — I just read 2379 off etcd's own manifest two minutes ago.
Professor Owl: One digit. Fix the file, don't touch anything else, and let the kubelet do what it's already built to do.
drill cluster and confirm it's healthykubectl get nodes shows drill-control-plane Ready, and all four control-plane mirror pods read 1/1 Running.grep output itself, that etcd listens on both 2379 (client) and 2380 (peer), and that kube-apiserver.yaml currently points at 2379.kube-apiserver.yamlgrep for --etcd-servers shows :2380 and the file still saves as valid YAML.kubectl get nodes, run from your own terminal, returns a connection refused error on at least two separate tries.crictl — catch the climbing restart count and the fatal log linecrictl logs line naming a storage-backend or etcd-connection failure, and you can point at the exact ATTEMPT number that proves it's a loop.journalctl -u kubelet and name the exact wrong portcrictl ps --name kube-apiserver shows a fresh container with ATTEMPT: 0, with no kubectl apply ever run.kubectl get nodes answers cleanly on the very first try, and kind delete cluster --name drill has run.1. Why does kube-apiserver exit fatally and repeatedly instead of starting up in some degraded mode once --etcd-servers points at the wrong port? 2. Name etcd's two default ports in a kubeadm-built cluster and what each one is actually for. 3. Why is kubectl useless for diagnosing this specific failure, and what two tools do you reach for instead, and why do they still work when kubectl doesn't? 4. In terms of what shows up in crictl ps -a, what's the practical difference between this drill's bug (a bad value) and a static pod manifest with invalid YAML? 5. How do you actually apply a fix to a static pod manifest, and why does kubectl apply not enter into it at all?
Check your answers
kube-apiservercannot construct its storage backend without successfully reaching etcd's client key-value API within its dial timeout at startup. Rather than run half-configured against a backend it can't confirm, it logs a fatal error and exits on purpose — and the kubelet, faithfully following the manifest exactly as written, restarts it, which fails identically, forever, until the manifest itself changes.- 2379 is the client port — the key-value API that
kube-apiserverand tools likeetcdctlactually talk to. 2380 is the peer port, used only for Raft consensus traffic between etcd cluster members; it speaks a different protocol and answers no client requests at all. kubectlonly knows how to talk tokube-apiserver, andkube-apiserveris the exact component that's down — asking it what's wrong with itself is circular.crictltalks directly to the container runtime andjournalctl -u kubeletreads the kubelet's own systemd logs; neither one routes through the API server, so both keep working no matter how broken the control plane gets.- A bad value (this drill) is still valid YAML, so the kubelet successfully parses and runs it — the container starts, fails, and shows up in
crictl ps -awith a climbingATTEMPTcount you can inspect and read logs from. Invalid YAML can't be parsed at all, so the kubelet never attempts to run it — no container, no restart count, nothing incrictl ps -a; the only evidence is a parse error injournalctl -u kubelet. - You edit the manifest file directly on the node, in place, inside
/etc/kubernetes/manifests/— the kubelet's own file-watch loop notices the change and recreates the pod within seconds, entirely on its own.kubectl applydoesn't enter into it because a static pod was never created through the API in the first place; there is nothing inetcdto apply against, only a file on a specific node's disk.
Fixed it, verified it, cleaned up the cluster? That's the whole drill. For the theory behind why this loop works at all, see Control Plane Internals; for the general six-ring method this single scenario is one worked example of, see A Troubleshooting Methodology. The kubeadm tool guide and the kind tool guide cover the platforms this drill ran on end to end, and Platform Engineering's CKA overview covers whether this exam is worth your time at all. Ready for a different single skill? Try Drill — Debug a Stuck Pod or Drill — Troubleshoot a Failed Upgrade, or step back to the six-part capstone for the continuity version. Chasing every CNCF cert past the CKA? Golden Astronaut picks up the rest of the Kubestronaut ladder where this course leaves off.