Drill — Troubleshoot a Failed Upgrade
Reading the correct worker-upgrade order in the kubeadm tool guide or CKA Domain 1, and actually catching yourself about to run it backwards when a teammate hands you a runbook that swapped two steps, are two different skills. This drill is the second one. It's completely self-contained — no dependency on the six-part capstone, about 25 to 40 minutes, and nothing here needs to survive past a kind delete cluster at the end. You'll stand up a small two-worker cluster, deploy a tiny app with a topology spread constraint and a PodDisruptionBudget so the scheduler's decisions are actually visible instead of theoretical, and then deliberately run one worker's upgrade steps in the exact wrong order — the disruptive kubelet bounce before the drain, not after — using the same runbook a well-meaning teammate swears still checks out. You'll watch the scheduler place new work on a node nobody told it was busy, hit the two real errors kubectl drain throws at a node that isn't ready for it, and catch the forgotten uncordon this course's own troubleshooting page already calls the "#1 self-inflicted outage."
Picture a hotel room that needs its carpet replaced. Done right: hang the "Being Cleaned — Do Not Book" sign on the door first, walk the current guest to a different room, then bring the crew in, and only take the sign down once the new carpet's dry and the room's actually ready again. Now picture the wrong order someone actually tried: the crew rips up the old carpet first, the sign never went up, so the front desk — which has no idea anything's happening in there — checks a brand-new guest straight into that room, suitcase and all, onto bare concrete. Someone notices, hangs the sign late, and now has to walk an already-checked-in guest back out on top of everything else. And even after the new carpet's finished, if nobody ever takes the sign back down, the front desk just keeps skipping that perfectly good room forever, cramming every other guest into whatever's left. Three separate mistakes, one root cause: the sign didn't go up before the work started, and might never come back down after.
How this drill works
☺ Like you're 10: Break one worker's upgrade sequence on purpose, in the wrong order a real teammate actually proposed, then fix it exactly the way you'd have to on a real node.
This is a single-skill, hands-on drill, not a reading exercise — every command below runs against a real, disposable two-worker kind cluster. The kubeadm tool guide and CKA Domain 1 already state the correct worker-upgrade order in full: upgrade the kubeadm package, drain the node, run kubeadm upgrade node, upgrade kubelet and kubectl, restart the kubelet, uncordon. This drill exists because reading that order under zero pressure and catching yourself about to run it backwards under a real one are not the same skill. You'll run the disruptive step — the kubelet bounce a real worker upgrade ends with — before the drain that's supposed to come first, watch exactly what that costs, fix it the hard way, and then catch the second half of the same class of mistake: forgetting the uncordon at the end.
You need Docker, kind, and kubectl installed locally — nothing else. Everything below is one disposable cluster, deleted at the end (kind delete cluster --name upgrade-drill); nothing here depends on the six-part capstone or costs money. One honesty note up front: kind pins every node in a cluster to a single Kubernetes version, so there's no real minor-version binary to swap. This drill isn't about the version number — it's about the sequencing mistake around it, and the exact disruptive moment a real worker upgrade ends with, restarting the kubelet service to pick up new binaries and config, is something you can trigger for real against a kind node's real systemd. kubectl drain's exact error text and flags also drift a little between versions — treat what's below as the shape of the answer, not a magic incantation.
Stand up the cluster and a small, honestly-configured app
☺ Like you're 10: Two small rooms instead of one, and a tiny app spread evenly across both, so you can actually see which room new work lands in.
One control-plane node and two workers — enough to make "which node did that Pod land on" a real, visible question instead of a rhetorical one:
# kind-config.yaml — 1 control plane, 2 workers, nothing fancy kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 name: upgrade-drill nodes: - role: control-plane - role: worker - role: worker
kind create cluster --config kind-config.yaml kubectl get nodes # NAME STATUS ROLES AGE VERSION # upgrade-drill-control-plane Ready control-plane 52s v1.31.0 # upgrade-drill-worker Ready <none> 37s v1.31.0 # upgrade-drill-worker2 Ready <none> 37s v1.31.0
Deploy beacon — four replicas, a topologySpreadConstraint so it actually lands evenly across both workers instead of wherever the scheduler feels like, a small emptyDir scratch volume (a real, common pattern for local nginx cache dirs, and important later), and a PodDisruptionBudget so there's a real budget for kubectl drain to respect:
# beacon.yaml
apiVersion: v1
kind: Namespace
metadata: { name: beacon }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: beacon, namespace: beacon }
spec:
replicas: 4
selector: { matchLabels: { app: beacon } }
template:
metadata: { labels: { app: beacon } }
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway # not DoNotSchedule — see the note below
labelSelector: { matchLabels: { app: beacon } }
containers:
- name: beacon
image: nginx:1.27-alpine
ports: [{ containerPort: 80 }]
readinessProbe:
httpGet: { path: /, port: 80 }
periodSeconds: 3
volumeMounts:
- { name: scratch, mountPath: /var/cache/nginx-scratch }
volumes:
- name: scratch
emptyDir: {}
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: beacon-pdb, namespace: beacon }
spec:
minAvailable: 3
selector: { matchLabels: { app: beacon } }ScheduleAnyway rather than the stricter DoNotSchedule is deliberate, not a default left on autopilot: with only two candidate nodes, ScheduleAnyway's scoring is more than enough to land a reliable even split while nothing else is competing for the scheduler's attention — and it means beacon doesn't deadlock on itself later in this drill, once one of those two nodes gets cordoned. Scheduling & Resource Management covers exactly this DoNotSchedule vs. ScheduleAnyway trade-off in full if it's new.
kubectl apply -f beacon.yaml kubectl -n beacon rollout status deploy/beacon kubectl get pods -n beacon -o wide # NAME READY STATUS NODE # beacon-7c9f8d6b5-b2h8k 1/1 Running upgrade-drill-worker # beacon-7c9f8d6b5-m8v2t 1/1 Running upgrade-drill-worker # beacon-7c9f8d6b5-x2k9p 1/1 Running upgrade-drill-worker2 # beacon-7c9f8d6b5-j4nq7 1/1 Running upgrade-drill-worker2 kubectl get pdb -n beacon # NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE # beacon-pdb 3 N/A 1 22s
Two on worker, two on worker2, and exactly one disruption currently allowed — hold onto both numbers, they matter in a few scenes.
The runbook someone actually followed
☺ Like you're 10: Before you break anything, read the instructions you're about to follow closely enough to catch what's backwards in them.
Nobody wrote this mistake down labeled as a mistake — it arrived the way most real ones do, as an inherited doc somebody found and trusted a little too fast:
#cluster-ops — Slack, 09:12
@gizmo: dug up the old node-upgrade runbook, still basically checks out — run
it top to bottom on upgrade-drill-worker2:
1. kubeadm upgrade node
2. restart kubelet
3. kubectl drain <node> --ignore-daemonsets
4. kubectl uncordon <node>
saves you a step vs. whatever the tools page says 🤑Put that next to the kubeadm tool guide's own command family and CKA Domain 1's lifecycle section, and the problem is visible before a single command runs: steps 2 and 3 are swapped. The correct doc drains first, specifically because kubeadm upgrade node and the kubelet restart that follows it are the disruptive part — the part a drain is supposed to happen before, not clean up after. Recreate it exactly as written, wrong order and all, so the next scene's evidence is real instead of hypothetical.
Recreate the mistake: the bounce before the drain
☺ Like you're 10: Start the work on the room before you've put the "do not book" sign on the door, and see what the front desk does next.
A real kubeadm upgrade node call mostly rewrites the local kubelet config and the CNI ConfigMap — it doesn't restart anything on its own. The genuinely disruptive moment is two steps later: the kubelet binary gets upgraded, and the service gets restarted to pick it up. That's the exact moment this drill is built around, and it's a completely real command against a completely real systemd unit inside the kind node container — no version bump needed to make the danger real:
kubectl get nodes # upgrade-drill-worker2 Ready <none> 4m v1.31.0 # — no SchedulingDisabled anywhere. Nothing has told the scheduler this node is busy. docker exec upgrade-drill-worker2 systemctl restart kubelet
Immediately, before anything settles, simulate the one thing that never waits for a convenient moment — a routine capacity bump landing mid-upgrade:
kubectl scale deployment/beacon -n beacon --replicas=6 kubectl get pods -n beacon -o wide # NAME READY STATUS NODE # beacon-7c9f8d6b5-b2h8k 1/1 Running upgrade-drill-worker # beacon-7c9f8d6b5-m8v2t 1/1 Running upgrade-drill-worker # beacon-7c9f8d6b5-x2k9p 1/1 Running upgrade-drill-worker2 # beacon-7c9f8d6b5-j4nq7 1/1 Running upgrade-drill-worker2 # beacon-7c9f8d6b5-q9d3f 1/1 Running upgrade-drill-worker # beacon-7c9f8d6b5-r5t1w 0/1 ContainerCreating upgrade-drill-worker2 ← landed here, mid-bounce
Two new replicas, one on each node, because ScheduleAnyway is still pulling toward an even split — and one of them landed squarely on the node whose kubelet just bounced. In kind, that bounce is over in a few seconds, so r5t1w settles into Running almost immediately and this looks almost harmless. A real worker upgrade's package install, config reload, and CNI re-init takes minutes, not seconds — the exact same mechanism, held open far longer, on a node that genuinely wasn't ready to receive it.
Decision: what single command, run before the systemctl restart above, would have stopped this from happening at all?
Check the response
A cordon — and specifically the one kubectl drain performs as its first act, before it evicts anything. Nothing about restarting a service marks a node as off-limits to the scheduler; Ready and "schedulable" are two completely independent signals, and the scheduler only reads the second one. The runbook's step 3, kubectl drain, is the step that actually flips that signal — running it after the bounce instead of before is why the scheduler had no reason not to place r5t1w exactly where it did.
Clean it up — and hit the two errors an unprepared node throws
☺ Like you're 10: Try to hang the sign late, and the door tells you exactly what else you forgot to check first.
Reach for the drain now, the way the runbook eventually gets to it:
kubectl drain upgrade-drill-worker2 # node/upgrade-drill-worker2 cordoned # error: unable to drain node "upgrade-drill-worker2" due to error:cannot delete # Pods declare no controller (kube-system/kindnet-8f3xr, kube-system/kube-proxy-4jvqz), # cannot delete Pods with local storage (beacon/beacon-7c9f8d6b5-r5t1w), ...continuing command... # There are pending nodes to be drained: # upgrade-drill-worker2 # cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore # DaemonSet-managed Pods): kube-system/kindnet-8f3xr, kube-system/kube-proxy-4jvqz # cannot delete Pods with local storage (use --delete-emptydir-data to override): # beacon/beacon-7c9f8d6b5-r5t1w
Two real, unrelated refusals in one shot. kindnet and kube-proxy run as DaemonSets on every node — a bare kubectl drain refuses to touch DaemonSet-managed Pods on purpose, because deleting one just makes it come right back on the same node the moment it's gone, so there's nothing eviction actually accomplishes there. And beacon's emptyDir scratch volume trips the second refusal: evicting that Pod destroys data living only on this node's local disk, and kubectl drain won't do that silently either. Notice the node is already cordoned even though the command as a whole errored — cordon happens first and unconditionally; only the eviction half failed.
kubectl drain upgrade-drill-worker2 --ignore-daemonsets --delete-emptydir-data # node/upgrade-drill-worker2 already cordoned # evicting pod beacon/beacon-7c9f8d6b5-x2k9p # evicting pod beacon/beacon-7c9f8d6b5-j4nq7 # evicting pod beacon/beacon-7c9f8d6b5-r5t1w # pod/beacon-7c9f8d6b5-x2k9p evicted # pod/beacon-7c9f8d6b5-j4nq7 evicted # pod/beacon-7c9f8d6b5-r5t1w evicted # node/upgrade-drill-worker2 drained
All three succeed cleanly — six total replicas, evicting down to three still satisfies beacon-pdb's minAvailable: 3 exactly, at the boundary, with nothing to wait on.
This is the actual mechanical reason order matters here, not just politeness: kubectl drain removes Pods through the Eviction API, which checks every affected PodDisruptionBudget before it's allowed to proceed. A raw systemctl restart kubelet never calls that API at all — nothing about restarting a service asks etcd whether a PDB would be violated, because a service bounce isn't a Pod deletion in the first place. The scene above wasn't dangerous because a Pod scheduled somewhere inconvenient; it was dangerous because the step that ran first was one PodDisruptionBudget has no visibility into whatsoever.
The other missing step: the forgotten uncordon
☺ Like you're 10: A clean room the front desk never un-blocked is a room that quietly disappears from every future guest, forever.
The drain finished. The node is now empty and correctly marked SchedulingDisabled — but suppose the runbook's step 4 never runs, or runs and gets missed in a busy on-call handoff. Watch what that costs the very next time capacity is needed, without touching upgrade-drill-worker2 at all:
kubectl get nodes # NAME STATUS ROLES # upgrade-drill-control-plane Ready control-plane # upgrade-drill-worker Ready <none> # upgrade-drill-worker2 Ready,SchedulingDisabled <none> kubectl scale deployment/beacon -n beacon --replicas=8 kubectl get pods -n beacon -o wide # every one of the 8 replicas lands on upgrade-drill-worker — none on worker2, # because SchedulingDisabled quietly rules it out of every placement decision
Decision: is there anything actually broken here from the cluster's point of view — any error, any alert — or is this exactly the kind of thing that goes unnoticed for days?
Check the response
Nothing is broken, which is precisely why this is dangerous. kubectl get pods shows eight healthy Running replicas. No alert fires, because nothing failed — the scheduler is doing exactly what a cordoned node tells it to do. What's missing is capacity: half the cluster is silently unavailable to every future scheduling decision, and the only visible trace is one easy-to-miss word, SchedulingDisabled, in a column of kubectl get nodes nobody's watching until something else runs out of room first.
Fix it, and notice what fixing it does and doesn't do on its own:
kubectl uncordon upgrade-drill-worker2 kubectl get nodes # upgrade-drill-worker2 Ready <none> — SchedulingDisabled cleared
The eight already-running replicas stay exactly where they are — uncordon only reopens the node to future placement decisions, it doesn't rebalance anything that already landed elsewhere while it was closed. This course's own troubleshooting page puts the general version of this lesson in one line worth remembering past this drill: "forgetting this is the #1 self-inflicted outage."
Where this maps to the CKA
☺ Like you're 10: This isn't an invented exercise — Troubleshooting is the single biggest domain on a real exam, worth more of your score than any other.
Node-level failure diagnosis and safe drain/upgrade/uncordon sequencing sits squarely inside Domain 5, Troubleshooting — the largest domain on the CKA curriculum at 30% — and the correct upgrade order itself is one of Domain 1's eight named competencies. This is an independent, unofficial study resource, not affiliated with the CNCF or the Linux Foundation — confirm exam format, pricing, and every other logistic detail on the Linux Foundation's own CKA page before you book anything. The CKA study plan and CKA practice tasks build the rest of the exam around exactly this kind of sequencing discipline.
upgrade-drill clusterkubectl get nodes shows all three nodes Ready.kubectl get pods -n beacon -o wide shows 2 Pods per worker, and kubectl get pdb -n beacon shows ALLOWED DISRUPTIONS: 1.systemctl restart kubelet has run inside upgrade-drill-worker2, with no cordon or drain run first.kubectl get pods -n beacon -o wide shows a brand-new replica scheduled onto upgrade-drill-worker2 right after the bounce.kubectl drain refusal errors on the unprepared nodekubectl drain upgrade-drill-worker2 has shown you both the DaemonSet error and the local-storage error in one pass.node/upgrade-drill-worker2 drained prints, and beacon still has 3 Pods Running the whole time.upgrade-drill-worker alone, with worker2 still SchedulingDisabled.kubectl get nodes shows no SchedulingDisabled anywhere, and kind delete cluster --name upgrade-drill has run.Sol the Sloth: The runbook says restart kubelet, then drain. I read that three times because it felt backwards. It is backwards.
Gizmo the Gremlin: It saves a step! Drain first and you're just sitting there waiting for Pods to move. Restart first and you're already working while that happens. 🤑
Timmy the Turtle: You're working on a node the scheduler still thinks is wide open, Gizmo. That's not saving a step. That's skipping the one step whose entire job is telling the scheduler to leave it alone.
Foxy: So the two drain errors weren't the drain being difficult with us — they were the drain telling us exactly what the raw kubelet restart skipped past for free.
Sol the Sloth: Slow and in order beats fast and out of order. I will say that as many times as it takes.
Timmy the Turtle: And uncordon isn't optional cleanup at the end. Skip it and that whole node just quietly stops existing, as far as the scheduler's concerned.
1. Why does restarting a node's kubelet directly, without draining first, let the scheduler place a brand-new Pod on that node anyway? 2. Name the two things kubectl drain refuses to do by default, and the exact flag that overrides each one. 3. What's the actual mechanical difference between a Pod removed by a raw kubelet restart and one removed by kubectl drain, in terms of what does or doesn't consult a PodDisruptionBudget? 4. What silently breaks if a node is drained correctly but never uncordoned afterward — and why doesn't anything alert on it? 5. Put the real sequence back in order: drain, the kubeadm package upgrade, kubeadm upgrade node plus the kubelet restart, and uncordon.
Check your answers
- Being
Readyand being schedulable are two independent signals, and the scheduler only reads the second one. A kubelet restart changes neither — nothing marks the node as off-limits — so the scheduler keeps placing new Pods on it exactly as it would any other healthy node, mid-bounce included. - It refuses to delete DaemonSet-managed Pods, overridden with
--ignore-daemonsets— deleting one just lets it immediately reschedule right back onto the same node. It also refuses to delete Pods using localemptyDirstorage, overridden with--delete-emptydir-data, since evicting one destroys data that exists only on that node's disk. kubectl drainremoves Pods through the Eviction API, which checks every affectedPodDisruptionBudgetbefore allowing the deletion to proceed. A rawsystemctl restart kubeletnever calls that API — a service bounce isn't a Pod deletion at all, so there's nothing for a PDB to intercept.- Half the cluster's scheduling capacity disappears silently — new Pods keep landing exclusively on whatever nodes remain schedulable. Nothing alerts because nothing failed: the scheduler is correctly honoring
SchedulingDisabled, and every Pod that does get placed comes up healthy. The only trace is that one word inkubectl get nodes. - Upgrade the
kubeadmpackage on the node, drain it (kubectl drain --ignore-daemonsets --delete-emptydir-data), runkubeadm upgrade node, upgradekubeletandkubectland restart the kubelet, then uncordon.
Fixed the order, watched both mistakes cost something real, cleaned up? That's the whole drill. The kubeadm tool guide covers the full command family this drill only used a slice of, and A Troubleshooting Methodology is the general six-ring method this scenario is one worked example of. Ready for a different single skill? Try Drill — Fix a Broken Cluster for a control-plane failure instead of a worker one, or Drill — Debug a Stuck Pod for five unrelated Pod-level root causes — or step back to the six-part capstone for the continuity version. Chasing every CNCF cert past the CKA? Golden Astronaut picks up the rest of the Kubestronaut ladder where this course leaves off.