Drill — Debug a Stuck Pod

This drill hands you a throwaway cluster and one YAML file: five Deployments, applied in a single kubectl apply, and every single one of them stuck. Not stuck the same way, either — one's Pending because it asked for more memory than the cluster has, a second is Pending for an entirely different reason, a third can't pull its image, a fourth keeps crashing because it's missing something it needs to even start, and a fifth is being killed by its own memory limit. Nothing here depends on the capstone cluster or any lesson before it — bring your own kind, your own kubectl, and about thirty minutes. The skill isn't memorizing what CrashLoopBackOff means; it's the discipline of reading the cheapest signal first, on five objects that all look equally broken from kubectl get pods alone and are not equally broken underneath.

How this drill works

☺ Like you're 10: Break five things on purpose, all at once, then fix each one for real before moving to the next.

You'll stand up a bare single-node kind cluster, apply one manifest containing five Deployments that are each broken in a different, self-inflicted way, and work them one at a time using nothing but kubectl get, kubectl describe, and kubectl logs — Ring 0 through Ring 2 of the method a troubleshooting methodology lays out in full. This drill doesn't re-teach that method; it makes you actually run it, five times, against five objects that share nothing except the fact that kubectl get pods shows all five as "not Running." For each case: read the symptom, name your hypothesis before you open the reveal, then check it against the fix. A hypothesis you were talked into by an answer key teaches you less than one you defended yourself, even wrongly.

Set up the scratch cluster

☺ Like you're 10: One tiny throwaway cluster, built just for this — nothing you'd be sad to delete an hour from now.

A one-line, default, single-node kind cluster is all this drill needs — nothing here is about networking or multi-node topology, so skip the config file entirely:

kind create cluster --name debug --image kindest/node:v1.31.0
kubectl create namespace stuck-pods
kubectl config set-context --current --namespace=stuck-pods

Save the broken bundle below as broken.yaml and apply it in one shot:

# broken.yaml — five Deployments, five self-inflicted problems, applied together
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ledger-writer
  namespace: stuck-pods
  labels: { app: ledger-writer }
spec:
  replicas: 1
  selector: { matchLabels: { app: ledger-writer } }
  template:
    metadata: { labels: { app: ledger-writer } }
    spec:
      containers:
        - name: ledger-writer
          image: nginx:1.27-alpine
          resources:
            requests: { cpu: "100m", memory: "64Gi" }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: promo-scheduler
  namespace: stuck-pods
  labels: { app: promo-scheduler }
spec:
  replicas: 1
  selector: { matchLabels: { app: promo-scheduler } }
  template:
    metadata: { labels: { app: promo-scheduler } }
    spec:
      nodeSelector:
        disktype: ssd
      containers:
        - name: promo-scheduler
          image: nginx:1.27-alpine
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: receipts-api
  namespace: stuck-pods
  labels: { app: receipts-api }
spec:
  replicas: 1
  selector: { matchLabels: { app: receipts-api } }
  template:
    metadata: { labels: { app: receipts-api } }
    spec:
      containers:
        - name: receipts-api
          image: nginx:1.27-alpne   # <- typo, baked in on purpose
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: session-cache
  namespace: stuck-pods
  labels: { app: session-cache }
spec:
  replicas: 1
  selector: { matchLabels: { app: session-cache } }
  template:
    metadata: { labels: { app: session-cache } }
    spec:
      containers:
        - name: session-cache
          image: busybox:1.36
          command: ["sh", "-c"]
          args:
            - |
              if [ -z "$CACHE_TOKEN" ]; then
                echo "FATAL: CACHE_TOKEN is required, refusing to start" >&2
                exit 1
              fi
              sleep 3600
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: report-builder
  namespace: stuck-pods
  labels: { app: report-builder }
spec:
  replicas: 1
  selector: { matchLabels: { app: report-builder } }
  template:
    metadata: { labels: { app: report-builder } }
    spec:
      containers:
        - name: report-builder
          image: busybox:1.36
          command: ["sh", "-c"]
          args:
            - "dd if=/dev/zero of=/dev/shm/pad bs=1M count=200; sleep 3600"
          resources:
            requests: { memory: "10Mi" }
            limits: { memory: "20Mi" }

kubectl apply -f broken.yaml
kubectl get pods
# give it 60-90 seconds, then confirm all five are unhealthy in five different ways:
NAME                              READY   STATUS             RESTARTS   AGE
ledger-writer-6bd9c7f9d4-k2m7p    0/1     Pending            0          75s
promo-scheduler-7f4d8b9c6-x9wjq   0/1     Pending            0          75s
receipts-api-5c8f6d4b7-ht2mn      0/1     ImagePullBackOff   0          75s
session-cache-84b6c9f5d-p4rkw     0/1     CrashLoopBackOff   3          75s
report-builder-9d5c7b8f6-m2vqz    0/1     CrashLoopBackOff   2          75s

Done when: all five Pods are listed, none of them 1/1 Running. Nothing above is a trick or a race condition — every one of these is deterministic and will look exactly like this on any machine, every time.

Case 1 — ledger-writer: Pending, and it's not a scheduler bug

☺ Like you're 10: The Pod isn't stuck waiting in line — it's waiting for a seat that doesn't exist anywhere in the building.

Pending means exactly one thing at Ring 0: no container has been created yet. It does not tell you why. Ring 1 — events — usually does, for free, with no container required to be running at all:

kubectl describe pod -l app=ledger-writer | tail -6

Before opening the reveal: what would make a Pod requesting only 100m CPU fail to schedule on any node at all?

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  90s   default-scheduler   0/1 nodes are available: 1 Insufficient memory.
                                                         preemption: 0/1 nodes are available: 1 No preemption
                                                         victims found for incoming pod.

kubectl set resources deployment/ledger-writer -c=ledger-writer --requests=cpu=100m,memory=128Mi
kubectl get pods -l app=ledger-writer
# → a brand new Pod appears, and within seconds: 1/1 Running

Case 2 — promo-scheduler: also Pending, for a completely different reason

☺ Like you're 10: This one isn't asking for a seat too big for the room — it's asking for a room that doesn't exist at all.

kubectl get pods shows this one as Pending too — identical status string to Case 1, and that's the whole point of running this drill instead of just reading about it: the same three characters can mean two Pods have nothing in common except that neither has been scheduled yet.

kubectl describe pod -l app=promo-scheduler | tail -4

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  70s   default-scheduler   0/1 nodes are available: 1 node(s) didn't match
                                                         Pod's node affinity/selector.

# Option A — the selector was copy-pasted from another environment and doesn't belong here:
kubectl patch deployment promo-scheduler --type=json \
  -p='[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]'

# Option B — the workload genuinely needs SSD-backed nodes, so label one to match instead:
kubectl get nodes -o name
kubectl label node debug-control-plane disktype=ssd

Case 3 — receipts-api: ImagePullBackOff, and no shell to blame

☺ Like you're 10: A node was found, a spot was reserved — the delivery truck just can't find a package with that exact label on it.

Unlike the two Pending cases, this Pod was scheduled — a node exists and a container was about to start. Something after scheduling, before the container ever ran, is the blocker. Ring 1 again, no exec required:

kubectl describe pod -l app=receipts-api | tail -6

Events:
  Type     Warning  Reason   Age   From     Message
  -------  -------  ------   ---   ----     -------
  Normal   Pulling  55s      kubelet  Pulling image "nginx:1.27-alpne"
  Warning  Failed   52s      kubelet  Failed to pull image "nginx:1.27-alpne": rpc error: code = NotFound
                                       desc = failed to resolve reference "docker.io/library/nginx:1.27-alpne":
                                       nginx:1.27-alpne: not found
  Warning  Failed   52s      kubelet  Error: ErrImagePull
  Normal   BackOff  40s      kubelet  Back-off pulling image "nginx:1.27-alpne"
  Warning  Failed   40s      kubelet  Error: ImagePullBackOff

kubectl set image deployment/receipts-api receipts-api=nginx:1.27-alpine
kubectl get pods -l app=receipts-api
# → 1/1 Running within a few seconds — the pull succeeds immediately once the tag is real

Case 4 — session-cache: CrashLoopBackOff, and the reason is in a log you have to catch fast

☺ Like you're 10: This one did start — it just took one look around, decided something was missing, and quit before it ever got comfortable.

Unlike Cases 1–3, a container here has actually run at least once — that's the whole definition of CrashLoopBackOff. That means Ring 2, logs, has something to say — but only if you ask for the crashed attempt specifically:

kubectl logs -l app=session-cache
# → often empty or "previous terminated container ... not found" if you're too slow —
#   the current attempt hasn't printed anything new yet, and the last one is gone.
#   Ask for the PREVIOUS container's logs explicitly instead:
kubectl logs -l app=session-cache --previous

FATAL: CACHE_TOKEN is required, refusing to start

kubectl set env deployment/session-cache CACHE_TOKEN=dev-local-token
kubectl get pods -l app=session-cache -w
# → STATUS cycles once more, restartCount stops climbing, then settles at 1/1 Running

Case 5 — report-builder: also CrashLoopBackOff — but not the same failure wearing the same label

☺ Like you're 10: Two toy cars both won't start, but one's out of gas and the other's engine straight-up seized — same symptom on the dashboard, opposite problem underneath.

kubectl get pods shows this one as CrashLoopBackOff too, identical to Case 4 — and it is not the same failure. CrashLoopBackOff only ever describes kubelet's restart-frequency wrapper; the actual cause lives one field deeper, in lastState.terminated.reason:

kubectl get pod -l app=report-builder \
  -o jsonpath='{.items[0].status.containerStatuses[0].lastState.terminated}'

{
  "reason": "OOMKilled",
  "exitCode": 137,
  "startedAt": "2026-08-27T09:14:02Z",
  "finishedAt": "2026-08-27T09:14:04Z"
}

# Option A — it's debug cruft nobody meant to ship: remove the step entirely
kubectl patch deployment report-builder --type=json \
  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/args","value":["sleep 3600"]}]'

# Option B — the workload genuinely needs to buffer that much: give it the memory instead
kubectl set resources deployment/report-builder -c=report-builder --limits=memory=256Mi

Verify: all five green, and staying green

☺ Like you're 10: "Fixed for a second" and "actually fixed" aren't the same thing — wait, then check again.

Once all five fixes are applied, confirm the whole namespace, not just one Pod at a time:

kubectl get deployments
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
ledger-writer     1/1     1            1           11m
promo-scheduler   1/1     1            1           11m
receipts-api      1/1     1            1           11m
session-cache     1/1     1            1           11m
report-builder    1/1     1            1           11m

READY 1/1 the instant after a fix only proves the new Pod started — it doesn't prove it stays started. Confirm restart counts are actually flat, not just recently reset, by checking twice, two minutes apart:

kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.restartCount}{"\n"}{end}' \
  2>/dev/null || kubectl get pods -o custom-columns=NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount
# run it, wait two minutes, run it again — every number should be identical both times

Done when: all five Deployments read 1/1, and every restart count is unchanged across two checks two minutes apart. When you're satisfied, tear the whole thing down — nothing here needs to outlive this drill:

kind delete cluster --name debug

Every fix in this drill reused a single discipline: read the cheapest signal first, and only escalate once it comes up empty — the full six-ring version of that method, plus the exact mechanism behind all five failure signatures used here, lives in a troubleshooting methodology. For the exam-paced version of the same four Ring 0–3 moves under a two-hour clock, see the Exam Blueprint's Troubleshooting domain. For the scheduling math behind why ledger-writer couldn't fit anywhere, see Scheduling & resource management. Ready for the messier, single-cluster version where every one of these failure types is mixed into one continuous outage instead of five neatly separated Deployments? Try Drill — Fix a Broken Cluster, or step back to the six-part capstone for the full continuity version.