Hands-On · Break-Fix Drills

Break-Fix Drills

Reading a troubleshooting guide teaches you the vocabulary of failure. Only breaking things teaches you the reflex. These fifteen drills each hand you an exact sabotage command, tell you the symptom to watch for, and then start a three-to-five minute clock. That inversion matters: on a performance-based exam nobody hands you a tidy question — you get a cluster that is already wrong, a task list, and a countdown. Every drill here is small, local, disposable, and repeatable, and every one ends with an objective check you can run to prove you actually fixed it rather than merely made the error message go away.

⚖ CNPA vs CNPE — This whole page is CNPE-only: CNPA is a closed-book, multiple-choice exam with no lab component, so you will never sabotage or repair a live cluster on it. The symptom-to-cause reasoning you drill here — what CrashLoopBackOff actually means, why a Service can have no Endpoints — still matters for CNPA’s closed-book recall, just tested as a question instead of a task.

☺ Explain it like I’m 10

Firefighters don’t learn by reading about fire. Someone lights a safe, small fire in a training building and says “go.” Then they do it again tomorrow, and again, until finding the fire takes seconds instead of minutes. These drills are that training building. You light the fire yourself — one command — then race to put it out. Nothing real burns, and every round makes you quicker.

🐘🐢Your hosts for this track: Ellie the Elephant & Timmy the Turtle — Ellie never forgets a symptom she has seen before (she is the one who says “I know that Events line”), and Timmy refuses to let you guess: no fix counts until a command proves it. Between them they will make you fast and honest.
⚠ Read this before you break anything

These drills assume a local, throwaway cluster — kind or minikube on your laptop — that you can delete and rebuild in two minutes. Several drills deliberately damage cluster-wide things (they taint every node, scale CoreDNS to zero, install an admission webhook that rejects pods). Never run them against anything you or anyone else depends on. Also: tool versions, install URLs, CRD apiVersions and CLI flags drift constantly — Kyverno has moved policy fields between releases, Argo Rollouts renames plugin subcommands, chart values get restructured. Treat every command here as a shape, and follow each project’s current quickstart for the exact incantation. When something does not exist the way this page says, that is drill zero: kubectl explain and kubectl api-resources are how you find out the truth on the day.

How to run a drill

Each drill has the same three beats. Break it — run the sabotage exactly as written, without reading ahead. Name it — start a timer and find the failure from the cluster alone, not from this page. Fix it — apply the smallest change that makes the objective check pass. Only then open the collapsible solution to compare your route to ours. If you opened it early, the drill did not happen; reset it and run it again tomorrow.

The clock is the point. Three minutes is not a stretch goal, it is roughly the budget a real exam task leaves you once you have read the question. If a drill takes ten minutes, that is useful data — repeat that one until it takes three. Almost every drill in this set is solved by the same short loop, which is worth burning into your fingers before you start:

# the 60-second loop — run these in this order, every single time
kubectl get pods -n drills -o wide            # what state, how many restarts, which node
kubectl describe pod -n drills <pod>          # STOP at Events: — the answer is usually there
kubectl logs -n drills <pod>                  # app-level failures
kubectl logs -n drills <pod> --previous       # the crash that already happened
kubectl get events -n drills --sort-by=.lastTimestamp | tail -20

# useful aliases for the whole session
alias k=kubectl
kubectl config set-context --current --namespace=drills

Set up the bench once. Everything below lives in a single namespace you can delete between rounds, plus one known-good app to compare against — half of troubleshooting is knowing what “working” looks like on this cluster.

# one-time bench setup
kind create cluster --name drills          # or: minikube start
kubectl create namespace drills
kubectl config set-context --current --namespace=drills

# a known-good reference app — leave it running all session
kubectl create deployment web --image=nginx:1.27-alpine
kubectl expose deployment web --port=80
kubectl get pods,svc                        # this is what healthy looks like

# a throwaway shell you will use constantly (busybox has wget + nslookup)
kubectl run tester --rm -it --image=busybox:1.36 --restart=Never -- sh
# ...and from inside it:  wget -qO- http://web   /   nslookup web

# nuke and reset between drills (this takes the reference app with it —
# re-run the two `create deployment` / `expose` lines above afterwards)
kubectl delete namespace drills && kubectl create namespace drills
0 / 15 drills complete

Round 1 · Pods that will not start

Drill 1CrashLoopBackOff from a bad command — 🐢 Timmy · 3 min
Break it: kubectl -n drills run crashy --image=busybox:1.36 --command -- /bin/sh -c "echo starting; exit 1". Symptom: within a minute the pod cycles RunningErrorCrashLoopBackOff, and the RESTARTS column climbs while the backoff between attempts doubles toward five minutes. Your job: without reading further, establish three facts — the container’s exit code, whether it ever ran at all, and what it printed on its last life. Use kubectl describe pod crashy and read Last State, then kubectl logs crashy --previous (plain logs may fail with “container is waiting to start” — that is why --previous exists). Then fix it by re-running the pod with a command that stays up: --command -- /bin/sh -c "sleep 3600".
Done when: kubectl -n drills get pod crashy shows Running with 0 restarts, and you can state the exit code the broken version returned without looking it up again.
Show the fix
  1. kubectl -n drills describe pod crashyLast State: Terminated, Reason: Error, Exit Code: 1. CrashLoopBackOff is never the cause — it is the kubelet’s reaction. The cause is always in Last State plus the previous logs.
  2. Exit code triage: 1 or another small number = the app itself failed (bad config, missing dependency, bad flag). 127 = command not found. 126 = found but not executable. 137 = SIGKILL (see Drill 2). 143 = SIGTERM.
  3. kubectl -n drills delete pod crashy, then re-run with --command -- /bin/sh -c "sleep 3600". In a real Deployment this is a command/args edit, not a pod delete.
# drill-2-oom.yaml — a container that asks for far more memory than its limit
apiVersion: v1
kind: Pod
metadata:
  name: hungry
  namespace: drills
spec:
  containers:
    - name: stress
      image: polinux/stress
      command: ["stress"]
      args: ["--vm", "1", "--vm-bytes", "200M", "--vm-hang", "1"]
      resources:
        requests:
          memory: "32Mi"
        limits:
          memory: "32Mi"        # <-- the sabotage
Drill 2OOMKilled behind a CrashLoop — 🐘 Ellie · 4 min
Break it: save the manifest above and kubectl apply -f drill-2-oom.yaml. Symptom: the pod looks identical to Drill 1 from kubectl get podsCrashLoopBackOff, restarts climbing — with no application logs at all, because the process is killed before it can complain. Your job: prove this is a memory kill and not an app bug in under two minutes. Run kubectl -n drills describe pod hungry and read Last State: Terminated, Reason: OOMKilled, Exit Code: 137. Confirm the ceiling with kubectl -n drills get pod hungry -o jsonpath='{.spec.containers[0].resources}'. Then fix it the honest way: a bare Pod’s resources cannot be patched, so kubectl -n drills delete pod hungry, raise limits.memory to 256Mi in the file, and kubectl apply -f drill-2-oom.yaml again.
Done when: kubectl -n drills get pod hungry is Running and kubectl -n drills describe pod hungry | grep -i oom returns nothing.
Show the fix
  1. OOMKilled + exit code 137 = the kernel’s cgroup killer, not your app. Logs are usually empty or truncated mid-sentence — that emptiness is itself the tell.
  2. Delete the pod, edit limits.memory to 256Mi, apply again — applying the change to the live pod is rejected with “Pod updates may not change fields other than …”, because a Pod’s resource block is effectively immutable; inside a Deployment the same edit rolls a fresh pod for you. Memory is incompressible: exceed the limit and you are killed instantly. CPU is compressible — exceed it and you are only throttled, which shows up as slowness and failing probes, never as OOMKilled.
  3. The other direction is just as examinable: if the requests are too high the pod never schedules at all (Drill 4). Requests decide where you land; limits decide when you die.
Drill 3ImagePullBackOff — two very different causes — 🐢 Timmy · 5 min
Break it (a): kubectl -n drills run typo --image=nginx:1.27-alpne. Break it (b): kubectl -n drills run private --image=docker.io/acmeinternal/checkout:1.0. Symptom: both pods sit in ErrImagePull, then ImagePullBackOff, and from kubectl get pods they are indistinguishable. Your job: run kubectl -n drills describe pod typo and kubectl -n drills describe pod private and read the Events line on each — they say two completely different things. One reports the manifest or tag was not found; the other reports pull access denied / “may require authorization”, meaning the registry answered but refused you. Fix (a) with kubectl -n drills set image pod/typo typo=nginx:1.27-alpine (or delete and re-run). For (b), write out the two-command repair even if you have no private registry handy: create a docker-registry secret and attach it as an imagePullSecrets entry on the pod spec or on the namespace’s default ServiceAccount.
Done when: kubectl -n drills get pod typo is Running, and you can quote the two Events strings from memory and say which one means “fix the tag” versus “attach a pull secret”.
Show the fix
  1. manifest for … not found” / “not found: manifest unknown” = a typo in the repository or tag. Verify the tag exists before you retype it; :latest hides this class of bug and is why policy engines ban it (Drill 12).
  2. pull access denied … may require ‘docker login’” or 401 Unauthorized = credentials, not spelling. The repair is a pull secret:
    kubectl -n drills create secret docker-registry regcred --docker-server=<registry> --docker-username=<user> --docker-password=<token>
    then either add imagePullSecrets: [{name: regcred}] to the pod spec, or attach it namespace-wide with kubectl -n drills patch serviceaccount default -p '{"imagePullSecrets":[{"name":"regcred"}]}' (existing pods must be recreated to pick it up).
  3. Two more causes worth recognising instantly: a rate-limited public registry (toomanyrequests), and an imagePullPolicy: Never image that was never loaded onto the node — on kind, kind load docker-image.
# drill-4c-pvc.yaml — a claim for a storage class this cluster does not have
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
  namespace: drills
spec:
  storageClassName: fast-ssd        # <-- the sabotage: no such class on kind/minikube
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: needs-disk
  namespace: drills
spec:
  containers:
    - name: app
      image: nginx:1.27-alpine
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data
Drill 4Pending, three ways — 🐘 Ellie · 5 min
One symptom, three unrelated causes, ninety seconds each. Round A — it does not fit: kubectl -n drills create deployment toobig --image=nginx:1.27-alpine then kubectl -n drills set resources deployment/toobig --requests=cpu=64. Round B — it is not welcome: kubectl taint nodes --all drill=hard:NoSchedule then kubectl -n drills run tainted --image=nginx:1.27-alpine. Round C — it is waiting on a disk: apply the manifest above. Symptom: in all three, kubectl get pods shows Pending with no node assigned and no container ever created. Your job: for each round run kubectl -n drills describe pod <name> and read the scheduler’s verdict verbatim from Events — it names the reason and counts the nodes. For Round C also run kubectl -n drills get pvc and kubectl -n drills describe pvc data, then kubectl get storageclass to learn what this cluster actually offers. Fix all three.
Done when: kubectl -n drills get pods shows every drill-4 pod Running, kubectl -n drills get pvc data shows Bound, and kubectl get nodes -o jsonpath='{.items[*].spec.taints}' prints nothing containing drill.
Show the fix
  1. A — Events: “0/1 nodes are available: 1 Insufficient cpu”. The scheduler compares your requests against node allocatable, not against actual usage. Fix: kubectl -n drills set resources deployment/toobig --requests=cpu=100m.
  2. B — Events: “1 node(s) had untolerated taint {drill: hard}”. Either add a matching tolerations block to the pod, or remove the taint with the trailing-dash form: kubectl taint nodes --all drill=hard:NoSchedule-. Taints repel; tolerations permit; nodeSelector/affinity attract — three different mechanisms with three different error strings.
  3. C — Pod Events: “pod has unbound immediate PersistentVolumeClaims”; the real reason is one level down, on the claim: describe pvc → “storageclass.storage.k8s.io "fast-ssd" not found”. Fix: kubectl get storageclass (kind and minikube both ship standard), then kubectl -n drills delete pod needs-disk and kubectl -n drills delete pvc data before recreating both with storageClassName: standard (or with the field omitted, which takes the default class) — storageClassName is immutable, so patching it in place is rejected, and the claim must go before the pod that references it can come back. Same story if a ReadWriteMany claim meets a provisioner that only does ReadWriteOnce.
  4. The habit that beats all three: Pending is a scheduling story, so read the scheduler. kubectl describe pod Events, every time, before touching anything.
# drill-5-config.yaml — a container that demands a key nobody created
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: drills
data:
  GREETING: "hello"                 # note: no APP_MODE key here
---
apiVersion: v1
kind: Pod
metadata:
  name: config-consumer
  namespace: drills
spec:
  containers:
    - name: app
      image: nginx:1.27-alpine
      env:
        - name: APP_MODE
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: APP_MODE         # <-- the sabotage: this key does not exist
Drill 5CreateContainerConfigError — 🐢 Timmy · 3 min
Break it: kubectl apply -f drill-5-config.yaml. Symptom: the pod is neither Pending nor CrashLooping — it sits in CreateContainerConfigError and never starts a container even once, so kubectl logs gives you nothing to work with. Your job: kubectl -n drills describe pod config-consumer and read the Events line, which names the missing key and the object it looked in. Then fix it in place without deleting the pod: kubectl -n drills patch configmap app-config --type merge -p '{"data":{"APP_MODE":"production"}}' and watch the kubelet retry on its own within a minute.
Done when: kubectl -n drills get pod config-consumer is Running and kubectl -n drills exec config-consumer -- printenv APP_MODE prints production.
Show the fix
  1. Events: “Error: couldn’t find key APP_MODE in ConfigMap drills/app-config”. The same status appears for a missing Secret, a missing ConfigMap entirely, or a malformed envFrom reference — the message tells you which.
  2. Add the key (as above), or declare the dependency optional with configMapKeyRef: {name: app-config, key: APP_MODE, optional: true} when the app has a sane default. The kubelet keeps retrying, so a patch is enough — no restart needed.
  3. Neighbouring statuses worth telling apart on sight: CreateContainerConfigError = config the kubelet cannot resolve. CreateContainerError = the runtime refused (bad command path, read-only filesystem). RunContainerError = it failed the instant it started.

Round 2 · Pods that start but do not serve

# drill-6-probes.yaml — a probe pointed at a path nginx does not serve
apiVersion: v1
kind: Pod
metadata:
  name: probey
  namespace: drills
  labels:
    app: probey
spec:
  containers:
    - name: web
      image: nginx:1.27-alpine
      ports:
        - containerPort: 80
      # ROUND A: only this probe is active as written
      readinessProbe:
        httpGet:
          path: /healthz           # <-- nginx returns 404 here
          port: 80
        periodSeconds: 5
      # ROUND B: delete the pod, comment out readinessProbe above,
      # uncomment the six lines below, and apply again
      # livenessProbe:
      #   httpGet:
      #     path: /healthz
      #     port: 80
      #   periodSeconds: 5
      #   failureThreshold: 3
Drill 6Readiness vs liveness — the same 404, two fates — 🐘 Ellie · 5 min
Break it (round A): apply the manifest above with only the readinessProbe present. Symptom A: the pod is Running but stuck at READY 0/1 forever, with RESTARTS 0 — and if you put it behind a Service it silently receives no traffic. Break it (round B): delete the pod, swap so only the livenessProbe is present, re-apply. Symptom B: the container is killed and restarted every ~15 seconds; RESTARTS climbs and you land back in CrashLoopBackOff. Your job: watch both with kubectl -n drills get pod probey -w, read the Events (“Readiness probe failed: HTTP probe failed with statuscode: 404” versus “Liveness probe failed…” followed by “Killing container”), and be able to state the difference in one sentence before you fix anything. Fix both by pointing the probe at /.
Done when: kubectl -n drills get pod probey shows 1/1 Running with 0 restarts, and you can explain why round A produced no restarts while round B produced many.
Show the fix
  1. Readiness answers “should traffic go here?” — failing it removes the pod from the Service’s EndpointSlice and nothing else. Symptom: 0/1 Running, zero restarts, requests quietly failing or hanging elsewhere.
  2. Liveness answers “is this container wedged?” — failing it failureThreshold times makes the kubelet kill and restart the container. Symptom: restart loop. A liveness probe that is too aggressive (short periodSeconds, no initialDelaySeconds, no startupProbe) will restart a slow-booting app forever — an outage entirely manufactured by the probe.
  3. Fix: path: /. In real life, expose a genuine health endpoint, use a startupProbe for slow starters, and never point a liveness probe at a dependency you do not own — otherwise a database blip restarts your whole fleet.
Drill 7A Service with no Endpoints — 🐢 Timmy · 4 min
Break it: on the bench app from setup, run kubectl -n drills patch svc web -p '{"spec":{"selector":{"app":"web-v2"}}}'. Symptom: the pods are perfectly healthy and kubectl get svc web looks completely normal — it has a ClusterIP and a port — but every request from another pod hangs and then times out. Your job: confirm the break from a client first: kubectl -n drills run tester --rm -it --image=busybox:1.36 --restart=Never -- wget -q -T 3 -O- http://web. Then follow the chain the way the exam expects — Service → EndpointSlice → pod labels: kubectl -n drills describe svc web (look for Endpoints: <none>), kubectl -n drills get endpointslices -l kubernetes.io/service-name=web, and kubectl -n drills get pods --show-labels. Repair the selector so it matches the real pod labels.
Done when: kubectl -n drills get endpointslices -l kubernetes.io/service-name=web lists pod IPs, and the busybox wget -qO- http://web returns nginx’s welcome HTML.
Show the fix
  1. kubectl -n drills patch svc web -p '{"spec":{"selector":{"app":"web"}}}' — match the labels the pods actually carry (--show-labels is faster than guessing).
  2. Empty endpoints has exactly four causes, and you should check them in this order: selector mismatch; no ready pods (a failing readiness probe — Drill 6 — removes pods from the slice); wrong namespace (a Service only selects pods in its own); port/targetPort mismatch, where endpoints exist but point at a port nothing listens on.
  3. The distinguishing signal: timeout usually means nowhere to send the packet or a policy is dropping it; connection refused means something answered and said no — a wrong port or a process not listening.
Drill 8DNS is down — 🐘 Ellie · 4 min
Break it: kubectl -n kube-system scale deployment coredns --replicas=0 (on minikube the deployment may be named coredns too — confirm with kubectl -n kube-system get deploy). Symptom: every name lookup in the cluster fails at once. From a test pod, wget -qO- http://web returns “bad address”, while wget -qO- http://<the Service ClusterIP> still works perfectly — that split is the whole diagnosis. Your job: prove it is DNS and not the app. Run kubectl -n drills run dnstest --rm -it --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default.svc.cluster.local, inspect the client’s resolver config with kubectl -n drills exec deploy/web -- cat /etc/resolv.conf (its nameserver line should be the kube-dns ClusterIP, which you can confirm with kubectl -n kube-system get svc kube-dns), then check the resolver itself: kubectl -n kube-system get pods -l k8s-app=kube-dns and kubectl -n kube-system get endpoints kube-dns. Restore it.
Done when: kubectl -n kube-system get endpoints kube-dns lists at least one address and the busybox nslookup kubernetes.default resolves to the API server’s ClusterIP.
Show the fix
  1. kubectl -n kube-system scale deployment coredns --replicas=2 (one replica is fine on a single-node kind cluster).
  2. The four-step DNS drill, in order: does the ClusterIP work when the name does not (→ DNS, not networking); is /etc/resolv.conf pointing at the right resolver; are the CoreDNS pods Running and in the kube-dns endpoints; do the CoreDNS logs (kubectl -n kube-system logs -l k8s-app=kube-dns) show errors such as a loop or an unreachable upstream.
  3. Real-world variants that produce the same symptom: CoreDNS is CrashLooping on a bad Corefile in its ConfigMap; a NetworkPolicy blocks egress to port 53 (Drill 9); or a pod uses dnsPolicy: Default and therefore never sees cluster DNS at all.

Round 3 · Blocked, denied, and misrouted

# drill-9-deny.yaml — a default-deny that also silently kills DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: drills
spec:
  podSelector: {}                  # every pod in the namespace
  policyTypes: ["Ingress", "Egress"]
---
# the repair: additive allow rules, applied WITHOUT removing default-deny
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-and-web
  namespace: drills
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    - to:
        - podSelector:
            matchLabels:
              app: web
      ports:
        - protocol: TCP
          port: 80
  ingress:
    - from:
        - podSelector: {}          # anything in this namespace
      ports:
        - protocol: TCP
          port: 80
Drill 9A NetworkPolicy that blocks in silence — 🐢 Timmy · 5 min
First, check your CNI: NetworkPolicy is only enforced if your network plugin implements it — kind’s default plugin historically does not, so use minikube start --cni=calico or a kind cluster with Calico or Cilium installed. (If nothing gets blocked, that is itself a superb lesson: a policy that is not enforced looks exactly like a policy that is working.) Break it: apply only the default-deny half of the manifest above. Symptom: nothing logs an error anywhere — packets are dropped, not refused — so your test pod hangs on every call, and the first failure you hit is name resolution, because egress denial takes DNS with it. Your job: reproduce with kubectl -n drills run tester --rm -it --image=busybox:1.36 --restart=Never -- wget -q -T 3 -O- http://web, then list what applies with kubectl -n drills get networkpolicy and kubectl -n drills describe networkpolicy default-deny. Now repair it additively: apply the second half (allow DNS egress to kube-system, allow egress to app: web, allow ingress on 80) while leaving default-deny in place.
Done when: with default-deny still applied, the busybox pod can both resolve (nslookup web) and fetch (wget -qO- http://web). Deleting the deny policy does not count as a fix.
Show the fix
  1. NetworkPolicies are allow-lists that combine by union: the moment any policy selects a pod for a direction, everything not explicitly allowed in that direction is denied. There is no “deny rule” to hunt for — you are looking for a missing allow.
  2. The classic omission is DNS egress. Add UDP and TCP port 53 toward the kube-dns pods in kube-system. Forget it and every failure looks like “the other service is down” when the traffic never left the pod.
  3. Read the YAML carefully: a list item with namespaceSelector and podSelector under one dash means “that pod in that namespace” (AND); two separate dashes means “either” (OR). That single character flips the meaning of the rule and is a favourite exam trap.
  4. Verify with intent, not vibes — kubectl -n drills describe networkpolicy to see the effective rules, and on Cilium cilium monitor --type drop to watch the drops as they happen.
# drill-10-ingress.yaml — a rule for a host you will not send
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
  namespace: drills
spec:
  ingressClassName: nginx
  rules:
    - host: shop.example.com       # <-- sabotage A: curl a different Host header
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 80
Drill 10Ingress 404 versus 503 — 🐘 Ellie · 5 min
Setup: get an ingress controller running — minikube addons enable ingress, or on kind apply the ingress-nginx “kind” provider manifest (kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml) onto a cluster created with extraPortMappings for 80/443, then kubectl -n ingress-nginx wait --for=condition=Ready pod -l app.kubernetes.io/component=controller --timeout=180s. Only kind-with-port-mappings answers on localhost: on minikube read minikube ip and use that address everywhere http://localhost/ appears below. Break it (a → 404): apply the Ingress above, then request it with the wrong host: curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: web.example.com' http://localhost/. Break it (b → 503): now send the right host but empty the backend: kubectl -n drills scale deployment web --replicas=0, then curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: shop.example.com' http://localhost/. Symptom: two different numbers that point at two different halves of the system. Your job: produce each code deliberately, then explain which layer each accuses before you repair anything. Check kubectl -n drills describe ingress shop and kubectl -n ingress-nginx logs -l app.kubernetes.io/component=controller --tail=20. Restore both.
Done when: curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: shop.example.com' http://localhost/ prints 200, and you can state the one-line rule that separates 404 from 503.
Show the fix
  1. 404 = the controller got your request but no rule matched: wrong host, wrong path/pathType, a missing or mistyped ingressClassName (so no controller claimed the Ingress at all), or an Ingress living in a different namespace from its Service. Fix: send the host the rule declares, or change the rule.
  2. 503 = the rule did match and the upstream has no ready endpoints. That immediately becomes Drill 7: scale back up (kubectl -n drills scale deployment web --replicas=1) and check the Service’s EndpointSlice.
  3. Remember the third one: connection refused / no response at all means you never reached a controller — no ingress controller installed, no port mapping on kind, or a Service of type LoadBalancer stuck Pending because a local cluster has no cloud provider.
  4. One-liner to memorise: 404 is a routing bug, 503 is a backend bug, no answer at all is an entry-point bug.
Drill 11“Forbidden” — reading an RBAC error — 🐢 Timmy · 4 min
Break it: build a deliberately under-privileged identity — kubectl -n drills create serviceaccount dev, kubectl -n drills create role pod-reader --verb=get,list,watch --resource=pods, kubectl -n drills create rolebinding dev-reads-pods --role=pod-reader --serviceaccount=drills:dev. Now make it do something it cannot: kubectl -n drills get secrets --as=system:serviceaccount:drills:dev. Symptom: Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:drills:dev" cannot list resource "secrets" in API group "" in the namespace "drills". Your job: that sentence contains five facts — who, verb, resource, API group, namespace — and the fix must match all five. Enumerate what the identity really has with kubectl auth can-i --list --as=system:serviceaccount:drills:dev -n drills, then grant the minimum: a new Role for secrets with get,list plus a RoleBinding. Binding cluster-admin is an automatic fail.
Done when: kubectl auth can-i list secrets --as=system:serviceaccount:drills:dev -n drills prints yes and the same command with -n default still prints no.
Show the fix
  1. kubectl -n drills create role secret-reader --verb=get,list --resource=secrets then kubectl -n drills create rolebinding dev-reads-secrets --role=secret-reader --serviceaccount=drills:dev.
  2. The three mistakes that produce this error even when a Role looks right: the RoleBinding is in the wrong namespace (a RoleBinding only grants inside its own); the API group is wrong (deployments live in apps, not ""); or you need a ClusterRole because the resource is cluster-scoped (nodes, PVs, namespaces) — a Role can never grant those, no matter how you write it.
  3. kubectl auth can-i with --as is the fastest instrument on the whole exam: it answers in one second what reading YAML answers in three minutes. Learn --list too.
# drill-12-policy.yaml — Kyverno rejects :latest at admission
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
  annotations:
    # Kyverno normally AUTO-GENERATES a twin of this rule for Deployments, Jobs and
    # CronJobs, which would reject round (b) at the Deployment itself. Turning autogen
    # off is what pushes that denial down onto the ReplicaSet — the whole point of (b).
    pod-policies.kyverno.io/autogen-controllers: none
spec:
  validationFailureAction: Enforce   # on Kyverno 1.13+ this moves DOWN to the rule as
                                     # validate.failureAction: Enforce — set one or the
                                     # other, never both, and check `kubectl explain
                                     # clusterpolicy.spec` on the version you installed
  background: false
  rules:
    - name: require-explicit-tag
      match:
        any:
          - resources:
              kinds: ["Pod"]
              namespaces: ["drills"]
      validate:
        message: "Images must use an explicit tag, not :latest."
        pattern:
          spec:
            containers:
              - image: "!*:latest"
Drill 12An admission webhook says no — 🐘 Ellie · 4 min
Setup: helm repo add kyverno https://kyverno.github.io/kyverno/, helm repo update, then helm install kyverno kyverno/kyverno -n kyverno --create-namespace and kubectl -n kyverno wait --for=condition=Ready pod --all --timeout=180s. Break it: apply the policy above, then kubectl -n drills run latest --image=nginx:latest. Symptom (a): the request is rejected synchronously — no object is created, and the denial text names the webhook and the policy rule. Break it (b) — the harder one: kubectl -n drills create deployment sneaky --image=nginx:latest. Now the Deployment is created happily and zero pods appear, because the denial happened one level down. Your job: find the (b) failure without guessing: kubectl -n drills get deploy,rs, then kubectl -n drills describe rs -l app=sneaky and read its Events, or kubectl -n drills get events --sort-by=.lastTimestamp | tail. Fix by pinning a real tag. Then inspect the machinery: kubectl get validatingwebhookconfigurations and kubectl -n drills get policyreport.
Done when: you have produced the denial from a bare kubectl run, located the identical message in the ReplicaSet’s Events for the Deployment case, and kubectl -n drills get pods -l app=sneaky shows a Running pod after you pin the tag.
Show the fix
  1. kubectl -n drills set image deployment/sneaky nginx=nginx:1.27-alpine. Policy engines are not the enemy — the fix is to comply, and only rarely to change the policy (switch to Audit to observe first, then Enforce).
  2. The exam-critical lesson from (b): controller-created pods fail one layer below where you are looking. A Deployment’s denial surfaces on its ReplicaSet; a Job’s on the Job; a CronJob’s on the Job it spawned. Always walk down the ownership chain with describe. Kyverno’s autogen exists to spare you this — it copies a Pod rule onto the pod controllers so the Deployment is rejected up front, which is why we had to switch it off with pod-policies.kyverno.io/autogen-controllers: none to reproduce the nastier failure.
  3. Two failure modes of the webhook itself: with failurePolicy: Fail, an unhealthy admission controller blocks every matching create in the cluster (the symptom is “nothing can be created any more” and the fix is to heal or remove the webhook configuration); a stale ValidatingWebhookConfiguration left behind after uninstall does exactly the same thing.

Round 4 · Delivery and signal

Drill 13Argo CD stuck OutOfSync on an immutable field — 🐢 Timmy · 5 min
Setup: install Argo CD (kubectl create namespace argocd, then kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml and kubectl -n argocd wait --for=condition=Available deploy --all --timeout=300s) and create an Application pointing at a small manifest folder in a repo you control — the guestbook example works. Let it go Synced/Healthy. Break it: in Git, change the Deployment’s spec.selector.matchLabels (for example app: guestbook-uiapp: guestbook-web) and update the pod template labels to match, then commit and push. Symptom: the app flips to OutOfSync and stays there; every sync attempt fails with Deployment.apps "guestbook-ui" is invalid: spec.selector: Invalid value: …: field is immutable. Self-heal retries forever and never wins. Your job: read the real error rather than the status word — kubectl -n argocd get app guestbook -o jsonpath='{.status.operationState.message}', or argocd app get guestbook — then choose a repair that can actually apply: enable Replace=true in the Application’s syncOptions, run argocd app sync guestbook --replace, or delete the live Deployment and let Argo recreate it from Git.
Done when: kubectl -n argocd get app guestbook reports Synced and Healthy again, and you can say in one sentence why no number of retries would ever have fixed it.
Show the fix
  1. spec.selector on a Deployment, clusterIP on a Service, storageClassName on a PVC, a Job’s spec.template, and most CRD spec fields marked immutable all reject a patch. A GitOps controller only ever runs the equivalent of apply/patch — so the desired state is simply unappliable, and the loop is working correctly while failing forever.
  2. Repairs, least to most invasive: syncOptions: [Replace=true] on that Application (or a Replace=true annotation on the resource), argocd app sync --replace, or delete the live object and let the next reconcile recreate it. Flux’s equivalent is spec.force: true on the Kustomization.
  3. Neighbouring stuck-sync causes to recognise: a resource waiting on a finalizer; SharedResourceWarning when two Applications claim the same object; permanent OutOfSync drift caused by a mutating webhook or another controller writing a field you also declare (fix with ignoreDifferences); and a ComparisonError, which is a repo/render failure, not a cluster failure.
# drill-14-rollout.yaml — a canary that will stop and wait for a human
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: demo
  namespace: drills
spec:
  replicas: 4
  strategy:
    canary:
      steps:
        - setWeight: 25
        - pause: {}                # <-- indefinite: waits for a manual promote
        - setWeight: 50
        - pause: {duration: 30s}
        - setWeight: 100
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
        - name: web
          image: nginx:1.27-alpine
          ports:
            - containerPort: 80
Drill 14A Rollout that stopped halfway — 🐘 Ellie · 4 min
Setup: install the controller — kubectl create namespace argo-rollouts, then kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml — plus the kubectl argo rollouts plugin from the project’s install page, then apply the Rollout above and let it reach Healthy. (The first revision of a Rollout skips the canary steps by design; the pause only bites on the second.) Break it: ship a new version — kubectl argo rollouts set image demo web=nginx:1.27 -n drills — and walk away. Symptom: the release freezes at 25%: one canary pod on the new image, three on the old, and it stays that way indefinitely while the developer messages you that “the deploy is stuck”. Your job: in under four minutes decide which kind of stuck it is. Run kubectl argo rollouts get rollout demo --watch -n drills and read the status message, then kubectl -n drills describe rollout demo. You are separating three cases: an indefinite pause: {} waiting on a human, a timed pause still counting down, and an aborted rollout where an AnalysisRun failed (that one reports Degraded, not Paused — check kubectl -n drills get analysisrun). Then release it.
Done when: kubectl argo rollouts get rollout demo -n drills shows the new revision Healthy at 100% stable, and you can name which of the three stall causes this was and the status string that told you.
Show the fix
  1. Status Paused with reason CanaryPauseStep = a deliberate pause: {}. Release it with kubectl argo rollouts promote demo -n drills (next step) or promote demo --full -n drills (skip all remaining steps). kubectl argo rollouts abort demo -n drills sends it back to stable.
  2. Status Degraded after an AnalysisRun reports Failed = the canary was judged and rejected. That is the system working; the fix belongs in the application or the metric threshold, not in the Rollout. Check kubectl -n drills describe analysisrun <name> for the measurement that failed.
  3. Two more stalls with the same look: the canary pods never become Ready (back to Drills 1–6 — describe the canary ReplicaSet’s pods), or the traffic-routing provider is misconfigured so weights never shift. And if you drive Rollouts from Git, remember Argo CD may show the app Progressing forever while the Rollout waits — the pause is the real state.
# drill-15-sm.yaml — a real target, plus a ServiceMonitor Prometheus will quietly ignore
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metricsapp
  namespace: drills
  labels:
    app: metricsapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: metricsapp
  template:
    metadata:
      labels:
        app: metricsapp
    spec:
      containers:
        - name: app
          image: quay.io/brancz/prometheus-example-app:v0.5.0   # serves /metrics on 8080
          ports:
            - name: metrics
              containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: metricsapp
  namespace: drills
  labels:
    app: metricsapp           # the ServiceMonitor selector must match THIS, not the pod labels
spec:
  selector:
    app: metricsapp
  ports:
    - name: metrics           # the ServiceMonitor endpoint must use this NAME
      port: 8080
      targetPort: metrics
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: metricsapp
  namespace: drills
  labels:
    team: platform            # <-- sabotage 1: missing the "release" label Prometheus selects on
spec:
  selector:
    matchLabels:
      app: web-v2             # <-- sabotage 2: no Service carries this label
  endpoints:
    - port: "8080"            # <-- sabotage 3: must be the Service PORT NAME, not a number
      path: /metrics
      interval: 15s
Drill 15A Prometheus target that never appears — 🐢 Timmy & 🐘 Ellie · 5 min
Setup: helm repo add prometheus-community https://prometheus-community.github.io/helm-charts, helm repo update, then helm install kps prometheus-community/kube-prometheus-stack -n monitoring --create-namespace and wait with kubectl -n monitoring wait --for=condition=Ready pod --all --timeout=300s. Break it: kubectl apply -f drill-15-sm.yaml — it ships a tiny app that really does serve /metrics, a Service with a properly named port, and a ServiceMonitor carrying three independent sabotages. Symptom: no alert, no error, no event: the job simply never shows up at all in Prometheus, and any dashboard or alert built on it silently reports “No data” rather than a problem. Your job: port-forward Prometheus (kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090) and work top-down through Status → Service Discovery and Status → Targets. First learn which ServiceMonitors this Prometheus will even look at: kubectl -n monitoring get prometheus -o yaml | grep -A6 serviceMonitorSelector. Then fix all three sabotages and confirm the target scrapes.
Done when: with the port-forward still running, curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=up{job="metricsapp"}' comes back with a single result whose value is 1 — and the same job reads UP under Status → Targets.
Show the fix
  1. Selection: the Prometheus CR’s serviceMonitorSelector (from the chart, usually release: kps) decides which ServiceMonitors are read at all. Add that label to the ServiceMonitor’s metadata.labels — miss it and your object exists, is valid, and is completely ignored.
  2. Matching: spec.selector.matchLabels must match the Service’s labels (not the pods’), and endpoints[].port must be the Service port’s name (e.g. metrics) — a number belongs in targetPort. Also check namespaceSelector if the Service lives in another namespace.
  3. Read the two states differently: a target that is missing is a discovery/selection bug (labels, selectors, namespace). A target that is present but DOWN is a scrape bug — wrong path, wrong port, connection refused, TLS, or a NetworkPolicy blocking Prometheus (Drill 9). Service Discovery shows dropped targets and the relabelling that dropped them.
🦆 Dot’s-eye view

“Every one of these fifteen has landed in my inbox as ‘it’s broken, can you look?’ — and honestly I can’t tell them apart. Pending, CrashLoop, 503, a rollout that never finished: from where I sit they’re all just ‘my thing isn’t up.’ The platform engineers I trust most aren’t the ones who know the most YAML. They’re the ones who get from my one-line complaint to the right describe command in about ten seconds.”

What you will have built

☺ Like you’re 10: Not a thing you can point at — a reflex. Fifteen kinds of broken that you can now recognise on sight instead of guessing at.

There is no artefact at the end of this page, and that is deliberate: what you build here is recall speed. Fifteen sabotages give you fifteen symptom-to-cause shortcuts and, more valuable still, a handful of discriminations — the pairs that look identical from kubectl get pods and diverge completely one command later. CrashLoopBackOff from a bad command versus CrashLoopBackOff from an OOM kill. A missing image tag versus a missing pull secret. Pending because it does not fit versus Pending because it is not welcome versus Pending because a volume never bound. Readiness (dropped from the endpoints, no restarts) versus liveness (killed and restarted). A 404 that accuses your routing rules versus a 503 that accuses your endpoints. A target that is missing versus a target that is down.

That maps directly onto the exam’s diagnose-and-remediate work, and onto real on-call. It also completes the loop with the rest of the site: the troubleshooting playbook gives you the universal triage order and a symptom index, the three deep playbooks (workloads, networking, delivery) give you the decision trees, the command reference gives you the exact flags, and this page is where you turn all of it into muscle. If you have not yet built the platform these drills break, do the lab track first — several drills assume Argo CD, Argo Rollouts, Kyverno and Prometheus are already installed. Then take the same skills into timed conditions with the practice task bank — its observability and security sets lean hardest on these reflexes — and the mock exams.

◆ Key idea

Almost every drill on this page is solved by the same three commands in the same order: kubectl get pods -o wide, kubectl describe (stop at Events), kubectl logs --previous. The skill is not knowing more commands — it is refusing to type anything else until those three have run.

🦫 Benny’s challenge · going further

Ready to raise the difficulty? One: write the fifteen sabotages into a single break.sh that picks one at random, applies it, and prints nothing — then have a friend run it while you diagnose blind. Two: chain two sabotages in one cluster (a NetworkPolicy and a selector mismatch) so the first fix does not restore service; multi-cause incidents are where confidence goes to die. Three: cut every time box in half. Four: for each drill, write the one-sentence incident note you would send Dot — symptom, cause, fix, prevention. Five: turn three drills into prevention instead of cure — a Kyverno policy that requires memory limits, a readiness probe convention in your Backstage template, and a PrometheusRule that pages on kube_pod_container_status_restarts_total climbing.

🎬 At the Platform Guild
🦆

Dot: It’s broken. It just says CrashLoopBackOff. That means the code is bad, right?

🐘

Ellie: It means the kubelet is restarting something that keeps dying. That’s the reaction, not the cause. describe it — Last State, exit code.

🦊

Foxy: Exit code 137. So… the code is really bad?

🐘

Ellie: 137 is SIGKILL. Reason says OOMKilled. Nobody’s code is bad — your memory limit is 32Mi and the app wants 200.

👺

Gizmo: Easy fix! Delete the limits. Delete all the limits. Give it the whole node! 😈

🐢

Timmy: And the next noisy pod evicts everything else on that node. Raise it to what the app measures, then prove it: describe shows no OOM, pod stays Running.

🦥

Sol: …I was going to restart it a few times and see if it fixed itself.

🐢

Timmy: That’s not a fix, Sol, that’s a coin flip with extra steps. Slow is smooth. Smooth is fast.

🐢 Timmy’s checkpoint

1. Two pods both show CrashLoopBackOff. Which single command, and which field in its output, tells you whether one was OOM-killed? 2. A Service has a ClusterIP and pods are Running, but requests time out. What are the first two things you check, in order? 3. Your readiness probe fails. How many restarts do you expect, and what else changes? 4. An Ingress returns 503 for the correct host. Which layer is broken — and which returns 404 instead? 5. A GitOps app is permanently OutOfSync with “field is immutable”. Why will retrying never help, and what are two repairs? 6. Your Prometheus target is not DOWN, it is absent entirely. What kind of bug is that?

Check your answers
  1. kubectl describe pod <name> → the container’s Last State block: Reason: OOMKilled, Exit Code: 137 versus Reason: Error, Exit Code: 1. CrashLoopBackOff is the kubelet’s reaction; Last State is the cause.
  2. First the Service’s EndpointSlice (kubectl describe svcEndpoints: <none>, or kubectl get endpointslices -l kubernetes.io/service-name=<svc>); then the pod labels versus the selector (kubectl get pods --show-labels). If endpoints exist but calls still fail, look at port/targetPort, then NetworkPolicy.
  3. Zero restarts. The pod stays Running at 0/1 and is removed from the Service’s EndpointSlice, so it silently receives no traffic. A failing liveness probe is the one that kills and restarts the container.
  4. 503 = the rule matched but the backend Service has no ready endpoints — a backend problem. 404 = no rule matched at all: wrong host, wrong path/pathType, wrong or missing ingressClassName, or the Ingress is in a different namespace from its Service. No response at all means you never reached a controller.
  5. Because a reconciler only ever applies/patches, and the API server rejects the patch every time — the desired state is unappliable, so the loop fails identically forever. Repairs: enable Replace=true (Argo CD syncOptions / argocd app sync --replace, or Flux’s spec.force), or delete the live resource and let the next reconcile recreate it.
  6. A discovery/selection bug, not a scrape bug — the serviceMonitorSelector label, the spec.selector against the Service’s labels, or the namespaceSelector. A target that appears but reads DOWN is the scrape failing: wrong port name, wrong path, refused connection, TLS, or a NetworkPolicy in the way.