Hands-On Labs · Guided Drills

Drill — Recover From a Storage Incident

Two small storage incidents, one skill: telling the difference between "this is broken" and "this is working exactly as designed, and you haven't given it what it's waiting for." First, a PersistentVolumeClaim that sits Pending forever for a reason kubectl describe won't hand you on a plate. Then, an etcd disaster-recovery exercise that starts with a snapshot command that reports success and ends with a restore that refuses to run at all — twice, for two completely different reasons. Both scenarios run on one disposable kind cluster on your own machine, no cloud account, nothing carried over from the five-part capstone. Give yourself 15 minutes for the PVC scenario and 20-25 minutes for the etcd scenario before you look anything up — the whole point is building the habit of checking what's actually true instead of what the first command's exit code implies.

☺ Explain it like I'm 10

Imagine a coat-check counter that only hands you a ticket if your coat's color is on its list. Hand over a color that isn't on the list at all, and the attendant doesn't yell at you or slam a door — they just never look up from their book, because as far as they're concerned nobody handed them anything. That's half of today: a request for storage that nothing in the building is listening for. The other half is bigger: imagine your whole town's memory lives in one giant ledger, and you photocopy a page of it every morning "just in case." Today you photocopy a page and leave it sitting on a desk that gets swept into the trash every night — and only find out when you actually need that copy and it's gone. Then, even once you copy it somewhere safe, you discover the copy machine you tried to use for putting pages back into the ledger got replaced with a different machine years ago, and nobody told you.

🐘🦫Your hosts for this drill: Ellie the Elephant & Benny the Beaver — Ellie is the platform's memory keeper, and both of today's incidents are really the same question she never stops asking: where does this state actually live, and does it survive the next restart? Benny built the manifests you'll be patching, so he's reading etcd.yaml over your shoulder the whole time.
⚠ Before you start

You need Docker (or Podman), kind, and kubectl. No cloud account, no real disks, no charges. Everything lives in one throwaway single-node cluster you delete when you're done: kind delete cluster --name storage-drill. Command flags for etcdctl/etcdutl and kind itself move between versions — if something below errors on a flag, check --help and adapt; that's a small rep of the exact same "don't trust the first assumption" skill this drill teaches.

How this drill works

☺ Like you're 10: Two incidents, no hints until you've actually poked at it yourself, and every fix gets proven, not assumed.

Both scenarios are self-contained bugs you create on purpose, then diagnose using only what the cluster tells you — no answer key until the end. Neither one depends on a StatefulSet, a CSI driver, or anything from the capstone's storage lab; that's deliberate, and it mirrors real on-call work, where a storage incident rarely announces itself with a clean error message pointing straight at the fix.

Set up the scratch cluster

☺ Like you're 10: One tiny throwaway cluster, running for real, on your own laptop.

kind bootstraps a real, single-node cluster via kubeadm inside a Docker container — which matters today, because it means a real static-pod etcd, a real /etc/kubernetes/manifests, and a real default StorageClass, all reachable without touching a cloud provider:

kind create cluster --name storage-drill
kubectl cluster-info --context kind-storage-drill
kubectl get storageclass
NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      DEFAULT
standard (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   true

Exactly one StorageClass, named standard, backed by Rancher's local-path-provisioner, with volumeBindingMode: WaitForFirstConsumer — the same default the exam blueprint's storage page tells you to prefer, and the reason it exists here at all. Remember that output; you'll need to compare against it in a minute.

Scenario 1 — the PVC that never bound

☺ Like you're 10: A parking permit for a lot that doesn't exist gets you nowhere, and the attendant never even bothers telling you why.

You're standing up storage for a small catalog-uploads service. Someone copy-pasted the PVC out of a tutorial written for a managed cloud cluster, where fast-ssd is a real, commonly-provisioned class name. On this kind cluster, it isn't:

# pvc.yaml — BROKEN, as given
apiVersion: v1
kind: Namespace
metadata:
  name: catalog
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: catalog-uploads
  namespace: catalog
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 2Gi
kubectl apply -f pvc.yaml
kubectl get pvc -n catalog
NAME              STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
catalog-uploads   Pending                                      fast-ssd       11s

Pending is expected for a second or two while dynamic provisioning kicks in. It's still Pending a full five minutes later. Reach for the tool that's supposed to explain everything — the object's own events:

kubectl describe pvc catalog-uploads -n catalog | tail -6
Access Modes:
VolumeMode:    Filesystem
Used By:       <none>
Events:        <none>

No events at all. Not a warning, not a "waiting for a volume" message — nothing. That's the trap: it feels like the diagnostic tool is broken, when what's actually happened is that no controller in this cluster ever picked the claim up in the first place. local-path-provisioner only reacts to PVCs naming the exact StorageClass it owns; a name it doesn't recognize isn't a failure it reports, it's a request it never sees. The silence itself is the finding, not a dead end.

Path A — nothing is listening storageClassName: fast-ssd (requested) class doesn't exist no provisioner is watching for that name at all nothing to report Pending forever Events: <none> Path B — working as designed storageClassName: standard WaitForFirstConsumer resolves, binding delayed Pending — one clear event "waiting for first consumer" Pod mounts the claim Bound

Confirm the finding directly instead of trusting the theory:

kubectl get storageclass fast-ssd
Error from server (NotFound): storageclasses.storage.k8s.io "fast-ssd" not found
◆ Key idea

An empty Events list on a Pending PVC is itself informative — most provisioners only act on a PVC once they can resolve its named StorageClass and confirm it's theirs; a name that resolves to nothing produces silence, not a warning. When describe has nothing to say, the next move is kubectl get storageclass, not staring harder at the object that has nothing left to tell you.

A PVC's storageClassName is immutable once the object exists — no in-place edit will save you here. Delete and recreate against the class that actually exists:

kubectl delete pvc catalog-uploads -n catalog
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: catalog-uploads
  namespace: catalog
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: standard
  resources:
    requests:
      storage: 2Gi
EOF
kubectl get pvc -n catalog
NAME              STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
catalog-uploads   Pending                                      standard       4s

Still Pending — and this time that's not a new bug. describe now shows exactly one event, "waiting for first consumer to be created before binding": standard's WaitForFirstConsumer mode deliberately delays provisioning until a Pod actually references the claim, so the disk lands in whatever zone the scheduler already picked (see the storage blueprint page for the full case for why that's the safer default). Give it a consumer:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: catalog-uploader
  namespace: catalog
spec:
  containers:
    - name: app
      image: busybox
      command: ["sleep", "3600"]
      volumeMounts:
        - name: data
          mountPath: /uploads
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: catalog-uploads
EOF
kubectl get pvc -n catalog
NAME              STATUS   VOLUME       CAPACITY   ACCESS MODES   STORAGECLASS   AGE
catalog-uploads   Bound    pvc-8f2c...  2Gi        RWO            standard       41s

Done when: catalog-uploads shows Bound, and you can explain out loud, without checking notes, why the first Pending had zero events and the second Pending had exactly one — and why only one of those two was ever actually broken.

Scenario 2 — the snapshot that saved, and the restore that wouldn't run

☺ Like you're 10: Copying a page isn't backing it up if you leave the copy on a desk that gets emptied every night — and the machine you reach for to put it back might not even do that job anymore.

Plant something worth protecting, the way the CKA lifecycle competency and its study plan both expect you to be able to recover:

kubectl create configmap mission-log \
  --from-literal=status="probe launched, T-minus 4:12, all systems nominal"
kubectl get configmap mission-log

Take a snapshot the way it's usually first demonstrated — exec into the running etcd static Pod and run etcdctl directly:

kubectl exec -n kube-system etcd-storage-drill-control-plane -- \
  etcdctl snapshot save /tmp/etcd-snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
{"level":"info","msg":"snapshot db and kv store are consistent"}
Snapshot saved at /tmp/etcd-snapshot.db

Success, in plain text. Now try to get that file somewhere safe — off the node entirely, onto your own laptop:

docker cp storage-drill-control-plane:/tmp/etcd-snapshot.db ./etcd-snapshot.db
Error: No such container:path: storage-drill-control-plane:/tmp/etcd-snapshot.db

The snapshot command didn't lie — the file really was written to /tmp/etcd-snapshot.db. It just wasn't written to the path you think it was. Confirm directly instead of arguing with the error message:

docker exec storage-drill-control-plane cat /tmp/etcd-snapshot.db
cat: /tmp/etcd-snapshot.db: No such file or directory

It genuinely isn't there — not because the save failed, but because "there" was two different filesystems the whole time. A kind cluster is containers inside a container: storage-drill-control-plane is the Docker container standing in for the node, and the etcd static Pod is a separate container running inside that node, managed by the node's own kubelet and containerd. /tmp inside the Pod's container is private to that container's own writable layer — it is not the node's /tmp, and docker cp only ever reaches the node's own filesystem.

YOUR LAPTOP — docker cp reaches this layer KIND NODE CONTAINER "storage-drill-control-plane" docker exec reaches here — this is the real disk ETCD POD'S OWN CONTAINER kubectl exec reaches here /tmp/etcd-snapshot.db saved here first — private to this one container, gone on restart /var/lib/etcd/etcd-snapshot.db hostPath-mounted — this path really lives on the node's own disk docker cp reaches this file — crosses both boundaries fine

Only two host directories are bind-mounted into that Pod at all, per the static manifest kubeadm generated: /etc/kubernetes/pki/etcd (the certs) and /var/lib/etcd (etcd's own data directory). Anything written under one of those really lands on the node's disk; anything written anywhere else — including /tmp — is gone the instant that Pod's container is replaced, which happens on every etcd restart. Redo the snapshot into the path that actually persists:

kubectl exec -n kube-system etcd-storage-drill-control-plane -- \
  etcdctl snapshot save /var/lib/etcd/etcd-snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

docker exec storage-drill-control-plane ls -lh /var/lib/etcd/etcd-snapshot.db
docker cp storage-drill-control-plane:/var/lib/etcd/etcd-snapshot.db ./etcd-snapshot.db

This time docker cp actually produces a file on your laptop — a real, portable copy, independent of the node and the Pod both. That's the artifact a real incident response would ship off to object storage; Velero automates exactly that shipping step for whole-cluster and whole-namespace backups, though it's worth being precise about the boundary: Velero backs up Kubernetes API objects and PersistentVolume data through CSI snapshots, not etcd's own internal state — the two tools solve adjacent but different problems.

Simulate the incident:

kubectl delete configmap mission-log
kubectl get configmap mission-log
Error from server (NotFound): configmaps "mission-log" not found

Reach for the command every tutorial shows first:

kubectl exec -n kube-system etcd-storage-drill-control-plane -- \
  etcdctl snapshot restore /var/lib/etcd/etcd-snapshot.db \
  --data-dir=/var/lib/etcd/restored-data
Error: unknown command "restore" for "etcdctl snapshot"

It doesn't fail because the snapshot is bad, or the path is wrong. It fails because the command doesn't exist anymore. etcd 3.6 removed snapshot restore from etcdctl entirely and moved it into a separate binary, etcdutl — bundled in the same official etcd image, but a different program with a different name. This is exactly the gap between what most walkthroughs still show and what current etcd ships; the storage & CSI deep-dive notes the same split from the CSI side of the storage stack.

◆ Key idea

etcdutl snapshot restore never talks to a running etcd process at all — it's an offline tool that reads a snapshot file and writes a brand-new data directory from scratch, which is exactly why it can't be run "into" the live /var/lib/etcd your cluster is currently using. But a fresh data directory also needs to agree with the member identity the static Pod is about to start it with — --name and --initial-cluster get baked into the restored directory at restore time, and if they don't match what etcd.yaml passes on boot, the new etcd process refuses to start at all. Read the running manifest before you restore; don't guess the flags.

Read the exact identity the current static Pod is running with, then match it:

docker exec storage-drill-control-plane \
  grep -E -- '--name=|--initial-cluster=|--initial-advertise-peer-urls=' \
  /etc/kubernetes/manifests/etcd.yaml
    - --name=storage-drill-control-plane
    - --initial-advertise-peer-urls=https://172.18.0.2:2380
    - --initial-cluster=storage-drill-control-plane=https://172.18.0.2:2380
kubectl exec -n kube-system etcd-storage-drill-control-plane -- \
  etcdutl snapshot restore /var/lib/etcd/etcd-snapshot.db \
  --data-dir=/var/lib/etcd/restored-data \
  --name=storage-drill-control-plane \
  --initial-cluster=storage-drill-control-plane=https://172.18.0.2:2380 \
  --initial-advertise-peer-urls=https://172.18.0.2:2380

That writes a fully independent, restored data directory — still sitting under /var/lib/etcd, so it's still hostPath-backed and survives the Pod restart you're about to trigger. Point the static Pod at it and let kubelet do the rest:

docker exec storage-drill-control-plane \
  sed -i 's#path: /var/lib/etcd#path: /var/lib/etcd/restored-data#' \
  /etc/kubernetes/manifests/etcd.yaml

Editing that file on the node's real disk is what matters — kubelet is watching /etc/kubernetes/manifests directly (this is what "static Pod" means: no Deployment, no scheduler, kubelet manages it alone), so it notices the change, kills the running etcd container, and starts a new one whose data directory is the one you just restored:

kubectl get pods -n kube-system -l component=etcd -w   # watch it restart, then Ctrl+C
kubectl get configmap mission-log
NAME          DATA   AGE
mission-log   1      9m14s

Done when: mission-log is back, its AGE reads older than the moment you deleted it — proof this is the object your snapshot actually captured, not one you recreated by hand — and you can point at the exact line in etcd.yaml that made the restored data directory take effect.

One habit, two disasters

☺ Like you're 10: Before you trust a tool's "success," ask it to prove the thing actually exists somewhere you can reach it.

Neither bug today announced itself honestly on the first try. Scenario 1's failure mode was total silence where you expected an error. Scenario 2's "success" message was completely true and still pointed at a file that was about to vanish, and the fix everyone reaches for first — etcdctl snapshot restore — doesn't exist on the version you're almost certainly running today. The transferable move is the same both times: don't stop at the first plausible signal, confirm the actual state directly — kubectl get storageclass, docker exec ... cat, a restored ConfigMap's real age — before you believe anything is fixed.

🦆 Dot's-eye view

"I don't want to know which binary etcd moved a subcommand into. I want a green checkmark that says my backup actually works — and the only way to earn that checkmark honestly is to delete something on purpose and watch it come back. A snapshot you've never restored isn't a backup, it's a hope with a timestamp on it."

🐘 Ellie's challenge · going further

Restore isn't finished when the ConfigMap comes back — clean up the way a real runbook would. Once mission-log is confirmed restored, delete the old /var/lib/etcd directory (not restored-data, the original) so a future you doesn't accidentally point the manifest back at stale data by typo. Then go further: schedule a second snapshot immediately after the restore, name it distinctly, and write yourself a one-line rule for how many generations of snapshot you'd keep on a real production control plane — and where off-node they'd actually need to live to survive the node itself disappearing.

0 / 10 steps complete

Scenario 1 — the PVC that never bound

1Create the scratch cluster and confirm the one real StorageClass
Done when: kubectl get storageclass shows exactly one class, named standard, marked default.
2Apply the PVC naming a StorageClass that doesn't exist
Done when: the claim sits Pending and kubectl describe shows Events: <none>.
3Diagnose the silence by checking what StorageClasses actually exist
Done when: kubectl get storageclass fast-ssd returns NotFound, confirming the root cause.
4Delete and recreate the PVC against the real class
Done when: the claim is still Pending, but now with exactly one "waiting for first consumer" event.
5Create the Pod that consumes the claim and watch it bind
Done when: kubectl get pvc -n catalog reports Bound.

Scenario 2 — the snapshot that saved, and the restore that wouldn't run

6Save a snapshot into /tmp and prove it never reached the node
Done when: docker exec ... cat /tmp/etcd-snapshot.db reports "No such file or directory" on the node.
7Redo the snapshot into a hostPath-backed path and copy it off the node
Done when: docker cp produces a real etcd-snapshot.db file sitting on your own laptop.
8Delete mission-log, then watch etcdctl snapshot restore fail outright
Done when: the command errors with unknown command "restore", not a data or path error.
9Read the running manifest's identity flags and restore offline with etcdutl
Done when: etcdutl snapshot restore completes into /var/lib/etcd/restored-data with matching --name and --initial-cluster.
10Repoint the static Pod manifest and confirm mission-log comes back
Done when: kubectl get configmap mission-log succeeds with an AGE older than your delete command.
🎬 At the Pod Squad
🦫

Benny the Beaver: My PVC's Events are just empty. Not even a warning. Is describe broken?

🐘

Ellie the Elephant: Nothing's broken, Benny — nothing's watching. A provisioner only speaks up about a class it recognizes as its own. Ask the cluster what classes actually exist instead of asking the claim why it's quiet.

🦊

Foxy: Wait, so a totally silent Pending and a Pending with one clean event can both be "normal, keep waiting"? How do you ever tell them apart under pressure?

🐘

Ellie the Elephant: You don't guess — you compare against kubectl get storageclass every single time. That one command answers "does this even have somewhere to go" before you spend one more minute reading events that were never going to arrive.

👺

Gizmo: Or, hear me out — since etcd's being difficult, just rm -rf /var/lib/etcd on the live node and let it regenerate a fresh cluster. Clean slate! 🤑

🐢

Timmy the Turtle: Absolutely not, Gizmo. That directory is the only copy of the cluster's entire state that exists anywhere at that moment. You never experiment on the only copy — you take a snapshot first, prove the restore works on a copy, and only then you're allowed to be brave.

🦫

Benny the Beaver: And apparently "etcdctl snapshot restore" doesn't even exist anymore, so being brave with the wrong command wouldn't have worked either way.

🐢 Timmy's checkpoint

1. Why did the first PVC's Events come back completely empty instead of showing an error, and what does that absence actually tell you? 2. Why couldn't you just kubectl edit the PVC to fix its storageClassName in place? 3. After pointing the PVC at a real StorageClass, it was still Pending — was that a second bug, and how do you tell the two Pending states apart from their events alone? 4. Why did the etcd snapshot report success and then turn out to be unreachable from docker cp? 5. Why did etcdctl snapshot restore fail immediately, regardless of the snapshot file or the path given? 6. What two pieces of information did you have to read out of the running etcd.yaml before calling etcdutl snapshot restore, and why do they have to match? 7. What's the one file edit that actually makes a kubeadm-managed cluster start using restored etcd data?

Check your answers
  1. Because no provisioner in the cluster was ever watching for a StorageClass named fast-ssd — since that object doesn't exist, nothing claims the request as its own, so nothing generates an event about it. Empty events here mean "nobody is even trying," not "something failed silently."
  2. A PersistentVolumeClaim's storageClassName is immutable after creation — the fix is to delete the claim and recreate it with the correct class, not edit the existing object.
  3. Yes, it's a genuinely different state, and it isn't a bug — it's WaitForFirstConsumer working as designed, waiting for a Pod to reference the claim before it provisions and binds. Tell them apart by the events: zero events means no provisioner ever recognized the request; exactly one "waiting for first consumer" event means the class resolved fine and it's deliberately waiting on a Pod.
  4. Because the snapshot was written inside the etcd static Pod's own private container filesystem at /tmp, which is separate from the kind node container's filesystem — a kind cluster nests one container inside another, and docker cp only ever reaches the outer node container's real disk, not the inner Pod's ephemeral layer.
  5. Because etcd 3.6 removed the snapshot restore subcommand from etcdctl entirely and moved it into a separate binary, etcdutl — the command simply doesn't exist anymore on a current cluster, independent of whether the snapshot file or the data-dir path is correct.
  6. The --name and --initial-cluster (and --initial-advertise-peer-urls) values the running static Pod manifest is passing at boot — the restored data directory bakes in a member identity at restore time, and if it disagrees with what the manifest passes on the next start, etcd refuses to come up.
  7. Editing /etc/kubernetes/manifests/etcd.yaml on the node's real disk to change the etcd data volume's hostPath to the restored directory — kubelet watches that path directly (that's what "static Pod" means) and automatically restarts etcd against the new data the moment the file changes.

Both incidents resolved? For the full RPC-by-RPC mechanics behind Scenario 1's fix, see Storage & the CSI; for the exam-blueprint depth version of the Pending-PVC diagnosis, see the storage blueprint page. Cluster Architecture, Installation & Configuration covers the same etcd snapshot and restore flags this drill just made you actually run, alongside the full kubeadm upgrade sequence; the Platform Engineering course's CKA page covers the exam-day flag list at reference depth if you want it in one place. Velero covers the adjacent, whole-cluster-object backup problem this drill deliberately didn't touch. For a different single skill entirely, try Drill — Diagnose a Networking Failure or Drill — Harden an RBAC Configuration, or step back to the five-part capstone for the continuity version of storage and stateful apps.