Storage & the CSI
The exam blueprint's storage domain teaches you to write a StorageClass, claim a PersistentVolumeClaim, and reason about access modes and reclaim policy — everything you need to pass a performance-based task. None of that explains what actually happens between kubectl apply and a formatted disk showing up inside a container. This page goes underneath the PVC: the Container Storage Interface (CSI) as a plugin contract, the exact sequence of remote procedure calls that turns "provision this" into a mounted filesystem, how StatefulSets pair stable network identity with stable per-replica storage so a database pod can find its own disk again after a reschedule, and how the same CSI contract extends to point-in-time volume snapshots. This is advanced, beyond-exam material — the kind of thing you reach for when a PVC is stuck and kubectl describe alone isn't enough to tell you why.
Imagine a hotel where every room needs a bed, but the hotel doesn't make beds itself — it just has a rule: any bed company can plug into the hotel's ordering system, as long as they answer three questions the same way every other bed company does. "Can you make a bed?" "Can you bring the bed to a room and set it up?" "Can you take a snapshot photo of exactly how the bed looked, so we can rebuild it later?" That's the CSI — Kubernetes doesn't know or care whether your disk is a cloud provider's block storage, a shared network drive, or a fancy distributed filesystem. It just asks the same three questions, in the same order, every single time, and lets a specialist plugin answer them. You never see the plugin's own private conversation with the disk company — you only ever see the finished bed, already made, in the room.
CSI: one contract, any storage system
☺ Like you're 10: Kubernetes doesn't build disks itself — it hires a specialist plugin for each storage system, and every plugin has to answer the exact same set of questions the exact same way.
Early Kubernetes shipped storage drivers in-tree — compiled directly into kube-controller-manager and kubelet, one if branch per cloud provider. That worked until it didn't: every new storage backend needed a Kubernetes core release to add support, every driver bug was a core Kubernetes bug, and vendors had no way to ship a fix without waiting on the project's own release cadence. The Container Storage Interface, a vendor-neutral gRPC specification co-developed with the Cloud Foundry and Mesos communities, fixed this by moving storage drivers out of Kubernetes core entirely. A CSI driver is an ordinary container that implements a small, fixed set of RPCs; Kubernetes talks to it the same way regardless of whether it's backed by a cloud block store, a SAN, or a distributed filesystem. Every in-tree volume plugin that used to ship inside Kubernetes has since been migrated to an out-of-tree CSI driver — the in-tree AWS EBS and GCE PD plugins, for two examples, were removed entirely in Kubernetes 1.27, years after CSI Migration made the switch transparent to existing manifests.
A CSI driver implements three gRPC services, and no driver is required to implement all of every method — a driver declares what it supports through its own capability RPCs. Identity answers "what are you and what do you support" — plugin name, version, and capabilities. Controller covers the operations that don't need to run on a specific node: CreateVolume, DeleteVolume, ControllerPublishVolume (attach), ControllerUnpublishVolume (detach), CreateSnapshot, and volume expansion. Node covers the operations that must run on the node where a Pod actually lands: NodeStageVolume, NodePublishVolume, and their unstage/unpublish counterparts. Kubernetes never calls a driver's gRPC endpoint directly — it delegates to a set of Kubernetes-maintained, driver-agnostic sidecar containers that watch the API server and translate object changes into the correct RPC, in the correct order, against the driver's Unix domain socket.
Kubernetes exposes two objects that make this plugin architecture inspectable rather than opaque. A CSIDriver is cluster-scoped and self-published by the driver on startup — it declares whether attach is even required (attachRequired: false for drivers like most NFS- and EFS-style network filesystems, which have nothing to physically attach), whether Pod metadata should be passed through on mount, and which volume lifecycle modes the driver supports. A CSINode is per-node and lists which drivers are registered there and each driver's node ID — the object the scheduler consults so it never places a Pod needing a volume on a node where the right driver isn't even running.
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
name: ebs.csi.aws.com
spec:
attachRequired: true # false for drivers with nothing to attach, e.g. NFS
podInfoOnMount: true # driver receives the requesting Pod's name/namespace
fsGroupPolicy: File # how fsGroup ownership is applied on mount
volumeLifecycleModes:
- Persistent # the normal PVC-backed path
- Ephemeral # optional: inline volumes with no PVC at allA CSIDriver with attachRequired: false means ControllerPublishVolume is never called for that driver — there's no block device to attach, so a stuck Pod's volume problem for an NFS-backed class is almost never an attach issue. Checking kubectl describe pvc events and the driver's own node-plugin logs will tell you more than staring at a VolumeAttachment object that was never going to exist in the first place.
Dynamic provisioning, RPC by RPC
☺ Like you're 10: "Create this disk" is really four separate handoffs — make the disk, attach it to the right room, get it ready inside that room, then finally hand it to the guest — and each handoff is a different phone call to a different specialist.
The blueprint page teaches what a StorageClass and volumeBindingMode: WaitForFirstConsumer do; this is how, one RPC at a time, from an unbound PVC to a container that can actually read and write its disk. Nothing here is optional plumbing to skip past — each stage is where a specific class of "my PVC is stuck" bug actually lives.
Two RPCs run on the control plane, watching API objects: CreateVolume, triggered by an unbound PVC, and ControllerPublishVolume, triggered once a Pod is actually scheduled and a VolumeAttachment object is created to represent the pending attach. Two more run on the node itself, called directly by kubelet: NodeStageVolume mounts the raw device once to a staging path private to that node — the step where formatting happens, if the volume is fresh — and NodePublishVolume bind-mounts that staging path into the specific directory the Pod's container will see. Splitting stage from publish matters the moment two containers in the same Pod share a volume: staging happens once per node, publishing happens once per Pod, so the expensive mount-and-format work is never repeated for a second consumer of the same disk.
$ kubectl get volumeattachments
NAME ATTACHER PV NODE ATTACHED
csi-3a9f... ebs.csi.aws.com pvc-8b21c4de-... node-3 true
$ kubectl get csinodes node-3 -o jsonpath='{.spec.drivers[*].name}'
ebs.csi.aws.comA Pod stuck Terminating after its node dies without a clean shutdown is very often waiting on ControllerUnpublishVolume — the VolumeAttachment object still says attached: true because Kubernetes can't safely confirm the disk was actually released by a node it can no longer reach, and mounting the same block device twice risks silent corruption. The correct fix is confirming the node is truly gone (not just briefly unreachable) and then deleting the Node object, which lets the attach-detach controller force the detach — never delete a VolumeAttachment by hand while the underlying node might still be alive and writing to that disk.
StatefulSets: stable identity paired with stable storage
☺ Like you're 10: A Deployment hands out interchangeable name tags and shrugs about who gets which locker. A StatefulSet gives out numbered name tags for life, and each number always gets its own locker back, even after a fire drill.
A Deployment's replicas are anonymous and interchangeable by design — web-7d9f4c8b6-xq2kp could vanish and be replaced by web-7d9f4c8b6-mz91t and nothing downstream should ever notice. That assumption breaks the moment replicas aren't interchangeable: a database primary and its replicas play different roles, a Kafka broker needs a stable identity so partition assignment survives a restart, and any clustered system that votes or elects a leader needs to reliably re-find the same peers after a crash. The object model covers the general desired-state pattern every workload type shares; StatefulSet is the one workload type built specifically to violate the "replicas are interchangeable" assumption on purpose, and it does that with two mechanisms working together.
Stable network identity comes from an ordinal index baked into the Pod name — db-0, db-1, db-2, never a random suffix — combined with a headless Service (clusterIP: None). A headless Service doesn't load-balance; it returns a DNS A record per Pod, so each ordinal gets its own stable, predictable DNS name: db-0.db-headless.default.svc.cluster.local resolves to exactly that one Pod, whether it's the Pod that's been running for a month or one recreated ninety seconds ago after a crash. A clustered database's peers configuration can hardcode those names once and never touch them again.
Stable storage comes from volumeClaimTemplates: instead of one shared PVC for the whole StatefulSet, Kubernetes stamps out a dedicated PVC per ordinal — data-db-0, data-db-1, data-db-2 — the first time each replica is created, and never again. If db-1's Pod is deleted and recreated, the new Pod comes back as db-1, on the same DNS name, and is bound to the exact same data-db-1 PVC it had before — not a fresh empty volume. That pairing, same name plus same disk every single time, is the entire mechanism that lets a StatefulSet replica "come back to itself" after a failure instead of rejoining the cluster as a blank node that has to resync everything from scratch.
apiVersion: v1
kind: Service
metadata:
name: db-headless
spec:
clusterIP: None # headless — one DNS record per Pod, no load-balancing
selector: { app: db }
ports: [{ port: 5432 }]
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: db
spec:
serviceName: db-headless # must match the headless Service above
replicas: 3
selector: { matchLabels: { app: db } }
template:
metadata: { labels: { app: db } }
spec:
containers:
- name: db
image: postgres:16
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
volumeClaimTemplates: # one PVC minted per ordinal, on first creation only
- metadata: { name: data }
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: fast-ssd
resources: { requests: { storage: 20Gi } }
persistentVolumeClaimRetentionPolicy:
whenDeleted: Retain # keep PVCs if the whole StatefulSet is deleted
whenScaled: Delete # drop data-db-2's PVC if you scale 3 → 2Creation and scale-up run strictly in order, 0 then 1 then 2, and each ordinal must reach Running and Ready before the next one starts — the default OrderedReady pod management policy, which exists so a member joining a cluster never races ahead of the peer it needs to join first. Scale-down and deletion run in strict reverse, highest ordinal first. The persistentVolumeClaimRetentionPolicy field, GA since Kubernetes 1.27, is the piece that finally answered the question every StatefulSet operator used to have to solve by hand: what happens to a replica's PVC when it disappears? whenScaled controls the PVC's fate when the StatefulSet shrinks — Delete is often right for a cache, wrong for anything holding data you can't regenerate — and whenDeleted controls it when the entire StatefulSet object is removed, where Retain is almost always the safer default.
| Deployment | StatefulSet | |
|---|---|---|
| Pod naming | Random hash suffix | Stable ordinal — db-0, db-1, … |
| DNS | Only via the Service's single load-balanced name | One stable per-Pod name via a headless Service |
| Storage | None by default, or one shared PVC if added manually | One dedicated PVC per ordinal, reused across reschedules |
| Startup/scaling order | Unordered — all replicas at once | Strictly ordered, one at a time by default |
StatefulSets give you the primitive; they don't give you a running, self-healing database on their own — failover, backup scheduling, and version upgrades are exactly what a database operator adds on top. This course's Stateful Workloads & Database Operators picks up from here with that pattern, and Platform Engineering's Storage & Stateful Workloads covers the same ground again from a platform team's altitude — storage as a self-service capability, backup and disaster recovery, and the cost side of data gravity.
"The first time I scaled a StatefulSet from 3 down to 1 and back up to 3, I expected db-1 and db-2 to come back empty. They didn't — because I'd set whenScaled: Retain, their old PVCs were just sitting there, unbound, waiting. When the replicas came back they rejoined with all their old data still attached, which was either exactly what I wanted or a very quiet way to reintroduce stale data into a cluster, depending entirely on whether I meant to set that field that way. Read your own retention policy before you scale down — it's not a detail, it's the whole decision."
Volume snapshots: point-in-time copies through the same interface
☺ Like you're 10: A snapshot is a photograph of exactly what a disk looked like at one instant — and just like a photograph, it's useless if the room was mid-earthquake when you took it.
CSI extends the same plugin contract to point-in-time copies, using three objects that mirror the PVC/PV/StorageClass triangle almost exactly. A VolumeSnapshotClass names a snapshotter driver and its parameters, the same role a StorageClass plays for provisioning. A VolumeSnapshot is the namespaced request — "snapshot this PVC right now" — the same role a PVC plays. Kubernetes creates a VolumeSnapshotContent to represent the real, driver-side snapshot object, the same role a PV plays for a real disk. The csi-snapshotter sidecar (paired with a cluster-wide snapshot-controller) watches for new VolumeSnapshot objects and calls the driver's CreateSnapshot RPC, exactly the same watch-and-translate pattern external-provisioner uses for PVCs.
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: fast-ssd-snapshots
driver: ebs.csi.aws.com
deletionPolicy: Retain # keep the snapshot even if the VolumeSnapshot is deleted
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: db-0-snap-pre-migration
spec:
volumeSnapshotClassName: fast-ssd-snapshots
source:
persistentVolumeClaimName: data-db-0
---
# restoring a snapshot means creating a brand-new PVC that points at it —
# there is no "restore in place" operation, only "clone into a new volume"
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-db-0-restored
spec:
storageClassName: fast-ssd
dataSource:
name: db-0-snap-pre-migration
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes: ["ReadWriteOnce"]
resources: { requests: { storage: 20Gi } }Restoring never edits an existing PVC in place — it always creates a fresh PVC whose dataSource points at the snapshot (or, via the same field, directly at another PVC for a live clone), and the CSI driver provisions a new volume pre-populated from that source. That's deliberate: it means a bad restore never destroys the volume you were trying to recover from, and it means you can restore the same snapshot into several new PVCs at once — handy for spinning up a scratch copy of production data to debug against without touching production at all.
A CSI volume snapshot is crash-consistent, not application-consistent, unless something upstream quiesces writes first. It captures the disk exactly as if the machine had lost power at that instant — fine for a filesystem, but a database mid-transaction can come back needing its own crash-recovery process on next start, the same as after a real power loss. It's also not a backup in the disaster-recovery sense: most CSI snapshots live in the same cloud account, often the same region, as the source disk, so they don't protect against an account compromise or a regional outage. Velero builds on exactly this CSI snapshot mechanism for its own snapshot-based backups, but adds the two things raw CSI snapshots don't give you on their own: off-cluster (and optionally cross-region) storage, and namespace/cluster-level restore that reconstructs the Kubernetes objects around the data, not just the disk contents.
On a cluster with a snapshot-capable CSI driver installed, create a small PVC, write a file to it from a throwaway Pod, then create a VolumeSnapshot of that PVC. Delete the original PVC entirely. Now create a new PVC with dataSource pointing at the snapshot, mount it from a fresh Pod, and confirm your file is still there — even though the volume it originally lived on no longer exists. Then check kubectl get volumesnapshotcontent and trace which object actually held the real, driver-side snapshot the whole time.
Benny: My PVC's been Pending for five minutes and I don't even know which of the four RPCs it's stuck on.
Foxy: Start with kubectl describe pvc — the events usually name the exact stage. Then check whether a VolumeAttachment even exists yet for it.
Gizmo the Gremlin: Or — hot take — just delete the PVC and make a new one. Fresh start, no debugging required! 🎉
Timmy the Turtle: If that PVC is bound to a StatefulSet ordinal holding real data, Gizmo, "fresh start" means the next Pod comes back with an empty disk instead of its own history. Diagnose the stuck RPC first — deleting is not a debugging step.
Ellie the Elephant: And if you're not sure whether it's worth saving, check the retention policy before you touch anything. I've seen "just delete it" turn into "we lost six months of metrics" more than once.
Professor Owl: Notice it's the same shape as everything else in this course — a desired-state object, a controller watching it, and a sequence of steps that has to run in order. Storage just has more steps, and a much less forgiving failure mode when you skip one.
Storage is the one area of Kubernetes where "just delete and recreate" — usually a perfectly safe reflex for a stuck Deployment — can genuinely destroy something you can't get back. Before you touch a production PVC, StatefulSet, or VolumeAttachment, know which RPC it's actually stuck on. Once the mechanics here feel solid, Stateful Workloads & Database Operators is the natural next stop for running a real database on top of everything covered here, and the exam blueprint's storage domain is worth a second pass now that the objects it tests aren't black boxes anymore.
1. Why did Kubernetes move storage drivers out of core and into CSI, and what does a CSIDriver object's attachRequired: false tell you about which RPC never runs for that driver? 2. Put the four provisioning-and-mount RPCs in order, and say which two run on the control plane versus which two run on the node. 3. What two mechanisms does a StatefulSet pair together to give a replica a way to "come back to itself" after a crash? 4. What does persistentVolumeClaimRetentionPolicy.whenScaled: Delete actually do, and why might that be the wrong default for a database? 5. Why is restoring a VolumeSnapshot always "create a new PVC," never "restore this PVC in place"? 6. What's the practical difference between a CSI volume snapshot and a Velero backup?
Check your answers
- In-tree drivers required a core Kubernetes release for every fix or new backend, and coupled vendor bugs to the project's own release cycle; CSI moves drivers into independently shippable containers implementing a fixed gRPC contract.
attachRequired: falsemeansControllerPublishVolume/ControllerUnpublishVolumeare never called for that driver — there's no block device to attach, so attach-related debugging steps don't apply. CreateVolume→ControllerPublishVolume(both control plane, watching PVC/VolumeAttachment objects and calling the cloud API) →NodeStageVolume→NodePublishVolume(both on the node, called directly by kubelet over the driver's Unix socket).- Stable network identity (an ordinal Pod name plus a headless Service giving each ordinal its own DNS record) paired with stable storage (a dedicated PVC per ordinal via
volumeClaimTemplates, rebound to the same ordinal every time it's recreated). - It deletes that ordinal's PVC — and the real disk behind it, if the reclaim policy is Delete — the moment the StatefulSet is scaled down past that ordinal. That's fine for disposable state like a cache, but for a database replica it means scaling down and back up loses that replica's data instead of resyncing an existing copy.
- Because restoring in place would risk destroying the very data you're trying to recover if anything went wrong mid-restore; creating a fresh PVC with
dataSourceset means the source snapshot is untouched no matter what happens, and the same snapshot can be restored into multiple new PVCs at once. - A CSI snapshot is a driver-level, usually same-account, same-region point-in-time copy of one disk — crash-consistent, not necessarily application-consistent. Velero uses that same CSI snapshot mechanism underneath but adds off-cluster (and often cross-region) storage and whole-namespace/cluster restore that reconstructs the surrounding Kubernetes objects, not just the raw disk contents.