Storage & Stateful Workloads

Storage is where the comforting fiction of “cattle, not pets” finally breaks down. A stateless web pod can die and be reborn on any node in the cluster and nobody notices. A database pod cannot: it carries the one copy of data the business actually cares about, and that data has weight, an identity, and an order it must come up in. This is the deep dive into running state on Kubernetes — the volume model and CSI beneath it, StatefulSets, the operators that turn a raw StatefulSet into a real database, backup & disaster recovery, and how a platform offers all of this as a one-click, self-service capability without letting anyone lose a byte.

Why state is hard on Kubernetes

☺ Like you’re 10: Kubernetes was built to shuffle apps around like playing cards. That works great — right up until one of the cards is holding something heavy it can’t put down.

Kubernetes earned its reputation on stateless workloads, and its whole scheduler is optimised for them: pack pods onto whatever nodes have room, kill and reschedule freely, scale horizontally by cloning. State violates almost every one of those assumptions at once. Before you touch a single manifest, it’s worth being precise about why — because every mechanism later in this page exists to answer one of these three problems.

The stateless assumption: pods are disposable, disks are not

A container’s root filesystem is ephemeral. When a pod is deleted, rescheduled, or its node dies, that filesystem is gone — by design. For a stateless service that’s a feature: no snowflakes, no drift, every replica identical. But it means anything a pod writes to its own filesystem is throwaway. The moment you need data to outlive a single pod instance, you must push that data outside the pod’s lifecycle onto a durable volume that the cluster tracks separately. Kubernetes deliberately keeps “the pod” and “the disk” as two independent objects so that one can be destroyed and recreated while the other persists.

Identity & ordering: some replicas are not interchangeable

A Deployment gives you N anonymous, interchangeable replicas with random names like web-7d9f-xq2k. That is exactly wrong for clustered data systems. A PostgreSQL primary and its replicas are not interchangeable — one accepts writes, the others follow it. A Kafka broker needs a stable identity so partitions can be reliably assigned to it. A ZooKeeper or etcd member votes in a quorum and must be individually addressable. These systems need stable identity (the same name and the same DNS record every time a pod restarts) and often a predictable start order (bring up the first member, let it form a cluster, then join the rest). Deployments provide neither; that gap is the entire reason StatefulSet exists.

Data gravity: compute must move to the data

Data has mass. A stateless pod can be scheduled anywhere because it carries nothing; a stateful pod is chained to the specific disk that holds its data, and that disk usually lives in exactly one availability zone. This is data gravity: the larger a dataset grows, the more everything else — the compute, the network, the backups — must orbit it, because moving terabytes is slow and expensive while moving a pod is instant and free. On Kubernetes this shows up as a hard scheduling constraint: a pod that uses a zonal block volume can only run on a node in that volume’s zone. Get the ordering wrong and the scheduler paints itself into a corner — which is why the binding rules in the next section matter so much.

The Kubernetes storage model

☺ Like you’re 10: A claim is a request slip: “I need a 10-gigabyte fast disk.” A volume is the real disk that shows up to satisfy the slip. A StorageClass is the vending machine that makes a matching disk on demand.

Kubernetes storage is a supply-and-demand market with a deliberate wall between the two sides. Developers express demand; the platform (and its drivers) provide supply; the control plane matches them. Understanding the four objects — and how they bind — is the foundation everything else rests on.

Volumes, PersistentVolumes & PersistentVolumeClaims

A plain Volume is scoped to a pod’s lifetime — an emptyDir for scratch space, a configMap or secret projected as files, an ephemeral CSI volume. Useful, but it dies with the pod. For durable storage there are two paired objects. A PersistentVolume (PV) is a cluster-scoped piece of real storage — an EBS volume, a Ceph RBD image, an NFS export — with its own independent lifecycle. A PersistentVolumeClaim (PVC) is a namespaced request for storage: “I want 20 Gi, ReadWriteOnce, from the fast-ssd class.” The control plane binds a PVC to a satisfying PV, and the pod mounts the claim, never the PV directly. That indirection is the whole point: developers write portable manifests that ask for storage, while the messy provider-specific details live in the PV and the driver.

StorageClass & dynamic provisioning

In the old world an administrator pre-created PVs by hand and hoped a claim would match — static provisioning, which does not scale. A StorageClass flips it to dynamic: the class names a provisioner (a CSI driver such as ebs.csi.aws.com or pd.csi.storage.gke.io) plus a bag of parameters (disk type, IOPS, filesystem, encryption). When a PVC references that class and no PV exists, the provisioner mints a brand-new volume on the fly and creates the PV to represent it. A platform typically publishes a small menu of classes — standard, fast-ssd, shared-file — as a curated golden path, and marks one as the cluster default.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "6000"
  throughput: "500"          # MiB/s
  encrypted: "true"
allowVolumeExpansion: true     # PVCs of this class can be grown online
reclaimPolicy: Delete          # delete the real disk when the PVC is deleted
volumeBindingMode: WaitForFirstConsumer   # bind only after a pod is scheduled
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pgdata
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 20Gi

Access modes: RWO, ROX, RWX, RWOP

Access modes describe how many nodes can mount a volume and in which direction. The subtlety that trips up even experienced engineers: the classic modes are counted per node, not per pod.

Two practical rules fall out of this. First, most block storage is RWO, so you cannot have three replicas of a Deployment all writing to the same block PVC across nodes — you need RWX (a network filesystem) for that, or a design where each replica owns its own volume. Second, RWO historically let two pods on the same node both open the volume, which can corrupt data systems that assume a single writer; ReadWriteOncePod (stable since Kubernetes 1.29) closes that hole by guaranteeing exactly one pod cluster-wide — the correct choice for a primary database.

Binding, topology & reclaim policies

Two fields decide the timing and the afterlife of a volume. volumeBindingMode: Immediate provisions the disk the instant the PVC is created — dangerous for zonal storage, because the disk may land in us-east-1a while the only spare capacity for the pod is in 1b, and now the pod is unschedulable forever. WaitForFirstConsumer delays provisioning until a pod actually references the claim, letting the scheduler pick a node first so the volume is created in the right zone. On any cluster with zonal block storage this should be the default. Meanwhile the reclaim policy governs what happens when the PVC is deleted: Delete (the dynamic default) destroys the underlying disk — convenient and dangerous; Retain keeps the disk and its data for manual recovery, releasing the PV into a Released state you clean up deliberately. For anything holding real data, Retain plus a real backup story is the grown-up choice.

CSI — the Container Storage Interface

☺ Like you’re 10: CSI is a standard-shaped plug. Any storage vendor who builds a plug of that shape works with Kubernetes, without Kubernetes needing to know anything about their storage.

Originally, storage drivers lived inside the Kubernetes source tree (“in-tree”), so adding support for a new storage system meant patching and releasing Kubernetes itself. The Container Storage Interface (CSI) ended that: it’s a vendor-neutral gRPC contract that lets storage providers ship out-of-tree drivers on their own schedule. Every modern volume — cloud disks, Ceph, Portworx, Longhorn, NetApp — is a CSI driver, and understanding the driver’s anatomy demystifies almost every “my volume is stuck attaching” incident.

The three plugin roles: identity, controller, node

A CSI driver implements three gRPC services, and Kubernetes runs them in two deployment shapes. The controller plugin (usually a Deployment, one active replica) handles cluster-wide operations — creating and deleting volumes, attaching them to nodes, taking snapshots. It runs the vendor’s driver container next to a set of Kubernetes-maintained sidecars that translate Kubernetes objects into CSI calls: external-provisioner watches PVCs and calls CreateVolume; external-attacher watches VolumeAttachment objects and calls ControllerPublishVolume; external-snapshotter and external-resizer handle snapshots and expansion. The node plugin runs as a DaemonSet on every node, next to a node-driver-registrar sidecar; when a pod lands, kubelet calls the node plugin’s NodeStageVolume and NodePublishVolume to format and mount the disk into the pod. The identity service simply reports the driver’s name and capabilities.

Snapshots & clones

CSI standardised point-in-time snapshots as first-class objects. You define a VolumeSnapshotClass (which driver, retention policy), then create a namespaced VolumeSnapshot pointing at a PVC; the external-snapshotter asks the driver to snapshot the volume and records a cluster-scoped VolumeSnapshotContent. To restore, you create a new PVC whose dataSource is that snapshot — the driver provisions a fresh volume pre-loaded with the snapshot’s data. The same mechanism, with a PVC as the dataSource instead of a snapshot, gives you volume cloning — handy for spinning a realistic staging database off production in seconds.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: pgdata-2026-07-12
spec:
  volumeSnapshotClassName: ebs-snap
  source:
    persistentVolumeClaimName: pgdata     # snapshot this live PVC
---
# Restore: a new PVC seeded from the snapshot
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pgdata-restored
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: fast-ssd
  dataSource:
    name: pgdata-2026-07-12
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  resources:
    requests:
      storage: 20Gi

Volume expansion & topology

Two more CSI capabilities matter for real platforms. Online expansion: if the StorageClass sets allowVolumeExpansion: true, you grow a volume simply by editing the PVC’s requested size upward — the external-resizer expands the backing disk and, for most filesystems, resizes it live without a restart. (You cannot shrink.) Topology: a CSI driver advertises which zones it can serve, and the scheduler uses that so a pod and its volume land together. This is the machinery that makes WaitForFirstConsumer work, and it’s why a well-built platform can offer zonal block storage and keep pods schedulable — a theme we return to under scheduling and the cluster substrate.

StatefulSets in depth

☺ Like you’re 10: A StatefulSet is like assigning kids permanent numbered lockers. Kid 0 always gets locker 0, and their stuff stays in it even if the kid goes home and comes back tomorrow.

The StatefulSet is Kubernetes’ answer to identity and ordering. It looks like a Deployment but adds three guarantees that clustered data systems depend on: stable names, stable per-replica storage, and controlled ordering.

Stable network identity & the headless Service

StatefulSet pods are named by ordinal: db-0, db-1, db-2 — and that name is sticky. If db-1 is deleted, its replacement is also called db-1, reattaches the same storage, and answers to the same DNS. That DNS comes from a headless Service (a Service with clusterIP: None), which is required and named in the StatefulSet’s serviceName. Instead of load-balancing behind one virtual IP, a headless Service publishes a per-pod DNS record: db-0.db.prod.svc.cluster.local. Now a replica can find the primary by a stable name rather than a random IP — exactly what Patroni, Kafka, or a Mongo replica set need to form a cluster. (More on how Services and DNS work in networking.)

volumeClaimTemplates: one durable disk per replica

A Deployment’s replicas share nothing; a StatefulSet’s replicas each need their own data. The volumeClaimTemplates field is a stamp: the controller creates one PVC per pod, named <template>-<statefulset>-<ordinal> — e.g. data-db-0, data-db-1. That PVC is sticky to the ordinal: reschedule db-0 to another node and it reattaches data-db-0, so the data follows the identity. Crucially, these PVCs are not deleted when you scale down or delete the StatefulSet — a deliberate safety default so a mistaken scale-down can’t vaporise your data. Since Kubernetes 1.27+, a persistentVolumeClaimRetentionPolicy lets you opt into deletion on whenScaled or whenDeleted if you truly want it.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db                 # MUST match a headless Service
  replicas: 3
  podManagementPolicy: OrderedReady   # 0 → 1 → 2, one at a time
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0                # bump to canary only the highest ordinals
  selector:
    matchLabels: { app: db }
  template:
    metadata:
      labels: { app: db }
    spec:
      containers:
        - name: postgres
          image: postgres:17
          ports: [{ containerPort: 5432, name: pg }]
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:           # one sticky PVC per replica
    - metadata: { name: data }
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-ssd
        resources: { requests: { storage: 20Gi } }
---
apiVersion: v1
kind: Service
metadata:
  name: db
spec:
  clusterIP: None                 # headless
  selector: { app: db }
  ports: [{ port: 5432, name: pg }]

Ordered & partitioned rollouts

By default (podManagementPolicy: OrderedReady) the controller creates pods in order — db-1 is not created until db-0 is Running and Ready — and scales down in reverse (db-2 first). That lets a clustered system bootstrap its leader before followers join. Set Parallel when your app doesn’t care about order and you want faster scaling. Updates get the same care: with updateStrategy: RollingUpdate and a partition of k, only pods with ordinal ≥ k are updated — a built-in canary. Set partition: 2 on a 3-replica set and only db-2 takes the new version; verify it, then drop the partition to 0 to roll the rest. OnDelete hands you full manual control, updating a pod only when you delete it. These knobs are why StatefulSets can update a quorum system without ever losing it.

Running databases on Kubernetes: the operator pattern for data

☺ Like you’re 10: A StatefulSet gives a database a body — stable name, its own disk. An operator gives it a brain: it knows how to fail over, take backups, and heal, the way a human DBA would.

A raw StatefulSet handles identity and storage, but it doesn’t know what a database is. It won’t promote a replica when the primary dies, won’t stream backups to object storage, won’t reconfigure connection routing after a failover. That domain knowledge is exactly what an operator encodes — a custom controller that watches a Custom Resource like Cluster or Kafka and continuously reconciles a full, production-grade data system. This is where “databases on Kubernetes” went from risky to routine.

Why a StatefulSet alone isn’t enough

Consider a Postgres primary that crashes. A StatefulSet will faithfully recreate db-0 and reattach its disk — but if that disk or its zone is gone, you have an outage until a human intervenes. A real database needs automated failover (elect a replica, promote it, repoint clients), continuous backup (base backups plus WAL/binlog archiving for point-in-time recovery), connection routing (a stable endpoint that always points at the current primary), and safe rolling upgrades that respect replication. Encoding all of that as a controller — so it happens in seconds, unattended, the same way every time — is the operator’s job. It is the reconciliation loop applied to a database.

The operator landscape

The pattern is identical across all of them: a small, expressive Custom Resource captures intent (“a 3-node HA Postgres with nightly backups to this bucket”), and the operator does the rest. Here is CloudNativePG — note how much production behaviour is compressed into a few lines:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: orders-db
spec:
  instances: 3                    # 1 primary + 2 streaming replicas, auto-failover
  storage:
    size: 50Gi
    storageClass: fast-ssd
  postgresql:
    parameters:
      max_connections: "200"
      shared_buffers: 512MB
  backup:
    barmanObjectStore:
      destinationPath: s3://acme-db-backups/orders
      s3Credentials:
        accessKeyId:     { name: backup-creds, key: ACCESS_KEY_ID }
        secretAccessKey: { name: backup-creds, key: SECRET_ACCESS_KEY }
      wal:
        compression: gzip         # continuous WAL archiving → point-in-time recovery
    retentionPolicy: "30d"
  bootstrap:
    initdb:
      database: orders
      owner: orders_app

Notice the backup credentials come from a Secret, never inline — feed them from the secrets pipeline, not a committed file.

Self-hosted vs a managed cloud database

The most senior decision here is whether to run the database at all. Running Postgres yourself with a mature operator is entirely viable and gives you portability, cost control at scale, and a single Kubernetes-native control plane. But a managed service — RDS/Aurora, Cloud SQL, Azure Database — hands the pager for storage durability, patching, and failover to the cloud provider. Reach for managed when the database is mission-critical, your team is small, and undifferentiated heavy lifting is a distraction; reach for self-hosted on Kubernetes when you need multi-cloud portability, have genuine scale that makes managed pricing painful, want everything under one GitOps workflow, or run on-prem/edge where no managed option exists. A strong platform often offers both behind one self-service API, so the choice becomes a field in a form rather than a re-architecture — which is exactly the trick the next section pulls off.

Data protection: backup & disaster recovery

☺ Like you’re 10: A snapshot is a photo of your disk. A backup is that photo kept in a different building, that you’ve actually checked you can print again. A backup you’ve never restored isn’t a backup — it’s a wish.

Storage durability is not backup. A replicated volume protects against a disk failing; it does nothing against a bad migration, a fat-fingered DROP TABLE, ransomware, or an accidental namespace deletion — those replicate the damage instantly. Real data protection is a separate, tested discipline, and the exam-beyond skill here is knowing the layers and their guarantees.

Snapshots vs backups: consistency matters

A CSI volume snapshot is fast and cheap, but by default it’s crash-consistent: it captures the disk exactly as it is, including half-written buffers, as if the machine lost power. Most databases recover from that via their write-ahead log — but not always cleanly. An application-consistent backup quiesces the database first (flush buffers, briefly freeze writes, or use the DB’s native backup command) so the captured state is a valid transaction boundary. That’s why the gold standard for databases is the operator’s own backup path (base backup + continuous WAL/binlog archiving), which is application-consistent and enables point-in-time recovery — restoring to 14:32:57, just before the bad migration. Snapshots are a great fast layer underneath that, not a replacement for it.

Velero: backing up the cluster, not just the disks

Velero is the standard tool for cluster-level backup and migration. It does two things at once: it backs up Kubernetes objects (your Deployments, Services, ConfigMaps, PVCs — the desired state) to object storage, and it backs up the volume data either via CSI snapshots or its built-in File System Backup (Kopia/restic). You can scope backups by namespace or label, run them on a schedule, and — critically — restore into a different cluster, which is what makes Velero a disaster-recovery and cluster-migration tool, not just a backup tool. For application consistency, Velero runs backup hooks — commands inside the pod (e.g. pg_backup_start, or an fsfreeze) before and after the snapshot.

# A nightly, application-consistent backup of the "orders" namespace,
# including volume data via CSI snapshots, kept for 30 days.
velero schedule create orders-nightly \
  --schedule="0 2 * * *" \
  --include-namespaces orders \
  --snapshot-volumes \
  --ttl 720h0m0s

# Disaster recovery: restore that namespace into a *different* cluster.
# (kubectl context now points at the DR cluster)
velero restore create --from-schedule orders-nightly \
  --include-namespaces orders

# Trust, but verify — inspect what actually got captured.
velero backup describe orders-nightly-20260712020000 --details

RPO, RTO & cross-region recovery

Two numbers turn “we have backups” into an actual strategy. RPO (Recovery Point Objective) is how much data you can afford to lose — the gap between your last recoverable point and the failure. RTO (Recovery Time Objective) is how long you can afford to be down. They drive the design, and they cost money in proportion to how small they are:

The part everyone skips is the drill. A backup that has never been restored is a liability disguised as safety. Cross-region and cross-cluster DR — restoring into a standby cluster in another region, or failing over to a synchronous replica — must be rehearsed on a schedule, timed against your RTO, and folded into your incident and reliability practice. Ellie never forgets a byte precisely because she tests that she can get it back.

Storage as a self-service platform capability

☺ Like you’re 10: Instead of Dot filing a ticket and waiting three days for a database, she clicks one button and a robot builds a real, backed-up database in two minutes.

Everything so far — StorageClasses, StatefulSets, operators, backup schedules — is plumbing. The platform-engineering move is to hide that plumbing behind a small, safe, self-service API so a developer gets a production-grade database without learning any of it. This is where storage stops being an ops burden and becomes a golden path.

Databases-as-a-service via operators

Because a database operator already turns a Custom Resource into a running cluster, you can expose that CR (or an even simpler wrapper around it) as the product. Publish a curated Cluster template with sane defaults — HA, encrypted storage, nightly backups, right-sized resources — and “get a database” becomes “add one small YAML file to your app’s repo,” reconciled by GitOps. The platform team owns the defaults and guardrails (backups are always on; storage class is always encrypted); the developer owns only the choices that matter to them (size, database name).

Crossplane claims for buckets & managed databases

To offer a uniform interface across both self-hosted and cloud databases — or to provision object-storage buckets, cloud caches, and queues — reach for Crossplane. You author a Composition that maps a friendly, namespaced claim to whatever really fulfils it (a CloudNativePG Cluster in one environment, an RDS instance in another), and developers only ever see the claim. The abstraction hides the provider entirely: Dot asks for a PostgresInstance and doesn’t know or care whether she got a pod or an Aurora endpoint.

# What a developer writes — a claim, in their own namespace.
# A Crossplane Composition decides how it's really fulfilled
# (CloudNativePG in dev, RDS in prod) and returns a connection Secret.
apiVersion: platform.acme.io/v1alpha1
kind: PostgresInstance
metadata:
  name: orders
spec:
  size: small          # → 2 vCPU / 8Gi, 50Gi fast-ssd  (platform decides the mapping)
  highAvailability: true
  backup:
    retentionDays: 30  # backups are mandatory; you may only tune retention
  writeConnectionSecretToRef:
    name: orders-db-conn   # the platform drops host/user/password here

Performance, cost & data gravity

☺ Like you’re 10: Fast disks cost more, and moving your data between cities costs money every time. So you keep the compute right next to the data — and you don’t pay for a race car when a bicycle will do.

Storage is where cloud bills quietly balloon and where the wrong performance choice silently throttles an application. The platform’s job is to expose the right tiers as named StorageClasses and to make the expensive choices visible — tying straight into FinOps.

IOPS & throughput: pick the class that fits the workload

Block storage is sold on three axes — capacity (GiB), IOPS (operations/second, which limits small-random workloads like a transactional database), and throughput (MiB/s, which limits large-sequential workloads like log ingestion or analytics scans). A platform models these as a small menu of classes so developers choose intent, not raw numbers:

Local vs network storage

The sharpest trade-off is local versus network storage. Local NVMe on the node is dramatically faster — no network hop — but it’s ephemeral in the worst way: if the node dies, the data is gone, and the pod cannot reschedule elsewhere because the disk was physically attached to that machine. Network block storage (EBS, PD) survives node failure and follows the pod to a new node in the same zone, at the cost of latency and IOPS caps. The rule of thumb: use network storage by default for anything whose only copy lives there; use local storage only when the application itself replicates data across pods (Kafka with replication factor 3, a sharded database, a distributed cache) so losing one node’s disk is survivable by design. Getting this wrong is how teams either burn money on provisioned IOPS they don’t need, or lose data to a “fast” local disk that was never meant to be durable.

The cost of data gravity: storage & egress

Data gravity has a line item. Storage itself accrues cost per GiB-month — and so do the snapshots and backups you (correctly) keep, which is why retention policy is a cost lever, not just a safety one. But the sneakiest charge is egress: moving data out of a zone, region, or cloud is billed per gigabyte, and cross-AZ replication traffic for a chatty database can quietly dwarf the storage bill. This is data gravity restated as economics: it’s cheap to keep data still and expensive to move it, so architect to keep compute next to its data, replicate deliberately, and think hard before a design shuttles terabytes across regions. Surface these costs through cost visibility (OpenCost can attribute PV and snapshot spend per namespace) so teams see the price of their storage choices — and fold the durability guarantees into your architecture and reliability planning rather than discovering them on the invoice.

State is the part of the platform where mistakes are permanent, so it’s the part where the platform’s guardrails earn their keep the most. Get the model right — durable volumes decoupled from pods, StatefulSets for identity, operators for the DBA brain, tested backups for the “oh no” moments, and a self-service front door so none of it lands on a developer — and you’ve tamed the hardest thing on the substrate. From here, it pairs naturally with policy guardrails that keep secrets out of Git and encryption always on.