Capstone Part 4 — Storage & Stateful Apps

This is Part 4 of the five-part capstone running through this whole course: one continuously evolving cluster and app, carried from a bare kind cluster in Part 1 through a running checkout-api Deployment in Part 2 and a reachable Service and Ingress in Part 3. Every Pod in that story so far has been disposable on purpose — kill any checkout-api Pod and the Deployment simply replaces it, no state lost because there was none to lose. This part introduces the first workload that can't get away with that: checkout-db, a StatefulSet-backed Postgres instance with its own PersistentVolumeClaim, wired to the app you already have running. By the end you'll have deleted a database Pod on purpose, watched Kubernetes bring it back, and proven — not assumed — that the data survived, plus a CronJob that backs it up somewhere the database's own disk can't take it. Part 5 locks down who's allowed to touch any of it.

What you're building on, and what this part adds

☺ Like you're 10: The cluster and the app already exist and already talk to the outside world — this part just gives the app somewhere to keep its memory.

This page assumes Part 1, Part 2, and Part 3 are all behind you: a kind cluster named checkout with one control-plane node and two workers; a checkout namespace; a checkout-api Deployment reading its configuration from a checkout-api-config ConfigMap and a checkout-api-secrets Secret, in the exact shape the Workloads & Scheduling blueprint page walks through; and a checkout-api Service fronted by an ingress-nginx Ingress, reachable at a local hostname. Every one of those objects is stateless by construction — kubectl rollout restart deployment/checkout-api is a completely safe thing to run at any time, because there was never anything inside a checkout-api Pod worth keeping. This page adds the first object in the whole capstone where that stops being true:

Everything below assumes the object-model and PVC/PV mechanics from The object model and the CKA blueprint's Storage domain — StorageClasses, access modes, reclaim policy, and how a claim actually binds. This page puts those objects to work on a real (if small) database instead of re-deriving them from scratch; read Storage & the CSI first if volumeClaimTemplates or "stable per-replica storage" are unfamiliar phrases.

What kind's default StorageClass actually gives you

☺ Like you're 10: Ask for storage and one small robot on the exact same node quietly carves out a folder on that node's own disk and hands it over.

Nothing in Parts 1–3 ever asked for a PersistentVolume, so this is the first page in the capstone where kind's bundled storage driver actually does anything. Every kind cluster ships with Rancher's local-path-provisioner pre-installed, exposed as a StorageClass named standard and marked default — no separate install step, no cloud account, nothing to provision by hand:

$ kubectl get storageclass
NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      DEFAULT
standard (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   true

rancher.io/local-path is deliberately simple: when a PVC using this class is claimed, the provisioner creates a plain directory on one specific node's local filesystem and backs the PersistentVolume with it — no network storage layer, no CSI driver attach/detach dance, just a folder. volumeBindingMode: WaitForFirstConsumer is exactly the behavior the Storage blueprint page calls the exam's default-correct answer, and here it matters for a very concrete reason: the provisioner can't pick which node's disk to use until it knows which node the Pod claiming the volume actually lands on. That single design choice — a local folder, pinned to whichever node happened to get the Pod first — is the reason the reschedule drill further down this page behaves the way it does, and it's worth understanding now rather than being surprised by it later.

The StatefulSet: one replica, one stable disk

☺ Like you're 10: Instead of a Deployment's identical, interchangeable copies, a StatefulSet hands out numbered nametags — and whichever Pod is wearing nametag zero always gets drawer zero back.

checkout-api's Deployment gives every replica an interchangeable, randomly-suffixed name because none of them need to be told apart. A StatefulSet exists for the opposite case: checkout-db-0 keeps that exact name across every restart, and its volumeClaimTemplates entry stamps out a PersistentVolumeClaim per ordinal — data-checkout-db-0 — that the same Pod identity reattaches to every time, rather than a fresh empty volume. A StatefulSet also needs a headless Service (clusterIP: None) to exist first; that's what gives each ordinal its own resolvable DNS name instead of one shared, load-balanced address:

# checkout-db-headless-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: checkout-db
  namespace: checkout
spec:
  clusterIP: None            # headless — required by StatefulSet's serviceName
  selector:
    app: checkout-db
  ports:
    - name: postgres
      port: 5432

# checkout-db-secret.yaml — Part 5 replaces this plaintext Secret with something properly managed
apiVersion: v1
kind: Secret
metadata:
  name: checkout-db-secrets
  namespace: checkout
type: Opaque
stringData:
  POSTGRES_PASSWORD: "capstone-local-only"

# checkout-db-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: checkout-db
  namespace: checkout
spec:
  serviceName: checkout-db      # must match the headless Service above
  replicas: 1
  selector:
    matchLabels: { app: checkout-db }
  template:
    metadata:
      labels: { app: checkout-db }
    spec:
      containers:
        - name: postgres
          image: postgres:16.4
          ports:
            - containerPort: 5432
              name: postgres
          env:
            - name: POSTGRES_DB
              value: checkout
            - name: POSTGRES_USER
              value: checkout
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef: { name: checkout-db-secrets, key: POSTGRES_PASSWORD }
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata   # see the subPath note below
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
          readinessProbe:
            exec: { command: ["pg_isready", "-U", "checkout"] }
            periodSeconds: 5
          livenessProbe:
            exec: { command: ["pg_isready", "-U", "checkout"] }
            initialDelaySeconds: 30
            periodSeconds: 10
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain           # deleting the StatefulSet must NOT delete the data
    whenScaled: Retain
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: standard
        resources:
          requests:
            storage: 5Gi

Two details are deliberate, not incidental. PGDATA is set one directory level below the mount point instead of at the mount root — Postgres refuses to initialize a data directory that already contains a filesystem's own reserved entries (like lost+found on a fresh ext4 volume), and a nested pgdata subdirectory sidesteps that entirely without needing a separate subPath volume mount. And persistentVolumeClaimRetentionPolicy is set explicitly rather than left at its default — as the Storage blueprint page covers, StatefulSet PVCs already outlive Pod deletion and scale-down by default, but being explicit here says out loud that deleting checkout-db itself, on purpose or by accident, must never be the thing that deletes the data too.

$ kubectl apply -f checkout-db-headless-svc.yaml -f checkout-db-secret.yaml -f checkout-db-statefulset.yaml
$ kubectl get pods -n checkout -l app=checkout-db -w
NAME             READY   STATUS    RESTARTS   AGE
checkout-db-0    1/1     Running   0          38s

$ kubectl get pvc -n checkout
NAME                    STATUS   VOLUME       CAPACITY   ACCESS MODES   STORAGECLASS
data-checkout-db-0      Bound    pvc-7a3f...  5Gi        RWO            standard

Wiring checkout-api to its new database

☺ Like you're 10: The app just needs one address to knock on — the StatefulSet's own headless Service already hands out a name that always means "the current Pod zero."

The headless Service above gives checkout-db-0 a resolvable DNS name in the shape every StatefulSet Pod gets: <pod-name>.<service-name>.<namespace>.svc.cluster.local. Patch that address straight into the same checkout-api-config ConfigMap Part 2 already created, and restart the Deployment so every Pod picks up the new value — a plain ConfigMap update never restarts anything on its own, exactly the gotcha the Workloads & Scheduling blueprint page already warned about:

$ kubectl patch configmap checkout-api-config -n checkout --type merge \
    -p '{"data":{"DB_HOST":"checkout-db-0.checkout-db.checkout.svc.cluster.local","DB_PORT":"5432"}}'

$ kubectl rollout restart deployment/checkout-api -n checkout
$ kubectl rollout status deployment/checkout-api -n checkout

$ kubectl exec -it deploy/checkout-api -n checkout -- curl -s localhost:8080/healthz/db
{"db":"connected"}

Proving state survives a reschedule

☺ Like you're 10: Write something down, knock the whole desk over on purpose, and check the notebook is still readable once the desk gets put back.

Don't take a StatefulSet's promise on faith — write a row, delete the Pod that wrote it, and read the row back once Kubernetes brings the Pod back. This is the entire point of this section, and it takes about ninety seconds:

# write something real, and note which node the Pod is currently on
$ kubectl exec -it checkout-db-0 -n checkout -- psql -U checkout -d checkout -c \
    "CREATE TABLE IF NOT EXISTS orders (id serial primary key, note text);
     INSERT INTO orders (note) VALUES ('part-4-proof');"
$ kubectl get pod checkout-db-0 -n checkout -o jsonpath='{.spec.nodeName}{"\n"}'
checkout-worker

# delete the Pod on purpose — not the StatefulSet, not the PVC
$ kubectl delete pod checkout-db-0 -n checkout
pod "checkout-db-0" deleted

$ kubectl get pod checkout-db-0 -n checkout -w
NAME             READY   STATUS              RESTARTS   AGE
checkout-db-0    0/1     ContainerCreating   0          4s
checkout-db-0    1/1     Running             0          11s

# same claim, same node, same row
$ kubectl get pvc data-checkout-db-0 -n checkout
$ kubectl get pod checkout-db-0 -n checkout -o jsonpath='{.spec.nodeName}{"\n"}'
checkout-worker
$ kubectl exec -it checkout-db-0 -n checkout -- psql -U checkout -d checkout -c "SELECT * FROM orders;"
 id |     note
----+---------------
  1 | part-4-proof

Nothing about that sequence is a coincidence. The StatefulSet controller — the same reconcile-loop pattern the controller pattern page covers in general — notices checkout-db-0 is gone and recreates a Pod with that exact name, which the Storage blueprint page's binding rules then match straight back to the one PVC already named data-checkout-db-0: no new claim gets created, because one with that name already exists and is already bound. And the node stayed the same for a reason worth naming directly, not glossing over.

A backup CronJob: a second, independent volume

☺ Like you're 10: The filing cabinet surviving a Pod restart is not the same thing as the filing cabinet surviving the whole room burning down — for that you need a photocopy living somewhere else entirely.

The drill above proves checkout-db-0's data survives a Pod being deleted. It proves nothing about the PVC itself being lost — a node failure, an accidental kubectl delete pvc, or the whole kind cluster being torn down all take data-checkout-db-0 with them. A backup has to live on different storage, so its failure is independent of the thing it's backing up. Create a second PVC that has nothing to do with the StatefulSet at all, and a CronJob that periodically dumps into it:

# checkout-db-backups-pvc.yaml — a completely separate volume from checkout-db-0's own disk
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: checkout-db-backups
  namespace: checkout
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: standard
  resources:
    requests:
      storage: 2Gi

# checkout-db-backup-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: checkout-db-backup
  namespace: checkout
spec:
  schedule: "0 3 * * *"          # 03:00 daily — the cluster's clock, not yours
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: pg-dump
              image: postgres:16.4
              command: ["/bin/sh", "-c"]
              args:
                - >
                  pg_dump -h checkout-db-0.checkout-db.checkout.svc.cluster.local
                  -U checkout -d checkout -F c
                  -f /backups/checkout-$(date +%Y%m%d-%H%M%S).dump
              env:
                - name: PGPASSWORD
                  valueFrom:
                    secretKeyRef: { name: checkout-db-secrets, key: POSTGRES_PASSWORD }
              volumeMounts:
                - name: backups
                  mountPath: /backups
          volumes:
            - name: backups
              persistentVolumeClaim:
                claimName: checkout-db-backups

Don't wait until 3 a.m. to find out whether this works. Every CronJob can be triggered on demand by stamping out one ordinary Job from its template — exactly how you'd rehearse an alert firing before trusting it, applied here to a backup instead:

$ kubectl apply -f checkout-db-backups-pvc.yaml -f checkout-db-backup-cronjob.yaml
$ kubectl create job --from=cronjob/checkout-db-backup checkout-db-backup-manual -n checkout
job.batch/checkout-db-backup-manual created

$ kubectl wait --for=condition=complete job/checkout-db-backup-manual -n checkout --timeout=60s
$ kubectl run -it --rm backup-check -n checkout --image=busybox --restart=Never \
    --overrides='{"spec":{"containers":[{"name":"backup-check","image":"busybox","command":["ls","-la","/backups"],"volumeMounts":[{"name":"b","mountPath":"/backups"}]}],"volumes":[{"name":"b","persistentVolumeClaim":{"claimName":"checkout-db-backups"}}]}}'
-rw-r--r--    1 root     root         8192 checkout-20260827-030112.dump

What "done" looks like for Part 4

☺ Like you're 10: A database that remembers things even after being knocked over, plus a photocopy of everything living somewhere else entirely.

At the end of this part, checkout-db runs as a one-replica StatefulSet with its own per-ordinal PersistentVolumeClaim, provisioned dynamically from kind's standard StorageClass; checkout-api talks to it over its stable per-Pod DNS name; a deliberately separate PersistentVolumeClaim holds nightly pg_dump archives written by a CronJob you've already triggered and verified by hand; and you've personally deleted checkout-db-0 and watched its data come back rather than taking any of this on faith. Nothing here gets thrown away — Part 5 builds directly on it:

Part 4 gave checkout-db a memory that survives more than it looks like it should, and a backup that doesn't share its fate. Continue to Capstone Part 5 — Security & RBAC, where access to everything built so far finally gets locked down. Or step back to the full lab track to see how this capstone fits the rest of the hands-on labs, revisit the CKA blueprint's own Storage domain for the exam-scoped version of everything on this page, and read Storage & the CSI or Velero for what this setup deliberately left for later.

Milestones

☺ Like you're 10: Tick each box only once you've actually watched it happen on your own cluster, not because the step "sounds right."

Work these in order — each depends on the cluster state from the one before. Progress saves in this browser.