Stateful Workloads & Database Operators
Kubernetes was built around a comforting assumption: any Pod can die at any moment and nothing downstream should care, because whatever it was doing is either idempotent or about to be redone by an identical replacement. A database breaks that assumption on purpose — it has to remember something, on a specific disk, and possibly agree with other replicas about who's allowed to accept writes right now. Storage & the CSI already covers how a StatefulSet pairs stable identity with a stable per-replica volume, and Operators & Custom Resource Definitions already covers the reconcile-loop pattern an operator runs. This page is where those two ideas meet a database specifically: why a PodDisruptionBudget has to reason about quorum instead of a plain replica count, what backup strategy looks like once an operator owns the loop instead of a cron job, what a database operator has to get right that a generic one doesn't, and an honest decision framework for when running the database yourself is the wrong call entirely.
Picture a hotel with a magic front desk: any guest can be moved to any identical room in seconds, because every room holds the same furniture and nobody keeps anything personal there. That's most of Kubernetes — rooms are interchangeable. Now picture one guest who checked in with a safe bolted to the floor, files that must never exist in two rooms at once, and a rule that if that room ever burns down, only a copy someone actually checked yesterday — not one merely promised to exist — gets the files back. Moving that guest isn't "assign any free room" anymore. It's "move the safe, prove the copy still opens, and make absolutely sure two rooms never both believe they're the one holding the real files at the same time." That's a database on Kubernetes.
A StatefulSet gives you identity and storage — not a brain
☺ Like you're 10: A StatefulSet hands a database guest the same numbered room and the same safe every time it checks back in. It does not hire anyone to actually run the hotel.
Storage & the CSI covers the mechanism in full: a StatefulSet pairs a stable, ordinal-numbered identity (db-0, db-1, db-2) with a stable, per-ordinal PersistentVolumeClaim stamped from volumeClaimTemplates, creates and scales Pods in strict order, and — since Kubernetes 1.27 — lets a persistentVolumeClaimRetentionPolicy say explicitly whether a replica's disk survives a scale-down or a full deletion. That's real, load-bearing machinery, and it's a large part of why running a database on Kubernetes went from "please don't" to routine over the last several years. It is also, entirely on its own, an inert guarantee of continuity — nobody has yet asked "is db-0 actually a healthy primary right now?", or done anything about the answer. A StatefulSet will faithfully recreate a crashed db-0 and reattach its exact old disk, and consider its job done — even if that disk belongs to a primary that crashed mid-write and needs recovery before it should answer a single query.
Closing that gap is exactly the job Operators & Custom Resource Definitions describes in general: a controller running the same watch → diff → act loop as anything built into Kubernetes core, except this time watching a Kind that represents a whole database cluster rather than a generic workload. What makes a database operator specifically hard, on top of the CRD-and-controller mechanics that page already walks through end to end, is everything the rest of this page covers: deciding who's allowed to be primary right now, keeping a client's connection string pointed at whoever that is, sequencing an upgrade so it never outruns replication, and treating backup as something that has to be provably restorable rather than merely scheduled.
PodDisruptionBudgets for a quorum, not a headcount
☺ Like you're 10: For an ordinary app, "keep at least one copy running" is enough of a rule. For a voting system, going from three voters to two doesn't just mean fewer votes — cross the wrong line and the group can't agree on anything at all.
SRE's Kubernetes reliability patterns covers the PodDisruptionBudget object itself in full — minAvailable vs. maxUnavailable, how kubectl drain and the eviction API respect disruptionsAllowed, and the line between voluntary and involuntary disruption. Assume that mechanism here and go straight to the number that actually matters for a clustered database, a Kafka broker set, or a self-hosted etcd: not "how many replicas are up," but "does the surviving set still hold quorum." A Raft- or Paxos-style system needs a strict majority of its members to agree before it can elect a leader or accept a write — for N members that's floor(N/2) + 1. A 3-node cluster needs 2 alive; a 5-node cluster needs 3. Drop below that floor and the system doesn't degrade gracefully the way a stateless app with one fewer replica does — it typically stops accepting writes entirely, or, worse, two surviving minorities can each convince themselves they're the legitimate majority.
The genuinely useful detail most PDB explanations skip: disruptionsAllowed is computed from currently healthy Pods, not from a counter of voluntary evictions since some reset. If one quorum member is already down — crashed, OOMKilled, a node that died overnight, nothing to do with kubectl drain at all — a PDB with maxUnavailable: 1 on a 3-member set already shows disruptionsAllowed: 0, and correctly refuses to let a routine, unrelated node drain take out a second member. That's the actual protection: the PDB doesn't care why a member is already unhealthy, only that letting one more go voluntarily would cross the quorum line.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: ledger-quorum-pdb
spec:
# 3 members, quorum needs 2 — this permits losing exactly the 1 you can spare.
# Resist the temptation to compute "N minus quorum" and stop there: that
# number is the bare mathematical floor, with zero margin left over for an
# unrelated involuntary failure landing in the same maintenance window.
maxUnavailable: 1
selector:
matchLabels: { app: ledger, role: quorum-member }A mature database operator very often creates and manages a PDB of its own, scoped to its own view of quorum health — CloudNativePG and Strimzi both do this for the clusters they own. Applying your own separate PDB against the same Pods, with a different maxUnavailable, doesn't add protection; it adds a second, possibly contradictory opinion the eviction API has to reconcile, and the more conservative of the two silently wins in ways that are annoying to debug. Check what the operator already manages before writing your own — this is the same "don't fight the reconciler" lesson Operators & CRDs covers for any managed child object.
Backup strategy once an operator owns the loop
☺ Like you're 10: An operator that promises nightly backups is like a smoke detector that promises to beep — the promise means nothing until you've actually stood in the room during a real fire and heard it go off.
DevOps' Database Change Management covers RPO and RTO as pipeline concerns in full — pgBackRest and WAL-G streaming write-ahead logs for point-in-time recovery, a pre-migration snapshot gate, and a scheduled, deploy-independent restore-and-verify drill that proves a backup actually restores instead of merely existing. Assume that framework and go one layer more specific: what changes once a database operator owns the backup lifecycle instead of a separate pipeline stage invoking a CLI tool by hand. The backup schedule, retention window, and WAL-archiving destination become fields on the database's own custom resource, kept true by the exact same reconcile loop that manages everything else about the cluster — declare it once, and the operator re-verifies and re-applies it on every pass, the same way it re-verifies replica count or storage. Continuous WAL (or binlog) archiving turns RPO into "roughly the time between log segment switches" — seconds — rather than "however long since the last nightly snapshot," with no separate cron job to keep alive.
Restore inherits the same declarative shape as everything else on this course's object model: rather than running a restore command against a live instance, you declare a new cluster object whose bootstrap source happens to be a prior backup, and the operator reconciles that desired state into existence from scratch.
# A restore is a new Cluster object, not a special "restore" verb — the
# operator reconciles it into existence exactly like any other desired state.
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: orders-db-restored
spec:
instances: 3
bootstrap:
recovery:
backup:
name: orders-db-backup-20260826-0300 # a Backup object the operator already reconciled
storage:
size: 50GiOne consistency distinction still matters even under an operator: a raw CSI volume snapshot, taken without the database's cooperation, is only crash-consistent — a valid image of the disk exactly as if the machine lost power, which most engines can recover from via their own write-ahead log but not always cleanly. An application-consistent backup, the kind an operator's own backup path produces, briefly coordinates with the database first, so what's captured is a clean transaction boundary rather than a snapshot mid-write. PE's Storage & Stateful Workloads goes deeper on exactly why that distinction matters and on the fuller operator landscape across engines.
An operator automates taking a backup. It does not automate knowing the backup actually restores. That gap doesn't close on its own just because a CustomResource says backup.enabled: true — the scheduled, deploy-independent restore drill DevOps' Database Change Management page describes is still a separate, human-owned habit, whether the backup underneath it came from a cron job or a reconcile loop.
What a database operator has to get right that a generic one doesn't
☺ Like you're 10: Any operator can restart a crashed thing. A database operator additionally has to know which crashed thing was "in charge" a second ago, and make sure exactly one replacement claims that job — never zero, and never two at once.
Four concerns separate a database operator from the generic reconcile-loop mechanics Operators & CRDs already covers: failover — detecting an unhealthy primary and promoting exactly one replica to take its place, without two replicas both deciding they're now in charge; connection routing — moving client traffic to whoever the new primary is without every client needing to know a Pod name changed; safe rolling upgrades — sequencing so an upgrade never outruns replication or takes the primary down before a replica is caught up enough to take over; and the backup lifecycle covered above. All four run on the identical watch → diff → act loop as a stateless controller — what's different is only what "diff" and "act" mean when the resource being reconciled is a live, opinionated database engine instead of a stateless container.
Connection routing is solved with plain Kubernetes primitives, not anything database-specific: a mature Postgres operator typically manages two Services over the same Pods — a read-write Service whose endpoint always points at the current primary, and a read-only Service load-balanced across replicas — and repoints the read-write Service's endpoint the instant it promotes a new primary. Clients keep using the same DNS name throughout a failover; only which Pod answers behind it changes. Services & Networking covers the endpoint mechanism a Service relies on to make that repointing work.
# Synchronous replication, declared once: promotion won't discard a
# transaction this replica hasn't confirmed, at the cost of write latency.
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: orders-db
spec:
instances: 3
postgresql:
synchronous:
method: any # any 1 of the configured standbys must confirm before commit returns
number: 1Failure detection itself often reuses a primitive this course has already met: this course's Control Plane Internals covers how kube-scheduler and kube-controller-manager use a Kubernetes-native Lease object for leader election — exactly one active holder, renewed on a clock, picked up by a standby the moment renewal lapses. Patroni, the HA agent several Postgres operators wrap, can use that same Kubernetes API — Endpoints or a native Lease — as its own distributed configuration store, so a database's leader election and Kubernetes' own control-plane leader election end up leaning on the identical mechanism, just pointed at a different set of processes.
Operators & CRDs' build-vs-adopt framework applies here directly, and its own example is a database — a mature, community-hardened operator is almost always the right call over writing one, and this is a domain where the correctness bar is unusually high to clear alone. CloudNativePG for Postgres and Strimzi for Kafka are two of the most widely deployed; PE's Storage & Stateful Workloads surveys the fuller landscape across engines — Vitess for sharded MySQL, the Zalando Postgres operator, Percona and MongoDB and Redis operators — in more depth than this page needs to repeat.
"The first operator CR I ever wrote by hand — before I learned to just adopt one — got the promotion order right and the RBAC wrong. I scoped the ServiceAccount to cluster-admin because I didn't want to debug a permissions error mid-build, told myself I'd tighten it later, and shipped it. 'Later' arrived the day someone asked what that ServiceAccount's token could actually touch, and the honest answer was 'everything, in every namespace, forever.' A generic operator being over-permissioned is bad. A database operator being over-permissioned is worse, because it already legitimately needs to create Secrets and manage volumes — which makes an audit of 'does it need this specific verb' actually necessary, not optional."
Timmy's concern in that story generalizes: a real database operator's ServiceAccount typically needs to create and read Secrets (connection credentials, TLS certs), manage PersistentVolumeClaims, and patch Services — real, sensitive verbs, not a rubber-stamp reason to reach for cluster-admin. This course's RBAC & Admission Control and DevSecOps' Kubernetes security deep dive cover scoping a controller's ServiceAccount down to exactly the verbs its reconcile loop actually calls.
When a managed database off-cluster is the better call
☺ Like you're 10: Running the database yourself buys you every single knob. It also means every one of those knobs is now something your own team, not somebody else's on-call rotation, has to actually understand at 2 a.m.
Everything on this page — quorum-aware PDBs, WAL-archiving RPO, failover and connection routing, RBAC scope — is a real, learnable set of knobs. It's also, honestly, a second full-time discipline layered on top of running Kubernetes at all, and a managed service (RDS/Aurora, Cloud SQL, Azure Database, or a fully-managed Postgres/Kafka offering) exists specifically to take every one of those knobs off your team's plate in exchange for a bill and somewhat less control. Deciding which side of that trade to take is a real engineering decision, not a referendum on whether Kubernetes "can" run a database — it obviously can.
| Dimension | Leans self-hosted (on Kubernetes) | Leans managed (off-cluster) |
|---|---|---|
| Operational bench depth | A team that can confidently answer "what's our RPO, and when did we last test a restore?" today | A small team, or one that would be answering that question for the first time under this page |
| Portability need | Genuine multi-cloud, hybrid, or on-prem/edge requirements a cloud-native managed service can't reach | Single-cloud is fine, and the provider's managed offering is a first-class product there |
| Scale & cost profile | Real scale where managed per-hour pricing becomes the dominant line item and self-hosting pays for itself | Scale where the operational cost of self-hosting (people-hours, incident risk) exceeds the price premium |
| Workflow | Everything already lives under one GitOps control loop, and a second, separate provisioning path is the odd one out | No strong pull toward GitOps-managed infrastructure specifically for this workload |
| Blast radius if it goes wrong | Acceptable, because the failure modes above are understood, tested, and rehearsed | Mission-critical data where "we've never tested our restore" is not a risk worth taking on |
A platform team operating at scale often doesn't have to choose once and live with it — PE's Storage & Stateful Workloads covers offering both self-hosted and managed database provisioning behind one self-service API, so the decision becomes a field in a form rather than a re-architecture. This course's forthcoming Cost & FinOps on Kubernetes covers the cost-comparison side of that call in more depth than fits here.
Self-hosting a database to save a modest monthly bill is a false economy the moment nobody on the team can confidently answer this page's own checkpoint questions. Running a database on Kubernetes is a standing commitment — to quorum-aware PDBs, to a rehearsed restore, to RBAC that isn't cluster-admin — not a checkbox you tick once during setup. Gizmo will always have a reason it's "basically the same thing, just cheaper." It usually isn't, once the pager actually goes off.
On a kind cluster, install a mainstream operator for whichever engine you know best — CloudNativePG for Postgres is a reasonable default — and stand up a 3-instance cluster. Once it's healthy, delete the primary Pod by hand with kubectl delete pod and time two things: how long until a replica is promoted (check kubectl get cluster -o wide or the operator's own status field), and how long until a client using the read-write Service's DNS name successfully reconnects. That measured number, not whatever the operator's README claims, is your actual RTO for this failure mode.
Gizmo the Gremlin: Whole StatefulSet-plus-operator thing is a lot of YAML for a database. Just run Postgres as one Pod with a plain PVC, and let Velero snapshot it nightly. Simpler, and it's still "backed up."
Benny the Beaver: One Pod means one primary and zero replicas — the instant that Pod's node has a bad night, you're not failing over, you're waiting for a reschedule and hoping the disk reattaches cleanly.
Ellie the Elephant: And a nightly volume snapshot with no WAL archiving in between gives you an RPO measured in "up to 24 hours," not seconds. I've got the receipts on every snapshot we've taken — none of them are continuous.
Timmy the Turtle: Nobody's even asked what that lone Pod's ServiceAccount can touch yet, either. "Simpler" keeps meaning "fewer things I checked," not fewer things that can actually go wrong.
Sol the Sloth: If three replicas feels like overkill for what this actually needs, that's a fair question — but it's a sizing conversation, not a reason to skip quorum and backups entirely. Work out what it really needs slowly, don't just round down to one.
Recon the Robot: Give me an operator and a proper CR and I'll keep failover, backups, and connection routing continuously true, the same loop as everything else I reconcile. Give me one bare Pod and there's nothing for me to watch — there's no gap left to close, because nothing's declared.
Benny the Beaver: Installing CloudNativePG properly this afternoon, then. Three instances, WAL archiving on, a PDB that actually respects quorum. Same amount of YAML as Gizmo's version, once you count what his was quietly skipping.
1. What does a StatefulSet guarantee on its own, and what does it explicitly not know or do about the database running inside it? 2. A 5-node quorum cluster loses one member to an unrelated crash. What should disruptionsAllowed read on a correctly-configured PDB at that moment, and why? 3. Why does continuous WAL (or binlog) archiving narrow RPO to roughly seconds, where a nightly-only snapshot doesn't? 4. What's the actual difference between a crash-consistent backup and an application-consistent one? 5. How does a database operator typically move client traffic to a newly-promoted primary without clients needing to know a Pod name changed? 6. Name two ServiceAccount verbs a real database operator legitimately needs that make "just grant cluster-admin" a worse idea than it would be for a stateless controller. 7. Give one condition that clearly favors a managed database over self-hosting, and one that clearly favors self-hosting on Kubernetes.
Check your answers
- A StatefulSet guarantees stable identity and stable per-replica storage — the same Pod name always gets the same disk back. It has no idea whether the workload inside is a healthy database primary, a crashed one mid-recovery, or anything in between, and does nothing about failover, backup, or connection routing on its own.
- It should read 0. Quorum for 5 members is 3; with one already down, only 4 are healthy, and losing one more voluntarily would drop the cluster to 3 — the bare floor with zero margin — so a correctly configured PDB (accounting for currently healthy Pods, not a separate voluntary-eviction counter) refuses any further voluntary disruption at that point.
- Because RPO is bounded by the gap between the last durable copy of data and the moment of loss. Continuous WAL archiving durably ships each log segment as it's produced, so the gap is roughly "time since the last segment switch" — seconds. A nightly-only snapshot's gap is "time since last night's snapshot," which can be nearly a full day.
- A crash-consistent backup captures the disk exactly as-is, as if power was cut mid-write — valid, but recovery depends on the engine's own crash-recovery logic doing the right thing. An application-consistent backup briefly coordinates with the database first, so the captured state is a clean transaction boundary rather than a mid-write snapshot.
- By managing a stable Service (a read-write endpoint) whose selector or endpoint the operator repoints to the new primary the instant it promotes one — clients keep using the same DNS name throughout, and only which Pod answers behind it changes.
- Creating and reading Secrets (connection credentials, TLS certificates) and managing PersistentVolumeClaims are both routine, necessary verbs for a database operator — which is exactly why scoping them precisely matters more, not less: an audit of "does it need this verb" is genuinely worth doing rather than a formality to skip with a blanket cluster-admin grant.
- Managed clearly wins when the data is mission-critical and the team can't yet confidently answer basic RPO/RTO and restore-testing questions. Self-hosting on Kubernetes clearly wins when there's a genuine multi-cloud or on-prem portability requirement, or real scale where managed per-hour pricing becomes the dominant cost and the team already has the operational depth this page assumes.