Drill — Design a Chaos Experiment

This is not a lesson — it's one exercise, worked all the way through. Wavelength's checkout stack had a near-miss last month: a routine database failover happened to coincide with a 90-second spike in checkout errors that quietly resolved itself before anyone finished loading a dashboard, and nobody has ever deliberately tested what actually happens to checkout when the database's primary changes underneath it. Your job is to design — not run — the chaos experiment that would answer that question properly: a falsifiable hypothesis, a blast radius and a set of abort conditions decided before anyone touches a keyboard, and an exact specification of the evidence that would prove you wrong. Nothing on this page gets executed against a real system. The deliverable is a design a chaos-engineering review board would actually approve, and writing one properly is the entire skill this drill exists to build.

The scenario: Wavelength's checkout stack, three tiers deep

☺ Like you're 10: A page you click, a program that does the work, and a filing cabinet that remembers the order — three tiers, and today's test targets the filing cabinet.

Wavelength's checkout flow is a plain three-tier design. checkout-web is the presentation tier: a stateless server-rendered frontend behind an ALB, autoscaled 12–20 instances across three AZs — nothing here holds state, and losing any single instance is already a well-tested, unremarkable event that happens routinely during normal deploys. checkout-svc is the application tier: 18 Go API pods, six per AZ, that validate the cart, reserve inventory, write the order record, and call out to Stripe to charge the card. orders-db is the data tier: Postgres 15, with automated failover managed by Patroni — one primary in us-east-1a, a synchronous replica in us-east-1b, and an asynchronous replica in us-east-1c — reached through a PgBouncer connection pool: three poolers, 200 max connections each, 600 total. Every one of checkout-svc's three sequential database calls per checkout attempt (validate cart, reserve inventory, write order) goes through that shared pool.

Nobody has deliberately failed orders-db's primary on purpose — not in staging with realistic load, and certainly not in production. The closest thing to a real signal is last month's near-miss noted in the table above: an unplanned failover during a routine patch window happened to coincide with a 90-second spike in checkout errors that self-resolved before anyone finished pulling up a dashboard. That is not evidence the system is fine. It is evidence that nobody knows yet, one way or the other — which is exactly the condition that makes this a worthwhile chaos experiment instead of a formality that confirms what everyone already believes.

What you're actually being asked to produce

☺ Like you're 10: Four things, written down, before anyone so much as opens a terminal.

Four deliverables, in order, are what this drill is grading — not a script, not a command, not a dashboard screenshot. (1) A falsifiable hypothesis with a named fault, a named scope, and a numeric steady-state threshold. (2) A blast radius, fixed in advance, that states exactly where this runs first and what has to hold true before it's allowed to run anywhere wider. (3) Abort conditions, also fixed in advance, naming the exact metric, threshold, and person with the authority to call it. (4) An exact specification of what evidence — which metric, which query, which numeric threshold — would count as proof the hypothesis is wrong, so that a maybe-it-held result can't quietly get scored as a pass.

Rocky's first draft — and everything a reviewer would send back

☺ Like you're 10: "Kill it and see" isn't a plan, even if the animal saying it is very enthusiastic.

Rocky drops this in the team's chaos-engineering channel on Monday morning, visibly proud of himself:

Plan: kill the orders-db primary in prod Wednesday afternoon
and see what happens to checkout. Should be fine, we have
replicas.

— Rocky

Read it against the four deliverables above and it fails all four, plus one thing it doesn't even realize it's ambiguous about:

• No falsifiable hypothesis. "Should be fine" names no metric and no threshold. There's no number that, if crossed, would make Rocky admit the hypothesis was wrong — which means whatever happens Wednesday, he can call it a success.
• No blast radius. "In prod" is the whole platform, at whatever traffic level Wednesday afternoon happens to be — which, per the baseline table, is close to peak. There's no staging step, no off-peak step, no stated path from small to large.
• No abort conditions. Nothing says what would make the experiment stop, at what threshold, or who has the authority to call it. "See what happens" is not a plan for what to do if what happens is bad.
• No evidence spec. Without a defined observation plan, an ambiguous result — checkout looked fine, mostly, for a bit — can't be told apart from a genuine pass. That's the exact failure mode chaos engineering warns about: a hypothesis only means something if there's a concrete way it could turn out to be false.
• "Kill" and "switchover" are two different faults, and Rocky's plan doesn't say which one he means. A graceful patronictl switchover is a coordinated handoff — Patroni already knows the change is coming, so there's no detection delay. A hard kill (kill -9 on the Postgres process, or terminating the instance outright) tests something extra: how long Patroni's own failure detection takes before it even starts failing over, typically another 10–30 seconds on top of the failover mechanics themselves, governed by Patroni's ttl and loop_wait settings. These are legitimately two different experiments with two different hypotheses. Picking one deliberately — and saying so — is part of the design, not a detail to leave implicit.

None of that means Rocky's underlying instinct is wrong — targeting orders-db's primary is, in fact, the right fault to test today, for reasons the next section makes explicit. The instinct just needs the four missing pieces before it's an experiment instead of a threat.

Step 1 — choose the fault and the target tier

☺ Like you're 10: Test the thing nobody's sure about and that would actually hurt if it broke — not the thing you already know is fine.

Chaos experiments are worth running in proportion to two things: how much genuine uncertainty exists about a failure mode, and how bad it would be if that uncertainty resolved the wrong way. Killing a single checkout-web instance scores low on both — it happens routinely during ordinary deploys, the ALB already handles it invisibly, and nobody has any real doubt about the outcome. A controlled failover of orders-db's primary scores high on both: it's never been deliberately tested, the retry policy in the baseline table (fixed 500ms delay, no jitter) is exactly the kind of gap reliability patterns warns produces synchronized retry waves against a shared resource, and the one real data point available — last month's near-miss — is inconclusive rather than reassuring. That combination is what makes this the right target, not simply "it's the database, databases are scary."

The specific fault: a graceful, operator-triggered switchover — patronictl switchover --candidate <replica> — that hands the primary role from us-east-1a to the synchronous replica in us-east-1b. This deliberately matches the shape of last month's near-miss (a primary changing under load) rather than a hard kill, because the open question the near-miss raised is specifically about what checkout-svc does during and immediately after a primary handoff — not about how long Patroni takes to notice a crash, which is a real but separate question saved for later.

Step 2 — write a hypothesis that can actually fail

☺ Like you're 10: Name the exact number that has to stay true, or you haven't really made a claim.

A well-formed hypothesis has three parts, per chaos engineering: the fault, the scope, and a testable steady-state. Assembled from Step 1 and the baseline table:

Those thresholds are deliberately looser than the 99.92% / 1.4s baseline — some real degradation during an active primary handoff is expected and acceptable. The hypothesis isn't "nothing changes," it's "the degradation stays bounded and the system recovers on its own." 99.5% and 5s are the line between "a database failover, working as designed" and "a database failover that took checkout down with it."

Step 3 — set the blast radius before anyone opens a terminal

☺ Like you're 10: Start in the quiet room, not the crowded one — the math below is exactly why.

The blast-radius ladder from chaos engineering applies directly here, and the baseline table gives enough to justify each rung with arithmetic instead of intuition. The mechanism under suspicion is specific: because retries use a fixed 500ms delay with no jitter, requests in flight when the primary disappears will tend to time out and retry in near-lockstep — and every request that arrives during the failover window queues up behind it, since there's no primary to write to until the promotion completes.

Queued DB calls during the failover window
  ≈ checkout attempts/sec × 3 DB calls/attempt × failover duration (s)

Peak traffic (23.3 req/s), worst-case failover (47s):
  23.3 × 3 × 47  ≈ 3,290 queued calls   — 5.5× the 600-connection pool

Peak traffic (23.3 req/s), average failover (29s):
  23.3 × 3 × 29  ≈ 2,029 queued calls   — 3.4× the 600-connection pool

Trough traffic (3.0 req/s), worst-case failover (47s):
  3.0 × 3 × 47   ≈   423 queued calls   — under the 600-connection pool

Trough traffic (3.0 req/s), average failover (29s):
  3.0 × 3 × 29   ≈   261 queued calls   — well under the 600-connection pool

That's a back-of-the-envelope estimate, not a guarantee — it assumes queued calls arrive as fast as clients attempt them, and doesn't model how PgBouncer actually queues versus rejects under pressure. That imprecision is itself the argument for staging first, not a reason to skip it. But the direction is unambiguous: at peak, even the average historical failover time triples the pool's capacity; at trough, even the worst historical failover time stays under it. That's a real, numeric reason to run off-peak first, not just a vague "traffic is lower then."

1. Stage A — staging, synthetic load. Replay production's peak traffic shape against a staging cluster with the same topology (3-node Patroni, same PgBouncer sizing), trigger the same switchover, and confirm the hypothesis's thresholds hold. No real users, no real money, unlimited retries at this stage.
2. Stage B — production, off-peak, single switchover. Only after Stage A holds cleanly, more than once. Runs at a specific scheduled window — Tuesday 03:30 UTC, inside the historical trough — with a same-day change freeze on checkout-svc, the on-call engineer and orders-db's owning team both watching dashboards live, and the switchover triggered by hand, not on a timer.
3. Stage C — production, peak traffic. Explicitly not part of this design. It's deferred to a later, separately-approved experiment, contingent on Stage B holding cleanly more than once — the same "expand scope only after confidence is earned at the smaller one" rule the blast-radius ladder exists to enforce, and the arithmetic above is exactly why peak isn't where this starts.

Step 4 — write the abort conditions before anyone opens a terminal

☺ Like you're 10: Once the fire alarm's pulled, you can't un-pull it — so most of the safety plan has to happen before you pull it.

Here's a nuance worth stating plainly: once patronictl switchover is triggered, it runs to completion on its own in roughly 20–45 seconds — there is no command that cancels a promotion already in progress. That means "abort conditions" for this specific fault live in two different places, not one:

Any responder watching the dashboards can call a live-phase breach; the on-call IC has final authority to declare the experiment over versus continuing to observe. Because there's no true "abort switch" once the switchover starts, the pre-agreed remediation has to be ready to fire the instant a threshold breaches — not improvised: scale PgBouncer's pool cap immediately, or feature-flag a fast-fail on non-critical checkout steps to shed load off the pool. That's a materially different plan from Step 5's second scenario below, where the fault is instantly reversible — the two "try it yourself" answers in the key contrast this deliberately.

Step 5 — specify exactly what would prove you wrong

☺ Like you're 10: Decide, before you look, exactly what "it broke" would look like on the screen — otherwise you'll talk yourself into "close enough."

The hypothesis from Step 2 is falsified if, at any point in the 10-minute observation window, either of these queries crosses its threshold:

# Falsifies the completion-rate half of the hypothesis
sum(rate(checkout_requests_total{status=~"2.."}[30s]))
  / sum(rate(checkout_requests_total[30s])) < 0.995

# Falsifies the latency half of the hypothesis
histogram_quantile(0.99, rate(checkout_duration_seconds_bucket[1m])) > 5

# Supporting signal — not the hypothesis itself, but explains WHY if either fires
pgbouncer_pools_cl_waiting > 0

The hypothesis holds only if neither query crosses its line for the full window — not "mostly stayed under," not "dipped once and recovered." A single 30-second window at 99.4% is a clean falsification, not a rounding error near 99.5%; the same discipline the SLO & error-budget drill insists on when a number looks close enough to wave through.

The finished design, assembled

☺ Like you're 10: Same four pieces, now stapled into one document a reviewer could actually sign off on.

CHAOS EXPERIMENT DESIGN — orders-db primary failover
Owner: on-call SRE + orders-db team   Status: awaiting Stage B approval

FAULT      patronictl switchover, orders-db primary
           us-east-1a → sync replica us-east-1b

SCOPE      orders-db only. checkout-web / checkout-svc observed, not faulted.

HYPOTHESIS For 10 min bracketing the switchover (T-1min .. T+9min):
             checkout completion rate stays ≥ 99.5% (rolling 30s)
             AND p99 checkout latency stays < 5s (rolling 1min)

BLAST      Stage A: staging, synthetic peak-shaped load        — required first
RADIUS     Stage B: prod, Tue 03:30 UTC, single switchover      — this run
           Stage C: prod, peak traffic                          — NOT in scope

PRE-FLIGHT completion ≥ 99.85% · replica lag < 2s · pools < 50%
GATE       no active incident · IC + orders-db team present
           any gate fails → do not trigger, reschedule

LIVE-PHASE completion < 95% for 30s, OR queue > 100 for 15s,
BREACH     OR p99 > 8s for 60s
           → hypothesis disproven, execute remediation, no fault cancellation exists

EVIDENCE   falsified if either PromQL threshold in Step 5 crosses,
           at any point, for the full window — no partial credit

Try it yourself: the payments dependency

☺ Like you're 10: Same four steps, different target — and this time the fault has an actual off switch.

This drill designs the experiment; it doesn't run it. For the mechanics of actually executing a graduated chaos program — tooling, stakeholder buy-in, and the staging-to-production maturity path — see chaos engineering and chaos engineering at scale. The full version of this exercise, run against a real throwaway environment, is Capstone Part 6 — Chaos Engineer It. For the retry-and-pool reasoning this drill leaned on, see reliability patterns; for the SLO arithmetic behind the error-budget aside, see the SLO & error-budget drill.