Capstone Part 6 — Chaos Engineer It

This is the last of six parts building one continuous project. Part 1 gave checkout-api an SLO and a budget; Part 2 gave it alerting that pages on that budget burning fast; Part 3 gave the page somewhere real to go; Part 4 broke it for real — a database connection-pool leak, fourteen minutes, about 30% of a month's budget gone — and closed with three action items, one of which was a promise: "Build a recurring chaos experiment that injects this exact connection-pool failure mode on a schedule." That promise is this page. Today you turn Part 4's one-off, manually-supervised fault-injection script into a real chaos experiment — a falsifiable hypothesis, a blast radius you can defend, an automated rollback trigger independent of the thing you're testing — and then you run it, on purpose, against the exact fault that already hurt this service once, to find out whether everything built in Parts 1 through 5 catches it faster this time. Or catches it without waking anyone up at all.

What you're carrying in from Parts 1–5

☺ Like you're 10: Nothing on this page is invented fresh — every number and every tool below already exists because an earlier part built it.

Keep this table in view; every section below cites one row of it by name rather than re-deriving it.

You'll need everything already running from Parts 2–4 — Prometheus, Alertmanager, Grafana, and a live PagerDuty integration, all pointed at kubectl config current-context reading kind-sre-dev and the checkout-api namespace — plus Litmus core installed fresh today, and jq, bc, and k6 already on your path from earlier parts.

Why re-inject a fault Part 5 already added a CI test for

☺ Like you're 10: A test that runs before code ships and a test that runs against the thing that's already running answer two different questions, and only one of them protects you from a hotfix, a manual edit, or a rollback gone sideways.

It's fair to ask why this page exists at all once Part 5's k6 concurrency-reuse check is sitting in CI — doesn't that already close the exact gap Part 4's five-whys found? It closes the first gap: a future pull request that reintroduces DB_POOL_MODE=per-request now fails a build before it ever merges. It does nothing for every path that bypasses a pull request entirely — an emergency kubectl edit during a different incident, a config value drifting back during a rushed rollback, a Helm values file getting reverted by accident. Part 4's own scenario script was exactly that kind of bypass: a direct kubectl patch against a live Deployment, no CI in the loop at all. Shift-left testing (Part 5's CI gate) and shift-right testing (this page) are complementary, not redundant — one keeps a known bug from merging again, the other keeps checking that the running system still survives it if it does.

That's also exactly what Part 4's action item #3 asked for, and it's worth being precise about the word "recurring." A chaos experiment run once and filed away answers "did the system survive this on the day I checked." A scheduled one keeps re-asking the question after every deploy, every config change, every dependency upgrade — which is the only way to catch the day someone quietly undoes Part 4's fix without meaning to.

The hypothesis

☺ Like you're 10: Before anything gets broken on purpose, write down exactly what's supposed to still be true afterward — and today that includes something new: whether a human needs to be involved at all.

Per chaos engineering's three-part structure — fault, scope, expected steady-state — here's today's, built specifically to re-test Part 4's incident rather than a generic textbook example:

Notice this hypothesis has two clauses, deliberately, because it's really asking two different questions Part 4 left open: can this be caught without a human at all, and, if it does reach a human, is the response actually faster than last time or did Parts 2 and 3 just look good on paper. Both clauses are falsifiable — either number, wrong, is a real finding worth writing down, not a reason to quietly redefine "success."

Blast radius: a fault this shape can't be scoped by percentage

☺ Like you're 10: You can't half-flip one setting — the moment you change it, every copy of the app sees the new value at once, so the ladder's rungs have to be built out of time and rehearsal instead of "just hit fewer of the pods."

Chaos engineering's blast-radius ladder is usually a percentage — one pod, then a slice of traffic, then everything. This fault doesn't offer that knob. DB_POOL_MODE lives on the checkout-api Deployment's pod template, so patching it changes the setting for every replica the instant the rollout completes — there's no version of "25% of pods still pooled, 75% not," the way a Litmus PODS_AFFECTED_PERC field controls for a pod-delete experiment. When a fault can't be fractionally scoped, the controls that matter shift to two things instead: how long it's allowed to run, and how fast — and how independently of the thing you're testing — it gets reverted. Both get built explicitly below, rather than skipped because the usual percentage knob isn't available.

What the ladder can still give you here is a rehearsal rung. Before this fault ever touches the real checkout-api Deployment, prove the whole patch → saturate → detect → revert cycle against a disposable scratch copy that the real alerting and paging can't see at all:

ctx="$(kubectl config current-context)"
[[ "$ctx" == "kind-sre-dev" ]] || { echo "wrong context: $ctx — stop"; exit 1; }

kubectl -n checkout-api get deploy checkout-api -o yaml \
  | sed 's/name: checkout-api/name: checkout-api-scratch/' \
  | kubectl -n checkout-api apply -f -
kubectl -n checkout-api rollout status deploy/checkout-api-scratch

checkout-api-scratch doesn't carry the app: checkout-api label the recording rules and PagerDuty routing key are scoped to, so nothing you do to it can page anyone or move a real budget number — it exists purely to prove the mechanics below actually work before they're pointed at the Deployment that can.

Building the experiment: a custom ChaosEngine plus an independent watchdog

☺ Like you're 10: Litmus's off-the-shelf toolbox doesn't have a "flip this one setting" tool, so you build a small one yourself and wrap it in the same safety scaffolding every other Litmus experiment gets — plus a second, completely separate off-switch that doesn't trust the first one alone.

Litmus's ChaosHub ships pod-kill, network, and resource-pressure faults — nothing that flips a single environment variable on a named Deployment. That means today's experiment is a small custom ChaosExperiment (a job-pod image wrapping the exact kubectl patch Part 4 already wrote) wired into the standard ChaosEngine/ChaosResult contract Litmus documents. Scope its service account to exactly what it needs — the RBAC-creep gotcha that page warns about applies doubly to a hand-rolled experiment nobody else has already hardened:

# chaos/rbac-pool-drift.yaml — scoped to one verb, one resource, one namespace
apiVersion: v1
kind: ServiceAccount
metadata: { name: pool-drift-sa, namespace: checkout-api }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: pool-drift-role, namespace: checkout-api }
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "patch"]          # nothing else — no delete, no create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: pool-drift-rb, namespace: checkout-api }
subjects: [{ kind: ServiceAccount, name: pool-drift-sa, namespace: checkout-api }]
roleRef: { kind: Role, name: pool-drift-role, apiGroup: rbac.authorization.k8s.io }

# chaos/chaosengine-pool-drift.yaml — illustrative shape; confirm exact
# ChaosEngine fields against Litmus's current CRD before applying for real.
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: checkout-pool-drift
  namespace: checkout-api
spec:
  appinfo:
    appns: checkout-api
    applabel: "app=checkout-api"        # swap to app=checkout-api-scratch for the dry run
    appkind: deployment
  chaosServiceAccount: pool-drift-sa
  annotationCheck: "true"               # the opt-in gate from the section above
  engineState: active
  jobCleanUpPolicy: delete
  experiments:
    - name: checkout-pool-drift         # custom — not from the public ChaosHub
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "180"               # hard ceiling: 3 minutes, no matter what
            - name: TARGET_ENV_VALUE
              value: "per-request"        # what the job pod patches DB_POOL_MODE to
        probe:
          - name: fast-burn-would-have-fired
            type: promProbe
            mode: Continuous             # records, doesn't abort — see the watchdog below
            runProperties: { probeTimeout: 10, interval: 15, retry: 1 }
            promProbe/inputs:
              endpoint: "http://prometheus.checkout-api.svc.cluster.local:9090"
              query: "sre:checkout_requests:error_ratio5m"
              comparator: { criteria: ">", value: "0.0144" }

That promProbe is deliberately a verifier, not an abort switch — it records into the ChaosResult whether the exact condition CheckoutErrorBudgetBurnFast pages on actually got crossed, which is half of today's hypothesis. It is not what stops the experiment. TOTAL_CHAOS_DURATION is Litmus's own hard ceiling — the fault dies at 180 seconds no matter what else happens. The real, fast abort is a third, completely independent mechanism:

# chaos/watchdog.sh — polls the exact signal Part 4's postmortem said the
# runbook should have checked first, and reverts the instant it's pinned.
#!/usr/bin/env bash
set -euo pipefail
NS=checkout-api; PROM=http://localhost:9090
NEEDED=2 SEEN=0   # two consecutive 15s polls above threshold ≈ 30s, not one noisy scrape

while true; do
  ratio=$(curl -s "$PROM/api/v1/query?query=checkout_api_db_pool_in_use/checkout_api_db_pool_max" \
    | jq -r '.data.result[0].value[1] // "0"')
  if (( $(echo "$ratio > 0.95" | bc -l) )); then SEEN=$((SEEN+1)); else SEEN=0; fi
  if [[ "$SEEN" -ge "$NEEDED" ]]; then
    echo "[watchdog] pool pinned for ${NEEDED} polls — reverting now"
    kubectl -n "$NS" patch deployment checkout-api --type merge -p \
      '{"spec":{"template":{"spec":{"containers":[{"name":"checkout-api","env":[{"name":"DB_POOL_MODE","value":"pooled"}]}]}}}}'
    kubectl -n "$NS" rollout status deploy/checkout-api
    break
  fi
  sleep 15
done

The dry run

☺ Like you're 10: Run the whole thing against the harmless copy first and watch every piece work — the patch lands, the pool saturates, the watchdog notices, the revert lands — before it ever touches anything a real page could come from.

Point the ChaosEngine above at app=checkout-api-scratch, annotate only that Deployment, and run it end to end with a small k6 burst against its own port-forward:

kubectl -n checkout-api annotate deployment checkout-api-scratch litmuschaos.io/chaos="true"
kubectl -n checkout-api apply -f chaos/rbac-pool-drift.yaml
kubectl -n checkout-api apply -f chaos/chaosengine-pool-drift.yaml   # applabel: app=checkout-api-scratch
./chaos/watchdog.sh &
kubectl -n checkout-api port-forward svc/checkout-api-scratch 8081:80 &
k6 run --env TARGET=http://localhost:8081 chaos/checkout-load.js

kubectl get chaosresult checkout-pool-drift-checkout-pool-drift -n checkout-api \
  -o jsonpath='{.status.experimentStatus.verdict}'
# expect: the promProbe verdict confirms error_ratio5m did cross 0.0144 on the
# scratch copy's own traffic — proving the probe wiring works — while
# Grafana's real checkout-api dashboard and PagerDuty stay completely silent

That silence on the real dashboard is the entire point of the dry run: it proves the experiment's plumbing — the patch, the load, the probe query, the watchdog's poll-and-revert loop — all function correctly, with zero possibility of paging anyone or spending a single second of the real 30.3-minute budget, because nothing about checkout-api-scratch feeds the app=checkout-api-scoped recording rules at all.

Running it for real, and reading the timeline

☺ Like you're 10: Now point the exact same thing at the real service, once with the automatic fix turned on and once with it deliberately delayed, and time both against the fourteen minutes Part 4 actually took.

Re-point applabel to app=checkout-api, annotate the real Deployment, and run two passes.

Pass A — watchdog live. Start the watchdog, apply the ChaosEngine, and drive load. Because Prometheus scrapes every 15 seconds (Part 2's scrape_interval) and the watchdog needs two consecutive 15-second polls above 95% pool utilization before it acts, expect a revert roughly 30–45 seconds after the pool actually saturates — call it under a minute end to end once the rollout confirms. Compare that to Part 4's postmortem: 8 minutes from page (14:02) to the revert actually applied (14:10), three of which the postmortem itself attributes to checking the wrong dashboard first. The watchdog doesn't beat that time because it's smarter — it beats it because it never had a wrong dashboard to check in the first place; it went straight at the one signal Part 4's five-whys already identified as the real root-cause tell.

PASS A — watchdog live (worked example; record your own real timestamps)
T+0:00  ChaosEngine applies the patch; DB_POOL_MODE=per-request live
T+0:00  k6 load begins against checkout-api
T+0:18  pool utilization crosses 95%, watchdog's SEEN=1
T+0:33  second consecutive poll above threshold, SEEN=2 — revert fires
T+0:33  kubectl patch reverts DB_POOL_MODE to "pooled"
T+0:47  rollout status confirms; pool utilization falling
T+0:52  error ratio back under threshold
-----------------------------------------------------------------
Part 4's real incident, for comparison: page at 14:02, revert applied
14:10 (8 min), fully resolved 14:16 (14 min).

Pass B — watchdog deliberately held back. Comment out the watchdog, run the identical fault, and let it reach a human on purpose — this is the half of the hypothesis that checks whether Part 3's escalation policy and Part 4's runbook edit actually hold up, not just whether automation can outrun them. Confirm the page fires (CheckoutErrorBudgetBurnFast, both windows, for: 2m — the same timing Part 2 already proved), acknowledge it, and open the runbook's DB-pool entry. Time from acknowledgment to the pool-gauge query: it should now be the very first thing you run, not the third dashboard you check, because that's the exact edit Part 4's action item #2 made. Time the full page-to-mitigation-applied window and compare directly to Part 4's 8 minutes — a real, faster number here is the concrete proof that Part 4's postmortem action item actually changed behavior, not just a document.

What the experiment proved, what it didn't, and the new action item

☺ Like you're 10: The fast automated fix is genuinely good news — but a fix nobody ever hears about is also how the same bug quietly comes back a dozen more times before anyone notices the pattern.

Write this up the way postmortems & blameless culture teaches, scaled to an experiment rather than an incident:

Confirmed: the fault reproduces exactly as Part 4 described it — pool saturation under concurrent load, same signal, same shape. The human-facing path is measurably faster than Part 4's real incident once Pass B's runbook edit is in place, because the responder no longer spends the three minutes Part 4's postmortem flagged as avoidable. And, checked the same way Part 4 checked it: no duplicate or missing payment charge occurred in either pass — the pool exhausts on the write step, after the payment call, but every failure surfaces as a clean timeout or 5xx on the order write, never a retried authorization, so Part 1's zero-tolerance invariant stays intact both times, which is itself worth confirming explicitly rather than assuming it still holds just because it held once before.

Not confirmed the way you might expect: in Pass A, CheckoutErrorBudgetBurnFast never actually reaches firing. The watchdog reverts in well under the alert's own for: 2m requirement, so the alert sits in pending and clears on its own — nobody gets paged, because nobody needed to be. That's the hypothesis's best-case outcome technically holding, and it's also a real, new problem: a fault that self-heals fast enough to never page also never leaves a record anyone will see. Run this scheduled experiment — or a real recurrence of the same drift — every week for a month, and it could revert itself every single time without a single human ever learning the pattern is happening at all, which is exactly the kind of invisible, creeping risk alert design & alert fatigue warns a purely silence-optimized system drifts toward.

File that as the experiment's own action item, the same way Part 4 filed three:

ACTION ITEM — from Part 6's chaos experiment
Route the watchdog's revert action through Alertmanager's existing
file-ticket receiver (severity: ticket, #checkout-reliability),
so an auto-remediated recurrence is logged even when it never
reaches CheckoutErrorBudgetBurnFast's firing state.
Owner: you   Due: before this experiment's next scheduled run

Once that's wired, schedule the whole thing to actually recur — a Kubernetes CronJob re-applying the ChaosEngine weekly, or, if you've stood up ChaosCenter, a Litmus CronWorkflow doing the same — so this stops being a chaos experiment you ran once for this page and becomes the recurring verification Part 4's action item #3 actually asked for.

What "done" looks like — closing all six parts

☺ Like you're 10: Six pages ago, checkout-api had no target, no alarm, no phone number, no scar tissue, and no proof any of it would hold under a second hit. Now it has all five.

Six parts, one service, nothing thrown away along the way:

That last column matters more than any single number above it: the hypothesis didn't just "pass." It passed in a way that surfaced something Parts 1 through 5 couldn't have shown you on their own — that a system fast enough to fix itself is also fast enough to hide the fact that it had to. That's the actual discipline chaos engineering is teaching here, not just "break things on purpose."

Part 6 closes the capstone: a real, scheduled chaos experiment against the exact fault that already cost checkout-api fourteen real minutes, an independent rollback fast enough that a human sometimes never needs to get involved, and the honest finding that "sometimes never needs to" is a result worth logging, not just celebrating. Step back to Run a Reliable Service — start here to see all six parts as one arc, revisit chaos engineering and chaos engineering at scale for the concepts behind what you built today, or get a faster, standalone rep with Drill — Design a Chaos Experiment. For the credential path, SRE Practitioner covers this course's own certification, and Gremlin Enterprise Chaos Engineering Certification is worth reading if you'd rather learn the commercial tool this page's alternatives table mentions. And for where all of this actually came from at scale, Netflix & the Simian Army is the origin story this entire course's chaos-engineering material traces back to.