Capstone Part 4 — Run the Incident & Write the Postmortem

This is the fourth of six parts building one continuous project: checkout, the service you defined SLOs for in Part 1, instrumented and alerted on in Part 2, and wrote a real PagerDuty escalation and runbook for in Part 3. Everything before this page was preparation — a target on paper, an alarm proved against a synthetic spike, a phone tree drilled once on purpose. This part is where that preparation gets tested against something it didn't see coming: you'll run a provided fault-injection script that reproduces a genuine connection-pool failure, get paged for real by Part 3's escalation policy, declare formal incident command using the Incident Commander/Comms Lead split from Incident Command for Large-Scale Incidents — even though one service technically doesn't need the full structure — mitigate it using the exact runbook entry Part 3 already wrote for this, and then write the postmortem all the way through a five-whys root cause and a set of action items with real owners and due dates, two of which become the opening move of Part 5 and Part 6. By the end you'll have spent real minutes against the 43.2-minute error budget you calculated in Part 1, on purpose, and have the paperwork to show exactly where they went.

What this part assumes, and what it produces

☺ Like you're 10: Everything from Parts 1–3 still needs to be standing — this page doesn't rebuild any of it, it breaks it on purpose.

This part assumes Parts 1–3 are done and still standing: the SLI/SLO/error-budget artifacts from Part 1, the docker compose stack, metrics, and four burn-rate alerts from Part 2, and the real PagerDuty escalation policy and runbook entries from Part 3. You'll need docker and docker compose (the exact stack from Part 2 — nothing here needs Kubernetes), the k6 load-testing tool, and access to the real PagerDuty escalation Part 3 wired up (the mechanics are identical if you substituted Opsgenie or Grafana OnCall). Nothing here needs a second person, though a study partner makes the role-split section considerably more real — more on that below.

One deliberate boundary, carried over from Part 1's own note: if anything below ever looked like a duplicate or missing payment charge, that's not this scenario — Part 1 marked payment-correctness as a zero-tolerance invariant, outside any error budget, always a full incident regardless of what this page's SLO math says. Today's incident is a clean availability breach: real 5xx responses and timeouts, nothing double-charged.

The scenario: a connection-pool leak, live

☺ Like you're 10: A provided script quietly breaks how checkout talks to its own database, then throws real traffic at it until the cracks show.

Part 2's checkout-svc stub proved the alerting pipeline end to end without ever touching a real datastore — CHAOS_ERROR_RATE was enough. Part 1's architecture and Part 3's own runbook entry for "checkout's own order database" both already assumed a real one exists, so this part is where it actually shows up: a small Postgres service, and a genuine connection pool that a bad config change can genuinely exhaust. Add one service to Part 2's docker-compose.yml, and two new lines to the checkout service already in it:

# docker-compose.yml — additions only, nothing else in Part 2's file changes
services:
  checkout:
    environment:
      - CHAOS_ERROR_RATE=${CHAOS_ERROR_RATE:-0}
      - DB_POOL_MODE=${DB_POOL_MODE:-pooled}                          # NEW
      - DATABASE_URL=postgresql://postgres:checkout@order-db:5432/orders  # NEW
    depends_on: [order-db]                                            # NEW

  order-db:                                                           # NEW
    image: postgres:16
    environment:
      - POSTGRES_PASSWORD=checkout
      - POSTGRES_DB=orders
    ports: ["5432:5432"]

And in checkout-svc/app.py, a connection pool with exactly one deliberate bug — the fault this whole part exists to trigger:

# checkout-svc/app.py — additions to Part 2's file
import psycopg2
from psycopg2.pool import SimpleConnectionPool

DATABASE_URL = os.environ["DATABASE_URL"]
DB_POOL_MODE = os.environ.get("DB_POOL_MODE", "pooled")   # "pooled" or "per-request"
_pool = SimpleConnectionPool(1, 10, DATABASE_URL)          # 10 connections, same as Postgres's own default

def get_conn():
    if DB_POOL_MODE == "per-request":
        return psycopg2.connect(DATABASE_URL)   # BUG: opened outside the pool, never returned to it
    return _pool.getconn()

def put_conn(conn):
    if DB_POOL_MODE != "per-request":
        _pool.putconn(conn)

# inside the existing /checkout handler: acquire before the simulated work, release after
conn = get_conn()
try:
    with conn.cursor() as cur:
        cur.execute("INSERT INTO orders (cart_id) VALUES (%s)", (request.json.get("cart_id"),))
    conn.commit()
finally:
    put_conn(conn)

Under light traffic this is invisible — Postgres's own default connection ceiling is generous, and a handful of leaked connections goes unnoticed. Under real concurrent load, ten pooled connections run out within seconds, new requests queue for one that never comes back, and they time out or get refused outright. This is the exact failure mode from the five-whys worked example on postmortems & blameless culture — you're about to live through the incident that example was abstracted from, against your own service, with your own dashboards.

The provided scenario script flips DB_POOL_MODE the same way Part 2 flipped CHAOS_ERROR_RATE — an env var and a container recreate, nothing more exotic — then hands you a k6 script to generate the concurrent load that actually triggers the saturation. The bug is latent until traffic makes it real, same as almost every capacity-shaped incident:

# chaos/scenario-connection-leak.sh — provided for Part 4
#!/usr/bin/env bash
set -euo pipefail

echo "[scenario] flipping DB_POOL_MODE to per-request..."
echo "DB_POOL_MODE=per-request" >> .env
docker compose up -d checkout     # recreates only the checkout container, same as Part 2's own CHAOS_ERROR_RATE flip

echo "[scenario] fault is live and harmless at idle - the pool only starves under concurrency."
echo "[scenario] open a second terminal and start the load generator:"
echo "  k6 run chaos/checkout-load.js"

// chaos/checkout-load.js — the same generator, run much bigger, in Part 5
import http from 'k6/http';

export const options = { vus: 40, duration: '10m' };

export default function () {
  http.post(
    'http://localhost:8080/checkout',
    JSON.stringify({ cart_id: 'cart_9182' }),
    { headers: { 'Content-Type': 'application/json' } }
  );
}

Getting paged, and declaring command on purpose

☺ Like you're 10: The phone you wired to a real number in Part 3 is about to actually ring — and instead of just fixing it quietly, you're going to run the whole formal process on purpose, for the practice.

Within roughly a minute of the load generator starting, Part 2's CheckoutErrorBudgetBurnFast rule — the 1-hour/5-minute window pair at the 14.4× threshold from multi-window, multi-burn-rate alerting — should fire, and this time it goes somewhere real: Part 3's tested checkout-primary-ep escalation policy rings whoever's on the checkout-primary-oncall schedule this week. That's you, for this exercise:

ALERT FIRING: CheckoutErrorBudgetBurnFast
  service:     checkout  (PagerDuty service: checkout-api)
  severity:    page
  windows:     1h / 5m  (both breaching)
  burn_rate:   approx 920x   (threshold: 14.4x)
  summary:     error-budget burn rate is ~64x past the paging
               threshold - at this rate the entire 30-day budget
               empties in well under an hour if it keeps going
  runbook_url: https://runbooks.acme.io/checkout-api#rb-db-pool

That 920× isn't a typo, and it's worth sitting with for a second: the rule pages at 14.4×, which corresponds to "the budget would empty in about two days if sustained." A near-total outage like this one is nowhere near that boundary case — it's roughly 64 times past it, which is exactly why an alert firing this hard should never be mistaken for noise. Notice, too, that the runbook_url annotation isn't decorative this time — it resolves straight to the exact order-database entry Part 3 wrote, the #rb-db-pool anchor, because that page's whole point was making sure this alert would never fire without somewhere useful to send you.

Per Part 3's first-five-minutes checklist, the very first action is acknowledge — before opening any dashboard. Do that now: acknowledging stops the 15-minute escalation clock from paging Marco on the secondary schedule while you're still just looking. Then declare. Per Incident Command for Large-Scale Incidents, "the primary responder asks for it" is always sufficient on its own to declare formal command — no blast-radius or headcount trigger needs to fire first. Treat choosing to run this lab as asking for it. That page also names the one rule that matters most: the Incident Commander decides, and never personally debugs. A single service doesn't need all three of that page's formal roles running in parallel, so this capstone collapses them to exactly two — the two the brief for this page names on purpose:

With a partner: split exactly those two roles, and enforce the IC rule strictly — the moment the IC opens a terminal to "just check one thing," call it out, the same IC-drift anti-pattern the source page warns about. Solo: keep both hats, but make the split real by narrating it — before running any diagnostic or mitigating command, say (and log) the IC decision first, then switch hats and type it as Ops/Comms. It feels artificial for about ninety seconds and then stops being artificial, which is the whole point of building the habit here, on a service small enough that getting it wrong costs nothing, before an incident where getting it wrong costs a lot.

Running the room: the action log and the comms cadence

☺ Like you're 10: Write down every real action with a timestamp as you go — not because it's fun, but because it's the only honest memory you'll have once you sit down to write the postmortem.

Keep a single running log for the whole incident, timestamped in UTC, the same discipline Incident Command for Large-Scale Incidents calls the single shared operating picture. A shared doc, the #checkout-reliability Slack channel Part 3 already routes tickets and Part 1's early-warning trigger into, or a plain text file all work — what matters is that it's one place, appended to in real time, not reconstructed from memory afterward. Comms/Ops posts a status update on a fixed cadence — every 5 minutes for an incident this size — whether or not anything materially changed:

14:02 UTC  CheckoutErrorBudgetBurnFast fires (1h/5m windows both
           breaching, burn rate approx 920x). Paged.
14:02 UTC  Acknowledged. Escalation clock stopped before Marco.
14:03 UTC  Declared. IC: [you/partner]. Ops/Comms: [partner/you].
14:04 UTC  Comms: "Investigating checkout 5xx spike. No
           customer-facing message yet. Next update 14:09."
14:05 UTC  Checked recent deploys/config (first-five-minutes step
           2): DB_POOL_MODE flipped to per-request ~6 min ago.
14:06 UTC  Dependency triage: payment-processor and
           inventory-service both read clean - fault is internal.
14:07 UTC  Ran Part 3's order-database entry: pg_stat_activity
           shows 10/10 connections active, none idle. Pool
           exhausted, matches the recent config change.
14:09 UTC  Comms: "Root cause suspected: DB connection pool,
           tied to a recent config flip. Mitigation in
           progress. Next update 14:14."
14:10 UTC  IC approves reverting the recent change (Part 3's
           runbook: "roll it back first"). Ops flips DB_POOL_MODE
           back to pooled, docker compose up -d checkout.
14:12 UTC  Error ratio falling: 41% -> 8% -> 0.3%.
14:16 UTC  SLIs back inside SLO, sustained 4 minutes. IC declares
           resolved.
14:17 UTC  Comms: final update, incident closed.

Notice the cadence held even when there was nothing new to say at 14:09 — "still investigating, mitigation in progress" is itself information, and a missed update reads as "nobody's driving," which is precisely the anxious silence the Comms role exists to prevent. This exact log, unedited, is the raw material the postmortem's timeline section is built from a few sections down.

Diagnosing, mitigating, and spending real error budget

☺ Like you're 10: Follow the same triage order Part 3 already wrote down, land on the same runbook entry it already prepared, and then do the arithmetic on how much of the month's "allowed to be broken" time you just used up.

Diagnosis follows Part 3's own triage order, not a fresh investigation from scratch — rule out the two external dependencies before assuming the fault is internal:

# Part 3's dependency triage - is it payment-processor or inventory-service?
sum(rate(http_client_requests_total{service="checkout-api",code=~"5.."}[5m])) by (dependency)
# both read clean -> per Part 3's runbook, the fault is inside checkout itself

# Availability SLI - error ratio over the last 5 minutes (Part 2's own recording rule)
sre:checkout_requests:error_ratio5m

With both external dependencies clean, pull up the exact runbook entry Part 3 wrote for this — "checkout's own order database" — and run its diagnostic command as written:

docker compose exec checkout psql "$DATABASE_URL" -c \
  "SELECT count(*) FROM pg_stat_activity WHERE state != 'idle';"
# (swap for: kubectl -n checkout exec -it deploy/checkout-api -- psql ...
#  if your own Part 3 runbook commands assumed Kubernetes instead of
#  Part 2's docker compose - same query either way)

Ten active connections out of a ten-connection pool is the number that ends the guessing. Per Part 3's own runbook, mitigation is ordered by speed, and this incident matches its first case exactly: "if the deploy check found a recent change, roll it back first — a bad query shipped an hour ago is faster to undo than to fix live." DB_POOL_MODE flipping six minutes before the alert fired is precisely that recent change:

echo "DB_POOL_MODE=pooled" >> .env
docker compose up -d checkout

Don't declare resolved the instant the graph looks better. Per Incident Command for Large-Scale Incidents, the IC confirms the SLI is back inside SLO and the mitigation looks durable — not merely that the last five minutes looked fine — so watch sre:checkout_requests:error_ratio5m stay clean for a sustained window (the log above uses 4 minutes) before calling it.

Now do the arithmetic Part 1 set up and Part 2 alerted on. A 14-minute incident (14:02–14:16) at a 92% average error ratio doesn't consume 14 full minutes of budget — it consumes a weighted 14 minutes, because 8% of traffic was still succeeding the whole time:

Incident duration:          14 minutes
Average error ratio:        92%
Weighted minutes consumed:  0.92 x 14   = 12.9 minutes
Monthly availability budget: 43.2 minutes (from Part 1)
Fraction of budget spent:   12.9 / 43.2 = ~30%

Same math in requests (5,000,000 req / 43,200 min = 115.7 req/min):
Requests during window:     115.7 x 14  = ~1,620
Failed requests:            0.92 x 1,620 = ~1,490
Fraction of the 5,000-req budget:  1,490 / 5,000 = ~30%

Budget remaining after this one event: ~70%
(30.3 of 43.2 min / 3,510 of 5,000 req, for the rest of the 30-day window)

Writing the postmortem

☺ Like you're 10: Fill in the same sections every time, in the same order, using only what's actually in your timestamped log — not what you remember an hour later.

Draft the postmortem the moment the incident closes, using the exact template from postmortems & blameless culture and nothing but the action log above as source material. Below is what a complete one looks like for this exact incident — write your own from your own numbers if a real run differs, but keep every section:

POSTMORTEM - checkout availability incident
Date: [your run date]  ·  Author: [you]  ·  Status: Reviewed

SUMMARY
checkout returned 5xx responses or timed out on the large
majority of checkout traffic for 14 minutes (14:02-14:16 UTC)
after a config change (DB_POOL_MODE=per-request) stopped the
service from returning database connections to its pool.
Mitigated by rolling the change back. Roughly 1,490 checkout
attempts failed - about 30% of this month's availability error
budget spent in a single event.

TIMELINE (UTC)
14:02  CheckoutErrorBudgetBurnFast fires (burn rate ~920x).
       Paged and acknowledged immediately.
14:03  Incident declared. IC and Ops/Comms assigned.
14:04  First Comms update posted.
14:05  Recent-change check (first-five-minutes step 2):
       DB_POOL_MODE flipped ~6 minutes before the alert fired.
14:06  Dependency triage: payment-processor and
       inventory-service both clean - fault is internal.
14:07  Ran the order-database runbook entry: pg_stat_activity
       shows 10/10 connections active. Pool exhausted.
14:09  Comms update: root cause suspected, mitigation in
       progress.
14:10  Reverted DB_POOL_MODE to pooled, recreated the container.
14:12  Error ratio falling: 41% -> 8% -> 0.3%.
14:16  SLIs back inside SLO, sustained 4 minutes. IC declares
       resolved.
14:17  Final Comms update and close.

ROOT CAUSE(S)
1. A config change (DB_POOL_MODE=per-request) shipped without a
   load test exercising connection-pool behavior under
   concurrent traffic - CI only checks functional correctness.
2. The change was reviewed as a one-line env-var tweak, not with
   the same scrutiny a schema or query change would get, even
   though it touched how every request holds a shared resource.
3. No recurring fault-injection test exists that would have
   caught this failure mode before a real deploy did.

IMPACT
~1,490 of ~1,620 checkout requests during the window failed
(5xx or timeout) - about 30% of the monthly availability error
budget (43.2 min / 5,000 req) spent in one event. No
payment-correctness invariant was broken - every failure was a
clean 5xx or timeout, never a duplicate or missing charge.

WHAT WENT WELL
- The Part 3 runbook's order-database entry pointed straight at
  the right pg_stat_activity query - diagnosis, once triage
  reached the internal branch, took under two minutes.
- Acknowledging immediately stopped the escalation clock before
  it ever reached Marco on the secondary schedule.
- The mitigation (revert the config change) was correct on the
  first try - no back-and-forth between competing fixes.

WHAT WENT POORLY
- DB_POOL_MODE shipped as a plain env-var change with no
  concurrency test and no extra review, despite touching shared
  connection-pool behavior.
- The Comms cadence slipped to 6 minutes once, against a
  5-minute target.

ACTION ITEMS
1. Add a k6 load test exercising DB connection reuse under
   concurrent traffic to CI.
   Owner: you   Due: before Part 5
2. Require config flags that touch shared resources (pools,
   caches, rate limits) to go through the same review as a code
   change, not a one-line env-var diff.
   Owner: you   Due: this week
3. Build a recurring chaos experiment that injects this exact
   connection-pool failure mode on a schedule.
   Owner: you   Due: Part 6

Read the root-cause section again and notice what it never says: it never says "whoever flipped the config should have tested it more carefully." Every line lands on a gap in CI, in review process, or in ongoing verification — fixable things that protect whoever touches this config path next, not a verdict on whoever touched it this time.

Five whys: getting past the first answer

☺ Like you're 10: Keep asking "why" about your own last answer until you stop landing on a person and start landing on a missing check.

The root-cause section above didn't arrive fully formed — it's the output of running the five whys against the symptom in the log, refusing to stop at the first plausible-sounding answer:

Symptom: checkout returned 5xx/timeouts to ~92% of traffic for
         14 minutes.

Why?    Because the database connection pool was fully
        saturated and new requests couldn't obtain a connection.
Why?    Because a config change (DB_POOL_MODE=per-request)
        stopped connections from being returned to the pool,
        opening a fresh one per request instead.
Why?    Because that change passed review as a plain env-var
        tweak - the same bar as changing a log level - even
        though it altered how every request holds a shared
        resource.
Why?    Because CI has no test that exercises connection-pool
        behavior under concurrent load - only functional
        correctness is tested, not resource usage under
        traffic.
Why?    Because this exact class of regression has never
        caused a production incident before, so a
        concurrency-load test was never prioritized into the
        pipeline.

Root cause: CI validates correctness but not resource behavior
under concurrent load, and review doesn't distinguish a
cosmetic config flag from one that touches a shared resource -
so a regression exactly like this one passes both gates and
merges undetected until real traffic finds it in production.

Notice the chain never terminates on a person, and notice it took five steps, not one or two — stopping at "a config change stopped connection reuse" would have produced a postmortem whose only real finding is a description of the bug, not an explanation of why the pipeline let it through. The pipeline gap is the part that's still true, and still dangerous, for the next config change too.

Action items that set up Part 5 and Part 6

☺ Like you're 10: Two of today's three fix-it tickets aren't just chores — they're literally the first step of the next two parts of this capstone.

Store every action item as a real ticket linked back to this postmortem, not as prose inside it — a sentence with no owner and no due date will be rediscovered, unfixed, in the next postmortem for the same root cause. What makes this particular set of three worth calling out is that they don't just close the loop in the abstract; two of them are the loop closing across the rest of this capstone:

Milestones

☺ Like you're 10: Tick each box only once you've actually watched it happen on your own screen — not because the step "sounds right."

Work these in order — later ones depend on the incident state from earlier ones. Progress saves in this browser.

Part 4 gave you a real fired-and-resolved incident, a role split you can now actually feel the value of, and a complete blameless postmortem with three tracked action items — two of which are the literal starting point for what comes next. Continue to Capstone Part 5 — Capacity Plan & Load Test, where the CI load-test action item above becomes real, or step back to Run a Reliable Service — start here for how all six parts fit together. For more reps outside this capstone's own story, the incident-response tabletop drill and the postmortem-writing drill both isolate one half of what this page just made you do end to end, and Etsy & the origin of blameless postmortems is worth reading for where the discipline you just practiced actually came from.