Drill — Run a Blameless Postmortem

Every other drill in this set hands you a terminal. This one hands you a blank document. Culture & collaboration already told you that a generative culture separates "what broke and why" from "whose fault is it" — this drill is where you actually do that separation yourself, against a real incident's raw evidence, instead of reading someone else's finished postmortem and nodding along. You'll work the same catalog-api cache-collapse incident diagnosed in Drill — Diagnose a Production Incident — the diagnosis is handed to you below as settled fact, because finding a root cause is a different skill than writing one up honestly, and this drill only tests the second. First you'll read a first draft that gets the facts right and the culture wrong, and name exactly what's wrong with it. Then you'll run your own five whys and write the real thing. Budget 35–40 minutes.

How this drill works

☺ Like you're 10: You'll read a badly-written note first so you know exactly what "wrong" looks like, then you'll write the good version yourself, in four short steps.

Four steps, in order. Step 1 hands you a real first draft of this incident's postmortem — the kind a tired, well-meaning engineer actually writes twenty minutes after a page — and asks you to find everything wrong with it before you're told. Step 2 has you run your own five whys against the incident's evidence down to a systemic root cause, in writing, before checking the model answer. Step 3 is the real deliverable: draft the complete postmortem using this platform's standard template, then compare it against a full model postmortem built from the same evidence. Step 4 is the detail most first drafts still get wrong even after they've stopped naming names — turning a finding into an action item a system can actually enforce, not a sentence that asks people to remember to be better.

The incident, recapped: catalog-api's cache collapse

☺ Like you're 10: Here's everything that happened, already laid out in order — your job today isn't to find these facts, it's to write them up right.

This is Northwind Retail's catalog-api, some months after the transformation described in the case study, diagnosed move-by-move in Drill — Diagnose a Production Incident. If you've already worked that drill, skip straight to Step 1 — this is the same incident, condensed to exactly the evidence a postmortem needs. If you haven't, everything below is self-contained; treat it the way you'd treat a completed on-call handoff.

CATALOG-API — product-detail availability incident, 2026-08-17

13:45  Growth sends the "Back to School" campaign email, 2.1M
       recipients, deep-linking straight into individual product
       pages instead of the usual homepage banner.
14:02  ALERT FIRING: CatalogApiHighErrorRate (6.1%, threshold 2%)
14:03  ALERT FIRING: CatalogApiP99LatencyHigh (1,840ms, threshold 500ms)
14:05  Blended: 1,410 req/s (baseline ~450), errors 9.8%, p99 2,210ms.
14:07  By route: /products/:id/related 68.5% errors (41.0% at
       14:02); /products/:id 11.2% (0.4% at 14:02); /cart-preview
       6.9% (0% at 14:02). /search/suggest and /health unaffected
       throughout.
14:05  Redis hit rate, product-detail routes: 38.4%, down from a
       92% baseline and falling since 13:55 — 640 distinct product
       IDs requested in this window, versus a normal Tuesday's
       top-20 "trending" set.
14:06  pgbouncer SHOW POOLS: sv_active pinned at 60/60, cl_waiting
       climbing. slow-query log: repeated sequential scans of
       product_views (47.9M rows, no index on product_id),
       900ms–2,400ms+ per call.
14:08  Recent-change check: v2.9.0 (4 days ago) added
       /products/:id/related behind the related_products_widget
       flag at 10% of traffic. The flag ramped 10% → 100% two days
       ago (2026-08-15 09:14 UTC) — config-only, no new deploy, and
       it does not appear in `gh run list`, only in flag history.
14:09  Flag flipped to 0% — the kill switch.
14:10  Backends already stuck mid-scan terminated by hand
       (pg_terminate_backend) to drain the pool immediately.
14:12  pgbouncer cl_waiting back to 0.
14:14  Error rates on all three affected routes back to baseline
       for a sustained 2+ minutes. Marked mitigated.
16:30  CREATE INDEX CONCURRENTLY on product_views(product_id)
       shipped through the standard pipeline, verified with
       EXPLAIN ANALYZE. Marked resolved.

That's the whole incident. Nothing below this point should ever introduce a fact that isn't already sitting in that block — a postmortem that quietly adds detail nobody actually observed is exactly the kind of thing culture & collaboration's generative-versus-bureaucratic distinction warns against: information should flow freely, not get embellished to make a tidier story.

Step 1 — read the first draft, and find everything wrong with it

☺ Like you're 10: This note gets every fact right and still fails, because it answers the wrong question.

Twenty minutes after mitigation, someone on the team drafts this. Read it once, all the way through, before doing anything else:

ROOT CAUSE (first draft)

An engineer added the /products/:id/related query without adding
an index for it, which should have been caught in review. Separately,
the growth team pushed the related-products flag to 100% and sent a
2.1M-recipient email campaign without warning the on-call engineer,
which is what actually triggered the incident.

Going forward, engineers need to be more careful about adding new
queries without indexes, and growth needs to loop in on-call before
large campaign sends.

ACTION ITEMS
1. Reviewers should catch missing indexes going forward.
2. Growth to notify on-call before big sends.

Every fact in that draft is true. Write down at least three things wrong with it anyway — specifically, not just "it's mean" — before you open the reveal below.

The blame path vs. the generative path

☺ Like you're 10: Answer "who did this" and the same trap catches the next person; answer "what let this happen" and the trap itself gets removed.

The first draft and the real postmortem don't just read differently — they produce different futures. This is the mechanism culture & collaboration describes through Westrum's typology: a pathological or bureaucratic culture searches for who to blame after a failure, and a generative one searches for what to fix. The question you ask determines which document you're capable of writing next.

Step 2 — run five whys to the systemic root cause

☺ Like you're 10: Keep asking "why" about your own last answer until you stop landing on a name and start landing on a missing check.

Start from the symptom, and keep asking why until you reach something a system — not a person — could have prevented. Write your own chain before opening the reveal; the discipline is refusing to stop at the first plausible-sounding answer, the same Lean, small-batch thinking what is DevOps traces back to the Toyota Production System.

Symptom: error rates and p99 latency spiked across three catalog-api routes between 14:02 and 14:14 UTC, even though only one of those routes had any code or config change behind it.

Why? A shared 60-connection pgbouncer pool filled completely
     (60/60 active, cl_waiting climbing) and stayed full, so every
     route drawing from that pool -- including two with nothing
     wrong in their own code -- had to queue for a connection that
     wasn't there.

Why? A flood of slow queries against product_views (900ms-2,400ms+
     each) was holding connections far longer than normal, because
     none of them could use an index -- product_views.product_id
     has no supporting index, so every one of these queries forces
     a sequential scan across all 47.9 million rows.

Why? Under normal traffic this query almost never ran. The
     product-detail Redis cache absorbs ~92% of requests to this
     route, so the unindexed query only ever surfaced on a rare
     miss -- never expensive enough in aggregate to trip a
     slow-query alert.

Why? A campaign email at 13:45 UTC deep-linked into 640 distinct
     product pages instead of the usual small, cache-warm set of
     trending products, which collapsed the cache hit rate from
     92% to 38% in about ten minutes and turned a rare query into
     a constant one.

Why? Nothing in the pipeline or the observability stack treats
     "traffic against many distinct, cache-cold keys" as a load
     scenario -- load tests exercise volume, not key cardinality --
     and the only alerts on this service watch the downstream
     symptoms (error rate, latency), not the cache hit rate that
     had already started falling several minutes before either one
     fired.

ROOT CAUSE: an unindexed query that is cheap to leave unnoticed
under a high cache-hit rate becomes an availability risk the
moment traffic shape changes -- and nothing in catalog-api's
testing or alerting was built to notice a shape change instead of
a volume change, so the first signal anyone had was the downstream
error-rate alert, minutes after the leading indicator had already
told the same story.

Step 3 — write the real postmortem

☺ Like you're 10: Same sections, same order, every single time — so nobody has to guess what's missing, and nothing important gets left out by accident.

Every postmortem on this platform follows one template: SUMMARY, TIMELINE, ROOT CAUSE(S), IMPACT, WHAT WENT WELL, WHAT WENT POORLY, ACTION ITEMS. Using only the recap block above and your Step 2 chain, draft the whole thing yourself — fifteen minutes, no more — before opening the model answer. Two rules while you write: every sentence has to trace back to a fact in the recap block, and no sentence gets a person's name as its subject.

POSTMORTEM - catalog-api product-detail availability incident
Date: 2026-08-17  ·  Author: [on-call responder]  ·  Status: Resolved

SUMMARY
A "Back to School" marketing email deep-linked 2.1M recipients
directly into 640 distinct product pages instead of the usual
small, cache-warm trending set. The resulting flood of Redis
cache misses hit an unindexed query on a 47.9-million-row table,
and the resulting slow queries filled catalog-api's shared
60-connection pgbouncer pool. Because every route shares that one
pool, two routes with no bug of their own -- /products/:id and
/cart-preview -- degraded alongside the one that actually
triggered it. Mitigated in ~7 minutes by disabling the feature
flag that had (separately) ramped this route to 100% of traffic
two days earlier, and by clearing the backends already stuck on
the slow query. Resolved the same day once a supporting index
shipped.

TIMELINE (UTC)
13:45  Campaign email sent; 640 distinct product IDs start
       receiving direct traffic.
14:02  CatalogApiHighErrorRate and CatalogApiP99LatencyHigh both
       fire. Paged and acknowledged.
14:05  Cache hit rate found already falling (92% -> 38%) -- a
       leading indicator, not itself alerted on.
14:06  pgbouncer pool confirmed full (60/60), cl_waiting
       climbing; slow-query log shows repeated sequential scans
       on product_views, no index on product_id.
14:08  A flag ramp two days prior found in flag history, not the
       deploy log -- the config-only change that set up today's
       incident was nearly missed during triage.
14:09  related_products_widget flag flipped to 0% -- stops new
       expensive queries from being requested.
14:10  Backends already stuck mid-scan terminated by hand to
       drain the pool immediately.
14:12  pgbouncer draining; cl_waiting back to 0.
14:14  Error rates on all three affected routes back to baseline
       for a sustained 2+ minutes. Marked mitigated.
16:30  CREATE INDEX CONCURRENTLY on product_views(product_id)
       shipped through the standard pipeline, verified with
       EXPLAIN ANALYZE. Marked resolved.

ROOT CAUSE(S)
1. product_views has no supporting index on product_id, so any
   real volume of traffic against the related-products query
   forces a full sequential scan of a 47.9M-row table.
2. All catalog-api routes share one fixed-size, 60-connection
   pgbouncer pool with no per-route isolation, so one slow route
   can exhaust connections needed by every other route.
3. Cache hit rate is not itself an alerted metric -- only its
   downstream symptoms (error rate, latency) are -- so the
   leading indicator was invisible until real damage had already
   started.
4. Feature-flag ramps are config-only changes with no deploy, and
   don't appear in the same change log the deploy pipeline and
   on-call already check first during triage.

IMPACT
Elevated errors and latency on /products/:id/related (peak
~68.5%), /products/:id (peak ~11.2%), and /cart-preview (peak
~6.9%) between 14:02 and 14:14 UTC (~12 minutes). /search/suggest
and /health were unaffected throughout. This flag ramp counts as
one change failure toward this month's DORA change failure rate.

WHAT WENT WELL
- Splitting the blended 9.8% error rate by route, then by the
  metric behind it (cache hit rate, pool state), found the real
  chokepoint in about six minutes instead of guessing from the
  alert text alone.
- The kill switch worked exactly as designed: flipping one flag
  stopped new damage in under a minute, no redeploy required.
- Blast radius never reached /search/suggest or /health -- this
  stayed a partial degradation, not a full outage.

WHAT WENT POORLY
- Nothing paged on the leading indicator (cache hit rate); the
  first signal anyone had was the downstream error-rate alert,
  several minutes after the real problem started.
- The flag ramp that set this incident up two days earlier had no
  visibility anywhere on-call naturally checks first.
- One slow route was able to degrade two unrelated, otherwise-
  healthy routes purely because they share one connection pool.

ACTION ITEMS
1. Add CREATE INDEX CONCURRENTLY on product_views(product_id).
   Owner: database/platform   Due: done, same day
2. Give /products/:id/related its own smaller, isolated
   connection pool, separate from the shared 60-connection pool.
   Owner: platform   Due: this sprint
3. Alert directly on cache hit-rate drop as a leading indicator,
   routed to the same rotation as the error-rate alert.
   Owner: observability   Due: this sprint
4. Route feature-flag ramp events into the same on-call-visible
   change feed the deploy pipeline already posts to.
   Owner: platform tooling   Due: this sprint

Step 4 — turn each finding into an owned action item, not an admonition

☺ Like you're 10: "Be more careful" and "remember to check next time" ask a person to do something a system should be doing instead.

The first draft's two action items and the model postmortem's four aren't just differently worded — they're a different kind of sentence. One kind describes a hoped-for future behavior from a person; the other describes a change to a system that holds regardless of who's on call next month.

The last two rows are the most important ones on this page. The blame-shaped draft didn't just phrase its findings badly — it never found them at all, because it stopped looking the moment it had "an engineer" and "the growth team" to write down. Naming a person is a stopping point. Naming a system rarely is, because a system can always be described more specifically — which pool, which index, which alert, which owner, which date — and that specificity is what makes an action item something a future postmortem can actually check off as done instead of rediscovering, unfixed, six weeks later.

That's the drill: one incident, two very different documents, and the only difference between them is which question you asked first. For the culture theory behind why this works — Westrum's typology, why "you build it, you run it" makes the feedback loop tighter, why blameless doesn't mean consequence-free — go back to culture & collaboration. Incident management covers where a postmortem sits in the five-stage incident lifecycle, and measuring success: the DORA metrics covers how an incident like this one counts toward change failure rate. For the diagnosis half of this exact incident, see Drill — Diagnose a Production Incident; for the alert that should have caught the leading indicator here, see Drill — Set Up Meaningful Alerts; and for the sustainable-rotation side of what makes honest incident reporting possible in the first place, see On-Call Culture & Sustainable Operations.