Drill — Diagnose a Production Incident
This drill hands you one incident and nothing else: two alerts, a dashboard, a deploy log, and a stack of raw evidence from a service you've never touched before. It is not an incident-command exercise — the capstone's Part 5 already put you through that choreography end to end, roles, cadence, and postmortem included, on its own single scripted incident. This drill is narrower, and in a lot of real on-call rotations it's the harder half: read symptoms that could plausibly point at four or five different root causes, rule out the ones that are wrong using evidence instead of instinct, land on the one that's actually true, and decide what to do about it before anyone hands you the answer. Work it like it's really 14:02 on a Tuesday and your phone just buzzed.
A kid runs up holding their stomach and says it hurts. A rushed guess says "you ate too much candy" and sends them back to recess. A good nurse checks instead — when did it start, does it hurt in one spot or everywhere, is there a fever, did anyone else who ate the same lunch feel sick too? Most of those checks come back clean and get crossed off the list on purpose, instead of stopping at the first guess that merely sounds plausible — and crossing them off deliberately is what makes the answer she eventually lands on trustworthy instead of lucky. This drill hands you the kid, the thermometer readings, and the lunch menu, and asks you to be the nurse.
Every block below is real enough to run real commands against — copy the log, query, and dashboard blocks into local files (alert.txt, app.log, slow-query.log, whatever names make sense) and actually grep, awk, or read them line by line rather than skimming for the punchline. Work the ten moves in order — later ones assume you've already ruled out what the earlier ones ruled out. Several moves end with a <details> reveal — write your own answer down first, on paper or in a scratch file, before you open it. A conclusion you were talked into by an answer key doesn't teach you anything a conclusion you defended on your own would have.
The setup: catalog-api, a few months after the transformation
☺ Like you're 10: Before you can spot something acting weird, you need to know what "normal" looks like for it.
This is Northwind Retail, the fictional e-commerce company from earlier in this course, some months past the 90-day transformation described there. The practices from that story are all still standing: catalog-api — Postgres-backed, fronted by a shared Redis cache — ships through the trunk-based, canary-gated pipeline from CI/CD pipelines, new capabilities go out behind flags per deployment strategies, and every route reports the four golden signals from monitoring & observability to a shared dashboard. Six replicas of catalog-api share one pgbouncer pool in front of the Postgres primary, sized at 60 connections total. Here's what a normal Tuesday looks like at 13:00 UTC, before any of this starts:
| Signal | Normal Tuesday, 13:00 UTC |
|---|---|
Traffic, all catalog-api routes | ~450 req/s |
Traffic, product-detail family (/products/:id + /products/:id/related) | ~140 req/s |
| Error rate, blended | 0.3% |
| p99 latency, blended | 180ms |
pgbouncer pool in use | 21 / 60 (35%) |
| Redis hit rate, product-detail family | 92% |
Keep this table open in another tab. Every number that follows only means something in contrast to it.
14:02 UTC — the page
☺ Like you're 10: Two alarms go off one minute apart on the same service — before opening a single dashboard, what would even cause that?
ALERT FIRING: CatalogApiHighErrorRate
service: catalog-api
fired: 14:02:11 UTC
value: error ratio 6.1% (threshold 2%, for 3m)
ALERT FIRING: CatalogApiP99LatencyHigh
service: catalog-api
fired: 14:03:04 UTC
value: p99 1,840ms (threshold 500ms, for 3m)Acknowledge both before you do anything else — the same rule incident management gives for any page, so a secondary on-call doesn't get paged behind you while you're still just looking. Then, before opening a single dashboard: write down at least three genuinely different explanations that would produce this exact pair of alerts. A bad deploy today. A traffic spike, hostile or not. A database or infrastructure event. A downstream dependency degrading. Keep that list next to you — the next ten moves exist to cross entries off it with evidence, not to confirm whichever one you liked first.
Move 1 — don't trust the blended number
☺ Like you're 10: "6% of everyone" and "95% of a fifth of everyone" can produce the exact same blended number — you can't tell which one you have without splitting it.
The dashboard at 14:05 UTC, blended across every catalog-api route:
catalog-api — all routes, 5m window, as of 14:05 UTC
traffic: 1,410 req/s (baseline ~450 req/s)
error rate: 9.8% (baseline 0.3%)
p99 latency: 2,210ms (baseline 180ms)
pod restarts: 0A blended 9.8% could mean every route is mildly sick, or one route is on fire and the rest are fine — those are very different incidents that call for very different responses, and the number above can't tell them apart on its own. Split it by route before you decide anything, at two points five minutes apart, and look for two things at once: which route is actually failing, and whether the problem is holding steady or spreading.
Reveal: the per-route split
by route, 1m error-rate window
14:02 UTC 14:07 UTC
/health 0% 0%
/products/:id 0.4% 11.2% ← was clean, now degrading too
/products/:id/related 41.0% 68.5%
/cart-preview 0% 6.9% ← starting to slip
/search/suggest 0% 1.1%Two facts, not one. First: at 14:02 this was concentrated almost entirely on /products/:id/related — the other four routes were still clean. Second, and more urgent: five minutes later it is not staying contained — /products/:id and /cart-preview, two routes with nothing obviously in common with the related-products feature, are now failing too. Whatever is wrong is either directly breaking multiple unrelated routes, or breaking one thing they all secretly share. That distinction is where Move 6 ends up.
Move 2 — check what actually changed (and how far back "recent" really goes)
☺ Like you're 10: "Nothing deployed today" feels like it clears the app of blame, but a change from two or four days ago can still be the reason today is different.
The instinctive first move in any incident is checking what shipped. Do that — but don't stop at "today":
$ gh run list --workflow deploy-catalog-api.yml --branch main --limit 5
STATUS WORKFLOW AGE NOTE
success deploy-catalog-api 4d2h v2.9.0 — add /products/:id/related endpoint
(flag: related_products_widget, 10% rollout)
success deploy-catalog-api 9d6h v2.8.3 — bump pg driver, no behavior change
success deploy-catalog-api 16d1h v2.8.2 — cart-preview caching fix
success deploy-catalog-api 23d4h v2.8.1 — routine dependency bumps
success deploy-catalog-api 30d0h v2.8.0 — checkout webhook retry fix
$ flags history related_products_widget --service catalog-api
2026-08-15 09:14 UTC 10% → 100% changed by: growth-team (config only, no deploy)
2026-08-13 14:30 UTC 0% → 10% changed by: release of v2.9.0Reveal: what this actually rules in and out
Nothing deployed today — that's true, and it's exactly the fact that would make a rushed responder cross "the app" off the list entirely and go hunting for infrastructure or an attack instead. But two things happened recently enough to matter: a code change 4 days ago that added the exact route now failing, and a config-only change 2 days ago that took that route from 10% of traffic to 100% — with no new deploy, so it wouldn't show up if you only ever check the deploy log and never check flag history separately. Neither one is today's change, and neither one is ruled out by "nothing deployed today." Keep both on the list; you haven't found the trigger yet, only a suspect that had the means and the opportunity days before today.
Move 3 — rule out infrastructure before you rule in application code
☺ Like you're 10: Before blaming the code, make sure the ground it's standing on didn't move.
Quick, cheap checks that either end the investigation here or clear the infrastructure layer entirely:
$ aws rds describe-events --source-identifier catalog-db-prod --duration 180
(no events)
$ kubectl -n catalog get pods -l app=catalog-api
NAME READY RESTARTS STATUS
catalog-api-7c9f6d-abc12 1/1 0 Running
catalog-api-7c9f6d-abc13 1/1 0 Running
catalog-api-7c9f6d-abc14 1/1 0 Running
catalog-api-7c9f6d-abc15 1/1 0 Running
catalog-api-7c9f6d-abc16 1/1 0 Running
catalog-api-7c9f6d-abc17 1/1 0 RunningNo RDS failover, no maintenance event, zero pod restarts across all six replicas. Whatever this is, it isn't a dead database, a bad node, or an OOM-killed pod — the process is up and the host layer is quiet. That's an important negative result, not a wasted step: it moves the investigation away from "something under the app broke" and toward "something the app is doing, or something hitting it, changed." Cross infrastructure off your list from Move 1 and move on.
Move 4 — a traffic spike that isn't an attack
☺ Like you're 10: More visitors than usual isn't automatically a break-in — sometimes it's just a really popular flyer.
Traffic is up more than 3x. Before assuming that's hostile, look at where it's landing and where it came from:
#growth-campaigns
[13:45] Priya (Growth): "Back to School" email just went out to the full
list — 2.1M sends, deep-links straight to each product's page
this time instead of the usual homepage banner. Watching
conversion on the product grid 👀
$ traffic, product-detail family (/products/:id + /products/:id/related), 1m buckets
13:40 148 req/s (baseline)
13:50 310 req/s
14:00 980 req/s
14:05 1,390 req/s ← ~3x baseline, still climbing
$ distinct product_id values requested, product-detail family, 14:00–14:05 UTC
640 distinct IDs
(a normal Tuesday's top-20 "trending" set usually covers ~85% of this
traffic; today's requests are spread across the email's entire grid)Reveal: legitimate spike, unusual shape
This traffic is not an attack — it traces cleanly to a real marketing send, timed almost exactly to when the alerts started. But "legitimate" doesn't mean "harmless," and the shape matters as much as the volume: a normal Tuesday's traffic concentrates on a small, cache-friendly set of trending products, while this traffic is spread across 640 distinct product IDs the email linked directly to. A system that's fine under 3x volume on a familiar set of keys can still fall over under the same 3x volume spread across an unfamiliar set — and a deploy log would never have shown you this, because nothing about it is a deploy. It's a Slack message and a traffic-shape change, not a code or infrastructure change, and it's the actual trigger.
Move 5 — the metric nobody thought to check
☺ Like you're 10: A pantry running out isn't visible from outside — you only notice once someone opens the door and finds it bare.
catalog-api's golden-signal dashboard only tracks latency, traffic, errors, and saturation on the app tier — nothing about the cache it depends on. Pull that metric directly:
$ cache hit-rate, product-detail family, 5m rolling window
13:55 UTC 91.6%
14:00 UTC 74.2%
14:05 UTC 38.4% ← collapsing, not just "traffic is up"Reveal: why this number matters more than the traffic count
If the cache were absorbing this spike the way it absorbs a normal one, the hit rate would hold near 92% no matter how high traffic climbed — a cache doesn't care how many requests arrive, only whether it has already seen that exact key. It's collapsing because Move 4's 640 distinct product IDs are mostly not in a cache warmed by the usual concentrated top-sellers. That turns most of this traffic into cache misses — real queries reaching Postgres that would normally never leave Redis. This is the same idea monitoring & observability calls a leading indicator: the hit rate was already telling the real story two full minutes before the pool metric in Move 6 confirms it.
Move 6 — follow it all the way to the database
☺ Like you're 10: Once you know the cache stopped catching the ball, the next question is what's happening to every ball that gets through.
Every cache miss from Move 5 becomes a real query. Check the connection pool and the query log next:
$ psql -h pgbouncer.internal -p 6432 pgbouncer -c "SHOW POOLS;"
database | cl_active | cl_waiting | sv_active | sv_idle | pool_size
-----------+-----------+------------+-----------+---------+-----------
catalog | 340 | 188 | 60 | 0 | 60
(sv_active pinned at pool_size; cl_waiting climbing on every poll)
$ tail -f /var/log/postgresql/slow-query.log
2026-08-17 14:03:02 UTC LOG: duration: 912.44 ms statement:
SELECT product_id, viewer_id FROM product_views
WHERE product_id = $1 ORDER BY viewed_at DESC LIMIT 50
2026-08-17 14:04:41 UTC LOG: duration: 1687.09 ms statement:
SELECT product_id, viewer_id FROM product_views
WHERE product_id = $1 ORDER BY viewed_at DESC LIMIT 50
2026-08-17 14:06:18 UTC LOG: duration: 2431.77 ms statement:
SELECT product_id, viewer_id FROM product_views
WHERE product_id = $1 ORDER BY viewed_at DESC LIMIT 50
$ psql -c "EXPLAIN ANALYZE SELECT product_id, viewer_id FROM product_views
WHERE product_id = 88214 ORDER BY viewed_at DESC LIMIT 50;"
Limit (actual time=2201.3..2431.6 rows=50 loops=1)
-> Sort (actual time=2201.2..2201.2 rows=50 loops=1)
Sort Key: viewed_at DESC
-> Seq Scan on product_views (actual rows=47,928,114)
Filter: (product_id = 88214)
Planning Time: 0.31 ms
Execution Time: 2431.9 msReveal: the chokepoint
sv_active pinned at exactly 60 — the pool's full size — with cl_waiting climbing is a pool that has run out of connections to hand out, not one that's merely busy. EXPLAIN ANALYZE shows why each one takes so long to give back: there is no index on product_views.product_id, so every one of these queries is a sequential scan across all 47.9 million rows of that table, and it's the query v2.9.0's related-products endpoint runs on every cache miss. At normal traffic and a 92% hit rate this query almost never ran — the bug was latent, not absent. At a 38% hit rate it's running hundreds of times a minute, each call pinning a connection in the pool for one to two-plus seconds. And because every catalog-api route shares this same 60-connection pool, a request to /products/:id — a fast, indexed, normally-cached lookup with no bug of its own — now has to queue behind these scans for a connection that isn't there. That's Move 1's spread explained: it isn't that /products/:id broke, it's that it's waiting in the same line as the route that did.
Move 7 — name the root cause, out loud, before you act
☺ Like you're 10: Before you touch anything, say the whole chain in one breath — if you can't, you don't have it yet.
You have every piece. Before opening the reveal, write the full causal chain yourself, start to finish, in one sentence you could say out loud to a second responder who's just joined the call.
Reveal: the causal chain, stated in full
A marketing email at 13:45 UTC drove a 3x traffic spike concentrated on 640 distinct product IDs instead of the usual small trending set, which collapsed the product-detail cache hit rate from 92% to 38%. The resulting flood of cache misses hit /products/:id/related — a route added 4 days ago and ramped to 100% of traffic 2 days ago — whose query against the 47.9-million-row product_views table has no supporting index on product_id, forcing a sequential scan on every call. Those scans, each pinning a database connection for one to over two seconds, filled the shared 60-connection pgbouncer pool and kept it full. Because every catalog-api route draws from that same pool, requests to unrelated, otherwise-healthy routes like /products/:id and /cart-preview began queueing behind the related-products scans for a connection that wasn't available, and started timing out too — which is exactly the spread Move 1 caught between 14:02 and 14:07.
Move 8 — classify it, then propose your mitigation
☺ Like you're 10: Once you know what's actually wrong, the fastest safe fix is rarely the same as the first fix that comes to mind.
First, classify it using the scale from incident management: at 14:02 this was one route failing hard while four stayed clean — a defensible SEV2, a meaningful subset of users on a core workflow. By 14:07 it's three routes degrading and still climbing, with no sign of leveling off — a defensible case for treating it as SEV1 even before it technically reaches "nearly all users," on the reasoning that an actively spreading failure with an unknown ceiling is exactly the kind of situation "declare higher, downgrade later if it turns out contained" exists for. Either call, made and stated out loud with the evidence behind it, beats silence while you decide.
Now propose your mitigation. Here are five real options a responder might reach for. Rank them, and write down why each one you reject is wrong — not just which one you'd pick — before opening the reveal:
- Roll back
catalog-apito the previous deploy. - Scale out
catalog-apireplicas (bump the HPA). - Raise
pgbouncer'spool_sizeon the fly. - Flip the
related_products_widgetfeature flag off. - Manually terminate the long-running backends pinned on the
product_viewsscan.
Reveal: evaluating all five
- Rollback — wrong lever, and slow. Nothing deployed today; the code that introduced the query merged 4 days ago. Rolling back to yesterday's build doesn't remove it, and a redeploy takes minutes you don't have while the pool is already full right now.
- Scale out replicas — wrong lever. The bottleneck is database connections, not application CPU or memory. More
catalog-apipods means more processes competing for the same fixed 60-connection pool — it can makecl_waitingworse, not better. - Raise pool_size — partial, and risky under pressure. It buys headroom without fixing anything, but it doesn't stop the scans themselves from getting slower as concurrency rises, and pushing more concurrent connections at a Postgres primary that's already running hundreds of sequential scans risks tipping the database itself over instead of just the pool in front of it. Worth having in the back pocket; not the first move.
- Flip the flag off — correct, and fastest. This is the fastest safe move available: it's a config change, not a deploy, and it stops new expensive queries from being requested at all within seconds. This is exactly the kill-switch pattern from Feature Flags & Progressive Delivery — the whole reason that flag existed in the first place.
- Terminate the stuck backends — correct, and necessary alongside #4. Flipping the flag off stops new scans from starting, but it does nothing for the connections already pinned mid-scan right now.
pg_terminate_backend()on backends running the offending query past a few seconds frees the pool immediately, instead of waiting for those scans to finish on their own or forpgbouncer's own timeouts to reclaim them one at a time.
The real mitigation is 4 and 5 together, in that order: kill the flag first so the bleeding stops growing, then clear the backends already stuck so the pool actually drains. Watch SHOW POOLS; and the by-route error rates from Move 1 settle before calling it mitigated — not resolved, only mitigated, the same distinction the capstone's own incident draws.
"Scale it out" is the correct reflex for a huge share of real incidents — but only when the thing that's saturated is compute. The moment saturation is a fixed-size shared resource — a connection pool, a thread pool, a rate limit, a queue depth — adding more callers to that resource makes the queue in front of it longer, not shorter. Check which kind of saturation you actually have (Move 6's SHOW POOLS; is exactly that check) before reaching for the scale-out button.
Move 9 — the real fix, once the fire is out
☺ Like you're 10: Turning off the smoke alarm's cause buys you time; you still have to actually fix the wiring.
With the flag off and the pool draining, nothing about the actual bug has changed yet — the capstone's own incident makes exactly this point about aborting a canary. The real fix has two parts, neither of them urgent enough to do under fire:
- Add the missing index.
CREATE INDEX CONCURRENTLY idx_product_views_product_id ON product_views (product_id);— built without locking the table for writes, exactly the online-migration pattern Database Change Management covers for exactly this situation: a schema change that must ship without another outage of its own. - Isolate the blast radius for next time. A single shared pool means one slow route can always starve every other route sharing it — the same bulkhead idea that shows up under a different name in SLOs, Error Budgets & Toil. Giving
/products/:id/relatedits own smaller, separately-sized pool (or a tighter query timeout than the shared default) means a future regression in that one route can only ever break that one route.
This exact shape — a fixed-size shared resource, one noisy caller, and collateral damage to callers who did nothing wrong — isn't unique to database connection pools. Thread pools, HTTP client connection pools, and message-queue consumer slots all fail the identical way. Once you've diagnosed it here, you'll recognize it anywhere a dashboard shows an unrelated route degrading in lockstep with a route that's actually broken.
Two longer-term changes would have caught this before a human ever ran these ten moves. First, an alert on cache hit-rate drop, not just on the downstream symptoms it eventually causes — Move 5's number was actionable two minutes before Move 6's was. Second, a chaos engineering game day that simulates a traffic-shape change (many distinct keys, not just more volume on familiar ones) instead of only ever testing raw throughput — the class of failure this drill just walked through is specifically a cardinality problem, and raw load tests routinely miss it.
Move 10 — where this hands off
☺ Like you're 10: Finding the problem and deciding what to do about it is one job; writing it up so it never happens again is a different one.
This drill stops here, on purpose, at mitigation and a proposed real fix. Two things this incident would also produce in a real rotation are deliberately somebody else's job on this platform: computing this incident's MTTM and MTTR and its contribution to change failure rate belongs to measuring success: the DORA metrics, and writing the actual blameless postmortem — timeline, five whys, owned action items — belongs to the blameless-postmortem drill. Splitting the two like this isn't busywork: diagnosis-and-mitigation is a different skill than incident choreography, which is a different skill again from writing a postmortem nobody reopens unfixed six weeks later, and each one deserves its own rep instead of being buried inside one long scripted story every time.
Ellie the Elephant: Blended error rate says 9.8%. I've got the per-route split too, if you want it.
Foxy: Always. And the deploy log — anything ship today?
Ellie the Elephant: Nothing today. Something four days ago, and a flag change two days ago. Neither one's "today," but neither one's cleared either.
Foxy: Then keep going. Cache hit rate just fell off a cliff — 92 down to 38 in ten minutes. That's not "more traffic," that's traffic hitting things it's never seen before.
Timmy the Turtle: And the pool's pinned at 60 out of 60. Don't scale the pods — you'll just grow the queue in front of the thing that's actually full.
Benny the Beaver: Flag's off, stuck backends killed, pool's draining. I'll put the real index on the migration board — CONCURRENTLY, no lock, no rush.
1. Why did checking "today's deploy log" alone almost lead you to the wrong conclusion, and what did the flag-history log add that the deploy log couldn't show? 2. Name the two metrics that were the real leading indicators here — the ones climbing or falling well before the pool itself showed as exhausted — and explain why each counts as "leading" rather than just another symptom. 3. Of the five mitigation options considered in Move 8, which two were the fastest safe fix, and why were "roll back" and "scale out pods" both the wrong lever here even though each is the correct move in other incidents? 4. What's the real, non-urgent fix, and name one longer-term monitoring or architecture change that would have caught this class of failure before a human had to run these ten moves.
Check your answers
- Nothing deployed today, which is true and would make a rushed responder cross the application off the list entirely. The flag-history log added a config-only change — the flag's ramp from 10% to 100% traffic, 2 days ago — that carries none of the visibility of a deploy but changed production behavior just as much as one would have.
- The Redis cache hit-rate drop (92% → 38%) and the
pgbouncercl_waitingclimb were both leading indicators — they were symptoms of the underlying resource running out before that exhaustion showed up as user-facing errors and latency. A saturation metric predicts the other golden signals are about to get worse; an error rate only reports that they already have. - Flipping the
related_products_widgetfeature flag off and manually terminating the stuck backends pinned on the slow scan. Rollback was wrong because the offending code shipped 4 days ago, not today, so reverting today's build doesn't remove it. Scaling out pods was wrong because the bottleneck was a fixed-size shared connection pool, not application compute — more replicas just means more processes competing for the same 60 connections. - Adding a supporting index on
product_views(product_id)via an online, non-locking migration (CREATE INDEX CONCURRENTLY), plus giving the related-products route its own isolated connection pool so a future regression there can't starve unrelated routes again. Longer term: alerting directly on cache hit-rate drop as a leading indicator, and running a chaos game day that tests a traffic-shape change (many distinct keys) rather than only raw throughput.
Every move in this drill reused a skill this course already named somewhere else — golden signals from monitoring & observability, the kill-switch pattern from feature flags, the online-migration discipline from database change management — the drill's whole point was making you reach for each one yourself, under a dashboard that didn't label which one you'd need. For more reps on the neighboring skills this incident deliberately left out, the blameless-postmortem drill picks up exactly where Move 10 stopped, and the meaningful-alerts drill is the other half of Move 9's closing point — building the alert that would have paged someone on the cache hit-rate drop instead of the pool exhaustion five minutes later.