Drill — Incident Response Tabletop
Reading about incident command and actually making the calls under a clock are two different skills, and most of this course teaches the first one. This drill exercises the second. You'll walk through one escalating outage against a fictional service, identity-api, minute by minute — and at five points the story stops and hands you exactly the decision a real on-call engineer would have to make right then, with only the information available at that moment, not a byte more: hold the severity or escalate it, invoke the full Incident Command structure or stay solo, page a team you don't own, tell customers something's wrong before you know why, and declare the fire out. Decide before you look — then open the answer key under each scene to see what a well-run response does at that exact juncture, and why, scored against the same severity matrix and command doctrine the rest of this course teaches, not against a gut feeling. Run it alone against the clock in your head, or read it aloud with your actual on-call rotation and argue about each call before revealing the answer.
Imagine a fire drill where nobody actually leaves the building — the teacher just says "the smoke alarm in the science wing went off once, on its own. What do you do?" and waits for your answer before saying what a well-drilled school actually does. Then the alarm goes off in three more rooms. Then someone says they can smell smoke by the gym. Then, a few minutes later, someone says it's fixed — do you believe them and go back to class, or wait one more minute to be sure? This page is that drill, but for a computer system: the "fire" is a service failing, the "rooms" are dashboards and error rates, and every pause is a moment where a real on-call engineer has to decide, right now, with only what they know so far.
How this drill works
☺ Like you're 10: Read each clock-stamped scene, pick your move, then open the box underneath to see what a well-drilled team would actually do and why.
This is a tabletop exercise, not a hands-on lab — nothing here runs against a cluster, and there's no kubectl in sight. That's deliberate: severity calls, paging decisions, and comms timing are judgment skills that live in an incident commander's head, not in a terminal, and the only way to build them under a clock is to actually be forced to decide before you're allowed to see the "right" answer. Below, the incident unfolds across five timestamped scenes, T+0 through T+31, delivered exactly the way a real on-call engineer receives them — one monitoring readout, one Slack thread, one support-queue count at a time, never the full picture at once. Each scene ends with a single bolded decision. Make the call in your head, or say it out loud if you're running this with your actual rotation, before you open the answer key underneath it — the exercise only works if you commit to a decision first, the same way a real page doesn't wait for you to feel ready. Each decision is worth points against the rubric near the end, scored against what incident management & on-call and Incident Command for Large-Scale Incidents both define as the correct response at that specific juncture — not against whatever felt safest in the moment.
Run it solo first, deciding all three incident-command roles yourself. Then run it again with two other people actually holding Incident Commander, Ops Lead, and Comms Lead for real — the same ten-minute exercise Incident Command for Large-Scale Incidents describes, this page is the fuller, scripted version of it. Enforce the one rule strictly: whoever is IC may only ask questions and make calls, never touch a keyboard to investigate. The interesting disagreements happen in the second run, when two people who'd each call themselves reasonable responders pick different answers to the same scene. If you want the hands-on version of this same territory — a real cluster, a real page, a postmortem written from a real timestamped log — that's Capstone Part 4; this page is the paper rehearsal for the same judgment calls, no cluster required.
The scenario: identity-api, on a Tuesday afternoon
☺ Like you're 10: Meet the service, its promise, its on-call rotation, and who owns which part of the system — before the alarm actually rings.
identity-api is the login and token-issuance service every other surface calls before it'll let a user do anything — it's the "primary API" example named directly in incident management & on-call's own severity matrix. It's owned and operated by the core-platform team, whose on-call rotation you're on today. It depends on a shared Redis fleet — the session cache — that identity-api reads from constantly but doesn't own: that fleet belongs to a separate team, cache-infra, with its own on-call, its own dashboards, and its own change log that core-platform can't see into.
| Thing | Value |
|---|---|
| Service | identity-api — login & token issuance, the critical path for every other surface |
| Availability SLO | 99.95% non-5xx, rolling 30 days — budget: 21.6 min / 30 days (43,200 min × 0.05%) |
| Fast-burn alert | 1h / 5m windows, 14.4× threshold — see multi-window, multi-burn-rate alerting |
| Primary on-call | core-platform-oncall — owns identity-api and its Postgres, today that's you |
| Dependency, not owned | Shared Redis session cache, owned and operated by cache-infra |
Downstream, don't own identity-api | web-frontend, mobile-clients, payments — all call it to validate or issue tokens |
Two definitions you'll need for the first two scenes, paraphrased from incident management & on-call's own matrix — read the full page for the complete tiering, this is only the slice this drill needs:
| Severity | What it actually requires |
|---|---|
| SEV1 | Full outage or critical-path failure — the core product is down or unusable for most users. Auto-declares an incident channel with an IC assigned within minutes. |
| SEV2 | Partial degradation — a subset of users affected, or an SLO threshold crossed without full unavailability. Pages the primary; doesn't by itself convene a formal incident structure. |
T+0 — the first page
☺ Like you're 10: The alarm just went off for real — not for nothing — but "real" and "everyone's affected" aren't the same question yet.
14:02 UTC. Your phone buzzes.
PAGERDUTY — core-platform-oncall
14:02 UTC ALERT TRIGGERED: identity-api fast-burn (page-fast tier)
service: identity-api
windows: 1h / 5m — BOTH BREACHING
5m window: error ratio 2.8% (burn rate ~56x)
1h window: error ratio 1.1% (burn rate ~22x, just crossed)
p95 latency: 210ms (baseline ~55ms)
impact: 97.2% of login attempts still succeeding
runbook: identity-api high-error-rate playbookBoth windows breaching means one thing clearly: this is a real, sustained signal, not a five-minute blip — the AND-gate between the two windows exists precisely so noise alone can't do this. What it does not settle by itself is severity. 2.8% failing is a genuine, budget-eating problem; it is not yet a full outage, and it's nowhere close to "unusable for most users." Decision: do you declare SEV1 right now, or hold at SEV2 while you investigate?
See what a well-run response does
Hold at SEV2. Acknowledge the page, start investigating as primary on-call immediately, and set yourself an explicit, short recheck — three to five minutes — rather than either ignoring the ambiguity or reaching straight for the loudest label available. Per incident management & on-call's own matrix, SEV1 requires a full outage or critical-path failure that's unusable for most users; 97.2% of logins are still succeeding, which is squarely "an SLO threshold crossed without full unavailability" — the textbook definition of SEV2. Declaring SEV1 here isn't reckless, but it is premature: it would auto-open a formal incident channel and put you in Incident Commander mode over a problem that, for all you know at 14:02, might be a single bad node that self-heals in ninety seconds. The two windows breaching already told you this is worth acting on now — that part isn't in question. What's in question is whether the response needs the full machinery yet, and at 2.8% it doesn't. The mistake to watch for isn't holding at SEV2 — it's holding at SEV2 and then not rechecking, which is exactly the "declaring too late" failure mode the deep-dive on Incident Command warns about: by the time someone finally escalates, span of control has already broken.
T+6 — confirmed, and getting worse
☺ Like you're 10: Now three different teams are saying the same thing at once — that's not ambiguous anymore.
14:08 UTC. Your recheck timer goes off. The picture has changed.
DASHBOARD — identity-api, 14:08 UTC
5m window: error ratio 34% (burn rate ~680x)
1h window: error ratio 4.9% (burn rate ~98x — comfortably breaching)
p95 latency: 1.1s
#on-call-general — Slack, last 3 minutes
14:06 @web-frontend-oncall: login button just spinning forever for a
chunk of our users, anyone else seeing this
14:07 @mobile-oncall: token refresh failing app-wide on iOS + Android,
users getting logged out mid-session
14:07 @payments-oncall: can't verify buyer sessions, checkout is
effectively blocked for anyone who isn't already mid-flowThree independent teams, three surfaces, one root cause. A third of all login attempts are failing and climbing. Decision: do you escalate to SEV1 and formally invoke Incident Command — a named Incident Commander, Ops Lead, and Comms Lead — or keep running it solo a little longer while the picture keeps clarifying?
See what a well-run response does
Escalate to SEV1, and invoke the full three-role structure, right now. Two separate thresholds just got crossed in the same six minutes. First, severity: 34% of logins failing, with three downstream teams independently confirming impact on their own surfaces, is exactly "critical-path failure, unusable for most users" — SEV1 is no longer a judgment call, it's what the matrix says this already is. Second, and separately, Incident Command for Large-Scale Incidents lists concrete triggers for escalating past the basic single-responder SEV1 model into the full Incident Commander / Ops Lead / Comms Lead split — and "blast radius spanning more than one team's service" is one of them, satisfied the moment web-frontend, mobile-clients, and payments all posted independently. Any one trigger is sufficient; you don't need to also wait out the 15–30 minute elapsed-time trigger once blast radius alone has already fired. Assign the three roles explicitly, out loud, right now: an Incident Commander who will not touch a keyboard to debug, an Ops Lead who pulls in subject-matter responders and tracks what's tried, and a Comms Lead who owns everything outward-facing from this second forward. The wrong move here is staying heads-down solo on "I've almost got it" — that's precisely the "declaring too late" pattern, and by the time it's obvious to everyone, coordination is already harder to impose than it would have been six minutes ago.
T+9 — the trail leads outside your walls
☺ Like you're 10: The broken part isn't in your house anymore — it's next door, and you don't have a key.
14:11 UTC. Roles are assigned. The Ops Lead has been digging.
# the metric that ends the guessing — identity-api's own view of its cache identity_api_token_cache_hit_ratio # 14:00 UTC (baseline): 0.971 (97.1% of lookups hit the cache) # 14:11 UTC (right now): 0.041 (4.1% — nearly every lookup now misses)
A cache-hit ratio collapsing from 97% to 4% means almost every request that used to be a sub-millisecond Redis lookup is now falling through to a full Postgres query instead — which is exactly what's saturating the connection pool and driving the latency and error spike. But identity-api's own dashboards only show the symptom: the hit-ratio metric reads from the cache, it doesn't show why the cache is unhealthy. The Redis fleet's own topology, replica health, and failover history live in cache-infra's Grafana folder, and core-platform doesn't have edit access to it — or the standing context to read it correctly under pressure. Decision: does the Ops Lead page cache-infra's on-call now, or does core-platform keep digging a little longer to be more certain first?
See what a well-run response does
Page cache-infra now. This is precisely the Ops Lead's job as defined in Incident Command for Large-Scale Incidents: "assembling the subject-matter responders the incident needs — the database owner, the team that shipped the suspect deploy" — a Redis fleet issue owned by a different team is that pattern exactly. You don't need certainty to page; you need a diagnosis that's narrowed to a system boundary you don't own. Paging to confirm or rule out is a legitimate, cheap action — the alternative is core-platform trying to read someone else's Redis topology and failover history from the outside, under pressure, without the standing context to interpret it correctly, which is slower and riskier than just asking the people who live in that dashboard every day. Reluctance to "bother" another team mid-incident is a common instinct and a costly one; every minute spent working around a boundary instead of through the team that owns it is a minute the fast-burn alert's 14.4× burn rate keeps compounding.
T+14 — the outside world notices
☺ Like you're 10: Waiting to say something until you know exactly what happened isn't careful — it just means customers find out from each other instead of from you.
14:16 UTC. No root cause confirmed yet — cache-infra is still triaging their own topology. But the incident is no longer contained to engineering channels.
SUPPORT QUEUE — 14:16 UTC Open tickets tagged "can't log in": 43 (baseline: ~3) Trend: +6 tickets/minute over the last 8 minutes App store reviews, last hour: 11 new 1-star reviews mentioning "won't let me log in" / "stuck on login screen" Status page: no update posted since last week's scheduled maintenance
The Comms Lead has been quiet — heads-down helping track the timeline instead. Decision: does the Comms Lead post an external status-page update right now, even without a confirmed root cause, or wait until the cause is known so the message can be fully accurate?
See what a well-run response does
Post now. An external status update's job is to say "we know, we're actively working on it" — not to explain the mechanism. Incident management & on-call defines the Comms role as owning "everything outward-facing... giving stakeholders a predictable update cadence... so responders aren't repeatedly interrupted to answer 'any update?'" — that job doesn't wait on diagnosis, it exists specifically so diagnosis can proceed uninterrupted while the outside world is still told something. A reasonable first post: "We're aware some users are experiencing login failures and are actively investigating. Next update by 14:31." — impact acknowledged, no speculation about cause, a concrete time for the next update. Waiting for certainty here is the same anti-pattern as "comms silence": stakeholders and customers fill an information vacuum with their own theories — the eleven 1-star reviews and the ticket count climbing at six a minute are exactly that vacuum being filled, and every additional silent minute adds more of both.
T+23 — a fix that looks like a fix
☺ Like you're 10: A green graph for ninety seconds is good news — it isn't yet proof the fire is actually out.
14:23 UTC. Cache-infra's on-call, Priya, finds it: a routine Redis patch triggered four failovers in ninety seconds, and every client — including identity-api — is holding connections to a primary that no longer exists. She repoints the client config at the confirmed-stable node and forces a reconnect.
DASHBOARD — identity-api
14:23 UTC Priya's fix rolls out. Error ratio falling: 51% -> 22% -> 9%
14:25 UTC Error ratio: 4%. Holding there for the last 90 seconds.
p95 latency: 190ms, close to baseline.Ninety seconds of green after twenty-three minutes of red is a good feeling. Decision: does the IC declare the incident resolved right now, or hold for a longer sustained window before calling it?
See what a well-run response does
Hold. Incident Command for Large-Scale Incidents is explicit: "the IC formally declares resolution only once the relevant SLI is confirmed back inside its SLO and the mitigation looks durable — not merely because things seem better for the last five minutes." Ninety seconds is even short of that five-minute bar the doctrine itself calls insufficient. Nothing about a fix rolling out cleanly guarantees every downstream effect of the original fault has finished playing out — a connection pool that was mid-saturation when the root cause got fixed doesn't necessarily recover instantly just because the thing that broke it is gone. The durable-recovery rule exists precisely for this shape of incident: the underlying dependency is fixed, but the blast radius it left behind hasn't finished settling. Watch what happens ninety seconds later before you decide whether "hold" was the right call in the abstract or the right call for a very concrete reason.
T+31 — sustained recovery, and what the postmortem will owe Priya
☺ Like you're 10: The graph dipped again right on schedule — and the postmortem that's coming next has to describe what Priya did without turning it into a verdict on her.
14:26:30 UTC — ninety seconds after the last scene — the graph relapses:
DASHBOARD — identity-api
14:26:30 UTC Error ratio jumps back up: 4% -> 13%.
Cause: identity-api's own local connection pool still
held stale sockets to the dead Redis primary; those
finally time out in a batch and reconnect badly.
14:29 UTC Pool fully cycled onto healthy connections. Error
ratio falling again: 13% -> 2% -> 0.06%.
14:33 UTC Error ratio: 0.04%, sustained for 5+ minutes straight.
IC declares resolved. Comms Lead posts the final
external update.If you held for a sustained window at T+23, this relapse cost you nothing but confirmed the call was correct for a concrete reason, not just a cautious one. If you declared resolved at 4%, you'd have had to walk it back at 14:26:30 — worse for credibility than holding an extra few minutes ever is. Total time from first page to formal resolution: 31 minutes. This incident is now heading straight into a full blameless postmortem, and it's worth sitting with one thing before it's written: Priya's manual repoint was the correct, decisive mitigation — and it's also exactly the kind of fast, judgment-call action a postmortem timeline has to name specifically, with her name on it, without that naming becoming an accusation. Blameless doesn't mean anonymous; it means the timeline can say "Priya repointed the client to the healthy node at 14:23" in exactly the same neutral tense it uses for every other fact, and the root-cause section still has to land on the actual systemic gap — in this case, that neither team had a runbook step for "confirm every downstream client reconnected cleanly" after a Redis failover — rather than on anyone's individual judgment under pressure. The postmortem-writing drill picks up exactly here if you want the next rep.
Score the drill
☺ Like you're 10: Add up your points, then see which band you landed in — and reread the answer keys for anything you got wrong before your next real on-call shift.
Score yourself honestly against what you actually decided before opening each answer key, not what you'd decide now having read it.
| Decision | Points | Full credit requires |
|---|---|---|
| 1. Severity at T+0 | 4 | Held at SEV2, started investigating immediately, set an explicit recheck |
| 2. Escalate at T+6 | 4 | Escalated to SEV1 and invoked the full IC / Ops Lead / Comms Lead split, without waiting for further confirmation |
| 3. Page cache-infra at T+9 | 4 | Paged the owning team the moment the fault localized outside identity-api's own boundary |
| 4. External comms at T+14 | 4 | Posted a status update immediately — impact and "investigating," no speculation on cause |
| 5. Declare resolved at T+23 | 4 | Held for a sustained recovery window instead of declaring on a 90-second improvement |
Notice that "the well-run answer" was never simply "always escalate everything as loudly as possible." At T+0 the correct call was to hold, and at T+6 it was to escalate hard and fast — same responder, same service, two different right answers six minutes apart, because the facts on the ground changed and the policy is written to be read against the facts, not applied as a reflex in either direction.
17–20: this is what a well-run response looks like — the calls you made match the policy, not just the vibe of the moment. 11–16: solid instincts, but go back and reread the decisions you scored lower on before your next real shift — you'll likely find the same pattern repeats. 0–10: reread incident management & on-call and Incident Command for Large-Scale Incidents in full before you're next on the pager for real — the gap here is usually a missing mental model, not a lack of effort.
Pip the Hummingbird: identity-api fast-burn, both windows lit, 2.8% error ratio. Paging core-platform now.
Professor Owl: Both windows breaching just tells us it's real. It doesn't tell us it's a SEV1 yet — 97% of logins are still going through.
Timmy the Turtle: So we hold at SEV2, and somebody actually sets a timer to recheck — not just hope it fixes itself.
Foxy: Six minutes later: three teams reporting, 34% failing. Different question now, isn't it?
Professor Owl: Completely different. SEV1, full structure, right now. I'm IC — and I am not opening a single dashboard to debug.
Timmy the Turtle: And when the graph goes green in ninety seconds — we don't declare resolved on ninety seconds. Five clean minutes, minimum, every time.
1. What's the exact difference between "an SLO threshold crossed without full unavailability" and "critical-path failure, unusable for most users" — and which one changed between T+0 and T+6 in this drill? 2. Name the three formal Incident Command roles T+6's decision invoked, and the one thing the person holding the first of those roles is explicitly forbidden from doing once they're assigned. 3. Why isn't "we don't have a confirmed root cause yet" a valid reason to hold back an external status-page update? 4. What's the exact rule for declaring an incident resolved, and how did this drill's own timeline prove that rule right rather than just theoretically correct?
Check your answers
- The first is SEV2 — a real, budget-eating problem, but most users are still succeeding. The second is SEV1 — the core product is down or unusable for most users. At T+0, 2.8% of logins were failing (SEV2); by T+6, 34% were failing with three separate teams confirming downstream impact, which is squarely SEV1.
- Incident Commander, Ops Lead, and Comms Lead. The Incident Commander is explicitly forbidden from personally debugging — their job is decisions and coordination, and the moment they open a terminal to investigate, nobody is left tracking what's been tried and what's currently in flight.
- Because the Comms Lead's job is communicating impact and status, not explaining mechanism — "we're aware and investigating" is a complete, honest, useful update on its own. Waiting for certainty just leaves customers and stakeholders filling the silence with their own theories, which is exactly what the climbing support queue and the 1-star reviews in this drill's T+14 scene were doing.
- An incident is declared resolved only once the relevant SLI is confirmed back inside its SLO and the mitigation looks durable — not merely because it looked better for the last few minutes. This drill's T+23 scene made that provably right rather than just theoretically right: the metric that looked fixed at 4% relapsed to 13% ninety seconds later, exactly the kind of delayed effect the "hold for a sustained window" rule exists to catch.
Five decisions, one incident, no cluster required — but the judgment underneath is the same judgment Capstone Part 4 asks you to exercise against a real, running service. If a decision here felt shaky, the deeper treatment is one click away: incident management & on-call for the severity matrix and the default single-responder shape, Incident Command for Large-Scale Incidents for exactly when and how to invoke the full three-role structure, and multi-window, multi-burn-rate alerting for why this drill's alert needed both windows breaching before it was allowed to page you at all. When you're ready to write up what happened at 14:33 UTC properly, five whys and all, the postmortem-writing drill is next.