Drill — Write the Postmortem

Notice what's absent: no line says a team "should have known better," no line editorializes about whether Priya's fix was tested enough, and the one genuine open question in the raw material — exactly when the patch went out — was resolved against the authoritative changelog (14:01:02 UTC) rather than Dana's unverified 2:15pm guess, which wasn't even the right time zone.

Root cause 1 (primary chain). Why did requests start failing at 14:02? Because the token-cache hit ratio collapsed from 97% to 4%, so nearly every request fell through to Postgres and saturated the connection pool. Why did the hit ratio collapse? Because cache-infra's scheduled maintenance window triggered four failovers on the Redis fleet within roughly 90 seconds, and identity-api's client kept issuing lookups against connections pointed at a primary that no longer existed. (This is the point where a draft that stops is stopping too early — "a Redis failover happened" doesn't explain why it became a customer-facing outage.) Why did the client keep using dead connections instead of detecting the failover? Because identity-api's Redis connection pool has no active health check — no periodic ping, no on-checkout validation — so a dead connection is only discovered when a live request tries to use it and times out. This is also the exact mechanism behind the 14:26:30 relapse: connections that survived the first fix as merely "idle" only failed once something finally tried to use them again, in a batch. Why was there no active health check on a pool this critical? Because reconnect-under-failover behavior was never part of identity-api's pre-production testing — there is no game-day or chaos exercise in its pipeline that actually kills the Redis primary and verifies the client recovers within an acceptable window; only normal-path functional correctness was ever tested. Why wasn't that ever required? Because identity-api's production readiness review treated Redis as infrastructure that was already someone else's problem to harden, and never asked cache-infra and core-platform to jointly verify how a dependent service behaves when that infrastructure fails over — failover testing was scoped entirely to cache-infra's own fleet, never to the services sitting downstream of it. Systemic root cause: identity-api's production readiness review had no requirement to test or configure reconnect behavior for its Redis dependency's failover, so a routine, well-tolerated failover event was free to become a 31-minute customer-facing outage with no client-side safety net to catch it.

Root cause 2 (secondary, contributing chain). Why did it take a page from core-platform — not cache-infra's own monitoring — to reveal that the fleet had just done four failovers in 90 seconds? Because cache-infra has no alert on failover frequency for this fleet; individual failovers are routine and expected, so nobody had ever defined "four in ninety seconds" as anomalous on cache-infra's own dashboards. Why was a burst like that never defined as anomalous? Because the fleet's alerting was designed around steady-state node health at the time it was stood up, and was never revisited once failover frequency became something that could genuinely damage a downstream client's connection pool. Contributing root cause: cache-infra had no burst-rate alert on its own failover events, so the fleet-side signal that would have let them investigate proactively — before any downstream team even noticed — simply didn't exist; they only learned about the storm because core-platform's Ops Lead paged them in. Note what this chain is not saying: it isn't "cache-infra should have caught this" as a criticism of the people on that team — it's a specific, missing piece of monitoring, which is exactly the kind of fixable gap a second root cause is supposed to surface, per postmortems & blameless culture's own guidance that real incidents rarely have just one.

Item 3 is a direct application of chaos engineering — see chaos engineering at scale for how to run a failover game day like this safely and repeatedly rather than as a one-off fire drill; the companion drill Design a Chaos Experiment is good practice for writing the actual experiment plan. Item 4 is the kind of gap production readiness reviews exists specifically to catch before launch, not after an outage. None of these four items is "be more careful next time" — every one of them is something that changes what happens automatically the next time a dependency fails over, whether or not anyone's paying close attention that day.

Summary. identity-api returned elevated error rates for 31 minutes (14:02–14:33 UTC, 2026-08-11) after a routine Redis maintenance window triggered four failovers within ~90 seconds; identity-api's client held stale connections to the dead primary, and a reconnect fix that appeared to resolve the incident within 90 seconds was followed by a second, related spike as the rest of the stale connection pool cycled out. Three downstream teams (web-frontend, mobile-clients, payments) reported customer-visible impact.

Timeline. See Step 1's model timeline above, 14:01:02–14:33:07 UTC.

Root cause(s). See Step 2's model root-cause section above — a primary chain (no active health check on identity-api's Redis pool, no game-day test of reconnect-under-failover, no PRR requirement for it) and a contributing chain (no burst-rate alert on cache-infra's own failover events).

Impact. 1,922,000 total requests during the window; 173,000 (9.0% volume-weighted, peaking at 34%) returned 5xx; ≈2.8 of the service's 21.6-minute monthly error budget consumed (~13% of a full month, in one incident); 43 support tickets (baseline ~3); 11 new 1-star app-store reviews; three downstream teams affected.

What went well. Severity was held correctly at SEV2 during the ambiguous first six minutes, then escalated decisively to full Incident Command the moment blast radius across three teams confirmed it. cache-infra was paged the moment the fault localized outside identity-api's own boundary, without wasted time trying to debug someone else's infrastructure from the outside. An external status update went out at 14:16 acknowledging impact before root cause was known, rather than waiting for certainty. The IC held for a sustained recovery window before declaring resolved, which caught the 14:26:30 relapse instead of being caught by it.

What went poorly. identity-api's Redis connection pool had no active health check, so a failed primary wasn't detected until a live request timed out — twice, once for the original failover and once for the relapse. Reconnect-under-failover behavior was never exercised in pre-production testing. identity-api's production readiness review never required verified failover behavior for this dependency before launch. cache-infra had no alert on its own failover frequency, so the fleet-side anomaly wasn't caught proactively.

Action items. See Step 3's model table above — four items, each with a named owner, a real date, and a ticket number.

• #2 — "Root cause: A Redis failover..." stops at the external event instead of asking why identity-api had no defense against it.
• #1 — "a bit of a repeat offender," "every time cache-infra touches Redis," and "doesn't seem like it was fully tested" all blame a team or a person instead of naming a systemic gap.
• #7 — "reportedly went out around 2:15pm" transcribes Dana's unverified guess — and gets it wrong; the actual authoritative time is 14:01:02 UTC (10:01am EDT), not 2:15pm anything.
• #6 — "Some users experienced login issues" instead of the actual tallied numbers — 173,000 failed requests, 43 tickets, 11 reviews.
• #3 — "We should add better health checks and probably also talk to cache-infra... at some point" has no owner, no date, and isn't specific enough to become a ticket.
• #5 — only one root cause is named at all; cache-infra's own missing burst-rate alert — the contributing chain — never gets asked about.
• #8 — the retro-scheduling reminder has nothing to do with the incident and shouldn't be anywhere near this document.

Seven of the eight mistakes, in one short paragraph — which is exactly why real postmortem review meetings exist: to catch precisely this, before it becomes the permanent record of what the team decided the incident meant.

1. Because it names an external event, not a systemic gap — a routine, well-tolerated Redis failover doesn't explain, by itself, why it was able to become a 31-minute customer-facing outage. The five whys has to keep going until it reaches something identity-api itself was missing: an active health check, a game-day test, and a PRR requirement for dependency-failover behavior.
2. The primary chain: identity-api's Redis client had no active health check, no game-day testing of reconnect-under-failover, and no PRR requirement for it. The secondary, contributing chain: cache-infra had no burst-rate alert on its own failover events, so the storm was only discovered because core-platform paged them in. Neither chain alone fully explains the incident — identity-api's gap let a normal failover reach customers, and cache-infra's gap meant nobody on the fleet side would have caught it proactively either way.
3. A named owner (a real person or specific team, not "we"), a real due date (not "soon" or "at some point"), and enough specificity to become an actual ticket someone could be held to — "add active health checks (periodic ping + idle-connection eviction) to identity-api's Redis client pool" rather than "add better health checks."
4. A guess like that must be verified against an authoritative source before it goes into the final document, not transcribed as-is. Here, that source is PagerDuty's own change-detection note, which recorded the maintenance window's actual start as 14:01:02 UTC — and Dana's guess turned out to be wrong, not just imprecise.