Case study: an outage post-mortem

Wavelength is a fictional video-streaming service with roughly 4 million daily viewers and a backend split across dozens of independently owned services. This case study follows one Tuesday morning there — a routine deploy, a slow memory leak, a cascading outage, and the incident response and blameless postmortem that followed — as one continuous sequence rather than a set of isolated concepts. Every practice used through the story is one covered earlier in this course, applied under real time pressure instead of in the abstract.

Wavelength, and the deploy that leaked

Wavelength runs its recommendation and playback stack as a set of independently deployed services. playback-api is the customer-facing edge: every app screen — home, browse, the player itself — routes through it, and at peak it serves around 28,000 requests per second. One layer behind it sits watch-history-svc, an internal service nobody outside the platform team thinks about day to day: it records what each viewer has watched and powers the "Continue watching" rail and personalized rankings on the home screen. playback-api calls it synchronously, once per home-screen load, with a 3-second timeout and no fallback response if the call is slow or fails — a design nobody had revisited since the integration shipped fourteen months earlier, back when watch-history-svc was a much smaller, much less load-bearing service.

At 10:14 UTC on a Tuesday, the watch-history team shipped a routine change: a new in-memory cache in front of the lookup that backs the "Continue watching" rail, meant to cut database load during peak hours. The cache key was derived from a per-session token rather than a stable viewer ID, and a bug in the key-generation logic meant almost every lookup produced a cache miss and inserted a new entry that nothing ever evicted. The deploy shipped as a single rolling update straight to 100% of the fleet — watch-history-svc had no canary stage in its pipeline, on the reasoning that an internal service posed little risk to anything outside itself. Every instance in the fleet started leaking memory at roughly the same rate: about 1.6 GB of additional heap per hour under normal traffic.

No circuit breaker, and the cascade begins

For the first eighty minutes, the leak was only a watch-history-svc problem. Heap usage climbed steadily from a normal 40% to 78% between 10:20 and 11:40, visible on a dashboard the team checked maybe once a day — nothing paged anyone, because saturation on that particular service had never been wired to an alert. At 11:42, garbage collection pauses started lengthening as the collector worked harder against a shrinking pool of reclaimable memory, and the lookup endpoint's p99 latency rose from its usual 40ms to over 2,200ms in the space of about five minutes.

playback-api kept calling it at the same rate it always had. With no circuit breaker in front of the call, every one of those slow requests occupied a thread for close to the full 3-second timeout instead of failing fast, and with no bulkhead isolating that call's resources from the rest of the service, the same shared thread pool handled every other endpoint too. By 11:47 the pool was exhausted. From that point, playback-api couldn't serve any request promptly — not just the home-screen rail that watch-history-svc actually powered, but login, browse, and starting playback, none of which had anything to do with watch history. A leak in one internal service, twenty minutes from being noticed at all, had become a platform-wide outage.

Detection, the Incident Commander, and the rollback

At 11:49, two minutes after the pool exhausted, a golden-signal alert fired: playback-api's error rate crossed the symptom-based 5% threshold that pages the primary on-call directly, rather than any threshold tied to watch-history-svc's internal state. The on-call engineer acknowledged within two minutes, spent a few minutes confirming the error rate was real and still climbing, and paged for backup — becoming the Incident Commander at 11:55 as the incident was declared a SEV1. Rather than debugging directly, the IC's first moves were pure coordination: open an incident channel, pull in watch-history-svc's owners as subject-matter responders, and assign a comms lead to post the first status-page update, which went out at 11:58.

By 12:03, the responders had confirmed watch-history-svc's heap was pegged at 97% across the fleet and matched the onset almost exactly to the 10:14 deploy. With that correlation and a service still visibly degrading, the IC made the call at 12:06 to roll back rather than attempt a forward fix under pressure — a live patch to a memory leak is slow to verify and easy to get wrong with the clock running. The rollback completed at 12:11, every watch-history-svc instance restarted on the previous build, and by 12:19 playback-api's error rate and latency were back at baseline. The IC declared the incident resolved at 12:25.

By the time it was over, playback-api had run in a degraded state for 32 minutes — from the pool exhausting at 11:47 to recovery at 12:19 — with error rates peaking near 9%. At Wavelength's peak request volume that translated to roughly 850,000 failed requests, and burned close to 40% of watch-history-svc's monthly error budget in one morning, once the team went back and calculated it — before the incident, nobody had defined an SLO for that service in the first place, because it had never been treated as customer-facing.

The blameless postmortem: five whys to the systemic gap

The postmortem was scheduled for the next morning, drafted jointly by the IC and the watch-history-svc engineer who wrote the caching change. Nobody in the review asked why that engineer didn't catch the bug — the document doesn't name them at all. The question the review kept returning to was what let a bug in one engineer's change reach 100% of production traffic and then take down services that had nothing to do with it. A five whys chain, run during the review, got there in five steps:

Symptom: playback-api was degraded or failing for 32 minutes,
         platform-wide.

Why?    Because playback-api's shared request-handling thread pool
        exhausted waiting on slow calls to watch-history-svc.
Why?    Because there was no circuit breaker in front of that call —
        a slow dependency was treated exactly like a healthy one,
        and playback-api kept dispatching calls into the same pool
        that served every other endpoint.
Why?    Because watch-history-svc itself had become slow: the 10:14
        deploy introduced an unbounded in-memory cache, and rising
        heap pressure lengthened GC pauses until lookup latency was
        roughly 50x normal.
Why?    Because the leaking deploy reached 100% of the fleet in one
        rolling update, with no canary stage and no automated gate
        comparing its signals to the previous build's baseline.
Why?    Because progressive delivery had only ever been adopted for
        customer-facing services at Wavelength — watch-history-svc
        was scoped as internal and low-risk when its pipeline was
        built, and nobody revisited that as playback-api came to
        depend on it synchronously, on every request.

Root cause: two gaps, not one — no circuit breaker to contain a
slow dependency, and no progressive rollout to catch a bad deploy
before it reached every instance at once. Either gap closed alone
would likely have kept this incident from reaching customers.

The chain lands on exactly the two gaps reliability patterns and progressive delivery exist to close, and it never lands on a name. That's not an accident of this particular incident — it's what a five-whys chain is supposed to do when it's run honestly: stop asking why a person made a choice, and keep asking why the system let that choice matter this much.

What Wavelength changed

Three changes came out of the review's action items, all shipped within the following six weeks. First, watch-history-svc got a formal SLO for the first time — 99.9% success and a 200ms p99 latency target on its lookup endpoint, both wired to page instead of living on a dashboard nobody watched in real time. Heap saturation crossing 70% is now its own alert, tuned specifically so the next slow leak gets caught in minutes, not the eighty it took this one.

Second, playback-api's client for watch-history-svc now sits behind a circuit breaker, with a defined fallback — an empty "Continue watching" rail instead of a hung request — for exactly the failure mode that took the platform down that morning:

resilience4j.circuitbreaker:
  instances:
    watchHistoryClient:
      slidingWindowSize: 20
      failureRateThreshold: 50
      waitDurationInOpenState: 30s
      permittedNumberOfCallsInHalfOpenState: 5
      fallbackMethod: emptyContinueWatchingRail

Third, watch-history-svc's pipeline now runs the same canary and automated rollback gate customer-facing services already had — 1% of the fleet for ten minutes, golden-signal deltas and error-budget burn checked against baseline, before a deploy is allowed anywhere near 100%. A leak shaped like this one would now show up as a failed canary on 1% of the fleet, not a fully-leaked fleet forty minutes before anyone looked at a dashboard.

And every quarter, the platform team now runs a game day that specifically rehearses this class of failure — degrading one internal dependency's latency without warning and watching whether the breaker trips, whether the fallback renders, and whether on-call notices before a customer does. The first one, run six weeks after the outage, confirmed the new breaker tripped at the 50% threshold as designed and playback-api stayed fully responsive with an empty rail. The incident that used to take the whole platform down now shows up as a slightly quieter home screen for the length of one game day.