Slack's January 2021 Outage
On Monday, January 4, 2021 — the first working day of the new year for most of Slack's customers — Slack went down, hard, for the better part of a morning. Slack's own engineering team published a detailed, technical account of what happened: "Slack's Outage on January 4th 2021," written by Laura Nolan, Senior Staff Engineer, on the Slack Engineering blog. This page follows that account closely. It's a capacity-planning story before it's anything else: the trigger wasn't a bug in the usual sense, it was a demand shape that a reasonable-sounding capacity model — "compare today to a typical Monday" — was never going to catch, followed by an autoscaling and networking cascade that turned one saturated link into a company-wide outage.
Imagine a school with one gate everyone uses each morning. Most Mondays the gate handles the rush fine, because most kids' lunch cards are already loaded from Friday. But after a two-week winter break, every kid arrives on the same morning and every lunch card needs refilling right at the gate — a rush the gate was never sized for on an ordinary Monday. Here's the twist: the gatekeeper doesn't watch the line at all; she watches how fast the turnstile clicks. Because the backup outside means fewer kids are actually reaching the turnstile each minute, she reads that as "quiet morning" and sends half her helpers home. Then she panics and calls in twelve hundred substitute helpers all at once — except the sign-in sheet for new helpers can only hold so many names before it runs out of room, so most of them stand around with no badge, unable to actually help. That's Slack's January 2021 outage: a demand spike an ordinary-Monday average never predicted, a monitoring signal that measured the wrong thing and made the problem worse, and a rescue effort that turned out to be too big for its own paperwork.
Where this account comes from
☺ Like you're 10: This is one real company's own detailed writeup of its own bad morning — worth reading exactly as carefully as that deserves, no more and no less.
The primary source for this page is Slack's own engineering postmortem, published on the Slack Engineering blog under the title "Slack's Outage on January 4th 2021" and credited to Laura Nolan, Senior Staff Engineer. It's unusually specific for a public postmortem — exact PST timestamps, the specific resource limits that got hit, the exact mechanism by which the autoscaler made things worse — and this page follows that timeline and mechanism closely rather than paraphrasing loosely. In one place this page also draws on independent, contemporaneous analysis: Lorin Hochstein's "Surfing Complexity" blog wrote a widely-read technical breakdown of the same incident shortly after Slack's post went up, and where this page borrows a framing from that analysis rather than from Slack's own words, it says so. Where Slack's postmortem doesn't give an exact number — how many times normal the traffic surge actually was, for instance — this page doesn't invent one.
The setup: a demand shape a day-of-week average can't see
☺ Like you're 10: "Compare to last Monday" sounds like a sensible rule right up until you ask which Monday you're actually about to have.
January 4, 2021 was the first working Monday after Slack's customers — mostly companies on a standard North American holiday calendar — had spent one to two weeks largely disconnected. Slack's own postmortem attributes the resulting surge to a specific, named mechanism: cold client caches. On an ordinary Monday, most users' Slack clients already hold a reasonably fresh local copy of their workspace state from the previous Friday, so reconnecting pulls down a comparatively small delta. After a multi-week break, that cache is stale for nearly the entire user base at once, so the first reconnect of the new year pulls down substantially more data per user than a routine Monday reconnect does — and it does so for almost everyone in the same narrow window, first thing in the morning. Worth noting only as background, since it isn't the mechanism Slack's own postmortem cites: January 2021 also sat in the middle of a stretch where many of Slack's customers were fully remote, which didn't change the cold-cache mechanism itself but raised how much of a normal workday was riding on Slack being reachable.
The capacity-planning lesson sits right here, before any infrastructure detail. A rolling historical average of "typical Monday traffic," however sophisticated, treats every Monday as a sample from the same distribution. The first Monday after an extended holiday isn't a sample from that distribution — it's a distinct, predictable in advance demand shape, the same way retail treats Black Friday as its own forecasting problem rather than an unusually busy Wednesday. A capacity plan that only asks "how does today compare to a typical day" has no way to see a demand shape it has, almost by definition, never sampled before. See capacity planning & performance for how to build a forecast that accounts for known, recurring-but-irregular events like this one, and Drill — Forecast the Bottleneck to practice spotting this kind of blind spot in a plan before it costs you a morning.
Stage 1 — the network buckles before anyone notices anything else
☺ Like you're 10: The very first sign of trouble wasn't "too many people using Slack" — it was one AWS-managed pipe between Slack's own data centers filling up.
Slack's backend runs across multiple separate AWS VPCs, linked together by AWS Transit Gateway (TGW) — a managed AWS networking service explicitly designed to scale transparently, without Slack's own engineers provisioning its capacity directly. At around 6:57am PST, Slack's external monitoring caught error rates climbing and message-send success dip to roughly 99%, down from Slack's normal baseline of well over 99.999%, as reported in the postmortem itself. The proximate cause was one of these Transit Gateways starting to drop packets under the cold-cache surge described above — a managed piece of infrastructure that was supposed to be Slack's problem to not think about had just become the single input that every downstream system's behavior traced back to.
Stage 2 — the autoscaler reads the wrong signal
☺ Like you're 10: CPU usage dropping looked like "things just got quieter." It actually meant "threads are stuck waiting on a broken network" — and the autoscaler had no way to tell the difference.
Slack's web tier autoscaling was driven by CPU utilization. As packet loss on the saturated Transit Gateway made calls to backend services slower, web-tier threads spent more of their time blocked waiting on the network rather than doing CPU work — so measured CPU utilization fell, exactly as it would if load had genuinely dropped. Per Slack's own postmortem, that drop "initially triggered some automated downscaling": the autoscaler did the only thing a CPU-based policy can do when its one signal falls, and started removing capacity from the web tier at almost the precise moment more capacity was needed, not less.
A scaling signal is a proxy for load — it is not load itself. CPU utilization is cheap to measure and reads correctly almost all the time, which is exactly what makes it dangerous the one time it doesn't: when the real bottleneck sits somewhere the CPU can't see, like a saturated network link, a contended lock, or a slow downstream dependency, "CPU is low" and "we're fine" quietly stop meaning the same thing.
Stage 3 — the overcorrection: roughly 1,200 servers, almost at once
☺ Like you're 10: Once the system finally noticed it badly needed more capacity, it asked for a huge amount of it in one breath instead of a little at a time.
Once a different, faster-reacting signal — thread utilization, not CPU — showed the web tier was genuinely saturated, an emergency scaling rule fired and Slack's infrastructure attempted to bring roughly 1,200 new servers online in the web tier within about a fifteen-minute window, 7:01–7:15am PST by the postmortem's own timeline. Lorin Hochstein's independent "Surfing Complexity" analysis of the incident later nicknamed this kind of blunt, high-magnitude emergency correction a "hammer rule" — not Slack's own term, but a useful name for the pattern: a scaling rule that trades precision for speed once a slower signal has already let things get bad. On paper, "add a lot of capacity fast" is exactly the right instinct once you're behind. In practice here it created an entirely new failure, because the machinery responsible for actually turning those 1,200 requests into working servers had never been asked to handle 1,200 requests in fifteen minutes before — and calling it during a live network incident meant it inherited the very degradation it existed to fix.
Stage 4 — the fix machinery becomes the next bottleneck
☺ Like you're 10: The system built to rescue the web tier turned out to have its own, much smaller capacity limit — and nobody found that limit until 1,200 requests hit it at once.
Slack's provision-service — the internal system that actually turns a requested new server into one that's live and serving traffic — hit two separate resource ceilings under that load, per Slack's own postmortem: primarily the Linux open-files limit on the machines running provision-service itself, and secondarily an AWS account-level quota limit. Both bottlenecks were invisible under normal, roughly-one-at-a-time provisioning traffic, and only surfaced once about 1,200 requests landed within minutes of each other. With provision-service degraded, most of those new instances came up incompletely provisioned — present in AWS, but not actually able to serve traffic — and Slack's load balancers, running their normal health checks against a suddenly huge pool of broken instances, saw failure rates spike. A large share of the "new capacity" that was supposed to fix the incident was, from the load balancer's point of view, simply more things to mark unhealthy.
The same underlying problem compounded the diagnosis itself. Slack's monitoring and dashboarding tooling lived in a separate VPC and depended on the same degraded Transit Gateway to reach the systems it was supposed to be watching — so for a stretch of the incident, the tools engineers needed to understand what was happening were themselves degraded by the exact thing they were trying to diagnose.
| System | What it hit | Why it wasn't caught earlier |
|---|---|---|
| Transit Gateway | Packet loss under a sharp packets-per-second increase | A managed AWS service meant to scale transparently — not something Slack load-tests directly |
| Web tier autoscaler | CPU-based signal read network-blocked threads as idle load | CPU is a proxy for load, not load itself, and had never been fooled this way before |
| provision-service | Linux open-files limit, then an AWS quota limit | Never tested at the burst rate an emergency scale-up event would actually demand |
| Monitoring & dashboards | Ran in a separate VPC, dependent on the same saturated TGW | Its own network dependency wasn't examined for shared fate with what it monitors |
Your emergency "add capacity fast" path is a service too, with its own dependencies and its own capacity limit — and it's usually the least-tested code you have, precisely because it's built to run rarely, at 3am, under conditions nobody wants to rehearse. If you've never fired your actual emergency scale-up rule at its actual emergency scale, you don't know its limit. Slack didn't, until 1,200 requests found it for them.
Timeline: how the outage actually unwound
☺ Like you're 10: Nothing here got fixed by one single change — it took four separate interventions, in sequence, over almost four hours.
| Time (PST) | What happened |
|---|---|
| ~6:57am | External monitoring catches rising error rates; message-send success drops to roughly 99%, down from a normal baseline well above 99.999% |
| ~7:00am | Network degradation from the saturated Transit Gateway is fully underway |
| 7:01–7:15am | An emergency scaling rule attempts to add roughly 1,200 servers to the web tier at once |
| ~8:15am | provision-service's bottlenecks clear enough to provision healthy instances again |
| ~9:15am | The web tier has enough functioning hosts that Slack is degraded, but no longer fully down |
| ~10:40am | AWS engineers manually increase Transit Gateway capacity; the network returns to normal |
End to end, that's roughly three hours and forty-three minutes from the first alert to the network normalizing, with a further stretch of elevated error rates while things fully stabilized. Recovery wasn't one fix — it was several, layered on top of each other: engineers disabled the CPU-based downscaling policy to stop losing capacity mid-incident, cleared out broken and unprovisioned instances rather than leaving them in the load-balancer pool, put load balancers into a more aggressive "panic mode" that spreads traffic across whatever's healthy instead of failing calls outright when too many backends look unhealthy at once, and leaned on circuit breaking and retries — see reliability patterns — to keep the system stable while capacity caught up. The last piece, notably, wasn't something Slack could do alone: Transit Gateway capacity is AWS-managed, so the final fix required AWS engineers to intervene directly.
What Slack changed afterward
☺ Like you're 10: Three concrete commitments, not just "we'll try to do better."
Slack's postmortem closes with specific remediation commitments rather than a general promise to be more careful:
- Move monitoring and dashboards out of the shared-fate path. Relocate monitoring and dashboarding infrastructure into the same VPC as the data stores it watches, removing its dependency on the same Transit Gateway path it needs in order to diagnose problems with that same path.
- Load-test the rescue path, not just steady state. Regularly load-test provision-service at the kind of burst scale an actual emergency scaling event could demand — not only the roughly-one-at-a-time rate it normally handles.
- Re-examine the autoscaling and health-checking configuration. Specifically revisit the risk that a load-based proxy signal like CPU utilization can be fooled by a bottleneck sitting elsewhere in the stack.
Slack's postmortem also reports that AWS, for its part, said it was reviewing its Transit Gateway scaling algorithms' handling of sharp increases in packets-per-second — a direct acknowledgment that the managed service's "scales transparently" promise hadn't held at the rate this surge demanded.
What to steal for your own capacity plan
☺ Like you're 10: None of these lessons need Slack's scale to matter at yours.
- Name the demand shapes your historical average can't see, and plan for them explicitly. Post-holiday Mondays, product launches, marketing pushes, and breaking-news traffic are all predictable well in advance, even though a rolling "typical day" average is structurally blind to them. Treat them the way retail treats Black Friday — with an explicit pre-provisioning plan — rather than trusting autoscaling to notice in time. Practice this on Drill — Forecast the Bottleneck.
- Ask what your autoscaling signal is actually a proxy for, and what could fool it. CPU utilization is cheap and usually right — which is exactly what makes the one time it's wrong dangerous. Cross-check it against a signal that reflects the same underlying reality a different way, the way thread utilization eventually caught what CPU had missed. See monitoring & observability and queueing theory for SRE for how saturation shows up in signals that CPU alone won't surface.
- Load-test the rescue path itself, not just the steady state. provision-service had a real capacity limit nobody had found, because nobody had ever asked it to handle 1,200 requests in fifteen minutes. See production readiness reviews and chaos engineering at scale for how to find that limit deliberately, before an incident finds it for you.
- Keep your monitoring's fate separate from the fate of what it monitors. If your dashboards share a single point of failure with the system they're meant to explain, you lose visibility at exactly the moment you need it most. See network reliability engineering.
- Treat autoscaling policy changes with the same rigor as any other production change. A scaling rule is code that makes fast, consequential decisions with no human in the loop — it deserves the review, testing, and kill-switch discipline this course argues for automation generally on what is SRE and toil & automation.
Pick one service you actually own. Name one predictable-but-irregular demand shape your normal capacity plan doesn't explicitly account for — a launch, a marketing send, a seasonal pattern, the first day back from a break. Now write down, in one sentence each: how much bigger than a routine day you expect that demand to be, and what specifically you'd pre-provision or pre-warm ahead of it rather than trust an autoscaler to catch in real time. If you can't answer the second sentence, you have a hunch, not a capacity plan.
Honest caveats: what doesn't transfer
☺ Like you're 10: A detailed, honest postmortem from one real company still isn't the same thing as an independently audited investigation — read it as what it is.
- This is one company's own account of its own incident. Slack's postmortem is candid and unusually specific, but it isn't an independently audited investigation. The exact magnitude of the traffic surge — how many times normal it actually was — isn't a figure Slack published; treat "significantly higher than a typical Monday, driven by cold caches" as accurate, and treat any specific multiplier you haven't seen sourced directly to Slack as unverified.
- AWS's side of the story is thinner. Slack's postmortem reports that AWS engineers manually increased Transit Gateway capacity and that AWS said it would review its scaling algorithms — but AWS itself never published its own detailed account of why the managed TGW service didn't scale automatically at the rate this event demanded. That half of the mechanism is necessarily seen through Slack's eyes only.
- The specific architecture won't match yours. VPC-per-service traffic connected by AWS Transit Gateway, CPU-based web-tier autoscaling, and a homegrown provision-service are Slack's own 2021 infrastructure choices, not a template. A team on a different cloud, a different network topology, or a different scaling signal won't hit this exact failure mode — even though the transferable lesson (a proxy signal can be fooled; a rescue path needs its own capacity plan) still applies.
- This account is inherently backward-looking. Slack's remediation commitments — relocating dashboards, load-testing provision-service, re-evaluating autoscaling — were stated intentions as of the postmortem's publication in early 2021. This page can't independently confirm they were fully carried out, or that they still describe Slack's current architecture years later.
Foxy: So the autoscaler saw CPU usage drop and concluded... business was slow?
Sol the Sloth: ...Correctly reading a wrong signal. CPU wasn't dropping because load was low. It was dropping because threads were stuck waiting on a saturated network. The number was accurate. The story it told was not.
Benny the Beaver: Then why not just add more servers the instant it noticed the real problem?
Timmy the Turtle: It tried. About twelve hundred of them, in fifteen minutes. And the thing meant to actually turn those requests into working servers had never been asked to handle twelve hundred requests in fifteen minutes before.
Foxy: So the rescue plan needed its own capacity plan.
Professor Owl: Every rescue plan does. The moment you build an "add capacity fast" button, that button becomes a service too — with its own limits, its own dependencies, and its own bad Monday waiting to happen, if nobody's ever pressed it this hard before.
Where this connects in the course
☺ Like you're 10: One outage touches five different lessons — read whichever matches what you're building next.
The forecasting problem at the root of this whole incident — a demand shape historical averages can't see — is capacity planning & performance. The specific failure of a load-based autoscaling signal is monitoring & observability and, for the saturation math underneath it, queueing theory for SRE. The network layer that failed first is network reliability engineering, and the discipline of testing your rescue path before you need it is production readiness reviews and chaos engineering at scale. What actually happens once someone gets paged for an incident like this is incident management & on-call, and writing it up afterward — the way Slack itself did, in public — is postmortems & blameless culture and this course's own worked example in Case study: an outage post-mortem. For other real, cascading-failure outages told the same way, see Meta's 2021 BGP Outage and The 2017 AWS S3 Outage; for another company's public postmortem culture specifically, see Cloudflare's Public Postmortem Culture.
1. What specific mechanism does Slack's own postmortem give for why traffic on January 4, 2021 exceeded what a typical Monday would predict? 2. Why did Slack's CPU-based autoscaler downscale the web tier at almost the worst possible moment, and what does that reveal about using CPU utilization as a capacity signal on its own? 3. What internal system became the next bottleneck once roughly 1,200 new servers were requested at once, and what two specific limits did it hit? 4. Name one way this incident made itself harder to diagnose while it was happening. 5. What's the core capacity-planning lesson for demand shapes that a rolling historical average can't see?
Check your answers
- Cold client caches: after a multi-week holiday break, nearly every user's Slack client held stale local state, so the first reconnect of the new year pulled down far more data per user than a routine Monday reconnect, concentrated in the same narrow morning window.
- Network packet loss from the saturated Transit Gateway made backend calls slower, so web-tier threads spent more time blocked waiting on the network rather than doing CPU work — measured CPU utilization fell as if load had genuinely dropped, and the CPU-based autoscaler had no way to distinguish "idle" from "busy waiting on a saturated network." It shows that a scaling signal is a proxy for load, not load itself, and can be fooled by a bottleneck the signal can't see.
- provision-service, the internal system that turns a requested server into a working one. It hit the Linux open-files limit on the machines running it (the more significant of the two) and a separate AWS account-level quota limit — both invisible under normal, roughly-one-at-a-time provisioning traffic.
- Slack's own monitoring and dashboarding tooling ran in a separate VPC that depended on the same degraded Transit Gateway to reach the systems it was watching, so the tools engineers needed to diagnose the incident were themselves degraded by it.
- A rolling "compare to a typical day" average treats every day as drawn from the same distribution and is structurally blind to predictable-but-irregular demand shapes — like the first working day after an extended holiday — that need their own explicit forecast and pre-provisioning plan rather than reliance on autoscaling alone.