Cloudflare's Public Postmortem Culture

This page sticks to what Cloudflare has said about itself, in public, on its own engineering blog — mainly the postmortem for the July 2, 2019 outage, a 27-minute global disruption traced to a single bad regular expression in a Web Application Firewall (WAF) rule. That incident is worth a whole case study on its own: a textbook example of catastrophic regex backtracking turning into a worldwide CPU-exhaustion event. But the more interesting fact isn't the bug — it's that Cloudflare wrote up exactly how it happened, named the offending rule, quoted the actual regex, and published it within hours, for anyone on the internet to read. This page covers the incident itself, and then the harder question: what changes about a postmortem's rigor and an organization's accountability when the intended reader isn't just the team that broke it, but every customer, competitor, and security researcher who can read a blog post.

July 2, 2019: twenty-seven minutes, one regex, the whole edge network

☺ Like you're 10: A brand-new security rule went out to every Cloudflare location on Earth at once, and it had one bad line in it that made every computer choke trying to read it.

On July 2, 2019, Cloudflare deployed a new WAF managed rule intended to detect cross-site scripting (XSS) attacks. Cloudflare's WAF rules at the time deployed differently from its normal software — a routine code change was staged out gradually, but a WAF rule update, because it might need to respond to an active attack within minutes, deployed globally and near-instantly to every machine in Cloudflare's edge network. That distinction is the whole story: a bug in an ordinary staged deploy would have shown up in one region first, giving engineers a chance to catch it before it reached everyone. This bug reached everyone at once.

The timeline, minute by minute

Cloudflare's own postmortem, published the same day, laid out a timeline precise to the minute — the level of precision this page is trying to model, not just describe:

Notice what the timeline actually shows: detection was fast (three minutes, via synthetic monitoring and PagerDuty — see alert design & alert fatigue for what makes an alert that fast possible) and mitigation was fast once the cause was found (seven minutes from root cause to global kill). The bulk of the 27 minutes — from 13:45 to 14:00 — was spent figuring out what was on fire, not putting it out. That's the normal shape of an incident: paging is nearly instant, diagnosis is where the clock actually runs.

What actually blew up the CPUs

The rule's regular expression contained a nested pattern that a backtracking regex engine — the kind most engines use by default, with no guaranteed upper bound on matching time — can turn into an exponential-time search on certain inputs. The specific fragment Cloudflare's postmortem singled out as the culprit is short enough to show in full:

.*(?:.*=.*)

/* Two unbounded ".*" wildcards, one nested inside a group that
   also contains an unbounded wildcard on each side of "=". For
   many inputs, there are an enormous number of ways to split the
   string across those wildcards and still fail to match — and a
   backtracking engine tries a huge fraction of them before giving
   up. Cloudflare's own numbers: a 20-character input could take
   the engine more than 555 steps to reject. Longer, adversarial-
   looking inputs made that number explode further, pinning a CPU
   core at 100% per request. */

This class of bug has a name — catastrophic backtracking — and it's a known hazard of any regex containing nested or adjacent unbounded quantifiers over overlapping character classes. It's not exotic; it's one of the most common ways for a regex to go from "correct" to "correct, but computationally explosive" without changing what it matches. What made this instance a global outage rather than a slow request on one server is the deployment mechanism described above: the rule reached every edge machine within the same few minutes, so every machine started burning CPU on the same pathological pattern at once. Cloudflare's postmortem also disclosed a second, compounding fact: a safeguard that limited how much CPU a single WAF rule was allowed to consume had been inadvertently removed in an earlier, unrelated change, sometime before this incident — so nothing caught the runaway regex before it took the CPU to 100%. A single bug plus a silently-missing safety net is a very common shape for a bad-enough outage; see distributed systems reliability fundamentals for why single points of failure so rarely act alone.

The fix, and the five commitments Cloudflare put in writing

☺ Like you're 10: Cloudflare didn't just say "sorry, fixed it" — it wrote down a specific list of promises, each with its own deadline, for anyone to check back on later.

The postmortem didn't stop at explaining the bug. It closed with a list of concrete engineering commitments — the same "action items with an owner and a due date" discipline covered in postmortems & blameless culture, except these owners and dates were published where any reader could later check whether they'd actually happened:

• Re-introduce the CPU-usage protection that had been silently removed — done the same day, before the rest of the fix.
• Manually audit all 3,868 existing WAF rules for the same backtracking pattern — a brute-force but honest response: don't assume this was the only rule with the flaw, go check every one of them.
• Add regex performance profiling to the rule test suite, with a public ETA (July 19, 2019) so a bad rule's CPU cost gets caught before it ships, not after.
• Move toward a regex engine with a runtime guarantee — evaluating options like RE2 or a Rust-based engine that can't backtrack exponentially by construction, rather than continuing to trust rule authors to hand-write safe regexes forever.
• Stage WAF rule rollouts the same way ordinary software deploys already were, while preserving a genuine emergency path for rules that must respond to an active attack within minutes.
• Build an emergency way to take Cloudflare's own dashboard and API off Cloudflare's own edge — a strikingly specific admission that during this outage, the tools engineers needed to fix Cloudflare were themselves running on the thing that was broken.
• Automate status-page updates instead of relying on someone to post to it by hand mid-incident.

What changes when the postmortem is public, not just internal

☺ Like you're 10: Writing something down for your own team is one level of honest. Writing the same thing down for strangers who will fact-check you is a much higher level.

An internal postmortem, covered in full in postmortems & blameless culture, is already supposed to be blameless, precise, and timestamped. Publishing that document externally adds pressures an internal-only version never faces:

• The precision has to survive outside scrutiny. An internal timeline that's off by a few minutes gets quietly corrected in the next meeting. A public timeline gets checked against customers' own monitoring, independent status-tracking sites, and a technical readership — Hacker News among them — that will say so loudly if a number doesn't add up. Writing for that audience raises the bar on getting the facts right the first time, not just eventually.
• The record becomes permanent and citable. An internal doc can be edited, archived, or quietly forgotten. A dated public blog post is a fixed reference point the company itself can't walk back without it being noticed — which, as the next section shows, Cloudflare later used against itself on purpose.
• Blamelessness gets tested at a distance, not just in the room. The internal discipline of naming systems, not people, matters even more publicly — a company that let a public postmortem name an individual engineer would be exposing that person to a scale of consequence no internal review ever could. Publishing forces the "system, not person" habit to hold up outside the room where it's easiest to maintain.
• It becomes a real, external accountability record for customers. A company selling reliability as its product — which is exactly what a CDN and security vendor is selling — has an unusual incentive to prove its own postmortem discipline works, because the readers deciding whether to trust it with their traffic are the same readers who can read the report.

Not a one-off: the practice across six years

☺ Like you're 10: Cloudflare kept doing this for years afterward — and later used its own worst day as the yardstick for how bad a new worst day was.

A single detailed postmortem could be a one-time PR exercise. Cloudflare's blog carries a running, dated archive of these reports stretching across years, covering incidents that range from embarrassing to merely inconvenient — including ones where the honest root cause reflects poorly on Cloudflare's own redundancy design, not just an unlucky external event. A November 2023 postmortem, for instance, disclosed a multi-day control-plane and analytics outage after a core data-center failure exposed a redundant-failover path that didn't work as designed — a considerably less flattering story than "we found the bug and fixed it in 27 minutes," published with the same level of named detail.

The clearest evidence that this is a sustained practice, not a single good day, came on November 18, 2025, when a database permissions change caused a configuration file feeding Cloudflare's Bot Management system to double in size, exceed an internal limit, and take down the core proxy across the network for roughly six hours — by Cloudflare's own account, a considerably larger disruption than 2019. CEO Matthew Prince personally signed that postmortem too, and in it, measured the new incident against the one this page is built around, calling it plainly "Cloudflare's worst outage since 2019." That's a company using its own six-year-old public postmortem as the benchmark for grading itself on a bad day — a comparison that's only possible because the 2019 report was detailed, dated, and never quietly memory-holed.

What to steal for your own postmortem practice

☺ Like you're 10: You don't need a global network to copy the habits — write it down like a stranger will read it, even if only your own team ever does.

• Timestamp everything, to the minute, in UTC. Cloudflare's timeline is legible precisely because every line has a time attached — the same standard practice covered in postmortems & blameless culture. A postmortem without minute-level timestamps can't be checked, internally or externally.
• Quote the actual artifact, not a paraphrase of it. Publishing the real regex, not a vague description of "a bad rule," is what let outside engineers actually verify the catastrophic-backtracking claim instead of taking Cloudflare's word for it. Specificity is what makes a postmortem checkable.
• Write the action items as a public promise with a date, even if you never publish them. Treat the same July-19-ETA discipline as the bar internally — an action item without a deadline attached to a name is a wish, not a fix, whether or not a stranger will ever read it.
• Size your safety nets to the deployment path's actual blast radius. A path that can reach every server on Earth in minutes needs a safety net built for that — not the lighter one that was fine when deploys were slow and staged.
• Practice writing as if a skeptical outsider will read it — before you're forced to. Even a company with no plans to ever publish externally gets a sharper internal postmortem by holding the draft to "would this survive a technical reader who assumes I'm hiding something," the exact bar a public postmortem has no choice but to clear.

Honest caveats

☺ Like you're 10: Cloudflare gets to pick what goes in the report and how it's worded — so read it as a very good, very detailed story the company chose to tell, not as neutral, independently checked fact.

• These are self-published, not independently audited. Every figure in this page — the timestamps, the 3,868 rules audited, the six-hour duration in 2025 — comes from Cloudflare's own blog, written and reviewed by Cloudflare before publication. Treat it as "Cloudflare has said," the same caveat this course applies to every vendor's own numbers, not as a third-party finding.
• A public postmortem is still a curated document. It goes through legal and communications review before it's published, same as any external company statement. That review can shape tone and framing even when the underlying facts are accurate — a genuinely useful public postmortem and a carefully managed one aren't mutually exclusive, but they're not the same thing either.
• Publishing externally doesn't guarantee internal blamelessness. A company can write an excellent, blameless-sounding public document while running a much less blameless conversation internally about who gets held accountable. The public document is evidence of communication discipline; it isn't proof of what happened in the room where the incident was actually reviewed.
• Not every company can afford this level of disclosure, and that's a legitimate constraint, not a moral failing. A regulated bank or a company mid-litigation may have real legal reasons to say much less publicly than Cloudflare does. The lesson to take from this case isn't "always publish everything" — it's "write it as if you might have to," which sharpens the internal document regardless of whether it ever leaves the building.

Where this connects in the course

☺ Like you're 10: This one incident touches several other lessons — read whichever one matches what you're building next.

The blameless-writing discipline this whole page assumes is covered in full in postmortems & blameless culture — read that first if you haven't. The detection-and-paging mechanics behind that three-minute first alert are monitoring & observability and, in more depth, alert design & alert fatigue. The organizational question of what it costs — and what it buys — to make reliability failures a matter of public record is organizational impact of SRE. And for a worked, fictional incident built to practice the same timeline-and-root-cause discipline without needing a real company's outage to learn from, see this course's own case study: an outage post-mortem. For other real, publicly documented incidents covered elsewhere in this course, see Meta's 2021 BGP outage, Slack's January 2021 outage, the 2017 AWS S3 outage, and — for where the blameless-postmortem idea itself came from before Cloudflare ever published one — Etsy & the origin of blameless postmortems.