Postmortems & blameless culture

A postmortem is the structured write-up you produce after an incident, and its entire value depends on one design choice: it investigates the conditions that let a failure happen, not the person who triggered it. This page covers why blame is actively counterproductive, what a real postmortem document contains, how to use the five whys to get from a symptom to a systemic cause, and why an incident review is worthless unless its action items actually get done.

Blameless means investigating the system, not the person

The blameless principle is not about being nice — it is a deliberate information-gathering strategy. Almost every outage traces back to a human action somewhere in the chain: someone ran a migration, merged a config change, fat-fingered a flag. If a postmortem's conclusion is "engineer X made a mistake," it has stopped investigating exactly where the useful information starts. The real question is never "who did this" but "what let this be possible" — why did the deploy pipeline allow an unreviewed change to reach production, why didn't the monitoring catch the regression before it paged a customer, why was there no dry-run mode on a command that deletes data.

The practical case for this is about what happens to information, not morale. When engineers expect blame, they hedge: they omit the detail that they were tired, that they skipped a step because the runbook was stale, that they'd noticed the same near-miss two weeks earlier and didn't say anything. Punishing the person who happened to be holding the pager when a systemic flaw finally triggered doesn't fix the flaw — it just teaches everyone else to hide the same flaw better, and guarantees it fires again under someone else's watch. John Allspaw's writing on this at Etsy and the SRE discipline's own origins at Google both converge on the same operating rule: the fixes that come out of a postmortem should be things like better defaults, better tooling, better checks in the deploy pipeline — the same category of countermeasure discussed for review gates in release engineering & progressive delivery — not "be more careful next time," which is not a fix at all because it doesn't change what happens the next time someone is tired, rushed, or new.

What triggers a postmortem, and who writes it

Not every page deserves a full postmortem — most teams set a threshold, commonly any incident that breached an SLO, burned a meaningful fraction of the error budget, paged more than a defined severity floor, or involved customer-visible data loss. Below that bar, a lighter incident note is often enough; above it, a full postmortem is mandatory, not optional, regardless of how minor the fix turned out to be. The document is normally drafted by the incident commander or the primary responder from the event covered in incident management & on-call, then reviewed collaboratively in a meeting with everyone who touched the incident — not written unilaterally by a manager after the fact, and not treated as a performance review input.

The standard postmortem template

A useful postmortem is a document, not a narrative essay, and it follows a consistent template so readers can scan for the part they need. The sections that show up across nearly every mature SRE program:

• Summary — three or four sentences: what broke, for how long, and who was affected. Written so someone skimming a list of fifty past postmortems gets the gist without opening the doc.
• Timeline — a timestamped sequence from the triggering change or event through detection, escalation, mitigation, and resolution, usually in UTC to avoid timezone confusion across a distributed team.
• Root cause(s) — plural, deliberately. Real incidents almost never have one root cause; they have a chain of contributing factors that each independently had to hold for the outage to happen. A migration that dropped a column is a trigger, not the whole story — the absence of a staging-environment check, the absence of a rollback plan, and an alert threshold set too loose to catch the early signal are also root causes, and each one gets its own action item.
• Impact — quantified: requests failed, customers affected, revenue or SLO minutes lost, support tickets opened. Vague impact statements ("some users had a bad experience") make it impossible to prioritize the fix against other work later.
• What went well — genuinely useful, not filler. Fast detection, an on-call runbook that worked as written, a rollback that executed cleanly — these are worth naming so they get reinforced rather than lost to survivorship bias.
• What went poorly — the honest counterpart: slow detection, a missing dashboard, an escalation that took three tries to reach the right person, a runbook step that was wrong.
• Action items — concrete, each with a named owner and a due date, tracked in the same system as regular engineering work (a ticket, not a paragraph of prose promising to "look into" something).

Five whys: drilling from symptom to system

The five whys technique, adapted from Sakichi Toyoda's work at Toyota, forces a postmortem past the first answer, which is almost always superficial. You ask "why" repeatedly, each time treating the previous answer as the new thing to explain, until you hit a cause that is systemic and fixable rather than an individual's action in the moment. Five is a guideline, not a hard rule — some chains resolve in three whys, some need seven — but stopping at the first or second why is the single most common postmortem failure mode, because that's almost always where "an engineer did X" still sits, before you reach the process gap that let X matter.

Symptom: The site was down for 22 minutes.

Why?    Because the database connection pool was exhausted and new
        requests couldn't get a connection.
Why?    Because a deploy introduced a code path that opened a new
        connection per request instead of reusing one from the pool.
Why?    Because the connection-pooling regression wasn't caught by
        the test suite before merge.
Why?    Because there is no load test in CI that exercises connection
        reuse under concurrent traffic — only functional correctness
        is tested, not resource behavior under load.
Why?    Because the team has never had a load-related production
        incident before, so load testing was never prioritized into
        the CI pipeline.

Root cause: CI validates correctness but not resource usage under
concurrent load, so a class of regression like this passes review
and merges undetected until it hits production traffic.

Notice what the chain does not conclude: it never says "the engineer who wrote the connection-handling code should have known better." It lands on a gap in CI coverage — something a fix can actually close, and something that protects the next engineer who writes similar code, not just this one. A five-whys chain that terminates on a person instead of a system or process has gone wrong somewhere upstream; back up and ask why that person was in a position for their individual action to cause an outage-scale failure at all.

Closing the loop: the part that makes the whole exercise worth it

An action item that never gets done is worse than not writing a postmortem at all, because it converts real engineering hours into a document that creates the appearance of a fix without the substance of one — and the next time the same failure mode fires, the postmortem archive will show it was already diagnosed and then dropped. Tracking action items to actual completion, in the same issue tracker as normal work, with the same visibility to the same leadership that reviewed the incident, is what separates a functioning postmortem program from theater. Some teams enforce this with a hard SLA on postmortem action items — for example, all items due within one quarter, tracked on a dashboard, escalated if they slip — and periodically audit the completion rate across all postmortems as a reliability metric in its own right, the same way they'd track any other item in measuring & reporting reliability.

The single clearest signal that a postmortem culture has decayed into ritual is a rising stack of postmortems whose action items sit open for months with no owner following up. If that's happening, the fix isn't writing better postmortems — it's fixing the tracking and accountability loop around the ones you already have.

Postmortem review, not postmortem punishment

The review meeting itself sets the tone for everything above it. A well-run review reads the timeline aloud, asks clarifying questions about the systemic conditions, and explicitly avoids phrasing that assigns individual fault — "the deploy lacked a canary step" instead of "you shipped without canarying." Facilitators who let the conversation drift toward who-did-what are undermining the same information flow the blameless principle exists to protect, even if no one in the room intends it that way. The output of a good review is a shared, accurate understanding of the incident and a short list of well-owned action items — not a verdict.