Infrastructure & Operations · Incident management

Incident management

Something will eventually break in production, no matter how good the pipeline or the infrastructure code is — and what determines whether that turns into a five-minute blip or a multi-hour outage is rarely the fix itself. It's whether the team already had a severity scale, an on-call rotation, a clear coordinator, and a communication plan in place before the page went off. This page covers all four, plus where the incident lifecycle ends and the post-incident review begins.

Severity levels: what gates response urgency

Every incident process starts with a severity (or priority) scale, because urgency has to be decided before anyone can decide who gets paged, how fast, and who needs to know. A typical four-tier scale looks like this. SEV1 — a complete outage or a security breach affecting all or nearly all users; every relevant responder is paged immediately, an incident commander is assigned within minutes, and a status page update goes out before root cause is even known. SEV2 — significant degradation: a major feature is down, or a meaningful subset of users can't complete a core workflow; the primary on-call is paged, with a response expected in roughly 15 minutes. SEV3 — a real bug with a workaround or a narrow blast radius; it goes into the team's queue and gets fixed on the next business day, no page required. SEV4 — cosmetic or negligible-impact issues; backlog, no urgency at all.

The scale matters because it's the single input that decides staffing, response-time expectations, and communication cadence all at once. Get the classification wrong in either direction and the rest of the process misfires: over-classifying burns out on-call with 3 a.m. pages for a typo in a marketing page's footer, while under-classifying leaves a genuine outage running for hours before anyone with the authority to fix it even knows it's happening.

On-call rotation: primary, secondary, and escalation

A basic on-call rotation has two roles staffed at any moment. The primary is the first person paged for anything within their team's ownership, usually via a tool like PagerDuty or Opsgenie that routes an alert from the monitoring stack — see monitoring & observability — straight to a phone. The secondary is the deliberate backup: if the primary doesn't acknowledge within a set window, commonly five to fifteen minutes, the tool auto-escalates to the secondary, and if that also goes unacknowledged, further up a defined escalation policy — team lead, then engineering manager, then a wider incident bridge. Without a codified policy, a page that lands on someone asleep or offline simply sits there, unacknowledged, while the incident keeps running.

Rotations typically run a week at a time — long enough to build real context on what's currently fragile, short enough to avoid burnout. The handoff between rotations is not just a calendar flip: the outgoing primary should hand the incoming one a short written note covering anything currently degraded, any risky change scheduled that week, and anything flaky enough to be a known false-alarm risk. A silent handoff with no context transfer is how an already-diagnosed issue gets treated as a fresh mystery by whoever picks up the pager next. Teams spanning multiple time zones often stagger rotations "follow-the-sun" so no single region absorbs every 3 a.m. page.

The incident commander: one person coordinates

Once an incident clears a certain severity, one person is assigned incident commander (IC). The IC's job is coordination, decisions, and communication — not, typically, hands-on-keyboard debugging. They decide what gets tried next when two possible fixes are on the table, decide when to pull in another team, decide when to declare the incident mitigated versus fully resolved, and own the outward-facing narrative of what's happening. The engineers with the deepest context on the failing system stay heads-down on the actual fix; the IC keeps everyone else — other responders, the status page, stakeholders — pointed at one consistent picture of reality instead of several conflicting ones.

A single point of coordination speeds resolution for a concrete reason: without it, incidents regress into duplicated or conflicting effort — two people independently restarting the same service, one engineer rolling back a deploy while another is mid-way through a different mitigation, both burning time re-establishing what the other already tried. The IC is also the one person with standing to make a call under uncertainty — "we're rolling back now, we'll chase root cause after" — and have it stick, rather than the team stalling on consensus while the outage continues. On a larger incident, many playbooks (Google's SRE incident-management practice among them) also name a separate scribe to keep the running timeline, freeing the IC to focus purely on decisions.

Communication during an incident: status pages and stakeholder updates

Two audiences need updates during an incident, and they need different things. External communication runs through a public status page (tools like Atlassian Statuspage or incident.io are common) — a short, factual line stating what's degraded, roughly since when, and that it's being worked on, updated as the picture changes. Internal communication runs through a dedicated channel or bridge call, so support, sales, and leadership can see current impact and expected timeline without pulling a responder out of the incident to ask directly.

Cadence is set by severity, not by how much has actually changed: a SEV1 typically gets an update every 15 to 30 minutes even when the honest content is "still investigating, no change" — silence during an active outage reads as either abandonment or a bigger problem than it actually is, and stakeholders fill an information vacuum with worse guesses than the truth would give them. Lower-severity incidents can update less often, or only at resolution. The IC usually owns this cadence directly or delegates it to the scribe, precisely so the engineers debugging the issue are never interrupted to answer "any update?" in six different channels at once. A good update states three things only: what's known, what's being done, and when the next update lands — not speculation about root cause before it's confirmed.

The incident lifecycle: from detection to review

Every incident moves through the same five stages, and the clock most teams actually optimize is not the one you'd first guess. Detect — an alert fires, ideally from the automated monitoring and alerting stack rather than a customer support ticket. Triage / ack — the on-call responder acknowledges the page and assigns a severity, starting the response-time clock from the scale above. Mitigate — the fastest available action that stops user impact: a rollback (see deployment strategies), a feature flag flipped off, traffic failed over to a healthy region. Mitigation is deliberately not the same as a full fix — it buys time and stops the bleeding, often before root cause is known. Resolve — the underlying issue is actually fixed and the service is confirmed healthy, not just no-longer-visibly-broken. Review — a blameless post-incident review, scheduled once the dust settles, that reconstructs the timeline and produces concrete follow-up actions instead of blame; that generative, blame-free approach is the same cultural mechanism covered in culture & collaboration, and the case study later in this course walks through a full post-incident review end to end.

Most teams track mean-time-to-mitigate (MTTM) as closely as mean-time-to-resolve (MTTR), because from a user's perspective the incident effectively ends at mitigation. Full resolution, and the review that follows it, matter enormously for the team's own long-term reliability, but by the time either happens, user-facing impact has already stopped.

Check your answers

1. Classification is driven by user impact and scope — a SEV1 is a complete outage or security breach affecting nearly all users, while a SEV3 is a real bug with a workaround and limited blast radius. It matters because severity is the single input that decides who gets paged, how fast, and how the incident is communicated — misclassifying it either burns out on-call with unnecessary pages or leaves a real outage under-resourced.
2. The primary is the first person paged for anything in their team's ownership. The secondary is the backup who gets auto-escalated to if the primary doesn't acknowledge within a set window, commonly five to fifteen minutes — defined by the team's escalation policy.
3. The IC's job is coordination, decisions, and communication, not hands-on debugging, because a single point of coordination prevents duplicated or conflicting mitigation effort and keeps everyone pointed at one consistent picture of what's happening. The engineers with the deepest technical context stay focused on the actual fix while the IC manages the rest.
4. Because silence during an active outage reads as either abandonment or a worse problem than it actually is — stakeholders left without updates fill the gap with worse guesses than the truth would give them, so a scheduled cadence, even with "no change" as the content, keeps everyone anchored to reality instead of speculation.