Incident management & on-call
Every production system eventually breaks, and the difference between a five-minute blip and a multi-hour outage is rarely the code — it's whether the humans responding know their role, their escalation path, and when to declare an emergency instead of filing a ticket for Monday. This page covers how to structure an on-call rotation that doesn't burn people out, the incident command roles that keep a live outage from turning into cross-talk, the severity matrix that decides who gets woken up at 3am, and the difference between a rotation that's merely surviving and one that's actually healthy.
The Incident Command System most software teams use during an outage isn't a tech invention — it's borrowed almost intact from wildland firefighting, formalized in the 1970s after commanders realized crews from different agencies couldn't coordinate a fast-moving fire without a shared chain of command. At the fire, the Incident Commander stands back from the flames and decides where crews go; nobody expects them to personally swing an axe. A Public Information Officer talks to the press and nearby residents so firefighters aren't stopping to answer questions mid-fight. Specialized crews handle the piece of the fire they're trained for. Software incident response uses the exact same shape: an Incident Commander who orchestrates without touching the keyboard, a Comms lead who shields responders from stakeholder pings, and subject-matter engineers who actually chase the bug.
Designing the on-call rotation
Most production rotations run two engineers deep: a primary who receives every page first, and a secondary who is paged automatically if the primary doesn't acknowledge within a set window — a common default is 5 minutes to acknowledge and 15 to actively engage, matching typical PagerDuty and Opsgenie escalation-policy defaults. The secondary covers exactly the failure mode a primary-only rotation misses: the primary is on a flight, in the shower, or their phone died, and without a fallback the page silently drops until someone happens to notice the dashboard is red.
Rotation length trades context retention against fatigue. A day-long rotation forces excessive handoffs and rarely lets anyone build real situational awareness of what changed recently; a month-long rotation concentrates burnout risk in one person and lets accumulating pager fatigue go unnoticed until it's severe. A one-week rotation is the convergence point most engineering organizations land on — long enough to build context on what's currently unstable, short enough that one bad week doesn't define someone's month.
The handoff itself should be a structured event, not a passing Slack message: a short synchronous call or a written doc covering open incidents, services running in a degraded state, recent deploys that haven't fully baked, and any alert currently flapping or known-noisy. A rotation with no real handoff forces every new primary to rediscover context the outgoing primary already had — see monitoring and observability for what should already be on the dashboard before that conversation starts.
Follow-the-sun rotations for global teams
A single-region rotation pages the same engineer at 3am local time roughly one week in every N, where N is the rotation's headcount — for a small team that's often every three or four weeks, indefinitely, for as long as the service exists. Follow-the-sun rotations solve this by staffing on-call from teams distributed across time zones — commonly US, EMEA, and APAC — so the active shift always lands during someone's normal working hours somewhere, with the handoff between regions happening roughly every eight hours as the day rolls forward.
The cost is real. Follow-the-sun requires enough trained engineers in each region to independently run production, not just cover a shift, which means duplicated on-call ramp-up, consistent runbooks and tooling across regions (see the SRE toolchain), and enough shared context that an APAC responder can pick up an incident a US engineer started six hours earlier without re-deriving what's already been tried. For a small team, one humane single-region rotation with a sane escalation policy is more realistic than three understaffed regional ones. Follow-the-sun earns its overhead once a service is large enough, and its 3am-local pages frequent enough, to justify the staffing cost of never having "the tired region."
Incident command: who does what during a live incident
Once a page fires, the responder who picks it up effectively becomes Incident Commander for the response, with two other roles filling in as the incident grows. The Incident Commander (IC) owns the incident: they coordinate the response, decide when to escalate severity, and make the final call on risky mitigations — roll back versus forward-fix — but the IC does not necessarily debug the problem themselves. Running the room — tracking who's doing what, deciding what to try next, keeping the incident moving — is a full cognitive load on its own, and an IC who's also staring at logs loses the coordination thread exactly when it matters most.
The Ops/Comms lead owns everything outward-facing: posting to the status page, answering the executive pinging the incident channel, and giving stakeholders a predictable update cadence — every 15 to 30 minutes is a common default — so responders aren't repeatedly interrupted to answer "any update?" Subject-matter responders are the engineers actually pulled into debugging: the database owner, the team that shipped the suspect deploy. They're brought in by the IC as the incident narrows, not paged by default for every SEV1. This division of labor is what keeps a major outage from degenerating into everyone talking over everyone else, and it's what the timeline built during the incident later feeds into the postmortem.
Severity levels: what gates urgency and who gets paged
Not every alert deserves a phone call at 2am, and a severity matrix is what encodes that judgment ahead of time instead of relitigating it mid-incident. A typical three-tier matrix:
- SEV1 — full outage or critical-path failure. The core product is down or unusable for most users — checkout, login, the primary API. Pages the primary on-call immediately, 24/7, and typically auto-declares an incident channel with an IC assigned within minutes.
- SEV2 — partial degradation. A subset of users or a non-critical feature is affected, or error rates and latency have crossed an SLO threshold without full unavailability. Pages the primary during business hours; overnight it may queue for the next shift, or page at reduced urgency, depending on the org's policy.
- SEV3 — minor issue. A cosmetic bug, a non-user-facing job failing, or a metric drifting but still well inside its error budget. Filed as a ticket, worked during business hours, and never pages anyone.
The matrix does two jobs at once: it protects sleep by refusing to page for anything below the line, and it protects trust in paging by guaranteeing that when the phone does ring, it's real. A team that pages SEV3-grade issues at 3am trains its own engineers to silence or ignore pages — which is exactly the failure mode a well-drawn matrix exists to prevent.
What healthy on-call looks like
A sustainable rotation is measurable, not a vibe. Industry on-call surveys and most mature SRE organizations converge on a rough ceiling somewhere around two or three actionable pages per shift as a health signal — not a hard SLA, but a number worth tracking per rotation over time. Below it, on-call is a manageable responsibility; sustained well above it, the correct conclusion isn't "the team needs to toughen up," it's that the alerting is broken, and the fix is better alert design — tighter thresholds, deduplication, actionable runbooks in monitoring and observability — or automating away the repeat cause outright.
Compensation matters for the same reason: on-call is work even during the hours nothing pages, because it constrains what an engineer can do with their evening — no long drives out of signal range, phone within reach, alcohol limited. Organizations that treat on-call as an unpaid tax on being senior enough to hold a pager burn out their best people fastest; a visible stipend or comp time per rotation is table stakes anywhere running a 24/7 service.
A pager that fires fifteen times in a shift isn't proof the system is fragile — it's proof the alerting is broken. Treat sustained high page volume as a bug to file and fix, never as a fact of on-call life that a rotation just has to endure. Teams that normalize noisy on-call train their best engineers to mute notifications, which is precisely the moment a real SEV1 gets missed.
Every SEV1, and most SEV2s, still deserve a full blameless postmortem — including "why did this page fire the way it did, and was the on-call load reasonable" as one of the questions the postmortem is required to answer, not an afterthought to it.
1. What's the purpose of the secondary on-call role, and what rotation length do most engineering organizations converge on? 2. During a live incident, why shouldn't the Incident Commander also be the one debugging the problem? 3. What separates a SEV1 from a SEV2 in a typical severity matrix, and how does that difference change who gets paged? 4. If a rotation is getting paged far more than a healthy ceiling, what should the team actually do about it?
Check your answers
- The secondary is paged automatically if the primary doesn't acknowledge within a set window, covering the case where the primary is unreachable. Most organizations converge on a one-week rotation as the balance between building context and avoiding burnout.
- Coordinating the response — tracking who's doing what, deciding next steps, keeping the incident moving — is a full cognitive load on its own; an IC who's also debugging loses the coordination thread exactly when it matters most.
- SEV1 is a full outage or critical-path failure affecting most users, paging the primary immediately 24/7 with an IC assigned within minutes. SEV2 is partial degradation or an SLO threshold crossed without full unavailability, which typically pages during business hours only and may queue overnight.
- Treat it as a bug in the alerting to fix — tighten thresholds, deduplicate, automate away the repeat cause — not as a fact of on-call life the rotation has to endure.