On-Call Culture & Sustainable Operations

Incident management covers what happens once a page fires — severity levels, the primary/secondary roles, the incident commander, how you communicate while it's live. This page covers the layer underneath that: whether the rotation answering that page is one a person can survive being part of for more than a few months. Three ideas carry the whole argument. First, a rotation is a designed system with its own math, not just a calendar with names filled in. Second, the fastest way to make on-call survivable is almost never to add more people to it — it's to cut the noise those people are answering in the first place. And third, when a team skips both of those and just tells people to tough it out, the pager doesn't only wear individuals down quietly — it reintroduces, from the inside, the exact incentive split DevOps was invented to dissolve: the engineer who owns the code starts dreading the release that might page them, and every deploy starts looking like risk to be minimized again instead of value to be shipped.

Rotation design: build the schedule so it survives contact with real life

☺ Like you're 10: A fair schedule beats a clever one — everyone takes a fair share of the hard weeks, everyone knows how to swap out of one, and nobody's phone rings without a plan for what happens if they can't answer.

Incident management already introduced the primary, the secondary, and the escalation policy that routes a page when the primary doesn't answer. This page is about designing the roster and the shift pattern underneath those roles so the rotation stays survivable across months, not just describing what happens during any one page. A weekly shift is the default at most organizations for a reason: long enough for a responder to build real context on what's currently fragile, short enough that the accumulated fatigue actually resets before the next one starts. Services with heavy off-hours volume sometimes split further — a day shift and a night shift within the same week — specifically to cap how many consecutive nights any one person absorbs, rather than assuming a week is a week regardless of when the pages land.

Depth matters as much as length. A rotation needs a primary, a secondary who auto-escalates to if the primary doesn't acknowledge, and a named tertiary — usually a lead or manager — for the case where both are unreachable, because "call the secondary" is not itself a complete escalation policy without an answer for what happens if the secondary is also asleep, on a flight, or mid-incident on something else. It also needs deliberate onboarding: someone new to a service shouldn't carry solo primary before shadowing a rotation or two alongside a senior secondary, both because they need the muscle memory and because a rotation that quietly relies on one or two people always being reachable has a bus factor problem wearing an on-call schedule's clothes.

For organizations with genuine engineering presence across two or three time zones, follow-the-sun — handing the pager to whichever region is in daylight — is the single highest-leverage fix available, because it removes the 3 a.m. page from the rotation entirely rather than just making it more bearable. But it only works if the handoff region can actually act, not merely acknowledge and wait. A follow-the-sun rotation where the "covering" region has no context on the service and no authority to touch it is worse than an honest single-region rotation, because it creates the appearance of coverage without the substance of it — and that gap surfaces at the worst possible moment, mid-incident, instead of during planning.

Two more mechanics round out a rotation people can actually live inside. Compensation — a flat weekly stipend, per-incident pay, or time off in lieu of the hours spent responding — should exist in some form; exact structures vary widely by company, role classification, and local labor law (some jurisdictions have specific statutory requirements for on-call compensation), so treat any specific number here as something to verify against your own company's policy and counsel rather than a universal norm. And swap and opt-out mechanisms need to be real, self-serve, and non-punitive — most scheduling tools built into PagerDuty or Opsgenie support one-click swaps, and a rotation with no accepted way to say "not this week" for a new baby, a move, or a family emergency doesn't eliminate that need, it just teaches people to silently under-respond instead of asking.

The headcount math: how small a roster before someone breaks

☺ Like you're 10: The more people share the schedule, the fewer hard weeks any one of them gets — but growing the group doesn't make the phone ring any less, it just spreads out who's holding it.

With N engineers in a single-region weekly rotation, any one of them is primary 1 week in N — a plain fraction, and the entire fairness argument for roster size sits on top of it. Google's Site Reliability Engineering book's chapter on Being On-Call offers two widely-cited outer bounds worth treating as a ceiling rather than a target: no one should spend more than roughly a quarter of their working time on-call, and a sustainable shift averages no more than about two events serious enough to require real action. Neither number is a law — team tolerance, page severity, and how much of the load is genuinely urgent all shift where your own line should sit — but both are useful as an early-warning marker, because crossing them tends to raise burnout risk well before anyone has said the word out loud.

Here's the part the roster math alone can't fix. If a service averages P avoidable pages per week, growing the roster from 4 to 8 halves how often any individual absorbs that P — but the organization still eats P pages every week in aggregate. Roster size manages how fairly the pain is distributed; it does nothing to the actual amount of pain in the system. That distinction is the pivot into the next section, and it's worth sitting with before reaching for "let's hire another rotation slot" as the first move.

Alert fatigue: cut the noise before you add another body to the rotation

☺ Like you're 10: If most alarms turn out to be nothing, people stop trusting any of them — and the one time it's real, they're slow to notice, because they've been trained to expect nothing.

Alert fatigue is the state where responders start treating every page as probably-nothing because most historically have been — which is exactly the state in which the one real page gets a slow, groggy, or missed response. It's the boy-who-cried-wolf failure mode, applied to production paging, and it's the reason this page argues for tuning signal before reaching for more headcount: a noisy alert paged out to eight people is still a noisy alert, just with more people learning to ignore it.

The field's most widely cited alerting philosophy — Rob Ewaschuk's "My Philosophy on Alerting," circulated for years and later published as an appendix to Google's SRE book — reduces to one test worth applying to every existing page rule: an alert should be urgent, important, actionable, and real. If a page's own runbook says "acknowledge and go back to sleep, it clears itself," that isn't an alert doing its job — that's a bug in the alert. This is also where symptom-based versus cause-based alerting earns its keep. A symptom-based alert — the burn-rate pattern from SLOs, error budgets & toil — fires because users are actually experiencing something bad right now. A cause-based alert — CPU above 80%, disk above 90% — fires on a condition that may or may not be hurting anyone yet, and in most organizations it's the single biggest source of noisy, low-value pages, because "CPU is high" is frequently just "the system is working," not "someone needs to act."

Turning that philosophy into a concrete practice is an alert audit: pull every page from the trailing 30 to 90 days and bucket each firing by what actually happened next.

# Pull the last 30 days of pages and bucket each by outcome,
# then look for any rule whose actionable rate is embarrassingly low.
curl -s "https://api.pagerduty.com/incidents?since=$(date -d '30 days ago' -I)" \
  -H "Authorization: Token token=$PD_TOKEN" \
  | jq -r '.incidents[] | [.service.summary, .urgency, .resolve_reason // "no-action-noted"] | @tsv' \
  | sort | uniq -c | sort -rn

# Example output — three rules worth an honest look:
#   41   disk-io-checkout    high   no-action-noted     <- fires, nobody acts, candidate for delete
#   9    checkout-5xx-burn   high   manual-rollback      <- real, actionable, keep
#   23   node-cpu-80pct      high   self-resolved         <- widen window, or downgrade to ticket

Anything outside "actioned" is a candidate to delete, downgrade, or add a suppression rule for — and doing this pass once, honestly, on the actual alert history rather than on gut feel about which alerts "seem important," is routinely the single highest-leverage hour a team can spend on rotation health, ahead of any staffing change.

The noise funnel: dedup, suppress, and route before anything reaches a phone

☺ Like you're 10: Fifty machines shouting about the same broken thing should sound like one voice, not fifty — and something you already know is happening, like a deploy, shouldn't be treated as a surprise.

An alert audit fixes which rules exist. The mechanics below decide what happens to a legitimate signal on its way to a human, and getting them right removes a huge amount of noise that no amount of threshold-tuning can touch on its own. Deduplication collapses every alert firing for the same underlying root cause into one — a hundred replicas complaining about a dead upstream dependency should generate one page, not a hundred. Correlation groups related alerts into a single incident so a shared dependency's outage produces one parent incident with children attached, rather than fanning out a separate page to every team that depends on it. Deploy-aware suppression mutes the noise a known rollout is expected to generate for its bake window, instead of leaving that noise live and quietly teaching the rotation to ignore pages during deploys specifically — which is its own, particularly dangerous form of alert fatigue. And severity-tiered routing applies the SEV scale from incident management to the infrastructure itself: SEV1/SEV2 reach a phone immediately, SEV3 opens a ticket for the next business day, SEV4 lands on a dashboard and pages no one at all.

Both PagerDuty and Opsgenie support this exact shape of routing and suppression natively, and it's also where the escalation policy from incident management physically lives — the policy is what the funnel routes into once a signal has actually cleared the "page a human" bar. Deploy-aware suppression works best as a pipeline step rather than a habit someone has to remember: a stage in CI/CD that opens a maintenance window scoped to the rollout's expected blast radius the moment a deploy starts, and closes it automatically once the canary bake time from your deployment strategy completes.

# Illustrative routing policy — the shape both PagerDuty and
# Opsgenie support natively; adapt field names to your own tool.
service: checkout
routing:
  - match: { severity: SEV1, or: SEV2 }
    action: page_primary_then_escalate
    escalation_policy: checkout-primary-secondary-manager
    dedup_key: "{{alert.fingerprint}}"     # collapse repeats of the same root cause
  - match: { severity: SEV3 }
    action: create_ticket
    queue: checkout-backlog
  - match: { severity: SEV4 }
    action: dashboard_only

suppression:
  - during: deploy_window          # opened/closed automatically by the CI/CD pipeline
    mute: [node-cpu-80pct, pod-restart-single]
    reason: "expected noise during rollout + canary bake"

Measuring on-call health before it breaks, not after

☺ Like you're 10: Check on the schedule the same way you'd check on a person — not just "did anything explode," but "how's everyone actually doing."

None of the above sticks if rotation health lives only in people's memory of their worst week. A handful of numbers, tracked on a normal cadence, catch a rotation drifting toward unsustainable before it costs someone a resignation letter.

That last one matters more than it looks. A rotation can be clean by every quantitative measure and still be miserable, because one dreaded 2 a.m. incident on a Tuesday can outweigh a whole otherwise-quiet week in how a person actually feels about their on-call life — a metric averaged over the week won't show that, but a two-line pulse survey collected right at handoff usually will. Treat rotation load with the same discipline SLOs, error budgets & toil applies to a reliability budget: review it on a schedule, not only after someone complains, and make sure the data actually reaches whoever owns headcount decisions — on-call pain that's invisible to that person never gets funded, no matter how real it is.

How bad on-call quietly reforms the wall of confusion

☺ Like you're 10: When carrying the pager stops feeling okay, people don't usually say so out loud — they just start quietly slowing down every release that might make it ring.

What is DevOps? named the wall of confusion and its fix in four words: you build it, you run it. The team that writes a service also deploys it, monitors it, and carries the pager for it, so release risk and operational risk land on the same people — which is precisely what dissolves the old Dev-versus-Ops incentive split. But that fix only holds if carrying the pager is actually bearable. The moment on-call becomes something people dread, the same alignment that made "you build it, you run it" work starts running in reverse. There's no other team left to throw the wall at — so instead of blaming Ops, the engineer who owns both the code and the pager starts quietly resisting the thing that's hurting them: the release itself.

Watch for the mechanism in this order, because it rarely announces itself. First, "let's not ship on Friday" reappears — not as a genuine, Lean, risk-based judgment call, but as a dread-avoidance reflex dressed up as caution. Then a manual sign-off gate creeps back "just to be safe," in front of a pipeline that used to be trusted to ship on its own. Then batch sizes quietly grow, because nobody particularly wants to ship more often than they strictly have to — which is a direct violation of the Lean pillar of CALMS, and per the DORA research this course's own measuring success page tracks, larger batches and lower deploy frequency correlate with a worse change failure rate and a longer time to restore, not a better one. The wall is back — just self-erected by one team instead of drawn between two, and it's usually visible on your own DORA dashboard as deployment frequency quietly declining and lead time quietly growing, months before anyone actually says the word "burnout" in a meeting.

Fixing it for real: runbooks, automation, and a toil budget with teeth

☺ Like you're 10: Instead of asking tired people to be tougher, teach the system to fix its own most common mistakes — then there's simply less for anyone to be tough about.

The fix is neither of the two easy-sounding answers. It isn't "hire more people" — the headcount math above already showed that only dilutes total pain, it doesn't shrink it. And it isn't "ask people to be more resilient," which quietly asks humans to absorb what is actually a systems problem. The real fix is paying down the toil that's generating the pages in the first place, funded and prioritized with the same deliberateness SLOs, error budgets & toil already gives reliability work during a feature freeze.

Made concrete, that means the on-call handoff from the rotation-design section earlier isn't just a status note — it carries a normal, expected line: "one thing I automated or killed this week," fed straight from the alert audit's non-actionable and self-resolved buckets. It means toil paydown gets the same explicit, named budget line that error-budget-triggered reliability work gets, because it's competing for the same engineering hours as the next roadmap deadline and will lose that fight by default if nobody protects it on purpose. And it means the rotation itself gets a periodic, blameless review separate from any single incident's postmortem — pulling the health metrics from earlier in this page and deciding, before the next quarter's schedule locks in, what actually gets fixed rather than merely logged. That's the same generative, no-blame instinct culture & collaboration already applies to incidents, aimed instead at the schedule that keeps producing them.