Alert Design & Alert Fatigue

Multi-window, multi-burn-rate alerting gave you the math for deciding when a burning error budget deserves to interrupt a human. This page steps back from the arithmetic and asks the harder, less comfortable question: once you have a page firing, was it ever worth building in the first place? Most alerting configurations don't fail by having too few alerts — they fail by accumulating too many, until the on-call engineer's honest, rational response to most pages is to glance, sigh, and move on. We'll define exactly what makes an alert worth a human's attention, make the case that a page nobody acts on is actively worse than silence, connect alert fatigue to the toil model from earlier in this course while showing where it does something ordinary toil doesn't, and end with a concrete method — bucket every alert from the last quarter by what actually happened after it fired — for taking a noisy configuration and cutting it down to one people trust again.

The actionability test: the one question every alert must answer

☺ Like you're 10: Before you let something interrupt a sleeping human, ask: is there actually something they can do about it right now? If the honest answer is "not really," it shouldn't be a page.

Google's SRE Workbook devotes an entire appendix to alerting philosophy, distilled from an internal essay by Google SRE Rob Ewaschuk called "My Philosophy on Alerting" that's worth reading in full if you can find it. The load-bearing idea, paraphrased rather than quoted, is a three-part test applied to every candidate alert before it's allowed to page a human: is it urgent (does it need attention now, not next week), is it actionable (is there a specific thing a human can do that changes the outcome), and is it real (does it reflect actual user or business impact, not an internal implementation detail nobody outside the team would notice)? Fail any one of the three and the condition doesn't belong on a pager. It might still deserve tracking — as a dashboard panel, a ticket queued for business hours, or a line in a weekly review — but tracking and paging are different mechanisms with different costs, and collapsing them into one is where most noisy configurations start.

This is also where the page/ticket split earns its keep, and it's the same split multi-window burn-rate alerting builds into its four severity tiers: a page interrupts a human's night; a ticket waits for a human's Monday. An alert that's real and actionable but not urgent — a certificate expiring in three weeks, a disk filling at a slow, predictable rate — should almost never be a page. Routing it to a page anyway is the single most common way a team backs into alert fatigue: not through one dramatic bad alert, but through a slow accumulation of perfectly reasonable-sounding conditions that each, individually, seemed worth knowing about right away.

Why a page nobody acts on is worse than no page at all

☺ Like you're 10: A broken smoke detector that never beeps is obviously useless. A smoke detector that beeps constantly for nothing is worse — it looks like protection while quietly training everyone to stop listening to it.

It's tempting to treat a low-value alert as a rounding error — annoying, but strictly better than having no coverage at all. That reasoning is wrong on three separate counts, and each one compounds the others. First, attention is a finite, shared resource on any on-call rotation, and every non-actionable page spends some of it. An engineer who's paged four times in a night for conditions that resolved themselves is measurably slower — and measurably less careful — responding to the fifth page, even if the fifth one is the real incident. Alert volume doesn't just cost the minutes spent acknowledging each page; it costs the sharpness of judgment applied to the one that matters.

Second, a noisy alert creates false coverage — the appearance of monitoring without the substance of it. "We have an alert for that" becomes an answer people give in postmortems and readiness reviews without checking whether the alert has ever once triggered a useful action. A service with twelve pageable alerts and a 4% actionability rate looks, on an inventory spreadsheet, better-monitored than a service with three alerts and a 95% actionability rate. It is almost certainly the opposite. Production readiness reviews exist partly to catch this — "monitoring & alerting" is one of the gated categories precisely because a review that only checks "does an alert exist" and not "does the alert do anything" will wave through exactly this failure mode.

Third, and most corrosively, a non-actionable page trains the humans who receive it. The first few times an engineer gets paged for something they can't act on, they investigate anyway, out of diligence. By the twentieth time, the rational response — and it is rational, given the evidence they've accumulated — is to acknowledge fast, glance without real engagement, and go back to sleep. That learned response doesn't stay scoped to the one bad alert; it generalizes to the pager as a category. This is the mechanism the rest of this page is about, and it's exactly why the next section treats alert fatigue as a form of toil with its own distinct, compounding cost.

Alert fatigue is toil — and it costs something ordinary toil doesn't: trust

☺ Like you're 10: Normal busywork wastes your time, but stops wasting it the moment someone automates it away. A false alarm wastes something that doesn't reset that cleanly — it wastes how much you believe the next alarm.

Toil & automation defines toil as recurring, manual, operational work that scales with the size of the service rather than shrinking through engineering — the kind of work SRE treats as a bug to eliminate, not a fact of life. Responding to a non-actionable page is toil by that exact definition: it's manual (a human has to acknowledge and investigate), it recurs (the same badly-tuned rule fires again next week), and it produces no lasting value (nothing about the system improved because the human looked at it). Cutting alert noise belongs on the same list as scripting a manual restart — it's automation-shaped work that removes recurring human effort from the operational loop.

But alert fatigue is toil with an extra property most other toil doesn't have: it compounds against something that doesn't reset when the underlying rule finally gets fixed. Fix a script that used to require a manual restart, and the very next incident benefits immediately — the toil is just gone. Fix a noisy alert rule after months of false pages, and the immediate technical problem is solved, but the on-call engineer's learned habit of treating pages skeptically doesn't un-learn itself on the same schedule. Trust in a paging system is built slowly, alert by alert, correct page by correct page — and it can be spent in a single bad week far faster than it's rebuilt. This asymmetry is the reason alert fatigue deserves to be treated as a distinct failure mode within the toil model, not just filed under "toil, general," and it's why fixing it is urgent even when each individual noisy page looks small.

The term "alert fatigue" itself is borrowed from clinical medicine, where it describes desensitization to monitor and equipment alarms in hospitals — a body of nursing and patient-safety literature has repeatedly found that the overwhelming majority of ICU alarms are clinically insignificant or false (commonly-cited figures run in the 80–99% range depending on the study and the ward; treat that as an order of magnitude worth knowing, not a number to cite precisely, and check current literature if you need an exact figure). The parallel to on-call is closer than it first sounds: both are environments where a real, rare, high-stakes event is deliberately buried inside a much larger stream of low-value ones, and in both, the documented failure mode isn't that people stop caring — it's that the signal-detection problem becomes genuinely, measurably harder the more noise surrounds the signal.

This is also why alert fatigue belongs in the same conversation as blameless postmortems. An on-call engineer who was slow to react to a real page because the last six pages that week were noise made a reasonable decision given the evidence in front of them — the postmortem's job is to trace that decision back to the alert configuration that produced the evidence, not to the engineer who trusted it.

The quarterly audit: bucket every page by what actually happened next

☺ Like you're 10: Once a quarter, go back through every single page and sort it into a pile based on what someone actually did about it — then look at which piles are huge and ask why.

Fixing a noisy configuration one complained-about alert at a time is slow and political — everyone remembers the alert that annoyed them personally, nobody has a clear picture of the whole rotation. The fix is a structured audit, run on a fixed cadence, that treats "was this page actionable?" as a measurable property of historical data rather than a matter of opinion:

1. Export every page that fired in the window. Ninety days is a reasonable default — long enough to smooth out a quiet or unusually busy month, short enough that the data is still relevant to the current system. Pull it from wherever pages actually live: Alertmanager's history, or your paging tool's incident log (PagerDuty, Opsgenie, Grafana OnCall, VictorOps). You need, at minimum: alert name, fire time, resolve time, who acknowledged it, and whatever resolution note exists.
2. Bucket every single firing — not every alert rule, every individual firing of that rule — into one of five outcomes, defined precisely enough that two different people categorizing the same page land on the same bucket. The table below is the taxonomy; the discipline is applying it consistently.
3. Roll up to a per-rule actionability rate: actioned firings ÷ total firings, for each alert rule. A rule that fired 40 times and was actioned twice has a 5% actionability rate, regardless of how the other 38 firings felt in the moment.
4. Set a floor, in advance, before you see the numbers. Somewhere around 60–75% actionable is a defensible starting bar for a page-tier alert — pick a number your team can defend and commit to it before the audit, so the results decide the outcome rather than whoever's most attached to a given rule.
5. Triage every rule under the floor using the disposition menu in the next section, and write the decision down against the rule, not just in someone's memory.

From bucket to disposition: cutting the noise down to size

☺ Like you're 10: Every noisy alarm gets one of four fates: fix what it's watching, teach it to stop repeating itself, move it off the pager, or delete it outright.

Each bucket from the audit maps to a specific fix, not a generic "reduce noise" instruction:

• Auto-resolved → either the threshold is too sensitive for this service's normal variance (tighten the window or the multiplier — see the sizing method in multi-window burn-rate alerting), or the condition is real but genuinely fixable by a script, in which case the right answer isn't a better alert at all — it's auto-remediation that runs before anyone gets paged, with an alert reserved only for when the auto-remediation itself fails.
• Duplicate / correlated → a structural fix, not a threshold fix. Group related pages together and inhibit the downstream ones once the root-cause alert is already firing — the mechanics are in the next section.
• Non-actionable / informational → downgrade from page to ticket, or replace the page with a dashboard panel someone reviews on a schedule. The underlying condition may be worth tracking; it was never worth a 3 a.m. interruption.
• False positive → a genuine bug in the rule. Fix the query, fix the data source, or — if the rule is measuring a cause instead of a user-visible symptom (see monitoring & observability on symptom-based vs. cause-based alerting) — delete it and replace it with a symptom-based alert that only fires when something a user or the business actually cares about is affected.

A worked slice of an audit, four real-shaped alert rules from one quarter's data:

Notice NodeNotReady: a raw actionability calculation says 89%, which would sail past almost any floor — but the honest number is "3 distinct incidents produced 46 pages," and that gap is itself the finding. The audit isn't complete until you've checked not just whether a firing was actioned, but whether the actions across many firings were actually the same action repeated. This is exactly why the audit has to be re-run — quarterly is the right cadence for most teams, tied to the same rhythm as a toil audit — rather than treated as a one-time cleanup. New services, new dependencies, and new traffic patterns all generate new noise, and an alert configuration that was clean last quarter can quietly drift back toward the state this whole page exists to fix.

Structural noise reduction: grouping, deduplication, and inhibition

☺ Like you're 10: If ten smoke detectors on the same floor all start beeping because of the same fire, you don't need ten separate people running to check it — you need one alarm for the floor.

The duplicate/correlated bucket has a fix that lives in the alerting pipeline, not in any individual rule's threshold. Prometheus Alertmanager's routing layer groups firing alerts that share labels into a single notification, waits a short window to catch related alerts arriving close together, and can be told to hold back — inhibit — a lower-priority alert entirely when a related higher-priority one is already firing:

route:
  receiver: default-pager
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s        # wait briefly for related alerts to arrive together
  group_interval: 5m     # batch further updates to an already-open group
  repeat_interval: 4h    # don't re-notify a still-firing page more often than this
  routes:
    - match:
        severity: ticket
      receiver: default-ticket
      repeat_interval: 24h

inhibit_rules:
  # if the node itself is down, don't also page every alert scoped to that node
  - source_match:
      alertname: NodeDown
    target_match:
      severity: page
    equal: ['cluster', 'node']

That single inhibit_rules block is the structural fix NodeNotReady needed in the worked table above: once a NodeDown alert is firing for a given node, every other page scoped to that same node is suppressed automatically, rather than relying on a human to recognize forty-one pages as one incident in the moment. PagerDuty and Opsgenie solve the same problem one layer up the stack, at the incident-management level rather than the metrics level: PagerDuty's Events API accepts a dedup_key so multiple triggers for the same underlying problem collapse into one open incident instead of one per trigger, and both tools offer alert-grouping features (time-based, content-based, or model-assisted, depending on the product and plan — verify current capabilities against each vendor's own docs, since these features evolve) that catch correlation Alertmanager's label-based rules miss. Layering both — inhibition close to the metric, grouping close to the human — is what turns "one outage, forty-one pages" into "one outage, one page, forty-one lines of context inside it."

The other half of actionability: runbooks and a clear escalation path

☺ Like you're 10: Knowing something's wrong isn't the same as knowing what to do about it — every page needs a "here's what to try" note attached, or it fails the actionability test no matter how well-designed the trigger is.

An alert can pass every test on this page — urgent, real, correctly deduplicated, firing at a sane rate — and still be functionally non-actionable if the person it wakes up has no idea what to do next. Actionability isn't purely a property of the trigger condition; it's a property of the trigger condition plus what the responder can do about it inside the time that matters. The cheapest fix with the highest leverage is a runbook link on every single page-tier alert, following the convention most Prometheus shops already use:

- alert: CheckoutErrorBudgetBurnFast
  expr: |
    sre:checkout_requests:error_ratio1h  > (14.4 * 0.001)
    and
    sre:checkout_requests:error_ratio5m  > (14.4 * 0.001)
  for: 2m
  labels: { severity: page }
  annotations:
    summary: "checkout burning error budget at 14.4x"
    runbook_url: "https://runbooks.acme.io/checkout-error-budget-burn-fast"

A minimally useful runbook entry answers four questions and nothing more — it's a field guide for 3 a.m., not documentation: what does this page mean in plain language; what's the fastest safe mitigation, even a temporary one; who do you escalate to and how if the mitigation doesn't work or isn't yours to run; and which dashboard shows you whether the mitigation is working. Anything beyond that belongs in a wiki, not in the page. Capstone Part 3 — write the on-call runbook is exactly this exercise end to end, and the on-call readiness checklist is the shorter, standing version of "does every page-tier alert actually have one of these" that's worth running before any service goes live, not just after the first bad night proves the gap.

Where this fits

☺ Like you're 10: The last few pages taught you when to page and how urgently; this page taught you how to tell whether the paging you already built is actually worth keeping.

Multi-window, multi-burn-rate alerting is the math that decides when to page; this page is the discipline that decides whether the resulting page deserves to exist at all, and it's worth revisiting after every real incident, not just once a quarter — the worked incident case study is a good place to see this reasoning applied to one full outage end to end. Incident management & on-call covers what happens once a well-designed page actually fires, and postmortems & blameless culture covers how to write up what a page — or a page's absence — revealed, without turning the write-up into blame directed at whoever was holding the pager. If you want to run the audit method on a built system rather than a worked example, Capstone Part 2 — build the monitoring & alerting is where you'll design the alerts this page teaches you to later interrogate, and the alert-design drill is a faster, standalone rep of redesigning one noisy alert from scratch.