Drill — Redesign a Noisy Alert

search-api — the same service this course's SLO drill already spent thirty days measuring — has a pager rule that fires roughly 40 times a week. It's a single threshold, no burn rate, no second window, written the way most real alerting rules actually get written: someone typed in the SLO's own error allowance as a percentage and called it done. Almost none of those forty pages are worth a human's night. This drill hands you that exact rule, the last week of pages it produced, and the same 30 days of traffic data the SLO drill already gave you — and asks you to replace it with a proper four-tier multi-window burn-rate configuration, then prove, with the actual numbers, that the replacement hits a pages-per-quarter target the old rule never came close to. Work each step before you check it against the walkthrough; the arithmetic is the entire skill being drilled, not just the shape of the YAML.

The scenario: search-api's pager never stops ringing

☺ Like you're 10: The alarm has been going off constantly for a service that's actually fine almost every single time.

The SLO & error-budget drill already established the facts you need: search-api carries a 99.9% non-5xx SLO over a rolling 30-day window, averages about 124.86 requests/minute once health-check traffic is filtered out, and that same 30-day window contained exactly two real incidents — a bad ranking-path deploy on 2026-07-29 and a reindex job that starved a dependency on 2026-08-10. What that drill didn't show you is the alert that was supposed to catch both of those in real time. Here it is, exactly as it's been running in production for months:

groups:
  - name: search-api-legacy-alerts
    rules:
      - alert: SearchAPIHighErrorRate
        expr: |
          sum(rate(http_requests_total{service="search-api",path!="/healthz",code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{service="search-api",path!="/healthz"}[5m]))
          > 0.001
        for: 1m
        labels: { severity: page }
        annotations:
          summary: "search-api 5xx rate above 0.1%"

Notice the threshold: 0.001, or 0.1% — which is exactly 1 − SLO, search-api's own error allowance, typed in directly as a raw percentage on a single 5-minute window. This is precisely the failure mode multi-window, multi-burn-rate alerting already named: "set the threshold at the SLO's raw value and you page constantly for noise." Your first job in this drill is to prove that in numbers, not just recognize the pattern.

Reading the legacy alert: why one stray error is enough

☺ Like you're 10: At this traffic level, a five-minute window is small enough that a single mistake already looks like an emergency.

At search-api's ~124.86 req/min average, a 5-minute window carries roughly 624 requests (124.86 × 5). A single 5xx error inside that window is a ratio of 1 ÷ 624 ≈ 0.16% — already above the 0.1% threshold, on its own, with nothing else wrong. The rule doesn't detect incidents; it detects the existence of any individual failed request, wearing a percentage's clothing. Below is a representative slice of last week's PagerDuty export for SearchAPIHighErrorRate, bucketed with the same five-outcome taxonomy alert design & alert fatigue uses for its own quarterly audit — actioned, auto-resolved, duplicate, non-actionable, false positive:

Step 1 — re-establish the budget you're protecting

☺ Like you're 10: Before you can build a smarter alarm, you need the same numbers the old, dumb one should have been using.

Carry forward exactly what the SLO & error-budget drill already computed, because the burn-rate math in the next step is built entirely on these four numbers: SLO = 99.9%, compliance window = 30 days = 43,200 minutes, 1 − SLO = 0.001, and average valid request rate = 124.86 req/min. That drill also confirmed the window's actual spend: 7,156 total 5xx errors, 57.31 minutes of actual bad-request time against a 43.20-minute allowance — a 14.11-minute overspend, driven almost entirely by the two incidents on 07-29 and 08-10. Nothing about redesigning the alert changes any of those facts; it only changes how fast a human would have found out about them.

Step 2 — set a pages-per-quarter target before you look at the data

☺ Like you're 10: Decide how many alarms are acceptable before you build the alarm, so the numbers don't just get bent to whatever comes out.

The quarterly-audit method in alert design & alert fatigue is explicit that a floor gets set before the audit runs, precisely so the results decide the outcome instead of whoever's most attached to a rule. Do the same thing here, in the other direction: set a ceiling on legitimate pages before backtesting the new design. That same course page has already validated a real benchmark — its worked audit table found CheckoutErrorBudgetBurnFast firing 8 times in a quarter at 88% actionability and called it "keep as-is... doing exactly what it was designed to." Adopt that as search-api's target: no more than about 8 legitimate page notifications per quarter, with the expectation that a smaller, quieter service than checkout should usually land well under it.

Step 3 — derive the four burn-rate tiers

☺ Like you're 10: Same formula this course already taught, plugged in with search-api's own numbers instead of checkout's.

Every multiplier falls out of one formula from multi-window, multi-burn-rate alerting: burn_rate = consumed_% × compliance_window ÷ long_window. Because search-api shares the exact same 30-day / 43,200-minute compliance window as the checkout example that deep dive already worked through, the four resulting multipliers come out identical — not by coincidence, but because the multiplier is a property of the compliance window and the policy percentages, never of the individual service:

The full rule set — recording rules at each of the seven windows the alerts need, then four alerts pairing them with a logical AND:

groups:
  - name: search-api-slo-burn-rate
    rules:
      - record: sre:search_api_requests:error_ratio5m
        expr: |
          sum(rate(http_requests_total{service="search-api",path!="/healthz",code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{service="search-api",path!="/healthz"}[5m]))
      - record: sre:search_api_requests:error_ratio30m
        expr: sum(rate(http_requests_total{service="search-api",path!="/healthz",code=~"5.."}[30m])) / sum(rate(http_requests_total{service="search-api",path!="/healthz"}[30m]))
      - record: sre:search_api_requests:error_ratio1h
        expr: sum(rate(http_requests_total{service="search-api",path!="/healthz",code=~"5.."}[1h])) / sum(rate(http_requests_total{service="search-api",path!="/healthz"}[1h]))
      - record: sre:search_api_requests:error_ratio2h
        expr: sum(rate(http_requests_total{service="search-api",path!="/healthz",code=~"5.."}[2h])) / sum(rate(http_requests_total{service="search-api",path!="/healthz"}[2h]))
      - record: sre:search_api_requests:error_ratio6h
        expr: sum(rate(http_requests_total{service="search-api",path!="/healthz",code=~"5.."}[6h])) / sum(rate(http_requests_total{service="search-api",path!="/healthz"}[6h]))
      - record: sre:search_api_requests:error_ratio24h
        expr: sum(rate(http_requests_total{service="search-api",path!="/healthz",code=~"5.."}[24h])) / sum(rate(http_requests_total{service="search-api",path!="/healthz"}[24h]))
      - record: sre:search_api_requests:error_ratio3d
        expr: sum(rate(http_requests_total{service="search-api",path!="/healthz",code=~"5.."}[3d])) / sum(rate(http_requests_total{service="search-api",path!="/healthz"}[3d]))

      - alert: SearchAPIErrorBudgetBurnFast
        expr: sre:search_api_requests:error_ratio1h  > (14.4 * 0.001) and sre:search_api_requests:error_ratio5m  > (14.4 * 0.001)
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "search-api burning error budget at 14.4x — 2% of the 30-day budget in 1h"
          runbook_url: "https://runbooks.acme.io/search-api-error-budget-burn-fast"

      - alert: SearchAPIErrorBudgetBurnModerate
        expr: sre:search_api_requests:error_ratio6h  > (6 * 0.001) and sre:search_api_requests:error_ratio30m > (6 * 0.001)
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "search-api burning error budget at 6x — 5% of the 30-day budget in 6h"
          runbook_url: "https://runbooks.acme.io/search-api-error-budget-burn-moderate"

      - alert: SearchAPIErrorBudgetBurnSlow
        expr: sre:search_api_requests:error_ratio24h > (3 * 0.001) and sre:search_api_requests:error_ratio2h  > (3 * 0.001)
        for: 15m
        labels: { severity: ticket }
        annotations:
          summary: "search-api burning error budget at 3x — 10% of the 30-day budget in 1d"

      - alert: SearchAPIErrorBudgetBurnSlowest
        expr: sre:search_api_requests:error_ratio3d  > (1 * 0.001) and sre:search_api_requests:error_ratio6h  > (1 * 0.001)
        for: 1h
        labels: { severity: ticket }
        annotations:
          summary: "search-api burning error budget at 1x (nominal) — 10% of the 30-day budget in 3d"

Step 4 — backtest against the same 30 days

☺ Like you're 10: Replay the new alarm against last month exactly as it happened, and count how many times it would really have gone off.

This is the discipline multi-window, multi-burn-rate alerting insists on: "backtest against historical SLI data" before trusting a threshold. Reuse the exact same 30-day dataset the SLO drill handed you — the same daily request and 5xx counts, the same two incidents — and check what each of the four tiers would actually have done.

The 28 ordinary days. Daily 5xx counts across the rest of the month run 18–30 against roughly 167,000–205,000 valid requests a day, which works out to a burn rate of about 0.09× to 0.18× — comfortably under even the slowest ticket tier's 1× threshold. None of the four tiers fires on any of these 28 days. Zero pages, zero tickets, from ordinary variance.

2026-07-29 — the bad deploy. 3,850 errors landed almost entirely inside the 47-minute window between the 14:03 UTC deploy and the 14:50 UTC rollback, against a day-specific rate of ~120.63 req/min — an in-incident error ratio of 3,850 ÷ (47 × 120.63) ≈ 67.9%, a burn rate of ~679×. Assuming that rate holds roughly constant once the bad code path goes fully live (a simplification — a real rolling deploy ramps over a minute or two, which would push this slightly later), solve the window-crossing formula t = threshold × window_minutes ÷ incident_error_rate for the binding 1-hour window: t = (14.4 × 0.001 × 60) ÷ 0.679 ≈ 1.27 minutes. Add the 2-minute for: debounce and the fast tier fires at roughly T+3.3 minutes — versus the 38 minutes it actually took a human to notice via a routine dashboard check. That's about 35 minutes reclaimed, on one incident alone.

2026-08-10 — the reindex job. 2,640 errors over 22 minutes against ~119.86 req/min is a near-total outage — error ratio ≈ 100%, burn rate above 1,000×. Run the same crossing-time formula and both windows breach within roughly a minute; the fast tier fires almost as fast as Prometheus can evaluate it.

Detection speed converts directly into error budget. If the fast tier had actually paged at T+3.3 minutes on 07-29 instead of a human noticing at T+38, and the rollback still took its actual ~9 minutes once someone was on it, the incident runs about 12 minutes instead of 47 — roughly 26% of its original duration. Scale 07-29's error count by that same fraction: 3,850 × 0.2617 ≈ 1,008 errors, or 1,008 ÷ 124.86 ≈ 8.07 minutes of budget-equivalent spend, down from the 30.83 minutes it actually cost. That alone drops the month's total actual spend from 57.31 to about 34.55 minutes — flipping the SLO drill's 14.11-minute overspend into roughly an 8.65-minute surplus. This is an illustrative estimate, not a certainty — it assumes a clean, constant error rate and an unchanged mitigation time — but the direction is the entire point: catching the same bug 35 minutes sooner was worth more budget than the incident itself spent.

Step 5 — close the loop: one inhibit rule and a runbook link

☺ Like you're 10: A severe incident is loud enough to trip two alarms at once — teach them to only wake someone up once.

Run the same crossing-time math on the moderate tier (6× over a 6-hour window) for 07-29's incident and it also breaches — at roughly T+5.2 minutes, just under two minutes behind the fast tier. Left alone, a severe incident like this trips both page-severity alerts within minutes of each other, so two real incidents in a 30-day sample produce four page notifications, not two. Scaled to a quarter, that's roughly 12 notifications — already above the 8-per-quarter target from Step 2, before anything else that quarter even happens. The fix is the same structural one alert design & alert fatigue used for NodeNotReady: an inhibit rule, not a threshold change.

inhibit_rules:
  # once the fast tier is already paging for this service, don't also
  # let the moderate tier open a second, redundant page for it
  - source_match:
      alertname: SearchAPIErrorBudgetBurnFast
    target_match:
      service: search-api
      severity: page
    equal: ['service']

With that one rule in place, two real incidents produce two page notifications, not four — projecting to about 6 per quarter, comfortably under the 8-per-quarter target with headroom for a slightly worse month. The last cheap fix costs nothing but a URL: every page-tier alert above carries a runbook_url annotation, following the same convention alert design & alert fatigue establishes — what the page means, the fastest safe mitigation, who to escalate to, and which dashboard confirms the fix is working. Writing that runbook for real is Capstone Part 3; this drill only requires that the link exists and points somewhere real before the rule ships.

One legacy rule, one afternoon of arithmetic, four tiers, one inhibit rule — and a pager that goes from 40 firings a week to roughly 6 a quarter without losing a single real incident. If any step here felt shaky, the full treatment is one click away: multi-window, multi-burn-rate alerting for the windowing math, alert design & alert fatigue for the audit taxonomy and the inhibition mechanics, and the SLO & error-budget drill for where this same month's numbers first came from. The incident-response tabletop picks up the moment one of these redesigned alerts actually fires for real, and Capstone Part 2 — build the monitoring & alerting is this exact exercise again, on a service and a dataset you build yourself instead of one handed to you.