Hands-On Labs · Guided Drills

Drill — Set Up Meaningful Alerts

You're about to inherit the most common failure mode in production alerting: a rule file that pages a human every time a gauge wiggles, whether or not a single customer noticed. This drill gives you a real service running on Docker Compose, a genuinely broken cause-based alert file that floods you with noise on purpose, and the tools to replace it with SLO-based, multi-window burn-rate alerts that stay silent under exactly that same noise and page fast the moment a customer is actually affected. No cluster, nothing carried over from the six-part capstone — just Docker, fifteen minutes, and one skill: telling a symptom from a cause before you wire it to someone's phone.

☺ Explain it like I'm 10

Imagine a smoke detector wired to go off every time the kitchen gets a little warmer than usual — the oven preheating, sunlight through the window, someone boiling pasta. Within a week you've unplugged it, because it's cried wolf so many times you can't tell a preheating oven from an actual fire. A good smoke detector doesn't care how warm the kitchen got — it watches for smoke, the thing that actually means something's burning. Today you're wiring a detector that watches the oven temperature (that's the broken version), and then swapping it for one that watches for smoke (that's the fix) — and you'll watch, with your own eyes, the difference between a phone that never stops buzzing and one that only buzzes when it should.

🐦🦥Your hosts for this drill: Pip the Hummingbird & Sol the Sloth — Pip carries the page the instant something fires, so Pip is the first to feel it when a rule file pages for nothing; Sol does the slow, careful arithmetic on the error budget that decides which alerts actually deserve Pip's wings.
⚠ Before you start

You need Docker and Docker Compose, curl, and a shell. promtool and amtool are used later for linting and routing checks — you don't need to install anything for them: the official prom/prometheus and prom/alertmanager images ship both binaries inside themselves, so docker compose exec prometheus promtool ... and docker compose exec alertmanager amtool ... work with zero extra setup. (If you'd rather run them natively, brew install prometheus alertmanager installs both CLIs directly.) Prometheus and Alertmanager's flags and config schema do drift between minors — if a command below errors, check docker compose exec prometheus promtool --version and the image tag pinned in the compose file below, and adapt. Everything here is a throwaway project — docker compose down -v when you're done and nothing lingers.

How this drill works

☺ Like you're 10: First you feel the pain of a noisy alert file for real, then you build the fix, then you prove the fix actually holds under a real fake outage.

You'll stand up one small service — checkout-api — with Prometheus scraping it and Alertmanager routing whatever fires. First you load a broken alert rule file that thresholds on internal implementation details: queue depth, CPU, GC pause. You'll watch it page nonstop while the service serves 100% of requests successfully, because none of those rules have any idea whether a user is happy. Then you'll define an actual SLO, write multi-window burn-rate alerts against it — the same formula and the same 99.9%/14.4×/6×/1× numbers from SLOs, Error Budgets & Toil, now pointed at a service you're actually running — and prove two things back to back: the new rules stay completely silent while the exact same internal noise keeps jittering, and they page fast the moment you inject a real, user-facing failure with a chaos endpoint. This is the hands-on counterpart to on-call culture & sustainable operations' section on alert fatigue — that page names the disease; this drill has you build the cure with your own hands.

Stand up the scratch service

☺ Like you're 10: One tiny checkout service, wired so Prometheus can watch it and you can flip a "break things" switch whenever you want.

Make a fresh project directory and drop in the four files below. checkout-api is a small Express service that serves GET /checkout/:id, exposes Prometheus metrics on /metrics, and has one deliberately dangerous knob — POST /admin/chaos — that dials up the error rate on demand.

mkdir alerting-drill && cd alerting-drill
mkdir checkout-api
npm init -y --prefix checkout-api
npm install --prefix checkout-api express prom-client
// checkout-api/server.js
const express = require("express");
const client = require("prom-client");

const app = express();
const register = client.register;

// --- the symptom plane: what a user actually experiences ---
const httpRequests = new client.Counter({
  name: "http_requests_total",
  help: "Total HTTP requests, labeled by outcome",
  labelNames: ["method", "route", "code"],
});
const httpDuration = new client.Histogram({
  name: "http_request_duration_seconds",
  help: "Request duration in seconds",
  labelNames: ["method", "route", "code"],
  buckets: [0.05, 0.1, 0.2, 0.3, 0.5, 1, 2],
});

let errorInjectRate = 0; // 0 = healthy. flip this with POST /admin/chaos

app.get("/checkout/:id", (req, res) => {
  const start = Date.now();
  const fail = Math.random() < errorInjectRate;
  const delay = 15 + Math.floor(Math.random() * 60);
  setTimeout(() => {
    const code = fail ? "500" : "200";
    const seconds = (Date.now() - start) / 1000;
    httpRequests.inc({ method: "GET", route: "/checkout/:id", code });
    httpDuration.observe({ method: "GET", route: "/checkout/:id", code }, seconds);
    res.status(fail ? 500 : 200).json(fail ? { error: "payment_failed" } : { id: req.params.id, status: "confirmed" });
  }, delay);
});

app.post("/admin/chaos", express.json(), (req, res) => {
  errorInjectRate = Math.max(0, Math.min(1, Number(req.body.rate) || 0));
  res.json({ errorInjectRate });
});

// --- the noise plane: real infra telemetry, simulated so this drill ---
// --- reproduces identically no matter whose laptop it runs on.     ---
// --- these three gauges are 100% decoupled from errorInjectRate —  ---
// --- that decoupling IS the point: they wiggle for reasons that    ---
// --- have nothing to do with whether a customer got their order.   ---
const queueDepth = new client.Gauge({ name: "checkout_queue_depth", help: "Pending background jobs" });
const cpuPercent = new client.Gauge({ name: "checkout_cpu_percent", help: "Simulated CPU utilization" });
const gcPauseMs = new client.Gauge({ name: "checkout_gc_pause_ms", help: "Simulated GC pause, milliseconds" });

setInterval(() => queueDepth.set(15 + Math.floor(Math.random() * 30)), 2000);   // always > 10 — never clears
setInterval(() => cpuPercent.set(55 + Math.floor(Math.random() * 40)), 2000);    // mostly > 60 — rarely dips clear
setInterval(() => gcPauseMs.set(20 + Math.floor(Math.random() * 100)), 3000);    // crosses 50 both ways — flaps

app.get("/metrics", async (req, res) => {
  res.set("Content-Type", register.contentType);
  res.end(await register.metrics());
});

app.listen(3000, () => console.log("checkout-api listening on :3000"));
# checkout-api/Dockerfile
cat > checkout-api/Dockerfile <<'EOF'
FROM node:22-slim
WORKDIR /app
COPY package*.json ./
RUN npm install --omit=dev
COPY server.js ./
EXPOSE 3000
CMD ["node", "server.js"]
EOF
# prometheus.yml — job_name "checkout" is what puts job="checkout" on every
# metric this target exports; that's the exact label the SLOs page's own
# PromQL examples already use, so those queries run here unmodified.
global:
  scrape_interval: 15s
  evaluation_interval: 15s
rule_files:
  - /etc/prometheus/alerts.yml
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]
scrape_configs:
  - job_name: checkout
    static_configs:
      - targets: ["checkout-api:3000"]
# docker-compose.yml — starts with alerts-broken.yml; you'll swap the mount later
services:
  checkout-api:
    build: ./checkout-api
    ports: ["3000:3000"]
  prometheus:
    image: prom/prometheus:v2.53.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alerts-broken.yml:/etc/prometheus/alerts.yml
    ports: ["9090:9090"]
    depends_on: [checkout-api]
  alertmanager:
    image: prom/alertmanager:v0.27.0
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    ports: ["9093:9093"]

Don't start it yet — the alert file it mounts doesn't exist until the next section, and starting Compose without it just errors on the missing volume.

Load the broken alert file and feel the noise

☺ Like you're 10: Three internal gauges, none of them checking whether anyone's actually unhappy, all three wired straight to a human's phone.

This is the alert file a well-meaning engineer actually ships: three thresholds on internal telemetry, every one of them severity: page, because "better safe than sorry" felt reasonable at the time it was written.

# alerts-broken.yml
groups:
  - name: checkout-internal-noise
    rules:
      - alert: CheckoutQueueDepthHigh
        expr: checkout_queue_depth > 10
        for: 1m
        labels: { severity: page }
        annotations:
          summary: "checkout-api background queue depth is high"
      - alert: CheckoutCPUHigh
        expr: checkout_cpu_percent > 60
        for: 1m
        labels: { severity: page }
        annotations:
          summary: "checkout-api CPU utilization is high"
      - alert: CheckoutGCPauseHigh
        expr: checkout_gc_pause_ms > 50
        for: 1m
        labels: { severity: page }
        annotations:
          summary: "checkout-api GC pause time is high"

Start everything, then start a steady stream of real, successful traffic — errorInjectRate is 0 by default, so every single request below succeeds:

docker compose up -d --build
sleep 5
while true; do curl -s -o /dev/null "http://localhost:3000/checkout/$((RANDOM % 1000))"; sleep 0.05; done &
echo "load generator running as job $!"

Give it two or three minutes, then open http://localhost:9090/alerts. You should see all three rules cycling through pending and firing — CheckoutQueueDepthHigh parked in firing permanently and never clearing, CheckoutCPUHigh mostly firing with the occasional brief gap, and CheckoutGCPauseHigh visibly flapping between firing and inactive every minute or two. Confirm the part that actually matters: zero requests are failing.

curl -s 'http://localhost:9090/api/v1/query?query=sum(rate(http_requests_total{job="checkout",code="500"}[5m]))' | grep -o '"value":\[[^]]*\]'
# → "value":[<timestamp>,"0"]  — literally zero errors, three alerts firing anyway
Before — cause-based queue_depth > 10 cpu_percent > 60 gc_pause_ms > 50 severity: page zero errors, three pages anyway — human paged every time After — symptom-based user success ratio (SLI) dual-window burn rate check 5m AND 1h must agree severity: page silent, dashboard only same internal noise underneath — never pages

Diagnose: which of these three should ever ring a phone

☺ Like you're 10: Ask one question about each alert — would a customer have noticed anything at all — and none of the three survive.

On-call culture & sustainable operations gives the exact test to run against every existing page rule: it should be urgent, important, actionable, and real. Run it against all three:

All three are cause-based: they threshold on an internal implementation detail that may or may not be hurting anyone. What you actually want is symptom-based: an alert that only fires because a real user request, right now, is failing or slow.

Define the SLO before you write a single alert rule

☺ Like you're 10: Decide first what "good enough" means to a customer — then, and only then, build the alarm around that number.

Give checkout-api the same target used throughout SLOs, Error Budgets & Toil: 99.9% of requests succeed, measured as code !~ "5.." over total requests. That makes the error budget 100% − 99.9% = 0.1%, or 0.001 as a fraction — the number every threshold below is built from. Compute the actual PromQL for the SLI itself first, so you can eyeball it live while the drill runs:

# checkout-api's rolling 5-minute success ratio
sum(rate(http_requests_total{job="checkout",code!~"5.."}[5m]))
/
sum(rate(http_requests_total{job="checkout"}[5m]))

Paste that into http://localhost:9090/graph right now, with the load generator still running and errorInjectRate still at 0 — it should read 1 (100%) solidly, the whole time the broken alert file was paging you for nothing.

Build the multi-window, multi-burn-rate alerts

☺ Like you're 10: One quick burn is worth an instant page; a slow, steady burn is worth a ticket for tomorrow — not the same alarm for both.

Burn rate is how fast the budget is being spent, as a multiple of the sustainable rate. Requiring two windows to agree — a short one to catch it fast, a longer one to confirm it isn't a two-minute blip — is what makes burn-rate alerting resistant to exactly the kind of noise the broken file was drowning in. Three tiers, same multipliers as the worked table in SLOs, Error Budgets & Toil:

# alerts-fixed.yml
groups:
  - name: checkout-slo-burn
    rules:
      - alert: CheckoutFastBurn
        expr: |
          (
            1 - (
              sum(rate(http_requests_total{job="checkout",code!~"5.."}[1h]))
              /
              sum(rate(http_requests_total{job="checkout"}[1h]))
            )
          ) > (14.4 * 0.001)
          and
          (
            1 - (
              sum(rate(http_requests_total{job="checkout",code!~"5.."}[5m]))
              /
              sum(rate(http_requests_total{job="checkout"}[5m]))
            )
          ) > (14.4 * 0.001)
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "checkout-api burning error budget 14.4x sustainable — page now"

      - alert: CheckoutSlowBurn
        expr: |
          (
            1 - (
              sum(rate(http_requests_total{job="checkout",code!~"5.."}[6h]))
              /
              sum(rate(http_requests_total{job="checkout"}[6h]))
            )
          ) > (6 * 0.001)
          and
          (
            1 - (
              sum(rate(http_requests_total{job="checkout",code!~"5.."}[30m]))
              /
              sum(rate(http_requests_total{job="checkout"}[30m]))
            )
          ) > (6 * 0.001)
        for: 5m
        labels: { severity: page }
        annotations:
          summary: "checkout-api burning error budget 6x sustainable — page now"

      - alert: CheckoutBudgetTicket
        expr: |
          (
            1 - (
              sum(rate(http_requests_total{job="checkout",code!~"5.."}[3d]))
              /
              sum(rate(http_requests_total{job="checkout"}[3d]))
            )
          ) > (1 * 0.001)
          and
          (
            1 - (
              sum(rate(http_requests_total{job="checkout",code!~"5.."}[6h]))
              /
              sum(rate(http_requests_total{job="checkout"}[6h]))
            )
          ) > (1 * 0.001)
        for: 15m
        labels: { severity: ticket }
        annotations:
          summary: "checkout-api steadily burning budget — file a ticket, no page"

Lint it before it ever touches Prometheus — a real, standard step, not optional ceremony:

docker compose exec prometheus promtool check rules /etc/prometheus/alerts.yml
# every rule must print "SUCCESS" — a syntax slip here fails silently
# at 3 a.m. otherwise, not at review time
◆ Key idea

The and between the two windows is the entire mechanism. A short, sharp blip trips the 5-minute window instantly but gets diluted away in the 1-hour window, so the and keeps it from paging — exactly the false alarm a single-window threshold would have fired. A sustained problem trips both windows together, because it's still elevated by the time the longer window catches up. Two clocks, one page, only when both agree.

Prove the AND logic with promtool, before waiting on real traffic

☺ Like you're 10: You can't actually wait around for a three-day alert to prove itself — so you fast-forward a fake clock instead.

Waiting a real hour to confirm CheckoutFastBurn behaves correctly isn't practical mid-drill, and waiting three real days to confirm CheckoutBudgetTicket never will be. promtool test rules solves exactly this: it feeds synthetic time series through your rule file on a fake clock and asserts what should and shouldn't have fired. Two scenarios, same rule file, opposite outcomes:

# rules_test.yml
rule_files:
  - alerts-fixed.yml
tests:
  # Scenario A — a one-minute error blip. Trips the 5m window hard,
  # but stays diluted under threshold in the 1h window. Must NOT page.
  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="checkout",code="200"}'
        values: '0+1000x64'
      - series: 'http_requests_total{job="checkout",code="500"}'
        values: '0+0x59 100+0x4'
    alert_rule_test:
      - eval_time: 64m
        alertname: CheckoutFastBurn
        exp_alerts: []

  # Scenario B — a sustained 2% error rate from minute zero.
  # Trips both windows together. Must page within "for: 2m".
  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="checkout",code="200"}'
        values: '0+980x20'
      - series: 'http_requests_total{job="checkout",code="500"}'
        values: '0+20x20'
    alert_rule_test:
      - eval_time: 15m
        alertname: CheckoutFastBurn
        exp_alerts:
          - exp_labels: { severity: page }
            exp_annotations: { summary: "checkout-api burning error budget 14.4x sustainable — page now" }
docker compose exec prometheus promtool test rules /etc/prometheus/rules_test.yml
# (mount rules_test.yml next to alerts-fixed.yml, or copy it in with `docker compose cp`)

Both must pass. Scenario A proves a brief spike alone can't page — the exact property that makes burn-rate alerting resistant to noise. Scenario B proves a genuine, sustained failure pages within the for: 2m window, so the same design that filters noise doesn't also filter real incidents.

Swap the file, confirm silence, then route by severity

☺ Like you're 10: Same noisy gauges still wiggling in the background — but now nobody's phone buzzes because of them.

Point Compose at the fixed file and reload:

sed -i.bak 's#alerts-broken.yml#alerts-fixed.yml#' docker-compose.yml
docker compose up -d
curl -s -X POST http://localhost:9090/-/reload   # or just: docker compose restart prometheus

Give it two minutes, then check http://localhost:9090/alerts again. checkout_queue_depth, checkout_cpu_percent, and checkout_gc_pause_ms are still jittering exactly as before — you never stopped scraping them, and they're still worth a Grafana panel for debugging — but nothing in alerts-fixed.yml references them, so nothing fires. That's the whole drill's thesis, made visible: the noise didn't go away, the alerting on it did.

Now wire severity to routing, so page and ticket actually mean something different downstream — the same SEV1/SEV2-pages-now, SEV3-tickets-tomorrow split from incident management, applied to infrastructure instead of a human-reported bug:

# alertmanager.yml
route:
  receiver: dashboard-only
  group_by: ["alertname"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: ['severity="page"']
      receiver: oncall-pager
      repeat_interval: 15m
    - matchers: ['severity="ticket"']
      receiver: eng-ticket-queue
      repeat_interval: 12h
receivers:
  - name: dashboard-only        # unlabeled / severity=info — visible in the UI, pages no one
  - name: oncall-pager          # wire this webhook to a real PagerDuty or Opsgenie integration key
    webhook_configs:
      - url: "http://127.0.0.1:5001/paging-stub"
  - name: eng-ticket-queue      # next business day, not urgent
    webhook_configs:
      - url: "http://127.0.0.1:5001/ticket-stub"

Verify the routing tree deterministically, without waiting for a real alert to prove it — amtool answers "who gets this?" for any label set in one call:

docker compose exec alertmanager amtool config routes test --config.file=/etc/alertmanager/alertmanager.yml severity=page
# → oncall-pager
docker compose exec alertmanager amtool config routes test --config.file=/etc/alertmanager/alertmanager.yml severity=ticket
# → eng-ticket-queue
docker compose exec alertmanager amtool config routes test --config.file=/etc/alertmanager/alertmanager.yml severity=info
# → dashboard-only

Two valid designs for the retired internal rules, know both: delete alerts-broken.yml's three rules outright, since the same signal already lives in a Grafana dashboard — see Grafana — with nobody paged by it; or keep them, but relabel every one severity: info so they route to dashboard-only and never reach oncall-pager. Either is correct. Deleting them and relabeling them are the same decision — "not this human's phone" — expressed two different ways.

Verify: prove the silence, then prove the page

☺ Like you're 10: First confirm nothing rings for a fake problem, then confirm something rings for a real one.

With the fixed file live and the load generator still running, check for firing alerts — there should be none:

curl -s 'http://localhost:9090/api/v1/query?query=ALERTS{alertstate="firing"}' | grep -c '"metric"'
# → 0, even though the internal gauges are still bouncing around above their old thresholds

Now inject a real, user-facing failure — this is the moment the whole rest of the drill was built to reach:

curl -s -X POST http://localhost:3000/admin/chaos -H 'content-type: application/json' -d '{"rate":0.5}'
# 50% of requests now fail — a real outage, not a gauge wiggle

Watch http://localhost:9090/alerts — within a few minutes CheckoutFastBurn goes pending, then firing, exactly as promtool's Scenario B predicted. Confirm it reached Alertmanager and got routed correctly:

curl -s http://localhost:9093/api/v2/alerts | grep -o '"alertname":"[^"]*"'
docker compose exec alertmanager amtool --alertmanager.url=http://localhost:9093 alert query --receiver=oncall-pager

Then close the loop the way you'd close a real incident — fix it, and confirm the page clears on its own instead of trusting your memory that it should:

curl -s -X POST http://localhost:3000/admin/chaos -H 'content-type: application/json' -d '{"rate":0}'
# wait a few minutes, then:
curl -s 'http://localhost:9090/api/v1/query?query=ALERTS{alertstate="firing"}' | grep -c '"metric"'
# → 0 again

Done when: the fixed alert file produces zero firing alerts at idle despite the same internal noise that flooded the broken file, CheckoutFastBurn fires within a few minutes of a real 50% error injection and reaches the oncall-pager receiver, and it clears on its own once the chaos rate returns to zero.

0 / 10 steps complete
1Build checkout-api, Prometheus and Alertmanager on Docker Compose
Done when: docker compose ps shows all three services running.
2Start the load generator and confirm zero request errors
Done when: the 5-minute success-ratio query reads exactly 1.
3Load alerts-broken.yml and watch it page for pure internal noise
Done when: all three broken alerts are firing or flapping in the Prometheus UI at zero errors.
4Run the urgent/important/actionable/real test against each broken alert
Done when: you can name which of the four properties each of the three alerts fails.
5Define the 99.9% SLO and confirm the SLI query live in Prometheus
Done when: the SLI query reads 1 while the load generator runs error-free.
6Write the three burn-rate rules and pass promtool check rules
Done when: promtool check rules prints SUCCESS for alerts-fixed.yml.
7Prove the dual-window AND logic with promtool test rules
Done when: Scenario A asserts no page, Scenario B asserts a page, and both pass.
8Swap in alerts-fixed.yml and confirm silence under the same noise
Done when: ALERTS{alertstate="firing"} returns zero results with the gauges still jittering.
9Wire severity-based routing and verify it with amtool config routes test
Done when: page routes to oncall-pager, ticket routes to eng-ticket-queue, info routes to dashboard-only.
10Inject a real 50% error rate, watch it page, then watch it clear
Done when: CheckoutFastBurn fires and reaches oncall-pager, then clears once chaos is reset to 0.
🎬 At the Ship-It Guild
🐦

Pip the Hummingbird: Three alerts, zero errors, and I've already been paged eleven times this morning. I'm starting to just glance at my phone and go back to sleep.

🦥

Sol the Sloth: That's the actual damage, Pip — not the eleven pages, the going-back-to-sleep part. Once you've learned to ignore a page, you'll ignore the real one too.

🦊

Foxy: So why not just delete the queue-depth alert and call it done?

🦥

Sol the Sloth: Give me a second... deleting one rule fixes one rule. The actual fix is asking the same question of every rule you've ever written: does this fire because a user is hurting, or because a number moved? Only one of those deserves Pip's wings.

👺

Gizmo: Or — hear me out — just set every alert to severity: page and let the humans sort it out. Nobody checks. 😈

🐢

Timmy the Turtle: Pip checks, Gizmo. Every single one. Until Pip stops checking — which is exactly what "sort it out" gets you, eventually.

🐦

Pip the Hummingbird: Burn rate's live. Both windows agree, so it's real — I'm already three channels ahead of it, same as always. Just now for something that's actually true.

✓ Checkpoint

1. Why does CheckoutQueueDepthHigh fail all four parts of the urgent/important/actionable/real test, and what specifically is missing that a symptom-based alert has? 2. What does requiring both a 5-minute and a 1-hour window to breach accomplish that a single-window threshold can't, and which promtool test scenario proves it? 3. In the fixed setup, what happens to checkout_queue_depth, checkout_cpu_percent, and checkout_gc_pause_ms — are they deleted, ignored, or something else? 4. What's the practical difference between routing an alert to severity: page versus severity: ticket, and which SEV levels from incident management does each correspond to?

Check your answers
  1. It fails real most directly — it's been firing continuously since the drill began at zero user impact, which means it isn't signaling a real problem, just a permanently elevated internal number. The thing missing is any connection at all to the SLI: a symptom-based alert only fires when the user-facing success ratio itself is degraded, regardless of what's happening inside the process.
  2. It filters out short blips that a single-window threshold would page on immediately: a brief spike trips the short window but gets diluted below threshold in the longer window, so the and keeps it silent. Scenario A in the promtool test proves this directly — a one-minute error burst asserts exp_alerts: [] and passes. Scenario B proves the flip side: a sustained problem trips both windows and does page, so the same design doesn't also swallow real incidents.
  3. They're neither deleted nor ignored in the sense of no longer being collected — Prometheus keeps scraping them, and they still belong on a Grafana dashboard for debugging. What changes is that nothing in alerts-fixed.yml thresholds on them, so they never generate an ALERTS series and never reach Alertmanager. The telemetry stays; the paging on it stops.
  4. severity: page routes to oncall-pager with a short repeat_interval — it wakes a human now, the equivalent of SEV1/SEV2 in incident management. severity: ticket routes to eng-ticket-queue with a long repeat_interval — it opens a queue item for the next business day with nobody paged, the equivalent of SEV3. (Anything left unlabeled falls through to dashboard-only, the SEV4 equivalent — visible, urgent to no one.)

Alert file fixed, routed, and proven under a real fake outage — that's the whole drill. For the arithmetic behind the 14.4×/6×/1× multipliers and how an error budget gates a release, see SLOs, Error Budgets & Toil; for the rest of what makes a rotation survive contact with reality — headcount, escalation, the noise funnel beyond just rule design — see on-call culture & sustainable operations. Prometheus and Grafana cover the stack this drill ran on; PagerDuty and Opsgenie cover what a real oncall-pager receiver plugs into. Ready for a different single skill? Try Drill — Diagnose a Production Incident, or step back to Ship It — Start Here for the six-part continuity version.