Capstone · Part 2 of 6 · Build the Monitoring & Alerting

Capstone Part 2 — Build the Monitoring & Alerting

This is the second of six parts building one continuous project: running the checkout service — the same request-based, payments-adjacent example used throughout this course's SLI/SLO and monitoring lessons — as a reliable production service, with your own hands on every piece. Part 1 defined what "reliable" means for it in numbers: a 99.9% availability SLO over a rolling 30-day window, and a latency SLO alongside it. This page turns those numbers into something that can actually page a human. By the end, checkout exposes real Prometheus metrics, Grafana renders a burn-rate dashboard against them, Prometheus and Alertmanager run the exact four-tier multi-window burn-rate rule set from multi-window, multi-burn-rate alerting, and you will have watched — with your own eyes, on your own screen — a synthetic error spike make the fast-burn alert fire, and then clear.

☺ Explain it like I'm 10

Part 1 was writing down the rule: "the bathtub is allowed to lose water this fast, and no faster." A rule nobody's watching is just a sentence on paper. Today you install the two clocks from the alerting lesson — the twitchy five-minute one and the patient one-hour one — glue them to the actual drain, and wire them to a bell that only rings when both clocks agree the drain's wide open. Then, to prove the bell actually works and isn't just wired to nothing, you open the drain yourself on purpose, on a system nobody depends on, and listen for it to ring. Then you close the drain again and make sure the bell actually stops.

🐘🦥Your hosts for this part: Ellie the Elephant & Sol the Sloth — Ellie instruments the metrics and wires the dashboards, and Sol carries over the exact burn-rate arithmetic from the alerting lesson to make sure the thresholds you deploy are the ones that were actually derived, not approximated from memory.

⚠ Where you are arriving from, and where you're headed

Arriving: a repo — call it reliability-capstone — holding nothing but the SLO document from Part 1: checkout's 99.9% availability SLO (30-day rolling window, error budget 43.2 minutes/month) and its latency SLO (99% of requests under 300ms). No running service yet, no metrics, no alerts. Leaving this page: a small containerized checkout service instrumented with real request-count and latency metrics; Prometheus, Alertmanager, and Grafana running locally via docker compose; the four-tier multi-window burn-rate rule set loaded and evaluating against real scraped data; a Grafana dashboard showing burn rate and budget remaining; and a synthetic error-rate spike you triggered yourself, watched page, and watched clear. Part 3 picks up exactly here and writes the runbook the page you just proved actually points to.

What this part assumes, and what it produces

☺ Like you're 10: The SLO numbers from last time, plus Docker on your laptop — nothing else has to exist yet.

You need Docker (or Podman) and curl; a working jq install makes the verification steps far more readable but isn't strictly required. You need the checkout SLOs from Part 1 — if you generalized the exercise to a different demo service there, substitute its name and numbers everywhere below, but the shape of every command stays identical. Nothing from a later part is required yet: no real paging tool, no k6 load-test harness, no chaos-injection framework. This part builds its own small chaos knob — a single environment variable — specifically so it doesn't have to borrow from Part 6 early.

The world model this part adds

Building on Part 1's SLO document, here's what gets born on this page:

Thing	Name / value	Introduced
The service	`checkout` — instrumented, containerized, listening on `:8080`	Part 2 — this page
Availability SLO	99.9% non-5xx over a rolling 30-day window (43.2 min/month budget)	Part 1 → alerted on here
Latency SLO	99% of requests < 300ms	Part 1 (instrumented here; alerting on it follows the same pattern, left as an exercise)
Metrics	`http_requests_total{route,code}`, `http_request_duration_seconds{route}`	Part 2 — this page
Synthetic chaos knob	`CHAOS_ERROR_RATE` env var, 0.0–1.0	Part 2 — this page
Alerting stack	Prometheus (rules) + Alertmanager (routing) + Grafana (dashboards), via `docker compose`	Part 2 — this page
Burn-rate alerts	`CheckoutErrorBudgetBurnFast/Moderate/Slow/Slowest`	Part 2 — this page

Instrumenting the `checkout` service's SLIs

☺ Like you're 10: Two counters and one stopwatch, exposed on a page a robot can read — that's the entire instrumentation job.

The SLI from SLIs, SLOs & error budgets is good-events over valid-events — for checkout's availability SLO, that's non-5xx requests over all requests. Instrumenting it means exposing exactly two shapes of data on a /metrics endpoint: a counter of requests by route and status code, and a histogram of request duration by route — the same two metric shapes the multi-window-burn-rate-alerting lesson's PromQL was already written against, so nothing about the rules you'll load later needs translating. Create checkout-svc/app.py:

from flask import Flask, jsonify, Response
from prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST
import os, random, time

app = Flask(__name__)

REQS = Counter(
    "http_requests_total", "Total HTTP requests to checkout",
    ["route", "code"],
)
LATENCY = Histogram(
    "http_request_duration_seconds", "checkout request latency",
    ["route"],
    buckets=(0.025, 0.05, 0.1, 0.15, 0.2, 0.3, 0.5, 1, 2, 5),
)

# The one knob this whole part exists to flip: 0.0 is normal traffic,
# 0.25 means roughly one in four requests fails on purpose. Nothing else
# about the request path changes when this is nonzero.
CHAOS_ERROR_RATE = float(os.environ.get("CHAOS_ERROR_RATE", "0"))

@app.route("/checkout", methods=["POST"])
def checkout():
    start = time.time()
    time.sleep(random.uniform(0.03, 0.09))          # simulated processing
    code = 500 if random.random() < CHAOS_ERROR_RATE else 200
    LATENCY.labels(route="/checkout").observe(time.time() - start)
    REQS.labels(route="/checkout", code=str(code)).inc()
    return jsonify(status=("ok" if code == 200 else "error")), code

@app.route("/metrics")
def metrics():
    return Response(generate_latest(), mimetype=CONTENT_TYPE_LATEST)

@app.route("/healthz")
def healthz():
    return "ok", 200

checkout-svc/requirements.txt and a minimal Dockerfile alongside it:

flask==3.0.*
prometheus_client==0.20.*
gunicorn==22.0.*

# checkout-svc/Dockerfile
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
ENV CHAOS_ERROR_RATE=0
EXPOSE 8080
CMD ["gunicorn", "-b", "0.0.0.0:8080", "-w", "2", "app:app"]

◆ Key idea

CHAOS_ERROR_RATE lives inside the service on purpose, as a plain environment variable, not behind a feature flag service or a separate fault-injection sidecar. This part is about proving the alerting path works end to end, not about building a production-grade fault-injection framework — that arrives properly in Part 6, aimed at a service that already trusts its own monitoring, which is exactly what today's exercise is establishing.

Standing up Prometheus, Alertmanager, and Grafana

☺ Like you're 10: One file tells Docker to start four little programs at once: your service, the thing that watches it, the thing that pages about it, and the thing that draws pretty pictures of it.

Wire all four containers together with a single docker-compose.yml at the repo root:

services:
  checkout:
    build: ./checkout-svc
    environment:
      - CHAOS_ERROR_RATE=${CHAOS_ERROR_RATE:-0}
    ports: ["8080:8080"]

  prometheus:
    image: prom/prometheus:v2.53.0
    volumes:
      - ./prometheus:/etc/prometheus
    ports: ["9090:9090"]
    command: ["--config.file=/etc/prometheus/prometheus.yml"]
    depends_on: [checkout]

  alertmanager:
    image: prom/alertmanager:v0.27.0
    volumes:
      - ./alertmanager:/etc/alertmanager
    ports: ["9093:9093"]

  grafana:
    image: grafana/grafana:11.1.0
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    ports: ["3000:3000"]
    depends_on: [prometheus]

prometheus/prometheus.yml — scrapes checkout every 15 seconds and loads the rule file the next two sections build:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/checkout-burn-rate.rules.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

scrape_configs:
  - job_name: checkout
    static_configs:
      - targets: ["checkout:8080"]

echo "CHAOS_ERROR_RATE=0" > .env
docker compose up -d --build
docker compose ps
curl -s http://localhost:8080/healthz
curl -s http://localhost:9090/-/ready

Recording rules and the four-tier burn-rate alerts

☺ Like you're 10: This is the same eight-window, four-alarm recipe from the alerting lesson, copied in whole, with 0.001 in every threshold because that's exactly what a 99.9% SLO's 1 − SLO comes out to.

This is not a simplified version of multi-window, multi-burn-rate alerting's worked example — it's the same rule set, deployed. Write prometheus/checkout-burn-rate.rules.yml:

groups:
  - name: checkout-slo-burn-rate
    rules:
      - record: sre:checkout_requests:error_ratio5m
        expr: |
          sum(rate(http_requests_total{route="/checkout",code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{route="/checkout"}[5m]))
      - record: sre:checkout_requests:error_ratio30m
        expr: |
          sum(rate(http_requests_total{route="/checkout",code=~"5.."}[30m]))
          /
          sum(rate(http_requests_total{route="/checkout"}[30m]))
      - record: sre:checkout_requests:error_ratio1h
        expr: |
          sum(rate(http_requests_total{route="/checkout",code=~"5.."}[1h]))
          /
          sum(rate(http_requests_total{route="/checkout"}[1h]))
      - record: sre:checkout_requests:error_ratio2h
        expr: |
          sum(rate(http_requests_total{route="/checkout",code=~"5.."}[2h]))
          /
          sum(rate(http_requests_total{route="/checkout"}[2h]))
      - record: sre:checkout_requests:error_ratio6h
        expr: |
          sum(rate(http_requests_total{route="/checkout",code=~"5.."}[6h]))
          /
          sum(rate(http_requests_total{route="/checkout"}[6h]))
      - record: sre:checkout_requests:error_ratio24h
        expr: |
          sum(rate(http_requests_total{route="/checkout",code=~"5.."}[24h]))
          /
          sum(rate(http_requests_total{route="/checkout"}[24h]))
      - record: sre:checkout_requests:error_ratio3d
        expr: |
          sum(rate(http_requests_total{route="/checkout",code=~"5.."}[3d]))
          /
          sum(rate(http_requests_total{route="/checkout"}[3d]))
      # extra: full compliance-window ratio, for the Grafana "budget remaining"
      # panel only — 30d is far too slow a window to ever page on directly.
      - record: sre:checkout_requests:error_ratio30d
        expr: |
          sum(rate(http_requests_total{route="/checkout",code=~"5.."}[30d]))
          /
          sum(rate(http_requests_total{route="/checkout"}[30d]))

      - alert: CheckoutErrorBudgetBurnFast
        expr: |
          sre:checkout_requests:error_ratio1h > (14.4 * 0.001)
          and
          sre:checkout_requests:error_ratio5m > (14.4 * 0.001)
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "checkout burning error budget at 14.4x — 2% of the 30-day budget in 1h"

      - alert: CheckoutErrorBudgetBurnModerate
        expr: |
          sre:checkout_requests:error_ratio6h  > (6 * 0.001)
          and
          sre:checkout_requests:error_ratio30m > (6 * 0.001)
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "checkout burning error budget at 6x — 5% of the 30-day budget in 6h"

      - alert: CheckoutErrorBudgetBurnSlow
        expr: |
          sre:checkout_requests:error_ratio24h > (3 * 0.001)
          and
          sre:checkout_requests:error_ratio2h  > (3 * 0.001)
        for: 15m
        labels: { severity: ticket }
        annotations:
          summary: "checkout burning error budget at 3x — 10% of the 30-day budget in 1d"

      - alert: CheckoutErrorBudgetBurnSlowest
        expr: |
          sre:checkout_requests:error_ratio3d > (1 * 0.001)
          and
          sre:checkout_requests:error_ratio6h > (1 * 0.001)
        for: 1h
        labels: { severity: ticket }
        annotations:
          summary: "checkout burning error budget at 1x (nominal) — 10% of the 30-day budget in 3d"

Reload without restarting the whole stack, then confirm Prometheus accepted every rule:

curl -s -X POST http://localhost:9090/-/reload
curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[].rules[].name'
# every recording rule and all four alert names should be listed, with no "health":"err"

Routing alerts: page vs. ticket in Alertmanager

☺ Like you're 10: One filing rule decides everything: the label that says page goes to the loud pile, the one that says ticket goes to the quiet pile — and today both piles are just stand-in mailboxes on your own laptop.

The severity label on each alert — set once, in the rule file above — is the only thing Alertmanager needs to route correctly. Write alertmanager/alertmanager.yml:

route:
  receiver: default
  group_by: ["alertname"]
  group_wait: 30s
  group_interval: 5m
  routes:
    - matchers: ["severity = page"]
      receiver: page-oncall
      repeat_interval: 15m
    - matchers: ["severity = ticket"]
      receiver: file-ticket
      repeat_interval: 4h

receivers:
  - name: default
  - name: page-oncall
    webhook_configs:
      - url: http://host.docker.internal:5001/   # stand-in — Part 3 swaps this for a real PagerDuty/Opsgenie integration key
  - name: file-ticket
    webhook_configs:
      - url: http://host.docker.internal:5002/   # stand-in — a real deployment points this at a ticket queue

A stand-in webhook receiver is enough to prove routing without standing up a real paging tool yet — a one-line Python listener is plenty:

python3 -m http.server 5001 &   # catches page-oncall deliveries; watch its access log
python3 -m http.server 5002 &   # catches file-ticket deliveries

Part 3 replaces both stand-ins with a real escalation policy in PagerDuty or Opsgenie — the routing rule above doesn't change at all when that happens, only the receiver URL does, which is exactly the point of keeping severity as the single decision variable.

Building the burn-rate dashboard in Grafana

☺ Like you're 10: A dashboard isn't the alarm — it's the window someone looks through once the alarm's already ringing, to see exactly how bad "bad" is.

Log into Grafana at localhost:3000 (admin/admin, forced reset on first login), add Prometheus (http://prometheus:9090) as a data source, and build four panels on one dashboard, all against the recording rules from two sections ago — nothing here needs a raw PromQL rewrite, only the already-recorded ratios:

Panel	Query	Notes
Error ratio, all 7 windows	`sre:checkout_requests:error_ratio5m` … `error_ratio3d`, one series each	Time-series panel; this is the exact multi-lens view from the alerting lesson's schematic, rendered live.
Burn rate (5m)	`sre:checkout_requests:error_ratio5m / 0.001`	Gauge; thresholds at 1 (green), 6 (yellow), 14.4 (red) mirror the four-tier table directly.
Budget remaining, this window	`100 * (1 - sre:checkout_requests:error_ratio30d / 0.001)`	Stat panel. At a sustained burn rate of exactly 1× for the whole window this reads 0% — spent, not spare, exactly as SLIs, SLOs & error budgets works out for the checkout API sitting precisely on its line.
Alert state	`ALERTS{alertname=~"CheckoutErrorBudgetBurn.*"}`	Table panel; shows which of the four alerts is currently `firing` versus `pending` versus absent.

Verifying the alert actually fires: a synthetic error-rate spike

☺ Like you're 10: Open the drain yourself, on purpose, and time how long it takes the bell to ring.

Everything above is unverified until you've watched the page-tier alert actually trip. Start steady baseline traffic first, so the dashboard has something to compare the spike against — a tiny loop is enough:

# load.sh — steady synthetic traffic against checkout, roughly 15-20 req/s
#!/usr/bin/env bash
while true; do curl -s -o /dev/null -X POST http://localhost:8080/checkout; sleep 0.05; done

chmod +x load.sh
./load.sh &
LOADPID=$!
# let a few minutes of clean baseline accumulate before you touch CHAOS_ERROR_RATE

Now open the drain. A 25% injected error rate is deliberately extreme against a 0.1% budget threshold — burn rate around 250× — because this test is checking that the wiring works at all, not re-validating the multiplier tuning itself (that backtesting exercise belongs to the alerting lesson and, for a rep of your own, the alert-design drill):

echo "CHAOS_ERROR_RATE=0.25" > .env
docker compose up -d checkout            # recreates only the checkout container with the new env
date -u                                  # note the wall-clock time you flipped the knob

Watch both windows the fast tier depends on cross threshold, then watch the alert itself:

watch -n5 'curl -s "http://localhost:9090/api/v1/query?query=sre:checkout_requests:error_ratio5m" | jq ".data.result[0].value[1]"'
# should climb past 0.0144 (14.4 * 0.001) within a couple of scrape intervals

curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.labels.alertname=="CheckoutErrorBudgetBurnFast") | {state, activeAt}'
# "state":"pending" first (inside the for: 2m window), then "state":"firing"

curl -s http://localhost:9093/api/v2/alerts | jq '.[] | select(.labels.alertname=="CheckoutErrorBudgetBurnFast") | .status.state'
# "active" — confirms Alertmanager received it, not just Prometheus

# and confirm it actually reached the page-oncall receiver:
# check the terminal running `python3 -m http.server 5001` for a POST logged around for: + group_wait later

Record the wall-clock gap between the timestamp you flipped CHAOS_ERROR_RATE and the moment Alertmanager shows "active". It should land close to the alert's for: 2m plus Alertmanager's group_wait: 30s — if it's dramatically longer, re-check that your rule file actually reloaded (the curl -X POST .../-/reload step from two sections ago) and that Alertmanager's route matchers line matches the label spelling exactly.

Proving it clears too, not just that it fires

☺ Like you're 10: A bell that never turns off is just as broken as one that never rings — close the drain and time that too.

Close the drain and confirm the recovery behaviour the alerting lesson's schematic predicted — the short window clears fast, the long window stays lit far longer, and the alert as a whole tracks the short window because it's an AND, not an OR:

echo "CHAOS_ERROR_RATE=0" > .env
docker compose up -d checkout
date -u   # note this timestamp too

watch -n5 'curl -s http://localhost:9090/api/v1/alerts | jq ".data.alerts[] | select(.labels.alertname==\"CheckoutErrorBudgetBurnFast\") | .state"'
# the 5m window clears within about 5-10 minutes of real time; expect "state" to
# disappear from the /alerts response entirely once it fully resolves

curl -s "http://localhost:9090/api/v1/query?query=sre:checkout_requests:error_ratio1h" | jq '.data.result[0].value[1]'
# still elevated well above 0.0144 here — it will keep sliding back down for
# up to the full hour, because the bad minutes are still inside its trailing average

⚠ Don't stop at "it fired"

It's tempting to declare victory the moment the page lands and move on. That only proves detection works — it says nothing about whether the alert is self-healing on the schedule it was designed for. If you skip this half of the exercise and the AND was silently typed as OR, or a stray for: got bolted onto the long window somewhere along the way, you won't find out until a real incident's page keeps nagging the on-call engineer for forty-five minutes after the actual fix has already landed — exactly the failure mode the alerting lesson spends a whole section warning about.

What "done" looks like for Part 2

☺ Like you're 10: A service that tells the truth about itself, a rulebook watching that truth, and proof — not just a claim — that the rulebook actually works.

At the end of this part: checkout is running in a container and exposing real http_requests_total and http_request_duration_seconds metrics; Prometheus is scraping it every 15 seconds and evaluating all seven recording rules and all four burn-rate alerts from multi-window, multi-burn-rate alerting; Alertmanager is routing severity: page and severity: ticket to separate receivers; Grafana renders a live burn-rate and budget-remaining dashboard against the same recording rules; and you have watched, with timestamps, the fast-burn alert go from silent to pending to firing to cleared, driven by nothing but a single environment variable you flipped yourself. Nothing here gets thrown away — Part 3 starts from this exact page tier and writes the runbook it should have pointed to the whole time, then swaps the stand-in webhook receivers for a real paging tool.

🎬 At the Reliability Watch
🐘Ellie the Elephant: /metrics is live — the error counter and the latency histogram are both scraping clean every fifteen seconds, no gaps.
🦥Sol the Sloth: ...And the recording rules match all eight windows from the alerting lesson exactly. Twenty-five percent injected against a zero-point-one-percent budget... that's a burn rate near two hundred and fifty. Nothing subtle about this test.
🦊Foxy: So blunt on purpose, or blunt because we didn't want to do the math properly?
🦥Sol the Sloth: ...On purpose. We're not re-tuning the multipliers today — that's backtesting work, and it needs months of real data, not one afternoon. We're only proving the wiring isn't broken.
🐢Timmy the Turtle: Before anyone touches that environment variable — confirmed it's AND between the two windows, not OR? And the for: is two minutes, not fifteen, on this tier?
🐘Ellie the Elephant: Confirmed both, straight from the file.
🐦Pip the Hummingbird: PAGE RECEIVED — CheckoutErrorBudgetBurnFast, severity page, active two minutes thirty-one seconds after the spike started. That's the for: and the group wait doing exactly their job.
🦊Foxy: Now turn it off. If the short window doesn't clear in a few minutes, we didn't build what the lesson taught us — we built something that only looks like it.
🐘Ellie the Elephant: Short window's back under threshold, alert's cleared. Long window's still elevated — it'll keep sliding down for most of the hour, exactly like the schematic said.

✓ Checkpoint

1. What two metric shapes does checkout need to expose for the burn-rate rules to work at all, and which SLI does each one back? 2. Why does the dashboard need a 30-day recording rule when none of the four alerts ever evaluate against a 30-day window? 3. Why is a 25% injected error rate a reasonable way to prove the alert wiring works, even though it doesn't validate whether 14.4× is the right multiplier for real production traffic? 4. After clearing CHAOS_ERROR_RATE, why does the fast/page tier's alert clear within minutes while the 1-hour window's own error ratio stays elevated much longer — and why is that expected, not a bug? 5. What's the one label that decides whether an alert becomes a page or a ticket in Alertmanager, and where is it actually set?

Check your answers

A counter (http_requests_total, labeled by route and status code) backs the availability SLI — good events are non-5xx, valid events are all requests. A histogram (http_request_duration_seconds, labeled by route) backs the latency SLI, letting you compute percentiles rather than a single misleading average.
None of the four burn-rate alerts should ever run against a full 30-day window — far too slow to page on. The 30-day recording rule exists only for the Grafana "budget remaining" stat panel, which needs the ratio over the exact compliance window to compute what fraction of the whole month's budget has actually been spent so far.
The test is validating that the pipeline — metrics, recording rules, alert evaluation, Alertmanager routing, receiver delivery — works end to end, not whether the specific multiplier thresholds are correctly tuned for real traffic patterns. An extreme, obvious spike removes any ambiguity about whether the *wiring* is broken; tuning the actual thresholds requires backtesting against months of real historical data, a separate exercise the alerting lesson describes.
The alert requires both windows above threshold, joined with AND. The 5-minute short window only looks back five minutes, so it clears almost as soon as clean traffic resumes; the 1-hour long window still contains the bad minutes inside its trailing average and only rolls them out gradually, over up to the full hour. The alert as a whole clears with the short window specifically because that's what the AND-join and the short window were designed to do — this is the exact "slow reset" fix from the alerting lesson, verified rather than just read about.
The severity label — set on each alert rule in checkout-burn-rate.rules.yml (page for the two page-tier alerts, ticket for the two ticket-tier alerts) — is the only thing Alertmanager's route.routes[].matchers reads to decide which receiver an alert goes to.

Part 2 gave checkout a real voice — metrics, dashboards, and alerts that have actually been proven to fire and clear. Continue to Capstone Part 3 — Write the On-Call Runbook, where the page you just watched land gets a real destination and real response steps. Or revisit multi-window, multi-burn-rate alerting and monitoring & observability for the concepts behind what you just built, step back to the capstone's start page for the full six-part map, or get a faster, standalone rep of alert design with the alert-design drill.