Capstone Part 3 — Write the On-Call Runbook

This is the third of six parts building one continuous project: checkout-api, the same 99.9%-availability service from Part 1 and Part 2, now gets somewhere real for its pages to go. Part 2 left both severity tiers — page and ticket — routed to a Python http.server stand-in on your own laptop, and explicitly promised that this page would replace it. Today you register checkout-api as a real PagerDuty service, close the single point of failure a lone escalation rung always has, swap the stand-in webhooks for a tested paging path, and write the runbook a bleary responder actually reads in the first five minutes: what to do before diagnosing anything, and documented steps for the two or three dependencies most likely to be the actual root cause when CheckoutErrorBudgetBurnFast fires.

Where Part 3 picks up

☺ Like you're 10: Two severities that currently page and ticket into an empty laptop terminal — today they get a real destination, and the destination gets a name, a backup, and a script.

Nothing below invents new facts about checkout-api — it's entirely built from what Parts 1 and 2 already established. Keep this table in your head; every section on this page cites one row of it by name.

Registering checkout-api as a PagerDuty service — and naming what's still missing

☺ Like you're 10: One name, one policy — but today's policy has only a single rung at the end before it gives up, and that's the exact gap this page exists to close.

This course's own PagerDuty page already walked through registering a checkout-api service — reuse its pagerduty_user "asha", its pagerduty_schedule "checkout_primary" (named checkout-primary-oncall, weekly rotation, America/New_York), its pagerduty_service "checkout" (PagerDuty display name checkout-api), and its pagerduty_service_integration against the Prometheus vendor, exactly as written there. One small reconciliation worth stating out loud: Part 2's container, metric labels, and alert names all say checkout; PagerDuty's own service record says checkout-api. That's not a naming inconsistency to fix — it's completely normal for a short container name and a formal service-catalog entry to differ, and the Terraform resource address (pagerduty_service.checkout) versus its name attribute ("checkout-api") is exactly where that gap is supposed to live.

What that page's escalation policy doesn't have is the real problem. Its two rules were: Level 1, the checkout_primary schedule, 15-minute delay; Level 2, a single user_reference called team_lead, full stop. That single named person at Level 2 — no schedule, no rotation, no backup of their own — is precisely the failure mode a production readiness review flags as blocking: a lone name at the end of a chain is a single point of failure wearing a policy's clothing. Closing it is this section's whole job.

Closing the gap: a genuine secondary schedule and a named backstop

☺ Like you're 10: Same two engineers, same seven-day rotation — just started a week later, so whoever's "secondary" this week is never the same tired person who's already "primary."

Add a second schedule built from the same two-person pool, with its rotation start offset by exactly one turn. With only two people cycling weekly, that single offset guarantees the secondary is always the person not currently primary — never the same human getting called twice:

# NEW — closes "no secondary rotation staffed": a real second schedule,
# not just a lone name sitting at the end of the chain.
resource "pagerduty_schedule" "checkout_secondary" {
  name      = "checkout-secondary-oncall"
  time_zone = "America/New_York"

  layer {
    name                          = "Weekly rotation, offset one turn from primary"
    start                         = "2026-01-12T09:00:00-05:00"   # exactly 7 days after checkout_primary's start
    rotation_virtual_start        = "2026-01-12T09:00:00-05:00"
    rotation_turn_length_seconds  = 604800                        # same 7-day cadence, same two people
    users = [
      pagerduty_user.asha.id,
      pagerduty_user.marco.id,
    ]
  }
}

resource "pagerduty_user" "priya" {
  name  = "Priya Shah"
  email = "priya@acme.io"
  role  = "manager"
}

# Replaces the two-rule policy from the PagerDuty tool page wholesale —
# same resource address, three real rungs instead of two.
resource "pagerduty_escalation_policy" "checkout" {
  name      = "checkout-primary-ep"
  num_loops = 1   # one real backstop beats two loops through a policy that used to dead-end

  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.checkout_primary.id
    }
  }
  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.checkout_secondary.id
    }
  }
  rule {
    escalation_delay_in_minutes = 5   # inert once num_loops = 1 — nothing follows this rule
    target {
      type = "user_reference"
      id   = pagerduty_user.priya.id
    }
  }
}

num_loops drops from 2 to 1 on purpose: once the chain ends in a real human with manager authority instead of dead-ending, looping the whole three-rung chain a second time before anyone with actual decision power gets involved just delays real escalation by another half hour for no benefit. One clean pass through three real rungs beats two passes through a policy that used to have only two.

Swapping the stand-in webhooks for the real routing key

☺ Like you're 10: Only the receiver's address changes — the severity label that decides page versus ticket doesn't move at all, exactly like Part 2 promised it wouldn't.

The pagerduty_service_integration resource from the PagerDuty tool page mints a routing key the moment it's applied. Alertmanager's own YAML has no built-in environment-variable interpolation, and pasting a live routing key straight into a config file checked into Git is a bad habit worth not starting — so pull it into a file Alertmanager reads at startup instead, the same pattern the _file variants exist for on every Alertmanager receiver type that takes a secret:

mkdir -p alertmanager/secrets
terraform apply
terraform output -raw checkout_routing_key > alertmanager/secrets/checkout-routing-key
chmod 600 alertmanager/secrets/checkout-routing-key

Mount that directory into the container — one line added to Part 2's docker-compose.yml, nothing else in that file changes:

  alertmanager:
    image: prom/alertmanager:v0.27.0
    volumes:
      - ./alertmanager:/etc/alertmanager
      - ./alertmanager/secrets:/etc/alertmanager/secrets:ro   # NEW

Then replace both receivers in alertmanager/alertmanager.yml — page-oncall now points at PagerDuty for real, and file-ticket lands in the same #checkout-reliability Slack channel Part 1's error-budget policy already uses for its early-warning trigger, so the team has one place to watch every tier of degradation instead of two:

receivers:
  - name: default
  - name: page-oncall
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/checkout-routing-key
        severity: critical
        description: '{{ .CommonAnnotations.summary }}'
        details:
          alertname: '{{ .CommonLabels.alertname }}'
          runbook:   '{{ .CommonAnnotations.runbook_url }}'
  - name: file-ticket
    slack_configs:
      - channel: "#checkout-reliability"
        api_url_file: /etc/alertmanager/secrets/slack-webhook-url   # your own incoming webhook, saved the same way
        title: '{{ .CommonAnnotations.summary }}'
        text: "ticket-tier burn — reviewed at next standup, not paged"

curl -s -X POST http://localhost:9093/-/reload
# no restart needed — Alertmanager hot-reloads its own config file

Verify with a real drill, the same Events API v2 shape the PagerDuty page already showed, now pointed at your own service instead of a hypothetical one. Fire it, then deliberately don't acknowledge, so you actually watch the fifteen-minute fall-through instead of taking it on faith:

ROUTING_KEY=$(terraform output -raw checkout_routing_key)

curl -X POST https://events.pagerduty.com/v2/enqueue \
  -H 'Content-Type: application/json' \
  -d '{
    "routing_key": "'"$ROUTING_KEY"'",
    "event_action": "trigger",
    "dedup_key": "checkout-api-runbook-drill",
    "payload": {
      "summary": "DRILL — verifying the escalation policy actually rings a phone",
      "source": "manual-runbook-drill",
      "severity": "critical"
    }
  }'
date -u   # note when you fired it

# ...wait, without acknowledging, and confirm the fall-through:
curl -s "https://api.pagerduty.com/incidents?statuses[]=triggered" \
  -H "Authorization: Token token=$PD_API_KEY" \
  -H 'Accept: application/vnd.pagerduty+json;version=2' \
  | jq '.incidents[] | {id, status, last_status_change_by}'

# resolve the drill once you've watched it fall through once
curl -X POST https://events.pagerduty.com/v2/enqueue \
  -H 'Content-Type: application/json' \
  -d '{"routing_key":"'"$ROUTING_KEY"'","event_action":"resolve","dedup_key":"checkout-api-runbook-drill"}'

The first five minutes

☺ Like you're 10: Stop the clock before you start diagnosing — acknowledging takes five seconds and buys you the entire fifteen-minute window to actually think instead of racing it.

This is the section every runbook_url below points to. Whatever fired, whatever the eventual root cause turns out to be, these steps come first, in this order, every time:

Triage: which of the three dependencies is actually on fire

☺ Like you're 10: checkout-api only calls three things — the alert that pages you can't tell you which one broke, so the runbook has to.

CheckoutErrorBudgetBurnFast and Moderate are deliberately symptom-based — they fire on checkout-api's own overall error ratio, exactly the way monitoring & observability says a page-worthy alert should, and exactly the way Part 2 built them: one number, agnostic to cause. That's correct design, but it means the alert itself never tells you which of the three dependencies from Part 1's schematic is actually responsible. A production checkout-api would label its outbound-call metric by dependency (http_client_requests_total{dependency="payment-processor",code=~"5.."}, and so on) precisely so this triage step is a single query instead of a guessing game — if your own deployment hasn't added that label yet, add it before you need it at 3 a.m., not during the incident. With it, isolating the culprit is one comparison:

sum by (dependency) (
  rate(http_client_requests_total{service="checkout-api", code=~"5.."}[5m])
)
/
sum by (dependency) (
  rate(http_client_requests_total{service="checkout-api"}[5m])
)
# whichever "dependency" label has the highest error ratio is where you start —
# if none of the three are elevated, the fault is likely inside checkout-api itself

Whichever dependency's ratio is elevated, jump straight to its entry below — don't read all three in order under pressure.

Runbook entry — the external payment processor

Symptom: error ratio elevated specifically on the payment-processor dependency label; POST /v1/checkout requests hang until timeout or come back 5xx right after the card-authorization call.

Diagnose:

# p99 latency to the payment processor specifically
curl -s "http://localhost:9090/api/v1/query?query=histogram_quantile(0.99,sum(rate(http_client_request_duration_seconds_bucket{service=%22checkout-api%22,dependency=%22payment-processor%22}[5m]))by(le))" | jq '.data.result[0].value[1]'

# check the payment processor's own published status page first —
# this is an external vendor; you cannot fix their outage from here

# pull recent traces to confirm the failure is isolated to this one call
curl -s "localhost:16686/api/traces?service=checkout-api&tags=%7B%22error%22%3A%22true%22%7D&minDuration=2s" | jq .

Mitigate: if the vendor's status page confirms a full outage, this is SEV1 — checkout has no fallback path for an authorization call, so declare the incident and communicate rather than continuing to dig. If it's degraded rather than fully down (elevated latency, intermittent timeouts), do not blindly retry failed authorization attempts — a retry that lands after the original actually succeeded is exactly the duplicate-charge scenario Part 1's zero-tolerance invariant covers, and that invariant overrides this runbook's normal escalation entirely, regardless of remaining error budget.

Escalate: loop in whoever owns the vendor relationship for a formal status inquiry if the outage passes 10 minutes — this is outside the team's direct control, so pulling in help earlier than usual is the right call, not a sign you're stuck.

Runbook entry — the inventory service

Symptom: error ratio elevated on the inventory-service dependency label; failures cluster on cart validation, before the payment call is ever reached.

Diagnose:

curl -s "http://localhost:9090/api/v1/query?query=sum(rate(http_client_requests_total{service=%22checkout-api%22,dependency=%22inventory-service%22,code=~%225..%22}[5m]))" | jq .

# check the inventory service's own dashboard / on-call channel —
# is this checkout-api-specific, or is inventory-service down for everyone?

Mitigate: unlike the payment processor, skipping the inventory check is technically possible — but it trades a hard failure for a soft one: overselling a sold-out item. That trade is a business decision, not an engineering one, and this runbook does not authorize it unilaterally. Flag it to the Incident Commander (or Priya, if the incident has reached Level 3) and get explicit sign-off before bypassing inventory validation, the same named-approver discipline Part 1's exception path already requires for anything shipped during a freeze.

Escalate: page inventory-service's own on-call rotation directly rather than waiting for them to notice their own dashboards — a dependency failing silently for hours because nobody told the owning team is a preventable delay, not bad luck.

Runbook entry — checkout-api's own order database

Symptom: error ratio elevated with no dependency label showing a spike — both the payment processor and inventory service read clean. This is the internal case: the fault is inside checkout-api or its own datastore.

Diagnose:

kubectl -n checkout exec -it deploy/checkout-api -- \
  psql "$DATABASE_URL" -c "SELECT count(*) FROM pg_stat_activity WHERE state != 'idle';"

# find the specific query holding connections open
kubectl -n checkout exec -it deploy/checkout-api -- \
  psql "$DATABASE_URL" -c "SELECT pid, now() - query_start AS duration, state, query FROM pg_stat_activity ORDER BY duration DESC LIMIT 5;"

Mitigate, in order of speed: (1) if the deploy check from the first-five-minutes table found a recent change, roll it back first — a bad query shipped an hour ago is faster to undo than to fix live; (2) if a single long-running query is holding the pool, terminate it: SELECT pg_terminate_backend(pid); for the offending pid from the query above; (3) if the pool is simply undersized for current traffic with no single culprit query, bump the pool size — treat this as an emergency change requiring the same sign-off as any other production change, not a quiet edit.

Escalate: pull in whoever owns the order database if none of the above recovers it within about 10 minutes — a saturated connection pool with no obvious single cause is exactly the kind of problem that benefits from a second set of eyes on replication lag, disk I/O, or a query plan regression you haven't thought to check.

Escalation authority: Part 1's error-budget policy, quoted verbatim

☺ Like you're 10: This runbook doesn't get to invent its own rules about when to freeze shipping or who's allowed to override that — Part 1 already wrote that rule down, before anyone was under pressure, and this page just quotes it.

Per Part 1's explicit promise — "Part 3 quotes this page's policy trigger and exception path verbatim as the runbook's escalation section" — here it is, unedited:

TRIGGER 1 — early warning
  ≤25% of budget remains (≤10.8 min / ≤1,250 failed requests left)
  with more than 5 days remaining in the window.
  -> Post a heads-up in #checkout-reliability. No freeze yet.

TRIGGER 2 — freeze
  Budget fully consumed (0 minutes / 0 requests remaining)
  before the 30-day window closes.
  -> FREEZE: no new checkout-api feature launches, no non-critical
     dependency upgrades, no checkout-flow experiments.
  -> REDIRECT: the owning team's next sprint goes to reliability
     work tied to whatever actually burned the budget — not
     general cleanup chosen after the fact.
  -> RESUME: automatically, once the rolling 30-day window has
     aged the bad minutes out and the trailing budget is > 0 again.

EXCEPTION PATH
  A change may still ship during a freeze only with written
  sign-off from the on-call incident commander or the team's EM,
  logged in the incident channel, and limited to: security
  patches, active-incident mitigations, legally mandated fixes.

Wiring runbook_url into Part 2's alert rules

☺ Like you're 10: An alert that pages you without telling you where the instructions live just traded one problem for a worse one — a phone ringing with no next step attached.

Add one annotation line to each of Part 2's four alert rules in prometheus/checkout-burn-rate.rules.yml — the query, the for:, and the severity label don't change at all:

      - alert: CheckoutErrorBudgetBurnFast
        expr: |
          sre:checkout_requests:error_ratio1h > (14.4 * 0.001)
          and
          sre:checkout_requests:error_ratio5m > (14.4 * 0.001)
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "checkout burning error budget at 14.4x — 2% of the 30-day budget in 1h"
          runbook_url: "https://runbooks.acme.io/checkout-api#first-five-minutes"   # NEW

The same runbook_url goes on all four alerts, page and ticket tier alike, and it deliberately points at the top-level first-five-minutes section rather than any single dependency entry — the alert can't tell you which of the three dependencies is involved, so its runbook_url shouldn't pretend to guess either. The triage step you just read is what routes a responder to the payment-processor entry, the inventory entry, or the database entry from there. Reload the same way Part 2 already showed:

curl -s -X POST http://localhost:9090/-/reload
curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[].rules[] | select(.type=="alerting") | {name, annotations}'
# every one of the four alerts should now show a runbook_url in its annotations

This is also the concrete artifact behind one line of the on-call readiness checklist: "every alert has been tested for actionability: it links to a runbook, and a recent audit confirms it isn't firing without a required response." checkout-api can now check that box honestly, with a URL that resolves to real content instead of a promise to write one someday.

What "done" looks like for Part 3

☺ Like you're 10: A phone that actually rings, a backup that actually answers if it doesn't, and a page that finally tells the person holding it what to do next.

At the end of this part: checkout-api is a real PagerDuty service behind a three-level escalation policy you have personally watched fall through from Asha to Marco under a real drill event; Alertmanager's page-oncall and file-ticket receivers point at a real routing key and a real Slack channel instead of two localhost stand-ins; a first-five-minutes checklist exists that starts with "acknowledge" and ends with "escalate on purpose if the clock is about to do it for you"; three runbook entries document the payment processor, the inventory service, and the order database — the three dependencies Part 1 named as where Part 4's incident and Part 5's capacity plan will both trace back to; and every one of Part 2's four alert rules carries a runbook_url that resolves to real content. Nothing here gets thrown away:

Part 3 gave checkout-api's page somewhere real to go: a tested escalation policy with a genuine backup, a first-five-minutes checklist, and three runbook entries that turn "the phone is ringing" into "here's exactly what to check first." Continue to Capstone Part 4 — Run the Incident & Write the Postmortem, where this exact runbook gets used for real against a scenario that spends real minutes of Part 1's budget. Or step back to Run a Reliable Service — start here for the full six-part map, revisit incident management & on-call and PagerDuty for the concepts behind what you built today, or get a faster, standalone rep of the escalation-design instinct with Drill — Incident Response Tabletop.