PagerDuty

PagerDuty is the tool most of this course's escalation-policy language was written against before it ever had a vendor name attached to it. The primary-then-secondary rotation, the acknowledgement window, the automatic hand-off to a second responder described in incident management & on-call — that's the abstract shape of the mechanic. PagerDuty is the concrete machine that implements it: a service receives an event, points at one escalation policy, and that policy works down an ordered list of on-call schedules and people, waiting a fixed number of minutes between each try before it moves to the next. Founded in 2009 by three engineers who'd lived through enough 3am pages at Amazon to build something better, and public on the NYSE (ticker PD) since 2019, PagerDuty became the long-standing market leader at exactly this job. This page covers that routing model in full, then the layer most teams under-use once a human is already on the line: Event Orchestration, incident workflows, status pages, and postmortems.

What PagerDuty is and the problem it solves

☺ Like you're 10: It's the phone tree that never forgets to call the next person if the first one doesn't pick up — and these days it also runs a chunk of the response after somebody finally does.

PagerDuty was founded in 2009 by Alex Solomon, Andrew Miklas, and Baskar Puvanathasan, three engineers who'd carried a pager at Amazon and built the product they wished they'd had, going through Y Combinator before growing into the category's default choice. It went public on the NYSE under ticker PD in 2019. Over the years the core paging mechanic has been folded into a broader suite the company now markets as the PagerDuty Operations Cloud — Incident Management, On-Call Management, Automation (Runbook Automation, built on Rundeck, which PagerDuty acquired in 2022), AIOps/Event Intelligence, Status Pages, and Analytics, all sharing one incident record and one permission model. Feature names and packaging shift with product releases, so treat the shape described on this page as durable and verify current naming against PagerDuty's own docs before you build against it.

PagerDuty doesn't generate alerts — it receives them. A monitoring tool (Prometheus Alertmanager, Datadog, CloudWatch, a homegrown script) sends an event in through one of several hundred built-in integrations or a generic webhook, and PagerDuty's job starts the instant that event lands: decide which service it belongs to, decide whether it's new or a repeat of something already in flight, and if it's new, work through that service's escalation policy until a human acknowledges it. Everything else on this page — Event Orchestration, incident workflows, status pages — is built on top of that one core loop.

The escalation-policy model: services, schedules, levels, and teams

☺ Like you're 10: An alert lands on one service, that service points at one escalation policy, and the policy is just an ordered list of "try this person, then that person, then this whole team," with a timer between each try.

Four objects do essentially all the work, and precision about which one owns which job is what makes the rest of this page make sense.

• Service — the thing being monitored: one microservice, one component, one team-owned system (checkout-api). A service has exactly one escalation policy at a time, but can carry many Integrations underneath it — one per upstream monitoring tool — each stamped with its own unique integration key (also called a routing key), so an Alertmanager webhook and a Datadog monitor can both feed the same service without PagerDuty confusing which is which.
• On-call schedule — who's up this week. A schedule is built from one or more layers: a base layer defines the rotation itself (daily, weekly, or a custom N-day cadence, with an explicit start time and time zone), and an override layer sits on top of it for one-off swaps — a PTO cover, a schedule swap — without touching the underlying rotation. Because the time zone lives on the schedule, not scattered across each engineer's head, daylight-saving transitions are handled once, centrally, instead of by a dozen people doing their own arithmetic.
• Escalation policy — an ordered list of levels (PagerDuty's API calls them rules). Each level names one or more targets — a schedule, a specific user, or in nested setups another escalation policy — and an escalation_delay_in_minutes: how long PagerDuty waits for an acknowledgement before moving to the next level. A num_loops setting controls how many times the whole chain repeats once it reaches the last level before PagerDuty gives up and simply leaves the incident open, unacknowledged.
• Team — the organizational and permission boundary. A team groups a set of services, the schedules and escalation policies behind them, and the users who belong to it, and scopes who can view or edit that slice of the account. This is the literal mechanism behind "service-to-team routing": a service is owned by a team, and everything else — who's allowed to silence its alerts, whose schedule it escalates through — inherits from that ownership.

Event Orchestration, alert grouping, and the urgency/priority split

☺ Like you're 10: Before anyone gets paged, PagerDuty first has to decide whether this event is a brand-new problem or just one more symptom of a fire it's already paging someone about.

Raw events don't walk straight into an escalation policy — they pass through Event Orchestration first, a rules engine (the successor to PagerDuty's older Event Rules and Rulesets) evaluated top-to-bottom against the incoming payload, at both a global level and per service. A rule can route an event to a specific service, set its priority, suppress it entirely, override its severity, or extract structured fields out of unstructured payload text with a regex — all before an escalation policy ever sees it.

The single most important field in that payload is the dedup key. Every event sent to PagerDuty's Events API carries an event_action — trigger, acknowledge, or resolve — and a dedup_key, and PagerDuty treats every event sharing the same key as the same underlying problem rather than a new one. A monitoring rule that fires trigger fifteen times in ten minutes for one flapping check should send the same dedup_key every time; PagerDuty updates the one open alert instead of opening fifteen incidents and paging someone fifteen times over. Beyond dedup, each service also has an alert grouping setting that decides how multiple distinct alerts become one page-worthy incident: content-based (group alerts sharing identical field values — the same error signature), time-based (group anything arriving within a configurable window), or Intelligent Alert Grouping, a machine-learning feature under PagerDuty's AIOps/Event Intelligence add-on that clusters related alerts without a human writing the grouping rule by hand.

The config you actually write

☺ Like you're 10: Four blocks of Terraform build the whole chain: a person, a rotation for that person, a policy that says who to call and when, and a service that points at the policy.

Nearly every mature PagerDuty account manages schedules, escalation policies, services, and integrations through the official Terraform provider rather than the web UI, for the same reason any other production config lives in git: reviewable diffs, no click-ops drift, and a rollback path. Field names below match the provider's shape at the time of writing — the provider has renamed a few attributes across major versions, so check the current schema on the Terraform Registry before copying this into a real module.

resource "pagerduty_user" "asha" {
  name  = "Asha Patel"
  email = "asha@acme.io"
  role  = "user"
}

resource "pagerduty_schedule" "checkout_primary" {
  name      = "checkout-primary-oncall"
  time_zone = "America/New_York"      # DST handled once, here — not per engineer

  layer {
    name                          = "Weekly rotation"
    start                         = "2026-01-05T09:00:00-05:00"
    rotation_virtual_start        = "2026-01-05T09:00:00-05:00"
    rotation_turn_length_seconds  = 604800            # 7 days
    users = [
      pagerduty_user.asha.id,
      pagerduty_user.marco.id,
    ]
  }
}

resource "pagerduty_escalation_policy" "checkout" {
  name      = "checkout-primary-ep"
  num_loops = 2                       # repeat the whole chain twice before giving up

  rule {
    escalation_delay_in_minutes = 15  # matches the default this course assumes elsewhere
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.checkout_primary.id
    }
  }
  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "user_reference"
      id   = pagerduty_user.team_lead.id
    }
  }
}

resource "pagerduty_service" "checkout" {
  name                    = "checkout-api"
  escalation_policy       = pagerduty_escalation_policy.checkout.id
  alert_creation          = "create_alerts_and_incidents"
  acknowledgement_timeout = 1800      # re-escalates if acked but not resolved in 30m
  auto_resolve_timeout    = 14400     # auto-resolves a stale, untouched incident in 4h

  incident_urgency_rule {
    type    = "constant"
    urgency = "high"
  }
}

data "pagerduty_vendor" "alertmanager" {
  name = "Prometheus"
}

resource "pagerduty_service_integration" "checkout_prometheus" {
  name    = "Prometheus Alertmanager"
  service = pagerduty_service.checkout.id
  vendor  = data.pagerduty_vendor.alertmanager.id
}

The integration resource is what actually mints the integration/routing key Alertmanager's pagerduty_configs receiver — or any other source — needs to send events at this specific service.

Day-to-day operations: the REST API, overrides, and the CLI gap

☺ Like you're 10: There's no single command-line tool for PagerDuty the way there is for Kubernetes — instead there are two HTTP APIs, one for sending events in and one for asking questions or making changes, plus the web UI for everything else.

Unlike Prometheus's promtool or Helm's own binary, PagerDuty ships no first-party CLI. Day-to-day interaction happens through the web UI, the Terraform provider for anything declarative, and two REST-ish HTTP APIs for anything scripted: the Events API v2 for sending alerts in, and the general-purpose REST API for reading and mutating almost everything else — incidents, schedules, overrides, users. (Community-maintained CLIs exist; none is officially supported, so treat them as convenience wrappers over the same APIs shown here, not a source of truth.)

# Events API v2 — trigger a new incident (or update one sharing the same dedup_key)
$ curl -X POST https://events.pagerduty.com/v2/enqueue \
    -H 'Content-Type: application/json' \
    -d '{
      "routing_key": "R0ABC1234567890DEADBEEF00112233",
      "event_action": "trigger",
      "dedup_key": "checkout-api-5xx-us-east-1",
      "payload": {
        "summary": "checkout-api error rate above SLO burn threshold",
        "source": "alertmanager/prod-us-east-1",
        "severity": "critical",
        "timestamp": "2026-08-16T03:14:00Z",
        "custom_details": { "burn_rate_1h": 14.2, "slo": "checkout-availability-99.9" }
      }
    }'

# same dedup_key, event_action resolve — closes it out once the burn rate recovers
$ curl -X POST https://events.pagerduty.com/v2/enqueue \
    -H 'Content-Type: application/json' \
    -d '{"routing_key":"R0ABC...","event_action":"resolve","dedup_key":"checkout-api-5xx-us-east-1"}'

# REST API — who's actually on call right now for a given schedule
$ curl -s "https://api.pagerduty.com/oncalls?schedule_ids[]=PSCHEDULEID" \
    -H 'Authorization: Token token=YOUR_API_KEY' \
    -H 'Accept: application/vnd.pagerduty+json;version=2'

# REST API — acknowledge an incident (mutating calls require a valid From user email)
$ curl -X PUT https://api.pagerduty.com/incidents/PINCIDENTID \
    -H 'Authorization: Token token=YOUR_API_KEY' \
    -H 'Content-Type: application/json' \
    -H 'From: asha@acme.io' \
    -d '{"incident": {"type": "incident_reference", "status": "acknowledged"}}'

# REST API — a one-off schedule override, covering someone's shift for a day
$ curl -X POST https://api.pagerduty.com/schedules/PSCHEDULEID/overrides \
    -H 'Authorization: Token token=YOUR_API_KEY' \
    -H 'Content-Type: application/json' \
    -d '{"overrides":[{"start":"2026-08-20T00:00:00Z","end":"2026-08-21T00:00:00Z",
        "user":{"id":"PUSERID","type":"user_reference"}}]}'

$ terraform plan   # review before shipping any escalation-policy or schedule change
$ terraform apply

Incident-response workflow beyond raw paging

☺ Like you're 10: Getting someone's phone to ring is the easy 10% — the rest of PagerDuty's product is everything that happens between "phone rings" and "postmortem gets filed."

This is the half of PagerDuty most teams that adopted it purely for paging never turn on, and it's the half the content this page sits alongside — incident management & on-call — describes in the abstract. Incident Workflows are automations triggered on lifecycle events (an incident is triggered, acknowledged, escalated, resolved) that can spin up a dedicated Slack channel, invite the right subject-matter responders, post a structured update, or kick off a Runbook Automation job, without a human doing any of it by hand mid-incident. Status updates are a structured field on the incident itself, distinct from ad hoc chat messages, meant to be the single source of truth an Ops/Comms lead posts outward on a cadence — the exact role described in incident command for large incidents. Post-Incident Reviews (PagerDuty's postmortem tooling) auto-populate a timeline from the incident's own log entries — who acknowledged, who escalated, when status updates went out — so the blameless postmortem starts from a real timestamped record instead of everyone's imperfect memory the next morning. Runbook Automation, built on the Rundeck acquisition, lets a responder trigger a pre-approved remediation job — restart a service, roll back a deploy, scale a pool — directly from the incident, the same automate-the-repeat-fix instinct toil and automation teaches applied at the moment of the page itself. And Status Pages is PagerDuty's own native public-status-page product bundled into the same platform — worth distinguishing from Atlassian's separately owned Statuspage.io, a similarly named but unrelated product some organizations also run — letting the incident's status updates publish outward to customers without the Comms lead re-typing them into a second tool. Analytics rolls all of this up into org-level numbers: mean time to acknowledge (MTTA), mean time to resolve (MTTR), and per-responder page load, the same health signals a sustainable on-call rotation is measured against.

Gotchas and failure modes

☺ Like you're 10: Almost every PagerDuty incident-about-PagerDuty traces back to one of a handful of settings nobody double-checked before the real incident arrived.

Urgency, not priority, controls whether the phone rings

Covered above and worth repeating as a gotcha in its own right: teams that set priority and assume it drives notification behavior discover during a live SEV1 that the field they configured never touched the escalation policy at all. If a call needs to happen, urgency is the field to check.

An exhausted escalation policy doesn't page anyone else on its own

When num_loops runs out and nobody has acknowledged, PagerDuty doesn't invent a new target — the incident simply sits open and unacknowledged, visible in the dashboard but silent to any human who isn't already looking. The fix is a deliberate final level (page the whole team, or a manager-escalation target) rather than trusting the loop count alone, and it's exactly the gap the on-call readiness checklist asks a team to verify before go-live, not discover during an incident.

A missing dedup key turns one flapping check into a dozen pages

Every monitoring integration that constructs its own Events API calls has to supply a stable dedup_key per underlying condition. Omit it, or derive it from something that changes on every fire (a timestamp, a random ID), and a single flapping alert opens a new incident — and a fresh page — on every trigger, which is precisely the noisy-pager failure mode alert design & alert fatigue covers in depth.

Auto-resolve can quietly close a stuck incident

The auto_resolve_timeout shown in the Terraform example above exists to clean up incidents nobody will ever act on, but the same setting will silently resolve a real, still-broken incident if it's set too aggressively relative to how long a genuine fix can take — a resolved incident stops paging and stops showing up as "open" in dashboards, so a too-short timeout can make an unresolved outage look finished.

Schedule layers and time-zone math are the most common on-call-config bug

An override layer stacks on top of a base rotation layer rather than replacing it outright, and a schedule built by several people over time without anyone reviewing it end to end can quietly develop an uncovered gap — a window where the resolved on-call user is nobody, because a rotation and an override disagree about who owns it. PagerDuty's own schedule preview UI is the fastest way to check this before it becomes a live incident with no primary; treating a schedule change like any other production change, reviewed before it ships, catches most of these.

PagerDuty vs. the alternatives

☺ Like you're 10: Other tools ask the same "call the right person, in the right order" question — the real differences are depth of incident-response features, ecosystem fit, and price.

The practical rule most teams land on: PagerDuty earns its price when the incident-workflow, status-page, and runbook-automation features actually get switched on and used — otherwise a team is paying suite pricing for what a leaner competitor's raw escalation policy would have covered just as well. See the SRE toolchain for how the on-call-and-paging category sits next to monitoring and chaos tooling, and verify current plans, seat pricing, and feature-tier boundaries on PagerDuty's own site before committing a budget, since packaging shifts more often than the underlying escalation-policy model does.

Going further

☺ Like you're 10: This page is enough to configure a real escalation policy and read a real incident — PagerDuty's own docs are where the newest feature names and current pricing actually live.

The canonical sources are PagerDuty's product documentation at support.pagerduty.com, the API reference at developer.pagerduty.com (Events API v2 and the REST API both documented there in full), and the official Terraform provider on the Terraform Registry. Pair this page with incident management & on-call for the human process this tool exists to support, alert design & alert fatigue for how to keep what reaches PagerDuty actionable in the first place, and SRE Tools & Automation for where this category sits in the exam blueprint. If you want hands-on reps writing an actual runbook against a schedule and an escalation policy rather than just reading about one, Capstone Part 3 — write the on-call runbook is exactly that exercise.