PagerDuty

PagerDuty is the layer that sits between "something is wrong" and "the right person's phone is ringing." Incident management already assumes that layer exists — a severity gets set, a primary on-call gets paged, an escalation policy backs them up if they don't answer within a few minutes. This page is about the tool that makes that assumption true: how a raw event from a monitoring system becomes a routed, deduplicated incident with an audit trail; how services, schedules, and escalation policies compose into something that reliably reaches a human; and the configuration and integration details that separate an escalation policy that works from one that quietly fails the first time it's tested for real.

What PagerDuty is and the problem it solves

☺ Like you're 10: PagerDuty is the switchboard that decides who to call, in what order, how loud to make the phone ring, and what to do if nobody picks up.

PagerDuty was founded in 2009 as an on-call scheduling tool and has since grown into a broader incident-response and automation platform (it went public on the NYSE under ticker PD in April 2019), but the core mechanic that made it indispensable hasn't changed: take an event from any monitoring source and turn it into a human response with a guaranteed, auditable path, rather than hoping the right person happens to see a Slack message or an email at the right moment. It is not a monitoring tool itself — it generates nothing to alert on. Prometheus, Datadog, New Relic, and dozens of other systems decide whether something is wrong; PagerDuty decides who finds out, how, and what happens if they don't respond.

The problem before a dedicated on-call layer

Before a tool owns this job, teams reach for whatever's already lying around: a spreadsheet roster nobody updates after the third schedule change, a distribution-list email that lands in an inbox nobody's watching at 3 a.m., a phone tree someone has to dial manually, or a Slack thread asking "who's on call this week?" in the middle of an actual outage. Monitoring can detect a real problem perfectly and it still goes nowhere, because nothing owns the "find a human, right now, and prove it happened" step with any reliability. That's a structural failure, not a people failure — the fix is a system that treats paging as seriously as the deploy pipeline treats a release, with the same kind of guardrails and audit trail.

The three nouns: Service, Escalation Policy, Schedule

Almost every PagerDuty configuration question reduces to these three, and they compose in one direction. A Service represents one monitored thing — checkout-api, payments-db — and is the object that events actually get routed to. Every Service has exactly one Escalation Policy: an ordered list of rules, each naming a target and a delay before moving to the next rule if nobody's acknowledged. A rule's target is almost always a Schedule — a definition of who's on call at any given moment, built from one or more rotating Layers, temporary Overrides for one-off swaps, and optional Restrictions that limit a layer to specific hours. Read it top to bottom: an event hits a Service, the Service's Escalation Policy pages whoever a Schedule says is on call, and if nobody answers, the policy moves to its next rule on a timer.

Where PagerDuty fits: the alerting layer incident-management.html assumes

☺ Like you're 10: Incident-management.html writes the rules — SEV1 pages everyone, the secondary backs up the primary; PagerDuty is the software that actually enforces those rules while everyone's asleep.

Every mechanic named in incident management has a direct PagerDuty implementation. A severity scale becomes a Service's urgency setting (and, per user, which notification rules even fire — more below). The primary/secondary on-call rotation is a Schedule with two layers, or two Schedules chained across escalation rules. The "auto-escalates after five to fifteen minutes" behavior is a rule's escalation delay, in minutes, configured directly. And the incident commander, status page, and stakeholder-communication cadence that page describes as process are, in PagerDuty, backed by real objects: an assigned responder, a status update posted to the incident, and — as covered below — a fully automatic timeline recording every step regardless of whether anyone remembers to write it down. This page assumes that process context and goes deep on the tool mechanics underneath it. For the deeper, org-level side of sustainable rotations — compensation, swap etiquette, and burnout — see On-Call Culture & Sustainable Operations; this page stays on the configuration that makes a rotation actually work when the page fires.

PagerDuty sits strictly downstream of whatever's covered in monitoring & observability — it never decides whether something is broken, only what happens once a monitoring system has already decided that for it. Two distinct APIs mediate that boundary, and mixing them up is a common first mistake: the Events API v2 (events.pagerduty.com) is the narrow, high-volume endpoint monitoring tools call to report a raw event; the REST API v2 (api.pagerduty.com) is the full CRUD API humans and infrastructure-as-code tools use to manage services, schedules, and users. The architecture section below walks the first one in detail; the operations section covers the second.

Architecture: how an event becomes a phone call

☺ Like you're 10: One alert takes a short trip through PagerDuty's own machinery before it ever reaches a phone — and every stop on that trip gets written down.

The diagram below is the whole pipeline. A monitoring tool posts a JSON event to the Events API v2; PagerDuty's Event Orchestration layer evaluates that event against a set of conditions and routes it (or suppresses it, or enriches it) to a Service; the Service creates an Incident and hands it to its Escalation Policy; the policy pages whoever the target Schedule says is currently on call, using that person's own per-urgency notification rules to decide push, SMS, phone call, or email, in what order and after what delay. If nobody acknowledges before the rule's escalation delay expires, the policy moves to its next rule automatically — no human has to notice the silence.

The incident lifecycle itself is a small state machine: triggered when the event first creates it, acknowledged when a responder takes ownership (which pauses further escalation), and resolved when it's closed out — manually, or automatically if the source system sends a matching resolve event. The dedup_key is what ties a whole stream of related events to one incident: as long as an incident with that key is open, more trigger events carrying the same key add to its history instead of opening a new incident, and a resolve event with that key closes it. Send another trigger with the same key after it's resolved, though, and PagerDuty opens a brand-new incident — a resolved incident never reopens, it just gets succeeded by a new one with a fresh ID and the same key.

The building blocks you configure: schedules, escalation policies, services

☺ Like you're 10: Three small files describe almost the whole on-call setup: who's up this week, what order to try them in, and which alarm they're attached to.

PagerDuty's own web UI is a perfectly reasonable way to build a first schedule, but any team running more than a handful of services quickly wants this in version control, reviewed like any other production config — which is exactly what PagerDuty's official Terraform provider is for. The three resources below are the ones you'll actually write.

Schedules: layers, restrictions, overrides

A schedule's layer block is one rotation: a start time, a rotation length in seconds, and an ordered list of users who cycle through it. restriction blocks narrow a layer to specific hours — the standard way to build a "business hours" schedule that only pages during the day, layered underneath a 24/7 catch-all for genuine emergencies.

resource "pagerduty_schedule" "checkout_primary" {
  name      = "Checkout — Primary"
  time_zone = "America/New_York"     # pick the team's actual zone, not UTC — handoff math gets confusing otherwise

  layer {
    name                         = "Weekly rotation"
    start                        = "2026-01-05T09:00:00-05:00"
    rotation_virtual_start       = "2026-01-05T09:00:00-05:00"
    rotation_turn_length_seconds = 604800   # 7 days
    users = [
      pagerduty_user.alice.id,
      pagerduty_user.bilal.id,
      pagerduty_user.carmen.id,
    ]

    # only page this layer during business hours; a second, 24/7 layer underneath covers the rest
    restriction {
      type              = "weekly_restriction"
      start_time_of_day  = "09:00:00"
      duration_seconds   = 32400         # 9 hours
      start_day_of_week  = 1             # Monday
    }
  }
}

A one-off swap — someone's sick, someone's traveling — is an override, applied through the UI, the mobile app, or the REST API, never by editing the base layer. Overrides sit on top of the computed rotation without touching the Terraform-managed layer definition at all, which is exactly why they exist: the schedule's long-term shape stays reviewable and stable while short-term reality stays flexible.

Escalation policies: rules, delays, and loops

Each rule block is one rung of the ladder: a target (almost always a schedule reference) and how many minutes to wait before trying the next rung. num_loops controls what happens if the policy reaches the bottom with nobody having acknowledged — repeat the whole ladder that many more times before giving up.

resource "pagerduty_escalation_policy" "checkout" {
  name      = "Checkout Escalation Policy"
  num_loops = 2     # if nobody acks after the last rule, start over from the top, twice

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.checkout_primary.id
    }
  }
  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.checkout_secondary.id
    }
  }
  rule {
    # the final rung is a named person on purpose — an engineering manager who
    # needs to know an incident has escalated this far, not a rotating schedule
    target {
      type = "user_reference"
      id   = pagerduty_user.eng_manager.id
    }
  }
}

Services and integrations

A service ties a monitored thing to the escalation policy that protects it, and its incident_urgency_rule is what decides whether pages from it are high or low urgency by default — which, as the notification-rules section below covers, is the single setting that most often silently determines whether a responder's phone actually makes noise.

resource "pagerduty_service" "checkout" {
  name                    = "checkout-api"
  escalation_policy       = pagerduty_escalation_policy.checkout.id
  alert_creation          = "create_alerts_and_incidents"
  auto_resolve_timeout    = 14400   # 4h — auto-resolve a stale incident nobody closed
  acknowledgement_timeout = 1800    # 30m — re-escalate if an ack goes stale with no further update

  incident_urgency_rule {
    type    = "constant"
    urgency = "high"
  }
}

# vendor name strings must match PagerDuty's own vendor catalog exactly — verify
# the current name against the account's Service Directory before relying on it
data "pagerduty_vendor" "alertmanager" {
  name = "Prometheus"
}

resource "pagerduty_service_integration" "prometheus" {
  name    = "Prometheus Alertmanager"
  service = pagerduty_service.checkout.id
  vendor  = data.pagerduty_vendor.alertmanager.id
}

Incident timelines and automated actions

☺ Like you're 10: PagerDuty writes down everything that happens to an incident the moment it happens, so nobody has to reconstruct the story from memory afterward — and it can even take some actions on its own.

Every state change on an incident — triggered, acknowledged by whom, escalated to whom, reassigned, priority set, a note added, resolved — lands automatically in that incident's timeline, timestamped and attributed to whichever human or automation caused it. This is the underlying record that feeds both a post-incident review and PagerDuty's own Analytics module, which rolls timelines up into per-service and per-team metrics like mean time to acknowledge (MTTA) and mean time to resolve.

Two separate mechanisms let PagerDuty act, not just record and page, and it's worth keeping them apart because they trigger at different points in an incident's life. Automation Actions — marketed today as PagerDuty Process Automation at the fuller end (built on PagerDuty's 2022 acquisition of Rundeck) and as lighter-weight actions attached directly to a service at the simpler end; confirm current product names on PagerDuty's own site, since the branding here has shifted more than once — let a responder trigger a predefined script, webhook, or runbook step directly from inside an open incident, with the run and its output logged straight to that incident's timeline. Incident Workflows are the newer, broader mechanism: a no-code sequence of steps — post to a Slack channel, open a Jira ticket, run an Automation Action, send a stakeholder update, page a second team — that fires automatically the moment an incident matches a condition, such as any P1 on a specific service.

The distinction that actually matters when designing either one: Event Orchestration (covered next) acts before an incident exists, deciding how a raw event gets routed, suppressed, or enriched. Incident Workflows act after an incident already exists, orchestrating the response around it. Confusing the two produces rules in the wrong place — someone tries to "route to Slack" inside Event Orchestration when that's really a workflow's job, or tries to dedupe noisy events inside a workflow when that has to happen upstream, at the router, before an incident is even created.

Integration patterns with monitoring tools

☺ Like you're 10: Any tool that can send a webhook can talk to PagerDuty, but the popular ones ship a pre-built adapter so you don't have to hand-map every field yourself.

The universal integration path is the Events API v2 directly — any system that can make an HTTP POST can page through PagerDuty with no special adapter at all:

# trigger — opens (or updates) an incident identified by dedup_key
$ curl -X POST https://events.pagerduty.com/v2/enqueue \
    -H 'Content-Type: application/json' \
    -d '{
      "routing_key": "R0ABCDEF1234567890ABCDEF12345678",
      "event_action": "trigger",
      "dedup_key": "checkout-5xx-us-east-1",
      "payload": {
        "summary": "Checkout API 5xx rate above 5% for 5m",
        "source": "prometheus-alertmanager",
        "severity": "critical",
        "custom_details": { "runbook": "https://wiki.internal/checkout-5xx" }
      }
    }'

# resolve — same dedup_key, closes the incident it opened
$ curl -X POST https://events.pagerduty.com/v2/enqueue \
    -H 'Content-Type: application/json' \
    -d '{"routing_key":"R0ABCDEF1234567890ABCDEF12345678","event_action":"resolve","dedup_key":"checkout-5xx-us-east-1"}'

Most teams never hand-write that JSON, though, because the tool already producing the alert usually ships a purpose-built PagerDuty integration. Prometheus's Alertmanager has a native pagerduty_configs receiver — worth setting dedup_key explicitly here, since Alertmanager's default grouping key doesn't always line up with the incident boundary you actually want:

# alertmanager.yml
receivers:
  - name: pagerduty-oncall
    pagerduty_configs:
      - routing_key: "${PAGERDUTY_ROUTING_KEY}"
        severity: '{{ .CommonLabels.severity }}'
        dedup_key: '{{ .GroupLabels.alertname }}-{{ .GroupLabels.cluster }}'
        description: '{{ .CommonAnnotations.summary }}'

Datadog pages through a mention in the monitor's own message — @pagerduty-checkout — once the Datadog-PagerDuty integration tile is configured with that service's routing key; Grafana's unified alerting has a pagerduty contact-point type that takes the same routing key directly; New Relic routes through its own notification-workflow destinations; and CloudWatch typically fans out through an SNS topic PagerDuty's AWS integration subscribes to. The field names differ across all of them, but every single one is doing the same thing under the hood: filling in routing_key, dedup_key, and a severity before handing the event to the Events API.

Whatever arrives at that endpoint can be filtered, rerouted, or enriched centrally by an Event Orchestration router before it ever reaches a service — useful when several noisy upstream tools all need the same suppression or dedup logic applied consistently, rather than trusting every tool's own config to get it right:

# the exact resource and attribute names for Event Orchestration have moved more
# than once as the feature matured — check the current PagerDuty provider docs
# before shipping this; the shape below matches the schema at time of writing
resource "pagerduty_event_orchestration" "checkout" {
  name = "Checkout Event Orchestration"
}

resource "pagerduty_event_orchestration_router" "checkout" {
  event_orchestration = pagerduty_event_orchestration.checkout.id

  set {
    id = "start"
    rule {
      label = "Route known flaky health-check noise to a low-urgency service"
      condition {
        expression = "event.summary matches part 'health check timeout'"
      }
      actions {
        route_to = pagerduty_service.checkout_low_urgency.id
      }
    }
  }

  catch_all {
    actions {
      route_to = pagerduty_service.checkout.id
    }
  }
}

Day-to-day operations: the API, Terraform, and ChatOps

☺ Like you're 10: There's no single command-line tool everyone uses day to day — most of the real interaction happens through the phone in your pocket, a Slack command, or a config file in Git.

Unlike Kubernetes or Terraform, PagerDuty has no single dominant CLI most teams reach for. Day-to-day interaction splits across four surfaces instead. The mobile app is where a responder actually acknowledges, resolves, reassigns, or escalates a page the moment it arrives — usually within seconds of the notification, not from a laptop. ChatOps commands through the Slack integration cover the common lookups without leaving a channel:

/pagerduty trigger "Checkout is throwing 500s"      # open an incident from Slack directly
/pagerduty ack PD-1234                               # acknowledge it
/pagerduty resolve PD-1234                            # close it out
/pagerduty oncall "Checkout — Primary"                # who's actually on call right now

The REST API v2 is what scripts, dashboards, and one-off operational tasks use, authenticated with an API token in the request header:

$ curl -H 'Authorization: Token token=YOUR_API_KEY' \
    'https://api.pagerduty.com/oncalls?schedule_ids[]=P1234AB'   # who's on call, right now, for a schedule

$ curl -X POST -H 'Authorization: Token token=YOUR_API_KEY' -H 'Content-Type: application/json' \
    https://api.pagerduty.com/schedules/P1234AB/overrides \
    -d '{"override":{"start":"2026-08-18T00:00:00Z","end":"2026-08-19T00:00:00Z","user":{"id":"PUSER01","type":"user_reference"}}}'

And the Terraform provider is how the resources from the previous section actually get applied — terraform plan against a schedule or escalation-policy change is a pull request like any other, reviewable before it's live, which matters enormously for a config that decides who gets woken up.

Gotchas and failure modes

☺ Like you're 10: Most PagerDuty failures aren't the software breaking — they're a setting that looked reasonable on the day someone configured it and quietly stopped being true.

Notification rules configured for only one urgency. Every user's own notification rules — push after zero minutes, SMS after two, phone call after five — are set separately for high urgency and low urgency. It's common for someone to configure high-urgency rules carefully on day one and never touch the low-urgency tab at all, leaving it empty. Any incident routed as low urgency then notifies that person through nothing — not a bug, just a config gap nobody noticed because low-urgency incidents are, by definition, the ones nobody's watching closely.

An escalation delay set too aggressively short. A one- or two-minute delay sounds responsive, but it's often shorter than the time it physically takes a person to unlock a phone and read a notification. The policy escalates past the primary before they've had a real chance to acknowledge, defeating the entire point of having a primary — and training the secondary to expect (and dread) pages that were never really theirs.

Hand-edited schedules and policies drift from Terraform. An engineer opens the UI to fix something under pressure during an incident — perfectly reasonable in the moment — and the fix never makes it back into the Terraform config. The next terraform apply silently reverts it, and the "fix" is gone with no diff anyone reviewed either way. Overrides are the sanctioned escape hatch for exactly this reason: use them for anything temporary, and treat the base schedule and policy as something that only changes through a reviewed pull request.

A maintenance window that nobody remembers to end. Suppressing alerts during planned work is correct; forgetting to lift the suppression afterward means the team believes monitoring is healthy when it's simply muted. This is the same failure shape as an ignored alert that Prometheus's Alertmanager silence mechanism warns about — a maintenance window is a deliberate, temporary decision, not a permanent fix for a chronically noisy check that should be tuned or removed instead.

Alternatives and when to choose it

☺ Like you're 10: A few other tools do the same core job — find a human, page them, escalate if they don't answer — and the real choice usually comes down to what your team already pays for.

The practical rule most teams land on: reach for PagerDuty or Opsgenie once more than one team needs paging and the escalation logic has any real shape to it — the moment "who gets called" stops being a single person's job, the audit trail and reviewable-config story pay for themselves the first time an incident review needs to reconstruct exactly what happened and when. See the DevOps toolchain for how this slot fits among the rest of the stack, and Incident & Event Response for the broader process this tool implements. Put the escalation-policy design from this page into practice in Drill — Set Up Meaningful Alerts and Capstone Part 5 — Incident Response. PagerDuty also runs free, self-paced credentials through PagerDuty University — a practitioner-level course is the usual entry point — but confirm current names, format, and availability on PagerDuty's own site rather than trusting a fixed description here, since these have been renamed before.