Grafana OnCall

Grafana OnCall is Grafana Labs' answer to a question every team eventually asks after adopting Prometheus and Grafana for free: why is the last mile — turning a firing alert into a phone call that keeps escalating until a human answers — the one piece of the stack with a recurring per-seat invoice attached? OnCall is an open-source-first paging engine that plugs directly into Grafana's own unified alerting and into Prometheus Alertmanager, lets you define schedules and escalation chains as version-controlled code instead of clicking them into a vendor's console, and is available either fully managed inside Grafana Cloud or as an engine you run yourself. This page covers its data model, how to provision it as code, the day-to-day API and chatops surface, and the tradeoff that makes self-hosting it a genuine engineering decision rather than a free lunch: once OnCall is the thing that pages you, OnCall's own uptime is now part of your incident response, and something has to watch it.

What Grafana OnCall is, and where it fits in the paging landscape

☺ Like you're 10: It's the piece that turns "an alert fired" into "a phone is ringing," and unlike most of that category it's software you're allowed to read, run yourself, and never pay a per-person bill for.

OnCall began life as a product called Amixr, which Grafana Labs acquired in 2021 and relaunched under its own name as an open-source-first member of the Grafana stack — worth knowing, because older blog posts, GitHub issues, and Stack Overflow answers still sometimes refer to it by the original name. Architecturally it sits exactly where Grafana itself stops: Grafana's unified alerting engine can evaluate a query and decide something is wrong, and it can hand that decision to a contact point, but nothing in Grafana knows what a rotation is, who's covering this week, or how long to wait before trying someone else. That's OnCall's entire job — scheduling, escalation, acknowledgement, and the notification channels (Slack, mobile push, SMS, phone calls) that actually reach a human — described in general terms for every vendor in this category in incident management & on-call.

The open-core shape matters more here than for most tools on this list, because it's the whole reason the tool exists. The engine — the Django backend that owns schedules, escalation chains, and alert-group state — is open source under the Apache-2.0 license at the time of writing (confirm on the repo; Grafana Labs has revisited licensing terms for other parts of the stack before, so don't assume it's permanent). You can run that engine yourself, for the cost of the infrastructure under it and zero seat licensing. Or you can let Grafana Labs run it for you as a bundled feature of Grafana Cloud, trading that operational ownership for usage-based Cloud billing. Both paths present the identical product and API; the decision between them is the subject of most of this page.

The core nouns: integration, route, escalation chain, schedule

☺ Like you're 10: A doorbell button, a rule for which room to ring it in, the list of people it tries in order, and the calendar that says whose turn it is — four different things that are easy to mix up until you've said their names a few times.

Almost every OnCall misconfiguration is a confusion between these five terms. Learn them once and the UI, the API, and the Terraform resources all stop feeling arbitrary.

Architecture: what the OSS engine actually runs

☺ Like you're 10: It isn't one program — it's a small fleet of pieces that all have to be up at once for a page to actually happen, and every one of them is now something you're responsible for if you self-host.

Self-hosted OnCall is not a single binary. The engine is a Django application serving the API and the integration webhook endpoints; a PostgreSQL or MySQL database holds schedules, escalation chains, users, and alert-group history; Redis backs both the Celery broker and OnCall's caching; and one or more Celery worker processes, plus a Celery beat scheduler, are what actually execute the timed parts of an escalation chain — the "wait 5 minutes, then try the secondary" step is a Celery task sitting in a queue, not a database poll. The part people find surprising is the front end: there's no separate OnCall web app to bookmark. The UI ships as a plugin installed inside a Grafana instance (self-hosted Grafana or Grafana Cloud), which simply talks to your engine's API — so "OnCall" is something you access through Grafana, not next to it.

Schedules and escalation chains as code

☺ Like you're 10: Instead of clicking a rotation into existence in a browser and hoping nobody clicks it back out of shape, you write it down as a file and let a tool apply it the same way every time.

The whole appeal of OnCall to a platform team is the same appeal Grafana's own dashboard-as-code provisioning has: an escalation chain built by hand in the UI is one accidental click away from silently paging the wrong person, while one defined in Terraform gets reviewed in a pull request and has a diff. The grafana Terraform provider ships grafana_oncall_* resources covering integrations, routes, escalation chains, individual escalation steps, schedules, and shifts.

# --- escalation chain: what happens once an alert group opens ---
resource "grafana_oncall_escalation_chain" "checkout_primary" {
  name = "checkout-primary"
}

resource "grafana_oncall_escalation" "notify_primary_oncall" {
  escalation_chain_id          = grafana_oncall_escalation_chain.checkout_primary.id
  type                         = "notify_on_call_from_schedule"
  notify_on_call_from_schedule = grafana_oncall_schedule.checkout_primary.id
  position                     = 0
}

resource "grafana_oncall_escalation" "wait_before_secondary" {
  escalation_chain_id = grafana_oncall_escalation_chain.checkout_primary.id
  type                 = "wait"
  duration             = 300              # seconds — 5 min to acknowledge, matching the platform's PagerDuty-era default
  position             = 1
}

resource "grafana_oncall_escalation" "notify_secondary_important" {
  escalation_chain_id          = grafana_oncall_escalation_chain.checkout_primary.id
  type                         = "notify_on_call_from_schedule"
  notify_on_call_from_schedule = grafana_oncall_schedule.checkout_secondary.id
  important                    = true      # forces the IMPORTANT personal notification policy — SMS/call, not a Slack ping
  position                     = 2
}

# --- schedule: who "checkout_primary" resolves to at any given moment ---
resource "grafana_oncall_schedule" "checkout_primary" {
  name      = "checkout-primary-rotation"
  type      = "calendar"
  time_zone = "America/New_York"
}

resource "grafana_oncall_on_call_shift" "checkout_weekly_rotation" {
  name           = "checkout-weekly"
  type           = "rolling_users"
  start          = "2026-01-05T09:00:00"
  duration       = 604800                 # seconds — one full week per rotation
  rotation_start = "2026-01-05T09:00:00"
  users          = [data.grafana_oncall_user.alice.id, data.grafana_oncall_user.ben.id]
}

# --- integration: the unique inbound endpoint, wired to the chain above ---
resource "grafana_oncall_integration" "checkout_from_grafana_alerting" {
  name = "checkout-grafana-alerting"
  type = "grafana_alerting"
  default_route {
    escalation_chain_id = grafana_oncall_escalation_chain.checkout_primary.id
  }
}

The other legitimate path to configuration-as-code is the one built into the product itself: a calendar-type schedule can instead be an iCal-type schedule pointed at a URL — a Google Calendar feed a team already maintains, or a feed generated by a separate rotation-planning script — and OnCall simply reads it on a polling interval rather than owning the rotation logic at all. That trades Terraform's review workflow for whatever review workflow already governs the calendar, which is sometimes none — know which one you're actually relying on.

Day-to-day operations

☺ Like you're 10: Most days you don't touch it at all — it only gets interesting when someone's swapping a shift, checking who's actually on call, or scripting something instead of clicking through the UI.

Beyond the Grafana-embedded UI, OnCall exposes a REST API and a Slack app that most teams end up using more than the API directly.

# base URL differs by deployment: your self-hosted engine, or a Cloud org's own subdomain
$ export ONCALL_TOKEN=...                 # created under OnCall → Settings → API Tokens
$ export ONCALL_URL=https://oncall.example.internal/api/v1

# who is actually on call right now, across every schedule
$ curl -s -H "Authorization: $ONCALL_TOKEN" "$ONCALL_URL/schedules" | jq '.results[] | {name, on_call_now}'

# the open incident queue
$ curl -s -H "Authorization: $ONCALL_TOKEN" "$ONCALL_URL/alert_groups?state=firing" | jq '.results[].id'

# acknowledge from a script instead of the Slack button or the UI
$ curl -s -X POST -H "Authorization: $ONCALL_TOKEN" "$ONCALL_URL/alert_groups/<id>/acknowledge"

# list escalation chains and schedules — what you'd otherwise click through to audit
$ curl -s -H "Authorization: $ONCALL_TOKEN" "$ONCALL_URL/escalation_chains" | jq '.results[].name'

# register a one-off override — someone is swapping tomorrow's shift
$ curl -s -X POST -H "Authorization: $ONCALL_TOKEN" "$ONCALL_URL/schedules/<id>/override" \
    -d '{"start":"2026-08-20T09:00:00Z","end":"2026-08-20T17:00:00Z","user":"<user_id>"}'

In practice the interface most engineers actually use is Slack: an alert group posts as a threaded message with Acknowledge and Resolve buttons, slash commands can trigger a manual escalation or ask "who's on call for checkout," and each user's personal notification policy is what decides whether that Slack message is the whole notification or just the first of several — a mobile push and an SMS may already be in flight by the time the Slack message renders, if the triggering step was flagged important.

The tradeoff: your pager now has its own pager

☺ Like you're 10: The smoke detector's calling machine needs its own smoke detector — and building that second one is the part a vendor contract used to quietly include.

This is the decision this page keeps circling back to, because it's the one that actually matters. Choosing Grafana Cloud's managed OnCall is choosing to keep paying — usage-based, not per-seat, but real — in exchange for someone else's SRE team owning exactly the problem this section describes. Choosing to self-host the OSS engine is choosing to take that problem on yourself, and it is a genuine, non-trivial reliability engineering exercise, not a checkbox.

Concretely, self-hosting commits a team to several things a commercial vendor bundles invisibly into their fee:

• Postgres/MySQL, Redis, and Celery all need to be highly available. If Celery workers stall or the queue backs up, "wait 5 minutes, then escalate" steps simply never fire — silently. Nothing in the UI screams about it; an alert group just sits there un-escalated until someone happens to notice the page never came.
• The engine cannot share a failure domain with what it pages for. Running OnCall's database in the same region, cluster, or cloud account as the checkout service it's meant to page about means a region-wide outage can take both down together — the one moment paging matters most.
• SMS and voice calling are not free or automatic. Grafana Cloud's managed OnCall bundles carrier delivery; a self-hosted engine needs its own Twilio (or equivalent) account, phone numbers, and budget wired in before a personal notification policy's "call me" step does anything at all.
• Something outside OnCall has to watch OnCall. The standard mitigation is a dead man's switch — an external heartbeat check, on a separate provider, that pages a human through a channel independent of OnCall entirely the moment it stops hearing from the engine. Skip this and the failure mode is silent by construction: the system that would normally tell you something is broken is the thing that's broken.

Gotchas and failure modes

☺ Like you're 10: Most of the pain isn't the calling machine breaking outright — it's the machine quietly doing something slightly different from what everyone assumed.

Alert-group grouping silently determines your page volume

OnCall deduplicates repeated firings into one alert group using a grouping key derived from the payload — usually a template over labels. Get that template wrong and the two failure modes go in opposite, equally bad directions: too loose, and genuinely different incidents collapse into one alert group and get resolved together by mistake; too strict, and every retrigger of the same underlying problem opens a brand-new alert group, re-running the full escalation chain from step one and paging a fresh round of humans for something already being worked. This is the same alert-fatigue failure mode covered generally in monitoring & observability, just implemented as a specific Jinja2 template worth testing before it ships.

Default versus important notification policies

Covered above as a warning, but worth repeating as a gotcha in its own right: a user who never configures their important personal notification policy — because the UI doesn't force it — falls back to whatever OnCall's default for that variant is, which may not match what the team assumed "critical page" means for that person. Audit this per person, not per team, before trusting a chain's important flag to actually wake someone up.

Plugin/engine version skew

Because the UI is a Grafana plugin talking to a separately versioned engine, upgrading one without the other is a real failure mode unique to the self-hosted, decoupled deployment — a newer plugin can render UI for API fields an older engine doesn't yet expose, or vice versa. Pin and upgrade both together, and read the engine's release notes for breaking API changes the way you'd read a Helm chart's upgrade notes before bumping its CRDs.

Timezone traps in rotations

A schedule's time_zone governs when a calendar-type rotation's shift boundaries fall, but shift durations are specified in raw seconds — a "one week" rotation is exactly 604800 seconds regardless of a daylight-saving transition inside it, which can shift a handoff by an hour twice a year in regions that observe it. Sanity-check a new rotation's actual handoff times in the UI after defining it in code, not just its intent.

Alternatives and when to choose it

☺ Like you're 10: Every option in this category makes the same basic promise — call the right person until someone answers — and differs mainly in who's running the calling machine and who's billing you for it.

The pattern most cost-conscious SRE teams converge on mirrors the one Grafana's own alternatives section describes for dashboards: open tooling for the layers you're comfortable operating, a commercial vendor for the layer where an outage in the tool itself is least tolerable. Paging is frequently the layer teams decide is worth paying to outsource precisely because of the dead-man's-switch problem above — it's one thing to accept a stale dashboard during an incident, and another to accept a pager that might not ring.