VictorOps (Splunk On-Call)

VictorOps is a paging and incident-collaboration tool built around one central idea: an incident should have exactly one running record, not an alert list in one system, a chat thread in another, and a hand-written summary stitched together afterward from both. That record is called the Timeline, and it is still the product's defining feature more than a decade after launch — even though the product itself has since changed owners twice and changed names once. Splunk acquired VictorOps in 2018 and rebranded it Splunk On-Call around 2020, folding it into a much larger observability and security portfolio; Splunk itself was then acquired by Cisco, a deal that closed in early 2024. This page covers what VictorOps/Splunk On-Call actually does — routing keys, escalation policies, the Timeline, the REST and ChatOps interfaces you touch day to day — the gotchas that bite real rotations, how it compares to PagerDuty, Opsgenie, and Grafana OnCall, and exactly where it now sits alongside Splunk's logging and APM products.

What it is, and why it has two names

☺ Like you're 10: One product, built in 2012, bought by a bigger company in 2018 that gave it a new name — and both names still show up in docs and search results, which trips people up more than it should.

VictorOps was founded in 2012 as a standalone on-call and incident-collaboration product, built to compete directly with the then-dominant PagerDuty on a specific bet: that the incident record itself — not just the paging mechanism — was the underserved part of the problem. Splunk acquired VictorOps in 2018, reportedly for around $120 million, and around 2020 rebranded the product Splunk On-Call. The rebrand was cosmetic in the sense that the underlying application, its API, and the Timeline concept didn't change; it was substantive in the sense that the product stopped being sold or supported as an independent thing and became one SKU inside Splunk's much larger commercial portfolio, priced and procured the way the rest of Splunk is. Cisco then acquired Splunk itself in a deal announced in September 2023 and closed in March 2024, for roughly $28 billion — so the tool this page covers has now passed through two acquisitions in six years. Splunk's own marketing and documentation still use "VictorOps" and "Splunk On-Call" close to interchangeably in places — legacy URLs, community integration guides, and even parts of the product's own UI carry the older name — so treat the two as the same product under two labels, and expect that naming inconsistency to be a live source of confusion rather than a historical footnote. Whatever the current packaging is called on Splunk's pricing page by the time you read this, verify it there before you budget or write a runbook around a specific tier name; vendor SKUs move faster than the underlying mechanism does.

The Timeline — one running record instead of several

☺ Like you're 10: Every alert, every "I've got it," every chat message about the incident, and every deploy that happened during it all land on the same scrolling page, in the order they occurred — so nobody has to reconcile four different logs to reconstruct what happened.

The Timeline is a per-incident, reverse-chronological feed that interleaves everything relevant to that incident as it happens: the inbound alert that opened it, who was paged and when, who acknowledged and when, every escalation step that fired, chat messages sent through VictorOps's own collaboration surface or relayed from Slack/Teams, manual annotations an engineer types in mid-incident ("rolled back the 14:02 deploy, watching error rate now"), and the eventual resolution. Competing tools mostly separate these into different views — an alert list here, a chat integration there, a manually written summary afterward — and leave reconstructing the actual sequence of events to whoever writes the postmortem. VictorOps's bet was that the sequence itself is valuable enough to capture as a first-class object while it's happening, not reconstructed from memory and Slack scrollback two days later.

Read the auto-drafted report for what it actually is: a well-organized transcript, not an analysis. It tells you exactly when the alert fired, who was paged, how long acknowledgement took, and what was said along the way — genuinely useful raw material for the timeline section of a postmortem. It does not tell you the root cause, the contributing factors, or the corrective actions, because none of those exist yet at the moment the incident closes. Treat the Timeline export as the evidence you hand to the blameless postmortem process, not as a substitute for writing one — a distinction Foxy would insist on the moment anyone tried to skip the write-up because "it's all in the Timeline already."

Routing keys, escalation policies, and on-call schedules

☺ Like you're 10: A routing key is the address label on an alert, an escalation policy is the phone-tree it triggers, and a schedule is whose turn it actually is this week — three separate settings that all have to agree before a page reaches a human.

Three objects do essentially all the configuration work, and they compose the same way in VictorOps as they do in every competitor in this category, which is worth knowing because the vocabulary transfers directly.

Alerts reach VictorOps two main ways. Dozens of pre-built integrations (Prometheus Alertmanager, Datadog, Nagios, New Relic, AWS CloudWatch, and many more) ship as configuration you paste a routing key into. Anything without a pre-built integration goes through the generic REST endpoint, which is the one worth knowing cold because it's what every homegrown script or unsupported monitoring tool ends up using:

# the REST endpoint URL embeds your org's API key and (optionally) the routing key
POST https://alert.victorops.com/integrations/generic/20131114/alert/<api-key>/<routing-key>
Content-Type: application/json

{
  "message_type": "CRITICAL",
  "entity_id": "checkout-p99-latency",
  "entity_display_name": "Checkout p99 latency",
  "state_message": "p99 2400ms, SLO threshold 500ms, burn rate 14.2x",
  "monitoring_tool": "prometheus-alertmanager"
}

message_type is the field that matters most: CRITICAL and WARNING open or update an incident, INFO annotates the Timeline without paging, RECOVERY auto-resolves the matching entry, and ACKNOWLEDGEMENT marks it acked from the monitoring side rather than a human clicking a button. entity_id is the de-duplication key — VictorOps groups repeated alerts sharing the same entity_id into one Timeline entry instead of opening a new incident on every retrigger, which is the single setting most responsible for whether a flapping check pages once or twenty times. Field names and the endpoint path are stable but not guaranteed forever; verify the current integration guide in Splunk On-Call's own docs before wiring a new source, exactly as you would for any vendor API. Prometheus shops typically skip hand-writing this payload entirely and point Alertmanager's webhook_configs at a small adapter, or at the REST endpoint directly with a template — see multi-window, multi-burn-rate alerting for the Alertmanager routing this ultimately plugs into.

Day to day: acknowledging, ChatOps, and the API

☺ Like you're 10: Most days you tap "acknowledge" on your phone or type a short command in Slack — the full API only comes out when you're scripting something or building a status page integration.

There's no equivalent of a rich local CLI here — VictorOps/Splunk On-Call is a hosted SaaS product, and you reach it through three surfaces: the mobile app (the one that actually rings your phone and keeps escalating until you swipe), a ChatOps integration in Slack or Microsoft Teams, and a REST API for anything scripted. The mobile app and web Timeline are where most acknowledgements happen in practice; the API is where automation and integrations live.

# public API — every request needs both header keys, generated per user or per integration
$ export VO_API_ID="a1b2c3d4-..."
$ export VO_API_KEY="xxxxxxxxxxxxxxxx"

# open an incident manually against an escalation policy (paging test, or a human-noticed problem
# with no monitoring alert behind it yet)
$ curl -X POST https://api.victorops.com/api-public/v1/incidents \
    -H "X-VO-Api-Id: $VO_API_ID" -H "X-VO-Api-Key: $VO_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "summary": "Checkout p99 latency breach",
      "details": "p99 2400ms vs 500ms SLO threshold, burn rate 14.2x",
      "userName": "sre-oncall-bot",
      "targets": [{"type": "EscalationPolicy", "slug": "checkout-primary"}]
    }'

# who's on call right now, for a given team — the question every runbook needs answered fast
$ curl -s -H "X-VO-Api-Id: $VO_API_ID" -H "X-VO-Api-Key: $VO_API_KEY" \
    https://api.victorops.com/api-public/v1/team/checkout/oncall/schedule | jq '.schedules[0].onCallUser'

# acknowledge or resolve programmatically — the same actions ChatOps and the app trigger
$ curl -X PATCH https://api.victorops.com/api-public/v1/incidents/ack \
    -H "X-VO-Api-Id: $VO_API_ID" -H "X-VO-Api-Key: $VO_API_KEY" \
    -H "Content-Type: application/json" -d '{"incidentNames": ["INC-4821"]}'

Endpoint paths and payload shapes above follow the documented public API's general contract; confirm exact field names against Splunk On-Call's current API reference before you script against it, the same caveat as the REST endpoint above. The ChatOps side works the mirror-image way: a Slack app posts new Timeline entries into a channel as they happen and exposes a small set of slash commands — acknowledging, resolving, or reassigning an incident without leaving Slack — with the exact command syntax configured per workspace install, so check your own workspace's help output rather than assume a specific string. The practical pattern most teams converge on: monitoring fires into the REST endpoint, the mobile app pages the primary, ChatOps carries the human coordination the Timeline is quietly recording anyway, and the API is reserved for the handful of things worth automating — status-page updates, ticket creation, or a manual page from a script that noticed something monitoring didn't.

Where it sits in Splunk's product family today

☺ Like you're 10: The alarm, the sensor archive, and the "how fast did the building respond" gauge are now three different products from the same company — buying one doesn't hand you the other two.

The brief in the SRE toolchain overview — "now part of Splunk's observability suite" — undersells how many separate acquisitions actually built that suite. Splunk's flagship product, Splunk Enterprise / Splunk Cloud Platform, is a log-search and indexing engine built around its own query language (SPL) and predates every acquisition below; it's also the foundation of Splunk's security line (Enterprise Security as a SIEM, SOAR from the 2018 Phantom acquisition), which is a genuinely separate go-to-market from the observability side even though it shares the same underlying platform. Splunk Infrastructure Monitoring and Splunk APM both trace back to SignalFx, acquired in 2019 for roughly $1.05 billion, and provide the metrics and distributed-tracing pieces. VictorOps, acquired the year before in 2018, became the incident-response and paging piece, marketed as Splunk On-Call. Splunk IT Service Intelligence (ITSI) is Splunk's own AIOps layer, built in-house rather than acquired, correlating KPIs across services into a service-health score.

Commercially, Infrastructure Monitoring, APM, RUM, and log observability are typically bundled and sold together as Splunk Observability Cloud; Splunk On-Call has at different points been sold as part of that bundle and as a separate line item, and that packaging has already moved more than once since the 2018 acquisition. The architectural point worth taking away is more durable than any specific SKU name: Splunk's logging product indexes what already happened, APM shows you how a request moved through your services, and Splunk On-Call is the layer that turns a detected problem into a paged human — three different jobs, three different data models, sold by the same company but not one integrated tool. A team standardizing on Splunk for logs gets no architectural obligation to also use Splunk On-Call for paging, and plenty of shops run Splunk for logs alongside PagerDuty or Opsgenie for paging with no friction at all. As with every vendor packaging question in this course, verify current bundling and pricing on Splunk's own site — it has changed shape at least three times already (independent VictorOps, Splunk On-Call as a standalone SKU, Splunk On-Call inside Observability Cloud) and the Cisco acquisition makes a fourth reshuffle plausible.

Gotchas and failure modes

☺ Like you're 10: Nearly every real-world complaint about this class of tool traces back to one of three things: a de-dup setting nobody tuned, a schedule that got the clock wrong, or a policy with no safety net at the bottom.

Alert floods from a missing or wrong entity_id

If a monitoring source omits entity_id, or generates a new one on every fire instead of a stable identifier for "this specific check on this specific resource," VictorOps has no basis for grouping and opens a fresh Timeline entry — and a fresh page — on every single alert. A flapping check that fires every sixty seconds for twenty minutes doesn't become one incident with twenty updates; it becomes twenty incidents, each triggering its own escalation policy from the top. This is the same underlying failure mode covered generally in alert design & alert fatigue, and in VictorOps specifically it's almost always traceable to one missing field in an integration's payload template.

Routing key typos fail silently

A routing key is just a string match. Paste the wrong one — a stale key from a decommissioned team, a typo, a key copied from a staging integration into a production one — and the alert doesn't error out or bounce; it either lands on whatever default/unrouted team exists or vanishes into a routing key nobody's escalation policy is attached to. There's no exception thrown anywhere in the pipeline for a human to notice. The only reliable defense is testing every new integration end to end with a synthetic alert before trusting it in production, and periodically auditing which routing keys actually have an escalation policy attached versus which are orphaned.

Rotation timezone and DST edges

Schedules are defined against a timezone, and handoff moments computed from that timezone are exactly where daylight-saving transitions and multi-region teams produce the "who's actually on call right now" confusion that every paging tool in this category is vulnerable to. A follow-the-sun rotation spanning three timezones with a DST transition in only one of them can hand off an hour early or late relative to what everyone assumed, and the first anyone notices is usually a page nobody answers because the person who thinks it's their shift hasn't started it yet by their own clock.

The Timeline is not a compliance archive by itself

Treat the Timeline as operational memory, not a system you can point an auditor or a regulator at directly for long-term retention guarantees — export what matters (the auto-drafted report, the raw event log) into wherever your organization's actual record of incidents lives, on whatever cadence your postmortem process requires. This is the same lesson as trusting a dashboard tool's own storage for anything that must outlive the tool: don't.

Alternatives and when to choose it

☺ Like you're 10: All four tools in this category do the same core job — route an alert to a human and escalate if nobody answers — so the real choice comes down to who else you already buy software from and how much the collaboration layer matters to you.

The category is narrower than most tool comparisons in this course: every serious option routes an alert, runs an escalation policy, and pages a human, so the differentiators are integration depth, existing vendor relationships, and how much the tool tries to own the incident narrative versus just the paging mechanism.

The practical decision rule most teams land on: if you're choosing on a green field with no existing vendor relationship, PagerDuty's integration breadth or Grafana OnCall's zero-incremental-cost fit (if you already run Grafana) usually win the evaluation outright. VictorOps/Splunk On-Call earns its place specifically when a Splunk relationship already exists elsewhere in the organization, or when the Timeline's single-record incident view is a requirement you've already identified rather than a nice-to-have discovered mid-evaluation. Whichever you choose, the underlying practice this whole category supports — a sustainable rotation, a policy with a real floor, and alerts tuned so they page for something actionable — is covered independent of any specific vendor in incident management & on-call and the on-call readiness checklist.