Grafana

Grafana is the layer an SRE actually looks at during an incident, but it is not where the incident's evidence lives. It queries Prometheus for metrics, Loki for logs, Tempo or Jaeger for traces, and a dozen other backends besides, and turns whatever comes back into dashboards, ad-hoc queries, and — since it grew its own unified alerting engine — pages. It never replaces the stores it queries; deleting Grafana loses you a window, not the room behind it. This page covers what that split actually buys you, how to provision dashboards and alert rules as version-controlled code instead of clicking them into existence, how its alerting engine relates to Prometheus's own Alertmanager, and where Grafana OnCall and Grafana Cloud extend the same open-core project into paging and a fully hosted stack.

What Grafana is, and what it deliberately is not

☺ Like you're 10: It draws pictures of other people's data on request — it keeps a tiny notebook of its own settings, but none of your actual metrics or logs.

Grafana is an open-source visualization and alerting server. You point it at one or more data sources — each a plugin that knows how to translate a query into that backend's dialect — and it gives you three surfaces on top of them: dashboards (saved, parameterized, shareable panels), Explore (a query scratchpad for the moment you're mid-incident and don't want to build a panel first), and alerting (rules that evaluate a query on a schedule and notify someone when the result crosses a threshold). None of that requires Grafana to store your telemetry, and it doesn't: its own database — SQLite by default, MySQL or PostgreSQL in any deployment that matters — holds only dashboards, users, folders, API keys, and alert rule definitions. Prometheus still owns your metrics; Loki still owns your logs. Delete the Grafana pod and you've lost a UI, not history — restore it from provisioning files, and every dashboard comes back exactly as it was, because the definitions live in Git, not in Grafana's memory.

The other half of that same sentence matters just as much: Grafana has no built-in notion of an SLO, a compliance window, or a burn rate. It will plot whatever PromQL expression you hand it exactly as accurately as Prometheus computed it, and it will alert on that expression crossing a number — but the arithmetic behind multi-window, multi-burn-rate alerting has to be computed somewhere else first, by hand-written Prometheus recording rules or by a generator like Sloth. Purpose-built SLO tools such as Nobl9 hold the declarative SLO target itself; Grafana just draws whatever number that produces. Confusing the two — expecting Grafana to "know" a target is 99.9% because a panel happens to be plotting an error ratio — is a fast way to ship a dashboard that quietly lies about how much budget is left.

Where it sits: Prometheus, Loki, and the rest of the stack it queries

☺ Like you're 10: One screen, many sensors — metrics from one place, logs from another, traces from a third, all drawn on the same wall so nobody has to remember four different control panels.

An on-call engineer mid-incident does not have the patience for four separate query languages in four separate browser tabs. Grafana's whole value proposition is collapsing that into one: click a latency spike on a metrics panel and land in the exact log lines from the same window, click a log line with a trace ID in it and land in the exact span. The table below is the mapping every SRE ends up memorizing.

Grafana Labs also ships Loki, Tempo, Mimir, and an OpenTelemetry-based collector distribution called Alloy — siblings built to be queried from the same window, which is why Loki deliberately reuses Prometheus's label model: the same {namespace="checkout"} selector filters both. None of that changes the underlying architectural fact: Grafana is a read path. It fans a query out to whichever data source owns the answer, waits, and draws what comes back.

Dashboard-as-code: provisioning instead of clicking

☺ Like you're 10: Instead of dragging panels around in the browser and hoping you remember what you did, you write the dashboard down as a file, and Grafana just reads the file every time it starts.

Everything you can build with the mouse in Grafana's UI, you can also describe as a file and hand to Grafana at startup — and on a real platform you should, for the same reason toil & automation argues for automating any manual step done more than once: a dashboard built by hand disappears the moment someone rebuilds the box, and a dashboard built from a file in Git survives every rebuild identically, gets reviewed in a pull request, and has a diff when someone changes an alert threshold. Grafana calls this provisioning: it reads YAML from a configured directory (conventionally /etc/grafana/provisioning/) at startup and on a polling interval, split into datasources/, dashboards/, and alerting/ subdirectories.

# provisioning/datasources/prometheus.yaml
apiVersion: 1                        # Grafana's own provisioning schema, not a Kubernetes apiVersion
datasources:
  - name: Prometheus
    uid: prom                        # pin this — dashboards and alert rules reference it by uid
    type: prometheus
    access: proxy                    # the Grafana SERVER queries it, never the browser directly
    url: http://prometheus.monitoring.svc:9090
    isDefault: true
    jsonData:
      timeInterval: 30s              # match your real scrape interval or $__rate_interval lies to you
      httpMethod: POST

  - name: Loki
    uid: loki
    type: loki
    access: proxy
    url: http://loki-gateway.monitoring.svc
    jsonData:
      derivedFields:                 # spot a trace id in a log line, turn it into a jump link
        - name: TraceID
          matcherRegex: 'trace_id=(\w+)'
          url: '${__value.raw}'
          datasourceUid: tempo

# provisioning/dashboards/platform.yaml — tells Grafana WHERE to find dashboard JSON on disk
apiVersion: 1
providers:
  - name: reliability
    orgId: 1
    folder: Reliability
    type: file
    disableDeletion: true           # the UI cannot delete a file-backed dashboard
    allowUiUpdates: false           # UI edits don't persist — the file wins, always
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards/reliability
      foldersFromFilesStructure: true

The dashboard JSON itself is verbose enough that hand-authoring the whole thing is rare — the normal workflow is build it in the UI once, export the JSON, and commit that as the source of truth from then on. Two knobs in the provider above are what make that commitment stick: allowUiUpdates: false turns the UI into a design surface rather than a save target, so a well-meaning click-fix in production evaporates on the next restart instead of silently diverging from Git. Treat any dashboard that matters — an SLO burn-rate view, an on-call landing page — this way from day one; the tooling to keep dashboard JSON tidy across many services (grafonnet/Jsonnet templates, or the Grafana Terraform provider's grafana_dashboard resource) is worth adopting once you're maintaining more than a handful by hand.

The unified alerting engine

☺ Like you're 10: Grafana can ring its own alarm bell now instead of only drawing a picture — but you still have to tell it exactly which number, from which sensor, counts as trouble.

Since Grafana 9, unified alerting is the only alerting engine Grafana ships — the older, dashboard-attached "legacy alerting" it replaced has since been removed entirely. An alert rule is a small pipeline: one or more queries against a data source, an expression that reduces them to a single number, and a threshold condition that number either satisfies or doesn't. Grafana evaluates every rule on its own schedule, independently of whether any dashboard happens to be open — alerting was decoupled from dashboards specifically so a rule can exist without a panel behind it.

Grafana actually shows you two kinds of rule side by side, and conflating them is the single most common alerting mistake teams make with this tool:

• Grafana-managed rules — stored in Grafana's own database, evaluated by Grafana itself, and routed through Grafana's embedded Alertmanager. They can query any data source, including ones Prometheus's own rule engine can't reach — Loki, a SQL table, a mix of several in one expression — which is the only real reason to prefer this path.
• Data-source-managed rules — ordinary Prometheus (or Mimir/Loki ruler) alerting rules that Grafana merely displays and lets you edit remotely. They are evaluated by Prometheus itself and routed by your Alertmanager, not Grafana's. If your rule is pure PromQL against one data source, this is usually the right home — it keeps the same rule authoritative in exactly one place.

Extending the checkout API's burn-rate rules from multi-window, multi-burn-rate alerting: the fast-burn tier there is defined as a Prometheus recording rule plus alert, evaluated by Prometheus and routed by Alertmanager — a textbook data-source-managed case, since it's pure PromQL against one backend. A Grafana-managed rule earns its keep when you need to combine sources — for example, only paging on the fast-burn condition if Loki also shows a matching spike in 5xx log lines, guarding against a metrics-pipeline glitch producing a false burn signal on its own.

# provisioning/alerting/checkout-burn-fast.yaml — a Grafana-managed rule combining two data sources
apiVersion: 1
groups:
  - orgId: 1
    name: checkout-slo
    folder: Reliability
    interval: 1m
    rules:
      - uid: checkout-burn-fast-confirmed
        title: "Checkout fast burn (14.4x/1h) confirmed in logs"
        condition: C
        for: 2m                      # short — the burn-rate windows already size the detection time
        data:
          - refId: A                 # the same recording rule Prometheus already computes
            datasourceUid: prom
            relativeTimeRange: { from: 3600, to: 0 }
            model: { refId: A, expr: "sre:checkout_requests:error_ratio1h > (14.4 * 0.001)" }
          - refId: B                 # a Loki query — the reason this is Grafana-managed at all
            datasourceUid: loki
            relativeTimeRange: { from: 300, to: 0 }
            model: { refId: B, expr: 'count_over_time({app="checkout"} |= "level=error" [5m])' }
          - refId: C
            datasourceUid: __expr__
            model: { refId: C, type: math, expression: "$A && $B > 5" }
        noDataState: Alerting        # NoData | Alerting | OK | KeepLast
        execErrState: Alerting       # OK | Alerting | Error | KeepLast
        labels: { severity: page, team: checkout, tier: fast-burn }
        annotations:
          summary: "checkout burning error budget at 14.4x, confirmed by matching Loki error volume"
contactPoints:
  - orgId: 1
    name: checkout-oncall
    receivers:
      - uid: cp-oncall
        type: webhook
        settings:
          url: https://oncall.example.grafana.net/integrations/v1/alertmanager/${ONCALL_TOKEN}/
policies:
  - orgId: 1
    receiver: platform-default
    group_by: [alertname, team]
    routes:
      - receiver: checkout-oncall
        object_matchers: [['team', '=', 'checkout'], ['severity', '=', 'page']]
        group_wait: 30s
        repeat_interval: 4h

A contact point is somewhere to send a firing alert — webhook, Slack, PagerDuty, email — and a notification policy is a routing tree matching on labels, deliberately shaped the same way Alertmanager's own route tree is, right down to group_by, group_wait, and repeat_interval. The webhook contact point above is the standard way Grafana's alerting hands a firing alert to Grafana OnCall, covered next.

Grafana OnCall and Grafana Cloud — the ecosystem's paging and hosted extensions

☺ Like you're 10: The alarm can ring — OnCall is what actually calls a person's phone until someone answers, and Cloud is the version where somebody else runs the whole building for you.

Grafana itself only gets you as far as "notify a contact point." Turning that into a phone call that escalates if nobody answers, a rotation that knows whose week it is, and a timeline of who acknowledged what, is a separate job — the same job PagerDuty, Opsgenie, and VictorOps do, described in full in incident management & on-call. Grafana OnCall is Grafana Labs' own entry in that category: an open-source-first scheduling and escalation tool that plugs natively into unified alerting through the webhook contact point shown above, and can also ingest alerts directly from Prometheus Alertmanager, so it works whether or not Grafana's own engine is the one deciding something is broken. See Grafana OnCall for schedules, escalation chains, and how it compares directly against PagerDuty.

Grafana Cloud is the other extension, orthogonal to OnCall: a fully hosted version of the whole open-source stack — Grafana itself, Mimir for long-term Prometheus-compatible metrics storage, Loki, Tempo, OnCall, plus synthetic monitoring and k6 load testing bundled in — so a team can adopt the same open dashboard and query model without operating any of the storage backends. It ships a genuinely free tier that's useful for evaluating the stack, and paid tiers priced on metrics/logs/traces ingested and users seated; treat any specific number here as a starting point to verify on Grafana Labs' own pricing page before you budget against it; it changes.

Day-to-day operations

☺ Like you're 10: Mostly you poke it with a web address — there's no big command-line tool the way there is for some other pieces of the stack.

Grafana has no rich CLI for daily work the way, say, a version-control tool does. In practice you reach it three ways: whatever process manager runs it (systemd, a container runtime, or Kubernetes), its HTTP API, and grafana-cli inside the box for plugin management and password recovery.

# --- reaching it ---
kubectl -n monitoring get svc,pod -l app.kubernetes.io/name=grafana
kubectl -n monitoring port-forward svc/grafana 3000:80        # Grafana listens on 3000 by default

# credentials live in a Secret (or your provisioned admin_password), never in your notes
kubectl -n monitoring get secret grafana -o jsonpath='{.data.admin-password}' | base64 -d; echo

# locked out? reset it from inside the running instance
grafana-cli --homepath /usr/share/grafana admin reset-admin-password 'a-better-password'

# --- the HTTP API: what CI pipelines and health checks actually use ---
curl -s localhost:3000/api/health                              # put this in a smoke test
curl -s -H "Authorization: Bearer $TOKEN" localhost:3000/api/datasources | jq '.[].name'
curl -s -H "Authorization: Bearer $TOKEN" localhost:3000/api/datasources/uid/prom/health

# round-trip a dashboard through git — export it, edit the file, push it back
curl -s -H "Authorization: Bearer $TOKEN" localhost:3000/api/dashboards/uid/checkout-slo \
  | jq '.dashboard' > dashboards/checkout-slo.json
curl -s -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"dashboard": '"$(cat dashboards/checkout-slo.json)"', "overwrite": true}' \
  localhost:3000/api/dashboards/db

# alert rules and current firing state
curl -s -H "Authorization: Bearer $TOKEN" localhost:3000/api/v1/provisioning/alert-rules
curl -s -H "Authorization: Bearer $TOKEN" localhost:3000/api/alertmanager/grafana/api/v2/alerts

# --- Terraform, the other common way to apply provisioning as code in CI ---
# resource "grafana_dashboard" "checkout_slo" { config_json = file("dashboards/checkout-slo.json") }
# resource "grafana_rule_group"  "checkout_slo" { ... }
terraform plan -out=plan.tfplan && terraform apply plan.tfplan

Gotchas and failure modes

☺ Like you're 10: Most Grafana pain comes from one of three places: someone clicked instead of committing, the time window fooled the query, or a panel is confidently drawing nothing at all.

UI edits versus provisioned files

This is the single most common Grafana failure, and it's entirely self-inflicted: an engineer improves a panel in the browser under pressure, hits save, and the live dashboard now disagrees with what's in Git. The next restart — or the next deploy, if allowUiUpdates: false is set the way it should be — silently reverts the change, and a genuine improvement evaporates because it was never committed. The fix is the policy shown earlier: provisioned dashboards are read-only in the UI, so a fix has to leave as a pull request, not stay as a click.

Time windows, rate intervals, and the confidently-empty panel

Grafana computes its $__interval variable from the visible time range divided by the panel's pixel width, so the same query returns different granularity at different zoom levels. Hard-code rate(x[1m]) against a 60-second scrape interval and zoom out to a week, and the graph goes blank — each window now holds at most one sample, and rate() needs two to compute anything. Use $__rate_interval instead, and make sure the data source's configured timeInterval matches your real scrape interval so Grafana can size it correctly. A quieter version of the same trap: a dashboard saved with an absolute time range shows everyone a frozen picture from whenever it was saved, forever, until someone notices.

Evaluation intervals and alert flap

An alert rule's own evaluation interval — how often Grafana actually re-runs the query — is a separate knob from the burn-rate window sizes and the for: duration inside the rule. Set the group's evaluation interval too coarse relative to a fast-burn tier's 5-minute short window and you lose most of the detection speed that window was bought for in the first place; set it aggressively short against an expensive cross-data-source expression and you can genuinely overload the backend you're querying, which is the next problem.

Grafana as a production dependency in its own right

Three risks are easy to forget because Grafana feels like "just a UI." Statelessness: the default SQLite database sits on local disk — no persistent volume or external Postgres/MySQL, and a restart wipes every hand-made dashboard, user, and silence; provisioned files survive, anything made only in the UI does not. Blast radius: a heavy dashboard on short auto-refresh, viewed by many people at once, can throw enough concurrent load at Prometheus or Loki to become the outage rather than the tool watching it — sensible refresh intervals and recording rules for expensive expressions aren't optional at scale. Alerting single point of failure: if every rule in your organization is Grafana-managed and Grafana itself goes down, nothing pages — which is exactly the argument for keeping the highest-severity, simplest burn-rate tiers as data-source-managed rules evaluated by Prometheus directly, independent of whether Grafana is even up.

Alternatives and when to choose it

☺ Like you're 10: Other screens exist too — some are welded to one company's sensors, some you have to run and patch yourself.

The real decision is rarely "Grafana or a competitor" in isolation — it's whether you want a vendor-neutral window you operate yourself, over storage you also operate, or a vendor suite where sensor and screen arrive as one billed product.

The pattern most mature SRE organizations converge on is exactly the layering this page describes: open storage backends you control the retention and cost of, Grafana or Grafana Cloud as the query and alert surface over them, and a dedicated paging tool — OnCall or a competitor — owning the human escalation path. Swapping any one layer rarely means rewriting the others, which is the entire point of keeping the roles this separate.