Grafana

Grafana is the platform’s window: one web application that queries every telemetry store you own — Prometheus for metrics, Loki for logs, Tempo or Jaeger for traces, even a SQL database for business numbers — and renders them as dashboards, ad-hoc explorations and alerts that a human can actually read. It solves the problem every platform hits at about its fifth service: the data to answer “is this thing healthy?” already exists, but it is scattered across four query languages and four unrelated UIs, so nobody looks at it, and the outage is found by a customer instead.

What Grafana is and the problem it solves

☺ Like you’re 10: It’s a picture-drawing window onto other people’s data — it stores nothing itself, it just asks and draws.

Grafana is an open-source (AGPLv3 since Grafana 8) visualisation and alerting server from Grafana Labs. You point it at one or more data sources, and it gives you three things on top of them: dashboards (saved, shareable, parameterised views), Explore (a scratchpad for one-off queries during an incident), and alerting (rules that evaluate a query on a schedule and notify someone when it goes bad).

The problem before a single pane

Without it, every telemetry backend brings its own console. Prometheus ships a query box that is fine for one expression and hopeless for a service overview; Loki and Tempo ship no UI at all; Jaeger has a decent trace view that knows nothing about metrics. So an on-call engineer at 03:12 holds four tabs, four query dialects and one mental join in their head, under stress, while the error budget drains. Grafana collapses that to one tab where a click on a latency spike lands you in the exact log lines, and a click on a log line lands you in the exact trace.

What Grafana is not

This is the most important sentence on the page: Grafana does not store your telemetry. It has a small relational database (SQLite by default, or MySQL/PostgreSQL in production) but that holds only Grafana’s own state — dashboards, users, folders, alert rules, annotations. Your metrics still live in Prometheus. Your logs still live in Loki. If Grafana falls over, you have lost your window, not your data; if Prometheus falls over, the window shows nothing because there is nothing to show. Teams confuse these two failures constantly, and the fix is different every time.

The wider family, and who owns what

Grafana Labs also builds Loki (logs), Tempo (traces), Mimir (long-term Prometheus storage) and Alloy (a telemetry collector distribution built on OpenTelemetry). They are siblings designed to be queried from the same window, and Loki deliberately borrows Prometheus’s label model so the same {namespace="checkout"} selector works in both heads. Worth knowing for the exam: Grafana, Loki and Tempo are Grafana Labs projects, not CNCF ones — Prometheus, Jaeger and OpenTelemetry are the CNCF members in this neighbourhood. See the CNCF landscape for why that distinction keeps coming up.

Where it fits in a platform

☺ Like you’re 10: It sits at the very end of the pipeline, where data finally turns into something a person looks at.

Every platform diagram has a telemetry column: collect → store → query → see. Grafana owns the last box, and only the last box. It is part of the platform’s observability plane, a shared service the platform team runs once so that fifty product teams don’t each stand up their own. Grafana is also, quietly, part of the developer experience surface: for most application engineers it is the only piece of the observability stack they ever open, which makes its dashboards the platform’s public face.

Its neighbours

Prometheus stores metrics, Loki stores logs, Tempo and Jaeger store traces, and OpenTelemetry produces and ships all three into them — Grafana reads every one. Alertmanager, a Prometheus component, routes, groups, deduplicates and silences alerts; Grafana can either hand its alerts to your Alertmanager or route them itself with its own embedded one. OpenCost exposes cost as Prometheus metrics, so your FinOps dashboard is just another Grafana dashboard. The full picture lives in the Observability & Monitoring lesson; the projects around it are catalogued in The Tool Landscape.

CNPE domain relevance

Grafana sits squarely in the exam’s Observability & Operations domain (20% of the total) — visualising platform and workload health, and understanding how alerting connects a signal to a human. It also brushes Platform APIs & Self-Service (25%: a labelled dashboard ConfigMap is a self-service API), Developer Experience (a dashboard is a product surface), and Reliability & Incident Response (SLO burn-rate panels and the alert that pages you). Expect it as a component you must install, provision, reach and read — not as something you must build from scratch.

How it works — architecture and components

☺ Like you’re 10: One web server, one little notebook of its own, and a list of phone numbers for everyone else’s data.

Architecturally Grafana is refreshingly boring, which is a compliment. It is a single Go binary serving an HTTP API and a React front end on port 3000, plus a database for its own state, plus a set of data source definitions.

The query path

When a panel refreshes, the browser asks the Grafana server, and the Grafana server asks the data source. That indirection matters. Data sources have an access mode: proxy (the default and effectively the only sane choice) means the server makes the call, so the backend never needs to be reachable from the user’s laptop and credentials stay server-side; the legacy direct mode had the browser call the backend itself and is deprecated. In-cluster, this is why your data source URL is a Kubernetes Service DNS name like http://prometheus-operated.monitoring.svc:9090 — the browser could never resolve that, but the Grafana pod can.

Provisioning — configuration as code

You can click everything into existence in the UI, and the state lands in Grafana’s database. Do not do that on a platform. Grafana reads YAML files from /etc/grafana/provisioning/ at start-up (and, for dashboards, on a polling interval). The subdirectories you will meet are datasources/, dashboards/, alerting/, plugins/ and access-control/. Anything defined there is owned by the files, so the whole observability surface becomes a reviewable directory in Git and rebuilds identically on a fresh cluster. This is the same argument as GitOps, applied to dashboards.

The sidecar pattern in kube-prometheus-stack

On Kubernetes almost nobody mounts those files by hand. The kube-prometheus-stack chart (which bundles Grafana) runs a small sidecar container next to Grafana that watches the cluster for ConfigMaps and Secrets carrying a particular label — by convention grafana_dashboard: "1" for dashboards and grafana_datasource: "1" for data sources — and writes their contents into the provisioning directory. The effect is a genuine platform API: a product team ships a labelled ConfigMap in their own namespace and their dashboard appears, with no ticket and no access to Grafana’s config. That is self-service in about twelve lines of YAML.

The resources you will actually write

☺ Like you’re 10: Three files: who to ask, where the pictures live, and when to wake somebody up.

Grafana introduces no CRDs of its own in the core project — its “resources” are provisioning YAML files and dashboard JSON, usually wrapped in ConfigMaps. (The separate Grafana Operator project does add Grafana, GrafanaDashboard and GrafanaDatasource custom resources if you want the Kubernetes-native flavour, but the files below are what the exam and the common chart use.)

Data sources, provisioned

This is the single most valuable file to be able to write from memory. Note apiVersion: 1 — that is Grafana’s provisioning file version, nothing to do with Kubernetes — and the split between jsonData (visible config) and secureJsonData (encrypted at rest, write-only).

apiVersion: 1                      # Grafana provisioning schema, NOT a k8s apiVersion
datasources:
  - name: Prometheus
    uid: prom                      # pin the uid — dashboards reference it by uid
    type: prometheus
    access: proxy                  # the Grafana server queries, not the browser
    url: http://prometheus-operated.monitoring.svc:9090
    isDefault: true
    jsonData:
      timeInterval: 30s            # match your scrape interval, or $__rate_interval lies
      httpMethod: POST
      exemplarTraceIdDestinations: # click an exemplar dot → jump to the trace
        - name: trace_id
          datasourceUid: tempo

  - name: Loki
    uid: loki
    type: loki
    access: proxy
    url: http://loki-gateway.monitoring.svc
    jsonData:
      derivedFields:               # find a trace id in a log line and make it a link
        - name: TraceID
          matcherRegex: 'trace_id=(\w+)'
          url: '${__value.raw}'
          datasourceUid: tempo

  - name: Tempo
    uid: tempo
    type: tempo
    access: proxy
    url: http://tempo-query-frontend.monitoring.svc:3100
    jsonData:
      tracesToLogsV2:              # from a span → the logs of that pod, same time window
        datasourceUid: loki
        spanStartTimeShift: '-2m'
        spanEndTimeShift: '2m'
        tags: [{ key: 'k8s.pod.name', value: 'pod' }]
      serviceMap:
        datasourceUid: prom        # service graph drawn from metrics

Dashboards as files, and the ConfigMap that ships one

A dashboard provider tells Grafana where to find dashboard JSON on disk; the ConfigMap is how that JSON gets there on Kubernetes.

# /etc/grafana/provisioning/dashboards/platform.yaml
apiVersion: 1
providers:
  - name: platform
    orgId: 1
    folder: Platform               # the folder shown in the UI
    type: file
    disableDeletion: true          # UI cannot delete file-backed dashboards
    allowUiUpdates: false          # UI edits cannot be saved — files win. Do this.
    updateIntervalSeconds: 30      # re-read the directory every 30s
    options:
      path: /var/lib/grafana/dashboards/platform
      foldersFromFilesStructure: true
---
# ↑ that file is read by Grafana itself. ↓ THIS is a separate Kubernetes manifest you kubectl apply.
# The sidecar in kube-prometheus-stack watches for this label cluster-wide
apiVersion: v1
kind: ConfigMap
metadata:
  name: checkout-golden-signals
  namespace: checkout             # team's own namespace — no Grafana access needed
  labels:
    grafana_dashboard: "1"        # the label the sidecar selects on
  annotations:
    grafana_folder: "Teams"       # only honoured if the chart sets sidecar.dashboards.folderAnnotation
data:
  checkout.json: |
    { "title": "Checkout · Golden Signals", "uid": "checkout-gs", ... }

Inside the dashboard JSON

You will rarely author JSON by hand end to end — build it in the UI, then export and commit it. But you must be able to read it in a review, because that is where the interesting mistakes hide: a hard-coded datasource uid, a variable with no All option, a threshold nobody agreed to.

{
  "title": "Checkout · Golden Signals",
  "uid": "checkout-gs",
  "schemaVersion": 39,
  "time": { "from": "now-6h", "to": "now" },
  "templating": { "list": [
    { "name": "ds", "type": "datasource", "query": "prometheus" },
    { "name": "namespace", "type": "query", "datasource": { "uid": "${ds}" },
      "query": "label_values(kube_pod_info, namespace)",
      "refresh": 2, "includeAll": true, "multi": true },
    { "name": "pod", "type": "query", "datasource": { "uid": "${ds}" }, "refresh": 2,
      "query": "label_values(kube_pod_info{namespace=~\"$namespace\"}, pod)" }
  ]},
  "panels": [
    { "type": "timeseries", "title": "Error rate (5xx)",
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
      "datasource": { "uid": "${ds}" },
      "targets": [ { "refId": "A", "legendFormat": "{{pod}}",
        "expr": "sum by (pod) (rate(http_requests_total{namespace=~\"$namespace\",code=~\"5..\"}[$__rate_interval])) / sum by (pod) (rate(http_requests_total{namespace=~\"$namespace\"}[$__rate_interval]))" } ],
      "fieldConfig": { "defaults": {
        "unit": "percentunit",
        "thresholds": { "mode": "absolute", "steps": [
          { "color": "green", "value": null },
          { "color": "red",   "value": 0.01 } ] } } },
      "transformations": [ { "id": "organize", "options": {} } ] },
    { "type": "logs", "title": "Errors for $pod",
      "datasource": { "uid": "loki" },
      "targets": [ { "refId": "A",
        "expr": "{namespace=~\"$namespace\", pod=~\"$pod\"} |= \"error\"" } ] }
  ],
  "annotations": { "list": [
    { "name": "Deploys", "enable": true, "datasource": { "uid": "${ds}" },
      "expr": "changes(kube_deployment_status_observed_generation{namespace=~\"$namespace\"}[5m]) > 0",
      "iconColor": "purple" } ] }
}

Three things to notice. $__rate_interval is Grafana’s built-in variable that picks a rate window guaranteed to be at least four scrape intervals wide — use it in every rate() and your graphs stop going mysteriously empty when someone zooms in. ${ds} makes the whole dashboard portable across clusters instead of hard-wiring one uid. And the annotation query paints a vertical line on every panel whenever a Deployment rolls, which answers “did this start when we shipped?” before anyone has to ask it.

Alert rules, contact points and notification policies

Unified alerting became the default in Grafana 9, and the old legacy alerting engine has since been removed entirely — on any currently supported release, unified alerting is what you get. An alert rule is a set of queries and expressions ending in a boolean condition; a contact point is somewhere to send (Slack, PagerDuty, email, webhook); a notification policy is a routing tree that matches on labels and decides which contact point gets it, with grouping and timing — deliberately the same shape as Alertmanager’s route tree.

# /etc/grafana/provisioning/alerting/checkout.yaml
# groups, contactPoints and policies all live in ONE YAML document —
# Grafana parses each provisioning file as a single document, so no `---` here.
apiVersion: 1
groups:
  - orgId: 1
    name: checkout-slo
    folder: Platform
    interval: 1m                       # how often the group is evaluated
    rules:
      - uid: checkout-burn-fast
        title: Checkout fast burn (14.4x over 1h)
        condition: C                   # the refId that must be true
        for: 5m                        # must stay firing this long → Alerting
        data:
          - refId: A
            datasourceUid: prom
            relativeTimeRange: { from: 3600, to: 0 }
            model:
              refId: A
              expr: |
                sum(rate(http_requests_total{job="checkout",code=~"5.."}[1h]))
                  / sum(rate(http_requests_total{job="checkout"}[1h]))
          - refId: C
            datasourceUid: __expr__    # the built-in expression engine
            model: { refId: C, type: threshold, expression: A,
                     conditions: [ { evaluator: { type: gt, params: [0.0144] } } ] }
        noDataState: NoData            # NoData | Alerting | OK | KeepLast
        execErrState: Alerting         # OK | Alerting | Error | KeepLast
        labels: { severity: page, team: checkout }
        annotations:
          summary: Checkout 1h error ratio exceeds 14.4x the 99.9% SLO budget burn rate
          runbook_url: https://runbooks.internal/checkout/burn
contactPoints:
  - orgId: 1
    name: checkout-oncall
    receivers:
      - uid: cp-pd
        type: pagerduty
        settings:
          integrationKey: ${PD_KEY}    # Grafana expands env vars here — never a literal
policies:
  - orgId: 1
    receiver: platform-default
    group_by: [alertname, namespace]
    routes:
      - receiver: checkout-oncall
        object_matchers: [['team', '=', 'checkout'], ['severity', '=', 'page']]
        group_wait: 30s
        repeat_interval: 4h

Day-to-day commands

☺ Like you’re 10: Open a tunnel to port 3000, fetch the password out of a Secret, then mostly poke it with curl.

Grafana has no rich kubectl-style CLI for daily work. In a cluster you interact with it three ways: kubectl to reach and inspect the pod, curl against its HTTP API, and grafana-cli inside the container for plugins and password recovery.

Reaching it, and the credential question

# Find it — the chart names it <release>-grafana, service port 80 → container 3000
kubectl -n monitoring get svc,pod -l app.kubernetes.io/name=grafana

# Port-forward: local 3000 → the Service's port 80
kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80

# The admin password lives in a Secret, NOT in your notes.
kubectl -n monitoring get secret kube-prometheus-stack-grafana \
  -o jsonpath='{.data.admin-user}' | base64 -d; echo
kubectl -n monitoring get secret kube-prometheus-stack-grafana \
  -o jsonpath='{.data.admin-password}' | base64 -d; echo
# Vanilla Grafana defaults to admin/admin and forces a change at first login.
# kube-prometheus-stack defaults to admin/prom-operator — change it.

# Locked out? Reset from inside the container.
# --homepath matters: grafana-cli needs to find the config and DB it is editing.
kubectl -n monitoring exec -it deploy/kube-prometheus-stack-grafana -c grafana -- \
  grafana-cli --homepath /usr/share/grafana admin reset-admin-password 'a-better-password'

# Is it healthy, and did provisioning work?
kubectl -n monitoring logs deploy/kube-prometheus-stack-grafana -c grafana | grep -i provision
kubectl -n monitoring logs deploy/kube-prometheus-stack-grafana -c grafana-sc-dashboard

The HTTP API

Everything the UI does, the API does — which is what CI pipelines and health checks use. Create a service account token in the UI (API keys are deprecated) and export it as $TOKEN.

# Liveness — this is the one to put in a smoke test
curl -s localhost:3000/api/health
# {"commit":"...","database":"ok","version":"<whatever you run>"}

# What data sources does it actually think it has?
curl -s -H "Authorization: Bearer $TOKEN" localhost:3000/api/datasources | jq '.[].name'

# Does a data source actually answer? (the "is it Grafana or is it Prometheus" test)
curl -s -H "Authorization: Bearer $TOKEN" localhost:3000/api/datasources/uid/prom/health

# Search, export and import dashboards — the CI round-trip
curl -s -H "Authorization: Bearer $TOKEN" 'localhost:3000/api/search?query=Checkout'
curl -s -H "Authorization: Bearer $TOKEN" localhost:3000/api/dashboards/uid/checkout-gs \
  | jq '.dashboard' > dashboards/checkout.json
curl -s -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"dashboard": '"$(cat dashboards/checkout.json)"', "overwrite": true}' \
  localhost:3000/api/dashboards/db

# Alert rules and current alert state
curl -s -H "Authorization: Bearer $TOKEN" localhost:3000/api/v1/provisioning/alert-rules
curl -s -H "Authorization: Bearer $TOKEN" localhost:3000/api/alertmanager/grafana/api/v2/alerts

# Mark a deploy on every dashboard — worth wiring into your pipeline
curl -s -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"text":"deploy checkout v1.4.3","tags":["deploy","checkout"]}' \
  localhost:3000/api/annotations

Explore mode — the incident tool

Explore is a query scratchpad with no dashboard to maintain and no panel chrome — you type, you look, you move on — and it is where you will actually spend an outage. Its superpower is the split view: PromQL on the left, LogQL on the right, one shared time range, so narrowing the metric narrows the logs. With the data sources wired as above, you go metric → exemplar dot → trace → span → “show logs for this span,” without typing a single label selector by hand. Practise that chain before you need it; the pattern is drilled in Triage: Workloads and the troubleshooting playbook.

Gotchas and failure modes

☺ Like you’re 10: Most Grafana pain is one of four things: someone edited in the UI, a variable wasn’t what you thought, the time window fooled you, or the data was never there.

Dashboards edited in the UI, and the drift that follows

This is the number one Grafana failure and it is entirely self-inflicted. Someone improves a panel in the browser, hits Save, and now the live dashboard disagrees with the file in Git. The next chart upgrade — or the next pod restart with allowUiUpdates: false — silently reverts it, and a week of tuning evaporates. Worse, nobody knows which version was “right.” The fix is a policy, not a feature: set allowUiUpdates: false and disableDeletion: true on every provisioned provider, so the UI becomes a design surface — build it, export the JSON, open a pull request. Grafana has since grown a first-party Git Sync feature (introduced in Grafana 12, initially opt-in) aimed at exactly this workflow — check its maturity on the release you run before betting on it, and remember that the discipline matters more than the mechanism.

Variables and scoping

Variables are where dashboards quietly lie. A few sharp edges: a multi or includeAll variable interpolates as a regex alternation, so your PromQL matcher must be =~"$namespace", not ="$namespace" — get that wrong and the panel is empty for everything except a single selection. Chained variables ($pod depending on $namespace) only refresh if refresh is set to on-time-range-change or on-dashboard-load; leave it at never and you will be filtering by pods that were deleted in March. label_values() only returns labels that exist in the selected time range, so a scoped-down window makes options disappear. And repeated panels or rows (repeat by variable) multiply queries by the option count — a 20-namespace “All” can fire 200 queries at Prometheus in one refresh and take the whole dashboard down with it.

Time ranges, intervals and the empty graph

Grafana computes $__interval from the time range divided by the panel’s pixel width, so the same query returns different granularity at different zoom levels. Hard-code rate(x[1m]) against a 60s scrape interval and zooming out gives you a graph of nothing, because each window holds at most one sample and rate() needs two. Always use $__rate_interval, and set timeInterval on the data source to your real scrape interval so Grafana can compute it correctly. Two related traps: dashboards default to browser timezone while your logs are UTC, which makes correlation feel haunted; and a dashboard saved with an absolute time range shows everyone a frozen week-old picture, forever.

“The dashboard is red but the service is fine”

Before believing a panel, ask which of four layers broke. (1) Grafana — check /api/health and the pod logs. (2) The data source — hit its health endpoint; an expired token or a renamed Service DNS name shows up as an empty panel, not an error banner. (3) The query — paste it into Explore against the raw store; if it fails there, Grafana is innocent. (4) The data — the metric may genuinely be absent because the ServiceMonitor doesn’t match, in which case the graph is telling the truth. Only the fourth is an application problem. Working that ladder in order, out loud, is most of what separates a calm incident from a noisy one — see Reliability & Incidents.

Alternatives and when to choose it

☺ Like you’re 10: Other windows exist — some come welded to one company’s data, some you have to run yourself.

The real choice is not “Grafana or a competitor” so much as “open, multi-backend visualisation that we run, versus a vendor suite where storage and UI arrive welded together.” Both are defensible; they price and lock in very differently.

A practical rule

Choose Grafana when you already own more than one telemetry backend, or expect to — that is the problem no vendor console solves for you. Choose a SaaS suite when your team is small enough that running observability costs more engineer-hours than the licence, and you can live with the exit cost. Note the sensible middle: many large estates keep vendor agents for storage and still put Grafana on top, so dashboards and alert definitions stay portable text in their own repository. The Tool Landscape places these against the rest of the stack; FinOps is where the SaaS bill eventually shows up.

Exam relevance and going further

☺ Like you’re 10: Grafana is on the exam list, but its own website is locked on exam day — so learn the file shapes, not the docs URL.

Grafana is the visualisation layer you should expect behind anything in the Observability & Operations domain (20% of the exam). Realistically it shows up as a component you must reach, inspect and reason about rather than one you build from nothing: find the Grafana pod and Service, port-forward to it, retrieve credentials from a Secret, confirm a data source is configured and healthy, add or fix a provisioned data source or dashboard ConfigMap, or explain how an alert gets from a query to a human. Read the Observability & Monitoring lesson alongside this page — the concepts there (the three signals, RED and USE, SLOs and burn rate) are what the panels are for, and the exam tests understanding more than clicking.

The documentation allowlist — read this twice

⚖ CNPA vs CNPE — That allowlist is a CNPE-only mechanic: CNPE is hands-on and permits those narrow lookups mid-task. CNPA is stricter, not looser — a fully closed-book multiple-choice exam with zero external references of any kind, not even kubernetes.io. Even so, the Grafana concepts on this page — the datasource/dashboard provisioning shapes, the alert-rule anatomy, the four-layer debugging ladder — are exactly the kind of concept-level knowledge CNPA's closed-book recall draws on.

What to be able to do without notes

Say in one sentence that Grafana visualises and alerts but stores no telemetry of its own. Write a datasources.yaml from an empty file: apiVersion: 1, a datasources list, and per entry name, uid, type, access: proxy, url, isDefault. Know that Grafana listens on 3000, that the chart’s Service usually maps port 80 to it, and the exact kubectl port-forward and get secret … | base64 -d incantations. Explain the sidecar/labelled-ConfigMap dashboard pattern and why it is a self-service API. Distinguish Grafana-managed alert rules from data-source-managed ones, and say where Alertmanager fits. Name the parts of a rule: query, expression, condition, for, labels, annotations, noDataState. Explain why $__rate_interval exists and what breaks without it. And be able to walk the four-layer ladder — Grafana, data source, query, data — when a panel is empty. Any unfamiliar term is in the glossary; the wider stack sits in the tool landscape.

Official resources for after the exam

When you are not sitting the exam, the canonical sources are the Grafana documentation at grafana.com/docs/grafana/latest — the Provisioning, Variables and Alerting sections are the three that repay reading end to end — the dashboard gallery at grafana.com/grafana/dashboards (import by ID, then treat it as a starting point, not a deliverable), the alerting provisioning reference at grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources, the HTTP API reference at grafana.com/docs/grafana/latest/developers/http_api, the source at github.com/grafana/grafana, and the sibling projects at grafana.com/oss/loki and grafana.com/oss/tempo. Pair this page with Prometheus for the query language behind most panels, Loki and OpenTelemetry for the other two signals, and Platform Best Practices for the dashboard conventions worth enforcing before you have four hundred of them.