Grafana

Prometheus and the rest of this course's exam ladder end at a number a machine understands; Grafana is where that number becomes a picture a human can act on at 3am. It queries live — never storing a metric, log line, or trace itself — and draws whatever it gets back from Prometheus, Loki, Tempo, or a dozen other backends onto the same panel, the same time axis, the same click. On the PCA blueprint, "Dashboarding basics," "Configuring Alerting rules," and "Understand and Use Alertmanager" sit inside one 18%-weighted domain — Grafana is where all three become real config you write rather than an abstract multiple-choice answer. This page covers the architecture that makes it a read-only layer over everything else, PromQL panels and the visualization types built for them, template variables that turn one dashboard into one-per-service, the genuinely confusing overlap between Grafana's own alerting engine and Prometheus's Alertmanager, and how to keep every dashboard defined as a file a GitOps controller can reconcile instead of state trapped in one server's clicked-together database.

Architecture: a read path, not a storage layer

☺ Like you're 10: One web server, a tiny notebook for its own settings, and a phone book of everyone else's data — it never keeps a copy of what it draws.

Grafana is a single Go binary serving an HTTP API and a web front end on port 3000, backed by a small relational database — SQLite by default, Postgres or MySQL in any real deployment — that holds only Grafana's own state: dashboards, users, folders, alert rules, annotations. Every data source is a live connection queried fresh on every panel refresh, through the server itself rather than the browser — the default access: proxy mode — so credentials for Prometheus, Loki, and Tempo live server-side, never in a laptop the whole team can inspect, and a backend reachable only inside the cluster network still works, because Grafana's own pod sits inside that network too. Delete every dashboard tomorrow and not one metric or trace anywhere else is affected; lose Prometheus and Grafana boots fine, it just has nothing left to draw.

On a Kubestronaut-shaped cluster, Grafana rarely arrives alone. The kube-prometheus-stack Helm chart is the common on-ramp: it deploys Prometheus, its own Prometheus Operator, a dedicated Alertmanager, and Grafana together, wired to each other by default. For managing Grafana's own resources — dashboards, data sources, alert rules — as native Kubernetes objects rather than files on a Deployment's disk, the separate Grafana Operator project adds CRDs (Grafana, GrafanaDashboard, GrafanaDatasource, and in current releases GrafanaAlertRuleGroup) that a controller reconciles the same way Argo CD's Application CRD reconciles a workload.

PromQL panels and the visualization types built for them

☺ Like you're 10: A panel is just one question asked in PromQL plus a decision about how to draw the answer — a line over time, a single big number, or a table.

Underneath the drag-and-drop editor, a dashboard is a JSON document: a list of panels, each naming a data source, one or more queries, and a visualization type. A timeseries panel wants an instant vector per refresh, redrawn as a line or bar over the selected time range — the natural home for a rate(). A stat or gauge panel wants a single number right now, good for "requests per second this instant." A table wants one row per label combination, good for a per-pod restart count nobody wants graphed. And a heatmap is the panel built specifically for a Prometheus histogram's bucketed shape, coloring each latency bucket by how many requests landed there per time slice — read it before reaching for histogram_quantile() alone, because a heatmap shows the whole distribution shifting, not just one percentile's line.

{
  "title": "Mission Log — golden signals",
  "panels": [
    {
      "title": "Error ratio (5m)",
      "type": "timeseries",
      "datasource": { "type": "prometheus", "uid": "prom" },
      "targets": [{
        "refId": "A",
        "expr": "sum(rate(http_requests_total{job=\"mission-log\",code=~\"5..\"}[5m])) / sum(rate(http_requests_total{job=\"mission-log\"}[5m]))",
        "legendFormat": "{{job}} error ratio"
      }],
      "fieldConfig": { "defaults": { "unit": "percentunit" } }
    },
    {
      "title": "p99 latency, le preserved",
      "type": "timeseries",
      "targets": [{
        "refId": "A",
        "expr": "histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job=\"mission-log\"}[5m])))"
      }]
    },
    {
      "title": "Requests right now",
      "type": "stat",
      "targets": [{ "refId": "A", "expr": "sum(rate(http_requests_total{job=\"mission-log\"}[1m]))" }]
    }
  ]
}

What actually earns the phrase "one pane" is a mixed data source panel: two queries in one panel, pointed at two different uids. A latency spike from Prometheus sitting on the same time axis as a jump in log volume from Loki is the fastest way to confirm the two are the same incident before an on-call engineer even opens a second tab. The Prometheus Model covers the query language itself in depth — this page assumes it and focuses on what happens once the query becomes a picture.

Template variables — one dashboard, every environment

☺ Like you're 10: A variable is a fill-in-the-blank dropdown at the top of the dashboard — pick a value and every panel's query updates to match, without anyone editing a query by hand.

A query variable asks a data source what values exist right now — label_values(kube_pod_info, namespace) returns every namespace Prometheus currently sees, refreshed on a schedule you choose. Variables can chain: a pod variable's own query references $namespace, so picking a namespace first narrows which pods show up next, exactly like a cascading form field. Set multi and includeAll together and the dropdown becomes a multi-select with an "All" option — the single most useful setting on a shared dashboard, and the single most common source of a support ticket.

"templating": { "list": [
  {
    "name": "namespace", "type": "query", "multi": true, "includeAll": true,
    "datasource": { "type": "prometheus", "uid": "prom" },
    "query": "label_values(kube_pod_info, namespace)",
    "refresh": 2
  },
  {
    "name": "pod", "type": "query", "multi": true, "includeAll": true,
    "datasource": { "type": "prometheus", "uid": "prom" },
    "query": "label_values(kube_pod_info{namespace=~\"$namespace\"}, pod)",
    "refresh": 2
  }
]}

Alerting: Grafana's own engine vs. Prometheus's Alertmanager

☺ Like you're 10: Two different alarms can both watch the same number — one lives inside Grafana, one lives inside Prometheus — and mixing them without deciding which one is in charge is how a team gets paged twice for one fire, or never paged at all.

The PCA blueprint's own model is the Prometheus-native one: a rule file holding recording rules and alerting rules, evaluated by Prometheus itself on a schedule, with anything that fires forwarded to Alertmanager — a separate process handling grouping, inhibition, silences, and routing to receivers. That's exactly what a kube-prometheus-stack install ships out of the box, Alertmanager included, and it's the model to have cold for "Understand and Use Alertmanager" on exam day.

Grafana adds a second, independent path since its unified alerting engine landed in version 9. A rule shown in Grafana's alert list is one of two very different things. A Grafana-managed rule lives in Grafana's own database, is evaluated by Grafana's own scheduler, and — uniquely — can query and combine multiple data sources at once in a single condition, something no single backend's own rule engine can do. A data-source-managed rule is the opposite: an ordinary Prometheus (or Mimir, or Loki ruler) rule that Grafana only displays and edits remotely, exactly the rule file above, still evaluated and fired by Prometheus's own Alertmanager whether or not Grafana is even running. And Grafana ships its own embedded Alertmanager, forked from Prometheus's, for routing Grafana-managed alerts — administrators can point it at that built-in engine or reconfigure it to hand notifications to an external Alertmanager instead, such as the one kube-prometheus-stack already deployed.

# Prometheus's OWN rule file — evaluated by Prometheus, fired through its Alertmanager
groups:
  - name: mission-log-slos
    rules:
      - record: job:http_request_errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
            / sum by (job) (rate(http_requests_total[5m]))
      - alert: MissionLogHighErrorRate
        expr: job:http_request_errors:ratio_rate5m{job="mission-log"} > 0.02
        for: 10m
        labels: { severity: page }
        annotations:
          summary: "mission-log error ratio above 2% for 10 minutes"
          runbook_url: https://runbooks.internal/mission-log/high-error-rate

A Grafana-managed equivalent, provisioned as a file rather than clicked into the UI, looks different — note field names have shifted across provisioning schema versions, so check the schema your Grafana release ships before copying this verbatim:

# /etc/grafana/provisioning/alerting/mission-log.yaml — GRAFANA-managed, own scheduler
apiVersion: 1
groups:
  - orgId: 1
    name: mission-log-slo
    folder: Mission Control
    interval: 1m
    rules:
      - uid: mission-log-burn-fast
        title: "mission-log fast burn (14.4x over 1h)"
        condition: C
        for: 5m
        data:
          - refId: A
            datasourceUid: prom
            relativeTimeRange: { from: 3600, to: 0 }
            model:
              expr: |
                sum(rate(http_requests_total{job="mission-log",code=~"5.."}[1h]))
                  / sum(rate(http_requests_total{job="mission-log"}[1h]))
          - refId: C
            datasourceUid: __expr__          # the built-in threshold expression engine
            model: { type: threshold, expression: A,
                     conditions: [ { evaluator: { type: gt, params: [0.0144] } } ] }
        labels: { severity: page }
        annotations:
          summary: "mission-log's 1-hour error ratio exceeds its SLO burn-rate budget"

Provisioning dashboards and alerts as code

☺ Like you're 10: A dashboard and an alert rule are both just text files underneath, so they can live in Git next to the manifests this whole ladder already taught you to reconcile.

Two Kubernetes-native paths get a dashboard into a running Grafana without anyone clicking "New Dashboard" in production, and both plug into the exact reconciliation loop covered on the Argo CD and Flux pages. The kube-prometheus-stack chart's built-in sidecar watches for ConfigMaps carrying a specific label and mounts their contents straight into Grafana's provisioning directory:

apiVersion: v1
kind: ConfigMap
metadata:
  name: mission-log-golden-signals
  namespace: monitoring
  labels:
    grafana_dashboard: "1"          # the kube-prometheus-stack sidecar watches for this label
data:
  mission-log-golden-signals.json: |
    { "title": "Mission Log — golden signals", "panels": [ ] }

Commit that ConfigMap to the same repository Argo CD or Flux already syncs, and a new dashboard becomes exactly as reviewable as a Deployment. The Grafana Operator takes the same idea further with a dedicated GrafanaDashboard CRD, which can hold the JSON inline or reference a Git URL directly — worth reaching for once dashboards, data sources, and alert rules all want to be first-class Kubernetes objects rather than files inside a ConfigMap:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: mission-log-golden-signals
  namespace: monitoring
spec:
  instanceSelector:
    matchLabels: { dashboards: mission-control }
  resyncPeriod: 5m
  json: |
    { "title": "Mission Log — golden signals", "panels": [ ] }

Either path needs the same discipline this course's GitOps Philosophy page argues for any reconciled resource: build and iterate in the browser, then export the JSON and open a pull request, rather than treating a browser edit as the final answer. Set allowUiUpdates: false on file-based provisioning, or simply don't grant broad edit access outside the pipeline, so the running server can't quietly diverge from what Git says it should be.

Day-to-day: the CLI barely exists — you live in Explore and the API

☺ Like you're 10: There's almost no command-line tool — you reach Grafana in a browser, poke it with the same HTTP API the UI uses, and do real incident work in a query scratchpad called Explore.

# reach Grafana without an Ingress
$ kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80

# the kube-prometheus-stack admin password lives in a Secret, not a default you can guess
$ kubectl -n monitoring get secret kube-prometheus-stack-grafana -o jsonpath='{.data.admin-password}' | base64 -d

# liveness — the one worth putting in a smoke test
$ curl -s localhost:3000/api/health

# what data sources does it actually see, and are they healthy?
$ curl -s -H "Authorization: Bearer $TOKEN" localhost:3000/api/datasources | jq '.[].name'
$ curl -s -H "Authorization: Bearer $TOKEN" localhost:3000/api/datasources/uid/prom/health

# export a dashboard you built in the UI, ready to commit
$ curl -s -H "Authorization: Bearer $TOKEN" localhost:3000/api/dashboards/uid/mission-log-gs \
    | jq '.dashboard' > dashboards/mission-log.json

# current GRAFANA-managed alert state, without opening a browser
$ curl -s -H "Authorization: Bearer $TOKEN" localhost:3000/api/alertmanager/grafana/api/v2/alerts

# vs. Prometheus's OWN Alertmanager — a completely separate API, on a completely separate pod
$ curl -s http://alertmanager.monitoring.svc:9093/api/v2/alerts

Create a service-account token in the UI for that $TOKEN — API keys are deprecated in its favor. Explore is where an actual incident happens: no dashboard to maintain, a query box per data source, and a shared time range, with a split view that puts a PromQL panel beside a LogQL panel so narrowing the metric narrows the logs beside it in the same motion.

Gotchas and failure modes

☺ Like you're 10: Most Grafana pain is one of three things: a hard-coded time window, a browser edit that never made it to Git, or a picture everyone expected that three separate systems quietly failed to agree to draw.

A hard-coded rate window goes blank the moment someone zooms out

rate(x[1m]) against a 30-second scrape interval needs at least two samples inside that window to return anything. Zoom a dashboard out from an hour to seven days and the same fixed window can land on a single sample, and the panel goes empty with no error to explain why. Grafana's built-in $__rate_interval variable computes a window guaranteed wide enough for the current zoom level and scrape interval — use it in place of a hard-coded duration inside every rate() or increase() call on a dashboard meant to be zoomed.

UI edits and provisioned files disagree, and the file wins eventually

Someone improves a panel in the browser mid-incident and hits Save. The next redeploy, or the next resync of a GrafanaDashboard with resyncPeriod set, silently reverts it — the exact same selfHeal argument the Argo CD page makes about a hand-edited Deployment applies here without modification. The fix is the same discipline, not a feature: build in the browser, export, open a pull request.

Exemplars need three separate systems to agree before a dot shows up

An exemplar is a sample on a metric carrying a trace ID, which is what lets a click on a latency spike jump straight into the exact slow trace — one of the OTCA curriculum's own named Metrics concepts. It needs Prometheus configured to store exemplars, the client library or OpenTelemetry Collector actually attaching a trace ID to the metric at record time, and the Grafana data source's exemplar config pointed at the right trace data source's uid. Any one of the three left unwired and the metric graphs perfectly well — just without the one feature that turns a spike into an answer.

Grafana vs. its alternatives

☺ Like you're 10: Other windows exist onto the same data — some skip the extra server entirely, one is a newer CNCF project trying the same idea with no proprietary export step.

Grafana itself isn't a named domain on the core Kubernetes exams this course assumes — CKA, CKAD and CKS test the workloads it draws, not the drawing tool — but "Dashboarding basics," "Configuring Alerting rules," and "Understand and Use Alertmanager" together make up 18% of the PCA blueprint, and exemplars linking a metric to a trace reappear on OTCA's own Metrics competency. Put it into practice in Capstone Part 4 — Observability, then go break your own rate-window and regex-variable mistakes on purpose in the broken-telemetry-pipeline drill before an incident teaches you the same lesson live. For the deeper, multi-cloud version of this same page — CloudWatch and Amazon Managed Grafana included — the Platform Engineering course's own Grafana reference goes further once the CNPE, not just the badge ladder, is the goal.