Grafana

Grafana doesn't collect a single metric or store a single log line of its own. It's the visualization and alerting layer that sits on top of whichever backend a team already runs — Prometheus for infrastructure metrics, CloudWatch for anything living in AWS, Loki or CloudWatch Logs for the log lines, even a Postgres table for a business number nobody thought belonged next to server metrics — and draws all of it as one dashboard instead of four browser tabs speaking four query languages. This page covers the three things that make it a genuine DevOps tool rather than a pretty picture: dashboards defined as version-controlled files instead of clicked-together artifacts that drift the moment someone edits one in the browser, a data-source model that lets one panel span backends nothing else in the room bridges automatically, and alerting rules that turn a query into a page landing on somebody's phone.

What Grafana is, and the problem it solves

☺ Like you're 10: Grafana is a picture-drawing window onto other people's data — it stores nothing itself, it just asks and draws.

Grafana is an open-source visualization and alerting server from Grafana Labs, first released in 2014 as a fork of an early Kibana dashboarding UI aimed squarely at time-series data instead of just logs. Point it at one or more data sources and it gives you three things on top of them: dashboards (saved, shareable, parameterized panels), Explore (a query scratchpad for the middle of an incident, with no dashboard to maintain), and alerting (rules that evaluate a query on a schedule and notify someone when it crosses a line). Recent major versions are dual-licensed under AGPLv3 and a Grafana-specific source-available license, and the project remains widely deployed either way — check the license terms that apply to the specific version you run before redistributing a modified build.

Before a shared pane exists, every backend brings its own console, and none of them talk to each other. Prometheus ships an expression browser that's fine for one query and useless for a service overview; CloudWatch needs its own login and speaks its own metric-math syntax; Loki and CloudWatch Logs ship no dashboarding UI worth the name at all. An on-call engineer at 3 a.m. holds three or four tabs, three or four query dialects, and does the mental join between them by hand, under stress, while the error budget drains. Grafana collapses that to one tab: a spike on a latency panel backed by Prometheus sits next to a request-count panel backed by CloudWatch, on the same time axis, refreshed together.

Where Grafana fits in the DevOps toolchain

☺ Like you're 10: It's the last stop on the Measurement pillar of CALMS — the step where numbers a system was already producing finally turn into something a human can look at and act on.

Grafana sits at the end of the observability chain covered in monitoring & observability: metrics, logs, and traces get collected and stored somewhere, and Grafana is the layer that turns raw telemetry into a shape a team actually looks at. That places it squarely under the Measurement letter of CALMS — a DORA metric or an SLO burn rate is only useful once someone can see it, and Grafana is where the DORA metrics and the burn-rate math from SLOs, error budgets & toil usually end up as an actual chart rather than a spreadsheet formula. It also sits directly upstream of incident management: an alert rule defined in Grafana is frequently the thing that turns a degraded metric into a page in the first place, which is why Pip the Hummingbird's job in that lesson so often starts with a rule written on this page.

Because it queries rather than stores, Grafana is genuinely tool-agnostic about what runs underneath it — a team that provisions its infrastructure with Terraform, ships containers via containers & orchestration, and already standardized on Prometheus and Loki for the CNCF side of its stack can point the exact same Grafana instance at CloudWatch the moment a workload moves onto a managed AWS service. See the DevOps toolchain for how it sits alongside the rest of the pipeline.

Architecture: one server, no storage of its own

☺ Like you're 10: One web server, a tiny notebook for its own settings, and a list of phone numbers for everyone else's data.

Architecturally Grafana is deliberately boring. It's a single Go binary serving an HTTP API and a web front end on port 3000, plus a small relational database for its own state, plus a set of data-source definitions it queries live on every panel refresh. When a panel loads, the browser doesn't talk to Prometheus or CloudWatch directly — it asks the Grafana server, and the server makes the call on the panel's behalf. That indirection is the default access: proxy mode, and it matters for two reasons: credentials for every backend live server-side, never in a browser the whole team can inspect, and a data source that's only reachable from inside a private network (a Prometheus instance with no public endpoint, say) still works, because the Grafana server sits inside that network even when nobody's laptop does.

Running the server yourself is the default path — a container, a VM, or a small fleet behind a load balancer, backed by an external Postgres or MySQL for anything beyond a single-node lab. On AWS specifically there's also a fully managed path: Amazon Managed Grafana runs Grafana Enterprise as a workspace AWS operates for you, authenticated through AWS IAM Identity Center or SAML rather than Grafana's own user database, with a purpose-built IAM role that can reach CloudWatch and Amazon Managed Service for Prometheus across accounts and regions without a static credential anywhere. It bills per active editor and viewer per month rather than per host — worth checking AWS's current pricing page before assuming a number, since active-user tiers have shifted before and will again.

The multi-data-source model: Prometheus, CloudWatch, Loki, and everything else in one pane

☺ Like you're 10: Every backend plugs into Grafana through the same kind of socket, so one dashboard can pull from several of them without anyone writing a translator.

A data source is a plugin plus a connection: a type (which query language and API it speaks), a url, and whatever authentication that backend needs. Clicking through the UI is fine for a single laptop; anything a team depends on should be provisioned — defined in YAML files Grafana reads at startup — so the whole observability surface is a reviewable directory in Git rather than state trapped in one server's database.

# /etc/grafana/provisioning/datasources/main.yaml
apiVersion: 1                        # Grafana's provisioning schema version, unrelated to Kubernetes
datasources:
  - name: Prometheus
    uid: prom                        # pin the uid — dashboards reference it by this, not by name
    type: prometheus
    access: proxy                    # the server queries, never the browser
    url: http://prometheus.monitoring.svc:9090
    isDefault: true
    jsonData:
      timeInterval: 30s              # match your real scrape interval

  - name: CloudWatch
    uid: cloudwatch
    type: cloudwatch
    jsonData:
      authType: default              # use the IAM role Grafana itself runs under — no static keys
      defaultRegion: us-east-1

  - name: Loki
    uid: loki
    type: loki
    access: proxy
    url: http://loki-gateway.monitoring.svc
    jsonData:
      derivedFields:                 # turn a trace id spotted in a log line into a clickable link
        - name: TraceID
          matcherRegex: 'trace_id=(\w+)'
          url: '${__value.raw}'

Three data sources with three completely different authentication stories sit side by side in that one file. Prometheus and Loki are reached over the network with no credentials at all, typical for a self-hosted backend living inside the same private network as Grafana. CloudWatch is reached through AWS's own identity system — authType: default tells the CloudWatch plugin to use whatever IAM role or instance profile the Grafana process is already running under, which is why the far more common production pattern is an IAM role scoped to exactly cloudwatch:GetMetricData, cloudwatch:ListMetrics, cloudwatch:GetMetricStatistics, logs:StartQuery/GetQueryResults (for CloudWatch Logs Insights), and the handful of ec2:Describe*/tag:GetResources calls Grafana uses to populate dimension pickers, rather than a long-lived access key pasted into a config file.

What actually earns the phrase "one pane" is a mixed data source panel: a single graph whose two queries point at two different uids. A deployment's request volume, reported by an Application Load Balancer straight into CloudWatch, can sit on the same timeseries as the same service's internally instrumented request rate from Prometheus — genuinely different collection paths, drawn on one shared time axis, which is the fastest way to catch a gap between what AWS's edge sees and what the application itself thinks it's handling.

Dashboards as code

☺ Like you're 10: A dashboard is just a text file describing what to draw, so it belongs in Git next to the code it's watching, not only inside one server's private database.

A dashboard is, underneath the drag-and-drop editor, a JSON document: a list of panels, each with a query, a visualization type, and formatting. That's the whole reason "dashboards as code" is a real practice and not a buzzword — export the JSON, commit it, and a dashboard gets the same review, the same diff, and the same rollback path as anything else in the pipeline. Two mechanisms get that file into a running Grafana without anyone clicking "New Dashboard" in production:

File-based provisioning points Grafana at a directory of dashboard JSON on disk, re-read on an interval — the natural fit when Grafana runs as a container alongside a Compose stack or on a plain VM. The Terraform provider (grafana/grafana on the Terraform Registry) is the natural fit for a team that's already managing the rest of its infrastructure with Terraform: data sources, folders, and dashboards all become ordinary resources reviewed in the same pull request as the infrastructure they watch. Argument names shift between provider versions more than core Terraform resources do, so treat the shape below as illustrative and check the registry docs for the version you pin.

resource "grafana_data_source" "prometheus" {
  type       = "prometheus"
  name       = "Prometheus"
  url        = "http://prometheus.monitoring.svc:9090"
  is_default = true
}

resource "grafana_data_source" "cloudwatch" {
  type = "cloudwatch"
  name = "CloudWatch"
  json_data_encoded = jsonencode({
    authType      = "default"
    defaultRegion = "us-east-1"
  })
}

resource "grafana_folder" "platform" {
  title = "Platform"
}

resource "grafana_dashboard" "checkout_golden_signals" {
  folder      = grafana_folder.platform.id
  config_json = file("${path.module}/dashboards/checkout-golden-signals.json")
}

Either mechanism needs the same discipline to actually hold: set allowUiUpdates: false on a file-based provider (or, on the Terraform path, simply never grant broad edit permissions to anyone outside the pipeline) so a well-meant browser edit can't quietly diverge from the file that's supposed to be the source of truth. That's the exact same lesson Terraform teaches about a hand-edited resource — a manual fix in the UI isn't wrong for finding the right panel, it's wrong for where the fix ends up living afterward. Build it in the browser, export the JSON, open a pull request.

Alerting rules: turning a query into a page

☺ Like you're 10: An alert rule is just a question Grafana asks on a schedule, plus a phone number to call the moment the answer is "yes, this is bad."

Unified alerting has been the default alerting engine since Grafana 9, and the older legacy engine has since been removed entirely — on any currently supported release, this is the only alerting model you'll meet. An alert rule is one or more queries plus an expression that reduces them to a boolean, evaluated by a rule group on a fixed interval; when the condition stays true for the rule's for duration, it fires. A contact point is somewhere to send the notification — Slack, email, a webhook, or a paging tool like PagerDuty or Opsgenie. A notification policy is a routing tree matching on labels that decides which contact point gets which alert, with its own grouping and repeat timing — deliberately the same shape as Prometheus's own Alertmanager, because Grafana ships an embedded Alertmanager that follows the identical model.

# /etc/grafana/provisioning/alerting/checkout.yaml
apiVersion: 1
groups:
  - orgId: 1
    name: checkout-slo
    folder: Platform
    interval: 1m
    rules:
      - uid: checkout-burn-fast
        title: Checkout fast burn (14.4x over 1h)
        condition: C
        for: 5m
        data:
          - refId: A
            datasourceUid: prom
            relativeTimeRange: { from: 3600, to: 0 }
            model:
              expr: |
                sum(rate(http_requests_total{job="checkout",code=~"5.."}[1h]))
                  / sum(rate(http_requests_total{job="checkout"}[1h]))
          - refId: C
            datasourceUid: __expr__            # the built-in threshold expression engine
            model: { type: threshold, expression: A,
                     conditions: [ { evaluator: { type: gt, params: [0.0144] } } ] }
        labels: { severity: page, team: checkout }
        annotations:
          summary: Checkout's 1-hour error ratio exceeds the SLO burn-rate budget
          runbook_url: https://runbooks.internal/checkout/burn

One distinction matters more than any single field in that file: Grafana shows two kinds of rule in the same list, and they behave very differently. Grafana-managed rules — the file above — are stored in Grafana's own database, evaluated by Grafana, and can query any data source at once, including joining a metric threshold with a log-based condition no single backend could express alone. Data-source-managed rules are ordinary Prometheus (or Mimir, or Loki ruler) alert rules that Grafana only displays and edits remotely; the backend itself evaluates and fires them through its own Alertmanager, with or without Grafana in the picture at all. Mixing both without deciding which one is authoritative for production paging is how a team ends up double-paged for one incident and silently unpaged for the next — pick one home, usually the backend's own rules for anything a single system can already express, and Grafana-managed rules for conditions that genuinely need more than one data source.

Day-to-day usage

☺ Like you're 10: There's barely a CLI — you reach it in a browser, poke it with the same HTTP API the UI itself uses, and do incident work in Explore.

# run it locally against an existing Prometheus/Loki
$ docker run -d -p 3000:3000 --name grafana grafana/grafana-oss

# liveness — the one worth putting in a smoke test
$ curl -s localhost:3000/api/health

# what does it actually think its data sources are, and are they healthy?
$ curl -s -H "Authorization: Bearer $TOKEN" localhost:3000/api/datasources | jq '.[].name'
$ curl -s -H "Authorization: Bearer $TOKEN" localhost:3000/api/datasources/uid/prom/health

# export a dashboard you built in the UI, ready to commit
$ curl -s -H "Authorization: Bearer $TOKEN" localhost:3000/api/dashboards/uid/checkout-gs \
    | jq '.dashboard' > dashboards/checkout.json

# current alert state, without opening a browser
$ curl -s -H "Authorization: Bearer $TOKEN" localhost:3000/api/alertmanager/grafana/api/v2/alerts

Create a service-account token in the UI for that $TOKEN — API keys are deprecated in favor of them. Explore is where an actual incident happens: no dashboard to maintain, no panel chrome, just a query box per data source and a shared time range, with a split view that puts a PromQL panel beside a LogQL panel so narrowing the metric narrows the logs alongside it. Practice that motion before an incident forces you to learn it live.

Gotchas and failure modes

☺ Like you're 10: Most Grafana pain is one of four things: someone edited in the browser, a variable wasn't what you thought, the time window fooled you, or CloudWatch got tired of being asked.

UI edits and provisioned files disagree, and the file always wins eventually. Someone improves a panel in the browser and hits Save; the live dashboard now disagrees with what's in Git. The next redeploy, or the next restart of a provider with allowUiUpdates: false, silently reverts the change, and a week of tuning evaporates with nobody sure which version was "right." The fix is policy, not a feature: build in the browser, export, PR the file — the same discipline this page already covered under dashboards as code.

Multi-select variables interpolate as a regex, not a literal. A dashboard variable with includeAll or multi-select turned on expands into a regex alternation like (dev|staging|prod), so a PromQL matcher written as namespace="$namespace" only ever matches a single selection — it needs =~"$namespace" to work once "All" is picked. This is the single most common "why is my panel empty for some selections and not others" ticket.

A fixed rate window makes graphs go blank when someone zooms out. rate(x[1m]) against a scrape interval of 30 seconds needs at least two samples inside that window; zoom the dashboard out to seven days and the window can land on a single sample, and rate() returns nothing. Grafana's built-in $__rate_interval variable computes a window guaranteed wide enough for the current zoom level and scrape interval — use it in place of a hard-coded duration in every rate() call.

Grafana vs. its alternatives

☺ Like you're 10: Other windows exist onto the same data — some come welded to one company's storage, some you have to run and patch yourself.

The practical rule: reach for Grafana — self-hosted or the managed AWS workspace — the moment more than one telemetry backend exists and a team wants one pane over all of them, config kept as reviewable text. Reach for CloudWatch's own dashboards when the estate really is single-cloud and adding a second tool costs more than it buys. Reach for a SaaS suite when the team is small enough that running observability infrastructure costs more engineer-hours than the license. Put this into practice in Capstone Part 4 — Observability, then go tighten a noisy alert set in Drill — Set Up Meaningful Alerts.

Grafana itself isn't an AWS-native service, so it won't be the headline subject of a DOP-C02 exam question — CloudWatch, X-Ray, and Amazon Managed Service for Prometheus carry that weight directly in the exam's Monitoring and Logging domain, covered in Monitoring & Logging. But the concepts on this page — dashboards as versioned artifacts, alerting on symptoms rather than causes, one pane spanning several data sources — are exactly what that domain is testing understanding of, regardless of which specific tool a question names; see the DOP-C02 exam guide and the service reference for how the weighting actually breaks down, and verify current percentages on AWS's own exam guide before studying against a fixed number printed here.