Prometheus

Prometheus is the cloud native metrics system: a single binary that walks your cluster on a timer, reads a plain-text scoreboard off every service’s /metrics endpoint, stores those numbers as labelled time series, lets you interrogate them in a query language called PromQL — and fires alerts when the answers get ugly. It solves the platform problem of “something is slow and nobody can prove what, when, or how much” by turning a running system into numbers every team can chart and alert on, with nothing to buy and nothing to configure per app.

What Prometheus is and the problem it solves

☺ Like you’re 10: It’s a robot that reads everybody’s numbers on a timer and remembers them, so you can draw pictures of the past and get warned about the present.

Prometheus was built at SoundCloud in 2012, donated to the CNCF in 2016, and became the second project ever to graduate — right after Kubernetes itself. That lineage matters: it was designed for the world Kubernetes creates, where targets are ephemeral and nobody can maintain a hand-written list of what to monitor. Today it is the default metrics backend of essentially every Kubernetes platform, and its query language and exposition format are de-facto standards other systems deliberately imitate.

The problem before Prometheus

Classical monitoring assumed stable hosts with names, checked against a static config, producing one number per check — “host web-03: CPU 78%, OK.” Kubernetes breaks every assumption in that sentence. Pods vanish in seconds, IP addresses are meaningless, and the interesting question is never “is web-03 up?” but “what is p99 checkout latency for version 1.4.3 in the EU region?” You need dimensions, not hostnames; discovery, not a hand-edited file; a query language, not a threshold.

The pull model and the exposition format

Prometheus’s defining architectural choice is that it pulls. Your application exposes an HTTP endpoint — conventionally /metrics — serving a simple line-based text format, and Prometheus fetches it on a schedule you control. Pull sounds like a detail; it is the whole design. Because Prometheus initiates, it controls the load and cannot be overwhelmed by a misbehaving client. A failed scrape is itself a signal — the synthetic up metric goes to 0, so target-down detection needs no heartbeat. No application needs monitoring credentials, and you can curl the endpoint yourself to see exactly what Prometheus sees. The format is deliberately boring:

# HELP http_requests_total Total HTTP requests handled.
# TYPE http_requests_total counter
http_requests_total{method="GET",route="/checkout",code="200"} 41822
http_requests_total{method="GET",route="/checkout",code="500"} 17

# HELP http_request_duration_seconds Request latency.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{route="/checkout",le="0.1"} 39120
http_request_duration_seconds_bucket{route="/checkout",le="0.5"} 41500
http_request_duration_seconds_bucket{route="/checkout",le="+Inf"} 41839
http_request_duration_seconds_sum{route="/checkout"} 5210.4
http_request_duration_seconds_count{route="/checkout"} 41839

That is the entire contract, which is why the ecosystem is enormous. Anything that cannot speak it natively gets an exporter that translates: node_exporter for machine CPU, memory, disk and network; kube-state-metrics for the state of Kubernetes objects (replicas wanted versus available, whether a Job failed); blackbox_exporter for probing a URL from outside; plus Postgres, Redis, Kafka and hundreds more.

What Prometheus deliberately is not

Prometheus is not a logging system, a tracing system, or a billing-grade record. It samples; it does not capture every event. Its storage is deliberately short-lived and single-node — no clustering, sharding or replication. It will tell you error rate rose to 4% at 14:32; it will never tell you which customer’s order failed. That is a feature: cheap, dimensional metrics are what make it affordable to watch every service. For per-request detail you go to logs and traces, as the observability lesson lays out.

Where Prometheus fits in a platform

☺ Like you’re 10: Prometheus sits in the platform’s watchtower. It doesn’t build or deploy anything — it watches everything else and tells other tools when to act.

In the reference architecture, Prometheus lives in the observability plane: a shared capability the platform team operates and every tenant consumes without installing anything. Its job is to be the metrics substrate — where numbers land — while other tools read from it. Because it is configured through Kubernetes objects, it is deployed and managed through GitOps like any other workload.

Its neighbours, and who does what

Prometheus is almost never used alone, and confusing it with its neighbours is a common exam stumble. Grafana stores nothing — it queries Prometheus and draws the result. OpenTelemetry is the vendor-neutral instrumentation standard whose Collector exports metrics into Prometheus, so the two are complements, not rivals. Alertmanager is a separate binary that receives fired alerts and decides who gets woken. KEDA autoscales on PromQL queries; OpenCost emits cost as Prometheus metrics; Argo Rollouts and Flagger query it mid-canary to promote or roll back. Prometheus is the shared numeric truth they all lean on.

CNPE domain relevance

Prometheus sits squarely inside the exam’s Observability domain — 20% of the blueprint — and is named on the official CNPE tool list. It leaks into other domains too, and the exam likes those seams: SLOs and error budgets are computed in PromQL, canary analysis queries Prometheus, FinOps dashboards are Prometheus queries, and custom-metric autoscaling reads from it. Treat it as one of the few tools you must be able to write, not merely recognise.

How it works — architecture, components, CRDs

☺ Like you’re 10: One program does four jobs in a loop: find the targets, read their numbers, store them, and run little rules over them.

The Prometheus server is a single Go binary with four internal jobs running continuously, plus a handful of friends around it. Understanding the loop makes almost every failure mode obvious.

Service discovery, relabelling, and the TSDB

Service discovery is how Prometheus learns what exists. On Kubernetes it queries the API server through kubernetes_sd_configs with a role — pod, service, endpoints, endpointslice, node or ingress — and gets a live list of candidates decorated with metadata labels like __meta_kubernetes_pod_label_app. Relabelling then filters and rewrites that list: relabel_configs run before the scrape and decide which targets survive and what labels they carry; metric_relabel_configs run after it and drop noisy series before storage. It is the most powerful and most confusing part of the config, and where cardinality problems get fixed.

Scraped samples land in the TSDB, Prometheus’s local time series database. Recent data sits in an in-memory head block protected by a write-ahead log; every two hours the head is compacted into an immutable on-disk block, and old blocks are merged then deleted. Retention defaults to 15 days (--storage.tsdb.retention.time, or a size cap with --storage.tsdb.retention.size). There is no clustering: HA means two identical replicas that both scrape everything, and long retention means shipping data elsewhere.

The four metric types

The exposition format labels every metric with a type, and choosing the wrong one is the most common instrumentation bug.

Histogram versus summary is a favourite exam question. A histogram ships raw bucket counts, so Prometheus can add ten pods’ buckets together and compute a fleet-wide p99 correctly. A summary ships an already-computed p99 per pod, and there is no valid mathematics for averaging percentiles — so summaries are effectively unusable in a distributed system. Prefer histograms, with bucket boundaries straddling your SLO.

The Prometheus Operator and its CRDs

On Kubernetes you almost never hand-edit prometheus.yml. The Prometheus Operator — packaged with everything else as the kube-prometheus-stack Helm chart — turns configuration into custom resources, so scrape config becomes a declarative, GitOps-managed, self-service thing a tenant can create in their own namespace. The Operator watches those CRs, generates the real config into a Secret, and reloads Prometheus.

All of them live in the monitoring.coreos.com API group, but not all at the same version: Prometheus, ServiceMonitor, PodMonitor, PrometheusRule, Alertmanager and Probe are served at v1, while AlertmanagerConfig and ScrapeConfig have historically trailed on an alpha/beta version. Before writing one from memory, confirm what your cluster actually serves with kubectl api-resources --api-group=monitoring.coreos.com.

The queries and resources you will actually write

☺ Like you’re 10: Two things you type all the time: little questions in PromQL, and little YAML files that say “watch this” and “ring the bell when that.”

PromQL essentials

PromQL has two result shapes you must tell apart. An instant vector is one value per series at a single moment — http_requests_total. A range vector adds a bracketed duration and returns a set of samples over time per series — http_requests_total[5m]. Range vectors cannot be graphed directly; they exist to be fed to a function that collapses them back to an instant vector, which is why rate() always takes a bracketed range.

# Per-second rate of a COUNTER, averaged over a 5-minute window.
# rate() handles counter resets, and needs at least two samples in the window —
# so keep the range >= 4x your scrape interval (30s scrape -> [2m] minimum).
rate(http_requests_total[5m])

# irate() uses only the LAST TWO samples: spiky, good for zooming into a
# fast-moving graph, bad for alerts (it misses everything in between).
irate(http_requests_total[1m])

# increase() = total growth over the window (it is rate() * window seconds).
increase(http_requests_total{code=~"5.."}[1h])

# Aggregate ACROSS series. "by" keeps the listed labels; "without" drops them.
sum by (route, code) (rate(http_requests_total[5m]))
sum without (instance, pod) (rate(http_requests_total[5m]))

# Error ratio: aggregate numerator and denominator the SAME way, then divide.
  sum by (route) (rate(http_requests_total{code=~"5.."}[5m]))
/ sum by (route) (rate(http_requests_total[5m]))

# p99 latency from a histogram. The "le" label MUST survive the aggregation.
histogram_quantile(0.99,
  sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))

# offset = look back in time. Compare now against the same moment last week.
  sum(rate(http_requests_total[5m]))
/ sum(rate(http_requests_total[5m] offset 7d))

# Subquery: run an inner range query over an outer window.
# "the highest 5m error rate seen at any point in the last hour, sampled each minute"
max_over_time( sum(rate(http_requests_total{code=~"5.."}[5m]))[1h:1m] )

# Selectors: = exact, != not, =~ regex match, !~ regex not-match.
up{job="checkout", namespace!="kube-system"} == 0

Two rules carry most of PromQL. First, rate() only makes sense on a counter — on a gauge it produces confident nonsense. Second, rate first, aggregate second: sum(rate(x[5m])) is correct; summing counters before rating them mishandles pod restarts. Binary operators match series by label set, which is why the error-ratio example aggregates both sides identically — mismatched labels silently return an empty result rather than an error.

A ServiceMonitor (and its PodMonitor twin)

This is the resource an application team writes to get scraped. Note the three things that must line up: the selector matches labels on the Service, the port is the Service port’s name (not a number), and the resource’s own labels must match what the Prometheus CR is selecting.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: checkout
  namespace: checkout
  labels:
    release: kube-prometheus-stack   # <-- the label the Prometheus CR selects on
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: checkout   # must match the SERVICE's labels
  namespaceSelector:
    matchNames: [checkout]
  endpoints:
    - port: http-metrics       # the NAME of a port on the Service, not 9090
      path: /metrics           # default is /metrics
      interval: 30s
      scrapeTimeout: 10s
      metricRelabelings:       # drop a known-noisy series before storing it
        - sourceLabels: [__name__]
          regex: 'go_gc_duration_seconds.*'
          action: drop
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor          # same idea, but selects Pods directly (no Service needed)
metadata:
  name: batch-workers
  namespace: checkout
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: batch-worker   # matches POD labels
  podMetricsEndpoints:
    - port: metrics        # the NAME of a containerPort on the Pod
      interval: 30s

A PrometheusRule with recording and alerting rules

Recording rules pre-compute an expensive expression on a schedule and store the result as a new series — dashboards and alerts then query the cheap pre-computed series. The convention for their names is level:metric:operations. Alerting rules evaluate an expression and fire when it returns any series; for requires the condition to hold continuously before firing, which is what kills flapping.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: checkout-slo
  namespace: checkout
  labels:
    release: kube-prometheus-stack   # same adoption label as the ServiceMonitor
spec:
  groups:
    - name: checkout.rules
      interval: 30s
      rules:
        # RECORDING RULE — compute once, query many times.
        - record: job:checkout_request_error_ratio:rate5m
          expr: |
            sum by (job) (rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
              / sum by (job) (rate(http_requests_total{job="checkout"}[5m]))

        # ALERTING RULE — a SYMPTOM alert: users are actually being hurt.
        - alert: CheckoutHighErrorRate
          expr: job:checkout_request_error_ratio:rate5m > 0.02
          for: 10m                     # must hold 10 minutes before it fires
          labels:
            severity: critical         # Alertmanager routes on these
            team: payments
          annotations:                 # for humans; templating is allowed
            summary: "Checkout error ratio is {{ $value | humanizePercentage }}"
            runbook_url: "https://runbooks.acme.dev/checkout-errors"

        # A latency alert built from the histogram.
        - alert: CheckoutLatencySLOBreach
          expr: |
            histogram_quantile(0.99,
              sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout"}[5m]))
            ) > 0.75
          for: 15m
          labels: { severity: warning, team: payments }
          annotations:
            summary: "p99 checkout latency above the 750ms SLO"

Alertmanager routing

Prometheus decides what is wrong; Alertmanager decides who hears about it, and how often. Its config is a routing tree: an alert enters at the root and walks down until it matches a node, inheriting whatever it does not override. group_by collapses related alerts into one notification, group_wait briefly holds the first send so siblings can join, and inhibit_rules suppress a lesser alert while a severe one fires. Silences are created at runtime via the UI or amtool, not in this file.

global:
  # Slack receivers need a webhook URL from here or from the receiver itself.
  slack_api_url_file: /etc/alertmanager/secrets/slack/url

route:
  receiver: platform-default
  group_by: [alertname, namespace, cluster]   # one notification per group
  group_wait: 30s            # wait for siblings before the first send
  group_interval: 5m         # wait before sending updates to an existing group
  repeat_interval: 4h        # re-nag interval for a still-firing group
  routes:
    - matchers: [ 'severity="critical"' ]
      receiver: pagerduty-oncall
      continue: false        # the default; stop here, do not try later siblings
    - matchers: [ 'team="payments"' ]
      receiver: slack-payments

inhibit_rules:               # if the whole cluster is down, don't also page
  - source_matchers: [ 'alertname="ClusterDown"' ]  # for every service in it
    target_matchers: [ 'severity=~"warning|critical"' ]
    equal: [cluster]         # only inhibit alerts sharing this label's value

receivers:
  - name: platform-default
    slack_configs:
      - channel: '#platform-alerts'
        send_resolved: true
  - name: pagerduty-oncall
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pd/key   # Events API v2
  - name: slack-payments
    slack_configs:
      - channel: '#payments-alerts'

Day-to-day commands

☺ Like you’re 10: Mostly you check your rules are spelled right, open the web page, and look at the list of things it’s watching.

promtool — validate before you commit

promtool ships in the Prometheus image and is your CI gate. Plain CRD schema validation checks structure, not PromQL, so unless the Prometheus Operator’s validating admission webhook is deployed and reachable (the kube-prometheus-stack chart installs one for PrometheusRule by default), a rule with a broken expression is accepted by the API server and then simply never loads. Validating in CI is the belt-and-braces fix — but note that promtool check rules expects a plain Prometheus rule file, so lift .spec out of the custom resource first.

promtool check config /etc/prometheus/prometheus.yml   # whole server config
promtool check rules rules/*.yaml                      # syntax of PLAIN rule files
promtool test rules tests/checkout_test.yaml           # unit-test alerts against fake data
promtool check metrics < scraped.txt                   # lint exposition output

# A PrometheusRule is not a rule file — lift .spec out before linting it.
kubectl -n checkout get prometheusrule checkout-slo -o jsonpath='{.spec}' > /tmp/rules.yaml
promtool check rules /tmp/rules.yaml

promtool query instant  http://localhost:9090 'up == 0'
promtool query range    http://localhost:9090 --start=$(date -u -d '-1 hour' +%Y-%m-%dT%H:%M:%SZ) \
  'sum(rate(http_requests_total[5m]))'

# The Operator stores generated config in a Secret — extract it to lint it.
kubectl -n monitoring get secret prometheus-kube-prometheus-stack-prometheus \
  -o jsonpath='{.data.prometheus\.yaml\.gz}' | base64 -d | gunzip | head -40

Reaching the UI and reading its status pages

# The Operator always creates a headless Service called prometheus-operated.
kubectl -n monitoring port-forward svc/prometheus-operated 9090
#   http://localhost:9090/targets   <- is my target UP? what was the scrape error?
#   http://localhost:9090/config    <- did my ServiceMonitor become scrape config?
#   http://localhost:9090/alerts    <- inactive / pending / firing
#   http://localhost:9090/tsdb-status <- top label/metric cardinality offenders

kubectl -n monitoring port-forward svc/alertmanager-operated 9093

# Query the HTTP API directly (handy inside a debug pod or a script).
curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=up{job="checkout"}'
curl -s  'http://localhost:9090/api/v1/targets?state=active' | jq '.data.activeTargets[].health'
curl -s  http://localhost:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName[:10]'
curl -s  http://localhost:9090/-/healthy ; curl -s http://localhost:9090/-/ready

Inspecting the Operator’s objects, and Alertmanager

kubectl get servicemonitor,podmonitor,prometheusrule -A
kubectl -n monitoring get prometheus -o yaml | grep -A6 'serviceMonitorSelector'  # THE gotcha
kubectl -n monitoring logs sts/prometheus-kube-prometheus-stack-prometheus -c prometheus
kubectl -n monitoring logs deploy/kube-prometheus-stack-operator   # config generation errors

# Prove an app is actually exposing metrics, from inside the cluster.
kubectl -n checkout exec deploy/checkout -- wget -qO- localhost:8080/metrics | head

amtool --alertmanager.url=http://localhost:9093 alert query
amtool --alertmanager.url=http://localhost:9093 silence add alertname=CheckoutHighErrorRate \
  --duration=2h --comment="planned migration"
amtool config routes test --config.file=alertmanager.yml severity=critical team=payments

Gotchas and failure modes

☺ Like you’re 10: Here are the ways Prometheus quietly does nothing and lets you think it’s working.

“No metrics” — the label mismatch that causes 80% of tickets

This is the most common Prometheus problem on Kubernetes, and it is worth memorising as a chain. A Prometheus CR only adopts monitors matching its serviceMonitorSelector and living in a namespace matching its serviceMonitorNamespaceSelector. The kube-prometheus-stack chart defaults that selector to release: <your-helm-release-name>, so a perfectly valid ServiceMonitor without that label is simply ignored — no error, no event, nothing in /targets. Work the chain in order: does it carry the release label? Does its selector match the Service’s labels (not the Deployment’s or Pod’s)? Is the namespaceSelector right? Is endpoints[].port the port name on the Service? And does that Service actually have ready backends (kubectl get endpointslices -l kubernetes.io/service-name=checkout)? Workload triage walks the same ladder for pods that never go ready.

Cardinality explosions

Each unique label-value combination is a separate series held in memory. Put an unbounded value in a label — a user ID, an email, a request ID, a full URL with query string — and series count jumps from thousands to millions. Queries slow, then time out, then Prometheus is OOM-killed and restarts into the same problem. The cure is bounded labels only (status code, method, route template, region, version) plus metricRelabelings to drop known offenders at ingestion. Diagnose with /tsdb-status, topk(10, count by (__name__)({__name__=~".+"})), and prometheus_tsdb_head_series. High-cardinality identifiers belong on traces and logs, never on metric labels.

rate() on the wrong thing, or the wrong window

rate() requires a counter and at least two samples inside the window. Ask for rate(x[15s]) with a 30-second scrape interval and you get an empty result — not an error, an empty graph, which people read as “no traffic.” The rule of thumb is a range of at least four times the scrape interval. Applying rate() to a gauge is silently meaningless. And a low-traffic endpoint that gets one request an hour will show rate values that look like zero, so use increase() over a long window for rare events.

Storage, HA, and the things Prometheus will not do for you

Prometheus keeps roughly 15 days locally and has no replication: if the volume dies, the history is gone. Two replicas give availability, not durability, and they disagree slightly because they scrape at different instants — which is why HA pairs are deduplicated downstream. For long retention, durable object storage and a global query view you add Thanos or Grafana Mimir, fed by the Thanos sidecar or by remote_write. Two further traps: remote_write queues fill and drop samples when the receiver is slow (watch prometheus_remote_storage_samples_dropped_total), and alerting still depends on the local Prometheus — if it is down, nothing fires at all, so alert on Prometheus itself from somewhere else. The troubleshooting playbook has the wider decision tree.

Alternatives and when to choose it

☺ Like you’re 10: Other notebooks exist. Most of them copy Prometheus’s handwriting so you can switch without relearning.

The remarkable thing about the alternatives is how many speak PromQL and accept Prometheus remote_write. That is a real platform-engineering advantage: dashboards, rules and instrumentation stay portable, so the backend becomes a swappable detail rather than a decade of lock-in.

The comparison that decides it

A practical rule

Start with Prometheus per cluster, run it through the Operator so tenants self-serve their own ServiceMonitors, and add Thanos or Mimir only when a real requirement arrives — “13 months for a compliance report,” or one dashboard across six clusters, which is also a multi-cluster conversation. Do not start with the distributed system. See The Tool Landscape for how Prometheus sits among the other named CNPE projects.

Exam relevance and going further

☺ Like you’re 10: On exam day you cannot open Prometheus’s website — so the YAML and the queries have to already live in your head.

Prometheus is on the official CNPE tool list and is the most likely vehicle for an Observability task. Expect to be asked to make a workload scraped, to write or repair a PrometheusRule, to read a PromQL expression and say what it means, to explain pull versus push, or to diagnose a target that is not appearing.

The documentation allowlist — read this twice

⚖ CNPA vs CNPE — That allowlist is a CNPE-only mechanic: CNPE is hands-on, so it permits those narrow live lookups mid-task. CNPA is stricter, not looser — a fully closed-book, multiple-choice exam with zero external resources and zero lookups of any kind, so neither prometheus.io nor kubernetes.io would be reachable there either. Even so, the concept-level knowledge above — the pull model, the four metric types, and reading a PromQL expression like histogram_quantile over a rate — is exactly the kind of thing CNPA's closed-book recall draws on.

What to be able to do without notes

Write a ServiceMonitor from a blank file: apiVersion: monitoring.coreos.com/v1, the adoption label, selector.matchLabels matching the Service, a namespaceSelector, and endpoints with a port name, path and interval. Write a PrometheusRule with one recording rule (record/expr) and one alerting rule (alert/expr/for/labels/annotations). Name the four metric types and say why a summary cannot be aggregated but a histogram can. Write sum by (le) (rate(..._bucket[5m])) inside histogram_quantile without hesitating. Explain up == 0, the pull model, and what Alertmanager adds. Know the diagnostic path: /targets, then selector labels, then Service endpoints. The CLI spine sits in the command reference.

Official resources for after the exam

Outside the exam the canonical sources are prometheus.io/docs (the Querying and Best practices sections repay slow reading), the Operator API reference at prometheus-operator.dev, github.com/prometheus-operator/kube-prometheus, the exposition standard at openmetrics.io, and cncf.io/projects/prometheus. Pair this page with Observability for the pillars and SLOs, Grafana for the dashboards on top, OpenTelemetry for the instrumentation underneath, and the glossary when a term stops making sense.