Prometheus

Every PromQL expression you've already seen in this course — histogram_quantile() on the checkout API's latency, rate(http_requests_total[5m]) feeding a burn-rate alert — assumes one specific piece of software underneath it, and this course has never actually stopped to introduce it by name. That software is Prometheus: an open-source, CNCF-graduated systems-monitoring toolkit built around one deliberately narrow idea — it reaches out and pulls a number off every target it knows about, on a fixed schedule, and stores what it collects in a purpose-built time-series database it also wrote. This page covers that pull model in full, the TSDB and PromQL that sit on top of it, how Prometheus finds what to scrape in a world where pods move constantly, why its own storage engine was never meant to hold years of history, and the whole ecosystem of long-term-storage projects that gap spawned.

What Prometheus is and the problem it solves

☺ Like you're 10: It's one program that walks around asking things "what's your number?" on a timer, remembers the answers on its own hard disk, and comes with its own language for asking questions about what it remembers.

Prometheus began at SoundCloud in 2012, modeled loosely on an internal Google system called Borgmon, and joined the Cloud Native Computing Foundation in 2016 as its second hosted project after Kubernetes — which is no accident; the two were built to fit together. A single Prometheus binary does four jobs at once: it discovers what to monitor, it scrapes (pulls) numeric data from those targets on a schedule, it stores what it collects in its own on-disk time-series database, and it evaluates rules against that data to compute derived series and fire alerts. No separate agent, no separate database, no separate query engine to install — one binary, one config file, one data directory.

The problem it solves is the one every one of this course's earlier pages has assumed was already handled: given a fleet of services that scale up and down and get rescheduled onto different hosts constantly, how do you get a consistent, queryable, alertable numeric history out of all of them without hand-wiring every service to know where to send its data? Prometheus's answer is to flip the direction of that relationship entirely — the monitoring system finds the services, not the other way around.

Pull versus push, and why it matters more than it sounds

Older and rival systems — Graphite, StatsD, most commercial APM agents, Datadog's agent — are push-based: something running next to your application actively sends metrics outward to a collector. Prometheus is pull-based: your application (or an exporter sitting beside it) exposes a plain-text HTTP endpoint, conventionally /metrics, and Prometheus itself decides when to visit it. This single design choice has consequences that ripple through everything else on this page:

• Liveness is free. If a scrape fails, Prometheus knows immediately and sets the built-in up metric to 0 for that target — a dead or unreachable service is a first-class, queryable, alertable fact, not something you have to infer from the absence of pushed data (which push-based systems have to solve separately and less cleanly).
• The exposition format is trivially simple. A target's /metrics endpoint is just plain text — a metric name, an optional set of {label="value"} pairs, and a number, one per line, with # HELP and # TYPE comments for documentation. Instrumenting anything that can serve HTTP takes a client library and a few lines of code.
• Central control over load. Prometheus, not a thousand independently-configured agents, decides the scrape interval — so you can never accidentally get metrics-DDoSed by a misconfigured fleet all pushing at once.
• It genuinely doesn't fit everything. A batch job that runs for eleven seconds and exits before the next scrape interval has nothing to be pulled from — Prometheus's answer is the Pushgateway, a small intermediary that short-lived jobs push a final result to, which Prometheus then scrapes like any other target. It exists specifically for this one exception; reaching for it to make normal long-running services "push" defeats the entire model and is one of the most common Prometheus misuses in the wild.

How it works — architecture and the scrape pipeline

☺ Like you're 10: Find the targets, walk to each one on a timer and copy down its numbers, save the copies to disk in two-hour bundles, and answer questions about all of it in its own question language.

Everything Prometheus does traces back to one loop: discover targets, scrape them, write what comes back to local storage, evaluate rules against that storage, repeat. The diagram below is the shape that loop takes in a real deployment, including the piece — the arrow leaving on the bottom — that this page spends its second half on.

Service discovery: finding a fleet that never sits still

Hard-coding target IP addresses (static_configs) works for a handful of stable hosts and nothing else — it's the first thing every real deployment outgrows. Prometheus ships service discovery integrations for most infrastructure a team actually runs on: kubernetes_sd_configs (with a role of pod, service, endpoints, endpointslice, node, or ingress), consul_sd_configs, ec2_sd_configs, azure_sd_configs, gce_sd_configs, file_sd_config (for anything without a native integration — an external script writes a JSON file, Prometheus watches it), and a generic http_sd_config for custom discovery sources. On Kubernetes specifically, the dominant pattern is annotation-based: a pod carries prometheus.io/scrape: "true", and every discovered pod — scrapeable or not — is filtered down by a relabel_configs stanza that keeps only the annotated ones.

This is the single most misunderstood piece of a Prometheus config, so it's worth being precise about the two relabeling stages by name, because they run at different times and do different jobs:

• relabel_configs runs before the scrape, against the labels service discovery produced (all prefixed __meta_* and dropped before storage unless kept). It decides whether a discovered target gets scraped at all, and can rewrite __address__ and __metrics_path__ to point the scrape somewhere non-default.
• metric_relabel_configs runs after the scrape, against the actual samples that came back. It's where you drop a noisy, high-cardinality metric an exporter insists on emitting, or strip a label before it ever touches disk.

# prometheus.yml — annotation-based Kubernetes pod discovery for the checkout service
scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # only scrape pods that opted in
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      # honor a custom metrics path if the pod declares one
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      # honor a custom port if the pod declares one
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      # carry useful metadata onto every sample as real labels
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app
    metric_relabel_configs:
      # this exporter emits one series per raw request path — drop it before it hits disk
      - source_labels: [__name__]
        regex: "http_request_duration_seconds_bucket_by_raw_path"
        action: drop

The TSDB: what actually happens on disk

Prometheus writes every incoming sample to a write-ahead log (WAL) first, for crash safety, while holding the last roughly two hours of data in an in-memory head block. Every couple of hours that head block is flushed to disk as an immutable block — a directory containing compressed chunks, an index, and metadata — and a background compactor continuously merges adjacent small blocks into larger ones, up to the configured retention limit. Retention is most commonly bounded by time (--storage.tsdb.retention.time, historically defaulting to around 15 days) but can also be bounded by disk size (--storage.tsdb.retention.size) — check the flag defaults against your installed version, since they've shifted across Prometheus releases. Compression is aggressive: Prometheus's own documentation has long cited roughly 1–2 bytes per sample on typical production workloads, which is what makes millions of active series per server tractable at all on ordinary disks.

PromQL: the query language built for exactly this shape of data

PromQL has two selector shapes that everything else builds on. An instant vector — http_requests_total{route="/checkout"} — returns one value per matching series at a single point in time. A range vector — the same selector with a duration appended, http_requests_total{route="/checkout"}[5m] — returns every sample for each matching series over the trailing window, and exists almost exclusively to be fed into a rate-style function. rate() computes the per-second average rate of increase over the range, correctly handling counter resets (a process restarting and its counter dropping back to zero); irate() does the same using only the last two points, more responsive but noisier; increase() returns the total increase over the window rather than a per-second rate. Aggregation operators — sum(), avg(), max(), count(), topk() — collapse many series into fewer, almost always paired with a by (label, ...) or without (label, ...) clause to control which labels survive the aggregation. histogram_quantile() is the function this course's SLO pages lean on hardest: given a histogram's _bucket series (each carrying a le — "less than or equal" — label marking its upper boundary), it interpolates a percentile like p99 across the buckets.

Recording rules pre-compute an expensive or frequently-reused expression on a schedule and save the result back into the TSDB as a new series — exactly the mechanism behind every sre:checkout_requests:error_ratio* series in multi-window, multi-burn-rate alerting. Alerting rules are the same idea aimed at a boolean condition instead of a value: when the expression's result set is non-empty, Prometheus fires an alert and pushes it to a separate binary, Alertmanager, which handles deduplication, grouping, silencing, and routing to a paging tool — the connective tissue that makes symptom-based alerting actually reach a human.

The config and rules you actually write

☺ Like you're 10: One file tells Prometheus how often to check things and where to look; separate rule files tell it what math to pre-compute and when to yell for help.

prometheus.yml is the one file that governs everything on this page: how often to scrape, which discovery mechanisms to use, where Alertmanager lives, and which rule files to load. A realistic top-level shape, extending the checkout SLO already worked through in SLIs, SLOs & error budgets:

global:
  scrape_interval: 15s        # how often targets get pulled
  evaluation_interval: 15s    # how often recording/alerting rules run
  external_labels:
    cluster: prod-us-east
    replica: prometheus-a     # distinguishes this replica from its HA twin — see Gotchas

rule_files:
  - "rules/*.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["10.0.1.11:9100", "10.0.1.12:9100"]
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs: [ ... ]     # the annotation-filter block shown above

# stream every sample onward the moment it's written — see "Why local storage..." below
remote_write:
  - url: "https://mimir.example.com/api/v1/push"
    queue_config:
      max_samples_per_send: 5000

Rule files hold the actual PromQL. See the full worked recording-and-alerting-rule set for the checkout API's 99.9% SLO in multi-window, multi-burn-rate alerting — it's the most realistic, production-shaped Prometheus rules file this course has, and it's worth reading straight after this page.

Day-to-day commands

☺ Like you're 10: A handful of commands cover almost everything: check the config for typos, ask it a question directly, and tell it to re-read its config without restarting.

# run it (a real deployment uses a Deployment/StatefulSet + PVC, not this — but this is the fastest way to see it)
$ docker run -d -p 9090:9090 \
    -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
    prom/prometheus --config.file=/etc/prometheus/prometheus.yml \
    --web.enable-lifecycle          # required for the /-/reload endpoint below

# validate BEFORE you ship a config change — catches typos, bad regex, dangling rule_files
$ promtool check config prometheus.yml
$ promtool check rules rules/checkout-slo.yml

# unit-test rules against synthetic input series — no live cluster required
$ promtool test rules rules_test.yml

# hot-reload config without restarting (needs --web.enable-lifecycle, or send SIGHUP instead)
$ curl -X POST localhost:9090/-/reload

# ask it something directly, the same way Grafana would
$ promtool query instant http://localhost:9090 'up{job="node"}'
$ promtool query range http://localhost:9090 'rate(http_requests_total{route="/checkout"}[5m])' \
    --start=2026-08-15T00:00:00Z --end=2026-08-16T00:00:00Z --step=1m

# a target's own exposition format — this is literally what Prometheus itself scrapes
$ curl -s localhost:9090/metrics | head -20

# liveness / readiness for load balancers and Kubernetes probes
$ curl -s localhost:9090/-/healthy
$ curl -s localhost:9090/-/ready

# inspect what's actually stored, block by block, when disk usage looks wrong
$ promtool tsdb list /var/lib/prometheus/data

The web UI at :9090 is worth knowing even if Grafana is where dashboards actually live: Status → Targets shows every discovered target's up state and the exact error from its last scrape, and Status → Service Discovery shows the __meta_* labels a target had before relabeling ran — the single fastest way to debug "why isn't this pod being scraped."

Why local storage isn't built for years of retention — the remote-write ecosystem

☺ Like you're 10: Prometheus's own notebook only has room for a couple of weeks, on purpose — so if you need years of history, it photocopies every page the moment it's written and mails the copies to a bigger warehouse built by someone else.

This is the gap the content brief for this page points straight at, and it's not an accident or an oversight — it's a deliberate design boundary. A single Prometheus server is a single node with local disk and no replication: lose that disk and you lose that server's history, there's no clustering to fail over to, and the on-disk format was optimized for the query patterns of the last few weeks (dashboards, active alerting), not for cheaply scanning years of cold data. Pushing --storage.tsdb.retention.time out to a year or more on a single instance is technically possible and a well-documented way to learn this the hard way — disk usage and query latency both grow with retained series count and time range, and a lone server has no mechanism to shard that load across machines.

Prometheus's own answer to "then how does anyone keep years of data" is not to fix the local TSDB — it's an API: remote_write. Configured with a URL and some tuning knobs (as shown in the config above), Prometheus streams every sample it ingests onward, in near real time, over a protobuf-and-snappy-compressed HTTP protocol, straight out of its own WAL. The local TSDB keeps functioning exactly as before — remote_write is additive, not a replacement — but now a separate system downstream is receiving the same firehose and can do with it whatever the local TSDB structurally can't: replicate it, shard it across machines, downsample it for cheap long-range queries, and retain it for years.

That gap is precisely what spawned an entire category of software, and knowing the shape of it is worth more than memorizing any one vendor's name:

• Thanos — a CNCF project that wraps Prometheus rather than replacing it: a Sidecar uploads each local TSDB block to object storage (S3, GCS, or similar) as soon as it's written, a Store Gateway serves queries against that historical data, a Compactor downsamples old blocks to keep long-range queries cheap, and a Querier presents one global PromQL endpoint across every Prometheus instance and the object-store history behind it.
• Cortex and Grafana Mimir — horizontally scalable, multi-tenant systems built to directly receive remote_write traffic at very large scale; Mimir is Grafana Labs' newer, largely Cortex-derived project and has become the more actively developed of the two.
• VictoriaMetrics — a purpose-built, notably efficient TSDB that accepts remote_write directly (or runs as a drop-in Prometheus-API-compatible replacement outright), commonly reached for specifically because of its disk-footprint and query-speed advantages at high cardinality.
• Managed cloud services — Amazon Managed Service for Prometheus, Google Cloud Managed Service for Prometheus, and Grafana Cloud all offer the remote_write receiving end as a hosted product, trading operational effort for a bill.

Every one of these keeps PromQL as the query language and, in most cases, Prometheus itself as the thing doing the scraping — they replace the storage tier, not the collection model. That's precisely why this whole category exists rather than a wholesale switch to a different monitoring system: teams want Prometheus's pull model, its Kubernetes-native service discovery, and PromQL, but need retention and scale the single-node local TSDB was never designed to provide alone.

Gotchas and failure modes

☺ Like you're 10: Most Prometheus incidents come from one root cause — someone let a label take on too many different values, and now there are a million tiny notebooks instead of one big one.

Cardinality explosion

Because every unique combination of label values is a distinct stored series, a label that takes an effectively unbounded number of values — a raw request path, a user ID, a UUID, a full email address — multiplies the number of series by however many distinct values pass through it. A metric with ten sane labels and one user_id label can turn a few thousand series into tens of millions almost overnight, and the damage isn't cosmetic: memory usage on the head block, disk usage on compacted blocks, and query latency (PromQL has to touch every matching series) all scale with series count. The fix is architectural, not clever PromQL: never put unbounded values in a label, and use metric_relabel_configs with action: labeldrop or action: drop as a backstop against an exporter that does it anyway.

rate() on the wrong metric type, and absent data

rate() and increase() are meaningful only on counters — applying them to a gauge produces a number that looks plausible and means nothing. Equally common: PromQL functions silently return no data for a series that's stopped existing (a pod rescheduled with new labels, a target that's been down since before the query window), which means a naive alert can develop a silent blind spot exactly when something is most wrong. The absent() function exists specifically to catch this — absent(up{job="checkout"}) fires when the series itself has vanished, which a plain threshold check on that series never will.

No built-in HA, and how teams actually get it

A single Prometheus server is a single point of failure with no clustering of its own. The standard mitigation is running two (or more) identical Prometheus replicas, each independently scraping the exact same targets with the exact same config — not sharing storage, not coordinating with each other — distinguished only by an external_labels.replica value like the config example above. Alertmanager then deduplicates alerts that arrive from both replicas for the same underlying condition, so a real incident pages once, not twice, while either replica can vanish without anyone losing observability.

The other operational sharp edges

• WAL corruption after an unclean shutdown (an OOM-kill mid-write, a hard node failure) can leave the WAL in a state Prometheus can't fully replay on restart; it usually self-recovers by truncating the damaged segment, at the cost of losing whatever hadn't been flushed to a block yet.
• Relabeling order matters. relabel_configs entries run top to bottom, and a keep/drop rule earlier in the list can silently remove a target before a later rule ever gets to see it — when a target that should be scraped isn't, Status → Service Discovery in the UI, not guesswork, is the fast way to find which rule ate it.
• Scrape timeout must stay below scrape interval. A scrape_timeout longer than scrape_interval lets scrapes of a slow target overlap and queue, which is its own quiet source of missing or duplicated-looking data.
• Federation is not a long-term-storage substitute. The /federate endpoint lets one Prometheus scrape an aggregated subset of another's series — useful for hierarchical rollups across teams, but it's still pull, still local storage on the receiving end, and solves a different problem than remote_write and the systems built on it above.

Where Prometheus sits under this course's stack

☺ Like you're 10: Nearly every math-heavy page in this course was quietly written assuming Prometheus was already running underneath it — this is the page where that assumption finally gets a name.

Prometheus is the layer monitoring & observability's four golden signals get collected into, the query engine SLIs, SLOs & error budgets defines an SLI against, and the exact rule format multi-window, multi-burn-rate alerting hand-writes as YAML. Sloth exists specifically to generate that YAML from a shorter declarative spec rather than hand-maintaining it at scale; Grafana is almost always the dashboard sitting on top querying it over PromQL; and a commercial platform like Nobl9 typically ingests its data from a Prometheus (or Thanos/Mimir) deployment rather than replacing it outright. See the SRE toolchain for how Prometheus sits alongside the rest of the metrics, tracing, paging, and chaos categories, and Monitoring & Service Level Indicators for how this material maps onto the exam blueprint. If you want hands-on reps with the config and rules on this page rather than just reading about them, Capstone Part 2 — build the monitoring & alerting is exactly that exercise.

Alternatives and when to reach for something else

☺ Like you're 10: Other tools ask the same "what's your number" question — some by shouting instead of walking around, some by having someone else run the whole warehouse for you.

The practical rule most teams land on: start with plain Prometheus for anything Kubernetes-native, add Alertmanager and Grafana as the paging and dashboarding layer from day one, and only reach for Thanos, Mimir, Cortex, VictoriaMetrics, or a managed service once retention needs, query-federation needs, or multi-tenant scale actually arrive — not preemptively. Nearly every production Prometheus deployment past a certain size ends up running some form of the long-term-storage layer this page describes; the question is rarely whether, only when.

Going further

☺ Like you're 10: This page is enough to read a real config and a real PromQL query — the official docs are where you go to become genuinely fluent.

The canonical source is the documentation at prometheus.io/docs — the querying basics and functions pages are worth reading end to end once this page's shape is familiar — alongside the source at github.com/prometheus/prometheus and the CNCF project page at cncf.io/projects/prometheus. If you're preparing for a vendor-neutral credential built specifically around this tool, see Prometheus Certified Associate (PCA) — verify current exam format and pricing on the Linux Foundation's own page before registering, the way this course's other certification pages recommend. Pair this page with monitoring & observability for the signals Prometheus is usually collecting, and multi-window, multi-burn-rate alerting for the fullest worked PromQL example this course has to offer.