Prometheus

Prometheus is the tool most people mean when they say "metrics" in a cloud-native stack: a single self-contained binary that reaches out and pulls numbers from every system you point it at on a schedule, stores them in its own time-series database, lets you query and combine them with a purpose-built language called PromQL, and evaluates alerting rules against that same data continuously. It has no required database, no required message broker, and no vendor to sign up with — which, alongside its enormous exporter ecosystem, is most of why it became the de facto open-source metrics standard and the data source Grafana most commonly visualizes. Monitoring & observability covered the tool-neutral shape of this problem — what a metric is good for, why it's cheap and pre-aggregated; this page goes deep on the specific tool: the pull model and why it was a deliberate design choice, PromQL from a bare selector to a working alert, the exporter pattern, and the failure modes — cardinality chief among them — that separate a Prometheus deployment that scales from one that gets OOM-killed by its own success.

What Prometheus is and the problem it solves

☺ Like you're 10: Before Prometheus, every company built its own private metrics tool from scratch — Prometheus gave everyone the same free one, and it caught on because it was genuinely good, not because a vendor pushed it.

Prometheus was built in 2012 at SoundCloud by Julius Volz and Matt T. Proud, explicitly modeled on Google's internal monitoring system Borgmon — the same lineage that later produced Google's SRE practice and the golden-signals framing this course covers in monitoring & observability. SoundCloud open-sourced it in 2015, and it became the second project the Cloud Native Computing Foundation ever accepted, graduating in 2018 right behind Kubernetes itself — not a coincidence, since Prometheus's dynamic service discovery was built for exactly the kind of environment where containers appear and disappear on their own schedule, not a fixed inventory of long-lived hosts.

What Prometheus actually is: one Go binary containing a scrape engine, a purpose-built time-series database, an HTTP API and PromQL query engine, and a rule evaluator for alerts and precomputed aggregations — all running as a single process with no required external dependency. That's a deliberate design choice worth sitting with: you can download one binary, point it at a config file, and have working metrics collection, storage, querying, and alerting in the time it takes to run docker run. Nothing else in this stack is mandatory to get real value; Grafana, Alertmanager-driven paging, and long-term remote storage are all things you add on top, not things you need on day one.

The pull-based model and architecture

☺ Like you're 10: Prometheus doesn't wait for anyone to mail it numbers — it walks around and asks for them itself, on its own clock, so a silence is instantly meaningful instead of ambiguous.

Every target Prometheus scrapes exposes a plain-text HTTP endpoint — by convention /metrics — listing its current metric values. Prometheus's scrape loop issues a plain GET against that endpoint on an interval it controls (scrape_interval, commonly 15s or 30s), parses the response, and stores every sample with the timestamp of the scrape itself. Nothing is ever pushed to Prometheus uninvited. That single design choice, "pull instead of push," produces several practical wins that are easy to undervalue until you've operated the alternative: a target that stops responding is unambiguous, because Prometheus generates its own synthetic up{job="...",instance="..."} metric — 1 for a successful scrape, 0 for a failed one — so "the service went quiet" and "the service is fine and just has nothing new to say" are never confused the way they can be with a push-based agent that simply stops sending; Prometheus alone controls its own ingestion rate, so it can never be knocked over by a fleet of misbehaving clients all pushing at once; and you can run Prometheus straight from a laptop against a service in a private network with zero inbound firewall changes on the target side, because the target only ever answers a request, it never has to know where to send anything.

The data model: metrics, labels, and the four types

☺ Like you're 10: Every number Prometheus stores comes with a name and a set of little tags describing it — and there are exactly four shapes a number can come in, each one telling you it can be used differently.

A Prometheus time series is identified by a metric name plus a set of key-value labels — http_requests_total{job="checkout", method="GET", status="200"} is one distinct series; change any label value and you get a different series entirely. Every unique combination of label values is its own independent stream of timestamped samples, which is the whole basis of PromQL's aggregation — and, as covered below, the whole basis of Prometheus's most common outage.

A counter only makes sense as a rate of change — the raw value is just "how many since the process started," which is meaningless on its own and actively misleading to graph directly, since a process restart resets it to zero and looks like traffic cratered. A histogram's buckets are cumulative — le="0.5" ("less than or equal to 0.5 seconds") counts every observation that landed in that bucket or below it, up through a final le="+Inf" bucket that always equals the total count — which is exactly what lets histogram_quantile() reconstruct an approximate percentile after the fact, and what lets a histogram, unlike a summary, be summed across every replica of a service before that percentile is computed. That last distinction is the practical reason most teams default to histograms for latency: you almost always want "p99 across the whole fleet," and only a histogram can answer that honestly.

PromQL: querying the time series

☺ Like you're 10: PromQL is a small language for asking "how much, over what stretch of time, sliced which way" — a handful of pieces combine into almost every question you'll ever ask it.

A bare metric name with a label selector — http_requests_total{job="checkout", status=~"5.."} — is an instant vector: one current value per matching series (=~ is a regex match, != and !~ negate). Append a duration in square brackets and it becomes a range vector — http_requests_total{job="checkout"}[5m] is every sample from each matching series over the last five minutes, which exists almost entirely to be fed into a function. rate() is the one you'll type the most: it computes the per-second average rate of increase of a counter over the range, correctly handling counter resets from restarts along the way, and it's what turns a monotonically climbing total into a graphable, alertable number. irate() is its high-resolution sibling — the rate between only the last two data points, better for volatile graphs, worse for alerting because a single noisy sample can trigger it.

# the canonical error-rate query — errors as a fraction of total requests, sliced per route
sum(rate(http_requests_total{job="checkout", status=~"5.."}[5m])) by (route)
/
sum(rate(http_requests_total{job="checkout"}[5m])) by (route)

# p99 latency from a histogram, across every replica of the service
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket{job="checkout"}[5m])) by (le)
)

# top 5 busiest routes right now
topk(5, sum(rate(http_requests_total[5m])) by (route))

# capacity planning: will this disk fill up in the next 4 hours at the current trend?
predict_linear(node_filesystem_free_bytes{mountpoint="/"}[6h], 4 * 3600) < 0

Aggregation operators — sum, avg, min, max, count, topk, bottomk — collapse many series into fewer, and the by (...) / without (...) clause controls which labels survive that collapse; get the clause wrong and a query either loses the dimension you needed (everything mashed into one line) or keeps a dimension so granular the aggregation didn't actually aggregate anything. A query that gets expensive to run on every dashboard refresh — a heavy histogram_quantile over a long range, evaluated by many viewers — is exactly what a recording rule is for: precompute it on a schedule and save the result as a new, ordinary time series under its own name, so dashboards and alerts read a cheap pre-aggregated number instead of recomputing the expensive query every time. Recording rules following the community's level:metric:operations naming convention are also the standard mechanism behind SLO burn-rate math, covered in depth in SLOs, Error Budgets & Toil.

# recording_rules/checkout.yml
groups:
  - name: checkout.recording
    interval: 30s
    rules:
      - record: job:http_errors:ratio5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
          /
          sum(rate(http_requests_total[5m])) by (job)

Alerting rules and Alertmanager

☺ Like you're 10: Prometheus decides when something's wrong; a separate program called Alertmanager decides who actually hears about it, and makes sure five related alarms don't turn into five separate phone calls.

An alerting rule is a PromQL expression Prometheus itself evaluates on a schedule; whenever it returns a non-empty result, the alert transitions from inactive to pending, and only fires as firing once the condition has held continuously for the rule's for duration — the single most important flap-suppression mechanism in the whole system, because it means one noisy sample doesn't page anyone.

# alerts/checkout.yml
groups:
  - name: checkout.alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{job="checkout", status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="checkout"}[5m])) > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "Checkout error rate above 5% for 10m"
          runbook: "https://runbooks.acme.internal/checkout-error-rate"

Prometheus itself never sends a page. Firing alerts are handed to Alertmanager, a separate process whose entire job is what happens between "this condition is true" and "the right human finds out": grouping combines related alerts (e.g. every route breaching at once during a real outage) into a single notification instead of a flood; routing sends different severity or team labels to different receivers — a page label straight to PagerDuty or Opsgenie, a warn label to a Slack channel nobody's asleep waiting on; silencing mutes a known, expected alert (a maintenance window) without touching the underlying rule; and inhibition suppresses a downstream symptom alert when its known root-cause alert is already firing, so a database outage doesn't also page five separate teams for every service that depends on it.

# alertmanager.yml
route:
  receiver: default-slack
  group_by: [alertname, cluster]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match: { severity: page }
      receiver: pagerduty-oncall

receivers:
  - name: default-slack
    slack_configs:
      - channel: "#checkout-alerts"
  - name: pagerduty-oncall
    pagerduty_configs:
      - routing_key: "${PAGERDUTY_ROUTING_KEY}"

inhibit_rules:
  - source_match: { severity: critical }
    target_match: { severity: warning }
    equal: [alertname, cluster]

What fires and how it's routed is a detection-and-plumbing question; what a human actually does once paged is a separate discipline covered in full in incident management — Prometheus and Alertmanager get you to "the right person's phone is buzzing," not past it.

The exporter ecosystem

☺ Like you're 10: An exporter is a small translator that sits in front of something that doesn't natively speak Prometheus's language and converts it into a page Prometheus knows how to read.

Most software doesn't expose a Prometheus-shaped /metrics endpoint on its own, especially software you didn't write. An exporter is a small, purpose-built process that queries a system some other way and re-exposes what it finds in Prometheus's plain-text exposition format. A handful cover the overwhelming majority of real infrastructure:

The distinction worth remembering: cAdvisor answers "what is this container doing" (CPU, memory, network I/O), kube-state-metrics answers "what does the Kubernetes API server think the state of this object should be" (a Deployment's spec.replicas vs. status.availableReplicas) — a service that looks perfectly healthy on cAdvisor's resource graphs can still be failing its rollout, which is exactly what kube-state-metrics catches and cAdvisor structurally cannot. Instrumenting your own code directly with a client library is preferred over an exporter wherever you can, since a client library can expose application-specific detail — a queue depth, a cache-hit ratio, a business metric like orders processed — that no generic exporter could ever infer from the outside:

from prometheus_client import Counter, Histogram, start_http_server

REQUEST_COUNT = Counter(
    "http_requests_total", "Total HTTP requests", ["method", "status"]
)
REQUEST_LATENCY = Histogram(
    "http_request_duration_seconds", "Request latency in seconds"
)

@REQUEST_LATENCY.time()
def handle_request(method):
    # ... handle the request ...
    REQUEST_COUNT.labels(method=method, status="200").inc()

start_http_server(8000)  # now scrapeable at :8000/metrics

On Kubernetes specifically, hand-editing scrape_configs for every new service doesn't scale, which is what the Prometheus Operator (installed as part of the kube-prometheus-stack Helm chart) exists to fix: a ServiceMonitor custom resource declares which Service's endpoints should be scraped, and the Operator reconciles that declaration into Prometheus's live scrape configuration automatically — the same declarative, reconciler-driven pattern covered generally in infrastructure as code, applied to scrape targets instead of servers.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: checkout
  labels:
    release: kube-prometheus-stack   # must match the Operator's selector
spec:
  selector:
    matchLabels: { app: checkout }
  endpoints:
    - port: metrics
      interval: 15s

Day-to-day: config, promtool, and running it

☺ Like you're 10: One config file lists what to scrape and where the alert rules live; one command-line tool checks both for mistakes before you ever restart anything.

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: prod-use1

rule_files:
  - "alerts/*.yml"
  - "recording_rules/*.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

scrape_configs:
  - job_name: "node"
    static_configs:
      - targets: ["10.0.1.11:9100", "10.0.1.12:9100"]

  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: "(.+)"

# quick local run
$ docker run -d --name prometheus -p 9090:9090 \
    -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
    prom/prometheus

# validate before you ever restart a running server
$ promtool check config prometheus.yml
$ promtool check rules alerts/checkout.yml

# query from the CLI, same engine the web UI and Grafana use
$ promtool query instant http://localhost:9090 'up{job="checkout"}'
$ curl -s localhost:9090/api/v1/query --data-urlencode 'query=up' | jq

# read a target's raw exposition output directly — the same text Prometheus parses
$ curl -s localhost:9100/metrics | grep node_cpu_seconds_total | head

# reload scrape configs and rule files without a restart (needs --web.enable-lifecycle)
$ curl -X POST http://localhost:9090/-/reload

Gotchas and failure modes

☺ Like you're 10: The mistakes that hurt aren't exotic — they're "I added one more label" and "I only ever ran one copy of it," and both take weeks to show up as a real problem.

Summary quantiles don't aggregate. Averaging two instances' reported p99 values, or worse, averaging their p99s together and calling the result a fleet-wide p99, is mathematically meaningless — a percentile computed locally on each instance's own sample set says nothing about the distribution across the whole fleet combined. This is exactly why the data-model section above steers toward histograms for anything you intend to aggregate across replicas: a histogram's raw bucket counts can be honestly summed first, and the quantile computed only once, on the true combined distribution.

No built-in high availability. A single Prometheus server is a single point of failure for both scraping and alerting — if it goes down, that window of history is simply gone, and no alert fires while it's down. The common pattern is running two identical Prometheus replicas scraping the same targets independently; that buys redundancy against one instance dying, but the two are not deduplicated against each other, so a dashboard or an alerting layer needs to know it's looking at (or Alertmanager needs to be told to deduplicate) two copies of the same signal. Genuine deduplication, a long-term retention window past the local default of roughly 15 days, and a single global query view across many clusters is what Thanos, Cortex, Mimir, and VictoriaMetrics add on top via Prometheus's remote_write — none of them replace Prometheus, they all still rely on a Prometheus server (or a lightweight agent mode of one) doing the actual scraping.

Staleness and restarts. If a target vanishes — a Kubernetes pod gets rescheduled under a new name — Prometheus marks its series stale after roughly five minutes of missed scrapes rather than showing a flat, misleadingly "last known value" line forever; a dashboard that looks flat instead of gapped for longer than that is usually querying a metric name that no active target is producing under the labels you expect, not a metric genuinely holding steady.

Alternatives and when to choose it

☺ Like you're 10: Other tools measure the same kind of thing — they just trade Prometheus's "you run it, you own it" model for someone else running it, at a price.

On AWS specifically, this is a real head-to-head rather than a hypothetical one — Monitoring & Logging covers CloudWatch's push-based, fully-managed model in exam depth, and it competes with Prometheus directly for the same job on the same cloud. AWS's own answer to that competition is Amazon Managed Service for Prometheus (AMP): a managed, horizontally-scaled, Prometheus-compatible remote-write target, so you keep the pull-based scraping, PromQL, and exporter ecosystem covered on this page while AWS operates the storage tier and durability. Most teams don't pick one exclusively — CloudWatch stays the default for native AWS service metrics (an ALB's request count, an RDS instance's CPU) that AWS already publishes for free, while Prometheus (self-hosted or via AMP) covers Kubernetes and any workload with the CNCF exporter ecosystem already built for it, and Grafana or Amazon Managed Grafana queries both as data sources side by side on the same dashboard.

Prometheus deliberately does not do everything: it has no logs — see the ELK Stack for that half of observability — and no distributed tracing of its own, covered tool-neutrally in Distributed Tracing & Telemetry. Put the scrape-and-alert patterns from this page into practice in Capstone Part 4 — Observability, then tune real alerting thresholds against a noisy signal in Drill — Set Up Meaningful Alerts. If you want a credential to go with the material, the Linux Foundation's Prometheus Certified Associate (PCA) covers this exact scope — verify its current format and pricing on the Linux Foundation's own site before planning around anything printed here.