Prometheus

The PCA blueprint spends 20% of its weight on Prometheus Fundamentals — Configuration and Scraping, Data Model and Labels, and a named competency called Understanding Prometheus Limitations — and this page is that domain turned into a config file you can actually write. The Prometheus model already covers why Prometheus pulls and why it labels instead of nesting a hierarchy; PCA — the exam already covers PromQL and the four metric types. Neither one hands you a working prometheus.yml. This page does: the scrape loop and its local TSDB, the scrape_configs block and the service-discovery mechanisms that feed it, the two relabeling stages that decide first who gets scraped and then what gets kept, a recording rule paired with an alerting rule the way rule_files actually stores them, federation versus remote_write as two genuinely different ways to move data off one server, the promtool/HTTP-API commands worth having in muscle memory, and the failure modes that catch people who only ever ran the happy path. On a Kubernetes platform this same configuration usually gets generated by the Prometheus Operator's CRDs rather than hand-written — Platform Engineering's Prometheus page covers that layer; PCA examines the config underneath it, so that's where this page stays.

Architecture: the loop, in the order a scrape actually happens

☺ Like you're 10: One program does the same four things on repeat, forever: find out who's out there, ask them for numbers, write the numbers down, then check if any of them mean trouble.

Prometheus is a single Go binary with no required external dependency — no database to install first, no message broker, nothing else that has to be up for it to start doing its job. On a schedule set by global.scrape_interval, it re-derives its target list from whatever service-discovery mechanisms are configured, applies relabel_configs to decide which of those targets actually get scraped at all, issues an HTTP GET against each survivor's /metrics endpoint, applies metric_relabel_configs to decide which of the returned series actually get kept, and appends whatever's left to a local TSDB. Separately, on global.evaluation_interval, it evaluates every group in rule_files against that same TSDB, writing new series for recording rules and pushing anything that fires to Alertmanager. Recent samples live in an in-memory head block protected by a write-ahead log; roughly every two hours the head compacts into an immutable on-disk block, older blocks merge, and anything past --storage.tsdb.retention.time (default 15 days) or --storage.tsdb.retention.size gets deleted. Nothing about this loop involves a second server — that single-binary, dependency-free design is deliberate, and it's exactly what makes horizontal scale, clustering and long retention someone else's job, covered further down.

Scrape configs and service discovery

☺ Like you're 10: One file lists every kind of place numbers might come from — a fixed address, a directory that updates itself, or a plain list in another file.

Every job Prometheus scrapes lives under scrape_configs, and every job needs some way to find its targets. static_configs is a hand-typed list — fine for a handful of stable addresses, painful at any real scale. kubernetes_sd_configs asks the API server directly for a role — pod, service, endpoints, endpointslice, node or ingress — and returns a live, continuously-refreshed list decorated with metadata labels like __meta_kubernetes_pod_annotation_* and __meta_kubernetes_namespace, which is what the relabeling stage below reads from. file_sd_configs watches one or more JSON or YAML files on disk and reloads automatically when they change — the escape hatch for anything that isn't Kubernetes and isn't static, from a Consul export to a config-management job that writes a fresh target list nightly. dns_sd_configs, consul_sd_configs and cloud-specific mechanisms (ec2_sd_configs and similar) round out the list for everything else.

global:
  scrape_interval: 15s        # default for every job below, unless a job overrides it
  scrape_timeout: 10s         # MUST be <= scrape_interval — see the gotcha further down
  evaluation_interval: 15s    # how often rule_files gets re-checked
  external_labels:            # stamped onto every series this server ships out —
    cluster: eu-west-1-prod   # federation and remote_write both rely on this
    replica: a

rule_files:
  - /etc/prometheus/rules/*.yaml

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["10.0.1.11:9100", "10.0.1.12:9100"]
        labels: { env: prod }

  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod              # also: service, endpoints, endpointslice, node, ingress
    relabel_configs:           # PRE-scrape — who actually gets asked for numbers
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"          # opt-in: only pods carrying this annotation get scraped
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        regex: (.+)
        target_label: __metrics_path__   # override the default /metrics path
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace          # promote a meta label to a real one
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app

  - job_name: legacy-hosts
    file_sd_configs:
      - files: ["/etc/prometheus/targets/*.json"]
        refresh_interval: 5m

Every __meta_* label that service discovery produces is scraped away before storage unless a relabel_configs rule explicitly promotes it — that's the point of the namespace and app lines above: without them, __meta_kubernetes_namespace and __meta_kubernetes_pod_label_app would simply vanish, and every series scraped this way would carry no namespace or app label at all.

Relabeling: filters and rewrites, before and after the scrape

☺ Like you're 10: The same short list of instructions — keep this, drop that, rename the other thing — gets used twice, once to pick who answers the door and once to pick which answers were actually worth writing down.

Both relabeling stages share one action vocabulary, applied against a source_labels list joined by a separator and matched with regex. keep and drop decide whether something survives at all; replace writes a new value into target_label, usually built from a regex capture group; labelmap copies every label matching a pattern to a new name, typically used to promote a whole family of __meta_kubernetes_* labels at once; labeldrop and labelkeep remove or retain labels by name rather than by value; and hashmod writes a deterministic hash of the source into a temporary label, which — paired with a keep on a specific remainder — is the standard way to shard one enormous job's scrape load across several Prometheus servers.

# Shard a huge job across 4 servers — this instance keeps only bucket 0
relabel_configs:
  - source_labels: [__address__]
    modulus: 4
    target_label: __tmp_hash
    action: hashmod
  - source_labels: [__tmp_hash]
    regex: "0"          # each of the 4 servers runs the same rule with a different regex
    action: keep

# metric_relabel_configs — POST-scrape, on the SAME job — trims what actually lands in the TSDB
metric_relabel_configs:
  - source_labels: [__name__]
    regex: "go_gc_duration_seconds.*"
    action: drop                     # a whole metric family nobody queries
  - source_labels: [request_id]
    action: labeldrop                # a label that should never have existed at all

The distinction that actually matters operationally: a relabel_configs rule that drops a target means Prometheus never even sends the HTTP request — cheapest possible fix for a target you don't want at all. A metric_relabel_configs rule that drops a series still pays the full network and parse cost of the scrape; it only saves the storage and query cost afterward. If an exporter is expensive to scrape and half its output is unwanted, filtering it with params or a dedicated low-noise endpoint beats dropping it after the fact every time.

Recording and alerting rules

☺ Like you're 10: One kind of rule does the expensive math once and remembers the answer; the other kind watches an answer and decides whether it's bad enough to wake somebody.

Rules live in plain YAML files named by rule_files — not inside prometheus.yml itself — organized into named groups, each with its own optional interval overriding evaluation_interval. A recording rule (record + expr) computes an expression on that schedule and stores the result as a brand-new series, conventionally named level:metric:operations, so dashboards and alerts query the cheap pre-computed answer instead of re-running an expensive rate()-and-sum() every time. An alerting rule (alert + expr) fires whenever its expression returns any series at all, and for: requires that to stay true continuously across that many evaluation cycles before the alert actually moves from pending to firing — the mechanism that kills one-evaluation blips before they page anyone.

# /etc/prometheus/rules/checkout.yaml
groups:
  - name: checkout.rules
    interval: 30s
    rules:
      # RECORDING rule — compute once, query everywhere, cheaply
      - record: job:http_request_errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
            / sum by (job) (rate(http_requests_total[5m]))

      # ALERTING rule — a SYMPTOM, built on the recording rule above
      - alert: CheckoutHighErrorRate
        expr: job:http_request_errors:ratio_rate5m{job="checkout"} > 0.05
        for: 10m                     # must hold for 10 straight minutes before firing
        labels:
          severity: page             # Alertmanager routes on labels, not on the expr
        annotations:
          summary: "Checkout is serving {{ $value | humanizePercentage }} errors"
          runbook_url: "https://runbooks.example.com/checkout-high-error-rate"

Structuring an alert this way — recording rule first, alert expression second — isn't just tidy, it's also what keeps for: cheap: re-evaluating a pre-aggregated series thirty times over ten minutes costs nothing next to re-running sum(rate(...)) over raw series thirty times. Alertmanager itself — grouping, silencing, inhibition, routing to a receiver — is a separate binary and a separate config file; this page stays on the Prometheus side of that boundary, where the Alerting & Dashboarding domain of the PCA blueprint picks it back up.

Federation and remote_write: two very different ways to get data out

☺ Like you're 10: One is a monthly summary folder someone asks for on purpose. The other is a truck that drives away with every single thing collected, all day, automatically.

Federation is one Prometheus server scraping another Prometheus server's /federate endpoint — it's the pull model applied one level up. A match[] parameter tells the source server exactly which series to return, and a global or regional server typically uses it to pull only already-aggregated series — recording-rule output, not raw per-instance data — from each cluster below it. remote_write is the opposite direction and the opposite shape: the local server pushes every sample, as it's scraped, continuously, to an external HTTP endpoint, batched and compressed and queued for retry. Federation is Prometheus's own scrape mechanism reused for hierarchical rollups of a small, deliberate slice of data; remote_write is a purpose-built streaming pipeline for shipping everything to long-term or multi-cluster storage — Thanos, Cortex, Mimir, VictoriaMetrics, or a managed vendor.

# On a GLOBAL Prometheus — federation, pulling ONLY aggregated series
scrape_configs:
  - job_name: federate
    honor_labels: true            # trust the SOURCE server's labels — see the gotcha below
    metrics_path: /federate
    params:
      match[]:
        - '{__name__=~"job:.*"}'  # recording-rule output ONLY — never raw series
        - 'up'
    static_configs:
      - targets: ["prometheus-eu-west:9090", "prometheus-us-east:9090"]

# On a LOCAL Prometheus — remote_write, shipping EVERY sample onward
remote_write:
  - url: https://thanos-receive.mission.internal/api/v1/receive
    queue_config:
      capacity: 10000              # samples buffered per shard before it blocks
      max_shards: 30                # parallel send workers
      max_samples_per_send: 2000
    write_relabel_configs:          # filter what actually leaves, same vocabulary as before
      - source_labels: [__name__]
        regex: "go_.*"
        action: drop
    basic_auth:
      username: prometheus
      password_file: /etc/prometheus/secrets/remote-write-password

remote_read:
  - url: https://thanos-query.mission.internal/api/v1/read
    read_recent: false              # let local TSDB answer recent queries; reach out only for old data

Which brings up limitations worth stating plainly, because PCA names them as their own competency. Prometheus's own documentation is explicit that it is not a long-term store — that's what remote_write plus an external system exists to solve; not clustered or replicated — two servers scraping the same targets are two independent, slightly-disagreeing copies, not one highly-available one; not billing-grade — it's a sampling system, and a scrape reads a counter's state at one instant rather than recording every event that moved it; and federation specifically does not scale to full data export — pulling every raw series through /federate instead of a deliberately aggregated slice is a documented anti-pattern that degrades the source server under load for no benefit remote_write wouldn't already give you for free. The Prometheus model covers why these limits are structural rather than configuration mistakes; this page is where you act on that fact.

Day-to-day commands

☺ Like you're 10: Mostly you check your files are spelled right before you commit them, then ask the running server questions through its web address instead of guessing.

# Validate BEFORE you ship — promtool ships in the Prometheus image, put it in CI
$ promtool check config /etc/prometheus/prometheus.yml
$ promtool check rules  /etc/prometheus/rules/*.yaml
$ promtool test rules   tests/checkout_test.yaml        # unit-test alerts on synthetic data
$ promtool check metrics < scraped.txt                  # lint a raw exposition payload

# Reload config/rules WITHOUT restarting — needs --web.enable-lifecycle at startup
$ curl -X POST http://localhost:9090/-/reload
$ kill -HUP $(pgrep prometheus)              # the other way in, no flag required

# The HTTP API — script it, or just open it in a browser
$ curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=up==0'
$ curl -s 'http://localhost:9090/api/v1/targets?state=active' | jq '.data.activeTargets[].health'
$ curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName[:5]'
$ curl -s http://localhost:9090/-/healthy ; curl -s http://localhost:9090/-/ready

# The UI pages worth knowing by URL, not just by clicking around
#   /targets      <- is my target UP, and what was the scrape error if not?
#   /config       <- what did prometheus.yml actually parse into?
#   /rules        <- is my recording rule producing anything?
#   /alerts       <- inactive / pending / firing, per alert
#   /tsdb-status  <- top label and metric-name cardinality offenders, right now

Gotchas that bite in production

☺ Like you're 10: Most surprises come from one setting quietly disagreeing with another, or from a safety feature doing exactly what it was built to do in a moment you didn't expect it.

scrape_timeout longer than scrape_interval is a config error, not a suggestion

Prometheus validates this relationship at load time — a job whose scrape_timeout exceeds its scrape_interval fails promtool check config outright, because a scrape that's still allowed to be running when the next one starts would need overlapping in-flight requests the design doesn't support. The fix is never "give it more time"; it's either shortening the timeout or lengthening the interval, and the honest question underneath both is why a single scrape is slow enough to need either.

honor_labels: true quietly lets the target win label conflicts

By default, if a scraped exporter exposes a label that collides with one Prometheus would add itself — job, instance, anything set via relabel_configs — Prometheus renames the exporter's version with an exported_ prefix and keeps its own. honor_labels: true flips that: the target's label wins outright, and Prometheus's own gets dropped instead. It's the correct setting for federation, precisely because you want the source server's job and instance to survive the hop untouched — but set it on an ordinary application scrape and a misbehaving exporter can silently overwrite instance for every series it sends, making two genuinely different pods look like the same one in every query.

Rare, low-traffic counters make rate() look like zero traffic

A counter that increments once an hour, queried with rate(x[5m]), will show 0 for almost every window that doesn't happen to contain the increment — which looks exactly like "no traffic" rather than "traffic exists but is sparse." increase() over a window long enough to reliably contain at least one event is the fix for anything that fires that rarely; a five-minute rate window was never built for an hourly event.

Cardinality has no built-in ceiling, and sharding hides the total instead of shrinking it

Every unique combination of label values is its own series with its own memory footprint, and nothing in the data model stops a single unbounded label — a raw user ID, a full request path — from multiplying series count by orders of magnitude the instant real traffic hits it. Sharding a job across servers with hashmod spreads that memory pressure around several machines; it does not reduce how many series exist across the fleet. The actual fix is removing the offending label at the instrumentation source — metric_relabel_configs with labeldrop is the emergency brake after the fact, not the repair.

Prometheus vs the alternatives

☺ Like you're 10: A few different notebooks exist for keeping numbers at bigger scale. Most of them copy Prometheus's handwriting on purpose, so switching later doesn't mean relearning everything.

The pattern worth internalizing: nearly every alternative here speaks Prometheus's own exposition format and remote_write protocol rather than inventing a competing one — a point the Prometheus model covers from the other direction, as the reason this specific combination of decisions became cloud native's default rather than one option among equals. Start with plain Prometheus per cluster, reach for federation only to roll up a small, deliberate slice of aggregated data, and reach for remote_write plus Thanos, Mimir or VictoriaMetrics the day a real requirement — a compliance retention window, one dashboard across six clusters — actually shows up. Don't start with the distributed system on day one.