The Prometheus Model
The PCA blueprint hands you Prometheus already decided: learn PromQL, the four metric types, an alerting rule, and you're tested on using them correctly. It never asks why Prometheus is built the specific way it's built — why, of every metrics system that existed when it was created, this exact combination of decisions is the one nearly every cloud native project now ships a /metrics endpoint for by default. That's this page. Reduce Prometheus to two independent bets and almost everything else follows: it pulls metrics from targets instead of having targets push them, and it stores every metric as a name plus an open-ended set of labels instead of a fixed position in a dotted hierarchy. Neither was the obvious choice at the time. Both paid off for mechanical, traceable reasons — and both create real architectural ceilings that an associate exam has no reason to test, but that anyone actually running Prometheus in anger eventually hits.
Picture two ways to check on campers scattered across a summer camp. In the first, every camper is trusted to run to the office and shout "I'm fine!" whenever they feel like it — and if one camper falls asleep and never shouts, the office has no way to tell "sleeping" apart from "never existed." In the second, a counselor walks to every single campsite, in order, every fifteen minutes, and looks. If a campsite doesn't answer, the counselor knows immediately and specifically which one has a problem — that's Prometheus's pull model. Now picture how the counselor writes each check down: not one messy sentence per camper, but a little card with tags — which cabin, which camper, which activity — so later you can ask "how many campers in Cabin 3 were cold last night" without redesigning the whole notebook. That card-with-tags habit is Prometheus's labelled data model, and it's the second half of the whole idea.
Two decisions, not one tool
☺ Like you're 10: "Prometheus" sounds like one clever idea. It's actually two separate bets that happened to be made by the same project — and either one could have gone the other way.
In 2012, when Prometheus was created at SoundCloud by engineers who had previously worked at Google, at least two other schools of monitoring were already established and popular: push-based agents that phoned metrics home on their own schedule (StatsD's model), and metric names arranged in a fixed, dotted hierarchy that the operator designed up front (Graphite's model). Prometheus's design — reportedly influenced by Google's internal Borgmon system — rejected both defaults at once. Split the two decisions apart, because they're genuinely independent and each one is worth reasoning about on its own: pull instead of push is a decision about who initiates a metrics exchange and when. Labels instead of a hierarchy is a decision about how a single data point is described. You could build a system with either one alone. Prometheus made both bets, and the combination is what the rest of this page is actually about.
A pull model wasn't the obvious choice
☺ Like you're 10: If the counselor walks the rounds instead of waiting to be shouted at, a silent campsite is itself the alarm — no separate "I'm still alive" signal needed.
In Prometheus, the server initiates every exchange: on a schedule, using service discovery to find out what currently exists, it issues an HTTP GET against each target's /metrics endpoint and reads back whatever that target has to say for itself at that instant. The target does nothing but sit there and answer when asked. That single design choice buys three things that are easy to take for granted once you've only ever used Prometheus:
- "Is this thing alive" comes free. A scrape either succeeds within its timeout or it doesn't. If it doesn't, Prometheus writes
up{job="...", instance="..."} 0to the very same time series database the metric itself would have landed in — no separate heartbeat protocol, no dead-man's-switch service, no agent-side retry queue to keep a push channel alive. Liveness is a byproduct of the collection mechanism itself, not a feature someone had to build on top of it. - The server, not thousands of clients, owns collection policy. Scrape interval, timeout, and which targets exist at all are configured in one place and can change without touching a single instrumented service. A push-based fleet has to get that same policy right independently, on every client, or ship a change to all of them at once.
- There's no fan-in stampede to defend against. A push system's receiver has to survive every client deciding, on its own clock, to send data at once — a genuinely hard availability problem at scale. A puller controls its own pace; it can throttle, shard, or back off without asking anyone's permission.
None of that makes pull unconditionally correct — it's a trade, and the trade shows up as friction in two specific places. First, the collector has to be able to reach every target directly, which fights NAT boundaries, egress-only networks, and anything that doesn't hold still long enough to be scraped — which is exactly why Prometheus ships one narrow, deliberate exception: the Pushgateway, for short-lived batch jobs that would otherwise die before a scrape could ever reach them. Second, exposing a /metrics endpoint is itself a small attack surface and an information-disclosure question someone has to own. Pull is a bet that "the collector can always reach the target, and the target is willing to be asked" is true for the overwhelming majority of workloads — which, inside a Kubernetes cluster where service discovery already knows about everything, it usually is.
The labelled model: a dimension cube, not a filing tree
☺ Like you're 10: A dotted name is like one long address you have to get exactly right when you first write it down. A card with tags lets you ask new questions about the same information later, without redoing the filing.
Graphite's dotted hierarchy — stats.prod.checkout.us-east.pod-7.http.requests.count — bakes every dimension into a single, ordered path chosen at instrumentation time. It works, and it's genuinely simple to reason about. But every dimension you might ever want to slice by — environment, service, region, pod — has to be a path segment decided in advance, in a fixed order, because a wildcard query can only match the shape the tree was actually built in. Ask for "every region, regardless of pod" and it works only if pod happened to come after region in the path you chose two years ago. Prometheus instead stores one metric name — http_requests_total — attached to an open, unordered set of labels: {env="prod", service="checkout", region="us-east", pod="pod-7"}. Every unique combination of label values is its own time series, and PromQL can group, filter, or aggregate along any label, in any combination, chosen at query time — not baked in at write time. The data model is really a multi-dimensional cube: pick any subset of its axes to collapse or slice, after the fact, without touching how the metric was instrumented.
This is also where the SLI/SLO/SLA vocabulary lives structurally rather than as three memorized terms. Measuring an SLI is nothing more exotic than a PromQL ratio over this same labelled data — usually a "good events over total events" recording rule. The SLO is the target ratio a team commits to over a window; the SLA is the contractual version with consequences attached; and the error budget — the gap between the SLO and perfect reliability — only makes sense as a number you can compute per service, per region, per environment, because the underlying data was never collapsed into one global count that couldn't be pulled back apart. A fixed hierarchy could compute a global success ratio too. It's the dimensional model that lets a team ask "what's checkout's error budget in us-east this week" without a special-purpose metric having been invented for exactly that slice in advance.
Why this combination became the de facto standard
☺ Like you're 10: Once nearly everyone was already speaking the same little language for describing numbers, other tools found it easier to learn that language than to invent their own.
Prometheus joined the CNCF in 2016 as its second hosted project, right behind Kubernetes, and graduated in 2018 — a timing accident with enormous consequences, because it tied Prometheus's adoption curve directly to Kubernetes's own. Kubernetes shipped cAdvisor and kube-state-metrics speaking Prometheus's exposition format from early on, and once the ecosystem's default scraper spoke one specific text format, exposing metrics in that exact format became the path of least resistance for practically every CNCF project that came after — independent of whether anyone involved ever ran the Prometheus server itself. That's the actual mechanism behind "de facto standard," and it kept compounding in ways worth naming individually:
- OpenMetrics, a CNCF sandbox specification, formalized Prometheus's plain-text exposition format into a standalone, vendor-neutral standard that other systems could target without depending on Prometheus at all.
- remote_write — the protocol Prometheus uses to forward samples to an external store — became the common ingestion API that Thanos, Cortex, Mimir and VictoriaMetrics all standardized on, rather than each inventing a bespoke one.
- The OpenTelemetry Collector ships a Prometheus receiver and a Prometheus/remote-write exporter as first-class components, letting OTel-instrumented systems cross into the Prometheus world and back — see the OpenTelemetry data model for the other side of that bridge.
- Commercial observability vendors routinely support "just point your existing
/metricsendpoints at us" as an onboarding path, specifically because so much software already emits that exact format with zero extra integration work.
The exposition format won even in places the Prometheus server didn't. Prometheus-the-server is one implementation choice among several today; the labelled text format plus remote_write became the assumed interchange contract that lets a team swap the storage backend — Prometheus itself, Thanos, Mimir, Cortex, VictoriaMetrics, a managed vendor — without touching a single instrumented service. That's a stronger, stickier kind of standard than "the most popular tool": it's a protocol nobody has to agree to use on purpose anymore, because everything around it already does.
The limitations that follow from the same two bets — beyond what PCA asks
☺ Like you're 10: Every choice that makes something good at one job makes it structurally bad at a different job. Knowing which job Prometheus was never trying to do is the real skill here.
PCA's Prometheus Fundamentals domain names a competency called "Understanding Prometheus Limitations" — real, and worth the exam credit — but an associate exam has room to ask that Prometheus has limits, not to make you reason through why they're structural rather than fixable with better configuration. Four are worth actually understanding:
- Cardinality is architectural, not a mistake you configure your way out of. Because every unique combination of label values is a fully separate series — its own chunk in the in-memory head block, its own entries in the write-ahead log — the labelled data model has no built-in ceiling on how many series a metric can produce. A dotted hierarchy implicitly bounds cardinality, because an operator has to physically type out every path segment that will ever exist. A label has no such brake: add
user_idto a metric with a million users and you've created a million series the instant real traffic arrives, and Prometheus has no way to warn you before memory pressure does it for you. Sharding a workload across multiple Prometheus servers spreads that cardinality around; it does not reduce the total amount of it that exists. - A single Prometheus server deliberately does not cluster, and does not retain data long-term on its own. This is itself part of the original philosophy, not an oversight: each server is a self-sufficient binary with local disk that keeps working even if every other distributed system around it is on fire — including the distributed systems that would otherwise be needed just to keep monitoring itself alive. The cost is that horizontal scale, cross-cluster aggregation, and retention past local disk are explicitly out of scope for the core project, solved instead by an entire adjacent ecosystem — Thanos's sidecar-plus-object-storage pattern, Cortex and Mimir's push-based multi-tenant architecture, VictoriaMetrics's own storage engine — all speaking
remote_write/remote_read, but representing a genuinely different reliability model than the single dependency-free binary underneath them. Running that stack well is CNPE-level, performance-based skill, not associate-level knowledge — which is exactly why PCA stops well short of it. - Pull has a scale ceiling of its own. With enough targets, a single server's own scrape scheduling — the CPU and network budget to issue and parse thousands of concurrent HTTP requests inside one interval — becomes the bottleneck, which is why very large fleets shard scraping functionally (by job, by hash) across several Prometheus servers rather than running one that pulls everything. And pull's requirement that the collector reach every target directly still fights NAT, egress-only networks, and workloads too ephemeral to be caught mid-scrape — precisely the shape of problem the narrow Pushgateway exception, and increasingly a push-friendly OpenTelemetry Collector pipeline sitting in front of a
remote_writereceiver, exist to route around. - It's a sampling system, not an event log. A scrape reads a counter's current value at one instant; it does not record the individual events that moved it between scrapes. That means Prometheus structurally cannot answer "did this one specific request at 14:32:07 succeed" — a scrape sees an aggregate state, never a single event. That's the tradeoff that keeps the whole system cheap enough to run at high frequency across an entire fleet: metrics answer how much, how often; logs answer what specifically happened; traces answer which request went where. That three-way split is exactly why PCA's Observability Concepts domain lists "Understand logs and events" and "Tracing and Spans" as separate competencies — Prometheus, by design, was never going to be the right answer to either.
"We'll fix the cardinality problem by sharding across more Prometheus servers" is a real, common production plan — and it fixes memory pressure on any one server without fixing the underlying number of series that exist across the fleet. If the root cause is a label that shouldn't be a label (a raw user ID, a request ID, an unbounded free-text field), sharding buys time, not a solution. The actual fix is almost always removing the offending label at the instrumentation source — metric_relabel_configs can drop it after the fact as an emergency brake, but a metric with too many labels was miswired at the point it was written, not at the point it was scraped.
Where the model is showing its age
☺ Like you're 10: The two big decisions haven't changed. What's built on top of them keeps quietly getting better at the exact spots that used to hurt.
Neither of the two founding bets has been abandoned — everything below is refinement on top of pull and labelled dimensional data, not a replacement for either, and none of it is on the PCA blueprint because an associate exam is deliberately scoped to the stable, long-standing surface rather than the moving edge. Native histograms store a compact, high-resolution histogram as a single sparse structure computed server-side, rather than forcing every instrumented service to hand-pick fixed bucket boundaries at write time — attacking the exact quantile-approximation-error problem that classic histogram_quantile() has always had. Expanded UTF-8 support for metric and label names, alongside matching PromQL quoting syntax, relaxed decades-old identifier restrictions specifically so names arriving from OpenTelemetry instrumentation — which never shared Prometheus's original naming rules — can cross into Prometheus without lossy renaming. And the remote_write protocol itself keeps evolving, in the same spirit that turned the original exposition format into OpenMetrics: a wire format under active refinement by the same ecosystem of storage backends that standardized on it in the first place. Treat the specifics as a moving target rather than a fixed fact to memorize — read the current Prometheus documentation and changelog rather than trusting a version number on any study page, including this one.
Find any live /metrics endpoint you have legitimate access to — your own service, node_exporter running locally, or a cluster you already operate — and curl it raw, without a client library formatting anything for you. Count the distinct metric_name values (the exposition format's # TYPE lines make this easy), then pick the single metric with the most label keys and count how many actual series that one metric name produced. Now do the arithmetic: if you added one more label with a hundred possible values, multiply. That number, computed by hand once, is worth more than a paragraph about cardinality — it's the moment "the label set is the series count" stops being an abstraction.
Foxy: Wait, doesn't pull sound backwards? Shouldn't my app just tell Prometheus what's going on, the moment it happens?
Ellie the Elephant: That's the intuition push-based systems run on, and it works — right up until a client dies quietly. Under pull, a missed scrape is the alarm. up flips to zero automatically, in the same database as everything else, with nothing extra built.
Gizmo the Gremlin: Great, so labels are basically free then? Let's tag every metric with the request ID too. Way better dashboards. 📊
Timmy the Turtle: Absolutely not. A label isn't a note you jot down — it's a brand-new time series for every value it can take. Request ID has essentially unlimited values. You wouldn't be building a better dashboard, you'd be building an outage.
Professor Owl: Which is really the whole trade laid bare — a fixed hierarchy bounds cardinality by construction because someone has to type every path out by hand. Labels trade that safety net for query-time flexibility. You can slice by anything later, but nothing stops you from creating a million series by accident.
Nutty the Squirrel: And the interesting part is how far past Prometheus-the-server this all traveled! Thanos, Cortex, Mimir, VictoriaMetrics — none of them reinvented the wire format. They all just speak remote_write, because it was already sitting there.
Ellie: Which is why none of this is on the PCA paper, and all of it is worth knowing anyway. The exam certifies you can use the notepad correctly. Running the notepad at real scale, for years, is a different skill — and it's built entirely out of the same two decisions we just walked through.
1. Name the two independent design decisions behind Prometheus, and say what conventional approach each one rejected. 2. Why does the pull model give you "this target is down" as a free signal, and what would a naive push-based system need instead to get the same signal? 3. In terms of when you decide which dimensions you can query by, what's the practical difference between a dotted-hierarchy metrics system and Prometheus's labelled model? 4. Why is cardinality explosion a property of the data model itself rather than a simple misconfiguration you can just patch? 5. Name three things a single Prometheus server deliberately does not do on its own, and roughly what kind of project usually fills each gap. 6. Why can Prometheus alone never answer "did this one specific request succeed" — and what does answer that question instead? 7. What does "the exposition format became a lingua franca" actually mean in practice? Name two ways it's true for systems that aren't Prometheus itself.
Check your answers
- Pull instead of push (the server initiates every metrics exchange, rejecting the StatsD-style push-agent model), and labelled dimensional data instead of a fixed dotted hierarchy (rejecting the Graphite-style model where every dimension is a path segment chosen at instrumentation time).
- A scrape either succeeds within its timeout or it doesn't; a failed or timed-out scrape writes
up{...} = 0to the same TSDB automatically. A naive push system's receiver has no way to distinguish "client crashed" from "client simply had nothing to report" without a separate heartbeat or dead-man's-switch mechanism built on top. - A dotted hierarchy fixes every dimension as a path segment chosen when the metric is first named — a query can only slice along dimensions the tree was shaped for in advance. Prometheus's labelled model lets any label be the query axis chosen at query time, after the fact, with no redesign of the metric needed.
- Every unique combination of label values is a fully separate time series with its own memory footprint; the data model has no built-in ceiling on how many combinations can exist. A hierarchy implicitly bounds cardinality because an operator has to type out every path in advance; a label has no equivalent brake, so a single unbounded label (like a raw user ID) can multiply series count without warning.
- Clustering/horizontal scale (filled by systems like Thanos, Cortex or Mimir with a push-based or sidecar architecture), long-term retention beyond local disk (the same tools, or VictoriaMetrics), and cross-cluster or multi-tenant aggregation (also the remote-write ecosystem) — a single Prometheus server deliberately stays a self-sufficient, dependency-free binary instead.
- Prometheus scrapes on an interval, so it reads a counter's aggregate state at one instant rather than recording individual events between scrapes — it structurally cannot see one specific request. Logs answer "what specifically happened," and traces answer "which request went where"; metrics answer "how much, how often."
- It means systems that were never Prometheus itself default to speaking its exposition format and protocols because so much of the ecosystem already does. Two concrete examples: OpenMetrics formalized the text exposition format as a standalone spec independent of running Prometheus at all, and Thanos/Cortex/Mimir/VictoriaMetrics all standardized on Prometheus's own
remote_writeprotocol as their common ingestion API rather than inventing separate ones.