Infrastructure & Operations · Monitoring & observability

Monitoring & observability

Monitoring tells you when something you already anticipated goes wrong. Observability lets you figure out what went wrong when you didn't anticipate it — which, for any system past a certain size, is most incidents. This page draws that line precisely, covers the three telemetry pillars that make observability possible, walks through the four golden signals every service should expose regardless of what it does, and lays out the alerting principles that keep a paging system trusted instead of ignored.

Monitoring: watching for failure modes you already predicted

Monitoring is the practice of collecting a predetermined set of signals and alerting when they cross a predetermined threshold. A team decides in advance which failure modes matter — CPU above 90%, disk above 85%, HTTP 5xx rate above 1% — builds a dashboard and an alert rule for each, and monitoring does its job precisely when one of those known conditions occurs. This is the model behind classic tools like Nagios and Zabbix, and it's the model most teams start with because it maps directly onto the failure modes everyone already knows about: a full disk, a dead process, a saturated CPU.

The structural limit of monitoring is that it can only tell you about the questions it was built to ask. If a new failure mode shows up — a subtle interaction between a cache eviction policy and a downstream retry storm, say, something nobody wrote a dashboard for — monitoring stays green while the system is actually degraded, because green was defined as "none of the metrics we thought to collect have crossed their threshold," not "the system is healthy." This isn't a flaw in execution; it's the definition of the practice. Monitoring answers questions you already knew to ask, before you knew you'd need to ask them.

That limit is exactly what observability is built to address, and it's why the two practices are complementary rather than competing — most production systems on this platform's toolchain run both at once.

Observability: enough raw telemetry to ask questions you didn't predict

Observability, a term borrowed from control theory, is a property of a system: how well its internal state can be inferred from its external outputs. Applied to software, it means collecting telemetry that is raw and high-cardinality enough — not pre-aggregated into a fixed set of dashboard panels — that an engineer can ask a novel question during an incident and get an answer, without having predicted that question in advance and instrumented specifically for it. "Show me every request from customer ID 4471 that touched the checkout service and took over 800ms in the last hour, broken down by which pod handled it" is an observability query. It requires the underlying events to carry enough distinct dimensions (customer ID, pod name, route, latency, as separate queryable fields) that this exact slice can be constructed on demand, which is why high-cardinality data — fields with many distinct values, like user ID or request ID, not just a handful of fixed categories — is the defining requirement, not an optional nice-to-have.

Monitoring and observability aren't a strict hierarchy where one replaces the other. A well-run service still keeps dashboards and alerts on its known failure modes — that's still the fastest way to get paged about a disk filling up. Observability is what you reach for once an alert fires and the dashboard tells you that something is wrong but not why: it's the difference between "error rate is elevated" (monitoring told you that) and "the errors are concentrated on one shard, correlate with a deploy nine minutes ago, and only affect requests carrying a specific feature flag" (only queryable raw telemetry gets you there). Charity Majors and Honeycomb popularized this framing specifically because increasingly distributed, increasingly dynamic systems — services that autoscale, get rescheduled by an orchestrator, and depend on a dozen other services — produce failure modes too numerous and too novel to pre-enumerate as dashboards.

A useful test for telling the two apart in practice: if answering a new question about production requires shipping new code to add a metric or a log line, you have monitoring but not observability for that question. If you can answer it by querying data you were already collecting, you have observability.

The three pillars: metrics, logs, and traces

Observability is built from three distinct kinds of telemetry, each answering a different question and each with a different cost profile:

• Metrics are numeric time series — a value, a timestamp, and a set of labels, such as http_requests_total{status="500",route="/checkout"}. They're cheap to store and query because they're pre-aggregated by design (Prometheus, StatsD, and CloudWatch Metrics are the common implementations), which makes them excellent for dashboards and threshold alerts but poor for root-causing a single anomalous request — a metric can tell you the error rate went from 0.1% to 4%, not which specific requests failed or why.
• Logs are discrete, timestamped events, typically one line per occurrence: a request handled, an exception thrown, a job completed. Structured logs (JSON lines with consistent fields, rather than free-text strings) are what make logs queryable at observability-grade granularity — tools like the ELK stack (Elasticsearch, Logstash, Kibana) or Loki index and search them. Logs are the richest single-event detail but the most expensive pillar to store and search at high volume, since every request can generate several lines.
• Traces follow one request's path across every service it touches in a distributed system, as a tree of timed spans — one span per hop — linked by a shared trace ID that propagates through headers from the edge inward. A trace answers "where did the 800ms go" directly: 40ms in the API gateway, 600ms waiting on the recommendations service, 160ms writing to the database. OpenTelemetry has become the standard instrumentation layer for producing traces (and increasingly metrics and logs too) in a vendor-neutral format, exported to backends like Jaeger, Tempo, or a commercial APM.

No single pillar is sufficient alone. Metrics tell you something is wrong and roughly how much; traces tell you where in a multi-service call the time or the error is concentrated; logs tell you the exact detail of what happened at that specific point. A typical incident flow moves through all three: a metric-based alert fires, a trace narrows the problem to one service and one span, and a log line from that service at that timestamp shows the actual exception and stack trace.

The four golden signals

Google's Site Reliability Engineering book proposes four signals as the minimum set worth watching on any service, regardless of what the service does internally. They're deliberately generic — the same four apply to a REST API, a message queue consumer, or a batch job — which is what makes them a useful default when a team is deciding what to instrument on a new service and doesn't yet know its specific failure modes.

• Latency — how long requests take. Report this as a distribution (p50, p95, p99), never a single average — an average of 120ms can hide a p99 of 4 seconds affecting one user in a hundred, and that tail is often exactly where the real problem lives. It's also worth splitting the latency of successful requests from failed ones, since a fast error and a slow error tell you different things.
• Traffic — the demand placed on the service, measured in whatever unit fits it: requests per second for an HTTP API, messages consumed per second for a queue worker. Traffic is the denominator that makes the other three signals interpretable — 50 errors a second means something completely different at 100 req/s traffic versus 100,000 req/s.
• Errors — the rate of requests that failed, whether by an explicit failure code (HTTP 5xx), an implicit one (a 200 response with the wrong content), or a policy violation (a response that took longer than an agreed budget). What counts as an error should match what the service's callers actually care about, not just what's easy to count.
• Saturation — how full the service is relative to its limit: CPU and memory utilization, but also less obvious constraints like queue depth, connection pool usage, or thread pool occupancy. Saturation is a leading indicator — a service at 92% memory utilization isn't failing yet, but it's the signal that predicts the other three are about to get worse.

The four signals map cleanly onto the three pillars: latency and errors are usually visible in traces down to the individual span, traffic and errors are usually cheapest to track as metrics, and saturation is almost always a metric pulled from the host or runtime. None of the four require a novel failure mode to be useful — they're the baseline every service should expose before anyone builds a more bespoke dashboard on top.

Alerting principles: actionable, symptom-based, and fatigue-aware

Collecting telemetry is necessary but not sufficient — someone has to get paged at the right moment, and alerting is where most of the day-to-day trust in an observability setup is won or lost. Three principles, again straight from the SRE book's alerting chapter, keep a paging system worth trusting:

1. Every alert must be actionable. If a human gets paged and there's nothing to do except acknowledge and go back to sleep, the alert shouldn't exist — either fix the underlying condition so it stops firing, downgrade it to a dashboard, or delete it. An alert with no corresponding action trains whoever's on call to skim and dismiss, which is exactly the reflex that causes a real page to get missed later.
2. Page on symptoms, not causes. Page when users are actually affected — elevated latency, an elevated error rate, a golden-signal threshold breached — not on every internal condition that might eventually cause that. "Database CPU at 85%" is a cause that may or may not ever become a symptom; "p99 latency above 2 seconds for 5 minutes" is a symptom that always means someone is having a bad time right now. Causes belong on dashboards for investigation after a symptom-based page has already woken someone up, not as pages of their own — otherwise every one of a service's dozen dependencies gets its own alert, and a single real incident pages the same engineer a dozen times for one root cause.
3. Set thresholds to avoid alert fatigue. A threshold set too sensitively fires on routine noise — a brief traffic spike, a single slow request during a deploy — and trains responders to distrust and ignore pages, which is the single biggest predictor of a missed real incident. Multi-window, multi-burn-rate alerting (comparing a short window like 5 minutes against a longer window like 1 hour, both breaching before paging) is the standard technique for catching real, sustained degradation while filtering out brief blips that self-resolve. This same discipline connects directly to error budgets and SLOs, covered from the reliability side in this platform's SRE course, and to how a fired alert becomes a tracked incident in incident management.

Putting it together in practice

A service that's well-instrumented in the sense this page describes has metrics dashboards for the four golden signals with symptom-based alerts on latency and error-rate thresholds, structured logs searchable by request ID and other high-cardinality fields, and distributed tracing wired through OpenTelemetry so a slow or failing request can be followed hop by hop. None of these three pillars is optional if the goal is real observability rather than monitoring with extra steps — a service with beautiful dashboards but no traces still can't answer "why is this specific customer's checkout slow," and a service with rich traces but no alerting still won't page anyone when it matters.

This is also where monitoring and observability investment differ in maturity, not in kind: a new service usually starts with metrics and golden-signal dashboards (monitoring, cheap to set up, catches the obvious failure modes immediately), then layers in structured logging and tracing as it grows more distributed dependencies and the failure modes stop being predictable in advance. The DevOps toolchain page covers where specific tools — Prometheus, Grafana, the ELK stack, OpenTelemetry backends — sit in this stack; this page is about the concepts those tools implement, which outlast any specific vendor choice.

Check your answers

1. Monitoring watches a predetermined set of signals for predetermined failure modes; observability is the property of having enough raw, high-cardinality telemetry to answer novel questions about failure modes nobody predicted in advance. High cardinality — fields with many distinct values, like request ID or customer ID, queryable individually rather than pre-aggregated — is the property that makes that possible.
2. Metrics (numeric time series) are best for "is something wrong and roughly how much"; logs (discrete timestamped events) are best for the exact detail of what happened at one point; traces (a request's path across distributed services as linked spans) are best for "where in the call chain did the time or error occur."
3. Latency, traffic, errors, and saturation. Saturation is a leading indicator because a service running hot on CPU, memory, or a connection pool isn't failing yet, but that fullness is what predicts rising latency and errors are about to follow.
4. Every alert must be actionable, page on symptoms rather than causes, and set thresholds deliberately to avoid alert fatigue. "Page on symptoms, not causes" means alerting on user-visible conditions like elevated latency or error rate, not on every internal condition (like one dependency's CPU usage) that might eventually lead to a symptom — causes go on a dashboard for investigation, not a page.