Monitoring & observability

Almost every service, regardless of what it does, is worth watching along the same four dimensions. This page covers Google's four golden signals in full, two practical framings built on top of them — USE and RED — and the single most consequential decision in alert design: paging on symptoms your users actually feel, not on every internal condition that might eventually cause one.

The four golden signals

Google's Site Reliability Engineering book names four signals as the minimum set worth a dashboard and, for most of them, an alert — on almost any service you'll ever run. They're deliberately generic: the same four apply to a REST API, a gRPC backend, a queue consumer, or a nightly batch job, which is exactly what makes them the right default when you're instrumenting a new service and don't yet know its specific failure modes.

• Latency — the time it takes to service a request. Report it as a distribution (p50, p95, p99), never a single average: an average of 150ms can hide a p99 of 3 seconds hitting one request in a hundred, and that tail is usually where the real complaints come from. Just as important: split successful latency from failed latency. A request that fails fast (a 400 in 8ms) and a request that fails slow (a 500 after a 30-second timeout) are different problems, and averaging them together into one "latency" number hides both.
• Traffic — the demand being placed on the system, measured in whatever unit fits it: requests per second for an HTTP API, transactions per second for a database, messages per second for a queue consumer. Traffic is the denominator that makes the other three signals mean anything — 200 errors a second is background noise at 500,000 req/s and a full-blown outage at 300 req/s.
• Errors — the rate of requests that failed. That includes explicit failures (HTTP 5xx, a gRPC error code), implicit ones (a 200 response with a malformed or wrong body), and policy failures (a response that took longer than an agreed budget, which some teams count as an error even though nothing "broke"). What counts as an error should reflect what callers actually care about, not just what's cheapest to count.
• Saturation — how full the service is relative to its limit: CPU and memory utilization, but just as often a less obvious ceiling like queue depth, connection-pool usage, thread-pool occupancy, or disk headroom. Saturation is a leading indicator — a service sitting at 90% memory isn't failing yet, but it's the signal that predicts latency and errors are about to get worse, often before either one has moved.

These four are the reason a well-run on-call rotation can hand a brand-new engineer a service they've never seen and have them make a reasonable call about its health in under a minute: pull up latency, traffic, errors, and saturation, and you have a first-order picture regardless of what the service actually does. See SLIs, SLOs & error budgets for how a subset of these same signals — usually latency and errors — become the metrics an SLO is defined against.

USE and RED: two practical framings on the same idea

The golden signals are the right mental model but not, by themselves, a checklist you can hand someone instrumenting a specific box. Two narrower methods fill that gap, each optimized for a different kind of thing you're watching, and both are just the golden signals sliced for a specific purpose.

USE — Utilization, Saturation, Errors — was popularized by performance engineer Brendan Gregg for watching resources: CPU, disk, memory, network interfaces, a database connection pool, a queue. For every resource you ask three questions: Utilization — the percentage of time the resource was busy doing work; Saturation — the extra work it's queued up because it couldn't keep up (run-queue length, wait-queue depth); and Errors — the count of error events on that resource (dropped packets, disk I/O errors, retransmits). USE is deliberately resource-centric and is the method you reach for when you're root-causing "something upstream is slow" down to a specific saturated box — a full disk queue, a maxed-out connection pool — rather than watching a service end to end.

RED — Rate, Errors, Duration — was popularized by Weaveworks for watching request-driven services: Rate is requests per second (traffic), Errors is the rate of failing requests, and Duration is the distribution of time each request takes (latency). RED is essentially the golden signals with saturation dropped, because for a stateless request-handling service saturation usually shows up one layer down, on the resources RED doesn't directly track — which is exactly where USE picks it up. USE adds utilization in its place, a question that doesn't quite apply to a raw resource the way traffic and latency apply to a service, but that distinguishes "busy but coping" from "busy and about to fall over" for the resource itself. Neither method is a different theory from the golden signals — both are the same four signals pre-filtered for what you're looking at. A typical production stack runs both together: RED dashboards on every service in the request path, USE dashboards on the resources underneath them (hosts, databases, message brokers), and the golden signals are the shared vocabulary that makes both legible to anyone on the team regardless of which one a given dashboard uses.

Symptom-based alerting vs. cause-based alerting

Collecting the right signals doesn't automatically produce a good on-call experience — that depends on what you choose to page a human for, and this is where most alerting setups go wrong. There are two ways to write an alert rule, and only one of them scales.

Cause-based alerting pages on every internal condition that might eventually lead to a problem: database CPU above 85%, one replica's disk above 90%, a single upstream dependency's error rate above 2%. It feels thorough — you're covering every way things could go wrong — but it doesn't scale, because a single real incident usually trips a dozen internal causes at once, each with its own alert rule, and now one outage pages the on-call engineer a dozen times for one root cause. Worse, many of those causes never actually become user-visible: a database can run hot at 85% CPU for hours without a single request getting slower, because it has headroom nobody documented. That alert fires anyway, gets acknowledged, and does nothing — training whoever's on call to skim and dismiss.

Symptom-based alerting pages only when users are actually affected: p99 latency above an agreed threshold for five minutes, error rate above an agreed threshold, a golden-signal breach that a real person outside the company would notice. "p99 latency above 2 seconds for 5 minutes" is a symptom — it always means someone out there is having a bad time right now. "Database CPU at 85%" is a cause that may or may not ever become one. Symptom-based alerting produces dramatically fewer pages, and every one of them is, by construction, actionable: if users are affected, there is always something worth doing about it, even if that something is just triage.

This doesn't mean cause-level signals are worthless — quite the opposite. They belong on a dashboard for the human who's already been paged by a symptom alert to investigate why. That's the distinction worth holding onto precisely: dashboards are for humans who are already looking, actively digging through data to understand a problem; alerts and pages are for pulling a human's attention toward the problem in the first place, at 3 a.m. if necessary. Conflating the two — paging on something that's really dashboard material, or leaving a genuine symptom sitting silently on a dashboard nobody's watching until a customer files a ticket — is the single most common alerting design mistake. Get this distinction right first, before touching thresholds or notification channels; see incident management & on-call for what happens after a symptom-based alert actually fires.

# Cause-based (avoid as a page — dozens of these can fire for one incident)
alert: DatabaseCPUHigh
expr: db_cpu_percent > 85
for: 5m
severity: page          # wrong — CPU alone isn't user-visible

# Symptom-based (page on this instead)
alert: CheckoutLatencyHigh
expr: histogram_quantile(0.99, http_request_duration_seconds{route="/checkout"}) > 2
for: 5m
severity: page          # right — a real user is waiting 2+ seconds right now

# The cause-based signal still has value — just not as a page
alert: DatabaseCPUHigh
expr: db_cpu_percent > 85
for: 5m
severity: dashboard     # visible to whoever investigates the page above, not a page itself

Putting the four signals into practice

A service that's well-monitored in the sense this page describes has a RED dashboard for its request path (rate, errors, duration split by success/failure), USE dashboards for the resources underneath it (CPU, memory, disk, connection pools, queues), and a small number of symptom-based alerts wired to page — typically one for elevated error rate and one for elevated latency, both measured against what users actually experience, not against an internal cause. Everything else — the individual resource utilizations, the queue depths, the replica lag — lives on dashboards for triage, not on the pager.

This is also where the four golden signals connect back to SLIs, SLOs, and error budgets: an SLO is usually built from exactly one or two of the four signals (most often latency and errors), turned into a formal target with a time window, so that the symptom-based alert isn't just "does this feel bad" but "has the service burned error budget fast enough that someone needs to act now." The golden signals are the raw material; an SLO is what happens when you commit to a number for one of them.

The same discipline scales down as well as up. A single-node batch job still has a traffic (jobs processed per hour), an error rate (jobs failed), a latency (time per job), and a saturation (queue backlog) — and the same rule applies: alert on the symptom (jobs are failing or backing up), keep the resource-level causes on a dashboard for when that alert fires.