Distributed Tracing & Telemetry

Monitoring & observability introduced traces as one of three telemetry pillars and moved on. This page stays. A real request today might cross a gateway, three internal services, a message queue, and a database before it returns — and the moment it's slow or wrong, "grep the logs around that timestamp" stops working almost immediately, because nothing yet says which log lines across which services belong to which request. This page is the machinery that fixes that: what a span actually contains, how a trace ID survives a hop across a network call or a queue (and how it doesn't, if nobody wires it up), what a log line needs before it can be joined against a trace, and OpenTelemetry — the vendor-neutral instrumentation standard that makes all of it portable across Jaeger, AWS X-Ray, Datadog APM, and whichever backend you're running next year.

From a telemetry pillar to a load-bearing system

☺ Like you're 10: The last page told you traces exist. This page is the actual plumbing that keeps one trace glued together as a request bounces between services — and the standard almost everyone has converged on for building that plumbing instead of hand-rolling it.

Metrics, logs, and traces got one paragraph each in the observability survey, and that's the right amount of space for an introduction — but traces are structurally different from the other two in a way that deserves its own page. A metric is emitted and stored by one process; a log line is written and shipped by one process. A trace, by definition, is not: it only exists as a coherent object if every process a request touches agrees to carry the same identifier forward and hand it to the next process in line. Nothing in TCP/IP, HTTP, or a message broker does that for you automatically. It is an engineering discipline someone has to actually implement, and most of the ways production tracing quietly breaks trace back to exactly one hop where that discipline lapsed.

This page covers that discipline in three layers: the shape of a trace and a span, the standard for carrying a trace ID across a boundary (HTTP call, queue hop, or otherwise), and how a structured log line earns the right to be pulled up next to a trace instead of grepped for by hand. Then it covers OpenTelemetry, the project that has absorbed most of this into a library you install rather than protocol you re-derive from a spec every time you pick up a new language.

Anatomy of a trace: spans, context, and the waterfall

☺ Like you're 10: A trace is the whole relay race. A span is one runner's leg of it — with a start time, an end time, a name for what they were doing, and a note about who handed them the baton.

A trace is a set of spans that share one trace_id. A span represents one unit of work — an HTTP request, a database query, a function call you cared enough to name — and carries a fixed shape regardless of which language or library produced it:

• trace_id — a 128-bit ID shared by every span in the trace. This is the sticker from the analogy above.
• span_id — a 64-bit ID unique to this one span.
• parent_span_id — the span_id of whoever called this span into existence. The root span, the very first one, has none.
• name — a human-readable operation name (POST /checkout, SELECT orders, apply_discount_rules).
• span kind — one of SERVER (received a call), CLIENT (made a call), INTERNAL (work with no network hop), PRODUCER (published a message), or CONSUMER (picked one up). This is what lets a backend draw a clean line between "waiting on the network" and "doing real work."
• start / end timestamps — duration is just the difference, but both ends are stored so spans can be laid out on a shared timeline.
• attributes — arbitrary key/value tags (http.status_code: 402, db.system: postgresql) that make a span filterable and comparable to every other span of the same kind.
• events — timestamped points within a span's lifetime (an exception thrown, a retry attempted) — a span's own tiny embedded log.
• status — OK, ERROR, or UNSET, plus an optional description. This is what "keep every failed trace" filters key off.
• links — pointers to spans in a different trace, or to a span that isn't a direct parent — the escape hatch for relationships that don't fit a clean tree, which is exactly what an asynchronous queue hop needs.

Stitch every span's parent_span_id back to its parent and you get a tree — usually drawn as a waterfall, one horizontal bar per span, nested under its parent and positioned on a shared time axis. Reading a waterfall is the single most common motion in a tracing-driven incident: find the longest bar, that's roughly where the time went; find the red bar, that's roughly where the error happened.

Propagating a trace ID across a service boundary

☺ Like you're 10: The trace's sticker number has to physically travel inside the request itself — as a header — or the next service has no way to know it's part of the same race at all.

Context propagation is the actual mechanism, and it comes down to one rule: the calling span's trace_id and span_id travel as request headers, and the receiving service reads them before creating its own span as a child. The W3C Trace Context specification standardized the header format in 2020, and it's what OpenTelemetry uses by default:

GET /checkout HTTP/1.1
Host: api.acme.io
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
tracestate: dd=s:1;congo=t61rcWkgMzE

# 00                                 spec version
# 4bf92f3577b34da6a3ce929d0e0e4736   trace-id    (32 hex chars / 16 bytes)
# 00f067aa0ba902b7                   parent-id   (16 hex chars / 8 bytes — the CALLER's span_id)
# 01                                 trace-flags (bit 0 = sampled: 01 sampled, 00 not)

The receiving service parses traceparent, keeps the same trace_id, treats the incoming parent-id as its new span's parent, mints its own fresh span_id, and — critically — forwards a rewritten traceparent (pointing at its own span_id) to whatever it calls next. tracestate rides alongside it as an ordered, vendor-specific extension slot, letting multiple tracing systems annotate the same request without stepping on each other. A separate, related header — baggage — propagates arbitrary business context the same way (baggage: customer.tier=gold,checkout.experiment=v2), so any downstream service can read it and tag its own spans, without that context needing to be threaded through every function signature by hand.

W3C Trace Context is the current standard, but it's not the only format you'll meet in the wild: B3 propagation (from Zipkin, still emitted by older Spring Cloud Sleuth and Envoy configs) uses separate X-B3-TraceId / X-B3-SpanId / X-B3-ParentSpanId / X-B3-Sampled headers, or a single condensed b3 header. OpenTelemetry's SDKs can read and write both formats, which matters the moment your trace crosses from a newly instrumented service into a legacy one, or through a proxy layer (Envoy, an API gateway) that only speaks B3.

Asynchronous hops need one more step, because there's no synchronous call to attach a header to. Publishing to a queue means manually copying the current traceparent into the message's own headers or attributes (Kafka record headers, an SQS MessageAttribute) at publish time, and reading it back out at consume time to start the next span as a link rather than a direct child — which is exactly the branch drawn in the diagram above. Most OpenTelemetry instrumentation libraries for Kafka, SQS, and RabbitMQ do this automatically once installed; hand-rolled queue clients are where this quietly gets skipped.

Structured logs a trace can actually join against

☺ Like you're 10: A log line becomes useful to a trace the moment it writes down the same sticker number the trace is using — as its own labeled field, not buried in a sentence.

A free-text log line like ERROR payment authorization failed for order 88213 is readable by a human and useless to a machine trying to join it against a trace. Structured logging — one JSON object per line, with consistent field names — fixes that, but only if trace_id and span_id are among the fields, written by the logging library at the moment the log call happens, from whatever span is currently active in that thread or async context:

{"timestamp":"2026-08-16T14:02:11.408Z","level":"error","service":"checkout",
 "message":"payment authorization failed","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736",
 "span_id":"00f067aa0ba902b7","customer_id":"4471","http.status_code":402,
 "payment.provider":"stripe","payment.decline_code":"insufficient_funds"}

Once every service's logger is configured this way, the join is trivial in either direction: from a slow or errored span in your tracing backend, pivot straight to every log line across every service carrying that exact trace_id, in whatever log backend holds them (the ELK stack, Loki, or a hosted equivalent) — no more grepping by approximate timestamp and hoping you've got the right five-minute window on the right host. The wiring is almost always automatic once it's set up once: Python's logging module, Node's pino/winston, and Java's MDC (Mapped Diagnostic Context) can all be configured to pull trace_id/span_id from OpenTelemetry's active context and stamp every log call with it, with zero changes to the call sites themselves.

The same correlation idea shows up one layer over in metrics, under the name exemplars: a single data point in a Prometheus histogram bucket can carry the trace_id of one specific request that landed in that bucket, so a spike on a latency graph in Prometheus or Grafana isn't just a number — it's a clickable link straight into one representative trace of the requests that caused it. That's the three pillars closing the loop the observability survey promised: a metric says something's wrong, an exemplar or a trace query narrows it to specific requests, and the joined log lines say exactly what happened.

OpenTelemetry: one instrumentation standard, many backends

☺ Like you're 10: Instead of every tool inventing its own private way to measure things, OpenTelemetry is the one shared measuring tape everyone agreed to use — so you can point the same measurements at whichever tool you like.

OpenTelemetry (often shortened to OTel) is a CNCF project formed in 2019 by merging two earlier, competing efforts — OpenTracing (the tracing-focused API that Jaeger and Zipkin grew up around) and OpenCensus (Google's combined metrics-and-tracing library) — into one vendor-neutral standard for producing metrics, logs, and traces. It has since become one of the most active projects in the CNCF, and for good reason: before it existed, instrumenting an application for Jaeger versus Datadog versus X-Ray meant three separate SDKs, three separate sets of decorators wrapped around your code, and re-instrumenting everything if you ever switched. OpenTelemetry's whole value proposition is decoupling how you instrument code from where the telemetry ends up.

The architecture has three distinct layers, and confusing them is the most common source of OTel confusion for newcomers:

• The API — the interfaces your code calls (tracer.start_span(...), span.set_attribute(...)). Stable across versions by design, so instrumented library code doesn't break when you upgrade your telemetry backend.
• The SDK — the actual implementation behind that API: samplers, span processors, and exporters that decide what happens to a span once it's finished (batch it, sample it, ship it somewhere).
• The Collector — a separate, standalone process (not a library — you deploy it like any other service) that receives telemetry over OTLP (the OpenTelemetry Protocol, over gRPC or HTTP), runs it through configurable processors, and exports it to one or more backends. Routing telemetry through a Collector instead of exporting straight from every service means sampling policy, redaction, and backend routing live in one place you can change without redeploying application code.

Two ways to actually get spans out of your code, and most real services use both: auto-instrumentation patches common libraries (HTTP clients and servers, database drivers, queue clients) with zero code changes — you install an agent or wrap your start command, and every HTTP call and DB query gets a span for free. Manual instrumentation uses the SDK API directly, for the business logic no library can see on your behalf:

# Auto-instrumentation: no code changes, wrap the start command —
# every HTTP call, DB query, and queue publish gets a span for free.
opentelemetry-instrument \
  --traces_exporter otlp \
  --service_name checkout \
  python app.py

# Manual span, for the part auto-instrumentation can't see —
# your own business logic, not a library call:
from opentelemetry import trace
tracer = trace.get_tracer("checkout.pricing")

with tracer.start_as_current_span("apply_discount_rules") as span:
    span.set_attribute("customer.tier", customer.tier)
    span.set_attribute("discount.rules_evaluated", len(rules))
    result = apply_rules(customer, cart)
    span.set_attribute("discount.amount_cents", result.amount_cents)

One more piece makes cross-service, cross-language traces actually comparable: semantic conventions, OpenTelemetry's standardized attribute names (http.method, http.status_code, db.system, messaging.system). Without them, a Java team's "did this call fail" attribute and a Python team's version would drift into two different key names, and no query or dashboard could treat them as the same thing. With them, a span from any instrumented service in any supported language uses the same vocabulary — which is what lets one Collector, one set of sampling rules, and one dashboard work across an entire polyglot fleet.

Sampling: you can't keep everything, so decide on purpose

☺ Like you're 10: Recording every single request forever gets expensive fast, so you have to choose which ones to keep — either by rolling dice before you know what happens, or by watching the whole thing and then deciding.

A busy service can generate more trace data per day than makes sense to store, let alone pay a vendor per span for, so nearly every real deployment samples — keeps some fraction of traces and discards the rest. There are two fundamentally different ways to decide which ones survive:

Tail-based sampling is strictly more useful for incident response — the traces you want most, the failing and the slow ones, are exactly the ones a random head-based roll is likeliest to throw away — but it costs more, because the Collector has to buffer entire traces in memory until it can see whether they qualify, and in a horizontally scaled fleet that means all of a trace's spans have to be routed to the same Collector instance to be evaluated together. Plenty of teams run head-based sampling for routine traffic and lean on tail-based rules specifically for "always keep errors" as the two aren't mutually exclusive.

Choosing (and switching) a tracing backend

☺ Like you're 10: Because everyone agreed on the same measuring tape, you can hand your measurements to a different workshop later without measuring everything all over again.

The backend is where spans actually land, get stored, and get queried — and because OpenTelemetry decouples instrumentation from destination, this is close to a config change, not a re-instrumentation project:

The practical upshot: a team that instruments with the OTel API and SDK and routes everything through a Collector can start on self-hosted Jaeger, add a second export to a commercial backend for a proof of concept, and drop Jaeger later — all by editing the Collector's exporters block, with zero changes to application code:

receivers:
  otlp:
    protocols: { grpc: {}, http: {} }

processors:
  batch: {}
  tail_sampling:
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 500 }
      - name: sample-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

exporters:
  otlp/jaeger:
    endpoint: jaeger-collector:4317
  otlp/datadog:
    endpoint: otel-intake.datadoghq.com:4317
    headers: { "dd-api-key": "${DD_API_KEY}" }

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlp/jaeger, otlp/datadog]

Putting it together: what good coverage looks like

☺ Like you're 10: All four pieces — spans, propagation, joinable logs, and one shared standard — only pay off together. Skip any one and there's a hole an incident will eventually find.

A service with real tracing coverage has all four pieces from this page working at once: spans with span kinds and semantic-convention attributes, a traceparent that survives every hop it makes — including its queue hops, wired by hand or by instrumentation — structured logs stamped with trace_id and span_id on every line, and a sampling policy that's a deliberate choice rather than an accident of default settings. None of this is optional if the goal is answering "why was this one specific request slow" rather than "the p99 went up sometime this afternoon" — which is precisely the gap between monitoring and observability the survey page opened with.

This is also where tracing stops being a standalone concept and starts touching almost everything else in this course: a canary in deployment strategies is far easier to judge when you can compare trace latency between the old and new version directly instead of waiting on aggregate dashboards; an incident gets diagnosed in minutes instead of hours once the responder can jump from an alert straight to the one trace that explains it; a error budget burns on the same latency and error data a trace is built from; and a game day is the cheapest way to find out your queue hop was never actually propagating context, before a real incident finds out for you.