Jaeger

Jaeger is the most widely deployed open-source backend for distributed tracing: it receives the spans your services emit, files them under the request that produced them, and lets you pull up one slow or failed request and see exactly which of the dozen services it touched actually spent the time. For an SRE it converts an incident question that metrics can only gesture at — "p99 doubled, but where in the call chain did the four seconds go?" — into a question with a literal answer: this span, in this service, at this timestamp. That's the whole value proposition, and it's why tracing sits beside metrics and logs as the third pillar an on-call engineer reaches for mid-incident, not an afterthought filed during the postmortem.

What Jaeger is, and the SRE question it answers

☺ Like you're 10: Metrics tell you something's wrong somewhere; a trace tells you which exact service, in which exact call, is where the time actually went.

Jaeger began at Uber in 2015, was donated to the CNCF in 2017, and graduated in 2019 — the same maturity tier as Kubernetes and Prometheus. On a platform it plays exactly one role, and it's worth being precise about the boundary: Jaeger ingests, stores, queries, and visualizes traces. It does not instrument your code — that's OpenTelemetry's job — and it does not hold your latency or error-rate dashboards, which is Prometheus and Grafana's job, covered in monitoring & observability. Tracing answers a different question than either of those: not that a golden signal moved, but where in the call chain the request actually spent its time, per the glossary's definition of distributed tracing — instrumentation that follows one request across process boundaries so latency and errors can be attributed to the specific hop that caused them, instead of guessed at from an aggregate.

Three words carry the whole vocabulary. A span is one unit of work — an HTTP handler, a database query, a queue publish — with a start time, a duration, and a set of key/value tags. A trace is the tree of every span that belongs to one request, tied together by a shared trace ID. And context propagation is the mechanism that keeps the tree connected: a small packet — trace ID, span ID, a sampling flag — travels between services in a request header, so the next hop knows which trace it's joining. Lose that propagation at one hop and the trace doesn't get slightly worse; it splits into two unrelated traces, which is the single most common reason a trace looks broken and Jaeger gets blamed for it.

Architecture: the collector, storage, and the query service

☺ Like you're 10: One part catches the stamped slips, one part files them away, and one part draws the picture when you go looking for a specific visit.

Jaeger is a small number of stateless services sitting in front of exactly one stateful store, and knowing which piece is which turns most Jaeger debugging into a two-minute job — because whatever's wrong lives in only one of them. The jaeger-collector receives spans, validates them, runs them through a short internal queue, and writes them to storage. The jaeger-query service reads from that same storage and serves both the web UI and a JSON HTTP API on port 16686 — the only port you'll actually type into a browser. Everything else in the deployment exists to make the write path resilient at volume: the optional jaeger-ingester, paired with Kafka, lets the collector write to a queue instead of storage directly, so a storage outage buffers instead of silently dropping spans. And the historical jaeger-agent — a per-node DaemonSet or sidecar that used to batch UDP thrift from Jaeger's own client libraries — is deprecated and removed entirely in Jaeger v2; nothing new should be built around it. The OpenTelemetry Collector, run as a DaemonSet or sidecar, is the modern answer to the same local-batching problem.

Storage is a pluggable, deliberate decision, not an implementation detail — it's the difference between a demo and something you can trust during an incident. In-memory keeps everything in RAM and loses it the moment the pod restarts; fine for a five-minute local test, a genuine liability anywhere near production. Elasticsearch (or OpenSearch) is the most common production backend: rich ad-hoc tag search, and index lifecycle management gives you a real retention policy. Cassandra is the other serious option: excellent sustained write throughput and a native TTL for expiry, at the cost of weaker ad-hoc tag querying than Elasticsearch offers. Whichever you run, Jaeger's query latency, retention window, and the size of the bill are properties of the store you chose underneath it, not of Jaeger itself — see the Elastic Stack if Elasticsearch is the direction you're weighing.

OTLP is the front door now, not the legacy client SDKs

☺ Like you're 10: Jaeger used to need its own translator installed in every service; today it just listens on the same shared language every other tracing backend already speaks.

For years, getting spans into Jaeger meant importing a jaeger-client-<language> library into every service and pointing it at Jaeger-native ingest ports. Those client libraries were deprecated in 2022 in favor of the vendor-neutral OpenTelemetry SDKs, and since Jaeger v1.35 the collector accepts OTLP natively, on gRPC 4317 and HTTP 4318 — the identical ports and protocol every other OTLP-compatible backend accepts. Jaeger v2 goes further still and is built directly on the OpenTelemetry Collector framework, configured with the same receivers, processors, exporters, and extensions vocabulary, with Jaeger's own storage, query, and remote-sampling logic running as collector extensions. Jaeger v1 reached end of life at the end of 2025, so v2 is the version to actually deploy; v1 is what you inherit in an older cluster, not what you'd choose today.

The practical consequence for an SRE writing instrumentation in this decade: you should never be adding a Jaeger-specific client to a service. Instrument with OpenTelemetry once, export over OTLP, and Jaeger becomes one exporter target among several — reachable through the exact same pipeline that also feeds your metrics and logs. The old Jaeger-native ports still exist behind the OTLP front door, but only for compatibility: 14268 (Jaeger thrift over HTTP), 14250 (Jaeger proto over gRPC), and 9411 (a Zipkin-compatible endpoint, useful if you're migrating a legacy Zipkin producer — see Zipkin — without an immediate rewrite). Treat all three as legacy paths for systems you haven't finished migrating, not as a design choice for anything new.

Sampling: head-based vs. tail-based, and why the difference decides your bill

☺ Like you're 10: Keeping a permanent record of every single request forever is a real cost, every day — sampling decides which requests earn one.

Tracing every request at full volume is not a hypothetical concern at scale; it's a line item. Every stored span is a write and an index entry on whatever backend you picked, and retention costs money continuously, not once. Sampling exists to decide, deliberately, which fraction of traffic gets a permanent record — and where that decision gets made changes what you're actually able to guarantee.

Head-based sampling decides at the very first span, before any downstream work has happened, and propagates the decision as a sampling flag inside the span context so every service further down the chain agrees. Jaeger supports several head strategies: const (always on or off — fine for a single dev service, ruinous at scale), probabilistic (keep a fixed fraction, the usual production default), ratelimiting (at most N traces per second regardless of traffic), and remote, where clients poll the collector's sampling-config port 5778 for a per-service, per-operation policy — so you retune sampling centrally from a config change instead of a redeploy. Head sampling's advantage is consistency and cost predictability; its unavoidable flaw is that the decision is made before the outcome is known, so the one request in ten thousand that actually failed is very likely not among the ones you kept.

Tail-based sampling flips the order: the decision is deferred until the whole trace has been observed, which means it doesn't live in Jaeger at all — it runs in the OpenTelemetry Collector's tail_sampling processor, upstream of Jaeger. Typical policies keep every trace that errored, every trace slower than your latency SLO threshold, and a small percentage of the unremarkable rest. The catch is structural: a tail-sampling decision needs every span of one trace to land on the same collector instance, which in practice means a two-tier gateway — a first tier whose only job is a loadbalancing exporter keyed by trace ID, feeding a second tier that actually makes the keep/drop call.

The config you'll actually write

☺ Like you're 10: Three files carry most real Jaeger work: one that runs Jaeger against real storage, one that points the collector at it, and one that decides how much to keep.

A production Jaeger install via the maintained Helm chart, with Elasticsearch storage and an explicit retention policy — nothing expires on its own, so skipping the cleaner job below means an index that grows until the cluster falls over:

# values-production.yaml — helm install jaeger jaegertracing/jaeger -f values-production.yaml
collector:
  replicaCount: 3                 # stateless — scale on ingest volume, not on storage size
  service:
    otlp:
      grpc: { name: otlp-grpc, port: 4317 }
      http: { name: otlp-http, port: 4318 }

query:
  replicaCount: 2                 # serves the UI/API on 16686

storage:
  type: elasticsearch
  elasticsearch:
    host: elasticsearch.observability.svc
    port: 9200
    indexPrefix: jaeger

esIndexCleaner:
  enabled: true                   # RETENTION. Without this, indices grow forever.
  numberOfDays: 14
  schedule: "10 23 * * *"

esRollover:                       # index-lifecycle rollover, so query stays fast as volume grows
  enabled: true
  schedule: "*/30 * * * *"

Applications almost never point straight at Jaeger — they point at an OpenTelemetry Collector, which batches, protects its own memory, and only then forwards on. That indirection is what lets you change a sampling policy or a backend without redeploying every service that emits spans:

# otel-collector-config.yaml — the gateway tier that feeds Jaeger
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  memory_limiter:                          # ALWAYS first — sheds load before an OOMKill
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 800 }       # matches the service's latency SLO threshold
      - name: sample-the-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 2 }
  batch: { timeout: 5s, send_batch_size: 1024 }   # ALWAYS last

exporters:
  otlp/jaeger:                              # plain otlp — the dedicated "jaeger" exporter was removed
    endpoint: jaeger-collector.observability.svc:4317
    tls: { insecure: true }                 # in-cluster; use real TLS across trust domains

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters:  [otlp/jaeger]

And a head-based remote-sampling policy, served over 5778, for the services that reach Jaeger directly rather than through a tail-sampling gateway:

{
  "service_strategies": [
    {
      "service": "checkout-api",
      "type": "probabilistic",
      "param": 0.05,
      "operation_strategies": [
        { "operation": "GET /healthz", "type": "probabilistic", "param": 0.0 },
        { "operation": "POST /orders", "type": "probabilistic", "param": 1.0 }
      ]
    },
    { "service": "search-api", "type": "ratelimiting", "param": 15 }
  ],
  "default_strategy": { "type": "probabilistic", "param": 0.01 }
}

Read it the way an SRE should: the default keeps one request in a hundred, checkout-api keeps 5% overall but never samples its health check and always keeps its order-creation path, and search-api is capped at fifteen traces a second no matter how much traffic arrives.

Day-to-day commands and reading the UI like an SRE

☺ Like you're 10: A handful of things you type to stand it up, ask it questions from a terminal, and read the picture it draws.

# Fastest local Jaeger — UI on 16686, OTLP on 4317/4318, sampling config on 5778
docker run --rm --name jaeger \
  -p 16686:16686 -p 4317:4317 -p 4318:4318 -p 5778:5778 \
  jaegertracing/jaeger:latest

# On a cluster, via the maintained Helm chart
helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm repo update
helm install jaeger jaegertracing/jaeger \
  --namespace observability --create-namespace -f values-production.yaml

kubectl -n observability port-forward svc/jaeger-query 16686:16686
# then open http://localhost:16686

# Confirm ingest is actually happening
kubectl -n observability logs deploy/jaeger-collector --tail=50
kubectl -n observability get svc -l app.kubernetes.io/component=collector

Everything the UI shows is a JSON API you can curl — the difference between "I think it's fixed" and proof, useful mid-incident or in a drill:

# Which services have ever reported a span — the first thing to check
curl -s localhost:16686/api/services | jq .

# Traces for one service, newest first — no start/end defaults to roughly the last two days
curl -s "localhost:16686/api/traces?service=checkout-api&limit=20" | jq -r '.data[].traceID'

# Only errored traces slower than 2 seconds — the exact shape an incident search takes
curl -s "localhost:16686/api/traces?service=checkout-api&tags=%7B%22error%22%3A%22true%22%7D&minDuration=2s" | jq .

# One specific trace, in full
curl -s localhost:16686/api/traces/4bf92f3577b34da6a3ce929d0e0e4736 | jq .

The UI itself is four screens worth knowing cold. Search takes a service, an operation, tag filters, and a min/max duration — that last field is the professional's shortcut, since "checkout-api traces over 2s in the last hour" finds the incident in one click rather than scrolling. Trace detail is the Gantt-style waterfall: spans are bars indented under their parent, and you scan for the bar visibly fatter than its siblings — one long bar with nothing nested inside it usually means time was spent in that service itself (or in an uninstrumented call it made), while a long stack of short sequential bars is the signature of an N+1 pattern. Compare diffs two trace IDs structurally, which is how you prove a slow trace has 340 database spans and a fast one has three. And the dependency graph renders the service call map built entirely from observed traffic — the only architecture diagram in most organizations that's actually still accurate, because nobody has to remember to update it.

Gotchas and failure modes

☺ Like you're 10: Here's why the record book sometimes looks empty, and why that's almost never the record book's own fault.

When the search screen comes back empty, work the chain outward from the application rather than suspecting Jaeger first — the fault is almost always in the first two links. First: is the service actually exporting — is the OTLP endpoint set, is the SDK initialized, is service.name populated? An empty service name files everything under unknown_service, which people then search for by the wrong name. Second: is context being propagated — does every outbound call, including the one through a hand-rolled HTTP client or a queue publish, forward the incoming trace header? A single hop that drops it doesn't corrupt the trace; it splits the trace in two, and the interesting half looks like it simply ends there. Third: is the exporter aimed at the right port and protocol — 4317 is gRPC, 4318 is HTTP, and crossing them produces a connection that looks fine and never delivers a span. Only after ruling out all three does a collector queue or storage issue become the likely suspect.

Sampling is the second most common trap, and it's the one that bites hardest during an actual incident: at 1% probabilistic sampling, the single failing request a customer just reported is very likely not in Jaeger at all, and an engineer under pressure concludes tracing doesn't work. The fix, in order, is raising the sampling rate for the affected service temporarily via remote sampling (a config change, not a redeploy), setting a critical operation to 100%, or — the durable fix — adopting tail-based sampling upstream so every error and every slow trace is kept unconditionally, exactly as described above.

Retention is the quiet operational trap. Nothing in Jaeger expires a trace by itself: on Elasticsearch you need the index cleaner or an ILM policy, on Cassandra you set a TTL, and skipping either means an index that grows until it takes the cluster down with it. Meanwhile the collector's own internal queue is bounded — under storage backpressure, spans get dropped silently from the application's point of view, with no error surfaced to the service that sent them. Scrape the collector's own metrics for dropped spans and queue depth and alert on them; Jaeger is a production service that carries incident-response weight, and it deserves the same golden-signal treatment as anything else you'd page on.

A shorter list of quieter traps worth knowing: clock skew between hosts can make a child span appear to start before its parent (the UI flags and adjusts for it, but check NTP before chasing a "time-travelling" span); high-cardinality operation names — an order ID baked into the span name instead of a tag — destroy the operation dropdown and make search useless; and full URLs with query strings routinely land in tags carrying PII, so redact at the collector before it lands somewhere your whole company can query.

Jaeger against the alternatives

☺ Like you're 10: Other record rooms exist. Here's how to pick between them.

Because everything speaks OTLP now, the backend is a genuinely reversible decision — which is the entire point of instrumenting through OpenTelemetry rather than a vendor's own library in the first place. Choose on operations and cost, not on lock-in.

Choose Jaeger when a self-hosted, vendor-neutral tracing UI and the dependency graph matter, and you already run — or are willing to run — Elasticsearch or Cassandra. Choose Tempo when object-storage economics dominate and Grafana is already the team's one pane of glass. Either way, instrument with OpenTelemetry so the backend stays a config change rather than a rewrite — see the SRE toolchain for how tracing sits next to the rest of the stack, and capacity planning & performance for how observability storage itself becomes a capacity line item once you're running it at scale.