Jaeger

Jaeger is the CNCF-graduated distributed tracing backend: it receives the spans your services emit, stores them, stitches them back into whole request journeys, and gives you a UI on port 16686 where a single slow checkout can be opened up like a waterfall and the guilty service pointed at by name. It solves the platform problem that metrics and logs cannot touch — “the request took four seconds, and it crossed eleven services; which one ate the time?”

What Jaeger is and the problem it solves

☺ Like you’re 10: It’s a place that keeps the timesheets from every request, and draws them as bars so you can see which bit was slow.

Jaeger began at Uber in 2015, was donated to the CNCF in 2017, and graduated in 2019 — the same tier as Kubernetes, Prometheus and Argo. It is the reference open-source implementation of the tracing ideas in Google’s Dapper paper, and on a platform it plays exactly one role: the backend for traces. It does not instrument your code (that is OpenTelemetry’s job) and it does not draw your CPU graphs (that is Prometheus and Grafana). It ingests, stores, queries and visualises traces, and it is very good at it.

The problem before tracing

In a microservice platform, one user action fans out across a dozen HTTP and gRPC calls, two queues and three databases, each owned by a different team. Metrics tell you that p99 latency doubled. Logs tell you what one service said, in isolation, with no reliable way to correlate line 40,000 in service A with line 12 in service F. Neither answers the only question that matters during an incident: where did the time go? Traces answer it directly, because a trace is the causal shape of one request across every service it touched.

The vocabulary you must have

Five words carry the whole subject. A span is one unit of work — an HTTP handler, a database query, a queue publish — with a name, a start time, a duration, and a set of key/value tags (OpenTelemetry calls them attributes) plus timestamped logs (OTel calls them events). A trace is the whole tree of spans that belong to one request, tied together by a shared trace ID. Span context is the small packet — trace ID, span ID, sampling flag — that travels between services in a header so the next service knows which trace it belongs to. And spans relate to each other by references: CHILD_OF for the normal “the parent is waiting on me” case, and FOLLOWS_FROM for fire-and-forget work the parent did not wait for (a queue consumer, an async retry). Every one of these terms is also in the glossary.

Where it fits in a platform

☺ Like you’re 10: Jaeger sits at the end of the pipe. Other tools collect the timesheets; Jaeger files them and shows them to you.

In the layered model from Platform Architecture, Jaeger belongs to the observability plane, which is cross-cutting rather than a layer: it watches the substrate, the delivery plane and the workloads alike. Within that plane it owns exactly the traces pillar of the three described in Observability & Operations.

Its neighbours

Upstream sits OpenTelemetry, and the division of labour is worth memorising: OTel instruments and exports; Jaeger stores and visualises. Your application uses an OTel SDK to produce spans, ships them over OTLP to an OpenTelemetry Collector, and the Collector exports them onward to Jaeger. Beside Jaeger sits Prometheus, holding the metrics that tell you when to go looking, and Grafana, which can query Jaeger as a data source so a latency spike on a panel links straight to the offending trace. A service mesh — Istio or Linkerd — generates spans for every hop without touching application code, but still cannot propagate context through your process for you.

Why a platform team runs it centrally

Tracing is a textbook platform capability: enormously valuable, and something no single application team will stand up correctly alone. The platform runs one Jaeger, publishes one OTLP endpoint, ships a default sampling policy, and puts “your service is traced” on the golden path — so a team gets it by adopting the template, not by reading a tracing tutorial. That is self-service applied to observability, and it is the difference between three teams having traces and forty.

CNPE domain relevance

Jaeger is named explicitly on the official CNPE tool list and lives in Domain 4 — Observability & Operations (20%) on the exam blueprint. The pairing the exam loves is the chain: instrument with OpenTelemetry → export via the Collector → land in Jaeger → find the trace in the UI. Rehearse it on the observability practice tasks, where task O3 is precisely that wiring job.

How it works — architecture and components

☺ Like you’re 10: One part catches the timesheets, one part files them in a big cabinet, and one part draws the picture when you ask.

Jaeger is a small number of stateless services in front of one stateful store. Knowing which is which turns most Jaeger debugging into a two-minute job, because the logs you need live in exactly one of them.

The components, and the one that is deprecated

The jaeger-collector receives spans, validates them, runs them through a processing queue, and writes them to storage. The jaeger-query service reads from that storage and serves both the web UI and a JSON HTTP API on 16686. The jaeger-ingester exists only in the streaming deployment: the collector writes to Kafka instead of storage, and the ingester drains Kafka into storage, so a storage outage buffers instead of dropping. And the jaeger-agent — historically a per-host DaemonSet or sidecar that batched UDP thrift from clients — is deprecated and removed in Jaeger v2. Build nothing new around it; the OpenTelemetry Collector, run as a DaemonSet or sidecar, is the modern answer to the same problem.

OTLP is now the front door

Since v1.35 Jaeger accepts OTLP natively, on gRPC 4317 and HTTP 4318, exactly like any other OpenTelemetry backend. The old Jaeger client libraries were deprecated in 2022 in favour of the OpenTelemetry SDKs, and the Jaeger-native ingest ports are legacy. Jaeger v2 goes further and is built on the OpenTelemetry Collector framework — configured with receivers, processors, exporters and extensions, with Jaeger’s storage, query and remote sampling becoming collector extensions (jaeger_storage, jaeger_query, remote_sampling). Jaeger v1 reached end of life at the end of 2025, so v2 is the line to learn and to install; v1 is what you inherit, not what you choose. The ports, the vocabulary and the UI are unchanged across the two.

Storage backends — a real decision

Jaeger is deliberately storage-pluggable, and the choice is the difference between a demo and a platform. In-memory keeps everything in RAM and loses it on restart — fine for a lab, catastrophic in production. Badger is an embedded on-disk store: single node, survives a restart, still not a cluster. Elasticsearch and OpenSearch are the most common production choices, with rich tag search and index lifecycle management for retention. Cassandra is the other production option: excellent write throughput and native TTL, weaker ad-hoc tag querying. Whichever you pick, Jaeger’s query speed, retention window and monthly bill are really properties of the store, not of Jaeger.

The resources you will actually write

☺ Like you’re 10: Here is the actual YAML you type — one to run Jaeger, one to point the collector at it, one to decide how many timesheets to keep.

Three configs carry most real Jaeger work: the Jaeger deployment itself, the OpenTelemetry Collector that feeds it, and the sampling policy that keeps the volume sane.

A Jaeger instance via the operator

The Jaeger Operator introduces one CRD — Jaeger in the jaegertracing.io/v1 group — with three strategy values: allInOne (one pod, in-memory or badger; dev only), production (separate collector and query Deployments against a real store), and streaming (production plus Kafka and an ingester). Learn to read this manifest, because you will inherit it in existing clusters — but know its status: the Jaeger Operator manages Jaeger v1 only, and it is itself deprecated now that v1 is end-of-life. New installs use the Jaeger Helm chart, or run Jaeger v2 under the OpenTelemetry Operator, describing the deployment with a collector-style config file instead of this CRD.

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: platform-tracing
  namespace: observability
spec:
  strategy: production            # allInOne | production | streaming
  collector:
    replicas: 3                   # stateless — scale on ingest volume
    resources:
      limits: { cpu: "1", memory: 1Gi }
  query:
    replicas: 2                   # serves the UI/API on 16686
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: https://elasticsearch.observability.svc:9200
        index-prefix: jaeger
        num-shards: 3
        num-replicas: 1
    secretName: jaeger-es-credentials     # ES_USERNAME / ES_PASSWORD keys
    esIndexCleaner:
      enabled: true
      numberOfDays: 7             # RETENTION. Without this, indices grow forever.
      schedule: "55 23 * * *"
    dependencies:
      enabled: true               # the Spark job that builds the service dependency graph
      schedule: "@daily"

The OpenTelemetry Collector that feeds it

You rarely point applications straight at Jaeger. Point them at an OpenTelemetry Collector, which batches, limits memory, enriches with Kubernetes metadata, and only then exports onward — so you can change backends without redeploying forty services.

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: observability
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc: { endpoint: 0.0.0.0:4317 }   # apps send here
          http: { endpoint: 0.0.0.0:4318 }
    processors:
      memory_limiter:                        # ALWAYS first — protects the collector itself
        check_interval: 1s
        limit_percentage: 80
        spike_limit_percentage: 20
      k8sattributes: {}                      # add pod/namespace/node labels to every span
      batch:
        timeout: 5s
        send_batch_size: 1024
    exporters:
      otlp/jaeger:
        endpoint: platform-tracing-collector.observability.svc:4317
        tls: { insecure: true }              # in-cluster; use real TLS across trust domains
    service:
      pipelines:
        traces:
          receivers:  [otlp]
          processors: [memory_limiter, k8sattributes, batch]
          exporters:  [otlp/jaeger]

Note the exporter is plain otlp, not a Jaeger-specific one — the deprecated jaeger exporter was removed from the Collector distributions. Order matters in processors: memory_limiter first, batch last.

A sampling strategies file

Tracing every request at full volume is usually unaffordable. Jaeger’s collector serves a per-service sampling policy to clients over remote sampling (port 5778), so you tune sampling centrally in Git instead of redeploying services.

{
  "service_strategies": [
    {
      "service": "checkout",
      "type": "probabilistic",
      "param": 0.5,
      "operation_strategies": [
        { "operation": "/healthz", "type": "probabilistic", "param": 0.0 },
        { "operation": "POST /orders", "type": "probabilistic", "param": 1.0 }
      ]
    },
    {
      "service": "inventory",
      "type": "ratelimiting",
      "param": 20
    }
  ],
  "default_strategy": {
    "type": "probabilistic",
    "param": 0.001
  }
}

Mount that as a ConfigMap. In Jaeger v1 you point the collector at it with --sampling.strategies-file=/etc/jaeger/sampling.json; in v2 the same file is named by the remote_sampling extension’s file.path in the collector-style config. Read it carefully: the default keeps one request in a thousand, checkout keeps half, its health check is sampled at zero (never trace a probe), its order-creation endpoint at 100%, and inventory is capped at twenty traces per second regardless of load.

Day-to-day commands and the UI

☺ Like you’re 10: A few things you type to run Jaeger, open its website, and ask it for traces without clicking anything.

Stand one up, and reach it

# Fastest possible local Jaeger — UI on 16686, OTLP on 4317/4318, sampling on 5778
# (the v1 image was jaegertracing/all-in-one; v1 is end-of-life)
docker run --rm --name jaeger \
  -p 16686:16686 -p 4317:4317 -p 4318:4318 -p 5778:5778 \
  jaegertracing/jaeger:latest

# On a cluster, via the Helm chart. The chart deploys a single all-in-one pod
# and provisions no datastore, so the default is in-memory — configure storage
# before you rely on it.
helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm repo update
helm install jaeger jaegertracing/jaeger \
  --namespace observability --create-namespace

# Reach the UI without an Ingress. With this chart the Service is the release
# name; an operator-managed v1 install exposes INSTANCE-query instead.
kubectl -n observability port-forward svc/jaeger 16686:16686
# then open http://localhost:16686

# Is anything arriving? Check what the ingest side is doing.
kubectl -n observability logs deploy/jaeger --tail=50
kubectl -n observability get svc jaeger -o wide       # confirm 4317/4318 are exposed

Query it from the terminal

Everything the UI shows is a JSON API you can curl — which is how you assert “a trace arrived” in a test, a pipeline, or an exam task without a browser.

# Which services have ever reported a span?  (The first thing to check.)
curl -s localhost:16686/api/services | jq .

# Operations known for one service
curl -s "localhost:16686/api/services/checkout/operations" | jq .

# Recent traces for a service. Results come back newest first; with no
# start/end the API defaults to roughly the last two days.
curl -s "localhost:16686/api/traces?service=checkout&limit=20" | jq -r '.data[].traceID'

# Only traces that errored, taking longer than 2 seconds
curl -s "localhost:16686/api/traces?service=checkout&tags=%7B%22error%22%3A%22true%22%7D&minDuration=2s" | jq .

# One specific trace, in full
curl -s localhost:16686/api/traces/4bf92f3577b34da6a3ce929d0e0e4736 | jq .

# The service dependency graph, as data
curl -s "localhost:16686/api/dependencies?endTs=$(date +%s000)&lookback=86400000" | jq .

Reading the UI like an SRE

The UI is four screens and you should be fluent in all of them. Search takes a service, an operation, tag filters (error=true, http.status_code=500), a lookback window, and min/max duration — that last one is the professional’s shortcut, because “show me checkout traces over 2s in the last hour” finds the incident in one click. Trace detail is the Gantt/flame waterfall: each span is a bar indented under its parent, and you scan down for the bar visibly fatter than its siblings, then expand it to read tags and events. Two shapes are worth recognising instantly — one long bar with nothing inside it means time was spent in that service (or in an uninstrumented call it made), while a long stack of short sequential bars means an N+1 pattern. Compare diffs two trace IDs structurally, which is how you show the slow trace has 340 database spans and the fast one has 3. And System Architecture renders the service dependency DAG built from observed traces — the only honest architecture diagram you will ever have, because it comes from real traffic rather than a wiki nobody updated.

Gotchas and failure modes

☺ Like you’re 10: Here is why the timesheets sometimes don’t show up, and why it is almost never the filing cabinet’s fault.

“No traces” is almost never Jaeger

This is the single most valuable thing on the page. When the UI is empty, work the chain from the application outward; expect the fault in the first two links. One: is the app actually exporting — is OTEL_EXPORTER_OTLP_ENDPOINT set, is the SDK initialised, is a service name set (an empty service.name lands traces under unknown_service, which people then fail to find)? Two: is context propagated — does every service forward the incoming traceparent header onto its outbound calls? A hand-rolled HTTP client or a queue publish that drops headers severs the trace. Three: is the exporter aimed at a reachable endpoint with the right protocol and port — 4317 is gRPC, 4318 is HTTP, and crossing them produces a confident-looking connection that never delivers a span. Four: is sampling so low you simply have not caught one yet. Only after all four suspect the collector queue or storage. The general shape of this triage is in the troubleshooting playbook; if the pod itself is unhealthy start at workload triage.

Sampling hides the trace you came for

At 0.1% probabilistic sampling, the one failing request in ten thousand that a customer complained about is almost certainly not in Jaeger at all — and engineers conclude tracing is useless. The fixes, in order: raise the rate for that service while you investigate (remote sampling makes this a config change, not a deploy); set operation-level 100% sampling on the few endpoints that matter; or adopt tail sampling in the OTel Collector so every errored or slow trace is kept regardless of the head decision. Also worth knowing: Prometheus exemplars can hand you an exact trace ID for a slow request, letting you jump straight to a trace you would never have found by searching.

Retention and storage will surprise you

Nothing in Jaeger expires traces by itself. On Elasticsearch you need the index cleaner (or an ILM policy) or indices grow until the cluster falls over; on Cassandra you set a TTL; on badger you set a span-store TTL; on in-memory there is no retention at all. Meanwhile the collector has a bounded in-memory queue: when storage slows, the queue fills and spans are dropped silently from the application’s point of view. Scrape the collector’s own metrics and alert on dropped spans and queue length — Jaeger is a production service and deserves the same treatment as everything else in Reliability & Incidents.

The quieter traps

Clock skew between nodes makes child spans appear to start before their parents; the UI applies a skew adjustment and marks it, so check NTP before chasing a “time travelling” span. High-cardinality operation names — an order ID in the span name instead of a tag — destroy the operation dropdown and make search useless. An empty System Architecture tab usually means the dependency-building job never ran, not that your services do not talk. Traces stopping at the mesh boundary happen because a sidecar generates hop spans but cannot propagate context through your process — the application still forwards the header. And PII in tags is real: spans routinely capture full URLs with query strings, so redact in the Collector before it lands in a store your whole company can read.

Alternatives and when to choose it

☺ Like you’re 10: Other filing cabinets exist. Here is how to pick.

Because everything now speaks OTLP, the backend is a genuinely swappable decision — the whole point of instrumenting with OpenTelemetry. Choose on operations and cost, not on lock-in.

A practical rule

Choose Jaeger when you want an excellent self-hosted, vendor-neutral tracing UI and you already run — or will run — Elasticsearch or Cassandra. Choose Tempo when object-storage economics dominate and Grafana is already your single pane of glass. Either way, instrument with OpenTelemetry so the choice stays reversible. See The Tool Landscape for how tracing sits among the other named projects, and Observability & Operations for how traces combine with metrics and logs during a real incident.

Exam relevance and going further

☺ Like you’re 10: On exam day you can’t open Jaeger’s own website — so the wiring has to already be in your head.

Jaeger is named on the official CNPE tool list and sits in Domain 4 — Observability & Operations (20%). The realistic task shape is not “tune Jaeger’s storage” but “telemetry is being produced and nothing is arriving — connect it.” Expect to deploy or inspect a Jaeger instance, write or repair an OpenTelemetry Collector exporter pointing at it, port-forward 16686, and show that a trace for a named service is findable.

The documentation allowlist — read this twice

⚖ CNPA vs CNPE — That allowlist is a CNPE-only mechanic: CNPE is hands-on, so a narrow set of live lookups stays reachable mid-task. CNPA is stricter, not looser — a fully closed-book, multiple-choice exam with zero external resources and zero lookups of any kind, so neither jaegertracing.io nor kubernetes.io would be reachable there either. Even so, the concept-level knowledge above — the OTel/Jaeger split, the port map, why empty search results point to propagation before storage — is exactly the kind of thing CNPA's closed-book recall draws on.

What to be able to do without notes

Recall the ports cold: 16686 UI and query API, 4317 OTLP/gRPC, 4318 OTLP/HTTP, 5778 remote sampling. Write an OTel Collector traces pipeline with an otlp receiver, memory_limiter/batch processors and an otlp exporter aimed at jaeger-collector:4317 — that block is the most likely thing you will be asked to produce. Name the components (collector, query, ingester) and what the agent was deprecated in favour of. Define trace, span, span context, and CHILD_OF versus FOLLOWS_FROM. Explain the OTel/Jaeger split in one sentence. State the sampling types and why head-based sampling cannot preferentially keep errors. And say without hesitating that empty search results mean propagation or exporter configuration long before they mean a broken backend. Related commands live in the command reference.

Official resources for after the exam

Outside the exam, the canonical sources are jaegertracing.io/docs (the Architecture, Deployment and Sampling pages repay careful reading), the source at github.com/jaegertracing/jaeger, the Helm charts at github.com/jaegertracing/helm-charts, the project page at cncf.io/projects/jaeger, and the W3C propagation standard at w3.org/TR/trace-context. Pair this page with Observability & Operations for the three pillars, OpenTelemetry for the instrumentation half of the chain, and the glossary when a term stops making sense.