OpenTelemetry

OpenTelemetry is the vendor-neutral standard — with its APIs, SDKs, wire protocol and Collector — for producing traces, metrics and logs from your software once, in one shape, and shipping them anywhere. It solves the problem that used to define observability budgets: every backend wanted its own agent and its own library, so instrumentation was a permanent, expensive marriage to one vendor. With OpenTelemetry the instrumentation belongs to you, and the backend becomes a line in a config file you can change on a Tuesday afternoon.

What OpenTelemetry is and the problem it solves

☺ Like you’re 10: It’s one shared language for programs to say what they’re doing, plus a sorting office that delivers those messages wherever you want.

OpenTelemetry (usually written OTel) is a CNCF project formed from the merger of OpenTracing and OpenCensus. By contributor count it is the second-largest project in the CNCF, behind only Kubernetes itself — which tells you something useful about the exam: it is treated less as “a tool you might pick” and more as the assumed substrate of the observability domain.

The problem before OTel

Instrumentation used to be the stickiest lock-in in the industry. You chose a vendor, imported their library into every service, deployed their agent, and learned their attribute names. Changing vendor meant a code change in every repository — so nobody changed vendor, and pricing knew it. Worse, a polyglot estate ran three agents per node with three sets of field names, so a Java service and a Go service described the same HTTP request in different words and no dashboard could join them. OTel splits that knot: instrumentation becomes a standard you own and compile in, delivery becomes a config you change at will, and the application never learns the backend’s name.

The API / SDK split

This is the most misunderstood thing about OTel. The API is the tiny, stable surface your code (and every library author’s code) calls: start a span, record a measurement, read the current context. It is a no-op by default — if nothing configures an SDK, API calls do nothing and cost almost nothing. The SDK is the implementation you wire up at process start, deciding sampling, batching, resource attributes and destination. That separation is what lets a library author instrument their HTTP client without imposing a telemetry pipeline, or any runtime cost, on users who haven’t opted in.

Three signals, one context

OTel covers three signals. Traces describe one request as a tree of spans across services. Metrics are aggregated numbers — counters, gauges, histograms. Logs are timestamped records, and unusually OTel mostly bridges existing logging libraries rather than replacing them. (Profiling is the newest signal and still maturing.) What makes the three worth more together is that they share a resource and a context — the same service.name, the same trace and span IDs stamped on the log line — so “this error log” and “this slow trace” are one click apart.

Where it fits in a platform

☺ Like you’re 10: OTel is the plumbing between the apps and the dashboards — it doesn’t store anything, it just moves and tidies.

In the layered platform model OpenTelemetry occupies the observability plane, end to end: in-process libraries at the application edge, an agent on every node of the Kubernetes substrate, and a gateway tier in front of the storage backends. It is the one piece of platform machinery that every workload touches.

Its neighbours

Upstream sit your applications and the Istio or Linkerd sidecars, which emit spans of their own and — crucially — propagate trace headers between services. Alongside sits the Kubernetes API, which the Collector queries through its k8sattributes processor to stamp pod, namespace and deployment names onto every record. Downstream sit the stores: Prometheus for metrics, Jaeger or Tempo for traces, Loki for logs, with Grafana on top — and OpenCost and FinOps dashboards downstream of that, since cost attribution is only as good as your resource attributes.

Why platform teams reach for it

Three reasons, in the order teams discover them. Negotiating power: changing observability vendor becomes an exporter change, not a year-long migration. A golden path: the platform injects instrumentation automatically, so a team gets traces on day one without reading a tracing tutorial — a genuine developer-experience win. And a throttle: the Collector is the one place to drop noisy attributes, sample hard, or redact a field that should never have been recorded, without asking twenty teams to redeploy.

CNPE domain relevance

OpenTelemetry is on the official CNPE tool list and sits in Domain 4 — Observability & Operations (20%) alongside Prometheus and Grafana. It brushes Domain 1 too (it appears in every reference platform design) and the reliability material, because trace context is what turns a postmortem from archaeology into reading. The primary lesson here is Observability & Reliability; drill it with the observability practice tasks.

How it works — architecture and components

☺ Like you’re 10: The app writes messages, adds a little tag so messages about the same request stick together, and posts them to the sorting office.

Four moving parts: what the app produces, how requests stay stitched together across boundaries, how much you keep, and what the Collector does with the result.

Instrumentation and the OTLP wire

Three routes get telemetry out of an application, and most estates use all three. Manual instrumentation means calling the API in your own code — the only way to describe business logic (“this span is charge card”). Instrumentation libraries are ready-made wrappers for common frameworks — an HTTP server, a database driver — producing spans that follow the semantic conventions. Zero-code instrumentation attaches those libraries at startup without touching source: a -javaagent JAR for the JVM, the opentelemetry-instrument wrapper for Python, a --require preload for Node.js, the CLR profiler for .NET. eBPF-based instrumentation is the newest arrival, needing no runtime support at all at the cost of less semantic detail.

Whichever route you take, everything travels over OTLP, the OpenTelemetry Protocol. Two transports, and their default ports are worth memorising because you will type them constantly:

Set the transport with OTEL_EXPORTER_OTLP_PROTOCOL (grpc or http/protobuf). Mismatching the two — endpoint on 4318 while the SDK still speaks gRPC — is the most common “no data arrives, no error logged” failure there is.

Conventions, context propagation, and sampling

A resource is the set of attributes describing who is emitting: service.name (the one genuinely required attribute — omit it and most backends file you under unknown_service), service.version, deployment.environment.name, plus k8s.pod.name and friends. Semantic conventions are the agreed names for everything else — http.request.method, url.path, http.response.status_code, db.system.name. They are why a dashboard built for one service works for the next; inventing your own attribute names is legal and quietly ruinous.

Context propagation is how a trace survives a network hop. A span context — trace ID, span ID, trace flags — is serialised into the W3C traceparent header; the receiving service parses it and makes its spans children of yours. Baggage is a separate header carrying arbitrary key–value pairs alongside the trace: handy for a tenant ID, dangerous for anything secret, since it crosses every boundary in plaintext.

# The W3C Trace Context header, as it appears on the wire
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^^ ^                                ^                ^
             |  trace-id (16 bytes, 32 hex)      |                sampled flag (01 = keep)
             version                             parent span-id (8 bytes, 16 hex)

tracestate: acme=t61rcWkgMzE          # vendor-specific, optional
baggage:    tenant.id=acme,env=prod   # user-defined key/values that ride along

Choose propagators with OTEL_PROPAGATORS=tracecontext,baggage — the default in current SDKs, and what you want unless bridging to a legacy B3 or Jaeger-header estate.

The last lever is sampling — how much of all this you actually keep — and it comes in two flavours that sit at opposite ends of the pipeline. Head sampling decides at the very start of a trace, in the SDK, before any work happens. It is cheap and predictable — parentbased_traceidratio at 10% keeps a tenth of traces, and because the decision rides in the trace flags, every downstream service agrees. Its flaw is obvious: you cannot know in advance that this is the trace that will error.

Tail sampling decides after the whole trace has been seen, in the Collector, via the tail_sampling processor: keep everything that errored, everything slower than 500ms, and 5% of the boring rest. Far better data per stored byte — but it requires that every span of a trace reaches the same Collector instance, which is why it always implies a gateway tier with the loadbalancing exporter routing by trace ID, plus buffering memory proportional to traffic.

The Collector: five component types and the wiring

The Collector is a single Go binary (otelcol, shipped in Core, Contrib and Kubernetes distributions) that receives, transforms and forwards telemetry. Its config declares components — and then, separately, wires them into pipelines.

Deployment modes and the Operator’s CRDs

Two shapes, usually both. The agent runs as a DaemonSet, one per node: apps send to a short, reliable local hop, and only the agent can attach node-level attributes and read container logs. The gateway runs as a horizontally scaled Deployment behind a Service, centralising tail sampling, egress auth, backend fan-out and rate limiting. Agent → gateway → backend is the reference topology.

The OpenTelemetry Operator manages both, and the two of its CRDs you must be able to name are OpenTelemetryCollector (a Collector, with spec.mode of daemonset, deployment, statefulset or sidecar, and the config inline) and Instrumentation (an auto-instrumentation profile the Operator injects into annotated Pods via an init container and environment variables — zero code, zero rebuild). Note the naming convention: for a CR called otel-agent the Operator creates the workload and Service as otel-agent-collector. The standard release manifest expects cert-manager to issue the certificates for its admission webhooks.

The resources you will actually write

☺ Like you’re 10: Three files: one tells the sorting office what to do, one runs it on Kubernetes, one teaches your apps to speak without changing their code.

A Collector config — the shape to memorise

This is the plain config.yaml the otelcol binary reads; the same structure appears inside the CRD. When debugging someone else’s config, read the service: block first — it is the only part that determines what actually happens.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317      # bind 0.0.0.0, NOT localhost, or nothing off-pod reaches it
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:                   # ALWAYS first: sheds load before the process is OOMKilled
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  k8sattributes:                    # enrich with pod/namespace/deployment (needs RBAC to list pods)
    passthrough: false
    extract:
      metadata: [k8s.namespace.name, k8s.pod.name, k8s.deployment.name, k8s.node.name]
  resource:
    attributes:
      - key: deployment.environment.name
        value: prod
        action: upsert
  batch:                            # ALWAYS last: amortises network cost
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlphttp/gateway:
    endpoint: http://otel-gateway.observability.svc:4318   # 4318 because this is otlphttp
  debug:
    verbosity: detailed             # prints telemetry to stdout — for debugging only

extensions:
  health_check:                     # :13133 — wire this to your liveness probe
  zpages:                           # :55679/debug/pipelinez — live pipeline introspection

service:
  extensions: [health_check, zpages]          # extensions must be listed here to run
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, k8sattributes, resource, batch]
      exporters:  [otlphttp/gateway]
    metrics:
      receivers:  [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters:  [otlphttp/gateway]
    logs:
      receivers:  [otlp]
      processors: [memory_limiter, batch]
      exporters:  [otlphttp/gateway]
  telemetry:
    logs:
      level: info

The otlphttp/gateway naming matters: type/name lets you declare several instances of one component type and point different pipelines at each. And note that debug is declared but unused — declaring it costs nothing, and adding it to a pipeline’s exporters list is the fastest way to see whether data is arriving at all.

The gateway, as an OpenTelemetryCollector

Here the Operator builds the Deployment, Service and ConfigMap for you. This gateway demonstrates two things worth stealing: tail sampling, and the spanmetrics connector generating RED metrics from traces, so you get request/error/duration series without instrumenting them separately.

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-gateway
  namespace: observability
spec:
  mode: deployment            # daemonset | deployment | statefulset | sidecar
  replicas: 3
  resources:
    limits: { memory: 2Gi }
  config:
    receivers:
      otlp:
        protocols:
          grpc: { endpoint: 0.0.0.0:4317 }
          http: { endpoint: 0.0.0.0:4318 }
    connectors:
      spanmetrics: {}           # traces in → RED metrics out
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 20
      tail_sampling:
        decision_wait: 10s      # how long to hold a trace before deciding
        num_traces: 100000
        policies:
          - name: keep-errors
            type: status_code
            status_code: { status_codes: [ERROR] }
          - name: keep-slow
            type: latency
            latency: { threshold_ms: 500 }
          - name: sample-the-rest
            type: probabilistic
            probabilistic: { sampling_percentage: 5 }
      batch: { timeout: 10s, send_batch_size: 8192 }
    exporters:
      otlp/jaeger:
        endpoint: jaeger-collector.observability.svc:4317
        tls: { insecure: true }
      prometheusremotewrite:
        endpoint: http://prometheus.observability.svc:9090/api/v1/write
    service:
      pipelines:
        traces:
          receivers:  [otlp]
          processors: [memory_limiter, tail_sampling, batch]
          exporters:  [otlp/jaeger, spanmetrics]   # connector used AS an exporter here
        metrics:
          receivers:  [otlp, spanmetrics]          # ...and AS a receiver here
          processors: [memory_limiter, batch]
          exporters:  [prometheusremotewrite]

Zero-code instrumentation with the Instrumentation CRD

This is what makes OTel a platform capability rather than a per-team project: the platform ships one Instrumentation object per namespace, and app teams opt in with a single Pod annotation.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: default
  namespace: payments
spec:
  exporter:
    endpoint: http://otel-agent.observability.svc:4318   # http/protobuf → port 4318
  propagators:
    - tracecontext        # W3C traceparent
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "0.1"       # head-sample 10% of root traces
  java:
    env:
      - name: OTEL_EXPORTER_OTLP_PROTOCOL
        value: http/protobuf
  python:
    env:
      - name: OTEL_EXPORTER_OTLP_PROTOCOL
        value: http/protobuf
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  namespace: payments
spec:
  replicas: 2
  selector:
    matchLabels:
      app: checkout          # selector and template labels MUST match, or it will not apply
  template:
    metadata:
      labels:
        app: checkout
      annotations:
        # "true" = the Instrumentation named "default" in THIS namespace.
        # Also valid: "observability/default" (cross-namespace) or "false" (opt out).
        instrumentation.opentelemetry.io/inject-java: "true"
    spec:
      containers:
        - name: checkout
          image: ghcr.io/acme/checkout:1.4.3
          env:
            - name: OTEL_SERVICE_NAME          # set it explicitly; do not rely on defaults
              value: checkout
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: service.version=1.4.3,deployment.environment.name=prod

The Operator’s mutating webhook adds an init container that copies the Java agent into a shared volume, then injects JAVA_TOOL_OPTIONS and the OTLP environment. The annotation only takes effect at Pod creation — adding it to a running Deployment does nothing until the Pods are recreated.

Day-to-day commands

☺ Like you’re 10: A short list of things you type to install it, check the sorting office is working, and prove a message got through.

Install

# The Operator needs cert-manager for its webhooks
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml
kubectl -n cert-manager rollout status deploy/cert-manager-webhook

# Install the OpenTelemetry Operator
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
kubectl -n opentelemetry-operator-system rollout status deploy/opentelemetry-operator-controller-manager

# Or Helm, if you prefer values files
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm upgrade --install otel-agent open-telemetry/opentelemetry-collector \
  --namespace observability --create-namespace \
  --set mode=daemonset --set image.repository=otel/opentelemetry-collector-contrib

kubectl get opentelemetrycollectors,instrumentations -A     # short names: otelcol, otelinst

Validate and introspect the Collector

# Static config check before you ship it (the binary, e.g. in a CI job or a container)
otelcol validate --config=file:/etc/otelcol/config.yaml
otelcol components                     # list every receiver/processor/exporter in THIS build

# Is it healthy, and is data actually moving?
# NOTE: substitute your own Service name below — the Operator calls it <cr-name>-collector,
# the Helm chart calls it <release>-opentelemetry-collector.
kubectl -n observability logs -l app.kubernetes.io/instance=otel-agent --tail=100
kubectl -n observability port-forward svc/otel-agent 13133:13133 &
curl -s localhost:13133

# The Collector's own metrics — the numbers that answer "where is my data being dropped?"
kubectl -n observability port-forward svc/otel-agent 8888:8888 &
curl -s localhost:8888/metrics | grep -E 'otelcol_(receiver|exporter|processor)'
#   otelcol_receiver_accepted_spans     ← arriving
#   otelcol_receiver_refused_spans      ← rejected (usually memory_limiter back-pressure)
#   otelcol_exporter_sent_spans         ← leaving
#   otelcol_exporter_send_failed_spans  ← backend rejecting or unreachable
#   processor-level counters exist too, but their names have churned across
#   Collector releases — grep the prefix rather than memorising one metric.

# Live pipeline view without redeploying anything
kubectl -n observability port-forward svc/otel-agent 55679:55679 &
open http://localhost:55679/debug/pipelinez

Prove the path end to end

# Configure any SDK entirely by environment — no code change
export OTEL_SERVICE_NAME=checkout
export OTEL_RESOURCE_ATTRIBUTES="service.version=1.4.3,deployment.environment.name=dev"
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-agent.observability.svc:4318
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf     # must MATCH the port you chose
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1
export OTEL_PROPAGATORS=tracecontext,baggage

# Zero-code launch wrappers
opentelemetry-instrument python app.py                       # Python
java -javaagent:/otel/opentelemetry-javaagent.jar -jar app.jar   # Java

# Hand-post one span over OTLP/HTTP+JSON — the fastest "is the receiver alive?" test
kubectl -n observability port-forward svc/otel-agent 4318:4318 &
curl -i -X POST http://localhost:4318/v1/traces \
  -H 'Content-Type: application/json' \
  -d '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"curl-test"}}]},
       "scopeSpans":[{"spans":[{"traceId":"5b8efff798038103d269b633813fc60c",
       "spanId":"eee19b7ec3c1b174","name":"hello","kind":1,
       "startTimeUnixNano":"1700000000000000000","endTimeUnixNano":"1700000000100000000"}]}]}]}'
# HTTP 200 with an empty success body means the Collector accepted it; a populated
# "partialSuccess" object means some spans were rejected, and it says why.

Gotchas and failure modes

☺ Like you’re 10: Here are the ways it goes quiet on you — and OTel’s failures are almost always silent, not loud.

The component that isn’t in a pipeline

Once more, because it is the number one bug: a receiver, processor or exporter present in the top-level config but absent from service.pipelines does nothing at all, with no warning. Symptoms are “I configured the filter and it isn’t filtering,” “tail sampling isn’t sampling,” “the second exporter never gets data.” The same trap applies to extensions and service.extensions. When a Collector behaves as if a component is missing, read the service: block first.

Port and protocol mismatches

The 4317/4318 pairing catches everyone. Point an SDK configured for grpc at 4318 (or http/protobuf at 4317) and you get connection errors buried in SDK debug output, or a hang. Two variants worth knowing: the otlp exporter is gRPC and otlphttp is HTTP — different components, not aliases; and the signal-specific OTEL_EXPORTER_OTLP_TRACES_ENDPOINT needs the full /v1/traces path spelled out, while the generic OTEL_EXPORTER_OTLP_ENDPOINT has it appended for you. Check too that the receiver binds 0.0.0.0, not localhost — see networking triage when connections are refused across a Pod boundary.

Traces that stop at a boundary

A trace that ends abruptly almost always means propagation was lost. Usual suspects: a proxy, gateway or broker that strips the traceparent header; an async hop (queue, cron, batch) where nobody carried context into the message; an uninstrumented HTTP client; or two services on different propagators (one B3, one W3C), each speaking a header the other ignores. The diagnostic is blunt and effective — log inbound headers at the boundary and look for traceparent. Absent means the sender is at fault; present but ignored means the receiver’s SDK isn’t extracting it.

Cost, cardinality, and the OOMKilled Collector

Telemetry is data, and data is a bill. Attribute cardinality: a user ID, a request ID, or a URL with IDs in the path placed on a metric attribute detonates your time-series count — high-cardinality identifiers belong on spans and logs, never on metrics. Memory: a Collector without memory_limiter first in every pipeline gets OOMKilled under a traffic spike and drops everything while it restarts — check restarts with workload triage. Processor order: batch belongs last, after any sampling, or you pay to batch data you were about to discard.

Alternatives and when to choose it

☺ Like you’re 10: There are other ways to collect this stuff, but most of them now speak OTel’s language too.

OpenTelemetry has largely won the instrumentation layer — every serious vendor ingests OTLP — so the real question is which pieces you adopt and what you pair them with, not whether to adopt it at all.

The comparison that decides it

A practical rule

Instrument with OpenTelemetry, run the Collector, and choose your storage separately and reversibly. Use head sampling early (cheap, simple) and graduate to tail sampling only once you have the gateway tier and the traffic to justify it. See The Tool Landscape for how OTel sits among the other named projects, and Observability & Reliability for what to do with the data once it lands.

Exam relevance and going further

☺ Like you’re 10: On exam day you can’t open OTel’s website — so the ports, the pipeline shape and the CRD names have to already be in your head.

OpenTelemetry is on the official CNPE tool list, in the observability domain. Expect to deploy or repair a Collector, read a config and say what it does (and what it silently doesn’t), fix a pipeline that drops data, annotate a workload for auto-instrumentation, or explain how trace context crosses a boundary. Config reading is the likeliest shape: OTel configs are long, and a small omission changes everything — excellent exam material. See the exam guide for the blueprint and the observability practice tasks for reps.

The documentation allowlist — read this twice

⚖ CNPA vs CNPE — That allowlist is a CNPE-only mechanic: CNPE is hands-on, so a narrow set of live lookups stays reachable mid-task. CNPA is stricter, not looser — a fully closed-book, multiple-choice exam with zero external resources and zero lookups of any kind, so neither opentelemetry.io nor kubernetes.io would be reachable there either. Even so, the concept-level knowledge above — the four Collector config blocks, the OTLP ports, head versus tail sampling — is exactly the kind of thing CNPA's closed-book recall draws on.

What to be able to do without notes

Write a minimal Collector config from a blank file with all four blocks — receivers, processors, exporters, service.pipelines — knowing the last is what makes the others real. State the OTLP ports (4317 gRPC, 4318 HTTP) and the signal paths. Name the five component types. Put memory_limiter first and batch last, and say why. Name the two Operator CRDs above and the four collector modes, and recall the annotation form instrumentation.opentelemetry.io/inject-<language>: "true". Explain head vs tail sampling and why tail sampling needs trace affinity. Name the header (traceparent) and the required resource attribute (service.name). Keep the CLI spine — otelcol validate, :8888 metrics, :13133 health, :55679/debug/pipelinez — in the command reference.

Official resources for after the exam

Outside the exam the canonical sources are opentelemetry.io/docs (the Collector Configuration and Language APIs & SDKs sections repay careful reading), the semantic conventions at opentelemetry.io/docs/specs/semconv, the component catalogue at opentelemetry-collector-contrib, the Operator repo, the W3C standard at w3.org/TR/trace-context, and cncf.io/projects/opentelemetry. Pair this page with Observability & Reliability for the concepts, Jaeger and Prometheus for the stores, and the glossary when a term stops making sense.