OpenTelemetry

OpenTelemetry is the CNCF-governed, vendor-neutral standard for producing traces, metrics, and logs from your own code — an agreed shape for telemetry, not a place to store or view it. It formed in 2019 when the CNCF merged two competing projects, OpenTracing and OpenCensus, into one specification with one set of SDKs, and it has since grown into one of the CNCF's most active projects by contributor count, commonly cited as second only to Kubernetes itself. For an SRE team the appeal isn't the acronym — it's economic: instrument a service once, against a neutral API that no vendor controls, and the traces that come out can be shipped to a self-hosted Jaeger, to a commercial platform like Honeycomb or Datadog, or to whatever replaces either of them in three years — by editing a config file, not by touching a single line of application code.

What OpenTelemetry is — and, pointedly, what it refuses to be

☺ Like you're 10: It's a rulebook two rival standards merged into, run by a committee no single company controls — and rule one is that it never keeps a single thing it collects.

OpenTelemetry (almost always written OTel) exists because instrumentation used to be the stickiest form of vendor lock-in in the industry. Before it, you picked an observability vendor, imported their proprietary library into every service, deployed their agent, and learned their attribute names. Changing vendor meant a code change in every repository, so almost nobody changed vendor — pricing knew it, and renewal conversations reflected that leverage. OTel's governance is built specifically to prevent that from happening again: it's hosted by the CNCF, its specification is set by a technical committee drawn from dozens of competing companies (AWS, Google, Microsoft, and yes, commercial observability vendors themselves all have seats), and it graduated to the CNCF's top project-maturity tier in 2021 — a small club that also includes Kubernetes and Prometheus. No single vendor can quietly bend the standard toward their own product, because their competitors are in the same room deciding it.

OTel covers three kinds of telemetry, called signals. Traces describe one request as a tree of connected spans across every service it touched. Metrics are aggregated numbers — counters, gauges, histograms — the same shapes monitoring and observability already builds golden-signal dashboards from. Logs are timestamped records, and unusually OTel mostly bridges existing logging libraries rather than replacing them outright. What makes the three worth more together than apart is that they share a common resource — the same service.name, the same trace and span IDs stamped onto the log line — so a slow trace and the error log it produced are one click apart instead of two separate archaeology projects.

The architecture: API, SDK, OTLP, and the Collector

☺ Like you're 10: Your code calls a thin, do-nothing-by-default API; a separate piece called the SDK does the real work; and everything travels to a router called the Collector over one shared wire protocol.

The most misunderstood thing about OTel is the split between its API and its SDK, and it's worth getting straight before anything else clicks. The API is the tiny, stable surface your code — and every library author's code — calls to start a span or record a measurement. By design it is a no-op until something configures an SDK: if nobody wires one up, API calls cost almost nothing and go nowhere. The SDK is the implementation you actually install and configure at process start, and it's where the real decisions live — sampling rate, batching, which resource attributes get attached, and where the data goes. That split is what lets a library author instrument their own HTTP client without forcing a telemetry pipeline, or any runtime cost, onto every user who hasn't opted in.

Whichever route the telemetry takes to get out of the process, it travels over OTLP, the OpenTelemetry Protocol — the one wire format every SDK, Collector, and OTLP-compatible backend agrees on.

The Collector is the piece that turns "a standard exists" into "you can actually act on it." It's a single, self-contained binary (otelcol) that receives OTLP, reshapes it in flight, and forwards it onward — and its config is built from four kinds of named component: receivers pull data in, processors reshape it (batch it, strip a field, enrich it with Kubernetes metadata), exporters push it out to one or more backends, and extensions add capabilities like a health-check endpoint that touch no telemetry at all. Declaring a component isn't the same as running it — nothing executes until it's named in a service.pipelines block, a distinction the Collector config section below makes concrete.

Manual spans vs. auto-instrumentation

☺ Like you're 10: You can hand-write exactly what a piece of code means, or you can attach a wrapper at startup that describes the obvious stuff automatically without you touching the code at all — most real fleets do both.

Getting telemetry out of a running service happens two ways, and a mature SRE org uses both deliberately rather than picking one. Manual instrumentation means calling the OTel API directly in your own code, and it's the only route that can describe business meaning — a span literally named charge-card with an attribute for the payment provider, something no automatic wrapper could infer. Auto-instrumentation attaches ready-made spans for common frameworks — an HTTP server, a database driver, a message queue client — at process startup, without a single line of source code changing: a -javaagent JAR for the JVM, the opentelemetry-instrument wrapper for Python, a --require preload for Node.js, the CLR profiler for .NET. The newest route, eBPF-based instrumentation, needs no runtime hooks at all — useful for a legacy fleet nobody wants to touch — at the cost of shallower, less business-meaningful spans.

# Manual span — describes something auto-instrumentation could never know on its own
from opentelemetry import trace

tracer = trace.get_tracer("checkout-service")

def charge_card(order):
    with tracer.start_as_current_span("charge-card") as span:
        span.set_attribute("payment.provider", order.provider)
        span.set_attribute("payment.amount_cents", order.amount_cents)
        result = payment_gateway.charge(order)
        if not result.ok:
            span.set_status(trace.Status(trace.StatusCode.ERROR, result.reason))
        return result

# Auto-instrumentation — zero code changes, attached at process start
opentelemetry-instrument python app.py                              # Python
java -javaagent:/otel/opentelemetry-javaagent.jar -jar checkout.jar # Java
node --require @opentelemetry/auto-instrumentations-node/register app.js  # Node.js

The practical rule most SRE teams converge on: turn auto-instrumentation on everywhere as a baseline — it's what makes an unfamiliar service traceable on day one of an incident — and add manual spans only where the automatic ones don't say enough, which is almost always at a business or domain boundary rather than a network one.

A Collector config that ships one trace to two backends

☺ Like you're 10: This is the actual file that proves the whole promise — the same trace, sent to two different filing cabinets, from one small block of config.

Here's a minimal but genuinely production-shaped Collector config. Read the service.pipelines block first — it's the only part that decides what actually runs, and it's deliberately routing the same traces pipeline to two exporters at once: a self-hosted Jaeger and a commercial Honeycomb, side by side, so the fan-out from the schematic above isn't hypothetical.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317      # bind 0.0.0.0, not localhost, or nothing off-host reaches it
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:                   # ALWAYS first: sheds load before the process gets OOM-killed
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  resource:
    attributes:
      - key: deployment.environment.name
        value: prod
        action: upsert
  batch:                            # ALWAYS last: amortises network cost
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlp/jaeger:                                    # self-hosted, inside our own cluster
    endpoint: jaeger-collector.observability.svc:4317
    tls: { insecure: true }
  otlphttp/honeycomb:                             # commercial, outside our cluster
    endpoint: https://api.honeycomb.io
    headers: { "x-honeycomb-team": "${env:HONEYCOMB_API_KEY}" }

extensions:
  health_check:                     # :13133 — wire this to your liveness probe

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, resource, batch]
      exporters:  [otlp/jaeger, otlphttp/honeycomb]   # BOTH — the hedge, in one line

Day-to-day: running and proving the pipeline

☺ Like you're 10: A short list of things you type to start the router, ask it if it's alive, and prove a message actually got through.

# Run the Collector locally against the config above
docker run --rm -p 4317:4317 -p 4318:4318 -p 13133:13133 \
  -v "$(pwd)/config.yaml:/etc/otelcol/config.yaml" \
  otel/opentelemetry-collector-contrib:latest \
  --config=/etc/otelcol/config.yaml

# Validate a config before you ship it — catches typos before a 3am surprise
otelcol validate --config=file:./config.yaml

# Is it up?
curl -s localhost:13133

# The Collector's own metrics — the numbers that answer "where is my data being dropped?"
curl -s localhost:8888/metrics | grep -E 'otelcol_(receiver|exporter)'
#   otelcol_receiver_accepted_spans     ← arriving from the SDK
#   otelcol_exporter_sent_spans         ← leaving toward a backend
#   otelcol_exporter_send_failed_spans  ← a backend rejecting or unreachable

# Hand-post one span over OTLP/HTTP+JSON — the fastest "is anything listening?" test
curl -i -X POST http://localhost:4318/v1/traces \
  -H 'Content-Type: application/json' \
  -d '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"curl-test"}}]},
       "scopeSpans":[{"spans":[{"traceId":"5b8efff798038103d269b633813fc60c",
       "spanId":"eee19b7ec3c1b174","name":"hello","kind":1,
       "startTimeUnixNano":"1700000000000000000","endTimeUnixNano":"1700000000100000000"}]}]}]}'
# HTTP 200 with an empty body means the Collector accepted it. Check the Jaeger UI and
# Honeycomb's query view a moment later — the same trace ID should be sitting in both.

Gotchas and failure modes

☺ Like you're 10: Almost every way this breaks is quiet, not loud — nothing crashes, data just stops arriving somewhere and nobody notices for a while.

Beyond the declared-but-not-wired trap above, four failure modes account for most of the pages this pipeline generates. Port and protocol mismatches: point an SDK configured for gRPC at port 4318, or an HTTP client at 4317, and you get connection errors buried in SDK debug output rather than an obvious error — check OTEL_EXPORTER_OTLP_PROTOCOL matches the port every time. A missing service.name: it's the one resource attribute every backend genuinely requires, and its absence doesn't fail loudly — it files everything under unknown_service, which is how a team discovers three "mystery" services in their trace view that turn out to be one real service with a typo. Cardinality explosions: a user ID or a raw URL path placed on a metric attribute (not a span or log attribute) can multiply your time-series count by orders of magnitude and blow through a backend's ingest quota or your own bill overnight — high-cardinality identifiers belong on spans and logs, never on metrics. Broken context propagation: a trace that ends abruptly at a service boundary almost always means the W3C traceparent header got dropped — by a proxy that strips unknown headers, by a message queue where nobody carried context into the message body, or by two services on different propagators (one legacy B3, one W3C) that don't understand each other's header. The diagnostic is blunt: log inbound headers at the boundary and check whether traceparent is present.

The hedge: why adopt OTel before you've even picked a backend

☺ Like you're 10: Deciding to speak the shared language now costs you almost nothing today — and it's what makes changing your mind about the filing cabinet, later, cheap instead of catastrophic.

The argument for OpenTelemetry rarely comes down to any one feature; it comes down to what economists would call optionality. A team that instruments with a proprietary vendor's library has, whether they've thought about it that way or not, made a multi-year bet on that vendor's pricing, roadmap, and continued existence — and the exit cost isn't the contract, it's re-instrumenting every service by hand. A team that instruments with OpenTelemetry has deferred that bet indefinitely: the backend becomes a Collector exporter block, changeable in a pull request, reversible, and testable by running two exporters side by side — exactly what the config above does — before cutting traffic over for real. That option is worth adopting now, even for a team perfectly happy with its current backend, because the cost of adopting it later — after every service has grown its own bespoke, vendor-specific spans — is dramatically higher than the cost of adopting it up front.

See the SRE toolchain for how OpenTelemetry sits next to the backends it can point at — Jaeger, Honeycomb, Zipkin, Datadog — and monitoring and observability for what to actually do with the telemetry once it lands somewhere. Readers who want to formalize this into a credential can look at the vendor-neutral OpenTelemetry Certified Associate (OTCA) — as with any certification, verify current format and pricing on the Linux Foundation's own training page before committing.