OpenTelemetry Collector

The OTCA blueprint puts 26% of its weight on the Collector alone — more than any other single domain on that exam — and for a simple reason: it's the one piece of an OpenTelemetry pipeline that is pure infrastructure rather than application code, which makes it the platform team's problem the moment more than one service exists. This page is that infrastructure made concrete: the five component types a config declares, the one blunt rule that decides whether any of them actually run, the agent-vs-gateway topology nearly every real deployment converges on, the OTTL transforms that let you redact and reshape telemetry centrally instead of chasing twenty application teams, the commands that prove a pipeline is alive, and the failure modes that turn a valid-looking YAML file into a Collector that does nothing. It assumes the data model and SDK side of the story — Resource, spans, OTLP on the wire — are settled elsewhere; this one starts where most real incidents start: a Collector, its config, and the gap between what it declares and what it runs.

Five component types, and the rule that decides what actually runs

☺ Like you're 10: Five kinds of building block, one blunt rule: nothing runs unless a checklist further down the file explicitly names it.

The Collector is a single Go binary — otelcol, shipped as Core, Contrib and a Kubernetes-flavoured distribution with the k8s-aware components already in — that receives telemetry, reshapes it, and sends it on. Its config splits neatly into declaring components and, separately, wiring them into pipelines, and mixing those two steps up is the single most common way to lose an afternoon.

Connectors are worth pausing on because they're the least intuitive of the five: spanmetrics above is simultaneously the exporter of the traces pipeline and a receiver of the metrics pipeline — it watches spans go by, derives request/error/duration counters from them, and hands those counters into a completely separate pipeline as if they'd arrived over the wire. That's how you get Prometheus-shaped RED metrics without instrumenting them separately anywhere in application code — a genuinely free byproduct of tracing you've already paid for.

Agent vs gateway: the topology nearly every fleet converges on

☺ Like you're 10: One relay rides along with every crew member for the short, cheap hop; one bigger relay sits above the whole fleet doing the expensive jobs that only work if everything passes through the same place.

Two deployment shapes, and most production platforms run both. The agent runs as a DaemonSet, one Collector per node: application pods send OTLP to a short, reliable local hop, and only the agent can attach node-level context — pod name, namespace, node name — because it's the only thing that actually lives on that node. The gateway runs as a horizontally scaled Deployment behind a Service that every agent forwards to, centralising the work that's either expensive or needs a whole-fleet view: tail sampling, egress authentication to a paid backend, rate limiting, and fan-out to multiple destinations. agent → gateway → backend is the reference topology, and it maps directly onto the OTLP ports you'll type constantly:

The OpenTelemetry Operator manages both shapes through one CRD: OpenTelemetryCollector, whose spec.mode is one of daemonset, deployment, statefulset or sidecar, with the whole otelcol config inline under spec.config. For a CR named otel-agent the Operator creates the workload and Service as otel-agent-collector — a naming convention worth knowing before you go hunting for a Service that isn't spelled the way you expect. Its second CRD, Instrumentation, is what lets application teams opt into zero-code tracing with a single Pod annotation rather than a library import; that's the API/SDK side of the story, and Platform Engineering's OpenTelemetry page covers it end to end.

The config you actually write

☺ Like you're 10: One file: what to listen on, what to tidy up and in what order, and where to send the tidy version.

This is the plain config.yaml the otelcol binary reads — an agent shape, small and local. The same four blocks appear again, inline, inside every OpenTelemetryCollector CR.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317      # bind 0.0.0.0, not localhost, or no pod off-node reaches it
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:                   # ALWAYS first: sheds load before the process gets OOMKilled
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  k8sattributes:                    # enrich with pod/namespace/node — needs RBAC to list pods
    passthrough: false
    extract:
      metadata: [k8s.namespace.name, k8s.pod.name, k8s.deployment.name, k8s.node.name]
  batch:                            # ALWAYS last: amortises the network cost of small telemetry
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlp/gateway:
    endpoint: otel-gateway.observability.svc:4317   # forward on, unencrypted inside the mesh
    tls: { insecure: true }
  debug:
    verbosity: detailed             # prints telemetry to stdout — for debugging only

extensions:
  health_check:                     # :13133 — wire this to your liveness probe
  zpages:                           # :55679/debug/pipelinez — live pipeline introspection

service:
  extensions: [health_check, zpages]         # extensions must ALSO be listed here to run
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters:  [otlp/gateway]
    metrics:
      receivers:  [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters:  [otlp/gateway]
    logs:
      receivers:  [otlp]
      processors: [memory_limiter, batch]
      exporters:  [otlp/gateway]

Note that debug is declared but sits outside every exporters list above — deliberately, to make the point stick: it costs nothing to declare, and adding it to one pipeline's exporters is the fastest way to see whether telemetry is arriving at all, before you go looking anywhere else.

The gateway tier is the same shape with two extra pieces doing the fleet-wide work — tail sampling and the spanmetrics connector — deployed through the Operator so the Deployment, Service and ConfigMap come free:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-gateway
  namespace: observability
spec:
  mode: deployment                  # daemonset | deployment | statefulset | sidecar
  replicas: 3
  resources:
    limits: { memory: 2Gi }
  config:
    receivers:
      otlp:
        protocols:
          grpc: { endpoint: 0.0.0.0:4317 }
          http: { endpoint: 0.0.0.0:4318 }
    connectors:
      spanmetrics: {}                # traces in → RED metrics out
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 20
      tail_sampling:
        decision_wait: 10s           # how long to hold a trace before deciding
        num_traces: 100000
        policies:
          - name: keep-errors
            type: status_code
            status_code: { status_codes: [ERROR] }
          - name: keep-slow
            type: latency
            latency: { threshold_ms: 500 }
          - name: sample-the-rest
            type: probabilistic
            probabilistic: { sampling_percentage: 5 }
      batch: { timeout: 10s, send_batch_size: 8192 }
    exporters:
      otlp/backend:
        endpoint: jaeger-collector.observability.svc:4317
        tls: { insecure: true }
      prometheusremotewrite:
        endpoint: http://prometheus.observability.svc:9090/api/v1/write
    service:
      pipelines:
        traces:
          receivers:  [otlp]
          processors: [memory_limiter, tail_sampling, batch]
          exporters:  [otlp/backend, spanmetrics]   # connector used AS an exporter here
        metrics:
          receivers:  [otlp, spanmetrics]           # ...and AS a receiver here
          processors: [memory_limiter, batch]
          exporters:  [prometheusremotewrite]

Two naming details worth carrying: type/name component IDs — otlp/backend, otlp/gateway — let a config declare several instances of one component type and point different pipelines at each; and a metrics store fed by prometheusremotewrite here is the same store the Prometheus model pulls into shape — the Collector is pushing into a system whose native instinct is to pull, which is exactly the kind of impedance mismatch worth understanding rather than just configuring around.

OTTL transforms: reshaping telemetry centrally

☺ Like you're 10: A tiny find-and-replace language for messages passing through — redact a word, add a tag, decide a message isn't worth keeping.

OTTL, the OpenTelemetry Transformation Language, is what the transform and filter processors and the routing connector all speak underneath. Every OTTL statement runs against a declared context — span, resource, metric, datapoint, or log — and calls small functions like set(), delete_key(), keep_keys() and truncate_all(), optionally guarded by a where clause so the statement only fires on records that match. This is the Collector's superpower and the reason it's a platform capability rather than a per-team library: one config change reshapes telemetry for every service behind it, with no application redeploy.

processors:
  transform/redact-and-tag:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          # strip a header that should never have left the process
          - delete_key(attributes, "http.request.header.authorization")
          # explicit error status from a status code the client library didn't set
          - set(status.code, STATUS_CODE_ERROR) where attributes["http.response.status_code"] >= 500
      - context: resource
        statements:
          # only keep the identity fields dashboards actually key on
          - keep_keys(attributes, ["service.name", "service.version", "k8s.namespace.name"])

  filter/drop-health-checks:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.route"] == "/healthz"'
        - 'attributes["http.route"] == "/readyz"'

Wire both into the traces pipeline — [memory_limiter, transform/redact-and-tag, filter/drop-health-checks, batch] — and every service behind this Collector stops leaking bearer tokens and stops paying to store liveness-probe noise, without a single line of application code changing. That's also why filter and transform belong before batch: batching data you're about to drop wastes the exact CPU you're trying to save.

Installing and upgrading with the Operator

☺ Like you're 10: A couple of commands install the manager that reads your Collector recipes; from then on, editing the recipe is the whole job.

The Operator's admission webhooks need cert-manager installed first. Once it's running, an OpenTelemetryCollector or Instrumentation object is just another manifest — which means the whole fleet, agent config included, is one reviewable YAML tree, manageable through Argo CD or Flux the same way any other GitOps-managed workload on this course is.

# The Operator needs cert-manager for its webhooks
$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml
$ kubectl -n cert-manager rollout status deploy/cert-manager-webhook

# Install the OpenTelemetry Operator
$ kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
$ kubectl -n opentelemetry-operator-system rollout status deploy/opentelemetry-operator-controller-manager

# Or Helm, if a values file suits your GitOps flow better
$ helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
$ helm upgrade --install otel-agent open-telemetry/opentelemetry-collector \
    --namespace observability --create-namespace \
    --set mode=daemonset --set image.repository=otel/opentelemetry-collector-contrib

$ kubectl get opentelemetrycollectors,instrumentations -A   # short names: otelcol, otelinst

Day-to-day commands: proving a pipeline is alive

☺ Like you're 10: A short list of things you type to check the sorting office is open, healthy, and actually moving mail.

# Static config check before you ship it — catches typos and missing pipeline references
$ otelcol validate --config=file:/etc/otelcol/config.yaml
$ otelcol components                      # list every receiver/processor/exporter in THIS build

# Is it up? (NOTE: the Operator names the Service <cr-name>-collector; Helm uses <release>-opentelemetry-collector)
$ kubectl -n observability port-forward svc/otel-agent-collector 13133:13133 &
$ curl -s localhost:13133

# The Collector's OWN metrics — the numbers that answer "where is my data being dropped?"
$ kubectl -n observability port-forward svc/otel-agent-collector 8888:8888 &
$ curl -s localhost:8888/metrics | grep -E 'otelcol_(receiver|exporter)'
#   otelcol_receiver_accepted_spans     ← arriving
#   otelcol_receiver_refused_spans      ← rejected, usually memory_limiter back-pressure
#   otelcol_exporter_sent_spans         ← leaving
#   otelcol_exporter_send_failed_spans  ← backend rejecting or unreachable

# Live pipeline view, no redeploy needed
$ kubectl -n observability port-forward svc/otel-agent-collector 55679:55679 &
$ open http://localhost:55679/debug/pipelinez

# Hand-post one span over OTLP/HTTP+JSON — the fastest "is the receiver even alive?" test
$ kubectl -n observability port-forward svc/otel-agent-collector 4318:4318 &
$ curl -i -X POST http://localhost:4318/v1/traces \
    -H 'Content-Type: application/json' \
    -d '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"curl-test"}}]},
         "scopeSpans":[{"spans":[{"traceId":"5b8efff798038103d269b633813fc60c",
         "spanId":"eee19b7ec3c1b174","name":"hello","kind":1,
         "startTimeUnixNano":"1700000000000000000","endTimeUnixNano":"1700000000100000000"}]}]}]}'
# HTTP 200 with an empty body: accepted. A populated "partialSuccess" object means
# some spans were rejected — and it says exactly why.

Gotchas and failure modes

☺ Like you're 10: Nearly every surprise here is quiet: a forgotten checklist entry, a mismatched pair of numbers, the wrong order, or one identifier in the wrong place.

The component that isn't in a pipeline

Worth a third mention because it really is the number one bug: a receiver, processor, exporter or connector present in the top-level config but absent from service.pipelines does nothing at all, with no warning. Symptoms read like everything else: "I configured the filter and nothing's filtered," "tail sampling isn't sampling," "the second exporter never gets data." The same trap applies to extensions and service.extensions.

Port and protocol mismatches

4317/4318 catches everyone eventually. Point an SDK or an agent's forwarding exporter configured for grpc at 4318, or http/protobuf at 4317, and you get connection errors buried in debug-level logs or a plain hang — never a clean, obvious failure. The otlp exporter is gRPC and otlphttp is HTTP; they are different components, not aliases for the same thing with a flag.

Processor order: memory_limiter first, batch last

A Collector without memory_limiter first in every pipeline gets OOMKilled under a traffic spike and drops everything mid-restart, instead of shedding load gracefully. Put batch anywhere before a sampling or filtering processor and you pay to batch data you were about to throw away — order the chain so the cheap, protective step runs first and the amortising step runs last, with everything that decides what to keep in between.

Cardinality: identifiers belong on spans, never on metrics

A user ID, a request ID, or a raw URL with IDs still in the path, attached to a metric attribute, detonates your time-series count — each unique value mints a brand-new series that a metrics store has to store forever. High-cardinality identifiers are exactly what spans and logs are for; keep them off anything that becomes a metric label, and use the keep_keys or delete_key OTTL functions above to enforce it centrally rather than trusting every team to remember on their own.

The Collector vs the alternatives

☺ Like you're 10: A few other sorting offices exist. They trade off how many kinds of mail they handle, and how much they already speak everyone else's language.

OpenTelemetry has effectively won the wire format — nearly every serious backend ingests OTLP now — so the real decision is rarely "OTel or not," it's how much of the Collector's centralising power a given team actually needs yet. See Platform Engineering's OpenTelemetry page for the SDK, sampling and context-propagation half of this story, Prometheus and Grafana for where the metrics this Collector exports actually land, and The OpenTelemetry Data Model for why the Resource/Scope/record shape is built the way it's built in the first place.