Drill — Trace a Broken Telemetry Pipeline

A teammate adds a filter processor to your OpenTelemetry Collector, meant to strip the constant liveness- and readiness-probe spans before they ever reach the tracing backend — cheap, sensible cost control. The config applies cleanly. The Collector reports Ready. Nothing anywhere logs an error, warns, or so much as raises an eyebrow. And weeks later the backend's storage bill has crept up in a way nobody can explain, because the filter that was supposed to be doing real work has, this whole time, been running for absolutely nothing — declared in the config, never actually wired into the pipeline that processes spans. This drill hands you that exact silence and asks you to find it using nothing but two of the Collector's own metrics, the same evidence a real on-call engineer would reach for first. It's fully self-contained: a throwaway kind cluster and a single Collector, no dependency on the capstone. Budget 25-40 minutes before you open the walkthrough below.

Declared vs wired — the bug hiding in plain sight

☺ Like you're 10: A rule taped to the wall doesn't count until it's on today's actual checklist — the Collector obeys the checklist, never the wall.

Every OpenTelemetry Collector config does two genuinely separate things. The top-level receivers:, processors: and exporters: blocks only declare components — they say "here is a thing that exists and here is how it's configured," nothing more. Separately, service.pipelines wires a subset of those declared components into an actual signal pipeline, in an explicit, ordered list. A component that's declared but never named inside service.pipelines is fully valid YAML, passes every schema check, and does precisely nothing at runtime — no warning at startup, no error in the logs, no difference in the Collector's Ready status. It just quietly never runs.

Build the scratch world

☺ Like you're 10: One cluster, cert-manager, the Operator, one Collector — all thrown away when you're done.

Stand up a fresh kind cluster and the OpenTelemetry Operator, which needs cert-manager installed first for its admission webhooks:

kind create cluster --name pipeline-drill
kubectl create namespace observability

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml
kubectl -n cert-manager rollout status deploy/cert-manager-webhook

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
kubectl -n opentelemetry-operator-system rollout status deploy/opentelemetry-operator-controller-manager

Now the Collector itself. It's meant to filter out /healthz and /readyz spans before they leave the cluster — the exact OTTL filter processor pattern, correctly written. Save this as collector.yaml. Notice the filter/drop-health-checks processor is fully declared with both its where-style conditions, and then look closely at the pipeline at the bottom:

# collector.yaml — BROKEN, as shipped
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: pipeline-drill
  namespace: observability
spec:
  mode: deployment
  replicas: 1
  image: otel/opentelemetry-collector-contrib:latest   # contrib build — filter needs it
  config:
    receivers:
      otlp:
        protocols:
          grpc: { endpoint: 0.0.0.0:4317 }
          http: { endpoint: 0.0.0.0:4318 }
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 80
        spike_limit_percentage: 25
      filter/drop-health-checks:
        error_mode: ignore
        traces:
          span:
            - 'attributes["http.route"] == "/healthz"'
            - 'attributes["http.route"] == "/readyz"'
      batch:
        timeout: 5s
        send_batch_size: 8192
    exporters:
      debug:
        verbosity: normal
    service:
      telemetry:
        metrics:
          level: detailed        # exposes the Collector's own metrics on :8888
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch]   # filter/drop-health-checks isn't here
          exporters: [debug]

Apply it and wait for it to come up. The Operator names the Deployment and Service <cr-name>-collector:

kubectl apply -f collector.yaml
kubectl -n observability rollout status deploy/pipeline-drill-collector

kubectl -n observability port-forward svc/pipeline-drill-collector 4318:4318 >/tmp/otlp-pf.log 2>&1 &
kubectl -n observability port-forward svc/pipeline-drill-collector 8888:8888 >/tmp/metrics-pf.log 2>&1 &

Generate mixed traffic — signal and noise, together

☺ Like you're 10: Send exactly fifteen spans that should get dropped and five that shouldn't, so the right answer is a number you already know.

Define a small helper that posts one span over OTLP/HTTP with a chosen http.route, then fire fifteen fake health-check spans and five fake real-request spans — the same shape of traffic a checkout service under a Kubernetes liveness probe actually produces:

send_span() {
  local route="$1"
  local trace_id span_id now end
  trace_id=$(openssl rand -hex 16)
  span_id=$(openssl rand -hex 8)
  now=$(date +%s%N)
  end=$((now + 5000000))
  curl -s -o /dev/null -X POST http://localhost:4318/v1/traces \
    -H 'Content-Type: application/json' \
    -d '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"checkout"}}]},
         "scopeSpans":[{"spans":[{"traceId":"'"$trace_id"'","spanId":"'"$span_id"'",
         "name":"'"$route"'","kind":2,
         "attributes":[{"key":"http.route","value":{"stringValue":"'"$route"'"}}],
         "startTimeUnixNano":"'"$now"'","endTimeUnixNano":"'"$end"'"}]}]}]}'
}

for i in $(seq 1 15); do send_span "/healthz";  done
for i in $(seq 1 5);  do send_span "/checkout"; done

Twenty spans total, sent once, straight into the receiver. If filter/drop-health-checks were actually wired in, exactly fifteen of them should never make it out the other side.

Diagnose it from two counters, nothing else

☺ Like you're 10: Don't reopen the YAML yet. Ask the Collector's own metrics whether anything is actually being thrown away.

Scrape the Collector's self-telemetry — no dashboard, no backend query, just the raw Prometheus text the Collector itself exposes on :8888:

curl -s localhost:8888/metrics | grep -E 'otelcol_(receiver_accepted_spans|exporter_sent_spans)'

otelcol_receiver_accepted_spans{receiver="otlp",transport="http"} 20
otelcol_exporter_sent_spans{exporter="debug"} 20

Twenty accepted, twenty sent — a perfect one-to-one match. That match is the entire bug, stated in two numbers. If filter/drop-health-checks were running, fifteen of those twenty spans should have been discarded before they ever reached the exporter, and sent_spans should read 5, not 20. It doesn't. Nothing crashed, nothing logged an error, the receiver accepted every span exactly as it should — the gap that should exist between "accepted" and "sent" simply never opened.

Underneath that second bar chart the arithmetic is simple and worth doing by hand once: a fifteen-span gap between a bar that stopped at 5 and one that reached 20 is exactly the fifteen /healthz spans you sent — not roughly, not approximately, exactly, because you know precisely what you fed the receiver.

Confirm against the resource, then fix it

☺ Like you're 10: The metrics told you something's wrong. The resource's own spec tells you exactly what.

The metrics prove a component isn't dropping anything. Confirm which one, and why, straight from the object Kubernetes is actually running:

kubectl -n observability get opentelemetrycollector pipeline-drill -o yaml | grep -A6 'pipelines:'

    pipelines:
      traces:
        exporters:
        - debug
        processors:
        - memory_limiter
        - batch

There it is: filter/drop-health-checks is declared, fully configured, sitting right there in the CR's own processors: block — and it is absent from this list. Add it, in the position that actually matters: after memory_limiter (so an overloaded Collector still sheds load first) and before batch (so you don't spend CPU batching spans you're about to throw away):

# collector.yaml — the fix
    service:
      telemetry:
        metrics:
          level: detailed
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, filter/drop-health-checks, batch]
          exporters: [debug]

Reapply, and let the Operator roll the Deployment:

kubectl apply -f collector.yaml
kubectl -n observability rollout status deploy/pipeline-drill-collector

Prove the fix with the same two counters

☺ Like you're 10: Run the identical batch again and watch the gap between the two numbers finally open up.

Because both metrics are cumulative Prometheus counters, note the baseline before you send anything new — it should still read the old totals from the broken run:

curl -s localhost:8888/metrics | grep -E 'otelcol_(receiver_accepted_spans|exporter_sent_spans)'
# otelcol_receiver_accepted_spans{receiver="otlp",transport="http"} 20
# otelcol_exporter_sent_spans{exporter="debug"} 20   ← unchanged since the redeploy, as expected

for i in $(seq 1 15); do send_span "/healthz";  done
for i in $(seq 1 5);  do send_span "/checkout"; done

curl -s localhost:8888/metrics | grep -E 'otelcol_(receiver_accepted_spans|exporter_sent_spans)'
# otelcol_receiver_accepted_spans{receiver="otlp",transport="http"} 40   (+20 — the receiver still accepts everything)
# otelcol_exporter_sent_spans{exporter="debug"} 25                       (+5 — only the /checkout spans got through)

Done when: the second batch's exporter delta is exactly +5, the receiver's delta is exactly +20, and the gap between them — 15 — matches the number of /healthz spans you sent, exactly. Optionally, confirm a second way: port-forward 55679 and open /debug/pipelinez to see filter/drop-health-checks now listed as an active stage of the traces pipeline, not just a declared, orphaned component.

The transferable habit

☺ Like you're 10: When nothing errors and nothing looks broken, count what's actually happening before you trust what's declared.

The fix today was one word in one list — adding filter/drop-health-checks where it was always supposed to be. The skill worth keeping is the sequence that found it: don't trust a config file's stated intent, measure a known input against the Collector's own output, notice exactly where the numbers disagree with what a correctly wired pipeline should produce, and only then go confirm the precise cause in the resource's own spec. That sequence generalizes past filters completely — a tail_sampling processor that never trims a bill, a k8sattributes processor whose enrichment never shows up on a span, a second exporter that never receives a single record. Every one of those is the identical shape: a component sitting fully configured and completely silent, findable only by counting, never by reading.

Fixed and confirmed? Good — that's the whole drill. For the full component model this bug lives inside, see OpenTelemetry Collector; for the exam that puts 26% of its weight on exactly this domain, see OTCA — the exam. For the data model underneath every span you generated today, see The OpenTelemetry Data Model; for where metrics like these eventually land in a real fleet, see The Prometheus Model. Ready for a different single skill? Try Drill — Lock Down a Mesh Namespace or Drill — Write an Enforcing Kyverno Policy, or go build the full loop in Build Your Cert Tracker — Start Here.