Honeycomb

Honeycomb is a commercial observability platform built on a bet its founders — Christine Yen and Charity Majors, both formerly of the mobile backend Parse — made explicitly against the industry's default architecture. Instead of splitting telemetry into three separately-stored signals (pre-aggregated metrics, narrow log lines, and sampled traces), Honeycomb stores one thing: a wide, structured event per unit of work — one row per HTTP request, one row per span — carrying however many fields you choose to attach to it, with no limit on how many distinct values any field can take. That single decision is this page's throughline. It's what makes BubbleUp, Honeycomb's signature debugging workflow, possible at all, and it's why the tool is worth understanding as a different architecture, not just a different vendor selling the same three pillars.

What Honeycomb is and the problem it solves

☺ Like you're 10: Most tools make you guess your questions before the incident happens and write a dashboard for each one. Honeycomb keeps everything so you can ask new questions after the incident starts.

Honeycomb was founded in 2016, and its founders' pitch was blunt: the "three pillars" of observability — metrics, logs, and traces, stored separately, in separately-optimized systems — is an architecture that makes sense for the tools that grew up around it, not for the engineer trying to debug an unfamiliar failure at 3am. A metrics system throws away the individual events the moment it computes an aggregate; a log system is usually narrow, unstructured, and grep-shaped; a tracing system shows you one request's timeline but rarely lets you ask "which other requests looked like this one, and what did they have in common?" across millions of them. Honeycomb's answer, and the idea Charity Majors and her co-authors later formalized in the book Observability Engineering (O'Reilly), is that the real test of observability isn't which signals you collect — it's whether you can answer a question about a failure mode you've never seen before, using data you already have, without shipping new instrumentation first. Monitoring & observability and Monitoring & Service Level Indicators cover that distinction in general; this page covers the specific architecture Honeycomb built to pass that test.

The product surface built on top of that architecture is smaller than a suite like Datadog's: one event store, one Query Builder, Boards (saved queries/dashboards), Triggers (alerts), and SLOs computed directly against the raw event stream. There's no separate metrics product and no separate logs product bolted on — everything is a query over the same wide events, which is a deliberate simplicity, not a missing feature.

The wide-event data model: one event, arbitrary width, arbitrary cardinality

☺ Like you're 10: Instead of writing five separate short notes about one request, you write one long note with everything on it — who it was, where it ran, how long it took, and anything else that might matter later.

A wide event is one structured record — in practice a JSON-shaped object with dozens to hundreds of key-value fields — representing one unit of work: an HTTP request, a background job execution, or one span in a distributed trace. Rather than emitting a counter increment, a gauge, and a separate unstructured log line for the same request, an instrumented service attaches everything it knows about that request onto one event as it's handled: the route, the status code, the duration, the user ID, the pricing-experiment cohort, the availability zone, the database shard, the cache-hit outcome, the build SHA that's running. Some teams call the practice of building one of these per request a canonical log line — a pattern Honeycomb's own writing has done a lot to popularize even though the term predates the company — and the discipline is the same either way: don't scatter twelve narrow log lines through a request's code path, accumulate one wide one and emit it at the end.

That's the mechanical reason a user_id, an order ID, a Kubernetes pod name, or a customer's account ID are all perfectly normal fields to put on a Honeycomb event, where the same fields would be a well-known way to take down a Prometheus server or blow up a Datadog bill. High-cardinality fields aren't a hazard here — they're the whole point, because they're exactly the fields that let you isolate one weird customer, one bad pod, or one misbehaving order out of millions of otherwise-identical requests.

Architecture: the columnar store and the query fan-out

☺ Like you're 10: Nothing gets summarized when it arrives — every detail is kept, sorted into neat columns, and a fleet of machines races through those columns whenever you ask a question.

Honeycomb's backend — described in its own engineering writing under the internal name Retriever — is a purpose-built, columnar event store, not a repurposed metrics or log database. Incoming events are written into per-column, compressed segments rather than pre-aggregated at write time, and a query fans out across a distributed set of query nodes, each scanning its own segments in parallel and streaming partial results back to be merged. Nothing about which fields are "indexed" or "important" is decided in advance — every column is queryable the same way, which is precisely what makes an ad hoc group-by on a field nobody planned to graph both possible and reasonably fast.

The tradeoff that follows is symmetrical, and worth stating plainly rather than as a marketing claim: a metrics system pushes cost to write time (aggregate once, query the small aggregate forever) and pays for that with inflexibility after the fact — you cannot retroactively slice a percentile you never recorded a breakdown for. Honeycomb pushes cost to query time (store everything raw, scan for every question) and pays for that with query cost that scales with data volume and time range, mitigated by columnar compression, parallel fan-out, and — critically — sampling before ingest, covered under Refinery below. Neither approach is free; they're different bets about which cost you'd rather carry.

Instrumenting a service: OpenTelemetry, not a proprietary SDK

☺ Like you're 10: You don't need Honeycomb's own toolkit — you use the same standard toolkit everyone else does, and just tell it to mail its notes to Honeycomb's address.

Honeycomb shipped its own instrumentation libraries early on, called Beelines, but has since deprecated them in favor of standard OpenTelemetry — the same SDKs and the same OTLP wire protocol that Jaeger, Zipkin, and every commercial APM vendor now speak. In practice that means pointing an unmodified OTel SDK's OTLP exporter at Honeycomb's endpoint with an API key header, with no vendor-specific code in the application:

# environment variables any standards-compliant OTel SDK already reads —
# no Honeycomb-specific import, no Beeline, no vendor lock-in at the SDK layer
export OTEL_SERVICE_NAME=checkout
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=${HONEYCOMB_API_KEY}"
# EU-region accounts point at https://api.eu1.honeycomb.io instead — check your account's region

Honeycomb's current model automatically creates one dataset per unique service.name it sees arrive on an API key's environment, so there's no separate dataset-provisioning step for a standard OTLP trace — the resource attribute you're already sending does that job. (Honeycomb's older Events API, for anything not going through OTLP, still accepts an explicit x-honeycomb-dataset header; the OTLP path above is the one to reach for on any new service.)

The instrumentation habit that actually matters — the one that produces a wide event worth BubbleUp-ing later — is adding business and infrastructure context onto the span OTel's auto-instrumentation already created, rather than treating that span as finished the moment the HTTP framework populated it:

from opentelemetry import trace

def handle_checkout(request):
    span = trace.get_current_span()
    # OTel auto-instrumentation already attached http.route, http.status_code,
    # and the request's duration. Everything below turns it from a narrow
    # trace span into a genuinely WIDE event.
    span.set_attribute("app.user_id", request.user.id)
    span.set_attribute("app.cart_value_cents", request.cart.total_cents)
    span.set_attribute("app.shard_id", request.user.shard)
    span.set_attribute("app.feature_flag.new_pricing", is_enabled("new_pricing", request.user))
    span.set_attribute("cloud.availability_zone", CURRENT_AZ)
    span.set_attribute("app.build_sha", BUILD_SHA)
    # no separate log line, no separate metric emission — this IS the event

The BubbleUp debugging workflow

☺ Like you're 10: You circle the weird dots on a graph, and the tool instantly tells you what those dots have in common that the normal dots don't.

This is the feature the wide-event model exists to make possible. The workflow starts the same way any investigation does — a heatmap of, say, request duration over time, with one dot per event. When something looks wrong — a cluster of dots sitting well above the rest — you drag a selection box around just that cluster. BubbleUp then automatically compares every field on the events inside your selection against every field on the events outside it (the baseline), and returns a ranked list of which fields differ most between the two groups: perhaps aws.az is 94% us-east-1c inside the selection and 12% outside it, or app.shard_id is overwhelmingly a single shard, or a specific db.statement shows up almost exclusively among the slow requests. You didn't write a query for any of those specific fields in advance — you asked one open-ended question, "what's different about these," and got an answer computed across every field on the event, not just the ones you happened to have graphed.

The reason a system built on pre-aggregated metrics structurally cannot reproduce this, even in principle, is what got thrown away at write time. Once a metrics pipeline has computed "p99 latency for checkout in this one-minute bucket was 340ms," the individual requests that produced that number are gone — there is no underlying event to go back and group by aws.az, because the aggregate never carried aws.az as a dimension in the first place, and even a metric that was tagged by availability zone only tells you the AZ's own aggregate, not which specific slow requests shared some other unrecorded trait. A dashboard, however well built, only ever answers the questions its author anticipated when they chose which fields to graph and group by — what SRE literature calls known unknowns. BubbleUp is built for unknown unknowns: the dimension nobody thought to alert on, discovered after the fact, from data that was never thrown away.

SLOs, Triggers, and Boards: the operational surface

☺ Like you're 10: The alert rules and the reliability scorecard both read from the exact same raw events you'd use for BubbleUp, so when one fires you're never more than one click from the events that caused it.

An SLO in Honeycomb is defined against a Derived Column — a small boolean expression evaluated over raw event fields, computed at query time rather than stored — that says whether a given event met the service level indicator: for instance, "duration under 300ms and not an error." Because it's derived at query time from data already stored, a derived column is retroactive in a way a metrics-based SLI generally isn't — define a new SLI expression today and Honeycomb can immediately evaluate it against every historical event still in retention, with no need to have deployed anything ahead of time to start "measuring" it. SLIs, SLOs & error budgets covers the underlying math this feeds, and SLO windows & composite SLOs covers rolling-vs-calendar windows that apply the same way here as anywhere else.

A Trigger is Honeycomb's alert rule — a saved query with a threshold, evaluated on a schedule, that notifies a destination (Slack, PagerDuty, a webhook) when crossed — the direct equivalent of a Datadog Monitor or a Prometheus alerting rule. The property worth knowing is that an SLO's burn-rate Trigger and BubbleUp share the same underlying data: a page that fires because the error budget is burning too fast links straight back to the exact events currently violating the SLI, one click from "we're paging" to "here's what's actually different about the requests failing the SLO" — see multi-window, multi-burn-rate alerting for the alerting design this composes with. A Board is Honeycomb's dashboard object: a saved collection of queries, the pre-built-views half of the architecture diagram above.

These objects are managed either by hand in the UI or, at any real scale, through Honeycomb's Terraform provider, reviewed in pull requests the same way infrastructure is:

resource "honeycombio_derived_column" "checkout_sli" {
  dataset     = "checkout"
  alias       = "sli_fast_and_ok"
  expression  = "AND(LT($duration_ms, 300), NOT($error))"
  description = "true when a checkout request met the SLI"
}

resource "honeycombio_slo" "checkout_availability" {
  name              = "checkout: 99.9% fast-and-ok"
  dataset           = "checkout"
  sli               = honeycombio_derived_column.checkout_sli.alias
  target_percentage = 99.9
  time_period_days  = 30
}

resource "honeycombio_trigger" "checkout_burn_rate" {
  name    = "checkout SLO burn rate — page"
  dataset = "checkout"
  query_json = jsonencode({
    calculation = [{ op = "COUNT" }]
    filter      = [{ column = "sli_fast_and_ok", op = "=", value = false }]
  })
  threshold { op = ">", value = 50 }
  frequency = 900
  recipient { type = "pagerduty_recipient" }
}

Day-to-day commands and workflows

☺ Like you're 10: There's no big command-line tool to learn — most day-to-day work is either clicking through the Query Builder, calling a well-documented web API, or letting Terraform manage the reviewed stuff.

Honeycomb doesn't ship a heavyweight bespoke CLI the way some observability vendors do; the day-to-day surface is the UI's Query Builder for exploration, a REST API for anything scripted or automated, and Terraform for anything that should be reviewed and versioned:

# drop a deploy marker so every graph in the dataset shows exactly when this shipped —
# the fastest way to answer "did the deploy cause this" from inside a BubbleUp session
$ curl https://api.honeycomb.io/1/markers/checkout \
    -H "X-Honeycomb-Team: ${HONEYCOMB_API_KEY}" \
    -d '{"message":"checkout v1.4.4","type":"deploy"}'

# run a saved query headlessly — e.g. from a runbook step or a CI gate — and get raw results back
$ curl https://api.honeycomb.io/1/queries/checkout \
    -H "X-Honeycomb-Team: ${HONEYCOMB_API_KEY}" \
    -d @query.json

# list every Trigger currently configured on a dataset — "what can actually page me"
$ curl https://api.honeycomb.io/1/triggers/checkout \
    -H "X-Honeycomb-Team: ${HONEYCOMB_API_KEY}"

# the reviewed, versioned path for SLOs/Triggers/Boards — see the Terraform block above
$ terraform plan -target=honeycombio_slo.checkout_availability
$ terraform apply

The actual investigative workflow — build a heatmap, group by a suspect field, drag a BubbleUp selection, read the ranked comparison, drill into individual events, drop a marker once you understand what shipped — happens in the UI itself, not at a terminal, which is a genuine departure from most of the tools elsewhere in the SRE toolchain.

Gotchas and failure modes

☺ Like you're 10: Almost every disappointment with this tool comes from either treating it like a narrow logging tool, or sampling away the exact request you needed later.

Sampling has to be tail-based, or you throw away the outlier you need

At real production traffic volumes, ingesting every single raw event is often not affordable, so Honeycomb's open-source companion project, Refinery, sits in front of ingest as a fleet of proxies that make a keep-or-drop decision per trace — critically, after the trace has finished, so the decision can depend on how the request actually turned out:

# refinery_rules.yaml (illustrative — verify current syntax against Refinery's own docs)
Samplers:
  __default__:
    RulesBasedSampler:
      Rules:
        - Name: "keep every error, unsampled"
          Conditions:
            - Field: error
              Operator: "="
              Value: true
          SampleRate: 1
        - Name: "keep every slow request, unsampled"
          Conditions:
            - Field: duration_ms
              Operator: ">"
              Value: 300
          SampleRate: 1
        - Name: "sample the boring, fast, successful stuff"
          SampleRate: 20   # keep roughly 1 in 20

A naive, head-based, random sample rate applied uniformly — the kind a metrics pipeline uses without much consequence — is actively dangerous here: if the one weird slow request you'd need for BubbleUp had, say, a 1-in-20 chance of being kept regardless of how interesting it was, most incidents' most interesting evidence never arrives at all. Tail-based sampling that always keeps errors and slow requests, and only thins out the boring, fast, successful traffic, is what keeps the data BubbleUp needs while still controlling cost — get this backwards and the tool quietly stops working exactly when you need it most.

Pricing is per event ingested, not per host or per metric series

Honeycomb's billing model is shaped around event volume (with a free tier and paid tiers gated on ingest volume and retention, subject to change — verify current figures on Honeycomb's own pricing page rather than trusting a number here). The consequence is the mirror image of a Datadog cardinality bill: a wide event with two hundred fields costs the same to ingest as a narrow one with five, because dimensionality isn't what's billed — but raw request volume at a high-traffic service, unsampled, gets expensive fast, which is exactly why Refinery exists as a first-class part of the architecture rather than an afterthought.

Raw retention is finite, and it's tied to your plan

Because nothing is pre-aggregated, "how far back can I query" is a direct function of how much raw data your plan retains, not an independent knob the way a metrics system's long-term-storage tier (Thanos, Mimir, Cortex) can be sized separately from its hot-path retention. A postmortem written a month after an incident may find the specific raw events already aged out, which is worth knowing before you promise a stakeholder "we can BubbleUp that from three months ago."

The workflow is genuinely proprietary

OpenTelemetry softens vendor lock-in at the instrumentation layer — the same OTLP traffic can be routed to Honeycomb, Jaeger, or a competitor by editing a Collector config, not by re-instrumenting application code, exactly as OpenTelemetry covers. What doesn't travel is the analysis workflow: Query Builder, Boards, Triggers, and BubbleUp itself are Honeycomb-specific constructs with no open equivalent, so switching vendors means rebuilding the saved queries, dashboards, and alert rules even though the underlying telemetry pipeline barely changes.

Honeycomb vs. the metrics-plus-traces split

☺ Like you're 10: One tool keeps every detail and searches through it fresh each time; the other set of tools decides in advance what to remember and forgets the rest.

Most organizations don't choose one architecture exclusively — a very common pattern is exactly the fan-out shown on OpenTelemetry's own page: the same OTel Collector routes traffic to a cheap, well-understood metrics backend like Prometheus for the four golden signals and always-on dashboards, while also routing to Honeycomb for the wide-event stream an engineer actually opens once an incident is underway and the question has stopped being "is it broken" and started being "why."

The honest framing, the same one reliability economics applies elsewhere on this course: neither architecture is strictly better, they're optimized for different questions, and the real decision is which one your team is actually reaching for during an incident. A team that has excellent Prometheus dashboards but still can't answer "why is it slow for this one customer" in under twenty minutes is paying the cost of the metrics-only architecture whether or not it shows up as a line item.

Where Honeycomb fits in the SREF blueprint

☺ Like you're 10: The exam wants you to recognize "this is a high-cardinality event-based observability tool" from a description, not to know which button does what inside it.

The DevOps Institute SRE Foundation (SREF) exam is closed-book and tests tool categories, not vendor specifics, as SRE Tools & Automation covers in full — Honeycomb is one of the representative tools listed for distributed tracing and high-cardinality observability, alongside OpenTelemetry, Jaeger, and Zipkin, and Monitoring & Service Level Indicators is the domain actually being tested. What's worth knowing cold for the exam is the conceptual distinction this page is built around — high-cardinality, high-dimensionality event data vs. pre-aggregated metrics, and why that distinction changes what kind of question you can answer after the fact — not the Terraform resource names or the Refinery config syntax above, which are for the job, not the test.