Hubble
Cilium's eBPF datapath already inspects every packet to decide whether it's allowed — routing, load balancing and policy enforcement all happen by reading that same flow. Hubble is what you get when someone points at that inspection point and asks it to also keep a record. It's the observability layer built directly into Cilium: instead of bolting a separate agent onto every pod or sniffing traffic on the side, Hubble reads flow events straight out of the eBPF maps the agent was already populating, and turns them into a queryable, cluster-wide history of who talked to whom, over what protocol, and whether the datapath let it through. This page is the tool-guide companion to Cilium and to CCA's Network Observability domain: the architecture behind hubble-relay and hubble-ui, the CiliumNetworkPolicy rules that actually turn on Layer 7 parsing, the hubble observe filter language you reach for in every real incident, the metrics that feed Prometheus and Grafana, and the gotchas that catch people who assume the CLI is cluster-wide the moment they type it.
Picture mission control's radio room. Every call between a capsule and the ground already has to pass through one switchboard, because that's the only way to route it to the right console — so the operator is already looking at who's calling and who they're calling. Hubble is what happens when you hand that operator a logbook and say: "since you're already looking anyway, write it down — who called whom, on which channel, and whether the call actually connected, or got cut off." Nobody built a second radio room just to keep that log. It's the same switchboard, the same operator, just also taking notes. That's why Hubble is nearly free to run: Cilium was never going to stop inspecting your traffic, so recording what it saw costs almost nothing extra.
Architecture: from eBPF datapath to a cluster-wide flow API
☺ Like you're 10: One recorder rides inside every node's agent, keeping its own local diary. One fan-in service turns "every node's diary" into a single diary anyone can query.
Hubble has no separate DaemonSet of its own — it lives inside cilium-agent, one instance per node, reading flow events out of the same eBPF maps the agent already populates to enforce policy. Each of those local Hubble servers keeps a small, in-memory, per-node ring buffer of recent flows and exposes them over a local API — node-scoped, and gone the moment that agent restarts. hubble-relay is the piece that turns "one diary per node" into "one diary for the cluster": a separate Deployment that opens a connection to every node's local Hubble server and fans them into a single aggregated gRPC API, which is what the hubble CLI and hubble-ui actually query for a cluster-wide view. Metrics take a third, entirely separate path — each agent exposes its own Prometheus-scrapeable HTTP endpoint, independent of relay, which matters more than it sounds like the first time relay is unhealthy and the dashboards keep updating anyway.
| Component | Shape | What it does |
|---|---|---|
| Hubble (embedded) | Inside each cilium-agent | Reads flow events out of the eBPF datapath the agent already populates; keeps a local, in-memory ring buffer scoped to that one node. |
hubble-relay | Deployment | Connects to every node's local Hubble server and fans them into one aggregated gRPC API — the thing the hubble CLI and hubble-ui talk to for a cluster-wide view. |
hubble-ui | Deployment (frontend + backend) | A live service map — namespaces and the connections between them, drawn from hubble-relay's stream, with allowed and dropped verdicts drawn in. |
hubble CLI | Client binary, ships inside the agent image and as a standalone download | Talks to hubble-relay for the cluster-wide view, or directly to one agent's local socket for a single node's view — the distinction that trips people up most, covered below. |
| Hubble metrics exporter | HTTP endpoint on each agent, plus a separate one on relay | Turns the same flow stream into Prometheus-scrapeable counters and histograms — an independent path out of the same data, not a re-query of relay. |
Hubble is cheap for exactly one reason: Cilium's eBPF programs already have to inspect every packet's identity to decide whether it's allowed, so exporting what happened and why alongside that decision costs almost nothing extra. That's also why Hubble can never see more than Cilium's datapath sees — it has no independent view of the network, only a running commentary on the packets Cilium was already handling.
Turning on L7 visibility: the CiliumNetworkPolicy you actually write
☺ Like you're 10: The operator only reads the note inside an envelope if a rule tells her to open it. Otherwise she just logs "envelope passed" and moves on.
By default, Hubble reports flows at L3/L4 only — source, destination, port, protocol, verdict. A flow only gets parsed above that, into an HTTP method and path, a DNS query, or a Kafka topic, when a CiliumNetworkPolicy attaches an L7 rules block to matching traffic — which is exactly what routes it through the per-node Envoy proxy for parsing. No L7 policy touching a flow means no L7 detail in hubble observe, full stop. Older Cilium releases had a pod annotation (io.cilium.proxy-visibility) that turned on parsing without writing a policy at all; it was removed in Cilium 1.15, so a policy is now the only way in.
The trick for visibility without narrowing what's allowed is to write rules that match everything rather than something specific — an empty http: [{}] block matches every method and path, and dns: [{ matchPattern: "*" }] matches every lookup:
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: cadet-dashboard-l7-visibility
namespace: prod
spec:
endpointSelector:
matchLabels:
app: cadet-dashboard
egress:
- toEndpoints: # DNS first, or nothing downstream ever resolves
- matchLabels:
io.kubernetes.pod.namespace: kube-system
k8s-app: kube-dns
toPorts:
- ports: [{ port: "53", protocol: ANY }]
rules:
dns: [{ matchPattern: "*" }] # every lookup, now visible in Hubble
- toEndpoints:
- matchLabels:
app: mission-control-api
io.kubernetes.pod.namespace: prod
toPorts:
- ports: [{ port: "8080", protocol: TCP }]
rules:
http: [{}] # match EVERY method/path — parse, don't narrowThis is the single most common way a well-meaning observability change turns into an incident. The moment this manifest's endpointSelector matches cadet-dashboard, that pod's egress direction becomes default-deny — same as any CiliumNetworkPolicy, no exception for policies whose only intent was to add visibility. If cadet-dashboard calls anything beyond DNS and mission-control-api, this policy silently blocks it, and the person who "just wanted to see the HTTP paths" has instead cut off half the pod's traffic. Before shipping any visibility policy, list every real egress destination the pod has — hubble observe --pod prod/cadet-dashboard --last 200 against the L3/L4-only view is the fast way to build that list before you narrow anything.
hubble observe — the filter language
☺ Like you're 10: One command, and a pile of ways to say exactly which page of the logbook you want — this pod, that verdict, this last five minutes only.
hubble observe is the one command worth knowing cold — it's the fast path from "something's wrong" to "here's the exact rule that did it." Every flow carries a verdict:
| Verdict | Means |
|---|---|
FORWARDED | Allowed and delivered — the boring, default case, and most of what you'll see. |
DROPPED | Denied — by a network policy, an unready backend, or the datapath itself — and printed with a drop reason. |
ERROR | The datapath tried and failed for a reason that isn't a policy denial — an L7 parser error is the common case. |
AUDIT | Would have been dropped, if the matching policy were enforcing rather than in audit mode — a genuinely useful staged-rollout signal, covered below. |
# Cluster-wide (via hubble-relay) — the default once the CLI is pointed at relay hubble observe --verdict DROPPED -f # every drop, live hubble observe --namespace prod --pod cadet-dashboard --last 100 hubble observe --from-pod prod/cadet-dashboard --to-pod prod/mission-control-api -f hubble observe --from-namespace prod --to-fqdn "api.spacex-telemetry.example" hubble observe --protocol http --http-status 403 # only failed L7 calls hubble observe --type l7 --namespace prod # only L7-parsed events hubble observe --type policy-verdict -f # policy decisions specifically hubble observe --type drop --output json | jq . # full record, drop reason included hubble observe --since 10m --until 2m # a fixed historical window hubble observe --node-name ip-10-0-4-201 -f # one node's slice, still a cluster view hubble observe -o compact -f # one line per flow — good on a narrow terminal # Node-scoped (no relay involved at all) — exactly ONE node's local ring buffer kubectl exec -n kube-system <cilium-agent-pod> -- hubble observe --last 20
Before flipping a new policy straight to enforcing in production, audit mode is worth the extra minute: put the relevant endpoint into policy audit mode (the exact mechanism — an agent-wide flag or a per-endpoint cilium-dbg call — varies by version, so check the release you're running) and the flows that policy would have dropped show up as AUDIT instead of actually being dropped. Watch hubble observe --verdict AUDIT -f for a realistic window, confirm nothing you didn't expect shows up, and only then flip enforcement on — a dry run against real traffic instead of a guess against a diagram.
hubble-relay and hubble-ui in practice
☺ Like you're 10: Turning the recorder, the fan-in, and the dashboard on is a few lines in one file — the same file that installs Cilium itself.
All three pieces are Helm values on the same chart that installs Cilium — there's no separate install step, which keeps the whole stack under the same GitOps discipline as everything else in this ladder.
# values.yaml — turning on agent-side Hubble, relay, and the UI together
hubble:
enabled: true
relay:
enabled: true
ui:
enabled: true
tls:
auto:
method: helm # or "cronJob" (a rotating CronJob) or "certmanager"
metrics:
enabled: [dns, drop, tcp, flow, "httpV2:exemplars=true"]Relay-to-agent traffic is mTLS by default on current releases — hubble.tls.auto.method just picks how the certificates get minted and rotated, from a one-shot Helm-generated pair through to a full cert-manager-issued chain. On a cluster you don't fully trust yet, that mTLS boundary is what stops an arbitrary pod from impersonating relay and reading every flow in the cluster.
cilium hubble enable --ui # a Helm upgrade under the hood — prefer the values file above on GitOps-managed clusters cilium hubble port-forward & # forwards relay's aggregated API to a local port hubble status # is relay actually seeing every node? cilium hubble ui # opens a browser straight to the service map kubectl port-forward -n kube-system svc/hubble-ui 12000:80 # the manual equivalent, scriptable in CI
Anyone who can reach the hubble-ui Service sees the entire cluster's live topology and flow history — every namespace, every identity, every drop. That's an enormous amount of internal detail to hand out for free. Reach it through kubectl port-forward or a Gateway/Ingress sitting behind your own authenticating proxy — never publish it directly, the same discipline any other cluster-internal dashboard needs and rarely gets by default.
Metrics: what Prometheus and Grafana actually get
☺ Like you're 10: The logbook also keeps running tallies — how many calls dropped this hour, by which rule — and those tallies live on even after the detailed pages get erased.
Raw flows are for one incident, right now. Metrics are for the trend line over the last month — and they're a genuinely separate export, not a Prometheus scrape of relay. Each agent exposes its own counters and histograms on an HTTP endpoint (metrics default to port 9965 on the agent, a separate port on relay — treat exact ports as version-checkable rather than memorized, and confirm with kubectl get svc -n kube-system -l k8s-app=cilium -o yaml before writing a ServiceMonitor), so a metrics scrape keeps working even during a window where relay itself is unhealthy.
hubble:
metrics:
enabled:
- dns
- drop
- tcp
- flow
- icmp
- port-distribution
- "httpV2:exemplars=true;labelsContext=source_namespace,destination_namespace"# Drop rate by reason and source namespace, last 5 minutes
sum(rate(hubble_drop_total[5m])) by (reason, source_namespace)
# 5xx rate for one service, once httpV2 metrics are enabled
sum(rate(hubble_http_requests_total{status_code=~"5..", destination_app="mission-control-api"}[5m]))Grafana dashboards built on top of these are the right home for "is drop rate climbing" alerting — PCA covers the query side in depth, and Prometheus and Grafana on this same Tools list cover the collection and dashboard side. Use flows for "what exactly happened to this one request," and metrics for "should anyone be paged" — they answer different questions, and Hubble is genuinely the only place both come from the same source.
Day-to-day commands
☺ Like you're 10: Is the recorder healthy, what got denied just now, and — if support needs it — bundle up the whole logbook.
cilium status --wait | grep -i hubble # is relay healthy, alongside the agents and operator? hubble status # per-node connectivity into relay — X/Y nodes seeing? hubble observe --verdict DROPPED --last 20 # the fastest "what just got denied" check hubble observe --type policy-verdict --last 20 -o json | jq '.flow.policy_match_type' cilium sysdump # one archive: agent, relay, and Hubble state, for support
On a throwaway cluster with Cilium and Hubble already enabled: apply the cadet-dashboard-l7-visibility policy above exactly as written, then curl mission-control-api from cadet-dashboard and watch the request's method and path appear in hubble observe --type l7 -f. Now curl a third destination the pod has never called before — a public URL, another Service, anything not in the policy — and watch it get silently denied in hubble observe --verdict DROPPED -f, exactly the gotcha above happening live. Add a matching rule, confirm it clears, then delete the whole policy and confirm L7 detail disappears from new flows while L3/L4 visibility keeps working regardless. The mesh-namespace drill on this course is a guided version of exactly this exercise, with Timmy walking the checklist.
"I filed a ticket saying 'the platform is broken, my service can't reach mission control.' The platform engineer ran one command — hubble observe --verdict DROPPED — and showed me my own team's policy denying it, with the rule name right there. Forty seconds. I stopped guessing about the network that day."
Gotchas and failure modes
☺ Like you're 10: Most surprises come from forgetting which of the two paths — relay's queries, or the agent's metrics — you're actually looking at, or how long the record you're reading is going to last.
The CLI without relay only sees one node
Run hubble observe straight after kubectl exec-ing into one cilium-agent pod and it works — but it's reading that pod's own local socket, which only ever held that one node's flows. It's an easy trap to fall into mid-incident: the output looks exactly like a normal hubble observe session, so nothing about the command itself signals "you are only seeing a fraction of the cluster." Point the CLI at hubble-relay — via cilium hubble port-forward or an explicit --server — before trusting an absence of a flow as evidence it didn't happen anywhere.
A "visibility-only" policy is still an enforcing policy
Covered in full above, but it belongs on this list too: any CiliumNetworkPolicy that selects a pod flips that direction to default-deny, with zero exception for policies whose stated purpose was only to add L7 parsing. Audit every real destination before narrowing anything, and prefer the empty-block match-everything pattern (http: [{}]) over a policy that happens to also be narrow.
The ring buffer is memory, not a database
Each node's local flow history is a fixed-size, in-memory ring buffer. Under sustained high flow volume it rolls off older entries; on an agent restart it's gone entirely. Neither means packets were actually dropped — only that Hubble's own record of them is gone. For genuine retention, Hubble's flow export writes flow logs out to a file or stdout for your own log pipeline to pick up (a static or dynamic export config, not the default), and metrics — being aggregate counters rather than individual flows — persist in Prometheus long after the matching raw flows have rolled off the buffer.
L7 parsing costs a hop, and Hubble only sees what Cilium handles
Every L7-parsed flow is routed through the per-node Envoy proxy — still sidecar-free, but a real hop and real CPU, not free the way L3/L4 visibility is. Reserve broad L7 visibility policies for boundaries you're actively debugging rather than leaving them permanently attached to every internal call. And Hubble has no view at all of traffic Cilium's datapath never handled — host-network pods that bypass the normal path in some configurations, a different CNI entirely, or a peer cluster in a Cluster Mesh that hasn't deployed Hubble of its own.
Hubble vs the alternatives
☺ Like you're 10: Other ways to watch the network exist — they just trade identity and policy context for something else, like raw bytes or compliance-grade logs.
| Option | Model | Best when | Costs you |
|---|---|---|---|
| Hubble | Reads flow events straight out of Cilium's own eBPF inspection; per-node, fanned in by relay | You already run Cilium and want identity-aware, near-free flow visibility with the policy verdict attached | Only ever sees what Cilium's datapath handles; raw flow history is ephemeral unless export is wired up |
| tcpdump / Wireshark | Raw packet capture at an interface, no identity or policy context | A genuinely low-level protocol mystery, or a non-Cilium environment | No labels, no verdicts, no policy correlation — you're reading bytes and rebuilding context by hand |
| Istio/Envoy access logs & telemetry | Per-sidecar proxy logs and metrics at L7, mesh-wide via the control plane | You're already running a sidecar mesh and want request-level tracing, not just a flow verdict | A proxy per pod, not per node — real memory and CPU tax, and nothing at all if the mesh isn't running |
| Cloud VPC flow logs | Provider-level, IP-and-port summaries at the network boundary | Coarse, compliance-grade evidence of what left the VPC | No labels, no L7, minutes of delay, and nothing for pod-to-pod traffic that never leaves one node |
Choose Hubble whenever Cilium is already the CNI — it's depth layered on inspection that's already happening, so turning it on is close to free. Reach for tcpdump when a mystery lives below the layer Hubble's model even represents, including debugging Cilium's own datapath. Where Istio or another sidecar mesh is also in the stack, its own telemetry complements Hubble rather than replacing it — Hubble is the identity-aware truth about whether a packet moved at all, the mesh's telemetry is the deeper request-level trace once it did; Service Mesh Architecture covers how the two layers actually divide the work.
Foxy: cadet-dashboard can't reach mission-control-api any more. Nothing in the deploy changed.
Pip: Give me one second. hubble observe --namespace prod --pod cadet-dashboard --verdict DROPPED --last 20… there it is. Policy denied, egress, port 8080.
Benny the Beaver: That's from the L7-visibility policy I merged this morning. It was only supposed to make Hubble show the HTTP paths.
Timmy the Turtle: It selected the pod. The moment it did, egress went default-deny — and your policy only listed DNS and one destination. Everything else this pod ever called is blocked now too.
Gizmo: Easy — delete the policy, ship it, nobody double-checks a rollback. 🤑
Timmy: Or list the actual destinations properly and keep the visibility. Delete it and we're blind again the next time this happens — same mistake, just invisible.
1. What does Hubble actually read its flow data from, and why does that make it nearly free to run? 2. Why might hubble observe, run right after kubectl exec-ing into one cilium-agent pod, show far less than the whole cluster — with no obvious sign that it's doing so? 3. You write a CiliumNetworkPolicy purely to turn on L7 visibility for a pod's egress. What's the classic mistake, and what happens to that pod's other traffic? 4. Name Hubble's four flow verdicts, and what AUDIT specifically means. 5. What happens to Hubble's flow history when an agent restarts, and what's the one thing that survives it? 6. Does Prometheus scrape hubble-relay, or each agent directly — and why does that distinction matter operationally?
Check your answers
- Straight out of the eBPF maps
cilium-agentalready populates to enforce policy on every packet. Because that inspection is happening regardless, exporting what it saw costs almost nothing extra — Hubble adds a record, not a new inspection point. - Because that command is talking to the local agent's own socket, which only ever held that one node's flows — and the output looks identical to a normal cluster-wide session, so nothing about it signals the narrower scope. Point the CLI at
hubble-relay(via port-forward or--server) for a genuine cluster-wide view. - The moment the policy's
endpointSelectormatches the pod, that direction becomes default-deny — with no exception for a policy whose intent was only visibility. Any real destination the policy didn't list, DNS included if forgotten, gets silently denied alongside the ones it did list. FORWARDED(allowed and delivered),DROPPED(denied, with a reason),ERROR(the datapath failed for a non-policy reason, e.g. an L7 parse error), andAUDIT— a flow that would have been dropped had the matching policy been enforcing rather than in audit mode, a safe way to dry-run a new policy against real traffic.- The node's in-memory ring buffer of recent flows is lost — it was never persisted. Metrics (aggregate counters, exported to Prometheus on a separate path) survive, because they were already outside the agent's own memory by the time the restart happened; raw flow history only survives if flow export to an external log pipeline was explicitly configured.
- Each agent directly, on its own metrics HTTP endpoint — never through
hubble-relay. That matters because metrics keep flowing (and alerting keeps working) even during a window where relay itself is unhealthy or a node can't reach it.