eBPF & the Cilium Datapath

The CCA blueprint gives eBPF one domain worth ten percent, and tells you just enough to say a verifier proves programs safe, a JIT compiles them to native code, and maps hold shared state. That's the right amount of depth for an associate exam — and the wrong amount for actually reasoning about a datapath you're on call for. This page goes underneath the exam: what a kernel hook actually is and which ones Cilium attaches to, what the verifier can and can't prove and why it rejects the programs it rejects, what a JIT buys you that an interpreter doesn't, what BTF and CO-RE solve, and the mechanical reason a hash-map lookup keeps a flat cost curve where iptables' rule chain does not. None of this is asked on the CCA. All of it is what separates knowing the vocabulary from being able to read a bpftool prog show dump during an incident and know what you're looking at.

eBPF is four things, not one

☺ Like you're 10: "eBPF" sounds like one gadget. It's actually four separate pieces working together — a program, a bouncer, a translator, and a filing cabinet.

eBPF (extended Berkeley Packet Filter — the name is a historical fossil; it now runs far past packet filtering) lets you load small programs into the Linux kernel at runtime, with no kernel module to compile and no reboot. What makes that safe enough to do in production, on a shared multi-tenant kernel, is that it's never really "one thing" — it's a pipeline of four distinct stages, and each stage exists specifically to make the next one trustworthy:

• Bytecode — your program, written in a restricted subset of C (or Rust, via aya), compiled by Clang/LLVM down to eBPF's own instruction set: a small, RISC-like set of ~110 opcodes operating on eleven 64-bit registers.
• The verifier — a static analyzer built into the kernel that walks every possible execution path of that bytecode before it's allowed to run, and rejects anything it can't prove terminates and stays memory-safe.
• The JIT compiler — once verified, translates the bytecode into native machine code for the host's actual CPU (x86-64, arm64, …), so the loaded program runs at compiled speed, not interpreted speed.
• Maps — key/value data structures living in kernel memory that a program reads and writes as it runs, and that userspace can also read and write via the bpf() syscall — the one sanctioned channel between kernel-resident state and the control plane that manages it.

Cilium's cilium-agent is that control plane. It doesn't move packets itself — it compiles eBPF programs from the cluster's current policy and identity state, loads them onto the node's kernel hooks, and keeps their maps populated as pods, Services, and CiliumNetworkPolicy objects change. The actual packet-moving work happens entirely in kernel space, at line rate, without ever crossing back into userspace for a decision.

Kernel hooks: where in the stack a program actually runs

☺ Like you're 10: A packet doesn't just teleport into a pod — it passes several checkpoints on the way in, and Cilium can bolt a program onto more than one of them, choosing the earliest checkpoint that already has enough information to decide.

"eBPF" names the mechanism; a program type names which kernel hook a given program is allowed to attach to, and what context (which struct, which fields) it's handed when it runs. Cilium uses several, and picking the earliest usable one is the whole performance story:

• XDP (eXpress Data Path) — runs in the network driver, before the kernel has even allocated an sk_buff for the packet. The earliest possible hook, and the cheapest: Cilium uses it at the node's ingress for its fastest DDoS-drop and load-balancing paths, when the NIC driver supports native XDP.
• tc (traffic control) ingress/egress — attaches to the classifier/action layer already used for QoS. This is Cilium's main workhorse: bpf_lxc runs on a pod's veth pair, bpf_host on the physical NIC, and bpf_overlay on the VXLAN/Geneve interface when running in tunnel mode. It has a full sk_buff to work with — later than XDP, but with more of the packet's context already built.
• Socket-level hooks — cgroup/connect4, cgroup/connect6, and sockops attach at the socket layer itself, before a packet is even constructed. This is how kube-proxy replacement does its earliest work: when a pod calls connect() to a Service's ClusterIP, a socket-level program can rewrite the destination to a live backend's real pod IP right there, so the kernel never builds a packet addressed to a virtual IP that has to be NAT'd back out later.
• Tracing hooks — kprobes, tracepoints, and perf_events attach to arbitrary kernel functions or static tracepoints. Cilium doesn't move packets here, but Hubble's deeper visibility and tools like bpftrace lean on exactly this category to observe kernel behavior with no source changes.

The pattern across all of them: eBPF doesn't replace the network stack with something else — it lets you observe and redirect at whichever existing layer already has the cheapest, earliest answer to the question you're asking, and skip every layer downstream of that answer entirely.

The verifier: proof by inspection, not by running it

☺ Like you're 10: Before your program ever runs for real, a very thorough reader goes through it line by line and refuses to let it start unless it can already prove — just from reading — that it will finish and won't reach outside its own sandbox.

Loading arbitrary code into the kernel is normally how you get a kernel module, and a kernel module can crash the whole machine. The verifier is what makes eBPF a different, much safer category: it statically proves a set of safety properties about a program before it's allowed to run at all, and it proves them without executing a single instruction. Concretely, it walks every reachable path through the bytecode's control-flow graph and checks:

• Bounded execution — historically, no unbounded loops at all (every backward jump had to prove termination by other means); recent kernels (5.3+) allow bounded loops via the bpf_loop() helper, where the iteration count itself is proven finite. Either way, "does this halt" has to be answerable before load time, not discovered at runtime.
• Memory safety — every pointer dereference is checked against what the verifier can prove about that pointer's type and valid range at that exact point in the program. Reading past the end of a packet buffer, or reading uninitialized stack memory, is a load-time rejection, not a runtime fault.
• Register and stack discipline — a fixed 512-byte stack, a bounded number of tracked register states, and no raw pointer arithmetic that the verifier can't follow.
• A restricted helper allowlist — a program can't call arbitrary kernel functions; it can only call from a small, explicitly exposed set of BPF helper functions (map lookups, checksum updates, redirect calls), each type-checked against the calling program type.
• A total instruction budget — historically capped at 4096 instructions per program (much higher today, on the order of a million on modern kernels, since the verifier's own complexity budget became the real constraint rather than a flat instruction count).

The practical consequence, and the one thing worth internalizing past what the CCA asks: a verifier rejection is not a bug in your program in the normal sense — it's the verifier failing to construct a proof, which is a narrower thing than the program actually being unsafe. A loop you know terminates can still be rejected if you wrote it in a shape the verifier's path-exploration can't follow; the fix is usually restructuring the code so the safety argument becomes visible to the analysis, not the underlying logic being wrong. This is the single most common source of "why won't my eBPF program load" confusion once you're past introductory material — and exactly why bpftool prog load or a Cilium agent log showing a verifier rejection is worth reading closely rather than treating as an opaque failure.

JIT compilation: from a portable instruction set to native speed

☺ Like you're 10: Once the inspector clears your program, a translator rewrites it from a universal instruction booklet into the exact machine language the computer's processor actually speaks — so it runs at full speed instead of being re-read one line at a time, forever.

Verified bytecode is still just bytecode — an interpreter could run it directly, decoding and executing one instruction at a time, but that cost is paid on every single packet, forever. The JIT (just-in-time) compiler instead translates the whole verified program, once, into native machine code for the host's actual architecture, and that compiled version is what actually runs on every subsequent packet. On Linux this has been the default since kernel 4.x for the major architectures; the bytecode interpreter still exists mainly as a fallback and a reference implementation for architectures without a mature eBPF JIT backend.

The instruction set eBPF bytecode targets was deliberately designed to be a close cousin of real CPU instruction sets — eleven general-purpose 64-bit registers, a calling convention, comparison and arithmetic ops that map almost one-to-one onto x86-64 or arm64 equivalents. That closeness is what makes the JIT simple and fast to run (compilation happens at load time, not per-packet) and what makes the compiled output competitive with hand-written C — there isn't a large abstraction gap between the portable bytecode and the native code it becomes. This is the concrete mechanism behind a line the CCA blueprint states as a fact without deriving it: eBPF programs run "at line rate." They run at line rate because by the time a packet crosses the hook, there's no interpretation, no bytecode dispatch loop, and no userspace round-trip in the hot path — just compiled native instructions and a map lookup.

BTF and CO-RE: portable programs without recompiling per kernel

☺ Like you're 10: Kernel internals shift slightly from version to version, like furniture getting rearranged between houses. Instead of rebuilding your program separately for every single house, BTF is a floor plan the kernel hands over, and CO-RE lets your program read that floor plan and adjust itself on the spot.

An eBPF program that reads kernel data structures directly — inspecting a struct sk_buff or a task_struct field, the way Cilium's deeper tracing programs and Hubble's socket-level visibility do — has a portability problem: the exact byte offset of a given field inside those structs can shift between kernel versions and distro configs. The traditional fix was compiling a separate binary per target kernel, using that kernel's own headers. BTF (BPF Type Format) is a compact, embedded description of every kernel type's actual layout on the running kernel, exposed at /sys/kernel/btf/vmlinux. CO-RE (Compile Once – Run Everywhere) is the technique built on top of it: the compiler emits special relocation records for every field access instead of a fixed offset, and at load time — not compile time — the loader reads the running kernel's own BTF and patches those offsets to match, before the program is ever handed to the verifier.

The result is one compiled artifact that loads correctly on any BTF-enabled kernel, without recompiling against that kernel's headers or shipping a kernel-specific build per node. This is exactly what lets a single Cilium container image work across a fleet of nodes running different kernel patch levels — which is the practical reason CO-RE is the default assumption in Cilium's own build going back several major releases, quietly replacing what used to require an in-cluster compiler sidecar (Cilium's older "eBPF probe compile at runtime" model) with a load-time relocation step.

Why a hash-map datapath outscales a rule chain

☺ Like you're 10: iptables checks a growing list of rules one at a time for every single packet, like reading a longer and longer instruction sheet from the top every time. eBPF instead reaches straight into a filing cabinet and pulls the answer out in one motion, no matter how big the cabinet gets.

The CCA blueprint states this contrast as a fact; here's the mechanism underneath it. iptables (and its Kubernetes usage inside kube-proxy's iptables mode) represents rules as an ordered, linear chain that the kernel walks sequentially for every packet, evaluating each rule's match conditions until one hits. Every Service, every NetworkPolicy-equivalent rule, every additional endpoint adds more entries to that chain — so the cost of evaluating a single packet grows with the total size of the ruleset, which itself grows with cluster size. That's algorithmically O(n) in the number of rules, walked fresh on every packet, and it's the specific, mechanical reason large iptables rule sets became a documented scaling ceiling for kube-proxy well before Cilium existed.

A BPF map — specifically the hash-table map type Cilium uses for policy verdicts, identities, and connection tracking — replaces sequential matching with a hash lookup: compute a key (say, a source/destination identity pair, or a 5-tuple), hash it, and read the bucket directly. That's O(1) in the expected case, regardless of how many total entries the map holds. Doubling the number of Services or policies doesn't make an existing packet's lookup any slower; it just makes the map itself larger. Combine that with the socket-level early redirection covered above — a pod-to-Service connection gets its destination rewritten to a real backend IP at connect() time, so the packet that eventually gets constructed is already addressed correctly and never needs a NAT/conntrack table lookup to translate a virtual IP at all — and you get a datapath whose per-packet cost stays essentially flat as the cluster grows, instead of climbing with it.

Reading it live: bpftool, cilium monitor, and what a verifier failure looks like

☺ Like you're 10: All of this stops being theory the moment you actually look — the tools that show you a real, loaded program running on a real node.

bpftool, maintained in the kernel source tree, is the general-purpose inspection tool for anything loaded on a box, Cilium or otherwise:

# List every loaded eBPF program on this node, by type and attach point
sudo bpftool prog show

# Dump one program's JIT-compiled instructions
sudo bpftool prog dump jited id <PROG_ID>

# List every loaded map, and peek at one's live contents
sudo bpftool map show
sudo bpftool map dump id <MAP_ID>

# Cilium's own agent CLI wraps the same underlying state
# in domain-specific terms — endpoints, identities, policy
cilium-dbg endpoint list
cilium-dbg bpf policy get <ENDPOINT_ID>
cilium-dbg monitor --type drop     # live-stream every packet an eBPF program drops, and why

A verifier rejection surfaces in the kernel log and in whichever tool attempted the load — for Cilium, that generally means an error in the cilium-agent logs at program-load time, naming the specific instruction and the safety property it couldn't prove. It reads unfamiliar the first time and is genuinely useful the second: it is telling you exactly which pointer, at exactly which instruction offset, it couldn't bound — not "your program is bad" in the abstract.