Cilium

Every CNI plugin has to satisfy the same flat, no-NAT Pod network that Networking & the CNI lays out — what separates them is what they do with the CPU cycles once that baseline is met. Cilium's answer is eBPF: instead of the kernel walking a chain of iptables rules for every packet, small, verified programs run directly inside the kernel and make forwarding and policy decisions from a hash-map lookup keyed on a workload's identity rather than its ephemeral IP address. That one architectural choice is why Cilium can replace kube-proxy outright, enforce policy on HTTP methods and DNS names instead of just ports, and hand you Hubble — a live, per-flow record of exactly which rule allowed or dropped every connection in the cluster. This page stays at the level you'll actually operate day to day: what you install, what you write, what breaks at 2 a.m. Cilium's own CNCF certification and the wider Kubestronaut ladder get their full treatment elsewhere on this platform, linked out to rather than repeated here.

What Cilium replaces, and why eBPF

☺ Like you're 10: Same job as any other CNI — get every Pod an address and let it talk — just done by teaching the kernel itself the rules instead of handing it a paper checklist.

At its floor, Cilium is an ordinary CNI plugin: on ADD it allocates a Pod IP, wires up the network namespace, and programs routes, exactly the contract Networking & the CNI walks through for any plugin. What makes it worth a page of its own is everything built on top of that floor. A conventional datapath — Flannel's overlay, or Calico in its default mode — still leans on iptables or IPVS to turn a Service into forwarding rules, evaluated by walking a list or hashing on IP address for every packet. Cilium instead attaches eBPF programs at hook points in the kernel's own packet path — the network interface (XDP) and the traffic-control layer (tc) — that make the same decisions via a kernel hash-map lookup, keyed on a security identity Cilium derives from a Pod's labels rather than its IP.

Because Cilium's datapath already knows what a packet is and where it's going, adding "and log the verdict" or "and check the HTTP path" is cheap — which is exactly where Hubble and Layer-7 policy come from later on this page. Cilium is a CNCF-graduated project (the first CNI to reach that maturity level), and it's increasingly the datapath underneath managed offerings you may already be running without knowing it — GKE's Dataplane V2 is Cilium under a different name.

Architecture: the agent, the operator, and Hubble

☺ Like you're 10: One helper rides along on every machine teaching its kernel the rules; one boss in the cluster hands out ID badges; one camera crew watches it all happen.

Three pieces do almost all the work. cilium-agent runs as a DaemonSet, one Pod per node — it watches kube-apiserver for Services, Pods, and CiliumNetworkPolicy objects, compiles what it sees into eBPF programs and maps, and loads them into that node's kernel; it's also the process that installs the CNI plugin binary and drives that node's Envoy proxy for anything a policy asks to inspect at Layer 7. cilium-operator runs as a small Deployment handling cluster-wide bookkeeping that doesn't belong on every node — IPAM block allocation and garbage-collecting stale identity objects chief among them; it sits off the datapath entirely, so if it's briefly unavailable, traffic already flowing keeps flowing, though new Pods may stall waiting on an IP. Hubble is embedded in every agent, reading flow events straight out of the eBPF datapath; hubble-relay fans those per-node streams out into one cluster-wide API, and hubble-ui turns it into a live service map.

The Helm values and CiliumNetworkPolicy you actually write

☺ Like you're 10: One file to switch the smart scanners on, and a couple of small rule files saying who's allowed through which door, and about what.

Cilium ships as a Helm chart, so the entire datapath configuration is one reviewable, GitOps-managed values file rather than a pile of one-off kubectl flags — install with helm install cilium cilium/cilium -f values.yaml, or with the cilium CLI, which is a thin wrapper around the same chart.

# values.yaml — a reasonable production-shaped starting point
kubeProxyReplacement: true          # eBPF handles ClusterIP/NodePort/LoadBalancer — delete kube-proxy
k8sServiceHost: api.cluster.internal   # REQUIRED once kube-proxy is gone: how the agent reaches the API
k8sServicePort: 6443

routingMode: tunnel                 # "tunnel" (portable default) or "native" (faster, needs a cooperating fabric
tunnelProtocol: vxlan                #   or BGP) — see the routing-strategy callout in Networking & the CNI

ipam:
  mode: cluster-pool                 # kubernetes | cluster-pool | eni | azure
  operator:
    clusterPoolIPv4PodCIDRList: ["10.244.0.0/16"]

hubble:
  enabled: true
  relay: { enabled: true }
  ui:    { enabled: true }
  metrics:
    enabled: [dns, drop, tcp, flow]

The object you'll actually hand-write most often is CiliumNetworkPolicy — the same grammar as a standard NetworkPolicy, extended with the parts identity makes possible. The rule below allows cart-api to call exactly one HTTP method and path on inventory-api, and nothing else — a plain NetworkPolicy can only say "TCP port 8080, yes or no," never "which method, which path."

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: inventory-api-ingress
  namespace: shop
spec:
  endpointSelector:                  # which Pods this policy protects
    matchLabels:
      app: inventory-api
  ingress:
    - fromEndpoints:                 # who may call — by LABELS, never by IP
        - matchLabels:
            app: cart-api
            io.kubernetes.pod.namespace: shop
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:                    # L7: parsed by the per-node Envoy proxy the agent drives
              - method: "GET"
                path: "/v1/stock/[0-9]+"

Kube-proxy replacement: one eBPF program instead of two systems

☺ Like you're 10: Instead of one worker wiring the hallway and a second, separate worker manually forwarding every phone call down it, one system does both jobs at once and never has to check with anyone else.

Networking & the CNI already covers kube-proxy's iptables and IPVS dataplanes as two independent watchers of the same Service and EndpointSlice objects, each reprogramming a node's rules on its own schedule. Cilium's eBPF datapath can absorb that job entirely: set kubeProxyReplacement: true and ClusterIP, NodePort, LoadBalancer, and session affinity are all implemented as eBPF programs, and kube-proxy is deleted from the cluster outright. Because ClusterIP translation happens at the socket layer — a Pod connecting to a Service gets rewritten to a real backend address before a packet is ever built — there's no per-packet DNAT cost the way there is with iptables, and no separate ruleset that can drift out of sync with what the CNI already knows about the Pod network.

$ kubectl -n kube-system exec ds/cilium -- cilium status | grep -i KubeProxyReplacement
KubeProxyReplacement:   True

$ kubectl -n kube-system get daemonset kube-proxy
Error from server (NotFound): daemonsets.apps "kube-proxy" not found   # expected — it's gone

Hubble and day-to-day commands

☺ Like you're 10: One command asks "is everything healthy?", one runs a full obstacle course to prove it, and one shows you every hallway trip and whether it made it through.

This is the capability that changes how an incident actually feels. Instead of arguing about whether a policy is blocking traffic, you watch the verdict.

# Install / upgrade — pin a version explicitly and read that release's notes first
$ cilium install --version 1.16.4
$ cilium status --wait                         # agents, operator, Hubble all healthy?
$ cilium connectivity test                     # deploys a test namespace, runs the full check suite

# The single most useful command in this entire page
$ hubble observe --verdict DROPPED -f
shop/cart-api-7d9f:52344 (ID:4102) -> shop/inventory-api-6b4c:8080 (ID:5590) Policy denied DROPPED (TCP Flags: SYN)

# Narrow it down
$ hubble observe --namespace shop --pod cart-api --verdict DROPPED --last 50
$ hubble observe --protocol http --http-status 403     # L7 verdicts, if an http rule applies
$ hubble observe --to-fqdn "api.stripe.com"             # did an FQDN egress rule resolve it?

# Ground truth straight from one node's agent
$ CILIUM_POD=$(kubectl -n kube-system get pods -l k8s-app=cilium -o name | head -1)
$ kubectl -n kube-system exec -it $CILIUM_POD -- cilium-dbg endpoint list   # every Pod: identity + policy state
$ kubectl -n kube-system exec -it $CILIUM_POD -- cilium-dbg identity list  # numeric identity ↔ label-set table
$ kubectl -n kube-system exec -it $CILIUM_POD -- cilium-dbg bpf lb list    # the Service load-balancing map itself

Gotchas

☺ Like you're 10: Four things bite almost everyone once: forgetting DNS, an old kernel, expecting the fancy rules to be free, and trying to swap CNIs while the plane's still in the air.

• The DNS trap, restated. Covered above and worth restating because it causes more outages than everything else on this list combined — an egress policy with no DNS rule looks exactly like "the database is down," not "I broke name resolution."
• Kernel version is a real constraint. Cilium's eBPF datapath needs a reasonably modern kernel, and the floor climbs with every release — check the current requirement against whatever your node images actually ship, especially on a long-lived enterprise distro or an older managed node pool. cilium status --verbose reports which features the running kernel actually supports.
• L7 rules aren't free. Plain L3/L4 policy is pure eBPF, no extra hop. The moment a rule includes an http: or dns: block, matching traffic gets redirected through a userspace Envoy proxy on that node — still fast, still sidecar-free, but a real hop with real CPU cost. Reach for L7 rules at genuine trust boundaries, not on every internal call.
• Swapping a CNI is not a rolling upgrade. Replacing an already-running CNI with Cilium on a live cluster is genuinely dangerous — the two plugins disagree about routes while both are installed, and every Pod needs a new interface from the new one. The safe path is a fresh cluster with Cilium from day zero and a workload migration; if you must migrate in place, follow Cilium's own documented per-node procedure and rehearse it on a disposable cluster first.
• Tunnel mode still costs MTU. Left at its default, Cilium's routingMode: tunnel encapsulates every packet in VXLAN or Geneve — the exact MTU tax Networking & the CNI already warns about for any overlay. Switching to routingMode: native removes it, but only where the underlay will actually route your Pod CIDRs.

Choosing Cilium vs. Calico vs. Flannel

☺ Like you're 10: Three road crews with three different budgets — a bare road, a road crew that checks IDs, and a road crew that wired cameras into every corner.

Flannel isn't in that table because it isn't really competing on the same axis — no policy engine at all, chosen purely for "make Pods reach each other" simplicity on a lab cluster. Networking & the CNI covers the full three-way comparison and the overlay-vs-native-routing tradeoff underneath all three plugins; this page assumes you've already decided Cilium is the answer and need to actually run it. Cilium's Cluster Mesh feature is also the concrete implementation behind this course's own Multi-Cluster & Fleet Management, and its optional sidecar-free mesh mode overlaps with Service Mesh Fundamentals' territory — for the platform-engineering depth on installation modes, Cluster Mesh prerequisites, and where Cilium sits relative to Istio and Linkerd, Platform Engineering's own Cilium tool page goes considerably further than this page needs to.

Cilium isn't a named domain on the CKA blueprint, but the concepts it embodies — CNI, default-deny NetworkPolicy, kube-proxy's job, identity-based vs. IP-based thinking — sit squarely inside it. Cilium has its own dedicated CNCF credential, the Cilium Certified Associate (CCA), covered in full on Platform Engineering's CCA page — domains, weights, and a study plan — and the wider Golden Kubestronaut ladder it belongs to is mapped on the sibling Golden Astronaut course.