Cilium

Cilium is the CNCF-graduated CNI that replaces the Linux kernel’s old iptables plumbing with eBPF programs loaded directly into the kernel — and in doing so turns the pod network from a dumb pipe into a platform capability that routes, load balances, enforces security policy at L3 through L7, and explains itself. It solves the problem that on a large cluster the network is simultaneously your worst performance bottleneck, your least observable component, and your most important security boundary — and traditional CNIs give you almost no leverage on any of the three.

What Cilium is and the problem it solves

☺ Like you’re 10: The part of the cluster that carries messages between apps — but a much smarter version that also checks IDs and takes notes.

Cilium is a CNCF-graduated project (it graduated in 2023, the first CNI to do so) providing networking, observability and security for Kubernetes. At its most basic it is a CNI plugin: it gives every pod an IP, wires it into a flat network, and makes pod-to-pod traffic work. What makes it interesting is how it does that — and everything it can do once the plumbing is in place.

The problem: iptables was never designed for this

A conventional Kubernetes datapath leans on iptables. Every Service becomes a chain of DNAT rules; every NetworkPolicy becomes more rules. That model is battle-tested and it works beautifully at small scale. At large scale it develops three chronic ailments. Rule evaluation is broadly linear, so a cluster with tens of thousands of Services burns real CPU walking chains for every new connection. Updates are worse than lookups: changing the ruleset means rewriting large blocks of it, so endpoint churn during a rollout can produce seconds of control-plane lag and dropped connections. And iptables reasons entirely in IP addresses — which in Kubernetes are ephemeral, recycled, and completely meaningless as a description of who a workload is.

The answer: programs in the kernel, identities instead of IPs

Cilium attaches eBPF programs at strategic hook points in the kernel — the network interface (tc/XDP), the socket layer, the cgroup layer — where they make forwarding, load balancing and policy decisions using hash-map lookups rather than list traversal. Cost stops scaling with the number of rules. Better still, the maps are keyed on a security identity that Cilium derives from a pod’s labels, not its IP. Every pod carrying app=checkout, env=prod shares one numeric identity cluster-wide; policy is expressed and enforced between identities. Reschedule the pod, change its IP, scale it to fifty replicas — the identity is unchanged and the policy still means what you wrote.

What Cilium is not

Cilium is not a GitOps tool, a policy admission controller, or a replacement for Kyverno and OPA Gatekeeper — those govern what manifests may be admitted; Cilium governs what packets may flow. It is not a general-purpose observability stack either: Hubble sees network flows, not application traces, so it complements OpenTelemetry and Prometheus rather than replacing them. And while Cilium Service Mesh is real, it is not a drop-in swap for every Istio feature.

Where Cilium fits in a platform

☺ Like you’re 10: Right at the bottom, underneath everything. If Cilium is unwell, nothing above it works.

Which plane and which domain it serves

Cilium sits in the substrate layer of the platform architecture — below your apps, below your mesh, below your ingress. Its control plane is the cilium-operator Deployment plus each node’s agent watching the Kubernetes API and compiling desired state into BPF maps; its data plane is the eBPF bytecode in the kernel of every node. It is a classic platform-team-owned, application-team-invisible capability: developers write a Service and a policy, and never learn that XDP exists.

Its neighbours

• Underneath: the nodes and their kernels — see The Kubernetes Substrate. Cilium is one of the few platform components with a genuine kernel version dependency.
• Alongside: Prometheus and Grafana. The agent, the operator and Hubble all export metrics; Hubble’s per-flow metrics are among the most useful signals in Observability.
• Above (and sometimes instead of): Istio and Linkerd. Cilium can carry the L4 and mTLS-ish parts of a mesh without sidecars, or simply be the network those meshes ride on.
• Governed by: Security & Policy Enforcement. Network policy is the “segmentation” half of zero trust; admission policy is the other half.
• Reaching across: Multi-Cluster, via Cluster Mesh — global Services and cross-cluster failover.

CNPE domain relevance

Cilium is not on the official CNPE tool list — do not expect a question that names it. But the exam absolutely tests the concepts it embodies: CNI and the flat pod network, NetworkPolicy and default-deny, kube-proxy and Service load balancing, egress control, and multi-cluster connectivity. Cilium is the canonical modern implementation of all of them, so studying it is the fastest way to make the Networking lesson concrete. Treat this page as depth for real platform work and as a mental model for exam questions phrased generically.

How Cilium works

☺ Like you’re 10: One helper on every machine teaches the machine’s brain the rules; one boss in the cluster hands out ID numbers and IP addresses.

The components

Datapath choices: routing mode, kube-proxy replacement, IPAM

Three install-time decisions shape everything afterwards, and they are the ones a platform engineer must be able to reason about.

• Routing mode. In tunnel mode (the portable default) pod traffic is encapsulated in VXLAN or Geneve between nodes, so the underlying network only needs node-to-node IP reachability — it never has to know your pod CIDRs. In native routing mode packets are forwarded unencapsulated, which removes the per-packet overhead and the MTU tax but requires the underlay to route pod CIDRs. Cilium can arrange that itself with autoDirectNodeRoutes on a flat L2 network, or announce pod CIDRs and LoadBalancer VIPs to your physical fabric using its built-in BGP control plane.
• kube-proxy replacement. With kubeProxyReplacement: true Cilium implements ClusterIP, NodePort, LoadBalancer, ExternalIPs and session affinity entirely in eBPF and you can delete kube-proxy. ClusterIP translation happens at the socket layer, so a pod connecting to a Service is rewritten to the backend address before a packet is ever built — zero per-packet DNAT cost. Because the agent must reach the API server without kube-proxy, you set k8sServiceHost and k8sServicePort explicitly.
• IPAM mode. kubernetes uses the per-node PodCIDR that the controller-manager assigns. cluster-pool (Cilium’s default) has the operator carve per-node blocks out of a cluster-wide pool it owns. The cloud modes — eni on AWS, azure, alibabacloud — allocate real VPC addresses so pods are first-class citizens of the cloud network — powerful, and firmly bounded by your cloud’s per-node IP limits.

The CRDs it introduces

The resources you will actually write

☺ Like you’re 10: One file to switch the smart network on, and a couple of small rule files saying who may talk to whom — and about what.

Installing with an opinionated values file

Cilium is installed with Helm (or the cilium CLI, which wraps Helm), which means the whole datapath is one reviewable, GitOps-managed file — exactly the “infrastructure through the same reconciler as apps” idea from GitOps Workflows.

# values.yaml — a production-shaped Cilium install (helm install cilium cilium/cilium -f values.yaml)
kubeProxyReplacement: true       # eBPF handles ClusterIP/NodePort/LoadBalancer; delete kube-proxy
k8sServiceHost: api.acme.internal   # REQUIRED without kube-proxy: how the agent reaches the API
k8sServicePort: 6443

routingMode: native              # "tunnel" (default) or "native"
ipv4NativeRoutingCIDR: 10.244.0.0/16   # required in native mode: what NOT to masquerade
autoDirectNodeRoutes: true       # only valid when all nodes share an L2 segment
# routingMode: tunnel
# tunnelProtocol: vxlan          # or geneve — costs ~50 bytes of MTU per packet

ipam:
  mode: cluster-pool             # kubernetes | cluster-pool | eni | azure
  operator:
    clusterPoolIPv4PodCIDRList: ["10.244.0.0/16"]
    clusterPoolIPv4MaskSize: 24  # a /24 block per node → roughly 250 usable pod IPs per node

bpf:
  masquerade: true               # do SNAT in eBPF rather than iptables

hubble:
  enabled: true
  relay: { enabled: true }       # cluster-wide flow API for the CLI
  ui:    { enabled: true }       # the service map
  metrics:
    enabled: [dns, drop, tcp, flow, "httpV2:exemplars=true"]

cluster:
  name: eu-west-1-prod           # MUST be unique across a Cluster Mesh
  id: 3                          # MUST be unique, 1-255

gatewayAPI:
  enabled: true                  # needs the Gateway API CRDs installed first

Identity-based policy with L7 and DNS awareness

Here is where Cilium visibly outgrows standard NetworkPolicy. The rule below allows checkout to call exactly two HTTP endpoints on payments — a GET on a path pattern and a POST — and nothing else. A plain NetworkPolicy can only say “TCP port 8080, yes or no.”

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: payments-api
  namespace: prod
spec:
  endpointSelector:                 # WHICH pods this policy protects
    matchLabels:
      app: payments
  ingress:
    - fromEndpoints:                # WHO may call — by LABELS, never by IP
        - matchLabels:
            app: checkout
            io.kubernetes.pod.namespace: prod
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:                   # L7: parsed by the per-node Envoy proxy
              - method: "GET"
                path: "/v1/balance/[0-9]+"    # a regex, not a glob
              - method: "POST"
                path: "/v1/charge"
    - fromEntities:
        - cluster                   # allow kubelet probes and in-cluster scrapers
      toPorts:
        - ports: [{ port: "9090", protocol: TCP }]

And here is DNS-aware egress: instead of chasing a SaaS provider’s rotating IP ranges, you name the hostname. Cilium proxies the pod’s DNS request, learns the answer, and programs exactly that IP into the policy map for the record’s TTL.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: payments-egress
  namespace: prod
spec:
  endpointSelector:
    matchLabels: { app: payments }
  egress:
    # 1. DNS must be allowed FIRST, and must be visible to the DNS proxy,
    #    otherwise toFQDNs can never learn any addresses.
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports: [{ port: "53", protocol: ANY }]
          rules:
            dns:
              - matchPattern: "*"           # observe every lookup this pod makes
    # 2. Now allow egress to hostnames, not IP ranges.
    - toFQDNs:
        - matchName: "api.stripe.com"
        - matchPattern: "*.s3.eu-west-1.amazonaws.com"
      toPorts:
        - ports: [{ port: "443", protocol: TCP }]
---
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy    # the platform team's non-negotiable baseline
metadata:
  name: baseline-egress-default-deny
spec:
  # Simply SELECTING a pod is what flips egress to default-deny for it.
  # This baseline allows only DNS; every other destination has to be opened
  # by a namespace-scoped CNP like the one above.
  endpointSelector:
    matchExpressions:
      - key: io.kubernetes.pod.namespace
        operator: NotIn
        values: [kube-system]
  egress:
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports: [{ port: "53", protocol: ANY }]
          rules:
            dns:
              - matchPattern: "*"

Cilium also has explicit ingressDeny/egressDeny rules, which standard NetworkPolicy cannot express — but reach for them deliberately, because deny rules take precedence over every allow rule anywhere in the cluster. A blanket egressDeny: [{toEntities: [world]}] would not merely set a floor; it would cancel the toFQDNs allow above and every other team’s egress rule with it. Build baselines out of default-deny-by-selection, which composes, and save explicit deny for the handful of destinations nobody may ever reach.

Global Services with Cluster Mesh

Once two clusters are meshed, making a Service global is a single annotation — applied to a Service of the same name and namespace in each participating cluster, since the mesh works by merging their endpoint sets. Traffic then prefers local backends and fails over to the remote cluster only when local endpoints are gone — cross-cluster failover without a global load balancer.

apiVersion: v1
kind: Service
metadata:
  name: payments
  namespace: prod
  annotations:
    service.cilium.io/global: "true"      # merge endpoints from ALL meshed clusters
    service.cilium.io/affinity: "local"   # prefer local backends; fail over only if none are ready
spec:
  type: ClusterIP
  selector: { app: payments }
  ports:
    - port: 8080
      targetPort: 8080

Day-to-day commands

☺ Like you’re 10: One command asks “are you OK?”, one runs a full health check, and one shows you every letter and whether it was delivered.

Install, status and the connectivity test

# Install / upgrade (the CLI wraps Helm; a values file is more GitOps-friendly)
# Always pin a version explicitly and read that release's upgrade notes first.
cilium install --version <x.y.z>
cilium upgrade --version <x.y.z>

# The first question, always: are agents, operator and Hubble healthy?
cilium status --wait
# /¯¯\__/¯¯\    Cilium:             OK
# \__/¯¯\__/    Operator:           OK
#    \__/       Hubble Relay:       OK
# DaemonSet  cilium   Desired: 6, Ready: 6/6, Available: 6/6

# The end-to-end proof: deploys a test namespace and runs a large suite of checks
# (pod-to-pod, pod-to-service, DNS, egress to world, policy enforcement, node-port).
cilium connectivity test
# Flags for narrowing or relocating the run (--test, --test-namespace) and for
# concurrency differ between CLI versions — read `cilium connectivity test --help`
# on the binary you actually have before wiring this into CI.

# Turn Hubble on (this performs a Helm upgrade — on a GitOps-managed cluster
# prefer the values file above) and open the service map
cilium hubble enable --ui
cilium hubble port-forward &
cilium hubble ui

# Cluster Mesh
cilium clustermesh enable --context eu-west --service-type LoadBalancer
cilium clustermesh connect --context eu-west --destination-context us-east
cilium clustermesh status --context eu-west --wait

Hubble: proving which rule dropped the packet

This is the capability that changes how incidents feel. Instead of arguing about whether a policy is blocking traffic, you watch the verdict.

# Every dropped flow in the cluster, live — the single most useful Cilium command
hubble observe --verdict DROPPED -f

# ...and the drop reason is printed for you, e.g.:
# prod/checkout-7d9f:52344 (ID:4211) -> prod/payments-6b4c:8080 (ID:7788) Policy denied DROPPED (TCP Flags: SYN)

# Narrow it down
hubble observe --namespace prod --pod checkout --verdict DROPPED --last 50
hubble observe --from-pod prod/checkout --to-pod prod/payments -f
hubble observe --protocol http --http-status 403           # L7 verdicts, if an http rule applies
hubble observe --type l7 --namespace prod                  # only L7 events
hubble observe --to-fqdn "api.stripe.com"                  # did the FQDN policy resolve?
hubble observe --type drop --output json | jq .          # full flow record, drop reason included

hubble status                                              # is relay seeing all nodes?

Digging into the datapath with cilium-dbg

When you need ground truth about what the kernel actually believes, exec into the agent. cilium-dbg is the in-pod debug CLI (it was simply cilium inside the agent in older releases).

CILIUM_POD=$(kubectl -n kube-system get pods -l k8s-app=cilium -o name | head -1)
K="kubectl -n kube-system exec -it $CILIUM_POD -- cilium-dbg"

$K status --verbose                 # datapath mode, IPAM usage, BPF map pressure, proxy status
$K endpoint list                    # every pod on this node: identity + policy enforcement state
$K identity list                    # the numeric identity ↔ label-set table
$K policy get                       # the compiled policy the agent is actually enforcing
$K bpf lb list                      # the Service load-balancing map — is your backend really in it?
$K bpf tunnel list                  # tunnel endpoints (tunnel mode only)
$K bpf policy get <endpoint-id>     # per-endpoint allow/deny map, keyed by identity
$K monitor --type drop              # raw datapath drop events on THIS node
$K service list                     # Services as the datapath sees them

# Is kube-proxy really gone and is eBPF really handling Services?
$K status | grep -i "KubeProxyReplacement"

Add the two you will reach for under pressure — cilium status and hubble observe --verdict DROPPED — to the muscle memory you are building on Command Reference.

Gotchas and failure modes

☺ Like you’re 10: Four things bite: forgetting to allow DNS, overlapping addresses, an old kernel, and expecting L7 rules to be free.

Default-deny and the DNS trap

Restating it because it causes more outages than everything else combined: policy in Kubernetes is additive allow with implicit deny once selected. Write an egress policy that lists your database and nothing else, and your pod loses name resolution instantly — every hostname lookup fails, and the symptom looks like “the database is down,” not “I broke DNS.” Always allow UDP/TCP 53 to kube-dns, and if you use toFQDNs, the DNS rule must also be present so the DNS proxy can observe the answers. If it looks like a mystery, hubble observe --verdict DROPPED ends the mystery in seconds.

Overlapping PodCIDRs and the Cluster Mesh that will not connect

Cluster Mesh has hard prerequisites that are painful to retrofit: every cluster needs a unique cluster.name, a unique numeric cluster.id, non-overlapping PodCIDRs, and node-to-node reachability between them. Almost every cluster in the world was bootstrapped with 10.244.0.0/16, so two default clusters overlap perfectly and the mesh either refuses to connect or routes traffic to the wrong pod. There is no clever fix — you re-IP a cluster, which in practice means rebuilding it. Allocate CIDRs from a registry before the first cluster exists; it is a five-minute decision that saves a rebuild. See Multi-Cluster and the reference architecture.

Kernel versions and where L7 costs you

• Kernel requirements are real. Cilium needs a reasonably modern kernel, and the documented floor has climbed release by release — current versions expect a 5.x kernel, want 5.10 or newer for a comfortable feature set, and want newer still for things like WireGuard transparent encryption, BIG TCP or the bandwidth manager. Read the system-requirements page for the exact release you are installing rather than trusting a number you once memorised. Managed node images and long-lived enterprise distros are where this bites. cilium status --verbose tells you which features the running kernel actually supports — check it before promising a feature to a tenant.
• L7 rules are not free. L3/L4 policy is pure eBPF in the kernel. The moment a rule includes an http:, dns: or kafka: block, matching traffic is redirected to a userspace proxy on that node — Envoy for http: and kafka:, Cilium’s own DNS proxy for dns:. That is still fast and still sidecar-free, but it adds a hop and real CPU. Use L7 rules where they earn their keep — API boundaries and egress — not on every internal call.
• FQDN policy inherits DNS’s weaknesses. Rules resolve to the addresses the DNS proxy observed, bounded by TTL. Very short TTLs, aggressive in-app DNS caching or clients that bypass cluster DNS entirely will all produce intermittent, maddening denials.
• Standard NetworkPolicy still works. Cilium enforces it faithfully alongside CNPs. Prefer plain NetworkPolicy for ordinary L3/L4 segmentation — it is portable and every engineer already reads it — and reach for a CNP only when you need something it cannot express. Portability is a real asset; do not spend it casually. See Best Practices.

Two owners for one network

If Cilium is also serving Gateway API or mesh duties while Istio or an NGINX ingress controller owns the same traffic path, you get two systems with opinions about the same packet — asymmetric routing, doubled retries, contradictory policy. Pick one owner per layer and write it down in the architecture doc. It is the same one-resource-two-controllers anti-pattern that stalks progressive delivery.

Alternatives and when to choose it

☺ Like you’re 10: There are a few different postal systems. They differ in speed, how much they can inspect, and how much you have to learn.

The field

The honest decision rule

Choose Cilium when at least one of these is true: your Service count or connection churn is large enough that the iptables datapath is measurably hurting you; you need policy expressed in something other than IP addresses — HTTP methods and paths, Kafka topics, or SaaS hostnames; you need to prove what the network did during an incident; or you are building toward multi-cluster with global Services. Choose Calico when a mature, conventional policy engine and BGP are enough and you would rather not take on eBPF operations. Choose Flannel when the cluster is disposable. And be honest about the cost: Cilium has more surface area than the CNI it replaces, it puts a kernel-version constraint on your node images, and it is very hard to swap out later. That is a strategic commitment, not a checkbox — which is precisely why it belongs in the architecture conversation and not in a ticket.

Exam relevance and going further

☺ Like you’re 10: Cilium isn’t named on the exam — but the ideas inside it are. And its website is locked during the test anyway.

What to know cold

Cilium is not on the official CNPE tool list, so nothing here needs to be recalled by product name. What you must own is the concept layer it implements — and the standard Kubernetes objects that express it. Without notes, be able to:

• Say what a CNI is responsible for, and that NetworkPolicy is only enforced if your CNI implements it — Flannel silently ignores it.
• Hand-write a default-deny NetworkPolicy and an allow rule, and explain that a pod is default-deny for a direction the moment any policy selects it.
• Remember to allow DNS egress in any default-deny egress policy.
• Explain what kube-proxy does and the ways to do it: kube-proxy’s own iptables, IPVS and nftables modes, or replacing kube-proxy entirely with an eBPF datapath — and why the last scales best.
• Contrast tunnel/overlay (VXLAN, Geneve — portable, MTU cost) with native routing (fast, requires the underlay to route pod CIDRs).
• Explain identity-based versus IP-based policy in one sentence, and why IPs are a bad identity in Kubernetes.
• Name what a cluster mesh buys you — global Services, cross-cluster failover, shared identity — and its hard prerequisite of non-overlapping PodCIDRs.

The documentation allowlist — read this twice

⚖ CNPA vs CNPE — That allowlist is a CNPE-only mechanic — CNPA permits none of it. CNPA is fully closed-book multiple-choice: no kubernetes.io, no man pages, no lookups of any kind, which makes it stricter than CNPE, not looser. Even so, the concept layer above — what a CNI does, default-deny semantics, identity-based policy — is exactly the kind of concept-level knowledge that pays off in CNPA's closed-book recall.

Practically: budget one study session for hand-writing a default-deny NetworkPolicy plus a DNS-allow rule on a blank page, then checking yourself against Know Cold. Rehearse the diagnosis half with Triage: Networking and the troubleshooting playbook, and read the parent lesson on Networking & Service Mesh at least twice.

Official links (for study time, not exam time)

• docs.cilium.io — the canonical documentation, including the full CiliumNetworkPolicy reference and every Helm value.
• Kubernetes without kube-proxy — the kube-proxy replacement guide, including the k8sServiceHost requirement.
• Cluster Mesh setup — the prerequisites list worth reading before you allocate any CIDRs.
• Hubble observability and github.com/cilium/cilium — the source, release notes and the connectivity-test suite.
• kubernetes.io — Network Policies — the one page in this list that is on the exam allowlist. Know your way around it.
• Related on this site: Networking & Service Mesh (the parent lesson), Security & Policy, Observability, Multi-Cluster, and Linkerd or Istio for what may sit above it.