Cilium

The CCA blueprint spends 38% of its weight — Architecture and Network Policy together — on ideas this page turns into running clusters: an eBPF datapath compiled and loaded by cilium-agent, IPAM and routing-mode choices you set once and live with for a long time, and a CiliumNetworkPolicy that enforces on a pod's identity rather than its ever-changing IP. Cilium is the CNCF-graduated CNI underneath a growing share of production Kubernetes — the datapath GKE's Dataplane V2 is built on, and the reference implementation the CCA exam tests by name rather than by concept. This page is the operational half of that story: the agent/operator split, what IPAM and routing mode actually decide, the policy YAML and CLI verbs you'll type for real, day-to-day Hubble commands, the failure modes that catch people who only read the happy path, and where Cilium sits against Calico, Flannel and a cloud-native CNI. The kernel machinery underneath all of it — the verifier, the JIT, BTF/CO-RE — has its own page; this one assumes eBPF exists and gets straight to running it.

Architecture: cilium-agent, cilium-operator, and the rest of the fleet

☺ Like you're 10: One crew member rides on every single node doing the real work; one supervisor sits above the whole fleet doing the jobs that must happen exactly once, not once per node.

A Cilium install lays down a small, purpose-built set of components, and knowing which one owns which job is most of what makes an incident fast instead of slow. cilium-agent runs as a DaemonSet, one pod per node — it watches the Kubernetes API, computes a security identity for every pod on its node from that pod's labels, and compiles the current set of Services, endpoints and CiliumNetworkPolicy objects into eBPF programs and maps that it loads straight into the node's kernel. It also installs the CNI plugin binary and drives a per-node Envoy proxy for anything that needs Layer 7 parsing. cilium-operator runs as a small Deployment, once per cluster, and does exactly the work that must not be duplicated per node: allocating IPAM blocks in cluster-pool mode, and garbage-collecting stale CiliumIdentity and CiliumEndpoint objects nobody references any more. It is not on the datapath — kill the operator and traffic already flowing keeps flowing; new pods may simply stall waiting on an IP.

Hubble is embedded directly inside every agent, reading flow events out of the eBPF datapath as they happen; hubble-relay is a small Deployment that fans out to every node's local Hubble and presents one cluster-wide flow API, which is what the hubble CLI and hubble-ui actually talk to. And when a cluster joins a Cluster Mesh, an optional clustermesh-apiserver Deployment exposes this cluster's identities, endpoints and Services to its peers.

Cilium began inside Isovalent (originally Covalent), open-sourced in 2015, and became a CNCF-graduated project in October 2023 — the first CNI to reach that bar. Isovalent itself was acquired by Cisco two months later. None of that history is examinable, but it explains why Cilium's pace of feature delivery — kube-proxy replacement, Gateway API, Cluster Mesh, a full service mesh — has looked less like a CNI plugin's roadmap and more like a platform vendor's.

IPAM and routing mode: the two decisions that shape everything after

☺ Like you're 10: One choice decides how pods get their addresses; the other decides whether traffic between nodes travels inside a sealed tube or straight down the open road.

IPAM mode decides where a pod's IP comes from. kubernetes mode uses the per-node PodCIDR the controller-manager already assigns — the simplest option, and the one with the least Cilium-specific behavior. cluster-pool, Cilium's own default, has the operator carve fixed-size blocks (a /24 is typical, roughly 250 usable pod IPs per node) out of one cluster-wide pool it owns, and hand a block to each node's agent as it joins. The cloud modes — eni on AWS, azure, alibabacloud — skip an overlay network entirely and hand pods real VPC addresses, which is powerful and firmly bounded by your cloud's per-node interface and IP limits.

Routing mode decides how a packet actually gets from one node to another. Tunnel mode — the portable default — encapsulates pod traffic in VXLAN or Geneve between nodes, so the underlay only ever needs to route node IPs to each other; it never has to learn a single pod CIDR. Native routing forwards packets unencapsulated, which removes the per-packet encapsulation cost and the roughly 50-byte MTU tax tunnel mode pays — at the price of requiring the underlay to actually route your pod CIDRs, either because Cilium can lay down direct routes itself on a flat L2 segment (autoDirectNodeRoutes), or by announcing pod CIDRs and LoadBalancer VIPs to real routers with Cilium's built-in BGP control plane.

Alongside those two sits kube-proxy replacement: with kubeProxyReplacement: true, eBPF implements ClusterIP, NodePort, LoadBalancer and session affinity entirely in the kernel, and kube-proxy can be deleted outright. ClusterIP translation happens at the socket layer — a pod's connect() call is rewritten to a live backend's real address before a packet is ever built — so there is no per-packet DNAT cost to pay at all. Because the agent must be able to reach the API server without kube-proxy in the picture, k8sServiceHost and k8sServicePort have to be set explicitly.

The CiliumNetworkPolicy you actually write

☺ Like you're 10: One file says which pods it protects, then a list of who may knock and what they're allowed to ask for — by badge, never by room number.

A standard Kubernetes NetworkPolicy can say "port 8080, yes or no." CiliumNetworkPolicy (CNP) is the same underlying idea with far more grammar: an endpointSelector names which pods the policy protects, then ingress/egress entries pair a peer selector — fromEndpoints/toEndpoints by label, fromEntities/toEntities for special sets like world or cluster, or toFQDNs by hostname — with toPorts, which can optionally carry L7 rules for HTTP, DNS or Kafka, parsed by the per-node Envoy proxy.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: payments-api
  namespace: prod
spec:
  endpointSelector:                 # WHICH pods this policy protects
    matchLabels:
      app: payments
  ingress:
    - fromEndpoints:                # WHO may call — by LABELS, never by IP
        - matchLabels:
            app: checkout
            io.kubernetes.pod.namespace: prod
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:                   # L7: parsed by the per-node Envoy proxy
              - method: "GET"
                path: "/v1/balance/[0-9]+"
              - method: "POST"
                path: "/v1/charge"
  egress:
    - toEndpoints:                  # DNS FIRST — see the warning below
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports: [{ port: "53", protocol: ANY }]
          rules:
            dns: [{ matchPattern: "*" }]
    - toFQDNs:                      # egress by hostname, not by IP range
        - matchName: "api.stripe.com"
      toPorts:
        - ports: [{ port: "443", protocol: TCP }]

Three policy enforcement modes govern when any of this applies at all: default (a pod allows everything until a policy's selector picks it out, at which point that direction becomes default-deny), always (default-deny everywhere from the start), and never. default is what almost every cluster runs, and it's also the source of the single most common Cilium outage.

Standard Kubernetes NetworkPolicy still works — Cilium enforces it faithfully alongside CNPs, and it's the more portable choice for ordinary L3/L4 segmentation. Reach for a CNP specifically when you need something the standard object cannot express: L7 rules, toFQDNs, entity selectors, explicit deny, or the cluster-scoped CiliumClusterwideNetworkPolicy a platform team uses as a baseline tenants can't delete. Network policy governs what packets may flow; it is a different plane from what Kyverno governs — which manifests may be admitted in the first place — and the two are complementary halves of the same zero-trust argument, not substitutes for each other.

Installing and upgrading with the Cilium CLI

☺ Like you're 10: One file switches the smart network on with the settings you actually want; a couple of commands install it, prove it's healthy, and check every kind of trip actually works.

Cilium installs via Helm — the cilium CLI is a thin, opinionated wrapper around exactly that — which means the whole datapath configuration is one reviewable values file, manageable through Argo CD or Flux the same way any other GitOps-managed workload is.

# values.yaml — a production-shaped Cilium install
kubeProxyReplacement: true          # eBPF handles ClusterIP/NodePort/LoadBalancer
k8sServiceHost: api.mission.internal   # REQUIRED without kube-proxy in the picture
k8sServicePort: 6443

routingMode: native                 # "tunnel" (default) or "native"
ipv4NativeRoutingCIDR: 10.244.0.0/16   # required in native mode: what NOT to masquerade
autoDirectNodeRoutes: true          # only valid when every node shares an L2 segment

ipam:
  mode: cluster-pool                # kubernetes | cluster-pool | eni | azure
  operator:
    clusterPoolIPv4PodCIDRList: ["10.244.0.0/16"]
    clusterPoolIPv4MaskSize: 24     # a /24 per node — roughly 250 usable pod IPs

hubble:
  enabled: true
  relay: { enabled: true }
  ui:    { enabled: true }

cluster:
  name: eu-west-1-prod              # MUST be unique across a Cluster Mesh
  id: 3                             # MUST be unique, 1–255

gatewayAPI:
  enabled: true                     # needs the Gateway API CRDs installed first

# Install / upgrade — always pin a version, always read that release's upgrade notes
$ cilium install --version <x.y.z>
$ cilium upgrade --version <x.y.z>

# The first question, always: are agents, operator and Hubble healthy?
$ cilium status --wait
# /¯¯\__/¯¯\    Cilium:             OK
# \__/¯¯\__/    Operator:           OK
#    \__/       Hubble Relay:       OK
# DaemonSet  cilium   Desired: 6, Ready: 6/6, Available: 6/6

# The end-to-end proof: pod-to-pod, pod-to-service, DNS, egress, policy enforcement
$ cilium connectivity test

# Query and MODIFY live config (edits the cilium-config ConfigMap and bounces agents —
# on a GitOps-managed cluster, change the values file instead and let the reconciler do it)
$ cilium config view
$ cilium config set enable-l7-proxy true
$ cilium sysdump                    # one archive with everything support will ask for

# Cluster Mesh — see the Cluster Mesh domain on the CCA blueprint for the full model
$ cilium clustermesh enable  --context eu-west --service-type LoadBalancer
$ cilium clustermesh connect --context eu-west --destination-context us-east
$ cilium clustermesh status  --context eu-west --wait

Hubble: watching every flow live

☺ Like you're 10: Instead of arguing about whether the network dropped something, you just ask it — and it tells you the exact rule that said no.

Because the agent already inspects every packet to enforce policy, exporting what happened and why is close to free — source and destination identity, port, protocol, verdict (FORWARDED, DROPPED, AUDIT) and a drop reason when there is one. That's what makes the single most useful command on this page also the shortest one to type:

# Every dropped flow in the cluster, live
$ hubble observe --verdict DROPPED -f
# prod/checkout-7d9f:52344 (ID:4211) -> prod/payments-6b4c:8080 (ID:7788) \
#   Policy denied DROPPED (TCP Flags: SYN)     <-- identities, not just IPs

$ hubble observe --namespace prod --pod checkout --last 50   # scope it down
$ hubble observe --type l7 --protocol http --http-status 403  # L7 verdicts
$ hubble observe --to-fqdn "api.stripe.com"                   # did toFQDNs resolve?
$ hubble status                                                # is relay seeing every node?

L7 visibility isn't free by default — a flow only gets parsed above L4 when a policy attaches an L7 rules block to it, which is the mechanism that steers matching traffic through Envoy in the first place. This page covers only the CLI verbs worth having in muscle memory; the full Network Observability domain — the Hubble UI's service map, enabling L7 visibility without narrowing what's allowed, and metrics wired into Prometheus and Grafana — gets its own page in Hubble, next in this section.

Gotchas and failure modes

☺ Like you're 10: Most surprises come from forgetting the DNS rule, two ships picking the same address, an old engine that can't run the new part, or expecting the fancy inspection to be free.

Overlapping PodCIDRs stall Cluster Mesh forever

Cluster Mesh has hard prerequisites that are painful to retrofit: every participating cluster needs a globally unique cluster.name, a unique numeric cluster.id, strictly non-overlapping PodCIDRs, and node-to-node reachability between them. Nearly every cluster on Earth was bootstrapped with 10.244.0.0/16, so two default clusters overlap perfectly and the mesh either refuses to connect or, worse, silently routes to the wrong pod. There is no clever fix once two live clusters collide — you re-IP one, which in practice means rebuilding it. Allocate CIDRs from a registry before the first cluster exists; it's a five-minute decision that saves a rebuild months later.

Kernel version skew changes behavior node to node

Cilium's documented minimum kernel has climbed release by release — current versions expect a 5.x kernel, want 5.10 or newer for a comfortable feature set, and want newer still for WireGuard transparent encryption, BIG TCP or the bandwidth manager. Managed node images and long-lived enterprise distros are where this quietly bites: the same Cilium image can behave differently on two nodes with different kernel minor versions, and it isn't always obvious that's the cause. cilium status --verbose on a given node reports which features its running kernel actually supports — check it before promising a feature to a tenant, and see eBPF & the Cilium Datapath for why the verifier itself is part of what shifts between kernel versions.

L7 rules are not free

L3/L4 policy is pure eBPF, resolved entirely in the kernel. The instant a rule includes an http:, dns: or kafka: block, matching traffic gets redirected to the per-node Envoy proxy (or Cilium's own DNS proxy for dns:) — still sidecar-free and still fast, but it adds a real hop and real CPU that L3/L4-only policy never pays. Reach for L7 rules where they earn their keep — API boundaries, sensitive egress — not on every internal call by default.

FQDN policy inherits DNS's own weaknesses

toFQDNs rules resolve to whatever address the DNS proxy actually observed, bounded by that record's TTL. Very short TTLs, aggressive in-app DNS caching, or a client that bypasses cluster DNS entirely will all produce intermittent, maddening denials that look like a Cilium bug and are actually a caching mismatch one layer up.

Cilium vs the alternatives

☺ Like you're 10: A few different postal systems exist. They trade off speed, how closely they can inspect a letter, and how much you have to learn to run one.

The honest version of that decision: Cilium is a strategic commitment, not a checkbox. It has more surface area than the CNI it replaces, it puts a real kernel-version floor on your node images, and — per the CNI-swap warning above — it is genuinely hard to walk back once workloads depend on identity-based policy and toFQDNs egress. That's exactly why it belongs in a platform's architecture conversation and not in a ticket, and it's most of why the CCA exists as its own credential in the first place: Cilium is examined by name, the way CGOA examines a vendor-neutral GitOps specification instead. If this page's job was the CCA blueprint made operational, two others cover the same tool from different angles worth knowing about: the sibling Kubernetes course's own Cilium page approaches it from inside a running cluster rather than an exam curriculum, and Platform Engineering's deeper operational reference goes further into production-scale gotchas for anyone continuing on to the CNPE afterward.