Calico
This course's Networking & the CNI page already covers the CNI contract in general — the ADD/DEL verbs, veth pairs, and why Flannel, Calico, and Cilium all exist to answer the same question differently. This page goes one level deeper into a single answer: Calico, maintained as an open source project by Tigera. Calico bundles two jobs most teams eventually need separately anyway — wiring the flat Pod network, by default with real BGP routes rather than an encapsulated tunnel, and enforcing NetworkPolicy once that network exists, with its own richer policy CRDs layered on top of the stock Kubernetes API. What follows is the architecture that makes both jobs work, the resources you actually write (IPPool, BGPPeer, GlobalNetworkPolicy, HostEndpoint), the CLI split between kubectl and calicoctl, and the handful of gotchas — an immutable CIDR, a forgotten NAT flag, an MTU nobody adjusted — that separate a calm Calico rollout from a confusing one.
Most delivery services route every package through one central sorting depot, even if the sender and the receiver live two doors apart — that's an overlay network, and it works, but it's a detour. Calico's favorite trick is to skip the depot: it teaches the actual road signs (the real network routers, using a language called BGP) where every house's mailbox is, so a package can drive straight there. When the roads themselves can't be taught new signs — say, on a rented street where the landlord controls all the road markings, like a lot of clouds do — Calico falls back to a delivery van with a shipping label wrapped around the package instead (that's the IP-in-IP or VXLAN overlay). Either way, once a package is moving, a guard at every single mailbox checks it against a list before letting it in — that's NetworkPolicy, and it's Calico's other whole job, completely separate from how the package got there.
What Calico actually is: a routing engine and a policy engine, bundled
☺ Like you're 10: One project, two jobs — getting a package to the right mailbox, and deciding who's allowed to knock once it's there.
Unlike Cilium, which this course's CNCF project landscape page notes graduated the CNCF in 2023, Calico is not itself a CNCF-hosted project — it's developed in the open by Tigera, which also sells a commercial Calico Enterprise / Calico Cloud layer (a UI, compliance reporting, multi-cluster federation) on top of the same free, open source core this page covers. That core ships with a choice of dataplanes: the default is Linux iptables, backed by ipsets for fast set matching rather than long linear rule chains; an eBPF dataplane is available as a faster, kube-proxy-replacing alternative with its own kernel-version floor; and a Windows HNS dataplane means Calico is one of the few CNI options with genuine native Windows-node support, which matters more than it sounds once a cluster mixes Linux and Windows node pools.
Keep the two jobs mentally separate, because the rest of this page is organized around exactly that split. Routing is "how does a packet addressed to a Pod IP physically get there" — Calico's answer is BGP by default, encapsulation as a fallback. Policy is "should that packet have been allowed to arrive at all" — Calico's answer is a full NetworkPolicy implementation plus its own extended CRDs. A cluster can use Calico for routing and something else for policy, or the reverse (Calico-for-policy-only on top of a cloud's native CNI is a documented, common pattern) — the bundle is a convenience, not a requirement.
Architecture: Felix, BIRD, Typha, and where state actually lives
☺ Like you're 10: Every node runs its own little pair of workers — one who programs the local traffic rules, one who tells the neighborhood which roads lead here — and a middleman keeps them from all pestering headquarters at once.
The calico-node DaemonSet puts one Pod on every node, and that Pod bundles two long-running processes plus a small glue component. Felix is the per-node agent: it watches the datastore for this node's workloads and policies, then programs the actual dataplane — iptables/ipset rules, routes, and (in eBPF mode) compiled BPF programs — to match. BIRD is a general-purpose BGP daemon that Calico drives to advertise this node's Pod IP ranges to its BGP peers and to learn routes to every other node's ranges in return; confd is the small piece that watches the datastore and renders BIRD's config from it. The datastore itself is usually the Kubernetes API server today — Calico's own resource types (IPPool, BGPPeer, NetworkPolicy, and the rest) are registered as CRDs under the projectcalico.org/v3 API group, so no separate etcd cluster is required, though the older standalone-etcd datastore mode still exists for clusters that were built on it before Kubernetes-datastore mode matured.
One more component sits off to the side: Typha, a fan-out proxy that batches many Felix instances' watches into one shared connection to the API server. Below roughly fifty nodes, every Felix can watch the API server directly without much strain; past that, hundreds of direct long-lived watch connections start to cost the control plane real CPU, and Typha is the fix — the Tigera operator install enables it automatically once cluster size crosses its threshold, but a raw manifest install may need it turned on by hand. A separate, cluster-scoped calico-kube-controllers Deployment rounds out the picture: it reconciles plain Kubernetes objects — Namespaces, ServiceAccounts, NetworkPolicies, and crucially deleted Nodes — into Calico's own datastore state, which is what actually reclaims IPAM allocations left behind when a node disappears.
BGP routing: full mesh, route reflectors, and when to skip the overlay entirely
☺ Like you're 10: Every node can either shout its address to every other node directly, or tell it to just two well-connected "town criers" instead — the second option scales a lot better.
By default, every calico-node's BIRD instance peers with every other node's BIRD instance — a node-to-node full mesh, using a shared private ASN (Calico's default is 64512) since a full mesh doesn't need each node to be a distinct autonomous system. That's zero configuration and works fine on a small cluster, but a full mesh means each node holds N-1 BGP sessions — session count grows roughly with the square of node count, and somewhere around fifty to a hundred nodes that starts costing real convergence time and control-plane churn on every node join or leave. The documented fix is route reflectors: designate a small number of nodes (or dedicated RR instances) as reflectors, disable the mesh, and have every other node peer only with the reflectors — session count per node drops to a handful regardless of cluster size, at the cost of one extra layer to reason about when a BGP session actually goes down.
Separate from peering topology is encapsulation, an IPPool-level choice between three modes: Never (native routing — BIRD advertises the raw Pod CIDR block, no wrapping at all, fastest but needs the underlying network, or a BGP peer to it, willing to route those addresses), Always (wrap every Pod packet in IP-in-IP or VXLAN, safest default when you're not sure what the fabric will tolerate), and CrossSubnet (route natively within a subnet, encapsulate only when a packet must cross one — the common middle ground on multi-rack on-prem clusters). On most managed clouds, native routing straight into the underlying SDN isn't an option Calico can negotiate at all, so Always or CrossSubnet VXLAN is the realistic default there — full BGP-to-the-fabric is squarely an on-prem and self-managed-cluster capability, and it's the single biggest reason a team picks Calico specifically over an equally capable overlay-only CNI.
# BGPConfiguration — one cluster-wide object, name is always "default" apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: logSeverityScreen: Info nodeToNodeMeshEnabled: false # turn the full mesh OFF once BGPPeers below take over asNumber: 64512 # cluster's own private ASN (Calico's default) --- # BGPPeer — every node peers with the two nodes labeled as route reflectors apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: peer-with-route-reflectors spec: peerSelector: route-reflector == 'true' nodeSelector: all() --- # BGPPeer — this rack's nodes also peer straight into the physical ToR router, no overlay apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: rack-1-tor-router spec: peerIP: 10.0.1.1 asNumber: 65001 nodeSelector: rack == 'rack-1'
Setting nodeToNodeMeshEnabled: false takes effect immediately across the whole cluster. If the matching BGPPeer objects pointing at route reflectors aren't already applied and Established, every node loses its only source of routes to every other node's Pod CIDR block at once — existing connections keep working off cached kernel routes for a while, but new Pods on other nodes become unreachable. Apply and verify the BGPPeers first, confirm with calicoctl node status from more than one node, then flip the mesh off.
IPAM: IPPool, block allocation, and the NAT flag everyone forgets once
☺ Like you're 10: Instead of asking headquarters for one address at a time, each node grabs a whole street of addresses up front, so it can hand them out fast without calling in constantly.
Calico's own IPAM plugin (calico-ipam, the default — the plain CNI reference plugin host-local also works but skips Calico's smarter allocation entirely) allocates addresses in blocks, not one at a time. Each IPPool's CIDR is carved into fixed-size blocks — blockSize: 26 is the IPv4 default, a /26 giving 64 addresses per block — and a node claims a whole block for itself the first time it needs an address there, then hands out individual Pod IPs from that local block without touching the datastore again until the block runs low. A node can hold more than one block, and can even borrow a block that was originally claimed by a different, now-idle node, which is what keeps a cluster from stranding IPs on a node that scaled down. natOutgoing: true on a pool is what makes Pod traffic leaving the cluster get source-NAT'd to the node's own IP — without it, a packet leaving toward the public internet still carries the Pod's un-routable cluster-internal source address, and it simply never gets a reply.
apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: default-ipv4-ippool spec: cidr: 192.168.0.0/16 blockSize: 26 # /26 per node-claimed block = 64 Pod IPs each ipipMode: CrossSubnet # encapsulate only when a packet must leave this subnet vxlanMode: Never # pick ONE encapsulation mode, never both at once natOutgoing: true # SNAT Pod egress to the node IP — see the gotcha below nodeSelector: all() --- # a second pool, scoped to one node group, with its own block size — easy to forget natOutgoing here apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: gpu-nodes-pool spec: cidr: 192.168.128.0/20 blockSize: 28 # smaller blocks — fewer, shorter-lived Pods per GPU node ipipMode: Never # this rack has a flat L2 fabric — route natively natOutgoing: true nodeSelector: workload-type == 'gpu'
Pools can also be scoped by nodeSelector the way the second example shows — useful for giving a distinct node group its own address range, its own block size, or its own encapsulation mode without touching the cluster-wide default. The trade-off is that every new pool is a genuinely fresh set of choices: nothing about natOutgoing, ipipMode, or the block size is inherited from the pool sitting next to it, which is exactly the gap the gotcha below walks through.
The policy resources you actually write: NetworkPolicy, GlobalNetworkPolicy, HostEndpoint
☺ Like you're 10: The plain guest list works everywhere; Calico's own, fancier guest list can also rank the rules in order and even guard the building's front door, not just the apartments — but only Calico's buildings understand it.
Calico fully implements the stock Kubernetes networking.k8s.io/v1 NetworkPolicy API — the namespaced, podSelector-based object Services & Networking and RBAC & Admission Control already cover, and the one worth writing whenever portability across CNIs matters. Calico's own projectcalico.org/v3 API group adds a genuinely richer, Calico-specific set of CRDs on top: its own namespaced NetworkPolicy kind supports an order field (lower numbers evaluate first) and explicit action: Allow | Deny | Log | Pass rules instead of pure allow-lists; GlobalNetworkPolicy is the cluster-scoped equivalent — not bound to one namespace, and able to select Pods across the whole cluster or even the nodes themselves; GlobalNetworkSet holds a reusable named list of external CIDRs a policy can reference; and HostEndpoint represents a node's own network interface, letting a GlobalNetworkPolicy secure host-level traffic — SSH, the kubelet port, a node's own outbound connections — the same way an ordinary policy secures a Pod.
# Stock Kubernetes NetworkPolicy — portable to any CNI that implements the upstream API
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: checkout-default-deny
namespace: checkout
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
---
# Calico's own GlobalNetworkPolicy — cluster-scoped, ordered, NOT portable to another CNI
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: platform-baseline
spec:
selector: all() # every Pod, every namespace, cluster-wide
order: 100 # low number = evaluated early, ahead of app-team policies
types: [Egress]
egress:
- action: Allow # always allow DNS, whatever else this policy denies
protocol: UDP
destination: { selector: 'k8s-app == "kube-dns"', ports: [53] }
- action: Deny # everything else egress-bound is denied by this baseline
---
# HostEndpoint — lets a policy secure the NODE's own interface, not just the Pods on it
apiVersion: projectcalico.org/v3
kind: HostEndpoint
metadata:
name: node1-eth0
labels: { role: k8s-node }
spec:
node: node1
interfaceName: eth0
expectedIPs: ["10.0.1.5"]order, action: Deny, GlobalNetworkPolicy, and HostEndpoint only exist because Calico defined them — swap the CNI to Cilium or Antrea and none of it parses. A manifest written entirely in the stock networking.k8s.io/v1 NetworkPolicy shape stays portable; reach for Calico's extended CRDs deliberately, for capability the stock API genuinely lacks (explicit ordering, cluster-wide scope, securing the node itself), not out of habit.
Day-to-day commands: kubectl, calicoctl, and which one owns what
☺ Like you're 10: Most of Calico's paperwork is filed in the same cabinet kubectl already knows how to open — but a couple of drawers only Calico's own key fits.
Running in Kubernetes-datastore mode — the common default — means IPPool, BGPPeer, NetworkPolicy, GlobalNetworkPolicy, and the rest are genuine CRDs, so plain kubectl get/apply/describe works on all of them without installing anything extra. calicoctl, Calico's own CLI, earns its place for two things kubectl structurally can't do: reading live per-node status straight from Felix's local status API — BGP session state, in particular, isn't stored as a Kubernetes object anywhere — and IPAM introspection deep enough to spot and reclaim addresses Felix never released cleanly. It also becomes mandatory, not just convenient, on any cluster still running the older standalone-etcd datastore mode.
# which namespace calico-node runs in depends on install method # (Tigera operator install → calico-system; raw manifest install → kube-system) $ kubectl -n calico-system get pods -l k8s-app=calico-node $ kubectl -n calico-system logs -l k8s-app=calico-node -c calico-node --tail=50 # BGP peer status — lives on the node, not in the datastore, so only calicoctl shows it $ calicoctl node status Calico process is running. IPv4 BGP status +--------------+-------------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+-------------------+-------+----------+-------------+ | 10.0.1.1 | node specific | up | 09:14:02 | Established | +--------------+-------------------+-------+----------+-------------+ # IPAM introspection — find and reclaim IPs Felix never released (e.g. after a hard node loss) $ calicoctl ipam show --show-blocks $ calicoctl ipam check $ calicoctl ipam release --ip=192.168.4.12 # plain kubectl works fine for CRUD on Calico's own CRDs $ kubectl get ippools.crd.projectcalico.org $ kubectl get bgppeers.crd.projectcalico.org $ kubectl apply -f platform-baseline-policy.yaml $ kubectl describe globalnetworkpolicy platform-baseline # per-Pod interface + IP mapping — useful the moment a Pod IP simply won't route $ calicoctl get workloadendpoint -n checkout -o wide
Bring up a kind cluster with its default CNI disabled (disableDefaultCNI: true in the kind config), apply a Calico manifest on top, and run calicoctl node status once Pods are scheduled — with a single node there's no BGP peer to show yet, so add a second kind node and watch a session appear. Then apply the checkout-default-deny policy above, confirm with kubectl exec that Pod-to-Pod traffic in that namespace actually stops, and only then add an explicit allow rule back. If your kubectl reflexes feel slow going in, this course's kubectl tool guide and fluency baseline are the right warm-up first.
Gotchas that bite in production
☺ Like you're 10: A few of Calico's settings only ever surprise you once — after that, you check them first, every time.
The one to internalize before any other: an IPPool's cidr is immutable after creation. Realize the pool was sized wrong and there's no in-place fix — the documented path is creating a new, correctly-sized pool, letting new Pod allocations land in it, and only deleting the old pool once nothing is still using addresses from it, which on a live cluster means a genuine, sometimes multi-day migration rather than an edit. Encapsulation carries its own quiet tax: IP-in-IP adds roughly 20 bytes of overhead per packet and VXLAN roughly 50, and if the node or NIC's MTU isn't adjusted downward to account for it — Calico can auto-detect this in most installs, but not always — large packets get silently fragmented or dropped while small ones sail through untouched, which is exactly the kind of bug that only shows up under real payload sizes, days after a change nobody suspects.
Every field on an IPPool — natOutgoing very much included — is independent per pool. Add a new pool for a new node group and forget to set natOutgoing: true on it, and Pods landing in that pool can reach every other Pod in the cluster fine, but time out reaching anything outside it: their packets leave carrying an un-routable cluster-internal source address that nothing on the public internet can send a reply back to. It reads exactly like a firewall problem and is actually one missing boolean.
Two more worth carrying into any real rollout. Running eBPF mode is an operational commitment, not a config flag flip — it needs a recent-enough Linux kernel (5.3 or newer is the commonly cited floor) and typically means stopping kube-proxy entirely, since Calico's own eBPF programs take over Service load-balancing. And Typha isn't automatic on every install path: a cluster that started small on a raw manifest install and grew past a few dozen nodes without anyone revisiting that choice can end up with every Felix hammering the API server directly, showing up as control-plane latency that has nothing obviously to do with networking until someone thinks to check.
"The first time a GPU node pool couldn't reach the internet, I spent twenty minutes convinced it was a firewall or a proxy config, because every other Pod on every other pool was completely fine. It wasn't a firewall at all — it was one IPPool with natOutgoing left at its default. Now the very first thing I check when 'this one node group can't reach the internet, but everything else can' shows up is kubectl get ippool -o yaml and a scan for that one field, before I go anywhere near actual network tooling."
Calico vs. the alternatives — and when teams actually choose it
☺ Like you're 10: Every option trades some power for some simplicity — Calico's trade is real BGP muscle for teams who already have somewhere to point it.
| Option | Model | Best when | Costs you |
|---|---|---|---|
| Calico | iptables + BGP by default (native routing or overlay); eBPF and Windows HNS dataplanes also available | On-prem/hybrid with a real BGP fabric to peer into; rich NetworkPolicy without eBPF's kernel/ops bar; mixed Linux+Windows node pools | Extended CRDs (order, GlobalNetworkPolicy) don't travel to another CNI; full mesh needs route reflectors past a few dozen nodes |
| Cilium (see also PE's Cilium tool page) | eBPF, in-kernel, identity-based policy | L7-aware policy, Hubble flow observability, or kube-proxy-replacement is a hard requirement | Kernel-version floor; more moving parts to run and tune correctly |
| Flannel | VXLAN overlay only | A lab cluster, or "just make Pods reach each other" with nothing else required | No NetworkPolicy engine of its own at all |
| Cloud-native CNI (AWS VPC CNI, Azure CNI, GKE native) | Hands Pods real VPC IPs directly — no overlay | Deep native integration with the cloud's own routing and security groups matters more than portability | Consumes VPC IP address space fast; usually needs Calico (or similar) layered on for policy alone |
The rule of thumb most platform teams settle on: reach for Calico specifically when BGP is already a language your network team speaks, or when you want mature, well-understood NetworkPolicy enforcement without committing to eBPF's kernel requirements and operational surface everywhere at once. This course's own Networking & the CNI page has the fuller Flannel/Calico/Cilium comparison and the CNI spec underneath all three; NetworkPolicy design itself is CKS territory, covered in this course's own CKS blueprint and, for default-deny namespace patterns specifically, DevSecOps' Kubernetes security deep dive. Calico itself isn't part of the CNCF's project-specific certification ladder the way Cilium (CCA) is — the sibling Golden Astronaut course maps the certifications that do exist for CNI-adjacent projects, if that's the direction you're headed next.
Pip the Hummingbird: The new GPU node pool's Pods can reach every other Pod in the cluster fine — but the model-registry download outside the cluster just times out, every time.
Foxy: Is it DNS not resolving, or is the packet leaving and just never getting an answer back?
Nutty the Squirrel: Let me dig — those Pods landed in gpu-nodes-pool, a different IPPool from the default one. Comparing the two specs now.
Nutty: Found it. The default pool has natOutgoing: true. The GPU pool never got that field set at all — it's false by default. Egress packets are leaving with an un-routable Pod IP as the source.
Gizmo: Skip the YAML — just set hostNetwork: true on the GPU Pods. They'll use the node's own IP, problem solved in one line. 🤑
Timmy the Turtle: No. hostNetwork takes those Pods clean out of the Pod network namespace — every NetworkPolicy selecting them by Pod IP stops applying, and now they share the node's own network stack entirely. Patch the one field that's actually wrong: natOutgoing: true on gpu-nodes-pool.
Benny the Beaver: Patched and applied. calicoctl ipam show and a fresh Pod both confirm egress traffic now carries the node's IP.
1. Name Calico's two jobs, and which component — Felix or BIRD — does which. 2. What does Typha actually solve, and roughly when does a cluster start needing it? 3. An IPPool is set to ipipMode: Never on a cluster where the physical network won't route Pod CIDR blocks. What breaks? 4. Why can't a GlobalNetworkPolicy manifest be applied unchanged against a Cilium cluster? 5. New Pods on a freshly added IPPool can reach other Pods but not the public internet. What's the single most likely field to check first? 6. Name one thing calicoctl can show you that plain kubectl cannot, even in Kubernetes-datastore mode.
Check your answers
- Routing the flat Pod network, and enforcing
NetworkPolicy. Felix programs the local dataplane (iptables/ipset or eBPF) for both jobs; BIRD handles BGP peering and route advertisement for the routing half specifically. - Typha fans out many Felix instances' watches of the API server into one shared connection, so hundreds of nodes don't each hold their own direct watch. Clusters typically start needing it somewhere around fifty nodes, though the Tigera operator install often enables it automatically past a size threshold.
- Pod-to-Pod traffic that has to leave the node fails to route, because
Nevermeans BIRD advertises the raw, un-encapsulated Pod CIDR block and nothing in the physical network knows what to do with those addresses — there's no fallback wrapping to fall back on. - Because
order,action: Deny, and theGlobalNetworkPolicykind itself are Calico-specific CRDs underprojectcalico.org/v3— Cilium doesn't register or understand that API group at all, so the manifest simply fails to apply. natOutgoingon that pool — if it's left at its default (unset/false) instead of explicitly set totrue, egress packets leave carrying an un-routable cluster-internal source IP and never get a reply.- Live BGP peer session state (
calicoctl node status) — it's read from Felix's local status API on that node, not stored as a Kubernetes object anywherekubectlcould show it.