Platform Engineering in Depth · Multi-Cluster & Fleet Management

Multi-Cluster & Fleet Management

Sooner or later, one cluster stops being enough. A regulator asks where the data lives; a bad upgrade takes down everything at once; a team wants an isolated blast radius; latency demands a presence on three continents. The moment you have a second cluster, you have a new problem that no single cluster ever posed: how do you treat many clusters as one fleet — created the same way, configured the same way, patched the same way, observed the same way — so that none of them quietly becomes a snowflake? This page is the deep tour of fleet engineering: why you split, how you carve the fleet up, how you stamp clusters out like cattle with Cluster API, how you push config to hundreds of them from Git, how you knit their networks together, how you keep the guardrails identical everywhere, how you place a workload across them, and — just as important — when the honest answer is “don’t add another one.”

☺ Explain it like I’m 10

Imagine you run one big treehouse for the whole neighbourhood. It’s great — until it gets so crowded that one broken plank drops everyone, and kids from far away have to walk an hour to reach it. So you build more treehouses: one per park, one for the little kids, one across town. Now the hard part isn’t building treehouses — it’s making sure all of them have the same ladder, the same safety net, and the same rules, so no single treehouse turns into the weird dangerous one nobody understands. Fleet management is the job of running many treehouses as if they were one — building them from the same blueprint and keeping them all in step.

🦉🦫Your hosts for this topic: Professor Owl & Benny the Beaver — Owl draws the fleet blueprint (how many clusters, sliced which way, and why), while Benny lays the rails that reconcile it: Cluster API turning clusters into cattle and GitOps fanning one source of truth out to the whole fleet. Between them, “many clusters” becomes “one platform.”

Why one cluster is never enough

☺ Like you’re 10: One giant treehouse is easy to run but risky — if it breaks, everyone falls. Splitting into a few gives each group its own safe space.

Multi-cluster is not a fashion; it’s a response to forces a single cluster cannot satisfy at once. Before you add a cluster you should be able to name which of these forces is driving you, because each one implies a different split, a different topology, and a different amount of ongoing tax. The five recurring drivers are blast-radius isolation, hard tenancy, data residency, latency & disaster recovery, and the raw scale ceiling of a single cluster. Most real fleets are the sum of two or three of these, layered.

Blast-radius isolation

A Kubernetes cluster is a shared-fate unit. It has one API server, one etcd quorum, one set of cluster-wide controllers, one CNI, one ingress path, and one version of everything at the control-plane layer. A single mistake at that layer — a botched upgrade, a runaway admission webhook that starts denying every request, a misconfigured CRD conversion that corrupts stored objects, an etcd disk filling up — degrades every workload on the cluster simultaneously. Splitting workloads across clusters turns one catastrophic outage into several independent, smaller ones. The prize is that the failure domain of “the whole platform” shrinks to “one cluster,” and you can roll changes cluster by cluster instead of all-at-once. This is the single most common honest reason to run more than one cluster, and it’s why reliability-minded teams treat a cluster as a blast-radius boundary, not just a bin of compute.

Hard tenancy: when a namespace isn’t a wall

Kubernetes gives you soft multi-tenancy inside a cluster: namespaces, RBAC, ResourceQuotas, NetworkPolicies, and Pod Security Standards separate teams reasonably well. But a namespace is not a hard security boundary. Tenants still share one API server, one kernel on each node, one set of CRDs, and one admission-control stack; a node-level container escape, a cluster-scoped CRD collision, or a greedy tenant exhausting the API server’s request budget crosses namespace lines. When your tenants are mutually distrusting — different customers in a SaaS, or workloads at genuinely different trust levels (PCI vs. general) — the defensible boundary is a separate cluster, not a namespace. The rule of thumb: soft tenancy for teams that trust each other, hard tenancy (separate clusters) for parties that don’t.

☺ Like you’re 10: Rooms in one house share the same locks and the same roof. If two families really don’t trust each other, you give them separate houses, not separate rooms.

Data residency, sovereignty & compliance

Regulation frequently dictates where bytes may physically sit and who may operate on them. GDPR, schemes like India’s DPDP, financial-sector data-localisation rules, and government “sovereign cloud” requirements can all mandate that EU users’ data stays in the EU, that a regulated workload runs in an isolated environment with a separate audit boundary, or that an operator in one jurisdiction cannot touch data in another. A single global cluster cannot honour “this data never leaves region X” — the control plane and its operators span regions by design. A per-region or per-jurisdiction cluster, with its own storage and its own access boundary, makes the compliance story provable rather than aspirational. Fleet engineering then becomes: keep the platform identical across jurisdictions while keeping the data firmly separated — a theme that runs straight into governance & compliance.

Latency, disaster recovery & the single-cluster ceiling

Two more forces are purely physical. Latency: users in Sydney talking to a cluster in Virginia pay ~200 ms round-trip no matter how good your code is; a regional cluster near them fixes it. Disaster recovery: a cluster (and often the availability zones under it) can fail together; surviving a regional outage means having a second region already running, not scrambling to build one. And finally the scale ceiling: a single cluster has real, published limits — on the order of 5,000 nodes, 150,000 pods, and 300,000 containers in upstream conformance guidance, but you’ll feel etcd pressure, API-server latency, and controller lag long before those numbers. Very large platforms shard into multiple clusters not for isolation but simply because one control plane cannot hold the whole footprint. Read scaling & scheduling for where those ceilings actually bite.

◆ Key idea

Never add a cluster “because multi-cluster is best practice.” Add one because you can name the force — blast radius, hard tenancy, residency, latency/DR, or scale ceiling — that a single cluster cannot satisfy. The driver you pick determines the topology you should choose next, and every cluster you add is a standing bill you’ll pay forever (the last section is entirely about that bill).

Fleet topologies — how to carve it up

☺ Like you’re 10: Once you need more than one treehouse, you have to decide how to split them — one per group of friends, one per neighbourhood, or a “main” treehouse that helps run all the others.

How you slice the fleet is an architecture decision with long consequences: it sets your blast radius, your cost, your operational surface, and how hard cross-cluster communication becomes. The common axes are per-team, per-environment, per-region, and — layered on top — hub-and-spoke and cell-based designs. Real platforms combine them: e.g. per-environment × per-region, managed from a hub, with each production region internally structured as cells.

Per-team, per-environment, per-region

Per-team clusters give each team a hard boundary and full autonomy, at the cost of multiplying clusters fast (and duplicating every add-on N times). They shine when teams are mutually distrusting or wildly different in shape; they hurt when you have fifty small teams and now fifty control planes to patch. Per-environment (dev / staging / prod, often further split prod from non-prod at the account level) is nearly universal and cheap in concept: it isolates the risky, fast-changing environments from the one that must stay up, and lets you give developers broad access in dev while locking prod down. Per-region is dictated by latency, DR, and residency; each region is a near-identical copy of the platform serving nearby users, with data kept local. These three axes are orthogonal, so a mature fleet often reads as {prod, non-prod} × {eu-west, us-east, ap-south} — six clusters before you’ve even split by team.

Hub-and-spoke: a management cluster

As the count grows, you need somewhere to run the machinery that runs the fleet. The hub-and-spoke pattern designates one (usually HA, usually not running business workloads) management cluster — the hub — that hosts the fleet control plane: Cluster API creating and upgrading the workload clusters, the GitOps controllers pushing config, the fleet-wide observability and policy backends, and the cluster inventory. The spokes are the workload clusters, each deliberately kept dumb and reproducible. The hub is powerful and therefore precious: it can create and destroy clusters and holds credentials to the whole fleet, so it becomes your highest-value target and must be hardened and, itself, recoverable from Git. Crucially, the hub should not be a runtime dependency of the spokes — if the hub is down, workloads keep serving; you just can’t make fleet-wide changes until it’s back.

Git fleet source of truth 🦫 Management cluster Cluster API · Argo CD policy · inventory · obs the “hub” prod · eu-west workload spoke prod · us-east workload spoke prod · ap-south workload spoke non-prod workload spoke reconcile create & configure

Cell-based architecture

At the top end, hyperscalers slice a region into cells: multiple identical, independent clusters (or cluster-plus-datastore units), each serving a fixed subset of tenants or traffic, with a thin routing layer in front that pins each customer to a cell. The point is bounded, predictable blast radius: a bad deploy or a poisoned tenant can only take down one cell — say 1/20th of traffic — and you can roll changes cell-by-cell as a natural canary. Cells also make scale linear: to grow, you add cells rather than making any one cluster bigger. The cost is real: you now run many clusters, a cell-routing layer, and per-cell capacity headroom, and you must keep tenants from spanning cells. Cells are overkill for most shops and indispensable for the few running enormous multi-tenant platforms — a classic case of matching the topology to the actual driver.

TopologyPrimary driverBlast radiusCost / ops surfaceBest when
Per-environmentIsolate risky change from prodOne environmentLow — a handful of clustersAlmost always; the baseline split
Per-teamHard tenancy / autonomyOne teamHigh — add-ons ×N teamsFew, distrusting, or very different teams
Per-regionLatency · DR · residencyOne regionMedium — a copy per regionGlobal users, or data-localisation law
Hub-and-spokeManage many clusters at onceHub can’t break running spokesMedium + a precious hubOnce cluster count > a handful
Cell-basedBounded, linear-scaling blast radiusOne cell (e.g. 1/20 of traffic)High — many cells + routingHuge multi-tenant platforms

Clusters as cattle — Cluster API

☺ Like you’re 10: Instead of building each treehouse by hand and remembering how you did it, you write the blueprint down once and a machine builds every treehouse the same way — and rebuilds any that break.

The moment clusters multiply, hand-building them (or worse, click-building them in a cloud console) stops working: every cluster is subtly different, upgrades are terrifying one-offs, and nobody can recreate a cluster from scratch. The fix is the same one that fixed servers a decade ago — treat clusters as cattle, not pets — and the tool that makes that real in the Kubernetes world is Cluster API (CAPI), a CNCF project that manages the lifecycle of clusters using Kubernetes’ own declarative machinery. It ties directly to the ideas in infrastructure as code & control planes: a cluster becomes just another declarative API object that a controller reconciles.

Management vs. workload clusters

CAPI runs a set of controllers on a management cluster (the hub from the previous section). You give the management cluster desired-state objects — a Cluster, a control plane, and one or more pools of machines — and its controllers call out to the infrastructure (AWS, Azure, GCP, vSphere, bare metal…) to create the real thing: a brand-new workload cluster that will actually run your apps. The management cluster never runs your business workloads; its job is to birth, upgrade, scale, and repair workload clusters. Because the desired state is ordinary Kubernetes objects, everything you already know applies: you can put those objects in Git, review them in a PR, and let a GitOps controller apply them — which is exactly how “create a new region” becomes a merge.

Providers, control planes & MachineDeployments

CAPI is deliberately layered into pluggable providers so the core stays cloud-agnostic. Three provider types compose to make a cluster: an infrastructure provider (e.g. CAPA for AWS, CAPZ for Azure) that creates VMs, networks and load balancers; a bootstrap provider (usually Kubeadm) that turns a raw machine into a Kubernetes node by generating cloud-init; and a control-plane provider (e.g. KubeadmControlPlane) that manages the API-server/etcd tier, including its HA and upgrades. Worker nodes are grouped into a MachineDeployment — the cluster-level analogue of a Deployment: it owns a set of Machines, and changing its replicas scales the pool while changing its version triggers a rolling replacement. That symmetry is the whole trick: the abstractions you use to run pods now run machines and clusters.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: prod-eu-west
  namespace: fleet
spec:
  clusterNetwork:
    pods: { cidrBlocks: ["192.168.0.0/16"] }
  controlPlaneRef:                         # the control-plane provider
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: prod-eu-west-cp
  infrastructureRef:                       # the infrastructure provider (AWS)
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSCluster
    name: prod-eu-west
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment                    # a worker pool = clusters’ “Deployment”
metadata:
  name: prod-eu-west-md-0
  namespace: fleet
spec:
  clusterName: prod-eu-west
  replicas: 6                              # scale the pool like any Deployment
  template:
    spec:
      clusterName: prod-eu-west
      version: v1.30.2                     # bump this → CAPI rolls new nodes
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: KubeadmConfigTemplate
          name: prod-eu-west-md-0
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
        kind: AWSMachineTemplate
        name: prod-eu-west-md-0

Declarative upgrades & immutable nodes

The reason CAPI is transformative for fleets is upgrades. To move a workload cluster from 1.30 to 1.31 you don’t SSH anywhere and run apt upgrade; you change the version field on the control plane and the MachineDeployment. CAPI then performs a rolling, immutable replacement: it provisions new nodes at the new version, waits for them to become Ready, cordons and drains the old ones, and deletes them — the same rollout choreography a Deployment uses for pods, but for entire machines. Nothing is upgraded in place; a node is never patched, only replaced, which eliminates configuration drift between nodes. Multiply that across a fleet and the payoff is enormous: upgrading fifty clusters becomes fifty reviewed, reconciled, rollback-able version bumps instead of fifty artisanal, sweaty maintenance windows.

⚠ The management cluster is a single point of leverage

The hub that runs Cluster API can create and destroy every cluster in the fleet and holds credentials to all of them — it is your most dangerous asset. Protect it accordingly: make it HA, restrict who can reach its API, and — because CAPI objects are just YAML — keep the fleet’s desired state in Git so the hub itself is reproducible. A subtle failure mode is the pivot problem: bootstrap CAPI on a temporary cluster, use it to build the real management cluster, then “pivot” the CAPI objects onto it. If you ever lose the hub without a backup or a Git source of truth, you can still reach the workload clusters, but you’ve lost the ability to manage the fleet declaratively.

Fleet configuration at scale

☺ Like you’re 10: You wrote one rulebook. Now you need a way to hand the same rulebook to every treehouse automatically — and give each one the small tweaks it needs — without walking to each one.

Cluster API births empty clusters; something else has to fill them with the platform — the ingress controller, cert-manager, the mesh, monitoring agents, policy, tenant namespaces — and keep them in step forever. This is GitOps stretched across a fleet: one Git source of truth, many reconcilers, and a way to say “this config goes to these clusters, with these per-cluster values.” Four tools dominate, and they trade off differently on where the reconciler runs and how you target clusters.

Argo CD ApplicationSets with the cluster generator

The most common fleet pattern is a single Argo CD on the hub with an ApplicationSet driven by the cluster generator. Every workload cluster is registered with Argo CD (CAPI can even auto-register it), carrying labels like region=eu-west or fleet=true. The cluster generator then produces one Application per matching cluster from a single template, and those label values flow into the template so each cluster can pull a region- or tier-specific overlay. Onboarding a brand-new cluster becomes automatic: register it, and the platform’s entire add-on stack is reconciled onto it without anyone writing a new manifest. Combine it with the matrix generator (clusters × apps) and one ApplicationSet can blanket a whole fleet.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-addons
  namespace: argocd
spec:
  generators:
    - clusters:                          # one App per cluster registered in Argo CD
        selector:
          matchLabels: { fleet: "true" }
  template:
    metadata:
      name: 'addons-{{name}}'            # {{name}}, {{server}}, {{metadata.labels...}}
    spec:
      project: default
      source:
        repoURL: https://github.com/acme/fleet-config.git
        targetRevision: main
        path: 'addons/overlays/{{metadata.labels.region}}'   # per-region overlay
      destination:
        server: '{{server}}'             # each cluster’s API endpoint
        namespace: platform-system
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [ CreateNamespace=true ]

Flux, Rancher Fleet & Sveltos

The other three take different stances. Flux favours a reconciler running in each cluster that pulls the same Git repo and applies a cluster-specific path or overlay; there’s no central controller to become a bottleneck, and each cluster is independent — attractive for large fleets and disconnected edge, at the cost of no single pane of glass. Rancher Fleet was designed from day one for fleets: a central manager pushes bundles to thousands of downstream agents, targeting clusters by label, and is Rancher’s answer to edge scale. Sveltos is a fleet add-on manager that shines at expressive targeting: you write a rule like “on every cluster labelled env=prod and running Kubernetes ≥ 1.29, deploy this Helm chart,” and it reconciles matches as clusters come and go — a lighter-weight, policy-driven complement that pairs well with Cluster API. There’s no single winner: the axis that matters is central push (Argo/Fleet — one place to see everything) vs. in-cluster pull (Flux — scales and survives partitions), and how rich the cluster-targeting language needs to be.

EngineWhere the reconciler runsTargeting modelSweet spot
Argo CD ApplicationSetsCentral (hub) pushCluster/Git/matrix generators + labelsOne pane of glass; small–large fleets
FluxIn each cluster (pull)Per-cluster path / Kustomize overlayLarge / disconnected / edge fleets
Rancher FleetCentral manager → agentsBundle + cluster-group labelsVery large fleets, Rancher shops, edge
SveltosCentral add-on controllerRich cluster selectors (labels, version…)Policy-driven add-ons alongside CAPI
🦆 Dot’s-eye view

“I don’t know or care how many clusters there are. I opened a PR that added my service to the prod overlay, and it showed up in eu-west, us-east, and ap-south — three continents — from one merge. The fleet is invisible to me, and that’s exactly how it should be. My mental model is still ‘change Git, it deploys.’ The platform team hid the multiplication.”

Multi-cluster networking & service discovery

☺ Like you’re 10: Each treehouse has its own phone book. If a kid in one treehouse wants to call someone in another, the treehouses need to share phone books and lay a wire between them.

Inside one cluster, a Service name resolves through CoreDNS and packets reach any pod on the flat network. Across clusters, neither is true by default: each cluster has its own DNS, its own pod CIDR, and no route to the others. Multi-cluster networking closes that gap along two independent seams — a data plane that makes cross-cluster pod-to-pod traffic actually route (and stay encrypted), and a discovery plane that lets a name in one cluster resolve to endpoints in another. On top sits global load balancing for user-facing failover.

Cluster mesh & flat cross-cluster networking

The data-plane job is to let a pod in cluster A dial a pod in cluster B as if they shared a network — securely, without exposing everything to the public internet. Cilium Cluster Mesh does this at the CNI/eBPF layer: it connects clusters’ agents so pod identities and endpoints are shared, and cross-cluster traffic is transparently encrypted (WireGuard/IPsec) and subject to network policy that spans clusters. Service meshes offer the alternative: Istio multi-cluster (and Linkerd multi-cluster) extend mTLS and traffic policy across clusters via east-west gateways, giving you identity, retries, and observability on cross-cluster calls. A hard prerequisite either way: non-overlapping pod/service CIDRs across clusters — if two clusters both use 10.244.0.0/16, cross-cluster routing is ambiguous and breaks. Plan the fleet’s address space up front.

# 1) Join two clusters into one mesh (Cilium Cluster Mesh, cilium CLI)
cilium clustermesh enable  --context eu-west
cilium clustermesh enable  --context us-east
cilium clustermesh connect --context eu-west --destination-context us-east
# each cluster needs a UNIQUE name + cluster-id and NON-overlapping CIDRs

# 2) Export a Service so other clusters can discover it (Multi-Cluster Services API)
cat <<'EOF' | kubectl --context eu-west apply -f -
apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceExport            # publishes "checkout" to the whole ClusterSet
metadata:
  name: checkout
  namespace: shop
EOF
# consumers in any member cluster now resolve:
#   checkout.shop.svc.clusterset.local  →  endpoints in ALL exporting clusters

The Multi-Cluster Services (MCS) API

Discovery is standardised by the Kubernetes Multi-Cluster Services API (SIG-Multicluster). You group clusters into a ClusterSet, then use two CRDs: a ServiceExport in the owning cluster marks a Service as shareable, and the implementation auto-creates a matching ServiceImport in the other members. Exported services become resolvable at a well-known domain — <svc>.<ns>.svc.clusterset.local — and, importantly, endpoints from all exporting clusters are merged, so the name naturally load-balances (and fails over) across clusters. MCS is the vendor-neutral contract; the actual plumbing is provided by an implementation (Cilium, Istio, Submariner, cloud MCS controllers). The value is that your app code targets a stable clusterset name and stops caring which cluster answers.

Global load balancing & failover

MCS handles east-west (service-to-service) traffic; north-south (users → the platform) needs a layer above all clusters. Global server load balancing (GSLB) — geo-aware DNS, anycast, or a global load balancer — sends each user to the nearest healthy region and, when a region fails its health check, steers traffic to another. This is what turns per-region clusters into genuine DR: the failover is a routing decision at the edge, not a human scramble. The subtlety is state: shifting stateless traffic is easy, but if a region owns data, failover has to account for replication lag and data locality — which is precisely where multi-cluster meets storage & state and residency rules again.

🌐 Global load balancer geo-DNS · nearest healthy region Cluster · eu-west checkout ServiceExport pods · 192.168.0.0/16 unique cluster-id Cluster · us-east checkout ServiceImport pods · 10.10.0.0/16 non-overlapping CIDR failover cluster mesh encrypted east-west resolve: checkout.shop.svc.clusterset.local → endpoints in both clusters

Consistent guardrails everywhere

☺ Like you’re 10: Every treehouse needs the same safety net, the same “who’s allowed in” rules, and the same alarm. If one treehouse skips the net, that’s the one where someone gets hurt.

A fleet’s greatest risk is drift in the guardrails: cluster #37 quietly missing a policy, running an old RBAC config, or not shipping logs. Attackers and outages find the weakest cluster. So the same three control planes you build for one cluster — policy, identity, and observability & secrets — must be reconciled onto every cluster identically, from the same source, and any cluster that falls out of compliance must be visibly flagged. This is where multi-cluster and governance fuse.

Fleet-wide policy-as-code

Admission policy (via OPA Gatekeeper or Kyverno, or the built-in Validating Admission Policy) is only as strong as its coverage. In a fleet you ship the exact same policy set to all clusters as just another add-on in your GitOps stream, so “no privileged pods,” “images only from our registry,” and “every namespace has a NetworkPolicy” hold everywhere by construction. Then you close the loop with fleet-level reporting: a hub-side controller that aggregates each cluster’s policy status so you can answer “which clusters are non-compliant right now?” at a glance. A policy that exists on 49 of 50 clusters is a policy you don’t really have — the 50th is where the breach happens.

Identity & RBAC across the fleet

Access has to be coherent across clusters or it becomes ungovernable. The pattern is central identity, federated to every cluster: wire each cluster’s API server to the same OIDC provider so a person’s group membership (say team-payments) maps to the same RBAC in every cluster, provisioned by the same GitOps stream rather than hand-edited per cluster. Workload identity federates the same way — SPIFFE/SPIRE or cloud workload identity gives services a verifiable identity that’s trusted fleet-wide, which is what makes cross-cluster mTLS meaningful. The anti-pattern to hunt down is per-cluster local admin accounts and hand-crafted kubeconfigs: they’re invisible, they drift, and they’re how a fleet accumulates snowflakes with god-mode.

Secrets & observability at fleet scale

Two more must be fleet-uniform. Secrets: the External Secrets Operator (or a per-cluster Vault agent) lets every cluster sync the secrets it needs from a central manager, so a rotated credential propagates fleet-wide without anyone copying values between clusters — the reference lives in Git, the plaintext never does. Observability: per-cluster silos defeat the purpose, so agents on every cluster ship metrics, logs, and traces to a central or federated backend (Thanos/Mimir/Cortex for metrics, Loki for logs), each stream labelled with its cluster so you can slice one cluster out or view the fleet as a whole. Without that, an incident spanning clusters is invisible; with it, the fleet has one nervous system. See observability for the golden-signals backbone this rides on.

⚠ The snowflake cluster

The failure mode of every fleet is the one cluster that’s almost like the others — created before you had CAPI, patched by hand during an incident and never reconciled back, missing a policy because someone “temporarily” disabled it. It passes casual inspection and fails at the worst moment. Defend against it structurally: every cluster comes from the same CAPI template and the same GitOps stream, self-heal is on, and a fleet dashboard flags any cluster that’s off-version or non-compliant. If you can’t recreate a cluster from Git, it’s already a snowflake.

Cross-cluster workload placement

☺ Like you’re 10: Sometimes you don’t want to pick which treehouse a kid goes to — you want a helper who looks at all of them and puts each kid where there’s room. That’s scheduling across the whole fleet.

Everything so far replicates config to clusters. A harder, more ambitious goal is to schedule workloads across a fleet: submit a Deployment once and have a fleet scheduler decide which clusters run it and with how many replicas — spreading for HA, packing for cost, or steering by region. This is genuinely hard (it fights CAP-theorem realities and adds a control plane above the clusters), and the ecosystem has iterated on it for years. Know the lineage, because it explains why the field looks the way it does.

The KubeFed history — and why it stalled

The first serious attempt, Kubernetes Federation (“KubeFed,” v1 then v2), tried to present a fleet as one big virtual cluster: you defined federated versions of resources with placement and override rules, and a host cluster propagated them. It taught the community a lot but never reached broad production adoption — the federated-type model was heavy and awkward, it duplicated API surface, and running a reliable control plane above clusters proved thorny. KubeFed is effectively archived. Its most important legacy is conceptual: the ideas of propagation (push this resource to these clusters) and override (but tweak it per cluster) survive in every modern tool, and much of the community concluded that for configuration, plain GitOps + ApplicationSets is simpler and good enough — reserving true multi-cluster scheduling for the cases that actually need it.

Karmada — propagation & overrides done right

Karmada (a CNCF project) is the mature successor. Crucially, it keeps the native Kubernetes API: you submit an ordinary Deployment to Karmada’s control plane, then attach a PropagationPolicy that says which clusters it targets and how replicas are scheduled — duplicated (full copy in each cluster) or divided (replicas split across clusters, optionally weighted by capacity), with spread constraints for HA and OverridePolicy for per-cluster tweaks (image registry, env). Because inputs are stock resources, existing tooling and CI still work. Karmada is the go-to when you genuinely need one logical workload scheduled across many clusters with real placement logic — global HA, capacity-aware bin-packing, or region steering — rather than the same manifest copied everywhere.

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: checkout
  namespace: shop
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: checkout           # a plain, unmodified Deployment
  placement:
    clusterAffinity:
      clusterNames: [eu-west, eu-central, us-east]
    spreadConstraints:
      - spreadByField: region  # ensure it lands in ≥ 2 regions (HA)
        maxGroups: 2
        minGroups: 2
    replicaScheduling:
      replicaSchedulingType: Divided          # split replicas across clusters
      replicaDivisionPreference: Weighted     # weight by cluster capacity

Admiralty & the lightweight approaches

Admiralty takes a different, elegant tack: instead of a heavy control plane, it uses a virtual-kubelet-style mechanism so a pod that can’t be scheduled locally is delegated to another cluster via a proxy pod — the fleet looks, to the scheduler, like extra capacity, and cross-cluster scheduling reuses the normal Kubernetes scheduler. It’s a natural fit for cloud-bursting and batch. At the other end of the spectrum, many teams deliberately avoid a fleet scheduler entirely: they use Argo CD ApplicationSets as “placement by config” (a generator decides which clusters get the app) and a global load balancer to steer live traffic — no new control plane, at the price of no automatic capacity-aware rebalancing. The honest default is to reach for real placement (Karmada/Admiralty) only when config-level replication plus GSLB can’t express what you need.

ApproachMechanismInputReach for it when
KubeFed (archived)Federated resource types + propagationFederated CRDsHistorical only — study, don’t adopt
KarmadaCentral plane + Propagation/OverridePolicyNative K8s resourcesTrue fleet scheduling, weighted spread
AdmiraltyVirtual-kubelet proxy podsNative podsCloud-bursting, batch overflow
ApplicationSets + GSLBConfig replication + edge routingGit manifestsMost cases — no extra control plane

The operational & cost tax

☺ Like you’re 10: Every extra treehouse looks free until you count the ladders, the safety nets, the paint, and the grown-up who has to inspect each one. Ten treehouses is ten times the chores.

Here is the truth the vendor slides skip: every cluster you add is a permanent bill, paid in money and — more painfully — in operational surface. The right instinct for a platform engineer is not “how do I run more clusters?” but “how do I add the fewest clusters that satisfy my real drivers?” This is where multi-cluster meets FinOps and anti-patterns, and where discipline beats enthusiasm.

Every cluster is real, recurring overhead

A cluster is never just its nodes. Managed control planes carry a per-cluster fee (roughly $70–75/month each on the big clouds — trivial at three clusters, a real line item at three hundred). Each cluster also needs its own baseline of system capacity that does no business work: control-plane nodes if self-managed, plus a per-node overhead for the kubelet, CNI, CSI, log/metric agents, and the mesh sidecars — commonly 10–20% of every node’s resources. Every cluster needs its own copy of the platform add-ons, its own certificates and their rotation, its own upgrade cadence, and its own place in your on-call surface. None of that ships a single feature; it’s the fixed cost of existing.

The N-clusters × M-add-ons multiplication

The tax isn’t linear in a comforting way — it multiplies. If your platform is 15 add-ons and you run 40 clusters, that’s 600 running instances to keep patched, compatible, and healthy; a CVE in the ingress controller is now 40 upgrades, and a breaking API change at a Kubernetes bump is 40 clusters × M add-ons of compatibility testing. This is exactly why the earlier machinery isn’t optional: without Cluster API making upgrades declarative and GitOps making add-ons uniform, an N×M fleet is unmanageable by hand. The reason to invest in fleet automation is that it converts a multiplying manual cost into a roughly fixed one — you maintain one template and one config stream regardless of N.

When NOT to add a cluster

Before adding a cluster, ask whether a cheaper boundary solves the actual driver. For team isolation, a well-configured namespace with quotas, NetworkPolicy, and RBAC is often plenty — reserve separate clusters for genuinely distrusting tenants. For stronger-than-namespace isolation without a full cluster’s tax, a virtual cluster (vCluster) gives a team its own API server and CRDs inside a host cluster — most of the isolation, a fraction of the overhead. Only reach for a real cluster when the driver truly demands it: hard security tenancy, a compliance/residency boundary, a distinct region for latency/DR, or a genuine scale ceiling. The discipline is to make “add a cluster” a deliberate, justified decision — not the reflex answer to every isolation request.

◆ Key idea

Escalate the boundary only as far as the driver requires: namespace → virtual cluster → separate cluster → separate region. Each step buys more isolation and costs more tax. Adding a cluster when a namespace would do is a classic over-engineering anti-pattern; refusing to add one when residency law demands it is negligence. The craft is matching the boundary to the real force — and then, whatever you choose, running it as part of the fleet so it never becomes a snowflake.

🦫 Benny’s workshop · 25 min

Simulate a fleet locally. Create three kind clusters (fleet-hub, fleet-eu, fleet-us) with deliberately non-overlapping pod CIDRs. On the hub, install Argo CD and register the other two as clusters, labelling them region=eu and region=us. Write one ApplicationSet with a cluster generator that deploys a tiny demo app to both, pulling a per-region overlay. Watch one merge land on two clusters. Now add a third cluster, register it, and confirm the app appears with no new manifest — that’s fleet onboarding. For a bonus round, enable a cluster mesh between eu and us and resolve a service across them by its clusterset.local name. Three tiny clusters, and the whole page becomes muscle memory.

🎬 At the Platform Guild
🦊

Foxy: We’ve got one cluster and it works great. Why not just make it bigger and put everything on it? One cluster is way less to babysit.

🦉

Professor Owl: Because one cluster is one blast radius. A bad upgrade, a runaway webhook, an etcd hiccup — and everything falls at once. And the regulator won’t let EU data sit in Virginia. Splitting isn’t optional; it’s the answer to forces one cluster can’t hold.

🦫

Benny: And I don’t hand-build the extras. Cluster API stamps each one from a blueprint, GitOps fans the same config to all of them, and upgrading is a version bump in a PR — cattle, not pets.

👺

Gizmo: Ugh, so much machinery. Just spin up a fresh cluster by hand whenever someone asks, tweak it live, and move on. Who’s counting? A cluster per person — freedom! 🤑

🐢

Timmy: That’s how cluster #37 becomes the snowflake with a disabled policy and no logs — the exact one that gets breached. Same template, same guardrails, same source of truth, or it doesn’t join the fleet.

🦆

Dot: Honestly I never see any of this. I opened one PR and my service showed up on three continents. If you’d given me a hand-built cluster to babysit, I’d have quit. Keep the fleet invisible.

🦉

Professor Owl: And add the fewest clusters that satisfy the real drivers. When a namespace or a vCluster will do, use it. Every cluster is a bill we pay forever.

Multi-cluster is where the whole course comes together: declarative infrastructure births the clusters, GitOps configures them, networking connects them, policy and governance keep them honest, and FinOps keeps the count sane. A fleet done well feels, to a developer, exactly like one big well-behaved cluster — which is the highest compliment fleet engineering can earn. Zoom back out with the platform architecture overview to see where the fleet sits in the bigger picture.

🐢 Timmy’s checkpoint

1. Name three distinct forces that justify a second cluster, and note how the driver changes the topology you’d pick. 2. In hub-and-spoke, what runs on the management cluster, and why must it not be a runtime dependency of the spokes? 3. In Cluster API, what does changing the version field on a MachineDeployment actually do to the nodes? 4. Which Argo CD ApplicationSet generator fans one config out to every registered cluster, and how do per-cluster values reach the template? 5. What two CRDs does the Multi-Cluster Services API use, and what clusterset name does an exported service resolve at? 6. When would you choose Karmada over “ApplicationSets + a global load balancer”? 7. Give two cheaper boundaries to try before adding a whole cluster.

Check your answers
  1. Any three of: blast-radius isolation (split → per-cluster/cell), hard tenancy (→ per-team/per-tenant clusters), data residency/compliance (→ per-region/per-jurisdiction), latency & DR (→ per-region), single-cluster scale ceiling (→ shard into more clusters). The driver dictates the slice.
  2. It runs the fleet control plane — Cluster API, the GitOps controllers, policy/observability backends, and the cluster inventory. It must not be a runtime dependency so that if the hub is down, workloads keep serving; you just can’t make fleet-wide changes until it recovers (and it should be reproducible from Git).
  3. It triggers a rolling, immutable node replacement: CAPI provisions new nodes at the new version, waits for Ready, then cordons/drains and deletes the old ones. Nodes are never patched in place, which prevents drift.
  4. The cluster generator. Each cluster is registered with labels; those labels (and name/server) are substituted into the template — e.g. {{metadata.labels.region}} selects a per-region overlay and {{server}} sets the destination.
  5. ServiceExport (marks a Service shareable in its owning cluster) and ServiceImport (auto-created in other members of the ClusterSet). It resolves at <svc>.<ns>.svc.clusterset.local, merging endpoints from all exporting clusters.
  6. Choose Karmada when you need one logical workload truly scheduled across clusters with real placement logic — divided/weighted replicas, capacity-aware spread, region steering — rather than the same manifest copied everywhere with traffic merely steered at the edge.
  7. A namespace (with quotas, NetworkPolicy, RBAC) for teams that trust each other, and a virtual cluster (vCluster) for stronger isolation (own API server & CRDs) without a full cluster’s tax. Reserve a real cluster for hard tenancy, residency, latency/DR, or a scale ceiling.