A Media Company's Multi-Cluster Migration

This is a composite, illustrative case study, not a real, identifiable company. "Northlight Media" is a stand-in assembled from patterns that repeat across real streaming and broadcast platforms as they outgrow a single Kubernetes cluster — no single company is Northlight, but the shape of the story is real. Northlight ran one cluster for two years: live-event video transcoding, an editorial CMS, and ad-serving, all sharing one control plane and one pool of nodes. That worked until a live match's ingest pipeline starved the CMS during breaking news, a routine upgrade took ad-serving down for forty minutes, an EU licensing deal demanded viewer data never leave the region, and three platform teams started fighting over the same namespaces. This page is the story of what Northlight built in response — a small regional fleet stood up with Cluster API instead of hand-rolled scripts, kept in sync with a fleet-wide GitOps rollout — and the three specific things that broke on the way there.

Northlight Media: what "the cluster" used to mean

☺ Like you're 10: One kitchen, one set of burners, three completely different meals cooking on it at once.

For its first two years on Kubernetes, "the cluster" at Northlight meant exactly one thing: a self-managed cluster, hand-built with kubeadm and a Terraform module nobody had touched since the person who wrote it left, running on AWS in a single region. It carried three genuinely different workloads on the same control plane and the same node pools: a live-event ingest and transcoding pipeline that turned incoming video feeds into the adaptive-bitrate streams viewers actually pulled, an editorial CMS that publishing staff used to write and schedule articles and metadata, and an ad-decision service that picked which ad to serve a given viewer in a given territory. Nothing about that was unreasonable at the time — one cluster is simpler to operate, cheaper to run, and easier to reason about than a fleet, and Anti-Patterns & Pitfalls is right that reaching for extra clusters before you need them is its own failure mode. Northlight's problem wasn't that one cluster was wrong on day one. It was that nobody had named the point at which it would stop being enough.

The triggers: three forces, one bad month

☺ Like you're 10: Not one big disaster — four separate annoyances that all pointed at the same fix.

Multi-Cluster & Fleet Management puts it plainly: never split because "multi-cluster is best practice" — split because you can name the specific force one cluster can't satisfy. Northlight could eventually name four, and they arrived close enough together that the platform team stopped treating them as separate incidents and started treating them as one signal.

Choosing the topology: named forces, not org chart

☺ Like you're 10: Split by the actual reason you need more kitchens — not by drawing one kitchen per cook.

With four named forces on the table, the platform team resisted the two easiest wrong answers: a cluster per team (which would have meant five-plus clusters solving an org-chart problem, not a technical one, and multiplying the fleet's add-on tax for no isolation benefit) and a single cluster with ever-more-elaborate namespace tooling bolted on (which does nothing for shared-fate upgrades or data residency, no matter how sophisticated the tooling gets). The topology that actually matched the four forces was region × environment: one production cluster per region that needs its own residency or latency boundary, plus a shared non-production cluster per major region for staging and load testing. Three teams could keep sharing a region's production cluster, separated by namespace, RBAC, and NetworkPolicy — soft isolation was genuinely sufficient there, because the three teams trusted each other and none of the four triggers pointed at hard tenancy between them.

Cluster API: turning "build a cluster" into a pull request

☺ Like you're 10: Instead of one person hand-mixing the concrete for every new kitchen, you write the blueprint once and a machine pours every foundation exactly the same way.

The original cluster had been built by hand, and rebuilding that process twice more — once for eu-west, once for non-prod — was exactly how a fleet accumulates snowflakes before it's even finished being built. Northlight adopted Cluster API for lifecycle from the start of the migration: a small management cluster running the CAPI controllers, the AWS infrastructure provider (CAPA) for the actual VMs and load balancers, and KubeadmControlPlane for the control-plane tier. Every new cluster became a reviewed pull request against a shared ClusterClass template instead of a person following a runbook from memory.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: prod-eu-west
  namespace: fleet
spec:
  clusterNetwork:
    pods: { cidrBlocks: ["10.32.0.0/16"] }     # explicit, unique per cluster — see "what broke" below
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: prod-eu-west-cp
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSCluster
    name: prod-eu-west
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: prod-eu-west-transcode-pool
  namespace: fleet
spec:
  clusterName: prod-eu-west
  replicas: 12
  template:
    spec:
      clusterName: prod-eu-west
      version: v1.30.2
      bootstrap:
        configRef: { apiVersion: bootstrap.cluster.x-k8s.io/v1beta1, kind: KubeadmConfigTemplate, name: prod-eu-west-transcode-pool }
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
        kind: AWSMachineTemplate
        name: prod-eu-west-transcode-pool

This page doesn't re-derive how CAPI works — Multi-Cluster & Fleet Management covers the provider split and the MachineDeployment/Deployment symmetry in full, and Platform Engineering's Cluster API tool guide goes deeper still on clusterctl and managed topologies. What matters for this story is what changed operationally: standing up prod-eu-west went from an estimated four days of a senior engineer's undivided attention to a same-day pull request that a controller reconciled unattended.

GitOps rollout: the same addons, every region, one ApplicationSet

☺ Like you're 10: A machine-poured foundation is still an empty room — something has to stock every kitchen with the exact same pans and spice rack, automatically.

Cluster API gets a cluster to "control plane up, nodes joined" and stops there. Filling it with ingress, cert-manager, metrics-server, the Kyverno policy bundle, and Northlight's own transcode-queue operator — and keeping all of that identical across three clusters forever — is GitOps stretched across a fleet. Northlight ran a single Argo CD instance on the management cluster with an ApplicationSet driven by the cluster generator, so registering a cluster with the right labels was the only step needed for it to inherit the entire addon stack:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-addons
  namespace: argocd
spec:
  generators:
    - clusters: { selector: { matchLabels: { fleet: "true" } } }
  template:
    metadata: { name: 'addons-{{name}}' }
    spec:
      source:
        repoURL: https://git.northlight.example/fleet-config.git
        path: 'addons/overlays/{{metadata.labels.region}}'
      destination: { server: '{{server}}', namespace: platform-system }
      syncPolicy: { automated: { prune: true, selfHeal: true } }

This is the same shape the deep-dive page walks through generically — the specific payoff for Northlight was that admission policy stopped being three separate Kyverno installs that could quietly drift apart. "No unsigned images in the transcode namespace" became one rule, shipped as one addon, present on every cluster the moment it registered, whether that cluster existed when the rule was written or not.

What broke: three incidents from the transition

☺ Like you're 10: Building three kitchens from one blueprint doesn't mean nothing ever burns — it means each fire teaches you something the blueprint didn't cover yet.

None of the mechanisms above are new — Cluster API and fleet-wide GitOps are both covered mechanically elsewhere in this course. What a case file can show that a mechanism page can't is what actually goes wrong the first time a team runs them for real. Three incidents did the most to shape Northlight's eventual playbook.

Incident 1: the CIDR collision that mis-routed ads

The prod-eu-west ClusterClass was cloned from an early draft that hadn't yet had its pod CIDR block parameterized, and it defaulted to 10.244.0.0/16 — the exact range still in use by an old hand-built non-prod-us-east-2 cluster that hadn't been decommissioned yet. Nobody noticed at creation time, because within a single cluster an unused CIDR default causes no symptom at all. The problem only surfaced once the ad-decision service was exported through the Multi-Cluster Services API so prod-us-east and prod-eu-west could share ad-inventory lookups: cross-cluster requests routed through the fleet's transit gateway started landing on the wrong cluster's Pods whenever both clusters happened to have a Pod at the same overlapping address. The first visible symptom wasn't a Kubernetes error at all — it was European viewers occasionally seeing US-only ad creative, caught three hours later when ad-revenue reconciliation flagged mismatched territory codes.

# What actually found it: comparing pod CIDR blocks across every Cluster object in the fleet
$ kubectl get clusters.cluster.x-k8s.io -A \
    -o jsonpath='{range .items[*]}{.metadata.name}{"  "}{.spec.clusterNetwork.pods.cidrBlocks}{"\n"}{end}'
prod-us-east         [10.244.0.0/16]
prod-eu-west         [10.244.0.0/16]   # <- collision with the still-running non-prod-us-east-2 cluster
non-prod-us-east-2   [10.244.0.0/16]

Incident 2: GitOps and the HPA, fighting over the same field

The day fleet-wide selfHeal: true went live, the transcoding Deployment's manifest in Git pinned spec.replicas: 12 as a sane baseline. It had an HPA attached that legitimately scaled that same field between 8 and 40 based on queue depth. Every few minutes, Argo CD would notice live replicas didn't match Git's 12 and "self-heal" it back down; the HPA would immediately scale it back up in response to the still-high queue depth it was watching. During a live event, that fight manifested as capacity flapping every three to five minutes — exactly when the transcoding pipeline could least afford it — and a visible backlog of dropped transcode jobs during the flap windows.

The fix is a documented pattern GitOps on Kubernetes describes generically as field-ownership conflict: tell the GitOps controller to stop asserting an opinion about a field another controller legitimately owns.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: addons-prod-us-east
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: transcode-worker
      jsonPointers:
        - /spec/replicas   # the HPA owns this field; Argo CD must stop fighting it for

Northlight's platform lead added a rule to the fleet's PR template after this one: any Deployment carrying an HPA must ship its ignoreDifferences entry in the same pull request, not as a follow-up once someone notices the flapping.

Incident 3: a MachineHealthCheck loop that made a capacity crunch worse

During the same live event, the transcoding pool's autoscaler tried to add worker capacity in a specific availability zone that had, unluckily, hit an AWS capacity limit that hour. New Machines were provisioned but took longer than the pool's configured MachineHealthCheck nodeStartupTimeout to reach Ready. CAPI's health-check controller did exactly what it was configured to do: it marked each slow Machine unhealthy and remediated it — deleted it and asked the infrastructure provider for a replacement, in the same constrained zone, which then took just as long and got marked unhealthy in turn. The loop burned through cloud spend replacing Machines that were never actually broken, and briefly made the shortage worse instead of better, because none of the churned Machines ever lived long enough to join the pool and take load.

An engineer broke the loop by hand — pausing the MachineHealthCheck, letting the in-flight Machines finish booting on their own, and only then re-enabling remediation. The lasting fix was two changes: a longer nodeStartupTimeout that reflected the pool's real worst-case boot time under load rather than its best-case, and a MachineDeployment spread across two availability zones instead of one, so a single zone's capacity limit no longer had the power to trigger a fleet-wide remediation storm.

What changed

☺ Like you're 10: The kitchens stopped catching each other's fires, and stocking a brand-new one stopped taking a week.

Because Northlight is composite, there is no specific published metric to cite — treat the following as the shape of outcome that recurs across migrations like this one, not a verified number.

• Blast radius actually shrank. A CMS incident in prod-us-east no longer has any path to touching ad-serving or transcoding in prod-eu-west — they don't share a control plane, an etcd quorum, or a node.
• EU residency became provable, not asserted. European viewer data and stream logs live only on prod-eu-west infrastructure, and that boundary is enforced by cluster topology rather than a policy someone has to remember to check.
• New-cluster time collapsed. What used to be days of one engineer's undivided, error-prone attention became a same-day, reviewed pull request against a shared ClusterClass template.
• Drift stopped being invisible. Every cluster in the fleet inherits the same addon stack and Kyverno policy bundle from the same ApplicationSet the moment it registers — no more discovering months later that one cluster quietly never got cert-manager installed.

What to steal for your own migration

☺ Like you're 10: Even if you don't run a streaming service, the same handful of tricks make any fleet migration less painful.

• Parameterize CIDR allocation before your second cluster exists, and enforce it in CI. A default value in a shared template is exactly how a collision ships unnoticed — make an overlap a failed check, not a code-review hope.
• Ship ignoreDifferences in the same pull request as any Deployment carrying an HPA, VPA, or other field-owning controller. Turning on fleet-wide selfHeal without it is a guaranteed fight, and it will surface at the worst possible moment, not a quiet one.
• Set MachineHealthCheck timeouts to your real worst-case boot time, and spread pools across zones. A remediation loop that keeps retrying the same failing placement doesn't fix a capacity crunch — it accelerates it.
• Split by named force, not by team or by fear. Northlight's fleet stayed at three clusters because that's how many distinct forces it could actually name; a cluster added "just in case" is pure standing cost with no isolation payoff.
• Track every hand-built cluster still running during a migration on an explicit retirement list. The CIDR collision existed only because a "temporary" old cluster was still alive and unaccounted for months after the migration notionally started.

Honest caveats — what doesn't transfer

☺ Like you're 10: This story worked for one kitchen chain with cooks willing to change how they worked. Not every chain will be.

• No number in this case file is a verified, published metric. Treat "what changed" as a pattern this kind of migration tends to produce, not a citation — a real vendor talk quoting a specific percentage belongs to that organization's context, not a guarantee of yours.
• This depends on genuine platform-team capacity to adopt CAPI and GitOps properly. A team without the headcount to build and maintain the ClusterClass template, the CI CIDR check, and the ApplicationSet correctly will find the fleet drifting anyway — the tooling doesn't run itself.
• Multi-cluster is a permanent tax, not a one-time decision. Three clusters means three times the add-on footprint and three times the surface a fleet dashboard has to watch, forever. A smaller media operation with no residency requirement and no history of shared-fate incidents may be entirely correctly served by one well-isolated cluster for years — see Anti-Patterns & Pitfalls on reaching for a cluster before you need one.
• Data-residency requirements vary by jurisdiction and by contract. Northlight's EU reading of its licensing terms is illustrative, not legal guidance — verify what your own regulatory or contractual obligations actually require before assuming a second cluster in a second region automatically satisfies them.
• A legacy estate migrates unevenly. Northlight's old non-prod-us-east-2 cluster was the slowest thing to retire precisely because nobody owned that decision — expect your own least-glamorous cluster to be the last one gone, and put someone's name on retiring it rather than assuming it will happen on its own.

For the mechanisms this case file assumes without re-deriving, see Multi-Cluster & Fleet Management and GitOps on Kubernetes; for the fleet-scale cost tax and topology decision tree in full, see Platform Engineering's Multi-Cluster & Fleet Management.