CERN & Platform Engineering at Extreme Scale

CERN is the odd file on this evidence board, and it's worth saying so before anything else: almost everything CERN's engineers have published in public — conference papers, KubeCon and OpenStack Summit talks, their own official Kubernetes case study — is about infrastructure at extraordinary scale, not about the developer-experience style of platform engineering this course otherwise teaches. There is no public talk about a Backstage-style portal for physicists, no published golden-path tutorial, no "time to first commit" metric. What is richly documented is how a small team runs hundreds of thousands of CPU cores, brought Kubernetes into a twenty-year-old grid-computing system built for a completely different kind of user, and had to solve for reproducibility across decades rather than sprints. That is still a platform-engineering story — just one about a very different platform, for a very different customer.

The starting situation: a twenty-year-old grid meets a wall of data

☺ Like you're 10: Physics experiments make an almost unimaginable pile of readings every second, and a huge, decades-old network of computers around the world has to store and crunch through all of it together.

The Worldwide LHC Computing Grid: a federation, not a single platform

The Worldwide LHC Computing Grid (WLCG) exists to store, distribute, and analyse the data produced by the Large Hadron Collider's four big experiments. Per WLCG's own public site, it combines roughly 1.4 million computer cores and 1.5 exabytes of storage contributed by more than 170 sites across 42 countries. Crucially — and this is the detail that shapes everything else in this case study — a KubeCon EU 2020 talk by CERN's Lukas Heinrich and the University of Manchester's Alessandra Forti, "Reimagining the Worldwide LHC Computing Grid on Kubernetes," described the WLCG candidly as it existed at the time: a federated, multi-cloud and multi-cluster deployment built up over roughly twenty years, made mostly of independent research-institute batch systems running on bare metal or VMs. Nobody designed the WLCG as one coherent platform from a blueprint. It grew as a federation of a hundred and seventy separately-operated sites agreeing to a shared set of standards — the polar opposite of "one platform team serves one company's developers."

Batch is king, and HTCondor is the workhorse

Inside CERN's own data centre, the workload shape is equally distinctive. CERN's official Kubernetes case study (published on kubernetes.io) states that batch workloads make up more than 80% of resource usage — a single physics analysis project alone has consumed 250,000 cores. This is about as far from a typical platform-engineering workload as it gets: no request/response latency budget, no user-facing SLA, no rolling deploy — just enormous numbers of independent, non-interactive jobs (simulate a particle collision, reconstruct an event, reprocess a dataset) that need to be scheduled, run to completion, and cleaned up. CERN's batch service runs on HTCondor as its scheduler; a CHEP 2019 conference paper on the service put its scale at more than 200,000 cores serving roughly 500 monthly unique users, provisioned at the time across some 20,000 virtual machines spread over more than 40 OpenStack projects.

Architecture & technology decisions: OpenStack first, Kubernetes for the batch layer

☺ Like you're 10: First they built one enormous, shared electricity grid for the whole science fair. Only later did they add a smarter, more automated way to plug new equipment into it.

An OpenStack private cloud sized in the hundreds of thousands of cores

Before Kubernetes entered the picture, CERN IT had already built one of the largest OpenStack deployments anywhere. The Cloud Infrastructure Service went into production in July 2013 with a few hundred compute nodes, according to a CERN paper on the ATLAS-on-Kubernetes work, and grew from there; the OpenStack Foundation's own "10 years of OpenStack" retrospective with CERN's Tim Bell puts the cloud at roughly 300,000 cores by its tenth anniversary, and CERN's official Kubernetes case study separately reports 10,000 hypervisors serving 4,300 projects and 3,300 users. To keep that scale operable, CERN split the deployment into dozens of independent Nova cells (an OpenStack Superuser article on CERN's cloud cites more than 280,000 cores split across 60 cells) so that a failure in one cell's control plane can't take down the rest of the cloud — the same "bound the blast radius" instinct this course covers in Multi-Cluster & Fleet Management, just applied a decade earlier and to plain compute cells instead of Kubernetes clusters.

That same Superuser piece describes a second, quieter shift that matters as much as any container technology: CERN moved from hand-provisioning virtual machines on request toward a self-service resource allocation model based on predefined quotas — a project gets a quota, and within it, requests a VM without waiting on a human. It's the same instinct behind self-service platforms everywhere: stop being the ticket queue, start being the API.

Bringing Kubernetes into the batch layer

A 2019/2020 CHEP conference paper, "Managing the CERN Batch System with Kubernetes," documents the next step: rather than keep provisioning batch capacity as hand-managed VMs, CERN prototyped running the HTCondor batch workers themselves inside Kubernetes. Half of a test capacity pool was provisioned as a bare-metal Kubernetes cluster built with OpenStack Magnum (OpenStack's "Kubernetes-as-a-service" component, which turns "give me a cluster" into an API call against the existing cloud); Helm then deployed the pieces needed to make that cluster behave like a batch worker pool — a Consul agent for service discovery and an HTCondor startd process running as a Kubernetes DaemonSet, so every node in the cluster automatically joined the HTCondor pool the moment it existed. In the shape the paper describes, provisioning a new batch worker stopped being "image a VM, register it with HTCondor, wait" and became "add a node to a Kubernetes cluster that already knows how to be an HTCondor worker":

# Illustrative shape of the pattern described in CERN's CHEP2020 paper —
# not a reproduction of CERN's actual manifests, which aren't public.
#
# 1. OpenStack Magnum turns "I need a Kubernetes cluster" into one API call
#    against capacity the cloud team already manages:
openstack coe cluster create batch-pool \
  --cluster-template k8s-baremetal-batch \
  --node-count 500

# 2. Helm installs the two pieces that turn plain nodes into HTCondor workers —
#    both running as a DaemonSet, so every node gets one automatically:
helm install condor-startd ./charts/htcondor-startd \
  --set collector.host=condor-collector.cern.ch
helm install consul-agent ./charts/consul-agent

# From here, "add batch capacity" is "add nodes to the cluster" —
# Kubernetes and the DaemonSet controller do the rest.

Federation across clouds, and reimagining the grid itself

CERN's Kubernetes adoption didn't stop at one cluster. Their official case study timeline records evaluation beginning in late 2015, a roughly six-month prototype phase, production deployment in October 2016, and Kubernetes federation reaching production in February 2018 — used to burst batch capacity out to public clouds (GKE, AKS, AWS were all named in a 2018 OpenStack Summit Vancouver talk, "CERN experiences with Multi Cloud, Federated Kubernetes") to absorb the workload spikes that hit before major physics conferences, when every analysis group wants results at once. Ricardo Rocha, a CERN software engineer quoted in the official case study, put the appeal of federation this way: work that took roughly a decade to stabilise using earlier distributed-computing approaches was demoed by an intern across CERN and public clouds together "in a couple of days" once Kubernetes gave everyone the same API.

By the KubeCon EU 2020 talk mentioned above, CERN and collaborators were going a step further: not just running batch workers on Kubernetes, but prototyping the entire WLCG federation model reimagined on Kubernetes — using containerd snapshotters to distribute container images efficiently to sites worldwide, and Helm as a GitOps mechanism for reproducible multi-cluster configuration across federation members. The talk's live demonstration deployed an actual WLCG federation member on Kubernetes that received and processed real ATLAS collision data — evidence the idea worked past the whiteboard, not just a slide.

Organisational & team-design choices: federation as an org model, not just an architecture

☺ Like you're 10: A tiny crew of caretakers looks after a mountain of machines using a lot of automation, and a hundred-and-seventy other schools each look after their own equipment — nobody is trying to run it all from one office.

A small central team, an enormous ratio of automation to headcount

The clearest organisational data point CERN has put in public is about team size, not org charts. An OpenStack Superuser spotlight on CERN's cloud describes the production support team inside CERN IT as around seven engineers (plus rotating students and fellows), and states plainly that this team scaled the cloud from roughly 30,000 to 300,000 cores without growing in size, crediting heavy automation for making that possible. Whatever the precise current headcount is today, the reported shape of the story is unambiguous: this is a platform run by a very small group whose real product is automation, not a large operations organisation whose product is hands-on-keyboard toil. That's the same bet the Team Topologies model makes about a well-run platform team generally — just tested at a scale where the alternative (headcount growing with core count) was never affordable in the first place.

The WLCG is 170 institutions, not one platform team

Zoom out from CERN's own data centre to the WLCG as a whole, and the organisational model inverts completely from anything else on this evidence board. Spotify, Mercedes-Benz, adidas, Zalando, Monzo and Netflix are each, however large, one company choosing to run one internal platform for its own developers. The WLCG is a federation of more than 170 independently-operated sites across 42 countries, each running its own hardware and reporting into a shared set of grid standards and interfaces. There is no single "platform team" whose job is developer experience for a hundred and seventy separate institutions' system administrators — there's a shared specification, and each site's own staff implement it against their own infrastructure. Kubernetes's role in the "reimagined WLCG" story is specifically to make that federation more uniform — a common API and a common deployment mechanism (Helm) across sites that would otherwise each run their own bespoke stack — without pretending it can turn a federation of institutions into a single organisation.

What's genuinely different about a scientific-computing platform

☺ Like you're 10: A normal company wants its app to work right now. A physics experiment wants an old measurement to be re-checkable ten years from now, exactly as it was the first time.

Reproducibility measured in decades, not sprints

Every platform in this course cares about reproducibility over the length of a deploy — roll back to yesterday's version. Particle physics needs something qualitatively harder: the ability to exactly re-run an analysis years or decades after the original run, against new data, using the original code, in the original computational environment. CERN's own REANA project (Reusable Analyses) — an open-source platform documented on CERN's HSF training pages and its own GitHub repository — exists precisely to solve this: analyses are expressed as declarative workflows (using engines like CWL, Snakemake, or Yadage) that run in containers on Kubernetes, HTCondor, or Slurm, so that a physicist can publish an analysis, share it with colleagues, and, in the platform's own stated goal, redo the exact same analysis with new data years later. RECAST builds on the same idea for a specific, high-value use case: reinterpreting an old, archived search for new physics under a brand-new theoretical model, without re-deriving the original analysis from scratch. Nothing in the rest of this course's AI/ML & data platforms lesson demands reproducibility on this timescale — even careful MLOps shops version data, code and environment to reproduce a model from last quarter, not from a hardware generation ago.

Batch-first, not request/response — and what that does to the toolbox

The other structural difference is workload shape. This course's Scaling, Scheduling & Performance lesson is built around the assumptions of long-lived, request/response services: HPA reacting to traffic, rolling updates that must never drop a live connection, PodDisruptionBudgets protecting availability. CERN's dominant workload — batch jobs run to completion, at more than 80% of resource usage — cares about almost none of that. It cares about throughput (how many jobs finish per hour), fair-share scheduling across many users and experiments (HTCondor's specialty), and squeezing utilisation out of every core, including opportunistic capacity borrowed when it's idle. That's precisely why CERN's Kubernetes story reaches for a DaemonSet running a batch scheduler's worker process rather than a Deployment running a stateless web service — the underlying primitive (reconcile toward a declared state, on every matching node) is the same one this course teaches, aimed at a workload the rest of the industry rarely has to schedule at this volume.

What changed: the metrics CERN has actually published

☺ Like you're 10: Some jobs that used to take hours now take minutes, and the same team looks after ten times more machines than before.

CERN's official Kubernetes case study is the source for the clearest before/after numbers on this evidence board, and this case study uses only those published figures — no estimates filled in around them.

Alongside those, the same case study reports the scale figures already used throughout this page — 330 petabytes stored at time of writing (with a roughly 10x increase expected as the accelerator complex is upgraded), 10,000 hypervisors, 320,000 cores, 4,300 projects and 3,300 users — plus the qualitative shift Ricardo Rocha described: Kubernetes gave CERN "a uniform API across heterogeneous resources," collapsing what had been bespoke, per-team distributed-systems engineering into something a new team member could reproduce in days.

What to steal for your own platform

☺ Like you're 10: You don't need a particle accelerator to borrow CERN's tricks — automation-per-engineer and honest data-lifetime thinking travel just fine to a normal company.

• Automation is what lets a small team own a huge estate — plan for it on purpose. CERN's ~7-engineer cloud team scaling 10x in core count without growing in headcount wasn't an accident; it was the explicit bet that automation, not more people, is how a platform scales. If your platform team's growth curve looks like your cluster count's growth curve, you haven't automated yet — you've just hidden the ticket queue behind a smaller mask.
• A DaemonSet is a more general tool than "run my sidecar." CERN's HTCondor-startd-as-DaemonSet pattern is a reminder that "run this one process on every matching node, and keep it running" is a primitive you can point at almost anything, not just log shippers and CNI agents. If you have a fleet-wide agent today provisioned by hand or by config management, ask whether a DaemonSet inside a cluster you already run would replace it.
• Separate "self-service for infrastructure operators" from "self-service for end users" honestly. CERN's quota-based OpenStack self-service is real and valuable, and it is explicitly a different, more modest claim than a Backstage-style, one-click, end-user developer portal. Know which one you're actually building, and don't let a true "we have self-service" claim about the former quietly get read as the latter.
• If your platform's output needs to be re-derivable years from now, design reproducibility as a first-class requirement, not an afterthought. Regulated industries (see Monzo and the regulated enterprise case files) already know this instinct from an audit angle; REANA/RECAST show the same discipline — versioned workflow, versioned code, pinned container environment — applied to scientific truth instead of compliance evidence. The mechanism this course teaches under reproducibility for ML is the same one; only the required time horizon changes.
• Bound your blast radius the way Nova cells do, at whatever scale you're actually at. Splitting a 300,000-core cloud into dozens of independently-failing Nova cells is the same idea as splitting a Kubernetes fleet into namespaces, clusters, or regions covered in Multi-Cluster & Fleet Management — pick the size of "one blast radius" deliberately, at any scale, instead of discovering it during an outage.

Honest caveats: what doesn't transfer, and where the record runs out

☺ Like you're 10: This story is a brilliant lesson about running gigantic infrastructure — it's a much thinner lesson about the "make life easy for one developer" side of platform engineering.

• This is not, publicly, a CNCF-style internal developer platform story. Say this plainly: nothing CERN has published describes a Backstage-equivalent catalog for physics analyses, a golden-path scaffolder for a new physicist's first analysis, or developer-experience metrics like time-to-first-deploy. The public talks are about infrastructure operations at scale — clusters, cores, federation, batch throughput — and this case study should not be read as proof that CERN has solved (or even attempted, in public) the same developer-experience problems Spotify or Netflix have.
• CERN's more recent, and separate, platform-engineering thread lives in a different part of the organisation. As of 2025, CERN's careers site has advertised "Platform Engineer" roles explicitly for its Open Science services — systems like INSPIRE, SCOAP3, and CERN Analysis Preservation — described in terms familiar from the rest of this course: documentation, automation, self-service, observability, modern tooling (Kubernetes, PostgreSQL, OpenSearch). That's a genuinely CNCF-flavoured platform-engineering effort at CERN — but it's for a research-library and open-science software estate, not the LHC's physics batch grid this case study is mostly built from, and it isn't (yet) documented in a public talk or case study the way the OpenStack/Kubernetes infrastructure story is. Don't blur the two.
• Ratios like "seven engineers per 300,000 cores" describe an outcome, not a recipe you can copy directly. That ratio reflects years of specific automation investment, a workload that's unusually uniform (mostly batch jobs against a shared scheduler) compared to a typical company's zoo of different services, and a mission-funded organisation without commercial deadline pressure. A small team running a small platform badly and a small team running a huge platform brilliantly can look identical on an org chart — the automation is the entire difference, and CERN's specific automation isn't public in enough detail to copy line for line.
• CERN's incentives and constraints are not a typical company's. CERN is a publicly-funded, mission-driven research organisation without shareholders, a sales quota, or a competitor about to ship a rival product. Decisions that make sense there — open-sourcing REANA, spending years perfecting reproducibility for its own sake, running infrastructure with the patience of an institution built to outlast any one experiment's timeline — don't automatically make business sense at a company answering to a quarterly roadmap. Borrow the engineering ideas; don't assume the organisational patience travels with them.
• Some of the sharpest figures in this case are a few years old. The before/after metrics table above, the 2018 federation date, and the ~7-engineer team figure all come from sources published between roughly 2016 and 2020. CERN has not published an equivalent, fully updated scorecard since; treat this case as a well-documented snapshot of how the migration went, not a live status report.

Where this connects in the course

☺ Like you're 10: This story mostly plugs into "how big platforms are built," not "how developers are made happy" — read it alongside the lessons that share that focus.

Read this case file next to Platform Architecture & Infrastructure, whose compute, multi-tenancy and scale concerns are exactly what CERN's OpenStack and Kubernetes layers are built to answer, and Scaling, Scheduling & Performance, whose scheduler-and-autoscaler machinery underpins the batch-first workload shape this case study leans on hardest. The reproducibility thread connects straight to Platforms for AI/ML & Data, which teaches the same versioned-data-plus-versioned-code-plus-pinned-environment discipline at a commercial timescale rather than a decades-long one. Multi-Cluster & Fleet Management covers the blast-radius thinking behind Nova cells and cluster federation in more general terms, and The Cloud Native Landscape & Choosing Tools is the right lens for judging CNCF-adjacent, research-grown projects like REANA the way you'd judge any tool before adopting it. For the rest of the evidence board — including the platforms whose stories run closer to typical developer experience — return to the case studies hub.