Platforms for AI/ML & Data

Machine learning is the workload that breaks every comfortable assumption a platform makes. A normal service is small, stateless, cheap to run, and identical from one replica to the next. A training job is enormous, stateful, needs a dozen machines to move in lockstep, and rides on accelerators that cost more per hour than a developer’s laptop costs to buy. The data it feeds on is measured in terabytes; the “build” is a multi-hour experiment; the artifact isn’t a container but a set of learned weights whose behaviour drifts as the world changes. This page goes past the exam blueprint into the vast domain of platforms for AI/ML and data: why this work needs a platform at all, the ML lifecycle on Kubernetes, how GPUs are scheduled and shared, the training and serving stacks, the new LLMOps/GenAI layer, the data plumbing underneath it, and the platform engineer’s job of paving golden paths for it all while keeping the accelerator bill from swallowing the company.

Why AI/ML needs a platform

☺ Like you’re 10: A science project that only works once, on one kid’s desk, isn’t really finished. ML has that problem times a thousand — so we build a shared lab where every experiment can be repeated, and the fancy equipment gets shared instead of hoarded.

It is tempting to think of ML as “just another app that happens to use a GPU.” It isn’t, and the differences are exactly the things a platform exists to tame. Three properties — irreproducibility, scarce and costly accelerators, and a yawning gap between the notebook and production — turn ad-hoc ML into a swamp, and turn a good platform into a competitive advantage. The discipline that spans them has a name, MLOps: applying the same automation, versioning and reconciliation ideas from the rest of this course to the messy, data-driven world of models.

The reproducibility problem

A trained model is a function of three things: the code, the data, and the environment (framework versions, CUDA drivers, random seeds, hyperparameters). Change any one silently and you get a different model — sometimes subtly worse, sometimes catastrophically. In classic software, we pinned the first and third long ago (Git for code, containers for environment). ML adds the second, and data is the hardest to pin: it’s huge, it lives in object stores and warehouses, and it changes underneath you. If you can’t say exactly which snapshot of the data, which commit of the training code, and which container image produced model-v7, then you can’t reproduce it, can’t debug it, and can’t prove to an auditor how it decides. A platform’s first job is to make every training run reproducible by construction — versioned data, versioned code, a pinned image, recorded hyperparameters — so “re-run it” is a button, not an archaeology project.

Scarce, expensive accelerators

GPUs (and TPUs, and other accelerators) are the defining constraint. They are expensive — a single high-end training GPU can cost several dollars an hour in the cloud, and clusters of them run to hundreds of dollars an hour — and they are scarce: cloud capacity for the newest chips is frequently sold out, and on-prem you have exactly as many as you bought. Unlike CPU, a GPU is not overcommittable by default and, out of the box, is claimed whole by one container even if that container uses 5% of it. The economic consequence is brutal: idle GPUs are the single largest source of waste in most ML platforms, and a fleet of half-used cards burns money at a rate that dwarfs the rest of the cluster combined. Everything later on this page — sharing, gang scheduling, scale-to-zero serving, spot capacity — exists to push GPU utilization up and the bill down. This is where the platform earns its keep, and where FinOps stops being a nice-to-have and becomes survival.

The notebook-to-production gap

Most ML is born in a Jupyter notebook: a data scientist explores, plots, and trains interactively on a laptop or a shared VM. Notebooks are wonderful for discovery and terrible for production — they hide global state, run cells out of order, pin nothing, and can’t be reviewed, tested, or scheduled like real code. The chasm between “it works in my notebook” and “it serves a million requests a day, retrains weekly, and pages someone when accuracy drops” is where most ML projects die. Bridging it is the core MLOps job: promote the notebook’s logic into versioned pipeline steps, run them on the cluster instead of a laptop, register the resulting model as a first-class artifact, and serve it behind an autoscaling endpoint with monitoring. A platform makes that promotion a golden path rather than a bespoke rewrite each time.

The ML lifecycle on Kubernetes

☺ Like you’re 10: Making a model is an assembly line — clean the ingredients, cook, label the jar, put it on the shelf, then taste-test it forever. And if it starts tasting off, you go back and cook a fresh batch.

ML on a platform is best understood as a lifecycle, not an event: data preparation, training, registration, serving, and monitoring, with monitoring feeding back into the next round of training. Kubernetes is a natural home for it because every stage is a containerised workload, and the loop is exactly the kind of declarative, reconciled process the rest of this platform already runs. The goal is to make each stage a reusable, tracked, self-service step so that shipping a model looks like flowing work down a pipeline rather than assembling a one-off by hand.

Pipelines as DAGs

The backbone of the lifecycle is the pipeline: a directed acyclic graph (DAG) of steps — fetch data, validate it, engineer features, train, evaluate, register — where each step is a container and the edges are data dependencies. Two engines dominate on Kubernetes. Kubeflow Pipelines (KFP) lets data scientists define the DAG in Python; it compiles that into Argo Workflows under the hood, so every step becomes a pod, artifacts flow between steps through object storage, and the whole run is tracked with lineage metadata. Argo Workflows itself is the lower-level, general-purpose workflow engine many teams use directly. The win over “a big training script” is that a DAG is cacheable (skip steps whose inputs didn’t change), parallelisable (fan out across data shards or hyperparameters), and reproducible (each step pins its image and inputs). Turning a notebook into a pipeline is the single most important promotion in MLOps.

Experiment tracking & metadata

Training is empirical: you run the same code dozens of times with different data, features, and hyperparameters, and you need to know which run won and why. That’s experiment tracking — tools like MLflow Tracking, Weights & Biases, or Kubeflow’s own metadata store record, for every run, the parameters, the code version, the dataset version, and the resulting metrics (accuracy, loss, F1). Without it, a team is flying blind, unable to compare runs or explain a regression. This is squarely Ellie’s territory: the platform should make tracking automatic — every pipeline run logs its lineage without the scientist wiring it up — so that months later anyone can answer “which data and settings produced the model we shipped in March?” Lineage is not bureaucracy; it is the difference between an auditable system and a liability.

The model registry

A trained model needs a home that is more than a file in a bucket. A model registry (MLflow Model Registry, the Kubeflow Model Registry, or a vendor equivalent) is the promotion gate between training and serving: it versions each model, attaches its metrics and lineage, and moves it through stages — None → Staging → Production → Archived. This is the exact analogue of a container registry for ML: serving reads from the registry, never straight from a training job, so “what’s in production” is always a named, versioned, promotable thing. It’s also the natural place to enforce policy — require an evaluation to pass before a model can reach Production, or demand a sign-off — which ties model promotion into the same policy-as-code guardrails the rest of the platform uses.

GPU & accelerator scheduling

☺ Like you’re 10: The magic crayons don’t plug themselves in. Kubernetes needs a special helper to even notice the crayons exist, and then some clever tricks to let more than one kid share a single box safely.

Kubernetes has no native idea of a GPU — to the scheduler, a node offers CPU and memory and nothing else. Making accelerators first-class citizens takes a specific stack, and sharing them without waste takes several more mechanisms. This is the most platform-specific, highest-leverage part of an ML platform: get GPU scheduling right and utilization climbs while the bill falls; get it wrong and you pay full price for cards that sit 90% idle.

The device plugin & GPU Operator

GPUs enter the cluster through the device plugin framework. The NVIDIA device plugin runs as a DaemonSet on GPU nodes, discovers the cards, and advertises them to the kubelet as an extended resource called nvidia.com/gpu. A pod then requests whole GPUs the same way it requests CPU — except a GPU is integer-only and non-overcommittable, so its request must equal its limit. Getting there by hand is painful: you need the right NVIDIA driver on the host, the container toolkit, the device plugin, monitoring, and (for MIG) a partition manager, all version-matched. The NVIDIA GPU Operator automates the whole stack as a single operator — it installs and manages the driver, the container runtime hooks, the device plugin, the DCGM exporter (which feeds GPU metrics to your observability stack), node feature discovery, and the MIG manager — so a node labelled “has GPUs” becomes fully ready without a human touching it. For a platform, the GPU Operator is table stakes.

apiVersion: v1
kind: Pod
metadata:
  name: train-resnet
spec:
  # Land only on the dedicated GPU pool, and tolerate its taint so batch
  # jobs can't squat on the expensive silicon (see Scaling & Scheduling).
  nodeSelector:
    nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB
  tolerations:
    - { key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }
  containers:
    - name: trainer
      image: registry.acme.io/ml/trainer:2.4.0
      resources:
        limits:
          nvidia.com/gpu: 2        # whole GPUs — request MUST equal limit (integer, no overcommit)

Sharing a GPU — MIG, time-slicing & MPS

Claiming a whole GPU for a job that uses a fraction of it is the default, and the default is wasteful. Three sharing mechanisms let multiple workloads share one physical card, and choosing between them is a real trade-off between isolation and flexibility.

MIG (Multi-Instance GPU), available on data-center cards like the A100 and H100, partitions one physical GPU into up to seven fully isolated instances in hardware — each with its own slice of compute (streaming multiprocessors) and its own dedicated memory. Because the isolation is physical, one tenant can’t starve or crash another, and each MIG instance is advertised as its own schedulable resource (for example nvidia.com/mig-1g.10gb). MIG is the right answer for multi-tenant inference and for giving many small jobs guaranteed, isolated slices.

Time-slicing is the opposite philosophy: the device plugin simply advertises the GPU as several replicas and lets pods take turns on it, context-switching in software. There is no memory isolation and no fault isolation — two pods that time-share a card can exhaust each other’s memory or drag each other down — but it works on any GPU, needs no special hardware, and is perfect for development, notebooks, and bursty low-stakes inference where oversubscription is fine. MPS (Multi-Process Service) sits in between: it runs GPU work from multiple processes concurrently with some spatial sharing and modest isolation. The platform rule of thumb: MIG for isolated multi-tenant production, time-slicing for cheap dev/test, MPS when you need concurrency without MIG-capable hardware.

# GPU Operator time-slicing config: advertise each physical GPU as 4 shares,
# so four pods can request one "nvidia.com/gpu" and oversubscribe the card.
apiVersion: v1
kind: ConfigMap
metadata: { name: time-slicing-config, namespace: gpu-operator }
data:
  any: |-
    version: v1
    sharing:
      timeSlicing:
        resources:
          - name: nvidia.com/gpu
            replicas: 4        # 1 physical GPU → 4 schedulable shares (NO memory isolation)

Scarcity, cost & keeping GPUs busy

Even with sharing, GPUs remain the scarce, costly resource the whole platform strains to keep fed. Three habits matter. First, dedicated, tainted GPU node pools (as in the earlier manifest) stop non-GPU work from occupying GPU nodes and blocking real jobs — the taint/toleration pattern from Scaling & Scheduling is exactly the tool. Second, measure real utilization, not allocation: the DCGM exporter reveals the uncomfortable truth that a GPU can be “allocated” (a pod holds it) while its actual SM occupancy is near zero — allocation is not usage, and the gap is money. Third, reclaim idle accelerators: pair GPU pools with Karpenter or the cluster autoscaler so idle GPU nodes scale to zero, use spot/preemptible GPUs for interruption-tolerant training, and scale inference endpoints to zero when no one is calling. GPU cost is not a footnote; it is frequently the largest line item, and driving utilization is the platform’s single biggest financial lever — the through-line into FinOps.

Training platforms

☺ Like you’re 10: Some pictures are too big for one kid to colour, so a whole team colours different corners at once — but they all have to start together, or the ones who started early just sit around waiting and wasting crayons.

Training ranges from a single-GPU fine-tune to a hundred-GPU distributed job that must move in perfect lockstep. A training platform gives data scientists a paved way to run all of it — interactive notebooks, one-shot jobs, and large distributed runs — without hand-assembling the orchestration each time. Three technologies define this space on Kubernetes, and one scheduling concept, gang scheduling, makes distributed training possible at all.

Kubeflow — the training toolkit

Kubeflow is the closest thing to an “ML platform in a box” on Kubernetes, and it’s really a suite. Its Training Operator introduces first-class CRDs for distributed training — PyTorchJob, TFJob, MPIJob, XGBoostJob — each of which knows how to launch the right topology of master and worker pods and wire up the environment variables those frameworks expect for multi-node communication. Kubeflow Notebooks gives scientists managed, GPU-backed Jupyter environments on the cluster (not their laptops), so exploration already runs where production runs. Katib automates hyperparameter tuning and neural architecture search by launching many trials in parallel. And Kubeflow Pipelines (met earlier) strings it all into DAGs. A platform team rarely adopts all of Kubeflow; more often it cherry-picks the Training Operator and Pipelines and wraps them in a thinner, opinionated golden path.

apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata: { name: bert-finetune, namespace: team-nlp }
spec:
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      template:
        spec:
          containers:
            - name: pytorch
              image: registry.acme.io/ml/bert:1.2
              resources: { limits: { nvidia.com/gpu: 1 } }
    Worker:
      replicas: 3              # 1 master + 3 workers = 4 pods that must run TOGETHER
      template:
        spec:
          schedulerName: volcano         # gang-schedule: all 4 or none (see below)
          containers:
            - name: pytorch
              image: registry.acme.io/ml/bert:1.2
              resources: { limits: { nvidia.com/gpu: 1 } }

Ray & KubeRay

Ray takes a different angle: rather than a per-framework CRD, it’s a general-purpose distributed-compute framework with a Pythonic API and a set of libraries — Ray Train (distributed training), Ray Tune (hyperparameter search), Ray Data (distributed data loading), and Ray Serve (model serving). You write ordinary Python and Ray fans it across a cluster of workers. KubeRay is the operator that runs Ray on Kubernetes through three CRDs: RayCluster (a head pod plus autoscaling worker pods), RayJob (run a job, then tear the cluster down), and RayService (a long-lived Ray Serve deployment). Ray shines when your workload doesn’t fit neatly into one framework’s box — reinforcement learning, large-scale batch inference, mixed data-plus-training pipelines, or LLM fine-tuning — and it has become the default substrate for a lot of frontier-model work precisely because of that flexibility.

Gang scheduling — Volcano & MPI

Here is the concept that makes or breaks distributed training. A 4-pod PyTorchJob only works if all four pods run at once — they form a communication group and block waiting for each other. The default Kubernetes scheduler places pods one at a time, so on a busy cluster it can happily schedule three of your four workers, find no room for the fourth, and leave you with three expensive GPUs held but idle, deadlocked, waiting forever for a partner that never arrives. The fix is gang scheduling: schedule the whole group all-or-nothing. Volcano (a CNCF project) is the batch scheduler that provides this on Kubernetes — via a PodGroup with a minMember, it won’t place any pod of the gang until it can place them all — and it adds queues, fair-share, and priority for multi-team GPU sharing. MPI-style jobs (via MPIJob) rely on the same guarantee. The alternative, Kueue, is a newer Kubernetes-native job-queueing layer that adds admission and quota control on top. Without gang scheduling, a shared GPU cluster running distributed jobs will eventually deadlock itself; with it, jobs wait politely in a queue and only start when they can fully run.

Model serving & inference

☺ Like you’re 10: Once a picture is finished, you hang it where everyone can ask it questions. But a picture that answers giant word-questions (an LLM) needs a very special, very clever frame — and you don’t want to keep the lights on when nobody’s looking at it.

Training produces a model; inference is putting it to work behind an endpoint. Serving is where ML meets classic platform concerns — autoscaling, rollouts, latency SLOs — plus a few that are unique to models: expensive-to-load weights, GPU-bound request handling, and, for large language models, a completely different performance profile. This is where Mira’s self-service promise gets real: a data scientist should get a live, autoscaling, canaryable endpoint from a registered model without writing a serving stack.

KServe & Seldon

KServe (formerly KFServing) is the leading model-serving layer on Kubernetes. Its InferenceService CRD turns a model reference into a production endpoint with batteries included: a predictor that loads the model, optional transformer (pre/post-processing) and explainer components, standardised prediction protocols, and — crucially — autoscaling including scale-to-zero via its Knative-based serverless mode, so an idle endpoint costs nothing until a request wakes it. It supports many runtimes out of the box (scikit-learn, XGBoost, PyTorch, NVIDIA Triton, and LLM runtimes like vLLM) and does canary rollouts natively by splitting traffic between model revisions. Seldon Core is the other major player; its strength is inference graphs — chaining models, transformers, routers, and explainers into a DAG — plus first-class A/B tests and multi-armed bandits for routing traffic between competing models. Reach for KServe for standardised single-model serving with scale-to-zero; reach for Seldon when you need complex multi-model graphs and experimentation baked in.

Serving LLMs — vLLM & TGI

Large language models break ordinary serving, and understanding why is senior-level knowledge. An LLM generates text one token at a time, and each step must attend to every previous token — so the server holds a growing KV cache (the attention keys and values) in GPU memory whose size scales with sequence length and batch size. Naive serving fragments and wastes this memory badly, capping how many requests fit on a card. vLLM solved this with PagedAttention, which manages the KV cache like an operating system pages virtual memory — in fixed blocks, near-zero waste — dramatically raising the number of concurrent requests a GPU can hold. It pairs that with continuous (in-flight) batching: instead of waiting to assemble a fixed batch, requests join and leave the running batch every decoding step, keeping the GPU saturated. Text Generation Inference (TGI) from Hugging Face offers the same class of optimisations — continuous batching, tensor parallelism to shard a model across GPUs, quantization. The platform lesson: LLM serving is not “a bigger Flask app.” Its throughput comes from cache management and batching, it is memory-bound on the KV cache, and you serve it through a purpose-built runtime — increasingly vLLM, often wrapped by KServe.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata: { name: llama-chat, namespace: genai }
spec:
  predictor:
    minReplicas: 0             # scale-to-zero when idle — no GPU burned overnight
    maxReplicas: 8
    scaleTarget: 10            # target concurrent requests per replica
    model:
      modelFormat: { name: vllm }        # serve via the vLLM runtime
      storageUri: s3://models/llama-3-8b-instruct/
      resources:
        limits: { nvidia.com/gpu: 1 }
  # Send 10% of traffic to this new revision; promote by raising the percentage.
  canaryTrafficPercent: 10

Autoscaling, scale-to-zero & canaries

Serving inherits the autoscaling toolbox from Scaling & Scheduling, with GPU-specific twists. Scale-to-zero is enormously valuable for GPU endpoints — an idle model that holds a card costs real money, so dropping to zero replicas when traffic stops can save the majority of an inference bill — but the cold start is brutal: loading multi-gigabyte weights onto a GPU can take tens of seconds, a penalty the first caller pays. So reserve scale-to-zero for spiky, latency-tolerant, or internal endpoints, and keep a warm floor (minReplicas: 1) for hot user-facing paths, or invest in faster model loading. For scaling up, the right signal is rarely CPU — it’s concurrent requests or queue depth (KServe’s concurrency target, or KEDA on a request queue). And because a bad model is worse than a slow one, rollouts must be progressive: shift a small slice of traffic to the new model, watch quality and latency, and promote or roll back — the exact progressive-delivery pattern, plus ML-specific tactics like shadow deployment (send the new model a copy of live traffic but discard its answers) to validate a model on real requests before it serves a single user.

The LLMOps / GenAI stack

☺ Like you’re 10: A giant word-robot doesn’t know your facts, so you give it a super-fast librarian who finds the right pages first and hands them over. And you always keep a polite bouncer at the door checking what goes in and comes out.

Generative AI added a whole new layer of platform concerns on top of classic MLOps. When teams consume large models — often ones they didn’t train — the hard problems shift from “how do I train this?” to “how do I ground it in our data, evaluate whether it’s any good, keep it safe, and afford it?” That layer is LLMOps, and a modern AI platform must provide it as paved infrastructure or every team will reinvent it badly.

Vector databases & RAG

The dominant pattern for grounding an LLM in private knowledge is Retrieval-Augmented Generation (RAG). You chop your documents into chunks, run each through an embedding model to get a vector (a list of numbers capturing meaning), and store those vectors in a vector database — Milvus, Qdrant, Weaviate, Chroma, or Postgres with pgvector. At query time you embed the user’s question, find the nearest vectors (an approximate-nearest-neighbour search using indexes like HNSW or IVF), pull back the matching chunks, and stuff them into the prompt as context. The model then answers from your documents rather than its frozen training data — which cuts hallucination and lets you update knowledge by re-indexing instead of retraining. As a platform capability, RAG means running a vector database as a stateful, backed-up, scalable service (its state is your knowledge base), plus an embedding-and-indexing pipeline that keeps it fresh.

Prompt & evaluation pipelines, guardrails

Prompts are now artifacts: they change behaviour as surely as code, so a mature platform versions them, reviews them, and tests them. That demands evaluation pipelines — because you can’t eyeball whether a change made the system better across thousands of cases. Evals run a model against a golden dataset and score it, often using LLM-as-judge (a strong model grades another model’s answers), regression suites (did this prompt change break the cases that used to pass?), and RAG-specific metrics (was the retrieved context actually relevant; is the answer faithful to it?). Frameworks like Ragas and promptfoo formalise this, and it belongs in CI: no prompt or model change reaches production without passing evals, exactly as no code change ships without tests. Alongside evals sit guardrails — the safety layer that filters inputs and outputs: blocking prompt-injection attempts, redacting PII, refusing disallowed content, and constraining outputs to a schema. Tools like NeMo Guardrails and Llama Guard implement this, and it’s the LLMOps face of the same policy-as-code discipline that guards the rest of the platform.

Token-cost observability

GenAI introduced a cost unit the rest of the platform never had: the token. Every request consumes prompt tokens (the context you send — RAG can make this large) and completion tokens (what the model generates), and whether you pay a per-token API price or the amortised cost of self-hosted GPUs, cost scales with tokens. Without visibility this spirals: a single verbose RAG prompt or a runaway agent loop can cost more than a thousand ordinary requests, and nobody notices until the invoice. So the platform must meter tokens as a first-class signal — per request, per team, per feature — feeding it into the same observability and FinOps systems that watch everything else, with alerts on anomalies. Two levers cut the bill directly: prompt caching (reuse the model’s work on a repeated prefix) and semantic caching (return a stored answer when a new question is close enough to a previous one). This is Ellie’s newest job — remembering not just metrics and logs, but every token and what it cost.

Data platforms on Kubernetes

☺ Like you’re 10: Models are only as good as what they eat. Under the fancy AI kitchen there’s a whole pantry and delivery system — fresh ingredients streaming in, a shelf of prepared ingredients everyone reuses, and a giant warehouse you can ask questions of.

No model is better than its data, and the data layer underneath an ML platform is a substantial platform in its own right. It streams events in real time, stores enormous volumes cheaply, serves features to both training and inference, and orchestrates the jobs that move data around. Much of it now runs on Kubernetes through operators, giving the data plane the same declarative, self-healing management as everything else.

Streaming — Kafka & Strimzi

Real-time ML — fraud detection, recommendations, live features — runs on event streams, and Apache Kafka is the backbone. Running stateful Kafka on Kubernetes by hand is hard, which is why Strimzi (a CNCF project) exists: it’s an operator that manages Kafka clusters, topics, users, and connectors as Kubernetes custom resources. You declare a Kafka resource and Strimzi provisions and heals the brokers; you declare a KafkaTopic and it creates the topic — Kafka administration becomes GitOps. This turns a notoriously operationally heavy system into a declarative one, and makes streaming a self-service capability teams can request rather than a bespoke deployment the platform babysits.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: clickstream-events
  namespace: streaming
  labels: { strimzi.io/cluster: analytics-kafka }
spec:
  partitions: 12            # parallelism for consumers (and KEDA scale target)
  replicas: 3               # survive a broker loss
  config:
    retention.ms: "604800000"   # keep 7 days of events for replay/backfill

Feature stores

A subtle, expensive ML bug is training/serving skew: the code that computes a feature during training differs from the code that computes it at inference, so the model sees inconsistent inputs and quietly underperforms. A feature store (Feast is the open-source standard) fixes this by defining each feature once and serving it from two synchronised stores: an offline store (the warehouse — large historical values for training) and an online store (a low-latency key-value store like Redis — the latest values for real-time inference). Training reads point-in-time-correct history; serving reads fresh values; both come from the same definition, so the skew disappears. A feature store also makes features reusable — one team’s well-built “customer 30-day spend” feature becomes shared platform infrastructure instead of being re-derived, differently, by everyone.

Lakehouse, query engines & orchestration

The storage foundation has converged on the lakehouse: cheap object storage (the data lake) plus open table formats — Apache Iceberg, Delta Lake, Apache Hudi — that add warehouse-grade features (ACID transactions, schema evolution, time travel) on top of files. Over that sit query engines — Trino/Presto for interactive SQL across sources, Apache Spark for large-scale processing — which increasingly run on Kubernetes via operators (the Spark Operator turns a Spark job into a CRD). Tying it together is orchestration: Apache Airflow is the long-standing DAG scheduler for data pipelines, while Argo Workflows is the Kubernetes-native alternative (and, recall, what Kubeflow Pipelines compiles to). The platform pattern is consistent with everything else on this page: object storage for state, open formats to avoid lock-in, engines and orchestrators as declarative Kubernetes workloads, all self-service.

The platform engineer’s role

☺ Like you’re 10: The platform engineer is the grown-up who builds the whole crayon table so the kids never have to think about it — they just draw. And who quietly makes sure the school can still afford crayons next month.

Step back from the technologies and the job is familiar: turn a sprawl of powerful-but-sharp tools into a paved golden path that ML teams can walk safely and fast, while keeping the (enormous) cost under control. Everything above — GPUs, training, serving, LLMOps, data — is raw material. The platform engineer’s value is assembling it into a small number of opinionated, secure-by-default, self-service experiences, so a data scientist ships a model without becoming a Kubernetes, CUDA, and vLLM expert first.

Paving golden paths for ML/AI teams

The golden-path test for an ML platform is concrete: can a data scientist go from “a model in the registry” to “a live, autoscaling, canaryable endpoint” with a single self-service action — no ticket, no hand-written serving YAML, no GPU-pool archaeology? The way you get there is the same abstraction pattern from Platform APIs & CRDs: hide the intricate InferenceService, GPU requests, node selectors, autoscaling, and guardrails behind a thin platform API or a portal template, and give teams sensible, GPU-aware defaults. Mira’s whole job is turning the caterpillar of raw ML infrastructure into a butterfly of one-button self-service — a “deploy model” action that quietly does everything correct underneath.

Taming GPU cost

If there is one number an AI platform engineer is judged on, it’s GPU efficiency, because the accelerators dominate the bill. The levers are everything this page has quietly been assembling: share cards with MIG and time-slicing so small jobs don’t each hold a whole GPU; gang-schedule and queue so distributed jobs don’t hold GPUs while deadlocked; scale inference to zero when idle and keep only warm floors where latency demands; put interruption-tolerant training on spot; right-size requests and let idle GPU nodes scale down; and meter tokens so GenAI spend is visible and attributable. Measure utilization, not allocation — an allocated-but-idle GPU is pure waste — and make cost a first-class, per-team signal. This is where the AI platform meets FinOps and Scaling & Scheduling head-on, and where a good platform engineer saves the company more than their own cost many times over.

Bringing it together

A mature AI/ML platform is the whole course applied to a demanding new workload: GitOps reconciling model and data infrastructure from Git, CRDs and operators (KServe, KubeRay, Strimzi, the GPU Operator) turning ML concepts into Kubernetes-native APIs, self-service golden paths hiding the sharp edges, observability extended to GPUs, models, and tokens, policy extended to model promotion and guardrails, and FinOps extended to the most expensive compute in the building. The AI/ML platform isn’t a separate thing bolted on the side — it’s your platform’s hardest tenant, and serving it well is the same discipline you’ve been building all along, turned up to eleven.

AI/ML is your platform’s most demanding tenant — the biggest data, the scarcest hardware, the highest bill, and a brand-new safety surface. But the tools that tame it are the ones you already know: declarative APIs, self-service golden paths, reconciliation, observability, and cost discipline, pointed at a harder problem. Keep following the money in FinOps for Platforms, the placement machinery in Scaling & Scheduling, or the abstractions that hide it all in Self-Service.