MLOps & AI Infrastructure

Training a model is a project; keeping it working is a job. MLOps is the discipline of shipping machine-learning models to production and keeping them reliable there — the data pipelines, serving stack, monitoring, versioning, and retraining loops that turn “it works in my notebook” into “it works for a million users at 3 a.m.” If you’ve built an agent or a RAG system, you’ve already met the surface; this page is the plumbing underneath.

MLOps = DevOps for machine learning

☺ Like you’re 10: Regular software is a recipe that never changes — same steps, same cake, every time. A model is a recipe that learned itself from ingredients, so if the ingredients drift, the cake quietly changes too. MLOps is the extra rulebook for cooking with a recipe that can surprise you.

You may already know DevOps: the practices that let software teams ship code quickly and safely — version control, automated testing, continuous integration and delivery (CI/CD), infrastructure-as-code, monitoring. MLOps is DevOps carried over to machine learning, plus the extra headaches that come from the fact that an ML system isn’t just code — it’s code and data and a trained model, all of which can change independently.

That third axis is the whole difference. Ordinary software is deterministic: same input, same output, forever, until someone edits the code. A model’s behavior is baked in from data during training, so it can “rot” even when nobody touches the code — because the world the data described has moved on. That’s why MLOps adds concerns that classic DevOps never had to worry about:

The same instinct powers LLMOps — the version of this discipline aimed at large language models and the agents built on them. Whether you’re serving a small classifier you trained in-house or calling a frontier model like Claude, GPT, or Gemini through an API, the reliability questions rhyme: is it up, is it fast enough, is it still good, and can I roll back when it isn’t? This page focuses on the classic model-serving case; the agent-specific slice is covered in Production & Operations.

The ML lifecycle: data → train → deploy → monitor → retrain

☺ Like you’re 10: It’s a circle, not a finish line. Gather ingredients, bake, put the cake in the shop, taste-check every day, and when it starts going stale, go bake a fresh batch. Round and round — a model is never “done.”

MLOps is best understood as a loop, not a one-way pipeline. A model that ships is at the start of its real life, not the end. The stages:

1. Data. Collect, clean, label, and validate the data the model learns from. This is where most of the real work lives — see Data Engineering. Garbage in, garbage model out.
2. Train. Fit the model and evaluate it on held-out data to estimate how well it’ll do on inputs it’s never seen. The mechanics are in Training Models; MLOps cares that this step is repeatable and tracked.
3. Deploy. Package the trained model and put it somewhere it can answer requests — behind an API, in a batch job, or on a device. This is “serving,” the next section.
4. Monitor. Watch the live model for slowdowns, errors, and — the sneaky one — quality decay as the world drifts. Covered below.
5. Retrain. When monitoring says quality has slipped (or on a schedule), gather fresh data and run the loop again, then ship the new version. Back to step one.

The arrow that makes it a loop is the last one. In traditional software you might deploy and forget; in ML, deploying is when the clock starts ticking on the model’s freshness. A well-run team automates as much of this circle as possible so that “the model is drifting” leads, with minimal human toil, to “a retrained model is live.”

Model serving: getting predictions to users

☺ Like you’re 10: Once the cake recipe is perfect, how do you hand out cake? Bake-to-order the moment someone walks in (fast, one at a time), or bake a giant tray overnight for the whole school (cheaper per slice, but nobody eats until morning). Serving is choosing how the cake reaches people.

Serving (or inference serving) is running the trained model to produce predictions for real requests. The first big fork is when predictions are needed:

The most common real-time pattern is to wrap the model in an HTTP API: the model loads into memory once when the server starts, then each request runs a forward pass and returns the prediction as JSON. This is exactly the shape you already use when you call a hosted model — the Claude, OpenAI, or Gemini APIs are enormous, highly optimized versions of this same “model behind an endpoint” idea.

# A real-time model server, in spirit
model = load_model("churn-v7.bin")     # loaded ONCE at startup, kept in memory

on_request(input):                     # runs PER request
    features = preprocess(input)
    score    = model.predict(features) # the forward pass — the expensive part
    return { "churn_risk": score }     # JSON back to the caller, fast

Two forces dominate real-time serving. Latency vs throughput is a constant trade: batching incoming requests together lets a GPU crunch many at once (great throughput) but makes each caller wait a moment for the batch to fill (worse latency). And hardware matters: deep models — especially large language models — run far faster on GPUs (or specialized accelerators like TPUs and various inference chips) than on ordinary CPUs, because a forward pass is mostly big matrix multiplications that those chips are built to do in parallel. Small classical models (see Classical ML) often serve happily on plain CPUs; a big deep-learning model usually needs an accelerator to hit acceptable latency.

Scaling & cost: serving without going broke

☺ Like you’re 10: A bakery hires more staff at lunch rush and sends them home when it’s quiet (autoscaling), uses a smaller cheaper oven when the fancy one is overkill (quantization), keeps popular cakes on the counter so it doesn’t re-bake them (caching), and grabs day-old-price oven time when it can wait (spot compute). All tricks to serve more for less.

GPUs are expensive, and traffic is spiky, so a huge part of MLOps is serving well without a runaway bill. The main levers:

• Autoscaling. Add serving instances when traffic rises and remove them when it falls, so you pay for capacity roughly matching demand rather than for peak load around the clock. Scaling GPU workloads is trickier than web servers — instances are pricey and slow to warm up — so teams tune this carefully.
• Quantization & distillation. Shrink the model itself. Quantization stores the weights at lower numeric precision (say 8- or 4-bit integers instead of 16-bit floats), cutting memory and speeding inference with usually small accuracy loss. Distillation trains a smaller “student” model to mimic a big one. Both are why a capable model can run on a laptop or phone — the same techniques behind local LLMs.
• Caching. If the same (or similar) request comes in repeatedly, don’t recompute — return the stored answer. For LLMs this shows up as prompt caching (reusing the processed prefix of a repeated prompt) and semantic caching (matching similar questions), both of which cut cost and latency dramatically for repetitive traffic.
• Spot / preemptible compute. Cloud providers rent spare capacity at a steep discount, with the catch that they can reclaim it at short notice. Fault-tolerant, restartable work — especially batch inference and training — can run on spot instances for a fraction of the price.

These stack. A well-tuned production model might be quantized to fit more copies per GPU, autoscaled to match a daily traffic curve, front by a cache that absorbs repeat questions, with heavy batch jobs pushed onto spot compute overnight. The art is squeezing cost without quietly degrading quality or latency past what users will tolerate.

Monitoring: why models “get worse” silently

☺ Like you’re 10: Nobody broke the cake recipe — but the customers changed. Last year everyone loved chocolate; this year they want vanilla, and your chocolate-only bakery is suddenly getting bad reviews without changing a thing. A model can go stale the same quiet way, so you have to keep tasting.

This is the concern that surprises people coming from ordinary software. A deployed model can degrade with zero code changes and zero bugs, purely because the live data has drifted away from what it was trained on. The failure is silent: the model keeps returning confident predictions, they’re just increasingly wrong. Monitoring is how you catch it. The main things to watch:

Data drift and concept drift are the two flavors of the same rot. In data drift the questions change; in concept drift the correct answers change. Either way, a model trained on last year’s world is answering this year’s — and quality decays. The hard part is that you often can’t measure accuracy directly in real time, because the true answer (did this customer actually churn? was that transaction really fraud?) arrives days or weeks later, if ever. So teams watch proxies: shifts in the input distribution, drops in prediction confidence, changes in the mix of outputs — early-warning signs that ground truth, when it arrives, will look worse.

For LLMs and agents the same idea applies but the metric is fuzzier: there’s rarely a single “accuracy” number, so teams lean on evals (running the model against a curated test set and scoring it), sampled human review, and automated judges. The through-line with classic ML is identical — measure quality continuously in production, because it will not stay put. The agent-flavored version of all this lives in Production & Operations.

Versioning & reproducibility: CI/CD for ML

☺ Like you’re 10: Every batch of cakes gets a label saying exactly which recipe, which flour, and which oven made it. So when one batch is amazing — or awful — you can make it again, or never make it again. Without labels, you’re guessing.

Because an ML system is code and data and a model, reproducibility means tracking all three together. The goal is to be able to answer, for any prediction in production: exactly which model made this, trained on which data, by which code, with which settings? If you can’t answer that, you can’t debug it, audit it, or reliably roll back. What gets versioned:

• Model versions. Each trained model is an artifact with a version, stored in a model registry so you always know which one is live and can redeploy an older one instantly. (This is what lets Benny roll back to v7.)
• Data versions. The exact dataset a model was trained on, snapshotted, so a training run can be reproduced. Two models trained on “the customer table” months apart are not comparable unless you know which snapshot each saw.
• Experiment tracking. Every training run logs its hyperparameters, code commit, data version, and resulting metrics — so you can compare runs and know why one model beat another instead of guessing.
• Config & environment. Library versions, random seeds, and settings, usually pinned in a container image, so the run isn’t at the mercy of whatever happened to be installed that day.

CI/CD for ML then automates the loop the way DevOps automates code deploys — but with extra gates. A push might trigger: retrain on the latest data snapshot → run the eval suite → only if quality clears a bar, register the new model version and deploy it (often gradually — canary or shadow first, then full traffic). The crucial addition over plain CI/CD is that the passing test is statistical: “the new model scores at least as well as the current one on the held-out set,” not merely “the code compiles.” A model that builds fine but scores worse should never reach users.

How this differs from Production & Operations

☺ Like you’re 10: This page is about running the bakery — keeping the ovens hot and the cakes fresh. The other page is about running the delivery robot that takes cake to customers, makes decisions on the way, and might do something silly if you’re not watching. Same spirit, different layer.

These two lessons are close cousins, and it’s worth being precise about the split so you know which to reach for. This page (MLOps) is about serving models — the training-to-serving-to-retraining loop for an ML model as a component. Production & Operations is about operating agents and LLM applications — the systems built on top of models that reason, call tools, and take actions.

In practice they layer: a production agent often calls one or more served models (some you trained, some you rent from a provider), so a real system runs both disciplines at once. If you’re building agents, start with Building Agents and Agentic AI, operate them with Production & Operations, and treat this page as the layer beneath — how the models those agents depend on stay fast, cheap, and correct. For the security angle that cuts across both, see AI Security and Responsible AI; to see where MLOps sits in the wider toolscape, the Ecosystem and Further Reading pages point to the mainstream platforms and papers.