AI Advanced · Training & Fine-Tuning Models

Training & Fine-Tuning Models

Almost everything in this course has treated the model as a finished engine you drive — you prompt it, ground it with RAG, wrap it in agents. This page steps into the other seat: how the engine itself is built and tuned. You’ll get the honest, oriented version — pretraining, fine-tuning, and alignment; when to change the weights versus just changing the prompt; and the tricks (LoRA, distillation, quantization) that make custom models practical instead of ruinously expensive.

Two paths: the model-user and the model-builder

☺ Like you’re 10: Most of this course taught you to drive the car really well. This page pops the hood and shows you how the engine gets made — and when it’s worth learning to tune one yourself instead of just driving.

Nearly every page so far has been about the model-user path: take a capable, already-trained model — Claude, GPT, Gemini, Llama — and get the most out of it with prompting, retrieval, tools, and agent design. That path is where most real value is created today, and it’s where most careers in applied AI live. You never touch the weights; you change what’s in front of the model.

This page is your orientation to the model-builder path: the work of actually shaping the weights so the model itself behaves differently. You almost certainly won’t pretrain a frontier model from scratch — that’s a hundred-million-dollar, thousand-GPU undertaking reserved for a handful of labs. But fine-tuning an existing open model, or an API-hosted one, is very much within reach of a small team, and knowing when it’s the right tool is a genuine skill.

Think of this lesson as the bridge between the applied world you already know and the deeper fields next door: deep learning is the machinery underneath, MLOps is how you run training and serving as a real system, and data engineering is where the training data comes from. Here we stay at the orientation altitude: enough to make good build-vs-buy decisions and know where to dig deeper.

Pretraining, fine-tuning, alignment: the three stages

☺ Like you’re 10: First the robot reads the whole library so it can talk (pretraining). Then it practises specific homework so it’s good at a job (fine-tuning). Then it learns manners — be helpful, be honest, don’t say scary things (alignment). Three different lessons, in order.

A modern chat model is built in stages, and they do genuinely different things. Mixing them up is the source of a lot of confusion, so here’s the clean version:

Pretraining is the expensive, once-per-model step. The result is a “base” model that has soaked up an astonishing amount of knowledge but doesn’t know it’s supposed to be an assistant — feed it “The capital of France is” and it happily continues the sentence, but ask it a question and it might just write more questions. (Why next-token prediction produces so much capability is the story in How models work.)

Supervised fine-tuning takes that base and trains it further on examples that demonstrate the behaviour you want — thousands of high-quality “here’s a request, here’s an ideal answer” pairs. This is what turns an autocomplete engine into something that follows instructions.

Alignment is the manners class, and it’s where the two acronyms you’ll hear live:

• RLHF (Reinforcement Learning from Human Feedback) — humans rank model outputs, those rankings train a reward model, and the main model is then optimised to score well against it. This is the classic recipe behind the first ChatGPT and much of what made assistants feel helpful and safe.
• DPO (Direct Preference Optimization) — a newer, simpler approach that learns directly from “A is better than B” preference pairs without training a separate reward model. It’s become popular because it’s more stable and cheaper to run, and it’s a common choice for teams doing their own alignment on open models.

Labs also layer in related techniques — Anthropic’s Constitutional AI uses a written set of principles to have the model help critique and improve its own responses, reducing how much human labelling is needed. The details move fast; the shape is stable: learn language → learn the task → learn the preferences. The Responsible AI page goes deeper on what “aligned” should even mean.

When to fine-tune (vs RAG or just better prompting)

☺ Like you’re 10: Three ways to help your robot: tell it clearly what you want (prompting), hand it the right pages to read (RAG), or send it to a private class so the new habit is baked in (fine-tuning). Try the cheap ones first — you bake in a habit only when the first two aren’t enough.

This is the single most important practical decision on this page, and people get it backwards constantly. The reflex to fine-tune is usually wrong. Reach for the cheapest tool that solves your problem, in this order:

1. Better prompting first. A clearer instruction, a few examples in the prompt (few-shot), or a system prompt that sets tone and format solves a huge fraction of “the model won’t do X” problems — with zero training, instantly editable. Start here always. (See Prompting.)
2. RAG for knowledge. If the gap is that the model doesn’t know your facts — your docs, your product, this week’s prices — retrieval is almost always the answer. You can update the knowledge the instant a doc changes, and the model can cite its sources.
3. Fine-tuning for behaviour. If the gap is how the model behaves — a consistent house tone, a strict output format, a specialised skill or domain style it can’t reliably hit even with good prompts — then baking it into the weights can win.

Here’s the crux: fine-tuning is a poor way to add knowledge. It’s expensive to redo whenever facts change, it can’t easily point at a source, and models still hallucinate confidently about baked-in facts. It shines instead at consistency of behaviour — matching a brand voice on every single reply, always producing valid JSON in a fixed schema, classifying support tickets in your taxonomy, or writing in a narrow professional style. And these aren’t exclusive: a real system often fine-tunes for style and uses RAG for facts at the same time.

One more angle: fine-tuning can also make a smaller, cheaper model good enough at one narrow task to replace a big expensive one — you trade a training cost up front for lower per-call cost forever. That economic case, not knowledge, is often the best reason to do it.

Parameter-efficient fine-tuning: LoRA & PEFT

☺ Like you’re 10: Instead of rewriting the robot’s whole giant brain to teach one new trick, you clip on a tiny extra notebook of changes. The huge brain stays exactly as it was; the little notebook holds just what’s different — and it’s cheap to make, store, and swap out.

A full fine-tune updates every weight in the model. For a modern model that’s billions of parameters — you’d need the memory and compute of the original training run, plus a full-size copy of the model saved for every variant you make. It’s so expensive that almost nobody outside the big labs does it. The answer is parameter-efficient fine-tuning (PEFT): change only a tiny slice of the model and freeze the rest.

The dominant PEFT method is LoRA (Low-Rank Adaptation). The intuition: instead of editing the enormous weight matrices directly, you freeze them and train small “adapter” matrices alongside that capture just the difference your task needs. Because those adapters are tiny — often well under 1% of the model’s parameters — LoRA fine-tuning fits on modest hardware and produces an adapter file measured in megabytes rather than a full model copy in the tens of gigabytes.

That last row is the quietly huge win: you can keep one base model in memory and attach a different LoRA adapter per customer, per task, or per tone — “tune once per behaviour, swap cheaply.” Variants like QLoRA push this further by also quantizing the frozen base (more on quantization below), letting you fine-tune surprisingly large models on a single consumer-grade GPU. This is exactly why the open-model ecosystem exploded: PEFT made custom models a weekend project instead of a data-centre project. Hosted fine-tuning APIs from the major providers use these efficient methods under the hood too, so even when you never see the word “LoRA,” it’s often what’s running.

Data: the part that actually decides everything

☺ Like you’re 10: A fine-tune only ever learns from the examples you show it. Show it a few hundred excellent examples and it learns the good habit. Show it a huge pile of sloppy or contradictory ones and it learns to be sloppy. Neat, honest examples beat lots of messy ones.

If you take one thing from this page: fine-tuning quality is data quality. The model copies the patterns in your examples — including the mistakes. A small set of clean, consistent, on-target examples routinely beats a large noisy one. What matters:

• Quality over quantity. A few hundred carefully curated examples that all demonstrate the exact behaviour you want often outperform tens of thousands of inconsistent ones. Every bad example is a bad habit you’re teaching.
• Consistency. If your examples disagree with each other — two different tones, two different formats — the model learns the average of the confusion. Pick the target behaviour and make every example show it.
• Format. Data is usually structured as prompt→completion or chat-message pairs (a JSONL file is the common shape), matching how you’ll actually call the model. Format your training data the way you’ll query in production, or you’ll get a mismatch.
• Coverage. Include the edge cases and the “say no” cases, not just the happy path — otherwise the model won’t know how to handle them.
• Licensing & privacy. You are baking this data into weights. Make sure you have the right to train on it, that it’s free of PII you shouldn’t retain, and that scraped or third-party content is properly licensed. This is a legal and ethical question, not just a technical one — see Responsible AI and AI Security.

Building this dataset is real work, and it overlaps heavily with data engineering. A common and effective shortcut: use a strong model to help draft candidate examples, then have humans review and correct them — you get scale from the model and quality from the humans. (Watch the licence terms: some providers restrict using their outputs to train competing models.)

Distillation & quantization: shrinking models to fit

☺ Like you’re 10: Two ways to make a big heavy robot small enough to carry. One: a clever small student robot watches the big one and copies how it thinks (distillation). Two: you round off the numbers in the robot’s brain so they take less room — a little less precise, a lot lighter (quantization).

These two techniques answer a different question from fine-tuning: not “how do I change the behaviour?” but “how do I make this model small and cheap enough to run where I need it?” — on a phone, at the edge, or just at lower cost and latency. They’re central to the whole local & on-device story.

• Distillation trains a small “student” model to imitate a large “teacher” model — the student learns from the teacher’s outputs (and often its probability distributions) rather than from raw data alone. The result is a much smaller model that keeps a surprising amount of the big one’s quality on the tasks it was distilled for. Many of the fast, cheap tiers you use every day are distilled from larger siblings.
• Quantization stores the model’s weights at lower numerical precision — e.g. 8-bit or 4-bit integers instead of 16-bit floats. That roughly halves or quarters the memory and speeds up inference, for a usually-small quality cost. Quantization is what lets a model that needs a data-centre GPU at full precision run on a laptop or a good phone; formats like GGUF and 4-bit schemes are what power the local-model boom.

They stack, and they stack with everything above: you might take an open base model, fine-tune it with LoRA for your task, then quantize the result to 4-bit so it runs cheaply — a fully custom, deployable model built without ever pretraining anything. For how these shrunken models get served in production, see Production & Ops.

Evaluating a fine-tune — and the pitfalls

☺ Like you’re 10: Always test the robot before and after the extra lesson, on the same questions. Otherwise you can’t tell if the lesson helped, did nothing, or accidentally made it forget things it used to know.

A fine-tune isn’t done when training finishes — it’s done when you’ve measured that it’s better and not secretly worse. The non-negotiable habit is a held-out evaluation set the model never trained on, run against both the original and the tuned model so you have a real before/after. Judge it on the behaviour you tuned for (does it match the target style/format?) and on the general skills you didn’t want to break. The main ways fine-tunes go wrong:

• Catastrophic forgetting. Push too hard on your narrow data and the model can lose general abilities it used to have — great at your tone now, but suddenly worse at basic reasoning or other tasks. This is why you always evaluate general capability too, not just your target behaviour.
• Overfitting. With too few or too repetitive examples, the model memorises your training set instead of learning the pattern — it parrots your examples back and flops on anything slightly different. A held-out set is exactly how you catch this.
• Dating quickly. A fine-tune is frozen to whatever base model and data you used. When the underlying model gets a big upgrade, an off-the-shelf newer model may leapfrog your carefully tuned older one — and any facts you baked in are already going stale. Plan to re-tune, and keep your eval set so re-tuning is quick to validate.
• Distribution mismatch. If your training examples don’t look like real production traffic, your eval will look great and the deployment won’t. Draw eval data from the real world.

Evaluation is a discipline of its own — building good eval sets, using LLM-as-judge carefully, tracking metrics over time — and it lives at the heart of MLOps and Production & Ops. Treat a fine-tuned model like any other software artifact: versioned, tested, monitored, and re-validated whenever the world underneath it moves.

Check your answers

1. The three stages: pretraining (train on a vast pile of text by predicting the next token, giving a base model), supervised fine-tuning (continue training on curated instruction→response examples so it follows instructions), and alignment. RLHF (Reinforcement Learning from Human Feedback) has humans rank outputs to train a reward model, then optimises the model to score well against it; DPO (Direct Preference Optimization) learns directly from “A is better than B” preference pairs without a separate reward model, making it simpler, more stable, and cheaper.
2. Latest pricing vs brand voice: use RAG for the pricing — that’s missing knowledge, and retrieval updates the instant a doc changes and can cite its source, whereas fine-tuning is a poor way to add facts that go stale. For matching brand voice, fine-tune — a consistent tone is a habit/behaviour, and baking it into the weights makes every reply land in-voice rather than drifting as prompting does.
3. What LoRA freezes vs trains: LoRA freezes the enormous base weights and trains only small “adapter” matrices alongside that capture just the difference your task needs — often well under 1% of the parameters. That makes custom models cheap because training fits on modest hardware (often a single GPU) and produces a tiny adapter file (megabytes) instead of a full model copy, and you can hot-swap adapters on one shared base.
4. Two failure modes and the one habit: a fine-tune can suffer catastrophic forgetting (losing general abilities), overfitting (memorising the training set), dating quickly, or distribution mismatch — name any two. The one habit that catches them is a held-out evaluation set the model never trained on, run against both the original and tuned model for a real before/after that checks both the target behaviour and the general skills you didn’t want to break.