How AI Models Work
Before an AI can be an agent, wield tools, or search your codebase, it is just a model — a program that predicts text. This is the gentle on-ramp: what a large language model (LLM) actually is, how it reads and trains and guesses, why it sometimes makes things up, and what dials you get to turn. Everything else in this course — agents, MCP, retrieval — is scaffolding built around this one box. Learn the box first.
An AI model is like the world’s most well-read autocomplete. It has read a giant mountain of text, and when you give it some words, it guesses the next word — then the next, then the next — one at a time, until it has written you an answer. It isn’t looking anything up in a book, and it isn’t alive. It’s just really good at guessing what usually comes next.
A model is a next-word predictor
☺ Like you’re 10: Play the game where a friend says “Twinkle twinkle little…” and you blurt out “star.” You didn’t look it up — you’ve just heard the song so many times that “star” feels obvious. A model plays that game with every sentence ever written.
Strip away all the mystique and a large language model does exactly one thing: given some text, it predicts what token (roughly, the next chunk of a word) is most likely to come next. Then it adds that token to the text and predicts again. And again. It writes a whole essay the same way you finish the sentence “The sky is ___” — by feeling out the most probable continuation.
That’s it. There is no little librarian inside looking up facts in a database. There is no ledger of true statements it consults. When it writes “Paris is the capital of France,” it isn’t retrieving that fact from a table — it’s that, across everything it read, the words “the capital of France is” were followed by “Paris” overwhelmingly often, so “Paris” is the highest-probability next token.
| The myth | What’s actually happening |
|---|---|
| “It looks things up in a database.” | It predicts likely text from patterns baked into billions of numbers (its weights). Nothing is looked up; the “knowledge” is smeared across the model. |
| “It reasons like a person, step by step.” | It produces plausible next tokens. Chain-of-thought “reasoning” is itself just more predicted text — useful, but not a proof engine. |
| “It knows when it’s right.” | It has no built-in truth signal. A confident-sounding sentence and a wrong one are produced by the very same machinery. |
| “Each answer is stored somewhere.” | Answers are generated fresh, token by token, every time — which is why the same prompt can yield different wording. |
This one idea — predict the next token from learned patterns — explains almost everything else on this page. Hold onto it. Every model family you’ll meet (Claude, GPT, Gemini, Llama) is a next-token predictor at heart; they differ in size, training data, and tuning, not in the basic trick.
Tokens: how models read and count
☺ Like you’re 10: A model doesn’t read whole words — it reads them in little LEGO bricks. “Cat” might be one brick, but “unbelievable” could be three: “un,” “believ,” “able.” It builds and reads everything out of these bricks.
Models don’t see letters or words the way you do. Text is first chopped into tokens — common chunks of characters. A token is often a whole short word, sometimes a word-piece, sometimes just punctuation or a space. A rough rule of thumb for English: ~4 characters per token, or about ¾ of a word — so 100 tokens ≈ 75 words.
Text: "Tokenization isn't magic." Tokens: ["Token", "ization", " isn", "'t", " magic", "."] Count: 6 tokens for 4 "words"
Why should you care about invisible word-bricks? Because tokens are the unit that everything is measured in:
- Pricing is per token. Providers bill by input tokens (your prompt) and output tokens (the reply) — never by the word. A verbose prompt with a long pasted file costs real money in input tokens before the model writes a single word back.
- Context limits are in tokens. The “200K context” or “1M context” you see advertised is a token budget, not a word count. A big source file can eat tokens fast.
- Odd behaviour is often a token artefact. Models are famously bad at “how many r’s are in strawberry?” partly because they see tokens, not individual letters. They can’t easily count characters they never see separately.
1 token ≈ ¾ of a word ≈ 4 characters of English. When someone asks “will this fit in the context window?” or “what will this cost?”, the honest answer is “let’s count the tokens.” Code, non-English text, and unusual formatting tokenize differently (often less efficiently) than plain English prose.
Training: where the “smarts” come from
☺ Like you’re 10: First a puppy reads the whole library so it learns how words fit together (that’s pretraining). Then a trainer gives it treats for being helpful and honest, and a gentle “no” for being rude or making stuff up (that’s tuning). The library gives it knowledge; the treats give it manners.
A model isn’t programmed with facts — it’s trained. Training happens in two big stages, and understanding them explains both the model’s power and its limits.
1. Pretraining. The model is shown an enormous amount of text — books, articles, code, web pages — and plays the next-token game trillions of times, adjusting its internal numbers (weights) whenever it guesses wrong. Slowly it soaks up grammar, facts, coding patterns, styles of reasoning, and a great deal of world knowledge. This stage is where the raw capability comes from, and it is staggeringly expensive in compute.
2. Instruction-tuning & alignment. A freshly pretrained model is a wild autocomplete — it will happily continue text in unhelpful or unsafe ways. So it’s refined to become an assistant: shown examples of good answers (instruction-tuning), and then nudged toward being helpful, honest, and careful using human (and AI) feedback — the family of techniques often called RLHF (Reinforcement Learning from Human Feedback) and related methods. This is why Claude, GPT, and Gemini feel like they’re following your instructions rather than just parroting the internet.
| Stage | What it does | What it produces |
|---|---|---|
| Pretraining | Learns language & world knowledge by predicting next tokens over a huge corpus | Raw capability, but no manners; a “base model” |
| Instruction-tuning | Learns to follow instructions from curated examples | A model that answers your question instead of continuing it |
| Alignment (RLHF etc.) | Rewards helpful, honest, harmless behaviour | The polite, cautious assistant you actually talk to |
Professor Owl: Here is the whole secret, class. When I ask the model to finish “The first person on the Moon was…,” it doesn’t find the answer — it predicts the most likely next word: “Armstrong.” One word at a time.
Foxy: But wait — does it KNOW when it’s right? If it’s just guessing, how does it tell a true guess from a made-up one?
Professor Owl: Wonderful question, Foxy — and the honest answer is: it doesn’t, not by itself. A true sentence and a confident fib come out of the exact same guessing machine. When it invents something false, we call that a hallucination.
Nutty the Squirrel: All those good guesses came from somewhere! I gathered mountains of text — books, code, web pages — and the model practised on them until the patterns stuck. But my pile has a date on it: nothing newer than its knowledge cutoff.
Timmy the Turtle: So before you trust a shiny answer, you check it. I look it up in a real source, or hand the model the facts to work from. Trust, then verify — never the other way round.
Because a model learns from a fixed pile of text collected up to a certain date, it has a knowledge cutoff — it genuinely doesn’t know about events, libraries, or releases after that point, and will often confidently guess instead of admitting it. That’s a core reason we bolt on retrieval and tools: to feed the model fresh, real information at question time.
Why models hallucinate
☺ Like you’re 10: Imagine a super-confident friend who never says “I don’t know.” Ask them anything and they’ll give a smooth, sure-sounding answer — even when they’re making it up. The words sound right; that doesn’t make them true.
A hallucination is when a model states something false or invented as if it were fact — a made-up citation, a function that doesn’t exist, a plausible-but-wrong date. Newcomers assume this is a rare bug. It isn’t: it falls straight out of how models work.
Remember the core trick — the model produces the most likely-sounding next tokens, not verified truth. Fluency and accuracy are different things. When the model has strong patterns (common facts, popular libraries) its guesses are usually right. When it’s in thin territory (an obscure API, a niche person, something after its cutoff), it still produces a smooth, confident answer — because producing smooth confident text is all it does. It has no internal “I’m not sure” alarm wired to the output.
So hallucinations are worst exactly where they’re most dangerous: specific facts, exact quotes, API signatures, numbers, anything recent. The fix is not “find a model that never lies” — it’s to stop asking the model to be a source of truth and instead ground it:
- Grounding / retrieval. Give the model the real documents to answer from, so it summarizes real text instead of guessing. That’s the whole idea behind retrieval-augmented generation (RAG).
- Tools. Let it call a search engine, run code, or query a database — so answers rest on something checkable rather than on vibes. This is a big part of what turns a model into an agent.
- Verification. Check the important claims before you act on them (Timmy’s job). Run the code, click the citation, confirm the date. Treat unverified model output as a draft, not a fact.
A hallucination doesn’t look like an error — it looks like a good answer. There’s no red squiggle. The tone is identical whether the model is certain, guessing, or completely fabricating. For anything load-bearing — legal, medical, financial, security, or code that ships — verify against a real source. The polish of the prose tells you nothing about whether it’s true.
Temperature & sampling: the randomness knob
☺ Like you’re 10: At every step the model has a bag of possible next words, each with a size. Temperature decides how boldly it reaches into the bag. Low temperature: it almost always grabs the biggest, safest word. High temperature: it’s willing to grab a smaller, surprising one — more creative, but riskier.
If a model always picked the single most-likely next token, it would be repetitive and robotic. So instead it samples from the probability distribution over possible next tokens — and temperature is the dial that controls how adventurous that sampling is.
| Temperature | Behaviour | Good for |
|---|---|---|
| Low (≈ 0–0.3) | Focused, deterministic, repeatable. Picks the safe, high-probability tokens almost every time. | Code, extraction, classification, factual answers, anything you want reproducible |
| Medium (≈ 0.5–0.7) | Balanced — some variety without going off the rails. | General chat, drafting, explanations |
| High (≈ 0.9–1.2) | Creative, varied, surprising — and more likely to wander or hallucinate. | Brainstorming, story ideas, marketing copy, breaking out of a rut |
Two things worth internalizing. First, higher temperature does not make the model “smarter” — it makes it more random. If you want a correct, consistent answer, turn it down. Second, this is why the same prompt can give different replies: unless temperature is at (or near) zero, sampling introduces genuine variation run to run. Related knobs like top_p (nucleus sampling) trim which tokens are eligible to be sampled at all, but temperature is the one you’ll reach for most. In a chat product you rarely see the slider; in an API or SDK, it’s a parameter you set deliberately.
Ask any chat model the same creative prompt (“write a two-line poem about coffee”) three times. Notice the variation — that’s sampling at a non-zero temperature. Then ask a factual question three times (“what’s 17 × 23?”). The wording may differ but the answer shouldn’t — because there’s one high-probability continuation and the model piles almost all its confidence there.
The context window — the model’s short-term memory
☺ Like you’re 10: 🐘 Ellie can hold a lot in her trunk, but only so much at once. Everything you want the model to “remember” right now — your question, the files, the earlier chat — has to fit in that trunk. Overfill it and the oldest stuff spills out the back.
A model has no memory of you between requests. The only thing it “knows” in a given moment is the text you send it this time — and that text has to fit inside the context window, a fixed maximum number of tokens the model can attend to at once. Think of it as short-term working memory, and 🐘 Ellie the Elephant as its keeper.
The context window has to hold everything the model considers for this turn:
- The system prompt / instructions that shape its behaviour.
- The conversation so far — every earlier message you want it to remember.
- Any documents, code, or data you’ve pasted or attached.
- Its own reply, which is generated into the same budget.
When you exceed the window, something has to give: older messages get dropped or summarized, and the model quietly “forgets” them. This is why a long chat can start losing track of what you said at the top, and why pasting a giant file can crowd out your actual question. Bigger windows (hundreds of thousands, even a million-plus tokens in modern families) help — but bigger isn’t free: more context costs more tokens, and models can still lose focus on details buried in the middle of a very long context.
Two important consequences that the rest of this course builds on:
- Because the window is finite, you can’t just stuff an entire wiki or codebase in and hope. Instead you retrieve only the relevant slices at question time — that’s the motivation for RAG.
- Different models expose different window sizes, and local / self-hosted models often have smaller windows than the big hosted ones — a real constraint when you choose a model for a task.
The context window resets every request. Anything an assistant seems to “remember” across sessions — your preferences, a project’s conventions — is being re-fed into context each time (from saved notes, a memory file, or a database), not recalled from the model itself. The model is stateless; the system around it supplies continuity. That distinction becomes central when you start building agents.
So what? From model to agent
☺ Like you’re 10: A model on its own is a brain in a jar: brilliant at guessing words, but with no hands, no eyes, and a memory that resets every time. Give it hands (tools), a way to look at what happened, and a loop to keep trying — and the brain becomes a helper that can actually get things done.
Let’s zoom out. Everything above describes the model by itself: a stateless next-token predictor with a knowledge cutoff, a token budget, a randomness dial, and a tendency to sound sure even when it’s wrong. Powerful, but limited. It can’t look anything up, can’t act, can’t check its own work, and forgets you the moment the request ends.
The rest of this course is about what happens when you wrap that model in machinery to cover those gaps:
| Model’s built-in limit | What we add around it | Where you’ll learn it |
|---|---|---|
| Knowledge cutoff; can’t see your data | Retrieval — fetch real, current documents to answer from | Retrieval & RAG |
| Can’t act on the world | Tools the model can call (run code, query, search) | MCP, Building agents |
| Hallucinates; no truth signal | Grounding + a verify-before-you-trust loop | RAG, Production ops |
| Stateless; forgets between turns | Memory: re-feeding context, saved state | Building agents |
| Answers once and stops | An act → observe → reason loop | Agentic AI |
Here is the one-line bridge to remember: a model becomes an agent the moment you wrap it in a loop with tools — letting it act, see what happened, and decide what to do next, over and over, until the goal is met. Same next-token predictor at the core; a whole new class of behaviour once it can reach out and try things. Getting comfortable? Head to Prompting next to learn how to talk to the box, then Agentic AI to see it grow hands.
(1) In one sentence, what is the single thing an LLM does? (2) Roughly how many words is 1,000 tokens, and why does that number matter for cost and context? (3) Why do models hallucinate, and name two ways to reduce it. (4) You want the same answer every time you run a prompt — do you turn temperature up or down? (5) What’s the difference between the context window and long-term memory — and what actually turns a model into an agent?
Check your answers
- What an LLM does: Given some text, it predicts the most likely next token, adds it, and predicts again — writing an answer one token at a time. There is no lookup or stored ledger of facts; the “knowledge” is smeared across its weights.
- 1,000 tokens in words: Roughly 750 words, since 1 token ≈ ¾ of a word (≈ 4 characters of English). It matters because tokens are the unit everything is measured in: pricing is billed per input and output token, and context limits (e.g. “200K context”) are token budgets, not word counts.
- Why hallucinations happen & two fixes: The model produces the most likely-sounding next tokens, not verified truth, and has no built-in “I’m not sure” alarm — so in thin territory it still emits smooth, confident, but wrong text. Reduce it by grounding/retrieval (give it the real documents to answer from) and by verification (check important claims — run the code, click the citation); using tools to fetch checkable answers also helps.
- Same answer every time — temperature: Turn temperature down (toward 0). Low temperature is focused, deterministic, and repeatable; higher temperature just adds randomness, which is why the same prompt can otherwise give different replies.
- Context window vs. long-term memory (and what makes an agent): The context window is the model’s short-term working memory — a fixed token budget for this single request — and it resets every turn, so the model is stateless. Anything “remembered” across sessions is re-fed into context by the system around the model, not recalled by the model itself. A model becomes an agent the moment you wrap it in a loop with tools — letting it act, observe what happened, and decide what to do next until the goal is met.