Context Engineering

A model doesn’t “remember” a conversation or “know” your docs — on every single call, it sees exactly one thing: the block of text you hand it, the context window. Context engineering is the craft of assembling that window deliberately, under a token budget: deciding what goes in, in what order, and what to cut when it won’t all fit. It’s the difference between an app that answers from the right facts and one that drowns the model in noise.

The window is a budget

☺ Like you’re 10: The index card has edges. Everything you want to tell your friend has to share that one card — the rules, the clues, the story so far, and the question. If you fill it with junk, there’s no room left for the part that matters.

Every model call has a hard limit on how much text it can take in at once, measured in tokens (roughly, chunks of a few characters — see How models work). That limit is the context window, and it is a fixed budget you spend on every request. It doesn’t matter that the model “saw” something three messages ago — if it isn’t in the window this call, the model cannot use it. There is no hidden memory; there is only what you packed.

Here’s the catch: a lot of different things all want a seat in that same window, and they compete for the space:

• System prompt. The standing instructions — who the assistant is, its rules, its output format. Present on every call.
• Retrieved chunks. The documents your retriever pulled in for this question (see Retrieval engineering). Often the biggest single consumer.
• Memory. Facts carried across sessions — the user’s name, preferences, prior decisions (the types of memory are covered in Agent memory).
• Tool results. Output the model gets back after calling a tool — an API response, a database row, a search result. These can be huge and arrive mid-conversation.
• Chat history. The earlier turns of this conversation, which grow with every exchange.
• The user turn. The actual question being asked right now — the one thing that must always be in.

Add these up naively and you blow the budget fast — a long history plus a fat tool result plus five retrieved chunks can exceed the window before the user has even finished typing. And even when it all technically fits, a stuffed window costs more money and more latency on every call (see Cost & latency ops) and, as you’ll see below, actually makes the model worse at using what’s there. So the budget isn’t just a ceiling you occasionally hit — it’s a constraint you design around from the start.

Assembling the context

☺ Like you’re 10: Writing one great card by hand is a nice trick. But a real app writes a fresh card every single turn, automatically — grabbing the right clues, trimming them to fit, and laying them out in the same tidy order every time. That little card-writer is the real machine.

Writing one prompt carefully is prompting — a human craft you do by hand. Context engineering is different: it’s the runtime code that builds the window on every call, from moving parts. Think of it as a little scheduler that, for each request, decides what to include, trims each piece to size, fills it into a template, and orders it — all under the token budget. The user never sees this; it happens between “user hits send” and “model receives text.”

A context assembler does three jobs on every turn:

1. Select. Decide what gets in — which memories are relevant, how many retrieved chunks, how much history — and drop the rest. This is where the budget is enforced.
2. Template. Wrap each piece in a consistent, labelled structure so the model can tell instructions from evidence from history. (This is where prompt templates live — more below.)
3. Order. Arrange the pieces in a deliberate sequence, because — as the next section shows — where something sits in the window changes how well the model uses it.

In code, the output of all this is best thought of as a context object: a structured value you build up, measure, and then flatten into the final prompt. Building it explicitly (rather than string-concatenating as you go) is what lets you enforce the budget and reorder pieces safely:

# PSEUDOCODE — a context assembler run once per turn
BUDGET = 8000                      # tokens we're allowed to spend

ctx = ContextObject()
ctx.add("system",  system_prompt,          priority=1)   # never dropped
ctx.add("memory",  relevant_memories(user), priority=2)   # facts across sessions
ctx.add("docs",    retrieve(user_question, k=4), priority=3)   # evidence
ctx.add("history", recent_turns(conversation),  priority=4)   # trimmable
ctx.add("user",    user_question,           priority=1)   # never dropped

# enforce the budget: drop / shrink lowest-priority pieces until it fits
while ctx.token_count() > BUDGET:
    ctx.trim_lowest_priority()     # e.g. summarize old history, cut a chunk

prompt = ctx.render(order=["system", "memory", "docs", "history", "user"])
answer = model(prompt)

Notice what this is not: it’s not the memory taxonomy from Agent memory (episodic vs. semantic vs. procedural — that page is about what kinds of memory exist). Context engineering is the layer that consumes memory and retrieval as inputs and packs the final window. Memory decides what could be remembered; context engineering decides what actually makes it onto today’s card.

Order matters

☺ Like you’re 10: When someone reads you a long list, you remember the first few things and the last few things — the stuff in the middle turns to mush. Models do the exact same thing. So put the important instructions and clues where they’ll actually be noticed: near the top or near the bottom, not buried in the middle.

You might assume a model reads its whole window with equal attention. It doesn’t. A well-documented effect called “lost in the middle” shows that models are best at using information placed at the beginning or end of a long context, and noticeably worse at using facts buried in the middle. A crucial retrieved chunk sitting in the exact center of a big window can be effectively ignored even though it’s right there.

That turns ordering from a cosmetic choice into an engineering decision. Practical placement rules that follow from it:

• Instructions go at the edges. Put your core system instructions up top, and it’s often worth restating the key ask (or the output format) at the very bottom, right before the model answers — the two positions it attends to most.
• Put the most important evidence first or last. If your retriever ranks chunks, don’t dump them middle-out; place the top-ranked chunk where it won’t get lost.
• Don’t bury the user’s actual question. Keep the live user turn at the end, after the evidence, so the model reads the question with the facts fresh.
• Shorter is safer. The “middle” only gets big when the window is big. Trimming the context (next section) shrinks the danger zone.

Compaction

☺ Like you’re 10: A conversation keeps growing, but the card stays the same size. So instead of carrying every word you’ve ever said, you scribble a short summary — “we agreed on the blue plan, user’s name is Sam” — and toss the long transcript. You keep the plot, drop the padding.

Conversations and tool outputs grow without limit; the window doesn’t. Compaction is the set of techniques for shrinking the context back under budget without losing the thread. The main moves:

• Summarize history. When the chat history gets long, replace the oldest turns with a compact running summary (“Earlier: user is booking a trip to Lisbon in June, budget €1500, prefers direct flights.”). You keep the durable facts and decisions and drop the verbatim back-and-forth.
• Prune tool output. A tool might return a 5,000-token JSON blob when the model only needs three fields. Extract or truncate before it enters the window — never paste raw dumps in wholesale.
• Drop the stale. Retrieved chunks or intermediate reasoning that a later turn made irrelevant can simply be removed rather than carried forever.
• Roll a rolling window. Keep the last N turns verbatim (recency matters) plus one summary of everything older — a common, robust default.

The tension to manage: summarizing is itself lossy. Compact too aggressively and you drop the one detail a later turn needed; compact too little and you blow the budget or hit “lost in the middle.” Good systems summarize the old and keep the recent verbatim, and they test what breaks (this is exactly where evals earn their keep — you measure whether compaction quietly degraded answers).

# PSEUDOCODE — compaction to stay in budget
KEEP_RECENT = 6                       # last 6 turns stay word-for-word

recent = history[-KEEP_RECENT:]
older  = history[:-KEEP_RECENT]

if older:
    summary = model("Summarize the durable facts and decisions:\n" + older)
    history = [summary] + recent      # one summary + recent verbatim

for result in tool_results:           # never paste raw dumps
    result.text = extract_relevant_fields(result.text)

Prompt caching in one paragraph

☺ Like you’re 10: If the first half of your card is the same every single time — the same rules, the same instructions — you shouldn’t rewrite it from scratch each turn. You keep a copy ready and just add the new bit. Faster and cheaper.

A lot of your window is stable across calls — the system prompt, tool definitions, maybe a fixed set of reference docs — while only the tail (the latest user turn) changes. Prompt caching lets a provider reuse the work already done on that stable prefix instead of reprocessing it every call, which cuts both cost and latency substantially when the prefix is large and reused often. The catch that ties it straight back to this page: caching only helps if the prefix is byte-for-byte identical each time, so you must put the stable stuff first and the changing stuff last — the same ordering discipline you already want. Different providers expose caching differently (some automatic, some you mark explicitly), so treat the mechanics as provider-specific; the design rule is universal. See Cost & latency ops for when the savings are worth it.

Prompts as versioned software

☺ Like you’re 10: The rules on your card aren’t random scribbles — they’re a real thing you save, label “version 3,” and can go back to if version 4 makes your friend answer worse. Prompts deserve the same care as code.

The templates your assembler fills in aren’t throwaway strings — they’re software, and they should be managed like it. This discipline is often called prompt ops. The core practices:

• Templates, not hardcoded text. Keep your prompts as named templates with slots ({{system}}, {{docs}}, {{user}}) rather than strings smeared through your app code. One place to read, one place to change.
• Version them. Give each prompt a version and store it in source control. When behavior changes, you can see which prompt change caused it — and diff it like any commit.
• A/B test changes. Don’t swap a production prompt on a hunch. Run the old and new versions side by side against your eval set (and, carefully, real traffic) and keep the one that scores better.
• Be able to roll back. If v4 regresses in production, you want to revert to v3 instantly — which is only possible if v3 is a saved, addressable artifact, not code you already overwrote.

The payoff is that prompt changes stop being scary. A prompt tweak is a code change with a diff, a test run, a deploy, and a rollback path — exactly the workflow that makes any other part of your system safe to touch.

Pitfalls

☺ Like you’re 10: Three ways the card goes wrong: you cram it so full the important bit gets lost, you leave old crossed-out notes on it that aren’t true anymore, or someone quietly rewrites the rules and nobody notices your friend started answering differently.

Context engineering has its own set of classic failures — and they’re sneaky because the app usually still runs, it just answers worse:

• Context stuffing / dilution. The instinct “more context = better answers” is wrong past a point. Cramming in every possibly-relevant chunk buries the actually-relevant one, worsens “lost in the middle,” and raises cost and latency — often while lowering quality. More is not better; relevant is better.
• Stale context. Carrying old summaries, outdated retrieved docs, or superseded memory forward means the model confidently answers from information that’s no longer true — worse than a gap, because it looks authoritative. Compaction and memory both need a freshness story.
• Silent prompt drift. Someone edits the template, or a summarizer’s behavior shifts, and outputs quietly change with no version bump and no test — so a regression ships and nobody can point to what caused it. This is exactly what versioning and evals are for.

The through-line: because the model only ever sees the window, every one of these is invisible until you look at the assembled context itself. Log the final prompt, watch its token count, and evaluate changes. The Cost & latency ops page goes deeper on measuring the budget, and Evals on catching quality regressions before users do.