Retrieval Engineering

The Retrieval & RAG lesson taught you the cartoon: embed your docs, drop them in a vector store, grab the top-k nearest chunks. That’s enough to demo — and nowhere near enough to ship. This page turns the cartoon into a production system: chunk deliberately, blend keyword and vector search, rerank for precision, rewrite the query, and — the part everyone skips — measure retrieval on its own, before you ever blame the model. You won’t train anything; you’ll wire together hosted embedding and rerank APIs and get the plumbing right.

Recap & the gap

☺ Like you’re 10: Grabbing “the three most similar cards” sounds smart, but similar-looking isn’t the same as correct. A card that rhymes with your question can beat the card that actually answers it — so the librarian confidently hands you the wrong thing.

Recall the RAG pipeline: chunk the docs, embed each chunk, store the vectors, and at question time embed the query and return the top-k nearest chunks by cosine similarity. That single pipeline is where Retrieval & RAG stopped. It’s the right mental model, but naive top-k semantic search quietly underperforms the moment real users and real documents show up:

• Semantic search misses exact matches. Embeddings are great at meaning and bad at literals. A part number like SKU-4471-B, a person’s name, an error code, or an acronym can score lower than a fluffy paragraph that merely sounds on-topic. Meaning-only search has no special respect for “this exact string.”
• Top-k similarity ≠ top-k usefulness. The embedding score ranks by overall resemblance, not by whether a chunk contains the answer. The chunk that truly answers the question is often sitting at rank 8, below seven near-misses.
• The user’s words aren’t the document’s words. People ask short, messy, pronoun-laden questions (“does it cover that?”). Documents are written formally. One embedding of a vague query rarely lands where the answer lives.
• You can’t tell if retrieval or generation failed. When the final answer is wrong, was it because the right chunk was never retrieved, or because the model fumbled a chunk it did get? If you only ever look at the final answer, you can’t know — and you’ll “fix” the wrong half.

Retrieval engineering is the applied craft of closing each of these gaps with off-the-shelf, hosted components — better chunking, hybrid search, a reranker, query transformation — and then proving it worked with retrieval-specific metrics. None of it requires training a model.

Chunking, done properly

☺ Like you’re 10: If you tear a book into cards, where you tear matters. Rip a sentence in half and neither piece makes sense. Make one card hold ten pages and the good bit gets buried. Cut along the natural seams — chapters, headings — and write a little label on each card so you can find it later.

A chunk is the unit that gets embedded, retrieved, and shown to the model. Get chunking wrong and everything downstream inherits the damage — most “our RAG is bad” complaints trace back here. Three levers:

• Size. Too large and a single chunk mixes several topics, so its embedding is a muddy average and the relevant sentence drowns. Too small and a fact gets split across chunks, so no single chunk holds the whole answer. A common working range is a few hundred tokens per chunk — big enough to be self-contained, small enough to be about one thing.
• Overlap. Chunks that share a little text at their boundaries (say, 10–20%) keep a sentence straddling a boundary from being lost by both neighbours. Overlap costs a bit of storage and duplication in exchange for not slicing an answer clean in two.
• Structure-aware splitting. Don’t split on a blind character count. Split on the document’s own structure — Markdown headings, HTML sections, PDF pages, code function boundaries — so each chunk is a coherent thought instead of an arbitrary window. A heading-aware splitter that never crosses an h2 boundary beats a fixed 500-character cut almost every time.

Just as important as the text is the metadata you attach to each chunk: source URL, title, section heading, author, last-updated date, product line, access level. Metadata is what lets you filter before you rank — “only chunks from the current product version,” “only docs this user is allowed to see.” Filtering shrinks the haystack and kills a whole class of wrong-but-similar hits.

# INDEX ONCE — chunk on structure, carry metadata (neutral pseudocode)
for doc in corpus:
    for section in split_on_headings(doc):          # structure-aware, not blind count
        for chunk in window(section, size=350, overlap=60):   # size + overlap in tokens
            index.add(
                vector   = embed(chunk.text),        # hosted embedding API
                text     = chunk.text,
                metadata = {                          # filterable fields
                    "source":  doc.url,
                    "section": section.heading,
                    "product": doc.product,
                    "updated": doc.updated_at,
                },
            )

Hybrid search

☺ Like you’re 10: Two librarians with different superpowers. One finds cards that mean the same thing even with different words. The other finds cards with the exact words you said, like a product code. Ask both, then merge their piles — you catch what either one alone would miss.

Vector search is strong on meaning and weak on literals. Old-fashioned keyword search is the mirror image: it nails exact terms — names, codes, rare acronyms — but is blind to paraphrase. The production answer is to run both and combine them. That’s hybrid search.

The keyword side is usually BM25, a decades-old, battle-tested scoring function that ranks documents by how often the query’s terms appear, weighted so that rare words count more and long documents don’t get an unfair boost. It ships in virtually every search engine and needs no model at all. So you get two ranked lists for the same query:

• Dense (vector) list. Ranked by embedding similarity — good for “means the same thing.”
• Sparse (BM25) list. Ranked by keyword overlap — good for “contains this exact term.”

Now you must fuse them into one list. You can’t just add the scores — a cosine similarity of 0.82 and a BM25 score of 14.6 live on totally different scales. The clean, popular trick is Reciprocal Rank Fusion (RRF), which throws away the raw scores and uses only each item’s rank in each list. An item’s fused score is the sum, across lists, of 1 / (k + rank) (a small constant k, often around 60, keeps the top ranks from dominating). Anything that ranks high in either list floats up; anything high in both floats highest.

# RETRIEVE — run both searches, fuse by rank (neutral pseudocode)
dense  = index.vector_search(embed(query), k=50)   # list of chunk ids, best-first
sparse = index.bm25_search(query,          k=50)   # keyword list, best-first

def rrf(lists, k=60):
    score = {}
    for lst in lists:
        for rank, doc_id in enumerate(lst):          # rank 0 = best
            score[doc_id] = score.get(doc_id, 0) + 1 / (k + rank)
    return sorted(score, key=score.get, reverse=True)

fused = rrf([dense, sparse])[:20]                    # one merged shortlist

Hybrid search is close to a free lunch: BM25 is cheap, the fusion is a few lines, and you recover exactly the exact-match cases where pure vectors fail. It’s the single highest-leverage upgrade over naive top-k for most real corpora.

Reranking

☺ Like you’re 10: First you grab a big stack of maybe-good cards, fast. Then a careful checker reads each card next to your actual question and re-sorts them so the truly-right ones are on top. Slower, but only on the small stack — and way more accurate.

Fusion gives you a solid shortlist, but it still ranks by cheap signals. To squeeze out real precision, add a reranking stage. The retriever’s job is recall — cast a wide net, get 20–50 candidates, don’t miss the answer. The reranker’s job is precision — take that shortlist and put the genuinely best few on top.

The magic is in how a reranker scores. Your embeddings are a bi-encoder: the query and each document are embedded separately, then compared with a dot product. Fast (you can pre-compute all the document vectors), but the two never actually “see” each other. A cross-encoder reranker instead feeds the query and one candidate together into a model and outputs a single relevance score for that exact pair. Because it reads them jointly, it catches subtle relevance a bi-encoder misses — but it’s far more expensive, so you can only run it on a shortlist, never the whole corpus.

So the production shape is a funnel: hybrid retrieval fetches ~50 candidates, a hosted rerank API scores each one against the query, and you keep the top handful to hand the model. You get the reranker’s accuracy without paying its cost across everything.

# RERANK the shortlist with a hosted cross-encoder (neutral pseudocode)
candidates = fused[:50]                               # from hybrid + RRF
scored = rerank(query=query, documents=[c.text for c in candidates])
top    = [c for c, _ in sort_by_score(candidates, scored)][:5]   # keep the best 5
context = "\n\n".join(c.text for c in top)            # → into the prompt

Query transformation

☺ Like you’re 10: Sometimes the problem isn’t the library — it’s how you asked. “Does it cover that?” is too vague to find anything. So before searching, you fix up the question: spell it out, ask it a few different ways, or even guess what a perfect answer would look like and go hunting for cards like that.

Everything so far assumed the query is fine and the retriever is the weak link. Often it’s the opposite: a terse, ambiguous, or context-dependent question that no retriever can serve well. Query transformation uses a cheap LLM call to reshape the query before it hits the index. The main moves:

• Query rewriting. Turn a messy, conversational question into a clean, standalone search query — resolving pronouns from the chat history (“does it cover that?” → “does the premium plan cover international shipping?”). Essential in multi-turn chat, where the real question is spread across several messages.
• Query expansion. Add synonyms and related terms the documents might actually use (“car” → also “vehicle, automobile”), so keyword search stops missing paraphrases.
• Multi-query. Generate several different phrasings of the same question, retrieve for each, and fuse the results (RRF again). Different phrasings hit different chunks; the union has far better recall than any single query.
• HyDE (Hypothetical Document Embeddings). A neat inversion: instead of embedding the question, ask an LLM to draft a hypothetical answer, then embed and search with that. Since answers look more like documents than questions do, the search vector lands closer to real answer-chunks. The draft can be wrong in its facts — you’re only using its shape to steer retrieval.

# QUERY TRANSFORMATION before retrieval (neutral pseudocode)
clean   = llm(f"Rewrite as a standalone search query, using this chat:\n{history}\nUser: {query}")
variants = llm(f"Give 3 alternative phrasings of: {clean}")   # multi-query
hyde    = llm(f"Write a short passage that would answer: {clean}")  # HyDE draft

results = []
for q in [clean] + variants:
    results.append(index.hybrid_search(q, k=30))
results.append(index.vector_search(embed(hyde), k=30))         # search by hypothetical answer
shortlist = rrf(results)[:50]                                  # fuse everything, then rerank

These aren’t all-or-nothing — start with rewriting (cheap, high-impact for chat) and add multi-query or HyDE only if evals show retrieval is still missing answers. Which brings us to the part nobody wants to do.

Measuring retrieval separately from generation

☺ Like you’re 10: If the whole answer is wrong, you need to know whose fault it was: did the librarian bring the wrong cards, or did the writer mess up cards that were right? So you grade them separately — one score for “did the right card get pulled,” another for “did the answer stick to the cards.”

Here is the core discipline of this entire lesson. A RAG system has two stages that can each fail — retrieval and generation — and if you only ever look at the final answer (an “end-to-end” eval), a low score can’t tell you which stage broke. So you evaluate the retriever on its own. Build a small test set of (question → the chunk(s) that truly contain the answer), run your retriever, and score how well it found them:

• Recall@k. Of the questions where a relevant chunk exists, for what fraction did it appear somewhere in the top-k? This is the make-or-break number: if the answer isn’t retrieved at all, no model can save you. Tune retrieval to maximize recall@k first.
• MRR (Mean Reciprocal Rank). How high up was the first relevant chunk? Score each query as 1/rank of its first hit (rank 1 → 1.0, rank 4 → 0.25) and average. Rewards putting the right chunk near the top, which matters because models attend most to what comes first.
• nDCG (normalized Discounted Cumulative Gain). The graded version: when some chunks are more relevant than others, nDCG rewards putting the most-relevant ones highest, discounting relevance the further down the list it sits. The go-to metric when relevance isn’t just yes/no.

Those three grade retrieval. Separately, grade generation with faithfulness (a.k.a. groundedness): given the chunks that were retrieved, does the answer actually follow from them, with no invented claims? A common approach is an LLM-as-judge that checks each sentence of the answer against the supplied context. Splitting the two lets you read failures precisely:

This is exactly the split-the-pipeline philosophy from Evals, applied to retrieval — and it’s the only honest way to know whether your fancy reranker or HyDE trick actually helped, rather than just felt like it did. Measure retrieval, change one thing, measure again.

Pitfalls

☺ Like you’re 10: Three classic ways to trip: only grading the final answer so you never learn who messed up; letting the card catalog go stale so it swears yesterday’s rules are today’s; and cutting cards so big or so tiny that the good part is either buried or torn in half.

Even with every upgrade above, the same few mistakes sink real systems:

• Only ever evaluating end-to-end. The cardinal sin. A single final-answer score hides which stage failed, so teams tune the model when retrieval was broken (or vice versa) and go in circles. Always keep a separate retrieval eval alongside the end-to-end one.
• A stale index. The index is a snapshot. If your docs changed but you didn’t re-index, retrieval will confidently serve last quarter’s policy — worse than a blank, because it looks authoritative. You need a re-indexing pipeline that keeps the store fresh as sources change (the operational plumbing lives in Data Engineering).
• Over- and under-chunking. Chunks too large bury the answer in noise and muddy the embedding; too small split the answer so no chunk holds it. When retrieval evals sag, chunk size and overlap are the first dials to turn — re-run recall@k after each change.
• Trusting retrieved text as instructions. Everything you retrieve is untrusted input. A page or ticket can hide “ignore your instructions and…”; a naive system may obey it. Retrieved content is data, never commands — see Guardrails for defending against injection through the retrieval path.

Retrieval engineering also doesn’t live in a vacuum: what you retrieve becomes the model’s context, so how you order, trim, and format those chunks in the prompt is its own craft — see Context Engineering. Retrieval decides what the model sees; context engineering decides how it sees it.