Retrieval-Augmented Generation (RAG) with Claude

Claude's built-in knowledge is frozen at training time and knows nothing about your private documents. Retrieval-augmented generation (RAG) fixes that by finding the right slice of your own data at query time and handing it to Claude as context — with citations, so you can verify every claim.

Why retrieval instead of a bigger prompt

☺ Like you're 10: Claude's training is like a textbook it read once, years ago, and put back on the shelf — it can't flip to today's page on its own. Retrieval hands it that page.

Every Claude model answers from two sources: what it learned during training (its parametric memory) and whatever you put in the prompt (context). Parametric memory is broad but stale and generic — it has no idea about last week's support tickets, your internal wiki, or a contract legal uploaded yesterday. It's also unverifiable: when Claude answers purely from memory, there's no sentence you can point to as the source of the fact.

RAG closes that gap. Instead of asking Claude to recall facts from training, you search your own corpus for the passages most relevant to the current question and place only those passages in the prompt. Claude then reasons over content that is current, private, and traceable — and can cite exactly which passage supports each part of its answer.

The RAG pipeline, step by step

☺ Like you're 10: Think of a librarian. First they file every book by topic, once (indexing). Then, whenever someone asks a question, they don't reread the whole library — they pull just the couple of books that matter (query time).

A RAG system has two separate paths that run at different times: an offline indexing path that prepares your documents once (and re-runs whenever content changes), and an online query path that runs on every request.

Concretely: split each source document into chunks (a paragraph, a heading section, a fixed token window); embed every chunk into a vector with an embedding model; store the vectors in a vector index alongside the original chunk text. At query time, embed the incoming question with the same embedding model, compare it against the index (typically by cosine similarity), pull back the top few matching chunks, and insert them into the prompt sent to Claude — along with an instruction to answer only from those chunks and cite which one supports each claim.

Embedding chunks and queries with Voyage AI

Anthropic doesn't build its own embedding model. Its documentation names Voyage AI as its recommended embeddings partner, citing dataset size and domain fit, inference performance, and customization as reasons to pick a specific Voyage model. The lineup includes general-purpose models (voyage-4-large, voyage-4, voyage-4-lite, voyage-4-nano), domain-specific models (voyage-law-2, voyage-code-3, voyage-finance-2), a multimodal model (voyage-multimodal-3.5), contextualized-chunk models called via contextualized_embed() (voyage-context-4, voyage-context-3), and rerankers (rerank-2.5, rerank-2.5-lite).

import voyageai
import numpy as np

vo = voyageai.Client()  # reads VOYAGE_API_KEY from the environment

chunks = [
    "Revenue grew 3% year-over-year in Q2.",
    "The board approved a $50M buyback program.",
    "Headcount increased by 12% to support the new EU region.",
]

# Embed the corpus once, at indexing time. input_type="document"
# prepends a fixed instruction tuned for things you're indexing.
doc_embeddings = vo.embed(
    chunks, model="voyage-4", input_type="document"
).embeddings

# Embed each incoming query at request time with input_type="query".
query = "How much did the company spend on the buyback?"
query_embedding = vo.embed(
    [query], model="voyage-4", input_type="query"
).embeddings[0]

# Voyage embeddings are normalized to length 1, so dot product
# ranks identically to cosine similarity — no extra normalization needed.
similarities = np.dot(doc_embeddings, query_embedding)
top_k = np.argsort(similarities)[::-1][:3]
retrieved_chunks = [chunks[i] for i in top_k]

A few knobs are worth knowing before you scale this up: quantization (float, int8, uint8, binary, ubinary) trades a little accuracy for much smaller, cheaper-to-store vectors, and Matryoshka embeddings let you truncate a vector to fewer dimensions without re-embedding. The input_type parameter ("query" vs. "document") matters more than it looks — matching it on each side of the pipeline materially improves retrieval quality, because Voyage prepends a different fixed instruction for each.

Fixing ambiguous chunks: contextual retrieval

☺ Like you're 10: A chunk that just says "revenue grew 3%" is like a photo with no caption — three percent of what, whose, and when? You need the caption stapled back on before it's useful.

Naive chunking has a subtle failure mode: a chunk that says "revenue grew 3% this quarter" is nearly useless on its own — three percent compared to what, for which company, in which period? Split out of its surrounding document, the chunk that would have answered a specific query no longer carries enough signal to be retrieved for it.

Anthropic's Contextual Retrieval technique fixes this by prepending a short, Claude-generated explanation to each chunk before it's indexed — 50 to 100 tokens that situate the chunk within the full document (which company, which filing, which section) — and doing this for both the embedding index (Contextual Embeddings) and a keyword index (Contextual BM25), combining dense semantic search with sparse keyword search in a hybrid retriever.

import anthropic

client = anthropic.Anthropic()

def situate_chunk(full_document: str, chunk: str) -> str:
    """Ask Claude for a short, chunk-specific context blurb (50-100 tokens),
    then prepend it to the chunk before embedding or BM25 indexing."""
    prompt = f"""<document>{full_document}</document>
Here is a chunk from the document above:
<chunk>{chunk}</chunk>
In 1-2 sentences, give the short context needed to situate this chunk
within the overall document (e.g. the company, time period, or section
it refers to), so the chunk makes sense read on its own. Answer with
only the context, nothing else."""
    response = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}],
    )
    context = next(b.text for b in response.content if b.type == "text")
    return f"{context}\n\n{chunk}"

Because this runs once per chunk at indexing time rather than per query, a fast, inexpensive model like Claude Haiku 4.5 is a reasonable choice — you're generating a short blurb for every chunk in the corpus, and that cost adds up across a large document set. The results, measured against a baseline of embedding standalone chunks with no added context:

Each layer is additive: contextualizing the chunks helps on its own, combining it with a keyword index helps more (dense embeddings can miss exact terms like part numbers or error codes that BM25 catches), and adding a reranking pass on top of the retrieved candidates helps most.

Grounding answers with citations

☺ Like you're 10: It's like an open-book test where you don't just have to give the answer — you have to underline the exact sentence you copied it from.

Once you've retrieved the right chunks, the last step is asking Claude to show its work. Anthropic's Citations API lets Claude ground its answer directly in source documents: you pass in plain text or PDF content and enable citations on that document, Claude's platform auto-chunks it into sentences internally, and the response includes citations pointing back to the exact passages that support each claim.

{
  "model": "claude-sonnet-5",
  "max_tokens": 1000,
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "document",
          "source": { "type": "text", "media_type": "text/plain", "data": "..." },
          "citations": { "enabled": true }
        },
        { "type": "text", "text": "What is our refund window?" }
      ]
    }
  ]
}

The field that does the work is "citations": {"enabled": true} on the document block. Output tokens that simply echo a quoted source passage aren't charged, and Anthropic reports this beats hand-rolled citation implementations by up to 15% on recall accuracy in internal evals. Citations are available on the direct Anthropic API as well as through Vertex AI and Amazon Bedrock.

If you're assembling retrieved chunks yourself rather than passing whole documents through the Citations API, you can get similar behavior with plain prompting: wrap each retrieved chunk in a tagged block and instruct Claude to cite the chunk id for every claim, and to say so explicitly if the chunks don't contain the answer.

import anthropic

client = anthropic.Anthropic()

context_block = "\n\n".join(
    f"<chunk id='{i}'>{c}</chunk>" for i, c in enumerate(retrieved_chunks)
)

message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1000,
    system=(
        "Answer only using the numbered <chunk> tags in the user message. "
        "For every claim, cite the chunk id it came from, like [chunk 2]. "
        "If the chunks don't contain the answer, say so instead of guessing."
    ),
    messages=[
        {
            "role": "user",
            "content": f"<chunks>{context_block}</chunks>\n\nQuestion: {query}",
        }
    ],
)

for block in message.content:
    if block.type == "text":
        print(block.text)

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

const contextBlock = retrievedChunks
  .map((c, i) => `<chunk id='${i}'>${c}</chunk>`)
  .join("\n\n");

const message = await client.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 1000,
  system:
    "Answer only using the numbered <chunk> tags in the user message. " +
    "For every claim, cite the chunk id it came from, like [chunk 2]. " +
    "If the chunks don't contain the answer, say so instead of guessing.",
  messages: [
    {
      role: "user",
      content: `<chunks>${contextBlock}</chunks>\n\nQuestion: ${query}`,
    },
  ],
});

for (const block of message.content) {
  if (block.type === "text") {
    console.log(block.text);
  }
}

Retrieval done right vs. dumping everything in the prompt

It's tempting to skip retrieval entirely and paste your whole knowledge base into the system prompt on every call — "Claude has everything, so it can't miss anything." In practice that's the anti-pattern, and targeted retrieval of a handful of relevant chunks is the pattern.

Anthropic's own RAG cookbook walks through evaluating a pipeline the same way you'd evaluate any other production system: build a synthetic eval set of questions paired with ground-truth chunks, score retrieval on its own (precision, recall, F1, MRR) separately from end-to-end answer accuracy graded by an LLM judge, and iterate on each half independently. In their worked example, adding summary indexing and a Claude-based reranking step on top of basic retrieval lifted end-to-end accuracy from 71% to 81%. Their general guidance is to prioritize recall over precision at the retrieval stage — it's cheaper to let Claude's generation step filter out an irrelevant chunk than to have retrieval silently drop the one chunk that actually contained the answer.