RAG: Giving a Model an Open-Book Exam

Ask a general-purpose chatbot whether you can still return the lamp you bought 45 days ago, and it will answer: politely, fluently, quite possibly with a number. “Most stores allow 30 days.” It has never read your shop’s refund policy. It doesn’t even know which shop you mean. It answers anyway, because producing a plausible next token is the one thing it always does.

Part 1 showed that a model knows things in exactly two ways: what training baked into its weights, and what is in the prompt right now. Your refund policy is in neither. Training it in is slow and expensive, and the policy will change next quarter. So do what a sensible person does with a document they’ve never read: look it up, put the page on the desk, then answer. Give the model an open-book exam.

In the RAG you’ll meet in practice, that’s meant literally: there is no special channel into the model. The application finds a few relevant paragraphs and pastes them into the prompt, above the question.

Look it up, paste it in, answer

Retrieval-augmented generation, or RAGRAGRetrieval-Augmented Generation. Retrieve relevant text and add it to the prompt before generating., puts a search step in front of the model. It connects a retrieval system, usually a vector databaseVector databaseDatabase that stores embeddings with their source text and finds nearest neighbours by similarity. like the ones in Part 11, to an LLM, and every question goes through three moves:

1. The application retrieves the most relevant chunks of text: passages a few paragraphs long, cut from your documents ahead of time.
2. It inserts them into the prompt alongside the question.
3. The model writes an answer grounded in that text, built from what is in front of it rather than from what it half-remembers.

That is how a model can answer questions about documents it was never trained on: an internal wiki, a contract signed this morning, yesterday’s sales report.

Here is the worked example from my notes, with the search computed for real. A customer of Acme, a fictional lamp shop, asks: “Can I get a refund after 45 days?” Acme’s help center (a bigger cousin of Part 11’s eight-article toy, with the same 60-day refund policy) has already been cut into 22 chunks and indexed.

The LLM runs once, in step 6, after all the searching is done. Everything before it is search, and search needs no chatbot-sized model: a small, separate embedding model turned the question into a vector. In my toy that’s refund 0.58 and time 0.81 (to it, the question is more about days than refunds; remember that for Fig. 2). The refund-policy chunk is refund 0.54 and time 0.81, plus crumbs. Both vectors have length 1, so their cosine similarity (Part 11) is just matching dimensions multiplied and added: 0.58 × 0.54 + 0.81 × 0.81 = 0.31 + 0.66 = 0.97. A chunk about refund processing times came second at 0.74, and a shipping chunk third at 0.56, entirely on time. Those three were pasted in, numbered, and this block of text is the entire interface between search and model:

System: Answer using only the context below. Cite the source.
Context:
[1] Refund policy: Refunds are available within 60 days of purchase...
[2] Returns: Once we receive your return, the refund goes back to...
[3] Shipping: Orders ship within 2 business days...
Question: Can I get a refund after 45 days?

The answer, as my notes give it (the replies in these figures are written, not generated): “Yes. Refunds are available within 60 days, and 45 days is inside that window [1].”

The step that decides whether the answer can be right is step 4, the ranking. Had the refund-policy chunk come fourth, no model, however clever, could have quoted it. Most of what goes wrong with RAG goes wrong there.

Now replay it as a support agent. A staff-only chunk (“agents may approve refunds up to 90 days with a team lead’s sign-off”) joins the search, scores 0.99, and the answer grows a second sentence. Same question, same model, different answer, because a different person may see different text. Notice where that happened: in the search, before anything reached the prompt.

Why not just fine-tune the facts in?

Fine-tuningFine-tuningFurther training of a pre-trained model on a smaller, curated dataset. (Part 7) can teach a model facts too, but slowly and unreliably. RAG gives you three things it can’t, and you just watched two. The answer cited a source: a fact spread across billions of weights has no address, and a retrieved chunk does. The agent saw a chunk the customer didn’t: access control happens at retrieval time, so one shared model sees only what each user may see. And knowledge stays current without retraining, because you update the index, not the model: when the policy changes at 9:00, you re-embed one chunk, and the 9:01 answer uses it. Fine-tuning is better at changing how a model behaves (tone, format, a narrow task); RAG is better at changing what it knows right now. Plenty of systems use both.

The index is built before anyone asks

Every step in Fig. 1 started from an index that existed before the customer typed anything. Building it is the offline half of RAG, called ingestion, and it runs when documents change, not when questions arrive:

1. Load the documents: wiki pages, PDFs, help-center articles, support tickets.
2. Clean them: strip navigation, cookie banners, footers and other boilerplate. Junk that gets indexed gets retrieved.
3. Chunk them: split each document into passages, typically 200 to 1,000 tokens, with some overlap between neighbors.
4. Embed each chunk with an embedding model.
5. Store the vector together with the chunk’s text and its metadata: source document, section, last update, who may see it.

The online half uses all three: it searches by vector, pastes the text, and filters and cites with the metadata. One rule carries over from Part 11: questions must be embedded with the same model as the chunks, because vectors from different models can’t be compared. Switch embedding models and you re-embed everything.

Why chunk at all? Part 11 ended on this: a vector is a fixed-size summary. Squeeze a twenty-page policy into one vector and the sentence you need is averaged in with everything else, if it gets in at all: an embedding model reads at most a fixed number of tokens (commonly 512 to 8,192) and may simply cut off the rest. And whatever you retrieve has to fit in the context windowContext windowMaximum number of tokens a model can process in one call (input + output)., where every token is paid for. So you cut documents into pieces small enough to be about one thing, yet big enough to make sense alone. Those demands pull against each other.

Fig. 2 runs chunkingChunkingSplitting documents into passages before embedding them for RAG. on one whole help-center page. You control the chunker; the search embeds every chunk and hands the single best match to the model. A question counts as answerable only if that one chunk holds everything the answer needs. (Top 1 is stricter than Fig. 1’s top 3, and makes each cut visible. With k = 3 the other half of a split answer may come along too, but only by luck.)

First, why it fails at 12 words. The heading “Damaged on arrival” lands at the end of chunk 2, and the sentence it explains, “Email a photo within 7 days of delivery”, starts chunk 3. The search picks chunk 2, which has the heading but not the answer; chunk 3 has the answer but never says it’s about damage. Drag to 14 words and the restocking-fee table breaks the same way: the heading ends chunk 3, the rows start chunk 4, and “Furniture 10%” no longer says 10% of what.

Bigger chunks do fix it: at 16 words all five questions work. At 18, two others break. Overlap is just as fickle: at 12 words, two words of overlap rescue two questions and break the sofa question, because every boundary moved. A fixed-size chunker counts words and nothing else, so whether a given fact survives is luck.

Now switch to By section. All five questions work at every size the slider allows, because the cuts follow the document’s own structure: a section becomes a chunk, a long section is split between sentences, and every piece starts with its heading. Careful pipelines do exactly this: split on headings and paragraphs first, keep tables whole or repeat the header row, and often prefix each chunk with its document and section titles. Overlap is the blunt fallback for text with no structure.

Last experiment: in Every N words, drag the size to 80. Chunk 1 now holds all but the last two words of the page, every answer included, and the refund question still fails: the leftover “24 hours.” wins, 0.81 to 0.48. My toy exaggerates here, since it reads the question as more time than refund and “24 hours” is pure time; a real model would probably pick the big chunk. But the mechanism is real. The big chunk’s vector averages refunds, damage, fees and gift cards, so it matches none of them strongly. This is dilution: the bigger the chunk, the blurrier its vector. (Many real chunkers merge tiny leftovers like that one into a neighbor.) And even when a big chunk wins, the model gets a whole page to find one sentence, and you pay for all of it.

What Fig. 1 glossed over

Fig. 1 walked the online half in order: embed the question with the same model as the chunks, retrieve the top k (filtered by metadata or permissions), build the prompt, generate, show citations. Real pipelines often add one more step between retrieving and building the prompt, re-ranking with a cross-encoder, which Fig. 4 puts in your hands. Two of the other steps deserve a closer look, starting with the last.

Citations work by a division of labor. The model only sees text and writes “[1]”. The application remembers which chunk it pasted in as [1], metadata and all, so it can turn “[1]” into a link to refund-policy.md. The model needs no URLs; it just has to cite honestly, and the reader can check.

Filter in the search, not in the prompt

In Fig. 1 the customer never saw the staff-only chunk because the search skipped it: every chunk carries an audience field, and the query only considered chunks this user may see. The filter has to live there. Had the staff chunk reached the prompt with an instruction not to reveal it, it would already be in the model’s context, and models can be talked into repeating their context. Text that is never retrieved can’t leak. The same filters do humbler jobs too: only this customer’s orders, only this year’s documents, only the manual for the product they own.

When retrieval hands over the wrong page

The model in a RAG system is a very good reader and a terrible skeptic. It does its best with whatever lands in its prompt, and sounds equally sure either way. Fig. 3 keeps the question fixed and changes what retrieval hands over.

Missed it. The refund-policy chunk didn’t make the top three (an odd phrasing, a bad chunk), so the model got refund processing times, shipping times and a note about gift cards. It answered anyway, with a plausible “30 days”, citing a chunk that says nothing of the kind. This is the classic RAG hallucinationHallucinationConfident but false output, often caused by missing or unretrieved information.: confident, fluent, false, caused by information that was never retrieved. It happens for the reason Part 1 gives for everything: the model produces a likely continuation, and a likely answer to a refund question contains a number of days. The safer prompt says what to do when the context comes up empty, and the model says “I don’t know” instead. Better, but not a fix: models don’t always comply, and the customer still has no answer. The real fix is upstream, in retrieval.

Stale copy. Here the model did everything right. Every claim is backed by the text it was given, and the answer is still wrong, because the chunk was embedded in February and the policy changed to 60 days in June. An index is a copy of your documents, frozen when you built it, and no prompt can fix a stale copy. The fix is in ingestion: track each document’s last-modified time or content hash, re-chunk and re-embed what changed, and delete the chunks of removed pages (the step everyone forgets). It’s the same fact that keeps RAG current, seen from the other side: knowledge is exactly as fresh as the last re-index.

Poisoned. Knowledge bases ingest text other people wrote: reviews, forum posts, emails, web pages. This chunk carries instructions aimed at the model, and to the model, a retrieved chunk and the system prompt are both tokens in its context. Chat formats label them differently and train models to favor the system prompt, but that’s a learned habit, not a wall: well-written text inside a chunk can still win. This is prompt injectionPrompt injectionMalicious instructions hidden in content the model reads (web pages, documents, tool results)., and RAG is a delivery route for it: the attacker needs no access to your app, only to something it indexes, and can word the text to match likely questions so it gets retrieved. The safer prompt fences the context and says never to follow instructions inside it. That helps; it is not a guarantee, because no known prompt reliably stops injection.

Notice what the claim check separates. Faithful asks whether every claim is supported by the text the model was handed; correct asks whether the answer matches today’s policy. The stale answer is perfectly faithful and wrong: faithfulness says the model stuck to its sources, not that the sources were any good. The poisoned answer passes my word-matching check too, since “refunds ended in 2025” appears in [2], but a careful judge would fail it: [2] doesn’t claim refunds ended, it orders the model to say so, and [1] says the opposite.

Grade the search and the answer separately

Every failure in Fig. 3 had the same symptom, a wrong answer, but the fixes lived in different places: the retriever, the index, the prompt, the sources you let into the index. If all you measure is “was the final answer right?”, you can’t tell which to fix. So you evaluate the two stages separately:

The top-left cell has a blind spot, and Fig. 3’s stale copy sits in it: right chunk, faithful answer, still wrong. The grid checks answers against their context, not against the truth. Catching a context that is itself wrong takes answers checked against known-good ones (answer quality, below) and a watch on how old your chunks are.

For retrieval you need a small test set: real questions, each labeled with the chunk that answers it. Then you measure recall@k: for what fraction of questions is the right chunk somewhere in the top k?

This assumes one right chunk per question. When an answer needs several, recall@k is usually the fraction of them in the top k, averaged over questions.

Fig. 4 ranks ten labeled questions with the same toy retriever as Fig. 1. At k = 3, nine of the ten hit. How far do you have to raise k to catch the tenth, and what does that cost every other question? Try it before reading on.

You need k = 8. The straggler is “Can I return a lamp if I threw the box away?”, whose answer is a side remark in the refund-window chunk (“in their original packaging”). My toy knows neither box nor packaging, where a real model would link them, and the chunk is mostly about something else, so it ranks eighth. (Real models miss subtler side remarks the same way.) Going from k = 3 to 8 lifts recall from 90% to 100%, and the retrieved context from 1,200 to 3,200 tokens (at an illustrative 400 per chunk) for every question, including the nine that were already fine. This is the too much context pitfall, in three parts:

• Cost. Every chunk is billed input on every question. If the app keeps retrieved chunks in the chat history, they’re re-sent and billed again on every later turn (Part 1).
• Noise. At k = 8, at most one chunk in eight is the answer. The other seven are near-misses that look relevant, exactly the kind of text that leads a model astray.
• Lost in the middle. Models tend to use material at the start and end of a long context more reliably than material in the middle, and the more chunks you send, the more of them sit in the middle.

A second, sharper look: re-ranking

Vector search compares two summaries made separately: each chunk was squeezed into one vector at ingestion, long before the question existed. That’s why it’s fast (the vectors are ready, and an approximate-nearest-neighbor index finds almost the closest among millions in milliseconds), and also why it’s blurry. A cross-encoder is a model that reads the question and one chunk together, as a single input, and outputs one relevance score. Seeing both at once, it can notice that a chunk says “E-104” rather than “E-207”, or that “not covered” answers “is it covered?”. But it must run once per question-chunk pair, so you can’t point it at the whole index. Hence the standard two stages: retrieve wide with vectors, re-rank those candidates, keep the best few.

Before you flip Re-rank in Fig. 4: can a sharper scorer ever make a question worse?

With re-ranking on, recall@1 jumps from 60% to 80%, and every question is in by k = 5 instead of 8. Take E-104, and a confession about my toys. My embedding has no dimension for codes, so it can’t see “E-104” at all; the E-207 chunk has more error-words, so it wins, 0.98 to 0.96. (A real embedding sees the code but blurs it with every other code: Part 11’s E-4012 problem.) My re-ranker isn’t a transformer: it matches question words against chunk words, weighting exact matches and rare words most, then blends that with the vector score. That makes it closer to the keyword half of hybrid search (Part 11), the usual fix for error codes, SKUs and names, than to a real cross-encoder, which catches far more than exact strings. And yes, a sharper scorer can make a question worse: here it drops “Delivery to Canada” from second to third, so recall@2 falls from 90% to 80%, at the price of ten extra question-chunk reads per question. Real cross-encoders are far better than my toy, but “usually better” is the honest summary for them too. So you measure.

Then grade the answers

For generation, you score the answers to the same questions for faithfulness (is every claim supported by the retrieved text?) and answer quality (does it answer the question, correctly and completely?). Fig. 3’s claim check is the crude version, plain string matching. In practice a judge model checks each answer against its context claim by claim, and people spot-check the judge, because judge models make mistakes too. Put the two report cards side by side and a wrong answer tells you where to look: low recall, at search; high recall but unfaithful answers, at the prompt or the model; high recall and faithful answers that are still wrong, at the chunks themselves, which are stale, wrong or planted.

Retrieval as a tool the model calls

Everything so far retrieves on every message, automatically. Agents often turn that around: “search the knowledge base” becomes a tool the model decides to call, through tool callingTool callingThe model outputs structured JSON asking the application to run a named tool with arguments., which Part 13 covers next. The model is told the tool exists:

{
  "name": "search_knowledge_base",
  "description": "Search Acme's help center. Returns the most relevant passages with their sources.",
  "parameters": {
    "type": "object",
    "properties": { "query": { "type": "string" } },
    "required": ["query"]
  }
}

When it needs facts, it replies with a request to run that tool instead of an answer; the app runs the same retrieval pipeline and hands back the chunks, and the model continues. That buys three things: no search for “thanks!”, a query the model can rewrite to be better than the user’s raw message, and a second search when the first one misses (the student finally gets to ask for more pages). It costs extra model calls, and adds a new failure: the model can decide not to search when it should, and answer from memory.

In the agent version, retrieval became a tool the model chooses to call. That raises an odd question: how does a model that only ever produces text “call” anything? The next part answers it.