Retrieval-Augmented Generation

A model may need information that was absent or out of date when it was trained. Retrieval supplies relevant source material as context, giving the model evidence it can use when answering a question.

14 sections · 10 questions · 3 coding challenges

1 / 14

The arrow keys move between sections too.

Could the Model Consult a Reference Before Answering?

BOTIE

A language model's weights are an awkward place to keep every fact we might later need. Updating a document does not automatically update the model, and an answer drawn only from its parameters may provide no clear reference for checking where the information came from.

Lewis and colleagues' 2020 RAG work combined a pretrained generator with retrieval from an external document index. It addressed knowledge-intensive tasks by allowing the model to condition generation on retrieved passages as well as its learned parameters.

A retriever first finds material related to the request. The generator then uses that material as part of its context when producing an answer. The documents can be maintained separately from the model, but this creates several things to check: whether retrieval found the right material, whether the material supports the answer, and whether the generator used it faithfully. Access to a reference does not guarantee a correct reading of it.

Sources for this history

Lewis and colleagues (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

What If the Answer Lives in a Document the Model Needs to Read?

Suppose someone asks an assistant when a library opens on Saturday. The answer lives in the library's current schedule, but we cannot assume it is correctly stored in the model's parameters. We need to give the model relevant information when answering.

Retrieval-augmented generation, or RAG, separates finding that information from writing an answer. A retriever selects useful passages from a collection. Those passages accompany the question as context for a language model.

Step 1 of 4

Keep the source information outside the model

Store the schedule and related material as passages with identifiable sources. The passages can be updated without retraining the language model. We still need a way to locate the passage relevant to a particular question.

Source 1: Weekday opening hours
Source 2: Saturday opening hours
Source 3: How to renew books

An invented library collection.

Step 2 of 4

Search for passages that might answer the question

A retriever compares the question with the collection and ranks candidate passages. Our small example uses shared words. More capable retrieval systems can use learned text vectors, but a high similarity score still does not prove that a passage contains the answer.

Question terms: library
Saturday
Candidate passages: Weekday hours
Highlighted: Saturday hours
Renewals

For a Saturday question, the Saturday passage is the desired evidence.

Step 3 of 4

Put the question and evidence into the model's context

The language model receives both the question and the selected passages. It can use those passages while composing the answer. Retrieving information changes the input context; it does not by itself update the model's weights.

1
Prompt
- Instructions
- Question
- Source passages
2
Language model
Generate an answer

The retrieval result becomes part of the input.

Step 4 of 4

Check whether the answer is actually supported

A fluent answer can still misread a passage, omit a qualification, or cite irrelevant evidence. Check that the sources support the answer. If retrieval found no useful passage, the system needs an explicit missing-evidence behavior rather than an invented fact.

Retrieval check: Did we find the needed evidence?
Answer check: Does the response agree with that evidence?

Two separate checks are needed.

Inspect retrieval before adding fluent generation

Search the small collection and read the passages selected for the model. This activity ends at retrieval, where we can check whether the supplied evidence would help answer the question.

Try Saturday, renew, and a word absent from the collection. The results expose both the useful behavior and the limits of exact word overlap.

Part 1. Keep Retrieval and Answering Separate

1. Prepare passages with recoverable sources

Suppose we ask for the library's Saturday hours. A model's parameters may reflect an old schedule or no reliable schedule at all. A confident answer would not establish that it is current. We can instead retrieve the relevant passage from the library's information and include it with the question. That gives the model evidence to work from, which we still need to check it uses correctly.

First, retrieval selects passages that may answer the question. Then generation uses the question and those passages to write a response. Keeping track of each stage helps us investigate mistakes. Did the search miss the right passage, or did the model misread information it was given? We need different fixes for those failures.

The collection on this page is three invented passages, each with a source number.

Source	Passage
1	The library opens at nine on weekdays. It closes at six.
2	The library opens at ten on Saturday. It closes at four.
3	Borrowed books may be renewed online using your library account.

The whole collection behind the live example.

A collection needs passages that are small enough to retrieve and include in a prompt, but large enough to preserve the information an answer needs. A sentence split away from its heading or qualifying paragraph can become misleading.

Source 2 shows what is at stake. Its second sentence says that it closes at four. Kept beside the first sentence, that is the Saturday closing time. Stored as a passage of its own, it no longer says what closes or on which day. It also shares no word with a question such as “When does the library close on Saturday?”, so a retriever that matches words would never return it. The last practice problem makes that split and runs the question.

Keep the original source and location alongside each passage. A generated citation is useful only if it points to the material actually used and that material supports the claim. Updating the collection changes available evidence without necessarily changing model parameters.

The source number is what makes an answer checkable. An answer that says ten and cites Source 2 can be held against the sentence in Source 2. And because the schedule lives in the collection, a change of hours is an edit to one passage. No parameter of the model has to change.

2. Retrieve candidate evidence

The retriever ranks passages for the question. Exact term matching works when question and source use similar words. Learned embeddings can compare broader semantic relationships. Some systems combine lexical and vector retrieval or rerank initial candidates.

Ask the live collection the full question, “When does the library open on Saturday?”, and it scores all three passages. Part 2 shows how the scores are calculated.

Source	What it is about	Score
2	Saturday hours	0.555
1	Weekday hours	0.416
3	Renewing books	0.158

Word-overlap scores of the three passages for the question about Saturday opening.

Similarity is a selection signal, not proof of relevance. A passage about weekday hours can share many words with a Saturday question and still be the wrong evidence. Read the content, especially dates, conditions, and exceptions.

The table is that warning in numbers. The right passage is ranked first, and the weekday passage is not far behind it at 0.416, because it shares the words the, library and on with the question. One word, Saturday, separates the passage that answers this question from the passage that answers a different one. A retriever asked for two passages would hand over both, and it would then be up to the model to notice the difference in days.

Part 2. Calculate a Small Retrieval Baseline

3. Represent text with word counts

The example case-folds text, extracts word-like pieces, and builds a vocabulary from three invented passages. Each text becomes a count vector. The query is compared against passage vectors using cosine similarity.

Each of those terms is a small idea. Case-folding treats capital and small letters as the same, so Saturday and saturday are one word. The vocabulary is the list of every different word in the passages, which comes to 22 words for this collection. A count vector has one entry for each vocabulary word, holding how many times that word occurs in the text. Cosine similarity measures how nearly two such vectors point the same way. It multiplies matching entries and adds them up, then divides by the lengths of the two vectors, so that a passage is not favored merely for holding more words.

For an illustrative two-word vocabulary, a query can exactly match one direction while partially matching another.

Vocabulary = [library, Saturday]
Query q = [1, 1]
Passage A = [1, 1]
Passage B = [1, 0]

cos(q, A) = (1 × 1 + 1 × 1) / (√2 × √2) = 1
cos(q, B) = (1 × 1 + 1 × 0) / (√2 × 1) = 1 / √2

The live collection contains more words, so its scores differ from these two-coordinate illustrations. Only positive matches are returned. If the query has no words in the indexed vocabulary, its vector has no length and the retriever returns no passages.

Here is the same calculation on the live collection, for the query the example opens with, library Saturday. The query vector has a one at library and a one at saturday, so its length is √2. Sources 1 and 2 each hold eleven words with the word at appearing twice, which gives each a squared length of 13. Source 3 holds ten different words, for a squared length of 10. The numerator of each score counts the shared words.

Source 2 shares library and saturday:  2 / (√2 × √13) ≈ 0.392
Source 3 shares library:               1 / (√2 × √10) ≈ 0.224
Source 1 shares library:               1 / (√2 × √13) ≈ 0.196

The Saturday passage is first, as hoped. Second place is the surprise. The passage about renewing books outranks the passage about weekday hours, although neither mentions Saturday and each shares only the word library with the query. It ranks higher because it is shorter, and cosine divides by length. At the example’s default of two passages, the context handed on would be the Saturday hours and the renewal instructions. Nothing in that second passage bears on the question, and its score had no way to say so.

Exact matching has a second limit, and the other searches the example suggests run into it.

Query	What comes back
library Saturday	Source 2 at 0.392, then Source 3 at 0.224
renewed	Source 3 at 0.316
renew	nothing
elephant	nothing

Four searches of the live collection, each asking for up to two passages.

Elephant returns nothing because no passage contains it, which is the honest result. Renew returns nothing for a worse reason. Source 3 is about renewing books, but the word in it is renewed, and to an exact matcher those are two different words. Learned embeddings are meant to close that gap. This baseline leaves it open so that it can be seen.

4. Put evidence into the answer context

A complete RAG application combines the question with selected passages and instructions about using them. The language model then performs normal conditional generation on that prompt. A simple prompt structure is:

Task: Answer using the supplied sources.
If the sources do not provide the answer, say so.

Question: When does the library open on Saturday?

[Source 2]
The library opens at ten on Saturday. It closes at four.

Answer with supporting source identifiers:

Each line of the template has a job. The task line tells the model to answer from the sources and not from memory. The second line gives it something to say when the sources are silent. That matters because a word-matching retriever returns its best matches whether or not any of them holds the answer. The question is restated so the model knows what to look for. The passage arrives with its source number, so the answer can cite it, and the closing line asks for that citation.

This is a prompt template, not a measured model response. Instructions alone cannot guarantee source-faithful output. The application must evaluate whether the answer agrees with the passages and whether citations support their associated claims.

Part 3. Find the Failure Before Changing the Model

5. Inspect the retrieval result

The SDK baseline returns source indices, text, and cosine scores. It keeps retrieval inspectable without introducing a hosted model or claiming that a generated answer has been verified.

from oop_ml.numpy.modern import TermOverlapRetriever

retriever = TermOverlapRetriever([
    "The library opens at nine on weekdays.",
    "The library opens at ten on Saturday.",
])
for passage in retriever.retrieve("library Saturday", limit=2):
    print(passage.index, passage.score, passage.text)

The snippet indexes two shortened passages and prints two lines. The Saturday sentence comes first with index 1 and a score near 0.535, and the weekday sentence follows with index 0 and a score near 0.267. Indices count from zero, so index 1 is the second passage in the list.

Those scores are higher than the 0.392 and 0.196 the live collection gave the same query and the same two opening sentences. The words shared are the same. The passages are shorter. Each holds seven different words, so the passage length in the denominator is √7 where the live passages had √13.

Saturday sentence:  2 / (√2 × √7) ≈ 0.535
Weekday sentence:   1 / (√2 × √7) ≈ 0.267

A score can be compared with other scores for the same query on the same collection, and with nothing else. Trimming a sentence from each passage raised the right passage’s score from 0.392 to 0.535 without making it any more relevant. No fixed score means that a passage answers the question.

6. Evaluate evidence and generation independently

If the needed passage was not retrieved, changing the answer model may not fix the failure. Check source coverage, chunk boundaries, retrieval ranking, and the context limit. If the right passage was supplied but the answer was wrong, inspect the generation step and its instructions.

What you find	Where the failure is	What to inspect
The needed passage is not among those retrieved	retrieval	source coverage, chunk boundaries, ranking, the context limit
The needed passage was retrieved and the answer is still wrong	generation	the instructions and how the model read the passage

Look at what was retrieved before looking at what was written.

The live collection supplies two different retrieval failures. A search for renew retrieves nothing although the collection covers renewals, which is the retriever’s failure. A question about Sunday opening cannot be answered from any of the three passages, which is a gap in source coverage. No change to the retriever or to the model will supply a fact the collection does not hold. In both cases a more capable answer model would be working from the wrong context, or from none.

Source text is evidence to read, not permission to follow arbitrary instructions embedded in a document. An application should preserve that distinction when constructing prompts and handling external material.

RAG does not automatically make facts current. The collection must be maintained, and the answer must use the relevant version. It also does not remove the need to say when evidence is missing or contradictory.

The library makes the point about currency concrete. Suppose the Saturday hours change and Source 2 is not edited. The retriever will go on returning the old sentence with the same score, and a faithful model will go on citing it. Every step will have worked, and the answer will be wrong.

This paper combines retrieval with generation. The lesson’s word-count retriever is a small baseline, not a reproduction of its learned retrieval architecture. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.

Continue with Evaluating Generative Models.

Split a Document without Separating a Claim from Its Conditions

A passage should contain enough context to interpret its claim. Splitting a maintenance instruction from the warning immediately below it can produce a misleading retrieved fragment. Keep document identity, version, location and neighbouring context available.

Chunk size and overlap trade coverage against repetition and context cost. Inspect retrieval on questions that depend on tables, qualifications and headings. A fixed number of characters is a starting implementation choice, not a definition of a useful evidence unit.

Check Whether Retrieval Supplied the Required Evidence

Before blaming the generator, identify the source passage needed to answer each evaluation question and check whether retrieval included it. The ranker cannot use evidence that the candidate stage never returned.

A simple top-k hit check is a starting diagnostic. Questions requiring several passages need an evidence-set criterion rather than a single relevant ID.

Measure retrieval before generating an answer

Each question has one required passage for this small diagnostic.

Python

import numpy as np
required = ['p2','p5','p1']
retrieved = [['p2','p3'], ['p4','p6'], ['p7','p1']]
hit = [answer in candidates for answer,candidates in zip(required,retrieved)]
print(hit)
print(round(float(np.mean(hit)), 3))

Output

[True, False, True]
0.667

The second question lacks its required evidence. A fluent answer cannot repair that retrieval failure without another information source.

Build the Context with Source Boundaries Intact

Select passages within the context budget, retain their identifiers and make clear which text came from which source. Deduplicate near-identical chunks so repetition does not crowd out a different necessary fact.

Retrieved text is evidence to examine, not an instruction with authority over the application. Keep user instructions and source content structurally separate. Source documents can contain irrelevant directions or malicious instructions as well as useful facts.

A Citation Must Support the Claim beside It

An answer can mention a real document while making a claim that document does not support. Evaluate whether each substantive claim follows from the cited passage, whether the evidence is current and whether important qualifications survived.

Include questions with no adequate source and questions with conflicting sources. A useful system should identify missing or conflicting evidence rather than manufacture a confident answer. Test retrieval, context construction and answer grounding separately so failures suggest a concrete next change.