What is retrieval-augmented generation?
Retrieval-augmented generation (RAG) is a method in which an AI system first searches a collection of documents or data for material relevant to a request, then passes that material to a language model together with the request, so the answer is based on specific, current sources rather than only on what the model learned in training.
The approach was named in a 2020 research paper by Lewis and colleagues and has since become the most common way organizations put their own content behind an AI assistant. Its appeal is a clean division of labor. Knowledge lives in documents that the organization controls and can update in minutes, while the model supplies language ability. Changing what the system knows means changing the documents, not retraining the model.
How retrieval-augmented generation works
A RAG system runs four steps for every request, and each has its own way of going wrong.
| Step | What happens | What can go wrong |
|---|---|---|
| Ingest | Documents are split into passages, converted into numerical representations called embeddings and stored in an index | Outdated, duplicated or contradictory documents enter the index |
| Retrieve | The request is compared with the index and the closest passages are selected | The right passage is missed, or a similar but wrong one is chosen |
| Augment | The selected passages are inserted into the prompt with instructions to answer from them | Too much or conflicting material crowds out what matters |
| Generate | The model writes the answer, ideally citing the passages it used | The model adds claims the passages do not support |
In practice, the quality of a RAG system depends more on its content and its retrieval than on the model. Teams improve results by combining keyword and semantic search, re-ranking candidates before they reach the model, filtering by the user's permissions and instructing the model to say when the sources do not contain an answer.
Why retrieval-augmented generation matters
Grounding answers in sources helps, but it is not a guarantee. When Stanford researchers tested AI legal research tools built on retrieval, they found the tools hallucinated less than a general-purpose model, yet each still produced hallucinated answers between 17% and 33% of the time on their test queries.
The lesson is that retrieval narrows the problem without closing it. A system that shows its sources lets a reader check the answer, and one that declines when nothing relevant is found avoids the worst failures. Where a wrong answer has consequences, such as a fee quoted to a customer or a policy term explained to an employee, review and testing remain part of the design.
Where retrieval-augmented generation is used
- Internal knowledge assistants. Answering staff questions from policies, procedures and product documentation.
- Customer help. Answering questions about products, fees and processes from approved content.
- Document question answering. Finding and explaining clauses in contracts, policies and filings.
- Research. Pulling together material from many reports or records for a person to review.
- Within AI agents. Answering a side question in the middle of a task, such as what a document requirement means, before the agent returns to the process.
RAG answers questions; it does not run processes. A task with required steps in a required order, such as completing an application, is better served by a defined flow that uses retrieval only where a question comes up. And figures that change by the minute, such as a balance or an order's status, should come from the system of record through a direct query, not from documents.
What good retrieval-augmented generation looks like
- Curated sources. Every document in the index has an owner, a date and a reason to be there.
- Permission-aware retrieval. Users only receive content they are entitled to see.
- Visible citations. Answers show which passages they came from.
- Willingness to abstain. The system says when its sources do not cover the question.
- A test set of real questions. Retrieval and answers are checked against known-correct results before changes go live.
- Gap tracking. Questions that found nothing are logged and fed back to content owners.
How to measure retrieval-augmented generation
- Retrieval recall. How often the passage containing the answer appears among the top results.
- Faithfulness. The share of claims in an answer that the retrieved passages support.
- Answer correctness. Agreement with known-correct answers on the test set.
- Abstention rate. How often the system declines, checked for both missed answers and correct refusals.
- Source freshness. The age of the documents behind answers, against their review dates.
Data and licensing rules for RAG
A RAG system copies an organization's documents, and often its customers' data, into a new index that people can query. The US points below are the common ones; specific industries and other countries impose further rules.
- Access to documents. Financial companies covered by the FTC Safeguards Rule must restrict each user's access to customer information to what that user needs. A retrieval index that ignores document permissions can surface records to people who should not see them.
- Deletion requests. Under the California Consumer Privacy Act, a consumer can ask a business to delete their personal information, and the business must pass the request on to its service providers. Copies held in a retrieval index count.
- Licensed content. Material from data providers, publishers and research services is often licensed for internal use only. Placing it behind a customer-facing assistant can exceed the license.
Frequently asked questions
What is the difference between RAG and fine-tuning?
RAG supplies information at the moment of each request without changing the model. Fine-tuning changes the model itself through further training, which suits consistent style or format better than facts that change.
Does RAG stop AI models from making things up?
It reduces the problem but does not remove it. Models can still misread a passage or add unsupported claims, which is why citations, abstention and testing matter.
What is a vector database?
A database built to store embeddings and find the ones closest to a query quickly. It is the usual home of a RAG index.
Can RAG use live data such as account balances?
Live, customer-specific figures are better fetched directly from the system of record through an interface. RAG is designed for documents and other content that change occasionally.
How much content does RAG need?
There is no minimum. A small set of well-maintained documents usually outperforms a large set of outdated ones.
Related terms
Answers are a step. Outcomes are the goal.
Encore AI's agents are not retrieval assistants.
Scale your best to be the best.
Book a Demo