What is RAG? How AI can answer from your own documents.

RAG, short for retrieval-augmented generation, is a way to make an AI model answer from your own documents. Before the model writes a reply, the system searches your files for the passages that matter and hands them over with the question. The model answers from that material and can show where each part came from.

What problem does RAG fix?

A language model only knows what it saw in training. It does not know your prices, your policies or last week's change to the returns process, and if asked anyway it may give a confident answer that is wrong. RAG fixes this by giving the model the right source material at the moment it answers.

This matters most where a wrong answer has a cost: a support assistant quoting a refund rule, a sales assistant promising a delivery time, or an internal assistant explaining leave to staff. In each case you want the answer to come from the current document, and you want to be able to see which one.

Where did RAG come from?

The term comes from a 2020 research paper by Patrick Lewis and colleagues, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. They paired a language model with a searchable index of Wikipedia, so the model could look facts up rather than rely only on what it had absorbed during training.

Two ideas from that paper still matter to a business. The authors named provenance, being able to show where an answer came from, and keeping a model's knowledge up to date as open problems for models that rely on memory alone. They also showed that swapping the document index changes what the system knows, with no retraining. That is the practical appeal. When a policy changes, you update the document, not the model.

How does RAG work?

RAG works in two phases. Ahead of time, your documents are split into chunks, each chunk is turned into an embedding, and the results are stored in a searchable index. At question time, the question becomes an embedding too, the closest chunks are retrieved, and the model writes an answer from them with its sources.

Search first, then write. That is the whole trick.

• Chunking. Documents are cut into passages small enough to be precise. Anthropic describes chunks as usually no more than a few hundred tokens. Cut too small and a passage loses its context. Cut too large and it drags in noise.
• Embeddings. An embedding model turns each chunk into a long list of numbers. Claude's documentation describes text embeddings as numerical representations of text that let you measure semantic similarity, which is why a question about refunds can find a passage about returns.
• Retrieval. The system finds the chunks closest to the question. Good systems often add keyword search for exact terms such as product codes, which Anthropic notes embeddings can miss, and a reranking step that keeps only the most relevant chunks.
• Grounded answers with citations. The model is told to answer only from the retrieved passages, to say so when they do not contain the answer, and to name the document behind each claim.

The citations are what make RAG useful at work. A person can click through and check the answer against the source. An answer nobody can check is a confident guess, however well it is written.

When is RAG the right tool?

RAG fits when the answers live in documents that change, there are too many of them to give the model at once, and people need to see the source. Typical examples are a support knowledge base, product manuals, internal policies and past proposals. Retrieval can also be limited to the documents each person is allowed to see.

When is RAG not the right tool?

RAG is the wrong tool when the answer sits in a database, when your material is small enough to fit in the prompt, or when you need to change how the model behaves rather than what it knows. Each of those has a simpler or better option, and it is worth ruling them out first.

• A plain database lookup. Questions such as where is my order, or how many units are in stock, have one exact answer in a system you already run. Query that system directly. An agent can call the query as a tool. Searching documents for it is slower and less reliable.
• Long context. If the whole knowledge base fits in the model's context window, you can send all of it with every question. In its 2024 article on retrieval, Anthropic suggested this for knowledge bases under 200,000 tokens, about 500 pages, and noted that prompt caching makes it faster and cheaper.
• Fine-tuning. Training a model further on examples changes its tone, format or behaviour on a narrow task. It is a poor way to add facts that change, because every change means training again. The two can be combined: fine-tuning for how to answer, RAG for what to say.

We start with the simplest option that could work. If the documents fit in the prompt, we test that first. If they do not, or they change every week, RAG earns its extra moving parts.

How do you know a RAG system is working?

Test it on real questions before anyone relies on it. Collect questions your team or customers actually ask, write down the correct answer and its source for each, and run the system against that set. Then check two things separately: did it retrieve the right passage, and did it answer faithfully from it?

Separating the two tells you where to fix. A wrong passage means the search needs work: chunk sizes, keyword matching or reranking. A right passage with a wrong answer means the instructions need work. Keep the question set and run it again every time something changes.

RAG is also how agents look things up. Our guide to how AI agents work shows where retrieval sits among an agent's tools and memory, and AI agents for a growing business covers proving a system on real data before it goes live.

If you want an assistant that answers from your own documents with a source for every answer, that is part of our custom AI and agents work. Tell us what your team keeps asking and we will check whether RAG, long context or a plain lookup fits best.