RAG (retrieval-augmented generation)
In shortRAG (retrieval-augmented generation) first looks up relevant passages in your documents, then gives them to the model so it answers from them.
In plain words
A model knows only what it learned in training plus what you put into the prompt. RAG adds a search step in between. The idea comes from a 2020 research paper by Lewis and colleagues, which combined a language model with a searchable text store. Anthropic's glossary describes it this way: relevant information is retrieved at runtime and passed to the model together with the original question, so the model can draw on facts beyond its training data.
The usual steps:
- Split your documents into short chunks.
- Turn each chunk into an embedding and store it in an index.
- When a question arrives, find the chunks closest in meaning.
- Put the question and those chunks into the prompt.
- The model answers from them, ideally with a quote or page number.
Why it matters
- Fresh and private facts: The model can use today's price list or your handbook without new training. Compare fine-tuning, which changes behaviour more than it adds facts.
- Fewer wrong answers: Grounding in text lowers the risk of hallucination. It does not remove it, and the answer is only as good as the passages found.
- Lower cost: You send a few chunks, not the whole archive, so you stay inside the context window and pay for fewer tokens.
- Privacy: The retrieved chunks go to the model provider. Check what you index.
- Security: Retrieved text can contain hidden instructions. OWASP notes that RAG does not fully prevent prompt injection.
Example
A firm indexes its 500-page staff handbook. An employee asks: "How many days of notice do I give for leave?" The system finds the three closest paragraphs and sends them with the question. The model answers in two sentences and names the section. If the rule is not in the handbook, a good setup says so instead of guessing.
Common misconceptions
- "RAG trains the model." It does not change the model. It changes the prompt.
- "RAG needs a vector database." Many setups use one, but keyword search or a mix also works.
- "RAG always answers correctly." Bad search gives bad passages, and bad passages give bad answers.
Questions and answers
What does RAG stand for?
Retrieval-augmented generation: retrieve relevant text first, then let the model generate an answer from it.
RAG or fine-tuning: which one do I need?
Use RAG when the model must use facts that change or are private. Fine-tuning suits style, format or task behaviour. Many projects start with RAG.
Does RAG stop hallucinations?
It lowers the risk by grounding answers in text, but errors remain. Ask for quotes and check key facts.
Read next
- Embedding
An embedding is a list of numbers that represents the meaning of a text or image, so that similar items end up close together.
- Context window
The context window is the maximum amount of text, counted in tokens, that a model can read and write in one request.
- Hallucination
A hallucination is a fluent, confident AI answer that is false, invented or not backed by the sources it was given.
- Fine-tuning
Fine-tuning trains an existing model further on your own examples, so it follows a style, format or task more reliably.
- Prompt injection
Prompt injection is an attack where text a model reads, such as a web page or file, contains instructions that make it act against its user.