What is RAG (retrieval-augmented generation)?

Why it exists

A language model only knows what was in its training data. It doesn’t know your company’s policies, your product manual or last week’s price list, and when asked, it may confidently make something up. Retraining the model on your documents is expensive and slow to update. RAG takes a simpler route: give the model the right pages at the moment it answers.

How it works, step by step

1. Split your documents into passages of a few hundred words, called chunks.
2. Embed each chunk: turn it into a list of numbers that captures its meaning.
3. Store the embeddings in a vector database or index.
4. Retrieve: when a question comes in, embed it the same way and find the closest chunks.
5. Generate: send the question and those chunks to a language model, with an instruction to answer only from them and cite where each fact came from.

Steps 1 to 3 happen once, and again whenever documents change. Steps 4 and 5 happen for every question, in a second or two.

Why it’s popular

• Fewer made-up answers: the model works from real passages, and can say “I don’t know” when nothing relevant is found.
• Sources you can check: every answer can link to the page it came from.
• Easy to update: change a document and re-embed it; there’s no retraining.
• No training data needed: you can build a useful first version in a day.

Where RAG goes wrong

Most bad answers come from retrieval, not from the language model: the right passage wasn’t found, because it was split badly, used different words or was buried in a table. So test retrieval on its own. Write 30 to 50 real questions with the passage that answers each one (a golden set), and measure how often that passage appears in the top results. Fix chunking and search first; switch language models last.

RAG or fine-tuning?

Use RAG when the model needs to know facts from your documents, especially ones that change. Use fine-tuning when the model needs to behave differently: follow a specific format, tone or task it struggles with. Many teams start with RAG and only fine-tune if a clear need remains.