RAG that actually answers: why the demo works and the rollout doesn't

RAG, retrieval-augmented generation, is the standard way to make an AI answer from your own knowledge instead of guessing. Before it answers, the model looks up the relevant passages in your documents and writes from those. Think of it as an open-book exam: it does not have to memorise your handbook, it just has to find the right page.

That is why the demo is so easy and the rollout is so hard. A clean PDF and three test questions look magical. A thousand real documents and real users are a different problem.

Where RAG rollouts break

• Retrieval, not generation, is the bottleneck. If the search step pulls the wrong passage, the model writes a confident answer from the wrong source. Most ‘the AI is wrong’ complaints are really ‘the retrieval was wrong’.
• Chunking decides quality. Split documents badly and the answer loses the context it needed. It is unglamorous plumbing, and it is most of the work.
• Messy sources in, messy answers out. Scanned PDFs, tables, duplicated policies, outdated versions. Real document sets are not clean, and the pipeline has to handle that.
• No source, no trust. If the answer cannot show which document it came from, nobody in a regulated or high-stakes setting will rely on it.

What makes it hold up

A production RAG system is mostly retrieval engineering, not prompting. Good chunking and metadata. A retrieval step you can measure, with an eval set of real questions and known-correct sources. Every answer linked to its source passage. A way to keep the index fresh as documents change. And a fallback for ‘I do not know’, because a confident wrong answer is worse than an honest gap.

When RAG is not the answer

If the knowledge fits in a single prompt, you do not need RAG, just paste it in. If the task is reasoning rather than lookup, retrieval will not help. And if your documents are a mess, fixing the source content often beats building a pipeline on top of chaos. RAG earns its place when the knowledge is large, keeps changing, and has to be answered with a source.