Why RAG works in the demo and fails in production

RAG fails in production because the demo measured whether the system could answer questions you already knew the answers to, on documents you already knew contained them. Production asks questions nobody curated, over data nobody cleaned, and the gap between those two is where the pilot dies. The model is almost never the problem. Retrieval is.

A retrieval-augmented generation system has two halves: find the right context, then write an answer from it. Demos stress the second half, which large models already do well. Production stresses the first half, which is an information-retrieval problem your team probably has not built for. When a RAG pilot underperforms, the answer is usually confident and fluent and wrong, and it is wrong because the model was handed the wrong passages, not because it reasoned badly over the right ones.

So the diagnosis almost always starts in the same place: not the prompt, not the model, but what got retrieved. Before changing anything else, log the exact chunks passed into the context for a batch of failing queries and read them. In most stalled pilots the failure is visible in that log within ten examples.

Underperforming RAG systems fail for a small number of recurring reasons. Most pilots have three or four of these at once, which is why swapping the model changes nothing.

You cannot fix retrieval by staring at final answers, because a wrong answer looks the same whether retrieval missed or the model ignored good context. Separate the two halves and measure each. This is the single most useful thing a stalled team can do.

First, evaluate retrieval in isolation. Take a set of real questions, label the passages that genuinely contain each answer, and measure whether retrieval returns them. If the right passage is not in the top-k, no prompt change will help, and you are looking at a chunking, embedding, or indexing problem. If the right passage is present but the answer is still wrong, the failure moved downstream to generation.

Second, read the retrieved chunks by hand for the failures. Cut tables, orphaned headings, and boilerplate are obvious on sight and invisible in an aggregate score. Third, check freshness: when was the index last built, and does the source of truth change faster than that. A large share of production RAG complaints are simply stale answers from a never-refreshed index.

The fixes are unglamorous and they compound. In rough order of how often they matter: fix chunking, then retrieval, then evaluation, then generation, then cost. Most teams reach for the model swap, which is last on the list.

Chunk on structure, not on a fixed token count. Split on headings, sections, and logical units, keep tables and their captions together, and attach metadata such as title, section, and date so retrieval and filtering have something to work with. Then improve retrieval: hybrid search that combines keyword and vector matching recovers exact terms, product codes, and names that pure embeddings miss, and a reranking pass over a larger candidate set lifts the genuinely relevant passage to the top before it reaches the model.

Build the evaluation harness before you tune anything else, because without it every other change is a guess. A modest set of real questions with labeled correct sources lets you measure retrieval recall and answer correctness on every change and catch regressions before they ship. For generation, ground the model explicitly: instruct it to answer only from the provided context and to say when the context does not contain the answer, which turns a confident hallucination into an honest gap you can then close at the retrieval layer. For cost and latency, retrieve fewer and better passages rather than more, cache what repeats, and measure at production traffic, not at demo scale.

The difference between a pilot that demos well and a system that survives production is not the presence of RAG, it is what surrounds it. The contrast is concrete.

A naive pipeline chunks by fixed size, retrieves by vector similarity alone, passes the top-k straight to the model, has no evaluation, and refreshes the index by hand when someone remembers. A production system chunks on document structure with metadata, retrieves with hybrid search and reranking, grounds generation against the context, runs a labeled evaluation set on every change, refreshes the index on a schedule tied to how fast the source changes, and is instrumented so you can see retrieval quality, latency, and cost per query in real traffic. Same three letters, different engineering problem.

That surrounding system is the work, and it is the work Stallwart does: retrieval built around your data, your documents, and your freshness requirements, with evaluation and observability treated as part of the build rather than a later phase. If a RAG pilot underperformed, the fix is almost never a better model. It is the retrieval system the pilot was allowed to skip.