What is RAG? A first-principles explanation

Retrieval-augmented generation (RAG) is a technique where a language model answers a question using text that was fetched from an external source at the moment you asked, rather than relying only on what it learned during training. In one loop: you take the question, retrieve the most relevant documents from a knowledge store, paste them into the prompt, and let the model generate an answer grounded in that text.

The whole idea rests on a distinction between two kinds of memory. A trained model has parametric memory: facts are compressed into billions of weights during training, blended together, and impossible to point at or update precisely. RAG adds non-parametric memory: an explicit, searchable store of text that lives outside the model and can be edited, versioned, and cited. RAG is the plumbing that connects the two.

A language model is trained to predict the next token given the previous ones. Along the way it absorbs an enormous amount of world knowledge, but that knowledge has three structural problems. It is frozen at the training cutoff, so anything newer is invisible. It is lossy, because facts are averaged across the training set rather than stored verbatim, so rare or specific details get smeared. And it is unattributed, because the model cannot tell you which source a claim came from.

When a model has no stored fact for a question but must still produce fluent text, it fills the gap with something plausible-sounding. That is what people call hallucination, and it is not a bug you can fully patch inside the weights. It is the expected behavior of a system optimized to continue text rather than to be correct.

You could fix some of this by fine-tuning, that is, continuing to train the model on new data. But fine-tuning bakes knowledge back into parametric memory, which means it is still lossy, still hard to attribute, and expensive to redo every time a document changes. RAG takes the opposite approach: leave the weights alone and change the text you show the model at question time.

RAG runs in two phases. There is an offline indexing phase you do once (and repeat when content changes), and an online query phase that runs on every question.

Indexing means turning your source material into something searchable. You split documents into chunks of a few hundred words, convert each chunk into a numeric vector called an embedding, and store those vectors in a database built for similarity search. At query time you embed the user's question the same way, find the chunks whose vectors are closest to the question's vector, and hand the top few to the model alongside the original question.

The generation step is then ordinary prompting. The model sees an instruction like answer using only the context below, followed by the retrieved chunks and the question. Because the relevant facts are now sitting in the prompt, the model can quote them rather than reconstruct them from memory.

An embedding is a list of numbers, say 768 or 1536 of them, that represents the meaning of a piece of text as a point in high-dimensional space. Models that produce embeddings are trained so that texts with similar meaning land near each other, even when they share no words. Dog and puppy end up close; dog and invoice end up far apart.

To rank chunks by relevance you need a way to measure closeness between two vectors. The standard choice is cosine similarity, which measures the angle between them rather than their length. In plain text: cos(θ) = (A·B) / (‖A‖ ‖B‖), where A·B is the dot product of the two vectors (multiply matching components and sum them) and ‖A‖ is the length of vector A. The result runs from 1 for pointing the same direction, through 0 for unrelated, to -1 for opposite.

Using the angle rather than raw distance matters because it makes the comparison about direction, that is, meaning, and not about how long a passage happens to be. Retrieval is then just: embed the query, compute similarity against every stored chunk (or an approximation of that, for speed), and return the highest-scoring ones.

RAG helps because it changes the task. Instead of asking the model to recall a fact, you ask it to read a fact you have placed in front of it, which is something models do far more reliably. It also makes answers auditable: because you know which chunks were retrieved, you can cite sources and let a person verify them. And it makes knowledge updatable, since fixing an answer means editing a document, not retraining a model.

But RAG only moves the hard problem, it does not delete it. If retrieval returns the wrong chunks, or misses the right one, the model answers confidently from bad context, which is arguably worse than an honest gap. The model can still ignore the context and fall back on its training. And nothing forces it to say I don't know when the retrieved text simply doesn't contain the answer.

So the accuracy of a RAG system is bounded by the quality of retrieval, not by the eloquence of the model. Most of the engineering effort, and most of the failures, live in the retrieval half.

A working demo is a hundred lines of code. A RAG system you can trust in production is mostly the decisions around those lines. Chunking strategy shapes what can be retrieved at all: chunks too large dilute the signal, too small lose context. Retrieval usually combines vector similarity with keyword search (hybrid search) and a re-ranking pass, because pure vector search misses exact terms like part numbers or names.

Then come the parts that separate a system from a script: access control so a user only retrieves documents they are allowed to see, evaluation so you can measure whether retrieval is actually returning the right chunks, freshness so the index reflects the current state of the source, and guardrails so the model abstains instead of inventing when context is thin. Each is an engineering discipline, not a prompt.

This is the gap between a RAG tutorial and a RAG system built around a specific organization's data, permissions, and correctness requirements. At Stallwart we treat RAG as infrastructure to be engineered against those constraints rather than a feature to be switched on. If you are learning it, learn the retrieval half deeply, because that is where correctness is won or lost.