RAG Architecture Explained, Stage by Stage

Retrieval augmented generation is a pattern where a language model answers using text fetched from your own data at query time, rather than only what it learned during training. The pipeline runs in two phases: an offline phase that ingests documents, splits them, embeds them, and writes them to a store, and an online phase that embeds the user query, retrieves the most relevant chunks, optionally reranks them, and passes them to the model as context for generation.

The reason RAG exists is grounding. A model asked a question about your contracts, your codebase, or last quarter's tickets has no reliable knowledge of them, and asking it anyway produces confident invention. RAG changes the question from "what do you know" to "here are the relevant passages, answer using only these," which makes answers checkable against a source.

Every stage below is a place where a design decision either preserves or destroys the signal that reaches the model. A RAG system fails far more often in chunking and retrieval than in the model itself, so the engineering effort belongs there.

Ingestion is parsing source documents into clean text plus metadata. The hard part is not reading a PDF, it is preserving structure: headings, tables, code blocks, and the parent document a passage came from. Tables flattened into run-on text and headings dropped are information you can never retrieve later, because it is no longer in the index.

Chunking then splits that text into retrievable units. This is the single highest-leverage decision in the pipeline, because a chunk is the smallest thing retrieval can return. Too large, and one chunk mixes several topics so the embedding is a blurred average and the model wastes context on irrelevant text. Too small, and a chunk loses the surrounding context needed to make it meaningful on its own.

There is no universal chunk size. The design decision is to chunk along the document's own structure rather than by a fixed character count. A few strategies, from crudest to most faithful:

An embedding model turns each chunk into a vector, a list of numbers positioning that text in a high-dimensional space where similar meanings sit close together. At query time the same model embeds the question, and retrieval becomes a nearest-neighbor search: find the chunk vectors closest to the query vector.

Closeness is usually cosine similarity, the cosine of the angle between two vectors, cos(θ) = (A·B)/(‖A‖‖B‖). It ranges from -1 to 1 and ignores magnitude, so it compares direction, that is, meaning, rather than text length. This is why a two-line answer and a paragraph on the same topic can score as highly similar.

The embedding model choice matters more than the vector database. Use the same model for indexing and querying, match it to your domain and languages, and remember its context limit caps useful chunk size. Dimensionality is a tradeoff: more dimensions can capture more nuance but cost more memory and search time.

The vector store holds these vectors and serves approximate nearest-neighbor search, trading a small amount of recall for large speed gains at scale using an index such as HNSW. Choose it on operational fit: metadata filtering, hybrid keyword-plus-vector search, update and delete behavior, and whether you want a managed service or a library embedded in your own service.

Retrieval fetches the top-k candidate chunks for a query. Pure vector search is strong on meaning but weak on exact terms like error codes, product SKUs, or names, where lexical search wins. Hybrid retrieval runs both a keyword search and a vector search and fuses the results, which is why it is the common production default rather than vector search alone.

Reranking is a second, more precise pass. The first retrieval optimizes for speed and casts a wide net, say the top 50 candidates. A reranker, typically a cross-encoder that reads the query and each candidate together rather than comparing precomputed vectors, then scores and reorders them so the best few reach the model. It is slower per item, which is exactly why it runs only on the shortlist, not the whole corpus.

Generation is the final stage: the reranked chunks go into the prompt with an instruction to answer only from the provided context and to cite or abstain when the context does not contain the answer. The design decisions here are the prompt contract, how much context to include before quality degrades, and whether to return citations back to the source chunks so a human can verify the answer.

A useful way to see the pipeline is as a series of filters that each narrow a large corpus down to the handful of passages the model reads. Retrieval trades recall for speed, reranking trades speed for precision, and generation trades context length for focus. Get the balance wrong at any one stage and the others cannot recover it.

A RAG system that is never measured drifts silently as data and queries change. Evaluation is what turns "it seems to work" into a number you can defend, and it splits cleanly into retrieval quality and generation quality, because a wrong answer can come from either.

Measure retrieval on its own first, since the generator cannot answer from context it never received. Standard measures are recall at k, whether the relevant chunk is in the retrieved set, and precision, how much of what was retrieved is actually relevant. If recall is low, no prompt engineering will fix the answer.

Then measure generation against the retrieved context: faithfulness, whether the answer is supported by the context rather than invented, and answer relevance, whether it addresses the question. Build a fixed evaluation set of real questions with known-good answers, run it on every change, and you convert tuning from guesswork into a controlled experiment. This is also where using a model as an automated judge, scored against that reference set, earns its place.

This is the discipline that separates a demo from a system you can run in production. At Stallwart we treat the evaluation set as a first-class deliverable built around the customer's real queries and data, because a RAG pipeline you cannot measure is one you cannot safely change.

When a RAG answer is wrong, the cause is usually upstream of the model. Diagnosing in pipeline order saves time: