Vector Databases Explained From First Principles

A vector database stores embeddings, which are lists of numbers that represent the meaning of a piece of text, an image, or audio as a point in high-dimensional space. When you embed a chunk of text with a model, you get back a fixed-length array, often 384, 768, or 1536 numbers. The database keeps that array alongside an identifier and usually some metadata, and its one core job is to answer the question: given a new query vector, which stored vectors are closest to it?

The reason this is useful is that good embedding models place semantically similar things near each other. Two sentences that mean roughly the same thing land close together in the space even if they share no words. So 'nearest in vector space' becomes a usable proxy for 'most relevant to the query'. Retrieval stops being keyword matching and becomes geometry.

It helps to be precise about what a vector database is not. It is not a general-purpose store for your source of truth, and it is not magic. It is an index over points in space plus the machinery to search that index quickly. Everything else, the metadata filtering, the hybrid keyword scoring, the sharding, is built around that one geometric operation.

The naive way to find the closest vectors is brute force: compare the query against every stored vector, compute a distance for each, and keep the smallest. This is exact, it always returns the true nearest neighbors, and for a few thousand vectors it is completely fine. Many teams over-engineer this step when a linear scan would have served them for a year.

The problem is the cost. With n stored vectors of dimension d, a single query costs on the order of n times d arithmetic operations, which is O(n*d) per query. Doubling your data doubles your query time. At a million vectors of dimension 768, one query touches hundreds of millions of numbers, and you pay that for every query, from every user, every time. Linear scaling sounds gentle until you plot it against a growing corpus and a latency budget measured in tens of milliseconds.

There is also a subtler tax called the curse of dimensionality. Classic tree structures like k-d trees, which make low-dimensional nearest-neighbor search fast, degrade toward brute force as dimension grows, because in high dimensions almost every point sits at a similar distance from every other point. The clean partitions that speed up 2D or 3D search stop separating anything useful. That is why high-dimensional search needs a different family of methods entirely.

Before you can find the nearest vector you have to define near. The three common metrics are cosine similarity, dot product, and Euclidean (L2) distance, and the right choice depends on how your embedding model was trained.

Cosine similarity measures the angle between two vectors and ignores their length: cos(θ) = (A·B) / (‖A‖ ‖B‖). The dot product A·B is the sum of element-wise products, and each norm ‖A‖ is the square root of the sum of squares of that vector's components. Cosine ranges from -1 to 1, where 1 means the vectors point the same direction. It is the default for text embeddings because it cares about direction, meaning, rather than magnitude.

Dot product alone keeps the magnitude in play, which matters for some models that encode confidence or importance in vector length. Euclidean distance measures straight-line distance between the two points. A key practical fact: if all vectors are normalized to unit length, then ranking by cosine, by dot product, and by Euclidean distance all produce the same neighbor order, so many systems normalize on write and then use the cheapest metric. The one rule that is not optional is to use the metric the embedding model was trained with, because mixing them silently degrades relevance.

The escape from O(n) is to stop insisting on the exact answer. Approximate nearest neighbor (ANN) search accepts occasionally missing a true neighbor in exchange for searching a small fraction of the data. The quality of an ANN index is measured by recall, the fraction of the true top-k neighbors it actually returns. A well-tuned index often reaches recall in the high 0.9s while touching a tiny slice of the corpus, and that trade is what makes vector search viable at scale.

HNSW (Hierarchical Navigable Small World) is a graph. Each vector becomes a node connected to its near neighbors, and the graph is built in layers: sparse long-range links at the top, dense short-range links at the bottom. A search starts at the top layer and greedily walks toward the query, hopping to whichever neighbor is closer, then drops a layer and refines. It is the same idea as skimming a map at country scale to find the region, then zooming in street by street. You reach a good answer in roughly logarithmic hops instead of scanning everything. HNSW gives excellent recall and low latency, at the cost of high memory, because the graph edges have to live in RAM.

IVF (Inverted File index) partitions instead of linking. During training it runs clustering, often k-means, over the vectors to find a set of centroids, and every vector is assigned to its nearest centroid's bucket. At query time you compare the query only against the few centroids, pick the closest handful of buckets, and search only inside those. If you split a million vectors into a thousand buckets and probe ten of them, you look at roughly one percent of the data. The number of buckets you probe is a dial: probe more for higher recall and higher latency, fewer for the reverse. IVF is often paired with product quantization (PQ), which compresses each vector into a compact code so far more vectors fit in memory, trading a little accuracy for a large memory saving.

Every index choice is a point in a triangle of recall, latency, and memory, and you cannot maximize all three at once. Understanding which corner you are pulling toward is most of what tuning a vector store is about. The parameters are not mysterious once you see what each one buys.

HNSW's main knobs are the number of edges per node (often called M) and the size of the search frontier at query time (efSearch). More edges and a wider frontier raise recall and memory or latency. IVF's knobs are the number of buckets and how many you probe (nprobe). Quantization adds a fourth axis: compress harder to save memory and lose a little recall. There is no universally best setting, only the setting that fits your corpus size, your latency budget, and your hardware.

The honest way to choose is to measure on your own data, because published benchmarks use datasets and hardware that are probably not yours. Build a small labeled set of queries with known correct answers, then sweep the parameters and plot recall against latency. Pick the cheapest configuration that clears your relevance bar. Treat any single vendor benchmark number as marketing until you have reproduced the shape of the curve on your workload.

Real systems rarely want the globally nearest vector. They want the nearest vector that also belongs to this tenant, is newer than last quarter, and has the right document type. That is metadata filtering, and how a database combines it with vector search matters a great deal. Pre-filtering narrows the candidate set first and then searches, which is exact but can be slow if the filter is not indexed. Post-filtering searches first and discards non-matching results, which is fast but can return too few results when the filter is selective, because the nearest neighbors were filtered away. Mature engines maintain filterable indexes so the two run together rather than fighting.

In a production RAG pipeline the vector database is one stage, not the whole system. The flow is: chunk your documents, embed each chunk, write vectors plus metadata to the store, then at query time embed the user's question, retrieve the top-k nearest chunks under any filters, and pass those chunks to the language model as grounding context. The retrieval quality sets a ceiling on the answer quality. If the right chunk is not in the top-k, no amount of prompt engineering downstream will recover it, which is why recall, chunking, and filtering deserve as much attention as the model itself.

Two failure modes dominate in practice, and neither is fixed by swapping databases. The first is bad chunking, where content is split so that the answer is scattered across chunks that never co-retrieve. The second is embedding drift, where the model used to index the corpus differs from the model used to embed queries, so the geometry no longer lines up. Getting these right is unglamorous engineering around your specific data and constraints, which is exactly where a retrieval system earns or loses its reliability. At Stallwart this is the layer we treat as production infrastructure rather than a demo, but the principles here hold whoever builds it.

You do not automatically need a dedicated vector database the moment you touch embeddings. The deciding factors are corpus size, query volume, latency budget, and whether you need filtering and updates at scale. Below roughly a hundred thousand vectors with modest traffic, a brute-force search in memory or a vector extension on a database you already run is often simpler and fast enough, and it removes a whole system from your stack.

The case for a dedicated store grows with scale. If you have millions or hundreds of millions of vectors, tight latency targets, high query concurrency, frequent inserts and deletes, or heavy metadata filtering, the specialized indexing, sharding, and memory management of a purpose-built engine start to pay for themselves. A useful comparison framing: a vector extension on your existing relational database keeps everything in one place and one backup story, while a dedicated vector database gives better index tuning, scaling, and filtering at the cost of another system to operate and keep in sync.

The first-principles rule is to add infrastructure when a measured limit forces it, not in anticipation. Start with the simplest thing that meets your recall and latency numbers on your real data, measure, and move to a dedicated engine when brute force or a lightweight extension actually stops clearing the bar. Most early RAG systems fail on chunking and evaluation long before they fail on which vector database they chose.