Embeddings Explained From First Principles

An embedding is a list of numbers, a vector, that stands in for a piece of data such as a word, a sentence, an image, or a user. A model maps each input to a fixed-length point in a high-dimensional space, and the arrangement of those points is the whole trick: inputs that a model has learned to treat as similar end up close together, and inputs it treats as different end up far apart. So the numbers themselves are not meaningful in isolation. What carries meaning is the relative geometry, how points sit in relation to each other.

Concretely, an embedding of dimension d is just an ordered array of d real numbers, for example a length-768 or length-1536 vector. You can picture the small case: a 2-dimensional embedding is a point (x, y) on a plane, and a 3-dimensional one is a point in a room. Real embeddings live in hundreds or thousands of dimensions, which you cannot visualize, but the intuition transfers. Nearness in that space is a proxy for similarity in whatever the model was trained to care about.

The reason this is useful is that computers compare numbers cheaply and reliably, while they cannot compare raw meaning at all. Once text becomes a vector, questions like "which of these ten thousand documents is most like this query" become arithmetic on arrays rather than an open-ended reading task.

Nothing about the numbers 0 to 1535 knows anything about language. The proximity of related concepts is not a built-in property of vectors, it is a property the model is trained to produce. The space starts as noise, and training reshapes it until distance lines up with the notion of similarity the training signal rewards.

The underlying idea is old and is often called the distributional hypothesis: words that appear in similar contexts tend to have similar meanings. "Dog" and "puppy" show up around the same neighboring words (leash, bark, vet, walk), so a model that predicts context from a word, or a word from its context, is pushed to give them similar internal representations. Extend that from single words to whole sentences and you get sentence embeddings, where two paraphrases that never share a word can still land near each other because they predict similar continuations or were labeled as a matching pair.

So the honest one-line answer to "why does similar meaning map to nearby vectors" is: because the training objective penalizes the model when related inputs land far apart and rewards it when they land close. Meaning is not stored in the vector, it is compressed into the geometry by optimization.

Two intuitions cover most of what is happening, without needing any specific model's internals. The first is the next-token, or context-prediction, intuition. A model reads text and is trained to predict what comes next, or to fill in a masked word from its surroundings. To predict well, it has to build internal representations where words and phrases that behave alike are represented alike, because that is what lets a single learned rule generalize across many contexts. The embeddings fall out as a byproduct of getting good at prediction.

The second is the contrastive intuition, which is the more direct way to train embeddings for search. You show the model pairs that should be close, such as a question and a passage that answers it, and pairs that should be far, such as that same question and an unrelated passage. The loss pulls the matching pair together and pushes the mismatched pairs apart. Repeat over many examples and the space organizes itself so that queries land near their good answers. This is why embedding models built for retrieval often behave differently from generic ones: they were shaped on exactly the query-versus-document task.

A useful mental model for the mechanics: training nudges each vector a little on every example, tightening true pairs and loosening false ones, and after enough nudges a stable geometry emerges. The specific architectures, datasets, and loss functions vary by model, and the exact recipes are usually proprietary, so it is better to reason about these two objectives than to assume details you cannot verify.

Once you have vectors, you need a way to score how alike two of them are. Three measures dominate, and the difference between them comes down to whether you care about direction, magnitude, or both.

Cosine similarity measures the angle between two vectors and ignores their length. The formula is cos(theta) = (A . B) / (||A|| ||B||), where A . B is the dot product (sum of A_i times B_i over all dimensions) and ||A|| is the length of A, computed as sqrt(A . A). It ranges from -1 (opposite) through 0 (unrelated, orthogonal) to 1 (same direction). Cosine is the default for text because a longer document should not automatically score higher than a short one just for having a larger vector.

Dot product is the raw A . B, the same numerator as cosine but without dividing by the lengths. It rewards both alignment and magnitude, so a longer vector can outscore a shorter one even at the same angle. This is sometimes what you want, for instance when vector magnitude has been trained to encode confidence or popularity, and it is cheaper to compute because it skips the normalization.

Euclidean distance (L2) is the straight-line distance between the two points: sqrt(sum over i of (A_i - B_i)^2). Unlike the other two it is a distance, so smaller means more similar. It is sensitive to magnitude and is the natural choice when the absolute position of points matters, such as in clustering.

The key relationship to remember: if all vectors are normalized to unit length, then cosine similarity, dot product, and Euclidean distance rank neighbors in the same order. They stop agreeing only when magnitudes differ. So the metric you pick matters most precisely when your vectors are not normalized.

Normalizing a vector means dividing it by its own length so it becomes a unit vector: A_normalized = A / ||A||. After this step every vector sits on the surface of a unit sphere and carries only direction, not magnitude. Doing this makes dot product and cosine similarity identical, which is why many vector databases normalize on ingest and then use the fast dot product internally while you think in cosine terms.

There is a practical reason to be deliberate here. If you store raw vectors and query with a metric that is sensitive to length, an unusually long document vector can dominate results for reasons that have nothing to do with relevance. Normalization removes that failure mode. The rule of thumb: decide on one convention, normalize consistently on both the stored vectors and the query vector, and make sure your database's configured metric matches how the embedding model was trained to be compared.

Embeddings compress meaning into a fixed number of dimensions, and compression is lossy by definition. A single vector for a paragraph cannot represent every distinction in that paragraph, so fine-grained detail, exact numbers, dates, names, and negation are often smeared out. "The contract was signed" and "the contract was not signed" can sit dangerously close, because the surrounding words are nearly identical and the model was rarely rewarded for separating them.

Similarity is also not the same as truth or relevance. Two passages can be near each other because they share a topic while one directly answers the query and the other does not. Embeddings capture aboutness, not correctness, so a nearest-neighbor hit is a candidate, not an answer. They also inherit whatever the training data emphasized: a model trained mostly on general web text will represent a niche legal or medical corpus more coarsely, and it will carry the biases and blind spots of its data.

Finally, a vector reflects the model that produced it. Vectors from two different embedding models are not comparable, and re-embedding a corpus with a new model means every stored vector has to be regenerated. Treat the embedding model as a fixed dependency of your index, not an interchangeable part.

In production, embeddings power semantic search and retrieval-augmented generation. You embed every document (usually split into chunks) once and store the vectors in an index. At query time you embed the incoming question with the same model, then ask the index for the nearest vectors using an approximate nearest-neighbor search, which trades a little accuracy for speed so you can search millions of vectors in milliseconds. The top matches become the context you hand to a language model to answer from.

Because embeddings capture aboutness rather than exactness, the strongest systems do not rely on vectors alone. Hybrid retrieval combines embedding search with keyword search, so exact terms, names, and codes are not lost, and a reranking step then re-scores the top candidates with a heavier model that reads the query and passage together. This directly addresses the limits above: vector search casts a wide semantic net, keyword search catches the literal matches, and reranking sorts out relevance from mere topical nearness.

The engineering discipline is mostly in the details around the embedding, not the embedding call itself: how you chunk documents so a vector represents a coherent idea, keeping the query and document models identical, matching the database metric to the model, and re-embedding when you change models. Those choices decide whether a retrieval system returns the right passage or a plausible-looking wrong one. Building retrieval that holds up on a real corpus, with the customer's own data and constraints rather than a clean demo set, is the kind of production work Stallwart focuses on.