Embedding Dimensions

What do you actually lose when you shrink an embedding from 1536 numbers to 256?

Not facts. Distances.

An embedding A learned list of numbers (coordinates) that represents the meaning of a word, sentence or document. Similar meanings get similar numbers. looks like a list of facts, but it is not one. No dimension means “PostgreSQL”. The meaning is spread across all of them, and what a search engine uses is not the numbers themselves. It uses the distances between them.

So when you cut a 1536-dimensional vector down to 256, you are not removing 83% of the meaning. You are removing 83% of the space. Every point gets projected into a smaller room, and points that were apart can land close together, or exactly on top of each other.

That is the loss that matters in practice: not information about what a document is about, but the geometric distance that separated it from documents that are almost the same.

The photo analogy helps. Take a 4K image down to 480p and you still see a car. What disappears is the license plate, the logo on the hubcap, the small text on the door. Broad meaning survives compression very well. Fine distinctions do not.

At 1536 dimensions the list is cleanly spread: 0.96, 0.81, 0.78, 0.41. At 256 the document about a different database climbs to second place, because the block that separated it was cut. At 64 all four cosines are 1.000, and the index can no longer tell termination from pooling from MySQL. Nothing was deleted from the documents. The geometry that separated them is simply not in those 64 numbers.

Keep the first k coordinates, then compare.

Truncation is exactly what it sounds like: keep the first k numbers of every vector and compute cosine similarity on those. Geometrically it is a projection onto the first k axes.

A small example, with made-up axes so the picture is visible. Say dimension one means “database”, two means “PostgreSQL”, three means “termination”:

• Query: [1, 1, 1]
• A: [1, 1, 1] (terminate PostgreSQL connections)
• B: [1, 1, 0] (PostgreSQL connection pooling)

Drop the third dimension and both documents become [1, 1]. The query cannot tell them apart anymore, and no amount of tuning downstream brings that difference back.

Why do the scores compress? Shared meaning lives everywhere, while the distinctions live in a few dimensions. Cut those dimensions and every vector keeps mostly the part it has in common with the query. The cosines move toward each other. In the lab the spread between the best and worst document goes from 0.553 at full size, to 0.187 at 256, to 0.000 at 64.

Why Matryoshka exists, and what rerankers are for.

Naive truncation is a gamble, because you do not know which dimensions carry the distinctions. Matryoshka Representation Learning removes the gamble: the model is trained so that the first 256 dimensions are useful on their own, the first 512 are better, and the full vector is best. One model, several usable sizes, chosen at query time.

This is why hosted models can offer a dimensions parameter (OpenAI’s text-embedding-3 family is the best-known example), and why open models such as Nomic Embed are trained with Matryoshka objectives. You pay for resolution, not for a different model.

And this is where the reranker comes back in. When 256 dimensions squeeze three different documents into a 0.05 band, the first stage has done its job if all three are in the shortlist. The reranker then reads query and document together, at full attention, and puts the right one first. Cheap recall at low resolution, expensive precision at the top. The dimension choice and the reranker are two ends of the same trade.

The fine print.

• Cutting is not monotone. Removing dimensions can raise a score, as document D does in the lab. If a document looks more relevant after compression, that is usually a sign, not an improvement.
• The right size is empirical. Build a small set of queries where the top candidates are near-duplicates, then measure recall@k at 1536, 768, 256 and 128. The number that matters is the smallest size that still finds the right document.
• Never mix sizes in one index. Vectors of different dimensions are not comparable, and thresholds do not transfer between sizes. Truncate consistently, at index time and at query time.
• Matryoshka is not magic. The first 256 dimensions are trained to be useful, but they are still lower resolution. Fine distinctions are the first thing to go, by design.
• Dimension count is not quality. Compare models at the size you will actually use, not at their maximum.
• The reranker cannot rescue recall. If compression pushes the right document out of the shortlist entirely, no second stage brings it back. Compression first hurts recall, then ordering.
• Measure the cost honestly. Six times less memory and a cheaper dot product are real wins at a million vectors. Whether they are worth the lost distinctions depends on your queries, and only your queries can answer that.