Distillation
Reranking
If vector search already found the right document, why do production systems pay for a second pass?
Finding is easy. Ordering is hard.
The first pass of a search system is built for speed. It compares your query against millions of precomputed summaries and returns the closest ones — approximately. That word matters: it is optimized to find, not to judge.
The problem is that “close in meaning” and “actually answers the question” are different things. A document can repeat your topic’s vocabulary and contain nothing useful.
query · “What do I get if the service keeps going down?”
The fix is a second stage. A cross-encoder A model that reads two texts together — query and document — and outputs a single relevance score. Slower, because nothing can be precomputed. takes the query and one candidate and reads them together, like a person opening the book. It is far more accurate and far more expensive, so you only run it on a shortlist.
Think of it as two librarians. The first one walks you to the right shelf in a second — approximate, but fast. The second opens a handful of books and checks the actual pages.
Deeper
Why not rerank everything? Because the cross-encoder needs one full model pass per candidate. A vector search over a million chunks is a lookup; a cross-encoder over a million chunks is a million model calls. So the architecture is always the same shape: cheap recall, expensive precision.
Precomputed comparison vs joint reading.
The two stages differ in one structural way: when the query meets the document.
A bi-encoder (vector search) embeds the query and the document independently:
score(q, d) = cos(E(q), E(d))
Both sides were computed long before they met — the document side at index time, the query side at search time. That is what makes it fast, and also what makes it approximate: each side is a summary.
A cross-encoder scores them as one input:
score(q, d) = f(q ⊕ d)
The ⊕ is a concatenation. Because query and document are read together, the model can weigh individual words against each other — “does the document actually answer this?” — but nothing can be precomputed.
Math
For a shortlist of k candidates, the cost of reranking is k forward passes of the cross-encoder. Vector search over n chunks is, in comparison, roughly independent of model depth; that is why n can be millions while k stays around 25–100.
Reranker scores are only comparable within one query, within one model. Do not mix them across models or compare them to cosine values — they answer different questions on different scales.
When lexical and vector stages are combined, their scores are often fused by rank rather than value — reciprocal rank fusion is the common default:
RRF(d) = Σ 1 / (60 + rank_i(d))
Rank fusion avoids the impossible question of whether a BM25 score of 12.4 is “better” than a cosine of 0.83.
Deeper
Practical shortlist numbers come from budgets, not theory. 50 candidates with a small cross-encoder on GPU is a common starting point; 25 on CPU. The upper bound is latency, not quality — quality usually flattens long before cost does.
Two retrieval ways in, one small list out.
The same query goes to both retrievers. Each one has different blind spots.
Deeper
When is a reranker not worth it? When candidates are already few (a small internal corpus), when queries are exact lookups (order numbers, error codes), or when latency is the product. Reranking is a precision tool for fuzzy questions over large corpora — applied everywhere else it is just a tax.
A reranker fixes order — nothing else.
- It inherits chunking damage. If the answer was cut in half at index time, the reranker sees half an answer and scores it honestly: badly.
- It cannot rescue recall. If the right document is not in the shortlist, no reranker will find it. Bad K is a recall bug, not a ranking bug.
- Domain mismatch. A cross-encoder trained on web text may not understand your legal or medical vocabulary. Rerankers are models; they have blind spots too.
- Latency and cost. Every candidate is a model call. The bill and the delay both scale with the shortlist size.
- Score worship. Reranker scores are relevance estimates, not truth. Thresholds copied from a blog post rarely survive contact with real data.
- Diminishing returns. After roughly the first 50 candidates, extra shortlist mostly buys latency, not quality.
The honest summary: reranking is the cheapest large quality win in retrieval — after the boring work of chunking and recall is done properly.
If you remember one thing
A reranker is a slower, more careful reader that looks at your query and each candidate together — and fixes the order the fast pass got approximately right.