What is Reranking?
Reranking is a second retrieval stage in which a reranker — usually a cross-encoder model — reads each candidate passage together with the query and rescores it, so the handful of chunks passed to the language model are the genuinely most relevant ones.
At a glance
- A reranker scores query + passage pairs jointly, unlike embeddings which are compared independently.
- Standard pattern: retrieve 50–100 candidates, rerank, keep the top 3–8 for the prompt.
- Fixes the common failure where the right document was retrieved but ranked too low to be used.
- Cross-encoders are far more accurate than vector similarity and far more expensive — hence two stages.
- Adds roughly 100–300 ms; the accuracy gain is usually the best latency trade in a RAG pipeline.
Bi-encoders vs. cross-encoders
Vector search uses a bi-encoder: the document and the query are embedded separately, and relevance is the cosine distance between two fixed vectors. That is what makes it fast — every document embedding is precomputed — and also what limits it, because the model never sees the query and the passage at the same time.
A cross-encoder reranker concatenates the query with a candidate passage and runs both through a transformer to produce a single relevance score. It can weigh negation, qualifiers, dates and entity matches that a cosine distance flattens. The cost is that nothing can be precomputed: scoring is one model call per candidate, so you can only afford it on a shortlist.
Where it fits in the pipeline
The production shape is retrieve-then-rerank. Hybrid search pulls 50–100 candidates optimised for recall, the reranker rescores them for precision, and only the top 3–8 chunks are assembled into the prompt. Retrieving a small top-k directly from the vector index — the naive pattern — forces you to be right on the first, weakest signal.
Options range from hosted APIs (Cohere Rerank, Voyage) to open cross-encoders you serve yourself (bge-reranker, mxbai-rerank). Some teams also use an LLM as the reranker for small candidate sets, or a lightweight ColBERT-style late-interaction model as a middle ground. Whichever you pick, measure it: reranking should visibly move context precision and answer faithfulness in your evaluation set, or it is only buying latency.
Frequently asked questions
What is a reranker?
A reranker is a model that takes the query and a candidate passage together and outputs a relevance score. Because it sees both texts at once, it judges relevance far more accurately than the independent vector comparison used during initial retrieval.
Why not just retrieve the top 3 chunks directly?
Vector similarity is a coarse signal, so the best passage often sits at rank 12 rather than rank 2. Retrieving 50–100 candidates and reranking them recovers those cases; retrieving only three means any ranking error is unrecoverable.
How much latency does reranking add?
Typically 100–300 ms for a shortlist of 50–100 passages, depending on the model and whether it is hosted or self-served. That is usually small next to generation time and is the highest-value accuracy spend in the pipeline.
Is reranking worth it for small knowledge bases?
Less so. With a few hundred well-chunked documents on a narrow topic, hybrid search alone often puts the right passage first. Reranking pays off as the corpus grows, topics overlap, or queries become conversational and ambiguous.
Putting Reranking to work?
We help data & AI teams design and ship this in production. Tell us what you're building and we'll point you at the shortest path.
Talk to our team