Joint attention spends more work on fewer candidates
A cross-encoder places the query and one candidate in the same model input. Because their tokens can interact through attention, the model can judge details that a single geometric comparison between independently produced vectors may hide.
The cost is repeated inference: every query-candidate pair needs a forward pass. That makes cross-encoders natural rerankers and poor first-stage retrievers for a large collection.
A negated requirement can reverse the vector order
For the query “authentication without API keys,” a semantically similar API-key guide may rank above a page about workload identity. A cross-encoder can inspect the negation and the candidate text together, then move the workload-identity page upward.
A bi-encoder creates reusable vectors for candidate retrieval. A large generative model can also score documents, but generation is not required for the pairwise relevance task.
The candidate set limits what reranking can repair
The RAG reranking guide explains the two-stage architecture. Cross-encoder reranking does not apply if the relevant document never enters the candidate set, and the common misuse is tuning the reranker while ignoring first-stage recall.
A useful diagnosis records the relevant document’s position before and after reranking. That separates a reranker that fails to reorder a good candidate from a retriever that never supplied one.
Reranking depth is a budget decision
Scoring more candidates gives the cross-encoder more opportunities to recover the best passage, but the request cost grows with the number and length of pairs. Documents may need trimming or passage-level scoring to fit the model input.
Evaluation should separate candidate recall from reranker ordering. If the answer-bearing passage appears in the candidate set but finishes too low, the reranker is implicated; if it never appears, changing the cross-encoder cannot fix the retrieval failure.
Continue with these glossary entries: