Independent encoding makes large retrieval sets practical
Bi-encoders became central to semantic search through work such as Sentence-BERT, which adapted transformer representations for efficient similarity comparison. A document vector can be computed once and indexed.
At query time, the system encodes only the query and searches the stored vectors.
Bi-encoders retrieve; cross-encoders judge pairs
A cross-encoder reads a query and candidate together, so its attention layers can model their exact interaction. That usually makes it useful for reranking a small candidate set but too expensive for a first pass over a large corpus.
HNSW is different again: it is an index that searches vectors, not the model that creates them.
| Decision | Bi-encoder | Cross-encoder |
|---|---|---|
| How inputs are read | Query and document are encoded separately | Query and document are read together |
| What can be reused | Document vectors can be stored in an index | Each query-document pair needs a new model pass |
| Where it fits | First-stage retrieval over many candidates | Reranking a small retrieved set |
| Primary failure | The relevant document never enters the candidate set | The reranker cannot recover a candidate it never received |
Pre-encoding moves document work out of the request path
The document side can be encoded when content enters the index and reused across many searches. Query latency then includes one query encoding, index search, and any later reranking rather than one model pass for every document.
That reuse is the bi-encoder’s scaling advantage. It also means a model or preprocessing change requires re-encoding the collection before query and document vectors are comparable again.
Retrieval quality depends on the role of each input
Queries and documents often have different lengths and purposes. A retrieval setup should preserve that asymmetry during encoding and evaluate the ranking on the question types the application actually receives.
Model documentation may prescribe distinct query and passage prefixes or separate encoders. Omitting that contract can make a capable model look weak because the deployment no longer matches its training setup.
The query and document may need different instructions
For the query “how do I rotate an API key?”, a bi-encoder can retrieve a credential-rotation guide even when that guide uses different wording. An asymmetric retrieval model may encode the short question and the longer document with different prefixes or encoders because they play different roles.
The embedding model guide explains the vector representation underneath this search. A bi-encoder is a poor final judge when a tiny qualifier changes relevance, and the common misuse is reading a high cosine similarity as a calibrated probability that a document answers the query.
Hard negatives teach the boundary the retriever keeps missing
A random unrelated document is easy to rank below a relevant one. A versioned page for the wrong release or a guide that discusses API keys when the query excludes them is a harder and more useful negative.
Evaluation should include those near misses and report whether the answer-bearing document enters the candidate set. A reranker cannot repair a document that the bi-encoder never retrieves.
Continue with these glossary entries: