Reranking

ContextOS utilizes a two-stage retrieval pipeline. The first stage uses Hybrid Retrieval to quickly gather a wide pool of candidate memories. The second stage uses a Cross-Encoder Reranker to precisely re-score those candidates.

Why is Retrieval Ranking Insufficient?

Dense embeddings (bi-encoders) are incredibly fast because the document vector and the query vector are computed independently, allowing for rapid cosine similarity searches across millions of records.

However, this speed comes at the cost of precision. Because the query and the document are embedded separately, the model cannot capture the deep, word-by-word interaction between them. Bi-encoders often surface documents that share similar vocabulary but answer the wrong question.

Cross-Encoder Behavior

ContextOS solves this by taking the top K candidates from the initial hybrid retrieval pool and passing them to a Cross-Encoder.

A Cross-Encoder passes both the query and the candidate document into the Transformer network simultaneously. This allows the self-attention mechanisms to compare words in the query directly against words in the document, resulting in a significantly more accurate relevance score.

The Latency Tradeoff

Cross-encoders are computationally expensive. Running a cross-encoder across your entire database would take minutes or hours.

ContextOS navigates this tradeoff via the Candidate Pool:

1. Retrieve a wide net of ~100 candidates using fast Hybrid Retrieval.
2. Apply Reciprocal Rank Fusion (RRF) to merge the lists.
3. Slice the top pre_rerank_k candidates.
4. Late Hydration: ContextOS only fetches the full text content from the database for these top candidates, saving significant I/O time.
5. Pass only this small, highly-relevant subset into the Cross-Encoder.

By confining the reranker to a small pool, ContextOS improves precision dramatically while adding only a small latency penalty (~50-150ms depending on the model and hardware).

Configuration

Reranking is enabled by default. You can control the depth of the reranking stage with the following settings:

enable_reranker=True
pre_rerank_k=50       # How many candidates to fuse
reranker_top_n=20     # How many candidates to re-score