Evaluation

ContextOS includes a built-in evaluation harness designed to measure the effectiveness of the entire pipeline, from retrieval to final context planning. We evaluate against metrics that matter for LLM context windows, not just traditional search engines.

The Evaluation Dataset

Our benchmarks run against a synthetic dataset of realistic software engineering team communications (Slack messages, standup notes, Jira tickets).

Why is the dataset repetitive? The dataset is intentionally designed with highly overlapping and repetitive statements (e.g. "Next.js app router caching is causing stale data" vs "The app router is caching stale data on the dashboard"). This forces the Retrieval Engine to work hard to distinguish between slight variations and tests the Planner's ability to prioritize when faced with many similar candidates.

Measured Metrics

• MRR (Mean Reciprocal Rank): Evaluates the absolute ranking quality of the first correct answer in the candidate pool.
• NDCG (Normalized Discounted Cumulative Gain): Evaluates the overall ranking quality across all retrieved items.
• Candidate Recall: The percentage of correct targets that were successfully retrieved into the candidate pool.
• Planner Recall: The percentage of correct targets that actually survived the memory planner and made it into the final token budget. (This is the ultimate metric for ContextOS).
• Latency: Measured in milliseconds across embedding, retrieval, fusion, reranking, and planning.

Evaluated Baselines

The harness compares multiple configurations:

1. Naive Dense: Pure pgvector cosine similarity.
2. Hybrid (Dense + BM25): Combining semantic and lexical search via RRF.
3. Hybrid + Reranking: Applying a cross-encoder to the top hybrid candidates.

Reproducing the Benchmarks

We do not publish static benchmark numbers because they depend heavily on your specific embedding models, hardware, and postgres configuration. Instead, we encourage you to run the evaluation harness yourself.

To reproduce the evaluation locally:

# 1. Start a fresh database
docker compose down -v
docker compose up -d
alembic upgrade head

# 2. Run the evaluation script
python -m eval.run_eval

Current Limitations

• The evaluation currently relies on single-hop QA pairs. It does not heavily test multi-hop reasoning or cross-document synthesis.
• The generated dataset uses a fixed vocabulary distribution which may over-reward exact-match BM25 retrieval compared to real-world noisy data.