ContextOS
Try ContextOS

Token Budgets

How ContextOS treats the context window as a constrained, finite resource.

The core philosophy of ContextOS is that retrieval must be decoupled from context selection. Just because you retrieve 50 relevant memories does not mean you can—or should—send them all to the LLM. Every LLM has a maximum context window, and large contexts increase latency and cost while degrading reasoning performance.

The Token Budget

ContextOS enforces a strict Token Budget. This is a hard limit on the total number of tokens that can be included in the final context payload.

When you initialize the ContextOrchestrator, you define the budget:

orchestrator = create_orchestrator(db=db, token_budget=4000)

Token Counting

ContextOS does not guess token sizes. It calculates the exact token count for every ContextItem at the time of insertion using `tiktoken`. These counts are stored in PostgreSQL, allowing the Memory Planner to make rapid decisions without recalculating string lengths.

The Selection Process

Once candidates are retrieved, fused, and reranked, they enter the Memory Planner. The Memory Planner executes the following algorithm:

  1. Guarantee System Context: Any candidate with source=SYSTEM is automatically selected. Their token counts are deducted from the available budget first.
  2. Score Candidates: All remaining candidates are given a final score based on semantic relevance (from the reranker) and chronological recency.
  3. Sort by Score: Candidates are sorted in descending order of their final hybrid score.
  4. Allocate Tokens: The planner iterates through the sorted list. If a candidate's token_count fits within the remaining budget, it is selected.
  5. Budget Exhaustion: Once a candidate exceeds the remaining budget, it is rejected with the reason BUDGET_EXHAUSTED. The planner continues to see if smaller candidates can fit into the remaining gap (greedy knapsack packing).
Note: ContextOS implements a greedy selection approach rather than a perfect 0/1 knapsack algorithm to optimize for latency.

Trace Decisions

Because ContextOS treats the context window as a constrained resource, it is highly observable. Every candidate evaluated by the planner is logged in the ExecutionTrace with an explicit decision reason:

  • SYSTEM_PROMPT - Guaranteed inclusion.
  • HIGH_SEMANTIC_SCORE - Selected primarily due to relevance.
  • HIGH_RECENCY - Selected primarily because it occurred recently in the session.
  • BUDGET_EXHAUSTED - Rejected because it would exceed the token budget.