Context Window: What It Is and Why It Isn't Enough

Technically, a context window is the maximum sequence length a model was trained and architected to process in a single forward pass, measured in tokens (roughly, pieces of words) rather than characters or words directly. A 200,000-token window can hold a lot of text, but the exact amount varies by content, since tokenization isn't one token per word.

Every part of a request competes for that same budget: system instructions, the conversation so far, any documents or memories retrieved for the current query, and the space left over for the model's own response. A context window isn't a place things get stored, it's a limited amount of space everything has to be squeezed into for one single request.

By 2026, context windows of one to two million tokens are common among frontier models, enough to hold an entire codebase or a long document collection in a single request. That's genuinely useful for bounded tasks: reviewing one large file end to end, or reasoning across a document without a retrieval step potentially missing something relevant.

It changed the calculus for smaller tasks too, letting teams skip building retrieval infrastructure for use cases where the whole relevant document set fits comfortably in a window. That's a real win for simplicity, but it's easy to over-generalize from it to something a large context window doesn't actually provide.

A context window resets between sessions no matter how large it is. Nothing about token capacity changes the fact that a new conversation starts with an empty window, so anything the model needs to "remember" from a prior session still has to come from somewhere outside the window entirely, which is exactly what a dedicated memory layer provides.

Cost scales with window size too. Processing more input tokens costs more per request, and attention computation scales worse than linearly as context length grows, so treating the context window as unlimited storage isn't just architecturally wrong, it gets expensive fast at any real usage volume.

Fitting more text into a window doesn't guarantee the model uses all of it equally well. The "Lost in the Middle" research found that models tend to attend more reliably to information near the beginning or end of a long context, with reduced accuracy for facts buried in the middle, even when everything technically fits within the limit.

That means padding a context window with marginally relevant material doesn't just cost more, it can actively hurt accuracy by diluting attention away from what actually matters. A smaller, carefully retrieved context frequently outperforms a larger, less curated one for exactly this reason.

stored doesn't try to replace the context window, it feeds it. Rather than dumping an agent's entire memory into every request, hybrid retrieval returns a compact, ranked set of the memories actually relevant to the current query, so what reaches the window is small, relevant, and cheap to process, regardless of how large the underlying memory graph has grown.

That's the practical answer to "why not just use a bigger window": a larger window raises the ceiling on what a single session can hold, but AI agent memory still needs a retrieval layer to decide what belongs in that window on any given request, and stored is built to be exactly that layer.

A token isn't a word or a character, it's a small chunk of text that a model's tokenizer decided to treat as one indivisible unit, and the mapping between the two varies quite a bit by content. Common, everyday English words are often a single token; rare words, unusual formatting, and non-English text frequently split into several tokens each, which is why two sentences of roughly the same length in characters can end up costing noticeably different amounts depending on what they're actually made of.

This matters for budgeting, not just theory. Code, JSON, and dense technical documentation tend to tokenize less efficiently than plain prose, so a limit that comfortably fits a novel's worth of narrative text can fill up considerably faster with a codebase or a table of structured data. Estimating capacity from word counts alone tends to be optimistic for exactly that reason, and it's easy to be caught off guard the first time a request that looked well within budget on paper gets rejected for exceeding the limit.

In practice, a request's token budget splits into a few competing categories: fixed overhead like system instructions and tool definitions, the conversation so far, anything retrieved for the current query, and headroom reserved for the model's response. Treating all of that as one undifferentiated pool tends to produce requests that either overflow unpredictably or leave far less room for retrieved content than a team assumes.

A reasonable starting discipline is to budget each category explicitly: cap fixed overhead, cap how much conversation history gets carried forward before it's summarized or trimmed, and treat whatever's left as the actual retrieval budget for the current query. That budget, not the advertised maximum window size, is the number that determines how much relevant memory can realistically be included on any given request.

It's worth revisiting that budget periodically rather than setting it once. Tool definitions grow as an agent gains capabilities, system instructions accumulate edge cases, and conversation summaries drift longer than intended, all of which quietly eat into the retrieval budget without anyone deciding that tradeoff on purpose. A budget that made sense at launch can silently starve retrieval of room six months later if nothing is watching it, so treating the split between overhead and retrieval as a metric worth tracking on a recurring basis, not a one-time decision made and forgotten, tends to catch the drift before it becomes a production problem.