Context Compression: Fitting More Into a Finite Window

Agent context accumulates fast: every tool call adds output, every retrieved memory adds text, and long conversation histories pile up across a session. Left unmanaged, that accumulation eventually exceeds the context window entirely, or, more commonly, stays technically within the limit while padding the request with far more than the current task actually needs.

Both outcomes are costly. Hitting the window limit means the model silently loses access to early information. Staying under the limit with excess padding still costs more per request and, per research on how models handle long inputs, can dilute attention away from what actually matters, a pattern documented in the "Lost in the Middle" study.

Summarization compresses a long history into a shorter representation that preserves the gist, trading some detail for a large reduction in tokens; it works well for conversation history where exact wording rarely matters as much as the underlying facts and decisions. Selective retrieval compresses differently: instead of shortening everything, it excludes what's irrelevant to the current query entirely, which is effectively what hybrid retrieval does for memory.

A third approach, structural compression, replaces verbose raw text with a more compact representation of the same information, storing an extracted fact or a graph relationship instead of the full paragraph it was derived from. This is often the highest-leverage form of compression for memory specifically, since a fact usually carries the same value in ten words as it did in a hundred.

Research on this problem has moved quickly. A January 2026 paper on active context compression reported a 22.7% reduction in tokens used while maintaining the same task accuracy, a meaningful result given how directly token count maps to inference cost at scale. The core idea in that line of research is dynamic: compress more aggressively where content is less relevant to the current task, and preserve detail where it matters.

That dynamic approach is the direction most production systems are converging on: static, uniform summarization tends to lose detail unevenly, cutting corners on content that later turns out to matter. Compression that's aware of what the current request actually needs consistently outperforms compression applied blindly to everything.

It's worth distinguishing compression from the blunter alternative: just truncating old context once the window fills up. Truncation is compression with zero regard for relevance, it drops whatever happens to be oldest, regardless of whether it's still needed. That's rarely the right tradeoff, since relevance and recency aren't the same thing; a decision made weeks ago can matter more than a message sent five minutes ago.

Good context compression is relevance-aware rather than recency-aware: it keeps what's still useful for the task at hand and compresses or drops what isn't, independent of how long ago it was written. That distinction is exactly why retrieval-based systems tend to outperform simple sliding-window truncation for anything beyond a short conversation.

Token reduction on its own is a misleading metric, since dropping content indiscriminately also reduces tokens and clearly makes retrieval worse. A useful evaluation of context compression has to measure both sides at once: how much smaller the request got, and whether task accuracy held steady on the same benchmark before and after compression was applied. Token reduction without a paired accuracy check is close to meaningless as a signal of whether compression actually worked.

In practice, teams that get this right run compression changes through the same evaluation set they use for retrieval changes generally: a fixed set of representative queries with known-good answers, so a compression technique that quietly drops something load-bearing shows up as a regression immediately rather than surfacing later as a confusing production issue. Tracking both token reduction and accuracy on that same fixed set, over time, also makes it obvious when a compression setting has drifted too aggressive.

The most common mistake is compressing uniformly: applying the same summarization or truncation to everything in the request regardless of how central it is to the current task. This tends to produce a fixed, predictable amount of token reduction but degrades accuracy unevenly, since the content that happens to get summarized away is sometimes exactly what the model needed for the specific question being asked.

A second common mistake is compressing too early, before it's known what the current query actually needs. Compressing at write time, when a memory is stored, forecloses information that a future, differently-worded query might have needed in full. It's usually better to store detail once and compress or filter it at retrieval time, when the actual information need is known, rather than guessing upfront what to discard, since a guess made at write time can't anticipate every future use of that same fact.

A third, subtler mistake is treating compression as a one-time pipeline step rather than an ongoing property of the system. Content that was safe to compress heavily six months ago can become load-bearing later as an agent's tasks evolve, so compression settings tuned once and left alone tend to drift out of sync with what the system actually needs over time, quietly degrading answer quality until someone notices and re-tunes them.

stored takes the structural and retrieval-based approach by default rather than trying to compress an ever-growing raw transcript after the fact. Every memory is stored as a compact, structured fact with graph relationships, not a verbatim paragraph, and hybrid retrieval returns only what's relevant to the current query, so compression happens naturally as a byproduct of good retrieval rather than as a separate summarization pass.

That means the compression problem most teams solve with a dedicated summarization pipeline is handled at write and query time instead: what reaches the model through AI agent memory is already close to the minimum useful representation of what the agent needs to know.