Short-Term vs Long-Term Memory in AI Agents

Short-term memory is the model's context window: the current conversation, recent tool outputs, and whatever's been explicitly included in the active prompt. It's fast to access, since it's already sitting in front of the model, but it's bounded by the window size and disappears entirely when the session ends.

This layer is where an agent does its actual reasoning: following the thread of the current task, referencing what was just said, and building on the last few tool calls. It's not meant to be durable, and trying to make it durable by simply never starting a new session runs straight into the context-window limits every long-context model still has.

Long-term memory lives outside the context window entirely, in a durable memory store the agent writes to and reads from across sessions. It's slower to access, since it requires a retrieval step, but it doesn't disappear when a session ends and isn't bounded by a fixed window size the way short-term memory is.

This is the layer that makes an agent feel consistent over time: it's where a user's preferences, a project's decisions, and past corrections live, available on the next session and the one after that, not just within the conversation where they were first mentioned.

The MemGPT paper made this split explicit by comparing it to how operating systems manage memory: a context window is like RAM, fast, directly accessible, but limited and volatile, while long-term memory is like disk, larger and durable, but requiring an explicit read or write operation to access rather than being immediately available.

That analogy extends usefully: just as an operating system pages data between RAM and disk as needed, a well-built agent needs a mechanism for moving relevant information from long-term memory into the context window when it's needed, and letting it fall out of the window (though not out of long-term storage) when it isn't.

In a working system, the two layers aren't independent, they're constantly exchanging information. At the start of a request, relevant long-term memories get retrieved and loaded into the short-term context window. At the end of a task, whatever's worth keeping from the short-term conversation gets written back out to long-term storage.

Getting this exchange right is most of what context engineering actually is: deciding what to retrieve into the window, how much of it, and what's worth writing back out once the task is done. Neither layer does much good without a disciplined process moving information between them.

stored is built specifically as the long-term layer: a durable, hybrid-searchable memory graph your agents write to as they work, reachable over a remote MCP endpoint so any client's short-term context window can pull relevant memories in on demand. The context window itself stays the client's responsibility; stored is what makes sure it isn't starting from nothing every session.

Because retrieval runs hybrid search across the graph, vectors, and keywords, what gets pulled into short-term context is a compact, ranked set, not a dump of everything the agent has ever learned, keeping the exchange between the two layers efficient as long-term memory grows.

A useful test for short-term vs long-term memory is durability: would this information still matter in a different session, with a different starting prompt, possibly in a different tool entirely? If yes, it belongs in long-term storage. The current step of a multi-step task, the exact wording of the last tool call, or a detail that's only relevant to finishing the request in front of the agent right now, all of that is fine to leave in short-term context and let disappear when the session ends.

Getting this wrong in either direction has a visible cost. Writing too much to long-term storage turns retrieval into a search problem over noise, since the memory store fills up with session-specific detail nobody will ever query for again. Writing too little means the agent relearns the same preferences and corrections in every new session, which is the exact failure long-term memory exists to prevent in the first place.

Applying the short-term vs long-term memory test consistently is more of a habit than a one-time design decision. Teams that revisit it regularly, checking every few weeks whether the store still reflects the durability rule rather than a pile of accumulated exceptions, tend to keep both layers useful far longer than teams that set the boundary once at launch and never look at it again.

The most common mistake is treating a large context window as a substitute for long-term memory: keeping every session's transcript in an ever-growing prompt instead of writing the durable parts out to a separate store. It works for a while, then quietly breaks once the transcript outgrows what fits, or once a new session starts with none of that history available at all, regardless of how large the window happened to be during the session that generated it.

The opposite mistake is retrieving too aggressively into short-term context: pulling in every long-term memory that's even loosely related to the current request, rather than a small, ranked, genuinely relevant set. That doesn't just waste tokens, it actively degrades output quality, since models attend less reliably to information buried in a long, cluttered context than to a short, well-chosen one.

A third, quieter mistake is letting the two layers drift out of sync: an agent updates a preference mid-session, that update stays in short-term context for the rest of the conversation, but nothing ever writes it back to long-term storage. The next session starts from the old preference again, and the correction has to happen all over, which is a frustrating, entirely avoidable failure mode for anyone using the agent regularly and expecting it to remember what it was just told.