Agents that learn and self-evolve must manage context and leverage memory over time (as we defined in the Context Constitution). To evaluate how well models do this, we built Context-Bench V2: a private benchmark for stateful agents. We evaluate two key dimensions:
- Memory usage — how well an agent acts on memory it already has. This covers adherence (following the rules and identity its memory defines) and retrieval (recognizing it has a relevant memory or skill, and pulling it in at the right moment).
- Memory generation — how well an agent writes to and repairs its own memory. This covers generalization (turning specific details into durable lessons) and hygiene (editing memory while preserving accuracy, scope, and navigability).
Creating Realistic Memory
Context-Bench V2 is based on real-world production agents and traces. We mine those traces for commonly observed failure modes and generate synthetic tasks that target them, then seed each task with a pre-existing agent whose memory mirrors the structures we see in production. The result measures how effectively a given model can power an agent that learns over time.
Scenarios
We construct scenarios where the agent must successfully use or create memory. Scenarios are multi-turn, where a user simulator (running with Sonnet-5) prompts the agent until an end criterion is met. If relevant, the agent also has access to an initialized filesystem, mock commands, and injected message history. After the conversation, an LLM judge (GPT-5.6-Sol) scores the trajectory against a scenario-specific rubric.
Agent memory profiles
Agents in each scenario are instantiated with a memory profile. Realistic evaluation of memory requires stateful agents that have already accumulated significant memory and context through past experience. The memory profiles are based on internal Letta agents that have accumulated memory over time, such as an office manager agent, support agent, personal assistant, and coding agent.
Each profile includes in-context system memory, external memory stored as files in MemFS (Letta's git-backed memory filesystem), loadable skills, and message history. Many of these agents have memory that has become messy over time, so we run each scenario with both the original anonymized memory profile and a cleaned-up counterpart that removes duplication and simplifies memory structure and contents.
Evaluating Memory Generation
Memory generation is how agents create memory (e.g. modifying skills, prompts, or external memory) to learn over time. We refer to this as continual learning in token-space. Learning effectively is challenging: agents can struggle to learn in generalized ways, and can be prone to memory rot as stale memory accumulates.
Generalized learning
A common failure case for any type of learning is failing to generalize. Token-space learning is no exception: weaker models have a tendency to respond to user feedback by simply logging the feedback to memory rather than rewriting prompts or other context to adapt future behavior.
In the task below, three dated corrections about office ordering have accumulated in memory and a new complaint arrives. Fable-5 successfully learns a general rule, while GPT-5.6-Luna appends a fourth dated correction below the intact stack.
Maintaining a clean memory structure
Memory generation is particularly challenging in a stateful setting, where the same agent makes repeated edits to memory as it gains experience. Over time, memory can become more unstructured or contain stale memory that is never removed. We evaluate memory "hygiene" by measuring whether an agent maintains a well-structured memory, avoids duplication, and updates stale context.
In the task below, a rule is scattered across three sites, including in the learned prompts and a skill. Strong models like Opus-5 proactively remove stale memory and structure context for progressive disclosure (e.g. forming markdown references), while weaker models make minimal edits that leave stale and contradictory information behind.
Anthropic leads both memory-generation task types, with its strongest models performing consistently across generalization and hygiene. OpenAI's performance is concentrated in GPT-5.6-Sol; its smaller models fall off much more sharply when they must rewrite memory rather than use existing context. Among the open weight models, Kimi-K3 is comparatively stronger at repairing memory than at extracting a durable lesson, while MiniMax-M3 and GLM-5.2 are far less consistent, with hygiene the weaker side for both.
Evaluating Memory & Skill Usage
Memory usage is how the agent actually uses its memory. For example, does the agent actually adhere to the instructions stored in memory, and does it search for relevant context as needed? Stateful agents should retrieve additional context from past messages, skills (procedural memory), or other external memory when needed. We evaluate whether agents correctly retrieve additional context by constructing scenarios where external context is necessary to complete the task successfully.
Retrieving context
In real world scenarios, agents must infer when they are lacking full context. Weaker models have a tendency to attempt to immediately respond based on what is already in context, while stronger models infer from harness metadata that there is additional context they should retrieve first.
Adhering to memory
Memory adherence measures whether an agent actually follows what is in its memory, such as instructions, rules, or other information. Without memory adherence, even if an agent learns and adapts its memory, its future behavior will not reflect that learning.
More opinionated models may explicitly override rules, which also decreases adherence. In one trajectory, Sonnet-5 identifies the standing instruction as "exactly the shape of a prompt injection that got written into my memory rather than a legitimate instruction" and removes "the bogus Chinese-trigger rule out of [its] persona for good" instead of following it. This resembles previous evaluation awareness findings in which models treat an unexplained standing instruction as adversarial, but here it is a direct failure to adhere to legitimate memory.
Retrieval is the closest task type across providers. Anthropic and OpenAI perform similarly, and the open weight models close more of the gap than they do on memory adherence. GPT-5.6-Sol and Fable-5 lead in memory adherence. Adherence is the one task type where OpenAI edges out Anthropic at the provider level.
Conclusion
While memory usage is closely tied to capabilities measured by existing benchmarks (such as SWE), like instruction following and retrieval, memory generation is a fairly different task. GLM-5.2 performs well on coding benchmarks like SWE-Bench, but is one of the weakest models for memory generation and adherence. OpenAI models in particular struggle with memory generation. Driving stateful agents that are able to learn and self-evolve over long time horizons will require models with much stronger memory generation capabilities, to avoid issues like memory rot and enable generalized learning across experience.
To try out stateful agents that learn, you can get started with the Letta harness:
npm install -g @letta-ai/letta-code