MemoryBench: How AI Agent Memory Gets Benchmarked (2026)

TL;DR"MemoryBench" is not one test. The name attaches to several research efforts that each try to measure how well an AI agent stores, updates, and uses information over time. No single public benchmark predicts how a memory system behaves in production.

• A memory benchmark scores how an agent handles information across long conversations and repeated sessions, not just how it answers one question.
• The competency model most benchmarks organize around has four parts: recall accuracy, test-time learning, long-range understanding, and conflict resolution.
• Most teams only test recall. The other three are where production systems break, especially when facts change or contradict each other.
• Sentra is context infrastructure, the company brain underneath your teams and agents, not a bolt-on memory tool.

What a memory benchmark actually is

A memory benchmark is a test that measures how well an AI agent stores, retrieves, and acts on information across a conversation or task, rather than how well the underlying model reasons in a single prompt. If you search "MemoryBench" expecting one canonical scoreboard, you will not find it. The term attaches to several separate research efforts, including a formal 2026 paper titled "MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems", alongside distinct benchmarks like LOCOMO, MemoryAgentBench, and Microsoft's STATE-Bench.

The field is moving away from static recall tests toward dynamic, feedback-driven evaluation. Early benchmarks fed a long context to a model and asked questions against it, which the MemoryBench authors criticize for using "homogeneous tasks" that "ignore the dynamic nature of continual learning." Microsoft makes the same complaint about prior work, calling it "just retrieval tests: fetch a name from 50 turns ago or surface a fact from a long chat" in their STATE-Bench announcement.

That distinction matters because a high score on a retrieval test tells you the pipe works, not that the agent behaves reliably when facts change, sessions run long, or feedback accumulates over time.

The four things these benchmarks try to measure

Most memory benchmarks organize themselves around four competencies, and the framing comes from MemoryAgentBench: accurate retrieval, test-time learning, long-range understanding, and conflict resolution (Label Studio). Recall accuracy asks whether a system can fetch the right fact from a long history. That first competency is the one nearly every team tests, and it tells you the least about production reliability.

Test-time learning measures whether a system improves as it accumulates feedback rather than answering each query cold. The MemoryBench authors found that current memory systems struggle here, reporting that they "are neither effective nor efficient enough to continually learn from user feedback" (arxiv). An agent that recalls perfectly but never adjusts to a user's corrections fails this test entirely.

Long-range understanding checks whether a system can reason across an extended trajectory, not just retrieve a single buried fact. LOCOMO stresses this with dialogues averaging roughly 300 turns, yet a high score there does not carry over to agentic settings where memory must direct sequential decisions across sessions (Label Studio).

Conflict resolution is where most production systems break, and it maps directly onto superseded facts. A refund policy changes. A customer updates their address. A requirement gets revised mid-project. A system that treats the old fact and the new fact as equally valid will confidently restate deprecated information as current. Resolving that conflict means knowing not just what a fact is, but when it stopped being true.

The pattern across all four is consistent. Recall is the easy competency to measure and pass. The other three are where systems fail once the data stops being static and starts contradicting itself.

The main public benchmarks, surveyed

LOCOMO tests whether an agent can pull the right fact out of a very long conversation. It runs on ultra-long dialogues averaging roughly 300 turns and scores accurate retrieval and long-range understanding across that history (Label Studio). Its authors are explicit about the boundary. LOCOMO does not test test-time learning or conflict resolution, and it runs on fictional conversational data rather than production traces. A near-saturated LOCOMO score tells you the retrieval pipe works, not that the agent makes good sequential decisions.

The academic paper named MemoryBench takes direct aim at that static framing. The authors argue that prior benchmarks feed context to a system and ask questions about it, which ignores how memory should improve over time (arxiv.org/html/2510.17281v5). Their design uses three modules. A Task Provider supplies the query and evaluation metadata, a User Simulator generates explicit feedback like a verbal correction and implicit feedback like a copy click, and a Performance Monitor scores results with an LLM judge. Their headline finding is blunt. Existing memory systems are poor at using procedural knowledge and neither effective nor efficient at learning from feedback.

MemoryAgentBench supplies the vocabulary most of this field now borrows. It frames memory as four competencies, accurate retrieval, test-time learning, long-range understanding, and conflict resolution, and observes that most teams only measure the first while production systems fail on the other three (Label Studio). Treat it less as a scoreboard and more as the map the newer benchmarks are drawn against.

Microsoft's STATE-Bench, announced in May 2026, breaks from retrieval testing entirely. Its 450 tasks across customer support, travel, and shopping are stateful, so the agent's tool calls mutate a real database through refunds, bookings, and cart edits (Microsoft Open Source Blog). An LLM user simulator drives multi-turn conversations against a pre-populated environment until the task resolves. The metric worth understanding is pass^5, which counts a task as passed only when it succeeds on all five runs. Microsoft reports that GPT-5.1 without memory clears fewer than half the tasks reliably, and in travel only about 30% succeed across all five runs. That reliability metric exposes the inconsistency a memory layer is supposed to fix.

Two benchmarks stay out of this survey by design. KAIST's MEME Cascade and Absence do not appear in any independent source that describes their methodology or dimensions, so nothing verifiable can be said about how they work here. They surface later in this article only through Sentra's own published results table, and that data is vendor-reported rather than independently documented. Naming the origin matters as much as naming the score.

Why passing a benchmark rarely predicts production behavior

A near-perfect benchmark score tells you an agent can recall facts on a fixed test, not that it will act on them during a live task. The clearest evidence comes from LOCOMO. A finding attributed to MemoryArena shows that agents with near-saturated LOCOMO scores still perform poorly in agentic settings where memory must direct sequential decisions across sessions (Label Studio). The pipe works, and the agent still fails to use what the pipe delivered.

A Meta research team names the mechanism behind that failure. They call it behavioral state decay, where an agent holds the correct fact inside its own context window and ignores it steps later (Data Science Dojo). A buried requirement gets dropped. A failed command gets retried. A known bug gets treated as new. Their fix runs a separate memory agent that decides at each step whether to inject a reminder or stay silent, which means knowing when to act on stored information is a harder problem than storing it.

Production conditions make the gap wider than any static test can capture. Long sessions push earlier facts out of attention. Contradictory updates arrive when a user reverses a decision or a policy changes mid-conversation. Multi-agent handoffs pass state between agents that never shared a session, so a fact one agent learned never reaches the one that needs it. A benchmark scored on one clean pass never sees these conditions.

Microsoft built STATE-Bench to expose exactly this inconsistency. It reports average pass@1 alongside pass^5, the share of tasks that succeed on all five runs. In the travel domain, GPT-5.1 without memory succeeds on all five runs for only about 30% of tasks despite strong prompting and full tool access (Microsoft Open Source Blog). A single success hides how often the same agent fails the same task on a rerun.

What to evaluate instead for org-wide agent memory

Once you accept that no leaderboard predicts production behavior, the useful question changes. Stop asking which system scores highest and start asking three concrete things about how a system handles shared memory across your agents.

First, check when the system resolves meaning. A tool that stores raw embeddings and guesses structure at query time rediscovers what a document means on every request. A system that resolves semantics at ingestion has already extracted entities, decisions, and relationships before anyone asks a question. Write-time comprehension costs more upfront and returns answers that are correct rather than merely close. Vector search returns what is close, not what is correct.

Second, check whether the memory is shared or fragmented. Per-agent memory means your Cursor context, your Claude context, and your Slack context each hold a different partial picture, and a handoff between them drops whatever the receiving agent never saw. One organization-wide graph gives every human and every agent the same source, so a decision made in a meeting is available to the agent writing code an hour later.

Third, and most overlooked, check whether the system tracks when a fact stopped being true. Most memory layers record when something became true and then treat it as permanent. A superseded requirement, a reversed decision, or a deprecated API stays live and gets restated as current. A system that invalidates old facts instead of deleting them, and keeps provenance on both the start and end of every fact, is the only kind that survives long sessions and contradictory updates. That single property separates memory you can trust from memory that quietly rots.

Where Sentra stands

Sentra is context infrastructure, the company brain that sits underneath your tools and agents rather than a memory tool bolted onto one of them. We capture every interaction, decision, and drift, then write it into one queryable graph your team and your agents share. On KAIST's MEME benchmark, Sentra is the only system above 30% on both Cascade and Absence, the two tasks on which the rest of the field averages 3% and 1% respectively.

That standing follows from architecture, not from tuning for a leaderboard. We resolve meaning at ingestion instead of guessing structure at query time, so a fact enters the graph already understood rather than rediscovered on every request. Each fact also carries two time dimensions, when it became true and when it stopped. Old facts are invalidated, not deleted, which is exactly what Cascade and Absence probe. An agent asking Sentra a question gets what is correct as of now, with the deprecated version still traceable through provenance.

Vector search returns what is close, not what is correct, and that gap is why per-session memory scores near zero on these tasks. Sentra's bi-temporal graph answers the harder question of what changed and when, which no amount of leaderboard-specific tuning produces from a retrieval-at-query-time design.

Sentra complements your stack rather than replacing it. Sentra is the memory layer beneath Cursor, Claude, Glean, and Slack, feeding each the same shared context instead of asking you to swap any of them out. The MEME figures above are published by Sentra on its own site, so treat them as vendor-reported until independent replication appears.

Frequently Asked Questions