MemoryBench Explained: The 2026 AI Memory Benchmarks, Published Scores, and How to Test Your Own

• The Tsinghua MemoryBench measures continual learning from explicit and implicit feedback, and its authors report that current memory systems are neither effective nor efficient at learning from that feedback.
• LongMemEval (500 questions, ICLR 2025) and LoCoMo (conversations averaging 300 turns) are the most quoted chat-memory benchmarks, and most scores published on them were run by the vendor being scored.
• MEME (KAIST AI with Tübingen and NAVER AI Lab, May 2026) scores Cascade and Absence, two tasks its authors say no earlier benchmark scores, and the six memory systems in the paper average 3% and 1% on them.
• Sentra scores above 30% on both MEME Cascade and Absence (KAIST), the only system to clear that bar on both, in a Sentra-run evaluation described in the scores section below.

What Is MemoryBench?

MemoryBench most often refers to "MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems" by Qingyao Ai, Yichen Tang, Changyue Wang, Jianming Long, Weihang Su and Yiqun Liu of Tsinghua University, first posted to arXiv on 20 October 2025 (arXiv 2510.17281). It asks a different question from most memory tests: whether a system gets better at serving users by learning from their feedback while it runs, rather than whether it can recall a fact buried in a long input.

• Data. The authors collect 11 public datasets, including LoCoMo, DialSim, LexEval, JuDGE and WritingBench, across three domains (open, academic and legal), four input and output length formats and two languages, for about 20,000 cases.
• Feedback simulation. An LLM playing the user generates explicit feedback, such as a sentence saying whether an answer was good, and implicit feedback, such as a click on a copy button. Each dataset is split 4:1 into training and test sets, feedback is simulated on the training split, and the system is scored on the test split.
• Memory types. It scores both declarative memory, the facts a system has been given, and procedural memory, knowledge of how to do a task better.
• Finding. The authors report that existing memory systems are not good at using procedural knowledge to improve, and that the effectiveness and efficiency of state-of-the-art baselines are far from satisfying.
• Code and data. The pipeline, user simulators and baselines are released at github.com/THUIR/MemoryBench, with the dataset on Hugging Face.

The second MemoryBench is Supermemory's open-source framework, MIT licensed and first published in September 2025. It is a test harness rather than a dataset. It runs LoCoMo, LongMemEval and ConvoMem against providers including Supermemory, Mem0 and Zep, checkpoints every stage from ingestion to evaluation, and lets you swap the judge between GPT-4o, Claude and Gemini (Supermemory docs). A vendor score labelled "MemoryBench" usually means one of those datasets run through this harness.

One more name is easy to confuse with both. MemBench (Tan et al., ACL 2025 Findings) evaluates factual and reflective memory in participation and observation scenarios (arXiv 2506.21605) and is unrelated to either MemoryBench.

Decision rule: when someone quotes a MemoryBench score, ask whether it is the Tsinghua dataset or a dataset run through Supermemory's harness, and which judge model scored it, before comparing it with anything.

Which AI Memory Benchmarks Matter in 2026?

Seven benchmarks account for most published memory results in 2026, and they split by setting: chat histories, agent tasks that act on an environment, and evolving knowledge where facts depend on each other, which is closest to how an organization's facts change. Each row links to the primary paper or announcement.

None of the seven uses real organizational data or tests who is allowed to see a fact. MEME comes closest on dynamics, because its facts depend on one another, so a change to one should change or cast doubt on others. The rest score memory of one user's conversations or one agent's tasks. A longer breakdown of the five LongMemEval abilities is in LongMemEval: what it measures.

Decision rule: use LongMemEval or LoCoMo to compare with vendor numbers, MemoryArena or STATE-Bench when memory drives agent actions, and MEME when your facts change and depend on each other.

What Scores Have Been Published, and How Far Can You Trust Them?

Most published memory scores were run by the vendor that built the system, scored by an LLM judge the vendor chose, on a configuration the vendor tuned, so two numbers are comparable only when the dataset, answering model and judge match. The table lists figures that trace to a primary page, with who produced each one.

• The same system scores differently depending on who runs it. Mem0's paper put Zep at 65.99% on LoCoMo, and Zep's rerun put Zep at 75.14%, a gap Zep attributes to implementation errors in Mem0's setup. Neither run is independent.
• The judge is part of the score. MEME publishes its judge agreement (98.6% on 144 hand-labelled samples). Most vendor posts publish no agreement figure, so a few points of difference can come from the judge alone.
• Full context is a strong baseline. In Mem0's own LoCoMo table, the full-context run (72.90%) scored above both Mem0 variants, and in MEME, gpt-4.1-mini reading everything in context (0.36 overall) beat five of the six memory systems. Memory systems in these papers mostly win on cost and latency rather than accuracy.
• Sentra's figures are vendor-reported as well. The MEME result is Sentra's run of the published tasks, the Terminal-Bench result is an internal evaluation pending official verification, and Sentra's Harvey LAB write-up treats its 0.6-point mean-pass margin over Engram as within noise; its finding is that the training step was not needed.

Sentra scores above 30% on both MEME Cascade and Absence (KAIST), the only system to clear that bar on both. In the paper's main table, no memory system or in-context baseline exceeds 6% on Cascade, and only Sonnet 4.6 reading the full context passes 30% on Absence (35%, with 5% on Cascade). The paper's one configuration that partially closes the gap, a file-based agent using Claude Opus 4.7 at ingestion, costs about 70 times the baseline. Cascade and Absence are the tasks where a change to one fact must alter or cast doubt on others, which is what a bi-temporal knowledge graph records: when each fact became true and when it stopped.

Decision rule: compare two memory scores only when the dataset, answering model and judge all match, and give a third-party run more weight than any vendor's own, including Sentra's.

Why Does a High Memory Benchmark Score Rarely Predict Production Behavior?

A high score on a recall benchmark shows that retrieval works on a fixed test, and production failures come from three things those tests do not exercise: facts that change, inconsistent behavior across reruns, and memory that must direct actions. The published evidence for each is direct.

• Changed facts. A refund policy changes, a customer moves, or a requirement is revised mid-project. A system that holds the old fact and the new one as equally valid restates the old one as current. Of the benchmarks above, LongMemEval scores knowledge updates, MemoryAgentBench scores selective forgetting, and only MEME scores the downstream effects of an update.
• Reruns. Microsoft reports that GPT-5.1 without memory completes fewer than half of STATE-Bench tasks reliably, and only about 30% of travel tasks succeed on all five runs (Microsoft Open Source Blog). A single pass hides that rate.
• Memory that drives actions. The MemoryArena authors report that agents with near-saturated LoCoMo performance perform poorly when memory from earlier sessions has to guide later actions (arXiv 2602.16313).
• Shared state. Every benchmark above scores one user or one agent. When several agents and people work from different copies of memory, a fact one of them learned never reaches the one that needs it, and no public benchmark measures that handoff.

Decision rule: treat a recall score as a minimum requirement, and decide on how a system handles changed facts and reruns.

How Should You Benchmark AI Memory for Your Own Team?

Build a test set from your own sources, weighted toward questions whose correct answer changes over time or depends on who is asking, and score it with a validated judge over enough questions to separate real differences from noise. Public benchmarks tell you which systems deserve a pilot; this procedure tells you which one works on your data.

• 1. Freeze a snapshot of three to five real sources. Use the systems your agents will read, such as chat, a ticket tracker, a CRM and a document store, and record the snapshot date so every system under test ingests the same data.
• 2. Write at least 200 questions with gold answers and a cited source for each. At 200 questions, a 70% score has a 95% confidence interval of about plus or minus 6 points, so smaller gaps between systems are noise; 400 questions narrow it to about 4.5 points.
• 3. Allocate questions to the failure modes public benchmarks miss. Include current-value questions where a price, owner or date changed after the snapshot, dependency questions where one change should alter another fact, absence questions where the honest answer is that the fact is unknown or no longer certain, deletion questions about something removed that must not return, and conflict questions where two sources disagree and one is newer.
• 4. Test permissions with paired askers. Ask the same question as two people with different access, and score any answer that reveals restricted content as a failure regardless of accuracy, because none of the public benchmarks test access control.
• 5. Change 20 to 30 facts at the source after ingestion. Re-ask those questions after a fixed interval, and record the freshness lag, the time until each system returns the new value, per source.
• 6. Hold the answering model fixed and validate the judge. Give every system the same answering model and prompt, score with an LLM judge and a written rubric, and hand-label at least 100 answers to measure judge agreement before trusting the scores. MEME's 98.6% agreement on 144 samples is a useful reference point.
• 7. Run every question several times and report pass^k. Report the share of questions answered correctly on every run alongside the mean, as STATE-Bench does with pass^5, because a system that is right on average and wrong on reruns fails in production.
• 8. Record cost and latency per answer. Track ingestion cost, tokens per answer and p95 latency, and include a full-context baseline, since full context often matches memory systems on accuracy and loses on cost.

Run the current-value and freshness questions again each month after rollout, because accuracy on changing facts degrades in ways a one-time test cannot show. How agent memory accuracy holds up over time covers the patterns to watch.

Decision rule: shortlist on public benchmarks, choose on your own test set, treat any permission leak as disqualifying, and rank the remaining systems on current-value and dependency accuracy before overall accuracy.

Frequently Asked Questions