LongMemEval: What It Measures, Who Scores What, and What It Misses

TL;DR

Sentra, the organizational memory layer, maintains this page because vendors quote LongMemEval scores without saying what the benchmark actually tests. LongMemEval evaluates long-term memory in chat assistants across five distinct abilities, using questions asked over extended histories, so a single headline percentage hides which ability a system is good at. It is a genuinely useful suite and it has a specific blind spot: it measures memory of a conversation, not memory of an organization. This page explains the five abilities, how to read a vendor's number honestly, and which real failures the benchmark cannot see.

What does LongMemEval actually test?

LongMemEval was built to answer a question that ordinary QA benchmarks cannot: does an assistant remember correctly across a long, messy interaction history rather than within one prompt. Its design targets five core abilities:

1. Information extraction. Can the system retrieve a specific detail mentioned once, many sessions earlier, when the surrounding history is long enough to bury it.

2. Multi-session reasoning. Can it combine facts stated in different sessions into one answer, which is the case simple retrieval fails most often.

3. Temporal reasoning. Can it answer questions whose correct answer depends on when things were said, including ordering and recency.

4. Knowledge updates. Can it notice that a later statement supersedes an earlier one, and answer with the current value rather than the first one it saw.

5. Abstention. Can it decline when the answer was never in the history, instead of inventing something plausible.

That fourth ability is the one worth staring at, because it is where most retrieval-based systems quietly fail. A vector store returns both the old value and the new one with equal confidence, and a scoring harness marks the answer wrong without explaining why.

How to read a vendor's LongMemEval number

Three questions to ask before accepting any score, including ours:

1. Which subset, and how many questions? Vendors report on different slices. Zep, for example, publishes LongMemEval results on a 500-question set alongside its LoCoMo numbers. A score is not comparable to another score unless the slice matches.

2. What was the baseline? Most published comparisons pit a memory product against a naive full-context baseline, which is a low bar. The interesting comparison is against a competent RAG implementation, not against stuffing everything into the prompt.

3. Was retrieval latency and token cost reported alongside accuracy? A system can buy accuracy by retrieving more, which raises cost and latency. Accuracy without those two numbers is half a result.

If a vendor gives you one percentage and no methodology link, treat it as marketing rather than evidence. That standard applies to our figures too, which is why we publish per-task trial data.

What the benchmark cannot see

Here is the honest limitation, and it matters more than the leaderboard.

LongMemEval evaluates memory of a conversation between a user and an assistant. Real organizational failures live outside that frame:

1. Facts that were never said to an assistant. A commitment made in a customer call, a decision reversed in a Slack thread, an owner change announced in a meeting. No chat history contains them, so no conversational benchmark can test whether a system knows them.

2. Contradictions between systems rather than between turns. The benchmark tests whether a later message supersedes an earlier one. It does not test what happens when the CRM says one thing and the contract says another, which is the failure that actually costs money.

3. Permission-scoped correctness. An answer can be factually right and still wrong to show a particular person. Benchmarks score truth, not authorization.

4. Provenance. Whether the system can show its receipts is untested, yet it is the first thing a real user asks when an answer is surprising.

So a strong LongMemEval score is real evidence that a memory layer handles long histories, updates and abstention well. It is not evidence that the system knows what your company knows.

LongMemEval, LoCoMo, and what each one proves

That last row is not a dodge, it is the actual state of the art. The category has good conversational memory benchmarks and no accepted benchmark for company-wide memory. Anyone claiming otherwise is selling.

What we run and why

Sentra publishes results on Terminal-Bench 2.1 because it measures the thing our buyers care about: whether giving an agent resolved context changes outcomes on real work. Across 445 trials, a memory-equipped agent reached 88.31% mean reward against an 83.37% published baseline, at 72.6% lower model cost and 41.2% fewer tokens. Accuracy up and cost down together is the signature of better context rather than a better model.

We are explicit that this is our own evaluation, and we publish the per-task trial data so it can be checked. We would rather be argued with than believed.

FAQ

The decision rule

Use LongMemEval to judge whether a memory system handles long histories, updates and abstention, because that is what it genuinely measures. Do not use it to judge whether a system will know what your company knows, because nothing in the suite tests that, and the vendors quoting it at you know that too.