We corrected our own memory benchmark

From 27 June to 3 July, our benchmarks page showed a bar chart of LoCoMo answer accuracy: Kognite (61.2% by the end of that week), and next to it “Mem0 (pub.)” at 91.8%. The caption called the comparison indicative because the judge models differed. That caption was doing far more work than a caption should. On 3 July we took the chart down and replaced it with a comparison we had actually measured. Both numbers in the new one are lower. They are also true.

This is what was wrong, what replaced it, and what the replacement still doesn’t tell you.

What was wrong with the chart

The 61.2% was ours: our harness, our answer prompt, deepseek-chat as the judge. The 91.8% was a figure taken from Mem0’s website, produced by their own setup. Putting them on one axis implied they measured the same thing. They didn’t, and the chart wasn’t even flattering — it put us 30 points behind. Wrong in the unflattering direction is still wrong.

A LoCoMo accuracy score is not a property of a memory system alone. It is the system plus every decision the harness makes around it: how the conversations are stored, how many retrieved memories reach the answering model, which model answers, with what prompt, and which model grades the answer. We had direct evidence of this in our own history. On 28 June our published number went from 52.0% to 61.2% in a single evening, on the same corpus with the same judge. The second run existed because the first had been started with the harness defaults; the re-run used a larger reranking pool and handed the answering model 30 retrieved memories instead of 5. Nine points from settings. Compared with that, a number from someone else’s harness is not a baseline. It is a different experiment.

What replaced it

We ran Mem0 ourselves — the open-source library, version 2.0.11 — through the same harness as Kognite, and held everything we could identical:

The only thing that differs is the retrieval engine: Kognite’s hybrid full-text + vector search with knowledge-graph expansion and a cross-encoder reranker, against Mem0’s default vector search.

Our own number went from 61.2% to 54.9%, largely because the fair run uses the plain configuration for both systems rather than the settings that had flattered ours. That 54.9%, with the +11.0 points beside it, is the figure on our home page today.

What the numbers mean

Recall@5

For each question, LoCoMo marks which parts of the conversation contain the evidence. Recall@5 is the share of questions where at least one of those supporting turns came back in the top 5 search results (matched on the text of the turn). It measures retrieval and nothing else — no language model is involved in scoring it. It is only well-defined when memories are stored verbatim, which is one reason this comparison stores them that way.

Answer accuracy

An answering model gets the question and only the retrieved memories, and writes an answer. A second model call — the judge — compares that answer with LoCoMo’s gold answer and says yes or no. Accuracy is the share of yeses. It measures the whole chain, so it depends on the answering model, the prompt and the judge as well as on retrieval.

The gap between the two is informative. Kognite found the evidence for 71.3% of questions but answered 54.9% correctly: the difference is questions lost after the right memory was already retrieved. The per-conversation chart on /benchmarks shows that gap for every conversation.

The judge model

Judges disagree. On 15 July we ran a different configuration — conversations ingested through our real memory-formation pipeline, which extracts atomic facts — and graded that run with two different judges: 71.4% with deepseek-chat as judge, 70.0% with GPT-4o. Same retrieval, same answering model, 1.4 points apart. A score reported without its judge is missing part of its definition.

Why like-for-like matters

Every knob above can move a score by more than the difference between two good systems. If one side of a comparison had 30 memories of context and the other had 5, or one was graded by a lenient judge and the other by a strict one, the chart measures the harnesses. Holding everything identical except the engine is the only way the difference means something about the engine.

The same caution applies to our own end-to-end result. /benchmarks shows the 70.0% GPT-4o-judged figure beside Mem0’s published LoCoMo results, which were graded by the same judge model. Sharing a judge removes one variable, not all of them — the harness, prompts and answer model still differ — which is why the head-to-head on our home page is the same-harness run instead.

Checking it yourself

Since 7 August the harness has been public under the MIT licence at github.com/global-software-development-eu/kognite-benchmark. It contains public-run.mjs, which runs the same stages — ingest, retrieve, score recall, answer, judge — against our public API with an ordinary API key, the LoCoMo adapter, a small sample dataset, and the result files behind our published numbers.

git clone https://github.com/global-software-development-eu/kognite-benchmark
cd kognite-benchmark

export KOGNITE_API_KEY=kgn_...     # app.kognite.dev → API keys
export OPENAI_API_KEY=sk-...       # or DEEPSEEK_API_KEY / ANTHROPIC_API_KEY

node public-run.mjs                # bundled sample: 18 questions, a smoke test

For the full LoCoMo set:

curl -O https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json
node adapt.mjs --source locomo --in locomo10.json --out datasets/locomo.json

# grade with a model that did not write the answers
export JUDGE_PROVIDER=openai JUDGE_MODEL=gpt-4o JUDGE_API_KEY=sk-...
DATASET=datasets/locomo.json node public-run.mjs

Three things to know before you do, all stated in the repository’s README as well:

• The sample is not a result. Eighteen questions prove the pipeline runs; don’t quote a number from it.
• The full set needs a paid plan. LoCoMo is roughly 5,882 memories; the free plan creates 5,000 a month. The runner stops and says so rather than reporting a degraded score.
• public-run.mjs did not produce our published figures. Those came from our internal runner, which isolates each conversation in its own scope — a privilege a public key doesn’t have. Expect your numbers to differ. The runner we used for the Mem0 side is not in the repository either; the table above lists the settings you would need to replicate it.

What this number does not tell you

• It is a retrieval comparison. Both systems received the same verbatim turns. Mem0’s own memory extraction was switched off, and so was ours. It answers “given the same memories, which engine finds the right ones”, not “which product remembers better end to end”.
• Mem0 ran with our embedder and its default store. Its usual default is OpenAI embeddings, and it supports vector stores with keyword scoring. A stronger embedder or a hybrid store might narrow the gap. We fixed the embedder on purpose, to isolate the architecture, but it is a choice a Mem0 user wouldn’t necessarily make.
• It is one dataset. Ten long conversations. Some LoCoMo categories are small — open-domain has 96 questions, adversarial has 2 — so per-category numbers are noisier than the overall one.
• The judge was deepseek-chat, and LLM judges wobble. We saw about a point of run-to-run variation with the same judge. Kognite hasn’t used DeepSeek anywhere since 4 September, so a run using today’s defaults is not judge-comparable with these numbers unless you choose the same judge.
• It is dated. The run is from July 3, 2026. The system has changed since — faster reranking, a different extraction model — and the numbers describe the deployment as it was that day.
• The latency isn’t yours. Search p50 of 870 ms and p95 of 1,454 ms were measured next to the API. Your round trip adds to both.
• It says nothing about your data. Chat transcripts between two people are one shape of memory. An agent’s notes about a codebase are another. The only benchmark that settles that is yours.

The rule we took from this: every number we publish points at something you can check, and when a claim and its evidence disagree, the evidence wins — including when it makes us look worse. The full history is in the changelog entry for 3 July.