Our A/B test measured a background job, not our change

On 22 September we shipped three new retrieval signals: a small bonus for recent memories, time windows for queries like “last week”, and a lookup that finds memories linked to entities named in the query. Then we measured them on LoCoMo. The first result said accuracy went up 5.5 points and recall went up 10.6. We withdrew it the same day. The benchmark hadn’t measured our change at all. It had measured a background job that was still running.

This post shows what that looked like, how we caught it, and the three-step check we now run first. If your memory layer enriches data asynchronously (entity extraction, graph building, summarisation), your benchmark can make the same mistake.

The result that looked too good

We ran four arms on one deployed build, with the same 235 LoCoMo questions and the same judge model (gpt-4o-mini). The only difference between arms was which retrieval signals the search request enabled.

Read the last row. The arm with fewer signals than NEW beat it by 15 points of accuracy. It also beat every other arm, and every arm beat the one before it. The scores rise with run order, and have nothing to do with configuration. A difference between configurations can’t explain that pattern. Something that changes over time can.

What was moving

When a memory is written to Kognite, it is stored and searchable at once. A background job then extracts entities and relationships from it and adds them to the knowledge graph, which retrieval uses to reach related memories. That job had not finished with the benchmark corpus. When the runs started, one of the two conversations had 150 of its 419 memories processed. Between database checks taken during the runs, its entity count climbed from 46 to 60 to 64.

So every arm searched a richer corpus than the arm before it. Both graph expansion and the new entity lookup get stronger as extraction progresses, so each arm’s score included all the enrichment done while the earlier arms ran. We were ranking four snapshots of a moving corpus and calling the result a comparison of configurations.

Two numbers from that session survive, because they were measured against the corpus as it stood at that moment rather than compared across runs. The entity lookup grows the candidate pool from 32 to 56 memories. Because more candidates reach the reranker, it adds 273 ms at p50. And 940 queries across the four arms returned 0 errors.

The run that counts

We let extraction finish on every benchmark conversation. Then we took a fingerprint of the corpus: memories, memories processed, entities and links per conversation. We ran three arms in the order OLD, NEW, OLD, and took the fingerprint again. The two fingerprints were identical, so the corpus did not move.

The two OLD rows are the point of the exercise. Same code, same settings, same corpus, and accuracy still differs by 2.1 points, because the answering and judging models are not deterministic. That gap is the noise floor, and any difference smaller than it means nothing.

• Accuracy: NEW is 0.25 points below the OLD average. Inside the noise, so no change.
• Recall@5: 0.5 points higher, on a metric that didn’t move at all between the two OLD runs, but from a single NEW run: marginal at best.
• Latency: 140 ms slower at p50. Real, and a cost.

So the honest result is no regression, no demonstrated gain and a measurable price. We had predicted it before the run. In LoCoMo every memory is written at the same moment, so a recency bonus has nothing to separate. The dataset’s dates live inside the text, not in the timestamps, so time windows either match everything or nothing. And on this corpus the graph already reaches most of what the entity lookup finds.

What the moving corpus did to absolute numbers

The same OLD configuration scored 31.1% accuracy with the corpus 150/419 processed, and 51.5% with it 419/419 processed. That is a 20-point swing from nothing but a background job finishing.

We have to apply this to ourselves. The LoCoMo run on our benchmarks page dates from 3 July. Its results file records the model, the timings and the scores, but not how far graph extraction had got when the questions ran. So we cannot tell you whether that number was measured on a finished corpus. The next number we publish will include the corpus fingerprint.

The check, before any benchmark of a memory layer

1. Drain every asynchronous stage first. Extraction, graph building, summarisation, embedding backfill. Then confirm the corpus is still: take the same counts twice, minutes apart, and require them to be identical.
2. Run an order control. Run the baseline, then the candidate, then the baseline again. If the two baseline runs disagree by more than the difference you’re trying to measure, you haven’t measured anything yet.
3. Report the fingerprint and the noise floor with the result. A score without the state of the corpus it ran on can’t be reproduced, even by you.

We also changed how we A/B at all. The signals can now be switched per request on the same deployed build, so a comparison differs in one variable only. Before, it compared two deployments, sometimes judged by two different models.

Why the signals stayed

Real memory stores have memories of different ages, and real users ask “what did we decide last week?” LoCoMo contains neither, so it can’t see these signals work, and we make no benchmark claim for them. Whether the entity lookup is worth 140 ms by default is still open. Answering it needs a dataset with real age differences and real time-based questions. That is a data problem, and we’d rather say so than tune until a benchmark agrees.

The first result was the one that flattered the change, and it was an artifact. The most useful habit this left us with is simple: when a number looks good, rerun the baseline before you believe it.