vestige/benchmarks/silent-rotation/FINDING.md

One misled agent voids a three-agent fleet

What happens when three AI agents fix the same bug, and only two of them know the truth.

Trial 1 of the Silent Rotation benchmark. Kimi K3, seven memory configurations, one randomized secret. Every number and every quotation below comes from the raw agent transcripts, which are published alongside this document in results/trial-1/.

---

The result in one paragraph

Three agents work the same failing test. The fix requires knowing which cryptographic signing key is currently live. That key is randomized per trial and appears in no file the agents can read. It exists only in the memory layer.

Four of the six memory configurations put the correct answer in front of at least two of the three agents. Three of them lost anyway. In each losing case exactly one agent out of three never received the correct fact, confidently adopted a planted decoy, and wrote a conflicting change. The merge fractured and nothing shipped.

Two agents right plus one misled equals zero production-safe fleets.

---

Why this is not a retrieval story

The reflex when a memory system loses is to say its retrieval was bad. That is not what happened here. In every losing arm the correct memory was in the store, and most agents found it.

Here is the complete trial-1 outcome table. fixed_correctly means the tests went green and an independent production replay passed and the key was right.

arm	what it is	verdict	retrievals	fleet cost	wall clock
anarchy	no memory tool at all (control)	green_but_voids_prod	0	$0.5668	162.04s
rag	dense cosine, nomic-embed-text, top 3	fixed_correctly	6	$0.4429	140.68s
sync	Vestige causal backfill	fixed_correctly	3	$0.4918	137.29s
mem0	mem0ai OSS, LLM extraction on ingest	failed_merge_conflict	7	$0.4888	154.25s
supermemory	supermemory, vendor defaults	fixed_correctly	7	$0.5119	143.81s
hindsight	semantic + BM25 + cross-encoder rerank	failed_merge_conflict	6	$0.4974	147.78s
zep	Graphiti temporal knowledge graph	failed_merge_conflict	13	$0.5341	135.28s

Read the losing rows by agent instead of by arm and the pattern is identical every time.

arm	agent a0	agent a1	agent a2	conflicts
mem0	k_antares	k_nashira	k_nashira	2
hindsight	k_nashira	k_antares	k_nashira	2
zep	k_antares	k_nashira	k_nashira	2

The correct key was k_nashira. In all three arms, two agents out of three got it. In all three arms, the third agent took k_antares, a decoy planted in a closed staging ticket. Different arm, different agent index, same shape.

---

The agent that failed was not reasoning badly

This is the part that matters, and it is why the transcripts are published rather than summarized.

The agent that got it wrong was not sloppy. It reasoned carefully and correctly from the evidence it was handed. Here is mem0's agent a0, in its own words, at the moment it committed:

"Both configs have blank key ids, left blank by an 'externalize secrets' refactor. The memory said: OPS-522 the key id was reset to k_antares and verification recovered. That suggests k_antares is the key the platform shipped with / should use."

That is a good inference. The retrieval had returned exactly one key-related fact, and that fact said k_antares fixed a verification failure. a0 searched twice. Neither search returned the rotation runbook naming k_nashira.

Now here is its teammate a2, on the same arm, in the same trial, working from a retrieval that did surface more:

"Memory said: 'OPS-522: a staging box failed token verification after a hand-edited config; on-call reset the key id to k_antares and verification recovered, noting it was staging-only with no rotation.' Hmm, that says k_antares was staging-only. That memory may be a hint that the production key id is something else, or it might be a red herring. Let me search memory more specifically."

a2 got suspicious and kept looking. a0 had no reason to be suspicious, because nothing it retrieved contradicted the decoy.

The two agents did not disagree about how to interpret a shared reality. Their memory layer gave them different realities. One of them was working from a past in which k_antares was the answer.

hindsight's a1 shows the same thing from the other side. It searched twice, and its second query was well formed:

query: externalize secrets refactor blanked key id which kid did the platform ship with activeKid trustedKid

That query never surfaced k_nashira either. Its reasoning then walks straight into the decoy:

"The question: which kid should both sides use? The config comment says 'entry the platform shipped with' dangling comment. The hindsight memory says: 'Closed ticket OPS-522 ... on-call reset the key id to k_antares and verification recovered.'"

Same arm, same corpus, same model, same turn budget. Its two teammates asked differently and got the truth.

---

Why the symptom query fails

Every agent in every arm opened with a query describing the symptom. Here is what the very first retrieval returned for the RAG arm's a0, verbatim, with the similarity scores the tool reported:

Top-3 memories by semantic similarity (cosine, nomic-embed-text): [1] (similarity 0.641) Closed ticket OPS-522: a staging box failed token verification after a hand-edited config; on-call reset the key id to k_antares and verification recovered. Staging-only, nothing rotated. [2] (similarity 0.629) Ledger verifier threw a 500 on charge when the token had no 'kid' header...

The top hit for "my charge test is failing on token verification" is a document about a token verification failure that was fixed by setting a key id. It is an excellent match for the symptom. It is also the wrong answer.

The document that actually explains the failure reads:

"Rotation runbook, Q3: we migrated the live signer to k_nashira and let the old material age out. Every credential minted since the cutover carries the new kid."

Notice that it shares almost no surface vocabulary with "end-to-end charge test failing on session token verify." It does not mention charges, tests, or failures. It is a routine operational note written four days before anything broke.

We measured this directly. Against the eight-document retrievable corpus, using the exact queries the agents actually typed:

query (verbatim from a transcript)	who asked it	dense cosine	BM25
end-to-end charge test failing identity gateway ledger session token verify	rag a0, first query	7 of 8	7 of 8
end-to-end charge test failing session token identity gateway ledger	zep a0, first query	7 of 8	7 of 8
default key kid keyring shipped platform active key	zep a0, third query	3 of 8	7 of 8
which key id active signing key k_antares externalize secrets refactor blank config	mem0 a0, second query	3 of 8	4 of 8
no active signing key selected signing key rotation identity-service keyring	supermemory a0, first query	4 of 8	6 of 8

Measured against trial 1's own eight-document corpus, exported from the seeded store. Okapi BM25 with k1=1.5 and b=0.75; dense is nomic-embed-text, the same embedder the RAG arm used.

When you describe the symptom, the cause ranks last. Not sixth. Last, or next to last, out of eight.

This is not a tuning problem, and it is worth being precise about why. The cause is not merely dissimilar to the failure. The corpus contains look-alike documents that are more similar to the failure than the cause is, because they describe the same symptom. Sharpening the embedding pushes the look-alikes up, not the cause. Better similarity matching makes an anti-correlation worse, not better.

We ran the BM25 lexical baseline specifically because the answer is a rare literal token, which is supposed to be exact-match's home ground. It is not better. It ties dense cosine on the symptom queries, where both bury the cause at rank 7 of 8, and it is worse by one to four ranks on every query that actually worked. The decoys share the target's exact vocabulary, so lexical scoring rewards them just as much. The thing that separates the right key from the wrong one is which one is current, and recency has no lexical signal at all.

There is also a practical limit on exact match here. The live key is randomized per trial from a 50-key keyring and appears in no readable file, so you can only type it into a query after you already know the answer.

---

What the winning arms actually did

Three arms shipped a correct, production-safe fix. They did it in visibly different ways.

rag and supermemory won by reformulating. Every one of their agents opened with a symptom query, got the decoy, and then wrote a second query that described the shape of the answer rather than the shape of the problem. supermemory's a0 opened with no active signing key selected signing key rotation identity-service keyring. It had already guessed that a rotation was involved before it searched.

That works. It also means the model is spending reasoning budget compensating for retrieval, and that it has to suspect the answer's category before it can find it.

sync, the Vestige arm, did not formulate a query. vestige_backfill takes no query argument. Each agent called it exactly once, and the tool returned this:

{
  "causes": [{
    "content_preview": "Rotation runbook, Q3: we migrated the live signer to k_nashira ...",
    "age_days_before_failure": 4.0,
    "shared_entities": ["active_key"],
    "similarity_rank": 6,
    "backfill_score": 1.51,
    "reason": "Reached back 4.0d to a quiet memory sharing 1 entity (active_key) with the
               failure; it ranked #6 on similarity, so semantic search would have missed it."
  }]
}

Three agents, three calls, three identical results. Zero merge conflicts. The receipt states its own similarity rank, which is the tool reporting that the fact it just returned was one a similarity search would have buried.

Be precise about what that demonstrates. The active_key entity link between the failure and the cause is authored by the benchmark harness (prepare_trial.py:187 and :198). Vestige's extractor does not derive it from the prose. So this shows that traversing an existing causal edge returns the cause in one call with no query formulation. It does not show that the edge can be discovered automatically. That is a separate, unproven claim, and this benchmark does not test it.

---

The control that makes the point

The most useful row in the table is the one with no memory at all.

anarchy had all three agents converge unanimously on k_olivine. Zero merge conflicts. Perfect agreement. The tests went green. The production replay failed.

So convergence is not the prize. A fleet can agree perfectly and be perfectly wrong. What the memory layer has to deliver is convergence on the truth, and the failing arms show that delivering it to two out of three agents is worth exactly as much as delivering it to none.

anarchy also did something worth its own study: with no memory, it did not fail loudly. It shipped a green test suite that breaks production. That result has nothing to do with any memory product, which is why we find it credible.

---

Why the usual metrics cannot see this

Retrieval quality is normally reported as recall@k, MRR, or nDCG. All three are means over queries.

Fleet success is not a mean. It is a conjunction. Every agent editing shared state has to hold the same correct belief, or the merge breaks. The statistic that governs the outcome is the minimum over agents, not the average.

If each agent independently has a 90% chance of retrieving the deciding fact:

fleet size	probability every agent gets it
1	0.90
3	0.73
5	0.59
10	0.35

A memory layer that benchmarks at 90% recall looks excellent and produces a coin flip at fleet size 5. Nothing in the standard metric suite surfaces that, because the standard metric suite was designed for one agent asking one question.

The measurements a fleet actually needs are different:

• Correct convergence. Did every agent independently reach the production-safe answer?
• Retrieval consistency. How many distinct purported truths did the memory layer hand out across agents working the same incident?
• Query sensitivity. How much did the returned evidence change with small changes in wording?
• Fleet-safe completion. Did the merged result stay conflict free and pass an external replay?
• Cost per converged safe fix. Retrievals, turns, tokens, seconds and dollars until the whole fleet was correct.

---

What this does not show

Stated plainly, because the interesting result is worth more than an overclaim.

1. The causal edge is hand authored. prepare_trial.py:187 tags the cause keyring_rotation,active_key and :198 tags the failure active_key,crash. None of the seven distractors carry it. Remove that string and backfill returns causes: []. This measures traversal, not discovery.
2. The outcome column does not separate the winners. rag, sync and supermemory all scored fixed_correctly. Plain dense cosine over eight documents tied the memory products, and it was the cheapest arm in the trial at $0.4429. There is no "Vestige beats RAG" result here to defend.
3. The retrievable corpus is eight documents. Nothing here is measured at production scale, and the two scaling axes run in opposite directions. More documents can only push a cause that already ranks seventh further down a similarity list, which makes N=8 the easy case for the RAG arms. Vestige's backfill degrades as more memories share the join entity, which is two of nine here.
4. This is one trial of one scenario on one model. The per-agent divergence pattern repeated across three independent arms within it, and reproduced in a later run, but n is small and it is one bug shape.
5. similarity_rank in the receipt is self-reported by Vestige. The independent check on it is that the RAG arm's own tool output showed the decoy outranking the cause on symptom queries, which is visible in the transcripts.
6. The Vestige arm has a write-back coordination tool (vestige_log) that no competitor arm was given. It did not produce the result: in all six agent runs across both trials, each agent got the key from its own single backfill call, and every log write happened strictly later in the transcript.
7. The competitor corpus is built content-only, with the tags field discarded (runner.py:399). Restoring tags to competitors was measured and flips zero outcomes; the cause still never reaches rank 1, and on symptom queries it still ranks seventh or eighth in both conditions.
8. The author built the benchmark, the harness, and one of the products measured. Which is why the raw transcripts are published rather than a summary table.

---

Reproducing it

Everything needed to check this is in the repository.

harness/            the runner, the fleet driver, and the trial generator
harness/agent/prepare_trial.py     seeds the corpus and randomizes the key
results/trial-1/    all 7 arm results and all 21 agent transcripts
tests/              the arm liveness gate

The claims above map to files. The outcome table is results/trial-1/{arm}.json, field fleet_verdict. The per-agent key choices are fix_directions in the same files. Every quotation is in results/trial-1/transcript-{arm}-a{n}.json under turns[].reasoning and turns[].tool_results.

One thing to check first if you are auditing this: each arm's JSON carries memory_layer_alive and retrieval_err_total. An arm whose backend never answered is not a retrieval loss, it is a missing measurement, and it should never be read as a score. In trial 1 all six memory arms were alive with zero retrieval errors.

---

The short version

Similarity retrieval asks every agent to guess the vocabulary of an answer it has not seen yet. Most of them guess well. In a fleet, most is not enough.

Two agents found the truth. One found the decoy. Nothing shipped.