# One misled agent voids a three-agent fleet **What happens when three AI agents fix the same bug, and only two of them know the truth.** Trial 1 of the Silent Rotation benchmark. Kimi K3, seven memory configurations, one randomized secret. Every number and every quotation below comes from the raw agent transcripts, which are published alongside this document in `results/trial-1/`. --- ## The result in one paragraph Three agents work the same failing test. The fix requires knowing which cryptographic signing key is currently live. That key is randomized per trial and appears in no file the agents can read. It exists only in the memory layer. Four of the six memory configurations put the correct answer in front of at least two of the three agents. Three of them lost anyway. In each losing case exactly one agent out of three never received the correct fact, confidently adopted a planted decoy, and wrote a conflicting change. The merge fractured and nothing shipped. Two agents right plus one misled equals zero production-safe fleets. --- ## Why this is not a retrieval story The reflex when a memory system loses is to say its retrieval was bad. That is not what happened here. In every losing arm the correct memory was in the store, and most agents found it. Here is the complete trial-1 outcome table. `fixed_correctly` means the tests went green **and** an independent production replay passed **and** the key was right. | arm | what it is | verdict | retrievals | fleet cost | wall clock | |---|---|---|---|---|---| | anarchy | no memory tool at all (control) | green_but_voids_prod | 0 | $0.5668 | 162.04s | | rag | dense cosine, nomic-embed-text, top 3 | **fixed_correctly** | 6 | $0.4429 | 140.68s | | sync | Vestige causal backfill | **fixed_correctly** | 3 | $0.4918 | 137.29s | | mem0 | mem0ai OSS, LLM extraction on ingest | failed_merge_conflict | 7 | $0.4888 | 154.25s | | supermemory | supermemory, vendor defaults | **fixed_correctly** | 7 | $0.5119 | 143.81s | | hindsight | semantic + BM25 + cross-encoder rerank | failed_merge_conflict | 6 | $0.4974 | 147.78s | | zep | Graphiti temporal knowledge graph | failed_merge_conflict | 13 | $0.5341 | 135.28s | Read the losing rows by agent instead of by arm and the pattern is identical every time. | arm | agent a0 | agent a1 | agent a2 | conflicts | |---|---|---|---|---| | mem0 | **k_antares** | k_nashira | k_nashira | 2 | | hindsight | k_nashira | **k_antares** | k_nashira | 2 | | zep | **k_antares** | k_nashira | k_nashira | 2 | The correct key was `k_nashira`. In all three arms, two agents out of three got it. In all three arms, the third agent took `k_antares`, a decoy planted in a closed staging ticket. Different arm, different agent index, same shape. --- ## The agent that failed was not reasoning badly This is the part that matters, and it is why the transcripts are published rather than summarized. The agent that got it wrong was not sloppy. It reasoned carefully and correctly from the evidence it was handed. Here is mem0's agent a0, in its own words, at the moment it committed: > "Both configs have blank key ids, left blank by an 'externalize secrets' refactor. The memory said: > OPS-522 the key id was reset to `k_antares` and verification recovered. That suggests `k_antares` is > the key the platform shipped with / should use." That is a good inference. The retrieval had returned exactly one key-related fact, and that fact said `k_antares` fixed a verification failure. a0 searched twice. Neither search returned the rotation runbook naming `k_nashira`. Now here is its teammate a2, on the same arm, in the same trial, working from a retrieval that did surface more: > "Memory said: 'OPS-522: a staging box failed token verification after a hand-edited config; on-call > reset the key id to k_antares and verification recovered, noting it was staging-only with no > rotation.' Hmm, that says k_antares was staging-only. That memory may be a hint that the production > key id is something else, or it might be a red herring. Let me search memory more specifically." a2 got suspicious and kept looking. a0 had no reason to be suspicious, because nothing it retrieved contradicted the decoy. The two agents did not disagree about how to interpret a shared reality. **Their memory layer gave them different realities.** One of them was working from a past in which `k_antares` was the answer. hindsight's a1 shows the same thing from the other side. It searched twice, and its second query was well formed: > query: `externalize secrets refactor blanked key id which kid did the platform ship with activeKid trustedKid` That query never surfaced `k_nashira` either. Its reasoning then walks straight into the decoy: > "The question: which kid should both sides use? The config comment says 'entry the platform shipped > with' dangling comment. The hindsight memory says: 'Closed ticket OPS-522 ... on-call reset the key > id to k_antares and verification recovered.'" Same arm, same corpus, same model, same turn budget. Its two teammates asked differently and got the truth. --- ## Why the symptom query fails Every agent in every arm opened with a query describing the symptom. Here is what the very first retrieval returned for the RAG arm's a0, verbatim, with the similarity scores the tool reported: > Top-3 memories by semantic similarity (cosine, nomic-embed-text): > **[1] (similarity 0.641)** Closed ticket OPS-522: a staging box failed token verification after a > hand-edited config; on-call reset the key id to k_antares and verification recovered. Staging-only, > nothing rotated. > **[2] (similarity 0.629)** Ledger verifier threw a 500 on charge when the token had no 'kid' header... The top hit for "my charge test is failing on token verification" is a document about a token verification failure that was fixed by setting a key id. It is an excellent match for the symptom. It is also the wrong answer. The document that actually explains the failure reads: > "Rotation runbook, Q3: we migrated the live signer to k_nashira and let the old material age out. > Every credential minted since the cutover carries the new kid." Notice that it shares almost no surface vocabulary with "end-to-end charge test failing on session token verify." It does not mention charges, tests, or failures. It is a routine operational note written four days before anything broke. We measured this directly. Against the eight-document retrievable corpus, using the exact queries the agents actually typed: | query (verbatim from a transcript) | who asked it | dense cosine | BM25 | |---|---|---|---| | `end-to-end charge test failing identity gateway ledger session token verify` | rag a0, first query | 7 of 8 | 7 of 8 | | `end-to-end charge test failing session token identity gateway ledger` | zep a0, first query | 7 of 8 | 7 of 8 | | `default key kid keyring shipped platform active key` | zep a0, third query | 3 of 8 | 7 of 8 | | `which key id active signing key k_antares externalize secrets refactor blank config` | mem0 a0, second query | 3 of 8 | 4 of 8 | | `no active signing key selected signing key rotation identity-service keyring` | supermemory a0, first query | 4 of 8 | 6 of 8 | Measured against trial 1's own eight-document corpus, exported from the seeded store. Okapi BM25 with k1=1.5 and b=0.75; dense is nomic-embed-text, the same embedder the RAG arm used. **When you describe the symptom, the cause ranks last.** Not sixth. Last, or next to last, out of eight. This is not a tuning problem, and it is worth being precise about why. The cause is not merely dissimilar to the failure. The corpus contains look-alike documents that are *more* similar to the failure than the cause is, because they describe the same symptom. Sharpening the embedding pushes the look-alikes up, not the cause. **Better similarity matching makes an anti-correlation worse, not better.** We ran the BM25 lexical baseline specifically because the answer is a rare literal token, which is supposed to be exact-match's home ground. It is not better. It ties dense cosine on the symptom queries, where both bury the cause at rank 7 of 8, and it is worse by one to four ranks on every query that actually worked. The decoys share the target's exact vocabulary, so lexical scoring rewards them just as much. The thing that separates the right key from the wrong one is *which one is current*, and recency has no lexical signal at all. There is also a practical limit on exact match here. The live key is randomized per trial from a 50-key keyring and appears in no readable file, so you can only type it into a query after you already know the answer. --- ## What the winning arms actually did Three arms shipped a correct, production-safe fix. They did it in visibly different ways. **rag** and **supermemory** won by reformulating. Every one of their agents opened with a symptom query, got the decoy, and then wrote a second query that described the *shape of the answer* rather than the shape of the problem. supermemory's a0 opened with `no active signing key selected signing key rotation identity-service keyring`. It had already guessed that a rotation was involved before it searched. That works. It also means the model is spending reasoning budget compensating for retrieval, and that it has to suspect the answer's category before it can find it. **sync**, the Vestige arm, did not formulate a query. `vestige_backfill` takes no query argument. Each agent called it exactly once, and the tool returned this: ```json { "causes": [{ "content_preview": "Rotation runbook, Q3: we migrated the live signer to k_nashira ...", "age_days_before_failure": 4.0, "shared_entities": ["active_key"], "similarity_rank": 6, "backfill_score": 1.51, "reason": "Reached back 4.0d to a quiet memory sharing 1 entity (active_key) with the failure; it ranked #6 on similarity, so semantic search would have missed it." }] } ``` Three agents, three calls, three identical results. Zero merge conflicts. The receipt states its own similarity rank, which is the tool reporting that the fact it just returned was one a similarity search would have buried. **Be precise about what that demonstrates.** The `active_key` entity link between the failure and the cause is authored by the benchmark harness (`prepare_trial.py:187` and `:198`). Vestige's extractor does not derive it from the prose. So this shows that *traversing an existing causal edge* returns the cause in one call with no query formulation. It does **not** show that the edge can be discovered automatically. That is a separate, unproven claim, and this benchmark does not test it. --- ## The control that makes the point The most useful row in the table is the one with no memory at all. **anarchy** had all three agents converge unanimously on `k_olivine`. Zero merge conflicts. Perfect agreement. The tests went green. The production replay failed. So convergence is not the prize. A fleet can agree perfectly and be perfectly wrong. What the memory layer has to deliver is convergence *on the truth*, and the failing arms show that delivering it to two out of three agents is worth exactly as much as delivering it to none. anarchy also did something worth its own study: with no memory, it did not fail loudly. It shipped a green test suite that breaks production. That result has nothing to do with any memory product, which is why we find it credible. --- ## Why the usual metrics cannot see this Retrieval quality is normally reported as recall@k, MRR, or nDCG. All three are **means over queries**. Fleet success is not a mean. It is a conjunction. Every agent editing shared state has to hold the same correct belief, or the merge breaks. The statistic that governs the outcome is the **minimum** over agents, not the average. If each agent independently has a 90% chance of retrieving the deciding fact: | fleet size | probability every agent gets it | |---|---| | 1 | 0.90 | | 3 | 0.73 | | 5 | 0.59 | | 10 | 0.35 | A memory layer that benchmarks at 90% recall looks excellent and produces a coin flip at fleet size 5. Nothing in the standard metric suite surfaces that, because the standard metric suite was designed for one agent asking one question. The measurements a fleet actually needs are different: - **Correct convergence.** Did *every* agent independently reach the production-safe answer? - **Retrieval consistency.** How many distinct purported truths did the memory layer hand out across agents working the same incident? - **Query sensitivity.** How much did the returned evidence change with small changes in wording? - **Fleet-safe completion.** Did the merged result stay conflict free and pass an external replay? - **Cost per converged safe fix.** Retrievals, turns, tokens, seconds and dollars until the whole fleet was correct. --- ## What this does not show Stated plainly, because the interesting result is worth more than an overclaim. 1. **The causal edge is hand authored.** `prepare_trial.py:187` tags the cause `keyring_rotation,active_key` and `:198` tags the failure `active_key,crash`. None of the seven distractors carry it. Remove that string and backfill returns `causes: []`. This measures traversal, not discovery. 2. **The outcome column does not separate the winners.** rag, sync and supermemory all scored `fixed_correctly`. Plain dense cosine over eight documents tied the memory products, and it was the **cheapest arm in the trial** at $0.4429. There is no "Vestige beats RAG" result here to defend. 3. **The retrievable corpus is eight documents.** Nothing here is measured at production scale, and the two scaling axes run in opposite directions. More documents can only push a cause that already ranks seventh further down a similarity list, which makes N=8 the *easy* case for the RAG arms. Vestige's backfill degrades as more memories share the join entity, which is two of nine here. 4. **This is one trial of one scenario on one model.** The per-agent divergence pattern repeated across three independent arms within it, and reproduced in a later run, but n is small and it is one bug shape. 5. **`similarity_rank` in the receipt is self-reported by Vestige.** The independent check on it is that the RAG arm's own tool output showed the decoy outranking the cause on symptom queries, which is visible in the transcripts. 6. **The Vestige arm has a write-back coordination tool** (`vestige_log`) that no competitor arm was given. It did not produce the result: in all six agent runs across both trials, each agent got the key from its own single backfill call, and every log write happened strictly later in the transcript. 7. **The competitor corpus is built content-only**, with the `tags` field discarded (`runner.py:399`). Restoring tags to competitors was measured and flips zero outcomes; the cause still never reaches rank 1, and on symptom queries it still ranks seventh or eighth in both conditions. 8. **The author built the benchmark, the harness, and one of the products measured.** Which is why the raw transcripts are published rather than a summary table. --- ## Reproducing it Everything needed to check this is in the repository. ``` harness/ the runner, the fleet driver, and the trial generator harness/agent/prepare_trial.py seeds the corpus and randomizes the key results/trial-1/ all 7 arm results and all 21 agent transcripts tests/ the arm liveness gate ``` The claims above map to files. The outcome table is `results/trial-1/{arm}.json`, field `fleet_verdict`. The per-agent key choices are `fix_directions` in the same files. Every quotation is in `results/trial-1/transcript-{arm}-a{n}.json` under `turns[].reasoning` and `turns[].tool_results`. One thing to check first if you are auditing this: each arm's JSON carries `memory_layer_alive` and `retrieval_err_total`. An arm whose backend never answered is not a retrieval loss, it is a missing measurement, and it should never be read as a score. In trial 1 all six memory arms were alive with zero retrieval errors. --- ## The short version Similarity retrieval asks every agent to guess the vocabulary of an answer it has not seen yet. Most of them guess well. In a fleet, most is not enough. Two agents found the truth. One found the decoy. Nothing shipped.