From c2d0606f2c62cf585fbf198baf1ac1e51cf04b1f Mon Sep 17 00:00:00 2001 From: Sam Valladares Date: Wed, 22 Jul 2026 00:14:51 +0700 Subject: [PATCH] fix(silent-rotation): recount first-call rates across all 25 trials and publish the method The first-call figures in QUOTES.md were computed on the five-model set before kimi-k2.7-code was restored and before mem0 and Zep were re-run on the repaired harness. Recounted directly from results/*/transcript-*.json by walking turns[].tool_results[], taking the first entry whose name is a memory tool, and testing whether manifest.json's correct_kid appears in its output: arm correct n rate sync (Vestige, no query) 65 65 100% rag (dense cosine) 6 66 9.1% supermemory 1 18 5.6% mem0 0 15 0% hindsight 0 9 0% zep 0 6 0% all query-based combined 7 114 6.1% Previously published as 56/56 and 6/87. Vestige is perfect on a larger denominator; the query-based aggregate moves 6.9% -> 6.1% on 114 calls instead of 87. The anarchy arm has no memory tool and is excluded, which accounts for the 67 transcripts with no memory call. EVIDENCE.md now carries the table and the exact recount procedure so a reader can reproduce it without rerunning any trials. Co-Authored-By: Claude Opus 4.8 --- benchmarks/silent-rotation/EVIDENCE.md | 19 +++++++++++++++++++ benchmarks/silent-rotation/QUOTES.md | 6 +++--- 2 files changed, 22 insertions(+), 3 deletions(-) diff --git a/benchmarks/silent-rotation/EVIDENCE.md b/benchmarks/silent-rotation/EVIDENCE.md index 4ac9fad..7093457 100644 --- a/benchmarks/silent-rotation/EVIDENCE.md +++ b/benchmarks/silent-rotation/EVIDENCE.md @@ -26,6 +26,25 @@ it the right one?** | zep | 0/2 | 1/2 | 1/2 | 2 | Models: Kimi K3, Kimi K2.7-code, MiniMax M3, GLM 5.2, GPT-5.6 Sol, DeepSeek V4 Flash. + +### First memory call + +For every agent, the FIRST memory-tool call it made: did the returned payload contain the trial's +correct key? This isolates the memory substrate from the model's ability to recover on later turns. + +| arm | first call correct | n | rate | +|---|---|---|---| +| **sync (Vestige, no query)** | **65** | **65** | **100%** | +| rag (dense cosine) | 6 | 66 | 9.1% | +| supermemory | 1 | 18 | 5.6% | +| mem0 | 0 | 15 | 0% | +| hindsight | 0 | 9 | 0% | +| zep | 0 | 6 | 0% | +| **all query-based combined** | **7** | **114** | **6.1%** | + +The `anarchy` arm has no memory tool and is excluded. Reproduce: for each +`results/*/transcript-*.json`, walk `turns[].tool_results[]`, take the first entry whose `name` is a +memory tool, and test whether `manifest.json`'s `correct_kid` appears in its `output`. Reproduce: read `fix_directions` and `correct_kid` out of every `trial-*/{arm}.json` and `manifest.json`. ### There are two failure modes and only one of them is visible diff --git a/benchmarks/silent-rotation/QUOTES.md b/benchmarks/silent-rotation/QUOTES.md index 104f347..c7f7a02 100644 --- a/benchmarks/silent-rotation/QUOTES.md +++ b/benchmarks/silent-rotation/QUOTES.md @@ -71,7 +71,7 @@ two failed queries: It concluded the memory layer had nothing and went to the source. It needed six queries in total and first surfaced the correct key on call four. -This is what a 6.9% first-call hit rate feels like from inside the agent. +This is what a 6.1% first-call hit rate feels like from inside the agent. --- @@ -152,8 +152,8 @@ configured one. Fixed and re-run: mem0 converged correctly in `runB-trial-3` and **It proves:** at turn zero an agent knows only the symptom, so it queries the symptom. In this incident class the symptom query returns a document that resembles the failure and buries the one that explains it, and that holds under dense cosine and BM25 alike. Across 6 models and 25 trials, a query-based arm's -first memory call surfaced the deciding fact 6 times out of 87. Anchoring on the failure event instead -of a query did it 56 out of 56. In a fleet, that first-call gap becomes either a merge conflict or a +first memory call surfaced the deciding fact 7 times out of 114. Anchoring on the failure event instead +of a query did it 65 out of 65. In a fleet, that first-call gap becomes either a merge conflict or a unanimous wrong answer that passes tests and breaks production. **It does not prove:** that Vestige discovers causal relationships. The edge it traverses is authored by