diff --git a/benchmarks/silent-rotation/EVIDENCE.md b/benchmarks/silent-rotation/EVIDENCE.md index 4ac9fad..7093457 100644 --- a/benchmarks/silent-rotation/EVIDENCE.md +++ b/benchmarks/silent-rotation/EVIDENCE.md @@ -26,6 +26,25 @@ it the right one?** | zep | 0/2 | 1/2 | 1/2 | 2 | Models: Kimi K3, Kimi K2.7-code, MiniMax M3, GLM 5.2, GPT-5.6 Sol, DeepSeek V4 Flash. + +### First memory call + +For every agent, the FIRST memory-tool call it made: did the returned payload contain the trial's +correct key? This isolates the memory substrate from the model's ability to recover on later turns. + +| arm | first call correct | n | rate | +|---|---|---|---| +| **sync (Vestige, no query)** | **65** | **65** | **100%** | +| rag (dense cosine) | 6 | 66 | 9.1% | +| supermemory | 1 | 18 | 5.6% | +| mem0 | 0 | 15 | 0% | +| hindsight | 0 | 9 | 0% | +| zep | 0 | 6 | 0% | +| **all query-based combined** | **7** | **114** | **6.1%** | + +The `anarchy` arm has no memory tool and is excluded. Reproduce: for each +`results/*/transcript-*.json`, walk `turns[].tool_results[]`, take the first entry whose `name` is a +memory tool, and test whether `manifest.json`'s `correct_kid` appears in its `output`. Reproduce: read `fix_directions` and `correct_kid` out of every `trial-*/{arm}.json` and `manifest.json`. ### There are two failure modes and only one of them is visible diff --git a/benchmarks/silent-rotation/QUOTES.md b/benchmarks/silent-rotation/QUOTES.md index 104f347..c7f7a02 100644 --- a/benchmarks/silent-rotation/QUOTES.md +++ b/benchmarks/silent-rotation/QUOTES.md @@ -71,7 +71,7 @@ two failed queries: It concluded the memory layer had nothing and went to the source. It needed six queries in total and first surfaced the correct key on call four. -This is what a 6.9% first-call hit rate feels like from inside the agent. +This is what a 6.1% first-call hit rate feels like from inside the agent. --- @@ -152,8 +152,8 @@ configured one. Fixed and re-run: mem0 converged correctly in `runB-trial-3` and **It proves:** at turn zero an agent knows only the symptom, so it queries the symptom. In this incident class the symptom query returns a document that resembles the failure and buries the one that explains it, and that holds under dense cosine and BM25 alike. Across 6 models and 25 trials, a query-based arm's -first memory call surfaced the deciding fact 6 times out of 87. Anchoring on the failure event instead -of a query did it 56 out of 56. In a fleet, that first-call gap becomes either a merge conflict or a +first memory call surfaced the deciding fact 7 times out of 114. Anchoring on the failure event instead +of a query did it 65 out of 65. In a fleet, that first-call gap becomes either a merge conflict or a unanimous wrong answer that passes tests and breaks production. **It does not prove:** that Vestige discovers causal relationships. The edge it traverses is authored by