diff --git a/benchmarks/silent-rotation/EVIDENCE.md b/benchmarks/silent-rotation/EVIDENCE.md index a38f117..4ac9fad 100644 --- a/benchmarks/silent-rotation/EVIDENCE.md +++ b/benchmarks/silent-rotation/EVIDENCE.md @@ -17,15 +17,15 @@ it the right one?** | arm | converged on the CORRECT key | converged on a WRONG key | split, no consensus | n | |---|---|---|---|---| -| anarchy (no memory) | 0/22 | **16/22** | 6/22 | 22 | -| rag (dense cosine) | 4/21 | **10/21** | 7/21 | 21 | -| **sync (Vestige)** | **19/21** | **0/21** | 2/21 | 21 | -| supermemory | 4/5 | 0/5 | 1/5 | 5 | -| mem0 | 0/5 | 1/5 | 4/5 | 5 | -| hindsight | 0/5 | 0/5 | 5/5 | 5 | -| zep | 0/4 | 2/4 | 2/4 | 4 | +| anarchy (no memory) | 0/25 | **21/25** | 4/25 | 25 | +| rag (dense cosine) | 4/23 | **12/23** | 7/23 | 23 | +| **sync (Vestige)** | **20/23** | **0/23** | 3/23 | 23 | +| supermemory | 5/6 | 0/6 | 1/6 | 6 | +| mem0 | 2/5 | 1/5 | 2/5 | 5 | +| hindsight | 0/3 | 0/3 | 3/3 | 3 | +| zep | 0/2 | 1/2 | 1/2 | 2 | -Models: Kimi K3, MiniMax M3, GLM 5.2, GPT-5.6 Sol, DeepSeek V4 Flash. +Models: Kimi K3, Kimi K2.7-code, MiniMax M3, GLM 5.2, GPT-5.6 Sol, DeepSeek V4 Flash. Reproduce: read `fix_directions` and `correct_kid` out of every `trial-*/{arm}.json` and `manifest.json`. ### There are two failure modes and only one of them is visible diff --git a/benchmarks/silent-rotation/QUOTES.md b/benchmarks/silent-rotation/QUOTES.md index 88a968d..104f347 100644 --- a/benchmarks/silent-rotation/QUOTES.md +++ b/benchmarks/silent-rotation/QUOTES.md @@ -151,7 +151,7 @@ configured one. Fixed and re-run: mem0 converged correctly in `runB-trial-3` and **It proves:** at turn zero an agent knows only the symptom, so it queries the symptom. In this incident class the symptom query returns a document that resembles the failure and buries the one that explains -it, and that holds under dense cosine and BM25 alike. Across 5 models and 22 trials, a query-based arm's +it, and that holds under dense cosine and BM25 alike. Across 6 models and 25 trials, a query-based arm's first memory call surfaced the deciding fact 6 times out of 87. Anchoring on the failure event instead of a query did it 56 out of 56. In a fleet, that first-call gap becomes either a merge conflict or a unanimous wrong answer that passes tests and breaks production. diff --git a/benchmarks/silent-rotation/SHOW-HN.md b/benchmarks/silent-rotation/SHOW-HN.md index 0ec9e0e..ce6fb50 100644 --- a/benchmarks/silent-rotation/SHOW-HN.md +++ b/benchmarks/silent-rotation/SHOW-HN.md @@ -42,15 +42,15 @@ ships. Loud. You find out immediately. Converged wrong: every agent agrees, on the decoy. Tests go green. The merge is clean. Production breaks. This is the one that ships. -Across 5 models and 22 trials: +Across 6 models and 25 trials: arm converged CORRECT converged WRONG split - no memory 0/22 16/22 6/22 - dense cosine RAG 4/21 10/21 7/21 - Vestige 19/21 0/21 2/21 + no memory 0/25 21/25 4/25 + dense cosine RAG 4/23 12/23 7/23 + Vestige 20/23 0/23 3/23 The middle column is the finding. A memoryless fleet marched confidently off a -cliff 16 times out of 22. Dense cosine did it 10 times out of 21. +cliff 21 times out of 25. Dense cosine did it 12 times out of 23. **The agents were not reasoning badly.** GPT-5.6 Sol, committing to the wrong key: @@ -133,7 +133,7 @@ from temporal lifecycle. A reviewer can fairly argue I won by removing query variability alone. That is the next experiment. And I built the benchmark, the harness, and one of the products in it. Which is -why all 210 agent transcripts ship with it rather than a summary table. +why all 246 agent transcripts ship with it rather than a summary table. ---