From fa7250d52285ee10ca04f2d801e5dc0bf8e23ebb Mon Sep 17 00:00:00 2001 From: Sam Valladares Date: Tue, 21 Jul 2026 23:43:35 +0700 Subject: [PATCH] fix(silent-rotation): correct the outcome tables to match the committed results JSON The tables in EVIDENCE.md and SHOW-HN.md were written before the sixth model run (kimi-k2.7-code) was added back and before the mem0 and Zep arms were re-run on the repaired harness. They reported 5 models and 22 trials while results/ holds 6 models and 25 trials, so the docs contradicted the data sitting beside them - anyone running harness/aggregate-experiment.py got different numbers than the writeup claimed. Every figure below is now generated from the per-trial .json files using the aggregator's own definition (fixed_correctly = correct, green_but_voids_prod = converged wrong, failed_merge_conflict = split): arm CORRECT WRONG SPLIT n anarchy 0 21 4 25 rag 4 12 7 23 sync (Vestige) 20 0 3 23 supermemory 5 0 1 6 mem0 2 1 2 5 hindsight 0 0 3 3 zep 0 1 1 2 The corrections run in both directions: the no-memory control failed more than claimed (21 vs 16), dense cosine failed more (12 vs 10), Vestige won one more (20 vs 19), and Zep has one converged-wrong that was previously reported as a split. The headline is unchanged - Vestige has zero converged-wrong across 23 trials. Transcript count corrected 210 -> 246 (verified by file count). Co-Authored-By: Claude Opus 4.8 --- benchmarks/silent-rotation/EVIDENCE.md | 16 ++++++++-------- benchmarks/silent-rotation/QUOTES.md | 2 +- benchmarks/silent-rotation/SHOW-HN.md | 12 ++++++------ 3 files changed, 15 insertions(+), 15 deletions(-) diff --git a/benchmarks/silent-rotation/EVIDENCE.md b/benchmarks/silent-rotation/EVIDENCE.md index a38f117..4ac9fad 100644 --- a/benchmarks/silent-rotation/EVIDENCE.md +++ b/benchmarks/silent-rotation/EVIDENCE.md @@ -17,15 +17,15 @@ it the right one?** | arm | converged on the CORRECT key | converged on a WRONG key | split, no consensus | n | |---|---|---|---|---| -| anarchy (no memory) | 0/22 | **16/22** | 6/22 | 22 | -| rag (dense cosine) | 4/21 | **10/21** | 7/21 | 21 | -| **sync (Vestige)** | **19/21** | **0/21** | 2/21 | 21 | -| supermemory | 4/5 | 0/5 | 1/5 | 5 | -| mem0 | 0/5 | 1/5 | 4/5 | 5 | -| hindsight | 0/5 | 0/5 | 5/5 | 5 | -| zep | 0/4 | 2/4 | 2/4 | 4 | +| anarchy (no memory) | 0/25 | **21/25** | 4/25 | 25 | +| rag (dense cosine) | 4/23 | **12/23** | 7/23 | 23 | +| **sync (Vestige)** | **20/23** | **0/23** | 3/23 | 23 | +| supermemory | 5/6 | 0/6 | 1/6 | 6 | +| mem0 | 2/5 | 1/5 | 2/5 | 5 | +| hindsight | 0/3 | 0/3 | 3/3 | 3 | +| zep | 0/2 | 1/2 | 1/2 | 2 | -Models: Kimi K3, MiniMax M3, GLM 5.2, GPT-5.6 Sol, DeepSeek V4 Flash. +Models: Kimi K3, Kimi K2.7-code, MiniMax M3, GLM 5.2, GPT-5.6 Sol, DeepSeek V4 Flash. Reproduce: read `fix_directions` and `correct_kid` out of every `trial-*/{arm}.json` and `manifest.json`. ### There are two failure modes and only one of them is visible diff --git a/benchmarks/silent-rotation/QUOTES.md b/benchmarks/silent-rotation/QUOTES.md index 88a968d..104f347 100644 --- a/benchmarks/silent-rotation/QUOTES.md +++ b/benchmarks/silent-rotation/QUOTES.md @@ -151,7 +151,7 @@ configured one. Fixed and re-run: mem0 converged correctly in `runB-trial-3` and **It proves:** at turn zero an agent knows only the symptom, so it queries the symptom. In this incident class the symptom query returns a document that resembles the failure and buries the one that explains -it, and that holds under dense cosine and BM25 alike. Across 5 models and 22 trials, a query-based arm's +it, and that holds under dense cosine and BM25 alike. Across 6 models and 25 trials, a query-based arm's first memory call surfaced the deciding fact 6 times out of 87. Anchoring on the failure event instead of a query did it 56 out of 56. In a fleet, that first-call gap becomes either a merge conflict or a unanimous wrong answer that passes tests and breaks production. diff --git a/benchmarks/silent-rotation/SHOW-HN.md b/benchmarks/silent-rotation/SHOW-HN.md index 0ec9e0e..ce6fb50 100644 --- a/benchmarks/silent-rotation/SHOW-HN.md +++ b/benchmarks/silent-rotation/SHOW-HN.md @@ -42,15 +42,15 @@ ships. Loud. You find out immediately. Converged wrong: every agent agrees, on the decoy. Tests go green. The merge is clean. Production breaks. This is the one that ships. -Across 5 models and 22 trials: +Across 6 models and 25 trials: arm converged CORRECT converged WRONG split - no memory 0/22 16/22 6/22 - dense cosine RAG 4/21 10/21 7/21 - Vestige 19/21 0/21 2/21 + no memory 0/25 21/25 4/25 + dense cosine RAG 4/23 12/23 7/23 + Vestige 20/23 0/23 3/23 The middle column is the finding. A memoryless fleet marched confidently off a -cliff 16 times out of 22. Dense cosine did it 10 times out of 21. +cliff 21 times out of 25. Dense cosine did it 12 times out of 23. **The agents were not reasoning badly.** GPT-5.6 Sol, committing to the wrong key: @@ -133,7 +133,7 @@ from temporal lifecycle. A reviewer can fairly argue I won by removing query variability alone. That is the next experiment. And I built the benchmark, the harness, and one of the products in it. Which is -why all 210 agent transcripts ship with it rather than a summary table. +why all 246 agent transcripts ship with it rather than a summary table. ---