fix(silent-rotation): correct the outcome tables to match the committed results JSON

The tables in EVIDENCE.md and SHOW-HN.md were written before the sixth model
run (kimi-k2.7-code) was added back and before the mem0 and Zep arms were
re-run on the repaired harness. They reported 5 models and 22 trials while
results/ holds 6 models and 25 trials, so the docs contradicted the data
sitting beside them - anyone running harness/aggregate-experiment.py got
different numbers than the writeup claimed.

Every figure below is now generated from the per-trial <arm>.json files using
the aggregator's own definition (fixed_correctly = correct, green_but_voids_prod
= converged wrong, failed_merge_conflict = split):

  arm             CORRECT  WRONG  SPLIT   n
  anarchy               0     21      4  25
  rag                   4     12      7  23
  sync (Vestige)       20      0      3  23
  supermemory           5      0      1   6
  mem0                  2      1      2   5
  hindsight             0      0      3   3
  zep                   0      1      1   2

The corrections run in both directions: the no-memory control failed more than
claimed (21 vs 16), dense cosine failed more (12 vs 10), Vestige won one more
(20 vs 19), and Zep has one converged-wrong that was previously reported as a
split. The headline is unchanged - Vestige has zero converged-wrong across 23
trials. Transcript count corrected 210 -> 246 (verified by file count).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Sam Valladares 2026-07-21 23:43:35 +07:00
parent fd77ff31a6
commit fa7250d522
3 changed files with 15 additions and 15 deletions

View file

@ -17,15 +17,15 @@ it the right one?**
| arm | converged on the CORRECT key | converged on a WRONG key | split, no consensus | n | | arm | converged on the CORRECT key | converged on a WRONG key | split, no consensus | n |
|---|---|---|---|---| |---|---|---|---|---|
| anarchy (no memory) | 0/22 | **16/22** | 6/22 | 22 | | anarchy (no memory) | 0/25 | **21/25** | 4/25 | 25 |
| rag (dense cosine) | 4/21 | **10/21** | 7/21 | 21 | | rag (dense cosine) | 4/23 | **12/23** | 7/23 | 23 |
| **sync (Vestige)** | **19/21** | **0/21** | 2/21 | 21 | | **sync (Vestige)** | **20/23** | **0/23** | 3/23 | 23 |
| supermemory | 4/5 | 0/5 | 1/5 | 5 | | supermemory | 5/6 | 0/6 | 1/6 | 6 |
| mem0 | 0/5 | 1/5 | 4/5 | 5 | | mem0 | 2/5 | 1/5 | 2/5 | 5 |
| hindsight | 0/5 | 0/5 | 5/5 | 5 | | hindsight | 0/3 | 0/3 | 3/3 | 3 |
| zep | 0/4 | 2/4 | 2/4 | 4 | | zep | 0/2 | 1/2 | 1/2 | 2 |
Models: Kimi K3, MiniMax M3, GLM 5.2, GPT-5.6 Sol, DeepSeek V4 Flash. Models: Kimi K3, Kimi K2.7-code, MiniMax M3, GLM 5.2, GPT-5.6 Sol, DeepSeek V4 Flash.
Reproduce: read `fix_directions` and `correct_kid` out of every `trial-*/{arm}.json` and `manifest.json`. Reproduce: read `fix_directions` and `correct_kid` out of every `trial-*/{arm}.json` and `manifest.json`.
### There are two failure modes and only one of them is visible ### There are two failure modes and only one of them is visible

View file

@ -151,7 +151,7 @@ configured one. Fixed and re-run: mem0 converged correctly in `runB-trial-3` and
**It proves:** at turn zero an agent knows only the symptom, so it queries the symptom. In this incident **It proves:** at turn zero an agent knows only the symptom, so it queries the symptom. In this incident
class the symptom query returns a document that resembles the failure and buries the one that explains class the symptom query returns a document that resembles the failure and buries the one that explains
it, and that holds under dense cosine and BM25 alike. Across 5 models and 22 trials, a query-based arm's it, and that holds under dense cosine and BM25 alike. Across 6 models and 25 trials, a query-based arm's
first memory call surfaced the deciding fact 6 times out of 87. Anchoring on the failure event instead first memory call surfaced the deciding fact 6 times out of 87. Anchoring on the failure event instead
of a query did it 56 out of 56. In a fleet, that first-call gap becomes either a merge conflict or a of a query did it 56 out of 56. In a fleet, that first-call gap becomes either a merge conflict or a
unanimous wrong answer that passes tests and breaks production. unanimous wrong answer that passes tests and breaks production.

View file

@ -42,15 +42,15 @@ ships. Loud. You find out immediately.
Converged wrong: every agent agrees, on the decoy. Tests go green. The merge is Converged wrong: every agent agrees, on the decoy. Tests go green. The merge is
clean. Production breaks. This is the one that ships. clean. Production breaks. This is the one that ships.
Across 5 models and 22 trials: Across 6 models and 25 trials:
arm converged CORRECT converged WRONG split arm converged CORRECT converged WRONG split
no memory 0/22 16/22 6/22 no memory 0/25 21/25 4/25
dense cosine RAG 4/21 10/21 7/21 dense cosine RAG 4/23 12/23 7/23
Vestige 19/21 0/21 2/21 Vestige 20/23 0/23 3/23
The middle column is the finding. A memoryless fleet marched confidently off a The middle column is the finding. A memoryless fleet marched confidently off a
cliff 16 times out of 22. Dense cosine did it 10 times out of 21. cliff 21 times out of 25. Dense cosine did it 12 times out of 23.
**The agents were not reasoning badly.** GPT-5.6 Sol, committing to the wrong key: **The agents were not reasoning badly.** GPT-5.6 Sol, committing to the wrong key:
@ -133,7 +133,7 @@ from temporal lifecycle. A reviewer can fairly argue I won by removing query
variability alone. That is the next experiment. variability alone. That is the next experiment.
And I built the benchmark, the harness, and one of the products in it. Which is And I built the benchmark, the harness, and one of the products in it. Which is
why all 210 agent transcripts ship with it rather than a summary table. why all 246 agent transcripts ship with it rather than a summary table.
--- ---