fix(silent-rotation): correct the outcome tables to match the committed results JSON

The tables in EVIDENCE.md and SHOW-HN.md were written before the sixth model
run (kimi-k2.7-code) was added back and before the mem0 and Zep arms were
re-run on the repaired harness. They reported 5 models and 22 trials while
results/ holds 6 models and 25 trials, so the docs contradicted the data
sitting beside them - anyone running harness/aggregate-experiment.py got
different numbers than the writeup claimed.

Every figure below is now generated from the per-trial <arm>.json files using
the aggregator's own definition (fixed_correctly = correct, green_but_voids_prod
= converged wrong, failed_merge_conflict = split):

  arm             CORRECT  WRONG  SPLIT   n
  anarchy               0     21      4  25
  rag                   4     12      7  23
  sync (Vestige)       20      0      3  23
  supermemory           5      0      1   6
  mem0                  2      1      2   5
  hindsight             0      0      3   3
  zep                   0      1      1   2

The corrections run in both directions: the no-memory control failed more than
claimed (21 vs 16), dense cosine failed more (12 vs 10), Vestige won one more
(20 vs 19), and Zep has one converged-wrong that was previously reported as a
split. The headline is unchanged - Vestige has zero converged-wrong across 23
trials. Transcript count corrected 210 -> 246 (verified by file count).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Sam Valladares 2026-07-21 23:43:35 +07:00
parent fd77ff31a6
commit fa7250d522
3 changed files with 15 additions and 15 deletions

View file

@ -17,15 +17,15 @@ it the right one?**
| arm | converged on the CORRECT key | converged on a WRONG key | split, no consensus | n |
|---|---|---|---|---|
| anarchy (no memory) | 0/22 | **16/22** | 6/22 | 22 |
| rag (dense cosine) | 4/21 | **10/21** | 7/21 | 21 |
| **sync (Vestige)** | **19/21** | **0/21** | 2/21 | 21 |
| supermemory | 4/5 | 0/5 | 1/5 | 5 |
| mem0 | 0/5 | 1/5 | 4/5 | 5 |
| hindsight | 0/5 | 0/5 | 5/5 | 5 |
| zep | 0/4 | 2/4 | 2/4 | 4 |
| anarchy (no memory) | 0/25 | **21/25** | 4/25 | 25 |
| rag (dense cosine) | 4/23 | **12/23** | 7/23 | 23 |
| **sync (Vestige)** | **20/23** | **0/23** | 3/23 | 23 |
| supermemory | 5/6 | 0/6 | 1/6 | 6 |
| mem0 | 2/5 | 1/5 | 2/5 | 5 |
| hindsight | 0/3 | 0/3 | 3/3 | 3 |
| zep | 0/2 | 1/2 | 1/2 | 2 |
Models: Kimi K3, MiniMax M3, GLM 5.2, GPT-5.6 Sol, DeepSeek V4 Flash.
Models: Kimi K3, Kimi K2.7-code, MiniMax M3, GLM 5.2, GPT-5.6 Sol, DeepSeek V4 Flash.
Reproduce: read `fix_directions` and `correct_kid` out of every `trial-*/{arm}.json` and `manifest.json`.
### There are two failure modes and only one of them is visible

View file

@ -151,7 +151,7 @@ configured one. Fixed and re-run: mem0 converged correctly in `runB-trial-3` and
**It proves:** at turn zero an agent knows only the symptom, so it queries the symptom. In this incident
class the symptom query returns a document that resembles the failure and buries the one that explains
it, and that holds under dense cosine and BM25 alike. Across 5 models and 22 trials, a query-based arm's
it, and that holds under dense cosine and BM25 alike. Across 6 models and 25 trials, a query-based arm's
first memory call surfaced the deciding fact 6 times out of 87. Anchoring on the failure event instead
of a query did it 56 out of 56. In a fleet, that first-call gap becomes either a merge conflict or a
unanimous wrong answer that passes tests and breaks production.

View file

@ -42,15 +42,15 @@ ships. Loud. You find out immediately.
Converged wrong: every agent agrees, on the decoy. Tests go green. The merge is
clean. Production breaks. This is the one that ships.
Across 5 models and 22 trials:
Across 6 models and 25 trials:
arm converged CORRECT converged WRONG split
no memory 0/22 16/22 6/22
dense cosine RAG 4/21 10/21 7/21
Vestige 19/21 0/21 2/21
no memory 0/25 21/25 4/25
dense cosine RAG 4/23 12/23 7/23
Vestige 20/23 0/23 3/23
The middle column is the finding. A memoryless fleet marched confidently off a
cliff 16 times out of 22. Dense cosine did it 10 times out of 21.
cliff 21 times out of 25. Dense cosine did it 12 times out of 23.
**The agents were not reasoning badly.** GPT-5.6 Sol, committing to the wrong key:
@ -133,7 +133,7 @@ from temporal lifecycle. A reviewer can fairly argue I won by removing query
variability alone. That is the next experiment.
And I built the benchmark, the harness, and one of the products in it. Which is
why all 210 agent transcripts ship with it rather than a summary table.
why all 246 agent transcripts ship with it rather than a summary table.
---