mirror of
https://github.com/samvallad33/vestige.git
synced 2026-07-24 23:41:01 +02:00
fix(silent-rotation): correct the outcome tables to match the committed results JSON
The tables in EVIDENCE.md and SHOW-HN.md were written before the sixth model run (kimi-k2.7-code) was added back and before the mem0 and Zep arms were re-run on the repaired harness. They reported 5 models and 22 trials while results/ holds 6 models and 25 trials, so the docs contradicted the data sitting beside them - anyone running harness/aggregate-experiment.py got different numbers than the writeup claimed. Every figure below is now generated from the per-trial <arm>.json files using the aggregator's own definition (fixed_correctly = correct, green_but_voids_prod = converged wrong, failed_merge_conflict = split): arm CORRECT WRONG SPLIT n anarchy 0 21 4 25 rag 4 12 7 23 sync (Vestige) 20 0 3 23 supermemory 5 0 1 6 mem0 2 1 2 5 hindsight 0 0 3 3 zep 0 1 1 2 The corrections run in both directions: the no-memory control failed more than claimed (21 vs 16), dense cosine failed more (12 vs 10), Vestige won one more (20 vs 19), and Zep has one converged-wrong that was previously reported as a split. The headline is unchanged - Vestige has zero converged-wrong across 23 trials. Transcript count corrected 210 -> 246 (verified by file count). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
fd77ff31a6
commit
fa7250d522
3 changed files with 15 additions and 15 deletions
|
|
@ -17,15 +17,15 @@ it the right one?**
|
|||
|
||||
| arm | converged on the CORRECT key | converged on a WRONG key | split, no consensus | n |
|
||||
|---|---|---|---|---|
|
||||
| anarchy (no memory) | 0/22 | **16/22** | 6/22 | 22 |
|
||||
| rag (dense cosine) | 4/21 | **10/21** | 7/21 | 21 |
|
||||
| **sync (Vestige)** | **19/21** | **0/21** | 2/21 | 21 |
|
||||
| supermemory | 4/5 | 0/5 | 1/5 | 5 |
|
||||
| mem0 | 0/5 | 1/5 | 4/5 | 5 |
|
||||
| hindsight | 0/5 | 0/5 | 5/5 | 5 |
|
||||
| zep | 0/4 | 2/4 | 2/4 | 4 |
|
||||
| anarchy (no memory) | 0/25 | **21/25** | 4/25 | 25 |
|
||||
| rag (dense cosine) | 4/23 | **12/23** | 7/23 | 23 |
|
||||
| **sync (Vestige)** | **20/23** | **0/23** | 3/23 | 23 |
|
||||
| supermemory | 5/6 | 0/6 | 1/6 | 6 |
|
||||
| mem0 | 2/5 | 1/5 | 2/5 | 5 |
|
||||
| hindsight | 0/3 | 0/3 | 3/3 | 3 |
|
||||
| zep | 0/2 | 1/2 | 1/2 | 2 |
|
||||
|
||||
Models: Kimi K3, MiniMax M3, GLM 5.2, GPT-5.6 Sol, DeepSeek V4 Flash.
|
||||
Models: Kimi K3, Kimi K2.7-code, MiniMax M3, GLM 5.2, GPT-5.6 Sol, DeepSeek V4 Flash.
|
||||
Reproduce: read `fix_directions` and `correct_kid` out of every `trial-*/{arm}.json` and `manifest.json`.
|
||||
|
||||
### There are two failure modes and only one of them is visible
|
||||
|
|
|
|||
|
|
@ -151,7 +151,7 @@ configured one. Fixed and re-run: mem0 converged correctly in `runB-trial-3` and
|
|||
|
||||
**It proves:** at turn zero an agent knows only the symptom, so it queries the symptom. In this incident
|
||||
class the symptom query returns a document that resembles the failure and buries the one that explains
|
||||
it, and that holds under dense cosine and BM25 alike. Across 5 models and 22 trials, a query-based arm's
|
||||
it, and that holds under dense cosine and BM25 alike. Across 6 models and 25 trials, a query-based arm's
|
||||
first memory call surfaced the deciding fact 6 times out of 87. Anchoring on the failure event instead
|
||||
of a query did it 56 out of 56. In a fleet, that first-call gap becomes either a merge conflict or a
|
||||
unanimous wrong answer that passes tests and breaks production.
|
||||
|
|
|
|||
|
|
@ -42,15 +42,15 @@ ships. Loud. You find out immediately.
|
|||
Converged wrong: every agent agrees, on the decoy. Tests go green. The merge is
|
||||
clean. Production breaks. This is the one that ships.
|
||||
|
||||
Across 5 models and 22 trials:
|
||||
Across 6 models and 25 trials:
|
||||
|
||||
arm converged CORRECT converged WRONG split
|
||||
no memory 0/22 16/22 6/22
|
||||
dense cosine RAG 4/21 10/21 7/21
|
||||
Vestige 19/21 0/21 2/21
|
||||
no memory 0/25 21/25 4/25
|
||||
dense cosine RAG 4/23 12/23 7/23
|
||||
Vestige 20/23 0/23 3/23
|
||||
|
||||
The middle column is the finding. A memoryless fleet marched confidently off a
|
||||
cliff 16 times out of 22. Dense cosine did it 10 times out of 21.
|
||||
cliff 21 times out of 25. Dense cosine did it 12 times out of 23.
|
||||
|
||||
**The agents were not reasoning badly.** GPT-5.6 Sol, committing to the wrong key:
|
||||
|
||||
|
|
@ -133,7 +133,7 @@ from temporal lifecycle. A reviewer can fairly argue I won by removing query
|
|||
variability alone. That is the next experiment.
|
||||
|
||||
And I built the benchmark, the harness, and one of the products in it. Which is
|
||||
why all 210 agent transcripts ship with it rather than a summary table.
|
||||
why all 246 agent transcripts ship with it rather than a summary table.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue