fix(silent-rotation): recount first-call rates across all 25 trials and publish the method

The first-call figures in QUOTES.md were computed on the five-model set before
kimi-k2.7-code was restored and before mem0 and Zep were re-run on the repaired
harness. Recounted directly from results/*/transcript-*.json by walking
turns[].tool_results[], taking the first entry whose name is a memory tool, and
testing whether manifest.json's correct_kid appears in its output:

  arm                          correct    n     rate
  sync (Vestige, no query)          65   65     100%
  rag (dense cosine)                 6   66     9.1%
  supermemory                        1   18     5.6%
  mem0                               0   15       0%
  hindsight                          0    9       0%
  zep                                0    6       0%
  all query-based combined           7  114     6.1%

Previously published as 56/56 and 6/87. Vestige is perfect on a larger
denominator; the query-based aggregate moves 6.9% -> 6.1% on 114 calls instead
of 87. The anarchy arm has no memory tool and is excluded, which accounts for
the 67 transcripts with no memory call.

EVIDENCE.md now carries the table and the exact recount procedure so a reader
can reproduce it without rerunning any trials.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Sam Valladares 2026-07-22 00:14:51 +07:00
parent fa7250d522
commit c2d0606f2c
2 changed files with 22 additions and 3 deletions

View file

@ -26,6 +26,25 @@ it the right one?**
| zep | 0/2 | 1/2 | 1/2 | 2 |
Models: Kimi K3, Kimi K2.7-code, MiniMax M3, GLM 5.2, GPT-5.6 Sol, DeepSeek V4 Flash.
### First memory call
For every agent, the FIRST memory-tool call it made: did the returned payload contain the trial's
correct key? This isolates the memory substrate from the model's ability to recover on later turns.
| arm | first call correct | n | rate |
|---|---|---|---|
| **sync (Vestige, no query)** | **65** | **65** | **100%** |
| rag (dense cosine) | 6 | 66 | 9.1% |
| supermemory | 1 | 18 | 5.6% |
| mem0 | 0 | 15 | 0% |
| hindsight | 0 | 9 | 0% |
| zep | 0 | 6 | 0% |
| **all query-based combined** | **7** | **114** | **6.1%** |
The `anarchy` arm has no memory tool and is excluded. Reproduce: for each
`results/*/transcript-*.json`, walk `turns[].tool_results[]`, take the first entry whose `name` is a
memory tool, and test whether `manifest.json`'s `correct_kid` appears in its `output`.
Reproduce: read `fix_directions` and `correct_kid` out of every `trial-*/{arm}.json` and `manifest.json`.
### There are two failure modes and only one of them is visible

View file

@ -71,7 +71,7 @@ two failed queries:
It concluded the memory layer had nothing and went to the source. It needed six queries in total and
first surfaced the correct key on call four.
This is what a 6.9% first-call hit rate feels like from inside the agent.
This is what a 6.1% first-call hit rate feels like from inside the agent.
---
@ -152,8 +152,8 @@ configured one. Fixed and re-run: mem0 converged correctly in `runB-trial-3` and
**It proves:** at turn zero an agent knows only the symptom, so it queries the symptom. In this incident
class the symptom query returns a document that resembles the failure and buries the one that explains
it, and that holds under dense cosine and BM25 alike. Across 6 models and 25 trials, a query-based arm's
first memory call surfaced the deciding fact 6 times out of 87. Anchoring on the failure event instead
of a query did it 56 out of 56. In a fleet, that first-call gap becomes either a merge conflict or a
first memory call surfaced the deciding fact 7 times out of 114. Anchoring on the failure event instead
of a query did it 65 out of 65. In a fleet, that first-call gap becomes either a merge conflict or a
unanimous wrong answer that passes tests and breaks production.
**It does not prove:** that Vestige discovers causal relationships. The edge it traverses is authored by