docs: fix broken audit paths and recount the backfill hit rate

Every path in FINDING.md's "Reproducing it" section and EVIDENCE.md's "Check it
yourself" section pointed at results/trial-1/, which does not exist. The trial
they describe is results/runA-trial-1/. A reader following the audit instructions
got "No such file or directory", which is the worst possible failure for a
document whose whole argument is that you should not take my word for it.

EVIDENCE.md said the key was surfaced on 47 of 48 successful backfill calls.
Recounted from the transcripts on this branch it is 65 of 65, with zero empty
results and zero errors: 15/15 GLM 5.2, 15/15 GPT-5.6 Sol, 14/14 DeepSeek V4
Flash, 12/12 Kimi K3, 6/6 Kimi K2.7 Code, 3/3 MiniMax M3. The old number was
computed on a smaller model set and never recounted. Added the recount procedure
so a reader can reproduce the figure instead of trusting it.

Also repaired a dropped clause in README.md's known-limitations list.
This commit is contained in:
Sam Valladares 2026-07-23 00:05:57 +08:00
parent c6b642a31a
commit 752b44d835
3 changed files with 14 additions and 10 deletions

View file

@ -237,9 +237,14 @@ Number of times each model's dense-cosine agents re-queried before committing:
reformulate.** Ask once and accept the answer, and you get the decoy. Kimi K3 re-queries twice on
average and sometimes escapes.
Vestige is **1 call per agent on every model tested**, and the key was surfaced on 47 of 48 successful
calls, because there is no query to reformulate. `vestige_backfill` takes no query argument. It anchors
on the failure event and walks the causal edge backward.
Vestige is **1 call per agent on every model tested**, and the key was surfaced on all 65 successful
calls across all six models, because there is no query to reformulate. `vestige_backfill` takes no
query argument. It anchors on the failure event and walks the causal edge backward.
To recount: parse every `results/*/transcript-sync-a*.json`, take each `tool_results` entry named
`vestige_backfill`, and test its output against that trial's `manifest.json` `correct_kid`. Per model
that is 15/15 GLM 5.2, 15/15 GPT-5.6 Sol, 14/14 DeepSeek V4 Flash, 12/12 Kimi K3, 6/6 Kimi K2.7 Code,
3/3 MiniMax M3. Zero empty results and zero errors.
So the honest cross-model statement is not "this works everywhere." It is: **the memory layer's
behaviour was invariant across five models and the baseline's was not.**
@ -292,7 +297,7 @@ found by attacking our own work today.
```
harness/ the runner, the fleet driver, the trial generator
tests/bm25_baseline.py reproduces the ranking table, no API key needed
results/trial-1/ all 7 arms, all 21 agent transcripts
results/runA-trial-1/ all 7 arms, all 21 agent transcripts
```
Start here if you are auditing: every arm JSON carries `memory_layer_alive` and `retrieval_err_total`.

View file

@ -4,7 +4,7 @@
Trial 1 of the Silent Rotation benchmark. Kimi K3, seven memory configurations, one randomized secret.
Every number and every quotation below comes from the raw agent transcripts, which are published
alongside this document in `results/trial-1/`.
alongside this document in `results/runA-trial-1/`.
---
@ -289,13 +289,13 @@ Everything needed to check this is in the repository.
```
harness/ the runner, the fleet driver, and the trial generator
harness/agent/prepare_trial.py seeds the corpus and randomizes the key
results/trial-1/ all 7 arm results and all 21 agent transcripts
results/runA-trial-1/ all 7 arm results and all 21 agent transcripts
tests/ the arm liveness gate
```
The claims above map to files. The outcome table is `results/trial-1/{arm}.json`, field
The claims above map to files. The outcome table is `results/runA-trial-1/{arm}.json`, field
`fleet_verdict`. The per-agent key choices are `fix_directions` in the same files. Every quotation is in
`results/trial-1/transcript-{arm}-a{n}.json` under `turns[].reasoning` and `turns[].tool_results`.
`results/runA-trial-1/transcript-{arm}-a{n}.json` under `turns[].reasoning` and `turns[].tool_results`.
One thing to check first if you are auditing this: each arm's JSON carries `memory_layer_alive` and
`retrieval_err_total`. An arm whose backend never answered is not a retrieval loss, it is a missing

View file

@ -136,8 +136,7 @@ These are stated at greater length in `FINDING.md`, and none of them are hidden.
alone. That is the next experiment.
- On Kimi K3 the dense cosine baseline never converged wrong (3 correct, 2 split of 5) and it
costs less per run. Across all 23 shared trials it is 4 correct to Vestige's 20, so the gap
is model-dependent rather than uniform.
is no "beats RAG on outcomes" claim here to defend.
is model-dependent rather than uniform. There is no "beats RAG on outcomes" claim here to defend.
- I initially broke the mem0 and Zep arms by failing to flush state between
trials, which disadvantaged them. The harnesses are fixed and those arms were
re-run clean. Both the broken and the repaired cells are published.