mirror of
https://github.com/samvallad33/vestige.git
synced 2026-07-24 23:41:01 +02:00
docs: fix broken audit paths and recount the backfill hit rate
Every path in FINDING.md's "Reproducing it" section and EVIDENCE.md's "Check it yourself" section pointed at results/trial-1/, which does not exist. The trial they describe is results/runA-trial-1/. A reader following the audit instructions got "No such file or directory", which is the worst possible failure for a document whose whole argument is that you should not take my word for it. EVIDENCE.md said the key was surfaced on 47 of 48 successful backfill calls. Recounted from the transcripts on this branch it is 65 of 65, with zero empty results and zero errors: 15/15 GLM 5.2, 15/15 GPT-5.6 Sol, 14/14 DeepSeek V4 Flash, 12/12 Kimi K3, 6/6 Kimi K2.7 Code, 3/3 MiniMax M3. The old number was computed on a smaller model set and never recounted. Added the recount procedure so a reader can reproduce the figure instead of trusting it. Also repaired a dropped clause in README.md's known-limitations list.
This commit is contained in:
parent
c6b642a31a
commit
752b44d835
3 changed files with 14 additions and 10 deletions
|
|
@ -237,9 +237,14 @@ Number of times each model's dense-cosine agents re-queried before committing:
|
|||
reformulate.** Ask once and accept the answer, and you get the decoy. Kimi K3 re-queries twice on
|
||||
average and sometimes escapes.
|
||||
|
||||
Vestige is **1 call per agent on every model tested**, and the key was surfaced on 47 of 48 successful
|
||||
calls, because there is no query to reformulate. `vestige_backfill` takes no query argument. It anchors
|
||||
on the failure event and walks the causal edge backward.
|
||||
Vestige is **1 call per agent on every model tested**, and the key was surfaced on all 65 successful
|
||||
calls across all six models, because there is no query to reformulate. `vestige_backfill` takes no
|
||||
query argument. It anchors on the failure event and walks the causal edge backward.
|
||||
|
||||
To recount: parse every `results/*/transcript-sync-a*.json`, take each `tool_results` entry named
|
||||
`vestige_backfill`, and test its output against that trial's `manifest.json` `correct_kid`. Per model
|
||||
that is 15/15 GLM 5.2, 15/15 GPT-5.6 Sol, 14/14 DeepSeek V4 Flash, 12/12 Kimi K3, 6/6 Kimi K2.7 Code,
|
||||
3/3 MiniMax M3. Zero empty results and zero errors.
|
||||
|
||||
So the honest cross-model statement is not "this works everywhere." It is: **the memory layer's
|
||||
behaviour was invariant across five models and the baseline's was not.**
|
||||
|
|
@ -292,7 +297,7 @@ found by attacking our own work today.
|
|||
```
|
||||
harness/ the runner, the fleet driver, the trial generator
|
||||
tests/bm25_baseline.py reproduces the ranking table, no API key needed
|
||||
results/trial-1/ all 7 arms, all 21 agent transcripts
|
||||
results/runA-trial-1/ all 7 arms, all 21 agent transcripts
|
||||
```
|
||||
|
||||
Start here if you are auditing: every arm JSON carries `memory_layer_alive` and `retrieval_err_total`.
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@
|
|||
|
||||
Trial 1 of the Silent Rotation benchmark. Kimi K3, seven memory configurations, one randomized secret.
|
||||
Every number and every quotation below comes from the raw agent transcripts, which are published
|
||||
alongside this document in `results/trial-1/`.
|
||||
alongside this document in `results/runA-trial-1/`.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -289,13 +289,13 @@ Everything needed to check this is in the repository.
|
|||
```
|
||||
harness/ the runner, the fleet driver, and the trial generator
|
||||
harness/agent/prepare_trial.py seeds the corpus and randomizes the key
|
||||
results/trial-1/ all 7 arm results and all 21 agent transcripts
|
||||
results/runA-trial-1/ all 7 arm results and all 21 agent transcripts
|
||||
tests/ the arm liveness gate
|
||||
```
|
||||
|
||||
The claims above map to files. The outcome table is `results/trial-1/{arm}.json`, field
|
||||
The claims above map to files. The outcome table is `results/runA-trial-1/{arm}.json`, field
|
||||
`fleet_verdict`. The per-agent key choices are `fix_directions` in the same files. Every quotation is in
|
||||
`results/trial-1/transcript-{arm}-a{n}.json` under `turns[].reasoning` and `turns[].tool_results`.
|
||||
`results/runA-trial-1/transcript-{arm}-a{n}.json` under `turns[].reasoning` and `turns[].tool_results`.
|
||||
|
||||
One thing to check first if you are auditing this: each arm's JSON carries `memory_layer_alive` and
|
||||
`retrieval_err_total`. An arm whose backend never answered is not a retrieval loss, it is a missing
|
||||
|
|
|
|||
|
|
@ -136,8 +136,7 @@ These are stated at greater length in `FINDING.md`, and none of them are hidden.
|
|||
alone. That is the next experiment.
|
||||
- On Kimi K3 the dense cosine baseline never converged wrong (3 correct, 2 split of 5) and it
|
||||
costs less per run. Across all 23 shared trials it is 4 correct to Vestige's 20, so the gap
|
||||
is model-dependent rather than uniform.
|
||||
is no "beats RAG on outcomes" claim here to defend.
|
||||
is model-dependent rather than uniform. There is no "beats RAG on outcomes" claim here to defend.
|
||||
- I initially broke the mem0 and Zep arms by failing to flush state between
|
||||
trials, which disadvantaged them. The harnesses are fixed and those arms were
|
||||
re-run clean. Both the broken and the repaired cells are published.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue