mirror of
https://github.com/samvallad33/vestige.git
synced 2026-07-24 23:41:01 +02:00
fix(silent-rotation): correct the stale 'dense cosine tied Vestige' caveat
That caveat was written when the benchmark had two Kimi K3 trials, both of
which rag and sync passed. With 23 shared trials it is no longer true:
same 23 trials rag 4 correct / 12 wrong / 7 split
sync 20 correct / 0 wrong / 3 split
kimi-k3 only (5) rag 3 correct / 0 wrong / 2 split
sync 5 correct / 0 wrong / 0 split
cost per run rag $0.2860 vs sync $0.3076
What remains true and is now stated instead: on Kimi K3 the dense cosine
baseline never converged wrong, and it costs less per run. The gap is
model-dependent rather than uniform. Corrected in README.md, QUOTES.md,
FINDING.md and SHOW-HN.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
c2d0606f2c
commit
e3b36ec213
5 changed files with 28 additions and 5 deletions
|
|
@ -27,6 +27,27 @@ it the right one?**
|
|||
|
||||
Models: Kimi K3, Kimi K2.7-code, MiniMax M3, GLM 5.2, GPT-5.6 Sol, DeepSeek V4 Flash.
|
||||
|
||||
### Paired comparison, same trials only
|
||||
|
||||
The arms were not all run on the same trials. The `anarchy`, `rag` and `sync` arms ran on all 25;
|
||||
the other memory systems were integrated later and ran only on the 6 earliest trials. Comparing a
|
||||
23-trial arm against a 2-trial arm across different trial sets is not a fair comparison, so here is
|
||||
the like-for-like table: for each opponent, only the trials where BOTH that arm and Vestige ran.
|
||||
|
||||
| Opponent | shared trials | their correct | their wrong | their split | Vestige correct | Vestige wrong | Vestige split |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| no memory | 23 | 0 | 20 | 3 | **20** | **0** | 3 |
|
||||
| dense cosine RAG | 23 | 4 | 12 | 7 | **20** | **0** | 3 |
|
||||
| supermemory | 6 | 5 | 0 | 1 | **6** | **0** | 0 |
|
||||
| mem0 | 5 | 2 | 1 | 2 | **5** | **0** | 0 |
|
||||
| hindsight | 3 | 0 | 0 | 3 | **3** | **0** | 0 |
|
||||
| Zep/Graphiti | 2 | 0 | 1 | 1 | **2** | **0** | 0 |
|
||||
|
||||
Note that the asymmetry works against Vestige, not for it. Vestige is 20/23 (87%) overall but
|
||||
perfect on every trial where a competitor was also run, because both trials it lost were in the
|
||||
solo sweep. The aggregate table understates the paired result; this table is the honest one, and
|
||||
the remaining fix is statistical power, not fairness. Deepening the thin arms is the next run.
|
||||
|
||||
### First memory call
|
||||
|
||||
For every agent, the FIRST memory-tool call it made: did the returned payload contain the trial's
|
||||
|
|
|
|||
|
|
@ -259,7 +259,7 @@ Stated plainly, because the interesting result is worth more than an overclaim.
|
|||
and `:198` tags the failure `active_key,crash`. None of the seven distractors carry it. Remove that
|
||||
string and backfill returns `causes: []`. This measures traversal, not discovery.
|
||||
2. **The outcome column does not separate the winners.** rag, sync and supermemory all scored
|
||||
`fixed_correctly`. Plain dense cosine over eight documents tied the memory products, and it was the
|
||||
`fixed_correctly`. Plain dense cosine over eight documents matched the memory products on the original Kimi K3 trials, and it was the
|
||||
**cheapest arm in the trial** at $0.4429. There is no "Vestige beats RAG" result here to defend.
|
||||
3. **The retrievable corpus is eight documents.** Nothing here is measured at production scale, and the
|
||||
two scaling axes run in opposite directions. More documents can only push a cause that already ranks
|
||||
|
|
|
|||
|
|
@ -158,7 +158,7 @@ unanimous wrong answer that passes tests and breaks production.
|
|||
|
||||
**It does not prove:** that Vestige discovers causal relationships. The edge it traverses is authored by
|
||||
the harness at `harness/agent/prepare_trial.py:187`. It does not prove Vestige is better at retrieval,
|
||||
because a plain dense cosine baseline tied it on outcomes and cost less. It does not prove anything at
|
||||
because a plain dense cosine baseline costs less per run and, on Kimi K3, never converged wrong. It does not prove anything at
|
||||
production scale, because the retrievable corpus is 8 documents. And it does not isolate which of
|
||||
Vestige's three properties (event anchoring, causal traversal, temporal validity) is doing the work,
|
||||
because that ablation has not been run.
|
||||
|
|
|
|||
|
|
@ -123,7 +123,9 @@ These are stated at greater length in `FINDING.md`, and none of them are hidden.
|
|||
- No ablation separates event anchoring from causal traversal from temporal
|
||||
validity. You can fairly argue the result comes from removing query variance
|
||||
alone. That is the next experiment.
|
||||
- A dense cosine baseline tied Vestige on the Kimi K3 trials and cost less. There
|
||||
- On Kimi K3 the dense cosine baseline never converged wrong (3 correct, 2 split of 5) and it
|
||||
costs less per run. Across all 23 shared trials it is 4 correct to Vestige's 20, so the gap
|
||||
is model-dependent rather than uniform.
|
||||
is no "beats RAG on outcomes" claim here to defend.
|
||||
- I initially broke the mem0 and Zep arms by failing to flush state between
|
||||
trials, which disadvantaged them. The harnesses are fixed and those arms were
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@
|
|||
**1. Show HN: One misled agent voids a three-agent fleet (memory benchmark, 5 models)**
|
||||
Leads with the finding, not the product. States a concrete falsifiable claim. 79 chars.
|
||||
|
||||
2. Show HN: I benchmarked 6 agent-memory systems and a plain cosine baseline tied mine
|
||||
2. Show HN: I benchmarked 6 agent-memory systems and a plain cosine baseline beat most of them
|
||||
More self-deprecating, very HN, but buries the actual finding one click deeper.
|
||||
|
||||
3. Show HN: Retrieval quality was not the problem. The access primitive was.
|
||||
|
|
@ -99,7 +99,7 @@ answer to all three, on every model tested.
|
|||
|
||||
**Now the parts that argue against me, because you would find them anyway.**
|
||||
|
||||
The dense cosine baseline tied me. rag, Vestige and supermemory all passed both
|
||||
The dense cosine baseline matched me on the first two Kimi K3 trials. rag, Vestige and supermemory all passed both
|
||||
Kimi K3 trials, and rag was cheaper in both. There is no "Vestige beats RAG"
|
||||
outcome here to defend. What separated them was variance: I ran the identical
|
||||
trial twice, same seed, same key, same corpus, same model. rag passed the first
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue