apunkt/vestige
1
0
Fork 0
mirror of https://github.com/samvallad33/vestige.git synced 2026-07-24 23:41:01 +02:00
Code Issues Projects Releases Packages Wiki Activity

docs: fix v2.2.0 drift and purge private content

Browse source 
Drift corrections:
- server.rs: advertised-tool comment 12 -> 13 (matches the code + test)
- crates/vestige-mcp/README.md: replace stale pre-consolidation tool
  list with the real 13 advertised tools + hidden-alias note
- tool-consolidation-v2.2.0.md: reconcile 12 (fold) vs 13 (fold +
  net-new backfill), matching test_tools_list_returns_all_tools
- VESTIGE_STATE_AND_PLAN.md: current release shape v2.1.21 -> v2.2.0
- CONFIGURATION.md: stale --version example -> v2.2.0
- README.md / CONTRIBUTING.md: LOC 86,000+ -> 92,000+, binary 22MB -> 23MB
  (test badge 1,550 verified accurate via full suite: 1550 passed / 0 failed)

Private content removed:
- marketing/funding collateral (docs/launch/{blog-post,demo-script,
  show-hn,opencode-adoption,reddit-cross-reference}, demo pitch/funding files)
- unshipped-cloud planning cluster (docs/plans, docs/prd, docs/adr) which
  carried real private local paths and machine names
- README: repoint a broken/private demo link to the public demo

Kept public: demo/README.md + postdict-demo.sh, tool-consolidation doc,
blackbox-proof pack. Guard (check-no-private-cloud) passes; no private
path markers remain; no broken cross-references.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B3RxH4JxAQYsuA9kFVnR5v
This commit is contained in:
Claude 2026-07-01 15:03:47 +00:00
parent 7aade490d8
commit e89a922cf8
No known key found for this signature in database
35 changed files with 74 additions and 19889 deletions
Download patch file Download diff file Expand all files Collapse all files

2
CONTRIBUTING.md
View file

@ -79,7 +79,7 @@ cd apps/dashboard && pnpm build
# Debug build
cargo build -p vestige-mcp
# Release build (22MB binary with embedded dashboard)
# Release build (23MB binary with embedded dashboard)
cargo build --release -p vestige-mcp
# The release binary is at target/release/vestige-mcp

6
README.md
View file

@ -31,7 +31,7 @@ So that's what Vestige is. Everyone else built a memory that **remembers**. I tr
It's one Rust binary. It runs entirely on your machine. It never phones home. And there's a 60-second start right below.
> 🎙️ **The 60-second version** of this whole story, the one I give in person, lives in [`demo/PITCH-v2-causebench.md`](demo/PITCH-v2-causebench.md). If you've got a minute, read that first. It's the clearest way to *get* why this matters.
> 🎙️ **Want the 60-second version?** Run the [Postdict demo](demo/README.md): one command, a throwaway DB, and you watch Vestige reach backward through time to a root cause a vector search can't find. It's the clearest way to *get* why this matters.
---
@ -272,7 +272,7 @@ Registering the server exposes the tools; a short instruction tells the agent *w
| | |
|---|---|
| **Language** | Rust 2024 (MSRV 1.91), **86,000+ lines** |
| **Language** | Rust 2024 (MSRV 1.91), **92,000+ lines** |
| **Binary** | ~23MB, single file |
| **Embeddings** | Nomic Embed Text v1.5 (768d→256d Matryoshka, 8192 ctx); Qwen3 optional |
| **Vector search** | USearch HNSW (≈20× faster than FAISS) |
@ -299,7 +299,7 @@ Registering the server exposes the tools; a short instruction tells the agent *w
### If your agent should remember what you taught it yesterday, star it. ⭐
<sub><b>86,000+ lines of Rust · 13 tools · 30 cognitive modules · 130 years of memory research · one 23MB binary that never phones home.</b></sub>
<sub><b>92,000+ lines of Rust · 13 tools · 30 cognitive modules · 130 years of memory research · one 23MB binary that never phones home.</b></sub>
<sub>Built by <a href="https://github.com/samvallad33">@samvallad33</a> · AGPL-3.0 · 100% local, 100% yours</sub>

33
crates/vestige-mcp/README.md
View file

@ -54,17 +54,30 @@ HTTP and dashboard bearer tokens are generated locally; see
## Current Tool Surface
The server exposes the current unified MCP tools from
[`src/server.rs`](src/server.rs), including:
As of v2.2, the server advertises **13 consolidated tools** from
[`src/server.rs`](src/server.rs):
- `session_context`
- `search`, `smart_ingest`, `memory`, `codebase`, `intention`
- `deep_reference`, `cross_reference`, `contradictions`
- `dream`, `explore_connections`, `predict`
- `memory_health`, `memory_graph`, `composed_graph`, `system_status`
- `importance_score`, `find_duplicates`
- `consolidate`, `memory_timeline`, `memory_changelog`
- `backup`, `export`, `restore`, `gc`, `suppress`
- `recall` — unified retrieval (folds `search` + `deep_reference` +
`cross_reference` + `contradictions`); modes `lookup`/`reason`/`contradictions`
- `smart_ingest` — Prediction-Error-gated ingestion (single + batch)
- `memory` — get / edit / promote / demote / state / purge
- `graph` — chains, associations, bridges, predictions, force-directed export
(folds `explore_connections` + `predict` + `memory_graph` + `composed_graph`)
- `maintain` — consolidate / dream / gc / importance_score / backup / export / restore
- `dedup` — scan / plan_merge / plan_supersede / apply / undo / protect / policy
(folds `find_duplicates` + the 7 Phase-3 merge tools)
- `memory_status` — health / retention / timeline / changelog
(folds `system_status` + `memory_health` + `memory_timeline` + `memory_changelog`)
- `suppress` — top-down active forgetting (Anderson 2025 + Davis Rac1)
- `backfill` — Retroactive Salience Backfill (Cai 2024 Nature), the flagship
- `codebase` — per-project patterns and decisions
- `intention` — set / check / update / list future triggers
- `source_sync` — external-source connectors (GitHub, Redmine)
- `session_start` — one-call session initialization (renamed from `session_context`)
The ~22 pre-consolidation names (`search`, `deep_reference`, `dream`,
`consolidate`, `memory_graph`, …) still work as **hidden back-compat aliases**
dispatched in `handle_tools_call`, so existing configs keep working.
See the root [`README.md`](../../README.md) and
[`docs/AGENT-MEMORY-PROTOCOL.md`](../../docs/AGENT-MEMORY-PROTOCOL.md) for

7
crates/vestige-mcp/src/server.rs
View file

@ -240,8 +240,11 @@ impl McpServer {
/// Handle tools/list request
async fn handle_tools_list(&self) -> Result<serde_json::Value, JsonRpcError> {
// v2.2: 12 advertised tools after Layer-1 Tool Consolidation
// (verified by `tools.len() == 12` in test_tools_list_returns_all_tools).
// v2.2: 13 advertised tools after Layer-1 Tool Consolidation
// (12 consolidated: dedup + memory_status + graph + maintain + recall,
// session_context renamed; plus the flagship `backfill`, a distinct
// cognitive primitive kept separate from `maintain`).
// Verified by `tools.len() == 13` in test_tools_list_returns_all_tools.
// 22 deprecated/folded names still work as hidden redirects in
// handle_tools_call. See docs/launch/tool-consolidation-v2.2.0.md.
let mut tools = vec![

77
demo/PITCH-FINAL-spoken.md
View file

@ -1,77 +0,0 @@
# Vestige — THE PITCH THAT CHANGED EVERYTHING (final, condensed cut)
Sam's stage pitch — ~45-50s spoken, for a multi-pitch event with a separate
networking hour afterward. Solo founder, first person, no slides/terminal on
stage (the demo happens 1:1 at networking on Sam's pre-loaded laptop). Two seams
closed from the draft: opener "hours"->"days" (matches "days ago"), and "AI
agents that touch" (plural). NAME = Vestige.
---
## THE PITCH
> Your bug was born days before it crashed — but you just can't remember where.
>
> When production breaks, you lose days hunting a bug. The cause is usually a
> quiet change you made days ago that looks nothing like the error it created.
>
> Today's memory systems fail here, because they look for what *looks like* the
> problem. But a root cause never looks like the bug it creates.
>
> **[beat]**
>
> It's like blaming a goalie for a 90th-minute World Cup goal — when the real
> failure was a silent midfield turnover fifteen minutes earlier. They're
> searching the goal line. Vestige traces the match backward.
>
> A 2024 Nature paper proved the human brain rewinds time to link a sudden shock
> to the quiet memory that caused it. Vestige does the exact same thing for
> software.
>
> Everyone else built a memory that remembers. I built the first one that
> *realizes.* Millions of AI agents that touch production will hit this wall.
> That's not a feature — that's the next layer of the dev stack.
>
> I'm Sam. I created Vestige from my tiny Chicago apartment — the first memory
> that finds the cause, not the lookalike.
>
> Come find me after. I'll show you in thirty seconds.
---
## THE LINES THAT WIN THE ROOM
- **Opener (pattern interrupt):** "Your bug was born days before it crashed — but you just can't remember where."
- **The detonation (THE line):** "A root cause never looks like the bug it creates." → beat of silence after.
- **The soccer proof (the secret weapon — makes ANYONE get it):** "...blaming the goalie for a 90th-minute goal when the real failure was a midfield turnover fifteen minutes earlier. They're searching the goal line. Vestige traces the match backward."
- **The category (the quotable):** "Everyone else built a memory that remembers. I built the first one that realizes."
- **The TAM (the money line):** "That's not a feature — that's the next layer of the dev stack."
- **The closer (fits the networking format):** "Come find me after. I'll show you in thirty seconds."
## STAGE RULES
1. NEVER open with "Hi, I'm Sam, thanks." Walk out, hold the room one beat, THEN
the opener. Your name comes near the END.
2. The detonation line — say it, then a real beat of silence before the soccer line.
3. The soccer analogy is your weapon — deliver it conversationally, like you're
explaining it to a friend. It's what makes a NON-engineer get it instantly.
4. "Built it alone, from a Chicago apartment" is a flex. Own it.
5. End on "I'll show you in thirty seconds." Don't say thank you. Let it hang.
## NETWORKING (the demo happens HERE, not on stage)
- The stage pitch's only job: be the MOST memorable founder so they seek you out
in the networking hour, out of all the other pitches.
- Have the seeded demo (`./demo/postdict-demo.sh`) ALREADY RUNNING / pre-warmed on
your laptop so when someone walks up you show it in ~10 seconds, no boot fumble.
- Demo from STRENGTH: show YOUR seeded scenario (instant, reproducible). Do NOT
promise to debug their codebase live — Vestige has to be recording their history
first. If asked "does it work on mine?": "it has to record your history first —
give me your repo for a day and I'll show you it catch a real one." (That's the
pilot hook, said 1:1, never from stage.)
## DELIVERY MAP
- "born days before it crashed" — land "born" with weight.
- "a root cause never looks like the bug it creates" — then a real beat of silence.
- soccer line — conversational, build to "Vestige traces the match backward."
- "It runs locally" energy on "Vestige does the exact same thing for software."
- "the first one that realizes" — let it land.
- "the next layer of the dev stack" — slow, final. Mic-drop.
- "I'll show you in thirty seconds." — quiet, confident, then stop.

129
demo/funding-demo-script.md
View file

@ -1,129 +0,0 @@
# Postdict — the 60-second funding demo
**Audience:** investors. **Goal:** they see a category, a moat, and a market — not a feature.
**The thesis, in four words:** *relevance does not equal resemblance.*
Format: live terminal, one take, you talking over it. ~60s. Punchy. No slides.
**Rule:** the monologue earns ~20 seconds. The rest is the terminal *doing the impossible* on screen.
---
## THE SCRIPT (what's on screen · what you say)
### [0:00–0:15] — THE FLAWED AXIOM (your cold open, verbatim)
**On screen:** a clean dark terminal, one line:
`relevance ≠ resemblance`
**You say:**
> "Every major AI memory framework on Earth — every VC-backed startup, every
> native platform layer — is built on one flawed assumption: that **relevance
> equals resemblance.** They turn your text into vector embeddings, and when
> something breaks, they search for memories that *look similar* to the problem.
>
> Here's what blows that foundation to pieces: **a root cause never looks like
> the bug it creates.** So the entire industry is searching in the one place the
> answer can never be."
*(Stop. Let it sit one beat. "Relevance equals resemblance" is the line they repeat to their partners.)*
---
### [0:15–0:28] — MAKE IT CONCRETE (type it live)
**On screen:**
```
$ vestige ingest "Set API_TIMEOUT=2 in the deploy env" --ago-days 3 # the quiet cause
$ vestige ingest "500 error in the billing service" --ago-days 20 # an old lookalike
$ vestige ingest "Service crashed: 500 on the auth endpoint" # today's crash
```
**You say:**
> "Watch. A one-line config change three days ago — forgotten. An old, unrelated
> 500 error weeks back. And today, the auth service crashes. Now ask: which past
> memory *caused* today's crash? A vector database ranks by resemblance — so to
> it, today's crash looks most like that old billing 500. **The thing that looks
> similar is never the thing that caused it.**"
---
### [0:28–0:45] — THE PROOF (the money shot · slow down here)
**On screen:** `$ vestige backfill --contrast` →
```
── 1. SIMILARITY SEARCH · keyword (BM25) ──
1. 500 error in the billing service ← top match (WRONG)
→ ranked by RESEMBLANCE. its top hit is a lookalike, not the cause.
── 2. POSTDICT (reach backward for the CAUSE) ──
#1 Set API_TIMEOUT=2 in the deploy env
↩ reached back 3.0 days before the failure
🔗 causal join: api_timeout (RIGHT)
```
**You say (let the `↩ reached back 3.0 days` line hold in silence for a full beat):**
> "Same database. Same query. Similarity search returns the lookalike — confident,
> and wrong. Postdict reaches **backward three days** and finds the actual cause.
> Not because it's similar — because it's **causally upstream.** This is memory
> with hindsight: the 'ohhh, *that's* why' moment, automatic."
---
### [0:45–0:55] — THE MOAT (kill the "can't they just add this?" objection)
**You say:**
> "Two reasons this is defensible. One: it's a faithful port of a 2024 *Nature*
> result — the brain reaches *backward* in time to find causes, and it's
> backward-*only*, which is exactly right, because a root cause is always in the
> past. We didn't invent this; we ported the algorithm evolution already
> perfected. Two: the incumbents can't bolt this on. Their entire architecture
> **is** the flawed axiom. To do this you rebuild memory from the cognitive
> science up — which we already did, and it's running locally, today."
---
### [0:55–1:00] — THE MARKET + THE ASK
**On screen:** `the first memory that finds the cause, not the lookalike.`
**You say:**
> "Every AI agent that writes code, runs infrastructure, or touches production
> hits root causes it can't explain. That's the entire agentic market, and it's
> on fire. We're not a better memory — we're the first memory that **reasons
> backward.** Local-first, reproducible, running now. We're raising [X] to make
> every agent debug like a senior engineer. The seed's in the repo — run it
> yourself."
---
## WHY THIS OPENING IS STRONGER (and how to deliver it)
Your framing beats "category error" because it names the **mechanism** of the
error, not just that one exists:
- **"Relevance equals resemblance"** is a *diagnosis* — it tells the investor
precisely what's broken (the axiom) in four words. "Category error" only says
*that* something's broken.
- **The one-two punch:** state the flawed axiom → detonate it with the fact
("a root cause never looks like the bug it creates"). The investor *feels* the
foundation crack. That's the moment they lean in.
**Delivery rules:**
1. **The monologue is 20 seconds, max.** Investors fund what they *see* work, not
what they hear claimed. Get to the terminal fast; let the contrast carry the
weight.
2. **Memorize two sentences.** The axiom: *"They believe relevance equals
resemblance."* The detonation: *"A root cause never looks like the bug it
creates."* Everything else can be loose.
3. **Silence is the tool at 0:28–0:45.** When `↩ reached back 3.0 days` hits the
screen, say nothing for a full second. The image does the selling.
4. **The moat answer is non-optional.** Every investor thinks "can't Mem0 add
this?" Answer it *before* they ask — their architecture is the axiom. That's
what converts "neat" into "fundable."
5. **End on the market, not the demo.** "Every agent that touches production" is
the TAM. The demo earns the right to say it; don't bury it under the feature.
## THE THREE LINES THAT DO THE WORK
- **The axiom (the hook):** "Every AI memory framework believes relevance equals resemblance."
- **The detonation (the thesis):** "A root cause never looks like the bug it creates."
- **The category (the close):** "We're not a better memory. We're the first memory that reasons backward."

117
demo/funding-demo-v2-blow-it-open.md
View file

@ -1,117 +0,0 @@
# Postdict — the 60-second SPOKEN pitch (solo founder, first person)
**This is a SPEECH.** You, on stage, looking at 50-100 engineers / founders /
investors. No terminal. No slides. No screen. Just your voice and the room.
**First person. "I built this." One person.** For a solo founder that's a
*strength* — it tells investors you can ship alone, which is exactly what they
bet on. Never say "we."
Built for the *ear*: short sentences, hard stops, one line they'll repeat in the
hallway. Read the **[beats]** as pauses — silence is your loudest move on stage.
---
## THE PITCH (≈60 seconds, spoken)
> Every engineer in this room has lost a day to the same thing.
>
> Production breaks. You burn hours hunting. And the cause turns out to be one
> line you changed three days ago — and forgot.
>
> **[pause]**
>
> The bug looked nothing like that change. So you never connected them. You just
> suffered until you stumbled onto it.
>
> **[pause]**
>
> Now we're handing that exact job — debugging — to AI agents. And every AI
> memory system on Earth is about to fail at it. Mem0, Zep, all of them. Because
> they're built on one assumption: that *relevance equals resemblance.* They
> search your memory for whatever *looks like* the problem.
>
> **[slow down — this is the line]**
>
> But a root cause never looks like the bug it creates.
>
> **[full stop. let it hang.]**
>
> So the entire industry is searching in the one place the answer can never be.
>
> **[pause]**
>
> So I built the opposite. When your agent hits a failure, my memory reaches
> *backward in time* — and finds the quiet change, days earlier, that actually
> caused it. The one no similarity search will ever surface, because it isn't
> *similar* — it's *upstream.*
>
> I didn't invent this. I ported it from your brain. There's a 2024 *Nature*
> paper: when something goes wrong, the brain reaches backward to find the cause
> — backward only, because a cause is always in the past. I turned that into
> software. It runs locally. Today.
>
> **[pause]**
>
> And the incumbents can't copy it — their whole architecture *is* the wrong
> assumption. To do this, you rebuild memory from the brain up. I already did.
>
> **[land it]**
>
> Everyone else built a memory that *remembers.* I built the first one that
> *realizes.* Every AI agent that touches production needs it — that's the whole
> market, and it's on fire.
>
> I'm Sam. This is Postdict. The first memory that finds the *cause*, not the
> lookalike. Come find me — I'll prove it on a laptop in thirty seconds.
---
## HOW TO DELIVER IT (this is 80% of the win on a stage)
1. **Memorize three sentences. Improvise the rest.** If you only nail three,
nail these:
- **The wound:** *"The cause turns out to be one line you changed three days ago — and forgot."*
- **The detonation:** *"A root cause never looks like the bug it creates."*
- **The category:** *"Everyone else built a memory that remembers. I built the first one that realizes."*
2. **The detonation line is the whole pitch.** Walk *toward* the audience as you
say it. Then STOP. Say nothing for two full seconds. That silence is you
letting 100 people independently realize you're right. Do not rush it.
3. **Open about THEM, not you.** The first 15 seconds is their pain, not your
product. When heads nod because they've lived it, the room is yours — you
haven't even said what you do yet.
4. **"I built this" is a flex, not a weakness.** Solo means you can ship without
a team — investors fund that. Say "I built," "I ported," "I already did it."
Own it.
5. **Short sentences. Hard stops.** On a stage, a long sentence loses the back
row. Every sentence above is built to be said in one breath. Trust the
periods.
6. **Don't demo on stage — *promise* the demo.** "Come find me, I'll prove it on
a laptop in 30 seconds" is stronger than fumbling a terminal in front of 100
people. It pulls the serious ones to you afterward, one-on-one, where deals
actually start.
7. **End on your name + the one-liner.** "I'm Sam. This is Postdict. The first
memory that finds the cause, not the lookalike." That's what they Google in
the parking lot.
---
## THE 30-SECOND VERSION (if you only get a hallway / elevator)
> "Every AI memory tool searches for what your bug *looks like.* But a root cause
> never looks like the bug it creates — it's a config change from three days ago
> that looks nothing like the crash. So they all miss it. I built the first
> memory that reaches *backward in time* to find the actual cause. Ported it
> straight from a *Nature* paper on how the brain does it. Every agent that
> touches production needs it. I'll show you on a laptop right now."
## THE ONE LINE (if you get five seconds)
> "I built the first AI memory that finds the *cause* of a bug, not the lookalike
> — because a cause never looks like the bug it creates."

2
docs/CONFIGURATION.md
View file

@ -310,7 +310,7 @@ vestige update --sandwich-companion
**Pin to specific version:**
```bash
vestige update --version v2.1.21
vestige update --version v2.2.0
```
**Manage the optional Cognitive Sandwich layer without updating binaries:**

30
docs/VESTIGE_STATE_AND_PLAN.md
View file

@ -15,14 +15,29 @@ For current user-facing release information, use:
## Current Release Shape
Vestige v2.1.21 is the "Agent-Neutral Hardening" release. Its public scope is:
Vestige v2.2.0 is the "Retroactive Salience + Tool Consolidation" release. Its
public scope is:
- Retroactive Salience Backfill (Cai 2024 *Nature*): on a recorded failure,
reach backward in time to promote the causally-upstream root cause. Auto-fires
in the consolidation pass; also exposed as the `backfill` MCP tool and
`vestige backfill` CLI
- MCP tool consolidation: 34 tools collapse to 13 advertised
(`recall`, `smart_ingest`, `memory`, `graph`, `maintain`, `dedup`,
`memory_status`, `suppress`, `backfill`, `codebase`, `intention`,
`source_sync`, `session_start`); the pre-consolidation names remain
dispatchable as hidden back-compat aliases
- `deep_reference`/`recall` retrieval engine upgrades: F32 embeddings (was I8),
Reciprocal Rank Fusion, and claim-vs-memory contradiction checks
It carries forward the local-first hardening baseline from prior releases:
- stdio MCP as the default agent transport, with HTTP MCP opt-in only
- binary-only `vestige update` by default
- delete and purge confirmation parity for destructive memory removal
- portable sync fixes for purge tombstones, UPSERT merge, and vector index
- portable sync handling for purge tombstones, UPSERT merge, and vector index
reloads
- safer release packaging with dashboard freshness checks and checksums
- release packaging with dashboard freshness checks and checksums
- agent-neutral memory instructions for any MCP-compatible client
The release keeps the local-first baseline intact. Heavy model hooks, local
@ -69,7 +84,14 @@ Vestige is organized as:
- `packages/vestige-init`: installer helper
- `docs`: user and integration documentation
## v2.1.21 Implementation Notes
## v2.2.0 Implementation Notes
The advertised MCP surface is 13 tools (see `handle_tools_list` in
`crates/vestige-mcp/src/server.rs`, verified by `test_tools_list_returns_all_tools`).
Consolidation is a facade: the ~22 folded/renamed names still dispatch through
`handle_tools_call` as hidden aliases, so existing client configs keep working.
`backfill` is deliberately advertised as its own tool rather than folded into
`maintain`, because backward causal promotion is a distinct cognitive primitive.
HTTP MCP is disabled unless the user passes `--http`, passes `--http-port`, or
sets `VESTIGE_HTTP_ENABLED=1`. The stdio MCP server remains the portable default

303
docs/adr/0001-pluggable-storage-and-network-access.md
View file

@ -1,303 +0,0 @@
# ADR 0001: Pluggable Storage Backend, Network Access, and Emergent Domains
**Status**: Accepted
**Date**: 2026-04-21
**Related**: [docs/prd/001-getting-centralized-vestige.md](../prd/001-getting-centralized-vestige.md)
---
## Context
Vestige v2.x runs as a per-machine local process: stdio MCP transport, SQLite +
FTS5 + USearch HNSW in `~/.vestige/`, fastembed locally for embeddings. This is
ideal for single-machine single-agent use but blocks three real needs:
- **Multi-machine access** -- same memory brain from laptop, desktop, server
- **Multi-agent access** -- multiple AI clients against one store concurrently
- **Future federation** -- syncing memory between decentralized nodes (MOS /
Threefold grid)
SQLite's single-writer model and lack of a native network protocol make it
unsuitable as a centralized server. PostgreSQL + pgvector collapses our three
storage layers (SQLite, FTS5, USearch) into one engine with MVCC concurrency,
auth, and replication.
Separately, Vestige today has no notion of domain or project scope -- all memories
share one namespace. For a multi-machine brain, users want soft topical
boundaries ("dev", "infra", "home") without manual tenanting. HDBSCAN clustering
on embeddings produces these boundaries from the data itself.
The PRD at `docs/prd/001-getting-centralized-vestige.md` sketches the full design.
This ADR records the architectural decisions and resolves the open questions from
that document.
---
## Decision
Introduce two new trait boundaries, a network transport layer, and a domain
classification module. All four changes ship in parallel phases.
**Trait boundaries:**
1. `MemoryStore` -- single trait covering CRUD, hybrid search, FSRS scheduling,
graph edges, and domains. One big trait, not four.
2. `Embedder` -- separate trait for text-to-vector encoding. Storage never calls
fastembed directly. Callers (cognitive engine locally, HTTP server remotely)
compute embeddings and pass them into the store.
**Backends:**
- `SqliteMemoryStore` -- existing code refactored behind the trait, no behavior
change.
- `PgMemoryStore` -- new, using sqlx + pgvector + tsvector. Selectable at runtime
via `vestige.toml`.
**Network:**
- MCP over Streamable HTTP on the existing Axum server.
- API key auth middleware (blake3-hashed, stored in `api_keys` table).
- Dashboard uses the same API keys for login, then signed session cookies for
subsequent requests.
**Domain classification:**
- HDBSCAN clustering over embeddings to discover domains automatically.
- Soft multi-domain assignment -- raw similarity scores stored per memory, every
domain above a threshold is assigned.
- Conservative drift handling -- propose splits/merges, never auto-apply.
---
## Architecture Overview
### Component Breakdown
1. **`Embedder` trait** (new module `crates/vestige-core/src/embedder/`)
- `async fn embed(&self, text: &str) -> Result<Vec<f32>>`
- `fn model_name(&self) -> &str`
- `fn dimension(&self) -> usize`
- Impls: `FastembedEmbedder` (local ONNX, today), future `JinaEmbedder`,
`OpenAiEmbedder`, etc.
- Stays pluggable forever -- no lock-in to fastembed or to nomic-embed-text.
2. **`MemoryStore` trait** (new module `crates/vestige-core/src/storage/trait.rs`)
- One trait, ~25 methods across CRUD, search, FSRS, graph, domain sections.
- Uses `trait_variant::make` to generate a `Send`-bound variant for
`Arc<dyn MemoryStore>` in Axum/tokio contexts.
- The 29 cognitive modules operate exclusively through this trait. No direct
SQLite or Postgres access from the modules.
3. **`SqliteMemoryStore`** (refactor of existing `crates/vestige-core/src/storage/sqlite.rs`)
- Existing rusqlite + FTS5 + USearch code, wrapped behind the trait.
- Add `domains TEXT[]` equivalent (JSON-encoded array column in SQLite).
- Add `domain_scores` JSON column.
- No behavioral change for current users.
4. **`PgMemoryStore`** (new `crates/vestige-core/src/storage/postgres.rs`)
- `sqlx::PgPool` with compile-time checked queries.
- pgvector HNSW index for vector search, tsvector + GIN for FTS.
- Native array columns for `domains`, JSONB for `domain_scores` and `metadata`.
- Hybrid search via RRF (Reciprocal Rank Fusion) in a single SQL query.
5. **Model registry**
- Per-database table `embedding_model` with `(name, dimension, hash, created_at)`.
- Both backends refuse writes from an embedder whose signature doesn't match
the registered row.
- Model swap = `vestige migrate --reembed --model=<new>`, O(n) cost, explicit.
6. **`DomainClassifier` cognitive module** (new `crates/vestige-core/src/neuroscience/domain_classifier.rs`)
- Owns the HDBSCAN discovery pass (using the `hdbscan` crate).
- Computes soft-assignment scores for every memory against every centroid.
- Stores raw `domain_scores: HashMap<String, f64>` per memory; thresholds into
the `domains` array using `assign_threshold` (default 0.65).
- Runs discovery on demand (`vestige domains discover`) or during dream
consolidation passes.
7. **HTTP MCP transport** (extension of existing Axum server in `crates/vestige-mcp/src/`)
- New route `POST /mcp` for Streamable HTTP JSON-RPC.
- New route `GET /mcp` for SSE (for long-running operations).
- REST API under `/api/v1/` for direct HTTP clients (non-MCP integrations).
- Auth middleware validates `Authorization: Bearer ...` or `X-API-Key`, plus
signed session cookies for dashboard.
8. **Key management** (new `crates/vestige-mcp/src/auth/`)
- `api_keys` table -- blake3-hashed keys, scopes, optional domain filter,
last-used timestamp.
- CLI: `vestige keys create|list|revoke`.
9. **FSRS review event log** (future-proofing for federation)
- New table `review_events` -- append-only `(memory_id, timestamp, rating,
prior_state, new_state)`.
- Current `scheduling` table becomes a materialized view over the event log
(reconstructible from events).
- Phase 5 federation merges event logs, not derived state. Zero cost today,
avoids lock-in tomorrow.
### Data Flow
**Local mode (stdio MCP, unchanged UX):**
```
stdio client -> McpServer -> CognitiveEngine -> FastembedEmbedder -> MemoryStore (SQLite)
```
**Server mode (HTTP MCP, new):**
```
Remote client -> Axum HTTP -> auth middleware -> CognitiveEngine
-> FastembedEmbedder (server-side) -> MemoryStore (Postgres)
```
The cognitive engine is backend-agnostic. The embedder and the store are both
swappable. The 7-stage search pipeline (overfetch -> cross-encoder rerank ->
temporal -> accessibility -> context match -> competition -> spreading activation)
sits *above* the `MemoryStore` trait and works identically against either backend.
### Orthogonality of HDBSCAN and Reranking
HDBSCAN and the cross-encoder reranker solve different problems and both stay:
- **HDBSCAN** discovers domains by clustering embeddings. Runs once per discovery
pass. Produces centroids. Used to *filter* search candidates, not to rank them.
- **Cross-encoder reranker** (Jina Reranker v1 Turbo) scores query-document pairs
at search time. Runs on every search. Produces ranked results.
Domain membership is a filter applied before or during overfetch; reranking runs
on whatever candidate set survives the filter.
---
## Alternatives Considered
| Alternative | Pros | Cons | Why Not |
|-------------|------|------|---------|
| Split into 4 traits (`MemoryStore + SchedulingStore + GraphStore + DomainStore`) | Cleaner interface segregation | Every module holds 4 trait objects, coordinates transactions across them | One trait is fine in Rust; extract sub-traits later if a genuine need appears |
| Embedding computed inside the backend | Simpler call sites for callers | Backend becomes aware of embedding models; can't support remote clients without local fastembed | Keep storage pure; separate `Embedder` trait handles pluggability |
| Unconstrained pgvector `vector` (no dimension) | Flexible for model swaps | HNSW still needs fixed dims at index creation; hides a meaningful migration as "silent" | Fixed dimension per install, explicit `--reembed` migration |
| Dashboard separate auth (cookies only, no keys) | Simpler dashboard UX | Two auth systems to maintain | Shared API keys with session cookie layer on top |
| Auto-tuned `assign_threshold` targeting an unclassified ratio | Adapts to corpus | Hard to debug ("why did this memory change domain?"); magical | Static 0.65 default, config-tunable, dashboard shows `domain_scores` for manual retuning |
| Aggressive drift (auto-reassign memories whose scores drifted) | Always up-to-date domains | Breaks user muscle memory; silent reshuffling | Conservative: always propose, user accepts |
| CRDTs for federation state | Mathematically clean merges | Massive complexity, performance cost, overkill | Defer; design FSRS as event log now so any future sync model works |
---
## Consequences
### Positive
- Single memory brain accessible from every machine.
- Multi-agent concurrent access via Postgres MVCC.
- Natural topical scoping emerges from data, not manual tenants.
- Future embedding model swaps are a config + migration, not a rewrite.
- Federation has a clean on-ramp (event log merge) without committing now.
- The `Embedder` / `MemoryStore` split unlocks other storage backends later
(Redis, Qdrant, Iroh-backed blob store, etc.) with minimal work.
### Negative
- Operating a Postgres instance is more work than managing a SQLite file.
- Users who stay on SQLite gain nothing from this ADR (but lose nothing either).
- Migration (`vestige migrate --from sqlite --to postgres`) is a sensitive
operation for users with months of memories -- needs strong testing.
- HDBSCAN + re-soft-assignment runs in O(n) over all embeddings. At 100k+
memories this starts to matter; manageable but not free.
### Risks
- **Trait abstraction leaks**: a cognitive module might need backend-specific
behavior (e.g., Postgres triggers for tsvector). Mitigation: keep such logic
inside the backend impl; the trait stays pure.
Escalation: if a module genuinely cannot express what it needs through the
trait, the trait grows, not the module bypasses.
- **Embedding model drift**: users on older fastembed versions silently
producing slightly different vectors after a fastembed upgrade. Mitigation:
model hash in the registry, refuse mismatched writes, surface a clear error.
- **Auth misconfiguration**: a user binds to `0.0.0.0` without setting
`auth.enabled = true`. Mitigation: refuse to start with non-localhost bind
and auth disabled. Hard error, not a warning.
- **Re-clustering feedback loop**: dream consolidation proposes re-clusters,
which the user accepts, which changes classifications, which affects future
retrievals, which affect future dreams. Mitigation: cap re-cluster frequency
(every 5th dream by default), require explicit user acceptance of proposals.
- **Cross-domain spreading activation weight (0.5 default)**: arbitrary choice;
could be too aggressive or too lax. Mitigation: config-tunable; instrument
retrieval quality metrics in the dashboard so the user sees impact.
---
## Resolved Decisions (from Q&A)
| # | Question | Resolution |
|---|----------|------------|
| 1 | Trait granularity | Single `MemoryStore` trait |
| 2 | Embedding on insert | Caller provides; separate `Embedder` trait for pluggability |
| 3 | pgvector dimension | Fixed per install, derived from `Embedder::dimension()` at schema init |
| 4 | Federation sync | Defer algorithm; store FSRS reviews as append-only event log now |
| 5 | Dashboard auth | Shared API keys + signed session cookie |
| 6 | HDBSCAN `min_cluster_size` | Default 10; user reruns with `--min-cluster-size N`; no auto-sweep |
| 7 | Domain drift | Conservative -- always propose splits/merges, never auto-apply |
| 8 | Cross-domain spreading activation | Follow with decay factor 0.5 (tunable) |
| 9 | Assignment threshold | Static 0.65 default, config-tunable, raw `domain_scores` stored for introspection |
---
## Implementation Plan
Five phases, each independently shippable.
### Phase 1: Storage trait extraction
- Define `MemoryStore` and `Embedder` traits in `vestige-core`.
- Refactor `SqliteMemoryStore` to implement `MemoryStore`; no behavior change.
- Refactor `FastembedEmbedder` to implement `Embedder`.
- Add `embedding_model` registry table; enforce consistency on write.
- Add `domains TEXT[]`-equivalent and `domain_scores` JSON columns to SQLite
(empty for all existing rows).
- Convert all 29 cognitive modules to operate via the traits.
- **Acceptance**: existing test suite passes unchanged. Zero warnings.
### Phase 2: PostgreSQL backend
- `PgMemoryStore` with sqlx, pgvector, tsvector.
- sqlx migrations (`crates/vestige-core/migrations/postgres/`).
- Backend selection via `vestige.toml` `[storage]` section.
- `vestige migrate --from sqlite --to postgres` command.
- `vestige migrate --reembed` command for model swaps.
- **Acceptance**: full test suite runs green against Postgres with a testcontainer.
### Phase 3: Network access
- Streamable HTTP MCP route on Axum (`POST /mcp`, `GET /mcp` for SSE).
- REST API under `/api/v1/`.
- API key table + blake3 hashing + `vestige keys create|list|revoke`.
- Auth middleware (Bearer, X-API-Key, session cookie).
- Refuse non-localhost bind without auth enabled.
- **Acceptance**: MCP client over HTTP works from a second machine; dashboard
login flow works; unauth requests return 401.
### Phase 4: Emergent domain classification
- `DomainClassifier` module using the `hdbscan` crate.
- `vestige domains discover|list|rename|merge` CLI.
- Automatic soft-assignment pipeline (compute `domain_scores` on ingest, threshold
into `domains`).
- Re-cluster every Nth dream consolidation (default 5); surface proposals in the
dashboard.
- Context signals (git repo, IDE) as soft priors on classification.
- Cross-domain spreading activation with 0.5 decay.
- **Acceptance**: on a corpus of 500+ mixed memories, discover produces sensible
clusters; search scoped to a domain returns tightly relevant results.
### Phase 5: Federation (future, explicitly out of scope for this ADR's
acceptance)
- Node discovery (Mycelium / mDNS).
- Memory sync protocol over append-only review events and LWW-per-UUID for
memory records.
- Explicit follow-up ADR before any code.
---
## Open Questions
None at ADR acceptance time. Follow-up items that are *implementation choices*,
not architectural:
- Precise cross-domain decay weight (start at 0.5, instrument, tune)
- Dashboard histogram of `domain_scores` (UX design detail)
- Whether to gate Postgres behind a Cargo feature flag (`postgres-backend`) or
always compile it in (lean toward feature flag to keep SQLite-only builds small)

545
docs/adr/0002-phase-2-execution.md
View file

@ -1,545 +0,0 @@
# ADR 0002: Phase 2 Execution -- Postgres Backend Integration, Phase 1 Amendment
**Status**: Accepted
**Date**: 2026-05-26
**Related**: [docs/adr/0001-pluggable-storage-and-network-access.md](0001-pluggable-storage-and-network-access.md), [docs/plans/0002-phase-2-postgres-backend.md](../plans/0002-phase-2-postgres-backend.md)
---
## Context
ADR 0001 set the architectural direction: introduce `MemoryStore` and `Embedder`
traits, ship a Postgres backend behind a feature flag, and reach a single shared
memory brain across machines. Phase 1 (storage trait extraction) shipped on
`feat/storage-trait-phase1` (790c0c8). The Phase 2 master plan at
`docs/plans/0002-phase-2-postgres-backend.md` was drafted before Phase 1 was
frozen.
Starting Phase 2 surfaces a small set of execution-level decisions that ADR 0001
did not cover and that the master plan now disagrees with the live code on.
These decisions are too big to silently absorb into a per-step plan and too
small to amend ADR 0001. They live here.
Three concrete realities driving this ADR:
1. **Trait shape mismatch.** Master plan 0002 assumed `trait_variant::make`
produced distinct `MemoryStore` (Send-bound) and `LocalMemoryStore`
(non-Send) variants, and that errors were `StoreError`. Phase 1 froze on
`#[async_trait::async_trait]` with `pub use MemoryStore as LocalMemoryStore`
and an error type called `MemoryStoreError`. The Postgres backend has to
follow Phase 1, not the master plan -- but we should record that explicitly.
2. **`SqliteMemoryStore` is monolithic.**
`crates/vestige-core/src/storage/sqlite.rs` is ~8200 lines. Phase 1 appended
the trait impl block at the bottom of the same file. Adding a similarly
large `postgres.rs` perpetuates the pattern; this is the natural moment to
decide whether the SQLite file gets split.
3. **Constructor surface drift.** Master plan 0002 specifies
`PgMemoryStore::connect(url, max_connections, &dyn Embedder)`. The Phase 1
`SqliteMemoryStore` constructor takes no embedder -- registry consistency
runs through `registered_model()` / `register_model()` on the trait,
invoked by the caller. The two backends should look the same to a caller;
right now they would not.
4. **Multi-tenancy is a one-way door.** The Postgres schema is the place to
reserve user/group/visibility columns *now*, even though Phase 3 is the
phase that wires the auth filter using them. Adding `owner_user_id` and
GIN indexes to a populated, HNSW-indexed `knowledge_nodes` table later is an
expensive online migration; reserving NULL-defaulted columns at schema
creation is ~10 lines of SQL. The same logic applies to per-memory
context capture (codebase, MCP caller, session) -- promoting `codebase`
to a first-class column now keeps the door open for context-aware
sharing rules in Phase 4 without touching `knowledge_nodes`. See D7 and D8.
This ADR is also the umbrella under which Phase 2 sub-plans (`0002a-...`,
`0002b-...`, etc.) sit. The intent is: ADR + sub-plans land as one PR for
review; the implementation lands as a second PR with many commits inside.
---
## Already Decided (carried in by reference)
These are settled by ADR 0001 or by explicit agreement during this session.
Listed here so the discussion frame is clear; not re-litigated below.
- Postgres backend ships behind a `postgres-backend` Cargo feature, default
OFF. Mutually compilable with SQLite. (ADR 0001.)
- Single big `MemoryStore` trait. `PgMemoryStore` implements the same surface
as `SqliteMemoryStore`. (ADR 0001.)
- pgvector HNSW + tsvector + GIN + RRF hybrid search in one SQL statement.
(Master plan 0002, D4-D5.)
- sqlx 0.8 + pgvector 0.4 + compile-time-checked queries + offline `.sqlx/`
cache committed. (Master plan 0002.)
- Two sqlx migration files: `0001_init` (extensions, tables, non-vector
indexes) and `0002_hnsw` (HNSW separated for re-embed drop/recreate).
(Master plan 0002, D4.)
- `vestige migrate --from sqlite --to postgres` and
`vestige migrate --reembed --model=<new>` CLI subcommands. (ADR 0001 +
master plan 0002, D8-D10.)
- PR cadence: PR #1 carries this ADR plus all sub-plans; PR #2 carries the
implementation as many commits.
- Sub-plans use `0002a-`, `0002b-`, ... suffixes off `0002-`.
- `PgMemoryStore::connect` lands as `todo!()` in the skeleton; real body
comes later.
---
## Decisions
### D1. Sunset async_trait across the Phase 1 traits
Phase 1 froze with `#[async_trait::async_trait]` on both the `MemoryStore`
trait (`storage/memory_store.rs:194`) and the `Embedder` trait
(`embedder/mod.rs:27`), plus their SQLite and Fastembed impl blocks. async_trait
boxes every async fn into `Pin<Box<dyn Future + Send>>` -- one heap allocation
per call inside the hottest code path. We are amending Phase 1 to remove
async_trait entirely and replace it with `trait_variant::make`, so each trait
becomes two real generated variants (`MemoryStore` / `LocalMemoryStore`,
`Embedder` / `LocalEmbedder`) with `Send` bounds on the outer variant.
Scope split across three Phase 1 amendment sub-plans:
- **`0001a-trait-rewrite.md`** -- Rewrite `MemoryStore` only. Touches
`storage/memory_store.rs` (trait declaration) and `storage/sqlite.rs`
(impl block attribute). Leaves async_trait in place on the embedder side
so the diff stays focused.
- **`0001b-sqlite-split.md`** -- Pure code motion. Splits the
~8200-line `sqlite.rs` into a `sqlite/` directory. Independent of D1; can
land in either order relative to `0001a`.
- **`0001c-async-trait-sunset.md`** -- Rewrite `Embedder` the same way, then
remove `async-trait = "0.1"` from `crates/vestige-core/Cargo.toml`. Final
amendment commit removes the dependency entirely. After this lands, the
workspace contains zero references to `async_trait`.
All three sub-plans land on the existing `feat/storage-trait-phase1` branch
(790c0c8 has not been opened upstream yet; amend in place, no force-push to a
public PR).
### D2. PgMemoryStore::connect mirrors SqliteMemoryStore::new
```rust
impl PgMemoryStore {
pub async fn connect(url: &str, max_connections: u32) -> MemoryStoreResult<Self>;
pub async fn from_pool(pool: PgPool) -> MemoryStoreResult<Self>;
}
```
No `Embedder` in the constructor. The pgvector-specific
`ALTER TABLE knowledge_nodes ALTER COLUMN embedding TYPE vector($N)` DDL lives
inside the trait method `register_model(&ModelSignature)`. That method is
called by the caller (cognitive engine bootstrap, migrate CLI, tests) after
construction, exactly as it is for `SqliteMemoryStore`.
`MemoryStoreError` gains two variants behind the feature flag (added during
the Postgres impl, not during the Phase 1 amendment):
```rust
#[cfg(feature = "postgres-backend")]
#[error("postgres error: {0}")]
Postgres(#[from] sqlx::Error),
#[cfg(feature = "postgres-backend")]
#[error("postgres migration error: {0}")]
Migrate(#[from] sqlx::migrate::MigrateError),
```
### D3. Split sqlite.rs into a sqlite/ directory as Phase 1 amendment
Pure code motion, no behavioural change. Target layout:
```
crates/vestige-core/src/storage/sqlite/
mod.rs -- SqliteMemoryStore struct, new(), reader/writer locks
crud.rs -- insert/get/update/delete
search.rs -- fts_search, vector_search, hybrid search
scheduling.rs -- FSRS state methods
graph.rs -- edges, neighbors
domain.rs -- domain CRUD, classify stub
registry.rs -- embedding_model table + register_model
portable_sync.rs -- portable archive backend bridge
trait_impl.rs -- impl LocalMemoryStore for SqliteMemoryStore
```
Cognitive-module imports stay on `crate::storage::SqliteMemoryStore` and
related re-exports from `storage/mod.rs`; the split is private to the
module. Each motion commit must keep `cargo test -p vestige-core` green for
bisectability.
This lands in the Phase 1 amendment PR alongside D1 (separate commit, same
branch).
### D4. Postgres backend as a directory from day one
```
crates/vestige-core/src/storage/postgres/
mod.rs -- PgMemoryStore struct, connect, from_pool, trait impl
pool.rs -- PgPool construction from PostgresConfig
migrations.rs -- sqlx::migrate! wrapper
registry.rs -- ensure_registry, ALTER COLUMN TYPE vector(N)
search.rs -- RRF query + row mapping
migrate_cli.rs -- SQLite -> Postgres streaming copy
reembed.rs -- O(n) re-encode + HNSW rebuild
```
D1+D2 of the master plan land first as a skeleton in `mod.rs` with `todo!()`
bodies; later sub-plans fill in the other files.
### D5. Sub-plan layout: two phases worth of sub-plans
Phase 1 amendment sub-plans (under `docs/plans/`):
- `0001a-trait-rewrite.md` -- MemoryStore async_trait -> trait_variant, call-site audit
- `0001b-sqlite-split.md` -- sqlite.rs -> sqlite/ directory, commit-by-commit
- `0001c-async-trait-sunset.md` -- Embedder rewrite + drop async-trait dep from Cargo.toml
Phase 2 sub-plans (under `docs/plans/`):
- `0002a-skeleton-and-feature-gate.md` -- master plan D1 + D2 (todo!() bodies)
- `0002b-pool-and-config.md` -- master plan D3 + D7
- `0002c-migrations.md` -- master plan D4
- `0002d-store-impl-bodies.md` -- master plan D2 real bodies + D6 registry
- `0002e-hybrid-search.md` -- master plan D5
- `0002f-migrate-cli.md` -- master plan D8 + D10
- `0002g-reembed.md` -- master plan D9
- `0002h-testing-and-benches.md` -- master plan D14 + D15
- `0002i-runbook.md` -- master plan D16
Each sub-plan is a self-contained brief sized to fit one focused
implementation session (handed to Claude Code as a `/goal` instruction
without requiring the agent to load the master plan).
### D6. SQLite split does not get its own ADR
The split is pure code motion; no public types, behaviour, or paths change.
`0001b-sqlite-split.md` is enough.
### D7. Multi-tenancy schema reservation (L1-L3)
Phase 2 reserves the columns and tables needed for future per-user / per-group
visibility, so Phase 3 (auth) does not require a column-add migration over a
populated, HNSW-indexed `knowledge_nodes` table. Single-user behaviour is unchanged
in both backends.
New tables in `0001_init.up.sql`:
```sql
CREATE TABLE users (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
handle TEXT NOT NULL UNIQUE,
display_name TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
metadata JSONB NOT NULL DEFAULT '{}'::jsonb
);
CREATE TABLE groups (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
handle TEXT NOT NULL UNIQUE,
display_name TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
metadata JSONB NOT NULL DEFAULT '{}'::jsonb
);
CREATE TABLE group_memberships (
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
group_id UUID NOT NULL REFERENCES groups(id) ON DELETE CASCADE,
role TEXT NOT NULL DEFAULT 'member', -- 'member' | 'admin'
joined_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (user_id, group_id)
);
INSERT INTO users (id, handle, display_name)
VALUES ('00000000-0000-0000-0000-000000000001', 'local', 'Local User');
```
New columns on `knowledge_nodes`:
```sql
owner_user_id UUID NOT NULL DEFAULT '00000000-0000-0000-0000-000000000001'
REFERENCES users(id),
visibility TEXT NOT NULL DEFAULT 'private', -- 'private' | 'group' | 'public'
shared_with_groups UUID[] NOT NULL DEFAULT '{}'
CREATE INDEX idx_knowledge_nodes_owner ON knowledge_nodes (owner_user_id);
CREATE INDEX idx_knowledge_nodes_shared_groups ON knowledge_nodes USING GIN (shared_with_groups);
```
Phase 3 visibility filter (declared here for reference; implemented in Phase 3):
```sql
WHERE
(visibility = 'private' AND owner_user_id = $me)
OR (visibility = 'group'
AND (owner_user_id = $me OR shared_with_groups && $my_group_ids))
OR visibility = 'public'
```
Why tri-state enum and not just `shared_with_groups[] + is_public`: the
explicit `visibility` field documents intent at the row level. A `'private'`
row with a non-empty `shared_with_groups` is detectable inconsistency
(a CHECK constraint can enforce it later) rather than silent data.
SQLite parity: same tables and columns with identical defaults.
`shared_with_groups` is a JSON `'[]'` text encoding (no array type).
Single-user mode never changes any of these values; the trait surface ignores
the visibility filter for SQLite because there is exactly one user.
Sharing automation (matching by domain, tag, repo, MCP caller, ...) is
explicitly **not** in Phase 2. See D8 for context capture, and the Follow-ups
section for the Phase 4 `sharing_rules` design sketch.
RLS policies are not declared in Phase 2. Phase 3 decides whether to add
RLS as defense-in-depth on top of the app-layer filter.
### D8. Context-aware ingest
Every memory carries its ingest context, so future automation (sharing rules,
domain scoping, audit) can match on it without a schema migration. Most of
this is already happening in the Phase 1 ingest pipeline; D8 promotes it to
ADR-level commitment so Phase 2 cannot drop it on the way to Postgres.
Context dimensions and where they live:
- **`codebase`** -- promoted to a first-class indexed column on `knowledge_nodes`.
High-frequency query path (`SELECT ... WHERE codebase = 'vestige'`) for
both human exploration and Phase 4 HDBSCAN scoping. Direct B-tree index
beats JSONB extraction.
```sql
codebase TEXT, -- nullable; populated from ingest context
CREATE INDEX idx_knowledge_nodes_codebase ON knowledge_nodes (codebase) WHERE codebase IS NOT NULL;
```
`MemoryRecord` gains `pub codebase: Option<String>`.
- **`mcp_client_id`** -- which MCP caller created this. Persistent identity
once Phase 3 API keys exist. Lives in `metadata.mcp_client_id` (JSONB).
Not query-hot enough to deserve a column.
- **`session_id`** -- ephemeral; identifies the calling session for runtime
override scoping. Lives in `metadata.session_id` (JSONB). Sessions die
fast; storing them as rows or indexed columns is waste.
- **`file` / `topics`** -- existing optional context already accepted by the
ingest pipeline. Stay in metadata JSONB.
Phase 2's job for D8 is operational, not architectural: audit the ingest
path from MCP request to row write to ensure none of these fields gets
dropped when crossing the SQLite -> Postgres backend boundary.
---
## PR Cadence
Two work streams, three PRs total:
1. **PR A: Phase 1 amendment**
- Branch: `feat/storage-trait-phase1` (existing, amended in place)
- Commits: MemoryStore trait rewrite (0001a) + sqlite split (0001b, multiple
motion commits) + Embedder rewrite & async-trait dep removal (0001c).
- Sub-plans `0001a-`, `0001b-`, `0001c-` are committed on this branch.
2. **PR B: ADR 0002 + Phase 2 sub-plans (this document + the 9 sub-plans)**
- New branch off PR A's tip once that is reviewed.
- No code; docs only.
3. **PR C: Phase 2 implementation**
- New branch off PR B's tip.
- One PR with many commits clustered by sub-plan.
PR B is the "let's discuss execution before writing code" gate. PR C is the
"now we write code" gate. If PR A is itself sizable enough that it needs the
amendments reviewed in stages, the three sub-plans (`0001a`, `0001b`, `0001c`)
can split into separate PRs; that's a tactical call at PR time.
---
## Architecture Overview
Final layout after the Phase 1 amendment (PR A) and Phase 2 implementation
(PR C):
```
crates/vestige-core/src/storage/
mod.rs -- re-exports, Storage alias for BC
memory_store.rs -- trait_variant-generated MemoryStore + LocalMemoryStore, types, error
migrations.rs -- SQLite migration registry (Phase 1, unchanged)
portable.rs -- portable archive format (Phase 1, unchanged)
sqlite/ -- was sqlite.rs (D3, Phase 1 amendment)
mod.rs -- SqliteMemoryStore struct, new(), reader/writer locks
crud.rs -- insert/get/update/delete
search.rs -- fts/vector/hybrid
scheduling.rs -- FSRS state
graph.rs -- edges, neighbors
domain.rs -- domain CRUD, classify stub
registry.rs -- embedding_model table + register_model
portable_sync.rs -- portable backend bridge
trait_impl.rs -- impl LocalMemoryStore for SqliteMemoryStore
postgres/ -- D4, Phase 2
mod.rs -- PgMemoryStore struct, connect, from_pool, trait impl
pool.rs -- PgPool construction from config
migrations.rs -- sqlx::migrate! wrapper
registry.rs -- register_model body, ALTER COLUMN TYPE vector(N)
search.rs -- RRF query + row mapping
migrate_cli.rs -- SQLite -> Postgres streaming copy
reembed.rs -- O(n) re-encode + HNSW rebuild
crates/vestige-core/migrations/
sqlite/ -- Phase 1, with V15 migration for D7+D8 columns/tables
postgres/ -- Phase 2
0001_init.up.sql -- includes D7 tables + columns, D8 codebase column
0001_init.down.sql
0002_hnsw.up.sql
0002_hnsw.down.sql
```
Tables in the Postgres schema after migration 0001:
| Table | Purpose | Phase that populates |
|-------|---------|----------------------|
| `embedding_model` | One-row registry of name/dim/hash | Phase 2 (first connect) |
| `knowledge_nodes` | Core records + owner/visibility/codebase | Phase 2 ingest; Phase 4 fills `domains` |
| `scheduling` | FSRS state | Phase 2 |
| `edges` | Spreading activation graph | Phase 2 |
| `review_events` | Append-only FSRS review log | Phase 2; Phase 5 federation reads |
| `domains` | Phase 4 cluster centroids | Phase 4 |
| `users` | L1 identities (D7) | Phase 3 |
| `groups` | L3 groups (D7) | Phase 3 |
| `group_memberships` | L3 user-group links (D7) | Phase 3 |
`sharing_rules` (Phase 4) and `api_keys` (Phase 3) are added later by their
own migrations.
---
## Alternatives Considered
| Alternative | Why not |
|-------------|---------|
| Keep async_trait on the Phase 1 trait | One heap allocation per trait call inside the hottest code path in Vestige. Boxing every future also obscures the actual return type, which makes lifetimes and Send-ness harder to reason about. The Phase 1 PR is not opened upstream yet, so amending is free. |
| Take `&dyn Embedder` into `connect` | Couples constructor to embedder; breaks ADR 0001's separation; can't be used by callers that don't have an embedder yet (tests, migrate CLI). |
| Defer SQLite split | Postgres lands alongside an 8K-line peer; the pattern compounds; future readers see "backends are huge here". |
| Single `postgres.rs` | Master plan calls out 7 sub-files; we know it's getting split; doing it twice is waste. |
| Per-deliverable sub-plans (16 docs) | Review fatigue; many sub-plans would be 3-5 lines of Cargo or one migration each. Logical groups cluster naturally with PR commits. |
| One rolling sub-plan with checkboxes | Moving target; doesn't serve as a `/goal` brief for a fresh Claude Code session. |
| Separate ADR for the SQLite split | Pure code motion with no public-surface change; doesn't constrain future decisions. ADRs are for decisions that bind. |
| Punt multi-tenancy schema entirely to Phase 3 | Adding `owner_user_id` and indexes to a populated, HNSW-indexed `knowledge_nodes` table later is an expensive online migration. Reserving NULL-defaulted columns now is ~10 lines of SQL. |
| `shared_with_groups[] + is_public` instead of tri-state visibility enum | More compact but `visibility = 'private'` documents intent at the row level; a CHECK constraint can later enforce array/enum consistency. Two columns conveying one fact is fine when both are referenced often. |
| Add `shared_with_users[]` for direct user-to-user sharing | A "group of one" subsumes it without an extra column and GIN index. Phase 3 CLI can auto-create singleton groups if a user requests direct shares. |
| Bake per-domain or per-tag sharing defaults into Phase 2 schema | Sharing automation needs real usage data before committing to fuzzy (domain centroids) vs crisp (tags) vs context (codebase / MCP caller). Phase 4 designs a generic `sharing_rules` table that matches on any context dimension; deferring costs nothing because rules live in a new table, not new columns. |
| `codebase` stays in JSONB metadata | High-frequency query path (HDBSCAN scoping, codebase-wide searches, future `sharing_rules` match). B-tree on a real column beats GIN on a JSONB key for this access pattern. Cost is one nullable TEXT column. |
---
## Consequences
### Positive
- Phase 1 trait stops boxing futures on every call. Lifetimes and Send-ness
become inspectable instead of hidden inside an `async_trait` macro expansion.
- `connect` stays backend-agnostic; tests and CLI tools stand up either backend
without an `Embedder` in scope.
- Cognitive module imports never change paths -- the SQLite split is private
to `storage/sqlite/`, public re-exports through `storage/mod.rs` unchanged.
- Postgres backend lands already-modular; future SQL changes touch one of
seven small files, not one of eight thousand lines.
- Phase 2 master plan stays archival; ADR 0002 + sub-plans are the live source
of truth for execution.
- Multi-tenancy columns reserved now means Phase 3 auth is purely additive --
no online migration over a populated, HNSW-indexed `knowledge_nodes` table.
- Context-aware ingest (D8) keeps the door open for repo / session /
MCP-caller-scoped sharing rules in Phase 4 without changing `knowledge_nodes`.
### Negative
- The Phase 1 amendment expands a "finished" branch. It is a real cost: the
trait rewrite touches every cognitive module that holds a store handle.
- SQLite split is a pure-motion diff. Annoying to review even when safe.
- Three PRs (amendment, ADR+plans, implementation) instead of one or two.
Discipline tax in exchange for reviewability.
- Multi-tenancy reservation adds three never-queried tables and three
always-default columns to the SQLite schema. Real but small storage cost in
single-user mode (a single bootstrap row + empty tables + NULL/empty
defaults per memory).
### Risks
- **Trait rewrite breaks a cognitive module's Send-ness expectation.**
Mitigation: `cargo test --workspace` runs after each call-site edit;
trait_variant-generated `MemoryStore` is the Send variant and matches the
current `Arc<dyn ...>` usage everywhere except thread-local impls (none
exist today).
- **SQLite motion commit introduces a silent semantic change.** Mitigation:
each commit keeps `cargo test -p vestige-core` green; reviewer can bisect.
- **Sub-plan boundaries don't match how implementation wants to commit.**
Mitigation: sub-plans are advisory; the implementation PR clusters commits
however it ends up needing to.
- **Reserved columns get used in Phase 3 in a way that mismatches Phase 2
defaults.** Mitigation: Phase 3 owns the auth filter; Phase 2 defaults
(`owner_user_id = local`, `visibility = 'private'`) are intentionally the
"no access for anyone but the owner" worst-case; widening at Phase 3 is
safe, narrowing would be the dangerous direction.
- **Memory: PR A amendment invalidates the locally-deployed Phase 1 binary's
ABI.** Not a real risk -- the trait change is purely source-level Rust; the
on-disk DB schema is unchanged. The rebuilt binary slots in over the
current one without DB migration.
---
## Resolved Decisions
| # | Question | Resolution |
|---|----------|------------|
| Q1 | Phase 1 trait shape | Rewrite with trait_variant::make. Amend Phase 1 PR. |
| Q2 | PgMemoryStore::connect signature | Mirror SqliteMemoryStore::new; no Embedder. register_model does the pgvector typmod stamp. |
| Q3 | Split sqlite.rs | Yes, as Phase 1 amendment. sqlite.rs -> sqlite/ directory; pure code motion. |
| Q4 | Postgres module layout | Directory from day one. |
| Q5 | Sub-plan granularity | Logical groups, ~9 docs for Phase 2 plus 2 for the Phase 1 amendment. |
| Q6 | ADR for SQLite split | No. Sub-plan `0001b-sqlite-split.md` is sufficient. |
| Q7 | Multi-tenancy schema | Reserve users / groups / group_memberships tables and owner_user_id / visibility / shared_with_groups columns on knowledge_nodes in Phase 2. Single-user defaults; Phase 3 fills in real values. |
| Q8 | Visibility encoding | Tri-state enum `'private' \| 'group' \| 'public'` plus `shared_with_groups[]`. No `shared_with_users[]`; no RLS in Phase 2. |
| Q9 | Sharing automation grain | Per-memory only in Phase 2. Phase 4 ships a generic `sharing_rules` table matching on codebase / tag / node_type / mcp_client_id. |
| Q10 | Context capture on ingest | `codebase` promoted to a first-class indexed column; `mcp_client_id` and `session_id` stay in metadata JSONB. |
---
## Follow-ups
- Phase 1 amendment sub-plans drafted: `0001a-trait-rewrite.md`,
`0001b-sqlite-split.md`, `0001c-async-trait-sunset.md`. Ready to execute on
`feat/storage-trait-phase1`.
- Phase 2 sub-plans drafted: `0002a-` through `0002i-` against the accepted
decisions above. Ready to execute on a new branch off PR A's tip.
- Decide branch placement for this ADR before it gets committed -- it cannot
live on `feat/storage-trait-phase1` (that branch is now PR A's code-only
amendment branch). Likely a new branch off PR A's tip for PR B (docs only).
- Validate local Postgres dev cluster before PR C work begins. Recipe at
`docs/plans/local-dev-postgres-setup.md` is correct but needs to be applied
on this machine (delandtj-home): cluster is not initdb'd, pgvector is not
installed. Containerized `pgvector/pgvector:pg18` is a viable alternative
if pgvector packaging is friction. See open discussion thread.
### Phase 4 sketch: `sharing_rules` and the precedence chain
Recorded here so the Phase 4 author does not have to rediscover the design.
Phase 2 does **not** implement any of this; it only ensures the schema and
ingest context capture make this possible without a `knowledge_nodes` migration.
```sql
-- Phase 4 migration (not Phase 2)
CREATE TABLE sharing_rules (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
owner_user_id UUID NOT NULL REFERENCES users(id),
-- Match: any subset; all set fields must match conjunctively
match_codebase TEXT,
match_tag TEXT,
match_node_type TEXT,
match_api_key_id UUID REFERENCES api_keys(id), -- MCP caller identity
-- Policy
visibility TEXT NOT NULL,
shared_with_groups UUID[] NOT NULL DEFAULT '{}',
-- Conflict resolution
priority INTEGER NOT NULL DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
```
Precedence on ingest, first match wins:
1. Caller-explicit visibility in the MCP request
2. Active session override held by the MCP server (per-session, in-memory,
not persisted; matched by `session_id`)
3. Highest-priority `sharing_rules` row whose match fields all hold
4. User's `default_visibility` (typically `'private'`)
Per-session overrides do not persist; storing ephemeral session IDs as DB
rows is waste. Per-codebase / per-MCP-caller rules do persist as
`sharing_rules` rows.

395
docs/launch/blog-post.md
View file

@ -1,395 +0,0 @@
# Building a Cognitive Memory System with FSRS-6 and Three.js -- What 130 Years of Neuroscience Taught Us About AI Memory
Your AI assistant does not remember anything.
Every conversation starts from zero. You explain your project structure, your preferences, the bug you fixed last Tuesday, the architectural decision you made last month. The context window is a goldfish bowl -- 200K tokens of short-term memory, then nothing. RAG systems bolt on a vector database and call it "memory," but what they actually build is a search engine. Search is not memory. Memory is a living system that decays, strengthens, connects, and dreams.
Vestige is an open-source Rust MCP server that gives AI agents persistent memory modeled on real neuroscience. Not metaphorical neuroscience. Actual published algorithms from Ebbinghaus (1885), Collins & Loftus (1975), Bjork & Bjork (1992), Frey & Morris (1997), and the FSRS-6 spaced repetition scheduler trained on 700 million Anki reviews.
77,840+ lines of Rust. 29 cognitive modules. 734 tests. Single binary deployment with an embedded SvelteKit dashboard. AGPL-3.0 licensed.
Here is how we built it.
---
## The Problem: Session Boundaries are Amnesia
Every AI conversation today has the same failure mode. The model is stateless. Context windows are large but finite, and they reset between sessions. The industry's answer has been Retrieval-Augmented Generation -- embed documents, stuff them into the prompt, let the model figure it out.
RAG works for document Q&A. It does not work for memory, and here is why:
1. **No forgetting curve.** Every chunk in a vector database has equal weight forever. A configuration snippet from six months ago has the same retrieval priority as the bug fix from yesterday.
2. **No consolidation.** Memories are never merged, connected, or synthesized. You get back isolated chunks, not understanding.
3. **No retroactive importance.** If you flag something as important today, RAG cannot go back and strengthen the memories from last week that suddenly matter.
4. **No surprise detection.** Every insert is treated the same. Duplicate information bloats the database. Contradictions pile up silently.
We wanted something different. We wanted memory that behaves like a brain -- where memories compete, strengthen through use, decay through neglect, and form connections during idle periods.
---
## The Solution: Treat Memory Like a Brain, Not a Database
Vestige implements a cognitive architecture with three core principles:
1. **Memories have a lifecycle.** They are born, they strengthen through retrieval, they decay over time, they can be revived, and they eventually fade below the retrieval threshold. This is FSRS-6.
2. **Storage is gated by novelty.** Not everything deserves to be remembered. Prediction Error Gating compares new information against existing memories and decides whether to create, update, merge, or supersede. This is the hippocampal bouncer.
3. **Retrieval changes memory.** Every search strengthens the memories it finds (the Testing Effect) and weakens competitors (retrieval-induced forgetting). Memory is not a read-only operation.
### Architecture Overview
```
┌─────────────────────────────────────────────────────┐
│ AI Agent (Claude, GPT, etc.) │
│ ↕ JSON-RPC over stdio (MCP protocol) │
├─────────────────────────────────────────────────────┤
│ vestige-mcp 19 MCP tools │
│ ├── Axum HTTP server (dashboard + WebSocket) │
│ ├── CognitiveEngine (29 stateful modules) │
│ └── Tool handlers (one file per tool) │
├─────────────────────────────────────────────────────┤
│ vestige-core cognitive algorithms │
│ ├── fsrs/ FSRS-6 spaced repetition │
│ ├── neuroscience/ 10 modules (STC, spreading │
│ │ activation, hippocampal │
│ │ index, importance signals...) │
│ ├── search/ hybrid, HyDE, reranker, │
│ │ keyword, vector, temporal │
│ ├── advanced/ 11 modules (dreams, PE gate, │
│ │ chains, compression, │
│ │ cross-project learning...) │
│ ├── embeddings/ fastembed (Nomic v1.5, 768d) │
│ └── storage/ SQLite + FTS5 + USearch HNSW │
└─────────────────────────────────────────────────────┘
```
The entire system compiles to a single binary. The SvelteKit dashboard is embedded at compile time using Rust's `include_dir!` macro. No external services. No cloud dependencies. Your memories live on your machine.
---
## Deep Dive: FSRS-6 -- A Power Law Forgetting Curve
The Free Spaced Repetition Scheduler (FSRS-6) is the mathematical backbone of Vestige. Where traditional systems store memories with static priority scores, every memory in Vestige has two dynamic properties: **stability** (how deeply encoded it is) and **difficulty** (how hard it is to retain). These evolve over time according to a 21-parameter model trained on 700 million real-world Anki reviews.
The core formula is the power forgetting curve:
```
R(t, S) = (1 + factor * t / S) ^ (-w20)
where factor = 0.9 ^ (-1 / w20) - 1
```
- `R` is retrievability -- the probability you can recall this memory right now
- `t` is elapsed time since last access (in days)
- `S` is stability -- the number of days for R to drop to 90%
- `w20` is the personalizable decay parameter (default 0.1542)
Why power law instead of exponential? Because Ebbinghaus was right in 1885, and the data from 700 million reviews confirms it: human forgetting follows a power curve, not an exponential one. Power law decay has a heavier tail -- memories hang around longer than exponential models predict, which matches real behavior.
Here is the actual Rust implementation:
```rust
pub fn retrievability_with_decay(stability: f64, elapsed_days: f64, w20: f64) -> f64 {
if stability <= 0.0 { return 0.0; }
if elapsed_days <= 0.0 { return 1.0; }
let factor = 0.9_f64.powf(-1.0 / w20) - 1.0;
let r = (1.0 + factor * elapsed_days / stability).powf(-w20);
r.clamp(0.0, 1.0)
}
```
FSRS-6 also introduced three new parameters (w17, w18, w19) for **same-day reviews** -- a gap in earlier versions that caused instability when memories were accessed multiple times within 24 hours. In an AI agent context where the same memory might be retrieved dozens of times in one session, this matters enormously.
Each memory exists in one of four states based on its accessibility score:
| State | Accessibility | Behavior |
|-------|--------------|----------|
| **Active** | >= 70% | Immediately retrievable, surfaces in searches |
| **Dormant** | 40-70% | Retrievable with effort, lower search priority |
| **Silent** | 10-40% | Rarely surfaces, needs direct access to revive |
| **Unavailable** | < 10% | Below retrieval threshold, candidate for GC |
Memories are never hard-deleted. They fade. And any access -- even a search that returns them as a secondary result -- strengthens them back toward Active.
---
## Deep Dive: Prediction Error Gating -- The Hippocampal Bouncer
The brain does not store everything it perceives. The hippocampus acts as a novelty filter, comparing incoming information against existing memories and only consolidating what is genuinely new. This is prediction error: the gap between what you expected and what you got.
Vestige implements this as `PredictionErrorGate`. When `smart_ingest` is called with new content:
1. The content is embedded into a 768-dimensional vector (Nomic Embed v1.5)
2. Existing memories are searched for candidates above a similarity threshold
3. Cosine similarity determines the prediction error: `PE = 1.0 - similarity`
4. A decision is made:
```rust
pub enum GateDecision {
Create { reason, prediction_error, related_memory_ids },
Update { target_id, similarity, update_type, prediction_error },
Supersede { old_memory_id, similarity, supersede_reason, prediction_error },
Merge { memory_ids, avg_similarity, strategy },
}
```
The thresholds:
| Similarity | Decision | Rationale |
|-----------|----------|-----------|
| > 0.92 | **Reinforce** | Near-identical content. Strengthen existing memory. |
| > 0.75 | **Update/Merge** | Related content. Merge information into existing memory. |
| 0.70-0.75 + contradiction detected | **Supersede** | Correction. New content replaces outdated memory. |
| < 0.75 | **Create** | Novel content. Store as new memory. |
Contradiction detection uses heuristic NLP -- looking for negation patterns ("don't" vs. "do", "avoid" vs. "use") and correction phrases ("actually", "the right way", "should be"). This catches the common case where a user corrects earlier advice.
The result: you can call `smart_ingest` aggressively without worrying about duplicates. The gate handles deduplication, merging, and conflict resolution automatically. The cost of a false positive (saving something redundant) is near zero because the gate will catch it. The cost of a false negative (losing knowledge) is permanent.
---
## Deep Dive: HyDE Search -- Query Expansion Without an LLM
Hypothetical Document Embeddings (HyDE) is a technique from Gao et al. (2022) where you use an LLM to generate a hypothetical answer to a query, embed that hypothetical answer, and use it for vector search. The intuition: a hypothetical answer is closer in embedding space to the real answer than the raw question is.
Full HyDE requires an LLM call at search time. That is too slow for a local-first system with sub-50ms search targets. Vestige implements a zero-latency approximation:
1. **Intent classification.** The raw query is classified into one of six intents: Definition, HowTo, Reasoning, Temporal, Lookup, or Technical.
2. **Template expansion.** Based on the intent, 3-5 variant queries are generated:
```rust
QueryIntent::Definition => {
variants.push(format!("{clean} is a concept that involves"));
variants.push(format!("The definition of {clean} in the context of"));
variants.push(format!("{clean} refers to a type of"));
}
```
3. **Centroid embedding.** All variants are embedded, and the centroid (average) of the embedding vectors is computed and L2-normalized.
4. **Broadened search.** The centroid embedding captures a wider semantic space than any single query, improving recall for conceptual and question-style queries.
This gives approximately 60% of full HyDE quality improvement with zero latency overhead. The embedding model (Nomic v1.5 running locally via fastembed) generates all variant embeddings in a single batch.
The search pipeline then runs seven stages:
1. **Overfetch** -- Pull 3x results from hybrid search (BM25 keyword + semantic vector)
2. **Rerank** -- Re-score by relevance using a cross-encoder-style reranker
3. **Temporal boost** -- Recent memories get a recency bonus
4. **Accessibility filter** -- FSRS-6 retention threshold gates results (Ebbinghaus curve)
5. **Context match** -- Tulving's encoding specificity (1973): match current context to encoding context
6. **Competition** -- Anderson's retrieval-induced forgetting (1994): winners strengthen, competitors weaken
7. **Spreading activation** -- Collins & Loftus (1975): activate related memories as a side effect
That last stage is the critical differentiator. Every search does not just return results -- it reshapes the memory landscape.
---
## Deep Dive: Synaptic Tagging and Capture -- Retroactive Importance
This is the feature that no other AI memory system has.
In 1997, Frey and Morris published a landmark paper in Nature describing Synaptic Tagging and Capture (STC). The finding: weak stimulation creates a temporary "synaptic tag" at a synapse. If a strong stimulation occurs within a temporal window (up to 9 hours), Plasticity-Related Products (PRPs) are produced that can be "captured" by the tagged synapses, consolidating them to long-term storage.
Translation for AI: **memories can become important retroactively.**
You have a conversation with a coworker about their vacation plans. Trivial. Three hours later, you learn they are leaving the company. Suddenly that vacation conversation is important context. In a traditional memory system, the vacation memory has already been classified as low-priority and buried. With STC, the "leaving the company" event triggers a backward sweep that captures and promotes the vacation conversation.
Vestige implements this with a 9-hour backward window and a 2-hour forward window:
```rust
const DEFAULT_BACKWARD_HOURS: f64 = 9.0;
const DEFAULT_FORWARD_HOURS: f64 = 2.0;
```
When an importance event occurs (user explicitly flags something, a novelty spike is detected, or repeated access patterns emerge), the STC system sweeps for tagged memories within the capture window. Capture probability decays with temporal distance using one of four configurable decay functions (exponential, linear, power law, or logarithmic).
Different event types have different capture characteristics:
| Event Type | Base Strength | Capture Radius | Use Case |
|-----------|--------------|----------------|----------|
| UserFlag | 1.0 | 1.0x | "Remember this" |
| NoveltySpike | 0.9 | 0.7x (narrow) | High prediction error |
| EmotionalContent | 0.8 | 1.5x (wide) | Sentiment detection |
| RepeatedAccess | 0.75 | 1.2x | Pattern of retrieval |
Captured memories are grouped into **importance clusters** -- temporal neighborhoods of memories that collectively provide context around a significant moment. This models how biological memory works: you do not remember isolated facts, you remember episodes.
---
## Deep Dive: Memory Dreaming -- Offline Consolidation
During sleep, the hippocampus replays recent experiences and transfers consolidated memories to the neocortex. This process discovers hidden connections between memories, strengthens important patterns, and prunes weak connections.
Vestige simulates this with a 5-stage dream cycle:
```
Stage 1 - Replay: Replay recent memories in chronological order
Stage 2 - Cross-reference: Compare all memory pairs for hidden connections
Stage 3 - Strengthen: Reinforce connections that co-activate
Stage 4 - Prune: Decay weak connections, remove below threshold
Stage 5 - Transfer: Identify memories ready for semantic storage
```
The dreaming system maintains a `ConnectionGraph` -- a weighted bidirectional graph where edges represent discovered relationships between memories. Edges have strength (0.0 to 2.0) and decay over time (factor 0.95 per consolidation cycle). Connections below 0.1 strength are pruned.
During Phase 2, the system evaluates memory pairs and discovers connections via multiple signals:
```rust
pub enum DiscoveredConnectionType {
Semantic, // High embedding similarity (> 0.8)
SharedConcept, // 2+ shared tags
Temporal, // Created within 24 hours + similarity > 0.6
Complementary, // Moderate similarity, different angles
CausalChain, // Cause-effect relationship detected
}
```
Phase 3 generates synthesized insights -- new knowledge that emerges from combining existing memories:
```rust
pub enum InsightType {
HiddenConnection, // "X and Y are related in ways you didn't notice"
RecurringPattern, // "You keep encountering this theme"
Generalization, // "These specific cases suggest a general rule"
Contradiction, // "These two memories conflict"
KnowledgeGap, // "You know X and Z but not Y"
TemporalTrend, // "This topic has evolved over the past month"
Synthesis, // "Combining A + B + C yields new understanding"
}
```
Dreams are triggered automatically: at session start if more than 24 hours have passed since the last dream, or after every 50 memory saves. The consolidation scheduler also monitors activity patterns and runs during detected idle periods (30+ minutes of inactivity).
---
## The Dashboard: Three.js Makes Memory Visible
Memory is invisible by default. You cannot debug what you cannot see. Vestige includes an embedded dashboard that renders the memory graph as a 3D force-directed visualization powered by Three.js with WebGL bloom post-processing.
Every memory is a glowing sphere. Size maps to retention strength. Color maps to node type (fact, concept, decision, pattern, event). Opacity fades as memories decay. Edges represent discovered connections, with opacity proportional to connection weight.
The visualization is event-driven via WebSocket. The Axum HTTP server runs alongside the MCP stdio transport, broadcasting `VestigeEvent` variants to all connected dashboard clients:
```rust
pub enum VestigeEvent {
MemoryCreated { id, content_preview, node_type, tags, timestamp },
SearchPerformed { query, result_count, result_ids, duration_ms, timestamp },
DreamStarted { memory_count, timestamp },
DreamProgress { phase, memory_id, progress_pct, timestamp },
ConnectionDiscovered { source_id, target_id, connection_type, weight, timestamp },
RetentionDecayed { id, old_retention, new_retention, timestamp },
Heartbeat { uptime_secs, memory_count, avg_retention, timestamp },
// ... 13 event types total
}
```
On the frontend, each event type triggers a distinct visual effect:
- **MemoryCreated:** Particle spawn burst (60 particles expanding outward) + expanding shockwave ring
- **SearchPerformed:** Blue pulse ripple across all nodes
- **DreamStarted:** Purple wash, bloom intensity increases to 1.5, rotation slows
- **DreamProgress:** Individual memories light up as they are "replayed"
- **ConnectionDiscovered:** Golden flash line between two nodes
- **RetentionDecayed:** Red pulse on the decaying node
- **ConsolidationCompleted:** Golden shimmer across all nodes
The force-directed layout uses a Fibonacci sphere distribution for initial positions with repulsion-attraction dynamics: nodes repel each other (Coulomb's law), edges attract connected nodes (spring force), and a centering force prevents drift. The simulation runs for 300 frames then settles.
The entire dashboard ships inside the Vestige binary. No separate frontend deployment. No CDN. `include_dir!` embeds the SvelteKit build output at compile time, and Axum serves it with proper MIME types and cache headers:
```rust
static DASHBOARD_DIR: Dir<'_> =
include_dir!("$CARGO_MANIFEST_DIR/../../apps/dashboard/build");
```
---
## Architecture: Why Rust, SQLite, and Local Embeddings
### Rust
Memory is infrastructure. It runs on every interaction, on every search, on every save. Latency matters. We need sub-50ms search over thousands of memories, with embedding generation, FSRS calculations, and seven-stage pipeline execution. Rust gives us zero-cost abstractions, fearless concurrency (the `CognitiveEngine` is `Arc<Mutex<CognitiveEngine>>` shared across async handlers), and compile-time guarantees that the 29 stateful cognitive modules do not have data races.
### SQLite + FTS5 + USearch
SQLite is the most deployed database in the world for a reason. WAL mode gives us concurrent reads alongside writes. FTS5 gives us BM25 keyword search with zero operational overhead. USearch provides a Rust-native HNSW index for approximate nearest-neighbor vector search. The entire memory store is a single file at `~/.vestige/vestige.db`.
### fastembed (Nomic Embed v1.5)
All embeddings run locally. The Nomic Embed v1.5 model produces 768-dimensional vectors, runs via ONNX Runtime, and is competitive with OpenAI's ada-002. The model is cached in the platform user cache directory after first download (~130MB), with `./.fastembed_cache` as a fallback. No API keys. No network calls during operation. Your memories never leave your machine.
### Performance
| Memories | Search Time | Memory Usage |
|----------|-------------|--------------|
| 100 | < 10ms | ~50MB |
| 1,000 | < 50ms | ~100MB |
| 10,000 | < 200ms | ~300MB |
---
## Results: 734 Tests and What They Cover
Vestige has 734 tests across the workspace (313 in vestige-core, 338 in vestige-mcp, plus e2e tests). Every cognitive module has dedicated test coverage:
- **FSRS-6 algorithm:** Retrievability monotonic decay, round-trip interval calculation, sentiment boost, same-day review stability, difficulty mean reversion, fuzzing determinism
- **Prediction Error Gating:** Empty candidates, near-identical reinforcement, demoted memory supersession, orthogonal content creation, contradiction detection, force-create/force-update intents
- **Synaptic Tagging:** Tag creation and capture, PRP triggering, weak event rejection, clustering, tag decay and cleanup, batch operations, capture window probability
- **Spreading Activation:** Network creation, edge addition, BFS propagation with decay, activation thresholds, edge reinforcement
- **Memory Dreaming:** Full dream cycle, tag similarity, connection graph CRUD, consolidation scheduling, activity tracking
### Comparison with Existing Approaches
| Feature | RAG (Pinecone/Chroma) | mem0 | Vestige |
|---------|----------------------|------|---------|
| Forgetting curve | No | No | FSRS-6 (21-param power law) |
| Duplicate detection | Manual | Basic | Prediction Error Gating |
| Retroactive importance | No | No | Synaptic Tagging & Capture |
| Retrieval strengthening | No | No | Testing Effect + spreading activation |
| Dream consolidation | No | No | 5-stage sleep model |
| Query expansion | No | No | HyDE (template-based) |
| 3D visualization | No | No | Three.js + WebSocket |
| Local embeddings | Optional | Cloud | Always local (Nomic v1.5) |
| Single binary | No | No | include_dir! embedded dashboard |
| License | Proprietary/OSS | OSS | AGPL-3.0 |
---
## What We Learned
**Neuroscience is an engineering goldmine.** The literature on human memory is vast, detailed, and largely untapped by the AI systems community. Papers from the 1970s through 2000s describe algorithms that directly translate into code -- Collins & Loftus's spreading activation is literally a BFS with weighted edges and decay. FSRS-6 is a parameterized forgetting curve. STC is a temporal window query with capture probability.
**The Testing Effect changes everything.** Making search a write operation (not just read) transforms the memory dynamics. Frequently accessed memories get stronger. Competitors get weaker. The system self-organizes toward surfacing what matters most.
**Prediction Error Gating eliminates the "save or not" problem.** The single hardest UX question in AI memory is: what should be saved? The answer from neuroscience is: whatever is surprising. PE Gating compares against existing knowledge and only stores what is genuinely novel. This eliminates both the "save everything" bloat and the "save nothing" amnesia.
**Dreams are not a gimmick.** Offline consolidation consistently discovers connections that real-time search misses. When you replay 50 memories and compare all pairs, patterns emerge that individual searches would never find. The insight generation is simple (tag overlap + temporal proximity + embedding similarity), but the results are surprisingly useful.
---
## What's Next
Vestige v1.9 is targeting autonomic features: a retention target system that automatically adjusts consolidation frequency, adaptive embedding model selection based on content type, and a proactive suggestion engine that surfaces relevant memories before you search for them.
Further out: emotional memory tagging via sentiment analysis (the amygdala module), multi-agent memory sharing (let your coding agent share memories with your research agent), and a training loop that personalizes the FSRS-6 weights to your individual forgetting curve.
Memory is the missing layer between context windows and persistent knowledge. We think treating it as a cognitive system -- not a database -- is the right approach.
---
Vestige is open source under AGPL-3.0 at [github.com/samvallad33/vestige](https://github.com/samvallad33/vestige).
### References
- Ebbinghaus, H. (1885). *Uber das Gedachtnis*. Duncker & Humblot.
- Collins, A. M., & Loftus, E. F. (1975). A spreading-activation theory of semantic processing. *Psychological Review*, 82(6), 407-428.
- Bjork, R. A., & Bjork, E. L. (1992). A new theory of disuse and an old theory of stimulus fluctuation. In *From learning processes to cognitive processes: Essays in honor of William K. Estes* (Vol. 2, pp. 35-67).
- Tulving, E., & Thomson, D. M. (1973). Encoding specificity and retrieval processes in episodic memory. *Psychological Review*, 80(5), 352-373.
- Frey, U., & Morris, R. G. M. (1997). Synaptic tagging and long-term potentiation. *Nature*, 385, 533-536.
- Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. *Psychological Science*, 17(3), 249-255.
- Anderson, M. C., Bjork, R. A., & Bjork, E. L. (1994). Remembering can cause forgetting: Retrieval dynamics in long-term memory. *Journal of Experimental Psychology: Learning, Memory, and Cognition*, 20(5), 1063-1087.
- Redondo, R. L., & Morris, R. G. M. (2011). Making memories last: the synaptic tagging and capture hypothesis. *Nature Reviews Neuroscience*, 12(1), 17-30.
- Gao, L., et al. (2022). Precise Zero-Shot Dense Retrieval without Relevance Labels. *arXiv:2212.10496*.
- Ye, J., et al. (2024). FSRS-6: A spaced repetition algorithm based on free recall. *github.com/open-spaced-repetition*.

484
docs/launch/demo-script.md
View file

@ -1,484 +0,0 @@
# Vestige v2.0 "Cognitive Leap" — MCP Dev Summit NYC Demo Script
**Event:** MCP Dev Summit NYC, April 1-3, 2026
**Presenter:** Sam Valladares
**Project:** Vestige — The cognitive engine that gives AI a brain.
**Tagline:** 130 years of memory research. One Rust binary. Zero cloud.
---
## Pre-Demo Checklist (Do This Before the Conference)
### Hardware
- [ ] MacBook charged, charger packed
- [ ] USB-C to HDMI adapter tested with venue projector (ask AV team for resolution)
- [ ] Phone hotspot configured as backup (embedding model already cached = no network needed)
### Software
- [ ] Vestige v2.0 binary installed: `vestige-mcp --version` shows `2.0.0`
- [ ] Claude Code installed and authenticated
- [ ] Terminal font size: 18pt minimum (audience readability)
- [ ] Browser zoom: 150% for dashboard views
### Pre-Load for Offline Demo
```bash
# 1. Ensure embedding model is cached (130MB, downloads on first use)
# Run any search to trigger download:
vestige health
# 2. Verify cache exists:
ls ~/.fastembed_cache/
# Should show: nomic-ai--nomic-embed-text-v1.5/
# 3. Pre-load ~20 diverse memories so the graph looks alive:
vestige ingest "TypeScript is my preferred language for frontend development" --tags preference,typescript
vestige ingest "React Server Components eliminate client-side data fetching waterfalls" --tags pattern,react
vestige ingest "Decided to use Axum over Actix-web for Vestige's HTTP layer because of tower middleware ecosystem" --tags decision,architecture
vestige ingest "BUG FIX: SQLite WAL mode requires separate reader/writer connections for concurrent access" --tags bug-fix,sqlite
vestige ingest "FSRS-6 uses 21 parameters trained on 700M+ Anki reviews, achieving 30% better efficiency than SM-2" --tags fsrs,science
vestige ingest "Prediction Error Gating: similarity > 0.92 = reinforce, > 0.75 = update, < 0.75 = create new" --tags pe-gating,science
vestige ingest "Synaptic Tagging (Frey & Morris 1997): memories become important retroactively when a significant event occurs within 9 hours" --tags science,synaptic-tagging
vestige ingest "Three.js InstancedMesh renders 1000+ nodes at 60fps using GPU instancing" --tags pattern,threejs
vestige ingest "MCP protocol uses JSON-RPC 2.0 over stdio — no HTTP overhead, native tool integration" --tags mcp,architecture
vestige ingest "Bjork dual-strength model: storage strength never decreases, retrieval strength decays with time" --tags science,bjork
vestige ingest "HyDE query expansion classifies intent into 6 types and generates 3-5 hypothetical document variants" --tags hyde,search
vestige ingest "Ebbinghaus forgetting curve: R = e^(-t/S) where R=retrievability, t=time, S=stability" --tags science,ebbinghaus
vestige ingest "USearch HNSW index is 20x faster than FAISS for nearest neighbor search" --tags performance,search
vestige ingest "Reconsolidation (Nader 2000): retrieved memories enter a labile state for 24-48 hours where they can be modified" --tags science,reconsolidation
vestige ingest "Anderson 1994 retrieval-induced forgetting: retrieving one memory suppresses competing memories" --tags science,competition
vestige ingest "Einstein & McDaniel 1990 prospective memory: remember to do X when Y happens, with time/context/event triggers" --tags science,prospective-memory
vestige ingest "Vestige search pipeline: overfetch -> rerank -> temporal boost -> accessibility filter -> context match -> competition -> spreading activation" --tags architecture,search-pipeline
vestige ingest "Vestige has 734 tests, 77,840 lines of Rust, 29 cognitive modules, and ships as a 22MB binary" --tags vestige,stats
vestige ingest "The difference between Vestige and every other AI memory tool: we implemented the actual neuroscience, not just a vector database with timestamps" --tags vestige,philosophy
vestige ingest "Spreading activation: when you search for one thing, related memories light up automatically, like how thinking of 'doctor' primes 'nurse'" --tags science,spreading-activation
# 4. Run a dream cycle to create connections between memories:
# (Do this through Claude Code so the CognitiveEngine processes it)
# In Claude Code, say: "Dream about my recent memories"
# 5. Verify dashboard is running:
open http://localhost:3927/dashboard
# Should see 3D graph with ~20 nodes, connections from the dream
# 6. Test the full demo flow once (time it):
# - Open terminal, open browser side by side
# - Run through the 3-minute version
# - Target: under 3 minutes with natural pacing
```
### Browser Tabs to Pre-Open
1. `http://localhost:3927/dashboard` (3D Graph view)
2. `http://localhost:3927/dashboard/feed` (Real-time event feed)
3. `http://localhost:3927/dashboard/stats` (Stats with retention histogram)
4. GitHub repo: `https://github.com/samvallad33/vestige`
### Terminal Setup
- Split terminal: left pane for Claude Code, right pane for commands
- Dark background, high contrast
- Cursor blink off (less distracting on projector)
---
## VERSION 1: 30-Second Elevator Pitch
**Use this:** Hallway conversations, after-party, meeting someone at the coffee line, anyone who asks "what are you working on?"
### The Script
> Your AI forgets everything between sessions. Every conversation starts from zero.
>
> I built Vestige — a single Rust binary that gives AI persistent memory based on real cognitive science. Not a vector database with a timestamp column. Actual FSRS-6 spaced repetition trained on 700 million reviews. Prediction error gating. Synaptic tagging. Memory dreaming. The same algorithms your brain uses.
>
> It runs as an MCP server. One command to install, one command to connect. Your AI remembers your preferences, your decisions, your bug fixes. And memories decay on the Ebbinghaus curve unless they're used — just like yours do.
>
> Twenty-two megabyte binary. Seven hundred thirty-four tests. Zero cloud dependencies. I'm at the summit if you want to see the 3D brain visualization.
### Key Points to Hit
- "Real neuroscience, not just embeddings" — this is the differentiator
- "Single Rust binary" — simplicity, performance
- "MCP server" — relevant to this audience specifically
- "Zero cloud" — privacy, local-first resonates
- End with an invitation to see the dashboard — creates a follow-up
### If They Ask One Follow-Up Question
**"How is this different from Mem0?"**
> Mem0 is a cloud memory API. Great product, well-funded. But it's fundamentally a vector store with categories. Vestige implements the actual cognitive science — memories decay on the Ebbinghaus curve, get strengthened by retrieval, get consolidated in dream cycles, compete for activation. It's the difference between a filing cabinet and a brain.
**"What's the MCP integration like?"**
> One command: `claude mcp add vestige vestige-mcp -s user`. That's it. Twenty-one tools, but they're organized into five subsystems that Claude uses automatically. You don't even think about it — your AI just starts remembering.
**"Is it open source?"**
> AGPL-3.0. Fully open. The neuroscience is the moat, not the code.
---
## VERSION 2: 3-Minute Demo — "Watch Me Think"
**Use this:** Lightning talk slot, booth demo, small group gathered around your laptop.
**Tone:** Fast. Visual. Punchy. Let the 3D graph do the talking.
**Setup:** Terminal on left half of screen. Browser with dashboard on right half. Dashboard open to the Graph page.
### [0:00-0:20] The Hook
> *[Point at 3D graph on screen, nodes floating and pulsing]*
>
> This is a brain. Not a metaphor — an actual implementation of how memory works. Every node is a memory. Every connection was discovered by a dream cycle. That glow you see? That's retention decaying on the Ebbinghaus curve in real time.
>
> This is Vestige. Let me show you what happens when I talk to Claude.
### [0:20-0:50] The Live Memory
*[Switch to terminal with Claude Code]*
```
You: Remember that I'm presenting at MCP Dev Summit NYC and my talk is about cognitive memory systems
```
*[While Claude processes, switch to browser — point at the Feed tab]*
> Watch the feed. See that? "MemoryCreated" event just fired over WebSocket. And if you look at the graph...
>
> *[Switch to Graph tab — a new node appears with a burst animation]*
>
> There. A new neuron, literally being born in real time.
### [0:50-1:30] The Search
*[Back to terminal]*
```
You: What do you know about how I'm using memory science in my work?
```
> Now watch what happens during a search.
>
> *[Switch to Feed — show SearchPerformed event]*
>
> Seven-stage pipeline just fired. It overfetched 3x results, ran them through a cross-encoder reranker, applied temporal boost, checked FSRS retention, matched context using Tulving's encoding specificity principle, ran competition — Anderson's retrieval-induced forgetting — and spread activation to related memories.
>
> *[Point at Claude's response showing recalled context]*
>
> And it just pulled back exactly the right context. Not because of keyword matching — because of cognitive science.
### [1:30-2:15] The Dream
*[Back to terminal]*
```
You: Dream about my recent memories
```
> Now the wild part. This triggers a dream cycle — inspired by how your hippocampus replays memories during sleep to find hidden connections.
>
> *[Switch to Graph — show dream mode: purple ambient, nodes pulsing sequentially]*
>
> Watch the nodes light up one by one. It's replaying memories, looking for patterns it missed. See those golden lines appearing? Those are connections it just discovered between memories that were stored at completely different times.
>
> *[Switch to Stats tab, point at retention histogram]*
>
> And here's the retention distribution. FSRS-6 — trained on 700 million Anki reviews. Every memory has a predicted decay curve. Memories that aren't accessed fade. Memories that get used get stronger. Exactly like your brain.
### [2:15-2:50] The Punchline
> *[Switch to terminal, run:]*
```bash
vestige-mcp --version
# → vestige-mcp 2.0.0
```
```bash
wc -l $(find /path/to/vestige/crates -name "*.rs") | tail -1
# → 77,840 total
```
> Seventy-eight thousand lines of Rust. Seven hundred thirty-four tests. Twenty-two megabyte binary. Ships with the dashboard embedded. Install is one npm command:
```bash
npm install -g vestige-mcp-server
claude mcp add vestige vestige-mcp -s user
```
> Two lines. Your AI now has a brain.
### [2:50-3:00] Close
> Vestige v2.0, "Cognitive Leap." Open source, AGPL-3.0. The repo is `samvallad33/vestige`. Come talk to me if you want to see the neuroscience under the hood.
---
## VERSION 3: 10-Minute Deep Dive — Full Walkthrough
**Use this:** Breakout session, workshop demo, recorded talk, anyone who gives you a stage.
**Tone:** Authoritative. Technical depth but accessible. Build from simple to mind-blowing.
**Setup:** Terminal fullscreen. Browser in separate space (swipe to switch). Have the GitHub repo open as a backup.
---
### ACT 1: The Problem [0:00-1:30]
> I want to start with a question. How many of you use Claude, or GPT, or Cursor every day?
>
> *[Hands go up]*
>
> And how many of you have had this experience: you spent two hours debugging a problem with your AI, you finally fixed it, and a week later you hit the exact same bug and your AI has absolutely no memory of the solution?
>
> *[Nods]*
>
> That's because every major AI assistant has the memory of a goldfish. Every session starts from absolute zero. Claude literally tells you this: "I don't have the ability to remember previous conversations."
>
> Now here's what's strange. We've known how to build memory systems for over a century. Hermann Ebbinghaus published the forgetting curve in 1885. Bjork and Bjork formalized the dual-strength model in 1992. FSRS-6 — the state of the art in spaced repetition — was trained on 700 million reviews and published in 2024. The science exists.
>
> Nobody implemented it for AI. So I did.
### ACT 2: Install and First Memory [1:30-3:30]
> Let me show you how fast this is. I'm starting from scratch.
```bash
# Install (macOS Apple Silicon)
npm install -g vestige-mcp-server
```
> Three binaries. The MCP server, the CLI admin tool, and a restore utility. Twenty-two megabytes total. No Docker. No Python. No node_modules. No cloud API key.
```bash
# Connect to Claude Code
claude mcp add vestige vestige-mcp -s user
```
> That's it. One command connects Vestige to Claude as a user-scoped MCP server. Let me restart Claude Code and show you what happens.
*[Open Claude Code]*
```
You: Remember that I prefer Rust over Go for systems programming, and TypeScript for frontend work.
```
> Watch what Claude does. It calls `smart_ingest` — Vestige's primary storage tool. But this isn't just shoving text into a database. Let me walk you through what just happened under the hood:
>
> 1. **Prediction Error Gating** checked if this memory already exists. It compared the new content against all stored memories using embedding similarity. Since it's novel — similarity below 0.75 — it creates a new memory.
> 2. **Importance scoring** ran four channels: novelty (is this new?), arousal (is this emotionally significant?), reward (will this be useful?), attention (is the user focused?). User preferences score high on reward.
> 3. **Intent detection** classified this as a "preference" statement.
> 4. **Synaptic tagging** marked this memory for potential retroactive strengthening if something related happens in the next 9 hours.
> 5. **Hippocampal indexing** created a fast-lookup pointer.
>
> All of that in under 50 milliseconds. In a single Rust binary. No API calls.
*[New Claude Code session]*
```
You: What programming languages do I prefer?
```
> New session. Clean context. But Vestige remembers.
>
> *[Show Claude responding with the saved preference]*
>
> And this search just ran a 7-stage cognitive pipeline. Let me show you what's happening visually.
### ACT 3: The Dashboard [3:30-5:30]
> *[Switch to browser: `localhost:3927/dashboard`]*
>
> This is the Vestige dashboard. SvelteKit 2, Three.js, WebSocket connection to the running MCP server.
*[Graph page — nodes floating in 3D space with bloom post-processing]*
> Every node is a memory. The brightness represents retention strength — how accessible this memory is right now. That's driven by FSRS-6, the same algorithm that powers modern Anki. Brighter means recently accessed or frequently used. Dimmer means it's fading.
>
> The connections between nodes? Those were discovered during dream cycles — offline consolidation where Vestige replays memories and finds hidden relationships, just like your hippocampus does during sleep.
>
> I can click any node to see its details — content, type, tags, retention strength, when it was created, when it was last accessed, its predicted decay curve.
*[Click a node, show detail panel with retention curve]*
> See this curve? That's `R(t) = (1 + FACTOR * t / S)^(-w20)` — the FSRS-6 forgetting curve. Right now this memory has 94% retention. In 7 days without access, it drops to 71%. In 30 days, 43%. But every time it's retrieved, the stability increases and the curve flattens. The more you use a memory, the harder it is to forget. Exactly like your brain.
*[Switch to Feed page]*
> This is the real-time event feed. Every cognitive operation — memory creation, search, dreaming, consolidation — fires a WebSocket event. Let me show you.
*[Switch back to terminal, run a search in Claude Code]*
```
You: How does FSRS-6 work?
```
*[Switch to Feed — show SearchPerformed event with details]*
> There — `SearchPerformed`. Query: "How does FSRS-6 work?" Results: 3 memories returned. And on the graph, watch the nodes involved pulse.
>
> *[Switch to Graph — show nodes pulsing from the search]*
>
> The blue pulse you see is spreading activation. When you search for FSRS, it doesn't just find memories about FSRS — it activates related memories about spaced repetition, Ebbinghaus, Bjork. Like how thinking about "doctor" primes "nurse" in your mind.
### ACT 4: The Dream [5:30-7:30]
> Now let me show you the feature I'm most proud of. Memory dreaming.
>
> Your brain doesn't just store memories and retrieve them. During sleep, your hippocampus replays the day's experiences, compresses them, finds connections you missed while awake, and consolidates the important ones into long-term storage. This is the science of memory consolidation — Diekelmann and Born, 2010.
>
> Vestige does the same thing.
*[In Claude Code:]*
```
You: Dream about my recent memories
```
*[Switch to Dashboard Graph — show dream mode activate]*
> Watch this. Purple ambient wash — we're entering dream mode. The graph slows down. And now watch the nodes light up one at a time.
>
> *[Point as nodes pulse sequentially]*
>
> It's replaying each memory, computing similarity to every other memory, looking for connections that weren't obvious at ingest time. See that golden line that just appeared? It just discovered that two memories stored days apart are semantically related.
*[Switch to Feed — show DreamStarted, then DreamCompleted events]*
> The dream cycle produces insights — natural language descriptions of the connections it found. And here's the critical part: the memories that get replayed during dreaming are strengthened. Their FSRS stability increases. Memories that aren't replayed continue to decay. Over time, the system naturally retains what matters and forgets what doesn't.
>
> This is not a cron job. This is not garbage collection. This is the actual computational equivalent of memory consolidation during sleep.
### ACT 5: HyDE and the Search Pipeline [7:30-9:00]
> One more thing I want to show you. The search isn't just keyword matching plus cosine similarity.
>
> Vestige v2.0 added HyDE — Hypothetical Document Embeddings. When you search for something conceptual, the system first classifies your intent: are you asking a definition question? A how-to? Reasoning about a problem? Looking up a specific fact?
>
> Based on that classification, it generates 3 to 5 hypothetical documents — what a perfect answer might look like — and creates a centroid embedding from all of them. That centroid is what gets compared against your stored memories.
>
> The result: dramatically better recall on conceptual queries. If you search "how does memory decay work," you don't just get memories that contain those words. You get memories about Ebbinghaus, FSRS, retention curves, Bjork's dual-strength model — because the hypothetical documents capture the concept, not just the keywords.
>
> This runs on top of the 7-stage pipeline:
>
> 1. Overfetch 3x results from hybrid search (BM25 keyword + semantic embedding)
> 2. Cross-encoder reranker re-scores by deep relevance
> 3. Temporal boost for recent memories
> 4. FSRS-6 retention filter — memories below threshold are inaccessible
> 5. Context matching — Tulving 1973, encoding specificity
> 6. Competition — Anderson 1994, retrieval-induced forgetting
> 7. Spreading activation — activate related memories, update predictive model
>
> And critically: searching strengthens the memories you find. This is called the Testing Effect — retrieval practice is the single most effective way to consolidate memory. Every search is a workout for the memories involved.
### ACT 6: The Numbers and the Close [9:00-10:00]
> Let me leave you with the numbers.
*[Terminal:]*
```bash
vestige-mcp --version
# → vestige-mcp 2.0.0
# Stats
# 77,840 lines of Rust
# 734 tests, zero failures
# 29 cognitive modules
# 22MB release binary with embedded dashboard
# 21 MCP tools across 5 subsystems
# 12 published neuroscience principles implemented
# <50ms typical ingest latency
# <300ns cosine similarity (benchmarked with Criterion)
# Zero cloud dependencies
# Zero API keys required
# One command to install
```
> This is what I've been building for the past three months. I'm one person, I'm twenty-one years old, and I believe this is how AI memory should work — grounded in real science, running locally, open source.
>
> Vestige v2.0, "Cognitive Leap." The repo is `github.com/samvallad33/vestige`. The dashboard is running at `localhost:3927`. I'll be around all three days — come find me if you want to talk about FSRS, or synaptic tagging, or why I think every AI assistant on the planet should have a forgetting curve.
>
> Thank you.
---
## Anticipated Audience Questions and Answers
### Technical Questions
**Q: How does this compare to RAG? Isn't this just RAG with extra steps?**
> RAG is retrieval-augmented generation — you search a corpus and inject results into the prompt. Vestige does that, but with a cognitive layer on top. RAG doesn't have retention decay. RAG doesn't have memory consolidation. RAG doesn't have prediction error gating to prevent duplicates. RAG doesn't suppress competing memories on retrieval. Vestige is to RAG what human memory is to a filing cabinet — the retrieval mechanism is similar, but the memory lifecycle is completely different.
**Q: What embedding model do you use?**
> Nomic Embed Text v1.5 by default — 768 dimensions truncated to 256 via Matryoshka representation learning. All local via ONNX through fastembed. v2.0 also supports Nomic v2 MoE (475M params, 8 experts) as an opt-in feature. The reranker is Jina v1 Turbo, with Qwen3-Reranker-0.6B available as opt-in.
**Q: What's the storage backend?**
> SQLite with WAL mode. FTS5 for keyword search with Porter stemming. USearch HNSW for vector search — 20x faster than FAISS. Separate reader/writer connections for concurrent access. Single file database. I8 vector quantization for 2x storage savings with under 1% recall loss.
**Q: How does FSRS-6 actually work? What are the 21 parameters?**
> FSRS models memory as a power-law forgetting curve: `R(t) = (1 + FACTOR * t / S)^(-w20)` where S is stability and w20 is the decay parameter. The 21 parameters were trained on 700 million Anki reviews using machine learning. They encode how difficulty changes with repeated reviews, how stability grows based on review quality (Again/Hard/Good/Easy), and how same-day reviews affect long-term retention. It's 30% more efficient than SM-2, which is what Anki has used for decades.
**Q: Does it work with Claude Desktop? Other AI clients?**
> Yes. It speaks MCP — the Model Context Protocol. One config change and it works with Claude Desktop, Cursor, VS Code Copilot, JetBrains, Windsurf, Xcode 26.3. Anything that speaks MCP.
**Q: What about multi-user or team memory?**
> The current Pro plan is more pragmatic: prove portable sync/storage first, then ship Solo and Team workflows around managed sync, backups, onboarding, and support. The open-source core stays local-first; paid team features should stay in the separate Pro/commercial boundary.
**Q: How does Prediction Error Gating prevent duplicate memories?**
> When you ingest a new memory, it computes embedding similarity against all existing memories. If similarity is above 0.92, it reinforces the existing memory (bumps FSRS stability). Between 0.75 and 0.92, it updates/merges. Below 0.75, it creates a new memory. The thresholds come from computational neuroscience research on prediction error signals — the brain stores what's surprising, reinforces what's familiar, and updates what's partially known. Same principle.
**Q: What's the performance like at scale? How many memories can it handle?**
> I'm currently running with 700+ memories and everything is under 50ms. HNSW scales logarithmically, so 10x the memories doesn't mean 10x the latency. SQLite handles millions of rows. The practical limit is probably in the hundreds of thousands before you'd want to shard — but for personal AI memory, that's years of heavy use.
### Business / Community Questions
**Q: Why AGPL and not MIT?**
> Because I don't want AWS or Google hosting Vestige as a service without contributing back. AGPL means if you serve it over a network, you must open-source your modifications. Local use is completely free and unrestricted. Cloud and team features are proprietary — the MongoDB/HashiCorp playbook.
**Q: How do I contribute?**
> GitHub: `samvallad33/vestige`. The codebase is well-tested — 734 tests, zero warnings. Good first issues are labeled. The science is documented in `docs/SCIENCE.md`. PRs welcome, especially for new cognitive modules.
**Q: Are you funded? Is this a company?**
> Not yet. This is a solo project. I built it because I believe AI memory should work like human memory — not because a VC told me to. If the right opportunity comes along, I'm open to it. But the open source core isn't going anywhere.
### Skeptical Questions
**Q: Isn't this over-engineered? Do you really need 29 cognitive modules?**
> Fair question. You could build a memory system with just embeddings and timestamps, and it would work for simple cases. But it would miss things. It wouldn't know that a memory stored last week is related to one stored last month unless you searched for both. It wouldn't automatically forget outdated information. It wouldn't retroactively strengthen a memory when something important happens later. Each module solves a specific problem that real users hit. The 29 modules aren't bloat — they're the necessary completeness of a system that actually models cognition.
**Q: How do I know the neuroscience is real and not just marketing?**
> Every principle I've implemented has a citation. Ebbinghaus 1885. Bjork and Bjork 1992. Frey and Morris 1997. Anderson 1994. Tulving 1973. Einstein and McDaniel 1990. Nader 2000. Diekelmann and Born 2010. FSRS was published with full methodology and trained on public Anki data. I didn't make any of this up — I translated peer-reviewed cognitive science into Rust.
**Q: Claude Code just added built-in memory. Doesn't that make Vestige obsolete?**
> Claude's built-in memory is a flat text file. No spaced repetition, no decay, no dreaming, no cognitive search pipeline, no visualization. It's a good first step — it validates that persistent memory matters. But it's the difference between a notepad and a brain. Vestige doesn't compete with Claude's memory — it replaces and extends it.
---
## Recovery Plans
### If the Dashboard Doesn't Load
- Vestige-mcp must be running (it auto-starts the dashboard on port 3927)
- Fall back to terminal-only demo: `vestige health` and `vestige stats` show system state
- Say: "The dashboard is SvelteKit embedded in the binary — it starts automatically. Let me show you the data from the terminal instead."
### If the Embedding Model Downloads Mid-Demo
- This only happens on first-ever use. The pre-demo checklist prevents this.
- If it happens: "The embedding model is 130MB and downloads once on first use. After that, Vestige is fully offline. Let me switch to the dashboard while it downloads."
### If Claude Code Takes Too Long to Respond
- Have a second terminal tab with pre-typed commands ready
- Switch to showing the dashboard and explaining the architecture
- Say: "While Claude processes that, let me show you what's happening in the visualization."
### If Someone Asks Something You Don't Know
- "That's a great question and I honestly don't have a good answer for it yet. Find me after the talk and let's dig into it."
- Never bluff. The audience is technical. Authenticity wins.
---
## Stage Presence Notes
- **Start from the dashboard.** The 3D graph is the hook. It's visual, it's unusual, it makes people lean in.
- **Don't rush the dream sequence.** The purple wash and sequential node pulses are the most visually impressive moment. Let it breathe for 3-4 seconds.
- **Say the scientists' names.** "Ebbinghaus," "Bjork," "Frey and Morris" — this signals that you've done the reading. The MCP Dev Summit audience respects depth.
- **Make eye contact during the punchline.** "One command. Your AI now has a brain." Look at the audience, not the screen.
- **Own your age.** Twenty-one, solo developer, zero funding. This is an asset, not a liability. You built something that the well-funded competitors haven't.
- **The dashboard is your co-presenter.** Every time Claude does something, the dashboard should be showing the corresponding event. Practice the terminal-to-browser switch until it's seamless.
- **Don't apologize.** Not for bugs, not for the AGPL, not for being solo. Confident but not arrogant. The work speaks.

123
docs/launch/opencode-adoption.md
View file

@ -1,123 +0,0 @@
# OpenCode Adoption Plan
Status: Vestige was tested with OpenCode `1.16.2` on June 8, 2026. The working config uses OpenCode's top-level `mcp.vestige` schema, not `mcpServers`.
Public promotion started:
- Vestige PR #70: `https://github.com/samvallad33/vestige/pull/70`
- OpenCode issue: `https://github.com/anomalyco/opencode/issues/31402`
- OpenCode docs/ecosystem PR: `https://github.com/anomalyco/opencode/pull/31405`
- awesome-opencode PR: `https://github.com/awesome-opencode/awesome-opencode/pull/418`
- opencode.cafe listing request: `https://github.com/R44VC0RP/opencode.cafe/issues/6`
- OpenCode persistent memory comment: `https://github.com/anomalyco/opencode/issues/16077#issuecomment-4652064625`
## Release Gate
- PR #67 is merged upstream and should be treated as the contributor-driven starting point.
- Ship the corrected OpenCode config docs and `@vestige/init` migration from stale `mcpServers.vestige` to `mcp.vestige`.
- Ship the background embedding initialization fix before making direct `npx` the main OpenCode install path. A cold published `2.1.23` package can still time out while OpenCode waits for tools.
- After release, verify all three OpenCode paths again:
- installed binary: `command: ["vestige-mcp"]`
- project memory: `command: ["vestige-mcp", "--data-dir", "./.vestige"]`
- direct npm: `command: ["npx", "-y", "-p", "vestige-mcp-server@latest", "vestige-mcp"]` with `timeout: 60000`
## Official OpenCode PR
Target repo: `https://github.com/anomalyco/opencode`
Files:
- `packages/web/src/content/docs/mcp-servers.mdx`
- `packages/web/src/content/docs/ecosystem.mdx`
MCP docs snippet:
```json
{
"$schema": "https://opencode.ai/config.json",
"mcp": {
"vestige": {
"type": "local",
"command": ["npx", "-y", "-p", "vestige-mcp-server@latest", "vestige-mcp"],
"enabled": true,
"timeout": 60000
}
}
}
```
Ecosystem row:
```md
| [Vestige](https://github.com/samvallad33/vestige) | Local MCP memory server for OpenCode that remembers project decisions, preferences, and previous fixes across sessions |
```
Positioning: local, inspectable MCP memory for OpenCode. Avoid claiming Vestige fixes OpenCode's process memory or session resume behavior.
## Awesome OpenCode
Target repo: `https://github.com/awesome-opencode/awesome-opencode`
Suggested entry, with category to confirm against maintainer preference (`data/projects/vestige.yaml` or `data/resources/vestige.yaml`):
```yaml
name: Vestige
repo: https://github.com/samvallad33/vestige
tagline: Local persistent memory for OpenCode
description: Local MCP server that lets OpenCode remember project decisions, preferences, architecture context, and previous fixes across sessions.
scope:
- global
- project
tags:
- mcp
- memory
- local-first
- sqlite
- opencode
min_version: 1.16.2
homepage: https://github.com/samvallad33/vestige/blob/main/docs/integrations/opencode.md
installation: |
npm install -g vestige-mcp-server@latest
npx @vestige/init
```
## MCP Directories
Current state:
- Official MCP Registry already lists `io.github.samvallad33/vestige` at `https://registry.modelcontextprotocol.io/v0/servers?search=vestige`.
- Smithery already lists Vestige and indexes 25 tools: `https://smithery.ai/server/@samvallad33/vestige`.
- Glama already lists Vestige, but the listing needs a refresh/fix if it shows no tools: `https://glama.ai/mcp/servers/samvallad33/vestige`.
- `mcp.so` does not show Vestige under the expected slugs yet; submit manually at `https://mcp.so/submit`.
Priority order:
1. Official MCP Registry: `https://github.com/modelcontextprotocol/registry`
2. Awesome MCP Servers: `https://github.com/punkpeye/awesome-mcp-servers`
3. Glama MCP directory: `https://glama.ai/mcp/servers`
4. Smithery: `https://smithery.ai`
5. PulseMCP: `https://www.pulsemcp.com`
Registry metadata is mostly ready: `server.json` exists and `packages/vestige-mcp-npm/package.json` has `mcpName: "io.github.samvallad33/vestige"`. Publish only when the package version and `server.json` version match the released npm package.
## Community Launch
Use tested technical copy, not hype:
> Vestige now works with OpenCode as a local MCP memory server. It gives OpenCode persistent memory for project decisions, preferences, architecture context, and previous fixes across sessions. Install with `npm install -g vestige-mcp-server@latest`, run `npx @vestige/init`, then verify with `opencode mcp list`.
High-signal channels after release:
- OpenCode Discord: `https://opencode.ai/discord`
- opencode.cafe MCP Server listing: `https://opencode.cafe`
- OpenCode memory-related GitHub issues, only where directly relevant
- Hacker News and Lobsters with a technical post about the tested OpenCode integration and failure modes
- npm keyword/discovery after the next package release includes `opencode`
## Proof Checklist
- `opencode debug config` accepts `mcp.vestige`.
- `opencode mcp list` shows `vestige connected`.
- Stale `mcpServers.vestige` examples fail in OpenCode and are migrated by `@vestige/init`.
- OpenCode tools are prefixed as `vestige_search`, `vestige_smart_ingest`, `vestige_session_context`, and `vestige_deep_reference`.
- The OpenCode guide says `timeout: 60000` for direct `npx` and `timeout: 10000` for installed binaries.

233
docs/launch/reddit-cross-reference.md
View file

@ -1,233 +0,0 @@
# Reddit Launch Posts — cross_reference Tool
## Post 1: r/ClaudeAI (Primary)
**Title:** `I built a tool that catches when Claude's memories contradict each other before it answers. It's been wrong 0 times since.`
---
I've been building Vestige — an MCP memory server that gives Claude persistent memory across sessions. FSRS-6 spaced repetition (the Anki algorithm), 29 neuroscience-inspired modules, single Rust binary. 1,111 memories stored. It works.
But last week it almost cost me hours of debugging.
Claude confidently told me a benchmark notebook should use `--enable-prefix-caching` with vLLM. I trusted it. The notebook crashed. Burned a daily submission.
The problem? I had TWO memories:
- **January**: "prefix caching crashes with our vLLM build"
- **March**: "prefix caching works with the newer vLLM build"
Claude found both. Picked the wrong one. Gave me a confident wrong answer based on the January memory. The March memory was correct — but Claude had no way to know they conflicted.
So I built `cross_reference`.
Now before answering any factual question, Claude calls:
```
cross_reference({ query: "should I use --enable-prefix-caching with vLLM?" })
```
And gets back:
```json
{
"status": "contradictions_found",
"confidence": 0.35,
"guidance": "WARNING: 1 contradiction detected across 12 memories.
The newer memory is likely more accurate.",
"contradictions": [
{
"newer": {
"date": "2026-03-18",
"preview": "Switched to a newer vLLM build that supports --enable-prefix-caching..."
},
"older": {
"date": "2026-01-15",
"preview": "prefix caching crashed with our older local vLLM build..."
},
"recommendation": "Trust the newer memory. Consider demoting the older one."
}
],
"timeline": [ /* 12 memories sorted newest-first */ ]
}
```
Claude sees the contradiction BEFORE answering. Trusts the March memory. Gets it right.
**I've been using it for 2 days. It's caught contradictions I didn't even know I had.** Old project decisions that got reversed. Config values that changed. Library versions that got upgraded. All sitting in memory, waiting to give me wrong answers.
### How it works under the hood:
1. **Broad retrieval** — pulls up to 50 memories related to the query
2. **Cross-encoder reranking** — filters for quality (Jina Reranker v1 Turbo)
3. **Pairwise contradiction detection** — every memory pair checked for negation patterns ("don't" vs "do", "deprecated" vs "recommended") and correction signals ("now uses", "switched to", "replaced by")
4. **Topic gating** — only compares memories that share significant words (prevents false positives between unrelated memories)
5. **Confidence scoring** — agreements boost confidence, contradictions tank it, recency helps
6. **Timeline** — everything sorted newest-first so Claude sees the evolution
7. **Superseded list** — explicitly identifies which old memories should be demoted
### The bigger picture:
Context windows are now 1M tokens (Claude Opus 4.6, GPT-5.4). But bigger context doesn't fix this problem. The "Lost in the Middle" research shows accuracy drops from 92% to 78% at 1M tokens. More context = more chances for contradictions to slip through.
Memory systems need to be SMARTER, not just bigger. That's what Vestige does — 29 cognitive modules implementing real neuroscience:
- **FSRS-6** — memories naturally decay when unused, strengthen when accessed (21 parameters trained on 700M+ reviews)
- **Prediction Error Gating** — only stores what's genuinely new (no duplicates)
- **Dream consolidation** — replays memories to discover hidden connections (yes, like sleep)
- **Spreading activation** — search for "auth bug" and find the related JWT update from last week
- **cross_reference** — the new tool that catches contradictions before they become wrong answers
### Stats:
- 22 MCP tools
- 746 tests, 0 failures
- Zero `unsafe` code
- Clean security audit (0 findings — AgentAudit verified)
- Single 22MB Rust binary — no Docker, no PostgreSQL, no cloud
- Works with Claude Code, Cursor, VS Code, Xcode 26.3, JetBrains, Windsurf
### Install (30 seconds):
```bash
# macOS Apple Silicon
npm install -g vestige-mcp-server
sudo mv vestige-mcp /usr/local/bin/
claude mcp add vestige vestige-mcp -s user
```
Then add to your CLAUDE.md:
```
Before answering factual questions, call cross_reference({ query: "the topic" })
to verify your memories don't contradict each other.
```
That's it. Your AI now fact-checks itself.
**GitHub:** https://github.com/samvallad33/vestige
I searched every memory MCP server out there — Mem0 (47K stars), Hindsight, Zep, Letta, OMEGA, Solitaire, MuninnDB. Some detect contradictions at ingest time (when you save). Nobody else gives the AI an on-demand tool to verify its own memories before answering, with confidence scoring and guidance on which memory to trust.
Happy to answer questions about the neuroscience, the architecture, or how to set it up.
---
## Post 2: r/LocalLLaMA (Cross-post, 2h later)
**Title:** `Your AI has 1000 memories. Some contradict each other. It doesn't know. I built the fix — 100% local, single Rust binary, zero cloud.`
---
The problem nobody talks about with AI memory:
You use a memory system. Over months, you accumulate 1000+ memories. Your project config changed 3 times. A library got deprecated. A decision got reversed. All those old memories are still there.
When your AI searches memory, it finds 5 results. Two of them disagree. The AI picks one — maybe the wrong one — and gives you a confident answer based on outdated information.
I built `cross_reference` to fix this. It's tool #22 in Vestige, my cognitive memory MCP server.
```
cross_reference({ query: "what database does the project use?" })
```
Returns:
```json
{
"status": "contradictions_found",
"confidence": 0.35,
"contradictions": [{
"newer": { "date": "2026-03", "preview": "Migrated to PostgreSQL..." },
"older": { "date": "2026-01", "preview": "Using SQLite for the backend..." }
}],
"guidance": "WARNING: Trust the newer memory."
}
```
The AI sees the conflict. Picks the right one. Every time.
**What makes this different from RAG/vector search:**
| | RAG | Vestige |
|---|---|---|
| Storage | Store everything | Prediction Error Gating — only stores what's new |
| Retrieval | Nearest neighbor | 7-stage cognitive pipeline with reranking |
| Contradictions | Returns both, hopes for the best | **Detects and flags before answering** |
| Decay | Nothing expires | FSRS-6 — memories fade naturally |
| Duplicates | Manual dedup | Self-healing via semantic comparison |
**The stack:**
- Rust 2024 edition. Single 22MB binary. No Python, no Docker, no cloud.
- FSRS-6 spaced repetition (21 parameters, 700M+ reviews)
- 29 cognitive modules (spreading activation, synaptic tagging, dream consolidation, prediction error gating)
- 3D neural dashboard at localhost:3927 (Three.js, real-time WebSocket)
- MCP server — works with any AI that speaks MCP
**100% local. Your data never leaves your machine.**
```bash
npm install -g vestige-mcp-server
sudo mv vestige-mcp /usr/local/bin/
claude mcp add vestige vestige-mcp -s user
```
746 tests. Zero unsafe code. Clean security audit. AGPL-3.0.
GitHub: https://github.com/samvallad33/vestige
---
## Post 3: r/rust (Optional, technical audience)
**Title:** `I built a 22MB Rust binary that gives AI agents a brain — FSRS-6, 29 cognitive modules, 3D dashboard, and a new contradiction detection tool. 746 tests, zero unsafe.`
---
Vestige is a cognitive memory engine for AI agents. MCP server (stdio JSON-RPC + HTTP transport), SQLite WAL backend, USearch HNSW vector search, Nomic Embed v1.5 via fastembed (local ONNX, no API).
The latest addition: `cross_reference` — pairwise contradiction detection across memories at retrieval time. The AI calls it before answering factual questions to verify its memories don't disagree.
**Why Rust:**
- Single static binary (22MB with embedded SvelteKit dashboard via `include_dir!`)
- No runtime, no GC pauses during real-time search
- `tokio::sync::Mutex` for the cognitive engine, `std::sync::Mutex` for SQLite reader/writer split
- Zero `unsafe` blocks in the entire codebase
- `cargo test` runs 746 tests in 11 seconds
**Architecture:**
```
Axum 0.8 (dashboard + HTTP transport)
↕ WebSocket event bus (tokio::broadcast, 1024 capacity)
MCP Server (stdio JSON-RPC)
→ 22 tools dispatched via match on tool name
→ Arc<Storage> + Arc<Mutex<CognitiveEngine>>
SQLite WAL + FTS5 + USearch HNSW
→ fastembed 5.11 (Nomic Embed v1.5, 768D → 256D Matryoshka)
→ Jina Reranker v1 Turbo (cross-encoder, 38M params)
```
**Key crates:** rusqlite 0.38, axum 0.8, tokio 1, fastembed 5.11, usearch 2, chrono 0.4, serde 1, uuid 1, include_dir 0.7
**What I learned building it:**
- `OnceLock<Result<Mutex<TextEmbedding>, String>>` for lazy model initialization — the embedding model downloads ~130MB on first run, `OnceLock` ensures it only happens once and caches the error if it fails
- `floor_char_boundary()` saved me from a UTF-8 panic (content truncation with multi-byte chars)
- SQLite `PRAGMA journal_mode = WAL` + `synchronous = NORMAL` + `mmap_size = 268435456` gives surprisingly good concurrent read performance
- `try_lock()` pattern for cognitive features in search — if the lock is held (by dream consolidation), search falls back to simpler scoring instead of blocking
Clean security audit. Parameterized SQL everywhere. CSP headers on the dashboard. Constant-time auth comparison (`subtle::ConstantTimeEq`). File permissions 0o600/0o700.
GitHub: https://github.com/samvallad33/vestige
AGPL-3.0 | 746 tests | 79K+ LOC
---
## Posting Strategy
| Subreddit | When | Expected Engagement |
|-----------|------|-------------------|
| r/ClaudeAI | First — 12-14 UTC (US morning) | High — direct audience for MCP tools |
| r/LocalLLaMA | 2h after r/ClaudeAI | High — local-first angle resonates here |
| r/rust | Same day, evening UTC | Medium — technical deep dive audience |
| r/MachineLearning | Next day if first two do well | Lower but prestigious |
**Title formula that works on Reddit:** Personal story + specific problem + "nobody else has this"
**DO NOT do:** Product-name-first titles, marketing speak, "introducing X", "check out my project"
**DO:** Lead with the PAIN ("my AI gave me wrong answers"), show the FIX (the JSON output), then reveal the tool.

547
docs/launch/show-hn.md
View file

@ -1,547 +0,0 @@
# Vestige v2.0 Launch — Show HN + Cross-Posts
---
## 1. Hacker News — Show HN
### Title (76 chars)
```
Show HN: Vestige – FSRS-6 spaced repetition as long-term memory for AI agents
```
### Body (first comment)
```
Hi HN,
I built Vestige because every AI conversation starts from zero. Your AI has no
memory of yesterday. I wanted to fix that with real science, not just a vector
database with a wrapper.
**What it is:** A memory server for AI agents (MCP protocol). It sits between
you and your AI — Claude, Cursor, VS Code Copilot, etc. — and gives it genuine
long-term memory with cognitive-science-backed forgetting, strengthening, and
retrieval. Written in Rust, 100% local, single 22MB binary.
**The neuroscience stack:**
- **FSRS-6 spaced repetition** — the same 21-parameter power-law forgetting
model behind Anki's best algorithm, trained on 700M+ reviews. Memories decay
along empirically-validated curves instead of living forever with equal weight.
- **Prediction Error Gating** — on ingest, new content is compared against all
existing memories. If similarity >92%, it reinforces. 75-92%, it merges.
<75%, it creates. This is inspired by how the brain decides what's worth
encoding vs. what's redundant.
- **Dual-strength model** (Bjork & Bjork, 1992) — each memory tracks storage
strength (how well-encoded, only increases) and retrieval strength (how
accessible right now, decays over time). A memory can be well-stored but hard
to retrieve, like a name on the tip of your tongue.
- **Testing Effect** — every search automatically strengthens matching memories.
Using memory makes it stronger. This is one of the most robust findings in
cognitive psychology (Roediger & Karpicke, 2006).
- **Synaptic Tagging** (Frey & Morris, 1997) — when something important happens,
it retroactively strengthens memories from the surrounding time window (default:
9 hours back, 2 hours forward). This models how the brain consolidates memories
during waking hours.
- **Spreading Activation** (Collins & Loftus, 1975) — searching for "React hooks"
surfaces "useEffect" memories through semantic similarity, even without keyword
overlap.
- **Memory Dreaming** — offline consolidation that replays recent memories to
discover hidden connections and synthesize insights. Inspired by hippocampal
replay during sleep.
**v2.0 adds:**
- 3D neural visualization dashboard (SvelteKit + Three.js) — watch memories
pulse when accessed, burst particles on creation, golden flash lines when
connections form. GPU instanced rendering handles 1000+ nodes at 60fps.
- WebSocket event bus — every cognitive operation (search, dream, consolidation,
decay) broadcasts real-time events to the dashboard.
- HyDE query expansion — template-based Hypothetical Document Embeddings that
classify query intent into 6 types, expand into 3-5 variants, and average
the embedding vectors. Dramatically improves conceptual search.
- Everything compiles into a single 22MB binary. The SvelteKit dashboard is
embedded via Rust's `include_dir!` macro. No Docker, no Node runtime, no
external services.
**Numbers:** 77,840 lines of Rust, 734 tests, 29 cognitive modules, 21 MCP
tools, search under 50ms for 1000 memories (SQLite FTS5 + USearch HNSW).
**What it is NOT:** This is not RAG. RAG treats memory as a static database —
chunk everything, embed it, top-k retrieve. Vestige treats memory as a dynamic
cognitive system. Memories decay. Using them makes them stronger. Important
events retroactively strengthen recent memories. Irrelevant memories fade. The
system evolves.
The embedding model (Nomic Embed Text v1.5) runs locally via ONNX. After the
first-run model download (~130MB), there are zero network requests. No
telemetry, no analytics, no phoning home.
I've been using this daily for 2 months and the experience is genuinely different.
Claude remembers my coding patterns, my architectural decisions, my preferences.
New sessions start with context instead of a blank slate.
Source: https://github.com/samvallad33/vestige
Happy to answer any questions about the cognitive science, the Rust architecture,
or MCP in general.
```
---
## 2. Prepared FAQ for HN Comments
### Q: "How is this different from just shoving everything into a vector database?"
```
The core difference: a vector database gives everything equal weight forever.
Vestige applies forgetting.
In a vector DB, a note from 6 months ago you never referenced sits alongside
critical context from yesterday, both equally retrievable. Over time your
retrieval quality degrades because the signal-to-noise ratio gets worse.
Vestige uses FSRS-6 (the same algorithm as Anki's best spaced repetition mode)
to model forgetting curves. Memories you use get stronger (Testing Effect).
Memories you ignore fade (power-law decay). Important events retroactively
strengthen nearby memories (Synaptic Tagging).
The result is a system where retrieval quality improves over time instead of
degrading. The AI surfaces what's actually relevant, not just what's
semantically closest.
It also deduplicates on ingest (Prediction Error Gating) — if you try to store
something 92%+ similar to existing memory, it reinforces instead of creating a
duplicate. This keeps the knowledge base clean without manual maintenance.
```
### Q: "Why not just use Claude's built-in memory / ChatGPT memory?"
```
Two reasons: control and science.
Control: Native AI memory is a black box on someone else's servers. You can't
see what was stored, how it decays, or export it. Vestige stores everything in
a local SQLite database you own. You can query it directly, back it up, export
as JSON, or delete it entirely.
Science: Native memory implementations are proprietary. We have no idea what
algorithm they use for retention or retrieval. Vestige uses published research
— FSRS-6 (power-law forgetting, 21 parameters, trained on 700M Anki reviews),
dual-strength model (Bjork & Bjork 1992), encoding specificity (Tulving 1973).
These are well-studied, empirically-validated models.
They also work simultaneously — Claude's native memory handles general context,
Vestige handles structured knowledge with explicit cognitive science.
```
### Q: "77K lines of Rust seems like a lot for a memory system. What's in there?"
```
Fair question. Roughly:
- ~22K: fastembed (vendored fork of the embedding library, ONNX inference)
- ~15K: 29 cognitive modules (FSRS-6, prediction error gating, synaptic
tagging, spreading activation, dreaming, hippocampal index, etc.)
- ~12K: MCP server + 21 tool implementations
- ~8K: Storage layer (SQLite, FTS5, HNSW vector index, migrations)
- ~7K: SvelteKit dashboard (TypeScript/Svelte, embedded in binary)
- ~6K: Tests (734 tests across core + mcp + e2e + doctests)
- ~5K: Search pipeline (hybrid BM25+semantic, RRF fusion, HyDE, reranker)
- ~3K: Dashboard backend (Axum, WebSocket, REST API, event system)
Is it over-engineered? Maybe. But each cognitive module implements a specific
finding from memory research. The complexity comes from faithfully modeling how
memory actually works, not from unnecessary abstraction.
```
### Q: "Does FSRS-6 actually make a difference, or is it just a gimmick?"
```
FSRS-6 is the state of the art in spaced repetition. It was developed by the
open-spaced-repetition group, trained on 700M+ Anki reviews, and benchmarks
30% more efficient than SM-2 (the algorithm most SRS apps use, which dates
to 1987).
The key insight is the forgetting model. SM-2 uses exponential decay, which
doesn't match empirical data. FSRS-6 uses a power-law curve:
R(t, S) = (1 + factor * t / S)^(-w20)
Power-law forgetting has been consistently demonstrated in memory research
since Wixted & Ebbesen (1991). The difference matters practically — exponential
decay predicts memories fall off a cliff, while power-law decay predicts a long
tail where old memories can still be retrieved.
For an AI memory system, this means old but important memories don't vanish.
They fade slowly and can be revived by accessing them (the Testing Effect).
```
### Q: "Does this work with models other than Claude?"
```
Yes. Vestige speaks MCP (Model Context Protocol), which is supported by Claude,
Cursor, VS Code Copilot, JetBrains, Windsurf, and others. Any MCP-compatible
client can use it.
The CLAUDE.md configuration in the repo tells the AI when and how to use the
memory tools, but the underlying server is model-agnostic. You could write
equivalent instructions for any model that supports MCP tool calling.
```
### Q: "Why AGPL-3.0?"
```
To prevent cloud providers from hosting Vestige as a competing service without
contributing back. AGPL requires that if you serve the software over a network,
you must open-source your modifications.
The core memory system is fully open source and always will be. If you run it
locally (which is the intended use case), AGPL is functionally identical to GPL.
```
### Q: "What's the performance like? SQLite seems like it would be slow for vectors."
```
SQLite handles the keyword search (FTS5 — very fast). Vector search uses USearch
HNSW with int8 quantization, which is separate from SQLite.
Benchmarks on an M1 MacBook Pro:
100 memories: <10ms search
1,000 memories: <50ms search
10,000 memories: <200ms search
100,000 memories: <1s search
cosine_similarity: 296ns
RRF fusion: 17µs
Embedding generation: ~100ms per memory (only on ingest)
For personal use (hundreds to a few thousand memories), search is essentially
instant. The bottleneck is embedding generation, which only happens when storing
new memories.
```
### Q: "How does the 3D dashboard work? Is it practical or just eye candy?"
```
Both, honestly. The 3D force-directed graph is genuinely useful for seeing
clusters of related knowledge, discovering memories you forgot about, and
watching the dreaming process in real-time (memories pulse purple as they're
replayed).
But the primary UX for interacting with memories is through your AI. The
dashboard is a monitoring and exploration tool, not the main interface. You
talk to Claude/Cursor/etc normally, and Vestige handles memory in the
background via MCP tool calls.
The dashboard is embedded in the binary via include_dir! — no separate server.
It's served on localhost:3927/dashboard alongside the MCP stdio server.
```
### Q: "Is the 'dreaming' feature actually doing anything useful?"
```
It does three things:
1. Replays recent memories and computes pairwise semantic similarity to
discover connections you didn't explicitly create.
2. Identifies memories that should be linked based on content overlap,
creating an associative network.
3. Synthesizes short insights from clusters of related memories.
Is it as sophisticated as hippocampal replay during NREM sleep? No. It's an
engineering approximation that runs the same kind of "offline processing" —
reviewing, connecting, consolidating — that biological memory consolidation
does during sleep. The connections it discovers are sometimes genuinely
surprising and useful.
```
---
## 3. Reddit Cross-Posts
### r/rust
**Title:** `Vestige v2.0 — 77K LOC Rust memory system with FSRS-6, HNSW, Axum WebSockets, and an embedded SvelteKit dashboard in a 22MB binary`
**Body:**
```markdown
I've been building Vestige for the past few months and just shipped v2.0. It's
a cognitive memory system for AI agents that implements neuroscience-backed
memory algorithms in pure Rust.
**Rust-specific highlights:**
- **Single binary deployment**: SvelteKit dashboard compiled to static files,
then embedded into the Rust binary via `include_dir!`. The entire system —
MCP server, HTTP dashboard, WebSocket event bus, embedding inference — ships
as one 22MB binary.
- **fastembed vendored fork**: We vendor a fork of the fastembed-rs crate for
ONNX embedding inference (Nomic Embed Text v1.5, 768-dim vectors truncated
to 256 via Matryoshka). Feature flags for Nomic v2 MoE and Metal GPU
acceleration.
- **Axum 0.8 + tokio broadcast**: Dashboard runs on Axum with WebSocket
upgrade at `/ws`. A single `tokio::broadcast::channel(1024)` propagates
events from the MCP stdio server to all connected dashboard clients.
- **USearch HNSW**: Vector similarity search via USearch with int8
quantization (M=16, efConstruction=128, efSearch=64). Criterion benchmarks:
cosine_similarity at 296ns, RRF fusion at 17µs.
- **SQLite + FTS5**: rusqlite 0.38 with WAL mode, reader/writer connection
split, FTS5 porter tokenizer for keyword search. Interior mutability via
`Mutex<Connection>` — all Storage methods take `&self`.
- **Rust 2024 edition**: Using `use<'_>` captures in RPITIT and the latest
edition features. MSRV 1.85.
- **Release profile**: `lto = true`, `codegen-units = 1`, `opt-level = "z"`,
`strip = true` gets the binary down to 22MB including embedded assets.
- **734 tests**: 352 core + 378 mcp + 4 doctests. Zero warnings.
**Architecture:**
```
MCP Client (Claude/Cursor/etc)
| stdio JSON-RPC
v
McpServer (rmcp 0.14)
|
+---> Arc<Storage> (SQLite + HNSW)
|
+---> Arc<Mutex<CognitiveEngine>> (29 modules)
|
+---> broadcast::Sender<VestigeEvent>
|
v
Axum Dashboard (port 3927)
|
+---> /ws (WebSocket)
+---> /api/* (REST)
+---> /dashboard/* (SvelteKit static)
```
The cognitive engine implements FSRS-6 spaced repetition, prediction error
gating, synaptic tagging, spreading activation, and memory dreaming. Each
module is a stateful struct initialized once at startup and shared via Arc.
**What I'd do differently:** The fastembed vendoring is the ugliest part of the
codebase. ONNX Runtime bindings are notoriously painful in Rust, and I spent
more time fighting `ort` than any other dependency. If I started over, I might
explore `candle` as the primary backend instead of ORT.
Source: https://github.com/samvallad33/vestige
License: AGPL-3.0
Happy to discuss any of the Rust architecture decisions.
```
---
### r/ClaudeAI
**Title:** `Vestige v2.0 "Cognitive Leap" — give Claude real long-term memory with neuroscience-backed forgetting, a 3D dashboard, and 21 MCP tools`
**Body:**
```markdown
Vestige gives Claude persistent memory across sessions using real cognitive
science instead of just dumping everything into a database.
**The problem it solves:** Every Claude conversation starts from zero. Even with
native memory, you have no control over what's stored, how it decays, or where
it lives. Vestige gives Claude a proper long-term memory system that runs 100%
locally on your machine.
**What makes it different from other MCP memory servers:**
- **Memories decay like real memories.** FSRS-6 spaced repetition (the same
algorithm Anki uses) models forgetting curves. Memories you use get stronger.
Memories you ignore fade. Important events retroactively strengthen recent
memories.
- **Smart deduplication.** When Claude tries to save something similar to what
it already knows, Prediction Error Gating decides whether to create, merge,
or just reinforce the existing memory. No manual cleanup needed.
- **29 cognitive modules** implementing findings from memory research: dual-
strength model, testing effect, synaptic tagging, spreading activation,
context-dependent retrieval, memory dreaming.
**v2.0 new features:**
- **3D Memory Dashboard** at localhost:3927/dashboard — watch Claude's mind in
real-time. Memories pulse when accessed, burst particles on creation, golden
lines when connections form. SvelteKit + Three.js with bloom post-processing.
- **Real-time event bus** — every cognitive operation (search, dream,
consolidation) broadcasts WebSocket events to the dashboard.
- **HyDE query expansion** — dramatically better search for conceptual queries.
- **Single 22MB binary** — everything embedded, no Docker, no Node, no cloud.
**Setup (2 minutes):**
```bash
npm install -g vestige-mcp-server
claude mcp add vestige vestige-mcp -s user
```
Then add the CLAUDE.md instructions from the repo to tell Claude how to use
memory tools automatically.
**What it's like in practice:** After 2 months of daily use, Claude remembers
my coding patterns, my architectural decisions, my preferences across every
project. New sessions start with context instead of a blank slate. It knows I
prefer Rust over Go, that I use Tailwind, and that my last debugging session
on Project X ended with a tricky race condition in the WebSocket handler.
It's the difference between talking to someone with amnesia vs. someone who
actually knows you.
21 MCP tools. 77,840 lines of Rust. 734 tests. Works with Claude Code, Claude
Desktop, Cursor, VS Code Copilot, JetBrains, Windsurf, and Xcode.
Source: https://github.com/samvallad33/vestige
Happy to answer questions or help with setup.
```
---
### r/LocalLLaMA
**Title:** `Vestige v2.0 — local-first AI memory server with FSRS-6 spaced repetition, ONNX embeddings, and zero cloud dependency (77K LOC Rust, 22MB binary)`
**Body:**
```markdown
Vestige is a memory system for AI agents that runs entirely on your machine.
No cloud, no API keys, no telemetry. After the first-run embedding model
download (~130MB), it makes zero network requests.
**Why this matters for local LLM setups:**
Most memory/RAG solutions assume cloud embeddings (OpenAI, Cohere, etc.).
Vestige embeds locally via ONNX (Nomic Embed Text v1.5, 768-dim vectors) and
stores everything in a local SQLite database. If you're already running local
models, your memory system should be local too.
**The cognitive science angle:**
This isn't just another vector database wrapper. It implements real memory
algorithms:
- **FSRS-6**: The state-of-the-art spaced repetition algorithm (power-law
forgetting, 21 parameters, trained on 700M+ Anki reviews). Memories decay
naturally instead of living forever.
- **Prediction Error Gating**: On ingest, compares new content against existing
memories. Creates/merges/reinforces based on novelty. Prevents bloat.
- **Testing Effect**: Searching for a memory automatically strengthens it.
Memory improves through use.
- **Dual-strength model**: Storage strength (how well-encoded, never decreases)
vs. retrieval strength (how accessible now, decays over time). Mimics the
tip-of-tongue phenomenon.
- **Memory Dreaming**: Offline consolidation that replays memories to discover
connections. Inspired by hippocampal replay during sleep.
**Technical stack:**
- Rust, single 22MB binary
- Nomic Embed Text v1.5 via ONNX (local, ~130MB model)
- Optional: Nomic v2 MoE (475M params, feature flag), Metal GPU (Apple Silicon)
- SQLite FTS5 for keyword search + USearch HNSW for vector search
- MCP protocol (works with Claude, Cursor, VS Code Copilot, etc.)
- Axum HTTP + WebSocket for dashboard
- SvelteKit + Three.js 3D visualization (embedded in binary)
**v2.0 highlights:**
- 3D force-directed memory graph with real-time WebSocket events
- HyDE query expansion (template-based hypothetical document embeddings)
- FSRS decay visualization with retention curves
- 734 tests, 29 cognitive modules, 21 tools
- fastembed 5.11 with feature flags for Nomic v2 MoE + Qwen3 reranker
**Performance:**
- Search: <50ms for 1K memories, <200ms for 10K
- Embedding: ~100ms per memory (ingest only)
- cosine_similarity: 296ns (Criterion benchmark)
- Memory: ~100MB for 1K memories, ~300MB for 10K
**For local model users specifically:** Vestige speaks MCP (Model Context
Protocol). If your local model setup supports MCP tool calling, it can use
Vestige directly. The CLAUDE.md instructions in the repo tell the model when
and how to use memory tools — you can adapt these for any model.
The embedding model downloads from Hugging Face on first run and caches
locally. After that, fully air-gapped.
Source: https://github.com/samvallad33/vestige
License: AGPL-3.0 (use freely for local/personal use, cloud service requires
source disclosure)
This is a solo project — feedback, issues, and contributions are very welcome.
```
---
## 4. Posting Strategy
### Timing
- **Show HN**: Tuesday or Wednesday, 8-10 AM EST (peak engagement window)
- **r/rust**: Same day, 1-2 hours after HN post goes up
- **r/ClaudeAI**: Same day, stagger by 1 hour
- **r/LocalLLaMA**: Same day or next morning
### Rules to Follow
- **HN**: Title is the post. First comment is the body above. Respond to every
comment within 30 minutes for the first 3 hours. Be humble, technical,
transparent about limitations. Never say "AI-powered" or "game-changer."
- **Reddit**: Each subreddit gets a tailored post emphasizing what that
community cares about. No cross-linking between posts. Engage authentically.
- **General**: Lead with the science, not the product. Let the tech speak.
Acknowledge competitors honestly. Never disparage alternatives.
### Key Messaging Points
1. This is a solo project built on published research, not a startup pitch
2. The neuroscience is real but honestly described (some modules are faithful
implementations, some are engineering heuristics inspired by research)
3. 100% local, zero cloud — this is a feature, not a limitation
4. The 3D dashboard is a genuine exploration tool, not just eye candy
5. FSRS-6 is the differentiator — no other AI memory system uses real spaced
repetition
### What NOT to Say
- "Revolutionary" / "game-changing" / "paradigm shift"
- "AI-powered" (it IS AI infrastructure, don't label it that way)
- Anything negative about Mem0, Cognee, or other competitors
- Claims about being "the best" at anything
- Marketing language of any kind

20
docs/launch/tool-consolidation-v2.2.0.md
View file

@ -4,6 +4,11 @@
> tool, then make the few always-on tools deterministic. Two layers: Layer 1
> (this release) collapses 34 advertised tools to 12; Layer 2 (follow-up) shrinks
> the *default* surface and enforces the memory loop with hooks.
>
> Note: v2.2.0 also ships one **net-new** tool from a separate value stream — the
> flagship `backfill` (Retroactive Salience, Cai 2024 *Nature*). It is not part of
> this fold, so the shipped advertised surface is **13** (12 from consolidation +
> `backfill`), as asserted by `test_tools_list_returns_all_tools`.
## Why (frontier evidence)
@ -21,7 +26,7 @@ ignores make the 5 it uses harder to choose:
Vestige had **34** advertised tools. This is the correction.
## Layer 1 — Count reduction (THIS RELEASE): 34 → 12 advertised
## Layer 1 — Count reduction (THIS RELEASE): 34 → 12 (fold result)
Principle: **one consolidation per commit, one change per submission.** Each
consolidation is its own commit, landed in a safe order with the hot retrieval
@ -42,9 +47,12 @@ habits keep working) and is removed in **v2.3.0**.
`recall` is committed **last** because it is the hot path.
### Final advertised surface (12)
### Final advertised surface (13)
| Standalone (6) | Consolidated (6) |
The fold itself lands at 12; the release adds the net-new flagship `backfill`,
for **13 advertised**.
| Standalone (7) | Consolidated (6) |
|---|---|
| `smart_ingest` | `recall` |
| `memory` | `dedup` |
@ -52,6 +60,7 @@ habits keep working) and is removed in **v2.3.0**.
| `intention` | `graph` |
| `source_sync` | `maintain` |
| `suppress` | `session_start` |
| `backfill` | |
### Action / mode / view maps
@ -66,8 +75,9 @@ habits keep working) and is removed in **v2.3.0**.
- **`search` is folded, not kept standalone.** `recall` with no `mode` (the
default) *is* search — a zero-overhead pass-through to `search_unified`. Keeping
both `search` and `recall` advertised would be the exact RAG-MCP anti-pattern.
Final count is a clean **12**, leaving 2 slots of headroom toward a future
always-on `save` surface rather than spending them on a redundant verb.
The fold lands at a clean **12**; adding the flagship `backfill` brings the
shipped surface to **13**, still well inside the ~30-tool safe zone and leaving
headroom toward a future always-on `save` surface.
- **`graph` actions are flat peers, not nested.** `explore`'s `chain` /
`associations` / `bridges` sit alongside `predict` / `memory_graph` /
`composed_graph` actions in a single `action` enum — matching the existing

1026
docs/plans/0001-phase-1-storage-trait-extraction.md
View file

File diff suppressed because it is too large Load diff

689
docs/plans/0001a-trait-rewrite.md
View file

@ -1,689 +0,0 @@
# Phase 1 Amendment Sub-Plan: async_trait -> trait_variant
**Status**: Draft
**Depends on**: Phase 1 (already on `feat/storage-trait-phase1`, tip 790c0c8)
**Followed by**: `docs/plans/0001c-async-trait-sunset.md` (Embedder rewrite + async-trait crate removal)
**Related**: docs/adr/0002-phase-2-execution.md (decision D1), docs/plans/0001-phase-1-storage-trait-extraction.md
---
## Context
Phase 1 froze with the storage trait declared as
`#[async_trait::async_trait] pub trait MemoryStore: Send + Sync + 'static`
and a `pub use MemoryStore as LocalMemoryStore;` alias. `async_trait` boxes
every `async fn` in the trait into `Pin<Box<dyn Future + Send>>`. That is one
heap allocation per call inside the hottest code path in Vestige (insert,
search, update_scheduling are all on this surface). It also collapses the
two intended trait shapes -- a Send-bound `MemoryStore` for tokio/axum and a
non-Send `LocalMemoryStore` for thread-local backends -- into a single trait
behind an alias, removing the type-system signal we will want for Phase 2's
Postgres backend.
ADR 0002 decision D1 supersedes this. The amendment lands on the existing
`feat/storage-trait-phase1` branch before Phase 2 starts, before the PR is
opened upstream. The end state:
- `LocalMemoryStore` is the source-of-truth trait, declared with native
async-fn-in-trait (stable on MSRV 1.91), no `Sync` bound on the trait
itself, and `Sync + 'static` on the trait object.
- `MemoryStore` is auto-generated by `#[trait_variant::make(MemoryStore: Send)]`
with `Send` bounds on every returned future, so `Arc<dyn MemoryStore>` is
movable across `tokio::spawn`.
- `trait-variant` 0.1 is already present in `crates/vestige-core/Cargo.toml`.
The `async-trait` crate dependency stays in place through this sub-plan
(it is still in use by the embedder impl); removing it is the job of
`0001c-async-trait-sunset.md`.
- No production caller changes. Every production call site holds
`Arc<Storage>` (the concrete `SqliteMemoryStore` alias), which the trait
rewrite does not touch. Only the trait declaration and impl blocks change.
---
## Scope
### In scope
- Rewrite `MemoryStore` / `LocalMemoryStore` declaration in
`crates/vestige-core/src/storage/memory_store.rs` to use
`#[trait_variant::make(MemoryStore: Send)] pub trait LocalMemoryStore`.
- Remove the `pub use MemoryStore as LocalMemoryStore;` alias from the same
file. `LocalMemoryStore` becomes the real trait; `MemoryStore` is the
generated Send variant.
- Drop the `#[async_trait::async_trait]` attribute on the SQLite impl block
in `crates/vestige-core/src/storage/sqlite.rs` (line 6274). The impl block
switches from `impl MemoryStore for SqliteMemoryStore` (currently spelled
through the `LocalMemoryStore` alias) to `impl LocalMemoryStore for
SqliteMemoryStore`. `trait_variant` provides a blanket `impl<T:
LocalMemoryStore + Send> MemoryStore for T` so `&dyn MemoryStore` and
`Arc<dyn MemoryStore>` keep working unchanged.
- Update doc comments in `memory_store.rs` to drop
references to `#[async_trait::async_trait]` and instead describe the
`trait_variant` mechanism.
- Keep the existing Phase 1 test suite (`tests/phase_1/*.rs`) green. The
tests already use `Arc<dyn MemoryStore>` and `Box<dyn Embedder>`, which is
exactly the surface the rewrite is meant to preserve.
### Out of scope -- moved to 0001c
- Rewriting the `Embedder` / `LocalEmbedder` trait declaration -- handled by
`0001c-async-trait-sunset.md`.
- Dropping `#[async_trait::async_trait]` from
`crates/vestige-core/src/embedder/fastembed.rs`.
- Removing `async-trait = "0.1"` from `crates/vestige-core/Cargo.toml`.
- The hard `grep -rn 'async_trait' crates/` zero-match gate (async_trait is
still present in the embedder module after 0001a alone).
### Out of scope
- The SQLite file split (`sqlite.rs` -> `sqlite/` directory). That is the
sibling sub-plan `0001b-sqlite-split.md`.
- Any production-side refactor that switches `Arc<Storage>` to
`Arc<dyn MemoryStore>`. Production code keeps using the concrete alias.
- Adding `register_model` / `from_pool` / Postgres-specific variants of
`MemoryStoreError`. Those land with the Postgres backend in Phase 2.
- Removing the `pub type Storage = SqliteMemoryStore;` alias.
---
## Prerequisites
### Current state (verified)
- `crates/vestige-core/Cargo.toml` already declares
`trait-variant = "0.1"` (line 117). `async-trait = "0.1"` (line 119)
remains in place for the duration of this sub-plan; `0001c` removes it.
- `crates/vestige-core/src/storage/memory_store.rs:194` declares the trait
with `#[async_trait::async_trait] pub trait MemoryStore: Send + Sync +
'static`.
- `crates/vestige-core/src/storage/memory_store.rs:262` has
`pub use MemoryStore as LocalMemoryStore;`.
- `crates/vestige-core/src/storage/sqlite.rs:6274` declares
`#[async_trait::async_trait] impl
crate::storage::memory_store::LocalMemoryStore for SqliteMemoryStore`.
- Production call sites use `Arc<Storage>` (the concrete
`SqliteMemoryStore` alias). Confirmed by grep:
`grep -rn "dyn MemoryStore\|dyn LocalMemoryStore" --include="*.rs"`
returns hits only in `tests/phase_1/*.rs`, in two comments inside
`memory_store.rs` and `sqlite.rs`, and in one test-only `&dyn MemoryStore`
cast inside `sqlite.rs::tests` (line 8669).
- Workspace-wide `async_trait` usages: only the trait declarations and
impl blocks in `memory_store.rs`, `sqlite.rs`, `embedder/mod.rs`, and
`embedder/fastembed.rs` (verified by
`grep -rn async_trait --include="*.rs"`). This sub-plan touches only the
first two; the embedder files are addressed by `0001c`.
### Required crates
| Crate | Version | Action |
|-------|---------|--------|
| `trait-variant` | `0.1` | Already declared. Verify present after edit. |
| `async-trait` | `0.1` | Unchanged for this sub-plan (still used by embedder impl). Removed by `0001c`. |
| `blake3`, `thiserror`, `chrono`, `uuid`, `serde`, `serde_json` | unchanged | unchanged |
---
## Files Touched
Grouped by category. Every change is listed; nothing else gets touched.
### Trait declarations (vestige-core)
| File | Lines (approx) | Change |
|------|----------------|--------|
| `crates/vestige-core/src/storage/memory_store.rs` | 183-262 | Replace `#[async_trait::async_trait]` block with `#[trait_variant::make(MemoryStore: Send)] pub trait LocalMemoryStore`. Drop the `pub use MemoryStore as LocalMemoryStore;` alias. Update doc comments. |
### Impl blocks (vestige-core)
| File | Lines (approx) | Change |
|------|----------------|--------|
| `crates/vestige-core/src/storage/sqlite.rs` | 6274 (impl block header only) | Delete the `#[async_trait::async_trait]` attribute. Keep the `impl crate::storage::memory_store::LocalMemoryStore for SqliteMemoryStore { ... }` body verbatim. |
### Cargo dependency cleanup
None in this sub-plan. The `async-trait` crate stays declared in
`crates/vestige-core/Cargo.toml` because the embedder impl still uses it.
`0001c-async-trait-sunset.md` removes the dependency after the embedder
side is rewritten.
### Cognitive module call sites
No changes. The 29 cognitive modules under `crates/vestige-core/src/neuroscience/`
and `crates/vestige-core/src/advanced/` already operate on concrete
`SqliteMemoryStore` (via the `Storage` alias) or on plain data types
(`KnowledgeNode`, `Vec<f32>`, `ConnectionRecord`). Grep verified zero
production references to `dyn MemoryStore` or `dyn LocalMemoryStore`.
### MCP call sites
No changes. All ~30 `vestige-mcp/src/**.rs` files holding `Arc<Storage>`
keep working because:
- `Storage` is still `pub type Storage = SqliteMemoryStore;` (unchanged).
- They call inherent methods on the concrete type, never via a trait object.
- `SqliteMemoryStore` implements `LocalMemoryStore`; `trait_variant` auto-
generates a blanket `impl<T: LocalMemoryStore + Send> MemoryStore for T`,
so the concrete type also satisfies `MemoryStore` for any future caller
that wants the trait-object form.
### Test files (vestige-core integration tests)
| File | Lines | Change |
|------|-------|--------|
| `tests/phase_1/trait_round_trip.rs` | 7-18, 134 | No change. Already uses `Arc<dyn MemoryStore>` and `use vestige_core::storage::MemoryStore`. trait_variant emits a `MemoryStore` trait at the same path, so the imports resolve. |
| `tests/phase_1/send_bound_variant.rs` | 10-12, 36, 57 | No change. Already asserts the trait_variant Send invariant; the rewrite is what makes the doc comment on lines 3-4 actually true. |
| `tests/phase_1/cognitive_module_isolation.rs` | 11, 37, 76, 102, 115 | No change. |
| `tests/phase_1/embedding_model_registry.rs` | 10 | No change. |
| `tests/phase_1/domain_column_migration.rs` | 98 | No change. |
| `crates/vestige-core/src/storage/sqlite.rs::tests` | 8666-8675 | No change. The existing test casts `&s` to `&dyn MemoryStore` and calls trait methods through that vtable; trait_variant preserves this exact dyn-compatible surface. |
### Documentation
| File | Change |
|------|--------|
| `crates/vestige-core/src/storage/memory_store.rs` | Rewrite the trait-level doc comment (lines 185-193) to describe trait_variant rather than async_trait. |
| `CLAUDE.md` | No change. The repo-level architecture notes do not name the trait attribute. |
---
## Trait Declaration Rewrite
### Before (current state on `feat/storage-trait-phase1`)
`crates/vestige-core/src/storage/memory_store.rs:183-262`:
```rust
// ----------------------------------------------------------------------------
// TRAIT
// ----------------------------------------------------------------------------
/// The single storage abstraction.
///
/// `#[async_trait::async_trait]` makes every `async fn` return a
/// `Pin<Box<dyn Future + Send>>`, which is required for `Arc<dyn MemoryStore>`
/// to be movable across `tokio::spawn` boundaries.
///
/// `LocalMemoryStore` is a type alias kept for source compatibility with code
/// that refers to the non-send variant. In Phase 1 both names refer to the same
/// (dyn-compatible, Send-safe) trait.
#[async_trait::async_trait]
pub trait MemoryStore: Send + Sync + 'static {
// --- Lifecycle ---
async fn init(&self) -> MemoryStoreResult<()>;
async fn health_check(&self) -> MemoryStoreResult<HealthStatus>;
// ... 25 more async fn ...
async fn vacuum(&self) -> MemoryStoreResult<()>;
}
/// Type alias kept for source compatibility. Both names refer to the same
/// `async_trait`-annotated trait that is dyn-compatible and `Send + Sync`.
pub use MemoryStore as LocalMemoryStore;
```
### After
`crates/vestige-core/src/storage/memory_store.rs:183-262`:
```rust
// ----------------------------------------------------------------------------
// TRAIT
// ----------------------------------------------------------------------------
/// The single storage abstraction.
///
/// `LocalMemoryStore` is the source-of-truth trait. The
/// `#[trait_variant::make(MemoryStore: Send)]` attribute auto-generates a
/// `MemoryStore` trait whose returned futures are `Send`, so
/// `Arc<dyn MemoryStore>` is movable across `tokio::spawn` boundaries while
/// `Arc<dyn LocalMemoryStore>` remains usable on single-threaded executors
/// and thread-local backends.
///
/// Every method is native async-fn-in-trait (stable on MSRV 1.91); no
/// per-call heap allocation, no boxed futures.
#[trait_variant::make(MemoryStore: Send)]
pub trait LocalMemoryStore: Sync + 'static {
// --- Lifecycle ---
async fn init(&self) -> MemoryStoreResult<()>;
async fn health_check(&self) -> MemoryStoreResult<HealthStatus>;
// --- Embedding model registry ---
async fn registered_model(&self) -> MemoryStoreResult<Option<ModelSignature>>;
async fn register_model(&self, sig: &ModelSignature) -> MemoryStoreResult<()>;
// --- CRUD ---
async fn insert(&self, record: &MemoryRecord) -> MemoryStoreResult<Uuid>;
async fn get(&self, id: Uuid) -> MemoryStoreResult<Option<MemoryRecord>>;
async fn update(&self, record: &MemoryRecord) -> MemoryStoreResult<()>;
async fn delete(&self, id: Uuid) -> MemoryStoreResult<()>;
// --- Search ---
async fn search(&self, query: &SearchQuery) -> MemoryStoreResult<Vec<SearchResult>>;
async fn fts_search(&self, text: &str, limit: usize) -> MemoryStoreResult<Vec<SearchResult>>;
async fn vector_search(
&self,
embedding: &[f32],
limit: usize,
) -> MemoryStoreResult<Vec<SearchResult>>;
// --- FSRS Scheduling ---
async fn get_scheduling(
&self,
memory_id: Uuid,
) -> MemoryStoreResult<Option<SchedulingState>>;
async fn update_scheduling(&self, state: &SchedulingState) -> MemoryStoreResult<()>;
async fn get_due_memories(
&self,
before: DateTime<Utc>,
limit: usize,
) -> MemoryStoreResult<Vec<(MemoryRecord, SchedulingState)>>;
// --- Graph (spreading activation) ---
async fn add_edge(&self, edge: &MemoryEdge) -> MemoryStoreResult<()>;
async fn get_edges(
&self,
node_id: Uuid,
edge_type: Option<&str>,
) -> MemoryStoreResult<Vec<MemoryEdge>>;
async fn remove_edge(&self, source: Uuid, target: Uuid) -> MemoryStoreResult<()>;
async fn get_neighbors(
&self,
node_id: Uuid,
depth: usize,
) -> MemoryStoreResult<Vec<(MemoryRecord, f64)>>;
// --- Domains ---
async fn list_domains(&self) -> MemoryStoreResult<Vec<Domain>>;
async fn get_domain(&self, id: &str) -> MemoryStoreResult<Option<Domain>>;
async fn upsert_domain(&self, domain: &Domain) -> MemoryStoreResult<()>;
async fn delete_domain(&self, id: &str) -> MemoryStoreResult<()>;
async fn classify(&self, embedding: &[f32]) -> MemoryStoreResult<Vec<(String, f64)>>;
// --- Bulk / Maintenance ---
async fn count(&self) -> MemoryStoreResult<usize>;
async fn get_stats(&self) -> MemoryStoreResult<StoreStats>;
async fn vacuum(&self) -> MemoryStoreResult<()>;
}
```
Notes:
- The `pub use MemoryStore as LocalMemoryStore;` line on the current
`memory_store.rs:262` is **deleted** entirely. `MemoryStore` is now the
generated trait that `trait_variant::make` emits at the same module path;
`LocalMemoryStore` is the source-of-truth declaration. Both names export
from `storage/mod.rs` already (see lines 10-14 of that file).
- `Sync + 'static` on `LocalMemoryStore` (and no `Send` bound on the trait
itself) is correct: `Send` on the trait is what `trait_variant::make`
inserts when it emits the `MemoryStore` variant. The current trait carries
`Send + Sync + 'static`; the rewrite drops the `Send` bound from the
local variant. `Arc<dyn LocalMemoryStore>` is `Sync` but not `Send`;
`Arc<dyn MemoryStore>` (the generated variant) is `Send + Sync`.
- `trait_variant` 0.1 does **not** require any attribute on the impl block.
The attribute lives only on the trait declaration. See the next section.
The Embedder trait rewrite uses the identical `trait_variant::make` pattern
and is fully specified in `0001c-async-trait-sunset.md`.
---
## Impl Block Migration
`trait_variant` 0.1 attaches the attribute only to the trait declaration.
The impl side is plain `impl LocalMemoryStore for SqliteMemoryStore`; no
attribute on the impl, no `#[trait_variant::make(MemoryStore: Send)]` on the
impl block. The crate auto-generates the blanket
`impl<T: LocalMemoryStore + Send> MemoryStore for T`, so any concrete type
that implements `LocalMemoryStore` automatically also implements
`MemoryStore` provided it is `Send`.
`SqliteMemoryStore` already is `Send + Sync` (it holds `Mutex<Connection>`
fields whose `Mutex` is `std::sync::Mutex`, which is `Send + Sync` when its
guarded type is `Send`; `rusqlite::Connection` is `Send`). No new bound is
needed.
### Before
`crates/vestige-core/src/storage/sqlite.rs:6274`:
```rust
#[async_trait::async_trait]
impl crate::storage::memory_store::LocalMemoryStore for SqliteMemoryStore {
async fn init(&self) -> crate::storage::memory_store::MemoryStoreResult<()> {
// Migrations run in `new`; this is a no-op for the SQLite backend.
Ok(())
}
// ... ~2400 lines of method bodies, unchanged ...
}
```
### After
`crates/vestige-core/src/storage/sqlite.rs:6274`:
```rust
impl crate::storage::memory_store::LocalMemoryStore for SqliteMemoryStore {
async fn init(&self) -> crate::storage::memory_store::MemoryStoreResult<()> {
// Migrations run in `new`; this is a no-op for the SQLite backend.
Ok(())
}
// ... ~2400 lines of method bodies, unchanged ...
}
```
Diff is exactly one removed line (the `#[async_trait::async_trait]` attribute).
Every method body, every `async fn` signature, every `use` statement inside
the impl block stays verbatim. No `Box::pin(async move { ... })`, no manual
`Pin<Box<dyn Future>>`, no `'async_trait` lifetime markers; native
async-fn-in-trait does this directly.
The Embedder impl block rewrite follows the identical "remove one
`#[async_trait::async_trait]` attribute" pattern and is fully specified in
`0001c-async-trait-sunset.md`.
### Why the impl block does not need an attribute
`trait_variant::make` generates two things from the source trait:
1. The source trait itself (`LocalMemoryStore`), with native async fns.
2. A second trait (`MemoryStore`) whose method signatures return
`impl Future<Output = ...> + Send` instead of `impl Future<Output = ...>`,
plus a blanket impl wiring concrete types through.
Both are emitted at the macro-call site. `SqliteMemoryStore` only writes one
impl block (against `LocalMemoryStore`); the macro-generated blanket
guarantees `SqliteMemoryStore: MemoryStore` for free. The current `&dyn
MemoryStore` casts (sqlite.rs:8669; tests under tests/phase_1/) keep
type-checking unchanged.
---
## Call Site Audit
Verified via:
```bash
grep -rn "dyn MemoryStore\|dyn LocalMemoryStore" --include="*.rs" \
/home/delandtj/prppl/vestige-phase2/
grep -rn "Arc<Storage>\|Arc<SqliteMemoryStore>" --include="*.rs" \
/home/delandtj/prppl/vestige-phase2/
grep -rn "use.*MemoryStore;\|use.*LocalMemoryStore;" --include="*.rs" \
/home/delandtj/prppl/vestige-phase2/
```
### Files that reference the trait object form (`dyn MemoryStore` / `dyn LocalMemoryStore`)
All test-only or doc-comment-only:
| File | Line | Use | Required change |
|------|------|-----|-----------------|
| `tests/phase_1/trait_round_trip.rs` | 7-18 | `Arc<dyn MemoryStore>` in `make_store()` and test bodies | None. `MemoryStore` is the generated Send variant; signature stays. |
| `tests/phase_1/trait_round_trip.rs` | 134 | `Arc<dyn MemoryStore>` upcast inside a test body | None. |
| `tests/phase_1/send_bound_variant.rs` | 10-97 | `Arc<dyn MemoryStore>` moved across `tokio::spawn` | None. This test becomes meaningfully correct only after the rewrite (currently it relies on async_trait boxing; after the rewrite it relies on trait_variant's Send variant -- same observable outcome). |
| `tests/phase_1/cognitive_module_isolation.rs` | 11, 33-115 | `Arc<dyn MemoryStore>` passed into cognitive module-style closures | None. |
| `tests/phase_1/embedding_model_registry.rs` | 10 | `Arc<dyn MemoryStore>` in `make_store()` | None. |
| `tests/phase_1/domain_column_migration.rs` | 98 | `Arc<dyn MemoryStore>` cast inside a migration assertion | None. |
| `crates/vestige-core/src/storage/sqlite.rs` | 8666-8675 | `let dyn_s: &dyn MemoryStore = &s;` inside `mod tests` | None. The cast is testing that the dyn-vtable still resolves the trait methods correctly; after the rewrite it resolves through the `MemoryStore` trait that `trait_variant::make` emits at the same path. |
| `crates/vestige-core/src/storage/memory_store.rs` | 188 (doc) | Doc comment mentioning `Arc<dyn MemoryStore>` | Replaced as part of the doc rewrite (see Trait Declaration section). |
### Files that hold the concrete type (`Arc<Storage>` / `Arc<SqliteMemoryStore>`)
35 files, 116 hits. Every one of them keeps working unchanged because the
concrete `SqliteMemoryStore` type stays exactly as it is. Listed here for
completeness so a reviewer can confirm none of them needs an edit:
```
crates/vestige-core/src/storage/mod.rs (alias declaration)
crates/vestige-core/src/storage/sqlite.rs (impl block)
crates/vestige-mcp/src/server.rs (Arc<Storage> in McpServer)
crates/vestige-mcp/src/cognitive.rs (hydrate(&Storage))
crates/vestige-mcp/src/autopilot.rs
crates/vestige-mcp/src/protocol/http.rs
crates/vestige-mcp/src/dashboard/mod.rs
crates/vestige-mcp/src/dashboard/state.rs
crates/vestige-mcp/src/dashboard/handlers.rs
crates/vestige-mcp/src/resources/codebase.rs
crates/vestige-mcp/src/resources/memory.rs
crates/vestige-mcp/src/tools/changelog.rs
crates/vestige-mcp/src/tools/codebase_unified.rs
crates/vestige-mcp/src/tools/context.rs
crates/vestige-mcp/src/tools/contradictions.rs
crates/vestige-mcp/src/tools/cross_reference.rs
crates/vestige-mcp/src/tools/dedup.rs
crates/vestige-mcp/src/tools/dream.rs
crates/vestige-mcp/src/tools/explore.rs
crates/vestige-mcp/src/tools/feedback.rs
crates/vestige-mcp/src/tools/graph.rs
crates/vestige-mcp/src/tools/health.rs
crates/vestige-mcp/src/tools/importance.rs
crates/vestige-mcp/src/tools/intention_unified.rs
crates/vestige-mcp/src/tools/maintenance.rs
crates/vestige-mcp/src/tools/memory_states.rs
crates/vestige-mcp/src/tools/memory_unified.rs
crates/vestige-mcp/src/tools/predict.rs
crates/vestige-mcp/src/tools/restore.rs
crates/vestige-mcp/src/tools/review.rs
crates/vestige-mcp/src/tools/search_unified.rs
crates/vestige-mcp/src/tools/session_context.rs
crates/vestige-mcp/src/tools/smart_ingest.rs
crates/vestige-mcp/src/tools/suppress.rs
crates/vestige-mcp/src/tools/tagging.rs
crates/vestige-mcp/src/tools/timeline.rs
```
Each holds `Arc<Storage>` and dispatches to inherent methods on
`SqliteMemoryStore`. None of them goes through a trait vtable. Required
change for every one of them: **none**.
### Files that `use ...MemoryStore` from production code
```
grep -rn "use.*MemoryStore;\|use.*LocalMemoryStore;" --include="*.rs" \
| grep -v "memory_store.rs\|sqlite.rs\|tests/phase_1"
```
returns nothing. Production code does not import the trait by name.
### Conclusion
The rewrite is a strictly local change to two source files
(`storage/memory_store.rs` and `storage/sqlite.rs`). Zero production call
sites need edits. The integration tests that consume `Arc<dyn MemoryStore>`
keep their current form; the rewrite is what gives that signature its
no-box semantics on the storage side. The `Box<dyn Embedder>` surface is
addressed by `0001c-async-trait-sunset.md`.
---
## Commit Sequence
Two commits, each green on `cargo test -p vestige-core --no-default-features`
and `cargo test -p vestige-core --features embeddings,vector-search`.
### Commit 1: rewrite MemoryStore / LocalMemoryStore trait declaration
- Touches: `crates/vestige-core/src/storage/memory_store.rs` only.
- Action: replace lines 183-262 per the "Trait Declaration Rewrite" section
above. Delete the `pub use MemoryStore as LocalMemoryStore;` line.
- Green after: `cargo check -p vestige-core` (the impl block in `sqlite.rs`
still has `#[async_trait::async_trait]` on it, but it still resolves
through the `LocalMemoryStore` trait which is now native-async; the
`async_trait` macro is harmless when applied to a trait that the impl
block targets by path, because the macro rewrites the impl's async fns
into boxed-future fns whose signatures still match the native-async
declarations after trait_variant lowering). If `cargo check` complains
here, fold commit 2 into commit 1.
**Mitigation if check fails between commits 1 and 2:** combine the two
into a single commit. The split is offered for review convenience; the
build must be green after every commit lands.
### Commit 2: drop `#[async_trait::async_trait]` from SqliteMemoryStore impl
- Touches: `crates/vestige-core/src/storage/sqlite.rs` only.
- Action: delete line 6274 (`#[async_trait::async_trait]`).
- Green after: `cargo test -p vestige-core --features embeddings,vector-search`,
including all `trait_*` tests inside `sqlite.rs::tests` (lines 8643-8712)
and the trait-object cast on line 8669.
### Combined alternative
If the per-step split feels artificial, commits 1 and 2 can collapse into
a single commit covering both the trait rewrite and the impl-attribute
drop for `MemoryStore`. That is acceptable; the two-commit form is
preferred only because it lets a reviewer bisect trait-rewrite failures
separately from impl-rewrite failures.
The Embedder / fastembed commits and the `async-trait` Cargo dependency
removal live in `0001c-async-trait-sunset.md`.
---
## Verification
Every command runs from the repo root unless noted otherwise.
```bash
# 1. Vestige-core, default features (embeddings + vector-search).
cargo test -p vestige-core --features embeddings,vector-search
# 2. Vestige-core, minimal features (no embeddings, no vector-search).
cargo test -p vestige-core --no-default-features
# 3. Workspace build, release mode (catches any feature-gated regression
# in the vestige-mcp tools tree).
cargo build --workspace --release
# 4. Whole-workspace test (vestige-mcp 406 tests + vestige-core 352 tests
# per the CLAUDE.md baseline).
cargo test --workspace
# 5. Phase 1 integration tests (these are the trait-shape contract).
cargo test --test trait_round_trip --features embeddings,vector-search
cargo test --test send_bound_variant --features embeddings,vector-search
cargo test --test cognitive_module_isolation --features embeddings,vector-search
cargo test --test embedding_model_registry --features embeddings,vector-search
cargo test --test domain_column_migration --features embeddings,vector-search
# 6. Clippy gate, deny warnings (matches Phase 1 PR policy of zero warnings).
cargo clippy --workspace --all-targets --features embeddings,vector-search -- -D warnings
# 7. Storage-side dependency hygiene check (must return zero hits).
# Scoped to the storage module only -- the embedder module still uses
# async_trait until 0001c lands.
grep -rn "async_trait\|async-trait" crates/vestige-core/src/storage/
# 8. Confirm trait_variant attribute is in place on the storage trait
# (must return exactly one hit in memory_store.rs).
grep -rn "trait_variant::make" crates/vestige-core/src/storage/
```
Expected outcomes:
- Command 1: 352 tests pass (matches baseline).
- Command 2: smaller test count, all pass.
- Command 3: workspace compiles in release mode.
- Command 4: 758 total tests pass (matches CLAUDE.md baseline).
- Command 5: each phase_1 integration test binary runs green. The
`send_bound_variant::arc_dyn_memory_store_moves_across_tokio_tasks` test
is the canary; if `MemoryStore` lost its Send-bound future variant, this
fails to compile with "future cannot be sent between threads safely".
- Command 6: zero clippy warnings. The rewrite must not introduce a new
`clippy::needless_lifetimes` or `clippy::async_yields_async`.
- Command 7: empty output. async_trait is gone from the storage module.
The embedder module still contains async_trait; that is removed by
`0001c-async-trait-sunset.md`.
- Command 8: one hit, in `memory_store.rs`.
---
## Acceptance Criteria
A reviewer should be able to check every box:
- [ ] `crates/vestige-core/src/storage/memory_store.rs` declares the trait
with `#[trait_variant::make(MemoryStore: Send)] pub trait
LocalMemoryStore: Sync + 'static`, no `async_trait` attribute, no
`Send` bound on `LocalMemoryStore` itself.
- [ ] `crates/vestige-core/src/storage/memory_store.rs` no longer contains
`pub use MemoryStore as LocalMemoryStore;`.
- [ ] `crates/vestige-core/src/storage/sqlite.rs:6274` is plain
`impl crate::storage::memory_store::LocalMemoryStore for
SqliteMemoryStore` -- no attribute on the impl block.
- [ ] `grep -rn "async_trait\|async-trait" crates/vestige-core/src/storage/`
returns zero hits.
- [ ] `grep -rn "trait_variant::make" crates/vestige-core/src/storage/`
returns exactly one hit (the storage trait in `memory_store.rs`).
- [ ] All 758 workspace tests pass (`cargo test --workspace`).
- [ ] Phase 1 integration tests pass with the trait-object surface
(`Arc<dyn MemoryStore>`) intact.
- [ ] `cargo clippy --workspace --all-targets --features
embeddings,vector-search -- -D warnings` is clean.
- [ ] No production source file under `crates/vestige-mcp/` or
`crates/vestige-core/src/{neuroscience,advanced,consolidation,codebase,
memory,embeddings,embedder}/` was modified by this sub-plan.
- [ ] `Arc<Storage>` still type-checks at every existing call site (verified
by the workspace test pass).
- [ ] Doc comments on the storage trait declaration describe
`trait_variant`, not `async_trait`.
---
## Risks and Mitigations
- **`trait_variant` 0.1 macro emits unexpected diagnostics on MSRV 1.91.**
Mitigation: the master Phase 1 plan already prescribed this exact pattern
(`#[trait_variant::make(MemoryStore: Send)] pub trait LocalMemoryStore:
Sync + 'static`, see plan `0001-...` line 274-275); the crate has been in
`vestige-core/Cargo.toml` since Phase 1 landed. If a diagnostic appears,
pin to the exact known-good version with `trait-variant = "=0.1.2"` and
open an upstream issue.
- **Native async-fn-in-trait makes the trait no longer dyn-compatible.**
Mitigation: `trait_variant::make` is specifically the workaround for this
-- it emits both the source trait (for static dispatch) and a Send-bound
variant whose returned futures use `Pin<Box<dyn Future + Send>>` only at
the dyn boundary. `Arc<dyn MemoryStore>` keeps working because the
generated `MemoryStore` trait is dyn-compatible by construction. Verified
by the existing `send_bound_variant::*` tests, which intentionally move
`Arc<dyn MemoryStore>` across `tokio::spawn` from inside a
`multi_thread` runtime.
- **A cognitive module silently relied on the boxed-future return type.**
Mitigation: grep verified no cognitive module imports `MemoryStore` /
`LocalMemoryStore` or holds an `Arc<dyn ...>` form; all of them use the
concrete `Storage` alias. There is no Send-ness expectation downstream to
break.
- **Future bodies inside the SQLite impl capture non-Send locals.**
Mitigation: every method body in `sqlite.rs:6274..` runs synchronous
rusqlite calls inside the same `async fn` frame; no `.await` points
exist inside the bodies that we ship today. The `Send` bound on the
generated `MemoryStore` variant is therefore satisfied automatically.
If a future change adds `.await` inside an impl method, the new
trait_variant surface will surface that as a compile error at the call
site, which is the correct outcome.
- **`async-trait` crate is left declared after this sub-plan.**
This is intentional: the embedder impl still depends on it. The
`0001c-async-trait-sunset.md` sub-plan removes the crate after the
embedder side is rewritten. Grep on the whole workspace returns only
the storage and embedder files; no downstream crate pulls `async-trait`.
---
## Out-of-Band Notes
- This sub-plan amends `feat/storage-trait-phase1` (tip 790c0c8). The
branch has not been opened upstream yet, so amending in place is safe;
no force-push to a public PR.
- The companion sub-plan `0001b-sqlite-split.md` lands after this one on
the same branch. The trait-rewrite landing first is intentional: the
SQLite split moves the impl block into `storage/sqlite/trait_impl.rs`,
and it is cleaner to move a small attribute-free impl than a
macro-decorated one.
- The companion sub-plan `0001c-async-trait-sunset.md` lands after this
one (order with `0001b` is independent) and finishes the
async_trait -> trait_variant transition for the Embedder trait, then
removes the `async-trait` crate dependency.
- After the Phase 1 amendment sub-plans (`0001a`, `0001b`, `0001c`) land,
the branch is reviewed and merged before Phase 2 sub-plans (`0002a-`
through `0002i-`) begin implementation.

732
docs/plans/0001b-sqlite-split.md
View file

@ -1,732 +0,0 @@
# Sub-Plan 0001b: Split sqlite.rs into a sqlite/ directory
**Status**: Draft
**Branch**: `feat/storage-trait-phase1` (Phase 1 amendment, PR A)
**Depends on**: `0001a-trait-rewrite.md` (must land first; it carries the
`trait_variant`-generated trait declaration that `trait_impl.rs` matches)
**Related**: `docs/adr/0002-phase-2-execution.md` (D3, D6)
---
## Context
`crates/vestige-core/src/storage/sqlite.rs` is the single SQLite backend file
that Phase 1 inherited from pre-trait Vestige and then appended the
`LocalMemoryStore` trait impl block to. The file is 8713 lines as of
commit 790c0c8 on `feat/storage-trait-phase1`. ADR 0002 D3 decides to split
it into a `sqlite/` directory before Phase 2 lands `postgres/` as a peer.
Reasoning, in one paragraph:
The Postgres backend is going in as a directory of seven small files
(`postgres/{mod,pool,migrations,registry,search,migrate_cli,reembed}.rs`).
If SQLite stays as one 8K-line file alongside that, the repo says "backends
look like one big file or seven small ones, pick a side", which forces
every future maintainer to re-litigate the layout. Splitting now -- as
**pure code motion**, no public-API change, no behavioural change, no
migration -- lets both backends look the same, keeps each surface mappable
in a single editor tab, and shrinks the diffs Phase 2 has to review.
This sub-plan is sized as one focused implementation session.
The split is **private** to `storage/sqlite/`. Cognitive modules continue
to `use crate::storage::SqliteMemoryStore`; the existing re-exports in
`crates/vestige-core/src/storage/mod.rs` keep working without touching
any caller. Tests stay green commit-by-commit.
This sub-plan depends on `0001a-trait-rewrite.md` landing first because
`sqlite/trait_impl.rs` is the file that picks up the new trait_variant
attribute. Doing the split first would force a second rewrite of
`trait_impl.rs` when the trait rewrite arrives. Order matters; this is
the cheap-to-respect ordering.
---
## Target Layout
Final directory after this sub-plan:
```
crates/vestige-core/src/storage/sqlite/
mod.rs -- module root: SqliteMemoryStore struct, new(),
reader/writer locks, error types, shared helpers,
portable-sync-related types, record types
crud.rs -- ingest/smart_ingest/get/update/delete/purge/search-by-id
search.rs -- fts, semantic, hybrid, time-based queries
scheduling.rs -- FSRS state, decay, consolidation, review, promote/demote,
suppression, gc, retention, waking tags
graph.rs -- memory_connections (edges), subgraph, neighbors
domain.rs -- domains/domain_scores column readers, classify stub
(Phase 4 will expand this file)
registry.rs -- embedding_model table, enforce_model, register_model body
portable_sync.rs -- portable export/import/sync + merge helpers
trait_impl.rs -- impl LocalMemoryStore for SqliteMemoryStore
```
`storage/mod.rs` is unchanged in spirit: it still does `mod sqlite;` and
`pub use sqlite::{...};` -- the only difference is that `sqlite` is now a
directory module instead of a leaf file. The re-export list does not
change.
---
## Current File Map (line numbers from commit 790c0c8)
The current `sqlite.rs` is structurally:
| Region | Lines | Contents |
|--------|-------|----------|
| Header | 1-43 | Imports, feature-gated imports |
| Error types | 45-89 | `StorageError`, `Result`, `SmartIngestResult`, `MergeWrite` |
| Portable sync types | 97-211 | `PortableSyncBackend` trait, `FilePortableSyncBackend` struct, `PortableSyncReport`, `PurgeReport` |
| Constants | 233-273 | `PORTABLE_TABLES`, `PORTABLE_USER_DATA_TABLES`, `PortableMergeState`, env constants |
| Struct decl | 287-301 | `SqliteMemoryStore` struct fields |
| Impl block 1 | 303-3740 | Constructor + bulk of native API |
| Record structs | 3747-3866 | `IntentionRecord`, `InsightRecord`, `ConnectionRecord`, `MemoryStateRecord`, `StateTransitionRecord`, `ConsolidationHistoryRecord`, `DreamHistoryRecord`, `Default for InsightRecord` |
| Impl block 2 | 3868-6133 | Intentions / Insights / Connections / States / History / Backup / Portable / GC / Subgraph |
| Impl block 3 | 6139-6272 | Trait-helper methods (`node_to_record`, `read_domain_columns`, `enforce_model`) |
| Trait impl | 6274-7110 | `impl LocalMemoryStore for SqliteMemoryStore` |
| Tests | 7112-8713 | `#[cfg(test)] mod tests`: native API tests + trait round-trip tests |
---
## Mapping Table
Every public method, every private helper, every struct, every test module
in the current file -- with the destination file in the new layout. Line
ranges cite the current `sqlite.rs` (commit 790c0c8 on
`feat/storage-trait-phase1`, viewed through the
`/home/delandtj/prppl/vestige-phase2` worktree).
### Header and shared types (-> `sqlite/mod.rs`)
| Item | Lines | Destination | Notes |
|------|-------|-------------|-------|
| Module-level `use` imports | 1-43 | `sqlite/mod.rs` | Trimmed per-file in destination; what does not fit in `mod.rs` moves with its consumers |
| `StorageError` enum + `Result` alias | 49-71 | `sqlite/mod.rs` | Re-exported through `pub use` chain; called from every sub-module |
| `SmartIngestResult` struct | 73-89 | `sqlite/mod.rs` | Returned by `crud::smart_ingest`; defined here because other code depends on the type |
| `MergeWrite` enum | 91-95 | `sqlite/portable_sync.rs` | Only used by merge helpers |
| `PortableSyncBackend` trait | 97-109 | `sqlite/portable_sync.rs` | Public trait; re-exported through `mod.rs` |
| `FilePortableSyncBackend` struct + `impl` | 111-194 | `sqlite/portable_sync.rs` | |
| `PortableSyncReport` struct | 196-211 | `sqlite/portable_sync.rs` | |
| `PurgeReport` struct | 213-231 | `sqlite/crud.rs` | Returned by `purge_node` |
| `PORTABLE_TABLES` constant | 237-254 | `sqlite/portable_sync.rs` | |
| `PORTABLE_USER_DATA_TABLES` constant | 256-272 | `sqlite/portable_sync.rs` | |
| `PortableMergeState` struct | 274-277 | `sqlite/portable_sync.rs` | |
| `DATA_DIR_ENV` constant | 279 | `sqlite/mod.rs` | Read by constructor |
| `DATABASE_FILE` constant | 280 | `sqlite/mod.rs` | Read by constructor |
| `SqliteMemoryStore` struct decl | 282-301 | `sqlite/mod.rs` | All fields stay public-to-crate within the directory |
### Constructor and config (-> `sqlite/mod.rs`)
These are foundational; they live in `mod.rs` because every sub-module
calls them or operates on the struct they build.
| Item | Lines | Destination | Notes |
|------|-------|-------------|-------|
| `fn data_dir_from_env` | 304-313 | `sqlite/mod.rs` | private helper |
| `fn expand_tilde` | 314-332 | `sqlite/mod.rs` | private helper |
| `fn prepare_data_dir` | 333-346 | `sqlite/mod.rs` | private helper |
| `pub fn db_path_for_data_dir` | 347-355 | `sqlite/mod.rs` | |
| `pub fn default_db_path` | 356-368 | `sqlite/mod.rs` | |
| `fn configure_connection` | 369-396 | `sqlite/mod.rs` | |
| `pub fn new` | 397-457 | `sqlite/mod.rs` | The constructor |
| `pub fn db_path` | 458-462 | `sqlite/mod.rs` | |
| `pub fn data_dir` | 463-467 | `sqlite/mod.rs` | |
| `pub fn sidecar_dir` | 468-473 | `sqlite/mod.rs` | |
| `fn load_embeddings_into_index` | 474-552 | `sqlite/mod.rs` | Called by `new`; touches vector index |
### CRUD: ingest, get, update, delete, purge (-> `sqlite/crud.rs`)
| Item | Lines | Destination | Notes |
|------|-------|-------------|-------|
| `pub fn ingest` | 553-643 | `sqlite/crud.rs` | |
| `pub fn smart_ingest` | 644-864 | `sqlite/crud.rs` | Calls vector search via `self.semantic_search`; cross-module call resolved by impl block being on the same struct |
| `pub fn get_node_embedding` (vector-search) | 865-887 | `sqlite/crud.rs` | embedding read for one node |
| `pub fn get_all_embeddings` (vector-search) | 888-914 | `sqlite/crud.rs` | |
| `pub fn get_node_embedding` (no vector-search stub) | 915-919 | `sqlite/crud.rs` | feature-gated alternative |
| `pub fn update_node_content` | 920-951 | `sqlite/crud.rs` | |
| `fn generate_embedding_for_node` | 952-999 | `sqlite/crud.rs` | private helper; only called by ingest and update_node_content |
| `pub fn get_node` | 1000-1011 | `sqlite/crud.rs` | |
| `fn parse_timestamp` | 1012-1027 | `sqlite/mod.rs` | **Shared helper**: row_to_node uses it, intention/insight rows also parse timestamps. Bump to `pub(super) fn` |
| `fn row_to_node` | 1028-1119 | `sqlite/mod.rs` | **Shared helper**: crud reads single nodes; search.rs builds node lists from rows; scheduling.rs returns nodes for review queue. Bump to `pub(super) fn`. Static method (no `&self`) so a free function in `mod.rs` is fine |
| `pub fn delete_node` | 1842-1869 | `sqlite/crud.rs` | |
| `pub fn purge_node` | 1870-1987 | `sqlite/crud.rs` | |
| `fn node_exists` | 1988-1996 | `sqlite/crud.rs` | static helper, called only by purge |
| `fn record_sync_tombstone` | 1997-2014 | `sqlite/crud.rs` | static helper, called by delete and purge |
| `pub fn get_all_nodes` | 2268-2291 | `sqlite/crud.rs` | bulk read |
| `pub fn get_nodes_by_type_and_tag` | 2292-2342 | `sqlite/crud.rs` | bulk read |
### Search: fts, semantic, hybrid, temporal (-> `sqlite/search.rs`)
| Item | Lines | Destination | Notes |
|------|-------|-------------|-------|
| `pub fn recall` | 1120-1147 | `sqlite/search.rs` | top-level recall path |
| `fn keyword_search` | 1148-1180 | `sqlite/search.rs` | private |
| `pub fn search` | 2015-2043 | `sqlite/search.rs` | |
| `pub fn search_terms` | 2044-2075 | `sqlite/search.rs` | |
| `pub fn concrete_search_filtered` | 2076-2172 | `sqlite/search.rs` | |
| `fn upsert_concrete_result` | 2173-2197 | `sqlite/search.rs` | static helper |
| `fn normalize_literal_query` | 2198-2210 | `sqlite/search.rs` | static helper |
| `fn escape_like` | 2211-2224 | `sqlite/search.rs` | static helper |
| `fn literal_match_score` | 2225-2248 | `sqlite/search.rs` | static helper |
| `fn node_matches_type_filters` | 2249-2267 | `sqlite/search.rs` | static helper |
| `pub fn is_embedding_ready` (both feature variants) | 2343-2354 | `sqlite/search.rs` | both versions move together |
| `pub fn init_embeddings` (both feature variants) | 2355-2367 | `sqlite/search.rs` | both versions move together |
| `fn get_query_embedding` | 2368-2400 | `sqlite/search.rs` | private; uses `query_cache` field |
| `pub fn semantic_search` | 2401-2434 | `sqlite/search.rs` | |
| `pub fn hybrid_search` (feature on) | 2435-2452 | `sqlite/search.rs` | |
| `pub fn hybrid_search_filtered` (feature on) | 2453-2581 | `sqlite/search.rs` | |
| `pub fn hybrid_search` (feature off) | 2582-2593 | `sqlite/search.rs` | feature-gated stub |
| `pub fn hybrid_search_filtered` (feature off) | 2594-2635 | `sqlite/search.rs` | feature-gated stub |
| `fn keyword_search_with_scores` | 2636-2726 | `sqlite/search.rs` | |
| `fn semantic_search_raw` | 2727-2765 | `sqlite/search.rs` | |
| `pub fn generate_embeddings` | 2766-2819 | `sqlite/search.rs` | populates embeddings post hoc |
| `fn embedding_regeneration_candidates` | 2820-2891 | `sqlite/search.rs` | called by generate_embeddings |
| `pub fn query_at_time` | 2892-2933 | `sqlite/search.rs` | temporal query |
| `pub fn query_time_range` | 2934-3005 | `sqlite/search.rs` | temporal query |
| `fn embedding_model_matches_active` (associated fn) | 5652-5671 | `sqlite/search.rs` | static helper for hybrid_search; promoted to `pub(super)` (test references it) |
| `fn embedding_model_supports_matryoshka` | 5672-5677 | `sqlite/search.rs` | static helper |
| `fn embedding_vector_for_active_model` | 5678-5697 | `sqlite/search.rs` | static helper |
| `fn active_embedding_model_like_pattern` | 5698-5713 | `sqlite/search.rs` | static helper |
### Scheduling: FSRS, decay, consolidation, review, promote/demote, suppression, gc, retention (-> `sqlite/scheduling.rs`)
This is the busiest destination file. The grouping rule is: anything that
touches FSRS scheduling fields (`stability`, `difficulty`, `retrievability`,
`reps`, `lapses`, `retention_strength`, `retrieval_strength`) or the
review/decay/consolidation/gc lifecycle lives here.
| Item | Lines | Destination | Notes |
|------|-------|-------------|-------|
| `pub fn mark_reviewed` | 1181-1275 | `sqlite/scheduling.rs` | FSRS state mutation |
| `pub fn strengthen_on_access` | 1276-1344 | `sqlite/scheduling.rs` | |
| `pub fn strengthen_batch_on_access` | 1345-1357 | `sqlite/scheduling.rs` | |
| `pub fn mark_memory_useful` | 1358-1377 | `sqlite/scheduling.rs` | |
| `fn log_access` | 1378-1393 | `sqlite/scheduling.rs` | private |
| `pub fn promote_memory` | 1394-1425 | `sqlite/scheduling.rs` | |
| `pub fn demote_memory` | 1426-1472 | `sqlite/scheduling.rs` | |
| `pub fn suppress_memory` | 1473-1504 | `sqlite/scheduling.rs` | active forgetting |
| `pub fn reverse_suppression` | 1505-1552 | `sqlite/scheduling.rs` | |
| `pub fn count_suppressed` | 1553-1567 | `sqlite/scheduling.rs` | |
| `pub fn get_recently_suppressed` | 1568-1594 | `sqlite/scheduling.rs` | |
| `pub fn apply_rac1_cascade` | 1595-1641 | `sqlite/scheduling.rs` | active forgetting cascade |
| `pub fn run_rac1_cascade_sweep` | 1642-1657 | `sqlite/scheduling.rs` | |
| `pub fn get_review_queue` | 1658-1681 | `sqlite/scheduling.rs` | |
| `pub fn preview_review` | 1682-1712 | `sqlite/scheduling.rs` | |
| `pub fn get_stats` | 1713-1841 | `sqlite/scheduling.rs` | reports retention/lapses/review counts; lives here for symmetry with the FSRS reporters next door |
| `pub fn apply_decay` | 3006-3095 | `sqlite/scheduling.rs` | core decay loop |
| `fn get_fsrs_w20` | 3096-3119 | `sqlite/scheduling.rs` | |
| `pub fn run_consolidation` | 3120-3407 | `sqlite/scheduling.rs` | |
| `fn auto_dedup_consolidation` | 3408-3538 | `sqlite/scheduling.rs` | called by run_consolidation |
| `fn compute_act_r_activations` | 3539-3605 | `sqlite/scheduling.rs` | called by run_consolidation |
| `fn prune_access_log` | 3606-3620 | `sqlite/scheduling.rs` | called by run_consolidation |
| `fn optimize_w20_if_ready` | 3621-3720 | `sqlite/scheduling.rs` | called by run_consolidation |
| `fn generate_missing_embeddings` | 3721-3740 | `sqlite/scheduling.rs` | called by run_consolidation |
| `pub fn get_state_transitions` | 5714-5748 | `sqlite/scheduling.rs` | audit trail tied to scheduling state |
| `pub fn get_avg_retention` | 5780-5792 | `sqlite/scheduling.rs` | |
| `pub fn get_retention_distribution` | 5794-5825 | `sqlite/scheduling.rs` | |
| `pub fn get_retention_trend` | 5826-5858 | `sqlite/scheduling.rs` | |
| `pub fn save_retention_snapshot` | 5859-5878 | `sqlite/scheduling.rs` | |
| `pub fn count_memories_below_retention` | 5879-5892 | `sqlite/scheduling.rs` | |
| `pub fn gc_below_retention` | 5893-5936 | `sqlite/scheduling.rs` | |
| `pub fn auto_promote_frequent_access` | 5937-5985 | `sqlite/scheduling.rs` | |
| `pub fn set_waking_tag` | 5986-5998 | `sqlite/scheduling.rs` | waking SWR tagging |
| `pub fn clear_waking_tags` | 5999-6011 | `sqlite/scheduling.rs` | |
| `pub fn get_waking_tagged_memories` | 6012-6028 | `sqlite/scheduling.rs` | |
| `pub fn get_recent_state_transitions` | 6105-6132 | `sqlite/scheduling.rs` | |
### Graph: edges (memory_connections), neighbors, subgraph (-> `sqlite/graph.rs`)
| Item | Lines | Destination | Notes |
|------|-------|-------------|-------|
| `pub fn save_connection` | 4180-4202 | `sqlite/graph.rs` | |
| `pub fn get_connections_for_memory` | 4203-4220 | `sqlite/graph.rs` | |
| `pub fn get_all_connections` | 4221-4236 | `sqlite/graph.rs` | |
| `pub fn strengthen_connection` | 4237-4259 | `sqlite/graph.rs` | |
| `pub fn apply_connection_decay` | 4260-4272 | `sqlite/graph.rs` | |
| `pub fn prune_weak_connections` | 4273-4284 | `sqlite/graph.rs` | |
| `fn row_to_connection` | 4285-4305 | `sqlite/graph.rs` | private |
| `pub fn get_most_connected_memory` | 6029-6046 | `sqlite/graph.rs` | |
| `pub fn get_memory_subgraph` | 6048-6103 | `sqlite/graph.rs` | calls `get_connections_for_memory`, `get_node`, `get_all_connections` -- all resolvable through `self` |
### Domain (-> `sqlite/domain.rs`)
Phase 1 keeps domain logic to JSON-column reads + classify stub. Phase 4
expands this file. Keeping the file in the split so Phase 4 has an
obvious place to add to.
| Item | Lines | Destination | Notes |
|------|-------|-------------|-------|
| `fn read_domain_columns` | 6167-6196 | `sqlite/domain.rs` | private helper used by trait `get`. Bump to `pub(super)` |
The trait methods `list_domains` / `get_domain` / `upsert_domain` /
`delete_domain` / `classify` live in `sqlite/trait_impl.rs`; they
delegate to thin helpers that, in Phase 1, are essentially noops or
JSON reads. Phase 4 will move the substance of those methods into
`sqlite/domain.rs` as real functions.
### Registry: embedding_model table (-> `sqlite/registry.rs`)
| Item | Lines | Destination | Notes |
|------|-------|-------------|-------|
| `fn enforce_model` | 6203-6272 | `sqlite/registry.rs` | private helper used by trait `insert` and `update`. Bump to `pub(super)` |
The trait methods `registered_model` and `register_model` themselves
live in `sqlite/trait_impl.rs`. Phase 2's `postgres/registry.rs` will
mirror this layout.
### Intentions, Insights, Memory States, Consolidation History, Dream History, Backup (-> `sqlite/mod.rs`)
These were tacked onto `SqliteMemoryStore` over time as the cognitive
modules needed somewhere to persist their state. They are not part of the
trait surface, they are not naturally grouped with crud/search/scheduling,
and they are each fairly small and self-contained. They live in `mod.rs`
under labelled sections (one big impl block can carry them) rather than
inventing extra files like `intentions.rs` and `insights.rs`. Postgres
will mirror this once Phase 5 picks up the work; for now they have no
peer.
Rationale: every one of these methods writes to a single table, the
bodies are short, and grouping them next to the constructor preserves
"open `mod.rs` to see the whole struct" as the navigation default.
| Item | Lines | Destination | Notes |
|------|-------|-------------|-------|
| `IntentionRecord` struct | 3747-3766 | `sqlite/mod.rs` | re-exported through `storage/mod.rs` |
| `InsightRecord` struct + `Default` | 3767-3797 | `sqlite/mod.rs` | re-exported |
| `ConnectionRecord` struct | 3799-3809 | `sqlite/mod.rs` | re-exported; consumed by `graph.rs` |
| `MemoryStateRecord` struct | 3811-3821 | `sqlite/mod.rs` | |
| `StateTransitionRecord` struct | 3823-3833 | `sqlite/mod.rs` | re-exported |
| `ConsolidationHistoryRecord` struct | 3835-3846 | `sqlite/mod.rs` | |
| `DreamHistoryRecord` struct | 3848-3866 | `sqlite/mod.rs` | re-exported |
| `pub fn save_intention` etc. (intention block) | 3874-4058 | `sqlite/mod.rs` | one impl block, in-section labelled |
| `fn row_to_intention` | 4023-4058 | `sqlite/mod.rs` | private |
| insights block (`save_insight`, `get_insights`, etc.) | 4065-4174 | `sqlite/mod.rs` | |
| `fn row_to_insight` | 4153-4173 | `sqlite/mod.rs` | private |
| memory-state block | 4306-4459 | `sqlite/mod.rs` | |
| `fn row_to_memory_state` | 4431-4459 | `sqlite/mod.rs` | private |
| consolidation-history block | 4465-4540 | `sqlite/mod.rs` | |
| dream-history block | 4546-4638 | `sqlite/mod.rs` | |
| `pub fn count_memories_since` | 4639-4651 | `sqlite/mod.rs` | |
| `fn scan_last_backup_timestamp` | 4652-4682 | `sqlite/mod.rs` | |
| `pub fn last_backup_timestamp` | 4683-4688 | `sqlite/mod.rs` | |
| `pub fn get_last_backup_timestamp` (associated) | 4689-4696 | `sqlite/mod.rs` | |
| `pub fn backup_to` | 5749-5774 | `sqlite/mod.rs` | sqlite VACUUM INTO; called by backup tool |
### Portable export/import/sync (-> `sqlite/portable_sync.rs`)
This is the second-largest destination after `scheduling.rs` and the most
self-contained.
| Item | Lines | Destination | Notes |
|------|-------|-------------|-------|
| `pub fn export_portable_archive` | 4699-4755 | `sqlite/portable_sync.rs` | |
| `pub fn export_portable_archive_to_path` | 4756-4806 | `sqlite/portable_sync.rs` | |
| `pub fn import_portable_archive` | 4807-4978 | `sqlite/portable_sync.rs` | |
| `pub fn import_portable_archive_from_path` | 4979-4996 | `sqlite/portable_sync.rs` | |
| `pub fn sync_portable_archive` (generic over backend) | 4997-5025 | `sqlite/portable_sync.rs` | |
| `pub fn sync_portable_archive_file` | 5026-5030 | `sqlite/portable_sync.rs` | |
| `fn merge_portable_table` | 5031-5073 | `sqlite/portable_sync.rs` | |
| `fn merge_knowledge_nodes` | 5074-5126 | `sqlite/portable_sync.rs` | |
| `fn merge_sync_tombstones` | 5127-5204 | `sqlite/portable_sync.rs` | |
| `fn merge_deletion_tombstones` | 5205-5245 | `sqlite/portable_sync.rs` | |
| `fn merge_keyed_table` | 5246-5281 | `sqlite/portable_sync.rs` | |
| `fn row_references_locally_newer_node` | 5282-5302 | `sqlite/portable_sync.rs` | |
| `fn merge_append_only_table` | 5303-5338 | `sqlite/portable_sync.rs` | |
| `fn parent_rows_exist` | 5339-5370 | `sqlite/portable_sync.rs` | |
| `fn insert_or_replace_row` | 5371-5386 | `sqlite/portable_sync.rs` | |
| `fn merge_key_columns` | 5387-5398 | `sqlite/portable_sync.rs` | |
| `fn upsert_row_with_columns` | 5399-5446 | `sqlite/portable_sync.rs` | |
| `fn insert_row_with_columns` | 5447-5469 | `sqlite/portable_sync.rs` | |
| `fn merge_row_exists` | 5470-5487 | `sqlite/portable_sync.rs` | |
| `fn row_exists_by_values` | 5488-5507 | `sqlite/portable_sync.rs` | |
| `fn row_values_for_columns` | 5508-5528 | `sqlite/portable_sync.rs` | |
| `fn portable_value` | 5529-5540 | `sqlite/portable_sync.rs` | |
| `fn portable_text` | 5541-5551 | `sqlite/portable_sync.rs` | |
| `fn portable_timestamp` | 5552-5559 | `sqlite/portable_sync.rs` | |
| `fn parse_rfc3339_opt` | 5560-5565 | `sqlite/portable_sync.rs` | |
| `fn tombstone_timestamp` | 5566-5580 | `sqlite/portable_sync.rs` | |
| `fn current_schema_version` | 5581-5589 | `sqlite/portable_sync.rs` | static helper |
| `fn ensure_portable_import_target_empty` | 5590-5604 | `sqlite/portable_sync.rs` | static helper |
| `fn table_exists` | 5605-5613 | `sqlite/portable_sync.rs` | static helper |
| `fn table_row_count` | 5614-5618 | `sqlite/portable_sync.rs` | static helper |
| `fn table_columns` | 5619-5630 | `sqlite/portable_sync.rs` | static helper |
| `fn portable_value_from_ref` | 5631-5646 | `sqlite/portable_sync.rs` | static helper |
| `fn quote_ident` | 5647-5651 | `sqlite/portable_sync.rs` | static helper |
### Trait helpers and trait impl (-> `sqlite/trait_impl.rs`)
| Item | Lines | Destination | Notes |
|------|-------|-------------|-------|
| `fn node_to_record` | 6142-6164 | `sqlite/trait_impl.rs` | associated fn used only by trait body; co-locate |
| `impl LocalMemoryStore for SqliteMemoryStore` block | 6274-7110 | `sqlite/trait_impl.rs` | full trait impl; attribute changes from `#[async_trait::async_trait]` to whatever 0001a settles on (`#[trait_variant::make(...)]` is on the trait declaration; the impl block carries no attribute under trait_variant) |
### Tests
The current `#[cfg(test)] mod tests` block at lines 7112-8713 contains
**two** distinct test families:
1. **Native API tests** (7120-8198): unit tests against the legacy
`pub fn` surface (`test_ingest_and_get`, `test_search`, `test_review`,
`test_delete`, `test_dream_history_save_and_get_last`,
`test_portable_archive_exact_round_trip`, `test_keyword_search_*`,
`test_concrete_search_*`, `test_purge_*`, etc.).
2. **Trait round-trip tests** (8200-8712, after the
`// ===== Phase 1: LocalMemoryStore trait round-trip tests =====`
banner): `trait_init_is_idempotent`, `trait_register_model_*`,
`trait_insert_*`, `trait_get_*`, `trait_update_*`, `trait_delete_*`,
`trait_fts_search_*`, `trait_hybrid_search_*`,
`trait_scheduling_*`, `trait_add_edge_*`, `trait_get_edges_*`,
`trait_remove_edge_*`, `trait_get_neighbors_*`, `trait_list_domains_*`,
`trait_upsert_*`, `trait_classify_*`, `trait_count_*`,
`trait_get_stats_*`, `trait_vacuum_*`,
`trait_insert_refuses_dimension_mismatch`.
See the Test Relocation section below for the resolution.
---
## Visibility Changes
The split moves items into sibling files inside one module. Helpers that
were `fn ...` (i.e. crate-private but file-private under the current
layout, since the file *is* the module) need their visibility lifted
just enough that sibling files can call them. The principle is: smallest
bump that makes the call site compile.
`pub(super)` is sufficient for everything below; nothing needs
`pub(crate)`. The trait `LocalMemoryStore` exposure does not change --
sub-modules call `self.method(...)` on `SqliteMemoryStore`, which
resolves through the impl blocks defined in their own files; visibility
is automatic at impl-block scope.
Items that need a visibility bump (currently private fn, become
`pub(super) fn`):
- `parse_timestamp` (1012): called by `row_to_node` and by intention /
insight row mappers.
- `row_to_node` (1028): called by `crud.rs`, `search.rs`,
`scheduling.rs`. Static associated fn.
- `read_domain_columns` (6167): called by `trait_impl.rs`.
- `enforce_model` (6203): called by `trait_impl.rs`.
- `embedding_model_matches_active` (5652): currently called by
`hybrid_search_filtered`; tests also reference it. Has to remain
`pub(super) fn` and be `pub` only if the existing test names reach it
through a re-export. (See Test Relocation.)
- `embedding_model_supports_matryoshka` (5672): private; only callers in
`search.rs` after the move; stays `fn` (no bump needed).
- `embedding_vector_for_active_model` (5678): same as the matches
function -- a test references it. Bump to `pub(super)`.
- `active_embedding_model_like_pattern` (5698): private; only used by
search; stays `fn`.
- `generate_embedding_for_node` (952): currently called by `ingest` and
`update_node_content`. Both move to `crud.rs`; stays `fn`.
- `get_query_embedding` (2368): only used inside `search.rs`; stays `fn`.
- `keyword_search_with_scores` (2636): only used inside `search.rs`;
stays `fn`.
- `semantic_search_raw` (2727): only used inside `search.rs`; stays `fn`.
- `embedding_regeneration_candidates` (2820): used by
`generate_embeddings`; both move to `search.rs`; stays `fn`. The
existing test (line 7167) references it through `storage.method()`,
which will continue to work because the test file can move with it.
- `log_access` (1378): private to `scheduling.rs`; stays `fn`.
- All the `auto_dedup_consolidation` / `compute_act_r_activations` /
`prune_access_log` / `optimize_w20_if_ready` /
`generate_missing_embeddings` helpers (3408-3740): private to
`scheduling.rs`; stays `fn`.
- `row_to_intention` / `row_to_insight` / `row_to_memory_state` /
`row_to_connection`: all stay private in their destination file (only
one caller each).
- All `merge_*` / `portable_*` / `parse_rfc3339_opt` / `quote_ident`:
private to `portable_sync.rs`; stays `fn`.
- `node_exists` (1988): private to `crud.rs`; stays `fn`.
- `record_sync_tombstone` (1997): private to `crud.rs`; stays `fn`.
- `get_fsrs_w20` (3096): private to `scheduling.rs`; stays `fn`.
Items already `pub fn` (or `pub(crate) fn`) stay as they are -- no
visibility regression.
Field visibility on `SqliteMemoryStore` itself: currently all fields are
private. The sub-modules access them via `self.field`. Because impl
blocks for `SqliteMemoryStore` are written in sibling files of the same
module, `self.field` reaches private fields without a visibility bump.
**No field visibility changes are required.** Confirm this during the
first motion commit; if Rust disagrees, mark the relevant fields
`pub(super)` and document in the commit message.
---
## Public Re-exports
`crates/vestige-core/src/storage/mod.rs` currently exports:
```rust
mod memory_store;
mod migrations;
mod portable;
mod sqlite;
pub use memory_store::{...};
pub use migrations::MIGRATIONS;
pub use portable::{...};
pub use sqlite::{
ConnectionRecord, ConsolidationHistoryRecord, DreamHistoryRecord, FilePortableSyncBackend,
InsightRecord, IntentionRecord, PortableSyncBackend, PortableSyncReport, Result,
SmartIngestResult, SqliteMemoryStore, StateTransitionRecord, StorageError,
};
pub type Storage = SqliteMemoryStore;
```
After the split, `mod sqlite;` resolves to the new directory module
(`storage/sqlite/mod.rs`). The `pub use sqlite::{...}` block resolves
against the items re-exported by `storage/sqlite/mod.rs`.
`storage/sqlite/mod.rs` therefore needs the same names visible at its
top level. Add at the end of `mod.rs`:
```rust
mod crud;
mod search;
mod scheduling;
mod graph;
mod domain;
mod registry;
mod portable_sync;
mod trait_impl;
pub use portable_sync::{FilePortableSyncBackend, PortableSyncBackend, PortableSyncReport};
// SqliteMemoryStore, StorageError, Result, SmartIngestResult, IntentionRecord,
// InsightRecord, ConnectionRecord, StateTransitionRecord,
// ConsolidationHistoryRecord, DreamHistoryRecord are defined in mod.rs itself,
// so they are already in scope and do not need a re-export.
```
The `crates/vestige-core/src/storage/mod.rs` file does not change. The
`pub type Storage = SqliteMemoryStore;` alias keeps working.
If `cargo build` complains that `storage/mod.rs` cannot resolve a name
in its `pub use sqlite::{...}` block, the fix is to add the missing name
to `sqlite/mod.rs`'s re-export tail; no change to `storage/mod.rs`.
---
## Test Relocation
Two test families, two destinations.
**Native API tests** (current lines 7120-8198) cover the legacy `pub fn`
surface. They live close to their subject:
- Tests that touch the constructor, common helpers, and shared setup
(`create_test_storage`, `create_test_storage_at`,
`test_storage_creation`, `test_get_last_backup_timestamp_no_panic`)
move to `sqlite/mod.rs` in a `#[cfg(test)] mod tests` block.
- `test_ingest_and_get`, `test_delete`,
`test_purge_scrubs_insight_json_orphans_children_and_writes_tombstone`
go to `sqlite/crud.rs` as a `#[cfg(test)] mod tests` block.
- `test_search`, `test_keyword_search_with_include_types`,
`test_keyword_search_with_exclude_types`,
`test_include_types_takes_precedence_over_exclude`,
`test_type_filter_with_no_matches_returns_empty`,
`test_hybrid_search_backward_compat`,
`test_concrete_search_literal_identifier_lands_first`,
`test_embedding_model_family_matching`,
`test_embedding_regeneration_candidates_include_entire_mismatched_corpus`
go to `sqlite/search.rs`.
- `test_review` goes to `sqlite/scheduling.rs`.
- `test_dream_history_save_and_get_last`, `test_dream_history_empty`,
`test_count_memories_since` go to `sqlite/mod.rs` (history tables live
there).
- All `test_portable_*` go to `sqlite/portable_sync.rs`.
- `test_file_portable_sync_round_trips_between_devices` goes to
`sqlite/portable_sync.rs`.
**Trait round-trip tests** (current lines 8200-8712) test the
`LocalMemoryStore` trait impl. Two viable layouts:
A. Co-locate with the impl in `sqlite/trait_impl.rs` (one big
`#[cfg(test)] mod trait_tests`).
B. Keep them as a single `tests.rs` file in the sqlite directory.
**Decision: A.** Co-locate. The trait round-trip tests are explicitly
testing the `impl LocalMemoryStore for SqliteMemoryStore` block;
co-location means a reader who edits the trait impl sees its tests in
the same file. Option B would mean two places to look every time a
trait method changes shape. For an 8K-line collapse the tradeoff
favours co-location.
Concretely: `sqlite/trait_impl.rs` ends with a
`#[cfg(test)] mod trait_tests { ... }` block that contains all 30+
`trait_*` tests, plus the shared `make_record`, `rt`, and small helpers
defined inside the current test mod for trait tests (lines 8208-8226).
---
## Commit Sequence
Each commit moves one logical group. After each commit:
```
cargo build -p vestige-core
cargo test -p vestige-core
cargo clippy -p vestige-core -- -D warnings
```
must pass. Order is chosen so that each move is small, the next move
does not depend on the previous having grown surprising visibility, and
the largest / riskiest move (the trait impl, with the new
trait_variant attribute) lands last.
| # | Commit | What moves | Tests touched |
|---|--------|-----------|----------------|
| 1 | `refactor(sqlite): scaffold sqlite/ directory` | Convert `sqlite.rs` -> `sqlite/mod.rs` verbatim (rename + create empty sibling files `crud.rs`, `search.rs`, `scheduling.rs`, `graph.rs`, `domain.rs`, `registry.rs`, `portable_sync.rs`, `trait_impl.rs` each with `use super::*;`). At this point `mod.rs` declares the new modules but they are empty. | None move. Build proves the rename works. |
| 2 | `refactor(sqlite): split out portable sync` | Move all `merge_*`, `portable_*`, `export_*`, `import_*`, `sync_*` items + `MergeWrite`, `PortableSyncBackend`, `FilePortableSyncBackend`, `PortableSyncReport`, `PortableMergeState`, `PORTABLE_TABLES`, `PORTABLE_USER_DATA_TABLES`, helper statics into `sqlite/portable_sync.rs`. Add `pub use portable_sync::{...}` in `mod.rs` for the public types. | `test_portable_*` and `test_file_portable_sync_round_trips_between_devices` move too. |
| 3 | `refactor(sqlite): split out graph / connections` | Move `save_connection`, `get_connections_for_memory`, `get_all_connections`, `strengthen_connection`, `apply_connection_decay`, `prune_weak_connections`, `row_to_connection`, `get_most_connected_memory`, `get_memory_subgraph` to `sqlite/graph.rs`. | None move (no native graph tests; trait edge tests still in trait_tests). |
| 4 | `refactor(sqlite): split out scheduling / fsrs / consolidation` | Move all items listed in the Scheduling row to `sqlite/scheduling.rs`. | `test_review` moves. |
| 5 | `refactor(sqlite): split out search / fts / semantic / hybrid` | Move all items listed in the Search row to `sqlite/search.rs`. Add `pub(super)` to the four `embedding_model_*` helpers that tests reference. | `test_search`, `test_keyword_search_*`, `test_include_types_*`, `test_type_filter_*`, `test_hybrid_search_*`, `test_concrete_search_*`, `test_embedding_model_family_matching`, `test_embedding_regeneration_candidates_include_entire_mismatched_corpus` move. |
| 6 | `refactor(sqlite): split out crud / ingest / get / update / delete / purge` | Move `ingest`, `smart_ingest`, `get_node`, `update_node_content`, `delete_node`, `purge_node`, `get_all_nodes`, `get_nodes_by_type_and_tag`, `node_exists`, `record_sync_tombstone`, `generate_embedding_for_node`, `get_node_embedding`, `get_all_embeddings`, `PurgeReport` to `sqlite/crud.rs`. Bump `row_to_node` and `parse_timestamp` to `pub(super) fn` in `mod.rs`. | `test_ingest_and_get`, `test_delete`, `test_purge_scrubs_insight_json_orphans_children_and_writes_tombstone` move. |
| 7 | `refactor(sqlite): split out registry helper` | Move `enforce_model` to `sqlite/registry.rs`, bumped to `pub(super)`. | None move. |
| 8 | `refactor(sqlite): split out domain helper` | Move `read_domain_columns` to `sqlite/domain.rs`, bumped to `pub(super)`. | None move. |
| 9 | `refactor(sqlite): split out trait impl + tests` | Move `node_to_record` and the full `impl LocalMemoryStore for SqliteMemoryStore` block to `sqlite/trait_impl.rs`. Move the entire trait round-trip test module (lines 8200-8712, including `make_record` and `rt` helpers) to a `#[cfg(test)] mod trait_tests` block at the bottom of `trait_impl.rs`. This is the commit where the trait_variant attribute (from sub-plan 0001a) is observed: the impl block on `SqliteMemoryStore` uses whatever syntax the rewritten trait expects (no `#[async_trait::async_trait]`). | All `trait_*` tests move. |
Commit 1 is the only commit that creates new files; the rest move
existing code into them. Reviewers can bisect through this list to
find any silent-semantic change.
---
## Verification
Run after every commit. All three must pass before pushing:
```
cargo build -p vestige-core
cargo test -p vestige-core
cargo clippy -p vestige-core -- -D warnings
```
The Phase 1 amendment branch must also build with the no-default-features
configuration that the release binary uses for the alternative feature
set:
```
cargo build -p vestige-core --no-default-features
cargo test -p vestige-core --no-default-features
```
Some of the methods being moved (`get_node_embedding`, `is_embedding_ready`,
`init_embeddings`, the feature-on/feature-off `hybrid_search` pair) have
two definitions guarded by feature flags. The split must preserve both
copies in the same destination file with their existing `#[cfg(...)]`
attributes; the no-default-features build confirms.
After the last commit, run the workspace-wide check that Phase 1 promised:
```
cargo build --workspace
cargo test --workspace
```
This catches downstream consumers (`vestige-mcp`, `vestige`,
`vestige-restore`) that might depend on a specific module path (they
should not -- they import from `crate::storage::SqliteMemoryStore` and
the re-exports in `storage/mod.rs` -- but the workspace build is the
authoritative confirmation).
---
## Acceptance Criteria
1. `crates/vestige-core/src/storage/sqlite.rs` no longer exists. In its
place is `crates/vestige-core/src/storage/sqlite/` with the eight
files listed in the Target Layout section, each below 2000 lines.
2. `crates/vestige-core/src/storage/mod.rs` is unchanged (or
functionally unchanged -- the `pub use sqlite::{...}` block contains
the same identifiers in the same order).
3. Every cognitive module and binary in the workspace
(`vestige-core`, `vestige-mcp`, `vestige`, `vestige-restore`)
compiles with no source edits other than the ones in
`crates/vestige-core/src/storage/sqlite/`.
4. `cargo build -p vestige-core`,
`cargo test -p vestige-core`,
`cargo clippy -p vestige-core -- -D warnings`,
`cargo build -p vestige-core --no-default-features`, and
`cargo test -p vestige-core --no-default-features` all pass at the
end of every commit in the sequence (bisectability).
5. `cargo test --workspace` matches the Phase 1 baseline test count
(758 tests, of which 352 are in `vestige-core`). No new tests are
added by this sub-plan; no existing test is renamed or deleted.
6. The on-disk SQLite schema is unchanged. A live database created on
the pre-split build opens cleanly on the post-split build and round
trips a memory.
7. `git log --follow` works for at least one method in each destination
file (i.e. `git mv` was used where the line range constitutes most
of the file content of the destination, otherwise a `git log -p`
on the new file shows the history before the rename through the
block-move detection that recent `git log` versions support).
8. No public symbol disappears from `crate::storage::*`. A reviewer can
verify with:
```
cargo doc -p vestige-core --no-deps
```
before and after the split, and `diff` the generated
`target/doc/vestige_core/storage/index.html` lists.
---
## Non-Goals (explicit)
- No public API change. The trait surface (`LocalMemoryStore`,
`MemoryStore`), the legacy `pub fn` surface on `SqliteMemoryStore`,
the re-exports through `storage/mod.rs`, and the `pub type Storage =
SqliteMemoryStore;` alias are all preserved.
- No behavioural change. No SQL is rewritten, no FSRS parameter is
retuned, no embedding model is touched, no migration is added.
- No new tests. Tests move with their subject; no new tests are
written.
- No clippy fix-ups that pre-date this sub-plan. If `cargo clippy
-- -D warnings` was passing before the split, it must continue to
pass; if it was not passing, the failures stay where they are and
are addressed in a separate commit (out of scope here).
- No removal of the `pub type Storage = SqliteMemoryStore;` BC alias.
That happens at the end of Phase 4 per ADR 0001.
- No reorganisation of `storage/memory_store.rs`,
`storage/migrations.rs`, or `storage/portable.rs`. Those files are
out of scope; the split is private to `storage/sqlite/`.
---
## Risks and Mitigations
| Risk | Mitigation |
|------|------------|
| Silent semantic change introduced by a motion commit | Per-commit `cargo test -p vestige-core` keeps the bisect window to a single commit. Reviewer bisects with `cargo test -p vestige-core` as the witness. |
| Sibling-file `self.field` accesses fail because Rust enforces module visibility on tuple-struct or named fields | Confirmed: associated impl blocks in sibling files of the same `mod sqlite` reach private fields. If the compiler disagrees, bump the affected fields to `pub(super)` in `mod.rs` and note the bump in the commit message. |
| Test-only helpers (`create_test_storage`, `make_record`, `rt`) get duplicated across test modules | Lift them once into a `#[cfg(test)] mod test_support { ... }` sub-module in `sqlite/mod.rs` and re-export with `pub(super) use`. Do this in commit 1 (scaffold), not later. |
| `#[cfg(all(feature = "embeddings", feature = "vector-search"))]` items end up in `mod.rs` where they pollute the shared layer | Audit during commit 5 (search split); items behind both feature flags belong in `search.rs`. The `query_cache` field stays in `mod.rs` because the struct definition is there; the field declaration is feature-gated and that gate moves with the struct as-is. |
| `git log --follow` blame chains break on the moved methods | Use `git mv sqlite.rs sqlite/mod.rs` in commit 1 so commit 1 looks like a rename (`git log --follow` keeps working). Subsequent commits are content moves inside the module; modern `git log --follow -M -C` heuristics still trace the lines. Reviewers who need pristine blame should bisect to before commit 1. |
| Sub-plan 0001a (trait rewrite) has not landed when this work starts | Block: do not start commits 1-9 until 0001a is on the same branch (`feat/storage-trait-phase1`) and tests pass. `trait_impl.rs` lands the new attribute in commit 9; if 0001a is not in, commit 9 fails. |
---
## Self-Contained Brief (for /goal)
A fresh Claude Code session can execute this sub-plan by:
1. Reading this file end to end.
2. Reading `crates/vestige-core/src/storage/sqlite.rs` (the file to be
split) in full, using line ranges from the Mapping Table to confirm
the current shape matches the brief.
3. Reading `crates/vestige-core/src/storage/mod.rs` (the re-export
surface that must continue to work).
4. Reading `crates/vestige-core/src/storage/memory_store.rs` (the
trait surface that `trait_impl.rs` implements).
5. Confirming sub-plan 0001a has landed on the current branch by
checking that `memory_store.rs` no longer carries
`#[async_trait::async_trait]` on the trait declaration.
6. Working through the Commit Sequence in order, running the
Verification commands after each commit.
The session does not need to read ADR 0002 or the master Phase 2 plan
to do this work. The split is purely mechanical relative to the
mapping table above.

847
docs/plans/0001c-async-trait-sunset.md
View file

@ -1,847 +0,0 @@
# Sub-Plan 0001c: Sunset the `async-trait` crate dependency
**Status**: Draft
**Branch**: `feat/storage-trait-phase1` (Phase 1 amendment, PR A)
**Depends on**:
- `0001a-trait-rewrite.md` (rewrites `MemoryStore` / `LocalMemoryStore` and
the SQLite impl; lands first)
- `0001b-sqlite-split.md` (moves `sqlite.rs` into `sqlite/`; lands second)
**Related**: `docs/adr/0002-phase-2-execution.md` (decision D1 closing line:
"async-trait dependency stays in Cargo.toml only if other code uses it;
otherwise removed"). This sub-plan operationalises the removal.
---
## Context
This is the third and final Phase 1 amendment sub-plan. Sub-plan `0001a`
rewrote `MemoryStore` / `LocalMemoryStore` using
`#[trait_variant::make(MemoryStore: Send)]` and dropped the
`#[async_trait::async_trait]` attribute from the SQLite impl block.
Sub-plan `0001b` then split `sqlite.rs` into a `sqlite/` directory; the
trait impl now lives in `sqlite/trait_impl.rs`. After `0001a` lands, the
only remaining `async_trait` usage in the workspace is the embedder pair
(`embedder/mod.rs` declares the trait; `embedder/fastembed.rs` implements
it). This sub-plan rewrites those two files following the exact pattern
from `0001a`, then removes `async-trait = "0.1"` from
`crates/vestige-core/Cargo.toml`. End state: zero `async_trait` references
anywhere under `crates/`, zero direct dependency on the `async-trait`
crate, workspace tests and clippy green.
The rewrite is mechanically tiny -- one trait declaration, one impl block,
one Cargo.toml line -- but it is gated behind `0001a` and `0001b` so the
trait-rewrite pattern is already settled and so the SQLite trait impl
attribute has already been dropped. Doing the embedder rewrite without
that pre-work would leave the `async-trait` dep behind for the SQLite
side and force the Cargo.toml deletion into a separate commit later.
---
## Scope
### In scope
- Rewrite `LocalEmbedder` declaration in
`crates/vestige-core/src/embedder/mod.rs` to use
`#[trait_variant::make(Embedder: Send)] pub trait LocalEmbedder`.
- Delete the `pub use LocalEmbedder as Embedder;` alias from the same file.
The `Embedder` symbol becomes the trait that `trait_variant::make` emits
at the same module path, so the existing
`pub use embedder::{Embedder, ..., LocalEmbedder};` line in
`crates/vestige-core/src/lib.rs:167` keeps working untouched.
- Drop the `#[async_trait::async_trait]` attribute from the
`FastembedEmbedder` impl block in
`crates/vestige-core/src/embedder/fastembed.rs`.
- Update doc comments on the trait declaration to describe
`trait_variant` rather than `async_trait`.
- Remove `async-trait = "0.1"` from
`crates/vestige-core/Cargo.toml` (line 119 area). Use
`cargo rm async-trait` from inside the crate directory.
- Verify with `grep -rn "async_trait" crates/` returning zero hits.
### Out of scope
- Any change to the `MemoryStore` trait or `SqliteMemoryStore` impl;
those were handled by `0001a`.
- Any file moves in `embedder/` (no parallel of `0001b` is required;
`embedder/` already has the `mod.rs` + `fastembed.rs` shape).
- Touching `crates/vestige-mcp/` or any cognitive module. None of them
hold `Arc<dyn Embedder>` or `Box<dyn Embedder>` in production.
- Renaming the `Embedder` / `LocalEmbedder` symbols or changing the
re-exports in `crates/vestige-core/src/lib.rs`.
---
## Prerequisites
### State assumed at start
- `0001a` is merged onto the branch. After `0001a`:
- `crates/vestige-core/src/storage/memory_store.rs` declares
`#[trait_variant::make(MemoryStore: Send)] pub trait LocalMemoryStore`.
- The SQLite impl block has no `#[async_trait::async_trait]` attribute.
- `grep -rn async_trait crates/` returns exactly three hits, all in
`crates/vestige-core/src/embedder/` (two in `mod.rs`, one in
`fastembed.rs`), and one Cargo.toml hit.
- `0001b` is merged onto the branch. After `0001b`:
- `crates/vestige-core/src/storage/sqlite.rs` no longer exists as a
single file; the impl lives in `crates/vestige-core/src/storage/sqlite/trait_impl.rs`.
- The embedder files are untouched.
### Required crates
| Crate | Version | Action |
|----------------|---------|-----------------------------------------------------------------|
| `trait-variant`| `0.1` | Already declared (line 117 of Cargo.toml). Verify present. |
| `async-trait` | `0.1` | Remove. Only the two embedder files still use it after `0001a`. |
### Workspace-wide audit before starting
Run from `/home/delandtj/prppl/vestige-phase2/` (or the equivalent
worktree where this sub-plan executes):
```bash
grep -rn "async_trait\|async-trait" crates/ tests/
```
Expected hits before this sub-plan starts (after `0001a` + `0001b`):
```
crates/vestige-core/Cargo.toml:119:async-trait = "0.1"
crates/vestige-core/src/embedder/mod.rs:24:/// `#[async_trait::async_trait]` makes every `async fn` return a
crates/vestige-core/src/embedder/mod.rs:27:#[async_trait::async_trait]
crates/vestige-core/src/embedder/mod.rs:56:/// Both names refer to the same `async_trait`-annotated trait.
crates/vestige-core/src/embedder/fastembed.rs:44:#[async_trait::async_trait]
```
Five hits across two source files and one Cargo.toml. After this sub-plan,
the same grep must return zero hits.
```bash
grep -rn "async-trait\|async_trait" --include="Cargo.toml" crates/
```
Expected: exactly one hit (`crates/vestige-core/Cargo.toml:119`). No other
workspace crate declares `async-trait` as a direct dependency. This is
the precondition that lets us delete the line cleanly.
---
## Files Touched
### Trait declaration (vestige-core)
| File | Lines (approx) | Change |
|-------------------------------------------------|----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `crates/vestige-core/src/embedder/mod.rs` | 21-57 | Replace `#[async_trait::async_trait] pub trait LocalEmbedder: Send + Sync + 'static` with `#[trait_variant::make(Embedder: Send)] pub trait LocalEmbedder: Sync + 'static`. Delete the `pub use LocalEmbedder as Embedder;` alias on line 57. Update doc comments (lines 21-26, 55-56). |
### Impl block (vestige-core)
| File | Lines (approx) | Change |
|-------------------------------------------------|----------------|--------------------------------------------------------------------------------------------------------------|
| `crates/vestige-core/src/embedder/fastembed.rs` | 44 | Delete the `#[async_trait::async_trait]` attribute. Keep the `impl LocalEmbedder for FastembedEmbedder { ... }` body verbatim. No `Box::pin`, no `'async_trait` lifetimes, no manual `Pin<Box<dyn Future>>`. |
### Other Embedder impls
None. `grep -rn "impl.*LocalEmbedder\|impl.*Embedder for" crates/ tests/`
returns exactly one production hit:
`crates/vestige-core/src/embedder/fastembed.rs:45: impl LocalEmbedder for FastembedEmbedder`.
There is no test mock implementing `Embedder` in the test tree (the only
test that touches the trait, `tests/phase_1/embedder_trait.rs`, uses the
concrete `FastembedEmbedder` boxed as `Box<dyn Embedder>`).
### Call sites (production)
Verified by:
```bash
grep -rn "dyn Embedder\|dyn LocalEmbedder" crates/ tests/ --include="*.rs"
grep -rn "Box<dyn Embedder>\|Arc<dyn Embedder>" crates/ tests/ --include="*.rs"
grep -rn "use.*Embedder" crates/ tests/ --include="*.rs"
```
Production call sites that may need verification (and the expected change
for each, even though we have already verified that none need an edit):
| File | Use | Required change |
|------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------|-----------------|
| `crates/vestige-core/src/lib.rs:167` | `pub use embedder::{Embedder, EmbedderError, EmbedderResult, FastembedEmbedder, LocalEmbedder};` | None. Both names still exist at `crate::embedder::*` after the rewrite; `Embedder` is now the `trait_variant`-generated trait, `LocalEmbedder` is the source-of-truth trait. The re-export keeps resolving. |
| `crates/vestige-core/src/embedder/fastembed.rs:7` | `use super::{EmbedderError, EmbedderResult, LocalEmbedder};` | None. `LocalEmbedder` is still the source-of-truth trait name. |
| `crates/vestige-core/src/embedder/mod.rs:5` | `pub use fastembed::FastembedEmbedder;` | None. Concrete type, untouched. |
| `crates/vestige-mcp/src/**` | No file imports `Embedder` or `LocalEmbedder` by name; none hold `Arc<dyn Embedder>` or `Box<dyn Embedder>`. | None. Verified by grep returning empty for `dyn Embedder` and `dyn LocalEmbedder` under `crates/vestige-mcp/`. |
| Cognitive modules under `crates/vestige-core/src/advanced/` and `crates/vestige-core/src/neuroscience/` | No file imports `Embedder` or `LocalEmbedder` by name. `advanced/adaptive_embedding.rs` defines its own unrelated `AdaptiveEmbedder` struct. | None. The name collision is purely surface-level; the two types live in different modules and never resolve to each other. |
| `crates/vestige-core/src/embeddings/**` | No file imports `Embedder` or `LocalEmbedder`. The `EmbeddingService` struct is what `FastembedEmbedder` wraps internally. | None. |
The production audit returns zero files that need editing.
### Call sites (tests)
| File | Lines | Use | Required change |
|------------------------------------------------------------|-------|--------------------------------------------------------------------|-----------------|
| `tests/phase_1/embedder_trait.rs` | 3, 19 | `use vestige_core::embedder::{Embedder, FastembedEmbedder};`<br>`let e: Box<dyn Embedder> = Box::new(FastembedEmbedder::new());` | None. `Embedder` is the `trait_variant`-generated Send variant; `Box<dyn Embedder>` keeps compiling. `FastembedEmbedder` implements `LocalEmbedder`; the blanket `impl<T: LocalEmbedder + Send> Embedder for T` that `trait_variant::make` emits gives the boxing for free. |
The test audit returns zero files that need editing.
### Cargo dependency cleanup
| File | Lines | Change |
|-------------------------------------|-----------|-----------------------------------------------------------------------------------------------------|
| `crates/vestige-core/Cargo.toml` | 119 | Remove `async-trait = "0.1"`. Run `cargo rm async-trait` from inside `crates/vestige-core/` so `Cargo.lock` updates atomically with the manifest. |
### Documentation
| File | Change |
|---------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| `crates/vestige-core/src/embedder/mod.rs` | Rewrite the trait-level doc comment (lines 21-26) and the `pub use` doc comment (lines 55-56) to describe `trait_variant`, not `async_trait`. See "Trait declaration rewrite" below for the exact new text. |
| `CLAUDE.md` | No change. The repo-level architecture notes do not name the trait attribute. |
---
## Trait Declaration Rewrite
### Before (state after `0001a` and `0001b` land)
`crates/vestige-core/src/embedder/mod.rs:1-58`:
```rust
//! Text-to-vector encoding trait. Pluggable per-install.
mod fastembed;
pub use fastembed::FastembedEmbedder;
/// Error returned by every `Embedder` method.
#[non_exhaustive]
#[derive(Debug, thiserror::Error)]
pub enum EmbedderError {
#[error("embedder initialization failed: {0}")]
Init(String),
#[error("embedding generation failed: {0}")]
EmbedFailed(String),
#[error("invalid input: {0}")]
InvalidInput(String),
}
pub type EmbedderResult<T> = std::result::Result<T, EmbedderError>;
/// Pluggable embedder. The storage layer NEVER calls fastembed directly;
/// callers compute vectors via this trait and pass them into `MemoryStore`.
///
/// `#[async_trait::async_trait]` makes every `async fn` return a
/// `Pin<Box<dyn Future + Send>>`, which is required for `Box<dyn Embedder>`
/// and `Arc<dyn Embedder>` to be dyn-compatible.
#[async_trait::async_trait]
pub trait LocalEmbedder: Send + Sync + 'static {
async fn embed(&self, text: &str) -> EmbedderResult<Vec<f32>>;
fn model_name(&self) -> &str;
fn dimension(&self) -> usize;
/// Stable blake3 hash of (model_name || dimension || vestige-core crate version).
/// Lowercase hex, 64 chars.
///
/// Used by `MemoryStore::register_model` to detect silent model drift
/// (e.g. a fastembed minor upgrade that changes vector output).
fn model_hash(&self) -> String;
async fn embed_batch(&self, texts: &[&str]) -> EmbedderResult<Vec<Vec<f32>>>;
/// Returns the `ModelSignature` describing this embedder. Convenience
/// wrapper over the three accessors above.
fn signature(&self) -> crate::storage::ModelSignature {
crate::storage::ModelSignature {
name: self.model_name().to_string(),
dimension: self.dimension(),
hash: self.model_hash(),
}
}
}
/// Type alias: `Embedder` is the dyn-compatible, Send+Sync variant.
/// Both names refer to the same `async_trait`-annotated trait.
pub use LocalEmbedder as Embedder;
```
### After
`crates/vestige-core/src/embedder/mod.rs:1-55` (approximately):
```rust
//! Text-to-vector encoding trait. Pluggable per-install.
mod fastembed;
pub use fastembed::FastembedEmbedder;
/// Error returned by every `Embedder` method.
#[non_exhaustive]
#[derive(Debug, thiserror::Error)]
pub enum EmbedderError {
#[error("embedder initialization failed: {0}")]
Init(String),
#[error("embedding generation failed: {0}")]
EmbedFailed(String),
#[error("invalid input: {0}")]
InvalidInput(String),
}
pub type EmbedderResult<T> = std::result::Result<T, EmbedderError>;
/// Pluggable embedder. The storage layer NEVER calls fastembed directly;
/// callers compute vectors via this trait and pass them into `MemoryStore`.
///
/// `LocalEmbedder` is the source-of-truth trait. The
/// `#[trait_variant::make(Embedder: Send)]` attribute auto-generates an
/// `Embedder` variant whose returned futures are `Send`, so
/// `Box<dyn Embedder>` and `Arc<dyn Embedder>` are usable on tokio/axum
/// runtimes, while `Box<dyn LocalEmbedder>` remains usable on single-
/// threaded executors and thread-local backends.
///
/// Every method is native async-fn-in-trait (stable on MSRV 1.91); no
/// per-call heap allocation, no boxed futures at the static-dispatch
/// boundary.
#[trait_variant::make(Embedder: Send)]
pub trait LocalEmbedder: Sync + 'static {
async fn embed(&self, text: &str) -> EmbedderResult<Vec<f32>>;
fn model_name(&self) -> &str;
fn dimension(&self) -> usize;
/// Stable blake3 hash of (model_name || dimension || vestige-core crate version).
/// Lowercase hex, 64 chars.
///
/// Used by `MemoryStore::register_model` to detect silent model drift
/// (e.g. a fastembed minor upgrade that changes vector output).
fn model_hash(&self) -> String;
async fn embed_batch(&self, texts: &[&str]) -> EmbedderResult<Vec<Vec<f32>>>;
/// Returns the `ModelSignature` describing this embedder. Convenience
/// wrapper over the three accessors above.
fn signature(&self) -> crate::storage::ModelSignature {
crate::storage::ModelSignature {
name: self.model_name().to_string(),
dimension: self.dimension(),
hash: self.model_hash(),
}
}
}
```
### Both halves of the macro-generated output (for reviewer clarity)
`trait_variant::make(Embedder: Send)` expands the source-of-truth
`LocalEmbedder` declaration above into the equivalent of:
```rust
// 1. The source-of-truth trait, exactly as written.
pub trait LocalEmbedder: Sync + 'static {
fn embed(&self, text: &str) -> impl Future<Output = EmbedderResult<Vec<f32>>>;
fn model_name(&self) -> &str;
fn dimension(&self) -> usize;
fn model_hash(&self) -> String;
fn embed_batch(&self, texts: &[&str]) -> impl Future<Output = EmbedderResult<Vec<Vec<f32>>>>;
fn signature(&self) -> crate::storage::ModelSignature { /* default impl unchanged */ }
}
// 2. The generated Send variant.
pub trait Embedder: Sync + 'static {
fn embed(&self, text: &str) -> impl Future<Output = EmbedderResult<Vec<f32>>> + Send;
fn model_name(&self) -> &str;
fn dimension(&self) -> usize;
fn model_hash(&self) -> String;
fn embed_batch(&self, texts: &[&str]) -> impl Future<Output = EmbedderResult<Vec<Vec<f32>>>> + Send;
fn signature(&self) -> crate::storage::ModelSignature { /* default impl unchanged */ }
}
// 3. The blanket impl that wires any LocalEmbedder + Send through to Embedder.
impl<T> Embedder for T
where
T: LocalEmbedder + Send,
// (all returned futures of LocalEmbedder's async fns are required to be Send,
// which is satisfied for FastembedEmbedder -- see "Risks" below)
{ /* forwarders */ }
```
Notes:
- The `pub use LocalEmbedder as Embedder;` line on the current
`embedder/mod.rs:57` is **deleted** entirely. `Embedder` is now the
trait that `trait_variant::make` emits at the same module path; the
re-export in `crates/vestige-core/src/lib.rs:167`
(`pub use embedder::{Embedder, ..., LocalEmbedder};`) keeps resolving
unchanged.
- `Sync + 'static` on `LocalEmbedder` (and no `Send` bound on the trait
itself) mirrors the `0001a` pattern for `LocalMemoryStore`. The current
trait carries `Send + Sync + 'static`; the rewrite drops the `Send`
bound from the local variant. `Box<dyn LocalEmbedder>` is `Sync` but
not `Send`; `Box<dyn Embedder>` (the generated variant) is `Send + Sync`.
- `trait_variant` 0.1 does **not** require any attribute on the impl
block. The attribute lives only on the trait declaration. See next
section.
---
## Impl Block Migration
`trait_variant` 0.1 attaches the attribute only to the trait declaration.
The impl side is plain `impl LocalEmbedder for FastembedEmbedder`; no
attribute on the impl, no `#[trait_variant::make(Embedder: Send)]` on the
impl block. The macro auto-generates the blanket
`impl<T: LocalEmbedder + Send> Embedder for T`, so any concrete type that
implements `LocalEmbedder` automatically also implements `Embedder`
provided it is `Send`.
`FastembedEmbedder` is `Send + Sync` because:
- `inner: EmbeddingService` is `Send + Sync` (it wraps fastembed's
`TextEmbedding` which is `Send + Sync` after fastembed 4.x; verified
in `crates/vestige-core/src/embeddings/mod.rs`).
- `cached_hash: std::sync::OnceLock<String>` is `Send + Sync` for `T: Send + Sync`.
- The `#[cfg(not(feature = "embeddings"))]` branch carries only
`cached_hash`, which is unconditionally `Send + Sync`.
No new bound is needed.
### Before
`crates/vestige-core/src/embedder/fastembed.rs:38-100` (relevant header):
```rust
impl Default for FastembedEmbedder {
fn default() -> Self {
Self::new()
}
}
#[async_trait::async_trait]
impl LocalEmbedder for FastembedEmbedder {
async fn embed(&self, text: &str) -> EmbedderResult<Vec<f32>> {
// ... body unchanged ...
}
fn model_name(&self) -> &str { /* ... */ }
fn dimension(&self) -> usize { /* ... */ }
fn model_hash(&self) -> String { /* ... */ }
async fn embed_batch(&self, texts: &[&str]) -> EmbedderResult<Vec<Vec<f32>>> {
// ... body unchanged ...
}
}
```
### After
`crates/vestige-core/src/embedder/fastembed.rs:38-99` (one fewer line):
```rust
impl Default for FastembedEmbedder {
fn default() -> Self {
Self::new()
}
}
impl LocalEmbedder for FastembedEmbedder {
async fn embed(&self, text: &str) -> EmbedderResult<Vec<f32>> {
// ... body unchanged ...
}
fn model_name(&self) -> &str { /* ... */ }
fn dimension(&self) -> usize { /* ... */ }
fn model_hash(&self) -> String { /* ... */ }
async fn embed_batch(&self, texts: &[&str]) -> EmbedderResult<Vec<Vec<f32>>> {
// ... body unchanged ...
}
}
```
Diff is exactly one removed line (the `#[async_trait::async_trait]`
attribute on line 44). Every method body, every `async fn` signature,
every `use` statement inside the impl block stays verbatim. No
`Box::pin(async move { ... })`, no manual `Pin<Box<dyn Future>>`, no
`'async_trait` lifetime markers; native async-fn-in-trait does this
directly.
### Why the impl block does not need an attribute
`trait_variant::make` generates two things from the source trait
(see the "macro-generated output" subsection above):
1. The source trait itself (`LocalEmbedder`), with native async fns.
2. A second trait (`Embedder`) whose method signatures return
`impl Future<Output = ...> + Send` instead of `impl Future<Output = ...>`,
plus a blanket impl wiring concrete types through.
Both are emitted at the macro-call site. `FastembedEmbedder` writes one
impl block (against `LocalEmbedder`); the macro-generated blanket
guarantees `FastembedEmbedder: Embedder` for free. The
`Box<dyn Embedder>` cast on `tests/phase_1/embedder_trait.rs:19` keeps
type-checking unchanged.
---
## Call Site Audit
Verified via, from the phase2 worktree root:
```bash
grep -rn "async_trait\|LocalEmbedder\|dyn Embedder" crates/
grep -rn "use.*Embedder" crates/ tests/ --include="*.rs"
grep -rn "Box<dyn Embedder>\|Arc<dyn Embedder>\|&dyn Embedder" crates/ tests/ --include="*.rs"
grep -rn "Box<dyn LocalEmbedder>\|Arc<dyn LocalEmbedder>\|&dyn LocalEmbedder" crates/ tests/ --include="*.rs"
grep -rn "impl LocalEmbedder for\|impl Embedder for" crates/ tests/ --include="*.rs"
```
### Files that reference the trait object form
Exactly one, test-only:
| File | Line | Use | Required change |
|--------------------------------------|------|------------------------------------------------------------------------------------------|-----------------|
| `tests/phase_1/embedder_trait.rs` | 3 | `use vestige_core::embedder::{Embedder, FastembedEmbedder};` | None. `Embedder` is the generated Send variant at the same path. |
| `tests/phase_1/embedder_trait.rs` | 19 | `let e: Box<dyn Embedder> = Box::new(FastembedEmbedder::new());` | None. `FastembedEmbedder: LocalEmbedder + Send` -> blanket gives `: Embedder` -> `Box<dyn Embedder>` is well-formed. |
### Files that import `Embedder` or `LocalEmbedder` by name
| File | Line | Use | Required change |
|-----------------------------------------------------|------|----------------------------------------------------------------------------------------------------------------|-----------------|
| `crates/vestige-core/src/lib.rs` | 167 | `pub use embedder::{Embedder, EmbedderError, EmbedderResult, FastembedEmbedder, LocalEmbedder};` | None. Both names still resolve. |
| `crates/vestige-core/src/embedder/mod.rs` | 5 | `pub use fastembed::FastembedEmbedder;` | None. |
| `crates/vestige-core/src/embedder/fastembed.rs` | 7 | `use super::{EmbedderError, EmbedderResult, LocalEmbedder};` | None. |
| `tests/phase_1/embedder_trait.rs` | 3 | `use vestige_core::embedder::{Embedder, FastembedEmbedder};` | None. |
### Files that implement the trait
| File | Line | Impl | Required change |
|-----------------------------------------------------|------|-----------------------------------------------------------------------|----------------------------------------------|
| `crates/vestige-core/src/embedder/fastembed.rs` | 45 | `impl LocalEmbedder for FastembedEmbedder` (currently `#[async_trait]`) | Drop the `#[async_trait::async_trait]` attr. |
No other impls exist. There is no test mock implementing `Embedder` or
`LocalEmbedder` anywhere in the workspace.
### Files that import `async_trait` directly
After `0001a` lands, only the embedder pair:
```
crates/vestige-core/src/embedder/mod.rs:24 (doc comment)
crates/vestige-core/src/embedder/mod.rs:27 (attribute)
crates/vestige-core/src/embedder/mod.rs:56 (doc comment)
crates/vestige-core/src/embedder/fastembed.rs:44 (attribute)
```
Plus the Cargo manifest:
```
crates/vestige-core/Cargo.toml:119:async-trait = "0.1"
```
### Production files that hold a concrete embedder
`FastembedEmbedder` is constructed and used by concrete name (not via
trait object) in: the dashboard / MCP layer if it needs to embed query
strings ad-hoc. None of those call sites need an edit because the
concrete type is what they hold, and the concrete type is untouched by
this sub-plan.
### Conclusion
| Category | Count |
|--------------------------------------------------|-------|
| Production source files modified | 2 |
| Test source files modified | 0 |
| Cargo manifests modified | 1 |
| Production source files importing `Embedder` / `LocalEmbedder` (verified unchanged) | 3 |
| Test source files importing `Embedder` (verified unchanged) | 1 |
| Direct `async_trait` uses outside the embedder module after `0001a` | 0 |
---
## Cargo.toml Change
From inside `crates/vestige-core/`:
```bash
cargo rm async-trait
```
This removes line 119 of `Cargo.toml` and updates `Cargo.lock` in one
step. Manual editing is acceptable as a fallback if `cargo rm` is
unavailable in the agent environment; in that case, follow up with
`cargo check -p vestige-core` to refresh the lockfile.
### Verification
```bash
# Direct dependency must be gone.
grep -rn "async-trait\|async_trait" --include="Cargo.toml" crates/
# Expected: empty.
# Transitive presence is permitted (e.g. via a third-party crate).
cargo tree -p vestige-core --depth 2 | grep async-trait
# Expected: empty for the direct edges; if a sub-dependency still pulls
# async-trait transitively, the output may contain it deeper than depth 2,
# which is fine. We only care about removing the DIRECT edge.
```
If `cargo tree --depth 2` returns any `async-trait` line, inspect with
`cargo tree -p vestige-core -i async-trait` to see what is pulling it.
Acceptable parents: any third-party crate. Unacceptable parent: anything
under `vestige-*`, which would mean a missed file.
---
## Commit Sequence
Three commits, each green on
`cargo test -p vestige-core --features embeddings,vector-search` and
`cargo test -p vestige-core --no-default-features`.
### Commit 1: rewrite LocalEmbedder trait declaration
- Touches: `crates/vestige-core/src/embedder/mod.rs` only.
- Action: replace lines 21-57 per the "Trait Declaration Rewrite"
section above. Delete the `pub use LocalEmbedder as Embedder;` line.
- Green after: `cargo check -p vestige-core` (the impl block in
`fastembed.rs` still has its `#[async_trait::async_trait]` attribute;
the macro is harmless when applied to a trait that the impl block
targets by path, because async_trait rewrites the impl's async fns
into boxed-future fns whose signatures still match the native-async
declarations after trait_variant lowering, just as it did for the
SQLite intermediate state in `0001a`'s commit 1).
**Mitigation if check fails between commits 1 and 2:** combine the
two into a single commit. The split is offered for review convenience;
the build must be green after every commit lands.
### Commit 2: drop `#[async_trait::async_trait]` from FastembedEmbedder impl
- Touches: `crates/vestige-core/src/embedder/fastembed.rs` only.
- Action: delete line 44 (`#[async_trait::async_trait]`).
- Green after:
- `cargo test -p vestige-core --features embeddings,vector-search`.
- `cargo test -p vestige-core --no-default-features` (the
`#[cfg(not(feature = "embeddings"))]` branches inside the impl now
stand on their own).
- Phase 1 integration test: `cargo test --test embedder_trait
--features embeddings,vector-search`.
### Commit 3: drop the async-trait dependency
- Touches: `crates/vestige-core/Cargo.toml` (plus `Cargo.lock` as a
side effect).
- Action: from inside `crates/vestige-core/`, run `cargo rm async-trait`.
- Green after: `cargo build --workspace --all-targets` and
`cargo test --workspace`.
- Final hard ASCII gate: `! grep -rn "async_trait" crates/` must exit
with status 0 (i.e. the inverted grep finds nothing).
### Combined alternative
Commits 1 and 2 may fold into a single commit if the per-step split
feels artificial (the patterns are identical to `0001a`'s commits 3
and 4). Commit 3 (the Cargo.toml removal) should stay separate so the
dependency-removal diff is visible in isolation.
---
## Verification
Every command runs from the repo root unless noted otherwise.
```bash
# 1. Vestige-core, default features (embeddings + vector-search).
cargo test -p vestige-core --features embeddings,vector-search
# 2. Vestige-core, minimal features (no embeddings, no vector-search).
cargo test -p vestige-core --no-default-features
# 3. Workspace build, all targets (catches any feature-gated regression
# in the vestige-mcp tools tree).
cargo build --workspace --all-targets
# 4. Whole-workspace test (vestige-mcp 406 tests + vestige-core 352
# tests per the CLAUDE.md baseline).
cargo test --workspace
# 5. Phase 1 embedder integration test (the trait-shape contract).
cargo test --test embedder_trait --features embeddings,vector-search
# 6. Clippy gate, deny warnings (matches Phase 1 PR policy).
cargo clippy --workspace --all-targets --features embeddings,vector-search -- -D warnings
# 7. Hard ASCII gate -- async_trait must be gone from source.
! grep -rn "async_trait" crates/
# Inverted grep: exit 0 iff grep found nothing.
# 8. Hard ASCII gate -- async-trait must be gone from manifests.
! grep -rn "async-trait" --include="Cargo.toml" crates/
# 9. Confirm trait_variant attribute is in place at the embedder.
grep -rn "trait_variant::make" crates/vestige-core/src/embedder/
# Expected: exactly one hit, in embedder/mod.rs.
# 10. Workspace-wide trait_variant audit (should match the count after
# 0001a -- two hits total, one for storage, one for embedder).
grep -rn "trait_variant::make" crates/vestige-core/src/
# Expected: two hits.
```
Expected outcomes:
- Command 1: 352 vestige-core tests pass (matches baseline).
- Command 2: smaller test count, all pass.
- Command 3: workspace builds in dev mode for all targets.
- Command 4: 758 total tests pass (matches CLAUDE.md baseline).
- Command 5: `embedder_trait` integration test passes. The
`fastembed_implements_embedder_trait` assertion (`let e: Box<dyn
Embedder> = ...`) is the canary; if `trait_variant::make` failed to
emit the `Embedder` Send variant, this fails to compile.
- Command 6: zero clippy warnings.
- Command 7: empty output. `async_trait` is fully gone from source.
- Command 8: empty output. `async-trait` is fully gone from manifests.
- Command 9: one hit.
- Command 10: two hits.
---
## Acceptance Criteria
A reviewer should be able to check every box:
- [ ] `crates/vestige-core/src/embedder/mod.rs` declares the embedder
trait with `#[trait_variant::make(Embedder: Send)] pub trait
LocalEmbedder: Sync + 'static`, no `async_trait` attribute, no
`Send` bound on `LocalEmbedder` itself.
- [ ] `crates/vestige-core/src/embedder/mod.rs` no longer contains
`pub use LocalEmbedder as Embedder;`.
- [ ] `crates/vestige-core/src/embedder/fastembed.rs` declares
`impl LocalEmbedder for FastembedEmbedder` with no attribute on
the impl block.
- [ ] `crates/vestige-core/Cargo.toml` does not declare `async-trait`
as a direct dependency.
- [ ] `grep -rn "async_trait" crates/` returns zero hits.
- [ ] `grep -rn "async-trait" --include="Cargo.toml" crates/` returns
zero hits.
- [ ] `grep -rn "trait_variant::make" crates/vestige-core/src/` returns
exactly two hits (storage trait + embedder trait).
- [ ] All 758 workspace tests pass (`cargo test --workspace`).
- [ ] `tests/phase_1/embedder_trait.rs` compiles and passes with the
`Box<dyn Embedder>` cast intact.
- [ ] `cargo clippy --workspace --all-targets --features
embeddings,vector-search -- -D warnings` is clean.
- [ ] No file under `crates/vestige-mcp/` or under
`crates/vestige-core/src/{neuroscience,advanced,consolidation,
codebase,memory,embeddings}/` was modified by this sub-plan.
- [ ] `Cargo.lock` was updated as a side effect of `cargo rm async-trait`
(it must no longer reference `async-trait`).
- [ ] Doc comments on the embedder trait declaration describe
`trait_variant`, not `async_trait`.
---
## Risks and Mitigations
- **`trait_variant::make` requires returned futures to be `Send` for the
blanket `impl<T: LocalEmbedder + Send> Embedder for T`. If any
`async fn embed`/`embed_batch` body inside `FastembedEmbedder` captures
a non-Send local, the blanket impl fails to type-check.**
Mitigation: the existing impl bodies call `self.inner.embed(text)` /
`self.inner.embed_batch(texts)`, where `inner: EmbeddingService` is
`Send + Sync` (verified in `crates/vestige-core/src/embeddings/mod.rs`).
No `.await` points exist inside the bodies in either feature branch;
the `EmbeddingService::embed` calls are synchronous. The futures are
trivially `Send`. If a future change introduces a non-Send local
(e.g. an `Rc` or a non-Send guard), the blanket impl will surface that
as a compile error at the dyn cast in `tests/phase_1/embedder_trait.rs`,
which is the correct outcome.
- **The macro's blanket impl interacts oddly with the default `signature`
method.**
Mitigation: `signature` is a synchronous method returning
`crate::storage::ModelSignature`, with no `Send` or `async` concerns.
`trait_variant::make` emits it on both variants as-is. The existing
Phase 1 test `signature_matches_memory_store_registry` exercises this
path and is part of the verification step.
- **`Box<dyn Embedder>` cast in `tests/phase_1/embedder_trait.rs` fails
to resolve after the rewrite.**
Mitigation: the rewrite preserves the `Embedder` symbol at the same
module path; only its provenance changes (now generated by
`trait_variant::make` instead of by `pub use LocalEmbedder as
Embedder;`). The macro is specifically designed so that the generated
trait is dyn-compatible at the Send-bound boundary. Verified by the
identical pattern already working for `MemoryStore` after `0001a`.
- **`cargo rm async-trait` updates `Cargo.lock` but accidentally bumps
other crates.**
Mitigation: run `cargo rm async-trait` and then immediately inspect
the resulting `Cargo.lock` diff. The expected diff is the removal of
the `[[package]] name = "async-trait"` block and its hash. Anything
else is a red flag and should be reverted before committing
(`git checkout -- Cargo.lock` then `cargo update -p async-trait
--precise=remove` -- or fall back to manual edit + `cargo check`).
- **A new workspace crate added in parallel with this work declares
`async-trait` and the dependency removal silently re-introduces it
later.**
Mitigation: the verification step `grep -rn "async-trait"
--include="Cargo.toml" crates/` is part of the acceptance criteria; a
rebase that reintroduces the line will fail this gate.
- **MCP server uses `Embedder` somewhere we missed.**
Mitigation: full workspace grep (`grep -rn "Embedder" crates/`)
returns no hits inside `crates/vestige-mcp/` for the trait names; the
MCP layer uses the concrete `EmbeddingService` from
`crates/vestige-core/src/embeddings/` for ad-hoc embedding calls. The
trait surface is purely internal to `vestige-core`.
---
## Out-of-Band Notes
- **No other workspace crate declares `async-trait` as a direct
dependency.** Verified by
`grep -rn "async-trait" --include="Cargo.toml" crates/` returning
exactly one hit at `crates/vestige-core/Cargo.toml:119`. There is
nothing to clean up in `crates/vestige-mcp/Cargo.toml` or elsewhere.
- **Order matters across the three Phase 1 amendment sub-plans:**
`0001a` (trait rewrite) -> `0001b` (sqlite split) -> `0001c` (this
one, async-trait sunset). Reversing the order is possible in
principle but would force re-editing the embedder rewrite twice and
leaves the `async-trait` dep behind until very late.
- **This sub-plan amends `feat/storage-trait-phase1` (tip 790c0c8 plus
whatever commits `0001a` and `0001b` added).** The branch has not
been opened upstream yet, so amending in place is safe; no force-push
to a public PR.
- **After this sub-plan lands, the branch is reviewed and merged before
Phase 2 sub-plans (`0002a-` through `0002i-`) begin implementation.**
Phase 2 introduces no async-trait usage; the Postgres backend follows
the same `trait_variant::make` pattern (see ADR 0002 D1).
- **`trait-variant` 0.1 stays in `Cargo.toml`.** It is the only crate
this sub-plan keeps; `async-trait` is the only one it removes.
---
## Self-Contained `/goal` Brief
For a fresh Claude Code session executing this sub-plan without prior
conversation context:
1. Check out branch `feat/storage-trait-phase1` (or a worktree off
of it after `0001a` and `0001b` are merged into it).
2. Read this file (`docs/plans/0001c-async-trait-sunset.md`) in full.
3. Read `docs/plans/0001a-trait-rewrite.md` sections "Trait declaration
rewrite" and "Impl block migration" -- they document the exact
pattern this sub-plan mirrors for the embedder.
4. Run the prerequisite audit grep listed under "Prerequisites". If it
returns more than the five hits documented there, stop and report;
the upstream state does not match what this sub-plan assumes.
5. Execute Commit 1 (rewrite `embedder/mod.rs`), then Commit 2 (drop
the attribute on the FastembedEmbedder impl), then Commit 3
(`cargo rm async-trait`). Run the verification commands listed
above after each commit; do not proceed if any test or clippy gate
fails.
6. Verify every box in "Acceptance Criteria" is ticked.
7. Report file paths touched, test counts, and the final two grep
results (commands 7 and 8 from "Verification") in the closing
message.

1278
docs/plans/0002-phase-2-postgres-backend.md
View file

File diff suppressed because it is too large Load diff

554
docs/plans/0002a-skeleton-and-feature-gate.md
View file

@ -1,554 +0,0 @@
# Phase 2 Sub-Plan 0002a -- Skeleton and Feature Gate
**Status**: Ready
**Depends on**: Phase 1 amendment (sub-plans `0001a-trait-rewrite.md` and
`0001b-sqlite-split.md`) merged. Specifically:
- `MemoryStore` trait declared with `#[trait_variant::make(MemoryStore: Send)]`,
generating a non-Send `LocalMemoryStore` companion trait. The
`pub use MemoryStore as LocalMemoryStore` alias from Phase 1 is gone.
- `crates/vestige-core/src/storage/sqlite.rs` has been split into
`crates/vestige-core/src/storage/sqlite/` with the same public surface.
This sub-plan covers Phase 2 master-plan deliverables D1 and D2 only:
the `postgres-backend` Cargo feature gate and a compilable `PgMemoryStore`
skeleton whose trait method bodies are `todo!()`. No real Postgres code, no
migrations, no SQL. Later sub-plans (`0002b-pool-and-config.md`,
`0002c-migrations.md`, `0002d-store-impl-bodies.md`, ...) fill the bodies in.
The success criterion is a clean build under both feature-flag configurations,
nothing more.
---
## Context
ADR 0002 D4 commits Phase 2 to a `crates/vestige-core/src/storage/postgres/`
directory from day one. The seven other files in that directory
(`pool.rs`, `migrations.rs`, `registry.rs`, `search.rs`, `migrate_cli.rs`,
`reembed.rs`) belong to subsequent sub-plans. This sub-plan creates only
`crates/vestige-core/src/storage/postgres/mod.rs` so the rest can be added
incrementally without breaking the build.
Per ADR 0002 D2, `PgMemoryStore::connect` mirrors `SqliteMemoryStore::new`:
no `Embedder` argument. The pgvector typmod DDL
(`ALTER TABLE memories ALTER COLUMN embedding TYPE vector($N)`) lives inside
the trait method `register_model`, invoked by the caller after construction.
In this sub-plan `register_model` is a `todo!()` body; `0002c-migrations.md`
and `0002d-store-impl-bodies.md` provide the real implementation.
The trait surface in `crates/vestige-core/src/storage/memory_store.rs` is the
source of truth for method signatures. Do NOT copy signatures from the master
plan -- they are stale in places (for example, master plan 0002 D2 lists
`remove_edge` as three-arg `(source, target, edge_type)`; the live trait has
two args `(source, target)`).
---
## Cargo manifest changes
Two optional crates and one new feature flag. Use `cargo add` per the global
CLAUDE.md preference; do not hand-edit `Cargo.toml`.
```bash
cd crates/vestige-core
cargo add sqlx@0.8 --optional --no-default-features \
--features runtime-tokio,tls-rustls,postgres,uuid,chrono,json,migrate,macros
cargo add pgvector@0.4 --optional --features sqlx
```
After both commands, open `crates/vestige-core/Cargo.toml` and add the
`postgres-backend` feature line in the `[features]` block. Place it after
the `metal` feature, before `[dependencies]`:
```toml
# Postgres backend (mutually compilable with the SQLite backend; default OFF).
# Compile with: --features postgres-backend
postgres-backend = ["dep:sqlx", "dep:pgvector"]
```
Do NOT add `tokio-stream`, `futures`, `indicatif`, or `toml` in this sub-plan.
The master plan D1 lists them in the `postgres-backend` feature for
convenience, but their consumers (streaming migrate, progress bar, config
parsing) land in later sub-plans. Adding them here pulls dead weight into the
feature gate.
Do NOT add the `vestige-mcp` pass-through feature in this sub-plan either.
The MCP crate gets its `postgres-backend` feature in `0002b-pool-and-config.md`
when `MemoryStoreConfig` lands and the binary needs a knob to pick a backend.
Verify the diff to `crates/vestige-core/Cargo.toml` looks like this and only
this:
```toml
[features]
# ...existing features unchanged...
postgres-backend = ["dep:sqlx", "dep:pgvector"]
[dependencies]
# ...existing deps unchanged...
sqlx = { version = "0.8", default-features = false, features = [
"runtime-tokio", "tls-rustls", "postgres", "uuid", "chrono",
"json", "migrate", "macros",
], optional = true }
pgvector = { version = "0.4", features = ["sqlx"], optional = true }
```
The exact ordering of the two new lines inside `[dependencies]` is not
significant; `cargo add` places them at the end. Leave the placement that
`cargo add` produces.
---
## Storage module export
Edit `crates/vestige-core/src/storage/mod.rs` to expose the new module behind
the feature flag. Two lines change.
Add to the module-declaration block (after `mod sqlite;`):
```rust
#[cfg(feature = "postgres-backend")]
mod postgres;
```
Add to the re-export block (after the `pub use sqlite::{ ... }` block):
```rust
#[cfg(feature = "postgres-backend")]
pub use postgres::PgMemoryStore;
```
Nothing else in `storage/mod.rs` changes. The `Storage` alias still points at
`SqliteMemoryStore`; the SQLite re-export block is untouched.
---
## Postgres module skeleton
Create `crates/vestige-core/src/storage/postgres/mod.rs` with the full content
below. This is the only new file in this sub-plan.
```rust
#![cfg(feature = "postgres-backend")]
//! Postgres-backed implementation of `MemoryStore`.
//!
//! Skeleton only. Every trait method is `todo!()`. Real bodies land in
//! subsequent Phase 2 sub-plans:
//! - `0002b-pool-and-config.md`: pool construction and config wiring
//! - `0002c-migrations.md`: sqlx migration files and `init`/`register_model`
//! - `0002d-store-impl-bodies.md`: CRUD, scheduling, edges, domains
//! - `0002e-hybrid-search.md`: RRF query and search bodies
//!
//! The directory grows companion files (`pool.rs`, `migrations.rs`,
//! `registry.rs`, `search.rs`, `migrate_cli.rs`, `reembed.rs`) in those
//! sub-plans; this skeleton only creates `mod.rs`.
use chrono::{DateTime, Utc};
use sqlx::PgPool;
use uuid::Uuid;
use crate::storage::memory_store::{
Domain, HealthStatus, LocalMemoryStore, MemoryEdge, MemoryRecord, MemoryStoreResult,
ModelSignature, SchedulingState, SearchQuery, SearchResult, StoreStats,
};
/// Postgres-backed implementation of `MemoryStore`.
///
/// Cheaply cloneable. Methods take `&self`; interior state lives inside the
/// `PgPool` (which already provides `Sync` via `Arc` internally).
#[derive(Clone)]
pub struct PgMemoryStore {
pool: PgPool,
/// Embedding vector dimension. Set to 0 in the skeleton; populated by
/// `register_model` in `0002d-store-impl-bodies.md` once the pgvector
/// `ALTER COLUMN TYPE vector(N)` DDL lands.
embedding_dim: i32,
}
impl PgMemoryStore {
/// Construct a new store from a connection URL.
///
/// Mirrors `SqliteMemoryStore::new`: no `Embedder` argument. The pgvector
/// `ALTER TABLE memories ALTER COLUMN embedding TYPE vector($N)` DDL lives
/// inside `register_model`, not here. The caller (cognitive engine
/// bootstrap, migrate CLI, tests) invokes `register_model` after this
/// returns, identically to the SQLite path.
///
/// Real body lands in `0002b-pool-and-config.md` (pool construction) and
/// `0002c-migrations.md` (initial migration run).
pub async fn connect(_url: &str, _max_connections: u32) -> MemoryStoreResult<Self> {
todo!("PgMemoryStore::connect lands in 0002b-pool-and-config.md")
}
/// Low-level constructor for tests: supply an existing pool, skip migrate.
///
/// Real body lands in `0002b-pool-and-config.md`.
pub async fn from_pool(_pool: PgPool) -> MemoryStoreResult<Self> {
todo!("PgMemoryStore::from_pool lands in 0002b-pool-and-config.md")
}
/// Accessor used by migrate/reembed CLI.
pub fn pool(&self) -> &PgPool {
&self.pool
}
/// Currently-registered vector dimension. Returns 0 until `register_model`
/// has been called (real body: `0002d-store-impl-bodies.md`).
pub fn embedding_dim(&self) -> i32 {
self.embedding_dim
}
}
// trait_variant::make on the trait declaration generates two traits:
// - `MemoryStore` (Send-bound)
// - `LocalMemoryStore` (non-Send companion)
// The implementer writes `impl LocalMemoryStore for ...` plainly; the Send
// variant is generated automatically from the non-Send impl.
impl LocalMemoryStore for PgMemoryStore {
// --- Lifecycle ---
async fn init(&self) -> MemoryStoreResult<()> {
todo!("PgMemoryStore::init lands in 0002c-migrations.md")
}
async fn health_check(&self) -> MemoryStoreResult<HealthStatus> {
todo!("PgMemoryStore::health_check lands in 0002d-store-impl-bodies.md")
}
// --- Embedding model registry ---
async fn registered_model(&self) -> MemoryStoreResult<Option<ModelSignature>> {
todo!("PgMemoryStore::registered_model lands in 0002d-store-impl-bodies.md")
}
async fn register_model(&self, _sig: &ModelSignature) -> MemoryStoreResult<()> {
todo!("PgMemoryStore::register_model lands in 0002d-store-impl-bodies.md")
}
// --- CRUD ---
async fn insert(&self, _record: &MemoryRecord) -> MemoryStoreResult<Uuid> {
todo!("PgMemoryStore::insert lands in 0002d-store-impl-bodies.md")
}
async fn get(&self, _id: Uuid) -> MemoryStoreResult<Option<MemoryRecord>> {
todo!("PgMemoryStore::get lands in 0002d-store-impl-bodies.md")
}
async fn update(&self, _record: &MemoryRecord) -> MemoryStoreResult<()> {
todo!("PgMemoryStore::update lands in 0002d-store-impl-bodies.md")
}
async fn delete(&self, _id: Uuid) -> MemoryStoreResult<()> {
todo!("PgMemoryStore::delete lands in 0002d-store-impl-bodies.md")
}
// --- Search ---
async fn search(&self, _query: &SearchQuery) -> MemoryStoreResult<Vec<SearchResult>> {
todo!("PgMemoryStore::search lands in 0002e-hybrid-search.md")
}
async fn fts_search(
&self,
_text: &str,
_limit: usize,
) -> MemoryStoreResult<Vec<SearchResult>> {
todo!("PgMemoryStore::fts_search lands in 0002e-hybrid-search.md")
}
async fn vector_search(
&self,
_embedding: &[f32],
_limit: usize,
) -> MemoryStoreResult<Vec<SearchResult>> {
todo!("PgMemoryStore::vector_search lands in 0002e-hybrid-search.md")
}
// --- FSRS Scheduling ---
async fn get_scheduling(
&self,
_memory_id: Uuid,
) -> MemoryStoreResult<Option<SchedulingState>> {
todo!("PgMemoryStore::get_scheduling lands in 0002d-store-impl-bodies.md")
}
async fn update_scheduling(&self, _state: &SchedulingState) -> MemoryStoreResult<()> {
todo!("PgMemoryStore::update_scheduling lands in 0002d-store-impl-bodies.md")
}
async fn get_due_memories(
&self,
_before: DateTime<Utc>,
_limit: usize,
) -> MemoryStoreResult<Vec<(MemoryRecord, SchedulingState)>> {
todo!("PgMemoryStore::get_due_memories lands in 0002d-store-impl-bodies.md")
}
// --- Graph (spreading activation) ---
async fn add_edge(&self, _edge: &MemoryEdge) -> MemoryStoreResult<()> {
todo!("PgMemoryStore::add_edge lands in 0002d-store-impl-bodies.md")
}
async fn get_edges(
&self,
_node_id: Uuid,
_edge_type: Option<&str>,
) -> MemoryStoreResult<Vec<MemoryEdge>> {
todo!("PgMemoryStore::get_edges lands in 0002d-store-impl-bodies.md")
}
async fn remove_edge(&self, _source: Uuid, _target: Uuid) -> MemoryStoreResult<()> {
todo!("PgMemoryStore::remove_edge lands in 0002d-store-impl-bodies.md")
}
async fn get_neighbors(
&self,
_node_id: Uuid,
_depth: usize,
) -> MemoryStoreResult<Vec<(MemoryRecord, f64)>> {
todo!("PgMemoryStore::get_neighbors lands in 0002d-store-impl-bodies.md")
}
// --- Domains (Phase 1: stubs return empty; full impl in Phase 4) ---
async fn list_domains(&self) -> MemoryStoreResult<Vec<Domain>> {
todo!("PgMemoryStore::list_domains lands in 0002d-store-impl-bodies.md")
}
async fn get_domain(&self, _id: &str) -> MemoryStoreResult<Option<Domain>> {
todo!("PgMemoryStore::get_domain lands in 0002d-store-impl-bodies.md")
}
async fn upsert_domain(&self, _domain: &Domain) -> MemoryStoreResult<()> {
todo!("PgMemoryStore::upsert_domain lands in 0002d-store-impl-bodies.md")
}
async fn delete_domain(&self, _id: &str) -> MemoryStoreResult<()> {
todo!("PgMemoryStore::delete_domain lands in 0002d-store-impl-bodies.md")
}
async fn classify(&self, _embedding: &[f32]) -> MemoryStoreResult<Vec<(String, f64)>> {
todo!("PgMemoryStore::classify lands in 0002d-store-impl-bodies.md")
}
// --- Bulk / Maintenance ---
async fn count(&self) -> MemoryStoreResult<usize> {
todo!("PgMemoryStore::count lands in 0002d-store-impl-bodies.md")
}
async fn get_stats(&self) -> MemoryStoreResult<StoreStats> {
todo!("PgMemoryStore::get_stats lands in 0002d-store-impl-bodies.md")
}
async fn vacuum(&self) -> MemoryStoreResult<()> {
todo!("PgMemoryStore::vacuum lands in 0002d-store-impl-bodies.md")
}
}
```
Notes on the skeleton:
- The file-level `#![cfg(feature = "postgres-backend")]` means the whole file
vanishes when the feature is off. The `mod postgres;` line in
`storage/mod.rs` is itself feature-gated, so this is belt-and-braces; both
gates are needed because the file-level attribute is what allows the file to
use `sqlx::PgPool` unconditionally inside it.
- `EmbeddingModelDescriptor` (a separate Postgres-internal type that the
master plan sketched on the struct) is dropped. The trait surface already
carries `ModelSignature` for the registry round-trip; the registry storage
layout is a private concern of `registry.rs`, which is added later. Keep
`PgMemoryStore` minimal until a real consumer needs the extra type.
- The struct only carries `pool` and `embedding_dim`. The model descriptor
field from the master plan sketch goes away with `EmbeddingModelDescriptor`.
If `register_model` later needs to cache the descriptor on the struct, it
can be added then; the skeleton does not speculate.
- The two trivial accessors (`pool`, `embedding_dim`) get real bodies. Every
other method is `todo!()` so it returns `!` and trivially coerces to the
declared return type at the type checker; this is what lets the build pass
with no error variants and no SQL.
---
## Connect signature
Per ADR 0002 D2:
```rust
pub async fn connect(url: &str, max_connections: u32) -> MemoryStoreResult<Self>;
pub async fn from_pool(pool: PgPool) -> MemoryStoreResult<Self>;
```
No `&dyn Embedder` argument. This deliberately differs from master plan 0002,
which predates the Phase 1 freeze. The pgvector-specific DDL
(`ALTER TABLE memories ALTER COLUMN embedding TYPE vector($N)`) does not run
inside `connect`; it runs inside `register_model(&ModelSignature)`, which the
caller invokes after `connect` returns.
In this sub-plan `register_model` is `todo!()`. The real body lands in
`0002d-store-impl-bodies.md` after `0002c-migrations.md` ships the
`0001_init.up.sql` migration that creates the `memories` table with a
placeholder `embedding vector` column (no typmod), against which
`register_model` later runs the typmod stamp.
---
## Error variant additions: deferred
`MemoryStoreError` does NOT gain `Postgres(sqlx::Error)` or
`Migrate(sqlx::migrate::MigrateError)` in this sub-plan.
The reason is mechanical: `todo!()` evaluates to the never type `!`, which
coerces to any `MemoryStoreResult<T>` regardless of the error variants
available. With every method body a `todo!()`, the skeleton has no expression
that needs to convert a `sqlx::Error` or `sqlx::migrate::MigrateError` into
`MemoryStoreError`. Adding the variants here would mean adding the
`#[cfg(feature = "postgres-backend")]` and `#[from]` plumbing to
`memory_store.rs` with no consumer yet -- dead code at every level except the
enum definition itself.
`0002d-store-impl-bodies.md` introduces both variants in the same commit that
turns the first `todo!()` into a real `sqlx::query!` call. That keeps the
diff to `memory_store.rs` next to the first usage site, which is easier to
review than adding variants ahead of need.
For reference, the variants that will be added in `0002d-store-impl-bodies.md`
look like this:
```rust
#[cfg(feature = "postgres-backend")]
#[error("postgres error: {0}")]
Postgres(#[from] sqlx::Error),
#[cfg(feature = "postgres-backend")]
#[error("postgres migration error: {0}")]
Migrate(#[from] sqlx::migrate::MigrateError),
```
Do not pre-add them here.
---
## Verification
Run these commands from the workspace root. All four must produce a clean
build, zero warnings on the diff-affected files, no test changes.
```bash
# 1. Default features (SQLite backend, postgres-backend OFF). Must build.
cargo build --workspace --all-targets
# 2. Workspace clippy with default features. Must be clean.
cargo clippy --workspace --all-targets -- -D warnings
# 3. Postgres feature enabled. Must build.
cargo build -p vestige-core --features postgres-backend
# 4. Clippy with postgres feature enabled. Must be clean.
cargo clippy -p vestige-core --features postgres-backend --all-targets -- -D warnings
```
Expected outcomes:
- `cargo build --workspace --all-targets` finishes with no compilation of
`sqlx` or `pgvector` (both are optional, no consumer with default features).
The `postgres` module is excluded entirely via `#[cfg]`.
- `cargo build -p vestige-core --features postgres-backend` compiles `sqlx`,
`pgvector`, and `storage/postgres/mod.rs`. The build succeeds because every
trait method is `todo!()`; nothing actually runs SQL.
- Both `clippy` invocations pass with `-D warnings`. The `todo!()` macro does
not emit a `dead_code` lint by itself, and the trivial accessors are used by
later sub-plans (clippy on the postgres feature alone may flag them as
unused if you run with `--lib` only; the `--all-targets` form keeps tests
and benches in scope so this does not fire).
- If clippy flags `unused_variables` on the underscore-prefixed parameters in
the `todo!()` bodies, the underscore prefix is already the standard
suppression; if a future clippy version disagrees, add
`#[allow(unused_variables)]` to the impl block, not to each method.
Tests are not modified in this sub-plan. The unit tests in
`memory_store.rs` (`memory_store_error_from_storage_error`,
`model_signature_serde_round_trip`, `memory_record_serde_round_trip`) keep
passing because no type they touch changes.
Do NOT run `cargo test` against the postgres feature -- there is no Postgres
running and no test exercises `PgMemoryStore` yet. The build check is the
contract.
---
## Acceptance criteria
1. `crates/vestige-core/Cargo.toml` declares `sqlx = "0.8"` and
`pgvector = "0.4"` as optional dependencies with the exact feature sets
specified above.
2. `crates/vestige-core/Cargo.toml` declares `postgres-backend = ["dep:sqlx",
"dep:pgvector"]` and nothing else inside that feature.
3. `crates/vestige-mcp/Cargo.toml` is unchanged.
4. `crates/vestige-core/src/storage/mod.rs` adds exactly two
feature-gated lines: `mod postgres;` and `pub use postgres::PgMemoryStore;`.
No other change.
5. `crates/vestige-core/src/storage/postgres/mod.rs` exists and contains the
`PgMemoryStore` struct, `impl PgMemoryStore` block with real `pool` and
`embedding_dim` accessors and `todo!()` bodies for `connect` and
`from_pool`, and the full `impl LocalMemoryStore for PgMemoryStore` block
with `todo!()` for every trait method.
6. The trait impl method signatures match `memory_store.rs` byte-for-byte
(including `remove_edge(&self, source: Uuid, target: Uuid)` two-arg form,
not the three-arg form from the master plan).
7. `MemoryStoreError` is unchanged.
8. No other files in the crate are touched. No new files in
`storage/postgres/` besides `mod.rs`.
9. The four verification commands above all succeed.
---
## Commit sequence
One commit is recommended. The two changes (Cargo manifest + module skeleton)
do not compile in isolation: the manifest change without the skeleton produces
unused-optional-dep warnings, and the skeleton without the manifest change
fails to find `sqlx`. Splitting them adds no review value, since the second
commit is the one that has to compile cleanly.
```bash
git add crates/vestige-core/Cargo.toml \
crates/vestige-core/Cargo.lock \
crates/vestige-core/src/storage/mod.rs \
crates/vestige-core/src/storage/postgres/mod.rs
git commit -m "feat(storage): scaffold postgres-backend feature and PgMemoryStore skeleton
Adds the postgres-backend Cargo feature gating sqlx 0.8 and pgvector 0.4.
Introduces crates/vestige-core/src/storage/postgres/mod.rs with the
PgMemoryStore struct, connect/from_pool/pool/embedding_dim, and a trait impl
whose method bodies are todo!() pending later Phase 2 sub-plans.
Builds clean with default features (SQLite only) and with --features
postgres-backend. No runtime behaviour change.
Refs ADR 0002 D1, D2, D4."
```
If for any reason the manifest change must be reviewed separately (for
example, a security review of the sqlx version pin), split as:
1. `cargo add` for sqlx and pgvector + manual feature line in Cargo.toml.
Build with default features will pass but `--features postgres-backend`
will fail (no module to satisfy the feature). This is acceptable for a
short-lived intermediate commit.
2. `storage/mod.rs` edits + `storage/postgres/mod.rs` creation. Both builds
pass.
Default to the single-commit form unless asked to split.
---
## Followups
- `0002b-pool-and-config.md` adds `pool.rs`, `PostgresConfig`, and the
`vestige-mcp` `postgres-backend` pass-through feature.
- `0002c-migrations.md` adds `crates/vestige-core/migrations/postgres/` with
`0001_init.{up,down}.sql` and `0002_hnsw.{up,down}.sql`, plus
`postgres/migrations.rs` invoking `sqlx::migrate!`. `init()` body lands here.
- `0002d-store-impl-bodies.md` introduces the two `MemoryStoreError` variants
and replaces every `todo!()` in CRUD / scheduling / edges / domains /
registry with real `sqlx::query!` bodies.
- `0002e-hybrid-search.md` fills the three search bodies via the RRF query.

886
docs/plans/0002b-pool-and-config.md
View file

@ -1,886 +0,0 @@
# Sub-plan 0002b -- Pool construction and VestigeConfig
**Status**: Draft
**Master plan**: [0002-phase-2-postgres-backend.md](0002-phase-2-postgres-backend.md)
**ADR**: [0002-phase-2-execution.md](../adr/0002-phase-2-execution.md)
**Predecessor**: [0002a-skeleton-and-feature-gate.md](0002a-skeleton-and-feature-gate.md)
---
## Context
This sub-plan delivers two of the master plan's deliverables now that the
`0002a` skeleton has landed:
- **D3** -- pool construction in
`crates/vestige-core/src/storage/postgres/pool.rs`. Replaces the `todo!()`
body of `PgMemoryStore::connect` with a real `PgPool` builder that reads a
`PostgresConfig`. Registry/migration calls remain `todo!()`; those are
filled in by sub-plans `0002c` (migrations) and `0002d` (store bodies +
registry).
- **D7** -- new module `crates/vestige-core/src/config.rs` containing
`VestigeConfig`, `StorageConfig`, `SqliteConfig`, `PostgresConfig`,
`EmbeddingsConfig`, plus a `ConfigError` enum and a loader that reads
`vestige.toml`. The loader is wired into `vestige-mcp` so the running
server picks SQLite or Postgres at startup based on the config file.
After this sub-plan:
- `cargo build` (default features, no `postgres-backend`) compiles and the
MCP server still defaults to SQLite when no `vestige.toml` is present.
- `cargo build --features postgres-backend` compiles, with
`PgMemoryStore::connect` now wiring through `pool.rs` (registry/migration
still `todo!()` until `0002c` and `0002d`).
- A `vestige.toml` example can be round-tripped through
`VestigeConfig::load` in a unit test.
This sub-plan deliberately does NOT:
- Add migrations (`0002c`).
- Fill in real CRUD/search bodies on `PgMemoryStore` (`0002d`, `0002e`).
- Add env-var override support (Phase 3 concern, called out in master plan
D7 behaviour notes).
---
## Dependencies
- `0002a-skeleton-and-feature-gate.md` must be merged. That sub-plan creates
`crates/vestige-core/src/storage/postgres/mod.rs` with:
- `PgMemoryStore` struct holding `pool: PgPool`.
- `PgMemoryStore::connect(url: &str, max_connections: u32) -> MemoryStoreResult<Self>`
body = `todo!()`.
- `PgMemoryStore::from_pool(pool: PgPool) -> MemoryStoreResult<Self>`
body = `todo!()`.
- The trait impl block with all methods routed to `todo!()`.
- The `postgres-backend` feature gate on the module declaration in
`storage/mod.rs`.
This sub-plan extends those bodies and adds two siblings: `pool.rs` and
`registry.rs` (the latter is a stub here, real body in `0002d`).
---
## Audit step (do this first)
Before adding `config.rs`, confirm there is no existing top-level config
loader. Run from the repo root:
```bash
rg -nF 'VestigeConfig' crates/
rg -nF 'toml::from_str' crates/
rg -n '#\[derive.*Deserialize.*\]' crates/vestige-core/src/
```
If a `VestigeConfig` struct already exists from Phase 1, treat the "Config
module" section below as additive: extend the existing struct rather than
creating a new file. The cross-cut additions in that case are:
1. Add the `StorageConfig` enum (gated and ungated branches).
2. Add `SqliteConfig`, `PostgresConfig`.
3. Add the `default_path()` helper if missing.
4. Add `ConfigError` if a different error enum is used today (rename/extend
instead of duplicating).
As of the audit at the time of this writing, no `VestigeConfig` exists in
`vestige-core`. `directories::ProjectDirs` is already used in
`vestige-core/src/embeddings/local.rs` and in
`vestige-mcp/src/protocol/auth.rs`, so the `directories` crate is already a
workspace dependency -- no new dep there.
---
## Cargo manifest additions
Add `toml` to `vestige-core`. `serde` and `thiserror` are already present
from Phase 1; `directories` is already a transitive dep but we add it
explicitly so `default_path()` is supported.
```bash
cd crates/vestige-core
cargo add toml@0.8
cargo add directories@5
```
No new deps on `vestige-mcp`; it already depends on `vestige-core`.
`sqlx` is already added by `0002a` behind the `postgres-backend` feature
with `runtime-tokio`, `tls-rustls`, `postgres`, `uuid`, `chrono`,
`json`, `macros`, `migrate` features. The pool module only uses what is
already pulled in.
---
## Config module
**File**: `crates/vestige-core/src/config.rs` (new).
**Re-exported** from `crates/vestige-core/src/lib.rs` as `pub mod config;` plus
`pub use config::{VestigeConfig, StorageConfig, SqliteConfig, EmbeddingsConfig, ConfigError};`
and `#[cfg(feature = "postgres-backend")] pub use config::PostgresConfig;`.
Full content:
```rust
//! Vestige top-level configuration.
//!
//! Loaded from `~/.vestige/vestige.toml` by default; the path is overridable
//! via `VestigeConfig::load(Some(&path))`. Parsing uses serde + toml; the
//! `[storage]` section is internally-tagged on a `backend` field so a single
//! enum dispatch picks SQLite or Postgres.
use std::path::{Path, PathBuf};
use serde::Deserialize;
/// Top-level configuration as parsed from `vestige.toml`.
#[derive(Debug, Clone, Deserialize, Default)]
#[serde(default, deny_unknown_fields)]
pub struct VestigeConfig {
pub embeddings: EmbeddingsConfig,
pub storage: StorageConfig,
/// Reserved for Phase 3. Empty in Phase 2.
pub server: ServerConfig,
/// Reserved for Phase 3. Empty in Phase 2.
pub auth: AuthConfig,
}
/// Embedding provider selection.
#[derive(Debug, Clone, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct EmbeddingsConfig {
/// Provider key. Phase 2 ships `"fastembed"` only.
pub provider: String,
/// Model name. For fastembed this is e.g. `"nomic-ai/nomic-embed-text-v1.5"`.
pub model: String,
}
impl Default for EmbeddingsConfig {
fn default() -> Self {
Self {
provider: "fastembed".to_string(),
model: crate::DEFAULT_EMBEDDING_MODEL.to_string(),
}
}
}
/// Storage backend selection. Internally tagged on the `backend` field:
///
/// ```toml
/// [storage]
/// backend = "sqlite"
///
/// [storage.sqlite]
/// path = "/home/user/.vestige/vestige.db"
/// ```
///
/// or, when compiled with `--features postgres-backend`:
///
/// ```toml
/// [storage]
/// backend = "postgres"
///
/// [storage.postgres]
/// url = "postgres://vestige:secret@localhost:5432/vestige"
/// max_connections = 10
/// acquire_timeout_secs = 30
/// ```
#[derive(Debug, Clone, Deserialize)]
#[serde(tag = "backend", rename_all = "lowercase", deny_unknown_fields)]
pub enum StorageConfig {
Sqlite(SqliteConfig),
#[cfg(feature = "postgres-backend")]
Postgres(PostgresConfig),
}
impl Default for StorageConfig {
fn default() -> Self {
StorageConfig::Sqlite(SqliteConfig::default())
}
}
/// SQLite backend configuration.
#[derive(Debug, Clone, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct SqliteConfig {
/// Path to the `vestige.db` file. If unset, the SqliteMemoryStore
/// constructor picks its platform default location.
#[serde(default)]
pub path: Option<PathBuf>,
}
impl Default for SqliteConfig {
fn default() -> Self {
Self { path: None }
}
}
/// Postgres backend configuration. Only present when the `postgres-backend`
/// Cargo feature is enabled.
#[cfg(feature = "postgres-backend")]
#[derive(Debug, Clone, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct PostgresConfig {
/// `postgres://user:pass@host:port/db` -- forwarded to
/// `PgConnectOptions::from_str`.
pub url: String,
/// Pool size. Default `10`.
#[serde(default)]
pub max_connections: Option<u32>,
/// Acquire timeout in seconds. Default `30`. Set above 30 so
/// testcontainer-based test fixtures do not race.
#[serde(default)]
pub acquire_timeout_secs: Option<u64>,
}
/// Reserved for Phase 3 (bind address, ports, TLS).
#[derive(Debug, Clone, Default, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct ServerConfig {}
/// Reserved for Phase 3 (API keys, claims).
#[derive(Debug, Clone, Default, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct AuthConfig {}
/// Errors raised while locating, reading, or parsing `vestige.toml`.
#[derive(Debug, thiserror::Error)]
pub enum ConfigError {
#[error("config io: {0}")]
Io(#[from] std::io::Error),
#[error("config toml: {0}")]
Toml(#[from] toml::de::Error),
#[error("config dir: could not locate user home")]
NoHome,
#[error("invalid config: {0}")]
Invalid(String),
}
impl VestigeConfig {
/// Load config from `path` or from `default_path()` when `None`.
///
/// Returns `VestigeConfig::default()` (SQLite + fastembed defaults) when
/// the file does not exist. Any other I/O or parse failure is surfaced
/// as a `ConfigError`.
pub fn load(path: Option<&Path>) -> Result<Self, ConfigError> {
let resolved: PathBuf = match path {
Some(p) => p.to_path_buf(),
None => Self::default_path()?,
};
match std::fs::read_to_string(&resolved) {
Ok(text) => {
let cfg: VestigeConfig = toml::from_str(&text)?;
cfg.validate()?;
Ok(cfg)
}
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
Ok(Self::default())
}
Err(e) => Err(ConfigError::Io(e)),
}
}
/// `~/.vestige/vestige.toml`. The directory is NOT created here; loading
/// a missing file falls back to defaults.
pub fn default_path() -> Result<PathBuf, ConfigError> {
let dirs = directories::ProjectDirs::from("", "vestige", "vestige")
.ok_or(ConfigError::NoHome)?;
// ProjectDirs::config_dir() varies per OS. Vestige convention is
// ~/.vestige/vestige.toml on Linux/macOS regardless of XDG, so we
// build the path off the home dir explicitly.
let home = directories::UserDirs::new()
.ok_or(ConfigError::NoHome)?
.home_dir()
.to_path_buf();
let _ = dirs; // keep the dep wired; future Phase 3 may use it
Ok(home.join(".vestige").join("vestige.toml"))
}
/// Light cross-field validation. Heavy validation (URL parsing,
/// directory existence) is left to the backend constructors.
fn validate(&self) -> Result<(), ConfigError> {
if self.embeddings.provider.is_empty() {
return Err(ConfigError::Invalid(
"embeddings.provider must not be empty".into(),
));
}
if self.embeddings.model.is_empty() {
return Err(ConfigError::Invalid(
"embeddings.model must not be empty".into(),
));
}
match &self.storage {
StorageConfig::Sqlite(_) => {}
#[cfg(feature = "postgres-backend")]
StorageConfig::Postgres(cfg) => {
if cfg.url.is_empty() {
return Err(ConfigError::Invalid(
"storage.postgres.url must not be empty".into(),
));
}
}
}
Ok(())
}
}
```
### Serde behaviour with `postgres-backend` off
`StorageConfig` is generated by serde only for the variants that are
compiled in. When `postgres-backend` is off and the user writes:
```toml
[storage]
backend = "postgres"
[storage.postgres]
url = "..."
```
serde returns a `toml::de::Error` of the form
`unknown variant `postgres`, expected `sqlite``. That error path goes
through `From<toml::de::Error> for ConfigError`, surfacing as
`ConfigError::Toml(..)`. The MCP server prints this once at startup and
exits with a non-zero code; there is no panic.
To make the error friendlier we wrap that specific case in a clearer
message via a thin post-parse check. Add this small helper after parsing
in `load()`:
```rust
// (Inside the Ok(text) arm in load(), wrapping the parse step.)
let cfg: VestigeConfig = match toml::from_str(&text) {
Ok(c) => c,
Err(e) => {
let msg = e.to_string();
if msg.contains("unknown variant `postgres`") {
return Err(ConfigError::Invalid(
"storage.backend = \"postgres\" requires building with --features postgres-backend".into(),
));
}
return Err(ConfigError::Toml(e));
}
};
```
This keeps the strict default deny_unknown_fields behaviour while giving the
user a one-line action item.
---
## Pool module
**File**: `crates/vestige-core/src/storage/postgres/pool.rs` (new).
```rust
#![cfg(feature = "postgres-backend")]
//! `PgPool` construction for the Postgres backend.
//!
//! Pool defaults follow ADR 0002 D2 + master plan D3:
//! - max_connections = 10
//! - acquire_timeout = 30s (must exceed testcontainer warmup)
//! - idle_timeout = 600s
//! - max_lifetime = 1800s
//! - test_before_acquire = false (cheap queries; saves a roundtrip)
use std::str::FromStr;
use std::time::Duration;
use sqlx::postgres::{PgConnectOptions, PgPoolOptions};
use sqlx::{ConnectOptions, PgPool};
use crate::config::PostgresConfig;
use crate::storage::memory_store::{MemoryStoreError, MemoryStoreResult};
const DEFAULT_MAX_CONNECTIONS: u32 = 10;
const DEFAULT_ACQUIRE_TIMEOUT_SECS: u64 = 30;
const IDLE_TIMEOUT_SECS: u64 = 600;
const MAX_LIFETIME_SECS: u64 = 1800;
const STATEMENT_CACHE_CAPACITY: usize = 256;
/// Build a Postgres connection pool from a `PostgresConfig`. Does NOT run
/// migrations or stamp the embedding registry; those are the caller's job
/// (`PgMemoryStore::connect`).
pub async fn build_pool(cfg: &PostgresConfig) -> MemoryStoreResult<PgPool> {
let opts = PgConnectOptions::from_str(&cfg.url)
.map_err(MemoryStoreError::from)?
.application_name("vestige")
.statement_cache_capacity(STATEMENT_CACHE_CAPACITY)
.log_statements(tracing::log::LevelFilter::Debug);
let max_conn = cfg.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
let acquire = cfg
.acquire_timeout_secs
.unwrap_or(DEFAULT_ACQUIRE_TIMEOUT_SECS);
let pool = PgPoolOptions::new()
.max_connections(max_conn)
.min_connections(0)
.acquire_timeout(Duration::from_secs(acquire))
.idle_timeout(Some(Duration::from_secs(IDLE_TIMEOUT_SECS)))
.max_lifetime(Some(Duration::from_secs(MAX_LIFETIME_SECS)))
.test_before_acquire(false)
.connect_with(opts)
.await
.map_err(MemoryStoreError::from)?;
Ok(pool)
}
```
### Wiring into `PgMemoryStore::connect`
In `crates/vestige-core/src/storage/postgres/mod.rs`, replace the
`todo!()` body left by `0002a` for `connect` and `from_pool` with:
```rust
// In crates/vestige-core/src/storage/postgres/mod.rs
use sqlx::PgPool;
use crate::config::PostgresConfig;
use crate::storage::memory_store::{MemoryStoreError, MemoryStoreResult};
mod pool;
mod registry; // see "Registry stub" section below
pub struct PgMemoryStore {
pool: PgPool,
}
impl PgMemoryStore {
/// Convenience constructor matching `SqliteMemoryStore::new` shape.
/// Takes a URL + pool size for the common case.
pub async fn connect(url: &str, max_connections: u32) -> MemoryStoreResult<Self> {
let cfg = PostgresConfig {
url: url.to_string(),
max_connections: Some(max_connections),
acquire_timeout_secs: None,
};
Self::connect_with(&cfg).await
}
/// Full-config constructor.
pub async fn connect_with(cfg: &PostgresConfig) -> MemoryStoreResult<Self> {
let pool = pool::build_pool(cfg).await?;
Self::from_pool(pool).await
}
/// Construct from an already-built pool (used by tests and the migrate
/// CLI to share a pool across operations).
pub async fn from_pool(pool: PgPool) -> MemoryStoreResult<Self> {
// Migrations are added by 0002c.
// todo!("run sqlx::migrate! once 0002c lands")
registry::ensure_registry_stub(&pool).await?;
Ok(Self { pool })
}
}
```
`connect_with` is the long-lived API; `connect` becomes a thin shim that
stays compatible with the master-plan-mandated signature.
### Registry stub
**File**: `crates/vestige-core/src/storage/postgres/registry.rs` (new, stub).
```rust
#![cfg(feature = "postgres-backend")]
//! Embedding registry. Real body lands in sub-plan 0002d.
use sqlx::PgPool;
use crate::storage::memory_store::MemoryStoreResult;
/// Placeholder. Real implementation in 0002d reads/writes `embedding_model`
/// and stamps `ALTER TABLE memories ALTER COLUMN embedding TYPE vector($N)`.
pub(crate) async fn ensure_registry_stub(_pool: &PgPool) -> MemoryStoreResult<()> {
// Intentionally a no-op until 0002c lands the table + 0002d lands the
// real body. Leaving this as todo!() would crash the MCP server at
// startup the moment a user switches `backend = "postgres"`, which is
// not what we want for the build verification step in this sub-plan.
Ok(())
}
```
The no-op keeps `cargo build --features postgres-backend` not just
compiling but also allowing the MCP server to *boot* against a Postgres
URL pointing at an already-migrated database (the local-dev-postgres-setup
docs cover bringing up such a DB by hand). Real init lands in `0002d`.
---
## Error variants
**File**: `crates/vestige-core/src/storage/memory_store.rs` (edit).
The Phase 1 enum `MemoryStoreError` gains two feature-gated variants. These
were deferred in `0002a` and become required as soon as `pool.rs` calls
`.map_err(MemoryStoreError::from)` on `sqlx::Error`.
```rust
// Within enum MemoryStoreError { ... } in memory_store.rs
#[cfg(feature = "postgres-backend")]
#[error("postgres error: {0}")]
Postgres(#[from] sqlx::Error),
#[cfg(feature = "postgres-backend")]
#[error("postgres migration error: {0}")]
Migrate(#[from] sqlx::migrate::MigrateError),
```
Both use thiserror's `#[from]` attribute so the `?` operator works in
`pool.rs`, the migrate module (`0002c`), and registry code (`0002d`).
Default-features build (no `postgres-backend`) sees neither variant; the
enum stays exhaustive on stable.
If clippy fires on `non_exhaustive` due to the gated variants, add
`#[non_exhaustive]` on the enum. That has no caller-side effect since the
enum is constructed only inside the crate.
---
## vestige-mcp wiring
### Cargo feature passthrough
**File**: `crates/vestige-mcp/Cargo.toml` (edit).
Add a feature that forwards through to `vestige-core`. Default features in
`vestige-mcp` stay unchanged.
```toml
[features]
default = ["embeddings", "vector-search"]
embeddings = ["vestige-core/embeddings"]
vector-search = ["vestige-core/vector-search"]
postgres-backend = ["vestige-core/postgres-backend"]
```
Verify with:
```bash
cargo build -p vestige-mcp --features postgres-backend
```
### Backend dispatch at startup
**File**: `crates/vestige-mcp/src/main.rs` (edit around the existing
`Storage::new(storage_path)` call -- see audit note above; in the current
worktree this is around line 285).
The current code is roughly:
```rust
let storage_path = match prepare_storage_path(config.data_dir) { ... };
let storage = match Storage::new(storage_path) { ... };
```
Replace that with a dispatch driven by `VestigeConfig`:
```rust
use std::sync::Arc;
use vestige_core::config::{StorageConfig, VestigeConfig};
use vestige_core::storage::SqliteMemoryStore;
#[cfg(feature = "postgres-backend")]
use vestige_core::storage::postgres::PgMemoryStore;
use vestige_core::storage::MemoryStore;
// Earlier: still call prepare_storage_path to honour --data-dir override.
let storage_path = match prepare_storage_path(config.data_dir.clone()) { ... };
// New: load vestige.toml (or fall back to defaults).
let vestige_cfg = match VestigeConfig::load(config.config_path.as_deref()) {
Ok(c) => c,
Err(e) => {
eprintln!("vestige: failed to load config: {e}");
std::process::exit(2);
}
};
let storage: Arc<dyn MemoryStore> = match &vestige_cfg.storage {
StorageConfig::Sqlite(sqlite_cfg) => {
// CLI flag --data-dir wins over the config file path.
let path = storage_path.clone().or_else(|| sqlite_cfg.path.clone());
let s = SqliteMemoryStore::new(path).unwrap_or_else(|e| {
eprintln!("vestige: sqlite init failed: {e}");
std::process::exit(3);
});
Arc::new(s)
}
#[cfg(feature = "postgres-backend")]
StorageConfig::Postgres(pg_cfg) => {
let s = PgMemoryStore::connect_with(pg_cfg).await.unwrap_or_else(|e| {
eprintln!("vestige: postgres init failed: {e}");
std::process::exit(3);
});
Arc::new(s)
}
};
```
The `config_path: Option<PathBuf>` field on the local `Config` (or
clap-derived `Args`) struct must be added if not present; it accepts
`--config <path>`. Default behaviour (no flag) goes through
`VestigeConfig::default_path()`.
If the existing main wires `Storage` through a concrete type rather than
`Arc<dyn MemoryStore>`, the dispatch above lives behind a helper:
```rust
async fn build_store(cfg: &VestigeConfig, cli_path: Option<PathBuf>)
-> Result<Arc<dyn MemoryStore>, anyhow::Error>
{ ... }
```
and the caller chains `.into()` as needed. Phase 1 already moved
cognitive modules to `Arc<dyn MemoryStore>` so this should be a pure
substitution; if a concrete-type holdout is found, fix it locally in this
sub-plan (separate commit) rather than punting.
---
## vestige.toml example
The canonical example to ship in `docs/` (Phase 2 docs land in `0002i`,
runbook), shown here for reference and used verbatim by the unit test
below.
```toml
# vestige.toml -- top-level configuration
#
# Default location: ~/.vestige/vestige.toml
# Override: vestige-mcp --config /path/to/vestige.toml
[embeddings]
provider = "fastembed"
model = "nomic-ai/nomic-embed-text-v1.5"
# --- SQLite backend (default) ---
[storage]
backend = "sqlite"
[storage.sqlite]
path = "/home/user/.vestige/vestige.db"
# --- Postgres backend (requires --features postgres-backend) ---
# [storage]
# backend = "postgres"
#
# [storage.postgres]
# url = "postgres://vestige:secret@localhost:5432/vestige"
# max_connections = 10
# acquire_timeout_secs = 30
[server]
# Reserved for Phase 3 (bind address, ports, TLS).
[auth]
# Reserved for Phase 3 (API keys, claims).
```
---
## Verification
Run all of these from the repo root. The first three are the gates that
must pass before this sub-plan is considered done.
### 1. Default build (no Postgres)
```bash
cargo build -p vestige-core
cargo build -p vestige-mcp
cargo test -p vestige-core --lib
```
Expected: clean build. `VestigeConfig::default()` selects SQLite; the MCP
server boots the same way it did pre-sub-plan.
### 2. Postgres-feature build
```bash
cargo build -p vestige-core --features postgres-backend
cargo build -p vestige-mcp --features postgres-backend
```
Expected: clean build. `PgMemoryStore::connect_with` resolves to
`pool::build_pool` + `registry::ensure_registry_stub`; no `todo!()` is
reachable on the build path. `connect` and `from_pool` are exported.
### 3. Clippy across both feature sets
```bash
cargo clippy -p vestige-core -- -D warnings
cargo clippy -p vestige-core --features postgres-backend -- -D warnings
cargo clippy -p vestige-mcp --features postgres-backend -- -D warnings
```
### 4. Unit test: round-trip the example
Add this test to `crates/vestige-core/src/config.rs`:
```rust
#[cfg(test)]
mod tests {
use super::*;
const EXAMPLE_SQLITE: &str = r#"
[embeddings]
provider = "fastembed"
model = "nomic-ai/nomic-embed-text-v1.5"
[storage]
backend = "sqlite"
[storage.sqlite]
path = "/home/user/.vestige/vestige.db"
"#;
#[cfg(feature = "postgres-backend")]
const EXAMPLE_POSTGRES: &str = r#"
[embeddings]
provider = "fastembed"
model = "nomic-ai/nomic-embed-text-v1.5"
[storage]
backend = "postgres"
[storage.postgres]
url = "postgres://vestige:secret@localhost:5432/vestige"
max_connections = 10
acquire_timeout_secs = 30
"#;
#[test]
fn parses_sqlite_example() {
let cfg: VestigeConfig = toml::from_str(EXAMPLE_SQLITE).expect("parse");
match cfg.storage {
StorageConfig::Sqlite(s) => assert!(s.path.is_some()),
#[cfg(feature = "postgres-backend")]
StorageConfig::Postgres(_) => panic!("wrong variant"),
}
assert_eq!(cfg.embeddings.provider, "fastembed");
}
#[cfg(feature = "postgres-backend")]
#[test]
fn parses_postgres_example() {
let cfg: VestigeConfig = toml::from_str(EXAMPLE_POSTGRES).expect("parse");
match cfg.storage {
StorageConfig::Postgres(p) => {
assert_eq!(p.url, "postgres://vestige:secret@localhost:5432/vestige");
assert_eq!(p.max_connections, Some(10));
assert_eq!(p.acquire_timeout_secs, Some(30));
}
StorageConfig::Sqlite(_) => panic!("wrong variant"),
}
}
#[cfg(not(feature = "postgres-backend"))]
#[test]
fn rejects_postgres_when_feature_off() {
let toml_text = r#"
[storage]
backend = "postgres"
[storage.postgres]
url = "postgres://x/y"
"#;
let res: Result<VestigeConfig, _> = toml::from_str(toml_text);
assert!(res.is_err(), "must fail without postgres-backend feature");
}
#[test]
fn defaults_pick_sqlite() {
let cfg = VestigeConfig::default();
assert!(matches!(cfg.storage, StorageConfig::Sqlite(_)));
}
#[test]
fn load_missing_file_returns_default() {
let tmp = std::env::temp_dir().join("vestige-no-such-file.toml");
let _ = std::fs::remove_file(&tmp);
let cfg = VestigeConfig::load(Some(&tmp)).expect("missing file is OK");
assert!(matches!(cfg.storage, StorageConfig::Sqlite(_)));
}
#[test]
fn load_roundtrip_from_disk() {
let tmp = std::env::temp_dir().join("vestige-roundtrip.toml");
std::fs::write(&tmp, EXAMPLE_SQLITE).unwrap();
let cfg = VestigeConfig::load(Some(&tmp)).expect("load");
assert!(matches!(cfg.storage, StorageConfig::Sqlite(_)));
let _ = std::fs::remove_file(&tmp);
}
}
```
Run:
```bash
cargo test -p vestige-core --lib config::
cargo test -p vestige-core --lib config:: --features postgres-backend
```
### 5. Smoke: server boots with default config
```bash
# default build, no vestige.toml on disk
cargo run -p vestige-mcp -- --help
# should print help, no panic
```
---
## Acceptance criteria
- [ ] `cargo build -p vestige-core` (default features) succeeds.
- [ ] `cargo build -p vestige-core --features postgres-backend` succeeds.
- [ ] `cargo build -p vestige-mcp` (default features) succeeds.
- [ ] `cargo build -p vestige-mcp --features postgres-backend` succeeds.
- [ ] `cargo clippy` with and without `postgres-backend` is clean on both
crates.
- [ ] `crates/vestige-core/src/config.rs` exists, exposes
`VestigeConfig`, `StorageConfig`, `SqliteConfig`, `EmbeddingsConfig`,
`ConfigError`, plus `PostgresConfig` when the feature is on.
- [ ] `VestigeConfig::load(None)` returns `Ok(default)` when
`~/.vestige/vestige.toml` is missing.
- [ ] `VestigeConfig::load(Some(&path))` round-trips both the SQLite and
Postgres example blocks above.
- [ ] With `postgres-backend` off, parsing `backend = "postgres"` returns
a clear `ConfigError::Invalid` mentioning the feature flag, NOT a
panic.
- [ ] `crates/vestige-core/src/storage/postgres/pool.rs` exists,
implementing `build_pool(&PostgresConfig) -> MemoryStoreResult<PgPool>`
with the documented defaults.
- [ ] `PgMemoryStore::connect`, `connect_with`, and `from_pool` all wire
through `pool::build_pool`. None of them is `todo!()`. The registry
step is a no-op stub documented as filled in by `0002d`.
- [ ] `MemoryStoreError::Postgres(sqlx::Error)` and
`MemoryStoreError::Migrate(sqlx::migrate::MigrateError)` exist
behind `#[cfg(feature = "postgres-backend")]` with `#[from]`.
- [ ] `vestige-mcp` has a `postgres-backend` feature that forwards to
`vestige-core/postgres-backend`.
- [ ] `vestige-mcp/src/main.rs` selects SQLite vs Postgres at startup
based on `VestigeConfig`. SQLite is the default when no config file
is present.
- [ ] Unit tests in the "Verification" section pass on both feature sets.
---
## Out of scope (handled by other sub-plans)
- Migrations (`crates/vestige-core/migrations/postgres/*.sql`) -- `0002c`.
- Real `PgMemoryStore` CRUD/search/scheduling/edges bodies -- `0002d`,
`0002e`.
- `ensure_registry` real body with `ALTER COLUMN TYPE vector(N)` -- `0002d`.
- `vestige migrate --from sqlite --to postgres` CLI -- `0002f`.
- Re-embed flow -- `0002g`.
- Env-var override (`VESTIGE_POSTGRES_URL`, etc.) -- Phase 3.
- RLS, multi-tenant column population -- Phase 3.

1119
docs/plans/0002c-migrations.md
View file

File diff suppressed because it is too large Load diff

1771
docs/plans/0002d-store-impl-bodies.md
View file

File diff suppressed because it is too large Load diff

825
docs/plans/0002e-hybrid-search.md
View file

@ -1,825 +0,0 @@
# Phase 2 Sub-Plan 0002e -- Hybrid RRF Search
**Status**: Ready
**Depends on**:
- `0002a-skeleton-and-feature-gate.md` (the `postgres-backend` feature flag
exists and `PgMemoryStore` compiles with `todo!()` bodies).
- `0002b-pool-and-config.md` (a working `PgPool` reaches the backend).
- `0002c-migrations.md` (migration `0001_init` has created the `knowledge_nodes`
table with the D7 columns -- `owner_user_id`, `visibility`,
`shared_with_groups` -- and the D8 column `codebase`; migration `0002_hnsw`
has built the HNSW index).
- `0002d-store-impl-bodies.md` (real CRUD bodies exist so the integration
tests below can seed data through the trait surface rather than raw SQL).
This sub-plan covers master plan 0002 deliverable D5: the hybrid RRF search
query implementation in `crates/vestige-core/src/storage/postgres/search.rs`,
plus the `search`, `fts_search`, and `vector_search` method bodies in
`crates/vestige-core/src/storage/postgres/mod.rs` that delegate into it.
---
## Context
This is one of the more performance-sensitive sub-plans in Phase 2. Every
search call from the cognitive engine -- the 7-stage retrieval pipeline,
`session_context`, `predict`, `deep_reference`, the dashboard -- bottoms out
in `MemoryStore::search`. The Postgres backend has to keep up with the
existing SQLite hybrid path, which combines BM25 over FTS5 with USearch HNSW
in two separate round trips and fuses the rankings in Rust.
The shape of the win on Postgres is that both branches and the fusion run
inside one statement. The planner sees both CTEs together, the round trip is
single, and the rerank stage runs over a cleanly overfetched candidate set.
Latency targets live in `0002h-testing-and-benches.md`. This sub-plan is
responsible for producing a correct, schema-stable query that the benches
can drive against. Do not optimise here; correctness and structure first.
Master plan 0002 D5 (around lines 522-628 of
`docs/plans/0002-phase-2-postgres-backend.md`) sketches the SQL. That
sketch is the starting point, not the finished product. The schema after
the D7 and D8 amendments has more columns than the sketch enumerates, and
the SQLite `search` method (around line 6503 of
`crates/vestige-core/src/storage/sqlite.rs` in the Phase 1 worktree)
documents the semantics this implementation must stay compatible with:
- Empty `query.limit` defaults to 10.
- `query.text == Some("")` is treated as no text query (degrade to vector).
- `query.embedding == None` is treated as no vector query (degrade to FTS).
- Both empty returns `Ok(vec![])`; not an error.
- The `MemoryRecord` in each `SearchResult` must be populated with all
fields the trait promises, including `domains` and `domain_scores` (Phase
4 will fill these in; Phase 2 returns the stored values, which may be
empty arrays / empty objects).
---
## Constants
```rust
/// Reciprocal Rank Fusion smoothing constant from Cormack, Clarke and Buettcher
/// 2009 ("Reciprocal Rank Fusion outperforms Condorcet and individual rank
/// learning methods"). 60 is the canonical default and is robust across most
/// fusion regimes. Do not tune this without a paper-citation-grade reason.
const RRF_K: i32 = 60;
/// Each branch (FTS, vector) is allowed to return OVERFETCH_MULT x final_limit
/// rows before fusion. Three matches the Phase 1 SQLite overfetch and gives
/// the fusion enough candidates to recover from any single branch's bad
/// recall on a given query.
const OVERFETCH_MULT: i64 = 3;
```
These live at module scope in
`crates/vestige-core/src/storage/postgres/search.rs`. They are `pub(crate)`
only if `0002h-testing-and-benches.md` needs to reference them from the
integration tests; otherwise private.
---
## Public API
```rust
#![cfg(feature = "postgres-backend")]
use pgvector::Vector;
use sqlx::PgPool;
use crate::storage::memory_store::{
MemoryStoreResult, SearchQuery, SearchResult,
};
/// Hybrid RRF search over Postgres FTS and pgvector cosine distance.
///
/// Branch behavior:
/// - empty text + null embedding -> Ok(vec![])
/// - empty text + Some(embedding) -> pure vector search (FTS CTE returns
/// zero rows; fusion equals the vector
/// branch)
/// - Some(text) + null embedding -> pure FTS search
/// - Some(text) + Some(embedding) -> full RRF fusion
///
/// `query.limit == 0` is treated as 10 (matches the SQLite default).
pub(crate) async fn rrf_search(
pool: &PgPool,
query: &SearchQuery,
) -> MemoryStoreResult<Vec<SearchResult>>;
/// FTS-only convenience search. Equivalent to calling `rrf_search` with
/// `query.embedding = None`, but uses a dedicated single-branch query that
/// avoids the FULL OUTER JOIN and the params CTE; faster by one planner pass
/// per call.
pub(crate) async fn fts_only(
pool: &PgPool,
text: &str,
limit: usize,
) -> MemoryStoreResult<Vec<SearchResult>>;
/// Vector-only convenience search. Dedicated single-branch query for the same
/// latency reason as `fts_only`.
pub(crate) async fn vector_only(
pool: &PgPool,
embedding: &[f32],
limit: usize,
) -> MemoryStoreResult<Vec<SearchResult>>;
```
### Parameter handling
In `rrf_search`:
```rust
let final_limit: i32 = if query.limit == 0 { 10 } else { query.limit as i32 };
let overfetch: i32 = (final_limit as i64 * OVERFETCH_MULT)
.min(i32::MAX as i64) as i32;
let q_text: &str = query.text.as_deref().unwrap_or("");
let q_vec: Option<Vector> = query.embedding.as_ref()
.map(|v| Vector::from(v.clone()));
let dom_filter: Option<&[String]> = query.domains.as_deref();
let nt_filter: Option<&[String]> = query.node_types.as_deref();
let tag_filter: Option<&[String]> = query.tags.as_deref();
let min_retr: Option<f64> = query.min_retrievability;
```
Both branches empty -- `q_text` is empty and `q_vec` is `None` -- returns
`Ok(vec![])` without hitting the database. The SQLite backend has the same
behavior and tests rely on it.
```rust
if q_text.is_empty() && q_vec.is_none() {
return Ok(Vec::new());
}
```
### `search` method body in `postgres/mod.rs`
```rust
#[async_trait::async_trait] // or trait_variant after the Phase 1 amendment
impl MemoryStore for PgMemoryStore {
async fn search(&self, query: &SearchQuery)
-> MemoryStoreResult<Vec<SearchResult>>
{
crate::storage::postgres::search::rrf_search(&self.pool, query).await
}
async fn fts_search(&self, text: &str, limit: usize)
-> MemoryStoreResult<Vec<SearchResult>>
{
crate::storage::postgres::search::fts_only(&self.pool, text, limit)
.await
}
async fn vector_search(&self, embedding: &[f32], limit: usize)
-> MemoryStoreResult<Vec<SearchResult>>
{
crate::storage::postgres::search::vector_only(&self.pool, embedding, limit)
.await
}
}
```
Everything below specifies the inside of those three free functions.
---
## SQL: the hybrid RRF query
The query is built as one `&'static str` (or `OnceCell<String>`; see "Use
of sqlx::query!" below) and reused. Bound parameters are kept to seven
through a `params` CTE that the rest of the query references by name --
this keeps the SQL readable and stops the bound-parameter count growing
with each filter clause.
Bound parameters:
- `$1`: text query (TEXT, may be empty)
- `$2`: embedding (pgvector::Vector, may be NULL)
- `$3`: overfetch limit per branch (INT)
- `$4`: final limit (INT)
- `$5`: domain filter (TEXT[] or NULL)
- `$6`: node_type filter (TEXT[] or NULL)
- `$7`: tag filter (TEXT[] or NULL)
If `min_retrievability.is_some()` the outer SELECT adds a JOIN on
`scheduling` and a WHERE clause; that path uses a different prepared
statement (see "min_retrievability filter" below) so the simple-path query
stays free of the join.
```sql
WITH params AS (
SELECT
$1::text AS q_text,
$2::vector AS q_vec,
$3::int AS overfetch,
$4::int AS final_limit,
$5::text[] AS dom_filter,
$6::text[] AS nt_filter,
$7::text[] AS tag_filter
),
fts AS (
SELECT
m.id,
ts_rank_cd(
m.search_vec,
websearch_to_tsquery('english', p.q_text)
) AS score,
ROW_NUMBER() OVER (
ORDER BY ts_rank_cd(
m.search_vec,
websearch_to_tsquery('english', p.q_text)
) DESC
) AS rank
FROM knowledge_nodes m
CROSS JOIN params p
WHERE p.q_text <> ''
AND m.search_vec @@ websearch_to_tsquery('english', p.q_text)
AND (p.dom_filter IS NULL OR m.domains && p.dom_filter)
AND (p.nt_filter IS NULL OR m.node_type = ANY(p.nt_filter))
AND (p.tag_filter IS NULL OR m.tags && p.tag_filter)
ORDER BY score DESC
LIMIT (SELECT overfetch FROM params)
),
vec AS (
SELECT
m.id,
1.0 - (m.embedding <=> p.q_vec) AS score,
ROW_NUMBER() OVER (
ORDER BY m.embedding <=> p.q_vec
) AS rank
FROM knowledge_nodes m
CROSS JOIN params p
WHERE m.embedding IS NOT NULL
AND p.q_vec IS NOT NULL
AND (p.dom_filter IS NULL OR m.domains && p.dom_filter)
AND (p.nt_filter IS NULL OR m.node_type = ANY(p.nt_filter))
AND (p.tag_filter IS NULL OR m.tags && p.tag_filter)
ORDER BY m.embedding <=> p.q_vec
LIMIT (SELECT overfetch FROM params)
),
fused AS (
SELECT
COALESCE(f.id, v.id) AS id,
COALESCE(1.0 / (60 + f.rank), 0.0)
+ COALESCE(1.0 / (60 + v.rank), 0.0) AS rrf_score,
f.score AS fts_score,
v.score AS vector_score
FROM fts f
FULL OUTER JOIN vec v ON f.id = v.id
)
SELECT
m.id AS "id!: uuid::Uuid",
m.owner_user_id AS "owner_user_id!: uuid::Uuid",
m.visibility AS "visibility!: String",
m.shared_with_groups AS "shared_with_groups!: Vec<uuid::Uuid>",
m.codebase AS "codebase: String",
m.domains AS "domains!: Vec<String>",
m.domain_scores AS "domain_scores!: serde_json::Value",
m.content AS "content!: String",
m.node_type AS "node_type!: String",
m.tags AS "tags!: Vec<String>",
m.embedding AS "embedding: pgvector::Vector",
m.metadata AS "metadata!: serde_json::Value",
m.created_at AS "created_at!: chrono::DateTime<chrono::Utc>",
m.updated_at AS "updated_at!: chrono::DateTime<chrono::Utc>",
fused.rrf_score AS "rrf_score!: f64",
fused.fts_score AS "fts_score: f64",
fused.vector_score AS "vector_score: f64"
FROM fused
JOIN knowledge_nodes m ON m.id = fused.id
ORDER BY fused.rrf_score DESC
LIMIT (SELECT final_limit FROM params);
```
Notes on the SELECT column list. The D7 columns (`owner_user_id`,
`visibility`, `shared_with_groups`) and the D8 column (`codebase`) are
selected even though Phase 2 does not filter on them yet, so:
1. The `MemoryRecord` returned to the trait can be populated with the
stored values from day one. Phase 3 will start writing real
`owner_user_id` / `visibility` values; Phase 2 always writes the
single-user defaults (`'00000000-...-0001'`, `'private'`, `'{}'`). The
`MemoryRecord` returned in Phase 2 simply carries those defaults.
2. The schema-drift integration tests (see "Verification") catch the case
where someone adds a NOT NULL column to `knowledge_nodes` without updating
this query.
Notes on the body:
- `CROSS JOIN params p` is used instead of the master-plan sketch's
`FROM knowledge_nodes m, params p`. Same plan, clearer intent.
- The `ORDER BY ... LIMIT` inside each branch CTE is there so the planner
can stop early once it has `overfetch` rows; without it the LIMIT is
applied after a full sort over all matches.
- `1.0 - (m.embedding <=> p.q_vec)` converts pgvector's cosine *distance*
to cosine *similarity* in [0, 1] for the `vector_score` output. RRF
itself does not need the similarity -- it uses ranks -- but the trait
surface exposes `vector_score: Option<f64>` for caller introspection.
- `RRF_K = 60` is inlined as `60` in the SQL string. A `const` formatter
feels tidier but `60` is a literature constant; spell it out and leave a
comment in the Rust source: `// 60 == RRF_K (Cormack 2009)`.
- `FULL OUTER JOIN` is required: a row that the FTS branch finds and the
vector branch does not must still appear in `fused`, and vice versa.
- `COALESCE(..., 0.0)` on each `1.0 / (60 + rank)` term handles the
no-match-from-this-branch case. The fusion score for a row that only the
FTS branch ranks is `1/(60 + f.rank)` exactly.
- `m.search_vec` is the generated `tsvector` column created in migration
`0001_init` (see D4 of the master plan).
---
## Result row mapping
`sqlx::query_as::<_, SearchRow>` reads each row into a private struct that
owns the column types exactly as they come back from Postgres. The struct
is converted into a `SearchResult` after fetch.
```rust
#[derive(sqlx::FromRow)]
struct SearchRow {
id: uuid::Uuid,
owner_user_id: uuid::Uuid,
visibility: String,
shared_with_groups: Vec<uuid::Uuid>,
codebase: Option<String>,
domains: Vec<String>,
domain_scores: serde_json::Value,
content: String,
node_type: String,
tags: Vec<String>,
embedding: Option<pgvector::Vector>,
metadata: serde_json::Value,
created_at: chrono::DateTime<chrono::Utc>,
updated_at: chrono::DateTime<chrono::Utc>,
rrf_score: f64,
fts_score: Option<f64>,
vector_score: Option<f64>,
}
impl SearchRow {
fn into_result(self) -> SearchResult {
use crate::storage::memory_store::MemoryRecord;
use std::collections::HashMap;
// domain_scores is JSONB; the column always exists, but may be the
// empty object {} when Phase 4 has not classified this memory yet.
let domain_scores: HashMap<String, f64> =
serde_json::from_value(self.domain_scores).unwrap_or_default();
let record = MemoryRecord {
id: self.id,
domains: self.domains,
domain_scores,
content: self.content,
node_type: self.node_type,
tags: self.tags,
// pgvector::Vector -> Vec<f32>
embedding: self.embedding.map(|v| v.to_vec()),
created_at: self.created_at,
updated_at: self.updated_at,
metadata: self.metadata,
// owner_user_id / visibility / shared_with_groups / codebase
// do not appear on MemoryRecord yet. Phase 3 will decide whether
// to extend MemoryRecord or surface them via a side channel.
// For Phase 2 they are read but discarded here.
};
SearchResult {
record,
score: self.rrf_score,
fts_score: self.fts_score,
vector_score: self.vector_score,
}
}
}
```
Type mapping summary:
| SQL type | Rust type | Notes |
|-------------------|--------------------------------------|------------------------------------------------|
| UUID | `uuid::Uuid` | requires sqlx `uuid` feature |
| TEXT | `String` | |
| TEXT NULL | `Option<String>` | used for `codebase` |
| TEXT[] | `Vec<String>` | for `tags`, `domains` |
| UUID[] | `Vec<uuid::Uuid>` | for `shared_with_groups` |
| JSONB | `serde_json::Value` | for `metadata`, `domain_scores` |
| TIMESTAMPTZ | `chrono::DateTime<chrono::Utc>` | requires sqlx `chrono` feature |
| VECTOR(N) NULL | `Option<pgvector::Vector>` | `.map(|v| v.to_vec())` to `Option<Vec<f32>>` |
| FLOAT8 | `f64` | |
| FLOAT8 NULL | `Option<f64>` | for `fts_score`, `vector_score` |
If `MemoryRecord` is extended in Phase 3 to carry `owner_user_id`,
`visibility`, `shared_with_groups`, and `codebase`, the conversion above
gets four more fields. Phase 2 reads them so the integration tests can
assert on them via SQL, but the trait surface does not expose them yet.
---
## `fts_only` and `vector_only` -- dedicated single-branch queries
The master plan offers two options for the convenience methods: reuse
`rrf_search` with one branch nulled, or write dedicated queries. The
dedicated queries win:
- One CTE instead of three. Planner picks the obvious plan.
- No FULL OUTER JOIN.
- No `params` indirection -- bound parameters used directly.
- The output `score` is the branch's native score (BM25-ish `ts_rank_cd` /
cosine similarity), not an RRF fusion score over one branch. Callers of
`fts_search` and `vector_search` get an intuitive score back.
### `fts_only`
Bound parameters:
- `$1`: text query (TEXT, must be non-empty; the caller guards `text.is_empty()`)
- `$2`: limit (INT)
```sql
SELECT
m.id AS "id!: uuid::Uuid",
m.owner_user_id AS "owner_user_id!: uuid::Uuid",
m.visibility AS "visibility!: String",
m.shared_with_groups AS "shared_with_groups!: Vec<uuid::Uuid>",
m.codebase AS "codebase: String",
m.domains AS "domains!: Vec<String>",
m.domain_scores AS "domain_scores!: serde_json::Value",
m.content AS "content!: String",
m.node_type AS "node_type!: String",
m.tags AS "tags!: Vec<String>",
m.embedding AS "embedding: pgvector::Vector",
m.metadata AS "metadata!: serde_json::Value",
m.created_at AS "created_at!: chrono::DateTime<chrono::Utc>",
m.updated_at AS "updated_at!: chrono::DateTime<chrono::Utc>",
ts_rank_cd(m.search_vec, websearch_to_tsquery('english', $1))
AS "fts_score!: f64"
FROM knowledge_nodes m
WHERE m.search_vec @@ websearch_to_tsquery('english', $1)
ORDER BY ts_rank_cd(m.search_vec, websearch_to_tsquery('english', $1)) DESC
LIMIT $2;
```
The Rust caller maps each row to a `SearchResult` with:
```rust
SearchResult {
record,
score: fts_score,
fts_score: Some(fts_score),
vector_score: None,
}
```
If `text.is_empty()` the caller returns `Ok(Vec::new())` before hitting
the database. `websearch_to_tsquery('english', '')` returns an empty
tsquery that matches nothing; the round-trip is wasted work otherwise.
### `vector_only`
Bound parameters:
- `$1`: embedding (pgvector::Vector)
- `$2`: limit (INT)
```sql
SELECT
m.id AS "id!: uuid::Uuid",
m.owner_user_id AS "owner_user_id!: uuid::Uuid",
m.visibility AS "visibility!: String",
m.shared_with_groups AS "shared_with_groups!: Vec<uuid::Uuid>",
m.codebase AS "codebase: String",
m.domains AS "domains!: Vec<String>",
m.domain_scores AS "domain_scores!: serde_json::Value",
m.content AS "content!: String",
m.node_type AS "node_type!: String",
m.tags AS "tags!: Vec<String>",
m.embedding AS "embedding: pgvector::Vector",
m.metadata AS "metadata!: serde_json::Value",
m.created_at AS "created_at!: chrono::DateTime<chrono::Utc>",
m.updated_at AS "updated_at!: chrono::DateTime<chrono::Utc>",
1.0 - (m.embedding <=> $1) AS "vector_score!: f64"
FROM knowledge_nodes m
WHERE m.embedding IS NOT NULL
ORDER BY m.embedding <=> $1
LIMIT $2;
```
The Rust caller maps each row to:
```rust
SearchResult {
record,
score: vector_score,
fts_score: None,
vector_score: Some(vector_score),
}
```
Both convenience methods ignore `SearchQuery.domains` / `tags` /
`node_types` / `min_retrievability` -- they take `&str` and `&[f32]`
respectively, not a `SearchQuery`. Callers that want filters on a
single-branch search should call `search` with the other branch input
left at its degrade-to-zero default.
---
## `min_retrievability` filter
`SearchQuery::min_retrievability: Option<f64>` is applied as a final
filter after fusion by joining on the `scheduling` table:
```sql
-- with-min-retrievability variant: identical CTEs to the base query, only
-- the final SELECT changes.
SELECT
... (same column list as the base query) ...
FROM fused
JOIN knowledge_nodes m ON m.id = fused.id
JOIN scheduling s ON s.memory_id = m.id
WHERE s.retrievability >= $8
ORDER BY fused.rrf_score DESC
LIMIT (SELECT final_limit FROM params);
```
This is a separate prepared statement -- the eight-parameter variant --
held alongside the seven-parameter base. The Rust dispatch:
```rust
if let Some(min_r) = query.min_retrievability {
sqlx::query_as::<_, SearchRow>(QUERY_WITH_MIN_R)
.bind(q_text)
.bind(q_vec)
.bind(overfetch)
.bind(final_limit)
.bind(dom_filter)
.bind(nt_filter)
.bind(tag_filter)
.bind(min_r)
.fetch_all(pool).await?
} else {
sqlx::query_as::<_, SearchRow>(QUERY_BASE)
.bind(q_text)
.bind(q_vec)
.bind(overfetch)
.bind(final_limit)
.bind(dom_filter)
.bind(nt_filter)
.bind(tag_filter)
.fetch_all(pool).await?
}
```
Why not unconditionally join: the `scheduling` join is expensive enough on
a large `knowledge_nodes` table that adding it to every search call regresses the
common path. `min_retrievability` is set by the cognitive engine's
accessibility filter and is `None` in most direct callers.
The same two-variant pattern repeats for `fts_only` and `vector_only`; in
practice callers of those methods rarely set `min_retrievability` (it is
not part of their argument list), so only the base variant is needed
unless the trait surface grows.
---
## Domain / tag / node_type filters
Each filter is expressed as a NULL-conditional clause inside both branch
CTEs, written using PostgreSQL array operators:
```sql
AND (p.dom_filter IS NULL OR m.domains && p.dom_filter)
AND (p.nt_filter IS NULL OR m.node_type = ANY(p.nt_filter))
AND (p.tag_filter IS NULL OR m.tags && p.tag_filter)
```
- `&&` is the PostgreSQL "arrays overlap" operator. Matches if any
element in `m.domains` is in the filter array. Index-friendly with a
GIN index on `m.domains` (created in `0001_init`).
- `= ANY(...)` matches `m.node_type` (a scalar) against any element of
the filter array. Index-friendly with a B-tree on `m.node_type`.
- `&&` is used again on `m.tags` (a `TEXT[]`).
The NULL-conditional form is critical: when the filter parameter is
`NULL`, the clause short-circuits to `TRUE` and contributes nothing to
the WHERE. This keeps a single query reusable across "no filter" and
"filter set" cases without rewriting SQL.
When the Rust caller passes `None` for a filter, sqlx binds it as `NULL`
of the column type (`text[]`). The cast `$5::text[]` in the `params` CTE
is what tells sqlx the binding type.
The master plan's draft has each filter clause duplicated across both
branch CTEs. That duplication is correct -- the planner cannot push a
WHERE clause across a FULL OUTER JOIN into both sides automatically; we
do it manually.
---
## Empty-string text query handling
The base query guards the FTS branch with `WHERE p.q_text <> ''`.
`websearch_to_tsquery('english', '')` returns an empty tsquery. An empty
tsquery has no lexemes and matches no document; the `@@` operator returns
false for every row. Without the guard, the FTS branch would still run --
sequential scan, tokenisation per row, comparison -- and return zero
rows. The guard short-circuits at planning time.
The guard does not affect the FULL OUTER JOIN: when the FTS branch
returns zero rows, the join degenerates to "every row that the vector
branch returned, with `f.id IS NULL` and `f.rank IS NULL`". The
`COALESCE(1.0 / (60 + f.rank), 0.0)` then evaluates to `0.0`, and the
fused score reduces to the vector branch's RRF term alone. This is the
"pure vector search" degrade path.
Symmetrically, the vector branch guards itself with
`WHERE m.embedding IS NOT NULL AND p.q_vec IS NOT NULL`, which gives the
"pure FTS search" degrade path when the caller passes no embedding.
The both-empty case (`q_text == ''` and `q_vec IS NULL`) is intercepted
in Rust before the query runs and returns `Ok(vec![])`. Returning empty
rather than error matches the SQLite behavior and is what the Phase 1
ingest pipeline relies on for "no signal, no results" fallback.
---
## Use of `sqlx::query!` versus `sqlx::query_as`
`sqlx::query!` and `sqlx::query_as!` are compile-time-checked: the SQL is
sent to a live Postgres at build time, the result schema is validated, and
the generated Rust struct fields are derived. That checking is the
default for every other query in `0002d-store-impl-bodies.md`.
For the RRF query, the macro path is impractical for two reasons:
1. **Two structurally different queries** -- the base (seven parameters,
no `scheduling` join) and the `min_retrievability` variant (eight
parameters, with the join). The macro would force two macro
invocations, each producing its own anonymous result struct, and the
result types would not unify. Manual `From` impls would be needed in
both directions.
2. **The dedicated `fts_only` and `vector_only` queries have a different
output column set** (`fts_score!` instead of `rrf_score! + fts_score? +
vector_score?`). Three macro invocations, three structs, three
conversion helpers.
The chosen pattern is `sqlx::query_as::<_, SearchRow>(SQL_CONST)` with a
single `SearchRow` struct that owns the column types and a single
`SearchRow::into_result` helper. The SQL is held in module-scope `&'static
str` constants:
```rust
const QUERY_BASE: &str = include_str!("search.rrf.sql");
const QUERY_WITH_MIN_R: &str = include_str!("search.rrf.min_retr.sql");
const QUERY_FTS_ONLY: &str = include_str!("search.fts.sql");
const QUERY_VECTOR_ONLY: &str = include_str!("search.vector.sql");
```
`include_str!` keeps the SQL out of the Rust source. The four `.sql`
files live next to `search.rs` in
`crates/vestige-core/src/storage/postgres/`.
The cost: schema drift (someone renames `m.codebase` to `m.repo_name`)
will not break the build. The integration tests in "Verification" below
are the safety net. This is a deliberate trade -- it is the one sub-plan
in Phase 2 where runtime flexibility beats compile-time checking.
If a future contributor wants compile-time checking back for the simple
case, the right move is to introduce a `#[cfg(test)]`-only macro-checked
variant of `QUERY_BASE` and assert at test build time that the macro
agrees with the string. That belongs in `0002h-testing-and-benches.md` if
anywhere.
---
## Verification
Integration tests live in
`crates/vestige-core/tests/postgres_search.rs`, gated by
`#[cfg(feature = "postgres-backend")]` and `#[ignore]` by default (the
test runner CI workflow in `0002h-testing-and-benches.md` runs them with
`--ignored` against a live Postgres).
Common harness for every test:
1. Spin up Postgres via `sqlx::PgPool::connect` against the test URL.
2. Run `sqlx::migrate!("./migrations/postgres").run(&pool)` to bring the
schema up.
3. Register a deterministic test embedder via `register_model` so
`embedding` columns can be written.
4. Seed 50 mixed memories through `MemoryStore::insert` -- mixed
`node_type` (`fact`, `concept`, `event`, `decision`), mixed `tags`
(`rust`, `postgres`, `search`, `dream`, `bug-fix`), mixed `codebase`,
embeddings drawn from three small clusters so vector recall has
structure to find.
Test cases:
**T1. Full RRF returns the seeded target.**
Insert a known memory with `content = "FSRS-6 spaced repetition cadence"`
and an embedding from cluster A. Query with
`text = Some("FSRS spaced repetition")` and an embedding near cluster A.
Assert the target memory is in the top 3 of the returned `SearchResult`s
and that both `fts_score` and `vector_score` are `Some` for it.
**T2. Pure FTS degrade.**
Same target as T1. Query with `text = Some("FSRS spaced repetition")` and
`embedding = None`. Assert the target appears, all results have
`vector_score == None`, `fts_score == Some(_)`, and `score` equals the
fused RRF score (which collapses to one branch's `1.0/(60 + rank)`).
**T3. Pure vector degrade.**
Same target as T1. Query with `text = Some("")` and
`embedding = Some(cluster_A_vector)`. Assert the target appears, all
results have `fts_score == None`, `vector_score == Some(_)`.
**T4. Both empty returns `Ok(vec![])`.**
Query with `text = Some("")` and `embedding = None`. Assert exactly an
empty result vector and that no SQL was executed (assert via a
`sqlx::PgPool` query-count handle if convenient; otherwise document that
the short-circuit lives in Rust).
**T5. `domains` filter.**
Insert one memory with `domains = vec!["domain-x"]` and 49 others without
it. Query with `domains = Some(vec!["domain-x"])` and a matching text.
Assert exactly one result is returned and it is the seeded memory.
**T6. `tags` filter.**
Same pattern as T5 with `tags = Some(vec!["bug-fix"])`.
**T7. `node_types` filter.**
Same pattern as T5 with `node_types = Some(vec!["decision"])`.
**T8. `min_retrievability` filter.**
Seed two memories with the same content + embedding. Write
`scheduling` rows so that one has `retrievability = 0.9` and the other
`0.1`. Query with `min_retrievability = Some(0.5)`. Assert exactly the
high-retrievability memory is returned.
**T9. `query.limit == 0` defaults to 10.**
Seed 30 matching memories. Query with `limit = 0`. Assert the result
contains exactly 10 entries.
**T10. `fts_only` and `vector_only` parity.**
For the same target memory, call `fts_only` and `vector_only` directly
and compare against `search` with the corresponding branch zeroed. The
top-1 result must match by id; the scores need only be of the same sign
and magnitude (not bit-identical, because RRF fusion changes the
absolute score).
**T11. Schema-drift canary.**
Run the base query against an empty `knowledge_nodes` table and `fetch_all`
into `Vec<SearchRow>`. Any added NOT NULL column on `knowledge_nodes` that is
not in the SELECT will fail the test at the `try_get` boundary with a
clear error. This is the test that compensates for not using
`sqlx::query!`.
**T12. Hostile inputs.**
Query with `text = Some("'; DROP TABLE knowledge_nodes; --")` and a normal
embedding. Assert no panic, results returned cleanly, `knowledge_nodes` table
still present. This is symbolic; `websearch_to_tsquery` is parameter-
bound and SQL injection is not actually possible, but the test is cheap
and the assertion is real.
---
## Acceptance criteria
A reviewer of the implementation PR should be able to confirm:
1. `crates/vestige-core/src/storage/postgres/search.rs` exists and is
compiled only when `feature = "postgres-backend"` is on.
2. The four `.sql` files (`search.rrf.sql`,
`search.rrf.min_retr.sql`, `search.fts.sql`, `search.vector.sql`)
exist in the same directory and are `include_str!`-ed into module-
scope `&'static str` constants.
3. `RRF_K = 60` and `OVERFETCH_MULT = 3` are defined as constants at
module scope with the Cormack 2009 citation in a comment.
4. The seven-parameter base query is one statement and uses a `params`
CTE; the eight-parameter `min_retrievability` variant adds exactly
one JOIN and one WHERE clause on top of the base.
5. Empty text degrades to pure vector; null embedding degrades to pure
FTS; both empty short-circuits to `Ok(vec![])` in Rust before the
query runs.
6. The SELECT column list in every query includes `owner_user_id`,
`visibility`, `shared_with_groups`, and `codebase` even though Phase 2
does not filter on them.
7. `SearchRow::into_result` populates a `MemoryRecord` with every field
the trait requires, including `domains` and `domain_scores` decoded
from JSONB.
8. `PgMemoryStore::search`, `PgMemoryStore::fts_search`, and
`PgMemoryStore::vector_search` each delegate to the corresponding
free function with one line of body.
9. All twelve integration tests (`T1` through `T12`) pass against a live
Postgres with the `0001_init` + `0002_hnsw` migrations applied.
10. `cargo build -p vestige-core` succeeds with
`--features postgres-backend` and with the feature off.
11. `cargo clippy -p vestige-core --features postgres-backend -- -D warnings`
is clean.
When all eleven are true, this sub-plan is done and
`0002f-migrate-cli.md` is unblocked.

1045
docs/plans/0002f-migrate-cli.md
View file

File diff suppressed because it is too large Load diff

843
docs/plans/0002g-reembed.md
View file

@ -1,843 +0,0 @@
# Sub-plan 0002g -- Re-embed driver and `vestige migrate reembed` CLI
**Status**: Draft
**Master plan**: [0002-phase-2-postgres-backend.md](0002-phase-2-postgres-backend.md)
**ADR**: [0002-phase-2-execution.md](../adr/0002-phase-2-execution.md)
**Predecessor**: [0002f-migrate-cli.md](0002f-migrate-cli.md)
---
## Context
This sub-plan delivers master plan deliverable **D9** -- the bulk re-embedding
driver -- and the `vestige migrate reembed` arm of the CLI scaffolded by
**D10** in sub-plan `0002f`. After this sub-plan lands, an operator can run:
```
vestige migrate reembed \
--postgres-url postgresql://localhost/vestige \
--model nomic-ai/nomic-embed-text-v1.5 \
--dimension 768
```
and the running Postgres backend will:
1. Stream every row out of `knowledge_nodes`.
2. Re-encode `content` with the requested `Embedder`.
3. Write the new vectors back.
4. Adjust the pgvector typmod if the new dimension differs from the old.
5. Rebuild the HNSW index.
6. Update the `embedding_model` registry row with the new
`(name, dimension, hash)` signature.
The whole operation runs as a single offline maintenance step. Search MUST NOT
be served during the window because partially re-embedded tables mix old and
new vector spaces and produce meaningless rankings.
This sub-plan deliberately does NOT:
- Migrate vectors between backends. That's `0002f` (SQLite -> Postgres copy).
- Invent new embedder constructors. The CLI resolves `--model` via the
existing `FastembedEmbedder::new()` constructor; the master plan's
`Embedder::from_name(&str)` factory does not exist yet (see "CLI wiring"
below for the actual call shape).
- Add a `vestige migrate reembed --sqlite-path ...` arm. SQLite re-embedding
is out of Phase 2 scope; the SQLite store's registry already handles model
drift detection via `MemoryStoreError::EmbeddingMismatch`, and the
recommended user path is "migrate to Postgres then re-embed there".
---
## Dependencies
- `0002a-skeleton-and-feature-gate.md` -- `PgMemoryStore` exists.
- `0002b-pool-and-config.md` -- `connect` builds a real `PgPool`.
- `0002c-migrations.md` -- `idx_knowledge_nodes_embedding_hnsw` and the
`embedding_model` registry row exist; `0002_hnsw.up.sql` defines the index.
- `0002d-store-impl-bodies.md` -- `register_model` and the internal
`update_registry_for_reembed` helper exist on `PgMemoryStore`.
- `0002e-hybrid-search.md` -- not technically required by reembed itself,
but the verification step at the bottom of this plan uses
`vector_search`.
- `0002f-migrate-cli.md` -- provides the `clap` scaffolding under
`vestige migrate ...`. This sub-plan adds the `reembed` subcommand and
does not redo the top-level wiring.
If `0002f` has not landed, the work order is: do the clap scaffolding from
`0002f` first (even the SQLite-to-Postgres half can be `todo!()` initially),
then this sub-plan.
---
## Audit step (do this first)
Before writing `reembed.rs`, confirm the live shape of the supporting code.
From the repo root:
```bash
rg -nF 'embed_batch' crates/vestige-core/src/
rg -nF 'register_model' crates/vestige-core/src/storage/
rg -nF 'idx_knowledge_nodes_embedding_hnsw' crates/vestige-core/migrations/postgres/
rg -nF 'update_registry_for_reembed' crates/vestige-core/src/storage/postgres/
```
Expected findings:
- `LocalEmbedder::embed_batch(&[&str]) -> Vec<Vec<f32>>` exists (Phase 1).
- `register_model` is on the `MemoryStore` trait (Phase 1) and has a real body
on `PgMemoryStore` after `0002d`.
- `idx_knowledge_nodes_embedding_hnsw` is the canonical HNSW index name. If
`0002c-migrations.md` chose a different name, update the SQL constants in
`reembed.rs` accordingly.
- `update_registry_for_reembed` is the helper added by `0002d` that updates
the existing registry row instead of inserting a new one. If it is not
present at audit time, this sub-plan adds it as part of the work (see
"Driver fn", step 7).
---
## Cargo manifest additions
No new crates. `sqlx`, `futures`, `uuid`, and `tokio` are already in
`vestige-core` from earlier sub-plans. `tracing` is already used throughout
Phase 2.
The CLI binary (`vestige-mcp/src/bin/cli.rs`) needs `clap` (already there),
`humantime` (already there for the migrate copy progress), and nothing else.
---
## Plan struct
`crates/vestige-core/src/storage/postgres/reembed.rs`:
```rust
#![cfg(feature = "postgres-backend")]
/// Tunables for the re-embed driver.
///
/// Defaults match the master plan's recommendation: medium batch, drop the
/// HNSW index before bulk writes, rebuild the index in plain mode (not
/// CONCURRENTLY) because the operator is expected to gate search anyway.
#[derive(Debug, Clone)]
pub struct ReembedPlan {
/// Number of memories embedded per `embed_batch` call and per `UPDATE`.
/// Default 128. Larger batches reduce SQL round-trips at the cost of
/// peak RAM (batch_size vectors of `4 * new_dim` bytes each, plus the
/// corresponding text strings).
pub batch_size: usize,
/// Drop `idx_knowledge_nodes_embedding_hnsw` before the bulk UPDATE pass so
/// each row write does not trigger an HNSW insertion. The index is
/// rebuilt after all rows are written. Default true.
pub drop_hnsw_first: bool,
/// Build the rebuilt HNSW index with `CREATE INDEX CONCURRENTLY`.
/// This avoids holding an `AccessExclusiveLock` on `knowledge_nodes`, at the
/// cost of running outside any transaction (see "CREATE INDEX
/// CONCURRENTLY caveats" below). Default false; flip it on when the
/// re-embed window has to overlap live traffic AND the operator has
/// already gated writes some other way.
pub concurrent_index: bool,
}
impl Default for ReembedPlan {
fn default() -> Self {
Self {
batch_size: 128,
drop_hnsw_first: true,
concurrent_index: false,
}
}
}
```
The defaults match the master plan. `concurrent_index = false` is the safer
operator-default because plain `CREATE INDEX` can run inside the same script
that drove the writes; `CONCURRENTLY` requires careful autocommit handling
(see caveats section).
---
## Report struct
```rust
/// Summary of one re-embed run. Returned by `run_reembed` and surfaced by
/// the CLI as a one-line summary (and as `--dry-run` output, where the
/// duration fields are estimates instead of measurements).
pub struct ReembedReport {
/// Number of `knowledge_nodes` rows whose `embedding` column was rewritten.
/// Includes rows whose embedding was previously NULL.
pub rows_updated: u64,
/// Wall time from the first row stream to the registry update,
/// excluding HNSW rebuild. Seconds with sub-millisecond precision.
pub duration_secs: f64,
/// Wall time of the HNSW rebuild step alone. Tracked separately
/// because it dominates total time on large tables and the operator
/// wants to know how much of the window was spent waiting for the
/// index versus encoding text.
pub index_rebuild_secs: f64,
}
```
The CLI prints all three fields. Tests assert on `rows_updated` only;
durations are non-deterministic.
---
## Driver fn
```rust
use std::sync::Arc;
use std::time::Instant;
use futures::TryStreamExt;
use sqlx::Row;
use uuid::Uuid;
use crate::embedder::Embedder;
use crate::storage::MemoryStoreResult;
use crate::storage::postgres::PgMemoryStore;
pub async fn run_reembed(
store: &PgMemoryStore,
new_embedder: Arc<dyn Embedder>,
plan: ReembedPlan,
) -> MemoryStoreResult<ReembedReport>;
```
Step-by-step:
### 1. No-op check (registry comparison)
Read the current registry row. If `(name, dimension, hash)` already matches
`new_embedder.signature()`, log "registry matches; nothing to re-embed" and
return `ReembedReport { rows_updated: 0, duration_secs: 0.0,
index_rebuild_secs: 0.0 }`.
```rust
let current = store.registered_model().await?; // Phase 1 trait method
let target = new_embedder.signature();
if current.is_some_and(|c| c == target) {
tracing::info!("registry already matches target embedder; no-op");
return Ok(ReembedReport { rows_updated: 0, duration_secs: 0.0, index_rebuild_secs: 0.0 });
}
```
This is the cheapest precondition. It also guards against accidental
double-runs after a successful re-embed.
### 2. Drop HNSW (optional)
If `plan.drop_hnsw_first`:
```sql
DROP INDEX IF EXISTS idx_knowledge_nodes_embedding_hnsw;
```
This avoids HNSW insert work on every UPDATE. Recommended default. The index
gets rebuilt in step 6.
If the operator declines (`drop_hnsw_first = false`), the UPDATE pass is much
slower on large tables but the index never goes through an empty/half state.
This is the safer-but-slower path used when the table is small enough that
rebuild cost matters more than write throughput.
### 3. Stream `(id, content)`
Stream all rows in primary-key order so progress reporting is monotone and
restarts can resume by id-greater-than:
```rust
let mut stream = sqlx::query!(
"SELECT id, content FROM knowledge_nodes ORDER BY id"
).fetch(store.pool());
let mut batch_ids: Vec<Uuid> = Vec::with_capacity(plan.batch_size);
let mut batch_texts: Vec<String> = Vec::with_capacity(plan.batch_size);
```
`fetch(pool)` returns a streaming cursor backed by a single connection;
rows arrive in chunks (sqlx default 50) without materialising the whole
result set in RAM.
### 4. Batched re-encode + UPDATE
For each row arriving from the stream:
```rust
while let Some(row) = stream.try_next().await? {
batch_ids.push(row.id);
batch_texts.push(row.content);
if batch_ids.len() >= plan.batch_size {
flush_batch(&new_embedder, store, &mut batch_ids, &mut batch_texts).await?;
}
}
if !batch_ids.is_empty() {
flush_batch(&new_embedder, store, &mut batch_ids, &mut batch_texts).await?;
}
```
`flush_batch` builds a `Vec<&str>` view, calls `new_embedder.embed_batch`,
then writes the result back. The Phase 1 `LocalEmbedder` trait exposes
`async fn embed_batch(&self, texts: &[&str]) -> Vec<Vec<f32>>`; this is
present on every embedder including `FastembedEmbedder`, so the loop never
needs to fall back to per-row `embed`. (If a future embedder lacks a real
batch implementation, the trait blanket impl is the place to add a per-row
fallback, not this driver.)
The write SQL:
```sql
UPDATE knowledge_nodes
SET embedding = v.embedding
FROM UNNEST($1::uuid[], $2::vector[]) AS v(id, embedding)
WHERE knowledge_nodes.id = v.id;
```
**Note on `UNNEST($2::vector[])`.** pgvector exposes `vector` as a base
type, and Postgres `UNNEST` does support arrays of base types. In practice,
sqlx's `pgvector::Vector` crate provides `PgHasArrayType` for `Vector`, so
`Vec<pgvector::Vector>` binds to `vector[]`. If a build catches the master
plan's snag where `vector[]` round-tripping is rejected by pgvector or by
sqlx (the master plan hedges on this), fall back to one UPDATE per row:
```sql
UPDATE knowledge_nodes SET embedding = $1::vector WHERE id = $2;
```
executed in a `sqlx::Transaction` batched per `plan.batch_size`. Slower by
a constant factor (~5x in benchmarking, dominated by per-statement overhead
rather than encoding) but always works. **Document the choice in the file
header** so a future reader knows why the slow path may be live.
### 5. Dimension change (relax-then-tighten)
If `new_embedder.dimension() != current.dimension`:
```sql
ALTER TABLE knowledge_nodes ALTER COLUMN embedding TYPE vector($NEW_DIM);
```
This MUST happen after every row has a vector of the new dimension. pgvector
validates the column typmod on write; mixing dimensions during the UPDATE
pass would be rejected. See "ALTER TABLE typmod relaxation" below for the
mechanics.
If the dimension is unchanged, skip this step.
### 6. Rebuild HNSW
```sql
CREATE INDEX idx_knowledge_nodes_embedding_hnsw
ON knowledge_nodes USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);
```
(Use the exact `WITH` parameters from `0002_hnsw.up.sql`. Do not invent new
ones here.)
If `plan.concurrent_index`, prepend `CONCURRENTLY` and run on a raw
autocommit connection -- see caveats section.
Time this step separately and record in `index_rebuild_secs`. On a
100k-row table at 768D, expect roughly 30-90 seconds on local fastembed
hardware; on 1M rows expect several minutes.
### 7. Update registry
Call the `update_registry_for_reembed` helper added by `0002d`:
```rust
store.update_registry_for_reembed(&new_embedder.signature()).await?;
```
If `0002d` lands without that helper (because at that point reembed wasn't
the use case), this sub-plan adds it. The body is a single SQL statement:
```sql
UPDATE embedding_model
SET model_name = $1,
dimension = $2,
model_hash = $3,
updated_at = now()
WHERE id = 1;
```
(`embedding_model` is a single-row table keyed by a fixed `id = 1`; the
master plan establishes this in D6.)
### 8. Return
```rust
Ok(ReembedReport {
rows_updated,
duration_secs: total_start.elapsed().as_secs_f64() - index_rebuild_secs,
index_rebuild_secs,
})
```
---
## Memory bounds
The driver is designed to use bounded memory regardless of table size.
In flight at any moment:
- `batch_ids: Vec<Uuid>` -- 16 bytes per id; 128 entries = 2 KB.
- `batch_texts: Vec<String>` -- average row content size, call it 1 KB;
128 entries = ~128 KB.
- `batch_vectors: Vec<Vec<f32>>` -- `dimension * 4 bytes` per vector;
768D * 4 * 128 = ~393 KB.
Worst case at 768D and batch 128: well under 1 MB of live heap. Multiply by
2 or 3 if the operator overrides `--batch-size` to thousands.
Crucially: the row stream from sqlx is a real cursor, not a buffered
`fetch_all`. The driver never loads the full table into RAM. Tested at 1M
rows on a 16 GB dev box; peak RSS for the reembed process stays under
200 MB, dominated by the embedder model weights, not the row data.
---
## ALTER TABLE typmod relaxation
pgvector columns carry a typmod -- the dimension. Writes against a column
declared as `vector(768)` are validated to be 768-dimensional; writes
against `vector` (no typmod) are accepted at any dimension.
To re-embed into a different dimension, the typmod has to be relaxed before
the writes and tightened after. Three approaches were considered:
### Approach A (recommended): write at the OLD dimension, then ALTER TYPE
If the new dimension equals the old dimension, this section is moot.
If the new dimension differs:
1. Drop HNSW.
2. Run the UPDATE pass writing vectors of the NEW dimension. **This works
because** pgvector's typmod check is liberal during the brief window
when a column is being mass-updated -- specifically, the per-row check
happens against the column's declared typmod, which is still the OLD
dimension. **This step fails** unless we widen the column first.
Approach A as stated does not actually work. Cross it out and use B.
### Approach B (recommended for real): widen to untyped `vector`, write, then tighten
1. Drop HNSW.
2. `ALTER TABLE knowledge_nodes ALTER COLUMN embedding TYPE vector;` -- removes
the typmod entirely. pgvector accepts this (the cast from `vector(768)`
to `vector` is identity at the storage level; only the metadata
changes). Verify on the live build that this DDL succeeds; pgvector
versions before 0.5 may reject it, in which case Approach C is the
fallback.
3. UPDATE pass writes new-dimension vectors. The column has no typmod
constraint to fight against.
4. `ALTER TABLE knowledge_nodes ALTER COLUMN embedding TYPE vector($NEW_DIM);`
-- reinstates the typmod at the new dimension. pgvector validates every
existing row; if any row has the wrong dimension the ALTER fails. This
is the integrity gate.
5. Rebuild HNSW with the new dimension implicitly in scope.
### Approach C (fallback): drop-and-add column
If Approach B fails on the live pgvector version:
1. Drop HNSW.
2. `ALTER TABLE knowledge_nodes ADD COLUMN embedding_new vector($NEW_DIM);`
3. UPDATE pass writes into `embedding_new`.
4. `ALTER TABLE knowledge_nodes DROP COLUMN embedding;`
5. `ALTER TABLE knowledge_nodes RENAME COLUMN embedding_new TO embedding;`
6. Rebuild HNSW.
Approach C is safer (it never relaxes the typmod) but slower (drop-column
is a full-table rewrite, then rename is metadata-only). It also briefly
doubles disk usage during step 3 because both columns coexist.
**Implementation:** start with Approach B. Add a code comment pointing at
Approach C as the fallback if a tested pgvector version refuses the
typmod relaxation in step 2. The migration SQL fragments for both
approaches live alongside each other in `reembed.rs` as private const
strings; the driver picks at runtime based on a probe query
(`SELECT atttypmod FROM pg_attribute WHERE ... ;` after step 2; if the
typmod is still nonzero, fall through to Approach C).
---
## CREATE INDEX CONCURRENTLY caveats
`CREATE INDEX CONCURRENTLY`:
- Cannot run inside a transaction. sqlx's default `query.execute(&pool)`
uses an implicit transaction in some configurations; explicit
autocommit is required.
- Takes roughly 2-3x as long as plain `CREATE INDEX` because it does
two table scans.
- Can fail late (after most of the work is done) if a concurrent write
conflicts; the resulting index is left in `INVALID` state and must be
dropped before retrying.
Implementation pattern:
```rust
async fn rebuild_hnsw_concurrent(pool: &PgPool) -> MemoryStoreResult<()> {
let mut conn = pool.acquire().await?;
// sqlx acquires a connection in autocommit mode; the trick is to
// NOT wrap this in a `begin().await?` transaction.
sqlx::query(
"CREATE INDEX CONCURRENTLY idx_knowledge_nodes_embedding_hnsw \
ON knowledge_nodes USING hnsw (embedding vector_cosine_ops) \
WITH (m = 16, ef_construction = 64)"
)
.execute(&mut *conn)
.await?;
Ok(())
}
```
If the index already exists (because a prior run partially succeeded),
the operator must run `DROP INDEX idx_knowledge_nodes_embedding_hnsw;`
themselves before retrying. The driver intentionally does NOT auto-drop
in CONCURRENTLY mode because that could mask a real schema problem.
For the default `concurrent_index = false` path, use plain
`CREATE INDEX ...` against `pool.execute(...)`; transactions are fine.
---
## dry_run mode
```rust
pub async fn dry_run_reembed(
store: &PgMemoryStore,
new_embedder: Arc<dyn Embedder>,
plan: &ReembedPlan,
) -> MemoryStoreResult<DryRunSummary>;
pub struct DryRunSummary {
pub rows_to_update: u64,
pub embedder_batches: u64,
pub estimated_walltime_secs: f64,
pub current_signature: ModelSignature,
pub target_signature: ModelSignature,
pub would_alter_typmod: bool,
}
```
Behaviour:
1. `SELECT COUNT(*) FROM knowledge_nodes;` to get `rows_to_update`.
2. `embedder_batches = ceil(rows_to_update / plan.batch_size)`.
3. `estimated_walltime_secs = rows_to_update / 50.0` -- the master plan's
50-rows-per-second baseline for local fastembed. Add a 30s flat fee for
the HNSW rebuild on tables under 100k rows; scale linearly past that.
4. `would_alter_typmod = current_signature.dimension != target_signature.dimension`.
5. Print everything to stderr in a human-friendly summary; emit JSON on
stdout if `--json` is set.
6. Return without writing anything.
The dry-run path performs zero embedder calls and zero `knowledge_nodes` writes.
It is safe to run against production at any time.
---
## CLI wiring
The `clap` subcommand surface, extending what `0002f` already added:
```rust
#[derive(Subcommand)]
#[cfg(feature = "postgres-backend")]
enum MigrateAction {
/// Copy SQLite -> Postgres. Owned by 0002f.
Copy { /* ... see 0002f ... */ },
/// Re-embed all memories in a Postgres backend with a new embedder.
Reembed(ReembedArgs),
}
#[derive(clap::Args)]
#[cfg(feature = "postgres-backend")]
struct ReembedArgs {
/// Postgres URL of the target backend.
#[arg(long)]
postgres_url: String,
/// Embedder model name. Today only `nomic-ai/nomic-embed-text-v1.5`
/// is supported (the FastembedEmbedder default). The argument is
/// kept so a future embedder factory can resolve other names
/// without changing the CLI surface.
#[arg(long)]
model: String,
/// Vector dimension produced by the embedder. Cross-checked against
/// the embedder's `dimension()` at startup; mismatch is a fatal
/// error before any writes occur.
#[arg(long)]
dimension: usize,
/// Embedder + UPDATE batch size. Default 128.
#[arg(long, default_value_t = 128)]
batch_size: usize,
/// Drop idx_knowledge_nodes_embedding_hnsw before the UPDATE pass.
/// Default true.
#[arg(long, default_value_t = true)]
drop_hnsw_first: bool,
/// Use CREATE INDEX CONCURRENTLY for the rebuild. Default false.
#[arg(long, default_value_t = false)]
concurrent_index: bool,
/// Print the plan without writing anything.
#[arg(long, default_value_t = false)]
dry_run: bool,
}
```
The handler:
```rust
async fn run_reembed_cli(args: ReembedArgs) -> anyhow::Result<()> {
let embedder: Arc<dyn Embedder> = resolve_embedder(&args.model)?;
if embedder.dimension() != args.dimension {
anyhow::bail!(
"embedder '{}' produces dimension {}, --dimension was {}",
embedder.model_name(), embedder.dimension(), args.dimension,
);
}
let store = PgMemoryStore::connect(&args.postgres_url, 4).await?;
let plan = ReembedPlan {
batch_size: args.batch_size,
drop_hnsw_first: args.drop_hnsw_first,
concurrent_index: args.concurrent_index,
};
if args.dry_run {
let summary = dry_run_reembed(&store, embedder, &plan).await?;
print_dry_run(&summary);
return Ok(());
}
let report = run_reembed(&store, embedder, plan).await?;
print_report(&report);
Ok(())
}
fn resolve_embedder(model: &str) -> anyhow::Result<Arc<dyn Embedder>> {
// Today, Phase 1 provides exactly one Embedder constructor:
// FastembedEmbedder::new(). The master plan calls out a future
// `Embedder::from_name(&str)` factory that does not yet exist.
// Until that factory lands, this function accepts only the
// FastembedEmbedder's `model_name()` value and errors on anything
// else. Adding a real registry is a follow-up task.
let candidate = FastembedEmbedder::new();
if candidate.model_name() == model {
return Ok(Arc::new(candidate));
}
anyhow::bail!(
"unknown embedder model '{}'. Known: {}",
model,
candidate.model_name(),
);
}
```
**Important honesty note for the implementer:** the master plan claims
`Embedder::from_name(&str)` already exists in Phase 1. As of audit (see
"Audit step" above), it does not. This sub-plan ships the
`FastembedEmbedder::new()` matcher and leaves the factory pattern for a
future change. Do not block on inventing the factory just to satisfy the
master plan's wording -- doing so expands scope without a real second
embedder to use it.
The CLI invocation matches the form requested in the master plan:
```
vestige migrate reembed \
--postgres-url postgresql://localhost/vestige \
--model nomic-ai/nomic-embed-text-v1.5 \
--dimension 768 \
--batch-size 128 \
--drop-hnsw-first \
--dry-run
```
---
## Failure handling
The driver makes a single, important promise: **between step 4 (UPDATE
pass) and step 7 (registry update), the database is in an inconsistent
state**. Specifically:
- Rows already processed in step 4 carry vectors in the NEW embedding
space.
- Rows not yet processed carry vectors in the OLD embedding space.
- The `embedding_model` registry still says OLD.
- The HNSW index is dropped (if `drop_hnsw_first = true`).
If the driver crashes, is killed, loses its DB connection, or the
operator hits Ctrl-C in this window, the partial state is broken in a
specific way: a `vector_search` against the table would mix vectors
from two different model spaces, producing nonsensical similarity
rankings. The operator MUST NOT serve search until the re-embed
completes.
**Recovery procedure** (document this loudly in the operator-facing log):
1. The CLI log already says, on every batch, `"reembed: wrote batch N
(M rows)"`. The last such log line indicates how far the pass got.
2. The recovery action is to **re-run reembed** with the same arguments.
The driver's step 1 (no-op check) will see that the registry still
says OLD and will re-do the work. The UPDATE pass overwrites rows
that were already re-embedded (harmless; the new vector is
deterministic per content), and processes the rest.
3. Once the second run completes through step 7, the table is
consistent again.
The driver logs a one-time WARNING at startup, before any writes:
```
WARN: vestige migrate reembed is starting. Search results will be
WARN: incorrect until this run completes. Stop the MCP server now if
WARN: it is connected to this database. Press Ctrl-C within 5 seconds
WARN: to abort.
```
The 5-second pause is implemented with `tokio::time::sleep` and can be
suppressed with `--no-confirm` for scripted use.
There is no "resume from row N" feature in this iteration. Re-embedding
is idempotent at the row level (same content + same embedder = same
vector), so a full re-run is correct, just wasteful. If the table grows
large enough that full re-runs are unacceptable, a follow-up adds a
checkpoint table; that is out of Phase 2 scope.
---
## Verification
### Unit tests (colocated in `reembed.rs`)
1. **`reembed_no_op_when_signature_matches`** -- seed a `PgMemoryStore`
via testcontainers, register a fake embedder dim=64, call
`run_reembed` with the same fake embedder, assert the returned
`ReembedReport.rows_updated == 0` and that no embedder calls were
made (use a counter-wrapped fake).
2. **`reembed_plan_defaults`** -- `ReembedPlan::default()` returns
`batch_size = 128`, `drop_hnsw_first = true`,
`concurrent_index = false`.
3. **`reembed_dry_run_returns_summary_without_writing`** -- seed 50
rows, call `dry_run_reembed`, assert `rows_to_update == 50` and
that the original embeddings are untouched.
### Integration test (under `tests/phase_2/pg_reembed.rs`)
Acceptance test that exercises the dimension-change path end to end:
```rust
#![cfg(feature = "postgres-backend")]
use std::sync::Arc;
mod common;
use common::test_embedder::{FakeEmbedder, FakeEmbedderConfig};
use common::pg_harness::PgHarness;
#[tokio::test]
async fn reembed_changes_dimension_and_search_still_works() {
let old = Arc::new(FakeEmbedder::new(FakeEmbedderConfig {
name: "fake-old",
dimension: 64,
}));
let harness = PgHarness::start(old.clone()).await.unwrap();
// Seed 100 memories. Each gets a 64-d vector from `old`.
for i in 0..100 {
let content = format!("memory number {i} talks about rust and async");
let vec = old.embed(&content).await.unwrap();
harness.store.insert(/* ... record with embedding = vec ... */).await.unwrap();
}
// Now re-embed with a different fake at dim 128.
let new = Arc::new(FakeEmbedder::new(FakeEmbedderConfig {
name: "fake-new",
dimension: 128,
}));
let report = run_reembed(
&harness.store,
new.clone(),
ReembedPlan::default(),
).await.unwrap();
assert_eq!(report.rows_updated, 100);
// (a) Every row has a 128-d vector.
let dims: Vec<i32> = sqlx::query_scalar(
"SELECT vector_dims(embedding) FROM knowledge_nodes"
).fetch_all(harness.store.pool()).await.unwrap();
assert!(dims.iter().all(|&d| d == 128));
// (b) Registry reflects the new signature.
let sig = harness.store.registered_model().await.unwrap().unwrap();
assert_eq!(sig.name, "fake-new");
assert_eq!(sig.dimension, 128);
// (c) vector_search returns results in the new space.
let probe = new.embed("memory number 5 talks about rust and async").await.unwrap();
let results = harness.store.vector_search(&probe, 10).await.unwrap();
assert!(!results.is_empty());
}
```
The `FakeEmbedder` from `common/test_embedder.rs` produces deterministic
vectors by hashing the input; both the seed and the search probe use the
same hash, so the test does not depend on actual semantic similarity.
### Bench (optional, not gating)
A simple benchmark in `crates/vestige-core/benches/reembed.rs` reports
throughput at 100k rows with `FakeEmbedder`. Useful for catching
regressions in the UPDATE-pass batching pattern. Not part of CI.
---
## Acceptance criteria
This sub-plan is complete when:
1. `crates/vestige-core/src/storage/postgres/reembed.rs` exists and
compiles under `--features postgres-backend`.
2. `ReembedPlan` and `ReembedReport` are public types matching the
shapes in this document.
3. `run_reembed` implements the eight numbered steps in the Driver fn
section, including the no-op short-circuit at step 1 and the
typmod relaxation logic at step 5.
4. `dry_run_reembed` returns counts and estimates without writing.
5. The `vestige migrate reembed ...` subcommand is wired through
`crates/vestige-mcp/src/bin/cli.rs`, gated on `--features
postgres-backend`, validating `--dimension` against
`embedder.dimension()`.
6. The three unit tests pass.
7. The `pg_reembed.rs` integration test passes against the
testcontainer harness from `0002h` (or against a locally provisioned
pgvector instance if `0002h` is not yet merged).
8. The operator-facing WARN banner is printed before any writes and
honours `--no-confirm`.
9. The recovery semantics from "Failure handling" are documented in
the module-level rustdoc of `reembed.rs`, so a future operator
reading `cargo doc` sees the "you must re-run to completion before
serving search" rule without finding this sub-plan first.
10. `cargo sqlx prepare --workspace` updates `.sqlx/` with the new
queries; the resulting JSON files are committed.
When all ten items are checked, sub-plan `0002g` lands. Master plan
deliverable D9 is satisfied. The remaining Phase 2 work is `0002h`
(testing and benches) and `0002i` (runbook).

1223
docs/plans/0002h-testing-and-benches.md
View file

File diff suppressed because it is too large Load diff

724
docs/plans/0002i-runbook.md
View file

@ -1,724 +0,0 @@
# Phase 2 Sub-Plan 0002i -- Postgres Ops Runbook
**Status**: Ready
**Depends on**: Phase 2 sub-plans 0002a through 0002h merged (or at least
their interfaces stable). The runbook documents behaviour produced by those
sub-plans: feature gate, config schema, migrations, `vestige migrate` CLI,
hybrid search, and the test harness. Nothing in this sub-plan compiles or
runs; the deliverable is a single Markdown file.
This sub-plan covers Phase 2 master-plan deliverable D16 only: a one-page
operator-facing runbook for deploying Vestige with the Postgres backend.
---
## Context
Why a runbook. The ADR (0002) and the master plan (0002) are written for
implementors. They settle execution-level decisions and itemise deliverables.
They are not deployable instructions. A separate document is needed for the
operator who has to install pgvector, take backups, recover from a failed
re-embed, and decide whether to roll a migration back. The runbook is that
document.
Who reads it. Ops people, not developers. Concretely: someone who has a
shell on a Linux host, knows how to use `psql` and `systemctl`, and has been
handed a built `vestige-mcp` binary plus a `vestige.toml`. They are not
expected to read Rust source or follow internal Cargo features. They do
know what a backup is, what a connection pool is, and how to read a
PostgreSQL log.
In scope: deployment of the Postgres backend on a single host or a small
cluster, day-to-day monitoring, scheduled and ad-hoc backups, embedding
migration via `vestige migrate reembed`, and troubleshooting the failure
modes most likely to land in an operator's lap.
Out of scope: local development setup -- that lives in
`docs/plans/local-dev-postgres-setup.md` and the runbook links to it for
developer onboarding only. Network exposure of the Vestige HTTP API
(Phase 3), federation (Phase 5), Postgres TLS / certificate handling, and
multi-tenant operation are also out of scope; the runbook explicitly
flags them as "see Phase N" so operators do not improvise.
This sub-plan is the plan for producing the runbook. It outlines the
runbook structure, inlines the runbook body as the canonical "this is what
the file should say" text, and lists acceptance criteria. The implementation
agent for D16 copies the inlined body into `docs/runbook/postgres.md`,
creating `docs/runbook/` if it does not already exist. No other files in the
repository are modified.
---
## Deliverable
The artifact produced by executing this sub-plan is exactly one new file:
```
docs/runbook/postgres.md
```
It is NOT under `docs/plans/`. Plans describe how Vestige gets built;
runbooks describe how Vestige gets operated. The two directories are
deliberately separated.
Side effect: create the directory `docs/runbook/` if it does not exist.
Do not add an index file, README, or any other content under `docs/runbook/`
in this sub-plan -- only `postgres.md`.
This sub-plan document (`docs/plans/0002i-runbook.md`) is itself NOT a
deliverable in the operator sense. It is the plan for producing the runbook,
and lives under `docs/plans/` with the other Phase 2 sub-plans.
---
## Runbook structure
The runbook is organised as a flat list of ten sections, in order. Operators
read it top to bottom on first deployment; subsequent visits jump to a
specific section. Section numbering matches the inlined body below.
1. **Prerequisites** -- what must already be installed and available on the
host before Vestige even tries to connect. PostgreSQL 16 or newer
(18 on Arch is fine), pgvector >= 0.5, pgcrypto (for `gen_random_uuid`),
sufficient disk for the HNSW index, OS user permissions on the data
directory.
2. **Initial setup** -- one-time tasks: create the database role, create
the database, install required extensions, and lay down an initial
`vestige.toml`. Includes the canonical `CREATE EXTENSION` calls and a
minimal config snippet.
3. **First connect** -- what happens the first time `vestige-mcp` starts
against an empty `vestige` database: sqlx applies the bundled
migrations, `register_model` stamps the embedding column type, and the
registry row is written. How an operator verifies each step succeeded
using `psql`.
4. **Connection pool tuning** -- default of 10 connections per
`vestige-mcp` instance, when to raise it, how to size the Postgres
server-side `max_connections` and `shared_buffers` accordingly. Cross-
reference to `vestige.toml` and to ADR 0002 D2 / open question Q5.
5. **Backup discipline** -- `pg_dump` and `pg_restore` invocations,
recommended frequency, which tables matter (knowledge_nodes and scheduling
are critical and not regenerable; review_events is append-only and
replayable from clients; edges are reconstructable from spreading
activation runs; domains can be recomputed by Phase 4 once it ships).
Also covers backup verification (restore-to-tmp drill).
6. **Migration between embeddings** -- the `vestige migrate reembed`
workflow: when an operator needs it (model upgrade, dim change),
downtime expectations, how to verify completion via the
`embedding_model` registry and HNSW presence, and how to recover from
an interrupted run.
7. **Re-clustering domains** -- a brief forward reference. Domain
clustering is owned by Phase 4 (`docs/plans/0004-phase-4-emergent-domain-classification.md`);
until Phase 4 ships, operators should not invoke any re-clustering
workflow manually. The runbook section is intentionally one paragraph
long and points at the Phase 4 plan.
8. **Monitoring** -- the small set of pg_catalog and pg_stat_* queries
that answer "is Vestige healthy?": `pg_stat_activity` for stuck queries,
`pg_stat_statements` for query patterns (if the extension is enabled),
index sizes for the HNSW, and how to spot a half-built HNSW after a
failed migration.
9. **Troubleshooting** -- a table of common errors with the symptom and
the fix. Extension missing, pool exhausted, embedding dimension
mismatch, FTS language config (`'english'` vs `'simple'`), migrations
partially applied.
10. **Rollback caveats** -- every `*.up.sql` has a `*.down.sql`, but
downgrades destroy data (HNSW gets dropped, vector column type
reverts, domain rows vanish). The runbook tells operators to always
take a backup before applying a new migration, even though sqlx will
do its best to be idempotent.
---
## Runbook body
The full text below is what should be copied verbatim into
`docs/runbook/postgres.md`. ASCII only. Code blocks use fenced syntax with
language hints. Operator-facing prose; second person ("you") for
instructions. Where a command requires sudo, the prompt shows it explicitly.
```markdown
# Vestige Postgres Backend -- Operator Runbook
This runbook covers deploying, operating, monitoring, and recovering a
Vestige installation that uses the Postgres backend. It is written for
operators handling a built `vestige-mcp` binary and a `vestige.toml`.
For local development setup, see
`docs/plans/local-dev-postgres-setup.md`. For the architectural rationale,
see `docs/adr/0001-pluggable-storage-and-network-access.md` and
`docs/adr/0002-phase-2-execution.md`. For the deliverable-level plan, see
`docs/plans/0002-phase-2-postgres-backend.md`.
---
## 1. Prerequisites
Before Vestige can connect:
- PostgreSQL server, version 16 or newer. Arch ships 18.x; Debian stable
ships 16.x; both work.
- `pgvector` extension, version 0.5 or newer. Distro packages:
`pgvector` on Arch, `postgresql-16-pgvector` on Debian/Ubuntu.
- `pgcrypto` extension, shipped with the PostgreSQL contrib package
(`postgresql-contrib` on Debian, included in the base `postgresql`
package on Arch). Vestige uses `gen_random_uuid()` from pgcrypto for
primary keys.
- Disk space: budget roughly 4x the size of your `knowledge_nodes.embedding`
column for the HNSW index. With 768-dim float32 vectors at 100k
memories, that is about 1.2 GB for the embeddings plus 4-5 GB for the
HNSW index. Plan accordingly.
- OS user: the `postgres` system user (or whatever user owns
`/var/lib/postgres/data`) must have read/write on the data directory.
Vestige itself does not need filesystem access to Postgres; it talks
TCP only.
- Network: Vestige and Postgres can be on the same host (loopback) or
different hosts. If different hosts, allow the Vestige host's IP in
`pg_hba.conf` and on any firewall.
---
## 2. Initial setup
These steps run once per Postgres cluster.
### 2.1 Install extensions
As the `postgres` superuser:
```sh
sudo -u postgres psql -d vestige <<'SQL'
CREATE EXTENSION IF NOT EXISTS vector;
CREATE EXTENSION IF NOT EXISTS pgcrypto;
SQL
```
Verify:
```sh
sudo -u postgres psql -d vestige -c \
"SELECT extname, extversion FROM pg_extension WHERE extname IN ('vector','pgcrypto');"
```
You should see two rows. If `vector` is missing, the pgvector package was
not installed for the right PostgreSQL major version; reinstall it.
### 2.2 Create the role and database
The `vestige` role owns its own database; it does NOT need superuser.
Extensions must be installed by `postgres`, not by `vestige`.
```sh
sudo -u postgres psql -v ON_ERROR_STOP=1 <<'SQL'
CREATE ROLE vestige WITH LOGIN CREATEDB PASSWORD 'CHANGE_ME';
CREATE DATABASE vestige OWNER vestige ENCODING 'UTF8';
GRANT ALL PRIVILEGES ON DATABASE vestige TO vestige;
SQL
sudo -u postgres psql -d vestige -v ON_ERROR_STOP=1 <<'SQL'
GRANT ALL ON SCHEMA public TO vestige;
ALTER SCHEMA public OWNER TO vestige;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO vestige;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO vestige;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON FUNCTIONS TO vestige;
SQL
```
Replace `CHANGE_ME` with a strong password and store it where Vestige can
read it (typically `~/.vestige_pg_pw`, mode 600, owned by the user running
`vestige-mcp`).
### 2.3 Minimal `vestige.toml`
```toml
[storage]
backend = "postgres"
[storage.postgres]
url = "postgresql://vestige:CHANGE_ME@127.0.0.1:5432/vestige"
max_connections = 10
```
The `url` field accepts a `${VAR}` placeholder; in practice operators
either inline the password or export `DATABASE_URL` and reference
`url = "${DATABASE_URL}"`. See `docs/CONFIGURATION.md` for the full
schema once Phase 3 lands.
---
## 3. First connect
When `vestige-mcp` starts against an empty `vestige` database, it:
1. Builds a `PgPool` of `max_connections` (default 10) connections.
2. Runs every migration in `crates/vestige-core/migrations/postgres/`
in order. The bundled migrations are `0001_init` (tables, non-vector
indexes) and `0002_hnsw` (HNSW index on `knowledge_nodes.embedding`).
3. Calls `register_model` once it knows the active embedder's dimension.
This issues `ALTER TABLE knowledge_nodes ALTER COLUMN embedding TYPE
vector($N)` and inserts a row into `embedding_model`.
4. Begins accepting MCP requests.
To verify after the first start:
```sh
sudo -u postgres psql -d vestige <<'SQL'
-- All expected tables present.
\dt
-- embedding_model has exactly one row.
SELECT name, dimension, hash FROM embedding_model;
-- The HNSW index exists.
SELECT indexname FROM pg_indexes
WHERE tablename = 'knowledge_nodes' AND indexname LIKE '%hnsw%';
SQL
```
Expected: `knowledge_nodes`, `scheduling`, `edges`, `domains`, `review_events`,
`embedding_model`, `users`, `groups`, `group_memberships`; one row in
`embedding_model`; one `idx_knowledge_nodes_embedding_hnsw` index.
If a migration fails mid-way, the partial state lands in
`_sqlx_migrations`. See section 9 for recovery.
---
## 4. Connection pool tuning
Defaults:
- Vestige client pool: `max_connections = 10` per `vestige-mcp` instance.
- Postgres server: `max_connections = 100` (default).
Math: one MCP client with the default pool uses up to 10 server slots.
Five concurrent MCP clients use up to 50 slots. The remaining 50 cover
`psql` sessions, background workers, and headroom for replication or
backup processes.
When to raise:
- More than three MCP clients connecting to one Postgres instance.
- Long-running queries (above 500ms p99) showing pool wait time in
Vestige logs (look for `pool acquire timed out` warnings).
- A noticeable number of concurrent dream/consolidation runs.
How to raise:
```toml
[storage.postgres]
max_connections = 20 # client side, per vestige-mcp instance
```
And on the Postgres server, edit `postgresql.conf`:
```conf
max_connections = 200
shared_buffers = 2GB # roughly 25 percent of RAM, never above 8GB
```
Then restart Postgres (`sudo systemctl restart postgresql`). Vestige
clients pick up their own `max_connections` change on next restart.
Do not raise pool sizes blindly. Past about 4x the CPU core count,
Postgres throughput drops; a small connection pooler (PgBouncer in
transaction mode) is the right answer above ~200 client connections,
but Vestige's expected scale rarely needs that.
---
## 5. Backup discipline
### 5.1 Which tables matter
| Table | Backup priority | Regenerable? |
|-------|-----------------|--------------|
| `knowledge_nodes` | Critical | No |
| `scheduling` | Critical | No (FSRS state) |
| `embedding_model` | Critical | No (one row, but stamps the column type) |
| `users`, `groups`, `group_memberships` | Critical | No (Phase 3 will populate) |
| `review_events` | Important | Replayable by clients but tedious |
| `edges` | Optional | Yes (recomputed by spreading activation) |
| `domains` | Optional | Yes (Phase 4 recomputes by clustering) |
For a typical single-operator install, dumping the whole database is
fastest and simplest. Skip the optional tables only if dump size becomes
a bandwidth problem.
### 5.2 Full logical backup
```sh
pg_dump --host=127.0.0.1 --username=vestige --format=custom \
--file=vestige-$(date -u +%Y%m%dT%H%M%SZ).dump \
vestige
```
The custom format compresses by default and works with parallel restore.
File size for 10k memories: roughly 80 MB.
Frequency recommendations:
- Daily for any installation with active ingest.
- Before every `vestige migrate reembed` run (see section 6).
- Before every Postgres major-version upgrade.
- Retain at least 7 daily, 4 weekly, 3 monthly dumps. Compress with
`--format=custom` (already gzipped) and keep them on different
storage from the database itself.
### 5.3 Restore
To a fresh database:
```sh
sudo -u postgres createdb -O vestige vestige_restore
pg_restore --host=127.0.0.1 --username=vestige --dbname=vestige_restore \
--jobs=4 vestige-20260301T030000Z.dump
```
To replace the live database (destructive; only after taking a fresh
dump):
```sh
sudo systemctl stop vestige-mcp # or however the service is run
sudo -u postgres dropdb vestige
sudo -u postgres createdb -O vestige vestige
pg_restore --host=127.0.0.1 --username=vestige --dbname=vestige \
--jobs=4 vestige-20260301T030000Z.dump
sudo systemctl start vestige-mcp
```
### 5.4 Restore drill
Run a restore-to-throwaway-database every month and run `vestige search`
or a manual `psql` count against it. A backup you have not restored is a
backup you do not have.
```sh
sudo -u postgres createdb -O vestige vestige_restore_drill
pg_restore --host=127.0.0.1 --username=vestige --dbname=vestige_restore_drill \
--jobs=4 vestige-latest.dump
PGPASSWORD="$(cat ~/.vestige_pg_pw)" psql -h 127.0.0.1 -U vestige \
-d vestige_restore_drill \
-c 'SELECT count(*) FROM knowledge_nodes;'
sudo -u postgres dropdb vestige_restore_drill
```
---
## 6. Migration between embeddings
Use `vestige migrate reembed` when:
- Upgrading to a new embedding model that produces a different dimension
(for example, swapping from `nomic-embed-text-v1.5` 768D to a 1024D
model).
- Switching providers and the model hash differs even at the same
dimension.
What it does:
1. Reads every row from `knowledge_nodes`, re-encodes the `content` column
through the new embedder, and writes the new vector back.
2. Drops the HNSW index before the re-encode loop (this is the default;
`--concurrent-index` keeps it during the run at the cost of speed).
3. Updates the `embedding_model` row with the new name, dimension, and
hash.
4. Rebuilds the HNSW index with the new vectors.
### 6.1 Before starting
- Take a fresh backup (section 5.2). The tool refuses to start without a
`--yes` flag if it detects no recent backup; ignore at your peril.
- Stop ingest. Vestige's MCP server can stay running for read-only
access, but pause any client that calls `smart_ingest` or
`update_scheduling`.
- Have the new embedder model available locally. The CLI loads it
before the first row is touched; if loading fails, no data is changed.
### 6.2 Running
```sh
vestige migrate reembed --model=<new-model-name> --yes
```
Add `--concurrent-index` if you cannot accept the brief window during
HNSW rebuild where queries do not use the index (sequential scan
fallback works but is slow).
The tool prints a progress bar via `indicatif`. Expected throughput:
roughly 200 memories per second per CPU core for a 768D ONNX model.
10,000 memories on an 8-core box: about 6 seconds, plus HNSW rebuild
(another 30-90 seconds at that scale).
### 6.3 Verifying completion
```sh
sudo -u postgres psql -d vestige <<'SQL'
-- Registry reflects the new model.
SELECT name, dimension, hash FROM embedding_model;
-- HNSW index is present and not partial.
SELECT indexname, indexdef
FROM pg_indexes
WHERE tablename = 'knowledge_nodes' AND indexname LIKE '%hnsw%';
-- All rows have a non-null embedding of the new dimension.
SELECT count(*) FILTER (WHERE embedding IS NULL) AS missing,
count(*) AS total
FROM knowledge_nodes;
SQL
```
Expected: registry shows the new model name and dimension, one HNSW
index, zero missing embeddings.
### 6.4 Recovering from an interrupted run
`vestige migrate reembed` is restartable. On interruption:
- The `embedding_model` row may or may not have been updated. Check it
manually and roll forward by re-running with `--yes --resume` (the
tool detects the inconsistency and finishes the rows that still hold
old embeddings).
- The HNSW index may be missing. Re-running the command rebuilds it as
its last step.
- If the system is in a state the tool refuses to reason about, restore
from the backup taken in 6.1.
---
## 7. Re-clustering domains
Domain clustering is owned by Phase 4
(`docs/plans/0004-phase-4-emergent-domain-classification.md`). Until
Phase 4 ships, the `domains` table is reserved schema and is populated
only by tests. Operators must not invoke any domain re-clustering
workflow manually; there is no supported one in Phase 2.
When Phase 4 lands, this section is replaced with the real procedure.
---
## 8. Monitoring
### 8.1 Quick health check
```sh
PGPASSWORD="$(cat ~/.vestige_pg_pw)" psql -h 127.0.0.1 -U vestige -d vestige <<'SQL'
SELECT count(*) AS memory_count FROM knowledge_nodes;
SELECT name, dimension FROM embedding_model;
SELECT pg_size_pretty(pg_database_size('vestige')) AS db_size;
SQL
```
### 8.2 In-flight queries
```sql
SELECT pid, now() - query_start AS runtime, state, query
FROM pg_stat_activity
WHERE datname = 'vestige' AND state <> 'idle'
ORDER BY runtime DESC NULLS LAST;
```
Anything over 5 seconds with `state = 'active'` deserves a look. HNSW
search queries should land well under 100ms on properly-sized hardware.
### 8.3 Query pattern analysis
If `pg_stat_statements` is loaded (`shared_preload_libraries =
'pg_stat_statements'` in `postgresql.conf`):
```sql
SELECT calls, mean_exec_time, query
FROM pg_stat_statements
WHERE query ILIKE '%knowledge_nodes%'
ORDER BY mean_exec_time DESC
LIMIT 20;
```
Look for hybrid-search queries that have drifted above 100ms p50. The
usual culprit is a missing or half-built HNSW index.
### 8.4 Index health
```sql
SELECT indexname, pg_size_pretty(pg_relation_size(indexrelid)) AS size,
idx_scan, idx_tup_read
FROM pg_indexes
JOIN pg_stat_user_indexes USING (indexrelid)
WHERE schemaname = 'public' AND relname = 'knowledge_nodes';
```
A HNSW index with `idx_scan = 0` after several hours of traffic usually
means the planner is preferring sequential scan -- either the table is
too small to bother with the index (fine) or the index is corrupt and
needs rebuilding (`REINDEX INDEX idx_knowledge_nodes_embedding_hnsw;`).
### 8.5 Spotting half-built HNSW
After a failed migration or a crashed `reembed`:
```sql
SELECT indexname, indisvalid, indisready
FROM pg_indexes
JOIN pg_index ON indexrelid = (schemaname || '.' || indexname)::regclass
WHERE tablename = 'knowledge_nodes';
```
Any row with `indisvalid = false` is broken. Drop and recreate:
```sql
DROP INDEX IF EXISTS idx_knowledge_nodes_embedding_hnsw;
CREATE INDEX idx_knowledge_nodes_embedding_hnsw
ON knowledge_nodes USING hnsw (embedding vector_cosine_ops);
```
---
## 9. Troubleshooting
| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| `ERROR: extension "vector" is not available` on start | pgvector not installed for this Postgres major version | Install the distro package matching `pg_config --version`, then `CREATE EXTENSION vector;` as superuser |
| `pool timed out while waiting for an open connection` in Vestige logs | Pool too small or stuck queries holding connections | Raise `max_connections` in `vestige.toml`; investigate `pg_stat_activity` for queries above 5s |
| `vector dimensions do not match` on insert | `embedding_model` was stamped at one dimension and a different embedder is now running | Re-run `vestige migrate reembed --model=<correct>` or fix the embedder configuration |
| Hybrid search returns the same row twice | Stale `.sqlx/` query cache from before D5 landed | Run `cargo sqlx prepare` in `crates/vestige-core/`, rebuild the binary |
| `text search configuration "english" does not exist` | Postgres locale build does not include the english dictionary (rare on Alpine) | Install the language-pack or override the FTS language in `vestige.toml` (see `[storage.postgres.fts]` once Phase 2 D5 lands) |
| `relation "_sqlx_migrations" exists, but migration X is in "applied" with no checksum` | Previous run died between `BEGIN` and `COMMIT` | Stop Vestige, restore from backup, restart |
| HNSW index very large compared to data | `m` and `ef_construction` defaults too high for the corpus | Acceptable for now; tuning lands as part of Phase 4 |
| `permission denied for schema public` on a new install | `vestige` role does not own `public` | Re-run the grants block in section 2.2 as `postgres` |
If a problem is not in this table, capture: PostgreSQL log
(`/var/log/postgres/`, journalctl `-u postgresql`), Vestige log
(`RUST_LOG=debug,sqlx=info` for a fresh run), the migration state
(`SELECT * FROM _sqlx_migrations ORDER BY version;`), and file a bug.
---
## 10. Rollback caveats
Every migration in `crates/vestige-core/migrations/postgres/` has a
matching `*.down.sql`. `sqlx migrate revert` walks them in reverse order.
This is not the same as risk-free. The `0002_hnsw.down.sql` drops the
HNSW index (rebuildable, expensive). The `0001_init.down.sql` drops
every table -- including `knowledge_nodes`, including data. Down migrations
exist for development, not for casual production use.
Before applying any new migration:
1. Take a backup (section 5.2).
2. Run the migration on a restored copy first if you can afford the time.
3. Read the new migration's `*.up.sql` and `*.down.sql` to understand
what changes.
To revert one migration manually:
```sh
sqlx migrate revert \
--database-url "postgresql://vestige:...@127.0.0.1:5432/vestige" \
--source crates/vestige-core/migrations/postgres
```
Note that Vestige's binary does not run `sqlx migrate revert`
automatically. Reverts are always an explicit operator decision.
If a revert fails partway through, treat the database as inconsistent:
restore from the backup taken in step 1.
```
---
## Cross-references
- `docs/adr/0001-pluggable-storage-and-network-access.md` -- ADR that
established the pluggable backend.
- `docs/adr/0002-phase-2-execution.md` -- ADR settling Phase 2 execution
decisions; section "Architecture Overview" lists every table the
runbook references.
- `docs/plans/0002-phase-2-postgres-backend.md` -- master plan; D16
(deliverables list) and the Open Implementation Questions section
(especially Q4 HNSW rebuild and Q5 pool sizing) inform the runbook's
recommendations.
- `docs/plans/local-dev-postgres-setup.md` -- developer-facing recipe
for a one-machine Arch / CachyOS dev cluster. The runbook links to it
as the "for development, see" pointer.
- `docs/CONFIGURATION.md` -- existing config doc; section 4 of the
runbook ("Connection pool tuning") cross-references it for the
authoritative `vestige.toml` schema.
---
## Verification
A reviewer is given:
- A fresh Linux VM (Debian 12 or Arch current; both must work) with
network access and no Postgres installed.
- A built `vestige-mcp` binary for that platform.
- The runbook (`docs/runbook/postgres.md`).
The reviewer follows the runbook top to bottom and reaches a state in
which Vestige answers MCP requests against the Postgres backend.
Checkpoints, in order:
1. After section 1 (Prerequisites): `pg_config --version` returns 16 or
newer; `pkg-config --modversion libpq` resolves; the `pgvector`
distro package is installed.
2. After section 2.1 (Extensions): two rows in
`SELECT extname FROM pg_extension WHERE extname IN ('vector', 'pgcrypto');`.
3. After section 2.2 (Role + DB): `psql -U vestige -h 127.0.0.1 -d vestige -c '\conninfo'`
succeeds.
4. After section 2.3 (Config): `vestige.toml` parses (test by
`vestige config print` once that subcommand lands, otherwise
`vestige-mcp --check-config`).
5. After section 3 (First connect): the eight expected tables are
present; `embedding_model` has exactly one row; the HNSW index
exists; `vestige-mcp` log shows "Postgres backend ready".
6. After section 5.2 (Backup): the dump file exists and `pg_restore -l`
on it lists the expected tables.
7. After section 5.4 (Restore drill): the drill database holds the same
row count as the source.
If any checkpoint fails, the runbook section that produced the failure
is the one that needs revision. Capture the exact command, exit code,
and log line; revise the runbook in a follow-up PR.
A second reviewer reads the runbook without executing it and checks for:
- ASCII only; no em dashes, no curly quotes, no Unicode arrows, no
ellipses, no bullets (`*`/`-` ASCII only).
- Every section number from 1 to 10 present and in order.
- Every cross-reference resolves to an existing file or to a Phase
number explicitly marked as "future".
- No code block longer than 30 lines; if longer, it should be split or
referenced from another file.
---
## Acceptance criteria
- [ ] `docs/runbook/` directory exists.
- [ ] `docs/runbook/postgres.md` exists and matches the inlined body
above byte-for-byte after stripping the outer code fence used in
this sub-plan to embed it.
- [ ] All ten sections from the "Runbook structure" outline are present
under their stated headings.
- [ ] No file other than `docs/runbook/postgres.md` is created or
modified by executing this sub-plan.
- [ ] ASCII only: no em dashes, no curly quotes, no Unicode arrows,
no ellipses, no Unicode bullets (`grep -P '[^\x00-\x7F]'
docs/runbook/postgres.md` returns no matches).
- [ ] Every cross-reference in the runbook points at a file that exists
in the repository at the time of merge, OR is explicitly framed
as "future Phase N" with a pointer to the relevant plan document.
- [ ] Every command block is copy-pastable: no `<placeholder>` syntax
that does not also have an inline note describing what to
substitute.
- [ ] A second pair of eyes confirms the verification checkpoints in the
preceding section are reproducible.
- [ ] The runbook is no longer than the inlined body in this sub-plan;
operators reach the end without losing patience.

1435
docs/plans/0003-phase-3-network-access.md
View file

File diff suppressed because it is too large Load diff

883
docs/plans/0004-phase-4-emergent-domain-classification.md
View file

@ -1,883 +0,0 @@
# Phase 4 Plan: Emergent Domain Classification
**Status**: Draft
**Depends on**: Phase 1 (domain columns on memories, `Domain` struct + `DomainStore` methods on `MemoryStore`, `Embedder` trait), Phase 2 (Postgres JSONB + TEXT[] support for domain fields, `embedding_model` registry parity), Phase 3 (Axum HTTP server, REST `/api/v1/` scaffolding, API key auth middleware, signed dashboard session cookies)
**Related**: docs/adr/0001-pluggable-storage-and-network-access.md (Phase 4), docs/prd/001-getting-centralized-vestige.md (Emergent Domain Model)
---
## Scope
### In scope
- `DomainClassifier` cognitive module under `crates/vestige-core/src/neuroscience/domain_classifier.rs`, alongside existing neuroscience modules (spreading_activation, synaptic_tagging, ...).
- HDBSCAN discovery pipeline using the `hdbscan` crate (v0.10): load all embeddings, cluster, extract centroids, extract top-terms via TF-IDF over cluster members, persist via the trait's `DomainStore` methods.
- Soft-assignment pipeline: for each memory, compute `cosine_similarity(memory.embedding, domain.centroid)` for every domain, store raw scores in `domain_scores` JSONB, threshold into `domains[]` using `assign_threshold` (default 0.65).
- Automatic classification on ingest: run through `CognitiveEngine` / `smart_ingest` so new memories get classified against existing centroids immediately; skip when `domain_count == 0` (Phase 0 accumulation).
- Re-cluster hook in dream consolidation: every Nth four-phase dream cycle (N=5 default) triggers a discovery pass and generates proposals (split / merge / none). Proposals land in a new `domain_proposals` table, surface in the dashboard, and are never auto-applied (conservative drift, ADR Q7).
- Context signals: `SignalSource` trait with `GitRepoSignal` (detects `.git` in CWD or `metadata.cwd`) and `IdeHintSignal` (reads `metadata.editor` / `metadata.ide`). Each returns a `boost_map` of `domain_id -> additive delta` (typical +0.05). Injected as a `signal_boost: Option<HashMap<String, f64>>` parameter into `DomainClassifier::classify`.
- Cross-domain spreading activation decay: `ActivationNetwork` traversal multiplies the edge's effective weight by `cross_domain_decay` (default 0.5) when `target.domains` and `source.domains` are disjoint. Strict "no overlap" policy, not graded.
- CLI subcommands (in `crates/vestige-mcp/src/bin/cli.rs`, under a new `Domains` command group): `list`, `discover [--min-cluster-size N] [--force]`, `rename <id> <new_label>`, `merge <a> <b> [--into <id>]`. Human-readable tables on stdout; JSON via `--json`.
- Dashboard UI additions (`apps/dashboard/src/routes/(app)/domains/`): list page, per-domain detail (memories, centroid top_terms, score histogram, proposal review controls).
- REST endpoints under `/api/v1/domains` (introduced by Phase 3 skeleton, implemented in Phase 4): list, discover, rename, merge, proposal list / accept / reject.
- Config additions: `[domains]` section in `vestige.toml` covering `assign_threshold`, `recluster_interval`, `min_cluster_size`, `cross_domain_decay`, `discovery_threshold`, `merge_threshold`, `signal_boost` (per-signal toggle).
### Out of scope
- Phase 5 federation (explicit separate ADR). Domain centroids are installation-local; no sync.
- Learned re-weighting of domain scores (future, only if retrieval-quality metrics show a need).
- Interactive cluster-membership editing in the UI (drag-and-drop reassign) -- future enhancement.
- Multi-user domain namespaces. One domain set per installation; API keys that carry `domain_filter` just restrict access, they do not create namespaces.
- Auto-sweep of `min_cluster_size` / auto-tuned `assign_threshold` (ADR resolution Q6 + Q9: static defaults, user tunes).
- Graded cross-domain decay (`|A intersect B| / max(|A|,|B|)`) -- strict "no overlap" is the Phase 4 rule.
---
## Prerequisites
Artifacts that Phases 1-3 are expected to have landed:
- In `vestige-core`:
- `Embedder` trait (`crates/vestige-core/src/embedder/`).
- `MemoryStore` trait (`crates/vestige-core/src/storage/trait.rs` or similar) including `DomainStore` methods: `list_domains`, `get_domain`, `upsert_domain`, `delete_domain`, `classify(&[f32]) -> Vec<(String, f64)>`, plus a bulk accessor such as `all_embeddings()` (already present in sqlite.rs as `get_all_embeddings`) and a `get_all_memories_with_embeddings()` iterator for discovery. The trait must expose a method to batch-update `(domains, domain_scores)` for a memory id.
- `Domain` struct: `{ id: String, label: String, centroid: Vec<f32>, top_terms: Vec<String>, memory_count: usize, created_at: DateTime<Utc> }`.
- Columns on memories in both SQLite and Postgres: `domains TEXT[]` (or JSON array on SQLite) and `domain_scores JSONB` (or TEXT JSON on SQLite).
- The `domains` table in both backends (see PRD schema sketch).
- In `vestige-mcp`:
- Axum `/api/v1/` router prefix with auth middleware.
- CLI skeleton (`bin/cli.rs`) using `clap`; Phase 4 adds a `Domains` subcommand tree.
- REST handlers file structure ready under `crates/vestige-mcp/src/dashboard/handlers.rs` (legacy) and a dedicated REST handler under `/api/v1/`; Phase 4 adds `domains.rs` handler module.
- SvelteKit dashboard (`apps/dashboard/`) with existing `(app)/memories`, `(app)/timeline`, `(app)/stats`, etc. Phase 4 adds `(app)/domains/`.
New workspace crate additions required (added manually to `Cargo.toml`, since `cargo add` is not run from the plan):
- `hdbscan = "0.10"` in `crates/vestige-core/Cargo.toml` (feature-gated behind `domain-classification`).
- Optional: a lightweight stop-word constant inline; no external stop-word crate -- the neuroscience modules already do tokenization on whitespace + length>3 (see `dreams.rs::content_similarity`). Reuse that style; no `ndarray` needed because `hdbscan` v0.10 accepts `&[Vec<f32>]` directly (verified from PRD snippet).
- No new deps in `vestige-mcp` for Phase 4 -- CLI reuses `clap` / `colored` / `comfy-table` if already present, otherwise a hand-rolled padded print. We pick hand-rolled to avoid adding a table crate; this matches the existing style of `run_stats` in `cli.rs`.
Test fixtures:
- A JSON seed corpus checked into `tests/phase_4/fixtures/seed_500.json` containing >= 500 memories drawn from three plausible clusters. A builder function `tests/phase_4/support/fixtures.rs::build_seed_corpus()` deterministically generates or loads this corpus. Each record has `content`, `tags`, `embedding` (768D bge-base-en-v1.5; use a committed vector or a deterministic mock embedder in tests). For deterministic tests we fake embeddings by hashing content -- acceptable as long as the fake preserves cluster separability (prefix-based: "DEV-...", "INFRA-...", "HOME-..." seeds three Gaussian blobs).
- Reuse `Embedder` mock from Phase 1 tests (`MockEmbedder`) for discovery tests that need real cosine similarity.
- A minimal git-repo fixture created in a tempdir (`tempfile::tempdir` + `std::process::Command::new("git").arg("init")`) for context-signal tests.
---
## Deliverables
1. `DomainClassifier` cognitive module: struct, defaults, `classify`, `classify_with_boost`, `reassign_all`, `discover`.
2. `domain_terms` helper (TF-IDF over cluster members, returning `top_k` terms).
3. `cli domains discover` subcommand.
4. `cli domains list` / `rename` / `merge` subcommands.
5. Auto-classify hook on ingest (wired into the cognitive engine's ingest pipeline before persistence).
6. Re-cluster hook in dream consolidation (`DreamEngine::run` orchestrator gets an optional `DomainReClusterHook`; triggers every Nth dream).
7. Context signal extractor module (`crates/vestige-core/src/neuroscience/context_signals.rs`) with `SignalSource` trait + `GitRepoSignal` + `IdeHintSignal`.
8. Cross-domain spreading activation decay in `ActivationNetwork::activate` (config-driven).
9. `vestige.toml` `[domains]` section + defaults loader.
10. Dashboard UI: SvelteKit routes `(app)/domains/+page.svelte` (list), `(app)/domains/[id]/+page.svelte` (detail), `(app)/domains/proposals/+page.svelte` (review).
11. REST endpoints under `/api/v1/domains` + `/api/v1/domains/proposals`.
12. `domain_proposals` table + migration + `DomainProposal` trait methods on `MemoryStore`.
13. WebSocket event `VestigeEvent::DomainProposalCreated` so the dashboard gets a live notification after a re-cluster fires.
---
## Detailed Task Breakdown
### 1. `DomainClassifier` cognitive module
**File**: `crates/vestige-core/src/neuroscience/domain_classifier.rs`
**Export**: in `crates/vestige-core/src/neuroscience/mod.rs`, add `pub mod domain_classifier;` and re-export `pub use domain_classifier::{DomainClassifier, ClassificationResult, DomainProposal, ProposalKind};`
**Deps**: `hdbscan = "0.10"`, `serde`, `serde_json`, `chrono`, `tracing`, existing `crate::storage::Domain`, `crate::storage::MemoryStore` trait.
Struct and defaults (match PRD exactly):
```rust
pub struct DomainClassifier {
pub assign_threshold: f64, // default 0.65
pub discovery_threshold: usize, // default 150
pub recluster_interval: usize, // default 5 (every 5th dream)
pub min_cluster_size: usize, // default 10
pub min_samples: usize, // default 5 (HDBSCAN)
pub cross_domain_decay: f64, // default 0.5
pub merge_threshold: f64, // default 0.90 (centroid cosine)
pub top_terms_k: usize, // default 10
}
impl Default for DomainClassifier { ... }
```
Result types:
```rust
#[derive(Debug, Clone)]
pub struct ClassificationResult {
pub scores: HashMap<String, f64>, // raw per-domain similarities
pub domains: Vec<String>, // above assign_threshold
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum ProposalKind {
Split { parent: String, children: Vec<String> },
Merge { targets: Vec<String>, suggested_label: String },
NewCluster { top_terms: Vec<String> },
}
#[derive(Debug, Clone)]
pub struct DomainProposal {
pub id: String, // uuid v4
pub kind: ProposalKind,
pub rationale: String,
pub confidence: f64,
pub created_at: DateTime<Utc>,
pub status: ProposalStatus, // Pending | Accepted | Rejected
}
```
Key methods (all pure where possible; all pub):
```rust
impl DomainClassifier {
pub fn classify(&self, embedding: &[f32], domains: &[Domain]) -> ClassificationResult;
pub fn classify_with_boost(
&self,
embedding: &[f32],
domains: &[Domain],
boost: Option<&HashMap<String, f64>>,
) -> ClassificationResult;
pub async fn reassign_all(
&self,
store: &dyn MemoryStore,
domains: &[Domain],
) -> Result<usize, StorageError>;
pub async fn discover(
&self,
store: &dyn MemoryStore,
) -> Result<Vec<Domain>, StorageError>;
pub async fn propose_changes(
&self,
store: &dyn MemoryStore,
existing: &[Domain],
newly_discovered: &[Domain],
) -> Result<Vec<DomainProposal>, StorageError>;
pub async fn apply_proposal(
&self,
store: &dyn MemoryStore,
proposal: &DomainProposal,
) -> Result<(), StorageError>;
}
```
Behavior notes:
- `classify` returns empty `{ scores: {}, domains: [] }` iff `domains.is_empty()` (accumulation phase). This matches the PRD snippet verbatim.
- `classify_with_boost` adds the boost delta to each score AFTER cosine, before thresholding. It clamps to `[0.0, 1.0]`. Boost keys not present in `domains` are ignored.
- `reassign_all` streams memories in batches of 500 (iterator on the store) to keep memory bounded; for each memory issues a single `UPDATE memories SET domains = ?, domain_scores = ? WHERE id = ?` call. Returns count of memories whose `domains` vector actually changed.
- `discover` loads all `(id, embedding)` pairs via an `all_embeddings()` method on the store (exists under `#[cfg(all(feature = "embeddings", feature = "vector-search"))]` in `sqlite.rs::get_all_embeddings`; Phase 1 should promote this onto the trait -- if not yet promoted, add the method). Then:
1. Build `Vec<Vec<f32>>` and index -> id map.
2. `Hdbscan::default_hyper_params(&embeddings).min_cluster_size(self.min_cluster_size).min_samples(self.min_samples).build()` (exact builder depends on hdbscan 0.10 surface; see Open Question).
3. `let labels = clusterer.cluster()?;`
4. `let centers = clusterer.calc_centers(Center::Centroid, &labels)?;`
5. Group indices by label ignoring -1 (noise). For each cluster compute `top_terms` via `compute_top_terms`.
6. Preserve stable IDs where possible: match each new cluster centroid to the closest existing domain by cosine; if similarity > 0.85, reuse the existing domain id + label. Otherwise generate a fresh id `cluster_{n}` with a label derived from the first 2 terms.
7. Upsert all resulting `Domain`s via the store.
- `propose_changes` compares old vs new clusters:
- **Split**: an old domain that best-matches two or more new domains each with >= `min_cluster_size` members. Rationale: "domain `dev` is now 2 clusters of >=10 memories: `systems` and `networking`".
- **Merge**: two old domains whose centroids now satisfy `cosine > merge_threshold` get a merge proposal.
- **NewCluster**: a new cluster that doesn't match any old domain above 0.85 similarity.
- `apply_proposal` runs the split or merge against the store (reassign memberships via `reassign_all`), then marks the proposal `Accepted`. It never runs automatically -- only via the CLI or dashboard.
Helper:
```rust
fn compute_top_terms(documents: &[&str], k: usize) -> Vec<String>;
```
Uses TF-IDF with IDF computed over the entire passed-in corpus (the `documents` slice), tokenization = whitespace split, lowercase, strip non-alphanumeric, drop tokens shorter than 4 chars and a small built-in stop-word list (`the`, `and`, `for`, `that`, `with`, ...). Matches the tokenizer used in `dreams.rs::content_similarity` and `dreams.rs::extract_patterns` so behavior is predictable.
Cosine similarity helper:
```rust
fn cosine_similarity(a: &[f32], b: &[f32]) -> f64;
```
Keep the existing crate-level `cosine_similarity` if already present (check `embeddings::` or `search::`); otherwise add a private one. Returns 0.0 on dimension mismatch, panics would be a bug.
### 2. Top-terms computation helper
**File**: same module, private section.
- `fn tokenize(text: &str) -> Vec<String>`: lowercase, split on non-alphanumeric, filter len >= 4, drop stop-words.
- `fn tfidf_top_k(docs: &[&str], k: usize) -> Vec<String>`:
1. `tf[doc_idx][term] = count / total_terms`.
2. `df[term] = docs containing term`.
3. `idf[term] = log((N + 1) / (df[term] + 1)) + 1` (smoothed).
4. For each term, average `tf` across docs in the cluster; multiply by `idf`; sort desc; return top `k`.
Cluster top-terms are computed over cluster members only, with IDF over the **whole corpus** (all memory contents), not the cluster, so common words get penalized globally. Recompute global IDF once per `discover` call.
### 3. CLI subcommand: `vestige domains discover`
**File**: `crates/vestige-mcp/src/bin/cli.rs`
Add to `enum Commands`:
```rust
/// Emergent domain management
Domains {
#[command(subcommand)]
action: DomainAction,
},
```
```rust
#[derive(clap::Subcommand)]
enum DomainAction {
/// List all discovered domains
List {
#[arg(long)] json: bool,
},
/// Run HDBSCAN discovery on all embeddings and propose domains
Discover {
#[arg(long, default_value_t = 10)] min_cluster_size: usize,
/// Skip the proposal flow and write new domains directly (first-time use)
#[arg(long)] force: bool,
#[arg(long)] json: bool,
},
/// Rename a domain (by id)
Rename {
id: String,
new_label: String,
},
/// Merge two domains
Merge {
a: String,
b: String,
#[arg(long)] into: Option<String>, // default: `a`
},
}
```
Handler plumbing lives in `run_domains(action)` dispatching to `run_domains_list`, `run_domains_discover`, `run_domains_rename`, `run_domains_merge`. Each opens the default `Storage`, constructs a `DomainClassifier::default()`, and invokes the appropriate method.
Output format for `list`:
```
ID LABEL MEMORIES TOP TERMS
dev Development 87 rust, trait, async, tokio, zinit
infra Infrastructure 47 bgp, sonic, vlan, frr, peering
home Home 31 solar, kwh, battery, pool, esphome
(unclassified) 12
```
Produced via plain `print!` with `%-15s %-18s %-10d %s` style padding. `--json` emits `serde_json::to_string_pretty(&domains)`.
Output format for `discover` with `--force`:
```
HDBSCAN: 500 embeddings, min_cluster_size=10, min_samples=5
Found 3 clusters (ignoring 14 noise points)
cluster_0 (N=47) top: bgp, sonic, vlan, frr, peering
cluster_1 (N=31) top: solar, kwh, battery, pool, esphome
cluster_2 (N=22) top: rust, trait, async, tokio, zinit
Writing 3 domains to the store...
Soft-assigning 500 memories against centroids...
multi-domain: 43
single-domain: 412
unclassified (below threshold 0.65): 45
Done in 7.4s.
```
Output format for `discover` without `--force` (post-Phase-0):
```
HDBSCAN: 623 embeddings, min_cluster_size=10
Comparing to existing 3 domains...
Proposals (pending, accept via dashboard or `vestige domains proposals`):
[split] dev -> (systems:34, networking:28) confidence 0.82
[new] cluster_5 (books, novels, reading) confidence 0.71
Run `vestige domains proposals` to review, or open the dashboard.
```
### 4. CLI: `list`, `rename`, `merge`
- `list`: calls `store.list_domains()`, fetches unclassified count via `store.count_memories_without_domains()` (Phase 1 should have provided this; if not, Phase 4 adds it to the trait and both backends).
- `rename`: `store.get_domain(id)` -> mutate `label` -> `store.upsert_domain`. No memory touch.
- `merge`: load both, compute blended centroid (weighted by `memory_count`), merge `top_terms` (union, recompute TF-IDF rank if both sides share the corpus), delete the non-`into` domain, call `reassign_all`. Wrapped in a transaction on Postgres; on SQLite rely on the existing writer-lock pattern.
### 5. Auto-classify on ingest
**File**: `crates/vestige-core/src/cognitive.rs` (or equivalent ingest entry in `vestige-mcp/src/tools/smart_ingest.rs`).
Integration point: just before the record is persisted in the smart-ingest path, after the embedder has produced `embedding` and before `storage.insert(...)`. Trace the current call site -- today `Storage::ingest(IngestInput)` computes embedding inside storage; in Phase 1 the embedder becomes external (ADR decision Q2), so classification can hook right there in the cognitive engine.
Pseudocode:
```rust
let embedding = embedder.embed(&input.content).await?;
let domains = store.list_domains().await?;
let (domains_assigned, domain_scores) = if domains.is_empty() {
(Vec::new(), HashMap::new())
} else {
let boost = context_signals.gather_boost(&input.metadata, &domains);
let result = classifier.classify_with_boost(&embedding, &domains, boost.as_ref());
(result.domains, result.scores)
};
record.embedding = Some(embedding);
record.domains = domains_assigned;
record.domain_scores = domain_scores;
store.insert(&record).await?;
```
Edge cases:
- Accumulation phase (`domains.is_empty()`): skip classification entirely. Zero overhead.
- Embedding failed / skipped: leave `domains = []`, `domain_scores = {}`. Never fail ingest because of classification.
- Metric: emit `VestigeEvent::MemoryClassified { id, domains, top_score }` on the WebSocket bus so the dashboard sees it live.
### 6. Re-cluster hook in dream consolidation
**File**: `crates/vestige-core/src/advanced/dreams.rs` (long file, 1131-line `dream()` entry on the `MemoryDreamer` impl) plus `crates/vestige-core/src/consolidation/phases.rs` (the `DreamEngine::run` orchestrator).
Design: the `DreamEngine::run(...)` returns `FourPhaseDreamResult`. It does not currently know how many times it has run. Phase 4 introduces a persistent counter on disk (column `dream_cycle_count` on a new singleton `system_state` table, or a simple row in the existing `metadata` / `embedding_model` registry). After the Integration phase finishes, the cognitive engine increments the counter and, if `counter % recluster_interval == 0`, launches discovery asynchronously:
Extension struct in `phases.rs`:
```rust
pub struct DreamReClusterHook<'a> {
pub classifier: &'a DomainClassifier,
pub store: &'a dyn MemoryStore,
pub event_tx: Option<&'a tokio::sync::mpsc::UnboundedSender<VestigeEvent>>,
}
impl<'a> DreamReClusterHook<'a> {
pub async fn tick(&self, cycle_count: usize) -> Result<Vec<DomainProposal>, StorageError> {
if cycle_count == 0 || cycle_count % self.classifier.recluster_interval != 0 {
return Ok(vec![]);
}
let existing = self.store.list_domains().await?;
let rediscovered = self.classifier.discover(self.store).await?;
let proposals = self
.classifier
.propose_changes(self.store, &existing, &rediscovered)
.await?;
for p in &proposals {
self.store.insert_domain_proposal(p).await?;
if let Some(tx) = self.event_tx {
let _ = tx.send(VestigeEvent::DomainProposalCreated {
id: p.id.clone(),
kind: format!("{:?}", p.kind),
confidence: p.confidence,
timestamp: Utc::now(),
});
}
}
Ok(proposals)
}
}
```
Caller wires `tick()` after `DreamEngine::run()` returns, at the ingest/consolidation orchestrator level. The hook never mutates existing domains -- it only writes proposals. The acceptance path is manual (CLI or dashboard).
Counter storage: add method `store.bump_dream_cycle_count() -> Result<usize>` returning the new count. Single-row table:
```sql
CREATE TABLE IF NOT EXISTS system_state (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
-- seed: ('dream_cycle_count', '0')
```
### 7. Context signal extractor
**File**: `crates/vestige-core/src/neuroscience/context_signals.rs`
```rust
pub trait SignalSource: Send + Sync {
/// Returns domain_id -> additive boost (positive or negative, typically in [-0.1, +0.1]).
fn boost_map(
&self,
input_metadata: &serde_json::Value,
domains: &[Domain],
) -> HashMap<String, f64>;
fn name(&self) -> &'static str;
}
pub struct GitRepoSignal {
pub boost: f64, // default +0.05
}
pub struct IdeHintSignal {
pub boost: f64,
}
pub struct ContextSignals {
sources: Vec<Box<dyn SignalSource>>,
}
impl ContextSignals {
pub fn gather_boost(
&self,
input_metadata: &serde_json::Value,
domains: &[Domain],
) -> Option<HashMap<String, f64>>;
}
```
Signal encoding convention (document in the module header):
- A signal is a **soft prior**. It nudges the post-cosine score by a small additive delta, clamped to `[-0.10, +0.10]` per signal.
- Multiple signals sum, then the final boost per domain is clamped to `[-0.15, +0.15]` so signals cannot by themselves push a memory into or out of a domain; the embedding similarity dominates.
- Signals target domains by heuristic: `GitRepoSignal` boosts any domain whose `top_terms` overlaps `{"rust","async","trait","function","class","def","git","commit","fn","code"}`. `IdeHintSignal` does the same for `{"file","line","editor","vscode","neovim","rust-analyzer","lsp"}`.
- All signal boosts are logged via `tracing::debug!` so users can audit why a memory picked up a domain.
`GitRepoSignal::boost_map` implementation:
```rust
fn boost_map(&self, meta: &Value, domains: &[Domain]) -> HashMap<String, f64> {
let is_git = meta.get("cwd")
.and_then(|v| v.as_str())
.map(|cwd| std::path::Path::new(cwd).join(".git").exists())
.unwrap_or(false)
|| meta.get("git_repo").is_some();
if !is_git { return HashMap::new(); }
let mut out = HashMap::new();
for d in domains {
let code_hits = d.top_terms.iter()
.filter(|t| CODE_TERMS.contains(t.as_str()))
.count();
if code_hits > 0 { out.insert(d.id.clone(), self.boost); }
}
out
}
```
Config knob in `[domains.signals]`: `git = true`, `ide = true`, `git_boost = 0.05`, `ide_boost = 0.05`.
### 8. Cross-domain spreading activation decay
**File**: `crates/vestige-core/src/neuroscience/spreading_activation.rs`
Modify `ActivationConfig`:
```rust
pub struct ActivationConfig {
pub decay_factor: f64,
pub max_hops: u32,
pub min_threshold: f64,
pub allow_cycles: bool,
pub cross_domain_decay: f64, // NEW, default 0.5
}
```
Domain metadata on nodes: the current `ActivationNode` has `id`, `activation`, `last_activated`, `edges: Vec<String>`. Phase 4 adds `pub domains: Vec<String>`. Populated when nodes get added (propagated from the memory's `domains` field). The network is rebuilt on each search from the store; if the in-memory network is persisted (check `ActivationNetwork` lifetime in `CognitiveEngine`), the population happens in the engine at boot and on insert.
Traversal change, in `ActivationNetwork::activate` loop, replacing the single line `let propagated = current_activation * edge.strength * self.config.decay_factor;`:
```rust
let cross_penalty = {
let src_doms = self.nodes.get(&current_id).map(|n| &n.domains);
let tgt_doms = self.nodes.get(&target_id).map(|n| &n.domains);
match (src_doms, tgt_doms) {
(Some(s), Some(t)) if !s.is_empty() && !t.is_empty() => {
let overlap = s.iter().any(|d| t.contains(d));
if overlap { 1.0 } else { self.config.cross_domain_decay }
}
_ => 1.0, // unclassified on either side: no penalty
}
};
let propagated = current_activation * edge.strength * self.config.decay_factor * cross_penalty;
```
Rationale for "unclassified -> no penalty": unclassified memories are Phase-0 or low-confidence corpus members; penalizing them would block useful cross-pollination during the accumulation ramp.
API to update a node's domains after reclassification:
```rust
pub fn set_node_domains(&mut self, id: &str, domains: Vec<String>);
```
Called by the reassignment pipeline after `reassign_all`.
### 9. `vestige.toml` `[domains]` section
**File**: wherever `vestige.toml` is loaded (search for `[storage]` / `[server]` loaders). Add:
```toml
[domains]
assign_threshold = 0.65
discovery_threshold = 150
recluster_interval = 5
min_cluster_size = 10
min_samples = 5
cross_domain_decay = 0.5
merge_threshold = 0.90
top_terms_k = 10
[domains.signals]
git = true
ide = true
git_boost = 0.05
ide_boost = 0.05
```
Rust-side: `DomainsConfig { ... }` struct with `serde(default)` so `vestige.toml` without a `[domains]` section falls back to hard-coded defaults. `DomainClassifier::from_config(cfg: &DomainsConfig) -> Self`.
### 10. Dashboard UI additions
**SvelteKit routes** (`apps/dashboard/src/routes/(app)/domains/`):
- `+page.svelte` (list): fetches `GET /api/v1/domains` and `GET /api/v1/domains/unclassified-count`. Renders a table: `label`, `memories`, `top_terms` chips, `created_at`. Each row links to `/domains/[id]`. A "Discover" button posts `POST /api/v1/domains/discover`.
- `[id]/+page.svelte` (detail): fetches `GET /api/v1/domains/:id`, `GET /api/v1/domains/:id/memories?limit=100`, `GET /api/v1/domains/:id/score-histogram`. Renders:
- Header: label (editable, triggers `PUT /api/v1/domains/:id`), top-terms chips, memory count, created_at.
- Histogram: a vertical bar chart of `domain_scores[:id]` buckets 0-0.1, 0.1-0.2, ..., 0.9-1.0 across all memories. Data source: server precomputes buckets so the client does not need to fetch all scores.
- Memory list: paginated, each row shows the raw score for this domain.
- `proposals/+page.svelte`: fetches `GET /api/v1/domains/proposals?status=pending`. Each pending proposal card shows `kind`, `rationale`, `confidence`, `created_at`, buttons "Accept" (posts `POST /api/v1/domains/proposals/:id/accept`) and "Reject" (`POST .../reject`). Live updates via the existing WebSocket channel (`/ws`) reacting to `DomainProposalCreated` events.
Styling reuses the existing Tailwind + shadcn-svelte conventions in `apps/dashboard/src/lib/components/`.
Existing `(app)/stats` and `(app)/feed` pages get a small "Domains" summary panel that links to `/domains`.
### 11. REST endpoints
**File**: `crates/vestige-mcp/src/protocol/http.rs` or a new `crates/vestige-mcp/src/api/domains.rs` module, wired into the `/api/v1/` router.
| Method | Path | Handler |
|--------|------|---------|
| GET | `/api/v1/domains` | `list_domains` -- returns `[Domain...]` + unclassified count |
| POST | `/api/v1/domains/discover` | `trigger_discover` -- body `{ min_cluster_size?: usize, force?: bool }`, returns proposals or applied domains |
| GET | `/api/v1/domains/:id` | `get_domain` |
| PUT | `/api/v1/domains/:id` | `update_domain` -- rename |
| DELETE | `/api/v1/domains/:id` | `delete_domain` -- with `?merge_into=other_id` |
| GET | `/api/v1/domains/:id/memories` | paginated memories in this domain |
| GET | `/api/v1/domains/:id/score-histogram` | precomputed buckets |
| GET | `/api/v1/domains/proposals` | `list_proposals?status=pending` |
| POST | `/api/v1/domains/proposals/:id/accept` | `accept_proposal` |
| POST | `/api/v1/domains/proposals/:id/reject` | `reject_proposal` |
All handlers go through the Phase 3 auth middleware (Bearer / X-API-Key / session cookie). Responses are JSON; error paths use `StatusCode::*` with a small `{"error": "..."}` body.
### 12. `domain_proposals` table + trait methods
Postgres migration (`crates/vestige-core/migrations/postgres/00XX_domain_proposals.sql`):
```sql
CREATE TABLE domain_proposals (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
kind TEXT NOT NULL, -- 'split' | 'merge' | 'new_cluster'
payload JSONB NOT NULL, -- serialized ProposalKind body
rationale TEXT NOT NULL,
confidence DOUBLE PRECISION NOT NULL,
status TEXT NOT NULL DEFAULT 'pending', -- pending|accepted|rejected
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
resolved_at TIMESTAMPTZ
);
CREATE INDEX idx_domain_proposals_status ON domain_proposals (status, created_at DESC);
```
SQLite migration: same table, `UUID` -> `TEXT`, `JSONB` -> `TEXT` with JSON-encoded bodies, `TIMESTAMPTZ` -> `TEXT` ISO-8601.
`MemoryStore` trait additions:
```rust
async fn insert_domain_proposal(&self, p: &DomainProposal) -> Result<()>;
async fn list_domain_proposals(&self, status: Option<&str>) -> Result<Vec<DomainProposal>>;
async fn get_domain_proposal(&self, id: &str) -> Result<Option<DomainProposal>>;
async fn set_proposal_status(&self, id: &str, status: &str) -> Result<()>;
```
### 13. WebSocket event for proposals
**File**: `crates/vestige-mcp/src/dashboard/events.rs`
Add variant:
```rust
pub enum VestigeEvent {
// ... existing ...
DomainProposalCreated {
id: String,
kind: String,
confidence: f64,
timestamp: DateTime<Utc>,
},
MemoryClassified {
id: String,
domains: Vec<String>,
top_score: f64,
timestamp: DateTime<Utc>,
},
}
```
The SvelteKit dashboard's WS client reacts to both events: classified events refresh any open domain-detail page; proposal events push a toast and a badge on the navbar.
---
## Test Plan
Test root: `tests/phase_4/` (a new member of the workspace; mirror the `tests/e2e` layout).
`tests/phase_4/Cargo.toml`:
```toml
[package]
name = "vestige-phase4-tests"
version = "0.0.0"
edition = "2024"
publish = false
[dependencies]
vestige-core = { path = "../../crates/vestige-core", features = ["embeddings", "vector-search", "domain-classification"] }
vestige-mcp = { path = "../../crates/vestige-mcp" }
tokio = { workspace = true }
anyhow = "1"
tempfile = "3"
serde_json = { workspace = true }
uuid = { workspace = true }
```
### Unit tests (colocated in `domain_classifier.rs::tests`, `context_signals.rs::tests`, `spreading_activation.rs::tests`)
Each public function must have at least one test:
- `classify_empty_domains_returns_empty`: `classify(&[0.0; 768], &[])` returns `ClassificationResult { scores: {}, domains: [] }`.
- `classify_single_domain_scores`: one `Domain` with a known centroid; input embedding equal to centroid; expect score 1.0 and `domains == [id]`.
- `classify_multi_domain_overlap`: two domains A, B; input halfway between centroids; expect both scores >= `assign_threshold`; expect `domains == [A, B]` (order not guaranteed).
- `classify_below_threshold_returns_empty_domains_but_scores_filled`: input orthogonal to all centroids; expect `scores` populated, `domains` empty.
- `classify_with_boost_adds_delta`: same input as above, with `boost = {A: 0.4}`; expect A now above threshold, B unchanged.
- `classify_boost_clamps_to_unit`: `boost = {A: 5.0}`; resulting `scores[A]` must be <= 1.0.
- `tfidf_top_k_returns_distinct_terms`: given three fake docs, `top_k=3` returns three non-duplicate strings, in descending TF-IDF order.
- `tfidf_top_k_drops_stopwords`: `["the and for"]` + real content -> stop-words absent.
- `compute_top_terms_handles_empty_cluster`: returns `vec![]` (no panic).
- `signal_git_present_vs_absent`: `GitRepoSignal` given metadata with `.git` in cwd returns non-empty map; without it returns empty.
- `signal_ide_present_vs_absent`: `IdeHintSignal` ditto for `metadata.editor == "vscode"`.
- `signal_combined_clamped`: two signals both firing each at +0.10 -> combined map values <= +0.15.
- `cross_domain_decay_full_weight_on_overlap`: graph with node A in domain `dev`, node B in domain `dev`, edge A->B strength 1.0; after `activate`, B's activation equals the standard `initial * strength * decay_factor` (no extra penalty).
- `cross_domain_decay_half_weight_no_overlap`: A in `dev`, B in `infra`, same edge -> B's activation is 0.5x that of the overlap case.
- `cross_domain_decay_unclassified_no_penalty`: A classified, B unclassified -> full weight.
- `propose_changes_detects_split`: existing domain `dev`; new discovery returns two clusters whose centroids both sit close to old `dev` centroid, each >= min_cluster_size members -> proposal of kind `Split { parent: "dev", children: [a, b] }`.
- `propose_changes_detects_merge`: two existing domains whose new centroids now have cosine > `merge_threshold` -> proposal of kind `Merge`.
- `propose_changes_detects_new_cluster`: a new cluster with no match >= 0.85 to any existing -> `NewCluster`.
- `apply_proposal_split_updates_memberships`: after accept, memories previously in `dev` get reassigned (some to child a, some to child b) via `reassign_all`.
### Integration tests (`tests/phase_4/tests/`)
One file per behavior listed in the Phase 4 acceptance sheet.
- `discover_seed_corpus.rs` -- loads the 500-memory fixture, runs `classifier.discover(&store).await`, asserts at least 3 clusters, asserts per-cluster intra-similarity mean > 0.6, asserts discovery wall time < 10s in release. Also asserts `top_terms` for each cluster contains at least one expected keyword per cluster (dev: contains any of `rust/trait/async`; infra: `bgp/vlan/network`; home: `solar/battery/pool`).
- `soft_assign_multi_domain.rs` -- inserts a memory "deploy zinit containers over BGP network"; after classify, `domains` contains both `dev` and `infra` (from a known centroid setup).
- `auto_classify_on_ingest.rs` -- with three existing domains, a fresh `smart_ingest` of a dev-ish sentence ends up with `domains == ["dev"]` and non-empty `domain_scores`.
- `reembed_triggers_recluster.rs` -- after `vestige migrate --reembed`, centroids must be recomputed; verify `list_domains()` returns fresh `centroid` values (different from pre-reembed).
- `dream_consolidation_recluster_hook.rs` -- run 5 dream cycles with heavy synthetic memory insertion; after the 5th, assert `list_domain_proposals("pending")` has at least one proposal.
- `proposal_accept_applies_changes.rs` -- accept a split proposal via `apply_proposal`; verify that memories in `dev` are now distributed across the new children and that the old `dev` domain is removed.
- `proposal_reject_leaves_state.rs` -- reject a proposal; verify all domains and memberships unchanged.
- `drift_is_proposal_only.rs` -- over 5 dream cycles with new inserts, never call accept; verify every memory's `domains` field equals its initial post-discovery value. No auto-apply.
- `cross_domain_activation_decay.rs` -- build a `ActivationNetwork` with two memories linked by a strength-1.0 edge, one in `dev`, one in `infra`; activate `dev` memory with 1.0; assert `infra` memory's activation == `0.5 * decay_factor` (0.35 with default decay_factor 0.7). Then set both to `dev` and reassert activation == `0.7`.
- `cli_domains_discover.rs` -- spawn `cargo run -- domains discover --force --json`, parse stdout, assert at least 3 clusters and valid JSON shape.
- `cli_domains_rename_merge.rs` -- happy-path rename then merge, with stdout assertions.
- `context_signal_git_repo.rs` -- ingest the same sentence from inside a tempdir with `.git` vs outside; assert the git-run produces slightly higher `domain_scores` for the code-related domain (diff >= 0.04, matches `git_boost = 0.05`).
- `threshold_tunable.rs` -- same memory, two runs with `assign_threshold = 0.40` vs `0.85`; the low-threshold run assigns more domains than the high-threshold run for the same content.
- `signal_boost_clamped.rs` -- artificially configure `git_boost = 5.0` and assert the resulting per-domain score is still <= 1.0.
- `discover_preserves_stable_ids.rs` -- run discover twice with no new memories; the second run's domain ids match the first's (via centroid-similarity stable-ID matching above 0.85).
### Dashboard UI tests (`tests/phase_4/ui/`)
Use curl-driven smoke tests (avoids adding Playwright as a new hard dep; Playwright already exists at `apps/dashboard/playwright.config.ts` and can be extended later).
- `domains_list_renders.sh` -- `curl -H "X-API-Key: $KEY" http://localhost:3927/api/v1/domains` returns 200 + JSON array with expected keys.
- `domain_detail_histogram.sh` -- `curl .../api/v1/domains/dev/score-histogram` returns 10 buckets.
- `proposal_review_flow.sh` -- create a pending proposal via SQL insert; `curl POST .../api/v1/domains/proposals/<id>/accept`; `curl GET .../proposals?status=accepted` shows it.
- `unauth_domain_list_rejected.sh` -- no auth header -> 401.
### Benchmarks (`tests/phase_4/benches/`)
Criterion benches:
- `bench_discover_10k.rs` -- synthetic 10k x 768D embeddings drawn from 5 blobs; assert `discover` wall p95 < 30s on a warm release build.
- `bench_auto_classify_single.rs` -- 20 domains in memory, classify one 768D vector; assert p99 < 5ms.
- `bench_reassign_all.rs` -- 10k memories, 5 domains; assert full `reassign_all` wall time < 90s (100 rows/ms baseline).
---
## Acceptance Criteria
- [ ] `cargo build -p vestige-core --features domain-classification` zero warnings.
- [ ] `cargo build -p vestige-mcp` zero warnings.
- [ ] `cargo clippy --workspace --all-targets --all-features -- -D warnings` clean.
- [ ] `cargo test -p vestige-phase4-tests` -- all tests in `tests/phase_4/` pass.
- [ ] On a 500+ memory seed corpus covering three natural clusters (dev / infra / home), `vestige domains discover --force` produces sensible top-terms matching the expected keyword sets and labels are stable on a second run.
- [ ] `vestige search` with domain filter `["dev"]` excludes any memory whose `domains` array does not include `dev`.
- [ ] After 5 dream cycles with ongoing inserts, no existing memory's `domains` has silently changed; proposals exist in `domain_proposals` table; accepting a proposal reassigns as described.
- [ ] Cross-domain spreading activation: a query in `dev` that crosses a single edge into an `infra`-only memory still returns the memory but with activation `cross_domain_decay * in-domain_activation`.
- [ ] `vestige domains discover --min-cluster-size 20` produces strictly fewer or equal clusters than the default, and with larger per-cluster membership.
- [ ] Dashboard `/dashboard/domains` route renders all domains within 2 seconds on the seed corpus.
- [ ] Proposal UI flow (open pending, accept, confirmed in store) works end-to-end.
- [ ] Benchmarks meet targets (discover 10k p95 < 30s, auto-classify p99 < 5ms).
---
## Rollback Notes
- **Feature gate**: add `domain-classification` to `crates/vestige-core/Cargo.toml`'s `[features]`. When disabled, the `DomainClassifier` module is not compiled, the classification call in the ingest path is a no-op (`#[cfg]`-guarded), and cross-domain decay collapses to `1.0`. The CLI `domains` subcommand emits "domain classification is disabled in this build".
- **Revert strategy**: drop the two new tables `domains` (if created in Phase 1 is retained) or `domain_proposals` (Phase 4). A DOWN migration clears `memories.domains` and `memories.domain_scores`. Existing memories simply lose their domain assignments; all search and retrieval paths work unchanged because `domains = []` is the documented "unclassified" state.
- **Idempotency**: rerunning `discover` is always safe. Cluster numeric IDs may differ between runs, but the stable-ID match by centroid similarity preserves user-assigned labels. Do not persist cluster ids in client-side bookmarks; link via the user-assigned label.
- **Data-loss risk**: `apply_proposal` is a destructive operation (it deletes the old parent domain in a split or merges two). The dashboard's accept button double-confirms with a modal that shows the number of affected memories.
---
## Open Implementation Questions
Each question + candidates + RECOMMENDATION.
### OQ1. Top-terms extraction: TF-IDF vs BM25 vs frequency?
- TF-IDF with smoothed IDF -- standard, cheap, good-enough.
- BM25 -- better for long-document discrimination, overkill for short memory contents.
- Raw frequency -- noisy; stop-words dominate.
**RECOMMENDATION**: TF-IDF with global IDF over the entire memory corpus (not just cluster members), recomputed once per `discover` call. Same tokenizer as the `dreams.rs::content_similarity` Jaccard for consistency.
### OQ2. Proposal persistence: DB table vs in-memory with dashboard notification?
- DB table (`domain_proposals`) -- durable, surfaces across restarts, enables audit.
- In-memory only -- simpler, but loses proposals on server restart.
**RECOMMENDATION**: DB table. Proposals are rare (every 5th dream) and valuable user-facing artifacts; durability is mandatory.
### OQ3. `hdbscan` crate: f32 vs f64 input, exact API surface?
- v0.10 historically takes `&[Vec<f64>]`; embeddings are `Vec<f32>`.
- Cost of converting f32 -> f64 at discovery time: `10k * 768 = 7.68M` f64 doubles ~ 60MB transient, acceptable.
**RECOMMENDATION**: verify v0.10's type signature at implementation time; if it requires f64, perform the conversion in `discover()` behind a single allocation. Document in module header. If the crate API diverged from the PRD snippet, fall back to the manual builder style (`HdbscanHyperParams::builder().min_cluster_size(n).min_samples(s).build()`).
### OQ4. Stable domain IDs across discover re-runs?
- Option A: numeric IDs from HDBSCAN labels -- unstable, re-runs shuffle them.
- Option B: hash(top_terms) -- stable if top-terms stable, but top-terms drift.
- Option C (recommended): after computing new centroids, match each to the closest existing domain by centroid cosine; if similarity > 0.85, reuse the existing domain's `id` and `label`. Otherwise mint a fresh `id = "cluster_<uuid>"`.
**RECOMMENDATION**: Option C. Preserves user-assigned labels across drift. Threshold 0.85 is config-tunable via `stable_id_threshold` if needed later.
### OQ5. Context signal injection site: ingest handler vs embedder vs classifier?
- Embedder -- would alter embedding; signals are not about embedding quality.
- Ingest handler -- signals known there, but then `DomainClassifier` cannot be tested in isolation.
- Classifier as a `classify_with_boost(boost: Option<&HashMap>)` parameter -- pure, testable, composable.
**RECOMMENDATION**: classifier parameter. The cognitive engine constructs the boost map via `ContextSignals::gather_boost(&metadata, &domains)` and hands it to the classifier. Keeps the classifier stateless w.r.t. signals.
### OQ6. Re-cluster proposal cadence: event-based (every Nth dream) vs time-based (weekly)?
- ADR resolution Q7: every Nth dream (N=5 default).
- Alternative: once per week regardless of dream cadence.
**RECOMMENDATION**: stick with every Nth dream. Users who dream rarely re-cluster rarely -- that matches the philosophy ("memory work triggers memory bookkeeping"). Note the alternative as future consideration; if users complain about never seeing proposals, add a time-based fallback.
### OQ7. Minimum corpus size for first discover?
- PRD default: 150.
- Too low -> noisy initial clusters, proposals every dream.
- Too high -> user waits forever for domains to appear.
**RECOMMENDATION**: 150 as the default discovery gate; HDBSCAN's `min_cluster_size=10` will produce 0 clusters for < 100 memories, so the system gracefully produces no domains until the corpus is large enough. Test with `N=80, 150, 500` in `threshold_tunable.rs` to confirm sensible behavior.
### OQ8. Cross-domain decay: strict no-overlap vs graded?
- Strict: `1.0` if any overlap, `cross_domain_decay` otherwise.
- Graded: `max(cross_domain_decay, |A intersect B| / max(|A|, |B|))`.
**RECOMMENDATION**: strict for Phase 4. Easier to reason about, easier to tune, easier to test. Graded is a marked future enhancement; file an issue if retrieval-quality metrics justify it.
### OQ9. Classifier invocation from remote HTTP clients?
- In server mode, an agent posts `smart_ingest` -> server embeds -> server classifies.
- All the work stays server-side; MCP clients never do classification.
**RECOMMENDATION**: confirmed server-side-only. Document in the MCP tool schema that `smart_ingest` now returns `domains` and `domain_scores` in its response so clients can display the classification to the user.
### OQ10. Where to store the dream-cycle counter?
- In-memory on `CognitiveEngine` -- lost on restart, miscounts cadence.
- New `system_state` singleton table.
**RECOMMENDATION**: `system_state` table. Survives restarts. Also useful for future metrics (total memories ever, total dreams ever).
### OQ11. Scope of `reassign_all` after a proposal accept vs a normal discover?
- On discover --force (first-time), run `reassign_all` against all memories.
- On proposal accept (split / merge), run `reassign_all` only on affected memories (parent's members for split; both parents' members for merge) to avoid touching unrelated records.
**RECOMMENDATION**: scoped reassignment where possible; fall back to full `reassign_all` only on `discover --force` or when the set of domains has fundamentally changed. Reduces write amplification on large corpora.
### OQ12. Proposal freshness?
- Multiple re-clusters could stack up pending proposals.
**RECOMMENDATION**: before inserting a new proposal, check for existing pending proposals with the same `kind + targets`; if present, bump `created_at` and `confidence` instead of creating a duplicate. Add a `confidence_history` array in the `payload` JSONB for audit.
---
## Implementation Sequencing (suggested order)
1. Land the `DomainClassifier` struct, `classify` / `classify_with_boost`, unit tests. (Day 1)
2. Add `compute_top_terms` + TF-IDF helper, tests. (Day 1)
3. Wire `discover` end-to-end against SQLite; `discover_seed_corpus` integration test. (Day 2)
4. Add `domain_proposals` table migrations + trait methods; both backends. (Day 2)
5. Implement `propose_changes` + `apply_proposal`; proposal unit tests. (Day 3)
6. Context signals module + tests. (Day 3)
7. Hook classifier into ingest path; `auto_classify_on_ingest` integration test. (Day 4)
8. Cross-domain decay in spreading activation; unit + integration tests. (Day 4)
9. Dream re-cluster hook + `system_state` counter; integration tests for drift-only behavior. (Day 5)
10. CLI subcommands. (Day 6)
11. REST endpoints. (Day 6)
12. SvelteKit dashboard routes + WebSocket event wiring. (Day 7-8)
13. Benchmarks + acceptance sweep on the 500-memory seed. (Day 9)
---
## File Map (everything Phase 4 touches or creates)
Creates:
- `crates/vestige-core/src/neuroscience/domain_classifier.rs`
- `crates/vestige-core/src/neuroscience/context_signals.rs`
- `crates/vestige-core/migrations/postgres/00XX_domain_proposals.sql`
- `crates/vestige-core/migrations/sqlite/00XX_domain_proposals.sql` (or inline in `storage/migrations.rs`)
- `crates/vestige-mcp/src/api/domains.rs` (REST handlers)
- `apps/dashboard/src/routes/(app)/domains/+page.svelte`
- `apps/dashboard/src/routes/(app)/domains/[id]/+page.svelte`
- `apps/dashboard/src/routes/(app)/domains/proposals/+page.svelte`
- `apps/dashboard/src/lib/api/domains.ts`
- `tests/phase_4/Cargo.toml`
- `tests/phase_4/tests/*.rs` (per the Integration test list)
- `tests/phase_4/fixtures/seed_500.json`
- `tests/phase_4/support/fixtures.rs`
Modifies:
- `crates/vestige-core/Cargo.toml` -- add `hdbscan = "0.10"` under a new `domain-classification` feature.
- `crates/vestige-core/src/neuroscience/mod.rs` -- register new modules, re-exports.
- `crates/vestige-core/src/neuroscience/spreading_activation.rs` -- `cross_domain_decay` field in `ActivationConfig`, `domains` field on `ActivationNode`, decay math in `activate`.
- `crates/vestige-core/src/consolidation/phases.rs` -- `DreamReClusterHook`.
- `crates/vestige-core/src/advanced/dreams.rs` -- accept a hook callback from the orchestrator (if the orchestration is done at this level).
- `crates/vestige-core/src/storage/trait.rs` -- add proposal + system_state methods.
- `crates/vestige-core/src/storage/sqlite.rs` -- implement proposal + system_state methods + `all_embeddings_with_meta` if not already on the trait.
- `crates/vestige-core/src/storage/postgres.rs` (Phase 2) -- same.
- `crates/vestige-core/src/lib.rs` -- re-exports.
- `crates/vestige-core/src/cognitive.rs` (or equivalent ingest orchestrator) -- auto-classify injection.
- `crates/vestige-mcp/src/bin/cli.rs` -- `Domains` subcommand + dispatch.
- `crates/vestige-mcp/src/dashboard/mod.rs` -- wire new REST routes.
- `crates/vestige-mcp/src/dashboard/events.rs` -- new event variants.
- `crates/vestige-mcp/src/dashboard/handlers.rs` -- if legacy dashboard gets a domains panel (optional).
- `vestige.toml` config loader -- `[domains]` section + struct + defaults.
- Root `Cargo.toml` workspace members -- add `tests/phase_4`.
---
## Risks
- **HDBSCAN determinism**: HDBSCAN is deterministic given input order; sorting embeddings by memory id before feeding the clusterer guarantees reproducibility across runs -- do this in `discover()` and document it.
- **Embedding dimension drift**: Phase 1's `embedding_model` registry blocks writes from mismatched embedders. If `discover()` ever sees two dimensions, it bails with a clear error and points at `vestige migrate --reembed`.
- **Classification latency on ingest**: for users with thousands of domains (unlikely but possible), `classify` is O(n_domains * dim). 20 domains * 768 f32 = 15k flops per classification, trivial. Still, expose a `classify_budget_ms` config knob for paranoia.
- **Re-cluster proposal storms**: if the corpus is borderline-stable, small changes can produce conflicting proposals on consecutive dreams. Mitigation: OQ12 (dedup by target set, bump confidence instead of stacking).
- **Dashboard feature gap**: if the SvelteKit app lands with the domains route but the REST endpoints are not yet deployed, the route 404s. Mitigation: ship the REST endpoints in the same release; a feature flag on the client toggles the nav entry.
---
## Non-Goals Reminder
- No Phase 5 federation concerns in this plan.
- No cross-installation domain sync.
- No automatic accept of proposals, ever.
- No graded cross-domain decay; strict only.
- No ML-based domain label suggestion (top-terms are enough for v1).
- No editing individual memory memberships from the UI in this phase.

279
docs/plans/local-dev-postgres-setup.md
View file

@ -1,279 +0,0 @@
# Local Dev Postgres Setup (container, hybrid approach)
**Status**: Applied on this machine on 2026-05-27 (rootless podman, Postgres 18.4 + pgvector 0.8.2).
**Related**: docs/plans/0002-phase-2-postgres-backend.md, docs/adr/0002-phase-2-execution.md, docs/adr/0001-pluggable-storage-and-network-access.md
Purpose: capture the minimum, repeatable steps to stand up a long-lived
Postgres 18 + pgvector instance on a local Linux dev box for Phase 2
(`PgMemoryStore`) development, `sqlx prepare`, and manual migration
testing. This is a single-operator dev recipe, not a production runbook.
ADR 0002 picked the **hybrid container** approach over a native install:
the `pgvector/pgvector:pg18` image ships pgvector pre-installed, matches
the image testcontainers will use in the Phase 2 test harness, and avoids
the AUR/build-from-source friction of native pgvector packaging on Arch.
---
## Current state on this machine
- Runtime: rootless `podman` 5.8.2 (Arch). `docker` 29.5.1 also installed but unused.
- Image: `docker.io/pgvector/pgvector:pg18` (PostgreSQL 18.4, pgvector 0.8.2).
- Container: `vestige-pg`, `--restart=always`, port `127.0.0.1:5432:5432`.
- Volume: named podman volume `vestige-pgdata`, mounted at
`/var/lib/postgresql/data` inside the container; `PGDATA` points at
`/var/lib/postgresql/data/pgdata` so the volume mount is non-empty at
init time (Postgres refuses to initdb into a non-empty directory).
- Listens on: `127.0.0.1:5432` only (port mapping is bound to loopback).
- Auth: `scram-sha-256` (image default for both local socket and host).
### Database + role
- Database: `vestige`, UTF8, owner `vestige`, `LC_COLLATE=C.UTF-8`, `LC_CTYPE=C.UTF-8`.
- Role: `vestige` with `LOGIN CREATEDB` (no superuser, no replication).
- Schema `public` re-owned to `vestige` with full default privileges on
future tables / sequences / functions.
- Extension: `vector` (pgvector 0.8.2) installed in the `vestige`
database by the superuser at setup time.
Net effect: the `vestige` role can create, alter, drop, and grant freely
inside the `vestige` database -- enough for `sqlx::migrate!`, ad-hoc
schema work, and the full Phase 2 `MemoryStore` surface. It cannot create
extensions; the superuser handled `CREATE EXTENSION vector` already.
### Passwords
Two passwords live in the dev user's home, mode 600:
- `~/.vestige_pg_superpw` -- the `postgres` superuser password inside the
container. Used for one-shot admin tasks (creating roles, installing
extensions, password rotation). Day-to-day app traffic does NOT use it.
- `~/.vestige_pg_pw` -- the `vestige` role password. This is the one the
Phase 2 backend, `sqlx prepare`, and ad-hoc `psql` invocations use.
### Connection
```
postgresql://vestige:<password>@127.0.0.1:5432/vestige
```
Recommended dev shell export (keep this OUT of the repo; use `.env` +
gitignore or a shell rc):
```sh
export DATABASE_URL="postgresql://vestige:$(cat ~/.vestige_pg_pw)@127.0.0.1:5432/vestige"
```
---
## Reproduce from scratch
On a fresh Linux box with `podman` installed and `python3` available:
```sh
# 1. Pull the image
podman pull docker.io/pgvector/pgvector:pg18
# 2. Create a persistent named volume
podman volume create vestige-pgdata
# 3. Generate the superuser password and stash it (mode 600)
SUPER_PW=$(python3 -c 'import secrets,string; a=string.ascii_letters+string.digits; print("".join(secrets.choice(a) for _ in range(32)))')
umask 077
printf '%s' "$SUPER_PW" > ~/.vestige_pg_superpw
chmod 600 ~/.vestige_pg_superpw
# 4. Start the container
podman run -d \
--name vestige-pg \
--restart=always \
-p 127.0.0.1:5432:5432 \
-e POSTGRES_PASSWORD="$SUPER_PW" \
-e PGDATA=/var/lib/postgresql/data/pgdata \
-v vestige-pgdata:/var/lib/postgresql/data \
docker.io/pgvector/pgvector:pg18
unset SUPER_PW
# 5. Wait for ready
until podman exec vestige-pg pg_isready -U postgres -h 127.0.0.1 >/dev/null 2>&1; do
sleep 1
done
# 6. Generate the vestige role password and stash it (mode 600)
VESTIGE_PW=$(python3 -c 'import secrets,string; a=string.ascii_letters+string.digits; print("".join(secrets.choice(a) for _ in range(32)))')
umask 077
printf '%s' "$VESTIGE_PW" > ~/.vestige_pg_pw
chmod 600 ~/.vestige_pg_pw
# 7. Create role + database + grants + extension (runs as superuser inside the container)
podman exec -i vestige-pg psql -U postgres -v ON_ERROR_STOP=1 <<SQL
CREATE ROLE vestige WITH LOGIN CREATEDB PASSWORD '${VESTIGE_PW}';
CREATE DATABASE vestige OWNER vestige ENCODING 'UTF8'
TEMPLATE template0 LC_COLLATE 'C.UTF-8' LC_CTYPE 'C.UTF-8';
GRANT ALL PRIVILEGES ON DATABASE vestige TO vestige;
SQL
podman exec -i vestige-pg psql -U postgres -d vestige -v ON_ERROR_STOP=1 <<'SQL'
GRANT ALL ON SCHEMA public TO vestige;
ALTER SCHEMA public OWNER TO vestige;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO vestige;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO vestige;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON FUNCTIONS TO vestige;
CREATE EXTENSION IF NOT EXISTS vector;
SQL
unset VESTIGE_PW
# 8. Smoke test as the vestige role
PGPASSWORD="$(cat ~/.vestige_pg_pw)" psql -h 127.0.0.1 -U vestige -d vestige \
-c "SELECT current_user, current_database(), version();" \
-c "SELECT extname, extversion FROM pg_extension WHERE extname = 'vector';" \
-c "SELECT '[1,2,3]'::vector <-> '[3,2,1]'::vector AS l2_distance;"
```
---
## Boot persistence (rootless podman)
`--restart=always` keeps the container alive across podman daemon
restarts, but rootless podman containers do NOT auto-start on system
boot unless the dev user has lingering enabled:
```sh
sudo loginctl enable-linger "$USER"
```
After that, the `podman-restart.service` user unit handles restart of
`--restart=always` containers when the user session starts at boot:
```sh
systemctl --user enable --now podman-restart.service
```
Skip both if you prefer to start the cluster manually each session with
`podman start vestige-pg`.
---
## Day-to-day operation
```sh
# Status
podman ps --filter name=vestige-pg
# Logs (follow)
podman logs -f vestige-pg
# psql as the app role
PGPASSWORD="$(cat ~/.vestige_pg_pw)" psql -h 127.0.0.1 -U vestige -d vestige
# psql as the superuser (for grants, extensions, role admin)
podman exec -it vestige-pg psql -U postgres
# Stop / start
podman stop vestige-pg
podman start vestige-pg
# Restart in place
podman restart vestige-pg
```
---
## Password rotation
```sh
# Rotate the vestige role password
NEW_PW=$(python3 -c 'import secrets,string; a=string.ascii_letters+string.digits; print("".join(secrets.choice(a) for _ in range(32)))')
umask 077
printf '%s' "$NEW_PW" > ~/.vestige_pg_pw
chmod 600 ~/.vestige_pg_pw
podman exec -i vestige-pg psql -U postgres -v ON_ERROR_STOP=1 \
-c "ALTER ROLE vestige WITH PASSWORD '${NEW_PW}';"
unset NEW_PW
# Rotate the superuser password (less common)
NEW_SUPER=$(python3 -c 'import secrets,string; a=string.ascii_letters+string.digits; print("".join(secrets.choice(a) for _ in range(32)))')
umask 077
printf '%s' "$NEW_SUPER" > ~/.vestige_pg_superpw
chmod 600 ~/.vestige_pg_superpw
podman exec -i vestige-pg psql -U postgres -v ON_ERROR_STOP=1 \
-c "ALTER ROLE postgres WITH PASSWORD '${NEW_SUPER}';"
unset NEW_SUPER
```
Then re-export `DATABASE_URL` in any live shells.
---
## Backup and restore (dev-grade)
`pg_dump` writes a plain-text SQL dump to host disk. For dev data this is
enough; production runbook lives in `0002i-runbook.md`.
```sh
# Dump
PGPASSWORD="$(cat ~/.vestige_pg_pw)" pg_dump -h 127.0.0.1 -U vestige -d vestige \
--format=plain --no-owner > vestige-$(date +%Y%m%d-%H%M%S).sql
# Restore (drops + recreates)
podman exec -i vestige-pg psql -U postgres -v ON_ERROR_STOP=1 \
-c 'DROP DATABASE IF EXISTS vestige;' \
-c 'CREATE DATABASE vestige OWNER vestige ENCODING UTF8 TEMPLATE template0;'
PGPASSWORD="$(cat ~/.vestige_pg_pw)" psql -h 127.0.0.1 -U vestige -d vestige < vestige-DUMP.sql
```
The named volume `vestige-pgdata` persists outside the container; the
container can be `podman rm`'d and recreated without losing data, as
long as the volume stays in place.
---
## Teardown
Destroys the cluster and all data in it:
```sh
podman stop vestige-pg
podman rm vestige-pg
podman volume rm vestige-pgdata
podman rmi docker.io/pgvector/pgvector:pg18
rm -f ~/.vestige_pg_pw ~/.vestige_pg_superpw
```
`enable-linger` and the user systemd unit can be undone with
`sudo loginctl disable-linger "$USER"` and
`systemctl --user disable podman-restart.service` if you turned them on.
---
## Notes for Phase 2
- `pgvector` is preinstalled in the image; the `CREATE EXTENSION vector`
in step 7 above makes it available inside the `vestige` DB. The
extension must be loaded BEFORE `sqlx::migrate!` runs the Phase 2
migration that declares typed `Vector` columns, otherwise the
migration fails.
- Testcontainer-based Phase 2 integration tests use the same
`pgvector/pgvector:pg18` image and spin up fresh containers per run;
they are independent of this long-lived cluster. This cluster exists
for `sqlx prepare`, `cargo run -- migrate --to postgres`, and manual
poking.
- `sqlx prepare` needs `DATABASE_URL` pointed at this cluster with
`vestige` migrations already applied. Run from `crates/vestige-core/`.
---
## Out of scope for this doc
- TLS, client-cert auth, non-localhost access. Phase 3 exposes the
Vestige HTTP API over the network, not Postgres directly.
- PITR, WAL archiving, replication, PgBouncer, tuned `postgresql.conf`.
Defaults are fine for Phase 2 development.
- Native (non-container) Postgres install. The prior version of this
doc covered native Arch packaging; superseded by ADR 0002's hybrid
decision.
- Making this the canonical Vestige backend. By default Vestige still
uses SQLite; this cluster exists so the `postgres-backend` feature
can be built and tested locally.

751
docs/prd/001-getting-centralized-vestige.md
View file

@ -1,751 +0,0 @@
# RFC: Pluggable Storage Backend + Network Access for Vestige
**Status**: Draft / Discussion
**Author**: Jan
**Date**: 2026-02-26
**Vestige version**: v2.x (current main)
## Summary
Add a pluggable storage backend trait to Vestige, enabling PostgreSQL (+pgvector) as an alternative to the current SQLite+FTS5+USearch stack. Simultaneously add HTTP MCP transport with API key authentication to enable centralized/remote deployment.
This keeps the existing local-first SQLite mode fully intact while opening up a server deployment model.
## Motivation
Vestige currently runs as a local process per machine (MCP via stdio, SQLite in `~/.vestige/`). This works great for single-machine use but doesn't support:
- **Multi-machine access**: Same memory brain from laptop, desktop, and server
- **Multi-agent access**: Multiple AI clients hitting one memory store concurrently
- **Future federation**: Syncing memory between decentralized nodes (e.g., MOS/Threefold grid)
SQLite's single-writer model and lack of native network protocol make it unsuitable as a centralized server. PostgreSQL is a natural fit: built-in concurrency (MVCC), authentication, replication, and with `pgvector` + built-in FTS it collapses three separate storage layers into one.
## Design
### Storage Trait
The core abstraction. All 29 cognitive modules interact with storage exclusively through this trait (or a small family of traits).
```rust
use std::collections::HashMap;
use uuid::Uuid;
/// Core memory record, backend-agnostic
#[derive(Debug, Clone)]
pub struct MemoryRecord {
pub id: Uuid,
pub domains: Vec<String>, // [] = unclassified, ["dev"], ["dev", "infra"], etc.
pub domain_scores: HashMap<String, f64>, // raw similarities: {"dev": 0.82, "infra": 0.71}
pub content: String,
pub node_type: String,
pub tags: Vec<String>,
pub embedding: Option<Vec<f32>>, // dimensionality is runtime config
pub created_at: chrono::DateTime<chrono::Utc>,
pub updated_at: chrono::DateTime<chrono::Utc>,
pub metadata: serde_json::Value,
}
/// FSRS scheduling state, stored alongside each memory
#[derive(Debug, Clone)]
pub struct SchedulingState {
pub memory_id: Uuid,
pub stability: f64,
pub difficulty: f64,
pub retrievability: f64,
pub last_review: Option<chrono::DateTime<chrono::Utc>>,
pub next_review: Option<chrono::DateTime<chrono::Utc>>,
pub reps: u32,
pub lapses: u32,
}
/// Hybrid search request
#[derive(Debug, Clone)]
pub struct SearchQuery {
pub domains: Option<Vec<String>>, // None = search all domains
pub text: Option<String>, // FTS query
pub embedding: Option<Vec<f32>>, // vector similarity
pub tags: Option<Vec<String>>, // tag filter
pub node_types: Option<Vec<String>>,
pub limit: usize,
pub min_retrievability: Option<f64>, // filter by FSRS state
}
#[derive(Debug, Clone)]
pub struct SearchResult {
pub record: MemoryRecord,
pub score: f64, // combined/fused score
pub fts_score: Option<f64>,
pub vector_score: Option<f64>,
}
/// Connection/edge between memories (for spreading activation)
#[derive(Debug, Clone)]
pub struct MemoryEdge {
pub source_id: Uuid,
pub target_id: Uuid,
pub edge_type: String,
pub weight: f64,
pub created_at: chrono::DateTime<chrono::Utc>,
}
/// Main storage trait — one impl per backend
/// trait_variant generates a Send-bound `MemoryStore` alias,
/// enabling Arc<dyn MemoryStore> without manual boxing.
#[trait_variant::make(MemoryStore: Send)]
pub trait LocalMemoryStore: Sync + 'static {
// --- Lifecycle ---
async fn init(&self) -> Result<()>;
async fn health_check(&self) -> Result<HealthStatus>;
// --- CRUD ---
async fn insert(&self, record: &MemoryRecord) -> Result<Uuid>;
async fn get(&self, id: Uuid) -> Result<Option<MemoryRecord>>;
async fn update(&self, record: &MemoryRecord) -> Result<()>;
async fn delete(&self, id: Uuid) -> Result<()>;
// --- Search ---
async fn search(&self, query: &SearchQuery) -> Result<Vec<SearchResult>>;
async fn fts_search(&self, text: &str, limit: usize) -> Result<Vec<SearchResult>>;
async fn vector_search(&self, embedding: &[f32], limit: usize) -> Result<Vec<SearchResult>>;
// --- FSRS Scheduling ---
async fn get_scheduling(&self, memory_id: Uuid) -> Result<Option<SchedulingState>>;
async fn update_scheduling(&self, state: &SchedulingState) -> Result<()>;
async fn get_due_memories(&self, before: chrono::DateTime<chrono::Utc>, limit: usize) -> Result<Vec<(MemoryRecord, SchedulingState)>>;
// --- Graph (spreading activation) ---
async fn add_edge(&self, edge: &MemoryEdge) -> Result<()>;
async fn get_edges(&self, node_id: Uuid, edge_type: Option<&str>) -> Result<Vec<MemoryEdge>>;
async fn remove_edge(&self, source: Uuid, target: Uuid) -> Result<()>;
async fn get_neighbors(&self, node_id: Uuid, depth: usize) -> Result<Vec<(MemoryRecord, f64)>>;
// --- Bulk / Maintenance ---
async fn count(&self) -> Result<usize>;
async fn get_stats(&self) -> Result<StoreStats>;
async fn vacuum(&self) -> Result<()>;
}
```
**Design notes:**
- `trait_variant::make` generates a `MemoryStore` trait alias with `Send`-bound futures, allowing `Arc<dyn MemoryStore>` for runtime backend selection. `LocalMemoryStore` is the base (usable in single-threaded contexts), `MemoryStore` is the Send variant for Axum/tokio.
- `embedding: Option<Vec<f32>>` — dimensions determined at runtime by the configured fastembed model. The backend stores whatever it gets.
- The trait is intentionally flat. The cognitive modules (FSRS-6, spreading activation, synaptic tagging, prediction error gating, etc.) sit *above* this trait and don't need to know about the backend.
- `search()` does hybrid RRF fusion at the backend level — both SQLite and Postgres implementations handle this internally.
### Backend: SQLite (existing, refactored)
Wraps the current implementation behind the trait:
```
SqliteMemoryStore
├── rusqlite connection pool (r2d2 or deadpool)
├── FTS5 virtual table (keyword search)
├── USearch HNSW index (vector search, behind RwLock)
└── WAL mode + busy timeout for concurrent readers
```
No behavioral changes — just the trait boundary.
### Backend: PostgreSQL (new)
```
PgMemoryStore
├── sqlx::PgPool (connection pool, compile-time checked queries)
├── tsvector + GIN index (keyword search)
├── pgvector + HNSW index (vector search)
└── Standard PostgreSQL MVCC concurrency
```
**Schema sketch:**
```sql
CREATE EXTENSION IF NOT EXISTS vector;
-- Domain registry — populated by clustering, not by user
CREATE TABLE domains (
id TEXT PRIMARY KEY, -- auto-generated or user-named
label TEXT NOT NULL, -- human label (suggested or user-provided)
centroid vector, -- mean embedding of domain members
top_terms TEXT[] NOT NULL DEFAULT '{}', -- top keywords for display
memory_count INTEGER NOT NULL DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
metadata JSONB NOT NULL DEFAULT '{}'
);
CREATE TABLE memories (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
domains TEXT[] NOT NULL DEFAULT '{}', -- [] = unclassified
domain_scores JSONB NOT NULL DEFAULT '{}', -- {"dev": 0.82, "infra": 0.71} raw similarities
content TEXT NOT NULL,
node_type TEXT NOT NULL DEFAULT 'general',
tags TEXT[] NOT NULL DEFAULT '{}',
embedding vector, -- dimension set at table creation or unconstrained
metadata JSONB NOT NULL DEFAULT '{}',
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
-- FTS: auto-maintained tsvector column
search_vec TSVECTOR GENERATED ALWAYS AS (
setweight(to_tsvector('english', content), 'A') ||
setweight(to_tsvector('english', coalesce(node_type, '')), 'B') ||
setweight(array_to_tsvector(tags), 'C')
) STORED
);
-- FTS index
CREATE INDEX idx_memories_fts ON memories USING GIN (search_vec);
-- Vector similarity (HNSW)
CREATE INDEX idx_memories_embedding ON memories
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);
-- Common filters
CREATE INDEX idx_memories_domains ON memories USING GIN (domains);
CREATE INDEX idx_memories_node_type ON memories (node_type);
CREATE INDEX idx_memories_tags ON memories USING GIN (tags);
CREATE INDEX idx_memories_created ON memories (created_at);
-- FSRS scheduling state
CREATE TABLE scheduling (
memory_id UUID PRIMARY KEY REFERENCES memories(id) ON DELETE CASCADE,
stability DOUBLE PRECISION NOT NULL DEFAULT 0.0,
difficulty DOUBLE PRECISION NOT NULL DEFAULT 0.0,
retrievability DOUBLE PRECISION NOT NULL DEFAULT 1.0,
last_review TIMESTAMPTZ,
next_review TIMESTAMPTZ,
reps INTEGER NOT NULL DEFAULT 0,
lapses INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX idx_scheduling_next ON scheduling (next_review);
-- Graph edges (spreading activation)
-- Edges can cross domain boundaries — spreading activation respects
-- domain filters when provided, traverses freely when searching all domains.
CREATE TABLE edges (
source_id UUID NOT NULL REFERENCES memories(id) ON DELETE CASCADE,
target_id UUID NOT NULL REFERENCES memories(id) ON DELETE CASCADE,
edge_type TEXT NOT NULL DEFAULT 'related',
weight DOUBLE PRECISION NOT NULL DEFAULT 1.0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (source_id, target_id, edge_type)
);
CREATE INDEX idx_edges_target ON edges (target_id);
-- API keys
CREATE TABLE api_keys (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
key_hash TEXT NOT NULL UNIQUE, -- blake3
label TEXT NOT NULL,
scopes TEXT[] NOT NULL DEFAULT '{read,write}',
domain_filter TEXT[] NOT NULL DEFAULT '{}', -- {} = access all domains
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
last_used TIMESTAMPTZ,
active BOOLEAN NOT NULL DEFAULT true
);
```
**Hybrid search in SQL:**
```sql
-- RRF (Reciprocal Rank Fusion) combining FTS + vector
-- $1 = query text, $2 = embedding, $3 = limit, $4 = domain filter (NULL for all)
WITH fts AS (
SELECT id, ts_rank_cd(search_vec, websearch_to_tsquery('english', $1)) AS score,
ROW_NUMBER() OVER (ORDER BY ts_rank_cd(search_vec, websearch_to_tsquery('english', $1)) DESC) AS rank
FROM memories
WHERE search_vec @@ websearch_to_tsquery('english', $1)
AND ($4::text[] IS NULL OR domains && $4) -- array overlap: any match
LIMIT 50
),
vec AS (
SELECT id, 1 - (embedding <=> $2::vector) AS score,
ROW_NUMBER() OVER (ORDER BY embedding <=> $2::vector) AS rank
FROM memories
WHERE embedding IS NOT NULL
AND ($4::text[] IS NULL OR domains && $4)
LIMIT 50
)
SELECT COALESCE(f.id, v.id) AS id,
COALESCE(1.0 / (60 + f.rank), 0) + COALESCE(1.0 / (60 + v.rank), 0) AS rrf_score,
f.score AS fts_score,
v.score AS vector_score
FROM fts f FULL OUTER JOIN vec v ON f.id = v.id
ORDER BY rrf_score DESC
LIMIT $3;
```
### Embedding Configuration
The embedding layer stays external to the storage backend. fastembed runs locally and produces vectors that get passed into `MemoryRecord.embedding`.
```toml
# vestige.toml
[embeddings]
provider = "fastembed" # only local for now
model = "BAAI/bge-base-en-v1.5" # 768 dimensions
# model = "BAAI/bge-large-en-v1.5" # 1024 dimensions
# model = "BAAI/bge-small-en-v1.5" # 384 dimensions
[storage]
backend = "postgres" # or "sqlite"
[storage.sqlite]
path = "~/.vestige/vestige.db"
[storage.postgres]
url = "postgresql://vestige:secret@localhost:5432/vestige"
max_connections = 10
```
On init, the backend reads the embedding dimension from the first stored vector (or from config) and validates consistency.
For pgvector: you can either create the column as `vector(768)` (fixed, faster) or unconstrained `vector` (flexible, slightly slower). Recommendation: fixed dimension derived from config, with a migration path if the model changes.
### Emergent Domain Model
Instead of user-defined tenants, domains emerge automatically from the data via clustering. The user never has to decide where a memory belongs — the system figures it out.
#### Pipeline
```
Phase 1: Accumulate (cold start, 0 → N memories)
│ All memories stored with domains = [] (unclassified)
│ No classification overhead, just embed and store
│ Threshold N is configurable, default ~150 memories
│
Phase 2: Discover (triggered once at threshold, or manually)
│ Run HDBSCAN on all embeddings:
│ - min_cluster_size: ~10
│ - min_samples: ~5
│ - No eps parameter needed (unlike DBSCAN)
│ - Automatically determines number of clusters
│ - Handles variable-density clusters
│ - Border points between clusters flagged naturally
│
│ For each cluster, extract:
│ - Centroid (mean embedding)
│ - Top terms (TF-IDF or frequency over cluster members)
│ - Suggested label from top terms
│
│ Present to user (via dashboard or CLI):
│ "I found 3 natural groupings in your memories:
│ ● cluster_0 (47 memories): BGP, SONiC, VLAN, FRR, peering...
│ ● cluster_1 (31 memories): solar, kWh, battery, pool, ESPHome...
│ ● cluster_2 (22 memories): Rust, trait, async, zinit, tokio..."
│
│ User can:
│ - Name them: cluster_0 → "infra", cluster_1 → "home", cluster_2 → "dev"
│ - Accept suggested names
│ - Merge clusters
│ - Do nothing (auto-names stick)
│
Phase 3: Soft-assign all existing memories
│ Now that centroids exist, re-score every memory (including
│ those from discovery) against all centroids.
│ This replaces HDBSCAN's hard labels with continuous scores:
│
│ For each memory:
│ similarities = [(domain, cosine_sim(embedding, centroid)) for each domain]
│ domains = [id for (id, score) in similarities if score >= threshold]
│
│ Memories in overlap zones get multiple domains.
│ Memories far from all centroids stay unclassified.
│
Phase 4: Classify (ongoing, after discovery)
│ New memory ingested:
│ 1. Compute embedding
│ 2. Compute similarity to ALL domain centroids
│ 3. Store raw scores in domain_scores JSONB
│ 4. Threshold into domains[] array
│ 5. Update domain centroids incrementally (running mean)
│
│ Context signals as soft priors:
│ - Git repo / IDE metadata → boost similarity to code-related domains
│ - No workspace context → slight boost toward non-technical domains
│ - These shift the score, never override the embedding distance
│
Phase 5: Re-cluster (periodic, during dream consolidation)
Re-run HDBSCAN on all embeddings including new ones
Detect:
- New clusters forming from previously unclassified memories
- Existing clusters splitting (domain grew too broad)
- Clusters merging (domains that were artificially separate)
Propose changes to user:
"Your 'dev' domain may have split into two groups:
- systems (zinit, MOS, containers, VMs) — 34 memories
- networking (BGP, SONiC, VLANs, MLAG) — 28 memories
Split them? [yes / no / later]"
Re-run soft assignment on all memories after structural changes
Centroid vectors are updated regardless
```
#### Domain Storage
```rust
#[derive(Debug, Clone)]
pub struct Domain {
pub id: String,
pub label: String,
pub centroid: Vec<f32>,
pub top_terms: Vec<String>,
pub memory_count: usize,
pub created_at: chrono::DateTime<chrono::Utc>,
}
```
Added to the `MemoryStore` trait:
```rust
// --- Domains ---
async fn list_domains(&self) -> Result<Vec<Domain>>;
async fn get_domain(&self, id: &str) -> Result<Option<Domain>>;
async fn upsert_domain(&self, domain: &Domain) -> Result<()>;
async fn delete_domain(&self, id: &str) -> Result<()>;
async fn classify(&self, embedding: &[f32]) -> Result<Vec<(String, f64)>>;
// Returns [(domain_id, similarity)] sorted by similarity desc.
// Caller decides threshold for assignment.
```
#### Classification Module
A new cognitive module alongside FSRS, spreading activation, etc.:
```rust
pub struct DomainClassifier {
/// Similarity threshold — domains scoring above this are assigned
pub assign_threshold: f64, // default: 0.65
/// Minimum memories before running initial discovery
pub discovery_threshold: usize, // default: 150
/// How often to re-cluster (in dream consolidation passes)
pub recluster_interval: usize, // default: every 5th consolidation
/// HDBSCAN min_cluster_size
pub min_cluster_size: usize, // default: 10
}
/// Raw classification result — all scores, before thresholding
#[derive(Debug, Clone)]
pub struct ClassificationResult {
/// Similarity to every known domain centroid
pub scores: HashMap<String, f64>, // {"dev": 0.82, "infra": 0.71, "home": 0.34}
/// Domains above assign_threshold
pub domains: Vec<String>, // ["dev", "infra"]
}
impl DomainClassifier {
/// Score a memory against all domain centroids.
/// Returns raw scores AND thresholded domain list.
pub fn classify(
&self,
embedding: &[f32],
domains: &[Domain],
) -> ClassificationResult {
if domains.is_empty() {
return ClassificationResult {
scores: HashMap::new(),
domains: vec![], // still in accumulation phase
};
}
let scores: HashMap<String, f64> = domains.iter()
.map(|d| (d.id.clone(), cosine_similarity(embedding, &d.centroid)))
.collect();
let assigned: Vec<String> = scores.iter()
.filter(|(_, &s)| s >= self.assign_threshold)
.map(|(id, _)| id.clone())
.collect();
ClassificationResult { scores, domains: assigned }
}
/// Soft-assign all existing memories after discovery or re-clustering.
/// Returns number of memories whose domains changed.
pub async fn reassign_all(
&self,
store: &dyn MemoryStore,
domains: &[Domain],
) -> Result<usize> {
// Load all memories, re-score, update domains + domain_scores
// Batched to avoid loading everything into memory at once
todo!()
}
}
```
**Key distinction from the previous design:** there's no "closest wins" or "margin" logic. Every domain gets a score, and *all* domains above threshold are assigned. A memory about "deploying zinit containers via BGP-routed network" might score 0.78 on "dev" and 0.72 on "infra" — it gets both. A memory about "solar panel output today" scores 0.85 on "home" and 0.31 on everything else — it only gets "home".
The raw `domain_scores` are always stored, so you (or the dashboard) can see *why* a memory was classified the way it was, and the threshold can be adjusted retroactively without re-computing embeddings.
#### Search Behavior
- **Default (no domain filter)**: searches all memories across all domains
- **Domain-scoped**: `domains: Some(vec!["dev"])` — only memories tagged with `dev`
- **Multi-domain**: `domains: Some(vec!["dev", "infra"])` — memories in either
- **MCP clients can set `X-Vestige-Domain` header** for default scoping, but the system works fine without it
#### HDBSCAN Implementation
HDBSCAN (Hierarchical DBSCAN) over the embedding vectors. Advantages over plain DBSCAN:
- **No `eps` parameter** — the hardest thing to tune in DBSCAN. HDBSCAN determines density thresholds from the data hierarchy.
- **Variable-density clusters** — a tight cluster of networking memories and a spread-out cluster of personal memories are both detected correctly.
- **Border points** — memories between clusters are identified as low-confidence members, which aligns perfectly with soft assignment.
Implementation: the `hdbscan` crate in Rust. Load all embeddings into memory (at 768d × f32 × 10k memories ≈ 30MB — fine), cluster, compute centroids, soft-assign all memories against the centroids.
```rust
use hdbscan::{Center, Hdbscan};
fn discover_domains(
embeddings: &[Vec<f32>],
min_cluster_size: usize,
) -> (Vec<Vec<usize>>, Vec<Vec<f32>>) { // (cluster → member indices, centroids)
let clusterer = Hdbscan::default(embeddings);
let labels = clusterer.cluster().unwrap();
let centroids = clusterer.calc_centers(Center::Centroid, &labels).unwrap();
// Group indices by label, ignoring noise (-1)
let mut clusters: HashMap<i32, Vec<usize>> = HashMap::new();
for (i, &label) in labels.iter().enumerate() {
if label >= 0 {
clusters.entry(label).or_default().push(i);
}
}
(clusters.into_values().collect(), centroids)
}
```
After HDBSCAN produces hard clusters, the soft-assignment pass (Phase 3) immediately re-scores all memories — including the ones HDBSCAN assigned — against the computed centroids. So HDBSCAN's hard labels are only used to *define* the centroids. The actual domain assignments always come from the continuous similarity scores.
This works identically for both SQLite and Postgres backends — clustering runs in Rust application code, results are written back to the storage layer.
### Network Transport
#### MCP over Streamable HTTP
Extend the existing Axum server:
```rust
// Alongside existing dashboard routes
let app = Router::new()
// Existing dashboard
.route("/api/health", get(health_handler))
.route("/dashboard/*path", get(dashboard_handler))
// New: MCP over HTTP
.route("/mcp", post(mcp_handler).get(mcp_sse_handler))
// New: REST API
// X-Vestige-Domain header optionally scopes to a domain
.route("/api/v1/memories", post(create_memory).get(list_memories))
.route("/api/v1/memories/:id", get(get_memory).put(update_memory).delete(delete_memory))
.route("/api/v1/search", post(search_memories))
.route("/api/v1/consolidate", post(trigger_consolidation))
.route("/api/v1/stats", get(get_stats))
.route("/api/v1/domains", get(list_domains))
.route("/api/v1/domains/discover", post(trigger_discovery))
.route("/api/v1/domains/:id", put(rename_domain).delete(merge_domain))
// Auth on everything except health
.layer(middleware::from_fn(api_key_auth));
```
#### Auth Middleware
```rust
async fn api_key_auth(
State(store): State<Arc<dyn MemoryStore>>,
request: axum::extract::Request,
next: middleware::Next,
) -> Result<Response, StatusCode> {
// Skip auth for health endpoint
if request.uri().path() == "/api/health" {
return Ok(next.run(request).await);
}
let key = request.headers()
.get("Authorization")
.and_then(|v| v.to_str().ok())
.and_then(|v| v.strip_prefix("Bearer "))
.or_else(|| request.headers()
.get("X-API-Key")
.and_then(|v| v.to_str().ok()));
match key {
Some(k) if verify_api_key(store.as_ref(), k).await => {
Ok(next.run(request).await)
}
_ => Err(StatusCode::UNAUTHORIZED),
}
}
```
#### Client Configuration
```json
// Claude Desktop / Claude Code — single key, all domains
{
"mcpServers": {
"vestige": {
"url": "http://vestige.local:3927/mcp",
"headers": {
"Authorization": "Bearer vst_a1b2c3..."
}
}
}
}
```
No domain header needed — searches all domains by default. The MCP tools include an optional `domain` parameter for scoped queries if the LLM or user wants to narrow down.
Alternatively, scope a connection to a specific domain:
```json
// Domain-scoped connection (e.g., for a home automation agent)
{
"mcpServers": {
"vestige-home": {
"url": "http://vestige.local:3927/mcp",
"headers": {
"Authorization": "Bearer vst_e5f6g7...",
"X-Vestige-Domain": "home"
}
}
}
}
```
### Server Configuration
```toml
# vestige.toml — full example for server mode
[server]
bind = "0.0.0.0:3927" # or mycelium IPv6 address
# tls_cert = "/path/to/cert.pem" # optional
# tls_key = "/path/to/key.pem"
[auth]
enabled = true
# If false, no key required (local-only mode)
[storage]
backend = "postgres"
[storage.postgres]
url = "postgresql://vestige:secret@localhost:5432/vestige"
max_connections = 10
[embeddings]
provider = "fastembed"
model = "BAAI/bge-base-en-v1.5"
```
### CLI Extensions
```bash
# Domain management (mostly automatic, but user can inspect/rename)
vestige domains list
# → dev Development (auto) memories: 87 top: Rust, trait, async, tokio
# → infra Infrastructure (auto) memories: 47 top: BGP, SONiC, VLAN, FRR
# → home Home (auto) memories: 31 top: solar, kWh, pool, ESPHome
# → (unclassified) memories: 12
vestige domains rename cluster_0 infra --label "Infrastructure"
vestige domains merge home personal --into home
vestige domains discover --force # re-run HDBSCAN now
# Key management
vestige keys create --label "macbook"
# → Created key: vst_a1b2c3d4... (store this, shown once)
vestige keys create --label "home-assistant" --scopes read --domains home
# → Created key: vst_e5f6g7h8... (read-only, home domain only)
vestige keys list
# → macbook vst_a1b2... scopes: [read,write] domains: [all]
# → home-assistant vst_e5f6... scopes: [read] domains: [home]
vestige keys revoke vst_a1b2c3d4...
# Migration
vestige migrate --from sqlite --to postgres \
--sqlite-path ~/.vestige/vestige.db \
--postgres-url postgresql://localhost/vestige
```
## Implementation Plan
### Phase 1: Storage Trait Extraction
- Define the `MemoryStore` trait (including domain methods)
- Refactor current SQLite code to implement it
- Add `domains TEXT[]` column to existing SQLite schema
- Verify all 29 modules work through the trait (no direct SQLite access)
- **No behavioral changes** — all memories start as unclassified
### Phase 2: PostgreSQL Backend
- Implement `PgMemoryStore`
- Schema migrations (sqlx or refinery)
- `vestige migrate` command for SQLite → Postgres
- Config file support for backend selection
### Phase 3: Network Access
- MCP Streamable HTTP endpoint on existing Axum server
- API key auth middleware + CLI management
- REST API endpoints
- Feature flags for stdio vs HTTP mode
### Phase 4: Emergent Domain Classification
- `DomainClassifier` cognitive module
- HDBSCAN clustering via `hdbscan` crate (runs on both backends)
- Soft assignment pass: score all memories against centroids, threshold into domains
- `domain_scores` JSONB stored per memory for transparency / retroactive re-thresholding
- Domain discovery CLI and dashboard UI
- Auto-classification on ingest (once domains exist)
- Re-clustering during dream consolidation passes
- Domain management CLI (rename, merge, inspect)
### Phase 5: Federation (future)
- Node discovery via Mycelium / mDNS
- Memory sync protocol (UUID-based, last-write-wins)
- Possibly Iroh for content-addressed replication
- FSRS state merge (review history append, not overwrite)
## Crate Dependencies (new)
```toml
# Phase 1 — trait abstraction
trait-variant = "0.1"
# Phase 2 — Postgres
sqlx = { version = "0.8", features = ["runtime-tokio", "postgres", "uuid", "chrono", "json"] }
pgvector = "0.4" # sqlx integration for vector type
# Phase 3 — Auth
blake3 = "1" # key hashing
rand = "0.8" # key generation
# Phase 4 — Domain clustering
hdbscan = "0.10" # HDBSCAN — no eps tuning, variable density, built-in centroid calc
```
## Open Questions
1. **Trait granularity**: One big `MemoryStore` trait or split into `MemoryStore + SchedulingStore + GraphStore + DomainStore`? Splitting is cleaner but means more `dyn` parameters threading through handlers.
2. **Embedding on insert**: Should the storage backend call fastembed, or should the caller always provide the embedding? Current design says caller provides it, keeping the backend pure storage. But this means every client needs fastembed locally even if the DB is remote. For the server model, having the server compute embeddings makes more sense.
3. **pgvector dimension**: Fixed (e.g., `vector(768)`) or unconstrained (`vector`)? Fixed is faster for HNSW but requires migration if model changes.
4. **Sync conflict resolution for federation**: LWW per-UUID is simple but lossy. CRDTs would be more correct but massively more complex. For FSRS state specifically, merging review event logs would be ideal.
5. **Dashboard auth**: The 3D dashboard currently runs unauthenticated on localhost. With remote access, it needs the same auth. Should it use the same API keys or have a separate session/cookie mechanism?
6. **HDBSCAN `min_cluster_size`**: The main tuning knob. Too small → noisy micro-clusters. Too large → distinct topics get merged. Default of 10 should work for most cases, but may need a manual override or auto-sweep (run with several values, pick the one with best silhouette score).
7. **Domain drift**: Over time, the character of a domain changes. How aggressively should re-clustering reshape existing domains? Conservative (only propose splits/merges, never auto-apply) vs. aggressive (auto-reassign memories whose scores drifted below threshold)?
8. **Spreading activation across domains**: When searching within a single domain, should graph edges that cross into other domains be followed? Probably yes for recall quality, but with decaying weight as you cross boundaries.
9. **Threshold tuning**: The `assign_threshold` (0.65 default) determines how many memories are multi-domain vs single-domain vs unclassified. Too low → everything is multi-domain (useless). Too high → too many unclassified. Could be auto-tuned per dataset by targeting a specific unclassified ratio (e.g., "keep fewer than 10% unclassified").