vestige/docs/adr/0001-pluggable-storage-and-network-access.md

ADR 0001: Pluggable Storage Backend, Network Access, and Emergent Domains

Status: Accepted Date: 2026-04-21 Related: docs/prd/001-getting-centralized-vestige.md

---

Context

Vestige v2.x runs as a per-machine local process: stdio MCP transport, SQLite + FTS5 + USearch HNSW in ~/.vestige/, fastembed locally for embeddings. This is ideal for single-machine single-agent use but blocks three real needs:

• Multi-machine access -- same memory brain from laptop, desktop, server
• Multi-agent access -- multiple AI clients against one store concurrently
• Future federation -- syncing memory between decentralized nodes (MOS / Threefold grid)

SQLite's single-writer model and lack of a native network protocol make it unsuitable as a centralized server. PostgreSQL + pgvector collapses our three storage layers (SQLite, FTS5, USearch) into one engine with MVCC concurrency, auth, and replication.

Separately, Vestige today has no notion of domain or project scope -- all memories share one namespace. For a multi-machine brain, users want soft topical boundaries ("dev", "infra", "home") without manual tenanting. HDBSCAN clustering on embeddings produces these boundaries from the data itself.

The PRD at docs/prd/001-getting-centralized-vestige.md sketches the full design. This ADR records the architectural decisions and resolves the open questions from that document.

---

Decision

Introduce two new trait boundaries, a network transport layer, and a domain classification module. All four changes ship in parallel phases.

Trait boundaries:

1. MemoryStore -- single trait covering CRUD, hybrid search, FSRS scheduling, graph edges, and domains. One big trait, not four.
2. Embedder -- separate trait for text-to-vector encoding. Storage never calls fastembed directly. Callers (cognitive engine locally, HTTP server remotely) compute embeddings and pass them into the store.

Backends:

• SqliteMemoryStore -- existing code refactored behind the trait, no behavior change.
• PgMemoryStore -- new, using sqlx + pgvector + tsvector. Selectable at runtime via vestige.toml.

Network:

• MCP over Streamable HTTP on the existing Axum server.
• API key auth middleware (blake3-hashed, stored in api_keys table).
• Dashboard uses the same API keys for login, then signed session cookies for subsequent requests.

Domain classification:

• HDBSCAN clustering over embeddings to discover domains automatically.
• Soft multi-domain assignment -- raw similarity scores stored per memory, every domain above a threshold is assigned.
• Conservative drift handling -- propose splits/merges, never auto-apply.

---

Architecture Overview

Component Breakdown

1. Embedder trait (new module crates/vestige-core/src/embedder/)

   • async fn embed(&self, text: &str) -> Result<Vec<f32>>
   • fn model_name(&self) -> &str
   • fn dimension(&self) -> usize
   • Impls: FastembedEmbedder (local ONNX, today), future JinaEmbedder, OpenAiEmbedder, etc.
   • Stays pluggable forever -- no lock-in to fastembed or to nomic-embed-text.

2. MemoryStore trait (new module crates/vestige-core/src/storage/trait.rs)

   • One trait, ~25 methods across CRUD, search, FSRS, graph, domain sections.
   • Uses trait_variant::make to generate a Send-bound variant for Arc<dyn MemoryStore> in Axum/tokio contexts.
   • The 29 cognitive modules operate exclusively through this trait. No direct SQLite or Postgres access from the modules.

3. SqliteMemoryStore (refactor of existing crates/vestige-core/src/storage/sqlite.rs)

   • Existing rusqlite + FTS5 + USearch code, wrapped behind the trait.
   • Add domains TEXT[] equivalent (JSON-encoded array column in SQLite).
   • Add domain_scores JSON column.
   • No behavioral change for current users.

4. PgMemoryStore (new crates/vestige-core/src/storage/postgres.rs)

   • sqlx::PgPool with compile-time checked queries.
   • pgvector HNSW index for vector search, tsvector + GIN for FTS.
   • Native array columns for domains, JSONB for domain_scores and metadata.
   • Hybrid search via RRF (Reciprocal Rank Fusion) in a single SQL query.

5. Model registry

   • Per-database table embedding_model with (name, dimension, hash, created_at).
   • Both backends refuse writes from an embedder whose signature doesn't match the registered row.
   • Model swap = vestige migrate --reembed --model=<new>, O(n) cost, explicit.

6. DomainClassifier cognitive module (new crates/vestige-core/src/neuroscience/domain_classifier.rs)

   • Owns the HDBSCAN discovery pass (using the hdbscan crate).
   • Computes soft-assignment scores for every memory against every centroid.
   • Stores raw domain_scores: HashMap<String, f64> per memory; thresholds into the domains array using assign_threshold (default 0.65).
   • Runs discovery on demand (vestige domains discover) or during dream consolidation passes.

7. HTTP MCP transport (extension of existing Axum server in crates/vestige-mcp/src/)

   • New route POST /mcp for Streamable HTTP JSON-RPC.
   • New route GET /mcp for SSE (for long-running operations).
   • REST API under /api/v1/ for direct HTTP clients (non-MCP integrations).
   • Auth middleware validates Authorization: Bearer ... or X-API-Key, plus signed session cookies for dashboard.

8. Key management (new crates/vestige-mcp/src/auth/)

   • api_keys table -- blake3-hashed keys, scopes, optional domain filter, last-used timestamp.
   • CLI: vestige keys create|list|revoke.

9. FSRS review event log (future-proofing for federation)

   • New table review_events -- append-only (memory_id, timestamp, rating, prior_state, new_state).
   • Current scheduling table becomes a materialized view over the event log (reconstructible from events).
   • Phase 5 federation merges event logs, not derived state. Zero cost today, avoids lock-in tomorrow.

Data Flow

Local mode (stdio MCP, unchanged UX):

stdio client -> McpServer -> CognitiveEngine -> FastembedEmbedder -> MemoryStore (SQLite)

Server mode (HTTP MCP, new):

Remote client -> Axum HTTP -> auth middleware -> CognitiveEngine
    -> FastembedEmbedder (server-side) -> MemoryStore (Postgres)

The cognitive engine is backend-agnostic. The embedder and the store are both swappable. The 7-stage search pipeline (overfetch -> cross-encoder rerank -> temporal -> accessibility -> context match -> competition -> spreading activation) sits above the MemoryStore trait and works identically against either backend.

Orthogonality of HDBSCAN and Reranking

HDBSCAN and the cross-encoder reranker solve different problems and both stay:

• HDBSCAN discovers domains by clustering embeddings. Runs once per discovery pass. Produces centroids. Used to filter search candidates, not to rank them.
• Cross-encoder reranker (Jina Reranker v1 Turbo) scores query-document pairs at search time. Runs on every search. Produces ranked results.

Domain membership is a filter applied before or during overfetch; reranking runs on whatever candidate set survives the filter.

---

Alternatives Considered

Alternative	Pros	Cons	Why Not
Split into 4 traits (MemoryStore + SchedulingStore + GraphStore + DomainStore)	Cleaner interface segregation	Every module holds 4 trait objects, coordinates transactions across them	One trait is fine in Rust; extract sub-traits later if a genuine need appears
Embedding computed inside the backend	Simpler call sites for callers	Backend becomes aware of embedding models; can't support remote clients without local fastembed	Keep storage pure; separate Embedder trait handles pluggability
Unconstrained pgvector vector (no dimension)	Flexible for model swaps	HNSW still needs fixed dims at index creation; hides a meaningful migration as "silent"	Fixed dimension per install, explicit --reembed migration
Dashboard separate auth (cookies only, no keys)	Simpler dashboard UX	Two auth systems to maintain	Shared API keys with session cookie layer on top
Auto-tuned assign_threshold targeting an unclassified ratio	Adapts to corpus	Hard to debug ("why did this memory change domain?"); magical	Static 0.65 default, config-tunable, dashboard shows domain_scores for manual retuning
Aggressive drift (auto-reassign memories whose scores drifted)	Always up-to-date domains	Breaks user muscle memory; silent reshuffling	Conservative: always propose, user accepts
CRDTs for federation state	Mathematically clean merges	Massive complexity, performance cost, overkill	Defer; design FSRS as event log now so any future sync model works

---

Consequences

Positive

• Single memory brain accessible from every machine.
• Multi-agent concurrent access via Postgres MVCC.
• Natural topical scoping emerges from data, not manual tenants.
• Future embedding model swaps are a config + migration, not a rewrite.
• Federation has a clean on-ramp (event log merge) without committing now.
• The Embedder / MemoryStore split unlocks other storage backends later (Redis, Qdrant, Iroh-backed blob store, etc.) with minimal work.

Negative

• Operating a Postgres instance is more work than managing a SQLite file.
• Users who stay on SQLite gain nothing from this ADR (but lose nothing either).
• Migration (vestige migrate --from sqlite --to postgres) is a sensitive operation for users with months of memories -- needs strong testing.
• HDBSCAN + re-soft-assignment runs in O(n) over all embeddings. At 100k+ memories this starts to matter; manageable but not free.

Risks

• Trait abstraction leaks: a cognitive module might need backend-specific behavior (e.g., Postgres triggers for tsvector). Mitigation: keep such logic inside the backend impl; the trait stays pure. Escalation: if a module genuinely cannot express what it needs through the trait, the trait grows, not the module bypasses.
• Embedding model drift: users on older fastembed versions silently producing slightly different vectors after a fastembed upgrade. Mitigation: model hash in the registry, refuse mismatched writes, surface a clear error.
• Auth misconfiguration: a user binds to 0.0.0.0 without setting auth.enabled = true. Mitigation: refuse to start with non-localhost bind and auth disabled. Hard error, not a warning.
• Re-clustering feedback loop: dream consolidation proposes re-clusters, which the user accepts, which changes classifications, which affects future retrievals, which affect future dreams. Mitigation: cap re-cluster frequency (every 5th dream by default), require explicit user acceptance of proposals.
• Cross-domain spreading activation weight (0.5 default): arbitrary choice; could be too aggressive or too lax. Mitigation: config-tunable; instrument retrieval quality metrics in the dashboard so the user sees impact.

---

Resolved Decisions (from Q&A)

#	Question	Resolution
1	Trait granularity	Single MemoryStore trait
2	Embedding on insert	Caller provides; separate Embedder trait for pluggability
3	pgvector dimension	Fixed per install, derived from Embedder::dimension() at schema init
4	Federation sync	Defer algorithm; store FSRS reviews as append-only event log now
5	Dashboard auth	Shared API keys + signed session cookie
6	HDBSCAN min_cluster_size	Default 10; user reruns with --min-cluster-size N; no auto-sweep
7	Domain drift	Conservative -- always propose splits/merges, never auto-apply
8	Cross-domain spreading activation	Follow with decay factor 0.5 (tunable)
9	Assignment threshold	Static 0.65 default, config-tunable, raw domain_scores stored for introspection

---

Implementation Plan

Five phases, each independently shippable.

Phase 1: Storage trait extraction

• Define MemoryStore and Embedder traits in vestige-core.
• Refactor SqliteMemoryStore to implement MemoryStore; no behavior change.
• Refactor FastembedEmbedder to implement Embedder.
• Add embedding_model registry table; enforce consistency on write.
• Add domains TEXT[]-equivalent and domain_scores JSON columns to SQLite (empty for all existing rows).
• Convert all 29 cognitive modules to operate via the traits.
• Acceptance: existing test suite passes unchanged. Zero warnings.

Phase 2: PostgreSQL backend

• PgMemoryStore with sqlx, pgvector, tsvector.
• sqlx migrations (crates/vestige-core/migrations/postgres/).
• Backend selection via vestige.toml [storage] section.
• vestige migrate --from sqlite --to postgres command.
• vestige migrate --reembed command for model swaps.
• Acceptance: full test suite runs green against Postgres with a testcontainer.

Phase 3: Network access

• Streamable HTTP MCP route on Axum (POST /mcp, GET /mcp for SSE).
• REST API under /api/v1/.
• API key table + blake3 hashing + vestige keys create|list|revoke.
• Auth middleware (Bearer, X-API-Key, session cookie).
• Refuse non-localhost bind without auth enabled.
• Acceptance: MCP client over HTTP works from a second machine; dashboard login flow works; unauth requests return 401.

Phase 4: Emergent domain classification

• DomainClassifier module using the hdbscan crate.
• vestige domains discover|list|rename|merge CLI.
• Automatic soft-assignment pipeline (compute domain_scores on ingest, threshold into domains).
• Re-cluster every Nth dream consolidation (default 5); surface proposals in the dashboard.
• Context signals (git repo, IDE) as soft priors on classification.
• Cross-domain spreading activation with 0.5 decay.
• Acceptance: on a corpus of 500+ mixed memories, discover produces sensible clusters; search scoped to a domain returns tightly relevant results.

Phase 5: Federation (future, explicitly out of scope for this ADR's

acceptance)

• Node discovery (Mycelium / mDNS).
• Memory sync protocol over append-only review events and LWW-per-UUID for memory records.
• Explicit follow-up ADR before any code.

---

Open Questions

None at ADR acceptance time. Follow-up items that are implementation choices, not architectural:

• Precise cross-domain decay weight (start at 0.5, instrument, tune)
• Dashboard histogram of domain_scores (UX design detail)
• Whether to gate Postgres behind a Cargo feature flag (postgres-backend) or always compile it in (lean toward feature flag to keep SQLite-only builds small)