-`PgMemoryStore` struct implementing the Phase 1 `MemoryStore` trait against `sqlx::PgPool`, including compile-time checked queries via `sqlx::query!` / `sqlx::query_as!`.
- First-class `pgvector` integration: typed `Vector` columns, HNSW index (`vector_cosine_ops`, `m = 16`, `ef_construction = 64`), and use of the cosine-distance operator `<=>`.
- First-class Postgres FTS: GENERATED `tsvector` column (`search_vec`) with `setweight` (A=content, B=node_type, C=tags), GIN index, and `websearch_to_tsquery` at query time.
- Hybrid search via Reciprocal Rank Fusion (RRF) expressed as a single SQL statement with CTEs for FTS and vector subqueries, with optional domain filter through array overlap (`&&`).
- sqlx migrations directory at `crates/vestige-core/migrations/postgres/`, numbered `{NNNN}_{name}.up.sql` / `{NNNN}_{name}.down.sql`, runnable by `sqlx::migrate!` at startup and by `sqlx-cli`.
- Offline query cache committed under `crates/vestige-core/.sqlx/` so a DATABASE_URL is not required at build time.
- Backend selection via `vestige.toml`: `[storage]` section with `backend = "sqlite" | "postgres"` plus the per-backend subsection (`[storage.sqlite]`, `[storage.postgres]`). Exclusive at compile time via `postgres-backend` feature, exclusive at runtime via the enum.
- CLI: `vestige migrate --reembed --model=<new>` -- O(n) re-embed under a new `Embedder`, registry update, HNSW rebuild.
- Testcontainer-based integration tests using the `pgvector/pgvector:pg16` image, behind the `postgres-backend` feature so SQLite-only builds remain untouched.
-`PgMemoryStore` parity with `SqliteMemoryStore` across every public `MemoryStore` method defined in Phase 1.
### Out of scope
- Phase 3 (network access): HTTP MCP transport, API key auth, `vestige keys` CLI. The `api_keys` DDL is declared by Phase 3; Phase 2 does not create it.
- Phase 4 (emergent domain classification): `DomainClassifier`, HDBSCAN, discover / rename / merge CLI. Phase 2 provisions the `domains` and `domain_scores` columns and the `domains` table structure so Phase 4 slots in without further migration, but does not compute or classify.
- Phase 5 (federation): cross-node sync. The `review_events` table is declared in Phase 1; Phase 2 only references it where FSRS writes happen.
- Changes to the cognitive engine, Phase 1 traits, or the embedding pipeline itself. Phase 2 only adds a backend.
- SQLCipher parity for Postgres. Operator responsibility (TLS to Postgres, pgcrypto, disk-level encryption) is out of scope for this phase.
---
## Prerequisites
### Expected Phase 1 artifacts (consumed, not produced)
Phase 2 treats all of the following as fixed interfaces. Each path is the expected Phase 1 location.
-`crates/vestige-core/src/storage/mod.rs` -- re-exports the trait and the two concrete backends.
-`crates/vestige-core/src/storage/memory_store.rs` -- defines the `MemoryStore` trait (generated by `trait_variant::make` from `LocalMemoryStore`) with the full CRUD, search, FSRS, graph, and domain surface from the PRD. Phase 2 implements every method here.
-`crates/vestige-core/src/storage/error.rs` -- `StoreError` enum plus `pub type StoreResult<T> = Result<T, StoreError>`. Phase 2 extends this with `StoreError::Postgres(sqlx::Error)` and `StoreError::Migrate(sqlx::migrate::MigrateError)` via `From` impls (the variants themselves MUST live behind `#[cfg(feature = "postgres-backend")]`).
-`crates/vestige-core/src/embedder/mod.rs` -- `Embedder` trait with `embed`, `model_name`, `dimension`, `model_hash`. Phase 2 calls `model_name()`, `dimension()`, and `model_hash()` for the registry.
-`crates/vestige-core/src/storage/sqlite.rs` -- `SqliteMemoryStore: MemoryStore`. Phase 2's `migrate --from sqlite --to postgres` uses this as the source.
-`crates/vestige-core/src/storage/registry.rs` -- `EmbeddingModelRegistry` abstraction that both backends implement. Phase 2 supplies a Postgres version writing to `embedding_model`.
-`crates/vestige-core/migrations/sqlite/` -- V12 (Phase 1) adds `domains TEXT` (JSON-encoded array), `domain_scores TEXT` (JSON), `embedding_model(name, dimension, hash, created_at)`, and `review_events(id, memory_id, timestamp, rating, prior_state, new_state)`. Phase 2 mirrors every column and table in Postgres.
If any of the above is missing when Phase 2 starts, the first action is to surface the gap back to Phase 1 -- do NOT backfill a partial trait in Phase 2.
### Required crates (declared in Phase 2, not installed by this doc)
The agent running Phase 2 uses `cargo add` in `crates/vestige-core/` for each dependency below. Exact versions and feature flags:
-`sqlx@0.8` with features `runtime-tokio`, `tls-rustls`, `postgres`, `uuid`, `chrono`, `json`, `migrate`, `macros`. Optional (gated by `postgres-backend`).
-`pgvector@0.4` with features `sqlx`. Optional (gated by `postgres-backend`).
-`deadpool` is NOT needed; `sqlx::PgPool` is the pool.
-`toml@0.8` (no features) for `vestige.toml` parsing. Moved to non-optional because both backends share the config surface.
-`figment@0.10` with features `toml`, `env` -- optional, only if Phase 1 has not already picked a config loader. If Phase 1 ships a loader, skip `figment` and reuse.
-`dirs@6` -- already a transitive `directories` dependency; reuse existing.
-`tokio-stream@0.1` (no features). Used by migrate commands for streamed iteration.
-`indicatif@0.17` (no features). Progress bars for the migrate CLI.
-`futures@0.3` with features `std`. Consumed by sqlx stream combinators.
Dev-only (under `[dev-dependencies]` in `crates/vestige-core/Cargo.toml`, gated by `postgres-backend`):
-`testcontainers@0.22` with features `blocking` off, `async` on (default).
-`testcontainers-modules@0.10` with features `postgres`.
-`tokio@1` features `macros`, `rt-multi-thread` (already present for core tests).
-`criterion@0.5` already present; add a new `[[bench]]` entry.
Feature additions in `crates/vestige-core/Cargo.toml`:
`postgres-backend` is OFF by default. `default = ["embeddings", "vector-search", "bundled-sqlite"]` stays unchanged. `vestige-mcp` forwards a new feature `postgres-backend = ["vestige-core/postgres-backend"]`.
### External tooling
- PostgreSQL 16 or newer (uses `gen_random_uuid()` from `pgcrypto` bundled via `CREATE EXTENSION pgcrypto` in migration 0001; pgvector HNSW indexes require pgvector 0.5+).
- The `pgvector` extension installed in the target database (our migration issues `CREATE EXTENSION IF NOT EXISTS vector`).
-`sqlx-cli@0.8` installed on the developer machine for `cargo sqlx prepare --workspace` and `cargo sqlx migrate add` (not a build-time requirement once `.sqlx/` is committed).
- Docker or Podman reachable by the test harness for `testcontainers-modules::postgres` to spin up `pgvector/pgvector:pg16`.
- A local Postgres cluster for `sqlx prepare`, manual migration work, and `vestige migrate --to postgres` smoke runs. The recipe for standing one up on Arch/CachyOS (install, initdb, role + db, pgvector, connection string at `~/.vestige_pg_pw`) lives in `docs/plans/local-dev-postgres-setup.md`. Postgres 18 from the Arch repo satisfies the "16 or newer" requirement above. Phase 2 work assumes `DATABASE_URL` points at that cluster once migrations are applied.
- MSRV 1.91 (per `CLAUDE.md`). `sqlx 0.8` is compatible.
-`rustflags` unchanged. No `nightly`-only features.
---
## Deliverables
1. Feature gate `postgres-backend` in `crates/vestige-core/Cargo.toml` and `crates/vestige-mcp/Cargo.toml` that cleanly disables all Postgres code paths when off.
8.`crates/vestige-core/migrations/postgres/0002_hnsw.up.sql` + `0002_hnsw.down.sql` -- HNSW index creation separated so it can be `CREATE INDEX CONCURRENTLY` during reembed.
9.`crates/vestige-core/src/config.rs` -- `VestigeConfig`, `StorageConfig`, `SqliteConfig`, `PostgresConfig`, `EmbeddingsConfig`, plus a single `VestigeConfig::load(path: Option<&Path>)` returning `Result<Self, ConfigError>`.
10.`crates/vestige-core/src/storage/postgres/migrate_cli.rs` -- streaming SQLite-to-Postgres copy, domain-aware, with `indicatif` progress.
11.`crates/vestige-core/src/storage/postgres/reembed.rs` -- `ReembedPlan` and its driver; re-encodes all memories via a supplied `Embedder`, updates `embedding_model`, rebuilds HNSW.
12.`crates/vestige-mcp/src/bin/cli.rs` -- two new `clap` subcommands `Migrate` (union of `--from/--to` and `--reembed` variants, one subcommand or two, see Open Questions) wired to deliverables 10 and 11.
14.`tests/phase_2/` -- six integration test files listed in the Test Plan.
15.`crates/vestige-core/benches/pg_hybrid_search.rs` -- Criterion benches for RRF search at 1k and 100k memories, gated by `postgres-backend`.
16.`docs/runbook/postgres.md` -- brief ops note covering extension install, `max_connections`, backup discipline, and rollback caveats. (Short; only required for the "rollback of migrate" deliverable.)
pgvector = { version = "0.4", features = ["sqlx"], optional = true }
tokio-stream = { version = "0.1", optional = true }
futures = { version = "0.3", optional = true }
toml = "0.8"
indicatif = "0.17"
```
- **Behavior notes**: keep the two backends mutually compilable per `CLAUDE.md`. Every `use sqlx::...` sits under `#[cfg(feature = "postgres-backend")]`. Every module under `crates/vestige-core/src/storage/postgres/` carries `#![cfg(feature = "postgres-backend")]` as its file-level attribute.
record.embedding.as_ref().map(|v| Vector::from(v.clone())) as Option<Vector>,
record.metadata,
record.created_at,
record.updated_at,
)
.execute(&self.pool)
.await?;
```
- **Behavior notes**:
-`StoreError` gets two new variants behind the feature:
```rust
#[cfg(feature = "postgres-backend")]
#[error("postgres error: {0}")]
Postgres(#[from] sqlx::Error),
#[cfg(feature = "postgres-backend")]
#[error("postgres migration error: {0}")]
Migrate(#[from] sqlx::migrate::MigrateError),
```
-`classify()` on Postgres implements the PRD's cosine-similarity-to-centroid computation inside SQL using `1 - (centroid <=> $1::vector)` over the `domains` table and returns rows sorted descending. This mirrors the behavior a `DomainClassifier` in Phase 4 uses; Phase 2 ships the backend capability but does not call it.
- **Behavior notes**: acquire timeout chosen to exceed the 30-second testcontainer spin-up requirement. `application_name = "vestige"` makes `pg_stat_activity` readable from `psql` during debugging.
memory_id UUID PRIMARY KEY REFERENCES memories(id) ON DELETE CASCADE,
stability DOUBLE PRECISION NOT NULL DEFAULT 0.0,
difficulty DOUBLE PRECISION NOT NULL DEFAULT 0.0,
retrievability DOUBLE PRECISION NOT NULL DEFAULT 1.0,
last_review TIMESTAMPTZ,
next_review TIMESTAMPTZ,
reps INTEGER NOT NULL DEFAULT 0,
lapses INTEGER NOT NULL DEFAULT 0
);
-- Graph edges (spreading activation)
CREATE TABLE edges (
source_id UUID NOT NULL REFERENCES memories(id) ON DELETE CASCADE,
target_id UUID NOT NULL REFERENCES memories(id) ON DELETE CASCADE,
edge_type TEXT NOT NULL DEFAULT 'related',
weight DOUBLE PRECISION NOT NULL DEFAULT 1.0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (source_id, target_id, edge_type)
);
-- FSRS review event log (Phase 1 creates this; Phase 2 mirrors it for Postgres).
-- Append-only. Used for future federation (Phase 5).
CREATE TABLE review_events (
id BIGSERIAL PRIMARY KEY,
memory_id UUID NOT NULL REFERENCES memories(id) ON DELETE CASCADE,
timestamp TIMESTAMPTZ NOT NULL DEFAULT now(),
rating SMALLINT NOT NULL,
prior_state JSONB NOT NULL,
new_state JSONB NOT NULL
);
-- Indexes on memories (vector index is declared separately in 0002_hnsw.up.sql)
CREATE INDEX idx_memories_fts ON memories USING GIN (search_vec);
CREATE INDEX idx_memories_domains ON memories USING GIN (domains);
CREATE INDEX idx_memories_tags ON memories USING GIN (tags);
CREATE INDEX idx_memories_node_type ON memories (node_type);
CREATE INDEX idx_memories_created ON memories (created_at);
CREATE INDEX idx_memories_updated ON memories (updated_at);
-- Indexes on scheduling
CREATE INDEX idx_scheduling_next_review ON scheduling (next_review);
CREATE INDEX idx_scheduling_last_review ON scheduling (last_review);
-- Indexes on edges
CREATE INDEX idx_edges_target ON edges (target_id);
CREATE INDEX idx_edges_source ON edges (source_id);
CREATE INDEX idx_edges_type ON edges (edge_type);
-- Indexes on review_events
CREATE INDEX idx_review_events_memory ON review_events (memory_id);
CREATE INDEX idx_review_events_ts ON review_events (timestamp);
-- Update trigger on memories.updated_at
CREATE OR REPLACE FUNCTION memories_set_updated_at() RETURNS TRIGGER AS $$
BEGIN
NEW.updated_at := now();
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER trg_memories_updated_at
BEFORE UPDATE ON memories
FOR EACH ROW EXECUTE FUNCTION memories_set_updated_at();
```
`0001_init.down.sql`:
```sql
DROP TRIGGER IF EXISTS trg_memories_updated_at ON memories;
DROP FUNCTION IF EXISTS memories_set_updated_at();
DROP INDEX IF EXISTS idx_review_events_ts;
DROP INDEX IF EXISTS idx_review_events_memory;
DROP INDEX IF EXISTS idx_edges_type;
DROP INDEX IF EXISTS idx_edges_source;
DROP INDEX IF EXISTS idx_edges_target;
DROP INDEX IF EXISTS idx_scheduling_last_review;
DROP INDEX IF EXISTS idx_scheduling_next_review;
DROP INDEX IF EXISTS idx_memories_updated;
DROP INDEX IF EXISTS idx_memories_created;
DROP INDEX IF EXISTS idx_memories_node_type;
DROP INDEX IF EXISTS idx_memories_tags;
DROP INDEX IF EXISTS idx_memories_domains;
DROP INDEX IF EXISTS idx_memories_fts;
DROP TABLE IF EXISTS review_events;
DROP TABLE IF EXISTS edges;
DROP TABLE IF EXISTS scheduling;
DROP TABLE IF EXISTS memories;
DROP TABLE IF EXISTS domains;
DROP TABLE IF EXISTS embedding_model;
```
`0002_hnsw.up.sql` (separated so reembed can drop-and-recreate without touching the rest of the schema):
```sql
-- HNSW index on memories.embedding.
-- pgvector requires the column to have a typmod (fixed dimension) for HNSW.
-- The dimension is stamped by the application at startup via ALTER TABLE
-- using the embedder's dimension() method (see PgMemoryStore::connect).
-- We express the index with the generic vector_cosine_ops operator class.
CREATE INDEX idx_memories_embedding_hnsw
ON memories USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);
```
`0002_hnsw.down.sql`:
```sql
DROP INDEX IF EXISTS idx_memories_embedding_hnsw;
```
- **Behavior notes**:
- pgvector HNSW requires a typmod. `PgMemoryStore::connect` runs `ALTER TABLE memories ALTER COLUMN embedding TYPE vector($N)` with `$N = embedder.dimension()` exactly once, guarded by a check against `embedding_model` (first startup ever) or validated against it on subsequent starts. If `embedder.dimension()` differs from the stored one and `embedding_model` is non-empty, return `StoreError::EmbeddingDimensionMismatch` -- the user must run `vestige migrate --reembed`.
-`ALTER COLUMN ... TYPE vector($N)` on a populated column fails unless the data fits; that is the desired safety net.
- The `tsvector` GENERATED column uses `array_to_string(tags, ' ')` rather than `array_to_tsvector` from the PRD sketch, because `array_to_tsvector` is not a core function in Postgres 16 and would require an extension. The behavior is equivalent for weight C.
-`gen_random_uuid()` comes from `pgcrypto`. In Postgres 13+ it is also available from core; we keep the extension for older compatibility paths.
- MVCC: all table writes are transactional; no explicit locks. `INSERT ... ON CONFLICT DO UPDATE` is used in `upsert_domain`, `update_scheduling`, and edge idempotency.
ts_rank_cd(m.search_vec, websearch_to_tsquery('english', p.q_text)) AS score,
ROW_NUMBER() OVER (
ORDER BY ts_rank_cd(m.search_vec, websearch_to_tsquery('english', p.q_text)) DESC
) AS rank
FROM memories m, params p
WHERE p.q_text <> ''
AND m.search_vec @@ websearch_to_tsquery('english', p.q_text)
AND (p.dom_filter IS NULL OR m.domains && p.dom_filter)
AND (p.nt_filter IS NULL OR m.node_type = ANY(p.nt_filter))
AND (p.tag_filter IS NULL OR m.tags && p.tag_filter)
LIMIT (SELECT overfetch FROM params)
),
vec AS (
SELECT m.id,
1 - (m.embedding <=> p.q_vec) AS score,
ROW_NUMBER() OVER (
ORDER BY m.embedding <=> p.q_vec
) AS rank
FROM memories m, params p
WHERE m.embedding IS NOT NULL
AND p.q_vec IS NOT NULL
AND (p.dom_filter IS NULL OR m.domains && p.dom_filter)
AND (p.nt_filter IS NULL OR m.node_type = ANY(p.nt_filter))
AND (p.tag_filter IS NULL OR m.tags && p.tag_filter)
LIMIT (SELECT overfetch FROM params)
),
fused AS (
SELECT COALESCE(f.id, v.id) AS id,
COALESCE(1.0 / (60 + f.rank), 0.0)
+ COALESCE(1.0 / (60 + v.rank), 0.0) AS rrf_score,
f.score AS fts_score,
v.score AS vector_score
FROM fts f FULL OUTER JOIN vec v ON f.id = v.id
)
SELECT m.id AS "id!: Uuid",
m.domains AS "domains!: Vec<String>",
m.domain_scores AS "domain_scores!: serde_json::Value",
m.content AS "content!",
m.node_type AS "node_type!",
m.tags AS "tags!: Vec<String>",
m.embedding AS "embedding?: Vector",
m.metadata AS "metadata!: serde_json::Value",
m.created_at AS "created_at!: chrono::DateTime<chrono::Utc>",
m.updated_at AS "updated_at!: chrono::DateTime<chrono::Utc>",
fused.rrf_score AS "rrf_score!: f64",
fused.fts_score AS "fts_score?: f64",
fused.vector_score AS "vector_score?: f64"
FROM fused
JOIN memories m ON m.id = fused.id
ORDER BY fused.rrf_score DESC
LIMIT (SELECT final_limit FROM params);
```
- **Behavior notes**:
-`OVERFETCH_MULT * query.limit` is passed as `$3`. Final `$4` is `query.limit`.
- Empty text query is allowed; the `fts` CTE returns zero rows (`p.q_text <> ''`) and the result degrades to pure vector search, which matches `vector_search` behavior.
- Null embedding is allowed; the `vec` CTE returns zero rows and the result degrades to pure FTS, which matches `fts_search` behavior.
-`fts_search` and `vector_search` are separate public methods on the trait. Each uses a simpler single-CTE query derived from the above by removing the other branch. Implementing them as thin wrappers over `rrf_search` with nullified inputs is acceptable but adds one extra plan per call; the explicit implementations win on latency.
-`min_retrievability` in `SearchQuery` is applied as a final filter by joining on `scheduling` in the outer `SELECT`. Adding that join unconditionally regresses simple searches; add it only when `query.min_retrievability.is_some()`.
- The serde representation matches the PRD: `[storage]` with `backend = "sqlite"` and a matching `[storage.sqlite]` or `[storage.postgres]` subsection.
- Because `StorageConfig` is `#[serde(tag = "backend")]`, an unknown backend string returns a clear error.
- If `postgres-backend` is compiled off and the user writes `backend = "postgres"`, deserialization returns "unknown variant `postgres`" -- loud failure. Phase 2 wraps this into `ConfigError::Invalid("postgres-backend feature not compiled in")`.
-`env`-override hooks (e.g., `VESTIGE_POSTGRES_URL`) are a Phase 3 concern; not added here.
use crate::storage::error::{StoreError, StoreResult};
use crate::storage::postgres::PgMemoryStore;
use crate::storage::sqlite::SqliteMemoryStore;
#[derive(Debug, Clone)]
pub struct SqliteToPostgresPlan {
pub sqlite_path: std::path::PathBuf,
pub postgres_url: String,
pub max_connections: u32,
pub batch_size: usize, // default 500
}
pub struct MigrationReport {
pub memories_copied: u64,
pub scheduling_rows: u64,
pub edges_copied: u64,
pub review_events_copied: u64,
pub domains_copied: u64,
pub errors: Vec<(Uuid, StoreError)>,
}
pub async fn run_sqlite_to_postgres(
plan: SqliteToPostgresPlan,
embedder: Arc<dynEmbedder>,
) -> StoreResult<MigrationReport>;
```
Algorithm:
1. Open source `SqliteMemoryStore` in read-only mode (`?mode=ro`).
2. Check source `embedding_model` registry; refuse if it disagrees with the supplied embedder unless the user also passed `--reembed`.
3. Open destination `PgMemoryStore` via `connect` (runs migrations, stamps dim).
4. Stream source rows in batches of `plan.batch_size` via a windowed query ordered by `created_at, id` (stable cursor; survives resume).
5. For each batch: begin a Postgres transaction, `INSERT INTO memories ... ON CONFLICT (id) DO NOTHING` for all rows, `INSERT INTO scheduling` likewise, commit. Copy domain assignments (`domains`, `domain_scores`) verbatim -- they are `[]` and `{}` for pre-Phase-4 SQLite data.
6. After memories finish, stream edges and review_events the same way.
7. Emit progress via `indicatif::ProgressBar` (one bar per table, multi-bar). Each 1000 rows log to tracing at INFO.
8. Return `MigrationReport` for the caller to print.
- **Behavior notes**:
- Memory-bounded: batch size 500 and sqlx streams mean memory usage stays O(batch * row_size), not O(total_rows).
- Idempotent: re-running replays only the rows not already present; `ON CONFLICT DO NOTHING` means partial runs recover.
- UUID strings from SQLite are parsed via `Uuid::parse_str` -- any mangled ID pushes to `errors` instead of aborting.
- The FTS `search_vec` is regenerated by Postgres via the GENERATED column; no data to copy.
-`review_events` may not exist in Phase 1 SQLite for pre-V12 databases. The migrator detects missing tables via `SELECT name FROM sqlite_master` and skips gracefully.
- A separate `--dry-run` flag prints the counts per table without writing.
pub concurrent_index: bool, // default false; use CREATE INDEX (not CONCURRENTLY)
}
pub struct ReembedReport {
pub rows_updated: u64,
pub duration_secs: f64,
pub index_rebuild_secs: f64,
}
pub async fn run_reembed(
store: &PgMemoryStore,
new_embedder: Arc<dynEmbedder>,
plan: ReembedPlan,
) -> StoreResult<ReembedReport>;
```
Algorithm:
1. Verify `new_embedder.dimension()` != stored dimension OR `new_embedder.model_hash()` != stored hash -- otherwise no-op and return `rows_updated = 0`.
2.`BEGIN; ALTER TABLE memories ALTER COLUMN embedding DROP NOT NULL`; not actually needed (column is already nullable) but shown here for documentation.
3. If `plan.drop_hnsw_first`, execute `DROP INDEX IF EXISTS idx_memories_embedding_hnsw;` so updates are not slowed by index maintenance. This is the recommended path; `REINDEX` is kept in the Open Questions as an alternative.
4. Stream all `id, content` from `memories` ordered by `id`.
5. For each batch of `plan.batch_size`: call `new_embedder.embed_batch(&texts)` (Phase 1 trait exposes batched embedding when available; otherwise loop single `embed`). Then:
```sql
UPDATE memories
SET embedding = v.embedding::vector
FROM UNNEST($1::uuid[], $2::real[][]) AS v(id, embedding)
WHERE memories.id = v.id;
```
6. After all rows updated: run `ALTER TABLE memories ALTER COLUMN embedding TYPE vector($NEW_DIM)` if dimension changed.
7. Rebuild HNSW. If `plan.concurrent_index`, execute `CREATE INDEX CONCURRENTLY idx_memories_embedding_hnsw ...`; else `CREATE INDEX idx_memories_embedding_hnsw ...`.
8.`update_registry` with the new embedder.
9. Return `ReembedReport`.
- **Behavior notes**:
- Memory-bounded: batch_size * 2 (old + new texts) vectors in RAM at any time.
- The dimension change must happen AFTER all rows are updated (pgvector validates typmod on write when a typmod is present; we relax-then-tighten).
-`CONCURRENTLY` builds do not hold `AccessExclusiveLock`, but fail inside a transaction. That's why the outer driver runs index DDL as an autocommit statement (sqlx `execute` outside a pool transaction).
- For `--dry-run`, emit what *would* happen (row count, estimated embedder calls, estimated time using `rows / 50`-per-second baseline for local fastembed) and exit.
An alternate top-level layout (single `vestige migrate` with flags `--from`, `--to`, `--reembed`) is equivalent; the subcommand split is preferred because the two flag sets are disjoint (see Open Question 1).
- **Behavior notes**:
-`--from`/`--to` values are validated; the current Phase 2 build accepts only `sqlite` and `postgres`.
- For `reembed`, the `--model` string resolves to an `Embedder` via a factory already provided by Phase 1 (`Embedder::from_name(&str)`); Phase 2 does not invent new embedder constructors.
- Progress output on `stderr`; machine-readable summary on `stdout` as one-line JSON when `--json` is set (skipped for Phase 2 unless trivial).
### D11. Offline query cache (`.sqlx/`)
- **File**: `crates/vestige-core/.sqlx/` (committed directory of `query-*.json`)
- **Depends on**: all `sqlx::query!` call sites being final.
- **Procedure**: the developer runs `cargo sqlx prepare --workspace` with a live Postgres having the schema applied. Output goes into `crates/vestige-core/.sqlx/`. This directory is committed. CI enforces freshness by running `cargo sqlx prepare --workspace --check` against the same live Postgres (or failing that, any dev can reproduce by setting `SQLX_OFFLINE=true`).
- **Behavior notes**: `SQLX_OFFLINE=true` in `build.rs` or env is the default on CI and for downstream consumers. The `vestige-core` docs add a one-liner in README for contributors: "if you change any SQL in Phase 2 modules, rerun `cargo sqlx prepare` with a live DB."
### D12. Testcontainer harness (integration)
- **File**: `tests/phase_2/common/mod.rs` (the `common` convention used in `tests/phase_2/` crates)
- **Depends on**: D2 through D11.
- **Signatures**:
```rust
#![cfg(feature = "postgres-backend")]
use std::sync::Arc;
use testcontainers_modules::postgres::Postgres;
use testcontainers::{runners::AsyncRunner, ContainerAsync};
use vestige_core::embedder::Embedder;
use vestige_core::storage::postgres::PgMemoryStore;
let port = container.get_host_port_ipv4(5432).await?;
let url = format!(
"postgresql://postgres:postgres@127.0.0.1:{}/postgres", port
);
let store = PgMemoryStore::connect(&url, 4, embedder.as_ref()).await?;
Ok(Self { container, store })
}
}
```
- **Behavior notes**:
- Image `pgvector/pgvector:pg16` bundles pgvector into the official postgres:16 image.
- Pool size 4 is enough for tests without starving the container's default `max_connections = 100`.
-`ContainerAsync` is held for the whole test scope; drop tears down the container.
- A fake `TestEmbedder` in `common/test_embedder.rs` provides a deterministic hash-based embedding (no ONNX dependency in CI).
---
## Test Plan
### Unit tests (colocated in `src/`)
Under `crates/vestige-core/src/storage/postgres/`:
-`pool.rs` -- one test per `build_pool` branch: defaults, explicit `max_connections`, invalid URL returns `StoreError::Postgres`.
-`registry.rs` -- three tests: first-init writes row and alters typmod, reopen with same embedder returns Ok, reopen with different dimension returns `EmbeddingMismatch`.
-`search.rs` -- query-builder unit tests for parameter packing: empty text, null embedding, all three filters null, all three filters populated.
- Each test is written once as `async fn roundtrip_<method>(store: &dyn MemoryStore)` and invoked from two wrappers, one for SQLite and one for Postgres.
- Acceptance: every method returns equal results (except for `Uuid` ordering in `list_domains` where the test sorts before comparing).
**`tests/phase_2/pg_hybrid_search_rrf.rs`**
- Inserts 20 memories with known content ("rust async trait", "postgres hnsw vector", "fastembed onnx model", ...).
- Case 1: pure FTS. `SearchQuery { text: Some("rust trait"), embedding: None, ... }` returns the three Rust-related rows in order; `fts_score` populated, `vector_score` null.
- Case 2: pure vector. `SearchQuery { text: None, embedding: Some(embed("rust trait")), ... }` returns the same three rows via cosine; `vector_score` populated, `fts_score` null.
- Case 3: hybrid. Both set -- top hit has both scores; `rrf_score >= 1/(60+1) + 1/(60+1) = 0.0328`.
- Case 4: domain filter. 10 memories tagged with `domains = ["dev"]`, 10 with `["home"]`. Query with `domains: Some(vec!["dev"])` returns only dev memories.
- Case 5: edge case -- empty FTS query plus an embedding behaves identically to `vector_search`; empty embedding plus FTS query behaves identically to `fts_search`.
- Spawn 2 tasks concurrently running `update_scheduling` on overlapping IDs -- last write wins (MVCC), neither errors.
- Assert: all 1,600 rows present, no deadlocks, every task returns `Ok`.
- Run time <10secondsonacoldcontainer.
### Compile-time query verification
- CI step: `cargo sqlx prepare --workspace --check` against a CI-provisioned Postgres (GitHub Actions / Forgejo Actions services block). Fails CI if any `query!` macro goes stale.
- Alternative offline run for contributors: `SQLX_OFFLINE=true cargo check -p vestige-core --features postgres-backend`. CI runs both forms to ensure `.sqlx/` is up to date.
-`.sqlx/` is committed to the repo. A `.gitattributes` entry marks it as `linguist-generated=true` so it doesn't inflate language stats.
### Benchmarks
Under `crates/vestige-core/benches/pg_hybrid_search.rs` (Criterion), gated by `postgres-backend`.
-`pg_search_1k` -- populate 1,000 memories once per bench suite, measure `rrf_search` p50/p99 over 500 iterations. Target: p50 <10ms,p99<30msonalocalcontainer.
- [ ]`vestige migrate copy --from sqlite --to postgres` on a 10,000-memory corpus completes without data loss: row count parity, content byte-parity on a 1 percent sample, FSRS state preserved (stability, difficulty, reps, lapses, next_review), edge count parity.
- [ ]`vestige migrate reembed` with a dimension-changing embedder returns to a fully queryable state: HNSW present, `embedding_model` updated, no stale vectors, memory IDs untouched.
- [ ] Trait parity: every method on `MemoryStore` has at least one passing test against `PgMemoryStore`.
- [ ] Phase 1's existing SQLite suite continues to pass with zero changes required (Phase 2 is additive).
- [ ] The `postgres-backend` feature does not compile in SQLCipher (`encryption`) simultaneously (mutually exclusive at compile time, per project rule).
---
## Rollback Notes
- Every `*.up.sql` has a matching `*.down.sql` in `crates/vestige-core/migrations/postgres/`. `sqlx migrate revert` walks them in reverse order. Manual operator procedure: `sqlx migrate revert --database-url $URL --source crates/vestige-core/migrations/postgres`.
-`vestige migrate copy` is a one-way operation. The source SQLite DB is read-only during the run and untouched afterward; users retain their original file indefinitely. Recommended discipline: copy the SQLite file aside before starting, retain for 30 days.
-`vestige migrate reembed` is destructive to the `embedding` column. Recommended discipline: take a logical backup (`pg_dump --table=memories --table=embedding_model --table=scheduling`) before a reembed run. The tool prints that recommendation before starting and exits non-zero unless `--yes` is passed or the user is on a TTY that confirms.
- Feature-gate strategy: the default build remains SQLite-only. Downstream users pull `postgres-backend` explicitly: `cargo install --features postgres-backend vestige-mcp`. If the Postgres implementation fails in the field, users fall back to SQLite simply by flipping `vestige.toml`'s `[storage] backend = "sqlite"` and restarting. No data re-migration is needed if they retained their SQLite file.
- The `docs/runbook/postgres.md` deliverable (D16) captures this discipline as a one-page ops note.
---
## Open Implementation Questions
Each item has a recommendation. Ship that unless a reviewer objects.
### Q1. CLI shape: subcommand split vs flag union
- **Options**: (a) `vestige migrate copy --from sqlite --to postgres ...` and `vestige migrate reembed --model=...` (subcommand split); (b) `vestige migrate --from sqlite --to postgres ...` and `vestige migrate --reembed --model=...` under one `clap` command with disjoint flag groups (flag union).
- **RECOMMENDATION**: (a) subcommand split. The flag sets do not overlap and clap expresses the constraint more cleanly. The ADR string `vestige migrate --from sqlite --to postgres` can still be documented as a canonical alias by having `copy` accept it verbatim when `--from` is present.
- **RECOMMENDATION**: `postgres-backend`. Matches the ADR text and is explicit in `Cargo.toml` feature listings.
### Q3. sqlx offline mode strategy
- **Options**: (a) commit `.sqlx/` so downstream builds never need DATABASE_URL; (b) require `DATABASE_URL` at build time.
- **RECOMMENDATION**: (a). The repo already ships as a library; many downstream users will build from crates.io with no Postgres available. Committing `.sqlx/` costs ~100 kB.
### Q4. HNSW rebuild strategy during reembed
- **Options**: (a) `DROP INDEX; CREATE INDEX`; (b) `REINDEX INDEX CONCURRENTLY`; (c) `CREATE INDEX CONCURRENTLY` on a new name then swap.
- **RECOMMENDATION**: (a) by default for speed on empty / near-empty tables; expose `--concurrent-index` for large production corpora where locking the table is unacceptable. `REINDEX CONCURRENTLY` on pgvector HNSW is supported in pgvector 0.6+ but the community still reports edge cases with `maintenance_work_mem` -- skip unless a user explicitly opts in.
### Q5. Connection pool sizing default
- **Options**: 4, 10, 20, `cpus() * 2`.
- **RECOMMENDATION**: 10. Matches the PRD example, covers a single-operator load, and does not exhaust the default Postgres `max_connections = 100`. Configurable via `vestige.toml`.
- **RECOMMENDATION**: (b) pin exact. The float tag `pg16` has shipped breaking changes in the past (e.g., pg 16.0 to 16.1 interop). Pin to a specific pgvector minor and Postgres patch. CI bumps the tag via a single-line change.
### Q7. Empty-text and null-embedding behavior in `search`
- **Options**: (a) return an error if both are missing; (b) return an empty result; (c) return all memories sorted by `created_at DESC`.
- **RECOMMENDATION**: (a). A `search` call with no query is a bug in the caller; returning empty silently would hide the bug. The existing Phase 1 SQLite behavior (TBD but likely errors) is the tiebreaker.
### Q8. `classify()` SQL vs Rust
- **Options**: (a) compute cosine to all centroids in SQL (`SELECT id, 1 - (centroid <=> $1::vector) FROM domains ORDER BY ...`); (b) load centroids, compute in Rust.
- **RECOMMENDATION**: (a). Leverages pgvector's SIMD paths and avoids round-tripping centroid vectors. At Phase 4 scale (tens of centroids) the difference is marginal, but the SQL path is simpler and matches the rest of the backend.
### Q9. FSRS `review_events` writes: trait method vs implicit on `update_scheduling`
- **Options**: (a) add an explicit `record_review(memory_id, rating, prior, new)` method to the Phase 1 trait; (b) have `update_scheduling` write the event atomically.
- **RECOMMENDATION**: this is a Phase 1 question, not Phase 2. Phase 2 implements whichever Phase 1 chose. If Phase 1 missed it, Phase 2 raises a blocker rather than deciding alone.
### Q10. `tsvector` weight for tags -- PRD used `array_to_tsvector`, we used `array_to_string`
- **Options**: (a) `array_to_tsvector(tags)` (requires the `tsvector_extra` extension or similar); (b) `to_tsvector('english', array_to_string(tags, ' '))` (plain core Postgres).
- **RECOMMENDATION**: (b). Equivalent ranking, zero extra extensions. If a future tag matches a stopword (`"the"`), it gets dropped, but that is correct behavior for ranking.
- **Options**: (a) always run `sqlx::migrate!` on connect; (b) require the user to run `vestige migrate-schema` explicitly before starting the server.
- **RECOMMENDATION**: (a) during Phase 2; revisit in Phase 3 when the server binary exists. Developer ergonomics win now, and the migrations are idempotent.
### Q12. Offline query cache freshness vs `sqlx-cli` version skew
- **Options**: (a) pin `sqlx-cli` version in CI `actions/cache` step; (b) let CI install whatever version `sqlx` depends on.
- **RECOMMENDATION**: (a) pin to the same 0.8.x as the crate. `sqlx prepare` output changes between 0.7 and 0.8 and must match the runtime.
---
## Sequencing
The Phase 2 agent executes deliverables in this order; deliverables not listed can run in any order relative to each other.
Each deliverable PR includes its own tests; the final Phase 2 PR stacks them (or lands as a single branch if the Phase 1 trait is stable enough to avoid rebase churn).
