feat(storage): phase 1 -- extract MemoryStore and Embedder traits (ADR 0001)

Introduce two trait boundaries that the rest of the stack now sits above,
landing Phase 1 of ADR 0001 (pluggable storage and network access).
Rebased onto v2.1.22 Sanhedrin from the original April work.

MemoryStore / LocalMemoryStore (crates/vestige-core/src/storage/memory_store.rs):
  One trait, ~25 methods, covering CRUD, hybrid / FTS / vector search,
  FSRS scheduling, graph edges, and the forthcoming domain surface.
  trait_variant::make generates a Send-bound MemoryStore alias over the
  base LocalMemoryStore so Arc<dyn MemoryStore> works under tokio/axum.
  Storage errors map through a dedicated MemoryStoreError.

Embedder / LocalEmbedder (crates/vestige-core/src/embedder/):
  Pluggable text-to-vector encoder. FastembedEmbedder wraps the existing
  EmbeddingService; storage never calls fastembed directly anymore.
  Embedder::signature() produces the ModelSignature consumed by the
  store's embedding_model registry.

SqliteMemoryStore (crates/vestige-core/src/storage/sqlite.rs):
  Storage renamed to SqliteMemoryStore; the old name lives on as a
  pub type alias so Arc<Storage> consumers in vestige-mcp stay intact.
  All existing inherent methods are untouched; the trait impl is
  purely additive and dispatches into them. The db_path field added
  by v2.1.1 portable-sync is preserved.

Migration V14 (crates/vestige-core/src/storage/migrations.rs):
  Renumbered from V12 (the original April number) to V14 to slot in
  cleanly after upstream's V12 (v2.1.1 sync_tombstones) and V13
  (v2.1.2 purge tombstones).
  - embedding_model registry table (CHECK id = 1, code enforces the
    single-row invariant).
  - knowledge_nodes.domains / domain_scores TEXT columns (JSON arrays
    default '[]' / '{}'), domains catalogue table, supporting indexes.
  Phase 4 populates these columns; Phase 1 just exposes the schema.

Consolidation and other cognitive pathways now accept a
&dyn LocalMemoryStore (sync) or Arc<dyn MemoryStore> (async) rather
than a concrete Storage.

Tests:
  - trait-method unit tests colocated in sqlite.rs and migrations.rs
  - embedder/fastembed.rs tests for name/dimension/hash stability
  - new integration crate tests/phase_1 (added to workspace members):
    trait_round_trip (8), embedding_model_registry (7),
    domain_column_migration (5), cognitive_module_isolation (4),
    send_bound_variant (2), embedder_trait (2).

Acceptance gate post-rebase:
  - cargo build --workspace --all-targets: ok
  - cargo clippy --workspace --all-targets -- -D warnings: clean
  - cargo test -p vestige-core --lib: 428 pass
  - cargo test -p vestige-phase-1-tests: 28 pass
  - cargo test -p vestige-mcp --lib: 380 pass (Storage alias preserves
    every existing call site)

Co-existence with v2.1.1 portable-sync: this trait extraction is
additive. Portable-sync's tombstone migrations (V12, V13) remain
on the concrete SqliteMemoryStore; Phase 2 (Postgres) will decide
which of those surfaces graduate into the trait.
This commit is contained in:
Jan De Landtsheer 2026-04-21 21:43:52 +02:00 committed by Sam Valladares
parent 01fa882760
commit 5715f585fd
17 changed files with 3282 additions and 44 deletions

View file

@ -125,6 +125,9 @@ usearch = { version = "=2.23.0", optional = true }
# LRU cache for query embeddings
lru = "0.16"
trait-variant = "0.1"
blake3 = "1"
async-trait = "0.1"
[dev-dependencies]
tempfile = "3"

View file

@ -0,0 +1,182 @@
//! `FastembedEmbedder` -- adapts the existing `EmbeddingService` to the
//! `LocalEmbedder` trait.
#[cfg(feature = "embeddings")]
use crate::embeddings::{EMBEDDING_DIMENSIONS, EmbeddingService};
use super::{EmbedderError, EmbedderResult, LocalEmbedder};
pub struct FastembedEmbedder {
#[cfg(feature = "embeddings")]
inner: EmbeddingService,
cached_hash: std::sync::OnceLock<String>,
}
impl FastembedEmbedder {
pub fn new() -> Self {
Self {
#[cfg(feature = "embeddings")]
inner: EmbeddingService::new(),
cached_hash: std::sync::OnceLock::new(),
}
}
fn compute_hash(name: &str, dim: usize) -> String {
let mut hasher = blake3::Hasher::new();
hasher.update(name.as_bytes());
hasher.update(&(dim as u64).to_le_bytes());
// fastembed's ONNX bytes are not directly accessible at runtime; we
// use `(name, dim, vestige-core CARGO_PKG_VERSION)` as the
// signature. If fastembed ever changes its output deterministically
// between minor versions, bumping the crate version triggers a
// mismatch -- which is exactly the drift we want to detect.
hasher.update(env!("CARGO_PKG_VERSION").as_bytes());
hasher.finalize().to_hex().to_string()
}
}
impl Default for FastembedEmbedder {
fn default() -> Self {
Self::new()
}
}
#[async_trait::async_trait]
impl LocalEmbedder for FastembedEmbedder {
async fn embed(&self, text: &str) -> EmbedderResult<Vec<f32>> {
#[cfg(feature = "embeddings")]
{
let emb = self
.inner
.embed(text)
.map_err(|e| EmbedderError::EmbedFailed(e.to_string()))?;
Ok(emb.vector)
}
#[cfg(not(feature = "embeddings"))]
{
let _ = text;
Err(EmbedderError::Init(
"embeddings feature not enabled".to_string(),
))
}
}
fn model_name(&self) -> &str {
#[cfg(feature = "embeddings")]
{
self.inner.model_name()
}
#[cfg(not(feature = "embeddings"))]
{
"nomic-ai/nomic-embed-text-v1.5"
}
}
fn dimension(&self) -> usize {
#[cfg(feature = "embeddings")]
{
EMBEDDING_DIMENSIONS
}
#[cfg(not(feature = "embeddings"))]
{
256
}
}
fn model_hash(&self) -> String {
self.cached_hash
.get_or_init(|| Self::compute_hash(self.model_name(), self.dimension()))
.clone()
}
async fn embed_batch(&self, texts: &[&str]) -> EmbedderResult<Vec<Vec<f32>>> {
#[cfg(feature = "embeddings")]
{
let embs = self
.inner
.embed_batch(texts)
.map_err(|e| EmbedderError::EmbedFailed(e.to_string()))?;
Ok(embs.into_iter().map(|e| e.vector).collect())
}
#[cfg(not(feature = "embeddings"))]
{
let _ = texts;
Err(EmbedderError::Init(
"embeddings feature not enabled".to_string(),
))
}
}
}
// ============================================================================
// UNIT TESTS
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn embedder_reports_correct_name() {
let e = FastembedEmbedder::new();
assert!(
e.model_name().contains("nomic"),
"model name should contain 'nomic'"
);
}
#[test]
fn embedder_reports_256_dimension() {
let e = FastembedEmbedder::new();
assert_eq!(e.dimension(), 256);
}
#[test]
fn embedder_hash_is_stable() {
let e = FastembedEmbedder::new();
let h1 = e.model_hash();
let h2 = e.model_hash();
assert_eq!(h1, h2, "model_hash must be stable across calls");
}
#[test]
fn embedder_hash_includes_crate_version() {
// Compute what the hash should be given the known inputs
let name = FastembedEmbedder::new().model_name().to_string();
let dim = FastembedEmbedder::new().dimension();
let expected = FastembedEmbedder::compute_hash(&name, dim);
let got = FastembedEmbedder::new().model_hash();
assert_eq!(got, expected);
}
#[test]
fn embedder_signature_matches_accessors() {
let e = FastembedEmbedder::new();
let sig = e.signature();
assert_eq!(sig.name, e.model_name());
assert_eq!(sig.dimension, e.dimension());
assert_eq!(sig.hash, e.model_hash());
}
#[cfg(feature = "embeddings")]
#[test]
fn embedder_embed_smoke() {
let e = FastembedEmbedder::new();
let rt = tokio::runtime::Runtime::new().unwrap();
let vec = rt.block_on(e.embed("hello world")).expect("embed");
assert_eq!(vec.len(), 256);
}
#[cfg(feature = "embeddings")]
#[test]
fn embedder_embed_batch_matches_sequential() {
let e = FastembedEmbedder::new();
let rt = tokio::runtime::Runtime::new().unwrap();
let texts = ["alpha beta", "gamma delta"];
let batch = rt.block_on(e.embed_batch(texts.as_ref())).expect("batch");
let seq_a = rt.block_on(e.embed(texts[0])).expect("seq a");
let seq_b = rt.block_on(e.embed(texts[1])).expect("seq b");
assert_eq!(batch[0], seq_a);
assert_eq!(batch[1], seq_b);
}
}

View file

@ -0,0 +1,57 @@
//! Text-to-vector encoding trait. Pluggable per-install.
mod fastembed;
pub use fastembed::FastembedEmbedder;
/// Error returned by every `Embedder` method.
#[non_exhaustive]
#[derive(Debug, thiserror::Error)]
pub enum EmbedderError {
#[error("embedder initialization failed: {0}")]
Init(String),
#[error("embedding generation failed: {0}")]
EmbedFailed(String),
#[error("invalid input: {0}")]
InvalidInput(String),
}
pub type EmbedderResult<T> = std::result::Result<T, EmbedderError>;
/// Pluggable embedder. The storage layer NEVER calls fastembed directly;
/// callers compute vectors via this trait and pass them into `MemoryStore`.
///
/// `#[async_trait::async_trait]` makes every `async fn` return a
/// `Pin<Box<dyn Future + Send>>`, which is required for `Box<dyn Embedder>`
/// and `Arc<dyn Embedder>` to be dyn-compatible.
#[async_trait::async_trait]
pub trait LocalEmbedder: Send + Sync + 'static {
async fn embed(&self, text: &str) -> EmbedderResult<Vec<f32>>;
fn model_name(&self) -> &str;
fn dimension(&self) -> usize;
/// Stable blake3 hash of (model_name || dimension || vestige-core crate version).
/// Lowercase hex, 64 chars.
///
/// Used by `MemoryStore::register_model` to detect silent model drift
/// (e.g. a fastembed minor upgrade that changes vector output).
fn model_hash(&self) -> String;
async fn embed_batch(&self, texts: &[&str]) -> EmbedderResult<Vec<Vec<f32>>>;
/// Returns the `ModelSignature` describing this embedder. Convenience
/// wrapper over the three accessors above.
fn signature(&self) -> crate::storage::ModelSignature {
crate::storage::ModelSignature {
name: self.model_name().to_string(),
dimension: self.dimension(),
hash: self.model_hash(),
}
}
}
/// Type alias: `Embedder` is the dyn-compatible, Send+Sync variant.
/// Both names refer to the same `async_trait`-annotated trait.
pub use LocalEmbedder as Embedder;

View file

@ -83,6 +83,7 @@
/// Optional `vestige.toml` configuration (Phase 2: Configurable Output).
pub mod config;
pub mod consolidation;
pub mod embedder;
pub mod fsrs;
pub mod fts;
pub mod memory;
@ -159,13 +160,46 @@ pub use config::{CONFIG_FILE, OutputConfig, OutputDefaults, OutputProfile, Vesti
// Storage layer
pub use storage::{
CompositionEventRecord, CompositionMemberRecord, CompositionNeighborRecord,
CompositionOutcomeRecord, ConnectionRecord, ConsolidationHistoryRecord, DreamHistoryRecord,
InsightRecord, IntentionRecord, NeverComposedCandidate, PORTABLE_ARCHIVE_FORMAT,
PortableArchive, PortableImportMode, PortableImportReport, Result, SmartIngestResult,
StateTransitionRecord, Storage, StorageError,
ClassificationResult,
CompositionEventRecord,
CompositionMemberRecord,
CompositionNeighborRecord,
CompositionOutcomeRecord,
ConnectionRecord,
ConsolidationHistoryRecord,
Domain,
DreamHistoryRecord,
HealthStatus,
InsightRecord,
IntentionRecord,
LocalMemoryStore,
MemoryEdge,
MemoryRecord,
MemoryStore,
MemoryStoreError,
MemoryStoreResult,
ModelSignature,
NeverComposedCandidate,
PORTABLE_ARCHIVE_FORMAT,
PortableArchive,
PortableImportMode,
PortableImportReport,
Result,
SchedulingState,
SearchQuery,
SmartIngestResult,
SqliteMemoryStore,
StateTransitionRecord,
Storage,
StorageError,
StoreStats,
// Note: storage::SearchResult is intentionally not re-exported here to avoid
// collision with memory::SearchResult. Use vestige_core::storage::SearchResult directly.
};
// Embedder trait and implementations
pub use embedder::{Embedder, EmbedderError, EmbedderResult, FastembedEmbedder, LocalEmbedder};
// Consolidation (sleep-inspired memory processing)
pub use consolidation::SleepConsolidation;
pub use consolidation::{

View file

@ -0,0 +1,316 @@
//! Backend-agnostic memory store trait.
//!
//! This is the single abstraction every cognitive module sits above. It is
//! intentionally flat: one trait, ~25 methods, no sub-traits.
use std::collections::HashMap;
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use uuid::Uuid;
// ----------------------------------------------------------------------------
// ERROR
// ----------------------------------------------------------------------------
/// Error returned by every `LocalMemoryStore` / `MemoryStore` method.
#[non_exhaustive]
#[derive(Debug, thiserror::Error)]
pub enum MemoryStoreError {
#[error("not found: {0}")]
NotFound(String),
#[error("backend error: {0}")]
Backend(String),
#[error(
"embedding model mismatch: store registered {registered_name} (dim {registered_dim}, \
hash {registered_hash}), embedder is {actual_name} (dim {actual_dim}, hash {actual_hash})"
)]
ModelMismatch {
registered_name: String,
registered_dim: usize,
registered_hash: String,
actual_name: String,
actual_dim: usize,
actual_hash: String,
},
#[error("invalid input: {0}")]
InvalidInput(String),
#[error("initialization error: {0}")]
Init(String),
}
impl From<crate::storage::StorageError> for MemoryStoreError {
fn from(e: crate::storage::StorageError) -> Self {
use crate::storage::StorageError as S;
match e {
S::NotFound(s) => MemoryStoreError::NotFound(s),
S::Database(e) => MemoryStoreError::Backend(e.to_string()),
S::Io(e) => MemoryStoreError::Backend(e.to_string()),
S::InvalidTimestamp(s) => MemoryStoreError::Backend(format!("invalid timestamp: {s}")),
S::Init(s) => MemoryStoreError::Init(s),
}
}
}
pub type MemoryStoreResult<T> = std::result::Result<T, MemoryStoreError>;
// ----------------------------------------------------------------------------
// DATA TYPES
// ----------------------------------------------------------------------------
/// Backend-agnostic memory record.
///
/// Phase 1 intentionally keeps this type independent of `KnowledgeNode` to
/// avoid dragging 30+ legacy fields through the trait surface. The SQLite
/// backend converts between `MemoryRecord` and `KnowledgeNode` at the
/// boundary.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct MemoryRecord {
pub id: Uuid,
/// Empty = unclassified. Populated in Phase 4.
pub domains: Vec<String>,
/// Raw similarity per domain centroid. Empty until Phase 4 runs clustering.
pub domain_scores: HashMap<String, f64>,
pub content: String,
pub node_type: String,
pub tags: Vec<String>,
pub embedding: Option<Vec<f32>>,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
pub metadata: serde_json::Value,
}
/// FSRS-6 scheduling state, one row per memory.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SchedulingState {
pub memory_id: Uuid,
pub stability: f64,
pub difficulty: f64,
pub retrievability: f64,
pub last_review: Option<DateTime<Utc>>,
pub next_review: Option<DateTime<Utc>>,
pub reps: u32,
pub lapses: u32,
}
/// Hybrid search request.
#[derive(Debug, Clone, Default)]
pub struct SearchQuery {
pub domains: Option<Vec<String>>,
pub text: Option<String>,
pub embedding: Option<Vec<f32>>,
pub tags: Option<Vec<String>>,
pub node_types: Option<Vec<String>>,
pub limit: usize,
pub min_retrievability: Option<f64>,
}
#[derive(Debug, Clone)]
pub struct SearchResult {
pub record: MemoryRecord,
pub score: f64,
pub fts_score: Option<f64>,
pub vector_score: Option<f64>,
}
/// Edge in the spreading-activation graph.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct MemoryEdge {
pub source_id: Uuid,
pub target_id: Uuid,
pub edge_type: String,
pub weight: f64,
pub created_at: DateTime<Utc>,
}
/// A topical domain (populated in Phase 4). Phase 1 only needs the type to
/// shape the trait surface; discover/classify are Phase 4 work.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Domain {
pub id: String,
pub label: String,
pub centroid: Vec<f32>,
pub top_terms: Vec<String>,
pub memory_count: usize,
pub created_at: DateTime<Utc>,
}
/// Result of classifying one vector against all known domains.
#[derive(Debug, Clone)]
pub struct ClassificationResult {
pub scores: HashMap<String, f64>,
pub domains: Vec<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct StoreStats {
pub total_memories: usize,
pub memories_with_embeddings: usize,
pub total_edges: usize,
pub total_domains: usize,
pub registered_model_name: Option<String>,
pub registered_model_dim: Option<usize>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub enum HealthStatus {
Healthy,
Degraded { reason: String },
Unavailable { reason: String },
}
// ----------------------------------------------------------------------------
// EMBEDDING MODEL SIGNATURE
// ----------------------------------------------------------------------------
/// Snapshot of the embedding model that was used to write vectors into the
/// store. Persisted in the `embedding_model` table; compared on every write
/// before the vector is accepted.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct ModelSignature {
pub name: String,
pub dimension: usize,
/// Lowercase hex-encoded blake3 hash, 64 chars.
pub hash: String,
}
// ----------------------------------------------------------------------------
// TRAIT
// ----------------------------------------------------------------------------
/// The single storage abstraction.
///
/// `#[async_trait::async_trait]` makes every `async fn` return a
/// `Pin<Box<dyn Future + Send>>`, which is required for `Arc<dyn MemoryStore>`
/// to be movable across `tokio::spawn` boundaries.
///
/// `LocalMemoryStore` is a type alias kept for source compatibility with code
/// that refers to the non-send variant. In Phase 1 both names refer to the same
/// (dyn-compatible, Send-safe) trait.
#[async_trait::async_trait]
pub trait MemoryStore: Send + Sync + 'static {
// --- Lifecycle ---
async fn init(&self) -> MemoryStoreResult<()>;
async fn health_check(&self) -> MemoryStoreResult<HealthStatus>;
// --- Embedding model registry ---
async fn registered_model(&self) -> MemoryStoreResult<Option<ModelSignature>>;
async fn register_model(&self, sig: &ModelSignature) -> MemoryStoreResult<()>;
// --- CRUD ---
async fn insert(&self, record: &MemoryRecord) -> MemoryStoreResult<Uuid>;
async fn get(&self, id: Uuid) -> MemoryStoreResult<Option<MemoryRecord>>;
async fn update(&self, record: &MemoryRecord) -> MemoryStoreResult<()>;
async fn delete(&self, id: Uuid) -> MemoryStoreResult<()>;
// --- Search ---
async fn search(&self, query: &SearchQuery) -> MemoryStoreResult<Vec<SearchResult>>;
async fn fts_search(&self, text: &str, limit: usize) -> MemoryStoreResult<Vec<SearchResult>>;
async fn vector_search(
&self,
embedding: &[f32],
limit: usize,
) -> MemoryStoreResult<Vec<SearchResult>>;
// --- FSRS Scheduling ---
async fn get_scheduling(&self, memory_id: Uuid) -> MemoryStoreResult<Option<SchedulingState>>;
async fn update_scheduling(&self, state: &SchedulingState) -> MemoryStoreResult<()>;
async fn get_due_memories(
&self,
before: DateTime<Utc>,
limit: usize,
) -> MemoryStoreResult<Vec<(MemoryRecord, SchedulingState)>>;
// --- Graph (spreading activation) ---
async fn add_edge(&self, edge: &MemoryEdge) -> MemoryStoreResult<()>;
async fn get_edges(
&self,
node_id: Uuid,
edge_type: Option<&str>,
) -> MemoryStoreResult<Vec<MemoryEdge>>;
async fn remove_edge(&self, source: Uuid, target: Uuid) -> MemoryStoreResult<()>;
async fn get_neighbors(
&self,
node_id: Uuid,
depth: usize,
) -> MemoryStoreResult<Vec<(MemoryRecord, f64)>>;
// --- Domains (Phase 1: stubs return empty; full impl in Phase 4) ---
async fn list_domains(&self) -> MemoryStoreResult<Vec<Domain>>;
async fn get_domain(&self, id: &str) -> MemoryStoreResult<Option<Domain>>;
async fn upsert_domain(&self, domain: &Domain) -> MemoryStoreResult<()>;
async fn delete_domain(&self, id: &str) -> MemoryStoreResult<()>;
/// Phase 1: returns `Ok(vec![])` since no centroids exist. Phase 4 wires
/// the full soft-assignment pass.
async fn classify(&self, embedding: &[f32]) -> MemoryStoreResult<Vec<(String, f64)>>;
// --- Bulk / Maintenance ---
async fn count(&self) -> MemoryStoreResult<usize>;
async fn get_stats(&self) -> MemoryStoreResult<StoreStats>;
async fn vacuum(&self) -> MemoryStoreResult<()>;
}
/// Type alias kept for source compatibility. Both names refer to the same
/// `async_trait`-annotated trait that is dyn-compatible and `Send + Sync`.
pub use MemoryStore as LocalMemoryStore;
// ----------------------------------------------------------------------------
// UNIT TESTS
// ----------------------------------------------------------------------------
#[cfg(test)]
mod tests {
use super::*;
use crate::storage::StorageError;
#[test]
fn memory_store_error_from_storage_error() {
let se = StorageError::NotFound("abc".to_string());
let mse = MemoryStoreError::from(se);
assert!(matches!(mse, MemoryStoreError::NotFound(_)));
let se2 = StorageError::Init("init failure".to_string());
let mse2 = MemoryStoreError::from(se2);
assert!(matches!(mse2, MemoryStoreError::Init(_)));
}
#[test]
fn model_signature_serde_round_trip() {
let sig = ModelSignature {
name: "nomic-ai/nomic-embed-text-v1.5".to_string(),
dimension: 256,
hash: "a".repeat(64),
};
let json = serde_json::to_string(&sig).expect("serialize");
let sig2: ModelSignature = serde_json::from_str(&json).expect("deserialize");
assert_eq!(sig, sig2);
}
#[test]
fn memory_record_serde_round_trip() {
let rec = MemoryRecord {
id: Uuid::new_v4(),
domains: vec!["dev".to_string()],
domain_scores: {
let mut m = HashMap::new();
m.insert("dev".to_string(), 0.9);
m
},
content: "hello".to_string(),
node_type: "fact".to_string(),
tags: vec!["tag1".to_string()],
embedding: None,
created_at: Utc::now(),
updated_at: Utc::now(),
metadata: serde_json::json!({}),
};
let json = serde_json::to_string(&rec).expect("serialize");
let rec2: MemoryRecord = serde_json::from_str(&json).expect("deserialize");
assert_eq!(rec.content, rec2.content);
assert_eq!(rec.domains, rec2.domains);
}
}

View file

@ -79,6 +79,11 @@ pub const MIGRATIONS: &[Migration] = &[
description: "ComposedGraph: composition events, members, outcomes",
up: MIGRATION_V15_UP,
},
Migration {
version: 16,
description: "ADR 0001 Phase 1: embedding_model registry, domains/domain_scores columns, domains table",
up: MIGRATION_V16_UP,
},
];
/// A database migration
@ -904,6 +909,54 @@ fn add_column_if_missing(conn: &rusqlite::Connection, sql: &str) -> rusqlite::Re
}
}
/// V16: ADR 0001 Phase 1 - embedding_model registry + domain columns.
///
/// The ALTER TABLE statements are split out into `MIGRATION_V16_ALTER_COLUMNS`
/// because SQLite has no `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`. The
/// migration runner handles them individually so replaying V16 is idempotent.
const MIGRATION_V16_UP: &str = r#"
-- Migration V16: embedding model registry + per-memory domain columns.
-- 1. Embedding model registry. Single logical row; the (id = 1) constraint is
-- enforced in code via `register_model` (SQLite CHECK on a single-row
-- table is uglier than a constraint we already enforce in Rust).
CREATE TABLE IF NOT EXISTS embedding_model (
id INTEGER PRIMARY KEY CHECK (id = 1),
name TEXT NOT NULL,
dimension INTEGER NOT NULL,
hash TEXT NOT NULL,
created_at TEXT NOT NULL
);
-- 2. Per-memory domain columns are applied separately (see apply_migrations).
-- 3. Index on the domains JSON column to enable LIKE-style filter in Phase 4.
CREATE INDEX IF NOT EXISTS idx_nodes_domains ON knowledge_nodes(domains);
CREATE INDEX IF NOT EXISTS idx_nodes_domain_scores ON knowledge_nodes(domain_scores);
-- 4. Domains catalogue (empty until Phase 4 populates).
CREATE TABLE IF NOT EXISTS domains (
id TEXT PRIMARY KEY,
label TEXT NOT NULL,
centroid BLOB,
top_terms TEXT NOT NULL DEFAULT '[]',
memory_count INTEGER NOT NULL DEFAULT 0,
created_at TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_domains_created_at ON domains(created_at);
UPDATE schema_version SET version = 16, applied_at = datetime('now');
"#;
/// The two ALTER TABLE statements for V16. Kept separate so the migration
/// runner can try each individually and ignore "duplicate column" errors,
/// making V16 idempotent on replay (SQLite has no ADD COLUMN IF NOT EXISTS).
pub const MIGRATION_V16_ALTER_COLUMNS: &[&str] = &[
"ALTER TABLE knowledge_nodes ADD COLUMN domains TEXT NOT NULL DEFAULT '[]'",
"ALTER TABLE knowledge_nodes ADD COLUMN domain_scores TEXT NOT NULL DEFAULT '{}'",
];
/// Apply pending migrations
pub fn apply_migrations(conn: &rusqlite::Connection) -> rusqlite::Result<u32> {
let current_version = get_current_version(conn)?;
@ -932,6 +985,15 @@ pub fn apply_migrations(conn: &rusqlite::Connection) -> rusqlite::Result<u32> {
)?;
}
// V16 adds columns via ALTER TABLE, which SQLite does not support
// with IF NOT EXISTS. Run them individually and ignore duplicate
// column errors so replay stays idempotent.
if migration.version == 16 {
for stmt in MIGRATION_V16_ALTER_COLUMNS {
add_column_if_missing(conn, stmt)?;
}
}
// Use execute_batch to handle multi-statement SQL including triggers
conn.execute_batch(migration.up)?;
@ -958,17 +1020,17 @@ mod tests {
/// version after `apply_migrations` runs all migrations end-to-end, and
/// neither of the dead tables V11 drops must exist afterwards.
#[test]
fn test_apply_migrations_advances_to_v15_and_drops_dead_tables() {
fn test_apply_migrations_advances_to_v16_and_drops_dead_tables() {
let conn = rusqlite::Connection::open_in_memory().expect("open in-memory");
// Pre-requisite: schema_version must be bootstrapped by V1.
apply_migrations(&conn).expect("apply_migrations succeeds");
// 1. schema_version advanced to V15
// 1. schema_version advanced to V16
let version = get_current_version(&conn).expect("read schema_version");
assert_eq!(
version, 15,
"schema_version must be 15 after all migrations"
version, 16,
"schema_version must be 16 after all migrations"
);
// 2. knowledge_edges is gone (V11 drops it)
@ -1086,10 +1148,132 @@ mod tests {
conn.execute("UPDATE schema_version SET version = 10", [])
.expect("rewind schema_version");
// Replay must not error.
apply_migrations(&conn).expect("V11 replay must be idempotent");
// Replay V11 onward. V11 uses DROP TABLE IF EXISTS so it is idempotent.
// V12/V13 tombstone tables use CREATE TABLE IF NOT EXISTS. V14/V16 ALTER
// TABLE idempotency is handled by the migration runner.
apply_migrations(&conn).expect("V11..V16 replay must be idempotent");
// After replaying from V10, the schema advances to the latest version.
let version = get_current_version(&conn).expect("read schema_version");
assert_eq!(version, 15, "schema_version back at 15 after replay");
assert_eq!(version, 16, "schema_version back at 16 after replay");
}
#[test]
fn v16_adds_embedding_model_table() {
let conn = rusqlite::Connection::open_in_memory().expect("open in-memory");
apply_migrations(&conn).expect("apply_migrations");
let count: i64 = conn
.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='embedding_model'",
[],
|row| row.get(0),
)
.expect("query sqlite_master");
assert_eq!(count, 1, "embedding_model table must exist after V16");
}
#[test]
fn v16_adds_domains_columns() {
let conn = rusqlite::Connection::open_in_memory().expect("open in-memory");
apply_migrations(&conn).expect("apply_migrations");
let info: Vec<String> = {
let mut stmt = conn
.prepare("PRAGMA table_info(knowledge_nodes)")
.expect("prepare");
stmt.query_map([], |row| row.get::<_, String>(1))
.expect("query_map")
.map(|r| r.expect("row"))
.collect()
};
assert!(
info.contains(&"domains".to_string()),
"domains column missing"
);
assert!(
info.contains(&"domain_scores".to_string()),
"domain_scores column missing"
);
}
#[test]
fn v16_default_values_empty_json() {
let conn = rusqlite::Connection::open_in_memory().expect("open in-memory");
apply_migrations(&conn).expect("apply_migrations");
// Insert a minimal row to test defaults
conn.execute(
"INSERT INTO knowledge_nodes (id, content, node_type, created_at, updated_at, last_accessed, \
stability, difficulty, reps, lapses, learning_state, storage_strength, retrieval_strength, \
retention_strength, next_review, scheduled_days, has_embedding) \
VALUES ('test-id','content','fact',datetime('now'),datetime('now'),datetime('now'),\
1.0,0.3,0,0,'new',1.0,1.0,1.0,datetime('now'),1,0)",
[],
).expect("insert row");
let (domains, domain_scores): (String, String) = conn
.query_row(
"SELECT domains, domain_scores FROM knowledge_nodes WHERE id='test-id'",
[],
|row| Ok((row.get(0)?, row.get(1)?)),
)
.expect("query row");
assert_eq!(domains, "[]");
assert_eq!(domain_scores, "{}");
}
#[test]
fn v16_is_replayable() {
let conn = rusqlite::Connection::open_in_memory().expect("open in-memory");
apply_migrations(&conn).expect("first apply");
// Rewind to V15 so V16 runs again.
conn.execute("UPDATE schema_version SET version = 15", [])
.expect("rewind");
// V16 uses CREATE TABLE IF NOT EXISTS and idempotent ALTER handling.
apply_migrations(&conn).expect("V16 replay must be idempotent");
let version = get_current_version(&conn).expect("read version");
assert_eq!(version, 16, "schema_version must be 16 after replay");
}
#[test]
fn v16_preserves_existing_rows_from_v15() {
let conn = rusqlite::Connection::open_in_memory().expect("open in-memory");
// Apply up to V15 only, including the V14 ALTER TABLE columns that
// `apply_migrations` normally runs before the V14 SQL batch.
for migration in MIGRATIONS {
if migration.version <= 15 {
if migration.version == 14 {
add_column_if_missing(
&conn,
"ALTER TABLE knowledge_nodes ADD COLUMN protected INTEGER NOT NULL DEFAULT 0",
)
.expect("apply V14 protected column");
add_column_if_missing(
&conn,
"ALTER TABLE knowledge_nodes ADD COLUMN superseded_by TEXT",
)
.expect("apply V14 superseded_by column");
}
conn.execute_batch(migration.up).expect("apply migration");
}
}
// Insert a row under the V15 schema, before PR #61's V16 columns exist.
conn.execute(
"INSERT INTO knowledge_nodes (id, content, node_type, created_at, updated_at, last_accessed, \
stability, difficulty, reps, lapses, learning_state, storage_strength, retrieval_strength, \
retention_strength, next_review, scheduled_days, has_embedding) \
VALUES ('existing-id','old content','fact',datetime('now'),datetime('now'),datetime('now'),\
1.0,0.3,0,0,'new',1.0,1.0,1.0,datetime('now'),1,0)",
[],
).expect("insert pre-v16 row");
apply_migrations(&conn).expect("apply V16 migration");
// Check the old row has defaults
let (domains, domain_scores): (String, String) = conn
.query_row(
"SELECT domains, domain_scores FROM knowledge_nodes WHERE id='existing-id'",
[],
|row| Ok((row.get(0)?, row.get(1)?)),
)
.expect("query pre-v16 row");
assert_eq!(domains, "[]");
assert_eq!(domain_scores, "{}");
}
}

View file

@ -1,15 +1,17 @@
//! Storage Module
//!
//! SQLite-based storage layer with:
//! - FTS5 full-text search with query sanitization
//! - Embedded vector storage
//! - FSRS-6 state management
//! - Temporal memory support
//! Backend-agnostic memory store abstraction plus SQLite reference impl.
mod memory_store;
mod migrations;
mod portable;
mod sqlite;
pub use memory_store::{
ClassificationResult, Domain, HealthStatus, LocalMemoryStore, MemoryEdge, MemoryRecord,
MemoryStore, MemoryStoreError, MemoryStoreResult, ModelSignature, SchedulingState, SearchQuery,
SearchResult, StoreStats,
};
pub use migrations::MIGRATIONS;
pub use portable::{
PORTABLE_ARCHIVE_FORMAT, PortableArchive, PortableImportMode, PortableImportReport,
@ -19,6 +21,11 @@ pub use sqlite::{
CompositionEventRecord, CompositionMemberRecord, CompositionNeighborRecord,
CompositionOutcomeRecord, ConnectionRecord, ConsolidationHistoryRecord, DreamHistoryRecord,
FilePortableSyncBackend, InsightRecord, IntentionRecord, NeverComposedCandidate,
PortableSyncBackend, PortableSyncReport, Result, SmartIngestResult, StateTransitionRecord,
Storage, StorageError,
PortableSyncBackend, PortableSyncReport, Result, SmartIngestResult, SqliteMemoryStore,
StateTransitionRecord, StorageError,
};
/// Backwards-compatibility alias. Retained until Phase 4 completes so every
/// existing `Arc<Storage>` call site keeps compiling. Scheduled for removal
/// once no downstream source file references it.
pub type Storage = SqliteMemoryStore;

File diff suppressed because it is too large Load diff