diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d06f1a..413a4eb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,9 +5,34 @@ All notable changes to Vestige will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [2.1.1] - 2026-05-01 — "Portable Sync" + +v2.1.1 focuses on user-controlled portability: exact storage archives, merge-safe file sync, pluggable sync backends, and explicit hook opt-ins. + +### Added + +- **Exact portable archives** — `vestige portable-export` / `vestige portable-import` preserve raw Vestige storage rows: memory IDs, FSRS state, graph edges, suppression state, audit rows, sessions, intentions, and embedding blobs. +- **Merge-safe imports** — `vestige portable-import --merge` can merge into non-empty databases. It applies `sync_tombstones`, keeps newer local memory rows on timestamp conflicts, preserves stable IDs, and rebuilds FTS after import. +- **File-backed two-way sync** — `vestige sync ` performs pull-merge-push through a shared portable archive. This works today with Dropbox, iCloud Drive, Syncthing, Git, network shares, and shared folders. +- **Pluggable portable-sync backend trait** — core now exposes `PortableSyncBackend`, `FilePortableSyncBackend`, and `PortableSyncReport`, so non-file backends can reuse the same merge semantics without reimplementing conflict handling. +- **Portable restore merge mode** — the MCP `restore` tool accepts `merge: true` for portable archives and returns inserted/updated/deleted/skipped/conflict counts. +- **Qwen3 embedding opt-in** — build-time and runtime support for Qwen3 embeddings, with model-aware retrieval safeguards so mixed embedding models are not compared in the same vector path. + +### Fixed + +- **Sanhedrin, preflight, and all Vestige Claude Code hooks are optional again.** The Cognitive Sandwich installer now activates no hooks by default and leaves every preflight hook, every Stop hook, the MLX launchd service, and the 19 GB Qwen model path behind explicit `--enable-preflight`, `--enable-sanhedrin`, or `--with-launchd` flags. +- **x86-friendly Sanhedrin path.** The verifier bridge now accepts any OpenAI-compatible chat endpoint via `VESTIGE_SANHEDRIN_ENDPOINT` and `VESTIGE_SANHEDRIN_MODEL`, so Linux and Intel Mac users can opt in without MLX or Apple Silicon. + +### Verified + +- `cargo test -p vestige-core portable --no-fail-fast` +- `cargo test -p vestige-mcp portable --no-fail-fast` +- `cargo test --workspace --no-fail-fast` +- Installer shell/Python/JSON validation and default/preflight/Sanhedrin migration dry-runs. + ## [2.1.0] - 2026-04-27 — "Cognitive Sandwich Goes Local" -The Sanhedrin Executioner — Vestige's veto layer for Claude Code responses — now runs entirely on a local MLX model (`mlx-community/Qwen3.6-35B-A3B-4bit`). Zero API cost per Claude turn, fully offline, no Anthropic round-trip on the critical path. Combined with four pre-cognitive UserPromptSubmit hooks (synthesis-preflight, cwd-state-injector, vestige-pulse-daemon, preflight-swarm), Vestige now ships a complete "Cognitive Sandwich" — Vestige memories injected before the model thinks, local Sanhedrin veto after the model speaks — installable in one command on a MacBook. +v2.1.0 ships the Cognitive Sandwich hook harness for Claude Code, with a hotfix that makes every hook layer opt-in. The default installer stages files, removes old Vestige hook wiring, starts no MLX service, downloads no model, and makes no automatic model calls. Users can opt into preflight context, Sanhedrin verification, or the Apple Silicon MLX backend separately. ### Added @@ -21,18 +46,18 @@ The Sanhedrin Executioner — Vestige's veto layer for Claude Code responses — - `synthesis-stop-validator.sh` — Stop hook regex against forbidden hedging patterns. - `veto-detector.sh` — fast 50ms regex pre-screen against `veto`-tagged Vestige memories. - `synthesis-gate.sh` — legacy v1 trigger (kept for backward compat). - - `settings.fragment.json` — JSON snippet merged into `~/.claude/settings.json` by the installer. + - `settings.fragment.json` — empty default JSON snippet; the installer only wires hooks from the opt-in preflight/Sanhedrin fragments. - **Dashboard `/api/changelog` endpoint** — bounded REST event feed for recent `DreamCompleted` and `ConnectionDiscovered` events, used by the Pulse hook to inject fresh synthesis into Claude Code context. - **`agents/`** — `executioner.md` (legacy/fallback Haiku 4.5 path), `lateral-thinker.md`, `synthesis-composer.md`. - **`launchd/com.vestige.mlx-server.plist.template`** — auto-start `mlx_lm.server` with the Qwen3.6-35B-A3B-4bit model on login. Templated with `__HOME__` and `__MODEL__` placeholders. -- **`scripts/install-sandwich.sh`** — one-command installer that stages hooks, agents, plist, jq-merges the settings fragment, and `launchctl load`s the plist. Backs up `settings.json` to `.bak.pre-sandwich`. Supports `--force`, `--no-launchd`, `--include-memory-loader`, `--src=PATH`. -- **`scripts/check-sandwich-prereqs.sh`** — comprehensive prereq verifier (Apple Silicon, Python 3.10+, jq, uv, mlx-lm, hf, claude, vestige-mcp, model on disk, MCP HTTP up, server up, plist installed, settings wired). +- **`scripts/install-sandwich.sh`** — one-command installer that stages hooks and agents, removes old Vestige hook wiring by default, and only wires optional layers with `--enable-preflight`, `--enable-sanhedrin`, or `--with-launchd`. Backs up `settings.json` to `.bak.pre-sandwich`. Supports `--force`, `--include-memory-loader`, and `--src=PATH`. +- **`scripts/check-sandwich-prereqs.sh`** — default verifier confirms no Vestige Claude Code hooks are wired. Optional `--preflight` and `--sanhedrin` modes check the corresponding enabled layer. - **`docs/COGNITIVE_SANDWICH.md`** — architecture diagram, install guide, performance notes (82 tok/s on M3 Max), uninstall, configuration env vars. - **PR #48** — `VESTIGE_DATA_DIR` env-var support + tilde expansion + secure unix perms (thanks @Jelloeater) — directly addresses the ghost env-vars exposed by v2.0.9 cleanup. ### Changed -- **Sanhedrin Executioner default backend swapped from Anthropic Haiku 4.5 → local `mlx_lm.server` + Qwen3.6-35B-A3B-4bit.** Anthropic API key no longer required for the post-cognitive layer. The `executioner.md` agent definition is retained as manual/fallback only when invoked explicitly via `Task(subagent_type='executioner')`. +- **Sanhedrin Executioner backend can run locally or remotely when explicitly enabled.** The bridge targets an OpenAI-compatible chat endpoint, with local `mlx_lm.server` + Qwen3.6-35B-A3B-4bit available behind `--with-launchd` on Apple Silicon. Anthropic API key no longer required for the post-cognitive layer. The `executioner.md` agent definition is retained as manual/fallback only when invoked explicitly via `Task(subagent_type='executioner')`. - **All hooks sanitized for public release** — replaced hardcoded personal absolute paths with `$HOME` / `$VESTIGE_*` env vars; removed personal regex tokens. - **NPM binary installer now follows package version** — `vestige-mcp-server@2.1.0` downloads release assets from `v2.1.0` instead of a stale hardcoded binary tag, while local workspace installs skip the release-asset download before the tag exists. @@ -40,22 +65,23 @@ The Sanhedrin Executioner — Vestige's veto layer for Claude Code responses — - `cargo test --workspace --release --no-fail-fast`: **1,229 passing, 0 failed** (366 vestige-core + 358 vestige-mcp lib + 4 vestige-mcp bin + 497 e2e + 4 doctests). - Sanhedrin bridge smoke checks: Python bytecode compilation passes, fail-open bridge invocation returns `yes`, and public hook settings validate as JSON. +- v2.1.0 hotfix installer matrix: default, preflight-only, Sanhedrin-only, full sandwich, legacy-all-hooks migration, and unrelated custom hooks preservation. - 8-day Sandwich dogfood: **84% pass rate, 16% legitimate vetoes** caught real hallucinations. ### Closes - #36 (Agent Hooks for Low-Effort Automatic Memory Capture) — Cognitive Sandwich is the answer. -### Prerequisites for the Cognitive Sandwich +### Prerequisites for optional local MLX Sanhedrin -- macOS Apple Silicon (M1+) — required for MLX +- macOS Apple Silicon (M1+) — required only for `--with-launchd` MLX autostart - Python 3.10+ - ~22 GB free RAM (Qwen3.6-35B-A3B-4bit at runtime) - First-run model download: ~19 GB from Hugging Face (cached locally thereafter) ### Migration -None required for existing Vestige users. The Cognitive Sandwich is opt-in via `scripts/install-sandwich.sh`. The MCP server, schema, and tool surface are bit-identical to v2.0.9. +None required for existing Vestige users. The Cognitive Sandwich is opt-in via `scripts/install-sandwich.sh`; running the default installer removes old Vestige hook wiring and leaves preflight, Sanhedrin, launchd, and the 19 GB model path disabled. The MCP server, schema, and tool surface are bit-identical to v2.0.9. --- diff --git a/Cargo.lock b/Cargo.lock index 2e15454..015f3ae 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4531,8 +4531,9 @@ checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" [[package]] name = "vestige-core" -version = "2.1.0" +version = "2.1.1" dependencies = [ + "candle-core", "chrono", "criterion", "directories", @@ -4566,7 +4567,7 @@ dependencies = [ [[package]] name = "vestige-mcp" -version = "2.1.0" +version = "2.1.1" dependencies = [ "anyhow", "axum", diff --git a/Cargo.toml b/Cargo.toml index 40cd33f..3ca3766 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -10,7 +10,7 @@ exclude = [ ] [workspace.package] -version = "2.1.0" +version = "2.1.1" edition = "2024" license = "AGPL-3.0-only" repository = "https://github.com/samvallad33/vestige" diff --git a/README.md b/README.md index 57b2f0d..ff34566 100644 --- a/README.md +++ b/README.md @@ -20,12 +20,21 @@ Built on 130 years of memory research — FSRS-6 spaced repetition, prediction e --- +## What's New in v2.1.1 "Portable Sync" + +v2.1.1 focuses on the biggest post-launch ask: move memories between machines without losing cognitive state. It also adds opt-in Qwen3 embeddings for higher-recall local retrieval. + +- **Exact portable archives.** `vestige portable-export` / `vestige portable-import` preserve IDs, FSRS state, graph edges, suppression state, audit rows, and embedding blobs for Vestige-to-Vestige device transfer. +- **Sync-safe merge storage.** `vestige portable-import --merge` and `vestige sync ` merge non-empty databases, apply delete tombstones, keep newer local memories, rebuild FTS, and push through a pluggable portable-sync backend. v2.1.1 ships the file backend for Dropbox, iCloud, Syncthing, Git, and shared folders. +- **Qwen3 embeddings.** Build with `qwen3-embeddings`, set `VESTIGE_EMBEDDING_MODEL=qwen3-0.6b`, and run `vestige consolidate` to re-embed existing memories. +- **Model-aware retrieval.** Vestige now avoids comparing Qwen and Nomic vectors in the same search/dedup path. + ## What's New in v2.1.0 "Cognitive Sandwich Goes Local" -v2.1.0 adds an opt-in Claude Code hook harness around the existing Vestige MCP server. The MCP tool surface and database schema stay backward compatible, while the new local Sanhedrin verifier and preflight hooks can inject trusted memory context before Claude answers and check drafts against high-trust Vestige evidence before delivery. +v2.1.0 adds an opt-in Claude Code hook harness around the existing Vestige MCP server. The MCP tool surface and database schema stay backward compatible, while preflight hooks can inject trusted memory context before Claude answers. The heavyweight Sanhedrin verifier is optional and can be enabled separately. -- **Local Sanhedrin Executioner.** The post-response verifier now runs through `mlx_lm.server` with `mlx-community/Qwen3.6-35B-A3B-4bit` by default, so the veto layer can run offline on Apple Silicon without Anthropic API calls. -- **One-command Cognitive Sandwich installer.** `scripts/install-sandwich.sh` stages hooks, agents, and a launchd plist, merges the Claude Code hooks block, and prints real verification commands. +- **Optional Sanhedrin Executioner.** The post-response verifier is off by default. Users can enable it with an OpenAI-compatible endpoint on x86/Linux/Intel Mac, or add `--with-launchd` on Apple Silicon to run the local MLX Qwen backend. +- **One-command Cognitive Sandwich installer.** `scripts/install-sandwich.sh` stages hook files and agents by default, removes old Vestige hook wiring, and leaves all Claude Code hook layers plus the 19 GB model path opt-in. - **Pulse hook backed by `/api/changelog`.** Fresh dream and connection events can be injected into the next Claude Code prompt context without blocking the prompt. - **`VESTIGE_DATA_DIR` support.** `--data-dir` now has an env-var fallback, tilde expansion, secure directory creation, and clear precedence docs. - **NPM release wrapper fixed.** `vestige-mcp-server@2.1.0` now downloads binaries from the matching `v2.1.0` GitHub release tag instead of an old hardcoded release. @@ -211,7 +220,7 @@ Vestige v2.0 ships with a real-time 3D visualization of your AI's memory. Every **Tech:** SvelteKit 2 + Svelte 5 + Three.js + Tailwind CSS 4 + WebSocket -The dashboard runs automatically at `http://localhost:3927/dashboard` when the MCP server starts. +Run `vestige dashboard` to open `http://localhost:3927/dashboard`, or set `VESTIGE_DASHBOARD_ENABLED=true` to start it with the MCP server. --- @@ -335,8 +344,8 @@ This isn't a key-value store with an embedding model bolted on. Vestige implemen | `consolidate` | Run FSRS-6 decay cycle (also auto-runs every 6 hours) | | `memory_timeline` | Browse chronologically, grouped by day | | `memory_changelog` | Audit trail of state transitions | -| `backup` / `export` / `gc` | Database backup, JSON export, garbage collection | -| `restore` | Restore from JSON backup | +| `backup` / `export` / `gc` | Database backup, JSON/JSONL/portable export, garbage collection | +| `restore` | Restore from JSON backup or portable archive | ### Deep Reference (v2.0.4) | Tool | What It Does | @@ -382,7 +391,7 @@ At the start of every session: | **Language** | Rust 2024 edition (MSRV 1.91) | | **Codebase** | 80,000+ lines, 1,292 tests (366 core + 425 mcp + 497 e2e + 4 doctests) | | **Binary size** | ~20MB | -| **Embeddings** | Nomic Embed Text v1.5 (768d → 256d Matryoshka, 8192 context) | +| **Embeddings** | Nomic Embed Text v1.5 by default (768d -> 256d Matryoshka, 8192 context); Qwen3 0.6B optional | | **Vector search** | USearch HNSW (20x faster than FAISS) | | **Reranker** | Jina Reranker v1 Turbo (38M params, +15-20% precision) | | **Storage** | SQLite + FTS5 (optional SQLCipher encryption) | @@ -395,17 +404,9 @@ At the start of every session: ### Optional Features ```bash -# Metal GPU acceleration (Apple Silicon — faster embedding inference) -cargo build --release -p vestige-mcp --features metal - -# Nomic Embed Text v2 MoE (475M params, 305M active, 8 experts) -cargo build --release -p vestige-mcp --features nomic-v2 - -# Qwen3 Reranker (Candle backend, high-precision cross-encoder) -cargo build --release -p vestige-mcp --features qwen3-reranker - -# SQLCipher encryption -cargo build --release -p vestige-mcp --no-default-features --features encryption,embeddings,vector-search +# Qwen3 embeddings (Candle backend; add metal on Apple Silicon) +cargo build --release -p vestige-mcp --features qwen3-embeddings,metal +VESTIGE_EMBEDDING_MODEL=qwen3-0.6b vestige consolidate ``` --- @@ -419,6 +420,10 @@ vestige stats --states # Cognitive state breakdown vestige health # System health check vestige consolidate # Run memory maintenance vestige restore # Restore from backup +vestige portable-export # Exact cross-device archive +vestige portable-import # Import archive into an empty database +vestige portable-import --merge # Merge archive into this database +vestige sync # Pull/merge/push via file backend vestige dashboard # Open 3D dashboard in browser ``` @@ -465,7 +470,7 @@ Cache: macOS `~/Library/Caches/com.vestige.core/fastembed` | Linux `~/.cache/ves
Dashboard not loading -The dashboard starts automatically on port 3927 when the MCP server runs. Check: +Run `vestige dashboard` or set `VESTIGE_DASHBOARD_ENABLED=true`, then check: ```bash curl http://localhost:3927/api/health # Should return {"status":"healthy",...} diff --git a/crates/vestige-core/Cargo.toml b/crates/vestige-core/Cargo.toml index 0171472..6aee2ba 100644 --- a/crates/vestige-core/Cargo.toml +++ b/crates/vestige-core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "vestige-core" -version = "2.1.0" +version = "2.1.1" edition = "2024" rust-version = "1.91" authors = ["Vestige Team"] @@ -51,8 +51,11 @@ ort-dynamic = ["embeddings", "fastembed/ort-load-dynamic"] # Requires: fastembed with nomic-v2-moe feature nomic-v2 = ["embeddings", "fastembed/nomic-v2-moe"] -# Qwen3 Reranker (Candle backend, high-precision cross-encoder) -qwen3-reranker = ["embeddings", "fastembed/qwen3"] +# Qwen3 Embeddings (Candle backend, opt-in for v2.1.1 re-embedding) +qwen3-embeddings = ["embeddings", "fastembed/qwen3", "dep:candle-core"] + +# Backwards-compatible feature alias from the original v2.1.0 naming. +qwen3-reranker = ["qwen3-embeddings"] # Metal GPU acceleration on Apple Silicon (significantly faster inference) metal = ["fastembed/metal"] @@ -96,8 +99,9 @@ notify = "8" # OPTIONAL: Embeddings (fastembed v5 - local ONNX inference, 2026 bleeding edge) # ============================================================================ # nomic-embed-text-v1.5: 768 dimensions, 8192 token context, Matryoshka support -# v5.11: Adds Nomic v2 MoE (nomic-v2-moe feature) + Qwen3 reranker (qwen3 feature) +# fastembed v5 provides Nomic v2 MoE and Qwen3 feature-gated model loaders. fastembed = { version = "5.11", default-features = false, features = ["hf-hub-native-tls", "image-models"], optional = true } +candle-core = { version = "0.10.2", optional = true } # ============================================================================ # OPTIONAL: Vector Search (USearch - HNSW, 20x faster than FAISS) diff --git a/crates/vestige-core/src/embeddings/local.rs b/crates/vestige-core/src/embeddings/local.rs index 3dfc363..a6ec555 100644 --- a/crates/vestige-core/src/embeddings/local.rs +++ b/crates/vestige-core/src/embeddings/local.rs @@ -7,7 +7,13 @@ //! - **Default**: Nomic Embed Text v1.5 (ONNX, 768d → 256d Matryoshka, 8192 context) //! - **Optional**: Nomic Embed Text v2 MoE (Candle, 475M params, 305M active, 8 experts) //! Enable with `nomic-v2` feature flag + `metal` for Apple Silicon acceleration. +//! - **Optional**: Qwen3 Embedding 0.6B (Candle, 1024d → 256d Matryoshka) +//! Enable with `qwen3-embeddings` and `VESTIGE_EMBEDDING_MODEL=qwen3-0.6b`. +#[cfg(feature = "qwen3-embeddings")] +use candle_core::{DType, Device}; +#[cfg(feature = "qwen3-embeddings")] +use fastembed::Qwen3TextEmbedding; use fastembed::{EmbeddingModel, InitOptions, TextEmbedding}; use std::sync::{Mutex, OnceLock}; @@ -31,7 +37,82 @@ pub const BATCH_SIZE: usize = 32; // ============================================================================ /// Result type for model initialization -static EMBEDDING_MODEL_RESULT: OnceLock, String>> = OnceLock::new(); +static EMBEDDING_BACKEND_RESULT: OnceLock, String>> = + OnceLock::new(); + +const NOMIC_V15_MODEL_ID: &str = "nomic-ai/nomic-embed-text-v1.5"; +#[cfg(feature = "qwen3-embeddings")] +const QWEN3_06B_MODEL_ID: &str = "Qwen/Qwen3-Embedding-0.6B"; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum EmbeddingModelSpec { + NomicV15, + #[cfg(feature = "qwen3-embeddings")] + Qwen3Embedding06B, +} + +impl EmbeddingModelSpec { + fn selected() -> Result { + let requested = std::env::var("VESTIGE_EMBEDDING_MODEL") + .ok() + .map(|value| value.trim().to_ascii_lowercase()) + .filter(|value| !value.is_empty()) + .unwrap_or_else(|| "nomic-v1.5".to_string()); + + match requested.as_str() { + "nomic" | "nomic-v1.5" | "nomic-embed-text-v1.5" | NOMIC_V15_MODEL_ID => { + Ok(Self::NomicV15) + } + "qwen3" | "qwen3-0.6b" | "qwen3-embedding-0.6b" | "qwen/qwen3-embedding-0.6b" => { + #[cfg(feature = "qwen3-embeddings")] + { + Ok(Self::Qwen3Embedding06B) + } + #[cfg(not(feature = "qwen3-embeddings"))] + { + Err( + "VESTIGE_EMBEDDING_MODEL requests Qwen3, but vestige-core was not built with the qwen3-embeddings feature" + .to_string(), + ) + } + } + other => Err(format!( + "Unsupported VESTIGE_EMBEDDING_MODEL '{}'. Expected 'nomic-v1.5' or 'qwen3-0.6b'.", + other + )), + } + } + + fn model_name(self) -> &'static str { + match self { + Self::NomicV15 => NOMIC_V15_MODEL_ID, + #[cfg(feature = "qwen3-embeddings")] + Self::Qwen3Embedding06B => QWEN3_06B_MODEL_ID, + } + } +} + +enum EmbeddingBackend { + NomicV15(TextEmbedding), + #[cfg(feature = "qwen3-embeddings")] + Qwen3Embedding06B(Qwen3TextEmbedding), +} + +impl EmbeddingBackend { + fn model_name(&self) -> &'static str { + match self { + Self::NomicV15(_) => NOMIC_V15_MODEL_ID, + #[cfg(feature = "qwen3-embeddings")] + Self::Qwen3Embedding06B(_) => QWEN3_06B_MODEL_ID, + } + } +} + +fn qwen3_format_query(query: &str) -> String { + format!( + "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: {query}" + ) +} /// Get the default cache directory for fastembed models. /// @@ -65,10 +146,10 @@ pub(crate) fn get_cache_dir() -> std::path::PathBuf { std::path::PathBuf::from(".fastembed_cache") } -/// Initialize the global embedding model -/// Using nomic-embed-text-v1.5 (768d) - 8192 token context, Matryoshka support -fn get_model() -> Result, EmbeddingError> { - let result = EMBEDDING_MODEL_RESULT.get_or_init(|| { +/// Initialize the global embedding backend selected by `VESTIGE_EMBEDDING_MODEL`. +fn get_backend() -> Result, EmbeddingError> { + let result = EMBEDDING_BACKEND_RESULT.get_or_init(|| { + let spec = EmbeddingModelSpec::selected()?; // Get cache directory (respects FASTEMBED_CACHE_PATH env var) let cache_dir = get_cache_dir(); @@ -77,31 +158,66 @@ fn get_model() -> Result, Embeddin tracing::warn!("Failed to create cache directory {:?}: {}", cache_dir, e); } - // nomic-embed-text-v1.5: 768 dimensions, 8192 token context - // Matryoshka representation learning, fully open source - let options = InitOptions::new(EmbeddingModel::NomicEmbedTextV15) - .with_show_download_progress(true) - .with_cache_dir(cache_dir); + match spec { + EmbeddingModelSpec::NomicV15 => { + let options = InitOptions::new(EmbeddingModel::NomicEmbedTextV15) + .with_show_download_progress(true) + .with_cache_dir(cache_dir); - TextEmbedding::try_new(options) - .map(Mutex::new) - .map_err(|e| { - format!( - "Failed to initialize nomic-embed-text-v1.5 embedding model: {}. \ - Ensure ONNX runtime is available and model files can be downloaded.", - e + TextEmbedding::try_new(options) + .map(EmbeddingBackend::NomicV15) + .map(Mutex::new) + .map_err(|e| { + format!( + "Failed to initialize {} embedding model: {}. \ + Ensure ONNX runtime is available and model files can be downloaded.", + spec.model_name(), + e + ) + }) + } + #[cfg(feature = "qwen3-embeddings")] + EmbeddingModelSpec::Qwen3Embedding06B => { + let device = qwen3_device(); + Qwen3TextEmbedding::from_hf( + QWEN3_06B_MODEL_ID, + &device, + DType::F32, + MAX_TEXT_LENGTH, ) - }) + .map(EmbeddingBackend::Qwen3Embedding06B) + .map(Mutex::new) + .map_err(|e| { + format!( + "Failed to initialize {} embedding model: {}. \ + Ensure Hugging Face model files can be downloaded.", + spec.model_name(), + e + ) + }) + } + } }); match result { - Ok(model) => model + Ok(backend) => backend .lock() .map_err(|e| EmbeddingError::ModelInit(format!("Lock poisoned: {}", e))), Err(err) => Err(EmbeddingError::ModelInit(err.clone())), } } +#[cfg(feature = "qwen3-embeddings")] +fn qwen3_device() -> Device { + #[cfg(feature = "metal")] + { + if let Ok(device) = Device::new_metal(0) { + return device; + } + } + Device::Cpu +} + // ============================================================================ // ERROR TYPES // ============================================================================ @@ -223,7 +339,7 @@ impl EmbeddingService { /// Check if the model is ready pub fn is_ready(&self) -> bool { - match get_model() { + match get_backend() { Ok(_) => true, Err(e) => { tracing::warn!("Embedding model not ready: {}", e); @@ -234,25 +350,26 @@ impl EmbeddingService { /// Check if the model is ready and return the error if not pub fn check_ready(&self) -> Result<(), EmbeddingError> { - get_model().map(|_| ()) + get_backend().map(|_| ()) } /// Initialize the model (downloads if necessary) pub fn init(&self) -> Result<(), EmbeddingError> { - let _model = get_model()?; // Ensures model is loaded and returns any init errors + let _model = get_backend()?; // Ensures model is loaded and returns any init errors Ok(()) } /// Get the model name pub fn model_name(&self) -> &'static str { - #[cfg(feature = "nomic-v2")] + if let Some(Ok(backend)) = EMBEDDING_BACKEND_RESULT.get() + && let Ok(backend) = backend.lock() { - "nomic-ai/nomic-embed-text-v2-moe" - } - #[cfg(not(feature = "nomic-v2"))] - { - "nomic-ai/nomic-embed-text-v1.5" + return backend.model_name(); } + + EmbeddingModelSpec::selected() + .unwrap_or(EmbeddingModelSpec::NomicV15) + .model_name() } /// Get the embedding dimensions @@ -268,7 +385,7 @@ impl EmbeddingService { )); } - let mut model = get_model()?; + let mut backend = get_backend()?; // Truncate if too long (char-boundary safe) let text = if text.len() > MAX_TEXT_LENGTH { @@ -281,9 +398,15 @@ impl EmbeddingService { text }; - let embeddings = model - .embed(vec![text], None) - .map_err(|e| EmbeddingError::EmbeddingFailed(e.to_string()))?; + let embeddings = match &mut *backend { + EmbeddingBackend::NomicV15(model) => model + .embed(vec![text], None) + .map_err(|e| EmbeddingError::EmbeddingFailed(e.to_string()))?, + #[cfg(feature = "qwen3-embeddings")] + EmbeddingBackend::Qwen3Embedding06B(model) => model + .embed(&[text]) + .map_err(|e| EmbeddingError::EmbeddingFailed(e.to_string()))?, + }; if embeddings.is_empty() { return Err(EmbeddingError::EmbeddingFailed( @@ -294,13 +417,26 @@ impl EmbeddingService { Ok(Embedding::new(matryoshka_truncate(embeddings[0].clone()))) } + /// Generate an embedding for retrieval queries. + /// + /// Qwen3 uses instruction-formatted queries against raw document embeddings; + /// Nomic remains symmetric and receives the query unchanged. + pub fn embed_query(&self, query: &str) -> Result { + if self.model_name().to_ascii_lowercase().contains("qwen3") { + let formatted = qwen3_format_query(query); + self.embed(&formatted) + } else { + self.embed(query) + } + } + /// Generate embeddings for multiple texts (batch processing) pub fn embed_batch(&self, texts: &[&str]) -> Result, EmbeddingError> { if texts.is_empty() { return Ok(vec![]); } - let mut model = get_model()?; + let mut backend = get_backend()?; let mut all_embeddings = Vec::with_capacity(texts.len()); // Process in batches for efficiency @@ -320,9 +456,15 @@ impl EmbeddingService { }) .collect(); - let embeddings = model - .embed(truncated, None) - .map_err(|e| EmbeddingError::EmbeddingFailed(e.to_string()))?; + let embeddings = match &mut *backend { + EmbeddingBackend::NomicV15(model) => model + .embed(truncated, None) + .map_err(|e| EmbeddingError::EmbeddingFailed(e.to_string()))?, + #[cfg(feature = "qwen3-embeddings")] + EmbeddingBackend::Qwen3Embedding06B(model) => model + .embed(&truncated) + .map_err(|e| EmbeddingError::EmbeddingFailed(e.to_string()))?, + }; for emb in embeddings { all_embeddings.push(Embedding::new(matryoshka_truncate(emb))); @@ -481,6 +623,14 @@ mod tests { } } + #[test] + fn test_qwen3_query_format() { + let formatted = qwen3_format_query("rust memory portability"); + assert!(formatted.starts_with("Instruct:")); + assert!(formatted.contains("retrieve relevant passages")); + assert!(formatted.ends_with("Query: rust memory portability")); + } + #[test] fn test_embedding_normalize() { let mut emb = Embedding::new(vec![3.0, 4.0]); diff --git a/crates/vestige-core/src/embeddings/mod.rs b/crates/vestige-core/src/embeddings/mod.rs index 5d89c10..d38497b 100644 --- a/crates/vestige-core/src/embeddings/mod.rs +++ b/crates/vestige-core/src/embeddings/mod.rs @@ -13,6 +13,7 @@ mod code; mod hybrid; mod local; +#[cfg(feature = "vector-search")] pub(crate) use local::get_cache_dir; pub use local::{ BATCH_SIZE, EMBEDDING_DIMENSIONS, Embedding, EmbeddingError, EmbeddingService, MAX_TEXT_LENGTH, diff --git a/crates/vestige-core/src/lib.rs b/crates/vestige-core/src/lib.rs index 306b2d2..640ba4a 100644 --- a/crates/vestige-core/src/lib.rs +++ b/crates/vestige-core/src/lib.rs @@ -153,7 +153,8 @@ pub use fsrs::{ // Storage layer pub use storage::{ ConnectionRecord, ConsolidationHistoryRecord, DreamHistoryRecord, InsightRecord, - IntentionRecord, Result, SmartIngestResult, StateTransitionRecord, Storage, StorageError, + IntentionRecord, PORTABLE_ARCHIVE_FORMAT, PortableArchive, PortableImportMode, + PortableImportReport, Result, SmartIngestResult, StateTransitionRecord, Storage, StorageError, }; // Consolidation (sleep-inspired memory processing) diff --git a/crates/vestige-core/src/memory/mod.rs b/crates/vestige-core/src/memory/mod.rs index e7c61d4..e8c3f32 100644 --- a/crates/vestige-core/src/memory/mod.rs +++ b/crates/vestige-core/src/memory/mod.rs @@ -247,8 +247,14 @@ pub struct MemoryStats { pub newest_memory: Option>, /// Number of nodes with semantic embeddings pub nodes_with_embeddings: i64, + /// Number of nodes with embeddings generated by the active embedding model family + pub nodes_with_active_embeddings: i64, + /// Number of nodes whose stored embeddings belong to a different model family + pub nodes_with_mismatched_embeddings: i64, /// Embedding model used (if any) pub embedding_model: Option, + /// Embedding model family currently configured for new queries and writes + pub active_embedding_model: Option, } impl Default for MemoryStats { @@ -262,7 +268,10 @@ impl Default for MemoryStats { oldest_memory: None, newest_memory: None, nodes_with_embeddings: 0, + nodes_with_active_embeddings: 0, + nodes_with_mismatched_embeddings: 0, embedding_model: None, + active_embedding_model: None, } } } diff --git a/crates/vestige-core/src/storage/migrations.rs b/crates/vestige-core/src/storage/migrations.rs index 2f1ac5b..fae52ae 100644 --- a/crates/vestige-core/src/storage/migrations.rs +++ b/crates/vestige-core/src/storage/migrations.rs @@ -59,6 +59,11 @@ pub const MIGRATIONS: &[Migration] = &[ description: "v2.0.7 Cleanup: drop dead knowledge_edges and compressed_memories tables", up: MIGRATION_V11_UP, }, + Migration { + version: 12, + description: "v2.1.1 Sync: tombstones for merge-capable portable storage", + up: MIGRATION_V12_UP, + }, ]; /// A database migration @@ -681,6 +686,26 @@ DROP TABLE IF EXISTS compressed_memories; UPDATE schema_version SET version = 11, applied_at = datetime('now'); "#; +/// V12: Merge-capable sync tombstones. +/// +/// Portable sync needs to propagate deletions between devices. `knowledge_nodes` +/// remains the source of truth for live memories; this table records deletes so +/// another device can remove the same memory during a merge import. +const MIGRATION_V12_UP: &str = r#" +CREATE TABLE IF NOT EXISTS sync_tombstones ( + table_name TEXT NOT NULL, + row_id TEXT NOT NULL, + deleted_at TEXT NOT NULL, + reason TEXT, + PRIMARY KEY (table_name, row_id) +); + +CREATE INDEX IF NOT EXISTS idx_sync_tombstones_deleted_at +ON sync_tombstones(deleted_at); + +UPDATE schema_version SET version = 12, applied_at = datetime('now'); +"#; + /// Get current schema version from database pub fn get_current_version(conn: &rusqlite::Connection) -> rusqlite::Result { conn.query_row( @@ -730,17 +755,17 @@ mod tests { /// version after `apply_migrations` runs all migrations end-to-end, and /// neither of the dead tables V11 drops must exist afterwards. #[test] - fn test_apply_migrations_advances_to_v11_and_drops_dead_tables() { + fn test_apply_migrations_advances_to_v12_and_drops_dead_tables() { let conn = rusqlite::Connection::open_in_memory().expect("open in-memory"); // Pre-requisite: schema_version must be bootstrapped by V1. apply_migrations(&conn).expect("apply_migrations succeeds"); - // 1. schema_version advanced to V11 + // 1. schema_version advanced to V12 let version = get_current_version(&conn).expect("read schema_version"); assert_eq!( - version, 11, - "schema_version must be 11 after all migrations" + version, 12, + "schema_version must be 12 after all migrations" ); // 2. knowledge_edges is gone (V11 drops it) @@ -768,6 +793,19 @@ mod tests { compressed_memories_rows, 0, "compressed_memories table must be dropped by V11" ); + + // 4. sync_tombstones exists (V12 creates it) + let sync_tombstone_rows: i64 = conn + .query_row( + "SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='sync_tombstones'", + [], + |row| row.get(0), + ) + .expect("query sqlite_master"); + assert_eq!( + sync_tombstone_rows, 1, + "sync_tombstones table must be created by V12" + ); } /// V11 must be idempotent on replay — if the tables were already dropped @@ -789,6 +827,6 @@ mod tests { apply_migrations(&conn).expect("V11 replay must be idempotent"); let version = get_current_version(&conn).expect("read schema_version"); - assert_eq!(version, 11, "schema_version back at 11 after replay"); + assert_eq!(version, 12, "schema_version back at 12 after replay"); } } diff --git a/crates/vestige-core/src/storage/mod.rs b/crates/vestige-core/src/storage/mod.rs index eb224fa..1660529 100644 --- a/crates/vestige-core/src/storage/mod.rs +++ b/crates/vestige-core/src/storage/mod.rs @@ -7,10 +7,16 @@ //! - Temporal memory support mod migrations; +mod portable; mod sqlite; pub use migrations::MIGRATIONS; -pub use sqlite::{ - ConnectionRecord, ConsolidationHistoryRecord, DreamHistoryRecord, InsightRecord, - IntentionRecord, Result, SmartIngestResult, StateTransitionRecord, Storage, StorageError, +pub use portable::{ + PORTABLE_ARCHIVE_FORMAT, PortableArchive, PortableImportMode, PortableImportReport, + PortableTable, PortableValue, +}; +pub use sqlite::{ + ConnectionRecord, ConsolidationHistoryRecord, DreamHistoryRecord, FilePortableSyncBackend, + InsightRecord, IntentionRecord, PortableSyncBackend, PortableSyncReport, Result, + SmartIngestResult, StateTransitionRecord, Storage, StorageError, }; diff --git a/crates/vestige-core/src/storage/portable.rs b/crates/vestige-core/src/storage/portable.rs new file mode 100644 index 0000000..dda0a74 --- /dev/null +++ b/crates/vestige-core/src/storage/portable.rs @@ -0,0 +1,171 @@ +//! Portable archive types for exact Vestige-to-Vestige transfer. +//! +//! This format preserves SQLite row data instead of re-ingesting memories. It is +//! intentionally storage-level: import can keep IDs, FSRS state, graph edges, +//! suppression state, embeddings, and audit/history rows intact. + +use chrono::{DateTime, Utc}; +use rusqlite::types::Value; +use serde::{Deserialize, Serialize}; + +/// Current portable archive format identifier. +pub const PORTABLE_ARCHIVE_FORMAT: &str = "vestige.portable.v1"; + +/// Full exact portable archive. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct PortableArchive { + /// Stable format marker used for compatibility checks. + pub archive_format: String, + /// Vestige version that produced the archive. + pub vestige_version: String, + /// SQLite schema version of the source database. + pub schema_version: u32, + /// Archive creation timestamp. + pub exported_at: DateTime, + /// Export mode. v1 only writes "exact". + pub mode: String, + /// Dumped storage tables in deterministic import order. + pub tables: Vec, +} + +impl PortableArchive { + /// Count all rows across all tables. + pub fn total_rows(&self) -> usize { + self.tables.iter().map(|table| table.rows.len()).sum() + } +} + +/// One table in a portable archive. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct PortableTable { + /// SQLite table name. + pub name: String, + /// Column names in row value order. + pub columns: Vec, + /// Raw rows. Each row has the same order as `columns`. + pub rows: Vec>, +} + +/// SQLite value encoded in JSON. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(tag = "type", content = "value", rename_all = "camelCase")] +pub enum PortableValue { + /// SQL NULL. + Null, + /// SQL INTEGER. + Integer(i64), + /// SQL REAL. + Real(f64), + /// SQL TEXT. + Text(String), + /// SQL BLOB, hex encoded. + Blob(String), +} + +impl PortableValue { + /// Convert this portable value back into a rusqlite owned value. + pub(crate) fn to_sql_value(&self) -> Result { + match self { + Self::Null => Ok(Value::Null), + Self::Integer(value) => Ok(Value::Integer(*value)), + Self::Real(value) => Ok(Value::Real(*value)), + Self::Text(value) => Ok(Value::Text(value.clone())), + Self::Blob(value) => decode_hex(value).map(Value::Blob), + } + } +} + +/// Import behavior for duplicate primary keys. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum PortableImportMode { + /// Reject import if user data already exists, then insert rows exactly. + EmptyOnly, + /// Merge archive rows into an existing database. + /// + /// This mode is intended for file-backed sync between devices. It applies + /// tombstones, upserts row-keyed state, and appends audit/history rows. + Merge, +} + +/// Summary of an exact portable import. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct PortableImportReport { + /// Number of imported tables. + pub tables_imported: usize, + /// Number of imported rows. + pub rows_imported: usize, + /// Number of archive tables skipped because the target schema lacks them. + pub tables_skipped: usize, + /// Whether FTS was rebuilt after import. + pub fts_rebuilt: bool, + /// Number of rows inserted. + #[serde(default)] + pub rows_inserted: usize, + /// Number of existing rows updated/replaced. + #[serde(default)] + pub rows_updated: usize, + /// Number of rows skipped because local state was newer or unsupported. + #[serde(default)] + pub rows_skipped: usize, + /// Number of local rows deleted by imported tombstones. + #[serde(default)] + pub rows_deleted: usize, + /// Number of merge conflicts resolved by keeping local state. + #[serde(default)] + pub conflicts_kept_local: usize, +} + +pub(crate) fn encode_hex(bytes: &[u8]) -> String { + const HEX: &[u8; 16] = b"0123456789abcdef"; + let mut out = String::with_capacity(bytes.len() * 2); + for &byte in bytes { + out.push(HEX[(byte >> 4) as usize] as char); + out.push(HEX[(byte & 0x0f) as usize] as char); + } + out +} + +fn decode_hex(input: &str) -> Result, String> { + if !input.len().is_multiple_of(2) { + return Err("hex blob has odd length".to_string()); + } + + let mut out = Vec::with_capacity(input.len() / 2); + let bytes = input.as_bytes(); + for chunk in bytes.chunks_exact(2) { + let high = hex_value(chunk[0])?; + let low = hex_value(chunk[1])?; + out.push((high << 4) | low); + } + Ok(out) +} + +fn hex_value(byte: u8) -> Result { + match byte { + b'0'..=b'9' => Ok(byte - b'0'), + b'a'..=b'f' => Ok(byte - b'a' + 10), + b'A'..=b'F' => Ok(byte - b'A' + 10), + _ => Err(format!("invalid hex byte: {}", byte as char)), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn hex_round_trip() { + let bytes = vec![0, 1, 2, 15, 16, 127, 128, 255]; + let encoded = encode_hex(&bytes); + assert_eq!(decode_hex(&encoded).unwrap(), bytes); + } + + #[test] + fn rejects_invalid_hex() { + assert!(decode_hex("f").is_err()); + assert!(decode_hex("zz").is_err()); + } +} diff --git a/crates/vestige-core/src/storage/sqlite.rs b/crates/vestige-core/src/storage/sqlite.rs index 398db9f..fbe3c7c 100644 --- a/crates/vestige-core/src/storage/sqlite.rs +++ b/crates/vestige-core/src/storage/sqlite.rs @@ -3,13 +3,15 @@ //! Core storage layer with integrated embeddings and vector search. use chrono::{DateTime, Duration, Utc}; -use directories::ProjectDirs; -#[cfg(feature = "embeddings")] +use directories::{BaseDirs, ProjectDirs}; +#[cfg(all(feature = "embeddings", feature = "vector-search"))] use lru::LruCache; -use rusqlite::{Connection, OptionalExtension, params}; -#[cfg(feature = "embeddings")] +use rusqlite::types::{Type, Value, ValueRef}; +use rusqlite::{Connection, OptionalExtension, params, params_from_iter}; +use std::io::Write; +#[cfg(all(feature = "embeddings", feature = "vector-search"))] use std::num::NonZeroUsize; -use std::path::PathBuf; +use std::path::{Component, Path, PathBuf}; use std::sync::Mutex; use uuid::Uuid; @@ -18,13 +20,20 @@ use crate::fsrs::{ }; use crate::fts::sanitize_fts5_query; use crate::memory::{ - ConsolidationResult, IngestInput, KnowledgeNode, MemoryStats, RecallInput, SearchMode, + ConsolidationResult, IngestInput, KnowledgeNode, MatchType, MemoryStats, RecallInput, + SearchMode, SearchResult, }; #[cfg(all(feature = "embeddings", feature = "vector-search"))] -use crate::memory::{EmbeddingResult, MatchType, SearchResult, SimilarityResult}; +use crate::memory::{EmbeddingResult, SimilarityResult}; +use crate::storage::portable::{ + PORTABLE_ARCHIVE_FORMAT, PortableArchive, PortableImportMode, PortableImportReport, + PortableTable, PortableValue, encode_hex, +}; #[cfg(feature = "embeddings")] -use crate::embeddings::{EMBEDDING_DIMENSIONS, Embedding, EmbeddingService, matryoshka_truncate}; +use crate::embeddings::EmbeddingService; +#[cfg(all(feature = "embeddings", feature = "vector-search"))] +use crate::embeddings::{EMBEDDING_DIMENSIONS, Embedding, matryoshka_truncate}; #[cfg(feature = "vector-search")] use crate::search::{VectorIndex, linear_combination}; @@ -78,16 +87,167 @@ pub struct SmartIngestResult { pub reason: String, } +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum MergeWrite { + Inserted, + Updated, +} + +/// Backend interface for portable sync storage. +/// +/// The first shipped backend is a local file, which works with Dropbox, iCloud, +/// Syncthing, Git, shared volumes, or any other folder sync tool. Remote stores +/// can implement this trait without changing merge semantics. +pub trait PortableSyncBackend { + /// Human-readable backend label for reports. + fn label(&self) -> String; + /// Read the current remote archive. `Ok(None)` means no remote exists yet. + fn read_archive(&self) -> Result>; + /// Atomically write the merged archive back to the backend when possible. + fn write_archive(&self, archive: &PortableArchive) -> Result<()>; +} + +/// File-backed portable sync backend. +#[derive(Debug, Clone)] +pub struct FilePortableSyncBackend { + path: PathBuf, +} + +impl FilePortableSyncBackend { + /// Create a file-backed sync backend for a portable archive path. + pub fn new(path: impl Into) -> Self { + Self { path: path.into() } + } + + /// Archive path backing this sync store. + pub fn path(&self) -> &Path { + &self.path + } +} + +impl PortableSyncBackend for FilePortableSyncBackend { + fn label(&self) -> String { + format!("file:{}", self.path.display()) + } + + fn read_archive(&self) -> Result> { + if !self.path.exists() { + return Ok(None); + } + let file = std::fs::File::open(&self.path)?; + let archive: PortableArchive = serde_json::from_reader(file).map_err(|e| { + StorageError::Init(format!( + "Failed to parse portable sync archive '{}': {}", + self.path.display(), + e + )) + })?; + Ok(Some(archive)) + } + + fn write_archive(&self, archive: &PortableArchive) -> Result<()> { + let parent = self.path.parent().unwrap_or_else(|| Path::new(".")); + std::fs::create_dir_all(parent)?; + let filename = self + .path + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or("vestige-sync.json"); + let temp_path = parent.join(format!(".{}.tmp-{}", filename, Uuid::new_v4())); + + let mut file = std::fs::File::create(&temp_path)?; + if let Err(e) = serde_json::to_writer_pretty(&mut file, archive) { + let _ = std::fs::remove_file(&temp_path); + return Err(StorageError::Init(format!( + "Failed to write portable sync archive '{}': {}", + self.path.display(), + e + ))); + } + file.flush()?; + file.sync_all()?; + drop(file); + + if let Err(rename_err) = std::fs::rename(&temp_path, &self.path) { + if self.path.exists() { + std::fs::remove_file(&self.path)?; + std::fs::rename(&temp_path, &self.path)?; + } else { + let _ = std::fs::remove_file(&temp_path); + return Err(rename_err.into()); + } + } + Ok(()) + } +} + +/// Summary of a pull-merge-push sync operation. +#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct PortableSyncReport { + /// Backend label that was synced. + pub backend: String, + /// Whether an existing remote archive was pulled before pushing. + pub pulled: bool, + /// Merge report from the pull phase, if a remote archive existed. + pub pull: Option, + /// Number of tables written to the backend during push. + pub pushed_tables: usize, + /// Number of rows written to the backend during push. + pub pushed_rows: usize, + /// Portable archive format written during push. + pub archive_format: String, +} + // ============================================================================ // STORAGE // ============================================================================ +const PORTABLE_TABLES: &[&str] = &[ + "knowledge_nodes", + "node_embeddings", + "fsrs_cards", + "memory_states", + "memory_connections", + "memory_access_log", + "state_transitions", + "intentions", + "insights", + "sessions", + "fsrs_config", + "consolidation_history", + "dream_history", + "retention_snapshots", + "sync_tombstones", +]; + +const PORTABLE_USER_DATA_TABLES: &[&str] = &[ + "knowledge_nodes", + "node_embeddings", + "fsrs_cards", + "memory_states", + "memory_connections", + "memory_access_log", + "state_transitions", + "intentions", + "insights", + "sessions", + "consolidation_history", + "dream_history", + "retention_snapshots", + "sync_tombstones", +]; + +const DATA_DIR_ENV: &str = "VESTIGE_DATA_DIR"; +const DATABASE_FILE: &str = "vestige.db"; + /// Main storage struct with integrated embedding and vector search /// /// Uses separate reader/writer connections for interior mutability. /// All methods take `&self` (not `&mut self`), making Storage `Send + Sync` /// so the MCP layer can use `Arc` instead of `Arc>`. pub struct Storage { + db_path: PathBuf, writer: Mutex, reader: Mutex, scheduler: Mutex, @@ -96,11 +256,75 @@ pub struct Storage { #[cfg(feature = "vector-search")] vector_index: Mutex, /// LRU cache for query embeddings to avoid re-embedding repeated queries - #[cfg(feature = "embeddings")] + #[cfg(all(feature = "embeddings", feature = "vector-search"))] query_cache: Mutex>>, } impl Storage { + fn data_dir_from_env() -> Option { + std::env::var_os(DATA_DIR_ENV).and_then(|value| { + if value.is_empty() { + None + } else { + Some(PathBuf::from(value)) + } + }) + } + + fn expand_tilde(path: PathBuf) -> PathBuf { + let rest = { + let mut components = path.components(); + match components.next() { + Some(Component::Normal(first)) if first == "~" => { + Some(components.as_path().to_path_buf()) + } + _ => None, + } + }; + + match rest { + Some(rest) => BaseDirs::new() + .map(|dirs| dirs.home_dir().join(rest)) + .unwrap_or(path), + None => path, + } + } + + fn prepare_data_dir(data_dir: PathBuf) -> Result { + let data_dir = Self::expand_tilde(data_dir); + std::fs::create_dir_all(&data_dir)?; + // Restrict directory permissions to owner-only on Unix + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + let perms = std::fs::Permissions::from_mode(0o700); + let _ = std::fs::set_permissions(&data_dir, perms); + } + Ok(data_dir.join(DATABASE_FILE)) + } + + /// Resolve a Vestige database path from an explicit data directory. + pub fn db_path_for_data_dir(data_dir: PathBuf) -> Result { + Self::prepare_data_dir(data_dir) + } + + /// Resolve the default Vestige database path. + /// + /// `VESTIGE_DATA_DIR` is treated as a directory and wins over the platform + /// per-user data directory. The database file is always `vestige.db` inside + /// that directory. + pub fn default_db_path() -> Result { + if let Some(data_dir) = Self::data_dir_from_env() { + return Self::prepare_data_dir(data_dir); + } + + let proj_dirs = ProjectDirs::from("com", "vestige", "core").ok_or_else(|| { + StorageError::Init("Could not determine project directories".to_string()) + })?; + + Self::prepare_data_dir(proj_dirs.data_dir().to_path_buf()) + } + /// Apply PRAGMAs and optional encryption to a connection fn configure_connection(conn: &Connection) -> Result<()> { // Apply encryption key if SQLCipher is enabled and key is provided @@ -133,22 +357,7 @@ impl Storage { pub fn new(db_path: Option) -> Result { let path = match db_path { Some(p) => p, - None => { - let proj_dirs = ProjectDirs::from("com", "vestige", "core").ok_or_else(|| { - StorageError::Init("Could not determine project directories".to_string()) - })?; - - let data_dir = proj_dirs.data_dir(); - std::fs::create_dir_all(data_dir)?; - // Restrict directory permissions to owner-only on Unix - #[cfg(unix)] - { - use std::os::unix::fs::PermissionsExt; - let perms = std::fs::Permissions::from_mode(0o700); - let _ = std::fs::set_permissions(data_dir, perms); - } - data_dir.join("vestige.db") - } + None => Self::default_db_path()?, }; // Open writer connection @@ -180,12 +389,13 @@ impl Storage { // Initialize LRU cache for query embeddings (capacity: 100 queries) // SAFETY: 100 is always non-zero, this cannot fail - #[cfg(feature = "embeddings")] + #[cfg(all(feature = "embeddings", feature = "vector-search"))] let query_cache = Mutex::new(LruCache::new( NonZeroUsize::new(100).expect("100 is non-zero"), )); let storage = Self { + db_path: path, writer: Mutex::new(writer_conn), reader: Mutex::new(reader_conn), scheduler: Mutex::new(FSRSScheduler::default()), @@ -193,7 +403,7 @@ impl Storage { embedding_service, #[cfg(feature = "vector-search")] vector_index: Mutex::new(vector_index), - #[cfg(feature = "embeddings")] + #[cfg(all(feature = "embeddings", feature = "vector-search"))] query_cache, }; @@ -203,6 +413,21 @@ impl Storage { Ok(storage) } + /// Absolute path of the SQLite database this storage instance uses. + pub fn db_path(&self) -> &Path { + &self.db_path + } + + /// Data directory containing the SQLite database and sidecar folders. + pub fn data_dir(&self) -> &Path { + self.db_path.parent().unwrap_or_else(|| Path::new(".")) + } + + /// Sidecar directory for files belonging to this storage instance. + pub fn sidecar_dir(&self, name: &str) -> PathBuf { + self.data_dir().join(name) + } + /// Load existing embeddings into vector index #[cfg(all(feature = "embeddings", feature = "vector-search"))] fn load_embeddings_into_index(&self) -> Result<()> { @@ -211,10 +436,10 @@ impl Storage { .lock() .map_err(|_| StorageError::Init("Reader lock poisoned".into()))?; - let mut stmt = reader.prepare("SELECT node_id, embedding FROM node_embeddings")?; + let mut stmt = reader.prepare("SELECT node_id, embedding, model FROM node_embeddings")?; - let embeddings: Vec<(String, Vec)> = stmt - .query_map([], |row| Ok((row.get(0)?, row.get(1)?)))? + let embeddings: Vec<(String, Vec, String)> = stmt + .query_map([], |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)))? .filter_map(|r| r.ok()) .collect(); @@ -227,11 +452,32 @@ impl Storage { .map_err(|_| StorageError::Init("Vector index lock poisoned".to_string()))?; let mut load_failures = 0u32; - for (node_id, embedding_bytes) in embeddings { + let mut skipped_model_mismatches = 0u32; + let active_model = self.embedding_service.model_name(); + for (node_id, embedding_bytes, model_name) in embeddings { + if !Self::embedding_model_matches_active(&model_name, active_model) { + skipped_model_mismatches += 1; + continue; + } + if let Some(embedding) = Embedding::from_bytes(&embedding_bytes) { - // Handle Matryoshka migration: old 768-dim → truncate to 256-dim + // Handle Matryoshka models explicitly. Do not silently truncate + // unknown embedding families into the active 256d index. let vector = if embedding.dimensions != EMBEDDING_DIMENSIONS { - matryoshka_truncate(embedding.vector) + let model_lower = model_name.to_ascii_lowercase(); + if model_lower.contains("nomic") || model_lower.contains("qwen3") { + matryoshka_truncate(embedding.vector) + } else { + load_failures += 1; + tracing::warn!( + node_id = %node_id, + model = %model_name, + dimensions = embedding.dimensions, + expected = EMBEDDING_DIMENSIONS, + "Skipping embedding with incompatible dimensions" + ); + continue; + } } else { embedding.vector }; @@ -248,6 +494,13 @@ impl Storage { load_failures ); } + if skipped_model_mismatches > 0 { + tracing::info!( + count = skipped_model_mismatches, + active_model = active_model, + "Vector index skipped embeddings from a different model family; run consolidation to re-embed them" + ); + } Ok(()) } @@ -569,14 +822,19 @@ impl Storage { .lock() .map_err(|_| StorageError::Init("Reader lock poisoned".into()))?; let mut stmt = - reader.prepare("SELECT embedding FROM node_embeddings WHERE node_id = ?1")?; + reader.prepare("SELECT embedding, model FROM node_embeddings WHERE node_id = ?1")?; - let embedding_bytes: Option> = stmt - .query_row(params![node_id], |row| row.get(0)) + let embedding_row: Option<(Vec, String)> = stmt + .query_row(params![node_id], |row| Ok((row.get(0)?, row.get(1)?))) .optional()?; - Ok(embedding_bytes - .and_then(|bytes| crate::embeddings::Embedding::from_bytes(&bytes).map(|e| e.vector))) + Ok(embedding_row.and_then(|(bytes, model)| { + Self::embedding_vector_for_active_model( + &bytes, + &model, + self.embedding_service.model_name(), + ) + })) } /// Get all embedding vectors for duplicate detection @@ -586,23 +844,32 @@ impl Storage { .reader .lock() .map_err(|_| StorageError::Init("Reader lock poisoned".into()))?; - let mut stmt = reader.prepare("SELECT node_id, embedding FROM node_embeddings")?; + let mut stmt = reader.prepare("SELECT node_id, embedding, model FROM node_embeddings")?; + let active_model = self.embedding_service.model_name(); let results: Vec<(String, Vec)> = stmt .query_map([], |row| { let node_id: String = row.get(0)?; let embedding_bytes: Vec = row.get(1)?; - Ok((node_id, embedding_bytes)) + let model: String = row.get(2)?; + Ok((node_id, embedding_bytes, model)) })? .filter_map(|r| r.ok()) - .filter_map(|(id, bytes)| { - crate::embeddings::Embedding::from_bytes(&bytes).map(|e| (id, e.vector)) + .filter_map(|(id, bytes, model)| { + Self::embedding_vector_for_active_model(&bytes, &model, active_model) + .map(|vector| (id, vector)) }) .collect(); Ok(results) } + /// Fallback for builds without local embeddings/vector search. + #[cfg(not(all(feature = "embeddings", feature = "vector-search")))] + pub fn get_node_embedding(&self, _node_id: &str) -> Result>> { + Ok(None) + } + /// Update the content of an existing node pub fn update_node_content(&self, id: &str, new_content: &str) -> Result<()> { let now = Utc::now(); @@ -645,6 +912,7 @@ impl Storage { .embedding_service .embed(content) .map_err(|e| StorageError::Init(format!("Embedding failed: {}", e)))?; + let model_name = self.embedding_service.model_name(); let now = Utc::now(); @@ -659,15 +927,15 @@ impl Storage { params![ node_id, embedding.to_bytes(), - EMBEDDING_DIMENSIONS as i32, - "nomic-embed-text-v1.5", + embedding.dimensions as i32, + model_name, now.to_rfc3339(), ], )?; writer.execute( - "UPDATE knowledge_nodes SET has_embedding = 1, embedding_model = 'nomic-embed-text-v1.5' WHERE id = ?1", - params![node_id], + "UPDATE knowledge_nodes SET has_embedding = 1, embedding_model = ?2 WHERE id = ?1", + params![node_id, model_name], )?; } @@ -1449,11 +1717,56 @@ impl Storage { |row| row.get(0), )?; - let embedding_model: Option = if nodes_with_embeddings > 0 { - Some("nomic-embed-text-v1.5".to_string()) - } else { - None + let embedding_model: Option = reader + .query_row( + "SELECT model + FROM node_embeddings + GROUP BY model + ORDER BY COUNT(*) DESC, model ASC + LIMIT 1", + [], + |row| row.get(0), + ) + .optional()?; + + #[cfg(feature = "embeddings")] + let active_embedding_model = Some(self.embedding_service.model_name().to_string()); + #[cfg(not(feature = "embeddings"))] + let active_embedding_model = None; + + #[cfg(feature = "embeddings")] + let (nodes_with_active_embeddings, nodes_with_mismatched_embeddings) = { + let active_model = active_embedding_model.as_deref().unwrap_or_default(); + let model_pattern = Self::active_embedding_model_like_pattern(active_model); + let active_count: i64 = reader.query_row( + "SELECT COUNT(*) + FROM knowledge_nodes kn + WHERE kn.has_embedding = 1 + AND EXISTS ( + SELECT 1 FROM node_embeddings ne + WHERE ne.node_id = kn.id + AND ne.model LIKE ?1 + )", + params![&model_pattern], + |row| row.get(0), + )?; + let mismatched_count: i64 = reader.query_row( + "SELECT COUNT(*) + FROM knowledge_nodes kn + WHERE kn.has_embedding = 1 + AND NOT EXISTS ( + SELECT 1 FROM node_embeddings ne + WHERE ne.node_id = kn.id + AND ne.model LIKE ?1 + )", + params![&model_pattern], + |row| row.get(0), + )?; + (active_count, mismatched_count) }; + #[cfg(not(feature = "embeddings"))] + let (nodes_with_active_embeddings, nodes_with_mismatched_embeddings) = + (nodes_with_embeddings, 0); Ok(MemoryStats { total_nodes: total, @@ -1472,7 +1785,10 @@ impl Storage { .ok() }), nodes_with_embeddings, + nodes_with_active_embeddings, + nodes_with_mismatched_embeddings, embedding_model, + active_embedding_model, }) } @@ -1482,6 +1798,9 @@ impl Storage { .writer .lock() .map_err(|_| StorageError::Init("Writer lock poisoned".into()))?; + if Self::node_exists(&writer, id)? { + Self::record_sync_tombstone(&writer, "knowledge_nodes", id, "delete_node")?; + } let rows = writer.execute("DELETE FROM knowledge_nodes WHERE id = ?1", params![id])?; // Clean up vector index to prevent stale search results @@ -1495,6 +1814,32 @@ impl Storage { Ok(rows > 0) } + fn node_exists(conn: &Connection, id: &str) -> Result { + let count: i64 = conn.query_row( + "SELECT COUNT(*) FROM knowledge_nodes WHERE id = ?1", + params![id], + |row| row.get(0), + )?; + Ok(count > 0) + } + + fn record_sync_tombstone( + conn: &Connection, + table_name: &str, + row_id: &str, + reason: &str, + ) -> Result<()> { + conn.execute( + "INSERT INTO sync_tombstones (table_name, row_id, deleted_at, reason) + VALUES (?1, ?2, ?3, ?4) + ON CONFLICT(table_name, row_id) DO UPDATE SET + deleted_at = excluded.deleted_at, + reason = excluded.reason", + params![table_name, row_id, Utc::now().to_rfc3339(), reason], + )?; + Ok(()) + } + /// Search with full-text search pub fn search(&self, query: &str, limit: i32) -> Result> { let sanitized_query = sanitize_fts5_query(query); @@ -1652,15 +1997,16 @@ impl Storage { } /// Get query embedding from cache or compute it - #[cfg(feature = "embeddings")] + #[cfg(all(feature = "embeddings", feature = "vector-search"))] fn get_query_embedding(&self, query: &str) -> Result> { + let cache_key = format!("{}\0{}", self.embedding_service.model_name(), query); // Check cache first { let mut cache = self .query_cache .lock() .map_err(|_| StorageError::Init("Query cache lock poisoned".to_string()))?; - if let Some(cached) = cache.get(query) { + if let Some(cached) = cache.get(&cache_key) { return Ok(cached.clone()); } } @@ -1668,7 +2014,7 @@ impl Storage { // Not in cache, compute embedding let embedding = self .embedding_service - .embed(query) + .embed_query(query) .map_err(|e| StorageError::Init(format!("Failed to embed query: {}", e)))?; // Store in cache @@ -1677,7 +2023,7 @@ impl Storage { .query_cache .lock() .map_err(|_| StorageError::Init("Query cache lock poisoned".to_string()))?; - cache.put(query.to_string(), embedding.vector.clone()); + cache.put(cache_key, embedding.vector.clone()); } Ok(embedding.vector) @@ -1864,6 +2210,60 @@ impl Storage { Ok(results) } + /// Keyword-only fallback for builds without local embeddings/vector search. + #[cfg(not(all(feature = "embeddings", feature = "vector-search")))] + pub fn hybrid_search( + &self, + query: &str, + limit: i32, + _keyword_weight: f32, + _semantic_weight: f32, + ) -> Result> { + self.hybrid_search_filtered(query, limit, 1.0, 0.0, None, None) + } + + /// Keyword-only fallback for builds without local embeddings/vector search. + #[cfg(not(all(feature = "embeddings", feature = "vector-search")))] + pub fn hybrid_search_filtered( + &self, + query: &str, + limit: i32, + _keyword_weight: f32, + _semantic_weight: f32, + include_types: Option<&[String]>, + exclude_types: Option<&[String]>, + ) -> Result> { + let nodes = self.search_terms(query, limit.max(1) * 4)?; + let mut results = Vec::new(); + + for node in nodes { + if let Some(includes) = include_types { + if !includes.iter().any(|t| t == &node.node_type) { + continue; + } + } else if let Some(excludes) = exclude_types + && excludes.iter().any(|t| t == &node.node_type) + { + continue; + } + + let score = 1.0 / (results.len() as f32 + 1.0); + results.push(SearchResult { + node, + keyword_score: Some(score), + semantic_score: None, + combined_score: score, + match_type: MatchType::Keyword, + }); + + if results.len() >= limit.max(1) as usize { + break; + } + } + + Ok(results) + } + /// Keyword search returning scores, with optional type filtering in the SQL query. #[cfg(all(feature = "embeddings", feature = "vector-search"))] fn keyword_search_with_scores( @@ -2008,8 +2408,10 @@ impl Storage { } let mut result = EmbeddingResult::default(); + let active_model = self.embedding_service.model_name(); + let model_pattern = Self::active_embedding_model_like_pattern(active_model); - let nodes: Vec<(String, String)> = { + let nodes: Vec<(String, String, Option)> = { let reader = self .reader .lock() @@ -2017,7 +2419,10 @@ impl Storage { if let Some(ids) = node_ids { let placeholders = ids.iter().map(|_| "?").collect::>().join(","); let query = format!( - "SELECT id, content FROM knowledge_nodes WHERE id IN ({})", + "SELECT kn.id, kn.content, COALESCE(ne.model, kn.embedding_model) AS embedding_model + FROM knowledge_nodes kn + LEFT JOIN node_embeddings ne ON ne.node_id = kn.id + WHERE kn.id IN ({})", placeholders ); @@ -2028,7 +2433,11 @@ impl Storage { ids.iter().map(|s| s as &dyn rusqlite::ToSql).collect(); let rows = stmt.query_map(params.as_slice(), |row| { - Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?)) + Ok(( + row.get::<_, String>(0)?, + row.get::<_, String>(1)?, + row.get::<_, Option>(2)?, + )) })?; for r in rows.flatten() { @@ -2037,37 +2446,58 @@ impl Storage { } result_nodes } else if force { - let mut stmt = reader.prepare("SELECT id, content FROM knowledge_nodes")?; + let mut stmt = + reader.prepare("SELECT id, content, embedding_model FROM knowledge_nodes")?; let rows = stmt.query_map([], |row| { - Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?)) + Ok(( + row.get::<_, String>(0)?, + row.get::<_, String>(1)?, + row.get::<_, Option>(2)?, + )) })?; rows.filter_map(|r| r.ok()).collect() } else { let mut stmt = reader.prepare( - "SELECT id, content FROM knowledge_nodes - WHERE has_embedding = 0 OR has_embedding IS NULL", + "SELECT kn.id, kn.content, COALESCE(ne.model, kn.embedding_model) AS embedding_model + FROM knowledge_nodes kn + LEFT JOIN node_embeddings ne ON ne.node_id = kn.id + WHERE kn.has_embedding = 0 + OR kn.has_embedding IS NULL + OR ne.node_id IS NULL + OR COALESCE(ne.model, kn.embedding_model, '') NOT LIKE ?1", )?; - let rows = stmt.query_map([], |row| { - Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?)) + let rows = stmt.query_map(params![model_pattern], |row| { + Ok(( + row.get::<_, String>(0)?, + row.get::<_, String>(1)?, + row.get::<_, Option>(2)?, + )) })?; rows.filter_map(|r| r.ok()).collect() } }; - for (id, content) in nodes { + for (id, content, stored_model) in nodes { if !force { - let has_emb: i32 = self + let (has_emb, stored_model): (i32, Option) = self .reader .lock() .map_err(|_| StorageError::Init("Reader lock poisoned".into()))? .query_row( - "SELECT COALESCE(has_embedding, 0) FROM knowledge_nodes WHERE id = ?1", - params![id], - |row| row.get(0), + "SELECT COALESCE(kn.has_embedding, 0), COALESCE(ne.model, kn.embedding_model) + FROM knowledge_nodes kn + LEFT JOIN node_embeddings ne ON ne.node_id = kn.id + WHERE kn.id = ?1", + params![&id], + |row| Ok((row.get(0)?, row.get(1)?)), ) - .unwrap_or(0); + .unwrap_or_else(|_| (0, stored_model)); - if has_emb == 1 { + if has_emb == 1 + && stored_model.as_deref().is_some_and(|model| { + Self::embedding_model_matches_active(model, active_model) + }) + { result.skipped += 1; continue; } @@ -2922,6 +3352,9 @@ impl Storage { return Ok(0); } + let active_model = self.embedding_service.model_name(); + let model_pattern = Self::active_embedding_model_like_pattern(active_model); + let nodes: Vec<(String, String)> = { let reader = self .reader @@ -2929,11 +3362,16 @@ impl Storage { .map_err(|_| StorageError::Init("Reader lock poisoned".into()))?; reader .prepare( - "SELECT id, content FROM knowledge_nodes - WHERE has_embedding = 0 OR has_embedding IS NULL + "SELECT kn.id, kn.content + FROM knowledge_nodes kn + LEFT JOIN node_embeddings ne ON ne.node_id = kn.id + WHERE kn.has_embedding = 0 + OR kn.has_embedding IS NULL + OR ne.node_id IS NULL + OR COALESCE(ne.model, kn.embedding_model, '') NOT LIKE ?1 LIMIT 100", )? - .query_map([], |row| Ok((row.get(0)?, row.get(1)?)))? + .query_map(params![model_pattern], |row| Ok((row.get(0)?, row.get(1)?)))? .filter_map(|r| r.ok()) .collect() }; @@ -3863,12 +4301,7 @@ impl Storage { Ok(count) } - /// Get last backup timestamp by scanning the backups directory. - /// Parses `vestige-YYYYMMDD-HHMMSS.db` filenames. - pub fn get_last_backup_timestamp() -> Option> { - let proj_dirs = directories::ProjectDirs::from("com", "vestige", "core")?; - let backup_dir = proj_dirs.data_dir().parent()?.join("backups"); - + fn scan_last_backup_timestamp(backup_dir: &Path) -> Option> { if !backup_dir.exists() { return None; } @@ -3897,6 +4330,859 @@ impl Storage { latest } + /// Get last backup timestamp for this storage instance. + /// Parses `vestige-YYYYMMDD-HHMMSS.db` filenames. + pub fn last_backup_timestamp(&self) -> Option> { + Self::scan_last_backup_timestamp(&self.sidecar_dir("backups")) + } + + /// Get last backup timestamp in the default backups directory. + /// Kept for compatibility with older callers. + pub fn get_last_backup_timestamp() -> Option> { + let backup_dir = Self::default_db_path().ok()?.parent()?.join("backups"); + Self::scan_last_backup_timestamp(&backup_dir) + } + + /// Export an exact portable archive preserving raw Vestige storage rows. + /// + /// Unlike the user-facing JSON export, this preserves IDs, timestamps, + /// FSRS state, graph edges, suppression state, history tables, and raw + /// embedding blobs. It is intended for Vestige-to-Vestige device transfer. + pub fn export_portable_archive(&self) -> Result { + let mut reader = self + .reader + .lock() + .map_err(|_| StorageError::Init("Reader lock poisoned".into()))?; + let tx = reader.transaction()?; + + let schema_version = Self::current_schema_version(&tx)?; + let mut tables = Vec::new(); + + for table_name in PORTABLE_TABLES { + if !Self::table_exists(&tx, table_name)? { + continue; + } + + let quoted_table = Self::quote_ident(table_name); + let mut stmt = tx.prepare(&format!("SELECT * FROM {} ORDER BY rowid", quoted_table))?; + let columns: Vec = stmt + .column_names() + .iter() + .map(|name| (*name).to_string()) + .collect(); + let column_count = columns.len(); + + let rows = stmt.query_map([], |row| { + let mut values = Vec::with_capacity(column_count); + for idx in 0..column_count { + values.push(Self::portable_value_from_ref(row.get_ref(idx)?)?); + } + Ok(values) + })?; + + let mut portable_rows = Vec::new(); + for row in rows { + portable_rows.push(row?); + } + + tables.push(PortableTable { + name: (*table_name).to_string(), + columns, + rows: portable_rows, + }); + } + + let archive = PortableArchive { + archive_format: PORTABLE_ARCHIVE_FORMAT.to_string(), + vestige_version: crate::VERSION.to_string(), + schema_version, + exported_at: Utc::now(), + mode: "exact".to_string(), + tables, + }; + tx.commit()?; + Ok(archive) + } + + /// Write an exact portable archive to a JSON file. + pub fn export_portable_archive_to_path( + &self, + path: &std::path::Path, + ) -> Result { + let archive = self.export_portable_archive()?; + let parent = path.parent().unwrap_or_else(|| std::path::Path::new(".")); + let filename = path + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or("vestige-portable.json"); + let temp_path = parent.join(format!(".{}.tmp-{}", filename, Uuid::new_v4())); + + let mut file = std::fs::File::create(&temp_path)?; + if let Err(e) = serde_json::to_writer_pretty(&mut file, &archive) { + let _ = std::fs::remove_file(&temp_path); + return Err(StorageError::Init(format!( + "Failed to write portable archive: {}", + e + ))); + } + file.flush()?; + file.sync_all()?; + drop(file); + + if let Err(rename_err) = std::fs::rename(&temp_path, path) { + if path.exists() { + std::fs::remove_file(path)?; + std::fs::rename(&temp_path, path)?; + } else { + let _ = std::fs::remove_file(&temp_path); + return Err(rename_err.into()); + } + } + Ok(archive) + } + + /// Import an exact portable archive. + /// + /// `EmptyOnly` preserves the conservative migration path. `Merge` is used + /// by portable sync to combine non-empty databases with tombstones and + /// newer-local conflict handling. + pub fn import_portable_archive( + &self, + archive: &PortableArchive, + mode: PortableImportMode, + ) -> Result { + if archive.archive_format != PORTABLE_ARCHIVE_FORMAT { + return Err(StorageError::Init(format!( + "Unsupported portable archive format '{}'", + archive.archive_format + ))); + } + if archive.mode != "exact" { + return Err(StorageError::Init(format!( + "Unsupported portable archive mode '{}'", + archive.mode + ))); + } + + let mut seen_tables = std::collections::HashSet::new(); + let mut tables_by_name = std::collections::HashMap::new(); + for table in &archive.tables { + if !PORTABLE_TABLES.contains(&table.name.as_str()) { + return Err(StorageError::Init(format!( + "Portable archive contains unsupported table '{}'", + table.name + ))); + } + if !seen_tables.insert(table.name.as_str()) { + return Err(StorageError::Init(format!( + "Portable archive contains duplicate table '{}'", + table.name + ))); + } + tables_by_name.insert(table.name.as_str(), table); + } + + let mut report = PortableImportReport { + tables_imported: 0, + rows_imported: 0, + tables_skipped: 0, + fts_rebuilt: false, + rows_inserted: 0, + rows_updated: 0, + rows_skipped: 0, + rows_deleted: 0, + conflicts_kept_local: 0, + }; + + { + let mut writer = self + .writer + .lock() + .map_err(|_| StorageError::Init("Writer lock poisoned".into()))?; + + let current_schema = Self::current_schema_version(&writer)?; + if archive.schema_version > current_schema { + return Err(StorageError::Init(format!( + "Archive schema version {} is newer than this Vestige database schema {}", + archive.schema_version, current_schema + ))); + } + + match mode { + PortableImportMode::EmptyOnly => { + Self::ensure_portable_import_target_empty(&writer)? + } + PortableImportMode::Merge => {} + } + + let tx = writer.transaction()?; + + for table_name in PORTABLE_TABLES { + let Some(table) = tables_by_name.get(table_name) else { + continue; + }; + + if !Self::table_exists(&tx, table_name)? { + report.tables_skipped += 1; + continue; + } + + if mode == PortableImportMode::Merge { + Self::merge_portable_table(&tx, table_name, table, &mut report)?; + report.tables_imported += 1; + continue; + } + + let target_columns = Self::table_columns(&tx, table_name)?; + let mut insert_columns = Vec::new(); + let mut source_indexes = Vec::new(); + + for (idx, column) in table.columns.iter().enumerate() { + if target_columns.iter().any(|target| target == column) { + insert_columns.push(column.clone()); + source_indexes.push(idx); + } + } + + if insert_columns.is_empty() { + report.tables_skipped += 1; + continue; + } + + let quoted_table = Self::quote_ident(table_name); + let quoted_columns = insert_columns + .iter() + .map(|column| Self::quote_ident(column)) + .collect::>() + .join(", "); + let placeholders = std::iter::repeat_n("?", insert_columns.len()) + .collect::>() + .join(", "); + let verb = if *table_name == "fsrs_config" { + "INSERT OR REPLACE" + } else { + "INSERT" + }; + let sql = format!( + "{} INTO {} ({}) VALUES ({})", + verb, quoted_table, quoted_columns, placeholders + ); + + for row in &table.rows { + if row.len() != table.columns.len() { + return Err(StorageError::Init(format!( + "Portable archive row in table '{}' has {} values for {} columns", + table_name, + row.len(), + table.columns.len() + ))); + } + + let values = source_indexes + .iter() + .map(|idx| row[*idx].to_sql_value()) + .collect::, _>>() + .map_err(|e| { + StorageError::Init(format!("Invalid portable value: {}", e)) + })?; + tx.execute(&sql, params_from_iter(values))?; + report.rows_imported += 1; + report.rows_inserted += 1; + } + + report.tables_imported += 1; + } + + if Self::table_exists(&tx, "knowledge_fts")? { + tx.execute( + "INSERT INTO knowledge_fts(knowledge_fts) VALUES('rebuild')", + [], + )?; + report.fts_rebuilt = true; + } + + tx.commit()?; + } + + #[cfg(all(feature = "embeddings", feature = "vector-search"))] + self.load_embeddings_into_index()?; + + Ok(report) + } + + /// Read and import an exact portable archive JSON file. + pub fn import_portable_archive_from_path( + &self, + path: &std::path::Path, + mode: PortableImportMode, + ) -> Result { + let file = std::fs::File::open(path)?; + let archive: PortableArchive = serde_json::from_reader(file) + .map_err(|e| StorageError::Init(format!("Failed to parse portable archive: {}", e)))?; + self.import_portable_archive(&archive, mode) + } + + /// Synchronize this database with a pluggable portable archive backend. + /// + /// Sync is pull-merge-push: + /// 1. read remote archive if present, + /// 2. merge it into the local database using tombstones and conflict rules, + /// 3. export the merged local database, + /// 4. write the archive back through the backend. + pub fn sync_portable_archive( + &self, + backend: &B, + ) -> Result { + let (pulled, pull) = match backend.read_archive()? { + Some(remote) => ( + true, + Some(self.import_portable_archive(&remote, PortableImportMode::Merge)?), + ), + None => (false, None), + }; + + let archive = self.export_portable_archive()?; + let pushed_tables = archive.tables.len(); + let pushed_rows = archive.total_rows(); + let archive_format = archive.archive_format.clone(); + backend.write_archive(&archive)?; + + Ok(PortableSyncReport { + backend: backend.label(), + pulled, + pull, + pushed_tables, + pushed_rows, + archive_format, + }) + } + + /// Synchronize this database with a file-backed portable archive. + pub fn sync_portable_archive_file(&self, path: &std::path::Path) -> Result { + let backend = FilePortableSyncBackend::new(path); + self.sync_portable_archive(&backend) + } + + fn merge_portable_table( + tx: &rusqlite::Transaction<'_>, + table_name: &str, + table: &PortableTable, + report: &mut PortableImportReport, + ) -> Result<()> { + match table_name { + "sync_tombstones" => Self::merge_sync_tombstones(tx, table, report), + "knowledge_nodes" => Self::merge_knowledge_nodes(tx, table, report), + "memory_access_log" + | "state_transitions" + | "consolidation_history" + | "dream_history" + | "retention_snapshots" => Self::merge_append_only_table(tx, table_name, table, report), + "node_embeddings" => { + Self::merge_keyed_table(tx, table_name, table, &["node_id"], report) + } + "fsrs_cards" | "memory_states" => { + Self::merge_keyed_table(tx, table_name, table, &["memory_id"], report) + } + "memory_connections" => { + Self::merge_keyed_table(tx, table_name, table, &["source_id", "target_id"], report) + } + "intentions" | "insights" | "sessions" => { + Self::merge_keyed_table(tx, table_name, table, &["id"], report) + } + "fsrs_config" => Self::merge_keyed_table(tx, table_name, table, &["key"], report), + _ => { + report.tables_skipped += 1; + Ok(()) + } + } + } + + fn merge_knowledge_nodes( + tx: &rusqlite::Transaction<'_>, + table: &PortableTable, + report: &mut PortableImportReport, + ) -> Result<()> { + for row in &table.rows { + let Some(id) = Self::portable_text(table, row, "id") else { + report.rows_skipped += 1; + continue; + }; + let incoming_updated = Self::portable_timestamp(table, row, "updated_at"); + + if let Some(deleted_at) = Self::tombstone_timestamp(tx, "knowledge_nodes", id)? + && incoming_updated.is_some_and(|updated| deleted_at >= updated) + { + report.conflicts_kept_local += 1; + report.rows_skipped += 1; + continue; + } + + let existing_updated: Option = tx + .query_row( + "SELECT updated_at FROM knowledge_nodes WHERE id = ?1", + params![id], + |row| row.get(0), + ) + .optional()?; + + if let (Some(existing), Some(incoming)) = ( + existing_updated + .as_deref() + .and_then(Self::parse_rfc3339_opt), + incoming_updated, + ) { + if existing > incoming { + report.conflicts_kept_local += 1; + report.rows_skipped += 1; + continue; + } + } + + let affected = Self::insert_or_replace_row(tx, "knowledge_nodes", table, row)?; + report.rows_imported += 1; + if affected == MergeWrite::Inserted { + report.rows_inserted += 1; + } else { + report.rows_updated += 1; + } + } + Ok(()) + } + + fn merge_sync_tombstones( + tx: &rusqlite::Transaction<'_>, + table: &PortableTable, + report: &mut PortableImportReport, + ) -> Result<()> { + for row in &table.rows { + let Some(table_name) = Self::portable_text(table, row, "table_name") else { + report.rows_skipped += 1; + continue; + }; + let Some(row_id) = Self::portable_text(table, row, "row_id") else { + report.rows_skipped += 1; + continue; + }; + let deleted_at = Self::portable_timestamp(table, row, "deleted_at"); + + let affected = Self::insert_or_replace_row(tx, "sync_tombstones", table, row)?; + report.rows_imported += 1; + if affected == MergeWrite::Inserted { + report.rows_inserted += 1; + } else { + report.rows_updated += 1; + } + + if table_name == "knowledge_nodes" { + let local_updated: Option = tx + .query_row( + "SELECT updated_at FROM knowledge_nodes WHERE id = ?1", + params![row_id], + |row| row.get(0), + ) + .optional()?; + let should_delete = match ( + local_updated.as_deref().and_then(Self::parse_rfc3339_opt), + deleted_at, + ) { + (Some(local), Some(deleted)) => deleted >= local, + (Some(_), None) => true, + (None, _) => false, + }; + if should_delete { + let deleted = + tx.execute("DELETE FROM knowledge_nodes WHERE id = ?1", params![row_id])?; + report.rows_deleted += deleted; + } + } + } + Ok(()) + } + + fn merge_keyed_table( + tx: &rusqlite::Transaction<'_>, + table_name: &str, + table: &PortableTable, + key_columns: &[&str], + report: &mut PortableImportReport, + ) -> Result<()> { + for row in &table.rows { + if !Self::parent_rows_exist(tx, table_name, table, row)? { + report.rows_skipped += 1; + continue; + } + if key_columns + .iter() + .any(|column| Self::portable_value(table, row, column).is_none()) + { + report.rows_skipped += 1; + continue; + } + let affected = Self::insert_or_replace_row(tx, table_name, table, row)?; + report.rows_imported += 1; + if affected == MergeWrite::Inserted { + report.rows_inserted += 1; + } else { + report.rows_updated += 1; + } + } + Ok(()) + } + + fn merge_append_only_table( + tx: &rusqlite::Transaction<'_>, + table_name: &str, + table: &PortableTable, + report: &mut PortableImportReport, + ) -> Result<()> { + for row in &table.rows { + if !Self::parent_rows_exist(tx, table_name, table, row)? { + report.rows_skipped += 1; + continue; + } + + let insert_columns: Vec = table + .columns + .iter() + .filter(|column| column.as_str() != "id") + .cloned() + .collect(); + if insert_columns.is_empty() { + report.rows_skipped += 1; + continue; + } + + let values = Self::row_values_for_columns(table, row, &insert_columns)?; + if Self::row_exists_by_values(tx, table_name, &insert_columns, &values)? { + report.rows_skipped += 1; + continue; + } + + Self::insert_row_with_columns(tx, table_name, &insert_columns, values)?; + report.rows_imported += 1; + report.rows_inserted += 1; + } + Ok(()) + } + + fn parent_rows_exist( + tx: &rusqlite::Transaction<'_>, + table_name: &str, + table: &PortableTable, + row: &[PortableValue], + ) -> Result { + match table_name { + "node_embeddings" | "memory_access_log" => Self::portable_text(table, row, "node_id") + .map(|id| Self::node_exists(tx, id)) + .transpose() + .map(|v| v.unwrap_or(false)), + "fsrs_cards" | "memory_states" | "state_transitions" => { + Self::portable_text(table, row, "memory_id") + .map(|id| Self::node_exists(tx, id)) + .transpose() + .map(|v| v.unwrap_or(false)) + } + "memory_connections" => { + let source_exists = Self::portable_text(table, row, "source_id") + .map(|id| Self::node_exists(tx, id)) + .transpose()? + .unwrap_or(false); + let target_exists = Self::portable_text(table, row, "target_id") + .map(|id| Self::node_exists(tx, id)) + .transpose()? + .unwrap_or(false); + Ok(source_exists && target_exists) + } + _ => Ok(true), + } + } + + fn insert_or_replace_row( + tx: &rusqlite::Transaction<'_>, + table_name: &str, + table: &PortableTable, + row: &[PortableValue], + ) -> Result { + let key_exists = Self::merge_row_exists(tx, table_name, table, row)?; + let values = Self::row_values_for_columns(table, row, &table.columns)?; + Self::insert_row_with_columns(tx, table_name, &table.columns, values)?; + Ok(if key_exists { + MergeWrite::Updated + } else { + MergeWrite::Inserted + }) + } + + fn insert_row_with_columns( + tx: &rusqlite::Transaction<'_>, + table_name: &str, + columns: &[String], + values: Vec, + ) -> Result<()> { + let quoted_table = Self::quote_ident(table_name); + let quoted_columns = columns + .iter() + .map(|column| Self::quote_ident(column)) + .collect::>() + .join(", "); + let placeholders = std::iter::repeat_n("?", columns.len()) + .collect::>() + .join(", "); + let sql = format!( + "INSERT OR REPLACE INTO {} ({}) VALUES ({})", + quoted_table, quoted_columns, placeholders + ); + tx.execute(&sql, params_from_iter(values))?; + Ok(()) + } + + fn merge_row_exists( + tx: &rusqlite::Transaction<'_>, + table_name: &str, + table: &PortableTable, + row: &[PortableValue], + ) -> Result { + let key_columns: &[&str] = match table_name { + "knowledge_nodes" | "intentions" | "insights" | "sessions" => &["id"], + "node_embeddings" => &["node_id"], + "fsrs_cards" | "memory_states" => &["memory_id"], + "memory_connections" => &["source_id", "target_id"], + "fsrs_config" => &["key"], + "sync_tombstones" => &["table_name", "row_id"], + _ => &[], + }; + if key_columns.is_empty() { + return Ok(false); + } + let mut columns = Vec::new(); + for key in key_columns { + columns.push((*key).to_string()); + } + let values = Self::row_values_for_columns(table, row, &columns)?; + Self::row_exists_by_values(tx, table_name, &columns, &values) + } + + fn row_exists_by_values( + tx: &rusqlite::Transaction<'_>, + table_name: &str, + columns: &[String], + values: &[Value], + ) -> Result { + let quoted_table = Self::quote_ident(table_name); + let where_clause = columns + .iter() + .map(|column| format!("{} IS ?", Self::quote_ident(column))) + .collect::>() + .join(" AND "); + let sql = format!( + "SELECT COUNT(*) FROM {} WHERE {}", + quoted_table, where_clause + ); + let count: i64 = tx.query_row(&sql, params_from_iter(values.iter()), |row| row.get(0))?; + Ok(count > 0) + } + + fn row_values_for_columns( + table: &PortableTable, + row: &[PortableValue], + columns: &[String], + ) -> Result> { + columns + .iter() + .map(|column| { + Self::portable_value(table, row, column) + .ok_or_else(|| { + StorageError::Init(format!( + "Portable archive row in table '{}' is missing column '{}'", + table.name, column + )) + })? + .to_sql_value() + .map_err(|e| StorageError::Init(format!("Invalid portable value: {}", e))) + }) + .collect() + } + + fn portable_value<'a>( + table: &PortableTable, + row: &'a [PortableValue], + column: &str, + ) -> Option<&'a PortableValue> { + table + .columns + .iter() + .position(|name| name == column) + .and_then(|idx| row.get(idx)) + } + + fn portable_text<'a>( + table: &PortableTable, + row: &'a [PortableValue], + column: &str, + ) -> Option<&'a str> { + match Self::portable_value(table, row, column) { + Some(PortableValue::Text(value)) => Some(value.as_str()), + _ => None, + } + } + + fn portable_timestamp( + table: &PortableTable, + row: &[PortableValue], + column: &str, + ) -> Option> { + Self::portable_text(table, row, column).and_then(Self::parse_rfc3339_opt) + } + + fn parse_rfc3339_opt(value: &str) -> Option> { + DateTime::parse_from_rfc3339(value) + .map(|dt| dt.with_timezone(&Utc)) + .ok() + } + + fn tombstone_timestamp( + tx: &rusqlite::Transaction<'_>, + table_name: &str, + row_id: &str, + ) -> Result>> { + let deleted_at: Option = tx + .query_row( + "SELECT deleted_at FROM sync_tombstones WHERE table_name = ?1 AND row_id = ?2", + params![table_name, row_id], + |row| row.get(0), + ) + .optional()?; + Ok(deleted_at.as_deref().and_then(Self::parse_rfc3339_opt)) + } + + fn current_schema_version(conn: &Connection) -> Result { + let version: i64 = conn.query_row( + "SELECT COALESCE(MAX(version), 0) FROM schema_version", + [], + |row| row.get(0), + )?; + Ok(version as u32) + } + + fn ensure_portable_import_target_empty(conn: &Connection) -> Result<()> { + for table_name in PORTABLE_USER_DATA_TABLES { + if Self::table_exists(conn, table_name)? { + let count = Self::table_row_count(conn, table_name)?; + if count > 0 { + return Err(StorageError::Init(format!( + "Portable import requires an empty target database; table '{}' has {} rows", + table_name, count + ))); + } + } + } + Ok(()) + } + + fn table_exists(conn: &Connection, table_name: &str) -> Result { + let exists: i64 = conn.query_row( + "SELECT COUNT(*) FROM sqlite_master WHERE type IN ('table', 'view') AND name = ?1", + params![table_name], + |row| row.get(0), + )?; + Ok(exists > 0) + } + + fn table_row_count(conn: &Connection, table_name: &str) -> Result { + let sql = format!("SELECT COUNT(*) FROM {}", Self::quote_ident(table_name)); + Ok(conn.query_row(&sql, [], |row| row.get(0))?) + } + + fn table_columns(conn: &Connection, table_name: &str) -> Result> { + let sql = format!("PRAGMA table_info({})", Self::quote_ident(table_name)); + let mut stmt = conn.prepare(&sql)?; + let rows = stmt.query_map([], |row| row.get::<_, String>(1))?; + + let mut columns = Vec::new(); + for row in rows { + columns.push(row?); + } + Ok(columns) + } + + fn portable_value_from_ref(value: ValueRef<'_>) -> rusqlite::Result { + Ok(match value { + ValueRef::Null => PortableValue::Null, + ValueRef::Integer(value) => PortableValue::Integer(value), + ValueRef::Real(value) => PortableValue::Real(value), + ValueRef::Text(value) => PortableValue::Text( + std::str::from_utf8(value) + .map_err(|e| { + rusqlite::Error::FromSqlConversionFailure(0, Type::Text, Box::new(e)) + })? + .to_string(), + ), + ValueRef::Blob(value) => PortableValue::Blob(encode_hex(value)), + }) + } + + fn quote_ident(identifier: &str) -> String { + format!("\"{}\"", identifier.replace('"', "\"\"")) + } + + #[cfg(all(feature = "embeddings", feature = "vector-search"))] + fn embedding_model_matches_active(stored_model: &str, active_model: &str) -> bool { + if stored_model == active_model { + return true; + } + + let stored = stored_model.to_ascii_lowercase(); + let active = active_model.to_ascii_lowercase(); + + if active.contains("qwen3") { + return stored.contains("qwen3"); + } + + if active.contains("nomic-embed-text-v1.5") { + return stored.contains("nomic") && stored.contains("v1.5"); + } + + false + } + + #[cfg(all(feature = "embeddings", feature = "vector-search"))] + fn embedding_model_supports_matryoshka(model_name: &str) -> bool { + let model = model_name.to_ascii_lowercase(); + model.contains("nomic") || model.contains("qwen3") + } + + #[cfg(all(feature = "embeddings", feature = "vector-search"))] + fn embedding_vector_for_active_model( + embedding_bytes: &[u8], + stored_model: &str, + active_model: &str, + ) -> Option> { + if !Self::embedding_model_matches_active(stored_model, active_model) { + return None; + } + + let embedding = Embedding::from_bytes(embedding_bytes)?; + if embedding.dimensions == EMBEDDING_DIMENSIONS { + Some(embedding.vector) + } else if Self::embedding_model_supports_matryoshka(stored_model) { + Some(matryoshka_truncate(embedding.vector)) + } else { + None + } + } + + #[cfg(feature = "embeddings")] + fn active_embedding_model_like_pattern(active_model: &str) -> String { + let active = active_model.to_ascii_lowercase(); + if active.contains("qwen3") { + "%qwen3%".to_string() + } else if active.contains("nomic-embed-text-v1.5") { + "%nomic%v1.5%".to_string() + } else { + active_model.to_string() + } + } + // ======================================================================== // STATE TRANSITIONS (Audit Trail) // ======================================================================== @@ -4077,8 +5363,7 @@ impl Storage { pub fn gc_below_retention(&self, threshold: f64, min_age_days: i64) -> Result { let cutoff = (Utc::now() - Duration::days(min_age_days)).to_rfc3339(); - // Collect IDs first for vector index cleanup - #[cfg(all(feature = "embeddings", feature = "vector-search"))] + // Collect IDs first for sync tombstones and vector index cleanup. let doomed_ids: Vec = { let reader = self .reader @@ -4096,6 +5381,9 @@ impl Storage { .writer .lock() .map_err(|_| StorageError::Init("Writer lock poisoned".into()))?; + for id in &doomed_ids { + Self::record_sync_tombstone(&writer, "knowledge_nodes", id, "gc_below_retention")?; + } let deleted = writer.execute( "DELETE FROM knowledge_nodes WHERE retention_strength < ?1 AND created_at < ?2", params![threshold, cutoff], @@ -4329,6 +5617,45 @@ mod tests { Storage::new(Some(db_path)).unwrap() } + fn create_test_storage_at(dir: &tempfile::TempDir, name: &str) -> Storage { + Storage::new(Some(dir.path().join(name))).unwrap() + } + + #[cfg(all(feature = "embeddings", feature = "vector-search"))] + #[test] + fn test_embedding_model_family_matching() { + assert!(Storage::embedding_model_matches_active( + "nomic-embed-text-v1.5", + "nomic-ai/nomic-embed-text-v1.5", + )); + assert!(Storage::embedding_model_matches_active( + "Qwen/Qwen3-Embedding-0.6B", + "Qwen/Qwen3-Embedding-0.6B", + )); + assert!(!Storage::embedding_model_matches_active( + "nomic-ai/nomic-embed-text-v1.5", + "Qwen/Qwen3-Embedding-0.6B", + )); + + let bytes = Embedding::new(vec![1.0; EMBEDDING_DIMENSIONS]).to_bytes(); + assert!( + Storage::embedding_vector_for_active_model( + &bytes, + "nomic-ai/nomic-embed-text-v1.5", + "Qwen/Qwen3-Embedding-0.6B", + ) + .is_none() + ); + assert!( + Storage::embedding_vector_for_active_model( + &bytes, + "Qwen/Qwen3-Embedding-0.6B", + "Qwen/Qwen3-Embedding-0.6B", + ) + .is_some() + ); + } + #[test] fn test_storage_creation() { let storage = create_test_storage(); @@ -4469,6 +5796,320 @@ mod tests { assert_eq!(count_future, 0); } + #[test] + fn test_portable_archive_exact_round_trip() { + let source_dir = tempdir().unwrap(); + let target_dir = tempdir().unwrap(); + let source = create_test_storage_at(&source_dir, "source.db"); + + let first = source + .ingest(IngestInput { + content: "Portable archive alpha memory".to_string(), + node_type: "fact".to_string(), + tags: vec!["portable".to_string()], + source: Some("test".to_string()), + ..Default::default() + }) + .unwrap(); + let second = source + .ingest(IngestInput { + content: "Portable archive beta memory".to_string(), + node_type: "fact".to_string(), + ..Default::default() + }) + .unwrap(); + source.mark_reviewed(&first.id, Rating::Good).unwrap(); + source + .save_connection(&ConnectionRecord { + source_id: first.id.clone(), + target_id: second.id.clone(), + strength: 0.75, + link_type: "semantic".to_string(), + created_at: Utc::now(), + last_activated: Utc::now(), + activation_count: 1, + }) + .unwrap(); + + let archive = source.export_portable_archive().unwrap(); + assert_eq!(archive.archive_format, PORTABLE_ARCHIVE_FORMAT); + assert!(archive.total_rows() >= 3); + assert!( + archive + .tables + .iter() + .any(|table| table.name == "knowledge_nodes" && table.rows.len() == 2) + ); + + let target = create_test_storage_at(&target_dir, "target.db"); + let report = target + .import_portable_archive(&archive, PortableImportMode::EmptyOnly) + .unwrap(); + assert!(report.rows_imported >= 3); + assert!(report.fts_rebuilt); + + let restored = target.get_node(&first.id).unwrap().unwrap(); + assert_eq!(restored.id, first.id); + assert_eq!(restored.content, first.content); + assert_eq!(restored.tags, first.tags); + assert_eq!(restored.reps, 1); + + let connections = target.get_connections_for_memory(&first.id).unwrap(); + assert_eq!(connections.len(), 1); + assert_eq!(connections[0].target_id, second.id); + + let results = target.search("alpha", 10).unwrap(); + assert_eq!(results.len(), 1); + assert_eq!(results[0].id, first.id); + } + + #[test] + fn test_portable_import_rejects_non_empty_target() { + let source_dir = tempdir().unwrap(); + let target_dir = tempdir().unwrap(); + let source = create_test_storage_at(&source_dir, "source.db"); + source + .ingest(IngestInput { + content: "Source memory".to_string(), + node_type: "fact".to_string(), + ..Default::default() + }) + .unwrap(); + let archive = source.export_portable_archive().unwrap(); + + let target = create_test_storage_at(&target_dir, "target.db"); + target + .ingest(IngestInput { + content: "Existing target memory".to_string(), + node_type: "fact".to_string(), + ..Default::default() + }) + .unwrap(); + + let err = target + .import_portable_archive(&archive, PortableImportMode::EmptyOnly) + .unwrap_err(); + assert!( + err.to_string() + .contains("requires an empty target database") + ); + } + + #[test] + fn test_portable_import_rejects_unknown_mode() { + let source_dir = tempdir().unwrap(); + let target_dir = tempdir().unwrap(); + let source = create_test_storage_at(&source_dir, "source.db"); + source + .ingest(IngestInput { + content: "Source memory".to_string(), + node_type: "fact".to_string(), + ..Default::default() + }) + .unwrap(); + let mut archive = source.export_portable_archive().unwrap(); + archive.mode = "merge".to_string(); + + let target = create_test_storage_at(&target_dir, "target.db"); + let err = target + .import_portable_archive(&archive, PortableImportMode::EmptyOnly) + .unwrap_err(); + assert!( + err.to_string() + .contains("Unsupported portable archive mode") + ); + } + + #[test] + fn test_portable_import_rejects_malformed_table_list() { + let source_dir = tempdir().unwrap(); + let target_dir = tempdir().unwrap(); + let source = create_test_storage_at(&source_dir, "source.db"); + source + .ingest(IngestInput { + content: "Source memory".to_string(), + node_type: "fact".to_string(), + ..Default::default() + }) + .unwrap(); + + let mut duplicate_archive = source.export_portable_archive().unwrap(); + let duplicate_table = duplicate_archive + .tables + .iter() + .find(|table| table.name == "knowledge_nodes") + .unwrap() + .clone(); + duplicate_archive.tables.push(duplicate_table); + + let target = create_test_storage_at(&target_dir, "target-duplicate.db"); + let err = target + .import_portable_archive(&duplicate_archive, PortableImportMode::EmptyOnly) + .unwrap_err(); + assert!( + err.to_string() + .contains("Portable archive contains duplicate table") + ); + + let mut unknown_archive = source.export_portable_archive().unwrap(); + unknown_archive.tables.push(PortableTable { + name: "sqlite_sequence".to_string(), + columns: vec!["name".to_string(), "seq".to_string()], + rows: vec![], + }); + + let target = create_test_storage_at(&target_dir, "target-unknown.db"); + let err = target + .import_portable_archive(&unknown_archive, PortableImportMode::EmptyOnly) + .unwrap_err(); + assert!( + err.to_string() + .contains("Portable archive contains unsupported table") + ); + } + + #[test] + fn test_portable_merge_import_combines_non_empty_databases() { + let source_dir = tempdir().unwrap(); + let target_dir = tempdir().unwrap(); + let source = create_test_storage_at(&source_dir, "source.db"); + let target = create_test_storage_at(&target_dir, "target.db"); + + let source_node = source + .ingest(IngestInput { + content: "Source sync memory".to_string(), + node_type: "fact".to_string(), + tags: vec!["sync".to_string()], + ..Default::default() + }) + .unwrap(); + let target_node = target + .ingest(IngestInput { + content: "Target local memory".to_string(), + node_type: "fact".to_string(), + tags: vec!["local".to_string()], + ..Default::default() + }) + .unwrap(); + + let archive = source.export_portable_archive().unwrap(); + let report = target + .import_portable_archive(&archive, PortableImportMode::Merge) + .unwrap(); + + assert!(report.rows_inserted > 0); + assert!(target.get_node(&source_node.id).unwrap().is_some()); + assert!(target.get_node(&target_node.id).unwrap().is_some()); + } + + #[test] + fn test_portable_merge_import_keeps_newer_local_memory() { + let source_dir = tempdir().unwrap(); + let target_dir = tempdir().unwrap(); + let source = create_test_storage_at(&source_dir, "source.db"); + let target = create_test_storage_at(&target_dir, "target.db"); + + let node = source + .ingest(IngestInput { + content: "Original shared memory".to_string(), + node_type: "fact".to_string(), + ..Default::default() + }) + .unwrap(); + let archive = source.export_portable_archive().unwrap(); + target + .import_portable_archive(&archive, PortableImportMode::EmptyOnly) + .unwrap(); + + let newer = (Utc::now() + Duration::hours(1)).to_rfc3339(); + { + let writer = target.writer.lock().unwrap(); + writer + .execute( + "UPDATE knowledge_nodes SET content = ?1, updated_at = ?2 WHERE id = ?3", + params!["Newer local edit", newer, &node.id], + ) + .unwrap(); + } + + let report = target + .import_portable_archive(&archive, PortableImportMode::Merge) + .unwrap(); + + assert!(report.conflicts_kept_local >= 1); + let restored = target.get_node(&node.id).unwrap().unwrap(); + assert_eq!(restored.content, "Newer local edit"); + } + + #[test] + fn test_portable_merge_import_applies_delete_tombstones() { + let source_dir = tempdir().unwrap(); + let target_dir = tempdir().unwrap(); + let source = create_test_storage_at(&source_dir, "source.db"); + let target = create_test_storage_at(&target_dir, "target.db"); + + let node = source + .ingest(IngestInput { + content: "Memory deleted on source".to_string(), + node_type: "fact".to_string(), + ..Default::default() + }) + .unwrap(); + let archive = source.export_portable_archive().unwrap(); + target + .import_portable_archive(&archive, PortableImportMode::EmptyOnly) + .unwrap(); + assert!(target.get_node(&node.id).unwrap().is_some()); + + source.delete_node(&node.id).unwrap(); + let delete_archive = source.export_portable_archive().unwrap(); + let report = target + .import_portable_archive(&delete_archive, PortableImportMode::Merge) + .unwrap(); + + assert!(report.rows_deleted >= 1); + assert!(target.get_node(&node.id).unwrap().is_none()); + } + + #[test] + fn test_file_portable_sync_round_trips_between_devices() { + let sync_dir = tempdir().unwrap(); + let first_dir = tempdir().unwrap(); + let second_dir = tempdir().unwrap(); + let sync_path = sync_dir.path().join("vestige-sync.json"); + let first = create_test_storage_at(&first_dir, "first.db"); + let second = create_test_storage_at(&second_dir, "second.db"); + + let first_node = first + .ingest(IngestInput { + content: "First device memory".to_string(), + node_type: "fact".to_string(), + tags: vec!["sync".to_string()], + ..Default::default() + }) + .unwrap(); + let first_push = first.sync_portable_archive_file(&sync_path).unwrap(); + assert!(!first_push.pulled); + assert!(sync_path.exists()); + + let second_node = second + .ingest(IngestInput { + content: "Second device memory".to_string(), + node_type: "fact".to_string(), + tags: vec!["sync".to_string()], + ..Default::default() + }) + .unwrap(); + let second_sync = second.sync_portable_archive_file(&sync_path).unwrap(); + assert!(second_sync.pulled); + assert!(second.get_node(&first_node.id).unwrap().is_some()); + + let first_sync = first.sync_portable_archive_file(&sync_path).unwrap(); + assert!(first_sync.pulled); + assert!(first.get_node(&second_node.id).unwrap().is_some()); + assert!(first_sync.pushed_rows >= 2); + } + #[test] fn test_get_last_backup_timestamp_no_panic() { // Static method should not panic even if no backups exist diff --git a/crates/vestige-mcp/Cargo.toml b/crates/vestige-mcp/Cargo.toml index 3916203..af6a663 100644 --- a/crates/vestige-mcp/Cargo.toml +++ b/crates/vestige-mcp/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "vestige-mcp" -version = "2.1.0" +version = "2.1.1" edition = "2024" description = "Cognitive memory MCP server for Claude - FSRS-6, spreading activation, synaptic tagging, 3D dashboard, and 130 years of memory research" authors = ["samvallad33"] @@ -21,6 +21,9 @@ ort-download = ["embeddings", "vestige-core/ort-download"] # Usage: cargo build --no-default-features --features ort-dynamic,vector-search # Runtime: export ORT_DYLIB_PATH=$(brew --prefix onnxruntime)/lib/libonnxruntime.dylib ort-dynamic = ["embeddings", "vestige-core/ort-dynamic"] +qwen3-embeddings = ["embeddings", "vestige-core/qwen3-embeddings"] +qwen3-reranker = ["qwen3-embeddings"] +metal = ["embeddings", "vestige-core/metal"] [[bin]] name = "vestige-mcp" @@ -44,7 +47,7 @@ path = "src/bin/cli.rs" # Only `bundled-sqlite` is always on. `embeddings` and `vector-search` are # toggled via vestige-mcp's own feature flags below so `--no-default-features` # actually works (previously hardcoded here, which silently defeated the flag). -vestige-core = { version = "2.1.0", path = "../vestige-core", default-features = false, features = ["bundled-sqlite"] } +vestige-core = { version = "2.1.1", path = "../vestige-core", default-features = false, features = ["bundled-sqlite"] } # ============================================================================ # MCP Server Dependencies diff --git a/crates/vestige-mcp/src/bin/cli.rs b/crates/vestige-mcp/src/bin/cli.rs index bc1f8ae..c8e5328 100644 --- a/crates/vestige-mcp/src/bin/cli.rs +++ b/crates/vestige-mcp/src/bin/cli.rs @@ -8,14 +8,13 @@ use std::io::{BufWriter, Write}; use std::path::Path; use std::path::PathBuf; use std::process::Command; -use std::sync::Arc; +use std::sync::{Arc, OnceLock}; use anyhow::Context; use chrono::{NaiveDate, Utc}; use clap::{Parser, Subcommand}; use colored::Colorize; -use directories::ProjectDirs; -use vestige_core::{IngestInput, Storage}; +use vestige_core::{IngestInput, PortableImportMode, Storage}; /// Vestige - Cognitive Memory System CLI #[derive(Parser)] @@ -27,10 +26,16 @@ use vestige_core::{IngestInput, Storage}; long_about = "Vestige is a cognitive memory system based on 130 years of memory research.\n\nIt implements FSRS-6, spreading activation, synaptic tagging, and more." )] struct Cli { + /// Use a specific Vestige data directory for this command. + #[arg(long, global = true, value_name = "DIR")] + data_dir: Option, + #[command(subcommand)] command: Commands, } +static CLI_DB_PATH: OnceLock = OnceLock::new(); + #[derive(Subcommand)] enum Commands { /// Show memory statistics @@ -52,7 +57,7 @@ enum Commands { /// Update Vestige binaries from the latest GitHub release Update { - /// Install a specific release tag instead of latest (example: v2.1.0) + /// Install a specific release tag instead of latest (example: v2.1.1) #[arg(long)] version: Option, @@ -92,6 +97,27 @@ enum Commands { since: Option, }, + /// Export an exact portable archive for Vestige-to-Vestige transfer + PortableExport { + /// Output archive path + output: PathBuf, + }, + + /// Import an exact portable archive + PortableImport { + /// Input archive path + input: PathBuf, + /// Merge into the current database instead of requiring an empty target + #[arg(long)] + merge: bool, + }, + + /// Two-way sync with a file-backed portable archive + Sync { + /// Sync archive path, often in Dropbox/iCloud/Syncthing/Git + archive: PathBuf, + }, + /// Garbage collect stale memories below retention threshold Gc { /// Minimum retention strength to keep (delete below this) @@ -150,6 +176,13 @@ enum Commands { fn main() -> anyhow::Result<()> { let cli = Cli::parse(); + if let Some(data_dir) = cli.data_dir { + let db_path = Storage::db_path_for_data_dir(data_dir)?; + CLI_DB_PATH + .set(db_path) + .map_err(|_| anyhow::anyhow!("data directory was initialized more than once"))?; + } + match cli.command { Commands::Stats { tagging, states } => run_stats(tagging, states), Commands::Health => run_health(), @@ -167,6 +200,9 @@ fn main() -> anyhow::Result<()> { tags, since, } => run_export(output, format, tags, since), + Commands::PortableExport { output } => run_portable_export(output), + Commands::PortableImport { input, merge } => run_portable_import(input, merge), + Commands::Sync { archive } => run_sync(archive), Commands::Gc { min_retention, max_age_days, @@ -465,7 +501,7 @@ fn run_update( /// Run stats command fn run_stats(show_tagging: bool, show_states: bool) -> anyhow::Result<()> { - let storage = Storage::new(None)?; + let storage = open_storage()?; let stats = storage.get_stats()?; println!("{}", "=== Vestige Memory Statistics ===".cyan().bold()); @@ -499,8 +535,20 @@ fn run_stats(show_tagging: bool, show_states: bool) -> anyhow::Result<()> { stats.nodes_with_embeddings ); + if let Some(model) = &stats.active_embedding_model { + println!("{}: {}", "Active Embedding Model".white().bold(), model); + } + if let Some(model) = &stats.embedding_model { - println!("{}: {}", "Embedding Model".white().bold(), model); + println!("{}: {}", "Stored Embedding Model".white().bold(), model); + } + + if stats.nodes_with_mismatched_embeddings > 0 { + println!( + "{}: {}", + "Mismatched Embeddings".white().bold(), + stats.nodes_with_mismatched_embeddings + ); } if let Some(oldest) = stats.oldest_memory { @@ -520,13 +568,13 @@ fn run_stats(show_tagging: bool, show_states: bool) -> anyhow::Result<()> { // Embedding coverage let embedding_coverage = if stats.total_nodes > 0 { - (stats.nodes_with_embeddings as f64 / stats.total_nodes as f64) * 100.0 + (stats.nodes_with_active_embeddings as f64 / stats.total_nodes as f64) * 100.0 } else { 0.0 }; println!( "{}: {:.1}%", - "Embedding Coverage".white().bold(), + "Active Embedding Coverage".white().bold(), embedding_coverage ); @@ -651,7 +699,7 @@ fn print_distribution_bar(label: &str, count: usize, total: usize, color: &str) /// Run health check fn run_health() -> anyhow::Result<()> { - let storage = Storage::new(None)?; + let storage = open_storage()?; let stats = storage.get_stats()?; println!("{}", "=== Vestige Health Check ===".cyan().bold()); @@ -690,13 +738,13 @@ fn run_health() -> anyhow::Result<()> { // Embedding coverage let embedding_coverage = if stats.total_nodes > 0 { - (stats.nodes_with_embeddings as f64 / stats.total_nodes as f64) * 100.0 + (stats.nodes_with_active_embeddings as f64 / stats.total_nodes as f64) * 100.0 } else { 0.0 }; println!( "{}: {:.1}%", - "Embedding Coverage".white(), + "Active Embedding Coverage".white(), embedding_coverage ); println!( @@ -721,14 +769,18 @@ fn run_health() -> anyhow::Result<()> { warnings.push("Many memories are due for review"); } - if stats.total_nodes > 0 && stats.nodes_with_embeddings == 0 { - warnings.push("No embeddings generated - semantic search unavailable"); + if stats.total_nodes > 0 && stats.nodes_with_active_embeddings == 0 { + warnings.push("No active-model embeddings generated - semantic search unavailable"); } if embedding_coverage < 50.0 && stats.total_nodes > 10 { warnings.push("Low embedding coverage - run consolidation to improve semantic search"); } + if stats.nodes_with_mismatched_embeddings > 0 { + warnings.push("Stored embeddings from another model are present - run consolidation after changing embedding models"); + } + if !warnings.is_empty() { println!(); println!("{}", "Warnings:".yellow().bold()); @@ -749,9 +801,9 @@ fn run_health() -> anyhow::Result<()> { recommendations.push("Review due memories to strengthen retention."); } - if stats.nodes_with_embeddings < stats.total_nodes { + if stats.nodes_with_active_embeddings < stats.total_nodes { recommendations - .push("Run 'vestige consolidate' to generate embeddings for better semantic search."); + .push("Run 'vestige consolidate' to generate active-model embeddings for better semantic search."); } if stats.total_nodes > 100 && stats.average_retention < 0.7 { @@ -788,7 +840,7 @@ fn run_consolidate() -> anyhow::Result<()> { println!("Running memory consolidation cycle..."); println!(); - let storage = Storage::new(None)?; + let storage = open_storage()?; let result = storage.run_consolidation()?; println!( @@ -834,7 +886,49 @@ fn run_restore(backup_path: PathBuf) -> anyhow::Result<()> { println!("Loading backup from: {}", backup_path.display()); // Read and parse backup - let backup_content = std::fs::read_to_string(&backup_path)?; + let backup_bytes = std::fs::read(&backup_path)?; + if backup_bytes.starts_with(b"SQLite format 3\0") { + anyhow::bail!( + "{} is a raw SQLite database backup, not a JSON restore file. Use portable-export/portable-import for cross-device transfer, or replace the database file manually while Vestige is stopped.", + backup_path.display() + ); + } + let backup_content = String::from_utf8(backup_bytes) + .with_context(|| format!("{} is not UTF-8 JSON", backup_path.display()))?; + + if let Ok(archive) = serde_json::from_str::(&backup_content) + && archive.archive_format == vestige_core::PORTABLE_ARCHIVE_FORMAT + { + println!("Detected portable archive."); + println!("{}: {}", "Format".white().bold(), archive.archive_format); + println!("{}: {}", "Schema".white().bold(), archive.schema_version); + println!("{}: {}", "Tables".white().bold(), archive.tables.len()); + println!("{}: {}", "Rows".white().bold(), archive.total_rows()); + println!(); + + let storage = open_storage()?; + let report = storage.import_portable_archive(&archive, PortableImportMode::EmptyOnly)?; + + println!( + "{}: {}", + "Tables imported".white().bold(), + report.tables_imported + ); + println!( + "{}: {}", + "Rows imported".white().bold(), + report.rows_imported + ); + println!( + "{}: {}", + "Tables skipped".white().bold(), + report.tables_skipped + ); + println!("{}: {}", "FTS rebuilt".white().bold(), report.fts_rebuilt); + println!(); + println!("{}", "Portable restore complete.".green().bold()); + return Ok(()); + } #[derive(serde::Deserialize)] struct BackupWrapper { @@ -857,16 +951,27 @@ fn run_restore(backup_path: PathBuf) -> anyhow::Result<()> { source: Option, } - let wrapper: Vec = serde_json::from_str(&backup_content)?; - let recall_result: RecallResult = serde_json::from_str(&wrapper[0].text)?; - let memories = recall_result.results; + let memories = if let Ok(wrapper) = serde_json::from_str::>(&backup_content) + { + let first = wrapper.first().context("backup wrapper is empty")?; + let recall_result: RecallResult = serde_json::from_str(&first.text)?; + recall_result.results + } else if let Ok(recall_result) = serde_json::from_str::(&backup_content) { + recall_result.results + } else if let Ok(memories) = serde_json::from_str::>(&backup_content) { + memories + } else { + anyhow::bail!( + "Unrecognized backup format. Expected portable archive, MCP wrapper, RecallResult, or array of memories." + ); + }; println!("Found {} memories to restore", memories.len()); println!(); // Initialize storage println!("Initializing storage..."); - let storage = Storage::new(None)?; + let storage = open_storage()?; println!("Generating embeddings and ingesting memories..."); println!(); @@ -916,8 +1021,8 @@ fn run_restore(backup_path: PathBuf) -> anyhow::Result<()> { println!("{}: {}", "Total Nodes".white(), stats.total_nodes); println!( "{}: {}", - "With Embeddings".white(), - stats.nodes_with_embeddings + "Active Embeddings".white(), + stats.nodes_with_active_embeddings ); Ok(()) @@ -925,9 +1030,20 @@ fn run_restore(backup_path: PathBuf) -> anyhow::Result<()> { /// Get the default database path fn get_default_db_path() -> anyhow::Result { - let proj_dirs = ProjectDirs::from("com", "vestige", "core") - .ok_or_else(|| anyhow::anyhow!("Could not determine project directories"))?; - Ok(proj_dirs.data_dir().join("vestige.db")) + if let Some(path) = CLI_DB_PATH.get() { + Ok(path.clone()) + } else { + Ok(Storage::default_db_path()?) + } +} + +/// Open storage using the CLI-selected data directory, if one was provided. +fn open_storage() -> anyhow::Result { + if let Some(path) = CLI_DB_PATH.get() { + Ok(Storage::new(Some(path.clone()))?) + } else { + Ok(Storage::new(None)?) + } } /// Fetch all nodes from storage using pagination @@ -963,7 +1079,7 @@ fn run_backup(output: PathBuf) -> anyhow::Result<()> { // Open storage to flush WAL before copying println!("Flushing WAL checkpoint..."); { - let storage = Storage::new(None)?; + let storage = open_storage()?; // get_stats triggers a read so the connection is active, then drop flushes let _ = storage.get_stats()?; } @@ -1050,7 +1166,7 @@ fn run_export( }) .unwrap_or_default(); - let storage = Storage::new(None)?; + let storage = open_storage()?; let all_nodes = fetch_all_nodes(&storage)?; // Apply filters @@ -1141,6 +1257,145 @@ fn run_export( Ok(()) } +/// Run exact portable archive export. +fn run_portable_export(output: PathBuf) -> anyhow::Result<()> { + println!("{}", "=== Vestige Portable Export ===".cyan().bold()); + println!(); + + if let Some(parent) = output.parent() + && !parent.exists() + { + std::fs::create_dir_all(parent)?; + } + + let storage = open_storage()?; + let archive = storage.export_portable_archive_to_path(&output)?; + + let file_size = std::fs::metadata(&output)?.len(); + let size_display = if file_size >= 1024 * 1024 { + format!("{:.2} MB", file_size as f64 / (1024.0 * 1024.0)) + } else if file_size >= 1024 { + format!("{:.1} KB", file_size as f64 / 1024.0) + } else { + format!("{} bytes", file_size) + }; + + println!("{}: {}", "Archive".white().bold(), output.display()); + println!("{}: {}", "Format".white().bold(), archive.archive_format); + println!("{}: {}", "Schema".white().bold(), archive.schema_version); + println!("{}: {}", "Tables".white().bold(), archive.tables.len()); + println!("{}: {}", "Rows".white().bold(), archive.total_rows()); + println!(); + println!( + "{}", + format!( + "Portable export complete: {} ({})", + output.display(), + size_display + ) + .green() + .bold() + ); + + Ok(()) +} + +/// Run exact portable archive import. +fn run_portable_import(input: PathBuf, merge: bool) -> anyhow::Result<()> { + println!("{}", "=== Vestige Portable Import ===".cyan().bold()); + println!(); + println!("{}: {}", "Archive".white().bold(), input.display()); + let mode = if merge { + PortableImportMode::Merge + } else { + PortableImportMode::EmptyOnly + }; + println!( + "{}", + if merge { + "Mode: merge into existing database".yellow() + } else { + "Mode: empty database only".yellow() + } + ); + println!(); + + let storage = open_storage()?; + let report = storage.import_portable_archive_from_path(&input, mode)?; + + println!( + "{}: {}", + "Tables imported".white().bold(), + report.tables_imported + ); + println!( + "{}: {}", + "Rows imported".white().bold(), + report.rows_imported + ); + println!( + "{}: {}", + "Tables skipped".white().bold(), + report.tables_skipped + ); + println!("{}: {}", "FTS rebuilt".white().bold(), report.fts_rebuilt); + if merge { + println!( + "{}: {} inserted, {} updated, {} deleted, {} skipped, {} kept local", + "Merge".white().bold(), + report.rows_inserted, + report.rows_updated, + report.rows_deleted, + report.rows_skipped, + report.conflicts_kept_local + ); + } + println!(); + println!("{}", "Portable import complete.".green().bold()); + + Ok(()) +} + +/// Run file-backed two-way sync. +fn run_sync(archive: PathBuf) -> anyhow::Result<()> { + println!("{}", "=== Vestige File Sync ===".cyan().bold()); + println!(); + println!("{}: {}", "Archive".white().bold(), archive.display()); + + let storage = open_storage()?; + let report = storage.sync_portable_archive_file(&archive)?; + + if let Some(pull) = &report.pull { + println!("{}", "Pull: merged remote archive".yellow()); + println!( + " {} inserted, {} updated, {} deleted, {} skipped, {} kept local", + pull.rows_inserted, + pull.rows_updated, + pull.rows_deleted, + pull.rows_skipped, + pull.conflicts_kept_local + ); + } else { + println!( + "{}", + "Pull: archive does not exist yet; creating it".yellow() + ); + } + + println!("{}", "Push: wrote merged local state".yellow()); + println!( + "{}", + format!( + "Sync complete: {} tables, {} rows", + report.pushed_tables, report.pushed_rows + ) + .green() + .bold() + ); + + Ok(()) +} + /// Run garbage collection command fn run_gc( min_retention: f64, @@ -1151,7 +1406,7 @@ fn run_gc( println!("{}", "=== Vestige Garbage Collection ===".cyan().bold()); println!(); - let storage = Storage::new(None)?; + let storage = open_storage()?; let all_nodes = fetch_all_nodes(&storage)?; let now = Utc::now(); @@ -1327,7 +1582,7 @@ fn run_ingest( valid_until: None, }; - let storage = Storage::new(None)?; + let storage = open_storage()?; // Try smart_ingest (PE Gating) if available, otherwise regular ingest #[cfg(all(feature = "embeddings", feature = "vector-search"))] @@ -1381,7 +1636,7 @@ fn run_dashboard(port: u16, open_browser: bool) -> anyhow::Result<()> { format!("http://127.0.0.1:{}", port).cyan() ); - let storage = Storage::new(None)?; + let storage = open_storage()?; // Try to initialize embeddings for search support #[cfg(feature = "embeddings")] @@ -1412,7 +1667,7 @@ fn run_serve(port: u16, with_dashboard: bool, dashboard_port: u16) -> anyhow::Re println!("{}", "=== Vestige HTTP Server ===".cyan().bold()); println!(); - let storage = Storage::new(None)?; + let storage = open_storage()?; #[cfg(feature = "embeddings")] { diff --git a/crates/vestige-mcp/src/cognitive.rs b/crates/vestige-mcp/src/cognitive.rs index 86aadcf..3d106ff 100644 --- a/crates/vestige-mcp/src/cognitive.rs +++ b/crates/vestige-mcp/src/cognitive.rs @@ -6,6 +6,7 @@ use vestige_core::neuroscience::predictive_retrieval::PredictiveMemory; use vestige_core::neuroscience::prospective_memory::{IntentionParser, ProspectiveMemory}; +#[cfg(feature = "vector-search")] use vestige_core::search::TemporalSearcher; use vestige_core::{ AccessibilityCalculator, @@ -31,9 +32,6 @@ use vestige_core::{ MemoryDreamer, NoveltySignal, ReconsolidationManager, - // Search modules - Reranker, - RerankerConfig, RewardSignal, SpeculativeRetriever, StateUpdateService, @@ -41,6 +39,8 @@ use vestige_core::{ Storage, SynapticTaggingSystem, }; +#[cfg(feature = "vector-search")] +use vestige_core::{Reranker, RerankerConfig}; /// Stateful cognitive engine holding all neuroscience modules. /// @@ -80,7 +80,9 @@ pub struct CognitiveEngine { pub consolidation_scheduler: ConsolidationScheduler, // -- Search -- + #[cfg(feature = "vector-search")] pub reranker: Reranker, + #[cfg(feature = "vector-search")] pub temporal_searcher: TemporalSearcher, } @@ -161,7 +163,9 @@ impl CognitiveEngine { consolidation_scheduler: ConsolidationScheduler::new(), // Search + #[cfg(feature = "vector-search")] reranker: Reranker::new(RerankerConfig::default()), + #[cfg(feature = "vector-search")] temporal_searcher: TemporalSearcher::new(), } } diff --git a/crates/vestige-mcp/src/main.rs b/crates/vestige-mcp/src/main.rs index 9aa59d0..27d37c7 100644 --- a/crates/vestige-mcp/src/main.rs +++ b/crates/vestige-mcp/src/main.rs @@ -467,7 +467,7 @@ async fn main() { } // Load cross-encoder reranker in the background (downloads ~150MB on first run) - #[cfg(feature = "embeddings")] + #[cfg(feature = "vector-search")] { let cog_clone = Arc::clone(&cognitive); tokio::spawn(async move { diff --git a/crates/vestige-mcp/src/tools/cross_reference.rs b/crates/vestige-mcp/src/tools/cross_reference.rs index 531e934..de58994 100644 --- a/crates/vestige-mcp/src/tools/cross_reference.rs +++ b/crates/vestige-mcp/src/tools/cross_reference.rs @@ -513,6 +513,7 @@ pub async fn execute( } let mut ranked = results; + #[cfg(feature = "vector-search")] if let Ok(mut cog) = cognitive.try_lock() { let candidates: Vec<_> = ranked .iter() diff --git a/crates/vestige-mcp/src/tools/dedup.rs b/crates/vestige-mcp/src/tools/dedup.rs index 8456629..ec93721 100644 --- a/crates/vestige-mcp/src/tools/dedup.rs +++ b/crates/vestige-mcp/src/tools/dedup.rs @@ -4,8 +4,10 @@ //! cosine similarity on stored embeddings. Uses union-find for //! efficient clustering. +#[cfg(all(feature = "embeddings", feature = "vector-search"))] use serde::Deserialize; use serde_json::Value; +#[cfg(all(feature = "embeddings", feature = "vector-search"))] use std::collections::HashMap; use std::sync::Arc; @@ -43,6 +45,7 @@ pub fn schema() -> Value { #[derive(Debug, Deserialize)] #[serde(rename_all = "camelCase")] +#[cfg(all(feature = "embeddings", feature = "vector-search"))] struct DedupArgs { #[serde(alias = "similarity_threshold")] similarity_threshold: Option, @@ -51,11 +54,13 @@ struct DedupArgs { } /// Simple union-find for clustering +#[cfg(all(feature = "embeddings", feature = "vector-search"))] struct UnionFind { parent: Vec, rank: Vec, } +#[cfg(all(feature = "embeddings", feature = "vector-search"))] impl UnionFind { fn new(n: usize) -> Self { Self { @@ -89,21 +94,22 @@ impl UnionFind { } pub async fn execute(storage: &Arc, args: Option) -> Result { - let args: DedupArgs = match args { - Some(v) => serde_json::from_value(v).map_err(|e| format!("Invalid arguments: {}", e))?, - None => DedupArgs { - similarity_threshold: None, - limit: None, - tags: None, - }, - }; - - let threshold = args.similarity_threshold.unwrap_or(0.80) as f32; - let limit = args.limit.unwrap_or(20); - let tag_filter = args.tags.unwrap_or_default(); - #[cfg(all(feature = "embeddings", feature = "vector-search"))] { + let args: DedupArgs = match args { + Some(v) => { + serde_json::from_value(v).map_err(|e| format!("Invalid arguments: {}", e))? + } + None => DedupArgs { + similarity_threshold: None, + limit: None, + tags: None, + }, + }; + let threshold = args.similarity_threshold.unwrap_or(0.80) as f32; + let limit = args.limit.unwrap_or(20); + let tag_filter = args.tags.unwrap_or_default(); + // Load all embeddings let all_embeddings = storage .get_all_embeddings() @@ -261,6 +267,8 @@ pub async fn execute(storage: &Arc, args: Option) -> Result Value { "properties": { "format": { "type": "string", - "description": "Export format: 'json' (default) or 'jsonl'", - "enum": ["json", "jsonl"], + "description": "Export format: 'json' (default), 'jsonl', or 'portable' for exact Vestige-to-Vestige transfer", + "enum": ["json", "jsonl", "portable"], "default": "json" }, "tags": { @@ -51,7 +51,7 @@ pub fn export_schema() -> Value { }, "path": { "type": "string", - "description": "Custom filename (not path). File is saved in ~/.vestige/exports/. Default: memories-{timestamp}.{format}" + "description": "Custom filename (not path). File is saved in the active Vestige data directory's exports/ folder. Default: memories-{timestamp}.{format}" } } }) @@ -117,7 +117,7 @@ pub async fn execute_system_status( }; let embedding_coverage = if stats.total_nodes > 0 { - (stats.nodes_with_embeddings as f64 / stats.total_nodes as f64) * 100.0 + (stats.nodes_with_active_embeddings as f64 / stats.total_nodes as f64) * 100.0 } else { 0.0 }; @@ -131,12 +131,17 @@ pub async fn execute_system_status( if stats.nodes_due_for_review > 10 { warnings.push("Many memories are due for review"); } - if stats.total_nodes > 0 && stats.nodes_with_embeddings == 0 { - warnings.push("No embeddings generated - semantic search unavailable"); + if stats.total_nodes > 0 && stats.nodes_with_active_embeddings == 0 { + warnings.push("No active-model embeddings generated - semantic search unavailable"); } if embedding_coverage < 50.0 && stats.total_nodes > 10 { warnings.push("Low embedding coverage - run consolidate to improve semantic search"); } + if stats.nodes_with_mismatched_embeddings > 0 { + warnings.push( + "Stored embeddings from another model are present - run consolidate after changing embedding models", + ); + } let mut recommendations = Vec::new(); if status == "critical" { @@ -146,8 +151,8 @@ pub async fn execute_system_status( if stats.nodes_due_for_review > 5 { recommendations.push("Review due memories to strengthen retention."); } - if stats.nodes_with_embeddings < stats.total_nodes { - recommendations.push("Run 'consolidate' to generate missing embeddings."); + if stats.nodes_with_active_embeddings < stats.total_nodes { + recommendations.push("Run 'consolidate' to generate active-model embeddings."); } if stats.total_nodes > 100 && stats.average_retention < 0.7 { recommendations.push("Consider running periodic consolidation."); @@ -233,7 +238,7 @@ pub async fn execute_system_status( Some(dt) => storage.count_memories_since(*dt).unwrap_or(0), None => stats.total_nodes, }; - let last_backup = Storage::get_last_backup_timestamp(); + let last_backup = storage.last_backup_timestamp(); Ok(serde_json::json!({ "tool": "system_status", @@ -249,8 +254,11 @@ pub async fn execute_system_status( "averageStorageStrength": stats.average_storage_strength, "averageRetrievalStrength": stats.average_retrieval_strength, "withEmbeddings": stats.nodes_with_embeddings, + "withActiveEmbeddings": stats.nodes_with_active_embeddings, + "mismatchedEmbeddings": stats.nodes_with_mismatched_embeddings, "embeddingCoverage": format!("{:.1}%", embedding_coverage), "embeddingModel": stats.embedding_model, + "activeEmbeddingModel": stats.active_embedding_model, "oldestMemory": stats.oldest_memory.map(|dt| dt.to_rfc3339()), "newestMemory": stats.newest_memory.map(|dt| dt.to_rfc3339()), // Distribution @@ -299,13 +307,7 @@ pub async fn execute_consolidate( /// Backup tool pub async fn execute_backup(storage: &Arc, _args: Option) -> Result { // Determine backup path - let vestige_dir = directories::ProjectDirs::from("com", "vestige", "core") - .ok_or("Could not determine data directory")?; - let backup_dir = vestige_dir - .data_dir() - .parent() - .unwrap_or(vestige_dir.data_dir()) - .join("backups"); + let backup_dir = storage.sidecar_dir("backups"); std::fs::create_dir_all(&backup_dir) .map_err(|e| format!("Failed to create backup directory: {}", e))?; @@ -354,13 +356,60 @@ pub async fn execute_export(storage: &Arc, args: Option) -> Resu }; let format = args.format.unwrap_or_else(|| "json".to_string()); - if format != "json" && format != "jsonl" { + if format != "json" && format != "jsonl" && format != "portable" { return Err(format!( - "Invalid format '{}'. Must be 'json' or 'jsonl'.", + "Invalid format '{}'. Must be 'json', 'jsonl', or 'portable'.", format )); } + if format == "portable" { + if args.tags.as_ref().is_some_and(|tags| !tags.is_empty()) || args.since.is_some() { + return Err( + "Portable export is exact and does not support tags or since filters.".to_string(), + ); + } + + let export_dir = storage.sidecar_dir("exports"); + std::fs::create_dir_all(&export_dir) + .map_err(|e| format!("Failed to create export directory: {}", e))?; + + let export_path = match args.path { + Some(ref p) => { + let filename = std::path::Path::new(p) + .file_name() + .ok_or("Invalid export filename: must be a simple filename, not a path")?; + let name_str = filename.to_str().ok_or("Invalid filename encoding")?; + if name_str.contains("..") { + return Err("Invalid export filename: '..' not allowed".to_string()); + } + export_dir.join(filename) + } + None => { + let timestamp = Utc::now().format("%Y%m%d-%H%M%S"); + export_dir.join(format!("vestige-portable-{}.json", timestamp)) + } + }; + + let archive = storage + .export_portable_archive_to_path(&export_path) + .map_err(|e| e.to_string())?; + let file_size = std::fs::metadata(&export_path) + .map(|m| m.len()) + .unwrap_or(0); + + return Ok(serde_json::json!({ + "tool": "export", + "path": export_path.display().to_string(), + "format": "portable", + "archiveFormat": archive.archive_format, + "schemaVersion": archive.schema_version, + "tablesExported": archive.tables.len(), + "rowsExported": archive.total_rows(), + "sizeBytes": file_size, + })); + } + // Parse since date let since_date = match &args.since { Some(date_str) => { @@ -412,13 +461,7 @@ pub async fn execute_export(storage: &Arc, args: Option) -> Resu .collect(); // Determine export path — always constrained to vestige exports directory - let vestige_dir = directories::ProjectDirs::from("com", "vestige", "core") - .ok_or("Could not determine data directory")?; - let export_dir = vestige_dir - .data_dir() - .parent() - .unwrap_or(vestige_dir.data_dir()) - .join("exports"); + let export_dir = storage.sidecar_dir("exports"); std::fs::create_dir_all(&export_dir) .map_err(|e| format!("Failed to create export directory: {}", e))?; @@ -729,4 +772,41 @@ mod tests { assert_eq!(triggers["savesSinceLastDream"], 3); assert!(triggers["lastDreamTimestamp"].is_null()); } + + #[tokio::test] + async fn test_portable_export_writes_archive_to_storage_exports_dir() { + let (storage, _dir) = test_storage().await; + storage + .ingest(vestige_core::IngestInput { + content: "Portable MCP export test memory".to_string(), + node_type: "fact".to_string(), + source: None, + sentiment_score: 0.0, + sentiment_magnitude: 0.0, + tags: vec!["portable".to_string()], + valid_from: None, + valid_until: None, + }) + .unwrap(); + + let result = execute_export( + &storage, + Some(serde_json::json!({ + "format": "portable", + "path": "portable-test.json" + })), + ) + .await + .unwrap(); + + let path = result["path"].as_str().unwrap(); + assert_eq!(result["format"], "portable"); + assert!(path.ends_with("exports/portable-test.json")); + assert!(std::path::Path::new(path).exists()); + assert_eq!( + result["archiveFormat"], + vestige_core::PORTABLE_ARCHIVE_FORMAT + ); + assert!(result["rowsExported"].as_u64().unwrap() > 0); + } } diff --git a/crates/vestige-mcp/src/tools/restore.rs b/crates/vestige-mcp/src/tools/restore.rs index 6f1bc04..ac7184c 100644 --- a/crates/vestige-mcp/src/tools/restore.rs +++ b/crates/vestige-mcp/src/tools/restore.rs @@ -6,9 +6,10 @@ use serde::Deserialize; use serde_json::Value; +use std::path::Path; use std::sync::Arc; -use vestige_core::{IngestInput, Storage}; +use vestige_core::{IngestInput, PortableArchive, PortableImportMode, Storage}; /// Input schema for restore tool pub fn schema() -> Value { @@ -18,6 +19,16 @@ pub fn schema() -> Value { "path": { "type": "string", "description": "Path to the backup JSON file to restore from" + }, + "allowAnyPath": { + "type": "boolean", + "description": "Allow restoring from a file outside the active Vestige backups/ or exports/ directories. Only set true for trusted local files.", + "default": false + }, + "merge": { + "type": "boolean", + "description": "For portable archives, merge into the current database instead of requiring an empty target. Applies sync tombstones and keeps newer local memory rows on conflict.", + "default": false } }, "required": ["path"] @@ -25,8 +36,13 @@ pub fn schema() -> Value { } #[derive(Debug, Deserialize)] +#[serde(rename_all = "camelCase")] struct RestoreArgs { path: String, + #[serde(default)] + allow_any_path: bool, + #[serde(default)] + merge: bool, } #[derive(Deserialize)] @@ -56,14 +72,57 @@ pub async fn execute(storage: &Arc, args: Option) -> Result return Err("Missing arguments".to_string()), }; - let path = std::path::Path::new(&args.path); + let path = Path::new(&args.path); if !path.exists() { return Err(format!("Backup file not found: {}", args.path)); } + if !args.allow_any_path { + ensure_restore_path_allowed(storage, path)?; + } + // Read and parse backup - let backup_content = - std::fs::read_to_string(path).map_err(|e| format!("Failed to read backup: {}", e))?; + let backup_bytes = std::fs::read(path).map_err(|e| format!("Failed to read backup: {}", e))?; + if backup_bytes.starts_with(b"SQLite format 3\0") { + return Err( + "Restore expected JSON, but this file is a raw SQLite database backup. Use portable export/import for cross-device transfer, or replace the database file manually while Vestige is stopped." + .to_string(), + ); + } + let backup_content = String::from_utf8(backup_bytes) + .map_err(|_| "Restore file is not UTF-8 JSON".to_string())?; + + if let Ok(archive) = serde_json::from_str::(&backup_content) + && archive.archive_format == vestige_core::PORTABLE_ARCHIVE_FORMAT + { + let rows = archive.total_rows(); + let tables = archive.tables.len(); + let mode = if args.merge { + PortableImportMode::Merge + } else { + PortableImportMode::EmptyOnly + }; + let report = storage + .import_portable_archive(&archive, mode) + .map_err(|e| e.to_string())?; + return Ok(serde_json::json!({ + "tool": "restore", + "success": true, + "mode": if args.merge { "portable-merge" } else { "portable" }, + "tables": tables, + "rows": rows, + "tablesImported": report.tables_imported, + "rowsImported": report.rows_imported, + "tablesSkipped": report.tables_skipped, + "rowsInserted": report.rows_inserted, + "rowsUpdated": report.rows_updated, + "rowsDeleted": report.rows_deleted, + "rowsSkipped": report.rows_skipped, + "conflictsKeptLocal": report.conflicts_kept_local, + "ftsRebuilt": report.fts_rebuilt, + "message": format!("Imported {} rows from portable archive.", report.rows_imported), + })); + } // Try parsing as wrapped format first (MCP response wrapper), // then fall back to direct RecallResult @@ -132,6 +191,33 @@ pub async fn execute(storage: &Arc, args: Option) -> Result Result<(), String> { + let canonical_path = path + .canonicalize() + .map_err(|e| format!("Failed to resolve restore path: {}", e))?; + + for dir in [ + storage.sidecar_dir("exports"), + storage.sidecar_dir("backups"), + ] { + if !dir.exists() { + continue; + } + let canonical_dir = dir + .canonicalize() + .map_err(|e| format!("Failed to resolve allowed restore directory: {}", e))?; + if canonical_path.starts_with(&canonical_dir) { + return Ok(()); + } + } + + Err(format!( + "MCP restore is restricted to {} and {} by default. Pass allowAnyPath=true only for trusted local files.", + storage.sidecar_dir("exports").display(), + storage.sidecar_dir("backups").display() + )) +} + #[cfg(test)] mod tests { use super::*; @@ -156,6 +242,7 @@ mod tests { let s = schema(); assert_eq!(s["type"], "object"); assert!(s["properties"]["path"].is_object()); + assert!(s["properties"]["allowAnyPath"].is_object()); assert!( s["required"] .as_array() @@ -193,12 +280,22 @@ mod tests { async fn test_malformed_json_fails() { let (storage, dir) = test_storage().await; let path = write_temp_file(&dir, "bad.json", "this is not json {{{"); - let args = serde_json::json!({ "path": path }); + let args = serde_json::json!({ "path": path, "allowAnyPath": true }); let result = execute(&storage, Some(args)).await; assert!(result.is_err()); assert!(result.unwrap_err().contains("Unrecognized backup format")); } + #[tokio::test] + async fn test_restore_rejects_arbitrary_path_by_default() { + let (storage, dir) = test_storage().await; + let path = write_temp_file(&dir, "outside_exports.json", "[]"); + let args = serde_json::json!({ "path": path }); + let result = execute(&storage, Some(args)).await; + assert!(result.is_err()); + assert!(result.unwrap_err().contains("restricted")); + } + #[tokio::test] async fn test_restore_direct_array_format() { let (storage, dir) = test_storage().await; @@ -207,7 +304,7 @@ mod tests { { "content": "Memory two", "nodeType": "concept" } ]); let path = write_temp_file(&dir, "backup.json", &backup.to_string()); - let args = serde_json::json!({ "path": path }); + let args = serde_json::json!({ "path": path, "allowAnyPath": true }); let result = execute(&storage, Some(args)).await; assert!(result.is_ok()); let value = result.unwrap(); @@ -229,7 +326,7 @@ mod tests { ] }); let path = write_temp_file(&dir, "recall.json", &backup.to_string()); - let args = serde_json::json!({ "path": path }); + let args = serde_json::json!({ "path": path, "allowAnyPath": true }); let result = execute(&storage, Some(args)).await; assert!(result.is_ok()); let value = result.unwrap(); @@ -237,12 +334,47 @@ mod tests { assert_eq!(value["total"], 3); } + #[tokio::test] + async fn test_restore_portable_archive_from_allowed_exports_dir() { + let (source, _source_dir) = test_storage().await; + let (target, _target_dir) = test_storage().await; + + source + .ingest(vestige_core::IngestInput { + content: "Portable MCP restore test memory".to_string(), + node_type: "fact".to_string(), + source: None, + sentiment_score: 0.0, + sentiment_magnitude: 0.0, + tags: vec!["portable".to_string()], + valid_from: None, + valid_until: None, + }) + .unwrap(); + + let export_dir = target.sidecar_dir("exports"); + std::fs::create_dir_all(&export_dir).unwrap(); + let archive_path = export_dir.join("portable-restore.json"); + source + .export_portable_archive_to_path(&archive_path) + .unwrap(); + + let args = serde_json::json!({ "path": archive_path }); + let result = execute(&target, Some(args)).await.unwrap(); + + assert_eq!(result["tool"], "restore"); + assert_eq!(result["success"], true); + assert_eq!(result["mode"], "portable"); + assert!(result["rowsImported"].as_u64().unwrap() > 0); + assert_eq!(target.get_stats().unwrap().total_nodes, 1); + } + #[tokio::test] async fn test_restore_empty_results_array() { let (storage, dir) = test_storage().await; let backup = serde_json::json!({ "results": [] }); let path = write_temp_file(&dir, "empty.json", &backup.to_string()); - let args = serde_json::json!({ "path": path }); + let args = serde_json::json!({ "path": path, "allowAnyPath": true }); let result = execute(&storage, Some(args)).await; assert!(result.is_ok()); let value = result.unwrap(); @@ -255,7 +387,7 @@ mod tests { // Empty [] parses as Vec first, which has no items → "Empty backup file" let (storage, dir) = test_storage().await; let path = write_temp_file(&dir, "empty_arr.json", "[]"); - let args = serde_json::json!({ "path": path }); + let args = serde_json::json!({ "path": path, "allowAnyPath": true }); let result = execute(&storage, Some(args)).await; assert!(result.is_err()); assert!(result.unwrap_err().contains("Empty backup file")); @@ -266,7 +398,7 @@ mod tests { let (storage, dir) = test_storage().await; let backup = serde_json::json!([{ "content": "No type specified" }]); let path = write_temp_file(&dir, "notype.json", &backup.to_string()); - let args = serde_json::json!({ "path": path }); + let args = serde_json::json!({ "path": path, "allowAnyPath": true }); let result = execute(&storage, Some(args)).await; assert!(result.is_ok()); assert_eq!(result.unwrap()["restored"], 1); diff --git a/crates/vestige-mcp/src/tools/search_unified.rs b/crates/vestige-mcp/src/tools/search_unified.rs index bfe419e..d467e38 100644 --- a/crates/vestige-mcp/src/tools/search_unified.rs +++ b/crates/vestige-mcp/src/tools/search_unified.rs @@ -305,7 +305,8 @@ pub async fn execute( .unwrap_or(std::cmp::Ordering::Equal) }); - // Rerank the remaining candidates + // Rerank the remaining candidates when the vector-search search stack is enabled. + #[cfg(feature = "vector-search")] let reranked_results: Vec = if rerank_candidates.is_empty() { Vec::new() } else if let Ok(mut cog) = cognitive.try_lock() { @@ -330,6 +331,11 @@ pub async fn execute( .cloned() .collect() }; + #[cfg(not(feature = "vector-search"))] + let reranked_results: Vec = rerank_candidates + .into_iter() + .map(|(result, _)| result) + .collect(); // Merge: bypass first, then reranked, trim to limit filtered_results = bypass_results; @@ -340,6 +346,7 @@ pub async fn execute( // ==================================================================== // STAGE 3: Temporal boosting (recency + validity windows) // ==================================================================== + #[cfg(feature = "vector-search")] if let Ok(cog) = cognitive.try_lock() { for result in &mut filtered_results { let recency = cog.temporal_searcher.recency_boost(result.node.created_at); diff --git a/crates/vestige-mcp/src/tools/session_context.rs b/crates/vestige-mcp/src/tools/session_context.rs index e7414eb..56c615b 100644 --- a/crates/vestige-mcp/src/tools/session_context.rs +++ b/crates/vestige-mcp/src/tools/session_context.rs @@ -223,7 +223,7 @@ pub async fn execute( Some(dt) => storage.count_memories_since(*dt).unwrap_or(0), None => stats.total_nodes, }; - let last_backup = Storage::get_last_backup_timestamp(); + let last_backup = storage.last_backup_timestamp(); let now = Utc::now(); let needs_dream = last_dream @@ -236,7 +236,7 @@ pub async fn execute( if include_status { let embedding_pct = if stats.total_nodes > 0 { - (stats.nodes_with_embeddings as f64 / stats.total_nodes as f64) * 100.0 + (stats.nodes_with_active_embeddings as f64 / stats.total_nodes as f64) * 100.0 } else { 0.0 }; diff --git a/crates/vestige-mcp/src/tools/smart_ingest.rs b/crates/vestige-mcp/src/tools/smart_ingest.rs index 72dc3a6..f5c60cd 100644 --- a/crates/vestige-mcp/src/tools/smart_ingest.rs +++ b/crates/vestige-mcp/src/tools/smart_ingest.rs @@ -315,7 +315,10 @@ async fn execute_batch( let mut results = Vec::new(); let mut created = 0u32; + #[cfg(all(feature = "embeddings", feature = "vector-search"))] let mut updated = 0u32; + #[cfg(not(all(feature = "embeddings", feature = "vector-search")))] + let updated = 0u32; let mut skipped = 0u32; let mut errors = 0u32; diff --git a/docs/COGNITIVE_SANDWICH.md b/docs/COGNITIVE_SANDWICH.md index f7a4eee..6a395d4 100644 --- a/docs/COGNITIVE_SANDWICH.md +++ b/docs/COGNITIVE_SANDWICH.md @@ -2,7 +2,7 @@ **Vestige's defense-in-depth safety architecture for Claude Code.** -The Cognitive Sandwich wraps every Claude Code response in two layers of cognitive scaffolding: +The default Cognitive Sandwich installer only stages files and removes old v2.1.0 hook wiring. It activates no Claude Code hooks and makes no automatic model calls. Both the preflight layer and the Stop-hook layer are explicit opt-ins: ``` ┌────────────────────────────────────────────────┐ @@ -15,32 +15,31 @@ The Cognitive Sandwich wraps every Claude Code response in two layers of cogniti ├────────────────────────────────────────────────┤ │ 🥩 MEAT — Claude Code reasons │ ├────────────────────────────────────────────────┤ -│ 🥪 BOTTOM BREAD — Stop hooks │ -│ • Veto-detector (fast 50ms regex pre-screen) │ -│ • Sanhedrin Executioner (LOCAL Qwen3.6-35B) │ -│ • Synthesis stop validator (hedge detector) │ +│ 🥪 OPTIONAL BOTTOM BREAD — Stop hooks │ +│ • Veto-detector / synthesis validator │ +│ • Sanhedrin Executioner verifier │ └────────────────────────────────────────────────┘ ``` -The Sanhedrin Executioner is the headline of v2.1.0. As of v2.1.0 it runs entirely on a local MLX model (`mlx-community/Qwen3.6-35B-A3B-4bit`), replacing the v2.0.x Haiku 4.5 subagent. **Zero API cost per Claude turn, fully offline, ~5–15s verdict latency on M-series Apple Silicon.** +Sanhedrin, preflight, and all Vestige Claude Code hooks are optional. The default installer wires none of them; it does not call Claude, start MLX, require a 19 GB model download, or require 20+ GB of RAM. Users who want preflight context can opt in with `--enable-preflight`. Users who want the post-response verifier can opt in with `--enable-sanhedrin` and point it at any OpenAI-compatible `/v1/chat/completions` endpoint. On Apple Silicon, an additional `--with-launchd` flag can auto-start the local MLX Qwen backend. --- ## How a single response flows through the Sandwich 1. **You type a prompt in Claude Code.** -2. **UserPromptSubmit hooks fire in parallel** (none can block — all fail-open): +2. **If explicitly enabled, UserPromptSubmit hooks fire in parallel** (none can block — all fail-open): - `load-all-memory.sh` (opt-in) — dumps every memory MD into context - `synthesis-preflight.sh` — POSTs your prompt to `vestige-mcp` `/api/deep_reference`, injects the trust-scored reasoning chain - `cwd-state-injector.sh` — captures git status, branch, open PRs/issues, modified files - `vestige-pulse-daemon.sh` — injects fresh Vestige dream insights from the past 20 min into the next prompt context - `preflight-swarm.sh` — spawns the `lateral-thinker` subagent in fresh context to surface cross-disciplinary structural parallels 3. **Claude reads the assembled context and generates a draft.** -4. **Stop hooks fire serially** (any can VETO with `exit 2`, forcing a rewrite): +4. **By default, no Vestige Stop hooks are installed.** If explicitly enabled, Stop hooks fire serially (any can VETO with `exit 2`, forcing a rewrite): - `veto-detector.sh` — fast regex against `veto`-tagged Vestige memories (~50ms) - - `sanhedrin.sh` → `sanhedrin-local.py` — single-shot local Qwen3.6-35B-A3B verdict + - `sanhedrin.sh` → `sanhedrin-local.py` — optional single-shot semantic verdict - `synthesis-stop-validator.sh` — regex against forbidden patterns (hedging, summary-instead-of-composition) -5. **If all 3 Stop hooks return `exit 0`, the response is delivered.** +5. **If all enabled Stop hooks return `exit 0`, the response is delivered.** --- @@ -73,7 +72,7 @@ False-positive guards (added v2.1.0 after dogfood): ### One-liner ```bash -curl -fsSL https://raw.githubusercontent.com/samvallad33/vestige/v2.1.0/scripts/install-sandwich.sh | sh +curl -fsSL https://raw.githubusercontent.com/samvallad33/vestige/v2.1.1/scripts/install-sandwich.sh | sh ``` ### From a checkout @@ -82,30 +81,65 @@ curl -fsSL https://raw.githubusercontent.com/samvallad33/vestige/v2.1.0/scripts/ git clone https://github.com/samvallad33/vestige cd vestige ./scripts/install-sandwich.sh # add --force to overwrite existing hooks -./scripts/check-sandwich-prereqs.sh # verify everything's wired +./scripts/check-sandwich-prereqs.sh # verify no Vestige hooks are wired by default +``` + +The default command does not activate any Claude Code hook. It removes old v2.1.0 Vestige hook wiring from `~/.claude/settings.json` while preserving unrelated user hooks. + +### Optional Preflight + +Preflight is a separate opt-in layer. It includes `preflight-swarm.sh`, which uses `claude -p --model claude-haiku-4-5-20251001`; it is not wired by default. + +```bash +./scripts/install-sandwich.sh --enable-preflight +./scripts/check-sandwich-prereqs.sh --preflight +``` + +### Optional Sanhedrin + +Sanhedrin is a separate opt-in layer. + +```bash +# Wire the Sanhedrin Stop hook, using the default OpenAI-compatible endpoint. +./scripts/install-sandwich.sh --enable-sanhedrin + +# Apple Silicon only, and only if the machine has enough memory: +./scripts/install-sandwich.sh --enable-sanhedrin --with-launchd + +# x86 / Linux / Intel Mac: use any OpenAI-compatible endpoint. +./scripts/install-sandwich.sh \ + --enable-sanhedrin \ + --sanhedrin-endpoint=http://127.0.0.1:11434/v1/chat/completions \ + --sanhedrin-model=qwen2.5:14b ``` ### Prerequisites | Tool | Install | |---|---| -| macOS Apple Silicon (M1+) | required for MLX | | Python 3.10+ | typically preinstalled | | `jq` | `brew install jq` | +| `vestige-mcp` | `cargo install vestige-mcp` | +| Claude Code | https://claude.ai/code | + +Optional Apple Silicon local Sanhedrin backend: + +| Tool | Install | +|---|---| +| macOS Apple Silicon (M1+) | required for MLX launchd only | | `uv` | `brew install uv` | | `mlx-lm` | `uv tool install mlx-lm` | | `huggingface_hub[cli]` | `uv tool install 'huggingface_hub[cli]'` | -| `vestige-mcp` | `cargo install vestige-mcp` | -| Claude Code | https://claude.ai/code | | Qwen3.6-35B-A3B-4bit | `hf download mlx-community/Qwen3.6-35B-A3B-4bit` (~19 GB) | ### What the installer does 1. Verifies prereqs (warnings for missing tools, fatal only on jq/python3). 2. Copies hooks to `~/.claude/hooks/`, agents to `~/.claude/agents/`. -3. Renders `launchd/com.vestige.mlx-server.plist.template` with your `$HOME` and chosen model, writes to `~/Library/LaunchAgents/`. -4. `launchctl load` the plist (auto-start mlx_lm.server with the Qwen model on boot). -5. Backs up existing `~/.claude/settings.json` to `.bak.pre-sandwich`, then `jq`-merges the hooks block. +3. Backs up existing `~/.claude/settings.json` to `.bak.pre-sandwich`, then removes old Vestige hook wiring from previous v2.1.0 installs. +4. With `--enable-preflight`, merges the UserPromptSubmit hooks block. +5. With `--enable-sanhedrin`, writes `~/.claude/hooks/vestige-sanhedrin.env` and merges a Sanhedrin-enabled Stop hooks block. +6. With `--enable-sanhedrin --with-launchd` on Apple Silicon, renders and loads `launchd/com.vestige.mlx-server.plist.template`. ### Uninstall @@ -120,7 +154,7 @@ cp ~/.claude/settings.json.bak.pre-sandwich ~/.claude/settings.json ## Performance notes -On M3 Max 16-core (400 GB/s memory bandwidth): +Optional local MLX backend on M3 Max 16-core (400 GB/s memory bandwidth): - Sanhedrin verdict: 5–15 seconds end-to-end (single deep_reference + single Qwen call) - mlx_lm.server token generation: ~82 tok/s - mlx_lm.server peak resident memory: ~19.7 GB @@ -134,11 +168,12 @@ On M3 Max 14-core or M2/M1 Max: closer to 3–7s prompt processing, ~50–60 tok | Env var | Default | Effect | |---|---|---| -| `VESTIGE_SANHEDRIN_ENABLED` | `1` | Set to `0` to disable Sanhedrin Stop hook entirely | +| `VESTIGE_SANHEDRIN_ENABLED` | `0` | Set to `1` to enable the optional Sanhedrin Stop hook | | `VESTIGE_SWARM_ENABLED` | `1` | Set to `0` to disable preflight lateral-thinker swarm | | `VESTIGE_DASHBOARD_PORT` | `3927` | Vestige MCP HTTP API port used by hooks | -| `MLX_ENDPOINT` | `http://127.0.0.1:8080/v1/chat/completions` | OpenAI-compatible chat completions endpoint for Sanhedrin | -| `VESTIGE_SANDWICH_MODEL` | `mlx-community/Qwen3.6-35B-A3B-4bit` | Model launchd serves and Sanhedrin requests | +| `VESTIGE_SANHEDRIN_ENDPOINT` | `http://127.0.0.1:8080/v1/chat/completions` | OpenAI-compatible chat completions endpoint for Sanhedrin | +| `VESTIGE_SANHEDRIN_MODEL` | `mlx-community/Qwen3.6-35B-A3B-4bit` | Model name sent to the Sanhedrin endpoint | +| `MLX_ENDPOINT` / `VESTIGE_SANDWICH_MODEL` | legacy aliases | Backward-compatible names still read by the bridge | | `VESTIGE_MEMORY_DIR` | (auto) | Override per-user Claude memory dir | --- @@ -151,11 +186,12 @@ Full architecture memory: search Vestige for `god-tier-plan` or `cognitive-sandw --- -## Linux / Intel Mac +## Linux / Intel Mac / x86 -The launchd layer is macOS-arm64-only. On Linux or Intel Mac: -- Hooks + agents install fine with `--no-launchd` -- The Sanhedrin Stop hook will fail-open (mlx-server unreachable → exit 0) -- Optional: run a remote mlx_lm.server / vLLM / Ollama OpenAI-compatible endpoint and set `MLX_ENDPOINT` to its `/v1/chat/completions` URL +The base hook harness runs on x86. The launchd MLX helper is macOS-arm64-only. -Future v2.2.0 will add Linux-native MLX equivalents. +On Linux, Windows under WSL, or Intel Mac: +- Run `scripts/install-sandwich.sh` normally to stage files and remove old Vestige hook wiring. No hooks are activated. +- If you want Sanhedrin, run an OpenAI-compatible endpoint such as vLLM, Ollama, llama.cpp server, or a remote MLX/vLLM box. +- Install with `--enable-sanhedrin --sanhedrin-endpoint= --sanhedrin-model=`. +- If the endpoint is unreachable, Sanhedrin fails open and does not block Claude Code. diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 63e9cdf..8b7c773 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -6,7 +6,7 @@ ## First-Run Network Requirement -Vestige downloads the **Nomic Embed Text v1.5** model (~130MB) from Hugging Face on first use. +Vestige downloads the **Nomic Embed Text v1.5** model (~130MB) from Hugging Face on first use. Qwen3 embeddings are opt-in and download their own Hugging Face model when selected. **All subsequent runs are fully offline.** @@ -25,6 +25,8 @@ Override with environment variable: export FASTEMBED_CACHE_PATH="/custom/path" ``` +Qwen3 currently uses Hugging Face Hub's Candle loader directly, so use the standard Hugging Face cache environment such as `HF_HOME` if you need to relocate that larger model cache. + --- ## Environment Variables @@ -32,6 +34,7 @@ export FASTEMBED_CACHE_PATH="/custom/path" | Variable | Default | Description | |----------|---------|-------------| | `VESTIGE_DATA_DIR` | OS per-user data directory | Storage directory fallback; overridden by `--data-dir`; database lives at `/vestige.db` | +| `VESTIGE_EMBEDDING_MODEL` | `nomic-v1.5` | Embedding backend selector. Use `qwen3-0.6b` with a build that enables `qwen3-embeddings` | | `RUST_LOG` | `info` (via tracing-subscriber) | Log verbosity + per-module filtering | | `FASTEMBED_CACHE_PATH` | `./.fastembed_cache` | Embedding model cache location | | `VESTIGE_DASHBOARD_PORT` | `3927` | Dashboard HTTP + WebSocket port | @@ -50,6 +53,7 @@ export FASTEMBED_CACHE_PATH="/custom/path" ```bash vestige-mcp --data-dir /custom/path # Custom storage location VESTIGE_DATA_DIR=~/.vestige vestige-mcp # Env fallback storage location +VESTIGE_DATA_DIR=./.vestige vestige stats # Point the CLI at the same custom DB vestige-mcp --help # Show all options ``` @@ -66,6 +70,10 @@ vestige stats --states # Cognitive state distribution vestige health # System health check vestige consolidate # Run memory maintenance vestige restore # Restore from backup +vestige portable-export # Exact Vestige-to-Vestige archive +vestige portable-import # Import exact archive into an empty database +vestige portable-import --merge # Merge exact archive into this database +vestige sync # Pull/merge/push through a file backend ``` --- @@ -169,7 +177,7 @@ vestige update **Pin to specific version:** ```bash -vestige update --version v2.1.0 +vestige update --version v2.1.1 ``` **Check your version:** diff --git a/docs/STORAGE.md b/docs/STORAGE.md index 004dcee..ef98830 100644 --- a/docs/STORAGE.md +++ b/docs/STORAGE.md @@ -24,6 +24,44 @@ Override precedence: --- +## Moving Memories Between Devices + +For device-to-device migration, use a portable archive instead of the normal JSON export: + +```bash +# On the source machine +vestige portable-export ~/Desktop/vestige-portable.json + +# On the destination machine, before adding memories +vestige portable-import ~/Desktop/vestige-portable.json +``` + +Portable archives preserve raw Vestige storage rows: memory IDs, FSRS state, graph connections, suppression state, timestamps, audit history, and embedding blobs. + +For one-time migration, keep the conservative empty-database import: + +```bash +vestige portable-import ~/Desktop/vestige-portable.json +``` + +For cross-device sync, use merge mode or the file-backed sync command: + +```bash +# Merge a portable archive into an existing database. +vestige portable-import ~/Dropbox/vestige/portable.json --merge + +# Pull, merge, and push through a shared archive file. +vestige sync ~/Dropbox/vestige/portable.json +``` + +`vestige sync` uses the same pluggable portable-sync backend interface as the core library. v2.1.1 ships a file backend, which works with Dropbox, iCloud Drive, Syncthing, Git, network shares, or any folder-sync system. The merge algorithm applies delete tombstones, keeps newer local memories on timestamp conflicts, preserves stable IDs, rebuilds FTS after import, and writes the pushed archive atomically when the filesystem supports rename. + +When using the MCP `export` tool with `format: "portable"`, Vestige writes the archive under the active data directory's `exports/` folder. The MCP `restore` tool only reads from that `exports/` or `backups/` folder by default; pass `allowAnyPath: true` only for a trusted local file you selected manually. + +The regular `vestige export` / `vestige restore` path remains useful for human-readable backups, partial exports, and older files, but it re-ingests memory content and does not preserve every storage-level relationship. + +--- + ## Storage Modes ### Option 1: Global Memory (Default) @@ -69,6 +107,13 @@ This creates `.vestige/vestige.db` in your project root. Add `.vestige/` to `.gi If both `VESTIGE_DATA_DIR` and `--data-dir` are set, the CLI flag wins. Use the env var for a machine-wide default and the CLI flag for per-client or per-project overrides. +The `vestige` CLI also honors `VESTIGE_DATA_DIR`, so use the same directory when inspecting or exporting a custom MCP instance: + +```bash +VESTIGE_DATA_DIR=./.vestige vestige stats +VESTIGE_DATA_DIR=./.vestige vestige portable-export ./vestige-portable.json +``` + **Multiple Named Instances:** For power users who want both global AND project memory: @@ -132,7 +177,7 @@ Claude Code config - for "Storm": ## Data Safety -**Important:** Vestige stores data locally with no cloud sync, redundancy, or automatic backup. +**Important:** Vestige stores data locally. v2.1.1 adds user-controlled file-backed sync through `vestige sync`, but Vestige does not run a hosted cloud service, background replication daemon, or automatic backup for you. | Use Case | Risk Level | Recommendation | |----------|------------|----------------| diff --git a/hooks/sanhedrin-local.py b/hooks/sanhedrin-local.py index 00eb29b..677ba60 100755 --- a/hooks/sanhedrin-local.py +++ b/hooks/sanhedrin-local.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# sanhedrin-local.py — Local Qwen3.6-35B-A3B Sanhedrin Executioner. +# sanhedrin-local.py — OpenAI-compatible Sanhedrin Executioner bridge. # Drop-in replacement for the Haiku 4.5 subagent that sanhedrin.sh used to spawn. # # Reads draft from stdin, prints single-line verdict to stdout: @@ -8,10 +8,10 @@ # # Architecture: # stdin (draft) -> Vestige /api/deep_reference (single semantic query) -# -> mlx_lm.server localhost:8080 (one-shot judgment) +# -> OpenAI-compatible chat endpoint (one-shot judgment) # -> stdout (single-line verdict) # -# Fail-open: if mlx-server unreachable, print "yes" and exit 0 (don't break +# Fail-open: if the endpoint is unreachable, print "yes" and exit 0 (don't break # the Cognitive Sandwich on infra errors). The wrapping sanhedrin.sh maps # "yes" to exit 0, so this preserves existing fail-open semantics. @@ -35,7 +35,11 @@ VESTIGE_BASE_URL = ( os.environ.get("VESTIGE_BASE_URL") or f"http://127.0.0.1:{DASHBOARD_PORT}" ).rstrip("/") -MLX_ENDPOINT = os.environ.get("MLX_ENDPOINT") or "http://127.0.0.1:8080/v1/chat/completions" +SANHEDRIN_ENDPOINT = ( + os.environ.get("VESTIGE_SANHEDRIN_ENDPOINT") + or os.environ.get("MLX_ENDPOINT") + or "http://127.0.0.1:8080/v1/chat/completions" +) VESTIGE_ENDPOINT = ( os.environ.get("VESTIGE_DEEP_REFERENCE_ENDPOINT") or f"{VESTIGE_BASE_URL}/api/deep_reference" @@ -43,8 +47,12 @@ VESTIGE_ENDPOINT = ( VESTIGE_HEALTH = ( os.environ.get("VESTIGE_HEALTH_ENDPOINT") or f"{VESTIGE_BASE_URL}/api/health" ) -MODEL = os.environ.get("VESTIGE_SANDWICH_MODEL") or "mlx-community/Qwen3.6-35B-A3B-4bit" -MLX_TIMEOUT = env_int("MLX_TIMEOUT", 45) +MODEL = ( + os.environ.get("VESTIGE_SANHEDRIN_MODEL") + or os.environ.get("VESTIGE_SANDWICH_MODEL") + or "mlx-community/Qwen3.6-35B-A3B-4bit" +) +SANHEDRIN_TIMEOUT = env_int("VESTIGE_SANHEDRIN_TIMEOUT", env_int("MLX_TIMEOUT", 45)) VESTIGE_TIMEOUT = env_int("VESTIGE_TIMEOUT", 5) THINK_RE = re.compile(r".*?", re.DOTALL | re.IGNORECASE) @@ -289,7 +297,7 @@ def judge(draft: str, evidence: str) -> str: "\n\nOn second thought", "\n\nOh wait", ], } - resp = post_json(MLX_ENDPOINT, body, MLX_TIMEOUT) + resp = post_json(SANHEDRIN_ENDPOINT, body, SANHEDRIN_TIMEOUT) if not isinstance(resp, dict): return "" try: diff --git a/hooks/sanhedrin.sh b/hooks/sanhedrin.sh index 39a2131..a875f1f 100755 --- a/hooks/sanhedrin.sh +++ b/hooks/sanhedrin.sh @@ -17,25 +17,37 @@ # sanhedrin.sh (2-8s Haiku subagent, may block) → # synthesis-stop-validator.sh (existing regex hedge check, may block) # -# Opt-in: set VESTIGE_SANHEDRIN_ENABLED=1 in parent shell. +# Opt-in: set VESTIGE_SANHEDRIN_ENABLED=1 in parent shell, or install with +# scripts/install-sandwich.sh --enable-sanhedrin. # Re-entrancy lock: VESTIGE_EXECUTIONER_ACTIVE=1 inside the subagent. # # Ship date 2026-04-20. set -u -# === OPT-OUT GATE === -# Post-Cognitive Sanhedrin is ON by default as of 2026-04-21 (birthday -# launch day). To disable, set VESTIGE_SANHEDRIN_ENABLED=0 in your -# environment. Default-on guarantees the Cognitive Sandwich fires on -# fresh machines, Docker containers, GUI-launched Claude Code, and -# shells without .zshrc — any case where the Claude Code process lacks -# a sourced profile. The re-entrancy guard (VESTIGE_EXECUTIONER_ACTIVE) -# below still prevents fork-bombs from the subagent's own Stop hook. -if [ "${VESTIGE_SANHEDRIN_ENABLED:-1}" = "0" ]; then - exit 0 +# === OPT-IN GATE === +# Sanhedrin is heavyweight: the default local backend is a ~19 GB model and +# needs roughly 20+ GB of free RAM. Keep it disabled unless the user explicitly +# opts in. The installer writes this env file only for --enable-sanhedrin. +SANHEDRIN_ENV="${VESTIGE_SANHEDRIN_ENV:-$HOME/.claude/hooks/vestige-sanhedrin.env}" +if [ -f "$SANHEDRIN_ENV" ]; then + set +u + set -a + # shellcheck disable=SC1090 + . "$SANHEDRIN_ENV" 2>/dev/null || { + set +a + set -u + exit 0 + } + set +a + set -u fi +case "${VESTIGE_SANHEDRIN_ENABLED:-0}" in + 1|true|TRUE|yes|YES|on|ON) ;; + *) exit 0 ;; +esac + # === RE-ENTRANCY GUARD === # The Executioner's own Stop hook will fire when it returns — prevent # recursive spawns that would fork-bomb the quota. @@ -114,11 +126,11 @@ if [ -z "$DRAFT" ]; then fi # === VERIFY local executioner bridge available === -# 2026-04-25: switched from Haiku 4.5 subagent to local Qwen3.6-35B-A3B -# via mlx_lm.server (launchd com.vestige.mlx-server). Bridge script -# fetches Vestige evidence via HTTP API (VESTIGE_DASHBOARD_PORT, default 3927) -# then judges via MLX_ENDPOINT (default port 8080). Zero per-token cost, fully offline, -# sub-second-to-15s verdict latency. Fail-open if mlx-server unreachable. +# 2026-04-25: switched from Haiku 4.5 subagent to an OpenAI-compatible +# local/remote endpoint. On Apple Silicon the optional launchd path starts +# mlx_lm.server; on x86 users can point VESTIGE_SANHEDRIN_ENDPOINT at vLLM, +# Ollama, llama.cpp, or any compatible /v1/chat/completions endpoint. +# Fail-open if the endpoint is unreachable. BRIDGE="$HOME/.claude/hooks/sanhedrin-local.py" if [ ! -x "$BRIDGE" ] && [ ! -f "$BRIDGE" ]; then exit 0 @@ -191,15 +203,15 @@ case "$TRIMMED" in $REASON -The Executioner (local Qwen3.6-35B-A3B via mlx_lm.server, fresh context, -fed Vestige deep_reference evidence over HTTP) judged your draft and +The Executioner (Sanhedrin endpoint, fresh context, fed Vestige +deep_reference evidence over HTTP) judged your draft and found a contradiction against a high-trust memory. You may NOT stop. Rewrite WITHOUT the contradicted claim. Use mcp__vestige__deep_reference to inspect the cited memory and cite the correct replacement pattern from its \`recommended\` field. -Local-only, zero API cost, fully offline. Bridge script: +Bridge script: ~/.claude/hooks/sanhedrin-local.py SANHEDRIN_MSG exit 2 diff --git a/hooks/settings.fragment.json b/hooks/settings.fragment.json index 12254bd..0967ef4 100644 --- a/hooks/settings.fragment.json +++ b/hooks/settings.fragment.json @@ -1,23 +1 @@ -{ - "hooks": { - "UserPromptSubmit": [ - { - "hooks": [ - { "type": "command", "command": "$HOME/.claude/hooks/synthesis-preflight.sh", "timeout": 8 }, - { "type": "command", "command": "$HOME/.claude/hooks/cwd-state-injector.sh", "timeout": 8 }, - { "type": "command", "command": "$HOME/.claude/hooks/vestige-pulse-daemon.sh", "timeout": 6 }, - { "type": "command", "command": "$HOME/.claude/hooks/preflight-swarm.sh", "timeout": 45 } - ] - } - ], - "Stop": [ - { - "hooks": [ - { "type": "command", "command": "$HOME/.claude/hooks/veto-detector.sh", "timeout": 6 }, - { "type": "command", "command": "$HOME/.claude/hooks/sanhedrin.sh", "timeout": 70 }, - { "type": "command", "command": "$HOME/.claude/hooks/synthesis-stop-validator.sh", "timeout": 6 } - ] - } - ] - } -} +{} diff --git a/hooks/settings.preflight.fragment.json b/hooks/settings.preflight.fragment.json new file mode 100644 index 0000000..7a0d829 --- /dev/null +++ b/hooks/settings.preflight.fragment.json @@ -0,0 +1,14 @@ +{ + "hooks": { + "UserPromptSubmit": [ + { + "hooks": [ + { "type": "command", "command": "$HOME/.claude/hooks/synthesis-preflight.sh", "timeout": 8 }, + { "type": "command", "command": "$HOME/.claude/hooks/cwd-state-injector.sh", "timeout": 8 }, + { "type": "command", "command": "$HOME/.claude/hooks/vestige-pulse-daemon.sh", "timeout": 6 }, + { "type": "command", "command": "$HOME/.claude/hooks/preflight-swarm.sh", "timeout": 45 } + ] + } + ] + } +} diff --git a/hooks/settings.sanhedrin.fragment.json b/hooks/settings.sanhedrin.fragment.json new file mode 100644 index 0000000..893d427 --- /dev/null +++ b/hooks/settings.sanhedrin.fragment.json @@ -0,0 +1,13 @@ +{ + "hooks": { + "Stop": [ + { + "hooks": [ + { "type": "command", "command": "$HOME/.claude/hooks/veto-detector.sh", "timeout": 6 }, + { "type": "command", "command": "VESTIGE_SANHEDRIN_ENABLED=1 $HOME/.claude/hooks/sanhedrin.sh", "timeout": 70 }, + { "type": "command", "command": "$HOME/.claude/hooks/synthesis-stop-validator.sh", "timeout": 6 } + ] + } + ] + } +} diff --git a/package.json b/package.json index 8b9f5a4..6336ff6 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "vestige", - "version": "2.1.0", + "version": "2.1.1", "private": true, "description": "Cognitive memory for AI - MCP server with FSRS-6 spaced repetition", "author": "Sam Valladares", diff --git a/scripts/check-sandwich-prereqs.sh b/scripts/check-sandwich-prereqs.sh index cb9b368..b6f3509 100755 --- a/scripts/check-sandwich-prereqs.sh +++ b/scripts/check-sandwich-prereqs.sh @@ -5,21 +5,53 @@ set -u ok() { printf ' \033[1;32m[ OK ]\033[0m %s\n' "$*"; } warn() { printf ' \033[1;33m[WARN]\033[0m %s\n' "$*"; FAIL=1; } miss() { printf ' \033[1;31m[MISS]\033[0m %s\n' "$*"; FAIL=1; } +info() { printf ' \033[1;36m[INFO]\033[0m %s\n' "$*"; } FAIL=0 +CHECK_PREFLIGHT=0 +CHECK_SANHEDRIN=0 DASHBOARD_PORT="${VESTIGE_DASHBOARD_PORT:-3927}" -MLX_ENDPOINT="${MLX_ENDPOINT:-http://127.0.0.1:8080/v1/chat/completions}" -MLX_ENDPOINT="${MLX_ENDPOINT%/}" -MLX_MODELS_URL="${MLX_ENDPOINT%/chat/completions}/models" +SANHEDRIN_ENV="${VESTIGE_SANHEDRIN_ENV:-$HOME/.claude/hooks/vestige-sanhedrin.env}" + +for arg in "$@"; do + case "$arg" in + --preflight|--enable-preflight) CHECK_PREFLIGHT=1 ;; + --sanhedrin|--enable-sanhedrin) CHECK_SANHEDRIN=1 ;; + -h|--help) + cat <<'EOF' +Usage: scripts/check-sandwich-prereqs.sh [--preflight] [--sanhedrin] + +Without flags, verifies that the default install has no Vestige hooks wired. +With --preflight, checks the optional UserPromptSubmit hook layer. +With --sanhedrin, checks the optional OpenAI-compatible verifier endpoint. +EOF + exit 0 + ;; + esac +done + +if [ -f "$SANHEDRIN_ENV" ]; then + set +u + set -a + # shellcheck disable=SC1090 + . "$SANHEDRIN_ENV" 2>/dev/null || true + set +a + set -u +fi + +SANHEDRIN_ENDPOINT="${VESTIGE_SANHEDRIN_ENDPOINT:-${MLX_ENDPOINT:-http://127.0.0.1:8080/v1/chat/completions}}" +SANHEDRIN_ENDPOINT="${SANHEDRIN_ENDPOINT%/}" +SANHEDRIN_MODELS_URL="${SANHEDRIN_ENDPOINT%/chat/completions}/models" echo "Vestige Cognitive Sandwich — Prereq Check" echo # Platform -if [ "$(uname -s)" = "Darwin" ] && [ "$(uname -m)" = "arm64" ]; then - ok "Apple Silicon macOS ($(sw_vers -productVersion 2>/dev/null || echo darwin))" -else - miss "Apple Silicon Mac required (M1+). Detected $(uname -s) $(uname -m)." +OS_NAME="$(uname -s)" +ARCH_NAME="$(uname -m)" +ok "Platform: $OS_NAME $ARCH_NAME" +if [ "$OS_NAME" != "Darwin" ] || [ "$ARCH_NAME" != "arm64" ]; then + info "Local MLX launchd is Apple Silicon-only; base hooks and endpoint-backed Sanhedrin can run on x86." fi # Python @@ -35,56 +67,95 @@ fi # CLI tools command -v jq >/dev/null && ok "jq" || miss "jq missing — brew install jq" -command -v uv >/dev/null && ok "uv" || miss "uv missing — brew install uv" -command -v mlx_lm.server >/dev/null && ok "mlx-lm" || miss "mlx-lm — uv tool install mlx-lm" -command -v hf >/dev/null && ok "huggingface_hub CLI" || miss "hf — uv tool install 'huggingface_hub[cli]'" -command -v claude >/dev/null && ok "claude CLI" || miss "claude CLI — install Claude Code" -command -v vestige-mcp >/dev/null && ok "vestige-mcp" || miss "vestige-mcp — cargo install vestige-mcp" +if [ "$CHECK_PREFLIGHT" -eq 1 ]; then + command -v claude >/dev/null && ok "claude CLI" || miss "claude CLI — install Claude Code" + command -v vestige-mcp >/dev/null && ok "vestige-mcp" || miss "vestige-mcp — cargo install vestige-mcp" -# Model on disk — HF cache uses `models----` (double-dash separators). -MODEL="${VESTIGE_SANDWICH_MODEL:-mlx-community/Qwen3.6-35B-A3B-4bit}" -HF_HOME_DEFAULT="${HF_HOME:-$HOME/.cache/huggingface}" -ENC_MODEL="models--$(printf '%s' "$MODEL" | sed 's|/|--|g')" -if [ -d "$HF_HOME_DEFAULT/hub/$ENC_MODEL" ]; then - ok "Model cached: $MODEL" -else - printf ' \033[1;33m[INFO]\033[0m Model not yet downloaded — first run will fetch ~19GB\n' - printf ' hf download %s\n' "$MODEL" - # NOT a failure — first-run download is expected. -fi - -# Vestige MCP HTTP API -if curl -fsS -m 2 "http://127.0.0.1:${DASHBOARD_PORT}/api/health" >/dev/null 2>&1; then - ok "vestige-mcp dashboard responding on :$DASHBOARD_PORT" -else - warn "vestige-mcp dashboard not responding on :$DASHBOARD_PORT" -fi - -# OpenAI-compatible local/remote model endpoint -if curl -fsS -m 2 "$MLX_MODELS_URL" >/dev/null 2>&1; then - ok "model endpoint responding at $MLX_MODELS_URL" -else - warn "model endpoint not responding at $MLX_MODELS_URL — install + load launchd plist or set MLX_ENDPOINT" -fi - -# launchd plist -if [ -f "$HOME/Library/LaunchAgents/com.vestige.mlx-server.plist" ]; then - ok "launchd plist installed" -else - warn "launchd plist missing — run: install-sandwich.sh" + # Vestige MCP HTTP API + if curl -fsS -m 2 "http://127.0.0.1:${DASHBOARD_PORT}/api/health" >/dev/null 2>&1; then + ok "vestige-mcp dashboard responding on :$DASHBOARD_PORT" + else + warn "vestige-mcp dashboard not responding on :$DASHBOARD_PORT" + fi fi # Settings hook wiring -if [ -f "$HOME/.claude/settings.json" ] && \ - jq -e '.hooks.UserPromptSubmit and .hooks.Stop' "$HOME/.claude/settings.json" >/dev/null 2>&1; then - ok "settings.json hooks block present" +if [ "$CHECK_PREFLIGHT" -eq 0 ] && [ "$CHECK_SANHEDRIN" -eq 0 ]; then + if [ -f "$HOME/.claude/settings.json" ] && \ + jq -e 'any((.hooks.UserPromptSubmit[]?.hooks[]?, .hooks.Stop[]?.hooks[]?); ((.command? // "") | test("synthesis-preflight\\.sh|cwd-state-injector\\.sh|vestige-pulse-daemon\\.sh|preflight-swarm\\.sh|load-all-memory\\.sh|veto-detector\\.sh|sanhedrin\\.sh|synthesis-stop-validator\\.sh|synthesis-gate\\.sh")))' "$HOME/.claude/settings.json" >/dev/null 2>&1; then + warn "Vestige hooks are still wired; run: install-sandwich.sh --force" + else + ok "no Vestige Claude Code hooks wired by default" + fi +fi + +if [ "$CHECK_PREFLIGHT" -eq 1 ]; then + echo + echo "Optional Preflight" + + if [ -f "$HOME/.claude/settings.json" ] && \ + jq -e 'any(.hooks.UserPromptSubmit[]?.hooks[]?; ((.command? // "") | contains("synthesis-preflight.sh"))) and any(.hooks.UserPromptSubmit[]?.hooks[]?; ((.command? // "") | contains("preflight-swarm.sh")))' "$HOME/.claude/settings.json" >/dev/null 2>&1; then + ok "preflight UserPromptSubmit hooks wired" + else + warn "preflight hooks not wired — run: install-sandwich.sh --enable-preflight" + fi + + info "preflight-swarm.sh uses claude -p with Haiku when enabled; default installs do not wire it." +fi + +if [ "$CHECK_SANHEDRIN" -eq 1 ]; then + echo + echo "Optional Sanhedrin" + + if [ -f "$SANHEDRIN_ENV" ]; then + ok "Sanhedrin env file present" + else + warn "Sanhedrin env file missing — run: install-sandwich.sh --enable-sanhedrin" + fi + + if [ "$OS_NAME" = "Darwin" ] && [ "$ARCH_NAME" = "arm64" ]; then + command -v uv >/dev/null && ok "uv" || warn "uv missing — brew install uv" + command -v mlx_lm.server >/dev/null && ok "mlx-lm" || warn "mlx-lm — uv tool install mlx-lm" + command -v hf >/dev/null && ok "huggingface_hub CLI" || warn "hf — uv tool install 'huggingface_hub[cli]'" + + MODEL="${VESTIGE_SANHEDRIN_MODEL:-${VESTIGE_SANDWICH_MODEL:-mlx-community/Qwen3.6-35B-A3B-4bit}}" + HF_HOME_DEFAULT="${HF_HOME:-$HOME/.cache/huggingface}" + ENC_MODEL="models--$(printf '%s' "$MODEL" | sed 's|/|--|g')" + if [ -d "$HF_HOME_DEFAULT/hub/$ENC_MODEL" ]; then + ok "Model cached: $MODEL" + else + info "Model not cached: $MODEL (local MLX path downloads ~19GB)" + fi + + if [ -f "$HOME/Library/LaunchAgents/com.vestige.mlx-server.plist" ]; then + ok "launchd plist installed" + else + info "launchd plist not installed; endpoint-backed Sanhedrin can still run" + fi + else + info "Skipping MLX/launchd checks on $OS_NAME $ARCH_NAME" + fi + + if curl -fsS -m 2 "$SANHEDRIN_MODELS_URL" >/dev/null 2>&1; then + ok "Sanhedrin model endpoint responding at $SANHEDRIN_MODELS_URL" + else + warn "Sanhedrin endpoint not responding at $SANHEDRIN_MODELS_URL" + fi + + if [ -f "$HOME/.claude/settings.json" ] && \ + jq -e '.hooks.Stop[]?.hooks[]?.command | contains("sanhedrin.sh")' "$HOME/.claude/settings.json" >/dev/null 2>&1; then + ok "Sanhedrin Stop hook wired" + else + warn "Sanhedrin Stop hook not wired — run: install-sandwich.sh --enable-sanhedrin" + fi else - warn "settings.json missing hooks block — run: install-sandwich.sh" + echo + info "Sanhedrin is optional and not checked. Use --sanhedrin to verify an enabled endpoint." fi echo if [ $FAIL -eq 0 ]; then - echo " Ready. Cognitive Sandwich will fire on next Claude Code prompt." + echo " Ready. Default install has no Vestige Claude Code hooks wired and makes no automatic model calls." exit 0 else echo " Fix the items above, then re-run." diff --git a/scripts/install-sandwich.sh b/scripts/install-sandwich.sh index 7b3c70b..fd2b4b9 100755 --- a/scripts/install-sandwich.sh +++ b/scripts/install-sandwich.sh @@ -2,28 +2,28 @@ # install-sandwich.sh — One-command installer for the Vestige Cognitive Sandwich. # # Usage: -# curl -fsSL https://raw.githubusercontent.com/samvallad33/vestige/v2.1.0/scripts/install-sandwich.sh | sh +# curl -fsSL https://raw.githubusercontent.com/samvallad33/vestige/v2.1.1/scripts/install-sandwich.sh | sh # # or, from a checkout: -# ./scripts/install-sandwich.sh [--force] [--no-launchd] [--include-memory-loader] +# ./scripts/install-sandwich.sh [--force] [--enable-preflight] [--enable-sanhedrin] [--with-launchd] [--include-memory-loader] +# ./scripts/install-sandwich.sh --enable-sanhedrin --sanhedrin-endpoint=http://127.0.0.1:11434/v1/chat/completions --sanhedrin-model=qwen2.5:14b # # What it does: # 1. Verifies required local tools -# 2. Stages ~/.claude/hooks/, ~/.claude/agents/, ~/Library/LaunchAgents/ +# 2. Stages ~/.claude/hooks/ and ~/.claude/agents/ # 3. Copies sanitized hooks + agents -# 4. Renders launchd plist template with $HOME and chosen MODEL -# 5. Merges hooks block into ~/.claude/settings.json (preserves existing keys) -# 6. launchctl load com.vestige.mlx-server (auto-starts mlx_lm.server with Qwen3.6-35B-A3B) -# 7. Prints next-steps for model download +# 4. Removes old Vestige hook wiring from ~/.claude/settings.json by default +# 5. Optionally enables preflight hooks and/or Sanhedrin. Only with --with-launchd on Apple Silicon, +# auto-starts mlx_lm.server with Qwen3.6-35B-A3B set -euo pipefail -VERSION="${VESTIGE_SANDWICH_VERSION:-v2.1.0}" +VERSION="${VESTIGE_SANDWICH_VERSION:-v2.1.1}" REPO="samvallad33/vestige" -MODEL_ID="${VESTIGE_SANDWICH_MODEL:-mlx-community/Qwen3.6-35B-A3B-4bit}" +MODEL_ID="${VESTIGE_SANHEDRIN_MODEL:-${VESTIGE_SANDWICH_MODEL:-mlx-community/Qwen3.6-35B-A3B-4bit}}" DASHBOARD_PORT="${VESTIGE_DASHBOARD_PORT:-3927}" -MLX_ENDPOINT="${MLX_ENDPOINT:-http://127.0.0.1:8080/v1/chat/completions}" -MLX_ENDPOINT="${MLX_ENDPOINT%/}" -MLX_MODELS_URL="${MLX_ENDPOINT%/chat/completions}/models" +SANHEDRIN_ENDPOINT="${VESTIGE_SANHEDRIN_ENDPOINT:-${MLX_ENDPOINT:-http://127.0.0.1:8080/v1/chat/completions}}" +SANHEDRIN_ENDPOINT="${SANHEDRIN_ENDPOINT%/}" +SANHEDRIN_MODELS_URL="${SANHEDRIN_ENDPOINT%/chat/completions}/models" HOOKS_DIR="$HOME/.claude/hooks" AGENTS_DIR="$HOME/.claude/agents" @@ -31,45 +31,70 @@ LAUNCHD_DIR="$HOME/Library/LaunchAgents" SETTINGS="$HOME/.claude/settings.json" FORCE=0 -NO_LAUNCHD=0 +ENABLE_PREFLIGHT=0 +ENABLE_SANHEDRIN=0 +WITH_LAUNCHD=0 INCLUDE_MEMORY_LOADER=0 SRC="" for arg in "$@"; do case "$arg" in --force) FORCE=1 ;; - --no-launchd) NO_LAUNCHD=1 ;; + --enable-preflight) ENABLE_PREFLIGHT=1 ;; + --enable-sandwich) ENABLE_PREFLIGHT=1; ENABLE_SANHEDRIN=1 ;; + --enable-sanhedrin) ENABLE_SANHEDRIN=1 ;; + --with-launchd) WITH_LAUNCHD=1 ;; + --no-launchd) WITH_LAUNCHD=0 ;; --include-memory-loader) INCLUDE_MEMORY_LOADER=1 ;; + --sanhedrin-endpoint=*|--endpoint=*) + SANHEDRIN_ENDPOINT="${arg#*=}" + SANHEDRIN_ENDPOINT="${SANHEDRIN_ENDPOINT%/}" + SANHEDRIN_MODELS_URL="${SANHEDRIN_ENDPOINT%/chat/completions}/models" + ;; + --sanhedrin-model=*|--model=*) + MODEL_ID="${arg#*=}" + ;; --src=*) SRC="${arg#--src=}" ;; -h|--help) - sed -n '2,20p' "$0" + sed -n '2,24p' "$0" exit 0 ;; esac done +if [ "$WITH_LAUNCHD" -eq 1 ] && [ "$ENABLE_SANHEDRIN" -eq 0 ]; then + ENABLE_SANHEDRIN=1 +fi + say() { printf '\033[1;36m[sandwich]\033[0m %s\n' "$*"; } warn() { printf '\033[1;33m[sandwich]\033[0m %s\n' "$*" >&2; } die() { printf '\033[1;31m[sandwich]\033[0m %s\n' "$*" >&2; exit 1; } -# --- Platform check (honors --no-launchd for Linux/Intel users) --- -if [ "$(uname -s)" != "Darwin" ]; then - if [ "$NO_LAUNCHD" -eq 0 ]; then - die "macOS required for the launchd auto-start of mlx_lm.server. Re-run with --no-launchd to install hooks only and run mlx_lm.server manually." - fi - warn "Non-Darwin platform — installing hooks/agents only (no launchd). Run an OpenAI-compatible model endpoint and set MLX_ENDPOINT if it is not $MLX_ENDPOINT." -elif [ "$(uname -m)" != "arm64" ]; then - warn "Apple Silicon recommended (M1+). Detected $(uname -m). The local Qwen3.6 model requires arm64 + Metal." +# --- Platform check --- +OS_NAME="$(uname -s)" +ARCH_NAME="$(uname -m)" +say "platform: $OS_NAME $ARCH_NAME" +if [ "$ENABLE_SANHEDRIN" -eq 1 ] && [ "$WITH_LAUNCHD" -eq 0 ]; then + say "Sanhedrin enabled without launchd; using OpenAI-compatible endpoint: $SANHEDRIN_ENDPOINT" +fi +if [ "$WITH_LAUNCHD" -eq 1 ] && { [ "$OS_NAME" != "Darwin" ] || [ "$ARCH_NAME" != "arm64" ]; }; then + warn "--with-launchd is Apple Silicon only; skipping local MLX autostart on $OS_NAME $ARCH_NAME" + warn "Sanhedrin can still run on x86 via --sanhedrin-endpoint or VESTIGE_SANHEDRIN_ENDPOINT." + WITH_LAUNCHD=0 fi # --- Prereqs (warnings only, install proceeds) --- command -v jq >/dev/null || die "jq required: brew install jq" command -v python3 >/dev/null || die "python3 required (3.10+)" -command -v claude >/dev/null || warn "'claude' CLI not found — install Claude Code first." -command -v vestige-mcp >/dev/null || warn "'vestige-mcp' not found — install with: cargo install vestige-mcp" -command -v uv >/dev/null || warn "'uv' not found — install with: brew install uv" -command -v mlx_lm.server >/dev/null || warn "mlx-lm not installed — run: uv tool install mlx-lm" -command -v hf >/dev/null || warn "'hf' not found — run: uv tool install 'huggingface_hub[cli]'" +if [ "$ENABLE_PREFLIGHT" -eq 1 ]; then + command -v claude >/dev/null || warn "'claude' CLI not found — preflight-swarm.sh will fail open." + command -v vestige-mcp >/dev/null || warn "'vestige-mcp' not found — Vestige preflight hooks will fail open." +fi +if [ "$WITH_LAUNCHD" -eq 1 ]; then + command -v uv >/dev/null || warn "'uv' not found — install with: brew install uv" + command -v mlx_lm.server >/dev/null || warn "mlx-lm not installed — run: uv tool install mlx-lm" + command -v hf >/dev/null || warn "'hf' not found — run: uv tool install 'huggingface_hub[cli]'" +fi # --- Resolve source: local checkout or release tarball --- if [ -n "$SRC" ]; then @@ -89,7 +114,21 @@ fi [ -d "$SCRIPT_DIR/hooks" ] || die "hooks/ not found in $SCRIPT_DIR — wrong source?" # --- Stage directories --- -mkdir -p "$HOOKS_DIR" "$AGENTS_DIR" "$LAUNCHD_DIR" +mkdir -p "$HOOKS_DIR" "$AGENTS_DIR" +if [ "$WITH_LAUNCHD" -eq 1 ]; then + mkdir -p "$LAUNCHD_DIR" +fi + +# v2.1.0 originally installed the MLX server as part of the default path. +# Default reinstalls now retire that job; users can restore it with --with-launchd. +if [ "$WITH_LAUNCHD" -eq 0 ] && [ "$OS_NAME" = "Darwin" ]; then + LEGACY_PLIST="$LAUNCHD_DIR/com.vestige.mlx-server.plist" + if [ -f "$LEGACY_PLIST" ]; then + launchctl unload "$LEGACY_PLIST" 2>/dev/null || true + rm -f "$LEGACY_PLIST" + say "removed old Sanhedrin launchd job (use --with-launchd to opt back in)" + fi +fi # --- Copy hooks --- copied=0; skipped=0 @@ -121,8 +160,25 @@ for f in "$SCRIPT_DIR/agents"/*.md; do done say "agents installed to $AGENTS_DIR" -# --- Render launchd plist (macOS only) --- -if [ "$NO_LAUNCHD" -eq 0 ]; then +# --- Persist optional Sanhedrin env --- +quote_env() { + printf "'%s'" "$(printf '%s' "$1" | sed "s/'/'\\\\''/g")" +} + +if [ "$ENABLE_SANHEDRIN" -eq 1 ]; then + SANHEDRIN_ENV="$HOOKS_DIR/vestige-sanhedrin.env" + { + printf 'VESTIGE_SANHEDRIN_ENABLED=1\n' + printf 'VESTIGE_SANHEDRIN_ENDPOINT=%s\n' "$(quote_env "$SANHEDRIN_ENDPOINT")" + printf 'VESTIGE_SANHEDRIN_MODEL=%s\n' "$(quote_env "$MODEL_ID")" + printf 'VESTIGE_DASHBOARD_PORT=%s\n' "$(quote_env "$DASHBOARD_PORT")" + } > "$SANHEDRIN_ENV" + chmod 0600 "$SANHEDRIN_ENV" + say "Sanhedrin opt-in config written to $SANHEDRIN_ENV" +fi + +# --- Render launchd plist (Apple Silicon opt-in only) --- +if [ "$WITH_LAUNCHD" -eq 1 ]; then PLIST="$LAUNCHD_DIR/com.vestige.mlx-server.plist" TEMPLATE="$SCRIPT_DIR/launchd/com.vestige.mlx-server.plist.template" [ -f "$TEMPLATE" ] || die "launchd template missing: $TEMPLATE" @@ -140,31 +196,80 @@ else cp "$SETTINGS" "$HOME/.claude/settings.json.bak.pre-sandwich" fi TMP_MERGE="$(mktemp)" -jq -s '.[0] * .[1]' "$SETTINGS" "$SCRIPT_DIR/hooks/settings.fragment.json" > "$TMP_MERGE" +PREFLIGHT_FRAGMENT="$SCRIPT_DIR/hooks/settings.fragment.json" +SANHEDRIN_FRAGMENT="$SCRIPT_DIR/hooks/settings.fragment.json" +if [ "$ENABLE_PREFLIGHT" -eq 1 ]; then + PREFLIGHT_FRAGMENT="$SCRIPT_DIR/hooks/settings.preflight.fragment.json" +fi +if [ "$ENABLE_SANHEDRIN" -eq 1 ]; then + SANHEDRIN_FRAGMENT="$SCRIPT_DIR/hooks/settings.sanhedrin.fragment.json" +fi +jq -s ' + def is_vestige_hook: + (.command? // "") as $cmd + | [ + "synthesis-preflight.sh", + "cwd-state-injector.sh", + "vestige-pulse-daemon.sh", + "preflight-swarm.sh", + "load-all-memory.sh", + "veto-detector.sh", + "sanhedrin.sh", + "synthesis-stop-validator.sh", + "synthesis-gate.sh" + ] | any(. as $needle | $cmd | contains($needle)); + + def scrub_vestige_hooks: + .hooks.UserPromptSubmit = ( + (.hooks.UserPromptSubmit // []) + | map(.hooks = ((.hooks // []) | map(select((is_vestige_hook | not))))) + | map(select(((.hooks // []) | length) > 0)) + ) + | if ((.hooks.UserPromptSubmit // []) | length) == 0 then del(.hooks.UserPromptSubmit) else . end + | .hooks.Stop = ( + (.hooks.Stop // []) + | map(.hooks = ((.hooks // []) | map(select((is_vestige_hook | not))))) + | map(select(((.hooks // []) | length) > 0)) + ) + | if ((.hooks.Stop // []) | length) == 0 then del(.hooks.Stop) else . end + | if ((.hooks // {}) | length) == 0 then del(.hooks) else . end; + + (.[0] | scrub_vestige_hooks) * .[1] * .[2] +' "$SETTINGS" "$PREFLIGHT_FRAGMENT" "$SANHEDRIN_FRAGMENT" > "$TMP_MERGE" mv "$TMP_MERGE" "$SETTINGS" -say "merged hooks block into $SETTINGS (backup at .bak.pre-sandwich)" +if [ "$ENABLE_PREFLIGHT" -eq 1 ] || [ "$ENABLE_SANHEDRIN" -eq 1 ]; then + enabled_layers="" + [ "$ENABLE_PREFLIGHT" -eq 1 ] && enabled_layers="${enabled_layers} preflight" + [ "$ENABLE_SANHEDRIN" -eq 1 ] && enabled_layers="${enabled_layers} sanhedrin" + say "merged optional hook layer(s) into $SETTINGS:${enabled_layers} (backup at .bak.pre-sandwich)" +else + say "removed Vestige hook wiring from $SETTINGS; default install activates no Claude Code hooks (backup at .bak.pre-sandwich)" +fi # --- Next steps --- cat <20 GB free RAM, add --with-launchd to auto-start + the local MLX Qwen server. On x86, point --sanhedrin-endpoint at vLLM, + Ollama, llama.cpp, or another OpenAI-compatible /v1/chat/completions URL. To uninstall: - launchctl unload $LAUNCHD_DIR/com.vestige.mlx-server.plist - rm $LAUNCHD_DIR/com.vestige.mlx-server.plist + launchctl unload $LAUNCHD_DIR/com.vestige.mlx-server.plist 2>/dev/null || true + rm -f $LAUNCHD_DIR/com.vestige.mlx-server.plist cp $HOME/.claude/settings.json.bak.pre-sandwich $HOME/.claude/settings.json EOF