From 6bbafc024191f2910d05d4ee43c3ae1f215ae7b4 Mon Sep 17 00:00:00 2001 From: Sam Valladares Date: Sun, 26 Jul 2026 00:20:35 +0800 Subject: [PATCH] fix(release): close the 3 pre-ship blockers from the final audit README: add the '## Vestige Pro' section so the #vestige-pro anchor that six shipped surfaces link to actually resolves. States the price and the zero-knowledge guarantee honestly, with no checkout link, because checkout is not open yet. SECURITY: 'Vulnerabilities: 0' was false at 5. Updated rustls-webpki, quinn-proto and crossbeam-epoch; cargo audit now genuinely exits 0. Supported versions now list 2.3.x. Migration V20: clears connector sync cursors so the V19 idempotency repair happens automatically on the next source_sync, instead of a changelog note telling users to run something that could not have worked. cloud-sync hardening: redact the sync key and passphrase from Debug, require https (loopback carve-out), enforce a 12 character passphrase minimum. Trace retention days are clamped so a hostile value cannot abort the process. release.yml: add 'connectors' to the Windows and Intel Mac rows so source_sync is a real tool on those platforms rather than a compiled-out stub. CHANGELOG: correct three claims the audit fact-checked as false, document the breaking plaintext-sync change, and add the two new env levers. Gates: cargo test 1588/0, cloud-sync suite 18/0, clippy clean, svelte-check 937/0/0, dashboard build green, cargo audit 0 vulnerabilities. Co-Authored-By: Claude Opus 5 --- .github/workflows/release.yml | 4 +- CHANGELOG.md | 75 ++++++-- Cargo.lock | 77 ++++++--- README.md | 22 ++- SECURITY.md | 38 ++++- crates/vestige-core/src/storage/cloud_sync.rs | 144 +++++++++++++++- crates/vestige-core/src/storage/migrations.rs | 160 ++++++++++++++++++ .../vestige-core/src/storage/trace_store.rs | 133 ++++++++++++++- docs/CONFIGURATION.md | 21 +++ 9 files changed, 621 insertions(+), 53 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 4512ac4..1aec977 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -38,7 +38,7 @@ jobs: # generates 0 embeddings, and no model download is ever attempted. # usearch builds cleanly on MSVC because vestige-core pins it with # features=["fp16lib"] (see crates/vestige-core/Cargo.toml). - cargo_flags: "--no-default-features --features embeddings,ort-download,vector-search,cloud-sync" + cargo_flags: "--no-default-features --features embeddings,ort-download,vector-search,cloud-sync,connectors" needs_onnxruntime: false # Intel Mac uses the ort-dynamic feature to runtime-link against a # system libonnxruntime (Homebrew), sidestepping the missing @@ -48,7 +48,7 @@ jobs: - target: x86_64-apple-darwin os: macos-latest archive: tar.gz - cargo_flags: "--no-default-features --features ort-dynamic,vector-search,cloud-sync" + cargo_flags: "--no-default-features --features ort-dynamic,vector-search,cloud-sync,connectors" - target: aarch64-apple-darwin os: macos-latest archive: tar.gz diff --git a/CHANGELOG.md b/CHANGELOG.md index 150b6fd..0bea99e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,7 +25,7 @@ graph page (an Observatory button beside Dream and Memory Cinema takes over full-bleed; Esc exits), and the field is clickable — GPU picking opens the same Memory Detail inspector Classic's picking drove. -### Added — Vestige Pro: zero-knowledge cloud-sync client + upgrade surfaces +### Added — Vestige Pro: zero-knowledge cloud-sync client + CLI/npm upgrade surfaces The public client for the hosted Vestige Cloud managed-sync service lands behind the optional `cloud-sync` cargo feature, and encryption is mandatory: @@ -40,17 +40,28 @@ have). The refusal cuts both ways: the client will not upload an unencrypted archive, and it rejects a plaintext blob on download instead of quietly accepting it. The transport is the existing pull-merge-push portable-sync engine over HTTP with ETag/`If-Match` optimistic concurrency, so two devices -converge without lost updates. The dashboard gains matching Pro upgrade -surfaces. **The hosted service is optional and paid; local-first stays free** — -the default build links no HTTP client and behaves exactly as before. +converge without lost updates. The Pro upgrade surfaces that ship with it are +**CLI and npm only** — the `vestige sync --cloud` subscribe wall, the `--help` +and `vestige health` footers, the npm README, and the postinstall banner. The +dashboard gains no Pro surface in this release. **The hosted service is optional +and paid; local-first stays free** — the default `vestige-core` library build +links no HTTP client at all, and in the shipped server binary (whose default +features do include `connectors` and `cloud-sync`, so it links `reqwest`) the +cloud client is inert unless you set `VESTIGE_CLOUD_ENDPOINT`, +`VESTIGE_CLOUD_SYNC_KEY`, and `VESTIGE_CLOUD_ENCRYPTION_KEY` and explicitly run +`vestige sync --cloud`. ### Changed — The living field is now the main graph renderer The main graph page defaults to the Observatory engine wherever WebGPU exists. A new **Field | Classic** toggle in the control bar keeps the Three.js view one click away and untouched (picking, colour modes, temporal scrubbing and legends -remain Classic features); no WebGPU forces Classic automatically, and the -choice persists in `localStorage`. Dream, Observatory takeover, Memory Cinema, +remain Classic features); a missing or non-functional WebGPU stack forces +Classic automatically — including the case where `'gpu' in navigator` is true +but `requestAdapter()` returns null, `requestDevice()` throws, or the device is +lost mid-session — and the choice persists in `localStorage` (the automatic +fallback deliberately does not persist, so a machine whose GPU recovers gets +the field back next session). Dream, Observatory takeover, Memory Cinema, and Reload work in both modes. ### Changed — `smart_ingest` batch mode honors an explicit `forceCreate: false` @@ -88,9 +99,15 @@ three is 100% finished. The remaining edges are stated per item. zero production callers, so `agent_traces` never populated. `handle_tools_call` now records the opening `mcp.call` event before dispatch and the memory events after, under a shared `run_id` — including in pure - stdio mode (was gated on a dashboard socket). **Not yet wired**: the - receipts and memory-PR producers — those Black Box surfaces still have no - production writer and stay empty for now. + stdio mode (was gated on a dashboard socket). **Tracing is on by default** + and writes rows to your local database on every MCP tool call; set + `VESTIGE_TRACE=0` (or `false`/`off`/`no`) to turn the recorder off. **Traces + are pruned after 30 days by default** — the consolidation cycle sweeps + `agent_traces` older than the retention window and drops any `agent_runs` + roll-up left with no events. Override with `VESTIGE_TRACE_RETENTION_DAYS`; + `0` keeps traces forever (sweep disabled). **Not yet wired**: the receipts + and memory-PR producers — those Black Box surfaces still have no production + writer and stay empty for now. - **Recurring intentions re-arm (#124)**: `mark_triggered` advances a recurring trigger's `next_occurrence` and returns it to Active (was firing once and staying Triggered forever). **Not yet done**: the re-arm logic is @@ -125,7 +142,11 @@ are replaced (`898bd33`) with real dashboard HTTP endpoints backed by existing core capabilities: `GET /api/duplicates` (dedup cluster detection), `GET /api/contradictions` (trust-weighted contradiction analysis), and `GET /api/patterns/cross-project` (cross-project pattern transfer). The "Live" -badge is now truthful. +badge is now truthful. Two more honesty fixes on the same routes: `/duplicates` +loses its no-op "Merge all" button (the copy now points at the `dedup` MCP +tool, which actually performs previewable, reversible merges), and the +`/contradictions` stat card reports the number of memories actually analyzed +instead of an inflated total. ### Fixed — Auto-consolidation merge: opt-out lever + protected pins honored (#142, via #143) @@ -161,12 +182,36 @@ ComposeBench roadmap note now points at the silent-rotation harness. ### Upgrade notes -- **Migration V19 runs automatically** and adds `source_project` to the - connector idempotency key and unique index. If you synced **two or more +- **BREAKING for 2.2.x cloud-sync users: encryption is no longer optional.** + In 2.2.x, `VESTIGE_CLOUD_ENCRYPTION_KEY` was optional and `vestige sync + --cloud` would upload a plaintext archive when it was unset. 2.3.0 makes + zero-knowledge encryption mandatory in **both** directions: it refuses to + start a cloud sync without a passphrase, and on download it **hard-rejects a + plaintext remote archive** ("refusing plaintext cloud archive") rather than + quietly accepting it. So if you synced plaintext under 2.2.x, 2.3.0 will not + read that blob. Recovery: on the machine that **still holds the local data**, + set `VESTIGE_CLOUD_ENCRYPTION_KEY` to a strong passphrase and re-run `vestige + sync --cloud` to re-upload an encrypted archive, then set that same + passphrase on every other device. Vestige never receives the passphrase and + cannot reset it for you. + +- **Black Box tracing is on by default and writes to your database.** Every MCP + tool call now records trace rows (`agent_traces` / `agent_runs`). Set + `VESTIGE_TRACE=0` (or `false`/`off`/`no`) to opt out. **Retention changed + too**: traces older than **30 days** are pruned during the consolidation + cycle; tune with `VESTIGE_TRACE_RETENTION_DAYS`, where `0` keeps them + forever. + +- **Migrations V19 and V20 run automatically.** V19 adds `source_project` to + the connector idempotency key and unique index. If you synced **two or more projects of the same source system** (e.g. two GitHub repos) under 2.2.x, - their overlapping issue ids clobbered each other's memories. After - upgrading, re-run `source_sync` once **per project** so each project's - records are re-indexed under its own key. + their overlapping issue ids clobbered each other's memories. V20 then clears + the connector sync cursors, so the next `source_sync` for each project does a + full re-scan and repairs the clobbered rows **automatically** — there is no + manual step beyond running your normal sync. One caveat for large projects: a + single `source_sync` run is bounded by `max_pages`, which defaults to `10`, so + a big backlog may take more than one call (or one call with a higher + `max_pages`) before the re-scan is complete. ### Credits diff --git a/Cargo.lock b/Cargo.lock index 4a05b8e..8d0c27f 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -456,7 +456,7 @@ dependencies = [ "num_cpus", "objc2-foundation", "objc2-metal", - "rand", + "rand 0.9.3", "rand_distr", "rayon", "safetensors 0.7.0", @@ -569,6 +569,17 @@ dependencies = [ "cpufeatures 0.2.17", ] +[[package]] +name = "chacha20" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.0", + "rand_core 0.10.1", +] + [[package]] name = "chacha20poly1305" version = "0.10.1" @@ -576,7 +587,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "10cd79432192d1c0f4e1a0fef9527696cc039165d729fb41b3f4f4f354c2dc35" dependencies = [ "aead", - "chacha20", + "chacha20 0.9.1", "cipher", "poly1305", "zeroize", @@ -861,9 +872,9 @@ dependencies = [ [[package]] name = "crossbeam-epoch" -version = "0.9.18" +version = "0.9.20" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e" +checksum = "2d6914041f254d6e9176c01941b21115dcfb7089e55135a35411081bd106ef3f" dependencies = [ "crossbeam-utils", ] @@ -1348,7 +1359,7 @@ checksum = "c2d1f04709a8ac06e8e8042875a3c466cc4832d3c1a18dbcb9dba3c6e83046bc" dependencies = [ "half", "num-traits", - "rand", + "rand 0.9.3", "rand_distr", ] @@ -1769,11 +1780,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" dependencies = [ "cfg-if", - "js-sys", "libc", "r-efi 5.3.0", "wasip2", - "wasm-bindgen", ] [[package]] @@ -1783,10 +1792,13 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0de51e6874e94e7bf76d726fc5d13ba782deca734ff60d5bb2fb2607c7406555" dependencies = [ "cfg-if", + "js-sys", "libc", "r-efi 6.0.0", + "rand_core 0.10.1", "wasip2", "wasip3", + "wasm-bindgen", ] [[package]] @@ -1849,7 +1861,7 @@ dependencies = [ "cfg-if", "crunchy", "num-traits", - "rand", + "rand 0.9.3", "rand_distr", "zerocopy", ] @@ -1915,7 +1927,7 @@ dependencies = [ "libc", "log", "native-tls", - "rand", + "rand 0.9.3", "reqwest", "serde", "serde_json", @@ -3480,14 +3492,15 @@ dependencies = [ [[package]] name = "quinn-proto" -version = "0.11.14" +version = "0.11.16" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "434b42fec591c96ef50e21e886936e66d3cc3f737104fdb9b737c40ffb94c098" +checksum = "2f4bfc015262b9df63c8845072ce59068853ff5872180c2ce2f13038b970e560" dependencies = [ "bytes", - "getrandom 0.3.4", + "getrandom 0.4.2", "lru-slab", - "rand", + "rand 0.10.2", + "rand_pcg", "ring", "rustc-hash", "rustls", @@ -3544,6 +3557,17 @@ dependencies = [ "rand_core 0.9.5", ] +[[package]] +name = "rand" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80" +dependencies = [ + "chacha20 0.10.1", + "getrandom 0.4.2", + "rand_core 0.10.1", +] + [[package]] name = "rand_chacha" version = "0.9.0" @@ -3572,6 +3596,12 @@ dependencies = [ "getrandom 0.3.4", ] +[[package]] +name = "rand_core" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" + [[package]] name = "rand_distr" version = "0.5.1" @@ -3579,7 +3609,16 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6a8615d50dcf34fa31f7ab52692afec947c4dd0ab803cc87cb3b0b4570ff7463" dependencies = [ "num-traits", - "rand", + "rand 0.9.3", +] + +[[package]] +name = "rand_pcg" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "caa0f4137e1c0a72f4c651489402276c8e8e1cf081f3b0ba156d2cbeef09e86a" +dependencies = [ + "rand_core 0.10.1", ] [[package]] @@ -3609,7 +3648,7 @@ dependencies = [ "num-traits", "paste", "profiling", - "rand", + "rand 0.9.3", "rand_chacha", "simd_helpers", "thiserror 2.0.18", @@ -3874,9 +3913,9 @@ dependencies = [ [[package]] name = "rustls-webpki" -version = "0.103.11" +version = "0.103.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "20a6af516fea4b20eccceaf166e8aa666ac996208e8a644ce3ef5aa783bc7cd4" +checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e" dependencies = [ "ring", "rustls-pki-types", @@ -4395,7 +4434,7 @@ dependencies = [ "monostate", "onig", "paste", - "rand", + "rand 0.9.3", "rayon", "rayon-cond", "regex", @@ -4631,7 +4670,7 @@ dependencies = [ "http", "httparse", "log", - "rand", + "rand 0.9.3", "sha1", "thiserror 2.0.18", "utf-8", diff --git a/README.md b/README.md index ac2d11f..921d3cd 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ Local-first long-term memory for AI agents, delivered over MCP. Vestige remember [![Binary](https://img.shields.io/badge/binary-25MB_single_file-informational)](https://github.com/samvallad33/vestige/releases/latest) [![License](https://img.shields.io/badge/license-AGPL--3.0-3b82f6)](LICENSE) -[What it is](#what-vestige-is) · [Install](#install) · [First interaction](#your-first-real-interaction) · [vs RAG](#how-it-differs-from-rag) · [Backward reach](#backward-reach-the-backfill-feature) · [Benchmark](#silent-rotation-a-reproducible-benchmark) · [Science](#the-science) · [Tools](#the-13-tools) · [Dashboard](#the-dashboard) · [Integrations](#works-with-every-agent) · [Docs](#go-deeper) +[What it is](#what-vestige-is) · [Install](#install) · [First interaction](#your-first-real-interaction) · [vs RAG](#how-it-differs-from-rag) · [Backward reach](#backward-reach-the-backfill-feature) · [Benchmark](#silent-rotation-a-reproducible-benchmark) · [Science](#the-science) · [Tools](#the-13-tools) · [Dashboard](#the-dashboard) · [Integrations](#works-with-every-agent) · [Pro](#vestige-pro) · [Docs](#go-deeper) --- @@ -248,6 +248,26 @@ This is opt-in. Vestige works fine with no protocol at all. --- +## Vestige Pro + +Everything above is free forever and never metered. The engine runs on your machine, with no account, no quota, and no upsell inside the product. + +Vestige Pro is for when that memory needs to follow you. It is managed, end-to-end encrypted continuity of your memory graph and your accountability history (Black Box traces, receipts, memory PRs) across every machine you work on. You record a decision on the laptop, and the agent on the desktop already knows it. + +| | Detail | +|---|---| +| Price | $19/month | +| What syncs | Your memory graph plus your accountability history | +| Encryption | XChaCha20-Poly1305, applied on your machine before anything is uploaded | +| Key derivation | Argon2id over a passphrase you choose | +| What the server holds | Ciphertext only | + +Zero-knowledge is the design, not a setting. You pick one passphrase, you use the same one on every device, and it never leaves your machine. The server stores bytes it cannot read, and the client refuses to sync anything in plaintext. If you lose that passphrase, the encrypted data is unrecoverable, by me and by anyone else. That is the property you are paying for, not a gap in it. + +**Availability.** Checkout is not open yet, so there is nothing to buy today and no payment link here pretending otherwise. The client half already ships in this release, which is why `vestige sync --cloud` exists and tells you what it needs. Subscriptions open shortly. To catch the announcement, watch [Releases](https://github.com/samvallad33/vestige/releases) or follow [Discussions](https://github.com/samvallad33/vestige/discussions). + +--- + ## Under the hood Vestige is a single Rust binary. No sidecar services, no external database, no cloud dependency. diff --git a/SECURITY.md b/SECURITY.md index c130b7b..dbc9abc 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -4,8 +4,10 @@ | Version | Supported | | ------- | ------------------ | -| 2.1.x | :white_check_mark: | -| 2.0.x | Critical fixes only | +| 2.3.x | :white_check_mark: | +| 2.2.x | Critical fixes only | +| 2.1.x | Critical fixes only | +| 2.0.x | :x: | | 1.x | :x: | ## Reporting a Vulnerability @@ -70,10 +72,38 @@ All MCP tool inputs are validated: ### Dependencies -We use well-maintained dependencies and run `cargo audit` regularly. Current status: +We use well-maintained dependencies and run `cargo audit` as part of the release gate. Status as of the v2.3.0 audit (2026-07-25, 564 crate dependencies scanned): - **Vulnerabilities**: 0 -- **Warnings**: 2 (unmaintained transitive dependencies with no known CVEs) +- **Warnings**: 9 (3 unmaintained, 5 unsound, 1 yanked), all transitive, none with a known exploit path in Vestige + +That zero is not the default state. The 2.3.0 lockfile carried 5 vulnerabilities +until three transitive dependencies were updated. Resolved for this release: + +| Advisory | Crate | Issue | Resolved by | +|---|---|---|---| +| RUSTSEC-2026-0185 (7.5 high) | `quinn-proto` | Remote memory exhaustion from unbounded out-of-order stream reassembly | 0.11.14 -> 0.11.16 | +| RUSTSEC-2026-0104 | `rustls-webpki` | Reachable panic parsing a certificate revocation list | 0.103.11 -> 0.103.13 | +| RUSTSEC-2026-0098 | `rustls-webpki` | Name constraints for URI names incorrectly accepted | 0.103.11 -> 0.103.13 | +| RUSTSEC-2026-0099 | `rustls-webpki` | Name constraints accepted for certificates asserting a wildcard name | 0.103.11 -> 0.103.13 | +| RUSTSEC-2026-0204 | `crossbeam-epoch` | Invalid pointer dereference in the `fmt::Pointer` impl | 0.9.18 -> 0.9.20 | + +`rustls-webpki` is the certificate validation path that Vestige Pro cloud sync +depends on, so these versions are pinned forward in `Cargo.lock` and a downgrade +is a release blocker, not a lockfile detail. + +The 9 remaining warnings are not vulnerabilities and have no available fix we can +apply, because each is pulled in transitively by `fastembed`, `candle`, +`usearch`, or `git2`: + +- **Unmaintained**: `core2` (RUSTSEC-2026-0105, also yanked), `number_prefix` + (RUSTSEC-2025-0119), `paste` (RUSTSEC-2024-0436) +- **Unsound**: `anyhow` (RUSTSEC-2026-0190), `cxx` (RUSTSEC-2026-0202), `git2` + (RUSTSEC-2026-0183, RUSTSEC-2026-0184), `memmap2` (RUSTSEC-2026-0186) + +We track these and will pull the fixes through as soon as our direct +dependencies publish releases that carry them. We would rather print the real +number here than a zero. ## Security Checklist diff --git a/crates/vestige-core/src/storage/cloud_sync.rs b/crates/vestige-core/src/storage/cloud_sync.rs index 2fb37e3..9eccfbe 100644 --- a/crates/vestige-core/src/storage/cloud_sync.rs +++ b/crates/vestige-core/src/storage/cloud_sync.rs @@ -38,12 +38,40 @@ const REQUEST_TIMEOUT: Duration = Duration::from_secs(60); /// derives the namespace from the key), so the client uses a fixed path. const BLOB_PATH: &str = "/v1/blob"; +/// Minimum zero-knowledge passphrase length. The server stores ciphertext it can +/// never decrypt, so the only attack left is offline brute force by whoever holds +/// that ciphertext. Argon2id slows each guess down; it cannot rescue a passphrase +/// with a handful of characters of entropy. +const MIN_ENCRYPTION_KEY_LEN: usize = 12; + +/// Hosts allowed to be reached over plaintext `http://`: loopback only, for local +/// development and the test mock. Everything else must be `https://`. +const LOOPBACK_HOSTS: [&str; 3] = ["127.0.0.1", "localhost", "[::1]"]; + +/// Whether it is safe to present the bearer sync key to `endpoint`. +/// +/// `https://` anywhere is fine. Plain `http://` is accepted only for a loopback +/// host, and only when the host ends there (so `http://127.0.0.1.evil.test` is +/// still rejected). +fn is_transport_secure(endpoint: &str) -> bool { + let lower = endpoint.to_ascii_lowercase(); + if lower.starts_with("https://") { + return true; + } + let Some(rest) = lower.strip_prefix("http://") else { + return false; + }; + LOOPBACK_HOSTS.iter().any(|host| { + rest.strip_prefix(host) + .is_some_and(|tail| tail.is_empty() || tail.starts_with(':') || tail.starts_with('/')) + }) +} + /// HTTP-backed portable sync backend for Vestige Cloud. /// /// Mirrors the shape of /// [`FilePortableSyncBackend`](super::sqlite::FilePortableSyncBackend) but reads /// and writes the archive over HTTPS with a per-user bearer key. -#[derive(Debug)] pub struct HttpPortableSyncBackend { /// Base endpoint, e.g. `https://sync.vestige.dev`. No trailing slash. endpoint: String, @@ -61,6 +89,20 @@ pub struct HttpPortableSyncBackend { last_etag: RefCell>, } +/// Hand-written so neither secret can be leaked by a `{:?}` in a log line, an +/// error chain, or a panic message. The bearer key and the zero-knowledge +/// passphrase are the two values that must never be printed. +impl std::fmt::Debug for HttpPortableSyncBackend { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("HttpPortableSyncBackend") + .field("endpoint", &self.endpoint) + .field("sync_key", &"") + .field("encryption_key", &"") + .field("last_etag", &self.last_etag) + .finish_non_exhaustive() + } +} + impl HttpPortableSyncBackend { /// Plaintext cloud sync is deliberately disabled. /// @@ -73,8 +115,11 @@ impl HttpPortableSyncBackend { /// /// The portable archive is encrypted with XChaCha20-Poly1305 /// (Argon2id-derived key) before upload and decrypted on download. The - /// passphrase never leaves this process. Missing or blank passphrases are - /// rejected rather than silently uploading plaintext. + /// passphrase never leaves this process. Missing, blank, or short + /// passphrases are rejected rather than silently uploading weakly protected + /// ciphertext, and a non-`https://` endpoint is rejected rather than + /// silently putting the bearer sync key on the wire in cleartext (plain + /// `http://` loopback is allowed for local development). pub fn new_with_encryption( endpoint: impl Into, sync_key: impl Into, @@ -87,6 +132,13 @@ impl HttpPortableSyncBackend { "cloud sync endpoint is empty (set VESTIGE_CLOUD_ENDPOINT)".to_string(), )); } + if !is_transport_secure(&endpoint) { + return Err(StorageError::Init(format!( + "cloud sync endpoint must use https:// (got `{endpoint}`). Plain http:// would \ + put your sync key on the wire in cleartext; only http://127.0.0.1 and \ + http://localhost are allowed, for local development." + ))); + } if sync_key.is_empty() { return Err(StorageError::Init( "cloud sync key is empty (set VESTIGE_CLOUD_SYNC_KEY)".to_string(), @@ -101,6 +153,15 @@ impl HttpPortableSyncBackend { .to_string(), ) })?; + if encryption_key.trim().chars().count() < MIN_ENCRYPTION_KEY_LEN { + return Err(StorageError::Init(format!( + "cloud sync encryption key must be at least {MIN_ENCRYPTION_KEY_LEN} characters. \ + The server only ever holds ciphertext, so anyone who obtains it can guess your \ + passphrase offline as fast as their hardware allows — Argon2id slows that down, \ + it does not make a short passphrase safe. Set VESTIGE_CLOUD_ENCRYPTION_KEY to a \ + long passphrase you can remember." + ))); + } let client = Client::builder() .timeout(REQUEST_TIMEOUT) .user_agent(concat!("vestige-cloud-sync/", env!("CARGO_PKG_VERSION"))) @@ -347,6 +408,83 @@ mod tests { assert!(encrypted_backend("https://x", "key").is_encrypted()); } + #[test] + fn debug_redacts_sync_key_and_passphrase() { + let be = encrypted_backend("https://sync.example", "super-secret-sync-key"); + let rendered = format!("{be:?}"); + assert!(!rendered.contains("super-secret-sync-key"), "{rendered}"); + assert!(!rendered.contains(TEST_PASSPHRASE), "{rendered}"); + assert!(rendered.contains(""), "{rendered}"); + // Non-secret fields stay visible for debugging. + assert!(rendered.contains("https://sync.example"), "{rendered}"); + } + + #[test] + fn new_rejects_plaintext_http_endpoint() { + // Remote plaintext would leak the bearer key on the wire. + let err = HttpPortableSyncBackend::new_with_encryption( + "http://sync.example", + "key", + Some(TEST_PASSPHRASE.to_string()), + ) + .unwrap_err(); + assert!(err.to_string().contains("must use https://"), "{err}"); + + // A loopback-lookalike host is not loopback. + assert!( + HttpPortableSyncBackend::new_with_encryption( + "http://127.0.0.1.evil.test", + "key", + Some(TEST_PASSPHRASE.to_string()), + ) + .is_err() + ); + // Schemeless / other schemes are rejected too. + assert!( + HttpPortableSyncBackend::new_with_encryption( + "sync.example", + "key", + Some(TEST_PASSPHRASE.to_string()), + ) + .is_err() + ); + + // Loopback carve-out for local development and the test mock. + assert!(encrypted_backend("http://127.0.0.1:8080", "key").is_encrypted()); + assert!(encrypted_backend("http://localhost:8080", "key").is_encrypted()); + assert!(encrypted_backend("https://sync.example", "key").is_encrypted()); + } + + #[test] + fn new_rejects_short_encryption_key() { + let err = HttpPortableSyncBackend::new_with_encryption( + "https://sync.example", + "key", + Some("a".to_string()), + ) + .unwrap_err(); + assert!(err.to_string().contains("at least 12 characters"), "{err}"); + + // Padding with whitespace does not buy entropy. + assert!( + HttpPortableSyncBackend::new_with_encryption( + "https://sync.example", + "key", + Some(" short ".to_string()), + ) + .is_err() + ); + // Exactly at the floor is accepted. + assert!( + HttpPortableSyncBackend::new_with_encryption( + "https://sync.example", + "key", + Some("abcdefghijkl".to_string()), + ) + .is_ok() + ); + } + #[test] fn endpoint_trailing_slash_trimmed() { let be = encrypted_backend("https://sync.example/", "k"); diff --git a/crates/vestige-core/src/storage/migrations.rs b/crates/vestige-core/src/storage/migrations.rs index 3440f61..a17a84c 100644 --- a/crates/vestige-core/src/storage/migrations.rs +++ b/crates/vestige-core/src/storage/migrations.rs @@ -99,6 +99,11 @@ pub const MIGRATIONS: &[Migration] = &[ description: "Scope the source idempotency key by source_project so same-system sources with overlapping ids no longer clobber each other", up: MIGRATION_V19_UP, }, + Migration { + version: 20, + description: "Finish the V19 repair: clear connector sync cursors so the next source_sync re-scans from scratch and re-keys every clobbered record", + up: MIGRATION_V20_UP, + }, ]; /// A database migration @@ -1155,6 +1160,36 @@ CREATE UNIQUE INDEX IF NOT EXISTS idx_nodes_source_key UPDATE schema_version SET version = 19, applied_at = datetime('now'); "#; +/// V20: make the V19 repair actually happen instead of asking the user to do it. +/// +/// V19 fixed the idempotency *key* going forward, but every record that had +/// already been clobbered under the old `(source_system, source_id)` key stays +/// clobbered until the connector re-fetches it. Re-running `source_sync` does +/// NOT re-fetch it: `run_sync` resumes from the saved per-(system, scope) +/// cursor minus only `CURSOR_OVERLAP_SECS` (see +/// [`crate::connectors::run_sync`]), so any record not touched upstream since +/// the last pre-upgrade sync falls outside the `since` window forever — while +/// the `SyncReport` still looks clean. +/// +/// Clearing the checkpoints makes the next sync start from `since = None`, i.e. +/// a full re-scan that re-upserts every record under the corrected key. +/// `upsert_by_source` is idempotent (Created / Updated / Unchanged, keyed by +/// `content_hash`), so the re-scan costs bandwidth and nothing else: untouched +/// records come back `Unchanged`. +/// +/// This is a NEW migration rather than an edit to V19 on purpose — a database +/// that already has V19 applied would never re-run it. +const MIGRATION_V20_UP: &str = r#" +-- Drop every incremental-sync checkpoint. `get_connector_cursor` returns a +-- zeroed cursor when no row exists, so the next run starts from the beginning +-- and re-keys the rows V19 could only protect going forward. Only sync +-- bookkeeping lives here (cursor / last-synced / reconcile / counters), so no +-- memory content is touched. +DELETE FROM connector_cursors; + +UPDATE schema_version SET version = 20, applied_at = datetime('now'); +"#; + /// Apply pending migrations /// /// Each migration is applied inside an explicit transaction so its schema @@ -1614,6 +1649,131 @@ mod tests { ); } + /// Apply every migration up to and including `through`, mirroring the + /// per-version ALTER TABLE handling `apply_migrations` does for V14/V16/V17. + /// Lets a test build a database that is genuinely *at* an older schema + /// version rather than rewinding `schema_version` on a current one. + fn apply_migrations_through(conn: &rusqlite::Connection, through: u32) { + for migration in MIGRATIONS { + if migration.version > through { + break; + } + if migration.version == 14 { + add_column_if_missing( + conn, + "ALTER TABLE knowledge_nodes ADD COLUMN protected INTEGER NOT NULL DEFAULT 0", + ) + .expect("V14 protected column"); + add_column_if_missing( + conn, + "ALTER TABLE knowledge_nodes ADD COLUMN superseded_by TEXT", + ) + .expect("V14 superseded_by column"); + } + if migration.version == 16 { + for stmt in MIGRATION_V16_ALTER_COLUMNS { + add_column_if_missing(conn, stmt).expect("V16 alter column"); + } + } + if migration.version == 17 { + for stmt in MIGRATION_V17_ALTER_COLUMNS { + add_column_if_missing(conn, stmt).expect("V17 alter column"); + } + } + conn.execute_batch(migration.up).expect("apply migration"); + } + } + + fn cursor_row_count(conn: &rusqlite::Connection) -> i64 { + conn.query_row("SELECT COUNT(*) FROM connector_cursors", [], |row| row.get(0)) + .expect("count connector_cursors") + } + + fn seed_connector_cursor(conn: &rusqlite::Connection, system: &str, scope: &str) { + conn.execute( + "INSERT INTO connector_cursors \ + (source_system, scope, cursor_updated_at, last_synced_at, records_seen) \ + VALUES (?1, ?2, '2026-07-01T00:00:00+00:00', '2026-07-01T00:00:00+00:00', 42)", + [system, scope], + ) + .expect("seed connector cursor"); + } + + /// V20 must empty `connector_cursors` on a database that is already at V19, + /// so the next `source_sync` starts from `since = None` and re-upserts every + /// record under the project-scoped key. Without this, records not touched + /// upstream since the last pre-upgrade sync stay clobbered forever — the + /// changelog's "just re-run source_sync" advice cannot work on its own, + /// because `run_sync` resumes from the saved cursor. + #[test] + fn v20_clears_connector_cursors_on_a_v19_database() { + let conn = rusqlite::Connection::open_in_memory().expect("open in-memory"); + apply_migrations_through(&conn, 19); + assert_eq!( + get_current_version(&conn).expect("version"), + 19, + "test fixture must be a genuine V19 database" + ); + + // Pre-upgrade sync state: checkpoints that would otherwise pin the next + // run's `since` window past the clobbered records. + seed_connector_cursor(&conn, "github", "octocat/repoA"); + seed_connector_cursor(&conn, "github", "octocat/repoB"); + assert_eq!(cursor_row_count(&conn), 2); + + let applied = apply_migrations(&conn).expect("V20 applies on a V19 database"); + assert_eq!(applied, 1, "exactly V20 should apply on a V19 database"); + assert_eq!( + get_current_version(&conn).expect("version"), + MIGRATIONS.last().unwrap().version + ); + assert_eq!( + cursor_row_count(&conn), + 0, + "V20 must clear every connector cursor so the next sync full re-scans" + ); + } + + /// Fresh database: all migrations apply cleanly through V20 and the cursor + /// table exists and is empty (nothing to clear, no error). + #[test] + fn v20_applies_cleanly_on_a_fresh_database() { + let conn = rusqlite::Connection::open_in_memory().expect("open in-memory"); + apply_migrations(&conn).expect("fresh migrations succeed through V20"); + assert_eq!( + get_current_version(&conn).expect("version"), + 20, + "latest migration must be V20" + ); + assert_eq!(cursor_row_count(&conn), 0); + } + + /// V20 is replayable: re-running it after new cursors were saved simply + /// clears them again, and a no-op `apply_migrations` on a current DB leaves + /// live cursors alone. + #[test] + fn v20_is_replayable_and_does_not_touch_cursors_when_already_applied() { + let conn = rusqlite::Connection::open_in_memory().expect("open in-memory"); + apply_migrations(&conn).expect("first apply"); + + // A post-upgrade sync saves a fresh checkpoint. + seed_connector_cursor(&conn, "github", "octocat/repoA"); + + // Already-current DB: no migration runs, so the live cursor survives. + assert_eq!(apply_migrations(&conn).expect("no-op replay"), 0); + assert_eq!(cursor_row_count(&conn), 1); + + // Forced replay from V19 clears it again without erroring. + conn.execute("UPDATE schema_version SET version = 19", []) + .expect("rewind to 19"); + apply_migrations(&conn).expect("V20 replay must be idempotent"); + assert_eq!(cursor_row_count(&conn), 0); + assert_eq!( + get_current_version(&conn).expect("version"), + MIGRATIONS.last().unwrap().version + ); + } + #[test] fn v16_preserves_existing_rows_from_v15() { let conn = rusqlite::Connection::open_in_memory().expect("open in-memory"); diff --git a/crates/vestige-core/src/storage/trace_store.rs b/crates/vestige-core/src/storage/trace_store.rs index da9e31a..8d5223e 100644 --- a/crates/vestige-core/src/storage/trace_store.rs +++ b/crates/vestige-core/src/storage/trace_store.rs @@ -18,6 +18,29 @@ use super::sqlite::SqliteMemoryStore; use super::{Result, StorageError}; use crate::trace::{MemoryPr, MemoryPrAction, MemoryPrStatus, MemoryTraceEvent, Receipt}; +/// Trace retention window used when `VESTIGE_TRACE_RETENTION_DAYS` is unset or +/// unusable. +const DEFAULT_TRACE_RETENTION_DAYS: i64 = 30; + +/// Upper bound on the configurable trace retention window (100 years). Beyond +/// this the window is meaningless — `0` is the documented "keep forever" — and +/// unbounded values overflow the `chrono::Duration` the sweep builds from them, +/// which panics (and the release profile aborts on panic). +const MAX_TRACE_RETENTION_DAYS: i64 = 36_500; + +/// Parse the `VESTIGE_TRACE_RETENTION_DAYS` value into a usable retention +/// window. Unset, empty, negative, and malformed values fall back to +/// [`DEFAULT_TRACE_RETENTION_DAYS`]; values above +/// [`MAX_TRACE_RETENTION_DAYS`] are clamped. Split out from +/// [`SqliteMemoryStore::prune_agent_traces`] so the parsing rules are testable +/// without touching process-global environment state. +fn resolve_trace_retention_days(raw: Option<&str>) -> i64 { + raw.and_then(|v| v.trim().parse::().ok()) + .filter(|d| *d >= 0) + .map(|d| d.min(MAX_TRACE_RETENTION_DAYS)) + .unwrap_or(DEFAULT_TRACE_RETENTION_DAYS) +} + /// A roll-up summary of one agent run, for the Black Box run list. #[derive(Debug, Clone, serde::Serialize, serde::Deserialize, PartialEq)] pub struct AgentRunSummary { @@ -216,15 +239,13 @@ impl SqliteMemoryStore { /// /// Retention defaults to 30 days. `VESTIGE_TRACE_RETENTION_DAYS` overrides /// it; `0` keeps traces forever (sweep disabled); unset, empty, negative, - /// or malformed values fall back to the default. Returns the number of - /// trace events deleted. + /// or malformed values fall back to the default, and absurdly large values + /// are clamped to [`MAX_TRACE_RETENTION_DAYS`]. Returns the number of trace + /// events deleted. pub fn prune_agent_traces(&self) -> Result { - const DEFAULT_TRACE_RETENTION_DAYS: i64 = 30; - let days = std::env::var("VESTIGE_TRACE_RETENTION_DAYS") - .ok() - .and_then(|v| v.trim().parse::().ok()) - .filter(|d| *d >= 0) - .unwrap_or(DEFAULT_TRACE_RETENTION_DAYS); + let days = resolve_trace_retention_days( + std::env::var("VESTIGE_TRACE_RETENTION_DAYS").ok().as_deref(), + ); self.prune_agent_traces_older_than_days(days) } @@ -234,7 +255,18 @@ impl SqliteMemoryStore { if days == 0 { return Ok(0); } - let cutoff_ms = (Utc::now() - chrono::Duration::days(days)).timestamp_millis(); + // Belt and braces against an out-of-range window: `Duration::days` + // panics on overflow and the release profile is panic = "abort", so an + // unclamped value would hard-kill the process from inside the periodic + // consolidation sweep. Skip the sweep instead — a cutoff that far back + // could not have deleted anything anyway. + let Some(cutoff) = chrono::Duration::try_days(days) + .and_then(|window| Utc::now().checked_sub_signed(window)) + else { + tracing::warn!("Trace retention of {days} days is out of range; skipping trace sweep"); + return Ok(0); + }; + let cutoff_ms = cutoff.timestamp_millis(); let writer = self .writer .lock() @@ -642,6 +674,89 @@ mod tests { assert!(s.get_agent_run("run_new").unwrap().is_some()); } + /// Retention parsing must survive hostile / fat-fingered env values. The + /// upper bound is the load-bearing one: an unclamped huge value overflows + /// the `chrono::Duration` the sweep builds, and `panic = "abort"` in the + /// release profile turns that into a hard process kill. + #[test] + fn trace_retention_env_value_is_clamped_and_never_overflows() { + // Sane values pass through untouched. + assert_eq!(resolve_trace_retention_days(Some("7")), 7); + assert_eq!(resolve_trace_retention_days(Some(" 90 ")), 90); + // 0 stays 0: the documented "keep forever" switch. + assert_eq!(resolve_trace_retention_days(Some("0")), 0); + // Unset / empty / malformed / negative fall back to the default. + for raw in [None, Some(""), Some(" "), Some("forever"), Some("-5")] { + assert_eq!( + resolve_trace_retention_days(raw), + DEFAULT_TRACE_RETENTION_DAYS, + "unusable value {raw:?} must fall back to the default" + ); + } + // Absurdly large values are clamped instead of overflowing. + for raw in ["999999999999", "9223372036854775807"] { + assert_eq!( + resolve_trace_retention_days(Some(raw)), + MAX_TRACE_RETENTION_DAYS, + "{raw} must be clamped, not passed through" + ); + } + } + + /// End-to-end: a hostile `VESTIGE_TRACE_RETENTION_DAYS` must not panic the + /// process, and must not sweep anything (the clamped window is 100 years, + /// far older than any trace). Also exercises the raw sweep at `i64::MAX` to + /// prove the second line of defence inside + /// `prune_agent_traces_older_than_days` holds even if a caller bypasses the + /// env clamp. + #[test] + fn prune_agent_traces_survives_hostile_retention_value() { + static ENV_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); + const KEY: &str = "VESTIGE_TRACE_RETENTION_DAYS"; + + let s = store(); + s.append_trace_event(&MemoryTraceEvent::McpCall { + run_id: "run_hostile".into(), + tool: "search".into(), + args_hash: "h".into(), + at: Utc::now().timestamp_millis(), + }) + .unwrap(); + + // Direct call: no clamp in play, so this is the overflow path itself. + assert_eq!( + s.prune_agent_traces_older_than_days(i64::MAX).unwrap(), + 0, + "an out-of-range window must skip the sweep, not panic" + ); + + // Env path: process env is global and unsafe to mutate under Rust 2024, + // so serialize on a local lock and restore the previous value. + let _guard = ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let previous = std::env::var_os(KEY); + unsafe { std::env::set_var(KEY, "999999999999") }; + let deleted = s.prune_agent_traces(); + unsafe { + match previous { + Some(prev) => std::env::set_var(KEY, prev), + None => std::env::remove_var(KEY), + } + } + + assert_eq!( + deleted.unwrap(), + 0, + "a 100-year clamped window sweeps nothing" + ); + assert_eq!( + s.get_trace("run_hostile").unwrap().len(), + 1, + "the fresh trace must survive" + ); + } + #[test] fn receipt_roundtrips() { let s = store(); diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 382eff4..7cecf26 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -47,9 +47,29 @@ Qwen3 currently uses Hugging Face Hub's Candle loader directly, so use the stand | `VESTIGE_CONSOLIDATION_INTERVAL_HOURS` | `6` | FSRS-6 decay cycle cadence | | `VESTIGE_BACKFILL_AUTOFIRE` | `on` | Retroactive Salience Backfill auto-fire during consolidation. On by default; set `0`/`false`/`off`/`no` to disable. The manual `backfill` tool + CLI stay available either way. When on, promotion is bounded (`stability = MIN(stability * 1.5, stability + 365)`) | | `VESTIGE_AUTO_CONSOLIDATE_MERGE` | `on` | Auto concat-merge of near-duplicate memories during consolidation (keeps the strongest, folds the rest in as `[MERGED]` blocks, deletes the originals). On by default; set `0`/`false`/`off`/`no` to disable. Protected (`dedup protect`) memories are never absorbed or deleted by this pass, on or off. | +| `VESTIGE_TRACE` | `on` | Agent Black Box trace recording. **On by default**: every MCP tool call writes rows to `agent_traces`/`agent_runs` in your local database. Set `0`/`false`/`off`/`no` to turn the recorder off. Read once per process, so changing it mid-process has no effect | +| `VESTIGE_TRACE_RETENTION_DAYS` | `30` | How long Black Box traces are kept. The consolidation cycle deletes trace events older than this and drops any `agent_runs` roll-up left with no events. `0` keeps traces forever (sweep disabled); unset, empty, negative, or malformed values fall back to `30` | +| `VESTIGE_DISABLE_VECTOR_SEARCH` | unset (vector search on) | Kill switch for the HNSW vector index. Set to `1`/`true`/`yes`/`on`/`enable`/`enabled` to force semantic/vector search off and fall back to keyword search. Useful on older x86 CPUs — the index also disables itself automatically when AVX2+FMA are missing | > **Storage location precedence:** `--data-dir ` wins over `VESTIGE_DATA_DIR`; if neither is set, Vestige uses your OS's per-user data directory: `~/Library/Application Support/com.vestige.core/` on macOS, `~/.local/share/vestige/core/` on Linux, `%APPDATA%\vestige\core\` on Windows. Custom paths are directories, are created if missing, expand a leading `~`, and store the database at `/vestige.db`. +### Vestige Pro (hosted cloud sync) + +These are read only by `vestige sync --cloud`. Leave them unset and Vestige stays fully local — nothing is uploaded and no network call is made. + +| Variable | Default | Description | +|----------|---------|-------------| +| `VESTIGE_CLOUD_ENDPOINT` | unset | Hosted managed-sync endpoint, issued when you subscribe. `--endpoint` on `vestige sync --cloud` takes precedence | +| `VESTIGE_CLOUD_SYNC_KEY` | unset | Per-user bearer key for the hosted service, issued when you subscribe. Authenticates the transport only — it is **not** the encryption passphrase | +| `VESTIGE_CLOUD_ENCRYPTION_KEY` | unset (**required** for cloud sync) | Passphrase for client-side zero-knowledge encryption (Argon2id KDF → XChaCha20-Poly1305, `VSTGENC1` envelope). Use the same passphrase on every device | + +> **The passphrase never leaves your machine.** The archive is encrypted on-device +> before upload and decrypted after download, so the hosted service only ever +> stores ciphertext. Vestige has no copy of `VESTIGE_CLOUD_ENCRYPTION_KEY` and +> **cannot reset or recover it** — if you lose it, the synced blob is +> unrecoverable by design. Encryption is mandatory: the client refuses to upload +> an unencrypted archive and rejects a plaintext archive on download. + --- ## Output Configuration (`vestige.toml`) @@ -165,6 +185,7 @@ vestige portable-export # Exact Vestige-to-Vestige archive vestige portable-import # Import exact archive into an empty database vestige portable-import --merge # Merge exact archive into this database vestige sync # Pull/merge/push through a file backend +vestige sync --cloud # Pull/merge/push through Vestige Pro (see cloud env vars) ``` ---