demo/README.md: the complete self-serve demo artifact — one-command run, the
seeded scenario explained, a "build your own scenario" section, the honest
boundary (won't invent a cause; can't reach a cause that was never recorded),
the Nature citation + the "field admits this is unsolved" sources, and the
recording playbook + paste-ready caption.
Writing/testing the README surfaced a real inconsistency, now fixed:
- The CLI's failure-finder used a hardcoded content-only marker subset and
ignored tags, so a "Checkout latency spiked" memory (regression tag, no crash
word in content) was never picked as the failure. The CLI now calls the SAME
public `looks_like_failure` (content + tags, full list) the backfill tool uses
— one definition, no drift.
- Extended FAILURE_MARKERS with performance/degradation failures (spiked,
latency, degraded, slow, hang, throttled, oom, 502/503/504, flaky, ...) so the
feature backfills from perf regressions, not just hard crashes.
clippy clean; 527 core + 453 mcp tests; both the main demo and the README's
custom scenario verified end-to-end.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
3-model rotation audit (DeepSeek V4-Pro / Kimi K2.7 / MiniMax M3, max thinking,
each model × each of 3 sections). Claude verified every finding against code.
CONFIRMED + FIXED:
- [FTS, consensus DeepSeek+MiniMax] sanitize_fts5_or_query split on
!is_alphanumeric()+'_', but the index uses tokenize='porter ascii' which
splits on '_' and non-ASCII. So "API_TIMEOUT"/"café" became single phrases that
could NEVER match. Now splits on !is_ascii_alphanumeric() + lowercases to mirror
the tokenizer; caps token count (64) and length (64) for DoS hardening. Also
fixes the pre-existing storage.search bug (multi-word queries silently returned
nothing). 5 new tests pin it.
- [Demo honesty, consensus Kimi+DeepSeek] the contrast labeled keyword search as
"SIMILARITY SEARCH" and asserted "NONE of these is the cause" universally. Now
prints the REAL engine ("keyword (BM25)" vs "semantic (vector + BM25 hybrid)")
and claims only what's true ("ranked by RESEMBLANCE; its top hit is a lookalike").
De-hardcoded the "Service crashed:" munging to a generic label-strip.
VERIFIED FALSE POSITIVE (not changed): MiniMax "fts.id non-existent column" —
the FTS5 table is declared `fts5(id, content, tags, ...)`, the JOIN is valid.
No injection found by any model (quote-doubling + operator-stripping confirmed safe).
clippy clean; 527 core + 453 mcp tests pass; demo verified.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CLI surface for Retroactive Salience Backfill so the seeded demo (and anyone who
clones it) can reproduce "memory with hindsight" from a terminal:
- `ingest --ago-days N`: backdate a memory N days (plant a dated cause/history).
- `backfill [--failure-id ID] [--manual] [--lookback-days N] [--no-promote]`:
reach backward from a failure and surface+promote the causal earlier memory,
with demo-grade colored output (↩ reached back N days, 🔗 causal join: <entity>,
similarity rank, ✅ promoted).
Verified live end-to-end on a real DB: plant a 3-day-old API_TIMEOUT env-var note
+ a semantically-similar 500 distractor + a crash, run `vestige backfill`, and it
surfaces the env-var note by the shared api_timeout entity (ignoring the similar
distractor) and promotes it. clippy clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The MCP surface for memory-with-hindsight. When a failure memory exists, the
`backfill` tool reaches backward across the real store and promotes the quiet
earlier cause that a vector search structurally cannot surface (not similar to
the failure, only causally upstream via a shared entity).
- tools/backfill.rs: builds BackfillCandidates from real KnowledgeNodes (entities
from tags + heuristic env-var/path/identifier extraction), computes real cosine
similarity from stored embeddings (to PROVE the cause ranks low on similarity),
runs the core RetroactiveBackfill, and promotes surfaced causes via
storage.promote_memory. Auto-finds the latest failure, or takes failure_id;
manual=true forces; promote=false for a dry run.
- registered + dispatched in server.rs (35 tools now); tool-list test updated.
- storage: added pub set_created_at (backdate created_at) so the demo/test can
plant a dated cause.
LIVE RECEIPT: live_backfill_surfaces_root_cause_through_storage ingests a
3-day-old API_TIMEOUT env-var note + a semantically-similar 500-error distractor
+ a crash into a REAL SQLite store, runs the backfill tool, and asserts it
surfaces + promotes the env-var note by the shared API_TIMEOUT entity (the root
cause RAG misses). clippy clean; 522 core + 453 mcp tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The headliner neuro-mechanism: when a salient FAILURE lands (bug/crash/
regression — the "aversive event"), reach BACKWARD in time and promote the
quiet earlier memory that caused it — the one a vector search structurally
cannot surface because it isn't *similar* to the failure, only causally upstream.
Faithful port of Zaki/Cai et al. 2024, Nature 637:145-155 ("Offline ensemble
co-reactivation links memories across days"), causally proven (hippocampal
silencing abolishes the linking). Ported faithfully:
- backward-only asymmetry (fear links retrospectively, never prospectively) —
also exactly correct for software: a root cause is always upstream in time.
- linking flows along the shared-entity overlap (same file/env-var/service),
NOT semantic similarity — that's the whole point (RAG already covers similarity).
- scoped to failure->backward-causal-backfill, not "all salience flows backward"
(mirrors the Cai aversive->neutral paradigm; honest about scope).
Trigger: auto-detect (high prediction-error + failure markers) OR manual override.
Promotion: boosts FSRS stability so the cause stops decaying and surfaces next time.
Receipt (4/4 tests): backfill_surfaces_the_cause_rag_misses proves it promotes a
sim=0.11 env-var note over a sim=0.82 distractor by the shared API_TIMEOUT entity;
backward-only (future memory never promoted); no shared entity => no fabricated
cause; non-salient doesn't fire; manual override works. clippy clean; 522 core
tests pass (no regressions).
Wires into existing primitives: prediction_error gate (salience), dreams/
consolidation (offline window), memory/strength (promotion). MCP tool + live
demo next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two deeper review findings (both blockers) + doc de-staling.
C2-deep: my earlier C2 made purge/delete TRACE as memory.write, but gate_writes
did `get_node(id) -> skip on None`, and purge had already DELETEd the row — so a
destructive removal still never opened a Memory PR (it was silently skipped).
The most security-critical write type couldn't be reviewed. Fix: a missing node
is now gateable for destructive decisions — gate_writes builds the WriteContext
from the decision itself (marks `forgets`, which classify_write gates), and the
PR records the removal with node.deleted=true. Proven live: purging a node opens
a PR (kind node_decayed, deleted true); test
gate_opens_pr_for_destructive_write_after_node_deleted_c2.
PRIV: gate_writes copied the FULL node.content into the PR diff + title, so a
real secret in a gated memory would leak into the memory_prs table, the
dashboard, and any exported proof bundle — defeating the point of gating
sensitive writes. Fix: the PR now stores a truncated content PREVIEW + an FNV
content HASH, and sensitive-topic/sensitive-node-type writes are fully REDACTED
("[redacted — sensitive content; review via risk signals]"). The reviewer still
sees the risk signals (why it opened) and a hash (to correlate), never the
secret. Tests gate_redacts_sensitive_content_in_pr_priv,
content_preview_redacts_sensitive_and_truncates, content_hash_is_stable. The
committed memory_pr.json + the whole proof bundle were re-captured and contain
no secret (verified by scan); the re-shot memory-prs.png shows the redaction.
DOC: REVIEW.md commit list is now git-log-based (no stale hashes); C2-deep + PRIV
added to the findings table; PROOF.md write/PR rows updated; test count -> 1007.
Gates: 1007 lib tests pass (+7 new regressions), clippy -D warnings clean,
dashboard check + build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two more review findings — both real, both blockers — plus stale-doc cleanup.
C1: the B1 release used reverse_suppression(subject_id, labile_hours), which
REFUSES once the 24h active-forgetting labile window has passed. So a Memory PR
reviewed late could be marked "promoted" while its memory stayed suppressed.
Approving a quarantined write is an explicit reviewer decision and must release
the memory regardless of elapsed time. New SqliteMemoryStore::release_quarantine
fully clears the suppression (count→0, suppressed_at→NULL) with NO time-window
limit; the PR handler now uses it. Proven: a test backdates suppressed_at to
+100h, shows reverse_suppression refuses, and release_quarantine still releases.
C2: memory(action="purge"|"delete") returns `action` + nodeId but those labels
weren't in is_write_decision, so destructive removal bypassed the memory.write
trace and the PR gate. Added purge/purged/delete/deleted/forget/forgotten.
Proven live: purging a node now records a second memory.write event
({"decision":"purge"}) under the run.
Docs: REVIEW.md de-staled — removed the fixed 140b15f diff-stat / "3 commits"
prose (it moved with each fix), listed all commits, added C1/C2 to the findings
table, updated the test count.
Gates: 1002 lib tests pass (+3 new regressions), clippy -D warnings clean,
dashboard check + build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A full multi-agent review found 7 real issues (4 blockers). All fixed + tested.
B1 (blocker): Promoting a Memory PR did not release the quarantined memory —
the UI said "promoted" while the memory stayed suppressed/out of retrieval.
act_on_memory_pr now calls reverse_suppression(subject_id) on accept actions;
MemoryPrAction::releases_memory() encodes the rule (promote/merge/supersede
release; forget/quarantine keep it held). Proven live: PR response
subjectReleased:true, SQLite suppression_count 0.
B2 (blocker): memory promote/demote (returns `action`, not `decision`) and
codebase remember_* writes bypassed the write-trace + PR gate. extract_writes
now reads `action` too, filtered by is_write_decision (reads like get/state
excluded); is_write_tool includes `codebase`.
B3 (blocker): receipt ids collided within a run (r_<date>_<runId> +
INSERT OR REPLACE overwrote earlier receipts). IDs are now
r_<date>_<runId8>_<unique6>; build() mints the suffix, build_with_unique()
keeps tests deterministic.
B4 (blocker): proof bundle was assembled from two runs (trace.json=run_proof,
websocket-events.jsonl=run_proof2). Re-captured the whole bundle from a single
run — trace, websocket, receipt, and memory_pr all carry run_proof now.
B5: Black Box receipts panel showed global latest, not the selected run.
Added list_receipts_for_run + /api/receipts?run= ; the page uses listForRun.
B6: SENSITIVE_TOPICS substring matching false-fired (tokenizer->token,
author->auth, secretary->secret). Switched to word-boundary matching; real
phrasings (auth token, security vulnerability, api key) still gate.
B7: set_review_mode now writes atomically (temp+rename via write_atomic);
export_trace sanitizes run_id in the Content-Disposition filename; memory-prs
static routes declared before the dynamic /{id} route.
Withdrawn: the /mode-vs-/{id} route order is NOT a functional bug (axum 0.8 /
matchit prioritizes static segments) — reordered for clarity only.
Gates: 999 lib tests pass (+9 new regressions), clippy -D warnings clean,
dashboard check + build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bounded follow-up (tight acceptance criteria, no scope expansion): flip the
dream.patch producer from "quiet because no dream ran" to a recorded live event.
The dream tool's `insights` array carries no per-item id, so the recorder
extracted zero proposals and dream.patch never fired even on a real dream.
Fix: derive a stable proposal id from each insight's REAL content (its
insight_type + the source memories it consolidated). The dream genuinely ran;
this just gives each real proposal a deterministic handle. Unit-tested against
the exact dream output shape.
Proven end to end (run_dream_proof, 6 memories consolidated):
- one dream.patch event: dream:RecurringPattern:5d941c7f+a41aca72+...
- SQLite + /api/traces/:runId: dream-trace.json (14 events, last is dream.patch)
- WebSocket: dream-websocket-events.jsonl (the dream.patch TraceEvent)
- dashboard: screenshots/dream-producers.png — the row flips to "fired this run"
PROOF.md updated: dream.patch moves from CAVEAT to REAL (still not live by
default — it fires only when a dream actually runs, and the UI says so).
sanhedrin.veto remains an honest CAVEAT (optional hook, off by default).
Gates: 957 lib tests pass, clippy -D warnings clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the receipt chain impossible to doubt. Freeze the claim surface, prove
every hop, and turn the two off-by-default producers into explicit UI states.
Frozen public claim: "Vestige records real MCP memory activity into a
replayable local trace, with receipts and reviewable risky writes." We do NOT
claim Sanhedrin vetoes or dream patches are live by default.
Regression — full-spine test (server.rs): one runId must cross, byte-identical,
MCP output -> SQLite trace -> WebSocket event -> API response shape ->
MCP resource. Fails if any hop drops or rewrites the id.
Honest UI states (Black Box "Event producers" panel):
- sanhedrin.veto -> "No veto producer connected (optional Sanhedrin hook, off
by default)" instead of empty mystery.
- dream.patch -> "No dream run in this trace" unless a dream actually ran.
- contradiction.detected -> "no contradiction in this run" when none fired.
Quarantine review (not pre-write blocking): risky writes are committed then
suppressed — audit history preserved, retrieval influence suspended until
reviewed. Reworded the server notice + UI copy to say exactly that.
Receipts UI gap closed: ReceiptCard is now mounted on the Black Box page
(retrieved/suppressed/trust-floor, activation path, "Open receipt in Cinema").
Proof pack (blackbox-proof-2026-06-22/): status.json, trace.json (the
.vestige-trace.json export), receipt.json, memory_pr.json (promoted via
UI->API->SQLite), websocket-events.jsonl (live TraceEvent x6 + PR opened/
decided), screenshots (Black Box, Receipts, Memory PRs, Graph), and PROOF.md
with real/caveat/stub per feature.
Gates: 988 lib tests pass, clippy -D warnings clean, dashboard check + build
clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The portable archive is encrypted on the client before upload and decrypted
after download, so the hosted service only ever stores ciphertext — true
zero-knowledge. The passphrase (VESTIGE_CLOUD_ENCRYPTION_KEY) is independent
of the bearer sync key and never leaves the device.
- new cloud_crypto module: Argon2id KDF + XChaCha20-Poly1305 AEAD, self-
describing envelope (MAGIC|version|salt|nonce|ciphertext+tag)
- HttpPortableSyncBackend encrypts on write / decrypts on read; transparent
upgrade of legacy plaintext archives; clear error if remote is encrypted
but no passphrase is set
- sync_portable_archive_cloud takes optional encryption_key
- CLI surfaces encryption status (on/off) on sync
- 6 crypto tests (roundtrip, wrong-key, tamper detection, non-determinism,
envelope detection); E2E verified: server blob is ciphertext, passphrase
device recovers, no-passphrase device cannot decrypt
491 core tests green, clippy -D warnings clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump all manifests 2.1.26 → 2.1.27 and date the CHANGELOG entry for the
GitHub + Redmine connector layer and source-aware search filters (#57, PR #78).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make Vestige a durable, local, semantically-searchable retrieval layer over an
external system of record (GitHub Issues first), citing back to the canonical
record. Unlike a live ticket-system MCP proxy, Vestige keeps a durable embedded
index: searchable offline, joinable with the rest of memory, temporally
versioned, and re-syncable idempotently with no duplication.
Phases 1-2 of #57 plus a GitHub reference connector and source-aware search:
- Source envelope on KnowledgeNode/IngestInput (source_system, source_id,
source_url, source_updated_at, content_hash, synced_at, source_project,
source_type, source_author). Migration V17: nullable columns (additive),
partial UNIQUE index on (source_system, source_id), connector_cursors table.
- Idempotent sync primitives in vestige-core: upsert_by_source (content-hash
change detection), connector cursor checkpoints, reconcile_source_tombstones
(invalidate-don't-delete via bitemporal valid_until).
- Connector contract + run_sync driver + GitHub Issues connector behind the
optional `connectors` feature (on by default in vestige-mcp, off in the core
library default so non-connector consumers link no HTTP client).
- source_sync MCP tool ({"repo": "owner/name"}); token from GITHUB_TOKEN env
only. Search results gain a sourceRecord citation for connector memories.
Adversarial review fixes: GitHub `since` Z-form (the `+00:00` offset corrupted
the cursor server-side), un-tombstone clears superseded_by too, cursor never
advances past a failing record, Link next-url host-pinned (token-leak guard),
records_seen counts new records only.
Verified: cargo check/test/clippy -D warnings green across the workspace
(default and connectors features); 483 core tests pass. Version bump to 2.1.27
and tag deferred to release.
Refs #57
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cargo rm async-trait. Last usage was the FastembedEmbedder impl attribute,
removed in the preceding 0001c commit; the MemoryStore side stopped using
async-trait at 0001a.
Verification: grep -rn async_trait crates/ returns zero hits. grep -rn
async-trait --include=Cargo.toml crates/ returns zero hits. Cargo.lock no
longer references the async-trait package.
Mirror of the 0001a pattern for the Embedder side.
- embedder/mod.rs: LocalEmbedder is the source trait declared with native
async-fn-in-trait. #[trait_variant::make(EmbedderSend: Send)] derives the
Send-bounded variant that backends implement. A hand-written Embedder
trait wraps each async method in BoxedEmbedderFuture<'a, T> and forwards
sync methods through a blanket impl<T: EmbedderSend> Embedder for T, so
Box<dyn Embedder> / Arc<dyn Embedder> stay dyn-safe -- trait_variant 0.1
alone does NOT produce a dyn-safe variant (RPITIT), so the hand-written
adapter is required.
- embedder/fastembed.rs: drop the #[async_trait::async_trait] attribute and
retarget the impl block to EmbedderSend. Adjust the top-level use to
bring EmbedderSend into scope (also keeps fastembed::tests' use super::*
trait lookups working).
- lib.rs: export EmbedderSend alongside the existing Embedder /
LocalEmbedder re-exports.
The async-trait Cargo dependency is dropped in a follow-up commit so the
manifest change stays visible on its own.
Verification: cargo test -p vestige-core --features embeddings,vector-search
(428) and --no-default-features (370) both green. cargo test --test
embedder_trait green (2/2 including Box<dyn Embedder> cast). cargo build
--workspace --release green. cargo clippy --workspace --features
embeddings,vector-search -- -D warnings clean. grep -rn async_trait crates/
returns zero.
Replaces #[async_trait::async_trait] on the storage trait with a
trait_variant-driven layout plus a hand-written dyn-compatible adapter.
- memory_store.rs: LocalMemoryStore is the source trait declared with
native async-fn-in-trait. #[trait_variant::make(MemoryStoreSend: Send)]
derives the Send-bounded variant that backends actually implement (the
blanket impl in 0.1.x goes variant -> source). A hand-written
MemoryStore trait wraps every method in
Pin<Box<dyn Future<Output = MemoryStoreResult<T>> + Send + 'a>> with
a BoxedStoreFuture<'a, T> alias, and a blanket impl<T: MemoryStoreSend>
MemoryStore for T adapts every Send-variant implementation. This keeps
Arc<dyn MemoryStore> dyn-safe for Phase 1 cognitive-module tests --
trait_variant 0.1 alone does NOT produce a dyn-safe variant (RPITIT),
so the hand-written adapter is required and supersedes the plan claim
that trait_variant gives dyn-compat for free.
- sqlite.rs: drop the #[async_trait::async_trait] attribute on the impl
block and retarget it to MemoryStoreSend. Two pre-existing clippy
issues that the macro had been masking are fixed in the same body
(return Ok(out) tail expression in vector_search; DomainRow tuple
alias in get_domain).
- mod.rs: export MemoryStoreSend alongside the existing LocalMemoryStore
and MemoryStore re-exports.
Verification: cargo test -p vestige-core --features embeddings,vector-search
passes (428 lib tests). All five Phase 1 integration test binaries pass
(trait_round_trip, send_bound_variant including
arc_dyn_memory_store_moves_across_tokio_tasks, cognitive_module_isolation,
embedding_model_registry, domain_column_migration). cargo test --workspace
green across every test binary. cargo build --workspace --release green.
cargo clippy --workspace --features embeddings,vector-search -- -D warnings
clean. grep -rn async_trait crates/vestige-core/src/storage/ returns
zero hits.
Supersedes plan claim in docs/plans/0001a-trait-rewrite.md about
trait_variant emitting a dyn-compatible Send variant; option (c) from
the design conversation (hand-written dyn adapter) was selected
explicitly because trait_variant 0.1.2 does not.
Introduce two trait boundaries that the rest of the stack now sits above,
landing Phase 1 of ADR 0001 (pluggable storage and network access).
Rebased onto v2.1.22 Sanhedrin from the original April work.
MemoryStore / LocalMemoryStore (crates/vestige-core/src/storage/memory_store.rs):
One trait, ~25 methods, covering CRUD, hybrid / FTS / vector search,
FSRS scheduling, graph edges, and the forthcoming domain surface.
trait_variant::make generates a Send-bound MemoryStore alias over the
base LocalMemoryStore so Arc<dyn MemoryStore> works under tokio/axum.
Storage errors map through a dedicated MemoryStoreError.
Embedder / LocalEmbedder (crates/vestige-core/src/embedder/):
Pluggable text-to-vector encoder. FastembedEmbedder wraps the existing
EmbeddingService; storage never calls fastembed directly anymore.
Embedder::signature() produces the ModelSignature consumed by the
store's embedding_model registry.
SqliteMemoryStore (crates/vestige-core/src/storage/sqlite.rs):
Storage renamed to SqliteMemoryStore; the old name lives on as a
pub type alias so Arc<Storage> consumers in vestige-mcp stay intact.
All existing inherent methods are untouched; the trait impl is
purely additive and dispatches into them. The db_path field added
by v2.1.1 portable-sync is preserved.
Migration V14 (crates/vestige-core/src/storage/migrations.rs):
Renumbered from V12 (the original April number) to V14 to slot in
cleanly after upstream's V12 (v2.1.1 sync_tombstones) and V13
(v2.1.2 purge tombstones).
- embedding_model registry table (CHECK id = 1, code enforces the
single-row invariant).
- knowledge_nodes.domains / domain_scores TEXT columns (JSON arrays
default '[]' / '{}'), domains catalogue table, supporting indexes.
Phase 4 populates these columns; Phase 1 just exposes the schema.
Consolidation and other cognitive pathways now accept a
&dyn LocalMemoryStore (sync) or Arc<dyn MemoryStore> (async) rather
than a concrete Storage.
Tests:
- trait-method unit tests colocated in sqlite.rs and migrations.rs
- embedder/fastembed.rs tests for name/dimension/hash stability
- new integration crate tests/phase_1 (added to workspace members):
trait_round_trip (8), embedding_model_registry (7),
domain_column_migration (5), cognitive_module_isolation (4),
send_bound_variant (2), embedder_trait (2).
Acceptance gate post-rebase:
- cargo build --workspace --all-targets: ok
- cargo clippy --workspace --all-targets -- -D warnings: clean
- cargo test -p vestige-core --lib: 428 pass
- cargo test -p vestige-phase-1-tests: 28 pass
- cargo test -p vestige-mcp --lib: 380 pass (Storage alias preserves
every existing call site)
Co-existence with v2.1.1 portable-sync: this trait extraction is
additive. Portable-sync's tombstone migrations (V12, V13) remain
on the concrete SqliteMemoryStore; Phase 2 (Postgres) will decide
which of those surfaces graduate into the trait.
Rebased on v2.1.25 merge/supersede and bumped the post-release metadata to v2.1.26 so this branch does not roll versions backward.
Adds local vestige.toml defaults, output profiles, and MCP response precedence for search, timeline, codebase context, and session context.
Verified:
- cargo metadata --format-version 1 --locked --no-deps
- cargo test -p vestige-core config --no-fail-fast
- cargo test -p vestige-mcp config --no-fail-fast
Tolerate SQLite-native timestamps from external writers while preserving RFC3339 as the canonical write format.
Verified locally on the merge result:
- cargo test -p vestige-core test_parse_timestamp_accepts_rfc3339_and_sqlite_native --no-fail-fast
CI/Test Suite on the updated PR branch are green.
* feat(mcp/system_status): add optional schema_introspection flag
Adds an optional `schema_introspection: bool` parameter to the
`system_status` MCP tool. When set to true, the response gains a
`schema` block carrying:
- `schemaVersion` (u32) — highest applied migration, mirrors
`Storage::current_schema_version` (now exposed via a typed public
method).
- `schemaVersionAppliedAt` (RFC3339, optional) — timestamp the
current schema_version row was applied.
- `tables` ([{name, rows, columns}]) — per-table row count + column
list, walked over the canonical PORTABLE_USER_DATA_TABLES set so
the surface stays stable across migrations rather than enumerating
arbitrary sqlite_master rows.
- `embeddingNullCount` (i64) — count of knowledge_nodes with NO row
in node_embeddings. Distinct from MemoryStats.nodes_with_embeddings
(which keys off the `has_embedding` flag column), so audit scripts
can detect drift between the flag and the join-based truth.
- `activeEmbeddingModel` (string, optional) + `activeEmbeddingDimensions`
(u32, optional) — mirrors the existing MemoryStats active-model
fields, included here so audits get schema_version + active model
in a single round-trip.
Motivation: external consumers (audit scripts, migration guards,
downstream binary upgrade scripts) currently must read SQLite
directly to learn the schema shape, which couples them to internals
Vestige owns and breaks on every migration. This PR closes that gap
with a first-class MCP surface.
Implementation:
- New `pub fn schema_introspection() -> Result<SchemaIntrospection>`
inherent method on `Storage` (sqlite.rs). Inherent, not on a
trait — schema-walk is SQLite-specific by nature, so this stays
out of any future MemoryStore trait extraction.
- New typed structs `SchemaIntrospection` + `TableIntrospection` in
memory/mod.rs (canonical home alongside MemoryStats), re-exported
from the crate root.
- MCP layer (maintenance.rs) parses `SystemStatusArgs`, conditionally
extends the existing response object with a `schema` key — additive,
default off, response shape unchanged when omitted.
Coupling assessment vs PR #61 (storage-trait-phase1):
This PR adds ONE new public inherent method on `Storage` plus uses
three already-existing private helpers (`current_schema_version`,
`table_exists`, `table_row_count`, `table_columns`). It does NOT
touch the existing inherent method signatures, does NOT add anything
to the prospective `MemoryStore` trait surface, and does NOT modify
any of the ~25 methods #61 lifts into the trait. PR #61 is purely
additive on the trait surface (per its description, `pub type
Storage = SqliteMemoryStore;` preserves all existing call sites);
this PR is additive on the inherent surface. Two purely-additive
changes to disjoint surfaces should rebase cleanly.
Tests:
- system_status_schema_has_schema_introspection_flag (schema
introspection: property present, type=boolean, default=false,
not required)
- system_status_without_schema_flag_omits_schema_block
(backwards-compat: unset/false → no `schema` key)
- system_status_with_schema_flag_emits_schema_block (positive case:
schema block present, schemaVersion >= 13, tables non-empty,
knowledge_nodes row count + columns sane, convenience fields
present)
- system_status_camelcase_alias (#[serde(rename_all="camelCase")] +
alias works for both snake and camel input)
- storage_schema_introspection_method (Storage-layer method tested
directly, independent of MCP)
Closes the second of two gaps surfaced in the knowledge-mgmt-sota-uplift
initiative. Companion to PR #68 (search.tag_prefix). The two PRs are
deliberately decoupled — this one carries the storage-layer surface
extension; the other is MCP-layer-only.
* fix(memory): derive Default on SchemaIntrospection to satisfy clippy
The manual `impl Default for SchemaIntrospection` tripped
`clippy::derivable_impls` under the workspace's `-D warnings` CI gate.
All fields are types with `Default` impls (`u32`, `Option<T>`, `Vec<T>`,
`i64`), so deriving is equivalent and clippy-clean. Matches the existing
style of `ConsolidationResult` further down in the same file.
Adds an optional `tag_prefix` string parameter to the `search` MCP tool.
When set, only results that carry at least one tag whose value starts
with the prefix are returned (case-sensitive, matching the existing
exact-tag semantics in memory_timeline / export / gc).
Motivation: external consumers that need "all memories tagged
`<class>:*`" (e.g. `meeting:standup`, `meeting:1-on-1`) currently have
three paths, all bad: (i) export everything and filter client-side
(heavy), (ii) enumerate the prefix space and pass exact tags as a list
(impractical for open-set tag classes), or (iii) read SQLite directly
(an anti-pattern that couples consumers to internal schema). This PR
closes that gap with a minimal, additive surface.
Implementation note: filter runs at the MCP layer, NOT in the storage
predicate. Rationale: (a) leaves crates/vestige-core/src/storage/
untouched, avoiding collision with PR #61's storage-trait extraction;
(b) `SearchResult.node.tags` is already loaded from the same JSON-array
column the brief's proposed SQL would scan, so the post-filter is
functionally equivalent; (c) post-filter applies BEFORE the reranker
so the cross-encoder does not waste cycles on memories the caller will
not receive, and BEFORE strengthen-on-access so dropped results do not
get a testing-effect boost they did not earn.
Headroom: when tag_prefix is set, the hybrid path doubles its overfetch
multiplier (capped at the existing 100 ceiling) and the concrete path
fetches 3x its normal limit, both to leave the post-filter enough pool
to still return ~limit results after thinning. The Stage 0
keyword-priority merge also re-applies the prefix filter so it cannot
re-introduce filtered-out memories.
Backwards-compat: parameter is optional, defaults to None; every existing
call shape and response shape is unchanged.
Tests:
- tags_match_prefix unit (prefix-vs-substring, case-sensitivity,
tagless-memory semantics, empty-prefix corner case)
- schema introspection (property present, type=string, not required)
- hybrid-path filter excludes non-matching tag-classes
- hybrid-path filter excludes tagless memories
- backwards-compat: no tag_prefix → behavior unchanged
- concrete-path filter (literal-query branch) honors tag_prefix
Closes a gap surfaced in the knowledge-mgmt-sota-uplift initiative
(KMSU Session 89 audit; ~3,300-memory production Vestige).
Claude Code v2.1.91+ honors the per-tool annotation
`_meta["anthropic/maxResultSizeChars"]` (up to 500_000) to override
its 50K default truncation of `CallToolResult`. Without it, large
Vestige payloads are silently truncated and spilled to disk, forcing
the parent agent to chunk-read them.
Empirically observed truncation under realistic default parameters
(measured on v1.3.0 against ~3,300 memories; v2.x tool surface
preserves the same names + payload shapes):
search(detail_level="full", limit=20) -> 134,824 chars -> truncated
search(detail_level="summary", limit=10) -> 71,318 chars -> truncated
memory_timeline(limit=30) -> 83,626 chars -> truncated
This patch:
1. Adds `meta: Option<serde_json::Value>` to `ToolDescription` with
`#[serde(rename = "_meta")]` so the wire shape matches the MCP
spec. Backwards-compatible (the field is optional +
`skip_serializing_if`; older MCP clients ignore unknown JSON keys
per the spec).
2. Derives `Default` on `ToolDescription` so existing call sites can
adopt the new field via struct-update syntax
(`..Default::default()`) without restating it.
3. Annotates the four high-payload tools per measurement-driven
discipline; the other 21 tools deliberately do NOT carry the
annotation (cargo-cult prevention — a generous cap on every tool
dilutes the signal and trains future maintainers that the value
is arbitrary):
- search -> 300_000 (2.2x headroom over observed peak)
- memory_timeline -> 200_000 (2.4x headroom over observed peak)
- memory -> 100_000 (single-record bounded)
- codebase -> 100_000 (future-growth bounded)
Tools that COULD plausibly grow into the annotated set with future
workload (`deep_reference`, `cross_reference`, `memory_graph`,
`explore_connections`, `session_context`) are left unannotated
until empirical measurement shows truncation under realistic use.
4. Adds three regression tests in `server::tests`:
- test_high_payload_tools_have_max_result_size_annotation:
pins each cap value + asserts <= 500K Anthropic ceiling
- test_other_tools_do_not_carry_max_result_size_annotation:
cargo-cult prevention; dynamically iterates `tools/list` and
asserts every tool NOT in the discipline-prescribed set lacks
the annotation (robust to new tools being added by future PRs)
- test_meta_wire_shape_uses_underscore_meta_field:
pins the serde rename to `_meta` (the spec'd wire name) so a
refactor of `ToolDescription` cannot silently drop the rename
All 22 `server::tests` pass on v2.1.22 base (19 pre-existing + 3 new).
Full lib test suite: 379/380 pass; the 1 unrelated failure
(`tools::maintenance::tests::test_portable_export_writes_archive_to_storage_exports_dir`)
is a pre-existing Windows path-separator assertion bug in
`tools/maintenance.rs:823` (`path.ends_with("exports/portable-test.json")`
fails on Windows where the path uses `\`) — unaffected by this PR.
References:
- Anthropic CC v2.1.91 release notes (April 2026): "Added MCP tool
result persistence override via _meta['anthropic/maxResultSizeChars']
annotation (up to 500K), allowing larger results like DB schemas
to pass through without truncation"
- claude-agent-sdk-python v0.1.55 #756: forward bookkeeping
establishing the on-Tool-definition (not on-CallToolResult)
semantics for this annotation
Co-authored-by: Peter Lauzon <inbijiburu@protonmail.com>
