cargo rm async-trait. Last usage was the FastembedEmbedder impl attribute,
removed in the preceding 0001c commit; the MemoryStore side stopped using
async-trait at 0001a.
Verification: grep -rn async_trait crates/ returns zero hits. grep -rn
async-trait --include=Cargo.toml crates/ returns zero hits. Cargo.lock no
longer references the async-trait package.
Mirror of the 0001a pattern for the Embedder side.
- embedder/mod.rs: LocalEmbedder is the source trait declared with native
async-fn-in-trait. #[trait_variant::make(EmbedderSend: Send)] derives the
Send-bounded variant that backends implement. A hand-written Embedder
trait wraps each async method in BoxedEmbedderFuture<'a, T> and forwards
sync methods through a blanket impl<T: EmbedderSend> Embedder for T, so
Box<dyn Embedder> / Arc<dyn Embedder> stay dyn-safe -- trait_variant 0.1
alone does NOT produce a dyn-safe variant (RPITIT), so the hand-written
adapter is required.
- embedder/fastembed.rs: drop the #[async_trait::async_trait] attribute and
retarget the impl block to EmbedderSend. Adjust the top-level use to
bring EmbedderSend into scope (also keeps fastembed::tests' use super::*
trait lookups working).
- lib.rs: export EmbedderSend alongside the existing Embedder /
LocalEmbedder re-exports.
The async-trait Cargo dependency is dropped in a follow-up commit so the
manifest change stays visible on its own.
Verification: cargo test -p vestige-core --features embeddings,vector-search
(428) and --no-default-features (370) both green. cargo test --test
embedder_trait green (2/2 including Box<dyn Embedder> cast). cargo build
--workspace --release green. cargo clippy --workspace --features
embeddings,vector-search -- -D warnings clean. grep -rn async_trait crates/
returns zero.
Replaces #[async_trait::async_trait] on the storage trait with a
trait_variant-driven layout plus a hand-written dyn-compatible adapter.
- memory_store.rs: LocalMemoryStore is the source trait declared with
native async-fn-in-trait. #[trait_variant::make(MemoryStoreSend: Send)]
derives the Send-bounded variant that backends actually implement (the
blanket impl in 0.1.x goes variant -> source). A hand-written
MemoryStore trait wraps every method in
Pin<Box<dyn Future<Output = MemoryStoreResult<T>> + Send + 'a>> with
a BoxedStoreFuture<'a, T> alias, and a blanket impl<T: MemoryStoreSend>
MemoryStore for T adapts every Send-variant implementation. This keeps
Arc<dyn MemoryStore> dyn-safe for Phase 1 cognitive-module tests --
trait_variant 0.1 alone does NOT produce a dyn-safe variant (RPITIT),
so the hand-written adapter is required and supersedes the plan claim
that trait_variant gives dyn-compat for free.
- sqlite.rs: drop the #[async_trait::async_trait] attribute on the impl
block and retarget it to MemoryStoreSend. Two pre-existing clippy
issues that the macro had been masking are fixed in the same body
(return Ok(out) tail expression in vector_search; DomainRow tuple
alias in get_domain).
- mod.rs: export MemoryStoreSend alongside the existing LocalMemoryStore
and MemoryStore re-exports.
Verification: cargo test -p vestige-core --features embeddings,vector-search
passes (428 lib tests). All five Phase 1 integration test binaries pass
(trait_round_trip, send_bound_variant including
arc_dyn_memory_store_moves_across_tokio_tasks, cognitive_module_isolation,
embedding_model_registry, domain_column_migration). cargo test --workspace
green across every test binary. cargo build --workspace --release green.
cargo clippy --workspace --features embeddings,vector-search -- -D warnings
clean. grep -rn async_trait crates/vestige-core/src/storage/ returns
zero hits.
Supersedes plan claim in docs/plans/0001a-trait-rewrite.md about
trait_variant emitting a dyn-compatible Send variant; option (c) from
the design conversation (hand-written dyn adapter) was selected
explicitly because trait_variant 0.1.2 does not.
Introduce two trait boundaries that the rest of the stack now sits above,
landing Phase 1 of ADR 0001 (pluggable storage and network access).
Rebased onto v2.1.22 Sanhedrin from the original April work.
MemoryStore / LocalMemoryStore (crates/vestige-core/src/storage/memory_store.rs):
One trait, ~25 methods, covering CRUD, hybrid / FTS / vector search,
FSRS scheduling, graph edges, and the forthcoming domain surface.
trait_variant::make generates a Send-bound MemoryStore alias over the
base LocalMemoryStore so Arc<dyn MemoryStore> works under tokio/axum.
Storage errors map through a dedicated MemoryStoreError.
Embedder / LocalEmbedder (crates/vestige-core/src/embedder/):
Pluggable text-to-vector encoder. FastembedEmbedder wraps the existing
EmbeddingService; storage never calls fastembed directly anymore.
Embedder::signature() produces the ModelSignature consumed by the
store's embedding_model registry.
SqliteMemoryStore (crates/vestige-core/src/storage/sqlite.rs):
Storage renamed to SqliteMemoryStore; the old name lives on as a
pub type alias so Arc<Storage> consumers in vestige-mcp stay intact.
All existing inherent methods are untouched; the trait impl is
purely additive and dispatches into them. The db_path field added
by v2.1.1 portable-sync is preserved.
Migration V14 (crates/vestige-core/src/storage/migrations.rs):
Renumbered from V12 (the original April number) to V14 to slot in
cleanly after upstream's V12 (v2.1.1 sync_tombstones) and V13
(v2.1.2 purge tombstones).
- embedding_model registry table (CHECK id = 1, code enforces the
single-row invariant).
- knowledge_nodes.domains / domain_scores TEXT columns (JSON arrays
default '[]' / '{}'), domains catalogue table, supporting indexes.
Phase 4 populates these columns; Phase 1 just exposes the schema.
Consolidation and other cognitive pathways now accept a
&dyn LocalMemoryStore (sync) or Arc<dyn MemoryStore> (async) rather
than a concrete Storage.
Tests:
- trait-method unit tests colocated in sqlite.rs and migrations.rs
- embedder/fastembed.rs tests for name/dimension/hash stability
- new integration crate tests/phase_1 (added to workspace members):
trait_round_trip (8), embedding_model_registry (7),
domain_column_migration (5), cognitive_module_isolation (4),
send_bound_variant (2), embedder_trait (2).
Acceptance gate post-rebase:
- cargo build --workspace --all-targets: ok
- cargo clippy --workspace --all-targets -- -D warnings: clean
- cargo test -p vestige-core --lib: 428 pass
- cargo test -p vestige-phase-1-tests: 28 pass
- cargo test -p vestige-mcp --lib: 380 pass (Storage alias preserves
every existing call site)
Co-existence with v2.1.1 portable-sync: this trait extraction is
additive. Portable-sync's tombstone migrations (V12, V13) remain
on the concrete SqliteMemoryStore; Phase 2 (Postgres) will decide
which of those surfaces graduate into the trait.
Rebased on v2.1.25 merge/supersede and bumped the post-release metadata to v2.1.26 so this branch does not roll versions backward.
Adds local vestige.toml defaults, output profiles, and MCP response precedence for search, timeline, codebase context, and session context.
Verified:
- cargo metadata --format-version 1 --locked --no-deps
- cargo test -p vestige-core config --no-fail-fast
- cargo test -p vestige-mcp config --no-fail-fast
Tolerate SQLite-native timestamps from external writers while preserving RFC3339 as the canonical write format.
Verified locally on the merge result:
- cargo test -p vestige-core test_parse_timestamp_accepts_rfc3339_and_sqlite_native --no-fail-fast
CI/Test Suite on the updated PR branch are green.
* feat(mcp/system_status): add optional schema_introspection flag
Adds an optional `schema_introspection: bool` parameter to the
`system_status` MCP tool. When set to true, the response gains a
`schema` block carrying:
- `schemaVersion` (u32) — highest applied migration, mirrors
`Storage::current_schema_version` (now exposed via a typed public
method).
- `schemaVersionAppliedAt` (RFC3339, optional) — timestamp the
current schema_version row was applied.
- `tables` ([{name, rows, columns}]) — per-table row count + column
list, walked over the canonical PORTABLE_USER_DATA_TABLES set so
the surface stays stable across migrations rather than enumerating
arbitrary sqlite_master rows.
- `embeddingNullCount` (i64) — count of knowledge_nodes with NO row
in node_embeddings. Distinct from MemoryStats.nodes_with_embeddings
(which keys off the `has_embedding` flag column), so audit scripts
can detect drift between the flag and the join-based truth.
- `activeEmbeddingModel` (string, optional) + `activeEmbeddingDimensions`
(u32, optional) — mirrors the existing MemoryStats active-model
fields, included here so audits get schema_version + active model
in a single round-trip.
Motivation: external consumers (audit scripts, migration guards,
downstream binary upgrade scripts) currently must read SQLite
directly to learn the schema shape, which couples them to internals
Vestige owns and breaks on every migration. This PR closes that gap
with a first-class MCP surface.
Implementation:
- New `pub fn schema_introspection() -> Result<SchemaIntrospection>`
inherent method on `Storage` (sqlite.rs). Inherent, not on a
trait — schema-walk is SQLite-specific by nature, so this stays
out of any future MemoryStore trait extraction.
- New typed structs `SchemaIntrospection` + `TableIntrospection` in
memory/mod.rs (canonical home alongside MemoryStats), re-exported
from the crate root.
- MCP layer (maintenance.rs) parses `SystemStatusArgs`, conditionally
extends the existing response object with a `schema` key — additive,
default off, response shape unchanged when omitted.
Coupling assessment vs PR #61 (storage-trait-phase1):
This PR adds ONE new public inherent method on `Storage` plus uses
three already-existing private helpers (`current_schema_version`,
`table_exists`, `table_row_count`, `table_columns`). It does NOT
touch the existing inherent method signatures, does NOT add anything
to the prospective `MemoryStore` trait surface, and does NOT modify
any of the ~25 methods #61 lifts into the trait. PR #61 is purely
additive on the trait surface (per its description, `pub type
Storage = SqliteMemoryStore;` preserves all existing call sites);
this PR is additive on the inherent surface. Two purely-additive
changes to disjoint surfaces should rebase cleanly.
Tests:
- system_status_schema_has_schema_introspection_flag (schema
introspection: property present, type=boolean, default=false,
not required)
- system_status_without_schema_flag_omits_schema_block
(backwards-compat: unset/false → no `schema` key)
- system_status_with_schema_flag_emits_schema_block (positive case:
schema block present, schemaVersion >= 13, tables non-empty,
knowledge_nodes row count + columns sane, convenience fields
present)
- system_status_camelcase_alias (#[serde(rename_all="camelCase")] +
alias works for both snake and camel input)
- storage_schema_introspection_method (Storage-layer method tested
directly, independent of MCP)
Closes the second of two gaps surfaced in the knowledge-mgmt-sota-uplift
initiative. Companion to PR #68 (search.tag_prefix). The two PRs are
deliberately decoupled — this one carries the storage-layer surface
extension; the other is MCP-layer-only.
* fix(memory): derive Default on SchemaIntrospection to satisfy clippy
The manual `impl Default for SchemaIntrospection` tripped
`clippy::derivable_impls` under the workspace's `-D warnings` CI gate.
All fields are types with `Default` impls (`u32`, `Option<T>`, `Vec<T>`,
`i64`), so deriving is equivalent and clippy-clean. Matches the existing
style of `ConsolidationResult` further down in the same file.
Adds an optional `tag_prefix` string parameter to the `search` MCP tool.
When set, only results that carry at least one tag whose value starts
with the prefix are returned (case-sensitive, matching the existing
exact-tag semantics in memory_timeline / export / gc).
Motivation: external consumers that need "all memories tagged
`<class>:*`" (e.g. `meeting:standup`, `meeting:1-on-1`) currently have
three paths, all bad: (i) export everything and filter client-side
(heavy), (ii) enumerate the prefix space and pass exact tags as a list
(impractical for open-set tag classes), or (iii) read SQLite directly
(an anti-pattern that couples consumers to internal schema). This PR
closes that gap with a minimal, additive surface.
Implementation note: filter runs at the MCP layer, NOT in the storage
predicate. Rationale: (a) leaves crates/vestige-core/src/storage/
untouched, avoiding collision with PR #61's storage-trait extraction;
(b) `SearchResult.node.tags` is already loaded from the same JSON-array
column the brief's proposed SQL would scan, so the post-filter is
functionally equivalent; (c) post-filter applies BEFORE the reranker
so the cross-encoder does not waste cycles on memories the caller will
not receive, and BEFORE strengthen-on-access so dropped results do not
get a testing-effect boost they did not earn.
Headroom: when tag_prefix is set, the hybrid path doubles its overfetch
multiplier (capped at the existing 100 ceiling) and the concrete path
fetches 3x its normal limit, both to leave the post-filter enough pool
to still return ~limit results after thinning. The Stage 0
keyword-priority merge also re-applies the prefix filter so it cannot
re-introduce filtered-out memories.
Backwards-compat: parameter is optional, defaults to None; every existing
call shape and response shape is unchanged.
Tests:
- tags_match_prefix unit (prefix-vs-substring, case-sensitivity,
tagless-memory semantics, empty-prefix corner case)
- schema introspection (property present, type=string, not required)
- hybrid-path filter excludes non-matching tag-classes
- hybrid-path filter excludes tagless memories
- backwards-compat: no tag_prefix → behavior unchanged
- concrete-path filter (literal-query branch) honors tag_prefix
Closes a gap surfaced in the knowledge-mgmt-sota-uplift initiative
(KMSU Session 89 audit; ~3,300-memory production Vestige).
Land the Postgres dev cluster recipe Jan provisioned on delandtj-home
(rootless podman + pgvector/pgvector:pg18, PG 18.4, pgvector 0.8.2) and
align all live ADR 0002 / Phase 2 sub-plan references from pg16 to pg18.
- docs/plans/local-dev-postgres-setup.md -- rewritten end-to-end:
podman container vestige-pg with --restart=always, named volume
vestige-pgdata, PGDATA=/var/lib/postgresql/data/pgdata, port mapping
127.0.0.1:5432:5432, two-password split (superuser + app role),
pgvector preinstalled, CREATE EXTENSION vector handled at setup,
day-to-day commands, password rotation, dev-grade backup/restore,
teardown, boot-persistence notes for rootless podman. Old native
Arch install recipe moved to Out-of-scope (covered by image now).
- docs/adr/0002-phase-2-execution.md -- the open-thread mention of
pgvector/pgvector:pg16 in the Follow-ups section now reads pg18.
- docs/plans/0002c-migrations.md -- container example in the local
dev section updated to pg18.
- docs/plans/0002d-store-impl-bodies.md -- testcontainers GenericImage
tag pg16 -> pg18; prose reference updated.
- docs/plans/0002h-testing-and-benches.md -- harness pg18 across
testcontainers Postgres builder, image-caching prose, CI workflow
example.
The archival master plan (docs/plans/0002-phase-2-postgres-backend.md)
keeps its original pg16 references intentionally; the supersession
notice already points readers to the live sub-plans.
Nine Phase 2 sub-plans operationalising ADR 0002 against the Phase 2
master plan, each sized to fit a focused implementation session and
handed to Claude Code as a /goal brief without requiring the agent to
load the master plan.
Order of execution (each depends on the previous unless noted):
- 0002a-skeleton-and-feature-gate.md -- postgres-backend Cargo feature
+ PgMemoryStore skeleton with todo!() bodies. D1+D2.
- 0002b-pool-and-config.md -- PgPool builder, VestigeConfig/
PostgresConfig, vestige.toml loader wired into vestige-mcp. D3+D7
(master plan numbering).
- 0002c-migrations.md -- sqlx migrations 0001_init/0002_hnsw including
D7 (users/groups/memberships, owner/visibility/shared_with_groups)
and D8 (codebase column). SQLite V15 parity migration. D4.
- 0002d-store-impl-bodies.md -- real CRUD + registry bodies; trivial
fts_search/vector_search bodies. D2+D6.
- 0002e-hybrid-search.md -- one-statement RRF query. D5.
- 0002f-migrate-cli.md -- vestige migrate copy (SQLite -> Postgres),
--dry-run, idempotent re-runs, --allow-source-upgrade for pre-V15
sources. D8+D10.
- 0002g-reembed.md -- vestige migrate reembed (offline rebuild).
D9 + D10 reembed arm. Ships resolve_embedder helper as a workaround
for the missing Embedder::from_name(&str) constructor.
- 0002h-testing-and-benches.md -- testcontainers harness, six
integration test files, Criterion bench at 1k/100k. D14+D15.
- 0002i-runbook.md -- operator-facing deployment + day-2 runbook. D16.
Supersession notice added to the master plan (0002-phase-2-postgres-
backend.md) pointing at ADR 0002; body retained as archival reference.
PR B carries this commit plus the previous two (ADR 0002 + Phase 1
amendment sub-plans); no code change.
Three sub-plans operationalising ADR 0002 D1 + D3 on the existing
feat/storage-trait-phase1 branch (790c0c8, not yet pushed upstream):
- 0001a-trait-rewrite.md -- rewrite MemoryStore with
#[trait_variant::make(MemoryStore: Send)] generating a non-Send
LocalMemoryStore companion. Production callers use Arc<Storage> and are
unaffected; only the trait declaration and SQLite impl block change.
- 0001b-sqlite-split.md -- pure code motion. Split sqlite.rs (8713 lines)
into a sqlite/ directory (mod, crud, search, scheduling, graph, domain,
registry, portable_sync, trait_impl). Public re-exports unchanged; tests
green commit-by-commit. Depends on 0001a so trait_impl.rs picks up the
trait_variant attribute once.
- 0001c-async-trait-sunset.md -- rewrite Embedder the same way, then
remove async-trait = "0.1" from crates/vestige-core/Cargo.toml. End
state: zero async_trait references in the workspace.
Together these three lands as PR A.
Binding ADR for Phase 2 Postgres backend integration plus the Phase 1
amendment that removes async_trait from the storage and embedder traits.
Decisions D1-D8:
- D1: sunset async_trait across MemoryStore + Embedder via trait_variant
- D2: PgMemoryStore::connect(url, max_connections) mirrors SqliteMemoryStore;
no Embedder in constructor; register_model handles pgvector typmod
- D3: split sqlite.rs into a sqlite/ directory as Phase 1 amendment
- D4: postgres/ as a directory from day one
- D5: sub-plan layout -- 3 Phase 1 amendment + 9 Phase 2 sub-plans
- D6: no separate ADR for the SQLite split (pure code motion)
- D7: reserve multi-tenancy schema (users/groups/group_memberships +
owner_user_id/visibility/shared_with_groups) in Phase 2 so Phase 3 auth
is additive, not an online migration over an HNSW-indexed table
- D8: codebase promoted to a first-class indexed column on knowledge_nodes;
mcp_client_id and session_id stay in metadata JSONB
PR cadence: PR A = Phase 1 amendment (code on feat/storage-trait-phase1);
PR B = this ADR + Phase 2 sub-plans (docs only); PR C = Phase 2
implementation. Phase 4 sharing_rules table sketched in Follow-ups.
Claude Code v2.1.91+ honors the per-tool annotation
`_meta["anthropic/maxResultSizeChars"]` (up to 500_000) to override
its 50K default truncation of `CallToolResult`. Without it, large
Vestige payloads are silently truncated and spilled to disk, forcing
the parent agent to chunk-read them.
Empirically observed truncation under realistic default parameters
(measured on v1.3.0 against ~3,300 memories; v2.x tool surface
preserves the same names + payload shapes):
search(detail_level="full", limit=20) -> 134,824 chars -> truncated
search(detail_level="summary", limit=10) -> 71,318 chars -> truncated
memory_timeline(limit=30) -> 83,626 chars -> truncated
This patch:
1. Adds `meta: Option<serde_json::Value>` to `ToolDescription` with
`#[serde(rename = "_meta")]` so the wire shape matches the MCP
spec. Backwards-compatible (the field is optional +
`skip_serializing_if`; older MCP clients ignore unknown JSON keys
per the spec).
2. Derives `Default` on `ToolDescription` so existing call sites can
adopt the new field via struct-update syntax
(`..Default::default()`) without restating it.
3. Annotates the four high-payload tools per measurement-driven
discipline; the other 21 tools deliberately do NOT carry the
annotation (cargo-cult prevention — a generous cap on every tool
dilutes the signal and trains future maintainers that the value
is arbitrary):
- search -> 300_000 (2.2x headroom over observed peak)
- memory_timeline -> 200_000 (2.4x headroom over observed peak)
- memory -> 100_000 (single-record bounded)
- codebase -> 100_000 (future-growth bounded)
Tools that COULD plausibly grow into the annotated set with future
workload (`deep_reference`, `cross_reference`, `memory_graph`,
`explore_connections`, `session_context`) are left unannotated
until empirical measurement shows truncation under realistic use.
4. Adds three regression tests in `server::tests`:
- test_high_payload_tools_have_max_result_size_annotation:
pins each cap value + asserts <= 500K Anthropic ceiling
- test_other_tools_do_not_carry_max_result_size_annotation:
cargo-cult prevention; dynamically iterates `tools/list` and
asserts every tool NOT in the discipline-prescribed set lacks
the annotation (robust to new tools being added by future PRs)
- test_meta_wire_shape_uses_underscore_meta_field:
pins the serde rename to `_meta` (the spec'd wire name) so a
refactor of `ToolDescription` cannot silently drop the rename
All 22 `server::tests` pass on v2.1.22 base (19 pre-existing + 3 new).
Full lib test suite: 379/380 pass; the 1 unrelated failure
(`tools::maintenance::tests::test_portable_export_writes_archive_to_storage_exports_dir`)
is a pre-existing Windows path-separator assertion bug in
`tools/maintenance.rs:823` (`path.ends_with("exports/portable-test.json")`
fails on Windows where the path uses `\`) — unaffected by this PR.
References:
- Anthropic CC v2.1.91 release notes (April 2026): "Added MCP tool
result persistence override via _meta['anthropic/maxResultSizeChars']
annotation (up to 500K), allowing larger results like DB schemas
to pass through without truncation"
- claude-agent-sdk-python v0.1.55 #756: forward bookkeeping
establishing the on-Tool-definition (not on-CallToolResult)
semantics for this annotation
Co-authored-by: Peter Lauzon <inbijiburu@protonmail.com>
