apunkt/omnigraph
1
0
Fork 0
mirror of https://github.com/ModernRelay/omnigraph.git synced 2026-06-18 02:24:27 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 5
omnigraph/docs/user/operations/maintenance.md
Ragnor Comerford b183db078f
Index materialization is derived state: defer off the write path, reconcile via optimize (iss-848) (#246) 
* test(engine): reproduce empty-table Vector @index aborting schema apply

A Vector (IVF) index trains k-means centroids over the column, so Lance
cannot build it on 0 vectors ("Creating empty vector indices with
train=False is not yet implemented"). schema apply reconciles a table's
whole index set whenever any @index on it changes, so adding an unrelated
scalar @index materializes the dormant empty vector index and aborts the
entire migration (all-or-nothing).

This regression test inits a 0-row Doc with a Vector @index, adds a scalar
@index, and asserts the apply succeeds (then loads one embedded row and
asserts the deferred index materializes). It fails today at the apply step
with the vector-index abort; the fix lands in the next commit.

Refs dev-graph iss-empty-vector-index-schema-apply, iss-848.

* fix(engine): defer Vector @index on an empty table instead of aborting schema apply

build_indices_on_dataset_for_catalog materialized a declared Vector @index
unconditionally. On a 0-row table Lance cannot train the IVF index
("Creating empty vector indices with train=False is not yet implemented"),
so any later migration that touches the table (e.g. adding an unrelated
scalar @index, which reconciles the table's whole index set) aborted the
entire migration on the dormant vector index — all-or-nothing.

Guard the vector arm with a row-count check, matching the guard
ensure_indices_for_branch and the branch-merge rebuild already use: an
untrainable column becomes a pending index that a later ensure_indices /
optimize materializes once the table has rows. Reads stay correct meanwhile
(vector search degrades to a brute-force scan).

Stop-gap: the residual rows-present-but-vectors-null window and the full
decoupling (intent recorded at apply, an idempotent coverage reconciler)
are dev-graph iss-848. Turns the green half of the regression test added in
the previous commit.

Refs dev-graph iss-empty-vector-index-schema-apply, iss-848, iss-687.

* docs(invariants): record the logical-contract-over-physical-state principle

The bug class behind the empty-table vector-index abort (and the schema-apply
vs optimize version drift) is one shape: a physical operation allowed to fail
a logical one. Several hard invariants (2, 5, 7, 13) and deny-list items are
already instances of this, but the unifying rule was never written down.

Add it to docs/dev/invariants.md as a "Governing principle" section above the
hard invariants, naming which invariants and deny-list items instantiate it
and the smell to watch for (a logical operation gated on a physical fact).
Add a one-line always-on rule (7) in AGENTS.md so it stays in working memory,
with the qualifier that genuine logical conflicts still fail loudly — the
licence to lag covers physical convergence, not correctness.

Audience-neutral: no private ticket refs. check-agents-md.sh passes.

* test(engine): index build must tolerate rows with null vectors (load-before-embed)

Loading rows whose vector column is null into a `Vector @index` table fails
today: build_indices (reached via the loader's prepare_updates_for_commit)
calls create_vector_index, and Lance's IVF KMeans errors "cannot train 1
centroids with 0 vectors". The same abort hits ensure_indices/optimize/schema
apply/merge, since they all funnel through build_indices_on_dataset_for_catalog.

This test loads two null-embedding rows and calls ensure_indices; it must not
abort (the untrainable vector column is deferred, sibling indexes still build).
Fails today at the load step; fixed in the next commit.

Refs dev-graph iss-848, iss-empty-vector-index-schema-apply.

* fix(engine): defer unbuildable index columns instead of aborting the write path

build_indices_on_dataset_for_catalog is the chokepoint every write path funnels
through (load/mutate via prepare_updates_for_commit, schema apply, ensure_indices,
optimize, branch merge). Its vector arm called create_vector_index
unconditionally, so a column with no trainable vectors yet — an empty table, or
rows loaded before `embed` populates them — aborted the whole operation with
Lance's IVF KMeans error.

Fault-isolate the vector build: on failure, record the column as a PendingIndex
(table, column, reason), log it, and continue building the sibling indexes; a
later ensure_indices/optimize materializes it once the column is trainable, and
reads use brute-force meanwhile. Manifest/CAS/IO errors at the publish boundary
still propagate. Isolating at the single chokepoint realizes the governing
principle (physical index state never fails a logical operation) for every write
path, and supersedes the earlier symptomatic count_rows==0 stop-gap (removed) —
closing the residual rows-present-but-vectors-null window it left open.

Surfacing pending index status rather than failing is the database norm
(Postgres indisvalid, LanceDB list_indices). ensure_indices and the build_indices
wrappers now return Vec<PendingIndex>; optimize surfaces it in a later commit.

Refs dev-graph iss-848, iss-951 (vector index stays inline-commit until lance#6666).

* test(engine): index-only schema apply must not touch table data

Adding an @index to an existing column should be a pure metadata change once
index materialization moves to the reconciler (iss-848): the apply records the
intent in the catalog/IR but builds nothing inline, so the table's manifest
version is unchanged. Today the indexed_tables block builds the index inline
and bumps the version (4 -> 5). Fixed in the next commit.

Refs dev-graph iss-848.

* fix(engine): schema apply records index intent only; index-only apply is metadata

Schema apply no longer builds indexes inline. The four build_indices calls
(added/renamed/rewritten/index-only tables) are removed; the @index/@key intent
is already persisted in the catalog/IR the apply writes, and the physical index
is materialized off the critical path by ensure_indices/optimize (iss-848).

Concretely:
- AddConstraint (an @index addition — every other added constraint plans as
  UnsupportedChange) becomes a pure metadata step alongside the metadata-only
  steps: it touches no table data, so the table version is unchanged.
- added/renamed/rewritten tables still write their data; only the trailing
  index build is gone. The rewritten table's coverage is restored later by
  optimize_indices.
- recovery_pins drops index-only tables (they no longer advance Lance HEAD) and
  keeps rewritten tables; their post_commit_pin = expected+1 is now exact (one
  rewrite commit), strengthening recovery classification.
- the now-orphaned Omnigraph::build_indices_on_dataset_for_catalog wrapper is
  removed.

A migration can no longer abort on an index build, for any index type at any
cardinality. Turns the green half of index_only_constraint_apply_touches_no_table_data.

Refs dev-graph iss-848.

* test(engine): optimize must converge a declared-but-unbuilt index

After iss-848, adding an @index post-data is a metadata-only apply that defers
the physical build, so the column is declared-indexed but unbuilt (reads scan).
`optimize` — the operator's cron reconciler — must materialize it. Today optimize
only maintains coverage of EXISTING indexes (optimize_indices) and never creates
missing ones, so the rank BTREE stays Degraded after optimize. Fixed next commit.

Refs dev-graph iss-848.

* fix(engine): optimize materializes declared-but-unbuilt indexes (the reconciler)

`omnigraph optimize` is the operator's cron reconciler. It already compacts and
folds new fragments into EXISTING indexes (optimize_indices); now it also builds
declared-but-missing indexes, so the indexes schema apply / load defer (iss-848)
converge on the next optimize.

Done inside optimize_one_table (not by composing the all-tables ensure_indices,
which is drift-blind and would re-publish the uncovered HEAD>manifest drift that
optimize deliberately skips): after the per-table drift/blob skips and under the
queue + Optimize sidecar already held, a needs_index_create gate (reusing
needs_index_work_node/edge — "declared index missing AND row_count > 0", so empty
tables stay no-ops) admits index-only work, and Phase B builds the missing index
over the just-compacted layout via the build chokepoint. An untrainable vector
column fault-isolates into the new TableOptimizeStats.pending_indexes (the
list_indices/indisvalid analog operators read), not a failure. committed now
reflects index commits, so the existing post-publish cache invalidation covers
them. LanceDB's optimize only maintains existing indexes; creating
declared-but-missing ones is the L2 behavior omnigraph's declarative @index needs.

Turns the green half of optimize_materializes_index_declared_but_unbuilt.

Refs dev-graph iss-848.

* docs: index materialization is deferred to the reconciler (iss-848)

Update the index-lifecycle docs to reflect the new contract: @index/@key
declares intent and the physical index is derived state that never fails a
logical operation. Schema apply builds nothing (records intent only);
load/mutate build inline through one chokepoint that defers an untrainable
Vector column as pending; optimize/ensure_indices is the reconciler that
creates declared-but-missing indexes and maintains coverage, reporting
still-pending columns.

Touches: dev/invariants.md (truth-matrix Index-lifecycle row), AGENTS.md
(capability matrix), user/search/indexes.md (L2 orchestration), user/operations/
maintenance.md (optimize reconciler bullet), dev/testing.md (new tests).

* test(server): schema_apply_route_can_add_index reflects deferred index build

iss-848 made schema apply record @index intent without building the physical
index inline. The route test asserted the index count increased after apply;
on an empty graph it now stays unchanged (the build is deferred to
ensure_indices/optimize). Assert the new contract: apply succeeds and the
physical index count is unchanged.

* fix(engine): precheck vector trainability — don't pin or swallow (PR review)

Two issues Cursor Bugbot caught in the chokepoint fault-isolation:

1. (HIGH) Pending vector pins roll back siblings. needs_index_work_node counted
   a missing vector index as work whenever the table had rows, so a column with
   no trainable vectors got pinned in the EnsureIndices recovery sidecar — but
   the build deferred it (zero commit). On a crash before manifest publish the
   classifier sees NoMovement and the all-or-nothing decision (recovery.rs
   decide()) rolls back the WHOLE sidecar, undoing a sibling table's committed
   index work.
2. (MED) Vector build swallowed fatal errors. The match arm converted every
   create_vector_index error into a deferred PendingIndex, hiding genuine
   I/O/manifest/Lance failures as "pending".

Fix both with one trainability precheck (vector_column_trainable: >=1 non-null
vector, the ivf_flat(1) minimum) used identically by needs_index_work_node and
the build arm: an untrainable column is never counted as work (so never pinned —
no zero-commit pin) and never attempted (so it can't fail); only a trainable
column is built, and then any error PROPAGATES (stays fatal). The deferred
column is still recorded as a PendingIndex with a clear reason.

Refs dev-graph iss-848.

* feat(cli): surface pending index column + reason in optimize output (PR review)

Codex (P2): pending_indexes was documented as visible in `optimize --json` but
the CLI projection never emitted it — operators would lose the only signal that
optimize has deferred index work. Greptile (P2): the stat dropped the reason, so
operators saw which column was stuck, not why.

Carry the reason: TableOptimizeStats.pending_indexes is now Vec<PendingIndex>
(column + reason), and `omnigraph optimize --json` emits {column, reason} per
pending index; human output prints a "↳ index pending on '<col>': <reason>" line.

Refs dev-graph iss-848.

* test: align CLI index-add test with deferred build; cover post-rename reconcile

- schema_apply_json_adds_index_for_existing_property (cli_schema_config.rs): the
  CLI analog of the server test — asserted the index count grew after apply;
  under iss-848 the apply defers the build, so the count is unchanged on an
  empty graph. Assert the deferred contract. (The only full-suite failure.)
- optimize_materializes_index_after_type_rename (maintenance.rs, new): covers
  the gap Greptile flagged — a RenameType writes the renamed table with rows but
  no indexes (inline build removed in Commit B); assert the rank index is
  Degraded post-rename and Indexed after optimize reconciles it.

Refs dev-graph iss-848.

* test(engine): in-source apply tests reflect deferred index materialization

The two db::omnigraph in-source unit tests asserted the old "schema apply builds
/ preserves indexes inline" behavior (the only remaining full-suite failures):

- test_apply_schema_defers_index_then_reconciler_builds_it (was
  test_apply_schema_adds_index_for_existing_property): apply records the @index
  intent but builds nothing; assert the BTREE on `age` is absent after apply and
  present after ensure_indices. (Uses `age`, unindexed in TEST_SCHEMA — `name
  @key` is already FTS-indexed at seed.)
- test_apply_schema_rewrite_defers_index_then_reconciler_restores (was
  test_apply_schema_rewrite_preserves_existing_indices): an AddProperty rewrite
  no longer rebuilds indexes inline; assert ensure_indices restores id BTREE +
  name FTS after the rewrite.

Verified by grep that these + the server/CLI tests are the complete set of
"apply builds an index" assertions; all other index-presence tests run after
load/ensure_indices/primitives, which still build.

Refs dev-graph iss-848.

* fix(engine): optimize always reports pending indexes, not only on create-work (PR review)

Cursor Bugbot (MED): pending_indexes was filled only when needs_index_create was
true, but the vector trainability precheck makes needs_index_work_node exclude an
untrainable Vector column. So a table whose sole missing index is untrainable, but
which optimize still compacts or reindexes, returned an empty pending_indexes —
contradicting the documented operator contract for deferred columns.

Run the (idempotent) build chokepoint unconditionally once past the no-op gate,
rather than gating it on needs_index_create. It skips existing indexes, builds
any buildable missing one, and reports an untrainable column as pending whether
the table entered for compaction, reindex, or index creation. needs_index_create
still gates the no-op decision (so an index-only table still enters the path).

Refs dev-graph iss-848.

* test(engine): reframe staged-BTREE-failure failpoint onto the reconciler path

ensure_indices_stage_btree_failure_leaves_existing_tables_writable fired
`ensure_indices.post_stage_pre_commit_btree` and expected `apply_schema` (adding
a type) to fail mid-BTREE-build. iss-848 removed apply's inline index build, so
that apply now succeeds and the test's unwrap_err panicked — it exercised a
removed code path.

Reframe onto where BTREE builds happen now: seed Person, add an `@index` on
`age` (apply records intent, defers the build), then `ensure_indices` builds the
deferred BTREE and the failpoint fires between stage and commit. Person's HEAD
is unchanged (no drift) and its EnsureIndices sidecar pins NoMovement; a write to
a different, unpinned table (Company) is unaffected (mutations/loads heal
roll-forward and proceed, unlike optimize/repair which refuse on a pending
sidecar). Preserves the original coverage (staged-index stage failure leaves
other tables writable, no drift) in the new architecture.

Refs dev-graph iss-848.

* feat(server): converge deferred indexes promptly after schema apply (iss-848)

Schema apply records @index intent but defers the physical build. On a
long-lived server, spawn a detached best-effort ensure_indices after a
successful apply so the indexes converge promptly instead of waiting for the
operator's next optimize. Fire-and-forget: it never blocks or fails the apply
response, and a failure is logged (the index still converges on the next
optimize). Guarded on result.applied. The CLI is one-shot, so it has no
equivalent; its convergence path is the optimize cadence.

handle.engine is already an Arc, so the spawn takes an owned clone. Convergence
itself is covered by the engine ensure_indices/optimize tests; the existing
empty-graph schema-apply route tests confirm the response is unaffected (the
spawn is a read-only no-op on an empty table).

Refs dev-graph iss-848.

* docs(maintenance): list pending_indexes in optimize per-table stats (consistency)
2026-06-15 18:48:43 +02:00

8.7 KiB
Raw Blame History

Maintenance: Optimize, Repair & Cleanup

Addressing. optimize, repair, and cleanup are direct (storage-native) CLI commands: they run with direct storage access against a positional file:///s3:// URI or --cluster <dir|s3://…> --graph <id> (which resolves the graph's storage URI from the served cluster state, so you needn't know the <storage>/graphs/<id>.omni layout). They never run through a server, and reject --server or a remote (http(s)://) URI with a declared error. There are no server routes for them by design — to maintain a server-backed graph, run them out-of-band against the graph's storage URI. See the Command capabilities section of cli-reference.md.

optimize — non-destructive

  • Compacts every node + edge table on main, then reindexes them, then publishes the resulting version to the __manifest so the manifest's recorded version tracks the compacted-and-reindexed state. Reads pin the manifest version, so without this publish the work would be invisible to readers and would break the version precondition of the next schema apply / strict update/delete ("stale view … refresh and retry"). The publish advances the graph version (a system-attributed commit) only for tables that actually changed.
  • Rewrites small fragments into fewer large ones; old fragments remain reachable via older versions until cleanup runs.
  • Reindex (index coverage maintenance). A scalar/FTS/vector index only covers the fragments it was built over. Rows appended after the index was built (e.g. by load --mode merge, whose commit does not rebuild an already-existing index) are scanned unindexed, and compaction itself rewrites fragments out of an index's coverage. optimize runs Lance's incremental optimize_indices after compaction to fold those fragments back in (a delta merge, not a full retrain), restoring full coverage so equality/range/traversal predicates stay index-accelerated. This is why a table with no compaction work but stale index coverage still commits a new version under optimize. Run optimize on a cadence at least as frequent as your freshness window so recently-loaded rows do not linger in the unindexed flat-scan tail.
  • Create declared-but-missing indexes (the index reconciler). @index/@key declares intent; schema apply records it but builds nothing, and load/mutate defer a column that cannot be built yet (a Vector column with no trainable vectors). optimize materializes any such declared-but-unbuilt index over the compacted layout — so it is the convergence path for an @index added after data exists, or a vector index whose embeddings arrived via a later embed. A column still not buildable (no vectors yet) is reported on the table's stat as pending_indexes (visible in --json), not treated as a failure; the next optimize retries. So optimize is the single operator-facing index reconciler: it compacts, restores coverage, and builds declared-but-missing indexes.
  • Each table's compact→reindex→publish serializes with concurrent mutations on the same table. A crash mid-operation is recovered automatically on the next open (both compaction and reindex are content-preserving, so roll-forward is always safe).
  • Requires a recovered graph. optimize refuses (errors) when a pending crash-recovery operation is present — operating on an unrecovered graph could publish a partial write that recovery would roll back. Reopen the graph to run recovery, then re-run optimize.
  • Uncovered drift is skipped, not interpreted. If a table's underlying version is ahead of the version recorded in __manifest and no crash-recovery record covers that movement, optimize reports skipped: DriftNeedsRepair with the manifest/head versions and leaves the table untouched. Run omnigraph repair to classify and explicitly publish that drift.
  • Bounded by OMNIGRAPH_MAINTENANCE_CONCURRENCY (default 8).
  • Returns per-table stats: table_key, fragments_removed, fragments_added, committed, skipped, manifest_version, lance_head_version, pending_indexes (the last lists any declared @index column the reconciler could not build this run, with the reason — e.g. a vector column with no trainable vectors yet).
  • Blob tables are skipped. A table that declares any Blob property is not compacted: it is reported with skipped: BlobColumnsUnsupportedByLance (and logged) instead of compacted, and the rest of the sweep proceeds normally. Reads and writes are unaffected — only compaction is. Consequence: fragment count and deleted-row space on blob tables are not reclaimed; query results are never affected. A skipped blob table is also not reindexed in the same sweep (the skip happens before the reindex step), so its index coverage on appended rows is not refreshed by optimize today.

repair — explicit

  • Handles uncovered manifest/head drift: a table's underlying version is ahead of the manifest pin and no crash-recovery record explains the movement.
  • Preview by default. omnigraph repair --json <uri> reports each table's classification, action, manifest/head versions, underlying operation names, and any classification error. --confirm publishes only verified maintenance drift; if any suspicious or unverifiable table is refused, the CLI prints the per-table output and exits non-zero. --force --confirm also publishes suspicious or unverifiable drift after operator review.
  • Classifies drift by reading the table's transaction history from manifest_version + 1 through the current head. Only fragment-reservation and rewrite (compaction) operations are verified maintenance. Semantic operations such as append, delete, update, merge, or missing transaction history are not auto-healed.
  • Publishes repair by advancing __manifest to the existing head; it does not rewrite data. If the publish succeeds, normal reads and strict writes use the repaired version. If it fails, no new data-side partial state was created.
  • Requires a clean recovery state. A pending crash-recovery operation still belongs to automatic recovery, not manual repair.

cleanup — destructive

  • Garbage-collects old versions per table.
  • Removes versions (and their unique fragments) older than the retention policy.
  • Policy options keep_versions and older_than — at least one is required.
  • Returns per-table stats: table_key, bytes_removed, old_versions_removed, error.
  • Fault-isolated per table. A single table's transient failure (version GC or orphan reclaim) is recorded on that table's stats row (with an error) and logged, and never aborts the healthy tables — cleanup is the convergence backstop, so it does as much as it can and converges on re-run. The CLI reports any failed tables; rerun cleanup to retry them.
  • CLI guards with --confirm; without it, prints a preview line.
  • Non-local consent (RFC-011 D9). Against a non-local target (an s3:// store/cluster), cleanup additionally requires --yes on top of --confirm: a TTY is prompted, and a non-interactive run (no TTY, or --json) refuses rather than destroying. A local (file://) target needs only --confirm. The same --yes gate applies to overwrite load and branch delete; every maintenance run echoes its resolved target to stderr (suppress with --quiet).
  • Recovery floor: --keep < 3 may garbage-collect versions that crash recovery needs as a rollback target. Default --keep 10 is safe.
  • Orphaned-branch reconciliation: before the version GC, cleanup reclaims any per-table or commit-graph branch absent from the manifest branch list. These orphans arise when a branch_delete flips the manifest authority but a downstream best-effort reclaim does not complete (see branches-commits.md). The reconciler is idempotent (it no-ops once nothing is orphaned), runs regardless of the keep_versions / older_than values (those gate version GC only), and never reclaims main or system-branch forks. Reclaimed forks are logged.

Tombstones

Logical sub-table delete markers in __manifest that exclude a sub-table version from snapshot reconstruction.

Internal schema migrations

Version evolutions of the on-disk __manifest shape are reconciled automatically on the first write under a new binary. An on-disk stamp records the shape; the binary migrates it forward before reading state, and reads are side-effect-free. No operator action is required for in-place upgrades. See storage.md → Internal schema versioning for the full mechanism.

A binary opening a manifest stamped at a version higher than it knows about refuses to publish with a clear "upgrade omnigraph first" error — old binaries cannot clobber a newer schema.