mirror of
https://github.com/ModernRelay/omnigraph.git
synced 2026-06-30 02:49:39 +02:00
0dcdcf5a9d
* test(engine): pin zero-row cascade delete must not drift an edge table (red) A delete <Node> cascades a delete_where into every incident edge type. The inline delete_where (Dataset::delete) advances Lance HEAD even when zero edges match, but the cascade records the new version only if deleted_rows > 0 — so a node with no incident edges leaves edge:Knows HEAD>manifest drift, which trips the next strict write's ExpectedVersionMismatch and repair refuses it. Red today: edge:Knows manifest=v5, Lance HEAD=v6. Goes green when delete moves to the staged two-phase path (iss-950, Lance 7.0 DeleteBuilder::execute_uncommitted), where a 0-row delete commits no Lance version and the deleted_rows>0 gate becomes correct by construction. * fix(engine): a zero-row delete must not advance Lance HEAD Lance's Dataset::delete commits a new version even when the predicate matches nothing (build_transaction always emits Operation::Delete), so a node delete that cascades a delete_where into an incident edge type with no matching edges advanced that edge table's Lance HEAD while the cascade skipped record_inline (gated on deleted_rows > 0) — leaving HEAD>manifest drift that wedged the next strict write and that repair refused as suspicious/unverifiable. Use Lance 7.0's two-phase DeleteBuilder::execute_uncommitted to read num_deleted_rows before committing: a no-match delete now advances nothing (no version, no drift) and the existing deleted_rows>0 gate is correct by construction. Non-zero deletes commit the staged transaction with skip_auto_cleanup + affected_rows (parity with the prior inline path). First step of the staged-delete migration (iss-950); turns the node_delete_with_no_incident_edges_leaves_no_edge_table_drift regression green. * feat(engine): stage_delete two-phase primitive (MR-A step 0) Add TableStore::stage_delete (Lance 7.0 DeleteBuilder::execute_uncommitted), the two-phase analogue of stage_merge_insert: writes deletion files without advancing Lance HEAD, returns Option<StagedWrite> (None on 0 rows = true no-op), carrying the deletion-vector updated_fragments as new_fragments and the superseded originals as removed_fragment_ids so combine_committed_with_staged makes the deletion visible to in-query reads. No affected_rows is threaded: like stage_merge_insert's Operation::Update commit, the staged delete relies on OmniGraph's per-table write queue + manifest CAS, not Lance's per-dataset conflict resolver (commit_staged is a single attempt). Flip the two residual guards to the staged path: staged_writes.rs now asserts stage_delete does NOT advance HEAD and that a staged delete is read-your-writes visible (the deletion-vector RYW proof D2 retirement depends on); the lance_surface_guards delete guard pins execute_uncommitted's UncommittedDelete. No behavior change yet (callers still use delete_where); Step 1 wires them. * feat(engine): TableStorage::stage_delete + migrate merge delete path (MR-A step 1a) Add stage_delete/Option<StagedHandle> to the TableStorage trait (delegates to TableStore::stage_delete). Migrate the two branch_merge delete sites (three-way RewriteMerged + adopt delta) from the inline delete_where residual to stage_delete + commit_staged — identical in shape to the stage_merge_insert + commit_staged pair above each. HEAD still advances within the merge sequence (via commit_staged), under the unchanged SidecarKind::BranchMerge Phase-B confirmation; the _pre_delete/_pre_index failpoints fire by position, unchanged. merge_truth_table, branching, composite_flow green. * feat(engine): migrate all delete sites to staged path, retire inline delete (MR-A step 1b/1c) Routes every delete through the staged write path so delete never advances Lance HEAD inline — the last inline-commit residual on the mutation path is gone. `MutationStaging` now accumulates delete predicates (`record_delete`) alongside pending write batches; at end-of-query `stage_all` combines a table's predicates into one `(p1) OR (p2) …` `stage_delete` (a deletion-vector transaction, no HEAD advance) and `commit_all` commits it through the same `commit_staged` path as inserts/updates. Deletes are now ordinary staged entries: one sidecar pin at `expected + 1`, no inline special-casing. Migrated callers (all 5): the 3 mutation.rs sites (delete-node, cascade, delete-edge) and the 2 merge.rs sites (already on stage_delete in step 1a). `affected_edges`/`affected` move from post-inline-commit `deleted_rows` to a committed `count_rows` at record time — exact under D₂, bounded by the cascade working set. A predicate matching zero rows stages nothing (the staged equivalent of the old "skip record_inline on 0 deleted rows"), so the zero-row edge-table drift class stays closed by construction. Retired scaffolding now that no caller remains: - `MutationStaging.inline_committed` + `record_inline` → `delete_predicates` + `record_delete`; `StagedMutation.inline_committed`/`paths` fields and all the `commit_all` inline handling (queue keys, sidecar pins with the `record_inline` table_version special-case, the inline recheck loop). - `open_table_for_mutation`'s post-inline-commit reopen branch (deletes no longer advance HEAD mid-query, so a second touch reopens at the pinned version like any write). - `InlineCommitResidual::delete_where` + its `TableStore` impl, the orphaned `TableStore::delete_where`, and `DeleteState`. `InlineCommitResidual` now carries only `create_vector_index` (Lance #6666 still open). D₂ stays for now: staged-delete read-your-writes doesn't yet compose into the pending accumulator (insert-then-delete on one table), so mixed insert/update/delete in one query is still rejected at parse time. Retiring D₂ is step 2. Doc comments updated to match across exec/, storage_layer, db/. Tests (all green): writes, consistency, validators, end_to_end, composite_flow, merge_truth_table, maintenance, recovery, staged_writes, forbidden_apis, lance_surface_guards, changes, point_in_time (286), plus failpoints (63). * docs: delete is a staged write, not an inline-commit residual (MR-A step 1) Update the docs that described `delete` as the inline-commit residual now that MR-A routes it through `stage_delete`. Always-loaded surfaces (AGENTS.md rule 4 / capability matrix, invariants.md Invariant 4 / truth matrix / known gaps) plus the dev write-path docs (writes.md, execution.md incl. its mutation sequence diagram, architecture.md) now state: deletes accumulate as predicates and stage like inserts/updates, no inline HEAD advance; `InlineCommitResidual` carries only `create_vector_index` (Lance #6666). The parse-time D₂ rule is documented as retained — not because delete inline-commits, but because staged-delete read-your-writes is not yet wired into the pending accumulator (MR-A step 2). lance.md's 7.0 audit note marked MR-A as landed. * docs: D₂ is a deliberate boundary, not temporary scaffolding (MR-A close-out) After MR-A staged the delete path, D₂ (a mutation query is insert/update-only OR delete-only) was left framed as temporary — "until Lance ships two-phase delete" / "retire in step 2". Lance shipped that and we used it for the inline-commit fix; D₂'s original justification is gone. It now stands for a different, permanent reason: keeping a query to one kind keeps its read-your-writes unambiguous and each table to one version per query. Retiring it would buy single-commit mixed atomicity (cheap workaround: split, or a branch) at the cost of an in-query delete view, pending pruning, edge id-resolution, and two-commit-per-table ordering in the hot mutation path — complexity not worth earning. Decision: keep D₂ as a deliberate boundary. Reframes the now-stale wording everywhere, no logic change: - The D₂ parse-time error message no longer promises "this restriction lifts when Lance exposes a two-phase delete API"; it states the boundary and points to a branch+merge for one atomic commit. - `enforce_no_mixed_destructive_constructive` doc, AGENTS.md, invariants.md (Invariant 4 / truth matrix / removed from the known-gaps), writes.md, architecture.md, lance.md, and the user mutations doc (which wrongly said deletes "commit through a different path" — both stage now). - Swept remaining stale `delete_where` mentions left from the Step-1 migration: the merge.rs "swap when upstream ships" comments (already swapped), the forbidden_apis / table_ops residual notes, the staged_writes vector-index guard doc (was "same as stage_delete's absence" — stage_delete now exists), and test comments/assert messages in recovery/maintenance/writes/failpoints. Genuinely-historical records (dated Lance audit, rfc-013, bug-case-fix) left. Verified: engine builds warning-free; check-agents-md OK; writes/maintenance/ recovery/staged_writes/forbidden_apis all green. Closes MR-A. * test(engine): overlapping delete predicates must not double-count affected_* (red) Reproduces a reporting regression from the staged-delete migration flagged in PR #308 review. Because deletes now stage (instead of inline-committing), two delete statements in one query both scan the same unchanged committed snapshot; counting each predicate independently over-reports `affected_*` when they overlap. The old inline path committed each delete before the next ran, so it counted distinct. `delete Person where name = "Alice"` then `delete Person where age > 29` over the standard fixture (Alice 30, Charlie 35) removes 2 distinct nodes and 3 distinct edges, but the buggy per-statement counting returns 3 nodes / 6 edges. RED at this commit (asserts left=3, right=2). * fix(engine): dedup overlapping delete predicates when counting affected_* Count each delete statement against the committed snapshot MINUS the predicates a prior delete statement on the same table already recorded: `(pred) AND NOT ((prior1) OR (prior2) …)`. Summed over statements this is inclusion-exclusion — `Σ |pₙ \ (p₁ ∪ …)| = |p₁ ∪ p₂ ∪ …|` — exactly the distinct count the combined `(p1) OR (p2)` staged delete removes. Works for nodes and edges alike with no edge identity needed; the node ID scan uses the same exclusion so a later statement also doesn't re-cascade already-deleted nodes. The ORIGINAL predicate is still what gets recorded (the staged delete removes the union); only the count uses the exclusion. The common single-delete path is unchanged (`prior` empty → filter is just the base predicate). New helper `dedup_delete_filter` + `MutationStaging::recorded_delete_predicates`. Turns the red regression test green (2 nodes / 3 edges); writes (33), end_to_end, validators, maintenance, recovery, composite_flow, merge_truth_table, consistency, changes, and failpoints (63) all stay green. * test(engine): delete dedup must not drop NULL-column rows (red) Follow-up to the overlapping-delete fix flagged in PR #308 review (Greptile P1): the `(base) AND NOT (prior)` exclusion breaks under SQL three-valued logic. If a prior delete predicate references a NULLable column, a later statement's matching row whose column is NULL makes `prior` evaluate to UNKNOWN, `NOT UNKNOWN` is UNKNOWN, and the row is filtered out of the scan — even though the prior delete never matched it. That drops it from `deleted_ids`, skipping its cascade (orphaned edges) or, if it is the only match, leaving the node undeleted. A data bug, not just a miscount. Data: Charlie(age 35), Zoe(age NULL); Knows Zoe→Charlie. `delete Person where age > 30` then `delete Person where name = "Zoe"`. Under the buggy `NOT`, Zoe's scan `(name='Zoe') AND NOT (age>30)` is UNKNOWN → Zoe survives. RED at this commit (Person count left=1, right=0). * fix(engine): NULL-safe delete dedup — exclude only definitely-matched prior rows Change `dedup_delete_filter` from `(base) AND NOT (prior)` to `(base) AND ((prior) IS NOT TRUE)`. `IS NOT TRUE` keeps both FALSE and UNKNOWN rows, so a prior predicate that evaluates to SQL UNKNOWN (a NULL in a referenced column) no longer drops a row this statement legitimately matches — only rows a prior predicate matched as definitely TRUE are excluded from the count/scan. The distinct-count semantics are unchanged for non-NULL data. Turns the red NULL-dedup test green (Zoe deleted, her edge cascaded), and the overlapping-dedup + writes/end_to_end/validators/maintenance/recovery/ composite_flow/consistency suites stay green. * docs(engine): note dedup_delete_filter's load-bearing dependency on D₂ Self-review follow-up: the overlapping-delete dedup assumes the committed snapshot is invariant across a query's statements, which holds only because D₂ forbids mixing writes with deletes (so a delete-touched table has no pending writes). Make that dependency explicit at the function so a future D₂ relaxation is forced to revisit the dedup. Comment-only. * Preserve staged write commit metadata
24 KiB
24 KiB
Lance Docs Index (for OmniGraph agents)
OmniGraph sits on top of Lance. Many problems — index lifecycle, branching, transactions, fragments, compaction, vector/FTS internals — are answered upstream in Lance's docs, not in this codebase.
This file is the curated entry point. When you hit a Lance-shaped problem, find the matching topic below and fetch the listed URL(s) before guessing. Don't grep our codebase for behavior that is documented authoritatively in Lance.
Base URL: https://lance.org. Fetch the FULL page content, not summaries — use curl -sL <url> | pandoc -f html -t markdown or paste the rendered page text manually. Tools that summarize pages (like Claude's WebFetch) routinely drop load-bearing details — defaults, pub(crate) blockers, sub-specs hidden behind navigation hubs. Never act on a summarized fetch alone. Keep this index curated to relevant material — the upstream sitemap has hundreds of URLs (notably the Namespace REST API model surface, Spark/Trino/Databricks integrations) that we don't use.
Substrate boundary check. Before fetching, recall docs/dev/invariants.md: if Lance already does the thing, we don't reimplement it. The most common reason to read these docs is to confirm a substrate behavior, not to learn what to clone.
Quick-start (read these once per project)
| Read when | URL |
|---|---|
| Onboarding to Lance — concepts in 10 min | https://lance.org/quickstart/ |
| Onboarding to vector search | https://lance.org/quickstart/vector-search/ |
| Onboarding to full-text search | https://lance.org/quickstart/full-text-search/ |
| Onboarding to versioning / time travel | https://lance.org/quickstart/versioning/ |
| Lance's own AGENTS.md (its agent guide) | https://lance.org/format/AGENTS/ |
By problem domain
Storage format & file layout
Touching db/manifest, fragment lifecycle, dataset reconstruction, or anything that reads/writes raw Lance state.
| Topic | URL |
|---|---|
| Lance file format overview | https://lance.org/format/ |
| File-level format spec | https://lance.org/format/file/ |
| File encoding | https://lance.org/format/file/encoding/ |
| File-level versioning | https://lance.org/format/file/versioning/ |
| Table layout (fragments, manifest) | https://lance.org/format/table/layout/ |
| Table schema metadata | https://lance.org/format/table/schema/ |
| Table-level versioning | https://lance.org/format/table/versioning/ |
| Transactions (commit semantics, conflict types) | https://lance.org/format/table/transaction/ |
| MemWAL (durability story) | https://lance.org/format/table/mem_wal/ |
| Row-ID lineage (stable row IDs) | https://lance.org/format/table/row_id_lineage/ |
| Branches & tags (Lance native) | https://lance.org/format/table/branch_tag/ |
Branching / tags / time travel
Touching graph-level branches, snapshots, run isolation, the commit graph.
| Topic | URL |
|---|---|
| Branch & tag format | https://lance.org/format/table/branch_tag/ |
| Tags & branches operational guide | https://lance.org/guide/tags_and_branches/ |
| Versioning quick-start | https://lance.org/quickstart/versioning/ |
| Table-level versioning spec | https://lance.org/format/table/versioning/ |
Indexes
Adding/changing index types, fixing coverage, debugging FTS or vector recall, designing the reconciler.
Reads & writes
Touching the bulk loader, mutation execution, merge_insert, WriteMode selection.
| Topic | URL |
|---|---|
| Read-and-write guide | https://lance.org/guide/read_and_write/ |
| Distributed write | https://lance.org/guide/distributed_write/ |
| Rust example: write & read a dataset | https://lance.org/examples/rust/write_read_dataset/ |
Schema evolution
Touching apply_schema, the migration planner, additive evolution.
| Topic | URL |
|---|---|
| Data-evolution guide | https://lance.org/guide/data_evolution/ |
| Migration guide | https://lance.org/guide/migration/ |
Object store / S3
Touching storage.rs, S3-compatible backends (RustFS, MinIO), env vars.
| Topic | URL |
|---|---|
| Object-store guide | https://lance.org/guide/object_store/ |
Data types
Touching schema-language scalar mappings, blob columns, JSON, list columns.
| Topic | URL |
|---|---|
| Data types overview | https://lance.org/guide/data_types/ |
| Arrays / list types | https://lance.org/guide/arrays/ |
| Blobs (LargeBinary) | https://lance.org/guide/blob/ |
| JSON | https://lance.org/guide/json/ |
Performance & tuning
Optimizing scans, fragment counts, cache behavior, memory pool sizing.
| Topic | URL |
|---|---|
| Performance guide | https://lance.org/guide/performance/ |
Compaction & cleanup
Touching omnigraph optimize / cleanup, the underlying compact_files / cleanup_old_versions.
| Topic | URL |
|---|---|
Read-and-write guide (covers compact_files, cleanup_old_versions) |
https://lance.org/guide/read_and_write/ |
| Performance (compaction tradeoffs) | https://lance.org/guide/performance/ |
| Fragment-reuse index | https://lance.org/format/index/system/frag_reuse/ |
DataFusion integration
The runtime substrate that may carry our query execution. See docs/dev/invariants.md: we don't rebuild relational machinery.
| Topic | URL |
|---|---|
| DataFusion integration | https://lance.org/integrations/datafusion/ |
SDK reference
Looking up a specific Rust API (signature, return type, error variant).
| Topic | URL |
|---|---|
| SDK docs landing | https://lance.org/sdk_docs/ |
What's not in this index (and why)
- Namespace REST API model surface (
/format/namespace/client/operations/models/...) — hundreds of REST schema docs for the Lance Namespace catalog API. Omnigraph does not run a Lance Namespace server, so these are not reachable from our problem space. - Spark / Trino / Databricks / Dataproc / Hive / Glue / Polaris / Iceberg / Unity / OneLake / Gravitino integrations — not part of OmniGraph's deployment surface.
- Python / TF / PyTorch / Hugging Face / Ray integrations — OmniGraph is Rust-only; Python notebooks aren't relevant.
- Community / governance / release / voting / PMC pages — meta, not technical.
If a future need pulls one of these into scope, add a row to the matching domain section above and link it from AGENTS.md's topic index.
Maintenance
When Lance ships a major release that changes any of the above (file format bump, new index type, transaction semantics change, new branching primitive), refresh this index in the same change as the omnigraph upgrade. Stale Lance pointers are worse than no pointers.
Last alignment audit: 2026-06-15 (Lance 7.0.0 upstream; omnigraph pinned at 7.0.0)
Migration from Lance 6.0.1 → 7.0.0 landed in this cycle. Arrow stayed 58, DataFusion stayed 53 (no change) — the only transitive bump is object_store 0.12.5 → 0.13.2. 141 upstream commits reviewed (6.0.1 → 7.0.0); no fixes lost (the 6.0.x release-branch backports are all forward-ported into 7.0.0). Behavior-affecting findings:
- object_store 0.13 moved convenience methods behind a new
ObjectStoreExttrait (get/put/head/rename/delete;list/list_with_delimiter/put_optsstay on the coreObjectStoretrait). Fix = adduse object_store::ObjectStoreExt;tostorage.rsanddb/manifest/namespace.rs; no call-site changes. Mirrors Lance's own migration in PR #6672. The local-FSPutMode::Updategap is unchanged (still unimplemented upstream), sostorage.rs::write_text_if_match's local content-token emulation stays. roaringmust be pinned to 0.11.4 (cargo update -p roaring --precise 0.11.4). Lance 7.0.0'sUpdatedFragmentOffsetsnewtype (PR #6650) derivesEqoverHashMap<u64, RoaringBitmap>, which needsRoaringBitmap: Eq— added only in roaring 0.11.4 (roaring-rs PR #341). Lance's looseroaring = "0.11"constraint otherwise resolves the broken 0.11.3 and lance itself fails to compile (RoaringBitmap: Eq is not satisfied). roaring is transitive (no direct workspace dep); the pin lives only inCargo.lock._row_created_at_versionfor merge-insert INSERT rows now = the commit version (PR #6774; was a fallback of 1 / dataset-creation version). Flippedlance_version_columns.rs::lance_merge_insert_new_row_stamps_created_at_versionto assert== v2. Production change-detection keys on_row_last_updated_at_version+ ID-set membership, so classification logic is unaffected (thechanges/mod.rsrationale comment was corrected).- BTREE range-query bound inclusiveness fixed (PR #6796, issue #6792):
x <= hi AND x > loreturned the wrong boundary row on 6.0.1. omnigraph today builds BTREE only on string@keycolumns (id/src/dst) and queries them by equality/IN, not range, so its current query patterns almost certainly never hit this bug — but the corrected boundary semantics are a contract we rely on the moment a BTREE-range path appears (BTREE-on-properties via the index-type tickets, or a range-on-key query). Pinned bylance_surface_guards.rs::btree_range_query_boundary_is_correct(reproduces #6792's 5-row + BTREE shape). WriteParams::auto_cleanupdefault flipped from on (every-20-commits) toNone(PR #6755). On 6.0.1 the on-by-default hook could GC versions the__manifestpins for snapshots/time-travel. omnigraph owns cleanup explicitly (optimize.rs::cleanup_all_tables). Two parts to the fix, becauseauto_cleanupis create-time config only and has no effect on existing datasets (Lancewrite.rsdocs): (1)auto_cleanup: Noneat all 11WriteParamssites so new datasets store no cleanup config; (2) — the load-bearing half —skip_auto_cleanup: trueon every commit path, because graphs created before the bump still carry the on-config in their datasets, and Lance's hook fires off the dataset's stored config at commit time (io/commit.rs:if !commit_config.skip_auto_cleanup). So the staged commit path (commit_staged→CommitBuilder::with_skip_auto_cleanup(true)), the__manifestpublisher (MergeInsertBuilder::skip_auto_cleanup(true)), and the directWriteParamspaths all skip the hook. Without this, an upgraded graph would still auto-cleanup and delete__manifest-pinned versions. Pinned bylance_surface_guards.rs::skip_auto_cleanup_suppresses_version_gc(negative control + with-skip survival).- Lance #6658 SHIPPED in 7.0.0 (
DeleteBuilder::execute_uncommitted, exposed via PR #6781) → MR-A (migratedeleteto the staged two-phase API) has since landed (dev-graphiss-950):delete_whereis retired, deletes stage viaTableStorage::stage_delete, and the guard was flipped to_compile_uncommitted_delete_field_shape(pinsexecute_uncommitted/UncommittedDelete).StagedWritemust carryUncommittedDelete.affected_rowsthroughcommit_stagedso Lance's row-level rebase metadata is preserved. The parse-time D2 rule is retained as a deliberate boundary (constructive XOR destructive per query), not as scaffolding awaiting further work. - The unenforced primary key is now immutable once set (
lance::dataset::transaction, ~L2472–2480:if !primary_key_before.is_empty() && (writes_primary_key || primary_key_after != primary_key_before) → "the unenforced primary key is a reserved key and cannot be changed once set"). omnigraph marks__manifest.object_idas the unenforced PK (lance-schema:unenforced-primary-key) for merge-insert row-level CAS — baked intomanifest_schema()at init, and added by themigrate_v1_to_v2internal-schema migration for pre-v0.4.0 graphs. The migration relied on Lance 6's idempotent re-apply for crash-recovery (a crash after the field-set but before the stamp bump re-enters the migration with the PK already present); under v7 that re-apply errors, so a real v1 graph could never finish migrating. Fixed by guarding the set on the manifest's unenforced-PK field (db/manifest/migrations.rs::migrate_v1_to_v2):["object_id"]→ no-op,[]→ set, any other PK field → loud refusal (the wrong CAS key, unchangeable under v7). Pinned bylance_surface_guards.rs::unenforced_primary_key_is_immutable_once_set(red if Lance relaxes immutability); regression:db::manifest::tests::test_publish_migrates_pre_stamp_manifest_to_current_version(was red under v7). - Native
DirectoryNamespaceno longer recognizes omnigraph's manifest-tracked tables (lance-namespace-implsdir.rs ~L1310):list/describe/create_table_versionroute throughcheck_table_status, which reports an omnigraph table absent →TableNotFound. The decoupling is contingent on omnigraph's legacy boolean PK key, not an unconditional v7 property: v7's namespace eagerly adds the newlance-schema:unenforced-primary-key:positionkey to any__manifestlacking it; that write hits the immutable-PK rule above (the boolean key already set the PK), soensure_manifest_table_up_to_dateerrors and the namespace silently falls back to directory listing. omnigraph keeps the boolean key deliberately — Lance honors it permanently (maps to PK position 0), and one uniform on-disk format beats a new-vs-old split (existing graphs can't be re-keyed to the position key under that same immutability rule). omnigraph production never uses Lance's native namespace (its publisher writes__manifestdirectly via merge_insert; its ownnamespace.rsimpls are custom), so this is test-only — thetest_directory_namespace_direct_publish_cannot_replace_native_omnigraph_write_pathsurface guard was realigned to the v7 behavior (it now asserts the native namespace is fully decoupled, which only strengthens the guard's thesis). - Still NOT fixed in 7.0.0: vector-index two-phase (Lance #6666 open) —
create_vector_indexinline residual retained; blob-column compaction —compact_files_still_fails_on_blob_columnsguard still red on a fix,optimizestill skips blob tables behindLANCE_SUPPORTS_BLOB_COMPACTION. - No Lance API surface omnigraph uses changed at compile time (the only compile break was object_store) — but two runtime behaviors did (the unenforced-PK immutability and the native-namespace
TableNotFound, above), each caught by the full engine test suite rather than the build.CleanupPolicy,WriteParams(apart from theauto_cleanupdefault),CompactionOptions, the namespace models (resolved vialance-namespace-reqwest-client0.7.7, unchanged across the bump),Operation,ManifestLocation, andMergeInsertBuildershapes are all stable. Lesson: a clean build is not a clean alignment — runcargo test --workspacebefore declaring a Lance bump done. - Two surface guards added by the v3→v4 migration-robustness follow-up (not a Lance bump, but they pin Lance error surfaces the migration now classifies on):
dataset_open_missing_returns_not_found_variant(a missingDataset::openreturnsDatasetNotFound/NotFound— the legacy-open read indb/commit_graph.rs::read_legacy_commit_cachetreats only those as "no legacy data" and propagates everything else) andlance_error_incompatible_transaction_variant_exists(a concurrentUpdateConfigstamp-bump loses withIncompatibleTransaction—db/manifest/migrations.rs::commit_v4_stamp_idempotentlymatches it to retry the benign same-value race). Re-run on a Lance bump like the others.
Bump this date stanza on the next alignment pass.
Prior alignment audit: 2026-05-22 (Lance 6.0.1 upstream; omnigraph pinned at 6.0.1)
Migration from Lance 4.0.0 → 6.0.1 landed in this cycle (DataFusion 52 → 53, Arrow 57 → 58, lance-tokenizer 6.0.1 added, tantivy* removed). Direct 4 → 6 jump; v5.x was not used as an intermediate (rationale in ~/.claude/plans/shimmering-percolating-duckling.md). Behavior-affecting findings:
- DatasetIndexExt moved from
lance-indextolance::index(Lance PR #6280, v5.0). Six import sites updated.lance-index::IndexTypeandlance-index::is_system_indexstayed inlance-index.omnigraph-cliandomnigraph-servergainedlance = { workspace = true }in their dev-dependencies. DescribeTableResponsegainedis_only_declared: Option<bool>(lance-namespace 6.0+, v5.0 PR #6186). Set toSome(false)in bothBranchManifestNamespace::describe_tableandStagedTableNamespace::describe_table— every table we return is physically materialized viaDataset::open, never "declared-only."MergeInsertBuilderexecute_reader return shape preserved(Arc<Dataset>, MergeStats); the publisher CAS chain atdb/manifest/publisher.rs:370-391works unchanged. Pinned bytests/lance_surface_guards.rs::_compile_merge_insert_builder_method_chain.LanceError::TooMuchWriteContentionvariant retained in v6.0.1 (no rename). The typed publisher translation atdb/manifest/publisher.rs:417-430continues to apply. Pinned bylance_surface_guards.rs::lance_error_too_much_write_contention_variant_exists.ManifestLocationfield shape stable:.path: object_store::path::Path,.size: Option<u64>,.e_tag: Option<String>,.naming_scheme: ManifestNamingScheme. Pinned bylance_surface_guards.rs::manifest_location_field_shape.LanceFileVersion::default()flipped V2_0 → V2_1 (v5.0). No effect — everydata_storage_versioncallsite explicitly pinsSome(LanceFileVersion::V2_2)(load-bearing for blob v2:Blob v2 requires file version >= 2.2enforced inlance/src/dataset/write.rs:748).Dataset::checkout_version(N).await?.restore().await?:restore()takes&mut selfand returnsResult<()>(mutates in place, does not consume + return a new dataset). The recovery rollback hammer atdb/manifest/recovery.rs:505-522continues to work. Pinned bylance_surface_guards.rs::_compile_checkout_version_then_restore_signature.DatasetBuilder::from_namespace(...).with_branch(...).with_version(...).load()surface preserved (the namespace builder chain atdb/manifest/namespace.rs:162-174). Pinned bylance_surface_guards.rs::_compile_dataset_builder_from_namespace_signature.compact_files(&mut ds, CompactionOptions::default(), None)signature stable.CompactionOptionsstill does not exposedata_storage_version;compact_filesbuilds its ownWriteParams { ..Default::default() }. Note:LanceFileVersion::default()is now V2_1 in v6, so optimize-rewritten fragments come out at V2_1 by default (was V2_0 in v4). Existing explicit V2_2 pins on creates/appends still apply.Dataset::optimize_indices(&mut self, &lance_index::optimize::OptimizeOptions)(viaDatasetIndexExt) is a depended-on surface as of the index-coverage work:db/omnigraph/optimize.rscalls it aftercompact_filesto fold appended/rewritten fragments into existing indexes (incremental merge, not retrain). It is a committing call (mutates in place, advances HEAD; no uncommitted variant in v6.0.1), so optimize treats it as an inline-commit residual under theSidecarKind::Optimizerecovery sidecar. Signature pinned bylance_surface_guards.rs::_compile_optimize_indices_signature; the incremental-coverage behavior pinned byoptimize_indices_extends_fragment_coverage(appended fragment uncovered before, covered after).Dataset::delete(predicate)returnsDeleteResult { new_dataset: Arc<Dataset>, num_deleted_rows: u64 }— unchanged shape. Pinned bylance_surface_guards.rs::_compile_delete_result_field_shape. MR-A will repurpose this guard to the staged two-phase variant onceDeleteBuilder::execute_uncommittedmigration lands.- File reader read methods now async (Lance PR #6710, v6.0). No effect — omnigraph reaches Lance exclusively through
Dataset::scanand the staged-write API. - Tokenizer vendored as
lance-tokenizer(Lance PR #6512, v6.0). No effect — no direct tokenizer imports. - Lance #6658 closed (2026-05-14) but
DeleteBuilder::execute_uncommitteddid not ship in v6.0.1 — binary search across the release stream shows it first appears inv7.0.0-beta.10(the closing commits landed on main but didn't backport to the 6.x line). Tracked as MR-A: migratedelete_whereto staged, retire the parse-time D2 mutation rule, extend recovery sidecar coverage. Gated on the Lance v7.x bump, not this PR. v7.0.0-rc.1 dropped 2026-05-21. - Lance #6666 still open (
build_index_metadata_from_segmentspublic): vector-index two-phase blocked; inlinecreate_vector_indexresidual retained. - Lance #6877 still open (
MergeInsertBuilderdup-rowid): PR #109'sSourceDedupeBehavior::FirstSeen+check_batch_unique_by_keysprecondition stay load-bearing. Dataset::force_delete_branch(branches().delete(name, force=true), dataset.rs:524) tolerates a missing branch-contents ref (vs plaindelete_branch'sRefNotFound), but on the local store still errorsNotFoundif the branchtree/directory is fully absent (remove_dir_all's NotFound is not caught for Lance's native error variant, refs.rs:526-549). Both variants still refuse a branch with referencing descendants (RefConflict).TableStore::force_delete_branchwraps this to be fully idempotent (tolerates already-absent). The single-authority branch-delete redesign uses it for orphan reclamation (eager best-effort reclaim + cleanup reconciler). Pinned bylance_surface_guards.rs::force_delete_branch_semantics. Branch delete is "flip the ref atomically, thenremove_dir_all(tree/{branch})"; branch-exclusive data lives undertree/{branch}/so a drop reclaims it immediately without touchingmain.- Lance blob-v2
compact_filesbug (no public issue found as of 2026-06):compact_filesdisables binary-copy for blob datasets and forcesBlobHandling::AllBinaryon the read side; the v2.1+ structural decoder then mis-counts column infos for the blob-v2 struct and fails withInvalid user input: there were more fields in the schema than provided column indices / infos(lance-encoding/src/decoder.rs::ColumnInfoIter::expect_next). This fails even a pristine uniform-V2_2 multi-fragment blob table; vector/list/scalar/ragged columns and mixed file versions all compact fine. Reads/queries use descriptor handling (BlobHandling::default()) and are unaffected.optimizeskips blob-bearing tables behindLANCE_SUPPORTS_BLOB_COMPACTION = false(db/omnigraph/optimize.rs), reportingSkipReason::BlobColumnsUnsupportedByLance. Pinned bylance_surface_guards.rs::compact_files_still_fails_on_blob_columns, which turns red when the bug is fixed → flip the gate, remove the skip branch + themaintenance.rs::optimize_skips_blob_table_and_reports_skipskip assertions.
Surface guards added: crates/omnigraph/tests/lance_surface_guards.rs (10 named guards; 5 runtime + 5 compile-only; plus the index-coverage work's _compile_optimize_indices_signature and optimize_indices_extends_fragment_coverage). Future Lance bumps re-run this file first as the smoke check. Two additional guards from the original plan deferred to follow-up (manifest_cas_returns_row_level_contention_variant needs full publisher-race harness; table_version_metadata_byte_compatible_with_v4 needs pub(crate) reach extension).