docs(lance): 9.0.0-beta.15 alignment stanza; close the vendor-pin and blob-compaction gaps

New dated audit stanza in lance.md (382 upstream commits across the 7->8
and 8->9-beta.15 legs; zero ecosystem churn, no format/reader-floor
movement, the FTS-v2 soft door, the merge_insert rewrite, the open
items #6666/#6914/#7508). invariants.md: the vendored-lance-table and
blob-column-compaction known gaps close per their documented removal
conditions. AGENTS.md: substrate line + compaction matrix row. testing.md:
the blob guard's new positive name.

AGENTS.md

@@ -18,7 +18,7 @@ Tools that support `@`-imports (Claude Code) auto-include all three files via th
 
 **Version surveyed:** 0.8.0
 **Workspace crates:** `omnigraph-compiler`, `omnigraph` (engine), `omnigraph-policy`, `omnigraph-api-types` (shared HTTP wire DTOs), `omnigraph-cluster`, `omnigraph-cli`, `omnigraph-server`
-**Storage substrate:** Lance 7.x (columnar, versioned, branchable)
+**Storage substrate:** Lance 9.x (columnar, versioned, branchable; 9.0.0-beta.15 git-rev pin until 9.0.0 stable)
 **License:** MIT
 **Toolchain:** Rust stable, edition 2024
 
@@ -253,7 +253,7 @@ omnigraph policy explain --cluster ./company-brain --graph knowledge --actor act
 | Per-dataset versioning + time travel | ✅ | `snapshot_at_version`, `entity_at`, snapshot-pinned reads across many tables |
 | Per-dataset branches | ✅ | **Graph-level** branches (atomic across all sub-tables), lazy fork, system branch filtering |
 | Atomic single-dataset commits | ✅ | **Multi-table publish via three layers**, NOT a single Lance primitive: (1) per-table Lance `commit_staged` for the data write, (2) `__manifest` row-level CAS via `ManifestBatchPublisher` for cross-table ordering, (3) the open-time recovery sweep for the residual gap between (1) and (2). All three layers ship; the five migrated writers (`MutationStaging::finalize`, `schema_apply`, `branch_merge`, `ensure_indices`, `optimize_all_tables`) write a `__recovery/{ulid}.json` sidecar before Phase B and delete it after Phase C. The next `Omnigraph::open` (gated on `OpenMode::ReadWrite`) runs the sweep in `db/manifest/recovery.rs`: classify, decide all-or-nothing per sidecar, roll forward via single `ManifestBatchPublisher::publish` or roll back via `Dataset::restore` followed by a manifest publish of the restored version (so both directions converge to `manifest == HEAD` — no residual drift), and record an audit row in `_graph_commit_recoveries.lance` (queryable via `omnigraph commit list --filter actor=omnigraph:recovery`). The write entry points (`load_as`, `mutate_as`, `apply_schema_as`, `branch_merge_as`) and `refresh` additionally run an in-process roll-forward-only heal (serialized against live writers via the per-table write queues), so a long-lived server converges on its next write without restart; only rollback-eligible sidecars still defer to the next read-write open (a future background reconciler's goal). Engine writes route through a sealed `TableStorage` trait (`db.storage()`) exposing only `stage_*` + `commit_staged` + reads; the sole inline-commit residual (`create_vector_index`) is split onto a separate sealed `InlineCommitResidual` trait reached via `db.storage_inline_residual()` (MR-854), so the default surface cannot couple a write with a HEAD advance — §1 holds by construction. `delete` migrated to the staged path in MR-A (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, [#6658](https://github.com/lance-format/lance/issues/6658)); `create_vector_index` stays inline until upstream Lance ships a public two-phase API ([#6666](https://github.com/lance-format/lance/issues/6666)); `LoadMode::Overwrite` uses Lance `Overwrite` staged transactions. |
-| Compaction (`compact_files`) + reindex (`optimize_indices`) | ✅ | `omnigraph optimize` orchestrates over all node/edge tables, bounded concurrency; per table runs `compact_files` **then Lance `optimize_indices`** (folds appended/rewritten fragments back into existing indexes — incremental merge, not retrain) and **publishes the resulting version to `__manifest`** (so the manifest tracks the Lance HEAD — required for reads to observe the work and for schema apply / strict writes to pass their HEAD-vs-manifest precondition), under the per-`(table, main)` write queue with `SidecarKind::Optimize` recovery coverage spanning both ops; **commits even with no compaction work if index coverage is stale**; **refuses on an unrecovered graph**; **skips uncovered HEAD > manifest drift** with `DriftNeedsRepair`; **skips blob-bearing tables** (reported via `TableOptimizeStats.skipped`, not silent; reindex is skipped for them too today), gated on `LANCE_SUPPORTS_BLOB_COMPACTION` until the upstream blob-v2 compaction-decode bug is fixed (see [docs/dev/invariants.md](docs/dev/invariants.md) Known Gaps) |
+| Compaction (`compact_files`) + reindex (`optimize_indices`) | ✅ | `omnigraph optimize` orchestrates over all node/edge tables, bounded concurrency; per table runs `compact_files` **then Lance `optimize_indices`** (folds appended/rewritten fragments back into existing indexes — incremental merge, not retrain) and **publishes the resulting version to `__manifest`** (so the manifest tracks the Lance HEAD — required for reads to observe the work and for schema apply / strict writes to pass their HEAD-vs-manifest precondition), under the per-`(table, main)` write queue with `SidecarKind::Optimize` recovery coverage spanning both ops; **commits even with no compaction work if index coverage is stale**; **refuses on an unrecovered graph**; **skips uncovered HEAD > manifest drift** with `DriftNeedsRepair`; **compacts blob-bearing tables** (the pre-9 `LANCE_SUPPORTS_BLOB_COMPACTION` skip was removed once Lance 8.0.0+ shipped blob-v2 compaction — see [docs/dev/invariants.md](docs/dev/invariants.md) Known Gaps) |
 | Repair uncovered drift | — | `omnigraph repair` explicitly classifies uncovered table `HEAD > manifest` drift: verified maintenance drift (`ReserveFragments`/`Rewrite`) can be published with `--confirm`; suspicious or unverifiable drift requires `--force --confirm`. Sidecar-covered crash residuals still recover automatically on open. |
 | Cleanup (`cleanup_old_versions`) | ✅ | `omnigraph cleanup` with `--keep` / `--older-than` policy |
 | BTREE / inverted (FTS) / vector indexes | ✅ | `@index`/`@key` declares intent; the physical index is derived state that never fails a logical op. Built per column through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector); idempotent; lazy across branches. **Schema apply builds nothing** (records intent only); `load`/`mutate` build inline but **defer an untrainable Vector column** (no trainable vectors yet) as *pending* rather than aborting. `ensure_indices`/`optimize` is the reconciler that materializes declared-but-missing indexes and restores coverage of appended/rewritten fragments (`optimize_indices`), reporting still-pending columns (see Compaction row). |

docs/dev/invariants.md

@@ -193,31 +193,28 @@ them explicit.
   and [writes.md](writes.md). New write paths should use the staged shape unless a
   documented Lance blocker applies.
 - **Vector indexes:** `create_vector_index` still advances Lance HEAD inline —
-  segment-commit needs `build_index_metadata_from_segments`, `pub(crate)` in Lance
-  7.0.0 (#6666 open). Keep recovery coverage in place until that residual is
+  segment-commit needs `build_index_metadata_from_segments`, still `pub(crate)` at Lance
+  9.0.0-beta.15 (#6666 open). Keep recovery coverage in place until that residual is
   removed. (`delete` is no longer a residual — staged in MR-A. D2 is not a gap:
   it is a deliberate constructive-XOR-destructive boundary, documented in
   Invariant 4 and the truth matrix.)
-- **Vendored lance-table pin (lance#7480):** `lance-table` 7.0.0 resolves to
-  `vendor/lance-table` via `[patch.crates-io]` — the pristine published source
-  plus ONLY the lance#7480 `rowids/index.rs` hunk, without which any filtered
-  read on a table that was merge-updated and then delete-touched fails
-  (lance#7444; `iss-merge-rowid-overlap-corrupts-filtered-reads`). The fix
-  ships in no Lance release ≤ 8.0.0. Remove the vendor dir + patch entry at
-  the first bump whose `lance-table` carries the fix (9.0.0, or a backported
-  8.0.1); `lance_surface_guards.rs::filtered_scan_tolerates_merge_update_row_id_overlap`
-  turns red if the pin is dropped early or a bump regresses it. See the
-  2026-07-02 stanza in [lance.md](lance.md) and
-  `vendor/lance-table/README.omnigraph.md`.
-- **Blob-column compaction:** Lance `compact_files` mis-decodes blob-v2 columns
-  under its forced `BlobHandling::AllBinary` read ("more fields in the schema
-  than provided column indices"), so `optimize` skips any table with a `Blob`
-  property — reporting `SkipReason::BlobColumnsUnsupportedByLance` (loud, not a
-  silent drop) behind the `LANCE_SUPPORTS_BLOB_COMPACTION` gate. Reads and writes
-  are unaffected; only space/fragment reclamation on blob tables is deferred.
-  Remove the skip when the upstream Lance fix lands — the
-  `lance_surface_guards.rs::compact_files_still_fails_on_blob_columns` guard
-  turns red on that bump to force it.
+- **Vendored lance-table pin — CLOSED (9.0.0-beta.15 bump):** lance#7480
+  shipped upstream in 9.0.0-beta.11, so the `vendor/lance-table` pin and its
+  `[patch.crates-io]` entry were removed per their documented removal
+  condition.
+  `lance_surface_guards.rs::filtered_scan_tolerates_merge_update_row_id_overlap`
+  passes on stock lance-table and remains the regression tripwire. Note the
+  release exposure: binaries ≤ v0.8.0 predate even the pin — the rebuild-free
+  remedy for fleets is upgrading the binary (see the 2026-07-05 stanza in
+  [lance.md](lance.md)).
+- **Blob-column compaction — CLOSED (9.0.0-beta.15 bump):** Lance 8.0.0+
+  compacts blob-v2 correctly (upstream #7017, hardened by #7618). The
+  `LANCE_SUPPORTS_BLOB_COMPACTION` gate, the optimize skip branch, and
+  `SkipReason::BlobColumnsUnsupportedByLance` were removed;
+  `lance_surface_guards.rs::compact_files_succeeds_on_blob_columns` and
+  `maintenance.rs::optimize_compacts_blob_table_alongside_plain_table` pin the
+  positive behavior (a red there means blob compaction regressed — restore the
+  skip machinery from git history).
 - **Recovery is serialized against live writers in-process only:** the
   write-entry heal (and `refresh`) serialize against a live writer's sidecar
   lifetime via the per-`(table, branch)` write queues plus the schema-apply

docs/dev/lance.md

@@ -156,7 +156,85 @@ If a future need pulls one of these into scope, add a row to the matching domain
 
 When Lance ships a major release that changes any of the above (file format bump, new index type, transaction semantics change, new branching primitive), refresh this index in the same change as the omnigraph upgrade. Stale Lance pointers are worse than no pointers.
 
-### Patch pin: 2026-07-02 (vendored lance-table 7.0.0 + lance#7480; omnigraph still pinned at 7.0.0)
+### Last alignment audit: 2026-07-05 (Lance 9.0.0-beta.15 upstream; omnigraph pinned at 9.0.0-beta.15 via git rev)
+
+Migration from Lance 7.0.0 → 9.0.0-beta.15 landed in this cycle. The 9.x betas
+are **git tags only** (crates.io carries ≤ 8.0.0), so every lance crate is a git
+dependency rev-pinned to the `v9.0.0-beta.15` tag commit (`f24e42c1`); switch to
+the crates.io release at 9.0.0 stable. 382 upstream commits reviewed across two
+audit legs (243 in 7→8, 139 in 8→9-beta.15). **Arrow stayed 58, DataFusion
+stayed 53, object_store stayed 0.13.2** — zero ecosystem churn. **No table/file
+format or minimum-reader-version movement in either leg**: data written by this
+binary under our settings (explicit V2_2 pins) round-trips to a 7.0.0 reader;
+the one soft door is FTS v2 index files (default for NEW inverted indexes since
+9.0.0-beta.11 — readable by 7.0.0+, rebuildable derived state). Behavior-affecting
+findings:
+
+- **lance#7480 shipped (9.0.0-beta.11)** → the `vendor/lance-table` pin and its
+  `[patch.crates-io]` entry are **retired** per their documented removal
+  condition; `filtered_scan_tolerates_merge_update_row_id_overlap` now passes on
+  stock lance-table and stays as the regression tripwire.
+- **lance#7320 shipped (9.0.0-beta.1)** → the sequential BTREE segment-merge
+  corruption (lance#7230: update-preserved row ids duplicated by
+  `build_stable_row_id_filter` when compaction skips the superseded fragment;
+  empirically reproduced on 7.0.0 via load → keyed update → optimize → broken
+  filtered reads AND broken keyed writes) is fixed upstream. No engine-side
+  mitigation needed.
+- **Blob-v2 compaction fixed** (8.0.0 PR #7017, hardened by #7618 in beta.15
+  after a beta.13 regression) → executed the documented removal plan:
+  `LANCE_SUPPORTS_BLOB_COMPACTION`, the optimize skip branch, and
+  `SkipReason::BlobColumnsUnsupportedByLance` deleted; the guard inverted to a
+  positive pin (`compact_files_succeeds_on_blob_columns`) and
+  `maintenance.rs::optimize_compacts_blob_table_alongside_plain_table` asserts
+  blob tables compact, publish, and keep every row.
+- **Filter-literal coercion improved** (v8 #6935 et al.): a width-mismatched
+  literal (`n32 = 5i64`) is now coerced to the column type BEFORE pushdown and
+  USES the BTREE (7.0.0 planned an index-defeating column cast).
+  `scalar_index_use_requires_matched_literal_type` re-pinned to the new truth;
+  `query.rs::literal_to_typed_expr` stays load-bearing.
+- **Branch-consistency enforcement on open** (9.x): `DatasetBuilder` now errors
+  loudly when the resolved manifest belongs to a different branch than
+  requested ("open of branch X resolved a manifest belonging to branch Y") —
+  the substrate enforcing what omnigraph's entry-owner resolution always did.
+  Production opens by owner-resolved location (unaffected); the lazy-fork
+  namespace test pins both the error and the owner-branch open.
+- **Native DirectoryNamespace churn** (#7222 removed
+  `table_version_storage_enabled` + the `__manifest` version-storage
+  experiment; #7176/#7191/#7234 rewrote manifest handling): the decoupling
+  guard's thesis holds at v9 (manifest-tracked tables still `TableNotFound` to
+  native tooling with dir-listing disabled) and was realigned.
+- **merge_insert substantially rewritten** (+1321 lines across #6878 composite
+  indexed join keys, #7484 WhenNotMatchedBySource::Delete/Fail fixed on the
+  indexed-scan path, #7410/#7359 stale scalar-index entries after
+  stable-row-id update): every edge-table merge (src+dst BTREE keys) now takes
+  the AND-folded index-probe path. Full suite green; the surfaces omnigraph
+  pins (`execute_reader` return shape, WhenMatched/WhenNotMatched,
+  `SourceDedupeBehavior`) are unchanged. **#6877 was fixed in 8.0.0 (#6965)**
+  — re-evaluate retiring `SourceDedupeBehavior::FirstSeen` +
+  `check_batch_unique_by_keys` in a follow-up.
+- **optimize_indices steady-state is a true no-op** (v8 #6905: no version bump
+  when there is no index work) and **FTS AND queries enforce required terms**
+  (#7385 — `match_text`/`bm25` result sets can legitimately change). Both
+  absorbed by the existing suites (68/68 green, incl. cost gates within slack).
+- **New failure surfaces**: every commit path has a default 30-minute timeout
+  (v8 #6773); `alter_columns` with a cast on an indexed column now hard-errors
+  (v8 #7158 — omnigraph's planner never emits casts today, OG-MF-106 refuses
+  type changes first).
+- **Still NOT fixed at 9.0.0-beta.15:** vector-index two-phase (lance#6666 —
+  `build_index_metadata_from_segments` still `pub(crate)`; the
+  `create_vector_index` inline residual and its recovery coverage are
+  retained), #6914 per-row version-metadata refresh (unshipped), and upstream
+  **#7508 is open** (FTS "record batch length" errors after frequent
+  `optimize_indices` — omnigraph's reconciler call pattern; watch it).
+- **RowAddrTreeMap moved** from `lance_core::utils::mask` to the new
+  `lance-select` crate — the single compile break in the whole migration.
+- Beta-line caveat: the blob read path regressed inside the line (beta.13,
+  fixed beta.15 by #7618) — pre-release churn is real; re-audit before any
+  beta→stable or beta→beta advance.
+
+Bump this date stanza on the next alignment pass.
+
+### Patch pin: 2026-07-02 (vendored lance-table 7.0.0 + lance#7480) — RETIRED at the 9.0.0-beta.15 bump (fix upstream since 9.0.0-beta.11)
 
 Not a version bump — a single-fix vendored pin. `[patch.crates-io] lance-table = { path = "vendor/lance-table" }` points at the pristine published 7.0.0 source carrying ONLY the lance#7480 `rowids/index.rs` hunk (merged upstream 2026-07-01, a few hours AFTER v8.0.0 was cut, so it ships in no release ≤ 8.0.0):
 

docs/dev/testing.md

@@ -26,7 +26,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
 | `staged_writes.rs` | TableStore staged-write primitives (`stage_append`, `stage_merge_insert`, `commit_staged`, `scan_with_staged`, `count_rows_with_staged`) — primitive-level only; engine code uses the in-memory `MutationStaging` accumulator instead |
 | `forbidden_apis.rs` | Defense-in-depth source-walk guard: engine code (`exec/`, `db/omnigraph/`, `loader/`, `changes/`) must not reach around the sealed storage trait to Lance inline-commit APIs, nor open datasets directly (`Dataset::open` / `DatasetBuilder::from_uri`/`from_namespace`) — reads route through `Snapshot::open` and the held-handle cache; `// forbidden-api-allow: <reason>` sentinel exempts reviewed lines |
 | `failpoint_names_guard.rs` | Source-walk guard (same defense-in-depth shape as `forbidden_apis.rs`): every failpoint call site (`maybe_fail`, `ScopedFailPoint::new`/`with_callback`, `Rendezvous::park_first`) must reference a `failpoints::names` const, never a bare string literal — a typo'd literal compiles and silently never fires |
-| `lance_surface_guards.rs` | Pins the Lance API surfaces omnigraph depends on (named runtime + compile-only guards; see [lance.md](lance.md)) — the first smoke check on any Lance version bump; e.g. `compact_files_still_fails_on_blob_columns` turns red when the upstream blob-compaction fix lands |
+| `lance_surface_guards.rs` | Pins the Lance API surfaces omnigraph depends on (named runtime + compile-only guards; see [lance.md](lance.md)) — the first smoke check on any Lance version bump; e.g. `compact_files_succeeds_on_blob_columns` pins blob-v2 compaction working (a red means a Lance regression) |
 | `warm_read_cost.rs` | Cost-budget tests for the warm read path (query-latency work), measured at the object-store boundary with Lance `IOTracker` (the LanceDB IO-counted pattern): a warm same-branch read does 0 manifest opens, 1 version probe, validates the schema once (Fix 1 / finding A / Fix 2 at commit-history depth); stale same-branch reads perform exactly 2 probes and refresh manifest-only; recreated non-main branches with the same Lance version refresh by incarnation; recreated branch-owned table handles are distinguished by table e_tag or refresh-time cache clearing; recreated traversal topology is protected by per-edge-table e_tag in the graph-index cache key or refresh-time cache clearing; a warm *repeat* read does 0 table opens via the held-handle cache and a write re-opens only the changed table at its new version/e_tag (Fix 3/6A). Also the CSR topology-build cost guards: `fresh_branch_traversal_reuses_main_graph_index` (A1 — a lazy-fork branch reuses main's cached CSR index, 0 rebuilds via `graph_build_count`) and `single_edge_query_builds_only_referenced_edge` (A2 — a one-edge query builds only that edge via `graph_edges_built`); both force CSR via the scoped `with_traversal_mode` seam, so they need no `#[serial]`. See "Cost-budget tests" below. |
 | `write_cost.rs` | Cost-budget tests for the WRITE path (RFC-013), the latency twin of `warm_read_cost.rs` on the **shared `helpers::cost` harness** (`measure`/`IoCounts`/`assert_flat`/`local_graph`). Runs on **local FS**; gates the **internal-table** term (`__manifest` scans flat in commit-history depth, lineage rows included — `internal_table_scans_are_flat_in_history`, now **green every-PR** since RFC-013 step 2 brought the internal tables into `optimize`; the test compacts at each depth before measuring) plus green every-PR guards (single-insert `data_writes` bounded, a per-write read-op ceiling that fails the moment a round-trip is added, and a `measure_with_staged` fitness assert that a keyed insert routes through `stage_merge_insert` once with no `stage_append`/vector-index build). The **data-table opener** term is S3-only — see `write_cost_s3.rs` and the backend-split note in "Cost-budget tests" below |
 | `write_cost_s3.rs` | Bucket-gated (skips without `OMNIGRAPH_S3_TEST_BUCKET`) twin of `write_cost.rs` on the same `helpers::cost` harness: gates the **data-table opener** term (per-write latest-version resolution flat across commit depth on a real object store — per-version GETs are invisible on local FS). A cost gate, not a correctness test — run on demand, not in the every-merge `rustfs_integration` job (see the backend-split note in "Cost-budget tests" below) |