docs(lance): record the lance-table #7480 patch pin — stanza, known gap, test-map row

Same-PR documentation for the vendored pin: a dated patch-pin stanza in
lance.md's alignment history (mechanism, pinning tests, removal
condition, and the adjacent v8-only #7251 finding that gates iss-986 on
the 7→8 bump), a Known Gaps entry in invariants.md so the pin cannot be
forgotten, and the writes.rs row in testing.md now names the row-id
overlap regression pair.
This commit is contained in:
aaltshuler 2026-07-02 02:17:25 +03:00 committed by Andrew Altshuler
parent b5c0c6238b
commit 26c420cb1a
3 changed files with 21 additions and 1 deletions

View file

@ -198,6 +198,17 @@ them explicit.
removed. (`delete` is no longer a residual — staged in MR-A. D2 is not a gap:
it is a deliberate constructive-XOR-destructive boundary, documented in
Invariant 4 and the truth matrix.)
- **Vendored lance-table pin (lance#7480):** `lance-table` 7.0.0 resolves to
`vendor/lance-table` via `[patch.crates-io]` — the pristine published source
plus ONLY the lance#7480 `rowids/index.rs` hunk, without which any filtered
read on a table that was merge-updated and then delete-touched fails
(lance#7444; `iss-merge-rowid-overlap-corrupts-filtered-reads`). The fix
ships in no Lance release ≤ 8.0.0. Remove the vendor dir + patch entry at
the first bump whose `lance-table` carries the fix (9.0.0, or a backported
8.0.1); `lance_surface_guards.rs::filtered_scan_tolerates_merge_update_row_id_overlap`
turns red if the pin is dropped early or a bump regresses it. See the
2026-07-02 stanza in [lance.md](lance.md) and
`vendor/lance-table/README.omnigraph.md`.
- **Blob-column compaction:** Lance `compact_files` mis-decodes blob-v2 columns
under its forced `BlobHandling::AllBinary` read ("more fields in the schema
than provided column indices"), so `optimize` skips any table with a `Blob`

View file

@ -156,6 +156,15 @@ If a future need pulls one of these into scope, add a row to the matching domain
When Lance ships a major release that changes any of the above (file format bump, new index type, transaction semantics change, new branching primitive), refresh this index in the same change as the omnigraph upgrade. Stale Lance pointers are worse than no pointers.
### Patch pin: 2026-07-02 (vendored lance-table 7.0.0 + lance#7480; omnigraph still pinned at 7.0.0)
Not a version bump — a single-fix vendored pin. `[patch.crates-io] lance-table = { path = "vendor/lance-table" }` points at the pristine published 7.0.0 source carrying ONLY the lance#7480 `rowids/index.rs` hunk (merged upstream 2026-07-01, a few hours AFTER v8.0.0 was cut, so it ships in no release ≤ 8.0.0):
- **Why:** an update-style `merge_insert` over a merge-written fragment legally reuses the updated rows' stable row ids (row-id-lineage spec: updates preserve `_rowid`) while the superseded fragment keeps its full sequence + a deletion vector. A later delete leaves the overlapping id range sparsely tiled, and unpatched `RowIdIndex::new` asserted dense tiling — every filtered read that builds the id→address map then fails ("Wrong range" debug assert; "all columns in a record batch must have the same length" or a silently-wrong batch in release). Upstream bug lance#7444; tracked as `iss-merge-rowid-overlap-corrupts-filtered-reads` / `blk-lance-7444` on the dev graph. The fix is read-side only: the on-disk overlap is spec-legal, so already-written graphs become readable as-is — no data repair.
- **Pinned by** `lance_surface_guards.rs::filtered_scan_tolerates_merge_update_row_id_overlap` (a faithful transcription of lance#7444's minimal repro — merge-seed → merge-update → delete → filter + `with_row_id`; the merge-on-merge seed and the filtered-with-row-id read are both load-bearing) and the engine-level `writes.rs::filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent` (+ its green append-only control).
- **Removal condition:** drop `vendor/lance-table` + the `[patch.crates-io]` entry at the first Lance bump whose `lance-table` ships lance#7480 (9.0.0, or a backported 8.0.1). The surface guard keeps the removal honest in both directions. Verify-the-delta instructions live in `vendor/lance-table/README.omnigraph.md`.
- **Related, found during the same investigation, NOT consumed by this pin:** Lance v8.0.0 (released 2026-07-01) fixes merge_insert's legacy-Merger silent match-dropping under a scalar-indexed join key with a partial-schema / all-null-leading-column source (PR #7251) — the path any iss-986 field-level-merge implementation would use, since omnigraph BTREE-indexes every merge join key. omnigraph's *current* full-schema batches dodge #7251 by construction (the compiler puts `id` / `src`+`dst` at the exact leading positions the buggy check inspects — catalog/mod.rs:220,275). Gate iss-986 on the 7→8 bump.
### Last alignment audit: 2026-06-15 (Lance 7.0.0 upstream; omnigraph pinned at 7.0.0)
Migration from Lance 6.0.1 → 7.0.0 landed in this cycle. **Arrow stayed 58, DataFusion stayed 53** (no change) — the only transitive bump is `object_store` 0.12.5 → 0.13.2. 141 upstream commits reviewed (6.0.1 → 7.0.0); no fixes lost (the 6.0.x release-branch backports are all forward-ported into 7.0.0). Behavior-affecting findings:

View file

@ -21,7 +21,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
| `end_to_end.rs` | Full init → load → query/mutate flow |
| `branching.rs` | Branch create / list / delete, lazy fork |
| `merge_truth_table.rs` | Merge-pair truth table (MR-786): all 9×9 `(left_op, right_op)` cells from `{noop, addNode, removeNode, addEdge, removeEdge, setProperty, dropProperty, addLabel, removeLabel}`. Adding a new op to `OpVariant` forces a compile error in `build_case` until the new row + column are dispositioned. 36 executable cells run through real `branch_merge` with a structured oracle (`MergeOutcome` / `MergeConflictKind` + graph-state assert); 45 cells involving `dropProperty`/`addLabel`/`removeLabel` are recorded as `Unsupported` until the mutation grammar grows. |
| `writes.rs` | Direct-publish writes: cancellation, non-strict insert/merge rebase under the per-table queue, strict stale-write conflicts, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery) |
| `writes.rs` | Direct-publish writes: cancellation, non-strict insert/merge rebase under the per-table queue, strict stale-write conflicts, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery); the lance#7444 row-id-overlap regression (`filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent` — merge-load → same-key merge-load → delete → keyed point lookup, green only under the vendored lance-table patch — plus its append-only control) |
| `staged_writes.rs` | TableStore staged-write primitives (`stage_append`, `stage_merge_insert`, `commit_staged`, `scan_with_staged`, `count_rows_with_staged`) — primitive-level only; engine code uses the in-memory `MutationStaging` accumulator instead |
| `forbidden_apis.rs` | Defense-in-depth source-walk guard: engine code (`exec/`, `db/omnigraph/`, `loader/`, `changes/`) must not reach around the sealed storage trait to Lance inline-commit APIs, nor open datasets directly (`Dataset::open` / `DatasetBuilder::from_uri`/`from_namespace`) — reads route through `Snapshot::open` and the held-handle cache; `// forbidden-api-allow: <reason>` sentinel exempts reviewed lines |
| `lance_surface_guards.rs` | Pins the Lance API surfaces omnigraph depends on (named runtime + compile-only guards; see [lance.md](lance.md)) — the first smoke check on any Lance version bump; e.g. `compact_files_still_fails_on_blob_columns` turns red when the upstream blob-compaction fix lands |