mirror of
https://github.com/ModernRelay/omnigraph.git
synced 2026-07-12 03:12:11 +02:00
b5c0c6238b
iss-merge-rowid-overlap-corrupts-filtered-reads / lance#7444: an
update-style merge_insert over a merge-written fragment legally reuses the
updated rows' stable row ids (row-id-lineage spec: updates preserve
_rowid) while the superseded fragment keeps its full sequence plus a
deletion vector. A later delete leaves the overlapping id range sparsely
tiled, and lance-table 7.0.0's RowIdIndex::new asserted dense tiling —
failing every filtered read that builds the id→address map ("Wrong range"
debug assert; "all columns in a record batch must have the same length"
or a silently-wrong batch in release).
The upstream fix (lance#7480, merged 2026-07-01) landed hours AFTER
v8.0.0 was cut, so no release ≤ 8.0.0 carries it. Consume it now as a
vendored pin: vendor/lance-table is the pristine published 7.0.0 source
plus ONLY the #7480 rowids/index.rs hunk (drop the false tiling assert;
hard-error on the true invariant — one live id claimed by two fragments)
and upstream's regression unit test, wired via [patch.crates-io]. The fix
is read-side only, so already-written graphs become readable as-is — no
data repair.
Removal condition (see vendor/lance-table/README.omnigraph.md): drop the
vendor dir + patch entry at the first Lance bump whose lance-table ships
lance#7480 (9.0.0, or a backported 8.0.1). The surface guard
filtered_scan_tolerates_merge_update_row_id_overlap keeps that honest in
both directions.
Turns the previous commit's red tests green. Full workspace gate passes
(cargo test --workspace --locked --no-fail-fast, 68 suites).
47 lines
1.2 KiB
Rust
47 lines
1.2 KiB
Rust
// SPDX-License-Identifier: Apache-2.0
|
|
// SPDX-FileCopyrightText: Copyright The Lance Authors
|
|
|
|
pub mod stream;
|
|
|
|
pub trait LanceIteratorExtension {
|
|
fn exact_size(self, size: usize) -> ExactSize<Self>
|
|
where
|
|
Self: Sized;
|
|
}
|
|
|
|
impl<I: Iterator> LanceIteratorExtension for I {
|
|
fn exact_size(self, size: usize) -> ExactSize<Self>
|
|
where
|
|
Self: Sized,
|
|
{
|
|
ExactSize { inner: self, size }
|
|
}
|
|
}
|
|
|
|
/// A iterator that is tagged with a known size. This is useful when we are
|
|
/// able to pre-compute the size of the iterator but the iterator implementation
|
|
/// isn't able to itself. A common example is when using `flatten()`.
|
|
///
|
|
/// This is inspired by discussion in <https://github.com/rust-lang/rust/issues/68995>
|
|
pub struct ExactSize<I> {
|
|
inner: I,
|
|
size: usize,
|
|
}
|
|
|
|
impl<I: Iterator> Iterator for ExactSize<I> {
|
|
type Item = I::Item;
|
|
|
|
fn next(&mut self) -> Option<Self::Item> {
|
|
match self.inner.next() {
|
|
None => None,
|
|
Some(x) => {
|
|
self.size -= 1;
|
|
Some(x)
|
|
}
|
|
}
|
|
}
|
|
|
|
fn size_hint(&self) -> (usize, Option<usize>) {
|
|
(self.size, Some(self.size))
|
|
}
|
|
}
|