- filtered_scan_tolerates_merge_update_row_id_overlap passes on stock
lance-table (vendor-pin retirement validated by its own guard).
- literal-coercion guard re-pinned: v8's coercion fixes (lance#6935 et
al.) coerce width-mismatched literals to the column type BEFORE
pushdown, so the widened case now USES the BTREE (was: index-defeating
column cast). literal_to_typed_expr stays load-bearing.
- namespace decoupling guard: table_version_storage_enabled removed
upstream (#7222); thesis unchanged and still pinned.
- lazy-fork namespace test: Lance 9 loudly rejects opening a main-owned
manifest as another branch — the substrate now enforces the invariant
entry-owner resolution protects; test pins both the error and the
owner-branch open.
The 9.x betas are git-tags only (crates.io carries <= 8.0.0), so all lance
crates become git dependencies rev-pinned to the v9.0.0-beta.15 tag commit
(f24e42c1). Carries lance#7480 (rowid overlap, closes#7444 — the
vendor/lance-table pin + [patch.crates-io] entry are retired per their
documented removal condition) and lance#7320 (BTREE segment merge, closes
#7230). Sole compile break in the workspace: RowAddrTreeMap moved to the
new lance-select crate.
Closes two coverage gaps from review of my own test surface: (1) the
lowering boundary — undirected lowers to Expand{direction: Both} both
top-level and inside not{} (the discarded-context-clone regression is
now pinned at the layer it lives at, not only end-to-end); (2) the
review-fix path — undirected both_modes equivalence with a
mutation-appended unindexed fragment degrading BTREE coverage, so
results stay correct and mode-equivalent whichever path the pessimistic
coverage pricing picks.
Devin: the cost model's index-coverage probe checked only the primary
endpoint column, so an undirected traversal with a degraded dst index
was priced as fully indexed; coverage is now the pessimistic combination
over every column endpoint_probes will scan. Greptile: the bulk
anti-join grouped Both with In for src_type_name but with Out for the
adjacency — safe only under T22's from==to guarantee; Both now groups
with Out in both arms so a future T22 relaxation cannot split them
silently.
CSR arm: an undirected hop chains csr(edge).neighbors with
csc(edge).neighbors under the existing visited/seen_dst gates, so
both-direction pairs and self-loops dedup for free (set semantics),
including per-hop in variable-length BFS. Indexed arm: endpoint_probes
returns both (src,dst) and (dst,src) orientations for Both; the per-hop
scan runs once per orientation into one neighbor_map, deduped by the
existing per-source seen_dst sets. Bulk anti-join: "no edge in EITHER
direction" (csr || csc has_neighbors).
Also fixes a lowering gap the anti-join test caught red: negation
inners are typechecked into a discarded context clone, so the
ResolvedTraversal direction lookup silently fell back to Out inside
not{} — undirectedness now travels on the AST node itself (the syntax
is the source of truth), immune to context-propagation gaps.
Tests: traversal.rs (out ∪ in vs the directional miss, both-ways dedup,
var-length from an incoming-only node, undirected anti-join vs
directional), traversal_indexed.rs both_modes equivalence, and the
proptest arm-equivalence property widened with <knows>{1,3}.
iss-gq-undirected-traversal, the Expand-internal design (no plan-level
Union; unblocked from iss-744/579): the grammar gains an undirected_edge
alternative in the traversal rule (angle brackets are collision-free —
comparisons live in the structurally separate filter production), the
AST Traversal carries `undirected`, Direction gains `Both`, and
typecheck resolves undirected patterns to Both after the new T22 rule:
undirected requires a same-endpoint-type edge (an asymmetric edge is
well-typed in at most one orientation, so the form is rejected with
guidance to use the directional pattern). Lowering passes Both through;
the reverse-expand orientation flip is a no-op for a symmetric
traversal. Bounds ({min,max}) and not{} compose unchanged.
Direction has no serde derives and IR never crosses a wire — no
compatibility surface. Parser/typecheck tests cover the bare, bounded,
inside-not forms, Both resolution on Knows (Person->Person), and the
T22 rejection on WorksAt (Person->Company).
Both review bots caught it: the new fns were inserted between run_once's
cfg(unix) attribute and the fn itself, leaving append_result, git_sha,
and run_once ungated (non-unix name-resolution break, again). All three
gated; results_path keeps the displaced attribute.
The harness's JSON lines were vanishing into terminal scrollback — the
numbers that justify decisions (the 6.9GB merge crash, the prefilter
recall pair) had no system of record, and the v7 baselines are needed
for before/after comparison across the Lance 9 bump. Every run now
appends its record, stamped with a unix ts and the current git sha, to
--out <path> / OMNIGRAPH_BENCH_RESULTS / benches/results.jsonl
(gitignored — host-specific, self-describing via host+params).
Greptile follow-up on the previous cfg(unix) fix: ungated scenario fns
still referenced the unix-only helpers module, failing name resolution
on non-unix hosts (cfg_attr(allow) suppresses warnings, not resolution).
Every item is now cfg(unix)-gated; the non-unix build compiles to the
stub main alone.
Greptile follow-up: the scenario ignored the flag and ran the full query
loop, so a baseline/delta pair measured ~0 instead of the queries'
memory cost. Baseline now exits after seeding + optimize, matching
merge-all-changed's shape.
Greptile review follow-ups: wait4 now retries on EINTR (a delivered
signal previously panicked the harness mid-run via the reaped-pid
assert); a failed setrlimit warns on inherited stderr instead of
silently running the child uncapped while the parent JSON records the
requested cap; and the Unix-only syscalls are cfg(unix)-gated with an
inert Windows stub main so cargo bench compiles everywhere.
The dedicated cost/perf instrument testing.md's write_cost_s3 note has been
promising: one cold, stateful macro-run per scenario in a fresh subprocess
(self-respawn via current_exe), reaped with libc::wait4 so ru_maxrss gives
kernel-exact peak RSS with no sampling; results are JSON lines and there are
deliberately no assertions — a decision instrument, never a CI gate.
Criterion is deliberately not used: statistics over warm in-process
iterations is the wrong model for multi-second stateful scenarios, it
measures no memory, and an OOM under --memory-cap-mb (setrlimit; enforced on
Linux, best-effort on macOS) is a data point that needs crash isolation.
Two scenarios ship with the skeleton:
- merge-all-changed: an embedding table whose branch changed EVERY row's
vector, merged three-way into a diverged main — the changed-delta
concat + hash-join cost of branch_merge. --baseline re-runs the identical
workload minus the merge so the peak-RSS delta isolates it (smoke,
20k rows x 256 dims: ~72 MB merge contribution on a 20.5 MB raw delta,
~3.5x).
- nearest-prefilter: selectivity-s filtered nearest() where matching rows sit
far from the query point — quantifies the post-filter ANN recall deficit
(smoke, 20k rows, s=0.05, k=10: 1000 matching rows exist, 0 returned);
becomes the prefilter latency comparison unchanged once the fix lands.
Greptile follow-up: the zero-norm guard computed the norm and then
normalize_vector recomputed it (a wasted pass at 1536+ dims), with the
inner zero guard unreachable by construction. Normalize in place with
the validated norm; normalize_vector stays for its other caller (the
mock provider).
The mock flag's precedence over OMNIGRAPH_EMBED_PROVIDER is deliberate
and stays (pinned by from_env_mock_flag_wins) — but the override was
silent, and mock vectors are indistinguishable from real ones (correct
dimension, unit norm, deterministic hash), so a leaked test env var
silently poisoned persisted embeds (CLI) and query-time nearest()
(server, for graphs without a bound cluster embedding profile). Emit a
tracing::warn naming the overridden provider; document the precedence
in the embeddings env-var table.
Closes iss-embeddings-mock-override-silent.
Replace the FixedSizeList arm's as_f64().unwrap_or(0.0) coercion with a
loud per-row OmniError::manifest in the loader's own style (property
name included, mutation path's "elements must be numeric" phrasing).
The red test turns green; the whole validators suite stays green.
Closes iss-loader-vector-element-coercion.
iss-loader-vector-element-coercion: the loader's FixedSizeList arm does
val.as_f64().unwrap_or(0.0) — a null element (what json! produces for a
non-finite float) or a string element silently zeroes the vector while
the mutation path rejects the same input loudly. No loader vector
validation had any test coverage; this also pins the existing dimension
check.
Fails as predicted against current code: the null-element load succeeds
(unwrap_err panics on Ok) instead of erroring.
Validate every component is_finite() (naming the offending index) and
reject zero-norm vectors before normalizing, via the existing
Err(String) channel — callers already map it through EmbedCallError
{retryable:false} into OmniError::manifest_internal, so no new error
plumbing. The red tests turn green; docs state the finite/non-zero
contract in embeddings.md.
Closes iss-embedding-nan-validation.
iss-embedding-nan-validation: validate_and_normalize_embedding checks
only dimension. The normalize guard (norm > f32::EPSILON) inverts on NaN
— normalization is skipped and the raw NaN vector returns Ok; an Inf
component normalizes to 0s + NaN; an all-zero vector (no direction,
undefined under cosine/ANN) passes. All were unpinned.
Fails as predicted against current code: unwrap_err() panics on the NaN
case — the poisoned vector is returned Ok.
Lance's scanner defaults to prefilter=false: a filter riding the same
scanner as nearest()/bm25() is applied AFTER the ANN/FTS top-k, so
`limit k` meant top-k of the whole table and a selective predicate
silently starved results (the deny-list's silent-partial-result shape;
measured by the nearest-prefilter bench scenario: 20k rows, s=0.05,
k=10 -> 1000 matching rows exist, 0 returned). Set prefilter(true)
whenever a structured filter is pushed to the scanner: one flag governs
both the vector and FTS sources, plain scans ignore it, and it
re-enables scalar-index acceleration for the predicate under nearest.
The red test turns green: filtered nearest now returns the top-k of
MATCHING rows. Docs state the filters-before-search contract explicitly
(docs/user/search/index.md).
Closes iss-nearest-postfilter-starves-results.
iss-nearest-postfilter-starves-results: a scalar match predicate combined
with nearest returns the post-filtered remainder of the GLOBAL ANN top-k,
not the top-k of matching rows. Fixture: 3 matching docs sit far from the
query vector while 3 non-matching docs occupy the global top-3 — the query
returns 0 rows despite limit 3 and 3 matches existing.
Fails as predicted against current code:
assertion left == right failed: left: 0, right: 3
Greptile review follow-up: a property renamed and widened in one
migration emits RenameProperty followed by ExtendEnum carrying the
post-rename name. That ordering/naming contract is now stated on the
ExtendEnum variant's doc and pinned by
plan_orders_rename_before_widening_and_names_the_new_property, so a
sequential step consumer (or a future apply-ordering change) can't
silently break it.
Same-PR docs for the ExtendEnum step: the supported-steps list in the
schema migration section (with the narrowing/rename/String-conversion
carve-outs and the order-insensitivity note), the OG-MF-106 lint-table
row narrowed to what it still covers, and the schema_apply.rs test-map
row extended with the two new tests.
The widening is planner-verified (superset, same shape), so every
committed row is already valid under the wider set: the apply arm joins
the AddConstraint metadata-only family — accepted-catalog update, no
table work, no version bump — and the unified validator accepts the new
variants on all three write surfaces immediately.
Tests: enum_widening_apply_is_metadata_only_and_accepts_new_variant
(no table-version bump; new variant accepted, original still accepted,
out-of-set still rejected) and enum_narrowing_apply_is_refused
(OG-MF-106 surfaced, graph left writable).
Adding variants to an enum property previously planned as an unsupported
property-type change (OG-MF-106): enum values live inside PropType and
the planner compared whole types. Special-case the one safe shape — same
scalar/list/nullability and the desired value set a strict superset of
the accepted one — into a new ExtendEnum step carrying the added values.
Narrowing, variant renames, and enum<->String conversions still plan as
OG-MF-106; a pure reorder was already a no-op (the schema IR sorts +
dedups enum values).
Planner matrix pinned in in-source tests (widening, reorder-as-no-change,
narrowing, rename, widen+nullability-flip, enum<->String both ways); CLI
plan rendering gains the ExtendEnum arm.
Devin review follow-up on the opener unification: StagedTableNamespace's
open_head opens a DATA table (its table_uri is the per-table physical
path), so it belongs on the table_wrapper/data_open_count bucket, not
the manifest one. Test-only impact (the namespace module is #[cfg(test)]
since RFC-013 step 3a), but the classification should be right where the
cost vocabulary is defined.
Greptile review follow-up on the session-threading commit: the
Arc::new(Session::default()) boilerplate at every test TableStore
construction collapses into helpers::test_session() (and a local twin in
staged_writes.rs, which is primitive-level and deliberately does not
include the helpers module) — a future session-constructor change is a
single-point edit.
Thread the graph's one Lance Session (previously read-path-only via
ReadCaches) into TableStore, so open_dataset_head, the branch ops, and
open_at_entry all attach it through the unified opener — the read path's
handle cache and the write side now warm the same Lance metadata/index
caches. diff_snapshots takes the graph's &TableStore instead of
constructing its own session-less store from a root uri.
Measured effect (write_cost.rs): the local data-table SCAN term — the
merge-insert/RI scan that re-read O(depth) immutable fragment metadata
per write — collapses from growing-with-depth to flat (depth 10: 1 read,
depth 100: 1 read). The opener/scan-split gate is re-pinned to the new
behavior and renamed
(data_table_reads_split_into_flat_opener_and_scan_flat_with_session):
a red there now means a write-side open dropped the session. The
Snapshot-without-caches fallback stays session-less by design (a
detached snapshot has no graph to share a session with); the S3
acceptance of the opener term remains owed to write_cost_s3 per the
RFC-013 handoff.
testing.md's backend-split note updated in the same change (the "local
scan grows with depth" claim is now stale).
The method was a delegating twin of open_dataset_head that discarded its
table_key argument ("retained for signature stability") — a dead
parameter every caller still had to supply, and a second name for the
same open policy on the sealed TableStorage surface. Remove it from the
trait, the TableStore impl, and all call sites, which also lets the two
needs_index_work_* helpers drop their now-unused table_key parameters.
Purely mechanical; no behavior change.
Unify the two instrumented openers (open_dataset_tracked latest-only,
open_table_dataset pinned-only) into a single
instrumentation::open_dataset whose version resolution (Latest | At(v))
is an explicit typed parameter, and route every production Dataset open
in the engine through it — including the previously-raw opens in the
branch ops (delete/list/force_delete_branch), manifest branch delete,
version-metadata resolution, recovery classification/restore, the
recovery-audit table, and schema-apply version GC (whose
forbidden-api-allow sentinel is now unnecessary and removed).
One chokepoint means three properties hold uniformly instead of
per-path: record_open feeds the cost probes on every open (the raw
sites were invisible to the gates), the per-query IO wrapper is
attached with the right class (manifest vs table) everywhere, and the
shared per-graph Session parameter exists on every path — threading it
into the write-side owners lands separately.
Behavior-preserving: every call site keeps its existing version
resolution, session argument, and wrapper class; in-source tests keep
raw opens (out of scope for the chokepoint by design).
Greptile review on #324: the lineage_projection row said the legacy
datasets 'hold ZERO commit rows' but the test asserts the directories are
never created — a strictly stronger invariant; say that. And
write_cost_s3.rs's module doc still claimed it runs in the rustfs CI job,
contradicting the (correct) new table row — align it with reality (run on
demand, cost gate).
The engine integration-test table drifted: failpoint_names_guard.rs,
lineage_projection.rs, merge_fast_forward.rs, scalar_indexes.rs, and
write_cost_s3.rs existed with no row (write_cost_s3 had prose mentions
only), and the file count said 28 where there are 36. testing.md is
@-imported into every agent session as the test-coverage map, so missing
rows defeat its "check what already covers it" contract. Row texts are
derived from each file's own doc comments; the write_cost_s3 row records
its run-on-demand (not every-merge CI) status per the existing
Cost-budget-tests note.
Greptile review on #323: the timeout path (server alive but never healthy)
panicked without the captured stderr — the harder failure mode was less
diagnosable than the port-race one. Read the log after kill+wait so the
final flush is captured. Hoist the io imports to the file's use block, and
reference the #327 design-fix follow-up (server-side :0 binding) at the
retry stop-gap.
spawn_server_process picked a port by binding :0 and DROPPING the
listener before the spawned omnigraph-server rebound it — a TOCTOU
window in which any concurrently-spawning test steals the port. The
loser exits "address in use", but stderr was nulled, so the test
panicked with a bare "server exited before becoming healthy: exit
status: 1" on whichever parity/system test lost the race that run
(~1-in-2 parallel runs locally; 12/12 pass serially).
Fix, two halves:
- Retry the spawn on a fresh port (up to 5 attempts) when the child
exits before passing the health check. A lost race becomes invisible;
a deterministic startup failure still panics loudly after the retries.
- Capture stderr to a temp file (not a pipe, which a chatty healthy
server would fill and block on; not null) and include it in the panic
and per-retry messages, so genuine startup failures carry the server's
own error text.
StdCommand isn't Clone and a reused command would trip clap's
duplicate --bind rejection, so retries respawn via clone_command
(program/args/envs/cwd).
Verified: parity_matrix 3x parallel clean (previously flaked ~every
other run), full omnigraph-cli suite green.
The generic export/import recipe never said what v0.8.0 actually changes.
Add the release-specific section: the internal-schema v4 on-disk deltas
(lineage rows in __manifest, the two retired commit-graph datasets, the
both-direction version gate) and the unified stricter validation with a
branch-based pre-flight recipe, plus the three ways to read the format
version.
Same-PR documentation for the vendored pin: a dated patch-pin stanza in
lance.md's alignment history (mechanism, pinning tests, removal
condition, and the adjacent v8-only #7251 finding that gates iss-986 on
the 7→8 bump), a Known Gaps entry in invariants.md so the pin cannot be
forgotten, and the writes.rs row in testing.md now names the row-id
overlap regression pair.
iss-merge-rowid-overlap-corrupts-filtered-reads / lance#7444: an
update-style merge_insert over a merge-written fragment legally reuses the
updated rows' stable row ids (row-id-lineage spec: updates preserve
_rowid) while the superseded fragment keeps its full sequence plus a
deletion vector. A later delete leaves the overlapping id range sparsely
tiled, and lance-table 7.0.0's RowIdIndex::new asserted dense tiling —
failing every filtered read that builds the id→address map ("Wrong range"
debug assert; "all columns in a record batch must have the same length"
or a silently-wrong batch in release).
The upstream fix (lance#7480, merged 2026-07-01) landed hours AFTER
v8.0.0 was cut, so no release ≤ 8.0.0 carries it. Consume it now as a
vendored pin: vendor/lance-table is the pristine published 7.0.0 source
plus ONLY the #7480 rowids/index.rs hunk (drop the false tiling assert;
hard-error on the true invariant — one live id claimed by two fragments)
and upstream's regression unit test, wired via [patch.crates-io]. The fix
is read-side only, so already-written graphs become readable as-is — no
data repair.
Removal condition (see vendor/lance-table/README.omnigraph.md): drop the
vendor dir + patch entry at the first Lance bump whose lance-table ships
lance#7480 (9.0.0, or a backported 8.0.1). The surface guard
filtered_scan_tolerates_merge_update_row_id_overlap keeps that honest in
both directions.
Turns the previous commit's red tests green. Full workspace gate passes
(cargo test --workspace --locked --no-fail-fast, 68 suites).
iss-merge-rowid-overlap-corrupts-filtered-reads / lance#7444: an
update-style merge over a merge-written fragment reuses the updated rows'
stable row ids (spec-legal overlap); a later delete leaves the overlapping
range sparsely tiled and unpatched lance-table 7.0.0's RowIdIndex::new
fails every filtered read that builds the id→address map.
Two boundaries:
- writes.rs::filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent
(engine walk: merge-load → same-key merge-load → delete → keyed point
lookups), plus the green isolation control
filtered_read_after_append_and_delete_is_consistent.
- lance_surface_guards.rs::filtered_scan_tolerates_merge_update_row_id_overlap
(faithful transcription of lance#7444's minimal repro at the raw Lance
API; pins the surface so a future bump that regresses lance#7480 turns
red).
Both fail on unpatched Lance 7.0.0 with the predicted symptom
(rowids/index.rs:50 "Wrong range" via the take path); green arrives with
the vendored lance-table patch in the next commit.
* ci: shard the RustFS S3 integration job across parallel runners
The RustFS S3 Integration job chronically hit its 75-minute timeout (e.g. on
the v0.8.0 release run) and got cancelled. Root cause is compile time, not test
time: the S3 tests each run in seconds (the write_cost_s3 step took 0.2m once the
engine was built), but the job ran six serial `cargo test` steps across four
crates plus a `--features failpoints` rebuild, and on a cold cache (any Cargo.lock
change, e.g. a release version bump) every suite must recompile the omnigraph-engine
+ Lance/DataFusion tree, summing to ~75m.
Split the suites into a `strategy.matrix.shard` (engine / server / cluster / cli /
failpoints), one suite per shard on its own runner with a per-shard rust-cache key
and `fail-fast: false`. Wall-clock becomes the slowest single shard (~40m cold,
~25m warm) instead of the sum. Bundling suites would not help — each crate adds its
own unique-dep compile on top of the shared substrate — so each gets its own shard;
the failpoints shard is isolated because its distinct feature set recompiles the
engine tree. Timeout lowered 75 -> 50 (headroom over the worst cold shard).
The job is renamed `RustFS S3 Integration (<shard>)`; it is not a required check,
so branch protection is unaffected. Docs updated in docs/dev/ci.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: drop the write_cost_s3 cost gate from the correctness job
The RustFS integration job is a correctness gate. write_cost_s3 is a
deterministic IO-count COST gate (RFC-013 step-3a data-table opener, flat
across commit depth) — a performance contract, not a correctness test.
Cost/perf contracts belong on a dedicated harness with a stable runner and
their own cadence, not on the every-merge correctness path. Remove the step
from the engine shard; a comment + testing.md record how to run it on demand
and note it's pending a dedicated cost harness. The local write_cost.rs
opener/scan-split guard still runs every-PR, so the split stays covered; only
the S3 acceptance of the opener term moves off the correctness path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(release): trim v0.8.0 notes, drop internal RFC references
Shorten the v0.8.0 release notes to a scannable highlights + upgrade format and
remove internal RFC-track references (RFC-013 Phase 7, step 3a, etc.), which are
not public-inspectable and don't belong in OSS-facing release notes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(release): widen rebuild note to any pre-v0.8.0 graph
Greptile P1: the version gate blocks any pre-v4 graph (v0.4.x-v0.7.x), not just
v0.7.x. The trim over-narrowed it and left older-release users without a signal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(release): cover #314 stricter validation + #316 linux-arm64 in v0.8.0 notes
Two changes landed after the version-bump commit (#313) that wrote the v0.8.0
release notes, so they were undocumented:
- #314 unified constraint validation across the loader, mutation, and merge
surfaces. Its behavior changes are all stricter (enum enforced on merge,
cross-version @unique rejection, precise within-batch vs across-batch dup-key
semantics, per-table overwrite validation) and are user-visible, so they get a
dedicated section.
- #316 added linux-arm64 (aarch64) as a first-class prebuilt target + Homebrew
bottle; note the new platform.
Also mention the stricter validation in the release intro. Docs-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(release): scope cross-version uniqueness to the write's visible state
Greptile P2: the prior wording implied a live-head @unique guarantee, but #314's
check is snapshot-scoped (probes the write's pinned base view via the
@key/@unique BTREEs), so a concurrent writer committing the same value after the
base was opened is not caught — matching the 'full cross-version uniqueness is
still a gap' note in docs/dev/invariants.md. Qualify with 'visible to that write'.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The test seeded the full fixture (test.jsonl, which carries Knows/WorksAt edges)
then fault-injected a per-table Overwrite of node:Person down to a single row.
Overwrite RI validation now correctly rejects that: dropping the seeded Persons
strands the retained edges, so the load failed with "src 'Alice' not found in
Person" during validation — before it reached the post-finalize failpoint the
test drives, so the recovery roll-forward was never exercised.
Seed Persons only (no edges) so the overwrite is a clean single-table
roll-forward, which is what the test asserts (RolledForward{node:Person}). The
overwrite still removes the seeded Persons, so F1's overwrite-removed-ids path is
still exercised — just without an orphan.
Feature-gated: this binary runs under `--features failpoints`, a separate CI
step from the workspace test run, so it's invisible to `cargo test -p
omnigraph-engine` and only the post-merge main CI exercises it.
The seeded knowledge graph (test.jsonl) carries Knows/WorksAt edges over its
Persons, so the policy test's per-table `--mode overwrite` of a lone Person
stranded those retained edges against the new node image. Overwrite RI
validation now correctly rejects that as an OrphanEdge, so the allowed-actor
load failed with "src 'Alice' not found in Person".
Replace the edge tables in the same overwrite (Knows LoadPolicy->LoadPolicy,
WorksAt LoadPolicy->Acme — Acme is a retained Company) so the load commits
cleanly. The test's subject — engine-layer policy enforcement on a destructive
overwrite — is unchanged; the loaded data was always incidental.
Surfaced only on main: cargo test --workspace fail-fasts at the first failing
binary, and the cli system suite runs only under the full workspace gate, not
-p omnigraph-engine.
* feat(engine): unify constraint validation across all write surfaces
Constraint enforcement (value/range/check, enum, uniqueness, edge
referential integrity, cardinality) was implemented three times — once
each in the bulk loader, the mutation executor, and the branch-merge
path — and had drifted: merge validated @range/@check but not enum, and
neither the mutation nor the load path enforced cross-version uniqueness
against already-committed rows.
Introduce one catalog-derived evaluator (`crate::validate`) that all
three surfaces route through. It is delta-scoped (checks only the change
set, not the whole graph) and index-backed (probes committed state
through the @key/@unique/src/dst BTREEs instead of full-scanning every
catalog table), reusing the existing leaf checks
(validate_value_constraints, validate_enum_constraints,
composite_unique_key) so the surfaces cannot drift again. A one-row-delta
merge now opens ~3 data tables instead of ~6+, and validation cost is
flat in graph size rather than O(V+E).
Behavior changes (all stricter, none relaxed):
- Enum constraints are now enforced on the merge path (was a gap).
- A write or load whose @unique value collides with an already-committed
different row is now rejected (cross-version uniqueness); re-upserting
an existing @key still upserts.
- Uniqueness distinguishes a duplicate key WITHIN one input batch (two
distinct records -> rejected, e.g. a bulk load listing a @key twice)
from the SAME id reappearing ACROSS batches (ordered supersession of
one logical row -> coalesced, e.g. a mutation insert-then-update).
- Overwrite loads validate per-table: a touched table's committed view is
its replacement image (empty), but a table absent from the batch keeps
its committed rows, so an edges-only overwrite still resolves
referential integrity against retained nodes.
Remove the per-surface validation orchestration the evaluator supersedes,
and the now-orphaned version-pinned dataset opener from the sealed
storage trait (reads route through the snapshot path). Docs (invariants,
testing) updated; full engine suite green.
* test(engine): pin orphan-edge validation on adopt-by-pointer merge
Regression for a gap in the unified merge validation: when a table is
adopted by pointer switch (AdoptSourceState) — source on main, target on a
branch — build_merge_changeset skips it, so referential integrity is never
checked for it. Merging main into a branch that deleted a node while main
added an edge to that node silently publishes the orphan edge.
This test merges main -> feature where feature deleted Bob and main added
Knows Alice->Bob, and asserts an OrphanEdge conflict. Red against HEAD
(merge returns Merged); turns green with the AdoptSourceState validation fix.
* fix(engine): validate adopt-by-pointer merge tables (AdoptSourceState)
The unified merge validator skipped any table classified AdoptSourceState
(a pointer switch / fork), so referential integrity, uniqueness, and
cardinality were never checked for it. Merging main into a branch that
deleted a node while main added an edge to that node silently published the
orphan edge — the prior full-scan validation caught it.
Root cause: classify_adopt keyed AdoptSourceState on the publish mechanism
("does it advance Lance HEAD") and returned before computing any delta, and
build_merge_changeset then skipped the table. Fix decouples the validation
input from the publish mechanism: classify_adopt now always computes the
source-vs-target delta (base == target on this path, so it is the right
validation delta) and carries it as AdoptSourceState { validation_delta };
build_merge_changeset validates it exactly like AdoptWithDelta. The publish
stays a pointer/fork (delta ignored) and remains excluded from recovery
pins, so publish/recovery semantics are unchanged — only validation is
restored. Closes the class: no publish optimization can bypass validation.
Turns the orphan-edge regression test green.
* test(engine): pin typed committed-uniqueness probe on non-String columns
The cross-version @unique check pushes a committed-state filter built from
the stringified key. On a non-String @unique column (e.g. Date) this compares
a Date32 column to a Utf8 literal — and the stringified key is the raw day
count, so the probe raises "Cannot cast string '20633' to Date32" for ANY
second write to the table (colliding or not).
Two regressions: a colliding Date value must surface a proper "@unique
violation" (not a coercion error), and a non-colliding write must succeed.
Both red against HEAD; green with the typed-literal probe fix.
* fix(engine): build committed uniqueness probe from typed column values
The cross-version @unique check pushed a Lance filter built with a
stringified key (lit(String)) against the real, typed column. On a
non-String @unique column this compared a Date32/numeric/bool column to a
Utf8 literal: a coercion error on Date/Bool (failing every write to the
table) or a silent miss on Float. For Date the stringified key was even the
raw day count, so the literal could never parse.
unique_holders now takes typed ScalarValues, built at the call site via
ScalarValue::try_from_array(group_column, row), so the pushed-down predicate
compares like-typed for any scalar @unique. The in-memory intra-delta dedup
keeps the stringified key (a type-agnostic equality grouping, unaffected).
Turns the Date @unique cross-version regression tests green.
* test(engine): pin id-keyed cardinality on merge-load edge moves/dups
Two cardinality drifts between validation and what commit persists:
- Move (B): a Merge-load that moves an edge to a new src only recounts the
new src, so vacating a src and dropping it below @card min is missed —
moving Alice's only WorksAt to Bob silently succeeds under @card(1..).
- Dup (A): a Merge-load batch listing one edge id under two srcs counts it
under both, but commit dedupes by id (last-wins). Alice gets a phantom
second edge and a spurious "has 2 edges (max 1)" violation under @card(0..1).
Both red against HEAD; green with the id-keyed last-wins cardinality model.
* fix(engine): key merge/load cardinality by edge id, last-wins
@card validation diverged from what commit persists in two ways: (1) it only
recounted the new src of a delta edge, so a Merge-load that moves an edge to a
new src never rechecked the vacated src and missed a drop below @card min; (2)
it counted raw delta rows, so the same edge id under two srcs in one batch was
counted under both, while commit dedupes by id (last-wins) — a phantom edge
and a spurious max violation.
evaluate_cardinality now coalesces the delta by edge id (last-wins, matching
dedupe_merge_batches_by_id) and builds the affected-src set from both the new
src of each delta edge AND the old committed src of each changed/deleted edge
id; a committed edge is dropped from its src when the delta deletes or
re-places it. The validated edge set per src now equals the committed image.
Turns the edge-move and duplicate-id cardinality regression tests green.
* docs(rfcs): add RFC 0001 — branch merge by fragment adoption
Proposed design for the by-design fix to merge cost/OOM: adopt the source
branch's Lance fragments by reference (base_paths) instead of re-materializing
rows, with a re-home reconciler + branch-delete reference guard closing the
dangling-reference lifecycle, and a reachability-complete cleanup sweep. Grounded
in the public Lance 7.0.0 multi-base APIs and the prior art (Delta shallow/deep
clone, Iceberg/lakeFS reachability GC). Status: Proposed.
* test(engine): pin @card validation on direct edge delete
Deletes stage as predicates, not constructive batches, so a delete-only
mutation produces an empty change-set and validate_changeset no-ops — a
`delete WorksAt where from = X` that removes a source's only edge commits
below @card(1..), while the merge path (which carries deleted_ids) rejects it.
Red against HEAD (the delete commits); green once the delete path resolves
its predicates into the validation change-set.
* fix(engine): validate edge cardinality on delete via resolved predicates
A delete-only mutation produced an empty change-set (deletes stage as
predicates, not constructive batches), so validate_changeset no-op'd and a
`delete Edge` that dropped a source below @card min committed silently — while
the merge path, which carries deleted_ids, rejects it.
validate_staged_mutation now resolves each staged delete predicate against the
live committed table (CommittedState::deleted_ids_matching, a SQL-filter scan
projecting id) and folds the matched ids into the change-set's deleted_ids for
that table. The existing evaluator then recounts the srcs a delete empties
(@card min) and sees removed rows for RI/node-delete — the same faithful
change-set the merge path already builds, so validation matches what commits.
Covers direct edge deletes, node deletes, and node-delete edge cascades
uniformly (all are staged predicates).
Turns the direct-edge-delete @card regression test green.
* refactor(engine): capture deleted ids at delete time, drop validation re-scan
The delete-cardinality fix resolved staged delete predicates a second time at
validation. Instead, capture the removed ids during the delete op's own scan:
execute_delete_edge and the node-delete edge cascade now scan id (not
count_rows), record the ids via MutationStaging::record_deleted_ids, and
to_changeset() folds them into the change-set's deleted_ids. validate_staged_
mutation reverts to plain to_changeset(); CommittedState::deleted_ids_matching
and scan_filtered_sql are removed.
Behavior-preserving (the @card-on-delete test stays green) and strictly fewer
scans — one scan at delete time replaces count-here + resolve-at-validation.
Node deletes already scanned their ids; this reuses that via a shared
ids_from_batches helper. Full engine suite green; workspace builds clean.
* test(engine): pin overwrite-removal RI + coalesced-unique final image
Two reviewer findings, both red against HEAD:
- F1 (High): overwriting a node table removes nodes without expressing them as
deleted_ids, so a retained edge in a non-overwritten table that references a
removed node is published as an orphan (edge-RI path-b never runs).
overwrite_node_removal_rejects_retained_orphan_edge.
- F2 (Medium): evaluate_unique accumulates superseded keys across batches, so a
mutation that frees a @unique value (Alice.email temp -> final) and reuses it
(insert Carol.email = temp) false-rejects a valid final image.
chained_unique_update_then_reuse_freed_value_is_not_a_violation.
* fix(engine): validate overwrite removals (orphan edges, emptied srcs)
An Overwrite load replaces each touched table, but to_changeset() only recorded
the new batch, never the committed rows the overwrite removes. So overwriting
node:Person to drop Bob while a retained edge:Knows(Alice->Bob) referenced him
published an orphan edge unchecked — edge-RI path-b is gated on the node's
deleted_ids, which were empty.
The loader now computes per overwritten table the removed ids (committed ids in
the pinned base minus the replacement batch's ids, via validate::
overwrite_removed_ids) and folds them into the change-set's deleted_ids. The
evaluator then runs RI path-b and cardinality against them — the same faithful
change-set the merge path builds. Overwrite is per-table, so a table absent from
the batch is untouched; a removed node referenced by a retained edge is now a
loud OrphanEdge.
Updates two tests that asserted the old silent-orphan behavior to
self-consistent overwrites (per-table Overwrite can't drop edge endpoints
without also overwriting the edge tables): end_to_end::overwrite_replaces_data
and writes::load_overwrite_with_bad_edge_reference_unblocks_next_load. The
orphan-rejection case itself is pinned by the new validators test.
* fix(engine): evaluate @unique against the coalesced final delta image
evaluate_unique iterated the raw delta batches and accumulated every key it saw
into one cross-batch map, so a coalesced write that frees then reuses a @unique
value within a query — update a row's email to 'temp', update the same row to
'final', insert a new row with 'temp' — false-rejected: 'temp' lingered in the
seen-set from the superseded first write though it no longer holds in the final
image that commits.
Restructure to validate the final coalesced image — the bytes that actually
publish:
- Pass 1 coalesces the delta by id (last-wins) into each id's final key, and
flags genuine within-ONE-batch duplicates (two distinct input records — the
bulk-load contract) before coalescing, so an unordered load batch with a real
dup still rejects.
- Pass 2 checks two distinct final ids holding the same key.
- Pass 3 does the committed cross-version lookup, excluding the delta's own ids.
Entries are sorted by id before the cross-row/committed passes so violation
order never depends on HashMap iteration. Coalescing first also drops the
redundant committed probes a superseded key used to issue.
Pinned by the chained-update red test; preserves intra-batch dup rejection
(consistency::loader_rejects_intra_batch_duplicate_keys) and cross-version
uniqueness (validators).
* style(engine): drop trailing blank line at staging.rs EOF
Left by a block-delete in an earlier refactor; flagged by git diff --check.
* docs(engine): refresh validate.rs module doc to current consumers
The module doc still said the merge path was the only consumer and the write
path a later, mechanical migration, and listed cardinality as a later
increment. Mutation and bulk load have since migrated onto the evaluator and
cardinality ships — correct both so the doc reflects that all three write
surfaces route through one evaluator.
* ci: add linux-arm64 (aarch64) prebuilt release target
Build an omnigraph-linux-arm64 archive in both the tagged-release and
edge-release matrices on the ubuntu-24.04-arm runner, and teach the
install script to map Linux/aarch64 to the new asset. Update the install
and CI docs to list the new platform.
Previously aarch64 Linux hit the install-script arch guard
("no prebuilt binary is available for Linux/aarch64") and could only
build from source; it is now a first-class prebuilt target.
* ci: emit a linux-arm64 bottle in the Homebrew formula
The formula generator only resolved macos-arm64 and linux-x86_64 and
emitted `on_linux { on_intel }`, so `brew install` on Linux/aarch64 had
no URL/sha and failed even though the release now ships an
omnigraph-linux-arm64 archive. Resolve that asset's digest and add an
`on_arm` block under `on_linux` so the documented Homebrew path matches
the new prebuilt target.
