omnigraph/docs/releases/v0.4.1.md

Omnigraph v0.4.1

Omnigraph v0.4.1 closes the multi-statement-mutation atomicity gap that v0.4.0 documented as a known limitation. Inserts and updates now route through an in-memory MutationStaging accumulator and commit via Lance's two-phase distributed-write API at end-of-query. A failed mid-query op no longer leaves Lance HEAD drifted on the touched table — the next mutation proceeds normally.

Highlights

• Staged-write rewire (MR-794): mutate_as and load (Append / Merge modes) accumulate insert/update batches into MutationStaging.pending per touched table. No Lance HEAD advance happens during op execution; one stage_* + commit_staged per table runs at end-of-query, then ManifestBatchPublisher::publish commits the manifest atomically. For op-execution failures (validation errors, missing endpoints, parse-time D₂ rejection), Lance HEAD on every staged table is untouched and the next mutation proceeds normally. A narrowed residual remains at the finalize→publisher boundary (multi-table commit_staged is not atomic with the manifest commit) — see docs/runs.md "Finalize → publisher residual" for details.
• D₂ parse-time rule: a single mutation query is either insert/update-only or delete-only. Mixed → rejected with a clear error directing the caller to split into two queries. Lance 4.0.0 has no public two-phase delete; deletes still inline-commit, and D₂ keeps that path safe.
• Read-your-writes via DataFusion MemTable: read sites in multi-statement mutations consume TableStore::scan_with_pending, which Lance-scans the committed snapshot at the captured expected_version and unions with a DataFusion MemTable over the pending batches. Replaces the previous "reopen at staged Lance version" pattern.
• Coordinator swap-restore eliminated from mutate_with_current_actor. Branch is threaded explicitly through the per-op execution path (execute_named_mutation, execute_insert, execute_update, execute_delete*, validate_edge_insert_endpoints, ensure_node_id_exists). The swap_coordinator_for_branch / restore_coordinator API and CoordinatorRestoreGuard are removed from mutation.rs. (merge.rs keeps its own swap pattern; that's a separate workflow tracked in MR-793.)
• docs/invariants.md §VI.25 flips from aspirational/open to upheld for inserts/updates. The within-query read-your-writes guarantee is now load-bearing for the publisher CAS contract.

Behavior changes

• A failed multi-statement mutation no longer surfaces ExpectedVersionMismatch on the next mutation against the same table. The next call proceeds normally — Lance HEAD on staged tables is unchanged.
• Mixed insert/update + delete in one query is rejected at parse time. Existing test queries that mixed both must be split.
• MutationStaging's shape changed: pending: HashMap<String, PendingTable>
  • inline_committed: HashMap<String, SubTableUpdate> replaces the previous latest: HashMap<String, StagedTable>. This is an internal type; no public API impact.

Residual / out of scope

• LoadMode::Overwrite keeps the legacy inline-commit path (truncate-then-append doesn't fit the staged shape). A mid-overwrite failure can still drift Lance HEAD on a partially-truncated table; the next overwrite replaces it. Operator-driven, rare.
• Delete-only multi-statement mutations still inline-commit per op. D₂ keeps inserts/updates from coexisting with deletes, so the inline path remains atomic per op but not per query for delete-only cascades. Closing this requires Lance to expose DeleteJob::execute_uncommitted; tracked in MR-793 / Lance-upstream.
• schema_apply, branch_merge_internal, ensure_indices still use Lance's inline-commit APIs. The two-phase pattern is in mutate_as and load only; hoisting it to a storage-trait invariant covering all writers is MR-793.

Tests added

• tests/runs.rs::partial_failure_leaves_target_queryable_and_unblocks_next_mutation (replaces the old partial_failure_observably_rolls_back_but_blocks_next_mutation_on_same_table)
• tests/runs.rs::mutation_rejects_mixed_insert_and_delete_at_parse_time
• tests/runs.rs::mixed_insert_and_update_on_same_person_coalesces_to_one_merge
• tests/runs.rs::multiple_appends_to_same_edge_coalesce_to_one_append
• tests/runs.rs::multi_statement_inserts_publish_exactly_once
• tests/runs.rs::load_with_bad_edge_reference_unblocks_next_load
• tests/runs.rs::load_with_cardinality_violation_unblocks_next_load

Files changed

• crates/omnigraph/src/exec/staging.rs (NEW) — MutationStaging, PendingTable, PendingMode, StagedTablePath, dedupe_merge_batches_by_id.
• crates/omnigraph/src/exec/mutation.rs — D₂ check; per-op rewires (execute_insert, execute_update, execute_delete*); branch threading; coordinator-swap removal; helper validate_edge_cardinality_with_pending; helper concat_match_batches_to_schema; apply_assignments updated to copy unassigned blob columns from full-schema scans.
• crates/omnigraph/src/loader/mod.rs — load_jsonl_reader split: staged path for Append/Merge, legacy inline-commit path for Overwrite. Helpers collect_node_ids_with_pending and validate_edge_cardinality_with_pending_loader.
• crates/omnigraph/src/table_store.rs — scan_with_pending, count_rows_with_pending (DataFusion MemTable-backed union with Lance scan).
• Cargo.toml (workspace) + crates/omnigraph/Cargo.toml — added datafusion = "52" direct dep (transitively pulled by Lance already; required for MemTable).
• docs/runs.md — removed "Known limitation" section; documented the new accumulator + D₂ + LoadMode::Overwrite residual.
• docs/invariants.md — §VI.25 status flipped to upheld for inserts/updates.
• docs/architecture.md — added "Mutation atomicity — in-memory accumulator (MR-794)" subsection; refreshed the engine + state diagrams to drop RunRegistry and add MutationStaging.
• docs/execution.md — rewrote the mutation flow sequence diagram for the staged-write path; updated the LoadMode table to call out per-mode commit semantics; rewrote load vs ingest.
• docs/query-language.md — documented the D₂ parse-time rule.
• docs/errors.md — added the D₂ BadRequest rejection path.
• docs/storage.md — dropped the live _graph_runs.lance reference (legacy from MR-771) from the layout diagram and prose.
• docs/branches-commits.md — moved __run__<id> to a legacy note; removed publish_run from the publish-trigger list.
• docs/audit.md — current _as API list refreshed; legacy RunRecord.actor_id moved to a historical note.
• docs/constants.md — marked the run registry / branch-prefix rows as legacy.
• docs/cli.md — replaced the legacy omnigraph run * quickstart block with omnigraph commit list/show.
• docs/testing.md — extended the runs.rs row to cover the new MR-794 contract tests; added the staged_writes.rs row.
• AGENTS.md (CLAUDE.md symlink) — updated the atomic-per-query description and the L2 capability matrix row.

Included Changes

• MR-794 step 2+ — rewire mutate_as and load via in-memory MutationStaging + stage_* / commit_staged per touched table at end-of-query.
• (MR-794 step 1 shipped in v0.4.0's PR #67 — StagedWrite, stage_append, stage_merge_insert, commit_staged, scan_with_staged, count_rows_with_staged — and is the substrate this release builds on.)