Implement RFC-022 unified graph write protocol (#343)

* Implement unified graph write protocol

* Preserve recovery error wire compatibility
This commit is contained in:
Andrew Altshuler 2026-07-11 14:02:54 +03:00 committed by GitHub
parent 0c8d769501
commit f758ff0d17
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
80 changed files with 13393 additions and 2050 deletions

View file

@ -170,30 +170,36 @@ Code paths:
- Graph index: `crates/omnigraph/src/graph_index/`
- Loader: `Omnigraph::ingest` at `crates/omnigraph/src/loader/mod.rs:74`
### Mutation atomicity — in-memory accumulator (MR-794)
### Mutation atomicity — in-memory accumulator + branch-wide OCC (MR-794 / RFC-022)
Inserts and updates inside `mutate_as` and the bulk loader's
Append/Merge modes go through `MutationStaging`
Mutations inside `mutate_as` and every bulk-load mode go through `MutationStaging`
([`crates/omnigraph/src/exec/staging.rs`](../../crates/omnigraph/src/exec/staging.rs)),
a per-query in-memory accumulator. No Lance HEAD advance happens during
op execution; one `stage_*` + `commit_staged` per touched table runs
at end-of-query, then the publisher commits the manifest atomically.
op execution. The attempt captures one native branch identity, exact graph head,
accepted schema identity/catalog, and base snapshot. At end-of-query it stages
one exact Lance transaction per touched table outside the effect gates, then
acquires the root-shared schema → branch → sorted-table gates and revalidates the
complete authority before arming recovery or advancing any table HEAD.
```
op-1 (insert/update) → push RecordBatch → MutationStaging.pending[table]
op-2 (insert/update) → read committed via Lance + pending via DataFusion
MemTable (read-your-writes) → push batch
op-N → push batch
─── end of query ───────────────────────────────────────
finalize: per pending table:
concat batches → stage_append OR stage_merge_insert OR stage_overwrite
→ commit_staged
publisher: ManifestBatchPublisher::publish (one cross-table CAS)
─── end of query ────────────────────────────────────────────────
stage_all: concat batches → one exact stage_* transaction per table (no HEAD move)
commit_all: acquire schema → branch → sorted-table gates
→ reject a new recovery intent → revalidate the complete branch token
→ arm schema-v3 recovery → commit_staged per table
publisher: publish pre-minted lineage under the exact native-branch/head + table precondition
```
A failed op leaves Lance HEAD untouched on the staged tables: the next
mutation proceeds normally with no drift to reconcile. Concrete
contracts:
A failed op leaves Lance HEAD untouched on the staged tables. If authority
changes before effects, insert-only mutations and Append/Merge loads discard the
whole attempt and fully reprepare with a bounded retry; Update/Delete/Overwrite
returns `ReadSetChanged`. After any physical effect, any later error returns
`RecoveryRequired` and leaves the v3 sidecar to converge the fixed outcome.
Concrete contracts:
- `D₂` parse-time rule: a query is either insert/update-only or
delete-only. Mixed → reject. Deletes now stage like inserts/updates
@ -262,7 +268,7 @@ flowchart LR
rec --> diff --> wp
```
Today, indexes are built explicitly via `ensure_indices`. Reads degrade gracefully when index coverage is partial — Lance's scanner unions indexed and scan paths automatically. The roadmap reconciler observes manifest state and converges coverage in the background.
Today, physical indexes are built explicitly via `ensure_indices`/`optimize`; schema apply, mutation, and load only record intent or publish their exact data effects and never build indexes inline. Reads degrade gracefully when index coverage is missing or partial — Lance's scanner unions indexed and scan paths automatically (vector search falls back to brute force). A future background reconciler may automate the same explicit convergence path.
### Server / CLI
@ -276,14 +282,14 @@ flowchart LR
pol[Cedar policy gate<br/>per request]:::l2
wl[WorkloadController<br/>per-actor admission]:::l2
eng[engine API<br/>Arc&lt;Omnigraph&gt;]:::l2
wq[WriteQueueManager<br/>per-(table, branch)]:::l2
wq[WriteQueueManager<br/>root-scoped schema → branch → table gates]:::l2
cli -.-> eng
srv_in --> auth --> pol --> wl --> eng
eng --> wq
```
The server applies Cedar policy at the HTTP boundary today. The roadmap, called out in [docs/dev/invariants.md](invariants.md) as a known gap, is to push policy into the planner as predicates. After Cedar, mutating handlers go through `WorkloadController` (per-actor admission cap + byte budget; PR 2 / MR-686) before reaching the engine. The engine itself holds an `Arc<WriteQueueManager>` so concurrent mutations on the same `(table, branch)` serialize at the queue, while disjoint keys run in parallel — see [docs/user/server.md](../user/operations/server.md) "Per-actor admission control" and [docs/dev/writes.md](writes.md). The CLI bypasses the HTTP layer (and admission) and calls the engine API directly.
The server applies Cedar policy at the HTTP boundary today. The roadmap, called out in [docs/dev/invariants.md](invariants.md) as a known gap, is to push policy into the planner as predicates. After Cedar, mutating handlers go through `WorkloadController` (per-actor admission cap + byte budget; PR 2 / MR-686) before reaching the engine. Every handle for one canonical graph root shares an `Arc<WriteQueueManager>` in-process. RFC-022-enrolled mutation/load attempts prepare outside the effect gates, then acquire the exclusive root schema gate, the target branch-effect gate, and sorted `(table, branch)` gates and hold them through publish. The root schema gate means enrolled effect windows on one graph currently serialize even across different branches; readers remain ungated, and pre-effect parsing, validation, and fragment staging can overlap. The branch gate preserves the complete per-branch validation authority, while table gates protect each concrete Lance effect and legacy adapter. These gates prevent same-process races and impose one deadlock-free order; the exact publisher precondition and durable recovery intent remain the correctness authorities, and the gates are not a cross-process lock. See [docs/user/server.md](../user/operations/server.md) "Per-actor admission control" and [docs/dev/writes.md](writes.md). The CLI bypasses the HTTP layer (and admission) and calls the engine API directly.
Code paths:
@ -303,8 +309,8 @@ Throughout the docs, capabilities are split into:
- **MVCC**: every Lance write bumps a per-dataset version; the OmniGraph manifest version coordinates which sub-table versions are visible together.
- **Snapshot isolation**: a query holds one `Snapshot` for its lifetime; concurrent writes don't leak in.
- **Cross-branch isolation**: copy-on-write means readers and writers on different branches don't block each other.
- **Per-query staging**: `mutate_as` and `load` (Append/Merge) accumulate insert/update batches in an in-memory `MutationStaging`; one `stage_*` + `commit_staged` per touched table runs at end-of-query, then the publisher commits the manifest atomically. A mid-query failure leaves Lance HEAD untouched on staged tables. (MR-794; pre-v0.4.0 used a `__run__<id>` staging branch + Run state machine, removed in MR-771.)
- **Cross-branch isolation**: Lance copy-on-write keeps branch data independent, and readers never acquire write gates. Enrolled writer preparation can overlap across branches, but their effect/publish windows currently share one exclusive root schema gate before the branch/table gates, so those windows serialize in-process even for different branches.
- **Per-query staging**: `mutate_as` and every `load` mode accumulate their work in an in-memory `MutationStaging`; `stage_all` prepares exact per-table transactions without moving HEAD. At commit, the root schema → branch → sorted-table gates protect complete-token revalidation, v3 recovery arming, zero-retry table effects, and one atomic manifest publish. A mid-query failure leaves Lance HEAD untouched on staged tables. (MR-794 / RFC-022; pre-v0.4.0 used a `__run__<id>` staging branch + Run state machine, removed in MR-771.)
- **Schema-apply lock**: `__schema_apply_lock__` system branch serializes schema migrations.
- **Fail-points** (`failpoints` cargo feature): `failpoints::maybe_fail("operation.step")?` in `branch_create`, publish, etc., for deterministic failure injection in tests.

View file

@ -79,16 +79,16 @@ Hybrid example: `order { rrf(nearest($d.embedding, $q), bm25($d.body, $q_text))
## Mutation execution (`exec/mutation.rs`)
Resolves expression values to literals, converts to typed Arrow arrays (`literal_to_typed_array(lit, DataType, num_rows)`), then writes via Lance's two-phase distributed-write API at end-of-query:
Resolves expression values to literals, converts to typed Arrow arrays (`literal_to_typed_array(lit, DataType, num_rows)`), then writes via Lance's two-phase distributed-write API at end-of-query. Before lowering/execution, one `WriteTxn` captures the target's Lance-native branch identity, exact optional graph head, accepted schema identity/catalog, and base table snapshot; every step in the attempt uses that immutable authority.
- `insert` (no `@key`, edges) → accumulate into `MutationStaging.pending` (Append mode); finalize calls `stage_append` once per touched table.
- `insert` (`@key` node) → accumulate into `pending` (Merge mode); finalize calls `stage_merge_insert` once per touched table.
- `insert` (no `@key`, edges) → accumulate into `MutationStaging.pending` (Append mode); `stage_all` later calls `stage_append` once per touched table.
- `insert` (`@key` node) → accumulate into `pending` (Merge mode); `stage_all` later calls `stage_merge_insert` once per touched table.
- `update` → scan committed via Lance + pending via DataFusion `MemTable` (read-your-writes), apply assignments, accumulate into `pending` (Merge mode).
- `delete` → records a predicate into `MutationStaging.delete_predicates` (count matching committed rows now for `affected_*`); finalize combines a table's predicates into one `stage_delete` (Lance 7.0 `DeleteBuilder::execute_uncommitted`, a deletion-vector transaction) committed via `commit_staged` — no inline HEAD advance (MR-A).
- `delete` → records a predicate into `MutationStaging.delete_predicates` (count matching committed rows now for `affected_*`); `stage_all` combines a table's predicates into one `stage_delete` (Lance 7.0 `DeleteBuilder::execute_uncommitted`, a deletion-vector transaction) — no inline HEAD advance (MR-A).
**D₂ parse-time rule.** A single mutation query is either insert/update-only or delete-only. Mixed → reject before any I/O. The check fires in `enforce_no_mixed_destructive_constructive(&ir)` inside `execute_named_mutation`.
Multi-statement mutations are atomic at the publisher commit boundary: every insert/update batch lives in memory until end-of-query, then exactly one `stage_*` + `commit_staged` runs per touched table, then `ManifestBatchPublisher::publish` commits the manifest atomically with per-table `expected_table_versions` CAS.
Multi-statement mutations are atomic at the publisher commit boundary. Every batch lives in memory until all statements and validation succeed; `stage_all` then prepares one exact transaction per touched table without advancing HEAD. `commit_all` acquires the root-shared schema → branch → sorted-table gates, rechecks for recovery intent, revalidates the complete branch authority, writes the schema-v3 recovery sidecar, and commits the table transactions with zero transparent conflict retries. The guards remain held while `ManifestBatchPublisher` publishes the pre-minted lineage under the same exact native-branch/head and table-version precondition.
### Mutation flow — sequence
@ -100,11 +100,12 @@ sequenceDiagram
participant cmp as omnigraph-compiler
participant stg as MutationStaging<br/>(exec/staging.rs)
participant ts as table_store
participant lance as Lance dataset
participant rec as schema-v3 recovery sidecar
participant pub as ManifestBatchPublisher
client->>og: mutate_as(branch, source, name, params, actor_id)
og->>cmp: parse + typecheck + lower_mutation_query
og->>og: heal/reject recovery intent; open_write_txn
og->>cmp: parse + typecheck + lower using txn catalog
cmp-->>og: MutationIR
og->>og: enforce_no_mixed_destructive_constructive (D₂)
loop for each mutation op
@ -116,23 +117,43 @@ sequenceDiagram
og->>ts: scan_with_pending (Lance + DataFusion MemTable union)
ts-->>og: matched batches
end
else delete (stage, D₂ keeps separate)
else delete (record predicate; D₂ keeps separate)
og->>ts: count_rows (committed match → affected_*)
og->>stg: ensure_path + record_delete (predicate)
end
end
og->>stg: finalize(db, branch)
loop per pending table
stg->>ts: stage_append OR stage_merge_insert (one per table)
ts-->>stg: StagedWrite (transaction + commit metadata + fragments)
stg->>ts: commit_staged (advances Lance HEAD)
ts-->>stg: new Dataset
og->>og: validate complete staged change-set against txn base
og->>stg: stage_all(db, branch)
loop per touched table
stg->>ts: stage_append OR stage_merge_insert OR stage_delete (one per table)
ts-->>stg: exact staged transaction (no HEAD movement)
end
stg->>stg: acquire schema → branch → sorted-table gates
stg->>og: recheck recovery barrier + revalidate complete WriteTxn
alt authority changed before effects
stg-->>og: ReadSetChanged
alt insert-only mutation
og->>og: discard complete attempt; bounded full reprepare
else Update/Delete
og-->>client: ReadSetChanged (409)
end
else authority unchanged
stg->>rec: persist fixed lineage + exact transaction identities
loop per touched table
stg->>ts: commit_staged (zero transparent retries)
ts-->>stg: confirm achieved transaction + table update
end
stg-->>og: updates + expected versions + sidecar + held gates
og->>pub: publish exact graph-head/table precondition
alt publish succeeds
pub-->>og: new manifest version
og->>rec: delete sidecar
og-->>client: MutationResult
else any error after an effect
pub-->>og: error
og-->>client: RecoveryRequired (sidecar remains authoritative)
end
end
stg-->>og: (updates: Vec<SubTableUpdate>, expected_versions)
og->>pub: commit_updates_on_branch_with_expected
pub->>pub: publisher CAS (cross-table OCC on __manifest)
pub-->>og: new manifest version
og-->>client: MutationResult
```
**Code paths:**
@ -143,11 +164,11 @@ sequenceDiagram
- Per-op execution: `execute_insert`, `execute_update`, `execute_delete_node`, `execute_delete_edge`
- Pending-aware reads: `TableStore::scan_with_pending` / `count_rows_with_pending` at `crates/omnigraph/src/table_store.rs`
- Edge cardinality with pending: `validate_edge_cardinality_with_pending` at `crates/omnigraph/src/exec/mutation.rs`
- Per-query accumulator: `crates/omnigraph/src/exec/staging.rs` (`MutationStaging`, `PendingTable`, `PendingMode`, `finalize`)
- End-of-query Lance commit: `TableStore::stage_append`, `stage_merge_insert`, `commit_staged` at `crates/omnigraph/src/table_store.rs`
- Manifest commit primitive: `commit_updates_on_branch_with_expected` at `crates/omnigraph/src/db/omnigraph/table_ops.rs`
- Per-query accumulator and protocol adapter: `crates/omnigraph/src/exec/staging.rs` (`MutationStaging::stage_all`, `StagedMutation::commit_all`)
- End-of-query Lance operations: `TableStore::stage_append`, `stage_merge_insert`, `stage_delete`, `commit_staged` at `crates/omnigraph/src/table_store.rs`
- Manifest commit primitive: `commit_updates_on_branch_with_expected` at `crates/omnigraph/src/db/omnigraph/table_ops.rs` (exact native-branch/head precondition plus expected table versions)
Atomicity guarantee for multi-statement mutations: a mid-query failure leaves Lance HEAD untouched on staged tables (no inline commit happened during op execution), so the next mutation proceeds normally with no `ExpectedVersionMismatch`. The publisher CAS at the very end either succeeds (manifest advances atomically across all touched sub-tables) or fails with a typed `ManifestConflictDetails::ExpectedVersionMismatch` (no partial publish). See [docs/dev/invariants.md](invariants.md) and [docs/dev/writes.md](writes.md).
Atomicity guarantee for multi-statement mutations: a mid-query failure leaves Lance HEAD untouched because no effect occurs during statement execution or staging. A pre-effect authority mismatch discards the complete attempt: insert-only mutations fully reprepare with a bounded retry, while Update/Delete returns typed `ReadSetChanged`. Once any `commit_staged` effect is durable, the attempt either makes every update visible together or returns `RecoveryRequired` with the fixed sidecar left for recovery; the engine never treats that post-effect error as an ordinary rebase. See [docs/dev/invariants.md](invariants.md) and [docs/dev/writes.md](writes.md).
## Bulk loader (`loader/mod.rs`)
@ -162,11 +183,11 @@ Atomicity guarantee for multi-statement mutations: a mid-query failure leaves La
| Mode | Semantics | Path (post-MR-794) |
|---|---|---|
| `Overwrite` | Replace all data in the target tables on the branch | Same accumulator; one `stage_overwrite` + `commit_staged` per touched table at end-of-load (a staged Lance `Operation::Overwrite` transaction — HEAD does not advance until commit; MR-793 Phase 2); publisher CAS. |
| `Append` | Strict insert; duplicates error | In-memory `MutationStaging` accumulator; one `stage_append` + `commit_staged` per touched table at end-of-load; publisher CAS. |
| `Merge` | Upsert by `id` (`merge_insert`) | Same accumulator; one `stage_merge_insert` per touched table at end-of-load (Merge mode dedupes by `id`, last-write-wins); publisher CAS. |
| `Overwrite` | Replace all data in the target tables on the branch | Same accumulator; one staged Lance `Operation::Overwrite` transaction per touched table. A pre-effect authority change is strict `ReadSetChanged`; no automatic replay. |
| `Append` | Strict insert; duplicates error | One `stage_append` transaction per touched table. A pre-effect authority change discards the whole parsed/validated attempt and fully reprepares with a bounded retry. |
| `Merge` | Upsert by `id` (`merge_insert`) | One `stage_merge_insert` transaction per touched table (deduped by `id`, last-write-wins). The same bounded full-reprepare rule as Append applies before effects. |
For all three modes, a mid-load failure (RI / cardinality violation, validation error) leaves Lance HEAD untouched on the staged tables — the next load on the same tables proceeds normally with no `ExpectedVersionMismatch`.
All three modes then use the same schema → branch → sorted-table gate, v3 recovery, zero-retry table commit, and exact publisher-precondition path as mutation. A parse, RI, cardinality, or validation failure leaves Lance HEAD untouched. After any table effect, any later error is `RecoveryRequired`. Load, mutation, and schema apply build no physical indexes inline; explicit `ensure_indices`/`optimize` reconciliation materializes declared intent later.
## `load` and the deprecated `ingest` shims

View file

@ -61,11 +61,14 @@ amplification = PR2's target.)
### A.2 What is LANDED on `main`
- **Step 2a**`optimize` compacts the internal tables too (`__manifest` / `_graph_commits` /
`_graph_commit_actors`), so a *periodically-compacted* graph keeps Term-1 flat. (Cleanup/version-GC
of them is the still-open PR1.)
- **Step 2a**#291 originally added compaction for all three then-current
internal tables (`__manifest`, `_graph_commits`, and `_graph_commit_actors`).
After strand-and-retire removed the two lineage datasets, the live behavior is
`__manifest`-only internal compaction. A periodically-compacted graph therefore
keeps the remaining Term-1 flat. (`__manifest` cleanup/version-GC is still open.)
- **Phase 7 / #299** (`1c5cb874`) — graph lineage lives in `__manifest` (`graph_commit` +
`graph_head:<branch>` rows in the same publish merge-insert; `_graph_commits` is now a projection;
`graph_head:<branch>` rows in the same publish merge-insert; `CommitGraph` is
now a projection and the former lineage datasets are gone;
v3→v4 internal-schema migration; schema-version floor). This removed the per-write commit-graph
scan and closed the manifest→commit-graph atomicity + commit-graph-parent-under-concurrency gaps.
**This is the base everything below builds on.**
@ -202,12 +205,14 @@ for the canonical list. Current reality:
- **Step 3a** — opener bypass: write opens go direct (`Dataset::open` by URI + version)
instead of the Lance-namespace builder (#288). **This already banked the dominant
depth win** — see §2 below; it reframes everything.
- **Step 2a** — internal-table compaction: `optimize` now compacts `__manifest` /
`_graph_commits` / `_graph_commit_actors` (#291). Plus the RFC latency-model
correction (#292).
- **Step 2a** — internal-table compaction: #291 originally covered
`__manifest` plus the two then-live lineage datasets. The current
post-retirement implementation compacts only `__manifest`. Plus the RFC
latency-model correction (#292).
- **Step 4 / Phase 7** — graph lineage moved into `__manifest` (#299 `1c5cb874`):
`graph_commit` + mutable `graph_head:<branch>` in the publish merge-insert,
`_graph_commits` now a projection. **The base for the live branch (§A).**
with `CommitGraph` projected directly from those rows and no separate lineage
dataset. **The base for the live branch (§A).**
- **Optimize-vs-write race** — optimize survives a cross-process write race on the
same table (#297, **LANDED** — origin/main `6d4606a8`; see §6 for why it's not
redundant with Design A). Step 3b stacks on top of this.
@ -312,7 +317,7 @@ single staleness feeds **two distinct failure modes**, both surfaced this cycle:
`writes.rs::served_strict_delete_after_external_optimize_advance_auto_refreshes`
(`#[ignore]` on branch `fix/write-path-stale-view-probe`). **The naive "just probe" fix is
proven wrong** — a blanket probe silently refreshes past *logical* advances too, breaking
`consistency::stale_handle_public_mutation_must_refresh_then_retry` (the deliberate
`consistency::stale_handle_strict_mutation_returns_read_set_changed_then_refresh_retry` (the deliberate
cross-process lost-update OCC primitive). The fix must **discriminate by op class**.
**Both fold into Design A (step 5), same as §1c.** `open_txn`'s one warm probe makes the base
@ -425,13 +430,14 @@ reopen/replan **semantics** are permanent. (Noted in RFC §6.6.)
## 4. DONE: Step 3b — capture-once `WriteTxn` (shipped on `rfc-013-step-3b-writetxn-v2`)
**Delivered:** on the **table-touch hot path**, a single `mutate`/`load` validates the schema
contract **once** and opens each touched data table **at most once** — a constant-factor/RTT
win (not a depth-slope win; 1a). Two cost gates in `write_cost.rs` lock it (both on a node
insert): `write_validates_schema_contract_once` (3 `read_text` / 2 `exists`, was 12/9) and
contract once at capture and once at the RFC-022 pre-effect gate, with one cheap trailing
identity-marker read fencing capture (never per table), and opens each touched data table
**at most once** — a constant-factor/RTT win (not a depth-slope win;
1a). Two cost gates in `write_cost.rs` lock it (both on a node insert):
`write_schema_io_is_bounded_to_capture_fence_and_effect_gate` (7 `read_text` / 4 `exists`) and
`keyed_insert_opens_table_at_most_once` (`data_open_count <= 1`, was 4). The carrier is the
minimal `WriteTxn { branch, base }`, threaded as `Option<&WriteTxn>` (`Some` on the hot
mutate/load path, `None` byte-identical everywhere else); it **converges into** step 5's
`PublishPlan`.
operation-local `WriteTxn { branch, base, authority, catalog }`; the catalog is built from the
exact accepted IR named by the authority token. It **converges into** step 5's `PublishPlan`.
**Not "once" everywhere (scope, not regression):** edge endpoint / cardinality RI validation
(`ensure_node_id_exists`, the loader's RI + cardinality) still resolves through
@ -444,8 +450,9 @@ gate yet (only the node gates above).
Commits (off merged-#297 main):
- **Stage 0** — scope `open_count``data_open_count`/`internal_open_count` by URI class
(the review fix: `open_dataset_tracked` also opens `__manifest`/`_graph_commits`, so the
raw counter conflated them and the gate was unreachable). Re-baselined RED 4.
(historically, before lineage-table retirement, `open_dataset_tracked` also opened
`__manifest`/`_graph_commits`, so the raw counter conflated them and the gate was
unreachable). Re-baselined RED 4.
- **Commit A (schema-once)** — capture `txn` once at entry (the single validation); the 4
validation sites collapse: S1 (entry `ensure_schema_state_valid`) removed; S3a
(`open_for_mutation_on_branch`) + S3b (`prepare_updates_for_commit`) source `txn.base`;
@ -587,7 +594,8 @@ for #298** (which built none of those constructs) but are **load-bearing constra
residual concurrent TOCTOU is the §7.1 gap (step 4) — un-widen here, don't over-reach.
- **Step 4 / Phase 7** (`iss-991`): **LANDED on `main` as #299 (`1c5cb874`).** Lineage now lives
in `__manifest` (`graph_commit` + mutable `graph_head:<branch>` in the same merge-insert;
`_graph_commits` is a projection). Removed the per-write `commit_graph.refresh`; closed the
`CommitGraph` is projected from those rows and the old lineage dataset is
gone). Removed the per-write `commit_graph.refresh`; closed the
manifest→commit-graph atomicity + commit-graph-parent-under-concurrency gaps. *(Historical note,
kept for the §7.1 framing it carried:)* it
carries the §7.1 *concurrent* write-skew fix (needs the `graph_head` contention row) —
@ -600,7 +608,7 @@ for #298** (which built none of those constructs) but are **load-bearing constra
false-fail** (§1d.2) — the op-class-aware precondition + `open_txn` probe. The contract is
two tests passing *together*: un-ignore
`writes.rs::served_strict_delete_after_external_optimize_advance_auto_refreshes` (goes green)
*while* `consistency::stale_handle_public_mutation_must_refresh_then_retry` stays green
*while* `consistency::stale_handle_strict_mutation_returns_read_set_changed_then_refresh_retry` stays green
(maintenance fast-forwards; logical fails loudly). Self-contained enough to ship standalone
like #297 if prod pain is acute; otherwise fold into the single PublishPlan delta-interpreter.
- **Step 2b** — internal-table cleanup + the Q8 monotonic watermark (a Lance boundary tag).

View file

@ -74,6 +74,15 @@ pattern, not just the outcome.
|---|---|
| camelCase property filters lowercased at runtime (#283) — two engine→Lance boundaries, two different fixes | [bug-case-fix.md](bug-case-fix.md) |
## Active Design Reviews
Review ledgers record open findings against proposed architecture. They remain
as durable disposition history after closure, so RFC backlinks stay valid.
| Area | Read |
|---|---|
| RFC-022027 split architecture review — open correctness blockers, dependency corrections, and required acceptance evidence | [rfc-022-027-architecture-review.md](rfc-022-027-architecture-review.md) |
## Active Implementation Plans
Working documents for in-flight feature work. Removed when the work lands.

View file

@ -146,7 +146,11 @@ converge the physical state.
drifts" and "manifest-derivable reconciler" items, are instances; so is
bounding a read's cost to its working set rather than the commit count. This
is the structural face of "engineering is programming integrated over time":
both failure modes are liabilities that compound as the system grows.
both failure modes are liabilities that compound as the system grows. At a
write/control boundary, schema identity, compiled catalog, and manifest
authority must come from one operation-local accepted view: validating a
fresh on-disk marker while planning or enumerating gates from a stale warm
catalog is not a coherent derivation.
## Current Truth Matrix
@ -155,11 +159,11 @@ converge the physical state.
| Multi-table commit | Manifest CAS plus recovery sidecars; not a single Lance primitive | [writes.md](writes.md), [architecture.md](architecture.md) |
| Constructive mutations | In-memory `MutationStaging`, one end-of-query table commit per touched table, then one manifest publish | [writes.md](writes.md), [execution.md](execution.md) |
| Deletes | Staged like inserts/updates (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, MR-A) — no inline HEAD advance; mixed insert/update/delete in one query rejected by D2 as a deliberate boundary (constructive XOR destructive per query; compose via separate mutations or a branch) | [query-language.md](../user/queries/index.md), [writes.md](writes.md) |
| Branch delete | Manifest is the single authority, flipped atomically first; per-table forks are derived state, reclaimed best-effort (`force_delete_branch`) with the `cleanup` reconciler as the guaranteed backstop. Reusing a name whose reclaim failed before `cleanup` surfaces an actionable error | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md) |
| Branch delete | Manifest is the single authority, flipped atomically first; per-table forks are derived state, reclaimed best-effort (`force_delete_branch`) with the `cleanup` reconciler as the guaranteed backstop. Under schema/target/all-table gates, a target-scoped unresolved sidecar may be made unreachable by deletion and is then audit-discarded by recovery; graph-global SchemaApply still blocks. Reusing a name whose fork reclaim failed before `cleanup` surfaces an actionable error | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) |
| Schema validation | Type checks, required fields, defaults, edge endpoint checks, and edge cardinality are enforced on write paths | [schema-language.md](../user/schema/index.md), [execution.md](execution.md) |
| Unique constraints | Value/enum, uniqueness, edge-RI, and cardinality route through ONE unified, catalog-derived evaluator (`crate::validate`) on ALL THREE write surfaces — branch-merge, mutation, and bulk load: Δ-scoped (checks the delta, not the whole graph) and index-backed (probes committed state through its BTREE rather than full-scanning every catalog table), reusing the leaf checks (`loader::validate_value_constraints`/`validate_enum_constraints`/`composite_unique_key`) so the surfaces cannot drift. This closed the prior merge bug (merge validated `@range`/`@check` but not enum) AND the **cross-version uniqueness gap** on the mutation and load paths (a duplicate of a committed `@unique` value is now rejected; the merge path always enforced it). The committed view is the merge target snapshot (merge), the write's pinned `txn.base` (mutation), or the pinned pre-load base (load — `Overwrite` validates the batch as the whole new image, committed view empty); `@card` reads LIVE HEAD per edge table on the mutation path only (the #298 stale-handle fix). `@key` is id-backed, so it is checked intra-delta only (a committed holder of a key value is always the same row — an upsert), skipping a wasted O(Δ) probe per keyed row; `@unique` (non-key) groups do the committed lookup. | [schema-language.md](../user/schema/index.md) |
| Unique constraints | Value/enum, uniqueness, edge-RI, and cardinality route through ONE unified, catalog-derived evaluator (`crate::validate`) on ALL THREE write surfaces — branch-merge, mutation, and bulk load: Δ-scoped (checks the delta, not the whole graph) and structured-filter-backed (committed probes use a BTREE when reconciled and remain correct by scanning while it is pending), reusing the leaf checks (`loader::validate_value_constraints`/`validate_enum_constraints`/`composite_unique_key`) so the surfaces cannot drift. This closed the prior merge bug (merge validated `@range`/`@check` but not enum) AND the **cross-version uniqueness gap** on the mutation and load paths (a duplicate of a committed `@unique` value is now rejected; the merge path always enforced it). The committed view is the merge target snapshot (merge), the write's pinned `txn.base` (mutation), or the pinned pre-load base (load — `Overwrite` validates the batch as the whole new image, committed view empty); `@card` refreshes the manifest-visible graph-branch snapshot on the mutation path only (the #298 stale-handle fix), then follows each entry's actual inherited/owned Lance ref. `@key` is id-backed, so it is checked intra-delta only (a committed holder of a key value is always the same row — an upsert), skipping a wasted O(Δ) probe per keyed row; `@unique` (non-key) groups do the committed lookup. | [schema-language.md](../user/schema/index.md) |
| Storage trait | `TableStorage` (via `db.storage()`) is staged-only; the sole inline-commit residual (`create_vector_index`) is split onto a separate sealed `InlineCommitResidual` trait reached via `db.storage_inline_residual()` (MR-854), so §1 holds by construction; capability/stat surfaces are roadmap | [writes.md](writes.md), [architecture.md](architecture.md) |
| Index lifecycle | `@index`/`@key` declares *intent*; the physical index is derived state and never fails a logical op. `schema apply` builds no indexes (records intent only; index-only changes touch no table data). `load`/`mutate` build inline through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector) that fault-isolates an untrainable Vector column into a *pending* index instead of aborting. `optimize`/`ensure_indices` is the reconciler: it creates declared-but-missing indexes and folds appended/rewritten fragments into existing ones (`optimize_indices`), reporting still-pending columns. Explicit maintenance call, not yet a background loop | [indexes.md](../user/search/indexes.md), [maintenance.md](../user/operations/maintenance.md) |
| Index lifecycle | `@index`/`@key` declares *intent*; the physical index is derived state and never fails a logical op. `schema apply`, `load`, and `mutate` build no indexes inline: RFC-022 mutation/load sidecars describe only their exact data effects, and index availability must never become a correctness prerequisite. `optimize`/`ensure_indices` is the reconciler: through the single `build_indices_on_dataset_for_catalog` chokepoint it creates declared-but-missing indexes (enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector), folds appended/rewritten fragments into existing ones (`optimize_indices`), and reports untrainable Vector columns as pending. Explicit maintenance call, not yet a background loop | [indexes.md](../user/search/indexes.md), [maintenance.md](../user/operations/maintenance.md) |
| Traversal IDs | Runtime still builds `TypeIndex`; Lance stable row-id based graph IDs are roadmap | [architecture.md](architecture.md), [query-language.md](../user/queries/index.md) |
| Auth | Bearer token hashing and server-side actor resolution are implemented at the HTTP boundary | [server.md](../user/operations/server.md), [policy.md](../user/operations/policy.md) |
| Tests | Tempdir-backed Lance tests are the current substrate; the storage adapter has an in-memory backend for adapter-level contract tests, but Lance datasets bypass it | [testing.md](testing.md) |
@ -215,12 +219,15 @@ them explicit.
`maintenance.rs::optimize_compacts_blob_table_alongside_plain_table` pin the
positive behavior (a red there means blob compaction regressed — restore the
skip machinery from git history).
- **Recovery is serialized against live writers in-process only:** the
write-entry heal (and `refresh`) serialize against a live writer's sidecar
lifetime via the per-`(table, branch)` write queues plus the schema-apply
serialization key — all in-process primitives. A recovery pass in one
process cannot serialize against a live writer in another (the open-time
sweep has the same exposure, and always has): it may roll a live foreign
- **Recovery is serialized against live writers in-process only:** every
`Omnigraph` handle for one canonical local root identity (lexically absolute,
with existing symlink ancestors resolved) shares a root-scoped queue manager;
object-store/custom scheme identities remain their normalized opaque URIs.
The write-entry heal, `refresh`, and Full ReadWrite-open sweep serialize
against a live writer's complete sidecar lifetime in schema → branch → sorted
table order and re-check sidecar existence after waiting. These remain
in-process primitives: a recovery pass cannot serialize against a live writer
in another process. It may roll a live foreign
writer's sidecar forward, which degrades to publisher-CAS contention for
data writes but can race the schema-staging promotion for a foreign live
schema apply. The roll-**forward** CAS contention is now
@ -238,22 +245,26 @@ them explicit.
cross-process serialization primitive (e.g. lease-based use of the
schema-apply lock branch) — design it before promoting multi-process write
topologies.
- **Fork reclaim is in-process-safe only:** the first write to a table on a
branch forks it (a Lance `create_branch` that advances state before the
manifest publish). An interrupted fork (crash, or a cancelled request
future) leaves a manifest-unreferenced branch ref. The next write self-heals
it — `reclaim_orphaned_fork_and_refork` (`force_delete_branch` + re-fork)
— but reclaim is only safe because the writer holds the per-`(table,
branch)` write queue from before the fork through the publish AND re-checks
the live manifest under it, so no *in-process* writer can be mid-fork. A
reclaim cannot serialize against a foreign-*process* in-flight fork: it may
force-delete a peer's just-created ref, which makes that peer's commit fail
and retry — the same one-winner-CAS exposure as above, not corruption. The
reclaim never fires unless in-process-queue + manifest authority both prove
the ref is manifest-unreferenced. `cleanup`'s per-table reconciler
(`reconcile_orphaned_branches`) is the guaranteed backstop for any fork the
write path never revisits. Both degrade to a no-op if Lance ships an atomic
multi-dataset branch op.
- **Fork ownership is durable, but Lance ref deletion is not conditional:** a
first-touch Mutation/Load table no longer creates its target ref while
preparing. Under schema → branch → table gates it revalidates, durably arms a
schema-v3 sidecar naming the target, creates the ref, then stages branch-local
files and commits. Both `reclaim_orphaned_fork_and_refork` and
`reconcile_orphaned_branches` consult pending sidecars before destruction; a
foreign claim is conflict/indeterminate, never permission to delete. Full
recovery accepts sidecar-before-ref crashes and deletes an unpublished fork
only when it is still exactly at the inherited version and no other sidecar
claims `(table_path, target ref)`. A no-effect sidecar with a competitor
discards only itself; the last survivor either cleans the untouched ref or
recovers its owned effect. Partial rollback performs no-effect cleanup before
publishing its fixed outcome. This closes the cross-handle live-ref
deletion bug and keeps cleanup as the backstop for truly unclaimed refs.
The remaining multi-process gap is narrower but real: Lance exposes
`force_delete_branch`, not compare-and-delete by `BranchIdentifier`, so a
foreign process can create intent/ref between the final list/check and the
delete. The documented single-writer-process support boundary remains until
Lance provides a conditional ref primitive (or OmniGraph adds a distributed
fence); process-local queues are not credited as that primitive.
- **Local `write_text_if_match` is not a cross-process CAS:** object-store
backends use a true conditional put (ETag If-Match; the in-memory test
backend too), but upstream `object_store` leaves `PutMode::Update`

View file

@ -36,7 +36,7 @@ The implementation spec's hard gate for this phase — failpoint recovery tests
What Phase 4 builds on (all shipped):
- **The engine's recovery discipline.** Writers that can advance Lance HEAD before manifest publish write `__recovery/{ulid}.json` sidecars carrying per-table pins (`expected_version`, `post_commit_pin`); `Omnigraph::open` in read-write mode classifies every pinned table (`NoMovement` / `RolledPastExpected` / `UnexpectedAtP1` / `UnexpectedMultistep` / `InvariantViolation`) and decides all-or-nothing: roll forward via one manifest publish, or roll back via `Dataset::restore`, recording an audit row attributed to `omnigraph:recovery`. The cluster inherits the *vocabulary* of this design but not its mechanics — see the roll-forward-only argument below.
- **The engine's recovery discipline.** Writers that can advance Lance HEAD before manifest publish write `__recovery/{ulid}.json` sidecars carrying per-table pins (`expected_version`, `post_commit_pin`); `Omnigraph::open` in read-write mode classifies every pinned table (`NoMovement` / `RolledPastExpected` / `UnexpectedAtP1` / `UnexpectedMultistep` / `InvariantViolation`) and decides all-or-nothing: roll forward via one manifest publish, or roll back via `Dataset::restore`. Every completed action records an audit row with the interrupted writer's actor in `recovery_for_actor`. A schema-v3 roll-forward publishes the sidecar's pre-minted lineage and therefore preserves that writer's commit id and actor; rollback and legacy recovery commits use `omnigraph:recovery`. The cluster inherits the *vocabulary* of this design but not its mechanics — see the roll-forward-only argument below.
- **The engine's schema-apply surface.** `apply_schema_as(desired_source, SchemaApplyOptions { allow_data_loss }, actor)` returns `SchemaApplyResult { supported, applied, manifest_version, steps }`; `preview_schema_apply_with_options` returns the migration plan plus desired catalog without applying; the `__schema_apply_lock__` branch serializes schema applies graph-wide and refuses to run while user branches exist. Policy enforcement (`enforce(SchemaApply, TargetBranch("main"), actor)`) happens before the lock.
- **Graph init.** `Omnigraph::init(uri, schema_source)` with a strict preflight (errors if schema artifacts exist) and an atomic `_schema.pg` claim. A documented gap: a failed init does not clean up Lance datasets or `__manifest/` it already created.
- **No engine graph-delete primitive.** Deleting a graph today means removing its object-store prefix. This RFC works with that fact rather than waiting on a primitive.

View file

@ -0,0 +1,549 @@
# Architecture review: RFC-022 through RFC-027
**Status:** Open review findings
**Date:** 2026-07-11
**Audience:** RFC authors, engine/storage maintainers, and release reviewers
**Reviewed against:** OmniGraph 0.8.1; Lance 9.0.0-beta.15 at
`f24e42c11a742581365e1cbe17c906ea2dac1bc6`; full Lance transaction,
branch/tag, MemWAL, row-lineage, read/write, compaction, and cleanup
specifications; pinned Rust implementation where the public specification is
not precise enough
This is a review ledger, not a competing specification. The RFCs remain the
normative proposals. A finding is closed only when the affected RFC either:
1. incorporates the required contract and tests; or
2. explicitly rejects the finding with evidence and records the resulting
support boundary.
Items marked **BLOCKER** must be dispositioned before the RFC that owns them is
accepted. Items marked **TIGHTENING** may be resolved during revision, but they
must not silently disappear from review.
This ledger is durable review history, not a throwaway implementation plan.
After every finding is dispositioned, change its status to closed and record
the resolving RFC sections/commits; keep the RFC backlinks valid.
## Overall architectural judgment
The central architecture remains the right one:
- Lance is the physical storage/versioning truth.
- `__manifest` is the graph-visible authority and one graph publish is the
visibility point for a logical graph commit.
- RFC-022 should define one correctness protocol with explicit physical-effect
adapters, not pretend that Lance offers one universal staged primitive.
- Key fencing, durable heads, retention, MemWAL, and lineage acceleration are
separate irreversible decisions and should keep separate evidence and format
gates.
The current blockers are boundary problems, not a reason to replace that core.
They concern crash classification, actual fencing, foreign-branch authority,
and capability activation.
> 💬 **Second-pass verification (2026-07-11):** I independently re-verified this
> ledger's two sharpest new substrate claims against the pinned checkout before
> commenting: BLOCKER-01's two-phase branch create is confirmed in Lance's own
> doc comment, and BLOCKER-04's lazy-branch exposure is confirmed
> architecturally — and appears to be a live bug in the shipped `cleanup`, not
> only an RFC gap (see the comment there). Overall judgment: concur with this
> ledger; three of its findings supersede corrections I applied to the RFCs on
> 2026-07-11 (noted inline at BLOCKER-07, BLOCKER-11, and tightening 5).
The latest revisions improved several important points and those changes should
remain:
- exact recovery-goal convergence is distinguished from adopting an unrelated
newer version;
- MemWAL is treated as Lance's strategic streaming architecture, with API and
format evolution as the integration risk;
- keyed fast-forward merge now has an explicit embedding-table memory gate;
- the legacy `__manifest` PK representation is preserved rather than
“normalized” into an incompatible form;
- cleanup's required operating cadence and the cost of indefinite deferral are
explicit;
- native branch refs are no longer described as cross-process-safe merely
because a process-local gate exists.
## Dependency correction
Stable table identity and incarnation are consumed by more than durable heads.
The intended dependency shape is therefore:
```text
RFC-022 unified write protocol
|
+-- stable identity/incarnation capability
| +-- RFC-024 durable heads
| +-- RFC-025 checkpoint rows
| `-- RFC-026 stream/reject authority
|
+-- RFC-023 key fencing
| `-- RFC-026 keyed streaming mode
|
+-- RFC-025 retention
+-- RFC-026 MemWAL
`-- RFC-027 merge-delta research
```
The public RFC process accepts whole RFCs, not independently accepted sections.
The identity capability must therefore be extracted into its own RFC, or
RFC-024 must be accepted in full before identity-dependent siblings. An
implementation may phase an accepted RFC internally, but it cannot treat a
sub-contract of an unaccepted RFC as a separately accepted decision.
## Blocking findings
### BLOCKER-01 — native graph-branch control is multi-phase
**Affected:** [RFC-022 §7](../rfcs/rfc-022-unified-write-path.md#7-native-graph-branch-ref-control-protocol)
RFC-022 currently models branch create/delete as one native ref mutation whose
completion is the visibility point. The pinned Lance implementation is more
specific:
- [`Dataset::create_branch`](https://github.com/lance-format/lance/blob/f24e42c11a742581365e1cbe17c906ea2dac1bc6/rust/lance/src/dataset.rs#L488-L535)
first commits a shallow-cloned branch dataset and only then writes
`BranchContents`. Lance explicitly calls this a non-atomic two-phase
operation and calls `BranchContents` the source of truth.
- [branch deletion](https://github.com/lance-format/lance/blob/f24e42c11a742581365e1cbe17c906ea2dac1bc6/rust/lance/src/dataset/refs.rs#L548-L584)
removes `BranchContents` before cleaning the branch directory.
Therefore create can crash with a zombie branch dataset and no authoritative
ref. A retry with the same name can fail until the zombie is reclaimed. Delete
can make the branch logically absent and then return a cleanup error.
**Required disposition:** define an idempotent control-operation classifier for
at least these states:
| Operation | Physical state | Logical result |
|---|---|---|
| create | no clone, no `BranchContents` | not started |
| create | clone only | zombie; reclaim before retry, or wait for an upstream completion API |
| create | clone plus matching `BranchContents` | complete |
| delete | `BranchContents` present | not deleted |
| delete | `BranchContents` absent, directory remains | deleted; reclaim pending |
| delete | both absent | complete |
The pinned public API cannot adopt the clone-only state: `BranchIdentifier` is
generated later by Lance's private `BranchContents` creation phase. The RFC
must therefore say whether a durable operation marker/sidecar is written before
the shallow clone, how retry distinguishes its own zombie from foreign state,
and whether it reclaims and retries or waits for a new upstream completion API.
It must also explain why cleanup failure after authoritative deletion is not
reported as if the branch still existed. Add failpoints at both create phases
and between delete authority removal and directory cleanup.
> 💬 **Verified (2026-07-11):** confirmed verbatim in the pinned source —
> `dataset.rs:488` documents "This is a two-phase operation… These two phases
> are not atomic. We consider `BranchContents` as the source of truth… may
> leave a zombie branch dataset", including the zombie-blocks-retry
> consequence and the cleanup duties this finding transcribes. The 2026-07-11
> RFC-022 §7 revision (single-writer-process boundary + conditional-ref
> upstream ask) addressed the *conditional-put* gap but not this crash
> classification — the two dispositions compose; both are needed.
### BLOCKER-02 — create-if-absent ownership is not a fencing lease
**Affected:** RFC-023 migration claim, RFC-024 migration claim, and
[RFC-025 §2.3](../rfcs/rfc-025-checkpoint-retention.md#23-cross-process-retention-claim)
`PutMode::Create` correctly elects one initial owner. A random token in that
object does not by itself fence a paused owner, because Lance tag, branch,
migration, and cleanup effects do not condition their commits on that token.
RFC-025 proposes a check-then-delete release; RFC-023 does not yet specify a
release protocol at all. For the check-then-delete shape, the race is:
1. old owner reads and verifies token A;
2. old owner pauses;
3. takeover removes A and creates token B;
4. old owner resumes and deletes B, or resumes a destructive physical effect.
“The stale process checks again” does not close the window after its final
check. Calling the random owner token a fencing token overstates the substrate
guarantee.
**Required disposition:** choose one honest contract:
- no takeover while the prior process can resume; operator recovery proves the
process/host is dead before removing the claim;
- a substrate-enforced lease/fencing primitive whose monotonically newer token
is checked by every protected effect, plus conditional compare-and-delete;
- or a non-takeoverable claim that requires explicit offline repair.
The `StorageAdapter` does not currently expose conditional delete, and its local
`write_text_if_match` is not a cross-process CAS. Any selected design must work
on local and S3 or explicitly narrow the supported backend/topology. For a
substrate-enforced lease, tests pause an owner after its last token check,
perform takeover, then resume it; the old owner must be unable to mutate state
or release the new claim. A host-death or non-takeoverable design instead proves
that takeover is refused until external fencing/death proof is established.
> 💬 **Concur (2026-07-11):** the paused-owner race is real and the "fencing
> token" label overstated the guarantee. Given the substrate as it stands (no
> conditional delete on the adapter; the documented local `write_text_if_match`
> gap), the only honest near-term contract is the boring one — no takeover
> while the prior process can resume; operator proves process/host death or
> uses explicit offline repair. The substrate-enforced lease is the right end
> state but should be written as an upstream/adapter work item, not assumed.
### BLOCKER-03 — key-conflict retry must first resolve partial effects
**Affected:** [RFC-022 §9](../rfcs/rfc-022-unified-write-path.md#9-concurrency-and-retry-semantics),
[RFC-023 §6](../rfcs/rfc-023-key-conflict-fencing.md#6-retry-and-validation-semantics),
and [RFC-026 §6](../rfcs/rfc-026-memwal-streaming-ingest.md#6-fold-protocol)
A retryable concurrent inserted-row-filter conflict is detected when a table
transaction attempts to commit. (`WhenMatched::Fail` may instead report a
pre-existing match during merge execution.) In a multi-table graph write or
MemWAL fold, table A may already have advanced under the armed sidecar before
table B reports the concurrent key conflict. Immediately “restart the entire
logical attempt” would replan around unresolved physical state, which RFC-022
forbids.
**Required disposition:** distinguish:
- a conflict before any physical table commit — finalize the already-durable
empty sidecar, then perform a safe full semantic restart;
- a conflict after any physical commit — keep the sidecar and resolve the
RFC-022 recovery outcome first. Normally the partial attempt must roll back
before a new semantic attempt can be prepared. If rollback is not safe, return
a typed recovery-required failure rather than retrying around it.
For strict insert, finalize an empty intent or retain a partial-effect sidecar,
as applicable, and return the terminal `KeyConflict`; RFC-022 permits partial
recovery to finish at the next synchronous barrier. Do not start another
attempt around that sidecar. For non-strict upsert, the barrier must resolve the
sidecar before the bounded automatic whole-operation retry. Add a failpoint/race
in which table N conflicts after table 1 has committed.
### BLOCKER-04 — live graph branches need physical GC protection
**Affected:** [RFC-025 §6](../rfcs/rfc-025-checkpoint-retention.md#6-cleanup-protocol-and-pruned-through-boundary)
The cleanup root set includes live graph branches, but RFC-025 creates Lance
tags only for named checkpoints. This is insufficient for lazy graph branches.
A graph branch may store, in its `__manifest`, a foreign reference to an old
version on a data table's `main` branch. Lance cleanup on that data table cannot
see the foreign OmniGraph row; it protects versions through its own branch and
tag references.
**Required disposition:** for every live graph-branch table reference, either:
- materialize a deterministic Lance tag/native ref that physically protects the
exact `(table branch, version)`; or
- choose a per-dataset cleanup cutoff no newer than the oldest live reference,
accepting that all intervening versions remain retained.
Checkpoint tags alone do not solve this. Add a test where a lazy branch pins an
old main-table version, main advances beyond the retention window, cleanup
runs, and the lazy branch still opens and reads the exact pinned state.
> 💬 **Escalation (2026-07-11):** this is very likely a live bug in the shipped
> `cleanup`, independent of RFC-025. Verified in code: `cleanup_all_tables`
> (`optimize.rs:802`) runs Lance `cleanup_old_versions` per data table with no
> lazy-branch pin logic, and a lazy graph branch creates **no Lance-native ref
> on the data table** until its first write to that table — its pin exists only
> as a foreign `__manifest` row Lance cannot see. Repro shape: create a branch,
> never write table X on it, advance main past the keep window, `cleanup
> --keep N` → the branch's pinned version of X is collected and the branch
> breaks. Per the repo's test-first rule this deserves a red regression test
> and an issue *now*; RFC-025's disposition should then build on that fix
> rather than owning the discovery. (Snapshot/time-travel pins share the
> mechanism but are history-trimming under an operator-confirmed policy; a
> live branch's working state is not history.)
### BLOCKER-05 — durable-head OCC must compare the full head token
**Affected:** [RFC-024 §5](../rfcs/rfc-024-durable-table-heads.md#5-atomic-publisher-contract)
The RFC says expected table versions are compared against table heads. Lance
version numbers can repeat across recreated datasets/branches, so version-only
comparison admits drop/recreate ABA.
**Required disposition:** define the OCC token as the complete logical head
identity, at least:
```text
(state, stable_table_id, incarnation_id, table_path,
table_branch, physical_ref_incarnation, table_version, schema_hash)
```
The publisher may encode that token differently, but retry/revalidation must
not reduce it to `table_version`. `physical_ref_incarnation` means an e_tag when
the backend supplies one, or another proven token that changes when a dataset
or native ref is deleted and recreated at the same path/branch/version. This is
distinct from the logical `incarnation_id`, which RFC-024 deliberately preserves
across an owner handoff. Add stale-writer races for both logical drop/recreate
and physical ref recreation at the same numeric version.
### BLOCKER-06 — MemWAL needs a capability/format activation barrier
**Affected:** [RFC-026 §3](../rfcs/rfc-026-memwal-streaming-ingest.md#3-enrollment-is-a-recoverable-inline-commit),
[§5](../rfcs/rfc-026-memwal-streaming-ingest.md#5-ack-path-validation-and-writer-lifecycle),
[§6](../rfcs/rfc-026-memwal-streaming-ingest.md#6-fold-protocol),
[§8](../rfcs/rfc-026-memwal-streaming-ingest.md#8-epoch-fenced-quiescence-barrier),
and [§12](../rfcs/rfc-026-memwal-streaming-ingest.md#12-phasing)
Enrollment creates persistent MemWAL metadata and `stream_state` changes the
correctness preconditions for schema, branch, maintenance, and data operations.
An older binary that understands key fencing but not stream lifecycle can ignore
`OPEN | DRAINING | SEALED`, mutate the base table, or perform a schema/branch
operation without draining acknowledged rows.
**Required disposition:** define:
- a graph capability/internal-format stamp written only after every enrolled
table and lifecycle authority is valid;
- old-binary/new-format and new-binary/partial-format refusal behavior;
- migration and rollback/roll-forward ordering;
- preservation rules for later heads/retention formats;
- genuine cross-version tests, not a stamp-rewind simulation.
Replica scope must also match RFC-023's recovery support boundary. Multiple
replicas may route acknowledgement traffic to one shard owner, but enrollment,
fold, and sidecar recovery remain single-writer-process operations until
foreign-process sidecar ownership is fenced. Do not advertise general replica
failover for those operations merely because MemWAL has a shard epoch.
### BLOCKER-07 — stable identity ownership contradicts sibling dependencies
**Affected:** [RFC-024 §3.1](../rfcs/rfc-024-durable-table-heads.md#31-identity-and-object-key),
[RFC-025 §2.1](../rfcs/rfc-025-checkpoint-retention.md#21-logical-authority-manifest-rows),
[RFC-026 §7](../rfcs/rfc-026-memwal-streaming-ingest.md#7-fold-time-rejection-is-atomic),
and [RFC-026 §8](../rfcs/rfc-026-memwal-streaming-ingest.md#8-epoch-fenced-quiescence-barrier)
RFC-024 now says it exclusively owns stable table identity and incarnation and
that identity-dependent sibling claims must wait for it. RFC-025 still calls
RFC-024 optional, and RFC-026 persists `stable-table-id` without declaring the
dependency.
**Required disposition:** extract the identity/incarnation format and migration
as a shared prerequisite RFC, or accept RFC-024 in full before RFC-025 and
RFC-026. Durable-head implementation can remain a later phase only after its
owning RFC and format contract are accepted.
The `_ingest_rejects` deterministic key must use stable table ID plus
incarnation rather than mutable `table_key`, or rename/recreate can change reject
identity and break replay idempotence.
> 💬 **Concur; supersedes a 2026-07-11 correction (2026-07-11):** the earlier
> pass placed identity ownership *inside* RFC-024 §3.1, which created exactly
> the contradiction described here (RFC-025 calls heads optional while
> consuming the identity RFC-024 claims to own). The extracted
> identity/incarnation RFC is the cleaner shape and should replace that
> ownership note. The `_ingest_rejects` key observation is an additional catch
> the earlier pass missed — endorsed.
### BLOCKER-08 — retention activation cannot precede migration selection
**Affected:** [RFC-025 §8](../rfcs/rfc-025-checkpoint-retention.md#8-migration-and-compatibility)
and [§11](../rfcs/rfc-025-checkpoint-retention.md#11-phasing)
RFC-025 leaves in-place migration versus export/import undecided. Its phase
table nevertheless puts “format activation” in Phase A and “selected migration
path” in Phase D. Activation cannot be implemented or reviewed before the
upgrade contract is known.
**Required disposition:** select the migration before Phase A, then specify
quiescence, partial-state refusal, main-stamp ordering, branch coverage, crash
recovery, and later capability preservation. If export/import is selected,
state the irreversible loss of branches, commit DAG, snapshots, and historical
checkpoints as part of the acceptance decision.
### BLOCKER-09 — decide whether these are public or internal RFCs
**Affected:** [public RFC process](../rfcs/README.md), RFC-022 through RFC-027
The files currently live in the public RFC track but use internal-style
`rfc-022-*` names, noncanonical `draft`/`research-blocked` statuses, and empty
owner metadata. In the public process, merging an RFC is acceptance; a merged
public RFC cannot simultaneously retain undispositioned acceptance blockers.
**Required disposition:** choose the track before merge:
- for the public track, rename to `NNNN-title.md`, use `Status: Proposed`, fill
the template's author/discussion/implementation metadata, and resolve every
blocker before the RFC PR merges; or
- move the proposals to `docs/dev/` and follow the maintainer-internal process,
leaving the public RFC directory for externally authorable accepted records.
RFC-027 may state “research-blocked” prominently as its technical state while
retaining the public lifecycle status `Proposed`.
> 💬 **Concur (2026-07-11):** same recommendation as the original family
> review — these are maintainer-internal proposals mid-revision; `docs/dev/`
> with the internal process is the low-friction disposition, keeping
> `docs/rfcs/` for its defined merge-equals-acceptance lifecycle. Whichever
> track is chosen, decide it before any of the set merges, since it defines
> what "dispositioned before acceptance" means for every other finding here.
## Protocol clarifications
### BLOCKER-10 — do not put foreign-branch facts in an atomic `ReadSet`
RFC-022 requires every `ReadSet` member to be arbitrated atomically by the
publish CAS. A CAS on reserved main cannot arbitrate a row on a named source
branch; a merge-target CAS cannot arbitrate a source-branch row.
Use the right category for each fact:
- checkpoint creation captures an immutable source version, revalidates it
before tagging, and then relies on the physical tag. A later source-head
advance is intentionally harmless, so the source head is an effect
precondition, not target-CAS authority;
- offline cleanup facts are protected by the selected fleet/retention barrier,
not by pretending one main-branch CAS covers every named branch;
- branch merge should define captured-source-commit semantics. If “latest
source at target publish” is required instead, it needs a real source-branch
fence held through the target CAS.
Keep target-branch values that must remain stable in the atomic `ReadSet`.
### BLOCKER-11 — RFC-022 and no-heads MemWAL need coarse OCC
**Affected:** [RFC-022 §10](../rfcs/rfc-022-unified-write-path.md#10-rollout-and-compatibility)
and [RFC-026 §6](../rfcs/rfc-026-memwal-streaming-ingest.md#6-fold-protocol)
[RFC-022's rollout](../rfcs/rfc-022-unified-write-path.md#10-rollout-and-compatibility)
currently says mutation/load read-set arbitration is likely blocked on
RFC-024 because probed-but-untouched tables lack mutable head rows.
For graph-content writes, `(branch incarnation, optional graph head)` is already
a conservative branch-wide authority token. An established branch changes its
`graph_head:<branch>` row on every graph commit; a fresh branch may initially
have no branch-specific head row, so absence plus incarnation is part of the
captured token. Schema identity needs its own atomically contended authority.
Any concurrent logical graph change forces full revalidation. RFC-024 table
heads can later reduce false contention by narrowing that token to the tables
actually read.
Today's publisher retries head-row contention by re-reading the live head and
reparenting the prepared write. That is insufficient for a
validation-sensitive plan. On every publisher retry, it must compare the
captured token and return control to full revalidation rather than reparenting
and continuing with stale validation.
The lower-liability rollout is therefore:
1. ship correct coarse OCC with `graph_head` plus schema identity;
2. introduce table-head tokens only after RFC-024 passes its independent format
and cost gates.
The no-heads RFC-026 fold path must carry the same captured branch token in its
`ReadSet`. Do not recouple RFC-022 or RFC-026 correctness to RFC-024
performance.
> 💬 **Concur; supersedes a 2026-07-11 correction (2026-07-11):** the premise
> is independently verified — `graph_head:<branch>` is one mutable row updated
> in every graph-content publish (the deliberate contention point pinned by
> the concurrent-disjoint-writers tests), so the coarse token genuinely
> arbitrates probed-but-untouched same-branch tables. This is a better design
> than the earlier RFC-022 §10 wording ("likely blocked on RFC-024"), which
> should be revised to the two-step rollout described here. One cost worth
> stating in the revision: coarse OCC makes *any* concurrent commit on a busy
> branch force full revalidation of in-flight writers — real throughput cost
> on agent-fleet branches, and precisely the honest motivation for RFC-024's
> later narrowing (rather than a correctness argument for it).
> **Implementation disposition (2026-07-11):** mutation/load now capture the
> native Lance `BranchIdentifier`, exact optional `graph_head`, and accepted
> schema identity; revalidate under a branch-then-table gate; and pass the same
> token to every publisher retry and schema-v3 recovery decision. Metadata-only
> schema-apply tests pin the required invariant that supported schema changes
> move `graph_head` even when no data-table version changes. This closes the
> coarse mutation/load cell without claiming a general schema-authority row or
> multi-process native-ref fencing. RFC-024 remains the false-contention
> narrowing step.
### TIGHTENING-01 — symmetric Lance conflicts are not obviously an activation gate
RFC-023 correctly identifies Lance's directional filtered/unfiltered behavior.
However, its own mandatory fleet outage, recovery drain, historical-duplicate
validation, stamp-last activation, and old-binary refusal are intended to make
unfiltered writers unreachable after activation.
The current RFC-023 routing table still permits a direct existing-row `Update`,
whose Lance operation has no inserted-row filter. Therefore unfiltered
transactions do remain reachable after activation. Symmetry can be demoted only
if every insertion-bearing keyed path routes through filtered merge-insert and
the RFC proves a direct update cannot insert a row or change `id`; affected-row
conflict metadata must continue to cover updates to the same existing row.
After that narrowing, upstream symmetry is useful defense in depth and a
valuable Lance surface guard, but it need not block activation for transaction
pairs that cannot violate key insertion. Two old unfiltered writers are not
protected by symmetry in either case.
> 💬 **Concur, with the deciding check named (2026-07-11):** the concrete
> unfiltered-`Update` reachable after activation is the matched-only
> partial-schema update merge (`WhenMatched::UpdateAll` +
> `WhenNotMatched::DoNothing`, the field-level-updates path in PR #342): it
> classifies zero inserts, so whether it emits an **empty** filter (`Some`
> with no keys — symmetric-safe) or **no** filter (`None` — the reachable
> unfiltered transaction) is exactly what decides whether this demotion is
> sound. That is a one-test question against the pinned revision and should be
> pinned as a surface guard before the demotion is accepted.
## Specification and acceptance tightening
These are smaller than the blockers above, but should be resolved before the
RFC set is merged:
1. **Checkpoint-name normalization:** define allowed bytes, Unicode
normalization, case sensitivity, maximum encoded length, reserved prefixes,
and normalization-version compatibility. Test collisions and reuse across
versions.
2. **RFC lifecycle values:** [`docs/rfcs/README.md`](../rfcs/README.md#status-values)
defines `Proposed`, `Accepted`, `Declined`, `Superseded by NNNN`, and
`Implemented`. Either use those values or amend the process before using
`draft` and `research-blocked` as machine-readable statuses. Research-blocked
can remain prominent in RFC-027's body while its lifecycle status is
`Proposed`.
3. **Provisional version naming:** sibling RFCs should say “RFC-024 heads
format” rather than treating `v5` as permanent while RFC-024 says the numeral
will change if another format lands first.
4. **Capability ordering:** RFC-025 must specify retention-first, heads-first,
and co-release preservation rules. RFC-026 is a conditional dependency when
streams are enrolled because cleanup must persistently quiesce them.
5. **Memory measurement:** `helpers::cost` measures I/O, not peak RSS. RFC-023's
adopted-merge and RFC-027's lineage memory gates should use the subprocess
`scenarios.rs` harness or an equivalent `wait4`/`ru_maxrss` instrument and
name dataset sizes, baseline, cap, and pass threshold.
> 💬 **Concur; catches a gap in a 2026-07-11 correction:** the RFC-023
> §11.4 memory gate added that day states the bound ("peak memory bounded by
> batch size") without naming an instrument that can measure it — this item
> is what makes that gate enforceable rather than aspirational.
6. **Derived-index fallbacks:** acceptance tests must force index-absent and
partially covered states for RFC-024 head lookup and RFC-027 candidate
discovery, assert identical logical results, and assert the promised
degraded-cost/fallback telemetry.
## Acceptance order after disposition
The review does not require all RFCs to land together. A safe order is:
1. accept RFC-022 after branch-control recovery, coarse `graph_head` OCC, and
foreign-branch fact classification are explicit;
2. accept and land the stable identity RFC/capability;
3. accept RFC-023 once partial-effect retry and its format/fleet barrier are
complete;
4. evaluate RFC-024 independently on its physical lookup cost gate;
5. accept RFC-025 after physical protection of both checkpoints and live graph
branches plus a selected upgrade path;
6. accept RFC-026 after its capability barrier and writer-ownership scope are
explicit;
7. keep RFC-027 in research until deletion-delta discovery passes its stated
correctness and flat-cost gates.
This ordering preserves the split's main benefit: a blocked performance
optimization or research result does not hold correctness work hostage.
> 💬 **Concur with the order; two sequencing notes (2026-07-11):**
> BLOCKER-04's escalation means step 5 has a prerequisite outside this list —
> the live lazy-branch cleanup exposure needs its red regression test and fix
> in the shipped `cleanup` first, so RFC-025 builds on a correct baseline. And
> step 3's TIGHTENING-01 demotion hinges on the partial-update filter check
> noted there; if that check lands `None`, upstream symmetry returns to the
> activation gate and RFC-023's timeline moves accordingly.

View file

@ -22,7 +22,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
| `branching.rs` | Branch create / list / delete, lazy fork |
| `merge_truth_table.rs` | Merge-pair truth table (MR-786): all 9×9 `(left_op, right_op)` cells from `{noop, addNode, removeNode, addEdge, removeEdge, setProperty, dropProperty, addLabel, removeLabel}`. Adding a new op to `OpVariant` forces a compile error in `build_case` until the new row + column are dispositioned. 36 executable cells run through real `branch_merge` with a structured oracle (`MergeOutcome` / `MergeConflictKind` + graph-state assert); 45 cells involving `dropProperty`/`addLabel`/`removeLabel` are recorded as `Unsupported` until the mutation grammar grows. |
| `merge_fast_forward.rs` | Fast-forward branch-merge cost + correctness: an append-only adopted-source merge routes *new* rows through `stage_append` instead of one whole-delta `stage_merge_insert` (the full-outer join that exhausted the DataFusion memory pool on embedding-bearing tables). The regression gate is structural — it asserts WHICH staged-write primitive the merge invokes via the task-local write probes (`omnigraph::instrumentation`), not a brittle size threshold; also: merge yields source state, defers vector indexes to the reconciler, streams blob columns |
| `writes.rs` | Direct-publish writes: cancellation, non-strict insert/merge rebase under the per-table queue, strict stale-write conflicts, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery); the lance#7444 row-id-overlap regression (`filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent` — merge-load → same-key merge-load → delete → keyed point lookup, green only under the vendored lance-table patch — plus its append-only control) |
| `writes.rs` | Direct-publish writes: cancellation, RFC-022 non-strict full-attempt reprepare from fresh branch authority, strict stale-write conflicts, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery); the lance#7444 row-id-overlap regression (`filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent` — merge-load → same-key merge-load → delete → keyed point lookup, green only under the vendored lance-table patch — plus its append-only control) |
| `staged_writes.rs` | TableStore staged-write primitives (`stage_append`, `stage_merge_insert`, `commit_staged`, `scan_with_staged`, `count_rows_with_staged`) — primitive-level only; engine code uses the in-memory `MutationStaging` accumulator instead |
| `forbidden_apis.rs` | Defense-in-depth source-walk guard: engine code (`exec/`, `db/omnigraph/`, `loader/`, `changes/`) must not reach around the sealed storage trait to Lance inline-commit APIs, nor open datasets directly (`Dataset::open` / `DatasetBuilder::from_uri`/`from_namespace`) — reads route through `Snapshot::open` and the held-handle cache; `// forbidden-api-allow: <reason>` sentinel exempts reviewed lines |
| `failpoint_names_guard.rs` | Source-walk guard (same defense-in-depth shape as `forbidden_apis.rs`): every failpoint call site (`maybe_fail`, `ScopedFailPoint::new`/`with_callback`, `Rendezvous::park_first`) must reference a `failpoints::names` const, never a bare string literal — a typo'd literal compiles and silently never fires |
@ -36,7 +36,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
| `changes.rs` | `diff_between` / `diff_commits` |
| `consistency.rs` | Cross-table snapshot isolation, atomic publish |
| `lineage_projection.rs` | RFC-013 Phase 7 acceptance gate: graph lineage lives ONLY in `__manifest` — over a realistic history (main commits, a branch, a merge, actors), the production `CommitGraph::open` projection reconstructs the full DAG (commit set, parents, merge parents + merge actor, per-branch heads, inline actors) from the `graph_commit`/`graph_head` rows, and the `_graph_commits.lance` / `_graph_commit_actors.lance` dataset directories are never created at all |
| `schema_apply.rs` | Migration plan + apply, schema-apply lock; index materialization deferred to the reconciler (iss-848): `apply_schema_defers_vector_index_on_empty_table` (an empty-table Vector `@index` never aborts the apply) and `index_only_constraint_apply_touches_no_table_data` (adding an `@index` is metadata-only — no table-version bump); enum widening (iss-enum-widening-migration): `enum_widening_apply_is_metadata_only_and_accepts_new_variant` (no table-version bump; new variant accepted, out-of-set still rejected) + `enum_narrowing_apply_is_refused` (OG-MF-106 with the graph left writable). The planner's widening/narrowing matrix lives in `schema_plan.rs`'s in-source tests |
| `schema_apply.rs` | Migration plan + apply, schema-apply lock; schema-contract publication is pinned by `read_only_open_holds_schema_gate_through_catalog_capture` and `refresh_holds_schema_gate_through_catalog_publication` (source, accepted IR/state, and compiled catalog are captured under one root schema gate). `long_lived_handle_uses_the_schema_catalog_bound_to_its_write_token` covers mutation/load plus a post-apply new node type merged through the pre-apply handle; `stale_handle_branch_delete_gates_tables_added_by_schema_apply` parks delete over that new type while a legacy index reconciler waits, proving merge planning and native-control table envelopes use an operation-local accepted catalog rather than stale ArcSwap state. Index materialization is deferred to the reconciler (iss-848): `apply_schema_defers_vector_index_on_empty_table` (an empty-table Vector `@index` never aborts the apply) and `index_only_constraint_apply_touches_no_table_data` (adding an `@index` is metadata-only — no table-version bump); enum widening (iss-enum-widening-migration): `enum_widening_apply_is_metadata_only_and_accepts_new_variant` (no table-version bump; new variant accepted, out-of-set still rejected) + `enum_narrowing_apply_is_refused` (OG-MF-106 with the graph left writable). The planner's widening/narrowing matrix lives in `schema_plan.rs`'s in-source tests |
| `search.rs` | FTS / vector / hybrid (`bm25`, `nearest`, `rrf`) |
| `scalar_indexes.rs` | Per-property index dispatch of `build_indices_on_dataset_for_catalog`: enums + orderable scalars get a BTREE (so `=`/range/IN/IS NULL are index-accelerated), free-text Strings keep FTS — observed via `TableStore::key_column_index_coverage`, the same helper the traversal chooser uses |
| `traversal.rs` | `Expand`, variable-length hops, anti-join, undirected traversal (`$a <edge> $b`, `Direction::Both` — out in with set-semantics dedup, both-direction anti-join) (CSR path — `OMNIGRAPH_TRAVERSAL_MODE` unset) |
@ -51,10 +51,10 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
| `validators.rs` | Schema constraint enforcement (enum, range, unique, cardinality) across JSONL load, mutation insert/update. ALL THREE write surfaces — mutation, bulk load, AND merge — route through the unified `crate::validate` evaluator (Δ-scoped, index-backed, reusing these leaf checks). Cross-version-uniqueness closure: `cross_version_unique_rejected_on_mutation_insert` + `reinsert_existing_key_is_upsert_not_unique_violation` (mutation path); `cross_version_unique_rejected_on_append_load` + `merge_load_reupsert_existing_key_is_not_unique_violation` (load path). Per-table `Overwrite`: `overwrite_load_validates_ri_against_new_image` (an edges-only overwrite still resolves RI against retained committed nodes) + `append_load_rejects_orphan_edge`. The evaluator's own unit tests live in `src/validate.rs` (`#[cfg(test)]`); its merge-conflict equivalence is pinned by `merge_truth_table.rs` (OrphanEdge) + `branching.rs` (Unique/Cardinality merge tests). Intra-batch duplicate-`@key` rejection on every load mode is pinned by `consistency.rs::loader_rejects_intra_batch_duplicate_keys`; the mutation-coalesce counterpart (insert+update / chained updates of one id are NOT a self-collision) by `writes.rs`. Non-String `@unique` columns probe committed state with a TYPED literal (not a stringified key): `cross_version_unique_rejected_on_date_column` + `noncolliding_write_to_date_unique_column_succeeds` (a `Date @unique` collision is a proper `@unique` violation, and a distinct value does not raise a Date32-vs-Utf8 coercion error). Cardinality is keyed by edge id, last-wins (matching commit's `dedupe_merge_batches_by_id`): `merge_load_edge_src_move_rechecks_vacated_src_cardinality` (a Merge-load moving an edge recounts the vacated src for `@card` min) + `merge_load_duplicate_edge_id_counts_once_per_card` (a dup edge id under two srcs in one batch counts once, no spurious max violation). Direct deletes capture the ids they remove (from the delete op's own scan) into the change-set's `deleted_ids`, so a delete emptying a src is validated: `mutation_delete_edge_below_card_min_rejected` (a `delete Edge` dropping a src below `@card` min is rejected, not silently committed). |
| `merge_cost.rs` | Cost-budget tests for branch MERGE on the shared `helpers::cost` harness: `merge_validation_is_delta_scoped` (a 1-row-delta merge opens ≤3 data tables — Δ-scoped, not the whole catalog; was ~6 pre-#5) and `merge_manifest_cost_grows_with_history` (the cross-branch `__manifest` open amplification still grows with commit depth — a separate, not-yet-addressed term — while validation `data_open_count` stays flat) |
| `policy_engine_chassis.rs` | Engine-layer Cedar enforcement (MR-722): allow + deny through every `_as` writer via the SDK directly — no HTTP — proving embedded and CLI callers hit the same gate as the server, with action × scope shapes matching `authorize_request` |
| `maintenance.rs` | `optimize` (compaction), `repair` (explicit uncovered-drift publish), and `cleanup` (version GC): empty/idempotent/no-op edges, policy validation, head preservation; `optimize` publishes its own compaction (`optimize_publishes_compaction_to_manifest_so_schema_apply_succeeds`), skips pre-existing uncovered drift (`optimize_skips_preexisting_manifest_head_drift`), and refuses to run while a `__recovery` sidecar is pending (`optimize_defers_when_recovery_sidecar_is_pending`); `repair` previews/heals verified maintenance drift, refuses raw semantic drift without `--force`, and forced repair publishes only by explicit operator choice; the index reconciler (iss-848): `index_build_tolerates_null_vector_rows` (an untrainable Vector column defers instead of aborting the build, sibling indexes still build) and `optimize_materializes_index_declared_but_unbuilt` (optimize creates a declared-but-deferred index) |
| `failpoints.rs` | Failure-injection coverage (gated on `failpoints` feature). Includes the five per-writer Phase B → recovery integration tests (`recovery_rolls_forward_after_finalize_publisher_failure`, `schema_apply_phase_b_failure_recovered_on_next_open`, `branch_merge_phase_b_failure_recovered_on_next_open`, `ensure_indices_phase_b_failure_recovered_on_next_open`, `optimize_phase_b_failure_recovered_on_next_open`) and the write-entry in-process heal contract (the four `*_after_finalize_publisher_failure_heals_without_reopen` tests — load, mutation, schema apply, branch merge: a follow-up write on the same handle rolls a sidecar-covered residual forward without reopen/refresh) and the storage-fault matrix for the sidecar lifecycle (`recovery.sidecar_{write,delete,list}` / `recovery.record_audit` failpoints: Phase A put failure aborts with zero drift, Phase D delete failure is swallowed and healed by the next write, list failures are loud at heal and open, audit-append failures are retried to exactly one audit row; plus the bucket-gated `s3_load_recovers_after_publisher_failure_without_reopen`). And the convergence-idempotent roll-forward regression (`open_sweep_roll_forward_converges_when_manifest_advances_concurrently`: two concurrent open-sweeps race one sidecar at the `recovery.before_roll_forward_publish` rendezvous; the CAS loser must converge, not fail the open — iss-schema-apply-reopen-recovery-race). |
| `maintenance.rs` | `optimize` (compaction), `repair` (explicit uncovered-drift publish), and `cleanup` (version GC): empty/idempotent/no-op edges, policy validation, head preservation; `optimize` publishes its own compaction (`optimize_publishes_compaction_to_manifest_so_schema_apply_succeeds`), skips pre-existing uncovered drift (`optimize_skips_preexisting_manifest_head_drift`), refuses to run while a `__recovery` sidecar is pending (`optimize_defers_when_recovery_sidecar_is_pending`), and compacts blob-v2 tables through the normal path (`optimize_compacts_blob_table_alongside_plain_table`); `repair` previews/heals verified maintenance drift, refuses raw semantic drift without `--force`, and forced repair publishes only by explicit operator choice; the index reconciler (iss-848): `index_build_tolerates_null_vector_rows` (logical load succeeds without inline index work; an untrainable Vector column then defers during reconciliation) and `optimize_materializes_index_declared_but_unbuilt` (optimize creates a declared-but-deferred index) |
| `failpoints.rs` | Failure-injection coverage (gated on `failpoints` feature). RFC-022 includes deterministic post-stage/pre-effect races for mutation/load uniqueness and strict disjoint-head changes, plus the cross-handle post-effect `RecoveryRequired` → read-write-open rollback cell. Control/recovery race cells include `first_touch_post_create_open_error_keeps_recovery_ownership`, `branch_delete_orphans_sidecar_armed_after_initial_barrier`, `branch_merge_fences_target_delete_recreate_aba`, `branch_merge_fences_concurrent_sync_on_same_handle`, `branch_merge_rejects_fresh_target_manifest_change_before_effects`, `branch_merge_rechecks_late_sidecar_after_table_gates`, `cleanup_rechecks_sidecars_under_gc_gates`, and `full_recovery_rereads_sidecar_body_after_discovery`. The suite also includes the five per-writer effect → manifest-CAS recovery tests, write-entry in-process heal contract, storage-fault matrix, S3 recovery twin, and convergence-idempotent roll-forward regression. |
| `failpoint_names_guard.rs` | Source-walk guard: every `maybe_fail(...)` call site (engine + cluster) must reference the compile-checked `failpoints::names` consts, never a bare string literal — a typo'd literal compiles but silently never fires; same defense-in-depth shape as `forbidden_apis.rs` |
| `recovery.rs` | Open-time recovery sweep — sidecar I/O, classifier dispatch (NoMovement / RolledPastExpected / UnexpectedAtP1 / UnexpectedMultistep / InvariantViolation), all-or-nothing decision, roll-forward via `ManifestBatchPublisher::publish`, roll-back via `Dataset::restore`, audit row in `_graph_commit_recoveries.lance`, `OpenMode::ReadOnly` skip path |
| `recovery.rs` | Open-time recovery sweep — legacy sidecars plus schema-v3 Mutation/Load exact transaction identity, fixed logical/rollback outcome IDs, branch-token comparison, fresh under-gate sidecar reread/reparse, all-or-nothing roll-forward/rollback/refusal, audit row in `_graph_commit_recoveries.lance`, and `OpenMode::ReadOnly` skip path |
| `composite_flow.rs` | Compositional/narrative end-to-end stories — multi-step flows that compose mechanics covered by other test files. Catches integration regressions where individual operations all pass their unit tests but their composition breaks (sequential merges, post-merge main writes, time-travel through merge DAG, reopen consistency over multi-merge histories, post-optimize and post-cleanup strict writes). |
## Fixtures

View file

@ -4,11 +4,14 @@
> removed in MR-771 (shipped v0.4.0). Writes now go directly to the target
> table; this document specifies that direct-publish path.
`mutate_as` and `load` write **directly to the target table**
and call `ManifestBatchPublisher::publish` once at the end with
`expected_table_versions` (the per-table manifest versions captured before
the first write). Cross-table OCC is enforced inside the publisher; the
publisher's row-level CAS on `__manifest` is the single fence.
`mutate_as` and `load` prepare against one immutable branch-authority token,
write **directly to the target table**, and call
`ManifestBatchPublisher::publish` once at the end. The token is
`(Lance branch identifier, exact optional graph_head, accepted schema identity)`;
the exact `graph_head` check protects validation dependencies on tables the
write does not touch. Publisher row-level CAS on `__manifest` is the visibility
fence. Process-local branch/table queues reduce retries but are not distributed
authority.
## What this means in practice
@ -19,15 +22,93 @@ publisher's row-level CAS on `__manifest` is the single fence.
`__run__*` branch on an upgraded graph is swept off `__manifest` by the
v2→v3 internal-schema migration on first read-write open. (The inert
`_graph_runs.lance` bytes remain until a `delete_prefix` primitive lands.)
- Cancelled mutation futures leave **no graph-visible state** — the manifest
is never advanced. They can leave two kinds of unreferenced residue, both
self-healing: orphaned Lance fragments (reclaimed by `omnigraph cleanup`),
and — on the *first* write to a table on a branch, which forks it before the
publish — a manifest-unreferenced branch ref. The next write to that table
reclaims the stale fork and re-forks (`reclaim_orphaned_fork_and_refork`),
and `cleanup`'s per-table reconciler is the guaranteed backstop; see the
- Cancelled mutation futures leave **no graph-visible state** unless the
manifest publish already completed. Before that point they can leave
reclaimable uncommitted Lance files, or sidecar-covered committed table
effects that the next quiesced recovery rolls forward/compensates. A
first-touch named-branch write can also leave a target table ref, but it is
never created without ownership: the schema-v3 sidecar is durable first and
names that `(table_path, target ref)`. Reclaim and `cleanup` treat any
matching pending sidecar as a hard stop. Quiesced full recovery accepts both
crash shapes — sidecar durable with no ref yet, or an exact untouched ref at
the inherited version — and removes the latter before deleting the empty
intent. If several pending intents claim one ref, a no-effect intent discards
only itself while any competitor remains; the last no-effect survivor cleans
an untouched ref, or the effect-owning survivor recovers normally. `cleanup`
remains the backstop for genuinely unclaimed legacy/stale refs; see the
fork-reclaim note in [invariants.md](invariants.md).
## Mutation/load coarse OCC (RFC-022 first adapter)
Mutation and load use a closed prepare → effect → publish attempt:
1. run the branch-aware recovery barrier, then capture the target branch's native Lance
`BranchIdentifier`, exact `graph_head:<branch>` (including absence on a
fresh branch), accepted schema identity, and table snapshot;
2. run the complete validator and prepare every effect outside the effect gate.
Existing-table transactions may stage reclaimable files here. A first-touch
named-branch table retains its batch/predicate and pre-mints the transaction
identity instead: Lance branch-local files cannot be staged until its target
ref exists;
3. acquire the schema gate, branch gate, then sorted table queues; re-check for
a relevant sidecar armed since step 1, then revalidate the token. Any
unresolved relevant intent returns typed `RecoveryRequired` before effects;
4. on a pre-effect mismatch, discard the complete attempt. Append/Insert/Merge
reprepare with a bounded retry; strict Update/Delete/Overwrite return typed
`ReadSetChanged`;
5. arm a schema-v3 recovery sidecar. For each deferred first-touch table, create
its target ref, stage branch-local files on that ref, and bind the staged
transaction to the pre-minted UUID. Then commit every planned transaction
with zero transparent conflict retries, confirm exact transaction UUIDs and
table updates, and publish the pre-minted lineage intent under the same token.
The publisher checks the exact head and native branch identity on every CAS
attempt. It never reparents a validation-sensitive intent after contention. A
mismatch after any physical effect returns `RecoveryRequired` and leaves the
sidecar intact; it is not an ordinary retry loop.
This adapter preserves the documented single-writer-process support boundary.
The native branch identifier detects delete/recreate ABA but is not a Lance
conditional-ref fence, and destructive recovery remains unsafe beside a live
foreign process.
### Branch-merge authority fence (adapter bridge)
Branch merge still uses its writer-specific multi-commit table effects and
confirmation sidecar; it has not yet been converted to the RFC-022 exact-effect
adapter. It does, however, join the closed control boundary needed by this first
slice: after the strict recovery barrier it acquires the root-shared schema gate
and the sorted source/target branch gates, performs the final sidecar check,
loads one operation-local catalog from the accepted contract, captures both graph
heads plus the base/source/target snapshots, and holds those gates through table
effects and manifest publication. Planning stays outside table queues. Before
Phase A, merge acquires the conservative all-catalog table envelope for both
source and target, re-lists sidecars, and compares fresh source/target manifest
versions with the captured snapshots. A stale warm handle catalog or coordinator
snapshot is never accepted as that revalidation.
That fence prevents a same-process target delete/recreate from reusing the branch
name underneath a merge plan. The race test deliberately recreates a target with
the same name and numeric Lance version but a different `BranchIdentifier`, so
version-only checking cannot accidentally satisfy it. This is a process-local
bridge, not a cross-process conditional-ref primitive and not a substitute for
the later full branch-merge read-set/reprepare adapter. `sync_branch` joins the
same root schema gate before replacing a handle's coordinator, so it cannot
overwrite merge's temporary target coordinator or change a native control's
active-branch authority mid-operation.
### Branch-delete orphaning exception
Branch deletion runs the healer first and then holds schema, the target branch,
and every accepted-catalog table gate through the native ref removal. An
unresolved sidecar scoped to that target does not permanently block deletion:
once those gates prove its in-process owner is no longer live, removing the
manifest branch makes its physical effects unreachable. The next write/open
records the orphan-discard recovery audit and deletes the sidecar. A
`SchemaApply` sidecar remains graph-global and blocks deletion. This exception
is specific to removing the authority that made the intent reachable; create,
merge, mutation, and load still reject relevant unresolved ownership.
## Read-your-writes within a multi-statement mutation
A `.gq` query with multiple ops (e.g. `insert Person … insert Knows …`)
@ -47,12 +128,20 @@ shared by both `mutate_as` and the bulk loader:
`TableStore::scan_with_pending`, which scans committed via Lance
and applies the same SQL filter to the pending batches via DataFusion
`MemTable`. Same-query writes are visible to subsequent reads.
- At end-of-query, `MutationStaging::finalize` issues exactly one
`stage_*` + `commit_staged` per touched table (concatenating
accumulated batches; merge-mode dedupes by `id`, last-write-wins),
and the publisher publishes the manifest atomically across all
touched sub-tables. Cross-table conflicts surface as
`ManifestConflictDetails::ExpectedVersionMismatch`.
- Blob-bearing updates use the materializing variant: Lance reads only matched
committed blob payloads as binary, the engine normalizes them back to the
logical blob schema, and the full rows join the same pending-shadow union.
Rewriting matched blob bytes costs I/O proportional to those bytes, but makes
correctness independent of whether a physical index selects a different
Lance merge plan.
- At end-of-query, `MutationStaging::stage_all` prepares exactly one staged
transaction per touched table and `commit_all` commits it (concatenating accumulated
batches; merge-mode dedupes by `id`, last-write-wins), and the publisher
publishes the manifest atomically across all touched sub-tables. Existing
tables stage before gate acquisition; a first-touch named-branch table stages
after sidecar + fork under the gates so its uncommitted files live in the
correct Lance branch tree. Cross-table conflicts surface as typed read-set or
manifest conflicts.
- **Deletes stage too (MR-A).** Lance 7.0's
`DeleteBuilder::execute_uncommitted` (#6658) makes delete a two-phase op,
so deletes no longer inline-commit. Each delete records a predicate in
@ -153,9 +242,9 @@ integrity, and edge cardinality before any Lance HEAD movement, stages
each touched table with Lance `Operation::Overwrite`, then runs
`commit_staged` under the normal `SidecarKind::Load` recovery sidecar
before publishing `__manifest`. `OMNIGRAPH_LOAD_CONCURRENCY` applies to the
fragment-writing stage only; the commit and manifest publish still run
under the per-table write queues. Empty-table overwrite is represented as
a valid zero-fragment Lance `Overwrite` transaction, not as
fragment-writing stage only; the commit and manifest publish run while holding
the root-shared schema → branch → sorted-table gates. Empty-table overwrite is
represented as a valid zero-fragment Lance `Overwrite` transaction, not as
truncate-then-append.
### Open-time recovery sweep
@ -170,8 +259,8 @@ test pins.
A second, narrower drift class — the **finalize → publisher window**
is closed across one open cycle by the open-time recovery sweep:
`MutationStaging::finalize` runs `stage_*` + `commit_staged` per touched
table sequentially, then the publisher commits the manifest. Lance has
`MutationStaging::stage_all` prepares the table transactions and `commit_all`
runs their independent HEAD advances before the publisher commits the manifest. Lance has
no multi-dataset atomic commit, so the per-table `commit_staged` calls
are independent operations: if commit_staged on table N+1 fails *after*
commit_staged on tables 1..N succeeded, or if the publisher's CAS
@ -179,30 +268,37 @@ pre-check rejects *after* every commit_staged succeeded, tables 1..N
are left at `Lance HEAD = manifest_pinned + 1`.
**Recovery protocol** (lifecycle of every staged-write writer —
`MutationStaging::finalize`, `schema_apply::apply_schema_with_lock`,
`MutationStaging::commit_all`, `schema_apply::apply_schema_with_lock`,
`branch_merge_on_current_target`, `ensure_indices_for_branch`,
`optimize_all_tables`):
1. **Phase A**: writer writes a sidecar JSON to
`__recovery/{ulid}.json` BEFORE its first HEAD-advancing commit
`__recovery/{ulid}.json` BEFORE its first independently durable physical
effect (including a first-touch Lance branch ref) or HEAD-advancing commit
(`commit_staged`, or `compact_files` for `optimize_all_tables`,
which advances the Lance HEAD via a reserve-fragments + rewrite
commit rather than a staged write). The
sidecar names every `(table_key, table_path, expected_version,
post_commit_pin)` it intends to commit + the writer kind +
actor_id.
For a first-touch named-branch Mutation/Load table, Phase A is followed by
target-ref creation and branch-local `stage_*`; the sidecar already carries
its pre-minted transaction identity.
2. **Phase B**: writer's per-table `commit_staged` loop runs.
- **Phase-B confirmation (`BranchMerge` only)**: a `BranchMerge` writer
- **Phase-B confirmation:** a `BranchMerge` writer
advances each table's HEAD by *several* commits (append → upsert →
delete), so a bare "HEAD moved" is ambiguous — it could be a complete
publish or one crashed mid-sequence. After the whole per-table loop
finishes, the writer re-writes the sidecar stamping each pin's
`confirmed_version` with the exact achieved version, then proceeds to
Phase C. This is the commit point of the recovery WAL: a crash *after*
confirmation rolls forward to those versions; a crash *during* Phase B
(sidecar still unconfirmed) rolls back. Other writers don't confirm —
their drift is derived state (index coverage, compaction) that a partial
roll-forward never corrupts.
Phase C. Schema-v3 Mutation/Load sidecars also confirm: each table must
match the staged Lance transaction's `(read_version, uuid)`, and the
sidecar records the exact `SubTableUpdate` plus original lineage intent.
This is the commit point of the recovery WAL: a crash *after* confirmation
rolls forward only when the captured branch token still matches; a crash
*during* Phase B (sidecar still unconfirmed) rolls back. Remaining legacy
writers don't confirm — their drift is derived state (index coverage,
compaction) that a partial roll-forward never corrupts.
3. **Phase C**: publisher commits the manifest.
4. **Phase D**: writer deletes the sidecar.
@ -226,6 +322,18 @@ recovery sweep in `crates/omnigraph/src/db/manifest/recovery.rs`:
`BranchMerge` sidecar, a moved HEAD with no `confirmed_version` classifies
as `IncompletePhaseB` (a partial multi-commit publish) and forces roll-back;
with a `confirmed_version`, roll-forward targets exactly that version.
Schema-v3 Mutation/Load additionally requires `EffectsConfirmed`, the exact
Lance transaction identity at the confirmed version, the original immutable
manifest delta, and a matching captured authority token. A changed token is
rollback-only; an unknown/foreign effect is refused rather than adopted.
An Armed first-touch intent with no owned transaction is deferred by live
roll-forward-only healing because another handle may still own it. Quiesced
full recovery tolerates an absent target ref (crash before fork), or removes
an exact unchanged unpublished ref after proving no other pending sidecar
claims it. With competing claims, the current no-effect sidecar discards
itself without touching the ref; the final survivor owns cleanup/recovery.
During partial rollback, no-effect refs are removed before the rollback
outcome is published so a retry cannot strand them.
- If any table is `InvariantViolation` (Lance HEAD < manifest pinned
should be impossible), **abort** with a loud error and leave the
sidecar on disk for operator review.
@ -243,14 +351,14 @@ recovery sweep in `crates/omnigraph/src/db/manifest/recovery.rs`:
each iteration. The audit row's `to_version` records the logical
rolled-back-to version (`manifest_pinned`); the manifest is published at the
restore commit (`manifest_pinned + 1`, same content).
- After a successful roll-forward or roll-back, an audit row is
recorded — the graph commit lineage (the `graph_commit` rows in `__manifest`
since RFC-013 Phase 7) carries a commit tagged
`actor_id = "omnigraph:recovery"`, and a sibling
`_graph_commit_recoveries.lance` row carries `recovery_kind`,
`recovery_for_actor` (the original sidecar's actor), `operation_id`,
per-table outcomes. Operators run `omnigraph commit list --filter
actor=omnigraph:recovery` to find recoveries.
- After a successful roll-forward or roll-back, an internal
`_graph_commit_recoveries.lance` row records `recovery_kind`,
`recovery_for_actor` (the original sidecar's actor), `operation_id`, and
exact per-table outcomes. A v3 roll-forward publishes the interrupted
writer's fixed lineage intent, including its original actor; rollback and
legacy recovery commits use `actor_id = "omnigraph:recovery"`. Ordinary
commit history is therefore not a complete recovery enumeration, and the
CLI currently has no public query for the recovery-audit table.
- Sidecar deleted as the final step.
Triggers for the residual: transient Lance write errors during finalize
@ -264,13 +372,24 @@ contention exceeding `PUBLISHER_RETRY_BUDGET = 5` retries.
Phase B → Phase C residual closes on the next write, without a
restart and without an explicit refresh. The heal lists `__recovery/`
(one `list_dir`; empty in the steady state) and, per sidecar, acquires
the same per-`(table_key, table_branch)` write queues every sidecar
writer holds from before `write_sidecar` until after `delete_sidecar`
so it serializes against a live writer instead of rolling its
schema → branch → sorted-table gates that overlap the writer's guarded
sidecar lifetime. RFC-022 mutation/load writers hold the complete order. Branch
merge now holds schema plus source/target branch authority for its whole attempt
and then the all-catalog source/target table envelope; other legacy adapters
serialize through their existing table or schema gate until their own adapter
slices land. The
manager is shared by every
`Omnigraph` handle for one canonical local root identity (relative, absolute,
and symlink aliases converge; object-store/custom schemes stay opaque), so this
also serializes a refresh or separately-opened handle against a live writer instead of rolling its
in-flight sidecar forward from under it (a sidecar whose queues can be
acquired belongs to a writer that finished or died; an existence
re-check after the wait skips the finished case). Lock order is
queues → coordinator, matching every writer's commit→publish path.
schema → branch → sorted tables → coordinator, matching the writer effect path.
Enrolled mutation/load attempts and branch merge perform one additional
`list_dir` after acquiring their authority gates; that final check closes the
pre-gate recovery TOCTOU without moving mutation/load validation or staged-file
construction under the gate.
Pinned by the four
`tests/failpoints.rs::*_after_finalize_publisher_failure_heals_without_reopen`
tests (load, mutation, schema apply, branch merge). The maintenance
@ -280,8 +399,11 @@ Phase-B commit (dropping its rows), and a branch merge publishes the
drift as an unattributed side effect — both while the stale sidecar
lingers to misclassify later.
Sidecars that would require a `Dataset::restore` (mixed / unexpected
state) are deferred to the next `OpenMode::ReadWrite` open: restore is
unsafe under concurrency because Lance's `check_restore_txn` accepts
state) are deferred to the next `OpenMode::ReadWrite` open. Full open-time
recovery uses the same root-scoped ordered gates and post-wait existence check,
so it cannot Restore/delete under a live writer owned by another handle in the
same process. Restore remains unsafe across processes because Lance's
`check_restore_txn` accepts
the restore against in-flight Append/Update/Delete commits and
silently orphans them (pinned by
`tests/staged_writes.rs::lance_restore_loses_to_concurrent_append_via_orphaning`).
@ -289,17 +411,24 @@ When such a deferred sidecar blocks a write, the commit-time drift
guard says so explicitly ("a pending recovery sidecar requires
rollback — reopen the graph read-write") instead of pointing at
`omnigraph repair`, which refuses while a sidecar is pending.
`cleanup` refuses pending sidecars at entry as well, before orphan reconciliation
or version GC: v3 ownership and compensation recovery may need the retained
Lance transaction/version history, so garbage collection cannot outrun the
recovery barrier.
Continuous in-process recovery for the rollback path is the goal of a
future background reconciler. `ensure_indices` does not heal at entry
itself — it runs inside the load / schema-apply flows after their
entry heal, and its strict preconditions still fail loudly on drift
when invoked directly.
future background reconciler. `ensure_indices` does not heal at entry itself;
it is an explicit maintenance/reconciliation call, separate from mutation,
load, and schema apply, and its strict preconditions fail loudly on drift.
The publisher-CAS contract is unchanged: a *concurrent writer* that
advances any of our touched tables between snapshot capture and
publisher commit produces exactly one winner. The residual above is
about *our* abandoned commits in the failure path, not about
concurrency races.
For enrolled mutation/load, the publisher rechecks the attempt's exact native
branch identity and `graph_head` as well as the touched-table versions. A
concurrent graph commit anywhere on the target branch therefore invalidates the
prepared authority instead of silently reparenting it. Before effects, an
insert-only mutation or Append/Merge load fully reprepares with a bounded retry; strict
Update/Delete/Overwrite returns `ReadSetChanged`; after any effect, any later
error returns `RecoveryRequired` and leaves the fixed v3 intent durable. Legacy
writers still arbitrate only their explicit touched-table expectations until
their adapters are enrolled.
**Sidecar I/O failure semantics** (all sidecar I/O goes through the
backend-generic `StorageAdapter`; the contracts below are pinned by the
@ -341,19 +470,25 @@ error, not a silent `false`.
## Conflict shape
Concurrent writers to the same `(table, branch)` produce exactly one
success and one failure. The losing writer's error is
`OmniError::Manifest` with kind `Conflict` and details
`ManifestConflictDetails::ExpectedVersionMismatch { table_key, expected,
actual }`. The HTTP server maps this to **409 Conflict** with body
`{"error": "...", "code": "conflict", "manifest_conflict": { "table_key":
"...", "expected": N, "actual": M }}` — see [docs/user/server.md](../user/operations/server.md).
For mutation/load, a changed authority detected before effects is
`ManifestConflictDetails::ReadSetChanged { member, expected, actual }`.
Retryable Insert/Merge/Append attempts handle this internally by fully
repreparing; strict writes surface **409 Conflict** with structured
`read_set_conflict` details. A changed authority discovered after a physical
effect, or any unresolved overlapping intent found at the synchronous recovery
barrier, is `OmniError::RecoveryRequired { operation_id, … }`, mapped to **503
Service Unavailable** with structured `recovery_required`; retry only after the
sidecar has been resolved. Legacy, not-yet-enrolled writers may still surface
`ExpectedVersionMismatch` and `manifest_conflict`.
## Audit
## Commit actor history
`actor_id` lands in the graph commit lineage — the `graph_commit` rows in
`__manifest`, written in the publish CAS (RFC-013 Phase 7; previously
`_graph_commits.lance`). Audit history is queried via `omnigraph commit list`.
`_graph_commits.lance`). Ordinary commit/actor history is queried via
`omnigraph commit list`. Crash-recovery actions additionally live in the internal
`_graph_commit_recoveries.lance` table described above; that exact recovery log
does not yet have a public CLI query.
## Storage versioning (no in-place migration)