Implement RFC-022 unified graph write protocol (#343)

* Implement unified graph write protocol

* Preserve recovery error wire compatibility

docs/dev/architecture.md

@@ -170,30 +170,36 @@ Code paths:
 - Graph index: `crates/omnigraph/src/graph_index/`
 - Loader: `Omnigraph::ingest` at `crates/omnigraph/src/loader/mod.rs:74`
 
-### Mutation atomicity — in-memory accumulator (MR-794)
+### Mutation atomicity — in-memory accumulator + branch-wide OCC (MR-794 / RFC-022)
 
-Inserts and updates inside `mutate_as` and the bulk loader's
-Append/Merge modes go through `MutationStaging`
+Mutations inside `mutate_as` and every bulk-load mode go through `MutationStaging`
 ([`crates/omnigraph/src/exec/staging.rs`](../../crates/omnigraph/src/exec/staging.rs)),
 a per-query in-memory accumulator. No Lance HEAD advance happens during
-op execution; one `stage_*` + `commit_staged` per touched table runs
-at end-of-query, then the publisher commits the manifest atomically.
+op execution. The attempt captures one native branch identity, exact graph head,
+accepted schema identity/catalog, and base snapshot. At end-of-query it stages
+one exact Lance transaction per touched table outside the effect gates, then
+acquires the root-shared schema → branch → sorted-table gates and revalidates the
+complete authority before arming recovery or advancing any table HEAD.
 
 ```
 op-1 (insert/update) → push RecordBatch → MutationStaging.pending[table]
 op-2 (insert/update) → read committed via Lance + pending via DataFusion
                        MemTable (read-your-writes) → push batch
 op-N → push batch
-─── end of query ───────────────────────────────────────
-finalize: per pending table:
-   concat batches → stage_append OR stage_merge_insert OR stage_overwrite
-                  → commit_staged
-publisher: ManifestBatchPublisher::publish (one cross-table CAS)
+─── end of query ────────────────────────────────────────────────
+stage_all: concat batches → one exact stage_* transaction per table (no HEAD move)
+commit_all: acquire schema → branch → sorted-table gates
+            → reject a new recovery intent → revalidate the complete branch token
+            → arm schema-v3 recovery → commit_staged per table
+publisher: publish pre-minted lineage under the exact native-branch/head + table precondition
 ```
 
-A failed op leaves Lance HEAD untouched on the staged tables: the next
-mutation proceeds normally with no drift to reconcile. Concrete
-contracts:
+A failed op leaves Lance HEAD untouched on the staged tables. If authority
+changes before effects, insert-only mutations and Append/Merge loads discard the
+whole attempt and fully reprepare with a bounded retry; Update/Delete/Overwrite
+returns `ReadSetChanged`. After any physical effect, any later error returns
+`RecoveryRequired` and leaves the v3 sidecar to converge the fixed outcome.
+Concrete contracts:
 
 - `D₂` parse-time rule: a query is either insert/update-only or
   delete-only. Mixed → reject. Deletes now stage like inserts/updates
@@ -262,7 +268,7 @@ flowchart LR
     rec --> diff --> wp
 ```
 
-Today, indexes are built explicitly via `ensure_indices`. Reads degrade gracefully when index coverage is partial — Lance's scanner unions indexed and scan paths automatically. The roadmap reconciler observes manifest state and converges coverage in the background.
+Today, physical indexes are built explicitly via `ensure_indices`/`optimize`; schema apply, mutation, and load only record intent or publish their exact data effects and never build indexes inline. Reads degrade gracefully when index coverage is missing or partial — Lance's scanner unions indexed and scan paths automatically (vector search falls back to brute force). A future background reconciler may automate the same explicit convergence path.
 
 ### Server / CLI
 
@@ -276,14 +282,14 @@ flowchart LR
     pol[Cedar policy gate<br/>per request]:::l2
     wl[WorkloadController<br/>per-actor admission]:::l2
     eng[engine API<br/>Arc&lt;Omnigraph&gt;]:::l2
-    wq[WriteQueueManager<br/>per-(table, branch)]:::l2
+    wq[WriteQueueManager<br/>root-scoped schema → branch → table gates]:::l2
 
     cli -.-> eng
     srv_in --> auth --> pol --> wl --> eng
     eng --> wq
 ```
 
-The server applies Cedar policy at the HTTP boundary today. The roadmap, called out in [docs/dev/invariants.md](invariants.md) as a known gap, is to push policy into the planner as predicates. After Cedar, mutating handlers go through `WorkloadController` (per-actor admission cap + byte budget; PR 2 / MR-686) before reaching the engine. The engine itself holds an `Arc<WriteQueueManager>` so concurrent mutations on the same `(table, branch)` serialize at the queue, while disjoint keys run in parallel — see [docs/user/server.md](../user/operations/server.md) "Per-actor admission control" and [docs/dev/writes.md](writes.md). The CLI bypasses the HTTP layer (and admission) and calls the engine API directly.
+The server applies Cedar policy at the HTTP boundary today. The roadmap, called out in [docs/dev/invariants.md](invariants.md) as a known gap, is to push policy into the planner as predicates. After Cedar, mutating handlers go through `WorkloadController` (per-actor admission cap + byte budget; PR 2 / MR-686) before reaching the engine. Every handle for one canonical graph root shares an `Arc<WriteQueueManager>` in-process. RFC-022-enrolled mutation/load attempts prepare outside the effect gates, then acquire the exclusive root schema gate, the target branch-effect gate, and sorted `(table, branch)` gates and hold them through publish. The root schema gate means enrolled effect windows on one graph currently serialize even across different branches; readers remain ungated, and pre-effect parsing, validation, and fragment staging can overlap. The branch gate preserves the complete per-branch validation authority, while table gates protect each concrete Lance effect and legacy adapter. These gates prevent same-process races and impose one deadlock-free order; the exact publisher precondition and durable recovery intent remain the correctness authorities, and the gates are not a cross-process lock. See [docs/user/server.md](../user/operations/server.md) "Per-actor admission control" and [docs/dev/writes.md](writes.md). The CLI bypasses the HTTP layer (and admission) and calls the engine API directly.
 
 Code paths:
 
@@ -303,8 +309,8 @@ Throughout the docs, capabilities are split into:
 
 - **MVCC**: every Lance write bumps a per-dataset version; the OmniGraph manifest version coordinates which sub-table versions are visible together.
 - **Snapshot isolation**: a query holds one `Snapshot` for its lifetime; concurrent writes don't leak in.
-- **Cross-branch isolation**: copy-on-write means readers and writers on different branches don't block each other.
-- **Per-query staging**: `mutate_as` and `load` (Append/Merge) accumulate insert/update batches in an in-memory `MutationStaging`; one `stage_*` + `commit_staged` per touched table runs at end-of-query, then the publisher commits the manifest atomically. A mid-query failure leaves Lance HEAD untouched on staged tables. (MR-794; pre-v0.4.0 used a `__run__<id>` staging branch + Run state machine, removed in MR-771.)
+- **Cross-branch isolation**: Lance copy-on-write keeps branch data independent, and readers never acquire write gates. Enrolled writer preparation can overlap across branches, but their effect/publish windows currently share one exclusive root schema gate before the branch/table gates, so those windows serialize in-process even for different branches.
+- **Per-query staging**: `mutate_as` and every `load` mode accumulate their work in an in-memory `MutationStaging`; `stage_all` prepares exact per-table transactions without moving HEAD. At commit, the root schema → branch → sorted-table gates protect complete-token revalidation, v3 recovery arming, zero-retry table effects, and one atomic manifest publish. A mid-query failure leaves Lance HEAD untouched on staged tables. (MR-794 / RFC-022; pre-v0.4.0 used a `__run__<id>` staging branch + Run state machine, removed in MR-771.)
 - **Schema-apply lock**: `__schema_apply_lock__` system branch serializes schema migrations.
 - **Fail-points** (`failpoints` cargo feature): `failpoints::maybe_fail("operation.step")?` in `branch_create`, publish, etc., for deterministic failure injection in tests.
 

docs/dev/execution.md

@@ -79,16 +79,16 @@ Hybrid example: `order { rrf(nearest($d.embedding, $q), bm25($d.body, $q_text))
 
 ## Mutation execution (`exec/mutation.rs`)
 
-Resolves expression values to literals, converts to typed Arrow arrays (`literal_to_typed_array(lit, DataType, num_rows)`), then writes via Lance's two-phase distributed-write API at end-of-query:
+Resolves expression values to literals, converts to typed Arrow arrays (`literal_to_typed_array(lit, DataType, num_rows)`), then writes via Lance's two-phase distributed-write API at end-of-query. Before lowering/execution, one `WriteTxn` captures the target's Lance-native branch identity, exact optional graph head, accepted schema identity/catalog, and base table snapshot; every step in the attempt uses that immutable authority.
 
-- `insert` (no `@key`, edges) → accumulate into `MutationStaging.pending` (Append mode); finalize calls `stage_append` once per touched table.
-- `insert` (`@key` node) → accumulate into `pending` (Merge mode); finalize calls `stage_merge_insert` once per touched table.
+- `insert` (no `@key`, edges) → accumulate into `MutationStaging.pending` (Append mode); `stage_all` later calls `stage_append` once per touched table.
+- `insert` (`@key` node) → accumulate into `pending` (Merge mode); `stage_all` later calls `stage_merge_insert` once per touched table.
 - `update` → scan committed via Lance + pending via DataFusion `MemTable` (read-your-writes), apply assignments, accumulate into `pending` (Merge mode).
-- `delete` → records a predicate into `MutationStaging.delete_predicates` (count matching committed rows now for `affected_*`); finalize combines a table's predicates into one `stage_delete` (Lance 7.0 `DeleteBuilder::execute_uncommitted`, a deletion-vector transaction) committed via `commit_staged` — no inline HEAD advance (MR-A).
+- `delete` → records a predicate into `MutationStaging.delete_predicates` (count matching committed rows now for `affected_*`); `stage_all` combines a table's predicates into one `stage_delete` (Lance 7.0 `DeleteBuilder::execute_uncommitted`, a deletion-vector transaction) — no inline HEAD advance (MR-A).
 
 **D₂ parse-time rule.** A single mutation query is either insert/update-only or delete-only. Mixed → reject before any I/O. The check fires in `enforce_no_mixed_destructive_constructive(&ir)` inside `execute_named_mutation`.
 
-Multi-statement mutations are atomic at the publisher commit boundary: every insert/update batch lives in memory until end-of-query, then exactly one `stage_*` + `commit_staged` runs per touched table, then `ManifestBatchPublisher::publish` commits the manifest atomically with per-table `expected_table_versions` CAS.
+Multi-statement mutations are atomic at the publisher commit boundary. Every batch lives in memory until all statements and validation succeed; `stage_all` then prepares one exact transaction per touched table without advancing HEAD. `commit_all` acquires the root-shared schema → branch → sorted-table gates, rechecks for recovery intent, revalidates the complete branch authority, writes the schema-v3 recovery sidecar, and commits the table transactions with zero transparent conflict retries. The guards remain held while `ManifestBatchPublisher` publishes the pre-minted lineage under the same exact native-branch/head and table-version precondition.
 
 ### Mutation flow — sequence
 
@@ -100,11 +100,12 @@ sequenceDiagram
     participant cmp as omnigraph-compiler
     participant stg as MutationStaging<br/>(exec/staging.rs)
     participant ts as table_store
-    participant lance as Lance dataset
+    participant rec as schema-v3 recovery sidecar
     participant pub as ManifestBatchPublisher
 
     client->>og: mutate_as(branch, source, name, params, actor_id)
-    og->>cmp: parse + typecheck + lower_mutation_query
+    og->>og: heal/reject recovery intent; open_write_txn
+    og->>cmp: parse + typecheck + lower using txn catalog
     cmp-->>og: MutationIR
     og->>og: enforce_no_mixed_destructive_constructive (D₂)
     loop for each mutation op
@@ -116,23 +117,43 @@ sequenceDiagram
                 og->>ts: scan_with_pending (Lance + DataFusion MemTable union)
                 ts-->>og: matched batches
             end
-        else delete (stage, D₂ keeps separate)
+        else delete (record predicate; D₂ keeps separate)
             og->>ts: count_rows (committed match → affected_*)
             og->>stg: ensure_path + record_delete (predicate)
         end
     end
-    og->>stg: finalize(db, branch)
-    loop per pending table
-        stg->>ts: stage_append OR stage_merge_insert (one per table)
-        ts-->>stg: StagedWrite (transaction + commit metadata + fragments)
-        stg->>ts: commit_staged (advances Lance HEAD)
-        ts-->>stg: new Dataset
+    og->>og: validate complete staged change-set against txn base
+    og->>stg: stage_all(db, branch)
+    loop per touched table
+        stg->>ts: stage_append OR stage_merge_insert OR stage_delete (one per table)
+        ts-->>stg: exact staged transaction (no HEAD movement)
+    end
+    stg->>stg: acquire schema → branch → sorted-table gates
+    stg->>og: recheck recovery barrier + revalidate complete WriteTxn
+    alt authority changed before effects
+        stg-->>og: ReadSetChanged
+        alt insert-only mutation
+            og->>og: discard complete attempt; bounded full reprepare
+        else Update/Delete
+            og-->>client: ReadSetChanged (409)
+        end
+    else authority unchanged
+        stg->>rec: persist fixed lineage + exact transaction identities
+        loop per touched table
+            stg->>ts: commit_staged (zero transparent retries)
+            ts-->>stg: confirm achieved transaction + table update
+        end
+        stg-->>og: updates + expected versions + sidecar + held gates
+        og->>pub: publish exact graph-head/table precondition
+        alt publish succeeds
+            pub-->>og: new manifest version
+            og->>rec: delete sidecar
+            og-->>client: MutationResult
+        else any error after an effect
+            pub-->>og: error
+            og-->>client: RecoveryRequired (sidecar remains authoritative)
+        end
     end
-    stg-->>og: (updates: Vec<SubTableUpdate>, expected_versions)
-    og->>pub: commit_updates_on_branch_with_expected
-    pub->>pub: publisher CAS (cross-table OCC on __manifest)
-    pub-->>og: new manifest version
-    og-->>client: MutationResult
 ```
 
 **Code paths:**
@@ -143,11 +164,11 @@ sequenceDiagram
 - Per-op execution: `execute_insert`, `execute_update`, `execute_delete_node`, `execute_delete_edge`
 - Pending-aware reads: `TableStore::scan_with_pending` / `count_rows_with_pending` at `crates/omnigraph/src/table_store.rs`
 - Edge cardinality with pending: `validate_edge_cardinality_with_pending` at `crates/omnigraph/src/exec/mutation.rs`
-- Per-query accumulator: `crates/omnigraph/src/exec/staging.rs` (`MutationStaging`, `PendingTable`, `PendingMode`, `finalize`)
-- End-of-query Lance commit: `TableStore::stage_append`, `stage_merge_insert`, `commit_staged` at `crates/omnigraph/src/table_store.rs`
-- Manifest commit primitive: `commit_updates_on_branch_with_expected` at `crates/omnigraph/src/db/omnigraph/table_ops.rs`
+- Per-query accumulator and protocol adapter: `crates/omnigraph/src/exec/staging.rs` (`MutationStaging::stage_all`, `StagedMutation::commit_all`)
+- End-of-query Lance operations: `TableStore::stage_append`, `stage_merge_insert`, `stage_delete`, `commit_staged` at `crates/omnigraph/src/table_store.rs`
+- Manifest commit primitive: `commit_updates_on_branch_with_expected` at `crates/omnigraph/src/db/omnigraph/table_ops.rs` (exact native-branch/head precondition plus expected table versions)
 
-Atomicity guarantee for multi-statement mutations: a mid-query failure leaves Lance HEAD untouched on staged tables (no inline commit happened during op execution), so the next mutation proceeds normally with no `ExpectedVersionMismatch`. The publisher CAS at the very end either succeeds (manifest advances atomically across all touched sub-tables) or fails with a typed `ManifestConflictDetails::ExpectedVersionMismatch` (no partial publish). See [docs/dev/invariants.md](invariants.md) and [docs/dev/writes.md](writes.md).
+Atomicity guarantee for multi-statement mutations: a mid-query failure leaves Lance HEAD untouched because no effect occurs during statement execution or staging. A pre-effect authority mismatch discards the complete attempt: insert-only mutations fully reprepare with a bounded retry, while Update/Delete returns typed `ReadSetChanged`. Once any `commit_staged` effect is durable, the attempt either makes every update visible together or returns `RecoveryRequired` with the fixed sidecar left for recovery; the engine never treats that post-effect error as an ordinary rebase. See [docs/dev/invariants.md](invariants.md) and [docs/dev/writes.md](writes.md).
 
 ## Bulk loader (`loader/mod.rs`)
 
@@ -162,11 +183,11 @@ Atomicity guarantee for multi-statement mutations: a mid-query failure leaves La
 
 | Mode | Semantics | Path (post-MR-794) |
 |---|---|---|
-| `Overwrite` | Replace all data in the target tables on the branch | Same accumulator; one `stage_overwrite` + `commit_staged` per touched table at end-of-load (a staged Lance `Operation::Overwrite` transaction — HEAD does not advance until commit; MR-793 Phase 2); publisher CAS. |
-| `Append` | Strict insert; duplicates error | In-memory `MutationStaging` accumulator; one `stage_append` + `commit_staged` per touched table at end-of-load; publisher CAS. |
-| `Merge` | Upsert by `id` (`merge_insert`) | Same accumulator; one `stage_merge_insert` per touched table at end-of-load (Merge mode dedupes by `id`, last-write-wins); publisher CAS. |
+| `Overwrite` | Replace all data in the target tables on the branch | Same accumulator; one staged Lance `Operation::Overwrite` transaction per touched table. A pre-effect authority change is strict `ReadSetChanged`; no automatic replay. |
+| `Append` | Strict insert; duplicates error | One `stage_append` transaction per touched table. A pre-effect authority change discards the whole parsed/validated attempt and fully reprepares with a bounded retry. |
+| `Merge` | Upsert by `id` (`merge_insert`) | One `stage_merge_insert` transaction per touched table (deduped by `id`, last-write-wins). The same bounded full-reprepare rule as Append applies before effects. |
 
-For all three modes, a mid-load failure (RI / cardinality violation, validation error) leaves Lance HEAD untouched on the staged tables — the next load on the same tables proceeds normally with no `ExpectedVersionMismatch`.
+All three modes then use the same schema → branch → sorted-table gate, v3 recovery, zero-retry table commit, and exact publisher-precondition path as mutation. A parse, RI, cardinality, or validation failure leaves Lance HEAD untouched. After any table effect, any later error is `RecoveryRequired`. Load, mutation, and schema apply build no physical indexes inline; explicit `ensure_indices`/`optimize` reconciliation materializes declared intent later.
 
 ## `load` and the deprecated `ingest` shims
 

docs/dev/handoff-rfc-013-write-path.md

@@ -61,11 +61,14 @@ amplification = PR2's target.)
 
 ### A.2 What is LANDED on `main`
 
-- **Step 2a** — `optimize` compacts the internal tables too (`__manifest` / `_graph_commits` /
-  `_graph_commit_actors`), so a *periodically-compacted* graph keeps Term-1 flat. (Cleanup/version-GC
-  of them is the still-open PR1.)
+- **Step 2a** — #291 originally added compaction for all three then-current
+  internal tables (`__manifest`, `_graph_commits`, and `_graph_commit_actors`).
+  After strand-and-retire removed the two lineage datasets, the live behavior is
+  `__manifest`-only internal compaction. A periodically-compacted graph therefore
+  keeps the remaining Term-1 flat. (`__manifest` cleanup/version-GC is still open.)
 - **Phase 7 / #299** (`1c5cb874`) — graph lineage lives in `__manifest` (`graph_commit` +
-  `graph_head:<branch>` rows in the same publish merge-insert; `_graph_commits` is now a projection;
+  `graph_head:<branch>` rows in the same publish merge-insert; `CommitGraph` is
+  now a projection and the former lineage datasets are gone;
   v3→v4 internal-schema migration; schema-version floor). This removed the per-write commit-graph
   scan and closed the manifest→commit-graph atomicity + commit-graph-parent-under-concurrency gaps.
   **This is the base everything below builds on.**
@@ -202,12 +205,14 @@ for the canonical list. Current reality:
 - **Step 3a** — opener bypass: write opens go direct (`Dataset::open` by URI + version)
   instead of the Lance-namespace builder (#288). **This already banked the dominant
   depth win** — see §2 below; it reframes everything.
-- **Step 2a** — internal-table compaction: `optimize` now compacts `__manifest` /
-  `_graph_commits` / `_graph_commit_actors` (#291). Plus the RFC latency-model
-  correction (#292).
+- **Step 2a** — internal-table compaction: #291 originally covered
+  `__manifest` plus the two then-live lineage datasets. The current
+  post-retirement implementation compacts only `__manifest`. Plus the RFC
+  latency-model correction (#292).
 - **Step 4 / Phase 7** — graph lineage moved into `__manifest` (#299 `1c5cb874`):
   `graph_commit` + mutable `graph_head:<branch>` in the publish merge-insert,
-  `_graph_commits` now a projection. **The base for the live branch (§A).**
+  with `CommitGraph` projected directly from those rows and no separate lineage
+  dataset. **The base for the live branch (§A).**
 - **Optimize-vs-write race** — optimize survives a cross-process write race on the
   same table (#297, **LANDED** — origin/main `6d4606a8`; see §6 for why it's not
   redundant with Design A). Step 3b stacks on top of this.
@@ -312,7 +317,7 @@ single staleness feeds **two distinct failure modes**, both surfaced this cycle:
    `writes.rs::served_strict_delete_after_external_optimize_advance_auto_refreshes`
    (`#[ignore]` on branch `fix/write-path-stale-view-probe`). **The naive "just probe" fix is
    proven wrong** — a blanket probe silently refreshes past *logical* advances too, breaking
-   `consistency::stale_handle_public_mutation_must_refresh_then_retry` (the deliberate
+   `consistency::stale_handle_strict_mutation_returns_read_set_changed_then_refresh_retry` (the deliberate
    cross-process lost-update OCC primitive). The fix must **discriminate by op class**.
 
 **Both fold into Design A (step 5), same as §1c.** `open_txn`'s one warm probe makes the base
@@ -425,13 +430,14 @@ reopen/replan **semantics** are permanent. (Noted in RFC §6.6.)
 ## 4. DONE: Step 3b — capture-once `WriteTxn` (shipped on `rfc-013-step-3b-writetxn-v2`)
 
 **Delivered:** on the **table-touch hot path**, a single `mutate`/`load` validates the schema
-contract **once** and opens each touched data table **at most once** — a constant-factor/RTT
-win (not a depth-slope win; 1a). Two cost gates in `write_cost.rs` lock it (both on a node
-insert): `write_validates_schema_contract_once` (3 `read_text` / 2 `exists`, was 12/9) and
+contract once at capture and once at the RFC-022 pre-effect gate, with one cheap trailing
+identity-marker read fencing capture (never per table), and opens each touched data table
+**at most once** — a constant-factor/RTT win (not a depth-slope win;
+1a). Two cost gates in `write_cost.rs` lock it (both on a node insert):
+`write_schema_io_is_bounded_to_capture_fence_and_effect_gate` (7 `read_text` / 4 `exists`) and
 `keyed_insert_opens_table_at_most_once` (`data_open_count <= 1`, was 4). The carrier is the
-minimal `WriteTxn { branch, base }`, threaded as `Option<&WriteTxn>` (`Some` on the hot
-mutate/load path, `None` byte-identical everywhere else); it **converges into** step 5's
-`PublishPlan`.
+operation-local `WriteTxn { branch, base, authority, catalog }`; the catalog is built from the
+exact accepted IR named by the authority token. It **converges into** step 5's `PublishPlan`.
 
 **Not "once" everywhere (scope, not regression):** edge endpoint / cardinality RI validation
 (`ensure_node_id_exists`, the loader's RI + cardinality) still resolves through
@@ -444,8 +450,9 @@ gate yet (only the node gates above).
 
 Commits (off merged-#297 main):
 - **Stage 0** — scope `open_count` → `data_open_count`/`internal_open_count` by URI class
-  (the review fix: `open_dataset_tracked` also opens `__manifest`/`_graph_commits`, so the
-  raw counter conflated them and the gate was unreachable). Re-baselined RED 4.
+  (historically, before lineage-table retirement, `open_dataset_tracked` also opened
+  `__manifest`/`_graph_commits`, so the raw counter conflated them and the gate was
+  unreachable). Re-baselined RED 4.
 - **Commit A (schema-once)** — capture `txn` once at entry (the single validation); the 4
   validation sites collapse: S1 (entry `ensure_schema_state_valid`) removed; S3a
   (`open_for_mutation_on_branch`) + S3b (`prepare_updates_for_commit`) source `txn.base`;
@@ -587,7 +594,8 @@ for #298** (which built none of those constructs) but are **load-bearing constra
   residual concurrent TOCTOU is the §7.1 gap (step 4) — un-widen here, don't over-reach.
 - **Step 4 / Phase 7** (`iss-991`): **LANDED on `main` as #299 (`1c5cb874`).** Lineage now lives
   in `__manifest` (`graph_commit` + mutable `graph_head:<branch>` in the same merge-insert;
-  `_graph_commits` is a projection). Removed the per-write `commit_graph.refresh`; closed the
+  `CommitGraph` is projected from those rows and the old lineage dataset is
+  gone). Removed the per-write `commit_graph.refresh`; closed the
   manifest→commit-graph atomicity + commit-graph-parent-under-concurrency gaps. *(Historical note,
   kept for the §7.1 framing it carried:)* it
   carries the §7.1 *concurrent* write-skew fix (needs the `graph_head` contention row) —
@@ -600,7 +608,7 @@ for #298** (which built none of those constructs) but are **load-bearing constra
   false-fail** (§1d.2) — the op-class-aware precondition + `open_txn` probe. The contract is
   two tests passing *together*: un-ignore
   `writes.rs::served_strict_delete_after_external_optimize_advance_auto_refreshes` (goes green)
-  *while* `consistency::stale_handle_public_mutation_must_refresh_then_retry` stays green
+  *while* `consistency::stale_handle_strict_mutation_returns_read_set_changed_then_refresh_retry` stays green
   (maintenance fast-forwards; logical fails loudly). Self-contained enough to ship standalone
   like #297 if prod pain is acute; otherwise fold into the single PublishPlan delta-interpreter.
 - **Step 2b** — internal-table cleanup + the Q8 monotonic watermark (a Lance boundary tag).

docs/dev/index.md

@@ -74,6 +74,15 @@ pattern, not just the outcome.
 |---|---|
 | camelCase property filters lowercased at runtime (#283) — two engine→Lance boundaries, two different fixes | [bug-case-fix.md](bug-case-fix.md) |
 
+## Active Design Reviews
+
+Review ledgers record open findings against proposed architecture. They remain
+as durable disposition history after closure, so RFC backlinks stay valid.
+
+| Area | Read |
+|---|---|
+| RFC-022–027 split architecture review — open correctness blockers, dependency corrections, and required acceptance evidence | [rfc-022-027-architecture-review.md](rfc-022-027-architecture-review.md) |
+
 ## Active Implementation Plans
 
 Working documents for in-flight feature work. Removed when the work lands.

docs/dev/invariants.md

@@ -146,7 +146,11 @@ converge the physical state.
     drifts" and "manifest-derivable reconciler" items, are instances; so is
     bounding a read's cost to its working set rather than the commit count. This
     is the structural face of "engineering is programming integrated over time":
-    both failure modes are liabilities that compound as the system grows.
+    both failure modes are liabilities that compound as the system grows. At a
+    write/control boundary, schema identity, compiled catalog, and manifest
+    authority must come from one operation-local accepted view: validating a
+    fresh on-disk marker while planning or enumerating gates from a stale warm
+    catalog is not a coherent derivation.
 
 ## Current Truth Matrix
 
@@ -155,11 +159,11 @@ converge the physical state.
 | Multi-table commit | Manifest CAS plus recovery sidecars; not a single Lance primitive | [writes.md](writes.md), [architecture.md](architecture.md) |
 | Constructive mutations | In-memory `MutationStaging`, one end-of-query table commit per touched table, then one manifest publish | [writes.md](writes.md), [execution.md](execution.md) |
 | Deletes | Staged like inserts/updates (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, MR-A) — no inline HEAD advance; mixed insert/update/delete in one query rejected by D2 as a deliberate boundary (constructive XOR destructive per query; compose via separate mutations or a branch) | [query-language.md](../user/queries/index.md), [writes.md](writes.md) |
-| Branch delete | Manifest is the single authority, flipped atomically first; per-table forks are derived state, reclaimed best-effort (`force_delete_branch`) with the `cleanup` reconciler as the guaranteed backstop. Reusing a name whose reclaim failed before `cleanup` surfaces an actionable error | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md) |
+| Branch delete | Manifest is the single authority, flipped atomically first; per-table forks are derived state, reclaimed best-effort (`force_delete_branch`) with the `cleanup` reconciler as the guaranteed backstop. Under schema/target/all-table gates, a target-scoped unresolved sidecar may be made unreachable by deletion and is then audit-discarded by recovery; graph-global SchemaApply still blocks. Reusing a name whose fork reclaim failed before `cleanup` surfaces an actionable error | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) |
 | Schema validation | Type checks, required fields, defaults, edge endpoint checks, and edge cardinality are enforced on write paths | [schema-language.md](../user/schema/index.md), [execution.md](execution.md) |
-| Unique constraints | Value/enum, uniqueness, edge-RI, and cardinality route through ONE unified, catalog-derived evaluator (`crate::validate`) on ALL THREE write surfaces — branch-merge, mutation, and bulk load: Δ-scoped (checks the delta, not the whole graph) and index-backed (probes committed state through its BTREE rather than full-scanning every catalog table), reusing the leaf checks (`loader::validate_value_constraints`/`validate_enum_constraints`/`composite_unique_key`) so the surfaces cannot drift. This closed the prior merge bug (merge validated `@range`/`@check` but not enum) AND the **cross-version uniqueness gap** on the mutation and load paths (a duplicate of a committed `@unique` value is now rejected; the merge path always enforced it). The committed view is the merge target snapshot (merge), the write's pinned `txn.base` (mutation), or the pinned pre-load base (load — `Overwrite` validates the batch as the whole new image, committed view empty); `@card` reads LIVE HEAD per edge table on the mutation path only (the #298 stale-handle fix). `@key` is id-backed, so it is checked intra-delta only (a committed holder of a key value is always the same row — an upsert), skipping a wasted O(Δ) probe per keyed row; `@unique` (non-key) groups do the committed lookup. | [schema-language.md](../user/schema/index.md) |
+| Unique constraints | Value/enum, uniqueness, edge-RI, and cardinality route through ONE unified, catalog-derived evaluator (`crate::validate`) on ALL THREE write surfaces — branch-merge, mutation, and bulk load: Δ-scoped (checks the delta, not the whole graph) and structured-filter-backed (committed probes use a BTREE when reconciled and remain correct by scanning while it is pending), reusing the leaf checks (`loader::validate_value_constraints`/`validate_enum_constraints`/`composite_unique_key`) so the surfaces cannot drift. This closed the prior merge bug (merge validated `@range`/`@check` but not enum) AND the **cross-version uniqueness gap** on the mutation and load paths (a duplicate of a committed `@unique` value is now rejected; the merge path always enforced it). The committed view is the merge target snapshot (merge), the write's pinned `txn.base` (mutation), or the pinned pre-load base (load — `Overwrite` validates the batch as the whole new image, committed view empty); `@card` refreshes the manifest-visible graph-branch snapshot on the mutation path only (the #298 stale-handle fix), then follows each entry's actual inherited/owned Lance ref. `@key` is id-backed, so it is checked intra-delta only (a committed holder of a key value is always the same row — an upsert), skipping a wasted O(Δ) probe per keyed row; `@unique` (non-key) groups do the committed lookup. | [schema-language.md](../user/schema/index.md) |
 | Storage trait | `TableStorage` (via `db.storage()`) is staged-only; the sole inline-commit residual (`create_vector_index`) is split onto a separate sealed `InlineCommitResidual` trait reached via `db.storage_inline_residual()` (MR-854), so §1 holds by construction; capability/stat surfaces are roadmap | [writes.md](writes.md), [architecture.md](architecture.md) |
-| Index lifecycle | `@index`/`@key` declares *intent*; the physical index is derived state and never fails a logical op. `schema apply` builds no indexes (records intent only; index-only changes touch no table data). `load`/`mutate` build inline through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector) that fault-isolates an untrainable Vector column into a *pending* index instead of aborting. `optimize`/`ensure_indices` is the reconciler: it creates declared-but-missing indexes and folds appended/rewritten fragments into existing ones (`optimize_indices`), reporting still-pending columns. Explicit maintenance call, not yet a background loop | [indexes.md](../user/search/indexes.md), [maintenance.md](../user/operations/maintenance.md) |
+| Index lifecycle | `@index`/`@key` declares *intent*; the physical index is derived state and never fails a logical op. `schema apply`, `load`, and `mutate` build no indexes inline: RFC-022 mutation/load sidecars describe only their exact data effects, and index availability must never become a correctness prerequisite. `optimize`/`ensure_indices` is the reconciler: through the single `build_indices_on_dataset_for_catalog` chokepoint it creates declared-but-missing indexes (enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector), folds appended/rewritten fragments into existing ones (`optimize_indices`), and reports untrainable Vector columns as pending. Explicit maintenance call, not yet a background loop | [indexes.md](../user/search/indexes.md), [maintenance.md](../user/operations/maintenance.md) |
 | Traversal IDs | Runtime still builds `TypeIndex`; Lance stable row-id based graph IDs are roadmap | [architecture.md](architecture.md), [query-language.md](../user/queries/index.md) |
 | Auth | Bearer token hashing and server-side actor resolution are implemented at the HTTP boundary | [server.md](../user/operations/server.md), [policy.md](../user/operations/policy.md) |
 | Tests | Tempdir-backed Lance tests are the current substrate; the storage adapter has an in-memory backend for adapter-level contract tests, but Lance datasets bypass it | [testing.md](testing.md) |
@@ -215,12 +219,15 @@ them explicit.
   `maintenance.rs::optimize_compacts_blob_table_alongside_plain_table` pin the
   positive behavior (a red there means blob compaction regressed — restore the
   skip machinery from git history).
-- **Recovery is serialized against live writers in-process only:** the
-  write-entry heal (and `refresh`) serialize against a live writer's sidecar
-  lifetime via the per-`(table, branch)` write queues plus the schema-apply
-  serialization key — all in-process primitives. A recovery pass in one
-  process cannot serialize against a live writer in another (the open-time
-  sweep has the same exposure, and always has): it may roll a live foreign
+- **Recovery is serialized against live writers in-process only:** every
+  `Omnigraph` handle for one canonical local root identity (lexically absolute,
+  with existing symlink ancestors resolved) shares a root-scoped queue manager;
+  object-store/custom scheme identities remain their normalized opaque URIs.
+  The write-entry heal, `refresh`, and Full ReadWrite-open sweep serialize
+  against a live writer's complete sidecar lifetime in schema → branch → sorted
+  table order and re-check sidecar existence after waiting. These remain
+  in-process primitives: a recovery pass cannot serialize against a live writer
+  in another process. It may roll a live foreign
   writer's sidecar forward, which degrades to publisher-CAS contention for
   data writes but can race the schema-staging promotion for a foreign live
   schema apply. The roll-**forward** CAS contention is now
@@ -238,22 +245,26 @@ them explicit.
   cross-process serialization primitive (e.g. lease-based use of the
   schema-apply lock branch) — design it before promoting multi-process write
   topologies.
-- **Fork reclaim is in-process-safe only:** the first write to a table on a
-  branch forks it (a Lance `create_branch` that advances state before the
-  manifest publish). An interrupted fork (crash, or a cancelled request
-  future) leaves a manifest-unreferenced branch ref. The next write self-heals
-  it — `reclaim_orphaned_fork_and_refork` (`force_delete_branch` + re-fork)
-  — but reclaim is only safe because the writer holds the per-`(table,
-  branch)` write queue from before the fork through the publish AND re-checks
-  the live manifest under it, so no *in-process* writer can be mid-fork. A
-  reclaim cannot serialize against a foreign-*process* in-flight fork: it may
-  force-delete a peer's just-created ref, which makes that peer's commit fail
-  and retry — the same one-winner-CAS exposure as above, not corruption. The
-  reclaim never fires unless in-process-queue + manifest authority both prove
-  the ref is manifest-unreferenced. `cleanup`'s per-table reconciler
-  (`reconcile_orphaned_branches`) is the guaranteed backstop for any fork the
-  write path never revisits. Both degrade to a no-op if Lance ships an atomic
-  multi-dataset branch op.
+- **Fork ownership is durable, but Lance ref deletion is not conditional:** a
+  first-touch Mutation/Load table no longer creates its target ref while
+  preparing. Under schema → branch → table gates it revalidates, durably arms a
+  schema-v3 sidecar naming the target, creates the ref, then stages branch-local
+  files and commits. Both `reclaim_orphaned_fork_and_refork` and
+  `reconcile_orphaned_branches` consult pending sidecars before destruction; a
+  foreign claim is conflict/indeterminate, never permission to delete. Full
+  recovery accepts sidecar-before-ref crashes and deletes an unpublished fork
+  only when it is still exactly at the inherited version and no other sidecar
+  claims `(table_path, target ref)`. A no-effect sidecar with a competitor
+  discards only itself; the last survivor either cleans the untouched ref or
+  recovers its owned effect. Partial rollback performs no-effect cleanup before
+  publishing its fixed outcome. This closes the cross-handle live-ref
+  deletion bug and keeps cleanup as the backstop for truly unclaimed refs.
+  The remaining multi-process gap is narrower but real: Lance exposes
+  `force_delete_branch`, not compare-and-delete by `BranchIdentifier`, so a
+  foreign process can create intent/ref between the final list/check and the
+  delete. The documented single-writer-process support boundary remains until
+  Lance provides a conditional ref primitive (or OmniGraph adds a distributed
+  fence); process-local queues are not credited as that primitive.
 - **Local `write_text_if_match` is not a cross-process CAS:** object-store
   backends use a true conditional put (ETag If-Match; the in-memory test
   backend too), but upstream `object_store` leaves `PutMode::Update`

docs/dev/rfc-004-cluster-graph-schema-apply.md

@@ -36,7 +36,7 @@ The implementation spec's hard gate for this phase — failpoint recovery tests
 
 What Phase 4 builds on (all shipped):
 
-- **The engine's recovery discipline.** Writers that can advance Lance HEAD before manifest publish write `__recovery/{ulid}.json` sidecars carrying per-table pins (`expected_version`, `post_commit_pin`); `Omnigraph::open` in read-write mode classifies every pinned table (`NoMovement` / `RolledPastExpected` / `UnexpectedAtP1` / `UnexpectedMultistep` / `InvariantViolation`) and decides all-or-nothing: roll forward via one manifest publish, or roll back via `Dataset::restore`, recording an audit row attributed to `omnigraph:recovery`. The cluster inherits the *vocabulary* of this design but not its mechanics — see the roll-forward-only argument below.
+- **The engine's recovery discipline.** Writers that can advance Lance HEAD before manifest publish write `__recovery/{ulid}.json` sidecars carrying per-table pins (`expected_version`, `post_commit_pin`); `Omnigraph::open` in read-write mode classifies every pinned table (`NoMovement` / `RolledPastExpected` / `UnexpectedAtP1` / `UnexpectedMultistep` / `InvariantViolation`) and decides all-or-nothing: roll forward via one manifest publish, or roll back via `Dataset::restore`. Every completed action records an audit row with the interrupted writer's actor in `recovery_for_actor`. A schema-v3 roll-forward publishes the sidecar's pre-minted lineage and therefore preserves that writer's commit id and actor; rollback and legacy recovery commits use `omnigraph:recovery`. The cluster inherits the *vocabulary* of this design but not its mechanics — see the roll-forward-only argument below.
 - **The engine's schema-apply surface.** `apply_schema_as(desired_source, SchemaApplyOptions { allow_data_loss }, actor)` returns `SchemaApplyResult { supported, applied, manifest_version, steps }`; `preview_schema_apply_with_options` returns the migration plan plus desired catalog without applying; the `__schema_apply_lock__` branch serializes schema applies graph-wide and refuses to run while user branches exist. Policy enforcement (`enforce(SchemaApply, TargetBranch("main"), actor)`) happens before the lock.
 - **Graph init.** `Omnigraph::init(uri, schema_source)` with a strict preflight (errors if schema artifacts exist) and an atomic `_schema.pg` claim. A documented gap: a failed init does not clean up Lance datasets or `__manifest/` it already created.
 - **No engine graph-delete primitive.** Deleting a graph today means removing its object-store prefix. This RFC works with that fact rather than waiting on a primitive.

docs/dev/rfc-022-027-architecture-review.md

@@ -0,0 +1,549 @@
+# Architecture review: RFC-022 through RFC-027
+
+**Status:** Open review findings
+**Date:** 2026-07-11
+**Audience:** RFC authors, engine/storage maintainers, and release reviewers
+**Reviewed against:** OmniGraph 0.8.1; Lance 9.0.0-beta.15 at
+`f24e42c11a742581365e1cbe17c906ea2dac1bc6`; full Lance transaction,
+branch/tag, MemWAL, row-lineage, read/write, compaction, and cleanup
+specifications; pinned Rust implementation where the public specification is
+not precise enough
+
+This is a review ledger, not a competing specification. The RFCs remain the
+normative proposals. A finding is closed only when the affected RFC either:
+
+1. incorporates the required contract and tests; or
+2. explicitly rejects the finding with evidence and records the resulting
+   support boundary.
+
+Items marked **BLOCKER** must be dispositioned before the RFC that owns them is
+accepted. Items marked **TIGHTENING** may be resolved during revision, but they
+must not silently disappear from review.
+
+This ledger is durable review history, not a throwaway implementation plan.
+After every finding is dispositioned, change its status to closed and record
+the resolving RFC sections/commits; keep the RFC backlinks valid.
+
+## Overall architectural judgment
+
+The central architecture remains the right one:
+
+- Lance is the physical storage/versioning truth.
+- `__manifest` is the graph-visible authority and one graph publish is the
+  visibility point for a logical graph commit.
+- RFC-022 should define one correctness protocol with explicit physical-effect
+  adapters, not pretend that Lance offers one universal staged primitive.
+- Key fencing, durable heads, retention, MemWAL, and lineage acceleration are
+  separate irreversible decisions and should keep separate evidence and format
+  gates.
+
+The current blockers are boundary problems, not a reason to replace that core.
+They concern crash classification, actual fencing, foreign-branch authority,
+and capability activation.
+
+> 💬 **Second-pass verification (2026-07-11):** I independently re-verified this
+> ledger's two sharpest new substrate claims against the pinned checkout before
+> commenting: BLOCKER-01's two-phase branch create is confirmed in Lance's own
+> doc comment, and BLOCKER-04's lazy-branch exposure is confirmed
+> architecturally — and appears to be a live bug in the shipped `cleanup`, not
+> only an RFC gap (see the comment there). Overall judgment: concur with this
+> ledger; three of its findings supersede corrections I applied to the RFCs on
+> 2026-07-11 (noted inline at BLOCKER-07, BLOCKER-11, and tightening 5).
+
+The latest revisions improved several important points and those changes should
+remain:
+
+- exact recovery-goal convergence is distinguished from adopting an unrelated
+  newer version;
+- MemWAL is treated as Lance's strategic streaming architecture, with API and
+  format evolution as the integration risk;
+- keyed fast-forward merge now has an explicit embedding-table memory gate;
+- the legacy `__manifest` PK representation is preserved rather than
+  “normalized” into an incompatible form;
+- cleanup's required operating cadence and the cost of indefinite deferral are
+  explicit;
+- native branch refs are no longer described as cross-process-safe merely
+  because a process-local gate exists.
+
+## Dependency correction
+
+Stable table identity and incarnation are consumed by more than durable heads.
+The intended dependency shape is therefore:
+
+```text
+RFC-022 unified write protocol
+    |
+    +-- stable identity/incarnation capability
+    |       +-- RFC-024 durable heads
+    |       +-- RFC-025 checkpoint rows
+    |       `-- RFC-026 stream/reject authority
+    |
+    +-- RFC-023 key fencing
+    |       `-- RFC-026 keyed streaming mode
+    |
+    +-- RFC-025 retention
+    +-- RFC-026 MemWAL
+    `-- RFC-027 merge-delta research
+```
+
+The public RFC process accepts whole RFCs, not independently accepted sections.
+The identity capability must therefore be extracted into its own RFC, or
+RFC-024 must be accepted in full before identity-dependent siblings. An
+implementation may phase an accepted RFC internally, but it cannot treat a
+sub-contract of an unaccepted RFC as a separately accepted decision.
+
+## Blocking findings
+
+### BLOCKER-01 — native graph-branch control is multi-phase
+
+**Affected:** [RFC-022 §7](../rfcs/rfc-022-unified-write-path.md#7-native-graph-branch-ref-control-protocol)
+
+RFC-022 currently models branch create/delete as one native ref mutation whose
+completion is the visibility point. The pinned Lance implementation is more
+specific:
+
+- [`Dataset::create_branch`](https://github.com/lance-format/lance/blob/f24e42c11a742581365e1cbe17c906ea2dac1bc6/rust/lance/src/dataset.rs#L488-L535)
+  first commits a shallow-cloned branch dataset and only then writes
+  `BranchContents`. Lance explicitly calls this a non-atomic two-phase
+  operation and calls `BranchContents` the source of truth.
+- [branch deletion](https://github.com/lance-format/lance/blob/f24e42c11a742581365e1cbe17c906ea2dac1bc6/rust/lance/src/dataset/refs.rs#L548-L584)
+  removes `BranchContents` before cleaning the branch directory.
+
+Therefore create can crash with a zombie branch dataset and no authoritative
+ref. A retry with the same name can fail until the zombie is reclaimed. Delete
+can make the branch logically absent and then return a cleanup error.
+
+**Required disposition:** define an idempotent control-operation classifier for
+at least these states:
+
+| Operation | Physical state | Logical result |
+|---|---|---|
+| create | no clone, no `BranchContents` | not started |
+| create | clone only | zombie; reclaim before retry, or wait for an upstream completion API |
+| create | clone plus matching `BranchContents` | complete |
+| delete | `BranchContents` present | not deleted |
+| delete | `BranchContents` absent, directory remains | deleted; reclaim pending |
+| delete | both absent | complete |
+
+The pinned public API cannot adopt the clone-only state: `BranchIdentifier` is
+generated later by Lance's private `BranchContents` creation phase. The RFC
+must therefore say whether a durable operation marker/sidecar is written before
+the shallow clone, how retry distinguishes its own zombie from foreign state,
+and whether it reclaims and retries or waits for a new upstream completion API.
+It must also explain why cleanup failure after authoritative deletion is not
+reported as if the branch still existed. Add failpoints at both create phases
+and between delete authority removal and directory cleanup.
+
+> 💬 **Verified (2026-07-11):** confirmed verbatim in the pinned source —
+> `dataset.rs:488` documents "This is a two-phase operation… These two phases
+> are not atomic. We consider `BranchContents` as the source of truth… may
+> leave a zombie branch dataset", including the zombie-blocks-retry
+> consequence and the cleanup duties this finding transcribes. The 2026-07-11
+> RFC-022 §7 revision (single-writer-process boundary + conditional-ref
+> upstream ask) addressed the *conditional-put* gap but not this crash
+> classification — the two dispositions compose; both are needed.
+
+### BLOCKER-02 — create-if-absent ownership is not a fencing lease
+
+**Affected:** RFC-023 migration claim, RFC-024 migration claim, and
+[RFC-025 §2.3](../rfcs/rfc-025-checkpoint-retention.md#23-cross-process-retention-claim)
+
+`PutMode::Create` correctly elects one initial owner. A random token in that
+object does not by itself fence a paused owner, because Lance tag, branch,
+migration, and cleanup effects do not condition their commits on that token.
+RFC-025 proposes a check-then-delete release; RFC-023 does not yet specify a
+release protocol at all. For the check-then-delete shape, the race is:
+
+1. old owner reads and verifies token A;
+2. old owner pauses;
+3. takeover removes A and creates token B;
+4. old owner resumes and deletes B, or resumes a destructive physical effect.
+
+“The stale process checks again” does not close the window after its final
+check. Calling the random owner token a fencing token overstates the substrate
+guarantee.
+
+**Required disposition:** choose one honest contract:
+
+- no takeover while the prior process can resume; operator recovery proves the
+  process/host is dead before removing the claim;
+- a substrate-enforced lease/fencing primitive whose monotonically newer token
+  is checked by every protected effect, plus conditional compare-and-delete;
+- or a non-takeoverable claim that requires explicit offline repair.
+
+The `StorageAdapter` does not currently expose conditional delete, and its local
+`write_text_if_match` is not a cross-process CAS. Any selected design must work
+on local and S3 or explicitly narrow the supported backend/topology. For a
+substrate-enforced lease, tests pause an owner after its last token check,
+perform takeover, then resume it; the old owner must be unable to mutate state
+or release the new claim. A host-death or non-takeoverable design instead proves
+that takeover is refused until external fencing/death proof is established.
+
+> 💬 **Concur (2026-07-11):** the paused-owner race is real and the "fencing
+> token" label overstated the guarantee. Given the substrate as it stands (no
+> conditional delete on the adapter; the documented local `write_text_if_match`
+> gap), the only honest near-term contract is the boring one — no takeover
+> while the prior process can resume; operator proves process/host death or
+> uses explicit offline repair. The substrate-enforced lease is the right end
+> state but should be written as an upstream/adapter work item, not assumed.
+
+### BLOCKER-03 — key-conflict retry must first resolve partial effects
+
+**Affected:** [RFC-022 §9](../rfcs/rfc-022-unified-write-path.md#9-concurrency-and-retry-semantics),
+[RFC-023 §6](../rfcs/rfc-023-key-conflict-fencing.md#6-retry-and-validation-semantics),
+and [RFC-026 §6](../rfcs/rfc-026-memwal-streaming-ingest.md#6-fold-protocol)
+
+A retryable concurrent inserted-row-filter conflict is detected when a table
+transaction attempts to commit. (`WhenMatched::Fail` may instead report a
+pre-existing match during merge execution.) In a multi-table graph write or
+MemWAL fold, table A may already have advanced under the armed sidecar before
+table B reports the concurrent key conflict. Immediately “restart the entire
+logical attempt” would replan around unresolved physical state, which RFC-022
+forbids.
+
+**Required disposition:** distinguish:
+
+- a conflict before any physical table commit — finalize the already-durable
+  empty sidecar, then perform a safe full semantic restart;
+- a conflict after any physical commit — keep the sidecar and resolve the
+  RFC-022 recovery outcome first. Normally the partial attempt must roll back
+  before a new semantic attempt can be prepared. If rollback is not safe, return
+  a typed recovery-required failure rather than retrying around it.
+
+For strict insert, finalize an empty intent or retain a partial-effect sidecar,
+as applicable, and return the terminal `KeyConflict`; RFC-022 permits partial
+recovery to finish at the next synchronous barrier. Do not start another
+attempt around that sidecar. For non-strict upsert, the barrier must resolve the
+sidecar before the bounded automatic whole-operation retry. Add a failpoint/race
+in which table N conflicts after table 1 has committed.
+
+### BLOCKER-04 — live graph branches need physical GC protection
+
+**Affected:** [RFC-025 §6](../rfcs/rfc-025-checkpoint-retention.md#6-cleanup-protocol-and-pruned-through-boundary)
+
+The cleanup root set includes live graph branches, but RFC-025 creates Lance
+tags only for named checkpoints. This is insufficient for lazy graph branches.
+A graph branch may store, in its `__manifest`, a foreign reference to an old
+version on a data table's `main` branch. Lance cleanup on that data table cannot
+see the foreign OmniGraph row; it protects versions through its own branch and
+tag references.
+
+**Required disposition:** for every live graph-branch table reference, either:
+
+- materialize a deterministic Lance tag/native ref that physically protects the
+  exact `(table branch, version)`; or
+- choose a per-dataset cleanup cutoff no newer than the oldest live reference,
+  accepting that all intervening versions remain retained.
+
+Checkpoint tags alone do not solve this. Add a test where a lazy branch pins an
+old main-table version, main advances beyond the retention window, cleanup
+runs, and the lazy branch still opens and reads the exact pinned state.
+
+> 💬 **Escalation (2026-07-11):** this is very likely a live bug in the shipped
+> `cleanup`, independent of RFC-025. Verified in code: `cleanup_all_tables`
+> (`optimize.rs:802`) runs Lance `cleanup_old_versions` per data table with no
+> lazy-branch pin logic, and a lazy graph branch creates **no Lance-native ref
+> on the data table** until its first write to that table — its pin exists only
+> as a foreign `__manifest` row Lance cannot see. Repro shape: create a branch,
+> never write table X on it, advance main past the keep window, `cleanup
+> --keep N` → the branch's pinned version of X is collected and the branch
+> breaks. Per the repo's test-first rule this deserves a red regression test
+> and an issue *now*; RFC-025's disposition should then build on that fix
+> rather than owning the discovery. (Snapshot/time-travel pins share the
+> mechanism but are history-trimming under an operator-confirmed policy; a
+> live branch's working state is not history.)
+
+### BLOCKER-05 — durable-head OCC must compare the full head token
+
+**Affected:** [RFC-024 §5](../rfcs/rfc-024-durable-table-heads.md#5-atomic-publisher-contract)
+
+The RFC says expected table versions are compared against table heads. Lance
+version numbers can repeat across recreated datasets/branches, so version-only
+comparison admits drop/recreate ABA.
+
+**Required disposition:** define the OCC token as the complete logical head
+identity, at least:
+
+```text
+(state, stable_table_id, incarnation_id, table_path,
+ table_branch, physical_ref_incarnation, table_version, schema_hash)
+```
+
+The publisher may encode that token differently, but retry/revalidation must
+not reduce it to `table_version`. `physical_ref_incarnation` means an e_tag when
+the backend supplies one, or another proven token that changes when a dataset
+or native ref is deleted and recreated at the same path/branch/version. This is
+distinct from the logical `incarnation_id`, which RFC-024 deliberately preserves
+across an owner handoff. Add stale-writer races for both logical drop/recreate
+and physical ref recreation at the same numeric version.
+
+### BLOCKER-06 — MemWAL needs a capability/format activation barrier
+
+**Affected:** [RFC-026 §3](../rfcs/rfc-026-memwal-streaming-ingest.md#3-enrollment-is-a-recoverable-inline-commit),
+[§5](../rfcs/rfc-026-memwal-streaming-ingest.md#5-ack-path-validation-and-writer-lifecycle),
+[§6](../rfcs/rfc-026-memwal-streaming-ingest.md#6-fold-protocol),
+[§8](../rfcs/rfc-026-memwal-streaming-ingest.md#8-epoch-fenced-quiescence-barrier),
+and [§12](../rfcs/rfc-026-memwal-streaming-ingest.md#12-phasing)
+
+Enrollment creates persistent MemWAL metadata and `stream_state` changes the
+correctness preconditions for schema, branch, maintenance, and data operations.
+An older binary that understands key fencing but not stream lifecycle can ignore
+`OPEN | DRAINING | SEALED`, mutate the base table, or perform a schema/branch
+operation without draining acknowledged rows.
+
+**Required disposition:** define:
+
+- a graph capability/internal-format stamp written only after every enrolled
+  table and lifecycle authority is valid;
+- old-binary/new-format and new-binary/partial-format refusal behavior;
+- migration and rollback/roll-forward ordering;
+- preservation rules for later heads/retention formats;
+- genuine cross-version tests, not a stamp-rewind simulation.
+
+Replica scope must also match RFC-023's recovery support boundary. Multiple
+replicas may route acknowledgement traffic to one shard owner, but enrollment,
+fold, and sidecar recovery remain single-writer-process operations until
+foreign-process sidecar ownership is fenced. Do not advertise general replica
+failover for those operations merely because MemWAL has a shard epoch.
+
+### BLOCKER-07 — stable identity ownership contradicts sibling dependencies
+
+**Affected:** [RFC-024 §3.1](../rfcs/rfc-024-durable-table-heads.md#31-identity-and-object-key),
+[RFC-025 §2.1](../rfcs/rfc-025-checkpoint-retention.md#21-logical-authority-manifest-rows),
+[RFC-026 §7](../rfcs/rfc-026-memwal-streaming-ingest.md#7-fold-time-rejection-is-atomic),
+and [RFC-026 §8](../rfcs/rfc-026-memwal-streaming-ingest.md#8-epoch-fenced-quiescence-barrier)
+
+RFC-024 now says it exclusively owns stable table identity and incarnation and
+that identity-dependent sibling claims must wait for it. RFC-025 still calls
+RFC-024 optional, and RFC-026 persists `stable-table-id` without declaring the
+dependency.
+
+**Required disposition:** extract the identity/incarnation format and migration
+as a shared prerequisite RFC, or accept RFC-024 in full before RFC-025 and
+RFC-026. Durable-head implementation can remain a later phase only after its
+owning RFC and format contract are accepted.
+
+The `_ingest_rejects` deterministic key must use stable table ID plus
+incarnation rather than mutable `table_key`, or rename/recreate can change reject
+identity and break replay idempotence.
+
+> 💬 **Concur; supersedes a 2026-07-11 correction (2026-07-11):** the earlier
+> pass placed identity ownership *inside* RFC-024 §3.1, which created exactly
+> the contradiction described here (RFC-025 calls heads optional while
+> consuming the identity RFC-024 claims to own). The extracted
+> identity/incarnation RFC is the cleaner shape and should replace that
+> ownership note. The `_ingest_rejects` key observation is an additional catch
+> the earlier pass missed — endorsed.
+
+### BLOCKER-08 — retention activation cannot precede migration selection
+
+**Affected:** [RFC-025 §8](../rfcs/rfc-025-checkpoint-retention.md#8-migration-and-compatibility)
+and [§11](../rfcs/rfc-025-checkpoint-retention.md#11-phasing)
+
+RFC-025 leaves in-place migration versus export/import undecided. Its phase
+table nevertheless puts “format activation” in Phase A and “selected migration
+path” in Phase D. Activation cannot be implemented or reviewed before the
+upgrade contract is known.
+
+**Required disposition:** select the migration before Phase A, then specify
+quiescence, partial-state refusal, main-stamp ordering, branch coverage, crash
+recovery, and later capability preservation. If export/import is selected,
+state the irreversible loss of branches, commit DAG, snapshots, and historical
+checkpoints as part of the acceptance decision.
+
+### BLOCKER-09 — decide whether these are public or internal RFCs
+
+**Affected:** [public RFC process](../rfcs/README.md), RFC-022 through RFC-027
+
+The files currently live in the public RFC track but use internal-style
+`rfc-022-*` names, noncanonical `draft`/`research-blocked` statuses, and empty
+owner metadata. In the public process, merging an RFC is acceptance; a merged
+public RFC cannot simultaneously retain undispositioned acceptance blockers.
+
+**Required disposition:** choose the track before merge:
+
+- for the public track, rename to `NNNN-title.md`, use `Status: Proposed`, fill
+  the template's author/discussion/implementation metadata, and resolve every
+  blocker before the RFC PR merges; or
+- move the proposals to `docs/dev/` and follow the maintainer-internal process,
+  leaving the public RFC directory for externally authorable accepted records.
+
+RFC-027 may state “research-blocked” prominently as its technical state while
+retaining the public lifecycle status `Proposed`.
+
+> 💬 **Concur (2026-07-11):** same recommendation as the original family
+> review — these are maintainer-internal proposals mid-revision; `docs/dev/`
+> with the internal process is the low-friction disposition, keeping
+> `docs/rfcs/` for its defined merge-equals-acceptance lifecycle. Whichever
+> track is chosen, decide it before any of the set merges, since it defines
+> what "dispositioned before acceptance" means for every other finding here.
+
+## Protocol clarifications
+
+### BLOCKER-10 — do not put foreign-branch facts in an atomic `ReadSet`
+
+RFC-022 requires every `ReadSet` member to be arbitrated atomically by the
+publish CAS. A CAS on reserved main cannot arbitrate a row on a named source
+branch; a merge-target CAS cannot arbitrate a source-branch row.
+
+Use the right category for each fact:
+
+- checkpoint creation captures an immutable source version, revalidates it
+  before tagging, and then relies on the physical tag. A later source-head
+  advance is intentionally harmless, so the source head is an effect
+  precondition, not target-CAS authority;
+- offline cleanup facts are protected by the selected fleet/retention barrier,
+  not by pretending one main-branch CAS covers every named branch;
+- branch merge should define captured-source-commit semantics. If “latest
+  source at target publish” is required instead, it needs a real source-branch
+  fence held through the target CAS.
+
+Keep target-branch values that must remain stable in the atomic `ReadSet`.
+
+### BLOCKER-11 — RFC-022 and no-heads MemWAL need coarse OCC
+
+**Affected:** [RFC-022 §10](../rfcs/rfc-022-unified-write-path.md#10-rollout-and-compatibility)
+and [RFC-026 §6](../rfcs/rfc-026-memwal-streaming-ingest.md#6-fold-protocol)
+
+[RFC-022's rollout](../rfcs/rfc-022-unified-write-path.md#10-rollout-and-compatibility)
+currently says mutation/load read-set arbitration is likely blocked on
+RFC-024 because probed-but-untouched tables lack mutable head rows.
+
+For graph-content writes, `(branch incarnation, optional graph head)` is already
+a conservative branch-wide authority token. An established branch changes its
+`graph_head:<branch>` row on every graph commit; a fresh branch may initially
+have no branch-specific head row, so absence plus incarnation is part of the
+captured token. Schema identity needs its own atomically contended authority.
+Any concurrent logical graph change forces full revalidation. RFC-024 table
+heads can later reduce false contention by narrowing that token to the tables
+actually read.
+
+Today's publisher retries head-row contention by re-reading the live head and
+reparenting the prepared write. That is insufficient for a
+validation-sensitive plan. On every publisher retry, it must compare the
+captured token and return control to full revalidation rather than reparenting
+and continuing with stale validation.
+
+The lower-liability rollout is therefore:
+
+1. ship correct coarse OCC with `graph_head` plus schema identity;
+2. introduce table-head tokens only after RFC-024 passes its independent format
+   and cost gates.
+
+The no-heads RFC-026 fold path must carry the same captured branch token in its
+`ReadSet`. Do not recouple RFC-022 or RFC-026 correctness to RFC-024
+performance.
+
+> 💬 **Concur; supersedes a 2026-07-11 correction (2026-07-11):** the premise
+> is independently verified — `graph_head:<branch>` is one mutable row updated
+> in every graph-content publish (the deliberate contention point pinned by
+> the concurrent-disjoint-writers tests), so the coarse token genuinely
+> arbitrates probed-but-untouched same-branch tables. This is a better design
+> than the earlier RFC-022 §10 wording ("likely blocked on RFC-024"), which
+> should be revised to the two-step rollout described here. One cost worth
+> stating in the revision: coarse OCC makes *any* concurrent commit on a busy
+> branch force full revalidation of in-flight writers — real throughput cost
+> on agent-fleet branches, and precisely the honest motivation for RFC-024's
+> later narrowing (rather than a correctness argument for it).
+
+> **Implementation disposition (2026-07-11):** mutation/load now capture the
+> native Lance `BranchIdentifier`, exact optional `graph_head`, and accepted
+> schema identity; revalidate under a branch-then-table gate; and pass the same
+> token to every publisher retry and schema-v3 recovery decision. Metadata-only
+> schema-apply tests pin the required invariant that supported schema changes
+> move `graph_head` even when no data-table version changes. This closes the
+> coarse mutation/load cell without claiming a general schema-authority row or
+> multi-process native-ref fencing. RFC-024 remains the false-contention
+> narrowing step.
+
+### TIGHTENING-01 — symmetric Lance conflicts are not obviously an activation gate
+
+RFC-023 correctly identifies Lance's directional filtered/unfiltered behavior.
+However, its own mandatory fleet outage, recovery drain, historical-duplicate
+validation, stamp-last activation, and old-binary refusal are intended to make
+unfiltered writers unreachable after activation.
+
+The current RFC-023 routing table still permits a direct existing-row `Update`,
+whose Lance operation has no inserted-row filter. Therefore unfiltered
+transactions do remain reachable after activation. Symmetry can be demoted only
+if every insertion-bearing keyed path routes through filtered merge-insert and
+the RFC proves a direct update cannot insert a row or change `id`; affected-row
+conflict metadata must continue to cover updates to the same existing row.
+
+After that narrowing, upstream symmetry is useful defense in depth and a
+valuable Lance surface guard, but it need not block activation for transaction
+pairs that cannot violate key insertion. Two old unfiltered writers are not
+protected by symmetry in either case.
+
+> 💬 **Concur, with the deciding check named (2026-07-11):** the concrete
+> unfiltered-`Update` reachable after activation is the matched-only
+> partial-schema update merge (`WhenMatched::UpdateAll` +
+> `WhenNotMatched::DoNothing`, the field-level-updates path in PR #342): it
+> classifies zero inserts, so whether it emits an **empty** filter (`Some`
+> with no keys — symmetric-safe) or **no** filter (`None` — the reachable
+> unfiltered transaction) is exactly what decides whether this demotion is
+> sound. That is a one-test question against the pinned revision and should be
+> pinned as a surface guard before the demotion is accepted.
+
+## Specification and acceptance tightening
+
+These are smaller than the blockers above, but should be resolved before the
+RFC set is merged:
+
+1. **Checkpoint-name normalization:** define allowed bytes, Unicode
+   normalization, case sensitivity, maximum encoded length, reserved prefixes,
+   and normalization-version compatibility. Test collisions and reuse across
+   versions.
+2. **RFC lifecycle values:** [`docs/rfcs/README.md`](../rfcs/README.md#status-values)
+   defines `Proposed`, `Accepted`, `Declined`, `Superseded by NNNN`, and
+   `Implemented`. Either use those values or amend the process before using
+   `draft` and `research-blocked` as machine-readable statuses. Research-blocked
+   can remain prominent in RFC-027's body while its lifecycle status is
+   `Proposed`.
+3. **Provisional version naming:** sibling RFCs should say “RFC-024 heads
+   format” rather than treating `v5` as permanent while RFC-024 says the numeral
+   will change if another format lands first.
+4. **Capability ordering:** RFC-025 must specify retention-first, heads-first,
+   and co-release preservation rules. RFC-026 is a conditional dependency when
+   streams are enrolled because cleanup must persistently quiesce them.
+5. **Memory measurement:** `helpers::cost` measures I/O, not peak RSS. RFC-023's
+   adopted-merge and RFC-027's lineage memory gates should use the subprocess
+   `scenarios.rs` harness or an equivalent `wait4`/`ru_maxrss` instrument and
+   name dataset sizes, baseline, cap, and pass threshold.
+
+   > 💬 **Concur; catches a gap in a 2026-07-11 correction:** the RFC-023
+   > §11.4 memory gate added that day states the bound ("peak memory bounded by
+   > batch size") without naming an instrument that can measure it — this item
+   > is what makes that gate enforceable rather than aspirational.
+6. **Derived-index fallbacks:** acceptance tests must force index-absent and
+   partially covered states for RFC-024 head lookup and RFC-027 candidate
+   discovery, assert identical logical results, and assert the promised
+   degraded-cost/fallback telemetry.
+
+## Acceptance order after disposition
+
+The review does not require all RFCs to land together. A safe order is:
+
+1. accept RFC-022 after branch-control recovery, coarse `graph_head` OCC, and
+   foreign-branch fact classification are explicit;
+2. accept and land the stable identity RFC/capability;
+3. accept RFC-023 once partial-effect retry and its format/fleet barrier are
+   complete;
+4. evaluate RFC-024 independently on its physical lookup cost gate;
+5. accept RFC-025 after physical protection of both checkpoints and live graph
+   branches plus a selected upgrade path;
+6. accept RFC-026 after its capability barrier and writer-ownership scope are
+   explicit;
+7. keep RFC-027 in research until deletion-delta discovery passes its stated
+   correctness and flat-cost gates.
+
+This ordering preserves the split's main benefit: a blocked performance
+optimization or research result does not hold correctness work hostage.
+
+> 💬 **Concur with the order; two sequencing notes (2026-07-11):**
+> BLOCKER-04's escalation means step 5 has a prerequisite outside this list —
+> the live lazy-branch cleanup exposure needs its red regression test and fix
+> in the shipped `cleanup` first, so RFC-025 builds on a correct baseline. And
+> step 3's TIGHTENING-01 demotion hinges on the partial-update filter check
+> noted there; if that check lands `None`, upstream symmetry returns to the
+> activation gate and RFC-023's timeline moves accordingly.

docs/dev/testing.md

@@ -22,7 +22,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
 | `branching.rs` | Branch create / list / delete, lazy fork |
 | `merge_truth_table.rs` | Merge-pair truth table (MR-786): all 9×9 `(left_op, right_op)` cells from `{noop, addNode, removeNode, addEdge, removeEdge, setProperty, dropProperty, addLabel, removeLabel}`. Adding a new op to `OpVariant` forces a compile error in `build_case` until the new row + column are dispositioned. 36 executable cells run through real `branch_merge` with a structured oracle (`MergeOutcome` / `MergeConflictKind` + graph-state assert); 45 cells involving `dropProperty`/`addLabel`/`removeLabel` are recorded as `Unsupported` until the mutation grammar grows. |
 | `merge_fast_forward.rs` | Fast-forward branch-merge cost + correctness: an append-only adopted-source merge routes *new* rows through `stage_append` instead of one whole-delta `stage_merge_insert` (the full-outer join that exhausted the DataFusion memory pool on embedding-bearing tables). The regression gate is structural — it asserts WHICH staged-write primitive the merge invokes via the task-local write probes (`omnigraph::instrumentation`), not a brittle size threshold; also: merge yields source state, defers vector indexes to the reconciler, streams blob columns |
-| `writes.rs` | Direct-publish writes: cancellation, non-strict insert/merge rebase under the per-table queue, strict stale-write conflicts, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery); the lance#7444 row-id-overlap regression (`filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent` — merge-load → same-key merge-load → delete → keyed point lookup, green only under the vendored lance-table patch — plus its append-only control) |
+| `writes.rs` | Direct-publish writes: cancellation, RFC-022 non-strict full-attempt reprepare from fresh branch authority, strict stale-write conflicts, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery); the lance#7444 row-id-overlap regression (`filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent` — merge-load → same-key merge-load → delete → keyed point lookup, green only under the vendored lance-table patch — plus its append-only control) |
 | `staged_writes.rs` | TableStore staged-write primitives (`stage_append`, `stage_merge_insert`, `commit_staged`, `scan_with_staged`, `count_rows_with_staged`) — primitive-level only; engine code uses the in-memory `MutationStaging` accumulator instead |
 | `forbidden_apis.rs` | Defense-in-depth source-walk guard: engine code (`exec/`, `db/omnigraph/`, `loader/`, `changes/`) must not reach around the sealed storage trait to Lance inline-commit APIs, nor open datasets directly (`Dataset::open` / `DatasetBuilder::from_uri`/`from_namespace`) — reads route through `Snapshot::open` and the held-handle cache; `// forbidden-api-allow: <reason>` sentinel exempts reviewed lines |
 | `failpoint_names_guard.rs` | Source-walk guard (same defense-in-depth shape as `forbidden_apis.rs`): every failpoint call site (`maybe_fail`, `ScopedFailPoint::new`/`with_callback`, `Rendezvous::park_first`) must reference a `failpoints::names` const, never a bare string literal — a typo'd literal compiles and silently never fires |
@@ -36,7 +36,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
 | `changes.rs` | `diff_between` / `diff_commits` |
 | `consistency.rs` | Cross-table snapshot isolation, atomic publish |
 | `lineage_projection.rs` | RFC-013 Phase 7 acceptance gate: graph lineage lives ONLY in `__manifest` — over a realistic history (main commits, a branch, a merge, actors), the production `CommitGraph::open` projection reconstructs the full DAG (commit set, parents, merge parents + merge actor, per-branch heads, inline actors) from the `graph_commit`/`graph_head` rows, and the `_graph_commits.lance` / `_graph_commit_actors.lance` dataset directories are never created at all |
-| `schema_apply.rs` | Migration plan + apply, schema-apply lock; index materialization deferred to the reconciler (iss-848): `apply_schema_defers_vector_index_on_empty_table` (an empty-table Vector `@index` never aborts the apply) and `index_only_constraint_apply_touches_no_table_data` (adding an `@index` is metadata-only — no table-version bump); enum widening (iss-enum-widening-migration): `enum_widening_apply_is_metadata_only_and_accepts_new_variant` (no table-version bump; new variant accepted, out-of-set still rejected) + `enum_narrowing_apply_is_refused` (OG-MF-106 with the graph left writable). The planner's widening/narrowing matrix lives in `schema_plan.rs`'s in-source tests |
+| `schema_apply.rs` | Migration plan + apply, schema-apply lock; schema-contract publication is pinned by `read_only_open_holds_schema_gate_through_catalog_capture` and `refresh_holds_schema_gate_through_catalog_publication` (source, accepted IR/state, and compiled catalog are captured under one root schema gate). `long_lived_handle_uses_the_schema_catalog_bound_to_its_write_token` covers mutation/load plus a post-apply new node type merged through the pre-apply handle; `stale_handle_branch_delete_gates_tables_added_by_schema_apply` parks delete over that new type while a legacy index reconciler waits, proving merge planning and native-control table envelopes use an operation-local accepted catalog rather than stale ArcSwap state. Index materialization is deferred to the reconciler (iss-848): `apply_schema_defers_vector_index_on_empty_table` (an empty-table Vector `@index` never aborts the apply) and `index_only_constraint_apply_touches_no_table_data` (adding an `@index` is metadata-only — no table-version bump); enum widening (iss-enum-widening-migration): `enum_widening_apply_is_metadata_only_and_accepts_new_variant` (no table-version bump; new variant accepted, out-of-set still rejected) + `enum_narrowing_apply_is_refused` (OG-MF-106 with the graph left writable). The planner's widening/narrowing matrix lives in `schema_plan.rs`'s in-source tests |
 | `search.rs` | FTS / vector / hybrid (`bm25`, `nearest`, `rrf`) |
 | `scalar_indexes.rs` | Per-property index dispatch of `build_indices_on_dataset_for_catalog`: enums + orderable scalars get a BTREE (so `=`/range/IN/IS NULL are index-accelerated), free-text Strings keep FTS — observed via `TableStore::key_column_index_coverage`, the same helper the traversal chooser uses |
 | `traversal.rs` | `Expand`, variable-length hops, anti-join, undirected traversal (`$a <edge> $b`, `Direction::Both` — out ∪ in with set-semantics dedup, both-direction anti-join) (CSR path — `OMNIGRAPH_TRAVERSAL_MODE` unset) |
@@ -51,10 +51,10 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
 | `validators.rs` | Schema constraint enforcement (enum, range, unique, cardinality) across JSONL load, mutation insert/update. ALL THREE write surfaces — mutation, bulk load, AND merge — route through the unified `crate::validate` evaluator (Δ-scoped, index-backed, reusing these leaf checks). Cross-version-uniqueness closure: `cross_version_unique_rejected_on_mutation_insert` + `reinsert_existing_key_is_upsert_not_unique_violation` (mutation path); `cross_version_unique_rejected_on_append_load` + `merge_load_reupsert_existing_key_is_not_unique_violation` (load path). Per-table `Overwrite`: `overwrite_load_validates_ri_against_new_image` (an edges-only overwrite still resolves RI against retained committed nodes) + `append_load_rejects_orphan_edge`. The evaluator's own unit tests live in `src/validate.rs` (`#[cfg(test)]`); its merge-conflict equivalence is pinned by `merge_truth_table.rs` (OrphanEdge) + `branching.rs` (Unique/Cardinality merge tests). Intra-batch duplicate-`@key` rejection on every load mode is pinned by `consistency.rs::loader_rejects_intra_batch_duplicate_keys`; the mutation-coalesce counterpart (insert+update / chained updates of one id are NOT a self-collision) by `writes.rs`. Non-String `@unique` columns probe committed state with a TYPED literal (not a stringified key): `cross_version_unique_rejected_on_date_column` + `noncolliding_write_to_date_unique_column_succeeds` (a `Date @unique` collision is a proper `@unique` violation, and a distinct value does not raise a Date32-vs-Utf8 coercion error). Cardinality is keyed by edge id, last-wins (matching commit's `dedupe_merge_batches_by_id`): `merge_load_edge_src_move_rechecks_vacated_src_cardinality` (a Merge-load moving an edge recounts the vacated src for `@card` min) + `merge_load_duplicate_edge_id_counts_once_per_card` (a dup edge id under two srcs in one batch counts once, no spurious max violation). Direct deletes capture the ids they remove (from the delete op's own scan) into the change-set's `deleted_ids`, so a delete emptying a src is validated: `mutation_delete_edge_below_card_min_rejected` (a `delete Edge` dropping a src below `@card` min is rejected, not silently committed). |
 | `merge_cost.rs` | Cost-budget tests for branch MERGE on the shared `helpers::cost` harness: `merge_validation_is_delta_scoped` (a 1-row-delta merge opens ≤3 data tables — Δ-scoped, not the whole catalog; was ~6 pre-#5) and `merge_manifest_cost_grows_with_history` (the cross-branch `__manifest` open amplification still grows with commit depth — a separate, not-yet-addressed term — while validation `data_open_count` stays flat) |
 | `policy_engine_chassis.rs` | Engine-layer Cedar enforcement (MR-722): allow + deny through every `_as` writer via the SDK directly — no HTTP — proving embedded and CLI callers hit the same gate as the server, with action × scope shapes matching `authorize_request` |
-| `maintenance.rs` | `optimize` (compaction), `repair` (explicit uncovered-drift publish), and `cleanup` (version GC): empty/idempotent/no-op edges, policy validation, head preservation; `optimize` publishes its own compaction (`optimize_publishes_compaction_to_manifest_so_schema_apply_succeeds`), skips pre-existing uncovered drift (`optimize_skips_preexisting_manifest_head_drift`), and refuses to run while a `__recovery` sidecar is pending (`optimize_defers_when_recovery_sidecar_is_pending`); `repair` previews/heals verified maintenance drift, refuses raw semantic drift without `--force`, and forced repair publishes only by explicit operator choice; the index reconciler (iss-848): `index_build_tolerates_null_vector_rows` (an untrainable Vector column defers instead of aborting the build, sibling indexes still build) and `optimize_materializes_index_declared_but_unbuilt` (optimize creates a declared-but-deferred index) |
-| `failpoints.rs` | Failure-injection coverage (gated on `failpoints` feature). Includes the five per-writer Phase B → recovery integration tests (`recovery_rolls_forward_after_finalize_publisher_failure`, `schema_apply_phase_b_failure_recovered_on_next_open`, `branch_merge_phase_b_failure_recovered_on_next_open`, `ensure_indices_phase_b_failure_recovered_on_next_open`, `optimize_phase_b_failure_recovered_on_next_open`) and the write-entry in-process heal contract (the four `*_after_finalize_publisher_failure_heals_without_reopen` tests — load, mutation, schema apply, branch merge: a follow-up write on the same handle rolls a sidecar-covered residual forward without reopen/refresh) and the storage-fault matrix for the sidecar lifecycle (`recovery.sidecar_{write,delete,list}` / `recovery.record_audit` failpoints: Phase A put failure aborts with zero drift, Phase D delete failure is swallowed and healed by the next write, list failures are loud at heal and open, audit-append failures are retried to exactly one audit row; plus the bucket-gated `s3_load_recovers_after_publisher_failure_without_reopen`). And the convergence-idempotent roll-forward regression (`open_sweep_roll_forward_converges_when_manifest_advances_concurrently`: two concurrent open-sweeps race one sidecar at the `recovery.before_roll_forward_publish` rendezvous; the CAS loser must converge, not fail the open — iss-schema-apply-reopen-recovery-race). |
+| `maintenance.rs` | `optimize` (compaction), `repair` (explicit uncovered-drift publish), and `cleanup` (version GC): empty/idempotent/no-op edges, policy validation, head preservation; `optimize` publishes its own compaction (`optimize_publishes_compaction_to_manifest_so_schema_apply_succeeds`), skips pre-existing uncovered drift (`optimize_skips_preexisting_manifest_head_drift`), refuses to run while a `__recovery` sidecar is pending (`optimize_defers_when_recovery_sidecar_is_pending`), and compacts blob-v2 tables through the normal path (`optimize_compacts_blob_table_alongside_plain_table`); `repair` previews/heals verified maintenance drift, refuses raw semantic drift without `--force`, and forced repair publishes only by explicit operator choice; the index reconciler (iss-848): `index_build_tolerates_null_vector_rows` (logical load succeeds without inline index work; an untrainable Vector column then defers during reconciliation) and `optimize_materializes_index_declared_but_unbuilt` (optimize creates a declared-but-deferred index) |
+| `failpoints.rs` | Failure-injection coverage (gated on `failpoints` feature). RFC-022 includes deterministic post-stage/pre-effect races for mutation/load uniqueness and strict disjoint-head changes, plus the cross-handle post-effect `RecoveryRequired` → read-write-open rollback cell. Control/recovery race cells include `first_touch_post_create_open_error_keeps_recovery_ownership`, `branch_delete_orphans_sidecar_armed_after_initial_barrier`, `branch_merge_fences_target_delete_recreate_aba`, `branch_merge_fences_concurrent_sync_on_same_handle`, `branch_merge_rejects_fresh_target_manifest_change_before_effects`, `branch_merge_rechecks_late_sidecar_after_table_gates`, `cleanup_rechecks_sidecars_under_gc_gates`, and `full_recovery_rereads_sidecar_body_after_discovery`. The suite also includes the five per-writer effect → manifest-CAS recovery tests, write-entry in-process heal contract, storage-fault matrix, S3 recovery twin, and convergence-idempotent roll-forward regression. |
 | `failpoint_names_guard.rs` | Source-walk guard: every `maybe_fail(...)` call site (engine + cluster) must reference the compile-checked `failpoints::names` consts, never a bare string literal — a typo'd literal compiles but silently never fires; same defense-in-depth shape as `forbidden_apis.rs` |
-| `recovery.rs` | Open-time recovery sweep — sidecar I/O, classifier dispatch (NoMovement / RolledPastExpected / UnexpectedAtP1 / UnexpectedMultistep / InvariantViolation), all-or-nothing decision, roll-forward via `ManifestBatchPublisher::publish`, roll-back via `Dataset::restore`, audit row in `_graph_commit_recoveries.lance`, `OpenMode::ReadOnly` skip path |
+| `recovery.rs` | Open-time recovery sweep — legacy sidecars plus schema-v3 Mutation/Load exact transaction identity, fixed logical/rollback outcome IDs, branch-token comparison, fresh under-gate sidecar reread/reparse, all-or-nothing roll-forward/rollback/refusal, audit row in `_graph_commit_recoveries.lance`, and `OpenMode::ReadOnly` skip path |
 | `composite_flow.rs` | Compositional/narrative end-to-end stories — multi-step flows that compose mechanics covered by other test files. Catches integration regressions where individual operations all pass their unit tests but their composition breaks (sequential merges, post-merge main writes, time-travel through merge DAG, reopen consistency over multi-merge histories, post-optimize and post-cleanup strict writes). |
 
 ## Fixtures

docs/dev/writes.md

@@ -4,11 +4,14 @@
 > removed in MR-771 (shipped v0.4.0). Writes now go directly to the target
 > table; this document specifies that direct-publish path.
 
-`mutate_as` and `load` write **directly to the target table**
-and call `ManifestBatchPublisher::publish` once at the end with
-`expected_table_versions` (the per-table manifest versions captured before
-the first write). Cross-table OCC is enforced inside the publisher; the
-publisher's row-level CAS on `__manifest` is the single fence.
+`mutate_as` and `load` prepare against one immutable branch-authority token,
+write **directly to the target table**, and call
+`ManifestBatchPublisher::publish` once at the end. The token is
+`(Lance branch identifier, exact optional graph_head, accepted schema identity)`;
+the exact `graph_head` check protects validation dependencies on tables the
+write does not touch. Publisher row-level CAS on `__manifest` is the visibility
+fence. Process-local branch/table queues reduce retries but are not distributed
+authority.
 
 ## What this means in practice
 
@@ -19,15 +22,93 @@ publisher's row-level CAS on `__manifest` is the single fence.
   `__run__*` branch on an upgraded graph is swept off `__manifest` by the
   v2→v3 internal-schema migration on first read-write open. (The inert
   `_graph_runs.lance` bytes remain until a `delete_prefix` primitive lands.)
-- Cancelled mutation futures leave **no graph-visible state** — the manifest
-  is never advanced. They can leave two kinds of unreferenced residue, both
-  self-healing: orphaned Lance fragments (reclaimed by `omnigraph cleanup`),
-  and — on the *first* write to a table on a branch, which forks it before the
-  publish — a manifest-unreferenced branch ref. The next write to that table
-  reclaims the stale fork and re-forks (`reclaim_orphaned_fork_and_refork`),
-  and `cleanup`'s per-table reconciler is the guaranteed backstop; see the
+- Cancelled mutation futures leave **no graph-visible state** unless the
+  manifest publish already completed. Before that point they can leave
+  reclaimable uncommitted Lance files, or sidecar-covered committed table
+  effects that the next quiesced recovery rolls forward/compensates. A
+  first-touch named-branch write can also leave a target table ref, but it is
+  never created without ownership: the schema-v3 sidecar is durable first and
+  names that `(table_path, target ref)`. Reclaim and `cleanup` treat any
+  matching pending sidecar as a hard stop. Quiesced full recovery accepts both
+  crash shapes — sidecar durable with no ref yet, or an exact untouched ref at
+  the inherited version — and removes the latter before deleting the empty
+  intent. If several pending intents claim one ref, a no-effect intent discards
+  only itself while any competitor remains; the last no-effect survivor cleans
+  an untouched ref, or the effect-owning survivor recovers normally. `cleanup`
+  remains the backstop for genuinely unclaimed legacy/stale refs; see the
   fork-reclaim note in [invariants.md](invariants.md).
 
+## Mutation/load coarse OCC (RFC-022 first adapter)
+
+Mutation and load use a closed prepare → effect → publish attempt:
+
+1. run the branch-aware recovery barrier, then capture the target branch's native Lance
+   `BranchIdentifier`, exact `graph_head:<branch>` (including absence on a
+   fresh branch), accepted schema identity, and table snapshot;
+2. run the complete validator and prepare every effect outside the effect gate.
+   Existing-table transactions may stage reclaimable files here. A first-touch
+   named-branch table retains its batch/predicate and pre-mints the transaction
+   identity instead: Lance branch-local files cannot be staged until its target
+   ref exists;
+3. acquire the schema gate, branch gate, then sorted table queues; re-check for
+   a relevant sidecar armed since step 1, then revalidate the token. Any
+   unresolved relevant intent returns typed `RecoveryRequired` before effects;
+4. on a pre-effect mismatch, discard the complete attempt. Append/Insert/Merge
+   reprepare with a bounded retry; strict Update/Delete/Overwrite return typed
+   `ReadSetChanged`;
+5. arm a schema-v3 recovery sidecar. For each deferred first-touch table, create
+   its target ref, stage branch-local files on that ref, and bind the staged
+   transaction to the pre-minted UUID. Then commit every planned transaction
+   with zero transparent conflict retries, confirm exact transaction UUIDs and
+   table updates, and publish the pre-minted lineage intent under the same token.
+
+The publisher checks the exact head and native branch identity on every CAS
+attempt. It never reparents a validation-sensitive intent after contention. A
+mismatch after any physical effect returns `RecoveryRequired` and leaves the
+sidecar intact; it is not an ordinary retry loop.
+
+This adapter preserves the documented single-writer-process support boundary.
+The native branch identifier detects delete/recreate ABA but is not a Lance
+conditional-ref fence, and destructive recovery remains unsafe beside a live
+foreign process.
+
+### Branch-merge authority fence (adapter bridge)
+
+Branch merge still uses its writer-specific multi-commit table effects and
+confirmation sidecar; it has not yet been converted to the RFC-022 exact-effect
+adapter. It does, however, join the closed control boundary needed by this first
+slice: after the strict recovery barrier it acquires the root-shared schema gate
+and the sorted source/target branch gates, performs the final sidecar check,
+loads one operation-local catalog from the accepted contract, captures both graph
+heads plus the base/source/target snapshots, and holds those gates through table
+effects and manifest publication. Planning stays outside table queues. Before
+Phase A, merge acquires the conservative all-catalog table envelope for both
+source and target, re-lists sidecars, and compares fresh source/target manifest
+versions with the captured snapshots. A stale warm handle catalog or coordinator
+snapshot is never accepted as that revalidation.
+
+That fence prevents a same-process target delete/recreate from reusing the branch
+name underneath a merge plan. The race test deliberately recreates a target with
+the same name and numeric Lance version but a different `BranchIdentifier`, so
+version-only checking cannot accidentally satisfy it. This is a process-local
+bridge, not a cross-process conditional-ref primitive and not a substitute for
+the later full branch-merge read-set/reprepare adapter. `sync_branch` joins the
+same root schema gate before replacing a handle's coordinator, so it cannot
+overwrite merge's temporary target coordinator or change a native control's
+active-branch authority mid-operation.
+
+### Branch-delete orphaning exception
+
+Branch deletion runs the healer first and then holds schema, the target branch,
+and every accepted-catalog table gate through the native ref removal. An
+unresolved sidecar scoped to that target does not permanently block deletion:
+once those gates prove its in-process owner is no longer live, removing the
+manifest branch makes its physical effects unreachable. The next write/open
+records the orphan-discard recovery audit and deletes the sidecar. A
+`SchemaApply` sidecar remains graph-global and blocks deletion. This exception
+is specific to removing the authority that made the intent reachable; create,
+merge, mutation, and load still reject relevant unresolved ownership.
+
 ## Read-your-writes within a multi-statement mutation
 
 A `.gq` query with multiple ops (e.g. `insert Person … insert Knows …`)
@@ -47,12 +128,20 @@ shared by both `mutate_as` and the bulk loader:
   `TableStore::scan_with_pending`, which scans committed via Lance
   and applies the same SQL filter to the pending batches via DataFusion
   `MemTable`. Same-query writes are visible to subsequent reads.
-- At end-of-query, `MutationStaging::finalize` issues exactly one
-  `stage_*` + `commit_staged` per touched table (concatenating
-  accumulated batches; merge-mode dedupes by `id`, last-write-wins),
-  and the publisher publishes the manifest atomically across all
-  touched sub-tables. Cross-table conflicts surface as
-  `ManifestConflictDetails::ExpectedVersionMismatch`.
+- Blob-bearing updates use the materializing variant: Lance reads only matched
+  committed blob payloads as binary, the engine normalizes them back to the
+  logical blob schema, and the full rows join the same pending-shadow union.
+  Rewriting matched blob bytes costs I/O proportional to those bytes, but makes
+  correctness independent of whether a physical index selects a different
+  Lance merge plan.
+- At end-of-query, `MutationStaging::stage_all` prepares exactly one staged
+  transaction per touched table and `commit_all` commits it (concatenating accumulated
+  batches; merge-mode dedupes by `id`, last-write-wins), and the publisher
+  publishes the manifest atomically across all touched sub-tables. Existing
+  tables stage before gate acquisition; a first-touch named-branch table stages
+  after sidecar + fork under the gates so its uncommitted files live in the
+  correct Lance branch tree. Cross-table conflicts surface as typed read-set or
+  manifest conflicts.
 - **Deletes stage too (MR-A).** Lance 7.0's
   `DeleteBuilder::execute_uncommitted` (#6658) makes delete a two-phase op,
   so deletes no longer inline-commit. Each delete records a predicate in
@@ -153,9 +242,9 @@ integrity, and edge cardinality before any Lance HEAD movement, stages
 each touched table with Lance `Operation::Overwrite`, then runs
 `commit_staged` under the normal `SidecarKind::Load` recovery sidecar
 before publishing `__manifest`. `OMNIGRAPH_LOAD_CONCURRENCY` applies to the
-fragment-writing stage only; the commit and manifest publish still run
-under the per-table write queues. Empty-table overwrite is represented as
-a valid zero-fragment Lance `Overwrite` transaction, not as
+fragment-writing stage only; the commit and manifest publish run while holding
+the root-shared schema → branch → sorted-table gates. Empty-table overwrite is
+represented as a valid zero-fragment Lance `Overwrite` transaction, not as
 truncate-then-append.
 
 ### Open-time recovery sweep
@@ -170,8 +259,8 @@ test pins.
 A second, narrower drift class — the **finalize → publisher window** —
 is closed across one open cycle by the open-time recovery sweep:
 
-`MutationStaging::finalize` runs `stage_*` + `commit_staged` per touched
-table sequentially, then the publisher commits the manifest. Lance has
+`MutationStaging::stage_all` prepares the table transactions and `commit_all`
+runs their independent HEAD advances before the publisher commits the manifest. Lance has
 no multi-dataset atomic commit, so the per-table `commit_staged` calls
 are independent operations: if commit_staged on table N+1 fails *after*
 commit_staged on tables 1..N succeeded, or if the publisher's CAS
@@ -179,30 +268,37 @@ pre-check rejects *after* every commit_staged succeeded, tables 1..N
 are left at `Lance HEAD = manifest_pinned + 1`.
 
 **Recovery protocol** (lifecycle of every staged-write writer —
-`MutationStaging::finalize`, `schema_apply::apply_schema_with_lock`,
+`MutationStaging::commit_all`, `schema_apply::apply_schema_with_lock`,
 `branch_merge_on_current_target`, `ensure_indices_for_branch`,
 `optimize_all_tables`):
 
 1. **Phase A**: writer writes a sidecar JSON to
-   `__recovery/{ulid}.json` BEFORE its first HEAD-advancing commit
+   `__recovery/{ulid}.json` BEFORE its first independently durable physical
+   effect (including a first-touch Lance branch ref) or HEAD-advancing commit
    (`commit_staged`, or `compact_files` for `optimize_all_tables`,
    which advances the Lance HEAD via a reserve-fragments + rewrite
    commit rather than a staged write). The
    sidecar names every `(table_key, table_path, expected_version,
    post_commit_pin)` it intends to commit + the writer kind +
    actor_id.
+   For a first-touch named-branch Mutation/Load table, Phase A is followed by
+   target-ref creation and branch-local `stage_*`; the sidecar already carries
+   its pre-minted transaction identity.
 2. **Phase B**: writer's per-table `commit_staged` loop runs.
-   - **Phase-B confirmation (`BranchMerge` only)**: a `BranchMerge` writer
+   - **Phase-B confirmation:** a `BranchMerge` writer
      advances each table's HEAD by *several* commits (append → upsert →
      delete), so a bare "HEAD moved" is ambiguous — it could be a complete
      publish or one crashed mid-sequence. After the whole per-table loop
      finishes, the writer re-writes the sidecar stamping each pin's
      `confirmed_version` with the exact achieved version, then proceeds to
-     Phase C. This is the commit point of the recovery WAL: a crash *after*
-     confirmation rolls forward to those versions; a crash *during* Phase B
-     (sidecar still unconfirmed) rolls back. Other writers don't confirm —
-     their drift is derived state (index coverage, compaction) that a partial
-     roll-forward never corrupts.
+     Phase C. Schema-v3 Mutation/Load sidecars also confirm: each table must
+     match the staged Lance transaction's `(read_version, uuid)`, and the
+     sidecar records the exact `SubTableUpdate` plus original lineage intent.
+     This is the commit point of the recovery WAL: a crash *after* confirmation
+     rolls forward only when the captured branch token still matches; a crash
+     *during* Phase B (sidecar still unconfirmed) rolls back. Remaining legacy
+     writers don't confirm — their drift is derived state (index coverage,
+     compaction) that a partial roll-forward never corrupts.
 3. **Phase C**: publisher commits the manifest.
 4. **Phase D**: writer deletes the sidecar.
 
@@ -226,6 +322,18 @@ recovery sweep in `crates/omnigraph/src/db/manifest/recovery.rs`:
   `BranchMerge` sidecar, a moved HEAD with no `confirmed_version` classifies
   as `IncompletePhaseB` (a partial multi-commit publish) and forces roll-back;
   with a `confirmed_version`, roll-forward targets exactly that version.
+  Schema-v3 Mutation/Load additionally requires `EffectsConfirmed`, the exact
+  Lance transaction identity at the confirmed version, the original immutable
+  manifest delta, and a matching captured authority token. A changed token is
+  rollback-only; an unknown/foreign effect is refused rather than adopted.
+  An Armed first-touch intent with no owned transaction is deferred by live
+  roll-forward-only healing because another handle may still own it. Quiesced
+  full recovery tolerates an absent target ref (crash before fork), or removes
+  an exact unchanged unpublished ref after proving no other pending sidecar
+  claims it. With competing claims, the current no-effect sidecar discards
+  itself without touching the ref; the final survivor owns cleanup/recovery.
+  During partial rollback, no-effect refs are removed before the rollback
+  outcome is published so a retry cannot strand them.
 - If any table is `InvariantViolation` (Lance HEAD < manifest pinned —
   should be impossible), **abort** with a loud error and leave the
   sidecar on disk for operator review.
@@ -243,14 +351,14 @@ recovery sweep in `crates/omnigraph/src/db/manifest/recovery.rs`:
   each iteration. The audit row's `to_version` records the logical
   rolled-back-to version (`manifest_pinned`); the manifest is published at the
   restore commit (`manifest_pinned + 1`, same content).
-- After a successful roll-forward or roll-back, an audit row is
-  recorded — the graph commit lineage (the `graph_commit` rows in `__manifest`
-  since RFC-013 Phase 7) carries a commit tagged
-  `actor_id = "omnigraph:recovery"`, and a sibling
-  `_graph_commit_recoveries.lance` row carries `recovery_kind`,
-  `recovery_for_actor` (the original sidecar's actor), `operation_id`,
-  per-table outcomes. Operators run `omnigraph commit list --filter
-  actor=omnigraph:recovery` to find recoveries.
+- After a successful roll-forward or roll-back, an internal
+  `_graph_commit_recoveries.lance` row records `recovery_kind`,
+  `recovery_for_actor` (the original sidecar's actor), `operation_id`, and
+  exact per-table outcomes. A v3 roll-forward publishes the interrupted
+  writer's fixed lineage intent, including its original actor; rollback and
+  legacy recovery commits use `actor_id = "omnigraph:recovery"`. Ordinary
+  commit history is therefore not a complete recovery enumeration, and the
+  CLI currently has no public query for the recovery-audit table.
 - Sidecar deleted as the final step.
 
 Triggers for the residual: transient Lance write errors during finalize
@@ -264,13 +372,24 @@ contention exceeding `PUBLISHER_RETRY_BUDGET = 5` retries.
 Phase B → Phase C residual closes on the next write, without a
 restart and without an explicit refresh. The heal lists `__recovery/`
 (one `list_dir`; empty in the steady state) and, per sidecar, acquires
-the same per-`(table_key, table_branch)` write queues every sidecar
-writer holds from before `write_sidecar` until after `delete_sidecar` —
-so it serializes against a live writer instead of rolling its
+schema → branch → sorted-table gates that overlap the writer's guarded
+sidecar lifetime. RFC-022 mutation/load writers hold the complete order. Branch
+merge now holds schema plus source/target branch authority for its whole attempt
+and then the all-catalog source/target table envelope; other legacy adapters
+serialize through their existing table or schema gate until their own adapter
+slices land. The
+manager is shared by every
+`Omnigraph` handle for one canonical local root identity (relative, absolute,
+and symlink aliases converge; object-store/custom schemes stay opaque), so this
+also serializes a refresh or separately-opened handle against a live writer instead of rolling its
 in-flight sidecar forward from under it (a sidecar whose queues can be
 acquired belongs to a writer that finished or died; an existence
 re-check after the wait skips the finished case). Lock order is
-queues → coordinator, matching every writer's commit→publish path.
+schema → branch → sorted tables → coordinator, matching the writer effect path.
+Enrolled mutation/load attempts and branch merge perform one additional
+`list_dir` after acquiring their authority gates; that final check closes the
+pre-gate recovery TOCTOU without moving mutation/load validation or staged-file
+construction under the gate.
 Pinned by the four
 `tests/failpoints.rs::*_after_finalize_publisher_failure_heals_without_reopen`
 tests (load, mutation, schema apply, branch merge). The maintenance
@@ -280,8 +399,11 @@ Phase-B commit (dropping its rows), and a branch merge publishes the
 drift as an unattributed side effect — both while the stale sidecar
 lingers to misclassify later.
 Sidecars that would require a `Dataset::restore` (mixed / unexpected
-state) are deferred to the next `OpenMode::ReadWrite` open: restore is
-unsafe under concurrency because Lance's `check_restore_txn` accepts
+state) are deferred to the next `OpenMode::ReadWrite` open. Full open-time
+recovery uses the same root-scoped ordered gates and post-wait existence check,
+so it cannot Restore/delete under a live writer owned by another handle in the
+same process. Restore remains unsafe across processes because Lance's
+`check_restore_txn` accepts
 the restore against in-flight Append/Update/Delete commits and
 silently orphans them (pinned by
 `tests/staged_writes.rs::lance_restore_loses_to_concurrent_append_via_orphaning`).
@@ -289,17 +411,24 @@ When such a deferred sidecar blocks a write, the commit-time drift
 guard says so explicitly ("a pending recovery sidecar requires
 rollback — reopen the graph read-write") instead of pointing at
 `omnigraph repair`, which refuses while a sidecar is pending.
+`cleanup` refuses pending sidecars at entry as well, before orphan reconciliation
+or version GC: v3 ownership and compensation recovery may need the retained
+Lance transaction/version history, so garbage collection cannot outrun the
+recovery barrier.
 Continuous in-process recovery for the rollback path is the goal of a
-future background reconciler. `ensure_indices` does not heal at entry
-itself — it runs inside the load / schema-apply flows after their
-entry heal, and its strict preconditions still fail loudly on drift
-when invoked directly.
+future background reconciler. `ensure_indices` does not heal at entry itself;
+it is an explicit maintenance/reconciliation call, separate from mutation,
+load, and schema apply, and its strict preconditions fail loudly on drift.
 
-The publisher-CAS contract is unchanged: a *concurrent writer* that
-advances any of our touched tables between snapshot capture and
-publisher commit produces exactly one winner. The residual above is
-about *our* abandoned commits in the failure path, not about
-concurrency races.
+For enrolled mutation/load, the publisher rechecks the attempt's exact native
+branch identity and `graph_head` as well as the touched-table versions. A
+concurrent graph commit anywhere on the target branch therefore invalidates the
+prepared authority instead of silently reparenting it. Before effects, an
+insert-only mutation or Append/Merge load fully reprepares with a bounded retry; strict
+Update/Delete/Overwrite returns `ReadSetChanged`; after any effect, any later
+error returns `RecoveryRequired` and leaves the fixed v3 intent durable. Legacy
+writers still arbitrate only their explicit touched-table expectations until
+their adapters are enrolled.
 
 **Sidecar I/O failure semantics** (all sidecar I/O goes through the
 backend-generic `StorageAdapter`; the contracts below are pinned by the
@@ -341,19 +470,25 @@ error, not a silent `false`.
 
 ## Conflict shape
 
-Concurrent writers to the same `(table, branch)` produce exactly one
-success and one failure. The losing writer's error is
-`OmniError::Manifest` with kind `Conflict` and details
-`ManifestConflictDetails::ExpectedVersionMismatch { table_key, expected,
-actual }`. The HTTP server maps this to **409 Conflict** with body
-`{"error": "...", "code": "conflict", "manifest_conflict": { "table_key":
-"...", "expected": N, "actual": M }}` — see [docs/user/server.md](../user/operations/server.md).
+For mutation/load, a changed authority detected before effects is
+`ManifestConflictDetails::ReadSetChanged { member, expected, actual }`.
+Retryable Insert/Merge/Append attempts handle this internally by fully
+repreparing; strict writes surface **409 Conflict** with structured
+`read_set_conflict` details. A changed authority discovered after a physical
+effect, or any unresolved overlapping intent found at the synchronous recovery
+barrier, is `OmniError::RecoveryRequired { operation_id, … }`, mapped to **503
+Service Unavailable** with structured `recovery_required`; retry only after the
+sidecar has been resolved. Legacy, not-yet-enrolled writers may still surface
+`ExpectedVersionMismatch` and `manifest_conflict`.
 
-## Audit
+## Commit actor history
 
 `actor_id` lands in the graph commit lineage — the `graph_commit` rows in
 `__manifest`, written in the publish CAS (RFC-013 Phase 7; previously
-`_graph_commits.lance`). Audit history is queried via `omnigraph commit list`.
+`_graph_commits.lance`). Ordinary commit/actor history is queried via
+`omnigraph commit list`. Crash-recovery actions additionally live in the internal
+`_graph_commit_recoveries.lance` table described above; that exact recovery log
+does not yet have a public CLI query.
 
 ## Storage versioning (no in-place migration)
 

docs/rfcs/rfc-0018-ingest-wal.md

@@ -2,7 +2,7 @@
 type: spec
 title: "RFC-018 — Streaming-ingest WAL on Lance MemWAL"
 description: Adds a durability-first streaming ingest path (ack on WAL durability, asynchronous fold into the graph commit chain) built entirely on Lance's MemWAL primitive; reconciled against Lance v8.0.0 and the v9 beta line; analyzed for composition with the upstream multi-table-commit RFCs.
-status: draft
+status: superseded
 tags: [eng, rfc, wal, ingest, lance, omnigraph]
 timestamp: 2026-07-02
 owner:
@@ -10,12 +10,19 @@ owner:
 
 # RFC-018 — Streaming-ingest WAL on Lance MemWAL
 
-**Status:** Draft / for discussion
+**Status:** Superseded by [RFC-026](rfc-026-memwal-streaming-ingest.md)
 **Date:** 2026-07-02
 **Surveyed version:** 0.7.2 (branch `dst-extract-crate`); Lance pinned at 7.0.0
 **Upstream surveyed:** Lance v8.0.0 (released; RC votes closed 2026-07-01), v9.0.0-beta.10; MemWAL spec (`lance.org/format/table/mem_wal/`, fetched in full 2026-07-02); discussions #7260, #7264, #7222, #7176
 **Audience:** OmniGraph maintainers
 
+> **Supersession note (2026-07-10):** RFC-026 carries the streaming-ingest
+> design forward under RFC-022's unified graph-write protocol. It also corrects
+> this draft's characterization of MemWAL: MemWAL is a strategic Lance
+> architecture and a major substrate investment, not an experimental direction.
+> The integration risk is its evolving API and format surface across Lance
+> releases.
+
 ---
 
 ## 0. TL;DR
@@ -125,8 +132,10 @@ fencing primitive.
 ## 3. Substrate inventory — what Lance provides and how we use all of it
 
 Per the lance.md protocol the MemWAL spec and adjacent pages were fetched in
-full (2026-07-02). The spec is **experimental** upstream — risk register in
-§10. Inventory, mapped to consumption:
+full (2026-07-02). MemWAL is a strategic Lance architecture with substantial
+upstream investment. Its API and format surface are still evolving across
+releases; §10 treats that maturity boundary as an integration risk. Inventory,
+mapped to consumption:
 
 | Lance tooling | Spec/PR | How this RFC uses it |
 |---|---|---|
@@ -323,8 +332,8 @@ this RFC phases on:
   matures it may fit *small in-place updates* better than WAL-upsert-fold;
   watch-listed as a possible Phase 4 refinement, not a dependency.
 - MemWAL fixes keep landing on v9 betas (#7489 cross-generation block-list on
-  in-memory scan arms) — confirming the experimental-spec churn risk (§10)
-  and the value of keeping our exposure transient-state-only.
+  in-memory scan arms) — confirming that its API/format integration surface is
+  still moving (§10) and the value of keeping our exposure transient-state-only.
 
 ## 7. Composition with upcoming Lance multi-table commits
 
@@ -464,8 +473,9 @@ participates in publication authority.
 
 ## 10. Risks
 
-- **MemWAL is experimental upstream** (spec banner; live format votes —
-  #7418 Status field mid-2026-06). Mitigation is structural: WAL state is
+- **MemWAL's API and format surface continues to evolve.** This risk concerns beta-era
+  API churn, not architectural commitment: Lance has made MemWAL a strategic
+  streaming-write investment. Mitigation is structural: WAL state is
   *transient* (folded then GC'd), so a format change between Lance versions
   can be handled by fold-to-quiescent before the bump; no long-lived on-disk
   state depends on the MemWAL format. This must stay true — resist any

docs/rfcs/rfc-0019-heads-and-fences.md

@@ -2,7 +2,7 @@
 type: spec
 title: "RFC-019 — Heads and Fences: structural O(1) writes without a warm-cache truth fork"
 description: Replaces the warm-publish/pinned-open machinery of PR #318 with two structural changes — durable per-table head rows ("refs, not replay") and substrate-native key-conflict fencing (Lance's unenforced-PK KeyExistenceFilter) — landing together as internal schema v5; composes with RFC-018 (WAL ingest) and the upstream multi-table-commit direction.
-status: draft
+status: superseded
 tags: [eng, rfc, write-path, manifest, lance, omnigraph]
 timestamp: 2026-07-04
 owner:
@@ -10,12 +10,21 @@ owner:
 
 # RFC-019 — Heads and Fences
 
-**Status:** Draft / for discussion
+**Status:** Superseded by [RFC-023](rfc-023-key-conflict-fencing.md) and [RFC-024](rfc-024-durable-table-heads.md)
 **Date:** 2026-07-04
 **Surveyed:** omnigraph `main` @ 98530a0e (0.8.0); Lance pinned 7.0.0 (+ vendored lance-table carrying lance#7480); upstream Lance v8.0.0 (released 2026-07-01), v9.0.0-beta.15; PR #318 at `2aab48ba` (reviewed 2026-07-04, 8 verified findings)
 **Companion docs:** RFC-018 (streaming-ingest WAL), PR #318's plan doc (`unlimited-history-latency-plan.md`, whose §9 "U2" this RFC promotes from follow-up to prerequisite)
 **Audience:** OmniGraph maintainers
 
+> **Supersession note (2026-07-10):** key-conflict fencing and durable table
+> heads are separate irreversible decisions with different substrate gates and
+> rollout requirements. RFC-023 and RFC-024 review them independently under
+> RFC-022's shared graph-write protocol. RFC-023 also corrects this draft's
+> load-bearing symmetry claim: on the surveyed Lance revision, an unfiltered
+> current transaction (including bare `Append`) can rebase after a filtered
+> update. Fencing therefore requires both transaction orders plus a fleet
+> compatibility barrier; it is not “unblocked for new tables” in a mixed graph.
+
 ---
 
 ## 0. TL;DR

docs/rfcs/rfc-022-unified-write-path.md

@@ -0,0 +1,773 @@
+---
+type: spec
+title: "RFC-022 — Unified graph-write protocol"
+description: One correctness protocol for graph-visible writes, with synchronous recovery, complete read-set arbitration, writer-specific physical-effect adapters, and explicit control-plane exceptions.
+status: draft
+tags: [eng, rfc, write-path, manifest, recovery, concurrency, lance, omnigraph]
+timestamp: 2026-07-10
+owner:
+---
+
+# RFC-022: Unified graph-write protocol
+
+**Status:** Draft / for team review
+**Date:** 2026-07-10
+**Surveyed:** OmniGraph 0.8.1 (`main`); Lance 9.0.0-beta.15, git rev `f24e42c1`
+**Audience:** engine and storage maintainers
+**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md).
+Findings marked **BLOCKER** must be dispositioned before acceptance.
+
+---
+
+## 0. Summary
+
+OmniGraph will have one **correctness protocol** for every operation that changes
+manifest-resolved graph state. The protocol does not require every writer to use
+the same Lance primitive. A mutation can commit one staged transaction per table,
+a branch merge can make several commits to a table, and optimize can use Lance
+operations that have no staged API. Writer-specific **effect adapters** describe
+those physical operations; one coordinator enforces the common safety rules.
+
+A graph-visible write follows this state machine:
+
+```text
+recovery barrier
+    → prepare pinned base + complete ReadSet + Effects
+    → acquire ordered process-local gates
+    → revalidate the complete ReadSet, or restart
+    → durably arm recovery
+    → apply writer-specific physical effects
+    → publish exactly one graph-visible __manifest CAS
+    → finalize derived state
+```
+
+The protocol has three hard boundaries:
+
+1. Recovery is a synchronous pre-write safety barrier. It may also run in the
+   background, but a writer never proceeds merely because recovery was scheduled.
+2. Validation and merge classification are valid only for their complete read set.
+   A changed probed table causes revalidation or a full restart; the writer never
+   refreshes expected versions underneath a plan computed from an older base.
+3. “One `__manifest` CAS” applies only to graph-visible commits. Native graph-branch
+   ref creation/deletion and physical-only maintenance have explicit, smaller
+   control protocols and do not manufacture graph commits.
+
+This RFC deliberately does not combine key fencing, durable table heads,
+checkpoint retention, MemWAL ingest, or lineage-based merge-delta discovery into
+one format and rollout. They are focused follow-ups:
+
+- [RFC-023 — Key-conflict fencing](rfc-023-key-conflict-fencing.md)
+- [RFC-024 — Durable table heads](rfc-024-durable-table-heads.md)
+- [RFC-025 — Checkpoint retention](rfc-025-checkpoint-retention.md)
+- [RFC-026 — MemWAL streaming ingest](rfc-026-memwal-streaming-ingest.md)
+- [RFC-027 — Lineage merge deltas](rfc-027-lineage-merge-deltas.md)
+
+## 1. Scope and authority
+
+### 1.1 Graph-visible writes
+
+A **graph-visible write** changes state resolved through a graph manifest snapshot:
+
+- a node or edge table version;
+- a registered or tombstoned table;
+- accepted schema identity or schema-visible table metadata;
+- a graph commit or graph head;
+- any future logical marker that changes query or time-travel semantics.
+
+For such an operation, the only visibility point is one successful `__manifest`
+commit containing the entire graph delta. Per-table Lance commits before that point
+are physical effects covered by recovery; they are not independently graph-visible.
+
+### 1.2 Control operations
+
+Two classes are not graph-visible commits:
+
+1. Native graph-branch ref create/delete. Lance stores these refs outside the
+   dataset-version chain and creates no new `__manifest` version for either action.
+2. Physical-only maintenance whose result is content-equivalent and is not selected
+   through a data-table pointer in `__manifest`, such as compacting `__manifest`
+   itself or reclaiming unreachable files.
+
+Sections 7 and 8 define their control protocols. Branch **merge** and a data-table
+optimize whose new version must be published are graph-visible writes and remain in
+the main protocol.
+
+### 1.3 What “unified” means
+
+Unified means one set of safety obligations, one coordinator state machine, and one
+closed registry of effect adapters. It does **not** mean:
+
+- one storage trait method for every Lance operation;
+- exactly one physical commit per table;
+- pretending native refs are manifest rows;
+- treating process-local queues as a distributed transaction manager;
+- moving commit recovery out of the correctness path.
+
+## 2. Protocol objects
+
+The names below are conceptual. Implementations may choose different Rust names,
+but they must preserve the represented information and transitions.
+
+```rust
+struct PreparedWrite {
+    operation_id: OperationId,
+    writer_kind: WriterKind,
+    target: BranchTarget,
+    base: BaseView,
+    read_set: ReadSet,
+    effects: Effects,
+    manifest_delta: ManifestDelta,
+    lineage_intent: Option<LineageIntent>,
+    recovery: RecoveryPlan,
+}
+```
+
+### 2.1 `BaseView`
+
+`BaseView` is the immutable state against which the operation was computed. It
+contains at least:
+
+- target manifest branch and its incarnation/freshness token;
+- pinned manifest version;
+- accepted schema identity;
+- pinned table entries used by the operation;
+- graph head when parentage, merge base, or branch semantics depend on it.
+
+The base is captured only after the recovery barrier completes. A recovery pass may
+advance the manifest or promote schema state, so a base captured before recovery is
+not valid write input.
+
+### 2.2 `ReadSet`
+
+`ReadSet` is every authority value whose stability is required for the prepared
+result to remain correct. It includes, as applicable:
+
+- target branch incarnation;
+- accepted schema identity;
+- graph head;
+- the table head for every table written;
+- the table head for every table probed by uniqueness, referential-integrity,
+  cardinality, policy-independent structural validation, or merge classification;
+- writer-specific authority, such as an enrolled stream's configuration and merge
+  generation.
+
+A table belongs in the read set because its value affected the decision, not because
+the writer happens to update it. Read-only dependencies are load-bearing.
+
+The publisher must arbitrate every read-set member atomically with the manifest
+commit. A fresh pre-check followed by an unconditional write is not a CAS. The
+implementation must ensure that a concurrent change after the check contends on a
+stable authority row or equivalent substrate token. If the current representation
+cannot arbitrate a read-only dependency, that writer has not completed this RFC and
+must not ship the outside-gate validation optimization.
+
+### 2.3 `Effects`
+
+`Effects` is an adapter-owned physical plan. Each effect declares:
+
+- physical dataset and Lance branch/ref it targets;
+- expected pre-state;
+- whether it is staged, inline, ref-only, or zero-commit;
+- the possible post-state shape: exact version, bounded range, or adapter-confirmed
+  version;
+- whether rollback is safe;
+- how recovery distinguishes no movement, partial movement, completed movement,
+  and an already-published result.
+
+The generic coordinator never assumes `expected_version + 1`. That assumption is
+valid for some mutation/load effects and false for branch merge, compaction, index
+work, schema metadata changes, and no-op plans.
+
+### 2.4 `ManifestDelta`
+
+`ManifestDelta` is the complete logical result to publish. It contains table-version
+journal entries, registrations/tombstones, mutable logical heads when present, schema
+identity, and lineage rows as required by the operation. Its logical contents are
+immutable for one prepared attempt. Physical version fields may be declared output
+slots that the adapter binds to an allowed, confirmed effect result; binding such a
+slot cannot widen or otherwise change the logical plan. A revalidation mismatch
+discards the delta and restarts preparation.
+
+### 2.5 `RecoveryPlan`
+
+`RecoveryPlan` supplies the writer-specific classifier and compensation/roll-forward
+rules. It must be serializable into a recovery sidecar before the first independently
+durable physical effect. It is part of the commit protocol, not a best-effort
+maintenance hint.
+
+## 3. Normative invariants
+
+1. **Recovery before base capture.** Every graph-visible writer runs and awaits the
+   recovery barrier before pinning `BaseView`.
+2. **No durable effect before durable intent.** On an existing physical ref,
+   reclaimable uncommitted files may be staged first. A first-touch named-table
+   transaction is different: Lance writes its uncommitted files into the opened
+   branch tree, so Prepare retains the logical batch/predicate and pre-mints its
+   transaction identity; after revalidation the recovery sidecar becomes durable,
+   then the writer creates the target ref and stages those branch-local files under
+   the held gates. In every case the sidecar precedes the first independently
+   persistent physical effect, including a native table ref, data-table HEAD
+   advance, or native tag/index mutation needed by the graph-visible result.
+3. **Complete read-set validation.** Physical effects may start only while the fresh
+   authority state equals the complete `ReadSet` used to prepare the operation.
+4. **Recompute, do not patch.** On a pre-effect mismatch, discard the prepared plan
+   and rerun validation/classification. Never refresh expected versions beneath an
+   old plan.
+5. **One graph visibility point.** A graph-visible write publishes one manifest CAS;
+   no subset of its table effects becomes graph-visible independently.
+6. **Adapters own effect truth.** Every effect-producing writer uses a registered
+   adapter. There is no generic fallback that guesses version movement or recovery
+   safety.
+7. **Queues are local optimization.** Ordered process-local gates prevent avoidable
+   in-process races and deadlocks. Cross-process correctness comes from Lance
+   conflicts, manifest arbitration, and recovery.
+8. **Recovery safety is synchronous.** Background sweeping is permitted, but a later
+   overlapping writer waits for, performs, or fails on unresolved recovery before it
+   can advance state.
+9. **Derived work follows visibility.** Expensive index reconciliation, cache warming,
+   and orphan reclaim may run asynchronously only when logical correctness does not
+   depend on their completion.
+
+These refine, and do not weaken, invariants 2, 3, 4, 5, 7, 9, 13, and 15 in
+[`docs/dev/invariants.md`](../dev/invariants.md).
+
+## 4. The graph-write state machine
+
+### 4.1 Stage A — recovery barrier
+
+Before accepting a base, the coordinator discovers pending recovery intents that can
+overlap this operation. It must reach one of three outcomes:
+
+1. all relevant intents are fully resolved;
+2. each remaining intent is proven already satisfied and can be finalized safely;
+3. the write fails with a typed recovery-required or live-writer-contention error.
+
+“Spawn recovery and continue” is not an outcome. Listing sidecars may run concurrently
+with non-authoritative I/O, but preparation cannot accept its base until recovery has
+completed and the post-recovery schema/manifest state is available.
+
+Recovery must not destructively act on a sidecar that may belong to a live writer
+without ownership/fencing proof. It waits, returns contention, or uses a protocol that
+proves the prior owner cannot continue.
+
+Recovery classifiers use exact effect identity or confirmed post-state. A numeric
+test such as `manifest_version >= observed_lance_head` is sufficient only when the
+adapter independently proves lineage containment; version ordering alone is not that
+proof. Exact identity cuts both ways: a roll-forward that loses its manifest CAS to
+a concurrent writer which already published this intent's exact goal state is
+**convergence, not failure** — the barrier records the audit outcome and removes the
+intent rather than failing the write (preserving the concurrent-advance convergence
+behavior fixed in #296). "Exact" forbids adopting an unrelated newer version; it does
+not forbid recognizing one's own goal already achieved.
+
+The barrier is a deliberate availability trade, stated plainly: an unresolvable
+overlapping recovery intent — including a live foreign writer's sidecar that cannot
+be fenced — fails every overlapping write with a typed error until resolved. Safety
+outranks availability here by design; operators observe the condition through the
+typed error and recovery telemetry rather than through silently degraded writes.
+
+### 4.2 Stage B — prepare
+
+Prepare runs without writer gates. It may:
+
+- capture `BaseView`;
+- evaluate the complete constraint set;
+- classify a merge;
+- compute embeddings or other deterministic payloads;
+- build staged Lance transactions and reclaimable uncommitted objects for targets
+  whose physical refs already exist;
+- for a first-touch named-table target, retain the complete logical stage input and
+  pre-mint its recovery transaction identity without creating the ref;
+- construct the complete `ReadSet`, `Effects`, `ManifestDelta`, and `RecoveryPlan`.
+
+Prepare must not advance a Lance HEAD or the graph's native branch refs. Staged files
+that are not referenced by a committed Lance manifest are permitted and are
+reclaimable if the attempt restarts.
+
+After Stage E arms recovery, a first-touch adapter may create its declared target ref
+under the held gates, stage branch-local files on that ref, bind the resulting Lance
+transaction to the pre-minted identity, and only then advance HEAD. The ref is itself
+an independently durable effect covered by the sidecar; recovery must classify both
+sidecar-before-ref and ref-before-HEAD crash states.
+
+Every validator registers its actual committed-state probes in `ReadSet`. Key fences
+are an additional same-key conflict signal; they do not replace read-set arbitration
+for non-key uniqueness, RI, cardinality, or merge-target stability.
+
+### 4.3 Stage C — acquire ordered gates
+
+The prepared effects declare their gate keys. One attempt acquires the complete set in
+this total order:
+
+1. durable global operation claims (migration, retention, or future claims) by
+   deterministic claim key;
+2. graph/schema-control key, if required;
+3. target branch-control key, if required;
+4. `(physical_branch, table_key)` keys in deterministic sorted order.
+
+The gates are held through revalidation, recovery arming, physical effects, and the
+manifest visibility decision. They may be released after a successful manifest CAS or
+after a failed post-effect attempt has safely left its sidecar for recovery.
+
+Acquiring a global claim is coordination, not a graph effect for §3's sidecar-order
+rule: the claim record must itself contain an owner/fencing token and an explicit
+crash-release/takeover contract. No data HEAD, tag, index, or logical authority may
+move merely because the claim was acquired.
+
+The merge-exclusive mutex may protect a coordinator swap, but it is not a semantic
+cross-process lock and must not substitute for a target read set.
+
+### 4.4 Stage D — revalidate or restart
+
+With all gates held, the coordinator loads fresh authority state and compares every
+member of `ReadSet`.
+
+- If all members match, the attempt may arm recovery.
+- If any member differs, no physical effect may run. The attempt releases its gates,
+  discards staged state, and restarts from Prepare.
+- A strict API may return a typed conflict instead of retrying, but it may not publish
+  the stale plan.
+
+Retries are bounded and observable. Retrying a merge means recomputing the merge base
+and reclassifying against the new target. Retrying a validation-sensitive mutation
+means rerunning the validators, including probes of tables the mutation does not
+write.
+
+### 4.5 Stage E — arm recovery
+
+After successful revalidation, write the recovery sidecar durably before the first
+independently durable physical effect. An effect-free or authority-first workflow
+described in §4.9 may omit the sidecar. Otherwise the sidecar contains at least:
+
+- operation id, writer kind, actor, and target branch/incarnation;
+- pinned schema identity and complete read set;
+- every physical target and expected pre-state;
+- adapter recovery strategy;
+- intended manifest delta and lineage intent, or a durable reference to them;
+- confirmed post-state once the adapter reaches its all-effects-complete boundary.
+
+A multi-step adapter first records its pre-state plan. After all its physical effects
+finish, it durably records the exact confirmed post-state before manifest publish.
+Until that confirmation exists, recovery treats the effect set as possibly partial.
+
+### 4.6 Stage F — apply effects
+
+The adapter applies its declared physical effects. A failure after recovery is armed
+leaves the sidecar intact. The request must not delete it, silently adopt live HEAD, or
+start a fresh plan around it.
+
+The adapter returns exact achieved state to bind the physical output slots declared by
+`ManifestDelta`. If achieved state differs from the prepared effect envelope, the
+operation fails into recovery rather than widening its plan in place.
+
+### 4.7 Stage G — manifest CAS
+
+A graph-visible operation performs exactly one `__manifest` CAS carrying its complete
+logical delta and lineage when the plan has `LineageIntent`. Metadata-only plans carry
+their explicit authority/operation rows without manufacturing graph lineage. On every
+CAS attempt, the commit authority re-reads and arbitrates the complete `ReadSet`.
+
+The following cases are distinct:
+
+- **Conflict before any physical effect:** safe bounded restart from Prepare.
+- **Conflict after physical effects:** recovery case. Keep the sidecar; do not simply
+  re-stage or point the manifest at whatever HEAD is now live.
+- **CAS success:** the graph commit is visible atomically.
+
+When durable table heads land under RFC-024, tombstoning a table must update its mutable
+head to an explicit deleted state in this same CAS. A stale live head plus an immutable
+tombstone history is not a valid O(tables) current-state representation.
+
+### 4.8 Stage H — finalize
+
+After CAS success:
+
+- delete the sidecar best-effort when the workflow has one;
+- refresh/invalidate process-local views;
+- enqueue derived index reconciliation and orphan reclaim;
+- record recovery/audit completion when applicable.
+
+Sidecar deletion failure does not turn a durable successful graph commit into a user
+error. The next recovery barrier proves the exact intent satisfied and removes the
+artifact.
+
+### 4.9 Authority-first control workflows
+
+Some metadata workflows have no independently durable physical effect before their
+manifest CAS. They may use an **authority-first** subtype:
+
+1. run the recovery barrier, prepare, acquire gates/claims, and revalidate exactly as
+   above;
+2. publish the metadata transition as the first durable effect;
+3. perform only idempotent work whose desired target is fully encoded by that
+   transition and whose interruption cannot expose incorrect graph data.
+
+No generic sidecar is required before step 2 because there is no pre-authority effect
+to recover. The authority row itself is the durable recovery cursor. Checkpoint
+deletion, a GC-boundary publish, and `OPEN -> DRAINING` stream intent are candidate
+examples; each follow-up RFC must prove its post-CAS work is convergent and safe.
+
+A long control workflow is not one giant RFC-022 attempt. Each lifecycle transition,
+fold, schema/branch operation, and resume transition is a separate prepared write or
+native-ref control step with its own read set and visibility point. If a later phase
+needs a non-idempotent or independently visible effect not fully described by the
+authority row, that phase uses a normal sidecar before the effect.
+
+## 5. Crash contract
+
+| Crash point | Required result |
+|---|---|
+| Before sidecar | No independently durable effect occurred; uncommitted objects are reclaimable. |
+| Sidecar durable, no effect | Recovery aborts/finalizes the empty intent. |
+| Some effects applied, not confirmed | The adapter rolls back, completes, or refuses safely according to its declared strategy. |
+| All effects confirmed, before manifest CAS | Recovery rolls forward the exact confirmed manifest delta, or applies the adapter's explicit all-or-nothing rule. |
+| Manifest CAS succeeded, sidecar remains | Recovery proves the exact intent visible, audits it, and removes the sidecar. |
+
+Rollback is not assumed safe. Lance `Restore`, schema-file promotion, native refs, and
+content-replacing operations have different concurrency properties; each adapter must
+state which recovery direction is legal and under what fencing.
+
+## 6. Writer-effect adapters
+
+The adapter registry is closed by default: adding a graph-visible writer requires a
+new adapter or an explicit use of an existing adapter. Code review and tests must be
+able to enumerate every adapter and every entry point that invokes it.
+
+### 6.1 Mutation and load
+
+- Construct one staged effect per touched table where the Lance API permits it.
+  Existing-table effects may stage in Prepare; first-touch named-table effects use
+  the sidecar → target ref → branch-local stage ordering above.
+- Put every uniqueness, RI, and cardinality probe in `ReadSet`.
+- Revalidate or restart when a probed-but-untouched table changes.
+- Preserve strict replacement semantics for overwrite/delete.
+- Treat key-conflict fencing and strict keyed Append semantics as RFC-023 concerns;
+  no fence is credited as protection until that RFC's rollout gates pass.
+
+### 6.2 Branch merge
+
+- Compute row classification outside the gates.
+- Include the target graph head and every target table used by classification or
+  validation in `ReadSet`.
+- Any target change before effects forces a complete reclassification; publishing a
+  result computed against an old target and parenting it to a new live head is
+  forbidden.
+- The adapter supports zero, one, or several physical commits per table and records
+  exact confirmed post-state before manifest publish.
+- Lineage-based candidate discovery may replace the classifier only under RFC-027;
+  this protocol does not assume it is O(delta).
+
+### 6.3 Schema apply and storage migration
+
+- Acquire the schema-control gate before effect application.
+- Include accepted schema identity and every affected table in `ReadSet`.
+- Cover schema staging-file promotion, data-table schema/field-metadata commits,
+  registrations, tombstones, and final schema identity with one recovery intent.
+- Write the sidecar before the first table HEAD advance, including unenforced-PK
+  metadata backfill or other inline metadata commits.
+- A branch-wide or graph-wide migration must enumerate every physical manifest/data
+  branch it changes; updating main does not implicitly migrate older branch manifests.
+
+### 6.4 Data-table optimize and index work
+
+- The adapter may describe zero or multiple inline, content-preserving Lance commits.
+- It records the exact achieved version rather than assuming one version of movement.
+- If the new data-table version is selected through `__manifest`, publishing that
+  pointer is a graph-visible commit and uses this protocol.
+- Logical operations never fail because a derived index is absent or behind.
+- Physical-only internal-table maintenance remains the exception in Section 8.
+
+### 6.5 MemWAL fold
+
+RFC-026 owns enrollment, acknowledgement, quiescence, fresh-read semantics, and the
+public ingest surface. Any fold that becomes graph-visible is an adapter here:
+
+- fold-time validation contributes its complete read set;
+- Lance `merged_generations` changes atomically with the base-table data commit;
+- the sidecar covers the data-commit-to-manifest gap;
+- one successful manifest CAS makes the folded graph state visible.
+
+## 7. Native graph-branch ref control protocol
+
+Creating or deleting a graph branch mutates a native Lance branch ref for
+`__manifest`. Lance specifies that these operations do not generate a dataset version.
+There is no target branch on which to publish before create, and no target remains on
+which to publish after delete. They therefore cannot truthfully be instances of the
+graph-visible manifest-CAS protocol.
+
+Their control protocol is:
+
+1. run and await the recovery barrier;
+2. quiesce enrolled streams as required by RFC-026;
+3. acquire any active global claim, such as RFC-025's retention claim, and then
+   the graph/branch-control gate in §4.3 order;
+4. freshly revalidate source ref, target existence, and branch incarnation;
+5. perform one native Lance ref mutation, which is the visibility point;
+6. release the gate and reclaim orphaned per-table forks asynchronously.
+
+Delete has one recovery disposition that create does not: after the complete
+schema/target-branch/all-table gate set has waited out any live in-process owner,
+an unresolved sidecar scoped to the branch being removed may be rendered
+unreachable by the native ref deletion. A later heal records the orphan-discard
+audit and retires it. Graph-global schema recovery still blocks the control, and
+create/merge may not adopt this exception.
+
+The native ref operation itself should enforce the freshly checked precondition or
+surface concurrent ref mutation as a conflict — but at the pinned Lance revision it
+does not: branch-ref creation is an existence check followed by an unconditional
+put, not a conditional primitive (the same fact for which RFC-025 §2.3 rejects a
+branch ref as a claim mechanism). Until Lance ships a conditional/CAS ref mutation,
+graph-branch create/delete therefore inherit the documented single-writer-process
+support boundary — the same disposition RFC-023 §10 applies to recovery ownership —
+and multi-process branch operations are not advertised. The upstream ask for a
+conditional ref primitive is filed alongside this RFC; a process-local branch gate
+remains a local optimization, not the missing cross-process guarantee.
+
+These operations do not emit a synthetic graph commit. If a future product contract
+requires a native ref mutation and manifest/audit rows to become atomic together, it
+needs a separate multi-authority recovery protocol; this RFC does not claim an
+atomicity the substrate does not provide.
+
+This exception applies only to graph-level create/delete. Branch merge is a
+graph-visible write. Lazy per-table forks created while preparing a branch write are
+declared physical effects of that writer and remain subject to its recovery/reclaim
+contract.
+
+## 8. Physical-maintenance control protocol
+
+Physical work that does not change manifest-resolved logical graph state does not
+create graph lineage merely to fit the main protocol. Examples include:
+
+- compacting `__manifest`, which is itself the authority and is read at its Lance HEAD;
+- deleting versions/files already proven unreachable under the active retention
+  contract;
+- reclaiming orphaned branch refs or uncommitted objects;
+- rebuilding derived physical state when no graph-visible data-table pointer changes.
+
+Such work must still be idempotent, bounded, observable, and safe under concurrent
+native Lance commits. It uses substrate conflict/retry semantics appropriate to the
+operation. It must never expose partial logical graph state.
+
+The exception is from graph-lineage publication, not from recovery safety. Maintenance
+must run the recovery barrier or refuse before it can replace or delete an artifact
+named by an unresolved sidecar. It must also acquire any relevant process-local gates;
+as elsewhere, those gates are an optimization rather than cross-process authority.
+
+Data-table compaction/index work that advances a version which graph reads must select
+through `__manifest` is not exempt; Section 6.4 applies. Checkpoint reachability and
+the mapping from graph checkpoint rows to Lance-native GC pins belong to RFC-025.
+
+## 9. Concurrency and retry semantics
+
+Process-local gates reduce same-process races and establish one deadlock-free order.
+They do not coordinate two servers or CLIs. A conforming implementation remains safe
+if every process has its own gate manager.
+
+Cross-process safety comes from:
+
+- complete read-set arbitration at the manifest authority;
+- Lance transaction conflicts for physical table effects;
+- durable recovery before physical HEAD movement;
+- refusal to continue past unresolved overlapping recovery.
+
+Retry rules are phase-specific:
+
+- before effects, a read-set mismatch discards and recomputes the whole attempt;
+- while applying an effect, the adapter may retry only when Lance guarantees the
+  operation can be safely replanned from fresh physical state;
+- after any effect, manifest contention is resolved through the armed recovery intent,
+  not by silently rebasing the logical plan;
+- retry exhaustion returns a typed, observable conflict.
+
+## 10. Rollout and compatibility
+
+This RFC authorizes a protocol refactor, not a manifest v5 format moment. RFC-023
+through RFC-027 own their respective format and public-surface changes.
+
+It also does not require a mutable-tip `GraphState` singleton. Three measured,
+local latency fixes can land independently of the adapter conversion:
+
+1. make the graph's shared Lance `Session` a required parameter of every
+   manifest open/publisher path, so remote opens do not rebuild clients and cold
+   metadata state;
+2. capture one immutable operation-local manifest/lineage view and pass it down
+   the call stack instead of reopening the same state repeatedly; and
+3. remove the verified-redundant branch-idle refresh and the back-to-back second
+   `branch_delete_as` refresh once their existing coverage asserts unchanged
+   behavior.
+
+These are narrow access-shape fixes, not a second commit-input authority. They
+must preserve snapshot pinning and still cross the recovery/read-set barriers
+defined above.
+
+Implementation proceeds in this order:
+
+1. Introduce `PreparedWrite`, `ReadSet`, effect-adapter, and recovery-plan concepts
+   while preserving existing behavior.
+2. Ship conservative branch-wide arbitration first. Mutation/load captures
+   `(Lance BranchIdentifier, exact optional graph_head, accepted schema identity)`;
+   every publisher retry compares that token instead of reparenting. Because every
+   supported graph-content and schema apply advances `graph_head:<branch>` before
+   schema promotion, the shared head row atomically arbitrates probed-but-untouched
+   same-branch dependencies. The native branch identifier detects delete/recreate
+   ABA under the documented single-writer-process branch-control boundary. RFC-024
+   later narrows false contention with table heads; it is not a correctness
+   prerequisite for this coarse step. Existing live committed-state validation
+   probes remain until the narrowed read set replaces them.
+
+   > **Implementation note (2026-07-11):** mutation/load now use this coarse
+   > token, schema-v3 exact-effect sidecars, fixed lineage/rollback outcome ids,
+   > zero transparent Lance commit retries, and bounded full reprepare before
+   > effects. Branch merge remains on its writer-specific multi-commit path, but
+   > now holds the root-shared schema plus source/target branch gates from its
+   > strict recovery barrier and authority capture through publication. It plans
+   > with an accepted-contract catalog captured under that schema gate, then
+   > acquires all catalog table gates for source and target, re-lists recovery,
+   > and compares fresh manifest versions before Phase A. This closes
+   > same-process delete/recreate ABA and legacy table-only-writer races while
+   > its full exact-effect adapter remains future work. Schema apply,
+   > optimize/index, and MemWAL fold remain on their writer-specific paths until
+   > their adapter slices land.
+3. Convert mutation/load, branch merge, schema apply/migration, data-table optimize,
+   and graph-visible index work one adapter at a time.
+4. Add static or runtime enumeration proving no graph-visible entry point bypasses the
+   coordinator.
+5. Delete superseded writer-specific orchestration only after its crash and
+   concurrency cells pass through the adapter.
+6. Optimize background recovery latency only after the synchronous barrier and all
+   recovery classifications remain intact.
+
+Mixed writer binaries are not made safe by process-local gates. A deployment may
+enable the new protocol only when every writer that can reach the graph obeys the same
+sidecar and manifest-arbitration contract, or when a compatibility gate rejects older
+writers.
+
+## 11. Required tests and cost gates
+
+### 11.1 Protocol conformance
+
+- Enumerate every graph-visible entry point and its adapter.
+- Assert no sidecar-backed adapter can create an independently durable physical
+  effect before its sidecar is durable.
+- Enumerate authority-first workflows and assert their CAS is the first durable
+  effect and every post-CAS action is idempotently derivable from its authority row.
+- Assert one graph-visible operation produces exactly one manifest visibility commit.
+- Assert branch create/delete and physical-maintenance exceptions produce no synthetic
+  graph lineage.
+
+### 11.2 Read-set races
+
+- Two distinct ids racing on the same non-key `@unique` value cannot both publish.
+- An edge insert racing deletion of an endpoint must revalidate or conflict.
+- Cardinality probes of an untouched table participate in arbitration.
+- A target advance after merge classification forces complete reclassification.
+- Run the same cells with separate `Omnigraph` handles sharing one root-scoped
+  process-local gate manager, then with separate processes that do not share it.
+
+### 11.3 Recovery
+
+- Fail before sidecar, after sidecar, after each physical effect, after confirmation,
+  after manifest CAS, and during sidecar deletion for every adapter.
+- A later overlapping writer blocks, heals, or returns recovery-required; it never
+  advances around the sidecar.
+- A live foreign writer's sidecar is not destructively recovered without fencing.
+- Recovery proves exact effect identity; a numerically newer unrelated version is not
+  accepted as proof of ancestry.
+
+### 11.4 Adapter-specific truth tables
+
+- Mutation/load: zero/one table, multi-table, strict, non-strict, and
+  probed-but-untouched dependency cases.
+- Merge: adopt, rewrite, multi-commit, no-op, target advance, and partial-Phase-B crash.
+- Schema: staging files, registration-only, metadata HEAD advance, partial multi-table
+  migration, and branch-local state.
+- Optimize/index: zero, one, and several Lance commits, including monotonic publish and
+  retryable physical contention.
+- MemWAL fold, when RFC-026 lands: merged-generation conflict and every fold crash
+  boundary.
+
+### 11.5 Cost gates
+
+The protocol must not move validation, classification, embedding computation, or
+staged-file construction under process-local gates. Measure gate hold time separately
+from Prepare. Correctness gates precede latency optimization: a cost regression can
+delay rollout, but it cannot justify touched-table-only validation or asynchronous
+recovery safety.
+
+## 12. Invariants and deny-list check
+
+This design reinforces the existing architecture:
+
+- Lance and `__manifest` remain the sources of truth; `PreparedWrite` is immutable
+  attempt-local state, not a shadow mutable tip.
+- Graph visibility remains manifest-atomic.
+- Recovery remains part of the commit protocol.
+- Logical constraints fail loudly and are revalidated when their inputs change.
+- Derived indexes and reclaim work converge without becoming logical commit points.
+- No custom WAL, transaction manager, buffer pool, or distributed lock is introduced.
+
+The design rejects two tempting deny-list violations: treating process-local queues as
+distributed correctness, and treating recovery as derivable background work that a
+new writer may outrun.
+
+Acceptance should also add one clarification to invariant 15: a view of immutable,
+version-pinned state may be cached, while an in-memory view of the mutable tip is only
+a hint. Every use of mutable-tip state as write input must be re-arbitrated by the
+commit authority. Durable heads under RFC-024 are one possible authoritative
+representation; this protocol does not require or bless a warm parallel truth path.
+
+## 13. Drawbacks and rejected alternatives
+
+### 13.1 One generic staged-storage method
+
+Rejected. Lance does not expose staged forms for every operation, and existing writers
+have materially different version movement and rollback safety. A generic method would
+either lie about those differences or accumulate writer-kind conditionals. One
+protocol plus explicit adapters has lower long-run liability.
+
+### 13.2 Asynchronous heal with optimistic continuation
+
+Rejected. Current recovery classification relies on later writers not advancing past
+an unresolved sidecar. Scheduling a sweep does not establish that fact. Recovery may
+be proactively asynchronous, but the next writer still crosses a synchronous barrier.
+
+### 13.3 Touched-table-only CAS
+
+Rejected. Non-key uniqueness, RI, cardinality, and merge classification read tables
+the operation may not write. Ignoring those dependencies admits commits that were never
+valid against one serial graph state.
+
+### 13.4 Treat every state change as a manifest commit
+
+Rejected. Native branch refs and physical maintenance have different substrate
+visibility points. Manufacturing manifest commits would add coordination without
+making the native mutation atomic with them.
+
+## 14. Reversibility
+
+The in-memory coordinator and adapter refactor is reversible. Recovery-sidecar schema
+changes must be versioned and backward-compatible during rollout. This RFC alone does
+not authorize a `__manifest` schema-stamp bump, a public wire change, or a new storage
+substrate.
+
+The correctness contract is intentionally difficult to reverse: after writers rely on
+complete read-set arbitration and recovery-before-write, weakening either would
+reintroduce silent integrity or recovery races. Focused irreversible changes are
+reviewed in RFC-023 through RFC-027.
+
+## 15. Follow-up RFC boundaries
+
+- **RFC-023** owns PK annotation, fenced merge routing, strict keyed Append behavior,
+  mixed fenced/unfenced rollout, conflict mapping, and both commit-order tests.
+- **RFC-024** owns mutable table-head rows, explicit live/tombstoned head state,
+  current-state read shape, migration, and cost gates.
+- **RFC-025** owns checkpoint rows, Lance-native physical pins, cleanup reachability,
+  pruned-through semantics, and checkpoint/cleanup crash ordering.
+- **RFC-026** owns MemWAL enrollment, durable acknowledgements, fold/dead-letter
+  behavior, stream quiescence, fresh reads, public surfaces, and upgrade fencing.
+- **RFC-027** owns candidate discovery from row lineage, deletion discovery, fallback
+  semantics, and evidence for any O(delta) merge claim.
+
+Those RFCs call this protocol when they produce a graph-visible write. None may weaken
+the recovery barrier, omit read dependencies from `ReadSet`, or create a second graph
+visibility point.

docs/rfcs/rfc-023-key-conflict-fencing.md

@@ -0,0 +1,480 @@
+---
+type: spec
+title: "RFC-023 — Substrate-native key-conflict fencing"
+description: Make concurrent keyed writes fail or retry through Lance's transaction conflict filters, forbid keyed Append, and activate the contract only behind an all-branch fleet/format barrier.
+status: draft
+tags: [eng, rfc, write-path, concurrency, lance, primary-key]
+timestamp: 2026-07-10
+owner:
+---
+
+# RFC-023: Substrate-native key-conflict fencing
+
+**Status:** Draft / for team review
+**Date:** 2026-07-10
+**Surveyed:** omnigraph 0.8.1 (`main`); Lance 9.0.0-beta.15 at git rev
+`f24e42c1`; full Lance transaction, table-schema, read/write, branching, and
+MemWAL specifications; pinned Rust conflict-resolver and merge-insert sources
+**Relationship to RFC-022:** this RFC is the fencing decision split from the
+earlier monolithic RFC-022 draft. [RFC-022](rfc-022-unified-write-path.md)
+defines the shared write/recovery protocol; this RFC owns the substrate,
+compatibility stamp, and rollout requirements for key conflicts. It may share a
+release with [RFC-024](rfc-024-durable-table-heads.md), but neither RFC depends
+on the other's storage decision.
+**Audience:** engine, storage, migration, and release maintainers
+**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md).
+Findings marked **BLOCKER** must be dispositioned before acceptance.
+
+---
+
+## 0. Decision summary
+
+OmniGraph will use Lance's unenforced-primary-key merge-insert filter as the
+key-conflict primitive. It will not emulate the filter in the engine and will
+not add a lock table or custom transaction manager.
+
+The guarantee is deliberately narrow and strong:
+
+> For a keyed table, two concurrent operations that may insert the same `id`
+> MUST NOT both commit silently. They either serialize, retry from a new graph
+> base, or fail loudly.
+
+That guarantee cannot be rolled out by annotating a few new tables in an
+otherwise v4 graph. Lance's current mixed filtered/unfiltered conflict handling
+is directional, and a bare `Append` can commit after a filtered update. The
+contract therefore activates only after all supported writers and every table
+state reachable from every graph branch have crossed a fencing-compatible
+format barrier.
+
+Normative decisions:
+
+1. Node and edge table PK = `id`.
+2. Bare Lance `Append` is forbidden for keyed tables.
+3. Every keyed insert/upsert path produces the same Lance key filter.
+4. Mixed-version serving is forbidden during activation; the fencing-compatible
+   format stamp is written last and older binaries refuse it.
+5. PK metadata is permanent and preserved by every later schema/data rewrite.
+6. Existing-table migration covers every graph branch, including lazy-inherited
+   table states, and is recoverable by rolling forward.
+7. Fencing does not replace read-set OCC for `@unique`, RI, or cardinality.
+
+## 1. Problem
+
+OmniGraph validates duplicate `@key` values inside one delta, but today two
+processes can both read a base where `id = K` is absent, stage disjoint Lance
+fragments containing `K`, and let Lance rebase both commits. The graph manifest
+CAS orders graph publication; it does not tell Lance that the two data-table
+transactions inserted the same logical key.
+
+The result is worse than an ordinary write conflict: both callers can receive
+success while a keyed table contains duplicate IDs. Every subsequent upsert,
+edge lookup, uniqueness check, and merge then operates on a broken identity
+relation.
+
+The desired primitive already exists in Lance. The missing work is to use it on
+every keyed path and to define a rollout that never admits an unfenced writer.
+
+## 2. Substrate facts and the directional asymmetry
+
+This section is load-bearing. Tests pin every statement before implementation.
+
+### 2.1 Filter attachment
+
+On the pinned Lance revision, a v2 merge-insert whose ON field IDs exactly equal
+the schema's unenforced PK field IDs attaches an `inserted_rows_filter` to its
+`Operation::Update`. The filter contains keys for rows classified as inserts;
+updates of existing rows continue to use Lance's affected-row / fragment
+conflict machinery.
+
+The legacy indexed merge path does not attach this filter. Therefore a BTREE on
+`id` can route an otherwise correct merge onto an unfenced path unless the
+caller disables that path or Lance wires the filter into it.
+
+### 2.2 Conflict compatibility is directional
+
+Lance transaction compatibility is evaluated from the transaction currently
+attempting to commit against transactions that committed after its read
+version. It is not implicitly symmetric.
+
+At `f24e42c1`, `check_update_txn` behaves as follows:
+
+- current `Some(filter)` vs committed `Some(filter)` — compare field IDs and
+  filter intersection; overlap or incompatible filter configuration retries;
+- current `Some(filter)` vs committed `None` — retry conservatively;
+- current `None` vs committed `Some(filter)` — no corresponding conservative
+  arm; it may rebase;
+- current bare `Append` vs committed filtered `Update` — `Append` treats the
+  `Update` as compatible.
+
+Consequently, "filtered vs unfiltered conflicts" is not a sufficient rollout
+argument. Commit order matters. A filtered writer can win first and a stale
+unfiltered writer or bare keyed append can still land second.
+
+### 2.3 What a PK annotation does not do
+
+The key is explicitly *unenforced*. Merely setting the metadata:
+
+- does not validate historical uniqueness;
+- does not make bare appends unique;
+- does not protect a merge whose ON set differs from the PK;
+- does not repair an existing duplicate;
+- does not replace OmniGraph's semantic validators.
+
+## 3. Scope and non-goals
+
+In scope:
+
+- node and edge data tables;
+- mutation, load, branch merge, recovery replay, and WAL fold paths;
+- new-table creation and all-branch existing-table activation;
+- schema/overwrite preservation of PK metadata;
+- typed retry behavior and coverage gates.
+
+Out of scope:
+
+- using a composite edge PK (`src`, `dst`);
+- enforcing arbitrary `@unique` groups through the Lance PK;
+- keyless streaming-table deduplication;
+- a custom OmniGraph WAL, lock table, or transaction manager;
+- declaring general multi-process writes supported before foreign-process
+  recovery-sidecar ownership is solved.
+
+## 4. Table classes and PK contract
+
+### 4.1 Keyed graph tables
+
+Every normal node and edge table has a non-null `id` field. Its Lance schema
+MUST mark exactly that field as the unenforced PK. For edges, `src` and `dst`
+remain ordinary fields governed by referential-integrity and cardinality
+validation. An edge endpoint move is an update of the row identified by `id`.
+
+The PK field is addressed by stable field ID, not column position or mutable
+name. Until rename-stable OmniGraph type/property identity is closed, the
+fencing migration cannot claim rename safety.
+
+### 4.2 Keyless append-only tables
+
+A table explicitly declared append-only may omit a PK. Such a table may use
+Lance `Append`, including MemWAL append-only operation. It receives no
+same-logical-key guarantee because it has no logical key.
+
+Current node and edge types are not in this class: both have graph identity on
+`id`. The class is reserved for an explicitly designed internal or future
+non-graph table surface.
+
+The distinction is catalog-derived and first-class. Callers do not choose it
+with an ad-hoc flag.
+
+## 5. Normative write routing
+
+| Logical operation | Keyed table | Keyless append-only table |
+|---|---|---|
+| Strict insert / load append | merge-insert ON exactly `id`, `WhenMatched::Fail`, filtered path | `Append` allowed |
+| Upsert / load merge | merge-insert ON exactly `id`, `WhenMatched::UpdateAll`, filtered path | workload-specific |
+| Fast-forward branch merge of new rows | filtered merge-insert with `WhenMatched::Fail`, even when every row was classified new | `Append` allowed |
+| WAL upsert fold | filtered merge-insert with `merged_generations` | append transaction allowed |
+| Update existing row | merge-insert/update with affected-row conflict metadata | workload-specific |
+| Delete | staged Lance delete; PK filter is not the delete-conflict primitive | staged Lance delete |
+| Overwrite | staged overwrite whose output schema preserves the exact PK | staged overwrite |
+
+### 5.1 No keyed `Append`
+
+The prohibition includes internal optimization paths. A caller may not infer
+"all rows are new" and switch a keyed table to `stage_append`: that inference
+was made against a snapshot and is exactly what a concurrent same-key writer can
+invalidate.
+
+Routing through merge-insert does not collapse strict and upsert semantics.
+Strict `load --mode append` and other insert-only surfaces use
+`WhenMatched::Fail`; a row already present at the pinned base or discovered at
+execution remains an error. Only declared upsert surfaces use `UpdateAll`.
+
+The storage trait and `forbidden_apis` guard MUST make a keyed append difficult
+to express accidentally. The fast-forward merge optimization is retained only
+for keyless append-only tables unless Lance ships a key-filtered append
+transaction.
+
+This prohibition knowingly retires a measured fix. The fast-forward append
+path exists because a whole-delta merge-insert join exhausted the query memory
+pool on embedding-bearing tables (#277); routing adopted rows back through a
+filtered merge-insert re-exposes that workload to join memory behavior. The
+regression class is therefore a named ship gate: the fenced bulk adopt-merge
+must pass the §11.4 memory/cost gate on embedding-bearing tables — via bounded
+batched fenced merges inside one staged transaction, a pool-bounded execution
+mode, or an upstream key-filtered append — before the keyed fast-forward path
+is removed. Correctness wins the ordering, but the memory bound is not
+optional.
+
+### 5.2 Routing choice
+
+There are two acceptable implementations:
+
+1. use the v2 merge path (`use_index(false)`) and pass its scale gate; or
+2. consume a pinned Lance revision whose indexed path emits the identical
+   filter and passes the same surface guards.
+
+If the v2 hash-join cost scales unacceptably at the Phase-B workload, fencing
+waits for option 2. Correctness is not traded for the old indexed-path speed.
+
+### 5.3 Symmetric mixed-transaction behavior
+
+Before activation, the pinned Lance revision MUST conservatively reject both
+orders of:
+
+- filtered Update vs unfiltered Update;
+- filtered Update vs bare Append.
+
+The preferred fix is upstream conflict-resolver symmetry. A workspace-only
+fork is not an accepted permanent design. The fleet barrier remains necessary
+even after that fix because two old, unfiltered writers still have no filters
+to compare.
+
+## 6. Retry and validation semantics
+
+A Lance retryable key conflict restarts the entire logical attempt:
+
+1. gather a new graph snapshot and schema identity;
+2. rerun all delta and committed-state validation;
+3. restage from that base;
+4. commit and publish through the normal recovery-covered pipeline.
+
+It is incorrect to retry only `commit_staged`: an insert may have become an
+update, defaults or checks may now differ, and cross-table validation may have
+changed.
+
+Upsert surfaces may perform the bounded semantic retry. Strict insert surfaces,
+including `load --mode append`, do not change meaning under contention: both an
+already-present match from `WhenMatched::Fail` and a concurrent same-key commit
+normalize to typed `KeyConflict` / HTTP 409 for the whole strict operation. They
+do not switch to `UpdateAll`; callers decide whether to resubmit. Other strict
+read-modify-write surfaces retain their typed write conflict. Retry exhaustion
+on a non-strict upsert remains a retryable 409.
+
+Fencing covers the PK insertion race only. `@unique` values on different IDs,
+edge RI, and cardinality depend on a read set. Their correctness requires the
+read-set-in-CAS design or equivalent revalidation before HEAD movement; this RFC
+does not claim that fences close those races.
+
+## 7. Version and fleet barrier
+
+### 7.1 No partial activation in the old format
+
+OmniGraph MUST NOT annotate a new data table and advertise fencing while the
+graph remains generally writable by older binaries. An older process can select
+the legacy merge path or keyed append and bypass the guarantee.
+
+This RFC owns its activation boundary:
+
+- operators quiesce every server, CLI writer, and embedded writer for the
+  graph;
+- one migration claimant holds an atomic create-if-absent claim with a random
+  owner/fencing token; a native Lance branch sentinel is not accepted as CAS;
+- only the dedicated migration binary may open the old graph for writes;
+- the fencing-compatible stamp is written after every branch/table verification;
+- normal serving begins only after the stamp; older binaries then refuse.
+
+The migration claim uses the storage adapter's `PutMode::Create` contract,
+records operation/owner token, and has no time-only takeover. Recovery under the
+fleet outage must classify the migration ledger/sidecars before replacing a
+stale token.
+
+The stamp is graph-wide and read from the reserved main manifest before any
+named-branch open; selecting a named branch cannot bypass the compatibility
+check.
+
+An in-process mutex is not a fleet barrier. A marker unknown to v4 binaries is
+also not a fleet barrier. The operator procedure and cluster control plane must
+keep old writers stopped until finalization.
+
+The exact format number is assigned when this RFC is accepted. If RFC-024 is
+also accepted and ready, the two migrations may deliberately share its v5
+release after a combined failure-matrix review. If durable heads fail their
+cost gate, fencing still proceeds with its own next compatible stamp; if
+fencing is blocked upstream, durable heads need not wait.
+
+If durable heads are already active when fencing migrates, every PK metadata
+version repoint also emits the identity-bearing journal event and matching
+`table_head` transition in the same manifest CAS. If fencing lands first, its
+format/stamp becomes an explicit predecessor that a later heads migration must
+preserve. Acceptance covers heads-first, fencing-first, and co-release orders.
+
+### 7.2 New graphs
+
+A graph created directly at the fencing-compatible format creates every keyed
+table with the PK metadata already present and enables only the write routing
+in §5. There is no post-create annotation window.
+
+## 8. All-branch PK migration
+
+Migration operates on graph states, not merely table roots. The unit is every
+reachable tuple:
+
+```
+(graph_branch, table_key, table_path, pinned_table_branch, pinned_table_version)
+```
+
+This matters for lazy branches: a graph branch may still point at an old main
+table version whose schema predates the PK, even after main HEAD is annotated.
+
+Under the fleet barrier, the migration:
+
+1. enumerates and incarnation-pins every live graph branch;
+2. folds each branch manifest and enumerates its live keyed tables;
+3. validates that every pinned table image has non-null, unique `id` values;
+4. acquires branch/table gates in RFC-022 order and freshly revalidates the
+   pinned tuple, schema identity, and migration claim;
+5. writes a per-unit RFC-022 sidecar declaring expected branch/table state, the
+   optional native fork effect, PK metadata effect, and intended manifest delta
+   before either effect can persist;
+6. for an owned table branch, commits a set-if-absent PK metadata update;
+7. for a lazy-inherited state, forks an owned table branch from the *exact
+   pinned version*, applies the PK metadata there, and leaves row contents
+   unchanged;
+8. records exact achieved fork identity and table version in the sidecar;
+9. publishes that graph branch's manifest to the annotated physical version
+   with an audited migration marker but no graph-content commit or graph-head
+   movement, including a table-head transition when heads are active;
+10. records a branch/table completion digest;
+11. re-enumerates branches and verifies every live branch before writing the
+   fencing-compatible stamp.
+
+PK installation advances a Lance table version before the graph manifest can
+publish it, and a lazy fork creates native ref state first. The sidecar covers
+both gaps and lets recovery reclaim or adopt the exact fork rather than infer
+from a branch name. Because Lance forbids clearing/changing a set PK,
+migration is roll-forward-only:
+
+- an already-correct PK is an idempotent success and is not rewritten;
+- an absent PK resumes installation;
+- a different PK is a loud, operator-visible refusal;
+- recovery never attempts to "undo" PK metadata.
+
+Branch create/delete, schema apply, and normal data writes remain blocked for
+the whole enumeration/install/verify interval. The migration ledger makes a
+crash resumable without treating partial annotation as a served graph.
+
+## 9. Preservation after activation
+
+Once set, the following are storage invariants:
+
+- a table overwrite carries the same PK field IDs and positions;
+- schema apply cannot remove, replace, reorder semantically, or make nullable a
+  PK field;
+- rename preserves the PK field ID and metadata;
+- branch fork/clone preserves it;
+- import/rebuild creates it before accepting data;
+- recovery restore may select an older data image only if that image is already
+  fencing/PK-compatible;
+- a table recreation uses a new table incarnation but installs the same
+  catalog-derived PK contract at creation;
+- `__manifest`'s existing legacy PK key form is preserved exactly as stored;
+  the migration never rewrites or "normalizes" it. Lance forbids changing a
+  set PK, and the native-namespace decoupling documented in the Lance
+  alignment audit depends on that legacy form remaining in place.
+
+Every open-for-write path verifies the physical schema matches the catalog PK
+contract. The check is against the pinned physical schema and is not a
+maintained parallel registry.
+
+## 10. Recovery and multi-process scope
+
+All data writes retain the existing Phase A-D sidecar protocol. The key filter
+does not close the table-HEAD-before-graph-manifest window.
+
+The fenced data-table transaction is cross-process safe in its failure-free
+commit path. OmniGraph's current recovery sweep, however, serializes with live
+writers only in-process; a foreign recovery process can still inspect a live
+sidecar, and destructive `Restore` cannot be made convergence-idempotent.
+
+Therefore this RFC MUST use one of two honest dispositions:
+
+1. retain the documented single-writer-process support boundary and describe
+   fences as closing the silent key-race primitive only; or
+2. land a cross-process sidecar claim/lease before advertising general
+   multi-process writes.
+
+Fences alone are not evidence for disposition 2.
+
+## 11. Tests and acceptance gates
+
+### 11.1 Lance surface guards
+
+- exact PK ON set + v2 path produces a non-empty filter for inserts;
+- `WhenMatched::Fail` preserves that filter and reports an existing match
+  without writing;
+- mismatched ON set produces no filter;
+- legacy/indexed path behavior is pinned until replaced;
+- filtered/filtered overlapping keys retry;
+- filtered/filtered disjoint keys may rebase;
+- filtered/unfiltered retries in **both** commit orders;
+- filtered Update/bare Append retries in **both** commit orders;
+- PK metadata cannot be changed or removed once installed.
+
+### 11.2 Engine concurrency tests
+
+- the same-key DST cell becomes a hard assertion with N concurrent writers;
+- different keys remain concurrently writable;
+- every keyed load/mutation/merge/fold path is observed to use the filtered
+  primitive;
+- strict append of an existing `id` still fails and never updates the row;
+- strict pre-existing-match and concurrent-insert cases normalize to the same
+  external `KeyConflict` while preserving `WhenMatched::Fail`;
+- a source-walk guard rejects keyed `stage_append`, including the former
+  fast-forward path;
+- a retry reruns validation rather than committing the stale staged batch.
+
+### 11.3 Migration and recovery tests
+
+- main plus owned and lazy-inherited graph branches all emerge PK-annotated;
+- duplicate historical IDs abort before the fencing-compatible stamp;
+- crash after each table annotation and before each manifest repoint resumes
+  without data change;
+- crash before/after lazy fork and PK metadata commit recovers the exact
+  sidecar-recorded ref/version;
+- branch enumeration is incarnation-safe;
+- old binary/new graph and new binary/partially migrated graph refuse loudly;
+- heads-first, fencing-first, and co-release upgrades preserve every active
+  format capability and produce identical logical rows;
+- overwrite, schema apply, branch fork, restore, and import preserve PK metadata.
+
+### 11.4 Cost gate
+
+Measure a small upsert into 10K, 100K, and 1M-row indexed tables using the
+shared cost harness. If `use_index(false)` makes work scale with table size
+beyond the accepted budget, the indexed-path upstream work is a ship blocker.
+
+Additionally measure the bulk adopt-merge shape that motivated the keyed
+fast-forward path (#277): a many-row, all-new-rows fenced merge into an
+embedding-bearing table, asserting peak memory bounded by batch size rather
+than table or delta width. If the fenced path cannot meet that bound, the
+keyed fast-forward removal waits for the mitigation named in §5.1.
+
+> 💬 **Instrument required (tightening 5 in the
+> [review ledger](../dev/rfc-022-027-architecture-review.md)):**
+> `helpers::cost` measures I/O, not peak RSS, so this memory bound is
+> unenforceable as written. Use the subprocess `scenarios.rs` harness or an
+> equivalent `wait4`/`ru_maxrss` instrument, and name dataset sizes, baseline,
+> cap, and pass threshold.
+
+## 12. Decisions and open gates
+
+### Decided
+
+- `id`, not `src`+`dst`, is the edge PK.
+- No keyed append, including optimization-only append.
+- No mixed-fleet or new-table-only v4 rollout.
+- PK migration is all-branch, offline, idempotent, and roll-forward-only.
+- A retryable upsert conflict retries the logical operation; strict insert maps
+  both existing and concurrent matches to `KeyConflict` without changing mode.
+- Read-set validation remains a separate required concurrency design.
+
+### Open ship gates
+
+1. Upstream symmetric filtered/unfiltered and filtered/Append conflict behavior.
+2. v2-path scale result versus indexed-path filter availability.
+3. Operator repair procedure for pre-existing duplicate IDs.
+4. Rename-stable field/type identity.
+5. Cross-process recovery ownership before any broadened topology claim.
+6. Final format-number/release sequencing and the all-branch fleet stamp.
+7. The fenced bulk adopt-merge memory/cost gate on embedding-bearing tables
+   (the #277 regression class) — see §5.1 and §11.4.

docs/rfcs/rfc-024-durable-table-heads.md

@@ -0,0 +1,558 @@
+---
+type: spec
+title: "RFC-024 — Durable table heads and the v5 manifest"
+description: Materialize one live-or-tombstoned current-state row per table inside each manifest branch, prove bounded physical lookup, and migrate every branch atomically before stamping v5.
+status: draft
+tags: [eng, rfc, manifest, write-path, versioning, migration, lance]
+timestamp: 2026-07-10
+owner:
+---
+
+# RFC-024: Durable table heads and the v5 manifest
+
+**Status:** Draft / for team review
+**Date:** 2026-07-10
+**Surveyed:** omnigraph 0.8.1 (`main`); Lance 9.0.0-beta.15 at git rev
+`f24e42c1`; full Lance table layout, transaction, branching, indexing,
+compaction, cleanup, and object-store specifications
+**Relationship to RFC-022:** this RFC is the durable-heads decision split from
+the earlier monolithic RFC-022 draft. [RFC-022](rfc-022-unified-write-path.md)
+defines the shared publisher/recovery protocol; this RFC owns the v5 format and
+migration boundary. It deliberately excludes checkpoint retention, which
+[RFC-025](rfc-025-checkpoint-retention.md) reviews separately. Key fencing in
+[RFC-023](rfc-023-key-conflict-fencing.md) is also independently reviewable;
+the two may share a release but do not block one another's evidence gates.
+**Audience:** engine, manifest, migration, branch, and release maintainers
+**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md).
+Findings marked **BLOCKER** must be dispositioned before acceptance.
+
+---
+
+Throughout this draft, **v5** means the next internal schema after today's v4.
+If another independently accepted format change lands first, the release number
+is reassigned; none of the head-row semantics depend on the numeral.
+
+## 0. Decision summary
+
+The current manifest is both an immutable graph journal and the place writers
+ask "what is current?" Current-state resolution folds history, so its physical
+cost grows with commit count even though the answer contains only one value per
+table.
+
+v5 adds one mutable, durable `table_head` row per stable table identity inside
+each native `__manifest` branch. The publisher updates the head in the same
+Lance merge-insert transaction as immutable table-version rows, tombstone
+events, graph lineage, and `graph_head`. The journal remains the history source;
+heads become the current-state source.
+
+The format does **not** ship merely because the logical result has O(tables)
+rows. A filtered scan over a history-sized Lance table is still O(history)
+physical work. v5 is gated on a Lance-native indexed lookup whose measured I/O
+is flat at history depth and whose uncovered tail is bounded and observable.
+
+Normative decisions:
+
+1. Heads live in `__manifest`; there is no second heads dataset and no warm
+   mutable-tip authority.
+2. Head state is explicitly `live` or `tombstoned` and carries an incarnation.
+3. Every publish and recovery outcome updates journal and head atomically.
+4. Current-state reads use heads; history reads use the journal.
+5. Missing or duplicate heads in v5 are corruption, not a reason to silently
+   return to the history fold.
+6. Migration covers every live manifest branch, then stamps main v5 last.
+7. If RFC-023 is co-released, the combined migration also verifies PK fencing;
+   durable heads do not depend on that decision.
+8. Checkpoint/retention markers are deferred to a separate RFC.
+
+## 1. Problem
+
+`__manifest` currently stores immutable table-version and tombstone rows. To
+resolve a branch tip, readers scan those rows, select the greatest version per
+table, and apply tombstones. This makes a write's coordination cost depend on
+the total graph history. Compaction reduces fragment count but cannot remove
+the semantic journal rows, so even a compacted manifest retains a row-volume
+slope.
+
+Caching that fold as mutable in-process state is the wrong authority shape for
+writes: invalidation becomes a correctness condition across processes and
+branch incarnations. Storing the folded answer durably in the same transaction
+as the journal removes both liabilities:
+
+- no parallel authority can drift; and
+- current-state work is proportional to catalog width, provided physical
+  lookup is also bounded.
+
+## 2. Scope and non-goals
+
+In scope:
+
+- v5 table-head schema and state transitions;
+- atomic publisher, recovery, and current-read semantics;
+- bounded physical access proof;
+- all-branch predecessor→heads migration and compatibility refusal;
+- optional co-release integration with RFC-023's all-branch PK activation.
+
+Out of scope:
+
+- a `GraphState` singleton or warm publish input;
+- a separate `__heads` Lance dataset;
+- deleting immutable journal rows;
+- commit-graph ancestry acceleration;
+- checkpoint retention and arbitrary-version GC;
+- using a physical index as a correctness precondition.
+
+Checkpoint retention is excluded because a row in `__manifest` does not by
+itself pin a version in another Lance dataset. Lance cleanup recognizes its own
+versions, tags, and branch references, not foreign references. A later RFC must
+define substrate-native per-table pins and their crash-safe lifecycle.
+
+## 3. v5 table-head schema
+
+Each native manifest branch contains exactly one current head row for each
+stable table identity known to that branch.
+
+### 3.1 Identity and object key
+
+```
+object_id   = "table_head:<stable_table_id>"
+object_type = "table_head"
+```
+
+`stable_table_id` survives a type rename. It is not the display name and is not
+derived anew from `kind:name`. Closing the current rename-stable identity gap is
+a v5 ship gate, and **this RFC owns it**: the stable-ID encoding, the compiler
+contract that mints and preserves it, and the incarnation baseline are defined
+here. RFC-023 §4.1 (rename-safe PK claims) and RFC-025 §2.1 (checkpoint table
+rows) consume this identity and must not ship identity-dependent claims before
+it lands. No other RFC in the family defines a competing identity scheme.
+
+> 💬 **Superseded by review (BLOCKER-07 in the
+> [review ledger](../dev/rfc-022-027-architecture-review.md)):** ownership
+> *inside* this RFC contradicts siblings that consume the identity while
+> calling this RFC optional (RFC-025) or without declaring the dependency
+> (RFC-026). Disposition: extract identity/incarnation into its own
+> prerequisite RFC, or accept this RFC in full before the identity-dependent
+> siblings. Replace this ownership note accordingly.
+
+The head row is mutable under `WhenMatched::UpdateAll`, just like
+`graph_head:<branch>`. There is one object ID per stable identity; recreation
+does not leave multiple candidate heads.
+
+### 3.2 Payload
+
+v5 reuses the manifest's typed columns where they already fit and stores the
+remaining versioned payload in a typed JSON structure:
+
+```text
+TableHeadMetadata {
+    state: "live" | "tombstoned",
+    stable_table_id: String,
+    incarnation_id: ULID,
+    schema_hash: String,
+    head_graph_commit_id: Option<ULID>,
+}
+```
+
+The row columns have these meanings:
+
+| Column | `live` | `tombstoned` |
+|---|---|---|
+| `table_key` | current public table key/name | last public key/name |
+| `location` | physical table location | last physical location, diagnostic only |
+| `table_version` | current visible Lance version | last live Lance version |
+| `table_branch` | physical owner branch, nullable for main | last owner branch |
+| `row_count` | current row count | null |
+| `metadata` | `TableHeadMetadata` | `TableHeadMetadata`; `head_graph_commit_id` names the tombstoning graph commit |
+
+The `state` field is authoritative. A tombstoned row MUST NOT become live merely
+because an older journal version has a greater data-table version than some
+other row.
+
+### 3.3 Incarnations
+
+`incarnation_id` distinguishes drop/recreate ABA:
+
+- rename preserves stable ID and incarnation;
+- ordinary writes preserve both;
+- physical owner handoff preserves both;
+- dropping the type transitions the one head to `tombstoned`;
+- recreating a logically new table mints a new incarnation and updates the same
+  stable-identity head to `live` only through an explicit schema operation.
+
+If recreation is assigned a new stable table identity by schema semantics, it
+gets a new head object and the old identity remains tombstoned. The schema
+planner, not a name comparison, chooses this outcome.
+
+### 3.4 Journal identity
+
+The mutable head is not the only place that records identity. Every v5
+table-version, registration, rename, and tombstone journal event carries
+`stable_table_id` and `incarnation_id`; otherwise drop/recreate followed by a
+new physical dataset whose Lance versions restart cannot be replayed
+unambiguously.
+
+Migration writes one immutable v5 incarnation-baseline row per known identity,
+bound to the source-tip digest. New registrations/recreations write an immutable
+incarnation transition. Head repair starts from that baseline and replays only
+identity-bearing v5 events. It does not infer identity from mutable names or
+compare unrelated Lance version numbers. If pre-v5 history cannot be mapped to
+one baseline identity deterministically, migration refuses before the stamp.
+
+## 4. State-transition rules
+
+| Event | Required head transition |
+|---|---|
+| Register table | absent → live, new incarnation |
+| Data write / optimize / index publish | live version N → live version M |
+| Owner-branch handoff | live owner A → live owner B, even if version is equal |
+| Rename | live key/name A → live key/name B; identity/incarnation unchanged |
+| Drop table | live → tombstoned in the same graph publish as the tombstone journal row |
+| Recreate | tombstoned → live with the schema-planned identity/incarnation outcome |
+| Recovery roll-forward | apply the failed writer's intended live/tombstone transition |
+| Recovery rollback | publish a head matching the restored physical version and logical pre-write state |
+
+No path may append a table-version or tombstone journal row without including
+its corresponding head mutation in the same publisher source batch.
+
+## 5. Atomic publisher contract
+
+The v5 publisher constructs one merge-insert source containing:
+
+- immutable, identity-bearing table-version rows;
+- immutable, identity-bearing table-tombstone/transition rows;
+- mutable table-head rows;
+- when the RFC-022 plan carries `LineageIntent`, the immutable `graph_commit`
+  and mutable `graph_head:<branch>` rows;
+- for metadata-only plans, their specific CAS authority/operation rows without
+  manufacturing graph lineage.
+
+One Lance manifest commit makes the entire set visible. The publisher still
+resolves the graph parent and re-reads commit authority inside every CAS retry.
+Expected table versions are compared against table heads, not reconstructed by
+folding the journal.
+
+Two graph-content writers touching disjoint data tables still contend on
+`graph_head:<branch>`, form one linear graph history, and re-parent on retry.
+Writers touching the same table also contend on its one `table_head` object.
+Metadata-only CASes contend on the stable authority rows named by their complete
+`ReadSet`; they do not update `graph_head` merely to create contention.
+
+The immutable journal remains necessary for snapshots, diffs, audit, migration
+verification, and head repair. Head rows do not replace or truncate it.
+
+## 6. Read contract
+
+### 6.1 Current state
+
+A v5 current-state read:
+
+1. derives the expected live stable table IDs from the pinned catalog and fixed
+   system-table registry;
+2. issues a structured lookup for the exact head object IDs;
+3. requires exactly one valid row per expected identity;
+4. includes only rows whose authoritative state is `live`;
+5. validates schema identity from the head payload;
+6. returns one immutable `Snapshot` used for the operation's lifetime.
+
+Missing, duplicate, unknown-state, or schema-mismatched **live** heads fail
+loudly. The hot path does not enumerate every identity ever dropped merely to
+prove all tombstone heads exist; the branch completion digest plus explicit
+`heads verify`/repair owns bounded tombstone-set validation. A missing or
+duplicate tombstone is still corruption, but normal reads do not regain an
+O(history) scan to discover it.
+
+### 6.2 History
+
+`snapshot_at_version`, commit resolution, change feeds, and audit continue to
+read immutable journal/lineage state at the requested manifest version. A v5
+manifest version contains the heads as they stood at that version, but the
+journal remains the normative explanation of how the state arose.
+
+### 6.3 Diagnostic repair
+
+An explicit offline repair loads the branch's v5 incarnation baseline, replays
+identity-bearing journal transitions from that baseline, compares the result to
+its heads/marker digest, and publishes corrected heads with an audited system
+actor. Repair is not part of the read hot path and never silently runs from a
+query.
+
+## 7. Bounded physical lookup is a ship gate
+
+Logical O(tables) output does not prove physical O(tables) work. Without an
+index, `object_id IN (...)` still scans the journal-bearing manifest fragments;
+compaction reduces files but not semantic row count.
+
+### 7.1 Required property
+
+At fixed catalog width, a reconciled v5 current-state lookup MUST have zero
+positive slope in:
+
+- manifest object-store reads;
+- bytes read;
+- fragments/pages scanned; and
+- rows decoded
+
+as commit history grows.
+
+The bound must hold on a real object store as well as local FS and must be shown
+for compacted and uncompacted histories. The test uses the shared IO-tracking
+harness and installs the tracker before the manifest handle opens.
+
+### 7.2 Candidate access shape
+
+The primary candidate is a structured exact lookup on `object_id` backed by a
+Lance scalar index. It is acceptable only if measurement proves:
+
+- indexed head lookup avoids journal-fragment scans;
+- newly committed head rows leave at most a bounded uncovered tail;
+- reconciliation restores coverage without synchronous expensive work in the
+  logical write path;
+- index absence or partial coverage remains logically correct and is surfaced
+  as an observable degraded-cost mode.
+
+The index is derived state. Queries MUST remain correct if it is missing, and a
+missing index cannot block a logical write. The performance promise applies to
+the reconciled serving state and includes an explicit bound on uncovered work;
+it is not inferred merely from the existence of an index declaration.
+
+### 7.3 Rejected access shape
+
+A separate heads dataset is rejected. Lance commits are per dataset, so it
+would reintroduce a journal→heads crash gap and require another sidecar protocol
+for the very pointer whose purpose is to remove drift.
+
+If no in-manifest Lance-native access shape passes the gate, v5 does not ship
+with head rows. The fallback is to retain v4 plus the local session/view-passing
+improvements, not to waive the cost claim.
+
+## 8. Recovery protocol
+
+Data-table writers still use the existing four phases:
+
+1. write sidecar before a Lance HEAD advance;
+2. commit staged/inline table work;
+3. publish `__manifest`;
+4. delete sidecar.
+
+The sidecar's logical intent in v5 includes the expected and desired table-head
+payload. Recovery behavior is therefore complete:
+
+- roll-forward publishes the data version, journal row, table head, lineage,
+  and graph head together;
+- rollback restores the physical version, then publishes journal/audit state
+  and a table head matching the restored logical state;
+- a stale sidecar whose goal is already represented by the exact head
+  incarnation/version converges idempotently;
+- a partially matching head is not treated as success.
+
+Recovery remains a synchronous barrier before any later writer advances a
+touched table. Index reconciliation may be asynchronous; unresolved commit
+protocol state may not.
+
+## 9. v5 boundary and compatibility
+
+v5 comprises:
+
+1. table-head rows and their publish/read semantics;
+2. branch-local v5 completion markers; and
+3. the graph-level internal-schema stamp written after verification.
+
+It does **not** comprise checkpoint/retention markers.
+
+If RFC-023 is independently accepted and ready for the same release, v5 may
+also carry its PK annotation and fencing-compatible marker after a combined
+migration review. That is release coordination, not a prerequisite of heads.
+
+Format sequencing is explicit:
+
+| Order | Result |
+|---|---|
+| Heads first | proposed v5 contains heads; later fencing migration preserves and atomically updates heads for every PK-version repoint |
+| Fencing first | fencing takes the next format number; this heads migration takes the following number, accepts that exact predecessor, and preserves its PK/stamp invariant |
+| Co-release | one format maps to both independently accepted capabilities and runs the combined failure matrix |
+
+Migration code dispatches from an exact supported predecessor feature set; it
+never assumes that “pre-heads” means pristine v4 or drops a capability it does
+not own.
+
+After upgrade, serving is strict-single-version:
+
+- v5 binaries refuse unstamped or partially migrated graphs in normal mode;
+- older binaries refuse v5 with the existing upgrade message;
+- every open reads the graph-wide main stamp before selecting a named branch;
+- only the dedicated offline migration command may open the exact supported
+  predecessor for conversion;
+- there is no mixed v4/v5 serving period.
+
+## 10. All-branch predecessor→heads migration
+
+### 10.1 Preconditions
+
+The operator stops every graph writer and acquires an atomic
+create-if-absent migration claim (with exact owner/fencing token; not a Lance
+branch sentinel). The barrier covers server, CLI, embedded, maintenance,
+branch, and schema writes. Because predecessor binaries do not understand the
+new in-graph migration marker, the offline fleet barrier remains mandatory
+until the final stamp.
+
+The claim uses `PutMode::Create`, records the migration operation and owner
+token, and permits no time-only takeover. Recovery classifies the durable
+ledger/sidecars under the fleet outage before replacing a stale token.
+
+After acquiring the claim, migration completes RFC-022's recovery barrier
+before pinning any branch source tip.
+
+If fencing is co-released, the migration first executes RFC-023's all-branch
+table-PK plan. Otherwise head migration neither adds nor changes PK metadata.
+
+### 10.2 Per-branch conversion
+
+For each live native `__manifest` branch:
+
+1. capture branch name, ref incarnation, manifest version, and e_tag/timestamp;
+2. run the predecessor journal+tombstone fold once at that pinned tip;
+3. construct exactly one live or tombstoned head plus one immutable incarnation
+   baseline per stable table identity;
+4. validate table schema hashes and, only in a combined release, RFC-023 PK state;
+5. publish all heads/baselines, an audited migration record
+   (`actor = omnigraph:migration/v5`), and a branch-local completion marker in
+   one manifest CAS that revalidates the captured ref incarnation/source tip;
+   this physical representation migration does not create a graph-content
+   commit or move `graph_head`;
+6. store in the marker the source tip, head count, identity-baseline digest,
+   head-set digest, and heads-format version; the marker is content-scoped and
+   deliberately does not embed the source branch incarnation; source tip is
+   provenance, while digest/format determine inherited-marker validity;
+7. verify by reopening the produced branch version through the v5 head reader.
+
+The completion marker has a deterministic object ID within each manifest
+branch. A retry that finds a matching source/digest is complete; a mismatching
+marker is a loud migration conflict.
+
+### 10.3 Finalization
+
+After all branches report completion, migration re-enumerates native manifest
+refs and verifies the same incarnation set. Branch create/delete is blocked, so
+any difference indicates out-of-band modification and aborts finalization.
+
+Only then does it stamp main as internal schema v5. The stamp is the fleet
+commit point: before it, no serving process may start; after it, only v5 code may
+open the graph.
+
+New branches created after v5 fork a source manifest that already contains
+complete heads and the content-scoped v5 marker. The inherited marker remains
+valid for the identical snapshot; branch open separately binds that content to
+the new native ref incarnation and validates the graph-wide stamp. No
+post-create marker rewrite is required.
+
+## 11. Migration recovery
+
+The migration keeps a durable ledger outside the ordinary read path with one
+record per branch and, in a combined release, per RFC-023 table conversion. It
+records expected source incarnations, achieved physical versions, produced
+manifest versions, and digests.
+
+Recovery is idempotent and roll-forward-only:
+
+- completed, digest-matching branch conversions are skipped;
+- a table HEAD advance not yet represented in its graph branch is recovered by
+  the normal sidecar and then branch conversion resumes;
+- an uncommitted head-row source leaves no visible state and is rebuilt;
+- a committed branch marker is authoritative for that branch conversion;
+- the main v5 stamp is never written while any ledger unit is incomplete;
+- co-released PK metadata is never cleared to simulate rollback.
+
+If migration crashes, the graph remains offline. Restarting an old serving
+binary is not a recovery procedure; the operator resumes the v5 migration.
+
+## 12. Tests and acceptance gates
+
+### 12.1 Head semantics
+
+- current state from heads is byte-equivalent to the predecessor fold across realistic
+  histories;
+- live→tombstoned never resurrects an older live version;
+- drop/recreate distinguishes incarnations;
+- identity-bearing journal replay remains unambiguous when a recreated physical
+  dataset restarts Lance version numbering;
+- rename preserves identity/incarnation and changes the public key only;
+- owner-branch handoff at an equal table version updates the head;
+- missing, duplicate, malformed, and schema-mismatched heads fail loudly.
+- `heads verify` detects a missing/duplicate tombstone against the baseline and
+  completion digest without adding that enumeration to every current read.
+
+### 12.2 Publisher and recovery
+
+- concurrent disjoint writers produce one linear graph chain and correct heads;
+- same-table writers contend on one head row;
+- failpoints after every table commit but before manifest publish recover to
+  matching physical version, journal, table head, and graph head;
+- rollback and roll-forward assertions include head payloads, not only table
+  versions;
+- a stale sidecar converges exactly once with one audit record.
+
+### 12.3 All-branch migration
+
+- main, owned named branches, and lazy-inherited branches all convert;
+- tombstoned types are represented on every relevant branch;
+- crashes before/after every per-branch CAS resume from the ledger;
+- branch ref deletion/recreation is caught by incarnation verification;
+- final stamp is impossible while any table or branch marker is absent;
+- each branch's graph head and content lineage are unchanged by head migration;
+- old/new binary refusal matrix covers every supported predecessor capability
+  set, partial heads migration, and complete heads format;
+- a post-v5 branch inherits and validates complete heads.
+- the inherited completion marker is content-scoped while ref-incarnation
+  validation is performed separately.
+
+### 12.4 Cost gates
+
+At fixed table count and increasing commit depth, assert flat curves for
+manifest reads, bytes, fragments/pages, and decoded rows:
+
+- local FS, compacted and uncompacted;
+- S3/RustFS with real e_tags;
+- warm repeated read;
+- cold operation-local open with shared Session;
+- one uncovered-head update before reconciliation;
+- reconciled steady state.
+
+The test must fail if the lookup silently falls back to scanning journal
+history in the claimed steady state.
+
+The decode term is part of the gate: parsing head rows — including the typed
+JSON `TableHeadMetadata` payload — must be bounded by catalog width. A
+per-read parse cost that grows with anything other than table count fails the
+gate even when I/O is flat.
+
+### 12.5 Format guards
+
+- exact v5 head metadata schema and object IDs;
+- one head row per stable identity;
+- incarnation-baseline and identity-bearing journal event schemas;
+- content-scoped branch completion marker schema/digest;
+- RFC-023 PK metadata on node and edge tables when the release combines them;
+- v5 publisher source always pairs a journal/tombstone event with a head row.
+
+## 13. Decisions and open gates
+
+### Decided
+
+- Heads and journal share one `__manifest` transaction.
+- Current reads use heads; historical reads keep the journal.
+- Heads represent `live | tombstoned` plus incarnation explicitly.
+- A separate heads dataset and a mutable in-process tip authority are rejected.
+- Migration is offline, all-branch, resumable, and stamps v5 last.
+- RFC-023 PK activation is verified by v5 only when deliberately co-released.
+- Checkpoint retention is deferred.
+
+### Open ship gates
+
+1. Rename-stable table/type identity and final stable-ID encoding — owned by
+   this RFC; consumed by RFC-023 and RFC-025 (§3.1).
+2. The in-manifest indexed lookup implementation and bounded uncovered-tail
+   proof.
+3. Passing local and object-store depth-slope cost gates.
+4. Atomic cross-process migration-claim implementation and operator runbook.
+5. Final v5 metadata JSON/typed-column compatibility review.
+6. Full all-branch migration/failpoint matrix.

docs/rfcs/rfc-025-checkpoint-retention.md

@@ -0,0 +1,444 @@
+---
+type: spec
+title: "RFC-025 — Checkpoint-pinned retention"
+description: Makes named graph checkpoints authoritative retention roots, materializes them as Lance-native manifest and data-table tags, and defines crash-safe reconciliation and cleanup ordering on the RFC-022 unified write path.
+status: draft
+tags: [eng, rfc, retention, checkpoint, cleanup, manifest, lance, omnigraph]
+timestamp: 2026-07-10
+owner:
+---
+
+# RFC-025 — Checkpoint-pinned retention
+
+**Status:** Draft / for team review
+**Date:** 2026-07-10
+**Depends on:** [RFC-022](rfc-022-unified-write-path.md)'s publisher and
+recovery-sidecar protocol. It composes with
+[RFC-024](rfc-024-durable-table-heads.md), but is not part of v5 by implication.
+**Surveyed:** omnigraph 0.8.1; Lance 9.0.0-beta.15 (`f24e42c1`)
+**Audience:** engine, storage, CLI, and operations maintainers
+**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md).
+Findings marked **BLOCKER** must be dispositioned before acceptance.
+
+---
+
+## 0. Decision
+
+OmniGraph adopts checkpoint-pinned retention. A checkpoint is a durable, named
+graph snapshot: one reference set in the reserved main-manifest registry
+containing every physical manifest/table lineage and version needed to
+reconstruct that graph state.
+
+The checkpoint rows are the **logical authority**. Lance tags are the
+**physical pins** that make that authority effective. This distinction is
+load-bearing: Lance `cleanup_old_versions` protects Lance tags and branches; it
+does not inspect OmniGraph rows in `__manifest`. A checkpoint row without the
+corresponding tags would therefore promise time travel to versions that Lance
+is free to delete.
+
+The safe ordering is asymmetric:
+
+- create physical tags first, publish checkpoint authority last;
+- tombstone checkpoint authority first, reclaim physical tags last.
+
+Every crash window consequently over-retains. None under-retains.
+
+## 1. Scope and non-goals
+
+This RFC specifies:
+
+1. checkpoint and GC-boundary manifest rows plus their own format activation;
+2. deterministic Lance tags for the pinned manifest and every pinned table version;
+3. create, delete, and reconciliation protocols;
+4. how `cleanup` computes and protects its root set;
+5. CLI, policy, audit, observability, migration, and acceptance contracts.
+
+It does not change snapshot isolation, invent a second transaction log, or
+replace Lance cleanup. It does not make every historical graph commit a
+checkpoint. History outside branch heads, named checkpoints, and the explicit
+retention window remains eligible for collection after operator confirmation.
+
+## 2. Sources of truth
+
+### 2.1 Logical authority: manifest rows
+
+A checkpoint is immutable after creation. Retargeting a name means deleting the
+old checkpoint and creating a new checkpoint ID; a name never silently moves.
+The stable name-reservation row carries `state = live | tombstoned` and a
+monotonic `generation`. Reuse is a CAS transition from the exact tombstoned
+generation to `live` with `generation + 1` and a fresh checkpoint ID. Thus a
+deleted name may be deliberately reused, but two concurrent re-creations cannot
+both win.
+
+All checkpoint authority rows live on the reserved **main** branch of
+`__manifest`, regardless of the logical branch being checkpointed. This is a
+global registry, not branch-local graph state: deleting the source branch must
+not delete the authority that protects its snapshot. One main-manifest publish
+writes:
+
+- `checkpoint_name:<normalized-name>` — unique name reservation pointing at the
+  immutable checkpoint ID;
+- `checkpoint:<checkpoint_id>` — header containing `name`, source logical
+  branch, `graph_commit_id`, source physical manifest branch and version,
+  manifest schema stamp, `created_at`, and `actor`;
+- `checkpoint_table:<checkpoint_id>:<stable-table-id>:<incarnation-hash>` — one
+  row per pinned table containing stable table identity, table key,
+  `table_path`, `table_branch`, `table_version`, and physical incarnation.
+
+The name reservation, header, and every table row land in one RFC-022 publisher
+CAS on main. First creation is insert-only; reuse requires the exact tombstoned
+reservation generation in the `ReadSet`. Two concurrent attempts for the same
+normalized name therefore conflict even when they captured different source
+branches. A missing or duplicate table row is an invalid checkpoint, not a
+partial checkpoint.
+
+Stable table identity/incarnation is a ship gate even when RFC-024 has not
+landed; checkpoint rows cannot key retention to mutable names. A deployment may
+reuse RFC-024's identity baseline, but durable heads themselves are not a
+dependency.
+
+Deletion changes the name reservation from `live` to `tombstoned` and writes
+manifest tombstones for the checkpoint header and all table rows in one CAS.
+The reservation and every tombstone carry the same fresh
+`delete_operation_id`; that CAS is the durable delete/recovery marker. The
+retention claim may repeat the operation ID while the writer is live, but it is
+only a fence, never recovery authority. Checkpoint IDs are never reused.
+
+Checkpoint create/delete are manifest metadata transactions. They do not create
+a graph-content commit or move `graph_head`: taking a checkpoint must not change
+the graph state it names. They still use RFC-022's read-set arbitration,
+manifest CAS, actor, and audit contracts. Creation uses a recovery sidecar for
+its pre-CAS tag effects; deletion is authority-first and embeds its idempotent
+operation marker in the tombstone CAS instead of writing a generic sidecar.
+
+The named snapshot is the source branch version and graph commit pinned in step
+2 of creation, not whatever is current when the later main-registry CAS lands.
+That immutable version may remain a valid checkpoint if the source branch
+advances after capture. The response always returns the exact captured commit
+and manifest version; it never implies that the checkpoint tracks a moving tip.
+
+### 2.2 Physical protection: Lance tags
+
+Every checkpoint owns deterministic Lance tags of two kinds:
+
+```text
+ogcp_<checkpoint-id-base32>_manifest_<branch-hash>
+ogcp_<checkpoint-id-base32>_<table-and-lineage-hash>
+```
+
+The manifest tag targets the exact `(manifest_branch, manifest_version)` named
+by the header. Each table tag targets the exact `(table_branch, table_version)`
+recorded by its table row. The spelling stays within Lance tag-name constraints
+and includes enough physical-lineage identity to avoid collisions across lazy
+forks. Pinning `__manifest` itself is mandatory: data-table tags alone cannot
+preserve the schema, registrations, and lineage needed to reconstruct the graph
+snapshot.
+
+Internal `ogcp_` tags are reserved. User-supplied tooling must not create,
+move, or delete them. If an existing deterministic tag points anywhere else,
+checkpoint creation and cleanup fail loudly with a typed corruption error.
+
+Tags are derived state. A live checkpoint row with a missing tag is repaired
+from the row. An internal tag with neither a live checkpoint nor an in-flight
+checkpoint sidecar is orphaned and may be removed.
+
+Tags protect versions from `cleanup_old_versions`; they do **not** protect a
+named branch from Lance branch deletion, which removes `tree/<branch>`. The
+initial contract therefore refuses physical deletion of any manifest or data
+branch named by a live checkpoint row. Graph branch deletion and orphan-branch
+reclamation read the main checkpoint registry under the retention claim and
+return `CheckpointPinsBranch` with the blocking checkpoint IDs. They never call
+`force_delete_branch` on a pinned lineage. Operators delete the checkpoints
+first; hidden checkpoint branches or full snapshot copies require a separate
+design.
+
+### 2.3 Cross-process retention claim
+
+Checkpoint create/delete, pin reconciliation, destructive cleanup, and graph
+branch deletion serialize through one graph-wide, substrate-backed retention
+claim. Acquisition uses the storage adapter's atomic create-if-absent primitive
+(`PutMode::Create`) on `__leases/retention.json`; Lance branch creation is
+explicitly rejected because its branch-metadata step is an existence check
+followed by an unconditional put.
+
+The claim payload records operation ID, action, actor, creation time, and a
+random fencing/owner token. Every protected phase re-reads and verifies that
+token. Normal release verifies the exact token before deleting the claim; no
+other owner can replace it while create-if-absent observes the object. There is
+no time-based lease stealing. Crash takeover requires an explicit fleet write
+outage, classification of the sidecar/manifest operation marker, removal of the
+stale claim, and acquisition with a new token. A resumed stale process must
+fail its next token check.
+
+A process-local mutex is only a contention optimization. The retention claim is
+held across tag creation or physical cleanup, preventing this race: cleanup
+computes roots, another process captures an untagged old version, then cleanup
+deletes it before the tag lands. A crash leaves the claim and therefore
+over-retains. Takeover first classifies the prior operation from its recovery
+sidecar and/or exact manifest operation marker, then may release the claim
+only after that operation is completed or safely aborted. It never steals on
+elapsed wall time alone.
+
+The claim is not presented as a distributed reader/writer lock for arbitrary
+graph writes. Initial destructive cleanup is an **offline** operation: operators
+quiesce every server, CLI, embedded writer, and MemWAL stream before acquiring
+the claim. New binaries also refuse to start a write while the claim exists,
+but that check is defense in depth rather than proof that an already-running
+foreign writer drained. Online cleanup requires a separate fleet writer-epoch
+or substrate lease design and is out of scope.
+
+## 3. Checkpoint create protocol
+
+Checkpoint creation is a writer on the RFC-022 pipeline:
+
+1. Run and await RFC-022's synchronous recovery barrier.
+2. Authorize, capture one immutable source-branch manifest snapshot and graph
+   commit, and prepare the complete header/table/tag set plus a `ReadSet` that
+   includes source branch incarnation, source manifest version, format/schema
+   identity, applicable GC boundaries, and the main-registry name reservation;
+   a source at or behind a pruned-through boundary is refused even if its files
+   happen to remain.
+3. Acquire the retention claim, checkpoint/cleanup process-local serialization,
+   source branch-control gate, and adapter-declared metadata gates in RFC-022's
+   canonical order. Ordinary data queues are not held merely because immutable
+   versions are being tagged.
+4. Freshly revalidate the complete `ReadSet`. A changed branch incarnation,
+   missing tag target, conflicting name, or format change releases the gates and
+   restarts before any tag exists.
+5. Write a generic recovery sidecar containing the intended rows and tags.
+6. Create the manifest tag and every table tag idempotently. Existing correct
+   tags are no-ops; conflicting tags fail.
+7. Verify every tag, then release source branch control. A source-head advance
+   is harmless because the tags already pin the exact captured version;
+   source-branch deletion takes the same retention claim, classifies overlapping
+   create sidecars, and then refuses if the published checkpoint pins the ref.
+8. Publish the name/header/table rows in one CAS on the main manifest registry.
+9. Delete the recovery sidecar. Sidecar deletion remains best-effort after the
+   authority publish, as in the existing write protocol.
+10. Release the retention claim after the outcome is durably classifiable.
+
+A crash before step 8 leaves tags but no checkpoint. Recovery may complete the
+publish when every precondition still holds or remove the orphan tags. A crash
+after step 8 leaves a valid checkpoint even if sidecar cleanup did not finish.
+
+## 4. Checkpoint delete protocol
+
+Deletion deliberately reverses the create order:
+
+1. run the recovery barrier, authorize, and prepare the complete checkpoint plus
+   name/format/registry `ReadSet`;
+2. acquire the retention claim and local gates in canonical order, then freshly
+   revalidate the complete checkpoint;
+3. publish the tombstoned name reservation plus header/table tombstones, all
+   carrying one fresh `delete_operation_id`, in one manifest CAS;
+4. release the claim and acknowledge once the authority change is durable;
+5. delete the deterministic Lance tags asynchronously and idempotently.
+
+A failed tag deletion only retains extra data. The checkpoint is already gone
+from the user-visible namespace. The reconciler and the next `cleanup` retry
+the physical reclaim.
+
+Delete is an RFC-022 authority-first metadata workflow: no independently durable
+effect precedes the atomic tombstone CAS, and later tag deletion is derived,
+over-retaining cleanup. It therefore needs no generic write sidecar. The
+`delete_operation_id` persisted by the tombstone CAS lets crash recovery
+distinguish pre-CAS from post-CAS state exactly; the retention claim does not.
+
+## 5. Pin reconciler
+
+Read-write open and every destructive `cleanup` run reconcile checkpoint pins
+from the main-manifest checkpoint registry.
+The reconciler:
+
+1. reads the live checkpoint rows once;
+2. classifies checkpoint sidecars before touching apparently orphaned tags;
+3. creates or verifies every required tag;
+4. deletes internal tags proven to have no live or in-flight authority;
+5. emits a typed result for repaired pins, reclaimed pins, and failures.
+
+`cleanup` is fail-closed: any missing, conflicting, unreadable, or ambiguous
+pin aborts deletion for the affected graph. It never treats reconciliation
+failure as permission to collect data.
+
+The reconciler runs under the retention claim and is serialized with
+checkpoint creation and table maintenance.
+It may run concurrently across independent graphs, but it must not delete a
+tag belonging to a foreign process's live create sidecar.
+
+## 6. Cleanup protocol and pruned-through boundary
+
+The cleanup root set is:
+
+- every live graph branch head;
+- every live checkpoint manifest and table row;
+- every version inside the operator-selected time/count retention window;
+- versions Lance itself protects through non-OmniGraph tags or branch refs.
+
+Read-only preview may run without a claim, but it is explicitly provisional.
+Confirmed execution uses this order:
+
+1. establish the operator write outage, persistently seal/fold streams through
+   RFC-026 **before** taking the retention claim, and complete the RFC-022
+   recovery barrier;
+2. acquire the retention claim, then revalidate the fleet outage, every stream's
+   `SEALED` cut, and absence of recovery sidecars;
+3. run pin reconciliation and recompute the exact root set and per-dataset
+   cutoffs under the claim;
+4. prepare and revalidate a complete RFC-022 `ReadSet` containing format/schema
+   identity, branch incarnations, current manifest table entries/heads, prior GC
+   boundaries, and the root digest;
+5. publish mutable `gc_boundary:<dataset-lineage-hash>` rows containing cutoff,
+   root digest, cleanup operation ID, and timestamp in one reserved-main
+   manifest CAS;
+6. invoke Lance `cleanup_old_versions`, relying on the verified tags to retain
+   sparse checkpoint versions;
+7. record per-table removal statistics/failures and release the claim only when
+   the operation is durably resumable or complete.
+
+Step 5 is an RFC-022 authority-first metadata transaction: it changes which
+versions recovery may select, carries no graph-content lineage, and needs no
+sidecar because no physical effect precedes its CAS. Step 6 is RFC-022 §8
+physical maintenance under the already-published boundary. The two are not one
+indistinguishable “cleanup commit.”
+
+The boundary advances before delete. Any later recovery, restore, or publisher path
+that could make an older physical version current must include the boundary in
+its RFC-022 `ReadSet` and revalidate it before the physical effect. A position
+at or behind the boundary is retried from current state or refused. Cleanup
+never starts with an armed recovery that may still need an older version.
+
+Per-table cleanup remains fault-isolated, but partial operational success is
+reported explicitly. A successful table does not hide another table's error.
+The claim metadata plus `gc_boundary` operation ID is the cleanup recovery
+marker: takeover resumes or reports the remaining table units under the same
+root digest before releasing the claim; it does not silently recompute a
+broader destructive plan after a partial run.
+
+### 6.1 Operational cadence — the cost of never running this
+
+Offline-only destructive cleanup has a predictable failure mode: operators
+defer it indefinitely, and the graph silently reacquires the unbounded-history
+cost class this program measured (per-version chains that grow one entry per
+commit; latest-version listings whose page size grows with history; on hosted
+deployments, seconds of added latency per operation). The docs and `cleanup`
+preview therefore state the deferral cost explicitly, and deployments are
+expected to schedule a periodic maintenance window — for continuously written
+graphs, the same cadence discipline as `optimize` — rather than treating
+cleanup as exceptional. Read-only preview and pin reconciliation remain online
+so drift is visible between windows.
+
+Online destructive cleanup — concurrent with live writers under a fleet
+writer-epoch or substrate lease — is the named successor design, deliberately
+out of scope here (§2.3). This RFC's offline barrier is the correct first
+delivery, not the end state; a deployment for which write outages are
+unacceptable should treat the successor design as the gating requirement for
+adopting aggressive retention policies.
+
+## 7. Public and policy surface
+
+Initial delivery is engine plus direct-storage CLI, matching `cleanup`:
+
+```text
+omnigraph checkpoint create <name> [--branch <branch>] <store>
+omnigraph checkpoint list <store>
+omnigraph checkpoint show <name-or-id> <store>
+omnigraph checkpoint delete <name-or-id> <store>
+```
+
+JSON output includes checkpoint ID, name, actor, branch, graph commit, creation
+time, table count, and pin-health status. `cleanup` preview reports which branch,
+checkpoint, or window protects each retained version and states that execution
+requires a graph-wide write outage.
+
+Checkpoint creation and deletion use dedicated Cedar actions. Embedded callers
+hit the same engine gate as the CLI. HTTP management endpoints are out of scope
+until server-side maintenance has a general design; no transport-only bypass is
+introduced here.
+
+## 8. Migration and compatibility
+
+This RFC owns its own internal-format activation. RFC-022 authorizes no format
+bump, and RFC-024 explicitly excludes checkpoint rows from its heads format.
+The retention format may receive the next schema version after heads, or the
+two may share one release only if both RFCs are independently accepted and
+RFC-024 is amended before implementation. A fresh activated graph starts with
+no named checkpoints and a zero pruned-through boundary. No older commit is
+silently promoted into a checkpoint.
+
+The upgrade mechanism must be chosen before implementation:
+
+- an in-place upgrade preserves branches and history and must define quiescence,
+  crash recovery, stamp ordering, and rollback;
+- export/import creates a fresh graph at the new format but loses the old commit
+  DAG, branches, snapshots, and time-travel history. Those losses must be shown
+  in the plan and confirmation output; there is nothing left to backfill as a
+  checkpoint.
+
+Mixed-version writers are unsupported. An old binary must not write after the
+retention format stamp or tag protocol becomes authoritative.
+
+## 9. Observability and bounds
+
+Expose:
+
+- live checkpoint count and pinned table-version count;
+- missing, conflicting, repaired, and orphan-tag counts;
+- oldest checkpoint age and retained bytes when Lance reports them;
+- current GC boundary and root digest per dataset lineage;
+- cleanup versions/bytes removed and per-table failures;
+- checkpoint create/delete/reconcile latency and retry counts.
+
+Checkpoint enumeration and cleanup planning must be bounded by live checkpoints
+times catalog width, not total commit history. Operators can limit checkpoint
+count by policy; the engine refuses names or payloads beyond configured bounds
+before creating tags.
+
+That is a physical-I/O claim, not just a logical row-count claim. Registry
+lookup must use a structured Lance access path with a measured bound on
+uncovered fragments, reusing RFC-024's in-manifest scalar-index work when it is
+available or proving an equivalent access shape here. A filtered scan that
+still reads history-sized manifest fragments does not pass this RFC's cost gate;
+a separate checkpoint dataset is rejected because its authority rows could not
+share the main-manifest CAS.
+
+## 10. Acceptance gates
+
+- A checkpoint on an old sparse version survives cleanup while adjacent
+  unpinned versions are removed.
+- Branch delete and orphan reclamation refuse a non-main manifest/data lineage
+  while a live checkpoint references it; after checkpoint deletion and tag
+  reconciliation, branch deletion succeeds. `force_delete_branch` is never used
+  to bypass the pin.
+- The source `__manifest` version survives cleanup; data-table tags without the
+  manifest tag fail the checkpoint-validity test.
+- Concurrent creation of the same normalized name yields exactly one authority
+  record; the losing attempt leaves only safely reclaimable tags.
+- After deletion, concurrent attempts to reuse the tombstoned name generation
+  yield exactly one fresh checkpoint ID; the old checkpoint ID never revives.
+- Create failpoints cover sidecar, tag, and manifest boundaries; delete
+  failpoints cover authority CAS, claim release, and tag reclaim. Every outcome
+  converges to valid pins or safe over-retention.
+- Missing live tags are repaired before cleanup; conflicting tags block cleanup.
+- The retention claim blocks checkpoint create/delete/reconcile and branch delete
+  during cleanup on local FS and S3; destructive cleanup refuses active streams
+  or recovery sidecars.
+- Two-process races prove the atomic retention claim, not a process-local gate,
+  closes checkpoint-vs-branch-delete windows; crash takeover never relies on
+  wall-clock expiry alone.
+- Local and S3 claim tests prove `PutMode::Create` admits exactly one owner and
+  that mismatched owner tokens cannot release another operation's claim.
+- A documented fleet write-outage test runs writers before and after cleanup;
+  online writer-vs-cleanup concurrency is explicitly not advertised.
+- Genuine cross-version tests cover the selected upgrade path.
+- Cost tests use `helpers::cost` at realistic commit depth and prove that a
+  steady checkpoint list/lookup and cleanup plan do not grow with commit count
+  once physical layout is held constant, including a bounded uncovered tail.
+
+## 11. Phasing
+
+| Phase | Content | Gate |
+|---|---|---|
+| A | row schemas, engine DTOs, format activation, deterministic tag encoding | schema and tag surface guards |
+| B | create sidecar, delete-operation fields in the authority CAS, and reconciler | crash matrix; sparse-pin correctness |
+| C | offline cleanup integration and GC boundary enforcement | quiescence/refusal tests; cost budgets |
+| D | CLI, policy, audit, docs, and selected migration path | CLI outputs; genuine upgrade test |

docs/rfcs/rfc-026-memwal-streaming-ingest.md

@@ -0,0 +1,415 @@
+---
+type: spec
+title: "RFC-026 — MemWAL streaming ingest"
+description: Adopts Lance MemWAL as OmniGraph's strategic streaming-write architecture, with durable per-row acknowledgement, graph-atomic folds, epoch-fenced quiescence, and explicit fresh-read cuts on the RFC-022 unified write path.
+status: draft
+tags: [eng, rfc, streaming, ingest, wal, memwal, lance, omnigraph]
+timestamp: 2026-07-10
+owner:
+---
+
+# RFC-026 — MemWAL streaming ingest
+
+**Status:** Draft / for team review
+**Date:** 2026-07-10
+**Depends on:** [RFC-022](rfc-022-unified-write-path.md)'s unified write and
+generic recovery-sidecar protocol, plus
+[RFC-023](rfc-023-key-conflict-fencing.md) for the initial keyed graph-stream
+mode. Durable heads from
+[RFC-024](rfc-024-durable-table-heads.md) are compatible but not required.
+**Surveyed:** omnigraph 0.8.1; Lance 9.0.0-beta.15 (`f24e42c1`); complete MemWAL format specification
+**Audience:** engine, server, CLI, policy, and operations maintainers
+**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md).
+Findings marked **BLOCKER** must be dispositioned before acceptance.
+
+---
+
+## 0. Decision and risk posture
+
+OmniGraph adopts Lance MemWAL as its strategic streaming-write architecture.
+MemWAL is a major Lance architectural bet: a sharded LSM write path with durable
+WAL entries, flushed Lance generations, merge progress committed with base-table
+data, maintained indexes, and epoch-fenced writers. OmniGraph consumes that
+architecture rather than building a WAL, shard protocol, or LSM reader.
+
+This RFC does **not** characterize the architecture as experimental. The risk is
+narrower: Rust API names, some format details, and operational helpers are still
+maturing across Lance releases. We manage that API/format-maturity risk with a
+small adapter, compile/runtime surface guards, a quiescence requirement before
+Lance upgrades, and a fresh full-spec alignment audit on every bump. It is not a
+reason to fork or reimplement MemWAL.
+
+The contract is:
+
+- stream acknowledgement means the row's WAL entry is durable;
+- acknowledgement does not mean graph visibility;
+- default queries see only the manifest-committed graph;
+- a fold is an ordinary RFC-022 graph writer and is the sole visibility point;
+- fresh reads are explicit and never claim cross-table atomicity.
+
+## 1. Scope and non-goals
+
+This RFC specifies enrollment, the public stream API, acknowledgement semantics,
+folding, fold-time integrity, dead-letter atomicity, branch/schema quiescence,
+fresh-read cuts, resource bounds, observability, testing, and upgrade posture.
+
+It does not replace `load` or `mutate`, provide cross-query transactions, store
+manifest mutations in MemWAL, create a second metadata authority, or weaken
+default snapshot isolation. Stream-mode
+deletes remain out of the first delivery and require the Lance tombstone surface
+plus a separate acceptance pass.
+
+## 2. Stream mode and key semantics
+
+Initial delivery exposes
+`@stream(mode="upsert", on_reject="strict")` on a node or edge type;
+`on_reject` accepts `strict` or `dead_letter`.
+It requires the table's immutable unenforced primary key to equal OmniGraph's
+merge key: `id` for nodes and edges. All occurrences of one key map to one shard
+and MemWAL applies last-write-wins ordering.
+
+Public append mode is deliberately out of scope. Nodes and edges always have
+logical identity; allowing a retry to append the same `id` twice would violate
+that contract. A future explicitly keyless, non-graph append-only table class
+may consume MemWAL append semantics under its own schema/API decision.
+
+Stream ordering intentionally differs from the interactive fence:
+
+- concurrent interactive same-key writes serialize or fail/retry loudly;
+- same-key stream entries resolve by MemWAL generation/position order;
+- duplicate keys inside one bulk-load input retain the existing load error.
+
+The schema and user docs state all three together.
+
+## 3. Enrollment is a recoverable inline commit
+
+Schema apply records `@stream` intent only. First stream use enrolls the physical
+table by creating the singleton `__lance_mem_wal` system index and its sharding
+configuration.
+
+Enrollment advances Lance HEAD inline. It therefore uses the RFC-022 generic
+recovery protocol, not an ad-hoc state machine:
+
+1. run and await RFC-022's synchronous recovery barrier;
+2. authorize, pin the manifest/schema/table state, and prepare a complete
+   `ReadSet` containing schema identity, table entry/head, PK metadata, stream
+   intent/configuration, and lifecycle-row absence;
+3. acquire any global claims and then the `(table, branch)` write queue in
+   RFC-022 order, then freshly revalidate the complete `ReadSet`; a mismatch restarts
+   before any inline effect;
+4. verify RFC-023's already-installed PK; enrollment never performs a first-use
+   PK migration; validate the sharding configuration;
+5. write a generic sidecar with writer kind `stream_enrollment` and pre-commit
+   table pin;
+6. create the MemWAL index, advancing Lance HEAD;
+7. publish the new table version plus `stream_state = OPEN` in one manifest CAS,
+   including the table-head row when RFC-024 is active;
+8. delete the sidecar best-effort after publication.
+
+Recovery rolls the enrollment forward or back under the same classification and
+audit machinery as other inline residuals. Repeating enrollment with identical
+metadata is a no-op. A different PK, sharding spec, maintained-index set, or
+writer-default configuration is a typed conflict.
+
+No row is acknowledged until enrollment is manifest-committed.
+
+Initial delivery supports one unsharded shard per `(table, main)`. Non-main
+branches remain refused until the Lance branch-scoping question is proven by a
+surface guard and end-to-end test. Later `bucket(id, N)` sharding must preserve
+the one-key-to-one-shard rule.
+
+## 4. New public API
+
+The shipped `POST /graphs/{id}/ingest` path remains the deprecated, compatible
+alias of `/load`. Streaming receives a new, non-conflicting surface:
+
+```text
+POST /graphs/{graph_id}/streams/{type_name}/ingest?branch=main
+GET  /graphs/{graph_id}/streams
+GET  /graphs/{graph_id}/streams/{type_name}
+POST /graphs/{graph_id}/streams/{type_name}/fold
+POST /graphs/{graph_id}/streams/{type_name}/quiesce
+POST /graphs/{graph_id}/streams/{type_name}/resume
+```
+
+The ingest request and response use `Content-Type: application/x-ndjson` and
+`Accept: application/x-ndjson`.
+
+Each input line is one row payload. Each output line corresponds to the same
+input ordinal:
+
+```json
+{"ordinal":17,"status":"durable","shard_id":"...","writer_epoch":8,"wal_position":42}
+```
+
+Synchronous validation failures return a per-row error before a WAL append.
+Previously acknowledged rows in the same request remain durable; the response
+is a stream, not an all-request transaction. Ordering, cancellation, and retry
+rules are explicit:
+
+- acknowledgements are emitted in input order for one HTTP stream;
+- disconnecting does not cancel entries whose durability waiter resolved;
+- a missing response is ambiguous; retrying the same `id` and payload may add
+  another WAL entry but produces the same last-write-wins graph state;
+- server shutdown stops admission, drains durability waiters up to a bound, and
+  reports any unacknowledged tail as unknown to the client.
+
+CLI commands mirror the new namespace rather than overloading deprecated
+`omnigraph ingest`:
+
+```text
+omnigraph stream ingest <type> --data <ndjson> ...
+omnigraph stream status [<type>] ...
+omnigraph stream fold [<type>] ...
+omnigraph stream quiesce [<type>] ...
+omnigraph stream resume [<type>] ...
+```
+
+Every endpoint has a dedicated OpenAPI operation and handler tests. Ingest
+passes the engine `stream_ingest` Cedar action and per-actor admission
+accounting before acquiring a shard writer; fold/quiesce/resume use a separate
+`stream_manage` action. Status is authorized like other graph operational
+metadata. The same engine gates apply to embedded and remote CLI use.
+
+## 5. Ack-path validation and writer lifecycle
+
+Before append, OmniGraph applies checks that need no base-table read: Arrow
+shape/type, required/default fields, enum/range/check constraints, reserved
+columns, and stream mode. RI, cardinality, cross-version uniqueness, and
+external embedding computation remain fold-time work.
+
+One warm `ShardWriter` is held per active shard behind a bounded registry. The
+registry has idle eviction and hard limits for resident writers, MemTable bytes,
+unflushed WAL bytes, pending generations, and per-actor inflight bytes. Exceeding
+a bound backpressures with a typed retryable response; it never drops a row.
+
+Initial topology has one active ingest owner for each `(graph, table, main)`
+shard. MemWAL's epoch fence makes restart/failover safe; it is not a load
+balancer. A deployment with multiple server replicas must route a shard to its
+current owner (or return a typed retry/redirect) instead of letting replicas
+reclaim the epoch per request. General multi-owner routing waits for the
+multi-shard phase and its ownership protocol.
+
+## 6. Fold protocol
+
+The fold consumes flushed generations in ascending order. Embeddings and
+base-dependent validation run outside the table queue and register every probed
+table plus stream configuration/generation in RFC-022's `ReadSet`. The commit
+phase then:
+
+1. stages accepted rows with Lance merge-insert and includes
+   `merged_generations` in that transaction;
+2. stages any rejection/audit rows required by §7;
+3. acquires every affected queue in canonical sorted order and revalidates the
+   complete `ReadSet`; any mismatch discards and replans the whole fold;
+4. writes one generic RFC-022 recovery sidecar before the first
+   `commit_staged` call;
+5. commits every staged Lance transaction;
+6. publishes all data/internal table versions and lineage in one `__manifest`
+   CAS, including table heads when RFC-024 is active;
+7. deletes the sidecar after successful publication.
+
+The sidecar is mandatory even though merge-insert is staged. After
+`commit_staged`, Lance HEAD and `merged_generations` have moved while the graph
+manifest has not. A failure in that window is the ordinary multi-table recovery
+gap, not invisible staged state.
+
+MemWAL generation GC starts only after the exact fold is graph-visible, its
+sidecar is resolved, index catchup permits reclamation, and no `FreshReadCut`
+retention guard references the generation. Data-HEAD merge progress alone is
+never permission to delete the only fresh-tier copy.
+
+Concurrent folders reload `merged_generations`: a generation already committed
+is skipped; otherwise the fold is replanned from current state. A forced fold
+stops new flush creation for its cut, waits for in-flight durability waiters,
+and never aborts an acknowledged row.
+
+Fold lineage uses `omnigraph:ingest` as the mechanism actor and persists the
+authenticated contributor actor for every folded WAL range in the same internal
+audit participant. Commit and status output can therefore answer both who ran
+the fold and who supplied the data after WAL GC.
+
+## 7. Fold-time rejection is atomic
+
+`strict` is the default. A permanent RI, cardinality, uniqueness, or embedding
+failure stops the fold at the offending generation, marks the shard blocked,
+and backpressures new ingestion once configured lag bounds are reached.
+
+`dead_letter` is explicit. `_ingest_rejects` is then a versioned internal Lance
+table and a participant in the same fold pipeline, not best-effort state outside
+the commit protocol. Every reject has deterministic identity:
+
+```text
+(table_key, shard_id, generation, wal_position)
+```
+
+The fold stages reject rows, accepted rows, and merge progress before committing
+any of them. The generic sidecar covers every participant; the single manifest
+publish records both the base-table and reject-table versions. Replay is
+idempotent by reject identity. There is no ordering in which progress can become
+visible while the corresponding rejection is lost.
+
+`stream status` reports blocked generations and typed reject details. Reject
+retention is explicit and cannot be shorter than the WAL/fold audit retention
+needed to explain a durable acknowledgement.
+
+## 8. Epoch-fenced quiescence barrier
+
+Branch operations, schema changes, stream teardown, and Lance upgrades require a
+real barrier, not an empty check.
+
+Each enrolled table has a durable
+`stream_state:<stable-table-id>:<incarnation>` row in its manifest branch with
+`OPEN | DRAINING | SEALED`, configuration hash, and epoch floor. The row is the
+logical lifecycle authority and is updated by an RFC-022 CAS; MemWAL shard
+epochs are the physical writer fence. Neither an in-memory registry nor an
+empty-generation observation can substitute for both.
+Lifecycle-only transitions are audited manifest metadata transactions; they do
+not create graph-content commits or move `graph_head`.
+
+The shared drain sequence is:
+
+1. publish stream intent `OPEN -> DRAINING` for the target table/branch;
+2. increment and persist each shard writer epoch and seal claims, fencing stale
+   writers across processes;
+3. reject or backpressure new appends;
+4. wait for in-flight durability waiters, flush active MemTables, and fold every
+   generation to empty;
+5. verify shard manifests and base `merged_generations` agree on emptiness;
+6. publish `DRAINING -> SEALED` with the verified generation/epoch cut;
+7. for an operation-scoped drain, perform the guarded operation; persistent
+   public quiesce stops after step 6.
+
+Each lifecycle CAS is an RFC-022 authority-first metadata write; each fold is a
+separate normal RFC-022 graph write. `DRAINING` fully encodes the target epoch
+floor, so an interrupted epoch mutation is idempotently resumed from that row.
+The sequence is not one giant sidecar spanning multiple commits.
+
+There are two dispositions after the drain reaches `SEALED`:
+
+- **operation-scoped drain** — branch/schema maintenance automatically publishes
+  `SEALED -> OPEN` with a newer epoch only after the guarded operation succeeds
+  and the stream contract remains compatible;
+- **persistent quiesce** — the public `quiesce` command leaves the stream
+  `SEALED`. It never auto-reopens. `stream resume` explicitly revalidates schema,
+  PK, configuration, MemWAL format, and epoch, then publishes a newer `OPEN`
+  state. Stream teardown deletes intent only from `SEALED`.
+
+The barrier never holds the table write queue while waiting for a fold that
+needs that queue. State transition and epoch fencing happen first; fold commit
+then acquires the normal queue. Crash recovery resumes from the durable state
+and epoch.
+
+Schema apply must drain every affected enrolled type before changing fields,
+constraints, PK, embeddings, or `@stream` and resumes only when compatible.
+
+A Lance version upgrade requires persistent `stream quiesce --all`, but empty
+generations alone are insufficient: the MemWAL system index, shard manifests,
+epoch records, and generation directories may still use the old format. Before
+the bump, the implementation must prove one of: (a) upstream guarantees and
+cross-version tests cover every retained MemWAL artifact, (b) a public Lance
+metadata migration converts them, or (c) OmniGraph tears down the enrolled
+MemWAL metadata under recovery and re-enrolls after the bump. Without one of
+those gates the upgrade refuses; `resume` never opens unverified old metadata.
+
+## 9. Fresh-read cuts
+
+Freshness is a first-class engine/IR enum:
+
+```text
+Committed
+Fresh
+```
+
+At query planning, `Fresh` captures one `FreshReadCut` containing:
+
+- the ordinary manifest snapshot;
+- each selected shard-manifest version and writer epoch;
+- included flushed-generation paths and maximum generation;
+- the active same-process MemTable row-position watermark, when available;
+- the base table's `merged_generations` and index-catchup state read from the
+  exact table version selected by the manifest snapshot, never from live HEAD.
+
+Capture uses a retrying handshake:
+
+1. read the selected shard manifests/epochs, acquire Lance generation retention
+   guards for the flushed files in the tentative cut, and under one
+   same-process writer snapshot capture/pin any active-MemTable watermark;
+2. pin the graph manifest snapshot and read `merged_generations` from each exact
+   base-table version it selects;
+3. re-read the shard manifest versions/epochs; any epoch/configuration change
+   restarts the whole capture;
+4. if a generation from step 1 disappeared, accept that only when the pinned
+   base's `merged_generations` proves it is included; otherwise release guards,
+   discard the whole graph snapshot, and retry from step 1;
+5. exclude generations that appeared after step 1 and hold the generation and
+   MemTable read guards captured in step 1 until query completion.
+
+If Lance exposes no guard that prevents generation GC for the query lifetime,
+cross-process `Fresh` does not ship. A missing generation is never interpreted
+as “probably folded” against an older pinned base.
+
+Execution never refreshes that cut mid-query. It excludes every flushed
+generation `<= merged_generations[shard]`; otherwise old WAL data could outrank
+or duplicate its newer base-table image.
+
+Fresh reads have no cross-table atomicity. Same-process active MemTables provide
+read-your-writes; other processes can promise only the latest flushed state
+captured by their shard-manifest reads. The HTTP request and query docs state
+those limits wherever the tier is exposed.
+
+## 10. Observability and resource contracts
+
+Per shard expose durable WAL position, replay position, active epoch, current
+generation, flushed and merged generation, index catchup, pending rows/bytes,
+oldest acknowledged age, last fold error, blocked reject, and quiescence state.
+
+Metrics cover ack latency, durability-wait batching, fenced writers, replayed
+entries, fold rows/bytes/generations, fold retries, lag, reject counts, and
+sidecar recovery. Defaults for every byte/count/time bound are documented and
+configuration changes are observable behavior.
+
+`stream status` resolves the exact lifecycle rows and MemWAL metadata through a
+structured, bounded access path; it may reuse RFC-024's scalar-index machinery
+but cannot claim history-flat cost while scanning manifest history.
+
+## 11. Acceptance gates
+
+- Surface guards pin claim, append, durability waiter, flush, epoch fencing,
+  staged merge with `merged_generations`, index catchup, and seal/reopen APIs.
+- A WAL append failure emits no durable acknowledgement. Every acknowledged row
+  survives crash, replay, fold, and recovery.
+- Failpoints cover enrollment's inline-commit gap and every fold participant
+  around sidecar, `commit_staged`, reject persistence, and manifest publish.
+- Enrollment restarts without an inline effect when schema, PK, table head, or
+  stream configuration changes between prepare and gated revalidation.
+- Two folders converge exactly once; a fenced stale writer can never produce a
+  false durable acknowledgement after WAL GC.
+- Two server replicas do not epoch-ping-pong one shard; owner failover fences
+  the old process before the new owner acknowledges.
+- Quiescence tests race appends with branch and schema operations across two
+  coordinators and prove no post-drain tail appears; failpoints cover every
+  lifecycle-row, epoch-fence, fold, and `SEALED` boundary.
+- Persistent quiesce never auto-reopens; explicit resume validates a newer
+  epoch. Upgrade tests cover every retained MemWAL artifact through declared
+  compatibility, migration, or teardown/re-enrollment.
+- Fresh reads race capture with fold/GC, retry on an unexplained disappearing
+  generation, hold generation/MemTable guards through execution, exclude merged
+  generations, and document cross-table inconsistency explicitly.
+- Server/OpenAPI tests preserve the old `/ingest` alias and cover the new route;
+  CLI parity covers embedded and remote stream commands.
+- Ack-path object-store operations are O(1) and flat in graph history and WAL
+  depth. Fold cost is bounded by generations/rows folded, not graph history.
+- S3 correctness runs against RustFS; API/format guards are rerun before every
+  Lance bump.
+
+## 12. Phasing
+
+| Phase | Content | Gate |
+|---|---|---|
+| A | MemWAL adapter, surface guards, enrollment sidecar | inline-commit crash matrix |
+| B | new ingest route/CLI, durable ack, strict fold | ack durability; API compatibility; cost budget |
+| C | atomic dead letter, audit provenance, status/bounds | reject crash matrix; backpressure tests |
+| D | epoch-fenced drain, persistent quiesce/resume, schema/branch/upgrade integration | two-coordinator race and format-transition suite |
+| E | fresh cuts and maintained-index reads; cross-process `Fresh` ships only if the substrate generation-retention guard exists (§9), otherwise same-process only | cut consistency; merged-generation exclusion |
+| F | multi-shard upsert and stream deletes | one-key-one-shard proof; Lance re-audit |

docs/rfcs/rfc-027-lineage-merge-deltas.md

@@ -0,0 +1,254 @@
+---
+type: spec
+title: "RFC-027 — Lineage-based merge deltas"
+description: Research specification for replacing full-width branch-merge classification with Lance row-version lineage, explicitly blocked on a sublinear deletion-delta source and enforceable I/O cost gates.
+status: research-blocked
+tags: [eng, rfc, merge, lineage, change-feed, performance, lance, omnigraph]
+timestamp: 2026-07-10
+owner:
+---
+
+# RFC-027 — Lineage-based merge deltas
+
+**Status:** Research / blocked on deletion-delta discovery
+**Date:** 2026-07-10
+**Depends on:** [RFC-022](rfc-022-unified-write-path.md)'s unified branch-merge
+pipeline and capture-once write view
+**Surveyed:** omnigraph 0.8.1; Lance 9.0.0-beta.15 (`f24e42c1`)
+**Audience:** merge, storage, and performance maintainers
+**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md).
+Findings marked **BLOCKER** must be dispositioned before acceptance.
+
+---
+
+## 0. Status and decision boundary
+
+The direction is recommended: branch merge should discover changed row IDs from
+storage lineage, then read wide values only for those candidates. The proposed
+replacement is **not implementation-ready** and this RFC does not authorize
+removing `OrderedTableCursor`.
+
+Two facts block the O(delta) claim:
+
+1. filtering `_row_last_updated_at_version` is still a physical O(rows) column
+   scan unless a substrate index or change-log makes it selective;
+2. a deleted row is absent from the target snapshot, so its version columns
+   cannot identify it. The current implementation finds deletions by scanning
+   and differencing both complete ID sets.
+
+The RFC advances only when both candidate discovery and deletion discovery have
+measured costs bounded by the changed working set. Until then the existing merge
+classifier remains the correctness fallback.
+
+## 1. Problem
+
+Current three-way merge classification streams base, source, and target tables
+and compares row signatures. A one-row change can therefore read every row and
+every wide property, including embeddings and blobs, while holding merge-wide
+coordination longer than necessary.
+
+`changes/mod.rs` is narrower but not yet asymptotically different:
+
+- changed live rows are filtered by `_row_last_updated_at_version`;
+- insert/update classification builds the full base ID set;
+- deletion classification scans both ID sets;
+- cross-branch fallback scans and compares complete ordered rows.
+
+Reusing that code unchanged is O(rows), not O(delta). This RFC replaces those
+specific scans rather than relabeling them.
+
+## 2. Target contract
+
+For each table touched since the merge base:
+
+1. discover the source and target candidate row IDs from Lance metadata;
+2. classify insert, update, and delete without loading unrelated user columns;
+3. join the two candidate sets into the existing merge truth table;
+4. fetch complete rows only for candidates whose disposition needs values;
+5. stage the resulting delta through RFC-022 and publish once.
+
+The semantic oracle remains `merge_truth_table`. This RFC changes candidate
+discovery and I/O, not conflict kinds, delete/update precedence, constraint
+validation, or manifest atomicity.
+
+## 3. Live-row candidates
+
+On a stable-row-ID table in one physical lineage, Lance exposes:
+
+- `_row_created_at_version` — first creation of the logical row;
+- `_row_last_updated_at_version` — latest modification of the logical row.
+
+For merge-base table version `Vb` and side version `Vs`, a live row is:
+
+- an insert when `Vb < created_at <= Vs`;
+- an update when `created_at <= Vb` and
+  `Vb < last_updated_at <= Vs`.
+
+This classification removes the current full base-ID membership set. Candidate
+scans project only `id`, edge endpoints when applicable, stable row ID, and the
+two version columns.
+
+It does **not** by itself make the scan O(delta). Phase R1 must prove one of:
+
+1. Lance can build and maintain a scalar index over the version columns with
+   correct partial-coverage fallback;
+2. transaction/fragment metadata exposes an equivalent bounded changed-row
+   iterator;
+3. an upstream Lance change-feed primitive supplies these IDs directly.
+
+An index is derived state. Missing or partial coverage falls back to a correct
+narrow scan and is reported; it never makes merge fail or omit candidates.
+
+## 4. Deletion delta is the blocker
+
+Deleted logical rows have no live record whose version columns can be filtered.
+The current `deleted_ids_by_set_diff` scans all IDs at base and side. That term
+alone prevents an O(delta) merge, even if inserts and updates become selective.
+
+Research must disposition these substrate-shaped options:
+
+### 4.1 Deletion-vector and stable-row-ID lineage
+
+Walk only transactions/fragments changed between `Vb` and `Vs`, compare their
+deletion vectors, and translate newly deleted offsets to stable row IDs. This is
+acceptable only if it handles merge-insert rewrites, updates that move rows,
+compaction, fragment reuse, and deletion-vector materialization without scanning
+unaffected fragments.
+
+### 4.2 Upstream Lance change log
+
+Consume a public Lance API that yields durable inserted/updated/deleted stable
+row IDs by version range. A source-level prototype or private API is evidence,
+not a dependency; the production surface must be public and pinned by
+`lance_surface_guards.rs`.
+
+### 4.3 Atomic OmniGraph deletion deltas
+
+Write immutable per-commit deleted-ID rows in the same manifest CAS as the graph
+commit. This is first-class commit metadata, not a side channel. It adds storage
+and format liability and therefore needs a separate format decision before use;
+it is not silently folded into v5.
+
+Until one option passes the cost and correctness gates, delete-bearing histories
+use the existing classifier. There is no "usually O(delta)" claim that excludes
+deletes without saying so in plan and metrics.
+
+## 5. Branch lineage and unsupported operations
+
+Lance version numbers are branch-local and can overlap. Candidate discovery must
+use each manifest entry's physical `(table_path, table_branch, table_version)`;
+it must not subtract graph commit numbers or compare equal numeric versions from
+different branch lineages.
+
+Research must specify behavior for:
+
+- a lazy fork that still physically reads its parent table branch;
+- the first branch-owned write after a fork;
+- compaction and index-only versions between base and side;
+- overwrite, restore, schema rewrite, and hard-drop operations;
+- tables without stable row IDs or version metadata;
+- a table dropped or introduced on only one side.
+
+Any shape not proven lineage-compatible takes the correctness fallback and
+records a typed reason. Physical metadata gaps never weaken the logical merge.
+
+## 6. Proposed planner shape
+
+```text
+ResolveMergeBase
+  -> DiscoverSideDelta(source)
+  -> DiscoverSideDelta(target)
+  -> JoinCandidateIds
+  -> FetchCandidateRows
+  -> ExistingTruthTableAndValidation
+  -> RFC022StageAndPublish
+```
+
+`DiscoverSideDelta` returns ordered, typed operations keyed by table and logical
+row ID. Candidate order is deterministic. Payload fetches use structured key
+lookups or SIP; they do not synthesize string `IN` filters or read embedding/blob
+columns for candidates whose disposition needs only identity.
+
+The merge base, physical entries, and candidate cuts are captured once before
+heavy work. After acquiring publish queues, RFC-022's OCC/read-set rule either
+confirms the cut or restarts discovery; it never publishes a delta classified
+against a moved side.
+
+## 7. Fallback and rollout safety
+
+`OrderedTableCursor` remains the universal fallback through the research and
+shadow phases. Fallback reasons are closed enum values, including:
+
+- `DeletionDeltaUnavailable`;
+- `VersionIndexUncovered`;
+- `CrossLineageUnsupported`;
+- `StableRowIdsUnavailable`;
+- `OverwriteOrRestoreInRange`;
+- `LineageMetadataInconsistent`.
+
+Before enabling the new path, shadow mode runs both classifiers on the same
+captured snapshots and compares ordered operation sets and merge outcomes. A
+mismatch fails the test or records a production diagnostic; it never silently
+chooses the new answer.
+
+## 8. Cost contract
+
+Use `helpers::cost` with fixed `delta = 1` and table sizes of at least 10k,
+100k, and 1M rows, both scalar-only and embedding-bearing. Measure candidate
+discovery separately from candidate payload fetch.
+
+The acceptance target for a true lineage path is:
+
+- insert and update discovery I/O is flat within named slack as rows grow;
+- delete discovery I/O is flat within the same discipline;
+- bytes read scale with candidate identity/version columns plus fetched delta
+  payload, not total row width;
+- peak RSS is bounded by candidate batch size, not table size;
+- graph-history depth and unrelated catalog width do not change the curve.
+
+Local tests gate scan/fragment terms. S3 tests gate object-store RPC and opener
+terms that local FS cannot expose. A benchmark without an asserted I/O slope is
+supporting evidence, not the acceptance gate.
+
+If only live-row indexing passes, this RFC may prototype or shadow an
+insert/update-only path, but it does not authorize production shipping. A
+partial fast path requires a separately accepted RFC that names its O(rows)
+delete fallback and operational value. This RFC and the default-classifier
+O(delta) claim remain blocked until R2 passes.
+
+## 9. Correctness gates
+
+- `merge_truth_table` remains byte-for-byte the semantic disposition oracle.
+- Property tests compare lineage and cursor classifiers over inserts, updates,
+  deletes, endpoint moves, cycles, and conflicting constraints.
+- Dedicated cases cover compaction, index maintenance, restore, overwrite,
+  lazy forks, and same-number/different-branch versions.
+- A one-row delete in a 1M-row table is the blocker test: the RFC cannot leave
+  research status until it is both correct and flat in table size.
+- Surface guards pin every Lance version-column, stable-row-ID, deletion-vector,
+  transaction, and index behavior used by the chosen implementation.
+- Fault tests move a side after discovery and prove OCC restarts instead of
+  publishing stale classification.
+
+## 10. Observability
+
+For every merge table report classifier path, fallback reason, source/target
+candidate counts, deleted-ID discovery mechanism, fragments and bytes read,
+payload columns fetched, discovery latency, and whether shadow results matched.
+
+Metrics distinguish logical delta size from physical discovery work. This is
+required to catch an apparently correct path silently regressing to O(rows).
+
+## 11. Research plan
+
+| Phase | Content | Exit criterion |
+|---|---|---|
+| R0 | Instrument current cursor and `changes` paths; build shadow comparison harness | semantic and I/O baselines |
+| R1 | Prove selective live-row discovery and branch-version mapping | insert/update flat-cost gate |
+| R2 | Prototype all deletion-delta options against pinned Lance | one option passes delete correctness + flat-cost gate |
+| R3 | Shadow new classifier across the full merge truth table and histories | zero mismatches; explicit fallback ledger |
+| R4 | Enable lineage path behind a scoped feature/config gate | production diagnostics within budgets |
+| R5 | Make lineage default and consider cursor retirement | all fallbacks dispositioned; no physical gap can break merge |
+
+The RFC remains **research / blocked** through R2. Choosing an OmniGraph commit
+delta format in §4.3 requires its own format amendment before R3.

docs/user/branching/index.md

@@ -33,8 +33,10 @@ conflict kinds are on the [merge](merge.md) page.
 
 ## L2 — Recovery audit trail
 
-Interrupted multi-table writes are recovered automatically the next time the graph is opened read-write. Recovery commits are recorded in the audit trail under the actor `omnigraph:recovery`, so you can find them with:
-
-```bash
-omnigraph commit list --filter actor=omnigraph:recovery
-```
+Interrupted multi-table writes are recovered automatically the next time the
+graph is opened read-write. Each completed recovery is recorded internally in
+`_graph_commit_recoveries.lance`. A roll-forward keeps the interrupted
+writer's original commit id and actor; rollback and legacy recovery commits use
+the reserved actor `omnigraph:recovery`. Consequently, `commit list` is not a
+complete recovery log, and the CLI does not currently expose a query for the
+internal recovery-audit table.

docs/user/branching/transactions.md

@@ -19,7 +19,7 @@ Two primitives, two scopes:
 | **One `.gq` query** (any number of statements inside) | The query itself — handled by the publisher's atomic manifest commit | Yes — all statements land together or none of them do | The publisher never publishes; target unchanged |
 | **Many queries that must succeed together** | Branches: `branch_create` → run N queries on the branch → `branch_merge` | Yes — the merge is a single atomic publish | Drop the branch (`branch_delete`); main is unaffected |
 
-Snapshot isolation is per-query — every read inside one query sees one consistent manifest version. Two concurrent queries on the same branch see independent snapshots; the publisher's CAS catches racing writes.
+Snapshot isolation is per-query — every read inside one query sees one consistent manifest version. Two concurrent queries on the same branch see independent snapshots. Mutation/load capture the branch head as coarse OCC authority, so a prepared plan is never silently reparented after another graph commit.
 
 ## Comparison with `BEGIN` / `COMMIT`
 
@@ -136,7 +136,9 @@ This is the workflow agentic loops are designed around: **branches are the unit
 | Scenario | What happens | Caller action |
 |---|---|---|
 | Single query fails mid-flight | Publisher never publishes; target unchanged | Read the error, decide whether to retry |
-| Concurrent writers race the same `(table, branch)` | Publisher CAS rejects the loser with a version-mismatch conflict | Refresh handle, retry the query |
+| Branch authority changes before physical effects | Retryable inserts/loads fully reprepare; strict writes return `read_set_conflict` | For a surfaced strict conflict, refresh and retry deliberately |
+| Authority changes after a physical effect | The write returns `recovery_required` and leaves its durable sidecar | Resolve recovery by read-write reopen/server restart before retrying |
+| An overlapping recovery intent remains unresolved before effects | The write returns `recovery_required` with that intent's operation id and does not advance a table | Resolve recovery by read-write reopen/server restart before retrying |
 | Branch with N successful mutations, then merge fails (three-way conflict) | Each individual mutation already committed on the branch; merge surfaces `MergeConflicts` | Inspect, decide whether to keep working on the branch, abandon it (`branch_delete`), or resolve and re-merge |
 | Process crashes mid-branch-workflow | Each completed mutation on the branch is durable | Re-open the graph, continue where you left off |
 

docs/user/cli/reference.md

@@ -21,7 +21,7 @@ Top-level command families and subcommands. Graph-targeting commands accept a po
 | `schema plan \| apply \| show (alias: get)` | migrations. `apply` refuses a cluster-managed graph (one whose storage is inside a cluster) and points at `cluster apply` — those graphs evolve through the cluster ledger, not a direct apply |
 | `lint` (alias: `check`) | offline / graph-backed query validation. Replaces `query lint` / `query check`, which are kept as deprecated argv-level shims that print a one-line warning and rewrite to `omnigraph lint` |
 | `cluster validate \| plan \| apply \| approve \| status \| refresh \| import \| force-unlock` | declarative cluster control plane. `validate` checks a local `cluster.yaml` folder and referenced schema/query/policy files; `plan` diffs it against local JSON state at `__cluster/state.json`, annotates dispositions, and embeds real schema-migration previews; `apply` converges the cluster — stored-query/policy catalog writes (content-addressed under `__cluster/resources/`), graph creates, schema updates (soft drops only; `--as` records the actor), and graph deletes behind a digest-bound approval from `cluster approve <resource> --as <actor>` (`apply`/`approve` default the actor from `~/.omnigraph/config.yaml`'s `operator.actor` when `--as` is omitted); what apply converges is what an `omnigraph-server --cluster <dir>` deployment serves on its next restart (`--cluster` is the server's only boot source — cluster-only); `status` reads the state ledger; `refresh`/`import` explicitly update local JSON state from read-only graph observations; `force-unlock <LOCK_ID>` manually removes a held local state lock by exact id |
-| `optimize` | non-destructive Lance compaction (skips tables with `Blob` columns or uncovered drift; `--json` reports `skipped`) |
+| `optimize` | non-destructive Lance compaction + index reconciliation (blob-bearing tables use the normal path; tables with uncovered drift are skipped and `--json` reports `skipped`) |
 | `repair [--confirm] [--force]` | preview or explicitly publish uncovered manifest/head drift. `--confirm` heals verified maintenance drift and exits non-zero if suspicious/unverifiable drift is refused; `--force --confirm` publishes suspicious/unverifiable drift after operator review |
 | `cleanup --keep N --older-than 7d --confirm` | destructive version GC (`--confirm` to execute; also needs `--yes` against a non-local `s3://` target — see *Write diagnostics & destructive confirmation*) |
 | `embed` | offline JSONL embedding pipeline |

docs/user/concepts/storage.md

@@ -7,7 +7,7 @@ Every node type and every edge type is its own Lance dataset:
 - **Columnar Arrow storage**: each property is a column; nullable per Arrow schema.
 - **Fragments**: data is partitioned into fragments; new writes create new fragments.
 - **Manifest versioning**: every commit produces a new dataset version; old versions remain readable.
-- **Stable row IDs**: stable row IDs are enabled on every Lance dataset OmniGraph creates — node and edge data tables, `__manifest`, the commit-graph datasets, and any future system tables. This is an architectural invariant: the flag is one-way at dataset create, so a future change that introduces a Lance dataset must preserve it. Consequences: `_row_created_at_version` and `_row_last_updated_at_version` are available on every dataset (load-bearing for change-feed validators); indices survive `omnigraph optimize`. Pre-0.4.x graphs created before this code path settled may have datasets without the flag and cannot be retrofitted in place — the supported path is dump-and-reload. The rewrite path used by `schema_apply` preserves the flag.
+- **Stable row IDs**: stable row IDs are enabled on every Lance dataset OmniGraph creates — node and edge data tables, `__manifest`, `_graph_commit_recoveries.lance`, and any future system tables. This is an architectural invariant: the flag is one-way at dataset create, so a future change that introduces a Lance dataset must preserve it. Consequences: `_row_created_at_version` and `_row_last_updated_at_version` are available on every dataset (load-bearing for change-feed validators); indices survive `omnigraph optimize`. Pre-0.4.x graphs created before this code path settled may have datasets without the flag and cannot be retrofitted in place — the supported path is dump-and-reload. The rewrite path used by `schema_apply` preserves the flag.
 - **Append / delete / `merge_insert`**: native Lance write modes.
 - **Per-dataset branches** (Lance native): copy-on-write at the dataset level.
 - **Object-store agnostic**: file://, s3://, gs://, az://, http (read-only via Lance) — OmniGraph wires file:// and s3://.
@@ -30,7 +30,7 @@ OmniGraph is **not** a single Lance dataset; it is a *graph* of datasets coordin
 - **Snapshot reconstruction**: latest visible `table_version` per `(table_key, table_branch)` minus tombstones — rows where `object_type = table_tombstone`, whose own `table_version` (acting as the tombstone version) is `>= the entry's table_version`.
 - **Atomic publish**: multi-dataset commits publish so that a single write to `__manifest` flips all the new sub-table versions visible at once.
 - **Row-level CAS on the merge-insert join key**: `object_id` carries an unenforced-primary-key annotation so Lance's bloom-filter conflict resolver rejects two concurrent commits that land the same `object_id` row. Without this annotation, Lance's transparent rebase would admit silent duplicates from racing publishers.
-- **Optimistic concurrency control on publish**: a publish asserts the manifest's current latest non-tombstoned version for each touched table is exactly what the caller observed; mismatches surface as an `ExpectedVersionMismatch` manifest conflict naming the table and the expected/actual versions. Concurrent advances surface as a conflict rather than being silently rebased through.
+- **Optimistic concurrency control on publish**: legacy writers assert the manifest's current latest non-tombstoned version for each touched table; a mismatch surfaces as `ExpectedVersionMismatch`. RFC-022-enrolled mutation/load attempts use a stronger, branch-wide contract: preparation captures the Lance-native branch identity, the exact `graph_head` (including absence), the accepted schema identity/catalog, and one base table snapshot. Under root-shared schema → branch → sorted-table gates, the engine revalidates that complete authority before any physical effect, then the publisher rechecks the exact native branch identity/head plus the touched-table versions. An insert-only mutation or Append/Merge load whose authority changed before effects discards and fully reprepares the bounded attempt; Update/Delete/Overwrite returns `ReadSetChanged`. Once any Lance effect is durable, any later failure leaves the recovery sidecar authoritative and returns `RecoveryRequired` instead of silently rebasing or replaying the prepared plan.
 
 ### Internal schema versioning
 
@@ -95,7 +95,7 @@ flowchart TB
 - **`__manifest/`** is a Lance dataset whose rows describe which sub-table version is published at which graph-branch. Reading a snapshot starts here.
 - **`nodes/`** and **`edges/`** are sibling directories holding one Lance dataset per declared type. Names are `fnv1a64-hex` of the type name to keep paths fixed-length and case-safe.
 - The graph commit DAG lives in **`__manifest`** as `graph_commit` / `graph_head` rows written in the publish CAS (RFC-013 Phase 7). The former `_graph_commits.lance` / `_graph_commit_actors.lance` lineage tables are retired — a graph this binary creates has neither.
-- **`_graph_commit_recoveries.lance`** — one row per crash-recovery action. Joined by `graph_commit_id` to the graph commit lineage (the `graph_commit` rows in `__manifest` since RFC-013 Phase 7); the linked commit carries `actor_id=omnigraph:recovery`. Operators correlate recoveries with the original mutations they rolled forward / back via this join.
+- **`_graph_commit_recoveries.lance`** — one internal row per completed crash-recovery action, including its exact per-table outcomes and the original actor. It joins by `graph_commit_id` to the graph commit lineage in `__manifest`. A v3 roll-forward keeps the interrupted writer's original actor; rollback and legacy recovery commits use `omnigraph:recovery`. The CLI does not currently expose this internal table.
 - **`__recovery/{ulid}.json`** — transient sidecar files written by a writer before it advances the underlying dataset, deleted once the matching manifest publish succeeds. A sidecar persisting after process exit means the writer crashed mid-commit; the next read-write open processes it. Steady-state directory is empty.
 - **`_refs/branches/{name}.json`** is graph-level branch metadata — pointers from a branch name to the manifest version it heads.
 - **Inside each Lance dataset** (orange): the standard Lance directory layout. `_versions/{n}.manifest` records every commit; `data/` holds the actual Arrow fragments; `_indices/{uuid}/` holds index segments with their own `fragment_bitmap` for partial coverage; `_refs/` holds Lance-native per-dataset branches and tags.

docs/user/mutations/index.md

@@ -21,6 +21,12 @@ properties in the assignment block (`insert WorksAt { person: $p, org: $o }`).
 
 `<value>` is a literal, `$param`, or `now()`.
 
+On a blob-bearing type, an update materializes and rewrites blob payloads only
+for the rows matched by its predicate, including blobs the update does not
+change. This keeps correctness independent of physical index state, but adds
+read/write I/O proportional to the matched blob bytes; use selective update
+predicates for large blobs.
+
 ## Atomicity
 
 A change query publishes **one commit** at the end of the query. Multiple
@@ -29,6 +35,19 @@ failure leaves the graph untouched. See [transactions](../branching/transactions
 for the per-query atomicity contract and [branches](../branching/index.md) for
 multi-query workflows.
 
+Concurrent changes use optimistic concurrency over the whole target branch.
+Insert/Merge/Append operations whose branch changed before physical effects are
+discarded and fully revalidated with a bounded internal retry. Strict
+Update/Delete/Overwrite operations instead return a structured conflict. This
+branch-wide token is deliberately conservative: a change to a different table
+can invalidate a prepared strict write because constraints may have read it.
+
+If the synchronous barrier finds an unresolved overlapping recovery intent, or
+if a conflict is discovered after a Lance table effect is durable, the request
+returns `recovery_required` with an operation id. Do not immediately retry that
+request; reopen the graph read-write (or restart the server) so the durable
+recovery intent is resolved first.
+
 ## Inserts/updates and deletes cannot mix in one query
 
 A single change query must be **either insert/update-only or delete-only**.

docs/user/operations/audit.md

@@ -31,13 +31,12 @@ List commits to see who made each change:
 omnigraph commit list graph.omni
 ```
 
-System-initiated writes use reserved actor ids — for example, automatic recovery
-of an interrupted write records `omnigraph:recovery`, so operator changes and
-machine repairs are distinguishable in the history:
-
-```bash
-omnigraph commit list --filter actor=omnigraph:recovery graph.omni
-```
+System-initiated writes use reserved actor ids. Rollback and legacy recovery
+commits use `omnigraph:recovery`, while a v3 roll-forward preserves the
+interrupted writer's original commit id and actor. Exact recovery actions and
+per-table outcomes are stored in the internal
+`_graph_commit_recoveries.lance` audit table; the CLI does not currently expose
+that table, so `commit list` alone is not a complete recovery log.
 
 ## What is tracked
 

docs/user/operations/errors.md

@@ -8,9 +8,11 @@
 - `Io(io::Error)`
 - `Manifest(ManifestError { kind: BadRequest|NotFound|Conflict|Internal, details: Option<ManifestConflictDetails>, … })`
   - `ManifestConflictDetails::ExpectedVersionMismatch { table_key, expected, actual }` — caller's `expected_table_versions` did not match the manifest's current latest non-tombstoned version (set by `OmniError::manifest_expected_version_mismatch`).
+  - `ManifestConflictDetails::ReadSetChanged { member, expected, actual }` — an RFC-022 prepared write's branch/head/table authority changed before physical effects. HTTP returns **409** with `read_set_conflict`. A retry must start from preparation; strict writes leave that choice to the caller.
   - `ManifestConflictDetails::RowLevelCasContention` — Lance row-level CAS rejected the publish because a concurrent writer landed the same `object_id`. Retried internally by the publisher; only surfaces if the retry budget exhausts.
   - **D₂ parse-time rejection**: a single mutation query that mixes inserts/updates with deletes errors out *before any I/O* with kind `BadRequest`. Message: `mutation '<name>' on the same query mixes inserts/updates and deletes; split into separate mutations: (1) inserts and updates, then (2) deletes`. See [query-language.md](../queries/index.md) for the rule.
 - `MergeConflicts(Vec<MergeConflict>)`
+- `RecoveryRequired { operation_id, reason }` — an overlapping durable recovery intent remains unresolved. Its physical effects may already have landed, or it may still be armed before the first effect. HTTP returns **503** with `recovery_required.operation_id`. Resolve the sidecar through a read-write reopen/server restart before retrying; this is intentionally not an ordinary OCC retry.
 
 Compiler-side `CompilerError` covers parse / catalog / type / storage / plan / execution / arrow / lance / IO / manifest / unique-constraint, each with structured spans (`SourceSpan { start, end }`) for ariadne-style diagnostics. The legacy `NanoError` name remains as a deprecated compatibility alias.
 

docs/user/operations/maintenance.md

@@ -9,13 +9,13 @@
 - **Also compacts the internal `__manifest` table** (RFC-013 step 2), which accumulates one fragment per commit — it now carries the graph lineage and actor rows inline (RFC-013 Phase 7: `graph_commit` / `graph_head` rows), so on the authenticated write path every commit's actor lands here too — and otherwise makes every write's metadata scan grow with history. (The `_graph_commits.lance` / `_graph_commit_actors.lance` tables are retired, so there is no separate lineage table to compact.) It takes a simpler path than data tables: `__manifest` is read at its latest version, so compaction just advances its version in place — **no manifest publish and no recovery sidecar**. (The sidecar-free property is not because it is one commit — `compact_files` can emit a `ReserveFragments` commit before the `Rewrite`, and the auto-cleanup strip below is a further commit — but because every one of those commits is content-preserving and the table is read at its latest version, so a crash at any point leaves it readable and content-identical and the next `optimize` re-plans.) It appears in the returned stats under `table_key` `"__manifest"`. It is **not yet covered by `cleanup`**, so its version chain still grows until the cleanup half lands (it requires a cleanup-resurrection safeguard first); run `optimize` on a cadence to keep per-write metadata scans flat.
 - **`optimize` is non-destructive by construction — it never garbage-collects versions, on any table (data or internal).** Compaction rewrites fragments and advances the version; old versions stay reachable until you run `cleanup`. This holds even for a graph created by an older binary that stored an on-by-default Lance `auto_cleanup` hook: `compact_files` / `optimize_indices` commit with the hook enabled and expose no skip override, so before compacting **any** table `optimize` strips its stale `lance.auto_cleanup.*` config first, so Lance's commit-time GC hook cannot fire and silently prune `__manifest`-pinned versions. (Graphs created by current binaries store no such config; the strip is the upgrade-path safety net.) The internal-table path additionally tolerates a concurrent live writer: it runs a **bounded** rebase-and-retry, so transient contention does not fail the operator's `optimize` or the live write — but sustained contention past the retry budget surfaces a loud conflict error rather than looping forever (bounded and observable, not a silent give-up). The data-table path holds the per-table write queue while it compacts, so it does not contend with mutations on that table in the first place.
 - **Reindex (index coverage maintenance).** A scalar/FTS/vector index only covers the fragments it was built over. Rows appended after the index was built (e.g. by `load --mode merge`, whose commit does not rebuild an already-existing index) are scanned unindexed, and compaction itself rewrites fragments out of an index's coverage. `optimize` runs Lance's incremental `optimize_indices` after compaction to fold those fragments back in (a delta merge, not a full retrain), restoring full coverage so equality/range/traversal predicates stay index-accelerated. This is why a table with **no compaction work but stale index coverage still commits** a new version under `optimize`. Run `optimize` on a cadence at least as frequent as your freshness window so recently-loaded rows do not linger in the unindexed flat-scan tail.
-- **Create declared-but-missing indexes (the index reconciler).** `@index`/`@key` declares intent; `schema apply` records it but builds nothing, and `load`/`mutate` defer a column that cannot be built yet (a `Vector` column with no trainable vectors). `optimize` materializes any such declared-but-unbuilt index over the compacted layout — so it is the convergence path for an `@index` added after data exists, or a vector index whose embeddings arrived via a later `embed`. A column still not buildable (no vectors yet) is reported on the table's stat as `pending_indexes` (visible in `--json`), not treated as a failure; the next `optimize` retries. So `optimize` is the single operator-facing index reconciler: it compacts, restores coverage, **and** builds declared-but-missing indexes.
+- **Create declared-but-missing indexes (the index reconciler).** `@index`/`@key` declares intent; `schema apply`, `load`, and `mutate` build no physical indexes inline. They record or publish only their exact logical/data effects and leave all index materialization to `ensure_indices`/`optimize`. `optimize` materializes every buildable declared-but-missing index over the compacted layout — so it is the convergence path for an `@index` added after data exists, or a vector index whose embeddings arrived via a later `embed`. A column still not buildable (no vectors yet) is reported on the table's stat as `pending_indexes` (visible in `--json`), not treated as a failure; the next `optimize` retries. So `optimize` is the single operator-facing index reconciler: it compacts, restores coverage, **and** builds declared-but-missing indexes.
 - Each table's compact→reindex→publish serializes with concurrent mutations on the same table. A crash mid-operation is recovered automatically on the next open (both compaction and reindex are content-preserving, so roll-forward is always safe).
 - **Requires a recovered graph.** `optimize` refuses (errors) when a pending crash-recovery operation is present — operating on an unrecovered graph could publish a partial write that recovery would roll back. Reopen the graph to run recovery, then re-run `optimize`.
 - **Uncovered drift is skipped, not interpreted.** If a table's underlying version is ahead of the version recorded in `__manifest` and no crash-recovery record covers that movement, `optimize` reports `skipped: DriftNeedsRepair` with the manifest/head versions and leaves the table untouched. Run `omnigraph repair` to classify and explicitly publish that drift.
 - Bounded by `OMNIGRAPH_MAINTENANCE_CONCURRENCY` (default 8).
 - Returns per-table stats: `table_key, fragments_removed, fragments_added, committed, skipped, manifest_version, lance_head_version, pending_indexes` (the last lists any declared `@index` column the reconciler could not build this run, with the reason — e.g. a vector column with no trainable vectors yet).
-- **Blob tables are skipped.** A table that declares any `Blob` property is not compacted: it is reported with `skipped: BlobColumnsUnsupportedByLance` (and logged) instead of compacted, and the rest of the sweep proceeds normally. **Reads and writes are unaffected** — only compaction is. Consequence: fragment count and deleted-row space on blob tables are not reclaimed; query results are never affected. A skipped blob table is also **not reindexed** in the same sweep (the skip happens before the reindex step), so its index coverage on appended rows is not refreshed by `optimize` today.
+- **Blob tables use the normal compaction and reindex path.** Lance 8.0.0+ supports blob-v2 compaction, so OmniGraph no longer has a blob-specific skip or capability gate. Fragment reclamation and index-coverage repair therefore apply to blob-bearing tables like every other table.
 
 ## `repair` — explicit
 
@@ -39,7 +39,12 @@
 - CLI guards with `--confirm`; without it, prints a preview line.
 - **Non-local consent.** Against a non-local target (an `s3://` store/cluster), `cleanup` additionally requires `--yes` on top of `--confirm`: a TTY is prompted, and a non-interactive run (no TTY, or `--json`) refuses rather than destroying. A local (`file://`) target needs only `--confirm`. The same `--yes` gate applies to overwrite `load` and `branch delete`; every maintenance run echoes its resolved target to stderr (suppress with `--quiet`).
 - **Recovery floor:** `--keep < 3` may garbage-collect versions that crash recovery needs as a rollback target. Default `--keep 10` is safe.
-- **Orphaned-branch reconciliation:** before the version GC, cleanup reclaims any per-table or commit-graph branch absent from the manifest branch list. These orphans arise when a `branch_delete` flips the manifest authority but a downstream best-effort reclaim does not complete (see [branches-commits.md](../branching/index.md)). The reconciler is idempotent (it no-ops once nothing is orphaned), runs regardless of the `keep_versions` / `older_than` values (those gate version GC only), and never reclaims `main` or system-branch forks. Reclaimed forks are logged.
+- **Requires clean recovery state.** If any durable recovery intent is pending,
+  cleanup refuses before orphan reconciliation or version GC. Reopen the graph
+  read-write (or restart the server) to resolve recovery, then rerun cleanup;
+  deleting transaction/version history while an intent is pending would make
+  exact effect ownership unverifiable.
+- **Orphaned-branch reconciliation:** before the version GC, cleanup reclaims any per-table Lance branch absent from the manifest branch list. These orphans arise when a `branch_delete` flips the manifest authority but a downstream best-effort reclaim does not complete (see [branches-commits.md](../branching/index.md)). The reconciler is idempotent (it no-ops once nothing is orphaned), runs regardless of the `keep_versions` / `older_than` values (those gate version GC only), and never reclaims `main` or system-branch forks. Reclaimed forks are logged. Graph lineage has no separate branch dataset: it lives in `__manifest`.
 
 ## Tombstones
 

docs/user/operations/server.md

@@ -162,25 +162,51 @@ Only `/export` streams (`application/x-ndjson`, MPSC channel + `Body::from_strea
 
 ## Error model
 
-Uniform `ErrorOutput { error, code?, merge_conflicts[], manifest_conflict? }` with `code ∈ unauthorized | forbidden | bad_request | not_found | conflict | too_many_requests | internal`. Merge conflicts attach structured `MergeConflictOutput { table_key, row_id?, kind, message }`.
+Uniform
+`ErrorOutput { error, code?, merge_conflicts[], manifest_conflict?, read_set_conflict?, recovery_required? }`
+with
+`code ∈ unauthorized | forbidden | bad_request | not_found | method_not_allowed | conflict | too_many_requests | internal`.
+Merge conflicts attach structured
+`MergeConflictOutput { table_key, row_id?, kind, message }`.
 
-`manifest_conflict` is set on **concurrent-write rejections** (HTTP 409): the
-caller's pre-write view of one table's manifest version was stale.
-`ManifestConflictOutput { table_key, expected, actual }` tells the client
-which table to refresh and retry. This is the conflict shape produced by
-concurrent `/mutate` (or its `/change` alias), `/load` (or its deprecated
-`/ingest` alias) calls landing the same `(table, branch)` race.
+`manifest_conflict` is set on legacy per-table manifest-version rejections
+(HTTP 409). `ManifestConflictOutput { table_key, expected, actual }` tells the
+client which table was stale. Mutation and load use the unified coarse-OCC
+adapter described next; other writers retain this older conflict shape until
+they are enrolled.
 
-HTTP status codes used: 200, 400, 401, 403, 404, 409, 429, 500.
+`read_set_conflict` is set when a prepared write is rejected before any table
+effect because its branch authority changed. The HTTP status is 409 and
+`ReadSetConflictOutput { member, expected, actual }` identifies the stale
+authority member. The engine already performs a bounded full-attempt retry for
+mutation inserts and load `append`/`merge`. Strict mutation updates/deletes and
+load `overwrite` return the 409 to the caller instead of being replayed.
+
+`recovery_required` is set when an overlapping durable recovery intent remains
+unresolved; its table effects may or may not have started. The HTTP status is 503 and
+`RecoveryRequiredOutput { operation_id }` names the durable recovery intent.
+The optional `code` field is omitted for this response: adding a new value to
+the closed error-code enum would break older clients, while the optional
+structured field is additive and rolling-safe.
+Do not blindly resubmit the write: let a read-write open or the recovery sweep
+resolve that operation first, then retry from a fresh snapshot.
+
+HTTP status codes used: 200, 400, 401, 403, 404, 405, 409, 429, 500, 503.
 
 ## Per-actor admission control
 
-Disjoint
-`(table, branch)` writes from different actors now run concurrently,
-guarded only by the engine's per-(table, branch) write queue. To keep
-one heavy actor from exhausting shared capacity (Lance I/O, manifest
-churn, network), the server gates mutating handlers through per-process
-admission limits configured from environment variables:
+RFC-022-enrolled mutation/load preparation runs outside the effect gates, so
+parsing, validation, and reclaimable fragment staging can overlap across branches.
+Readers acquire none of these gates. Before the first durable effect, however, an
+attempt acquires the exclusive root schema gate, then its branch-effect gate and
+sorted table queues, and holds all of them through manifest publication. The root
+schema gate means enrolled effect windows on one graph currently serialize
+in-process even across different branches; the branch gate preserves one atomic
+graph-head validation authority, while table queues protect each concrete Lance
+effect and legacy writer. These are process-local ordering gates, not a
+cross-process lock. To keep one heavy actor from exhausting shared capacity
+(Lance I/O, manifest churn, network), the server gates mutating handlers through
+per-process admission limits configured from environment variables:
 
 | Env var | Default | Purpose |
 |---|---|---|

docs/user/reference/constants.md

@@ -4,14 +4,13 @@
 |---|---|---|
 | `MANIFEST_DIR` | `__manifest` | manifest layout |
 | Commit graph dirs (retired) | `_graph_commits.lance` / `_graph_commit_actors.lance` | retired in Phase B; lineage lives in `__manifest` (`graph_commit` / `graph_head` rows) since RFC-013 Phase 7. A graph this binary creates has neither. |
-| Recovery audit dir | `_graph_commit_recoveries.lance` | one row per crash-recovery action (`omnigraph commit list --filter actor=omnigraph:recovery`) |
+| Recovery audit dir | `_graph_commit_recoveries.lance` | internal exact record of completed crash-recovery actions; no public CLI query yet |
 | Run branch prefix (legacy, removed) | `__run__` | pre-v0.4.0 Run state machine; no longer a reserved name. A graph still carrying `__run__*` branches is sub-v4 and refused on open (rebuild via export/import). |
 | Schema apply lock | `__schema_apply_lock__` | schema apply |
 | Manifest publisher retry budget | `PUBLISHER_RETRY_BUDGET = 5` | manifest publish |
 | Internal manifest schema version | `INTERNAL_MANIFEST_SCHEMA_VERSION = 4` | manifest migrations (v4 = graph lineage in `__manifest`, RFC-013 Phase 7) |
 | Merge stage batch | `MERGE_STAGE_BATCH_ROWS = 8192` | merge execution |
 | Maintenance concurrency | `OMNIGRAPH_MAINTENANCE_CONCURRENCY=8` | optimize/cleanup |
-| Lance blob compaction support | `LANCE_SUPPORTS_BLOB_COMPACTION = false` | optimize |
 | Graph index cache size | `8` (LRU) | runtime cache |
 | Expand indexed-path frontier ceiling | `OMNIGRAPH_EXPAND_INDEXED_MAX_FRONTIER=1024` | traversal |
 | Expand indexed-path hop ceiling | `OMNIGRAPH_EXPAND_INDEXED_MAX_HOPS=6` | traversal |

docs/user/search/indexes.md

@@ -27,7 +27,7 @@ list/`Blob` columns → none.
 
 ## L2 — OmniGraph orchestration
 
-- **`@index`/`@key` declares intent; the physical index is derived state.** A migration records the declaration in the catalog/IR and never fails on it — `schema apply` builds **no** indexes (adding an `@index` to an existing column is a pure metadata change that touches no table data). `load`/`mutate` build declared indexes inline as part of the write, but a column that can't be built yet (a `Vector` column with no trainable vectors — IVF k-means needs ≥1 vector, e.g. rows loaded before `embed` runs) is left **pending**, not fatal. Reads stay correct meanwhile: a missing/partial index degrades to a scan (vector search to brute-force). A later `ensure_indices`/`optimize` materializes the pending index once it is buildable. This mirrors how LanceDB builds indexes asynchronously and serves unindexed rows by brute-force.
+- **`@index`/`@key` declares intent; the physical index is derived state.** A migration records the declaration in the catalog/IR and never fails on it — `schema apply` builds **no** indexes. Mutation/load likewise publish only their exact data effects; they do not widen the recovery plan with index commits. Reads stay correct while an index is missing or partially covered by falling back to scans (vector search to brute-force). A later `ensure_indices`/`optimize` materializes every buildable declaration; an untrainable Vector column remains pending rather than failing the logical write.
 - `ensure_indices()` / `ensure_indices_on(branch)` — idempotent build of BTREE + inverted + vector indexes for the current head; safe to re-run; returns the columns it had to defer as pending. `optimize` runs it after compaction, so the maintenance cron is the convergence path for deferred indexes.
 - Indexes are built on the *branch head* (not on a snapshot), so reads always see the current index state.
 - **Lazy branch forking for indexes**: a branch that hasn't mutated a sub-table doesn't need its own index — the main lineage's index is reused until the first write triggers a copy-on-write fork.