Harden RFC-022 pre-arm recovery ownership (#346)

docs/dev/invariants.md

@@ -84,6 +84,11 @@ converge the physical state.
 
 5. **Recovery is part of the commit protocol.** Writers that can advance Lance
    HEAD before manifest publish must write `__recovery/{ulid}.json` sidecars.
+   Under their final schema → branch → table gates, they must first prove every
+   existing physical target still equals its manifest pin; a new sidecar must
+   never claim an older writer's effect or uncovered drift. First-touch refs use
+   sidecar-before-ref ordering. A non-noop SchemaApply writes a sidecar even
+   with zero table pins because schema-contract staging is durable state.
    `Omnigraph::open` in read-write mode runs the all-or-nothing sweep; the
    write entry points (`load_as`, `mutate_as`, `apply_schema_as`,
    `branch_merge_as`) and `refresh` run roll-forward-only recovery in-process,

docs/dev/writes.md

@@ -54,8 +54,11 @@ Mutation and load use a closed prepare → effect → publish attempt:
    identity instead: Lance branch-local files cannot be staged until its target
    ref exists;
 3. acquire the schema gate, branch gate, then sorted table queues; re-check for
-   a relevant sidecar armed since step 1, then revalidate the token. Any
-   unresolved relevant intent returns typed `RecoveryRequired` before effects;
+   a relevant sidecar armed since step 1, then revalidate the token and require
+   every existing physical target's live Lance HEAD to equal its manifest pin.
+   Any unresolved relevant intent returns typed `RecoveryRequired`; uncovered
+   HEAD drift points to `omnigraph repair`. Both fail before this attempt arms
+   recovery;
 4. on a pre-effect mismatch, discard the complete attempt. Append/Insert/Merge
    reprepare with a bounded retry; strict Update/Delete/Overwrite return typed
    `ReadSetChanged`;
@@ -89,9 +92,13 @@ RFC-022 adapter contract:
    the immutable base/source/target snapshots outside table gates;
 3. acquire the conservative all-catalog source/target table envelope, re-list
    recovery intent, revalidate the complete target token, and revalidate the
-   source incarnation. A target change returns typed `ReadSetChanged` before
-   effects. A later source-head advance is allowed: the contract is "merge the
-   captured source commit," never "substitute whatever source is latest";
+   source incarnation. Before arming, every existing target ref that will receive
+   a physical effect must also have live Lance HEAD equal to its captured target
+   manifest pin; the verified handle is carried into the effect instead of being
+   reopened. First-touch refs remain absent until after the sidecar. A target
+   change returns typed `ReadSetChanged` before effects. A later source-head
+   advance is allowed: the contract is "merge the captured source commit," never
+   "substitute whatever source is latest";
 4. pre-mint the merge lineage and each table's ordered Lance data-transaction
    chain, then arm a schema-v4 BranchMerge sidecar before the first HEAD advance
    or first-touch table ref. Logical data steps commit with those exact
@@ -341,6 +348,14 @@ are left at `Lance HEAD = manifest_pinned + 1`.
 `branch_merge_on_current_target`, `ensure_indices_for_branch`,
 `optimize_all_tables`):
 
+Before Phase A, under the writer's final schema → branch → table gates, existing
+physical targets must still match their manifest pins. Ahead drift is never folded
+or claimed by manufacturing a new sidecar; it is attributed to an existing recovery
+intent or refused with explicit `omnigraph repair` guidance. First-touch targets use
+the separate sidecar-before-ref protocol. SchemaApply also verifies that AddType and
+RenameType target dataset paths are absent, so recovery cannot register an orphan or
+foreign dataset as if this apply created it.
+
 1. **Phase A**: writer writes a sidecar JSON to
    `__recovery/{ulid}.json` BEFORE its first independently durable physical
    effect (including a first-touch Lance branch ref) or HEAD-advancing commit
@@ -355,7 +370,18 @@ are left at `Lance HEAD = manifest_pinned + 1`.
    carries its pre-minted transaction identity. Branch merge uses schema v4:
    it distinguishes multi-commit HEAD effects from ref-only forks, records each
    multi-commit effect's ordered exact transaction chain, and records the
-   complete intended manifest delta, including pointer-only slots.
+   complete intended manifest delta, including pointer-only slots. SchemaApply
+   uses schema v5 as a narrow bridge: every non-noop apply records its target
+   schema hash even when the table-pin set is empty, then durably marks the
+   sidecar manifest-published immediately after Phase C. This distinguishes a
+   pre-staging rollback from completed schema promotion whose Phase-D delete
+   failed; table effects remain on the legacy loose classifier until the full
+   exact SchemaApply adapter lands. EnsureIndices uses schema v6 as a second
+   narrow bridge: table effects are still loosely classified, but the sidecar
+   pre-mints its rollback commit id. Before the first restore, recovery durably
+   binds the original per-table rollback audit plan, so an interruption after
+   rollback publish retries as that same `RolledBack` outcome instead of being
+   mistaken for a stale roll-forward.
 2. **Phase B**: writer's per-table `commit_staged` loop runs.
    - **Phase-B confirmation:** a schema-v4 `BranchMerge` writer
      advances each table's HEAD by *several* exact commits (append → upsert →
@@ -440,8 +466,9 @@ recovery sweep in `crates/omnigraph/src/db/manifest/recovery.rs`:
   `recovery_for_actor` (the original sidecar's actor), `operation_id`, and
   exact per-table outcomes. Schema-v3 Mutation/Load and schema-v4 BranchMerge
   roll-forward publish the interrupted writer's fixed lineage intent,
-  including its original actor; rollback and legacy recovery commits use
-  `actor_id = "omnigraph:recovery"`. Ordinary
+  including its original actor. Schema-v6 EnsureIndices rollback reuses its
+  pre-minted recovery commit id and durable audit plan. Other rollback and
+  legacy recovery commits use `actor_id = "omnigraph:recovery"`. Ordinary
   commit history is therefore not a complete recovery enumeration, and the
   CLI currently has no public query for the recovery-audit table.
 - Sidecar deleted as the final step.
@@ -459,10 +486,15 @@ restart and without an explicit refresh. The heal lists `__recovery/`
 (one `list_dir`; empty in the steady state) and, per sidecar, acquires
 schema → branch → sorted-table gates that overlap the writer's guarded
 sidecar lifetime. RFC-022 mutation/load writers hold the complete order. Branch
-merge now holds schema plus source/target branch authority for its whole attempt
-and then the all-catalog source/target table envelope; other legacy adapters
-serialize through their existing table or schema gate until their own adapter
-slices land. The
+merge holds schema plus source/target branch authority for its whole attempt and
+then the all-catalog source/target table envelope. SchemaApply holds schema → main
+branch → every live table; EnsureIndices holds schema → target branch → every table
+in its durable work plan. SchemaApply's schema-v5 confirmation closes its zero-pin
+outcome ambiguity while retaining loose table classification. EnsureIndices'
+schema-v6 payload also retains loose table-effect classification, but gives rollback
+a fixed commit id and a durable pre-restore audit plan so recovery re-entry cannot
+flip the outcome. Both close the pre-arm ownership boundary while their full exact
+adapters remain future work. Optimize retains its legacy adapter. The
 manager is shared by every
 `Omnigraph` handle for one canonical local root identity (relative, absolute,
 and symlink aliases converge; object-store/custom schemes stay opaque), so this
@@ -471,9 +503,9 @@ in-flight sidecar forward from under it (a sidecar whose queues can be
 acquired belongs to a writer that finished or died; an existence
 re-check after the wait skips the finished case). Lock order is
 schema → branch → sorted tables → coordinator, matching the writer effect path.
-Enrolled mutation/load attempts and branch merge perform one additional
+Mutation/load, branch merge, SchemaApply, and EnsureIndices perform one additional
 `list_dir` after acquiring their authority gates; that final check closes the
-pre-gate recovery TOCTOU without moving mutation/load validation or staged-file
+pre-gate recovery TOCTOU without moving validation or reclaimable staged-file
 construction under the gate.
 Pinned by the four
 `tests/failpoints.rs::*_after_finalize_publisher_failure_heals_without_reopen`

docs/rfcs/rfc-022-unified-write-path.md

@@ -322,6 +322,15 @@ cross-process lock and must not substitute for a target read set.
 With all gates held, the coordinator loads fresh authority state and compares every
 member of `ReadSet`.
 
+Revalidation also proves the physical pre-state that the new recovery intent would
+claim. For every existing table ref in the effect envelope, live Lance `HEAD` must
+equal the prepared manifest pin. `HEAD < pin` is an internal invariant failure;
+`HEAD > pin` belongs either to an already-durable recovery intent or to uncovered
+drift that requires explicit repair. The attempt must reject that state **before**
+writing its own sidecar; a new sidecar may never retroactively claim a pre-existing
+physical effect. A first-touch ref is the deliberate exception: it does not exist at
+this point and follows the sidecar-before-ref protocol in §4.3 instead.
+
 - If all members match, the attempt may arm recovery.
 - If any member differs, no physical effect may run. The attempt releases its gates,
   discards staged state, and restarts from Prepare.
@@ -471,6 +480,15 @@ able to enumerate every adapter and every entry point that invokes it.
 - Include accepted schema identity and every affected table in `ReadSet`.
 - Cover schema staging-file promotion, data-table schema/field-metadata commits,
   registrations, tombstones, and final schema identity with one recovery intent.
+- A non-noop schema change still needs that recovery intent when it has no table
+  effects: schema-contract staging and promotion are independently durable state,
+  so an empty table-pin set means “metadata-only SchemaApply,” not “effect-free.”
+- Until SchemaApply receives the full exact adapter, its schema-v5 bridge records
+  the target schema identity at arm time and durably confirms Phase C before final
+  schema-file promotion. Recovery may classify an empty-pin Phase-D residue as
+  completed only when both that confirmation and the live target identity match;
+  numeric manifest movement alone is not an outcome discriminator because a
+  rollback recovery commit also advances the manifest.
 - Write the sidecar before the first table HEAD advance, including unenforced-PK
   metadata backfill or other inline metadata commits.
 - A branch-wide or graph-wide migration must enumerate every physical manifest/data
@@ -482,6 +500,14 @@ able to enumerate every adapter and every entry point that invokes it.
 - It records the exact achieved version rather than assuming one version of movement.
 - If the new data-table version is selected through `__manifest`, publishing that
   pointer is a graph-visible commit and uses this protocol.
+- Until EnsureIndices receives its full exact effect adapter, its schema-v6 bridge
+  retains legacy loose table classification but pre-mints a stable rollback commit
+  id. Recovery persists the observed rollback audit plan before the first restore;
+  a retry after rollback publish therefore finalizes the same `RolledBack` outcome
+  instead of inferring direction from aligned numeric pins.
+- Schema-v6 does not prove ownership of an index commit and does not add a fixed
+  forward outcome. Exact transaction chains, fixed original lineage, and exact
+  first-touch ref identity remain requirements of the full adapter.
 - Logical operations never fail because a derived index is absent or behind.
 - Physical-only internal-table maintenance remains the exception in Section 8.
 

docs/user/search/indexes.md

@@ -28,7 +28,7 @@ list/`Blob` columns → none.
 ## L2 — OmniGraph orchestration
 
 - **`@index`/`@key` declares intent; the physical index is derived state.** A migration records the declaration in the catalog/IR and never fails on it — `schema apply` builds **no** indexes. Mutation/load likewise publish only their exact data effects; they do not widen the recovery plan with index commits. Reads stay correct while an index is missing or partially covered by falling back to scans (vector search to brute-force). A later `ensure_indices`/`optimize` materializes every buildable declaration; an untrainable Vector column remains pending rather than failing the logical write.
-- `ensure_indices()` / `ensure_indices_on(branch)` — idempotent build of BTREE + inverted + vector indexes for the current head; safe to re-run; returns the columns it had to defer as pending. `optimize` runs it after compaction, so the maintenance cron is the convergence path for deferred indexes.
+- `ensure_indices()` / `ensure_indices_on(branch)` — idempotent build of BTREE + inverted + vector indexes for the current manifest head; safe to re-run; returns the columns it had to defer as pending. Existing Lance HEAD drift is refused with `omnigraph repair` guidance rather than silently adopted. On a lazy child branch, an ancestor-owned table stays inherited when there is no index work; real index work first verifies the target ref is absent, then creates it only after recovery intent is durable. `optimize` runs the reconciler after compaction, so the maintenance cron is the convergence path for deferred indexes.
 - Indexes are built on the *branch head* (not on a snapshot), so reads always see the current index state.
 - **Lazy branch forking for indexes**: a branch that hasn't mutated a sub-table doesn't need its own index — the main lineage's index is reused until the first write triggers a copy-on-write fork.
 - Vector index parameters (metric, nlist, nprobe, etc.) are not exposed in the schema; they default at the Lance layer and are picked up automatically when an index is asked for on a Vector column.