Harden branch recovery and cleanup retention (#344)

* Harden branch recovery and cleanup retention

* Fail closed on blocked partial rollback

docs/dev/invariants.md

@@ -159,7 +159,8 @@ converge the physical state.
 | Multi-table commit | Manifest CAS plus recovery sidecars; not a single Lance primitive | [writes.md](writes.md), [architecture.md](architecture.md) |
 | Constructive mutations | In-memory `MutationStaging`, one end-of-query table commit per touched table, then one manifest publish | [writes.md](writes.md), [execution.md](execution.md) |
 | Deletes | Staged like inserts/updates (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, MR-A) — no inline HEAD advance; mixed insert/update/delete in one query rejected by D2 as a deliberate boundary (constructive XOR destructive per query; compose via separate mutations or a branch) | [query-language.md](../user/queries/index.md), [writes.md](writes.md) |
-| Branch delete | Manifest is the single authority, flipped atomically first; per-table forks are derived state, reclaimed best-effort (`force_delete_branch`) with the `cleanup` reconciler as the guaranteed backstop. Under schema/target/all-table gates, a target-scoped unresolved sidecar may be made unreachable by deletion and is then audit-discarded by recovery; graph-global SchemaApply still blocks. Reusing a name whose fork reclaim failed before `cleanup` surfaces an actionable error | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) |
+| Branch create/delete | `__manifest` `BranchContents` is the single logical authority. Lance create is physically two-phase, so OmniGraph prevalidates names, enforces path-prefix-disjoint live graph names, reclaims an absent-ref clone-only tree, and uses a bounded completion classifier; delete removes authority before tree cleanup, so an absent ref is success and derived tree reclaim may converge later. Neither control emits graph lineage. Per-table forks are derived state, reclaimed best-effort with `cleanup` as backstop. Under schema/target/all-table gates, a target-scoped unresolved sidecar may be made unreachable by deletion and is then audit-discarded by recovery; graph-global SchemaApply still blocks | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) |
+| Cleanup retention | Explicit cleanup derives exact `keep` cutoffs from Lance's available version list, caps each main-table GC cutoff at the oldest exact main version inherited by any live lazy graph branch, and refuses uncovered main HEAD drift. Lance protects native per-table branch refs itself. The graph-wide live-reference preflight fails closed before the first table GC, after which individual table failures remain fault-isolated | [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) |
 | Schema validation | Type checks, required fields, defaults, edge endpoint checks, and edge cardinality are enforced on write paths | [schema-language.md](../user/schema/index.md), [execution.md](execution.md) |
 | Unique constraints | Value/enum, uniqueness, edge-RI, and cardinality route through ONE unified, catalog-derived evaluator (`crate::validate`) on ALL THREE write surfaces — branch-merge, mutation, and bulk load: Δ-scoped (checks the delta, not the whole graph) and structured-filter-backed (committed probes use a BTREE when reconciled and remain correct by scanning while it is pending), reusing the leaf checks (`loader::validate_value_constraints`/`validate_enum_constraints`/`composite_unique_key`) so the surfaces cannot drift. This closed the prior merge bug (merge validated `@range`/`@check` but not enum) AND the **cross-version uniqueness gap** on the mutation and load paths (a duplicate of a committed `@unique` value is now rejected; the merge path always enforced it). The committed view is the merge target snapshot (merge), the write's pinned `txn.base` (mutation), or the pinned pre-load base (load — `Overwrite` validates the batch as the whole new image, committed view empty); `@card` refreshes the manifest-visible graph-branch snapshot on the mutation path only (the #298 stale-handle fix), then follows each entry's actual inherited/owned Lance ref. `@key` is id-backed, so it is checked intra-delta only (a committed holder of a key value is always the same row — an upsert), skipping a wasted O(Δ) probe per keyed row; `@unique` (non-key) groups do the committed lookup. | [schema-language.md](../user/schema/index.md) |
 | Storage trait | `TableStorage` (via `db.storage()`) is staged-only; the sole inline-commit residual (`create_vector_index`) is split onto a separate sealed `InlineCommitResidual` trait reached via `db.storage_inline_residual()` (MR-854), so §1 holds by construction; capability/stat surfaces are roadmap | [writes.md](writes.md), [architecture.md](architecture.md) |
@@ -168,12 +169,23 @@ converge the physical state.
 | Auth | Bearer token hashing and server-side actor resolution are implemented at the HTTP boundary | [server.md](../user/operations/server.md), [policy.md](../user/operations/policy.md) |
 | Tests | Tempdir-backed Lance tests are the current substrate; the storage adapter has an in-memory backend for adapter-level contract tests, but Lance datasets bypass it | [testing.md](testing.md) |
 
-The branch-delete reconciler is authority-derived: it reclaims orphaned forks
-today and degrades to a no-op if Lance ships an atomic multi-dataset branch
-operation, so the design composes with that future rather than blocking it. This
-is the same shape as invariant 7 (indexes are derived state); prefer it over a
-recovery-sidecar-style approach for any new multi-dataset metadata operation,
-since the sidecar would be scaffolding to remove once the substrate closes the gap.
+The branch-control reconcilers are authority-derived: same-name create reclaims
+an absent-ref clone-only manifest tree, and delete/cleanup reclaim orphaned
+per-table forks. They degrade to no-ops if Lance closes the corresponding
+physical gaps, so the design composes with that future rather than blocking it.
+This is the same shape as invariant 7 (indexes are derived state); prefer it
+over a recovery-sidecar-style approach for metadata work whose logical target
+is fully derivable from one existing authority. A graph-visible table effect is
+different and still requires durable ownership before it can advance HEAD.
+For legacy path-prefix overlaps, an ancestor first-touch tree is not proven
+unreachable while a live child remains. Full recovery may leave the sidecar
+intact and allow the graph to open for leaf-first child deletion **only when the
+intent owns no physical table effect**. If the same sidecar owns any durable
+effect, cleanup is part of an all-or-nothing rollback: read-write open fails
+closed until the child is removed through an existing handle or an offline
+Lance-level branch tool, then the next Full sweep reclaims the untouched fork and
+compensates the owned effect. Returning a writable handle with that rollback
+incomplete would let a legacy writer prepare from unresolved physical state.
 
 ## Known Gaps
 
@@ -252,19 +264,24 @@ them explicit.
   files and commits. Both `reclaim_orphaned_fork_and_refork` and
   `reconcile_orphaned_branches` consult pending sidecars before destruction; a
   foreign claim is conflict/indeterminate, never permission to delete. Full
-  recovery accepts sidecar-before-ref crashes and deletes an unpublished fork
-  only when it is still exactly at the inherited version and no other sidecar
-  claims `(table_path, target ref)`. A no-effect sidecar with a competitor
+  recovery accepts sidecar-before-ref crashes. When `BranchContents` is absent,
+  the already-armed intent plus the single-writer-process gate makes a same-name
+  clone-only tree unreachable, so recovery force-reclaims it without pretending
+  it can inspect an unlisted version. When a ref exists, recovery deletes the
+  unpublished fork only after proving it is still exactly at the inherited
+  version and no other sidecar claims `(table_path, target ref)`. A no-effect
+  sidecar with a competitor
   discards only itself; the last survivor either cleans the untouched ref or
   recovers its owned effect. Partial rollback performs no-effect cleanup before
   publishing its fixed outcome. This closes the cross-handle live-ref
   deletion bug and keeps cleanup as the backstop for truly unclaimed refs.
-  The remaining multi-process gap is narrower but real: Lance exposes
-  `force_delete_branch`, not compare-and-delete by `BranchIdentifier`, so a
-  foreign process can create intent/ref between the final list/check and the
-  delete. The documented single-writer-process support boundary remains until
-  Lance provides a conditional ref primitive (or OmniGraph adds a distributed
-  fence); process-local queues are not credited as that primitive.
+  The remaining multi-process gap is narrower but real: Lance exposes neither
+  conditional native graph-branch create/delete nor compare-and-delete by
+  `BranchIdentifier` for per-table refs, so a foreign process can mutate a ref
+  between the final list/check and the operation. The documented
+  single-writer-process support boundary remains until Lance provides a
+  conditional ref primitive (or OmniGraph adds a distributed fence);
+  process-local queues are not credited as that primitive.
 - **Local `write_text_if_match` is not a cross-process CAS:** object-store
   backends use a true conditional put (ETag If-Match; the in-memory test
   backend too), but upstream `object_store` leaves `PutMode::Update`

docs/dev/lance.md

@@ -198,6 +198,20 @@ findings:
   the substrate enforcing what omnigraph's entry-owner resolution always did.
   Production opens by owner-resolved location (unaffected); the lazy-fork
   namespace test pins both the error and the owner-branch open.
+- **Native branch control remains two-phase at beta.15.**
+  `Dataset::create_branch` commits the shallow-cloned `tree/{branch}` dataset
+  before validating/writing authoritative `BranchContents`; its random
+  `BranchIdentifier` element is minted only in that second phase and cannot be
+  pre-minted through the public API. OmniGraph therefore validates first and
+  classifies an ambiguous result from exact parent metadata rather than
+  inventing an identifier. Delete removes `BranchContents` before tree cleanup
+  and exposes no compare-and-delete primitive. Slash-separated names share
+  nested physical directories; `force_delete_branch` deliberately leaves an
+  ancestor tree while a live path-child exists, so live graph names are
+  path-prefix-disjoint. The public format page does not currently spell out the
+  identifier field, so the pinned Rust shape remains load-bearing. Guard 9 in
+  `lance_surface_guards.rs` pins clone-only raw-create failure plus force reclaim;
+  `src/branch_control.rs` pins delete classification and JSON identity fencing.
 - **Native DirectoryNamespace churn** (#7222 removed
   `table_version_storage_enabled` + the `__manifest` version-storage
   experiment; #7176/#7191/#7234 rewrote manifest handling): the decoupling
@@ -287,7 +301,7 @@ Migration from Lance 4.0.0 → 6.0.1 landed in this cycle (DataFusion 52 → 53,
 - **Lance #6658 closed** (2026-05-14) but `DeleteBuilder::execute_uncommitted` did **not** ship in v6.0.1 — binary search across the release stream shows it first appears in `v7.0.0-beta.10` (the closing commits landed on main but didn't backport to the 6.x line). Tracked as MR-A: migrate `delete_where` to staged, retire the parse-time D2 mutation rule, extend recovery sidecar coverage. **Gated on the Lance v7.x bump**, not this PR. v7.0.0-rc.1 dropped 2026-05-21.
 - **Lance #6666 still open** (`build_index_metadata_from_segments` public): vector-index two-phase blocked; inline `create_vector_index` residual retained.
 - **Lance #6877 still open** (`MergeInsertBuilder` dup-rowid): PR #109's `SourceDedupeBehavior::FirstSeen` + `check_batch_unique_by_keys` precondition stay load-bearing.
-- **`Dataset::force_delete_branch`** (`branches().delete(name, force=true)`, dataset.rs:524) tolerates a missing branch-*contents* ref (vs plain `delete_branch`'s `RefNotFound`), but on the local store still errors `NotFound` if the branch `tree/` directory is fully absent (`remove_dir_all`'s NotFound is not caught for Lance's native error variant, refs.rs:526-549). Both variants still refuse a branch with referencing descendants (`RefConflict`). `TableStore::force_delete_branch` wraps this to be fully idempotent (tolerates already-absent). The single-authority branch-delete redesign uses it for orphan reclamation (eager best-effort reclaim + cleanup reconciler). Pinned by `lance_surface_guards.rs::force_delete_branch_semantics`. Branch delete is "flip the ref atomically, then `remove_dir_all(tree/{branch})`"; branch-exclusive data lives under `tree/{branch}/` so a drop reclaims it immediately without touching `main`.
+- **`Dataset::force_delete_branch`** (`branches().delete(name, force=true)`, dataset.rs:524) tolerates a missing branch-*contents* ref (vs plain `delete_branch`'s `RefNotFound`), but on the local store still errors `NotFound` if the branch `tree/` directory is fully absent (`remove_dir_all`'s NotFound is not caught for Lance's native error variant, refs.rs:526-549). Both delete variants still refuse a branch with referencing descendants (`RefConflict`). The current OmniGraph disposition for these still-present substrate behaviors is recorded in the beta.15 audit above.
 - **Lance blob-v2 `compact_files` bug** (no public issue found as of 2026-06): `compact_files` disables binary-copy for blob datasets and forces `BlobHandling::AllBinary` on the read side; the v2.1+ structural decoder then mis-counts column infos for the blob-v2 struct and fails with `Invalid user input: there were more fields in the schema than provided column indices / infos` (`lance-encoding/src/decoder.rs::ColumnInfoIter::expect_next`). This fails even a pristine uniform-V2_2 multi-fragment blob table; vector/list/scalar/ragged columns and mixed file versions all compact fine. Reads/queries use descriptor handling (`BlobHandling::default()`) and are unaffected. `optimize` skips blob-bearing tables behind `LANCE_SUPPORTS_BLOB_COMPACTION = false` (`db/omnigraph/optimize.rs`), reporting `SkipReason::BlobColumnsUnsupportedByLance`. Pinned by `lance_surface_guards.rs::compact_files_still_fails_on_blob_columns`, which turns red when the bug is fixed → flip the gate, remove the skip branch + the `maintenance.rs::optimize_skips_blob_table_and_reports_skip` skip assertions.
 
 Surface guards added: `crates/omnigraph/tests/lance_surface_guards.rs` (10 named guards; 5 runtime + 5 compile-only; plus the index-coverage work's `_compile_optimize_indices_signature` and `optimize_indices_extends_fragment_coverage`). Future Lance bumps re-run this file first as the smoke check. Two additional guards from the original plan deferred to follow-up (`manifest_cas_returns_row_level_contention_variant` needs full publisher-race harness; `table_version_metadata_byte_compatible_with_v4` needs `pub(crate)` reach extension).

docs/dev/rfc-022-027-architecture-review.md

@@ -37,17 +37,19 @@ The central architecture remains the right one:
   separate irreversible decisions and should keep separate evidence and format
   gates.
 
-The current blockers are boundary problems, not a reason to replace that core.
-They concern crash classification, actual fencing, foreign-branch authority,
-and capability activation.
+The remaining open blockers are boundary problems, not a reason to replace that
+core. They concern actual cross-process fencing, capability activation, format
+rollout, and public RFC lifecycle. The native branch crash classifier,
+foreign-source merge semantics, and shipped lazy-branch cleanup exposure found
+by this review are now dispositioned below.
 
 > 💬 **Second-pass verification (2026-07-11):** I independently re-verified this
 > ledger's two sharpest new substrate claims against the pinned checkout before
 > commenting: BLOCKER-01's two-phase branch create is confirmed in Lance's own
 > doc comment, and BLOCKER-04's lazy-branch exposure is confirmed
-> architecturally — and appears to be a live bug in the shipped `cleanup`, not
-> only an RFC gap (see the comment there). Overall judgment: concur with this
-> ledger; three of its findings supersede corrections I applied to the RFCs on
+> architecturally. Both findings produced shipped fixes on 2026-07-11 and their
+> durable dispositions are recorded below. Overall judgment: concur with this
+> ledger; three other findings supersede corrections applied to the RFCs on
 > 2026-07-11 (noted inline at BLOCKER-07, BLOCKER-11, and tightening 5).
 
 The latest revisions improved several important points and those changes should
@@ -98,7 +100,9 @@ sub-contract of an unaccepted RFC as a separately accepted decision.
 
 **Affected:** [RFC-022 §7](../rfcs/rfc-022-unified-write-path.md#7-native-graph-branch-ref-control-protocol)
 
-RFC-022 currently models branch create/delete as one native ref mutation whose
+**Status:** Closed in specification and implementation on 2026-07-11.
+
+At review time RFC-022 modeled branch create/delete as one native ref mutation whose
 completion is the visibility point. The pinned Lance implementation is more
 specific:
 
@@ -143,6 +147,21 @@ and between delete authority removal and directory cleanup.
 > upstream ask) addressed the *conditional-put* gap but not this crash
 > classification — the two dispositions compose; both are needed.
 
+> ✅ **Disposition (2026-07-11):** RFC-022 §7 now makes `BranchContents` the sole
+> logical authority, defines the bounded create/delete classifier, explicitly
+> rejects a graph-branch sidecar/lineage entry, and retains the
+> single-writer-process boundary. The implementation prevalidates names,
+> requires live graph names to be physical-path-prefix-disjoint, reclaims an
+> absent-ref clone before one bounded retry, accepts only a current-attempt
+> matching lost acknowledgement, and fences delete by exact identifier. Full
+> first-touch recovery also force-reclaims clone-only table residue under its
+> already-durable writer sidecar. Coverage includes flat and named-source
+> clone-only recovery, invalid-name-before-clone, lost acknowledgements,
+> absent-ref/tree-present delete, same-identifier error preservation,
+> recreated-identifier refusal, path-prefix collision, legacy no-effect
+> leaf-first delete, mixed-effect rollback fail-closed behavior, and no
+> manifest-version/lineage movement.
+
 ### BLOCKER-02 — create-if-absent ownership is not a fencing lease
 
 **Affected:** RFC-023 migration claim, RFC-024 migration claim, and
@@ -193,6 +212,9 @@ that takeover is refused until external fencing/death proof is established.
 [RFC-023 §6](../rfcs/rfc-023-key-conflict-fencing.md#6-retry-and-validation-semantics),
 and [RFC-026 §6](../rfcs/rfc-026-memwal-streaming-ingest.md#6-fold-protocol)
 
+**Status:** The RFC-022 mutation/load portion is closed in the shipped adapter
+as of 2026-07-11. RFC-023 and RFC-026 retain their own open dispositions.
+
 A retryable concurrent inserted-row-filter conflict is detected when a table
 transaction attempts to commit. (`WhenMatched::Fail` may instead report a
 pre-existing match during merge execution.) In a multi-table graph write or
@@ -217,10 +239,24 @@ attempt around that sidecar. For non-strict upsert, the barrier must resolve the
 sidecar before the bounded automatic whole-operation retry. Add a failpoint/race
 in which table N conflicts after table 1 has committed.
 
+> ✅ **RFC-022 disposition (2026-07-11):** mutation/load take the conservative
+> safe branch of this requirement. A pre-effect authority mismatch discards and
+> fully reprepares the attempt. Once the exact-effect sidecar is durable and the
+> commit phase begins, every Lance commit failure returns `RecoveryRequired` and
+> leaves that sidecar intact, even when Full recovery later proves that no table
+> effect landed. The adapter performs zero transparent Lance commit retries and
+> never prepares a new semantic attempt around unresolved ownership; the next
+> synchronous barrier classifies and resolves it first. Finalizing a proven-empty
+> intent and automatically retrying would improve ergonomics, but is not required
+> for safety. RFC-023's fenced key-conflict contract and RFC-026's MemWAL fold
+> protocol still need their adapter-specific resolutions.
+
 ### BLOCKER-04 — live graph branches need physical GC protection
 
 **Affected:** [RFC-025 §6](../rfcs/rfc-025-checkpoint-retention.md#6-cleanup-protocol-and-pruned-through-boundary)
 
+**Status:** Closed in the shipped baseline and RFC-025 on 2026-07-11.
+
 The cleanup root set includes live graph branches, but RFC-025 creates Lance
 tags only for named checkpoints. This is insufficient for lazy graph branches.
 A graph branch may store, in its `__manifest`, a foreign reference to an old
@@ -253,6 +289,18 @@ runs, and the lazy branch still opens and reads the exact pinned state.
 > mechanism but are history-trimming under an operator-confirmed policy; a
 > live branch's working state is not history.)
 
+> ✅ **Disposition (2026-07-11):** shipped cleanup now resolves main plus every
+> live non-system graph branch under schema → all-branch → all-table gates,
+> verifies every exact inherited-main version opens, and caps each dataset's
+> cutoff at the oldest such pin before the first table GC. It also derives
+> `keep=N` from Lance's actual ordered version list and refuses uncovered main
+> HEAD drift. Unclassifiable roots abort graph-wide; only later per-table GC
+> failures are fault-isolated. Regression coverage pins count- and time-based
+> retention, the oldest of multiple live pins, exact keep counts, graph-wide
+> fail-closed ordering, and uncovered drift. RFC-025 §6 now composes checkpoint
+> tags with this required lazy-branch floor rather than treating tags as a
+> replacement.
+
 ### BLOCKER-05 — durable-head OCC must compare the full head token
 
 **Affected:** [RFC-024 §5](../rfcs/rfc-024-durable-table-heads.md#5-atomic-publisher-contract)
@@ -382,6 +430,11 @@ retaining the public lifecycle status `Proposed`.
 
 ### BLOCKER-10 — do not put foreign-branch facts in an atomic `ReadSet`
 
+**Affected:** [RFC-022 §6.2](../rfcs/rfc-022-unified-write-path.md#62-branch-merge)
+
+**Status:** Closed in specification on 2026-07-11; full merge-adapter conversion
+remains rollout work rather than an architecture ambiguity.
+
 RFC-022 requires every `ReadSet` member to be arbitrated atomically by the
 publish CAS. A CAS on reserved main cannot arbitrate a row on a named source
 branch; a merge-target CAS cannot arbitrate a source-branch row.
@@ -400,6 +453,13 @@ Use the right category for each fact:
 
 Keep target-branch values that must remain stable in the atomic `ReadSet`.
 
+> ✅ **Disposition (2026-07-11):** RFC-022 §6.2 defines the exact captured source
+> commit/snapshot as an immutable effect precondition, not a member of the
+> target publisher's atomic `ReadSet`. A later source-head advance is harmless
+> under captured-source semantics; only a future latest-at-target-publish
+> contract would require a source fence through target CAS. The current bridge
+> remains conservatively stricter before effects while its full adapter lands.
+
 ### BLOCKER-11 — RFC-022 and no-heads MemWAL need coarse OCC
 
 **Affected:** [RFC-022 §10](../rfcs/rfc-022-unified-write-path.md#10-rollout-and-compatibility)
@@ -525,7 +585,9 @@ RFC set is merged:
 The review does not require all RFCs to land together. A safe order is:
 
 1. accept RFC-022 after branch-control recovery, coarse `graph_head` OCC, and
-   foreign-branch fact classification are explicit;
+   foreign-branch fact classification are explicit (all three are now
+   dispositioned; the public lifecycle decision in BLOCKER-09 still governs
+   what “accepted” means);
 2. accept and land the stable identity RFC/capability;
 3. accept RFC-023 once partial-effect retry and its format/fleet barrier are
    complete;
@@ -540,10 +602,8 @@ The review does not require all RFCs to land together. A safe order is:
 This ordering preserves the split's main benefit: a blocked performance
 optimization or research result does not hold correctness work hostage.
 
-> 💬 **Concur with the order; two sequencing notes (2026-07-11):**
-> BLOCKER-04's escalation means step 5 has a prerequisite outside this list —
-> the live lazy-branch cleanup exposure needs its red regression test and fix
-> in the shipped `cleanup` first, so RFC-025 builds on a correct baseline. And
-> step 3's TIGHTENING-01 demotion hinges on the partial-update filter check
+> 💬 **Order update (2026-07-11):** BLOCKER-04's shipped-cleanup prerequisite is
+> now complete and RFC-025 §6 incorporates its floor. Step 3's TIGHTENING-01
+> demotion still hinges on the partial-update filter check
 > noted there; if that check lands `None`, upstream symmetry returns to the
 > activation gate and RFC-023's timeline moves accordingly.

docs/dev/testing.md

@@ -19,14 +19,14 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
 | File | Covers |
 |---|---|
 | `end_to_end.rs` | Full init → load → query/mutate flow |
-| `branching.rs` | Branch create / list / delete, lazy fork |
+| `branching.rs` | Branch create/list/delete and lazy fork; native-control hardening includes main and named-source clone-only create recovery, invalid-name-before-clone, live path-prefix namespace rejection, legacy prefix-collision leaf-first delete, and delete/recreate first-write safety. The lower classifier truth cells (absent-ref/tree-present delete, same-identifier native refusal, recreated-identifier typed conflict with JSON details) live in `src/branch_control.rs` unit tests |
 | `merge_truth_table.rs` | Merge-pair truth table (MR-786): all 9×9 `(left_op, right_op)` cells from `{noop, addNode, removeNode, addEdge, removeEdge, setProperty, dropProperty, addLabel, removeLabel}`. Adding a new op to `OpVariant` forces a compile error in `build_case` until the new row + column are dispositioned. 36 executable cells run through real `branch_merge` with a structured oracle (`MergeOutcome` / `MergeConflictKind` + graph-state assert); 45 cells involving `dropProperty`/`addLabel`/`removeLabel` are recorded as `Unsupported` until the mutation grammar grows. |
 | `merge_fast_forward.rs` | Fast-forward branch-merge cost + correctness: an append-only adopted-source merge routes *new* rows through `stage_append` instead of one whole-delta `stage_merge_insert` (the full-outer join that exhausted the DataFusion memory pool on embedding-bearing tables). The regression gate is structural — it asserts WHICH staged-write primitive the merge invokes via the task-local write probes (`omnigraph::instrumentation`), not a brittle size threshold; also: merge yields source state, defers vector indexes to the reconciler, streams blob columns |
 | `writes.rs` | Direct-publish writes: cancellation, RFC-022 non-strict full-attempt reprepare from fresh branch authority, strict stale-write conflicts, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery); the lance#7444 row-id-overlap regression (`filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent` — merge-load → same-key merge-load → delete → keyed point lookup, green only under the vendored lance-table patch — plus its append-only control) |
 | `staged_writes.rs` | TableStore staged-write primitives (`stage_append`, `stage_merge_insert`, `commit_staged`, `scan_with_staged`, `count_rows_with_staged`) — primitive-level only; engine code uses the in-memory `MutationStaging` accumulator instead |
 | `forbidden_apis.rs` | Defense-in-depth source-walk guard: engine code (`exec/`, `db/omnigraph/`, `loader/`, `changes/`) must not reach around the sealed storage trait to Lance inline-commit APIs, nor open datasets directly (`Dataset::open` / `DatasetBuilder::from_uri`/`from_namespace`) — reads route through `Snapshot::open` and the held-handle cache; `// forbidden-api-allow: <reason>` sentinel exempts reviewed lines |
 | `failpoint_names_guard.rs` | Source-walk guard (same defense-in-depth shape as `forbidden_apis.rs`): every failpoint call site (`maybe_fail`, `ScopedFailPoint::new`/`with_callback`, `Rendezvous::park_first`) must reference a `failpoints::names` const, never a bare string literal — a typo'd literal compiles and silently never fires |
-| `lance_surface_guards.rs` | Pins the Lance API surfaces omnigraph depends on (named runtime + compile-only guards; see [lance.md](lance.md)) — the first smoke check on any Lance version bump; e.g. `compact_files_succeeds_on_blob_columns` pins blob-v2 compaction working (a red means a Lance regression) |
+| `lance_surface_guards.rs` | Pins the Lance API surfaces omnigraph depends on (named runtime + compile-only guards; see [lance.md](lance.md)) — the first smoke check on any Lance version bump; e.g. `compact_files_succeeds_on_blob_columns` pins blob-v2 compaction working, while Guard 9 proves an unlisted clone-only branch blocks raw create and `force_delete_branch` reclaims it, but also pins Lance's intentional no-tree-cleanup behavior when a slash-name path-child is live (a red means a Lance regression) |
 | `warm_read_cost.rs` | Cost-budget tests for the warm read path (query-latency work), measured at the object-store boundary with Lance `IOTracker` (the LanceDB IO-counted pattern): a warm same-branch read does 0 manifest opens, 1 version probe, validates the schema once (Fix 1 / finding A / Fix 2 at commit-history depth); stale same-branch reads perform exactly 2 probes and refresh manifest-only; recreated non-main branches with the same Lance version refresh by incarnation; recreated branch-owned table handles are distinguished by table e_tag or refresh-time cache clearing; recreated traversal topology is protected by per-edge-table e_tag in the graph-index cache key or refresh-time cache clearing; a warm *repeat* read does 0 table opens via the held-handle cache and a write re-opens only the changed table at its new version/e_tag (Fix 3/6A). Also the CSR topology-build cost guards: `fresh_branch_traversal_reuses_main_graph_index` (A1 — a lazy-fork branch reuses main's cached CSR index, 0 rebuilds via `graph_build_count`) and `single_edge_query_builds_only_referenced_edge` (A2 — a one-edge query builds only that edge via `graph_edges_built`); both force CSR via the scoped `with_traversal_mode` seam, so they need no `#[serial]`. See "Cost-budget tests" below. |
 | `write_cost.rs` | Cost-budget tests for the WRITE path (RFC-013), the latency twin of `warm_read_cost.rs` on the **shared `helpers::cost` harness** (`measure`/`IoCounts`/`assert_flat`/`local_graph`). Runs on **local FS**; gates the **internal-table** term (`__manifest` scans flat in commit-history depth, lineage rows included — `internal_table_scans_are_flat_in_history`, now **green every-PR** since RFC-013 step 2 brought the internal tables into `optimize`; the test compacts at each depth before measuring) plus green every-PR guards (single-insert `data_writes` bounded, a per-write read-op ceiling that fails the moment a round-trip is added, and a `measure_with_staged` fitness assert that a keyed insert routes through `stage_merge_insert` once with no `stage_append`/vector-index build). The **data-table opener** term is S3-only — see `write_cost_s3.rs` and the backend-split note in "Cost-budget tests" below |
 | `write_cost_s3.rs` | Bucket-gated (skips without `OMNIGRAPH_S3_TEST_BUCKET`) twin of `write_cost.rs` on the same `helpers::cost` harness: gates the **data-table opener** term (per-write latest-version resolution flat across commit depth on a real object store — per-version GETs are invisible on local FS). A cost gate, not a correctness test — run on demand, not in the every-merge `rustfs_integration` job (see the backend-split note in "Cost-budget tests" below) |
@@ -51,8 +51,8 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
 | `validators.rs` | Schema constraint enforcement (enum, range, unique, cardinality) across JSONL load, mutation insert/update. ALL THREE write surfaces — mutation, bulk load, AND merge — route through the unified `crate::validate` evaluator (Δ-scoped, index-backed, reusing these leaf checks). Cross-version-uniqueness closure: `cross_version_unique_rejected_on_mutation_insert` + `reinsert_existing_key_is_upsert_not_unique_violation` (mutation path); `cross_version_unique_rejected_on_append_load` + `merge_load_reupsert_existing_key_is_not_unique_violation` (load path). Per-table `Overwrite`: `overwrite_load_validates_ri_against_new_image` (an edges-only overwrite still resolves RI against retained committed nodes) + `append_load_rejects_orphan_edge`. The evaluator's own unit tests live in `src/validate.rs` (`#[cfg(test)]`); its merge-conflict equivalence is pinned by `merge_truth_table.rs` (OrphanEdge) + `branching.rs` (Unique/Cardinality merge tests). Intra-batch duplicate-`@key` rejection on every load mode is pinned by `consistency.rs::loader_rejects_intra_batch_duplicate_keys`; the mutation-coalesce counterpart (insert+update / chained updates of one id are NOT a self-collision) by `writes.rs`. Non-String `@unique` columns probe committed state with a TYPED literal (not a stringified key): `cross_version_unique_rejected_on_date_column` + `noncolliding_write_to_date_unique_column_succeeds` (a `Date @unique` collision is a proper `@unique` violation, and a distinct value does not raise a Date32-vs-Utf8 coercion error). Cardinality is keyed by edge id, last-wins (matching commit's `dedupe_merge_batches_by_id`): `merge_load_edge_src_move_rechecks_vacated_src_cardinality` (a Merge-load moving an edge recounts the vacated src for `@card` min) + `merge_load_duplicate_edge_id_counts_once_per_card` (a dup edge id under two srcs in one batch counts once, no spurious max violation). Direct deletes capture the ids they remove (from the delete op's own scan) into the change-set's `deleted_ids`, so a delete emptying a src is validated: `mutation_delete_edge_below_card_min_rejected` (a `delete Edge` dropping a src below `@card` min is rejected, not silently committed). |
 | `merge_cost.rs` | Cost-budget tests for branch MERGE on the shared `helpers::cost` harness: `merge_validation_is_delta_scoped` (a 1-row-delta merge opens ≤3 data tables — Δ-scoped, not the whole catalog; was ~6 pre-#5) and `merge_manifest_cost_grows_with_history` (the cross-branch `__manifest` open amplification still grows with commit depth — a separate, not-yet-addressed term — while validation `data_open_count` stays flat) |
 | `policy_engine_chassis.rs` | Engine-layer Cedar enforcement (MR-722): allow + deny through every `_as` writer via the SDK directly — no HTTP — proving embedded and CLI callers hit the same gate as the server, with action × scope shapes matching `authorize_request` |
-| `maintenance.rs` | `optimize` (compaction), `repair` (explicit uncovered-drift publish), and `cleanup` (version GC): empty/idempotent/no-op edges, policy validation, head preservation; `optimize` publishes its own compaction (`optimize_publishes_compaction_to_manifest_so_schema_apply_succeeds`), skips pre-existing uncovered drift (`optimize_skips_preexisting_manifest_head_drift`), refuses to run while a `__recovery` sidecar is pending (`optimize_defers_when_recovery_sidecar_is_pending`), and compacts blob-v2 tables through the normal path (`optimize_compacts_blob_table_alongside_plain_table`); `repair` previews/heals verified maintenance drift, refuses raw semantic drift without `--force`, and forced repair publishes only by explicit operator choice; the index reconciler (iss-848): `index_build_tolerates_null_vector_rows` (logical load succeeds without inline index work; an untrainable Vector column then defers during reconciliation) and `optimize_materializes_index_declared_but_unbuilt` (optimize creates a declared-but-deferred index) |
-| `failpoints.rs` | Failure-injection coverage (gated on `failpoints` feature). RFC-022 includes deterministic post-stage/pre-effect races for mutation/load uniqueness and strict disjoint-head changes, plus the cross-handle post-effect `RecoveryRequired` → read-write-open rollback cell. Control/recovery race cells include `first_touch_post_create_open_error_keeps_recovery_ownership`, `branch_delete_orphans_sidecar_armed_after_initial_barrier`, `branch_merge_fences_target_delete_recreate_aba`, `branch_merge_fences_concurrent_sync_on_same_handle`, `branch_merge_rejects_fresh_target_manifest_change_before_effects`, `branch_merge_rechecks_late_sidecar_after_table_gates`, `cleanup_rechecks_sidecars_under_gc_gates`, and `full_recovery_rereads_sidecar_body_after_discovery`. The suite also includes the five per-writer effect → manifest-CAS recovery tests, write-entry in-process heal contract, storage-fault matrix, S3 recovery twin, and convergence-idempotent roll-forward regression. |
+| `maintenance.rs` | `optimize` (compaction), `repair` (explicit uncovered-drift publish), and `cleanup` (version GC): empty/idempotent/no-op edges, policy validation, head preservation; cleanup pins exact keep-count behavior (including keep larger than history), count/time retention of a live lazy branch, the oldest of multiple lazy pins, graph-wide fail-closed ordering on an unopenable pin, and refusal of uncovered main HEAD drift before any GC; `optimize` publishes its own compaction (`optimize_publishes_compaction_to_manifest_so_schema_apply_succeeds`), skips pre-existing uncovered drift (`optimize_skips_preexisting_manifest_head_drift`), refuses to run while a `__recovery` sidecar is pending (`optimize_defers_when_recovery_sidecar_is_pending`), and compacts blob-v2 tables through the normal path (`optimize_compacts_blob_table_alongside_plain_table`); `repair` previews/heals verified maintenance drift, refuses raw semantic drift without `--force`, and forced repair publishes only by explicit operator choice; the index reconciler (iss-848): `index_build_tolerates_null_vector_rows` (logical load succeeds without inline index work; an untrainable Vector column then defers during reconciliation) and `optimize_materializes_index_declared_but_unbuilt` (optimize creates a declared-but-deferred index) |
+| `failpoints.rs` | Failure-injection coverage (gated on `failpoints` feature). RFC-022 includes deterministic post-stage/pre-effect races for mutation/load uniqueness and strict disjoint-head changes, plus the cross-handle post-effect `RecoveryRequired` → read-write-open rollback cell. Native controls are pinned by `native_branch_controls_reclassify_lost_acknowledgements` (matching create and absent-ref delete, with no version/lineage movement); `armed_first_touch_recovery_accepts_missing_target_ref` additionally forges and reclaims the clone-only/no-`BranchContents` table state. Legacy path overlap has both sides pinned: `armed_first_touch_recovery_defers_legacy_path_overlap_until_leaf_delete` permits open only for a proven no-effect intent, while `partial_first_touch_recovery_fails_closed_on_legacy_path_overlap` leaves one exact multi-table effect and verifies open fails closed until offline leaf cleanup lets rollback converge. Other control/recovery race cells include `first_touch_post_create_open_error_keeps_recovery_ownership`, `branch_delete_orphans_sidecar_armed_after_initial_barrier`, `branch_merge_fences_target_delete_recreate_aba`, `branch_merge_fences_concurrent_sync_on_same_handle`, `branch_merge_rejects_fresh_target_manifest_change_before_effects`, `branch_merge_rechecks_late_sidecar_after_table_gates`, `cleanup_rechecks_sidecars_under_gc_gates`, and `full_recovery_rereads_sidecar_body_after_discovery`. The suite also includes the five per-writer effect → manifest-CAS recovery tests, write-entry in-process heal contract, storage-fault matrix, S3 recovery twin, and convergence-idempotent roll-forward regression. |
 | `failpoint_names_guard.rs` | Source-walk guard: every `maybe_fail(...)` call site (engine + cluster) must reference the compile-checked `failpoints::names` consts, never a bare string literal — a typo'd literal compiles but silently never fires; same defense-in-depth shape as `forbidden_apis.rs` |
 | `recovery.rs` | Open-time recovery sweep — legacy sidecars plus schema-v3 Mutation/Load exact transaction identity, fixed logical/rollback outcome IDs, branch-token comparison, fresh under-gate sidecar reread/reparse, all-or-nothing roll-forward/rollback/refusal, audit row in `_graph_commit_recoveries.lance`, and `OpenMode::ReadOnly` skip path |
 | `composite_flow.rs` | Compositional/narrative end-to-end stories — multi-step flows that compose mechanics covered by other test files. Catches integration regressions where individual operations all pass their unit tests but their composition breaks (sequential merges, post-merge main writes, time-travel through merge DAG, reopen consistency over multi-merge histories, post-optimize and post-cleanup strict writes). |

docs/dev/writes.md

@@ -30,13 +30,16 @@ authority.
   never created without ownership: the schema-v3 sidecar is durable first and
   names that `(table_path, target ref)`. Reclaim and `cleanup` treat any
   matching pending sidecar as a hard stop. Quiesced full recovery accepts both
-  crash shapes — sidecar durable with no ref yet, or an exact untouched ref at
-  the inherited version — and removes the latter before deleting the empty
-  intent. If several pending intents claim one ref, a no-effect intent discards
-  only itself while any competitor remains; the last no-effect survivor cleans
-  an untouched ref, or the effect-owning survivor recovers normally. `cleanup`
-  remains the backstop for genuinely unclaimed legacy/stale refs; see the
-  fork-reclaim note in [invariants.md](invariants.md).
+  logical crash shapes — sidecar durable with no ref yet, or an exact untouched
+  ref at the inherited version. Because Lance creates a branch dataset before
+  writing its authoritative `BranchContents`, the first shape may still contain
+  a clone-only tree; recovery force-reclaims that absent-ref tree idempotently.
+  It removes an exact untouched ref before deleting the empty intent. If several
+  pending intents claim one ref, a no-effect intent discards only itself while
+  any competitor remains; the last no-effect survivor cleans an untouched ref,
+  or the effect-owning survivor recovers normally. `cleanup` remains the
+  backstop for genuinely unclaimed legacy/stale refs; see the fork-reclaim note
+  in [invariants.md](invariants.md).
 
 ## Mutation/load coarse OCC (RFC-022 first adapter)
 
@@ -87,6 +90,14 @@ source and target, re-lists sidecars, and compares fresh source/target manifest
 versions with the captured snapshots. A stale warm handle catalog or coordinator
 snapshot is never accepted as that revalidation.
 
+The source snapshot is a captured merge input, not authority that the target
+manifest CAS can arbitrate. The current process-local source gate is a stronger
+same-process fence around that capture, including delete/recreate ABA, but the
+semantic contract is still "merge the captured source commit." A later source
+advance does not invalidate an otherwise prepared target publish. Claiming
+"latest source at target publish" would instead require a cross-process source
+fence held through the target CAS.
+
 That fence prevents a same-process target delete/recreate from reusing the branch
 name underneath a merge plan. The race test deliberately recreates a target with
 the same name and numeric Lance version but a different `BranchIdentifier`, so
@@ -109,6 +120,47 @@ records the orphan-discard recovery audit and deletes the sidecar. A
 is specific to removing the authority that made the intent reachable; create,
 merge, mutation, and load still reject relevant unresolved ownership.
 
+### Native graph-branch control recovery
+
+Graph branch create/delete do not use the graph-visible table-effect sidecar or
+emit graph lineage. Their sole logical authority is Lance `BranchContents` for
+the `__manifest` dataset, and Lance mutates that authority in two physical
+phases:
+
+- create shallow-clones `tree/{branch}` before writing `BranchContents`;
+- delete removes `BranchContents` before reclaiming that tree.
+
+Under the schema/branch/table control gates, create validates the name before
+the clone and rejects a live graph name that is a physical path ancestor or
+descendant of another live name. It then force-reclaims any absent-ref same-name
+tree and performs at most two native attempts. An ambiguous result is accepted
+only when fresh metadata has
+the captured parent branch/version/incarnation plus exactly one new identifier
+element and the target opens. Foreign or broken authoritative refs are never
+deleted. Delete captures the exact target identifier; after an ambiguous error,
+an absent ref is logical success, the same identifier preserves the original
+error, and a different identifier is a typed delete/recreate conflict. Derived
+tree cleanup is retried best-effort.
+
+There is deliberately no branch-control sidecar: within the supported
+single-writer-process topology, an absent ref makes a same-name tree unreachable
+garbage; the path-prefix-disjoint namespace is what makes Lance's recursive
+force cleanup exact. Same-name create is therefore the targeted reconciler.
+First-touch data-table
+forks remain sidecar-owned because they are physical effects of a graph-visible
+mutation/load. Lance does not expose conditional ref create/delete, so this
+classifier is not advertised as a cross-process branch-control fence.
+
+Legacy prefix-overlap recovery is the one first-touch case that does not prove an
+entire nested tree unreachable. If a Full sweep finds an ancestor first-touch
+target with a live path-child, it keeps the sidecar. Open may complete for
+leaf-first deletion only when the sidecar owns no physical table effect. A mixed
+attempt that owns an effect plus an untouched fork must roll back as one recovery
+outcome, so open fails closed while the child blocks fork cleanup. After an
+existing handle or an offline Lance-level branch tool removes the child, a later
+Full sweep reclaims the untouched fork, compensates the owned effect, and retires
+the intent.
+
 ## Read-your-writes within a multi-statement mutation
 
 A `.gq` query with multiple ops (e.g. `insert Person … insert Knows …`)
@@ -328,12 +380,17 @@ recovery sweep in `crates/omnigraph/src/db/manifest/recovery.rs`:
   rollback-only; an unknown/foreign effect is refused rather than adopted.
   An Armed first-touch intent with no owned transaction is deferred by live
   roll-forward-only healing because another handle may still own it. Quiesced
-  full recovery tolerates an absent target ref (crash before fork), or removes
-  an exact unchanged unpublished ref after proving no other pending sidecar
-  claims it. With competing claims, the current no-effect sidecar discards
-  itself without touching the ref; the final survivor owns cleanup/recovery.
+  full recovery tolerates an absent target ref (either crash before clone or a
+  clone-only tree with no `BranchContents`) and force-reclaims that absent-ref
+  target idempotently. If an authoritative ref exists, recovery removes it only
+  when it is exactly unchanged and no other pending sidecar claims it. With
+  competing claims, the current no-effect sidecar discards itself without
+  touching the ref; the final survivor owns cleanup/recovery.
   During partial rollback, no-effect refs are removed before the rollback
-  outcome is published so a retry cannot strand them.
+  outcome is published so a retry cannot strand them. If a legacy live
+  path-child blocks that cleanup, rollback returns an error and read-write open
+  fails closed; only a sidecar proven to own no table effect may defer cleanup
+  while returning an open handle.
 - If any table is `InvariantViolation` (Lance HEAD < manifest pinned —
   should be impossible), **abort** with a loud error and leave the
   sidecar on disk for operator review.