Fence Optimize against late recovery intents (#347)

* Fence optimize against late recovery intents

* Preserve manifest compaction after table errors

docs/dev/testing.md

@@ -52,7 +52,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
 | `merge_cost.rs` | Cost-budget tests for branch MERGE on the shared `helpers::cost` harness: `merge_validation_is_delta_scoped` (a 1-row-delta merge opens ≤3 data tables — Δ-scoped, not the whole catalog; was ~6 pre-#5) and `merge_manifest_cost_grows_with_history` (the cross-branch `__manifest` open amplification still grows with commit depth — a separate, not-yet-addressed term — while validation `data_open_count` stays flat) |
 | `policy_engine_chassis.rs` | Engine-layer Cedar enforcement (MR-722): allow + deny through every `_as` writer via the SDK directly — no HTTP — proving embedded and CLI callers hit the same gate as the server, with action × scope shapes matching `authorize_request` |
 | `maintenance.rs` | `optimize` (compaction), `repair` (explicit uncovered-drift publish), and `cleanup` (version GC): empty/idempotent/no-op edges, policy validation, head preservation; cleanup pins exact keep-count behavior (including keep larger than history), count/time retention of a live lazy branch, the oldest of multiple lazy pins, graph-wide fail-closed ordering on an unopenable pin, and refusal of uncovered main HEAD drift before any GC; `optimize` publishes its own compaction (`optimize_publishes_compaction_to_manifest_so_schema_apply_succeeds`), skips pre-existing uncovered drift (`optimize_skips_preexisting_manifest_head_drift`), refuses to run while a `__recovery` sidecar is pending (`optimize_defers_when_recovery_sidecar_is_pending`), and compacts blob-v2 tables through the normal path (`optimize_compacts_blob_table_alongside_plain_table`); `repair` previews/heals verified maintenance drift, refuses raw semantic drift without `--force`, and forced repair publishes only by explicit operator choice; the index reconciler (iss-848): `index_build_tolerates_null_vector_rows` (logical load succeeds without inline index work; an untrainable Vector column then defers during reconciliation) and `optimize_materializes_index_declared_but_unbuilt` (optimize creates a declared-but-deferred index) |
-| `failpoints.rs` | Failure-injection coverage (gated on `failpoints` feature). RFC-022 includes deterministic post-stage/pre-effect races for mutation/load uniqueness and strict disjoint-head changes, plus the cross-handle post-effect `RecoveryRequired` → read-write-open rollback cell. Branch merge adds the captured-source advance cell; post-confirm target-winner compensation; mixed physical + pointer-only delta recovery with fixed commit id/actor/parents; and both sidecar-before-first-ref and ambiguous-ref-create recovery. Native controls are pinned by `native_branch_controls_reclassify_lost_acknowledgements` (matching create and absent-ref delete, with no version/lineage movement); `armed_first_touch_recovery_accepts_missing_target_ref` additionally forges and reclaims the clone-only/no-`BranchContents` table state. Legacy path overlap has both sides pinned: `armed_first_touch_recovery_defers_legacy_path_overlap_until_leaf_delete` permits open only for a proven no-effect intent, while `partial_first_touch_recovery_fails_closed_on_legacy_path_overlap` leaves one exact multi-table effect and verifies open fails closed until offline leaf cleanup lets rollback converge. Other control/recovery race cells include `first_touch_post_create_open_error_keeps_recovery_ownership`, `branch_delete_orphans_sidecar_armed_after_initial_barrier`, `branch_merge_fences_target_delete_recreate_aba`, `branch_merge_fences_concurrent_sync_on_same_handle`, `branch_merge_rejects_fresh_target_manifest_change_before_effects`, `branch_merge_rechecks_late_sidecar_after_table_gates`, `cleanup_rechecks_sidecars_under_gc_gates`, and `full_recovery_rereads_sidecar_body_after_discovery`. The suite also includes the five per-writer effect → manifest-CAS recovery tests, write-entry in-process heal contract, storage-fault matrix, S3 recovery twin, and convergence-idempotent roll-forward regression. |
+| `failpoints.rs` | Failure-injection coverage (gated on `failpoints` feature). RFC-022 includes deterministic post-stage/pre-effect races for mutation/load uniqueness and strict disjoint-head changes, plus the cross-handle post-effect `RecoveryRequired` → read-write-open rollback cell. Branch merge adds the captured-source advance cell; post-confirm target-winner compensation; mixed physical + pointer-only delta recovery with fixed commit id/actor/parents; and both sidecar-before-first-ref and ambiguous-ref-create recovery. Native controls are pinned by `native_branch_controls_reclassify_lost_acknowledgements` (matching create and absent-ref delete, with no version/lineage movement); `armed_first_touch_recovery_accepts_missing_target_ref` additionally forges and reclaims the clone-only/no-`BranchContents` table state. Legacy path overlap has both sides pinned: `armed_first_touch_recovery_defers_legacy_path_overlap_until_leaf_delete` permits open only for a proven no-effect intent, while `partial_first_touch_recovery_fails_closed_on_legacy_path_overlap` leaves one exact multi-table effect and verifies open fails closed until offline leaf cleanup lets rollback converge. Other control/recovery race cells include `first_touch_post_create_open_error_keeps_recovery_ownership`, `branch_delete_orphans_sidecar_armed_after_initial_barrier`, `branch_merge_fences_target_delete_recreate_aba`, `branch_merge_fences_concurrent_sync_on_same_handle`, `branch_merge_rejects_fresh_target_manifest_change_before_effects`, `branch_merge_rechecks_late_sidecar_after_table_gates`, `optimize_rechecks_late_schema_apply_sidecar_after_main_gate` (late zero-pin graph-global intent), `optimize_rechecks_late_disjoint_main_sidecar_after_main_gate` (table-disjoint intent sharing `graph_head:main`), `optimize_holds_main_gate_through_disjoint_table_effects` (post-relist branch-gate lifetime), `cleanup_rechecks_sidecars_under_gc_gates`, and `full_recovery_rereads_sidecar_body_after_discovery`. The suite also includes the five per-writer effect → manifest-CAS recovery tests, write-entry in-process heal contract, storage-fault matrix, S3 recovery twin, and convergence-idempotent roll-forward regression. |
 | `failpoint_names_guard.rs` | Source-walk guard: every `maybe_fail(...)` call site (engine + cluster) must reference the compile-checked `failpoints::names` consts, never a bare string literal — a typo'd literal compiles but silently never fires; same defense-in-depth shape as `forbidden_apis.rs` |
 | `recovery.rs` | Open-time recovery sweep — legacy sidecars; schema-v3 Mutation/Load exact transaction identity; schema-v4 BranchMerge exact multi-commit chains/ref-only effects plus complete logical delta; restartable compensation recognition; fixed logical/rollback outcome IDs; branch-token comparison; fresh under-gate sidecar reread/reparse; all-or-nothing roll-forward/rollback/refusal; audit row in `_graph_commit_recoveries.lance`; and `OpenMode::ReadOnly` skip path |
 | `composite_flow.rs` | Compositional/narrative end-to-end stories — multi-step flows that compose mechanics covered by other test files. Catches integration regressions where individual operations all pass their unit tests but their composition breaks (sequential merges, post-merge main writes, time-travel through merge DAG, reopen consistency over multi-merge histories, post-optimize and post-cleanup strict writes). |

docs/dev/writes.md

@@ -494,8 +494,18 @@ outcome ambiguity while retaining loose table classification. EnsureIndices'
 schema-v6 payload also retains loose table-effect classification, but gives rollback
 a fixed commit id and a durable pre-restore audit plan so recovery re-entry cannot
 flip the outcome. Both close the pre-arm ownership boundary while their full exact
-adapters remain future work. Optimize retains its legacy adapter. The
-manager is shared by every
+adapters remain future work. Optimize retains its legacy effect adapter and a
+branch-authority two-stage recovery barrier: the graph-wide entry probe is a fast
+path; after acquiring main's process-local branch-writer gate, Optimize relists and
+rejects every main-target sidecar plus graph-global SchemaApply before it reads table
+HEADs, classifies drift, or arms its own sidecars. The gate is retained through the
+internally parallel per-table effects/publishes because each table-pointer publish
+advances shared `graph_head:main`. It remains held through final physical-only
+`__manifest` compaction so a new main recovery intent cannot arm before raw manifest
+movement finishes. This coarse legacy-adapter fence makes same-process sidecar-enrolled
+main writers wait for the entire graph-wide Optimize, while Optimize's own table tasks
+remain parallel.
+The manager is shared by every
 `Omnigraph` handle for one canonical local root identity (relative, absolute,
 and symlink aliases converge; object-store/custom schemes stay opaque), so this
 also serializes a refresh or separately-opened handle against a live writer instead of rolling its
@@ -506,7 +516,8 @@ schema → branch → sorted tables → coordinator, matching the writer effect
 Mutation/load, branch merge, SchemaApply, and EnsureIndices perform one additional
 `list_dir` after acquiring their authority gates; that final check closes the
 pre-gate recovery TOCTOU without moving validation or reclaimable staged-file
-construction under the gate.
+construction under the gate. Optimize's separate final `list_dir` runs under the
+main branch gate because even table-disjoint main intents share graph-head authority.
 Pinned by the four
 `tests/failpoints.rs::*_after_finalize_publisher_failure_heals_without_reopen`
 tests (load, mutation, schema apply, branch merge). The maintenance

docs/rfcs/rfc-022-unified-write-path.md

@@ -508,6 +508,18 @@ able to enumerate every adapter and every entry point that invokes it.
 - Schema-v6 does not prove ownership of an index commit and does not add a fixed
   forward outcome. Exact transaction chains, fixed original lineage, and exact
   first-touch ref identity remain requirements of the full adapter.
+- Optimize retains its legacy loose effect adapter. Its synchronous overlap
+  barrier is two-stage: a broad entry probe, then a relist under main's
+  process-local branch-writer gate that rejects every main-target sidecar plus
+  graph-global SchemaApply before HEAD inspection, drift classification, or its
+  own recovery arm. The gate remains held through internally parallel table
+  effects/publishes because every published table pointer advances shared
+  `graph_head:main`, and through final physical `__manifest` compaction so a new
+  main intent cannot arm before raw manifest movement finishes. This coarse
+  legacy-adapter safety cost makes same-process sidecar-enrolled main writers
+  wait for the entire graph-wide Optimize, while Optimize's own table tasks stay
+  parallel. This is not the future exact Optimize adapter, a universal mutex for
+  every main publisher, or a distributed fence.
 - Logical operations never fail because a derived index is absent or behind.
 - Physical-only internal-table maintenance remains the exception in Section 8.
 
@@ -791,6 +803,13 @@ writers.
   after manifest CAS, and during sidecar deletion for every adapter.
 - A later overlapping writer blocks, heals, or returns recovery-required; it never
   advances around the sidecar.
+- If Optimize's entry probe passes and a SchemaApply or table-disjoint main intent
+  arms before its branch gate is acquired, Optimize returns recovery-required with
+  that intent's id; it does not create an Optimize sidecar, move another table,
+  advance `graph_head:main`, or compact `__manifest` afterward.
+- After Optimize's under-gate relist passes, a table-disjoint main writer started
+  while productive table compaction is paused remains queued until Optimize
+  releases its branch-wide effect envelope, then completes normally.
 - A live foreign writer's sidecar is not destructively recovered without fencing.
 - Recovery proves exact effect identity; a numerically newer unrelated version is not
   accepted as proof of ancestry.