Harden branch recovery and cleanup retention (#344)

* Harden branch recovery and cleanup retention

* Fail closed on blocked partial rollback

AGENTS.md

@@ -252,11 +252,11 @@ omnigraph policy explain --cluster ./company-brain --graph knowledge --actor act
 |---|---|---|
 | Columnar storage on object store | ✅ Arrow/Lance | URI normalization, S3 env-var plumbing |
 | Per-dataset versioning + time travel | ✅ | `snapshot_at_version`, `entity_at`, snapshot-pinned reads across many tables |
-| Per-dataset branches | ✅ | **Graph-level** branches (atomic across all sub-tables), lazy fork, system branch filtering |
+| Per-dataset branches | ✅ | **Graph-level** refs are logically atomic through authoritative `__manifest` `BranchContents`; native create/delete crash gaps are classified and reclaimed under a single-writer-process boundary; live names are path-prefix-disjoint; data-table forks are lazy; system branches are filtered |
 | Atomic single-dataset commits | ✅ | **Multi-table publish via three layers**, NOT a single Lance primitive: (1) per-table Lance `commit_staged` for the data write, (2) `__manifest` row-level CAS via `ManifestBatchPublisher` for cross-table ordering, (3) the open-time recovery sweep for the residual gap between (1) and (2). All three layers ship; the five migrated writers (`MutationStaging::commit_all`, `schema_apply`, `branch_merge`, `ensure_indices`, `optimize_all_tables`) write a `__recovery/{ulid}.json` sidecar before Phase B and delete it after Phase C. The next `Omnigraph::open` (gated on `OpenMode::ReadWrite`) runs the sweep in `db/manifest/recovery.rs`: classify, decide all-or-nothing per sidecar, roll forward via single `ManifestBatchPublisher::publish` or roll back via `Dataset::restore` followed by a manifest publish of the restored version (so both directions converge to `manifest == HEAD` — no residual drift), and record an internal audit row in `_graph_commit_recoveries.lance`. A v3 roll-forward preserves the interrupted writer's fixed commit lineage and actor; rollback and legacy recovery commits use `omnigraph:recovery`. There is currently no public CLI query for the recovery-audit table, and ordinary commit history is not a complete recovery enumeration. The write entry points (`load_as`, `mutate_as`, `apply_schema_as`, `branch_merge_as`) and `refresh` additionally run an in-process roll-forward-only heal, serialized against same-process live writers through the shared root-scoped gate manager, so a long-lived server converges on its next write without restart; only rollback-eligible sidecars still defer to the next read-write open (a future background reconciler's goal). Engine writes route through a sealed `TableStorage` trait (`db.storage()`) exposing only `stage_*` + `commit_staged` + reads; the sole inline-commit residual (`create_vector_index`) is split onto a separate sealed `InlineCommitResidual` trait reached via `db.storage_inline_residual()` (MR-854), so the default surface cannot couple a write with a HEAD advance — §1 holds by construction. `delete` migrated to the staged path in MR-A (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, [#6658](https://github.com/lance-format/lance/issues/6658)); `create_vector_index` stays inline until upstream Lance ships a public two-phase API ([#6666](https://github.com/lance-format/lance/issues/6666)); `LoadMode::Overwrite` uses Lance `Overwrite` staged transactions. |
 | Compaction (`compact_files`) + reindex (`optimize_indices`) | ✅ | `omnigraph optimize` orchestrates over all node/edge tables, bounded concurrency; per table runs `compact_files` **then Lance `optimize_indices`** (folds appended/rewritten fragments back into existing indexes — incremental merge, not retrain) and **publishes the resulting version to `__manifest`** (so the manifest tracks the Lance HEAD — required for reads to observe the work and for schema apply / strict writes to pass their HEAD-vs-manifest precondition), under the per-`(table, main)` write queue with `SidecarKind::Optimize` recovery coverage spanning both ops; **commits even with no compaction work if index coverage is stale**; **refuses on an unrecovered graph**; **skips uncovered HEAD > manifest drift** with `DriftNeedsRepair`; **compacts blob-bearing tables** (the pre-9 `LANCE_SUPPORTS_BLOB_COMPACTION` skip was removed once Lance 8.0.0+ shipped blob-v2 compaction — see [docs/dev/invariants.md](docs/dev/invariants.md) Known Gaps) |
 | Repair uncovered drift | — | `omnigraph repair` explicitly classifies uncovered table `HEAD > manifest` drift: verified maintenance drift (`ReserveFragments`/`Rewrite`) can be published with `--confirm`; suspicious or unverifiable drift requires `--force --confirm`. Sidecar-covered crash residuals still recover automatically on open. |
-| Cleanup (`cleanup_old_versions`) | ✅ | `omnigraph cleanup` with `--keep` / `--older-than` policy |
+| Cleanup (`cleanup_old_versions`) | ✅ | `omnigraph cleanup` derives requested `--keep` / `--older-than` cutoffs from each table's available versions; Lance refs plus OmniGraph's live-lazy-branch and recovery floors may retain additional versions. It fails closed on unopenable pins, recovery intent, or uncovered main-table HEAD drift |
 | BTREE / inverted (FTS) / vector indexes | ✅ | `@index`/`@key` declares intent; the physical index is derived state that never fails a logical op. Built per column through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector); idempotent; lazy across branches. **Schema apply and mutation/load build no indexes inline**: the latter publish only their exact data effects, leaving physical intent pending. `ensure_indices`/`optimize` materializes declared-but-missing indexes, restores fragment coverage, and continues to report untrainable Vector columns as pending. |
 | `merge_insert` upsert | ✅ | `LoadMode::Merge`, mutation `update`/`insert`/`delete` lowering |
 | Vector search | ✅ | `nearest()` query op; embedding pipeline (Gemini / OpenAI clients); `@embed` in schema |

crates/omnigraph/src/branch_control.rs

@@ -0,0 +1,699 @@
+//! Crash classification for Lance's native branch controls.
+//!
+//! Lance branch creation is deliberately two-phase: it first commits a shallow
+//! clone under `tree/{branch}` and then writes authoritative `BranchContents`.
+//! Deletion removes `BranchContents` before reclaiming that tree.  These helpers
+//! keep OmniGraph on Lance's public APIs while making both operations bounded
+//! and idempotent under the documented single-writer-process branch-control
+//! boundary. Live graph names are path-prefix-disjoint because Lance cannot
+//! recursively reclaim an ancestor tree while a path-child branch remains.
+
+use std::collections::HashMap;
+
+use lance::Dataset;
+use lance::dataset::refs::{BranchContents, BranchIdentifier, check_valid_branch};
+
+use crate::error::{OmniError, Result};
+
+/// Result of a recoverable native create attempt.
+pub(crate) enum BranchCreateOutcome {
+    /// The requested branch is durably authoritative and its dataset opens.
+    Created(Dataset),
+    /// The target ref existed before this invocation. Callers classify its
+    /// ownership at their own logical layer; this helper never deletes it.
+    /// A ref that appears after target absence was captured is instead a typed
+    /// authority-change error and cannot enter an orphan-reclaim path.
+    RefAlreadyExists,
+}
+
+fn lance_error(error: lance::Error) -> OmniError {
+    OmniError::Lance(error.to_string())
+}
+
+/// Lance's local backend reports `NotFound` when the target tree is already
+/// absent, while object stores commonly treat the empty-prefix delete as a
+/// success. Normalize both to the idempotent contract every OmniGraph branch
+/// reconciler needs.
+pub(crate) async fn force_delete_branch_idempotent(
+    dataset: &mut Dataset,
+    branch: &str,
+) -> Result<()> {
+    let branches = list_branch_contents(dataset).await?;
+    if let Some(child) = path_descendant(&branches, branch) {
+        return Err(OmniError::manifest_conflict(format!(
+            "cannot reclaim branch '{branch}' while live branch '{child}' shares its physical \
+             Lance path; delete the child branch first"
+        )));
+    }
+    force_delete_branch_tree_unchecked(dataset, branch).await
+}
+
+async fn force_delete_branch_tree_unchecked(dataset: &mut Dataset, branch: &str) -> Result<()> {
+    match dataset.force_delete_branch(branch).await {
+        Ok(()) | Err(lance::Error::RefNotFound { .. }) | Err(lance::Error::NotFound { .. }) => {
+            Ok(())
+        }
+        Err(error) => Err(lance_error(error)),
+    }
+}
+
+async fn list_branch_contents(dataset: &Dataset) -> Result<HashMap<String, BranchContents>> {
+    dataset.list_branches().await.map_err(lance_error)
+}
+
+async fn branch_contents(dataset: &Dataset, branch: &str) -> Result<Option<BranchContents>> {
+    Ok(list_branch_contents(dataset).await?.get(branch).cloned())
+}
+
+pub(crate) fn path_descendant<'a>(
+    branches: &'a HashMap<String, BranchContents>,
+    branch: &str,
+) -> Option<&'a str> {
+    let prefix = format!("{branch}/");
+    branches
+        .keys()
+        .map(String::as_str)
+        .filter(|candidate| candidate.starts_with(&prefix))
+        .min()
+}
+
+fn path_collision<'a>(
+    branches: &'a HashMap<String, BranchContents>,
+    branch: &str,
+) -> Option<&'a str> {
+    let branch_prefix = format!("{branch}/");
+    branches
+        .keys()
+        .map(String::as_str)
+        .filter(|candidate| {
+            *candidate != branch
+                && *candidate != "main"
+                && (candidate.starts_with(&branch_prefix)
+                    || branch.starts_with(&format!("{candidate}/")))
+        })
+        .min()
+}
+
+fn encode_identifier(identifier: &BranchIdentifier) -> Result<String> {
+    serde_json::to_string(identifier).map_err(|error| {
+        OmniError::manifest_internal(format!("failed to encode Lance branch identifier: {error}"))
+    })
+}
+
+async fn authority_appeared_after_absence(dataset: &Dataset, branch: &str) -> Result<OmniError> {
+    match branch_contents(dataset, branch).await? {
+        Some(contents) => Ok(OmniError::manifest_read_set_changed(
+            format!("branch_identifier:{branch}"),
+            None,
+            Some(encode_identifier(&contents.identifier)?),
+        )),
+        None => Ok(OmniError::manifest_conflict(format!(
+            "branch authority for '{branch}' changed during absent-only reclaim; refusing \
+             destructive cleanup"
+        ))),
+    }
+}
+
+/// Reclaim only when the authoritative ref is freshly absent. Returns `false`
+/// if a ref exists and therefore nothing was touched. A live path-child is a
+/// conflict because Lance deliberately keeps the ancestor directory in that
+/// state; the graph namespace prevents new instances of that overlap.
+pub(crate) async fn reclaim_ref_absent_tree(dataset: &mut Dataset, branch: &str) -> Result<bool> {
+    let branches = list_branch_contents(dataset).await?;
+    if branches.contains_key(branch) {
+        return Ok(false);
+    }
+    if let Some(child) = path_descendant(&branches, branch) {
+        return Err(OmniError::manifest_conflict(format!(
+            "cannot reclaim absent branch '{branch}' while live branch '{child}' shares its \
+             physical Lance path; delete the child branch first"
+        )));
+    }
+    // This is the final authority read before the destructive call. The
+    // supported single-writer-process gate boundary prevents a local create
+    // from interleaving after it; an exact ref observed here is never passed to
+    // Lance's broad force-delete primitive.
+    force_delete_branch_tree_unchecked(dataset, branch).await?;
+    Ok(true)
+}
+
+/// A completed create receives an identifier equal to the parent's complete
+/// identifier mapping plus one newly-generated `(parent_version, uuid)` entry.
+/// The UUID itself cannot be pre-minted through Lance's public API, so this is
+/// the strongest completion proof available inside the supported
+/// single-writer-process boundary.
+fn matches_create_expectation(
+    contents: &BranchContents,
+    parent_branch: Option<&str>,
+    parent_version: u64,
+    parent_identifier: &BranchIdentifier,
+) -> bool {
+    let expected_parent = parent_branch.filter(|branch| *branch != "main");
+    let mapping = &contents.identifier.version_mapping;
+    let parent_mapping = &parent_identifier.version_mapping;
+    contents.parent_branch.as_deref() == expected_parent
+        && contents.parent_version == parent_version
+        && mapping.len() == parent_mapping.len() + 1
+        && mapping.starts_with(parent_mapping)
+        && mapping
+            .last()
+            .is_some_and(|(version, uuid)| *version == parent_version && !uuid.is_empty())
+}
+
+/// Create a branch with a bounded recovery classifier.
+///
+/// The caller must already hold OmniGraph's schema → source/target branch →
+/// table gate envelope. An absent `BranchContents` ref makes any same-name tree
+/// derived garbage, so it is reclaimed before the first attempt. An ambiguous
+/// native error is then classified from fresh authority: matching contents are
+/// accepted as a lost acknowledgement, mismatching contents are never deleted,
+/// and a ref-less clone is reclaimed before one bounded retry.
+pub(crate) async fn create_branch_recoverably(
+    source: &mut Dataset,
+    branch: &str,
+    source_version: u64,
+) -> Result<BranchCreateOutcome> {
+    // Lance validates inside phase 2 today. Validate before phase 1 so an
+    // invalid name cannot leave a clone-only zombie.
+    check_valid_branch(branch).map_err(lance_error)?;
+    if source.version().version != source_version {
+        return Err(OmniError::manifest_conflict(format!(
+            "branch source moved before native create: expected version {}, current {}",
+            source_version,
+            source.version().version
+        )));
+    }
+
+    let parent_branch = source.manifest().branch.clone();
+    let parent_identifier = source.branch_identifier().await.map_err(lance_error)?;
+
+    let initial_branches = list_branch_contents(source).await?;
+    if initial_branches.contains_key(branch) {
+        return Ok(BranchCreateOutcome::RefAlreadyExists);
+    }
+    if let Some(conflicting) = path_collision(&initial_branches, branch) {
+        return Err(OmniError::manifest_conflict(format!(
+            "cannot create branch '{branch}' while live branch '{conflicting}' shares its \
+             physical Lance path; live graph branch names may not be ancestors or descendants"
+        )));
+    }
+    if !reclaim_ref_absent_tree(source, branch).await? {
+        return Err(authority_appeared_after_absence(source, branch).await?);
+    }
+
+    for attempt in 0..2 {
+        let native_error = match source
+            .create_branch(branch, source_version, None)
+            .await
+            .map_err(lance_error)
+        {
+            Ok(created) => match crate::failpoints::maybe_fail(
+                crate::failpoints::names::BRANCH_CREATE_POST_NATIVE,
+            ) {
+                Ok(()) => return Ok(BranchCreateOutcome::Created(created)),
+                Err(error) => error,
+            },
+            Err(error) => error,
+        };
+
+        let observed = branch_contents(source, branch).await.map_err(|classifier_error| {
+            OmniError::manifest_internal(format!(
+                "native create of branch '{branch}' returned an ambiguous error ({native_error}); \
+                 reading BranchContents to classify it also failed ({classifier_error})"
+            ))
+        })?;
+        if let Some(contents) = observed {
+            if !matches_create_expectation(
+                &contents,
+                parent_branch.as_deref(),
+                source_version,
+                &parent_identifier,
+            ) {
+                return Err(OmniError::manifest_read_set_changed(
+                    format!("branch_identifier:{branch}"),
+                    None,
+                    Some(encode_identifier(&contents.identifier)?),
+                ));
+            }
+            let created = source.checkout_branch(branch).await.map_err(|error| {
+                OmniError::manifest_internal(format!(
+                    "branch '{}' has matching authoritative metadata after an ambiguous create, \
+                     but its branch dataset cannot be opened: {}; original native error: {}",
+                    branch, error, native_error
+                ))
+            })?;
+            return Ok(BranchCreateOutcome::Created(created));
+        }
+
+        match reclaim_ref_absent_tree(source, branch).await {
+            Ok(true) => {}
+            Ok(false) => {
+                return Err(authority_appeared_after_absence(source, branch).await?);
+            }
+            Err(cleanup_error) => {
+                return Err(OmniError::manifest_internal(format!(
+                    "native create of branch '{}' failed before authoritative metadata was visible \
+                     ({native_error}); clone-only cleanup also failed ({cleanup_error})",
+                    branch
+                )));
+            }
+        }
+        if attempt == 1 {
+            return Err(native_error);
+        }
+    }
+
+    unreachable!("bounded native branch-create loop returns from every attempt")
+}
+
+/// Delete a branch and classify Lance's authority-removal → tree-cleanup gap.
+///
+/// A missing ref after an error is logical success. Tree reclamation is derived
+/// state and is retried best-effort; a recreated identifier is never deleted.
+pub(crate) async fn delete_branch_recoverably(
+    dataset: &mut Dataset,
+    branch: &str,
+    expected_identifier: &BranchIdentifier,
+) -> Result<()> {
+    let initial = list_branch_contents(dataset).await?;
+    let Some(initial_contents) = initial.get(branch) else {
+        match reclaim_ref_absent_tree(dataset, branch).await {
+            Ok(true) => {}
+            Ok(false) => {
+                return Err(authority_appeared_after_absence(dataset, branch).await?);
+            }
+            Err(cleanup_error) => {
+                tracing::warn!(
+                    target: "omnigraph::branch_control",
+                    branch,
+                    error = %cleanup_error,
+                    "branch authority is already deleted; derived tree reclaim remains pending",
+                );
+            }
+        }
+        return Ok(());
+    };
+    if initial_contents.identifier != *expected_identifier {
+        return Err(OmniError::manifest_read_set_changed(
+            format!("branch_identifier:{branch}"),
+            Some(encode_identifier(expected_identifier)?),
+            Some(encode_identifier(&initial_contents.identifier)?),
+        ));
+    }
+    if let Some(child) = path_descendant(&initial, branch) {
+        return Err(OmniError::manifest_conflict(format!(
+            "cannot delete branch '{branch}' while live branch '{child}' shares its physical \
+             Lance path; delete the child branch first"
+        )));
+    }
+
+    let native_result = match dataset.delete_branch(branch).await.map_err(lance_error) {
+        Ok(()) => {
+            crate::failpoints::maybe_fail(crate::failpoints::names::BRANCH_DELETE_POST_NATIVE)
+        }
+        Err(error) => Err(error),
+    };
+    let native_error = match native_result {
+        Ok(()) => return Ok(()),
+        Err(error) => error,
+    };
+
+    let observed = branch_contents(dataset, branch)
+        .await
+        .map_err(|classifier_error| {
+            OmniError::manifest_internal(format!(
+                "native delete of branch '{branch}' returned an ambiguous error ({native_error}); \
+             reading BranchContents to classify it also failed ({classifier_error})"
+            ))
+        })?;
+    match observed {
+        None => {
+            match reclaim_ref_absent_tree(dataset, branch).await {
+                Ok(true) => {}
+                Ok(false) => {
+                    return Err(authority_appeared_after_absence(dataset, branch).await?);
+                }
+                Err(cleanup_error) => {
+                    tracing::warn!(
+                        target: "omnigraph::branch_control",
+                        branch,
+                        error = %cleanup_error,
+                        "branch authority is deleted; derived tree reclaim remains pending",
+                    );
+                }
+            }
+            Ok(())
+        }
+        Some(contents) if contents.identifier == *expected_identifier => Err(native_error),
+        Some(contents) => Err(OmniError::manifest_read_set_changed(
+            format!("branch_identifier:{branch}"),
+            Some(encode_identifier(expected_identifier)?),
+            Some(encode_identifier(&contents.identifier)?),
+        )),
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::sync::Arc;
+
+    use arrow_array::{Int32Array, RecordBatch, RecordBatchIterator};
+    use arrow_schema::{DataType, Field, Schema};
+    use lance::dataset::{WriteMode, WriteParams};
+
+    async fn test_dataset(dir: &tempfile::TempDir) -> Dataset {
+        let schema = Arc::new(Schema::new(vec![Field::new("id", DataType::Int32, false)]));
+        let batch = RecordBatch::try_new(
+            Arc::clone(&schema),
+            vec![Arc::new(Int32Array::from(vec![1]))],
+        )
+        .unwrap();
+        let reader = RecordBatchIterator::new(vec![Ok(batch)], schema);
+        // forbidden-api-allow: test-only raw Lance fixture for native branch-control truth cells
+        Dataset::write(
+            reader,
+            dir.path().to_str().unwrap(),
+            Some(WriteParams {
+                mode: WriteMode::Create,
+                auto_cleanup: None,
+                skip_auto_cleanup: true,
+                ..Default::default()
+            }),
+        )
+        .await
+        .unwrap()
+    }
+
+    #[test]
+    fn create_match_requires_exact_parent_incarnation_prefix() {
+        let parent = BranchIdentifier {
+            version_mapping: vec![(2, "parent-a".to_string())],
+        };
+        let matching = BranchContents {
+            parent_branch: Some("source".to_string()),
+            identifier: BranchIdentifier {
+                version_mapping: vec![(2, "parent-a".to_string()), (7, "child".to_string())],
+            },
+            parent_version: 7,
+            create_at: 0,
+            manifest_size: 0,
+            metadata: Default::default(),
+        };
+        assert!(matches_create_expectation(
+            &matching,
+            Some("source"),
+            7,
+            &parent
+        ));
+
+        let foreign_parent = BranchIdentifier {
+            version_mapping: vec![(2, "parent-b".to_string())],
+        };
+        assert!(!matches_create_expectation(
+            &matching,
+            Some("source"),
+            7,
+            &foreign_parent
+        ));
+    }
+
+    #[tokio::test]
+    async fn delete_accepts_absent_authority_and_reclaims_remaining_tree() {
+        let dir = tempfile::tempdir().unwrap();
+        let mut dataset = test_dataset(&dir).await;
+        let version = dataset.version().version;
+        dataset
+            .create_branch("feature", version, None)
+            .await
+            .unwrap();
+        let identifier = dataset
+            .list_branches()
+            .await
+            .unwrap()
+            .get("feature")
+            .unwrap()
+            .identifier
+            .clone();
+        std::fs::remove_file(
+            dir.path()
+                .join("_refs")
+                .join("branches")
+                .join("feature.json"),
+        )
+        .unwrap();
+
+        delete_branch_recoverably(&mut dataset, "feature", &identifier)
+            .await
+            .unwrap();
+        assert!(!dir.path().join("tree").join("feature").exists());
+    }
+
+    #[tokio::test]
+    async fn delete_preserves_native_error_while_same_identifier_remains() {
+        let dir = tempfile::tempdir().unwrap();
+        let mut dataset = test_dataset(&dir).await;
+        let version = dataset.version().version;
+        let mut feature = dataset
+            .create_branch("feature", version, None)
+            .await
+            .unwrap();
+        let identifier = dataset
+            .list_branches()
+            .await
+            .unwrap()
+            .get("feature")
+            .unwrap()
+            .identifier
+            .clone();
+        let feature_version = feature.version().version;
+        feature
+            .create_branch("dependent", feature_version, None)
+            .await
+            .unwrap();
+
+        let error = delete_branch_recoverably(&mut dataset, "feature", &identifier)
+            .await
+            .expect_err("a lineage-dependent ref must preserve Lance's delete refusal");
+        assert!(error.to_string().contains("referenc"));
+        assert_eq!(
+            dataset
+                .list_branches()
+                .await
+                .unwrap()
+                .get("feature")
+                .unwrap()
+                .identifier,
+            identifier
+        );
+    }
+
+    #[tokio::test]
+    async fn delete_rejects_recreated_identifier_without_removing_it() {
+        let dir = tempfile::tempdir().unwrap();
+        let mut dataset = test_dataset(&dir).await;
+        let version = dataset.version().version;
+        dataset
+            .create_branch("feature", version, None)
+            .await
+            .unwrap();
+        let original = dataset
+            .list_branches()
+            .await
+            .unwrap()
+            .get("feature")
+            .unwrap()
+            .identifier
+            .clone();
+        dataset.delete_branch("feature").await.unwrap();
+        dataset
+            .create_branch("feature", version, None)
+            .await
+            .unwrap();
+        let recreated = dataset
+            .list_branches()
+            .await
+            .unwrap()
+            .get("feature")
+            .unwrap()
+            .identifier
+            .clone();
+        assert_ne!(original, recreated);
+
+        let error = delete_branch_recoverably(&mut dataset, "feature", &original)
+            .await
+            .expect_err("a recreated target must be fenced by its identifier");
+        let OmniError::Manifest(error) = error else {
+            panic!("expected typed manifest conflict");
+        };
+        match error.details {
+            Some(crate::error::ManifestConflictDetails::ReadSetChanged {
+                member,
+                expected: Some(expected),
+                actual: Some(actual),
+            }) => {
+                assert_eq!(member, "branch_identifier:feature");
+                assert_eq!(expected, serde_json::to_string(&original).unwrap());
+                assert_eq!(actual, serde_json::to_string(&recreated).unwrap());
+            }
+            other => panic!("expected identifier ReadSetChanged, got {other:?}"),
+        }
+        assert_eq!(
+            dataset
+                .list_branches()
+                .await
+                .unwrap()
+                .get("feature")
+                .unwrap()
+                .identifier,
+            recreated,
+            "classifier must never delete the recreated authority"
+        );
+    }
+
+    #[tokio::test]
+    async fn lower_create_chokepoint_rejects_prefix_collision_before_clone() {
+        let dir = tempfile::tempdir().unwrap();
+        let mut dataset = test_dataset(&dir).await;
+        let version = dataset.version().version;
+        dataset
+            .create_branch("feature", version, None)
+            .await
+            .unwrap();
+
+        let error = match create_branch_recoverably(&mut dataset, "feature/child", version).await {
+            Ok(_) => panic!("the lower branch-control surface must enforce prefix disjointness"),
+            Err(error) => error,
+        };
+        assert!(error.to_string().contains("feature"));
+        assert!(error.to_string().contains("physical Lance path"));
+        assert!(
+            !dataset
+                .list_branches()
+                .await
+                .unwrap()
+                .contains_key("feature/child"),
+            "admission refusal must precede authoritative ref creation"
+        );
+        assert!(
+            !dir.path()
+                .join("tree")
+                .join("feature")
+                .join("child")
+                .exists(),
+            "admission refusal must precede Lance's shallow-clone phase"
+        );
+    }
+
+    #[tokio::test]
+    async fn lower_create_preserves_preexisting_same_name_ref() {
+        let dir = tempfile::tempdir().unwrap();
+        let mut dataset = test_dataset(&dir).await;
+        let version = dataset.version().version;
+        dataset
+            .create_branch("feature", version, None)
+            .await
+            .unwrap();
+        let before = dataset
+            .list_branches()
+            .await
+            .unwrap()
+            .get("feature")
+            .unwrap()
+            .identifier
+            .clone();
+
+        let outcome = create_branch_recoverably(&mut dataset, "feature", version)
+            .await
+            .unwrap();
+        assert!(matches!(outcome, BranchCreateOutcome::RefAlreadyExists));
+        let after = dataset
+            .list_branches()
+            .await
+            .unwrap()
+            .get("feature")
+            .unwrap()
+            .identifier
+            .clone();
+        assert_eq!(
+            before, after,
+            "classification must not replace the live ref"
+        );
+    }
+
+    #[tokio::test]
+    async fn absent_only_reclaim_preserves_authority_that_appeared_after_prior_absence() {
+        let dir = tempfile::tempdir().unwrap();
+        let mut dataset = test_dataset(&dir).await;
+        let version = dataset.version().version;
+        assert!(
+            !dataset
+                .list_branches()
+                .await
+                .unwrap()
+                .contains_key("feature"),
+            "precondition: an earlier classifier observed target absence"
+        );
+        dataset
+            .create_branch("feature", version, None)
+            .await
+            .unwrap();
+        let before = dataset
+            .list_branches()
+            .await
+            .unwrap()
+            .get("feature")
+            .unwrap()
+            .identifier
+            .clone();
+
+        assert!(
+            !reclaim_ref_absent_tree(&mut dataset, "feature")
+                .await
+                .unwrap(),
+            "the final authority check must refuse destructive cleanup"
+        );
+        assert_eq!(
+            dataset
+                .list_branches()
+                .await
+                .unwrap()
+                .get("feature")
+                .unwrap()
+                .identifier,
+            before
+        );
+    }
+
+    #[tokio::test]
+    async fn force_reclaim_reports_lexically_first_live_physical_path_child() {
+        let dir = tempfile::tempdir().unwrap();
+        let mut dataset = test_dataset(&dir).await;
+        let version = dataset.version().version;
+        dataset
+            .create_branch("feature/zeta", version, None)
+            .await
+            .unwrap();
+        dataset
+            .create_branch("feature/alpha", version, None)
+            .await
+            .unwrap();
+        dataset
+            .create_branch("feature", version, None)
+            .await
+            .unwrap();
+
+        let error = force_delete_branch_idempotent(&mut dataset, "feature")
+            .await
+            .expect_err("ancestor reclaim must not report success while a path-child is live");
+        assert!(error.to_string().contains("feature/alpha"));
+        assert!(
+            dataset
+                .list_branches()
+                .await
+                .unwrap()
+                .contains_key("feature"),
+            "preflight refusal must preserve ancestor authority"
+        );
+    }
+}

crates/omnigraph/src/db/graph_coordinator.rs

@@ -238,10 +238,10 @@ impl GraphCoordinator {
         let branch = normalize_branch_name(name)?
             .ok_or_else(|| OmniError::manifest("cannot create branch 'main'".to_string()))?;
 
-        // Manifest is the single branch authority (it forks `__manifest` first).
+        // Manifest BranchContents is the single branch authority. Lance creates
-        // The commit graph is a pure `__manifest` projection (Phase B), so a
+        // it in two physical phases (shallow clone, then BranchContents); the
-        // branch create is one atomic manifest op — no derived commit-graph
+        // manifest coordinator classifies/reclaims a clone-only zombie before
-        // branch to fork, and nothing to roll back.
+        // a bounded retry. No graph-lineage branch is created or rolled back.
         self.manifest.create_branch(&branch).await
     }
 
@@ -255,10 +255,11 @@ impl GraphCoordinator {
             )));
         }
 
-        // Manifest is the single branch authority (Phase B): one atomic op makes
+        // Removing manifest BranchContents is the logical visibility point.
-        // the branch cease to exist. The commit graph is a pure `__manifest`
+        // Lance reclaims the branch tree afterward, so an error may still mean
-        // projection with no derived branch to reclaim; the per-table data forks
+        // logical deletion succeeded; the manifest coordinator reclassifies
-        // are reclaimed by `cleanup`, not here.
+        // that outcome from fresh authority. Per-table data forks remain
+        // derived state and are reclaimed by the engine afterward.
         self.manifest.delete_branch(&branch).await
     }
 

crates/omnigraph/src/db/manifest.rs

@@ -634,10 +634,14 @@ impl ManifestCoordinator {
 
     pub async fn create_branch(&mut self, name: &str) -> Result<()> {
         let mut ds = self.dataset.clone();
-        ds.create_branch(name, self.version(), None)
+        match crate::branch_control::create_branch_recoverably(&mut ds, name, self.version())
-            .await
+            .await?
-            .map_err(|e| OmniError::Lance(e.to_string()))?;
+        {
-        Ok(())
+            crate::branch_control::BranchCreateOutcome::Created(_) => Ok(()),
+            crate::branch_control::BranchCreateOutcome::RefAlreadyExists => Err(
+                OmniError::manifest_conflict(format!("branch '{}' already exists", name)),
+            ),
+        }
     }
 
     pub async fn delete_branch(&mut self, name: &str) -> Result<()> {
@@ -649,9 +653,17 @@ impl ManifestCoordinator {
             crate::instrumentation::manifest_wrapper(),
         )
         .await?;
-        ds.delete_branch(name)
+        let branches = ds
+            .list_branches()
             .await
-            .map_err(|e| OmniError::Lance(e.to_string()))?;
+            .map_err(|error| OmniError::Lance(error.to_string()))?;
+        let expected_identifier = branches
+            .get(name)
+            .ok_or_else(|| OmniError::manifest_not_found(format!("branch '{}' not found", name)))?
+            .identifier
+            .clone();
+        crate::branch_control::delete_branch_recoverably(&mut ds, name, &expected_identifier)
+            .await?;
         self.dataset = open_manifest_dataset(&self.root_uri, self.active_branch.as_deref()).await?;
         self.known_state = read_manifest_state(&self.dataset).await?;
         Ok(())

crates/omnigraph/src/db/manifest/recovery.rs

@@ -1817,8 +1817,28 @@ async fn process_sidecar(
                 return Ok(false);
             }
             if matches!(mode, RecoveryMode::Full) {
-                cleanup_unpublished_no_effect_forks(root_uri, storage.as_ref(), sidecar, &states)
+                if let NoEffectForkCleanup::DeferredPathChild {
-                    .await?;
+                    table_path,
+                    target_branch,
+                    path_child,
+                } = cleanup_unpublished_no_effect_forks(
+                    root_uri,
+                    storage.as_ref(),
+                    sidecar,
+                    &states,
+                )
+                .await?
+                {
+                    warn!(
+                        operation_id = sidecar.operation_id.as_str(),
+                        table_path,
+                        branch = target_branch,
+                        path_child,
+                        "recovery: deferring no-effect fork cleanup until legacy path-child \
+                         branches are deleted leaf-first"
+                    );
+                    return Ok(false);
+                }
             }
             warn!(
                 operation_id = sidecar.operation_id.as_str(),
@@ -2369,6 +2389,16 @@ enum EffectOwnership {
     Unverifiable,
 }
 
+#[derive(Debug, Clone, PartialEq, Eq)]
+enum NoEffectForkCleanup {
+    Complete,
+    DeferredPathChild {
+        table_path: String,
+        target_branch: String,
+        path_child: String,
+    },
+}
+
 /// Remove first-touch named-branch refs created by an Armed v3 attempt that
 /// never landed this table's planned transaction.
 ///
@@ -2386,9 +2416,9 @@ async fn cleanup_unpublished_no_effect_forks(
     storage: &dyn StorageAdapter,
     sidecar: &RecoverySidecar,
     states: &[ClassifiedTable],
-) -> Result<()> {
+) -> Result<NoEffectForkCleanup> {
     if sidecar.protocol_v3.is_none() {
-        return Ok(());
+        return Ok(NoEffectForkCleanup::Complete);
     }
 
     let all_sidecars = list_sidecars(root_uri, storage).await?;
@@ -2432,9 +2462,38 @@ async fn cleanup_unpublished_no_effect_forks(
             .list_branches()
             .await
             .map_err(|error| OmniError::Lance(error.to_string()))?;
+        if let Some(child) = crate::branch_control::path_descendant(&branches, target_branch) {
+            // Lance cannot reclaim an ancestor tree while a slash-separated
+            // path-child remains. Old stores could admit that namespace shape.
+            // Keep the ownership sidecar and let open complete so the operator
+            // can delete the child branch first; the next Full sweep then
+            // rechecks authority and reclaims the ancestor fork.
+            warn!(
+                operation_id = sidecar.operation_id.as_str(),
+                table_path = pin.table_path.as_str(),
+                branch = target_branch,
+                path_child = child,
+                "recovery: deferring unpublished fork cleanup for legacy path overlap"
+            );
+            return Ok(NoEffectForkCleanup::DeferredPathChild {
+                table_path: pin.table_path.clone(),
+                target_branch: target_branch.to_string(),
+                path_child: child.to_string(),
+            });
+        }
         let Some(contents) = branches.get(target_branch) else {
-            // Crash-before-fork, or a previous cleanup pass that crashed before
+            // BranchContents is authoritative, but Lance create writes the
-            // its rollback publish. Both are already converged physically.
+            // shallow-cloned target dataset first. The absent-ref state is
+            // therefore either crash-before-fork (idempotent no-op) or an
+            // exact clone-only zombie owned by this already-armed sidecar.
+            // Reclaim both through Lance's force API before retiring intent;
+            // merely skipping here leaves the zombie blocking every retry.
+            if !crate::branch_control::reclaim_ref_absent_tree(&mut dataset, target_branch).await? {
+                return Err(OmniError::manifest_conflict(format!(
+                    "target ref '{target_branch}' appeared during no-effect recovery; refusing \
+                     to retire its ownership intent"
+                )));
+            }
             continue;
         };
         if contents.parent_version != pin.expected_version {
@@ -2468,7 +2527,7 @@ async fn cleanup_unpublished_no_effect_forks(
             .await
             .map_err(|error| OmniError::Lance(error.to_string()))?;
     }
-    Ok(())
+    Ok(NoEffectForkCleanup::Complete)
 }
 
 fn table_requires_rollback_effect(state: &ClassifiedTable) -> bool {
@@ -2546,9 +2605,33 @@ async fn roll_back_sidecar(
     sidecar: &RecoverySidecar,
     states: &[ClassifiedTable],
 ) -> Result<()> {
+    // An Armed multi-table attempt can create every first-touch ref and then
+    // land effects on only a subset. No-effect refs are not selected by the
+    // rollback manifest publish, so they must be removed BEFORE that publish.
+    // If recovery crashed after publishing first, the fixed rollback outcome
+    // would make the next pass finalize/delete the sidecar without ever seeing
+    // the still-orphaned refs. A rollback that owns any physical effect may not
+    // defer and let read-write open succeed: legacy writers are not all enrolled
+    // in the v3 preparation barrier. Fail closed until the path child is removed.
+    if let NoEffectForkCleanup::DeferredPathChild {
+        table_path,
+        target_branch,
+        path_child,
+    } = cleanup_unpublished_no_effect_forks(root_uri, storage, sidecar, states).await?
+    {
+        return Err(OmniError::manifest_internal(format!(
+            "OCC recovery sidecar '{}' owns physical effects but cannot clean unpublished fork \
+             '{}:{}' while legacy path-child '{}' is live; refusing read-write open; delete the \
+             child branch leaf-first using an existing handle or an offline Lance-level branch \
+             tool, then reopen",
+            sidecar.operation_id, table_path, target_branch, path_child
+        )));
+    }
+
     // Once the fixed rollback commit is visible, early recovery finalization no
     // longer has pre-restore table observations. Persist the exact audit plan
-    // first so that path can replay it without fabricating outcomes from pins.
+    // after fork cleanup and before the first restore so that path can replay it
+    // without fabricating outcomes from pins.
     let prepared_v3 = if sidecar.protocol_v3.is_some() {
         Some(prepare_v3_rollback_audit_plan(root_uri, storage, sidecar, states).await?)
     } else {
@@ -2556,14 +2639,6 @@ async fn roll_back_sidecar(
     };
     let sidecar = prepared_v3.as_ref().unwrap_or(sidecar);
 
-    // An Armed multi-table attempt can create every first-touch ref and then
-    // land effects on only a subset. No-effect refs are not selected by the
-    // rollback manifest publish, so they must be removed BEFORE that publish.
-    // If recovery crashed after publishing first, the fixed rollback outcome
-    // would make the next pass finalize/delete the sidecar without ever seeing
-    // the still-orphaned refs.
-    cleanup_unpublished_no_effect_forks(root_uri, storage, sidecar, states).await?;
-
     // Restore every drifted table (RolledPastExpected / UnexpectedAtP1 /
     // UnexpectedMultistep) to its manifest-pinned content, then PUBLISH so
     // `manifest == Lance HEAD` for each — symmetric with roll-forward. The

crates/omnigraph/src/db/omnigraph.rs

@@ -1825,7 +1825,41 @@ impl Omnigraph {
         queue_keys
     }
 
+    fn ensure_branch_create_namespace_safe(target: &str, branches: &[String]) -> Result<()> {
+        if branches.iter().any(|candidate| candidate == target) {
+            return Err(OmniError::manifest_conflict(format!(
+                "branch '{}' already exists",
+                target
+            )));
+        }
+
+        let target_prefix = format!("{target}/");
+        if let Some(conflicting) = branches.iter().find(|candidate| {
+            candidate.as_str() != "main"
+                && (candidate.starts_with(&target_prefix)
+                    || target.starts_with(&format!("{candidate}/")))
+        }) {
+            return Err(OmniError::manifest_conflict(format!(
+                "cannot create branch '{target}' while live branch '{conflicting}' shares its \
+                 physical Lance path; live graph branch names may not be ancestors or descendants"
+            )));
+        }
+
+        Ok(())
+    }
+
     async fn ensure_branch_delete_safe(&self, branch: &str, branches: &[String]) -> Result<()> {
+        let path_prefix = format!("{branch}/");
+        if let Some(child) = branches
+            .iter()
+            .find(|candidate| candidate.starts_with(&path_prefix))
+        {
+            return Err(OmniError::manifest_conflict(format!(
+                "cannot delete branch '{branch}' while live branch '{child}' shares its physical \
+                 Lance path; delete the child branch first"
+            )));
+        }
+
         let descendants = self
             .coordinator
             .read()
@@ -1921,7 +1955,9 @@ impl Omnigraph {
             .map(|entry| (entry.table_key.clone(), entry.table_path.clone()))
             .collect::<Vec<_>>();
 
-        // Authority flip (+ best-effort commit-graph reclaim) — must succeed.
+        // Authority removal is the logical branch deletion. Lance tree cleanup
+        // follows that ref removal; the coordinator classifies an absent ref as
+        // success even if the physical cleanup acknowledgement failed.
         self.coordinator.write().await.branch_delete(branch).await?;
         // Best-effort per-table fork reclaim; cleanup reconciles any leftover.
         self.cleanup_deleted_branch_tables(branch, &owned_tables)
@@ -1998,20 +2034,8 @@ impl Omnigraph {
         self.refresh_coordinator_only().await?;
         self.ensure_schema_apply_not_locked("branch_create").await?;
         self.ensure_schema_state_valid().await?;
-        if self
+        let branches = self.coordinator.read().await.all_branches().await?;
-            .coordinator
+        Self::ensure_branch_create_namespace_safe(&target, &branches)?;
-            .read()
-            .await
-            .all_branches()
-            .await?
-            .iter()
-            .any(|branch| branch == &target)
-        {
-            return Err(OmniError::manifest_conflict(format!(
-                "branch '{}' already exists",
-                target
-            )));
-        }
         self.coordinator.write().await.branch_create(name).await
     }
 
@@ -2107,20 +2131,8 @@ impl Omnigraph {
         self.ensure_schema_apply_not_locked("branch_create_from")
             .await?;
         self.ensure_schema_state_valid().await?;
-        if self
+        let branches = self.coordinator.read().await.all_branches().await?;
-            .coordinator
+        Self::ensure_branch_create_namespace_safe(&target_branch, &branches)?;
-            .read()
-            .await
-            .all_branches()
-            .await?
-            .iter()
-            .any(|candidate| candidate == &target_branch)
-        {
-            return Err(OmniError::manifest_conflict(format!(
-                "branch '{}' already exists",
-                target_branch
-            )));
-        }
         // Operate on a freshly-opened source coordinator that's owned locally
         // — never touch `self.coordinator`. The pre-fix implementation used
         // `swap_coordinator_for_branch` + operate + `restore_coordinator` as

crates/omnigraph/src/db/omnigraph/optimize.rs

@@ -18,12 +18,13 @@
 //!   older manifest versions until `cleanup` runs.
 //! * `cleanup_all_tables` — Lance `cleanup_old_versions` on every table.
 //!   Removes manifests (and their unique fragments) older than the configured
-//!   retention. Destructive to version history — callers should gate this
+//!   retention, capped at the oldest main-table version inherited by any live
-//!   behind an explicit confirm flag at the CLI layer.
+//!   lazy graph branch. Destructive to unreferenced version history — callers
+//!   should gate this behind an explicit confirm flag at the CLI layer.
 //!
 //! Both orchestrate the graph's node + edge datasets from main authority;
-//! cleanup preserves Lance-referenced named-branch history according to its
+//! cleanup preserves both Lance-referenced native branch history and the
-//! retention policy.
+//! graph-level lazy-branch references Lance cannot observe.
 
 use std::time::Duration;
 
@@ -786,7 +787,8 @@ async fn compact_internal_table(
 
 /// Run Lance `cleanup_old_versions` on every node + edge table on `main`,
 /// using [`CleanupPolicyOptions`]. The latest manifest is always preserved
-/// regardless (Lance invariant).
+/// regardless (Lance invariant), and the requested cutoff is capped at the
+/// oldest main-table version inherited by a live lazy graph branch.
 pub async fn cleanup_all_tables(
     db: &mut Omnigraph,
     options: CleanupPolicyOptions,
@@ -896,8 +898,77 @@ pub async fn cleanup_all_tables(
         ));
     }
 
+    // Lance protects versions referenced by its own per-dataset branches, but
+    // an OmniGraph branch is lazy: until a table is first written on that
+    // branch its manifest entry points directly at an older MAIN version and
+    // no Lance branch ref exists on the data table. Resolve every live graph
+    // branch from fresh authority while schema + all branch/table gates are
+    // held, then cap each main dataset's GC cutoff at its oldest such pin.
+    // Main itself participates: its manifest-visible version must open and
+    // equal Lance HEAD, so uncovered drift is repaired before cleanup rather
+    // than letting HEAD-based GC collect graph-visible authority.
+    // Any branch snapshot read failure aborts before the first table GC: an
+    // unknown live reference is never evidence that a version is disposable.
+    let mut oldest_live_main_version_by_path = std::collections::HashMap::<String, u64>::new();
+    for branch_target in &graph_branches {
+        if branch_target
+            .as_deref()
+            .is_some_and(crate::db::is_internal_system_branch)
+        {
+            continue;
+        }
+        let branch_label = branch_target.as_deref().unwrap_or("main");
+        let branch_snapshot = db
+            .fresh_snapshot_for_branch(branch_target.as_deref())
+            .await
+            .map_err(|err| {
+                OmniError::manifest_conflict(format!(
+                    "cleanup could not classify live branch '{branch_label}'; refusing version GC: {err}"
+                ))
+            })?;
+        for entry in branch_snapshot
+            .entries()
+            .filter(|entry| entry.table_branch.is_none())
+        {
+            // Validate that the exact protected version is still openable
+            // before GC starts. This catches pre-existing damage from an older
+            // cleanup implementation and keeps the sweep fail-closed instead
+            // of deleting unrelated history around an already-broken branch.
+            entry.open(db.root_uri(), None).await.map_err(|err| {
+                OmniError::manifest_conflict(format!(
+                    "cleanup could not classify live branch '{branch_label}' table '{}' at main version {}; refusing version GC: {err}",
+                    entry.table_key, entry.table_version
+                ))
+            })?;
+            let full_path = format!("{}/{}", db.root_uri, entry.table_path);
+            if branch_target.is_none() {
+                let head = db.storage().open_dataset_head(&full_path, None).await?;
+                if head.version() != entry.table_version {
+                    return Err(OmniError::manifest_conflict(format!(
+                        "cleanup found uncovered HEAD drift for table '{}': manifest version {}, \
+                         Lance HEAD {}; run `omnigraph repair` before version GC",
+                        entry.table_key,
+                        entry.table_version,
+                        head.version()
+                    )));
+                }
+            }
+            oldest_live_main_version_by_path
+                .entry(full_path)
+                .and_modify(|oldest| *oldest = (*oldest).min(entry.table_version))
+                .or_insert(entry.table_version);
+        }
+    }
+
     let before_timestamp = options.older_than.map(|d| Utc::now() - d);
     let keep_versions = options.keep_versions;
+    let table_tasks = table_tasks
+        .into_iter()
+        .map(|(table_key, full_path)| {
+            let live_main_floor = oldest_live_main_version_by_path.get(&full_path).copied();
+            (table_key, full_path, live_main_floor)
+        })
+        .collect::<Vec<_>>();
 
     if table_tasks.is_empty() {
         return Ok(Vec::new());
@@ -911,7 +982,7 @@ pub async fn cleanup_all_tables(
     // cleanup is the convergence backstop, so it must do as much as it can and
     // converge on re-run rather than fail wholesale (invariant 13).
     let results: Vec<TableCleanupStats> = futures::stream::iter(table_tasks.into_iter())
-        .map(|(table_key, full_path)| async move {
+        .map(|(table_key, full_path, live_main_floor)| async move {
             let outcome: Result<RemovalStats> = async {
                 crate::failpoints::maybe_fail(crate::failpoints::names::CLEANUP_TABLE_GC)?;
                 // `cleanup_old_versions` is a Lance-only maintenance API not
@@ -919,9 +990,36 @@ pub async fn cleanup_all_tables(
                 // above for the same rationale. Unwrap via `into_dataset()`.
                 let handle = storage.open_dataset_head(&full_path, None).await?;
                 let ds = handle.into_dataset();
-                let before_version = keep_versions
+                let requested_before_version = if let Some(keep) = keep_versions {
-                    .map(|n| ds.version().version.saturating_sub(n as u64))
+                    // Lance versions are not safely derivable from HEAD
-                    .filter(|v| *v > 0);
+                    // arithmetic after prior GC. Use the actual ordered
+                    // version list so `keep=N` retains exactly the newest N
+                    // available versions (with HEAD as the unavoidable floor
+                    // when N=0).
+                    let versions = ds
+                        .versions()
+                        .await
+                        .map_err(|error| OmniError::Lance(error.to_string()))?;
+                    let retain = (keep as usize).max(1);
+                    let cutoff = if versions.len() <= retain {
+                        versions.first()
+                    } else {
+                        versions.get(versions.len() - retain)
+                    }
+                    .ok_or_else(|| {
+                        OmniError::manifest_internal(format!(
+                            "cleanup found no versions for open table '{table_key}'"
+                        ))
+                    })?;
+                    Some(cutoff.version)
+                } else {
+                    None
+                };
+                let before_version = match (requested_before_version, live_main_floor) {
+                    (Some(requested), Some(floor)) => Some(requested.min(floor)),
+                    (None, Some(floor)) => Some(floor),
+                    (requested, None) => requested,
+                };
                 let policy = CleanupPolicy {
                     before_timestamp,
                     before_version,

crates/omnigraph/src/exec/staging.rs

@@ -1155,6 +1155,10 @@ impl StagedMutation {
                 row_count: state.row_count,
                 version_metadata: state.version_metadata,
             });
+            crate::failpoints::maybe_fail(crate::failpoints::names::MUTATION_POST_TABLE_COMMIT)
+                .map_err(|error| {
+                    OmniError::recovery_required(operation_id.clone(), error.to_string())
+                })?;
         }
 
         if let Err(error) = confirm_occ_sidecar_phase_b(

crates/omnigraph/src/failpoints.rs

@@ -38,7 +38,15 @@ pub(crate) fn maybe_fail_retryable_contention(name: &str) -> Result<()> {
 /// reference these constants instead of bare string literals, so a typo is a
 /// compile error rather than a silently-never-firing failpoint.
 pub mod names {
+    /// After Lance returns success from its two-phase native create, before
+    /// OmniGraph acknowledges it. Recovery must classify the matching
+    /// BranchContents as a completed create (lost acknowledgement).
+    pub const BRANCH_CREATE_POST_NATIVE: &str = "branch_create.post_native";
     pub const BRANCH_DELETE_BEFORE_TABLE_CLEANUP: &str = "branch_delete.before_table_cleanup";
+    /// After Lance returns success from native delete, before OmniGraph
+    /// acknowledges it. Recovery must classify the absent BranchContents as a
+    /// completed logical deletion.
+    pub const BRANCH_DELETE_POST_NATIVE: &str = "branch_delete.post_native";
     /// Branch delete holds the schema, target-branch, and fresh-catalog table
     /// envelope and has completed its final recovery check, before the native
     /// manifest-ref mutation.
@@ -94,6 +102,11 @@ pub mod names {
     /// After every deferred first-touch table ref is created under a durable
     /// v3 sidecar, before any staged data transaction advances target HEAD.
     pub const MUTATION_POST_FORK_PRE_COMMIT: &str = "mutation.post_fork_pre_commit";
+    /// After each exact staged table transaction advances HEAD, before the next
+    /// table effect or Phase-B confirmation. Used to leave a real partial
+    /// multi-table v3 attempt whose remaining first-touch fork still needs
+    /// recovery cleanup.
+    pub const MUTATION_POST_TABLE_COMMIT: &str = "mutation.post_table_commit";
     /// After the v3 ownership sidecar is durable but before the first deferred
     /// named-table ref is created. Recovery must accept the absent target ref.
     pub const MUTATION_POST_SIDECAR_PRE_FORK: &str = "mutation.post_sidecar_pre_fork";

crates/omnigraph/src/lib.rs

@@ -7,6 +7,7 @@
 // future Lance bumps stop needing this.
 #![recursion_limit = "256"]
 
+mod branch_control;
 pub mod changes;
 pub mod db;
 pub mod embedding;

crates/omnigraph/src/storage_layer.rs

@@ -295,14 +295,12 @@ pub trait TableStorage: sealed::Sealed + Send + Sync + Debug {
         target_branch: &str,
     ) -> Result<ForkOutcome<SnapshotHandle>>;
 
-    async fn delete_branch(&self, dataset_uri: &str, branch: &str) -> Result<()>;
+    /// Idempotent branch-tree reclaim used by the best-effort fork cleanup
-
+    /// under branch delete (`db/omnigraph.rs::cleanup_deleted_branch_tables`)
-    /// Idempotent variant of `delete_branch` used by the best-effort fork
-    /// reclaim under branch delete (`db/omnigraph.rs::cleanup_deleted_branch_tables`)
     /// and by the orphan-fork reconciler in `optimize`. Tolerates an
     /// already-absent branch (both Lance's `RefNotFound` and the local-store
     /// `NotFound` quirk on a missing `tree/{branch}/` dir). A still-referenced
-    /// branch (`RefConflict`) still surfaces as `OmniError::Lance`.
+    /// branch (`RefConflict`) or live physical path-child remains an error.
     async fn force_delete_branch(&self, dataset_uri: &str, branch: &str) -> Result<()>;
 
     /// List the named Lance branches present on the dataset at `dataset_uri`.
@@ -591,10 +589,6 @@ impl TableStorage for TableStore {
         )
     }
 
-    async fn delete_branch(&self, dataset_uri: &str, branch: &str) -> Result<()> {
-        TableStore::delete_branch(self, dataset_uri, branch).await
-    }
-
     async fn force_delete_branch(&self, dataset_uri: &str, branch: &str) -> Result<()> {
         TableStore::force_delete_branch(self, dataset_uri, branch).await
     }

crates/omnigraph/src/table_store.rs

@@ -277,19 +277,6 @@ impl TableStore {
         }
     }
 
-    pub async fn delete_branch(&self, dataset_uri: &str, branch: &str) -> Result<()> {
-        let mut ds = crate::instrumentation::open_dataset(
-            dataset_uri,
-            crate::instrumentation::VersionResolution::Latest,
-            Some(&self.session),
-            crate::instrumentation::table_wrapper(),
-        )
-        .await?;
-        ds.delete_branch(branch)
-            .await
-            .map_err(|e| OmniError::Lance(e.to_string()))
-    }
-
     /// List the named Lance branches present on the dataset at `dataset_uri`.
     /// The `cleanup` orphan reconciler diffs this against the manifest branch
     /// set to find orphaned per-table forks. `main`/default is not a named
@@ -311,15 +298,16 @@ impl TableStore {
 
     /// Idempotently drop `branch` from the dataset at `dataset_uri`.
     ///
-    /// Unlike [`delete_branch`](Self::delete_branch), this tolerates an
+    /// This tolerates an already-absent branch — both a missing contents ref (Lance's
-    /// already-absent branch — both a missing contents ref (Lance's
     /// `force_delete_branch` handles that) and a missing `tree/{branch}/`
     /// directory (the local-store `NotFound` quirk pinned by
     /// `lance_surface_guards::force_delete_branch_semantics`). Safe to call on a
     /// possibly-orphaned or already-reclaimed fork.
     ///
-    /// A branch that still has referencing descendants (`RefConflict`) is NOT
+    /// A branch that still has referencing descendants (`RefConflict`) or a
-    /// tolerated: that is a real ordering error and surfaces as `OmniError::Lance`.
+    /// live physical path-child is NOT tolerated: those are real ordering
+    /// errors. The graph namespace prevents new path-prefix overlaps; surfacing
+    /// legacy ones keeps cleanup from falsely reporting a reclaim Lance skipped.
     /// Used by the eager best-effort reclaim in `cleanup_deleted_branch_tables`
     /// and the `cleanup` orphan reconciler.
     pub async fn force_delete_branch(&self, dataset_uri: &str, branch: &str) -> Result<()> {
@@ -330,11 +318,7 @@ impl TableStore {
             crate::instrumentation::table_wrapper(),
         )
         .await?;
-        match ds.force_delete_branch(branch).await {
+        crate::branch_control::force_delete_branch_idempotent(&mut ds, branch).await
-            Ok(()) => Ok(()),
-            Err(lance::Error::RefNotFound { .. }) | Err(lance::Error::NotFound { .. }) => Ok(()),
-            Err(e) => Err(OmniError::Lance(e.to_string())),
-        }
     }
 
     pub fn ensure_expected_version(
@@ -387,49 +371,28 @@ impl TableStore {
             .map_err(|e| OmniError::Lance(e.to_string()))?;
         self.ensure_expected_version(&source_ds, table_key, source_version)?;
 
-        if let Err(create_err) = source_ds
+        let created = match crate::branch_control::create_branch_recoverably(
-            .create_branch(target_branch, source_version, None)
+            &mut source_ds,
-            .await
+            target_branch,
+            source_version,
+        )
+        .await?
         {
-            // Disambiguate the failure: only a genuinely pre-existing ref is a
+            crate::branch_control::BranchCreateOutcome::Created(dataset) => dataset,
-            // reclaim candidate. Mapping EVERY create_branch failure to
+            crate::branch_control::BranchCreateOutcome::RefAlreadyExists => {
-            // `RefAlreadyExists` would route a transient I/O / version / Lance
-            // internal error into the destructive reclaim path. So check whether
-            // the ref actually exists; if not, the failure is real — propagate
-            // it (preserving error fidelity) rather than force-deleting.
-            //
-            // `list_branches` reads `_refs/branches/` from the store, so it sees
-            // a fully-formed manifest-unreferenced fork (our common case — a
-            // create_branch that completed but whose manifest publish did not).
-            // It does NOT see a phase-1-only Lance "zombie" (tree dir written,
-            // no BranchContents) — but neither does `cleanup`'s reconciler, also
-            // list_branches-based. A zombie only forms if create_branch is
-            // interrupted *between its two internal phases* (a far narrower
-            // window than the manifest-publish gap), and it surfaces here as the
-            // propagated create error requiring manual reclaim. We deliberately
-            // do NOT force-delete on a not-found-ref failure: it is
-            // indistinguishable from a transient error on a fresh create, and
-            // force-deleting there is the destructive overreach this guard
-            // removes. The caller holds the per-(table, branch) write queue, so
-            // no in-process writer races this fork; a cross-process create
-            // between our check and now is the documented one-winner-CAS gap and
-            // propagates as a retryable error.
-            let ref_exists = source_ds
-                .list_branches()
-                .await
-                .map(|b| b.contains_key(target_branch))
-                .unwrap_or(false);
-            if ref_exists {
                 return Ok(ForkOutcome::RefAlreadyExists);
             }
-            return Err(OmniError::Lance(create_err.to_string()));
+        };
-        }
 
         // The ref is now independently durable. Any error from this point is an
         // ambiguous/post-effect outcome to the caller and must retain an armed
         // recovery intent rather than being treated as a safe pre-effect retry.
         crate::failpoints::maybe_fail(crate::failpoints::names::FORK_POST_CREATE_PRE_OPEN)?;
 
+        // Re-open through the shared session for normal cache behavior. The
+        // returned handle above is used only as proof that the matching branch
+        // dataset was openable during classification.
+        drop(created);
         let ds = self
             .open_dataset_head(dataset_uri, Some(target_branch))
             .await?;

crates/omnigraph/tests/branching.rs

@@ -128,6 +128,19 @@ async fn branch_create_open_list_and_lazy_branching_work() {
     let uri = dir.path().to_str().unwrap();
     let mut main = init_and_load(&dir).await;
 
+    main.branch_create("feature").await.unwrap();
+    // Reproduce Lance's phase-1-only crash state: keep the shallow-cloned
+    // `tree/feature` dataset but remove BranchContents, its sole logical
+    // authority. A same-name graph create must reclaim the zombie and retry,
+    // rather than surfacing DatasetAlreadyExists forever.
+    std::fs::remove_file(
+        dir.path()
+            .join("__manifest")
+            .join("_refs")
+            .join("branches")
+            .join("feature.json"),
+    )
+    .unwrap();
     main.branch_create("feature").await.unwrap();
     assert_eq!(main.branch_list().await.unwrap(), vec!["main", "feature"]);
 
@@ -1044,6 +1057,18 @@ async fn branch_created_from_non_main_inherits_branch_state() {
         .branch_create_from(ReadTarget::branch("feature"), "experiment")
         .await
         .unwrap();
+    std::fs::remove_file(
+        dir.path()
+            .join("__manifest")
+            .join("_refs")
+            .join("branches")
+            .join("experiment.json"),
+    )
+    .unwrap();
+    feature
+        .branch_create_from(ReadTarget::branch("feature"), "experiment")
+        .await
+        .expect("non-main create must also reclaim a clone-only target");
 
     assert_eq!(
         feature.branch_list().await.unwrap(),
@@ -1530,6 +1555,17 @@ async fn branch_api_rejects_reserved_main_and_same_source_target_merge() {
     let err = db.branch_delete("main").await.unwrap_err();
     assert!(err.to_string().contains("cannot delete branch 'main'"));
 
+    let err = db.branch_create("bad branch").await.unwrap_err();
+    assert!(err.to_string().contains("invalid") || err.to_string().contains("allowed"));
+    assert!(
+        !dir.path()
+            .join("__manifest")
+            .join("tree")
+            .join("bad branch")
+            .exists(),
+        "branch names must be validated before Lance's shallow-clone phase"
+    );
+
     let err = db.branch_merge("main", "main").await.unwrap_err();
     assert!(err.to_string().contains("distinct source and target"));
 
@@ -1574,6 +1610,91 @@ async fn branch_delete_removes_owned_table_branches_and_allows_recreate() {
     assert_eq!(count_rows_branch(&main, "feature", "node:Person").await, 5);
 }
 
+#[tokio::test]
+async fn branch_namespace_rejects_live_physical_path_prefix_collisions() {
+    let dir = tempfile::tempdir().unwrap();
+    let mut db = init_and_load(&dir).await;
+
+    db.branch_create("feature").await.unwrap();
+    let err = db.branch_create("feature/child").await.unwrap_err();
+    assert!(
+        err.to_string().contains("physical Lance path")
+            && err.to_string().contains("ancestors or descendants"),
+        "prefix collision must be actionable; got: {err}"
+    );
+    assert!(
+        !dir.path()
+            .join("__manifest")
+            .join("tree")
+            .join("feature")
+            .join("child")
+            .exists(),
+        "prefix admission must reject before Lance creates the target clone"
+    );
+
+    db.branch_delete("feature").await.unwrap();
+    db.branch_create("feature/child").await.unwrap();
+    let err = db.branch_create("feature").await.unwrap_err();
+    assert!(
+        err.to_string().contains("physical Lance path")
+            && err.to_string().contains("feature/child"),
+        "ancestor creation must reject the inverse prefix collision; got: {err}"
+    );
+    assert!(
+        !lance::Dataset::open(&format!("{}/__manifest", dir.path().display()))
+            .await
+            .unwrap()
+            .list_branches()
+            .await
+            .unwrap()
+            .contains_key("feature"),
+        "inverse admission refusal must not create an ancestor ref"
+    );
+}
+
+#[tokio::test]
+async fn branch_delete_refuses_legacy_physical_path_children() {
+    let dir = tempfile::tempdir().unwrap();
+    let uri = dir.path().to_str().unwrap();
+    let mut db = init_and_load(&dir).await;
+    db.branch_create("feature/child").await.unwrap();
+
+    // Forge a graph written before the prefix-disjoint namespace invariant.
+    // Lance permits both refs but intentionally cannot reclaim the ancestor's
+    // dataset directory while the child path is live.
+    let mut manifest = lance::Dataset::open(&format!("{uri}/__manifest"))
+        .await
+        .unwrap();
+    let version = manifest.version().version;
+    manifest
+        .create_branch("feature", version, None)
+        .await
+        .unwrap();
+
+    let err = db.branch_delete("feature").await.unwrap_err();
+    assert!(
+        err.to_string().contains("feature/child")
+            && err.to_string().contains("delete the child branch first"),
+        "legacy prefix collisions must be deleted leaf-first; got: {err}"
+    );
+    assert!(
+        manifest
+            .list_branches()
+            .await
+            .unwrap()
+            .contains_key("feature"),
+        "refusal must not remove ancestor authority"
+    );
+
+    db.branch_delete("feature/child").await.unwrap();
+    db.branch_delete("feature").await.unwrap();
+    assert_eq!(
+        db.branch_list().await.unwrap(),
+        vec!["main"],
+        "legacy overlap must converge when deleted leaf-first"
+    );
+}
+
 #[tokio::test]
 async fn branch_delete_rejects_branches_still_referenced_by_descendants() {
     let dir = tempfile::tempdir().unwrap();

crates/omnigraph/tests/failpoints.rs

@@ -56,6 +56,50 @@ fn node_table_uri(root: &str, type_name: &str) -> String {
     format!("{}/nodes/{hash:016x}", root.trim_end_matches('/'))
 }
 
+// Lance can durably complete a native ref mutation while the caller observes
+// an error (for example, a lost object-store acknowledgement). BranchContents
+// is the logical authority in both directions: matching create metadata means
+// success, and an absent ref means delete succeeded even if tree cleanup or the
+// acknowledgement failed. Neither control emits graph lineage or a main-table
+// version.
+#[tokio::test]
+#[serial]
+async fn native_branch_controls_reclassify_lost_acknowledgements() {
+    let _scenario = FailScenario::setup();
+    let dir = tempfile::tempdir().unwrap();
+    let db = helpers::init_and_load(&dir).await;
+    let before_version = version_main(&db).await.unwrap();
+    let before_commits = db.list_commits(Some("main")).await.unwrap().len();
+
+    {
+        let _fp = ScopedFailPoint::new(names::BRANCH_CREATE_POST_NATIVE, "return");
+        db.branch_create("feature")
+            .await
+            .expect("matching BranchContents must classify a lost create acknowledgement");
+    }
+    assert!(
+        db.branch_list()
+            .await
+            .unwrap()
+            .iter()
+            .any(|branch| branch == "feature")
+    );
+
+    {
+        let _fp = ScopedFailPoint::new(names::BRANCH_DELETE_POST_NATIVE, "return");
+        db.branch_delete("feature")
+            .await
+            .expect("absent BranchContents must classify a lost delete acknowledgement");
+    }
+    assert_eq!(db.branch_list().await.unwrap(), vec!["main".to_string()]);
+    assert_eq!(version_main(&db).await.unwrap(), before_version);
+    assert_eq!(
+        db.list_commits(Some("main")).await.unwrap().len(),
+        before_commits,
+        "native branch controls must not manufacture graph lineage"
+    );
+}
+
 // Branch delete flips the manifest authority first, then reclaims the per-table
 // forks best-effort. A failure during that reclaim (here, the
 // `branch_delete.before_table_cleanup` failpoint, standing in for a transient
@@ -647,6 +691,31 @@ async fn armed_first_touch_recovery_accepts_missing_target_ref() {
         "precondition: crash happened before the target ref was created"
     );
 
+    // Materialize the narrower Lance crash window underneath the already-
+    // durable writer intent: shallow clone present, BranchContents absent.
+    // Full recovery must not equate "not listed" with "no physical fork".
+    let mut person = lance::Dataset::open(&person_uri).await.unwrap();
+    let person_version = person.version().version;
+    person
+        .create_branch("feature", person_version, None)
+        .await
+        .unwrap();
+    std::fs::remove_file(
+        std::path::Path::new(&person_uri)
+            .join("_refs")
+            .join("branches")
+            .join("feature.json"),
+    )
+    .unwrap();
+    assert!(
+        std::path::Path::new(&person_uri)
+            .join("tree")
+            .join("feature")
+            .exists(),
+        "precondition: clone-only target tree exists"
+    );
+    drop(person);
+
     // Stage A is a synchronous, branch-aware barrier. The Armed sidecar has no
     // table effect for a HEAD-drift check to discover, but it is still ownership:
     // another mutation on the same branch must name it and stop before preparing
@@ -727,6 +796,314 @@ async fn armed_first_touch_recovery_accepts_missing_target_ref() {
             .exists(),
         "full recovery must remove the empty armed intent"
     );
+    assert!(
+        !std::path::Path::new(&person_uri)
+            .join("tree")
+            .join("feature")
+            .exists(),
+        "full recovery must reclaim an unlisted clone-only table fork"
+    );
+}
+
+#[tokio::test]
+#[serial]
+async fn armed_first_touch_recovery_defers_legacy_path_overlap_until_leaf_delete() {
+    let _scenario = FailScenario::setup();
+    let dir = tempfile::tempdir().unwrap();
+    let uri = dir.path().to_str().unwrap().to_string();
+    let db = helpers::init_and_load(&dir).await;
+    let main_rows = helpers::count_rows(&db, "node:Person").await;
+
+    // Materialize the leaf first so its graph snapshot owns a matching
+    // per-table Lance branch. New OmniGraph versions reject the inverse live
+    // namespace, but old stores could contain it.
+    db.branch_create("feature/child").await.unwrap();
+    db.mutate(
+        "feature/child",
+        MUTATION_QUERIES,
+        "insert_person",
+        &mixed_params(&[("$name", "Leaf")], &[("$age", 22)]),
+    )
+    .await
+    .unwrap();
+
+    // Forge only the legacy graph-level ancestor admission. It inherits main,
+    // so an interrupted first write will own an unpublished table fork.
+    let mut manifest = lance::Dataset::open(&format!("{uri}/__manifest"))
+        .await
+        .unwrap();
+    let manifest_version = manifest.version().version;
+    manifest
+        .create_branch("feature", manifest_version, None)
+        .await
+        .unwrap();
+    drop(manifest);
+
+    {
+        let _failpoint = ScopedFailPoint::new(names::MUTATION_POST_SIDECAR_PRE_FORK, "return");
+        let error = db
+            .mutate(
+                "feature",
+                MUTATION_QUERIES,
+                "insert_person",
+                &mixed_params(&[("$name", "Ancestor")], &[("$age", 23)]),
+            )
+            .await
+            .expect_err("failure after arming must leave the ancestor recovery intent");
+        assert!(matches!(error, OmniError::RecoveryRequired { .. }));
+    }
+    let operation_id = single_sidecar_operation_id(dir.path());
+
+    // Reproduce Lance's narrower clone-only crash window under the already
+    // durable intent. The live leaf shares the ancestor's physical path, so a
+    // force-delete of the ancestor cannot safely complete yet.
+    let person_uri = node_table_uri(&uri, "Person");
+    let mut person = lance::Dataset::open(&person_uri).await.unwrap();
+    let person_version = person.version().version;
+    person
+        .create_branch("feature", person_version, None)
+        .await
+        .unwrap();
+    std::fs::remove_file(
+        std::path::Path::new(&person_uri)
+            .join("_refs")
+            .join("branches")
+            .join("feature.json"),
+    )
+    .unwrap();
+    assert!(
+        person
+            .list_branches()
+            .await
+            .unwrap()
+            .contains_key("feature/child"),
+        "precondition: live leaf table branch exists"
+    );
+    assert!(
+        !person
+            .list_branches()
+            .await
+            .unwrap()
+            .contains_key("feature"),
+        "precondition: ancestor is clone-only"
+    );
+    drop(person);
+    drop(db);
+
+    let recovered = Omnigraph::open(&uri)
+        .await
+        .expect("legacy path overlap must defer cleanup instead of wedging read-write open");
+    assert!(
+        dir.path()
+            .join("__recovery")
+            .join(format!("{operation_id}.json"))
+            .exists(),
+        "deferred cleanup must retain its ownership sidecar"
+    );
+    assert_eq!(
+        helpers::count_rows_branch(&recovered, "feature", "node:Person").await,
+        main_rows,
+        "the empty ancestor intent must remain unpublished"
+    );
+
+    recovered
+        .branch_delete("feature/child")
+        .await
+        .expect("open handle must permit the documented leaf-first remediation");
+    drop(recovered);
+
+    let recovered = Omnigraph::open(&uri)
+        .await
+        .expect("the next Full sweep must finish ancestor cleanup");
+    assert!(
+        !dir.path()
+            .join("__recovery")
+            .join(format!("{operation_id}.json"))
+            .exists(),
+        "recovery must retire the intent after the path child is gone"
+    );
+    assert!(
+        !std::path::Path::new(&person_uri)
+            .join("tree")
+            .join("feature")
+            .exists(),
+        "the clone-only ancestor tree must be reclaimed after leaf deletion"
+    );
+    assert_eq!(
+        helpers::count_rows_branch(&recovered, "feature", "node:Person").await,
+        main_rows
+    );
+}
+
+#[tokio::test]
+#[serial]
+async fn partial_first_touch_recovery_fails_closed_on_legacy_path_overlap() {
+    let _scenario = FailScenario::setup();
+    let dir = tempfile::tempdir().unwrap();
+    let uri = dir.path().to_str().unwrap().to_string();
+    let db = helpers::init_and_load(&dir).await;
+    let main_person_rows = helpers::count_rows(&db, "node:Person").await;
+    let main_edge_rows = helpers::count_rows(&db, "edge:Knows").await;
+
+    let main_snapshot = db.snapshot_of("main").await.unwrap();
+    let table_pins = ["node:Person", "edge:Knows"]
+        .into_iter()
+        .map(|table_key| {
+            let entry = main_snapshot.entry(table_key).unwrap();
+            (
+                table_key.to_string(),
+                format!("{uri}/{}", entry.table_path),
+                entry.table_version,
+            )
+        })
+        .collect::<Vec<_>>();
+
+    db.branch_create("feature").await.unwrap();
+    let operation_id = {
+        let _failpoint = ScopedFailPoint::new(names::MUTATION_POST_TABLE_COMMIT, "return");
+        let error = db
+            .mutate(
+                "feature",
+                MUTATION_QUERIES,
+                "insert_person_and_friend",
+                &mixed_params(
+                    &[("$name", "Ancestor"), ("$friend", "Alice")],
+                    &[("$age", 23)],
+                ),
+            )
+            .await
+            .expect_err("the first exact table effect must leave a partial v3 intent");
+        match error {
+            OmniError::RecoveryRequired { operation_id, .. } => operation_id,
+            other => panic!("expected RecoveryRequired, got {other}"),
+        }
+    };
+    assert_eq!(single_sidecar_operation_id(dir.path()), operation_id);
+
+    // All deferred first-touch refs are created before the table-commit loop.
+    // Staging order is intentionally non-semantic, so either table may own the
+    // one durable effect while its sibling remains at the exact fork point.
+    let mut head_deltas = Vec::new();
+    let mut heads_before_open = Vec::new();
+    for (table_key, table_uri, expected_version) in &table_pins {
+        let mut root = lance::Dataset::open(table_uri).await.unwrap();
+        let branches = root.list_branches().await.unwrap();
+        assert!(
+            branches.contains_key("feature"),
+            "precondition: {table_key} has its armed first-touch ref"
+        );
+        let head = root
+            .checkout_branch("feature")
+            .await
+            .unwrap()
+            .version()
+            .version;
+        heads_before_open.push((table_key.clone(), head));
+        head_deltas.push(head.checked_sub(*expected_version).unwrap());
+
+        // Forge the legacy namespace only after the partial effect exists. New
+        // OmniGraph versions reject this path-prefix overlap at branch create,
+        // but old stores can contain it. Clone from the exact feature HEAD so
+        // the child itself introduces no extra ancestor movement.
+        root.create_branch("feature/child", ("feature", head), None)
+            .await
+            .unwrap();
+    }
+    head_deltas.sort_unstable();
+    assert_eq!(
+        head_deltas,
+        vec![0, 1],
+        "exactly one table effect must be durable while its sibling remains an untouched fork"
+    );
+
+    let mut manifest = lance::Dataset::open(&format!("{uri}/__manifest"))
+        .await
+        .unwrap();
+    let feature_manifest_version = manifest
+        .checkout_branch("feature")
+        .await
+        .unwrap()
+        .version()
+        .version;
+    manifest
+        .create_branch("feature/child", ("feature", feature_manifest_version), None)
+        .await
+        .unwrap();
+    drop(manifest);
+
+    let open_error = match Omnigraph::open(&uri).await {
+        Ok(_) => panic!(
+            "Full recovery must not return a writable handle while owned effects remain unrolled"
+        ),
+        Err(error) => error,
+    };
+    assert!(
+        open_error.to_string().contains("owns physical effects")
+            && open_error.to_string().contains("feature/child"),
+        "failure must explain the safe leaf-first remediation boundary: {open_error}"
+    );
+    assert!(
+        dir.path()
+            .join("__recovery")
+            .join(format!("{operation_id}.json"))
+            .exists(),
+        "failed Full recovery must retain exact ownership"
+    );
+    let manifest_after_failed_open = lance::Dataset::open(&format!("{uri}/__manifest"))
+        .await
+        .unwrap()
+        .checkout_branch("feature")
+        .await
+        .unwrap()
+        .version()
+        .version;
+    assert_eq!(
+        manifest_after_failed_open, feature_manifest_version,
+        "failed rollback preflight must not publish the manifest"
+    );
+    for ((table_key, table_uri, _), (_, before)) in table_pins.iter().zip(heads_before_open.iter())
+    {
+        let root = lance::Dataset::open(table_uri).await.unwrap();
+        let after = root
+            .checkout_branch("feature")
+            .await
+            .unwrap()
+            .version()
+            .version;
+        assert_eq!(
+            after, *before,
+            "failed preflight must not restore or publish {table_key}"
+        );
+    }
+
+    // A handle that predates the interrupted attempt can remove the graph leaf
+    // under the normal branch-control gates. This fixture forged its table refs
+    // outside the manifest, so finish the documented offline leaf cleanup at
+    // Lance level. Once the physical overlap is gone, the next quiesced Full
+    // sweep can compensate the owned effect atomically.
+    db.branch_delete("feature/child").await.unwrap();
+    for (_, table_uri, _) in &table_pins {
+        lance::Dataset::open(table_uri)
+            .await
+            .unwrap()
+            .force_delete_branch("feature/child")
+            .await
+            .unwrap();
+    }
+    drop(db);
+
+    let recovered = Omnigraph::open(&uri)
+        .await
+        .expect("rollback must converge after leaf-first remediation");
+    assert!(helpers::recovery::sidecar_operation_ids(dir.path()).is_empty());
+    assert_eq!(
+        helpers::count_rows_branch(&recovered, "feature", "node:Person").await,
+        main_person_rows
+    );
+    assert_eq!(
+        helpers::count_rows_branch(&recovered, "feature", "edge:Knows").await,
+        main_edge_rows
+    );
 }
 
 #[tokio::test]

crates/omnigraph/tests/lance_surface_guards.rs

@@ -338,7 +338,7 @@ async fn _compile_uncommitted_merge_insert_field_shape() -> lance::Result<()> {
 // The branch-delete reconciler (`db/omnigraph/optimize.rs::reconcile_orphaned_branches`)
 // and the eager best-effort reclaim in `cleanup_deleted_branch_tables` call
 // `force_delete_branch` to drop orphaned branch refs. The single-authority
-// design relies on three facts pinned here:
+// design relies on five facts pinned here:
 //   1. plain `delete_branch` errors on a missing ref (so the design uses the
 //      force variant instead);
 //   2. `force_delete_branch` removes an existing (forked) branch — the orphan
@@ -347,7 +347,14 @@ async fn _compile_uncommitted_merge_insert_field_shape() -> lance::Result<()> {
 //      errors on the local store, because `remove_dir_all`'s NotFound is not
 //      caught for Lance's native error variant. `TableStore::force_delete_branch`
 //      wraps this to be fully idempotent. Pin the raw quirk so a future Lance
-//      fix (which would let us simplify the wrapper) is noticed.
+//      fix (which would let us simplify the wrapper) is noticed;
+//   4. a clone-only zombie (branch dataset present, BranchContents absent)
+//      blocks raw create and is reclaimed by `force_delete_branch`. Lance's
+//      create is explicitly two-phase, so this is the crash state OmniGraph's
+//      native branch-control wrapper must heal before retrying;
+//   5. a live slash-name path-child makes force delete remove an ancestor's
+//      BranchContents but intentionally retain its dataset files. OmniGraph's
+//      prefix-disjoint live-name invariant prevents this false-success shape.
 
 #[tokio::test]
 async fn force_delete_branch_semantics() {
@@ -379,6 +386,56 @@ async fn force_delete_branch_semantics() {
         "force_delete_branch on a fully-absent branch no longer errors — \
          TableStore::force_delete_branch's NotFound tolerance can be simplified."
     );
+
+    // (4) Exact phase-1-only create state: create the shallow-cloned branch
+    // dataset, then remove only its authoritative BranchContents ref. This is
+    // the same fixture Lance's own dataset-versioning test uses for a zombie.
+    ds.create_branch("zombie", base, None).await.unwrap();
+    std::fs::remove_file(
+        std::path::Path::new(uri)
+            .join("_refs")
+            .join("branches")
+            .join("zombie.json"),
+    )
+    .unwrap();
+    assert!(
+        !ds.list_branches().await.unwrap().contains_key("zombie"),
+        "BranchContents is the authority; the clone-only tree must not list as a branch"
+    );
+    assert!(
+        ds.create_branch("zombie", base, None).await.is_err(),
+        "the clone-only tree should block an unclassified raw create"
+    );
+    ds.force_delete_branch("zombie").await.unwrap();
+    assert!(
+        !std::path::Path::new(uri)
+            .join("tree")
+            .join("zombie")
+            .exists(),
+        "force_delete_branch must reclaim the clone-only tree"
+    );
+
+    // (5) Slash-separated names overlap physically. A path-child created from
+    // main is not a lineage descendant of its lexical ancestor, so raw force
+    // delete removes the ancestor ref but deliberately leaves its dataset
+    // files to avoid recursively deleting the child.
+    ds.create_branch("ancestor/child", base, None)
+        .await
+        .unwrap();
+    ds.create_branch("ancestor", base, None).await.unwrap();
+    ds.force_delete_branch("ancestor").await.unwrap();
+    assert!(
+        !ds.list_branches().await.unwrap().contains_key("ancestor"),
+        "raw force delete still removes authoritative ancestor metadata"
+    );
+    assert!(
+        std::path::Path::new(uri)
+            .join("tree")
+            .join("ancestor")
+            .join("_versions")
+            .exists(),
+        "Lance must retain ancestor dataset files while a physical path-child is live"
+    );
 }
 
 // --- Guard 10: blob-column compaction works in this Lance ------------------

crates/omnigraph/tests/maintenance.rs

@@ -17,8 +17,8 @@ use omnigraph::loader::{LoadMode, load_jsonl};
 use omnigraph::table_store::{IndexCoverage, TableStore};
 
 use helpers::{
-    MUTATION_QUERIES, TEST_DATA, TEST_SCHEMA, count_rows, init_and_load, mixed_params, mutate_main,
+    MUTATION_QUERIES, TEST_DATA, TEST_SCHEMA, count_rows, count_rows_branch, init_and_load,
-    snapshot_main,
+    mixed_params, mutate_main, snapshot_main,
 };
 
 /// Filesystem URI of a node sub-table, mirroring the engine's layout
@@ -925,7 +925,9 @@ async fn cleanup_without_any_policy_option_errors() {
 #[tokio::test]
 async fn cleanup_keep_one_preserves_head_and_table_remains_readable() {
     let dir = tempfile::tempdir().unwrap();
+    let uri = dir.path().to_str().unwrap().to_string();
     let mut db = init_and_load(&dir).await;
+    add_person_fragments(&mut db).await;
 
     let people_before = count_rows(&db, "node:Person").await;
     assert!(
@@ -933,9 +935,21 @@ async fn cleanup_keep_one_preserves_head_and_table_remains_readable() {
         "fixture should seed Person rows for this test to be meaningful"
     );
 
-    // Most aggressive version-based cleanup short of forcing keep=0. Lance's
+    let person_uri = node_table_uri(&uri, "Person");
-    // contract is that head is always preserved regardless, so the table
+    assert!(
-    // must remain openable and rows must still be visible.
+        Dataset::open(&person_uri)
+            .await
+            .unwrap()
+            .versions()
+            .await
+            .unwrap()
+            .len()
+            > 1,
+        "precondition: Person must have history to collect"
+    );
+
+    // Most aggressive version-based cleanup short of forcing keep=0. `keep`
+    // is exact over the available version list, not HEAD arithmetic.
     let _stats = db
         .cleanup(CleanupPolicyOptions {
             keep_versions: Some(1),
@@ -945,6 +959,52 @@ async fn cleanup_keep_one_preserves_head_and_table_remains_readable() {
         .unwrap();
 
     assert_eq!(count_rows(&db, "node:Person").await, people_before);
+    assert_eq!(
+        Dataset::open(&person_uri)
+            .await
+            .unwrap()
+            .versions()
+            .await
+            .unwrap()
+            .len(),
+        1,
+        "keep=1 must retain exactly the latest available version"
+    );
+}
+
+#[tokio::test]
+async fn cleanup_keep_exceeding_history_preserves_every_available_version() {
+    let dir = tempfile::tempdir().unwrap();
+    let uri = dir.path().to_str().unwrap().to_string();
+    let mut db = init_and_load(&dir).await;
+    let person_uri = node_table_uri(&uri, "Person");
+    let before = Dataset::open(&person_uri)
+        .await
+        .unwrap()
+        .versions()
+        .await
+        .unwrap()
+        .len();
+    assert!(before > 1, "fixture must contain version history");
+
+    db.cleanup(CleanupPolicyOptions {
+        keep_versions: Some(10),
+        older_than: None,
+    })
+    .await
+    .unwrap();
+
+    let after = Dataset::open(&person_uri)
+        .await
+        .unwrap()
+        .versions()
+        .await
+        .unwrap()
+        .len();
+    assert_eq!(
+        after, before,
+        "keep greater than available history must not become an unbounded cleanup"
+    );
 }
 
 #[tokio::test]
@@ -970,6 +1030,224 @@ async fn cleanup_older_than_zero_preserves_head() {
         .unwrap();
 }
 
+#[tokio::test]
+async fn cleanup_preserves_main_version_pinned_by_live_lazy_branch() {
+    let dir = tempfile::tempdir().unwrap();
+    let mut db = init_and_load(&dir).await;
+
+    db.branch_create("feature").await.unwrap();
+    let feature_before = db.snapshot_of(ReadTarget::branch("feature")).await.unwrap();
+    let feature_person = feature_before.entry("node:Person").unwrap();
+    assert_eq!(
+        feature_person.table_branch, None,
+        "precondition: Person must still be inherited lazily from main"
+    );
+    let pinned_main_version = feature_person.table_version;
+    let feature_people_before = count_rows_branch(&db, "feature", "node:Person").await;
+
+    // Move main far enough that keep=1 would collect the version inherited by
+    // the lazy branch unless cleanup accounts for graph-level branch pins.
+    add_person_fragments(&mut db).await;
+    let main_person_version = db
+        .snapshot_of(ReadTarget::branch("main"))
+        .await
+        .unwrap()
+        .entry("node:Person")
+        .unwrap()
+        .table_version;
+    assert!(
+        pinned_main_version < main_person_version.saturating_sub(1),
+        "precondition: lazy-branch pin must fall outside keep=1 retention"
+    );
+
+    db.cleanup(CleanupPolicyOptions {
+        keep_versions: Some(1),
+        older_than: None,
+    })
+    .await
+    .unwrap();
+
+    assert_eq!(
+        count_rows_branch(&db, "feature", "node:Person").await,
+        feature_people_before,
+        "cleanup must preserve the exact main-table version inherited by a live lazy branch"
+    );
+
+    // The same floor must constrain a time-only policy. Lance combines the
+    // timestamp and version predicates with AND, so injecting the branch pin
+    // as `before_version` keeps the inherited version even when every old
+    // manifest satisfies the timestamp cutoff.
+    db.cleanup(CleanupPolicyOptions {
+        keep_versions: None,
+        older_than: Some(Duration::from_secs(0)),
+    })
+    .await
+    .unwrap();
+    assert_eq!(
+        count_rows_branch(&db, "feature", "node:Person").await,
+        feature_people_before,
+        "time-only cleanup must honor the same live lazy-branch floor"
+    );
+}
+
+#[tokio::test]
+async fn cleanup_uses_oldest_pin_across_multiple_live_lazy_branches() {
+    let dir = tempfile::tempdir().unwrap();
+    let mut db = init_and_load(&dir).await;
+
+    db.branch_create("a-old").await.unwrap();
+    let old_rows = count_rows_branch(&db, "a-old", "node:Person").await;
+    add_person_fragments(&mut db).await;
+    db.branch_create("z-new").await.unwrap();
+    let new_rows = count_rows_branch(&db, "z-new", "node:Person").await;
+    mutate_main(
+        &mut db,
+        MUTATION_QUERIES,
+        "insert_person",
+        &mixed_params(&[("$name", "Ivan")], &[("$age", 44)]),
+    )
+    .await
+    .unwrap();
+
+    db.cleanup(CleanupPolicyOptions {
+        keep_versions: Some(1),
+        older_than: None,
+    })
+    .await
+    .unwrap();
+
+    assert_eq!(
+        count_rows_branch(&db, "a-old", "node:Person").await,
+        old_rows,
+        "the oldest live pin must win over later branch pins"
+    );
+    assert_eq!(
+        count_rows_branch(&db, "z-new", "node:Person").await,
+        new_rows,
+        "newer lazy pins must remain readable too"
+    );
+}
+
+#[tokio::test]
+async fn cleanup_fails_closed_when_live_lazy_branch_pin_is_unopenable() {
+    let dir = tempfile::tempdir().unwrap();
+    let uri = dir.path().to_str().unwrap().to_string();
+    let mut db = init_and_load(&dir).await;
+
+    db.branch_create("feature").await.unwrap();
+    let pinned_main_version = db
+        .snapshot_of(ReadTarget::branch("feature"))
+        .await
+        .unwrap()
+        .entry("node:Person")
+        .unwrap()
+        .table_version;
+    add_person_fragments(&mut db).await;
+
+    // Simulate damage created by an older cleanup implementation: raw Lance
+    // sees no native Person branch for the lazy graph branch and removes its
+    // inherited main manifest.
+    let person_uri = node_table_uri(&uri, "Person");
+    let ds = Dataset::open(&person_uri).await.unwrap();
+    let head = ds.version().version;
+    assert!(pinned_main_version < head, "precondition: main advanced");
+    let removed = lance::dataset::cleanup::cleanup_old_versions(
+        &ds,
+        lance::dataset::cleanup::CleanupPolicy {
+            before_timestamp: None,
+            before_version: Some(head),
+            delete_unverified: false,
+            error_if_tagged_old_versions: false,
+            clean_referenced_branches: false,
+            delete_rate_limit: None,
+        },
+    )
+    .await
+    .unwrap();
+    assert!(
+        removed.old_versions > 0,
+        "precondition: raw Lance cleanup removed old main versions"
+    );
+    let company_uri = node_table_uri(&uri, "Company");
+    let company_versions_before = Dataset::open(&company_uri)
+        .await
+        .unwrap()
+        .versions()
+        .await
+        .unwrap()
+        .len();
+
+    let err = db
+        .cleanup(CleanupPolicyOptions {
+            keep_versions: Some(1),
+            older_than: None,
+        })
+        .await
+        .expect_err("cleanup must fail closed when a live lazy pin cannot be opened");
+    let message = err.to_string();
+    assert!(
+        message.contains("could not classify live branch 'feature'")
+            && message.contains("node:Person"),
+        "error must identify the unclassifiable live reference; got: {message}"
+    );
+    assert_eq!(
+        Dataset::open(&company_uri)
+            .await
+            .unwrap()
+            .versions()
+            .await
+            .unwrap()
+            .len(),
+        company_versions_before,
+        "the graph-wide preflight must fail before any unrelated table GC"
+    );
+}
+
+#[tokio::test]
+async fn cleanup_refuses_uncovered_main_head_drift_before_any_version_gc() {
+    let dir = tempfile::tempdir().unwrap();
+    let root = dir.path().to_str().unwrap().to_string();
+    let mut db = init_and_load(&dir).await;
+    let (manifest_version, head_version, _) = forge_person_compaction_drift(&mut db, &root).await;
+    let company_uri = node_table_uri(&root, "Company");
+    let company_versions_before = Dataset::open(&company_uri)
+        .await
+        .unwrap()
+        .versions()
+        .await
+        .unwrap()
+        .len();
+
+    let err = db
+        .cleanup(CleanupPolicyOptions {
+            keep_versions: Some(1),
+            older_than: None,
+        })
+        .await
+        .expect_err("cleanup must not garbage-collect around uncovered HEAD drift");
+    let message = err.to_string();
+    assert!(
+        message.contains("uncovered HEAD drift")
+            && message.contains("node:Person")
+            && message.contains("repair"),
+        "drift refusal must be actionable; got: {message}"
+    );
+    let (manifest_after, head_after, _) = person_manifest_and_head(&db, &root).await;
+    assert_eq!(manifest_after, manifest_version);
+    assert_eq!(head_after, head_version);
+    assert_eq!(
+        Dataset::open(&company_uri)
+            .await
+            .unwrap()
+            .versions()
+            .await
+            .unwrap()
+            .len(),
+        company_versions_before,
+        "drift preflight must abort before unrelated table history is collected"
+    );
+}
+
 #[tokio::test]
 async fn cleanup_then_optimize_preserves_rows_and_table_remains_writable() {
     // Cleanup destroys version history; the concern is that subsequent

docs/dev/invariants.md

@@ -159,7 +159,8 @@ converge the physical state.
 | Multi-table commit | Manifest CAS plus recovery sidecars; not a single Lance primitive | [writes.md](writes.md), [architecture.md](architecture.md) |
 | Constructive mutations | In-memory `MutationStaging`, one end-of-query table commit per touched table, then one manifest publish | [writes.md](writes.md), [execution.md](execution.md) |
 | Deletes | Staged like inserts/updates (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, MR-A) — no inline HEAD advance; mixed insert/update/delete in one query rejected by D2 as a deliberate boundary (constructive XOR destructive per query; compose via separate mutations or a branch) | [query-language.md](../user/queries/index.md), [writes.md](writes.md) |
-| Branch delete | Manifest is the single authority, flipped atomically first; per-table forks are derived state, reclaimed best-effort (`force_delete_branch`) with the `cleanup` reconciler as the guaranteed backstop. Under schema/target/all-table gates, a target-scoped unresolved sidecar may be made unreachable by deletion and is then audit-discarded by recovery; graph-global SchemaApply still blocks. Reusing a name whose fork reclaim failed before `cleanup` surfaces an actionable error | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) |
+| Branch create/delete | `__manifest` `BranchContents` is the single logical authority. Lance create is physically two-phase, so OmniGraph prevalidates names, enforces path-prefix-disjoint live graph names, reclaims an absent-ref clone-only tree, and uses a bounded completion classifier; delete removes authority before tree cleanup, so an absent ref is success and derived tree reclaim may converge later. Neither control emits graph lineage. Per-table forks are derived state, reclaimed best-effort with `cleanup` as backstop. Under schema/target/all-table gates, a target-scoped unresolved sidecar may be made unreachable by deletion and is then audit-discarded by recovery; graph-global SchemaApply still blocks | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) |
+| Cleanup retention | Explicit cleanup derives exact `keep` cutoffs from Lance's available version list, caps each main-table GC cutoff at the oldest exact main version inherited by any live lazy graph branch, and refuses uncovered main HEAD drift. Lance protects native per-table branch refs itself. The graph-wide live-reference preflight fails closed before the first table GC, after which individual table failures remain fault-isolated | [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) |
 | Schema validation | Type checks, required fields, defaults, edge endpoint checks, and edge cardinality are enforced on write paths | [schema-language.md](../user/schema/index.md), [execution.md](execution.md) |
 | Unique constraints | Value/enum, uniqueness, edge-RI, and cardinality route through ONE unified, catalog-derived evaluator (`crate::validate`) on ALL THREE write surfaces — branch-merge, mutation, and bulk load: Δ-scoped (checks the delta, not the whole graph) and structured-filter-backed (committed probes use a BTREE when reconciled and remain correct by scanning while it is pending), reusing the leaf checks (`loader::validate_value_constraints`/`validate_enum_constraints`/`composite_unique_key`) so the surfaces cannot drift. This closed the prior merge bug (merge validated `@range`/`@check` but not enum) AND the **cross-version uniqueness gap** on the mutation and load paths (a duplicate of a committed `@unique` value is now rejected; the merge path always enforced it). The committed view is the merge target snapshot (merge), the write's pinned `txn.base` (mutation), or the pinned pre-load base (load — `Overwrite` validates the batch as the whole new image, committed view empty); `@card` refreshes the manifest-visible graph-branch snapshot on the mutation path only (the #298 stale-handle fix), then follows each entry's actual inherited/owned Lance ref. `@key` is id-backed, so it is checked intra-delta only (a committed holder of a key value is always the same row — an upsert), skipping a wasted O(Δ) probe per keyed row; `@unique` (non-key) groups do the committed lookup. | [schema-language.md](../user/schema/index.md) |
 | Storage trait | `TableStorage` (via `db.storage()`) is staged-only; the sole inline-commit residual (`create_vector_index`) is split onto a separate sealed `InlineCommitResidual` trait reached via `db.storage_inline_residual()` (MR-854), so §1 holds by construction; capability/stat surfaces are roadmap | [writes.md](writes.md), [architecture.md](architecture.md) |
@@ -168,12 +169,23 @@ converge the physical state.
 | Auth | Bearer token hashing and server-side actor resolution are implemented at the HTTP boundary | [server.md](../user/operations/server.md), [policy.md](../user/operations/policy.md) |
 | Tests | Tempdir-backed Lance tests are the current substrate; the storage adapter has an in-memory backend for adapter-level contract tests, but Lance datasets bypass it | [testing.md](testing.md) |
 
-The branch-delete reconciler is authority-derived: it reclaims orphaned forks
+The branch-control reconcilers are authority-derived: same-name create reclaims
-today and degrades to a no-op if Lance ships an atomic multi-dataset branch
+an absent-ref clone-only manifest tree, and delete/cleanup reclaim orphaned
-operation, so the design composes with that future rather than blocking it. This
+per-table forks. They degrade to no-ops if Lance closes the corresponding
-is the same shape as invariant 7 (indexes are derived state); prefer it over a
+physical gaps, so the design composes with that future rather than blocking it.
-recovery-sidecar-style approach for any new multi-dataset metadata operation,
+This is the same shape as invariant 7 (indexes are derived state); prefer it
-since the sidecar would be scaffolding to remove once the substrate closes the gap.
+over a recovery-sidecar-style approach for metadata work whose logical target
+is fully derivable from one existing authority. A graph-visible table effect is
+different and still requires durable ownership before it can advance HEAD.
+For legacy path-prefix overlaps, an ancestor first-touch tree is not proven
+unreachable while a live child remains. Full recovery may leave the sidecar
+intact and allow the graph to open for leaf-first child deletion **only when the
+intent owns no physical table effect**. If the same sidecar owns any durable
+effect, cleanup is part of an all-or-nothing rollback: read-write open fails
+closed until the child is removed through an existing handle or an offline
+Lance-level branch tool, then the next Full sweep reclaims the untouched fork and
+compensates the owned effect. Returning a writable handle with that rollback
+incomplete would let a legacy writer prepare from unresolved physical state.
 
 ## Known Gaps
 
@@ -252,19 +264,24 @@ them explicit.
   files and commits. Both `reclaim_orphaned_fork_and_refork` and
   `reconcile_orphaned_branches` consult pending sidecars before destruction; a
   foreign claim is conflict/indeterminate, never permission to delete. Full
-  recovery accepts sidecar-before-ref crashes and deletes an unpublished fork
+  recovery accepts sidecar-before-ref crashes. When `BranchContents` is absent,
-  only when it is still exactly at the inherited version and no other sidecar
+  the already-armed intent plus the single-writer-process gate makes a same-name
-  claims `(table_path, target ref)`. A no-effect sidecar with a competitor
+  clone-only tree unreachable, so recovery force-reclaims it without pretending
+  it can inspect an unlisted version. When a ref exists, recovery deletes the
+  unpublished fork only after proving it is still exactly at the inherited
+  version and no other sidecar claims `(table_path, target ref)`. A no-effect
+  sidecar with a competitor
   discards only itself; the last survivor either cleans the untouched ref or
   recovers its owned effect. Partial rollback performs no-effect cleanup before
   publishing its fixed outcome. This closes the cross-handle live-ref
   deletion bug and keeps cleanup as the backstop for truly unclaimed refs.
-  The remaining multi-process gap is narrower but real: Lance exposes
+  The remaining multi-process gap is narrower but real: Lance exposes neither
-  `force_delete_branch`, not compare-and-delete by `BranchIdentifier`, so a
+  conditional native graph-branch create/delete nor compare-and-delete by
-  foreign process can create intent/ref between the final list/check and the
+  `BranchIdentifier` for per-table refs, so a foreign process can mutate a ref
-  delete. The documented single-writer-process support boundary remains until
+  between the final list/check and the operation. The documented
-  Lance provides a conditional ref primitive (or OmniGraph adds a distributed
+  single-writer-process support boundary remains until Lance provides a
-  fence); process-local queues are not credited as that primitive.
+  conditional ref primitive (or OmniGraph adds a distributed fence);
+  process-local queues are not credited as that primitive.
 - **Local `write_text_if_match` is not a cross-process CAS:** object-store
   backends use a true conditional put (ETag If-Match; the in-memory test
   backend too), but upstream `object_store` leaves `PutMode::Update`

docs/dev/lance.md

@@ -198,6 +198,20 @@ findings:
   the substrate enforcing what omnigraph's entry-owner resolution always did.
   Production opens by owner-resolved location (unaffected); the lazy-fork
   namespace test pins both the error and the owner-branch open.
+- **Native branch control remains two-phase at beta.15.**
+  `Dataset::create_branch` commits the shallow-cloned `tree/{branch}` dataset
+  before validating/writing authoritative `BranchContents`; its random
+  `BranchIdentifier` element is minted only in that second phase and cannot be
+  pre-minted through the public API. OmniGraph therefore validates first and
+  classifies an ambiguous result from exact parent metadata rather than
+  inventing an identifier. Delete removes `BranchContents` before tree cleanup
+  and exposes no compare-and-delete primitive. Slash-separated names share
+  nested physical directories; `force_delete_branch` deliberately leaves an
+  ancestor tree while a live path-child exists, so live graph names are
+  path-prefix-disjoint. The public format page does not currently spell out the
+  identifier field, so the pinned Rust shape remains load-bearing. Guard 9 in
+  `lance_surface_guards.rs` pins clone-only raw-create failure plus force reclaim;
+  `src/branch_control.rs` pins delete classification and JSON identity fencing.
 - **Native DirectoryNamespace churn** (#7222 removed
   `table_version_storage_enabled` + the `__manifest` version-storage
   experiment; #7176/#7191/#7234 rewrote manifest handling): the decoupling
@@ -287,7 +301,7 @@ Migration from Lance 4.0.0 → 6.0.1 landed in this cycle (DataFusion 52 → 53,
 - **Lance #6658 closed** (2026-05-14) but `DeleteBuilder::execute_uncommitted` did **not** ship in v6.0.1 — binary search across the release stream shows it first appears in `v7.0.0-beta.10` (the closing commits landed on main but didn't backport to the 6.x line). Tracked as MR-A: migrate `delete_where` to staged, retire the parse-time D2 mutation rule, extend recovery sidecar coverage. **Gated on the Lance v7.x bump**, not this PR. v7.0.0-rc.1 dropped 2026-05-21.
 - **Lance #6666 still open** (`build_index_metadata_from_segments` public): vector-index two-phase blocked; inline `create_vector_index` residual retained.
 - **Lance #6877 still open** (`MergeInsertBuilder` dup-rowid): PR #109's `SourceDedupeBehavior::FirstSeen` + `check_batch_unique_by_keys` precondition stay load-bearing.
-- **`Dataset::force_delete_branch`** (`branches().delete(name, force=true)`, dataset.rs:524) tolerates a missing branch-*contents* ref (vs plain `delete_branch`'s `RefNotFound`), but on the local store still errors `NotFound` if the branch `tree/` directory is fully absent (`remove_dir_all`'s NotFound is not caught for Lance's native error variant, refs.rs:526-549). Both variants still refuse a branch with referencing descendants (`RefConflict`). `TableStore::force_delete_branch` wraps this to be fully idempotent (tolerates already-absent). The single-authority branch-delete redesign uses it for orphan reclamation (eager best-effort reclaim + cleanup reconciler). Pinned by `lance_surface_guards.rs::force_delete_branch_semantics`. Branch delete is "flip the ref atomically, then `remove_dir_all(tree/{branch})`"; branch-exclusive data lives under `tree/{branch}/` so a drop reclaims it immediately without touching `main`.
+- **`Dataset::force_delete_branch`** (`branches().delete(name, force=true)`, dataset.rs:524) tolerates a missing branch-*contents* ref (vs plain `delete_branch`'s `RefNotFound`), but on the local store still errors `NotFound` if the branch `tree/` directory is fully absent (`remove_dir_all`'s NotFound is not caught for Lance's native error variant, refs.rs:526-549). Both delete variants still refuse a branch with referencing descendants (`RefConflict`). The current OmniGraph disposition for these still-present substrate behaviors is recorded in the beta.15 audit above.
 - **Lance blob-v2 `compact_files` bug** (no public issue found as of 2026-06): `compact_files` disables binary-copy for blob datasets and forces `BlobHandling::AllBinary` on the read side; the v2.1+ structural decoder then mis-counts column infos for the blob-v2 struct and fails with `Invalid user input: there were more fields in the schema than provided column indices / infos` (`lance-encoding/src/decoder.rs::ColumnInfoIter::expect_next`). This fails even a pristine uniform-V2_2 multi-fragment blob table; vector/list/scalar/ragged columns and mixed file versions all compact fine. Reads/queries use descriptor handling (`BlobHandling::default()`) and are unaffected. `optimize` skips blob-bearing tables behind `LANCE_SUPPORTS_BLOB_COMPACTION = false` (`db/omnigraph/optimize.rs`), reporting `SkipReason::BlobColumnsUnsupportedByLance`. Pinned by `lance_surface_guards.rs::compact_files_still_fails_on_blob_columns`, which turns red when the bug is fixed → flip the gate, remove the skip branch + the `maintenance.rs::optimize_skips_blob_table_and_reports_skip` skip assertions.
 
 Surface guards added: `crates/omnigraph/tests/lance_surface_guards.rs` (10 named guards; 5 runtime + 5 compile-only; plus the index-coverage work's `_compile_optimize_indices_signature` and `optimize_indices_extends_fragment_coverage`). Future Lance bumps re-run this file first as the smoke check. Two additional guards from the original plan deferred to follow-up (`manifest_cas_returns_row_level_contention_variant` needs full publisher-race harness; `table_version_metadata_byte_compatible_with_v4` needs `pub(crate)` reach extension).

docs/dev/rfc-022-027-architecture-review.md

@@ -37,17 +37,19 @@ The central architecture remains the right one:
   separate irreversible decisions and should keep separate evidence and format
   gates.
 
-The current blockers are boundary problems, not a reason to replace that core.
+The remaining open blockers are boundary problems, not a reason to replace that
-They concern crash classification, actual fencing, foreign-branch authority,
+core. They concern actual cross-process fencing, capability activation, format
-and capability activation.
+rollout, and public RFC lifecycle. The native branch crash classifier,
+foreign-source merge semantics, and shipped lazy-branch cleanup exposure found
+by this review are now dispositioned below.
 
 > 💬 **Second-pass verification (2026-07-11):** I independently re-verified this
 > ledger's two sharpest new substrate claims against the pinned checkout before
 > commenting: BLOCKER-01's two-phase branch create is confirmed in Lance's own
 > doc comment, and BLOCKER-04's lazy-branch exposure is confirmed
-> architecturally — and appears to be a live bug in the shipped `cleanup`, not
+> architecturally. Both findings produced shipped fixes on 2026-07-11 and their
-> only an RFC gap (see the comment there). Overall judgment: concur with this
+> durable dispositions are recorded below. Overall judgment: concur with this
-> ledger; three of its findings supersede corrections I applied to the RFCs on
+> ledger; three other findings supersede corrections applied to the RFCs on
 > 2026-07-11 (noted inline at BLOCKER-07, BLOCKER-11, and tightening 5).
 
 The latest revisions improved several important points and those changes should
@@ -98,7 +100,9 @@ sub-contract of an unaccepted RFC as a separately accepted decision.
 
 **Affected:** [RFC-022 §7](../rfcs/rfc-022-unified-write-path.md#7-native-graph-branch-ref-control-protocol)
 
-RFC-022 currently models branch create/delete as one native ref mutation whose
+**Status:** Closed in specification and implementation on 2026-07-11.
+
+At review time RFC-022 modeled branch create/delete as one native ref mutation whose
 completion is the visibility point. The pinned Lance implementation is more
 specific:
 
@@ -143,6 +147,21 @@ and between delete authority removal and directory cleanup.
 > upstream ask) addressed the *conditional-put* gap but not this crash
 > classification — the two dispositions compose; both are needed.
 
+> ✅ **Disposition (2026-07-11):** RFC-022 §7 now makes `BranchContents` the sole
+> logical authority, defines the bounded create/delete classifier, explicitly
+> rejects a graph-branch sidecar/lineage entry, and retains the
+> single-writer-process boundary. The implementation prevalidates names,
+> requires live graph names to be physical-path-prefix-disjoint, reclaims an
+> absent-ref clone before one bounded retry, accepts only a current-attempt
+> matching lost acknowledgement, and fences delete by exact identifier. Full
+> first-touch recovery also force-reclaims clone-only table residue under its
+> already-durable writer sidecar. Coverage includes flat and named-source
+> clone-only recovery, invalid-name-before-clone, lost acknowledgements,
+> absent-ref/tree-present delete, same-identifier error preservation,
+> recreated-identifier refusal, path-prefix collision, legacy no-effect
+> leaf-first delete, mixed-effect rollback fail-closed behavior, and no
+> manifest-version/lineage movement.
+
 ### BLOCKER-02 — create-if-absent ownership is not a fencing lease
 
 **Affected:** RFC-023 migration claim, RFC-024 migration claim, and
@@ -193,6 +212,9 @@ that takeover is refused until external fencing/death proof is established.
 [RFC-023 §6](../rfcs/rfc-023-key-conflict-fencing.md#6-retry-and-validation-semantics),
 and [RFC-026 §6](../rfcs/rfc-026-memwal-streaming-ingest.md#6-fold-protocol)
 
+**Status:** The RFC-022 mutation/load portion is closed in the shipped adapter
+as of 2026-07-11. RFC-023 and RFC-026 retain their own open dispositions.
+
 A retryable concurrent inserted-row-filter conflict is detected when a table
 transaction attempts to commit. (`WhenMatched::Fail` may instead report a
 pre-existing match during merge execution.) In a multi-table graph write or
@@ -217,10 +239,24 @@ attempt around that sidecar. For non-strict upsert, the barrier must resolve the
 sidecar before the bounded automatic whole-operation retry. Add a failpoint/race
 in which table N conflicts after table 1 has committed.
 
+> ✅ **RFC-022 disposition (2026-07-11):** mutation/load take the conservative
+> safe branch of this requirement. A pre-effect authority mismatch discards and
+> fully reprepares the attempt. Once the exact-effect sidecar is durable and the
+> commit phase begins, every Lance commit failure returns `RecoveryRequired` and
+> leaves that sidecar intact, even when Full recovery later proves that no table
+> effect landed. The adapter performs zero transparent Lance commit retries and
+> never prepares a new semantic attempt around unresolved ownership; the next
+> synchronous barrier classifies and resolves it first. Finalizing a proven-empty
+> intent and automatically retrying would improve ergonomics, but is not required
+> for safety. RFC-023's fenced key-conflict contract and RFC-026's MemWAL fold
+> protocol still need their adapter-specific resolutions.
+
 ### BLOCKER-04 — live graph branches need physical GC protection
 
 **Affected:** [RFC-025 §6](../rfcs/rfc-025-checkpoint-retention.md#6-cleanup-protocol-and-pruned-through-boundary)
 
+**Status:** Closed in the shipped baseline and RFC-025 on 2026-07-11.
+
 The cleanup root set includes live graph branches, but RFC-025 creates Lance
 tags only for named checkpoints. This is insufficient for lazy graph branches.
 A graph branch may store, in its `__manifest`, a foreign reference to an old
@@ -253,6 +289,18 @@ runs, and the lazy branch still opens and reads the exact pinned state.
 > mechanism but are history-trimming under an operator-confirmed policy; a
 > live branch's working state is not history.)
 
+> ✅ **Disposition (2026-07-11):** shipped cleanup now resolves main plus every
+> live non-system graph branch under schema → all-branch → all-table gates,
+> verifies every exact inherited-main version opens, and caps each dataset's
+> cutoff at the oldest such pin before the first table GC. It also derives
+> `keep=N` from Lance's actual ordered version list and refuses uncovered main
+> HEAD drift. Unclassifiable roots abort graph-wide; only later per-table GC
+> failures are fault-isolated. Regression coverage pins count- and time-based
+> retention, the oldest of multiple live pins, exact keep counts, graph-wide
+> fail-closed ordering, and uncovered drift. RFC-025 §6 now composes checkpoint
+> tags with this required lazy-branch floor rather than treating tags as a
+> replacement.
+
 ### BLOCKER-05 — durable-head OCC must compare the full head token
 
 **Affected:** [RFC-024 §5](../rfcs/rfc-024-durable-table-heads.md#5-atomic-publisher-contract)
@@ -382,6 +430,11 @@ retaining the public lifecycle status `Proposed`.
 
 ### BLOCKER-10 — do not put foreign-branch facts in an atomic `ReadSet`
 
+**Affected:** [RFC-022 §6.2](../rfcs/rfc-022-unified-write-path.md#62-branch-merge)
+
+**Status:** Closed in specification on 2026-07-11; full merge-adapter conversion
+remains rollout work rather than an architecture ambiguity.
+
 RFC-022 requires every `ReadSet` member to be arbitrated atomically by the
 publish CAS. A CAS on reserved main cannot arbitrate a row on a named source
 branch; a merge-target CAS cannot arbitrate a source-branch row.
@@ -400,6 +453,13 @@ Use the right category for each fact:
 
 Keep target-branch values that must remain stable in the atomic `ReadSet`.
 
+> ✅ **Disposition (2026-07-11):** RFC-022 §6.2 defines the exact captured source
+> commit/snapshot as an immutable effect precondition, not a member of the
+> target publisher's atomic `ReadSet`. A later source-head advance is harmless
+> under captured-source semantics; only a future latest-at-target-publish
+> contract would require a source fence through target CAS. The current bridge
+> remains conservatively stricter before effects while its full adapter lands.
+
 ### BLOCKER-11 — RFC-022 and no-heads MemWAL need coarse OCC
 
 **Affected:** [RFC-022 §10](../rfcs/rfc-022-unified-write-path.md#10-rollout-and-compatibility)
@@ -525,7 +585,9 @@ RFC set is merged:
 The review does not require all RFCs to land together. A safe order is:
 
 1. accept RFC-022 after branch-control recovery, coarse `graph_head` OCC, and
-   foreign-branch fact classification are explicit;
+   foreign-branch fact classification are explicit (all three are now
+   dispositioned; the public lifecycle decision in BLOCKER-09 still governs
+   what “accepted” means);
 2. accept and land the stable identity RFC/capability;
 3. accept RFC-023 once partial-effect retry and its format/fleet barrier are
    complete;
@@ -540,10 +602,8 @@ The review does not require all RFCs to land together. A safe order is:
 This ordering preserves the split's main benefit: a blocked performance
 optimization or research result does not hold correctness work hostage.
 
-> 💬 **Concur with the order; two sequencing notes (2026-07-11):**
+> 💬 **Order update (2026-07-11):** BLOCKER-04's shipped-cleanup prerequisite is
-> BLOCKER-04's escalation means step 5 has a prerequisite outside this list —
+> now complete and RFC-025 §6 incorporates its floor. Step 3's TIGHTENING-01
-> the live lazy-branch cleanup exposure needs its red regression test and fix
+> demotion still hinges on the partial-update filter check
-> in the shipped `cleanup` first, so RFC-025 builds on a correct baseline. And
-> step 3's TIGHTENING-01 demotion hinges on the partial-update filter check
 > noted there; if that check lands `None`, upstream symmetry returns to the
 > activation gate and RFC-023's timeline moves accordingly.

docs/dev/testing.md

@@ -19,14 +19,14 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
 | File | Covers |
 |---|---|
 | `end_to_end.rs` | Full init → load → query/mutate flow |
-| `branching.rs` | Branch create / list / delete, lazy fork |
+| `branching.rs` | Branch create/list/delete and lazy fork; native-control hardening includes main and named-source clone-only create recovery, invalid-name-before-clone, live path-prefix namespace rejection, legacy prefix-collision leaf-first delete, and delete/recreate first-write safety. The lower classifier truth cells (absent-ref/tree-present delete, same-identifier native refusal, recreated-identifier typed conflict with JSON details) live in `src/branch_control.rs` unit tests |
 | `merge_truth_table.rs` | Merge-pair truth table (MR-786): all 9×9 `(left_op, right_op)` cells from `{noop, addNode, removeNode, addEdge, removeEdge, setProperty, dropProperty, addLabel, removeLabel}`. Adding a new op to `OpVariant` forces a compile error in `build_case` until the new row + column are dispositioned. 36 executable cells run through real `branch_merge` with a structured oracle (`MergeOutcome` / `MergeConflictKind` + graph-state assert); 45 cells involving `dropProperty`/`addLabel`/`removeLabel` are recorded as `Unsupported` until the mutation grammar grows. |
 | `merge_fast_forward.rs` | Fast-forward branch-merge cost + correctness: an append-only adopted-source merge routes *new* rows through `stage_append` instead of one whole-delta `stage_merge_insert` (the full-outer join that exhausted the DataFusion memory pool on embedding-bearing tables). The regression gate is structural — it asserts WHICH staged-write primitive the merge invokes via the task-local write probes (`omnigraph::instrumentation`), not a brittle size threshold; also: merge yields source state, defers vector indexes to the reconciler, streams blob columns |
 | `writes.rs` | Direct-publish writes: cancellation, RFC-022 non-strict full-attempt reprepare from fresh branch authority, strict stale-write conflicts, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery); the lance#7444 row-id-overlap regression (`filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent` — merge-load → same-key merge-load → delete → keyed point lookup, green only under the vendored lance-table patch — plus its append-only control) |
 | `staged_writes.rs` | TableStore staged-write primitives (`stage_append`, `stage_merge_insert`, `commit_staged`, `scan_with_staged`, `count_rows_with_staged`) — primitive-level only; engine code uses the in-memory `MutationStaging` accumulator instead |
 | `forbidden_apis.rs` | Defense-in-depth source-walk guard: engine code (`exec/`, `db/omnigraph/`, `loader/`, `changes/`) must not reach around the sealed storage trait to Lance inline-commit APIs, nor open datasets directly (`Dataset::open` / `DatasetBuilder::from_uri`/`from_namespace`) — reads route through `Snapshot::open` and the held-handle cache; `// forbidden-api-allow: <reason>` sentinel exempts reviewed lines |
 | `failpoint_names_guard.rs` | Source-walk guard (same defense-in-depth shape as `forbidden_apis.rs`): every failpoint call site (`maybe_fail`, `ScopedFailPoint::new`/`with_callback`, `Rendezvous::park_first`) must reference a `failpoints::names` const, never a bare string literal — a typo'd literal compiles and silently never fires |
-| `lance_surface_guards.rs` | Pins the Lance API surfaces omnigraph depends on (named runtime + compile-only guards; see [lance.md](lance.md)) — the first smoke check on any Lance version bump; e.g. `compact_files_succeeds_on_blob_columns` pins blob-v2 compaction working (a red means a Lance regression) |
+| `lance_surface_guards.rs` | Pins the Lance API surfaces omnigraph depends on (named runtime + compile-only guards; see [lance.md](lance.md)) — the first smoke check on any Lance version bump; e.g. `compact_files_succeeds_on_blob_columns` pins blob-v2 compaction working, while Guard 9 proves an unlisted clone-only branch blocks raw create and `force_delete_branch` reclaims it, but also pins Lance's intentional no-tree-cleanup behavior when a slash-name path-child is live (a red means a Lance regression) |
 | `warm_read_cost.rs` | Cost-budget tests for the warm read path (query-latency work), measured at the object-store boundary with Lance `IOTracker` (the LanceDB IO-counted pattern): a warm same-branch read does 0 manifest opens, 1 version probe, validates the schema once (Fix 1 / finding A / Fix 2 at commit-history depth); stale same-branch reads perform exactly 2 probes and refresh manifest-only; recreated non-main branches with the same Lance version refresh by incarnation; recreated branch-owned table handles are distinguished by table e_tag or refresh-time cache clearing; recreated traversal topology is protected by per-edge-table e_tag in the graph-index cache key or refresh-time cache clearing; a warm *repeat* read does 0 table opens via the held-handle cache and a write re-opens only the changed table at its new version/e_tag (Fix 3/6A). Also the CSR topology-build cost guards: `fresh_branch_traversal_reuses_main_graph_index` (A1 — a lazy-fork branch reuses main's cached CSR index, 0 rebuilds via `graph_build_count`) and `single_edge_query_builds_only_referenced_edge` (A2 — a one-edge query builds only that edge via `graph_edges_built`); both force CSR via the scoped `with_traversal_mode` seam, so they need no `#[serial]`. See "Cost-budget tests" below. |
 | `write_cost.rs` | Cost-budget tests for the WRITE path (RFC-013), the latency twin of `warm_read_cost.rs` on the **shared `helpers::cost` harness** (`measure`/`IoCounts`/`assert_flat`/`local_graph`). Runs on **local FS**; gates the **internal-table** term (`__manifest` scans flat in commit-history depth, lineage rows included — `internal_table_scans_are_flat_in_history`, now **green every-PR** since RFC-013 step 2 brought the internal tables into `optimize`; the test compacts at each depth before measuring) plus green every-PR guards (single-insert `data_writes` bounded, a per-write read-op ceiling that fails the moment a round-trip is added, and a `measure_with_staged` fitness assert that a keyed insert routes through `stage_merge_insert` once with no `stage_append`/vector-index build). The **data-table opener** term is S3-only — see `write_cost_s3.rs` and the backend-split note in "Cost-budget tests" below |
 | `write_cost_s3.rs` | Bucket-gated (skips without `OMNIGRAPH_S3_TEST_BUCKET`) twin of `write_cost.rs` on the same `helpers::cost` harness: gates the **data-table opener** term (per-write latest-version resolution flat across commit depth on a real object store — per-version GETs are invisible on local FS). A cost gate, not a correctness test — run on demand, not in the every-merge `rustfs_integration` job (see the backend-split note in "Cost-budget tests" below) |
@@ -51,8 +51,8 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
 | `validators.rs` | Schema constraint enforcement (enum, range, unique, cardinality) across JSONL load, mutation insert/update. ALL THREE write surfaces — mutation, bulk load, AND merge — route through the unified `crate::validate` evaluator (Δ-scoped, index-backed, reusing these leaf checks). Cross-version-uniqueness closure: `cross_version_unique_rejected_on_mutation_insert` + `reinsert_existing_key_is_upsert_not_unique_violation` (mutation path); `cross_version_unique_rejected_on_append_load` + `merge_load_reupsert_existing_key_is_not_unique_violation` (load path). Per-table `Overwrite`: `overwrite_load_validates_ri_against_new_image` (an edges-only overwrite still resolves RI against retained committed nodes) + `append_load_rejects_orphan_edge`. The evaluator's own unit tests live in `src/validate.rs` (`#[cfg(test)]`); its merge-conflict equivalence is pinned by `merge_truth_table.rs` (OrphanEdge) + `branching.rs` (Unique/Cardinality merge tests). Intra-batch duplicate-`@key` rejection on every load mode is pinned by `consistency.rs::loader_rejects_intra_batch_duplicate_keys`; the mutation-coalesce counterpart (insert+update / chained updates of one id are NOT a self-collision) by `writes.rs`. Non-String `@unique` columns probe committed state with a TYPED literal (not a stringified key): `cross_version_unique_rejected_on_date_column` + `noncolliding_write_to_date_unique_column_succeeds` (a `Date @unique` collision is a proper `@unique` violation, and a distinct value does not raise a Date32-vs-Utf8 coercion error). Cardinality is keyed by edge id, last-wins (matching commit's `dedupe_merge_batches_by_id`): `merge_load_edge_src_move_rechecks_vacated_src_cardinality` (a Merge-load moving an edge recounts the vacated src for `@card` min) + `merge_load_duplicate_edge_id_counts_once_per_card` (a dup edge id under two srcs in one batch counts once, no spurious max violation). Direct deletes capture the ids they remove (from the delete op's own scan) into the change-set's `deleted_ids`, so a delete emptying a src is validated: `mutation_delete_edge_below_card_min_rejected` (a `delete Edge` dropping a src below `@card` min is rejected, not silently committed). |
 | `merge_cost.rs` | Cost-budget tests for branch MERGE on the shared `helpers::cost` harness: `merge_validation_is_delta_scoped` (a 1-row-delta merge opens ≤3 data tables — Δ-scoped, not the whole catalog; was ~6 pre-#5) and `merge_manifest_cost_grows_with_history` (the cross-branch `__manifest` open amplification still grows with commit depth — a separate, not-yet-addressed term — while validation `data_open_count` stays flat) |
 | `policy_engine_chassis.rs` | Engine-layer Cedar enforcement (MR-722): allow + deny through every `_as` writer via the SDK directly — no HTTP — proving embedded and CLI callers hit the same gate as the server, with action × scope shapes matching `authorize_request` |
-| `maintenance.rs` | `optimize` (compaction), `repair` (explicit uncovered-drift publish), and `cleanup` (version GC): empty/idempotent/no-op edges, policy validation, head preservation; `optimize` publishes its own compaction (`optimize_publishes_compaction_to_manifest_so_schema_apply_succeeds`), skips pre-existing uncovered drift (`optimize_skips_preexisting_manifest_head_drift`), refuses to run while a `__recovery` sidecar is pending (`optimize_defers_when_recovery_sidecar_is_pending`), and compacts blob-v2 tables through the normal path (`optimize_compacts_blob_table_alongside_plain_table`); `repair` previews/heals verified maintenance drift, refuses raw semantic drift without `--force`, and forced repair publishes only by explicit operator choice; the index reconciler (iss-848): `index_build_tolerates_null_vector_rows` (logical load succeeds without inline index work; an untrainable Vector column then defers during reconciliation) and `optimize_materializes_index_declared_but_unbuilt` (optimize creates a declared-but-deferred index) |
+| `maintenance.rs` | `optimize` (compaction), `repair` (explicit uncovered-drift publish), and `cleanup` (version GC): empty/idempotent/no-op edges, policy validation, head preservation; cleanup pins exact keep-count behavior (including keep larger than history), count/time retention of a live lazy branch, the oldest of multiple lazy pins, graph-wide fail-closed ordering on an unopenable pin, and refusal of uncovered main HEAD drift before any GC; `optimize` publishes its own compaction (`optimize_publishes_compaction_to_manifest_so_schema_apply_succeeds`), skips pre-existing uncovered drift (`optimize_skips_preexisting_manifest_head_drift`), refuses to run while a `__recovery` sidecar is pending (`optimize_defers_when_recovery_sidecar_is_pending`), and compacts blob-v2 tables through the normal path (`optimize_compacts_blob_table_alongside_plain_table`); `repair` previews/heals verified maintenance drift, refuses raw semantic drift without `--force`, and forced repair publishes only by explicit operator choice; the index reconciler (iss-848): `index_build_tolerates_null_vector_rows` (logical load succeeds without inline index work; an untrainable Vector column then defers during reconciliation) and `optimize_materializes_index_declared_but_unbuilt` (optimize creates a declared-but-deferred index) |
-| `failpoints.rs` | Failure-injection coverage (gated on `failpoints` feature). RFC-022 includes deterministic post-stage/pre-effect races for mutation/load uniqueness and strict disjoint-head changes, plus the cross-handle post-effect `RecoveryRequired` → read-write-open rollback cell. Control/recovery race cells include `first_touch_post_create_open_error_keeps_recovery_ownership`, `branch_delete_orphans_sidecar_armed_after_initial_barrier`, `branch_merge_fences_target_delete_recreate_aba`, `branch_merge_fences_concurrent_sync_on_same_handle`, `branch_merge_rejects_fresh_target_manifest_change_before_effects`, `branch_merge_rechecks_late_sidecar_after_table_gates`, `cleanup_rechecks_sidecars_under_gc_gates`, and `full_recovery_rereads_sidecar_body_after_discovery`. The suite also includes the five per-writer effect → manifest-CAS recovery tests, write-entry in-process heal contract, storage-fault matrix, S3 recovery twin, and convergence-idempotent roll-forward regression. |
+| `failpoints.rs` | Failure-injection coverage (gated on `failpoints` feature). RFC-022 includes deterministic post-stage/pre-effect races for mutation/load uniqueness and strict disjoint-head changes, plus the cross-handle post-effect `RecoveryRequired` → read-write-open rollback cell. Native controls are pinned by `native_branch_controls_reclassify_lost_acknowledgements` (matching create and absent-ref delete, with no version/lineage movement); `armed_first_touch_recovery_accepts_missing_target_ref` additionally forges and reclaims the clone-only/no-`BranchContents` table state. Legacy path overlap has both sides pinned: `armed_first_touch_recovery_defers_legacy_path_overlap_until_leaf_delete` permits open only for a proven no-effect intent, while `partial_first_touch_recovery_fails_closed_on_legacy_path_overlap` leaves one exact multi-table effect and verifies open fails closed until offline leaf cleanup lets rollback converge. Other control/recovery race cells include `first_touch_post_create_open_error_keeps_recovery_ownership`, `branch_delete_orphans_sidecar_armed_after_initial_barrier`, `branch_merge_fences_target_delete_recreate_aba`, `branch_merge_fences_concurrent_sync_on_same_handle`, `branch_merge_rejects_fresh_target_manifest_change_before_effects`, `branch_merge_rechecks_late_sidecar_after_table_gates`, `cleanup_rechecks_sidecars_under_gc_gates`, and `full_recovery_rereads_sidecar_body_after_discovery`. The suite also includes the five per-writer effect → manifest-CAS recovery tests, write-entry in-process heal contract, storage-fault matrix, S3 recovery twin, and convergence-idempotent roll-forward regression. |
 | `failpoint_names_guard.rs` | Source-walk guard: every `maybe_fail(...)` call site (engine + cluster) must reference the compile-checked `failpoints::names` consts, never a bare string literal — a typo'd literal compiles but silently never fires; same defense-in-depth shape as `forbidden_apis.rs` |
 | `recovery.rs` | Open-time recovery sweep — legacy sidecars plus schema-v3 Mutation/Load exact transaction identity, fixed logical/rollback outcome IDs, branch-token comparison, fresh under-gate sidecar reread/reparse, all-or-nothing roll-forward/rollback/refusal, audit row in `_graph_commit_recoveries.lance`, and `OpenMode::ReadOnly` skip path |
 | `composite_flow.rs` | Compositional/narrative end-to-end stories — multi-step flows that compose mechanics covered by other test files. Catches integration regressions where individual operations all pass their unit tests but their composition breaks (sequential merges, post-merge main writes, time-travel through merge DAG, reopen consistency over multi-merge histories, post-optimize and post-cleanup strict writes). |

docs/dev/writes.md

@@ -30,13 +30,16 @@ authority.
   never created without ownership: the schema-v3 sidecar is durable first and
   names that `(table_path, target ref)`. Reclaim and `cleanup` treat any
   matching pending sidecar as a hard stop. Quiesced full recovery accepts both
-  crash shapes — sidecar durable with no ref yet, or an exact untouched ref at
+  logical crash shapes — sidecar durable with no ref yet, or an exact untouched
-  the inherited version — and removes the latter before deleting the empty
+  ref at the inherited version. Because Lance creates a branch dataset before
-  intent. If several pending intents claim one ref, a no-effect intent discards
+  writing its authoritative `BranchContents`, the first shape may still contain
-  only itself while any competitor remains; the last no-effect survivor cleans
+  a clone-only tree; recovery force-reclaims that absent-ref tree idempotently.
-  an untouched ref, or the effect-owning survivor recovers normally. `cleanup`
+  It removes an exact untouched ref before deleting the empty intent. If several
-  remains the backstop for genuinely unclaimed legacy/stale refs; see the
+  pending intents claim one ref, a no-effect intent discards only itself while
-  fork-reclaim note in [invariants.md](invariants.md).
+  any competitor remains; the last no-effect survivor cleans an untouched ref,
+  or the effect-owning survivor recovers normally. `cleanup` remains the
+  backstop for genuinely unclaimed legacy/stale refs; see the fork-reclaim note
+  in [invariants.md](invariants.md).
 
 ## Mutation/load coarse OCC (RFC-022 first adapter)
 
@@ -87,6 +90,14 @@ source and target, re-lists sidecars, and compares fresh source/target manifest
 versions with the captured snapshots. A stale warm handle catalog or coordinator
 snapshot is never accepted as that revalidation.
 
+The source snapshot is a captured merge input, not authority that the target
+manifest CAS can arbitrate. The current process-local source gate is a stronger
+same-process fence around that capture, including delete/recreate ABA, but the
+semantic contract is still "merge the captured source commit." A later source
+advance does not invalidate an otherwise prepared target publish. Claiming
+"latest source at target publish" would instead require a cross-process source
+fence held through the target CAS.
+
 That fence prevents a same-process target delete/recreate from reusing the branch
 name underneath a merge plan. The race test deliberately recreates a target with
 the same name and numeric Lance version but a different `BranchIdentifier`, so
@@ -109,6 +120,47 @@ records the orphan-discard recovery audit and deletes the sidecar. A
 is specific to removing the authority that made the intent reachable; create,
 merge, mutation, and load still reject relevant unresolved ownership.
 
+### Native graph-branch control recovery
+
+Graph branch create/delete do not use the graph-visible table-effect sidecar or
+emit graph lineage. Their sole logical authority is Lance `BranchContents` for
+the `__manifest` dataset, and Lance mutates that authority in two physical
+phases:
+
+- create shallow-clones `tree/{branch}` before writing `BranchContents`;
+- delete removes `BranchContents` before reclaiming that tree.
+
+Under the schema/branch/table control gates, create validates the name before
+the clone and rejects a live graph name that is a physical path ancestor or
+descendant of another live name. It then force-reclaims any absent-ref same-name
+tree and performs at most two native attempts. An ambiguous result is accepted
+only when fresh metadata has
+the captured parent branch/version/incarnation plus exactly one new identifier
+element and the target opens. Foreign or broken authoritative refs are never
+deleted. Delete captures the exact target identifier; after an ambiguous error,
+an absent ref is logical success, the same identifier preserves the original
+error, and a different identifier is a typed delete/recreate conflict. Derived
+tree cleanup is retried best-effort.
+
+There is deliberately no branch-control sidecar: within the supported
+single-writer-process topology, an absent ref makes a same-name tree unreachable
+garbage; the path-prefix-disjoint namespace is what makes Lance's recursive
+force cleanup exact. Same-name create is therefore the targeted reconciler.
+First-touch data-table
+forks remain sidecar-owned because they are physical effects of a graph-visible
+mutation/load. Lance does not expose conditional ref create/delete, so this
+classifier is not advertised as a cross-process branch-control fence.
+
+Legacy prefix-overlap recovery is the one first-touch case that does not prove an
+entire nested tree unreachable. If a Full sweep finds an ancestor first-touch
+target with a live path-child, it keeps the sidecar. Open may complete for
+leaf-first deletion only when the sidecar owns no physical table effect. A mixed
+attempt that owns an effect plus an untouched fork must roll back as one recovery
+outcome, so open fails closed while the child blocks fork cleanup. After an
+existing handle or an offline Lance-level branch tool removes the child, a later
+Full sweep reclaims the untouched fork, compensates the owned effect, and retires
+the intent.
+
 ## Read-your-writes within a multi-statement mutation
 
 A `.gq` query with multiple ops (e.g. `insert Person … insert Knows …`)
@@ -328,12 +380,17 @@ recovery sweep in `crates/omnigraph/src/db/manifest/recovery.rs`:
   rollback-only; an unknown/foreign effect is refused rather than adopted.
   An Armed first-touch intent with no owned transaction is deferred by live
   roll-forward-only healing because another handle may still own it. Quiesced
-  full recovery tolerates an absent target ref (crash before fork), or removes
+  full recovery tolerates an absent target ref (either crash before clone or a
-  an exact unchanged unpublished ref after proving no other pending sidecar
+  clone-only tree with no `BranchContents`) and force-reclaims that absent-ref
-  claims it. With competing claims, the current no-effect sidecar discards
+  target idempotently. If an authoritative ref exists, recovery removes it only
-  itself without touching the ref; the final survivor owns cleanup/recovery.
+  when it is exactly unchanged and no other pending sidecar claims it. With
+  competing claims, the current no-effect sidecar discards itself without
+  touching the ref; the final survivor owns cleanup/recovery.
   During partial rollback, no-effect refs are removed before the rollback
-  outcome is published so a retry cannot strand them.
+  outcome is published so a retry cannot strand them. If a legacy live
+  path-child blocks that cleanup, rollback returns an error and read-write open
+  fails closed; only a sidecar proven to own no table effect may defer cleanup
+  while returning an open handle.
 - If any table is `InvariantViolation` (Lance HEAD < manifest pinned —
   should be impossible), **abort** with a loud error and leave the
   sidecar on disk for operator review.

docs/rfcs/rfc-022-unified-write-path.md

@@ -446,7 +446,15 @@ able to enumerate every adapter and every entry point that invokes it.
 
 ### 6.2 Branch merge
 
-- Compute row classification outside the gates.
+- Capture the exact source graph commit/snapshot and compute row classification
+  against that immutable input outside the effect gates. The source commit is an
+  effect precondition, not a member of the target publisher's atomic `ReadSet`:
+  a target CAS cannot arbitrate a foreign source-branch row.
+- Revalidate that the captured source incarnation and commit still identify the
+  prepared input before effects. A later source-head advance is intentionally
+  harmless: merge means "merge this captured source commit," not "whatever is
+  latest when the target publishes." A future latest-at-publish contract would
+  require a real source-branch fence held through target CAS.
 - Include the target graph head and every target table used by classification or
   validation in `ReadSet`.
 - Any target change before effects forces a complete reclassification; publishing a
@@ -495,15 +503,82 @@ There is no target branch on which to publish before create, and no target remai
 which to publish after delete. They therefore cannot truthfully be instances of the
 graph-visible manifest-CAS protocol.
 
+The native control is not one physical mutation. At the pinned Lance revision:
+
+- create first commits a shallow-cloned branch dataset under `tree/{branch}` and
+  then writes `BranchContents`; the two phases are non-atomic and
+  `BranchContents` is the sole logical authority;
+- delete removes `BranchContents` first and then reclaims the branch directory.
+
+Accordingly, a clone without `BranchContents` is unreachable physical garbage,
+while an absent `BranchContents` after delete means the logical delete succeeded
+even if directory cleanup returned an error.
+
+Lance maps slash-separated branch names into nested physical directories and will
+not reclaim an ancestor directory while a live descendant name exists. Live graph
+branch names are therefore path-prefix-disjoint: `review/2026` is valid, but it
+cannot coexist with a live `review` branch. This is checked before native create,
+and legacy overlapping names must be deleted leaf-first. Without this namespace
+invariant, `force_delete_branch` may return success while deliberately leaving an
+ancestor clone-only dataset in place.
+
 Their control protocol is:
 
 1. run and await the recovery barrier;
 2. quiesce enrolled streams as required by RFC-026;
 3. acquire any active global claim, such as RFC-025's retention claim, and then
    the graph/branch-control gate in §4.3 order;
-4. freshly revalidate source ref, target existence, and branch incarnation;
+4. freshly capture source ref/version/incarnation and target absence (or the
-5. perform one native Lance ref mutation, which is the visibility point;
+   delete target's exact `BranchIdentifier`);
-6. release the gate and reclaim orphaned per-table forks asynchronously.
+5. validate a create name and the path-prefix-disjoint namespace before Lance's
+   shallow-clone phase;
+6. execute the bounded native classifier below; and
+7. reclaim owned per-table forks best-effort while the complete control-gate set
+   remains held, then release the gates. A reclaim failure is logged for the
+   cleanup reconciler; this implementation does not schedule asynchronous reclaim.
+
+| Operation | Fresh physical state | Classifier action |
+|---|---|---|
+| create | no ref, no clone | idempotent force-reclaim (no-op), then create |
+| create | clone only | force-reclaim the unreachable clone, then create |
+| create | matching ref + openable clone observed after this invocation's ambiguous native result | complete; accept a lost acknowledgement |
+| create | mismatching/broken ref | conflict; never adopt or delete it |
+| delete | captured identifier still present | not deleted; return the native error |
+| delete | ref absent, directory remains | logically deleted; retry reclaim best-effort |
+| delete | ref and directory absent | complete |
+| delete | different identifier present | delete/recreate ABA; conflict, never delete it |
+
+Create retries the native call at most once after reclaiming an absent-ref tree.
+It never adopts a matching ref that was already present before the invocation;
+that remains ordinary `AlreadyExists` because there is no operation marker.
+Matching completion requires the expected parent branch and version plus a target
+identifier whose mapping is exactly the captured parent identifier followed by one
+new `(parent_version, uuid)` element; the target dataset must also open. Lance does
+not let the caller pre-mint that UUID, so the proof is intentionally scoped to the
+supported single-writer-process boundary.
+
+No separate graph-branch sidecar is written. Under the held target gate,
+path-prefix-disjoint namespace, and supported topology, absent `BranchContents`
+means there is no logical branch and any same-name tree is safe derived garbage;
+the next same-name create is the targeted, idempotent reconciler. A recovery
+sidecar would introduce a second authority and would incorrectly pull native
+controls into graph lineage. First-touch **data-table** forks are different: their
+mutation/load sidecar is already durable before branch creation. Full recovery
+force-reclaims an absent-ref target as either a no-op (crash-before-clone) or that
+intent's clone-only zombie before retiring the intent.
+
+One compatibility case deliberately defers that cleanup. A legacy graph may
+already contain path-prefix-overlapping live names. If an unresolved first-touch
+intent targets an ancestor table tree while a live path-child still exists, Full
+recovery leaves the sidecar durable instead of recursively deleting the child's
+storage. Read-write open may complete for leaf-first remediation only when the
+intent owns no physical table effect. If a multi-table attempt owns any effect
+while another untouched fork is blocked by the child, rollback is
+complete-or-error: open fails closed, the sidecar remains authoritative, and an
+existing handle or offline Lance-level branch tool must remove the descendant
+before the next Full recovery reclaims the untouched fork, compensates the owned
+effect, and retires the sidecar. This distinction prevents legacy writers outside
+the v3 barrier from preparing against an unresolved partial rollback.
 
 Delete has one recovery disposition that create does not: after the complete
 schema/target-branch/all-table gate set has waited out any live in-process owner,
@@ -515,13 +590,14 @@ create/merge may not adopt this exception.
 The native ref operation itself should enforce the freshly checked precondition or
 surface concurrent ref mutation as a conflict — but at the pinned Lance revision it
 does not: branch-ref creation is an existence check followed by an unconditional
-put, not a conditional primitive (the same fact for which RFC-025 §2.3 rejects a
+put, and delete is not compare-and-delete by `BranchIdentifier` (the same substrate
-branch ref as a claim mechanism). Until Lance ships a conditional/CAS ref mutation,
+gap for which RFC-025 §2.3 rejects a branch ref as a claim mechanism). Until Lance
-graph-branch create/delete therefore inherit the documented single-writer-process
+ships conditional/CAS ref mutation, graph-branch create/delete therefore inherit the
-support boundary — the same disposition RFC-023 §10 applies to recovery ownership —
+documented single-writer-process support boundary — the same disposition RFC-023
-and multi-process branch operations are not advertised. The upstream ask for a
+§10 applies to recovery ownership — and multi-process branch operations are not
-conditional ref primitive is filed alongside this RFC; a process-local branch gate
+advertised. The upstream ask for a conditional ref primitive is filed alongside
-remains a local optimization, not the missing cross-process guarantee.
+this RFC; a process-local branch gate remains a local optimization, not the missing
+cross-process guarantee.
 
 These operations do not emit a synthetic graph commit. If a future product contract
 requires a native ref mutation and manifest/audit rows to become atomic together, it
@@ -628,6 +704,16 @@ Implementation proceeds in this order:
    > its full exact-effect adapter remains future work. Schema apply,
    > optimize/index, and MemWAL fold remain on their writer-specific paths until
    > their adapter slices land.
+
+   > **Branch-control/cleanup slice (2026-07-11):** native graph-branch create
+   > and delete now use the §7 authority classifier, including pre-clone name
+   > validation, prefix-disjoint live names, bounded clone-only recovery,
+   > lost-acknowledgement classification, exact delete-incarnation fencing, and
+   > no synthetic graph lineage. First-touch Full recovery also reclaims an
+   > absent-ref clone-only table fork. Cleanup now protects every exact lazy
+   > branch pin, computes `keep` from Lance's actual version list, refuses
+   > uncovered main HEAD drift, and completes its live-root preflight before the
+   > first table GC.
 3. Convert mutation/load, branch merge, schema apply/migration, data-table optimize,
    and graph-visible index work one adapter at a time.
 4. Add static or runtime enumeration proving no graph-visible entry point bypasses the
@@ -661,6 +747,10 @@ writers.
 - An edge insert racing deletion of an endpoint must revalidate or conflict.
 - Cardinality probes of an untouched table participate in arbitration.
 - A target advance after merge classification forces complete reclassification.
+- A source branch advancing after capture does not silently substitute its new
+  head: the merge either uses the captured source commit or fails its pre-effect
+  source-incarnation/commit check. A latest-at-target-publish variant requires a
+  real source fence and is not inferred from the target `ReadSet`.
 - Run the same cells with separate `Omnigraph` handles sharing one root-scoped
   process-local gate manager, then with separate processes that do not share it.
 
@@ -685,6 +775,19 @@ writers.
   retryable physical contention.
 - MemWAL fold, when RFC-026 lands: merged-generation conflict and every fold crash
   boundary.
+- Native graph branch control: invalid name before clone; live path-prefix
+  collision before clone; clone-only create recovery on main and a named source;
+  pre-existing matching ref remains `AlreadyExists`; current-call lost create
+  acknowledgement is accepted only with exact parent/incarnation metadata and an
+  openable target; delete with absent ref plus remaining tree is success and
+  reclaims best-effort; same identifier preserves the native error; a recreated
+  identifier is a typed conflict and survives; neither direction advances a
+  manifest version or emits graph lineage.
+- First-touch recovery with legacy path overlap: a proven no-effect intent may
+  defer clone-only ancestor cleanup and return an open handle for leaf-first
+  remediation; a mixed multi-table intent with one owned effect and one blocked
+  untouched fork must fail read-write open, retain the sidecar without restoring
+  or publishing, and converge after offline Lance-level leaf cleanup.
 
 ### 11.5 Cost gates
 

docs/rfcs/rfc-025-checkpoint-retention.md

@@ -274,6 +274,16 @@ The cleanup root set is:
 - every version inside the operator-selected time/count retention window;
 - versions Lance itself protects through non-OmniGraph tags or branch refs.
 
+Checkpoint tags do not physically protect a lazy graph branch whose table row
+still points at an exact version on that data table's `main`: Lance cannot see
+the foreign `__manifest` reference. For each physical main dataset, cleanup
+therefore caps `before_version` at the oldest exact inherited-main version among
+all live graph branches. Native per-table branch refs remain Lance-protected and
+do not need duplicate tags. Failure to resolve or open any live root aborts the
+graph-wide preflight before the first table GC. The current shipped baseline also
+requires each manifest-visible main version to equal Lance HEAD; uncovered drift
+must be repaired before retention work.
+
 Read-only preview may run without a claim, but it is explicitly provisional.
 Confirmed execution uses this order:
 
@@ -283,7 +293,8 @@ Confirmed execution uses this order:
 2. acquire the retention claim, then revalidate the fleet outage, every stream's
    `SEALED` cut, and absence of recovery sidecars;
 3. run pin reconciliation and recompute the exact root set and per-dataset
-   cutoffs under the claim;
+   cutoffs under the claim, including the oldest live lazy-branch inherited-main
+   floor above;
 4. prepare and revalidate a complete RFC-022 `ReadSet` containing format/schema
    identity, branch incarnations, current manifest table entries/heads, prior GC
    boundaries, and the root digest;

docs/user/branching/index.md

@@ -6,12 +6,34 @@ Lance supports branching at the dataset level: a branch is a named lineage of ve
 
 ## L2 — Graph-level branches
 
-OmniGraph builds *graph branches* on top by branching every sub-table coherently:
+OmniGraph builds *graph branches* on top with one authoritative `__manifest`
+ref whose table entries form a coherent graph snapshot; data-table forks are
+created lazily on first write:
 
-- **Create** (`branch create` / `branch create --from <target>`) — the name `main` is disallowed; fails if the branch exists. Atomic: the new branch becomes visible all-or-nothing, so a name never half-exists.
+- **Create** (`branch create` / `branch create --from <target>`) — the name `main` is disallowed; fails if the branch exists. Logically atomic: the branch becomes visible only when its authoritative ref exists, so a name never half-exists in graph reads. If storage interruption leaves only Lance's unreachable shallow-clone directory, the next same-name create reclaims it and retries automatically.
 - **List** (`branch list`) — returns public branches, **filtering the internal** `__schema_apply_lock__` branch.
-- **Delete** (`branch delete`) — refuses if there are descendants on the branch, or if it is the current branch. Once deleted, the branch is gone from every snapshot. The owned per-table forks are reclaimed best-effort; if that reclaim hits a transient object-store error, the leftover storage is reclaimed later by the [`cleanup`](../operations/maintenance.md) command. One consequence: if a delete's reclaim fails, reusing that branch name before the next `cleanup` surfaces a clear error pointing at `cleanup`.
+- **Delete** (`branch delete`) — refuses if there are descendants on the branch, or if it is the current branch. Once its authoritative ref is removed, the branch is gone from every snapshot even if reclaiming a now-unreachable storage directory needs a later retry. Owned per-table forks are reclaimed best-effort; same-name create/first write safely reconciles relevant leftovers, and [`cleanup`](../operations/maintenance.md) remains the general backstop.
 - **Lazy forking**: a branch only forks a sub-table when that sub-table is first mutated on it. Pure-read branches share storage with their source. If two writers race to first-write the same branch, the loser gets a retryable "refresh and retry".
+- **Names are path-prefix-disjoint while live.** Slash-separated names are
+  supported (`review/2026-07`), but `review` and `review/2026-07` cannot both be
+  live because Lance stores them in overlapping physical directories. Choose
+  sibling names, or delete the existing ancestor/descendant first.
+
+Graph branch create/delete are coordinated across handles in one writer
+process. Until Lance exposes conditional native ref mutation, separate writer
+processes must not concurrently control branches on the same graph.
+
+For a legacy graph that already contains path-prefix-overlapping live names,
+recovery also preserves the leaf-first escape hatch. If read-write open finds an
+unresolved first-touch sidecar for an ancestor table fork while a live path-child
+remains, it never deletes the child's storage. When the interrupted write owns no
+table effect, it leaves the sidecar in place and completes the open so you can
+delete the descendant branch leaf-first. When the same sidecar owns a partial
+table effect, open fails closed because cleanup and rollback must finish together;
+remove the descendant through an already-open handle or an offline Lance-level
+branch tool, then reopen. The next read-write open reclaims the ancestor residue,
+rolls back the partial effect, and retires the sidecar. `omnigraph repair` is not
+that offline tool: it correctly refuses to run while a recovery sidecar is pending.
 
 ## L2 — Commit graph
 

docs/user/operations/maintenance.md

@@ -30,28 +30,53 @@
 - Garbage-collects old versions per table.
 - Removes versions (and their unique fragments) older than the retention policy.
 - Policy options `keep_versions` and `older_than` — at least one is required.
+  `keep_versions=N` derives its requested cutoff from the newest `N` available
+  versions per table (the current HEAD is always retained, including for
+  `N=0`); live-branch safety floors may retain additional intervening versions.
+  When both options are set, a version is removed only if it is older than both
+  cutoffs.
 - Returns per-table stats: `table_key, bytes_removed, old_versions_removed, error`.
-- **Fault-isolated per table.** A single table's transient failure (version GC or
+- **Fault-isolated per table after the graph-wide safety preflight.** A single
-  orphan reclaim) is recorded on that table's stats row (with an `error`) and logged,
+  table's transient version-GC failure is recorded on that table's stats row
-  and never aborts the healthy tables — cleanup is the convergence
+  (with an `error`), logged, and reported by the CLI without aborting healthy
-  backstop, so it does as much as it can and converges on re-run. The CLI reports
+  tables. Orphan-reclaim failures are also logged and retried on a later cleanup,
-  any failed tables; rerun `cleanup` to retry them.
+  but the current stats/CLI surface does not attach them to
+  `TableCleanupStats.error`. The recovery and live-branch checks below are
+  preflight invariants instead: either must succeed before any version GC runs.
+  Rerun `cleanup` to converge either kind of per-table failure.
 - CLI guards with `--confirm`; without it, prints a preview line.
 - **Non-local consent.** Against a non-local target (an `s3://` store/cluster), `cleanup` additionally requires `--yes` on top of `--confirm`: a TTY is prompted, and a non-interactive run (no TTY, or `--json`) refuses rather than destroying. A local (`file://`) target needs only `--confirm`. The same `--yes` gate applies to overwrite `load` and `branch delete`; every maintenance run echoes its resolved target to stderr (suppress with `--quiet`).
-- **Recovery floor:** `--keep < 3` may garbage-collect versions that crash recovery needs as a rollback target. Default `--keep 10` is safe.
+- **Recovery floor:** `--keep < 3` may garbage-collect versions that a later
+  rollback would otherwise use. `--keep 10` is the recommended conservative
+  count; there is no implicit default, so pass a policy explicitly.
 - **Requires clean recovery state.** If any durable recovery intent is pending,
   cleanup refuses before orphan reconciliation or version GC. Reopen the graph
   read-write (or restart the server) to resolve recovery, then rerun cleanup;
   deleting transaction/version history while an intent is pending would make
   exact effect ownership unverifiable.
+- **Preserves live lazy branches.** A graph branch initially inherits each data
+  table directly from an exact main-table version; until that table is first
+  written on the branch, there is no native Lance data-table branch for Lance's
+  cleanup to discover. Under the same schema, branch, and table gates used for
+  version GC, cleanup therefore reads every live graph-branch snapshot and caps
+  each main dataset's cutoff at its oldest inherited version. Native per-table
+  forks remain protected by Lance itself. If any live branch snapshot cannot be
+  classified, cleanup refuses before garbage-collecting any table rather than
+  guessing that its referenced versions are disposable.
+- **Refuses uncovered main-table drift.** Every manifest-visible main version
+  must open and equal Lance HEAD during the graph-wide preflight. If an external
+  or interrupted operation advanced HEAD without a recovery sidecar, run
+  `omnigraph repair` before cleanup; version GC never guesses around that drift.
 - **Orphaned-branch reconciliation:** before the version GC, cleanup reclaims any per-table Lance branch absent from the manifest branch list. These orphans arise when a `branch_delete` flips the manifest authority but a downstream best-effort reclaim does not complete (see [branches-commits.md](../branching/index.md)). The reconciler is idempotent (it no-ops once nothing is orphaned), runs regardless of the `keep_versions` / `older_than` values (those gate version GC only), and never reclaims `main` or system-branch forks. Reclaimed forks are logged. Graph lineage has no separate branch dataset: it lives in `__manifest`.
 
 ## Tombstones
 
 Logical sub-table delete markers in `__manifest` that exclude a sub-table version from snapshot reconstruction.
 
-## Internal schema migrations
+## Internal schema versions
 
-Version evolutions of the on-disk `__manifest` shape are reconciled automatically on the first write under a new binary. An on-disk stamp records the shape; the binary migrates it forward before reading state, and reads are side-effect-free. No operator action is required for in-place upgrades. See [storage.md → Internal schema versioning](../concepts/storage.md) for the full mechanism.
+The on-disk format is strict-single-version. A binary refuses a graph whose
-
+internal schema stamp differs from the version it supports; storage-format
-A binary opening a manifest stamped at a version *higher* than it knows about refuses to publish with a clear "upgrade omnigraph first" error — old binaries cannot clobber a newer schema.
+changes use export/import rebuild rather than automatic in-place migration.
+See the [upgrade guide](upgrade.md) and
+[versioning policy](../../dev/versioning.md).