Harden branch recovery and cleanup retention (#344)

* Harden branch recovery and cleanup retention

* Fail closed on blocked partial rollback

docs/user/branching/index.md

@@ -6,12 +6,34 @@ Lance supports branching at the dataset level: a branch is a named lineage of ve
 
 ## L2 — Graph-level branches
 
-OmniGraph builds *graph branches* on top by branching every sub-table coherently:
+OmniGraph builds *graph branches* on top with one authoritative `__manifest`
+ref whose table entries form a coherent graph snapshot; data-table forks are
+created lazily on first write:
 
-- **Create** (`branch create` / `branch create --from <target>`) — the name `main` is disallowed; fails if the branch exists. Atomic: the new branch becomes visible all-or-nothing, so a name never half-exists.
+- **Create** (`branch create` / `branch create --from <target>`) — the name `main` is disallowed; fails if the branch exists. Logically atomic: the branch becomes visible only when its authoritative ref exists, so a name never half-exists in graph reads. If storage interruption leaves only Lance's unreachable shallow-clone directory, the next same-name create reclaims it and retries automatically.
 - **List** (`branch list`) — returns public branches, **filtering the internal** `__schema_apply_lock__` branch.
-- **Delete** (`branch delete`) — refuses if there are descendants on the branch, or if it is the current branch. Once deleted, the branch is gone from every snapshot. The owned per-table forks are reclaimed best-effort; if that reclaim hits a transient object-store error, the leftover storage is reclaimed later by the [`cleanup`](../operations/maintenance.md) command. One consequence: if a delete's reclaim fails, reusing that branch name before the next `cleanup` surfaces a clear error pointing at `cleanup`.
+- **Delete** (`branch delete`) — refuses if there are descendants on the branch, or if it is the current branch. Once its authoritative ref is removed, the branch is gone from every snapshot even if reclaiming a now-unreachable storage directory needs a later retry. Owned per-table forks are reclaimed best-effort; same-name create/first write safely reconciles relevant leftovers, and [`cleanup`](../operations/maintenance.md) remains the general backstop.
 - **Lazy forking**: a branch only forks a sub-table when that sub-table is first mutated on it. Pure-read branches share storage with their source. If two writers race to first-write the same branch, the loser gets a retryable "refresh and retry".
+- **Names are path-prefix-disjoint while live.** Slash-separated names are
+  supported (`review/2026-07`), but `review` and `review/2026-07` cannot both be
+  live because Lance stores them in overlapping physical directories. Choose
+  sibling names, or delete the existing ancestor/descendant first.
+
+Graph branch create/delete are coordinated across handles in one writer
+process. Until Lance exposes conditional native ref mutation, separate writer
+processes must not concurrently control branches on the same graph.
+
+For a legacy graph that already contains path-prefix-overlapping live names,
+recovery also preserves the leaf-first escape hatch. If read-write open finds an
+unresolved first-touch sidecar for an ancestor table fork while a live path-child
+remains, it never deletes the child's storage. When the interrupted write owns no
+table effect, it leaves the sidecar in place and completes the open so you can
+delete the descendant branch leaf-first. When the same sidecar owns a partial
+table effect, open fails closed because cleanup and rollback must finish together;
+remove the descendant through an already-open handle or an offline Lance-level
+branch tool, then reopen. The next read-write open reclaims the ancestor residue,
+rolls back the partial effect, and retires the sidecar. `omnigraph repair` is not
+that offline tool: it correctly refuses to run while a recovery sidecar is pending.
 
 ## L2 — Commit graph
 

docs/user/operations/maintenance.md

@@ -30,28 +30,53 @@
 - Garbage-collects old versions per table.
 - Removes versions (and their unique fragments) older than the retention policy.
 - Policy options `keep_versions` and `older_than` — at least one is required.
+  `keep_versions=N` derives its requested cutoff from the newest `N` available
+  versions per table (the current HEAD is always retained, including for
+  `N=0`); live-branch safety floors may retain additional intervening versions.
+  When both options are set, a version is removed only if it is older than both
+  cutoffs.
 - Returns per-table stats: `table_key, bytes_removed, old_versions_removed, error`.
-- **Fault-isolated per table.** A single table's transient failure (version GC or
-  orphan reclaim) is recorded on that table's stats row (with an `error`) and logged,
-  and never aborts the healthy tables — cleanup is the convergence
-  backstop, so it does as much as it can and converges on re-run. The CLI reports
-  any failed tables; rerun `cleanup` to retry them.
+- **Fault-isolated per table after the graph-wide safety preflight.** A single
+  table's transient version-GC failure is recorded on that table's stats row
+  (with an `error`), logged, and reported by the CLI without aborting healthy
+  tables. Orphan-reclaim failures are also logged and retried on a later cleanup,
+  but the current stats/CLI surface does not attach them to
+  `TableCleanupStats.error`. The recovery and live-branch checks below are
+  preflight invariants instead: either must succeed before any version GC runs.
+  Rerun `cleanup` to converge either kind of per-table failure.
 - CLI guards with `--confirm`; without it, prints a preview line.
 - **Non-local consent.** Against a non-local target (an `s3://` store/cluster), `cleanup` additionally requires `--yes` on top of `--confirm`: a TTY is prompted, and a non-interactive run (no TTY, or `--json`) refuses rather than destroying. A local (`file://`) target needs only `--confirm`. The same `--yes` gate applies to overwrite `load` and `branch delete`; every maintenance run echoes its resolved target to stderr (suppress with `--quiet`).
-- **Recovery floor:** `--keep < 3` may garbage-collect versions that crash recovery needs as a rollback target. Default `--keep 10` is safe.
+- **Recovery floor:** `--keep < 3` may garbage-collect versions that a later
+  rollback would otherwise use. `--keep 10` is the recommended conservative
+  count; there is no implicit default, so pass a policy explicitly.
 - **Requires clean recovery state.** If any durable recovery intent is pending,
   cleanup refuses before orphan reconciliation or version GC. Reopen the graph
   read-write (or restart the server) to resolve recovery, then rerun cleanup;
   deleting transaction/version history while an intent is pending would make
   exact effect ownership unverifiable.
+- **Preserves live lazy branches.** A graph branch initially inherits each data
+  table directly from an exact main-table version; until that table is first
+  written on the branch, there is no native Lance data-table branch for Lance's
+  cleanup to discover. Under the same schema, branch, and table gates used for
+  version GC, cleanup therefore reads every live graph-branch snapshot and caps
+  each main dataset's cutoff at its oldest inherited version. Native per-table
+  forks remain protected by Lance itself. If any live branch snapshot cannot be
+  classified, cleanup refuses before garbage-collecting any table rather than
+  guessing that its referenced versions are disposable.
+- **Refuses uncovered main-table drift.** Every manifest-visible main version
+  must open and equal Lance HEAD during the graph-wide preflight. If an external
+  or interrupted operation advanced HEAD without a recovery sidecar, run
+  `omnigraph repair` before cleanup; version GC never guesses around that drift.
 - **Orphaned-branch reconciliation:** before the version GC, cleanup reclaims any per-table Lance branch absent from the manifest branch list. These orphans arise when a `branch_delete` flips the manifest authority but a downstream best-effort reclaim does not complete (see [branches-commits.md](../branching/index.md)). The reconciler is idempotent (it no-ops once nothing is orphaned), runs regardless of the `keep_versions` / `older_than` values (those gate version GC only), and never reclaims `main` or system-branch forks. Reclaimed forks are logged. Graph lineage has no separate branch dataset: it lives in `__manifest`.
 
 ## Tombstones
 
 Logical sub-table delete markers in `__manifest` that exclude a sub-table version from snapshot reconstruction.
 
-## Internal schema migrations
+## Internal schema versions
 
-Version evolutions of the on-disk `__manifest` shape are reconciled automatically on the first write under a new binary. An on-disk stamp records the shape; the binary migrates it forward before reading state, and reads are side-effect-free. No operator action is required for in-place upgrades. See [storage.md → Internal schema versioning](../concepts/storage.md) for the full mechanism.
-
-A binary opening a manifest stamped at a version *higher* than it knows about refuses to publish with a clear "upgrade omnigraph first" error — old binaries cannot clobber a newer schema.
+The on-disk format is strict-single-version. A binary refuses a graph whose
+internal schema stamp differs from the version it supports; storage-format
+changes use export/import rebuild rather than automatic in-place migration.
+See the [upgrade guide](upgrade.md) and
+[versioning policy](../../dev/versioning.md).