omnigraph/docs/dev/handoff-rfc-013-write-path.md

# Handoff: finishing RFC-013 (write-path latency + correctness)

> **Status update (strand-and-retire work):** the `_graph_commits.lance` /
> `_graph_commit_actors.lance` datasets are **retired** (Phase B — lineage lives in
> `__manifest`; the `CommitGraph` is a pure projection). References to those tables
> below are historical: `optimize` now compacts **`__manifest` only** among the
> internal tables, and the per-write `_graph_commits` scan term is gone. See
> [versioning.md](versioning.md) and [invariants.md](invariants.md).

**Status:** living handoff. **Source of truth is [`rfc-013-write-path-latency.md`](rfc-013-write-path-latency.md)** —
this doc is the *current-state map + the decisions/validation from the latest work cycle
+ the concrete next actions*. When they disagree, the RFC wins (and fix this doc).

**Audience:** the engineer/agent who picks up RFC-013 next.

> **Two threads, one RFC.** RFC-013 has been worked in two overlapping lines by different
> cycles, with different sub-numbering. Don't let the numbering confuse you:
> - **Thread A — per-write call-count / RTT collapse + concurrency correctness** (steps 3b /
>   4 / 5, Design A / `PublishPlan`, #297 / #254 / #296 / #298). This is **§1–§10 below** (the
>   original body of this doc). Read it for the concurrency model and the convergent fix.
> - **Thread B — `__manifest` scan amplification + the unbounded `_versions/` chain** (the
>   investigation framed as **PR1 / PR2 / PR3 / PR4**). This is **§A below (read it first)** —
>   it is the most recent cycle (2026-06-26) and where the live branch sits.
>
> They meet at the internal-table maintenance area (Thread A's "step 2a/2b" == Thread B's
> "compaction is done / PR1 = bound the chain"). §A maps the two framings.

---

## §A. Current state — the `__manifest` scan + version-chain thread (2026-06-26) — READ FIRST

The latest cycle attacked **write latency that grows with graph age on object storage** and a
**hard open failure on a large `_versions/` chain**. The working plan (survives context resets,
**not in the repo**) is `~/.claude/plans/in-the-mean-time-humble-reef.md` — pull it for the full
PR1/PR2/PR3/PR4 decomposition, the assumptions-validated-against-code list, and the critical-files
map. This section is the durable summary.

### A.1 The problem (root cause)

Two interacting terms, both centered on the internal `__manifest` Lance dataset (the cross-table
catalog; one dataset, 217 tables on a real graph):

- **Term 1 — repeated full `__manifest` scans per write.** Each read of `__manifest` was a bare
  `dataset.scan()` (no filter/projection/index), cost **O(fragments F)**. A publish did **4** such
  scans (OCC re-capture + `load_publish_state` + the `use_index(false)` merge-insert join +
  post-publish read-back). `F` grows **+1 per write**, so per-write cost climbs with history.
- **Term 2 — unbounded `_versions/` chain.** `cleanup` version-GC **excludes** the internal tables
  (`all_table_keys` is node/edge-only, `db/omnigraph/optimize.rs`), so `__manifest/_versions/`
  grows without bound (768 objects measured). Lance **lists** that prefix on every open; once large,
  a store like RustFS times out serving the list page → branch ops take minutes or **fail outright**.
  Unfixable via the public CLI today (`cleanup --keep` can't target `__manifest`).

Validated three ways: a multi-agent workflow (readers + adversarial refutation), Lance 7.0.0 source,
and live branch-op probes on a RustFS mirror. **Key Lance fact (closed the open-path investigation):**
on standard S3-class stores (R2/RustFS, not S3 Express) Lance sets `list_is_lexically_ordered = true`,
so it **never** uses `latest_version_hint.json` — it always lists `_versions/`. So the version hint is
**not** a lever; only **bounding the chain (PR1)** fixes the open path. (Corollary: on production R2 the
open is one cheap list page regardless of chain length, so the chain barely affects R2 open latency —
the RustFS *failure* is a RustFS list-page limit; the R2 ~16s write latency is Term-1 fragment
amplification = PR2's target.)

### A.2 What is LANDED on `main`

- **Step 2a** — `optimize` compacts the internal tables too (`__manifest` / `_graph_commits` /
  `_graph_commit_actors`), so a *periodically-compacted* graph keeps Term-1 flat. (Cleanup/version-GC
  of them is the still-open PR1.)
- **Phase 7 / #299** (`1c5cb874`) — graph lineage lives in `__manifest` (`graph_commit` +
  `graph_head:<branch>` rows in the same publish merge-insert; `_graph_commits` is now a projection;
  v3→v4 internal-schema migration; schema-version floor). This removed the per-write commit-graph
  scan and closed the manifest→commit-graph atomicity + commit-graph-parent-under-concurrency gaps.
  **This is the base everything below builds on.**

### A.3 What is IN FLIGHT — branch `ragnorc/read-lance-table-docs` = **PR #307** (OPEN, base `main`)

Ten commits ahead of `origin/main`. PR #307 includes the scan-halving work, the fold
correctness fix, and PR2.1's ground-truth cost harness.

1. **PR2 — halve per-write `__manifest` scans** (`4ac3cde4` + tripwire `52a7e0cd`). Three moves:
   - **#1a probe-gate the OCC re-capture** (`occ_snapshot_for_branch`, mirrors the read path's
     `resolve_target_inner`): replace the cold re-scan with a cheap incarnation **probe**; reuse the
     warm coordinator on match, cold-scan only on mismatch.
   - **#1b in-memory post-publish fold** (`fold_inputs` in `publisher.rs`; `PublishOutcome.known_state`):
     build the new `ManifestState` from `existing_versions ∪ pending rows` instead of re-scanning.
   - **#1c projection** on `read_manifest_scan` (drop `base_objects` always, `object_id` off the
     table-state path).
   - Net: per-write `__manifest` scans **4 → 2**; the two inherent publisher scans
     (`load_publish_state` + the `use_index(false)` merge-join) remain O(F).
2. **Fold correctness fix** (`5537cd95` test, `245cb26d` fix) — a reviewer (Cursor Bugbot High +
   Codex P2) caught that `#1b`'s fold dropped a **same-version owner-branch handoff**: a `table_version`
   row UPDATEd in place at the same Lance version with a new `table_branch` (merge-insert `UpdateAll`
   on the deterministic `version_object_id`) was appended after `existing_versions`, and
   `assemble_manifest_state` kept the stale first entry, so the warm coordinator held the wrong fork
   until refresh. Fix: key the fold's version entries by `(table_key, table_version)` so a pending row
   **replaces** the existing one (mirroring `UpdateAll`). Test-first repro in
   `db/manifest/tests.rs::test_post_publish_fold_reflects_owner_branch_handoff` (red→green).
3. **PR2.1 — ground-truth cost harness** (`fd73f01b`, `59d9ff39`, `3cd2b2c1`, `383022e8`, `9f1e5b6e`).
   Rebuilt `tests/helpers/cost.rs` on lance's IO-counted idiom (`incremental_stats()` deltas; one
   `IOTracker` per class). Added `cost_harness(body)` / `GraphIoMeter`: it installs one `__manifest`
   tracker **before the coordinator opens**, so the tracker rides every handle (init + each
   post-publish reassignment at `db/manifest.rs:590`). `manifest_reads` is now **ground truth**
   (handle-age-irrelevant), closing the blind spot where a per-op tracker installed at measure time
   could not see reads on the long-lived warm handle. `last_manifest_reads()` dumps the read log for
   `assert_io_eq!`-style failure diagnostics. Outside `cost_harness`, `measure` falls back to
   fresh-per-op, so `write_cost_s3.rs` is untouched. (Kept the bespoke `PrefixCounter` for the
   opener/scan split — lance does the same with `throttle_store`/`failing_store`, and the
   request-log alternative would couple to unstable debug method-strings.)

### A.4 The accurate measurement (PR2.1's payoff — what it told us)

The old (fresh-only) harness **undercounted writes**: `#1a`'s probe rides the warm handle, and its
reads escaped the per-op tracker (they showed only as `version_probes=1`). Ground truth counts them
and reveals **a write's freshness probe does ~3 `__manifest` object-store RPCs** (a *read*'s probe is
a 0-IO cache hit). So, apples-to-apples (both ground truth), per-write `__manifest` ops:

| | depth 10 | depth 100 | slope |
|---|---|---|---|
| Pre-PR2 (4 cold scans) | 50 | 410 | +4/write |
| Post-PR2 (ground truth) | 28 | 208 | +2/write |

- PR2 roughly **halved** the per-write manifest work **and its growth slope** (+4 → +2/write).
- The **compacted/maintained floor is ~5 RPCs/write, flat in history** — the 3-RPC probe now dominates
  it (it is O(1), not O(F)). So `#1a` made the OCC re-capture O(1), it did not make it free.
- For latency: a periodically-compacted graph has bounded, history-independent per-write manifest
  cost; an unmaintained graph still grows at half the rate (PR1 flattens the residual). The probe and
  RFC #7264 are the levers for the compacted floor. (The harness measures op *count*, the latency proxy
  on object stores; the ~16s R2 figure is the open-path chain = PR1, separate.)
- Regression guard: `write_cost.rs::manifest_reads_capture_warm_probe` (fresh=11 vs ground-truth=14)
  goes red if the ground-truth wiring reverts.

### A.5 What is LEFT (priority order) — Thread B

1. **PR1a — manual `__manifest`-only cleanup** *(available now, no new invariant; HIGHEST priority —
   it is the only thing that fixes the hard open **failures**)*. Add `all_table_keys_internal()` +
   `cleanup_internal_tables()` reusing the generic `cleanup_all_tables` loop (`optimize.rs`); refuse on
   a pending recovery sidecar. Safe **only on a quiesced graph** (no concurrent writers → no Q8
   resurrection race). Shrinks `_versions/` (768 → keep-N). This is RFC **step 2b's available half**.
   Pair with a **V2-naming surface guard** (protects the one-page open fast-path).
2. **PR3 (the available half) — branch-op de-amplification.** Branch **merge** candidate-scoping
   (avoid 3 full cross-branch snapshots + union-all-keys upfront, `exec/merge.rs`); **parallelize** the
   branch-delete loop (`ensure_branch_delete_safe` snapshots every other branch — O(branches)). Each
   per-branch scan is already cheaper post-PR2 (#1c projection).
3. **Design-gated / deferred:**
   - **PR1b — the Q8 durable boundary watermark** for SAFE automated/cadenced GC under live writers
     (Lance version create is a bare `PutMode::Create` with no monotonic guard → a stalled writer can
     resurrect a GC'd version = silent lost write on R2/S3). Invariant-level, partially MTT-redundant.
     **This is the same design point as Thread A's "step 2b / Q8 watermark" in §8.** Design deliberately
     or wait for RFC #7264.
   - **PR3 branch-delete O(1)** — needs a cross-branch dependency index (the `table_branch` dependency
     is genuinely cross-branch with no index today).
   - **PR4 / RFC #7264** — Lance native branch-aware `BatchCreateTableVersions`; manifest read → O(1),
     per-write fragment append gone; retires most of PR1/PR2. Upstream-blocked.
4. **Low-leverage:** ~~retire the vestigial `_graph_commits`/`_graph_commit_actors` datasets~~ **DONE**
   (Phase B of the strand-and-retire work — the tables are gone, branch authority is `__manifest`, the
   `CommitGraph` is a pure `__manifest` projection). Still open: a bitmap index on `__manifest` (no builder
   exists; `use_index(false)` means it can't serve the CAS join anyway — a `graph_head:<branch>` point-lookup
   is the better variant).

### A.6 Critical files (Thread B)

- `db/manifest/state.rs` — `read_manifest_state` / `read_manifest_scan` / `assemble_manifest_state` (the
  shared reduction both the fold and the scan feed).
- `db/manifest/publisher.rs` — `fold_inputs` / `PublishOutcome` / `is_owner_branch_handoff` (publisher.rs:267,
  the same-version handoff the fold must honor) / the merge-insert CAS.
- `db/manifest.rs` — `commit_changes_with_lineage` (adopts the fold; `self.dataset = dataset` reassignment
  at :590, the reason the cost tracker must be installed before open) + the probes.
- `db/omnigraph.rs` — `occ_snapshot_for_branch` (#1a), `resolved_branch_target`, `ensure_branch_delete_safe` (PR3).
- `exec/staging.rs` `commit_all`; `exec/merge.rs` (PR3); `db/omnigraph/optimize.rs` (`all_table_keys`,
  `cleanup_all_tables` — PR1).
- `tests/helpers/cost.rs` (the harness), `tests/write_cost.rs` / `warm_read_cost.rs` / `write_cost_s3.rs`,
  `tests/writes.rs` / `consistency.rs` / `composite_flow.rs` (must stay green).

### A.7 Gotchas (Thread B, learned this cycle)

- **A per-op object-store wrapper cannot see a long-lived handle's reads.** That was the measurement
  blind spot. The fix is to install the tracker before the handle opens (`cost_harness`), not at measure
  time. A write's warm-handle probe is **3 RPCs** that hid behind `version_probes=1`.
- **`cost_harness` must wrap the WHOLE test body** (the graph must open inside it), and the body future
  must be **`Box::pin`-ed** — wrapping a whole test body in another async layer overflows the test
  thread's stack (these cost tests already raise `recursion_limit`).
- **The fold must mirror merge-insert identity.** `version_object_id(table_key, version)` is
  deterministic, so a same-version handoff is an in-place `UpdateAll`; the in-memory fold must key by
  `(table_key, version)` and replace, or the warm coordinator desyncs from a fresh re-scan. The
  byte-identity guard is `writes.rs::post_publish_fold_matches_fresh_reopen`.
- **`lance-io` `test-util`** is enabled in dev-deps (gives `IoStats.requests` + `assert_io_eq!`,
  diagnostics only); production builds exclude dev-deps so they never see it.

### A.8 Immediate next action

The natural next PR is **PR1a** (no design gate, fixes the RustFS open failures). Run and confirm
the relevant test gate before starting or stacking that follow-up.

---

## 0. TL;DR — where we are and what's next

RFC-013 makes the write path fast **and** correct on object storage (217 Lance tables
under one `__manifest` catalog, on R2/S3). It is sequenced as steps; read §9 of the RFC
for the canonical list. Current reality:

**Landed on `main`:**
- **Step 1** — Tier-1 cost gate + the shared `helpers::cost` harness (#288).
- **Step 3a** — opener bypass: write opens go direct (`Dataset::open` by URI + version)
  instead of the Lance-namespace builder (#288). **This already banked the dominant
  depth win** — see §2 below; it reframes everything.
- **Step 2a** — internal-table compaction: `optimize` now compacts `__manifest` /
  `_graph_commits` / `_graph_commit_actors` (#291). Plus the RFC latency-model
  correction (#292).
- **Step 4 / Phase 7** — graph lineage moved into `__manifest` (#299 `1c5cb874`):
  `graph_commit` + mutable `graph_head:<branch>` in the publish merge-insert,
  `_graph_commits` now a projection. **The base for the live branch (§A).**
- **Optimize-vs-write race** — optimize survives a cross-process write race on the
  same table (#297, **LANDED** — origin/main `6d4606a8`; see §6 for why it's not
  redundant with Design A). Step 3b stacks on top of this.

**Open PRs (land these; relationships in §7):**
- **#296** `correctness-by-design-fix` — recovery roll-forward converges on a concurrent
  manifest advance (the fix for the flaky `iss-schema-apply-reopen-recovery-race`).
  **MERGED to main and integrated into this branch** — the converge helper now threads
  Phase-7's manifest-CAS recovery `graph_commit_id` (see `converge_or_defer_roll_forward`).
- **#295** `docs/rfc-013-step-3b` — the step-3b RFC doc.
- **#254** `ragnorc/bug-4-schema-apply-occ` — schema-apply vs optimize false-fail
  (same op-class family as #297, logical side).

**Step 3b is DONE** (capture-once `WriteTxn`, schema-once + open-collapse; see §4) on
`rfc-013-step-3b-writetxn-v2`. **Phase 7 (step 4) has since LANDED on `main` (#299 `1c5cb874`)**
— lineage now lives in `__manifest` (see §A.2). **Next for Thread A: the big one — Design A /
`PublishPlan` unification (step 5)** — see §5, the convergent fix for the bug *class* this area
keeps generating, which also absorbs 3b's deferred session-aware write opens. **Next for Thread B
(the live branch, §A): PR1a** (manual `__manifest` cleanup — fixes the RustFS open failures).

---

## 1. The corrected mental model (read this before touching anything)

Three reframes from the latest cycle that the older RFC prose may not fully reflect:

### 1a. 3a already won the depth fight → the residual is constant-factor + RTT
Before 3a, the write re-opened each table through the lance-namespace builder ~13×, and
that path was **O(depth)** (it re-opened `__manifest` + `list_table_versions` per open —
**not** a Lance back-walk; the root cause was OmniGraph's own namespace round-trips, not
Lance — validated against Lance source). 3a swapped it for the direct opener, which is
**O(1)** (`from_uri(loc).with_version(N)` = arithmetic path + one HEAD). So:

- The dominant **O(depth) data-table** term is **gone**.
- Step 2a flattened the secondary **internal-table** scan term.
- What remains is the **~110-hop serial backbone × RTT + compute** — a constant in
  depth. The latency model is **`wall = (serial_hops + ops/effective_concurrency)·RTT
  + compute`**; on a capped store (R2) the op-count term re-enters wall-clock, on an
  unlimited store it parallelizes away. Measured: prod one-row write 27→15.76s after
  2a; the remaining 15.76s is the serial backbone — **step 3b's target**, not step 2's.
- Step 3b's win is therefore the **call-count/RTT collapse** (redundant opens, the
  flat-46 schema reads), NOT a depth slope. Don't expect a depth-slope improvement from
  3b; gate it on the constant-factor (S3 round-trips), not a curve.

### 1b. Two op classes, two commit models (the §6.6 principle)
Every concurrency bug in this area is **one op class using the other's commit model**:

| class | examples | commutes? | correct commit model |
|---|---|---|---|
| **maintenance** | compaction (`Rewrite`), `optimize_indices` | yes (content-preserving) | Lance native rebase + app reopen/replan on real overlap + **monotonic manifest fast-forward** — no epoch, no read-set |
| **logical mutation** | load / mutate / merge / delete | no (lost-update, write-skew) | strict cross-process OCC: read-set + write-set CAS under the `writer_epoch` fence |

Applying strict OCC + equality-CAS uniformly is the mistake: too strong for maintenance
(false conflicts — #297's bug), too weak for logical cross-process (§6.5 corruption).

### 1c. The root liability (what keeps generating these bugs)
Lance gives **per-table atomic commits** but **no cross-table/cross-step atomicity**, so
every multi-commit op advances per-table Lance HEAD **before** the manifest references it
(the "A-before-B window"). The resulting `HEAD vs manifest` delta is **ambiguous**
(external drift? my own in-flight work? a crashed writer?), and **many uncoordinated code
paths each re-interpret it** (4 writers + the maintenance path + recovery + the write-path
drift guard). Each interpreter is a fresh chance to misclassify. That is the bug class:
- §6.5 cross-process logical corruption,
- #297's own-HEAD-drift misclassification,
- the flaky write-path "HEAD ahead of manifest, run repair" guard,
- the recovery classifier edges.

**The convergent fix is Design A (one publish authority — step 5); Lance MTT eventually
retires the window entirely.** See §5.

### 1d. The second facet: the write base is a stale pin (no probe)
The READ path resolves its base behind a freshness probe (`resolve_target_inner`
omnigraph.rs:~1072 → `probe_latest_incarnation` → `refresh_manifest_only`); the WRITE path
does NOT (`resolved_branch_target` omnigraph.rs:~778 returns the warm `coord.snapshot()` for
the bound branch, no probe). So a long-lived server's write base lags the live manifest. That
single staleness feeds **two distinct failure modes**, both surfaced this cycle:

1. **Stale validation *reads* → integrity under-enforced.** Write-path RI checks read
   committed state off the stale base. 3b's collapse #1 made it worse for edge `@card`:
   `edge_cardinality_read_handle` (mutation.rs:~614) scans the pinned `txn.base` instead of
   live HEAD (was live HEAD pre-3b), so a concurrent edge committed after `txn` capture is
   uncounted → a `@card` max can be exceeded (cursor **High** / codex **P1** on #298,
   **VALID**). **#298 fix: restore the live-HEAD read for that scan** (un-regress; gate-safe —
   the `data_open_count` gate is a node insert) + a deterministic regression test (commit A's
   edge, then B validates → must see A) + correct the wrong "pinned base == live HEAD" doc
   comment (mutation.rs:~605-613, which assumes a single writer). The *structural* liability
   underneath: there is **no unified write-validation read-set** — endpoint
   (`ensure_node_id_exists`, warm `snapshot_for_branch`), cardinality (mutation: pinned
   `txn.base`; loader: warm `snapshot_for_branch` — the SAME check forks per write path),
   commit drift guard (live `fresh_snapshot_for_branch`), and uniqueness
   (`enforce_unique_constraints_intra_batch`, intra-batch only — cross-version uniqueness is a
   documented gap). Three freshness levels chosen ad hoc, none re-validated at commit → the
   §7.1 TOCTOU class, and each new constraint forks the pattern again.

2. **Stale OCC *pin* → false-fail on a maintenance advance.** A served strict update/delete
   pins the stale base version, then false-fails `ExpectedVersionMismatch` after an external
   `optimize` advanced `__manifest` — even though the advance was content-preserving
   compaction the logical write should fast-forward past (invariant 7). It's the **write-side
   mirror of #297/§6.6** (#297 made optimize fast-forward past a logical write; this is a
   logical write that must fast-forward past optimize). A served read clears it (the read
   probes the shared coordinator). Validated repro on prod (omnigraph.ragnor.co) +
   `writes.rs::served_strict_delete_after_external_optimize_advance_auto_refreshes`
   (`#[ignore]` on branch `fix/write-path-stale-view-probe`). **The naive "just probe" fix is
   proven wrong** — a blanket probe silently refreshes past *logical* advances too, breaking
   `consistency::stale_handle_public_mutation_must_refresh_then_retry` (the deliberate
   cross-process lost-update OCC primitive). The fix must **discriminate by op class**.

**Both fold into Design A (step 5), same as §1c.** `open_txn`'s one warm probe makes the base
fresh (absorbs maintenance advances cheaply); the **op-class-aware strict precondition** —
derive from Lance's per-version transaction metadata (all `Rewrite`/`ReserveFragments` =
maintenance → fast-forward the pin; any `Append`/`Update`/`Delete`/`Merge` = logical → fail
loudly; NO parallel marker, invariant 1/15) — is the correctness fence for anything that lands
after. And the §7.1 read-set-in-CAS unifies the validation read-set + re-validates it under the
`graph_head` contention. So **the stale-view false-fail, the cardinality/validation-read-set
liability, and #297's mirror are one bug** (the write base is a stale, un-probed, un-classified
pin) with **one home: the single PublishPlan delta-interpreter** (§1c + §5). Strong corroboration
of Design A — three symptoms, one fix.

---

## 2. Validated facts — do NOT re-derive these

Established this cycle against **Lance 7.0.0 source**
(`~/.cargo/registry/src/index.crates.io-*/lance-7.0.0`) and current engine code. Cited so
you can trust them without re-investigating.

**Lance (upstream):**
- `from_uri(loc).with_version(N).load()` and `checkout_version(N)` are **O(1)** (computed
  V2 path `_versions/{u64::MAX-N:020}.manifest` + one HEAD; no listing/back-walk).
  (`lance-table/src/io/commit.rs` `default_resolve_version`.)
- A shared `Arc<Session>` (`DatasetBuilder::with_session`) warms metadata/index caches
  keyed by `(URI, version, e_tag)`. Caveat: the *first* manifest read on open is uncached
  — the Session warms the *scan/index* metadata, not the first open. **`WriteParams` *does*
  carry a `session` field** (`lance/src/dataset/write.rs`), but it only matters on the
  `WriteDestination::Uri` arm; OmniGraph's staged path always drives off an **already-open
  `Dataset`**, and Lance takes the store/session from that handle. So to attach the shared
  Session to a write base, open read-style (`open_table_dataset` → `from_uri().with_version()
  .with_session()`) and drive the staged write off that handle.
- A held `Arc<Dataset>` at a pinned version is `Send + Sync`, immutable, safe to reuse for
  many scans/count/staged-write base in one txn (OmniGraph's `TableHandleCache` already
  relies on this).
- **No compaction `RetryExecutor`** (only Delete/MergeInsert/Update have one).
  `commit_compaction` commits a fixed `Rewrite` via `apply_commit` direct. In
  `commit_transaction`, a semantic `RetryableCommitConflict` **escapes the retry loop**
  via `?` at `io/commit.rs:979`; the loop only retries the OCC `CommitConflict`
  (`:1096`), and even that re-rebases the *same* transaction (never re-plans). ⇒
  **compaction needs app-level reopen+REPLAN; you cannot "set conflict_retries" and let
  Lance own it.**
- `check_rewrite_txn`: a `Rewrite` rebases **cleanly** past a concurrent `Append`/disjoint
  `Update`/`Delete` (preserving both); only a same-fragment overlap yields a retryable
  conflict. ⇒ the common concurrent insert/update/delete is rebased for free; the app
  retry fires only on real overlap.

**Engine (internal):**
- Read path (post-#268) already has the capture-once machinery: `Snapshot` (`db/manifest.rs`),
  warm `GraphCoordinator` behind a `latest_version_id`/incarnation probe, a held
  `TableHandleCache` keyed `(table,branch,version,e_tag)`, **one shared `Session` per
  graph** (`read_caches.session`). **Writes bypass all of it by construction**
  (`resolved_branch_target` returns `read_caches: None`; the 3a write opener attaches no
  session and opens by latest, not pinned version).
- A single write opens each table **3–4×** (accumulation → staging reopen → commit
  drift-guard → publish prepare), each a fresh cold open. `validate_schema_contract`
  (`db/schema_state.rs`, via `ensure_schema_state_valid`) runs uncached (~3 `read_text`
  + 2 `exists`) at every resolve point (~the flat-46). Both are constant-factor, flat in
  depth — 3b's targets.
- Strict-op guards are the lost-update floor (3 layers: pre-stage `ensure_expected_version`
  `table_store.rs`; commit-time strict drift `exec/staging.rs`; publisher CAS
  `publisher.rs`). Capture-once **supplies** the pinned operand — never remove a guard.
- Fork-on-first-write authority reads (`classify_fork_ref` → `fresh_snapshot_for_branch`)
  must stay **fresh** (not served from a pinned base).
- Cost harness: `helpers::cost` (`measure`/`measure_with_staged`/`IoCounts`/`assert_flat`/
  `local_graph`/`s3_graph`). The schema-once assert can reuse `CountingStorageAdapter`
  (`warm_read_cost.rs::warm_query_validates_schema_contract_once`) with **zero** prod
  change; an open-count assert wants a small `open_count` AtomicU64 in `QueryIoProbes`
  (copy the `probe_count`/`record_probe` pattern). The forbidden-API guard
  (`tests/forbidden_apis.rs`) makes an instrumentation-level counter complete.

---

## 3. The #297 cycle (this branch) — what it is, and the lesson

`fix-optimize-concurrency-race` (5 commits): a CLI `optimize` racing a served write on the
same table failed (Lance Rewrite lost, or the equality-CAS publish lost). Fix: unify both
compaction paths on the internal path's **reopen+replan** shape, with a **two-level retry**
— outer loop reopens+replans on a real Lance overlap; inner Phase-C loop makes the manifest
publish a **monotonic fast-forward** (advance to compacted version `N`, or no-op when the
manifest already moved to `≥ N`), never the strict equality CAS. Sidecar written once;
in-process queue kept as a contention reducer (not the cross-process guard); no `writer_epoch`.

**Two review rounds surfaced two follow-on bugs I introduced with the retry loop** — both
fixed, both regression-tested (own-HEAD-drift via negative control):
1. **Own-HEAD-drift misclassification** (`56d004e0`): the drift guard re-ran every
   iteration and, after a partial Phase-B commit (auto_cleanup strip or compact, then a
   later op conflicts), saw `HEAD > manifest` from *our own* covered work and deleted the
   sidecar + returned `skipped_for_drift` (stranding uncovered drift). Fix: track
   `head_advanced`; the drift guard fires only when `!head_advanced`.
2. **Publish exhaustion spurious error** (`e9d16a2c`): the publish loop returned `Err` on
   its final retry even if the conflict meant a concurrent writer already published `≥ N`
   (postcondition met). Fix: re-check `current >= state.version` on exhaustion.

**The lesson (write it on the wall):** *wrapping a sequence of side-effecting commits in a
retry silently converts every "checked once, before any side effect" precondition into
"re-checked after partial side effects."* That's a distinct bug class; it needs
fault-injection tests **at each commit boundary**, not just end-to-end concurrency tests.
(The `optimize.before_compact` / `optimize.inject_reindex_conflict` failpoints exist for
exactly this.)

**Temporary mechanism flag:** `head_advanced` is an in-memory proxy for "is this HEAD
movement mine." Under Design A the authority answers that from the plan/sidecar **identity**
— so `head_advanced` is the part that gets *replaced*, while the monotonic-publish +
reopen/replan **semantics** are permanent. (Noted in RFC §6.6.)

---

## 4. DONE: Step 3b — capture-once `WriteTxn` (shipped on `rfc-013-step-3b-writetxn-v2`)

**Delivered:** on the **table-touch hot path**, a single `mutate`/`load` validates the schema
contract **once** and opens each touched data table **at most once** — a constant-factor/RTT
win (not a depth-slope win; 1a). Two cost gates in `write_cost.rs` lock it (both on a node
insert): `write_validates_schema_contract_once` (3 `read_text` / 2 `exists`, was 12/9) and
`keyed_insert_opens_table_at_most_once` (`data_open_count <= 1`, was 4). The carrier is the
minimal `WriteTxn { branch, base }`, threaded as `Option<&WriteTxn>` (`Some` on the hot
mutate/load path, `None` byte-identical everywhere else); it **converges into** step 5's
`PublishPlan`.

**Not "once" everywhere (scope, not regression):** edge endpoint / cardinality RI validation
(`ensure_node_id_exists`, the loader's RI + cardinality) still resolves through
`snapshot_for_branch` and re-validates the schema — and reads **warm**, not live. Threading
`txn.base` there to make it "once" would re-introduce the stale-read class the #298 cardinality
fix removed (it now reads live HEAD). Doing schema-once *and* fresh reads for those validations
needs the unified, re-checked read-set — **step 4 §7.1** (§1d). So #298 **un-regresses
cardinality only; it does not close write-validation freshness.** No edge-insert/load schema-once
gate yet (only the node gates above).

Commits (off merged-#297 main):
- **Stage 0** — scope `open_count` → `data_open_count`/`internal_open_count` by URI class
  (the review fix: `open_dataset_tracked` also opens `__manifest`/`_graph_commits`, so the
  raw counter conflated them and the gate was unreachable). Re-baselined RED 4.
- **Commit A (schema-once)** — capture `txn` once at entry (the single validation); the 4
  validation sites collapse: S1 (entry `ensure_schema_state_valid`) removed; S3a
  (`open_for_mutation_on_branch`) + S3b (`prepare_updates_for_commit`) source `txn.base`;
  S4 (`commit_all`) uses new `fresh_snapshot_for_branch_unchecked` (the OCC manifest re-read
  minus the schema re-validation). `fresh_snapshot_for_branch{,_unchecked}` now read the
  manifest directly via `ManifestCoordinator` (drops a spurious commit-graph `exists` probe;
  same `Snapshot`).
- **Commit B (open collapse 4→1)** — #1 accumulation open ELIMINATED (the node path discarded
  the handle; read `txn.base.entry().table_version`); #2 staging open KEPT (the one open);
  #3 commit drift-guard reads live HEAD via `entry.dataset.dataset().latest_version_id()` (a
  cheap manifest-pointer probe off the staged handle, not a fresh open); #4 index build reuses
  the `commit_staged` handle threaded through `CommittedMutation`/`prepare_updates_for_commit`.
- **Commit B.1 + cleanup** — named the two positional returns (`OpenedForMutation`,
  `CommittedMutation`) + a `debug_assert` pinning the open-skip contract; **removed the
  unearned `WriteTxn.session` field** (the collapse uses skip/probe/reuse, not a session).

**RFC §4.1 corrections — how they resolved:**
1. *Thread the evolving handle, not a version-keyed cache* → realized as collapse #4 (carry
   the `commit_staged` handle forward into the index build).
2. *Don't forbid re-resolution* → honored: the commit-time OCC re-read
   (`fresh_snapshot_for_branch_unchecked` — fresh manifest, only schema-revalidation dropped)
   and the fork-authority reads stay fresh.
3. *Minimal carrier* → `WriteTxn { branch, base }` (even the `session` from the original
   sketch was dropped as unearned).

**Deferred to step 5 (NOT in this PR):** session-aware write base opens. The one remaining
open (#2) stays a HEAD open; warming the shared `Session` across writes is an object-store
(S3) phenomenon invisible on local FS, so it earns its own `write_cost_s3.rs` gate in step 5,
where `txn` becomes the non-optional publish carrier. No new concurrency test was needed here:
#2 stays a HEAD open (no pinned+session base introduced), so the publisher CAS + #3 live-HEAD
probe fences are unchanged (covered by the green `writes.rs`/`consistency.rs`).

**Guardrails (don't regress):** schema validation is deliberately uncached for drift
detection — collapse to 1 *per write*, never cache across writes on a long-lived handle
(`lifecycle::long_lived_handle_rejects_schema_*`). The commit-time fresh read is OCC
machinery, not redundancy. Keep all 3 strict-op guards. Keep fork-authority reads fresh.
Pin the *correct* branch (server-bound-to-main writing a feature branch falls to a fresh
open). A branch `rfc-013-step-3b-writetxn` exists off an earlier main; rebase onto the
post-#297 main before starting.

---

## 5. Design A — the `PublishPlan` unification (step 5) = the convergent fix

**This is the real fix for the bug class in §1c.** Collapse the four hand-rolled writers +
the maintenance path into **one `publish(txn, plan)` authority** where the CAS + bounded
retry is **unconditional and unbypassable** (no caller can "hold the queue → skip the CAS").
Properties:
- **One interpreter of the `HEAD vs manifest` delta** — and "is this my work?" is answered
  by the plan/sidecar **identity**, not a re-derived comparison. The own-HEAD-drift bug, the
  §6.5 writers, the write-path guard — all close *by construction*.
- **Recovery = the same `PublishPlan` re-applied** — the crash-recovery interpreter and the
  live interpreter become the same code (`iss-merge-recovery-partial-rollforward` gone).
- Each `TableAction` commits by its **class** (§1b): `Rewrite` = maintenance (Lance rebase
  + reopen/replan + monotonic fast-forward, **no epoch**); load/mutate = logical (strict OCC
  + `writer_epoch`).

**Why it composes with Lance MTT (don't over-build):**
- The **unification itself is convergent** — when MTT lands, it slots *underneath* the same
  authority; nothing wasted. Build this.
- The **`writer_epoch`** is the one MTT-redundant piece (MTT's commit-handler lease subsumes
  a cross-process fence). Build it *last and minimally*, gated on actually deploying
  multi-writer topologies. Per the deny-list, don't reimplement what the substrate will own.

**Sequencing judgment (this cycle's strongest signal):** the bug density here (this PR alone
= 3 review rounds, all "a writer re-interprets the delta") means the current N-writers interim
is high integrated-over-time liability. **Consider pulling the *convergent half* of step 5
(the single authority + recovery-as-plan) forward — possibly ahead of 3b** — because it stops
the bug class rather than patching instances. #297 + #254 are the *de-risking inputs*: they
validate the maintenance-class and logical-class commit models in isolation first, so Design
A implements a known spec rather than designing under refactor pressure. Do NOT build more
substrate-shaped scaffolding (custom WAL / job queue / second coordination table) to paper
over the window — strictly higher liability than either Design A or waiting for MTT.

**Deeper-than-A (post-MTT or as Lance exposes uncommitted variants):** all-uncommitted-fragments
+ one manifest commit would shrink the A-before-B window itself, blocked today by Lance not
exposing uncommitted variants for `compact_files` / `optimize_indices` / vector index (#6666
open; delete #6658 shipped). Track, don't build yet.

### 5.1 Step-5 design constraints inherited from the #295 spec review
3b shipped a **minimal** `WriteTxn { branch, base }` (schema-once + open-collapse via
eliminate/probe/thread) and **deferred** the full §4.1 opener-unification — the pinned-base
opener, the shared-`Session` open, the write-local **handle cache**, and the strict-op
conflict-timing move — to step 5. So the greptile-bot comments on the #295 *spec* were **moot
for #298** (which built none of those constructs) but are **load-bearing constraints for step
5** when it builds them. Bank them:
1. **Handle cache must be `Send + Sync`** (`Mutex<HashMap<…, Dataset>>`, not `RefCell`) if
   `WriteTxn::open(&self)` is shared across concurrent stage futures — a `RefCell` compiles
   but panics when two stages poll. Or make it `&mut self` (no parallel-stage sharing). This
   is the deny-list "in-process-only `Dataset` impls — `Send + Sync`" item.
2. **The strict-op timing move needs an explicit retry contract.** If step 5 moves
   strict-op conflict detection from open-time `ensure_expected_version` to commit-time CAS
   (the §4.1 pinned-base design), it MUST specify: the txn is **discarded after any commit**
   (success or conflict — the handle cache is commit-invalidated), and the retry **re-opens a
   fresh `WriteTxn` at the new HEAD** (never re-stages against the stale pinned base — that
   reproduces the lost-update). **This is the same retry/refresh contract as the stale-view
   false-fail (§1d.2)** — the op-class-aware precondition + "fresh base on retry" are one
   design point. Today (#298) strict ops keep open-at-HEAD + `ensure_expected_version`, so the
   contract is unchanged; step 5 owns it the moment it pins strict reads to the base.
3. **The opener-equivalence test must be non-trivial.** A differential test that only passes
   when `HEAD == base` proves nothing about pinning. To actually prove "`WriteTxn::open`
   returns the pinned base, not HEAD," the test must **advance the branch HEAD externally
   (direct Lance write), then assert the txn open still reads the base version** — and that a
   strict write then fails `ExpectedVersionMismatch` at commit (verifying the timing move).

---

## 6. Why #297 is still needed even if you do Design A
- Design A **relocates** #297's maintenance-class commit logic into the authority's
  `TableAction::Rewrite` path; it does not eliminate it. #297 is the *validated spec + tests*.
- The two regression tests + §6.6 are the **contract** Design A must keep green.
- The prod bug is **live**; Design A is the largest write-path change in the RFC. Don't hold a
  correctness fix hostage to a big refactor, and don't do a big refactor under bug-fix urgency.
- Genuinely throwaway under Design A: only the loop's *location* + the `head_advanced` proxy
  (~a dozen lines). Everything else relocates or persists. **#297 LANDED.**

---

## 7. Open PRs and their relationships
- **#297** — maintenance-class fix (optimize vs write). **LANDED** (origin/main `6d4606a8`);
  step 3b stacks on it.
- **#254** — logical-class fix (schema-apply vs optimize false-fail). Same op-class family;
  both are de-risking inputs for Design A's per-class commit models.
- **#296** — recovery roll-forward converges on concurrent manifest advance. The fix
  for the flaky `iss-schema-apply-reopen-recovery-race`. It touches `recovery.rs` and is
  *aligned* with #297's "postcondition is the state, not winning the CAS" principle. **#296
  landed on main first and is merged into this branch:** the converge helper
  (`converge_or_defer_roll_forward`) was reconciled with Phase-7's manifest-CAS roll-forward —
  on convergence the audit references the winner's folded `graph_commit_id` (the current
  `graph_head`), not a freshly minted one.
- **#295** — the step-3b RFC doc (apply §4's three corrections to it).

---

## 8. Remaining RFC steps after 3b (RFC §9 is canonical)
- **#298 follow-up (do on the 3b PR, before merge): the edge-`@card` stale-read regression**
  (§1d.1). Restore the live-HEAD cardinality scan, add the deterministic regression test, fix
  the wrong doc comment. Small, gate-safe, un-regresses an integrity check (invariant 9). The
  residual concurrent TOCTOU is the §7.1 gap (step 4) — un-widen here, don't over-reach.
- **Step 4 / Phase 7** (`iss-991`): **LANDED on `main` as #299 (`1c5cb874`).** Lineage now lives
  in `__manifest` (`graph_commit` + mutable `graph_head:<branch>` in the same merge-insert;
  `_graph_commits` is a projection). Removed the per-write `commit_graph.refresh`; closed the
  manifest→commit-graph atomicity + commit-graph-parent-under-concurrency gaps. *(Historical note,
  kept for the §7.1 framing it carried:)* it
  carries the §7.1 *concurrent* write-skew fix (needs the `graph_head` contention row) —
  **frame §7.1 as "unify the entire write-validation read-set" (endpoint + cardinality +
  cross-version uniqueness), not merely "add `graph_head`"** (§1d.1): the bespoke
  `edge_cardinality_read_handle` and the mutation-vs-loader freshness fork dissolve into one
  pinned read-set re-validated under the `graph_head` contention, or the liability survives as
  a second special-case.
- **Step 5 / Design A** — §5 above. **Acceptance item: the served-strict-write stale-view
  false-fail** (§1d.2) — the op-class-aware precondition + `open_txn` probe. The contract is
  two tests passing *together*: un-ignore
  `writes.rs::served_strict_delete_after_external_optimize_advance_auto_refreshes` (goes green)
  *while* `consistency::stale_handle_public_mutation_must_refresh_then_retry` stays green
  (maintenance fast-forwards; logical fails loudly). Self-contained enough to ship standalone
  like #297 if prod pain is acute; otherwise fold into the single PublishPlan delta-interpreter.
- **Step 2b** — internal-table cleanup + the Q8 monotonic watermark (a Lance boundary tag).
  Deferred: only the secondary version-count/space term, touches the read/open path, and is
  MTT-redundant. Land when version-count cost bites.
- **§7.1 sequential write-skew** (`iss-overwrite-orphans-committed-edges`) — inbound-RI
  validation on node removal; independent, ships anytime.
- **#20** — the prod per-write `storage.ops` span metric (RFC §5.3), still owed.
- Branch ops: Lance `Clone` for create (`iss-691`).

---

## 9. Gotchas / traps (learned the hard way)
- **In-process queue ≠ cross-process lock.** Any "I hold the queue → skip the retry/CAS"
  reasoning is a bug across processes. This is the recurring trap.
- **Monotonic publish must be `≥`-conditional, never "no assertion."** The `__manifest`
  merge-insert is unconditional `UpdateAll` keyed on `object_id` (`publisher.rs:379`), so
  the equality (or monotonic) pre-check is the *only* guard — dropping it lets `UpdateAll`
  regress a newer version = lost write.
- **The drift guard interprets an ambiguous delta.** Re-evaluating it in a retry over
  self-mutated state is how #297's follow-on bug happened. Gate any HEAD-vs-manifest
  interpretation on "have *we* committed yet."
- **`compact_files` fires Lance's auto_cleanup GC hook** (commits with
  `skip_auto_cleanup=false`, no override) — optimize strips stale `lance.auto_cleanup.*`
  config before compacting to stay non-destructive on upgraded graphs. The strip is a
  separate commit (relevant to the partial-commit retry trap).
- **Lance rebases the common concurrent case for free** — so the data-table conflict usually
  surfaces as the manifest fast-forward, not a Lance error. The Lance-Rewrite-overlap path is
  rare and needs failpoint injection to test.

---

## 10. Verification (the gate)
- `cargo test --workspace --locked` — the canonical gate (matches CI).
- `cargo test -p omnigraph-engine --features failpoints --test failpoints optimize` —
  the optimize concurrency/recovery tests.
- `cargo test -p omnigraph-engine --test write_cost` / `write_cost_s3` (bucket-gated) —
  cost gates (3b adds the schema-once + open-count asserts here).
- `cargo test -p omnigraph-engine --test maintenance` — optimize/repair/cleanup.
- Re-read [`invariants.md`](invariants.md), [`lance.md`](lance.md), [`testing.md`](testing.md)
  before each change (always-on requirement).

Lance source for re-validation:
`/Users/ragnor/.cargo/registry/src/index.crates.io-*/lance-7.0.0` (key files: `io/commit.rs`,
`io/commit/conflict_resolver.rs`, `dataset/optimize.rs`, `dataset/write/retry.rs`,
`dataset/builder.rs`).