apunkt/omnigraph
1
0
Fork 0
mirror of https://github.com/ModernRelay/omnigraph.git synced 2026-07-12 03:12:11 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

Implement RFC-022 unified graph write protocol (#343)

Browse source 
* Implement unified graph write protocol

* Preserve recovery error wire compatibility
This commit is contained in:
Andrew Altshuler 2026-07-11 14:02:54 +03:00 • committed by GitHub
parent 0c8d769501
commit f758ff0d17
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
80 changed files with 13393 additions and 2050 deletions
Download patch file Download diff file Expand all files Collapse all files

26
docs/rfcs/rfc-0018-ingest-wal.md
View file

@ -2,7 +2,7 @@
type: spec
title: "RFC-018 — Streaming-ingest WAL on Lance MemWAL"
description: Adds a durability-first streaming ingest path (ack on WAL durability, asynchronous fold into the graph commit chain) built entirely on Lance's MemWAL primitive; reconciled against Lance v8.0.0 and the v9 beta line; analyzed for composition with the upstream multi-table-commit RFCs.
status: draft
status: superseded
tags: [eng, rfc, wal, ingest, lance, omnigraph]
timestamp: 2026-07-02
owner:
@ -10,12 +10,19 @@ owner:
# RFC-018 — Streaming-ingest WAL on Lance MemWAL
**Status:** Draft / for discussion
**Status:** Superseded by [RFC-026](rfc-026-memwal-streaming-ingest.md)
**Date:** 2026-07-02
**Surveyed version:** 0.7.2 (branch `dst-extract-crate`); Lance pinned at 7.0.0
**Upstream surveyed:** Lance v8.0.0 (released; RC votes closed 2026-07-01), v9.0.0-beta.10; MemWAL spec (`lance.org/format/table/mem_wal/`, fetched in full 2026-07-02); discussions #7260, #7264, #7222, #7176
**Audience:** OmniGraph maintainers
> **Supersession note (2026-07-10):** RFC-026 carries the streaming-ingest
> design forward under RFC-022's unified graph-write protocol. It also corrects
> this draft's characterization of MemWAL: MemWAL is a strategic Lance
> architecture and a major substrate investment, not an experimental direction.
> The integration risk is its evolving API and format surface across Lance
> releases.
---
## 0. TL;DR
@ -125,8 +132,10 @@ fencing primitive.
## 3. Substrate inventory — what Lance provides and how we use all of it
Per the lance.md protocol the MemWAL spec and adjacent pages were fetched in
full (2026-07-02). The spec is **experimental** upstream — risk register in
§10. Inventory, mapped to consumption:
full (2026-07-02). MemWAL is a strategic Lance architecture with substantial
upstream investment. Its API and format surface are still evolving across
releases; §10 treats that maturity boundary as an integration risk. Inventory,
mapped to consumption:
| Lance tooling | Spec/PR | How this RFC uses it |
|---|---|---|
@ -323,8 +332,8 @@ this RFC phases on:
matures it may fit *small in-place updates* better than WAL-upsert-fold;
watch-listed as a possible Phase 4 refinement, not a dependency.
- MemWAL fixes keep landing on v9 betas (#7489 cross-generation block-list on
in-memory scan arms) — confirming the experimental-spec churn risk (§10)
and the value of keeping our exposure transient-state-only.
in-memory scan arms) — confirming that its API/format integration surface is
still moving (§10) and the value of keeping our exposure transient-state-only.
## 7. Composition with upcoming Lance multi-table commits
@ -464,8 +473,9 @@ participates in publication authority.
## 10. Risks
- **MemWAL is experimental upstream** (spec banner; live format votes —
#7418 Status field mid-2026-06). Mitigation is structural: WAL state is
- **MemWAL's API and format surface continues to evolve.** This risk concerns beta-era
API churn, not architectural commitment: Lance has made MemWAL a strategic
streaming-write investment. Mitigation is structural: WAL state is
*transient* (folded then GC'd), so a format change between Lance versions
can be handled by fold-to-quiescent before the bump; no long-lived on-disk
state depends on the MemWAL format. This must stay true — resist any

13
docs/rfcs/rfc-0019-heads-and-fences.md
View file

@ -2,7 +2,7 @@
type: spec
title: "RFC-019 — Heads and Fences: structural O(1) writes without a warm-cache truth fork"
description: Replaces the warm-publish/pinned-open machinery of PR #318 with two structural changes — durable per-table head rows ("refs, not replay") and substrate-native key-conflict fencing (Lance's unenforced-PK KeyExistenceFilter) — landing together as internal schema v5; composes with RFC-018 (WAL ingest) and the upstream multi-table-commit direction.
status: draft
status: superseded
tags: [eng, rfc, write-path, manifest, lance, omnigraph]
timestamp: 2026-07-04
owner:
@ -10,12 +10,21 @@ owner:
# RFC-019 — Heads and Fences
**Status:** Draft / for discussion
**Status:** Superseded by [RFC-023](rfc-023-key-conflict-fencing.md) and [RFC-024](rfc-024-durable-table-heads.md)
**Date:** 2026-07-04
**Surveyed:** omnigraph `main` @ 98530a0e (0.8.0); Lance pinned 7.0.0 (+ vendored lance-table carrying lance#7480); upstream Lance v8.0.0 (released 2026-07-01), v9.0.0-beta.15; PR #318 at `2aab48ba` (reviewed 2026-07-04, 8 verified findings)
**Companion docs:** RFC-018 (streaming-ingest WAL), PR #318's plan doc (`unlimited-history-latency-plan.md`, whose §9 "U2" this RFC promotes from follow-up to prerequisite)
**Audience:** OmniGraph maintainers
> **Supersession note (2026-07-10):** key-conflict fencing and durable table
> heads are separate irreversible decisions with different substrate gates and
> rollout requirements. RFC-023 and RFC-024 review them independently under
> RFC-022's shared graph-write protocol. RFC-023 also corrects this draft's
> load-bearing symmetry claim: on the surveyed Lance revision, an unfiltered
> current transaction (including bare `Append`) can rebase after a filtered
> update. Fencing therefore requires both transaction orders plus a fleet
> compatibility barrier; it is not “unblocked for new tables” in a mixed graph.
---
## 0. TL;DR

773
docs/rfcs/rfc-022-unified-write-path.md Normal file
View file

@ -0,0 +1,773 @@
---
type: spec
title: "RFC-022 — Unified graph-write protocol"
description: One correctness protocol for graph-visible writes, with synchronous recovery, complete read-set arbitration, writer-specific physical-effect adapters, and explicit control-plane exceptions.
status: draft
tags: [eng, rfc, write-path, manifest, recovery, concurrency, lance, omnigraph]
timestamp: 2026-07-10
owner:
---
# RFC-022: Unified graph-write protocol
**Status:** Draft / for team review
**Date:** 2026-07-10
**Surveyed:** OmniGraph 0.8.1 (`main`); Lance 9.0.0-beta.15, git rev `f24e42c1`
**Audience:** engine and storage maintainers
**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md).
Findings marked **BLOCKER** must be dispositioned before acceptance.
---
## 0. Summary
OmniGraph will have one **correctness protocol** for every operation that changes
manifest-resolved graph state. The protocol does not require every writer to use
the same Lance primitive. A mutation can commit one staged transaction per table,
a branch merge can make several commits to a table, and optimize can use Lance
operations that have no staged API. Writer-specific **effect adapters** describe
those physical operations; one coordinator enforces the common safety rules.
A graph-visible write follows this state machine:
```text
recovery barrier
→ prepare pinned base + complete ReadSet + Effects
→ acquire ordered process-local gates
→ revalidate the complete ReadSet, or restart
→ durably arm recovery
→ apply writer-specific physical effects
→ publish exactly one graph-visible __manifest CAS
→ finalize derived state
```
The protocol has three hard boundaries:
1. Recovery is a synchronous pre-write safety barrier. It may also run in the
background, but a writer never proceeds merely because recovery was scheduled.
2. Validation and merge classification are valid only for their complete read set.
A changed probed table causes revalidation or a full restart; the writer never
refreshes expected versions underneath a plan computed from an older base.
3. “One `__manifest` CAS” applies only to graph-visible commits. Native graph-branch
ref creation/deletion and physical-only maintenance have explicit, smaller
control protocols and do not manufacture graph commits.
This RFC deliberately does not combine key fencing, durable table heads,
checkpoint retention, MemWAL ingest, or lineage-based merge-delta discovery into
one format and rollout. They are focused follow-ups:
- [RFC-023 — Key-conflict fencing](rfc-023-key-conflict-fencing.md)
- [RFC-024 — Durable table heads](rfc-024-durable-table-heads.md)
- [RFC-025 — Checkpoint retention](rfc-025-checkpoint-retention.md)
- [RFC-026 — MemWAL streaming ingest](rfc-026-memwal-streaming-ingest.md)
- [RFC-027 — Lineage merge deltas](rfc-027-lineage-merge-deltas.md)
## 1. Scope and authority
### 1.1 Graph-visible writes
A **graph-visible write** changes state resolved through a graph manifest snapshot:
- a node or edge table version;
- a registered or tombstoned table;
- accepted schema identity or schema-visible table metadata;
- a graph commit or graph head;
- any future logical marker that changes query or time-travel semantics.
For such an operation, the only visibility point is one successful `__manifest`
commit containing the entire graph delta. Per-table Lance commits before that point
are physical effects covered by recovery; they are not independently graph-visible.
### 1.2 Control operations
Two classes are not graph-visible commits:
1. Native graph-branch ref create/delete. Lance stores these refs outside the
dataset-version chain and creates no new `__manifest` version for either action.
2. Physical-only maintenance whose result is content-equivalent and is not selected
through a data-table pointer in `__manifest`, such as compacting `__manifest`
itself or reclaiming unreachable files.
Sections 7 and 8 define their control protocols. Branch **merge** and a data-table
optimize whose new version must be published are graph-visible writes and remain in
the main protocol.
### 1.3 What “unified” means
Unified means one set of safety obligations, one coordinator state machine, and one
closed registry of effect adapters. It does **not** mean:
- one storage trait method for every Lance operation;
- exactly one physical commit per table;
- pretending native refs are manifest rows;
- treating process-local queues as a distributed transaction manager;
- moving commit recovery out of the correctness path.
## 2. Protocol objects
The names below are conceptual. Implementations may choose different Rust names,
but they must preserve the represented information and transitions.
```rust
struct PreparedWrite {
operation_id: OperationId,
writer_kind: WriterKind,
target: BranchTarget,
base: BaseView,
read_set: ReadSet,
effects: Effects,
manifest_delta: ManifestDelta,
lineage_intent: Option<LineageIntent>,
recovery: RecoveryPlan,
}
```
### 2.1 `BaseView`
`BaseView` is the immutable state against which the operation was computed. It
contains at least:
- target manifest branch and its incarnation/freshness token;
- pinned manifest version;
- accepted schema identity;
- pinned table entries used by the operation;
- graph head when parentage, merge base, or branch semantics depend on it.
The base is captured only after the recovery barrier completes. A recovery pass may
advance the manifest or promote schema state, so a base captured before recovery is
not valid write input.
### 2.2 `ReadSet`
`ReadSet` is every authority value whose stability is required for the prepared
result to remain correct. It includes, as applicable:
- target branch incarnation;
- accepted schema identity;
- graph head;
- the table head for every table written;
- the table head for every table probed by uniqueness, referential-integrity,
cardinality, policy-independent structural validation, or merge classification;
- writer-specific authority, such as an enrolled stream's configuration and merge
generation.
A table belongs in the read set because its value affected the decision, not because
the writer happens to update it. Read-only dependencies are load-bearing.
The publisher must arbitrate every read-set member atomically with the manifest
commit. A fresh pre-check followed by an unconditional write is not a CAS. The
implementation must ensure that a concurrent change after the check contends on a
stable authority row or equivalent substrate token. If the current representation
cannot arbitrate a read-only dependency, that writer has not completed this RFC and
must not ship the outside-gate validation optimization.
### 2.3 `Effects`
`Effects` is an adapter-owned physical plan. Each effect declares:
- physical dataset and Lance branch/ref it targets;
- expected pre-state;
- whether it is staged, inline, ref-only, or zero-commit;
- the possible post-state shape: exact version, bounded range, or adapter-confirmed
version;
- whether rollback is safe;
- how recovery distinguishes no movement, partial movement, completed movement,
and an already-published result.
The generic coordinator never assumes `expected_version + 1`. That assumption is
valid for some mutation/load effects and false for branch merge, compaction, index
work, schema metadata changes, and no-op plans.
### 2.4 `ManifestDelta`
`ManifestDelta` is the complete logical result to publish. It contains table-version
journal entries, registrations/tombstones, mutable logical heads when present, schema
identity, and lineage rows as required by the operation. Its logical contents are
immutable for one prepared attempt. Physical version fields may be declared output
slots that the adapter binds to an allowed, confirmed effect result; binding such a
slot cannot widen or otherwise change the logical plan. A revalidation mismatch
discards the delta and restarts preparation.
### 2.5 `RecoveryPlan`
`RecoveryPlan` supplies the writer-specific classifier and compensation/roll-forward
rules. It must be serializable into a recovery sidecar before the first independently
durable physical effect. It is part of the commit protocol, not a best-effort
maintenance hint.
## 3. Normative invariants
1. **Recovery before base capture.** Every graph-visible writer runs and awaits the
recovery barrier before pinning `BaseView`.
2. **No durable effect before durable intent.** On an existing physical ref,
reclaimable uncommitted files may be staged first. A first-touch named-table
transaction is different: Lance writes its uncommitted files into the opened
branch tree, so Prepare retains the logical batch/predicate and pre-mints its
transaction identity; after revalidation the recovery sidecar becomes durable,
then the writer creates the target ref and stages those branch-local files under
the held gates. In every case the sidecar precedes the first independently
persistent physical effect, including a native table ref, data-table HEAD
advance, or native tag/index mutation needed by the graph-visible result.
3. **Complete read-set validation.** Physical effects may start only while the fresh
authority state equals the complete `ReadSet` used to prepare the operation.
4. **Recompute, do not patch.** On a pre-effect mismatch, discard the prepared plan
and rerun validation/classification. Never refresh expected versions beneath an
old plan.
5. **One graph visibility point.** A graph-visible write publishes one manifest CAS;
no subset of its table effects becomes graph-visible independently.
6. **Adapters own effect truth.** Every effect-producing writer uses a registered
adapter. There is no generic fallback that guesses version movement or recovery
safety.
7. **Queues are local optimization.** Ordered process-local gates prevent avoidable
in-process races and deadlocks. Cross-process correctness comes from Lance
conflicts, manifest arbitration, and recovery.
8. **Recovery safety is synchronous.** Background sweeping is permitted, but a later
overlapping writer waits for, performs, or fails on unresolved recovery before it
can advance state.
9. **Derived work follows visibility.** Expensive index reconciliation, cache warming,
and orphan reclaim may run asynchronously only when logical correctness does not
depend on their completion.
These refine, and do not weaken, invariants 2, 3, 4, 5, 7, 9, 13, and 15 in
[`docs/dev/invariants.md`](../dev/invariants.md).
## 4. The graph-write state machine
### 4.1 Stage A — recovery barrier
Before accepting a base, the coordinator discovers pending recovery intents that can
overlap this operation. It must reach one of three outcomes:
1. all relevant intents are fully resolved;
2. each remaining intent is proven already satisfied and can be finalized safely;
3. the write fails with a typed recovery-required or live-writer-contention error.
“Spawn recovery and continue” is not an outcome. Listing sidecars may run concurrently
with non-authoritative I/O, but preparation cannot accept its base until recovery has
completed and the post-recovery schema/manifest state is available.
Recovery must not destructively act on a sidecar that may belong to a live writer
without ownership/fencing proof. It waits, returns contention, or uses a protocol that
proves the prior owner cannot continue.
Recovery classifiers use exact effect identity or confirmed post-state. A numeric
test such as `manifest_version >= observed_lance_head` is sufficient only when the
adapter independently proves lineage containment; version ordering alone is not that
proof. Exact identity cuts both ways: a roll-forward that loses its manifest CAS to
a concurrent writer which already published this intent's exact goal state is
**convergence, not failure** — the barrier records the audit outcome and removes the
intent rather than failing the write (preserving the concurrent-advance convergence
behavior fixed in #296). "Exact" forbids adopting an unrelated newer version; it does
not forbid recognizing one's own goal already achieved.
The barrier is a deliberate availability trade, stated plainly: an unresolvable
overlapping recovery intent — including a live foreign writer's sidecar that cannot
be fenced — fails every overlapping write with a typed error until resolved. Safety
outranks availability here by design; operators observe the condition through the
typed error and recovery telemetry rather than through silently degraded writes.
### 4.2 Stage B — prepare
Prepare runs without writer gates. It may:
- capture `BaseView`;
- evaluate the complete constraint set;
- classify a merge;
- compute embeddings or other deterministic payloads;
- build staged Lance transactions and reclaimable uncommitted objects for targets
whose physical refs already exist;
- for a first-touch named-table target, retain the complete logical stage input and
pre-mint its recovery transaction identity without creating the ref;
- construct the complete `ReadSet`, `Effects`, `ManifestDelta`, and `RecoveryPlan`.
Prepare must not advance a Lance HEAD or the graph's native branch refs. Staged files
that are not referenced by a committed Lance manifest are permitted and are
reclaimable if the attempt restarts.
After Stage E arms recovery, a first-touch adapter may create its declared target ref
under the held gates, stage branch-local files on that ref, bind the resulting Lance
transaction to the pre-minted identity, and only then advance HEAD. The ref is itself
an independently durable effect covered by the sidecar; recovery must classify both
sidecar-before-ref and ref-before-HEAD crash states.
Every validator registers its actual committed-state probes in `ReadSet`. Key fences
are an additional same-key conflict signal; they do not replace read-set arbitration
for non-key uniqueness, RI, cardinality, or merge-target stability.
### 4.3 Stage C — acquire ordered gates
The prepared effects declare their gate keys. One attempt acquires the complete set in
this total order:
1. durable global operation claims (migration, retention, or future claims) by
deterministic claim key;
2. graph/schema-control key, if required;
3. target branch-control key, if required;
4. `(physical_branch, table_key)` keys in deterministic sorted order.
The gates are held through revalidation, recovery arming, physical effects, and the
manifest visibility decision. They may be released after a successful manifest CAS or
after a failed post-effect attempt has safely left its sidecar for recovery.
Acquiring a global claim is coordination, not a graph effect for §3's sidecar-order
rule: the claim record must itself contain an owner/fencing token and an explicit
crash-release/takeover contract. No data HEAD, tag, index, or logical authority may
move merely because the claim was acquired.
The merge-exclusive mutex may protect a coordinator swap, but it is not a semantic
cross-process lock and must not substitute for a target read set.
### 4.4 Stage D — revalidate or restart
With all gates held, the coordinator loads fresh authority state and compares every
member of `ReadSet`.
- If all members match, the attempt may arm recovery.
- If any member differs, no physical effect may run. The attempt releases its gates,
discards staged state, and restarts from Prepare.
- A strict API may return a typed conflict instead of retrying, but it may not publish
the stale plan.
Retries are bounded and observable. Retrying a merge means recomputing the merge base
and reclassifying against the new target. Retrying a validation-sensitive mutation
means rerunning the validators, including probes of tables the mutation does not
write.
### 4.5 Stage E — arm recovery
After successful revalidation, write the recovery sidecar durably before the first
independently durable physical effect. An effect-free or authority-first workflow
described in §4.9 may omit the sidecar. Otherwise the sidecar contains at least:
- operation id, writer kind, actor, and target branch/incarnation;
- pinned schema identity and complete read set;
- every physical target and expected pre-state;
- adapter recovery strategy;
- intended manifest delta and lineage intent, or a durable reference to them;
- confirmed post-state once the adapter reaches its all-effects-complete boundary.
A multi-step adapter first records its pre-state plan. After all its physical effects
finish, it durably records the exact confirmed post-state before manifest publish.
Until that confirmation exists, recovery treats the effect set as possibly partial.
### 4.6 Stage F — apply effects
The adapter applies its declared physical effects. A failure after recovery is armed
leaves the sidecar intact. The request must not delete it, silently adopt live HEAD, or
start a fresh plan around it.
The adapter returns exact achieved state to bind the physical output slots declared by
`ManifestDelta`. If achieved state differs from the prepared effect envelope, the
operation fails into recovery rather than widening its plan in place.
### 4.7 Stage G — manifest CAS
A graph-visible operation performs exactly one `__manifest` CAS carrying its complete
logical delta and lineage when the plan has `LineageIntent`. Metadata-only plans carry
their explicit authority/operation rows without manufacturing graph lineage. On every
CAS attempt, the commit authority re-reads and arbitrates the complete `ReadSet`.
The following cases are distinct:
- **Conflict before any physical effect:** safe bounded restart from Prepare.
- **Conflict after physical effects:** recovery case. Keep the sidecar; do not simply
re-stage or point the manifest at whatever HEAD is now live.
- **CAS success:** the graph commit is visible atomically.
When durable table heads land under RFC-024, tombstoning a table must update its mutable
head to an explicit deleted state in this same CAS. A stale live head plus an immutable
tombstone history is not a valid O(tables) current-state representation.
### 4.8 Stage H — finalize
After CAS success:
- delete the sidecar best-effort when the workflow has one;
- refresh/invalidate process-local views;
- enqueue derived index reconciliation and orphan reclaim;
- record recovery/audit completion when applicable.
Sidecar deletion failure does not turn a durable successful graph commit into a user
error. The next recovery barrier proves the exact intent satisfied and removes the
artifact.
### 4.9 Authority-first control workflows
Some metadata workflows have no independently durable physical effect before their
manifest CAS. They may use an **authority-first** subtype:
1. run the recovery barrier, prepare, acquire gates/claims, and revalidate exactly as
above;
2. publish the metadata transition as the first durable effect;
3. perform only idempotent work whose desired target is fully encoded by that
transition and whose interruption cannot expose incorrect graph data.
No generic sidecar is required before step 2 because there is no pre-authority effect
to recover. The authority row itself is the durable recovery cursor. Checkpoint
deletion, a GC-boundary publish, and `OPEN -> DRAINING` stream intent are candidate
examples; each follow-up RFC must prove its post-CAS work is convergent and safe.
A long control workflow is not one giant RFC-022 attempt. Each lifecycle transition,
fold, schema/branch operation, and resume transition is a separate prepared write or
native-ref control step with its own read set and visibility point. If a later phase
needs a non-idempotent or independently visible effect not fully described by the
authority row, that phase uses a normal sidecar before the effect.
## 5. Crash contract
| Crash point | Required result |
|---|---|
| Before sidecar | No independently durable effect occurred; uncommitted objects are reclaimable. |
| Sidecar durable, no effect | Recovery aborts/finalizes the empty intent. |
| Some effects applied, not confirmed | The adapter rolls back, completes, or refuses safely according to its declared strategy. |
| All effects confirmed, before manifest CAS | Recovery rolls forward the exact confirmed manifest delta, or applies the adapter's explicit all-or-nothing rule. |
| Manifest CAS succeeded, sidecar remains | Recovery proves the exact intent visible, audits it, and removes the sidecar. |
Rollback is not assumed safe. Lance `Restore`, schema-file promotion, native refs, and
content-replacing operations have different concurrency properties; each adapter must
state which recovery direction is legal and under what fencing.
## 6. Writer-effect adapters
The adapter registry is closed by default: adding a graph-visible writer requires a
new adapter or an explicit use of an existing adapter. Code review and tests must be
able to enumerate every adapter and every entry point that invokes it.
### 6.1 Mutation and load
- Construct one staged effect per touched table where the Lance API permits it.
Existing-table effects may stage in Prepare; first-touch named-table effects use
the sidecar → target ref → branch-local stage ordering above.
- Put every uniqueness, RI, and cardinality probe in `ReadSet`.
- Revalidate or restart when a probed-but-untouched table changes.
- Preserve strict replacement semantics for overwrite/delete.
- Treat key-conflict fencing and strict keyed Append semantics as RFC-023 concerns;
no fence is credited as protection until that RFC's rollout gates pass.
### 6.2 Branch merge
- Compute row classification outside the gates.
- Include the target graph head and every target table used by classification or
validation in `ReadSet`.
- Any target change before effects forces a complete reclassification; publishing a
result computed against an old target and parenting it to a new live head is
forbidden.
- The adapter supports zero, one, or several physical commits per table and records
exact confirmed post-state before manifest publish.
- Lineage-based candidate discovery may replace the classifier only under RFC-027;
this protocol does not assume it is O(delta).
### 6.3 Schema apply and storage migration
- Acquire the schema-control gate before effect application.
- Include accepted schema identity and every affected table in `ReadSet`.
- Cover schema staging-file promotion, data-table schema/field-metadata commits,
registrations, tombstones, and final schema identity with one recovery intent.
- Write the sidecar before the first table HEAD advance, including unenforced-PK
metadata backfill or other inline metadata commits.
- A branch-wide or graph-wide migration must enumerate every physical manifest/data
branch it changes; updating main does not implicitly migrate older branch manifests.
### 6.4 Data-table optimize and index work
- The adapter may describe zero or multiple inline, content-preserving Lance commits.
- It records the exact achieved version rather than assuming one version of movement.
- If the new data-table version is selected through `__manifest`, publishing that
pointer is a graph-visible commit and uses this protocol.
- Logical operations never fail because a derived index is absent or behind.
- Physical-only internal-table maintenance remains the exception in Section 8.
### 6.5 MemWAL fold
RFC-026 owns enrollment, acknowledgement, quiescence, fresh-read semantics, and the
public ingest surface. Any fold that becomes graph-visible is an adapter here:
- fold-time validation contributes its complete read set;
- Lance `merged_generations` changes atomically with the base-table data commit;
- the sidecar covers the data-commit-to-manifest gap;
- one successful manifest CAS makes the folded graph state visible.
## 7. Native graph-branch ref control protocol
Creating or deleting a graph branch mutates a native Lance branch ref for
`__manifest`. Lance specifies that these operations do not generate a dataset version.
There is no target branch on which to publish before create, and no target remains on
which to publish after delete. They therefore cannot truthfully be instances of the
graph-visible manifest-CAS protocol.
Their control protocol is:
1. run and await the recovery barrier;
2. quiesce enrolled streams as required by RFC-026;
3. acquire any active global claim, such as RFC-025's retention claim, and then
the graph/branch-control gate in §4.3 order;
4. freshly revalidate source ref, target existence, and branch incarnation;
5. perform one native Lance ref mutation, which is the visibility point;
6. release the gate and reclaim orphaned per-table forks asynchronously.
Delete has one recovery disposition that create does not: after the complete
schema/target-branch/all-table gate set has waited out any live in-process owner,
an unresolved sidecar scoped to the branch being removed may be rendered
unreachable by the native ref deletion. A later heal records the orphan-discard
audit and retires it. Graph-global schema recovery still blocks the control, and
create/merge may not adopt this exception.
The native ref operation itself should enforce the freshly checked precondition or
surface concurrent ref mutation as a conflict — but at the pinned Lance revision it
does not: branch-ref creation is an existence check followed by an unconditional
put, not a conditional primitive (the same fact for which RFC-025 §2.3 rejects a
branch ref as a claim mechanism). Until Lance ships a conditional/CAS ref mutation,
graph-branch create/delete therefore inherit the documented single-writer-process
support boundary — the same disposition RFC-023 §10 applies to recovery ownership —
and multi-process branch operations are not advertised. The upstream ask for a
conditional ref primitive is filed alongside this RFC; a process-local branch gate
remains a local optimization, not the missing cross-process guarantee.
These operations do not emit a synthetic graph commit. If a future product contract
requires a native ref mutation and manifest/audit rows to become atomic together, it
needs a separate multi-authority recovery protocol; this RFC does not claim an
atomicity the substrate does not provide.
This exception applies only to graph-level create/delete. Branch merge is a
graph-visible write. Lazy per-table forks created while preparing a branch write are
declared physical effects of that writer and remain subject to its recovery/reclaim
contract.
## 8. Physical-maintenance control protocol
Physical work that does not change manifest-resolved logical graph state does not
create graph lineage merely to fit the main protocol. Examples include:
- compacting `__manifest`, which is itself the authority and is read at its Lance HEAD;
- deleting versions/files already proven unreachable under the active retention
contract;
- reclaiming orphaned branch refs or uncommitted objects;
- rebuilding derived physical state when no graph-visible data-table pointer changes.
Such work must still be idempotent, bounded, observable, and safe under concurrent
native Lance commits. It uses substrate conflict/retry semantics appropriate to the
operation. It must never expose partial logical graph state.
The exception is from graph-lineage publication, not from recovery safety. Maintenance
must run the recovery barrier or refuse before it can replace or delete an artifact
named by an unresolved sidecar. It must also acquire any relevant process-local gates;
as elsewhere, those gates are an optimization rather than cross-process authority.
Data-table compaction/index work that advances a version which graph reads must select
through `__manifest` is not exempt; Section 6.4 applies. Checkpoint reachability and
the mapping from graph checkpoint rows to Lance-native GC pins belong to RFC-025.
## 9. Concurrency and retry semantics
Process-local gates reduce same-process races and establish one deadlock-free order.
They do not coordinate two servers or CLIs. A conforming implementation remains safe
if every process has its own gate manager.
Cross-process safety comes from:
- complete read-set arbitration at the manifest authority;
- Lance transaction conflicts for physical table effects;
- durable recovery before physical HEAD movement;
- refusal to continue past unresolved overlapping recovery.
Retry rules are phase-specific:
- before effects, a read-set mismatch discards and recomputes the whole attempt;
- while applying an effect, the adapter may retry only when Lance guarantees the
operation can be safely replanned from fresh physical state;
- after any effect, manifest contention is resolved through the armed recovery intent,
not by silently rebasing the logical plan;
- retry exhaustion returns a typed, observable conflict.
## 10. Rollout and compatibility
This RFC authorizes a protocol refactor, not a manifest v5 format moment. RFC-023
through RFC-027 own their respective format and public-surface changes.
It also does not require a mutable-tip `GraphState` singleton. Three measured,
local latency fixes can land independently of the adapter conversion:
1. make the graph's shared Lance `Session` a required parameter of every
manifest open/publisher path, so remote opens do not rebuild clients and cold
metadata state;
2. capture one immutable operation-local manifest/lineage view and pass it down
the call stack instead of reopening the same state repeatedly; and
3. remove the verified-redundant branch-idle refresh and the back-to-back second
`branch_delete_as` refresh once their existing coverage asserts unchanged
behavior.
These are narrow access-shape fixes, not a second commit-input authority. They
must preserve snapshot pinning and still cross the recovery/read-set barriers
defined above.
Implementation proceeds in this order:
1. Introduce `PreparedWrite`, `ReadSet`, effect-adapter, and recovery-plan concepts
while preserving existing behavior.
2. Ship conservative branch-wide arbitration first. Mutation/load captures
`(Lance BranchIdentifier, exact optional graph_head, accepted schema identity)`;
every publisher retry compares that token instead of reparenting. Because every
supported graph-content and schema apply advances `graph_head:<branch>` before
schema promotion, the shared head row atomically arbitrates probed-but-untouched
same-branch dependencies. The native branch identifier detects delete/recreate
ABA under the documented single-writer-process branch-control boundary. RFC-024
later narrows false contention with table heads; it is not a correctness
prerequisite for this coarse step. Existing live committed-state validation
probes remain until the narrowed read set replaces them.
> **Implementation note (2026-07-11):** mutation/load now use this coarse
> token, schema-v3 exact-effect sidecars, fixed lineage/rollback outcome ids,
> zero transparent Lance commit retries, and bounded full reprepare before
> effects. Branch merge remains on its writer-specific multi-commit path, but
> now holds the root-shared schema plus source/target branch gates from its
> strict recovery barrier and authority capture through publication. It plans
> with an accepted-contract catalog captured under that schema gate, then
> acquires all catalog table gates for source and target, re-lists recovery,
> and compares fresh manifest versions before Phase A. This closes
> same-process delete/recreate ABA and legacy table-only-writer races while
> its full exact-effect adapter remains future work. Schema apply,
> optimize/index, and MemWAL fold remain on their writer-specific paths until
> their adapter slices land.
3. Convert mutation/load, branch merge, schema apply/migration, data-table optimize,
and graph-visible index work one adapter at a time.
4. Add static or runtime enumeration proving no graph-visible entry point bypasses the
coordinator.
5. Delete superseded writer-specific orchestration only after its crash and
concurrency cells pass through the adapter.
6. Optimize background recovery latency only after the synchronous barrier and all
recovery classifications remain intact.
Mixed writer binaries are not made safe by process-local gates. A deployment may
enable the new protocol only when every writer that can reach the graph obeys the same
sidecar and manifest-arbitration contract, or when a compatibility gate rejects older
writers.
## 11. Required tests and cost gates
### 11.1 Protocol conformance
- Enumerate every graph-visible entry point and its adapter.
- Assert no sidecar-backed adapter can create an independently durable physical
effect before its sidecar is durable.
- Enumerate authority-first workflows and assert their CAS is the first durable
effect and every post-CAS action is idempotently derivable from its authority row.
- Assert one graph-visible operation produces exactly one manifest visibility commit.
- Assert branch create/delete and physical-maintenance exceptions produce no synthetic
graph lineage.
### 11.2 Read-set races
- Two distinct ids racing on the same non-key `@unique` value cannot both publish.
- An edge insert racing deletion of an endpoint must revalidate or conflict.
- Cardinality probes of an untouched table participate in arbitration.
- A target advance after merge classification forces complete reclassification.
- Run the same cells with separate `Omnigraph` handles sharing one root-scoped
process-local gate manager, then with separate processes that do not share it.
### 11.3 Recovery
- Fail before sidecar, after sidecar, after each physical effect, after confirmation,
after manifest CAS, and during sidecar deletion for every adapter.
- A later overlapping writer blocks, heals, or returns recovery-required; it never
advances around the sidecar.
- A live foreign writer's sidecar is not destructively recovered without fencing.
- Recovery proves exact effect identity; a numerically newer unrelated version is not
accepted as proof of ancestry.
### 11.4 Adapter-specific truth tables
- Mutation/load: zero/one table, multi-table, strict, non-strict, and
probed-but-untouched dependency cases.
- Merge: adopt, rewrite, multi-commit, no-op, target advance, and partial-Phase-B crash.
- Schema: staging files, registration-only, metadata HEAD advance, partial multi-table
migration, and branch-local state.
- Optimize/index: zero, one, and several Lance commits, including monotonic publish and
retryable physical contention.
- MemWAL fold, when RFC-026 lands: merged-generation conflict and every fold crash
boundary.
### 11.5 Cost gates
The protocol must not move validation, classification, embedding computation, or
staged-file construction under process-local gates. Measure gate hold time separately
from Prepare. Correctness gates precede latency optimization: a cost regression can
delay rollout, but it cannot justify touched-table-only validation or asynchronous
recovery safety.
## 12. Invariants and deny-list check
This design reinforces the existing architecture:
- Lance and `__manifest` remain the sources of truth; `PreparedWrite` is immutable
attempt-local state, not a shadow mutable tip.
- Graph visibility remains manifest-atomic.
- Recovery remains part of the commit protocol.
- Logical constraints fail loudly and are revalidated when their inputs change.
- Derived indexes and reclaim work converge without becoming logical commit points.
- No custom WAL, transaction manager, buffer pool, or distributed lock is introduced.
The design rejects two tempting deny-list violations: treating process-local queues as
distributed correctness, and treating recovery as derivable background work that a
new writer may outrun.
Acceptance should also add one clarification to invariant 15: a view of immutable,
version-pinned state may be cached, while an in-memory view of the mutable tip is only
a hint. Every use of mutable-tip state as write input must be re-arbitrated by the
commit authority. Durable heads under RFC-024 are one possible authoritative
representation; this protocol does not require or bless a warm parallel truth path.
## 13. Drawbacks and rejected alternatives
### 13.1 One generic staged-storage method
Rejected. Lance does not expose staged forms for every operation, and existing writers
have materially different version movement and rollback safety. A generic method would
either lie about those differences or accumulate writer-kind conditionals. One
protocol plus explicit adapters has lower long-run liability.
### 13.2 Asynchronous heal with optimistic continuation
Rejected. Current recovery classification relies on later writers not advancing past
an unresolved sidecar. Scheduling a sweep does not establish that fact. Recovery may
be proactively asynchronous, but the next writer still crosses a synchronous barrier.
### 13.3 Touched-table-only CAS
Rejected. Non-key uniqueness, RI, cardinality, and merge classification read tables
the operation may not write. Ignoring those dependencies admits commits that were never
valid against one serial graph state.
### 13.4 Treat every state change as a manifest commit
Rejected. Native branch refs and physical maintenance have different substrate
visibility points. Manufacturing manifest commits would add coordination without
making the native mutation atomic with them.
## 14. Reversibility
The in-memory coordinator and adapter refactor is reversible. Recovery-sidecar schema
changes must be versioned and backward-compatible during rollout. This RFC alone does
not authorize a `__manifest` schema-stamp bump, a public wire change, or a new storage
substrate.
The correctness contract is intentionally difficult to reverse: after writers rely on
complete read-set arbitration and recovery-before-write, weakening either would
reintroduce silent integrity or recovery races. Focused irreversible changes are
reviewed in RFC-023 through RFC-027.
## 15. Follow-up RFC boundaries
- **RFC-023** owns PK annotation, fenced merge routing, strict keyed Append behavior,
mixed fenced/unfenced rollout, conflict mapping, and both commit-order tests.
- **RFC-024** owns mutable table-head rows, explicit live/tombstoned head state,
current-state read shape, migration, and cost gates.
- **RFC-025** owns checkpoint rows, Lance-native physical pins, cleanup reachability,
pruned-through semantics, and checkpoint/cleanup crash ordering.
- **RFC-026** owns MemWAL enrollment, durable acknowledgements, fold/dead-letter
behavior, stream quiescence, fresh reads, public surfaces, and upgrade fencing.
- **RFC-027** owns candidate discovery from row lineage, deletion discovery, fallback
semantics, and evidence for any O(delta) merge claim.
Those RFCs call this protocol when they produce a graph-visible write. None may weaken
the recovery barrier, omit read dependencies from `ReadSet`, or create a second graph
visibility point.

480
docs/rfcs/rfc-023-key-conflict-fencing.md Normal file
View file

@ -0,0 +1,480 @@
---
type: spec
title: "RFC-023 — Substrate-native key-conflict fencing"
description: Make concurrent keyed writes fail or retry through Lance's transaction conflict filters, forbid keyed Append, and activate the contract only behind an all-branch fleet/format barrier.
status: draft
tags: [eng, rfc, write-path, concurrency, lance, primary-key]
timestamp: 2026-07-10
owner:
---
# RFC-023: Substrate-native key-conflict fencing
**Status:** Draft / for team review
**Date:** 2026-07-10
**Surveyed:** omnigraph 0.8.1 (`main`); Lance 9.0.0-beta.15 at git rev
`f24e42c1`; full Lance transaction, table-schema, read/write, branching, and
MemWAL specifications; pinned Rust conflict-resolver and merge-insert sources
**Relationship to RFC-022:** this RFC is the fencing decision split from the
earlier monolithic RFC-022 draft. [RFC-022](rfc-022-unified-write-path.md)
defines the shared write/recovery protocol; this RFC owns the substrate,
compatibility stamp, and rollout requirements for key conflicts. It may share a
release with [RFC-024](rfc-024-durable-table-heads.md), but neither RFC depends
on the other's storage decision.
**Audience:** engine, storage, migration, and release maintainers
**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md).
Findings marked **BLOCKER** must be dispositioned before acceptance.
---
## 0. Decision summary
OmniGraph will use Lance's unenforced-primary-key merge-insert filter as the
key-conflict primitive. It will not emulate the filter in the engine and will
not add a lock table or custom transaction manager.
The guarantee is deliberately narrow and strong:
> For a keyed table, two concurrent operations that may insert the same `id`
> MUST NOT both commit silently. They either serialize, retry from a new graph
> base, or fail loudly.
That guarantee cannot be rolled out by annotating a few new tables in an
otherwise v4 graph. Lance's current mixed filtered/unfiltered conflict handling
is directional, and a bare `Append` can commit after a filtered update. The
contract therefore activates only after all supported writers and every table
state reachable from every graph branch have crossed a fencing-compatible
format barrier.
Normative decisions:
1. Node and edge table PK = `id`.
2. Bare Lance `Append` is forbidden for keyed tables.
3. Every keyed insert/upsert path produces the same Lance key filter.
4. Mixed-version serving is forbidden during activation; the fencing-compatible
format stamp is written last and older binaries refuse it.
5. PK metadata is permanent and preserved by every later schema/data rewrite.
6. Existing-table migration covers every graph branch, including lazy-inherited
table states, and is recoverable by rolling forward.
7. Fencing does not replace read-set OCC for `@unique`, RI, or cardinality.
## 1. Problem
OmniGraph validates duplicate `@key` values inside one delta, but today two
processes can both read a base where `id = K` is absent, stage disjoint Lance
fragments containing `K`, and let Lance rebase both commits. The graph manifest
CAS orders graph publication; it does not tell Lance that the two data-table
transactions inserted the same logical key.
The result is worse than an ordinary write conflict: both callers can receive
success while a keyed table contains duplicate IDs. Every subsequent upsert,
edge lookup, uniqueness check, and merge then operates on a broken identity
relation.
The desired primitive already exists in Lance. The missing work is to use it on
every keyed path and to define a rollout that never admits an unfenced writer.
## 2. Substrate facts and the directional asymmetry
This section is load-bearing. Tests pin every statement before implementation.
### 2.1 Filter attachment
On the pinned Lance revision, a v2 merge-insert whose ON field IDs exactly equal
the schema's unenforced PK field IDs attaches an `inserted_rows_filter` to its
`Operation::Update`. The filter contains keys for rows classified as inserts;
updates of existing rows continue to use Lance's affected-row / fragment
conflict machinery.
The legacy indexed merge path does not attach this filter. Therefore a BTREE on
`id` can route an otherwise correct merge onto an unfenced path unless the
caller disables that path or Lance wires the filter into it.
### 2.2 Conflict compatibility is directional
Lance transaction compatibility is evaluated from the transaction currently
attempting to commit against transactions that committed after its read
version. It is not implicitly symmetric.
At `f24e42c1`, `check_update_txn` behaves as follows:
- current `Some(filter)` vs committed `Some(filter)` — compare field IDs and
filter intersection; overlap or incompatible filter configuration retries;
- current `Some(filter)` vs committed `None` — retry conservatively;
- current `None` vs committed `Some(filter)` — no corresponding conservative
arm; it may rebase;
- current bare `Append` vs committed filtered `Update` — `Append` treats the
`Update` as compatible.
Consequently, "filtered vs unfiltered conflicts" is not a sufficient rollout
argument. Commit order matters. A filtered writer can win first and a stale
unfiltered writer or bare keyed append can still land second.
### 2.3 What a PK annotation does not do
The key is explicitly *unenforced*. Merely setting the metadata:
- does not validate historical uniqueness;
- does not make bare appends unique;
- does not protect a merge whose ON set differs from the PK;
- does not repair an existing duplicate;
- does not replace OmniGraph's semantic validators.
## 3. Scope and non-goals
In scope:
- node and edge data tables;
- mutation, load, branch merge, recovery replay, and WAL fold paths;
- new-table creation and all-branch existing-table activation;
- schema/overwrite preservation of PK metadata;
- typed retry behavior and coverage gates.
Out of scope:
- using a composite edge PK (`src`, `dst`);
- enforcing arbitrary `@unique` groups through the Lance PK;
- keyless streaming-table deduplication;
- a custom OmniGraph WAL, lock table, or transaction manager;
- declaring general multi-process writes supported before foreign-process
recovery-sidecar ownership is solved.
## 4. Table classes and PK contract
### 4.1 Keyed graph tables
Every normal node and edge table has a non-null `id` field. Its Lance schema
MUST mark exactly that field as the unenforced PK. For edges, `src` and `dst`
remain ordinary fields governed by referential-integrity and cardinality
validation. An edge endpoint move is an update of the row identified by `id`.
The PK field is addressed by stable field ID, not column position or mutable
name. Until rename-stable OmniGraph type/property identity is closed, the
fencing migration cannot claim rename safety.
### 4.2 Keyless append-only tables
A table explicitly declared append-only may omit a PK. Such a table may use
Lance `Append`, including MemWAL append-only operation. It receives no
same-logical-key guarantee because it has no logical key.
Current node and edge types are not in this class: both have graph identity on
`id`. The class is reserved for an explicitly designed internal or future
non-graph table surface.
The distinction is catalog-derived and first-class. Callers do not choose it
with an ad-hoc flag.
## 5. Normative write routing
| Logical operation | Keyed table | Keyless append-only table |
|---|---|---|
| Strict insert / load append | merge-insert ON exactly `id`, `WhenMatched::Fail`, filtered path | `Append` allowed |
| Upsert / load merge | merge-insert ON exactly `id`, `WhenMatched::UpdateAll`, filtered path | workload-specific |
| Fast-forward branch merge of new rows | filtered merge-insert with `WhenMatched::Fail`, even when every row was classified new | `Append` allowed |
| WAL upsert fold | filtered merge-insert with `merged_generations` | append transaction allowed |
| Update existing row | merge-insert/update with affected-row conflict metadata | workload-specific |
| Delete | staged Lance delete; PK filter is not the delete-conflict primitive | staged Lance delete |
| Overwrite | staged overwrite whose output schema preserves the exact PK | staged overwrite |
### 5.1 No keyed `Append`
The prohibition includes internal optimization paths. A caller may not infer
"all rows are new" and switch a keyed table to `stage_append`: that inference
was made against a snapshot and is exactly what a concurrent same-key writer can
invalidate.
Routing through merge-insert does not collapse strict and upsert semantics.
Strict `load --mode append` and other insert-only surfaces use
`WhenMatched::Fail`; a row already present at the pinned base or discovered at
execution remains an error. Only declared upsert surfaces use `UpdateAll`.
The storage trait and `forbidden_apis` guard MUST make a keyed append difficult
to express accidentally. The fast-forward merge optimization is retained only
for keyless append-only tables unless Lance ships a key-filtered append
transaction.
This prohibition knowingly retires a measured fix. The fast-forward append
path exists because a whole-delta merge-insert join exhausted the query memory
pool on embedding-bearing tables (#277); routing adopted rows back through a
filtered merge-insert re-exposes that workload to join memory behavior. The
regression class is therefore a named ship gate: the fenced bulk adopt-merge
must pass the §11.4 memory/cost gate on embedding-bearing tables — via bounded
batched fenced merges inside one staged transaction, a pool-bounded execution
mode, or an upstream key-filtered append — before the keyed fast-forward path
is removed. Correctness wins the ordering, but the memory bound is not
optional.
### 5.2 Routing choice
There are two acceptable implementations:
1. use the v2 merge path (`use_index(false)`) and pass its scale gate; or
2. consume a pinned Lance revision whose indexed path emits the identical
filter and passes the same surface guards.
If the v2 hash-join cost scales unacceptably at the Phase-B workload, fencing
waits for option 2. Correctness is not traded for the old indexed-path speed.
### 5.3 Symmetric mixed-transaction behavior
Before activation, the pinned Lance revision MUST conservatively reject both
orders of:
- filtered Update vs unfiltered Update;
- filtered Update vs bare Append.
The preferred fix is upstream conflict-resolver symmetry. A workspace-only
fork is not an accepted permanent design. The fleet barrier remains necessary
even after that fix because two old, unfiltered writers still have no filters
to compare.
## 6. Retry and validation semantics
A Lance retryable key conflict restarts the entire logical attempt:
1. gather a new graph snapshot and schema identity;
2. rerun all delta and committed-state validation;
3. restage from that base;
4. commit and publish through the normal recovery-covered pipeline.
It is incorrect to retry only `commit_staged`: an insert may have become an
update, defaults or checks may now differ, and cross-table validation may have
changed.
Upsert surfaces may perform the bounded semantic retry. Strict insert surfaces,
including `load --mode append`, do not change meaning under contention: both an
already-present match from `WhenMatched::Fail` and a concurrent same-key commit
normalize to typed `KeyConflict` / HTTP 409 for the whole strict operation. They
do not switch to `UpdateAll`; callers decide whether to resubmit. Other strict
read-modify-write surfaces retain their typed write conflict. Retry exhaustion
on a non-strict upsert remains a retryable 409.
Fencing covers the PK insertion race only. `@unique` values on different IDs,
edge RI, and cardinality depend on a read set. Their correctness requires the
read-set-in-CAS design or equivalent revalidation before HEAD movement; this RFC
does not claim that fences close those races.
## 7. Version and fleet barrier
### 7.1 No partial activation in the old format
OmniGraph MUST NOT annotate a new data table and advertise fencing while the
graph remains generally writable by older binaries. An older process can select
the legacy merge path or keyed append and bypass the guarantee.
This RFC owns its activation boundary:
- operators quiesce every server, CLI writer, and embedded writer for the
graph;
- one migration claimant holds an atomic create-if-absent claim with a random
owner/fencing token; a native Lance branch sentinel is not accepted as CAS;
- only the dedicated migration binary may open the old graph for writes;
- the fencing-compatible stamp is written after every branch/table verification;
- normal serving begins only after the stamp; older binaries then refuse.
The migration claim uses the storage adapter's `PutMode::Create` contract,
records operation/owner token, and has no time-only takeover. Recovery under the
fleet outage must classify the migration ledger/sidecars before replacing a
stale token.
The stamp is graph-wide and read from the reserved main manifest before any
named-branch open; selecting a named branch cannot bypass the compatibility
check.
An in-process mutex is not a fleet barrier. A marker unknown to v4 binaries is
also not a fleet barrier. The operator procedure and cluster control plane must
keep old writers stopped until finalization.
The exact format number is assigned when this RFC is accepted. If RFC-024 is
also accepted and ready, the two migrations may deliberately share its v5
release after a combined failure-matrix review. If durable heads fail their
cost gate, fencing still proceeds with its own next compatible stamp; if
fencing is blocked upstream, durable heads need not wait.
If durable heads are already active when fencing migrates, every PK metadata
version repoint also emits the identity-bearing journal event and matching
`table_head` transition in the same manifest CAS. If fencing lands first, its
format/stamp becomes an explicit predecessor that a later heads migration must
preserve. Acceptance covers heads-first, fencing-first, and co-release orders.
### 7.2 New graphs
A graph created directly at the fencing-compatible format creates every keyed
table with the PK metadata already present and enables only the write routing
in §5. There is no post-create annotation window.
## 8. All-branch PK migration
Migration operates on graph states, not merely table roots. The unit is every
reachable tuple:
```
(graph_branch, table_key, table_path, pinned_table_branch, pinned_table_version)
```
This matters for lazy branches: a graph branch may still point at an old main
table version whose schema predates the PK, even after main HEAD is annotated.
Under the fleet barrier, the migration:
1. enumerates and incarnation-pins every live graph branch;
2. folds each branch manifest and enumerates its live keyed tables;
3. validates that every pinned table image has non-null, unique `id` values;
4. acquires branch/table gates in RFC-022 order and freshly revalidates the
pinned tuple, schema identity, and migration claim;
5. writes a per-unit RFC-022 sidecar declaring expected branch/table state, the
optional native fork effect, PK metadata effect, and intended manifest delta
before either effect can persist;
6. for an owned table branch, commits a set-if-absent PK metadata update;
7. for a lazy-inherited state, forks an owned table branch from the *exact
pinned version*, applies the PK metadata there, and leaves row contents
unchanged;
8. records exact achieved fork identity and table version in the sidecar;
9. publishes that graph branch's manifest to the annotated physical version
with an audited migration marker but no graph-content commit or graph-head
movement, including a table-head transition when heads are active;
10. records a branch/table completion digest;
11. re-enumerates branches and verifies every live branch before writing the
fencing-compatible stamp.
PK installation advances a Lance table version before the graph manifest can
publish it, and a lazy fork creates native ref state first. The sidecar covers
both gaps and lets recovery reclaim or adopt the exact fork rather than infer
from a branch name. Because Lance forbids clearing/changing a set PK,
migration is roll-forward-only:
- an already-correct PK is an idempotent success and is not rewritten;
- an absent PK resumes installation;
- a different PK is a loud, operator-visible refusal;
- recovery never attempts to "undo" PK metadata.
Branch create/delete, schema apply, and normal data writes remain blocked for
the whole enumeration/install/verify interval. The migration ledger makes a
crash resumable without treating partial annotation as a served graph.
## 9. Preservation after activation
Once set, the following are storage invariants:
- a table overwrite carries the same PK field IDs and positions;
- schema apply cannot remove, replace, reorder semantically, or make nullable a
PK field;
- rename preserves the PK field ID and metadata;
- branch fork/clone preserves it;
- import/rebuild creates it before accepting data;
- recovery restore may select an older data image only if that image is already
fencing/PK-compatible;
- a table recreation uses a new table incarnation but installs the same
catalog-derived PK contract at creation;
- `__manifest`'s existing legacy PK key form is preserved exactly as stored;
the migration never rewrites or "normalizes" it. Lance forbids changing a
set PK, and the native-namespace decoupling documented in the Lance
alignment audit depends on that legacy form remaining in place.
Every open-for-write path verifies the physical schema matches the catalog PK
contract. The check is against the pinned physical schema and is not a
maintained parallel registry.
## 10. Recovery and multi-process scope
All data writes retain the existing Phase A-D sidecar protocol. The key filter
does not close the table-HEAD-before-graph-manifest window.
The fenced data-table transaction is cross-process safe in its failure-free
commit path. OmniGraph's current recovery sweep, however, serializes with live
writers only in-process; a foreign recovery process can still inspect a live
sidecar, and destructive `Restore` cannot be made convergence-idempotent.
Therefore this RFC MUST use one of two honest dispositions:
1. retain the documented single-writer-process support boundary and describe
fences as closing the silent key-race primitive only; or
2. land a cross-process sidecar claim/lease before advertising general
multi-process writes.
Fences alone are not evidence for disposition 2.
## 11. Tests and acceptance gates
### 11.1 Lance surface guards
- exact PK ON set + v2 path produces a non-empty filter for inserts;
- `WhenMatched::Fail` preserves that filter and reports an existing match
without writing;
- mismatched ON set produces no filter;
- legacy/indexed path behavior is pinned until replaced;
- filtered/filtered overlapping keys retry;
- filtered/filtered disjoint keys may rebase;
- filtered/unfiltered retries in **both** commit orders;
- filtered Update/bare Append retries in **both** commit orders;
- PK metadata cannot be changed or removed once installed.
### 11.2 Engine concurrency tests
- the same-key DST cell becomes a hard assertion with N concurrent writers;
- different keys remain concurrently writable;
- every keyed load/mutation/merge/fold path is observed to use the filtered
primitive;
- strict append of an existing `id` still fails and never updates the row;
- strict pre-existing-match and concurrent-insert cases normalize to the same
external `KeyConflict` while preserving `WhenMatched::Fail`;
- a source-walk guard rejects keyed `stage_append`, including the former
fast-forward path;
- a retry reruns validation rather than committing the stale staged batch.
### 11.3 Migration and recovery tests
- main plus owned and lazy-inherited graph branches all emerge PK-annotated;
- duplicate historical IDs abort before the fencing-compatible stamp;
- crash after each table annotation and before each manifest repoint resumes
without data change;
- crash before/after lazy fork and PK metadata commit recovers the exact
sidecar-recorded ref/version;
- branch enumeration is incarnation-safe;
- old binary/new graph and new binary/partially migrated graph refuse loudly;
- heads-first, fencing-first, and co-release upgrades preserve every active
format capability and produce identical logical rows;
- overwrite, schema apply, branch fork, restore, and import preserve PK metadata.
### 11.4 Cost gate
Measure a small upsert into 10K, 100K, and 1M-row indexed tables using the
shared cost harness. If `use_index(false)` makes work scale with table size
beyond the accepted budget, the indexed-path upstream work is a ship blocker.
Additionally measure the bulk adopt-merge shape that motivated the keyed
fast-forward path (#277): a many-row, all-new-rows fenced merge into an
embedding-bearing table, asserting peak memory bounded by batch size rather
than table or delta width. If the fenced path cannot meet that bound, the
keyed fast-forward removal waits for the mitigation named in §5.1.
> 💬 **Instrument required (tightening 5 in the
> [review ledger](../dev/rfc-022-027-architecture-review.md)):**
> `helpers::cost` measures I/O, not peak RSS, so this memory bound is
> unenforceable as written. Use the subprocess `scenarios.rs` harness or an
> equivalent `wait4`/`ru_maxrss` instrument, and name dataset sizes, baseline,
> cap, and pass threshold.
## 12. Decisions and open gates
### Decided
- `id`, not `src`+`dst`, is the edge PK.
- No keyed append, including optimization-only append.
- No mixed-fleet or new-table-only v4 rollout.
- PK migration is all-branch, offline, idempotent, and roll-forward-only.
- A retryable upsert conflict retries the logical operation; strict insert maps
both existing and concurrent matches to `KeyConflict` without changing mode.
- Read-set validation remains a separate required concurrency design.
### Open ship gates
1. Upstream symmetric filtered/unfiltered and filtered/Append conflict behavior.
2. v2-path scale result versus indexed-path filter availability.
3. Operator repair procedure for pre-existing duplicate IDs.
4. Rename-stable field/type identity.
5. Cross-process recovery ownership before any broadened topology claim.
6. Final format-number/release sequencing and the all-branch fleet stamp.
7. The fenced bulk adopt-merge memory/cost gate on embedding-bearing tables
(the #277 regression class) — see §5.1 and §11.4.

558
docs/rfcs/rfc-024-durable-table-heads.md Normal file
View file

@ -0,0 +1,558 @@
---
type: spec
title: "RFC-024 — Durable table heads and the v5 manifest"
description: Materialize one live-or-tombstoned current-state row per table inside each manifest branch, prove bounded physical lookup, and migrate every branch atomically before stamping v5.
status: draft
tags: [eng, rfc, manifest, write-path, versioning, migration, lance]
timestamp: 2026-07-10
owner:
---
# RFC-024: Durable table heads and the v5 manifest
**Status:** Draft / for team review
**Date:** 2026-07-10
**Surveyed:** omnigraph 0.8.1 (`main`); Lance 9.0.0-beta.15 at git rev
`f24e42c1`; full Lance table layout, transaction, branching, indexing,
compaction, cleanup, and object-store specifications
**Relationship to RFC-022:** this RFC is the durable-heads decision split from
the earlier monolithic RFC-022 draft. [RFC-022](rfc-022-unified-write-path.md)
defines the shared publisher/recovery protocol; this RFC owns the v5 format and
migration boundary. It deliberately excludes checkpoint retention, which
[RFC-025](rfc-025-checkpoint-retention.md) reviews separately. Key fencing in
[RFC-023](rfc-023-key-conflict-fencing.md) is also independently reviewable;
the two may share a release but do not block one another's evidence gates.
**Audience:** engine, manifest, migration, branch, and release maintainers
**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md).
Findings marked **BLOCKER** must be dispositioned before acceptance.
---
Throughout this draft, **v5** means the next internal schema after today's v4.
If another independently accepted format change lands first, the release number
is reassigned; none of the head-row semantics depend on the numeral.
## 0. Decision summary
The current manifest is both an immutable graph journal and the place writers
ask "what is current?" Current-state resolution folds history, so its physical
cost grows with commit count even though the answer contains only one value per
table.
v5 adds one mutable, durable `table_head` row per stable table identity inside
each native `__manifest` branch. The publisher updates the head in the same
Lance merge-insert transaction as immutable table-version rows, tombstone
events, graph lineage, and `graph_head`. The journal remains the history source;
heads become the current-state source.
The format does **not** ship merely because the logical result has O(tables)
rows. A filtered scan over a history-sized Lance table is still O(history)
physical work. v5 is gated on a Lance-native indexed lookup whose measured I/O
is flat at history depth and whose uncovered tail is bounded and observable.
Normative decisions:
1. Heads live in `__manifest`; there is no second heads dataset and no warm
mutable-tip authority.
2. Head state is explicitly `live` or `tombstoned` and carries an incarnation.
3. Every publish and recovery outcome updates journal and head atomically.
4. Current-state reads use heads; history reads use the journal.
5. Missing or duplicate heads in v5 are corruption, not a reason to silently
return to the history fold.
6. Migration covers every live manifest branch, then stamps main v5 last.
7. If RFC-023 is co-released, the combined migration also verifies PK fencing;
durable heads do not depend on that decision.
8. Checkpoint/retention markers are deferred to a separate RFC.
## 1. Problem
`__manifest` currently stores immutable table-version and tombstone rows. To
resolve a branch tip, readers scan those rows, select the greatest version per
table, and apply tombstones. This makes a write's coordination cost depend on
the total graph history. Compaction reduces fragment count but cannot remove
the semantic journal rows, so even a compacted manifest retains a row-volume
slope.
Caching that fold as mutable in-process state is the wrong authority shape for
writes: invalidation becomes a correctness condition across processes and
branch incarnations. Storing the folded answer durably in the same transaction
as the journal removes both liabilities:
- no parallel authority can drift; and
- current-state work is proportional to catalog width, provided physical
lookup is also bounded.
## 2. Scope and non-goals
In scope:
- v5 table-head schema and state transitions;
- atomic publisher, recovery, and current-read semantics;
- bounded physical access proof;
- all-branch predecessor→heads migration and compatibility refusal;
- optional co-release integration with RFC-023's all-branch PK activation.
Out of scope:
- a `GraphState` singleton or warm publish input;
- a separate `__heads` Lance dataset;
- deleting immutable journal rows;
- commit-graph ancestry acceleration;
- checkpoint retention and arbitrary-version GC;
- using a physical index as a correctness precondition.
Checkpoint retention is excluded because a row in `__manifest` does not by
itself pin a version in another Lance dataset. Lance cleanup recognizes its own
versions, tags, and branch references, not foreign references. A later RFC must
define substrate-native per-table pins and their crash-safe lifecycle.
## 3. v5 table-head schema
Each native manifest branch contains exactly one current head row for each
stable table identity known to that branch.
### 3.1 Identity and object key
```
object_id = "table_head:<stable_table_id>"
object_type = "table_head"
```
`stable_table_id` survives a type rename. It is not the display name and is not
derived anew from `kind:name`. Closing the current rename-stable identity gap is
a v5 ship gate, and **this RFC owns it**: the stable-ID encoding, the compiler
contract that mints and preserves it, and the incarnation baseline are defined
here. RFC-023 §4.1 (rename-safe PK claims) and RFC-025 §2.1 (checkpoint table
rows) consume this identity and must not ship identity-dependent claims before
it lands. No other RFC in the family defines a competing identity scheme.
> 💬 **Superseded by review (BLOCKER-07 in the
> [review ledger](../dev/rfc-022-027-architecture-review.md)):** ownership
> *inside* this RFC contradicts siblings that consume the identity while
> calling this RFC optional (RFC-025) or without declaring the dependency
> (RFC-026). Disposition: extract identity/incarnation into its own
> prerequisite RFC, or accept this RFC in full before the identity-dependent
> siblings. Replace this ownership note accordingly.
The head row is mutable under `WhenMatched::UpdateAll`, just like
`graph_head:<branch>`. There is one object ID per stable identity; recreation
does not leave multiple candidate heads.
### 3.2 Payload
v5 reuses the manifest's typed columns where they already fit and stores the
remaining versioned payload in a typed JSON structure:
```text
TableHeadMetadata {
state: "live" | "tombstoned",
stable_table_id: String,
incarnation_id: ULID,
schema_hash: String,
head_graph_commit_id: Option<ULID>,
}
```
The row columns have these meanings:
| Column | `live` | `tombstoned` |
|---|---|---|
| `table_key` | current public table key/name | last public key/name |
| `location` | physical table location | last physical location, diagnostic only |
| `table_version` | current visible Lance version | last live Lance version |
| `table_branch` | physical owner branch, nullable for main | last owner branch |
| `row_count` | current row count | null |
| `metadata` | `TableHeadMetadata` | `TableHeadMetadata`; `head_graph_commit_id` names the tombstoning graph commit |
The `state` field is authoritative. A tombstoned row MUST NOT become live merely
because an older journal version has a greater data-table version than some
other row.
### 3.3 Incarnations
`incarnation_id` distinguishes drop/recreate ABA:
- rename preserves stable ID and incarnation;
- ordinary writes preserve both;
- physical owner handoff preserves both;
- dropping the type transitions the one head to `tombstoned`;
- recreating a logically new table mints a new incarnation and updates the same
stable-identity head to `live` only through an explicit schema operation.
If recreation is assigned a new stable table identity by schema semantics, it
gets a new head object and the old identity remains tombstoned. The schema
planner, not a name comparison, chooses this outcome.
### 3.4 Journal identity
The mutable head is not the only place that records identity. Every v5
table-version, registration, rename, and tombstone journal event carries
`stable_table_id` and `incarnation_id`; otherwise drop/recreate followed by a
new physical dataset whose Lance versions restart cannot be replayed
unambiguously.
Migration writes one immutable v5 incarnation-baseline row per known identity,
bound to the source-tip digest. New registrations/recreations write an immutable
incarnation transition. Head repair starts from that baseline and replays only
identity-bearing v5 events. It does not infer identity from mutable names or
compare unrelated Lance version numbers. If pre-v5 history cannot be mapped to
one baseline identity deterministically, migration refuses before the stamp.
## 4. State-transition rules
| Event | Required head transition |
|---|---|
| Register table | absent → live, new incarnation |
| Data write / optimize / index publish | live version N → live version M |
| Owner-branch handoff | live owner A → live owner B, even if version is equal |
| Rename | live key/name A → live key/name B; identity/incarnation unchanged |
| Drop table | live → tombstoned in the same graph publish as the tombstone journal row |
| Recreate | tombstoned → live with the schema-planned identity/incarnation outcome |
| Recovery roll-forward | apply the failed writer's intended live/tombstone transition |
| Recovery rollback | publish a head matching the restored physical version and logical pre-write state |
No path may append a table-version or tombstone journal row without including
its corresponding head mutation in the same publisher source batch.
## 5. Atomic publisher contract
The v5 publisher constructs one merge-insert source containing:
- immutable, identity-bearing table-version rows;
- immutable, identity-bearing table-tombstone/transition rows;
- mutable table-head rows;
- when the RFC-022 plan carries `LineageIntent`, the immutable `graph_commit`
and mutable `graph_head:<branch>` rows;
- for metadata-only plans, their specific CAS authority/operation rows without
manufacturing graph lineage.
One Lance manifest commit makes the entire set visible. The publisher still
resolves the graph parent and re-reads commit authority inside every CAS retry.
Expected table versions are compared against table heads, not reconstructed by
folding the journal.
Two graph-content writers touching disjoint data tables still contend on
`graph_head:<branch>`, form one linear graph history, and re-parent on retry.
Writers touching the same table also contend on its one `table_head` object.
Metadata-only CASes contend on the stable authority rows named by their complete
`ReadSet`; they do not update `graph_head` merely to create contention.
The immutable journal remains necessary for snapshots, diffs, audit, migration
verification, and head repair. Head rows do not replace or truncate it.
## 6. Read contract
### 6.1 Current state
A v5 current-state read:
1. derives the expected live stable table IDs from the pinned catalog and fixed
system-table registry;
2. issues a structured lookup for the exact head object IDs;
3. requires exactly one valid row per expected identity;
4. includes only rows whose authoritative state is `live`;
5. validates schema identity from the head payload;
6. returns one immutable `Snapshot` used for the operation's lifetime.
Missing, duplicate, unknown-state, or schema-mismatched **live** heads fail
loudly. The hot path does not enumerate every identity ever dropped merely to
prove all tombstone heads exist; the branch completion digest plus explicit
`heads verify`/repair owns bounded tombstone-set validation. A missing or
duplicate tombstone is still corruption, but normal reads do not regain an
O(history) scan to discover it.
### 6.2 History
`snapshot_at_version`, commit resolution, change feeds, and audit continue to
read immutable journal/lineage state at the requested manifest version. A v5
manifest version contains the heads as they stood at that version, but the
journal remains the normative explanation of how the state arose.
### 6.3 Diagnostic repair
An explicit offline repair loads the branch's v5 incarnation baseline, replays
identity-bearing journal transitions from that baseline, compares the result to
its heads/marker digest, and publishes corrected heads with an audited system
actor. Repair is not part of the read hot path and never silently runs from a
query.
## 7. Bounded physical lookup is a ship gate
Logical O(tables) output does not prove physical O(tables) work. Without an
index, `object_id IN (...)` still scans the journal-bearing manifest fragments;
compaction reduces files but not semantic row count.
### 7.1 Required property
At fixed catalog width, a reconciled v5 current-state lookup MUST have zero
positive slope in:
- manifest object-store reads;
- bytes read;
- fragments/pages scanned; and
- rows decoded
as commit history grows.
The bound must hold on a real object store as well as local FS and must be shown
for compacted and uncompacted histories. The test uses the shared IO-tracking
harness and installs the tracker before the manifest handle opens.
### 7.2 Candidate access shape
The primary candidate is a structured exact lookup on `object_id` backed by a
Lance scalar index. It is acceptable only if measurement proves:
- indexed head lookup avoids journal-fragment scans;
- newly committed head rows leave at most a bounded uncovered tail;
- reconciliation restores coverage without synchronous expensive work in the
logical write path;
- index absence or partial coverage remains logically correct and is surfaced
as an observable degraded-cost mode.
The index is derived state. Queries MUST remain correct if it is missing, and a
missing index cannot block a logical write. The performance promise applies to
the reconciled serving state and includes an explicit bound on uncovered work;
it is not inferred merely from the existence of an index declaration.
### 7.3 Rejected access shape
A separate heads dataset is rejected. Lance commits are per dataset, so it
would reintroduce a journal→heads crash gap and require another sidecar protocol
for the very pointer whose purpose is to remove drift.
If no in-manifest Lance-native access shape passes the gate, v5 does not ship
with head rows. The fallback is to retain v4 plus the local session/view-passing
improvements, not to waive the cost claim.
## 8. Recovery protocol
Data-table writers still use the existing four phases:
1. write sidecar before a Lance HEAD advance;
2. commit staged/inline table work;
3. publish `__manifest`;
4. delete sidecar.
The sidecar's logical intent in v5 includes the expected and desired table-head
payload. Recovery behavior is therefore complete:
- roll-forward publishes the data version, journal row, table head, lineage,
and graph head together;
- rollback restores the physical version, then publishes journal/audit state
and a table head matching the restored logical state;
- a stale sidecar whose goal is already represented by the exact head
incarnation/version converges idempotently;
- a partially matching head is not treated as success.
Recovery remains a synchronous barrier before any later writer advances a
touched table. Index reconciliation may be asynchronous; unresolved commit
protocol state may not.
## 9. v5 boundary and compatibility
v5 comprises:
1. table-head rows and their publish/read semantics;
2. branch-local v5 completion markers; and
3. the graph-level internal-schema stamp written after verification.
It does **not** comprise checkpoint/retention markers.
If RFC-023 is independently accepted and ready for the same release, v5 may
also carry its PK annotation and fencing-compatible marker after a combined
migration review. That is release coordination, not a prerequisite of heads.
Format sequencing is explicit:
| Order | Result |
|---|---|
| Heads first | proposed v5 contains heads; later fencing migration preserves and atomically updates heads for every PK-version repoint |
| Fencing first | fencing takes the next format number; this heads migration takes the following number, accepts that exact predecessor, and preserves its PK/stamp invariant |
| Co-release | one format maps to both independently accepted capabilities and runs the combined failure matrix |
Migration code dispatches from an exact supported predecessor feature set; it
never assumes that “pre-heads” means pristine v4 or drops a capability it does
not own.
After upgrade, serving is strict-single-version:
- v5 binaries refuse unstamped or partially migrated graphs in normal mode;
- older binaries refuse v5 with the existing upgrade message;
- every open reads the graph-wide main stamp before selecting a named branch;
- only the dedicated offline migration command may open the exact supported
predecessor for conversion;
- there is no mixed v4/v5 serving period.
## 10. All-branch predecessor→heads migration
### 10.1 Preconditions
The operator stops every graph writer and acquires an atomic
create-if-absent migration claim (with exact owner/fencing token; not a Lance
branch sentinel). The barrier covers server, CLI, embedded, maintenance,
branch, and schema writes. Because predecessor binaries do not understand the
new in-graph migration marker, the offline fleet barrier remains mandatory
until the final stamp.
The claim uses `PutMode::Create`, records the migration operation and owner
token, and permits no time-only takeover. Recovery classifies the durable
ledger/sidecars under the fleet outage before replacing a stale token.
After acquiring the claim, migration completes RFC-022's recovery barrier
before pinning any branch source tip.
If fencing is co-released, the migration first executes RFC-023's all-branch
table-PK plan. Otherwise head migration neither adds nor changes PK metadata.
### 10.2 Per-branch conversion
For each live native `__manifest` branch:
1. capture branch name, ref incarnation, manifest version, and e_tag/timestamp;
2. run the predecessor journal+tombstone fold once at that pinned tip;
3. construct exactly one live or tombstoned head plus one immutable incarnation
baseline per stable table identity;
4. validate table schema hashes and, only in a combined release, RFC-023 PK state;
5. publish all heads/baselines, an audited migration record
(`actor = omnigraph:migration/v5`), and a branch-local completion marker in
one manifest CAS that revalidates the captured ref incarnation/source tip;
this physical representation migration does not create a graph-content
commit or move `graph_head`;
6. store in the marker the source tip, head count, identity-baseline digest,
head-set digest, and heads-format version; the marker is content-scoped and
deliberately does not embed the source branch incarnation; source tip is
provenance, while digest/format determine inherited-marker validity;
7. verify by reopening the produced branch version through the v5 head reader.
The completion marker has a deterministic object ID within each manifest
branch. A retry that finds a matching source/digest is complete; a mismatching
marker is a loud migration conflict.
### 10.3 Finalization
After all branches report completion, migration re-enumerates native manifest
refs and verifies the same incarnation set. Branch create/delete is blocked, so
any difference indicates out-of-band modification and aborts finalization.
Only then does it stamp main as internal schema v5. The stamp is the fleet
commit point: before it, no serving process may start; after it, only v5 code may
open the graph.
New branches created after v5 fork a source manifest that already contains
complete heads and the content-scoped v5 marker. The inherited marker remains
valid for the identical snapshot; branch open separately binds that content to
the new native ref incarnation and validates the graph-wide stamp. No
post-create marker rewrite is required.
## 11. Migration recovery
The migration keeps a durable ledger outside the ordinary read path with one
record per branch and, in a combined release, per RFC-023 table conversion. It
records expected source incarnations, achieved physical versions, produced
manifest versions, and digests.
Recovery is idempotent and roll-forward-only:
- completed, digest-matching branch conversions are skipped;
- a table HEAD advance not yet represented in its graph branch is recovered by
the normal sidecar and then branch conversion resumes;
- an uncommitted head-row source leaves no visible state and is rebuilt;
- a committed branch marker is authoritative for that branch conversion;
- the main v5 stamp is never written while any ledger unit is incomplete;
- co-released PK metadata is never cleared to simulate rollback.
If migration crashes, the graph remains offline. Restarting an old serving
binary is not a recovery procedure; the operator resumes the v5 migration.
## 12. Tests and acceptance gates
### 12.1 Head semantics
- current state from heads is byte-equivalent to the predecessor fold across realistic
histories;
- live→tombstoned never resurrects an older live version;
- drop/recreate distinguishes incarnations;
- identity-bearing journal replay remains unambiguous when a recreated physical
dataset restarts Lance version numbering;
- rename preserves identity/incarnation and changes the public key only;
- owner-branch handoff at an equal table version updates the head;
- missing, duplicate, malformed, and schema-mismatched heads fail loudly.
- `heads verify` detects a missing/duplicate tombstone against the baseline and
completion digest without adding that enumeration to every current read.
### 12.2 Publisher and recovery
- concurrent disjoint writers produce one linear graph chain and correct heads;
- same-table writers contend on one head row;
- failpoints after every table commit but before manifest publish recover to
matching physical version, journal, table head, and graph head;
- rollback and roll-forward assertions include head payloads, not only table
versions;
- a stale sidecar converges exactly once with one audit record.
### 12.3 All-branch migration
- main, owned named branches, and lazy-inherited branches all convert;
- tombstoned types are represented on every relevant branch;
- crashes before/after every per-branch CAS resume from the ledger;
- branch ref deletion/recreation is caught by incarnation verification;
- final stamp is impossible while any table or branch marker is absent;
- each branch's graph head and content lineage are unchanged by head migration;
- old/new binary refusal matrix covers every supported predecessor capability
set, partial heads migration, and complete heads format;
- a post-v5 branch inherits and validates complete heads.
- the inherited completion marker is content-scoped while ref-incarnation
validation is performed separately.
### 12.4 Cost gates
At fixed table count and increasing commit depth, assert flat curves for
manifest reads, bytes, fragments/pages, and decoded rows:
- local FS, compacted and uncompacted;
- S3/RustFS with real e_tags;
- warm repeated read;
- cold operation-local open with shared Session;
- one uncovered-head update before reconciliation;
- reconciled steady state.
The test must fail if the lookup silently falls back to scanning journal
history in the claimed steady state.
The decode term is part of the gate: parsing head rows — including the typed
JSON `TableHeadMetadata` payload — must be bounded by catalog width. A
per-read parse cost that grows with anything other than table count fails the
gate even when I/O is flat.
### 12.5 Format guards
- exact v5 head metadata schema and object IDs;
- one head row per stable identity;
- incarnation-baseline and identity-bearing journal event schemas;
- content-scoped branch completion marker schema/digest;
- RFC-023 PK metadata on node and edge tables when the release combines them;
- v5 publisher source always pairs a journal/tombstone event with a head row.
## 13. Decisions and open gates
### Decided
- Heads and journal share one `__manifest` transaction.
- Current reads use heads; historical reads keep the journal.
- Heads represent `live | tombstoned` plus incarnation explicitly.
- A separate heads dataset and a mutable in-process tip authority are rejected.
- Migration is offline, all-branch, resumable, and stamps v5 last.
- RFC-023 PK activation is verified by v5 only when deliberately co-released.
- Checkpoint retention is deferred.
### Open ship gates
1. Rename-stable table/type identity and final stable-ID encoding — owned by
this RFC; consumed by RFC-023 and RFC-025 (§3.1).
2. The in-manifest indexed lookup implementation and bounded uncovered-tail
proof.
3. Passing local and object-store depth-slope cost gates.
4. Atomic cross-process migration-claim implementation and operator runbook.
5. Final v5 metadata JSON/typed-column compatibility review.
6. Full all-branch migration/failpoint matrix.

444
docs/rfcs/rfc-025-checkpoint-retention.md Normal file
View file

@ -0,0 +1,444 @@
---
type: spec
title: "RFC-025 — Checkpoint-pinned retention"
description: Makes named graph checkpoints authoritative retention roots, materializes them as Lance-native manifest and data-table tags, and defines crash-safe reconciliation and cleanup ordering on the RFC-022 unified write path.
status: draft
tags: [eng, rfc, retention, checkpoint, cleanup, manifest, lance, omnigraph]
timestamp: 2026-07-10
owner:
---
# RFC-025 — Checkpoint-pinned retention
**Status:** Draft / for team review
**Date:** 2026-07-10
**Depends on:** [RFC-022](rfc-022-unified-write-path.md)'s publisher and
recovery-sidecar protocol. It composes with
[RFC-024](rfc-024-durable-table-heads.md), but is not part of v5 by implication.
**Surveyed:** omnigraph 0.8.1; Lance 9.0.0-beta.15 (`f24e42c1`)
**Audience:** engine, storage, CLI, and operations maintainers
**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md).
Findings marked **BLOCKER** must be dispositioned before acceptance.
---
## 0. Decision
OmniGraph adopts checkpoint-pinned retention. A checkpoint is a durable, named
graph snapshot: one reference set in the reserved main-manifest registry
containing every physical manifest/table lineage and version needed to
reconstruct that graph state.
The checkpoint rows are the **logical authority**. Lance tags are the
**physical pins** that make that authority effective. This distinction is
load-bearing: Lance `cleanup_old_versions` protects Lance tags and branches; it
does not inspect OmniGraph rows in `__manifest`. A checkpoint row without the
corresponding tags would therefore promise time travel to versions that Lance
is free to delete.
The safe ordering is asymmetric:
- create physical tags first, publish checkpoint authority last;
- tombstone checkpoint authority first, reclaim physical tags last.
Every crash window consequently over-retains. None under-retains.
## 1. Scope and non-goals
This RFC specifies:
1. checkpoint and GC-boundary manifest rows plus their own format activation;
2. deterministic Lance tags for the pinned manifest and every pinned table version;
3. create, delete, and reconciliation protocols;
4. how `cleanup` computes and protects its root set;
5. CLI, policy, audit, observability, migration, and acceptance contracts.
It does not change snapshot isolation, invent a second transaction log, or
replace Lance cleanup. It does not make every historical graph commit a
checkpoint. History outside branch heads, named checkpoints, and the explicit
retention window remains eligible for collection after operator confirmation.
## 2. Sources of truth
### 2.1 Logical authority: manifest rows
A checkpoint is immutable after creation. Retargeting a name means deleting the
old checkpoint and creating a new checkpoint ID; a name never silently moves.
The stable name-reservation row carries `state = live | tombstoned` and a
monotonic `generation`. Reuse is a CAS transition from the exact tombstoned
generation to `live` with `generation + 1` and a fresh checkpoint ID. Thus a
deleted name may be deliberately reused, but two concurrent re-creations cannot
both win.
All checkpoint authority rows live on the reserved **main** branch of
`__manifest`, regardless of the logical branch being checkpointed. This is a
global registry, not branch-local graph state: deleting the source branch must
not delete the authority that protects its snapshot. One main-manifest publish
writes:
- `checkpoint_name:<normalized-name>` — unique name reservation pointing at the
immutable checkpoint ID;
- `checkpoint:<checkpoint_id>` — header containing `name`, source logical
branch, `graph_commit_id`, source physical manifest branch and version,
manifest schema stamp, `created_at`, and `actor`;
- `checkpoint_table:<checkpoint_id>:<stable-table-id>:<incarnation-hash>` — one
row per pinned table containing stable table identity, table key,
`table_path`, `table_branch`, `table_version`, and physical incarnation.
The name reservation, header, and every table row land in one RFC-022 publisher
CAS on main. First creation is insert-only; reuse requires the exact tombstoned
reservation generation in the `ReadSet`. Two concurrent attempts for the same
normalized name therefore conflict even when they captured different source
branches. A missing or duplicate table row is an invalid checkpoint, not a
partial checkpoint.
Stable table identity/incarnation is a ship gate even when RFC-024 has not
landed; checkpoint rows cannot key retention to mutable names. A deployment may
reuse RFC-024's identity baseline, but durable heads themselves are not a
dependency.
Deletion changes the name reservation from `live` to `tombstoned` and writes
manifest tombstones for the checkpoint header and all table rows in one CAS.
The reservation and every tombstone carry the same fresh
`delete_operation_id`; that CAS is the durable delete/recovery marker. The
retention claim may repeat the operation ID while the writer is live, but it is
only a fence, never recovery authority. Checkpoint IDs are never reused.
Checkpoint create/delete are manifest metadata transactions. They do not create
a graph-content commit or move `graph_head`: taking a checkpoint must not change
the graph state it names. They still use RFC-022's read-set arbitration,
manifest CAS, actor, and audit contracts. Creation uses a recovery sidecar for
its pre-CAS tag effects; deletion is authority-first and embeds its idempotent
operation marker in the tombstone CAS instead of writing a generic sidecar.
The named snapshot is the source branch version and graph commit pinned in step
2 of creation, not whatever is current when the later main-registry CAS lands.
That immutable version may remain a valid checkpoint if the source branch
advances after capture. The response always returns the exact captured commit
and manifest version; it never implies that the checkpoint tracks a moving tip.
### 2.2 Physical protection: Lance tags
Every checkpoint owns deterministic Lance tags of two kinds:
```text
ogcp_<checkpoint-id-base32>_manifest_<branch-hash>
ogcp_<checkpoint-id-base32>_<table-and-lineage-hash>
```
The manifest tag targets the exact `(manifest_branch, manifest_version)` named
by the header. Each table tag targets the exact `(table_branch, table_version)`
recorded by its table row. The spelling stays within Lance tag-name constraints
and includes enough physical-lineage identity to avoid collisions across lazy
forks. Pinning `__manifest` itself is mandatory: data-table tags alone cannot
preserve the schema, registrations, and lineage needed to reconstruct the graph
snapshot.
Internal `ogcp_` tags are reserved. User-supplied tooling must not create,
move, or delete them. If an existing deterministic tag points anywhere else,
checkpoint creation and cleanup fail loudly with a typed corruption error.
Tags are derived state. A live checkpoint row with a missing tag is repaired
from the row. An internal tag with neither a live checkpoint nor an in-flight
checkpoint sidecar is orphaned and may be removed.
Tags protect versions from `cleanup_old_versions`; they do **not** protect a
named branch from Lance branch deletion, which removes `tree/<branch>`. The
initial contract therefore refuses physical deletion of any manifest or data
branch named by a live checkpoint row. Graph branch deletion and orphan-branch
reclamation read the main checkpoint registry under the retention claim and
return `CheckpointPinsBranch` with the blocking checkpoint IDs. They never call
`force_delete_branch` on a pinned lineage. Operators delete the checkpoints
first; hidden checkpoint branches or full snapshot copies require a separate
design.
### 2.3 Cross-process retention claim
Checkpoint create/delete, pin reconciliation, destructive cleanup, and graph
branch deletion serialize through one graph-wide, substrate-backed retention
claim. Acquisition uses the storage adapter's atomic create-if-absent primitive
(`PutMode::Create`) on `__leases/retention.json`; Lance branch creation is
explicitly rejected because its branch-metadata step is an existence check
followed by an unconditional put.
The claim payload records operation ID, action, actor, creation time, and a
random fencing/owner token. Every protected phase re-reads and verifies that
token. Normal release verifies the exact token before deleting the claim; no
other owner can replace it while create-if-absent observes the object. There is
no time-based lease stealing. Crash takeover requires an explicit fleet write
outage, classification of the sidecar/manifest operation marker, removal of the
stale claim, and acquisition with a new token. A resumed stale process must
fail its next token check.
A process-local mutex is only a contention optimization. The retention claim is
held across tag creation or physical cleanup, preventing this race: cleanup
computes roots, another process captures an untagged old version, then cleanup
deletes it before the tag lands. A crash leaves the claim and therefore
over-retains. Takeover first classifies the prior operation from its recovery
sidecar and/or exact manifest operation marker, then may release the claim
only after that operation is completed or safely aborted. It never steals on
elapsed wall time alone.
The claim is not presented as a distributed reader/writer lock for arbitrary
graph writes. Initial destructive cleanup is an **offline** operation: operators
quiesce every server, CLI, embedded writer, and MemWAL stream before acquiring
the claim. New binaries also refuse to start a write while the claim exists,
but that check is defense in depth rather than proof that an already-running
foreign writer drained. Online cleanup requires a separate fleet writer-epoch
or substrate lease design and is out of scope.
## 3. Checkpoint create protocol
Checkpoint creation is a writer on the RFC-022 pipeline:
1. Run and await RFC-022's synchronous recovery barrier.
2. Authorize, capture one immutable source-branch manifest snapshot and graph
commit, and prepare the complete header/table/tag set plus a `ReadSet` that
includes source branch incarnation, source manifest version, format/schema
identity, applicable GC boundaries, and the main-registry name reservation;
a source at or behind a pruned-through boundary is refused even if its files
happen to remain.
3. Acquire the retention claim, checkpoint/cleanup process-local serialization,
source branch-control gate, and adapter-declared metadata gates in RFC-022's
canonical order. Ordinary data queues are not held merely because immutable
versions are being tagged.
4. Freshly revalidate the complete `ReadSet`. A changed branch incarnation,
missing tag target, conflicting name, or format change releases the gates and
restarts before any tag exists.
5. Write a generic recovery sidecar containing the intended rows and tags.
6. Create the manifest tag and every table tag idempotently. Existing correct
tags are no-ops; conflicting tags fail.
7. Verify every tag, then release source branch control. A source-head advance
is harmless because the tags already pin the exact captured version;
source-branch deletion takes the same retention claim, classifies overlapping
create sidecars, and then refuses if the published checkpoint pins the ref.
8. Publish the name/header/table rows in one CAS on the main manifest registry.
9. Delete the recovery sidecar. Sidecar deletion remains best-effort after the
authority publish, as in the existing write protocol.
10. Release the retention claim after the outcome is durably classifiable.
A crash before step 8 leaves tags but no checkpoint. Recovery may complete the
publish when every precondition still holds or remove the orphan tags. A crash
after step 8 leaves a valid checkpoint even if sidecar cleanup did not finish.
## 4. Checkpoint delete protocol
Deletion deliberately reverses the create order:
1. run the recovery barrier, authorize, and prepare the complete checkpoint plus
name/format/registry `ReadSet`;
2. acquire the retention claim and local gates in canonical order, then freshly
revalidate the complete checkpoint;
3. publish the tombstoned name reservation plus header/table tombstones, all
carrying one fresh `delete_operation_id`, in one manifest CAS;
4. release the claim and acknowledge once the authority change is durable;
5. delete the deterministic Lance tags asynchronously and idempotently.
A failed tag deletion only retains extra data. The checkpoint is already gone
from the user-visible namespace. The reconciler and the next `cleanup` retry
the physical reclaim.
Delete is an RFC-022 authority-first metadata workflow: no independently durable
effect precedes the atomic tombstone CAS, and later tag deletion is derived,
over-retaining cleanup. It therefore needs no generic write sidecar. The
`delete_operation_id` persisted by the tombstone CAS lets crash recovery
distinguish pre-CAS from post-CAS state exactly; the retention claim does not.
## 5. Pin reconciler
Read-write open and every destructive `cleanup` run reconcile checkpoint pins
from the main-manifest checkpoint registry.
The reconciler:
1. reads the live checkpoint rows once;
2. classifies checkpoint sidecars before touching apparently orphaned tags;
3. creates or verifies every required tag;
4. deletes internal tags proven to have no live or in-flight authority;
5. emits a typed result for repaired pins, reclaimed pins, and failures.
`cleanup` is fail-closed: any missing, conflicting, unreadable, or ambiguous
pin aborts deletion for the affected graph. It never treats reconciliation
failure as permission to collect data.
The reconciler runs under the retention claim and is serialized with
checkpoint creation and table maintenance.
It may run concurrently across independent graphs, but it must not delete a
tag belonging to a foreign process's live create sidecar.
## 6. Cleanup protocol and pruned-through boundary
The cleanup root set is:
- every live graph branch head;
- every live checkpoint manifest and table row;
- every version inside the operator-selected time/count retention window;
- versions Lance itself protects through non-OmniGraph tags or branch refs.
Read-only preview may run without a claim, but it is explicitly provisional.
Confirmed execution uses this order:
1. establish the operator write outage, persistently seal/fold streams through
RFC-026 **before** taking the retention claim, and complete the RFC-022
recovery barrier;
2. acquire the retention claim, then revalidate the fleet outage, every stream's
`SEALED` cut, and absence of recovery sidecars;
3. run pin reconciliation and recompute the exact root set and per-dataset
cutoffs under the claim;
4. prepare and revalidate a complete RFC-022 `ReadSet` containing format/schema
identity, branch incarnations, current manifest table entries/heads, prior GC
boundaries, and the root digest;
5. publish mutable `gc_boundary:<dataset-lineage-hash>` rows containing cutoff,
root digest, cleanup operation ID, and timestamp in one reserved-main
manifest CAS;
6. invoke Lance `cleanup_old_versions`, relying on the verified tags to retain
sparse checkpoint versions;
7. record per-table removal statistics/failures and release the claim only when
the operation is durably resumable or complete.
Step 5 is an RFC-022 authority-first metadata transaction: it changes which
versions recovery may select, carries no graph-content lineage, and needs no
sidecar because no physical effect precedes its CAS. Step 6 is RFC-022 §8
physical maintenance under the already-published boundary. The two are not one
indistinguishable “cleanup commit.”
The boundary advances before delete. Any later recovery, restore, or publisher path
that could make an older physical version current must include the boundary in
its RFC-022 `ReadSet` and revalidate it before the physical effect. A position
at or behind the boundary is retried from current state or refused. Cleanup
never starts with an armed recovery that may still need an older version.
Per-table cleanup remains fault-isolated, but partial operational success is
reported explicitly. A successful table does not hide another table's error.
The claim metadata plus `gc_boundary` operation ID is the cleanup recovery
marker: takeover resumes or reports the remaining table units under the same
root digest before releasing the claim; it does not silently recompute a
broader destructive plan after a partial run.
### 6.1 Operational cadence — the cost of never running this
Offline-only destructive cleanup has a predictable failure mode: operators
defer it indefinitely, and the graph silently reacquires the unbounded-history
cost class this program measured (per-version chains that grow one entry per
commit; latest-version listings whose page size grows with history; on hosted
deployments, seconds of added latency per operation). The docs and `cleanup`
preview therefore state the deferral cost explicitly, and deployments are
expected to schedule a periodic maintenance window — for continuously written
graphs, the same cadence discipline as `optimize` — rather than treating
cleanup as exceptional. Read-only preview and pin reconciliation remain online
so drift is visible between windows.
Online destructive cleanup — concurrent with live writers under a fleet
writer-epoch or substrate lease — is the named successor design, deliberately
out of scope here (§2.3). This RFC's offline barrier is the correct first
delivery, not the end state; a deployment for which write outages are
unacceptable should treat the successor design as the gating requirement for
adopting aggressive retention policies.
## 7. Public and policy surface
Initial delivery is engine plus direct-storage CLI, matching `cleanup`:
```text
omnigraph checkpoint create <name> [--branch <branch>] <store>
omnigraph checkpoint list <store>
omnigraph checkpoint show <name-or-id> <store>
omnigraph checkpoint delete <name-or-id> <store>
```
JSON output includes checkpoint ID, name, actor, branch, graph commit, creation
time, table count, and pin-health status. `cleanup` preview reports which branch,
checkpoint, or window protects each retained version and states that execution
requires a graph-wide write outage.
Checkpoint creation and deletion use dedicated Cedar actions. Embedded callers
hit the same engine gate as the CLI. HTTP management endpoints are out of scope
until server-side maintenance has a general design; no transport-only bypass is
introduced here.
## 8. Migration and compatibility
This RFC owns its own internal-format activation. RFC-022 authorizes no format
bump, and RFC-024 explicitly excludes checkpoint rows from its heads format.
The retention format may receive the next schema version after heads, or the
two may share one release only if both RFCs are independently accepted and
RFC-024 is amended before implementation. A fresh activated graph starts with
no named checkpoints and a zero pruned-through boundary. No older commit is
silently promoted into a checkpoint.
The upgrade mechanism must be chosen before implementation:
- an in-place upgrade preserves branches and history and must define quiescence,
crash recovery, stamp ordering, and rollback;
- export/import creates a fresh graph at the new format but loses the old commit
DAG, branches, snapshots, and time-travel history. Those losses must be shown
in the plan and confirmation output; there is nothing left to backfill as a
checkpoint.
Mixed-version writers are unsupported. An old binary must not write after the
retention format stamp or tag protocol becomes authoritative.
## 9. Observability and bounds
Expose:
- live checkpoint count and pinned table-version count;
- missing, conflicting, repaired, and orphan-tag counts;
- oldest checkpoint age and retained bytes when Lance reports them;
- current GC boundary and root digest per dataset lineage;
- cleanup versions/bytes removed and per-table failures;
- checkpoint create/delete/reconcile latency and retry counts.
Checkpoint enumeration and cleanup planning must be bounded by live checkpoints
times catalog width, not total commit history. Operators can limit checkpoint
count by policy; the engine refuses names or payloads beyond configured bounds
before creating tags.
That is a physical-I/O claim, not just a logical row-count claim. Registry
lookup must use a structured Lance access path with a measured bound on
uncovered fragments, reusing RFC-024's in-manifest scalar-index work when it is
available or proving an equivalent access shape here. A filtered scan that
still reads history-sized manifest fragments does not pass this RFC's cost gate;
a separate checkpoint dataset is rejected because its authority rows could not
share the main-manifest CAS.
## 10. Acceptance gates
- A checkpoint on an old sparse version survives cleanup while adjacent
unpinned versions are removed.
- Branch delete and orphan reclamation refuse a non-main manifest/data lineage
while a live checkpoint references it; after checkpoint deletion and tag
reconciliation, branch deletion succeeds. `force_delete_branch` is never used
to bypass the pin.
- The source `__manifest` version survives cleanup; data-table tags without the
manifest tag fail the checkpoint-validity test.
- Concurrent creation of the same normalized name yields exactly one authority
record; the losing attempt leaves only safely reclaimable tags.
- After deletion, concurrent attempts to reuse the tombstoned name generation
yield exactly one fresh checkpoint ID; the old checkpoint ID never revives.
- Create failpoints cover sidecar, tag, and manifest boundaries; delete
failpoints cover authority CAS, claim release, and tag reclaim. Every outcome
converges to valid pins or safe over-retention.
- Missing live tags are repaired before cleanup; conflicting tags block cleanup.
- The retention claim blocks checkpoint create/delete/reconcile and branch delete
during cleanup on local FS and S3; destructive cleanup refuses active streams
or recovery sidecars.
- Two-process races prove the atomic retention claim, not a process-local gate,
closes checkpoint-vs-branch-delete windows; crash takeover never relies on
wall-clock expiry alone.
- Local and S3 claim tests prove `PutMode::Create` admits exactly one owner and
that mismatched owner tokens cannot release another operation's claim.
- A documented fleet write-outage test runs writers before and after cleanup;
online writer-vs-cleanup concurrency is explicitly not advertised.
- Genuine cross-version tests cover the selected upgrade path.
- Cost tests use `helpers::cost` at realistic commit depth and prove that a
steady checkpoint list/lookup and cleanup plan do not grow with commit count
once physical layout is held constant, including a bounded uncovered tail.
## 11. Phasing
| Phase | Content | Gate |
|---|---|---|
| A | row schemas, engine DTOs, format activation, deterministic tag encoding | schema and tag surface guards |
| B | create sidecar, delete-operation fields in the authority CAS, and reconciler | crash matrix; sparse-pin correctness |
| C | offline cleanup integration and GC boundary enforcement | quiescence/refusal tests; cost budgets |
| D | CLI, policy, audit, docs, and selected migration path | CLI outputs; genuine upgrade test |

415
docs/rfcs/rfc-026-memwal-streaming-ingest.md Normal file
View file

@ -0,0 +1,415 @@
---
type: spec
title: "RFC-026 — MemWAL streaming ingest"
description: Adopts Lance MemWAL as OmniGraph's strategic streaming-write architecture, with durable per-row acknowledgement, graph-atomic folds, epoch-fenced quiescence, and explicit fresh-read cuts on the RFC-022 unified write path.
status: draft
tags: [eng, rfc, streaming, ingest, wal, memwal, lance, omnigraph]
timestamp: 2026-07-10
owner:
---
# RFC-026 — MemWAL streaming ingest
**Status:** Draft / for team review
**Date:** 2026-07-10
**Depends on:** [RFC-022](rfc-022-unified-write-path.md)'s unified write and
generic recovery-sidecar protocol, plus
[RFC-023](rfc-023-key-conflict-fencing.md) for the initial keyed graph-stream
mode. Durable heads from
[RFC-024](rfc-024-durable-table-heads.md) are compatible but not required.
**Surveyed:** omnigraph 0.8.1; Lance 9.0.0-beta.15 (`f24e42c1`); complete MemWAL format specification
**Audience:** engine, server, CLI, policy, and operations maintainers
**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md).
Findings marked **BLOCKER** must be dispositioned before acceptance.
---
## 0. Decision and risk posture
OmniGraph adopts Lance MemWAL as its strategic streaming-write architecture.
MemWAL is a major Lance architectural bet: a sharded LSM write path with durable
WAL entries, flushed Lance generations, merge progress committed with base-table
data, maintained indexes, and epoch-fenced writers. OmniGraph consumes that
architecture rather than building a WAL, shard protocol, or LSM reader.
This RFC does **not** characterize the architecture as experimental. The risk is
narrower: Rust API names, some format details, and operational helpers are still
maturing across Lance releases. We manage that API/format-maturity risk with a
small adapter, compile/runtime surface guards, a quiescence requirement before
Lance upgrades, and a fresh full-spec alignment audit on every bump. It is not a
reason to fork or reimplement MemWAL.
The contract is:
- stream acknowledgement means the row's WAL entry is durable;
- acknowledgement does not mean graph visibility;
- default queries see only the manifest-committed graph;
- a fold is an ordinary RFC-022 graph writer and is the sole visibility point;
- fresh reads are explicit and never claim cross-table atomicity.
## 1. Scope and non-goals
This RFC specifies enrollment, the public stream API, acknowledgement semantics,
folding, fold-time integrity, dead-letter atomicity, branch/schema quiescence,
fresh-read cuts, resource bounds, observability, testing, and upgrade posture.
It does not replace `load` or `mutate`, provide cross-query transactions, store
manifest mutations in MemWAL, create a second metadata authority, or weaken
default snapshot isolation. Stream-mode
deletes remain out of the first delivery and require the Lance tombstone surface
plus a separate acceptance pass.
## 2. Stream mode and key semantics
Initial delivery exposes
`@stream(mode="upsert", on_reject="strict")` on a node or edge type;
`on_reject` accepts `strict` or `dead_letter`.
It requires the table's immutable unenforced primary key to equal OmniGraph's
merge key: `id` for nodes and edges. All occurrences of one key map to one shard
and MemWAL applies last-write-wins ordering.
Public append mode is deliberately out of scope. Nodes and edges always have
logical identity; allowing a retry to append the same `id` twice would violate
that contract. A future explicitly keyless, non-graph append-only table class
may consume MemWAL append semantics under its own schema/API decision.
Stream ordering intentionally differs from the interactive fence:
- concurrent interactive same-key writes serialize or fail/retry loudly;
- same-key stream entries resolve by MemWAL generation/position order;
- duplicate keys inside one bulk-load input retain the existing load error.
The schema and user docs state all three together.
## 3. Enrollment is a recoverable inline commit
Schema apply records `@stream` intent only. First stream use enrolls the physical
table by creating the singleton `__lance_mem_wal` system index and its sharding
configuration.
Enrollment advances Lance HEAD inline. It therefore uses the RFC-022 generic
recovery protocol, not an ad-hoc state machine:
1. run and await RFC-022's synchronous recovery barrier;
2. authorize, pin the manifest/schema/table state, and prepare a complete
`ReadSet` containing schema identity, table entry/head, PK metadata, stream
intent/configuration, and lifecycle-row absence;
3. acquire any global claims and then the `(table, branch)` write queue in
RFC-022 order, then freshly revalidate the complete `ReadSet`; a mismatch restarts
before any inline effect;
4. verify RFC-023's already-installed PK; enrollment never performs a first-use
PK migration; validate the sharding configuration;
5. write a generic sidecar with writer kind `stream_enrollment` and pre-commit
table pin;
6. create the MemWAL index, advancing Lance HEAD;
7. publish the new table version plus `stream_state = OPEN` in one manifest CAS,
including the table-head row when RFC-024 is active;
8. delete the sidecar best-effort after publication.
Recovery rolls the enrollment forward or back under the same classification and
audit machinery as other inline residuals. Repeating enrollment with identical
metadata is a no-op. A different PK, sharding spec, maintained-index set, or
writer-default configuration is a typed conflict.
No row is acknowledged until enrollment is manifest-committed.
Initial delivery supports one unsharded shard per `(table, main)`. Non-main
branches remain refused until the Lance branch-scoping question is proven by a
surface guard and end-to-end test. Later `bucket(id, N)` sharding must preserve
the one-key-to-one-shard rule.
## 4. New public API
The shipped `POST /graphs/{id}/ingest` path remains the deprecated, compatible
alias of `/load`. Streaming receives a new, non-conflicting surface:
```text
POST /graphs/{graph_id}/streams/{type_name}/ingest?branch=main
GET /graphs/{graph_id}/streams
GET /graphs/{graph_id}/streams/{type_name}
POST /graphs/{graph_id}/streams/{type_name}/fold
POST /graphs/{graph_id}/streams/{type_name}/quiesce
POST /graphs/{graph_id}/streams/{type_name}/resume
```
The ingest request and response use `Content-Type: application/x-ndjson` and
`Accept: application/x-ndjson`.
Each input line is one row payload. Each output line corresponds to the same
input ordinal:
```json
{"ordinal":17,"status":"durable","shard_id":"...","writer_epoch":8,"wal_position":42}
```
Synchronous validation failures return a per-row error before a WAL append.
Previously acknowledged rows in the same request remain durable; the response
is a stream, not an all-request transaction. Ordering, cancellation, and retry
rules are explicit:
- acknowledgements are emitted in input order for one HTTP stream;
- disconnecting does not cancel entries whose durability waiter resolved;
- a missing response is ambiguous; retrying the same `id` and payload may add
another WAL entry but produces the same last-write-wins graph state;
- server shutdown stops admission, drains durability waiters up to a bound, and
reports any unacknowledged tail as unknown to the client.
CLI commands mirror the new namespace rather than overloading deprecated
`omnigraph ingest`:
```text
omnigraph stream ingest <type> --data <ndjson> ...
omnigraph stream status [<type>] ...
omnigraph stream fold [<type>] ...
omnigraph stream quiesce [<type>] ...
omnigraph stream resume [<type>] ...
```
Every endpoint has a dedicated OpenAPI operation and handler tests. Ingest
passes the engine `stream_ingest` Cedar action and per-actor admission
accounting before acquiring a shard writer; fold/quiesce/resume use a separate
`stream_manage` action. Status is authorized like other graph operational
metadata. The same engine gates apply to embedded and remote CLI use.
## 5. Ack-path validation and writer lifecycle
Before append, OmniGraph applies checks that need no base-table read: Arrow
shape/type, required/default fields, enum/range/check constraints, reserved
columns, and stream mode. RI, cardinality, cross-version uniqueness, and
external embedding computation remain fold-time work.
One warm `ShardWriter` is held per active shard behind a bounded registry. The
registry has idle eviction and hard limits for resident writers, MemTable bytes,
unflushed WAL bytes, pending generations, and per-actor inflight bytes. Exceeding
a bound backpressures with a typed retryable response; it never drops a row.
Initial topology has one active ingest owner for each `(graph, table, main)`
shard. MemWAL's epoch fence makes restart/failover safe; it is not a load
balancer. A deployment with multiple server replicas must route a shard to its
current owner (or return a typed retry/redirect) instead of letting replicas
reclaim the epoch per request. General multi-owner routing waits for the
multi-shard phase and its ownership protocol.
## 6. Fold protocol
The fold consumes flushed generations in ascending order. Embeddings and
base-dependent validation run outside the table queue and register every probed
table plus stream configuration/generation in RFC-022's `ReadSet`. The commit
phase then:
1. stages accepted rows with Lance merge-insert and includes
`merged_generations` in that transaction;
2. stages any rejection/audit rows required by §7;
3. acquires every affected queue in canonical sorted order and revalidates the
complete `ReadSet`; any mismatch discards and replans the whole fold;
4. writes one generic RFC-022 recovery sidecar before the first
`commit_staged` call;
5. commits every staged Lance transaction;
6. publishes all data/internal table versions and lineage in one `__manifest`
CAS, including table heads when RFC-024 is active;
7. deletes the sidecar after successful publication.
The sidecar is mandatory even though merge-insert is staged. After
`commit_staged`, Lance HEAD and `merged_generations` have moved while the graph
manifest has not. A failure in that window is the ordinary multi-table recovery
gap, not invisible staged state.
MemWAL generation GC starts only after the exact fold is graph-visible, its
sidecar is resolved, index catchup permits reclamation, and no `FreshReadCut`
retention guard references the generation. Data-HEAD merge progress alone is
never permission to delete the only fresh-tier copy.
Concurrent folders reload `merged_generations`: a generation already committed
is skipped; otherwise the fold is replanned from current state. A forced fold
stops new flush creation for its cut, waits for in-flight durability waiters,
and never aborts an acknowledged row.
Fold lineage uses `omnigraph:ingest` as the mechanism actor and persists the
authenticated contributor actor for every folded WAL range in the same internal
audit participant. Commit and status output can therefore answer both who ran
the fold and who supplied the data after WAL GC.
## 7. Fold-time rejection is atomic
`strict` is the default. A permanent RI, cardinality, uniqueness, or embedding
failure stops the fold at the offending generation, marks the shard blocked,
and backpressures new ingestion once configured lag bounds are reached.
`dead_letter` is explicit. `_ingest_rejects` is then a versioned internal Lance
table and a participant in the same fold pipeline, not best-effort state outside
the commit protocol. Every reject has deterministic identity:
```text
(table_key, shard_id, generation, wal_position)
```
The fold stages reject rows, accepted rows, and merge progress before committing
any of them. The generic sidecar covers every participant; the single manifest
publish records both the base-table and reject-table versions. Replay is
idempotent by reject identity. There is no ordering in which progress can become
visible while the corresponding rejection is lost.
`stream status` reports blocked generations and typed reject details. Reject
retention is explicit and cannot be shorter than the WAL/fold audit retention
needed to explain a durable acknowledgement.
## 8. Epoch-fenced quiescence barrier
Branch operations, schema changes, stream teardown, and Lance upgrades require a
real barrier, not an empty check.
Each enrolled table has a durable
`stream_state:<stable-table-id>:<incarnation>` row in its manifest branch with
`OPEN | DRAINING | SEALED`, configuration hash, and epoch floor. The row is the
logical lifecycle authority and is updated by an RFC-022 CAS; MemWAL shard
epochs are the physical writer fence. Neither an in-memory registry nor an
empty-generation observation can substitute for both.
Lifecycle-only transitions are audited manifest metadata transactions; they do
not create graph-content commits or move `graph_head`.
The shared drain sequence is:
1. publish stream intent `OPEN -> DRAINING` for the target table/branch;
2. increment and persist each shard writer epoch and seal claims, fencing stale
writers across processes;
3. reject or backpressure new appends;
4. wait for in-flight durability waiters, flush active MemTables, and fold every
generation to empty;
5. verify shard manifests and base `merged_generations` agree on emptiness;
6. publish `DRAINING -> SEALED` with the verified generation/epoch cut;
7. for an operation-scoped drain, perform the guarded operation; persistent
public quiesce stops after step 6.
Each lifecycle CAS is an RFC-022 authority-first metadata write; each fold is a
separate normal RFC-022 graph write. `DRAINING` fully encodes the target epoch
floor, so an interrupted epoch mutation is idempotently resumed from that row.
The sequence is not one giant sidecar spanning multiple commits.
There are two dispositions after the drain reaches `SEALED`:
- **operation-scoped drain** — branch/schema maintenance automatically publishes
`SEALED -> OPEN` with a newer epoch only after the guarded operation succeeds
and the stream contract remains compatible;
- **persistent quiesce** — the public `quiesce` command leaves the stream
`SEALED`. It never auto-reopens. `stream resume` explicitly revalidates schema,
PK, configuration, MemWAL format, and epoch, then publishes a newer `OPEN`
state. Stream teardown deletes intent only from `SEALED`.
The barrier never holds the table write queue while waiting for a fold that
needs that queue. State transition and epoch fencing happen first; fold commit
then acquires the normal queue. Crash recovery resumes from the durable state
and epoch.
Schema apply must drain every affected enrolled type before changing fields,
constraints, PK, embeddings, or `@stream` and resumes only when compatible.
A Lance version upgrade requires persistent `stream quiesce --all`, but empty
generations alone are insufficient: the MemWAL system index, shard manifests,
epoch records, and generation directories may still use the old format. Before
the bump, the implementation must prove one of: (a) upstream guarantees and
cross-version tests cover every retained MemWAL artifact, (b) a public Lance
metadata migration converts them, or (c) OmniGraph tears down the enrolled
MemWAL metadata under recovery and re-enrolls after the bump. Without one of
those gates the upgrade refuses; `resume` never opens unverified old metadata.
## 9. Fresh-read cuts
Freshness is a first-class engine/IR enum:
```text
Committed
Fresh
```
At query planning, `Fresh` captures one `FreshReadCut` containing:
- the ordinary manifest snapshot;
- each selected shard-manifest version and writer epoch;
- included flushed-generation paths and maximum generation;
- the active same-process MemTable row-position watermark, when available;
- the base table's `merged_generations` and index-catchup state read from the
exact table version selected by the manifest snapshot, never from live HEAD.
Capture uses a retrying handshake:
1. read the selected shard manifests/epochs, acquire Lance generation retention
guards for the flushed files in the tentative cut, and under one
same-process writer snapshot capture/pin any active-MemTable watermark;
2. pin the graph manifest snapshot and read `merged_generations` from each exact
base-table version it selects;
3. re-read the shard manifest versions/epochs; any epoch/configuration change
restarts the whole capture;
4. if a generation from step 1 disappeared, accept that only when the pinned
base's `merged_generations` proves it is included; otherwise release guards,
discard the whole graph snapshot, and retry from step 1;
5. exclude generations that appeared after step 1 and hold the generation and
MemTable read guards captured in step 1 until query completion.
If Lance exposes no guard that prevents generation GC for the query lifetime,
cross-process `Fresh` does not ship. A missing generation is never interpreted
as “probably folded” against an older pinned base.
Execution never refreshes that cut mid-query. It excludes every flushed
generation `<= merged_generations[shard]`; otherwise old WAL data could outrank
or duplicate its newer base-table image.
Fresh reads have no cross-table atomicity. Same-process active MemTables provide
read-your-writes; other processes can promise only the latest flushed state
captured by their shard-manifest reads. The HTTP request and query docs state
those limits wherever the tier is exposed.
## 10. Observability and resource contracts
Per shard expose durable WAL position, replay position, active epoch, current
generation, flushed and merged generation, index catchup, pending rows/bytes,
oldest acknowledged age, last fold error, blocked reject, and quiescence state.
Metrics cover ack latency, durability-wait batching, fenced writers, replayed
entries, fold rows/bytes/generations, fold retries, lag, reject counts, and
sidecar recovery. Defaults for every byte/count/time bound are documented and
configuration changes are observable behavior.
`stream status` resolves the exact lifecycle rows and MemWAL metadata through a
structured, bounded access path; it may reuse RFC-024's scalar-index machinery
but cannot claim history-flat cost while scanning manifest history.
## 11. Acceptance gates
- Surface guards pin claim, append, durability waiter, flush, epoch fencing,
staged merge with `merged_generations`, index catchup, and seal/reopen APIs.
- A WAL append failure emits no durable acknowledgement. Every acknowledged row
survives crash, replay, fold, and recovery.
- Failpoints cover enrollment's inline-commit gap and every fold participant
around sidecar, `commit_staged`, reject persistence, and manifest publish.
- Enrollment restarts without an inline effect when schema, PK, table head, or
stream configuration changes between prepare and gated revalidation.
- Two folders converge exactly once; a fenced stale writer can never produce a
false durable acknowledgement after WAL GC.
- Two server replicas do not epoch-ping-pong one shard; owner failover fences
the old process before the new owner acknowledges.
- Quiescence tests race appends with branch and schema operations across two
coordinators and prove no post-drain tail appears; failpoints cover every
lifecycle-row, epoch-fence, fold, and `SEALED` boundary.
- Persistent quiesce never auto-reopens; explicit resume validates a newer
epoch. Upgrade tests cover every retained MemWAL artifact through declared
compatibility, migration, or teardown/re-enrollment.
- Fresh reads race capture with fold/GC, retry on an unexplained disappearing
generation, hold generation/MemTable guards through execution, exclude merged
generations, and document cross-table inconsistency explicitly.
- Server/OpenAPI tests preserve the old `/ingest` alias and cover the new route;
CLI parity covers embedded and remote stream commands.
- Ack-path object-store operations are O(1) and flat in graph history and WAL
depth. Fold cost is bounded by generations/rows folded, not graph history.
- S3 correctness runs against RustFS; API/format guards are rerun before every
Lance bump.
## 12. Phasing
| Phase | Content | Gate |
|---|---|---|
| A | MemWAL adapter, surface guards, enrollment sidecar | inline-commit crash matrix |
| B | new ingest route/CLI, durable ack, strict fold | ack durability; API compatibility; cost budget |
| C | atomic dead letter, audit provenance, status/bounds | reject crash matrix; backpressure tests |
| D | epoch-fenced drain, persistent quiesce/resume, schema/branch/upgrade integration | two-coordinator race and format-transition suite |
| E | fresh cuts and maintained-index reads; cross-process `Fresh` ships only if the substrate generation-retention guard exists (§9), otherwise same-process only | cut consistency; merged-generation exclusion |
| F | multi-shard upsert and stream deletes | one-key-one-shard proof; Lance re-audit |

254
docs/rfcs/rfc-027-lineage-merge-deltas.md Normal file
View file

@ -0,0 +1,254 @@
---
type: spec
title: "RFC-027 — Lineage-based merge deltas"
description: Research specification for replacing full-width branch-merge classification with Lance row-version lineage, explicitly blocked on a sublinear deletion-delta source and enforceable I/O cost gates.
status: research-blocked
tags: [eng, rfc, merge, lineage, change-feed, performance, lance, omnigraph]
timestamp: 2026-07-10
owner:
---
# RFC-027 — Lineage-based merge deltas
**Status:** Research / blocked on deletion-delta discovery
**Date:** 2026-07-10
**Depends on:** [RFC-022](rfc-022-unified-write-path.md)'s unified branch-merge
pipeline and capture-once write view
**Surveyed:** omnigraph 0.8.1; Lance 9.0.0-beta.15 (`f24e42c1`)
**Audience:** merge, storage, and performance maintainers
**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md).
Findings marked **BLOCKER** must be dispositioned before acceptance.
---
## 0. Status and decision boundary
The direction is recommended: branch merge should discover changed row IDs from
storage lineage, then read wide values only for those candidates. The proposed
replacement is **not implementation-ready** and this RFC does not authorize
removing `OrderedTableCursor`.
Two facts block the O(delta) claim:
1. filtering `_row_last_updated_at_version` is still a physical O(rows) column
scan unless a substrate index or change-log makes it selective;
2. a deleted row is absent from the target snapshot, so its version columns
cannot identify it. The current implementation finds deletions by scanning
and differencing both complete ID sets.
The RFC advances only when both candidate discovery and deletion discovery have
measured costs bounded by the changed working set. Until then the existing merge
classifier remains the correctness fallback.
## 1. Problem
Current three-way merge classification streams base, source, and target tables
and compares row signatures. A one-row change can therefore read every row and
every wide property, including embeddings and blobs, while holding merge-wide
coordination longer than necessary.
`changes/mod.rs` is narrower but not yet asymptotically different:
- changed live rows are filtered by `_row_last_updated_at_version`;
- insert/update classification builds the full base ID set;
- deletion classification scans both ID sets;
- cross-branch fallback scans and compares complete ordered rows.
Reusing that code unchanged is O(rows), not O(delta). This RFC replaces those
specific scans rather than relabeling them.
## 2. Target contract
For each table touched since the merge base:
1. discover the source and target candidate row IDs from Lance metadata;
2. classify insert, update, and delete without loading unrelated user columns;
3. join the two candidate sets into the existing merge truth table;
4. fetch complete rows only for candidates whose disposition needs values;
5. stage the resulting delta through RFC-022 and publish once.
The semantic oracle remains `merge_truth_table`. This RFC changes candidate
discovery and I/O, not conflict kinds, delete/update precedence, constraint
validation, or manifest atomicity.
## 3. Live-row candidates
On a stable-row-ID table in one physical lineage, Lance exposes:
- `_row_created_at_version` — first creation of the logical row;
- `_row_last_updated_at_version` — latest modification of the logical row.
For merge-base table version `Vb` and side version `Vs`, a live row is:
- an insert when `Vb < created_at <= Vs`;
- an update when `created_at <= Vb` and
`Vb < last_updated_at <= Vs`.
This classification removes the current full base-ID membership set. Candidate
scans project only `id`, edge endpoints when applicable, stable row ID, and the
two version columns.
It does **not** by itself make the scan O(delta). Phase R1 must prove one of:
1. Lance can build and maintain a scalar index over the version columns with
correct partial-coverage fallback;
2. transaction/fragment metadata exposes an equivalent bounded changed-row
iterator;
3. an upstream Lance change-feed primitive supplies these IDs directly.
An index is derived state. Missing or partial coverage falls back to a correct
narrow scan and is reported; it never makes merge fail or omit candidates.
## 4. Deletion delta is the blocker
Deleted logical rows have no live record whose version columns can be filtered.
The current `deleted_ids_by_set_diff` scans all IDs at base and side. That term
alone prevents an O(delta) merge, even if inserts and updates become selective.
Research must disposition these substrate-shaped options:
### 4.1 Deletion-vector and stable-row-ID lineage
Walk only transactions/fragments changed between `Vb` and `Vs`, compare their
deletion vectors, and translate newly deleted offsets to stable row IDs. This is
acceptable only if it handles merge-insert rewrites, updates that move rows,
compaction, fragment reuse, and deletion-vector materialization without scanning
unaffected fragments.
### 4.2 Upstream Lance change log
Consume a public Lance API that yields durable inserted/updated/deleted stable
row IDs by version range. A source-level prototype or private API is evidence,
not a dependency; the production surface must be public and pinned by
`lance_surface_guards.rs`.
### 4.3 Atomic OmniGraph deletion deltas
Write immutable per-commit deleted-ID rows in the same manifest CAS as the graph
commit. This is first-class commit metadata, not a side channel. It adds storage
and format liability and therefore needs a separate format decision before use;
it is not silently folded into v5.
Until one option passes the cost and correctness gates, delete-bearing histories
use the existing classifier. There is no "usually O(delta)" claim that excludes
deletes without saying so in plan and metrics.
## 5. Branch lineage and unsupported operations
Lance version numbers are branch-local and can overlap. Candidate discovery must
use each manifest entry's physical `(table_path, table_branch, table_version)`;
it must not subtract graph commit numbers or compare equal numeric versions from
different branch lineages.
Research must specify behavior for:
- a lazy fork that still physically reads its parent table branch;
- the first branch-owned write after a fork;
- compaction and index-only versions between base and side;
- overwrite, restore, schema rewrite, and hard-drop operations;
- tables without stable row IDs or version metadata;
- a table dropped or introduced on only one side.
Any shape not proven lineage-compatible takes the correctness fallback and
records a typed reason. Physical metadata gaps never weaken the logical merge.
## 6. Proposed planner shape
```text
ResolveMergeBase
-> DiscoverSideDelta(source)
-> DiscoverSideDelta(target)
-> JoinCandidateIds
-> FetchCandidateRows
-> ExistingTruthTableAndValidation
-> RFC022StageAndPublish
```
`DiscoverSideDelta` returns ordered, typed operations keyed by table and logical
row ID. Candidate order is deterministic. Payload fetches use structured key
lookups or SIP; they do not synthesize string `IN` filters or read embedding/blob
columns for candidates whose disposition needs only identity.
The merge base, physical entries, and candidate cuts are captured once before
heavy work. After acquiring publish queues, RFC-022's OCC/read-set rule either
confirms the cut or restarts discovery; it never publishes a delta classified
against a moved side.
## 7. Fallback and rollout safety
`OrderedTableCursor` remains the universal fallback through the research and
shadow phases. Fallback reasons are closed enum values, including:
- `DeletionDeltaUnavailable`;
- `VersionIndexUncovered`;
- `CrossLineageUnsupported`;
- `StableRowIdsUnavailable`;
- `OverwriteOrRestoreInRange`;
- `LineageMetadataInconsistent`.
Before enabling the new path, shadow mode runs both classifiers on the same
captured snapshots and compares ordered operation sets and merge outcomes. A
mismatch fails the test or records a production diagnostic; it never silently
chooses the new answer.
## 8. Cost contract
Use `helpers::cost` with fixed `delta = 1` and table sizes of at least 10k,
100k, and 1M rows, both scalar-only and embedding-bearing. Measure candidate
discovery separately from candidate payload fetch.
The acceptance target for a true lineage path is:
- insert and update discovery I/O is flat within named slack as rows grow;
- delete discovery I/O is flat within the same discipline;
- bytes read scale with candidate identity/version columns plus fetched delta
payload, not total row width;
- peak RSS is bounded by candidate batch size, not table size;
- graph-history depth and unrelated catalog width do not change the curve.
Local tests gate scan/fragment terms. S3 tests gate object-store RPC and opener
terms that local FS cannot expose. A benchmark without an asserted I/O slope is
supporting evidence, not the acceptance gate.
If only live-row indexing passes, this RFC may prototype or shadow an
insert/update-only path, but it does not authorize production shipping. A
partial fast path requires a separately accepted RFC that names its O(rows)
delete fallback and operational value. This RFC and the default-classifier
O(delta) claim remain blocked until R2 passes.
## 9. Correctness gates
- `merge_truth_table` remains byte-for-byte the semantic disposition oracle.
- Property tests compare lineage and cursor classifiers over inserts, updates,
deletes, endpoint moves, cycles, and conflicting constraints.
- Dedicated cases cover compaction, index maintenance, restore, overwrite,
lazy forks, and same-number/different-branch versions.
- A one-row delete in a 1M-row table is the blocker test: the RFC cannot leave
research status until it is both correct and flat in table size.
- Surface guards pin every Lance version-column, stable-row-ID, deletion-vector,
transaction, and index behavior used by the chosen implementation.
- Fault tests move a side after discovery and prove OCC restarts instead of
publishing stale classification.
## 10. Observability
For every merge table report classifier path, fallback reason, source/target
candidate counts, deleted-ID discovery mechanism, fragments and bytes read,
payload columns fetched, discovery latency, and whether shadow results matched.
Metrics distinguish logical delta size from physical discovery work. This is
required to catch an apparently correct path silently regressing to O(rows).
## 11. Research plan
| Phase | Content | Exit criterion |
|---|---|---|
| R0 | Instrument current cursor and `changes` paths; build shadow comparison harness | semantic and I/O baselines |
| R1 | Prove selective live-row discovery and branch-version mapping | insert/update flat-cost gate |
| R2 | Prototype all deletion-delta options against pinned Lance | one option passes delete correctness + flat-cost gate |
| R3 | Shadow new classifier across the full merge truth table and histories | zero mismatches; explicit fallback ledger |
| R4 | Enable lineage path behind a scoped feature/config gate | production diagnostics within budgets |
| R5 | Make lineage default and consider cursor retirement | all fallbacks dispositioned; no physical gap can break merge |
The RFC remains **research / blocked** through R2. Choosing an OmniGraph commit
delta format in §4.3 requires its own format amendment before R3.