omnigraph/docs/rfcs/rfc-024-durable-table-heads.md

type	title	description	status	tags	timestamp	owner
spec	RFC-024 — Durable table heads and the v5 manifest	Materialize one live-or-tombstoned current-state row per table inside each manifest branch, prove bounded physical lookup, and migrate every branch atomically before stamping v5.	draft	eng rfc manifest write-path versioning migration lance	2026-07-10

RFC-024: Durable table heads and the v5 manifest

Status: Draft / for team review Date: 2026-07-10 Surveyed: omnigraph 0.8.1 (main); Lance 9.0.0-beta.15 at git rev f24e42c1; full Lance table layout, transaction, branching, indexing, compaction, cleanup, and object-store specifications Relationship to RFC-022: this RFC is the durable-heads decision split from the earlier monolithic RFC-022 draft. RFC-022 defines the shared publisher/recovery protocol; this RFC owns the v5 format and migration boundary. It deliberately excludes checkpoint retention, which RFC-025 reviews separately. Key fencing in RFC-023 is also independently reviewable; the two may share a release but do not block one another's evidence gates. Audience: engine, manifest, migration, branch, and release maintainers Open architecture review: RFC-022–027 review ledger. Findings marked BLOCKER must be dispositioned before acceptance.

---

Throughout this draft, v5 means the next internal schema after today's v4. If another independently accepted format change lands first, the release number is reassigned; none of the head-row semantics depend on the numeral.

0. Decision summary

The current manifest is both an immutable graph journal and the place writers ask "what is current?" Current-state resolution folds history, so its physical cost grows with commit count even though the answer contains only one value per table.

v5 adds one mutable, durable table_head row per stable table identity inside each native __manifest branch. The publisher updates the head in the same Lance merge-insert transaction as immutable table-version rows, tombstone events, graph lineage, and graph_head. The journal remains the history source; heads become the current-state source.

The format does not ship merely because the logical result has O(tables) rows. A filtered scan over a history-sized Lance table is still O(history) physical work. v5 is gated on a Lance-native indexed lookup whose measured I/O is flat at history depth and whose uncovered tail is bounded and observable.

Normative decisions:

1. Heads live in __manifest; there is no second heads dataset and no warm mutable-tip authority.
2. Head state is explicitly live or tombstoned and carries an incarnation.
3. Every publish and recovery outcome updates journal and head atomically.
4. Current-state reads use heads; history reads use the journal.
5. Missing or duplicate heads in v5 are corruption, not a reason to silently return to the history fold.
6. Migration covers every live manifest branch, then stamps main v5 last.
7. If RFC-023 is co-released, the combined migration also verifies PK fencing; durable heads do not depend on that decision.
8. Checkpoint/retention markers are deferred to a separate RFC.

1. Problem

__manifest currently stores immutable table-version and tombstone rows. To resolve a branch tip, readers scan those rows, select the greatest version per table, and apply tombstones. This makes a write's coordination cost depend on the total graph history. Compaction reduces fragment count but cannot remove the semantic journal rows, so even a compacted manifest retains a row-volume slope.

Caching that fold as mutable in-process state is the wrong authority shape for writes: invalidation becomes a correctness condition across processes and branch incarnations. Storing the folded answer durably in the same transaction as the journal removes both liabilities:

• no parallel authority can drift; and
• current-state work is proportional to catalog width, provided physical lookup is also bounded.

2. Scope and non-goals

In scope:

• v5 table-head schema and state transitions;
• atomic publisher, recovery, and current-read semantics;
• bounded physical access proof;
• all-branch predecessor→heads migration and compatibility refusal;
• optional co-release integration with RFC-023's all-branch PK activation.

Out of scope:

• a GraphState singleton or warm publish input;
• a separate __heads Lance dataset;
• deleting immutable journal rows;
• commit-graph ancestry acceleration;
• checkpoint retention and arbitrary-version GC;
• using a physical index as a correctness precondition.

Checkpoint retention is excluded because a row in __manifest does not by itself pin a version in another Lance dataset. Lance cleanup recognizes its own versions, tags, and branch references, not foreign references. A later RFC must define substrate-native per-table pins and their crash-safe lifecycle.

3. v5 table-head schema

Each native manifest branch contains exactly one current head row for each stable table identity known to that branch.

3.1 Identity and object key

object_id   = "table_head:<stable_table_id>"
object_type = "table_head"

stable_table_id survives a type rename. It is not the display name and is not derived anew from kind:name. Closing the current rename-stable identity gap is a v5 ship gate, and this RFC owns it: the stable-ID encoding, the compiler contract that mints and preserves it, and the incarnation baseline are defined here. RFC-023 §4.1 (rename-safe PK claims) and RFC-025 §2.1 (checkpoint table rows) consume this identity and must not ship identity-dependent claims before it lands. No other RFC in the family defines a competing identity scheme.

💬 Superseded by review (BLOCKER-07 in the review ledger): ownership inside this RFC contradicts siblings that consume the identity while calling this RFC optional (RFC-025) or without declaring the dependency (RFC-026). Disposition: extract identity/incarnation into its own prerequisite RFC, or accept this RFC in full before the identity-dependent siblings. Replace this ownership note accordingly.

The head row is mutable under WhenMatched::UpdateAll, just like graph_head:<branch>. There is one object ID per stable identity; recreation does not leave multiple candidate heads.

3.2 Payload

v5 reuses the manifest's typed columns where they already fit and stores the remaining versioned payload in a typed JSON structure:

TableHeadMetadata {
    state: "live" | "tombstoned",
    stable_table_id: String,
    incarnation_id: ULID,
    schema_hash: String,
    head_graph_commit_id: Option<ULID>,
}

The row columns have these meanings:

Column	live	tombstoned
table_key	current public table key/name	last public key/name
location	physical table location	last physical location, diagnostic only
table_version	current visible Lance version	last live Lance version
table_branch	physical owner branch, nullable for main	last owner branch
row_count	current row count	null
metadata	TableHeadMetadata	TableHeadMetadata; head_graph_commit_id names the tombstoning graph commit

The state field is authoritative. A tombstoned row MUST NOT become live merely because an older journal version has a greater data-table version than some other row.

3.3 Incarnations

incarnation_id distinguishes drop/recreate ABA:

• rename preserves stable ID and incarnation;
• ordinary writes preserve both;
• physical owner handoff preserves both;
• dropping the type transitions the one head to tombstoned;
• recreating a logically new table mints a new incarnation and updates the same stable-identity head to live only through an explicit schema operation.

If recreation is assigned a new stable table identity by schema semantics, it gets a new head object and the old identity remains tombstoned. The schema planner, not a name comparison, chooses this outcome.

3.4 Journal identity

The mutable head is not the only place that records identity. Every v5 table-version, registration, rename, and tombstone journal event carries stable_table_id and incarnation_id; otherwise drop/recreate followed by a new physical dataset whose Lance versions restart cannot be replayed unambiguously.

Migration writes one immutable v5 incarnation-baseline row per known identity, bound to the source-tip digest. New registrations/recreations write an immutable incarnation transition. Head repair starts from that baseline and replays only identity-bearing v5 events. It does not infer identity from mutable names or compare unrelated Lance version numbers. If pre-v5 history cannot be mapped to one baseline identity deterministically, migration refuses before the stamp.

4. State-transition rules

Event	Required head transition
Register table	absent → live, new incarnation
Data write / optimize / index publish	live version N → live version M
Owner-branch handoff	live owner A → live owner B, even if version is equal
Rename	live key/name A → live key/name B; identity/incarnation unchanged
Drop table	live → tombstoned in the same graph publish as the tombstone journal row
Recreate	tombstoned → live with the schema-planned identity/incarnation outcome
Recovery roll-forward	apply the failed writer's intended live/tombstone transition
Recovery rollback	publish a head matching the restored physical version and logical pre-write state

No path may append a table-version or tombstone journal row without including its corresponding head mutation in the same publisher source batch.

5. Atomic publisher contract

The v5 publisher constructs one merge-insert source containing:

• immutable, identity-bearing table-version rows;
• immutable, identity-bearing table-tombstone/transition rows;
• mutable table-head rows;
• when the RFC-022 plan carries LineageIntent, the immutable graph_commit and mutable graph_head:<branch> rows;
• for metadata-only plans, their specific CAS authority/operation rows without manufacturing graph lineage.

One Lance manifest commit makes the entire set visible. The publisher still resolves the graph parent and re-reads commit authority inside every CAS retry. Expected table versions are compared against table heads, not reconstructed by folding the journal.

Two graph-content writers touching disjoint data tables still contend on graph_head:<branch>, form one linear graph history, and re-parent on retry. Writers touching the same table also contend on its one table_head object. Metadata-only CASes contend on the stable authority rows named by their complete ReadSet; they do not update graph_head merely to create contention.

The immutable journal remains necessary for snapshots, diffs, audit, migration verification, and head repair. Head rows do not replace or truncate it.

6. Read contract

6.1 Current state

A v5 current-state read:

1. derives the expected live stable table IDs from the pinned catalog and fixed system-table registry;
2. issues a structured lookup for the exact head object IDs;
3. requires exactly one valid row per expected identity;
4. includes only rows whose authoritative state is live;
5. validates schema identity from the head payload;
6. returns one immutable Snapshot used for the operation's lifetime.

Missing, duplicate, unknown-state, or schema-mismatched live heads fail loudly. The hot path does not enumerate every identity ever dropped merely to prove all tombstone heads exist; the branch completion digest plus explicit heads verify/repair owns bounded tombstone-set validation. A missing or duplicate tombstone is still corruption, but normal reads do not regain an O(history) scan to discover it.

6.2 History

snapshot_at_version, commit resolution, change feeds, and audit continue to read immutable journal/lineage state at the requested manifest version. A v5 manifest version contains the heads as they stood at that version, but the journal remains the normative explanation of how the state arose.

6.3 Diagnostic repair

An explicit offline repair loads the branch's v5 incarnation baseline, replays identity-bearing journal transitions from that baseline, compares the result to its heads/marker digest, and publishes corrected heads with an audited system actor. Repair is not part of the read hot path and never silently runs from a query.

7. Bounded physical lookup is a ship gate

Logical O(tables) output does not prove physical O(tables) work. Without an index, object_id IN (...) still scans the journal-bearing manifest fragments; compaction reduces files but not semantic row count.

7.1 Required property

At fixed catalog width, a reconciled v5 current-state lookup MUST have zero positive slope in:

• manifest object-store reads;
• bytes read;
• fragments/pages scanned; and
• rows decoded

as commit history grows.

The bound must hold on a real object store as well as local FS and must be shown for compacted and uncompacted histories. The test uses the shared IO-tracking harness and installs the tracker before the manifest handle opens.

7.2 Candidate access shape

The primary candidate is a structured exact lookup on object_id backed by a Lance scalar index. It is acceptable only if measurement proves:

• indexed head lookup avoids journal-fragment scans;
• newly committed head rows leave at most a bounded uncovered tail;
• reconciliation restores coverage without synchronous expensive work in the logical write path;
• index absence or partial coverage remains logically correct and is surfaced as an observable degraded-cost mode.

The index is derived state. Queries MUST remain correct if it is missing, and a missing index cannot block a logical write. The performance promise applies to the reconciled serving state and includes an explicit bound on uncovered work; it is not inferred merely from the existence of an index declaration.

7.3 Rejected access shape

A separate heads dataset is rejected. Lance commits are per dataset, so it would reintroduce a journal→heads crash gap and require another sidecar protocol for the very pointer whose purpose is to remove drift.

If no in-manifest Lance-native access shape passes the gate, v5 does not ship with head rows. The fallback is to retain v4 plus the local session/view-passing improvements, not to waive the cost claim.

8. Recovery protocol

Data-table writers still use the existing four phases:

1. write sidecar before a Lance HEAD advance;
2. commit staged/inline table work;
3. publish __manifest;
4. delete sidecar.

The sidecar's logical intent in v5 includes the expected and desired table-head payload. Recovery behavior is therefore complete:

• roll-forward publishes the data version, journal row, table head, lineage, and graph head together;
• rollback restores the physical version, then publishes journal/audit state and a table head matching the restored logical state;
• a stale sidecar whose goal is already represented by the exact head incarnation/version converges idempotently;
• a partially matching head is not treated as success.

Recovery remains a synchronous barrier before any later writer advances a touched table. Index reconciliation may be asynchronous; unresolved commit protocol state may not.

9. v5 boundary and compatibility

v5 comprises:

1. table-head rows and their publish/read semantics;
2. branch-local v5 completion markers; and
3. the graph-level internal-schema stamp written after verification.

It does not comprise checkpoint/retention markers.

If RFC-023 is independently accepted and ready for the same release, v5 may also carry its PK annotation and fencing-compatible marker after a combined migration review. That is release coordination, not a prerequisite of heads.

Format sequencing is explicit:

Order	Result
Heads first	proposed v5 contains heads; later fencing migration preserves and atomically updates heads for every PK-version repoint
Fencing first	fencing takes the next format number; this heads migration takes the following number, accepts that exact predecessor, and preserves its PK/stamp invariant
Co-release	one format maps to both independently accepted capabilities and runs the combined failure matrix

Migration code dispatches from an exact supported predecessor feature set; it never assumes that “pre-heads” means pristine v4 or drops a capability it does not own.

After upgrade, serving is strict-single-version:

• v5 binaries refuse unstamped or partially migrated graphs in normal mode;
• older binaries refuse v5 with the existing upgrade message;
• every open reads the graph-wide main stamp before selecting a named branch;
• only the dedicated offline migration command may open the exact supported predecessor for conversion;
• there is no mixed v4/v5 serving period.

10. All-branch predecessor→heads migration

10.1 Preconditions

The operator stops every graph writer and acquires an atomic create-if-absent migration claim (with exact owner/fencing token; not a Lance branch sentinel). The barrier covers server, CLI, embedded, maintenance, branch, and schema writes. Because predecessor binaries do not understand the new in-graph migration marker, the offline fleet barrier remains mandatory until the final stamp.

The claim uses PutMode::Create, records the migration operation and owner token, and permits no time-only takeover. Recovery classifies the durable ledger/sidecars under the fleet outage before replacing a stale token.

After acquiring the claim, migration completes RFC-022's recovery barrier before pinning any branch source tip.

If fencing is co-released, the migration first executes RFC-023's all-branch table-PK plan. Otherwise head migration neither adds nor changes PK metadata.

10.2 Per-branch conversion

For each live native __manifest branch:

1. capture branch name, ref incarnation, manifest version, and e_tag/timestamp;
2. run the predecessor journal+tombstone fold once at that pinned tip;
3. construct exactly one live or tombstoned head plus one immutable incarnation baseline per stable table identity;
4. validate table schema hashes and, only in a combined release, RFC-023 PK state;
5. publish all heads/baselines, an audited migration record (actor = omnigraph:migration/v5), and a branch-local completion marker in one manifest CAS that revalidates the captured ref incarnation/source tip; this physical representation migration does not create a graph-content commit or move graph_head;
6. store in the marker the source tip, head count, identity-baseline digest, head-set digest, and heads-format version; the marker is content-scoped and deliberately does not embed the source branch incarnation; source tip is provenance, while digest/format determine inherited-marker validity;
7. verify by reopening the produced branch version through the v5 head reader.

The completion marker has a deterministic object ID within each manifest branch. A retry that finds a matching source/digest is complete; a mismatching marker is a loud migration conflict.

10.3 Finalization

After all branches report completion, migration re-enumerates native manifest refs and verifies the same incarnation set. Branch create/delete is blocked, so any difference indicates out-of-band modification and aborts finalization.

Only then does it stamp main as internal schema v5. The stamp is the fleet commit point: before it, no serving process may start; after it, only v5 code may open the graph.

New branches created after v5 fork a source manifest that already contains complete heads and the content-scoped v5 marker. The inherited marker remains valid for the identical snapshot; branch open separately binds that content to the new native ref incarnation and validates the graph-wide stamp. No post-create marker rewrite is required.

11. Migration recovery

The migration keeps a durable ledger outside the ordinary read path with one record per branch and, in a combined release, per RFC-023 table conversion. It records expected source incarnations, achieved physical versions, produced manifest versions, and digests.

Recovery is idempotent and roll-forward-only:

• completed, digest-matching branch conversions are skipped;
• a table HEAD advance not yet represented in its graph branch is recovered by the normal sidecar and then branch conversion resumes;
• an uncommitted head-row source leaves no visible state and is rebuilt;
• a committed branch marker is authoritative for that branch conversion;
• the main v5 stamp is never written while any ledger unit is incomplete;
• co-released PK metadata is never cleared to simulate rollback.

If migration crashes, the graph remains offline. Restarting an old serving binary is not a recovery procedure; the operator resumes the v5 migration.

12. Tests and acceptance gates

12.1 Head semantics

• current state from heads is byte-equivalent to the predecessor fold across realistic histories;
• live→tombstoned never resurrects an older live version;
• drop/recreate distinguishes incarnations;
• identity-bearing journal replay remains unambiguous when a recreated physical dataset restarts Lance version numbering;
• rename preserves identity/incarnation and changes the public key only;
• owner-branch handoff at an equal table version updates the head;
• missing, duplicate, malformed, and schema-mismatched heads fail loudly.
• heads verify detects a missing/duplicate tombstone against the baseline and completion digest without adding that enumeration to every current read.

12.2 Publisher and recovery

• concurrent disjoint writers produce one linear graph chain and correct heads;
• same-table writers contend on one head row;
• failpoints after every table commit but before manifest publish recover to matching physical version, journal, table head, and graph head;
• rollback and roll-forward assertions include head payloads, not only table versions;
• a stale sidecar converges exactly once with one audit record.

12.3 All-branch migration

• main, owned named branches, and lazy-inherited branches all convert;
• tombstoned types are represented on every relevant branch;
• crashes before/after every per-branch CAS resume from the ledger;
• branch ref deletion/recreation is caught by incarnation verification;
• final stamp is impossible while any table or branch marker is absent;
• each branch's graph head and content lineage are unchanged by head migration;
• old/new binary refusal matrix covers every supported predecessor capability set, partial heads migration, and complete heads format;
• a post-v5 branch inherits and validates complete heads.
• the inherited completion marker is content-scoped while ref-incarnation validation is performed separately.

12.4 Cost gates

At fixed table count and increasing commit depth, assert flat curves for manifest reads, bytes, fragments/pages, and decoded rows:

• local FS, compacted and uncompacted;
• S3/RustFS with real e_tags;
• warm repeated read;
• cold operation-local open with shared Session;
• one uncovered-head update before reconciliation;
• reconciled steady state.

The test must fail if the lookup silently falls back to scanning journal history in the claimed steady state.

The decode term is part of the gate: parsing head rows — including the typed JSON TableHeadMetadata payload — must be bounded by catalog width. A per-read parse cost that grows with anything other than table count fails the gate even when I/O is flat.

12.5 Format guards

• exact v5 head metadata schema and object IDs;
• one head row per stable identity;
• incarnation-baseline and identity-bearing journal event schemas;
• content-scoped branch completion marker schema/digest;
• RFC-023 PK metadata on node and edge tables when the release combines them;
• v5 publisher source always pairs a journal/tombstone event with a head row.

13. Decisions and open gates

Decided

• Heads and journal share one __manifest transaction.
• Current reads use heads; historical reads keep the journal.
• Heads represent live | tombstoned plus incarnation explicitly.
• A separate heads dataset and a mutable in-process tip authority are rejected.
• Migration is offline, all-branch, resumable, and stamps v5 last.
• RFC-023 PK activation is verified by v5 only when deliberately co-released.
• Checkpoint retention is deferred.

Open ship gates

1. Rename-stable table/type identity and final stable-ID encoding — owned by this RFC; consumed by RFC-023 and RFC-025 (§3.1).
2. The in-manifest indexed lookup implementation and bounded uncovered-tail proof.
3. Passing local and object-store depth-slope cost gates.
4. Atomic cross-process migration-claim implementation and operator runbook.
5. Final v5 metadata JSON/typed-column compatibility review.
6. Full all-branch migration/failpoint matrix.