mirror of
https://github.com/ModernRelay/omnigraph.git
synced 2026-07-12 03:12:11 +02:00
Enroll branch merge in RFC-022 write path (#345)
This commit is contained in:
parent
bd4c614e42
commit
e0e145aa92
22 changed files with 3722 additions and 464 deletions
|
|
@ -258,10 +258,15 @@ them explicit.
|
|||
schema-apply lock branch) — design it before promoting multi-process write
|
||||
topologies.
|
||||
- **Fork ownership is durable, but Lance ref deletion is not conditional:** a
|
||||
first-touch Mutation/Load table no longer creates its target ref while
|
||||
preparing. Under schema → branch → table gates it revalidates, durably arms a
|
||||
schema-v3 sidecar naming the target, creates the ref, then stages branch-local
|
||||
files and commits. Both `reclaim_orphaned_fork_and_refork` and
|
||||
first-touch Mutation/Load or BranchMerge table never creates its target ref
|
||||
before recovery ownership is durable. Under schema → branch → table gates it
|
||||
revalidates and arms a schema-v3 mutation/load or schema-v4 merge sidecar
|
||||
naming the target; branch merge additionally records the exact fork version
|
||||
and a pre-minted ordered transaction chain for every logical data effect,
|
||||
commits that chain without transparent conflict retries, and confirms the
|
||||
minted target `BranchIdentifier`. Recovery proves the exact chain rather than
|
||||
treating numeric HEAD movement as ownership. Both
|
||||
`reclaim_orphaned_fork_and_refork` and
|
||||
`reconcile_orphaned_branches` consult pending sidecars before destruction; a
|
||||
foreign claim is conflict/indeterminate, never permission to delete. Full
|
||||
recovery accepts sidecar-before-ref crashes. When `BranchContents` is absent,
|
||||
|
|
|
|||
|
|
@ -36,7 +36,7 @@ The implementation spec's hard gate for this phase — failpoint recovery tests
|
|||
|
||||
What Phase 4 builds on (all shipped):
|
||||
|
||||
- **The engine's recovery discipline.** Writers that can advance Lance HEAD before manifest publish write `__recovery/{ulid}.json` sidecars carrying per-table pins (`expected_version`, `post_commit_pin`); `Omnigraph::open` in read-write mode classifies every pinned table (`NoMovement` / `RolledPastExpected` / `UnexpectedAtP1` / `UnexpectedMultistep` / `InvariantViolation`) and decides all-or-nothing: roll forward via one manifest publish, or roll back via `Dataset::restore`. Every completed action records an audit row with the interrupted writer's actor in `recovery_for_actor`. A schema-v3 roll-forward publishes the sidecar's pre-minted lineage and therefore preserves that writer's commit id and actor; rollback and legacy recovery commits use `omnigraph:recovery`. The cluster inherits the *vocabulary* of this design but not its mechanics — see the roll-forward-only argument below.
|
||||
- **The engine's recovery discipline.** Writers that can advance Lance HEAD before manifest publish write `__recovery/{ulid}.json` sidecars carrying per-table pins (`expected_version`, `post_commit_pin`); `Omnigraph::open` in read-write mode classifies every pinned table (`NoMovement` / `RolledPastExpected` / `UnexpectedAtP1` / `UnexpectedMultistep` / `InvariantViolation`) and decides all-or-nothing: roll forward via one manifest publish, or roll back via `Dataset::restore`. Every completed action records an audit row with the interrupted writer's actor in `recovery_for_actor`. Schema-v3 Mutation/Load and schema-v4 BranchMerge roll-forward publish the sidecar's pre-minted lineage and therefore preserve that writer's commit id and actor; v4 additionally proves each multi-commit data effect from its ordered pre-minted Lance transaction chain. Rollback and legacy recovery commits use `omnigraph:recovery`. The cluster inherits the *vocabulary* of this design but not its mechanics — see the roll-forward-only argument below.
|
||||
- **The engine's schema-apply surface.** `apply_schema_as(desired_source, SchemaApplyOptions { allow_data_loss }, actor)` returns `SchemaApplyResult { supported, applied, manifest_version, steps }`; `preview_schema_apply_with_options` returns the migration plan plus desired catalog without applying; the `__schema_apply_lock__` branch serializes schema applies graph-wide and refuses to run while user branches exist. Policy enforcement (`enforce(SchemaApply, TargetBranch("main"), actor)`) happens before the lock.
|
||||
- **Graph init.** `Omnigraph::init(uri, schema_source)` with a strict preflight (errors if schema artifacts exist) and an atomic `_schema.pg` claim. A documented gap: a failed init does not clean up Lance datasets or `__manifest/` it already created.
|
||||
- **No engine graph-delete primitive.** Deleting a graph today means removing its object-store prefix. This RFC works with that fact rather than waiting on a primitive.
|
||||
|
|
|
|||
|
|
@ -432,8 +432,9 @@ retaining the public lifecycle status `Proposed`.
|
|||
|
||||
**Affected:** [RFC-022 §6.2](../rfcs/rfc-022-unified-write-path.md#62-branch-merge)
|
||||
|
||||
**Status:** Closed in specification on 2026-07-11; full merge-adapter conversion
|
||||
remains rollout work rather than an architecture ambiguity.
|
||||
**Status:** Closed in specification and implementation on 2026-07-11. Branch
|
||||
merge captures an immutable source commit and revalidates only its incarnation;
|
||||
the target publishes and recovers under its own exact authority token.
|
||||
|
||||
RFC-022 requires every `ReadSet` member to be arbitrated atomically by the
|
||||
publish CAS. A CAS on reserved main cannot arbitrate a row on a named source
|
||||
|
|
@ -516,6 +517,14 @@ performance.
|
|||
> multi-process native-ref fencing. RFC-024 remains the false-contention
|
||||
> narrowing step.
|
||||
|
||||
> **Branch-merge implementation disposition (2026-07-11):** branch merge uses
|
||||
> the same coarse target token without pretending the source belongs to that
|
||||
> atomic read set. Schema-v4 recovery persists the captured source parent,
|
||||
> fixed merge/rollback ids, pre-minted exact transaction chains for multi-commit
|
||||
> data effects, ref-only physical effects, and the complete logical delta. A later source advance remains harmless; a target
|
||||
> advance after effects becomes `RecoveryRequired` and is compensated rather
|
||||
> than re-parented.
|
||||
|
||||
### TIGHTENING-01 — symmetric Lance conflicts are not obviously an activation gate
|
||||
|
||||
RFC-023 correctly identifies Lance's directional filtered/unfiltered behavior.
|
||||
|
|
|
|||
|
|
@ -52,9 +52,9 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
|
|||
| `merge_cost.rs` | Cost-budget tests for branch MERGE on the shared `helpers::cost` harness: `merge_validation_is_delta_scoped` (a 1-row-delta merge opens ≤3 data tables — Δ-scoped, not the whole catalog; was ~6 pre-#5) and `merge_manifest_cost_grows_with_history` (the cross-branch `__manifest` open amplification still grows with commit depth — a separate, not-yet-addressed term — while validation `data_open_count` stays flat) |
|
||||
| `policy_engine_chassis.rs` | Engine-layer Cedar enforcement (MR-722): allow + deny through every `_as` writer via the SDK directly — no HTTP — proving embedded and CLI callers hit the same gate as the server, with action × scope shapes matching `authorize_request` |
|
||||
| `maintenance.rs` | `optimize` (compaction), `repair` (explicit uncovered-drift publish), and `cleanup` (version GC): empty/idempotent/no-op edges, policy validation, head preservation; cleanup pins exact keep-count behavior (including keep larger than history), count/time retention of a live lazy branch, the oldest of multiple lazy pins, graph-wide fail-closed ordering on an unopenable pin, and refusal of uncovered main HEAD drift before any GC; `optimize` publishes its own compaction (`optimize_publishes_compaction_to_manifest_so_schema_apply_succeeds`), skips pre-existing uncovered drift (`optimize_skips_preexisting_manifest_head_drift`), refuses to run while a `__recovery` sidecar is pending (`optimize_defers_when_recovery_sidecar_is_pending`), and compacts blob-v2 tables through the normal path (`optimize_compacts_blob_table_alongside_plain_table`); `repair` previews/heals verified maintenance drift, refuses raw semantic drift without `--force`, and forced repair publishes only by explicit operator choice; the index reconciler (iss-848): `index_build_tolerates_null_vector_rows` (logical load succeeds without inline index work; an untrainable Vector column then defers during reconciliation) and `optimize_materializes_index_declared_but_unbuilt` (optimize creates a declared-but-deferred index) |
|
||||
| `failpoints.rs` | Failure-injection coverage (gated on `failpoints` feature). RFC-022 includes deterministic post-stage/pre-effect races for mutation/load uniqueness and strict disjoint-head changes, plus the cross-handle post-effect `RecoveryRequired` → read-write-open rollback cell. Native controls are pinned by `native_branch_controls_reclassify_lost_acknowledgements` (matching create and absent-ref delete, with no version/lineage movement); `armed_first_touch_recovery_accepts_missing_target_ref` additionally forges and reclaims the clone-only/no-`BranchContents` table state. Legacy path overlap has both sides pinned: `armed_first_touch_recovery_defers_legacy_path_overlap_until_leaf_delete` permits open only for a proven no-effect intent, while `partial_first_touch_recovery_fails_closed_on_legacy_path_overlap` leaves one exact multi-table effect and verifies open fails closed until offline leaf cleanup lets rollback converge. Other control/recovery race cells include `first_touch_post_create_open_error_keeps_recovery_ownership`, `branch_delete_orphans_sidecar_armed_after_initial_barrier`, `branch_merge_fences_target_delete_recreate_aba`, `branch_merge_fences_concurrent_sync_on_same_handle`, `branch_merge_rejects_fresh_target_manifest_change_before_effects`, `branch_merge_rechecks_late_sidecar_after_table_gates`, `cleanup_rechecks_sidecars_under_gc_gates`, and `full_recovery_rereads_sidecar_body_after_discovery`. The suite also includes the five per-writer effect → manifest-CAS recovery tests, write-entry in-process heal contract, storage-fault matrix, S3 recovery twin, and convergence-idempotent roll-forward regression. |
|
||||
| `failpoints.rs` | Failure-injection coverage (gated on `failpoints` feature). RFC-022 includes deterministic post-stage/pre-effect races for mutation/load uniqueness and strict disjoint-head changes, plus the cross-handle post-effect `RecoveryRequired` → read-write-open rollback cell. Branch merge adds the captured-source advance cell; post-confirm target-winner compensation; mixed physical + pointer-only delta recovery with fixed commit id/actor/parents; and both sidecar-before-first-ref and ambiguous-ref-create recovery. Native controls are pinned by `native_branch_controls_reclassify_lost_acknowledgements` (matching create and absent-ref delete, with no version/lineage movement); `armed_first_touch_recovery_accepts_missing_target_ref` additionally forges and reclaims the clone-only/no-`BranchContents` table state. Legacy path overlap has both sides pinned: `armed_first_touch_recovery_defers_legacy_path_overlap_until_leaf_delete` permits open only for a proven no-effect intent, while `partial_first_touch_recovery_fails_closed_on_legacy_path_overlap` leaves one exact multi-table effect and verifies open fails closed until offline leaf cleanup lets rollback converge. Other control/recovery race cells include `first_touch_post_create_open_error_keeps_recovery_ownership`, `branch_delete_orphans_sidecar_armed_after_initial_barrier`, `branch_merge_fences_target_delete_recreate_aba`, `branch_merge_fences_concurrent_sync_on_same_handle`, `branch_merge_rejects_fresh_target_manifest_change_before_effects`, `branch_merge_rechecks_late_sidecar_after_table_gates`, `cleanup_rechecks_sidecars_under_gc_gates`, and `full_recovery_rereads_sidecar_body_after_discovery`. The suite also includes the five per-writer effect → manifest-CAS recovery tests, write-entry in-process heal contract, storage-fault matrix, S3 recovery twin, and convergence-idempotent roll-forward regression. |
|
||||
| `failpoint_names_guard.rs` | Source-walk guard: every `maybe_fail(...)` call site (engine + cluster) must reference the compile-checked `failpoints::names` consts, never a bare string literal — a typo'd literal compiles but silently never fires; same defense-in-depth shape as `forbidden_apis.rs` |
|
||||
| `recovery.rs` | Open-time recovery sweep — legacy sidecars plus schema-v3 Mutation/Load exact transaction identity, fixed logical/rollback outcome IDs, branch-token comparison, fresh under-gate sidecar reread/reparse, all-or-nothing roll-forward/rollback/refusal, audit row in `_graph_commit_recoveries.lance`, and `OpenMode::ReadOnly` skip path |
|
||||
| `recovery.rs` | Open-time recovery sweep — legacy sidecars; schema-v3 Mutation/Load exact transaction identity; schema-v4 BranchMerge exact multi-commit chains/ref-only effects plus complete logical delta; restartable compensation recognition; fixed logical/rollback outcome IDs; branch-token comparison; fresh under-gate sidecar reread/reparse; all-or-nothing roll-forward/rollback/refusal; audit row in `_graph_commit_recoveries.lance`; and `OpenMode::ReadOnly` skip path |
|
||||
| `composite_flow.rs` | Compositional/narrative end-to-end stories — multi-step flows that compose mechanics covered by other test files. Catches integration regressions where individual operations all pass their unit tests but their composition breaks (sequential merges, post-merge main writes, time-travel through merge DAG, reopen consistency over multi-merge histories, post-optimize and post-cleanup strict writes). |
|
||||
|
||||
## Fixtures
|
||||
|
|
|
|||
|
|
@ -75,38 +75,55 @@ The native branch identifier detects delete/recreate ABA but is not a Lance
|
|||
conditional-ref fence, and destructive recovery remains unsafe beside a live
|
||||
foreign process.
|
||||
|
||||
### Branch-merge authority fence (adapter bridge)
|
||||
### Branch-merge authority and recovery adapter (RFC-022 v4)
|
||||
|
||||
Branch merge still uses its writer-specific multi-commit table effects and
|
||||
confirmation sidecar; it has not yet been converted to the RFC-022 exact-effect
|
||||
adapter. It does, however, join the closed control boundary needed by this first
|
||||
slice: after the strict recovery barrier it acquires the root-shared schema gate
|
||||
and the sorted source/target branch gates, performs the final sidecar check,
|
||||
loads one operation-local catalog from the accepted contract, captures both graph
|
||||
heads plus the base/source/target snapshots, and holds those gates through table
|
||||
effects and manifest publication. Planning stays outside table queues. Before
|
||||
Phase A, merge acquires the conservative all-catalog table envelope for both
|
||||
source and target, re-lists sidecars, and compares fresh source/target manifest
|
||||
versions with the captured snapshots. A stale warm handle catalog or coordinator
|
||||
snapshot is never accepted as that revalidation.
|
||||
Branch merge retains its writer-specific row classifier and multi-commit table
|
||||
algorithms, but its authority, recovery, and visibility boundary now use the
|
||||
RFC-022 adapter contract:
|
||||
|
||||
The source snapshot is a captured merge input, not authority that the target
|
||||
manifest CAS can arbitrate. The current process-local source gate is a stronger
|
||||
same-process fence around that capture, including delete/recreate ABA, but the
|
||||
semantic contract is still "merge the captured source commit." A later source
|
||||
advance does not invalidate an otherwise prepared target publish. Claiming
|
||||
"latest source at target publish" would instead require a cross-process source
|
||||
fence held through the target CAS.
|
||||
1. capture source and target as coherent `WriteTxn` snapshots. The target token
|
||||
is `(BranchIdentifier, exact optional graph_head, accepted schema identity)`;
|
||||
the effective lineage head is captured separately because a fresh named
|
||||
branch can inherit a parent while its own `graph_head:<branch>` row is absent;
|
||||
2. compute the merge base from those captured commit ids and classify against
|
||||
the immutable base/source/target snapshots outside table gates;
|
||||
3. acquire the conservative all-catalog source/target table envelope, re-list
|
||||
recovery intent, revalidate the complete target token, and revalidate the
|
||||
source incarnation. A target change returns typed `ReadSetChanged` before
|
||||
effects. A later source-head advance is allowed: the contract is "merge the
|
||||
captured source commit," never "substitute whatever source is latest";
|
||||
4. pre-mint the merge lineage and each table's ordered Lance data-transaction
|
||||
chain, then arm a schema-v4 BranchMerge sidecar before the first HEAD advance
|
||||
or first-touch table ref. Logical data steps commit with those exact
|
||||
`(read_version, uuid)` identities and zero transparent conflict retries. Its
|
||||
physical-effect set can be smaller than its intended manifest delta:
|
||||
pointer-only table updates are still recorded so recovery publishes the
|
||||
complete logical merge;
|
||||
5. after every multi-commit table effect completes, confirm exact final table
|
||||
versions, every logical `SubTableUpdate`, and every first-touch target
|
||||
`BranchIdentifier`; then publish once with `ExactGraphHead` and the captured
|
||||
table expectations.
|
||||
|
||||
That fence prevents a same-process target delete/recreate from reusing the branch
|
||||
name underneath a merge plan. The race test deliberately recreates a target with
|
||||
the same name and numeric Lance version but a different `BranchIdentifier`, so
|
||||
version-only checking cannot accidentally satisfy it. This is a process-local
|
||||
bridge, not a cross-process conditional-ref primitive and not a substitute for
|
||||
the later full branch-merge read-set/reprepare adapter. `sync_branch` joins the
|
||||
same root schema gate before replacing a handle's coordinator, so it cannot
|
||||
overwrite merge's temporary target coordinator or change a native control's
|
||||
active-branch authority mid-operation.
|
||||
Publisher retries cannot re-parent the prepared merge onto a newer target. Any
|
||||
failure after the v4 sidecar is durable returns `RecoveryRequired`. Full recovery
|
||||
rolls confirmed effects forward only while the captured target authority still
|
||||
matches; otherwise it compensates the owned effects while preserving the target
|
||||
winner, or fails closed when foreign/interleaved table state makes compensation
|
||||
unverifiable. An Armed first-touch ref with no data HEAD movement is reclaimed
|
||||
without manufacturing rollback lineage. Armed recovery accepts only a
|
||||
contiguous prefix of the pre-minted data chain. Rebuildable `CreateIndex`
|
||||
transactions may follow only the complete chain and are rollback-discardable
|
||||
derived state; any other, unreadable, or non-contiguous transaction fails
|
||||
closed. A compensating Lance `Restore` is also recognized by its exact target so
|
||||
a crash after restore but before the manifest publish resumes without restoring
|
||||
again.
|
||||
|
||||
The handle-local coordinator swap and `merge_exclusive` mutex remain an
|
||||
implementation detail until target-context extraction lands; neither is treated
|
||||
as persistent authority. Native ref create/delete still lack conditional CAS, so
|
||||
first-touch destructive recovery retains the documented single-writer-process
|
||||
boundary. `sync_branch` continues to join the schema gate and cannot replace the
|
||||
temporary coordinator during a merge.
|
||||
|
||||
### Branch-delete orphaning exception
|
||||
|
||||
|
|
@ -334,16 +351,19 @@ are left at `Lance HEAD = manifest_pinned + 1`.
|
|||
post_commit_pin)` it intends to commit + the writer kind +
|
||||
actor_id.
|
||||
For a first-touch named-branch Mutation/Load table, Phase A is followed by
|
||||
target-ref creation and branch-local `stage_*`; the sidecar already carries
|
||||
its pre-minted transaction identity.
|
||||
target-ref creation and branch-local `stage_*`; the schema-v3 sidecar already
|
||||
carries its pre-minted transaction identity. Branch merge uses schema v4:
|
||||
it distinguishes multi-commit HEAD effects from ref-only forks, records each
|
||||
multi-commit effect's ordered exact transaction chain, and records the
|
||||
complete intended manifest delta, including pointer-only slots.
|
||||
2. **Phase B**: writer's per-table `commit_staged` loop runs.
|
||||
- **Phase-B confirmation:** a `BranchMerge` writer
|
||||
advances each table's HEAD by *several* commits (append → upsert →
|
||||
delete), so a bare "HEAD moved" is ambiguous — it could be a complete
|
||||
publish or one crashed mid-sequence. After the whole per-table loop
|
||||
finishes, the writer re-writes the sidecar stamping each pin's
|
||||
`confirmed_version` with the exact achieved version, then proceeds to
|
||||
Phase C. Schema-v3 Mutation/Load sidecars also confirm: each table must
|
||||
- **Phase-B confirmation:** a schema-v4 `BranchMerge` writer
|
||||
advances each table's HEAD by *several* exact commits (append → upsert →
|
||||
delete). Recovery proves a contiguous prefix of the pre-armed transaction
|
||||
chain rather than inferring ownership from numeric HEAD movement. After the
|
||||
whole per-table loop finishes, the writer atomically confirms each exact
|
||||
achieved version, the complete logical manifest delta, and first-touch ref
|
||||
identities, then proceeds to Phase C. Schema-v3 Mutation/Load sidecars also confirm: each table must
|
||||
match the staged Lance transaction's `(read_version, uuid)`, and the
|
||||
sidecar records the exact `SubTableUpdate` plus original lineage intent.
|
||||
This is the commit point of the recovery WAL: a crash *after* confirmation
|
||||
|
|
@ -371,9 +391,16 @@ recovery sweep in `crates/omnigraph/src/db/manifest/recovery.rs`:
|
|||
Lance HEAD to the manifest pin. Classify per the all-or-nothing
|
||||
decision tree (RolledPastExpected / NoMovement / UnexpectedAtP1 /
|
||||
UnexpectedMultistep / IncompletePhaseB / InvariantViolation). For a
|
||||
`BranchMerge` sidecar, a moved HEAD with no `confirmed_version` classifies
|
||||
as `IncompletePhaseB` (a partial multi-commit publish) and forces roll-back;
|
||||
with a `confirmed_version`, roll-forward targets exactly that version.
|
||||
legacy `BranchMerge` sidecar, a moved HEAD with no `confirmed_version`
|
||||
classifies as `IncompletePhaseB` (a partial multi-commit publish) and forces
|
||||
roll-back; with a `confirmed_version`, roll-forward targets exactly that
|
||||
version. Schema-v4 BranchMerge recovery additionally requires the captured
|
||||
target token, fixed original/rollback lineage ids, the exact ordered data
|
||||
transaction chains, exact confirmed physical effects, first-touch ref
|
||||
identities, and the complete confirmed manifest delta. A changed target token
|
||||
is rollback-only and can never re-parent the merge onto the winner. Recovery
|
||||
refuses a foreign or non-contiguous transaction instead of restoring through
|
||||
it, and recognizes an already-landed exact compensation restore on restart.
|
||||
Schema-v3 Mutation/Load additionally requires `EffectsConfirmed`, the exact
|
||||
Lance transaction identity at the confirmed version, the original immutable
|
||||
manifest delta, and a matching captured authority token. A changed token is
|
||||
|
|
@ -411,9 +438,10 @@ recovery sweep in `crates/omnigraph/src/db/manifest/recovery.rs`:
|
|||
- After a successful roll-forward or roll-back, an internal
|
||||
`_graph_commit_recoveries.lance` row records `recovery_kind`,
|
||||
`recovery_for_actor` (the original sidecar's actor), `operation_id`, and
|
||||
exact per-table outcomes. A v3 roll-forward publishes the interrupted
|
||||
writer's fixed lineage intent, including its original actor; rollback and
|
||||
legacy recovery commits use `actor_id = "omnigraph:recovery"`. Ordinary
|
||||
exact per-table outcomes. Schema-v3 Mutation/Load and schema-v4 BranchMerge
|
||||
roll-forward publish the interrupted writer's fixed lineage intent,
|
||||
including its original actor; rollback and legacy recovery commits use
|
||||
`actor_id = "omnigraph:recovery"`. Ordinary
|
||||
commit history is therefore not a complete recovery enumeration, and the
|
||||
CLI currently has no public query for the recovery-audit table.
|
||||
- Sidecar deleted as the final step.
|
||||
|
|
@ -469,7 +497,7 @@ guard says so explicitly ("a pending recovery sidecar requires
|
|||
rollback — reopen the graph read-write") instead of pointing at
|
||||
`omnigraph repair`, which refuses while a sidecar is pending.
|
||||
`cleanup` refuses pending sidecars at entry as well, before orphan reconciliation
|
||||
or version GC: v3 ownership and compensation recovery may need the retained
|
||||
or version GC: v3/v4 ownership and compensation recovery may need the retained
|
||||
Lance transaction/version history, so garbage collection cannot outrun the
|
||||
recovery barrier.
|
||||
Continuous in-process recovery for the rollback path is the goal of a
|
||||
|
|
@ -477,15 +505,16 @@ future background reconciler. `ensure_indices` does not heal at entry itself;
|
|||
it is an explicit maintenance/reconciliation call, separate from mutation,
|
||||
load, and schema apply, and its strict preconditions fail loudly on drift.
|
||||
|
||||
For enrolled mutation/load, the publisher rechecks the attempt's exact native
|
||||
branch identity and `graph_head` as well as the touched-table versions. A
|
||||
For enrolled mutation/load and branch merge, the publisher rechecks the
|
||||
attempt's exact native branch identity and `graph_head` as well as table
|
||||
expectations. A
|
||||
concurrent graph commit anywhere on the target branch therefore invalidates the
|
||||
prepared authority instead of silently reparenting it. Before effects, an
|
||||
insert-only mutation or Append/Merge load fully reprepares with a bounded retry; strict
|
||||
Update/Delete/Overwrite returns `ReadSetChanged`; after any effect, any later
|
||||
error returns `RecoveryRequired` and leaves the fixed v3 intent durable. Legacy
|
||||
writers still arbitrate only their explicit touched-table expectations until
|
||||
their adapters are enrolled.
|
||||
Update/Delete/Overwrite and branch merge return `ReadSetChanged`; after any
|
||||
effect, any later error returns `RecoveryRequired` and leaves the fixed v3/v4
|
||||
intent durable. Schema apply and optimize/index remain on their writer-specific
|
||||
arbitration until their adapters are enrolled.
|
||||
|
||||
**Sidecar I/O failure semantics** (all sidecar I/O goes through the
|
||||
backend-generic `StorageAdapter`; the contracts below are pinned by the
|
||||
|
|
|
|||
|
|
@ -694,14 +694,19 @@ Implementation proceeds in this order:
|
|||
> **Implementation note (2026-07-11):** mutation/load now use this coarse
|
||||
> token, schema-v3 exact-effect sidecars, fixed lineage/rollback outcome ids,
|
||||
> zero transparent Lance commit retries, and bounded full reprepare before
|
||||
> effects. Branch merge remains on its writer-specific multi-commit path, but
|
||||
> now holds the root-shared schema plus source/target branch gates from its
|
||||
> strict recovery barrier and authority capture through publication. It plans
|
||||
> with an accepted-contract catalog captured under that schema gate, then
|
||||
> acquires all catalog table gates for source and target, re-lists recovery,
|
||||
> and compares fresh manifest versions before Phase A. This closes
|
||||
> same-process delete/recreate ABA and legacy table-only-writer races while
|
||||
> its full exact-effect adapter remains future work. Schema apply,
|
||||
> effects. Branch merge now captures an immutable source commit/snapshot and
|
||||
> the target coarse token, computes its merge base from those captured ids,
|
||||
> and revalidates target authority plus source incarnation under the ordered
|
||||
> gates. Its schema-v4 recovery envelope distinguishes multi-commit HEAD
|
||||
> effects from first-touch ref-only forks, persists an ordered pre-minted
|
||||
> Lance transaction chain for every logical data effect plus fixed
|
||||
> merge/rollback lineage, and confirms the complete manifest delta (including
|
||||
> pointer-only slots) before publishing with `ExactGraphHead`. Those data
|
||||
> transactions commit with zero transparent conflict retries; recovery accepts
|
||||
> only their contiguous exact prefix (plus a derived `CreateIndex` tail after
|
||||
> the complete chain) and fails closed on foreign movement. A pre-effect target change
|
||||
> returns `ReadSetChanged`; any post-arm failure returns `RecoveryRequired`
|
||||
> and recovery never re-parents onto a target winner. Schema apply,
|
||||
> optimize/index, and MemWAL fold remain on their writer-specific paths until
|
||||
> their adapter slices land.
|
||||
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ Two primitives, two scopes:
|
|||
| **One `.gq` query** (any number of statements inside) | The query itself — handled by the publisher's atomic manifest commit | Yes — all statements land together or none of them do | The publisher never publishes; target unchanged |
|
||||
| **Many queries that must succeed together** | Branches: `branch_create` → run N queries on the branch → `branch_merge` | Yes — the merge is a single atomic publish | Drop the branch (`branch_delete`); main is unaffected |
|
||||
|
||||
Snapshot isolation is per-query — every read inside one query sees one consistent manifest version. Two concurrent queries on the same branch see independent snapshots. Mutation/load capture the branch head as coarse OCC authority, so a prepared plan is never silently reparented after another graph commit.
|
||||
Snapshot isolation is per-query — every read inside one query sees one consistent manifest version. Two concurrent queries on the same branch see independent snapshots. Mutation/load and branch merge capture the target branch head as coarse OCC authority, so a prepared plan is never silently reparented after another graph commit. Merge additionally pins the exact source commit it classified; a later source advance is not substituted into the prepared result.
|
||||
|
||||
## Comparison with `BEGIN` / `COMMIT`
|
||||
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@
|
|||
| `MANIFEST_DIR` | `__manifest` | manifest layout |
|
||||
| Commit graph dirs (retired) | `_graph_commits.lance` / `_graph_commit_actors.lance` | retired in Phase B; lineage lives in `__manifest` (`graph_commit` / `graph_head` rows) since RFC-013 Phase 7. A graph this binary creates has neither. |
|
||||
| Recovery audit dir | `_graph_commit_recoveries.lance` | internal exact record of completed crash-recovery actions; no public CLI query yet |
|
||||
| Exact recovery history-scan ceiling | `MAX_EFFECT_IDENTITY_SCAN_VERSIONS = 1024` | v3/v4 recovery fails closed as unverifiable rather than scanning an unbounded Lance transaction history; a v4 logical merge chain is at most three commits today, with only derived `CreateIndex` commits allowed after it |
|
||||
| Run branch prefix (legacy, removed) | `__run__` | pre-v0.4.0 Run state machine; no longer a reserved name. A graph still carrying `__run__*` branches is sub-v4 and refused on open (rebuild via export/import). |
|
||||
| Schema apply lock | `__schema_apply_lock__` | schema apply |
|
||||
| Manifest publisher retry budget | `PUBLISHER_RETRY_BUDGET = 5` | manifest publish |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue