mirror of
https://github.com/ModernRelay/omnigraph.git
synced 2026-06-18 02:24:27 +02:00
6a2dfa7325
* chore: correct stale global-lock comments
The global Arc<RwLock<Omnigraph>> that once serialized every server write was
removed — the server holds the engine as a lockless Arc<Omnigraph> and write
methods are &self, so the per-(table_key, branch) write queues are now the
actual write-serialization mechanism (in-process only).
Correct comments that still claimed the global lock is 'still in place' /
'today', or framed the queues as MR-686 scaffolding: write_queue.rs module doc,
exec/merge.rs, db/omnigraph/schema_apply.rs, db/manifest/recovery.rs, and the
bench_concurrent_http.rs example (which also wrongly stated mutate_as is
&mut self). workload.rs is left as-is — its 'previous global RwLock' wording is
accurate history.
* test: regression for self-healing a manifest-unreferenced fork
An interrupted first-write fork (create_branch succeeded, the manifest publish
did not) leaves a fully-formed Lance branch ref the manifest never references.
The branch stays a valid manifest branch, so cleanup's reconciler never
reclaims it, and today the next write to that table wedges with 'incomplete
prior delete; run cleanup'.
Forge that exact residue (a live 'feature' branch + a directly-created
'feature' ref on the Person table the manifest doesn't reference) and assert
the next load AND mutate self-heal. Deterministic and local — no S3 or timing,
since the forge IS the post-crash state. Adds a shared node_table_uri helper.
This commit is RED: it reproduces the bug and fails against the unfixed engine
with the predicted symptom. The fix follows in the next commit.
* fix: self-heal manifest-unreferenced branch forks
The first write to a table on a branch lazily forks it via Lance create_branch,
a durable two-phase op that advances Lance state BEFORE the atomic manifest
publish. If the writer dies or its request future is cancelled between the fork
and the publish, the branch ref is fully formed but the manifest never
references it. The next write re-enters the fork path, create_branch collides,
and the engine wedged with 'orphaned table state ... incomplete prior delete;
run cleanup' — which cleanup could not even fix, because the branch is still a
live manifest branch. This hit load, mutate, ingest, and the merge fork path
(one shared engine chokepoint), so a routine deploy restart or client
disconnect could wedge a branch.
Fix: treat the per-table fork ref as derived state of the manifest. fork_branch_
from_state returns a typed ForkOutcome instead of a human 'incomplete prior
delete' error; on RefAlreadyExists the db layer reclaims the manifest-
unreferenced fork (force_delete_branch + re-fork, exactly once) and proceeds.
A live committed fork is still routed to a retryable conflict before the fork
path, so concurrent first-writes stay correct.
Reclaim is only safe if no in-process writer can be mid-fork, so the write
entry points (load, mutate) acquire the per-(table, branch) write queues for
all touched tables up front — before the fork, held through the publish — when
forking a non-main branch. commit_all accepts these pre-held guards instead of
re-acquiring (the queue is non-re-entrant). The merge fork path already holds
the queue and self-heals through the shared wrapper. Cross-process in-flight
forks remain the documented one-winner-CAS gap.
Mechanical prep folded in: mutation IR lowering is hoisted so the touched-table
set is known before execution; commit_all gains the held_guards parameter.
Flips recreate_over_orphaned_fork_before_cleanup_is_actionable to assert
self-heal; fork_collision_with_live_concurrent_fork_is_retryable still holds.
Docs: writes.md cancelled-future note, invariants.md cross-process known gap.
* fix(cleanup): reconcile per-table manifest-unreferenced forks
reconcile_orphaned_branches keyed orphans on the branch NAME (absent from the
manifest), so it only reclaimed forks from a fully-deleted branch. A fork left
on a still-live branch by an interrupted first-write was never reclaimed — the
backstop the handoff expected cleanup to provide did not cover that case.
Broaden it to a per-table authority test: a Lance branch B on table T is an
orphan iff B is not a live manifest branch (delete-leftover) OR the manifest's
branch-B snapshot does not place T on B (interrupted first-write). Per-branch
snapshots are resolved once and cached across tables. Legitimately-forked
tables, main, and internal/system branches are never reclaimed; children are
dropped before parents to avoid Lance's referenced-parent RefConflict. The
commit-graph half stays whole-branch (per-table doesn't apply there).
This is the guaranteed-convergence backstop to the write-path self-heal: it
reclaims any fork the write path never revisits, and is what Lance's own
create_branch docstring asks embedders to provide for zombie/orphan refs.
* fix: reclaim self-validates against fresh manifest authority
The fork reclaim force-deletes a Lance branch ref, gated on the caller's proof
that the manifest does not place the table on the branch. But the first-write
path obtains that proof via snapshot_for_branch, which returns the coordinator's
CACHED snapshot when the handle is bound to the branch (an embedded handle on
the branch, or branch_merge's target swap). If that snapshot is stale and a
concurrent writer already published a legitimate fork, the reclaim would
force-delete it and re-fork from source, stranding the manifest at a version the
recreated ref no longer has.
Make the destructive primitive own its safety precondition: re-derive it from a
FRESH manifest read (fresh_snapshot_for_branch, which bypasses the cache)
immediately before force-deleting. If fresh authority shows the table is on the
branch, refuse with a retryable conflict instead of destroying a valid fork.
Correct for any caller regardless of snapshot staleness. Also stop branching on
Lance's exact RefConflict prose (loosened match; typed-variant is the durable
follow-up). Addresses PR review (Codex P1, Greptile P2).
* fix: cover delete-cascade edges in up-front fork-queue acquisition
A node delete cascades to every edge table touching that node (execute_delete_
node), forking those edge tables during execution. But touched_table_keys
derived the up-front fork-queue set from the IR ops alone (just node:Type), so a
branch delete that forks node + cascade edges held only the node queue —
commit_all then saw cascade-edge keys it had no guard for.
The touched set is a pure function of (IR ops + catalog), so compute the
COMPLETE set: op types plus, for delete-node ops, the cascade edges derived the
same way the executor derives them (from_type/to_type match). Pre-computed now
equals actual by construction.
Also promote commit_all's held-guard coverage check out of debug_assert into an
all-builds check that fails the write with a typed manifest_internal error: a
load-bearing serialization invariant must fail loudly+safely in release, not
silently proceed unguarded if a future execution path ever touches a table
outside the pre-computed set.
Adds branch_cascade_delete_forks_node_and_edges_under_held_queues, which drives
the cascade path on a branch (the gap the existing insert/load tests missed).
Addresses PR review (Cursor medium, Greptile P2).
* fix(cleanup): serialize fork reclaim against in-process live writers
The broadened per-table reconciler force_delete'd an orphan candidate on a LIVE
branch without holding the per-(table, branch) write queue. An in-process
first-write fork in its fork->publish window holds that queue and has not yet
advanced the manifest, so it looks exactly like an origin-2 orphan — concurrent
cleanup could delete the ref the writer still holds and is about to publish.
(The old branch-name-based reconciler did not have this race: a deleted branch
cannot have a live first-write.)
Bring the reconciler under the same invariant the write-path reclaim already
obeys: never force_delete a fork ref without holding the (table, branch) write
queue AND confirming, under it, from a fresh read, that the ref is still
manifest-unreferenced. Acquire one key at a time (no lock-order inversion vs
multi-table acquire_many writers); if the writer published meanwhile, the fresh
re-check sees the table on the branch and skips. Cross-process writers remain
the documented one-winner-CAS gap. Addresses PR review (Cursor high).
* fix: classify create_branch failure by ref existence, not by failure
fork_branch_from_state mapped ANY create_branch failure to RefAlreadyExists,
routing transient I/O / version / Lance-internal errors into the destructive
reclaim path and masking the real error as a retryable conflict.
Branch on the actual fact instead: on create_branch failure, check whether the
ref exists (list_branches). Only a genuinely pre-existing ref — a fully-formed
manifest-unreferenced fork — is a reclaim candidate; any other failure
propagates with fidelity. We deliberately do NOT force-delete on a not-found-ref
failure: it is indistinguishable from a transient error on a fresh create, and
force-deleting there is the overreach the fresh-authority guard already removed.
A phase-1-only Lance zombie (rarer; create_branch interrupted mid its two
internal phases) surfaces as the propagated error for manual reclaim.
Addresses PR review (Cursor medium).
* fix(cleanup): skip (not delete) on a transient re-check error for a live branch
The reconcile pre-delete re-check treated ANY fresh_snapshot error as 'still an
orphan' and proceeded to force_delete. A transient manifest read failure on a
LIVE branch could therefore destroy a fork the manifest still considers
legitimate — inconsistent with the write-path reclaim (aborts on the same error)
and the candidate scan (skips on snapshot failure).
Distinguish the two origins under the queue: a branch absent from the manifest
authority (origin 1) is a confirmed orphan and is deleted without a fresh read
(no live writer can hold a deleted branch's queue); a LIVE branch (origin 2)
gets the fresh re-check and, on a transient read error, is SKIPPED — never
destroyed on ambiguity — converging on a later cleanup. Same don't-destroy-on-
ambiguous-error principle as the create_branch failure classification.
Addresses PR review (Cursor medium).
* fix(cleanup): unify fork-ref reclaim on fresh authority under the queue
Consolidates the reconcile/reclaim hardening from PR review (the earlier per-site
commits were collapsed when reconciling with the main merge). Both destructive
fork-ref sites — the write-path reclaim and the cleanup reconciler — now share
one classifier, classify_fork_ref -> ForkRefStatus { Legitimate, Orphan,
Indeterminate }, evaluated from FRESH manifest authority under the held
(table, branch) write queue. A fork ref is destroyed ONLY on a confirmed Orphan;
a Legitimate (concurrent writer published a real fork) or Indeterminate
(transient read) status is never destroyed — the write path maps it to a
retryable conflict, cleanup maps it to skip. This closes, by construction:
- reclaim trusting a possibly-cached caller proof (Codex P1);
- reconcile racing an in-process live fork without the queue (Cursor);
- delete-on-transient-error in the re-check (Cursor/Greptile);
- origin-1 trusting a stale live_branches capture for a created-since branch
(Cursor/Greptile P1).
Having one classifier removes the duplication that let the two sites drift.
ForkOutcome is made pub to match the sealed trait method returning it. Verified
green on Lance 7.0.0 (full engine suite + 48/48 failpoints).
* test(cleanup): pin classify_fork_ref decision (Legitimate / Orphan / ghost)
Both fork-ref reclaim sites (write-path reclaim + cleanup reconciler) route
their destroy/skip decision through classify_fork_ref, but it had no direct
test — reverting the fresh-authority logic was not test-detectable. Add a
deterministic in-source unit test that forges each state and asserts the status:
a manifest-placed fork -> Legitimate (never destroyed); a ref the manifest does
not place on the branch -> Orphan; a ref for a branch absent from the manifest
-> Orphan (ghost reclaim preserved). This makes the core fresh-authority
decision behind every reclaim fix revert-detectable in one place.
(The Indeterminate arm — transient read on a live branch -> skip — needs an
injected read failure and is left to the failpoints suite; the cross-process
cleanup-vs-writer and cached-snapshot reclaim races are the documented
one-winner-CAS gap, not reachable same-process bugs, so they are not faked here.)
* test(cleanup): pin the Indeterminate (transient re-check) reclaim arm
Closes the last untested classify_fork_ref arm. Adds a 'classify.fresh_read'
failpoint (no-op without the failpoints feature) that simulates a transient
failure of the fresh-authority read, and a failpoints test driving it through
cleanup: a genuine origin-2 orphan on a LIVE branch whose fresh re-check fails
classifies as Indeterminate, so the reconciler SKIPS it (never destroys on an
inconclusive read) and reclaims it on the next run once the read succeeds.
This makes the don't-destroy-on-ambiguity rule revert-detectable end-to-end.
The only paths now left untested are the cross-process cleanup-vs-writer and
reclaim-vs-publish races — the documented one-winner-CAS gap (cleanup is
&mut self / CLI-only, so no reachable same-process race), not faked here.
* test(server): avoid stale schema apply route handle
* fix(cleanup): report indeterminate fork authority clearly
368 lines
21 KiB
Markdown
368 lines
21 KiB
Markdown
# Direct-Publish Write Path
|
|
|
|
> History: the Run state machine and `__run__<id>` staging branches were
|
|
> removed in MR-771 (shipped v0.4.0). Writes now go directly to the target
|
|
> table; this document specifies that direct-publish path.
|
|
|
|
`mutate_as` and `load` write **directly to the target table**
|
|
and call `ManifestBatchPublisher::publish` once at the end with
|
|
`expected_table_versions` (the per-table manifest versions captured before
|
|
the first write). Cross-table OCC is enforced inside the publisher; the
|
|
publisher's row-level CAS on `__manifest` is the single fence.
|
|
|
|
## What this means in practice
|
|
|
|
- No `RunRecord`, no `_graph_runs.lance`, no `_graph_run_actors.lance`.
|
|
- No `omnigraph run *` CLI subcommands and no `/runs/*` HTTP endpoints.
|
|
- No `__run__<id>` staging branches; `__run__*` is no longer a reserved
|
|
name. The branch-name guard was removed in MR-770, and any stale
|
|
`__run__*` branch on an upgraded graph is swept off `__manifest` by the
|
|
v2→v3 internal-schema migration on first read-write open. (The inert
|
|
`_graph_runs.lance` bytes remain until a `delete_prefix` primitive lands.)
|
|
- Cancelled mutation futures leave **no graph-visible state** — the manifest
|
|
is never advanced. They can leave two kinds of unreferenced residue, both
|
|
self-healing: orphaned Lance fragments (reclaimed by `omnigraph cleanup`),
|
|
and — on the *first* write to a table on a branch, which forks it before the
|
|
publish — a manifest-unreferenced branch ref. The next write to that table
|
|
reclaims the stale fork and re-forks (`reclaim_orphaned_fork_and_refork`),
|
|
and `cleanup`'s per-table reconciler is the guaranteed backstop; see the
|
|
fork-reclaim note in [invariants.md](invariants.md).
|
|
|
|
## Read-your-writes within a multi-statement mutation
|
|
|
|
A `.gq` query with multiple ops (e.g. `insert Person … insert Knows …`)
|
|
must observe earlier ops' writes when validating later ops (referential
|
|
integrity, edge cardinality). After MR-794 step 2+ this is implemented
|
|
via an in-memory `MutationStaging` accumulator in
|
|
[`crates/omnigraph/src/exec/staging.rs`](../../crates/omnigraph/src/exec/staging.rs),
|
|
shared by both `mutate_as` and the bulk loader:
|
|
|
|
- On the first touch of each table, the pre-write manifest version is
|
|
captured into `expected_versions[table_key]` (the publisher's CAS
|
|
fence at end-of-query).
|
|
- Each insert/update op pushes a `RecordBatch` into the per-table
|
|
pending accumulator. Lance HEAD does **not** advance during op
|
|
execution.
|
|
- Read sites (validation, predicate matching for `update`) consume
|
|
`TableStore::scan_with_pending`, which scans committed via Lance
|
|
and applies the same SQL filter to the pending batches via DataFusion
|
|
`MemTable`. Same-query writes are visible to subsequent reads.
|
|
- At end-of-query, `MutationStaging::finalize` issues exactly one
|
|
`stage_*` + `commit_staged` per touched table (concatenating
|
|
accumulated batches; merge-mode dedupes by `id`, last-write-wins),
|
|
and the publisher publishes the manifest atomically across all
|
|
touched sub-tables. Cross-table conflicts surface as
|
|
`ManifestConflictDetails::ExpectedVersionMismatch`.
|
|
- **Deletes still inline-commit.** Lance's `Dataset::delete` is not
|
|
exposed as a two-phase op in 6.0.1; deletes go through `delete_where`
|
|
immediately and record their post-write state in
|
|
`MutationStaging.inline_committed`. The parse-time D₂ rule (below)
|
|
prevents inserts/updates from coexisting with deletes in one query,
|
|
so the inline path is safe for delete-only mutations.
|
|
|
|
This upholds the manifest-atomic mutation and read-your-writes invariants
|
|
tracked in [docs/dev/invariants.md](invariants.md).
|
|
|
|
### D₂ — parse-time mixed-mode rejection
|
|
|
|
A single mutation query is either insert/update-only or delete-only.
|
|
Mixed → rejected at parse time with a clear error directing the user to
|
|
split the query. Reason: mixing creates ordering hazards
|
|
(insert→delete on the same row would silently no-op because the staged
|
|
insert isn't visible to delete; cascading deletes of just-inserted
|
|
edges break referential integrity). Until Lance exposes a two-phase
|
|
delete API, the parse-time rejection keeps both paths atomic and
|
|
correct. Tracked: MR-793, plus a Lance-upstream ticket.
|
|
|
|
### MR-793 status (storage trait two-phase invariant) — partial
|
|
|
|
MR-793 hoists the staged-write pattern into a `TableStorage` trait
|
|
surface with sealed-trait enforcement and opaque `SnapshotHandle` /
|
|
`StagedHandle` types — see `crates/omnigraph/src/storage_layer.rs`.
|
|
The trait is the canonical surface for new engine code; existing call
|
|
sites still use the inherent `TableStore` methods (mechanical migration
|
|
deferred to a follow-up cycle — tracked).
|
|
|
|
Three writers have been migrated onto staged primitives:
|
|
|
|
* **`ensure_indices`** (`db/omnigraph/table_ops.rs::build_indices_on_dataset_for_catalog`)
|
|
— scalar indices (BTree, Inverted) use `stage_create_*_index` +
|
|
`commit_staged`. Which index a `@index`/`@key` property gets is dispatched by
|
|
type via `node_prop_index_kind` (enum + orderable scalar → BTree, free-text
|
|
String → Inverted/FTS, Vector → vector). Vector indices stay inline (residual
|
|
— Lance `build_index_metadata_from_segments` is `pub(crate)` in 6.0.1;
|
|
companion ticket to lance-format/lance#6658 needed). This build is
|
|
existence-gated (it creates a *missing* index over current fragments); folding
|
|
fragments appended afterward into an *existing* index is `optimize`'s
|
|
`optimize_indices` pass — an inline-commit residual, not a staged write (Lance
|
|
exposes no uncommitted index-optimize), covered by the optimize recovery
|
|
sidecar (see [maintenance.md](../user/operations/maintenance.md)).
|
|
* **`branch_merge::publish_rewritten_merge_table`**
|
|
(`exec/merge.rs`) — merge_insert now uses `stage_merge_insert` +
|
|
`commit_staged`. Deletes stay inline (Lance #6658 residual).
|
|
* **`schema_apply` rewritten_tables** (`db/omnigraph/schema_apply.rs`)
|
|
— rewrites use `stage_overwrite` + `commit_staged`, including empty-table
|
|
rewrites via a zero-fragment Lance `Operation::Overwrite`.
|
|
|
|
A defense-in-depth integration test (`tests/forbidden_apis.rs`) walks
|
|
engine source and fails if non-allow-listed code calls Lance's
|
|
inline-commit APIs directly. The trait surface itself is the primary
|
|
enforcement (sealed + only-callable-via-trait once call sites land);
|
|
the grep test catches type-system bypass attempts.
|
|
|
|
The "finalize → publisher residual" described below applies equally to
|
|
the migrated writers — Lance has no multi-dataset atomic commit
|
|
primitive, so the per-table commit_staged → manifest publish gap is
|
|
the same drift class. Closing it requires either upstream Lance
|
|
multi-dataset commit OR the omnigraph-side recovery-on-open reconciler
|
|
described in `.context/mr-793-design.md` §15 (deferred to MR-795).
|
|
|
|
### Inline-commit residuals live on `InlineCommitResidual`, not `db.storage()` (MR-793 acceptance §1, by construction)
|
|
|
|
MR-793's acceptance criterion §1 ("`TableStore` (or successor) public API has no method that performs a manifest commit as a side effect of writing") holds **by construction** after MR-854. `db.storage()` (`&dyn TableStorage`) exposes only staged primitives + reads; the inline-commit writes Lance cannot yet stage live on a separate `InlineCommitResidual` trait reached via `Omnigraph::storage_inline_residual()`. A new engine writer cannot couple a write with a Lance HEAD advance through the default surface — it would have to name the residual accessor explicitly. The dead legacy methods (trait `append_batch` / `merge_insert_batches`, inherent `merge_insert_batch{,es}`, `create_{btree,inverted}_index`) were removed; appends/merges and scalar index builds all use the `stage_*` primitives.
|
|
|
|
Two methods remain on `InlineCommitResidual`, each named honestly at its call site:
|
|
|
|
| Residual method | Inline-commit reason | Closes when |
|
|
|---|---|---|
|
|
| `delete_where` | `DeleteBuilder::execute_uncommitted` is not in Lance v6.0.1 (closed upstream as [#6658](https://github.com/lance-format/lance/issues/6658) but first ships in `v7.0.0-beta.10`); see [docs/dev/lance.md](lance.md) | MR-A: Lance v7.x bump migrates `delete_where` to staged, retires the parse-time D₂ mutation rule, and extends recovery sidecar coverage |
|
|
| `create_vector_index` | Vector indices take Lance's "segment commit path"; `build_index_metadata_from_segments` is `pub(crate)` (Lance [#6666](https://github.com/lance-format/lance/issues/6666) still open) | Lance #6666 lands and `stage_create_vector_index` joins the staged surface |
|
|
|
|
The `tests/forbidden_apis.rs` guard still catches direct `lance::*` inline-commit misuse outside the storage layer; the trait split makes the staged-only default a type-system guarantee on top of it.
|
|
|
|
### `LoadMode::Overwrite` uses staged Lance `Overwrite`
|
|
|
|
The bulk loader's Append, Merge, and Overwrite modes all use the
|
|
staged-write path described above. `LoadMode::Overwrite` accumulates
|
|
replacement batches in memory, validates node/edge constraints, referential
|
|
integrity, and edge cardinality before any Lance HEAD movement, stages
|
|
each touched table with Lance `Operation::Overwrite`, then runs
|
|
`commit_staged` under the normal `SidecarKind::Load` recovery sidecar
|
|
before publishing `__manifest`. `OMNIGRAPH_LOAD_CONCURRENCY` applies to the
|
|
fragment-writing stage only; the commit and manifest publish still run
|
|
under the per-table write queues. Empty-table overwrite is represented as
|
|
a valid zero-fragment Lance `Overwrite` transaction, not as
|
|
truncate-then-append.
|
|
|
|
### Open-time recovery sweep
|
|
|
|
The staged-write rewire eliminates one drift class **by construction at
|
|
the writer layer**: an op that fails before pushing to the in-memory
|
|
accumulator (validation errors, missing endpoints, parse-time D₂
|
|
rejection) leaves Lance HEAD untouched on every staged table. This is
|
|
the case the `partial_failure_leaves_target_queryable_and_unblocks_next_mutation`
|
|
test pins.
|
|
|
|
A second, narrower drift class — the **finalize → publisher window** —
|
|
is closed across one open cycle by the open-time recovery sweep:
|
|
|
|
`MutationStaging::finalize` runs `stage_*` + `commit_staged` per touched
|
|
table sequentially, then the publisher commits the manifest. Lance has
|
|
no multi-dataset atomic commit, so the per-table `commit_staged` calls
|
|
are independent operations: if commit_staged on table N+1 fails *after*
|
|
commit_staged on tables 1..N succeeded, or if the publisher's CAS
|
|
pre-check rejects *after* every commit_staged succeeded, tables 1..N
|
|
are left at `Lance HEAD = manifest_pinned + 1`.
|
|
|
|
**Recovery protocol** (lifecycle of every staged-write writer —
|
|
`MutationStaging::finalize`, `schema_apply::apply_schema_with_lock`,
|
|
`branch_merge_on_current_target`, `ensure_indices_for_branch`,
|
|
`optimize_all_tables`):
|
|
|
|
1. **Phase A**: writer writes a sidecar JSON to
|
|
`__recovery/{ulid}.json` BEFORE its first HEAD-advancing commit
|
|
(`commit_staged`, or `compact_files` for `optimize_all_tables`,
|
|
which advances the Lance HEAD via a reserve-fragments + rewrite
|
|
commit rather than a staged write). The
|
|
sidecar names every `(table_key, table_path, expected_version,
|
|
post_commit_pin)` it intends to commit + the writer kind +
|
|
actor_id.
|
|
2. **Phase B**: writer's per-table `commit_staged` loop runs.
|
|
3. **Phase C**: publisher commits the manifest.
|
|
4. **Phase D**: writer deletes the sidecar.
|
|
|
|
> **Phase letter convention.** Throughout the recovery code, log
|
|
> messages, failpoint names (e.g. `branch_merge.post_phase_b_pre_manifest_commit`),
|
|
> and the per-writer integration tests, "Phase A/B/C/D" refers
|
|
> exclusively to the four-step lifecycle above. The per-table
|
|
> staged-write contract (`stage_*` then `commit_staged`, two steps)
|
|
> is referred to by those API verbs — never by phase letters — so a
|
|
> reader of `recovery.rs`, `failpoints.rs`, or this document only
|
|
> encounters phase letters in the per-writer context.
|
|
|
|
A failure between Phase A and Phase D leaves the sidecar on disk. The
|
|
next `Omnigraph::open` (gated on `OpenMode::ReadWrite`) runs the
|
|
recovery sweep in `crates/omnigraph/src/db/manifest/recovery.rs`:
|
|
|
|
- For each sidecar in `__recovery/`, compare every named table's
|
|
Lance HEAD to the manifest pin. Classify per the all-or-nothing
|
|
decision tree (RolledPastExpected / NoMovement / UnexpectedAtP1 /
|
|
UnexpectedMultistep / InvariantViolation).
|
|
- If any table is `InvariantViolation` (Lance HEAD < manifest pinned —
|
|
should be impossible), **abort** with a loud error and leave the
|
|
sidecar on disk for operator review.
|
|
- Otherwise, if every table is `RolledPastExpected`, **roll forward**:
|
|
a single `ManifestBatchPublisher::publish` call extends every pin
|
|
atomically. `SchemaApply` sidecars are eligible only when schema-state
|
|
recovery promoted the matching staging files in the same recovery pass;
|
|
otherwise full open-time recovery rolls them back and refresh-time
|
|
recovery leaves them for the next read-write open.
|
|
- Otherwise **roll back**: per-table `Dataset::restore` to the
|
|
manifest-pinned table version, then a single `ManifestBatchPublisher::publish`
|
|
of the restored HEAD — symmetric with roll-forward, so `manifest == HEAD`
|
|
after recovery (no residual drift). This convergence is what lets a
|
|
failed-then-retried schema apply succeed instead of failing one version higher
|
|
each iteration. The audit row's `to_version` records the logical
|
|
rolled-back-to version (`manifest_pinned`); the manifest is published at the
|
|
restore commit (`manifest_pinned + 1`, same content).
|
|
- After a successful roll-forward or roll-back, an audit row is
|
|
recorded — `_graph_commits.lance` carries
|
|
a commit tagged `actor_id = "omnigraph:recovery"`, and a sibling
|
|
`_graph_commit_recoveries.lance` row carries `recovery_kind`,
|
|
`recovery_for_actor` (the original sidecar's actor), `operation_id`,
|
|
per-table outcomes. Operators run `omnigraph commit list --filter
|
|
actor=omnigraph:recovery` to find recoveries.
|
|
- Sidecar deleted as the final step.
|
|
|
|
Triggers for the residual: transient Lance write errors during finalize
|
|
(object-store retry budget exhaustion, disk full); persistent publisher
|
|
contention exceeding `PUBLISHER_RETRY_BUDGET = 5` retries.
|
|
|
|
**Long-running servers**: the write entry points (`load_as`,
|
|
`mutate_as`, `apply_schema_as`, `branch_merge_as`) and
|
|
`Omnigraph::refresh` run roll-forward-only recovery in-process
|
|
(`recovery::heal_pending_sidecars_roll_forward`) — the common
|
|
Phase B → Phase C residual closes on the next write, without a
|
|
restart and without an explicit refresh. The heal lists `__recovery/`
|
|
(one `list_dir`; empty in the steady state) and, per sidecar, acquires
|
|
the same per-`(table_key, table_branch)` write queues every sidecar
|
|
writer holds from before `write_sidecar` until after `delete_sidecar` —
|
|
so it serializes against a live writer instead of rolling its
|
|
in-flight sidecar forward from under it (a sidecar whose queues can be
|
|
acquired belongs to a writer that finished or died; an existence
|
|
re-check after the wait skips the finished case). Lock order is
|
|
queues → coordinator, matching every writer's commit→publish path.
|
|
Pinned by the four
|
|
`tests/failpoints.rs::*_after_finalize_publisher_failure_heals_without_reopen`
|
|
tests (load, mutation, schema apply, branch merge). The maintenance
|
|
entries need the heal for more than liveness: without it, a schema
|
|
apply re-plans rewrites from the manifest pin and orphans the drifted
|
|
Phase-B commit (dropping its rows), and a branch merge publishes the
|
|
drift as an unattributed side effect — both while the stale sidecar
|
|
lingers to misclassify later.
|
|
Sidecars that would require a `Dataset::restore` (mixed / unexpected
|
|
state) are deferred to the next `OpenMode::ReadWrite` open: restore is
|
|
unsafe under concurrency because Lance's `check_restore_txn` accepts
|
|
the restore against in-flight Append/Update/Delete commits and
|
|
silently orphans them (pinned by
|
|
`tests/staged_writes.rs::lance_restore_loses_to_concurrent_append_via_orphaning`).
|
|
When such a deferred sidecar blocks a write, the commit-time drift
|
|
guard says so explicitly ("a pending recovery sidecar requires
|
|
rollback — reopen the graph read-write") instead of pointing at
|
|
`omnigraph repair`, which refuses while a sidecar is pending.
|
|
Continuous in-process recovery for the rollback path is the goal of a
|
|
future background reconciler. `ensure_indices` does not heal at entry
|
|
itself — it runs inside the load / schema-apply flows after their
|
|
entry heal, and its strict preconditions still fail loudly on drift
|
|
when invoked directly.
|
|
|
|
The publisher-CAS contract is unchanged: a *concurrent writer* that
|
|
advances any of our touched tables between snapshot capture and
|
|
publisher commit produces exactly one winner. The residual above is
|
|
about *our* abandoned commits in the failure path, not about
|
|
concurrency races.
|
|
|
|
**Sidecar I/O failure semantics** (all sidecar I/O goes through the
|
|
backend-generic `StorageAdapter`; the contracts below are pinned by the
|
|
storage-fault failpoints `recovery.sidecar_{write,delete,list}` /
|
|
`recovery.record_audit` and their tests in `tests/failpoints.rs` and
|
|
`tests/recovery.rs`):
|
|
|
|
- **Phase A put fails** (S3 PutObject / fs write): the writer aborts
|
|
before its first HEAD-advancing commit — no sidecar, no drift,
|
|
nothing to recover; a transient fault never wedges later writes.
|
|
- **Phase D delete fails** (S3 DeleteObject): swallowed with a warning —
|
|
the write already published, so failing the caller would report an
|
|
error for a durable write. The stale sidecar is consumed by the next
|
|
write's entry heal (or the next open) via the stale-sidecar
|
|
audit-recovery path, recorded as `RolledForward`.
|
|
- **`__recovery/` list fails** (S3 ListObjectsV2): loud at every
|
|
consumer — the write-entry heal fails the write, the open-time sweep
|
|
fails the open. Silently skipping recovery would be consumer
|
|
tolerance of drift.
|
|
- **Corrupt / unparseable sidecar**: refused loudly by heal and open
|
|
alike; the file stays on disk for operator inspection (read-only
|
|
opens still work — the sweep is skipped there).
|
|
- **Audit append fails after a roll-forward publish**: that recovery
|
|
attempt errors and keeps the sidecar; re-entry sees the
|
|
already-published manifest, records exactly one `RolledForward`
|
|
audit row, and deletes the sidecar (the retry tolerance documented
|
|
on `record_audit`).
|
|
|
|
Backend notes (the adapter is one implementation over `object_store`
|
|
for every backend): local writes stage through `name#<digits>` temp
|
|
files that the backend filters from listings and refuses to address —
|
|
crash residue of that shape is invisible to the sweep, harmless, and
|
|
reclaimed by `delete_prefix`/manual cleanup. Storage errors are
|
|
backend-wrapped text without a typed NotFound discriminant — callers
|
|
that need missing-vs-error (the cluster store) probe `exists()` first.
|
|
`exists()` itself is object-store semantics everywhere: only objects
|
|
(or non-empty prefixes) exist, and a permission failure is a loud
|
|
error, not a silent `false`.
|
|
|
|
## Conflict shape
|
|
|
|
Concurrent writers to the same `(table, branch)` produce exactly one
|
|
success and one failure. The losing writer's error is
|
|
`OmniError::Manifest` with kind `Conflict` and details
|
|
`ManifestConflictDetails::ExpectedVersionMismatch { table_key, expected,
|
|
actual }`. The HTTP server maps this to **409 Conflict** with body
|
|
`{"error": "...", "code": "conflict", "manifest_conflict": { "table_key":
|
|
"...", "expected": N, "actual": M }}` — see [docs/user/server.md](../user/operations/server.md).
|
|
|
|
## Audit
|
|
|
|
`actor_id` lands in `_graph_commits.lance` via `record_graph_commit` (no
|
|
intermediate run record). Audit history is queried via `omnigraph commit
|
|
list`.
|
|
|
|
## Migration code
|
|
|
|
`db/manifest/migrations.rs` carries the v2→v3 internal-schema step (MR-770):
|
|
a one-time sweep that deletes legacy `__run__*` staging branches off
|
|
`__manifest`. It runs in `Omnigraph::open(ReadWrite)` (via
|
|
`manifest::migrate_on_open`, before the coordinator reads branch state) and
|
|
again on the publisher's write path; both are idempotent once the stamp is at
|
|
v3. Deleting the inert `_graph_runs.lance` / `_graph_run_actors.lance` dataset
|
|
*bytes* is still deferred — it needs a `StorageAdapter::delete_prefix`
|
|
primitive — but those bytes are invisible to graph-level state.
|
|
|
|
## Mid-query partial failure: closed by MR-794
|
|
|
|
The pre-MR-794 design had a known limitation: a multi-statement `.gq`
|
|
mutation where op-N inline-committed a Lance fragment and op-N+1 then
|
|
failed left the touched table at `Lance HEAD = manifest_version + 1`,
|
|
blocking the next mutation with `ExpectedVersionMismatch`.
|
|
|
|
MR-794 (step 1 + step 2+) closed this for inserts/updates **by
|
|
construction at the writer layer**: insert and update batches accumulate
|
|
in memory; no Lance HEAD advance happens during op execution; one
|
|
`stage_*` + `commit_staged` per touched table runs at end-of-query, and
|
|
only after every op succeeded. A failed op leaves Lance HEAD untouched
|
|
on the staged tables, so the next mutation proceeds normally with no
|
|
drift to reconcile.
|
|
|
|
The cancellation case (future drop mid-mutation) inherits the same
|
|
guarantee — the in-memory accumulator evaporates with the dropped task
|
|
and no Lance write was ever issued.
|
|
|
|
For delete-touching mutations the legacy inline-commit shape is
|
|
preserved (Lance has no public two-phase delete in 6.0.1) — the same
|
|
narrow window remains. The parse-time D₂ rule prevents inserts/updates
|
|
from coexisting with deletes in one query, so a pure-delete failure
|
|
cannot drift any staged-table state. If a delete-only multi-table
|
|
mutation fails mid-cascade, the same workaround as before applies
|
|
(retry; rely on `omnigraph cleanup` once a later successful commit
|
|
moves HEAD past the orphan version). Closing this requires Lance to
|
|
expose `DeleteJob::execute_uncommitted`; tracked in MR-793 and a
|
|
Lance-upstream ticket.
|