* docs(rfc-013): bank the #295 spec-review comments as step-5 constraints (§5.1)
3b shipped a minimal WriteTxn{branch,base} and deferred the full §4.1 opener
unification (pinned-base opener, shared Session, write-local handle cache,
strict-op conflict-timing move) to step 5. The greptile comments on the #295
spec were moot for #298 (none of those constructs were built) but are
load-bearing for step 5: (1) the handle cache must be Send+Sync (Mutex, not
RefCell); (2) the strict-op timing move needs an explicit retry contract — txn
discarded after any commit, retry re-opens a fresh base — which is the SAME
contract as the stale-view false-fail (§1d.2); (3) the opener-equivalence test
must advance HEAD externally then assert pinned-base, not the trivial HEAD==base.
* feat(engine): fold graph lineage into the __manifest publish CAS (RFC-013 Phase 7)
Graph lineage no longer lives in a second write to _graph_commits.lance. Each
commit's graph_commit + graph_head:<branch> rows now ride the SAME __manifest
merge-insert as the table-version rows (one atomic version), and CommitGraph reads
its cache from the manifest projection (read_graph_lineage). _graph_commits.lance is
no longer written commit rows (it remains only as a Lance branch-ref carrier).
Mechanism: a LineageIntent { graph_commit_id (ULID, minted once), branch, actor,
merged_parent, created_at } threads through ManifestBatchPublisher::publish. Inside
the publisher retry loop the parent is resolved per attempt from the just-loaded
branch-scoped manifest (the should_replace_head winner over the visible graph_commit
rows — branch-correct by Lance branch isolation; the graph_head row is written for
forward-compat + the §7.1 contention point but is not the parent source, so a
freshly-forked branch resolves the right fork-point parent). A CAS-conflict retry
re-reads the advanced head → correct new parent; the commit_id is stable across
retries.
Closes two known gaps BY CONSTRUCTION (one write, no second step to fail/ race):
- manifest→commit-graph atomicity (no crash window between manifest + lineage),
- commit-graph parent under concurrency (no refresh→append TOCTOU; the per-write
commit_graph.refresh() is gone).
Recovery, branch-merge, and genesis route their lineage through the same CAS
(merge: one commit_merge_with_actor; recovery: publish_recovery_commit folds the
recovery commit, actor=omnigraph:recovery; genesis rides the init __manifest write).
The dead _graph_commits write helpers (append_commit/_merge/_actor) are
#[allow(dead_code)] (the actor sidecar table is still enumerated by optimize).
Verified (sequential): build clean; the new lineage_projection gate (manifest-only —
_graph_commits/_actors have 0 rows; full lineage reconstructs via the projection);
branching/merge_truth_table (exhaustive, branch-aware)/composite_flow/point_in_time/
changes/consistency/recovery; failpoints (59, incl. recovery lifecycle + the
now-closed atomicity gap); full --workspace. Cost tests REVERT to their pre-fold
values (writes +1, write_cost ceiling 80) — the proof of true single-CAS (no extra
write). invariants.md marks both gaps CLOSED.
PENDING (next stages, this PR): the §7.1 concurrent graph_head one-winner gate (stage
5 — two concurrent same-branch commits, exactly one wins); the stamp bump v4 +
migrate_v3_to_v4 backfill + read-only refuse for EXISTING graphs (stage 4); full
doc-sync of storage.md/architecture.md/writes.md.
* feat(engine): migrate existing v3 graphs to manifest lineage (RFC-013 Phase 7 stage 4)
The Phase-7 fold made CommitGraph read lineage from the __manifest projection, so a
pre-Phase-7 (internal-schema v3) graph — lineage in _graph_commits.lance, none in
__manifest — would read an empty commit DAG. Stage 4 makes existing graphs upgrade
seamlessly and not break reads.
- Stamp 3 -> 4 + migrate_v3_to_v4: bumps INTERNAL_MANIFEST_SCHEMA_VERSION and adds the
3 => migrate_v3_to_v4 arm. The migration reads this branch's _graph_commits/_actors,
emits one graph_commit row per commit + exactly one graph_head:<branch> for the head
(should_replace_head winner, deterministic id-sort — no hash-map-order in migration
output), merge-inserts into __manifest, then set_stamp(4) LAST. Idempotency guard
first (read_graph_lineage non-empty -> just stamp); crash before set_stamp re-enters
at v3 and the guard completes it. Does NOT touch the unenforced-PK metadata. Runs per
branch: migrate_on_open backfills main; load_publish_state backfills each branch on
its first write (root_uri/branch threaded through migrate_internal_schema).
- v3-read fallback: CommitGraph version-gates the lineage source — stamp < 4 reads the
(re-activated) _graph_commits.lance; >= 4 uses the manifest projection. So a READ-ONLY
open of an un-migrated graph reads correct history with no write. Correctness catch:
the legacy _graph_commit_actors.lance was never branched, so the fallback reads it
FLAT (no branch checkout) while checking out the branch only on the commits dataset.
- Read-only stamp-refuse: a ReadOnly open of a FUTURE-stamped graph now refuses with the
same upgrade error (future-proofing the next format bump; the write path already
refused via migrate_internal_schema).
- Docs: storage/architecture/writes/invariants/constants updated to manifest-stored
lineage; release note docs/releases/v0.8.0.md (format v4, old writers clean-break,
data preserved, upgrade writers first).
6 new tests (v3 backfill, idempotent, v3 read-only fallback, future-stamp refuse in both
modes, crash-before-stamp completes, legacy branch+flat-actor read). Full engine suite +
failpoints (59) + cargo test --workspace --locked green; check-agents-md passes.
* test(engine): graph_head concurrency gate — disjoint same-branch writers form a linear commit DAG (RFC-013 Phase 7)
Two (or N) writers committing disjoint tables on one branch still share the
mutable `graph_head:<branch>` manifest row, so the only row-level CAS
contention is that row. The contract — exactly one writer wins each CAS round;
the loser retries inside the publisher, re-resolves its parent off the
freshly-advanced head, and re-commits, so every writer lands and the
graph_commit DAG stays a single LINEAR chain (no fork) — had no acceptance
test. This adds it.
- concurrent_disjoint_writes_share_head_and_form_linear_chain: two disjoint
writers + distinct LineageIntent, tokio::join!; both commit; the on-disk DAG
is genesis -> c -> c' (asserted linear: exactly one genesis, no two commits
share a parent, the head is the unique non-parent).
- n_concurrent_disjoint_writers_converge_to_one_linear_chain: N=8 disjoint
writers each with an app-level retry loop (the publisher's internal budget
can be exhausted under contention); all converge to one linear chain of 8.
- concurrent_disjoint_writes_form_linear_chain_on_s3: the same race on a real
object store (true conditional-put CAS), bucket-gated.
Cites both tests from the §7.1 contention note in invariants.md.
Test-only; no production change.
* perf(engine): fold the lineage parent scan into the publish path's single __manifest scan (RFC-013 P2)
Each lineage publish scanned `__manifest` twice: `load_publish_state` read
table state via one scan, then `resolve_lineage_rows` did a second full
`read_graph_lineage` scan only to find the parent commit. Fold the
`graph_commit` extraction into the existing scan.
- `read_manifest_scan` gains a `collect_lineage` flag. The publish path
(`read_publish_scan`) collects the `graph_commit` rows in the same pass; the
table-state hot path leaves them in the forward-compat skip arm, so it never
pays the O(commits) lineage JSON decode (it also skips reading the
`object_id` column entirely). One shared `decode_graph_commit_row` serves
both the folded path and the standalone `read_graph_lineage`, so the two
cannot drift.
- `resolve_lineage_rows` is now sync and takes the already-parsed rows; the
per-attempt re-read is preserved because `load_publish_state` runs once per
CAS attempt, so a retry still re-parents off the advanced head.
- `load_publish_state` returns a named `LoadedPublishState` instead of a
four-tuple; the thin `read_registered_table_locations` /
`read_tombstone_versions` accessors fold away. `read_manifest_entries` becomes
`#[cfg(test)]`: the fold removes its last production caller, leaving only the
test-only namespace module (`db/manifest.rs`: `#[cfg(test)] mod namespace`),
so gating it keeps it from becoming dead code in non-test builds.
Measured at depth ~5: per-write `__manifest` reads drop 44 -> 26 (total reads
54 -> 36). write_cost.rs gains a `manifest_reads <= 34` sub-ceiling that trips
if a publish-path scan is re-added, and its calibration comment is corrected.
* test(engine): red — transient legacy-open failure silently completes the v3→v4 migration
A pre-Phase-7 (internal schema v3) graph keeps its graph lineage in
`_graph_commits.lance`; the v3→v4 internal-schema migration backfills it into
`__manifest` and stamps v4. `read_legacy_commit_cache` currently maps EVERY
`Dataset::open` error to "no legacy data" (`Err(_) => empty`), so a transient or
corrupt open during the one-time migration backfills nothing and still stamps
v4 — orphaning the real lineage permanently (the migration runs once; the v3
fallback is then disabled).
Add a `migration.v3_to_v4.legacy_open` failpoint that injects a non-not-found
Lance error at the legacy open, and a fault-injection regression test in the
`failpoints` binary. Against the current swallow the migration completes anyway,
so the test fails on its "migration must abort" assertion — the predicted
symptom. The fix follows in the next commit.
Test support reachable from the `failpoints` integration binary (it compiles the
crate without `cfg(test)`): the v3-fixture helpers and a stamp/row-count reader
are gated `cfg(any(test, feature = "failpoints"))`, still excluded from release
builds. Failpoint tests stay in the integration binary because the fail registry
is process-global.
* fix(engine): propagate non-not-found legacy-open errors in the v3→v4 migration
`read_legacy_commit_cache` mapped EVERY `Dataset::open` error to an empty cache
(`Err(_) => empty`) on both the legacy commits dataset and its actor sidecar. The
v3→v4 internal-schema migration reads this once before stamping internal-schema
v4; a transient or corrupt open therefore backfilled nothing and stamped v4
anyway, orphaning the graph's real lineage permanently (the migration runs once,
and the stamp-gated v3 fallback is disabled at v4). This is the "no silent
failures" deny-list violation, and realistic on object storage.
Both opens now match the not-found variants — Lance maps an object-store NotFound
to `DatasetNotFound` — as the benign "no legacy data" / "no authors" signal, and
propagate anything else as a loud error. The two arms share the variant contract
but carry different rationale (commits-absent is the legitimate empty signal;
actor-sidecar-absent is benign, but a corrupt actor open silently wiping
authorship before stamping v4 is the same loss hole), commented at each site.
Pinned by the `lance_surface_guards.rs::dataset_open_missing_returns_not_found_variant`
guard (turns red if a Lance bump changes the absence variant) and greens the
fault-injection regression test from the previous commit.
* test(engine): cover the per-branch v3→v4 migration against a real Lance branch
`seed_legacy_v3_lineage` writes every commit (including the "feature"-tagged one)
to MAIN's `_graph_commits.lance` with `manifest_branch` as a mere field, so the
production per-branch migration path — `read_legacy_commit_cache` checking out a
real Lance branch, and a branch-scoped `__manifest` — was never exercised.
Add `seed_legacy_v3_lineage_with_branch`, which forks a real `feature` Lance
branch on BOTH `_graph_commits.lance` and `__manifest` (the branch inherits
main's stripped v3 state), and a test that migrates the BRANCH and asserts the
branch's lineage lands in the BRANCH's `__manifest` (genesis + A + branch commit,
`graph_head:feature` → branch commit, parents + actors intact) with main's
`__manifest` untouched.
This empirically resolves the open question behind the merge robustness work: the
fast-path `read_graph_lineage(dataset)` has no `manifest_branch` filter, but
`__manifest` is Lance-branched per graph-branch, so a branch reads only its own
lineage — the test confirms migrating one branch does not leak into another. No
branch filter is needed.
* refactor(engine): type the lineage-backfill merge conflict via the publisher classifier
`state::merge_lineage_rows` (the v3→v4 lineage backfill's standalone `__manifest`
merge-insert) stringified its `execute_reader` error, discarding the Lance
variant. Route it through the publisher's `map_lance_publish_error` (now
`pub(crate)`) so a concurrent first-open's row-level CAS loss surfaces as the
SAME typed `OmniError::Manifest{ details: RowLevelCasContention }` the publisher's
own retry consumes — one vocabulary, no raw-Lance matching in the migration.
Deliberately NOT unified with `optimize::is_retryable_lance_conflict`: that
classifier also matches `CommitConflict`/`RetryableCommitConflict` from the
compaction commit path, which a row-level merge-insert never emits. Cross-linked
with a comment at both sites.
Behavior-preserving: the only path that changes is the error TYPE on a CAS loss
(previously an opaque `Lance` string, now a typed conflict); no success/failure
outcome changes. The bounded re-open retry that consumes the new type lands next.
* test(engine): red — concurrent v3→v4 migrations error instead of converging
`migrate_v2_to_v3` is concurrent-runner idempotent by design; v3→v4 regressed it.
`merge_lineage_rows` uses `conflict_retries(0)` and `migrate_v3_to_v4` has no
app-level retry, so when two processes open the same legacy graph at once the
backfill's row-level CAS loser errors the whole open instead of converging.
The test opens two `__manifest` handles at the same pre-migration (v3,
empty-lineage) HEAD and runs both `migrate_internal_schema` calls under
`tokio::join!`, forcing the `graph_head:main` CAS to fire every run. Against the
current code the loser fails with `RowLevelCasContention` ("Attempted 0
retries.") — the predicted symptom — so the "both must converge" assertion
panics. The bounded re-open retry that makes both converge lands next.
* fix(engine): make the v3→v4 lineage backfill converge under concurrent runners
`migrate_v2_to_v3` is concurrent-runner idempotent; v3→v4 was not. Two processes
(or open-for-write handles) opening the same legacy graph at once both reach the
backfill merge, and `merge_lineage_rows`'s `conflict_retries(0)` made the
row-level CAS loser error the whole open instead of converging.
Two contention points, both now handled all-or-nothing:
1. The backfill merge on `graph_head:<branch>`. Wrap (fast-path re-read → read
legacy → merge) in a bounded re-open retry loop: a `RowLevelCasContention` loss
re-opens the manifest past the winner's (atomic) commit and re-loops; the
fast-path re-read then sees the winner's lineage and stamps. On budget
exhaustion it returns a `RowLevelCasContention`-typed error so the publisher's
OUTER retry loop completes it. The retry decision reuses the publisher's
`is_retryable_publish_conflict` so the two stay in lockstep.
2. The terminal stamp bump. Making the merge loser converge newly lets BOTH
runners reach `set_stamp(4)` — an `UpdateConfig` commit on the same key — so the
loser gets `lance::Error::IncompatibleTransaction` (NOT a row-level CAS, so the
merge loop doesn't catch it). This surfaced only under the concurrent
full-suite run, not the isolated test. Both write the SAME value, so the
conflict is benign: `commit_v4_stamp_idempotently` re-opens and, if the stamp
already reached the target, succeeds; else re-applies (bounded).
Greens the race test from the previous commit (3x isolated, 5x full-suite, no
flake). The new `IncompatibleTransaction` match is pinned by
`lance_surface_guards.rs::lance_error_incompatible_transaction_variant_exists`.
* fix(engine): refuse a future internal-schema stamp on the branch read path
`load_commit_cache_for_branch` dispatched on the branch's internal-schema stamp —
`< CURRENT` to the v3 legacy fallback, `>= CURRENT` to the manifest projection —
but never refused a `> CURRENT` branch stamp, so a newer-binary shape would be
misread by the projection rather than rejected.
Add `refuse_if_stamp_too_new(stamp)` (re-exported `pub(crate)` from `migrations`)
right after the branch stamp is read, mirroring the main read path's
`refuse_if_internal_schema_too_new`. This is defense-in-depth, not a live hole:
migrations run main-first (main migrates on open; each branch on its first write),
so main's stamp is always >= every branch's and the main path refuses first. The
guard closes the gap if that ordering invariant is ever weakened.
Tested by force-stamping a real branch past CURRENT and asserting the branch read
refuses with the upgrade error (the test misreads via the projection — returns Ok
— without the guard, confirmed by removing it).
* docs(rfc-013): record the v3→v4 migration robustness fixes
invariants.md Known Gaps: the `migrate_v3_to_v4` entry now states the migration is
loud on non-not-found legacy-open errors and concurrent-runner idempotent (bounded
re-open retry on the merge CAS + idempotent stamp bump), and that the branch read
path refuses a `> CURRENT` stamp.
lance.md: note the two new surface guards the migration depends on
(`dataset_open_missing_returns_not_found_variant`,
`lance_error_incompatible_transaction_variant_exists`).
testing.md: note the migration fault-injection test in the failpoints row.
* refactor: remove dead code and silence warnings across engine + cluster
Dead-code sweep follow-up to the RFC-013 stack. No behavior change.
- engine: delete the orphaned `validate_edge_cardinality` — the load path uses
`validate_edge_cardinality_with_pending_loader` for every mode (including
Overwrite, which it treats as the replacement table image), so the old
standalone validator had no caller — and correct its sibling's now-stale doc
reference. Gate `TableStore::append_batch` `#[cfg(test)]`: it is the inline-
commit residual kept only for recovery test setup, with no non-test caller.
- cluster: drop unused imports in `lib.rs`, delete the unused
`ClusterStore::payload_display`, and raise `LiveGraphObservation` /
`GraphObservationJson` / `PolicyTarget` to `pub(crate)` to match the functions
that return them.
Both lib crates now build warning-free.
* fix(engine): match Lance's typed DatasetAlreadyExists, not the message string
The internal create-or-open idempotency fallbacks in `db/commit_graph.rs` and
`db/recovery_audit.rs` classified the "already exists" race by
`err.to_string().contains("Dataset already exists")` — a Lance display string,
not an API contract. A wording change upstream would silently break the fallback
(a re-create would error instead of opening the existing table). Match the typed
`lance::Error::DatasetAlreadyExists { .. }` variant instead — the same discipline
as the v3→v4 migration's not-found classifier — pinned by the new
`lance_surface_guards.rs::lance_error_dataset_already_exists_variant_exists`
guard so a Lance rename turns red instead of silently regressing.
* refactor(engine): consolidate now_micros into one crate::db helper
Four `fn now_micros() -> Result<i64>` copies (commit_graph, recovery_audit,
graph_coordinator, manifest/graph) had already drifted: three mapped the
clock error to `OmniError::manifest("...UNIX_EPOCH...")` while recovery_audit
used `OmniError::manifest_internal("...unix epoch...")`. Replace all four with
one `pub(crate) fn now_micros()` in `db/mod.rs` (the majority `manifest`
variant), and repoint the eight call sites at `crate::db::now_micros()`. No
test asserts on the failure message, so unifying the variant is behavior-safe;
the timestamp-mapping contract can no longer fork across the rows it stamps.
* refactor(engine): drop the dead snapshot param from roll_back_sidecar
`roll_back_sidecar` took `snapshot: &Snapshot` only to discard it with
`let _ = snapshot;` — rollbacks now always publish (the restored HEAD plus a
recovery-commit lineage row), so the snapshot is never read to decide whether
to skip a publish. Remove the parameter, the two call-site arguments, and the
suppressor. A signature must not advertise inputs it does not consume. The
`Snapshot` import stays — `process_sidecar`, `roll_forward_all`, and
`record_audit_recovery_rollforward` still take it.
* test(engine): red — open_at_branch wedges a branch on a missing commit-graph ref
A v4 graph keeps its graph lineage in `__manifest` (RFC-013 Phase 7); the
`_graph_commits.lance` branch ref is a derived artifact. An interrupted
fork-reclaim or a `cleanup` race can drop that derived ref while the manifest
lineage stays intact. Per invariants 7 + 15 a missing derived ref must not fail
a logical read of the lineage.
This wedge builds a real v4 `feature` branch (its `graph_head:feature` row in
`__manifest`), force-deletes ONLY the `_graph_commits.lance` `feature` ref, then
asserts the branch reads (`open_at_branch` / list-commits / `merge_base`)
succeed from `__manifest` while a write that needs the derived ref
(`create_branch`) fails loudly with the typed actionable error.
Red against current code: `open_at_branch`'s hard `checkout_branch(branch)?` on
the missing ref errors `OmniError::Lance` (Lance "Not found:
_graph_commits.lance/tree/feature/_versions"), wedging the logical read.
* fix(engine): read manifest lineage independent of the derived _graph_commits ref
`CommitGraph::open_at_branch` did a hard `checkout_branch(branch)?` on the
`_graph_commits.lance` branch ref before reading lineage — so a missing derived
ref (an interrupted fork-reclaim, or a `cleanup` race) wedged the branch's
commit-list / merge-base / snapshot resolution even though the lineage is
readable from the authoritative `__manifest` (RFC-013 Phase 7). That is a
derived/physical artifact failing a logical read — invariants 7 and 15.
Make the held commits handle `Option<Dataset>` (mirroring `actor_dataset`).
`open_at_branch` and `refresh` check out the derived ref best-effort: a typed
not-found (`RefNotFound`/`NotFound`) yields a `None` handle while the read
re-syncs from `__manifest`; any other open error still propagates. The manifest
existence gate is unchanged — `load_commit_cache_for_branch` keeps its hard `?`,
so a truly absent branch still fails loudly at the manifest. `create_branch`
(the only writer that forks a ref) and the folded-in version lookup return a
loud, actionable error on `None`, deferring repair to `cleanup`'s existing
orphan reconciler rather than inlining a write on a read-side refresh. Reads
(`head_commit`/`load_commits`/`get_commit`/`merge_base`) never touch the handle.
Greens the wedge regression from the preceding commit.
* fix(engine): v3→v4 retry loops return retryable contention on exhaustion
`commit_v4_stamp_idempotently`'s retry loop used `0..=STAMP_RETRY_BUDGET`
(6 iterations) with an `attempt < STAMP_RETRY_BUDGET` guard, so the LAST
iteration's `IncompatibleTransaction` fell through to
`Err(e) => OmniError::Lance(...)` — stringified, non-retryable — instead of the
intended `RowLevelCasContention`, and the post-loop contention return was dead
code. The publisher's outer retry only re-runs `is_retryable_publish_conflict`,
so under sustained concurrent v3→v4 migration the one-time stamp bump could fail
instead of converging, defeating the idempotency the migration is supposed to add.
Fix the loop to `0..BUDGET` with an UNGUARDED `IncompatibleTransaction` arm: the
retryable variant is always handled inside the loop (re-open + same-value check +
retry), so it can never reach the stringifying catch-all, and the post-loop is the
SINGLE reachable exhaustion path — the typed `RowLevelCasContention`. The `Err(e)`
arm now catches only genuine non-contention errors. Apply the same range alignment
to the sibling merge loop in `migrate_v3_to_v4` (behaviorally correct today — its
`Err(err)` returns the already-typed contention — but it carried the identical
off-by-one structure the stamp loop was copied from; aligning both stops the next
copy from re-introducing it).
Test-first. The exhaustion path is otherwise near-unreachable — a real concurrent
winner stamps the same value, so the re-read returns Ok on the first retry — so a
new `migration.v4_stamp.force_incompatible` failpoint forces every stamp attempt to
lose, driving exhaustion deterministically. Against the pre-fix loop the new
`v4_stamp_exhaustion_returns_retryable_contention` test goes red with
`Lance("Incompatible transaction: injected failpoint triggered…")`; with the fix it
asserts the typed `RowLevelCasContention`. Found by automated review on #299.
* feat(engine): minimum-supported internal-schema floor + retirement tripwire
The internal-schema migration chain (`migrate_internal_schema`) had a too-new
ceiling but no floor, so every old `migrate_vN_…` arm and the v3 legacy readers
it needs stay forever — the pile grows by one migration + readers + tests every
schema version. Add `MIN_SUPPORTED_INTERNAL_SCHEMA_VERSION` (1 today, a pure
no-op: `read_stamp` floors an absent stamp at 1 and no real graph carries 0) as
the oldest stamp this binary opens; raising it is how the chain sheds old code.
Collapse the one-sided `refuse_if_stamp_too_new` into `refuse_if_stamp_unsupported`
checking both bounds, so the floor lands at all three stamp-enforcement sites —
the write-path migrate dispatcher, the read-only open guard, and the branch
lineage-read path (`commit_graph.rs`) — via one compiler-enforced rename. A
hand-wired floor twin would have had to touch each site, and the branch-read path
is easy to miss; one combined guard cannot half-enforce. Rename the read-only
wrapper `refuse_if_internal_schema_unsupported` to match.
A compile-time tripwire (`const _: () = assert!(LOWEST_REGISTERED_MIGRATION_SOURCE
== MIN_SUPPORTED…)`) fails the build if a future floor bump forgets to delete the
now-dead migration arm (or vice versa) — stronger than a runtime test, impossible
to skip, and it doubles as the use that keeps the mirror const live.
Tests: a sub-floor graph is refused in both open modes (twin of
`future_stamp_is_refused_in_both_open_modes`); the guard accepts exactly
[MIN, CURRENT]. No behavior change for any real graph. The retirement runbook
lives on the `MIN_SUPPORTED` doc-comment + invariants.md.
* fix(engine): compose migration contention with publisher retry; precise recovery-converge audit commit
Three review-surfaced fixes on the RFC-013 Phase 7 path.
Publisher retry vs migration contention: `publish()` propagated a
`load_publish_state` error fatally via `?`, so a `RowLevelCasContention` surfaced
by the v3->v4 migration's exhausted merge/stamp budgets aborted the publish
instead of being retried — only `merge_rows` conflicts hit the retry. This
contradicted the migration's own design, which returns that typed error
EXPECTING the publisher to re-run the load (by which point a concurrent winner
has usually finished the migration, so the next scan is a no-op). Route a
retryable load error through the same retry path as a retryable `merge_rows`
conflict. Regression test (failpoints): a one-shot retryable contention injected
into `load_publish_state` now commits via the retry; red without the fix (the
write fails with the injected contention).
Recovery-converge audit commit id: `converge_or_defer_roll_forward` recorded the
branch HEAD as the audit row's `graph_commit_id`, but a concurrent user write can
advance `graph_head` past the recovery commit between the winner's publish and
this read — attributing the audit to a later, wrong commit. Use the latest
`RECOVERY_ACTOR`-authored commit (what `publish_recovery_commit` mints), which is
the recovery commit by construction. The audit's actor was already correct (it
comes from `sidecar.actor_id`, not the commit).
Dead param: drop the unused `snapshot` from `record_audit_recovery_rollforward`
(removing the `let _ = snapshot;` suppressor). `storage` stays — it is used to
delete the sidecar.
* refactor(storage): gate test-only TableStore::append_batch behind cfg(test)
The inherent append_batch is used only by in-source recovery test setup, but
the non-test lib build (cfg(test) off) cannot see those callers and emitted a
dead_code warning. Gating the method #[cfg(test)] silences the false positive
and enforces its own doc contract ("no new engine call sites") by construction
— engine code physically cannot call a cfg(test) method.
* test(failpoints): harden fault-injection harness + reproduce roll-forward CAS race
Hardens the test infrastructure around the process-global `fail` registry, and
adds a deterministic red repro for the open-time recovery sweep's roll-forward
CAS race (iss-schema-apply-reopen-recovery-race). The fix lands in the next
commit — this commit is intentionally red (rule 12: red→green visible in log).
Harness:
- One `ScopedFailPoint` (engine) gaining `with_callback`; the cluster duplicate
is removed and cluster tests reuse the engine type via `omnigraph/failpoints`.
- `#[serial]` on every failpoint test (the registry is process-global, so shared
names interfere under parallelism); `serial_test` added to cluster dev-deps.
- `helpers::failpoint::Rendezvous` (park-first / wait-until-reached / release)
replaces fixed-`sleep` cross-thread coordination; the three concurrent tests
now rendezvous deterministically. The reached flag doubles as a fired-assert.
- Compile-checked `failpoints::names` catalog (engine + cluster); every call
site references a const, and `failpoint_names_guard.rs` enforces "no string
literal names" by source-walk, so a typo is a build error not a silent no-fire.
Red repro:
- New `recovery.before_roll_forward_publish` failpoint at the sweep's
classify -> publish-CAS window (the only injection point there).
- `open_sweep_roll_forward_converges_when_manifest_advances_concurrently`: two
concurrent open-sweeps race one pending sidecar; the sweep parked at the
failpoint loses its publish CAS to the other and fails the open with
`ExpectedVersionMismatch`. FAILS at this commit by design.
* fix(recovery): converge roll-forward when the manifest advances concurrently
The open-time recovery sweep classified a pending sidecar as RolledPastExpected,
then published a manifest CAS at the sidecar's pinned expected_version. Under a
concurrent writer that advanced the manifest past expected during the
classify -> publish window, the CAS failed with ExpectedVersionMismatch and
`?`-propagated, failing the whole Omnigraph::open.
iss-schema-apply-reopen-recovery-race.
A roll-forward's postcondition is "the manifest reflects the sidecar's committed
Lance state", not "this sweep won the CAS" (invariants 7 & 15). On an
ExpectedVersionMismatch, re-read the live manifest and check whether the
sidecar's intent is already satisfied (every pinned table at a version >= the
one we observed and tried to publish; added tables registered; tombstones gone
— sound under the heal-first invariant, documented at the check). If satisfied,
this is convergence: record the RolledForward audit + delete the sidecar
idempotently. If only partway, defer to the next pass. Either way the open no
longer fails. Other errors still propagate; a genuine logical conflict
resurfaces via the classifier's InvariantViolation.
Turns the red repro from the previous commit green. The roll-BACK twin
(iss-recovery-sweep-live-writer-rollback) is destructive (Lance Restore) and
still needs a cross-process lease — the known-gap is updated accordingly.
* Address PR review: harden failpoint name guard + dedupe converge audit
Two issues surfaced in PR review of the failpoint hardening + recovery fix:
1. Name guard had a line-split blind spot. It scanned per line, so a call
wrapped across lines (`park_first(\n "name",\n)`) put the literal on a
different line than the call prefix and bypassed the "no string-literal
failpoint names" check — and one such literal
(`mutation.delete_node_pre_primary_delete`) had slipped through. Make the
guard whitespace/newline-tolerant (skip past the open paren to the first
argument token) so wrapping can't hide a literal, and convert the bypassed
site to the `names::` const.
2. Convergence path could append a duplicate recovery audit. When a
roll-forward publish loses its CAS but the manifest already reached the
sidecar's goal, `converge_or_defer_roll_forward` recorded a RolledForward
audit unconditionally. Under the heal-first invariant, whoever advanced the
manifest already healed this sidecar (audit + delete), so a second row
landed in `_graph_commit_recoveries` for one recovery event. Gate the
audit+delete on the sidecar still being present: absent => the winner
completed it, return success with no duplicate row. The convergence
regression test now asserts exactly one audit row.
* docs(dev): remove the schema-apply recovery-flake handoff (fixed by this PR)
The handoff was a transient investigation note for
`iss-schema-apply-reopen-recovery-race`, which this PR fixes (the converge
helper + the red→green regression). Its rationale now lives durably in the
dev-graph issue, the PR/commit history, and invariants.md, so the handoff is
obsolete. Drop the doc, its dev-index row, and the dangling reference from the
RFC-013 handoff; the doc cross-link check stays green.
* fix(recovery): include added-table registrations in the converge audit
The CAS-loss convergence audit built outcomes only from `sidecar.tables`,
omitting the `additional_registrations` that the normal `roll_forward_all`
audit includes. For a SchemaApply sidecar with added types, a converge-path
audit row would be incomplete versus the normal roll-forward path for the same
recovery kind. Mirror the roll-forward outcome construction (append a
registration outcome per added table) so both paths emit the same audit shape.
* fix(cluster): stop cluster-apply crash-loops from the recovery-sidecar trap
A `cluster apply` carrying a schema change against a graph that has
non-main branches, or an unsupported "needs backfill" migration, armed a
recovery sidecar *before* calling the engine, then left it behind when the
engine rejected the apply pre-movement. The server refuses to boot while
any sidecar is pending, and re-running apply re-armed a fresh sidecar — an
unescapable crash loop. None of the engine rejections are bugs; the trap
is in the apply/serve choreography.
Three coordinated changes:
1. Preview before arming the sidecar. `cluster apply` now runs
`preview_schema_apply_with_options` before `write_recovery_sidecar`, so
parser/planner rejections (non-main branches, unsupported plan) fail
loudly without leaving recovery work behind. The post-preview engine
error path now deletes the sidecar when the live schema still matches
the recorded digest (nothing moved), and keeps it only on real
mid-movement failure — both branches covered by new engine-failpoint
tests (cluster failpoints now enable omnigraph/failpoints).
2. Per-graph quarantine at serve time instead of whole-cluster refusal.
A graph-attributed pending sidecar, an unopenable graph root, a query
parse failure, or an unresolvable embedding provider now quarantines
just that graph (logged loudly at every boot layer) while healthy
graphs serve; `/graphs` lists only ready graphs and quarantined routes
404. Cluster-global problems (missing/unreadable state, malformed or
unattributable sidecars, shared-catalog or cluster-policy errors, zero
healthy graphs) stay fail-fast. `--require-all-graphs` /
OMNIGRAPH_REQUIRE_ALL_GRAPHS=1 restores all-or-nothing boot.
3. Backfill embedding-provider profile metadata on apply. Mirrors the
existing policy-binding backfill: a pre-5A ledger missing
`embedding_profile` is now detected as a metadata-only change and
backfilled by a no-op apply, instead of bricking serve with
`embedding_provider_profile_missing` forever.
Tests: trap (no sidecar after a rejected apply), both digest-cleanup
branches, per-graph quarantine (cluster + server), embedding backfill.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: resilient cluster boot + recovery-sidecar trap fix
Amend RFC-005 D4 readiness posture (cluster-global fail-fast vs graph-local
quarantine; deviation #5 for --require-all-graphs), add the v0.7.0 release
note, and update the user cluster/server/deployment docs and the
OMNIGRAPH_REQUIRE_ALL_GRAPHS env var.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cluster): surface sidecar-cleanup failures; document severity promotion
Address Greptile review on PR #284:
- The pre-movement sidecar cleanup fast-path discarded `delete_object`'s
result, so a transient delete failure left the graph quarantined with no
signal. Add `try_delete_object` (Result-returning) and emit a
`recovery_sidecar_cleanup_failed` warning diagnostic on failure; the
fire-and-forget `delete_object` now delegates to it.
- Document why the serve-time loop promotes every `list_recovery_sidecars`
diagnostic to a cluster-fatal error (the listing only emits genuine
read/parse/version failures, as warnings, whose blast radius serving
cannot prove) and note the promote-by-code path if that ever changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When a server/cluster scope resolves with no --graph and no default_graph, the CLI auto-uses a sole graph (cluster) or errors listing the candidate graph ids (cluster catalog; multi-graph server via best-effort GET /graphs), never a silent pick. GraphClient::resolve becomes async; flat/single-graph servers and happy paths are unaffected.
* feat(cluster): cluster_root_for_graph_uri detection helper (RFC-010 Slice 3)
Public helper the CLI uses to refuse `init` into a cluster-managed location:
given a graph storage URI of the cluster layout (`<root>/graphs/<id>.omni`),
return the cluster root if `<root>` holds `__cluster/state.json`, else None.
Cheap by construction — a URI that doesn't match the `<root>/graphs/<id>.omni`
shape returns None with zero I/O, so ordinary `init` targets never probe
storage. Works for file:// and s3:// via the storage adapter. Adds two
ClusterStore accessors (`display_root`, `has_state`).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(cli): cluster-managed maintenance addressing + init signpost (RFC-010 Slice 3)
Two cluster-graph-aware CLI behaviors, sharing the cluster-resolution path.
Maintenance addressing. `optimize`/`repair`/`cleanup` gain
`--cluster <dir|s3://…> --cluster-graph <id>`, which resolves the graph's
storage URI from the served cluster snapshot (the same truth a `--cluster`
server boots from — `read_serving_snapshot*`) and opens it embedded. The
operator no longer hand-types `<storage>/graphs/<id>.omni`. A distinct flag is
required because the global `--graph` is `requires = server` and means a remote
multi-graph id. clap enforces both-or-neither and exclusion with the positional
URI / `--target`; an unserved graph errors loudly, pointing at `cluster apply`.
init signpost. `init` refuses a cluster-managed positional path (the
`<root>/graphs/<id>.omni` layout where `<root>` holds `__cluster/state.json`,
detected by `cluster_root_for_graph_uri`) and points at `cluster apply` — graphs
in an established cluster are created with ledger/recovery/approvals, not by
hand. The check is gated on the path shape, so ordinary `init` does no extra I/O
and existing pre-apply cluster-graph inits are unaffected.
planes guard remediation now also mentions `--cluster … --cluster-graph …`
(the two Slice-1 guard-string tests track it). Docs updated (cli-reference
Command planes, maintenance.md, cluster.md §7); the stale "no S3-hosted cluster
directories" limitation is dropped (RFC-006 landed it).
Tests (cli_cluster.rs, reusing the apply-a-cluster fixture): resolve by id,
unknown-id error, `--cluster` requires `--cluster-graph`, init refusal +
signpost, and ordinary init still works.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cli): resolve cluster graphs from the state ledger, not the serving snapshot
Addresses the Greptile review on #221. `read_serving_snapshot*` does
all-or-nothing serving validation — recovery-sidecar checks plus a digest
verify of every catalog payload (query .gq, policy blobs). Using it to resolve
a maintenance target coupled `optimize`/`repair`/`cleanup` to the readiness of
unrelated resources: a single corrupt policy blob, or a pending recovery sweep,
would block the command before it could touch the graph — worst for `repair`,
the tool you reach for *when the cluster is degraded*.
Add `omnigraph_cluster::resolve_graph_storage_uri(cluster, graph_id)`: read the
state ledger, confirm the graph is in the applied revision, return
`graph_root(id)` — the URI is deterministically derivable, no catalog
validation. The CLI's cluster resolver now calls it.
Test: `optimize --cluster … --cluster-graph …` still resolves after the catalog
payloads (`__cluster/resources/`) are removed — the ledger-only path is not
blocked by degraded/unrelated catalog state.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Two serving changes that complete RFC-006's read side:
ServingPolicy carries the policy bundle CONTENT (digest-verified at
snapshot read) instead of a blob path — the catalog may live on object
storage, and the server must not re-read mutable state after the
snapshot. The server grows a PolicySource enum: File for omnigraph.yaml
deployments (unchanged), Inline for cluster boots, wired through
PolicyEngine::load_{graph,server}_from_source.
read_serving_snapshot_from_storage(uri) reads the applied revision
straight from a storage root, and --cluster accepts a scheme-qualified
URI (s3://bucket/prefix): config-free serving — a serving box needs only
the URI and credentials; the ledger and catalog on the bucket ARE the
deployment artifact. Bare paths keep the config-directory behavior.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cluster.yaml gains an optional storage: URI deciding where everything the
cluster STORES lives: the state ledger, lock, content-addressed catalog,
recovery sidecars, approval artifacts, and the derived graph roots
(<storage>/graphs/<id>.omni). Absent, it defaults to the config directory
itself — the original layout, byte-compatible, so pre-existing clusters and
the whole test suite are untouched. Declared configuration always stays in
the working tree (Terraform's config-local/state-remote split); credentials
are env-only, never in cluster.yaml.
Every command resolves its store from the declared root (a bad root is a
loud invalid_storage_root). Graph-root derivation, the delete executor
(prefix delete via the adapter), the sweep's existence probes, the catalog
payload write/verify/read paths, and the serving snapshot all flow through
ClusterStore — the last raw-fs holdouts for stored state are gone, and the
deny-list gains the rule that keeps it that way.
Tests: default-layout byte-compat, a file:// root relocating the entire
cluster (ledger+catalog+graphs under the new root, nothing under the config
dir, serving snapshot follows), invalid-root validation. 98 in-crate + 9
failpoints + full workspace gate green. The s3:// flavor lands with PR 3's
gated RustFS e2e.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LocalStateBackend becomes ClusterStore: every stored byte — state ledger,
lock, recovery sidecars, approval artifacts — now flows through the
engine's StorageAdapter, making file:// and s3:// one code path. Behavior
on the file backend is byte-compatible (layout, CAS semantics, diagnostics,
lock release timing) and the entire pre-existing suite passes unchanged.
Mechanics: the ledger CAS keeps its public sha256 vocabulary while the
physical swap is token-conditioned (ETag If-Match on S3 via PR #186's
primitives; content-token + temp/rename locally — the pre-port semantics);
the lock is a create-only put (genuinely cross-machine on object stores)
with deterministic drop-release locally and best-effort spawned release on
S3; sidecars/approvals address by URI (SweepOutcome and the executors carry
strings); sweep row-1 retirement joins the uniform deferred post-CAS
cleanup. ClusterStore also gains the catalog-payload and graph-root
methods that commit 2 wires in.
Async ripple: status/force-unlock/serving-snapshot and the server's
settings loader chain go async (CLI dispatch and ~20 test hosts follow,
mechanically). tokio joins the cluster crate's runtime deps for the lock
guard's handle.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verbatim move of the public output/diagnostic types and the internal
state/sidecar/approval models; previously-private types and their fields
get pub(crate) (they were crate-visible by position before). lib.rs is now
the command pipeline + public API. 95 tests green; full workspace gate
green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verbatim move of cluster.yaml parsing, query discovery, source digesting,
header/id validation, path resolution, and live-graph observation. Two
helpers that the cut swept along were relocated to their right homes
(state-status helpers back to lib.rs, lock-file helpers to store.rs). 95
tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verbatim move of the Serving* types, read_serving_snapshot, and
read_verified_payload; public re-exports preserved (the server's imports
are unchanged). 95 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verbatim move of LocalStateBackend, StateSnapshot, StateLockGuard and their
impls — the single home for stored-state I/O (state ledger, lock, recovery
sidecars, approval artifacts), where the RFC-006 object-storage port lands
next as a focused diff. Visibility bumps (pub(crate)) only; 95 tests green
before and after.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verbatim move (indentation preserved — embedded raw-string fixtures are
content). lib.rs drops from 7,857 to ~4,750 lines; `use super::*` resolves
to the crate root through the #[path] module declaration unchanged. 95
tests green before and after.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
resolve_query_decls hands its file contents to the caller; the per-query
digest/typecheck pass reuses them instead of re-reading (a file with N
queries was read N+1 times), which also closes the window where a file
changing between enumeration and validation produced a confusing
query_key_mismatch for a just-discovered name. Explicit-map declarations
read as before.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cluster.yaml's graphs.<id>.queries previously accepted only an explicit
name->file map, forcing configs to re-enumerate every `query <name>` that
the .gq files already declare (the SPIKE cookbook needed 66 entries for 6
files). The files ARE the declaration now: `queries: queries/` discovers
every declaration in a directory's top-level *.gq (sorted), a list form
takes explicit files, and the map stays for fine-grained control.
Discovery is loud — unreadable/unparseable files and duplicate query names
fail validation (query_parse_error, duplicate_query_name). Downstream is
untouched: each discovered query is still an individually addressed
resource with the containing file's digest.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RFC-005 §D2/§D4: read_serving_snapshot reads the applied revision as
everything a server needs to boot — graphs at derived roots, stored-query
sources read from the content-addressed catalog and re-hashed against the
recorded digests, policy blob paths with their applied applies_to bindings.
All-or-nothing: missing state, pending recovery sidecars, missing/tampered
blobs, pre-5A entries without bindings, and an empty graph set each refuse
the snapshot with a remedy; no partial serving. Lock-free by design — the
state file is replaced atomically, so the read is a consistent
point-in-time ledger.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Slice 5A of RFC-005: the state ledger becomes serving-sufficient for the
Phase-5 server boot. StateResource gains an optional applies_to (normalized
typed refs: cluster | graph.<id>), written by apply for every applied policy
create/update from the desired config's validated bindings.
The hole this closes: applies_to is not part of the policy file digest, so a
binding-only edit previously produced NO plan change at all (a 4C e2e even
asserted that — the gap, not a contract). Binding changes are now
first-class: a post-diff pass emits an Update with equal before/after
digests and a binding_change marker (visible in plan/apply JSON and human
output as [bindings]), classification/execution treat it as an ordinary
catalog-tier applied change (payload skips naturally — the blob is
unchanged), and convergence requires zero binding divergence, so stale
bindings can never report converged. Pre-5A ledger entries (no bindings
recorded) surface as the same backfill Update; one apply heals them, exactly
the remedy RFC-005's boot-error path names.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stage 4C execution half (RFC-004 §D5/§D6 + sweep rows 7/7b/8): an approved
graph.<id> delete — and its riding schema/query deletes — classifies Applied
and executes LAST in the run, sidecar-fenced: pre-op manifest pin (best
effort; partial roots still delete), approval_id carried in the sidecar,
recursive root removal (NotFound tolerated), subtree tombstoned out of the
ledger with a tombstone observation, the approval consumed in the same state
CAS (ledger summary) and its artifact file rewritten with consumed_at only
after the CAS lands — a failed run consumes nothing and the approval stays
valid for the retry.
Sweep rows: already-tombstoned intents retire (7); a completed delete with a
stale ledger rolls forward — tombstone + approval consumption + audit entry
(7b, idempotent); a still-present root retires the stale intent with a
graph_delete_incomplete warning and the still-approved delete re-executes in
the same run (8) — prefix removal is idempotent, so retry IS the repair.
The multi-graph mixed e2e gets its conclusion: blocked without approval,
cluster approve graph.engineering --as andrew, converge, tombstone visible
in status. Phase 4's disposition matrix is now fully executable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RFC-004 §D4, gate half: graph deletes (and their subtree) now classify
Blocked/approval_required instead of Deferred; the new cluster approve
command (requires the global --as actor) writes
__cluster/approvals/{ulid}.json bound to the desired config digest and the
change's before/after digests, so config or state drift invalidates the
artifact automatically (approval_stale warning, never authorizes). One gate
per subtree: compute_approvals lists only the graph-level delete, and
ApprovalRequirement gains a satisfied flag surfaced by plan. Consumption and
the delete executor land next — until then approved deletes stay blocked so
a gate-only build can never strip state without removing the root.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stage 4B (RFC-004 §D1/§D5): schema.<id> Update changes classify Applied and
execute after graph creates, sequentially and sidecar-fenced — read-write
open (the engine's own recovery runs first), pre-op manifest pin recorded,
apply_schema_as with allow_data_loss: false (soft drops only; hard drops wait
for 4C's approval artifacts), post-op pin rewritten into the sidecar, sidecar
retired only after the final state CAS. Queries gated on a same-plan schema
update unblock (the migration lands first in the same run); failures —
unsupported migrations, lock contention, user branches — surface as
schema_apply_failed with the engine's message, demote dependents via the
origin-aware demotion helper, and stop further graph-moving work.
Schema evolution is now fully cluster-driven (the defer -> manual schema
apply -> refresh loop is gone), and out-of-band schema drift is converged
back by apply as an ordinary soft migration (axiom 8: drift correction is
gated like any change; the recoverable tier needs no approval) — both pinned
by reworked e2es. The multi-graph mixed e2e's deferred row is now
delete-shaped, pre-staging the 4C surface.
Actor: cluster apply accepts the CLI's global --as via the new ApplyOptions /
apply_config_dir_with_options (apply_config_dir delegates unchanged); the
actor is echoed in ApplyOutput and recorded in sidecars and audit entries,
and threads to apply_schema_as so Cedar fires wherever a checker is
installed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RecoverySidecarKind::SchemaApply with digest-based sweep classification
(robust to unrelated manifest movement; version pins stay forensic):
ledger-consistent -> sidecar retired (RFC-004 rows 1+2); live digest matches
the intended schema, state stale -> roll forward with composite recompute and
a recovery_records audit entry (row 3); unverifiable or unexpected digests ->
pending, kept, graph-moving work blocked (rows 1-unopenable/6).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RFC-004 §D7's data-aware preview: for every schema update, plan opens the
live graph read-only and embeds the engine's migration plan (supported flag
+ typed steps) in the change record; the human renderer prints the steps.
Preview failures (unreachable graph, planner error) degrade to the digest
diff with a schema_preview_unavailable warning — planning never blocks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mechanical conversion ahead of Stage 4B (plan will preview schema migrations
against live graphs): signature, CLI dispatch, and test callers. Zero
behavior change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stage 4A (RFC-004 §D1/§D5): graph.<id> Create — and its paired schema Create,
which the init carries — classify Applied and execute first in the run,
sequentially and sidecar-fenced: sidecar written before Omnigraph::init at
the derived root, rewritten with the post-init manifest pin, deleted only
after the final state CAS lands. Dependent queries and policies no longer
block on a graph create in the same plan — creates run first, so they apply
in the same run; a create failure demotes them to blocked
(dependency_not_applied) and stops further graph-moving work (loud partials),
with the sidecar left for the sweep to classify. Graphs with a kept recovery
sidecar (rows 5/6) classify Blocked/cluster_recovery_pending, and the sweep's
Drifted/Error statuses are never clobbered by a generic Blocked.
Schema source is re-read and digest-verified under the lock before the init
(the write_resource_payload TOCTOU posture). Plan previews the same
dispositions. e2e fallout updated: a fresh multi-graph config now converges
in one apply; a destroyed root is re-created as an EMPTY graph by the next
apply (declarative convergence — visible in plan, called out in docs); the
new cluster_e2e_declared_graph_created_by_apply pins the no-manual-init flow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RFC-004 §D2/§D3 for the graph_create kind. RecoverySidecar records intent
under __cluster/recoveries/{ulid}.json; the roll-forward-only sweep runs at
the start of apply/refresh/import under the state lock and classifies each
survivor by observation: root absent -> intent removed (row 1); outcome
already recorded -> retired (row 2); create completed but state stale ->
ledger rolled forward with a recovery_records audit entry (row 4); partial
root -> Error/graph_create_incomplete, kept, never auto-deleted (row 5);
unexpected schema -> Drifted/actual_applied_state_pending, kept (row 6).
Sweep mutations ride the command's existing CAS write; completed sidecars
are deleted only after that write lands. Read-only status/plan warn
(cluster_recovery_pending) without acting. The apply payload gate now counts
only payload-phase errors so kept-sidecar diagnostics don't abort the run
before their statuses persist.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mechanical conversion ahead of Stage 4A graph create (which calls the async
Omnigraph::init from inside apply): the fn signature, the CLI dispatch arm,
and every test caller (#[test] -> #[tokio::test]). Zero behavior change; all
60 lib tests and 3 failpoint tests green before and after.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Optional failpoints feature (dep:fail + fail/failpoints, deliberately NOT
enabling omnigraph/failpoints), a maybe_fail/ScopedFailPoint module returning
Diagnostic-typed injected errors, and two call sites in apply_config_dir:
cluster_apply.after_payload_phase (the crash point: blobs on disk, state
untouched) and cluster_apply.before_state_write (routes through the
persisted-statuses revert contract; a cfg_callback here can mutate state.json
to make the CAS check fail organically). Feature off compiles to Ok(()) —
zero behavior change. Tests live in a separate integration binary because the
fail registry is process-global. Also refresh the crate description (stale
'read-only' since Stage 3A).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Closes the Stage 3A product gap where a deleted or corrupted blob under
__cluster/resources/ went unnoticed forever (status reported converged and
apply could not repair it because the digests matched).
verify_catalog_payloads checks every query/policy digest in state against its
content-addressed blob (existence + full sha256 re-hash; graph/schema/unknown
addresses have no payloads and are skipped). status reports findings read-only
(warnings catalog_payload_missing/_mismatch; error catalog_payload_read_error
— an unverifiable catalog must not report healthy). refresh closes the
self-heal loop: missing/mismatched blobs mark the resource drifted and remove
its digest from state so the next plan proposes a create and the next apply
republishes; unreadable blobs keep the digest (no spurious republish), mark
error, and exit non-zero. Verification runs before graph observation so the
recomputed graph composite already excludes removed query digests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two review findings (greptile, PR #165):
- ApplyOutput.resource_statuses on a failed state write now carries the
pre-apply on-disk snapshot instead of the in-memory mutations that were
never persisted, so automation reading the field independently of `ok`
cannot see phantom applied/blocked statuses. Regression test forces the
state write to fail via a read-only __cluster dir (unix-only, skips when
permissions are not enforced).
- Human-mode `cluster apply` prints the classified changes list on failure
too, so an operator debugging a partial apply without --json sees what was
attempted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
apply_config_dir executes the query/policy subset of the plan: payloads are
written content-addressed under __cluster/resources/{query,policy}/... before
the state CAS (state is the publish point; orphaned blobs from a failed CAS
are inert and re-apply is the repair), then state.json is CAS-updated with
applied digests, Applied/Blocked statuses, and a revision bump. Graph/schema
changes are never executed here: schema content and graph lifecycle defer to
a later phase with loud warnings, while graph.<id> composite-digest updates
whose schema component is unchanged converge automatically via recomputation
from state's own components (without which apply could never converge).
Idempotent re-apply leaves state bytes and revision untouched.
PlanChange gains optional disposition/reason fields, populated by the same
classifier in cluster plan, so plan is an honest preview of what apply will
execute, derive, defer, or block.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
