Patch release over v0.7.0: three correctness fixes (#283 camelCase filters,
#284 cluster-apply crash-loop, #277 branch-merge OOM on embedding tables), the
#280 queries-list catalog-metadata improvement, and the #268 warm-read perf
fix. No breaking changes, no on-disk format change, no migration.
Version coherence: all 7 crate manifests + path-dep constraints, Cargo.lock,
openapi.json, and AGENTS.md surveyed version bumped 0.7.0 -> 0.7.1. Full
workspace gate green (1472 tests).
Claude-Session: https://claude.ai/code/session_01FQ1Hf4eXLsJmeLUkTYBEw7
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cluster): stop cluster-apply crash-loops from the recovery-sidecar trap
A `cluster apply` carrying a schema change against a graph that has
non-main branches, or an unsupported "needs backfill" migration, armed a
recovery sidecar *before* calling the engine, then left it behind when the
engine rejected the apply pre-movement. The server refuses to boot while
any sidecar is pending, and re-running apply re-armed a fresh sidecar — an
unescapable crash loop. None of the engine rejections are bugs; the trap
is in the apply/serve choreography.
Three coordinated changes:
1. Preview before arming the sidecar. `cluster apply` now runs
`preview_schema_apply_with_options` before `write_recovery_sidecar`, so
parser/planner rejections (non-main branches, unsupported plan) fail
loudly without leaving recovery work behind. The post-preview engine
error path now deletes the sidecar when the live schema still matches
the recorded digest (nothing moved), and keeps it only on real
mid-movement failure — both branches covered by new engine-failpoint
tests (cluster failpoints now enable omnigraph/failpoints).
2. Per-graph quarantine at serve time instead of whole-cluster refusal.
A graph-attributed pending sidecar, an unopenable graph root, a query
parse failure, or an unresolvable embedding provider now quarantines
just that graph (logged loudly at every boot layer) while healthy
graphs serve; `/graphs` lists only ready graphs and quarantined routes
404. Cluster-global problems (missing/unreadable state, malformed or
unattributable sidecars, shared-catalog or cluster-policy errors, zero
healthy graphs) stay fail-fast. `--require-all-graphs` /
OMNIGRAPH_REQUIRE_ALL_GRAPHS=1 restores all-or-nothing boot.
3. Backfill embedding-provider profile metadata on apply. Mirrors the
existing policy-binding backfill: a pre-5A ledger missing
`embedding_profile` is now detected as a metadata-only change and
backfilled by a no-op apply, instead of bricking serve with
`embedding_provider_profile_missing` forever.
Tests: trap (no sidecar after a rejected apply), both digest-cleanup
branches, per-graph quarantine (cluster + server), embedding backfill.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: resilient cluster boot + recovery-sidecar trap fix
Amend RFC-005 D4 readiness posture (cluster-global fail-fast vs graph-local
quarantine; deviation #5 for --require-all-graphs), add the v0.7.0 release
note, and update the user cluster/server/deployment docs and the
OMNIGRAPH_REQUIRE_ALL_GRAPHS env var.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cluster): surface sidecar-cleanup failures; document severity promotion
Address Greptile review on PR #284:
- The pre-movement sidecar cleanup fast-path discarded `delete_object`'s
result, so a transient delete failure left the graph quarantined with no
signal. Add `try_delete_object` (Result-returning) and emit a
`recovery_sidecar_cleanup_failed` warning diagnostic on failure; the
fire-and-forget `delete_object` now delegates to it.
- Document why the serve-time loop promotes every `list_recovery_sidecars`
diagnostic to a cluster-fatal error (the listing only emits genuine
read/parse/version failures, as warnings, whose blast radius serving
cannot prove) and note the promote-by-code path if that ever changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
All six crate manifests + their path-dependency constraints, Cargo.lock,
the regenerated openapi.json version metadata, AGENTS.md's surveyed
version, and the v0.7.0 release notes (object-storage clusters,
config-free --cluster serving, the operator config surface, keyed
credentials, operator targeting/aliases, and the omnigraph.yaml
deprecation stages).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LocalStateBackend becomes ClusterStore: every stored byte — state ledger,
lock, recovery sidecars, approval artifacts — now flows through the
engine's StorageAdapter, making file:// and s3:// one code path. Behavior
on the file backend is byte-compatible (layout, CAS semantics, diagnostics,
lock release timing) and the entire pre-existing suite passes unchanged.
Mechanics: the ledger CAS keeps its public sha256 vocabulary while the
physical swap is token-conditioned (ETag If-Match on S3 via PR #186's
primitives; content-token + temp/rename locally — the pre-port semantics);
the lock is a create-only put (genuinely cross-machine on object stores)
with deterministic drop-release locally and best-effort spawned release on
S3; sidecars/approvals address by URI (SweepOutcome and the executors carry
strings); sweep row-1 retirement joins the uniform deferred post-CAS
cleanup. ClusterStore also gains the catalog-payload and graph-root
methods that commit 2 wires in.
Async ripple: status/force-unlock/serving-snapshot and the server's
settings loader chain go async (CLI dispatch and ~20 test hosts follow,
mechanically). tokio joins the cluster crate's runtime deps for the lock
guard's handle.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Optional failpoints feature (dep:fail + fail/failpoints, deliberately NOT
enabling omnigraph/failpoints), a maybe_fail/ScopedFailPoint module returning
Diagnostic-typed injected errors, and two call sites in apply_config_dir:
cluster_apply.after_payload_phase (the crash point: blobs on disk, state
untouched) and cluster_apply.before_state_write (routes through the
persisted-statuses revert contract; a cfg_callback here can mutate state.json
to make the CAS check fail organically). Feature off compiles to Ok(()) —
zero behavior change. Tests live in a separate integration binary because the
fail registry is process-global. Also refresh the crate description (stale
'read-only' since Stage 3A).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
