* fix(cluster): stop cluster-apply crash-loops from the recovery-sidecar trap
A `cluster apply` carrying a schema change against a graph that has
non-main branches, or an unsupported "needs backfill" migration, armed a
recovery sidecar *before* calling the engine, then left it behind when the
engine rejected the apply pre-movement. The server refuses to boot while
any sidecar is pending, and re-running apply re-armed a fresh sidecar — an
unescapable crash loop. None of the engine rejections are bugs; the trap
is in the apply/serve choreography.
Three coordinated changes:
1. Preview before arming the sidecar. `cluster apply` now runs
`preview_schema_apply_with_options` before `write_recovery_sidecar`, so
parser/planner rejections (non-main branches, unsupported plan) fail
loudly without leaving recovery work behind. The post-preview engine
error path now deletes the sidecar when the live schema still matches
the recorded digest (nothing moved), and keeps it only on real
mid-movement failure — both branches covered by new engine-failpoint
tests (cluster failpoints now enable omnigraph/failpoints).
2. Per-graph quarantine at serve time instead of whole-cluster refusal.
A graph-attributed pending sidecar, an unopenable graph root, a query
parse failure, or an unresolvable embedding provider now quarantines
just that graph (logged loudly at every boot layer) while healthy
graphs serve; `/graphs` lists only ready graphs and quarantined routes
404. Cluster-global problems (missing/unreadable state, malformed or
unattributable sidecars, shared-catalog or cluster-policy errors, zero
healthy graphs) stay fail-fast. `--require-all-graphs` /
OMNIGRAPH_REQUIRE_ALL_GRAPHS=1 restores all-or-nothing boot.
3. Backfill embedding-provider profile metadata on apply. Mirrors the
existing policy-binding backfill: a pre-5A ledger missing
`embedding_profile` is now detected as a metadata-only change and
backfilled by a no-op apply, instead of bricking serve with
`embedding_provider_profile_missing` forever.
Tests: trap (no sidecar after a rejected apply), both digest-cleanup
branches, per-graph quarantine (cluster + server), embedding backfill.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: resilient cluster boot + recovery-sidecar trap fix
Amend RFC-005 D4 readiness posture (cluster-global fail-fast vs graph-local
quarantine; deviation #5 for --require-all-graphs), add the v0.7.0 release
note, and update the user cluster/server/deployment docs and the
OMNIGRAPH_REQUIRE_ALL_GRAPHS env var.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cluster): surface sidecar-cleanup failures; document severity promotion
Address Greptile review on PR #284:
- The pre-movement sidecar cleanup fast-path discarded `delete_object`'s
result, so a transient delete failure left the graph quarantined with no
signal. Add `try_delete_object` (Result-returning) and emit a
`recovery_sidecar_cleanup_failed` warning diagnostic on failure; the
fire-and-forget `delete_object` now delegates to it.
- Document why the serve-time loop promotes every `list_recovery_sidecars`
diagnostic to a cluster-fatal error (the listing only emits genuine
read/parse/version failures, as warnings, whose blast radius serving
cannot prove) and note the promote-by-code path if that ever changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reconcile the v0.7.0 release notes with what 0.7.0 actually ships. The draft was
mid-cycle; two facts changed and a late-cycle arc was missing.
- omnigraph.yaml is REMOVED (not deprecated): drop the deprecation-window
framing (config migrate, OMNIGRAPH_NO_LEGACY_CONFIG, OMNIGRAPH_SUPPRESS_YAML_
DEPRECATION); the two-surface config (cluster.yaml + ~/.omnigraph/config.yaml)
is the only config.
- Cluster-only server: the server boots only from --cluster; no single-graph
flat-route / positional-URI / omnigraph.yaml-graphs boot. Deprecated-route Link
headers are the sibling-relative form (<load>, not </load>).
- Add the RFC-011 tail: defaults.store, profile list/show, schema-apply refusal
(CLI signpost + server 409), read-only aliases, the any/served/direct/control/
local capability vocabulary, removed legacy data-plane addressing.
- New "Engine & substrate" section: Lance 6→7, indexed traversal, scalar-index/
query-latency, index-materialization-deferred, recovery liveness, branch-fork
self-heal, composite @unique.
- New "Embeddings (RFC-012)" section + breaking bullet: provider-independent
client (OpenRouter default), @embed same-space validation, the default-provider
flip (OMNIGRAPH_EMBED_PROVIDER=gemini for Gemini-direct; OMNIGRAPH_GEMINI_BASE_URL
dropped).
- Upgrade notes: replace the false "omnigraph.yaml keeps working / config migrate"
guidance with the manual cluster.yaml + operator-config path; add server
--cluster and embeddings notes.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
All six crate manifests + their path-dependency constraints, Cargo.lock,
the regenerated openapi.json version metadata, AGENTS.md's surveyed
version, and the v0.7.0 release notes (object-storage clusters,
config-free --cluster serving, the operator config surface, keyed
credentials, operator targeting/aliases, and the omnigraph.yaml
deprecation stages).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
