omnigraph/docs/releases/v0.5.0.md

Omnigraph v0.5.0

Omnigraph v0.5.0 is a substrate, security, and migration-safety release. It jumps the storage substrate from Lance 4 to Lance 6.0.1 (DataFusion 52 → 53, Arrow 57 → 58), introduces engine-wide Cedar policy enforcement on every authoring path, and ships a structured schema-lint v1 chassis with code-tagged diagnostics, soft drops, and an explicit --allow-data-loss flag for destructive migrations.

Highlights

• Lance 6.0.1 substrate: bump from Lance 4.0.0 → 6.0.1, DataFusion 52 → 53, Arrow 57 → 58. New optimizer rules (vectorized IN-list eq kernel, PhysicalExprSimplifier, push-limit-into-hash-join, CASE-NULL shortcut) reach predicates that flow through the engine. lance-tokenizer replaces tantivy internally; FTS behavior preserved.
• Cedar policy engine: a new omnigraph-policy crate wires Omnigraph::enforce(action, scope, actor) into every _as writer (mutate_as, load_as, apply_schema_as, branch_create_as, branch_merge_as, branch_delete_as, plus the load and change variants). The HTTP server defaults to deny-all when no Cedar policy is configured; a YAML policy file is required to enable writes. Actor identity comes only from signed token claims — clients cannot set actor identity directly.
• Schema lint v1 chassis: diagnostics now carry stable codes of the form OG-XXX-NNN instead of free-form messages. omnigraph schema plan and apply understand soft drops on properties and types — destructive drops require the new --allow-data-loss flag (Hard mode) at the CLI and an equivalent JSON flag over HTTP.
• Structured filter pushdown: query-language predicates lower to DataFusion Expr and push down through Lance's Scanner::filter_expr instead of being flattened to SQL strings. This unlocks CompOp::Contains pushdown (via array_has), which previously fell through to in-memory post-scan filtering, and lets the DataFusion 53 optimizer rules above act on our predicates.
• HTTP allow_data_loss parity: the destructive-drop guard now exists on both the CLI (--allow-data-loss) and HTTP (allow_data_loss: true in the schema-apply request body).
• Inline query strings on CLI and HTTP: omnigraph read / omnigraph mutate and the corresponding HTTP endpoints accept inline .gq source, not just a file path. Easier ad-hoc queries, clearer request logs.
• Browser CORS layer: optional CORS layer on omnigraph-server for browser-based UIs, gated by OMNIGRAPH_CORS_ORIGINS.
• Merge-insert dup-rowid fix: Lance's MergeInsertBuilder could surface spurious "Ambiguous merge inserts" errors on sequential merges against rows previously rewritten by merge_insert. The engine now opts into SourceDedupeBehavior::FirstSeen with a check_batch_unique_by_keys fail-fast precondition that guarantees source-side dedup happens before Lance sees the batch.
• Branch-merge error-path recovery: a branch merge that failed mid-flight could leave the in-process coordinator pointing at a stale active branch. The error path now restores the prior coordinator, matching the success path's invariant.
• Branch merge with blob columns: external blob URIs are now materialized correctly during branch merge instead of being dropped or pointing at the source branch.
• Lance API surface guards: a new test file (crates/omnigraph/tests/lance_surface_guards.rs) pins eight specific Lance API surfaces (LanceError::TooMuchWriteContention, ManifestLocation fields, MergeInsertBuilder return shape, WriteParams::default, compact_files signature, etc.) so the next Lance bump fails compile or runtime on any silent drift rather than producing wrong-state recovery in production.

Behavior changes

• On-disk format unchanged: existing v0.4.2 datasets open unchanged. The Lance file format pin stays at V2_2 (required by Lance's blob v2 feature).
• omnigraph-server defaults to deny-all under --policy: starting a server with the policy feature enabled but no Cedar YAML policy configured rejects every write. Operators must supply a policy file to authorize anything.
• Schema-lint diagnostics carry stable codes: messages now lead with OG-XXX-NNN. CI parsers or tooling that keyed off the v0.4.2 free-form text need to switch to code-based matching.
• Destructive schema drops require --allow-data-loss: dropping a property or type returns a structured diagnostic by default. omnigraph schema apply --allow-data-loss (CLI) or {"allow_data_loss": true} (HTTP) opts into Hard mode.
• HashJoinExec null-aware semantics on anti-join: a side effect of the DataFusion 53 bump — NOT IN semantics under null-valued anti-join columns are now correct per SQL standard. Queries that depended on the prior behavior would have been incorrect.

Upgrade Notes

Migration

• No data migration. v0.4.2 repos open directly on v0.5.0.

Clients

• HTTP and SDK clients should switch any string-matching schema-lint parsing to code-based matching against the OG-XXX-NNN prefix.
• Clients exercising destructive schema drops (DropProperty, DropType) must add the allow_data_loss request field (HTTP) or --allow-data-loss flag (CLI). Default is soft-drop-or-reject.
• Clients consuming mutate_as / load_as / apply_schema_as / branch authoring APIs now flow through the policy enforcer. Anything bypassing authorization on v0.4.2 will be rejected on v0.5.0 once a policy is configured.

Operators

• Configure a Cedar policy YAML for production servers before enabling writes; deny-all is the new default. The omnigraph policy validate / test / explain CLI commands are unchanged.
• Bearer tokens continue to be the actor-identity source; review the signed-token-claim-only invariant in docs/dev/invariants.md if you've built custom authentication.
• If your local CI uses RustFS for S3-compatible storage testing, our CI pins rustfs/rustfs:1.0.0-beta.3 (the last known-good tag before the upstream credentials-policy change). Mirror the pin or set RUSTFS_ALLOW_INSECURE_DEFAULT_CREDENTIALS=true for the new image versions.

Tests added or strengthened

• crates/omnigraph/tests/lance_surface_guards.rs — 8 named guards pinning Lance API surfaces against silent drift on future bumps.
• crates/omnigraph/tests/policy_engine_chassis.rs — engine-level policy enforcement coverage; complements the existing HTTP policy tests.
• Policy chassis e2e gap-fills — branch-merge, branch-create, branch-delete policy paths now have explicit end-to-end tests over HTTP and CLI.
• Merge-pair truth table — exhaustive op-variant matrix for three-way merge across noop, addNode, removeNode, addEdge, removeEdge, setProperty, dropProperty, addLabel, removeLabel; the build fails to compile when a new op variant is added without dispositioning every pairing.
• Merge-insert: regression for the dup-rowid bug class on the load surface (load_merge_repeated_against_overlapping_keys_succeeds), the update surface (second_sequential_update_on_same_row_succeeds), and the upstream-Lance-gap canary (load_merge_window_2_documents_upstream_lance_gap).
• Maintenance + destructive-migration coverage — omnigraph optimize / cleanup boundary cases, plus schema-apply soft-drop and Hard-mode paths.
• Stable-row-id preservation across stage_overwrite — pins the invariant that staged overwrites carry stable row IDs through to the committed fragment set.
• CompOp::Contains pushdown regression (ir_filter_with_list_contains_pushes_down) — pins the new structured Expr pushdown path that retired the in-memory fallback.

Included Changes

• Lance 4 → 6.0.1, DataFusion 52 → 53, Arrow 57 → 58 substrate upgrade.
• omnigraph-policy crate with engine-wide Cedar enforcement and signed-token-claim-only actor identity.
• Schema-lint v1 chassis with OG-XXX-NNN codes, soft DropProperty / DropType semantics, and --allow-data-loss for Hard mode.
• HTTP allow_data_loss request field parity with the CLI flag.
• Structured DataFusion Expr filter pushdown via Scanner::filter_expr, with CompOp::Contains lowered through array_has.
• Inline .gq source acceptance on CLI and HTTP read/mutate endpoints.
• Optional CORS layer on omnigraph-server for browser UIs.
• Bug fixes: merge-insert dup-rowid (FirstSeen + uniqueness precondition), branch-merge coordinator restore on error, blob-column materialization during branch merge.
• New Lance API surface-guard test file as the canary for future Lance bumps.
• Recovery-sidecar coverage extended across the four write paths (MutationStaging::finalize, schema_apply, branch_merge, ensure_indices) with failpoint regression tests.
• CI: pinned rustfs/rustfs:1.0.0-beta.3 after the upstream :latest introduced a credentials-policy change.
• Version bump to 0.5.0 across workspace crates, Cargo.lock, openapi.json, and the AGENTS.md surveyed version.