diff --git a/AGENTS.md b/AGENTS.md index b335955..87d6a46 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,7 +16,7 @@ Tools that support `@`-imports (Claude Code) auto-include all three files via th `CLAUDE.md` is a symlink to this file — there is exactly one source of truth. Edit `AGENTS.md`. -**Version surveyed:** 0.6.2 +**Version surveyed:** 0.7.0 **Workspace crates:** `omnigraph-compiler`, `omnigraph` (engine), `omnigraph-policy`, `omnigraph-cluster`, `omnigraph-cli`, `omnigraph-server` **Storage substrate:** Lance 6.x (columnar, versioned, branchable) **License:** MIT @@ -33,8 +33,8 @@ OmniGraph is a typed property-graph engine built as a coordination layer over ma - **Multi-modal querying**: vector ANN (`nearest`), full-text (`search`/`fuzzy`/`match_text`/`bm25`), Reciprocal Rank Fusion (`rrf`), and graph traversal (`Expand`, anti-join `not { … }`) in one runtime. - **Branches and commits across the whole graph**: Git-style — every successful publish appends to a commit DAG; merges are three-way at the row level. - **Atomic per-query writes**: `mutate_as` and `load` accumulate insert/update batches into an in-memory `MutationStaging.pending` per touched table; one `stage_*` + `commit_staged` per table runs at end-of-query, then `ManifestBatchPublisher::publish` commits the manifest atomically with per-table `expected_table_versions` CAS. A mid-query failure leaves Lance HEAD untouched on staged tables — no drift, no run state machine, no staging branches. Deletes still inline-commit; D₂ at parse time prevents inserts/updates and deletes from coexisting in one query. -- **HTTP server**: Axum + utoipa OpenAPI, bearer auth (SHA-256 hashed, optional AWS Secrets Manager). Cedar policy enforcement is engine-wide — every `_as` writer calls `Omnigraph::enforce(action, scope, actor)`, so HTTP, CLI, and embedded SDK consumers all hit the same gate. **Two modes** (v0.6.0+): single-graph (legacy flat routes) and multi-graph (`/graphs/{graph_id}/...` cluster routes + read-only `GET /graphs` enumeration). Per-graph + server-level Cedar policies. Runtime add/remove (`POST /graphs`, `DELETE /graphs/{id}`) is not exposed — operators edit `omnigraph.yaml` and restart. -- **CLI** driven by a single `omnigraph.yaml`; multi-format output (json/jsonl/csv/kv/table). +- **HTTP server**: Axum + utoipa OpenAPI, bearer auth (SHA-256 hashed, optional AWS Secrets Manager). Cedar policy enforcement is engine-wide — every `_as` writer calls `Omnigraph::enforce(action, scope, actor)`, so HTTP, CLI, and embedded SDK consumers all hit the same gate. **Two modes** (v0.6.0+): single-graph (legacy flat routes) and multi-graph (`/graphs/{graph_id}/...` cluster routes + read-only `GET /graphs` enumeration). Per-graph + server-level Cedar policies. Multi-graph mode boots from a cluster directory (`--cluster `, RFC-005) or the legacy `omnigraph.yaml` `graphs:` map. Runtime add/remove (`POST /graphs`, `DELETE /graphs/{id}`) is not exposed — operators run `cluster apply` (or edit the legacy file) and restart. +- **CLI** with two-surface config (RFC-008): the team-owned cluster directory (`cluster.yaml`) plus the per-operator `~/.omnigraph/config.yaml` (servers, credentials, actor, aliases). The legacy combined `omnigraph.yaml` still loads with per-key deprecation warnings — `config migrate` proposes the split, `OMNIGRAPH_NO_LEGACY_CONFIG=1` enforces strict mode. **Never extend `omnigraph.yaml`.** Multi-format output (json/jsonl/csv/kv/table). Throughout the docs, capabilities are split into **L1 — Inherited from Lance** vs **L2 — Added by OmniGraph**. @@ -90,7 +90,7 @@ Full diagram and concurrency model: [docs/dev/architecture.md](docs/dev/architec | Cedar policy actions, scopes, CLI | [docs/user/policy.md](docs/user/policy.md) | | HTTP server endpoints, auth, error model, body limits | [docs/user/server.md](docs/user/server.md) | | CLI quick-start | [docs/user/cli.md](docs/user/cli.md) | -| CLI command surface and `omnigraph.yaml` schema | [docs/user/cli-reference.md](docs/user/cli-reference.md) | +| CLI command surface and config schemas (`~/.omnigraph/config.yaml`, legacy `omnigraph.yaml`) | [docs/user/cli-reference.md](docs/user/cli-reference.md) | | Audit / actor tracking | [docs/user/audit.md](docs/user/audit.md) | | Error taxonomy and result serialization | [docs/user/errors.md](docs/user/errors.md) | | Install (binary / Homebrew / source / channels) | [docs/user/install.md](docs/user/install.md) | @@ -258,8 +258,8 @@ omnigraph policy explain --actor act-alice --action change --branch main | Three-way row-level merge | — | `OrderedTableCursor` + `StagedTableWriter`, structured `MergeConflictKind` | | Change feeds | — | `diff_between` / `diff_commits` with manifest fast path + ID streaming | | Cedar policy | — | Per-graph actions plus server-scoped actions (see [docs/user/policy.md](docs/user/policy.md) for the current list), branch / target_branch / protected scopes, validate/test/explain CLI. **Engine-wide enforcement** (MR-722): every `_as` writer (`apply_schema_as`, `mutate_as`, `load_as` — the deprecated `ingest_as` shims route through it — `branch_create_as` / `branch_create_from_as`, `branch_delete_as`, `branch_merge_as`) calls `Omnigraph::enforce(action, scope, actor)` — HTTP, CLI, embedded SDK all hit the same gate. | -| HTTP server | — | Axum, OpenAPI via utoipa, bearer auth (SHA-256, AWS Secrets Manager option), `authorize_request` at the HTTP boundary (resolves bearer→actor, applies admission control), NDJSON streaming export, **multi-graph mode (v0.6.0+) with cluster routes + read-only `GET /graphs` enumeration + per-graph + server-level Cedar policies. Add/remove graphs by editing `omnigraph.yaml` and restarting.** | -| CLI with config | — | `omnigraph.yaml`, aliases, multi-format output (json/jsonl/csv/kv/table) | +| HTTP server | — | Axum, OpenAPI via utoipa, bearer auth (SHA-256, AWS Secrets Manager option), `authorize_request` at the HTTP boundary (resolves bearer→actor, applies admission control), NDJSON streaming export, **multi-graph mode (v0.6.0+) with cluster routes + read-only `GET /graphs` enumeration + per-graph + server-level Cedar policies. Multi-graph boots from a cluster directory (`--cluster`) or the legacy `omnigraph.yaml`; add/remove graphs via `cluster apply` (or by editing the legacy file) and restarting.** | +| CLI with config | — | two-surface config (team `cluster.yaml` dir + per-operator `~/.omnigraph/config.yaml`; legacy `omnigraph.yaml` deprecated per RFC-008), aliases, multi-format output (json/jsonl/csv/kv/table) | | Audit / actor tracking | — | `_as` write APIs + actor map in commit graph | | Local RustFS bootstrap | — | `scripts/local-rustfs-bootstrap.sh` one-shot S3-backed dev environment | diff --git a/Cargo.lock b/Cargo.lock index b1cf0ef..2099055 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4543,7 +4543,7 @@ dependencies = [ [[package]] name = "omnigraph-cli" -version = "0.6.2" +version = "0.7.0" dependencies = [ "assert_cmd", "clap", @@ -4566,7 +4566,7 @@ dependencies = [ [[package]] name = "omnigraph-cluster" -version = "0.6.2" +version = "0.7.0" dependencies = [ "fail", "omnigraph-compiler", @@ -4584,7 +4584,7 @@ dependencies = [ [[package]] name = "omnigraph-compiler" -version = "0.6.2" +version = "0.7.0" dependencies = [ "ahash", "arrow-array", @@ -4605,7 +4605,7 @@ dependencies = [ [[package]] name = "omnigraph-engine" -version = "0.6.2" +version = "0.7.0" dependencies = [ "arc-swap", "arrow-array", @@ -4648,7 +4648,7 @@ dependencies = [ [[package]] name = "omnigraph-policy" -version = "0.6.2" +version = "0.7.0" dependencies = [ "cedar-policy", "clap", @@ -4661,7 +4661,7 @@ dependencies = [ [[package]] name = "omnigraph-server" -version = "0.6.2" +version = "0.7.0" dependencies = [ "arc-swap", "async-trait", diff --git a/crates/omnigraph-cli/Cargo.toml b/crates/omnigraph-cli/Cargo.toml index 901d69c..1670fb2 100644 --- a/crates/omnigraph-cli/Cargo.toml +++ b/crates/omnigraph-cli/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "omnigraph-cli" -version = "0.6.2" +version = "0.7.0" edition = "2024" description = "CLI for the Omnigraph graph database." license = "MIT" @@ -13,11 +13,11 @@ name = "omnigraph" path = "src/main.rs" [dependencies] -omnigraph = { package = "omnigraph-engine", path = "../omnigraph", version = "0.6.2" } -omnigraph-compiler = { path = "../omnigraph-compiler", version = "0.6.2" } -omnigraph-cluster = { path = "../omnigraph-cluster", version = "0.6.2" } -omnigraph-policy = { path = "../omnigraph-policy", version = "0.6.2" } -omnigraph-server = { path = "../omnigraph-server", version = "0.6.2" } +omnigraph = { package = "omnigraph-engine", path = "../omnigraph", version = "0.7.0" } +omnigraph-compiler = { path = "../omnigraph-compiler", version = "0.7.0" } +omnigraph-cluster = { path = "../omnigraph-cluster", version = "0.7.0" } +omnigraph-policy = { path = "../omnigraph-policy", version = "0.7.0" } +omnigraph-server = { path = "../omnigraph-server", version = "0.7.0" } clap = { workspace = true } color-eyre = { workspace = true } serde = { workspace = true } diff --git a/crates/omnigraph-cluster/Cargo.toml b/crates/omnigraph-cluster/Cargo.toml index 973de6d..05a9308 100644 --- a/crates/omnigraph-cluster/Cargo.toml +++ b/crates/omnigraph-cluster/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "omnigraph-cluster" -version = "0.6.2" +version = "0.7.0" edition = "2024" description = "Cluster configuration validation, planning, and config-only apply for Omnigraph." license = "MIT" @@ -14,8 +14,8 @@ documentation = "https://docs.rs/omnigraph-cluster" failpoints = ["dep:fail", "fail/failpoints"] [dependencies] -omnigraph-compiler = { path = "../omnigraph-compiler", version = "0.6.2" } -omnigraph = { package = "omnigraph-engine", path = "../omnigraph", version = "0.6.2" } +omnigraph-compiler = { path = "../omnigraph-compiler", version = "0.7.0" } +omnigraph = { package = "omnigraph-engine", path = "../omnigraph", version = "0.7.0" } fail = { workspace = true, optional = true } serde = { workspace = true } serde_json = { workspace = true } diff --git a/crates/omnigraph-compiler/Cargo.toml b/crates/omnigraph-compiler/Cargo.toml index 8db46e6..bbf03f1 100644 --- a/crates/omnigraph-compiler/Cargo.toml +++ b/crates/omnigraph-compiler/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "omnigraph-compiler" -version = "0.6.2" +version = "0.7.0" edition = "2024" description = "Schema/query compiler for Omnigraph. Zero Lance dependency." license = "MIT" diff --git a/crates/omnigraph-policy/Cargo.toml b/crates/omnigraph-policy/Cargo.toml index 0df2a12..907ce07 100644 --- a/crates/omnigraph-policy/Cargo.toml +++ b/crates/omnigraph-policy/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "omnigraph-policy" -version = "0.6.2" +version = "0.7.0" edition = "2024" description = "Policy / authorization layer for Omnigraph — Cedar-backed PolicyEngine, PolicyChecker trait, ResourceScope enum." license = "MIT" diff --git a/crates/omnigraph-server/Cargo.toml b/crates/omnigraph-server/Cargo.toml index 5393221..614711e 100644 --- a/crates/omnigraph-server/Cargo.toml +++ b/crates/omnigraph-server/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "omnigraph-server" -version = "0.6.2" +version = "0.7.0" edition = "2024" description = "HTTP server for the Omnigraph graph database." license = "MIT" @@ -19,10 +19,10 @@ default = [] aws = ["dep:aws-config", "dep:aws-sdk-secretsmanager"] [dependencies] -omnigraph = { package = "omnigraph-engine", path = "../omnigraph", version = "0.6.2" } -omnigraph-compiler = { path = "../omnigraph-compiler", version = "0.6.2" } -omnigraph-policy = { path = "../omnigraph-policy", version = "0.6.2" } -omnigraph-cluster = { path = "../omnigraph-cluster", version = "0.6.2" } +omnigraph = { package = "omnigraph-engine", path = "../omnigraph", version = "0.7.0" } +omnigraph-compiler = { path = "../omnigraph-compiler", version = "0.7.0" } +omnigraph-policy = { path = "../omnigraph-policy", version = "0.7.0" } +omnigraph-cluster = { path = "../omnigraph-cluster", version = "0.7.0" } axum = { workspace = true } clap = { workspace = true } color-eyre = { workspace = true } diff --git a/crates/omnigraph/Cargo.toml b/crates/omnigraph/Cargo.toml index a4a2fe0..7ee9bda 100644 --- a/crates/omnigraph/Cargo.toml +++ b/crates/omnigraph/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "omnigraph-engine" -version = "0.6.2" +version = "0.7.0" edition = "2024" description = "Runtime engine for the Omnigraph graph database." license = "MIT" @@ -16,8 +16,8 @@ default = [] failpoints = ["dep:fail", "fail/failpoints"] [dependencies] -omnigraph-compiler = { path = "../omnigraph-compiler", version = "0.6.2" } -omnigraph-policy = { path = "../omnigraph-policy", version = "0.6.2" } +omnigraph-compiler = { path = "../omnigraph-compiler", version = "0.7.0" } +omnigraph-policy = { path = "../omnigraph-policy", version = "0.7.0" } lance = { workspace = true } lance-datafusion = { workspace = true } datafusion = { workspace = true } @@ -52,7 +52,7 @@ chrono = { workspace = true } arc-swap = { workspace = true } [dev-dependencies] -omnigraph-compiler = { path = "../omnigraph-compiler", version = "0.6.2" } +omnigraph-compiler = { path = "../omnigraph-compiler", version = "0.7.0" } tokio = { workspace = true } lance-namespace-impls = { workspace = true } serial_test = "3" diff --git a/docs/releases/v0.7.0.md b/docs/releases/v0.7.0.md new file mode 100644 index 0000000..4048041 --- /dev/null +++ b/docs/releases/v0.7.0.md @@ -0,0 +1,90 @@ +# Omnigraph v0.7.0 + +v0.7.0 takes the cluster control plane to object storage and overhauls the +configuration architecture around two single-owner surfaces. A cluster — +state ledger, content-addressed catalog, and graph data — can now live +entirely on an S3-compatible bucket, and a server can boot from that bucket +with no local files at all. Operator identity, credentials, and personal +aliases move to a home-level config; the legacy combined `omnigraph.yaml` +enters a guided, staged deprecation. + +## Highlights + +- **Clusters on object storage (`storage:`).** `cluster.yaml` gains an + optional `storage: s3://bucket/prefix` root. Every stored byte — the + state ledger, lock, recovery sidecars, approval artifacts, catalog blobs, + and the derived graph roots (`/graphs/.omni`) — flows + through one storage layer, so `file://` (the default, byte-compatible + with existing clusters) and `s3://` are a single code path. The ledger's + compare-and-swap uses S3 conditional writes (`If-Match` / + `If-None-Match`), verified against AWS semantics, RustFS, and + Tigris-backed stores; the state lock is genuinely cross-machine on + object storage. +- **Config-free serving: `--cluster s3://bucket/prefix`.** The server + accepts a bare storage-root URI and boots from the applied revision on + the bucket — the ledger and catalog are the whole deployment artifact. + Policy bundles serve as digest-verified *content* from the catalog + (never re-read from disk), closing the last gap for fully remote + clusters. The preferred container shape becomes **bucket, no volume** + (see `docs/user/deployment.md`). +- **Per-operator configuration (`~/.omnigraph/`).** A home-level config + carries operator identity (`operator.actor`, the new last hop of the + `--as` chain), output defaults, named servers, and personal aliases. + `$OMNIGRAPH_HOME` relocates it; `$OMNIGRAPH_CONFIG` now stands in for + `--config` in both binaries. +- **Credentials keyed by server name.** `omnigraph login ` stores + a bearer token in `~/.omnigraph/credentials` (created `0600`; over- + permissive files are refused). Token resolution for a request whose URL + matches an operator-defined server: `OMNIGRAPH_TOKEN_` env → the + credentials file → the legacy `bearer_token_env` chain unchanged. A + token is only ever sent to the server it is keyed to. +- **Operator targeting and aliases.** `--server ` (with `--graph + ` for multi-graph servers) addresses operator-defined endpoints on + every remote-capable command. Operator aliases are pure *bindings* — + personal name → (server, graph, stored-query name, default params) — + invoking catalog-owned stored queries; they carry no query content. +- **`omnigraph.yaml` deprecation begins.** Loading the legacy file prints + a per-key notice naming each present key's new home + (`OMNIGRAPH_SUPPRESS_YAML_DEPRECATION=1` to silence in CI). + `omnigraph config migrate` proposes — and with `--write`, applies — the + split: team half to a ready-to-review `cluster.yaml`, personal half + merged into the operator config (existing entries always win). + `omnigraph init` no longer scaffolds the file. Migrated teams can set + `OMNIGRAPH_NO_LEGACY_CONFIG=1` to turn any legacy-file load into a hard + error. The file itself keeps working until its removal at the next + major version. + +## Breaking / behavior changes + +- `omnigraph init` no longer writes an `omnigraph.yaml` into the working + directory. Start cluster configs from the documentation templates, or + run `omnigraph config migrate` against an existing legacy file. +- Loading a legacy `omnigraph.yaml` now emits a deprecation block on + stderr (suppressible; see above). Output on stdout is unchanged. +- `ServingPolicy` (cluster crate API) carries verified policy *content* + instead of a blob path; `read_serving_snapshot` and several cluster + command entry points are now async. + +## Upgrade notes + +- Existing clusters need no migration: an absent `storage:` key keeps the + config-directory layout byte-for-byte. +- Existing `omnigraph.yaml` setups keep working through the deprecation + window; `omnigraph config migrate` produces the recommended split. +- Operator setup is three lines: + `mkdir -p ~/.omnigraph`, write `operator.actor` (and `servers:`) into + `~/.omnigraph/config.yaml`, then `echo $TOKEN | omnigraph login `. + +## Internals + +- The cluster, server, and CLI crates were modularized (the 7.9k-line + cluster `lib.rs` is now eight focused modules; the server and CLI test + monoliths split into per-area suites) — pure code movement, no behavior + change. +- New gated end-to-end suites run the full cluster lifecycle against a + real S3-compatible store in CI, including a lock-release regression and + a config-free server boot from a bare bucket URI. +- The deployment guide gains the bucket-no-volume container recipe for + AWS and Railway, validated against a live Railway deployment + (Railway buckets are S3-compatible and pass the conditional-write + contract test). diff --git a/openapi.json b/openapi.json index 85c5b8d..6e3dd03 100644 --- a/openapi.json +++ b/openapi.json @@ -7,7 +7,7 @@ "name": "MIT", "identifier": "MIT" }, - "version": "0.6.2" + "version": "0.7.0" }, "paths": { "/branches": {