mirror of
https://github.com/ModernRelay/omnigraph.git
synced 2026-06-18 02:24:27 +02:00
Merge remote-tracking branch 'origin/main' into ragnorc/shaping-config-integration
# Conflicts: # crates/omnigraph-cluster/src/lib.rs # crates/omnigraph-cluster/src/serve.rs # crates/omnigraph-server/src/lib.rs # crates/omnigraph-server/src/settings.rs # docs/user/clusters/config.md
This commit is contained in:
aaltshuler
commit
4f8c71fa23
75 changed files with 6557 additions and 6879 deletions
|
|
@ -1,6 +1,6 @@
|
|||
# Architecture
|
||||
|
||||
OmniGraph is a typed property-graph engine built as a coordination layer over many Lance datasets, with Git-style branches and commits across the whole graph, multi-modal querying (vector + FTS + BM25 + RRF + graph traversal) in one runtime, an HTTP server with Cedar policy, and a CLI driven by a single `omnigraph.yaml`.
|
||||
OmniGraph is a typed property-graph engine built as a coordination layer over many Lance datasets, with Git-style branches and commits across the whole graph, multi-modal querying (vector + FTS + BM25 + RRF + graph traversal) in one runtime, an HTTP server with Cedar policy, and a CLI driven by a per-operator `~/.omnigraph/config.yaml` plus team-owned cluster directories.
|
||||
|
||||
## Reading guide
|
||||
|
||||
|
|
|
|||
|
|
@ -15,6 +15,38 @@ Use it this way:
|
|||
- Keep implementation ledgers, roadmap detail, and historical MR notes in the
|
||||
per-area docs. This file is the filter, not the encyclopedia.
|
||||
|
||||
## Governing principle: logical contract over physical state
|
||||
|
||||
The hard invariants below are instances of one rule. Keep it in view whenever
|
||||
a change touches the boundary between what the graph *means* and how it is
|
||||
physically stored.
|
||||
|
||||
> **Logical state is the contract. Physical state — index coverage, fragment
|
||||
> layout, compaction versions, staged writes — is derived, rebuildable, and may
|
||||
> be produced asynchronously. A physical operation must never fail a logical
|
||||
> one. Preconditions are checked against logical state; physical reconciliation
|
||||
> is idempotent and may lag or retry. Genuine logical conflicts still fail
|
||||
> loudly: the licence to lag covers physical convergence, not correctness.**
|
||||
|
||||
Invariants that instantiate it: **2** (manifest-atomic visibility) and **5**
|
||||
(recovery is part of the commit protocol) — a partially-written physical layer
|
||||
never changes what a graph commit means; **7** (indexes are derived state) — a
|
||||
query is correct under partial index coverage, and expensive index work
|
||||
converges from manifest state instead of gating the write path; **13** (failures
|
||||
bounded and observable) — the licence to lag is not a licence to drop, so a
|
||||
physical step that cannot make progress is surfaced, not swallowed. Deny-list
|
||||
items that enforce it: synchronous inline vector/FTS index rebuilds on the
|
||||
commit path; state that drifts from Lance or the manifest when it can be
|
||||
derived; job queues for manifest-derivable state where a reconciler fits.
|
||||
|
||||
The failure shape it rules out: a legitimate background operation on the
|
||||
physical layer (compaction, an index build, an interrupted staged write) is
|
||||
allowed to break a logical operation (a query's correctness, a migration's
|
||||
success, a branch's writability). The smell to watch for is a logical operation
|
||||
whose precondition is a *physical* fact — a cached file version, an index's
|
||||
existence, a fragment count. Make the precondition logical and let a reconciler
|
||||
converge the physical state.
|
||||
|
||||
## Hard Invariants
|
||||
|
||||
1. **Respect the substrate.** Lance owns columnar storage, per-dataset
|
||||
|
|
@ -105,7 +137,7 @@ Use it this way:
|
|||
| Schema validation | Type checks, required fields, defaults, edge endpoint checks, and edge cardinality are enforced on write paths | [schema-language.md](../user/schema/index.md), [execution.md](execution.md) |
|
||||
| Unique constraints | Intra-batch and write-path checks exist; intake and branch-merge derive the composite key through one shared function (`loader::composite_unique_key`, a separator-free `Vec<String>` tuple) and fail loudly on an un-keyable column type rather than silently exempting it; full cross-version uniqueness against already-committed rows is still a gap | [schema-language.md](../user/schema/index.md) |
|
||||
| Storage trait | `TableStorage` (via `db.storage()`) is staged-only; the inline-commit residuals (`delete_where`, `create_vector_index`) are split onto a separate sealed `InlineCommitResidual` trait reached via `db.storage_inline_residual()` (MR-854), so §1 holds by construction; capability/stat surfaces are roadmap | [writes.md](writes.md), [architecture.md](architecture.md) |
|
||||
| Index lifecycle | Index *creation* per `@index`/`@key` property is dispatched by type (enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector) via `node_prop_index_kind`; index *coverage maintenance* exists — `optimize` runs Lance `optimize_indices` after compaction to fold appended/rewritten fragments into existing indexes (still an explicit maintenance call, not yet a background reconciler) | [indexes.md](../user/search/indexes.md), [maintenance.md](../user/operations/maintenance.md) |
|
||||
| Index lifecycle | `@index`/`@key` declares *intent*; the physical index is derived state and never fails a logical op. `schema apply` builds no indexes (records intent only; index-only changes touch no table data). `load`/`mutate` build inline through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector) that fault-isolates an untrainable Vector column into a *pending* index instead of aborting. `optimize`/`ensure_indices` is the reconciler: it creates declared-but-missing indexes and folds appended/rewritten fragments into existing ones (`optimize_indices`), reporting still-pending columns. Explicit maintenance call, not yet a background loop | [indexes.md](../user/search/indexes.md), [maintenance.md](../user/operations/maintenance.md) |
|
||||
| Traversal IDs | Runtime still builds `TypeIndex`; Lance stable row-id based graph IDs are roadmap | [architecture.md](architecture.md), [query-language.md](../user/queries/index.md) |
|
||||
| Auth | Bearer token hashing and server-side actor resolution are implemented at the HTTP boundary | [server.md](../user/operations/server.md), [policy.md](../user/operations/policy.md) |
|
||||
| Tests | Tempdir-backed Lance tests are the current substrate; the storage adapter has an in-memory backend for adapter-level contract tests, but Lance datasets bypass it | [testing.md](testing.md) |
|
||||
|
|
@ -165,6 +197,22 @@ them explicit.
|
|||
one-winner-CAS territory; closing this fully needs a cross-process
|
||||
serialization primitive (e.g. lease-based use of the schema-apply lock
|
||||
branch) — design it before promoting multi-process write topologies.
|
||||
- **Fork reclaim is in-process-safe only:** the first write to a table on a
|
||||
branch forks it (a Lance `create_branch` that advances state before the
|
||||
manifest publish). An interrupted fork (crash, or a cancelled request
|
||||
future) leaves a manifest-unreferenced branch ref. The next write self-heals
|
||||
it — `reclaim_orphaned_fork_and_refork` (`force_delete_branch` + re-fork)
|
||||
— but reclaim is only safe because the writer holds the per-`(table,
|
||||
branch)` write queue from before the fork through the publish AND re-checks
|
||||
the live manifest under it, so no *in-process* writer can be mid-fork. A
|
||||
reclaim cannot serialize against a foreign-*process* in-flight fork: it may
|
||||
force-delete a peer's just-created ref, which makes that peer's commit fail
|
||||
and retry — the same one-winner-CAS exposure as above, not corruption. The
|
||||
reclaim never fires unless in-process-queue + manifest authority both prove
|
||||
the ref is manifest-unreferenced. `cleanup`'s per-table reconciler
|
||||
(`reconcile_orphaned_branches`) is the guaranteed backstop for any fork the
|
||||
write path never revisits. Both degrade to a no-op if Lance ships an atomic
|
||||
multi-dataset branch op.
|
||||
- **Local `write_text_if_match` is not a cross-process CAS:** object-store
|
||||
backends use a true conditional put (ETag If-Match; the in-memory test
|
||||
backend too), but upstream `object_store` leaves `PutMode::Update`
|
||||
|
|
|
|||
|
|
@ -1,6 +1,9 @@
|
|||
# RFC-011: CLI refactoring — one addressing & config model
|
||||
|
||||
**Status:** Proposed
|
||||
**Status:** Accepted — implemented (the `omnigraph.yaml` excision landed as
|
||||
#250/#251/#252; D1–D4, D6, D7, D9, D10 shipped). Two items remain: **D11**
|
||||
(server-side maintenance jobs) is gated on the bulk-data-plane RFC #219; **D5**
|
||||
(combined admin scope) stays deferred by design.
|
||||
**Date:** 2026-06-14
|
||||
**Audience:** CLI/server maintainers
|
||||
**Builds on:** [rfc-007-operator-config.md](rfc-007-operator-config.md)
|
||||
|
|
@ -526,10 +529,9 @@ Non-blocking; settle when convenient.
|
|||
server scope and maintain via `--cluster`. A `deployments: { … }` object
|
||||
(server + cluster validated coherent, referenced by a profile) is revisited only
|
||||
if admin ergonomics demand it — and Decision 11 largely removes the need.
|
||||
- **D8 — the `profile` command surface.** `profile list` / `profile show`
|
||||
(read-only inspection) are additive diagnostics, shippable anytime; they don't
|
||||
touch the grammar or resolution. The *no sticky `profile use`* constraint holds
|
||||
regardless — it is a design principle, not a command.
|
||||
- **D8 — the `profile` command surface.** *Shipped:* `profile list` / `profile
|
||||
show [<name>]` (read-only inspection). The *no sticky `profile use`* constraint
|
||||
holds — it is a design principle, not a command.
|
||||
|
||||
## Safety
|
||||
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ This file is the always-on map of the test surface. **Consult it before every ta
|
|||
| Crate | Path | Style |
|
||||
|---|---|---|
|
||||
| `omnigraph` (engine) | `crates/omnigraph/tests/` | Integration tests (28 files), fixture-driven, share `tests/helpers/mod.rs` |
|
||||
| `omnigraph-cli` | `crates/omnigraph-cli/tests/` | Per-area suites (post-modularization): `cli_cluster.rs` (cluster command surface + operator-actor cascade), `cli_cluster_e2e.rs` (spawned-binary lifecycle compositions — lost-state re-import recovery, out-of-band drift, graph-root destruction, multi-graph mixed-disposition convergence), `cli_data.rs` (load/read/change/branch/commit/export/snapshot/policy/embed/maintenance + operator format cascade), `cli_schema_config.rs` (init/config, schema plan/apply, RFC-008 deprecation warnings + `config migrate` + strict mode), `cli_queries.rs`, `parity_matrix.rs` (RFC-009 Phase 1: the embedded-vs-remote referee — every forked verb run against both arms with matched Cedar policy and the same actor, scrubbed-JSON + exit-code equality; divergences are pinned in its `KNOWN_DIVERGENCES` ledger, never silently repaired), `system_local.rs` (full-cycle cluster lifecycle with a spawned `--cluster` server, applied-policy enforcement over HTTP, keyed-credential auth, operator aliases), `system_remote.rs`; share `tests/support/mod.rs` (hermetic `OMNIGRAPH_HOME` by default) |
|
||||
| `omnigraph-cli` | `crates/omnigraph-cli/tests/` | Per-area suites (post-modularization): `cli_cluster.rs` (cluster command surface + operator-actor cascade), `cli_cluster_e2e.rs` (spawned-binary lifecycle compositions — lost-state re-import recovery, out-of-band drift, graph-root destruction, multi-graph mixed-disposition convergence), `cli_data.rs` (load/read/change/branch/commit/export/snapshot/policy/embed/maintenance + operator format cascade), `cli_schema_config.rs` (init/config, schema plan/apply), `cli_queries.rs`, `parity_matrix.rs` (RFC-009 Phase 1: the embedded-vs-remote referee — every forked verb run against both arms with matched Cedar policy and the same actor, scrubbed-JSON + exit-code equality; divergences are pinned in its `KNOWN_DIVERGENCES` ledger, never silently repaired), `system_local.rs` (full-cycle cluster lifecycle with a spawned `--cluster` server, applied-policy enforcement over HTTP, keyed-credential auth, operator aliases), `system_remote.rs`; share `tests/support/mod.rs` (hermetic `OMNIGRAPH_HOME` by default) |
|
||||
| `omnigraph-cluster` | mostly in-source `#[cfg(test)] mod tests`; `tests/failpoints.rs` (feature-gated); `tests/s3_cluster.rs` (bucket-gated full lifecycle on object storage) | Cluster config parser, local JSON state diff, state CAS/lock handling/recovery, read-only validate/plan/status plus explicit refresh/import graph observations, config-only apply (content-addressed payload publish, disposition gating, composite-digest convergence, idempotent re-apply), catalog payload verification (status read-only, refresh drift + self-heal), failpoint crash-mid-apply / CAS-race coverage, Stage 4A graph creation (create executor, recovery sidecars + sweep rows, create crash windows), Stage 4B schema apply (migration previews in plan, schema executor, schema-apply sweep classification, schema crash windows), Stage 4C gated deletes (digest-bound approvals, delete executor + tombstones, delete sweep rows, delete crash windows), and 5A policy binding metadata (applies_to in the applied revision, binding-change diffing + convergence, pre-5A backfill), and the 5B serving-snapshot read API (converged read, refusal rows) |
|
||||
| `omnigraph-server` | `crates/omnigraph-server/tests/` | Per-area suites (post-modularization): `auth_policy.rs`, `data_routes.rs`, `schema_routes.rs`, `stored_queries.rs`, `multi_graph.rs` (cluster-mode boot — converged serving, policy binding wiring, boot refusals — + the concurrent branch-ops matrix), `boot_settings.rs` (mode inference, PolicySource), `s3.rs` (bucket-gated: single-graph serving + config-free `--cluster s3://` boot), `openapi.rs` (OpenAPI drift / regeneration); share `tests/support/mod.rs` |
|
||||
| `omnigraph-compiler` | mostly in-source `#[cfg(test)] mod tests` | Parser, type-checker, IR lowering, lint |
|
||||
|
|
@ -29,7 +29,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
|
|||
| `point_in_time.rs` | Snapshots, time travel (`snapshot_at_version`, `entity_at`) |
|
||||
| `changes.rs` | `diff_between` / `diff_commits` |
|
||||
| `consistency.rs` | Cross-table snapshot isolation, atomic publish |
|
||||
| `schema_apply.rs` | Migration plan + apply, schema-apply lock |
|
||||
| `schema_apply.rs` | Migration plan + apply, schema-apply lock; index materialization deferred to the reconciler (iss-848): `apply_schema_defers_vector_index_on_empty_table` (an empty-table Vector `@index` never aborts the apply) and `index_only_constraint_apply_touches_no_table_data` (adding an `@index` is metadata-only — no table-version bump) |
|
||||
| `search.rs` | FTS / vector / hybrid (`bm25`, `nearest`, `rrf`) |
|
||||
| `traversal.rs` | `Expand`, variable-length hops, anti-join (CSR path — `OMNIGRAPH_TRAVERSAL_MODE` unset) |
|
||||
| `traversal_indexed.rs` | BTREE-indexed Expand (`execute_expand_indexed`) forced via `OMNIGRAPH_TRAVERSAL_MODE`, asserted semantically equal to the CSR path; own binary, all `#[serial]` so env writes never race |
|
||||
|
|
@ -42,7 +42,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
|
|||
| `lance_version_columns.rs` | Per-row `_row_last_updated_at_version` behavior |
|
||||
| `validators.rs` | Schema constraint enforcement (enum, range, unique, cardinality) across JSONL, insert, update paths |
|
||||
| `policy_engine_chassis.rs` | Engine-layer Cedar enforcement (MR-722): allow + deny through every `_as` writer via the SDK directly — no HTTP — proving embedded and CLI callers hit the same gate as the server, with action × scope shapes matching `authorize_request` |
|
||||
| `maintenance.rs` | `optimize` (compaction), `repair` (explicit uncovered-drift publish), and `cleanup` (version GC): empty/idempotent/no-op edges, policy validation, head preservation; `optimize` publishes its own compaction (`optimize_publishes_compaction_to_manifest_so_schema_apply_succeeds`), skips pre-existing uncovered drift (`optimize_skips_preexisting_manifest_head_drift`), and refuses to run while a `__recovery` sidecar is pending (`optimize_defers_when_recovery_sidecar_is_pending`); `repair` previews/heals verified maintenance drift, refuses raw semantic drift without `--force`, and forced repair publishes only by explicit operator choice |
|
||||
| `maintenance.rs` | `optimize` (compaction), `repair` (explicit uncovered-drift publish), and `cleanup` (version GC): empty/idempotent/no-op edges, policy validation, head preservation; `optimize` publishes its own compaction (`optimize_publishes_compaction_to_manifest_so_schema_apply_succeeds`), skips pre-existing uncovered drift (`optimize_skips_preexisting_manifest_head_drift`), and refuses to run while a `__recovery` sidecar is pending (`optimize_defers_when_recovery_sidecar_is_pending`); `repair` previews/heals verified maintenance drift, refuses raw semantic drift without `--force`, and forced repair publishes only by explicit operator choice; the index reconciler (iss-848): `index_build_tolerates_null_vector_rows` (an untrainable Vector column defers instead of aborting the build, sibling indexes still build) and `optimize_materializes_index_declared_but_unbuilt` (optimize creates a declared-but-deferred index) |
|
||||
| `failpoints.rs` | Failure-injection coverage (gated on `failpoints` feature). Includes the five per-writer Phase B → recovery integration tests (`recovery_rolls_forward_after_finalize_publisher_failure`, `schema_apply_phase_b_failure_recovered_on_next_open`, `branch_merge_phase_b_failure_recovered_on_next_open`, `ensure_indices_phase_b_failure_recovered_on_next_open`, `optimize_phase_b_failure_recovered_on_next_open`) and the write-entry in-process heal contract (the four `*_after_finalize_publisher_failure_heals_without_reopen` tests — load, mutation, schema apply, branch merge: a follow-up write on the same handle rolls a sidecar-covered residual forward without reopen/refresh) and the storage-fault matrix for the sidecar lifecycle (`recovery.sidecar_{write,delete,list}` / `recovery.record_audit` failpoints: Phase A put failure aborts with zero drift, Phase D delete failure is swallowed and healed by the next write, list failures are loud at heal and open, audit-append failures are retried to exactly one audit row; plus the bucket-gated `s3_load_recovers_after_publisher_failure_without_reopen`). |
|
||||
| `recovery.rs` | Open-time recovery sweep — sidecar I/O, classifier dispatch (NoMovement / RolledPastExpected / UnexpectedAtP1 / UnexpectedMultistep / InvariantViolation), all-or-nothing decision, roll-forward via `ManifestBatchPublisher::publish`, roll-back via `Dataset::restore`, audit row in `_graph_commit_recoveries.lance`, `OpenMode::ReadOnly` skip path |
|
||||
| `composite_flow.rs` | Compositional/narrative end-to-end stories — multi-step flows that compose mechanics covered by other test files. Catches integration regressions where individual operations all pass their unit tests but their composition breaks (sequential merges, post-merge main writes, time-travel through merge DAG, reopen consistency over multi-merge histories, post-optimize and post-cleanup strict writes). |
|
||||
|
|
|
|||
|
|
@ -19,8 +19,14 @@ publisher's row-level CAS on `__manifest` is the single fence.
|
|||
`__run__*` branch on an upgraded graph is swept off `__manifest` by the
|
||||
v2→v3 internal-schema migration on first read-write open. (The inert
|
||||
`_graph_runs.lance` bytes remain until a `delete_prefix` primitive lands.)
|
||||
- Cancelled mutation futures leave **no graph-level state** — only orphaned
|
||||
Lance fragments, which the existing `omnigraph cleanup` pipe reclaims.
|
||||
- Cancelled mutation futures leave **no graph-visible state** — the manifest
|
||||
is never advanced. They can leave two kinds of unreferenced residue, both
|
||||
self-healing: orphaned Lance fragments (reclaimed by `omnigraph cleanup`),
|
||||
and — on the *first* write to a table on a branch, which forks it before the
|
||||
publish — a manifest-unreferenced branch ref. The next write to that table
|
||||
reclaims the stale fork and re-forks (`reclaim_orphaned_fork_and_refork`),
|
||||
and `cleanup`'s per-table reconciler is the guaranteed backstop; see the
|
||||
fork-reclaim note in [invariants.md](invariants.md).
|
||||
|
||||
## Read-your-writes within a multi-statement mutation
|
||||
|
||||
|
|
|
|||
|
|
@ -6,34 +6,43 @@
|
|||
omnigraph init --schema schema.pg graph.omni
|
||||
omnigraph load --data data.jsonl --mode overwrite graph.omni
|
||||
omnigraph snapshot graph.omni --branch main --json
|
||||
omnigraph query --uri graph.omni --query queries.gq --name get_person --params '{"name":"Alice"}'
|
||||
omnigraph mutate --uri graph.omni --query queries.gq --name insert_person --params '{"name":"Mina","age":28}'
|
||||
# Invoke a stored query BY NAME from the catalog (served — addressed by scope):
|
||||
omnigraph query get_person --params '{"name":"Alice"}'
|
||||
omnigraph mutate insert_person --params '{"name":"Mina","age":28}'
|
||||
```
|
||||
|
||||
`omnigraph query` is the canonical read command (pairs with `POST /query`);
|
||||
`omnigraph mutate` is the canonical write command (pairs with `POST /mutate`).
|
||||
The previous names `omnigraph read` and `omnigraph change` keep working as
|
||||
visible aliases — invocations emit a one-line deprecation warning to stderr
|
||||
and otherwise behave identically. See [Deprecated names](#deprecated-names)
|
||||
for the migration table.
|
||||
The positional argument is the **stored-query name**, invoked from the served
|
||||
catalog (RFC-011 D3) — the graph is addressed by scope (`--server` / `--profile`
|
||||
/ defaults), and the verb asserts the query's kind (`query` rejects a stored
|
||||
mutation, and vice-versa). The previous names `omnigraph read` and
|
||||
`omnigraph change` keep working as visible aliases — invocations emit a one-line
|
||||
deprecation warning to stderr. See [Deprecated names](#deprecated-names).
|
||||
|
||||
For ad-hoc reads and mutations (REPLs, AI agents, one-off scripts), pass the
|
||||
GQ source inline with `-e` / `--query-string` instead of a file path:
|
||||
For **ad-hoc** reads and mutations (REPLs, AI agents, one-off scripts, local dev),
|
||||
pass the GQ source with `-e` / `--query-string` (inline) or `--query <path>` (a
|
||||
file), and address a graph's storage directly with `--store`. By-name catalog
|
||||
invocation is served-only — a bare `--store` has no catalog, so it's the ad-hoc
|
||||
lane:
|
||||
|
||||
```bash
|
||||
omnigraph query --uri graph.omni \
|
||||
omnigraph query --store graph.omni \
|
||||
-e 'query find($name: String) { match { $p: Person { name: $name } } return { $p.name, $p.age } }' \
|
||||
--params '{"name":"Alice"}'
|
||||
|
||||
omnigraph mutate --uri graph.omni \
|
||||
omnigraph mutate --store graph.omni \
|
||||
-e 'query add($name: String, $age: I32) { insert Person { name: $name, age: $age } }' \
|
||||
--params '{"name":"Inline","age":42}'
|
||||
|
||||
# A multi-query file: the positional selects which query to run.
|
||||
omnigraph query --store graph.omni --query queries.gq get_person --params '{"name":"Alice"}'
|
||||
```
|
||||
|
||||
`-e` is mutually exclusive with `--query <path>` and `--alias <name>`; exactly
|
||||
one of the three must be provided. The inline source travels through the same
|
||||
parser, lint, params binding, and commit machinery as a file-based query —
|
||||
only the source loader changes.
|
||||
`-e` is mutually exclusive with `--query <path>`. With either, the positional
|
||||
name (optional) selects which query in the source to run. The inline source
|
||||
travels through the same parser, lint, params binding, and commit machinery as a
|
||||
file-based query — only the source loader changes.
|
||||
|
||||
## Branching And Reviewable Data Flows
|
||||
|
||||
|
|
@ -50,19 +59,18 @@ omnigraph commit show --uri graph.omni <commit-id> --json
|
|||
|
||||
## Remote Server Mode
|
||||
|
||||
Serve a graph:
|
||||
Serve a cluster-applied graph:
|
||||
|
||||
```bash
|
||||
omnigraph-server graph.omni --bind 127.0.0.1:8080
|
||||
omnigraph cluster apply --config ./company-brain
|
||||
omnigraph-server --cluster ./company-brain --bind 127.0.0.1:8080
|
||||
```
|
||||
|
||||
Read through the HTTP API:
|
||||
Read through the HTTP API — invoke a stored query by name from the catalog:
|
||||
|
||||
```bash
|
||||
omnigraph query \
|
||||
omnigraph query get_person \
|
||||
--server http://127.0.0.1:8080 \
|
||||
--query queries.gq \
|
||||
--name get_person \
|
||||
--params '{"name":"Alice"}'
|
||||
```
|
||||
|
||||
|
|
@ -71,25 +79,31 @@ literal URL); a positional `http(s)://` URI is rejected. If the server requires
|
|||
auth, set its bearer token and `omnigraph login <server>` (or
|
||||
`OMNIGRAPH_BEARER_TOKEN`).
|
||||
|
||||
## Multi-graph servers (v0.6.0+)
|
||||
## Multi-graph servers
|
||||
|
||||
Against a multi-graph server (started with `--config omnigraph.yaml` referencing a non-empty `graphs:` map), use `omnigraph graphs list` to enumerate the registered graphs. The server must configure bearer tokens and `server.policy.file` with a rule that allows `graph_list`; `/graphs` is closed by default even when the server runs with `--unauthenticated`.
|
||||
A server boots from a cluster directory (`omnigraph-server --cluster <dir>`) and
|
||||
serves every graph the cluster declares. Use `omnigraph graphs list` to enumerate
|
||||
them. The cluster's server-level policy must allow `graph_list`; `/graphs` is
|
||||
closed by default even when the server runs with `--unauthenticated`.
|
||||
|
||||
```bash
|
||||
OMNIGRAPH_BEARER_TOKEN=admin-token \
|
||||
omnigraph graphs list --uri http://server.example.com --json
|
||||
omnigraph graphs list --server http://server.example.com --json
|
||||
```
|
||||
|
||||
For config-driven clients, set the remote graph's `bearer_token_env` to an environment variable containing a token whose actor is authorized by `server.policy.file`.
|
||||
For an operator-defined server, store its token with `omnigraph login <name>` (or
|
||||
`OMNIGRAPH_TOKEN_<NAME>`); the actor must be authorized by the cluster's
|
||||
server-level policy.
|
||||
|
||||
`list` rejects local URI targets — it's for remote multi-graph servers only.
|
||||
`list` rejects local (`--store`) targets — it's for remote multi-graph servers only.
|
||||
|
||||
Runtime add/remove is **not** in v0.6.0. To add a graph, stop the server, add a `graphs.<id>` entry to `omnigraph.yaml`, then restart. To remove, stop the server, delete the entry, restart.
|
||||
Runtime add/remove via API is not exposed. To add or remove a graph, edit the
|
||||
cluster's `cluster.yaml`, run `omnigraph cluster apply`, then restart the server.
|
||||
|
||||
Per-graph URLs: hit a graph's cluster route from any subcommand by pointing `--uri` at it:
|
||||
Per-graph addressing: select a graph on a multi-graph server with `--graph`:
|
||||
|
||||
```bash
|
||||
omnigraph read --uri http://server.example.com/graphs/beta --query q.gq ...
|
||||
omnigraph query get_person --server http://server.example.com --graph beta --params '{"name":"Ada"}'
|
||||
```
|
||||
|
||||
## Runs, Policy, And Diagnostics
|
||||
|
|
@ -100,9 +114,9 @@ omnigraph check --query queries.gq graph.omni --json
|
|||
|
||||
omnigraph schema plan --schema next.pg graph.omni --json
|
||||
omnigraph schema apply --schema next.pg graph.omni --json
|
||||
omnigraph policy validate --config omnigraph.yaml
|
||||
omnigraph policy test --config omnigraph.yaml
|
||||
omnigraph policy explain --config omnigraph.yaml --actor act-alice --action read --branch main
|
||||
omnigraph policy validate --cluster ./company-brain --graph knowledge
|
||||
omnigraph policy test --cluster ./company-brain --graph knowledge --tests policy.tests.yaml
|
||||
omnigraph policy explain --cluster ./company-brain --graph knowledge --actor act-alice --action read --branch main
|
||||
|
||||
omnigraph commit list graph.omni --json
|
||||
omnigraph commit show --uri graph.omni <commit-id> --json
|
||||
|
|
@ -116,34 +130,29 @@ also pass `--schema`.
|
|||
|
||||
## Config
|
||||
|
||||
`omnigraph.yaml` lets the CLI and server share named graphs, defaults, and
|
||||
query roots:
|
||||
Configuration has two surfaces with single owners (see the
|
||||
[CLI reference](reference.md#config-surfaces) for the full schema):
|
||||
|
||||
- **`~/.omnigraph/config.yaml`** — your personal operator config: default actor
|
||||
(`--as`), named servers + credentials, clusters, profiles, aliases, and
|
||||
default scope (`defaults.server` / `defaults.store` / `default_graph`). It
|
||||
decides *who you are* and *what you address by default*.
|
||||
- **`cluster.yaml`** (a team-owned cluster directory) — declares *what the system
|
||||
is*: graphs, schemas, stored queries, policies, and storage. A server boots
|
||||
from it (`--cluster <dir>`); see the [cluster guide](../clusters/index.md).
|
||||
|
||||
```yaml
|
||||
graphs:
|
||||
local:
|
||||
uri: demo.omni
|
||||
# ~/.omnigraph/config.yaml
|
||||
operator:
|
||||
actor: act-andrew
|
||||
servers:
|
||||
dev:
|
||||
uri: http://127.0.0.1:8080
|
||||
bearer_token_env: OMNIGRAPH_BEARER_TOKEN
|
||||
|
||||
cli:
|
||||
graph: local
|
||||
branch: main
|
||||
|
||||
query:
|
||||
roots:
|
||||
- queries
|
||||
- .
|
||||
url: http://127.0.0.1:8080
|
||||
defaults:
|
||||
server: dev
|
||||
default_graph: knowledge
|
||||
```
|
||||
|
||||
The config file can also define:
|
||||
|
||||
- server bind defaults
|
||||
- auth env files
|
||||
- query aliases for common read and change commands
|
||||
- `policy.file` for Cedar authorization rules
|
||||
|
||||
When policy is enabled, `schema apply` is authorized through the
|
||||
`schema_apply` action and is typically limited to admins on protected `main`.
|
||||
|
||||
|
|
@ -161,6 +170,6 @@ one-line warning to stderr and otherwise behave identically.
|
|||
| `omnigraph query lint` | `omnigraph lint` | Same flags. The argv-level shim rewrites `query lint` to `lint`. |
|
||||
| `omnigraph query check` | `omnigraph check` | `check` is a visible alias of `omnigraph lint`. |
|
||||
|
||||
The `command:` field in `aliases.<name>` in `omnigraph.yaml` accepts both
|
||||
`read` / `change` (legacy) and `query` / `mutate` (canonical); the two
|
||||
The `command:` field in `aliases.<name>` in `~/.omnigraph/config.yaml` accepts
|
||||
both `read` / `change` (legacy) and `query` / `mutate` (canonical); the two
|
||||
spellings are interchangeable on the wire via serde aliases.
|
||||
|
|
|
|||
|
|
@ -1,31 +1,32 @@
|
|||
# CLI Reference (`omnigraph`)
|
||||
|
||||
A reference for the `omnigraph` binary's command surface and `omnigraph.yaml` schema. For a quick-start guide, see [cli.md](index.md).
|
||||
A reference for the `omnigraph` binary's command surface and the per-operator `~/.omnigraph/config.yaml` schema. For a quick-start guide, see [cli.md](index.md).
|
||||
|
||||
Top-level command families and subcommands. Graph-targeting commands accept a positional `file://`/`s3://` URI, `--server <name|url>` (an operator-defined server from `~/.omnigraph/config.yaml` by name, or a literal `http(s)://` URL, optionally with `--graph <id>` for multi-graph servers; exclusive with a positional URI), `--store <uri>` (a single graph's storage directly), or `--profile <name>` / `$OMNIGRAPH_PROFILE` (a named scope bundle; see [Scopes & profiles](#scopes--profiles-rfc-011)); `cluster` commands use `--config <dir>`. A remote server is addressed only with `--server` — a positional `http(s)://` URI is rejected.
|
||||
Top-level command families and subcommands. Graph-targeting commands accept a positional `file://`/`s3://` URI, `--server <name|url>` (an operator-defined server from `~/.omnigraph/config.yaml` by name, or a literal `http(s)://` URL, optionally with `--graph <id>` for multi-graph servers; exclusive with a positional URI), `--store <uri>` (a single graph's storage directly), or `--profile <name>` / `$OMNIGRAPH_PROFILE` (a named scope bundle; see [Scopes & profiles](#scopes--profiles-rfc-011)); `cluster` commands use `--config <dir>`, while `policy` and `queries` read a cluster's applied state via `--cluster <dir|uri>`. A remote server is addressed only with `--server` — a positional `http(s)://` URI is rejected. **`query`/`mutate` are the exception**: their positional is a stored-query *name* (RFC-011 D3), not a graph URI, so they address the graph only via `--store`/`--server`/`--profile`/defaults.
|
||||
|
||||
## Top-level commands
|
||||
|
||||
| Command | Purpose |
|
||||
|---|---|
|
||||
| `init` | `--schema <pg>` → initialize a graph (no longer scaffolds `omnigraph.yaml`; start cluster configs from the [cluster.md](../clusters/index.md) quick-start or `config migrate`) |
|
||||
| `init` | `--schema <pg>` → initialize a graph (start cluster configs from the [cluster.md](../clusters/index.md) quick-start) |
|
||||
| `load` | bulk load a branch, local or remote (`--mode overwrite\|append\|merge` is **required** — overwrite is destructive, so there is no default). Without `--from` the target branch must exist; `--from <base>` forks a missing `--branch` from `<base>` first |
|
||||
| `ingest` | deprecated alias of `load --from <base>` (defaults: `--from main --mode merge`); prints a one-line warning to stderr |
|
||||
| `query` (alias: `read`) | run named read query; source via `--query <path>`, `-e`/`--query-string <GQ>`, or `--alias <name>` (exactly one). `read` is the deprecated previous name and prints a one-line warning to stderr |
|
||||
| `mutate` (alias: `change`) | run mutation query; same `--query` / `-e` / `--alias` mutual-exclusion as `query`. `change` is the deprecated previous name and prints a one-line warning to stderr |
|
||||
| `query <name>` (alias: `read`) | run a read query. **Catalog lane** (default): `<name>` is a stored query invoked **by name** from the served catalog (served-only — address with `--server`/`--profile`; the verb asserts the query is a read). **Ad-hoc lane**: with `--query <path>` or `-e`/`--query-string <GQ>`, runs that source (the positional `<name>` then selects which query in it). No positional graph URI — address via `--store`/`--server`/`--profile`. `read` is the deprecated previous name (one-line stderr warning) |
|
||||
| `mutate <name>` (alias: `change`) | run a mutation query; same catalog (by-name, served-only, verb asserts mutation) / ad-hoc (`--query`/`-e`) lanes as `query`. `change` is the deprecated previous name (one-line stderr warning) |
|
||||
| `alias <name> [args]` | invoke an operator alias — a read-only personal binding (under `aliases:` in `~/.omnigraph/config.yaml`) to a stored query on a named server (RFC-011 D4; replaces the removed `--alias` flag; stored mutations are rejected before execution) |
|
||||
| `snapshot` | print current snapshot (per-table version + row count) |
|
||||
| `export` | dump to JSONL on stdout (`--type T`, `--table K` filters) |
|
||||
| `branch create \| list \| delete \| merge` | branching ops |
|
||||
| `commit list \| show` | inspect commit graph |
|
||||
| `schema plan \| apply \| show (alias: get)` | migrations |
|
||||
| `schema plan \| apply \| show (alias: get)` | migrations. `apply` refuses a cluster-managed graph (one whose storage is inside a cluster) and points at `cluster apply` — those graphs evolve through the cluster ledger, not a direct apply |
|
||||
| `lint` (alias: `check`) | offline / graph-backed query validation. Replaces `query lint` / `query check`, which are kept as deprecated argv-level shims that print a one-line warning and rewrite to `omnigraph lint` |
|
||||
| `config migrate` | propose (or `--write`: apply) the split of a legacy `omnigraph.yaml` — team half → ready-to-review `cluster.yaml`, personal half → `~/.omnigraph/config.yaml` (key-level merge, existing entries win), plus dropped-key reasons and manual steps |
|
||||
| `cluster validate \| plan \| apply \| approve \| status \| refresh \| import \| force-unlock` | declarative cluster control plane. `validate` checks a local `cluster.yaml` folder and referenced schema/query/policy files; `plan` diffs it against local JSON state at `__cluster/state.json`, annotates dispositions, and embeds real schema-migration previews; `apply` converges the cluster — stored-query/policy catalog writes (content-addressed under `__cluster/resources/`), graph creates, schema updates (soft drops only; `--as` records the actor), and graph deletes behind a digest-bound approval from `cluster approve <resource> --as <actor>` (`apply`/`approve` default the actor from the per-operator `omnigraph.yaml`'s `cli.actor` when `--as` is omitted; nothing else in that file affects cluster commands); what apply converges is what an `omnigraph-server --cluster <dir>` deployment serves on its next restart (omnigraph.yaml deployments are unaffected); `status` reads the state ledger; `refresh`/`import` explicitly update local JSON state from read-only graph observations; `force-unlock <LOCK_ID>` manually removes a held local state lock by exact id |
|
||||
| `cluster validate \| plan \| apply \| approve \| status \| refresh \| import \| force-unlock` | declarative cluster control plane. `validate` checks a local `cluster.yaml` folder and referenced schema/query/policy files; `plan` diffs it against local JSON state at `__cluster/state.json`, annotates dispositions, and embeds real schema-migration previews; `apply` converges the cluster — stored-query/policy catalog writes (content-addressed under `__cluster/resources/`), graph creates, schema updates (soft drops only; `--as` records the actor), and graph deletes behind a digest-bound approval from `cluster approve <resource> --as <actor>` (`apply`/`approve` default the actor from `~/.omnigraph/config.yaml`'s `operator.actor` when `--as` is omitted); what apply converges is what an `omnigraph-server --cluster <dir>` deployment serves on its next restart (`--cluster` is the server's only boot source — RFC-011 cluster-only); `status` reads the state ledger; `refresh`/`import` explicitly update local JSON state from read-only graph observations; `force-unlock <LOCK_ID>` manually removes a held local state lock by exact id |
|
||||
| `optimize` | non-destructive Lance compaction (skips tables with `Blob` columns or uncovered drift; `--json` reports `skipped`) |
|
||||
| `repair [--confirm] [--force]` | preview or explicitly publish uncovered manifest/head drift. `--confirm` heals verified maintenance drift and exits non-zero if suspicious/unverifiable drift is refused; `--force --confirm` publishes suspicious/unverifiable drift after operator review |
|
||||
| `cleanup --keep N --older-than 7d --confirm` | destructive version GC |
|
||||
| `cleanup --keep N --older-than 7d --confirm` | destructive version GC (`--confirm` to execute; also needs `--yes` against a non-local `s3://` target — see *Write diagnostics & destructive confirmation*) |
|
||||
| `embed` | offline JSONL embedding pipeline |
|
||||
| `policy validate \| test \| explain` | Cedar tooling. Selects `cli.graph`, else `server.graph`, else top-level `policy.file` |
|
||||
| `policy validate \| test \| explain` | Cedar tooling against a cluster's applied policies (`--cluster <dir>`; `--graph <id>` picks a graph's bundle when several apply). `test` takes `--tests <file>`; `explain` takes `--actor`/`--action`/`--branch`/`--target-branch` |
|
||||
| `profile list \| show [<name>]` | read-only inspection of `~/.omnigraph/config.yaml` profiles. `list` shows each profile's binding (server/cluster/store) + default graph and marks the `$OMNIGRAPH_PROFILE`-active one; JSON keeps `binding` and adds `scope_kind`, `target`, `valid`, and `error`; `show` resolves one profile's scope (endpoint + default graph), defaulting to the active profile, else the flat operator defaults |
|
||||
| `version` / `-v` | print `omnigraph 0.3.x` |
|
||||
|
||||
## Command capabilities
|
||||
|
|
@ -34,21 +35,30 @@ Every command declares the **capability** it needs — what it requires to reach
|
|||
|
||||
- **`any`** — `query`, `mutate`, `load`, `ingest`, `branch *`, `snapshot`, `export`, `commit *`, `schema show`, `schema apply`. Run against a graph **served (via a server) or embedded (direct against a store)**: accept a positional `file://`/`s3://` URI, `--server <name|url>` (+ `--graph <id>` for multi-graph servers), `--store <uri>`, or `--profile <name>`. A remote server is addressed with `--server` — a positional `http(s)://` URI does **not** dispatch to one.
|
||||
- **`served`** — `graphs list`. Requires a server (accepts `--server` / `--profile`).
|
||||
- **`direct`** — `init`, `optimize`, `repair`, `cleanup`, `schema plan`, `queries validate`, `lint`. Need **direct storage access** (`file://` / `s3://`), never through a server. They accept a positional `URI`, but **not** `--server` / `--graph`, and a remote (`http(s)://`) URI is rejected. `optimize` / `repair` / `cleanup` also accept **`--cluster <dir|s3://…> --cluster-graph <id>`**, which resolves the graph's storage URI from the served cluster state (so you needn't know the `<storage>/graphs/<id>.omni` layout).
|
||||
- **`control`** — `cluster *`. Operates on a cluster directory via `--config <dir>`.
|
||||
- **`local`** — `policy *`, `embed`, `login`, `logout`, `config`, `version`, `queries list`. Address no graph.
|
||||
- **`direct`** — `init`, `optimize`, `repair`, `cleanup`, `schema plan`, `lint`. Need **direct storage access** (`file://` / `s3://`), never through a server. They accept a positional `URI`, but **not** `--server`, and a remote (`http(s)://`) URI is rejected. `optimize` / `repair` / `cleanup` additionally accept **`--cluster <dir|s3://…> --graph <id>`** (`--cluster` is a cluster directory or storage-root URI, named via `clusters:` in `~/.omnigraph/config.yaml` or a literal root), which resolves the graph's storage URI from the served cluster state (so you needn't know the `<storage>/graphs/<id>.omni` layout). `--graph` is the one graph selector across all scopes — on these three verbs it picks the cluster graph; on the other `direct` verbs it does not apply.
|
||||
- **`control`** — `cluster *` via `--config <dir>`; `policy *` and `queries *` via `--cluster <dir|uri>` or a cluster profile.
|
||||
- **`local`** — `alias`, `embed`, `login`, `logout`, `profile`, `version`. Address no explicit graph scope.
|
||||
|
||||
These restrictions are enforced and reported, not silent:
|
||||
|
||||
- A served-graph flag (`--server` / `--graph`) on a verb that doesn't reach a graph through a server fails loudly, e.g.: ``optimize is a direct (storage-native) command; --server/--graph address a served graph and do not apply. Pass a storage URI, or --cluster <dir> --cluster-graph <id>.``
|
||||
- A scope flag on a verb that can't consume it fails loudly rather than being silently dropped — `--server` outside a served scope, `--cluster` outside cluster-scoped verbs, or `--graph` where no multi-graph scope applies, e.g.: ``optimize is a direct (storage-native) command; --server addresses a served graph and does not apply. Pass a storage URI, or --cluster <dir> --graph <id>.``
|
||||
- A `direct` verb pointed at a remote URI fails loudly, e.g.: ``optimize is a direct (storage-native) command and needs direct storage access; the resolved target is a remote server (https://…). Pass the graph's file:// or s3:// URI.``
|
||||
- A data verb pointed at a positional `http(s)://` URI fails loudly: ``a remote graph must be addressed with --server <url> — a positional (or --uri) http(s):// URL no longer dispatches to a server.``
|
||||
- `init` into an **established cluster's** storage layout (`<root>/graphs/<id>.omni` where `<root>` holds `__cluster/state.json`) is refused — graphs in a cluster are created by `cluster apply` (which records ledger / recovery / approvals), not `init`.
|
||||
|
||||
To maintain a server-backed graph, run the `direct` verbs from a host with storage access against the graph's storage URI (a positional URI, or `--cluster … --cluster-graph …`), out-of-band from the serving process — there are no server routes for `optimize` / `repair` / `cleanup` by design.
|
||||
To maintain a server-backed graph, run the `direct` verbs from a host with storage access against the graph's storage URI (a positional URI, or `--cluster … --graph …`), out-of-band from the serving process — there are no server routes for `optimize` / `repair` / `cleanup` by design.
|
||||
|
||||
`omnigraph --help` lists commands with a **capability legend** at the bottom (any / served / direct / control / local).
|
||||
|
||||
## Write diagnostics & destructive confirmation
|
||||
|
||||
Two global flags make writes self-documenting and guard the dangerous ones (RFC-011 Decision 9):
|
||||
|
||||
- **Every write echoes its resolved target to stderr** — `omnigraph load → s3://acme/brain/graphs/knowledge.omni (direct, remote)` — so you catch a scope that resolved somewhere unexpected (e.g. *prod*) before it lands. Applies to `load`, `ingest`, `mutate`, `branch create|delete|merge`, `schema apply`, `optimize`, `repair`, `cleanup`. The line is stderr, so `--json` consumers reading stdout are unaffected; suppress it with **`--quiet`**.
|
||||
- **Destructive writes against a non-local scope require confirmation.** `cleanup`, overwrite `load` (`--mode overwrite`), and `branch delete` proceed freely against a local (`file://`) graph, but when the resolved target is **not local** (a served `http(s)://` graph or an `s3://` store/cluster) they require explicit consent: pass **`--yes`** to confirm, an interactive terminal is prompted, and a non-interactive run (no TTY, or `--json`) **refuses with an error** rather than silently destroying. `cleanup` still also requires its existing `--confirm` (preview→execute); `--yes` is the additional non-local consent.
|
||||
|
||||
A "local" target is a bare path or a `file://` URI; `http(s)://`, `s3://`, and other object-store schemes are non-local.
|
||||
|
||||
## Config surfaces
|
||||
|
||||
Two config surfaces with single owners, plus a zero-config tier:
|
||||
|
|
@ -59,22 +69,20 @@ Two config surfaces with single owners, plus a zero-config tier:
|
|||
| Operator config | one person | `~/.omnigraph/config.yaml` (override dir with `$OMNIGRAPH_HOME`) | who **I** am: identity, ergonomics |
|
||||
| Flags / env | per invocation | — | everything, explicitly |
|
||||
|
||||
`omnigraph.yaml` (below) is the legacy combined file — fully supported
|
||||
today, slated for staged deprecation; its keys' future homes are
|
||||
listed there.
|
||||
|
||||
### `~/.omnigraph/config.yaml` (operator)
|
||||
|
||||
```yaml
|
||||
operator:
|
||||
actor: act-andrew # default identity for every --as cascade:
|
||||
# --as > legacy cli.actor > operator.actor > none
|
||||
actor: act-andrew # default identity for the --as cascade: --as > operator.actor > none
|
||||
servers: # operator-owned endpoints; names key the credentials
|
||||
prod:
|
||||
url: https://graph.example.com # no tokens in this file, ever
|
||||
defaults:
|
||||
output: table # read format default, below --json/--format/alias/legacy
|
||||
server: prod # the everyday scope when no address is given (RFC-011)
|
||||
output: table # read format default, below --json/--format/alias
|
||||
server: prod # the everyday SERVED scope when no address is given (RFC-011)
|
||||
# store: file:///data/dev.omni # OR a zero-flag LOCAL default (mutually
|
||||
# # exclusive with `server`); the local-dev
|
||||
# # counterpart of `server`
|
||||
default_graph: knowledge # graph selected in a server/cluster scope
|
||||
clusters: # admin-only: managed-cluster storage roots (RFC-011).
|
||||
brain: # the ONLY place a storage root lives in this file.
|
||||
|
|
@ -85,8 +93,8 @@ profiles: # named scope bundles (RFC-011); pick with --profile
|
|||
```
|
||||
|
||||
Absent file = empty layer. Unknown keys warn and load (a file written for a
|
||||
newer CLI works on an older one). `$OMNIGRAPH_CONFIG=<path>` stands in for
|
||||
`--config` (the flag wins) in both the CLI and the server.
|
||||
newer CLI works on an older one). Override the config directory with
|
||||
`$OMNIGRAPH_HOME`.
|
||||
|
||||
#### Scopes & profiles (RFC-011)
|
||||
|
||||
|
|
@ -95,20 +103,32 @@ graph in it; the served-vs-direct access path is derived from the scope, not
|
|||
toggled. The scope comes from one of (highest precedence first): an explicit
|
||||
address (a positional URI, `--server`, or `--store <uri>`); a named
|
||||
`--profile <name>` (or `$OMNIGRAPH_PROFILE`); or the flat `defaults.server` +
|
||||
`defaults.default_graph`. A **profile** binds exactly one of `server` / `cluster`
|
||||
/ `store` plus an optional default graph — config data, not state: every command
|
||||
resolves its scope fresh, there is no sticky "current" mode.
|
||||
`defaults.default_graph` (a served default) **or** `defaults.store` (a zero-flag
|
||||
*local* default — mutually exclusive with `defaults.server`). A **profile** binds
|
||||
exactly one of `server` / `cluster` / `store` plus an optional default graph —
|
||||
config data, not state: every command resolves its scope fresh, there is no
|
||||
sticky "current" mode. Inspect what is defined with `omnigraph profile list` and
|
||||
`omnigraph profile show [<name>]` (read-only).
|
||||
|
||||
- `--store <uri>` addresses a single graph's storage directly (ad-hoc / break-glass).
|
||||
- A `cluster`-bound profile reaches `optimize` / `repair` / `cleanup` for a managed
|
||||
graph (resolving its storage root from `clusters:`), the same as
|
||||
`--cluster <root> --cluster-graph <id>`.
|
||||
`--cluster <root> --graph <id>`. A `--graph` flag overrides the profile's default.
|
||||
- A `server`-bound scope on a maintenance verb, or a `cluster`-bound scope on a
|
||||
data verb, is rejected with a message pointing at the right addressing.
|
||||
- **No graph selected (RFC-011 D7).** When a scope has no `--graph` and no
|
||||
`default_graph`, the CLI never silently picks:
|
||||
- **Cluster scope** — exactly **one** applied graph is used automatically;
|
||||
**several** errors and lists the candidates (from the served catalog).
|
||||
- **Server scope** — a multi-graph server (any non-empty `GET /graphs`, even a
|
||||
single entry) errors and lists the candidates: you must pass `--graph <id>`.
|
||||
A single-graph / flat server (405 on `/graphs`), or one whose `/graphs` is
|
||||
policy-gated or unreachable, uses its bare URL as before.
|
||||
|
||||
`--target` and the positional-`http(s)://`→remote dispatch have been **removed**;
|
||||
the remaining legacy surfaces (`--cluster-graph`, `omnigraph.yaml`'s `cli.graph`
|
||||
default) still work and an explicit address always wins.
|
||||
`--target`, `--cluster-graph`, and the positional-`http(s)://`→remote dispatch
|
||||
have been **removed** (`--graph` is now the one graph selector across server and
|
||||
cluster scopes); operator `defaults`/`--profile` supply the no-flag scope and an
|
||||
explicit address always wins.
|
||||
|
||||
#### Credentials keyed by server name
|
||||
|
||||
|
|
@ -136,10 +156,11 @@ aliases:
|
|||
format: table
|
||||
```
|
||||
|
||||
`omnigraph query --alias triage 2026-06-01` invokes
|
||||
`omnigraph alias triage 2026-06-01` invokes
|
||||
`POST <server>/graphs/spike/queries/weekly_triage` with the keyed
|
||||
credential. A legacy `omnigraph.yaml` alias with the same name wins during
|
||||
the deprecation window (with a warning).
|
||||
credential. Aliases live in their own `alias` namespace (RFC-011 Decision 4),
|
||||
so an alias can never shadow — or be shadowed by — a built-in verb. (The old
|
||||
`--alias <name>` flag on `query`/`mutate` was removed.)
|
||||
|
||||
A remote command whose URL prefix-matches an operator server's `url` (the
|
||||
`gh` host model — no flags needed) resolves its token through:
|
||||
|
|
@ -148,64 +169,10 @@ A remote command whose URL prefix-matches an operator server's `url` (the
|
|||
|---|---|
|
||||
| 1 | `OMNIGRAPH_TOKEN_<NAME>` env (`prod` → `OMNIGRAPH_TOKEN_PROD`) |
|
||||
| 2 | `[<name>]` section in `~/.omnigraph/credentials` |
|
||||
| 3 | the legacy chain unchanged (`bearer_token_env` → `OMNIGRAPH_BEARER_TOKEN` → `auth.env_file`) |
|
||||
| 3 | the default `OMNIGRAPH_BEARER_TOKEN` env |
|
||||
|
||||
A token is only ever sent to the server it is keyed to: URLs matching no
|
||||
operator server use the legacy chain alone.
|
||||
|
||||
## `omnigraph.yaml` schema (legacy combined file)
|
||||
|
||||
> **Deprecated.** Loading this file prints a per-key notice
|
||||
> naming each present key's new home (suppress in CI with
|
||||
> `OMNIGRAPH_SUPPRESS_YAML_DEPRECATION=1`); `omnigraph config migrate`
|
||||
> produces the split. The file keeps working through the deprecation
|
||||
> window. Migrated teams can set `OMNIGRAPH_NO_LEGACY_CONFIG=1` to turn
|
||||
> any legacy-file load into a hard error (regression guard; the file's
|
||||
> absence is always fine).
|
||||
|
||||
```yaml
|
||||
project: { name }
|
||||
graphs:
|
||||
<name>:
|
||||
uri: <local|s3://|http(s)://>
|
||||
bearer_token_env: <ENV_NAME>
|
||||
queries: # per-graph stored-query registry (server-role; multi-graph mode)
|
||||
<query-name>: # key MUST equal the `query <name>` symbol inside the .gq
|
||||
file: <path-to-.gq> # relative to this config's directory
|
||||
mcp:
|
||||
expose: true # default true: listed in the MCP catalog (GET /queries); set false to hide (still HTTP-callable)
|
||||
tool_name: <name> # optional MCP tool-name override (defaults to <query-name>;
|
||||
# must be unique across exposed queries)
|
||||
server:
|
||||
graph: <name>
|
||||
bind: <ip:port>
|
||||
cli:
|
||||
graph: <name>
|
||||
branch: <name>
|
||||
output_format: json|jsonl|csv|kv|table
|
||||
table_max_column_width: 80
|
||||
table_cell_layout: truncate|wrap
|
||||
query:
|
||||
roots: [<dir>, …] # search path for .gq files
|
||||
auth:
|
||||
env_file: .env.omni
|
||||
aliases:
|
||||
<alias>:
|
||||
# accepted values: `read` / `query` (read alias), `change` / `mutate`
|
||||
# (write alias). `query` and `mutate` are recommended; `read` and
|
||||
# `change` remain accepted forever for back-compat.
|
||||
command: read|change|query|mutate
|
||||
query: <path-to-.gq>
|
||||
name: <query-name>
|
||||
args: [<positional-name>, …]
|
||||
graph: <name>
|
||||
branch: <name>
|
||||
format: <output-format>
|
||||
queries: # top-level registry — applies only to a bare-URI (anonymous) graph; a graph served by name uses its `graphs.<id>.queries`. Mirrors top-level `policy`.
|
||||
<query-name>: { file: <path-to-.gq> } # mcp.expose defaults to true
|
||||
policy:
|
||||
file: policy.yaml
|
||||
```
|
||||
A keyed token is only ever sent to the server it is keyed to: a URL matching no
|
||||
operator server falls back to `OMNIGRAPH_BEARER_TOKEN` alone.
|
||||
|
||||
## Cluster config preview
|
||||
|
||||
|
|
@ -228,8 +195,8 @@ apply, refresh, and import acquire `__cluster/lock.json` by default and release
|
|||
it before returning. `cluster apply` executes only stored-query/policy catalog
|
||||
writes (content-addressed under `__cluster/resources/`) and requires an
|
||||
existing `state.json`; graph/schema changes are deferred with warnings, and
|
||||
applied resources do not serve traffic — the server still boots from
|
||||
`omnigraph.yaml`. `cluster status` reads state only and reports any existing
|
||||
applied resources do not serve traffic until an `omnigraph-server --cluster
|
||||
<dir>` restart picks them up. `cluster status` reads state only and reports any existing
|
||||
lock metadata. `force-unlock` removes a lock only when the supplied id exactly
|
||||
matches the lock file. `refresh` requires an existing `state.json`; `import`
|
||||
creates one only when it is missing. Both observe declared graphs read-only at
|
||||
|
|
@ -248,7 +215,7 @@ embeddings, aliases, and bindings are reserved for later stages. See
|
|||
|
||||
## Param resolution
|
||||
|
||||
Precedence (high to low): explicit `--params` / `--params-file`, alias positional args, `omnigraph.yaml` defaults. JS-safe-integer handling is built in (`is_js_safe_integer_i64`, `JS_MAX_SAFE_INTEGER_U64`) so 64-bit ids round-trip safely through JSON clients.
|
||||
Precedence (high to low): explicit `--params` / `--params-file`, alias positional args. JS-safe-integer handling is built in (`is_js_safe_integer_i64`, `JS_MAX_SAFE_INTEGER_U64`) so 64-bit ids round-trip safely through JSON clients.
|
||||
|
||||
## Bearer token resolution (CLI)
|
||||
|
||||
|
|
|
|||
|
|
@ -32,26 +32,24 @@ omnigraph cluster force-unlock <LOCK_ID> --config company-brain --json
|
|||
`--config` points at a directory, not a file. The directory must contain
|
||||
`cluster.yaml`. When omitted, it defaults to the current directory.
|
||||
|
||||
## Relationship to `omnigraph.yaml`
|
||||
## Relationship to `~/.omnigraph/config.yaml`
|
||||
|
||||
`cluster.yaml` does not replace `omnigraph.yaml`, and the two never describe
|
||||
the same fact. `omnigraph.yaml` is the permanent **per-operator** layer (CLI
|
||||
defaults, the operator's identity and credential references, graph targets
|
||||
for data-plane commands); `cluster.yaml` is the shared desired state of a
|
||||
`cluster.yaml` and the per-operator `~/.omnigraph/config.yaml` never describe
|
||||
the same fact. The operator config is the permanent **per-operator** layer
|
||||
(the operator's identity and credential references, named servers/clusters,
|
||||
profiles, and CLI defaults); `cluster.yaml` is the shared desired state of a
|
||||
whole deployment, read only by the `cluster` commands via `--config`.
|
||||
|
||||
The exact contract:
|
||||
|
||||
- **Cluster commands read `omnigraph.yaml` for exactly one thing**: the
|
||||
`cli.actor` default used by `apply`/`approve` when `--as` is omitted —
|
||||
operator identity is a per-operator fact. With `--as` present, no config
|
||||
is read at all. Nothing else (its graph set, targets, bind, queries,
|
||||
policies) ever influences a cluster command; a malformed `omnigraph.yaml`
|
||||
breaks only the no-flag actor lookup, loudly.
|
||||
- **A `--cluster` server reads `omnigraph.yaml` for nothing** — not even the
|
||||
implicit current-directory search runs (mode-inference rule 0). Boot from
|
||||
cluster state XOR `omnigraph.yaml`, never a merge.
|
||||
- **The other direction is ergonomics, not coupling**: a per-operator
|
||||
- **Cluster commands read the operator config for exactly one thing**: the
|
||||
`operator.actor` default used by `apply`/`approve` when `--as` is omitted —
|
||||
operator identity is a per-operator fact. With `--as` present, the operator
|
||||
config is not needed. Nothing else in it influences a cluster command.
|
||||
- **No legacy `omnigraph.yaml`**: the CLI does not read `omnigraph.yaml` at
|
||||
all, and a `--cluster` server reads only the cluster catalog — boot is
|
||||
cluster-only.
|
||||
- **The other direction is ergonomics, not coupling**: per-operator
|
||||
data-plane commands address a cluster graph by its derived storage root
|
||||
(`company-brain/graphs/knowledge.omni`) with `--store <uri>` — an ordinary
|
||||
local path, no special handling.
|
||||
|
|
@ -269,12 +267,11 @@ Deletes remove the resource from state; their old payload blobs stay on disk
|
|||
(garbage collection is a later stage). Re-running a converged apply is a no-op:
|
||||
no state write, no revision change (`state_written: false`).
|
||||
|
||||
**Applied means serving — for deployments that opt in.** A server started
|
||||
with `--cluster <dir>` boots from the applied revision (see
|
||||
**Applied means serving.** A server started with `--cluster <dir>` boots from
|
||||
the applied revision (see
|
||||
[Serving from the cluster](#serving-from-the-cluster-the-mode-switch)); it
|
||||
picks up newly applied state on its next restart. Deployments still booting
|
||||
from `omnigraph.yaml` are untouched: for them, applied means recorded in the
|
||||
catalog, nothing more.
|
||||
picks up newly applied state on its next restart. Until that restart, applied
|
||||
means recorded in the catalog, nothing more.
|
||||
|
||||
### Graph creation
|
||||
|
||||
|
|
|
|||
|
|
@ -117,7 +117,7 @@ omnigraph cluster apply --config company-brain --as andrew
|
|||
|
||||
`--as <actor>` attributes the run: it is recorded in recovery sidecars and
|
||||
audit entries and threaded into the engine's commit history. Set
|
||||
`cli: { actor: <you> }` in your per-operator `omnigraph.yaml` to make it the
|
||||
`operator: { actor: <you> }` in your `~/.omnigraph/config.yaml` to make it the
|
||||
default when `--as` is omitted (the flag always wins; `approve` requires one
|
||||
of the two).
|
||||
|
||||
|
|
@ -244,12 +244,12 @@ with an in-flight apply.
|
|||
- **CI-driven convergence**: `validate` and `plan --json` are read-only and
|
||||
safe in pipelines; gate `apply --as ci` on plan review. Approvals are the
|
||||
human step by design — keep `cluster approve` out of automation.
|
||||
- **`omnigraph.yaml` still has a job**: per-operator settings — your
|
||||
`cli.actor` default for `--as`, CLI defaults, credentials, and data-plane
|
||||
ergonomics (address a cluster graph by its derived root like
|
||||
`company-brain/graphs/knowledge.omni` with `--store` for loads). It just no
|
||||
longer describes the deployment — a server boots from one source or the
|
||||
other, never a merge of both.
|
||||
- **`~/.omnigraph/config.yaml` is the per-operator config**: your
|
||||
`operator.actor` default for `--as`, named servers/clusters, credentials,
|
||||
profiles, and data-plane ergonomics (address a cluster graph by its derived
|
||||
root like `company-brain/graphs/knowledge.omni` with `--store` for loads). The
|
||||
cluster directory's `cluster.yaml` is the **sole deployment declaration** — the
|
||||
server boots from the cluster only.
|
||||
|
||||
## 7. Maintaining a cluster graph
|
||||
|
||||
|
|
@ -258,10 +258,11 @@ operation — it runs out-of-band, with direct storage access, against the graph
|
|||
roots. Address a cluster graph by name instead of hand-typing its storage path:
|
||||
|
||||
```bash
|
||||
omnigraph optimize --cluster ./company-brain --cluster-graph knowledge
|
||||
omnigraph cleanup --cluster ./company-brain --cluster-graph knowledge --keep 10 --confirm
|
||||
# --cluster also takes the storage-root URI directly (config-free):
|
||||
omnigraph optimize --cluster s3://bucket/clusters/company-brain --cluster-graph knowledge
|
||||
omnigraph optimize --cluster ./company-brain --graph knowledge
|
||||
omnigraph cleanup --cluster ./company-brain --graph knowledge --keep 10 --confirm
|
||||
# --cluster also takes the storage-root URI directly (config-free), and a
|
||||
# `clusters:` name from ~/.omnigraph/config.yaml:
|
||||
omnigraph optimize --cluster s3://bucket/clusters/company-brain --graph knowledge
|
||||
```
|
||||
|
||||
The graph's storage URI is resolved from the **served cluster state** (the same
|
||||
|
|
@ -270,6 +271,16 @@ not resolvable. Run these from a host with storage access — there are no serve
|
|||
routes for them. Conversely, **`init` refuses** a cluster-managed path: graphs in
|
||||
a cluster are created by `cluster apply`, not by hand.
|
||||
|
||||
If the cluster has exactly **one** applied graph you can omit `--graph` — it is
|
||||
used automatically. With **several**, omitting `--graph` errors and lists the
|
||||
candidates (RFC-011 D7); it never picks one for you.
|
||||
|
||||
Against an **`s3://`-backed cluster** the resolved graph storage is non-local, so a
|
||||
destructive `cleanup` additionally requires **`--yes`** (an interactive prompt
|
||||
otherwise, refusal without a TTY) on top of `--confirm` — see [cli-reference.md](../cli/reference.md)'s
|
||||
*Write diagnostics & destructive confirmation*. Every maintenance run also echoes
|
||||
its resolved target to stderr (suppress with `--quiet`).
|
||||
|
||||
## What the control plane does not do (yet)
|
||||
|
||||
- **No hot reload** — applied changes serve on the next restart.
|
||||
|
|
|
|||
|
|
@ -13,13 +13,10 @@ Omnigraph supports two broad deployment shapes:
|
|||
|
||||
The server binary and container image expose the same HTTP surface.
|
||||
|
||||
The server also has two **boot sources**: `omnigraph.yaml` (graph targets
|
||||
declared in the per-operator config) or a **cluster directory**
|
||||
(`omnigraph-server --cluster <dir>`), which serves the cluster control
|
||||
The server has a single **boot source**: a **cluster directory**
|
||||
(`omnigraph-server --cluster <dir | s3://…>`), which serves the cluster control
|
||||
plane's applied revision — see
|
||||
[cluster-config.md](clusters/config.md#serving-from-the-cluster-the-mode-switch).
|
||||
The two are exclusive per deployment; switching is a restart with a different
|
||||
flag.
|
||||
|
||||
## Binary Deployment
|
||||
|
||||
|
|
@ -30,21 +27,26 @@ Build or install:
|
|||
|
||||
On Windows, the binaries are `omnigraph.exe` and `omnigraph-server.exe`.
|
||||
|
||||
Run against a local graph:
|
||||
The server boots from a cluster only (RFC-011) — there is no positional
|
||||
`<URI>` / single-graph boot. Point it at a local cluster directory:
|
||||
|
||||
```bash
|
||||
omnigraph-server graph.omni --bind 0.0.0.0:8080
|
||||
omnigraph-server --cluster ./company-brain --bind 0.0.0.0:8080
|
||||
```
|
||||
|
||||
Run against an object-store-backed graph:
|
||||
Or boot config-free from an object-storage-rooted cluster:
|
||||
|
||||
```bash
|
||||
OMNIGRAPH_SERVER_BEARER_TOKEN="change-me" \
|
||||
AWS_REGION="us-east-1" \
|
||||
omnigraph-server s3://my-bucket/graphs/example/releases/2026-04-10-v0.1.0 \
|
||||
omnigraph-server --cluster s3://my-bucket/clusters/company-brain \
|
||||
--bind 0.0.0.0:8080
|
||||
```
|
||||
|
||||
The server serves every graph in the cluster's applied revision under
|
||||
`/graphs/{id}/...`. See [clusters](clusters/index.md) for authoring and
|
||||
applying a cluster.
|
||||
|
||||
## Cluster Mode in Containers (AWS, Railway)
|
||||
|
||||
A cluster-booted deployment has **two shapes** since the `storage:` root:
|
||||
|
|
@ -80,10 +82,8 @@ docker run -d \
|
|||
-p 8080:8080 <image>
|
||||
```
|
||||
|
||||
`OMNIGRAPH_CLUSTER` is exclusive: combining it with `OMNIGRAPH_TARGET_URI`,
|
||||
`OMNIGRAPH_CONFIG`, or `OMNIGRAPH_TARGET` fails fast (exit 64), the same
|
||||
rule the server itself enforces. The image also ships the `omnigraph` CLI,
|
||||
so the day-2 loop runs in-container with no `omnigraph.yaml`:
|
||||
`OMNIGRAPH_CLUSTER` is the server's only boot source. The image also
|
||||
ships the `omnigraph` CLI, so the day-2 loop runs in-container:
|
||||
|
||||
```bash
|
||||
docker exec -it <container> sh -c \
|
||||
|
|
@ -104,10 +104,10 @@ docker exec -it <container> sh -c \
|
|||
`omnigraph cluster apply --as <you> --config /var/lib/omnigraph/cluster`
|
||||
→ force a new deployment (restart).
|
||||
|
||||
For a deployment that doesn't need the cluster control plane, the classic
|
||||
stateless shape — `OMNIGRAPH_TARGET_URI=s3://bucket/graph.omni`, no volume —
|
||||
remains the simplest AWS architecture (see Binary/Container Deployment
|
||||
above).
|
||||
For a stateless, volume-free deployment, root the cluster on object
|
||||
storage and boot config-free with
|
||||
`OMNIGRAPH_CLUSTER=s3://bucket/clusters/<name>` (the bucket-no-volume
|
||||
shape above) — the simplest AWS architecture.
|
||||
|
||||
### Railway
|
||||
|
||||
|
|
@ -181,23 +181,24 @@ Build the image:
|
|||
docker build -t omnigraph-server:local .
|
||||
```
|
||||
|
||||
Run against a local graph:
|
||||
The server boots from a cluster only (RFC-011). Run against a cluster
|
||||
directory on a mounted volume:
|
||||
|
||||
```bash
|
||||
docker run --rm -p 8080:8080 \
|
||||
-v "$PWD/graph.omni:/data/graph.omni" \
|
||||
-v "$PWD/company-brain:/var/lib/omnigraph/cluster" \
|
||||
omnigraph-server:local \
|
||||
/data/graph.omni --bind 0.0.0.0:8080
|
||||
--cluster /var/lib/omnigraph/cluster --bind 0.0.0.0:8080
|
||||
```
|
||||
|
||||
Run against an S3-backed graph:
|
||||
Run config-free against an object-storage-rooted cluster:
|
||||
|
||||
```bash
|
||||
docker run --rm -p 8080:8080 \
|
||||
-e OMNIGRAPH_SERVER_BEARER_TOKEN="change-me" \
|
||||
-e AWS_REGION="us-east-1" \
|
||||
omnigraph-server:local \
|
||||
s3://my-bucket/graphs/example/releases/2026-04-10-v0.1.0 \
|
||||
--cluster s3://my-bucket/clusters/company-brain \
|
||||
--bind 0.0.0.0:8080
|
||||
```
|
||||
|
||||
|
|
@ -208,27 +209,14 @@ When no positional args are given, the image entrypoint
|
|||
|
||||
| Var | Effect |
|
||||
|---|---|
|
||||
| `OMNIGRAPH_TARGET_URI` | Graph URI, passed as the positional argument. |
|
||||
| `OMNIGRAPH_CONFIG` | Path to an `omnigraph.yaml`, passed as `--config`. Used to supply a `policy.file` (Cedar authorization). The config file and any relative `policy.file` must be mounted into the container. |
|
||||
| `OMNIGRAPH_TARGET` | Graph name to select from the config's `graphs:` block (with `OMNIGRAPH_CONFIG`, when no `OMNIGRAPH_TARGET_URI`). |
|
||||
| `OMNIGRAPH_CLUSTER` | Cluster boot source — a config directory or a storage-root URI, forwarded as `--cluster`. The only boot source. |
|
||||
| `OMNIGRAPH_BIND` | Listen address (default `0.0.0.0:8080`). |
|
||||
|
||||
`OMNIGRAPH_TARGET_URI` and `OMNIGRAPH_CONFIG` **compose**: set both to keep the
|
||||
graph URI in the env var while loading policy from the config file (the
|
||||
positional URI wins over any `graphs:` entry). To enable Cedar policy on a
|
||||
container otherwise driven by `OMNIGRAPH_TARGET_URI`, mount the config dir and
|
||||
add `OMNIGRAPH_CONFIG`:
|
||||
|
||||
```bash
|
||||
docker run --rm -p 8080:8080 \
|
||||
-e OMNIGRAPH_SERVER_BEARER_TOKEN="change-me" \
|
||||
-e OMNIGRAPH_TARGET_URI="s3://my-bucket/graphs/example/releases/2026-04-10-v0.1.0" \
|
||||
-e OMNIGRAPH_CONFIG="/etc/omnigraph/omnigraph.yaml" \
|
||||
-v "$PWD/config:/etc/omnigraph:ro" \
|
||||
omnigraph-server:local
|
||||
# /etc/omnigraph/omnigraph.yaml contains `policy: { file: policy.yaml }`;
|
||||
# policy.yaml (+ optional policy.tests.yaml) sit beside it in the mount.
|
||||
```
|
||||
Per-graph and server-level Cedar policy come from the cluster's applied
|
||||
revision (authored in `cluster.yaml` and published with `cluster apply`),
|
||||
not from a separate config file. The cluster docker shapes — volume vs.
|
||||
config-free object-storage root — are detailed under
|
||||
[Cluster Mode in Containers](#cluster-mode-in-containers-aws-railway) above.
|
||||
|
||||
## Auth
|
||||
|
||||
|
|
|
|||
|
|
@ -1,17 +1,18 @@
|
|||
# Maintenance: Optimize, Repair & Cleanup
|
||||
|
||||
**Addressing.** `optimize`, `repair`, and `cleanup` are **direct** (storage-native) CLI commands: they run with direct storage access against a positional `file://`/`s3://` URI or **`--cluster <dir|s3://…> --cluster-graph <id>`** (which resolves the graph's storage URI from the served cluster state, so you needn't know the `<storage>/graphs/<id>.omni` layout). They never run through a server, and reject `--server` / `--graph` or a remote (`http(s)://`) URI with a declared error. There are no server routes for them by design — to maintain a server-backed graph, run them out-of-band against the graph's storage URI. See the *Command capabilities* section of [cli-reference.md](../cli/reference.md).
|
||||
**Addressing.** `optimize`, `repair`, and `cleanup` are **direct** (storage-native) CLI commands: they run with direct storage access against a positional `file://`/`s3://` URI or **`--cluster <dir|s3://…> --graph <id>`** (which resolves the graph's storage URI from the served cluster state, so you needn't know the `<storage>/graphs/<id>.omni` layout). They never run through a server, and reject `--server` or a remote (`http(s)://`) URI with a declared error. There are no server routes for them by design — to maintain a server-backed graph, run them out-of-band against the graph's storage URI. See the *Command capabilities* section of [cli-reference.md](../cli/reference.md).
|
||||
|
||||
## `optimize` — non-destructive
|
||||
|
||||
- Compacts every node + edge table on `main`, then reindexes them, then **publishes the resulting version to the `__manifest`** so the manifest's recorded version tracks the compacted-and-reindexed state. Reads pin the manifest version, so without this publish the work would be invisible to readers *and* would break the version precondition of the next schema apply / strict update/delete ("stale view … refresh and retry"). The publish advances the graph version (a system-attributed commit) only for tables that actually changed.
|
||||
- Rewrites small fragments into fewer large ones; old fragments remain reachable via older versions until `cleanup` runs.
|
||||
- **Reindex (index coverage maintenance).** A scalar/FTS/vector index only covers the fragments it was built over. Rows appended after the index was built (e.g. by `load --mode merge`, whose commit does not rebuild an already-existing index) are scanned unindexed, and compaction itself rewrites fragments out of an index's coverage. `optimize` runs Lance's incremental `optimize_indices` after compaction to fold those fragments back in (a delta merge, not a full retrain), restoring full coverage so equality/range/traversal predicates stay index-accelerated. This is why a table with **no compaction work but stale index coverage still commits** a new version under `optimize`. Run `optimize` on a cadence at least as frequent as your freshness window so recently-loaded rows do not linger in the unindexed flat-scan tail.
|
||||
- **Create declared-but-missing indexes (the index reconciler).** `@index`/`@key` declares intent; `schema apply` records it but builds nothing, and `load`/`mutate` defer a column that cannot be built yet (a `Vector` column with no trainable vectors). `optimize` materializes any such declared-but-unbuilt index over the compacted layout — so it is the convergence path for an `@index` added after data exists, or a vector index whose embeddings arrived via a later `embed`. A column still not buildable (no vectors yet) is reported on the table's stat as `pending_indexes` (visible in `--json`), not treated as a failure; the next `optimize` retries. So `optimize` is the single operator-facing index reconciler: it compacts, restores coverage, **and** builds declared-but-missing indexes.
|
||||
- Each table's compact→reindex→publish serializes with concurrent mutations on the same table. A crash mid-operation is recovered automatically on the next open (both compaction and reindex are content-preserving, so roll-forward is always safe).
|
||||
- **Requires a recovered graph.** `optimize` refuses (errors) when a pending crash-recovery operation is present — operating on an unrecovered graph could publish a partial write that recovery would roll back. Reopen the graph to run recovery, then re-run `optimize`.
|
||||
- **Uncovered drift is skipped, not interpreted.** If a table's underlying version is ahead of the version recorded in `__manifest` and no crash-recovery record covers that movement, `optimize` reports `skipped: DriftNeedsRepair` with the manifest/head versions and leaves the table untouched. Run `omnigraph repair` to classify and explicitly publish that drift.
|
||||
- Bounded by `OMNIGRAPH_MAINTENANCE_CONCURRENCY` (default 8).
|
||||
- Returns per-table stats: `table_key, fragments_removed, fragments_added, committed, skipped, manifest_version, lance_head_version`.
|
||||
- Returns per-table stats: `table_key, fragments_removed, fragments_added, committed, skipped, manifest_version, lance_head_version, pending_indexes` (the last lists any declared `@index` column the reconciler could not build this run, with the reason — e.g. a vector column with no trainable vectors yet).
|
||||
- **Blob tables are skipped.** A table that declares any `Blob` property is not compacted: it is reported with `skipped: BlobColumnsUnsupportedByLance` (and logged) instead of compacted, and the rest of the sweep proceeds normally. **Reads and writes are unaffected** — only compaction is. Consequence: fragment count and deleted-row space on blob tables are not reclaimed; query results are never affected. A skipped blob table is also **not reindexed** in the same sweep (the skip happens before the reindex step), so its index coverage on appended rows is not refreshed by `optimize` today.
|
||||
|
||||
## `repair` — explicit
|
||||
|
|
@ -34,6 +35,7 @@
|
|||
backstop, so it does as much as it can and converges on re-run. The CLI reports
|
||||
any failed tables; rerun `cleanup` to retry them.
|
||||
- CLI guards with `--confirm`; without it, prints a preview line.
|
||||
- **Non-local consent (RFC-011 D9).** Against a non-local target (an `s3://` store/cluster), `cleanup` additionally requires `--yes` on top of `--confirm`: a TTY is prompted, and a non-interactive run (no TTY, or `--json`) refuses rather than destroying. A local (`file://`) target needs only `--confirm`. The same `--yes` gate applies to overwrite `load` and `branch delete`; every maintenance run echoes its resolved target to stderr (suppress with `--quiet`).
|
||||
- **Recovery floor:** `--keep < 3` may garbage-collect versions that crash recovery needs as a rollback target. Default `--keep 10` is safe.
|
||||
- **Orphaned-branch reconciliation:** before the version GC, cleanup reclaims any per-table or commit-graph branch absent from the manifest branch list. These orphans arise when a `branch_delete` flips the manifest authority but a downstream best-effort reclaim does not complete (see [branches-commits.md](../branching/index.md)). The reconciler is idempotent (it no-ops once nothing is orphaned), runs regardless of the `keep_versions` / `older_than` values (those gate version GC only), and never reclaims `main` or system-branch forks. Reclaimed forks are logged.
|
||||
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ Server-scoped action (v0.6.0+; binds to `Omnigraph::Server::"root"`):
|
|||
|
||||
10. `graph_list` — `GET /graphs` registry enumeration (multi-graph mode)
|
||||
|
||||
Server-scoped actions cannot use `branch_scope` or `target_branch_scope` — they operate on the registry, not on a graph's branches. A rule cannot mix server-scoped and per-graph actions; split into separate rules. (Runtime `graph_create` / `graph_delete` are reserved but not shipped in v0.6.0; operators add/remove graphs by editing `omnigraph.yaml` and restarting.)
|
||||
Server-scoped actions cannot use `branch_scope` or `target_branch_scope` — they operate on the registry, not on a graph's branches. A rule cannot mix server-scoped and per-graph actions; split into separate rules. (Runtime `graph_create` / `graph_delete` over HTTP are reserved but not shipped; operators add/remove graphs by editing the cluster's `cluster.yaml`, running `omnigraph cluster apply`, and restarting the server.)
|
||||
|
||||
## Scope kinds
|
||||
|
||||
|
|
@ -28,38 +28,34 @@ Server-scoped actions cannot use `branch_scope` or `target_branch_scope` — the
|
|||
- `target_branch_scope` — applied to destination (`schema_apply`, branch ops, run ops)
|
||||
- `protected_branches` — named list with special rules; rule scopes are `any | protected | unprotected`
|
||||
|
||||
## Per-graph vs. server-level policy (multi-graph mode)
|
||||
## Per-graph vs. server-level policy
|
||||
|
||||
In multi mode (`omnigraph.yaml` with a non-empty `graphs:` map), policy files attach at two levels:
|
||||
A server boots from a cluster (`--cluster <dir>`), and the cluster's
|
||||
`cluster.yaml` declares its policy bundles in a `policies:` section. Each bundle
|
||||
names the scopes it `applies_to`: a graph id (per-graph rules — `read`, `change`,
|
||||
`branch_*`, `schema_apply`) or the literal `cluster` (server-level rules —
|
||||
`graph_list`).
|
||||
|
||||
```yaml
|
||||
server:
|
||||
policy:
|
||||
file: server-policy.yaml # server-level: graph_list
|
||||
|
||||
graphs:
|
||||
# cluster.yaml
|
||||
policies:
|
||||
base:
|
||||
file: base.policy.yaml
|
||||
applies_to: [cluster, knowledge] # cluster-level + the `knowledge` graph
|
||||
alpha:
|
||||
uri: s3://tenant-bucket/alpha
|
||||
policy:
|
||||
file: policies/alpha.yaml # per-graph: read, change, branch_*, schema_apply
|
||||
beta:
|
||||
uri: s3://tenant-bucket/beta
|
||||
# no per-graph policy → no engine-layer Cedar enforcement on beta
|
||||
file: policies/alpha.yaml
|
||||
applies_to: [alpha] # per-graph: alpha only
|
||||
```
|
||||
|
||||
**Config follows graph identity, not server mode.** A graph served by **name**
|
||||
(`--target <name>` or `server.graph`) uses its own `graphs.<name>.policy.file`,
|
||||
exactly as in multi-graph mode. Top-level `policy.file` applies only to an
|
||||
**anonymous** graph — one served by a bare `<URI>` with no `graphs:` entry.
|
||||
Serving a **named** graph (single- or multi-graph mode) while top-level
|
||||
`policy.file` (or `queries:`) is populated **refuses boot**, naming the block,
|
||||
since the top-level value would otherwise be silently shadowed by the per-graph
|
||||
block. Move per-graph rules to `graphs.<graph_id>.policy.file` and `graph_list`
|
||||
rules to `server.policy.file`.
|
||||
A graph with no bundle bound to it has no engine-layer Cedar enforcement. Each
|
||||
graph's HTTP request flows through its bound bundle; the management endpoint
|
||||
(`GET /graphs`) flows through the `cluster`-scoped bundle. When no bundle binds
|
||||
`cluster`, `GET /graphs` is denied in every runtime state, including
|
||||
`--unauthenticated`; with bearer tokens configured it returns 403 after admission
|
||||
control because `graph_list` is not a `read`-equivalent action. The operator must
|
||||
bind a `cluster`-scoped bundle granting `graph_list` to expose `/graphs`.
|
||||
|
||||
Each graph's HTTP request flows through its own per-graph policy. The management endpoint (`GET /graphs`) flows through the server-level policy. When `server.policy.file` is unset, `GET /graphs` is denied in every runtime state, including `--unauthenticated`; with bearer tokens configured, it returns 403 after admission control because `graph_list` is not a `read`-equivalent action. The operator must explicitly authorize via `server-policy.yaml` to expose `/graphs`.
|
||||
|
||||
Example server-level policy:
|
||||
Example `cluster`-scoped bundle:
|
||||
|
||||
```yaml
|
||||
version: 1
|
||||
|
|
@ -72,38 +68,26 @@ rules:
|
|||
actions: [graph_list]
|
||||
```
|
||||
|
||||
## Configuration
|
||||
Each per-graph rule may use at most one of `branch_scope` or
|
||||
`target_branch_scope`. Server-scoped rules (`graph_list`) take neither — they
|
||||
have no branch context.
|
||||
|
||||
`omnigraph.yaml`:
|
||||
## Actor for direct-engine writes
|
||||
|
||||
```yaml
|
||||
policy:
|
||||
file: policy.yaml # Cedar rules + groups
|
||||
tests: policy.tests.yaml # declarative test cases
|
||||
|
||||
cli:
|
||||
actor: act-andrew # default actor for CLI direct-engine writes
|
||||
```
|
||||
|
||||
Each per-graph rule may use at most one of `branch_scope` or `target_branch_scope`. Server-scoped rules (`graph_list`) take neither — they have no branch context.
|
||||
|
||||
`cli.actor` is the default actor identity for CLI direct-engine writes
|
||||
when `policy.file` is configured. Override per-invocation with `--as
|
||||
<ACTOR>` (top-level flag) — `--as` wins, otherwise `cli.actor` is used,
|
||||
otherwise no actor. With policy configured and neither set, the
|
||||
engine-layer footgun guard intentionally denies the write (silent bypass
|
||||
via "I forgot the actor" is exactly what the guard prevents). Remote
|
||||
HTTP writes ignore both — they resolve their actor server-side from the
|
||||
bearer token.
|
||||
The default actor identity for CLI direct-engine (`--store`) writes is
|
||||
`operator.actor` in `~/.omnigraph/config.yaml`. Override per-invocation with
|
||||
`--as <ACTOR>` — `--as` wins, otherwise `operator.actor`, otherwise no actor.
|
||||
Remote HTTP writes ignore both — they resolve their actor server-side from the
|
||||
bearer token. (Direct-store access carries no Cedar policy under RFC-011; policy
|
||||
lives in the cluster/server.)
|
||||
|
||||
## CLI
|
||||
|
||||
Policy tooling resolves its graph like server single-mode policy: `cli.graph`
|
||||
wins, otherwise `server.graph` is used, otherwise the top-level `policy.file`
|
||||
is validated/tested/explained as the anonymous policy.
|
||||
Policy tooling reads a cluster's applied policy bundles: pass `--cluster <dir>`,
|
||||
and `--graph <id>` to pick a graph's bundle when several apply.
|
||||
|
||||
- `omnigraph policy validate` — parse + count actors, exit 1 on parse error.
|
||||
- `omnigraph policy test` — run cases in `policy.tests.yaml`, exit 1 on any expectation mismatch.
|
||||
- `omnigraph policy test --tests <file>` — run the declarative cases in `<file>` against the selected bundle, exit 1 on any expectation mismatch.
|
||||
- `omnigraph policy explain --actor … --action … [--branch …] [--target-branch …]` — show decision and matched rule.
|
||||
- `omnigraph --as <ACTOR> <subcommand>` — set the actor for the duration of one invocation. Effective for `change`, `load` (and its deprecated `ingest` alias), `branch create|delete|merge`, and `schema apply` against a direct (`--store`) graph. **Rejected** on a served write (`--server`): the actor is bearer-token-resolved server-side, so `--as` can't set it there.
|
||||
|
||||
|
|
@ -132,7 +116,7 @@ reaches the authorization gate without a matching policy permit.
|
|||
|---|---|---|---|
|
||||
| **Open** | no | no | Every request is permitted. Refuses to start unless `--unauthenticated` or `OMNIGRAPH_UNAUTHENTICATED=1` is set — the operator must explicitly opt in. |
|
||||
| **DefaultDeny** | yes | no | Every authenticated request for an action other than `read` is rejected with HTTP 403. Closes the "tokens but forgot the policy file" trap — an operator who sets up auth and forgot to point at a policy file used to ship the illusion of protection. |
|
||||
| **PolicyEnabled** | yes | yes | Authenticated requests that reach a configured policy engine are evaluated by Cedar. Server-scoped actions still require `server.policy.file`. |
|
||||
| **PolicyEnabled** | yes | yes | Authenticated requests that reach a configured policy engine are evaluated by Cedar. Server-scoped actions still require a `cluster`-scoped policy bundle. |
|
||||
|
||||
The server refuses to start for the "no tokens, no policy, no flag" cell
|
||||
and for "policy file, no tokens" — instead of silently shipping an open
|
||||
|
|
|
|||
|
|
@ -1,38 +1,29 @@
|
|||
# HTTP Server (`omnigraph-server`)
|
||||
|
||||
Axum 0.8 + tokio + utoipa-generated OpenAPI. **Two modes** (v0.6.0+): single-graph and multi-graph, with **two boot sources** for multi mode: `omnigraph.yaml` or — exclusively — a cluster directory (`--cluster`). Mode is inferred from CLI args + config shape.
|
||||
Axum 0.8 + tokio + utoipa-generated OpenAPI. **Cluster-only boot** (RFC-011): the server always boots from a cluster (`--cluster <dir | s3://…>`) and serves N graphs (N ≥ 1) under cluster routes. There is no longer a single-graph flat-route mode, no positional `<URI>` boot, no `--target`, and no `omnigraph.yaml`-`graphs:`-map boot. All HTTP is nested under `/graphs/{graph_id}/...`; `/healthz` and the management `/graphs` enumeration stay flat.
|
||||
|
||||
## Modes
|
||||
## Boot
|
||||
|
||||
### Single-graph mode
|
||||
### Cluster boot (the only boot)
|
||||
|
||||
`omnigraph-server <URI>` or `omnigraph-server --target <name> --config omnigraph.yaml`. Routes are flat — `/snapshot`, `/read`, `/branches`, etc.
|
||||
```bash
|
||||
omnigraph-server --cluster <dir | s3://…> --bind 0.0.0.0:8080
|
||||
```
|
||||
|
||||
**Config follows graph identity.** A bare `<URI>` is an *anonymous* graph and uses the **top-level** `policy.file` / `queries:`. A graph chosen by **name** (`--target` / `server.graph`) uses its own `graphs.<name>.{policy.file, queries}` — the same block multi-graph mode uses. ⚠️ *Changed from v0.6.0, which always used top-level config in single mode: a named-graph config that puts `policy`/`queries` at top-level now **refuses boot** and points you at `graphs.<name>.…` (move the block there). Bare-`<URI>` single mode is unchanged.*
|
||||
|
||||
### Multi-graph mode (v0.6.0+)
|
||||
|
||||
`omnigraph-server --config omnigraph.yaml` with a non-empty `graphs:` map and **no** single-mode selector (no `server.graph`, no `<URI>`, no `--target`). The server opens every configured graph in parallel at startup (bounded concurrency = 4, fail-fast on the first open error). Routes are nested under `/graphs/{graph_id}/...`. Bare flat paths return 404 in multi mode.
|
||||
|
||||
### Cluster-booted multi mode
|
||||
|
||||
`omnigraph-server --cluster <dir-or-uri>` boots from the cluster catalog's **applied
|
||||
revision** instead of
|
||||
`omnigraph.yaml` — an exclusive boot source: combining it with `<URI>`,
|
||||
`--target`, or `--config` is a startup error, and `omnigraph.yaml` is never
|
||||
read in this mode. Always multi-graph routing. See
|
||||
`omnigraph-server --cluster <dir-or-uri>` boots from the cluster catalog's
|
||||
**applied revision**. The server resolves that revision into per-graph
|
||||
startup configs (id, URI, optional per-graph policy, stored-query
|
||||
registry) plus an optional server-level policy, then opens every
|
||||
configured graph in parallel at startup (bounded concurrency = 4,
|
||||
fail-fast on the first open error). Routing is always multi-graph —
|
||||
requests to bare flat protected paths (`/read`, `/snapshot`, …) return
|
||||
404; the served surface is `/graphs/{graph_id}/...`. See
|
||||
[cluster-config.md](../clusters/config.md#serving-from-the-cluster-the-mode-switch)
|
||||
for what is read and the fail-fast readiness rules. `--bind`,
|
||||
`--unauthenticated`, and the bearer-token env vars work identically.
|
||||
for what is read and the fail-fast readiness rules.
|
||||
|
||||
Mode inference:
|
||||
|
||||
0. CLI `--cluster <dir | s3://…>` → **multi, cluster-booted** (exclusive; a scheme-qualified argument reads the ledger straight from the storage root, no local config)
|
||||
1. CLI positional `<URI>` → single
|
||||
2. CLI `--target <name>` → single
|
||||
3. `server.graph` in config → single
|
||||
4. `--config` + non-empty `graphs:` + no single-mode selector → **multi**
|
||||
5. otherwise → error with migration hint
|
||||
A scheme-qualified argument (`s3://…`) reads the ledger straight from the
|
||||
storage root, with no local config directory. `--bind`,
|
||||
`--unauthenticated`, and the bearer-token env vars all apply.
|
||||
|
||||
### Stored-query validation at startup
|
||||
|
||||
|
|
@ -40,36 +31,37 @@ If a graph declares a `queries:` registry (see [cli-reference](../cli/reference.
|
|||
|
||||
## Endpoint inventory
|
||||
|
||||
Per-graph endpoints — same body shape across modes; URLs differ:
|
||||
|
||||
| Method | Single-mode path | Multi-mode path | Auth | Action |
|
||||
|---|---|---|---|---|
|
||||
| GET | `/healthz` | `/healthz` | none | — |
|
||||
| GET | `/openapi.json` | `/openapi.json` | none | — (strips security if auth disabled; in multi mode emits cluster paths with `cluster_` operation-id prefix) |
|
||||
| GET | `/snapshot?branch=` | `/graphs/{id}/snapshot?branch=` | bearer + `read` | snapshot of branch |
|
||||
| POST | `/query` | `/graphs/{id}/query` | bearer + `read` | inline read query (canonical; clean field names `query`/`name`; mutations → 400) |
|
||||
| POST | `/read` | `/graphs/{id}/read` | bearer + `read` | **deprecated** alias of `/query` (legacy field names `query_source`/`query_name`, byte-stable response; carries `Deprecation: true` + `Link: </query>; rel="successor-version"`) |
|
||||
| POST | `/export` | `/graphs/{id}/export` | bearer + `export` | NDJSON stream |
|
||||
| POST | `/mutate` | `/graphs/{id}/mutate` | bearer + `change` | mutation (canonical; `query`/`name`; accepts legacy `query_source`/`query_name` as serde aliases) |
|
||||
| POST | `/change` | `/graphs/{id}/change` | bearer + `change` | **deprecated** alias of `/mutate` (carries `Deprecation: true` + `Link: </mutate>; rel="successor-version"`) |
|
||||
| GET | `/queries` | `/graphs/{id}/queries` | bearer + `read` | list the `mcp.expose` stored queries as a typed tool catalog |
|
||||
| POST | `/queries/{name}` | `/graphs/{id}/queries/{name}` | bearer + `invoke_query` (+ `change` for a stored mutation) | invoke a named query from the `queries:` registry; deny == 404 |
|
||||
| GET | `/schema` | `/graphs/{id}/schema` | bearer + `read` | get current `.pg` source |
|
||||
| POST | `/schema/apply` | `/graphs/{id}/schema/apply` | bearer + `schema_apply` (target=`main`) | migrate |
|
||||
| POST | `/load` | `/graphs/{id}/load` | bearer + `branch_create` (only when `from` is set and the branch is created) + `change` | bulk load (canonical); branch creation is opt-in via `from` — without it a missing `branch` is a 404, never an implicit fork (32 MB body limit) |
|
||||
| POST | `/ingest` | `/graphs/{id}/ingest` | bearer + `branch_create` (only when `from` is set and the branch is created) + `change` | **deprecated** alias of `/load` (carries `Deprecation: true` + `Link: </load>; rel="successor-version"`) (32 MB body limit) |
|
||||
| GET | `/branches` | `/graphs/{id}/branches` | bearer + `read` | list branches |
|
||||
| POST | `/branches` | `/graphs/{id}/branches` | bearer + `branch_create` | create |
|
||||
| DELETE | `/branches/{branch}` | `/graphs/{id}/branches/{branch}` | bearer + `branch_delete` | delete |
|
||||
| POST | `/branches/merge` | `/graphs/{id}/branches/merge` | bearer + `branch_merge` | merge `source → target` |
|
||||
| GET | `/commits?branch=` | `/graphs/{id}/commits?branch=` | bearer + `read` | list |
|
||||
| GET | `/commits/{commit_id}` | `/graphs/{id}/commits/{commit_id}` | bearer + `read` | show |
|
||||
|
||||
Server-level management endpoints (v0.6.0+):
|
||||
Per-graph endpoints — all nested under `/graphs/{id}/...`. `{id}` is the
|
||||
graph id from the cluster's applied revision:
|
||||
|
||||
| Method | Path | Auth | Action |
|
||||
|---|---|---|---|
|
||||
| GET | `/graphs` | bearer + `graph_list` on `Server::"root"` | list registered graphs (405 in single mode) |
|
||||
| GET | `/healthz` | none | — |
|
||||
| GET | `/openapi.json` | none | — (strips security if auth disabled; emits the nested cluster paths with `cluster_` operation-id prefix) |
|
||||
| GET | `/graphs/{id}/snapshot?branch=` | bearer + `read` | snapshot of branch |
|
||||
| POST | `/graphs/{id}/query` | bearer + `read` | inline read query (canonical; clean field names `query`/`name`; mutations → 400) |
|
||||
| POST | `/graphs/{id}/read` | bearer + `read` | **deprecated** alias of `/query` (legacy field names `query_source`/`query_name`, byte-stable response; carries `Deprecation: true` + `Link: <query>; rel="successor-version"`) |
|
||||
| POST | `/graphs/{id}/export` | bearer + `export` | NDJSON stream |
|
||||
| POST | `/graphs/{id}/mutate` | bearer + `change` | mutation (canonical; `query`/`name`; accepts legacy `query_source`/`query_name` as serde aliases) |
|
||||
| POST | `/graphs/{id}/change` | bearer + `change` | **deprecated** alias of `/mutate` (carries `Deprecation: true` + `Link: <mutate>; rel="successor-version"`) |
|
||||
| GET | `/graphs/{id}/queries` | bearer + `read` | list the `mcp.expose` stored queries as a typed tool catalog |
|
||||
| POST | `/graphs/{id}/queries/{name}` | bearer + `invoke_query` (+ `change` for a stored mutation) | invoke a named query from the `queries:` registry; deny == 404 |
|
||||
| GET | `/graphs/{id}/schema` | bearer + `read` | get current `.pg` source |
|
||||
| POST | `/graphs/{id}/schema/apply` | bearer + `schema_apply` (target=`main`) | disabled for cluster-backed serving; returns 409 and points operators at `omnigraph cluster apply` + restart |
|
||||
| POST | `/graphs/{id}/load` | bearer + `branch_create` (only when `from` is set and the branch is created) + `change` | bulk load (canonical); branch creation is opt-in via `from` — without it a missing `branch` is a 404, never an implicit fork (32 MB body limit) |
|
||||
| POST | `/graphs/{id}/ingest` | bearer + `branch_create` (only when `from` is set and the branch is created) + `change` | **deprecated** alias of `/load` (carries `Deprecation: true` + `Link: <load>; rel="successor-version"`) (32 MB body limit) |
|
||||
| GET | `/graphs/{id}/branches` | bearer + `read` | list branches |
|
||||
| POST | `/graphs/{id}/branches` | bearer + `branch_create` | create |
|
||||
| DELETE | `/graphs/{id}/branches/{branch}` | bearer + `branch_delete` | delete |
|
||||
| POST | `/graphs/{id}/branches/merge` | bearer + `branch_merge` | merge `source → target` |
|
||||
| GET | `/graphs/{id}/commits?branch=` | bearer + `read` | list |
|
||||
| GET | `/graphs/{id}/commits/{commit_id}` | bearer + `read` | show |
|
||||
|
||||
Server-level management endpoints:
|
||||
|
||||
| Method | Path | Auth | Action |
|
||||
|---|---|---|---|
|
||||
| GET | `/graphs` | bearer + `graph_list` on `Server::"root"` | list registered graphs |
|
||||
|
||||
### Stored-query catalog (`GET /queries`)
|
||||
|
||||
|
|
@ -88,13 +80,14 @@ Invoke a curated, server-side stored query by **name** — the source comes from
|
|||
- **Requires an explicit policy grant when auth is on.** In default-deny mode (bearer tokens but no `policy.file`), only `read` is permitted, so *every* `/queries/{name}` call returns `404` until an `invoke_query` rule is configured.
|
||||
- A stored mutation cannot target a `snapshot` (`400`); a parameter type error is a structured `400` naming the parameter.
|
||||
|
||||
## Adding and removing graphs (multi mode)
|
||||
## Adding and removing graphs
|
||||
|
||||
Runtime add/remove via API is **not** exposed in v0.6.0 — neither
|
||||
`POST /graphs` nor `DELETE /graphs/{id}` is implemented. Operators add
|
||||
or remove graphs by stopping the server, editing the `graphs:` map in
|
||||
`omnigraph.yaml`, then restarting. The server treats `omnigraph.yaml`
|
||||
as operator-owned configuration and never writes it.
|
||||
Runtime add/remove via API is **not** exposed — neither `POST /graphs`
|
||||
nor `DELETE /graphs/{id}` is implemented. Operators add or remove graphs
|
||||
by running `cluster apply` against the cluster (which publishes a new
|
||||
applied revision) and restarting the server so it boots from the new
|
||||
revision. The server treats the cluster source as operator-owned and
|
||||
never writes it.
|
||||
|
||||
A future release may introduce a managed registry and re-expose runtime
|
||||
mutation on top of it.
|
||||
|
|
@ -138,8 +131,8 @@ channels:
|
|||
- **Response headers (RFC 9745)**: every response carries `Deprecation: true`.
|
||||
- **Response headers (RFC 8288)**: every response carries a `Link` header
|
||||
pointing at the canonical successor:
|
||||
`Link: </query>; rel="successor-version"` for `/read`, and
|
||||
`Link: </mutate>; rel="successor-version"` for `/change`. SDKs and HTTP
|
||||
`Link: <query>; rel="successor-version"` for `/read`, and
|
||||
`Link: <mutate>; rel="successor-version"` for `/change`. SDKs and HTTP
|
||||
proxies can pick the successor up automatically.
|
||||
|
||||
Migration is purely cosmetic on the client side — swap the URL path, leave
|
||||
|
|
@ -226,4 +219,4 @@ See [deployment.md](../deployment.md) for token-source operational details.
|
|||
admission control" above). No global rate limiter is configured;
|
||||
add `tower_http::limit` if a graph-wide cap is needed.
|
||||
- Pagination — none (commits/branches return everything; export streams).
|
||||
- Runtime graph add/remove — edit `omnigraph.yaml` and restart.
|
||||
- Runtime graph add/remove — run `cluster apply` and restart.
|
||||
|
|
|
|||
|
|
@ -27,7 +27,8 @@ list/`Blob` columns → none.
|
|||
|
||||
## L2 — OmniGraph orchestration
|
||||
|
||||
- `ensure_indices()` / `ensure_indices_on(branch)` — idempotent build of BTREE + inverted indexes for the current head; safe to re-run.
|
||||
- **`@index`/`@key` declares intent; the physical index is derived state.** A migration records the declaration in the catalog/IR and never fails on it — `schema apply` builds **no** indexes (adding an `@index` to an existing column is a pure metadata change that touches no table data). `load`/`mutate` build declared indexes inline as part of the write, but a column that can't be built yet (a `Vector` column with no trainable vectors — IVF k-means needs ≥1 vector, e.g. rows loaded before `embed` runs) is left **pending**, not fatal. Reads stay correct meanwhile: a missing/partial index degrades to a scan (vector search to brute-force). A later `ensure_indices`/`optimize` materializes the pending index once it is buildable. This mirrors how LanceDB builds indexes asynchronously and serves unindexed rows by brute-force.
|
||||
- `ensure_indices()` / `ensure_indices_on(branch)` — idempotent build of BTREE + inverted + vector indexes for the current head; safe to re-run; returns the columns it had to defer as pending. `optimize` runs it after compaction, so the maintenance cron is the convergence path for deferred indexes.
|
||||
- Indexes are built on the *branch head* (not on a snapshot), so reads always see the current index state.
|
||||
- **Lazy branch forking for indexes**: a branch that hasn't mutated a sub-table doesn't need its own index — the main lineage's index is reused until the first write triggers a copy-on-write fork.
|
||||
- Vector index parameters (metric, nlist, nprobe, etc.) are not exposed in the schema; they default at the Lance layer and are picked up automatically when an index is asked for on a Vector column.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue