mirror of
https://github.com/ModernRelay/omnigraph.git
synced 2026-06-24 02:38:06 +02:00
docs(user): restructure user docs into topic sections (Phase 1) (#223)
Move the 23 flat docs/user/*.md files into topic subdirectories so the user guide is organized by area (schema, queries, search, branching, cli, operations, clusters, concepts, reference) instead of a flat list. This is a pure structural move — whole files relocated, every cross-doc link recomputed, no prose rewrites or content splits (those follow in Phase 2). - 19 `git mv`s (install.md, deployment.md stay top-level); history preserved (renames detected at 92–100% similarity). - All intra-doc links, AGENTS.md's topic table (52 pointers), and the docs/dev + docs/releases back-links recomputed via relpath from each file's new location. - docs/user/index.md rewritten as a sectioned nav hub. - Fixed 5 doc-path references in Rust (comments + two user-facing server settings error strings) to point at the new locations. Verified: zero broken .md links across tracked docs; check-agents-md.sh green (with the untracked scratch docs set aside); touched crates build. Note: the public site (omnigraph-web) imports docs/ via a flat-only script; its import-docs.mjs needs a subdir-aware update before the next re-sync. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
8726ca92ec
commit
d46e50dd6d
33 changed files with 126 additions and 109 deletions
7
docs/user/operations/audit.md
Normal file
7
docs/user/operations/audit.md
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
# Audit / Actor tracking
|
||||
|
||||
- `Omnigraph::audit_actor_id: Option<String>` is the actor in effect.
|
||||
- `_as` variants of every write API let callers override the actor: `mutate_as`, `load_as`, `branch_merge_as`, `apply_schema_as`, etc.
|
||||
- Actor IDs are persisted on `GraphCommit.actor_id` with split storage in `_graph_commit_actors.lance` (the commit graph is split into `_graph_commits.lance` for the linkage and `_graph_commit_actors.lance` for the actor map).
|
||||
- HTTP server uses the bearer-token actor automatically. The CLI resolves one actor chain everywhere: `--as` > legacy `cli.actor` in `omnigraph.yaml` > `operator.actor` in `~/.omnigraph/config.yaml` > none (RFC-007).
|
||||
- Pre-v0.4.0 graphs also stored actor IDs on `RunRecord.actor_id` in `_graph_runs.lance` / `_graph_run_actors.lance`. The Run state machine was removed in MR-771; those files are inert post-v0.4.0. The v2→v3 manifest migration sweeps any stale `__run__*` branches on first write-open (MR-770); the inert dataset bytes remain until a `delete_prefix` primitive lands.
|
||||
24
docs/user/operations/errors.md
Normal file
24
docs/user/operations/errors.md
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
# Errors and Result Serialization
|
||||
|
||||
## Error taxonomy (`omnigraph::error::OmniError`)
|
||||
|
||||
- `Compiler(...)` — schema/query parse/typecheck errors
|
||||
- `Lance(String)` — storage layer
|
||||
- `DataFusion(String)` — execution layer
|
||||
- `Io(io::Error)`
|
||||
- `Manifest(ManifestError { kind: BadRequest|NotFound|Conflict|Internal, details: Option<ManifestConflictDetails>, … })`
|
||||
- `ManifestConflictDetails::ExpectedVersionMismatch { table_key, expected, actual }` — caller's `expected_table_versions` did not match the manifest's current latest non-tombstoned version (set by `OmniError::manifest_expected_version_mismatch`).
|
||||
- `ManifestConflictDetails::RowLevelCasContention` — Lance row-level CAS rejected the publish because a concurrent writer landed the same `object_id`. Retried internally by the publisher; only surfaces if the retry budget exhausts.
|
||||
- **D₂ parse-time rejection** (MR-794): a single mutation query that mixes inserts/updates with deletes errors out *before any I/O* with kind `BadRequest`. Message: `mutation '<name>' on the same query mixes inserts/updates and deletes; split into separate mutations: (1) inserts and updates, then (2) deletes`. See [docs/user/query-language.md](../queries/index.md) for the rule and [docs/dev/writes.md](../../dev/writes.md) for the underlying staged-write rationale.
|
||||
- `MergeConflicts(Vec<MergeConflict>)`
|
||||
|
||||
Compiler-side `NanoError` covers parse / catalog / type / storage / plan / execution / arrow / lance / IO / manifest / unique-constraint, each with structured spans (`SourceSpan { start, end }`) for ariadne-style diagnostics.
|
||||
|
||||
## Result serialization (`omnigraph_compiler::result::QueryResult`)
|
||||
|
||||
- `to_arrow_ipc()` — efficient binary
|
||||
- `to_sdk_json()` — JS-safe JSON (large i64 wrapped in metadata)
|
||||
- `to_rust_json()` — Rust-friendly JSON
|
||||
- `batches()` — direct Arrow `RecordBatch` access
|
||||
|
||||
Mutation results: `{ affectedNodes: usize, affectedEdges: usize }` (also exposed as a tiny Arrow batch).
|
||||
49
docs/user/operations/maintenance.md
Normal file
49
docs/user/operations/maintenance.md
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
# Maintenance: Optimize, Repair & Cleanup
|
||||
|
||||
`db/omnigraph/optimize.rs` and `db/omnigraph/repair.rs`.
|
||||
|
||||
**Addressing (RFC-010).** `optimize`, `repair`, and `cleanup` are **storage-plane** CLI commands: they run with direct storage access against a positional `URI`, `--target`, or **`--cluster <dir|s3://…> --cluster-graph <id>`** (which resolves the graph's storage URI from the served cluster state, so you needn't know the `<storage>/graphs/<id>.omni` layout). They never run through a server, and reject `--server` / `--graph` or a `--target` that resolves to a remote (`http(s)://`) URL with a declared error. There are no server routes for them by design — to maintain a server-backed graph, run them out-of-band against the graph's storage URI. See the *Command planes* section of [cli-reference.md](../cli/reference.md).
|
||||
|
||||
## `optimize_all_tables(db)` — non-destructive
|
||||
|
||||
- Lance `compact_files()` on every node + edge table on `main`, then **publishes the compacted version to the `__manifest`** so the manifest's `table_version` tracks the compacted Lance HEAD. Reads pin the manifest version, so without this publish compaction would be invisible to readers *and* would break the HEAD-vs-manifest precondition of the next schema apply / strict update/delete ("stale view … refresh and retry"). The publish advances the graph version (a system-attributed commit) only for tables that actually compacted.
|
||||
- Rewrites small fragments into fewer large ones; old fragments remain reachable via older manifests until `cleanup` runs.
|
||||
- Each table's compact→publish runs under its per-`(table, main)` write queue (serializing with concurrent mutations — compaction is a Lance `Rewrite` op that retryable-conflicts with a concurrent merge/update/delete on overlapping fragments). The Lance-HEAD-before-manifest-publish gap is covered by a `SidecarKind::Optimize` recovery sidecar (loose-match): a crash in that window rolls the compacted version forward on the next `Omnigraph::open` (compaction is content-preserving, so roll-forward is always safe).
|
||||
- **Requires a recovered graph.** `optimize` refuses (errors) when an unresolved recovery sidecar is present under `__recovery` — operating on an unrecovered graph could publish a partial write the open-time recovery sweep would roll back. Reopen the graph to run the recovery sweep, then re-run `optimize`.
|
||||
- **Uncovered drift is skipped, not interpreted.** If a table's Lance HEAD is ahead of the version recorded in `__manifest` and no recovery sidecar covers that movement, `optimize` reports `skipped: Some(DriftNeedsRepair)` with the manifest/head versions and leaves the table untouched. Run `omnigraph repair` to classify and explicitly publish that drift.
|
||||
- Bounded by `OMNIGRAPH_MAINTENANCE_CONCURRENCY` (default 8).
|
||||
- Returns `[TableOptimizeStats { table_key, fragments_removed, fragments_added, committed, skipped, manifest_version, lance_head_version }]`.
|
||||
- **Blob tables are skipped.** A table that declares any `Blob` property is not compacted: it is reported with `skipped: Some(BlobColumnsUnsupportedByLance)` (and logged via `tracing::warn`) instead of compacted, and the rest of the sweep proceeds normally. The current Lance `compact_files` mis-decodes blob-v2 columns under its forced `BlobHandling::AllBinary` read; **reads and writes are unaffected** — only compaction is. This is gated by `LANCE_SUPPORTS_BLOB_COMPACTION` (`db/omnigraph/optimize.rs`) and removed when the upstream Lance fix lands (see [docs/dev/lance.md](../../dev/lance.md)). Consequence: fragment count and deleted-row space on blob tables are not reclaimed until then; query results are never affected.
|
||||
|
||||
## `repair_all_tables(db, options)` — explicit
|
||||
|
||||
- Handles **uncovered manifest/head drift**: a table's Lance HEAD is ahead of the manifest pin and no recovery sidecar records the writer intent.
|
||||
- Preview by default. `omnigraph repair --json <uri>` reports each table's `classification`, `action`, manifest/head versions, Lance operation names, and any classification error. `--confirm` publishes only verified maintenance drift; if any suspicious or unverifiable table is refused, the CLI prints the per-table output and exits non-zero. `--force --confirm` also publishes suspicious or unverifiable drift after operator review.
|
||||
- Classifies drift by reading Lance transactions from `manifest_version + 1` through `lance_head_version`. Only `ReserveFragments` and `Rewrite` are verified maintenance. Semantic operations such as `Append`, `Delete`, `Update`, `Merge`, or missing transaction history are not auto-healed.
|
||||
- Publishes repair by advancing `__manifest` to the existing Lance HEAD; it does **not** rewrite Lance data. If the publish succeeds, normal reads and strict writes use the repaired version. If it fails, no new data-side partial state was created.
|
||||
- Requires a clean recovery state. Pending `__recovery` sidecars still belong to automatic sidecar recovery, not manual repair.
|
||||
|
||||
## `cleanup_all_tables(db, options)` — destructive
|
||||
|
||||
- Lance `cleanup_old_versions()` per table.
|
||||
- Removes manifests (and their unique fragments) older than the retention policy.
|
||||
- `CleanupPolicyOptions { keep_versions: Option<u32>, older_than: Option<Duration> }` — at least one is required.
|
||||
- Returns `[TableCleanupStats { table_key, bytes_removed, old_versions_removed, error }]`.
|
||||
- **Fault-isolated per table.** A single table's transient failure (version GC or
|
||||
orphan reclaim) is recorded on that table's stats row (`error: Some(..)`, logged
|
||||
via `tracing`) and never aborts the healthy tables — cleanup is the convergence
|
||||
backstop, so it does as much as it can and converges on re-run. The CLI reports
|
||||
any failed tables; rerun `cleanup` to retry them.
|
||||
- CLI guards with `--confirm`; without it, prints a preview line.
|
||||
- **Recovery floor:** `--keep < 3` may garbage-collect Lance versions that the open-time recovery sweep needs as a rollback target (the sweep restores to the branch's manifest-pinned table version, which is HEAD-1 in the typical Phase B → Phase C drift case). Default `--keep 10` is safe.
|
||||
- **Orphaned-branch reconciliation:** before the version GC, cleanup runs `reconcile_orphaned_branches`, which `force_delete_branch`es any per-table or commit-graph Lance branch absent from the manifest branch list. These orphans arise when a `branch_delete` flips the manifest authority but a downstream best-effort reclaim does not complete (see [branches-commits.md](../branching/index.md)). The reconciler is authority-derived and idempotent (it no-ops once nothing is orphaned), runs regardless of the `keep_versions` / `older_than` values (those gate version GC only), and never reclaims `main` or system-branch forks. Reclaimed forks are logged via `tracing::info`.
|
||||
|
||||
## Tombstones
|
||||
|
||||
Logical sub-table delete markers in `__manifest`; `tombstone_object_id(table_key, version)` excludes a sub-table version from snapshot reconstruction.
|
||||
|
||||
## Internal schema migrations (`db/manifest/migrations.rs`)
|
||||
|
||||
Version evolutions of the on-disk `__manifest` shape are reconciled automatically on the first write under a new binary. `INTERNAL_MANIFEST_SCHEMA_VERSION` declares the shape the binary expects; the on-disk stamp `omnigraph:internal_schema_version` (Lance schema-level metadata) records the on-disk shape. The publisher's open-for-write path calls `migrate_internal_schema` before reading state; reads are side-effect-free. No operator action is required for in-place upgrades. See [storage.md → Internal schema versioning](../concepts/storage.md) for the full mechanism.
|
||||
|
||||
A binary opening a manifest stamped at a version *higher* than it knows about refuses to publish with a clear "upgrade omnigraph first" error — old binaries cannot clobber a newer schema.
|
||||
175
docs/user/operations/policy.md
Normal file
175
docs/user/operations/policy.md
Normal file
|
|
@ -0,0 +1,175 @@
|
|||
# Authorization (Cedar policy)
|
||||
|
||||
OmniGraph integrates AWS Cedar (`cedar-policy = 4.9`) for ABAC.
|
||||
|
||||
## Policy actions
|
||||
|
||||
Per-graph actions (bind to `Omnigraph::Graph::"<graph_id>"`):
|
||||
|
||||
1. `read` — query / snapshot / list branches & commits
|
||||
2. `export` — NDJSON export
|
||||
3. `change` — mutations
|
||||
4. `schema_apply` — apply schema migrations
|
||||
5. `branch_create`
|
||||
6. `branch_delete`
|
||||
7. `branch_merge`
|
||||
8. `admin` — reserved for policy-management surfaces (hot reload, audit log, approvals). No call site today; see MR-724 for the reservation rationale.
|
||||
9. `invoke_query` — gates invoking a server-side stored query (the `queries:` registry). Graph-scoped (like `admin`) — per-branch access is enforced by the inner `read` / `change` gate, so a rule that sets `branch_scope` on `invoke_query` is rejected. Coarse in this release: an `invoke_query` allow rule permits any stored query on the graph; a future, additive refinement adds an optional per-query-name scope without changing rules written against the coarse action. Enforced at `POST /queries/{name}` (see [server](server.md)). A stored *mutation* is double-gated: `invoke_query` to reach the tool, plus `change` for the write itself (the engine `_as` writers still enforce per the query body).
|
||||
|
||||
Server-scoped action (v0.6.0+; binds to `Omnigraph::Server::"root"`):
|
||||
|
||||
10. `graph_list` — `GET /graphs` registry enumeration (multi-graph mode)
|
||||
|
||||
Server-scoped actions cannot use `branch_scope` or `target_branch_scope` — they operate on the registry, not on a graph's branches. A rule cannot mix server-scoped and per-graph actions; split into separate rules. (Runtime `graph_create` / `graph_delete` are reserved but not shipped in v0.6.0; operators add/remove graphs by editing `omnigraph.yaml` and restarting.)
|
||||
|
||||
## Scope kinds
|
||||
|
||||
- `branch_scope` — applied to source branch (`read`, `export`, `change`)
|
||||
- `target_branch_scope` — applied to destination (`schema_apply`, branch ops, run ops)
|
||||
- `protected_branches` — named list with special rules; rule scopes are `any | protected | unprotected`
|
||||
|
||||
## Per-graph vs. server-level policy (multi-graph mode)
|
||||
|
||||
In multi mode (`omnigraph.yaml` with a non-empty `graphs:` map), policy files attach at two levels:
|
||||
|
||||
```yaml
|
||||
server:
|
||||
policy:
|
||||
file: server-policy.yaml # server-level: graph_list
|
||||
|
||||
graphs:
|
||||
alpha:
|
||||
uri: s3://tenant-bucket/alpha
|
||||
policy:
|
||||
file: policies/alpha.yaml # per-graph: read, change, branch_*, schema_apply
|
||||
beta:
|
||||
uri: s3://tenant-bucket/beta
|
||||
# no per-graph policy → no engine-layer Cedar enforcement on beta
|
||||
```
|
||||
|
||||
**Config follows graph identity, not server mode.** A graph served by **name**
|
||||
(`--target <name>` or `server.graph`) uses its own `graphs.<name>.policy.file`,
|
||||
exactly as in multi-graph mode. Top-level `policy.file` applies only to an
|
||||
**anonymous** graph — one served by a bare `<URI>` with no `graphs:` entry.
|
||||
Serving a **named** graph (single- or multi-graph mode) while top-level
|
||||
`policy.file` (or `queries:`) is populated **refuses boot**, naming the block,
|
||||
since the top-level value would otherwise be silently shadowed by the per-graph
|
||||
block. Move per-graph rules to `graphs.<graph_id>.policy.file` and `graph_list`
|
||||
rules to `server.policy.file`.
|
||||
|
||||
Each graph's HTTP request flows through its own per-graph policy. The management endpoint (`GET /graphs`) flows through the server-level policy. When `server.policy.file` is unset, `GET /graphs` is denied in every runtime state, including `--unauthenticated`; with bearer tokens configured, it returns 403 after admission control because `graph_list` is not a `read`-equivalent action. The operator must explicitly authorize via `server-policy.yaml` to expose `/graphs`.
|
||||
|
||||
Example server-level policy:
|
||||
|
||||
```yaml
|
||||
version: 1
|
||||
groups:
|
||||
admins: [act-andrew]
|
||||
rules:
|
||||
- id: admins-can-list-graphs
|
||||
allow:
|
||||
actors: { group: admins }
|
||||
actions: [graph_list]
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
`omnigraph.yaml`:
|
||||
|
||||
```yaml
|
||||
policy:
|
||||
file: policy.yaml # Cedar rules + groups
|
||||
tests: policy.tests.yaml # declarative test cases
|
||||
|
||||
cli:
|
||||
actor: act-andrew # default actor for CLI direct-engine writes
|
||||
```
|
||||
|
||||
Each per-graph rule may use at most one of `branch_scope` or `target_branch_scope`. Server-scoped rules (`graph_list`) take neither — they have no branch context.
|
||||
|
||||
`cli.actor` is the default actor identity for CLI direct-engine writes
|
||||
when `policy.file` is configured. Override per-invocation with `--as
|
||||
<ACTOR>` (top-level flag) — `--as` wins, otherwise `cli.actor` is used,
|
||||
otherwise no actor. With policy configured and neither set, the
|
||||
engine-layer footgun guard intentionally denies the write (silent bypass
|
||||
via "I forgot the actor" is exactly what the guard prevents). Remote
|
||||
HTTP writes ignore both — they resolve their actor server-side from the
|
||||
bearer token.
|
||||
|
||||
## CLI
|
||||
|
||||
Policy tooling resolves its graph like server single-mode policy: `cli.graph`
|
||||
wins, otherwise `server.graph` is used, otherwise the top-level `policy.file`
|
||||
is validated/tested/explained as the anonymous policy.
|
||||
|
||||
- `omnigraph policy validate` — parse + count actors, exit 1 on parse error.
|
||||
- `omnigraph policy test` — run cases in `policy.tests.yaml`, exit 1 on any expectation mismatch.
|
||||
- `omnigraph policy explain --actor … --action … [--branch …] [--target-branch …]` — show decision and matched rule.
|
||||
- `omnigraph --as <ACTOR> <subcommand>` — set the actor for the duration of one invocation. Effective for `change`, `load` (and its deprecated `ingest` alias), `branch create|delete|merge`, and `schema apply` against local URIs. No-op against remote HTTP URIs (actor is bearer-token-resolved server-side).
|
||||
|
||||
## Enforcement
|
||||
|
||||
Policy is a property of the **engine**, not the transport. Every mutating
|
||||
write — `mutate_as`, `load_as` (the deprecated `ingest_as` shims route
|
||||
through it), `apply_schema_as`,
|
||||
`branch_create_as`, `branch_create_from_as`, `branch_delete_as`,
|
||||
`branch_merge_as` — calls `Omnigraph::enforce(action, scope, actor)` at
|
||||
the head of the method. The gate fires identically whether the call
|
||||
originates from the HTTP server, the CLI, or an embedded SDK consumer.
|
||||
When no `PolicyChecker` is installed (the dev/embedded default) the gate
|
||||
is a strict no-op; when one is installed and the call site forgets to
|
||||
thread an actor through, the gate fails closed rather than silently
|
||||
bypassing.
|
||||
|
||||
## Server runtime states (MR-723)
|
||||
|
||||
The HTTP server classifies its startup configuration into one of three
|
||||
states based on whether bearer tokens are configured and whether a
|
||||
policy file is set. The state determines what happens to a request that
|
||||
reaches `authorize_request()` without a matching policy permit.
|
||||
|
||||
| State | Tokens | Policy file | Behavior |
|
||||
|---|---|---|---|
|
||||
| **Open** | no | no | Every request is permitted. Refuses to start unless `--unauthenticated` or `OMNIGRAPH_UNAUTHENTICATED=1` is set — the operator must explicitly opt in. |
|
||||
| **DefaultDeny** | yes | no | Every authenticated request for an action other than `read` is rejected with HTTP 403. Closes the "tokens but forgot the policy file" trap — an operator who sets up auth and forgot to point at a policy file used to ship the illusion of protection. |
|
||||
| **PolicyEnabled** | yes | yes | Authenticated requests that reach a configured policy engine are evaluated by Cedar. Server-scoped actions still require `server.policy.file`. |
|
||||
|
||||
The classifier is `classify_server_runtime_state` in
|
||||
`crates/omnigraph-server/src/lib.rs`; it returns `Err` for the "no
|
||||
tokens, no policy, no flag" cell and for "policy file, no tokens" so the
|
||||
server refuses to start instead of silently shipping an open instance or
|
||||
a policy-protected server that can only 401. Tests pin every cell of the
|
||||
matrix and the State-2 deny path.
|
||||
|
||||
Server-side, `authorize_request()` still runs at the HTTP boundary —
|
||||
that's where actor identity is resolved from the bearer token and where
|
||||
admission control / per-actor rate limits live. Engine-layer enforcement
|
||||
is the **defense in depth** layer: it catches CLI direct-engine writes,
|
||||
embedded SDK consumers, and any future transport that hasn't (or won't)
|
||||
re-implement HTTP's authorize_request. Both layers consult the same
|
||||
Cedar policy via the same `PolicyChecker` trait, so decisions cannot
|
||||
disagree.
|
||||
|
||||
## Coarse vs. fine enforcement
|
||||
|
||||
There are two enforcement points, each with non-overlapping
|
||||
responsibilities:
|
||||
|
||||
| Layer | Question it answers | Where it fires |
|
||||
|---|---|---|
|
||||
| **Engine-layer (coarse)** | Can this actor invoke this action against this branch / branch-transition? | `Omnigraph::enforce(action, scope, actor)` at the head of every `_as` writer; one Cedar decision per call. |
|
||||
| **Query-layer (fine)** | For the rows / types this action actually touches, which can the actor see or modify? | Per-row predicates pushed into DataFusion at plan time. **Not yet implemented — see MR-725.** |
|
||||
|
||||
The engine-layer gate keeps `ResourceScope` deliberately at branch
|
||||
granularity (`Graph`, `Branch`, `TargetBranch`, `BranchTransition`).
|
||||
Per-type and per-row authority is the query-layer's job; conflating them
|
||||
in `ResourceScope` would create two places per-type policy could be
|
||||
evaluated and a drift surface between them.
|
||||
|
||||
## Actor identity (signed-claim-only)
|
||||
|
||||
The actor identity used for every policy decision comes from the matched bearer token — never from a client-supplied request header, query parameter, or body field. The server resolves the token at the auth middleware boundary, looks up the actor it was minted for, and overwrites whatever the handler may have placed in the policy request. Clients cannot set `actor_id` directly.
|
||||
|
||||
This is intentional. Trusting client-supplied identity for authorization is "asking the attacker if they're an admin" — Supabase's RLS history names the same footgun. The chokepoint lives in `authorize_request` in `crates/omnigraph-server/src/lib.rs` and is named in `docs/dev/invariants.md` Hard Invariant 11. A regression test asserts the contract: a request with `Authorization: Bearer <token-for-actor-A>` plus `X-Actor-Id: actor-B` always evaluates as actor A, never as actor B.
|
||||
|
||||
If you find yourself wanting to let clients override `actor_id` for impersonation, delegation, or service-account flows — that's a feature, but it needs explicit design (e.g., signed delegation claims, an `On-Behalf-Of` audit trail). It is not a convenience knob.
|
||||
230
docs/user/operations/server.md
Normal file
230
docs/user/operations/server.md
Normal file
|
|
@ -0,0 +1,230 @@
|
|||
# HTTP Server (`omnigraph-server`)
|
||||
|
||||
Axum 0.8 + tokio + utoipa-generated OpenAPI. **Two modes** (v0.6.0+): single-graph (legacy) and multi-graph (MR-668), with **two boot sources** for multi mode: `omnigraph.yaml` or — exclusively — a cluster directory (`--cluster`, RFC-005). Mode is inferred from CLI args + config shape.
|
||||
|
||||
## Modes
|
||||
|
||||
### Single-graph mode (legacy)
|
||||
|
||||
`omnigraph-server <URI>` or `omnigraph-server --target <name> --config omnigraph.yaml`. Routes are flat — `/snapshot`, `/read`, `/branches`, etc.
|
||||
|
||||
**Config follows graph identity.** A bare `<URI>` is an *anonymous* graph and uses the **top-level** `policy.file` / `queries:`. A graph chosen by **name** (`--target` / `server.graph`) uses its own `graphs.<name>.{policy.file, queries}` — the same block multi-graph mode uses. ⚠️ *Changed from v0.6.0, which always used top-level config in single mode: a named-graph config that puts `policy`/`queries` at top-level now **refuses boot** and points you at `graphs.<name>.…` (move the block there). Bare-`<URI>` single mode is unchanged.*
|
||||
|
||||
### Multi-graph mode (v0.6.0+)
|
||||
|
||||
`omnigraph-server --config omnigraph.yaml` with a non-empty `graphs:` map and **no** single-mode selector (no `server.graph`, no `<URI>`, no `--target`). The server opens every configured graph in parallel at startup (bounded concurrency = 4, fail-fast on the first open error). Routes are nested under `/graphs/{graph_id}/...`. Bare flat paths return 404 in multi mode.
|
||||
|
||||
### Cluster-booted multi mode (Phase 5)
|
||||
|
||||
`omnigraph-server --cluster <dir-or-uri>` boots from the cluster catalog's **applied
|
||||
revision** (`state.json` + content-addressed blobs) instead of
|
||||
`omnigraph.yaml` — an exclusive boot source: combining it with `<URI>`,
|
||||
`--target`, or `--config` is a startup error, and `omnigraph.yaml` is never
|
||||
read in this mode. Always multi-graph routing. See
|
||||
[cluster-config.md](../clusters/config.md#serving-from-the-cluster-the-mode-switch)
|
||||
for what is read and the fail-fast readiness rules. `--bind`,
|
||||
`--unauthenticated`, and the bearer-token env vars work identically.
|
||||
|
||||
Mode inference:
|
||||
|
||||
0. CLI `--cluster <dir | s3://…>` → **multi, cluster-booted** (exclusive; a scheme-qualified argument reads the ledger straight from the storage root, no local config)
|
||||
1. CLI positional `<URI>` → single
|
||||
2. CLI `--target <name>` → single
|
||||
3. `server.graph` in config → single
|
||||
4. `--config` + non-empty `graphs:` + no single-mode selector → **multi**
|
||||
5. otherwise → error with migration hint
|
||||
|
||||
### Stored-query validation at startup
|
||||
|
||||
If a graph declares a `queries:` registry (see [cli-reference](../cli/reference.md)), the server **loads and type-checks every stored query against that graph's live schema at startup** and **refuses to boot** if any query references a type or property the schema lacks — the same fail-loud posture as a malformed policy file, so schema drift surfaces at the deploy boundary rather than at invocation. Two MCP-exposed queries claiming the same tool name is likewise a boot error. Non-blocking advisories (e.g. an MCP-exposed query with a vector parameter an agent cannot supply) are logged. Validate offline before deploying with `omnigraph queries validate`. Discover the exposed queries as a typed tool catalog with `GET /queries`, and invoke one over HTTP with `POST /queries/{name}` (both below).
|
||||
|
||||
## Endpoint inventory
|
||||
|
||||
Per-graph endpoints — same body shape across modes; URLs differ:
|
||||
|
||||
| Method | Single-mode path | Multi-mode path | Auth | Action | Handler |
|
||||
|---|---|---|---|---|---|
|
||||
| GET | `/healthz` | `/healthz` | none | — | `server_health` |
|
||||
| GET | `/openapi.json` | `/openapi.json` | none | — | `server_openapi` (strips security if auth disabled; in multi mode emits cluster paths with `cluster_` operation-id prefix) |
|
||||
| GET | `/snapshot?branch=` | `/graphs/{id}/snapshot?branch=` | bearer + `read` | snapshot of branch | `server_snapshot` |
|
||||
| POST | `/query` | `/graphs/{id}/query` | bearer + `read` | inline read query (canonical; clean field names `query`/`name`; mutations → 400) | `server_query` |
|
||||
| POST | `/read` | `/graphs/{id}/read` | bearer + `read` | **deprecated** alias of `/query` (legacy field names `query_source`/`query_name`, byte-stable response; carries `Deprecation: true` + `Link: </query>; rel="successor-version"`) | `server_read` |
|
||||
| POST | `/export` | `/graphs/{id}/export` | bearer + `export` | NDJSON stream | `server_export` |
|
||||
| POST | `/mutate` | `/graphs/{id}/mutate` | bearer + `change` | mutation (canonical; `query`/`name`; accepts legacy `query_source`/`query_name` as serde aliases) | `server_mutate` |
|
||||
| POST | `/change` | `/graphs/{id}/change` | bearer + `change` | **deprecated** alias of `/mutate` (carries `Deprecation: true` + `Link: </mutate>; rel="successor-version"`) | `server_change` |
|
||||
| GET | `/queries` | `/graphs/{id}/queries` | bearer + `read` | list the `mcp.expose` stored queries as a typed tool catalog | `server_list_queries` |
|
||||
| POST | `/queries/{name}` | `/graphs/{id}/queries/{name}` | bearer + `invoke_query` (+ `change` for a stored mutation) | invoke a named query from the `queries:` registry; deny == 404 | `server_invoke_query` |
|
||||
| GET | `/schema` | `/graphs/{id}/schema` | bearer + `read` | get current `.pg` source | `server_schema_get` |
|
||||
| POST | `/schema/apply` | `/graphs/{id}/schema/apply` | bearer + `schema_apply` (target=`main`) | migrate | `server_schema_apply` |
|
||||
| POST | `/load` | `/graphs/{id}/load` | bearer + `branch_create` (only when `from` is set and the branch is created) + `change` | bulk load (canonical); branch creation is opt-in via `from` — without it a missing `branch` is a 404, never an implicit fork | `server_load` (32 MB body limit) |
|
||||
| POST | `/ingest` | `/graphs/{id}/ingest` | bearer + `branch_create` (only when `from` is set and the branch is created) + `change` | **deprecated** alias of `/load` (carries `Deprecation: true` + `Link: </load>; rel="successor-version"`) | `server_ingest` (32 MB body limit) |
|
||||
| GET | `/branches` | `/graphs/{id}/branches` | bearer + `read` | list branches | `server_branch_list` |
|
||||
| POST | `/branches` | `/graphs/{id}/branches` | bearer + `branch_create` | create | `server_branch_create` |
|
||||
| DELETE | `/branches/{branch}` | `/graphs/{id}/branches/{branch}` | bearer + `branch_delete` | delete | `server_branch_delete` |
|
||||
| POST | `/branches/merge` | `/graphs/{id}/branches/merge` | bearer + `branch_merge` | merge `source → target` | `server_branch_merge` |
|
||||
| GET | `/commits?branch=` | `/graphs/{id}/commits?branch=` | bearer + `read` | list | `server_commit_list` |
|
||||
| GET | `/commits/{commit_id}` | `/graphs/{id}/commits/{commit_id}` | bearer + `read` | show | `server_commit_show` |
|
||||
|
||||
Server-level management endpoints (v0.6.0+):
|
||||
|
||||
| Method | Path | Auth | Action | Handler |
|
||||
|---|---|---|---|---|
|
||||
| GET | `/graphs` | bearer + `graph_list` on `Server::"root"` | list registered graphs | `server_graphs_list` (405 in single mode) |
|
||||
|
||||
### Stored-query catalog (`GET /queries`)
|
||||
|
||||
List the graph's **`mcp.expose`** stored queries as a typed tool catalog — enough for a client (e.g. an MCP server) to register each as a tool without fetching `.gq` source. Each entry: `{ name, tool_name, description, instruction, mutation, params }`, where each param is `{ name, kind, item_kind?, vector_dim?, nullable }`. `kind` is one of `string | bool | int | bigint | float | date | datetime | blob | vector | list` (decomposed so a consumer maps it with a closed `switch`, never re-parsing GQ type spelling). `bigint` (I64/U64), `date`, `datetime`, and `blob` are carried as JSON **strings** — a 64-bit integer loses precision as a JSON number, dates are ISO strings, and a blob is a URI string.
|
||||
|
||||
- **Read-gated** (works in default-deny mode). The catalog is **graph-wide** (branch-independent; `read` is authorized against `main`).
|
||||
- **`mcp.expose` defaults to `true`** — declaring a query in `queries:` lists it; set `mcp: { expose: false }` to keep it HTTP/service-callable but hidden from the catalog.
|
||||
- **Not Cedar-filtered per query (yet).** A caller with `read` but not `invoke_query` can *list* a query they can't *invoke* (which would 404). Closing that gap is future per-query authorization; for now the catalog is a discovery surface and `invoke_query` remains the invocation gate.
|
||||
|
||||
### Stored-query invocation (`POST /queries/{name}`)
|
||||
|
||||
Invoke a curated, server-side stored query by **name** — the source comes from the graph's `queries:` registry, so the client never sends `.gq`. The request body itself is optional; omit it for no-param queries, or send `{ "params": { … }, "branch": "main", "snapshot": null }`, where every field is optional and `params` keys match the query's declared parameters. The response is the **read envelope** (`ReadOutput`) for a stored read or the **mutation envelope** (`ChangeOutput`) for a stored mutation — serialized untagged, so the wire shape is identical to `/query` / `/mutate`.
|
||||
|
||||
- **Gate:** `invoke_query` (per-graph, graph-scoped) at the boundary. A stored *mutation* is **double-gated** — it also passes the engine's `change` gate, so an actor with `invoke_query` but not `change` gets `403`.
|
||||
- **Deny == unknown, for callers without `invoke_query`:** for a caller lacking the grant, an `invoke_query` denial and an unknown query name return the **same `404`** (identical body), so the catalog can't be probed. A caller that *holds* `invoke_query` may still get the inner gate's `403` for an existing query it can't `read`/`change` (the double-gate, above) — so existence is visible to grant-holders by design.
|
||||
- **Requires an explicit policy grant when auth is on.** In default-deny mode (bearer tokens but no `policy.file`), only `read` is permitted, so *every* `/queries/{name}` call returns `404` until an `invoke_query` rule is configured.
|
||||
- A stored mutation cannot target a `snapshot` (`400`); a parameter type error is a structured `400` naming the parameter.
|
||||
|
||||
## Adding and removing graphs (multi mode)
|
||||
|
||||
Runtime add/remove via API is **not** exposed in v0.6.0 — neither
|
||||
`POST /graphs` nor `DELETE /graphs/{id}` is implemented. Operators add
|
||||
or remove graphs by stopping the server, editing the `graphs:` map in
|
||||
`omnigraph.yaml`, then restarting. The server treats `omnigraph.yaml`
|
||||
as operator-owned configuration and never writes it.
|
||||
|
||||
A future release may introduce a managed registry (Lance-backed,
|
||||
catalog-style: reserve → init → publish with recovery sidecars) and
|
||||
re-expose runtime mutation on top of it.
|
||||
|
||||
## Inline read queries (`POST /query`)
|
||||
|
||||
`POST /query` is the read-only, agent-friendly twin of `POST /read`. The
|
||||
request body uses clean field names that match the CLI `-e` flag and the GQ
|
||||
`query` keyword:
|
||||
|
||||
```json
|
||||
{
|
||||
"query": "query find($n: String) { match { $p: Person { name: $n } } return { $p.name } }",
|
||||
"name": "find",
|
||||
"params": { "n": "Alice" },
|
||||
"branch": "main",
|
||||
"snapshot": null
|
||||
}
|
||||
```
|
||||
|
||||
Response shape is identical to `/read` (`ReadOutput`). If the inline source
|
||||
contains mutations (`insert` / `update` / `delete`), the request is rejected
|
||||
with HTTP 400 and an error pointing the caller at `POST /mutate` — the
|
||||
read-only contract is enforced at the URL.
|
||||
|
||||
`POST /mutate` is the canonical mutation endpoint. It accepts the same clean
|
||||
field names (`query`, `name`); the legacy field names `query_source` and
|
||||
`query_name` continue to deserialize as serde aliases so existing clients keep
|
||||
working without changes.
|
||||
|
||||
## Deprecated names (`/read`, `/change`)
|
||||
|
||||
`POST /read` and `POST /change` are kept for back-compat indefinitely — they
|
||||
are byte-stable on the request side and otherwise behave identically to
|
||||
`/query` / `/mutate`. They are flagged as deprecated through three independent
|
||||
channels:
|
||||
|
||||
- **OpenAPI**: the operations carry `deprecated: true` in `openapi.json`, so
|
||||
every OpenAPI codegen (typescript-fetch, openapi-generator, oapi-codegen,
|
||||
…) emits a `@deprecated` marker on the generated SDK method.
|
||||
- **Response headers (RFC 9745)**: every response carries `Deprecation: true`.
|
||||
- **Response headers (RFC 8288)**: every response carries a `Link` header
|
||||
pointing at the canonical successor:
|
||||
`Link: </query>; rel="successor-version"` for `/read`, and
|
||||
`Link: </mutate>; rel="successor-version"` for `/change`. SDKs and HTTP
|
||||
proxies can pick the successor up automatically.
|
||||
|
||||
Migration is purely cosmetic on the client side — swap the URL path, leave
|
||||
the request body and response handling alone.
|
||||
|
||||
## Streaming
|
||||
|
||||
Only `/export` streams (`application/x-ndjson`, MPSC channel + `Body::from_stream`). Everything else is buffered JSON.
|
||||
|
||||
## Error model
|
||||
|
||||
Uniform `ErrorOutput { error, code?, merge_conflicts[], manifest_conflict? }` with `code ∈ unauthorized | forbidden | bad_request | not_found | conflict | too_many_requests | internal`. Merge conflicts attach structured `MergeConflictOutput { table_key, row_id?, kind, message }`.
|
||||
|
||||
`manifest_conflict` is set on **publisher CAS rejections** (HTTP 409): the
|
||||
caller's pre-write view of one table's manifest version was stale.
|
||||
`ManifestConflictOutput { table_key, expected, actual }` tells the client
|
||||
which table to refresh and retry. This is the conflict shape produced by
|
||||
concurrent `/mutate` (or its `/change` alias) or `/ingest` calls landing
|
||||
the same `(table, branch)` race.
|
||||
|
||||
HTTP status codes used: 200, 400, 401, 403, 404, 409, 429, 500.
|
||||
|
||||
## Per-actor admission control
|
||||
|
||||
Disjoint
|
||||
`(table, branch)` writes from different actors now run concurrently,
|
||||
guarded only by the engine's per-(table, branch) write queue. To keep
|
||||
one heavy actor from exhausting shared capacity (Lance I/O, manifest
|
||||
churn, network), the server gates mutating handlers through a
|
||||
`WorkloadController` configured per-process from environment variables:
|
||||
|
||||
| Env var | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `OMNIGRAPH_PER_ACTOR_INFLIGHT_MAX` | 16 | Concurrent in-flight mutations per actor |
|
||||
| `OMNIGRAPH_PER_ACTOR_BYTES_MAX` | 4 GiB | In-flight estimated bytes per actor |
|
||||
|
||||
When an actor exceeds its in-flight count or byte budget, the server
|
||||
returns **HTTP 429 Too Many Requests** with `code: too_many_requests`
|
||||
and a `Retry-After` header (seconds). The actor should back off; other
|
||||
actors are unaffected.
|
||||
|
||||
Cedar policy authorization runs **before** admission accounting so
|
||||
denied requests don't consume admission slots.
|
||||
|
||||
Today admission gates every mutating handler: `/mutate` (and its
|
||||
deprecated alias `/change`), `/ingest`, `/branches/{create,delete,merge}`,
|
||||
and `/schema/apply`. Read-only endpoints (`/snapshot`, `/query`, `/read`,
|
||||
`/export`, `/branches` GET, `/commits`, `/schema` GET) are not
|
||||
admission-gated.
|
||||
|
||||
## Body limits
|
||||
|
||||
- Default: 1 MB
|
||||
- `/ingest`: 32 MB
|
||||
|
||||
## Auth model (`bearer + SHA-256`)
|
||||
|
||||
- Tokens are SHA-256 hashed on startup; plaintext is never persisted in memory.
|
||||
- Constant-time comparison via `subtle::ConstantTimeEq`.
|
||||
- Three sources, in precedence:
|
||||
1. `OMNIGRAPH_SERVER_BEARER_TOKENS_AWS_SECRET` — AWS Secrets Manager (build with `--features aws`)
|
||||
2. `OMNIGRAPH_SERVER_BEARER_TOKENS_FILE` or `OMNIGRAPH_SERVER_BEARER_TOKENS_JSON` — JSON `{actor_id: token, …}`
|
||||
3. `OMNIGRAPH_SERVER_BEARER_TOKEN` — single legacy token, actor `default`
|
||||
- If no tokens are configured, startup refuses unless `--unauthenticated` or
|
||||
`OMNIGRAPH_UNAUTHENTICATED=1` explicitly opts into open local-dev mode. A
|
||||
policy file without tokens is also rejected at startup. In open mode
|
||||
`/openapi.json` strips the security scheme.
|
||||
|
||||
See [deployment.md](../deployment.md) for token-source operational details.
|
||||
|
||||
## Tracing & observability
|
||||
|
||||
- `tower_http::TraceLayer::new_for_http()`
|
||||
- Policy decisions logged at INFO level with actor, action, branch, decision, matched rule
|
||||
- Startup logs: token source name, graph URI, bind address
|
||||
- Graceful SIGINT shutdown
|
||||
|
||||
## Not implemented (by design or "TBD")
|
||||
|
||||
- CORS — not configured; add `tower_http::cors` if needed.
|
||||
- Rate limiting — per-actor admission control gates `/mutate` (alias
|
||||
`/change`), `/ingest`, `/branches/{create,delete,merge}`,
|
||||
`/schema/apply` (see "Per-actor
|
||||
admission control" above). No global rate limiter is configured;
|
||||
add `tower_http::limit` if a graph-wide cap is needed.
|
||||
- Pagination — none (commits/branches return everything; export streams).
|
||||
- Runtime graph add/remove — edit `omnigraph.yaml` and restart.
|
||||
Loading…
Add table
Add a link
Reference in a new issue