From 77dffdae928331545b480edd96821ec27a36c8a9 Mon Sep 17 00:00:00 2001 From: Andrew Altshuler Date: Sun, 14 Jun 2026 14:39:25 +0300 Subject: [PATCH] =?UTF-8?q?docs(user):=20de-dev=20polish=20=E2=80=94=20str?= =?UTF-8?q?ip=20internal=20scaffolding=20from=20user=20docs=20(Phase=203a)?= =?UTF-8?q?=20(#226)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove developer-only scaffolding that leaked into the public user/operator docs, while preserving every user-facing behavior, command, flag, endpoint, constant, and env var. No behavior changes. Removed across 18 files: - internal ticket / sequencing refs (MR-NNN, RFC-NNN, "Phase N"); - source-code paths (crates/**/*.rs, *.pest) and internal struct/function dumps (e.g. the QueryIR / GraphCommit / SchemaMigrationPlan Rust types, internal fn names like fork_branch_from_state, optimize_all_tables); - Lance-internal blocker prose (upstream issue numbers, blob-decode cause, sidecar Phase-B/C mechanics) — keeping the user-visible behavior (e.g. "optimize skips Blob-column tables; reads/writes unaffected"); - pre-v0.4.0 Run-state-machine archaeology. Internal IR/lowering/recovery-internals sections were either trimmed to a brief user-facing note (e.g. "Traversal execution", "interrupted writes recover automatically; recovery commits are recorded under actor omnigraph:recovery") or removed. Kept: all language syntax, lint codes, Cedar actions/scopes, endpoints, error taxonomy, every constant and env var (verified none dropped from the constants cheat-sheet), and the operator-facing explanations of on-disk artifacts. Residual "legacy" mentions are all user-facing (the deprecated omnigraph.yaml, the legacy token chain, old command names). Verified: zero internal-scaffolding leaks (MR/RFC/Phase/.rs/.pest = 0) across docs/user; zero broken links; check-agents-md.sh green. Co-authored-by: Claude Opus 4.8 --- docs/user/branching/changes.md | 2 +- docs/user/branching/index.md | 49 ++++++-------------- docs/user/branching/transactions.md | 6 +-- docs/user/cli/index.md | 2 +- docs/user/cli/reference.md | 13 +++--- docs/user/clusters/config.md | 2 - docs/user/concepts/storage.md | 44 +++++++++--------- docs/user/deployment.md | 3 +- docs/user/operations/errors.md | 2 +- docs/user/operations/maintenance.md | 56 +++++++++++------------ docs/user/operations/policy.md | 40 ++++++++-------- docs/user/operations/server.md | 71 ++++++++++++++--------------- docs/user/queries/index.md | 33 ++------------ docs/user/reference/constants.md | 40 ++++++++-------- docs/user/schema/index.md | 37 ++++++--------- docs/user/schema/lint.md | 43 ++++++++--------- docs/user/search/embeddings.md | 4 +- docs/user/search/indexes.md | 11 ++--- 18 files changed, 192 insertions(+), 266 deletions(-) diff --git a/docs/user/branching/changes.md b/docs/user/branching/changes.md index 58739e2..a9bceec 100644 --- a/docs/user/branching/changes.md +++ b/docs/user/branching/changes.md @@ -1,6 +1,6 @@ # Change Detection / Diff -`changes/mod.rs`. Three-level algorithm: +Diffing two read targets uses a three-level algorithm: 1. **Manifest diff**: skip sub-tables whose `(table_version, table_branch)` is unchanged. 2. **Lineage check**: diff --git a/docs/user/branching/index.md b/docs/user/branching/index.md index a0f1a6e..20ea125 100644 --- a/docs/user/branching/index.md +++ b/docs/user/branching/index.md @@ -2,44 +2,24 @@ ## L1 — Lance per-dataset branches -Lance supports branching at the dataset level: a branch is a named lineage of versions, and `fork_branch_from_state(source_branch, target_branch, source_version)` creates a copy-on-write fork. +Lance supports branching at the dataset level: a branch is a named lineage of versions, and a copy-on-write fork creates a new branch from a source branch at a given version. ## L2 — Graph-level branches OmniGraph builds *graph branches* on top by branching every sub-table coherently: -- `branch_create(name)` / `branch_create_from(target, name)` — disallowed name `main`; fails if branch exists; ensures the schema-apply lock is idle. Atomic and authority-first like `branch_delete`: it flips the `__manifest` branch (authority), then creates the derived commit-graph branch, force-dropping any orphaned commit-graph ref left by an incomplete prior delete (the manifest branch is fresh, so a same-named commit-graph branch is provably a zombie). If commit-graph creation fails, the manifest branch is rolled back so the name never half-exists. -- `branch_list()` — returns public branches, **filters the internal** `__schema_apply_lock__` branch. -- `branch_delete(name)` — refuses if there are descendants on the branch, or if it is the current branch. The manifest is the single authority for branch existence: deletion flips the `__manifest` branch ref first (one atomic op), after which the branch is gone from every snapshot. The owned per-table forks and the commit-graph branch are derived state, reclaimed best-effort with `force_delete_branch` after the flip. A failure during that reclaim (transient object-store error) does not fail the call or block the authority flip; the leftover forks are unreachable orphans that the [`cleanup`](../operations/maintenance.md) reconciler converges. One consequence: if a delete's best-effort reclaim fails, reusing that branch name before the next `cleanup` surfaces a clear error pointing at `cleanup` (the stale fork would otherwise collide on first write). -- **Lazy forking**: a branch only forks a sub-table when that sub-table is first mutated on it. Pure-read branches share fragments with their source. A fork collision is classified by the manifest authority, not by Lance branch versions: if the live manifest already records the fork on the active branch, a concurrent first-write won and the caller gets a retryable "refresh and retry"; if the manifest does not, a physical branch there is an orphan and the caller is pointed at `cleanup`. -- `sync_branch(branch)` — re-binds the in-memory handle to the latest head of the branch. +- **Create** (`branch create` / `branch create --from `) — the name `main` is disallowed; fails if the branch exists. Atomic: the new branch becomes visible all-or-nothing, so a name never half-exists. +- **List** (`branch list`) — returns public branches, **filtering the internal** `__schema_apply_lock__` branch. +- **Delete** (`branch delete`) — refuses if there are descendants on the branch, or if it is the current branch. Once deleted, the branch is gone from every snapshot. The owned per-table forks are reclaimed best-effort; if that reclaim hits a transient object-store error, the leftover storage is reclaimed later by the [`cleanup`](../operations/maintenance.md) command. One consequence: if a delete's reclaim fails, reusing that branch name before the next `cleanup` surfaces a clear error pointing at `cleanup`. +- **Lazy forking**: a branch only forks a sub-table when that sub-table is first mutated on it. Pure-read branches share storage with their source. If two writers race to first-write the same branch, the loser gets a retryable "refresh and retry". -## L2 — Commit graph (`db/commit_graph.rs`) +## L2 — Commit graph -In-memory shape of a graph commit: +Each graph commit carries a ULID id, the manifest branch and version it published, its parent commit (two parents for a merge commit, one for a linear commit), the actor who made it, and a creation timestamp. -``` -GraphCommit { - graph_commit_id: ULID, - manifest_branch: Option, - manifest_version: u64, - parent_commit_id: Option, - merged_parent_commit_id: Option, // populated for merge commits - actor_id: Option, // joined in-memory from _graph_commit_actors.lance, NOT a column on _graph_commits.lance - created_at: i64 (microseconds since epoch), -} -``` - -Storage is split across two Lance datasets (both with stable row IDs): - -- `_graph_commits.lance` — every column above *except* `actor_id`. -- `_graph_commit_actors.lance` — optional separate `(graph_commit_id, actor_id)` map, created on demand. The `actor_id` field above is populated by joining this dataset in-memory at load time. - -Notes: - -- Every successful publish (load / change / merge / schema_apply) appends one commit. +- Every successful publish (load / change / merge / schema apply) appends one commit. - Merge commits have two parents; linear commits have one. -- API: `list_commits(branch)`, `get_commit(id)`, `head_commit_id_for_branch(branch)`. +- Inspect history with `commit list` and `commit show`. ## L2 — Snapshots & time travel @@ -49,13 +29,12 @@ conflict kinds are on the [merge](merge.md) page. ## L2 — Internal system branches -Internal or legacy branch refs: - -- `__schema_apply_lock__` — serializes schema migrations; filtered from `branch_list()` but visible to internals. -- `__run__` — legacy from the pre-v0.4.0 Run state machine (removed in MR-771). These are swept off `__manifest` on the first read-write open by the v2→v3 internal-schema migration (MR-770), and `__run__*` is no longer a reserved name. Known limitation: a pre-v0.4.0 graph opened **read-only** still surfaces any stale `__run__*` branch in `branch_list()` until its first read-write open (the migration is write-path-only, like all manifest migrations). +- `__schema_apply_lock__` — serializes schema migrations; filtered from `branch list` but used internally. ## L2 — Recovery audit trail -The five migrated writers (`MutationStaging::finalize`, `schema_apply`, `branch_merge`, `ensure_indices`, `optimize_all_tables`) protect their multi-table commits with a sidecar at `__recovery/{ulid}.json` written before Phase B and deleted after Phase C. The next `Omnigraph::open` (gated on `OpenMode::ReadWrite`) runs the recovery sweep in `crates/omnigraph/src/db/manifest/recovery.rs`: classify per-table state, decide all-or-nothing per sidecar, roll forward / back, record an audit row. +Interrupted multi-table writes are recovered automatically the next time the graph is opened read-write. Recovery commits are recorded in the audit trail under the actor `omnigraph:recovery`, so you can find them with: -Audit rows live in `_graph_commit_recoveries.lance` (sibling to `_graph_commits.lance`) and reference the commit graph by `graph_commit_id`. The linked recovery commit is identified by that same `graph_commit_id`, and `actor_id="omnigraph:recovery"` is stored in `_graph_commit_actors.lance` (joined by `graph_commit_id`) — `_graph_commits.lance` itself does not carry the `actor_id` column. To find recoveries for a specific original actor: `omnigraph commit list --filter actor=omnigraph:recovery`, then join to `_graph_commit_recoveries.lance` by `graph_commit_id` to read `recovery_for_actor`. Schema: see `crates/omnigraph/src/db/recovery_audit.rs`. +```bash +omnigraph commit list --filter actor=omnigraph:recovery +``` diff --git a/docs/user/branching/transactions.md b/docs/user/branching/transactions.md index a5515da..6e6b1c4 100644 --- a/docs/user/branching/transactions.md +++ b/docs/user/branching/transactions.md @@ -107,7 +107,7 @@ Properties: - Each query on the branch is its own publisher commit — so they're individually atomic. Per-query CAS works on branches just like on main. - The branch lives on disk. Process crash mid-workflow? Re-open and resume. - Multiple agents can work on different branches in parallel without blocking each other. -- The merge is a three-way merge at the row level. Conflicts surface as `OmniError::MergeConflicts(Vec)`, with structured kinds (`DivergentInsert`, `DivergentUpdate`, `DeleteVsUpdate`, …) so callers can handle them programmatically. +- The merge is a three-way merge at the row level. Conflicts surface as structured merge-conflict kinds (`DivergentInsert`, `DivergentUpdate`, `DeleteVsUpdate`, …) so callers can handle them programmatically. ### 4. Coordinating multiple agents @@ -129,14 +129,14 @@ omnigraph branch merge agent-b/work --into main graph.omni Each agent sees a consistent snapshot of `main` at the time it forked. The first merge to `main` lands as a fast-forward (or a no-op if no concurrent change). The second merge runs three-way: rows touched by both branches surface as `MergeConflict`s for the caller to resolve. -This is the workflow MR-797 / agentic loops are designed around: **branches are the unit of "an agent's working set."** +This is the workflow agentic loops are designed around: **branches are the unit of "an agent's working set."** ## Failure modes | Scenario | What happens | Caller action | |---|---|---| | Single query fails mid-flight | Publisher never publishes; target unchanged | Read the error, decide whether to retry | -| Concurrent writers race the same `(table, branch)` | Publisher CAS rejects the loser with `ManifestConflictDetails::ExpectedVersionMismatch` | Refresh handle, retry the query | +| Concurrent writers race the same `(table, branch)` | Publisher CAS rejects the loser with a version-mismatch conflict | Refresh handle, retry the query | | Branch with N successful mutations, then merge fails (three-way conflict) | Each individual mutation already committed on the branch; merge surfaces `MergeConflicts` | Inspect, decide whether to keep working on the branch, abandon it (`branch_delete`), or resolve and re-merge | | Process crashes mid-branch-workflow | Each completed mutation on the branch is durable | Re-open the graph, continue where you left off | diff --git a/docs/user/cli/index.md b/docs/user/cli/index.md index a6ce442..6813744 100644 --- a/docs/user/cli/index.md +++ b/docs/user/cli/index.md @@ -106,7 +106,7 @@ omnigraph commit list graph.omni --json omnigraph commit show --uri graph.omni --json ``` -(The legacy `omnigraph run list/show/publish/abort` subcommands were removed in MR-771; mutations and loads publish atomically and the commit graph (`omnigraph commit list`) is the audit surface.) +(Mutations and loads publish atomically; the commit graph (`omnigraph commit list`) is the audit surface.) `query lint` and `query check` are the same command surface. In v1, graph-backed lint uses local or `s3://` graph URIs; HTTP targets are only supported when you diff --git a/docs/user/cli/reference.md b/docs/user/cli/reference.md index bb73225..77feaf1 100644 --- a/docs/user/cli/reference.md +++ b/docs/user/cli/reference.md @@ -8,7 +8,7 @@ Top-level command families and subcommands. Graph-targeting commands accept a po | Command | Purpose | |---|---| -| `init` | `--schema ` → initialize a graph (no longer scaffolds `omnigraph.yaml` — RFC-008; start cluster configs from the [cluster.md](../clusters/index.md) quick-start or `config migrate`) | +| `init` | `--schema ` → initialize a graph (no longer scaffolds `omnigraph.yaml`; start cluster configs from the [cluster.md](../clusters/index.md) quick-start or `config migrate`) | | `load` | bulk load a branch, local or remote (`--mode overwrite\|append\|merge` is **required** — overwrite is destructive, so there is no default). Without `--from` the target branch must exist; `--from ` forks a missing `--branch` from `` first | | `ingest` | deprecated alias of `load --from ` (defaults: `--from main --mode merge`); prints a one-line warning to stderr | | `query` (alias: `read`) | run named read query; source via `--query `, `-e`/`--query-string `, or `--alias ` (exactly one). `read` is the deprecated previous name and prints a one-line warning to stderr | @@ -19,7 +19,7 @@ Top-level command families and subcommands. Graph-targeting commands accept a po | `commit list \| show` | inspect commit graph | | `schema plan \| apply \| show (alias: get)` | migrations | | `lint` (alias: `check`) | offline / graph-backed query validation. Replaces `query lint` / `query check`, which are kept as deprecated argv-level shims that print a one-line warning and rewrite to `omnigraph lint` | -| `config migrate` | propose (or `--write`: apply) the RFC-008 split of a legacy `omnigraph.yaml` — team half → ready-to-review `cluster.yaml`, personal half → `~/.omnigraph/config.yaml` (key-level merge, existing entries win), plus dropped-key reasons and manual steps | +| `config migrate` | propose (or `--write`: apply) the split of a legacy `omnigraph.yaml` — team half → ready-to-review `cluster.yaml`, personal half → `~/.omnigraph/config.yaml` (key-level merge, existing entries win), plus dropped-key reasons and manual steps | | `cluster validate \| plan \| apply \| approve \| status \| refresh \| import \| force-unlock` | declarative cluster control plane. `validate` checks a local `cluster.yaml` folder and referenced schema/query/policy files; `plan` diffs it against local JSON state at `__cluster/state.json`, annotates dispositions, and embeds real schema-migration previews; `apply` converges the cluster — stored-query/policy catalog writes (content-addressed under `__cluster/resources/`), graph creates, schema updates (soft drops only; `--as` records the actor), and graph deletes behind a digest-bound approval from `cluster approve --as ` (`apply`/`approve` default the actor from the per-operator `omnigraph.yaml`'s `cli.actor` when `--as` is omitted; nothing else in that file affects cluster commands); what apply converges is what an `omnigraph-server --cluster ` deployment serves on its next restart (omnigraph.yaml deployments are unaffected); `status` reads the state ledger; `refresh`/`import` explicitly update local JSON state from read-only graph observations; `force-unlock ` manually removes a held local state lock by exact id | | `optimize` | non-destructive Lance compaction (skips tables with `Blob` columns or uncovered drift; `--json` reports `skipped`) | | `repair [--confirm] [--force]` | preview or explicitly publish uncovered manifest/head drift. `--confirm` heals verified maintenance drift and exits non-zero if suspicious/unverifiable drift is refused; `--force --confirm` publishes suspicious/unverifiable drift after operator review | @@ -30,7 +30,7 @@ Top-level command families and subcommands. Graph-targeting commands accept a po ## Command planes -Every command lives on one **plane**, which determines how it reaches a graph and which addressing flags apply (RFC-010): +Every command lives on one **plane**, which determines how it reaches a graph and which addressing flags apply: - **Data plane** — `query`, `mutate`, `load`, `ingest`, `branch *`, `snapshot`, `export`, `commit *`, `schema show`, `schema apply` (and `graphs list`, remote-only today). Run against a graph **embedded or via a server**: accept a positional `URI` / `--target` / `--server` (+ `--graph` for multi-graph servers). - **Storage / maintenance plane** — `init`, `optimize`, `repair`, `cleanup`, `schema plan`, `queries validate`, `lint`. Run with **direct storage access** (`file://` / `s3://`), never through a server. They accept a positional `URI` or `--target`, but **not** `--server` / `--graph`, and a `--target` that resolves to a remote (`http(s)://`) server is rejected. (`init` takes only a positional `URI` today — no `--target`.) `optimize` / `repair` / `cleanup` also accept **`--cluster --cluster-graph `**, which resolves the graph's storage URI from the served cluster state (so you needn't know the `/graphs/.omni` layout). @@ -48,8 +48,7 @@ To maintain a server-backed graph, run the maintenance verbs from a host with st ## Config surfaces -Two config surfaces with single owners (RFC-007/RFC-008), plus a zero-config -tier: +Two config surfaces with single owners, plus a zero-config tier: | Surface | Owner | Location | Declares | |---|---|---|---| @@ -58,7 +57,7 @@ tier: | Flags / env | per invocation | — | everything, explicitly | `omnigraph.yaml` (below) is the legacy combined file — fully supported -today, slated for staged deprecation (RFC-008); its keys' future homes are +today, slated for staged deprecation; its keys' future homes are listed there. ### `~/.omnigraph/config.yaml` (operator) @@ -123,7 +122,7 @@ operator server use the legacy chain alone. ## `omnigraph.yaml` schema (legacy combined file) -> **Deprecated (RFC-008).** Loading this file prints a per-key notice +> **Deprecated.** Loading this file prints a per-key notice > naming each present key's new home (suppress in CI with > `OMNIGRAPH_SUPPRESS_YAML_DEPRECATION=1`); `omnigraph config migrate` > produces the split. The file keeps working through the deprecation diff --git a/docs/user/clusters/config.md b/docs/user/clusters/config.md index 5b2e0d5..63d9d8d 100644 --- a/docs/user/clusters/config.md +++ b/docs/user/clusters/config.md @@ -1,7 +1,5 @@ # Cluster Config -**Status:** Phase 5 — cluster-booted serving (`omnigraph-server --cluster`). - > New to the cluster tooling? Start with the operator how-to guide, > [cluster.md](index.md) — this document is the reference. diff --git a/docs/user/concepts/storage.md b/docs/user/concepts/storage.md index 9cc2356..68bfbcc 100644 --- a/docs/user/concepts/storage.md +++ b/docs/user/concepts/storage.md @@ -7,47 +7,45 @@ Every node type and every edge type is its own Lance dataset: - **Columnar Arrow storage**: each property is a column; nullable per Arrow schema. - **Fragments**: data is partitioned into fragments; new writes create new fragments. - **Manifest versioning**: every commit produces a new dataset version; old versions remain readable. -- **Stable row IDs**: `enable_stable_row_ids: true` is set on every Lance dataset OmniGraph creates — node and edge data tables, `__manifest`, `_graph_commits.lance`, `_graph_commit_recoveries.lance`, and any future system tables. This is an architectural invariant: the flag is one-way at dataset create per Lance's row-id-lineage spec, so a future change that introduces a Lance dataset must preserve it. Consequences: `_row_created_at_version` and `_row_last_updated_at_version` are available on every dataset (load-bearing for change-feed validators); `CreateIndex × Rewrite` is not a retryable conflict, so indices survive `omnigraph optimize` without needing the Fragment Reuse Index; readers must use a Lance build that recognises the flag (our pinned 4.0.0 is fine). Pre-0.4.x graphs created before this code path settled may have datasets without the flag and cannot be retrofitted in place — the supported path is dump-and-reload. The `stage_overwrite` rewrite path (used by `schema_apply`) preserves the flag through `Operation::Overwrite`; pinned by `stage_overwrite_preserves_stable_row_ids` in `crates/omnigraph/tests/staged_writes.rs`. +- **Stable row IDs**: stable row IDs are enabled on every Lance dataset OmniGraph creates — node and edge data tables, `__manifest`, the commit-graph datasets, and any future system tables. This is an architectural invariant: the flag is one-way at dataset create, so a future change that introduces a Lance dataset must preserve it. Consequences: `_row_created_at_version` and `_row_last_updated_at_version` are available on every dataset (load-bearing for change-feed validators); indices survive `omnigraph optimize`. Pre-0.4.x graphs created before this code path settled may have datasets without the flag and cannot be retrofitted in place — the supported path is dump-and-reload. The rewrite path used by `schema_apply` preserves the flag. - **Append / delete / `merge_insert`**: native Lance write modes. - **Per-dataset branches** (Lance native): copy-on-write at the dataset level. -- **Object-store agnostic**: file://, s3://, gs://, az://, http (read-only via Lance) — OmniGraph wires file:// and s3:// (`storage.rs`). +- **Object-store agnostic**: file://, s3://, gs://, az://, http (read-only via Lance) — OmniGraph wires file:// and s3://. ## L2 — Multi-dataset coordination via `__manifest` OmniGraph is **not** a single Lance dataset; it is a *graph* of datasets coordinated through one append-only manifest table. - **Manifest table**: `__manifest/` Lance dataset. -- **Layout** (`db/manifest/layout.rs`, `db/manifest/state.rs`): +- **Layout**: - `nodes/{fnv1a64-hex(type_name)}` — one Lance dataset per node type - `edges/{fnv1a64-hex(edge_type_name)}` — one Lance dataset per edge type - `__manifest/` — the catalog of all sub-tables and their published versions - `_graph_commits.lance` / `_graph_commit_actors.lance` — the commit graph and its actor map - - (legacy `_graph_runs.lance` / `_graph_run_actors.lance` from pre-v0.4.0 graphs are inert; the run state machine was removed in MR-771. The v2→v3 manifest migration sweeps stale `__run__*` branches on first write-open; the inert dataset bytes themselves remain until a `delete_prefix` storage primitive lands) + - (legacy `_graph_runs.lance` / `_graph_run_actors.lance` from pre-v0.4.0 graphs are inert; the run state machine was removed. The internal schema migration sweeps stale `__run__*` branches on first write-open; the inert dataset bytes themselves remain until a prefix-delete storage primitive lands) - **Manifest row schema** (`object_id, object_type, location, metadata, base_objects, table_key, table_version, table_branch, row_count`): - `object_type` ∈ `table | table_version | table_tombstone` - `table_key` ∈ `node: | edge:` - `table_branch` is `null` for the main lineage and the branch name otherwise - **Snapshot reconstruction**: latest visible `table_version` per `(table_key, table_branch)` minus tombstones — rows where `object_type = table_tombstone`, whose own `table_version` (acting as the tombstone version) is `>= the entry's table_version`. -- **Atomic publish**: multi-dataset commits publish via a `ManifestBatchPublisher` so a single write to `__manifest` flips all the new sub-table versions visible at once. -- **Row-level CAS on the merge-insert join key**: `object_id` carries `lance-schema:unenforced-primary-key=true` so Lance's bloom-filter conflict resolver rejects two concurrent commits that land the same `object_id` row. Without this annotation, Lance's transparent rebase would admit silent duplicates of `version:T@v=N` from racing publishers (see `.context/merge-insert-cas-granularity.md`). -- **Optimistic concurrency control on publish**: `ManifestBatchPublisher::publish` accepts a `expected_table_versions: HashMap` map. Each entry asserts the manifest's current latest non-tombstoned version for that table is exactly what the caller observed; mismatches surface as `OmniError::Manifest` with `ManifestConflictDetails::ExpectedVersionMismatch { table_key, expected, actual }`. Empty map preserves the legacy "best-effort publish" semantics. The publisher uses `conflict_retries(0)` against Lance and owns retry itself (`PUBLISHER_RETRY_BUDGET = 5`), re-running the pre-check on each iteration so concurrent advances surface as `ExpectedVersionMismatch` rather than being silently rebased through. +- **Atomic publish**: multi-dataset commits publish so that a single write to `__manifest` flips all the new sub-table versions visible at once. +- **Row-level CAS on the merge-insert join key**: `object_id` carries an unenforced-primary-key annotation so Lance's bloom-filter conflict resolver rejects two concurrent commits that land the same `object_id` row. Without this annotation, Lance's transparent rebase would admit silent duplicates from racing publishers. +- **Optimistic concurrency control on publish**: a publish asserts the manifest's current latest non-tombstoned version for each touched table is exactly what the caller observed; mismatches surface as an `ExpectedVersionMismatch` manifest conflict naming the table and the expected/actual versions. Concurrent advances surface as a conflict rather than being silently rebased through. -### Internal schema versioning (`db/manifest/migrations.rs`) +### Internal schema versioning -The on-disk shape of `__manifest` is reconciled with the binary via a single stamp + dispatcher. `INTERNAL_MANIFEST_SCHEMA_VERSION` declares the shape this binary writes; the on-disk stamp `omnigraph:internal_schema_version` lives in the manifest dataset's schema-level metadata (Lance `update_schema_metadata`). +The on-disk shape of `__manifest` is reconciled with the binary via a single version stamp held in the manifest dataset's schema-level metadata. -- **`init_manifest_graph`** stamps the current version at creation, so newly initialized graphs never need migration. -- **Publisher open-for-write path** (`load_publish_state`) calls `migrate_internal_schema(&mut dataset)` before reading state. When the on-disk stamp matches the binary, this is a single metadata read with no writes; otherwise the dispatcher walks `match`-arm steps forward (1→2, 2→3, …) until the stamp matches, then proceeds with the publish. Reads stay side-effect-free. +- **Graph creation** stamps the current version, so newly initialized graphs never need migration. +- **The open-for-write path** migrates the on-disk stamp before reading state. When the stamp matches the binary, this is a single metadata read with no writes; otherwise the migration walks steps forward (1→2, 2→3, …) until the stamp matches, then proceeds with the publish. Reads stay side-effect-free. - **Forward-version protection**: a stamp *higher* than the binary's known version triggers a clear "upgrade omnigraph first" error. An old binary cannot clobber a newer schema by silently treating "unknown stamp" as "missing stamp". -- **Idempotency**: each migration step is safe to re-run. A crash between two metadata updates inside a single step leaves the partial state; the next open re-runs the step and the second update lands. The dispatcher itself is a cheap stamp-read on the steady-state path. - -Adding a new on-disk shape change is one constant bump (`INTERNAL_MANIFEST_SCHEMA_VERSION`), one match arm in `migrate_internal_schema`, and one test. No code outside this module branches on the stamp. +- **Idempotency**: each migration step is safe to re-run. A crash between two metadata updates inside a single step leaves the partial state; the next open re-runs the step and the second update lands. | Stamp | Shape change | |---|---| -| v1 (implicit, pre-stamp) | `__manifest.object_id` had no PK annotation; publisher had no row-level CAS protection. | -| v2 | `__manifest.object_id` carries `lance-schema:unenforced-primary-key=true`; row-level CAS engaged. Stamped as `omnigraph:internal_schema_version=2`. | -| v3 | One-time sweep of legacy `__run__*` staging branches (pre-v0.4.0 Run state machine, removed MR-771) off `__manifest`. Runs at `Omnigraph::open(ReadWrite)` and on publish. Stamped as `omnigraph:internal_schema_version=3`. | +| v1 (implicit, pre-stamp) | `__manifest.object_id` had no PK annotation; no row-level CAS protection. | +| v2 | `__manifest.object_id` carries an unenforced-primary-key annotation; row-level CAS engaged. | +| v3 | One-time sweep of legacy `__run__*` staging branches (pre-v0.4.0 Run state machine, removed) off `__manifest`. Runs at read-write open and on publish. | ## On-disk layout @@ -92,20 +90,20 @@ flowchart TB - **Graph root** is one directory (or S3 prefix). Everything below is part of one OmniGraph graph. - **`__manifest/`** is a Lance dataset whose rows describe which sub-table version is published at which graph-branch. Reading a snapshot starts here. - **`nodes/`** and **`edges/`** are sibling directories holding one Lance dataset per declared type. Names are `fnv1a64-hex` of the type name to keep paths fixed-length and case-safe. -- **`_graph_commits.lance`** is an L2 dataset that records the graph-level commit DAG, with a paired `_graph_commit_actors.lance` for the actor map. (Pre-v0.4.0 graphs also have inert `_graph_runs.lance` / `_graph_run_actors.lance` from the removed Run state machine; the v2→v3 migration sweeps their stale `__run__*` branches, and the dataset bytes are reclaimed once `delete_prefix` lands.) -- **`_graph_commit_recoveries.lance`** — one row per recovery sweep action. Joined to `_graph_commits.lance` by `graph_commit_id`; the linked commit row carries `actor_id=omnigraph:recovery`. Operators correlate recoveries with the original mutations they rolled forward / back via this join. See `crates/omnigraph/src/db/recovery_audit.rs`. -- **`__recovery/{ulid}.json`** — transient sidecar files written by the five migrated writers (`MutationStaging::finalize`, `schema_apply`, `branch_merge`, `ensure_indices`, `optimize_all_tables`) before Phase B begins, deleted after Phase C succeeds. A sidecar persisting after process exit means the writer crashed in the Phase B → Phase C window; the next `Omnigraph::open` recovery sweep processes it. Steady-state directory is empty. See `crates/omnigraph/src/db/manifest/recovery.rs`. +- **`_graph_commits.lance`** is an L2 dataset that records the graph-level commit DAG, with a paired `_graph_commit_actors.lance` for the actor map. (Pre-v0.4.0 graphs also have inert `_graph_runs.lance` / `_graph_run_actors.lance` from the removed Run state machine; the internal schema migration sweeps their stale `__run__*` branches, and the dataset bytes are reclaimed once a prefix-delete primitive lands.) +- **`_graph_commit_recoveries.lance`** — one row per crash-recovery action. Joined to `_graph_commits.lance` by `graph_commit_id`; the linked commit row carries `actor_id=omnigraph:recovery`. Operators correlate recoveries with the original mutations they rolled forward / back via this join. +- **`__recovery/{ulid}.json`** — transient sidecar files written by a writer before it advances the underlying dataset, deleted once the matching manifest publish succeeds. A sidecar persisting after process exit means the writer crashed mid-commit; the next read-write open processes it. Steady-state directory is empty. - **`_refs/branches/{name}.json`** is graph-level branch metadata — pointers from a branch name to the manifest version it heads. - **Inside each Lance dataset** (orange): the standard Lance directory layout. `_versions/{n}.manifest` records every commit; `data/` holds the actual Arrow fragments; `_indices/{uuid}/` holds index segments with their own `fragment_bitmap` for partial coverage; `_refs/` holds Lance-native per-dataset branches and tags. The split — L2 owns the cross-dataset catalog; L1 owns the per-dataset internals — means that schema work (which adds or removes datasets) updates `__manifest`, while data work (which adds fragments) updates `_versions/` inside the affected dataset and then bumps `__manifest`. -## URI scheme support (`storage.rs`) +## URI scheme support | Scheme | Backend | Notes | |---|---|---| -| local path / `file://` | `ObjectStorageAdapter` over `object_store::LocalFileSystem` | Normalized to absolute paths; relative and dot-segment paths are lexically absolutized | -| `s3://bucket/prefix` | `ObjectStorageAdapter` over `object_store` S3 | Honors `AWS_ENDPOINT_URL_S3`, `AWS_ALLOW_HTTP`, `AWS_S3_FORCE_PATH_STYLE` | +| local path / `file://` | local filesystem | Normalized to absolute paths; relative and dot-segment paths are lexically absolutized | +| `s3://bucket/prefix` | S3 object store | Honors `AWS_ENDPOINT_URL_S3`, `AWS_ALLOW_HTTP`, `AWS_S3_FORCE_PATH_STYLE` | | `http(s)://host:port` | HTTP client to `omnigraph-server` | Used by CLI as a target, not a storage backend | ## Object-store env vars (S3-compatible) diff --git a/docs/user/deployment.md b/docs/user/deployment.md index 7f134c5..71cd5c8 100644 --- a/docs/user/deployment.md +++ b/docs/user/deployment.md @@ -47,8 +47,7 @@ omnigraph-server s3://my-bucket/graphs/example/releases/2026-04-10-v0.1.0 \ ## Cluster Mode in Containers (AWS, Railway) -A cluster-booted deployment has **two shapes** since the `storage:` root -(RFC-006): +A cluster-booted deployment has **two shapes** since the `storage:` root: - **Bucket, no volume (preferred for cloud)** — the cluster's ledger, catalog, and graph data live under an object-storage root diff --git a/docs/user/operations/errors.md b/docs/user/operations/errors.md index fad39a7..48f1fc9 100644 --- a/docs/user/operations/errors.md +++ b/docs/user/operations/errors.md @@ -9,7 +9,7 @@ - `Manifest(ManifestError { kind: BadRequest|NotFound|Conflict|Internal, details: Option, … })` - `ManifestConflictDetails::ExpectedVersionMismatch { table_key, expected, actual }` — caller's `expected_table_versions` did not match the manifest's current latest non-tombstoned version (set by `OmniError::manifest_expected_version_mismatch`). - `ManifestConflictDetails::RowLevelCasContention` — Lance row-level CAS rejected the publish because a concurrent writer landed the same `object_id`. Retried internally by the publisher; only surfaces if the retry budget exhausts. - - **D₂ parse-time rejection** (MR-794): a single mutation query that mixes inserts/updates with deletes errors out *before any I/O* with kind `BadRequest`. Message: `mutation '' on the same query mixes inserts/updates and deletes; split into separate mutations: (1) inserts and updates, then (2) deletes`. See [docs/user/query-language.md](../queries/index.md) for the rule and [docs/dev/writes.md](../../dev/writes.md) for the underlying staged-write rationale. + - **D₂ parse-time rejection**: a single mutation query that mixes inserts/updates with deletes errors out *before any I/O* with kind `BadRequest`. Message: `mutation '' on the same query mixes inserts/updates and deletes; split into separate mutations: (1) inserts and updates, then (2) deletes`. See [query-language.md](../queries/index.md) for the rule. - `MergeConflicts(Vec)` Compiler-side `NanoError` covers parse / catalog / type / storage / plan / execution / arrow / lance / IO / manifest / unique-constraint, each with structured spans (`SourceSpan { start, end }`) for ariadne-style diagnostics. diff --git a/docs/user/operations/maintenance.md b/docs/user/operations/maintenance.md index eeeb002..a804e31 100644 --- a/docs/user/operations/maintenance.md +++ b/docs/user/operations/maintenance.md @@ -1,49 +1,47 @@ # Maintenance: Optimize, Repair & Cleanup -`db/omnigraph/optimize.rs` and `db/omnigraph/repair.rs`. +**Addressing.** `optimize`, `repair`, and `cleanup` are **storage-plane** CLI commands: they run with direct storage access against a positional `URI`, `--target`, or **`--cluster --cluster-graph `** (which resolves the graph's storage URI from the served cluster state, so you needn't know the `/graphs/.omni` layout). They never run through a server, and reject `--server` / `--graph` or a `--target` that resolves to a remote (`http(s)://`) URL with a declared error. There are no server routes for them by design — to maintain a server-backed graph, run them out-of-band against the graph's storage URI. See the *Command planes* section of [cli-reference.md](../cli/reference.md). -**Addressing (RFC-010).** `optimize`, `repair`, and `cleanup` are **storage-plane** CLI commands: they run with direct storage access against a positional `URI`, `--target`, or **`--cluster --cluster-graph `** (which resolves the graph's storage URI from the served cluster state, so you needn't know the `/graphs/.omni` layout). They never run through a server, and reject `--server` / `--graph` or a `--target` that resolves to a remote (`http(s)://`) URL with a declared error. There are no server routes for them by design — to maintain a server-backed graph, run them out-of-band against the graph's storage URI. See the *Command planes* section of [cli-reference.md](../cli/reference.md). +## `optimize` — non-destructive -## `optimize_all_tables(db)` — non-destructive - -- Lance `compact_files()` on every node + edge table on `main`, then **publishes the compacted version to the `__manifest`** so the manifest's `table_version` tracks the compacted Lance HEAD. Reads pin the manifest version, so without this publish compaction would be invisible to readers *and* would break the HEAD-vs-manifest precondition of the next schema apply / strict update/delete ("stale view … refresh and retry"). The publish advances the graph version (a system-attributed commit) only for tables that actually compacted. -- Rewrites small fragments into fewer large ones; old fragments remain reachable via older manifests until `cleanup` runs. -- Each table's compact→publish runs under its per-`(table, main)` write queue (serializing with concurrent mutations — compaction is a Lance `Rewrite` op that retryable-conflicts with a concurrent merge/update/delete on overlapping fragments). The Lance-HEAD-before-manifest-publish gap is covered by a `SidecarKind::Optimize` recovery sidecar (loose-match): a crash in that window rolls the compacted version forward on the next `Omnigraph::open` (compaction is content-preserving, so roll-forward is always safe). -- **Requires a recovered graph.** `optimize` refuses (errors) when an unresolved recovery sidecar is present under `__recovery` — operating on an unrecovered graph could publish a partial write the open-time recovery sweep would roll back. Reopen the graph to run the recovery sweep, then re-run `optimize`. -- **Uncovered drift is skipped, not interpreted.** If a table's Lance HEAD is ahead of the version recorded in `__manifest` and no recovery sidecar covers that movement, `optimize` reports `skipped: Some(DriftNeedsRepair)` with the manifest/head versions and leaves the table untouched. Run `omnigraph repair` to classify and explicitly publish that drift. +- Compacts every node + edge table on `main`, then **publishes the compacted version to the `__manifest`** so the manifest's recorded version tracks the compacted state. Reads pin the manifest version, so without this publish compaction would be invisible to readers *and* would break the version precondition of the next schema apply / strict update/delete ("stale view … refresh and retry"). The publish advances the graph version (a system-attributed commit) only for tables that actually compacted. +- Rewrites small fragments into fewer large ones; old fragments remain reachable via older versions until `cleanup` runs. +- Each table's compact→publish serializes with concurrent mutations on the same table. A crash mid-operation is recovered automatically on the next open (compaction is content-preserving, so roll-forward is always safe). +- **Requires a recovered graph.** `optimize` refuses (errors) when a pending crash-recovery operation is present — operating on an unrecovered graph could publish a partial write that recovery would roll back. Reopen the graph to run recovery, then re-run `optimize`. +- **Uncovered drift is skipped, not interpreted.** If a table's underlying version is ahead of the version recorded in `__manifest` and no crash-recovery record covers that movement, `optimize` reports `skipped: DriftNeedsRepair` with the manifest/head versions and leaves the table untouched. Run `omnigraph repair` to classify and explicitly publish that drift. - Bounded by `OMNIGRAPH_MAINTENANCE_CONCURRENCY` (default 8). -- Returns `[TableOptimizeStats { table_key, fragments_removed, fragments_added, committed, skipped, manifest_version, lance_head_version }]`. -- **Blob tables are skipped.** A table that declares any `Blob` property is not compacted: it is reported with `skipped: Some(BlobColumnsUnsupportedByLance)` (and logged via `tracing::warn`) instead of compacted, and the rest of the sweep proceeds normally. The current Lance `compact_files` mis-decodes blob-v2 columns under its forced `BlobHandling::AllBinary` read; **reads and writes are unaffected** — only compaction is. This is gated by `LANCE_SUPPORTS_BLOB_COMPACTION` (`db/omnigraph/optimize.rs`) and removed when the upstream Lance fix lands (see [docs/dev/lance.md](../../dev/lance.md)). Consequence: fragment count and deleted-row space on blob tables are not reclaimed until then; query results are never affected. +- Returns per-table stats: `table_key, fragments_removed, fragments_added, committed, skipped, manifest_version, lance_head_version`. +- **Blob tables are skipped.** A table that declares any `Blob` property is not compacted: it is reported with `skipped: BlobColumnsUnsupportedByLance` (and logged) instead of compacted, and the rest of the sweep proceeds normally. **Reads and writes are unaffected** — only compaction is. Consequence: fragment count and deleted-row space on blob tables are not reclaimed; query results are never affected. -## `repair_all_tables(db, options)` — explicit +## `repair` — explicit -- Handles **uncovered manifest/head drift**: a table's Lance HEAD is ahead of the manifest pin and no recovery sidecar records the writer intent. -- Preview by default. `omnigraph repair --json ` reports each table's `classification`, `action`, manifest/head versions, Lance operation names, and any classification error. `--confirm` publishes only verified maintenance drift; if any suspicious or unverifiable table is refused, the CLI prints the per-table output and exits non-zero. `--force --confirm` also publishes suspicious or unverifiable drift after operator review. -- Classifies drift by reading Lance transactions from `manifest_version + 1` through `lance_head_version`. Only `ReserveFragments` and `Rewrite` are verified maintenance. Semantic operations such as `Append`, `Delete`, `Update`, `Merge`, or missing transaction history are not auto-healed. -- Publishes repair by advancing `__manifest` to the existing Lance HEAD; it does **not** rewrite Lance data. If the publish succeeds, normal reads and strict writes use the repaired version. If it fails, no new data-side partial state was created. -- Requires a clean recovery state. Pending `__recovery` sidecars still belong to automatic sidecar recovery, not manual repair. +- Handles **uncovered manifest/head drift**: a table's underlying version is ahead of the manifest pin and no crash-recovery record explains the movement. +- Preview by default. `omnigraph repair --json ` reports each table's `classification`, `action`, manifest/head versions, underlying operation names, and any classification error. `--confirm` publishes only verified maintenance drift; if any suspicious or unverifiable table is refused, the CLI prints the per-table output and exits non-zero. `--force --confirm` also publishes suspicious or unverifiable drift after operator review. +- Classifies drift by reading the table's transaction history from `manifest_version + 1` through the current head. Only fragment-reservation and rewrite (compaction) operations are verified maintenance. Semantic operations such as append, delete, update, merge, or missing transaction history are not auto-healed. +- Publishes repair by advancing `__manifest` to the existing head; it does **not** rewrite data. If the publish succeeds, normal reads and strict writes use the repaired version. If it fails, no new data-side partial state was created. +- Requires a clean recovery state. A pending crash-recovery operation still belongs to automatic recovery, not manual repair. -## `cleanup_all_tables(db, options)` — destructive +## `cleanup` — destructive -- Lance `cleanup_old_versions()` per table. -- Removes manifests (and their unique fragments) older than the retention policy. -- `CleanupPolicyOptions { keep_versions: Option, older_than: Option }` — at least one is required. -- Returns `[TableCleanupStats { table_key, bytes_removed, old_versions_removed, error }]`. +- Garbage-collects old versions per table. +- Removes versions (and their unique fragments) older than the retention policy. +- Policy options `keep_versions` and `older_than` — at least one is required. +- Returns per-table stats: `table_key, bytes_removed, old_versions_removed, error`. - **Fault-isolated per table.** A single table's transient failure (version GC or - orphan reclaim) is recorded on that table's stats row (`error: Some(..)`, logged - via `tracing`) and never aborts the healthy tables — cleanup is the convergence + orphan reclaim) is recorded on that table's stats row (with an `error`) and logged, + and never aborts the healthy tables — cleanup is the convergence backstop, so it does as much as it can and converges on re-run. The CLI reports any failed tables; rerun `cleanup` to retry them. - CLI guards with `--confirm`; without it, prints a preview line. -- **Recovery floor:** `--keep < 3` may garbage-collect Lance versions that the open-time recovery sweep needs as a rollback target (the sweep restores to the branch's manifest-pinned table version, which is HEAD-1 in the typical Phase B → Phase C drift case). Default `--keep 10` is safe. -- **Orphaned-branch reconciliation:** before the version GC, cleanup runs `reconcile_orphaned_branches`, which `force_delete_branch`es any per-table or commit-graph Lance branch absent from the manifest branch list. These orphans arise when a `branch_delete` flips the manifest authority but a downstream best-effort reclaim does not complete (see [branches-commits.md](../branching/index.md)). The reconciler is authority-derived and idempotent (it no-ops once nothing is orphaned), runs regardless of the `keep_versions` / `older_than` values (those gate version GC only), and never reclaims `main` or system-branch forks. Reclaimed forks are logged via `tracing::info`. +- **Recovery floor:** `--keep < 3` may garbage-collect versions that crash recovery needs as a rollback target. Default `--keep 10` is safe. +- **Orphaned-branch reconciliation:** before the version GC, cleanup reclaims any per-table or commit-graph branch absent from the manifest branch list. These orphans arise when a `branch_delete` flips the manifest authority but a downstream best-effort reclaim does not complete (see [branches-commits.md](../branching/index.md)). The reconciler is idempotent (it no-ops once nothing is orphaned), runs regardless of the `keep_versions` / `older_than` values (those gate version GC only), and never reclaims `main` or system-branch forks. Reclaimed forks are logged. ## Tombstones -Logical sub-table delete markers in `__manifest`; `tombstone_object_id(table_key, version)` excludes a sub-table version from snapshot reconstruction. +Logical sub-table delete markers in `__manifest` that exclude a sub-table version from snapshot reconstruction. -## Internal schema migrations (`db/manifest/migrations.rs`) +## Internal schema migrations -Version evolutions of the on-disk `__manifest` shape are reconciled automatically on the first write under a new binary. `INTERNAL_MANIFEST_SCHEMA_VERSION` declares the shape the binary expects; the on-disk stamp `omnigraph:internal_schema_version` (Lance schema-level metadata) records the on-disk shape. The publisher's open-for-write path calls `migrate_internal_schema` before reading state; reads are side-effect-free. No operator action is required for in-place upgrades. See [storage.md → Internal schema versioning](../concepts/storage.md) for the full mechanism. +Version evolutions of the on-disk `__manifest` shape are reconciled automatically on the first write under a new binary. An on-disk stamp records the shape; the binary migrates it forward before reading state, and reads are side-effect-free. No operator action is required for in-place upgrades. See [storage.md → Internal schema versioning](../concepts/storage.md) for the full mechanism. A binary opening a manifest stamped at a version *higher* than it knows about refuses to publish with a clear "upgrade omnigraph first" error — old binaries cannot clobber a newer schema. diff --git a/docs/user/operations/policy.md b/docs/user/operations/policy.md index 91684d8..159ed4d 100644 --- a/docs/user/operations/policy.md +++ b/docs/user/operations/policy.md @@ -13,7 +13,7 @@ Per-graph actions (bind to `Omnigraph::Graph::""`): 5. `branch_create` 6. `branch_delete` 7. `branch_merge` -8. `admin` — reserved for policy-management surfaces (hot reload, audit log, approvals). No call site today; see MR-724 for the reservation rationale. +8. `admin` — reserved for policy-management surfaces (hot reload, audit log, approvals). No call site today. 9. `invoke_query` — gates invoking a server-side stored query (the `queries:` registry). Graph-scoped (like `admin`) — per-branch access is enforced by the inner `read` / `change` gate, so a rule that sets `branch_scope` on `invoke_query` is rejected. Coarse in this release: an `invoke_query` allow rule permits any stored query on the graph; a future, additive refinement adds an optional per-query-name scope without changing rules written against the coarse action. Enforced at `POST /queries/{name}` (see [server](server.md)). A stored *mutation* is double-gated: `invoke_query` to reach the tool, plus `change` for the write itself (the engine `_as` writers still enforce per the query body). Server-scoped action (v0.6.0+; binds to `Omnigraph::Server::"root"`): @@ -113,20 +113,20 @@ Policy is a property of the **engine**, not the transport. Every mutating write — `mutate_as`, `load_as` (the deprecated `ingest_as` shims route through it), `apply_schema_as`, `branch_create_as`, `branch_create_from_as`, `branch_delete_as`, -`branch_merge_as` — calls `Omnigraph::enforce(action, scope, actor)` at -the head of the method. The gate fires identically whether the call +`branch_merge_as` — consults the policy gate at the head of the method. +The gate fires identically whether the call originates from the HTTP server, the CLI, or an embedded SDK consumer. -When no `PolicyChecker` is installed (the dev/embedded default) the gate +When no policy is installed (the dev/embedded default) the gate is a strict no-op; when one is installed and the call site forgets to thread an actor through, the gate fails closed rather than silently bypassing. -## Server runtime states (MR-723) +## Server runtime states The HTTP server classifies its startup configuration into one of three states based on whether bearer tokens are configured and whether a policy file is set. The state determines what happens to a request that -reaches `authorize_request()` without a matching policy permit. +reaches the authorization gate without a matching policy permit. | State | Tokens | Policy file | Behavior | |---|---|---|---| @@ -134,21 +134,17 @@ reaches `authorize_request()` without a matching policy permit. | **DefaultDeny** | yes | no | Every authenticated request for an action other than `read` is rejected with HTTP 403. Closes the "tokens but forgot the policy file" trap — an operator who sets up auth and forgot to point at a policy file used to ship the illusion of protection. | | **PolicyEnabled** | yes | yes | Authenticated requests that reach a configured policy engine are evaluated by Cedar. Server-scoped actions still require `server.policy.file`. | -The classifier is `classify_server_runtime_state` in -`crates/omnigraph-server/src/lib.rs`; it returns `Err` for the "no -tokens, no policy, no flag" cell and for "policy file, no tokens" so the -server refuses to start instead of silently shipping an open instance or -a policy-protected server that can only 401. Tests pin every cell of the -matrix and the State-2 deny path. +The server refuses to start for the "no tokens, no policy, no flag" cell +and for "policy file, no tokens" — instead of silently shipping an open +instance or a policy-protected server that can only 401. -Server-side, `authorize_request()` still runs at the HTTP boundary — +Server-side, request authorization still runs at the HTTP boundary — that's where actor identity is resolved from the bearer token and where admission control / per-actor rate limits live. Engine-layer enforcement is the **defense in depth** layer: it catches CLI direct-engine writes, embedded SDK consumers, and any future transport that hasn't (or won't) -re-implement HTTP's authorize_request. Both layers consult the same -Cedar policy via the same `PolicyChecker` trait, so decisions cannot -disagree. +re-implement the HTTP boundary's authorization. Both layers consult the same +Cedar policy, so decisions cannot disagree. ## Coarse vs. fine enforcement @@ -157,19 +153,19 @@ responsibilities: | Layer | Question it answers | Where it fires | |---|---|---| -| **Engine-layer (coarse)** | Can this actor invoke this action against this branch / branch-transition? | `Omnigraph::enforce(action, scope, actor)` at the head of every `_as` writer; one Cedar decision per call. | -| **Query-layer (fine)** | For the rows / types this action actually touches, which can the actor see or modify? | Per-row predicates pushed into DataFusion at plan time. **Not yet implemented — see MR-725.** | +| **Engine-layer (coarse)** | Can this actor invoke this action against this branch / branch-transition? | The policy gate at the head of every `_as` writer; one Cedar decision per call. | +| **Query-layer (fine)** | For the rows / types this action actually touches, which can the actor see or modify? | Per-row predicates pushed into the query plan. **Not yet implemented.** | -The engine-layer gate keeps `ResourceScope` deliberately at branch -granularity (`Graph`, `Branch`, `TargetBranch`, `BranchTransition`). +The engine-layer gate keeps its resource scope deliberately at branch +granularity (graph, branch, target branch, branch transition). Per-type and per-row authority is the query-layer's job; conflating them -in `ResourceScope` would create two places per-type policy could be +in the engine-layer scope would create two places per-type policy could be evaluated and a drift surface between them. ## Actor identity (signed-claim-only) The actor identity used for every policy decision comes from the matched bearer token — never from a client-supplied request header, query parameter, or body field. The server resolves the token at the auth middleware boundary, looks up the actor it was minted for, and overwrites whatever the handler may have placed in the policy request. Clients cannot set `actor_id` directly. -This is intentional. Trusting client-supplied identity for authorization is "asking the attacker if they're an admin" — Supabase's RLS history names the same footgun. The chokepoint lives in `authorize_request` in `crates/omnigraph-server/src/lib.rs` and is named in `docs/dev/invariants.md` Hard Invariant 11. A regression test asserts the contract: a request with `Authorization: Bearer ` plus `X-Actor-Id: actor-B` always evaluates as actor A, never as actor B. +This is intentional. Trusting client-supplied identity for authorization is "asking the attacker if they're an admin" — Supabase's RLS history names the same footgun. The chokepoint lives at the server's auth boundary: a request with `Authorization: Bearer ` plus `X-Actor-Id: actor-B` always evaluates as actor A, never as actor B. If you find yourself wanting to let clients override `actor_id` for impersonation, delegation, or service-account flows — that's a feature, but it needs explicit design (e.g., signed delegation claims, an `On-Behalf-Of` audit trail). It is not a convenience knob. diff --git a/docs/user/operations/server.md b/docs/user/operations/server.md index 8e63e99..0eb2ae8 100644 --- a/docs/user/operations/server.md +++ b/docs/user/operations/server.md @@ -1,10 +1,10 @@ # HTTP Server (`omnigraph-server`) -Axum 0.8 + tokio + utoipa-generated OpenAPI. **Two modes** (v0.6.0+): single-graph (legacy) and multi-graph (MR-668), with **two boot sources** for multi mode: `omnigraph.yaml` or — exclusively — a cluster directory (`--cluster`, RFC-005). Mode is inferred from CLI args + config shape. +Axum 0.8 + tokio + utoipa-generated OpenAPI. **Two modes** (v0.6.0+): single-graph and multi-graph, with **two boot sources** for multi mode: `omnigraph.yaml` or — exclusively — a cluster directory (`--cluster`). Mode is inferred from CLI args + config shape. ## Modes -### Single-graph mode (legacy) +### Single-graph mode `omnigraph-server ` or `omnigraph-server --target --config omnigraph.yaml`. Routes are flat — `/snapshot`, `/read`, `/branches`, etc. @@ -14,10 +14,10 @@ Axum 0.8 + tokio + utoipa-generated OpenAPI. **Two modes** (v0.6.0+): single-gra `omnigraph-server --config omnigraph.yaml` with a non-empty `graphs:` map and **no** single-mode selector (no `server.graph`, no ``, no `--target`). The server opens every configured graph in parallel at startup (bounded concurrency = 4, fail-fast on the first open error). Routes are nested under `/graphs/{graph_id}/...`. Bare flat paths return 404 in multi mode. -### Cluster-booted multi mode (Phase 5) +### Cluster-booted multi mode `omnigraph-server --cluster ` boots from the cluster catalog's **applied -revision** (`state.json` + content-addressed blobs) instead of +revision** instead of `omnigraph.yaml` — an exclusive boot source: combining it with ``, `--target`, or `--config` is a startup error, and `omnigraph.yaml` is never read in this mode. Always multi-graph routing. See @@ -42,34 +42,34 @@ If a graph declares a `queries:` registry (see [cli-reference](../cli/reference. Per-graph endpoints — same body shape across modes; URLs differ: -| Method | Single-mode path | Multi-mode path | Auth | Action | Handler | -|---|---|---|---|---|---| -| GET | `/healthz` | `/healthz` | none | — | `server_health` | -| GET | `/openapi.json` | `/openapi.json` | none | — | `server_openapi` (strips security if auth disabled; in multi mode emits cluster paths with `cluster_` operation-id prefix) | -| GET | `/snapshot?branch=` | `/graphs/{id}/snapshot?branch=` | bearer + `read` | snapshot of branch | `server_snapshot` | -| POST | `/query` | `/graphs/{id}/query` | bearer + `read` | inline read query (canonical; clean field names `query`/`name`; mutations → 400) | `server_query` | -| POST | `/read` | `/graphs/{id}/read` | bearer + `read` | **deprecated** alias of `/query` (legacy field names `query_source`/`query_name`, byte-stable response; carries `Deprecation: true` + `Link: ; rel="successor-version"`) | `server_read` | -| POST | `/export` | `/graphs/{id}/export` | bearer + `export` | NDJSON stream | `server_export` | -| POST | `/mutate` | `/graphs/{id}/mutate` | bearer + `change` | mutation (canonical; `query`/`name`; accepts legacy `query_source`/`query_name` as serde aliases) | `server_mutate` | -| POST | `/change` | `/graphs/{id}/change` | bearer + `change` | **deprecated** alias of `/mutate` (carries `Deprecation: true` + `Link: ; rel="successor-version"`) | `server_change` | -| GET | `/queries` | `/graphs/{id}/queries` | bearer + `read` | list the `mcp.expose` stored queries as a typed tool catalog | `server_list_queries` | -| POST | `/queries/{name}` | `/graphs/{id}/queries/{name}` | bearer + `invoke_query` (+ `change` for a stored mutation) | invoke a named query from the `queries:` registry; deny == 404 | `server_invoke_query` | -| GET | `/schema` | `/graphs/{id}/schema` | bearer + `read` | get current `.pg` source | `server_schema_get` | -| POST | `/schema/apply` | `/graphs/{id}/schema/apply` | bearer + `schema_apply` (target=`main`) | migrate | `server_schema_apply` | -| POST | `/load` | `/graphs/{id}/load` | bearer + `branch_create` (only when `from` is set and the branch is created) + `change` | bulk load (canonical); branch creation is opt-in via `from` — without it a missing `branch` is a 404, never an implicit fork | `server_load` (32 MB body limit) | -| POST | `/ingest` | `/graphs/{id}/ingest` | bearer + `branch_create` (only when `from` is set and the branch is created) + `change` | **deprecated** alias of `/load` (carries `Deprecation: true` + `Link: ; rel="successor-version"`) | `server_ingest` (32 MB body limit) | -| GET | `/branches` | `/graphs/{id}/branches` | bearer + `read` | list branches | `server_branch_list` | -| POST | `/branches` | `/graphs/{id}/branches` | bearer + `branch_create` | create | `server_branch_create` | -| DELETE | `/branches/{branch}` | `/graphs/{id}/branches/{branch}` | bearer + `branch_delete` | delete | `server_branch_delete` | -| POST | `/branches/merge` | `/graphs/{id}/branches/merge` | bearer + `branch_merge` | merge `source → target` | `server_branch_merge` | -| GET | `/commits?branch=` | `/graphs/{id}/commits?branch=` | bearer + `read` | list | `server_commit_list` | -| GET | `/commits/{commit_id}` | `/graphs/{id}/commits/{commit_id}` | bearer + `read` | show | `server_commit_show` | +| Method | Single-mode path | Multi-mode path | Auth | Action | +|---|---|---|---|---| +| GET | `/healthz` | `/healthz` | none | — | +| GET | `/openapi.json` | `/openapi.json` | none | — (strips security if auth disabled; in multi mode emits cluster paths with `cluster_` operation-id prefix) | +| GET | `/snapshot?branch=` | `/graphs/{id}/snapshot?branch=` | bearer + `read` | snapshot of branch | +| POST | `/query` | `/graphs/{id}/query` | bearer + `read` | inline read query (canonical; clean field names `query`/`name`; mutations → 400) | +| POST | `/read` | `/graphs/{id}/read` | bearer + `read` | **deprecated** alias of `/query` (legacy field names `query_source`/`query_name`, byte-stable response; carries `Deprecation: true` + `Link: ; rel="successor-version"`) | +| POST | `/export` | `/graphs/{id}/export` | bearer + `export` | NDJSON stream | +| POST | `/mutate` | `/graphs/{id}/mutate` | bearer + `change` | mutation (canonical; `query`/`name`; accepts legacy `query_source`/`query_name` as serde aliases) | +| POST | `/change` | `/graphs/{id}/change` | bearer + `change` | **deprecated** alias of `/mutate` (carries `Deprecation: true` + `Link: ; rel="successor-version"`) | +| GET | `/queries` | `/graphs/{id}/queries` | bearer + `read` | list the `mcp.expose` stored queries as a typed tool catalog | +| POST | `/queries/{name}` | `/graphs/{id}/queries/{name}` | bearer + `invoke_query` (+ `change` for a stored mutation) | invoke a named query from the `queries:` registry; deny == 404 | +| GET | `/schema` | `/graphs/{id}/schema` | bearer + `read` | get current `.pg` source | +| POST | `/schema/apply` | `/graphs/{id}/schema/apply` | bearer + `schema_apply` (target=`main`) | migrate | +| POST | `/load` | `/graphs/{id}/load` | bearer + `branch_create` (only when `from` is set and the branch is created) + `change` | bulk load (canonical); branch creation is opt-in via `from` — without it a missing `branch` is a 404, never an implicit fork (32 MB body limit) | +| POST | `/ingest` | `/graphs/{id}/ingest` | bearer + `branch_create` (only when `from` is set and the branch is created) + `change` | **deprecated** alias of `/load` (carries `Deprecation: true` + `Link: ; rel="successor-version"`) (32 MB body limit) | +| GET | `/branches` | `/graphs/{id}/branches` | bearer + `read` | list branches | +| POST | `/branches` | `/graphs/{id}/branches` | bearer + `branch_create` | create | +| DELETE | `/branches/{branch}` | `/graphs/{id}/branches/{branch}` | bearer + `branch_delete` | delete | +| POST | `/branches/merge` | `/graphs/{id}/branches/merge` | bearer + `branch_merge` | merge `source → target` | +| GET | `/commits?branch=` | `/graphs/{id}/commits?branch=` | bearer + `read` | list | +| GET | `/commits/{commit_id}` | `/graphs/{id}/commits/{commit_id}` | bearer + `read` | show | Server-level management endpoints (v0.6.0+): -| Method | Path | Auth | Action | Handler | -|---|---|---|---|---| -| GET | `/graphs` | bearer + `graph_list` on `Server::"root"` | list registered graphs | `server_graphs_list` (405 in single mode) | +| Method | Path | Auth | Action | +|---|---|---|---| +| GET | `/graphs` | bearer + `graph_list` on `Server::"root"` | list registered graphs (405 in single mode) | ### Stored-query catalog (`GET /queries`) @@ -96,9 +96,8 @@ or remove graphs by stopping the server, editing the `graphs:` map in `omnigraph.yaml`, then restarting. The server treats `omnigraph.yaml` as operator-owned configuration and never writes it. -A future release may introduce a managed registry (Lance-backed, -catalog-style: reserve → init → publish with recovery sidecars) and -re-expose runtime mutation on top of it. +A future release may introduce a managed registry and re-expose runtime +mutation on top of it. ## Inline read queries (`POST /query`) @@ -154,7 +153,7 @@ Only `/export` streams (`application/x-ndjson`, MPSC channel + `Body::from_strea Uniform `ErrorOutput { error, code?, merge_conflicts[], manifest_conflict? }` with `code ∈ unauthorized | forbidden | bad_request | not_found | conflict | too_many_requests | internal`. Merge conflicts attach structured `MergeConflictOutput { table_key, row_id?, kind, message }`. -`manifest_conflict` is set on **publisher CAS rejections** (HTTP 409): the +`manifest_conflict` is set on **concurrent-write rejections** (HTTP 409): the caller's pre-write view of one table's manifest version was stale. `ManifestConflictOutput { table_key, expected, actual }` tells the client which table to refresh and retry. This is the conflict shape produced by @@ -169,8 +168,8 @@ Disjoint `(table, branch)` writes from different actors now run concurrently, guarded only by the engine's per-(table, branch) write queue. To keep one heavy actor from exhausting shared capacity (Lance I/O, manifest -churn, network), the server gates mutating handlers through a -`WorkloadController` configured per-process from environment variables: +churn, network), the server gates mutating handlers through per-process +admission limits configured from environment variables: | Env var | Default | Purpose | |---|---|---| @@ -199,7 +198,7 @@ admission-gated. ## Auth model (`bearer + SHA-256`) - Tokens are SHA-256 hashed on startup; plaintext is never persisted in memory. -- Constant-time comparison via `subtle::ConstantTimeEq`. +- Constant-time comparison. - Three sources, in precedence: 1. `OMNIGRAPH_SERVER_BEARER_TOKENS_AWS_SECRET` — AWS Secrets Manager (build with `--features aws`) 2. `OMNIGRAPH_SERVER_BEARER_TOKENS_FILE` or `OMNIGRAPH_SERVER_BEARER_TOKENS_JSON` — JSON `{actor_id: token, …}` diff --git a/docs/user/queries/index.md b/docs/user/queries/index.md index c00d1a9..c8a70c5 100644 --- a/docs/user/queries/index.md +++ b/docs/user/queries/index.md @@ -1,7 +1,5 @@ # Query Language (`.gq`) -Pest grammar at `crates/omnigraph-compiler/src/query/query.pest`. AST in `query/ast.rs`. Type checker in `query/typecheck.rs`. Lowering in `ir/lower.rs`. - ## Query declarations ``` @@ -49,40 +47,19 @@ Param types reuse all schema scalars; trailing `?` makes a param optional. The c Write statements (`insert` / `update` / `delete`) are documented on the [mutations](../mutations/index.md) page. -## IR (Intermediate Representation) +## Traversal execution -`QueryIR { name, params, pipeline: Vec, return_exprs, order_by, limit }` +Variable-length traversals (`Expand`) are executed one of two ways, chosen per-expand by a cost model over cheap manifest counts (frontier size, edge count, source-vertex count, hops) plus index coverage: selective traversals (small frontier relative to the source set) resolve neighbors from the persisted `src`/`dst` BTREE (one indexed scan per hop); dense / deep / large-frontier traversals — or those whose BTREE coverage is degraded so a full scan would be paid per hop — use an in-memory CSR adjacency index. Both produce identical results. The `OMNIGRAPH_EXPAND_INDEXED_MAX_FRONTIER` / `OMNIGRAPH_EXPAND_INDEXED_MAX_HOPS` ceilings bound the *initial dispatch* frontier/hops (beyond them CSR is always used); the cost model estimates total indexed work as ~`hops × frontier × fanout` and prices dense fan-out toward CSR — they are not a hard per-hop bound. `OMNIGRAPH_TRAVERSAL_MODE=indexed|csr` forces a mode (see [constants](../reference/constants.md)). -Pipeline operations: - -- `NodeScan { variable, type_name, filters }` -- `Expand { src_var, dst_var, edge_type, direction (Out|In), dst_type, min_hops, max_hops, dst_filters }` — destination filters are pushed *into* the expand so Lance scalar pushdown can prune. Executed one of two ways, chosen per-expand by a cost model over cheap manifest counts (frontier size, |E|, source-vertex count, hops) plus index coverage: selective traversals (small frontier relative to the source set) resolve neighbors from the persisted `src`/`dst` BTREE (one indexed scan per hop); dense / deep / large-frontier traversals — or those whose BTREE coverage is degraded so a full scan would be paid per hop — use the in-memory CSR adjacency index. Both produce identical results. The `OMNIGRAPH_EXPAND_INDEXED_MAX_FRONTIER` / `OMNIGRAPH_EXPAND_INDEXED_MAX_HOPS` ceilings bound the *initial dispatch* frontier/hops (beyond them CSR is always used); the cost model estimates total indexed work as ~`hops × frontier × fanout` and prices dense fan-out toward CSR — they are not a hard per-hop bound. `OMNIGRAPH_TRAVERSAL_MODE=indexed|csr` forces a mode (see [constants](../reference/constants.md)). -- `Filter { left, op, right }` -- `AntiJoin { outer_var, inner: Vec }` — for `not { … }` - -Lowering: - -1. Partition MATCH clauses (bindings, traversals, filters, negations). -2. Identify "deferred" bindings (a destination of a traversal that has filters) so the Expand can carry the filter as a pushdown. -3. Emit NodeScan for the first binding, then Expand operations, then remaining Filter operations, then AntiJoins for negations. -4. Translate RETURN / ORDER expressions; preserve LIMIT. - -## Linting & validation (`query/lint.rs`) +## Linting & validation Codes seen so far: - **Q000** (Error): parse error - **L201** (Warning): nullable property never set by any UPDATE — "{type}.{prop} exists in schema but no update query sets it" - (Warning): mutation declares no params — hardcoded mutations are easy to miss -- Plus all type errors from `typecheck_query_decl()` (undefined types, mismatched operators, undefined edges, etc.) +- Plus all type errors from type checking (undefined types, mismatched operators, undefined edges, etc.) -Output: - -``` -QueryLintOutput { status, schema_source, query_path, - queries_processed, errors, warnings, infos, - results: [{ name, kind, status, error?, warnings[] }], - findings: [{ severity, code, message, type_name?, property?, query_names[] }] } -``` +Lint output reports an overall status, per-query results (name, kind, status, any error and warnings), and structured findings (severity, code, message, and the type/property/query they apply to). CLI exits non-zero only on `status = Error`. diff --git a/docs/user/reference/constants.md b/docs/user/reference/constants.md index f523042..2cad0d1 100644 --- a/docs/user/reference/constants.md +++ b/docs/user/reference/constants.md @@ -1,26 +1,26 @@ # Constants & Tunables (cheat sheet) -| Name | Value | Where | +| Name | Value | Area | |---|---|---| -| `MANIFEST_DIR` | `__manifest` | `db/manifest/layout.rs` | -| Commit graph dir | `_graph_commits.lance` | `db/commit_graph.rs` | -| Run registry dir (legacy, removed MR-771) | `_graph_runs.lance` | inert post-v0.4.0; bytes remain until a `delete_prefix` primitive lands | -| Run branch prefix (legacy, removed MR-771/MR-770) | `__run__` | swept off `__manifest` by the v2→v3 migration; no longer a reserved name | -| Schema apply lock | `__schema_apply_lock__` | `db/mod.rs` | -| Manifest publisher retry budget | `PUBLISHER_RETRY_BUDGET = 5` | `db/manifest/publisher.rs` | -| Internal manifest schema version | `INTERNAL_MANIFEST_SCHEMA_VERSION = 3` | `db/manifest/migrations.rs` | -| Merge stage batch | `MERGE_STAGE_BATCH_ROWS = 8192` | `exec/merge.rs` | -| Maintenance concurrency | `OMNIGRAPH_MAINTENANCE_CONCURRENCY=8` | `db/omnigraph/optimize.rs` | -| Lance blob compaction support | `LANCE_SUPPORTS_BLOB_COMPACTION = false` | `db/omnigraph/optimize.rs` | -| Graph index cache size | `8` (LRU) | `runtime_cache.rs` | -| Expand indexed-path frontier ceiling | `OMNIGRAPH_EXPAND_INDEXED_MAX_FRONTIER=1024` | `exec/query.rs` | -| Expand indexed-path hop ceiling | `OMNIGRAPH_EXPAND_INDEXED_MAX_HOPS=6` | `exec/query.rs` | -| Expand CSR-build cost factor | `CSR_BUILD_FACTOR = 1.5` | `exec/query.rs` | -| Expand mode override | `OMNIGRAPH_TRAVERSAL_MODE` (`indexed`\|`csr`; unset = cost-based auto) | `exec/query.rs` | -| Default body limit | `1 MB` | `omnigraph-server/lib.rs` | -| Ingest body limit | `32 MB` | `omnigraph-server/lib.rs` | -| Engine embed model | `gemini-embedding-2-preview` | `omnigraph/embedding.rs` | -| Compiler embed model | `text-embedding-3-small` | `omnigraph-compiler/embedding.rs` | +| `MANIFEST_DIR` | `__manifest` | manifest layout | +| Commit graph dir | `_graph_commits.lance` | commit graph | +| Run registry dir (legacy, removed) | `_graph_runs.lance` | inert post-v0.4.0; bytes remain until a prefix-delete primitive lands | +| Run branch prefix (legacy, removed) | `__run__` | swept off `__manifest` by the internal schema migration; no longer a reserved name | +| Schema apply lock | `__schema_apply_lock__` | schema apply | +| Manifest publisher retry budget | `PUBLISHER_RETRY_BUDGET = 5` | manifest publish | +| Internal manifest schema version | `INTERNAL_MANIFEST_SCHEMA_VERSION = 3` | manifest migrations | +| Merge stage batch | `MERGE_STAGE_BATCH_ROWS = 8192` | merge execution | +| Maintenance concurrency | `OMNIGRAPH_MAINTENANCE_CONCURRENCY=8` | optimize/cleanup | +| Lance blob compaction support | `LANCE_SUPPORTS_BLOB_COMPACTION = false` | optimize | +| Graph index cache size | `8` (LRU) | runtime cache | +| Expand indexed-path frontier ceiling | `OMNIGRAPH_EXPAND_INDEXED_MAX_FRONTIER=1024` | traversal | +| Expand indexed-path hop ceiling | `OMNIGRAPH_EXPAND_INDEXED_MAX_HOPS=6` | traversal | +| Expand CSR-build cost factor | `CSR_BUILD_FACTOR = 1.5` | traversal | +| Expand mode override | `OMNIGRAPH_TRAVERSAL_MODE` (`indexed`\|`csr`; unset = cost-based auto) | traversal | +| Default body limit | `1 MB` | HTTP server | +| Ingest body limit | `32 MB` | HTTP server | +| Engine embed model | `gemini-embedding-2-preview` | engine embedding | +| Compiler embed model | `text-embedding-3-small` | compiler embedding | | Embed timeout | `30 000 ms` | both clients | | Embed retries | `4` | both clients | | Embed retry backoff | `200 ms` | both clients | diff --git a/docs/user/schema/index.md b/docs/user/schema/index.md index 4250676..d0fcd1b 100644 --- a/docs/user/schema/index.md +++ b/docs/user/schema/index.md @@ -1,7 +1,5 @@ # Schema Language (`.pg`) -Pest grammar at `crates/omnigraph-compiler/src/schema/schema.pest`. AST at `schema/ast.rs`. Catalog at `catalog/mod.rs`. - ## Top-level declarations - `interface { property* }` — reusable property contracts. @@ -47,37 +45,28 @@ Edge bodies only allow `@unique` and `@index`. - `@` or `@()` on any declaration or property. - Known annotations: - - `@embed` on a Vector property — names the *source* property whose text gets embedded into this vector at ingest (`embed_sources` map in NodeType). + - `@embed` on a Vector property — names the *source* property whose text gets embedded into this vector at ingest. - `@description("…")`, `@instruction("…")` on query declarations (carried through to clients). - Custom annotations are accepted by the parser and surfaced in catalog metadata; unrecognized annotations don't fail compilation. -## Catalog construction +## Table layout -- Pass 0: collect interfaces. -- Pass 1: collect nodes, expand `implements`, build constraint and `@embed` mappings, build the Arrow schema for each node table (`id: Utf8` plus all properties; blob columns get `LargeBinary`). -- Pass 2: collect edges, validate that `from_type` / `to_type` exist, normalize edge names case-insensitively for lookup, validate constraints for edges. Edge Arrow schema: `id: Utf8, src: Utf8, dst: Utf8` plus edge properties. - -## Schema IR & stable type IDs - -- `SCHEMA_IR_VERSION = 1` (`catalog/schema_ir.rs`). -- Each interface/node/edge currently gets a `stable_type_id` from a kind+name hash. -- Rename-preserving accepted IDs are an architectural invariant, but the current hash-on-name implementation is a known gap until migration carries IDs across `@rename_from`. -- Serialized as JSON for diff/migration plans. +- Each node type compiles to a table with an `id: Utf8` column plus all declared properties (blob columns are stored as `LargeBinary`); `implements` clauses expand the interface's properties into the node. +- Each edge type compiles to a table with `id: Utf8, src: Utf8, dst: Utf8` plus the edge's own properties. Edge endpoint types (`from`/`to`) must exist, and edge names are matched case-insensitively. ## Schema migration planning -`plan_schema_migration(accepted, desired) -> SchemaMigrationPlan { supported, steps[] }` with step types: +A migration plan compares the accepted schema against the desired one and reports whether the change is supported plus the ordered steps it requires: -- `AddType { type_kind, name }` -- `RenameType { type_kind, from, to }` -- `AddProperty { type_kind, type_name, property_name, property_type }` -- `RenameProperty { type_kind, type_name, from, to }` -- `AddConstraint { type_kind, type_name, constraint }` -- `UpdateTypeMetadata { … annotations }` -- `UpdatePropertyMetadata { … annotations }` -- `UnsupportedChange { entity, reason }` (forces `supported=false`) +- Add a type +- Rename a type +- Add a property +- Rename a property +- Add a constraint +- Update type or property metadata (annotations) +- Unsupported change (reports the entity and reason; forces the plan to unsupported) -`apply_schema()` returns `SchemaApplyResult { supported, applied, manifest_version, steps }` and is gated by an internal `__schema_apply_lock__` system branch so concurrent schema applies serialize. +Applying a plan reports whether it was supported, the steps applied, and the resulting manifest version. Concurrent schema applies serialize so they can't interleave. ## Destructive drops — `--allow-data-loss` diff --git a/docs/user/schema/lint.md b/docs/user/schema/lint.md index a1495fd..6635e9f 100644 --- a/docs/user/schema/lint.md +++ b/docs/user/schema/lint.md @@ -2,29 +2,26 @@ The migration planner emits **code-tagged diagnostics** for every schema change it rejects. Codes have the form `OG-XXX-NNN` and identify the rule (not the message); operators reference them in suppression directives, severity overrides, and CI reports. -This page is the catalog of codes shipped today. The chassis behind it is tracked in [MR-694](https://linear.app/modernrelay/issue/MR-694). +This page is the catalog of codes shipped today. -## What's shipped in v0 +## What's shipped -- Stable code attached to every rejection the planner emits (today: 5 of 17 paths — the rest carry `code: None` and are tagged as future work). +- Stable code attached to every rejection the planner emits (today: 5 of 17 paths — the rest are tagged as future work). - Code appears in the user-visible error message: `[OG-DS-104] removing property 'Person.age' is not supported …`. - CLI `omnigraph schema plan` shows the code on `unsupported change …` lines. -- Tests in `tests/schema_apply.rs` assert on codes, not on free-text prose. ## What's not shipped yet -- Severity configuration in `omnigraph.yaml` (planned: `lint: { OG-DS-103: error }`). +- Severity configuration (planned: `lint: { OG-DS-103: error }`). - `@allow(OG-XXX-NNN, "rationale")` suppression directives. -- Pre-migration checks (the `migration_check { … }` block — MR-941). -- The CD / VE / LK / NM families (MR-942..945). -- CI integration (MR-946). -- Cost-class annotations (MR-944). +- Pre-migration checks (the `migration_check { … }` block). +- The CD / VE / LK / NM families. +- CI integration. +- Cost-class annotations. -See the parent chassis issue (MR-694) for the design and the per-family sub-issues for what's planned. +## Code catalog -## Code catalog (v0) - -The chassis defines ten families. Today only DS and MF have emitted codes. The remaining families are reserved for future PRs. +The chassis defines ten families. Today only DS and MF have emitted codes. The remaining families are reserved for future releases. | Code | Family | Tier | Default severity | Meaning | |---|---|---|---|---| @@ -37,24 +34,22 @@ The chassis defines ten families. Today only DS and MF have emitted codes. The r | `OG-MF-104` | Maybe-fail | validated | error | tighten nullable to non-nullable (reserved) | | `OG-MF-106` | Maybe-fail | destructive | error | narrowing scalar type | -The full code catalog source of truth lives in `crates/omnigraph-compiler/src/lint/codes.rs`. CI-level invariants (uniqueness, format, family coverage) are unit-tested in the same module. - ## Families The ten chassis families: | Prefix | Family | Status | |---|---|---| -| **DS** | Destructive (data-loss) | shipped, v0 | -| **MF** | Maybe-fail / data-dependent | shipped, v0 | -| **CD** | Constraint deletion (relaxation warning) | tracked in MR-942 | +| **DS** | Destructive (data-loss) | shipped | +| **MF** | Maybe-fail / data-dependent | shipped | +| **CD** | Constraint deletion (relaxation warning) | planned | | **BC** | Backward-incompatible (rename) | implicit in `@rename_from`; codify later | -| **NM** | Naming conventions | tracked in MR-945 | -| **OW** | Ownership (per-resource Cedar) | tracked in MR-722 | -| **NL** | Non-linear (branch-merge divergence) | stubbed in MR-947 | -| **VE** | Vector / embedding | tracked in MR-943 | -| **ED** | Edge / graph topology | tracked in MR-701, MR-943 | -| **LK** | Lock duration / cost | tracked in MR-944 | +| **NM** | Naming conventions | planned | +| **OW** | Ownership (per-resource Cedar) | planned | +| **NL** | Non-linear (branch-merge divergence) | planned | +| **VE** | Vector / embedding | planned | +| **ED** | Edge / graph topology | planned | +| **LK** | Lock duration / cost | planned | ## Prior art diff --git a/docs/user/search/embeddings.md b/docs/user/search/embeddings.md index 382e683..31455c4 100644 --- a/docs/user/search/embeddings.md +++ b/docs/user/search/embeddings.md @@ -2,14 +2,14 @@ OmniGraph has **two** embedding clients with different defaults and purposes. -## Compiler-side client (`omnigraph-compiler/src/embedding.rs`) — query-time normalization +## Compiler-side client — query-time normalization - Default model: `text-embedding-3-small` (OpenAI-style schema) - Env: `NANOGRAPH_EMBED_MODEL`, `OPENAI_API_KEY`, `OPENAI_BASE_URL` (default `https://api.openai.com/v1`), `NANOGRAPH_EMBEDDINGS_MOCK`, `NANOGRAPH_EMBED_TIMEOUT_MS=30000`, `NANOGRAPH_EMBED_RETRY_ATTEMPTS=4`, `NANOGRAPH_EMBED_RETRY_BACKOFF_MS=200` - Methods: `embed_text(input, expected_dim)`, `embed_texts(inputs, expected_dim)` - Mock mode: deterministic FNV-1a + xorshift64 → L2-normalized vectors -## Engine-side client (`omnigraph/src/embedding.rs`) — runtime ingest +## Engine-side client — runtime ingest - Model: `gemini-embedding-2-preview` - Env: `GEMINI_API_KEY`, `OMNIGRAPH_GEMINI_BASE_URL` (default Google generativelanguage v1beta), `OMNIGRAPH_EMBED_TIMEOUT_MS=30000`, `OMNIGRAPH_EMBED_RETRY_ATTEMPTS=4`, `OMNIGRAPH_EMBED_RETRY_BACKOFF_MS=200`, `OMNIGRAPH_EMBEDDINGS_MOCK` diff --git a/docs/user/search/indexes.md b/docs/user/search/indexes.md index fde9488..84b968d 100644 --- a/docs/user/search/indexes.md +++ b/docs/user/search/indexes.md @@ -15,12 +15,11 @@ - **Lazy branch forking for indexes**: a branch that hasn't mutated a sub-table doesn't need its own index — the main lineage's index is reused until the first write triggers a copy-on-write fork. - Vector index parameters (metric, nlist, nprobe, etc.) are not exposed in the schema; they default at the Lance layer and are picked up automatically when an index is asked for on a Vector column. -## L2 — Graph topology index (`graph_index/mod.rs`) +## L2 — Graph topology index This is OmniGraph-specific (not Lance): -- `TypeIndex`: dense `u32 ↔ String id` mapping per node type. -- `CsrIndex`: Compressed Sparse Row representation of edges per edge type — `offsets[i]..offsets[i+1]` slices into `targets`. -- `GraphIndex { type_indices, csr (out), csc (in) }` — built on demand from a snapshot's edge tables, **lazily**: only when an `Expand` the planner routes to the CSR path (dense / large frontier) or an `AntiJoin` actually needs it. -- Cached in `RuntimeCache::graph_indices` (LRU, max 8 entries, keyed by snapshot id + edge table versions). -- Selective `Expand`s resolve neighbors from the persisted `src`/`dst` BTREE instead (one indexed scan per hop) and never trigger the CSR build; see [query-language](../queries/index.md) → Expand. Pure scans, and queries served entirely by the indexed traversal path, skip it. +- A Compressed Sparse Row (CSR) adjacency representation of edges, with both out- (CSR) and in- (CSC) directions, plus a dense per-node-type id mapping. +- Built on demand from a snapshot's edge tables, **lazily**: only when an `Expand` the planner routes to the CSR path (dense / large frontier) or an `AntiJoin` actually needs it. +- Cached per snapshot (LRU, keyed by snapshot id + edge table versions), so repeat traversals over the same snapshot reuse it. +- Selective `Expand`s resolve neighbors from the persisted `src`/`dst` BTREE instead (one indexed scan per hop) and never trigger the CSR build; see [query-language](../queries/index.md) → Traversal execution. Pure scans, and queries served entirely by the indexed traversal path, skip it.