mirror of
https://github.com/ModernRelay/omnigraph.git
synced 2026-06-30 02:49:39 +02:00
0dcdcf5a9d
* test(engine): pin zero-row cascade delete must not drift an edge table (red) A delete <Node> cascades a delete_where into every incident edge type. The inline delete_where (Dataset::delete) advances Lance HEAD even when zero edges match, but the cascade records the new version only if deleted_rows > 0 — so a node with no incident edges leaves edge:Knows HEAD>manifest drift, which trips the next strict write's ExpectedVersionMismatch and repair refuses it. Red today: edge:Knows manifest=v5, Lance HEAD=v6. Goes green when delete moves to the staged two-phase path (iss-950, Lance 7.0 DeleteBuilder::execute_uncommitted), where a 0-row delete commits no Lance version and the deleted_rows>0 gate becomes correct by construction. * fix(engine): a zero-row delete must not advance Lance HEAD Lance's Dataset::delete commits a new version even when the predicate matches nothing (build_transaction always emits Operation::Delete), so a node delete that cascades a delete_where into an incident edge type with no matching edges advanced that edge table's Lance HEAD while the cascade skipped record_inline (gated on deleted_rows > 0) — leaving HEAD>manifest drift that wedged the next strict write and that repair refused as suspicious/unverifiable. Use Lance 7.0's two-phase DeleteBuilder::execute_uncommitted to read num_deleted_rows before committing: a no-match delete now advances nothing (no version, no drift) and the existing deleted_rows>0 gate is correct by construction. Non-zero deletes commit the staged transaction with skip_auto_cleanup + affected_rows (parity with the prior inline path). First step of the staged-delete migration (iss-950); turns the node_delete_with_no_incident_edges_leaves_no_edge_table_drift regression green. * feat(engine): stage_delete two-phase primitive (MR-A step 0) Add TableStore::stage_delete (Lance 7.0 DeleteBuilder::execute_uncommitted), the two-phase analogue of stage_merge_insert: writes deletion files without advancing Lance HEAD, returns Option<StagedWrite> (None on 0 rows = true no-op), carrying the deletion-vector updated_fragments as new_fragments and the superseded originals as removed_fragment_ids so combine_committed_with_staged makes the deletion visible to in-query reads. No affected_rows is threaded: like stage_merge_insert's Operation::Update commit, the staged delete relies on OmniGraph's per-table write queue + manifest CAS, not Lance's per-dataset conflict resolver (commit_staged is a single attempt). Flip the two residual guards to the staged path: staged_writes.rs now asserts stage_delete does NOT advance HEAD and that a staged delete is read-your-writes visible (the deletion-vector RYW proof D2 retirement depends on); the lance_surface_guards delete guard pins execute_uncommitted's UncommittedDelete. No behavior change yet (callers still use delete_where); Step 1 wires them. * feat(engine): TableStorage::stage_delete + migrate merge delete path (MR-A step 1a) Add stage_delete/Option<StagedHandle> to the TableStorage trait (delegates to TableStore::stage_delete). Migrate the two branch_merge delete sites (three-way RewriteMerged + adopt delta) from the inline delete_where residual to stage_delete + commit_staged — identical in shape to the stage_merge_insert + commit_staged pair above each. HEAD still advances within the merge sequence (via commit_staged), under the unchanged SidecarKind::BranchMerge Phase-B confirmation; the _pre_delete/_pre_index failpoints fire by position, unchanged. merge_truth_table, branching, composite_flow green. * feat(engine): migrate all delete sites to staged path, retire inline delete (MR-A step 1b/1c) Routes every delete through the staged write path so delete never advances Lance HEAD inline — the last inline-commit residual on the mutation path is gone. `MutationStaging` now accumulates delete predicates (`record_delete`) alongside pending write batches; at end-of-query `stage_all` combines a table's predicates into one `(p1) OR (p2) …` `stage_delete` (a deletion-vector transaction, no HEAD advance) and `commit_all` commits it through the same `commit_staged` path as inserts/updates. Deletes are now ordinary staged entries: one sidecar pin at `expected + 1`, no inline special-casing. Migrated callers (all 5): the 3 mutation.rs sites (delete-node, cascade, delete-edge) and the 2 merge.rs sites (already on stage_delete in step 1a). `affected_edges`/`affected` move from post-inline-commit `deleted_rows` to a committed `count_rows` at record time — exact under D₂, bounded by the cascade working set. A predicate matching zero rows stages nothing (the staged equivalent of the old "skip record_inline on 0 deleted rows"), so the zero-row edge-table drift class stays closed by construction. Retired scaffolding now that no caller remains: - `MutationStaging.inline_committed` + `record_inline` → `delete_predicates` + `record_delete`; `StagedMutation.inline_committed`/`paths` fields and all the `commit_all` inline handling (queue keys, sidecar pins with the `record_inline` table_version special-case, the inline recheck loop). - `open_table_for_mutation`'s post-inline-commit reopen branch (deletes no longer advance HEAD mid-query, so a second touch reopens at the pinned version like any write). - `InlineCommitResidual::delete_where` + its `TableStore` impl, the orphaned `TableStore::delete_where`, and `DeleteState`. `InlineCommitResidual` now carries only `create_vector_index` (Lance #6666 still open). D₂ stays for now: staged-delete read-your-writes doesn't yet compose into the pending accumulator (insert-then-delete on one table), so mixed insert/update/delete in one query is still rejected at parse time. Retiring D₂ is step 2. Doc comments updated to match across exec/, storage_layer, db/. Tests (all green): writes, consistency, validators, end_to_end, composite_flow, merge_truth_table, maintenance, recovery, staged_writes, forbidden_apis, lance_surface_guards, changes, point_in_time (286), plus failpoints (63). * docs: delete is a staged write, not an inline-commit residual (MR-A step 1) Update the docs that described `delete` as the inline-commit residual now that MR-A routes it through `stage_delete`. Always-loaded surfaces (AGENTS.md rule 4 / capability matrix, invariants.md Invariant 4 / truth matrix / known gaps) plus the dev write-path docs (writes.md, execution.md incl. its mutation sequence diagram, architecture.md) now state: deletes accumulate as predicates and stage like inserts/updates, no inline HEAD advance; `InlineCommitResidual` carries only `create_vector_index` (Lance #6666). The parse-time D₂ rule is documented as retained — not because delete inline-commits, but because staged-delete read-your-writes is not yet wired into the pending accumulator (MR-A step 2). lance.md's 7.0 audit note marked MR-A as landed. * docs: D₂ is a deliberate boundary, not temporary scaffolding (MR-A close-out) After MR-A staged the delete path, D₂ (a mutation query is insert/update-only OR delete-only) was left framed as temporary — "until Lance ships two-phase delete" / "retire in step 2". Lance shipped that and we used it for the inline-commit fix; D₂'s original justification is gone. It now stands for a different, permanent reason: keeping a query to one kind keeps its read-your-writes unambiguous and each table to one version per query. Retiring it would buy single-commit mixed atomicity (cheap workaround: split, or a branch) at the cost of an in-query delete view, pending pruning, edge id-resolution, and two-commit-per-table ordering in the hot mutation path — complexity not worth earning. Decision: keep D₂ as a deliberate boundary. Reframes the now-stale wording everywhere, no logic change: - The D₂ parse-time error message no longer promises "this restriction lifts when Lance exposes a two-phase delete API"; it states the boundary and points to a branch+merge for one atomic commit. - `enforce_no_mixed_destructive_constructive` doc, AGENTS.md, invariants.md (Invariant 4 / truth matrix / removed from the known-gaps), writes.md, architecture.md, lance.md, and the user mutations doc (which wrongly said deletes "commit through a different path" — both stage now). - Swept remaining stale `delete_where` mentions left from the Step-1 migration: the merge.rs "swap when upstream ships" comments (already swapped), the forbidden_apis / table_ops residual notes, the staged_writes vector-index guard doc (was "same as stage_delete's absence" — stage_delete now exists), and test comments/assert messages in recovery/maintenance/writes/failpoints. Genuinely-historical records (dated Lance audit, rfc-013, bug-case-fix) left. Verified: engine builds warning-free; check-agents-md OK; writes/maintenance/ recovery/staged_writes/forbidden_apis all green. Closes MR-A. * test(engine): overlapping delete predicates must not double-count affected_* (red) Reproduces a reporting regression from the staged-delete migration flagged in PR #308 review. Because deletes now stage (instead of inline-committing), two delete statements in one query both scan the same unchanged committed snapshot; counting each predicate independently over-reports `affected_*` when they overlap. The old inline path committed each delete before the next ran, so it counted distinct. `delete Person where name = "Alice"` then `delete Person where age > 29` over the standard fixture (Alice 30, Charlie 35) removes 2 distinct nodes and 3 distinct edges, but the buggy per-statement counting returns 3 nodes / 6 edges. RED at this commit (asserts left=3, right=2). * fix(engine): dedup overlapping delete predicates when counting affected_* Count each delete statement against the committed snapshot MINUS the predicates a prior delete statement on the same table already recorded: `(pred) AND NOT ((prior1) OR (prior2) …)`. Summed over statements this is inclusion-exclusion — `Σ |pₙ \ (p₁ ∪ …)| = |p₁ ∪ p₂ ∪ …|` — exactly the distinct count the combined `(p1) OR (p2)` staged delete removes. Works for nodes and edges alike with no edge identity needed; the node ID scan uses the same exclusion so a later statement also doesn't re-cascade already-deleted nodes. The ORIGINAL predicate is still what gets recorded (the staged delete removes the union); only the count uses the exclusion. The common single-delete path is unchanged (`prior` empty → filter is just the base predicate). New helper `dedup_delete_filter` + `MutationStaging::recorded_delete_predicates`. Turns the red regression test green (2 nodes / 3 edges); writes (33), end_to_end, validators, maintenance, recovery, composite_flow, merge_truth_table, consistency, changes, and failpoints (63) all stay green. * test(engine): delete dedup must not drop NULL-column rows (red) Follow-up to the overlapping-delete fix flagged in PR #308 review (Greptile P1): the `(base) AND NOT (prior)` exclusion breaks under SQL three-valued logic. If a prior delete predicate references a NULLable column, a later statement's matching row whose column is NULL makes `prior` evaluate to UNKNOWN, `NOT UNKNOWN` is UNKNOWN, and the row is filtered out of the scan — even though the prior delete never matched it. That drops it from `deleted_ids`, skipping its cascade (orphaned edges) or, if it is the only match, leaving the node undeleted. A data bug, not just a miscount. Data: Charlie(age 35), Zoe(age NULL); Knows Zoe→Charlie. `delete Person where age > 30` then `delete Person where name = "Zoe"`. Under the buggy `NOT`, Zoe's scan `(name='Zoe') AND NOT (age>30)` is UNKNOWN → Zoe survives. RED at this commit (Person count left=1, right=0). * fix(engine): NULL-safe delete dedup — exclude only definitely-matched prior rows Change `dedup_delete_filter` from `(base) AND NOT (prior)` to `(base) AND ((prior) IS NOT TRUE)`. `IS NOT TRUE` keeps both FALSE and UNKNOWN rows, so a prior predicate that evaluates to SQL UNKNOWN (a NULL in a referenced column) no longer drops a row this statement legitimately matches — only rows a prior predicate matched as definitely TRUE are excluded from the count/scan. The distinct-count semantics are unchanged for non-NULL data. Turns the red NULL-dedup test green (Zoe deleted, her edge cascaded), and the overlapping-dedup + writes/end_to_end/validators/maintenance/recovery/ composite_flow/consistency suites stay green. * docs(engine): note dedup_delete_filter's load-bearing dependency on D₂ Self-review follow-up: the overlapping-delete dedup assumes the committed snapshot is invariant across a query's statements, which holds only because D₂ forbids mixing writes with deletes (so a delete-touched table has no pending writes). Make that dependency explicit at the function so a future D₂ relaxation is forced to revisit the dedup. Comment-only. * Preserve staged write commit metadata
316 lines
14 KiB
Markdown
316 lines
14 KiB
Markdown
# Architecture
|
||
|
||
OmniGraph is a typed property-graph engine built as a coordination layer over many Lance datasets, with Git-style branches and commits across the whole graph, multi-modal querying (vector + FTS + BM25 + RRF + graph traversal) in one runtime, an HTTP server with Cedar policy, and a CLI driven by a per-operator `~/.omnigraph/config.yaml` plus team-owned cluster directories.
|
||
|
||
## Reading guide
|
||
|
||
Three views, increasing zoom:
|
||
|
||
1. **System context** — what OmniGraph is and what it touches.
|
||
2. **Layer view** — the eight-layer stack inside one OmniGraph process.
|
||
3. **Component zoom-ins** — what's inside each layer.
|
||
|
||
For runtime flows (read query, mutation), see [`docs/dev/execution.md`](execution.md). For the on-disk layout of a graph, see [`docs/user/storage.md`](../user/concepts/storage.md).
|
||
|
||
L1 (orange in the diagrams) is what we inherit from Lance; L2 (blue) is what OmniGraph adds. The L1/L2 framing is also called out in prose at the bottom of this doc.
|
||
|
||
## System context
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
classDef external fill:#fef3e8,stroke:#c46900,color:#000
|
||
classDef omnigraph fill:#e8f4fd,stroke:#1e6aa8,color:#000
|
||
classDef store fill:#f0f0f0,stroke:#555,color:#000
|
||
|
||
cli[CLI users]:::external
|
||
http[HTTP clients<br/>and SDKs]:::external
|
||
agents[Agents]:::external
|
||
embed[Embedding providers<br/>OpenAI / Gemini]:::external
|
||
|
||
og[OmniGraph<br/>kernel]:::omnigraph
|
||
|
||
cedar[Cedar policy<br/>engine]:::external
|
||
s3[Object store<br/>local FS / S3 / RustFS]:::store
|
||
|
||
cli --> og
|
||
http --> og
|
||
agents --> og
|
||
og --> embed
|
||
og --> cedar
|
||
og --> s3
|
||
```
|
||
|
||
OmniGraph runs as a single process (one binary, multiple crates). External dependencies are the embedding APIs (called during ingest and at query-time normalization), Cedar (called for every privileged action), and an object store (everything OmniGraph persists lands here).
|
||
|
||
## Layer view
|
||
|
||
Inside the OmniGraph process, work flows through these layers:
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
classDef l2 fill:#e8f4fd,stroke:#1e6aa8,color:#000
|
||
classDef l1 fill:#fef3e8,stroke:#c46900,color:#000
|
||
|
||
subgraph CLIs[CLI and HTTP server]
|
||
cli[omnigraph CLI]:::l2
|
||
srv[omnigraph-server<br/>Axum + Cedar]:::l2
|
||
end
|
||
|
||
subgraph compiler[omnigraph-compiler]
|
||
front[parse → AST → typecheck → catalog → IR]:::l2
|
||
end
|
||
|
||
subgraph engine[omnigraph engine]
|
||
plan[exec query and mutation]:::l2
|
||
gi[graph index CSR/CSC<br/>RuntimeCache LRU 8]:::l2
|
||
coord[coordinator<br/>ManifestCoordinator · CommitGraph]:::l2
|
||
end
|
||
|
||
subgraph storage[storage trait — wraps Lance]
|
||
ts[table_store · storage.rs<br/>direct lance::Dataset today]:::l2
|
||
end
|
||
|
||
subgraph lance_layer[Lance 4.x — substrate]
|
||
lance[per-dataset versions, fragments<br/>BTREE · Inverted FTS · IVF/HNSW vector<br/>merge_insert · compact_files · cleanup_old_versions]:::l1
|
||
end
|
||
|
||
subgraph object_store[Object store]
|
||
os[local FS · S3 · RustFS · MinIO]:::l1
|
||
end
|
||
|
||
CLIs -- "string + params" --> compiler
|
||
compiler -- IROp --> engine
|
||
engine -- "scan / write request" --> storage
|
||
storage -- "Stream of RecordBatch" --> engine
|
||
storage -- "Lance API calls" --> lance_layer
|
||
lance_layer -- bytes --> object_store
|
||
```
|
||
|
||
The storage seam is partly aspirational. `TableStorage` exists as the sealed staged-write trait, but capability/stat surfaces and full call-site migration are still roadmap. The diagram shows the intended boundary.
|
||
|
||
## Component zoom-ins
|
||
|
||
### Compiler — `omnigraph-compiler`
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
classDef l2 fill:#e8f4fd,stroke:#1e6aa8,color:#000
|
||
|
||
src[".gq source"]:::l2
|
||
p[parser Pest<br/>query.pest · schema.pest]:::l2
|
||
ast[AST<br/>QueryDecl · Mutation · Schema]:::l2
|
||
cat[catalog<br/>NodeType · EdgeType · Interface]:::l2
|
||
tc[typecheck<br/>typecheck_query]:::l2
|
||
low[lower<br/>lower_query]:::l2
|
||
ir[IROp pipeline<br/>NodeScan · Expand · Filter · AntiJoin]:::l2
|
||
|
||
src --> p --> ast --> tc
|
||
cat --> tc
|
||
tc --> low --> ir
|
||
```
|
||
|
||
The compiler crate has zero Lance dependency. It owns the schema language, the query language, and the AST → IR lowering.
|
||
|
||
Code paths:
|
||
|
||
- Parser: `crates/omnigraph-compiler/src/query/parser.rs`, `crates/omnigraph-compiler/src/query/query.pest`
|
||
- Typecheck: `crates/omnigraph-compiler/src/query/typecheck.rs:83` (`typecheck_query`)
|
||
- Lower: `crates/omnigraph-compiler/src/ir/lower.rs:11` (`lower_query`)
|
||
- Catalog: `crates/omnigraph-compiler/src/catalog/`
|
||
|
||
### Engine — `omnigraph` crate
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
classDef l2 fill:#e8f4fd,stroke:#1e6aa8,color:#000
|
||
|
||
subgraph exec[exec module]
|
||
eq[query · execute_query<br/>query.rs:347]:::l2
|
||
em[mutation · mutate<br/>mutation.rs:511]:::l2
|
||
ld[loader · ingest<br/>loader/mod.rs:74]:::l2
|
||
end
|
||
|
||
subgraph state[graph state]
|
||
coord[GraphCoordinator]:::l2
|
||
mr[ManifestCoordinator<br/>db/manifest.rs]:::l2
|
||
cg[CommitGraph<br/>projection of __manifest graph_commit/graph_head rows]:::l2
|
||
stg[MutationStaging<br/>per-query in-memory accumulator<br/>exec/staging.rs]:::l2
|
||
end
|
||
|
||
subgraph idx[graph index]
|
||
gi[GraphIndex<br/>CSR/CSC built per query]:::l2
|
||
rc[RuntimeCache LRU=8]:::l2
|
||
end
|
||
|
||
subgraph io[Lance I/O]
|
||
ts[table_store]:::l2
|
||
st[storage adapter<br/>storage.rs]:::l2
|
||
end
|
||
|
||
eq --> gi
|
||
eq --> ts
|
||
em --> stg
|
||
em --> ts
|
||
ld --> stg
|
||
ld --> ts
|
||
eq --> mr
|
||
em --> mr
|
||
coord --> mr
|
||
coord --> cg
|
||
ts --> st
|
||
```
|
||
|
||
The engine binds the compiler IR to Lance. It owns multi-dataset coordination, the graph topology index, the per-query staging accumulator, and the snapshot/manifest read path.
|
||
|
||
Code paths:
|
||
|
||
- Read entry: `Omnigraph::query` at `crates/omnigraph/src/exec/query.rs:7`
|
||
- Mutation entry: `Omnigraph::mutate` at `crates/omnigraph/src/exec/mutation.rs:511`
|
||
- Manifest commit: `ManifestCoordinator::commit` at `crates/omnigraph/src/db/manifest.rs:280`
|
||
- Graph index: `crates/omnigraph/src/graph_index/`
|
||
- Loader: `Omnigraph::ingest` at `crates/omnigraph/src/loader/mod.rs:74`
|
||
|
||
### Mutation atomicity — in-memory accumulator (MR-794)
|
||
|
||
Inserts and updates inside `mutate_as` and the bulk loader's
|
||
Append/Merge modes go through `MutationStaging`
|
||
([`crates/omnigraph/src/exec/staging.rs`](../../crates/omnigraph/src/exec/staging.rs)),
|
||
a per-query in-memory accumulator. No Lance HEAD advance happens during
|
||
op execution; one `stage_*` + `commit_staged` per touched table runs
|
||
at end-of-query, then the publisher commits the manifest atomically.
|
||
|
||
```
|
||
op-1 (insert/update) → push RecordBatch → MutationStaging.pending[table]
|
||
op-2 (insert/update) → read committed via Lance + pending via DataFusion
|
||
MemTable (read-your-writes) → push batch
|
||
op-N → push batch
|
||
─── end of query ───────────────────────────────────────
|
||
finalize: per pending table:
|
||
concat batches → stage_append OR stage_merge_insert OR stage_overwrite
|
||
→ commit_staged
|
||
publisher: ManifestBatchPublisher::publish (one cross-table CAS)
|
||
```
|
||
|
||
A failed op leaves Lance HEAD untouched on the staged tables: the next
|
||
mutation proceeds normally with no drift to reconcile. Concrete
|
||
contracts:
|
||
|
||
- `D₂` parse-time rule: a query is either insert/update-only or
|
||
delete-only. Mixed → reject. Deletes now stage like inserts/updates
|
||
(MR-A: `stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`),
|
||
so they no longer advance HEAD inline; D₂ is a deliberate boundary
|
||
(constructive XOR destructive per query) that keeps in-query read-your-writes
|
||
unambiguous — compose mixed operations via separate mutations or a branch.
|
||
- `LoadMode::Overwrite` uses Lance `Operation::Overwrite` through the
|
||
same staged path. Loader validation runs against the replacement
|
||
in-memory batches before any `commit_staged`, and the publish window is
|
||
covered by `SidecarKind::Load` recovery.
|
||
- Read sites consume `TableStore::scan_with_pending`, which Lance-scans
|
||
the committed snapshot at the captured `expected_version` and unions
|
||
with a DataFusion `MemTable` over the pending batches.
|
||
|
||
This pattern realizes read-your-writes within a multi-statement mutation
|
||
and keeps failure scope bounded for inserts/updates by construction at
|
||
the writer layer. See [docs/dev/invariants.md](invariants.md) and
|
||
[docs/dev/writes.md](writes.md) for the publisher CAS contract this builds on.
|
||
|
||
### Storage trait — today vs. roadmap
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
classDef now fill:#e8f4fd,stroke:#1e6aa8,color:#000
|
||
classDef future fill:#fff,stroke:#888,stroke-dasharray:5 5,color:#444
|
||
|
||
subgraph today[Today]
|
||
d1[table_store<br/>opens lance::Dataset directly]:::now
|
||
d2[storage.rs<br/>S3 / file URI plumbing]:::now
|
||
end
|
||
|
||
subgraph roadmap[Roadmap - storage capabilities]
|
||
t[trait Dataset<br/>schema · stats · placement<br/>capabilities · scan · write]:::future
|
||
impl1[LanceStorage]:::future
|
||
impl2[future test impl]:::future
|
||
end
|
||
|
||
today -.-> roadmap
|
||
t --> impl1
|
||
t --> impl2
|
||
```
|
||
|
||
The staged-write trait exists today as `TableStorage`, implemented by `TableStore`. Full engine migration plus capability and statistics surfaces remain roadmap, so the planner cannot yet reason about all pushdown opportunities through a documented trait surface.
|
||
|
||
### Index lifecycle — today vs. roadmap
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
classDef now fill:#e8f4fd,stroke:#1e6aa8,color:#000
|
||
classDef future fill:#fff,stroke:#888,stroke-dasharray:5 5,color:#444
|
||
|
||
subgraph today[Today]
|
||
ei[ensure_indices<br/>omnigraph.rs:445]:::now
|
||
manual[called manually<br/>or from optimize]:::now
|
||
end
|
||
|
||
subgraph roadmap[Roadmap - manifest reconciler]
|
||
rec[Reconciler<br/>observes manifest]:::future
|
||
diff[coverage diff<br/>fragments − fragment_bitmap]:::future
|
||
wp[worker pool<br/>builds index segments]:::future
|
||
end
|
||
|
||
manual --> ei
|
||
today -.-> roadmap
|
||
rec --> diff --> wp
|
||
```
|
||
|
||
Today, indexes are built explicitly via `ensure_indices`. Reads degrade gracefully when index coverage is partial — Lance's scanner unions indexed and scan paths automatically. The roadmap reconciler observes manifest state and converges coverage in the background.
|
||
|
||
### Server / CLI
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
classDef l2 fill:#e8f4fd,stroke:#1e6aa8,color:#000
|
||
|
||
cli[omnigraph CLI<br/>command families]:::l2
|
||
srv_in[Axum HTTP<br/>REST + OpenAPI]:::l2
|
||
auth[Bearer auth<br/>SHA-256 hashed tokens]:::l2
|
||
pol[Cedar policy gate<br/>per request]:::l2
|
||
wl[WorkloadController<br/>per-actor admission]:::l2
|
||
eng[engine API<br/>Arc<Omnigraph>]:::l2
|
||
wq[WriteQueueManager<br/>per-(table, branch)]:::l2
|
||
|
||
cli -.-> eng
|
||
srv_in --> auth --> pol --> wl --> eng
|
||
eng --> wq
|
||
```
|
||
|
||
The server applies Cedar policy at the HTTP boundary today. The roadmap, called out in [docs/dev/invariants.md](invariants.md) as a known gap, is to push policy into the planner as predicates. After Cedar, mutating handlers go through `WorkloadController` (per-actor admission cap + byte budget; PR 2 / MR-686) before reaching the engine. The engine itself holds an `Arc<WriteQueueManager>` so concurrent mutations on the same `(table, branch)` serialize at the queue, while disjoint keys run in parallel — see [docs/user/server.md](../user/operations/server.md) "Per-actor admission control" and [docs/dev/writes.md](writes.md). The CLI bypasses the HTTP layer (and admission) and calls the engine API directly.
|
||
|
||
Code paths:
|
||
|
||
- Server entry: `crates/omnigraph-server/src/lib.rs`
|
||
- Auth: `crates/omnigraph-server/src/auth.rs`
|
||
- Policy: `crates/omnigraph-server/src/policy.rs`
|
||
- CLI: `crates/omnigraph-cli/src/main.rs`
|
||
|
||
## L1 / L2 framing
|
||
|
||
Throughout the docs, capabilities are split into:
|
||
|
||
- **L1 — Inherited from Lance**: what OmniGraph gets "for free" by sitting on top of the Lance dataset format (columnar Arrow storage, per-dataset versions and branches, index types, `merge_insert`, `compact_files` / `cleanup_old_versions`).
|
||
- **L2 — Added by OmniGraph**: typing (schema language), graph semantics, multi-dataset coordination via `__manifest`, graph-level branches and commits, the `.gq` query language and IR, the topology index, the HTTP server, Cedar policy, the CLI.
|
||
|
||
## Concurrency model
|
||
|
||
- **MVCC**: every Lance write bumps a per-dataset version; the OmniGraph manifest version coordinates which sub-table versions are visible together.
|
||
- **Snapshot isolation**: a query holds one `Snapshot` for its lifetime; concurrent writes don't leak in.
|
||
- **Cross-branch isolation**: copy-on-write means readers and writers on different branches don't block each other.
|
||
- **Per-query staging**: `mutate_as` and `load` (Append/Merge) accumulate insert/update batches in an in-memory `MutationStaging`; one `stage_*` + `commit_staged` per touched table runs at end-of-query, then the publisher commits the manifest atomically. A mid-query failure leaves Lance HEAD untouched on staged tables. (MR-794; pre-v0.4.0 used a `__run__<id>` staging branch + Run state machine, removed in MR-771.)
|
||
- **Schema-apply lock**: `__schema_apply_lock__` system branch serializes schema migrations.
|
||
- **Fail-points** (`failpoints` cargo feature): `failpoints::maybe_fail("operation.step")?` in `branch_create`, publish, etc., for deterministic failure injection in tests.
|
||
|
||
## Workspace crates
|
||
|
||
- `omnigraph-compiler` — schema and query grammars, catalog, IR, lowering, type checker, lint, migration planner, OpenAI-style embedding client.
|
||
- `omnigraph` (engine, published as `omnigraph-engine` on crates.io since v0.2.2) — the Lance-backed runtime: manifest, commit graph, snapshot, exec (incl. per-query `MutationStaging` accumulator), merge, loader, Gemini embedding client.
|
||
- `omnigraph-cli` — the `omnigraph` binary.
|
||
- `omnigraph-server` — the `omnigraph-server` binary (Axum HTTP server).
|