omnigraph/docs/dev/architecture.md

Architecture

OmniGraph is a typed property-graph engine built as a coordination layer over many Lance datasets, with Git-style branches and commits across the whole graph, multi-modal querying (vector + FTS + BM25 + RRF + graph traversal) in one runtime, an HTTP server with Cedar policy, and a CLI driven by a per-operator ~/.omnigraph/config.yaml plus team-owned cluster directories.

Reading guide

Three views, increasing zoom:

1. System context — what OmniGraph is and what it touches.
2. Layer view — the eight-layer stack inside one OmniGraph process.
3. Component zoom-ins — what's inside each layer.

For runtime flows (read query, mutation), see docs/dev/execution.md. For the on-disk layout of a graph, see docs/user/storage.md.

L1 (orange in the diagrams) is what we inherit from Lance; L2 (blue) is what OmniGraph adds. The L1/L2 framing is also called out in prose at the bottom of this doc.

System context

flowchart LR
    classDef external fill:#fef3e8,stroke:#c46900,color:#000
    classDef omnigraph fill:#e8f4fd,stroke:#1e6aa8,color:#000
    classDef store fill:#f0f0f0,stroke:#555,color:#000

    cli[CLI users]:::external
    http[HTTP clients<br/>and SDKs]:::external
    agents[Agents]:::external
    embed[Embedding providers<br/>OpenAI / Gemini]:::external

    og[OmniGraph<br/>kernel]:::omnigraph

    cedar[Cedar policy<br/>engine]:::external
    s3[Object store<br/>local FS / S3 / RustFS]:::store

    cli --> og
    http --> og
    agents --> og
    og --> embed
    og --> cedar
    og --> s3

OmniGraph runs as a single process (one binary, multiple crates). External dependencies are the embedding APIs (called during ingest and at query-time normalization), Cedar (called for every privileged action), and an object store (everything OmniGraph persists lands here).

Layer view

Inside the OmniGraph process, work flows through these layers:

flowchart TB
    classDef l2 fill:#e8f4fd,stroke:#1e6aa8,color:#000
    classDef l1 fill:#fef3e8,stroke:#c46900,color:#000

    subgraph CLIs[CLI and HTTP server]
        cli[omnigraph CLI]:::l2
        srv[omnigraph-server<br/>Axum + Cedar]:::l2
    end

    subgraph compiler[omnigraph-compiler]
        front[parse → AST → typecheck → catalog → IR]:::l2
    end

    subgraph engine[omnigraph engine]
        plan[exec query and mutation]:::l2
        gi[graph index CSR/CSC<br/>RuntimeCache LRU 8]:::l2
        coord[coordinator<br/>ManifestCoordinator · CommitGraph]:::l2
    end

    subgraph storage[storage trait — wraps Lance]
        ts[table_store · storage.rs<br/>direct lance::Dataset today]:::l2
    end

    subgraph lance_layer[Lance 4.x — substrate]
        lance[per-dataset versions, fragments<br/>BTREE · Inverted FTS · IVF/HNSW vector<br/>merge_insert · compact_files · cleanup_old_versions]:::l1
    end

    subgraph object_store[Object store]
        os[local FS · S3 · RustFS · MinIO]:::l1
    end

    CLIs -- "string + params" --> compiler
    compiler -- IROp --> engine
    engine -- "scan / write request" --> storage
    storage -- "Stream of RecordBatch" --> engine
    storage -- "Lance API calls" --> lance_layer
    lance_layer -- bytes --> object_store

The storage seam is partly aspirational. TableStorage exists as the sealed staged-write trait, but capability/stat surfaces and full call-site migration are still roadmap. The diagram shows the intended boundary.

Component zoom-ins

Compiler — omnigraph-compiler

flowchart LR
    classDef l2 fill:#e8f4fd,stroke:#1e6aa8,color:#000

    src[".gq source"]:::l2
    p[parser Pest<br/>query.pest · schema.pest]:::l2
    ast[AST<br/>QueryDecl · Mutation · Schema]:::l2
    cat[catalog<br/>NodeType · EdgeType · Interface]:::l2
    tc[typecheck<br/>typecheck_query]:::l2
    low[lower<br/>lower_query]:::l2
    ir[IROp pipeline<br/>NodeScan · Expand · Filter · AntiJoin]:::l2

    src --> p --> ast --> tc
    cat --> tc
    tc --> low --> ir

The compiler crate has zero Lance dependency. It owns the schema language, the query language, and the AST → IR lowering.

Code paths:

• Parser: crates/omnigraph-compiler/src/query/parser.rs, crates/omnigraph-compiler/src/query/query.pest
• Typecheck: crates/omnigraph-compiler/src/query/typecheck.rs:83 (typecheck_query)
• Lower: crates/omnigraph-compiler/src/ir/lower.rs:11 (lower_query)
• Catalog: crates/omnigraph-compiler/src/catalog/

Engine — omnigraph crate

flowchart TB
    classDef l2 fill:#e8f4fd,stroke:#1e6aa8,color:#000

    subgraph exec[exec module]
        eq[query · execute_query<br/>query.rs:347]:::l2
        em[mutation · mutate<br/>mutation.rs:511]:::l2
        ld[loader · ingest<br/>loader/mod.rs:74]:::l2
    end

    subgraph state[graph state]
        coord[GraphCoordinator]:::l2
        mr[ManifestCoordinator<br/>db/manifest.rs]:::l2
        cg[CommitGraph<br/>projection of __manifest graph_commit/graph_head rows]:::l2
        stg[MutationStaging<br/>per-query in-memory accumulator<br/>exec/staging.rs]:::l2
    end

    subgraph idx[graph index]
        gi[GraphIndex<br/>CSR/CSC built per query]:::l2
        rc[RuntimeCache LRU=8]:::l2
    end

    subgraph io[Lance I/O]
        ts[table_store]:::l2
        st[storage adapter<br/>storage.rs]:::l2
    end

    eq --> gi
    eq --> ts
    em --> stg
    em --> ts
    ld --> stg
    ld --> ts
    eq --> mr
    em --> mr
    coord --> mr
    coord --> cg
    ts --> st

The engine binds the compiler IR to Lance. It owns multi-dataset coordination, the graph topology index, the per-query staging accumulator, and the snapshot/manifest read path.

Code paths:

• Read entry: Omnigraph::query at crates/omnigraph/src/exec/query.rs:7
• Mutation entry: Omnigraph::mutate at crates/omnigraph/src/exec/mutation.rs:511
• Manifest commit: ManifestCoordinator::commit at crates/omnigraph/src/db/manifest.rs:280
• Graph index: crates/omnigraph/src/graph_index/
• Loader: Omnigraph::ingest at crates/omnigraph/src/loader/mod.rs:74

Mutation atomicity — in-memory accumulator (MR-794)

Inserts and updates inside mutate_as and the bulk loader's Append/Merge modes go through MutationStaging (crates/omnigraph/src/exec/staging.rs), a per-query in-memory accumulator. No Lance HEAD advance happens during op execution; one stage_* + commit_staged per touched table runs at end-of-query, then the publisher commits the manifest atomically.

op-1 (insert/update) → push RecordBatch → MutationStaging.pending[table]
op-2 (insert/update) → read committed via Lance + pending via DataFusion
                       MemTable (read-your-writes) → push batch
op-N → push batch
─── end of query ───────────────────────────────────────
finalize: per pending table:
   concat batches → stage_append OR stage_merge_insert OR stage_overwrite
                  → commit_staged
publisher: ManifestBatchPublisher::publish (one cross-table CAS)

A failed op leaves Lance HEAD untouched on the staged tables: the next mutation proceeds normally with no drift to reconcile. Concrete contracts:

• D₂ parse-time rule: a query is either insert/update-only or delete-only. Mixed → reject. Deletes now stage like inserts/updates (MR-A: stage_delete via Lance 7.0 DeleteBuilder::execute_uncommitted), so they no longer advance HEAD inline; D₂ is a deliberate boundary (constructive XOR destructive per query) that keeps in-query read-your-writes unambiguous — compose mixed operations via separate mutations or a branch.
• LoadMode::Overwrite uses Lance Operation::Overwrite through the same staged path. Loader validation runs against the replacement in-memory batches before any commit_staged, and the publish window is covered by SidecarKind::Load recovery.
• Read sites consume TableStore::scan_with_pending, which Lance-scans the committed snapshot at the captured expected_version and unions with a DataFusion MemTable over the pending batches.

This pattern realizes read-your-writes within a multi-statement mutation and keeps failure scope bounded for inserts/updates by construction at the writer layer. See docs/dev/invariants.md and docs/dev/writes.md for the publisher CAS contract this builds on.

Storage trait — today vs. roadmap

flowchart LR
    classDef now fill:#e8f4fd,stroke:#1e6aa8,color:#000
    classDef future fill:#fff,stroke:#888,stroke-dasharray:5 5,color:#444

    subgraph today[Today]
        d1[table_store<br/>opens lance::Dataset directly]:::now
        d2[storage.rs<br/>S3 / file URI plumbing]:::now
    end

    subgraph roadmap[Roadmap - storage capabilities]
        t[trait Dataset<br/>schema · stats · placement<br/>capabilities · scan · write]:::future
        impl1[LanceStorage]:::future
        impl2[future test impl]:::future
    end

    today -.-> roadmap
    t --> impl1
    t --> impl2

The staged-write trait exists today as TableStorage, implemented by TableStore. Full engine migration plus capability and statistics surfaces remain roadmap, so the planner cannot yet reason about all pushdown opportunities through a documented trait surface.

Index lifecycle — today vs. roadmap

flowchart LR
    classDef now fill:#e8f4fd,stroke:#1e6aa8,color:#000
    classDef future fill:#fff,stroke:#888,stroke-dasharray:5 5,color:#444

    subgraph today[Today]
        ei[ensure_indices<br/>omnigraph.rs:445]:::now
        manual[called manually<br/>or from optimize]:::now
    end

    subgraph roadmap[Roadmap - manifest reconciler]
        rec[Reconciler<br/>observes manifest]:::future
        diff[coverage diff<br/>fragments − fragment_bitmap]:::future
        wp[worker pool<br/>builds index segments]:::future
    end

    manual --> ei
    today -.-> roadmap
    rec --> diff --> wp

Today, indexes are built explicitly via ensure_indices. Reads degrade gracefully when index coverage is partial — Lance's scanner unions indexed and scan paths automatically. The roadmap reconciler observes manifest state and converges coverage in the background.

Server / CLI

flowchart LR
    classDef l2 fill:#e8f4fd,stroke:#1e6aa8,color:#000

    cli[omnigraph CLI<br/>command families]:::l2
    srv_in[Axum HTTP<br/>REST + OpenAPI]:::l2
    auth[Bearer auth<br/>SHA-256 hashed tokens]:::l2
    pol[Cedar policy gate<br/>per request]:::l2
    wl[WorkloadController<br/>per-actor admission]:::l2
    eng[engine API<br/>Arc&lt;Omnigraph&gt;]:::l2
    wq[WriteQueueManager<br/>per-(table, branch)]:::l2

    cli -.-> eng
    srv_in --> auth --> pol --> wl --> eng
    eng --> wq

The server applies Cedar policy at the HTTP boundary today. The roadmap, called out in docs/dev/invariants.md as a known gap, is to push policy into the planner as predicates. After Cedar, mutating handlers go through WorkloadController (per-actor admission cap + byte budget; PR 2 / MR-686) before reaching the engine. The engine itself holds an Arc<WriteQueueManager> so concurrent mutations on the same (table, branch) serialize at the queue, while disjoint keys run in parallel — see docs/user/server.md "Per-actor admission control" and docs/dev/writes.md. The CLI bypasses the HTTP layer (and admission) and calls the engine API directly.

Code paths:

• Server entry: crates/omnigraph-server/src/lib.rs
• Auth: crates/omnigraph-server/src/auth.rs
• Policy: crates/omnigraph-server/src/policy.rs
• CLI: crates/omnigraph-cli/src/main.rs

L1 / L2 framing

Throughout the docs, capabilities are split into:

• L1 — Inherited from Lance: what OmniGraph gets "for free" by sitting on top of the Lance dataset format (columnar Arrow storage, per-dataset versions and branches, index types, merge_insert, compact_files / cleanup_old_versions).
• L2 — Added by OmniGraph: typing (schema language), graph semantics, multi-dataset coordination via __manifest, graph-level branches and commits, the .gq query language and IR, the topology index, the HTTP server, Cedar policy, the CLI.

Concurrency model

• MVCC: every Lance write bumps a per-dataset version; the OmniGraph manifest version coordinates which sub-table versions are visible together.
• Snapshot isolation: a query holds one Snapshot for its lifetime; concurrent writes don't leak in.
• Cross-branch isolation: copy-on-write means readers and writers on different branches don't block each other.
• Per-query staging: mutate_as and load (Append/Merge) accumulate insert/update batches in an in-memory MutationStaging; one stage_* + commit_staged per touched table runs at end-of-query, then the publisher commits the manifest atomically. A mid-query failure leaves Lance HEAD untouched on staged tables. (MR-794; pre-v0.4.0 used a __run__<id> staging branch + Run state machine, removed in MR-771.)
• Schema-apply lock: __schema_apply_lock__ system branch serializes schema migrations.
• Fail-points (failpoints cargo feature): failpoints::maybe_fail("operation.step")? in branch_create, publish, etc., for deterministic failure injection in tests.

Workspace crates

• omnigraph-compiler — schema and query grammars, catalog, IR, lowering, type checker, lint, migration planner, OpenAI-style embedding client.
• omnigraph (engine, published as omnigraph-engine on crates.io since v0.2.2) — the Lance-backed runtime: manifest, commit graph, snapshot, exec (incl. per-query MutationStaging accumulator), merge, loader, Gemini embedding client.
• omnigraph-cli — the omnigraph binary.
• omnigraph-server — the omnigraph-server binary (Axum HTTP server).