mirror of
https://github.com/ModernRelay/omnigraph.git
synced 2026-06-09 01:35:18 +02:00
Merge pull request #64 from ModernRelay/ragnorc/architecture-diagrams
docs: add Mermaid architecture diagrams (system / storage / read + write flows)
This commit is contained in:
Ragnor Comerford
GitHub
commit
034c49f82b
3 changed files with 388 additions and 39 deletions
|
|
@ -2,49 +2,248 @@
|
|||
|
||||
OmniGraph is a typed property-graph engine built as a coordination layer over many Lance datasets, with Git-style branches and commits across the whole graph, multi-modal querying (vector + FTS + BM25 + RRF + graph traversal) in one runtime, an HTTP server with Cedar policy, and a CLI driven by a single `omnigraph.yaml`.
|
||||
|
||||
## Stack
|
||||
## Reading guide
|
||||
|
||||
Three views, increasing zoom:
|
||||
|
||||
1. **System context** — what OmniGraph is and what it touches.
|
||||
2. **Layer view** — the eight-layer stack inside one OmniGraph process.
|
||||
3. **Component zoom-ins** — what's inside each layer.
|
||||
|
||||
For runtime flows (read query, mutation), see [`docs/execution.md`](execution.md). For the on-disk layout of a repo, see [`docs/storage.md`](storage.md).
|
||||
|
||||
L1 (orange in the diagrams) is what we inherit from Lance; L2 (blue) is what OmniGraph adds. The L1/L2 framing is also called out in prose at the bottom of this doc.
|
||||
|
||||
## System context
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
classDef external fill:#fef3e8,stroke:#c46900,color:#000
|
||||
classDef omnigraph fill:#e8f4fd,stroke:#1e6aa8,color:#000
|
||||
classDef store fill:#f0f0f0,stroke:#555,color:#000
|
||||
|
||||
cli[CLI users]:::external
|
||||
http[HTTP clients<br/>and SDKs]:::external
|
||||
agents[Agents]:::external
|
||||
embed[Embedding providers<br/>OpenAI / Gemini]:::external
|
||||
|
||||
og[OmniGraph<br/>kernel]:::omnigraph
|
||||
|
||||
cedar[Cedar policy<br/>engine]:::external
|
||||
s3[Object store<br/>local FS / S3 / RustFS]:::store
|
||||
|
||||
cli --> og
|
||||
http --> og
|
||||
agents --> og
|
||||
og --> embed
|
||||
og --> cedar
|
||||
og --> s3
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ CLI (omnigraph) HTTP Server (omnigraph-server, Axum) │
|
||||
│ - 13 cmd families - REST + OpenAPI │
|
||||
│ - Aliases, configs - Bearer auth + Cedar policy │
|
||||
└──────────────────────────────┬───────────────────────────────────┘
|
||||
│
|
||||
┌──────────────────────────────▼───────────────────────────────────┐
|
||||
│ omnigraph-compiler │
|
||||
│ - Pest grammars: schema.pest, query.pest │
|
||||
│ - Catalog (Node/Edge/Interface types) │
|
||||
│ - IR + lowering (NodeScan / Expand / Filter / AntiJoin) │
|
||||
│ - Schema migration planner │
|
||||
│ - Embedding client (OpenAI-style for query-time normalization) │
|
||||
└──────────────────────────────┬───────────────────────────────────┘
|
||||
│
|
||||
┌──────────────────────────────▼───────────────────────────────────┐
|
||||
│ omnigraph (engine) │
|
||||
│ - GraphCoordinator + ManifestRepo (__manifest) │
|
||||
│ - CommitGraph (_graph_commits.lance) │
|
||||
│ - RunRegistry (_graph_runs.lance, __run__ branches) │
|
||||
│ - GraphIndex (CSR/CSC) + RuntimeCache (LRU 8) │
|
||||
│ - exec::query / mutation / merge │
|
||||
│ - Embedding client (Gemini for runtime ingest) │
|
||||
└──────────────────────────────┬───────────────────────────────────┘
|
||||
│
|
||||
┌──────────────────────────────▼───────────────────────────────────┐
|
||||
│ Lance 4.x (per-table dataset) │
|
||||
│ - Columnar (Arrow) storage, fragments │
|
||||
│ - Manifest versions per dataset │
|
||||
│ - Per-dataset branches (copy-on-write) │
|
||||
│ - Indexes: BTREE, Inverted (FTS/BM25), IVF/HNSW vector │
|
||||
│ - merge_insert (upsert), append, delete │
|
||||
│ - compact_files, cleanup_old_versions │
|
||||
└──────────────────────────────┬───────────────────────────────────┘
|
||||
│
|
||||
┌──────────────────────────────▼───────────────────────────────────┐
|
||||
│ Object store: local FS, S3, RustFS, MinIO, S3-compatible │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
|
||||
OmniGraph runs as a single process (one binary, multiple crates). External dependencies are the embedding APIs (called during ingest and at query-time normalization), Cedar (called for every privileged action), and an object store (everything OmniGraph persists lands here).
|
||||
|
||||
## Layer view
|
||||
|
||||
Inside the OmniGraph process, work flows through these layers:
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
classDef l2 fill:#e8f4fd,stroke:#1e6aa8,color:#000
|
||||
classDef l1 fill:#fef3e8,stroke:#c46900,color:#000
|
||||
|
||||
subgraph CLIs[CLI and HTTP server]
|
||||
cli[omnigraph CLI]:::l2
|
||||
srv[omnigraph-server<br/>Axum + Cedar]:::l2
|
||||
end
|
||||
|
||||
subgraph compiler[omnigraph-compiler]
|
||||
front[parse → AST → typecheck → catalog → IR]:::l2
|
||||
end
|
||||
|
||||
subgraph engine[omnigraph engine]
|
||||
plan[exec query and mutation]:::l2
|
||||
gi[graph index CSR/CSC<br/>RuntimeCache LRU 8]:::l2
|
||||
coord[coordinator<br/>ManifestRepo · CommitGraph · RunRegistry]:::l2
|
||||
end
|
||||
|
||||
subgraph storage[storage trait — wraps Lance]
|
||||
ts[table_store · storage.rs<br/>direct lance::Dataset today]:::l2
|
||||
end
|
||||
|
||||
subgraph lance_layer[Lance 4.x — substrate]
|
||||
lance[per-dataset versions, fragments<br/>BTREE · Inverted FTS · IVF/HNSW vector<br/>merge_insert · compact_files · cleanup_old_versions]:::l1
|
||||
end
|
||||
|
||||
subgraph object_store[Object store]
|
||||
os[local FS · S3 · RustFS · MinIO]:::l1
|
||||
end
|
||||
|
||||
CLIs -- "string + params" --> compiler
|
||||
compiler -- IROp --> engine
|
||||
engine -- "scan / write request" --> storage
|
||||
storage -- "Stream of RecordBatch" --> engine
|
||||
storage -- "Lance API calls" --> lance_layer
|
||||
lance_layer -- bytes --> object_store
|
||||
```
|
||||
|
||||
The `storage trait` row is partly aspirational. Today the engine calls `lance::Dataset` methods through `table_store`; a capability-bearing `Dataset` trait per [`docs/invariants.md`](invariants.md) §I.4 is on the roadmap (MR-737). The diagram shows the intended seam.
|
||||
|
||||
## Component zoom-ins
|
||||
|
||||
### Compiler — `omnigraph-compiler`
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
classDef l2 fill:#e8f4fd,stroke:#1e6aa8,color:#000
|
||||
|
||||
src[".gq source"]:::l2
|
||||
p[parser Pest<br/>query.pest · schema.pest]:::l2
|
||||
ast[AST<br/>QueryDecl · Mutation · Schema]:::l2
|
||||
cat[catalog<br/>NodeType · EdgeType · Interface]:::l2
|
||||
tc[typecheck<br/>typecheck_query]:::l2
|
||||
low[lower<br/>lower_query]:::l2
|
||||
ir[IROp pipeline<br/>NodeScan · Expand · Filter · AntiJoin]:::l2
|
||||
|
||||
src --> p --> ast --> tc
|
||||
cat --> tc
|
||||
tc --> low --> ir
|
||||
```
|
||||
|
||||
The compiler crate has zero Lance dependency. It owns the schema language, the query language, and the AST → IR lowering.
|
||||
|
||||
Code paths:
|
||||
|
||||
- Parser: `crates/omnigraph-compiler/src/query/parser.rs`, `crates/omnigraph-compiler/src/query/query.pest`
|
||||
- Typecheck: `crates/omnigraph-compiler/src/query/typecheck.rs:83` (`typecheck_query`)
|
||||
- Lower: `crates/omnigraph-compiler/src/ir/lower.rs:11` (`lower_query`)
|
||||
- Catalog: `crates/omnigraph-compiler/src/catalog/`
|
||||
|
||||
### Engine — `omnigraph` crate
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
classDef l2 fill:#e8f4fd,stroke:#1e6aa8,color:#000
|
||||
|
||||
subgraph exec[exec module]
|
||||
eq[query · execute_query<br/>query.rs:347]:::l2
|
||||
em[mutation · mutate<br/>mutation.rs:511]:::l2
|
||||
ld[loader · ingest<br/>loader/mod.rs:74]:::l2
|
||||
end
|
||||
|
||||
subgraph state[graph state]
|
||||
coord[GraphCoordinator]:::l2
|
||||
mr[ManifestRepo<br/>db/manifest.rs]:::l2
|
||||
cg[CommitGraph<br/>_graph_commits.lance]:::l2
|
||||
rr[RunRegistry<br/>_graph_runs.lance]:::l2
|
||||
end
|
||||
|
||||
subgraph idx[graph index]
|
||||
gi[GraphIndex<br/>CSR/CSC built per query]:::l2
|
||||
rc[RuntimeCache LRU=8]:::l2
|
||||
end
|
||||
|
||||
subgraph io[Lance I/O]
|
||||
ts[table_store]:::l2
|
||||
st[storage adapter<br/>storage.rs]:::l2
|
||||
end
|
||||
|
||||
eq --> gi
|
||||
eq --> ts
|
||||
em --> ts
|
||||
ld --> ts
|
||||
eq --> mr
|
||||
em --> mr
|
||||
coord --> mr
|
||||
coord --> cg
|
||||
coord --> rr
|
||||
ts --> st
|
||||
```
|
||||
|
||||
The engine binds the compiler IR to Lance. It owns multi-dataset coordination, the graph topology index, the run registry, and the snapshot/manifest read path.
|
||||
|
||||
Code paths:
|
||||
|
||||
- Read entry: `Omnigraph::query` at `crates/omnigraph/src/exec/query.rs:7`
|
||||
- Mutation entry: `Omnigraph::mutate` at `crates/omnigraph/src/exec/mutation.rs:511`
|
||||
- Manifest commit: `ManifestRepo::commit` at `crates/omnigraph/src/db/manifest.rs:280`
|
||||
- Graph index: `crates/omnigraph/src/graph_index/`
|
||||
- Loader: `Omnigraph::ingest` at `crates/omnigraph/src/loader/mod.rs:74`
|
||||
|
||||
### Storage trait — today vs. roadmap
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
classDef now fill:#e8f4fd,stroke:#1e6aa8,color:#000
|
||||
classDef future fill:#fff,stroke:#888,stroke-dasharray:5 5,color:#444
|
||||
|
||||
subgraph today[Today]
|
||||
d1[table_store<br/>opens lance::Dataset directly]:::now
|
||||
d2[storage.rs<br/>S3 / file URI plumbing]:::now
|
||||
end
|
||||
|
||||
subgraph roadmap[Roadmap — invariants §I.4]
|
||||
t[trait Dataset<br/>schema · stats · placement<br/>capabilities · scan · write]:::future
|
||||
impl1[LanceStorage]:::future
|
||||
impl2[MemStorage for tests]:::future
|
||||
end
|
||||
|
||||
today -.-> roadmap
|
||||
t --> impl1
|
||||
t --> impl2
|
||||
```
|
||||
|
||||
The storage layer's trait surface is aspirational. Today the engine calls `lance::Dataset` methods directly. The roadmap (per [`docs/invariants.md`](invariants.md) §I.4 and MR-737) is a `Dataset` trait that surfaces capabilities and statistics so the planner can reason about pushdown opportunities.
|
||||
|
||||
### Index lifecycle — today vs. roadmap
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
classDef now fill:#e8f4fd,stroke:#1e6aa8,color:#000
|
||||
classDef future fill:#fff,stroke:#888,stroke-dasharray:5 5,color:#444
|
||||
|
||||
subgraph today[Today]
|
||||
ei[ensure_indices<br/>omnigraph.rs:445]:::now
|
||||
manual[called manually<br/>or from optimize]:::now
|
||||
end
|
||||
|
||||
subgraph roadmap[Roadmap — invariants §VII.35]
|
||||
rec[Reconciler<br/>observes manifest]:::future
|
||||
diff[coverage diff<br/>fragments − fragment_bitmap]:::future
|
||||
wp[worker pool<br/>builds index segments]:::future
|
||||
end
|
||||
|
||||
manual --> ei
|
||||
today -.-> roadmap
|
||||
rec --> diff --> wp
|
||||
```
|
||||
|
||||
Today, indexes are built explicitly via `ensure_indices`. Reads degrade gracefully when index coverage is partial — Lance's scanner unions indexed and scan paths automatically. The roadmap reconciler (per [`docs/invariants.md`](invariants.md) §VII.35) observes manifest state and converges coverage in the background.
|
||||
|
||||
### Server / CLI
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
classDef l2 fill:#e8f4fd,stroke:#1e6aa8,color:#000
|
||||
|
||||
cli[omnigraph CLI<br/>command families]:::l2
|
||||
srv_in[Axum HTTP<br/>REST + OpenAPI]:::l2
|
||||
auth[Bearer auth<br/>SHA-256 hashed tokens]:::l2
|
||||
pol[Cedar policy gate<br/>per request]:::l2
|
||||
eng[engine API]:::l2
|
||||
|
||||
cli -.-> eng
|
||||
srv_in --> auth --> pol --> eng
|
||||
```
|
||||
|
||||
The server applies Cedar policy at the HTTP boundary today (per [`docs/invariants.md`](invariants.md) §VII.45, the roadmap is to push policy into the planner as predicates). The CLI bypasses the HTTP layer and calls the engine API directly.
|
||||
|
||||
Code paths:
|
||||
|
||||
- Server entry: `crates/omnigraph-server/src/lib.rs`
|
||||
- Auth: `crates/omnigraph-server/src/auth.rs`
|
||||
- Policy: `crates/omnigraph-server/src/policy.rs`
|
||||
- CLI: `crates/omnigraph-cli/src/main.rs`
|
||||
|
||||
## L1 / L2 framing
|
||||
|
||||
Throughout the docs, capabilities are split into:
|
||||
|
|
|
|||
|
|
@ -9,6 +9,49 @@ Pipeline:
|
|||
3. If `Expand` or `AntiJoin` is present, build (or fetch from `RuntimeCache`) a `GraphIndex`.
|
||||
4. Run `execute_query` against the snapshot.
|
||||
|
||||
### Read flow — sequence
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant client as Client
|
||||
participant og as Omnigraph::query<br/>(query.rs:7)
|
||||
participant cmp as omnigraph-compiler
|
||||
participant exec as execute_query<br/>(query.rs:347)
|
||||
participant gi as GraphIndex<br/>(RuntimeCache)
|
||||
participant ts as table_store
|
||||
participant lance as Lance scanner
|
||||
|
||||
client->>og: query(target, source, name, params)
|
||||
og->>og: ensure_schema_state_valid()<br/>resolve target → snapshot
|
||||
og->>cmp: parse + typecheck_query (typecheck.rs:83)
|
||||
cmp-->>og: CheckedQuery
|
||||
og->>cmp: lower_query (lower.rs:11)
|
||||
cmp-->>og: QueryIR (pipeline of IROp)
|
||||
og->>exec: extract_search_mode + dispatch (query.rs:110)
|
||||
exec->>gi: build / fetch GraphIndex<br/>(if Expand or AntiJoin)
|
||||
gi-->>exec: CSR / CSC topology
|
||||
loop for each IROp in pipeline
|
||||
exec->>ts: scan with predicate / SIP
|
||||
ts->>lance: filter · nearest · full_text_search
|
||||
lance-->>ts: Stream of RecordBatch
|
||||
ts-->>exec: RecordBatch stream
|
||||
exec->>exec: factorize · expand · fuse · project
|
||||
end
|
||||
exec-->>og: QueryResult (RecordBatches)
|
||||
og-->>client: serialized result
|
||||
```
|
||||
|
||||
**Code paths:**
|
||||
|
||||
- Entry: `Omnigraph::query` at `crates/omnigraph/src/exec/query.rs:7`
|
||||
- Search-mode extraction: `extract_search_mode` at `crates/omnigraph/src/exec/query.rs:110`
|
||||
- Pipeline runner: `execute_query` at `crates/omnigraph/src/exec/query.rs:347`
|
||||
- RRF fan-out: `execute_rrf_query` at `crates/omnigraph/src/exec/query.rs:393`
|
||||
- Per-source-row BFS: `execute_expand` at `crates/omnigraph/src/exec/query.rs:675`
|
||||
- Lance scan + pushdown: `execute_node_scan` at `crates/omnigraph/src/exec/query.rs:1027`
|
||||
- Filter → SQL pushdown: `build_lance_filter` at `crates/omnigraph/src/exec/query.rs:1158`
|
||||
|
||||
### Multi-modal search modes (`SearchMode`)
|
||||
|
||||
The executor recognizes three modes that may be combined in a single query:
|
||||
|
|
@ -44,6 +87,64 @@ Resolves expression values to literals, converts to typed Arrow arrays (`literal
|
|||
|
||||
Multi-statement mutations are atomic at the manifest commit boundary.
|
||||
|
||||
### Mutation flow — sequence
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant client as Client
|
||||
participant og as Omnigraph::mutate<br/>(mutation.rs:511)
|
||||
participant cmp as omnigraph-compiler
|
||||
participant runs as RunRegistry
|
||||
participant ts as table_store
|
||||
participant lance as Lance dataset
|
||||
participant mr as ManifestRepo<br/>(manifest.rs:280)
|
||||
|
||||
client->>og: mutate(target, source, name, params)
|
||||
og->>cmp: parse + typecheck_query
|
||||
cmp-->>og: CheckedQuery (Mutation IR)
|
||||
og->>runs: begin_run(target, op_hash)<br/>fork __run__<id> from target head
|
||||
runs-->>og: RunRecord
|
||||
loop for each mutation statement (on __run__<id>)
|
||||
og->>og: resolve expression literals<br/>literal_to_typed_array(lit, type, n)
|
||||
alt insert
|
||||
og->>ts: append RecordBatches
|
||||
ts->>lance: WriteMode::Append → new fragment(s)
|
||||
else update
|
||||
og->>ts: merge_insert keyed by id
|
||||
ts->>lance: merge_insert(WhenMatched::Update)
|
||||
else delete
|
||||
og->>ts: merge_insert with delete predicate
|
||||
ts->>lance: merge_insert(WhenMatched::Delete)
|
||||
end
|
||||
lance-->>ts: new dataset version
|
||||
og->>mr: commit_updates(SubTableUpdate)<br/>per-statement commit on __run__<id>
|
||||
mr-->>og: ack
|
||||
end
|
||||
og->>og: OCC: target head unchanged since begin_run?
|
||||
og->>og: publish_run(run_id)
|
||||
alt fast path (target hasn't moved)
|
||||
og->>mr: commit_updates_on_branch(target, updates)<br/>promote run snapshot
|
||||
else merge path (target advanced)
|
||||
og->>og: branch_merge_internal(__run__<id>, target)<br/>three-way merge
|
||||
end
|
||||
mr-->>og: new target snapshot
|
||||
og->>runs: terminate_run(Published)
|
||||
og-->>client: MutationResult
|
||||
```
|
||||
|
||||
**Code paths:**
|
||||
|
||||
- Entry: `Omnigraph::mutate` at `crates/omnigraph/src/exec/mutation.rs:511`
|
||||
- Per-mutation orchestration: `mutate_with_current_actor` at `crates/omnigraph/src/exec/mutation.rs:539`
|
||||
- Per-statement commit on the run-branch: `commit_updates` (called from `execute_insert` / `execute_update` / `execute_delete` in `crates/omnigraph/src/exec/mutation.rs`)
|
||||
- Run publish: `Omnigraph::publish_run` at `crates/omnigraph/src/db/omnigraph.rs:858`
|
||||
- Manifest commit primitive: `ManifestRepo::commit` at `crates/omnigraph/src/db/manifest.rs:280` (called from both per-statement `commit_updates` and the publish path)
|
||||
|
||||
Multi-statement mutations don't get atomicity from a single final `commit` — they get it from the **run-branch + publish_run** pattern. By default a mutation forks a fresh `__run__<id>` branch (`begin_run`); each statement individually commits its sub-table updates to that run-branch. After all statements complete, an OCC pre-check verifies the target hasn't moved since the run started, then `publish_run` atomically promotes the run-branch into the target — either via the fast path (direct promotion if the target hasn't moved) or a three-way merge. That final publish is what gives multi-statement mutations their atomicity guarantee (per [`docs/invariants.md`](invariants.md) §VI.26). If anything fails mid-run, the run is failed and the run-branch is dropped without affecting the target.
|
||||
|
||||
One exception: if the caller already targets a `__run__<id>` branch (mutation.rs:555), the mutation runs directly on that branch with no nested run wrapping — the assumption is the caller is managing the surrounding run lifecycle themselves. See [runs.md](runs.md) for the full run lifecycle.
|
||||
|
||||
## Bulk loader (`loader/mod.rs`)
|
||||
|
||||
- **JSONL only** in v1, with two record shapes:
|
||||
|
|
|
|||
|
|
@ -48,6 +48,55 @@ Adding a new on-disk shape change is one constant bump (`INTERNAL_MANIFEST_SCHEM
|
|||
| v1 (implicit, pre-stamp) | `__manifest.object_id` had no PK annotation; publisher had no row-level CAS protection. |
|
||||
| v2 | `__manifest.object_id` carries `lance-schema:unenforced-primary-key=true`; row-level CAS engaged. Stamped as `omnigraph:internal_schema_version=2`. |
|
||||
|
||||
## On-disk layout
|
||||
|
||||
A repo on disk is a directory tree of Lance datasets. Each dataset follows the standard Lance layout (`_versions/`, `data/`, `_indices/`, `_refs/`); OmniGraph adds the multi-dataset coordination by keeping `__manifest/` alongside the per-type datasets.
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
classDef l1 fill:#fef3e8,stroke:#c46900,color:#000
|
||||
classDef l2 fill:#e8f4fd,stroke:#1e6aa8,color:#000
|
||||
|
||||
repo["repo URI<br/>file:// or s3://bucket/prefix"]:::l2
|
||||
|
||||
manifest["__manifest/<br/>L2 catalog of sub-tables"]:::l2
|
||||
nodes["nodes/{fnv1a64-hex}/<br/>one dataset per node type"]:::l2
|
||||
edges["edges/{fnv1a64-hex}/<br/>one dataset per edge type"]:::l2
|
||||
cgraph["_graph_commits.lance/<br/>_graph_commit_actors.lance/"]:::l2
|
||||
runs["_graph_runs.lance/<br/>_graph_run_actors.lance/"]:::l2
|
||||
refs["_refs/branches/{name}.json<br/>graph-level branches"]:::l2
|
||||
|
||||
repo --> manifest
|
||||
repo --> nodes
|
||||
repo --> edges
|
||||
repo --> cgraph
|
||||
repo --> runs
|
||||
repo --> refs
|
||||
|
||||
subgraph dataset[Inside each Lance dataset — L1]
|
||||
ds_v["_versions/{n}.manifest<br/>per-dataset versions"]:::l1
|
||||
ds_data["data/<br/>fragment files (Arrow IPC)"]:::l1
|
||||
ds_idx["_indices/{uuid}/<br/>BTREE · Inverted FTS · IVF/HNSW"]:::l1
|
||||
ds_refs["_refs/<br/>per-dataset Lance branches/tags"]:::l1
|
||||
ds_tx["_transactions/<br/>commit transaction logs"]:::l1
|
||||
end
|
||||
|
||||
nodes -.-> dataset
|
||||
edges -.-> dataset
|
||||
manifest -.-> dataset
|
||||
```
|
||||
|
||||
**What's where:**
|
||||
|
||||
- **Repo root** is one directory (or S3 prefix). Everything below is part of one OmniGraph repo.
|
||||
- **`__manifest/`** is a Lance dataset whose rows describe which sub-table version is published at which graph-branch. Reading a snapshot starts here.
|
||||
- **`nodes/`** and **`edges/`** are sibling directories holding one Lance dataset per declared type. Names are `fnv1a64-hex` of the type name to keep paths fixed-length and case-safe.
|
||||
- **`_graph_commits.lance` / `_graph_runs.lance`** are L2 datasets that record the graph-level commit DAG and run registry respectively (each has a paired `*_actors.lance` for the actor map).
|
||||
- **`_refs/branches/{name}.json`** is graph-level branch metadata — pointers from a branch name to the manifest version it heads.
|
||||
- **Inside each Lance dataset** (orange): the standard Lance directory layout. `_versions/{n}.manifest` records every commit; `data/` holds the actual Arrow fragments; `_indices/{uuid}/` holds index segments with their own `fragment_bitmap` for partial coverage; `_refs/` holds Lance-native per-dataset branches and tags.
|
||||
|
||||
The split — L2 owns the cross-dataset catalog; L1 owns the per-dataset internals — means that schema work (which adds or removes datasets) updates `__manifest`, while data work (which adds fragments) updates `_versions/` inside the affected dataset and then bumps `__manifest`.
|
||||
|
||||
## URI scheme support (`storage.rs`)
|
||||
|
||||
| Scheme | Backend | Notes |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue