mirror of
https://github.com/ModernRelay/omnigraph.git
synced 2026-06-09 01:35:18 +02:00
Lakehouse-native graph engine with git-style workflows
https://omnigraph.dev
272b70bfb4
Original harness used recall@K vs. SIFT1M as the correctness oracle, which gives
the agent incentive to overfit to one data distribution: a kernel that hits
recall@10 on SIFT-shaped clusters could regress on other distributions and
still pass the gate. This commit replaces both halves of the oracle.
Correctness phase (was: recall@K floor):
- Bit-equivalent (max_abs_err <= 1e-4) match against an immutable scalar
reference kernel, on a 5-distribution input battery (Gaussian, uniform,
sparse, large-dynamic-range, mostly-zero) crossed with all evaluated PQ
shapes. Top-K compared with tie-tolerant equivalence (TOPK_DIST_TOL=1e-4).
Lossy techniques (LUT u8/u16 quantization, etc.) fail this gate by
construction.
Speed phase (was: geomean ns over one synthetic dataset):
- Geomean ns/query measured across 3 PQ shapes x 3 data distributions:
(128, 16, 256) - SIFT-like
(256, 16, 256) - sub_vector_dim=16
(768, 96, 256) - BERT-like
crossed with clustered / uniform / sparse data. Fixed seed across trials
for reproducibility; per-combo timings reported alongside the global
geomean / worst / best so a kernel that wins on one combo and regresses
on another fails the worst-case guard.
Kernel API (was: const-DIM scalar functions):
- Generic over (dim, num_sub_vectors, num_centroids) via PqShape.
- PqKernel::new(shape, codebook) lets the agent pre-process the codebook
once (transpose, cache c.c, pack LUT, etc.) and amortize across queries.
Build cost is excluded from per-query timing - the bench measures
distance_table + probe_top_k only.
Other consequences:
- SIFT1M loader (src/fixture.rs), prepare_fixtures.sh, and the
cache-directory plumbing all delete - the harness is now fully
self-contained, no external download.
- src/inputs.rs replaces src/fixture.rs; deterministic per-trial
test-data + workload generation, no frozen artifacts.
- Cargo.toml gains an empty [workspace] block so cargo doesn't walk up to
the omnigraph parent workspace from inside research/.
Verified end-to-end:
- cargo build --release: clean
- cargo clippy --release --all-targets -- -D warnings: clean
- cargo run --release --bin run_experiment: correctness pass, geomean
1.22M ns, worst 4.82M ns ((768,96,256), sparse), best 596k ns, exit 0,
total wall-clock ~39s
- smoke test: kernel returning 0 distance -> correctness fail with
diagnostic, exit 2
- cargo test --release --lib: 2/2 unit tests pass
(correctness_battery_is_deterministic, speed_workloads_match_shapes)
https://claude.ai/code/session_01Aq8kBUcjmEPobcEufnWbW5
|
||
|---|---|---|
| .cargo | ||
| .context | ||
| .github | ||
| crates | ||
| docker | ||
| docs | ||
| research/lance-autoresearch | ||
| scripts | ||
| .dockerignore | ||
| .gitignore | ||
| AGENTS.md | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CLAUDE.md | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| Dockerfile | ||
| LICENSE | ||
| og-cheet-sheet.md | ||
| omnigraph.example.yaml | ||
| openapi.json | ||
| README.md | ||
| rust-toolchain.toml | ||
| SECURITY.md | ||
Omnigraph
Lakehouse-native graph engine with git-style workflows.
Branch, commit, and merge typed graph data like source code. Multi-modal, self-hosted, open source.
Built on Rust, Arrow, DataFusion and Lance.
Join the Omnigraph Slack community
Use Cases
- Company brains
- Context graphs
- Backbone for multi-agent research
- Incident response graphs
- Compliance & audit graphs
- Enterprise knowledge systems
Capabilities
- Typed schema, typed queries, and typed mutations
- Schema-as-code, query validation and linting
- Git-style graph workflows: branches, commits, merges, and transactional runs
- Local, on-prem & cloud S3-native storage with snapshot-pinned reads
- Graph traversal + text, fuzzy, BM25, vector, and RRF search in one runtime
- Policy-as-code for server-side access control
- Single CLI for multiple deployments
Quick Install
curl -fsSL https://raw.githubusercontent.com/ModernRelay/omnigraph/main/scripts/install.sh | bash
This installs omnigraph and omnigraph-server into ~/.local/bin from
published release binaries.
Or install with Homebrew:
brew tap ModernRelay/tap
brew install ModernRelay/tap/omnigraph
For starter graphs and agent skills to bootstrap and operate Omnigraph, see ModernRelay/omnigraph-cookbooks.
One-Command Local RustFS Bootstrap
curl -fsSL https://raw.githubusercontent.com/ModernRelay/omnigraph/main/scripts/local-rustfs-bootstrap.sh | bash
That bootstrap:
- starts RustFS on
127.0.0.1:9000 - creates a bucket and S3-backed repo
- loads the checked-in context fixture
- launches
omnigraph-serveron127.0.0.1:8080
Docker must be installed and running first.
The RustFS bootstrap prefers the rolling edge binaries and only falls back to
source builds when release assets are unavailable.
If a previous run left objects under the same repo prefix but did not finish
initializing the repo, rerun with RESET_REPO=1 or set PREFIX to a new
value.
Common Commands
The same URI works for local paths, s3://…, or http://host:port.
omnigraph init --schema ./schema.pg ./repo.omni
omnigraph load --data ./data.jsonl ./repo.omni
omnigraph read --query ./queries.gq --name get_person --params '{"name":"Alice"}' ./repo.omni
omnigraph change --query ./queries.gq --name insert_person --params '{"name":"Mina"}' ./repo.omni
omnigraph branch create --from main feature-x ./repo.omni
omnigraph branch merge feature-x --into main ./repo.omni
See docs/cli.md for schema apply, snapshots, ingest, runs, and policy commands.
Docs
Build And Test
cargo build --workspace
cargo check --workspace
cargo test --workspace
Notes:
- Rust stable toolchain, edition 2024
- CI runs
cargo test --workspace --locked - Full CI and some local test flows require
protobuf-compiler - S3 integration tests expect an S3-compatible endpoint such as RustFS
Workspace Crates
crates/omnigraph-compiler: shared schema/query parser, typechecker, catalog, and IR loweringcrates/omnigraph: storage/runtime, branching, merge, change detection, and query executioncrates/omnigraph-cli: CLI for init/load/ingest/read/change/branch/snapshot/export/policy operationscrates/omnigraph-server: Axum HTTP server for remote reads, changes, ingest, export, branches, commits, and runs
Contributing
Please open an issue, spec, or design discussion before sending large code changes. Design feedback and concrete problem statements are the fastest way to collaborate on the roadmap.
Community
Join the Omnigraph Slack community to ask questions, share feedback, and follow development.