mirror of
https://github.com/ModernRelay/omnigraph.git
synced 2026-06-09 01:35:18 +02:00
Lakehouse-native graph engine with git-style workflows
https://omnigraph.dev
c6827919ca
Three writers each follow the same shape established in Phase 5: build SidecarTablePin list before the per-table commit_staged loop, write the sidecar via recovery::write_sidecar, do the existing work, delete the sidecar after the manifest publish succeeds. Loose-match classifier (recovery.rs): The classifier now distinguishes strict vs. loose match per SidecarKind. Strict (Mutation, Load, BranchMerge): exactly one commit_staged per table; lance_head == manifest_pinned + 1 AND post_commit_pin == lance_head required. Loose (SchemaApply, EnsureIndices): the writer may run N >= 1 commit_staged calls per table — index builds + rewrites compound, and the exact N is hard to compute at sidecar-write time. Loose accepts any lance_head > manifest_pinned (with expected_version still matching the manifest pin) as RolledPastExpected. The risk it admits — an external agent advancing HEAD between sidecar write and recovery — is out of scope for the single-coordinator model (MR-668 territory). roll_forward_all now reads the CURRENT Lance HEAD per table (not the sidecar's post_commit_pin) so the manifest publish reflects whatever HEAD landed, even if the loose-match writer committed multiple times per table. Per-writer wiring: - schema_apply::apply_schema_with_lock: sidecar covers rewritten_tables ∪ indexed_tables (the tables that go through stage_overwrite/stage_create_index commit_staged). Skips added_tables (fresh datasets, no Phase B residual class) and renamed_tables (handled by the existing schema-state staging recovery in recover_schema_state_files). - branch_merge::branch_merge_on_current_target: sidecar covers every table in candidates (publish_adopted_source_state + publish_rewritten_merge_table do the per-table commit_staged work). Sidecar writes after validate_merge_candidates and deletes after commit_manifest_updates. - ensure_indices_for_branch: sidecar covers every node + edge type in the catalog with a manifest entry (build_indices_on_dataset is per-table-per-index commit_staged). Skips when the catalog has nothing — steady-state calls incur no sidecar I/O when the manifest already pins all expected types. Allow recovery_audit.rs in forbidden_apis.rs: The new db/recovery_audit.rs uses Dataset::write to bootstrap the _graph_commit_recoveries.lance dataset (same pattern as commit_graph.rs which is already allow-listed). Add it to the ALLOW_LIST_FILES list in tests/forbidden_apis.rs. 8 new unit tests in db::manifest::recovery cover the loose-match classifier branches (SchemaApply + EnsureIndices accept multi-commit drift, NoMovement and InvariantViolation behave the same as strict). All 20 test binaries pass (350+ tests across the workspace). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .cargo | ||
| .context | ||
| .github/workflows | ||
| crates | ||
| docker | ||
| docs | ||
| scripts | ||
| .dockerignore | ||
| .gitignore | ||
| AGENTS.md | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CLAUDE.md | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| Dockerfile | ||
| LICENSE | ||
| og-cheet-sheet.md | ||
| omnigraph.example.yaml | ||
| openapi.json | ||
| README.md | ||
| rust-toolchain.toml | ||
| SECURITY.md | ||
Omnigraph
Lakehouse-native graph engine with git-style workflows.
Branch, commit, and merge typed graph data like source code. Multi-modal, self-hosted, open source.
Use Cases
- On-prem & hybrid context graphs
- Backbone for multi-agent research
- Incident response graphs
- Compliance & audit graphs
- Enterprise knowledge systems
Capabilities
- Typed schema, typed queries, and typed mutations
- Schema-as-code, query validation and linting
- Git-style graph workflows: branches, commits, merges, and transactional runs
- Local, on-prem & cloud S3-native storage with snapshot-pinned reads
- Graph traversal + text, fuzzy, BM25, vector, and RRF search in one runtime
- Policy-as-code for server-side access control
- Single CLI for multiple deployments
Quick Install
curl -fsSL https://raw.githubusercontent.com/ModernRelay/omnigraph/main/scripts/install.sh | bash
This installs omnigraph and omnigraph-server into ~/.local/bin from
published release binaries.
Or install with Homebrew:
brew tap ModernRelay/tap
brew install ModernRelay/tap/omnigraph
For starter graphs and agent skills to bootstrap and operate Omnigraph, see ModernRelay/omnigraph-cookbooks.
One-Command Local RustFS Bootstrap
curl -fsSL https://raw.githubusercontent.com/ModernRelay/omnigraph/main/scripts/local-rustfs-bootstrap.sh | bash
That bootstrap:
- starts RustFS on
127.0.0.1:9000 - creates a bucket and S3-backed repo
- loads the checked-in context fixture
- launches
omnigraph-serveron127.0.0.1:8080
Docker must be installed and running first.
The RustFS bootstrap prefers the rolling edge binaries and only falls back to
source builds when release assets are unavailable.
If a previous run left objects under the same repo prefix but did not finish
initializing the repo, rerun with RESET_REPO=1 or set PREFIX to a new
value.
Common Commands
The same URI works for local paths, s3://…, or http://host:port.
omnigraph init --schema ./schema.pg ./repo.omni
omnigraph load --data ./data.jsonl ./repo.omni
omnigraph read --query ./queries.gq --name get_person --params '{"name":"Alice"}' ./repo.omni
omnigraph change --query ./queries.gq --name insert_person --params '{"name":"Mina"}' ./repo.omni
omnigraph branch create --from main feature-x ./repo.omni
omnigraph branch merge feature-x --into main ./repo.omni
See docs/cli.md for schema apply, snapshots, ingest, runs, and policy commands.
Docs
Build And Test
cargo build --workspace
cargo check --workspace
cargo test --workspace
Notes:
- Rust stable toolchain, edition 2024
- CI runs
cargo test --workspace --locked - Full CI and some local test flows require
protobuf-compiler - S3 integration tests expect an S3-compatible endpoint such as RustFS
Workspace Crates
crates/omnigraph-compiler: shared schema/query parser, typechecker, catalog, and IR loweringcrates/omnigraph: storage/runtime, branching, merge, change detection, and query executioncrates/omnigraph-cli: CLI for init/load/ingest/read/change/branch/snapshot/export/policy operationscrates/omnigraph-server: Axum HTTP server for remote reads, changes, ingest, export, branches, commits, and runs
Contributing
Please open an issue, spec, or design discussion before sending large code changes. Design feedback and concrete problem statements are the fastest way to collaborate on the roadmap.
Community
Join the Omnigraph Slack community to ask questions, share feedback, and follow development.