mirror of
https://github.com/ModernRelay/omnigraph.git
synced 2026-06-09 01:35:18 +02:00
Lakehouse-native graph engine with git-style workflows
https://omnigraph.dev
ca21e73d43
Implement the remaining half of the open-time recovery sweep. Roll-forward execution (db/manifest/recovery.rs::roll_forward_all): constructs a GraphNamespacePublisher directly (recovery runs inside Omnigraph::open before the engine struct exists, so we can't go through Omnigraph::commit_updates_on_branch_with_expected). Builds a ManifestChange::Update per sidecar table reading row_count and TableVersionMetadata from the dataset at post_commit_pin (cheap; manifest-level reads, not a row scan), then calls publisher.publish with expected_table_versions = sidecar.expected_version per table. Single __manifest CAS extends every pin atomically — all-or-nothing at the substrate. Persistent CAS contention surfaces as the typed ExpectedVersionMismatch error and leaves the sidecar in place for the next open's retry. Audit model (new crates/omnigraph/src/db/recovery_audit.rs + record_audit() in recovery.rs): each successful recovery sweep records a graph-commit row tagged with actor_id="omnigraph:recovery" plus a row in a new sibling table _graph_commit_recoveries.lance carrying recovery_kind (RolledForward | RolledBack), recovery_for_actor (the sidecar's original actor_id), operation_id (sidecar ULID), sidecar_writer_kind, per_table_outcomes (JSON-serialized for schema flexibility), and created_at. Operators investigating "did my mutation land?" can find the answer via `omnigraph commit list --filter actor=omnigraph:recovery` joined to the recoveries table by graph_commit_id. The sibling-table choice avoids bumping INTERNAL_MANIFEST_SCHEMA_VERSION or migrating _graph_commits.lance. Same not-atomic-pair-write shape as the existing _graph_commits + _graph_commit_actors split — a crash between the two sequential writes leaves an orphan commit row with no recovery row. Recovery sweep tolerates this: re-entry classifies already-restored / already-published tables as NoMovement, the action is a no-op, and the audit append is retried. Note on classifier: process_sidecar's RollBack arm now restores RolledPastExpected, UnexpectedAtP1, AND UnexpectedMultistep (any drift class). Earlier Phase 3 logic restricted to RolledPastExpected only, which left UnexpectedAtP1/UnexpectedMultistep tables drifted; the all-or-nothing decision rule per docs/invariants.md §VI.23 demands all drifted tables be restored. 3 new integration tests in tests/recovery.rs (7 total now): - recovery_rolls_forward_after_phase_b_completes — happy-path roll-forward; audit row recorded; idempotent on second open. - recovery_rolls_back_records_audit_row_with_recovery_actor — roll-back path also records an audit row with the original actor. - recovery_rolls_forward_with_null_actor — sidecar without actor_id still records the audit row (recovery_for_actor = None). 3 new unit tests in db::recovery_audit pin the round-trip + persistence + recovery_kind string parsing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .cargo | ||
| .context | ||
| .github/workflows | ||
| crates | ||
| docker | ||
| docs | ||
| scripts | ||
| .dockerignore | ||
| .gitignore | ||
| AGENTS.md | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CLAUDE.md | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| Dockerfile | ||
| LICENSE | ||
| og-cheet-sheet.md | ||
| omnigraph.example.yaml | ||
| openapi.json | ||
| README.md | ||
| rust-toolchain.toml | ||
| SECURITY.md | ||
Omnigraph
Lakehouse-native graph engine with git-style workflows.
Branch, commit, and merge typed graph data like source code. Multi-modal, self-hosted, open source.
Use Cases
- On-prem & hybrid context graphs
- Backbone for multi-agent research
- Incident response graphs
- Compliance & audit graphs
- Enterprise knowledge systems
Capabilities
- Typed schema, typed queries, and typed mutations
- Schema-as-code, query validation and linting
- Git-style graph workflows: branches, commits, merges, and transactional runs
- Local, on-prem & cloud S3-native storage with snapshot-pinned reads
- Graph traversal + text, fuzzy, BM25, vector, and RRF search in one runtime
- Policy-as-code for server-side access control
- Single CLI for multiple deployments
Quick Install
curl -fsSL https://raw.githubusercontent.com/ModernRelay/omnigraph/main/scripts/install.sh | bash
This installs omnigraph and omnigraph-server into ~/.local/bin from
published release binaries.
Or install with Homebrew:
brew tap ModernRelay/tap
brew install ModernRelay/tap/omnigraph
For starter graphs and agent skills to bootstrap and operate Omnigraph, see ModernRelay/omnigraph-cookbooks.
One-Command Local RustFS Bootstrap
curl -fsSL https://raw.githubusercontent.com/ModernRelay/omnigraph/main/scripts/local-rustfs-bootstrap.sh | bash
That bootstrap:
- starts RustFS on
127.0.0.1:9000 - creates a bucket and S3-backed repo
- loads the checked-in context fixture
- launches
omnigraph-serveron127.0.0.1:8080
Docker must be installed and running first.
The RustFS bootstrap prefers the rolling edge binaries and only falls back to
source builds when release assets are unavailable.
If a previous run left objects under the same repo prefix but did not finish
initializing the repo, rerun with RESET_REPO=1 or set PREFIX to a new
value.
Common Commands
The same URI works for local paths, s3://…, or http://host:port.
omnigraph init --schema ./schema.pg ./repo.omni
omnigraph load --data ./data.jsonl ./repo.omni
omnigraph read --query ./queries.gq --name get_person --params '{"name":"Alice"}' ./repo.omni
omnigraph change --query ./queries.gq --name insert_person --params '{"name":"Mina"}' ./repo.omni
omnigraph branch create --from main feature-x ./repo.omni
omnigraph branch merge feature-x --into main ./repo.omni
See docs/cli.md for schema apply, snapshots, ingest, runs, and policy commands.
Docs
Build And Test
cargo build --workspace
cargo check --workspace
cargo test --workspace
Notes:
- Rust stable toolchain, edition 2024
- CI runs
cargo test --workspace --locked - Full CI and some local test flows require
protobuf-compiler - S3 integration tests expect an S3-compatible endpoint such as RustFS
Workspace Crates
crates/omnigraph-compiler: shared schema/query parser, typechecker, catalog, and IR loweringcrates/omnigraph: storage/runtime, branching, merge, change detection, and query executioncrates/omnigraph-cli: CLI for init/load/ingest/read/change/branch/snapshot/export/policy operationscrates/omnigraph-server: Axum HTTP server for remote reads, changes, ingest, export, branches, commits, and runs
Contributing
Please open an issue, spec, or design discussion before sending large code changes. Design feedback and concrete problem statements are the fastest way to collaborate on the roadmap.
Community
Join the Omnigraph Slack community to ask questions, share feedback, and follow development.