apunkt/omnigraph
1
0
Fork 0
mirror of https://github.com/ModernRelay/omnigraph.git synced 2026-06-09 01:35:18 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
Lakehouse-native graph engine with git-style workflows https://omnigraph.dev
166 commits 23 branches 12 tags 9.3 MiB
Find a file
Ragnor Comerford 052b6e680f
MR-794 step 2: address PR #68 follow-up review (Cubic) — pending dedupe + projection guard + CI 
Three new findings from Cubic on commit 3223b51:

* **Pending edge cardinality counted within-input duplicates** (P2):
  count_src_per_edge's pending walk added every row to the count,
  including duplicate rows that finalize will collapse via
  dedupe_merge_batches_by_id. A LoadMode::Merge with the same edge id
  twice would over-count → spurious @card violation. Fix: when
  dedupe_key_column is Some, walk pending in reverse, track seen keys
  via HashSet, count only the kept (last-occurrence) rows. Mirrors
  finalize-time dedupe so cardinality counts what stage_merge_insert
  actually publishes.

* **scan_with_pending silently disabled merge-shadow when projection
  omitted key_column** (P2): if a caller passed Some("id") as
  key_column but their projection didn't include "id", the
  filter_out_rows_where_string_in helper passed batches through
  unchanged — silently degrading to union semantics. Fix: validate
  up front that projection contains key_column when both are Some;
  return a typed Lance error otherwise. Tightened the helper too:
  missing column is now an internal error (was a silent passthrough).

* **Cascade-vs-explicit delete test was too weak** (P2): asserted
  only that edge count decreased after delete. The cascade alone
  could satisfy that even if the explicit second-delete silently
  no-op'd. Strengthened: assert post_knows == 0, which only holds
  when both ops landed (Bob→Diana would survive if op-2 no-op'd).

CI gap: also added test_failpoints_feature job to .github/workflows/ci.yml.
The workspace test runs without --features failpoints (the feature is
behind a Cargo flag), so the failpoints test suite was never exercised
by CI before now. The new job builds + runs
`cargo test -p omnigraph-engine --features failpoints --test failpoints`
on every full CI run, mirroring the test_aws_feature pattern.

New tests on tests/runs.rs:

* load_merge_mode_dedupes_within_pending_for_cardinality_count
  (Cubic P2 #2 — pending-vs-pending dedup, distinct from the
  load_merge_mode_dedupes_edge_for_cardinality_count test which
  covers committed-vs-pending dedup).
* scan_with_pending_rejects_key_column_missing_from_projection
  (Cubic P2 #3 — verifies the up-front validation rejects bad
  callers and that the happy path still works correctly).

Local test results:

* tests/runs.rs: 23/23 passed
* tests/failpoints.rs --features failpoints: 7/7 passed (includes the
  two new finalize→publisher residual tests landed in 3223b51).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 20:47:45 +02:00
.cargo Raise LANCE_MEM_POOL_SIZE to 1 GB in .cargo/config.toml 2026-04-19 22:27:49 +03:00
.context Investigate Lance MergeInsertBuilder CAS granularity (MR-766 prereq) 2026-04-28 23:30:17 +00:00
.github/workflows MR-794 step 2: address PR #68 follow-up review (Cubic) — pending dedupe + projection guard + CI 2026-05-01 20:47:45 +02:00
crates MR-794 step 2: address PR #68 follow-up review (Cubic) — pending dedupe + projection guard + CI 2026-05-01 20:47:45 +02:00
docker Initial public Omnigraph repository 2026-04-10 20:49:41 +03:00
docs MR-794 step 2: address PR #68 review — merge semantics, cardinality, residual 2026-05-01 13:47:55 +02:00
scripts Merge pull request #60 from ModernRelay/ragnorc/omnigraph-spec 2026-04-29 00:15:19 +02:00
.dockerignore Initial public Omnigraph repository 2026-04-10 20:49:41 +03:00
.gitignore Ignore local-only working files 2026-04-26 16:41:15 +03:00
AGENTS.md MR-794 step 2: docs — runs/invariants/architecture/execution + cleanup 2026-05-01 10:43:19 +02:00
Cargo.lock MR-794 step 2: scaffold MutationStaging accumulator + scan_with_pending 2026-05-01 10:42:21 +02:00
Cargo.toml MR-794 step 2: scaffold MutationStaging accumulator + scan_with_pending 2026-05-01 10:42:21 +02:00
CLAUDE.md Add AGENTS.md as canonical agent guide; symlink CLAUDE.md to it 2026-04-28 23:10:09 +02:00
CODE_OF_CONDUCT.md Initial public Omnigraph repository 2026-04-10 20:49:41 +03:00
CONTRIBUTING.md Merge remote-tracking branch 'origin/main' into ragnorc/explore-api 2026-04-18 20:24:39 +02:00
Dockerfile Dockerfile: switch base from Docker Hub to ECR Public 2026-04-20 13:46:23 +03:00
LICENSE Initial public Omnigraph repository 2026-04-10 20:49:41 +03:00
og-cheet-sheet.md Add query lint and check commands 2026-04-13 00:37:44 +03:00
omnigraph.example.yaml example config: use graphs / cli.graph, matching the MR-603 rename 2026-04-18 23:40:35 +03:00
openapi.json MR-771: demote Run to direct-publish via expected_table_versions CAS 2026-04-30 08:52:50 +02:00
README.md Sharpen README tagline; add incident-response and compliance use cases (#58) 2026-04-28 03:46:21 +03:00
rust-toolchain.toml Initial public Omnigraph repository 2026-04-10 20:49:41 +03:00
SECURITY.md Initial public Omnigraph repository 2026-04-10 20:49:41 +03:00

README.md

Omnigraph

License: MIT Rust Crates.io CI

Lakehouse-native graph engine with git-style workflows.

Branch, commit, and merge typed graph data like source code. Multi-modal, self-hosted, open source.

Use Cases

  • On-prem & hybrid context graphs
  • Backbone for multi-agent research
  • Incident response graphs
  • Compliance & audit graphs
  • Enterprise knowledge systems

Capabilities

  • Typed schema, typed queries, and typed mutations
  • Schema-as-code, query validation and linting
  • Git-style graph workflows: branches, commits, merges, and transactional runs
  • Local, on-prem & cloud S3-native storage with snapshot-pinned reads
  • Graph traversal + text, fuzzy, BM25, vector, and RRF search in one runtime
  • Policy-as-code for server-side access control
  • Single CLI for multiple deployments

Quick Install

curl -fsSL https://raw.githubusercontent.com/ModernRelay/omnigraph/main/scripts/install.sh | bash

This installs omnigraph and omnigraph-server into ~/.local/bin from published release binaries.

Or install with Homebrew:

brew tap ModernRelay/tap
brew install ModernRelay/tap/omnigraph

For starter graphs and agent skills to bootstrap and operate Omnigraph, see ModernRelay/omnigraph-starters.

One-Command Local RustFS Bootstrap

curl -fsSL https://raw.githubusercontent.com/ModernRelay/omnigraph/main/scripts/local-rustfs-bootstrap.sh | bash

That bootstrap:

  • starts RustFS on 127.0.0.1:9000
  • creates a bucket and S3-backed repo
  • loads the checked-in context fixture
  • launches omnigraph-server on 127.0.0.1:8080

Docker must be installed and running first.

The RustFS bootstrap prefers the rolling edge binaries and only falls back to source builds when release assets are unavailable.

If a previous run left objects under the same repo prefix but did not finish initializing the repo, rerun with RESET_REPO=1 or set PREFIX to a new value.

Common Commands

The same URI works for local paths, s3://…, or http://host:port.

omnigraph init   --schema ./schema.pg ./repo.omni
omnigraph load   --data   ./data.jsonl ./repo.omni
omnigraph read   --query  ./queries.gq --name get_person --params '{"name":"Alice"}' ./repo.omni
omnigraph change --query  ./queries.gq --name insert_person --params '{"name":"Mina"}' ./repo.omni
omnigraph branch create --from main feature-x ./repo.omni
omnigraph branch merge  feature-x --into main ./repo.omni

See docs/cli.md for schema apply, snapshots, ingest, runs, and policy commands.

Docs

Build And Test

cargo build --workspace
cargo check --workspace
cargo test --workspace

Notes:

  • Rust stable toolchain, edition 2024
  • CI runs cargo test --workspace --locked
  • Full CI and some local test flows require protobuf-compiler
  • S3 integration tests expect an S3-compatible endpoint such as RustFS

Workspace Crates

  • crates/omnigraph-compiler: shared schema/query parser, typechecker, catalog, and IR lowering
  • crates/omnigraph: storage/runtime, branching, merge, change detection, and query execution
  • crates/omnigraph-cli: CLI for init/load/ingest/read/change/branch/snapshot/export/policy operations
  • crates/omnigraph-server: Axum HTTP server for remote reads, changes, ingest, export, branches, commits, and runs

Contributing

Please open an issue, spec, or design discussion before sending large code changes. Design feedback and concrete problem statements are the fastest way to collaborate on the roadmap.