2026-04-10 20:49:41 +03:00
# Omnigraph
2026-04-14 13:52:14 +03:00
[](LICENSE)
[](rust-toolchain.toml)
2026-04-14 22:48:19 +03:00
[](https://crates.io/crates/omnigraph-cli)
2026-04-14 19:37:05 +03:00
[](https://github.com/ModernRelay/omnigraph/actions/workflows/ci.yml)
2026-04-14 13:52:14 +03:00
docs: rewrite README opening + add AGENTS.md dev commands (#122)
* docs(agents): add build/test/lint dev-command section
AGENTS.md (CLAUDE.md) covered architecture and invariants but had no
developer command surface — only runtime `omnigraph` CLI usage. Add a
concise "Build, test, lint" section with the non-obvious gotchas:
- crate dir `crates/omnigraph` is package `omnigraph-engine` (the `-p` name)
- canonical CI gate is `cargo test --workspace --locked`
- how to run one file / one fn
- feature-gated suites (`failpoints`, server `aws`)
- S3 tests skip without `OMNIGRAPH_S3_TEST_BUCKET`
- the two non-test CI checks (check-agents-md, OpenAPI drift)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(readme): rewrite opening, dedupe, fix stale references
- New manifesto-style opening (tagline, X-as-code, features, core
use cases, coordination-layer line); drop the old prose intro,
Use Cases, and Capabilities sections.
- Remove Capabilities, which restated the new opening line-for-line.
- Harmonize heading case: "## Core Use Cases".
- Dedupe the verbatim Slack invite (kept the Community section) and
the double-linked cli.md (kept the contextual pointer).
- Fix stale references that no longer match the code:
- drop "transactional runs" / "and runs" — no run concept remains;
writes are atomic per-query, multi-query workflows use branches.
- update the CLI crate command list to canonical query/mutate plus
commit/lint/optimize/cleanup.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-30 00:53:42 +01:00
**Lakehouse native graph engine built for context assembly**
2026-04-25 16:16:24 +02:00
fix: self-heal manifest-unreferenced branch forks (stop wedged branches) (#231)
* chore: correct stale global-lock comments
The global Arc<RwLock<Omnigraph>> that once serialized every server write was
removed — the server holds the engine as a lockless Arc<Omnigraph> and write
methods are &self, so the per-(table_key, branch) write queues are now the
actual write-serialization mechanism (in-process only).
Correct comments that still claimed the global lock is 'still in place' /
'today', or framed the queues as MR-686 scaffolding: write_queue.rs module doc,
exec/merge.rs, db/omnigraph/schema_apply.rs, db/manifest/recovery.rs, and the
bench_concurrent_http.rs example (which also wrongly stated mutate_as is
&mut self). workload.rs is left as-is — its 'previous global RwLock' wording is
accurate history.
* test: regression for self-healing a manifest-unreferenced fork
An interrupted first-write fork (create_branch succeeded, the manifest publish
did not) leaves a fully-formed Lance branch ref the manifest never references.
The branch stays a valid manifest branch, so cleanup's reconciler never
reclaims it, and today the next write to that table wedges with 'incomplete
prior delete; run cleanup'.
Forge that exact residue (a live 'feature' branch + a directly-created
'feature' ref on the Person table the manifest doesn't reference) and assert
the next load AND mutate self-heal. Deterministic and local — no S3 or timing,
since the forge IS the post-crash state. Adds a shared node_table_uri helper.
This commit is RED: it reproduces the bug and fails against the unfixed engine
with the predicted symptom. The fix follows in the next commit.
* fix: self-heal manifest-unreferenced branch forks
The first write to a table on a branch lazily forks it via Lance create_branch,
a durable two-phase op that advances Lance state BEFORE the atomic manifest
publish. If the writer dies or its request future is cancelled between the fork
and the publish, the branch ref is fully formed but the manifest never
references it. The next write re-enters the fork path, create_branch collides,
and the engine wedged with 'orphaned table state ... incomplete prior delete;
run cleanup' — which cleanup could not even fix, because the branch is still a
live manifest branch. This hit load, mutate, ingest, and the merge fork path
(one shared engine chokepoint), so a routine deploy restart or client
disconnect could wedge a branch.
Fix: treat the per-table fork ref as derived state of the manifest. fork_branch_
from_state returns a typed ForkOutcome instead of a human 'incomplete prior
delete' error; on RefAlreadyExists the db layer reclaims the manifest-
unreferenced fork (force_delete_branch + re-fork, exactly once) and proceeds.
A live committed fork is still routed to a retryable conflict before the fork
path, so concurrent first-writes stay correct.
Reclaim is only safe if no in-process writer can be mid-fork, so the write
entry points (load, mutate) acquire the per-(table, branch) write queues for
all touched tables up front — before the fork, held through the publish — when
forking a non-main branch. commit_all accepts these pre-held guards instead of
re-acquiring (the queue is non-re-entrant). The merge fork path already holds
the queue and self-heals through the shared wrapper. Cross-process in-flight
forks remain the documented one-winner-CAS gap.
Mechanical prep folded in: mutation IR lowering is hoisted so the touched-table
set is known before execution; commit_all gains the held_guards parameter.
Flips recreate_over_orphaned_fork_before_cleanup_is_actionable to assert
self-heal; fork_collision_with_live_concurrent_fork_is_retryable still holds.
Docs: writes.md cancelled-future note, invariants.md cross-process known gap.
* fix(cleanup): reconcile per-table manifest-unreferenced forks
reconcile_orphaned_branches keyed orphans on the branch NAME (absent from the
manifest), so it only reclaimed forks from a fully-deleted branch. A fork left
on a still-live branch by an interrupted first-write was never reclaimed — the
backstop the handoff expected cleanup to provide did not cover that case.
Broaden it to a per-table authority test: a Lance branch B on table T is an
orphan iff B is not a live manifest branch (delete-leftover) OR the manifest's
branch-B snapshot does not place T on B (interrupted first-write). Per-branch
snapshots are resolved once and cached across tables. Legitimately-forked
tables, main, and internal/system branches are never reclaimed; children are
dropped before parents to avoid Lance's referenced-parent RefConflict. The
commit-graph half stays whole-branch (per-table doesn't apply there).
This is the guaranteed-convergence backstop to the write-path self-heal: it
reclaims any fork the write path never revisits, and is what Lance's own
create_branch docstring asks embedders to provide for zombie/orphan refs.
* fix: reclaim self-validates against fresh manifest authority
The fork reclaim force-deletes a Lance branch ref, gated on the caller's proof
that the manifest does not place the table on the branch. But the first-write
path obtains that proof via snapshot_for_branch, which returns the coordinator's
CACHED snapshot when the handle is bound to the branch (an embedded handle on
the branch, or branch_merge's target swap). If that snapshot is stale and a
concurrent writer already published a legitimate fork, the reclaim would
force-delete it and re-fork from source, stranding the manifest at a version the
recreated ref no longer has.
Make the destructive primitive own its safety precondition: re-derive it from a
FRESH manifest read (fresh_snapshot_for_branch, which bypasses the cache)
immediately before force-deleting. If fresh authority shows the table is on the
branch, refuse with a retryable conflict instead of destroying a valid fork.
Correct for any caller regardless of snapshot staleness. Also stop branching on
Lance's exact RefConflict prose (loosened match; typed-variant is the durable
follow-up). Addresses PR review (Codex P1, Greptile P2).
* fix: cover delete-cascade edges in up-front fork-queue acquisition
A node delete cascades to every edge table touching that node (execute_delete_
node), forking those edge tables during execution. But touched_table_keys
derived the up-front fork-queue set from the IR ops alone (just node:Type), so a
branch delete that forks node + cascade edges held only the node queue —
commit_all then saw cascade-edge keys it had no guard for.
The touched set is a pure function of (IR ops + catalog), so compute the
COMPLETE set: op types plus, for delete-node ops, the cascade edges derived the
same way the executor derives them (from_type/to_type match). Pre-computed now
equals actual by construction.
Also promote commit_all's held-guard coverage check out of debug_assert into an
all-builds check that fails the write with a typed manifest_internal error: a
load-bearing serialization invariant must fail loudly+safely in release, not
silently proceed unguarded if a future execution path ever touches a table
outside the pre-computed set.
Adds branch_cascade_delete_forks_node_and_edges_under_held_queues, which drives
the cascade path on a branch (the gap the existing insert/load tests missed).
Addresses PR review (Cursor medium, Greptile P2).
* fix(cleanup): serialize fork reclaim against in-process live writers
The broadened per-table reconciler force_delete'd an orphan candidate on a LIVE
branch without holding the per-(table, branch) write queue. An in-process
first-write fork in its fork->publish window holds that queue and has not yet
advanced the manifest, so it looks exactly like an origin-2 orphan — concurrent
cleanup could delete the ref the writer still holds and is about to publish.
(The old branch-name-based reconciler did not have this race: a deleted branch
cannot have a live first-write.)
Bring the reconciler under the same invariant the write-path reclaim already
obeys: never force_delete a fork ref without holding the (table, branch) write
queue AND confirming, under it, from a fresh read, that the ref is still
manifest-unreferenced. Acquire one key at a time (no lock-order inversion vs
multi-table acquire_many writers); if the writer published meanwhile, the fresh
re-check sees the table on the branch and skips. Cross-process writers remain
the documented one-winner-CAS gap. Addresses PR review (Cursor high).
* fix: classify create_branch failure by ref existence, not by failure
fork_branch_from_state mapped ANY create_branch failure to RefAlreadyExists,
routing transient I/O / version / Lance-internal errors into the destructive
reclaim path and masking the real error as a retryable conflict.
Branch on the actual fact instead: on create_branch failure, check whether the
ref exists (list_branches). Only a genuinely pre-existing ref — a fully-formed
manifest-unreferenced fork — is a reclaim candidate; any other failure
propagates with fidelity. We deliberately do NOT force-delete on a not-found-ref
failure: it is indistinguishable from a transient error on a fresh create, and
force-deleting there is the overreach the fresh-authority guard already removed.
A phase-1-only Lance zombie (rarer; create_branch interrupted mid its two
internal phases) surfaces as the propagated error for manual reclaim.
Addresses PR review (Cursor medium).
* fix(cleanup): skip (not delete) on a transient re-check error for a live branch
The reconcile pre-delete re-check treated ANY fresh_snapshot error as 'still an
orphan' and proceeded to force_delete. A transient manifest read failure on a
LIVE branch could therefore destroy a fork the manifest still considers
legitimate — inconsistent with the write-path reclaim (aborts on the same error)
and the candidate scan (skips on snapshot failure).
Distinguish the two origins under the queue: a branch absent from the manifest
authority (origin 1) is a confirmed orphan and is deleted without a fresh read
(no live writer can hold a deleted branch's queue); a LIVE branch (origin 2)
gets the fresh re-check and, on a transient read error, is SKIPPED — never
destroyed on ambiguity — converging on a later cleanup. Same don't-destroy-on-
ambiguous-error principle as the create_branch failure classification.
Addresses PR review (Cursor medium).
* fix(cleanup): unify fork-ref reclaim on fresh authority under the queue
Consolidates the reconcile/reclaim hardening from PR review (the earlier per-site
commits were collapsed when reconciling with the main merge). Both destructive
fork-ref sites — the write-path reclaim and the cleanup reconciler — now share
one classifier, classify_fork_ref -> ForkRefStatus { Legitimate, Orphan,
Indeterminate }, evaluated from FRESH manifest authority under the held
(table, branch) write queue. A fork ref is destroyed ONLY on a confirmed Orphan;
a Legitimate (concurrent writer published a real fork) or Indeterminate
(transient read) status is never destroyed — the write path maps it to a
retryable conflict, cleanup maps it to skip. This closes, by construction:
- reclaim trusting a possibly-cached caller proof (Codex P1);
- reconcile racing an in-process live fork without the queue (Cursor);
- delete-on-transient-error in the re-check (Cursor/Greptile);
- origin-1 trusting a stale live_branches capture for a created-since branch
(Cursor/Greptile P1).
Having one classifier removes the duplication that let the two sites drift.
ForkOutcome is made pub to match the sealed trait method returning it. Verified
green on Lance 7.0.0 (full engine suite + 48/48 failpoints).
* test(cleanup): pin classify_fork_ref decision (Legitimate / Orphan / ghost)
Both fork-ref reclaim sites (write-path reclaim + cleanup reconciler) route
their destroy/skip decision through classify_fork_ref, but it had no direct
test — reverting the fresh-authority logic was not test-detectable. Add a
deterministic in-source unit test that forges each state and asserts the status:
a manifest-placed fork -> Legitimate (never destroyed); a ref the manifest does
not place on the branch -> Orphan; a ref for a branch absent from the manifest
-> Orphan (ghost reclaim preserved). This makes the core fresh-authority
decision behind every reclaim fix revert-detectable in one place.
(The Indeterminate arm — transient read on a live branch -> skip — needs an
injected read failure and is left to the failpoints suite; the cross-process
cleanup-vs-writer and cached-snapshot reclaim races are the documented
one-winner-CAS gap, not reachable same-process bugs, so they are not faked here.)
* test(cleanup): pin the Indeterminate (transient re-check) reclaim arm
Closes the last untested classify_fork_ref arm. Adds a 'classify.fresh_read'
failpoint (no-op without the failpoints feature) that simulates a transient
failure of the fresh-authority read, and a failpoints test driving it through
cleanup: a genuine origin-2 orphan on a LIVE branch whose fresh re-check fails
classifies as Indeterminate, so the reconciler SKIPS it (never destroys on an
inconclusive read) and reclaims it on the next run once the read succeeds.
This makes the don't-destroy-on-ambiguity rule revert-detectable end-to-end.
The only paths now left untested are the cross-process cleanup-vs-writer and
reclaim-vs-publish races — the documented one-winner-CAS gap (cleanup is
&mut self / CLI-only, so no reachable same-process race), not faked here.
* test(server): avoid stale schema apply route handle
* fix(cleanup): report indeterminate fork authority clearly
2026-06-15 22:17:25 +02:00
Omnigraph acts as operational state & coordination layer for agents.
2026-06-15 17:22:28 +02:00
Hundreds of agents can enrich the graph on parallel isolated branches and changes can be reviewed and merged safely.
2026-05-30 10:05:30 +01:00
2026-05-30 12:45:26 +02:00
- Git-style versioning & branching
- Multimodal retrieval (graph+vector/fts+filters) optimized for context assembly
2026-05-30 10:05:30 +01:00
- Object storage native (S3, RustFS)
2026-05-30 12:45:26 +02:00
- Native blob-as-data support (docs, images, videos, etc)
2026-05-30 10:05:30 +01:00
- VPC, On-prem, hybrid deployment
2026-05-30 12:45:26 +02:00
- [`Lance` ](https://github.com/lance-format/lance ) format as open storage layer
2026-04-14 13:52:14 +03:00
2026-05-30 10:08:14 +01:00
| AS CODE | What it means |
2026-05-30 10:05:30 +01:00
|---|---|
| **Schema AS CODE** | Typed `.pg` schemas, planned, applied, enforced |
| **Context AS CODE** | Linted queries & agentic nudges, versioned and reusable |
| **Security AS CODE** | Cedar policies enforced server-side on every mutation |
| **Dashboards AS CODE** | Declarative views & controls over the graph *(coming)* |
2026-04-14 13:52:14 +03:00
2026-05-30 10:05:30 +01:00
## Core Use Cases
2026-04-28 03:46:21 +03:00
2026-05-30 12:45:26 +02:00
| Use case | What it's for
2026-05-30 10:05:30 +01:00
|---|---|
2026-05-30 12:45:26 +02:00
| **Company brain** | Org knowledge unified into one queryable graph |
2026-05-30 10:05:30 +01:00
| **Context graph** | Decision traces and codified tribal knowledge |
| **Agentic memory** | Durable, versioned memory for long-running agents |
| **Dev graph** | Issues & dependency model for coding agents |
| **R&D data layer** | Experiments & trials data written into branches |
| **ML workflows** | Versioned, branchable graphs for training & eval |
| **Karpathy's LLM wiki** | A living, agent-updatable knowledge base |
2026-04-28 03:46:21 +03:00
2026-04-10 20:49:41 +03:00
## Quick Install
```bash
2026-04-11 02:19:21 +03:00
curl -fsSL https://raw.githubusercontent.com/ModernRelay/omnigraph/main/scripts/install.sh | bash
2026-04-10 20:49:41 +03:00
```
2026-04-10 23:26:09 +03:00
This installs `omnigraph` and `omnigraph-server` into `~/.local/bin` from
2026-04-11 14:34:29 +03:00
published release binaries.
2026-04-10 23:26:09 +03:00
2026-04-11 04:01:39 +03:00
Or install with Homebrew:
```bash
brew tap ModernRelay/tap
brew install ModernRelay/tap/omnigraph
```
2026-05-02 23:37:09 +03:00
For starter graphs and agent skills to bootstrap and operate Omnigraph, see [`ModernRelay/omnigraph-cookbooks` ](https://github.com/ModernRelay/omnigraph-cookbooks ).
2026-04-14 19:37:05 +03:00
2026-04-10 20:49:41 +03:00
## One-Command Local RustFS Bootstrap
```bash
2026-04-11 02:19:21 +03:00
curl -fsSL https://raw.githubusercontent.com/ModernRelay/omnigraph/main/scripts/local-rustfs-bootstrap.sh | bash
2026-04-10 20:49:41 +03:00
```
That bootstrap:
- starts RustFS on `127.0.0.1:9000`
2026-05-24 16:46:00 +01:00
- creates a bucket and S3-backed graph
2026-04-10 20:49:41 +03:00
- loads the checked-in context fixture
- launches `omnigraph-server` on `127.0.0.1:8080`
Docker must be installed and running first.
2026-04-10 23:26:09 +03:00
The RustFS bootstrap prefers the rolling `edge` binaries and only falls back to
source builds when release assets are unavailable.
2026-05-24 16:46:00 +01:00
If a previous run left objects under the same graph prefix but did not finish
initializing the graph, rerun with `RESET_REPO=1` or set `PREFIX` to a new
2026-04-12 21:08:04 +03:00
value.
2026-04-10 20:49:41 +03:00
## Common Commands
2026-06-16 11:40:11 +03:00
A deployment is a **cluster** . A `cluster.yaml` declares its graphs, schemas,
stored queries, and policies; you converge it with `cluster apply` and serve it.
The server is cluster-first — it boots only from a cluster and serves every graph
under `/graphs/{id}/…` . Day-to-day work goes through that server: graphs are
addressed with `--server <name>` (+ `--graph <id>` ), and `query` /`mutate` invoke
a stored query from the catalog **by name** .
2026-04-10 20:49:41 +03:00
```bash
2026-06-16 11:40:11 +03:00
# 1. Converge the declared cluster, then serve it
omnigraph cluster apply --config ./company-brain
omnigraph-server --cluster ./company-brain --bind 0.0.0.0:8080
# or config-free from object storage — the bucket IS the deployment:
# omnigraph-server --cluster s3://my-bucket/company-brain --bind 0.0.0.0:8080
2026-06-16 11:18:16 +03:00
2026-06-16 11:40:11 +03:00
# 2. Work against the served graph — stored queries invoked by name
omnigraph query find_people --server prod --graph knowledge --params '{"q":"AI safety"}'
omnigraph mutate add_person --server prod --graph knowledge --params '{"name":"Mina"}'
omnigraph load --data ./data.jsonl --mode merge --server prod --graph knowledge
2026-06-16 11:18:16 +03:00
2026-06-16 11:40:11 +03:00
# 3. Branch and merge, Git-style across the whole graph
omnigraph branch create --from main review/2026-06 --server prod --graph knowledge
omnigraph branch merge review/2026-06 --into main --server prod --graph knowledge
2026-06-16 11:18:16 +03:00
```
2026-06-16 11:40:11 +03:00
Set a default scope (or a `--profile` ) in `~/.omnigraph/config.yaml` — operator
identity, named servers/clusters, credentials — and the `--server` /`--graph`
flags drop away (`omnigraph query find_people --params …` ).
2026-06-16 11:18:16 +03:00
2026-06-16 11:40:11 +03:00
**Local / ad-hoc.** For quick iteration on a standalone graph (no cluster, no
server), address storage directly with `--store` (or a positional `file://` /
`s3://` URI) and run ad-hoc `.gq` with `--query` (the positional then selects
which query in the file):
2026-06-16 11:18:16 +03:00
```bash
2026-06-16 11:40:11 +03:00
omnigraph init --schema ./schema.pg ./graph.omni
omnigraph load --data ./data.jsonl --mode merge --store ./graph.omni
omnigraph query --query ./queries.gq get_person --params '{"name":"Alice"}' --store ./graph.omni
2026-04-10 20:49:41 +03:00
```
2026-06-16 11:40:11 +03:00
See [docs/user/cli/index.md ](docs/user/cli/index.md ), the
[CLI reference ](docs/user/cli/reference.md ), the
[cluster guide ](docs/user/clusters/index.md ), and the
[deployment guide ](docs/user/deployment.md ) for schema apply, snapshots, commits,
profiles, and policy/queries tooling.
2026-04-10 20:49:41 +03:00
2026-05-30 14:29:49 +02:00
## Clients
For programmatic access to a running `omnigraph-server` :
- **TypeScript SDK** — [`@modernrelay/omnigraph` ](https://www.npmjs.com/package/@modernrelay/omnigraph ) ([source ](https://github.com/ModernRelay/omnigraph-ts/tree/main/packages/sdk )). Instance-per-client, typed errors, camelCase types, async-iterator streaming export.
```bash
npm install @modernrelay/omnigraph
```
- **Model Context Protocol server** — [`@modernrelay/omnigraph-mcp` ](https://www.npmjs.com/package/@modernrelay/omnigraph-mcp ) ([source ](https://github.com/ModernRelay/omnigraph-ts/tree/main/packages/mcp )). Bridges Omnigraph to LLM hosts (Claude Desktop, Claude Code, …) over stdio. Exposes tools and resources for schema, branches, queries, mutations, ingest, and bundles curated best-practices guidance from the cookbook.
```bash
npm install -g @modernrelay/omnigraph -mcp
```
Both packages are versioned in lockstep with `omnigraph-server` on major.minor: `@modernrelay/omnigraph@X.Y.*` targets `omnigraph-server@X.Y.*` . See [`ModernRelay/omnigraph-ts` ](https://github.com/ModernRelay/omnigraph-ts ) for the monorepo.
2026-04-10 20:49:41 +03:00
## Docs
2026-05-15 03:45:22 +03:00
- [Install guide ](docs/user/install.md )
- [Deployment guide ](docs/user/deployment.md )
2026-04-10 20:49:41 +03:00
## Build And Test
```bash
cargo build --workspace
cargo check --workspace
cargo test --workspace
```
Notes:
- Rust stable toolchain, edition 2024
- CI runs `cargo test --workspace --locked`
- Full CI and some local test flows require `protobuf-compiler`
- S3 integration tests expect an S3-compatible endpoint such as RustFS
## Workspace Crates
2026-06-16 11:18:16 +03:00
- `crates/omnigraph-compiler` : shared schema/query parser, typechecker, catalog, and IR lowering (zero Lance dependency)
- `crates/omnigraph` (package `omnigraph-engine` ): storage/runtime, branching, merge, change detection, query execution, and embeddings
- `crates/omnigraph-policy` : Cedar policy compilation and enforcement
- `crates/omnigraph-api-types` : shared HTTP wire DTOs used by both the server and the CLI
- `crates/omnigraph-cluster` : cluster config validation, planning, and apply (the control plane)
- `crates/omnigraph-server` : Axum HTTP server — cluster-first, serving N graphs under `/graphs/{id}/…`
- `crates/omnigraph-cli` : CLI for graph lifecycle (init/load), query/mutate, branch/commit/merge, schema/lint, snapshot/export, cluster control, policy/queries, profiles, and maintenance (optimize/repair/cleanup)
2026-04-10 20:49:41 +03:00
## Contributing
Please open an issue, spec, or design discussion before sending large code
changes. Design feedback and concrete problem statements are the fastest way to
collaborate on the roadmap.
2026-05-01 14:39:21 +02:00
## Community
Join the [Omnigraph Slack community ](https://join.slack.com/t/omnigraphworkspace/shared_invite/zt-3wfpglyxj-lHvJGhuySPfqLtN35uJZNw )
to ask questions, share feedback, and follow development.
