mirror of
https://github.com/ModernRelay/omnigraph.git
synced 2026-06-12 01:45:14 +02:00
Merge branch 'main' into devin/1779464281-mr-656-inline-query-strings
Resolve conflicts: keep query/mutate canonical CLI subcommands and top-level lint command (this branch) alongside the repo→graph terminology rename from main. Update test helpers (repo_path → graph_path, init_repo → init_graph, app_for_loaded_repo → app_for_loaded_graph) and align tempdir variable names so the merged tests compile. Drop the now- unused QueryCommand enum (Lint was promoted to a top-level Command). Co-Authored-By: Ragnor Comerford <ragnor.comerford@gmail.com>
This commit is contained in:
Devin AI
commit
9ff4af47fb
79 changed files with 2780 additions and 1894 deletions
|
|
@ -10,7 +10,7 @@ Three views, increasing zoom:
|
|||
2. **Layer view** — the eight-layer stack inside one OmniGraph process.
|
||||
3. **Component zoom-ins** — what's inside each layer.
|
||||
|
||||
For runtime flows (read query, mutation), see [`docs/dev/execution.md`](execution.md). For the on-disk layout of a repo, see [`docs/user/storage.md`](../user/storage.md).
|
||||
For runtime flows (read query, mutation), see [`docs/dev/execution.md`](execution.md). For the on-disk layout of a graph, see [`docs/user/storage.md`](../user/storage.md).
|
||||
|
||||
L1 (orange in the diagrams) is what we inherit from Lance; L2 (blue) is what OmniGraph adds. The L1/L2 framing is also called out in prose at the bottom of this doc.
|
||||
|
||||
|
|
@ -63,7 +63,7 @@ flowchart TB
|
|||
subgraph engine[omnigraph engine]
|
||||
plan[exec query and mutation]:::l2
|
||||
gi[graph index CSR/CSC<br/>RuntimeCache LRU 8]:::l2
|
||||
coord[coordinator<br/>ManifestRepo · CommitGraph]:::l2
|
||||
coord[coordinator<br/>ManifestCoordinator · CommitGraph]:::l2
|
||||
end
|
||||
|
||||
subgraph storage[storage trait — wraps Lance]
|
||||
|
|
@ -132,7 +132,7 @@ flowchart TB
|
|||
|
||||
subgraph state[graph state]
|
||||
coord[GraphCoordinator]:::l2
|
||||
mr[ManifestRepo<br/>db/manifest.rs]:::l2
|
||||
mr[ManifestCoordinator<br/>db/manifest.rs]:::l2
|
||||
cg[CommitGraph<br/>_graph_commits.lance]:::l2
|
||||
stg[MutationStaging<br/>per-query in-memory accumulator<br/>exec/staging.rs]:::l2
|
||||
end
|
||||
|
|
@ -166,7 +166,7 @@ Code paths:
|
|||
|
||||
- Read entry: `Omnigraph::query` at `crates/omnigraph/src/exec/query.rs:7`
|
||||
- Mutation entry: `Omnigraph::mutate` at `crates/omnigraph/src/exec/mutation.rs:511`
|
||||
- Manifest commit: `ManifestRepo::commit` at `crates/omnigraph/src/db/manifest.rs:280`
|
||||
- Manifest commit: `ManifestCoordinator::commit` at `crates/omnigraph/src/db/manifest.rs:280`
|
||||
- Graph index: `crates/omnigraph/src/graph_index/`
|
||||
- Loader: `Omnigraph::ingest` at `crates/omnigraph/src/loader/mod.rs:74`
|
||||
|
||||
|
|
|
|||
|
|
@ -16,12 +16,12 @@ This page explains what the policy says and how to change it.
|
|||
| **Disallow force pushes** | `true` | No history rewrites on `main`. |
|
||||
| **Disallow branch deletions** | `true` | `main` cannot be deleted. |
|
||||
| **Required conversation resolution** | `true` | All review comment threads must be resolved before merge. |
|
||||
| **Enforce on admins** | `true` | Even repo admins go through the gates. The point is no bypasses. |
|
||||
| **Enforce on admins** | `true` | Even repository admins go through the gates. The point is no bypasses. |
|
||||
| **Required signed commits** | not yet | Not enabled. Would lock out maintainers until everyone enrolls GPG/SSH commit signing. Tracked as a follow-up. |
|
||||
|
||||
## How to apply
|
||||
|
||||
Run from the repo root:
|
||||
Run from the repository root:
|
||||
|
||||
```bash
|
||||
./scripts/apply-branch-protection.sh
|
||||
|
|
@ -29,7 +29,7 @@ Run from the repo root:
|
|||
|
||||
The script reads `.github/branch-protection.json`, strips the human-readable `_comment` field (the GitHub API rejects unknown keys), and PUTs to `repos/ModernRelay/omnigraph/branches/main/protection`.
|
||||
|
||||
Requires `gh` authenticated with a token that has admin permissions on the repo.
|
||||
Requires `gh` authenticated with a token that has admin permissions on the repository.
|
||||
|
||||
To preview without applying:
|
||||
|
||||
|
|
@ -57,7 +57,7 @@ Outputs the live policy. Compare against `.github/branch-protection.json` to det
|
|||
|
||||
- **Audit trail**: `git log .github/branch-protection.json` shows every change with a reviewable diff and a merge commit.
|
||||
- **Disaster recovery**: if branch protection is accidentally removed or weakened via the UI, the JSON is the canonical recovery point.
|
||||
- **Consistency**: pairs with `.github/codeowners-roles.yml` (the CODEOWNERS source of truth). Repo policy lives in the repo.
|
||||
- **Consistency**: pairs with `.github/codeowners-roles.yml` (the CODEOWNERS source of truth). Repository policy lives in the repository.
|
||||
|
||||
## What this gates
|
||||
|
||||
|
|
@ -69,7 +69,7 @@ After branch protection is applied, every PR targeting `main` must:
|
|||
4. Have all review conversations resolved.
|
||||
5. Be squash- or rebase-merged (no merge commits).
|
||||
|
||||
Even repo admins are subject to these rules.
|
||||
Even repository admins are subject to these rules.
|
||||
|
||||
## Subsequent hardening (not in this PR)
|
||||
|
||||
|
|
|
|||
|
|
@ -2,9 +2,9 @@
|
|||
|
||||
`.github/workflows/`:
|
||||
|
||||
- **ci.yml**: text-only changes skip; otherwise `cargo test --workspace --locked` on ubuntu-latest with protobuf compiler. OpenAPI-drift check that auto-commits the regenerated `openapi.json` for same-repo PRs. Also runs the AGENTS.md cross-link integrity check (`scripts/check-agents-md.sh`).
|
||||
- **ci.yml**: text-only changes skip; otherwise `cargo test --workspace --locked` on ubuntu-latest with protobuf compiler. OpenAPI-drift check that auto-commits the regenerated `openapi.json` for same-repository PRs. Also runs the AGENTS.md cross-link integrity check (`scripts/check-agents-md.sh`).
|
||||
- **AWS feature build job**: `cargo build/test -p omnigraph-server --features aws` on ubuntu-latest.
|
||||
- **RustFS S3 integration**: spins up RustFS in Docker, runs `s3_storage`, `server_opens_s3_repo_directly_and_serves_snapshot_and_read`, and `local_cli_s3_end_to_end_init_load_read_flow`.
|
||||
- **RustFS S3 integration**: spins up RustFS in Docker, runs `s3_storage`, `server_opens_s3_graph_directly_and_serves_snapshot_and_read`, and `local_cli_s3_end_to_end_init_load_read_flow`.
|
||||
- **release-edge.yml**: on every push to main, retags `edge`, builds Linux/macOS-Intel/macOS-arm64 archives + sha256, publishes a rolling prerelease.
|
||||
- **release.yml**: on `v*` tags, builds the 3-platform matrix and updates the Homebrew tap (`scripts/update-homebrew-formula.sh`) by pushing the regenerated formula to `ModernRelay/homebrew-tap`.
|
||||
- **package.yml**: manual ECR image build; emits two image tags per commit (`<sha>`, `<sha>-aws`) via CodeBuild.
|
||||
|
|
|
|||
|
|
@ -2,13 +2,13 @@
|
|||
|
||||
`.github/CODEOWNERS` is **generated** — not hand-edited. The source of truth is `.github/codeowners-roles.yml`, expanded by `.github/scripts/render-codeowners.py`. CI rejects drift between the two and rejects direct edits to `CODEOWNERS` that don't accompany a yml change.
|
||||
|
||||
This setup gives every role change a reviewable PR and a permanent in-repo audit trail (`git log .github/codeowners-roles.yml`).
|
||||
This setup gives every role change a reviewable PR and a permanent in-repository audit trail (`git log .github/codeowners-roles.yml`).
|
||||
|
||||
## Current roles
|
||||
|
||||
| Role | Members | Scope |
|
||||
|---|---|---|
|
||||
| `engineering` | `@aaltshuler` | All code under `crates/**`, repo infrastructure, default for unmapped paths |
|
||||
| `engineering` | `@aaltshuler` | All code under `crates/**`, repository infrastructure, default for unmapped paths |
|
||||
| `docs` | `@aaltshuler`, `@ragnorc` | `docs/**`, README.md, AGENTS.md, CLAUDE.md, SECURITY.md |
|
||||
|
||||
GitHub treats multiple owners in a CODEOWNERS line as **"any one of them satisfies the review requirement"**. For docs, either named member can approve. To require N distinct approvers on a specific path, layer a CI check on top (not currently configured).
|
||||
|
|
@ -34,4 +34,4 @@ CI fails the PR if:
|
|||
- **Audit trail**: `git log .github/codeowners-roles.yml` is the canonical record of every role change. The rendered `CODEOWNERS` is a derived artifact.
|
||||
- **Roles are first-class**: paths reference roles, not raw handles. Renaming a person or rotating a role updates one place, not every path.
|
||||
- **Future extension**: scheduled rotation (weekly on-call, quarterly leads) plugs into the same yml without changing the path mappings. Not enabled today.
|
||||
- **Consistency with the product**: omnigraph itself enforces auditable Cedar policy. The repo's code-owner policy follows the same "policy as reviewed code" pattern.
|
||||
- **Consistency with the product**: omnigraph itself enforces auditable Cedar policy. The repository's code-owner policy follows the same "policy as reviewed code" pattern.
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Lance Docs Index (for OmniGraph agents)
|
||||
|
||||
OmniGraph sits on top of Lance. Many problems — index lifecycle, branching, transactions, fragments, compaction, vector/FTS internals — are answered upstream in Lance's docs, not in this repo.
|
||||
OmniGraph sits on top of Lance. Many problems — index lifecycle, branching, transactions, fragments, compaction, vector/FTS internals — are answered upstream in Lance's docs, not in this codebase.
|
||||
|
||||
This file is the curated entry point. **When you hit a Lance-shaped problem, find the matching topic below and fetch the listed URL(s) before guessing.** Don't grep our codebase for behavior that is documented authoritatively in Lance.
|
||||
|
||||
|
|
@ -156,13 +156,26 @@ If a future need pulls one of these into scope, add a row to the matching domain
|
|||
|
||||
When Lance ships a major release that changes any of the above (file format bump, new index type, transaction semantics change, new branching primitive), refresh this index in the same change as the omnigraph upgrade. Stale Lance pointers are worse than no pointers.
|
||||
|
||||
### Last alignment audit: 2026-05-02 (Lance 4.0.1 upstream; omnigraph pinned at 4.0.0)
|
||||
### Last alignment audit: 2026-05-22 (Lance 6.0.1 upstream; omnigraph pinned at 6.0.1)
|
||||
|
||||
A full read-through of every index page above was performed in the MR-793 cycle. Findings (no code changes required for PR #70):
|
||||
Migration from Lance 4.0.0 → 6.0.1 landed in this cycle (DataFusion 52 → 53, Arrow 57 → 58, lance-tokenizer 6.0.1 added, tantivy* removed). Direct 4 → 6 jump; v5.x was not used as an intermediate (rationale in `~/.claude/plans/shimmering-percolating-duckling.md`). Behavior-affecting findings:
|
||||
|
||||
- The MemWAL "three sub-pages" (Overview / Details / Implementation) turned out to be **anchor sections on the single existing page** at `https://lance.org/format/table/mem_wal/` — not separate URLs. Findings: MemWAL is opt-in (requires an unenforced primary key + explicit shard config; omnigraph doesn't use it), operates intra-table (LSM-tree for streaming writes into one Lance table), and does NOT overlap with MR-847's cross-table manifest-vs-Lance-HEAD recovery problem. MR-847's design is unaffected.
|
||||
- The distributed-indexing guide names Python APIs (`commit_existing_index_segments`, `merge_existing_index_segments`); the Rust analogues exist via `CreateIndexBuilder::execute_uncommitted` for scalar indices but **`build_index_metadata_from_segments` is `pub(crate)`** and blocks vector-index two-phase commits from outside the lance crate. Filed [lance-format/lance#6666](https://github.com/lance-format/lance/issues/6666) as a companion to [#6658](https://github.com/lance-format/lance/issues/6658).
|
||||
- "Stable Row ID for Index" is documented as **experimental** in lance-4.0.x. Our datasets enable stable row IDs at the dataset level (`WriteParams::enable_stable_row_ids = true`); confirming whether our created indices opt into stable-row-id mode is a follow-up worth doing before MR-848 (index reconciler) lands.
|
||||
- Fragment Reuse Index (FRI) is documented as one of three compaction strategies. omnigraph currently uses option 2 (immediate index rewrite at compaction time, via `omnigraph optimize`'s post-compaction rebuild). Adopting FRI is the explicit option for compaction-friendly index updates; relevant to MR-848.
|
||||
- **DatasetIndexExt moved** from `lance-index` to `lance::index` (Lance PR #6280, v5.0). Six import sites updated. `lance-index::IndexType` and `lance-index::is_system_index` stayed in `lance-index`. `omnigraph-cli` and `omnigraph-server` gained `lance = { workspace = true }` in their dev-dependencies.
|
||||
- **`DescribeTableResponse` gained `is_only_declared: Option<bool>`** (lance-namespace 6.0+, v5.0 PR #6186). Set to `Some(false)` in both `BranchManifestNamespace::describe_table` and `StagedTableNamespace::describe_table` — every table we return is physically materialized via `Dataset::open`, never "declared-only."
|
||||
- **`MergeInsertBuilder` execute_reader return shape preserved** `(Arc<Dataset>, MergeStats)`; the publisher CAS chain at `db/manifest/publisher.rs:370-391` works unchanged. Pinned by `tests/lance_surface_guards.rs::_compile_merge_insert_builder_method_chain`.
|
||||
- **`LanceError::TooMuchWriteContention` variant retained** in v6.0.1 (no rename). The typed publisher translation at `db/manifest/publisher.rs:417-430` continues to apply. Pinned by `lance_surface_guards.rs::lance_error_too_much_write_contention_variant_exists`.
|
||||
- **`ManifestLocation` field shape stable**: `.path: object_store::path::Path`, `.size: Option<u64>`, `.e_tag: Option<String>`, `.naming_scheme: ManifestNamingScheme`. Pinned by `lance_surface_guards.rs::manifest_location_field_shape`.
|
||||
- **`LanceFileVersion::default()` flipped V2_0 → V2_1** (v5.0). No effect — every `data_storage_version` callsite explicitly pins `Some(LanceFileVersion::V2_2)` (load-bearing for blob v2: `Blob v2 requires file version >= 2.2` enforced in `lance/src/dataset/write.rs:748`).
|
||||
- **`Dataset::checkout_version(N).await?.restore().await?`**: `restore()` takes `&mut self` and returns `Result<()>` (mutates in place, does not consume + return a new dataset). The recovery rollback hammer at `db/manifest/recovery.rs:505-522` continues to work. Pinned by `lance_surface_guards.rs::_compile_checkout_version_then_restore_signature`.
|
||||
- **`DatasetBuilder::from_namespace(...).with_branch(...).with_version(...).load()`** surface preserved (the namespace builder chain at `db/manifest/namespace.rs:162-174`). Pinned by `lance_surface_guards.rs::_compile_dataset_builder_from_namespace_signature`.
|
||||
- **`compact_files(&mut ds, CompactionOptions::default(), None)`** signature stable. `CompactionOptions` still does not expose `data_storage_version`; `compact_files` builds its own `WriteParams { ..Default::default() }`. Note: `LanceFileVersion::default()` is now V2_1 in v6, so optimize-rewritten fragments come out at V2_1 by default (was V2_0 in v4). Existing explicit V2_2 pins on creates/appends still apply.
|
||||
- **`Dataset::delete(predicate)` returns `DeleteResult { new_dataset: Arc<Dataset>, num_deleted_rows: u64 }`** — unchanged shape. Pinned by `lance_surface_guards.rs::_compile_delete_result_field_shape`. MR-A will repurpose this guard to the staged two-phase variant once `DeleteBuilder::execute_uncommitted` migration lands.
|
||||
- **File reader read methods now async** (Lance PR #6710, v6.0). No effect — omnigraph reaches Lance exclusively through `Dataset::scan` and the staged-write API.
|
||||
- **Tokenizer vendored as `lance-tokenizer`** (Lance PR #6512, v6.0). No effect — no direct tokenizer imports.
|
||||
- **Lance #6658 closed** (2026-05-14) but `DeleteBuilder::execute_uncommitted` did **not** ship in v6.0.1 — binary search across the release stream shows it first appears in `v7.0.0-beta.10` (the closing commits landed on main but didn't backport to the 6.x line). Tracked as MR-A: migrate `delete_where` to staged, retire the parse-time D2 mutation rule, extend recovery sidecar coverage. **Gated on the Lance v7.x bump**, not this PR. v7.0.0-rc.1 dropped 2026-05-21.
|
||||
- **Lance #6666 still open** (`build_index_metadata_from_segments` public): vector-index two-phase blocked; inline `create_vector_index` residual retained.
|
||||
- **Lance #6877 still open** (`MergeInsertBuilder` dup-rowid): PR #109's `SourceDedupeBehavior::FirstSeen` + `check_batch_unique_by_keys` precondition stay load-bearing.
|
||||
|
||||
Surface guards added: `crates/omnigraph/tests/lance_surface_guards.rs` (8 named guards; 3 runtime + 5 compile-only). Future Lance bumps re-run this file first as the smoke check. Two additional guards from the original plan deferred to follow-up (`manifest_cas_returns_row_level_contention_variant` needs full publisher-race harness; `table_version_metadata_byte_compatible_with_v4` needs `pub(crate)` reach extension).
|
||||
|
||||
Bump this date stanza on the next alignment pass.
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
|
|||
| `merge_truth_table.rs` | Merge-pair truth table (MR-786): all 9×9 `(left_op, right_op)` cells from `{noop, addNode, removeNode, addEdge, removeEdge, setProperty, dropProperty, addLabel, removeLabel}`. Adding a new op to `OpVariant` forces a compile error in `build_case` until the new row + column are dispositioned. 36 executable cells run through real `branch_merge` with a structured oracle (`MergeOutcome` / `MergeConflictKind` + graph-state assert); 45 cells involving `dropProperty`/`addLabel`/`removeLabel` are recorded as `Unsupported` until the mutation grammar grows. |
|
||||
| `runs.rs` | Direct-publish writes: cancellation, concurrent-writer CAS, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery) |
|
||||
| `staged_writes.rs` | TableStore staged-write primitives (`stage_append`, `stage_merge_insert`, `commit_staged`, `scan_with_staged`, `count_rows_with_staged`) — primitive-level only; engine code uses the in-memory `MutationStaging` accumulator instead |
|
||||
| `lifecycle.rs` | Repo lifecycle, schema state |
|
||||
| `lifecycle.rs` | Graph lifecycle, schema state |
|
||||
| `point_in_time.rs` | Snapshots, time travel (`snapshot_at_version`, `entity_at`) |
|
||||
| `changes.rs` | `diff_between` / `diff_commits` |
|
||||
| `consistency.rs` | Cross-table snapshot isolation, atomic publish |
|
||||
|
|
@ -31,7 +31,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
|
|||
| `traversal.rs` | `Expand`, variable-length hops, anti-join |
|
||||
| `aggregation.rs` | `count`, `sum`, `avg`, `min`, `max` |
|
||||
| `export.rs` | NDJSON streaming export filters |
|
||||
| `s3_storage.rs` | S3-backed repo (skipped unless `OMNIGRAPH_S3_TEST_BUCKET` is set) |
|
||||
| `s3_storage.rs` | S3-backed graph (skipped unless `OMNIGRAPH_S3_TEST_BUCKET` is set) |
|
||||
| `lance_version_columns.rs` | Per-row `_row_last_updated_at_version` behavior |
|
||||
| `validators.rs` | Schema constraint enforcement (enum, range, unique, cardinality) across JSONL, insert, update paths |
|
||||
| `maintenance.rs` | `optimize` (compaction) + `cleanup` (version GC): empty/idempotent/no-op edges, policy validation, head preservation |
|
||||
|
|
@ -45,7 +45,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
|
|||
|
||||
## Test helpers
|
||||
|
||||
- **Engine** — `crates/omnigraph/tests/helpers/mod.rs`: `init_and_load()` (bootstrap a temp repo + load standard fixture), `snapshot_main()`, `snapshot_branch()`, query/mutation runners, row collection and counting. Use these instead of hand-rolling.
|
||||
- **Engine** — `crates/omnigraph/tests/helpers/mod.rs`: `init_and_load()` (bootstrap a temp graph + load standard fixture), `snapshot_main()`, `snapshot_branch()`, query/mutation runners, row collection and counting. Use these instead of hand-rolling.
|
||||
- **CLI** — `crates/omnigraph-cli/tests/support/mod.rs`: `Command`-style wrapper for invoking `omnigraph`, server-process spawning, fixture resolution, output assertion helpers.
|
||||
- **Server** — no shared helpers; server tests call the `Omnigraph` engine API directly and exercise endpoints over the wire.
|
||||
|
||||
|
|
@ -63,14 +63,14 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav
|
|||
CI runs three S3-backed tests against a containerized RustFS server (`.github/workflows/ci.yml` → `rustfs_integration` job):
|
||||
|
||||
- `cargo test -p omnigraph-engine --test s3_storage`
|
||||
- `cargo test -p omnigraph-server --test server server_opens_s3_repo_directly_and_serves_snapshot_and_read`
|
||||
- `cargo test -p omnigraph-server --test server server_opens_s3_graph_directly_and_serves_snapshot_and_read`
|
||||
- `cargo test -p omnigraph-cli --test system_local local_cli_s3_end_to_end_init_load_read_flow`
|
||||
|
||||
Locally, set `OMNIGRAPH_S3_TEST_BUCKET` (and the usual `AWS_*` vars including `AWS_ENDPOINT_URL_S3` for non-AWS) before running. Without those, S3 tests skip gracefully.
|
||||
|
||||
## OpenAPI drift
|
||||
|
||||
`crates/omnigraph-server/tests/openapi.rs` regenerates `openapi.json` and diffs against the checked-in copy. CI auto-commits the regeneration on same-repo PRs and otherwise runs in strict-check mode (env: `OMNIGRAPH_UPDATE_OPENAPI`).
|
||||
`crates/omnigraph-server/tests/openapi.rs` regenerates `openapi.json` and diffs against the checked-in copy. CI auto-commits the regeneration on same-repository PRs and otherwise runs in strict-check mode (env: `OMNIGRAPH_UPDATE_OPENAPI`).
|
||||
|
||||
## Examples & benches
|
||||
|
||||
|
|
@ -79,7 +79,7 @@ Locally, set `OMNIGRAPH_S3_TEST_BUCKET` (and the usual `AWS_*` vars including `A
|
|||
|
||||
## Coverage tooling — what's missing
|
||||
|
||||
There is **no** coverage tooling in the repo today: no `tarpaulin.toml`, no `codecov.yml`, no coverage CI step. If you want to know whether your change is covered, the answer comes from reading and running the relevant integration tests, not from a tool.
|
||||
There is **no** coverage tooling in the repository today: no `tarpaulin.toml`, no `codecov.yml`, no coverage CI step. If you want to know whether your change is covered, the answer comes from reading and running the relevant integration tests, not from a tool.
|
||||
|
||||
If introducing coverage tooling is in scope for your task, the natural first step is `cargo-llvm-cov` wired into a separate CI job, and a per-crate threshold rather than a global one.
|
||||
|
||||
|
|
@ -97,7 +97,7 @@ How to check:
|
|||
- *Existing test covers the area but not your case* → **add an assertion or a fixture row to the existing test**, don't write a new function with `init_and_load()` again.
|
||||
- *No existing coverage in any test file* → only then write a new test; put it in the file that owns the area, or open a new file only if the area itself is new.
|
||||
|
||||
Three duplicated `init_and_load() → run_query → assert_eq` blocks where one parameterized test would do is the most common form of test rot in this repo. Don't add to it.
|
||||
Three duplicated `init_and_load() → run_query → assert_eq` blocks where one parameterized test would do is the most common form of test rot in this repository. Don't add to it.
|
||||
|
||||
## Before-every-task checklist
|
||||
|
||||
|
|
@ -106,7 +106,7 @@ When you pick up any change, walk through this:
|
|||
1. **Find existing coverage** (per the principle above). Don't just look at the first test file by name — grep for the symbol you're touching across every crate's `tests/`.
|
||||
2. **Run those tests locally before editing.** `cargo test --workspace --locked` for the broad pass; `-p <crate> --test <file>` for a focused loop. Confirm a clean baseline.
|
||||
3. **Decide extend-vs-new** explicitly. If you can extend an existing test (assertion, fixture row, parameterization), do that. Only add a new test fn or new file if no existing one owns the area.
|
||||
4. **Reuse the helpers.** `init_and_load()`, fixture files, the CLI `support` harness — re-use them. Don't bootstrap a fresh repo by hand if a helper exists.
|
||||
4. **Reuse the helpers.** `init_and_load()`, fixture files, the CLI `support` harness — re-use them. Don't bootstrap a fresh graph by hand if a helper exists.
|
||||
5. **Mind the boundary.** Per [docs/dev/invariants.md](invariants.md), test at the layer the change lives at — planner-level changes deserve planner-level tests, not just end-to-end.
|
||||
6. **For substrate-touching changes** (Lance behavior), reach for `failpoints` or fixture-driven scenarios, not stubbed-out mocks.
|
||||
7. **For server / API changes**, confirm the OpenAPI regeneration happens in `openapi.rs` and that the diff lands in `openapi.json`.
|
||||
|
|
|
|||
171
docs/releases/v0.5.0.md
Normal file
171
docs/releases/v0.5.0.md
Normal file
|
|
@ -0,0 +1,171 @@
|
|||
# Omnigraph v0.5.0
|
||||
|
||||
Omnigraph v0.5.0 is a substrate, security, and migration-safety release. It
|
||||
jumps the storage substrate from Lance 4 to Lance 6.0.1 (DataFusion 52 → 53,
|
||||
Arrow 57 → 58), introduces engine-wide Cedar policy enforcement on every
|
||||
authoring path, and ships a structured schema-lint v1 chassis with
|
||||
code-tagged diagnostics, soft drops, and an explicit `--allow-data-loss`
|
||||
flag for destructive migrations.
|
||||
|
||||
## Highlights
|
||||
|
||||
- **Lance 6.0.1 substrate**: bump from Lance 4.0.0 → 6.0.1, DataFusion 52 →
|
||||
53, Arrow 57 → 58. New optimizer rules (vectorized `IN`-list eq kernel,
|
||||
`PhysicalExprSimplifier`, push-limit-into-hash-join, CASE-NULL shortcut)
|
||||
reach predicates that flow through the engine. `lance-tokenizer` replaces
|
||||
tantivy internally; FTS behavior preserved.
|
||||
- **Cedar policy engine**: a new `omnigraph-policy` crate wires
|
||||
`Omnigraph::enforce(action, scope, actor)` into every `_as` writer
|
||||
(`mutate_as`, `load_as`, `apply_schema_as`, `branch_create_as`,
|
||||
`branch_merge_as`, `branch_delete_as`, plus the load and change
|
||||
variants). The HTTP server defaults to deny-all when no Cedar policy is
|
||||
configured; a YAML policy file is required to enable writes. Actor
|
||||
identity comes only from signed token claims — clients cannot set actor
|
||||
identity directly.
|
||||
- **Schema lint v1 chassis**: diagnostics now carry stable codes of the form
|
||||
`OG-XXX-NNN` instead of free-form messages. `omnigraph schema plan` and
|
||||
`apply` understand soft drops on properties and types — destructive drops
|
||||
require the new `--allow-data-loss` flag (Hard mode) at the CLI and an
|
||||
equivalent JSON flag over HTTP.
|
||||
- **Structured filter pushdown**: query-language predicates lower to
|
||||
DataFusion `Expr` and push down through Lance's `Scanner::filter_expr`
|
||||
instead of being flattened to SQL strings. This unlocks `CompOp::Contains`
|
||||
pushdown (via `array_has`), which previously fell through to in-memory
|
||||
post-scan filtering, and lets the DataFusion 53 optimizer rules above act
|
||||
on our predicates.
|
||||
- **HTTP `allow_data_loss` parity**: the destructive-drop guard now exists
|
||||
on both the CLI (`--allow-data-loss`) and HTTP (`allow_data_loss: true` in
|
||||
the schema-apply request body).
|
||||
- **Inline query strings on CLI and HTTP**: `omnigraph read` /
|
||||
`omnigraph mutate` and the corresponding HTTP endpoints accept inline
|
||||
`.gq` source, not just a file path. Easier ad-hoc queries, clearer
|
||||
request logs.
|
||||
- **Browser CORS layer**: optional CORS layer on `omnigraph-server` for
|
||||
browser-based UIs, gated by `OMNIGRAPH_CORS_ORIGINS`.
|
||||
- **Merge-insert dup-rowid fix**: Lance's `MergeInsertBuilder` could surface
|
||||
spurious `"Ambiguous merge inserts"` errors on sequential merges against
|
||||
rows previously rewritten by `merge_insert`. The engine now opts into
|
||||
`SourceDedupeBehavior::FirstSeen` with a `check_batch_unique_by_keys`
|
||||
fail-fast precondition that guarantees source-side dedup happens before
|
||||
Lance sees the batch.
|
||||
- **Branch-merge error-path recovery**: a branch merge that failed
|
||||
mid-flight could leave the in-process coordinator pointing at a stale
|
||||
active branch. The error path now restores the prior coordinator,
|
||||
matching the success path's invariant.
|
||||
- **Branch merge with blob columns**: external blob URIs are now
|
||||
materialized correctly during branch merge instead of being dropped or
|
||||
pointing at the source branch.
|
||||
- **Lance API surface guards**: a new test file
|
||||
(`crates/omnigraph/tests/lance_surface_guards.rs`) pins eight specific
|
||||
Lance API surfaces (`LanceError::TooMuchWriteContention`,
|
||||
`ManifestLocation` fields, `MergeInsertBuilder` return shape,
|
||||
`WriteParams::default`, `compact_files` signature, etc.) so the next
|
||||
Lance bump fails compile or runtime on any silent drift rather than
|
||||
producing wrong-state recovery in production.
|
||||
|
||||
## Behavior changes
|
||||
|
||||
- **On-disk format unchanged**: existing v0.4.2 datasets open unchanged.
|
||||
The Lance file format pin stays at V2_2 (required by Lance's blob v2
|
||||
feature).
|
||||
- **`omnigraph-server` defaults to deny-all under `--policy`**: starting a
|
||||
server with the policy feature enabled but no Cedar YAML policy
|
||||
configured rejects every write. Operators must supply a policy file to
|
||||
authorize anything.
|
||||
- **Schema-lint diagnostics carry stable codes**: messages now lead with
|
||||
`OG-XXX-NNN`. CI parsers or tooling that keyed off the v0.4.2 free-form
|
||||
text need to switch to code-based matching.
|
||||
- **Destructive schema drops require `--allow-data-loss`**: dropping a
|
||||
property or type returns a structured diagnostic by default.
|
||||
`omnigraph schema apply --allow-data-loss` (CLI) or
|
||||
`{"allow_data_loss": true}` (HTTP) opts into Hard mode.
|
||||
- **`HashJoinExec` null-aware semantics on anti-join**: a side effect of
|
||||
the DataFusion 53 bump — `NOT IN` semantics under null-valued anti-join
|
||||
columns are now correct per SQL standard. Queries that depended on the
|
||||
prior behavior would have been incorrect.
|
||||
|
||||
## Upgrade Notes
|
||||
|
||||
### Migration
|
||||
|
||||
- No data migration. v0.4.2 repos open directly on v0.5.0.
|
||||
|
||||
### Clients
|
||||
|
||||
- HTTP and SDK clients should switch any string-matching schema-lint
|
||||
parsing to code-based matching against the `OG-XXX-NNN` prefix.
|
||||
- Clients exercising destructive schema drops (`DropProperty`, `DropType`)
|
||||
must add the `allow_data_loss` request field (HTTP) or
|
||||
`--allow-data-loss` flag (CLI). Default is soft-drop-or-reject.
|
||||
- Clients consuming `mutate_as` / `load_as` / `apply_schema_as` / branch
|
||||
authoring APIs now flow through the policy enforcer. Anything bypassing
|
||||
authorization on v0.4.2 will be rejected on v0.5.0 once a policy is
|
||||
configured.
|
||||
|
||||
### Operators
|
||||
|
||||
- Configure a Cedar policy YAML for production servers before enabling
|
||||
writes; deny-all is the new default. The `omnigraph policy validate` /
|
||||
`test` / `explain` CLI commands are unchanged.
|
||||
- Bearer tokens continue to be the actor-identity source; review the
|
||||
signed-token-claim-only invariant in `docs/dev/invariants.md` if you've
|
||||
built custom authentication.
|
||||
- If your local CI uses RustFS for S3-compatible storage testing, our CI
|
||||
pins `rustfs/rustfs:1.0.0-beta.3` (the last known-good tag before the
|
||||
upstream credentials-policy change). Mirror the pin or set
|
||||
`RUSTFS_ALLOW_INSECURE_DEFAULT_CREDENTIALS=true` for the new image
|
||||
versions.
|
||||
|
||||
## Tests added or strengthened
|
||||
|
||||
- `crates/omnigraph/tests/lance_surface_guards.rs` — 8 named guards pinning
|
||||
Lance API surfaces against silent drift on future bumps.
|
||||
- `crates/omnigraph/tests/policy_engine_chassis.rs` — engine-level policy
|
||||
enforcement coverage; complements the existing HTTP policy tests.
|
||||
- Policy chassis e2e gap-fills — branch-merge, branch-create, branch-delete
|
||||
policy paths now have explicit end-to-end tests over HTTP and CLI.
|
||||
- Merge-pair truth table — exhaustive op-variant matrix for three-way
|
||||
merge across `noop`, `addNode`, `removeNode`, `addEdge`, `removeEdge`,
|
||||
`setProperty`, `dropProperty`, `addLabel`, `removeLabel`; the build
|
||||
fails to compile when a new op variant is added without dispositioning
|
||||
every pairing.
|
||||
- Merge-insert: regression for the dup-rowid bug class on the load surface
|
||||
(`load_merge_repeated_against_overlapping_keys_succeeds`), the update
|
||||
surface (`second_sequential_update_on_same_row_succeeds`), and the
|
||||
upstream-Lance-gap canary
|
||||
(`load_merge_window_2_documents_upstream_lance_gap`).
|
||||
- Maintenance + destructive-migration coverage — `omnigraph optimize` /
|
||||
`cleanup` boundary cases, plus schema-apply soft-drop and Hard-mode
|
||||
paths.
|
||||
- Stable-row-id preservation across `stage_overwrite` — pins the invariant
|
||||
that staged overwrites carry stable row IDs through to the committed
|
||||
fragment set.
|
||||
- `CompOp::Contains` pushdown regression
|
||||
(`ir_filter_with_list_contains_pushes_down`) — pins the new structured
|
||||
Expr pushdown path that retired the in-memory fallback.
|
||||
|
||||
## Included Changes
|
||||
|
||||
- Lance 4 → 6.0.1, DataFusion 52 → 53, Arrow 57 → 58 substrate upgrade.
|
||||
- `omnigraph-policy` crate with engine-wide Cedar enforcement and
|
||||
signed-token-claim-only actor identity.
|
||||
- Schema-lint v1 chassis with `OG-XXX-NNN` codes, soft `DropProperty` /
|
||||
`DropType` semantics, and `--allow-data-loss` for Hard mode.
|
||||
- HTTP `allow_data_loss` request field parity with the CLI flag.
|
||||
- Structured DataFusion `Expr` filter pushdown via
|
||||
`Scanner::filter_expr`, with `CompOp::Contains` lowered through
|
||||
`array_has`.
|
||||
- Inline `.gq` source acceptance on CLI and HTTP read/mutate endpoints.
|
||||
- Optional CORS layer on `omnigraph-server` for browser UIs.
|
||||
- Bug fixes: merge-insert dup-rowid (FirstSeen + uniqueness precondition),
|
||||
branch-merge coordinator restore on error, blob-column materialization
|
||||
during branch merge.
|
||||
- New Lance API surface-guard test file as the canary for future Lance
|
||||
bumps.
|
||||
- Recovery-sidecar coverage extended across the four write paths
|
||||
(`MutationStaging::finalize`, `schema_apply`, `branch_merge`,
|
||||
`ensure_indices`) with failpoint regression tests.
|
||||
- CI: pinned `rustfs/rustfs:1.0.0-beta.3` after the upstream `:latest`
|
||||
introduced a credentials-policy change.
|
||||
- Version bump to `0.5.0` across workspace crates, `Cargo.lock`,
|
||||
`openapi.json`, and the `AGENTS.md` surveyed version.
|
||||
19
docs/releases/v0.6.0.md
Normal file
19
docs/releases/v0.6.0.md
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
# Omnigraph v0.6.0
|
||||
|
||||
## Breaking Changes
|
||||
|
||||
- Renamed the Cedar resource entity from `Omnigraph::Repo` to `Omnigraph::Graph`.
|
||||
- Renamed policy API terminology from `repo_id` to `graph_id` on `PolicyCompiler::compile` and `PolicyEngine::load`.
|
||||
- Renamed query-lint schema source JSON from `"repo"` to `"graph"` for `schema_source.kind`.
|
||||
|
||||
## User Impact
|
||||
|
||||
- No on-disk migration is required. Existing `.omni` graphs continue to open with the same storage layout.
|
||||
- Supported YAML policy authoring is unchanged because the YAML schema does not expose the Cedar entity type name.
|
||||
- Operators with unsupported raw Cedar policy files should update `Omnigraph::Repo`
|
||||
resource references to `Omnigraph::Graph`.
|
||||
|
||||
## Documentation
|
||||
|
||||
- Public docs, CLI help, examples, server docs, and test helpers now consistently use "graph" for the OmniGraph data artifact.
|
||||
- GitHub/source repository terminology remains spelled out as "repository" where needed.
|
||||
|
|
@ -4,4 +4,4 @@
|
|||
- `_as` variants of every write API let callers override the actor: `mutate_as`, `ingest_as`, `branch_merge_as`, `apply_schema_as`, etc.
|
||||
- Actor IDs are persisted on `GraphCommit.actor_id` with split storage in `_graph_commit_actors.lance` (the commit graph is split into `_graph_commits.lance` for the linkage and `_graph_commit_actors.lance` for the actor map).
|
||||
- HTTP server uses the bearer-token actor automatically; CLI uses the local user / explicit env (no implicit actor).
|
||||
- Pre-v0.4.0 repos also stored actor IDs on `RunRecord.actor_id` in `_graph_runs.lance` / `_graph_run_actors.lance`. The Run state machine was removed in MR-771; those files are inert post-v0.4.0 and reclaimed by MR-770's production sweep.
|
||||
- Pre-v0.4.0 graphs also stored actor IDs on `RunRecord.actor_id` in `_graph_runs.lance` / `_graph_run_actors.lance`. The Run state machine was removed in MR-771; those files are inert post-v0.4.0 and reclaimed by MR-770's production sweep.
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ A reference for the `omnigraph` binary's command surface and `omnigraph.yaml` sc
|
|||
|
||||
| Command | Purpose |
|
||||
|---|---|
|
||||
| `init` | `--schema <pg>` → initialize a repo (also scaffolds `omnigraph.yaml` if missing) |
|
||||
| `init` | `--schema <pg>` → initialize a graph (also scaffolds `omnigraph.yaml` if missing) |
|
||||
| `load` | bulk load a branch (`--mode overwrite\|append\|merge`) |
|
||||
| `ingest` | branch-creating transactional load (`--from <base>`) |
|
||||
| `query` (alias: `read`) | run named read query; source via `--query <path>`, `-e`/`--query-string <GQ>`, or `--alias <name>` (exactly one). `read` is the deprecated previous name and prints a one-line warning to stderr |
|
||||
|
|
@ -19,7 +19,7 @@ A reference for the `omnigraph` binary's command surface and `omnigraph.yaml` sc
|
|||
| `commit list \| show` | inspect commit graph |
|
||||
| `run list \| show \| publish \| abort` | transactional run ops |
|
||||
| `schema plan \| apply \| show (alias: get)` | migrations |
|
||||
| `lint` (alias: `check`) | offline / repo-backed query validation. Replaces `query lint` / `query check`, which are kept as deprecated argv-level shims that print a one-line warning and rewrite to `omnigraph lint` |
|
||||
| `lint` (alias: `check`) | offline / graph-backed query validation. Replaces `query lint` / `query check`, which are kept as deprecated argv-level shims that print a one-line warning and rewrite to `omnigraph lint` |
|
||||
| `optimize` | non-destructive Lance compaction |
|
||||
| `cleanup --keep N --older-than 7d --confirm` | destructive version GC |
|
||||
| `embed` | offline JSONL embedding pipeline |
|
||||
|
|
|
|||
|
|
@ -1,13 +1,13 @@
|
|||
# CLI Guide
|
||||
|
||||
## Core Repo Flow
|
||||
## Core Graph Flow
|
||||
|
||||
```bash
|
||||
omnigraph init --schema ./schema.pg ./repo.omni
|
||||
omnigraph load --data ./data.jsonl --mode overwrite ./repo.omni
|
||||
omnigraph snapshot ./repo.omni --branch main --json
|
||||
omnigraph query --uri ./repo.omni --query ./queries.gq --name get_person --params '{"name":"Alice"}'
|
||||
omnigraph mutate --uri ./repo.omni --query ./queries.gq --name insert_person --params '{"name":"Mina","age":28}'
|
||||
omnigraph init --schema ./schema.pg ./graph.omni
|
||||
omnigraph load --data ./data.jsonl --mode overwrite ./graph.omni
|
||||
omnigraph snapshot ./graph.omni --branch main --json
|
||||
omnigraph query --uri ./graph.omni --query ./queries.gq --name get_person --params '{"name":"Alice"}'
|
||||
omnigraph mutate --uri ./graph.omni --query ./queries.gq --name insert_person --params '{"name":"Mina","age":28}'
|
||||
```
|
||||
|
||||
`omnigraph query` is the canonical read command (pairs with `POST /query`);
|
||||
|
|
@ -21,11 +21,11 @@ For ad-hoc reads and mutations (REPLs, AI agents, one-off scripts), pass the
|
|||
GQ source inline with `-e` / `--query-string` instead of a file path:
|
||||
|
||||
```bash
|
||||
omnigraph query --uri ./repo.omni \
|
||||
omnigraph query --uri ./graph.omni \
|
||||
-e 'query find($name: String) { match { $p: Person { name: $name } } return { $p.name, $p.age } }' \
|
||||
--params '{"name":"Alice"}'
|
||||
|
||||
omnigraph mutate --uri ./repo.omni \
|
||||
omnigraph mutate --uri ./graph.omni \
|
||||
-e 'query add($name: String, $age: I32) { insert Person { name: $name, age: $age } }' \
|
||||
--params '{"name":"Inline","age":42}'
|
||||
```
|
||||
|
|
@ -38,22 +38,22 @@ only the source loader changes.
|
|||
## Branching And Reviewable Data Flows
|
||||
|
||||
```bash
|
||||
omnigraph branch create --uri ./repo.omni --from main feature-x
|
||||
omnigraph branch list --uri ./repo.omni
|
||||
omnigraph branch merge --uri ./repo.omni feature-x --into main
|
||||
omnigraph branch create --uri ./graph.omni --from main feature-x
|
||||
omnigraph branch list --uri ./graph.omni
|
||||
omnigraph branch merge --uri ./graph.omni feature-x --into main
|
||||
|
||||
omnigraph ingest --data ./batch.jsonl --branch review/import-2026-04-09 ./repo.omni
|
||||
omnigraph export ./repo.omni --branch main --type Person > people.jsonl
|
||||
omnigraph commit list ./repo.omni --branch main --json
|
||||
omnigraph commit show --uri ./repo.omni <commit-id> --json
|
||||
omnigraph ingest --data ./batch.jsonl --branch review/import-2026-04-09 ./graph.omni
|
||||
omnigraph export ./graph.omni --branch main --type Person > people.jsonl
|
||||
omnigraph commit list ./graph.omni --branch main --json
|
||||
omnigraph commit show --uri ./graph.omni <commit-id> --json
|
||||
```
|
||||
|
||||
## Remote Server Mode
|
||||
|
||||
Serve a repo:
|
||||
Serve a graph:
|
||||
|
||||
```bash
|
||||
omnigraph-server ./repo.omni --bind 127.0.0.1:8080
|
||||
omnigraph-server ./graph.omni --bind 127.0.0.1:8080
|
||||
```
|
||||
|
||||
Read through the HTTP API:
|
||||
|
|
@ -73,22 +73,22 @@ and configure the matching `bearer_token_env` in `omnigraph.yaml`.
|
|||
|
||||
```bash
|
||||
omnigraph lint --query ./queries.gq --schema ./schema.pg --json
|
||||
omnigraph check --query ./queries.gq ./repo.omni --json
|
||||
omnigraph check --query ./queries.gq ./graph.omni --json
|
||||
|
||||
omnigraph schema plan --schema ./next.pg ./repo.omni --json
|
||||
omnigraph schema apply --schema ./next.pg ./repo.omni --json
|
||||
omnigraph schema plan --schema ./next.pg ./graph.omni --json
|
||||
omnigraph schema apply --schema ./next.pg ./graph.omni --json
|
||||
omnigraph policy validate --config ./omnigraph.yaml
|
||||
omnigraph policy test --config ./omnigraph.yaml
|
||||
omnigraph policy explain --config ./omnigraph.yaml --actor act-alice --action read --branch main
|
||||
|
||||
omnigraph commit list ./repo.omni --json
|
||||
omnigraph commit show --uri ./repo.omni <commit-id> --json
|
||||
omnigraph commit list ./graph.omni --json
|
||||
omnigraph commit show --uri ./graph.omni <commit-id> --json
|
||||
```
|
||||
|
||||
(The legacy `omnigraph run list/show/publish/abort` subcommands were removed in MR-771; mutations and loads publish atomically and the commit graph (`omnigraph commit list`) is the audit surface.)
|
||||
|
||||
`query lint` and `query check` are the same command surface. In v1, repo-backed
|
||||
lint uses local or `s3://` repo URIs; HTTP targets are only supported when you
|
||||
`query lint` and `query check` are the same command surface. In v1, graph-backed
|
||||
lint uses local or `s3://` graph URIs; HTTP targets are only supported when you
|
||||
also pass `--schema`.
|
||||
|
||||
## Config
|
||||
|
|
|
|||
|
|
@ -8,8 +8,8 @@ internal deploy automation.
|
|||
|
||||
Omnigraph supports two broad deployment shapes:
|
||||
|
||||
- local directory repos
|
||||
- `s3://` repos on AWS S3 or S3-compatible object stores
|
||||
- local directory graphs
|
||||
- `s3://` graphs on AWS S3 or S3-compatible object stores
|
||||
|
||||
The server binary and container image expose the same HTTP surface.
|
||||
|
||||
|
|
@ -20,18 +20,18 @@ Build or install:
|
|||
- `omnigraph`
|
||||
- `omnigraph-server`
|
||||
|
||||
Run against a local repo:
|
||||
Run against a local graph:
|
||||
|
||||
```bash
|
||||
omnigraph-server ./repo.omni --bind 0.0.0.0:8080
|
||||
omnigraph-server ./graph.omni --bind 0.0.0.0:8080
|
||||
```
|
||||
|
||||
Run against an object-store-backed repo:
|
||||
Run against an object-store-backed graph:
|
||||
|
||||
```bash
|
||||
OMNIGRAPH_SERVER_BEARER_TOKEN="change-me" \
|
||||
AWS_REGION="us-east-1" \
|
||||
omnigraph-server s3://my-bucket/repos/example/releases/2026-04-10-v0.1.0 \
|
||||
omnigraph-server s3://my-bucket/graphs/example/releases/2026-04-10-v0.1.0 \
|
||||
--bind 0.0.0.0:8080
|
||||
```
|
||||
|
||||
|
|
@ -46,7 +46,7 @@ curl -fsSL https://raw.githubusercontent.com/ModernRelay/omnigraph/main/scripts/
|
|||
The bootstrap:
|
||||
|
||||
- starts a local RustFS-backed object store
|
||||
- creates a bucket and S3-backed Omnigraph repo
|
||||
- creates a bucket and S3-backed Omnigraph graph
|
||||
- loads the checked-in context fixture
|
||||
- starts `omnigraph-server` on `127.0.0.1:8080`
|
||||
|
||||
|
|
@ -60,8 +60,8 @@ Useful overrides:
|
|||
|
||||
- `WORKDIR=/path/to/state`
|
||||
- `BUCKET=omnigraph-local`
|
||||
- `PREFIX=repos/context`
|
||||
- `RESET_REPO=1` to delete an existing partially initialized repo prefix before recreating it
|
||||
- `PREFIX=graphs/context`
|
||||
- `RESET_REPO=1` to delete an existing partially initialized graph prefix before recreating it
|
||||
- `BIND=127.0.0.1:8080`
|
||||
- `RUSTFS_CONTAINER_NAME=omnigraph-rustfs-demo`
|
||||
|
||||
|
|
@ -76,7 +76,7 @@ If `aws` is not installed, the script attempts a user-local AWS CLI install via
|
|||
running.
|
||||
|
||||
If a previous bootstrap left objects behind under the selected `PREFIX` but did
|
||||
not finish initializing the repo, rerun with `RESET_REPO=1` or choose a new
|
||||
not finish initializing the graph, rerun with `RESET_REPO=1` or choose a new
|
||||
`PREFIX`.
|
||||
|
||||
## Container Deployment
|
||||
|
|
@ -87,23 +87,23 @@ Build the image:
|
|||
docker build -t omnigraph-server:local .
|
||||
```
|
||||
|
||||
Run against a local repo:
|
||||
Run against a local graph:
|
||||
|
||||
```bash
|
||||
docker run --rm -p 8080:8080 \
|
||||
-v "$PWD/repo.omni:/data/repo.omni" \
|
||||
-v "$PWD/graph.omni:/data/graph.omni" \
|
||||
omnigraph-server:local \
|
||||
/data/repo.omni --bind 0.0.0.0:8080
|
||||
/data/graph.omni --bind 0.0.0.0:8080
|
||||
```
|
||||
|
||||
Run against an S3-backed repo:
|
||||
Run against an S3-backed graph:
|
||||
|
||||
```bash
|
||||
docker run --rm -p 8080:8080 \
|
||||
-e OMNIGRAPH_SERVER_BEARER_TOKEN="change-me" \
|
||||
-e AWS_REGION="us-east-1" \
|
||||
omnigraph-server:local \
|
||||
s3://my-bucket/repos/example/releases/2026-04-10-v0.1.0 \
|
||||
s3://my-bucket/graphs/example/releases/2026-04-10-v0.1.0 \
|
||||
--bind 0.0.0.0:8080
|
||||
```
|
||||
|
||||
|
|
@ -154,7 +154,7 @@ Manager secret whose `SecretString` is a JSON object of
|
|||
`{"actor_id": "token", ...}`:
|
||||
|
||||
```bash
|
||||
omnigraph-server-aws s3://my-bucket/repos/example ...
|
||||
omnigraph-server-aws s3://my-bucket/graphs/example ...
|
||||
# Environment:
|
||||
# OMNIGRAPH_SERVER_BEARER_TOKENS_AWS_SECRET=arn:aws:secretsmanager:us-east-1:123456789012:secret:omnigraph-tokens-AbCdEf
|
||||
```
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@ Mark a Vector property with `@embed("source_text_property")`. At ingest, the eng
|
|||
|
||||
## CLI `omnigraph embed` (offline file pipeline)
|
||||
|
||||
Operates on **JSONL files** (not on a repo). Three modes (mutually exclusive):
|
||||
Operates on **JSONL files** (not on a graph). Three modes (mutually exclusive):
|
||||
|
||||
- (default) `fill_missing` — only embed rows whose target field is empty
|
||||
- `--reembed-all` — overwrite all
|
||||
|
|
|
|||
|
|
@ -18,11 +18,11 @@ of MRs, internal recovery mechanics, or contributor-only invariants.
|
|||
| Write queries and mutations | [query-language.md](query-language.md) |
|
||||
| Use embeddings | [embeddings.md](embeddings.md) |
|
||||
|
||||
## Operate A Repo
|
||||
## Operate A Graph
|
||||
|
||||
| Goal | Read |
|
||||
|---|---|
|
||||
| Understand repo layout and URI support | [storage.md](storage.md) |
|
||||
| Understand graph layout and URI support | [storage.md](storage.md) |
|
||||
| Work with branches, commits, and snapshots | [branches-commits.md](branches-commits.md) |
|
||||
| Coordinate multi-query workflows | [transactions.md](transactions.md) |
|
||||
| Read diffs and change feeds | [changes.md](changes.md) |
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# HTTP Server (`omnigraph-server`)
|
||||
|
||||
Axum 0.8 + tokio + utoipa-generated OpenAPI. Single repo per process; deploy multiple processes for multi-tenant.
|
||||
Axum 0.8 + tokio + utoipa-generated OpenAPI. Single graph per process; deploy multiple processes for multi-tenant.
|
||||
|
||||
## Endpoint inventory
|
||||
|
||||
|
|
@ -136,7 +136,7 @@ See [deployment.md](deployment.md) for token-source operational details.
|
|||
|
||||
- `tower_http::TraceLayer::new_for_http()`
|
||||
- Policy decisions logged at INFO level with actor, action, branch, decision, matched rule
|
||||
- Startup logs: token source name, repo URI, bind address
|
||||
- Startup logs: token source name, graph URI, bind address
|
||||
- Graceful SIGINT shutdown
|
||||
|
||||
## Not implemented (by design or "TBD")
|
||||
|
|
@ -148,4 +148,4 @@ See [deployment.md](deployment.md) for token-source operational details.
|
|||
admission control" above). No global rate limiter is configured;
|
||||
add `tower_http::limit` if a graph-wide cap is needed.
|
||||
- Pagination — none (commits/branches return everything; export streams).
|
||||
- Multi-tenant routing — one repo per process.
|
||||
- Multi-tenant routing — one graph per process.
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ Every node type and every edge type is its own Lance dataset:
|
|||
- **Columnar Arrow storage**: each property is a column; nullable per Arrow schema.
|
||||
- **Fragments**: data is partitioned into fragments; new writes create new fragments.
|
||||
- **Manifest versioning**: every commit produces a new dataset version; old versions remain readable.
|
||||
- **Stable row IDs**: `enable_stable_row_ids: true` is set on every Lance dataset OmniGraph creates — node and edge data tables, `__manifest`, `_graph_commits.lance`, `_graph_commit_recoveries.lance`, and any future system tables. This is an architectural invariant: the flag is one-way at dataset create per Lance's row-id-lineage spec, so a future change that introduces a Lance dataset must preserve it. Consequences: `_row_created_at_version` and `_row_last_updated_at_version` are available on every dataset (load-bearing for change-feed validators); `CreateIndex × Rewrite` is not a retryable conflict, so indices survive `omnigraph optimize` without needing the Fragment Reuse Index; readers must use a Lance build that recognises the flag (our pinned 4.0.0 is fine). Pre-0.4.x repos created before this code path settled may have datasets without the flag and cannot be retrofitted in place — the supported path is dump-and-reload. The `stage_overwrite` rewrite path (used by `schema_apply`) preserves the flag through `Operation::Overwrite`; pinned by `stage_overwrite_preserves_stable_row_ids` in `crates/omnigraph/tests/staged_writes.rs`.
|
||||
- **Stable row IDs**: `enable_stable_row_ids: true` is set on every Lance dataset OmniGraph creates — node and edge data tables, `__manifest`, `_graph_commits.lance`, `_graph_commit_recoveries.lance`, and any future system tables. This is an architectural invariant: the flag is one-way at dataset create per Lance's row-id-lineage spec, so a future change that introduces a Lance dataset must preserve it. Consequences: `_row_created_at_version` and `_row_last_updated_at_version` are available on every dataset (load-bearing for change-feed validators); `CreateIndex × Rewrite` is not a retryable conflict, so indices survive `omnigraph optimize` without needing the Fragment Reuse Index; readers must use a Lance build that recognises the flag (our pinned 4.0.0 is fine). Pre-0.4.x graphs created before this code path settled may have datasets without the flag and cannot be retrofitted in place — the supported path is dump-and-reload. The `stage_overwrite` rewrite path (used by `schema_apply`) preserves the flag through `Operation::Overwrite`; pinned by `stage_overwrite_preserves_stable_row_ids` in `crates/omnigraph/tests/staged_writes.rs`.
|
||||
- **Append / delete / `merge_insert`**: native Lance write modes.
|
||||
- **Per-dataset branches** (Lance native): copy-on-write at the dataset level.
|
||||
- **Object-store agnostic**: file://, s3://, gs://, az://, http (read-only via Lance) — OmniGraph wires file:// and s3:// (`storage.rs`).
|
||||
|
|
@ -22,7 +22,7 @@ OmniGraph is **not** a single Lance dataset; it is a *graph* of datasets coordin
|
|||
- `edges/{fnv1a64-hex(edge_type_name)}` — one Lance dataset per edge type
|
||||
- `__manifest/` — the catalog of all sub-tables and their published versions
|
||||
- `_graph_commits.lance` / `_graph_commit_actors.lance` — the commit graph and its actor map
|
||||
- (legacy `_graph_runs.lance` / `_graph_run_actors.lance` from pre-v0.4.0 repos are inert; the run state machine was removed in MR-771 and these files are cleaned up via MR-770's production sweep)
|
||||
- (legacy `_graph_runs.lance` / `_graph_run_actors.lance` from pre-v0.4.0 graphs are inert; the run state machine was removed in MR-771 and these files are cleaned up via MR-770's production sweep)
|
||||
- **Manifest row schema** (`object_id, object_type, location, metadata, base_objects, table_key, table_version, table_branch, row_count`):
|
||||
- `object_type` ∈ `table | table_version | table_tombstone`
|
||||
- `table_key` ∈ `node:<TypeName> | edge:<EdgeName>`
|
||||
|
|
@ -36,7 +36,7 @@ OmniGraph is **not** a single Lance dataset; it is a *graph* of datasets coordin
|
|||
|
||||
The on-disk shape of `__manifest` is reconciled with the binary via a single stamp + dispatcher. `INTERNAL_MANIFEST_SCHEMA_VERSION` declares the shape this binary writes; the on-disk stamp `omnigraph:internal_schema_version` lives in the manifest dataset's schema-level metadata (Lance `update_schema_metadata`).
|
||||
|
||||
- **`init_manifest_repo`** stamps the current version at creation, so newly initialized repos never need migration.
|
||||
- **`init_manifest_graph`** stamps the current version at creation, so newly initialized graphs never need migration.
|
||||
- **Publisher open-for-write path** (`load_publish_state`) calls `migrate_internal_schema(&mut dataset)` before reading state. When the on-disk stamp matches the binary, this is a single metadata read with no writes; otherwise the dispatcher walks `match`-arm steps forward (1→2, 2→3, …) until the stamp matches, then proceeds with the publish. Reads stay side-effect-free.
|
||||
- **Forward-version protection**: a stamp *higher* than the binary's known version triggers a clear "upgrade omnigraph first" error. An old binary cannot clobber a newer schema by silently treating "unknown stamp" as "missing stamp".
|
||||
- **Idempotency**: each migration step is safe to re-run. A crash between two metadata updates inside a single step leaves the partial state; the next open re-runs the step and the second update lands. The dispatcher itself is a cheap stamp-read on the steady-state path.
|
||||
|
|
@ -50,14 +50,14 @@ Adding a new on-disk shape change is one constant bump (`INTERNAL_MANIFEST_SCHEM
|
|||
|
||||
## On-disk layout
|
||||
|
||||
A repo on disk is a directory tree of Lance datasets. Each dataset follows the standard Lance layout (`_versions/`, `data/`, `_indices/`, `_refs/`); OmniGraph adds the multi-dataset coordination by keeping `__manifest/` alongside the per-type datasets.
|
||||
A graph on disk is a directory tree of Lance datasets. Each dataset follows the standard Lance layout (`_versions/`, `data/`, `_indices/`, `_refs/`); OmniGraph adds the multi-dataset coordination by keeping `__manifest/` alongside the per-type datasets.
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
classDef l1 fill:#fef3e8,stroke:#c46900,color:#000
|
||||
classDef l2 fill:#e8f4fd,stroke:#1e6aa8,color:#000
|
||||
|
||||
repo["repo URI<br/>file:// or s3://bucket/prefix"]:::l2
|
||||
graph["graph URI<br/>file:// or s3://bucket/prefix"]:::l2
|
||||
|
||||
manifest["__manifest/<br/>L2 catalog of sub-tables"]:::l2
|
||||
nodes["nodes/{fnv1a64-hex}/<br/>one dataset per node type"]:::l2
|
||||
|
|
@ -66,12 +66,12 @@ flowchart TB
|
|||
recovery["__recovery/{ulid}.json<br/>recovery sidecars (transient)"]:::l2
|
||||
refs["_refs/branches/{name}.json<br/>graph-level branches"]:::l2
|
||||
|
||||
repo --> manifest
|
||||
repo --> nodes
|
||||
repo --> edges
|
||||
repo --> cgraph
|
||||
repo --> recovery
|
||||
repo --> refs
|
||||
graph --> manifest
|
||||
graph --> nodes
|
||||
graph --> edges
|
||||
graph --> cgraph
|
||||
graph --> recovery
|
||||
graph --> refs
|
||||
|
||||
subgraph dataset[Inside each Lance dataset — L1]
|
||||
ds_v["_versions/{n}.manifest<br/>per-dataset versions"]:::l1
|
||||
|
|
@ -88,10 +88,10 @@ flowchart TB
|
|||
|
||||
**What's where:**
|
||||
|
||||
- **Repo root** is one directory (or S3 prefix). Everything below is part of one OmniGraph repo.
|
||||
- **Graph root** is one directory (or S3 prefix). Everything below is part of one OmniGraph graph.
|
||||
- **`__manifest/`** is a Lance dataset whose rows describe which sub-table version is published at which graph-branch. Reading a snapshot starts here.
|
||||
- **`nodes/`** and **`edges/`** are sibling directories holding one Lance dataset per declared type. Names are `fnv1a64-hex` of the type name to keep paths fixed-length and case-safe.
|
||||
- **`_graph_commits.lance`** is an L2 dataset that records the graph-level commit DAG, with a paired `_graph_commit_actors.lance` for the actor map. (Pre-v0.4.0 repos also have inert `_graph_runs.lance` / `_graph_run_actors.lance` from the removed Run state machine; MR-770 sweeps these in production.)
|
||||
- **`_graph_commits.lance`** is an L2 dataset that records the graph-level commit DAG, with a paired `_graph_commit_actors.lance` for the actor map. (Pre-v0.4.0 graphs also have inert `_graph_runs.lance` / `_graph_run_actors.lance` from the removed Run state machine; MR-770 sweeps these in production.)
|
||||
- **`_graph_commit_recoveries.lance`** — one row per recovery sweep action. Joined to `_graph_commits.lance` by `graph_commit_id`; the linked commit row carries `actor_id=omnigraph:recovery`. Operators correlate recoveries with the original mutations they rolled forward / back via this join. See `crates/omnigraph/src/db/recovery_audit.rs`.
|
||||
- **`__recovery/{ulid}.json`** — transient sidecar files written by the four migrated writers (`MutationStaging::finalize`, `schema_apply`, `branch_merge`, `ensure_indices`) before Phase B begins, deleted after Phase C succeeds. A sidecar persisting after process exit means the writer crashed in the Phase B → Phase C window; the next `Omnigraph::open` recovery sweep processes it. Steady-state directory is empty. See `crates/omnigraph/src/db/manifest/recovery.rs`.
|
||||
- **`_refs/branches/{name}.json`** is graph-level branch metadata — pointers from a branch name to the manifest version it heads.
|
||||
|
|
|
|||
|
|
@ -48,7 +48,7 @@ query register_employee_with_team($name: String, $age: I32, $team: String) {
|
|||
|
||||
```bash
|
||||
omnigraph change --query ./mutations.gq --name register_employee_with_team \
|
||||
--params '{"name":"Alice","age":30,"team":"Acme"}' ./repo.omni
|
||||
--params '{"name":"Alice","age":30,"team":"Acme"}' ./graph.omni
|
||||
```
|
||||
|
||||
If the second statement fails (e.g. `Acme` doesn't exist), the publisher never publishes; `Alice` is not in the database. Atomic.
|
||||
|
|
@ -57,10 +57,10 @@ If the second statement fails (e.g. `Acme` doesn't exist), the publisher never p
|
|||
|
||||
```bash
|
||||
# Query 1
|
||||
omnigraph change --query ./mutations.gq --name register_employee --params '{"name":"Alice","age":30}' ./repo.omni
|
||||
omnigraph change --query ./mutations.gq --name register_employee --params '{"name":"Alice","age":30}' ./graph.omni
|
||||
|
||||
# Query 2 — runs after Query 1 has already published
|
||||
omnigraph change --query ./mutations.gq --name link_to_team --params '{"name":"Alice","team":"Acme"}' ./repo.omni
|
||||
omnigraph change --query ./mutations.gq --name link_to_team --params '{"name":"Alice","team":"Acme"}' ./graph.omni
|
||||
```
|
||||
|
||||
These are **two publishes** on `main`. If Query 2 fails, Query 1's effects are already visible. There is no `ROLLBACK` for Query 1.
|
||||
|
|
@ -75,32 +75,32 @@ The pattern when you need to run multiple queries — possibly across multiple c
|
|||
|
||||
```bash
|
||||
# Fork a working branch from main.
|
||||
omnigraph branch create --from main onboarding/2026-04-25 ./repo.omni
|
||||
omnigraph branch create --from main onboarding/2026-04-25 ./graph.omni
|
||||
|
||||
# Run any number of mutations on the branch — each one is its own publish on the branch.
|
||||
# Concurrent reads of `main` are unaffected.
|
||||
omnigraph change --branch onboarding/2026-04-25 \
|
||||
--query ./mutations.gq --name register_employee \
|
||||
--params '{"name":"Alice","age":30}' ./repo.omni
|
||||
--params '{"name":"Alice","age":30}' ./graph.omni
|
||||
|
||||
omnigraph change --branch onboarding/2026-04-25 \
|
||||
--query ./mutations.gq --name register_employee \
|
||||
--params '{"name":"Bob","age":25}' ./repo.omni
|
||||
--params '{"name":"Bob","age":25}' ./graph.omni
|
||||
|
||||
omnigraph change --branch onboarding/2026-04-25 \
|
||||
--query ./mutations.gq --name link_to_team \
|
||||
--params '{"name":"Alice","team":"Acme"}' ./repo.omni
|
||||
--params '{"name":"Alice","team":"Acme"}' ./graph.omni
|
||||
|
||||
# Inspect the branch — read queries work just like on main.
|
||||
omnigraph read --branch onboarding/2026-04-25 \
|
||||
--query ./queries.gq --name list_employees ./repo.omni
|
||||
--query ./queries.gq --name list_employees ./graph.omni
|
||||
|
||||
# Happy with what's on the branch? Merge it. This is one atomic publish:
|
||||
# `main` flips to include every commit on the branch.
|
||||
omnigraph branch merge onboarding/2026-04-25 --into main ./repo.omni
|
||||
omnigraph branch merge onboarding/2026-04-25 --into main ./graph.omni
|
||||
|
||||
# OR: not happy? Throw it away. `main` is untouched.
|
||||
# omnigraph branch delete onboarding/2026-04-25 ./repo.omni
|
||||
# omnigraph branch delete onboarding/2026-04-25 ./graph.omni
|
||||
```
|
||||
|
||||
Properties:
|
||||
|
|
@ -115,16 +115,16 @@ Two agents writing to the same graph independently:
|
|||
|
||||
```bash
|
||||
# Agent A
|
||||
omnigraph branch create --from main agent-a/work ./repo.omni
|
||||
omnigraph change --branch agent-a/work … ./repo.omni
|
||||
omnigraph branch create --from main agent-a/work ./graph.omni
|
||||
omnigraph change --branch agent-a/work … ./graph.omni
|
||||
# … many mutations …
|
||||
omnigraph branch merge agent-a/work --into main ./repo.omni
|
||||
omnigraph branch merge agent-a/work --into main ./graph.omni
|
||||
|
||||
# Agent B (running concurrently)
|
||||
omnigraph branch create --from main agent-b/work ./repo.omni
|
||||
omnigraph change --branch agent-b/work … ./repo.omni
|
||||
omnigraph branch create --from main agent-b/work ./graph.omni
|
||||
omnigraph change --branch agent-b/work … ./graph.omni
|
||||
# … many mutations …
|
||||
omnigraph branch merge agent-b/work --into main ./repo.omni
|
||||
omnigraph branch merge agent-b/work --into main ./graph.omni
|
||||
```
|
||||
|
||||
Each agent sees a consistent snapshot of `main` at the time it forked. The first merge to `main` lands as a fast-forward (or a no-op if no concurrent change). The second merge runs three-way: rows touched by both branches surface as `MergeConflict`s for the caller to resolve.
|
||||
|
|
@ -138,7 +138,7 @@ This is the workflow MR-797 / agentic loops are designed around: **branches are
|
|||
| Single query fails mid-flight | Publisher never publishes; target unchanged | Read the error, decide whether to retry |
|
||||
| Concurrent writers race the same `(table, branch)` | Publisher CAS rejects the loser with `ManifestConflictDetails::ExpectedVersionMismatch` | Refresh handle, retry the query |
|
||||
| Branch with N successful mutations, then merge fails (three-way conflict) | Each individual mutation already committed on the branch; merge surfaces `MergeConflicts` | Inspect, decide whether to keep working on the branch, abandon it (`branch_delete`), or resolve and re-merge |
|
||||
| Process crashes mid-branch-workflow | Each completed mutation on the branch is durable | Re-open the repo, continue where you left off |
|
||||
| Process crashes mid-branch-workflow | Each completed mutation on the branch is durable | Re-open the graph, continue where you left off |
|
||||
|
||||
## When to use what
|
||||
|
||||
|
|
@ -156,7 +156,7 @@ This is the workflow MR-797 / agentic loops are designed around: **branches are
|
|||
|
||||
- **Cross-query atomicity on `main` without a branch.** If you don't want to fork a branch, multiple queries on `main` publish independently. There is no implicit transaction.
|
||||
- **Long-running interactive transactions.** No `BEGIN` over a connection. Branches are the durable equivalent.
|
||||
- **Cross-graph (cross-repo) transactions.** Each repo is its own atomicity domain.
|
||||
- **Cross-graph transactions.** Each graph is its own atomicity domain.
|
||||
- **"Pessimistic" locks** that serialize writers before they reach the storage layer. Snapshot-MVCC + publisher CAS handles concurrency optimistically; the loser retries.
|
||||
|
||||
## See also
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue