omnigraph/docs/invariants.md

# Architectural Invariants & Policies

**Type:** Reference / standing document
**Status:** Living — updated as decisions accrue
**Audience:** anyone proposing, reviewing, or implementing a change to any part of OmniGraph

This document captures the invariants that hold across all of OmniGraph — storage, engine, server, schema, indexing, observability, and the OSS / Cloud product split. New RFCs, designs, and implementations are checked against this list.

These are not query-engine-specific. They apply to every layer.

> **Note on numbering.** §IV (Additivity / migration) and §VIII (OSS / Cloud kernel-product split) are referenced from §X but not yet drafted in this revision. Pending upstream fill-in.

## How to use

- **Writing an RFC or design proposal:** walk through the relevant sections and state how the proposal upholds each invariant — or why a documented exception is justified.
- **Reviewing a PR or design:** scan for invariants the change might violate. The deny-list (§IX) is the fastest first pass.
- **Debating a tradeoff:** invoke the relevant invariant and check whether the tradeoff respects it. Invariants here are load-bearing; breaking one is rare and requires explicit justification in the proposal.
- **Updating this document:** add to the deny-list freely. Removing or relaxing an invariant requires the same review process as any other architectural decision.

## I. Substrate boundaries — what we don't build

The most important question for any new component is "does the substrate already do this?" If yes, we don't.

1. **Lance is the storage substrate.** We do not build a competing storage format. We build above Lance via a trait. Where we want format-level behavior Lance doesn't have, we propose it upstream or work around it.
   *Check:* Does this proposal introduce a parallel storage format, custom on-disk pages, custom serialization?

2. **No own-format WAL or transaction manager.** Lance manifest MVCC plus (eventually) MemWAL is the durability story. We do not write durability code.
   *Check:* Does this proposal track its own write log, recovery state, or transaction journal?

3. **No own buffer pool.** Lance + `object_store` + Lance's scan-aware cache cover our needs.
   *Check:* Does this proposal cache Arrow batches or pages outside Lance's cache?

4. **The runtime substrate provides relational machinery.** Whether we choose DataFusion (current prior) or a custom executor, we do not rebuild joins, aggregations, parallelism, spill. We build only graph- or multi-modal-shaped operators on top.
   *Check:* Are we extending the substrate via traits, or reimplementing parts of it?

5. **Don't replace Lance's index lifecycle.** New fragments enter without index coverage; reads union indexed and scan paths via `fragment_bitmap`. Our reconciler observes the same primitive; we don't replicate it.
   *Check:* Does this proposal maintain its own version of "what's indexed and what isn't" parallel to Lance's?

## II. Layering — the seams that hold

6. **The IR is the contract between frontend and backend.** Frontends emit IR; planner / executor consume it. No frontend logic leaks downward; no executor concerns leak upward.
   *Check:* Does the proposal add to the IR, or to a layer? If to a layer, does it cross another layer's concern?

7. **Capabilities and statistics flow upward; data flows downward.** Lower layers expose what they can do (capabilities) and what they know (statistics). Upper layers consume both. Methods alone are insufficient — methods without capability advertisement force one-size-fits-all plans.
   *Check:* When adding a method to a layer trait, did we also expose the capability so the planner can reason about it?

8. **One trait boundary per layer.** Crossing a layer means going through its trait. Direct calls to lower-layer concrete types from upper layers are forbidden.
   *Check:* Does this code call `lance::Dataset` directly outside engine-storage? Call planner internals from the executor?

9. **No god modules.** Single-module concerns: storage, IR, planner, executor, frontend, reconciler, schema, policy. Each crate has a reference test suite that runs without the others.
   *Check:* Does this PR add a concern to a crate that already owns a different one?

## III. Distributability — kernel stays remote-friendly

These are technical constraints, independent of whether we ship a distributed product. They preserve the architectural seam.

10. Storage trait is `Send + Sync`. No in-process-only assumptions in `Dataset` impls.
11. `Dataset` impls accept remote descriptors (URI, snapshot ref, fragment ID) without requiring an open in-process Lance handle.
12. **IR is location-neutral.** No IR operator embeds an assumption about where data lives.
13. Cost model accepts a network-cost term as a future additive component. No place hard-codes "all cost is local I/O."
14. Reconciler trait admits alternate implementations. In-process tokio is the OSS default; a separable worker fleet is the distributed shape.

## IV. *(Additivity / migration — placeholder, not yet drafted)*

## V. Honesty — cost, observability, calibration

20. **Estimate-vs-actual logging on every estimator.** Cost models drift; calibration is a continuous process, not a one-off.
21. **Coverage and lag are first-class metrics.** Index coverage, reconciler lag, cost-model accuracy — surfaced through the storage trait's `capabilities()` and a unified observability API.
22. **Honest failure modes.** Cost-model misses degrade gracefully (spill, partial-result, bounded abort). No silent OOM.
23. **Per-query budgets propagate through operators.** Memory cap, wall-clock timeout, max-rows-scanned, max-fragments-scanned. Operators respect them; budgets exposed via explain.
24. **Plans are explainable.** Every executed query can be inspected as IR + physical plan + cost annotations. No "you'd have to read the source to know what this does."

## VI. Patterns — use the unified mechanism

When two features look similar, they probably share a mechanism. Use it.

25. **Reconciler pattern for derivable state.** Index coverage, statistics, anything derivable from manifest state — reconciled, not job-queued.
26. **Polymorphism via Union, not per-feature lowering.** Interfaces / wildcards / alternation on nodes and edges share one IR (`Polymorphism<T>`) and one lowering (Union of per-type concrete plans).
27. **Mutations wrap read subplans.** Insert / Update / Delete / Merge are operators that consume read-shaped subplans. Same planner, same cost model, same storage trait.
28. **SIP for cross-operator selectivity propagation.** Producers publish ID bitmaps; downstream scans consume them through structured pushdown. Don't ad-hoc IN-lists.
29. **Factorize multi-hop, flatten only at projection.** Lists carry multiplicity through intermediate operators. Flatten is inserted by the planner where required, not eagerly.
30. **Stable row IDs as dense graph IDs.** Don't maintain parallel string→u32 maps. Lance's stable row IDs are the substrate's identity layer; we use them directly.
31. **Rank and score are columns.** Retrieval operators emit `_score`, `_rank`. Fusion operators consume rank-bearing batches. Don't discard rank in pipeline-twice merges.
32. **Policy as predicates.** Authorization decisions are filter expressions injected into the planner, not enforcement at the API boundary.

## VII. Quality gates — every change passes

33. **Tests at every boundary.** `MemStorage` for engine tests; planner-only tests; executor-only tests with a stub storage. No layer tested only via end-to-end.
34. **Reference implementation per trait.** Every trait has a primary impl (Lance for storage) and at least a test impl.
35. **Documented capability surface.** New capabilities are documented with what they advertise, who consumes them, how the planner uses them.
36. **Benchmark before optimization.** New optimizations land with a benchmark that motivates them; if the motivating workload doesn't exist, the feature waits.

## VIII. *(OSS / Cloud kernel-product split — placeholder, not yet drafted)*

## IX. Anti-patterns — deny list

If a proposal fits one of these, the burden is on the proposer to justify why this case is the exception.

- Synchronous-inline index updates for indexes expensive to build (vector ANN, FTS). Reconciler pattern instead.
- Custom WAL / transaction manager / buffer pool. Lance owns these.
- Job queue for state derivable from manifest. Reconciler pattern instead.
- Per-feature lowering for shapes that share a structure (interfaces, wildcards, alternation). Use one mechanism.
- Eager materialization of cross-products in multi-hop. Factorize; flatten only when needed.
- Ad-hoc IN-list filtering when SIP fits.
- String-flattened SQL filter generation when structured pushdown is available.
- In-process-only `Dataset` impls. `Send + Sync`, remote descriptors.
- Cost-blind plan choice. Lowering-order execution is not a planner.
- Hidden statistics. If a metric matters for plan choice, it must be exposed through the trait surface.
- Side-channels for query semantics. Search modes, mutations, polymorphism — all first-class IR concepts.
- Discarding rank in retrieval. Score and rank propagate as columns.
- State that drifts from the manifest. Derive from observable state.
- Cloud-only correctness fixes. Correctness is always OSS.
- Forking the codebase for Cloud. Trait-extension only.
- Hand-rolling something Lance already does. Check the spec first.
- Mutating in place state that should be immutable (Lance fragments, index segments). New segments instead.
- Silent failures. OOM, timeout, partial result — all surfaced and bounded.

## X. Review checklist (use against any non-trivial change)

Print this when reviewing an RFC or PR. Each line is **yes / no / N/A**.

- Does it respect the substrate? (§I)
- Does it cross only one trait boundary per layer? (§II)
- Are capabilities and stats exposed for any new behavior? (§II.7)
- Storage trait stays `Send + Sync` and remote-friendly? (§III)
- Additive, not rewrite? Feature-flagged where behavior changes? (§IV)
- Any new estimator has estimate-vs-actual logging? (§V.20)
- Coverage / lag / budget metrics surfaced? (§V.21–23)
- Failure modes graceful, bounded, observable? (§V.22)
- Reuses an existing pattern (reconciler, Union, mutation-wrap-read, SIP, factorize) where applicable? (§VI)
- Tests at every boundary, not just end-to-end? (§VII.33)
- Reference impl + test impl for any new trait? (§VII.34)
- If commercial-relevant: kernel/product split preserved? (§VIII)
- None of the deny-list patterns apply? (§IX)

## XI. Living document policy

This document is updated when:

- A new architectural decision establishes a new invariant — add it.
- An existing invariant is challenged and either reaffirmed (with the case sharpened) or revised (with explicit migration of any affected code).
- A new anti-pattern surfaces in review and deserves a place on the deny-list — add it.

Updates require the same review process as code. Adding to the deny-list (§IX) is cheap; removing or relaxing an invariant (§I–VIII) requires explicit justification in the proposal.

When an invariant is contested in the moment, the resolution path is: (a) state the case in the relevant RFC or PR; (b) link it from this document; (c) update this document if the resolution changes the rule.

## XII. Source / origin

These invariants were extracted from the architectural decisions in:

- **MR-737** — Query Engine v2 RFC (the kernel scope and seams)
- **MR-738** — OSS / Cloud strategy (the commercial overlay)
- The schema migration program (**MR-694** family — additive evolution, safety tiers)
- The policy program (**MR-722 / MR-725** — predicate pushdown)
- The reconciler / index-lifecycle work (**MR-737 §5.16, MR-688, MR-679, MR-680**)
- The factorization and SIP work (**MR-737 §5.2, §5.3** — Kuzu / Ladybug inspiration)
- The polymorphic-bindings framing (**MR-737 §5.13** — one mechanism for eight cells)

General precedent: Lance + LanceDB Enterprise architecture; ClickHouse merge subsystem; Kubernetes controllers; Postgres autovacuum; the FDAL stack (Flight + DataFusion + Arrow + Lance).

Adding a new invariant here means we've learned something — either from a hard call we made and want to preserve, or from a mistake we don't want to repeat. Both are worth recording.