mirror of https://github.com/ModernRelay/omnigraph.git synced 2026-06-09 01:35:18 +02:00

aaltshuler 8393fd7946 docs: add datafusion-future-improvements.md to dev docs

Captures the post-PR #111 (Lance 4→6) + PR #113 (structured Expr
pushdown) DataFusion state in one place, so future maintainers don't
have to re-derive what's done, what's free, and what's still on the
table from chat history.

Structure:
- Direct touchpoints (only 2 — narrow surface)
- Shipped: PR-by-PR delta of what's landed
- Passive wins active on DF 53 (PR-linked, with where-it-bites-us
  notes)
- Still on the table, ranked by tier:
  - T1: structural, unblocked today (hydrate_nodes Expr pushdown)
  - T2: gated on Lance v7 (delete Expr via MR-A / issue #112)
  - T3: future-shape unlocks (extension planner, expression
    placement, etc.)
  - T4: won't reach us without major changes (custom ExecutionPlan
    territory)
- Upstream cadence note (Lance dictates the DF version)
- Maintenance section

Linked from docs/dev/index.md so the check-agents-md CI guard
passes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-23 12:52:21 +01:00

3 KiB

Raw Blame History

Developer Docs

Audience: contributors, maintainers, and coding agents

This is the contributor-facing entry point. These docs explain architecture, invariants, implementation contracts, test ownership, and upstream Lance constraints. User-facing behavior should still be documented through docs/user/index.md and the relevant public reference docs.

Required For Every Non-Trivial Change

Need	Read
Architectural rules, known gaps, deny-list	invariants.md
Upstream Lance source-of-truth index	lance.md
Existing test coverage and test placement	testing.md

Architecture And Storage

Area	Read
System structure, L1/L2 framing, component diagrams	architecture.md
On-disk layout, manifest schema, URI behavior	storage.md
Direct-publish writes, D2, staged writes, recovery sidecars	runs.md
Query execution, mutation execution, loader flow	execution.md
DataFusion: current state, passive wins, future improvements	datafusion-future-improvements.md
Index lifecycle and graph topology indexes	indexes.md
Branch and commit internals	branches-commits.md
Three-way merge implementation and conflicts	merge.md
Diff/change-feed implementation	changes.md
Branch protection policy	branch-protection.md
CODEOWNERS source of truth	codeowners.md

Language, Runtime, And Boundaries

Area	Read
Schema grammar, catalog, migration planner	schema-language.md
Query grammar, IR, lints, mutation restrictions	query-language.md
Embedding client and `@embed` integration	embeddings.md
Cedar policy surface and server gating	policy.md
Server auth, OpenAPI, endpoint handlers	server.md
Error taxonomy and serialization	errors.md
Constants and tunables	constants.md
Transaction model public contract	transactions.md

Project Operations

Area	Read
CI and release workflows	ci.md
Install and deployment packaging	install.md, deployment.md
Release history	releases/

Active Implementation Plans

Working documents for in-flight feature work. Removed when the work lands.

Area	Read
Schema-lint chassis v1 (MR-694) — `--allow-data-loss`, soft/hard drops	schema-lint-v1-plan.md

Boundary

Developer docs may mention implementation details, stale gaps, upstream Lance blockers, and review rules. User docs should not require that context unless the detail changes the public contract.

3 KiB Raw Blame History