docs: add datafusion-future-improvements.md to dev docs

Captures the post-PR #111 (Lance 4→6) + PR #113 (structured Expr
pushdown) DataFusion state in one place, so future maintainers don't
have to re-derive what's done, what's free, and what's still on the
table from chat history.

Structure:
- Direct touchpoints (only 2 — narrow surface)
- Shipped: PR-by-PR delta of what's landed
- Passive wins active on DF 53 (PR-linked, with where-it-bites-us
  notes)
- Still on the table, ranked by tier:
  - T1: structural, unblocked today (hydrate_nodes Expr pushdown)
  - T2: gated on Lance v7 (delete Expr via MR-A / issue #112)
  - T3: future-shape unlocks (extension planner, expression
    placement, etc.)
  - T4: won't reach us without major changes (custom ExecutionPlan
    territory)
- Upstream cadence note (Lance dictates the DF version)
- Maintenance section

Linked from docs/dev/index.md so the check-agents-md CI guard
passes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
aaltshuler 2026-05-23 12:52:21 +01:00
parent cb80fa40f1
commit 8393fd7946
2 changed files with 120 additions and 0 deletions

View file

@ -23,6 +23,7 @@ constraints. User-facing behavior should still be documented through
| On-disk layout, manifest schema, URI behavior | [storage.md](../user/storage.md) |
| Direct-publish writes, D2, staged writes, recovery sidecars | [runs.md](runs.md) |
| Query execution, mutation execution, loader flow | [execution.md](execution.md) |
| DataFusion: current state, passive wins, future improvements | [datafusion-future-improvements.md](datafusion-future-improvements.md) |
| Index lifecycle and graph topology indexes | [indexes.md](../user/indexes.md) |
| Branch and commit internals | [branches-commits.md](../user/branches-commits.md) |
| Three-way merge implementation and conflicts | [merge.md](merge.md) |