omnigraph/.context/experiments/bitmap-pushdown.md

Experiment 1.5 — Extending DataFusion dynamic-filter-pushdown to bitmap shape (code-dive)

Ticket: MR-925 §1.5 (validates MR-737 §5.3 "Sideways Information Passing (SIP) — extends DataFusion's dynamic filter pushdown" + §5.6 "Capability-bearing storage trait", with implications for §5.7 cost-model).

§-numbering note: the original MR-925 cross-reference said "§5.3 / §5.10". On a full re-read of MR-737: §5.3 is the SIP design and is the correct primary § for this experiment; §5.10 in current MR-737 is "First-class scores and rank fusion" (unrelated). The capability seam that the bitmap pushdown traverses on the storage side is §5.6 (scan_by_key_set capability), so this writeup also produces deltas for §5.6. Type: Code-dive only (no prototype crate). Substrate pin: DataFusion 52.5. Date: 2026-05-12.

---

Question

The dynamic-filter-pushdown (DFP) feature in DataFusion 52.5 ships with three pushdown strategies for hash-join build sides:

• InList(ArrayRef) — for small build sides (< 128 MB).
• HashTable(Arc<dyn JoinHashMapType>) — for large build sides.
• Empty — no rows, do not push.

Can a third-party operator (e.g. our NeighborExpand from §1.3, or the broader graph-engine BackJoin and NeighborSetIntersect from MR-737 §5.3) extend the same machinery to push a roaring-bitmap-shaped filter through DF's dynamic filter framework — without forking DF?

TL;DR

Yes, completely supported on DataFusion 52.5 as written. The extension footprint is roughly 200 LoC: implement a custom PhysicalExpr (e.g. BitmapMembershipExpr) and feed it to the existing public DynamicFilterPhysicalExpr::update(...) API. No fork, no pub(crate) work-around.

Findings

F1. DataFusion's DFP is expression-shaped, not enum-shaped.

The PushdownStrategy enum (InList | HashTable | Empty) is internal to joins::hash_join::shared_bounds — it is the HashJoinExec's own internal switch for selecting which physical-expr to construct for its dynamic filter. The framework itself does not care:

// datafusion-physical-expr-52.5.0/src/expressions/dynamic_filters.rs
pub fn update(&self, new_expr: Arc<dyn PhysicalExpr>) -> Result<()>

The update method takes any Arc<dyn PhysicalExpr>. So an operator outside the hash-join module is free to ship its own pushdown strategy (e.g. BitmapMembership) and pass it to update.

F2. Three things must hold for a custom dynamic-filter expr to work.

From reading dynamic_filters.rs:

1. Stable children at construction time. DynamicFilterPhysicalExpr::new(children, initial_expr) binds the column-leaves at construction. Subsequent updates may only swap the expression, not introduce new column references. For BitmapMembershipExpr { column: Column::new("id"), bitmap_bytes: ... }, the only child is column — stable.

2. Self-contained evaluate. The custom PhysicalExpr must implement fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> to return a BooleanArray of the same length as the input batch. For a roaring bitmap this is: deserialize once at first call, cache in OnceCell, then per-batch for i in 0..n: out[i] = bitmap.contains(col[i]).

3. Be a dyn PhysicalExpr — implement Debug, Display, data_type, nullable, evaluate, children, with_new_children, dyn_hash, dyn_eq. Standard boilerplate, mirroring InListExpr.

F3. Two pushdown paths exist; only one needs work for graph operators.

DataFusion 52.5 has two filter-pushdown phases (see ExecutionPlan::gather_filters_for_pushdown and ExecutionPlan::handle_child_pushdown_result in execution_plan.rs):

• Static pushdown (planning time): filters are pushed from FilterExec → HashJoinExec → DataSourceExec during the EnforceFilterPushdown physical optimizer rule.
• Dynamic pushdown (execution time): a DynamicFilterPhysicalExpr placeholder is left in the plan at planning time; at runtime, the producer operator calls update(new_expr) once its data is available (e.g. once the hash-join build side is materialized).

For graph operators that produce SIPs (NeighborExpand build phase, SemiJoin build phase, etc.), the dynamic path is the natural one. The producer pattern is:

// At plan-construction time (in our ExtensionPlanner):
let placeholder = lit(true);
let dynamic_filter = Arc::new(DynamicFilterPhysicalExpr::new(
    vec![Arc::new(Column::new("dst_id", 2))], // probe-side column refs
    placeholder,
));
// Wire dynamic_filter into both the probe-side scan (as a filter)
// and store an Arc<DynamicFilterPhysicalExpr> on our build-side operator.

// At execute time, once our build side completes:
let bitmap = build_roaring_from(build_ids)?;
let bitmap_bytes = serialize_to_vec(bitmap);
let pushdown_expr = Arc::new(BitmapMembershipExpr {
    column: probe_side_column_ref,
    bitmap_bytes,
});
self.dynamic_filter.update(pushdown_expr)?;

F4. The scan-side has two interception points.

A custom dynamic filter ends up at the scan. Two paths exist for the scan to consume it efficiently:

Path A. Generic predicate evaluation (works today, no DF fork).

The BitmapMembershipExpr::evaluate(batch) is called per batch. For each batch row, bitmap.contains(row.value) is invoked. The roaring crate's contains is O(log n) within a fragment-localized container and was measured at <0.1 µs per call in §1.4. For a 1024-row batch, this is ~100 µs of CPU, which is amortized against the I/O for that batch. This is enough for §5.6 as written.

Path B. Lance scan-level row-id mask (faster, needs Lance integration).

Lance's Scanner supports a RowIdMask that is applied at the scan level before any predicate evaluation. If our BitmapMembershipExpr targets a Lance row-ID column, we could extract the bitmap during the scan's handle_child_pushdown_result and convert it into a RowIdMask — completely bypassing the per-row predicate. This is the same trick Lance's full-text search uses today (see Lance's scalar.rs apply_full_text_search_index).

Path B requires changes to Lance's DataSourceExec or our wrapping adapter; Path A is zero-change to DataFusion or Lance.

F5. The static-pushdown phase passes-through unrecognized exprs cleanly.

FilterDescription::all_unsupported(parent_filters, &children) is the default for gather_filters_for_pushdown. Our custom BitmapMembershipExpr is just an unrecognized expr — it will not be pushed past operators that haven't opted into bitmap pushdown, and at the leaf DataSourceExec it falls back to per-batch evaluation (Path A above). No silent misbehavior, no crash, no need to teach DF about our expression shape.

F6. The framework does NOT support N pushdown sources to the same scan.

A DynamicFilterPhysicalExpr wraps one inner expression at a time. If two producers (e.g. Expand(a) and Expand(b)) both want to push bitmap filters onto the same probe scan, each calls update on its own dynamic filter; the scan must hold N separate dynamic filters and AND them at evaluation time. The plumbing for this (multiple Arc<DynamicFilterPhysicalExpr> on a scan) is standard BinaryExpr(AND) wrapping. No framework gap.

Concrete plan for §5.6 (RFC body delta)

The RFC §5.6 should specify:

1. Bitmap-shaped SIPs are propagated via the standard DynamicFilterPhysicalExpr API. No custom side-channel; reuse the framework. Producer calls update(new_expr); scan evaluates the resulting BooleanArray per batch.

2. A new public BitmapMembershipExpr lives in our graph crate (not in the DF tree). It is constructed with a Column child and an opaque Vec<u8> payload (roaring serialized bytes). Implements PhysicalExpr::evaluate by deserializing the bitmap once into a OnceCell<RoaringTreemap> and probing it per row.

3. Lance-aware scan adaptation is optional and incremental. Path A (per-batch evaluation) is the v1 implementation. Path B (scan-level RowIdMask) is a v2 optimization that requires a LanceDataSourceExec to special-case BitmapMembershipExpr in its handle_child_pushdown_result impl. The RFC should call out Path B as a follow-up, not a blocker.

4. N producers to one scan: AND-wrap. The scan holds N Arc<DynamicFilterPhysicalExpr>. At plan-construction time, the probe filter wires them all in via BinaryExpr::new(a, AND, BinaryExpr::new(b, AND, c)). No bespoke "multi-source" data structure.

What does NOT need a DataFusion fork

• Custom dynamic-filter expression shapes. Public via Arc<dyn PhysicalExpr> + update.
• Custom dynamic-filter producers. Public via DynamicFilterPhysicalExpr::new.
• Custom dynamic-filter consumers. All scans evaluate via the standard evaluate interface.
• Composition with existing DFP (InList/HashTable). Wrap with BinaryExpr(AND).

What WOULD need a DataFusion fork or upstream contribution

• Bitmap-aware FilterPushdownPolicy. If we want the static-pushdown pass to recognize BitmapMembershipExpr and route it specially (e.g. drop the InList variant when a bitmap is available), we'd need a FilterPushdownPolicy extension point that doesn't exist today. However, this is a planner optimization, not a correctness or capability issue. The plan still works without it.

• A typed BitmapPushdownStrategy in SharedBuildAccumulator. Only matters if we want graph-side BackJoins to share the HashJoinExec's build-side accumulator. We don't — graph operators have their own build phases.

Decision impact on MR-737 §5.6 and §5.7

§5.6 (SIP propagation) is achievable on DF 52.5 as written. The public DynamicFilterPhysicalExpr::update API is sufficient. No upstream contribution required for v1.

§5.7 (cross-operator filter sharing) is achievable as AND-wrapping of N dynamic filters on the same scan. No framework gap. The RFC should clarify that the scan accumulates filters via standard BinaryExpr composition, not via a bespoke multi-source channel.

Open Q3 ("can we share the SIP filter between operator stages?") — answered yes. Confirmed by reading datafusion-physical-expr-52.5.0/src/expressions/dynamic_filters.rs:227 (the update API) and datafusion-physical-plan-52.5.0/src/joins/hash_join/shared_bounds.rs:463 (the call site that proves the producer/consumer split works).

Caveats and follow-ups

• No prototype was built. Per the ticket, §1.5 is a code-dive only. The recommendation rests on reading DF source, not on a working end-to-end implementation. If RFC §5.6 lands with this plan, Phase 0 should include a smoke test that:

  1. Wires a BitmapMembershipExpr into a DynamicFilterPhysicalExpr.
  2. Runs a hash join with the bitmap as the dynamic filter.
  3. Compares row-output and timing against an InListExpr-shaped baseline. Estimated work: 1 day, no DF fork.

• Lance scan-level RowIdMask support is the right v2 follow-up — but is gated on the same plugin-registry blocker discussed in §1.2 for custom index types. The RowIdMask path uses a different mechanism (it's not a scalar index), so it may not be blocked the same way. Worth a quick code-dive against lance/src/index/scalar.rs and lance/src/dataset/scanner.rs to confirm before committing.

• DF 52.5 → 52.6 may rework parts of DFP. The PR thread for SharedBuildAccumulator shows active churn; pin to 52.5.x for now and re-validate when bumping to a 52.6+ minor.