mirror of
https://github.com/ModernRelay/omnigraph.git
synced 2026-07-21 03:41:03 +02:00
fix(engine): prefilter(true) for filtered vector/FTS search
Lance's scanner defaults to prefilter=false: a filter riding the same scanner as nearest()/bm25() is applied AFTER the ANN/FTS top-k, so `limit k` meant top-k of the whole table and a selective predicate silently starved results (the deny-list's silent-partial-result shape; measured by the nearest-prefilter bench scenario: 20k rows, s=0.05, k=10 -> 1000 matching rows exist, 0 returned). Set prefilter(true) whenever a structured filter is pushed to the scanner: one flag governs both the vector and FTS sources, plain scans ignore it, and it re-enables scalar-index acceleration for the predicate under nearest. The red test turns green: filtered nearest now returns the top-k of MATCHING rows. Docs state the filters-before-search contract explicitly (docs/user/search/index.md). Closes iss-nearest-postfilter-starves-results.
This commit is contained in:
parent
31134db440
commit
281525cf7a
2 changed files with 16 additions and 0 deletions
|
|
@ -1923,6 +1923,18 @@ async fn execute_node_scan(
|
||||||
// Apply the structured IR filter via Lance's Expr pushdown.
|
// Apply the structured IR filter via Lance's Expr pushdown.
|
||||||
if let Some(ref expr) = filter_expr {
|
if let Some(ref expr) = filter_expr {
|
||||||
scanner.filter_expr(expr.clone());
|
scanner.filter_expr(expr.clone());
|
||||||
|
// The filter must run BEFORE any ANN/FTS search on this
|
||||||
|
// scanner. Lance defaults to prefilter=false, which applies
|
||||||
|
// the filter to the search's top-k results — "you may get
|
||||||
|
// back fewer results than you ask for (or none at all)"
|
||||||
|
// (lance scanner.rs) — i.e. `limit k` would mean top-k of
|
||||||
|
// the whole table, silently starved by a selective filter.
|
||||||
|
// One flag governs both the vector and FTS sources, and it
|
||||||
|
// is unused by plain scans, so setting it whenever a filter
|
||||||
|
// is present is safe. Prefiltering also re-enables scalar-
|
||||||
|
// index acceleration for the predicate (Lance gates
|
||||||
|
// use_scalar_index on prefilter when a nearest is present).
|
||||||
|
scanner.prefilter(true);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Apply FTS queries from hoisted search filters (search/fuzzy/match_text in match clause)
|
// Apply FTS queries from hoisted search filters (search/fuzzy/match_text in match clause)
|
||||||
|
|
|
||||||
|
|
@ -20,6 +20,10 @@ rank).
|
||||||
- `nearest()` requires a `limit`. The query vector is resolved from the param map,
|
- `nearest()` requires a `limit`. The query vector is resolved from the param map,
|
||||||
or embedded from a text input at runtime via the configured
|
or embedded from a text input at runtime via the configured
|
||||||
[embedding client](embeddings.md).
|
[embedding client](embeddings.md).
|
||||||
|
- Match filters apply **before** the search: combining a `match` predicate with
|
||||||
|
`nearest()` (or `bm25()`) returns the top-`limit` of the *matching* rows —
|
||||||
|
never a post-filtered remainder of the global top-k. A selective filter
|
||||||
|
narrows the candidate set; it cannot starve the result count.
|
||||||
- Scores and ranks propagate as ordinary columns, so you can `return` a score and
|
- Scores and ranks propagate as ordinary columns, so you can `return` a score and
|
||||||
`order` by it.
|
`order` by it.
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue