feat: hybrid retrieval (BM25 + vector RRF fusion) for document-RAG (#875) (#1030)

Adds a sparse keyword retrieval path beside the existing vector path in
document-RAG, fused by weighted Reciprocal Rank Fusion on chunk_id, behind
--retrieval-mode (vector | keyword | hybrid, default vector).

The keyword index is a new pluggable service (KeywordIndexService /
KeywordIndexClientSpec); the first backend is SQLite FTS5, consuming Chunk
messages off the ingestion stream and answering BM25 queries from one
process, since the index is a single local file. Query text is sanitized
into per-term quoted phrases (raw text is not valid FTS5 syntax), which
also makes dotted clause numbers and error codes exact-match without a
trigram index. Indexes are scoped per (workspace, collection) and dropped
on collection deletion.

The keyword-index client spec is only registered when the sparse path is
enabled, so existing flow definitions without keyword-index queues are
untouched; with retrieval_mode=vector the retrieval path is unchanged. In
hybrid mode a keyword-path failure degrades to vector-only.
This commit is contained in:
Sunny Yang 2026-07-07 05:54:02 -06:00 committed by GitHub
parent e5206bddd0
commit 2bdc930b2a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
16 changed files with 1013 additions and 13 deletions

View file

@ -32,6 +32,14 @@ processors:
id: graph-embeddings-write
store_uri: http://localhost:6333
# Keyword (BM25) index: ingest-write and query in one processor, since
# the FTS5 index is a single local file.
- class: trustgraph.storage.kw_index.fts5.Processor
params:
<<: *defaults
id: kw-index
index_path: /tmp/tg-kw-index.db
- class: trustgraph.query.row_embeddings.qdrant.Processor
params:
<<: *defaults