mirror of
https://github.com/trustgraph-ai/trustgraph.git
synced 2026-07-09 21:32:10 +02:00
2bdc930b2a
Adds a sparse keyword retrieval path beside the existing vector path in document-RAG, fused by weighted Reciprocal Rank Fusion on chunk_id, behind --retrieval-mode (vector | keyword | hybrid, default vector). The keyword index is a new pluggable service (KeywordIndexService / KeywordIndexClientSpec); the first backend is SQLite FTS5, consuming Chunk messages off the ingestion stream and answering BM25 queries from one process, since the index is a single local file. Query text is sanitized into per-term quoted phrases (raw text is not valid FTS5 syntax), which also makes dotted clause numbers and error codes exact-match without a trigram index. Indexes are scoped per (workspace, collection) and dropped on collection deletion. The keyword-index client spec is only registered when the sparse path is enabled, so existing flow definitions without keyword-index queues are untouched; with retrieval_mode=vector the retrieval path is unchanged. In hybrid mode a keyword-path failure degrades to vector-only.
53 lines
1.5 KiB
YAML
53 lines
1.5 KiB
YAML
# Embeddings store. All Qdrant-backed vector query/write processors.
|
|
# One process owns the Qdrant driver pool for the whole group.
|
|
|
|
_defaults: &defaults
|
|
pubsub_backend: rabbitmq
|
|
rabbitmq_host: localhost
|
|
log_level: INFO
|
|
|
|
processors:
|
|
|
|
- class: trustgraph.query.doc_embeddings.qdrant.Processor
|
|
params:
|
|
<<: *defaults
|
|
id: doc-embeddings-query
|
|
store_uri: http://localhost:6333
|
|
|
|
- class: trustgraph.storage.doc_embeddings.qdrant.Processor
|
|
params:
|
|
<<: *defaults
|
|
id: doc-embeddings-write
|
|
store_uri: http://localhost:6333
|
|
|
|
- class: trustgraph.query.graph_embeddings.qdrant.Processor
|
|
params:
|
|
<<: *defaults
|
|
id: graph-embeddings-query
|
|
store_uri: http://localhost:6333
|
|
|
|
- class: trustgraph.storage.graph_embeddings.qdrant.Processor
|
|
params:
|
|
<<: *defaults
|
|
id: graph-embeddings-write
|
|
store_uri: http://localhost:6333
|
|
|
|
# Keyword (BM25) index: ingest-write and query in one processor, since
|
|
# the FTS5 index is a single local file.
|
|
- class: trustgraph.storage.kw_index.fts5.Processor
|
|
params:
|
|
<<: *defaults
|
|
id: kw-index
|
|
index_path: /tmp/tg-kw-index.db
|
|
|
|
- class: trustgraph.query.row_embeddings.qdrant.Processor
|
|
params:
|
|
<<: *defaults
|
|
id: row-embeddings-query
|
|
store_uri: http://localhost:6333
|
|
|
|
- class: trustgraph.storage.row_embeddings.qdrant.Processor
|
|
params:
|
|
<<: *defaults
|
|
id: row-embeddings-write
|
|
store_uri: http://localhost:6333
|