mirror of
https://github.com/trustgraph-ai/trustgraph.git
synced 2026-07-27 14:11:02 +02:00
1740e315f4
Adds batching to structured data row imports. Previously each row was sent as a separate WebSocket message, causing one embeddings service call per row. The bulk import clients now accumulate rows into batches (default 40) before sending, and the --batch-size CLI argument is wired through from tg-load-structured-data to the import call. Splits the schema "indexes" field into two separate configurations: - "query-indexes": fields for Cassandra exact-match retrieval - "vector-indexes": fields for vector embedding and semantic search This avoids wasting compute embedding primary key integers that only need exact lookup, while allowing each field to independently opt in to either or both retrieval patterns. Adds row_id (the primary key value) as a clustering column in the Cassandra rows table so multiple rows sharing the same index value no longer silently overwrite each other. Tests updated for new Cassandra schema. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| conftest.py | ||
| test_document_embeddings_processor.py | ||
| test_embedding_logic.py | ||
| test_embedding_utils.py | ||
| test_embeddings_client.py | ||
| test_embeddings_service_contract.py | ||
| test_embeddings_service_request.py | ||
| test_fastembed_dynamic_model.py | ||
| test_graph_embeddings_processor.py | ||
| test_huggingface_dynamic_model.py | ||
| test_ollama_dynamic_model.py | ||
| test_row_embeddings_processor.py | ||