trustgraph/trustgraph-flow/trustgraph
cybermaggedon 1740e315f4
feat: batch row imports and split index configuration (#1055)
Adds batching to structured data row imports. Previously each row was
sent as a separate WebSocket message, causing one embeddings service
call per row. The bulk import clients now accumulate rows into batches
(default 40) before sending, and the --batch-size CLI argument is
wired through from tg-load-structured-data to the import call.

Splits the schema "indexes" field into two separate configurations:
- "query-indexes": fields for Cassandra exact-match retrieval
- "vector-indexes": fields for vector embedding and semantic search

This avoids wasting compute embedding primary key integers that only
need exact lookup, while allowing each field to independently opt in
to either or both retrieval patterns.

Adds row_id (the primary key value) as a clustering column in the
Cassandra rows table so multiple rows sharing the same index value
no longer silently overwrite each other.

Tests updated for new Cassandra schema.
2026-07-22 10:20:06 +01:00
..
agent Per-flow librarian clients and per-workspace response queues (#865) 2026-05-06 12:01:01 +01:00
bootstrap feat: make bootstrapper initialiser timeouts configurable (#999) 2026-06-30 09:37:22 +01:00
chunking Per-flow librarian clients and per-workspace response queues (#865) 2026-05-06 12:01:01 +01:00
config/service fix: wire replication params through YAML/params path for Cassandra and Qdrant (#976) 2026-06-04 12:36:36 +01:00
cores fix: knowledge core round-trip for workspace-scoped librarian and all term types (#1049) 2026-07-16 00:31:47 +01:00
decoding fix: reject invalid PDF decoder input (#977) 2026-06-09 16:37:39 +01:00
direct Sanitize workspace name used as Cassandra keyspace (#1043) (#1053) 2026-07-20 10:34:55 +01:00
embeddings feat: batch row imports and split index configuration (#1055) 2026-07-22 10:20:06 +01:00
external Implement logging strategy (#444) 2025-07-30 23:18:38 +01:00
extract fix: preserve non-ASCII entity names in ontology URI normalization (#1036) 2026-07-09 00:05:35 +01:00
flow Per-flow librarian clients and per-workspace response queues (#865) 2026-05-06 12:01:01 +01:00
gateway feat: pluggable image-to-text service with OpenAI vision backend (#1038) 2026-07-12 12:47:04 +01:00
iam feat: pluggable image-to-text service with OpenAI vision backend (#1038) 2026-07-12 12:47:04 +01:00
librarian fix: wire replication params through YAML/params path for Cassandra and Qdrant (#976) 2026-06-04 12:36:36 +01:00
metering feat: workspace-based multi-tenancy, replacing user as tenancy axis (#840) 2026-04-21 23:23:01 +01:00
model feat: pluggable image-to-text service with OpenAI vision backend (#1038) 2026-07-12 12:47:04 +01:00
processing Fix/startup failure (#445) 2025-07-30 23:42:11 +01:00
prompt feat: LLM-native structured output via JSON schema enforcement (#1037) 2026-07-10 15:28:56 +01:00
query feat: batch row imports and split index configuration (#1055) 2026-07-22 10:20:06 +01:00
reranker feat: replace LLM edge scoring with cross-encoder reranker in GraphRAG (#1005) 2026-06-30 14:36:37 +01:00
retrieval feat: structured source document references in graph-rag responses (#1035) 2026-07-08 23:59:56 +01:00
rev_gateway Update rev-gateway for IAM integration (#940) 2026-05-19 21:45:43 +01:00
storage feat: batch row imports and split index configuration (#1055) 2026-07-22 10:20:06 +01:00
tables fix: knowledge core round-trip for workspace-scoped librarian and all term types (#1049) 2026-07-16 00:31:47 +01:00
template feat: LLM-native structured output via JSON schema enforcement (#1037) 2026-07-10 15:28:56 +01:00
tool_service feat: workspace-based multi-tenancy, replacing user as tenancy axis (#840) 2026-04-21 23:23:01 +01:00
__init__.py Feature/pkgsplit (#83) 2024-09-30 19:36:09 +01:00