trustgraph/docs/tech-specs
cybermaggedon 1740e315f4
feat: batch row imports and split index configuration (#1055)
Adds batching to structured data row imports. Previously each row was
sent as a separate WebSocket message, causing one embeddings service
call per row. The bulk import clients now accumulate rows into batches
(default 40) before sending, and the --batch-size CLI argument is
wired through from tg-load-structured-data to the import call.

Splits the schema "indexes" field into two separate configurations:
- "query-indexes": fields for Cassandra exact-match retrieval
- "vector-indexes": fields for vector embedding and semantic search

This avoids wasting compute embedding primary key integers that only
need exact lookup, while allowing each field to independently opt in
to either or both retrieval patterns.

Adds row_id (the primary key value) as a clustering column in the
Cassandra rows table so multiple rows sharing the same index value
no longer silently overwrite each other.

Tests updated for new Cassandra schema.
2026-07-22 10:20:06 +01:00
..
ar Structure the tech specs directory (#836) 2026-04-21 16:06:41 +01:00
es Structure the tech specs directory (#836) 2026-04-21 16:06:41 +01:00
he Structure the tech specs directory (#836) 2026-04-21 16:06:41 +01:00
hi Structure the tech specs directory (#836) 2026-04-21 16:06:41 +01:00
pt Structure the tech specs directory (#836) 2026-04-21 16:06:41 +01:00
ru Structure the tech specs directory (#836) 2026-04-21 16:06:41 +01:00
sw Structure the tech specs directory (#836) 2026-04-21 16:06:41 +01:00
tr Structure the tech specs directory (#836) 2026-04-21 16:06:41 +01:00
zh-cn Structure the tech specs directory (#836) 2026-04-21 16:06:41 +01:00
__TEMPLATE.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
active-flow-key-restructure.md Flow service lifecycle management (#822) 2026-04-16 17:19:39 +01:00
agent-explainability.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
agent-orchestration.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
architecture-principles.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
audit-events.md feat: add structured audit event system (#1027) 2026-07-06 10:47:49 +01:00
bootstrap.md feat: pluggable bootstrap framework with ordered initialisers (#847) 2026-04-22 18:03:46 +01:00
capabilities.md fix: list-my-workspaces permissions were broken (#1002) 2026-06-29 09:13:05 +01:00
cassandra-consolidation.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
cassandra-performance-refactor.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
collection-management.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
config-push-poke.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
data-ownership-model.md refactor(iam): pluggable IAM regime via authenticate/authorise contract (#853) 2026-04-28 16:19:41 +01:00
docling-decoder.md feat: add Docling-based document decoder as alternative to unstructured (#1054) 2026-07-20 21:25:33 +01:00
document-embeddings-chunk-id.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
embeddings-batch-processing.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
entity-centric-graph.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
explainability-cli.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
extraction-flows.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
extraction-provenance-subgraph.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
extraction-time-provenance.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
flow-blueprint-definition.md feat: pluggable image-to-text service with OpenAI vision backend (#1038) 2026-07-12 12:47:04 +01:00
flow-configurable-parameters.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
flow-service-queue-lifecycle.md Flow service lifecycle management (#822) 2026-04-16 17:19:39 +01:00
graph-contexts.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
graph-rag-semantic-filter.md feat: filter and cap GraphRAG reranker input across full stack (#1021) 2026-07-03 15:51:04 +01:00
graphql-query.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
graphrag-performance-optimization.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
iam-contract.md iam: self-service ops, optional workspace filters, Mux service routing (#855) 2026-04-28 22:13:12 +01:00
iam-protocol.md feat: add list-my-workspaces operation and document IAM in API specs (#961) 2026-05-29 19:17:37 +01:00
iam.md iam: self-service ops, optional workspace filters, Mux service routing (#855) 2026-04-28 22:13:12 +01:00
import-export-graceful-shutdown.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
jsonl-prompt-output.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
kafka-backend.md Add Kafka pub/sub backend (#830) 2026-04-18 11:18:34 +01:00
knowledge-core-completeness.md feat: complete knowledge core storage — named graphs, provenance, source material (#973) 2026-06-03 10:46:52 +01:00
large-document-loading.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
logging-strategy.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
mcp-tool-arguments.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
mcp-tool-bearer-token.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
minio-to-s3-migration.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
more-config-cli.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
multi-tenant-support.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
neo4j-user-collection-isolation.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
no-auth-regime.md feat: add no-auth IAM regime as a drop-in replacement for iam-svc (#933) 2026-05-18 14:10:05 +01:00
ontology-extract-phase-2.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
ontology.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
ontorag.md Fix ontology selector defaults, add bypass mode, enforce domain/range (#929) 2026-05-16 15:13:38 +01:00
openapi-spec.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
pubsub-abstraction.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
pubsub.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
python-api-refactor.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
query-time-explainability.md feat: structured source document references in graph-rag responses (#1035) 2026-07-08 23:59:56 +01:00
rag-streaming-support.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
schema-refactoring-proposal.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
sparql-query.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
streaming-llm-responses.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
structured-data-2.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
structured-data-descriptor.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
structured-data-index-split.md feat: batch row imports and split index configuration (#1055) 2026-07-22 10:20:06 +01:00
structured-data-schemas.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
structured-data.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
structured-diag-service.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
structured-output.md feat: LLM-native structured output via JSON schema enforcement (#1037) 2026-07-10 15:28:56 +01:00
tool-group.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
tool-services.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
universal-decoder.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
vector-store-lifecycle.md Feat: TrustGraph i18n & Documentation Translation Updates (#781) 2026-04-14 12:08:32 +01:00
workspace-scoped-services.md Per-workspace queue routing for workspace-scoped services (#862) 2026-05-04 10:30:03 +01:00