mirror of https://github.com/trustgraph-ai/trustgraph.git synced 2026-07-27 06:01:01 +02:00

feat: batch row imports and split index configuration (#1055 )

Adds batching to structured data row imports. Previously each row was
sent as a separate WebSocket message, causing one embeddings service
call per row. The bulk import clients now accumulate rows into batches
(default 40) before sending, and the --batch-size CLI argument is
wired through from tg-load-structured-data to the import call.

Splits the schema "indexes" field into two separate configurations:
- "query-indexes": fields for Cassandra exact-match retrieval
- "vector-indexes": fields for vector embedding and semantic search

This avoids wasting compute embedding primary key integers that only
need exact lookup, while allowing each field to independently opt in
to either or both retrieval patterns.

Adds row_id (the primary key value) as a clustering column in the
Cassandra rows table so multiple rows sharing the same index value
no longer silently overwrite each other.

Tests updated for new Cassandra schema.

2026-07-22 10:20:06 +01:00

2.8 KiB

Raw Blame History

Structured Data: Index Split and Composite Keys

Problem

The schema "indexes" field serves two purposes:

Cassandra row storage: creates denormalised copies for exact-match retrieval
Vector embeddings: sends field values to the embeddings service for semantic search

These are different capabilities with different costs. Embedding integer IDs is wasted compute; conversely, a field may benefit from semantic search without needing a Cassandra retrieval copy.

Additionally, the current Cassandra rows table uses index_value as the sole clustering column. Two rows sharing the same value for an indexed field (e.g. two stars named "Sirius") silently overwrite each other.

Changes

1. Split index configuration

Replace "indexes" with:

"query-indexes" -- fields for Cassandra exact-match lookup
"vector-indexes" -- fields for vector embedding and semantic search

A field can appear in both lists.

2. Composite clustering key

Add row_id (the value of the schema's primary key field) as a second clustering column:

Before:
  PRIMARY KEY ((collection, schema_name, index_name), index_value)

After:
  PRIMARY KEY ((collection, schema_name, index_name), index_value, row_id)

This allows multiple rows to share the same index value. Queries by index_value alone still work -- Cassandra allows omitting trailing clustering columns.

3. Schema config example

{
  "name": "stars",
  "fields": [
    {"name": "id", "type": "integer", "primary_key": true},
    {"name": "proper", "type": "string"},
    {"name": "spect", "type": "string"},
    {"name": "con", "type": "string"}
  ],
  "query-indexes": ["id", "proper", "con"],
  "vector-indexes": ["proper", "spect", "con"]
}

Future considerations

Composite primary keys: The primary key could span multiple columns (e.g. "primary_key": ["region", "id"]). The data model already supports this -- row_id in Cassandra is frozen<list<text>> and index_value is already a list. The schema config and writer would need to resolve multiple fields into the row_id value.
Composite query indexes: Each query index entry could cover multiple columns (e.g. "query-indexes": [["last_name", "first_name"], "email"]). This would allow exact-match lookup on field combinations. The Cassandra storage already uses index_value as a frozen<list<text>>, so the data model supports it -- the config parsing and index value construction would need updating.

Affected components

storage/rows/cassandra/write.py -- table schema, reads query-indexes, writes row_id
embeddings/row_embeddings/embeddings.py -- reads vector-indexes
query/row_embeddings/qdrant/service.py -- no change (consumes embeddings output)
Row query service -- must handle multiple rows per index value

2.8 KiB Raw Blame History