trustgraph/docs/tech-specs/document-embeddings-chunk-id.md

Document Embeddings Chunk ID

Overview

Document embeddings storage currently stores chunk text directly in the vector store payload, duplicating data that exists in Garage. This spec replaces chunk text storage with chunk_id references.

Current State

@dataclass
class ChunkEmbeddings:
    chunk: bytes = b""
    vectors: list[list[float]] = field(default_factory=list)

Vector store payload:

payload={"doc": chunk}  # Duplicates Garage content

Design

Schema Changes

ChunkEmbeddings - replace chunk with chunk_id:

@dataclass
class ChunkEmbeddings:
    chunk_id: str = ""
    vectors: list[list[float]] = field(default_factory=list)

DocumentEmbeddingsResponse - return chunk_ids instead of chunks:

@dataclass
class DocumentEmbeddingsResponse:
    error: Error | None = None
    chunk_ids: list[str] = field(default_factory=list)

Vector Store Payload

All stores (Qdrant, Milvus, Pinecone):

payload={"chunk_id": chunk_id}

Query Flow

1. Search vector store → get matching chunk_id values
2. Return chunk_ids to caller
3. Caller fetches content from Garage if needed

Files to Modify

Schema

• trustgraph-base/trustgraph/schema/knowledge/embeddings.py
• trustgraph-base/trustgraph/schema/services/query.py

Embeddings Service

• trustgraph-flow/trustgraph/embeddings/document_embeddings/embeddings.py

Storage Writers

• trustgraph-flow/trustgraph/storage/doc_embeddings/qdrant/write.py
• trustgraph-flow/trustgraph/storage/doc_embeddings/milvus/write.py
• trustgraph-flow/trustgraph/storage/doc_embeddings/pinecone/write.py

Query Services

• trustgraph-flow/trustgraph/query/doc_embeddings/qdrant/service.py
• trustgraph-flow/trustgraph/query/doc_embeddings/milvus/service.py
• trustgraph-flow/trustgraph/query/doc_embeddings/pinecone/service.py

Benefits

1. Single source of truth - chunk text only in Garage
2. Reduced vector store storage
3. Enables query-time provenance via chunk_id