mirror of
https://github.com/trustgraph-ai/trustgraph.git
synced 2026-04-25 00:16:23 +02:00
286f762369
The id field in pipeline Metadata was being overwritten at each processing stage (document → page → chunk), causing knowledge storage to create separate cores per chunk instead of grouping by document. Add a root field that: - Is set by librarian to the original document ID - Is copied unchanged through PDF decoder, chunkers, and extractors - Is used by knowledge storage for document_id grouping (with fallback to id) Changes: - Add root field to Metadata schema with empty string default - Set root=document.id in librarian when initiating document processing - Copy root through PDF decoder, recursive chunker, and all extractors - Update knowledge storage to use root (or id as fallback) for grouping - Add root handling to translators and gateway serialization - Update test mock Metadata class to include root parameter
13 lines
274 B
Python
13 lines
274 B
Python
from dataclasses import dataclass
|
|
|
|
@dataclass
|
|
class Metadata:
|
|
# Source identifier
|
|
id: str = ""
|
|
|
|
# Root document identifier (set by librarian, preserved through pipeline)
|
|
root: str = ""
|
|
|
|
# Collection management
|
|
user: str = ""
|
|
collection: str = ""
|