mirror of
https://github.com/trustgraph-ai/trustgraph.git
synced 2026-07-17 09:11:03 +02:00
The id field in pipeline Metadata was being overwritten at each processing (#686)
The id field in pipeline Metadata was being overwritten at each processing stage (document → page → chunk), causing knowledge storage to create separate cores per chunk instead of grouping by document. Add a root field that: - Is set by librarian to the original document ID - Is copied unchanged through PDF decoder, chunkers, and extractors - Is used by knowledge storage for document_id grouping (with fallback to id) Changes: - Add root field to Metadata schema with empty string default - Set root=document.id in librarian when initiating document processing - Copy root through PDF decoder, recursive chunker, and all extractors - Update knowledge storage to use root (or id as fallback) for grouping - Add root handling to translators and gateway serialization - Update test mock Metadata class to include root parameter
This commit is contained in:
parent
aa4f5c6c00
commit
286f762369
15 changed files with 48 additions and 4 deletions
|
|
@ -233,7 +233,7 @@ class KnowledgeTableStore:
|
|||
self.insert_triples_stmt,
|
||||
(
|
||||
uuid.uuid4(), m.metadata.user,
|
||||
m.metadata.id, when,
|
||||
m.metadata.root or m.metadata.id, when,
|
||||
[], triples,
|
||||
)
|
||||
)
|
||||
|
|
@ -265,7 +265,7 @@ class KnowledgeTableStore:
|
|||
self.insert_graph_embeddings_stmt,
|
||||
(
|
||||
uuid.uuid4(), m.metadata.user,
|
||||
m.metadata.id, when,
|
||||
m.metadata.root or m.metadata.id, when,
|
||||
[], entities,
|
||||
)
|
||||
)
|
||||
|
|
@ -297,7 +297,7 @@ class KnowledgeTableStore:
|
|||
self.insert_document_embeddings_stmt,
|
||||
(
|
||||
uuid.uuid4(), m.metadata.user,
|
||||
m.metadata.id, when,
|
||||
m.metadata.root or m.metadata.id, when,
|
||||
[], chunks,
|
||||
)
|
||||
)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue