mirror of
https://github.com/trustgraph-ai/trustgraph.git
synced 2026-07-10 13:52:11 +02:00
Extract-time provenance (#661)
1. Shared Provenance Module - URI generators, namespace constants,
triple builders, vocabulary bootstrap
2. Librarian - Emits document metadata to graph on processing
initiation (vocabulary bootstrap + PROV-O triples)
3. PDF Extractor - Saves pages as child documents, emits parent-child
provenance edges, forwards page IDs
4. Chunker - Saves chunks as child documents, emits provenance edges,
forwards chunk ID + content
5. Knowledge Extractors (both definitions and relationships):
- Link entities to chunks via SUBJECT_OF (not top-level document)
- Removed duplicate metadata emission (now handled by librarian)
- Get chunk_doc_id and chunk_uri from incoming Chunk message
6. Embedding Provenance:
- EntityContext schema has chunk_id field
- EntityEmbeddings schema has chunk_id field
- Definitions extractor sets chunk_id when creating EntityContext
- Graph embeddings processor passes chunk_id through to
EntityEmbeddings
Provenance Flow:
Document → Page (PDF) → Chunk → Extracted Facts/Embeddings
↓ ↓ ↓ ↓
librarian librarian librarian (chunk_id reference)
+ graph + graph + graph
Each artifact is stored in librarian with parent-child linking, and PROV-O
edges are emitted to the knowledge graph for full traceability from any
extracted fact back to its source document.
Also, updating tests
This commit is contained in:
parent
d8f0a576af
commit
cd5580be59
20 changed files with 1601 additions and 59 deletions
|
|
@ -128,10 +128,12 @@ class Processor(FlowProcessor):
|
|||
triples = []
|
||||
entities = []
|
||||
|
||||
# FIXME: Putting metadata into triples store is duplicated in
|
||||
# relationships extractor too
|
||||
for t in v.metadata.metadata:
|
||||
triples.append(t)
|
||||
# Get chunk document ID for provenance linking
|
||||
chunk_doc_id = v.document_id if v.document_id else v.metadata.id
|
||||
chunk_uri = v.metadata.id # The URI form for the chunk
|
||||
|
||||
# Note: Document metadata is now emitted once by librarian at processing
|
||||
# initiation, so we don't need to duplicate it here.
|
||||
|
||||
for defn in defs:
|
||||
|
||||
|
|
@ -159,22 +161,27 @@ class Processor(FlowProcessor):
|
|||
s=s_value, p=DEFINITION_VALUE, o=o_value
|
||||
))
|
||||
|
||||
# Link entity to chunk (not top-level document)
|
||||
triples.append(Triple(
|
||||
s=s_value,
|
||||
p=SUBJECT_OF_VALUE,
|
||||
o=Term(type=IRI, iri=v.metadata.id)
|
||||
o=Term(type=IRI, iri=chunk_uri)
|
||||
))
|
||||
|
||||
# Output entity name as context for direct name matching
|
||||
# Include chunk_id for embedding provenance
|
||||
entities.append(EntityContext(
|
||||
entity=s_value,
|
||||
context=s,
|
||||
chunk_id=chunk_doc_id,
|
||||
))
|
||||
|
||||
# Output definition as context for semantic matching
|
||||
# Include chunk_id for embedding provenance
|
||||
entities.append(EntityContext(
|
||||
entity=s_value,
|
||||
context=defn["definition"],
|
||||
chunk_id=chunk_doc_id,
|
||||
))
|
||||
|
||||
# Send triples in batches
|
||||
|
|
|
|||
|
|
@ -109,10 +109,12 @@ class Processor(FlowProcessor):
|
|||
|
||||
triples = []
|
||||
|
||||
# FIXME: Putting metadata into triples store is duplicated in
|
||||
# relationships extractor too
|
||||
for t in v.metadata.metadata:
|
||||
triples.append(t)
|
||||
# Get chunk document ID for provenance linking
|
||||
chunk_doc_id = v.document_id if v.document_id else v.metadata.id
|
||||
chunk_uri = v.metadata.id # The URI form for the chunk
|
||||
|
||||
# Note: Document metadata is now emitted once by librarian at processing
|
||||
# initiation, so we don't need to duplicate it here.
|
||||
|
||||
for rel in rels:
|
||||
|
||||
|
|
@ -168,19 +170,19 @@ class Processor(FlowProcessor):
|
|||
o=Term(type=LITERAL, value=str(o))
|
||||
))
|
||||
|
||||
# 'Subject of' for s
|
||||
# Link entity to chunk (not top-level document)
|
||||
triples.append(Triple(
|
||||
s=s_value,
|
||||
p=SUBJECT_OF_VALUE,
|
||||
o=Term(type=IRI, iri=v.metadata.id)
|
||||
o=Term(type=IRI, iri=chunk_uri)
|
||||
))
|
||||
|
||||
if rel["object-entity"]:
|
||||
# 'Subject of' for o
|
||||
# Link object entity to chunk
|
||||
triples.append(Triple(
|
||||
s=o_value,
|
||||
p=SUBJECT_OF_VALUE,
|
||||
o=Term(type=IRI, iri=v.metadata.id)
|
||||
o=Term(type=IRI, iri=chunk_uri)
|
||||
))
|
||||
|
||||
# Send triples in batches
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue