Extract-time provenance (#661)

1. Shared Provenance Module - URI generators, namespace constants,
   triple builders, vocabulary bootstrap
2. Librarian - Emits document metadata to graph on processing
   initiation (vocabulary bootstrap + PROV-O triples)
3. PDF Extractor - Saves pages as child documents, emits parent-child
   provenance edges, forwards page IDs
4. Chunker - Saves chunks as child documents, emits provenance edges,
   forwards chunk ID + content
5. Knowledge Extractors (both definitions and relationships):
   - Link entities to chunks via SUBJECT_OF (not top-level document)
   - Removed duplicate metadata emission (now handled by librarian)
   - Get chunk_doc_id and chunk_uri from incoming Chunk message
6. Embedding Provenance:
   - EntityContext schema has chunk_id field
   - EntityEmbeddings schema has chunk_id field
   - Definitions extractor sets chunk_id when creating EntityContext
   - Graph embeddings processor passes chunk_id through to
     EntityEmbeddings

Provenance Flow:
Document → Page (PDF) → Chunk → Extracted Facts/Embeddings
    ↓           ↓          ↓              ↓
  librarian  librarian  librarian    (chunk_id reference)
  + graph    + graph    + graph

Each artifact is stored in librarian with parent-child linking, and PROV-O
edges are emitted to the knowledge graph for full traceability from any
extracted fact back to its source document.

Also, updating tests
This commit is contained in:
cybermaggedon 2026-03-05 18:36:10 +00:00 committed by GitHub
parent d8f0a576af
commit cd5580be59
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
20 changed files with 1601 additions and 59 deletions

View file

@ -128,10 +128,12 @@ class Processor(FlowProcessor):
triples = []
entities = []
# FIXME: Putting metadata into triples store is duplicated in
# relationships extractor too
for t in v.metadata.metadata:
triples.append(t)
# Get chunk document ID for provenance linking
chunk_doc_id = v.document_id if v.document_id else v.metadata.id
chunk_uri = v.metadata.id # The URI form for the chunk
# Note: Document metadata is now emitted once by librarian at processing
# initiation, so we don't need to duplicate it here.
for defn in defs:
@ -159,22 +161,27 @@ class Processor(FlowProcessor):
s=s_value, p=DEFINITION_VALUE, o=o_value
))
# Link entity to chunk (not top-level document)
triples.append(Triple(
s=s_value,
p=SUBJECT_OF_VALUE,
o=Term(type=IRI, iri=v.metadata.id)
o=Term(type=IRI, iri=chunk_uri)
))
# Output entity name as context for direct name matching
# Include chunk_id for embedding provenance
entities.append(EntityContext(
entity=s_value,
context=s,
chunk_id=chunk_doc_id,
))
# Output definition as context for semantic matching
# Include chunk_id for embedding provenance
entities.append(EntityContext(
entity=s_value,
context=defn["definition"],
chunk_id=chunk_doc_id,
))
# Send triples in batches

View file

@ -109,10 +109,12 @@ class Processor(FlowProcessor):
triples = []
# FIXME: Putting metadata into triples store is duplicated in
# relationships extractor too
for t in v.metadata.metadata:
triples.append(t)
# Get chunk document ID for provenance linking
chunk_doc_id = v.document_id if v.document_id else v.metadata.id
chunk_uri = v.metadata.id # The URI form for the chunk
# Note: Document metadata is now emitted once by librarian at processing
# initiation, so we don't need to duplicate it here.
for rel in rels:
@ -168,19 +170,19 @@ class Processor(FlowProcessor):
o=Term(type=LITERAL, value=str(o))
))
# 'Subject of' for s
# Link entity to chunk (not top-level document)
triples.append(Triple(
s=s_value,
p=SUBJECT_OF_VALUE,
o=Term(type=IRI, iri=v.metadata.id)
o=Term(type=IRI, iri=chunk_uri)
))
if rel["object-entity"]:
# 'Subject of' for o
# Link object entity to chunk
triples.append(Triple(
s=o_value,
p=SUBJECT_OF_VALUE,
o=Term(type=IRI, iri=v.metadata.id)
o=Term(type=IRI, iri=chunk_uri)
))
# Send triples in batches