Cyber MacGeddon
33f031d664
The metadata field (list of triples) in the pipeline Metadata class
...
was redundant. Document metadata triples already flow directly from
librarian to triple-store via emit_document_provenance() - they don't
need to pass through the extraction pipeline.
Additionally, chunker and PDF decoder were overwriting metadata to []
anyway, so any metadata passed through the pipeline was being
discarded.
Changes:
- Remove metadata field from Metadata dataclass
(schema/core/metadata.py)
- Update all Metadata instantiations to remove metadata=[]
parameter
- Remove metadata handling from translators (document_loading,
knowledge)
- Remove metadata consumption from extractors (ontology, agent)
- Update gateway serializers and import handlers
- Update all unit, integration, and contract tests
2026-03-11 10:50:41 +00:00
cybermaggedon
7a6197d8c3
GraphRAG Query-Time Explainability ( #677 )
...
Implements full explainability pipeline for GraphRAG queries, enabling
traceability from answers back to source documents.
Renamed throughout for clarity:
- provenance_callback → explain_callback
- provenance_id → explain_id
- provenance_collection → explain_collection
- message_type "provenance" → "explain"
- Queue name "provenance" → "explainability"
GraphRAG queries now emit explainability events as they execute:
1. Session - query text and timestamp
2. Retrieval - edges retrieved from subgraph
3. Selection - selected edges with LLM reasoning (JSONL with id +
reasoning)
4. Answer - reference to synthesized response
Events stream via explain_callback during query(), enabling
real-time UX.
- Answers stored in librarian service (not inline in graph - too large)
- Document ID as URN: urn:trustgraph:answer:{session_id}
- Graph stores tg:document reference (IRI) to librarian document
- Added librarian producer/consumer to graph-rag service
- get_labelgraph() now returns (labeled_edges, uri_map)
- uri_map maps edge_id(label_s, label_p, label_o) →
(uri_s, uri_p, uri_o)
- Explainability data stores original URIs, not labels
- Enables tracing edges back to reifying statements via tg:reifies
- Added serialize_triple() to query service (matches storage format)
- get_term_value() now handles TRIPLE type terms
- Enables querying by quoted triple in object position:
?stmt tg:reifies <<s p o>>
- Displays real-time explainability events during query
- Resolves rdfs:label for edge components (s, p, o)
- Traces source chain via prov:wasDerivedFrom to root document
- Output: "Source: Chunk 1 → Page 2 → Document Title"
- Label caching to avoid repeated queries
GraphRagResponse:
- explain_id: str | None
- explain_collection: str | None
- message_type: str ("chunk" or "explain")
- end_of_session: bool
trustgraph-base/trustgraph/provenance/:
- namespaces.py - Added TG_DOCUMENT predicate
- triples.py - answer_triples() supports document_id reference
- uris.py - Added edge_selection_uri()
trustgraph-base/trustgraph/schema/services/retrieval.py:
- GraphRagResponse with explain_id, explain_collection, end_of_session
trustgraph-flow/trustgraph/retrieval/graph_rag/:
- graph_rag.py - URI preservation, streaming answer accumulation
- rag.py - Librarian integration, real-time explain emission
trustgraph-flow/trustgraph/query/triples/cassandra/service.py:
- Quoted triple serialization for query matching
trustgraph-cli/trustgraph/cli/invoke_graph_rag.py:
- Full explainability display with label resolution and source tracing
2026-03-10 10:00:01 +00:00
cybermaggedon
1809c1f56d
Structured data 2 ( #645 )
...
* Structured data refactor - multi-index tables, remove need for manual mods to the Cassandra tables
* Tech spec updated to track implementation
2026-02-23 15:56:29 +00:00
cybermaggedon
cf0daedefa
Changed schema for Value -> Term, majorly breaking change ( #622 )
...
* Changed schema for Value -> Term, majorly breaking change
* Following the schema change, Value -> Term into all processing
* Updated Cassandra for g, p, s, o index patterns (7 indexes)
* Reviewed and updated all tests
* Neo4j, Memgraph and FalkorDB remain broken, will look at once settled down
2026-01-27 13:48:08 +00:00
cybermaggedon
62b754d788
Fix flow loading ( #611 )
2026-01-14 16:23:15 +00:00
cybermaggedon
53cf5fd7f9
Fix test async warnings ( #601 )
...
* Fix tracemalloc async warnings
* Comment out debug, left in for use if needed
2026-01-06 22:09:34 +00:00
cybermaggedon
f79d0603f7
Update to add streaming tests ( #600 )
2026-01-06 21:48:05 +00:00
cybermaggedon
5304f96fe6
Fix tests ( #593 )
...
* Fix unit/integration/contract tests which were broken by messaging fabric work
2025-12-19 08:53:21 +00:00
cybermaggedon
ba95fa226b
Gateway queue overrides ( #584 )
2025-12-06 11:01:20 +00:00
cybermaggedon
0b7620bc04
Object batching ( #499 )
...
* Object batching
* Update tests
2025-09-05 15:59:06 +01:00
cybermaggedon
257a7951a7
Object import ( #497 )
...
* Object import dispatcher
* Add object import gateway test
2025-09-05 14:06:01 +01:00
cybermaggedon
96c2b73457
Fix import export graceful shutdown ( #476 )
...
* Tech spec for graceful shutdown
* Graceful shutdown of importers/exporters
* Update socket to include graceful shutdown orchestration
* Adding tests for conditions tracked in this PR
2025-08-28 13:39:28 +01:00
cybermaggedon
2f7fddd206
Test suite executed from CI pipeline ( #433 )
...
* Test strategy & test cases
* Unit tests
* Integration tests
2025-07-14 14:57:44 +01:00
