mirror of https://github.com/trustgraph-ai/trustgraph.git synced 2026-04-25 00:16:23 +02:00

cybermaggedon 7a6197d8c3 GraphRAG Query-Time Explainability (#677 ) Implements full explainability pipeline for GraphRAG queries, enabling traceability from answers back to source documents. Renamed throughout for clarity: - provenance_callback → explain_callback - provenance_id → explain_id - provenance_collection → explain_collection - message_type "provenance" → "explain" - Queue name "provenance" → "explainability" GraphRAG queries now emit explainability events as they execute: 1. Session - query text and timestamp 2. Retrieval - edges retrieved from subgraph 3. Selection - selected edges with LLM reasoning (JSONL with id + reasoning) 4. Answer - reference to synthesized response Events stream via explain_callback during query(), enabling real-time UX. - Answers stored in librarian service (not inline in graph - too large) - Document ID as URN: urn:trustgraph:answer:{session_id} - Graph stores tg:document reference (IRI) to librarian document - Added librarian producer/consumer to graph-rag service - get_labelgraph() now returns (labeled_edges, uri_map) - uri_map maps edge_id(label_s, label_p, label_o) → (uri_s, uri_p, uri_o) - Explainability data stores original URIs, not labels - Enables tracing edges back to reifying statements via tg:reifies - Added serialize_triple() to query service (matches storage format) - get_term_value() now handles TRIPLE type terms - Enables querying by quoted triple in object position: ?stmt tg:reifies <<s p o>> - Displays real-time explainability events during query - Resolves rdfs:label for edge components (s, p, o) - Traces source chain via prov:wasDerivedFrom to root document - Output: "Source: Chunk 1 → Page 2 → Document Title" - Label caching to avoid repeated queries GraphRagResponse: - explain_id: str \| None - explain_collection: str \| None - message_type: str ("chunk" or "explain") - end_of_session: bool trustgraph-base/trustgraph/provenance/: - namespaces.py - Added TG_DOCUMENT predicate - triples.py - answer_triples() supports document_id reference - uris.py - Added edge_selection_uri() trustgraph-base/trustgraph/schema/services/retrieval.py: - GraphRagResponse with explain_id, explain_collection, end_of_session trustgraph-flow/trustgraph/retrieval/graph_rag/: - graph_rag.py - URI preservation, streaming answer accumulation - rag.py - Librarian integration, real-time explain emission trustgraph-flow/trustgraph/query/triples/cassandra/service.py: - Quoted triple serialization for query matching trustgraph-cli/trustgraph/cli/invoke_graph_rag.py: - Full explainability display with label resolution and source tracing		2026-03-10 10:00:01 +00:00
..
api	GraphRAG Query-Time Explainability (#677 )	2026-03-10 10:00:01 +00:00
websocket	Row embeddings APIs exposed (#646 )	2026-02-23 21:52:56 +00:00
build-docs.sh	Add AsyncAPI spec for websocket (#613 )	2026-01-15 11:57:16 +00:00
README.md	Add AsyncAPI spec for websocket (#613 )	2026-01-15 11:57:16 +00:00

README.md

TrustGraph API Specifications

This directory contains formal specifications for the TrustGraph API Gateway.

Directory Structure

specs/
├── api/              # OpenAPI 3.1.0 specification for REST API
│   ├── openapi.yaml  # Main entry point
│   ├── paths/        # Endpoint definitions
│   ├── components/   # Reusable schemas, responses, parameters
│   └── security/     # Security scheme definitions
│
├── websocket/        # AsyncAPI 3.0.0 specification for WebSocket API
│   ├── asyncapi.yaml # Main entry point
│   ├── channels/     # Channel definitions
│   ├── components/   # Message and schema definitions
│   └── STREAMING.md  # Streaming patterns documentation
│
└── README.md         # This file

Specifications

REST API (OpenAPI 3.1.0)

Location: specs/api/openapi.yaml

The REST API specification documents:

5 Global Services: config, flow, librarian, knowledge, collection-management
16 Flow-Hosted Services: agent, RAG, embeddings, queries, loading, tools
Import/Export: Bulk data operations
Metrics: Prometheus monitoring

Features:

Modular structure with $ref to external files
Comprehensive request/response schemas
Authentication via Bearer tokens
Field naming in kebab-case

WebSocket API (AsyncAPI 3.0.0)

Location: specs/websocket/asyncapi.yaml

The WebSocket API specification documents:

Multiplexed async communication protocol
Request/response message envelopes with ID correlation
All services accessible via single WebSocket connection
Streaming response patterns

Features:

References REST API schemas (single source of truth)
Message-based routing (service + optional flow parameters)
Comprehensive streaming documentation
Full async/multiplexing behavior

Building Documentation

Prerequisites

npm install -g @redocly/cli @asyncapi/cli

Or use npx (no installation required).

Generate REST API Documentation

Using Redocly (HTML):

cd specs/api
npx @redocly/cli build-docs openapi.yaml -o ../../docs/api.html

Preview in browser:

cd specs/api
npx @redocly/cli preview-docs openapi.yaml

Opens interactive documentation at http://localhost:8080

Validate:

cd specs/api
npx @redocly/cli lint openapi.yaml

Generate WebSocket API Documentation

Using AsyncAPI (HTML):

cd specs/websocket
npx -p @asyncapi/cli asyncapi generate fromTemplate asyncapi.yaml @asyncapi/html-template@3.0.0 --use-new-generator -o /tmp/asyncapi-build -p singleFile=true --force-write
mv /tmp/asyncapi-build/index.html ../../docs/websocket.html
rm -rf /tmp/asyncapi-build

Notes:

The generator must run from the specs/websocket directory to properly resolve relative $ref paths to OpenAPI schemas
--use-new-generator flag enables generator v2 compatibility
-p singleFile=true parameter generates a single standalone HTML file

Validate:

cd specs/websocket
npx @asyncapi/cli validate asyncapi.yaml

Build All Documentation

Use the provided build script:

./specs/build-docs.sh

This generates:

docs/api.html - REST API documentation
docs/websocket.html - WebSocket API documentation

Viewing Documentation

After building:

REST API:

xdg-open docs/api.html
# or
firefox docs/api.html

WebSocket API:

xdg-open docs/websocket.html
# or
firefox docs/websocket.html

Schema Reuse

The WebSocket API specification references the REST API schemas using relative paths:

# In specs/websocket/components/messages/requests/AgentRequest.yaml
request:
  $ref: '../../../../api/components/schemas/agent/AgentRequest.yaml'

This ensures:

Single source of truth for all schemas
Consistency between REST and WebSocket APIs
Easy maintenance - update schemas in one place

Validation Status

Both specifications are validated and error-free:

✅ OpenAPI: Validated with Redocly CLI
✅ AsyncAPI: Validated with AsyncAPI CLI

Maintenance

Adding a New Service

Create schemas in specs/api/components/schemas/{service-name}/
- {ServiceName}Request.yaml
- {ServiceName}Response.yaml
Create path definition in specs/api/paths/ or specs/api/paths/flow/
Add path to main spec in specs/api/openapi.yaml
Create WebSocket message in specs/websocket/components/messages/requests/
- Reference the OpenAPI request schema
Add to ServiceRequest message in specs/websocket/components/messages/ServiceRequest.yaml

Validate both specs:

cd specs/api && npx @redocly/cli lint openapi.yaml
cd specs/websocket && npx @asyncapi/cli validate asyncapi.yaml

Modifying an Existing Service

Update schema in specs/api/components/schemas/{service-name}/
Changes automatically apply to WebSocket spec via $ref
Validate both specs to ensure consistency

Tools and Resources

OpenAPI Tools:

Redocly CLI - Linting, docs generation
Swagger Editor - Online editor
OpenAPI Generator - Client/server code generation

AsyncAPI Tools:

AsyncAPI CLI - Validation, docs generation
AsyncAPI Studio - Online editor
AsyncAPI Generator - Template-based generation

Online Validators:

OpenAPI: https://editor.swagger.io/
AsyncAPI: https://studio.asyncapi.com/

API Version

Current version: 1.8.0

Version is specified in both:

specs/api/openapi.yaml → info.version
specs/websocket/asyncapi.yaml → info.version

Update both when releasing a new API version.