trustgraph/docs/tech-specs/agent-explainability.md

Agent Explainability: Provenance Recording

Overview

Add provenance recording to the React agent loop so agent sessions can be traced and debugged using the same explainability infrastructure as GraphRAG.

Design Decisions:

• Write to urn:graph:retrieval (generic explainability graph)
• Linear dependency chain for now (iteration N → wasDerivedFrom → iteration N-1)
• Tools are opaque black boxes (record input/output only)
• DAG support deferred to future iteration

Provenance Model

AgentSession (urn:trustgraph:agent:{uuid})
    │
    │  tg:query = "User's question"
    │  prov:startedAtTime = timestamp
    │  rdf:type = tg:AgentSession
    │
    ↓ prov:wasGeneratedBy
    │
Iteration1 (urn:trustgraph:agent:{uuid}/i1)
    │
    │  tg:thought = "I need to query the knowledge base..."
    │  tg:action = "knowledge-query"
    │  tg:arguments = {"question": "..."}
    │  tg:observation = "Result from tool..."
    │  rdf:type = tg:AgentIteration
    │
    ↓ prov:wasDerivedFrom
    │
Iteration2 (urn:trustgraph:agent:{uuid}/i2)
    │  ...
    ↓ prov:wasDerivedFrom
    │
FinalAnswer (urn:trustgraph:agent:{uuid}/final)
    │
    │  tg:answer = "The final response..."
    │  rdf:type = tg:AgentFinal

Changes Required

1. Schema Changes

File: trustgraph-base/trustgraph/schema/services/agent.py

Add session_id and collection fields to AgentRequest:

@dataclass
class AgentRequest:
    question: str = ""
    state: str = ""
    group: list[str] | None = None
    history: list[AgentStep] = field(default_factory=list)
    user: str = ""
    collection: str = "default"  # NEW: Collection for provenance traces
    streaming: bool = False
    session_id: str = ""         # NEW: For provenance tracking across iterations

File: trustgraph-base/trustgraph/messaging/translators/agent.py

Update translator to handle session_id and collection in both to_pulsar() and from_pulsar().

2. Add Explainability Producer to Agent Service

File: trustgraph-flow/trustgraph/agent/react/service.py

Register an "explainability" producer (same pattern as GraphRAG):

from ... base import ProducerSpec
from ... schema import Triples

# In __init__:
self.register_specification(
    ProducerSpec(
        name = "explainability",
        schema = Triples,
    )
)

3. Provenance Triple Generation

Option A: Add to existing trustgraph-base/trustgraph/provenance/ module Option B: Create agent-specific provenance helpers in the agent module

Create helper functions (similar to GraphRAG's question_triples, exploration_triples, etc.):

def agent_session_triples(session_uri, query, timestamp):
    """Generate triples for agent session start."""
    return [
        Triple(s=session_uri, p=RDF_TYPE, o=TG_AGENT_SESSION),
        Triple(s=session_uri, p=TG_QUERY, o=query),
        Triple(s=session_uri, p=PROV_STARTED_AT_TIME, o=timestamp),
    ]

def agent_iteration_triples(iteration_uri, parent_uri, thought, action, arguments, observation):
    """Generate triples for one agent iteration."""
    return [
        Triple(s=iteration_uri, p=RDF_TYPE, o=TG_AGENT_ITERATION),
        Triple(s=iteration_uri, p=TG_THOUGHT, o=thought),
        Triple(s=iteration_uri, p=TG_ACTION, o=action),
        Triple(s=iteration_uri, p=TG_ARGUMENTS, o=json.dumps(arguments)),
        Triple(s=iteration_uri, p=TG_OBSERVATION, o=observation),
        Triple(s=iteration_uri, p=PROV_WAS_DERIVED_FROM, o=parent_uri),
    ]

def agent_final_triples(final_uri, parent_uri, answer):
    """Generate triples for agent final answer."""
    return [
        Triple(s=final_uri, p=RDF_TYPE, o=TG_AGENT_FINAL),
        Triple(s=final_uri, p=TG_ANSWER, o=answer),
        Triple(s=final_uri, p=PROV_WAS_DERIVED_FROM, o=parent_uri),
    ]

4. Integration in service.py

In agent_request() method:

async def agent_request(self, request, respond, next, flow):
    # Generate or retrieve session ID
    if not request.session_id:
        session_id = str(uuid.uuid4())
    else:
        session_id = request.session_id

    session_uri = f"urn:trustgraph:agent:{session_id}"
    iteration_num = len(history) + 1
    iteration_uri = f"{session_uri}/i{iteration_num}"

    # On first iteration, emit session triples
    if iteration_num == 1:
        triples = agent_session_triples(session_uri, request.question, timestamp)
        await flow("explainability").send(Triples(
            metadata=Metadata(user=request.user, collection=..., id=session_uri),
            triples=triples,
        ))

    # ... existing react() call ...

    if isinstance(act, Final):
        # Emit final answer triples
        final_uri = f"{session_uri}/final"
        parent_uri = f"{session_uri}/i{iteration_num - 1}" if iteration_num > 1 else session_uri
        triples = agent_final_triples(final_uri, parent_uri, act.final)
        await flow("explainability").send(...)
    else:
        # Emit iteration triples
        parent_uri = f"{session_uri}/i{iteration_num - 1}" if iteration_num > 1 else session_uri
        triples = agent_iteration_triples(iteration_uri, parent_uri, act.thought, act.name, act.arguments, act.observation)
        await flow("explainability").send(...)

        # Pass session_id to next iteration
        r = AgentRequest(
            ...,
            session_id=session_id,
        )

5. Predicate Definitions

File: trustgraph-base/trustgraph/provenance/namespaces.py

Add new predicates:

TG_THOUGHT = TG + "thought"
TG_ACTION = TG + "action"
TG_ARGUMENTS = TG + "arguments"
TG_OBSERVATION = TG + "observation"
TG_AGENT_SESSION = TG + "AgentSession"
TG_AGENT_ITERATION = TG + "AgentIteration"
TG_AGENT_FINAL = TG + "AgentFinal"
# TG_QUERY and TG_ANSWER likely already exist

Files to Modify

File	Change
trustgraph-base/trustgraph/schema/services/agent.py	Add session_id and collection to AgentRequest
trustgraph-base/trustgraph/messaging/translators/agent.py	Update translator for new fields
trustgraph-base/trustgraph/provenance/namespaces.py	Add agent predicates
trustgraph-base/trustgraph/provenance/__init__.py	Export new predicates
trustgraph-flow/trustgraph/agent/react/service.py	Add explainability producer + recording logic
trustgraph-cli/trustgraph/cli/show_explain_trace.py	Handle agent trace types
trustgraph-cli/trustgraph/cli/list_explain_traces.py	List agent sessions alongside GraphRAG

Files to Potentially Create

File	Purpose
trustgraph-base/trustgraph/provenance/agent.py	Agent-specific triple generators (optional, could inline in service.py)

CLI Updates Detail

list_explain_traces.py:

• Currently queries for tg:query predicate to find questions
• Agent sessions also use tg:query, so should work automatically
• May want to add a type indicator column (GraphRAG vs Agent)

show_explain_trace.py:

• Currently expects: question → exploration → focus → synthesis chain
• Agent traces have: session → iteration(s) → final chain
• Detection: check rdf:type of the root entity
  • tg:AgentSession → agent trace rendering
  • Otherwise → GraphRAG trace rendering (existing)
• Agent rendering shows:
  • Session info (question, time)
  • Each iteration: thought, action, args, observation
  • Final answer

Design Decisions

1. Collection handling: Add collection field to AgentRequest. Agent receives a collection parameter for provenance traces. Tools can access other collections per their config, but decision traces stay in the invoked collection.

2. CLI tool updates: Update existing tg-show-explain-trace to detect and handle both GraphRAG and agent trace types.

Implementation Order

0. Tech Spec: Write this plan to docs/tech-specs/agent-explainability.md
1. Schema: Add session_id and collection to AgentRequest + translator
2. Predicates: Add agent predicates to namespaces.py
3. Service: Add explainability producer to agent service
4. Provenance: Create/add agent triple generators
5. Integration: Wire up provenance recording in agent_request()
6. CLI: Update show_explain_trace.py to detect and render agent traces
7. Test: Run agent query and verify trace is recorded and viewable

Backwards Compatibility

• session_id defaults to "" - old requests work, just won't have provenance
• collection defaults to "default" - reasonable fallback
• CLI gracefully handles both trace types

Verification

# Run an agent query
tg-invoke-agent -q "What is the capital of France?"

# Check triples were written
tg-query-graph -p "https://trustgraph.ai/ns/query" -g "urn:graph:retrieval"

# List traces (should show agent sessions)
tg-list-explain-traces -U trustgraph -C default

# Show trace (may need CLI updates for agent entity types)
tg-show-explain-trace "urn:trustgraph:agent:xxx"

Future Work (Not This PR)

• DAG dependencies (when iteration N uses results from multiple prior iterations)
• Tool-specific provenance linking (KnowledgeQuery → its GraphRAG trace)
• CLI tool enhancements to better display agent traces
• Streaming provenance emission (emit as we go, not batch at end)