apunkt/trustgraph
1
0
Fork 0
mirror of https://github.com/trustgraph-ai/trustgraph.git synced 2026-04-25 00:16:23 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Updated tech specs for agent & explainability changes (#796)

Browse source
This commit is contained in:
cybermaggedon 2026-04-13 17:29:24 +01:00 • committed by GitHub
parent d2751553a3
commit 1515dbaf08
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
7 changed files with 205 additions and 233 deletions
Download patch file Download diff file Expand all files Collapse all files

361
docs/tech-specs/agent-explainability.md
View file

@ -1,272 +1,211 @@
# Agent Explainability: Provenance Recording # Agent Explainability: Provenance Recording
## Status
Implemented
## Overview ## Overview
Add provenance recording to the React agent loop so agent sessions can be traced and debugged using the same explainability infrastructure as GraphRAG. Agent sessions are traced and debugged using the same explainability infrastructure as GraphRAG and Document RAG. Provenance is written to `urn:graph:retrieval` and delivered inline on the explain stream.
**Design Decisions:** The canonical vocabulary for all predicates and types is published as an OWL ontology at `specs/ontology/trustgraph.ttl`.
- Write to `urn:graph:retrieval` (generic explainability graph)
- Linear dependency chain for now (analysis N → wasDerivedFrom → analysis N-1)
- Tools are opaque black boxes (record input/output only)
- DAG support deferred to future iteration
## Entity Types ## Entity Types
Both GraphRAG and Agent use PROV-O as the base ontology with TrustGraph-specific subtypes: All services use PROV-O as the base ontology with TrustGraph-specific subtypes.
### GraphRAG Types ### GraphRAG Types
| Entity | PROV-O Type | TG Types | Description | | Entity | TG Types | Description |
|--------|-------------|----------|-------------| |--------|----------|-------------|
| Question | `prov:Activity` | `tg:Question`, `tg:GraphRagQuestion` | The user's query | | Question | `tg:Question`, `tg:GraphRagQuestion` | The user's query |
| Exploration | `prov:Entity` | `tg:Exploration` | Edges retrieved from knowledge graph | | Grounding | `tg:Grounding` | Concept extraction from query |
| Focus | `prov:Entity` | `tg:Focus` | Selected edges with reasoning | | Exploration | `tg:Exploration` | Edges retrieved from knowledge graph |
| Synthesis | `prov:Entity` | `tg:Synthesis` | Final answer | | Focus | `tg:Focus` | Selected edges with reasoning |
| Synthesis | `tg:Synthesis`, `tg:Answer` | Final answer |
### Agent Types
| Entity | PROV-O Type | TG Types | Description |
|--------|-------------|----------|-------------|
| Question | `prov:Activity` | `tg:Question`, `tg:AgentQuestion` | The user's query |
| Analysis | `prov:Entity` | `tg:Analysis` | Each think/act/observe cycle |
| Conclusion | `prov:Entity` | `tg:Conclusion` | Final answer |
### Document RAG Types ### Document RAG Types
| Entity | PROV-O Type | TG Types | Description | | Entity | TG Types | Description |
|--------|-------------|----------|-------------| |--------|----------|-------------|
| Question | `prov:Activity` | `tg:Question`, `tg:DocRagQuestion` | The user's query | | Question | `tg:Question`, `tg:DocRagQuestion` | The user's query |
| Exploration | `prov:Entity` | `tg:Exploration` | Chunks retrieved from document store | | Grounding | `tg:Grounding` | Concept extraction from query |
| Synthesis | `prov:Entity` | `tg:Synthesis` | Final answer | | Exploration | `tg:Exploration` | Chunks retrieved from document store |
| Synthesis | `tg:Synthesis`, `tg:Answer` | Final answer |
**Note:** Document RAG uses a subset of GraphRAG's types (no Focus step since there's no edge selection/reasoning phase). ### Agent Types (React)
| Entity | TG Types | Description |
|--------|----------|-------------|
| Question | `tg:Question`, `tg:AgentQuestion` | The user's query (session start) |
| PatternDecision | `tg:PatternDecision` | Meta-router routing decision |
| Analysis | `tg:Analysis`, `tg:ToolUse` | One think/act cycle |
| Thought | `tg:Reflection`, `tg:Thought` | Agent reasoning (sub-entity of Analysis) |
| Observation | `tg:Reflection`, `tg:Observation` | Tool result (standalone entity) |
| Conclusion | `tg:Conclusion`, `tg:Answer` | Final answer |
### Agent Types (Orchestrator — Plan)
| Entity | TG Types | Description |
|--------|----------|-------------|
| Plan | `tg:Plan` | Structured plan of steps |
| StepResult | `tg:StepResult`, `tg:Answer` | Result from executing one plan step |
| Synthesis | `tg:Synthesis`, `tg:Answer` | Final synthesised answer |
### Agent Types (Orchestrator — Supervisor)
| Entity | TG Types | Description |
|--------|----------|-------------|
| Decomposition | `tg:Decomposition` | Question decomposed into sub-goals |
| Finding | `tg:Finding`, `tg:Answer` | Result from a sub-agent |
| Synthesis | `tg:Synthesis`, `tg:Answer` | Final synthesised answer |
### Mixin Types
| Type | Description |
|------|-------------|
| `tg:Answer` | Unifying type for terminal answers (Synthesis, Conclusion, Finding, StepResult) |
| `tg:Reflection` | Unifying type for intermediate commentary (Thought, Observation) |
| `tg:ToolUse` | Applied to Analysis when a tool is invoked |
| `tg:Error` | Applied to Observation events where a failure occurred (tool error or LLM parse error) |
### Question Subtypes ### Question Subtypes
All Question entities share `tg:Question` as a base type but have a specific subtype to identify the retrieval mechanism:
| Subtype | URI Pattern | Mechanism | | Subtype | URI Pattern | Mechanism |
|---------|-------------|-----------| |---------|-------------|-----------|
| `tg:GraphRagQuestion` | `urn:trustgraph:question:{uuid}` | Knowledge graph RAG | | `tg:GraphRagQuestion` | `urn:trustgraph:question:{uuid}` | Knowledge graph RAG |
| `tg:DocRagQuestion` | `urn:trustgraph:docrag:{uuid}` | Document/chunk RAG | | `tg:DocRagQuestion` | `urn:trustgraph:docrag:{uuid}` | Document/chunk RAG |
| `tg:AgentQuestion` | `urn:trustgraph:agent:{uuid}` | ReAct agent | | `tg:AgentQuestion` | `urn:trustgraph:agent:session:{uuid}` | Agent orchestrator |
This allows querying all questions via `tg:Question` while filtering by specific mechanism via the subtype. ## Provenance Chains
## Provenance Model All chains use `prov:wasDerivedFrom` links. Each entity is a `prov:Entity`.
### GraphRAG
``` ```
Question (urn:trustgraph:agent:{uuid}) Question → Grounding → Exploration → Focus → Synthesis
│
│ tg:query = "User's question"
│ prov:startedAtTime = timestamp
│ rdf:type = prov:Activity, tg:Question
│
↓ prov:wasDerivedFrom
│
Analysis1 (urn:trustgraph:agent:{uuid}/i1)
│
│ tg:thought = "I need to query the knowledge base..."
│ tg:action = "knowledge-query"
│ tg:arguments = {"question": "..."}
│ tg:observation = "Result from tool..."
│ rdf:type = prov:Entity, tg:Analysis
│
↓ prov:wasDerivedFrom
│
Analysis2 (urn:trustgraph:agent:{uuid}/i2)
│ ...
↓ prov:wasDerivedFrom
│
Conclusion (urn:trustgraph:agent:{uuid}/final)
│
│ tg:answer = "The final response..."
│ rdf:type = prov:Entity, tg:Conclusion
``` ```
### Document RAG Provenance Model ### Document RAG
``` ```
Question (urn:trustgraph:docrag:{uuid}) Question → Grounding → Exploration → Synthesis
│
│ tg:query = "User's question"
│ prov:startedAtTime = timestamp
│ rdf:type = prov:Activity, tg:Question
│
↓ prov:wasGeneratedBy
│
Exploration (urn:trustgraph:docrag:{uuid}/exploration)
│
│ tg:chunkCount = 5
│ tg:selectedChunk = "chunk-id-1"
│ tg:selectedChunk = "chunk-id-2"
│ ...
│ rdf:type = prov:Entity, tg:Exploration
│
↓ prov:wasDerivedFrom
│
Synthesis (urn:trustgraph:docrag:{uuid}/synthesis)
│
│ tg:content = "The synthesized answer..."
│ rdf:type = prov:Entity, tg:Synthesis
``` ```
## Changes Required ### Agent React
### 1. Schema Changes ```
Question → PatternDecision → Analysis(1) → Observation(1) → Analysis(2) → ... → Conclusion
**File:** `trustgraph-base/trustgraph/schema/services/agent.py`
Add `session_id` and `collection` fields to `AgentRequest`:
```python
@dataclass
class AgentRequest:
question: str = ""
state: str = ""
group: list[str] | None = None
history: list[AgentStep] = field(default_factory=list)
user: str = ""
collection: str = "default" # NEW: Collection for provenance traces
streaming: bool = False
session_id: str = "" # NEW: For provenance tracking across iterations
``` ```
**File:** `trustgraph-base/trustgraph/messaging/translators/agent.py` The PatternDecision entity records which execution pattern the meta-router selected. It is only emitted on the first iteration when routing occurs.
Update translator to handle `session_id` and `collection` in both `to_pulsar()` and `from_pulsar()`. Thought sub-entities derive from their parent Analysis. Observation entities derive from their parent Analysis (or from a sub-trace entity if the tool produced its own explainability, e.g. a GraphRAG query).
### 2. Add Explainability Producer to Agent Service ### Agent Plan-then-Execute
**File:** `trustgraph-flow/trustgraph/agent/react/service.py` ```
Question → PatternDecision → Plan → StepResult(0) → StepResult(1) → ... → Synthesis
Register an "explainability" producer (same pattern as GraphRAG):
```python
from ... base import ProducerSpec
from ... schema import Triples
# In __init__:
self.register_specification(
ProducerSpec(
name = "explainability",
schema = Triples,
)
)
``` ```
### 3. Provenance Triple Generation ### Agent Supervisor
**File:** `trustgraph-base/trustgraph/provenance/agent.py` ```
Question → PatternDecision → Decomposition → [fan-out sub-agents]
Create helper functions (similar to GraphRAG's `question_triples`, `exploration_triples`, etc.): → Finding(0) → Finding(1) → ... → Synthesis
```python
def agent_session_triples(session_uri, query, timestamp):
"""Generate triples for agent Question."""
return [
Triple(s=session_uri, p=RDF_TYPE, o=PROV_ACTIVITY),
Triple(s=session_uri, p=RDF_TYPE, o=TG_QUESTION),
Triple(s=session_uri, p=TG_QUERY, o=query),
Triple(s=session_uri, p=PROV_STARTED_AT_TIME, o=timestamp),
]
def agent_iteration_triples(iteration_uri, parent_uri, thought, action, arguments, observation):
"""Generate triples for one Analysis step."""
return [
Triple(s=iteration_uri, p=RDF_TYPE, o=PROV_ENTITY),
Triple(s=iteration_uri, p=RDF_TYPE, o=TG_ANALYSIS),
Triple(s=iteration_uri, p=TG_THOUGHT, o=thought),
Triple(s=iteration_uri, p=TG_ACTION, o=action),
Triple(s=iteration_uri, p=TG_ARGUMENTS, o=json.dumps(arguments)),
Triple(s=iteration_uri, p=TG_OBSERVATION, o=observation),
Triple(s=iteration_uri, p=PROV_WAS_DERIVED_FROM, o=parent_uri),
]
def agent_final_triples(final_uri, parent_uri, answer):
"""Generate triples for Conclusion."""
return [
Triple(s=final_uri, p=RDF_TYPE, o=PROV_ENTITY),
Triple(s=final_uri, p=RDF_TYPE, o=TG_CONCLUSION),
Triple(s=final_uri, p=TG_ANSWER, o=answer),
Triple(s=final_uri, p=PROV_WAS_DERIVED_FROM, o=parent_uri),
]
``` ```
### 4. Type Definitions Each sub-agent runs its own session with `wasDerivedFrom` linking back to the parent's Decomposition. Findings derive from their sub-agent's Conclusion.
**File:** `trustgraph-base/trustgraph/provenance/namespaces.py` ## Predicates
Add explainability entity types and agent predicates: ### Session / Question
```python | Predicate | Type | Description |
# Explainability entity types (used by both GraphRAG and Agent) |-----------|------|-------------|
TG_QUESTION = TG + "Question" | `tg:query` | string | The user's query text |
TG_EXPLORATION = TG + "Exploration" | `prov:startedAtTime` | string | ISO timestamp |
TG_FOCUS = TG + "Focus"
TG_SYNTHESIS = TG + "Synthesis"
TG_ANALYSIS = TG + "Analysis"
TG_CONCLUSION = TG + "Conclusion"
# Agent predicates ### Pattern Decision
TG_THOUGHT = TG + "thought" | Predicate | Type | Description |
TG_ACTION = TG + "action" |-----------|------|-------------|
TG_ARGUMENTS = TG + "arguments" | `tg:pattern` | string | Selected pattern (react, plan-then-execute, supervisor) |
TG_OBSERVATION = TG + "observation" | `tg:taskType` | string | Identified task type (general, research, etc.) |
TG_ANSWER = TG + "answer"
```
## Files Modified ### Analysis (Iteration)
| Predicate | Type | Description |
|-----------|------|-------------|
| `tg:action` | string | Tool name selected by the agent |
| `tg:arguments` | string | JSON-encoded arguments |
| `tg:thought` | IRI | Link to Thought sub-entity |
| `tg:toolCandidate` | string | Tool name available to the LLM (one per candidate) |
| `tg:stepNumber` | integer | 1-based iteration counter |
| `tg:llmDurationMs` | integer | LLM call duration in milliseconds |
| `tg:inToken` | integer | Input token count |
| `tg:outToken` | integer | Output token count |
| `tg:llmModel` | string | Model identifier |
| File | Change | ### Observation
|------|--------| | Predicate | Type | Description |
| `trustgraph-base/trustgraph/schema/services/agent.py` | Add session_id and collection to AgentRequest | |-----------|------|-------------|
| `trustgraph-base/trustgraph/messaging/translators/agent.py` | Update translator for new fields | | `tg:document` | IRI | Librarian document reference |
| `trustgraph-base/trustgraph/provenance/namespaces.py` | Add entity types, agent predicates, and Document RAG predicates | | `tg:toolDurationMs` | integer | Tool execution time in milliseconds |
| `trustgraph-base/trustgraph/provenance/triples.py` | Add TG types to GraphRAG triple builders, add Document RAG triple builders | | `tg:toolError` | string | Error message (tool failure or LLM parse error) |
| `trustgraph-base/trustgraph/provenance/uris.py` | Add Document RAG URI generators |
| `trustgraph-base/trustgraph/provenance/__init__.py` | Export new types, predicates, and Document RAG functions |
| `trustgraph-base/trustgraph/schema/services/retrieval.py` | Add explain_id, explain_graph, and explain_triples to DocumentRagResponse |
| `trustgraph-base/trustgraph/messaging/translators/retrieval.py` | Update DocumentRagResponseTranslator for explainability fields including inline triples |
| `trustgraph-flow/trustgraph/agent/react/service.py` | Add explainability producer + recording logic |
| `trustgraph-flow/trustgraph/retrieval/document_rag/document_rag.py` | Add explainability callback and emit provenance triples |
| `trustgraph-flow/trustgraph/retrieval/document_rag/rag.py` | Add explainability producer and wire up callback |
| `trustgraph-cli/trustgraph/cli/show_explain_trace.py` | Handle agent trace types |
| `trustgraph-cli/trustgraph/cli/list_explain_traces.py` | List agent sessions alongside GraphRAG |
## Files Created When `tg:toolError` is present, the Observation also carries the `tg:Error` mixin type.
| File | Purpose | ### Conclusion / Synthesis
|------|---------| | Predicate | Type | Description |
| `trustgraph-base/trustgraph/provenance/agent.py` | Agent-specific triple generators | |-----------|------|-------------|
| `tg:document` | IRI | Librarian document reference |
| `tg:terminationReason` | string | Why the loop stopped |
| `tg:inToken` | integer | Input token count (synthesis LLM call) |
| `tg:outToken` | integer | Output token count |
| `tg:llmModel` | string | Model identifier |
## CLI Updates Termination reason values:
- `final-answer` -- LLM produced a confident answer (react)
- `plan-complete` -- all plan steps executed (plan-then-execute)
- `subagents-complete` -- all sub-agents reported back (supervisor)
**Detection:** Both GraphRAG and Agent Questions have `tg:Question` type. Distinguished by: ### Decomposition
1. URI pattern: `urn:trustgraph:agent:` vs `urn:trustgraph:question:` | Predicate | Type | Description |
2. Derived entities: `tg:Analysis` (agent) vs `tg:Exploration` (GraphRAG) |-----------|------|-------------|
| `tg:subagentGoal` | string | Goal assigned to a sub-agent (one per goal) |
| `tg:inToken` | integer | Input token count |
| `tg:outToken` | integer | Output token count |
**`list_explain_traces.py`:** ### Plan
- Shows Type column (Agent vs GraphRAG) | Predicate | Type | Description |
|-----------|------|-------------|
| `tg:planStep` | string | Goal for a plan step (one per step) |
| `tg:inToken` | integer | Input token count |
| `tg:outToken` | integer | Output token count |
**`show_explain_trace.py`:** ### Token Counts on RAG Events
- Auto-detects trace type
- Agent rendering shows: Question → Analysis step(s) → Conclusion
## Backwards Compatibility Grounding, Focus, and Synthesis events on GraphRAG and Document RAG also carry `tg:inToken`, `tg:outToken`, and `tg:llmModel` for the LLM calls associated with that step.
- `session_id` defaults to `""` - old requests work, just won't have provenance ## Error Handling
- `collection` defaults to `"default"` - reasonable fallback
- CLI gracefully handles both trace types Tool execution errors and LLM parse errors are captured as Observation events rather than crashing the agent:
- The error message is recorded on `tg:toolError`
- The Observation carries the `tg:Error` mixin type
- The error text becomes the observation content, visible to the LLM on the next iteration
- The provenance chain is preserved (Observation derives from Analysis)
- The agent gets another iteration to retry or choose a different approach
## Vocabulary Reference
The full OWL ontology covering all classes and predicates is at `specs/ontology/trustgraph.ttl`.
## Verification ## Verification
```bash ```bash
# Run an agent query # Run an agent query with explainability
tg-invoke-agent -q "What is the capital of France?" tg-invoke-agent -q "What is quantum computing?" -x
# List traces (should show agent sessions with Type column) # Run with token usage
tg-list-explain-traces -U trustgraph -C default tg-invoke-agent -q "What is quantum computing?" --show-usage
# Show agent trace # GraphRAG with explainability
tg-show-explain-trace "urn:trustgraph:agent:xxx" tg-invoke-graph-rag -q "Tell me about AI" -x
# Document RAG with explainability
tg-invoke-document-rag -q "Summarize the findings" -x
``` ```
## Future Work (Not This PR)
- DAG dependencies (when analysis N uses results from multiple prior analyses)
- Tool-specific provenance linking (KnowledgeQuery → its GraphRAG trace)
- Streaming provenance emission (emit as we go, not batch at end)

2
docs/tech-specs/agent-orchestration.md
View file

@ -862,7 +862,7 @@ independently.
Response chunk fields: Response chunk fields:
message_id UUID for this message (groups chunks) message_id UUID for this message (groups chunks)
session_id Which agent session produced this chunk session_id Which agent session produced this chunk
chunk_type "thought" | "observation" | "answer" | ... message_type "thought" | "observation" | "answer" | ...
content The chunk text content The chunk text
end_of_message True on the final chunk of this message end_of_message True on the final chunk of this message
end_of_dialog True on the final message of the entire execution end_of_dialog True on the final message of the entire execution

4
docs/tech-specs/extraction-provenance-subgraph.md
View file

@ -203,3 +203,7 @@ def subgraph_provenance_triples(
This is a breaking change to the provenance model. Provenance has not This is a breaking change to the provenance model. Provenance has not
been released, so no migration is needed. The old `tg:reifies` / been released, so no migration is needed. The old `tg:reifies` /
`statement_uri` code can be removed outright. `statement_uri` code can be removed outright.
## Vocabulary Reference
The full OWL ontology covering all extraction and query-time classes and predicates is at `specs/ontology/trustgraph.ttl`.

7
docs/tech-specs/extraction-time-provenance.md
View file

@ -612,8 +612,13 @@ Link embedding entity IDs to chunk.
| Triple store | Use reification for triple → chunk provenance | | Triple store | Use reification for triple → chunk provenance |
| Embedding provenance | Link entity ID → chunk ID | | Embedding provenance | Link entity ID → chunk ID |
## Vocabulary Reference
The full OWL ontology covering all extraction and query-time classes and predicates is at `specs/ontology/trustgraph.ttl`.
## References ## References
- Query-time provenance: `docs/tech-specs/query-time-provenance.md` - Query-time provenance: `docs/tech-specs/query-time-explainability.md`
- Agent explainability: `docs/tech-specs/agent-explainability.md`
- PROV-O standard for provenance modeling - PROV-O standard for provenance modeling
- Existing source metadata in knowledge graph (needs audit) - Existing source metadata in knowledge graph (needs audit)

6
docs/tech-specs/python-api-refactor.md
View file

@ -704,17 +704,17 @@ class StreamingChunk:
@dataclasses.dataclass @dataclasses.dataclass
class AgentThought(StreamingChunk): class AgentThought(StreamingChunk):
"""Agent reasoning chunk""" """Agent reasoning chunk"""
chunk_type: str = "thought" message_type: str = "thought"
@dataclasses.dataclass @dataclasses.dataclass
class AgentObservation(StreamingChunk): class AgentObservation(StreamingChunk):
"""Agent tool observation chunk""" """Agent tool observation chunk"""
chunk_type: str = "observation" message_type: str = "observation"
@dataclasses.dataclass @dataclasses.dataclass
class AgentAnswer(StreamingChunk): class AgentAnswer(StreamingChunk):
"""Agent final answer chunk""" """Agent final answer chunk"""
chunk_type: str = "final-answer" message_type: str = "final-answer"
end_of_dialog: bool = False end_of_dialog: bool = False
@dataclasses.dataclass @dataclasses.dataclass

24
docs/tech-specs/query-time-explainability.md
View file

@ -138,6 +138,25 @@ Defined in `trustgraph-base/trustgraph/provenance/namespaces.py`:
| `TG_REASONING` | `https://trustgraph.ai/ns/reasoning` | | `TG_REASONING` | `https://trustgraph.ai/ns/reasoning` |
| `TG_CONTENT` | `https://trustgraph.ai/ns/content` | | `TG_CONTENT` | `https://trustgraph.ai/ns/content` |
| `TG_DOCUMENT` | `https://trustgraph.ai/ns/document` | | `TG_DOCUMENT` | `https://trustgraph.ai/ns/document` |
| `TG_IN_TOKEN` | `https://trustgraph.ai/ns/inToken` |
| `TG_OUT_TOKEN` | `https://trustgraph.ai/ns/outToken` |
| `TG_LLM_MODEL` | `https://trustgraph.ai/ns/llmModel` |
### Token Usage on Events
Grounding, Focus, and Synthesis events carry per-event LLM token counts:
| Predicate | Type | Present on |
|-----------|------|------------|
| `tg:inToken` | integer | Grounding, Focus, Synthesis |
| `tg:outToken` | integer | Grounding, Focus, Synthesis |
| `tg:llmModel` | string | Grounding, Focus, Synthesis |
- **Grounding**: tokens from the extract-concepts LLM call
- **Focus**: summed tokens from edge-scoring + edge-reasoning LLM calls
- **Synthesis**: tokens from the synthesis LLM call
Values are absent (not zero) when token counts are unavailable.
## GraphRagResponse Schema ## GraphRagResponse Schema
@ -261,8 +280,13 @@ Based on the provided knowledge statements...
| `trustgraph-flow/trustgraph/query/triples/cassandra/service.py` | Quoted triple query support | | `trustgraph-flow/trustgraph/query/triples/cassandra/service.py` | Quoted triple query support |
| `trustgraph-cli/trustgraph/cli/invoke_graph_rag.py` | CLI with explainability display | | `trustgraph-cli/trustgraph/cli/invoke_graph_rag.py` | CLI with explainability display |
## Vocabulary Reference
The full OWL ontology covering all classes and predicates is at `specs/ontology/trustgraph.ttl`.
## References ## References
- PROV-O (W3C Provenance Ontology): https://www.w3.org/TR/prov-o/ - PROV-O (W3C Provenance Ontology): https://www.w3.org/TR/prov-o/
- RDF-star: https://w3c.github.io/rdf-star/ - RDF-star: https://w3c.github.io/rdf-star/
- Extraction-time provenance: `docs/tech-specs/extraction-time-provenance.md` - Extraction-time provenance: `docs/tech-specs/extraction-time-provenance.md`
- Agent explainability: `docs/tech-specs/agent-explainability.md`

34
docs/tech-specs/streaming-llm-responses.md
View file

@ -393,28 +393,28 @@ The agent produces multiple types of output during its reasoning cycle:
- Answer (final response) - Answer (final response)
- Errors - Errors
Since `chunk_type` identifies what kind of content is being sent, the separate Since `message_type` identifies what kind of content is being sent, the separate
`answer`, `error`, `thought`, and `observation` fields can be collapsed into `answer`, `error`, `thought`, and `observation` fields can be collapsed into
a single `content` field: a single `content` field:
```python ```python
class AgentResponse(Record): class AgentResponse(Record):
chunk_type = String() # "thought", "action", "observation", "answer", "error" message_type = String() # "thought", "action", "observation", "answer", "error"
content = String() # The actual content (interpretation depends on chunk_type) content = String() # The actual content (interpretation depends on message_type)
end_of_message = Boolean() # Current thought/action/observation/answer is complete end_of_message = Boolean() # Current thought/action/observation/answer is complete
end_of_dialog = Boolean() # Entire agent dialog is complete end_of_dialog = Boolean() # Entire agent dialog is complete
``` ```
**Field Semantics:** **Field Semantics:**
- `chunk_type`: Indicates what type of content is in the `content` field - `message_type`: Indicates what type of content is in the `content` field
- `"thought"`: Agent reasoning/thinking - `"thought"`: Agent reasoning/thinking
- `"action"`: Tool/action being invoked - `"action"`: Tool/action being invoked
- `"observation"`: Result from tool execution - `"observation"`: Result from tool execution
- `"answer"`: Final answer to the user's question - `"answer"`: Final answer to the user's question
- `"error"`: Error message - `"error"`: Error message
- `content`: The actual streamed content, interpreted based on `chunk_type` - `content`: The actual streamed content, interpreted based on `message_type`
- `end_of_message`: When `true`, the current chunk type is complete - `end_of_message`: When `true`, the current chunk type is complete
- Example: All tokens for the current thought have been sent - Example: All tokens for the current thought have been sent
@ -428,27 +428,27 @@ class AgentResponse(Record):
When `streaming=true`: When `streaming=true`:
1. **Thought streaming**: 1. **Thought streaming**:
- Multiple chunks with `chunk_type="thought"`, `end_of_message=false` - Multiple chunks with `message_type="thought"`, `end_of_message=false`
- Final thought chunk has `end_of_message=true` - Final thought chunk has `end_of_message=true`
2. **Action notification**: 2. **Action notification**:
- Single chunk with `chunk_type="action"`, `end_of_message=true` - Single chunk with `message_type="action"`, `end_of_message=true`
3. **Observation**: 3. **Observation**:
- Chunk(s) with `chunk_type="observation"`, final has `end_of_message=true` - Chunk(s) with `message_type="observation"`, final has `end_of_message=true`
4. **Repeat** steps 1-3 as the agent reasons 4. **Repeat** steps 1-3 as the agent reasons
5. **Final answer**: 5. **Final answer**:
- `chunk_type="answer"` with the final response in `content` - `message_type="answer"` with the final response in `content`
- Last chunk has `end_of_message=true`, `end_of_dialog=true` - Last chunk has `end_of_message=true`, `end_of_dialog=true`
**Example Stream Sequence:** **Example Stream Sequence:**
``` ```
{chunk_type: "thought", content: "I need to", end_of_message: false, end_of_dialog: false} {message_type: "thought", content: "I need to", end_of_message: false, end_of_dialog: false}
{chunk_type: "thought", content: " search for...", end_of_message: true, end_of_dialog: false} {message_type: "thought", content: " search for...", end_of_message: true, end_of_dialog: false}
{chunk_type: "action", content: "search", end_of_message: true, end_of_dialog: false} {message_type: "action", content: "search", end_of_message: true, end_of_dialog: false}
{chunk_type: "observation", content: "Found: ...", end_of_message: true, end_of_dialog: false} {message_type: "observation", content: "Found: ...", end_of_message: true, end_of_dialog: false}
{chunk_type: "thought", content: "Based on this", end_of_message: false, end_of_dialog: false} {message_type: "thought", content: "Based on this", end_of_message: false, end_of_dialog: false}
{chunk_type: "thought", content: " I can answer...", end_of_message: true, end_of_dialog: false} {message_type: "thought", content: " I can answer...", end_of_message: true, end_of_dialog: false}
{chunk_type: "answer", content: "The answer is...", end_of_message: true, end_of_dialog: true} {message_type: "answer", content: "The answer is...", end_of_message: true, end_of_dialog: true}
``` ```
When `streaming=false`: When `streaming=false`:
@ -541,7 +541,7 @@ The following questions were resolved during specification:
populated and no other fields are needed. An error is always the final populated and no other fields are needed. An error is always the final
communication - no subsequent messages are permitted or expected after communication - no subsequent messages are permitted or expected after
an error. For LLM/Prompt streams, `end_of_stream=true`. For Agent streams, an error. For LLM/Prompt streams, `end_of_stream=true`. For Agent streams,
`chunk_type="error"` with `end_of_dialog=true`. `message_type="error"` with `end_of_dialog=true`.
3. **Partial Response Recovery**: The messaging protocol (Pulsar) is resilient, 3. **Partial Response Recovery**: The messaging protocol (Pulsar) is resilient,
so message-level retry is not needed. If a client loses track of the stream so message-level retry is not needed. If a client loses track of the stream