apunkt/trustgraph
1
0
Fork 0
mirror of https://github.com/trustgraph-ai/trustgraph.git synced 2026-05-25 15:25:12 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

Release/v1.2 (#457)

Browse source 
* Bump setup.py versions for 1.1

* PoC MCP server (#419)

* Very initial MCP server PoC for TrustGraph

* Put service on port 8000

* Add MCP container and packages to buildout

* Update docs for API/CLI changes in 1.0 (#421)

* Update some API basics for the 0.23/1.0 API change

* Add MCP container push (#425)

* Add command args to the MCP server (#426)

* Host and port parameters

* Added websocket arg

* More docs

* MCP client support (#427)

- MCP client service
- Tool request/response schema
- API gateway support for mcp-tool
- Message translation for tool request & response
- Make mcp-tool using configuration service for information
  about where the MCP services are.

* Feature/react call mcp (#428)

Key Features

  - MCP Tool Integration: Added core MCP tool support with ToolClientSpec and ToolClient classes
  - API Enhancement: New mcp_tool method for flow-specific tool invocation
  - CLI Tooling: New tg-invoke-mcp-tool command for testing MCP integration
  - React Agent Enhancement: Fixed and improved multi-tool invocation capabilities
  - Tool Management: Enhanced CLI for tool configuration and management

Changes

  - Added MCP tool invocation to API with flow-specific integration
  - Implemented ToolClientSpec and ToolClient for tool call handling
  - Updated agent-manager-react to invoke MCP tools with configurable types
  - Enhanced CLI with new commands and improved help text
  - Added comprehensive documentation for new CLI commands
  - Improved tool configuration management

Testing

  - Added tg-invoke-mcp-tool CLI command for isolated MCP integration testing
  - Enhanced agent capability to invoke multiple tools simultaneously

* Test suite executed from CI pipeline (#433)

* Test strategy & test cases

* Unit tests

* Integration tests

* Extending test coverage (#434)

* Contract tests

* Testing embeedings

* Agent unit tests

* Knowledge pipeline tests

* Turn on contract tests

* Increase storage test coverage (#435)

* Fixing storage and adding tests

* PR pipeline only runs quick tests

* Empty configuration is returned as empty list, previously was not in response (#436)

* Update config util to take files as well as command-line text (#437)

* Updated CLI invocation and config model for tools and mcp (#438)

* Updated CLI invocation and config model for tools and mcp

* CLI anomalies

* Tweaked the MCP tool implementation for new model

* Update agent implementation to match the new model

* Fix agent tools, now all tested

* Fixed integration tests

* Fix MCP delete tool params

* Update Python deps to 1.2

* Update to enable knowledge extraction using the agent framework (#439)

* Implement KG extraction agent (kg-extract-agent)

* Using ReAct framework (agent-manager-react)
 
* ReAct manager had an issue when emitting JSON, which conflicts which ReAct manager's own JSON messages, so refactored ReAct manager to use traditional ReAct messages, non-JSON structure.
 
* Minor refactor to take the prompt template client out of prompt-template so it can be more readily used by other modules. kg-extract-agent uses this framework.

* Migrate from setup.py to pyproject.toml (#440)

* Converted setup.py to pyproject.toml

* Modern package infrastructure as recommended by py docs

* Install missing build deps (#441)

* Install missing build deps (#442)

* Implement logging strategy (#444)

* Logging strategy and convert all prints() to logging invocations

* Fix/startup failure (#445)

* Fix loggin startup problems

* Fix logging startup problems (#446)

* Fix logging startup problems (#447)

* Fixed Mistral OCR to use current API (#448)

* Fixed Mistral OCR to use current API

* Added PDF decoder tests

* Fix Mistral OCR ident to be standard pdf-decoder (#450)

* Fix Mistral OCR ident to be standard pdf-decoder

* Correct test

* Schema structure refactor (#451)

* Write schema refactor spec

* Implemented schema refactor spec

* Structure data mvp (#452)

* Structured data tech spec

* Architecture principles

* New schemas

* Updated schemas and specs

* Object extractor

* Add .coveragerc

* New tests

* Cassandra object storage

* Trying to object extraction working, issues exist

* Validate librarian collection (#453)

* Fix token chunker, broken API invocation (#454)

* Fix token chunker, broken API invocation (#455)

* Knowledge load utility CLI (#456)

* Knowledge loader

* More tests
This commit is contained in:
cybermaggedon 2025-08-18 20:56:09 +01:00 • committed by GitHub
parent c85ba197be
commit 89be656990
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
509 changed files with 49632 additions and 5159 deletions
Download patch file Download diff file Expand all files Collapse all files

49
docs/apis/api-flow.md
View file

@ -210,6 +210,51 @@ Request schema:
Response schema:
`trustgraph.schema.FlowResponse`
## Flow Service Methods
Flow instances provide access to various TrustGraph services through flow-specific endpoints:
### MCP Tool Service - Invoke MCP Tools
The `mcp_tool` method allows invoking MCP (Model Control Protocol) tools within a flow context.
Request:
```json
{
"name": "file-reader",
"parameters": {
"path": "/path/to/file.txt"
}
}
```
Response:
```json
{
"object": {"content": "file contents here", "size": 1024}
}
```
Or for text responses:
```json
{
"text": "plain text response"
}
```
### Other Service Methods
Flow instances also provide access to:
- `text_completion` - LLM text completion
- `agent` - Agent question answering
- `graph_rag` - Graph-based RAG queries
- `document_rag` - Document-based RAG queries
- `embeddings` - Text embeddings
- `prompt` - Prompt template processing
- `triples_query` - Knowledge graph queries
- `load_document` - Document loading
- `load_text` - Text loading
## Python SDK
The Python SDK provides convenient access to the Flow API:
@ -233,6 +278,10 @@ flows = await client.list_flows()
# Stop a flow instance
await client.stop_flow("flow-123")
# Use flow instance services
flow = client.id("flow-123")
result = await flow.mcp_tool("file-reader", {"path": "/path/to/file.txt"})
```
## Features

57
docs/apis/api-librarian.md
View file

@ -12,6 +12,17 @@ The request contains the following fields:
- `operation`: The operation to perform (see operations below)
- `document_id`: Document identifier (for document operations)
- `document_metadata`: Document metadata object (for add/update operations)
- `id`: Document identifier (required)
- `time`: Unix timestamp in seconds as a float (required for add operations)
- `kind`: MIME type of document (required, e.g., "text/plain", "application/pdf")
- `title`: Document title (optional)
- `comments`: Document comments (optional)
- `user`: Document owner (required)
- `tags`: Array of tags (optional)
- `metadata`: Array of RDF triples (optional) - each triple has:
- `s`: Subject with `v` (value) and `e` (is_uri boolean)
- `p`: Predicate with `v` (value) and `e` (is_uri boolean)
- `o`: Object with `v` (value) and `e` (is_uri boolean)
- `content`: Document content as base64-encoded bytes (for add operations)
- `processing_id`: Processing job identifier (for processing operations)
- `processing_metadata`: Processing metadata object (for add-processing)
@ -38,7 +49,7 @@ Request:
"operation": "add-document",
"document_metadata": {
"id": "doc-123",
"time": 1640995200000,
"time": 1640995200.0,
"kind": "application/pdf",
"title": "Research Paper",
"comments": "Important research findings",
@ -46,9 +57,18 @@ Request:
"tags": ["research", "ai", "machine-learning"],
"metadata": [
{
"subject": "doc-123",
"predicate": "dc:creator",
"object": "Dr. Smith"
"s": {
"v": "http://example.com/doc-123",
"e": true
},
"p": {
"v": "http://purl.org/dc/elements/1.1/creator",
"e": true
},
"o": {
"v": "Dr. Smith",
"e": false
}
}
]
},
@ -77,7 +97,7 @@ Response:
{
"document_metadata": {
"id": "doc-123",
"time": 1640995200000,
"time": 1640995200.0,
"kind": "application/pdf",
"title": "Research Paper",
"comments": "Important research findings",
@ -85,9 +105,18 @@ Response:
"tags": ["research", "ai", "machine-learning"],
"metadata": [
{
"subject": "doc-123",
"predicate": "dc:creator",
"object": "Dr. Smith"
"s": {
"v": "http://example.com/doc-123",
"e": true
},
"p": {
"v": "http://purl.org/dc/elements/1.1/creator",
"e": true
},
"o": {
"v": "Dr. Smith",
"e": false
}
}
]
}
@ -129,7 +158,7 @@ Response:
"document_metadatas": [
{
"id": "doc-123",
"time": 1640995200000,
"time": 1640995200.0,
"kind": "application/pdf",
"title": "Research Paper",
"comments": "Important research findings",
@ -138,7 +167,7 @@ Response:
},
{
"id": "doc-124",
"time": 1640995300000,
"time": 1640995300.0,
"kind": "text/plain",
"title": "Meeting Notes",
"comments": "Team meeting discussion",
@ -157,10 +186,12 @@ Request:
"operation": "update-document",
"document_metadata": {
"id": "doc-123",
"time": 1640995500.0,
"title": "Updated Research Paper",
"comments": "Updated findings and conclusions",
"user": "alice",
"tags": ["research", "ai", "machine-learning", "updated"]
"tags": ["research", "ai", "machine-learning", "updated"],
"metadata": []
}
}
```
@ -197,7 +228,7 @@ Request:
"processing_metadata": {
"id": "proc-456",
"document_id": "doc-123",
"time": 1640995400000,
"time": 1640995400.0,
"flow": "pdf-extraction",
"user": "alice",
"collection": "research",
@ -229,7 +260,7 @@ Response:
{
"id": "proc-456",
"document_id": "doc-123",
"time": 1640995400000,
"time": 1640995400.0,
"flow": "pdf-extraction",
"user": "alice",
"collection": "research",

137
docs/apis/api-mcp-tool.md Normal file
View file

@ -0,0 +1,137 @@
# TrustGraph MCP Tool API
This is a higher-level interface to the MCP (Model Control Protocol) tool service. The input
specifies an MCP tool by name and parameters to pass to the tool.
## Request/response
### Request
The request contains the following fields:
- `name`: The MCP tool name
- `parameters`: A set of key/values describing the tool parameters
### Response
The response contains either of these fields:
- `text`: A plain text response
- `object`: A structured object response
## REST service
The REST service accepts `name` and `parameters` fields, with parameters
encoded as a JSON object.
e.g.
In this example, the MCP tool takes parameters and returns a
structured response in the `object` field.
Request:
```
{
"name": "file-reader",
"parameters": {
"path": "/path/to/file.txt"
}
}
```
Response:
```
{
"object": {"content": "file contents here", "size": 1024}
}
```
## Websocket
Requests have `name` and `parameters` fields.
e.g.
Request:
```
{
"id": "akshfkiehfkseffh-142",
"service": "mcp-tool",
"flow": "default",
"request": {
"name": "file-reader",
"parameters": {
"path": "/path/to/file.txt"
}
}
}
```
Responses:
```
{
"id": "akshfkiehfkseffh-142",
"response": {
"object": {"content": "file contents here", "size": 1024}
},
"complete": true
}
```
e.g.
An example which returns plain text
Request:
```
{
"id": "akshfkiehfkseffh-141",
"service": "mcp-tool",
"request": {
"name": "calculator",
"parameters": {
"expression": "2 + 2"
}
}
}
```
Response:
```
{
"id": "akshfkiehfkseffh-141",
"response": {
"text": "4"
},
"complete": true
}
```
## Pulsar
The Pulsar schema for the MCP Tool API is defined in Python code here:
https://github.com/trustgraph-ai/trustgraph/blob/master/trustgraph-base/trustgraph/schema/mcp_tool.py
Default request queue:
`non-persistent://tg/request/mcp-tool`
Default response queue:
`non-persistent://tg/response/mcp-tool`
Request schema:
`trustgraph.schema.McpToolRequest`
Response schema:
`trustgraph.schema.McpToolResponse`
## Pulsar Python client
The client class is
`trustgraph.clients.McpToolClient`
https://github.com/trustgraph-ai/trustgraph/blob/master/trustgraph-base/trustgraph/clients/mcp_tool_client.py

374
docs/cli/tg-delete-mcp-tool.md Normal file
View file

@ -0,0 +1,374 @@
# tg-delete-mcp-tool
## Synopsis
```
tg-delete-mcp-tool [OPTIONS] --name NAME
```
## Description
The `tg-delete-mcp-tool` command deletes MCP (Model Control Protocol) tools from the TrustGraph system. It removes MCP tool configurations by name from the 'mcp' configuration group. Once deleted, MCP tools are no longer available for agent use.
This command is useful for:
- Removing obsolete or deprecated MCP tools
- Cleaning up MCP tool configurations
- Managing MCP tool registry maintenance
- Updating MCP tool deployments by removing old versions
The command removes MCP tool configurations from the 'mcp' configuration group in the TrustGraph API.
## Options
- `-u, --api-url URL`
- TrustGraph API URL for configuration management
- Default: `http://localhost:8088/` (or `TRUSTGRAPH_URL` environment variable)
- Should point to a running TrustGraph API instance
- `--name NAME`
- **Required.** MCP tool name to delete
- Must match an existing MCP tool name in the registry
- MCP tool will be completely removed from the system
- `-h, --help`
- Show help message and exit
## Examples
### Basic MCP Tool Deletion
Delete a weather MCP tool:
```bash
tg-delete-mcp-tool --name weather
```
### Calculator MCP Tool Deletion
Delete a calculator MCP tool:
```bash
tg-delete-mcp-tool --name calculator
```
### Custom API URL
Delete an MCP tool from a specific TrustGraph instance:
```bash
tg-delete-mcp-tool --api-url http://trustgraph.example.com:8088/ --name custom-mcp
```
### Batch MCP Tool Deletion
Delete multiple MCP tools in a script:
```bash
#!/bin/bash
# Delete obsolete MCP tools
tg-delete-mcp-tool --name old-search
tg-delete-mcp-tool --name deprecated-calc
tg-delete-mcp-tool --name unused-mcp
```
### Conditional Deletion
Delete an MCP tool only if it exists:
```bash
#!/bin/bash
# Check if MCP tool exists before deletion
if tg-show-mcp-tools | grep -q "test-mcp"; then
tg-delete-mcp-tool --name test-mcp
echo "MCP tool deleted"
else
echo "MCP tool not found"
fi
```
## Deletion Process
The deletion process involves:
1. **Existence Check**: Verify the MCP tool exists in the configuration
2. **Configuration Removal**: Delete the MCP tool configuration from the 'mcp' group
The command performs validation before deletion to ensure the tool exists.
## Error Handling
The command handles various error conditions:
- **Tool not found**: If the specified MCP tool name doesn't exist
- **API connection errors**: If the TrustGraph API is unavailable
- **Configuration errors**: If the MCP tool configuration cannot be removed
Common error scenarios:
```bash
# MCP tool not found
tg-delete-mcp-tool --name nonexistent-mcp
# Output: MCP tool 'nonexistent-mcp' not found.
# Missing required field
tg-delete-mcp-tool
# Output: Exception: Must specify --name for MCP tool to delete
# API connection error
tg-delete-mcp-tool --api-url http://invalid-host:8088/ --name tool1
# Output: Exception: [Connection error details]
```
## Verification
The command provides feedback on the deletion process:
- **Success**: `MCP tool 'tool-name' deleted successfully.`
- **Not found**: `MCP tool 'tool-name' not found.`
- **Error**: `Error deleting MCP tool 'tool-name': [error details]`
## Advanced Usage
### Safe Deletion with Verification
Verify MCP tool exists before deletion:
```bash
#!/bin/bash
MCP_NAME="weather"
# Check if MCP tool exists
if tg-show-mcp-tools | grep -q "^$MCP_NAME"; then
echo "Deleting MCP tool: $MCP_NAME"
tg-delete-mcp-tool --name "$MCP_NAME"
# Verify deletion
if ! tg-show-mcp-tools | grep -q "^$MCP_NAME"; then
echo "MCP tool successfully deleted"
else
echo "MCP tool deletion failed"
fi
else
echo "MCP tool $MCP_NAME not found"
fi
```
### Backup Before Deletion
Backup MCP tool configuration before deletion:
```bash
#!/bin/bash
MCP_NAME="important-mcp"
# Export MCP tool configuration
echo "Backing up MCP tool configuration..."
tg-show-mcp-tools | grep -A 10 "^$MCP_NAME" > "${MCP_NAME}_backup.txt"
# Delete MCP tool
echo "Deleting MCP tool..."
tg-delete-mcp-tool --name "$MCP_NAME"
echo "MCP tool deleted, backup saved to ${MCP_NAME}_backup.txt"
```
### Cleanup Script
Clean up multiple MCP tools based on patterns:
```bash
#!/bin/bash
# Delete all test MCP tools
echo "Cleaning up test MCP tools..."
# Get list of test MCP tools
TEST_MCPS=$(tg-show-mcp-tools | grep "^test-" | cut -d: -f1)
for mcp in $TEST_MCPS; do
echo "Deleting $mcp..."
tg-delete-mcp-tool --name "$mcp"
done
echo "Cleanup complete"
```
### Environment-Specific Deletion
Delete MCP tools from specific environments:
```bash
#!/bin/bash
# Delete development MCP tools from production
export TRUSTGRAPH_URL="http://prod.trustgraph.com:8088/"
DEV_MCPS=("dev-mcp" "debug-mcp" "test-helper")
for mcp in "${DEV_MCPS[@]}"; do
echo "Removing development MCP tool: $mcp"
tg-delete-mcp-tool --name "$mcp"
done
```
### MCP Service Shutdown
Remove MCP tools when services are decommissioned:
```bash
#!/bin/bash
# Remove MCP tools for decommissioned service
SERVICE_NAME="old-service"
# Find MCP tools for this service
MCP_TOOLS=$(tg-show-mcp-tools | grep "$SERVICE_NAME" | cut -d: -f1)
for tool in $MCP_TOOLS; do
echo "Removing MCP tool for decommissioned service: $tool"
tg-delete-mcp-tool --name "$tool"
done
```
## Integration with Other Commands
### With MCP Tool Management
List and delete MCP tools:
```bash
# List all MCP tools
tg-show-mcp-tools
# Delete specific MCP tool
tg-delete-mcp-tool --name unwanted-mcp
# Verify deletion
tg-show-mcp-tools | grep unwanted-mcp
```
### With Configuration Management
Manage MCP tool configurations:
```bash
# View current configuration
tg-show-config
# Delete MCP tool
tg-delete-mcp-tool --name old-mcp
# View updated configuration
tg-show-config
```
### With MCP Tool Invocation
Ensure MCP tools can't be invoked after deletion:
```bash
# Delete MCP tool
tg-delete-mcp-tool --name deprecated-mcp
# Verify tool is no longer available
tg-invoke-mcp-tool --name deprecated-mcp
# Should fail with tool not found error
```
## Best Practices
1. **Verification**: Always verify MCP tool exists before deletion
2. **Backup**: Backup important MCP tool configurations before deletion
3. **Dependencies**: Check for MCP tool dependencies before deletion
4. **Service Coordination**: Coordinate with MCP service owners before deletion
5. **Testing**: Test system functionality after MCP tool deletion
6. **Documentation**: Document reasons for MCP tool deletion
7. **Gradual Removal**: Remove MCP tools gradually in production environments
8. **Monitoring**: Monitor for errors after MCP tool deletion
## Troubleshooting
### MCP Tool Not Found
If MCP tool deletion reports "not found":
1. Verify the MCP tool name is correct
2. Check MCP tool exists with `tg-show-mcp-tools`
3. Ensure you're connected to the correct TrustGraph instance
4. Check for case sensitivity in MCP tool name
### Deletion Errors
If deletion fails:
1. Check TrustGraph API connectivity
2. Verify API permissions
3. Check for configuration corruption
4. Retry the deletion operation
5. Check MCP service status
### Permission Errors
If deletion fails due to permissions:
1. Verify API access credentials
2. Check TrustGraph API permissions
3. Ensure proper authentication
4. Contact system administrator if needed
## Recovery
### Restore Deleted MCP Tool
If an MCP tool was accidentally deleted:
1. Use backup configuration if available
2. Re-register the MCP tool with `tg-set-mcp-tool`
3. Restore from version control if MCP tool definitions are tracked
4. Contact system administrator for recovery options
### Verify System State
After deletion, verify system state:
```bash
# Check MCP tool registry
tg-show-mcp-tools
# Verify no orphaned configurations
tg-show-config | grep "mcp\."
# Test MCP tool functionality
tg-invoke-mcp-tool --name remaining-tool
```
## MCP Tool Lifecycle
### Development to Production
Manage MCP tool lifecycle:
```bash
#!/bin/bash
# Promote MCP tool from dev to production
# Remove development version
tg-delete-mcp-tool --name dev-tool
# Add production version
tg-set-mcp-tool --name prod-tool --tool-url "http://prod.mcp.com/api"
```
### Version Management
Manage MCP tool versions:
```bash
#!/bin/bash
# Update MCP tool to new version
# Remove old version
tg-delete-mcp-tool --name tool-v1
# Add new version
tg-set-mcp-tool --name tool-v2 --tool-url "http://new.mcp.com/api"
```
## Security Considerations
When deleting MCP tools:
1. **Access Control**: Ensure proper authorization for deletion
2. **Audit Trail**: Log MCP tool deletions for security auditing
3. **Impact Assessment**: Assess security impact of tool removal
4. **Credential Cleanup**: Remove associated credentials if applicable
5. **Network Security**: Update firewall rules if MCP endpoints are no longer needed
## Related Commands
- [`tg-show-mcp-tools`](tg-show-mcp-tools.md) - Display registered MCP tools
- [`tg-set-mcp-tool`](tg-set-mcp-tool.md) - Configure and register MCP tools
- [`tg-invoke-mcp-tool`](tg-invoke-mcp-tool.md) - Execute MCP tools
- [`tg-delete-tool`](tg-delete-tool.md) - Delete regular agent tools
## See Also
- MCP Protocol Documentation
- TrustGraph MCP Integration Guide
- MCP Tool Management Manual

317
docs/cli/tg-delete-tool.md Normal file
View file

@ -0,0 +1,317 @@
# tg-delete-tool
## Synopsis
```
tg-delete-tool [OPTIONS] --id ID
```
## Description
The `tg-delete-tool` command deletes tools from the TrustGraph system. It removes tool configurations by ID from the agent configuration and updates the tool index accordingly. Once deleted, tools are no longer available for agent use.
This command is useful for:
- Removing obsolete or deprecated tools
- Cleaning up tool configurations
- Managing tool registry maintenance
- Updating tool deployments by removing old versions
The command removes both the tool from the tool index and deletes the complete tool configuration from the TrustGraph API.
## Options
- `-u, --api-url URL`
- TrustGraph API URL for configuration management
- Default: `http://localhost:8088/` (or `TRUSTGRAPH_URL` environment variable)
- Should point to a running TrustGraph API instance
- `--id ID`
- **Required.** Tool ID to delete
- Must match an existing tool ID in the registry
- Tool will be completely removed from the system
- `-h, --help`
- Show help message and exit
## Examples
### Basic Tool Deletion
Delete a weather tool:
```bash
tg-delete-tool --id weather
```
### Calculator Tool Deletion
Delete a calculator tool:
```bash
tg-delete-tool --id calculator
```
### Custom API URL
Delete a tool from a specific TrustGraph instance:
```bash
tg-delete-tool --api-url http://trustgraph.example.com:8088/ --id custom-tool
```
### Batch Tool Deletion
Delete multiple tools in a script:
```bash
#!/bin/bash
# Delete obsolete tools
tg-delete-tool --id old-search
tg-delete-tool --id deprecated-calc
tg-delete-tool --id unused-tool
```
### Conditional Deletion
Delete a tool only if it exists:
```bash
#!/bin/bash
# Check if tool exists before deletion
if tg-show-tools | grep -q "test-tool"; then
tg-delete-tool --id test-tool
echo "Tool deleted"
else
echo "Tool not found"
fi
```
## Deletion Process
The deletion process involves two steps:
1. **Index Update**: Remove the tool ID from the tool index
2. **Configuration Removal**: Delete the tool configuration data
Both operations must succeed for the deletion to be complete.
## Error Handling
The command handles various error conditions:
- **Tool not found**: If the specified tool ID doesn't exist
- **Missing configuration**: If tool is in index but configuration is missing
- **API connection errors**: If the TrustGraph API is unavailable
- **Partial deletion**: If index update or configuration removal fails
Common error scenarios:
```bash
# Tool not found
tg-delete-tool --id nonexistent-tool
# Output: Tool 'nonexistent-tool' not found in tool index.
# Missing required field
tg-delete-tool
# Output: Exception: Must specify --id for tool to delete
# API connection error
tg-delete-tool --api-url http://invalid-host:8088/ --id tool1
# Output: Exception: [Connection error details]
```
## Verification
The command provides feedback on the deletion process:
- **Success**: `Tool 'tool-id' deleted successfully.`
- **Not found**: `Tool 'tool-id' not found in tool index.`
- **Configuration missing**: `Tool configuration for 'tool-id' not found.`
- **Error**: `Error deleting tool 'tool-id': [error details]`
## Advanced Usage
### Safe Deletion with Verification
Verify tool exists before deletion:
```bash
#!/bin/bash
TOOL_ID="weather"
# Check if tool exists
if tg-show-tools | grep -q "^$TOOL_ID:"; then
echo "Deleting tool: $TOOL_ID"
tg-delete-tool --id "$TOOL_ID"
# Verify deletion
if ! tg-show-tools | grep -q "^$TOOL_ID:"; then
echo "Tool successfully deleted"
else
echo "Tool deletion failed"
fi
else
echo "Tool $TOOL_ID not found"
fi
```
### Backup Before Deletion
Backup tool configuration before deletion:
```bash
#!/bin/bash
TOOL_ID="important-tool"
# Export tool configuration
echo "Backing up tool configuration..."
tg-show-tools | grep -A 20 "^$TOOL_ID:" > "${TOOL_ID}_backup.txt"
# Delete tool
echo "Deleting tool..."
tg-delete-tool --id "$TOOL_ID"
echo "Tool deleted, backup saved to ${TOOL_ID}_backup.txt"
```
### Cleanup Script
Clean up multiple tools based on patterns:
```bash
#!/bin/bash
# Delete all test tools
echo "Cleaning up test tools..."
# Get list of test tools
TEST_TOOLS=$(tg-show-tools | grep "^test-" | cut -d: -f1)
for tool in $TEST_TOOLS; do
echo "Deleting $tool..."
tg-delete-tool --id "$tool"
done
echo "Cleanup complete"
```
### Environment-Specific Deletion
Delete tools from specific environments:
```bash
#!/bin/bash
# Delete development tools from production
export TRUSTGRAPH_URL="http://prod.trustgraph.com:8088/"
DEV_TOOLS=("dev-tool" "debug-tool" "test-helper")
for tool in "${DEV_TOOLS[@]}"; do
echo "Removing development tool: $tool"
tg-delete-tool --id "$tool"
done
```
## Integration with Other Commands
### With Tool Management
List and delete tools:
```bash
# List all tools
tg-show-tools
# Delete specific tool
tg-delete-tool --id unwanted-tool
# Verify deletion
tg-show-tools | grep unwanted-tool
```
### With Configuration Management
Manage tool configurations:
```bash
# View current configuration
tg-show-config
# Delete tool
tg-delete-tool --id old-tool
# View updated configuration
tg-show-config
```
### With Agent Workflows
Ensure agents don't use deleted tools:
```bash
# Delete tool
tg-delete-tool --id deprecated-tool
# Check agent configuration
tg-show-config | grep deprecated-tool
```
## Best Practices
1. **Verification**: Always verify tool exists before deletion
2. **Backup**: Backup important tool configurations before deletion
3. **Dependencies**: Check for tool dependencies before deletion
4. **Testing**: Test system functionality after tool deletion
5. **Documentation**: Document reasons for tool deletion
6. **Gradual Removal**: Remove tools gradually in production environments
7. **Monitoring**: Monitor for errors after tool deletion
## Troubleshooting
### Tool Not Found
If tool deletion reports "not found":
1. Verify the tool ID is correct
2. Check tool exists with `tg-show-tools`
3. Ensure you're connected to the correct TrustGraph instance
4. Check for case sensitivity in tool ID
### Partial Deletion
If deletion partially fails:
1. Check TrustGraph API connectivity
2. Verify API permissions
3. Check for configuration corruption
4. Retry the deletion operation
5. Manual cleanup may be required
### Permission Errors
If deletion fails due to permissions:
1. Verify API access credentials
2. Check TrustGraph API permissions
3. Ensure proper authentication
4. Contact system administrator if needed
## Recovery
### Restore Deleted Tool
If a tool was accidentally deleted:
1. Use backup configuration if available
2. Re-register the tool with `tg-set-tool`
3. Restore from version control if tool definitions are tracked
4. Contact system administrator for recovery options
### Verify System State
After deletion, verify system state:
```bash
# Check tool index consistency
tg-show-tools
# Verify no orphaned configurations
tg-show-config | grep "tool\."
# Test agent functionality
tg-invoke-agent --prompt "Test prompt"
```
## Related Commands
- [`tg-show-tools`](tg-show-tools.md) - Display registered tools
- [`tg-set-tool`](tg-set-tool.md) - Configure and register tools
- [`tg-delete-mcp-tool`](tg-delete-mcp-tool.md) - Delete MCP tools
- [`tg-show-config`](tg-show-config.md) - View system configuration
## See Also
- TrustGraph Tool Management Guide
- Agent Configuration Documentation
- System Administration Manual

448
docs/cli/tg-invoke-mcp-tool.md Normal file
View file

@ -0,0 +1,448 @@
# tg-invoke-mcp-tool
Invokes MCP (Model Control Protocol) tools through the TrustGraph API with parameter support.
## Synopsis
```bash
tg-invoke-mcp-tool [options] -n tool-name [-P parameters]
```
## Description
The `tg-invoke-mcp-tool` command invokes MCP (Model Control Protocol) tools through the TrustGraph API. MCP tools are external services that provide standardized interfaces for AI model interactions within the TrustGraph ecosystem.
MCP tools offer extensible functionality with consistent APIs, stateful interactions, and built-in security mechanisms. They can be used for various purposes including file operations, calculations, web requests, database queries, and custom integrations.
## Options
### Required Arguments
- `-n, --name TOOL_NAME`: MCP tool name to invoke
### Optional Arguments
- `-u, --url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
- `-f, --flow-id ID`: Flow instance ID to use (default: `default`)
- `-P, --parameters JSON`: Tool parameters as JSON-encoded dictionary
## Examples
### Basic Tool Invocation
```bash
tg-invoke-mcp-tool -n weather
```
### Tool with Parameters
```bash
tg-invoke-mcp-tool -n calculator -P '{"expression": "2 + 2"}'
```
### File Operations
```bash
tg-invoke-mcp-tool -n file-reader -P '{"path": "/path/to/file.txt"}'
```
### Web Request Tool
```bash
tg-invoke-mcp-tool -n http-client -P '{"url": "https://api.example.com/data", "method": "GET"}'
```
### Database Query
```bash
tg-invoke-mcp-tool -n database -P '{"query": "SELECT * FROM users LIMIT 10", "database": "main"}'
```
### Custom Flow and API URL
```bash
tg-invoke-mcp-tool -u http://custom-api:8088/ -f my-flow -n weather -P '{"location": "London"}'
```
## Parameter Format
### Simple Parameters
```bash
tg-invoke-mcp-tool -n calculator -P '{"operation": "add", "a": 10, "b": 5}'
```
### Complex Parameters
```bash
tg-invoke-mcp-tool -n data-processor -P '{
"input_data": [1, 2, 3, 4, 5],
"operations": ["sum", "average", "max"],
"output_format": "json"
}'
```
### File Input Parameters
```bash
tg-invoke-mcp-tool -n text-analyzer -P "{\"text\": \"$(cat document.txt)\", \"analysis_type\": \"sentiment\"}"
```
### Multiple Parameters
```bash
tg-invoke-mcp-tool -n report-generator -P '{
"template": "monthly-report",
"data_source": "sales_database",
"period": "2024-01",
"format": "pdf",
"recipients": ["admin@example.com"]
}'
```
## Common MCP Tools
### File Operations
```bash
# Read file content
tg-invoke-mcp-tool -n file-reader -P '{"path": "/path/to/file.txt"}'
# Write file content
tg-invoke-mcp-tool -n file-writer -P '{"path": "/path/to/output.txt", "content": "Hello World"}'
# List directory contents
tg-invoke-mcp-tool -n directory-lister -P '{"path": "/home/user", "recursive": false}'
```
### Data Processing
```bash
# JSON processing
tg-invoke-mcp-tool -n json-processor -P '{"data": "{\"key\": \"value\"}", "operation": "validate"}'
# CSV analysis
tg-invoke-mcp-tool -n csv-analyzer -P '{"file": "data.csv", "columns": ["name", "age"], "operation": "statistics"}'
# Text transformation
tg-invoke-mcp-tool -n text-transformer -P '{"text": "Hello World", "operation": "uppercase"}'
```
### Web and API
```bash
# HTTP requests
tg-invoke-mcp-tool -n http-client -P '{"url": "https://api.github.com/users/octocat", "method": "GET"}'
# Web scraping
tg-invoke-mcp-tool -n web-scraper -P '{"url": "https://example.com", "selector": "h1"}'
# API testing
tg-invoke-mcp-tool -n api-tester -P '{"endpoint": "/api/v1/users", "method": "POST", "payload": {"name": "John"}}'
```
### Database Operations
```bash
# Query execution
tg-invoke-mcp-tool -n database -P '{"query": "SELECT COUNT(*) FROM users", "database": "production"}'
# Schema inspection
tg-invoke-mcp-tool -n db-inspector -P '{"database": "main", "operation": "list_tables"}'
# Data migration
tg-invoke-mcp-tool -n db-migrator -P '{"source": "old_db", "target": "new_db", "table": "users"}'
```
## Output Formats
### String Response
```bash
tg-invoke-mcp-tool -n calculator -P '{"expression": "10 + 5"}'
# Output: "15"
```
### JSON Response
```bash
tg-invoke-mcp-tool -n weather -P '{"location": "New York"}'
# Output:
# {
# "location": "New York",
# "temperature": 22,
# "conditions": "sunny",
# "humidity": 45
# }
```
### Complex Object Response
```bash
tg-invoke-mcp-tool -n data-analyzer -P '{"dataset": "sales.csv"}'
# Output:
# {
# "summary": {
# "total_records": 1000,
# "columns": ["date", "product", "amount"],
# "date_range": "2024-01-01 to 2024-12-31"
# },
# "statistics": {
# "total_sales": 50000,
# "average_transaction": 50.0,
# "top_product": "Widget A"
# }
# }
```
## Error Handling
### Tool Not Found
```bash
Exception: MCP tool 'nonexistent-tool' not found
```
**Solution**: Check available tools with `tg-show-mcp-tools`.
### Invalid Parameters
```bash
Exception: Invalid JSON in parameters: Expecting property name enclosed in double quotes
```
**Solution**: Verify JSON parameter format and escape special characters.
### Missing Required Parameters
```bash
Exception: Required parameter 'input_data' not provided
```
**Solution**: Check tool documentation for required parameters.
### Flow Not Found
```bash
Exception: Flow instance 'invalid-flow' not found
```
**Solution**: Verify flow ID exists with `tg-show-flows`.
### Tool Execution Error
```bash
Exception: Tool execution failed: Connection timeout
```
**Solution**: Check network connectivity and tool service availability.
## Advanced Usage
### Batch Processing
```bash
# Process multiple files
for file in *.txt; do
echo "Processing $file..."
tg-invoke-mcp-tool -n text-analyzer -P "{\"file\": \"$file\", \"analysis\": \"sentiment\"}"
done
```
### Error Handling in Scripts
```bash
#!/bin/bash
# robust-tool-invoke.sh
tool_name="$1"
parameters="$2"
if ! result=$(tg-invoke-mcp-tool -n "$tool_name" -P "$parameters" 2>&1); then
echo "Error invoking tool: $result" >&2
exit 1
fi
echo "Success: $result"
```
### Pipeline Processing
```bash
# Chain multiple tools
data=$(tg-invoke-mcp-tool -n data-loader -P '{"source": "database"}')
processed=$(tg-invoke-mcp-tool -n data-processor -P "{\"data\": \"$data\", \"operation\": \"clean\"}")
tg-invoke-mcp-tool -n report-generator -P "{\"data\": \"$processed\", \"format\": \"pdf\"}"
```
### Configuration-Driven Invocation
```bash
# Use configuration file
config_file="tool-config.json"
tool_name=$(jq -r '.tool' "$config_file")
parameters=$(jq -c '.parameters' "$config_file")
tg-invoke-mcp-tool -n "$tool_name" -P "$parameters"
```
### Interactive Tool Usage
```bash
#!/bin/bash
# interactive-mcp-tool.sh
echo "Available tools:"
tg-show-mcp-tools
read -p "Enter tool name: " tool_name
read -p "Enter parameters (JSON): " parameters
echo "Invoking tool..."
tg-invoke-mcp-tool -n "$tool_name" -P "$parameters"
```
### Parallel Tool Execution
```bash
# Execute multiple tools in parallel
tools=("weather" "calculator" "file-reader")
params=('{"location": "NYC"}' '{"expression": "2+2"}' '{"path": "file.txt"}')
for i in "${!tools[@]}"; do
(
echo "Executing ${tools[$i]}..."
tg-invoke-mcp-tool -n "${tools[$i]}" -P "${params[$i]}" > "result-${tools[$i]}.json"
) &
done
wait
```
## Tool Management
### List Available Tools
```bash
# Show all registered MCP tools
tg-show-mcp-tools
```
### Register New Tools
```bash
# Register a new MCP tool
tg-set-mcp-tool weather-service "http://weather-api:8080/mcp" "Weather data provider"
```
### Remove Tools
```bash
# Remove an MCP tool
tg-delete-mcp-tool weather-service
```
## Use Cases
### Data Processing Workflows
```bash
# Extract, transform, and load data
raw_data=$(tg-invoke-mcp-tool -n data-extractor -P '{"source": "external_api"}')
clean_data=$(tg-invoke-mcp-tool -n data-cleaner -P "{\"data\": \"$raw_data\"}")
tg-invoke-mcp-tool -n data-loader -P "{\"data\": \"$clean_data\", \"target\": \"warehouse\"}"
```
### Automation Scripts
```bash
# Automated system monitoring
status=$(tg-invoke-mcp-tool -n system-monitor -P '{"checks": ["cpu", "memory", "disk"]}')
if echo "$status" | grep -q "warning"; then
tg-invoke-mcp-tool -n alert-system -P "{\"message\": \"System warning detected\", \"severity\": \"medium\"}"
fi
```
### Integration Testing
```bash
# Test API endpoints
endpoints=("/api/users" "/api/orders" "/api/products")
for endpoint in "${endpoints[@]}"; do
result=$(tg-invoke-mcp-tool -n api-tester -P "{\"endpoint\": \"$endpoint\", \"method\": \"GET\"}")
echo "Testing $endpoint: $result"
done
```
### Content Generation
```bash
# Generate documentation
code_analysis=$(tg-invoke-mcp-tool -n code-analyzer -P '{"directory": "./src", "language": "python"}')
tg-invoke-mcp-tool -n doc-generator -P "{\"analysis\": \"$code_analysis\", \"format\": \"markdown\"}"
```
## Performance Optimization
### Caching Tool Results
```bash
# Cache expensive tool operations
cache_dir="mcp-cache"
mkdir -p "$cache_dir"
invoke_with_cache() {
local tool="$1"
local params="$2"
local cache_key=$(echo "$tool-$params" | md5sum | cut -d' ' -f1)
local cache_file="$cache_dir/$cache_key.json"
if [ -f "$cache_file" ]; then
echo "Cache hit for $tool"
cat "$cache_file"
else
echo "Cache miss, invoking $tool..."
tg-invoke-mcp-tool -n "$tool" -P "$params" | tee "$cache_file"
fi
}
```
### Asynchronous Processing
```bash
# Non-blocking tool execution
async_invoke() {
local tool="$1"
local params="$2"
local output_file="$3"
tg-invoke-mcp-tool -n "$tool" -P "$params" > "$output_file" 2>&1 &
echo $! # Return process ID
}
# Execute multiple tools asynchronously
pid1=$(async_invoke "data-processor" '{"file": "data1.csv"}' "result1.json")
pid2=$(async_invoke "data-processor" '{"file": "data2.csv"}' "result2.json")
# Wait for completion
wait $pid1 $pid2
```
## Environment Variables
- `TRUSTGRAPH_URL`: Default API URL
## Related Commands
- [`tg-show-mcp-tools`](tg-show-mcp-tools.md) - List available MCP tools
- [`tg-set-mcp-tool`](tg-set-mcp-tool.md) - Register MCP tools
- [`tg-delete-mcp-tool`](tg-delete-mcp-tool.md) - Remove MCP tools
- [`tg-show-flows`](tg-show-flows.md) - List available flow instances
- [`tg-invoke-prompt`](tg-invoke-prompt.md) - Invoke prompt templates
## API Integration
This command uses the TrustGraph API flow interface to execute MCP tools within the context of specified flows. MCP tools are external services that implement the Model Control Protocol for standardized AI tool interactions.
## Best Practices
1. **Parameter Validation**: Always validate JSON parameters before execution
2. **Error Handling**: Implement robust error handling for production use
3. **Tool Discovery**: Use `tg-show-mcp-tools` to discover available tools
4. **Resource Management**: Consider performance implications of long-running tools
5. **Security**: Avoid passing sensitive data in parameters; use secure tool configurations
6. **Documentation**: Document custom tool parameters and expected responses
7. **Testing**: Test tool integrations thoroughly before production deployment
## Troubleshooting
### Tool Not Available
```bash
# Check tool registration
tg-show-mcp-tools | grep "tool-name"
# Verify tool service is running
curl -f http://tool-service:8080/health
```
### Parameter Issues
```bash
# Validate JSON format
echo '{"key": "value"}' | jq .
# Test with minimal parameters
tg-invoke-mcp-tool -n tool-name -P '{}'
```
### Flow Problems
```bash
# Check flow status
tg-show-flows | grep "flow-id"
# Verify flow supports MCP tools
tg-get-flow-class -n "flow-class" | jq '.interfaces.mcp_tool'
```
### Connection Issues
```bash
# Test API connectivity
curl -f http://localhost:8088/health
# Check environment variables
echo $TRUSTGRAPH_URL
```

267
docs/cli/tg-set-mcp-tool.md Normal file
View file

@ -0,0 +1,267 @@
# tg-set-mcp-tool
## Synopsis
```
tg-set-mcp-tool [OPTIONS] --name NAME --tool-url URL
```
## Description
The `tg-set-mcp-tool` command configures and registers MCP (Model Control Protocol) tools in the TrustGraph system. It allows defining MCP tool configurations with name and URL. Tools are stored in the 'mcp' configuration group for discovery and execution.
This command is useful for:
- Registering MCP tool endpoints for agent use
- Configuring external MCP server connections
- Managing MCP tool registry for agent workflows
- Integrating third-party MCP tools into TrustGraph
The command stores MCP tool configurations in the 'mcp' configuration group, separate from regular agent tools.
## Options
- `-u, --api-url URL`
- TrustGraph API URL for configuration storage
- Default: `http://localhost:8088/` (or `TRUSTGRAPH_URL` environment variable)
- Should point to a running TrustGraph API instance
- `--name NAME`
- **Required.** MCP tool name identifier
- Used to reference the MCP tool in configurations
- Must be unique within the MCP tool registry
- `--tool-url URL`
- **Required.** MCP tool URL endpoint
- Should point to the MCP server endpoint providing the tool functionality
- Must be a valid URL accessible by the TrustGraph system
- `-h, --help`
- Show help message and exit
## Examples
### Basic MCP Tool Registration
Register a weather service MCP tool:
```bash
tg-set-mcp-tool --name weather --tool-url "http://localhost:3000/weather"
```
### Calculator MCP Tool
Register a calculator MCP tool:
```bash
tg-set-mcp-tool --name calculator --tool-url "http://mcp-tools.example.com/calc"
```
### Remote MCP Service
Register a remote MCP service:
```bash
tg-set-mcp-tool --name document-processor \
--tool-url "https://api.example.com/mcp/documents"
```
### Custom API URL
Register MCP tool with custom TrustGraph API:
```bash
tg-set-mcp-tool -u http://trustgraph.example.com:8088/ \
--name custom-mcp --tool-url "http://custom.mcp.com/api"
```
### Local Development Setup
Register MCP tools for local development:
```bash
tg-set-mcp-tool --name dev-tool --tool-url "http://localhost:8080/mcp"
```
## MCP Tool Configuration
MCP tools are configured with minimal metadata:
- **name**: Unique identifier for the tool
- **url**: Endpoint URL for the MCP server
The configuration is stored as JSON in the 'mcp' configuration group:
```json
{
"name": "weather",
"url": "http://localhost:3000/weather"
}
```
## Advanced Usage
### Updating Existing MCP Tools
Update an existing MCP tool configuration:
```bash
# Update MCP tool URL
tg-set-mcp-tool --name weather --tool-url "http://new-weather-server:3000/api"
```
### Batch MCP Tool Registration
Register multiple MCP tools in a script:
```bash
#!/bin/bash
# Register a suite of MCP tools
tg-set-mcp-tool --name search --tool-url "http://search-mcp:3000/api"
tg-set-mcp-tool --name translate --tool-url "http://translate-mcp:3000/api"
tg-set-mcp-tool --name summarize --tool-url "http://summarize-mcp:3000/api"
```
### Environment-Specific Configuration
Configure MCP tools for different environments:
```bash
# Development environment
export TRUSTGRAPH_URL="http://dev.trustgraph.com:8088/"
tg-set-mcp-tool --name dev-mcp --tool-url "http://dev.mcp.com/api"
# Production environment
export TRUSTGRAPH_URL="http://prod.trustgraph.com:8088/"
tg-set-mcp-tool --name prod-mcp --tool-url "http://prod.mcp.com/api"
```
### MCP Tool Validation
Verify MCP tool registration:
```bash
# Register MCP tool and verify
tg-set-mcp-tool --name test-mcp --tool-url "http://test.mcp.com/api"
# Check if MCP tool was registered
tg-show-mcp-tools | grep test-mcp
```
## Error Handling
The command handles various error conditions:
- **Missing required arguments**: Both name and tool-url must be provided
- **Invalid URLs**: Tool URLs must be valid and accessible
- **API connection errors**: If the TrustGraph API is unavailable
- **Configuration errors**: If MCP tool data cannot be stored
Common error scenarios:
```bash
# Missing required field
tg-set-mcp-tool --name tool1
# Output: Exception: Must specify --tool-url for MCP tool
# Missing name
tg-set-mcp-tool --tool-url "http://example.com/mcp"
# Output: Exception: Must specify --name for MCP tool
# Invalid API URL
tg-set-mcp-tool -u "invalid-url" --name tool1 --tool-url "http://mcp.com"
# Output: Exception: [API connection error]
```
## Integration with Other Commands
### With MCP Tool Management
View registered MCP tools:
```bash
# Register MCP tool
tg-set-mcp-tool --name new-mcp --tool-url "http://new.mcp.com/api"
# View all MCP tools
tg-show-mcp-tools
```
### With Agent Workflows
Use MCP tools in agent workflows:
```bash
# Register MCP tool
tg-set-mcp-tool --name weather --tool-url "http://weather.mcp.com/api"
# Invoke MCP tool directly
tg-invoke-mcp-tool --name weather --input "location=London"
```
### With Configuration Management
MCP tools integrate with configuration management:
```bash
# Register MCP tool
tg-set-mcp-tool --name config-mcp --tool-url "http://config.mcp.com/api"
# View configuration including MCP tools
tg-show-config
```
## Best Practices
1. **Clear Naming**: Use descriptive, unique MCP tool names
2. **Reliable URLs**: Ensure MCP endpoints are stable and accessible
3. **Health Checks**: Verify MCP endpoints are operational before registration
4. **Documentation**: Document MCP tool capabilities and usage
5. **Error Handling**: Implement proper error handling for MCP endpoints
6. **Security**: Use secure URLs (HTTPS) when possible
7. **Monitoring**: Monitor MCP tool availability and performance
## Troubleshooting
### MCP Tool Not Appearing
If a registered MCP tool doesn't appear in listings:
1. Verify the MCP tool was registered successfully
2. Check MCP tool registry with `tg-show-mcp-tools`
3. Ensure the API URL is correct
4. Verify TrustGraph API is running
### MCP Tool Registration Errors
If MCP tool registration fails:
1. Check all required arguments are provided
2. Verify the tool URL is accessible
3. Ensure the MCP endpoint is operational
4. Check API connectivity
5. Review error messages for specific issues
### MCP Tool Connectivity Issues
If MCP tools aren't working as expected:
1. Verify MCP endpoint is accessible from TrustGraph
2. Check MCP server logs for errors
3. Ensure MCP protocol compatibility
4. Review network connectivity and firewall rules
5. Test MCP endpoint directly
## MCP Protocol
The Model Control Protocol (MCP) is a standardized interface for AI model tools:
- **Standardized API**: Consistent interface across different tools
- **Extensible**: Support for complex tool interactions
- **Stateful**: Can maintain state across multiple interactions
- **Secure**: Built-in security and authentication mechanisms
## Security Considerations
When registering MCP tools:
1. **URL Validation**: Ensure URLs are legitimate and secure
2. **Network Security**: Use HTTPS when possible
3. **Access Control**: Implement proper authentication for MCP endpoints
4. **Input Validation**: Validate all inputs to MCP tools
5. **Error Handling**: Don't expose sensitive information in error messages
## Related Commands
- [`tg-show-mcp-tools`](tg-show-mcp-tools.md) - Display registered MCP tools
- [`tg-delete-mcp-tool`](tg-delete-mcp-tool.md) - Remove MCP tool configurations
- [`tg-invoke-mcp-tool`](tg-invoke-mcp-tool.md) - Execute MCP tools
- [`tg-set-tool`](tg-set-tool.md) - Configure regular agent tools
## See Also
- MCP Protocol Documentation
- TrustGraph MCP Integration Guide
- Agent Tool Configuration Guide

321
docs/cli/tg-set-tool.md Normal file
View file

@ -0,0 +1,321 @@
# tg-set-tool
## Synopsis
```
tg-set-tool [OPTIONS] --id ID --name NAME --type TYPE --description DESCRIPTION [--argument ARG...]
```
## Description
The `tg-set-tool` command configures and registers tools in the TrustGraph system. It allows defining tool metadata including ID, name, description, type, and argument specifications. Tools are stored in the agent configuration and indexed for discovery and execution.
This command is useful for:
- Registering new tools for agent use
- Updating existing tool configurations
- Defining tool arguments and parameter types
- Managing the tool registry for agent workflows
The command updates both the tool index and stores the complete tool configuration in the TrustGraph API.
## Options
- `-u, --api-url URL`
- TrustGraph API URL for configuration storage
- Default: `http://localhost:8088/` (or `TRUSTGRAPH_URL` environment variable)
- Should point to a running TrustGraph API instance
- `--id ID`
- **Required.** Unique identifier for the tool
- Used to reference the tool in configurations and agent workflows
- Must be unique within the tool registry
- `--name NAME`
- **Required.** Human-readable name for the tool
- Displayed in tool listings and user interfaces
- Should be descriptive and clear
- `--type TYPE`
- **Required.** Tool type defining its functionality
- Valid types:
- `knowledge-query` - Query knowledge bases
- `text-completion` - Text completion/generation
- `mcp-tool` - Model Control Protocol tool
- `--description DESCRIPTION`
- **Required.** Detailed description of what the tool does
- Used by agents to understand tool capabilities
- Should clearly explain the tool's purpose and function
- `--argument ARG`
- Tool argument specification in format: `name:type:description`
- Can be specified multiple times for multiple arguments
- Valid argument types:
- `string` - String/text parameter
- `number` - Numeric parameter
- `-h, --help`
- Show help message and exit
## Examples
### Basic Tool Registration
Register a simple weather lookup tool:
```bash
tg-set-tool --id weather --name "Weather Lookup" \
--type knowledge-query \
--description "Get current weather information" \
--argument location:string:"Location to query" \
--argument units:string:"Temperature units (C/F)"
```
### Calculator Tool
Register a calculator tool with MCP type:
```bash
tg-set-tool --id calculator --name "Calculator" --type mcp-tool \
--description "Perform mathematical calculations" \
--argument expression:string:"Mathematical expression to evaluate"
```
### Text Completion Tool
Register a text completion tool:
```bash
tg-set-tool --id text-generator --name "Text Generator" \
--type text-completion \
--description "Generate text based on prompts" \
--argument prompt:string:"Text prompt for generation" \
--argument max_tokens:number:"Maximum tokens to generate"
```
### Custom API URL
Register a tool with custom API endpoint:
```bash
tg-set-tool -u http://trustgraph.example.com:8088/ \
--id custom-tool --name "Custom Tool" \
--type knowledge-query \
--description "Custom tool functionality"
```
### Tool Without Arguments
Register a simple tool with no arguments:
```bash
tg-set-tool --id status-check --name "Status Check" \
--type knowledge-query \
--description "Check system status"
```
## Tool Types
### knowledge-query
Tools that query knowledge bases, databases, or information systems:
- Used for information retrieval
- Typically return structured data or search results
- Examples: web search, document lookup, database queries
### text-completion
Tools that generate or complete text:
- Used for text generation tasks
- Process prompts and return generated content
- Examples: language models, text generators, summarizers
### mcp-tool
Model Control Protocol tools:
- Standardized tool interface for AI models
- Support complex interactions and state management
- Examples: external API integrations, complex workflows
## Argument Types
### string
Text or string parameters:
- Accept any text input
- Used for queries, prompts, identifiers
- Should include clear description of expected format
### number
Numeric parameters:
- Accept integer or floating-point values
- Used for limits, thresholds, quantities
- Should specify valid ranges when applicable
## Configuration Storage
The tool configuration is stored in two parts:
1. **Tool Index** (`agent.tool-index`)
- List of all registered tool IDs
- Updated to include new tools
- Used for tool discovery
2. **Tool Configuration** (`agent.tool.{id}`)
- Complete tool definition as JSON
- Includes metadata and argument specifications
- Used for tool execution and validation
## Advanced Usage
### Updating Existing Tools
Update an existing tool configuration:
```bash
# Update tool description
tg-set-tool --id weather --name "Weather Lookup" \
--type knowledge-query \
--description "Updated weather information service" \
--argument location:string:"Location to query"
```
### Batch Tool Registration
Register multiple tools in a script:
```bash
#!/bin/bash
# Register a suite of tools
tg-set-tool --id search --name "Web Search" --type knowledge-query \
--description "Search the web" \
--argument query:string:"Search query"
tg-set-tool --id summarize --name "Text Summarizer" --type text-completion \
--description "Summarize text content" \
--argument text:string:"Text to summarize"
tg-set-tool --id translate --name "Translator" --type mcp-tool \
--description "Translate text between languages" \
--argument text:string:"Text to translate" \
--argument target_lang:string:"Target language"
```
### Tool Validation
Verify tool registration:
```bash
# Register tool and verify
tg-set-tool --id test-tool --name "Test Tool" \
--type knowledge-query \
--description "Test tool for validation"
# Check if tool was registered
tg-show-tools | grep test-tool
```
## Error Handling
The command handles various error conditions:
- **Missing required arguments**: All required fields must be provided
- **Invalid tool types**: Only valid types are accepted
- **Invalid argument format**: Arguments must follow `name:type:description` format
- **API connection errors**: If the TrustGraph API is unavailable
- **Configuration errors**: If tool data cannot be stored
Common error scenarios:
```bash
# Missing required field
tg-set-tool --id tool1 --name "Tool 1"
# Output: Exception: Must specify --type for tool
# Invalid tool type
tg-set-tool --id tool1 --name "Tool 1" --type invalid-type
# Output: Exception: Type must be one of: knowledge-query, text-completion, mcp-tool
# Invalid argument format
tg-set-tool --id tool1 --name "Tool 1" --type knowledge-query \
--argument "bad-format"
# Output: Exception: Arguments should be form name:type:description
```
## Integration with Other Commands
### With Tool Management
View registered tools:
```bash
# Register tool
tg-set-tool --id new-tool --name "New Tool" \
--type knowledge-query \
--description "Newly registered tool"
# View all tools
tg-show-tools
```
### With Agent Invocation
Use registered tools with agents:
```bash
# Register tool
tg-set-tool --id weather --name "Weather" \
--type knowledge-query \
--description "Weather lookup"
# Use tool in agent workflow
tg-invoke-agent --prompt "What's the weather in London?"
```
### With Flow Configuration
Tools can be used in flow configurations:
```bash
# Register tool for flow use
tg-set-tool --id data-processor --name "Data Processor" \
--type mcp-tool \
--description "Process data in flows"
# View flows that might use the tool
tg-show-flows
```
## Best Practices
1. **Clear Naming**: Use descriptive, unique tool IDs and names
2. **Detailed Descriptions**: Provide comprehensive tool descriptions
3. **Argument Documentation**: Clearly describe each argument's purpose
4. **Type Selection**: Choose appropriate tool types for functionality
5. **Validation**: Test tools after registration
6. **Version Management**: Track tool configuration changes
7. **Documentation**: Document custom tools and their usage
## Troubleshooting
### Tool Not Appearing
If a registered tool doesn't appear in listings:
1. Verify the tool was registered successfully
2. Check the tool index with `tg-show-tools`
3. Ensure the API URL is correct
4. Verify TrustGraph API is running
### Tool Registration Errors
If tool registration fails:
1. Check all required arguments are provided
2. Verify argument format is correct
3. Ensure tool type is valid
4. Check API connectivity
5. Review error messages for specific issues
### Tool Configuration Issues
If tools aren't working as expected:
1. Verify tool arguments are correctly specified
2. Check tool type matches intended functionality
3. Ensure tool implementation is available
4. Review agent logs for tool execution errors
## Related Commands
- [`tg-show-tools`](tg-show-tools.md) - Display registered tools
- [`tg-delete-tool`](tg-delete-tool.md) - Remove tool configurations
- [`tg-set-mcp-tool`](tg-set-mcp-tool.md) - Configure MCP tools
- [`tg-invoke-agent`](tg-invoke-agent.md) - Use tools with agents
## See Also
- TrustGraph Tool Development Guide
- Agent Configuration Documentation
- MCP Tool Integration Guide

106
docs/tech-specs/ARCHITECTURE_PRINCIPLES.md Normal file
View file

@ -0,0 +1,106 @@
# Knowledge Graph Architecture Foundations
## Foundation 1: Subject-Predicate-Object (SPO) Graph Model
**Decision**: Adopt SPO/RDF as the core knowledge representation model
**Rationale**:
- Provides maximum flexibility and interoperability with existing graph technologies
- Enables seamless translation to other graph query languages (e.g., SPO → Cypher, but not vice versa)
- Creates a foundation that "unlocks a lot" of downstream capabilities
- Supports both node-to-node relationships (SPO) and node-to-literal relationships (RDF)
**Implementation**:
- Core data structure: `node → edge → {node | literal}`
- Maintain compatibility with RDF standards while supporting extended SPO operations
## Foundation 2: LLM-Native Knowledge Graph Integration
**Decision**: Optimize knowledge graph structure and operations for LLM interaction
**Rationale**:
- Primary use case involves LLMs interfacing with knowledge graphs
- Graph technology choices must prioritize LLM compatibility over other considerations
- Enables natural language processing workflows that leverage structured knowledge
**Implementation**:
- Design graph schemas that LLMs can effectively reason about
- Optimize for common LLM interaction patterns
## Foundation 3: Embedding-Based Graph Navigation
**Decision**: Implement direct mapping from natural language queries to graph nodes via embeddings
**Rationale**:
- Enables the simplest possible path from NLP query to graph navigation
- Avoids complex intermediate query generation steps
- Provides efficient semantic search capabilities within the graph structure
**Implementation**:
- `NLP Query → Graph Embeddings → Graph Nodes`
- Maintain embedding representations for all graph entities
- Support direct semantic similarity matching for query resolution
## Foundation 4: Distributed Entity Resolution with Deterministic Identifiers
**Decision**: Support parallel knowledge extraction with deterministic entity identification (80% rule)
**Rationale**:
- **Ideal**: Single-process extraction with complete state visibility enables perfect entity resolution
- **Reality**: Scalability requirements demand parallel processing capabilities
- **Compromise**: Design for deterministic entity identification across distributed processes
**Implementation**:
- Develop mechanisms for generating consistent, unique identifiers across different knowledge extractors
- Same entity mentioned in different processes must resolve to the same identifier
- Acknowledge that ~20% of edge cases may require alternative processing models
- Design fallback mechanisms for complex entity resolution scenarios
## Foundation 5: Event-Driven Architecture with Publish-Subscribe
**Decision**: Implement pub-sub messaging system for system coordination
**Rationale**:
- Enables loose coupling between knowledge extraction, storage, and query components
- Supports real-time updates and notifications across the system
- Facilitates scalable, distributed processing workflows
**Implementation**:
- Message-driven coordination between system components
- Event streams for knowledge updates, extraction completion, and query results
## Foundation 6: Reentrant Agent Communication
**Decision**: Support reentrant pub-sub operations for agent-based processing
**Rationale**:
- Enables sophisticated agent workflows where agents can trigger and respond to each other
- Supports complex, multi-step knowledge processing pipelines
- Allows for recursive and iterative processing patterns
**Implementation**:
- Pub-sub system must handle reentrant calls safely
- Agent coordination mechanisms that prevent infinite loops
- Support for agent workflow orchestration
## Foundation 7: Columnar Data Store Integration
**Decision**: Ensure query compatibility with columnar storage systems
**Rationale**:
- Enables efficient analytical queries over large knowledge datasets
- Supports business intelligence and reporting use cases
- Bridges graph-based knowledge representation with traditional analytical workflows
**Implementation**:
- Query translation layer: Graph queries → Columnar queries
- Hybrid storage strategy supporting both graph operations and analytical workloads
- Maintain query performance across both paradigms
---
## Architecture Principles Summary
1. **Flexibility First**: SPO/RDF model provides maximum adaptability
2. **LLM Optimization**: All design decisions consider LLM interaction requirements
3. **Semantic Efficiency**: Direct embedding-to-node mapping for optimal query performance
4. **Pragmatic Scalability**: Balance perfect accuracy with practical distributed processing
5. **Event-Driven Coordination**: Pub-sub enables loose coupling and scalability
6. **Agent-Friendly**: Support complex, multi-agent processing workflows
7. **Analytical Compatibility**: Bridge graph and columnar paradigms for comprehensive querying
These foundations establish a knowledge graph architecture that balances theoretical rigor with practical scalability requirements, optimized for LLM integration and distributed processing.

169
docs/tech-specs/LOGGING_STRATEGY.md Normal file
View file

@ -0,0 +1,169 @@
# TrustGraph Logging Strategy
## Overview
TrustGraph uses Python's built-in `logging` module for all logging operations. This provides a standardized, flexible approach to logging across all components of the system.
## Default Configuration
### Logging Level
- **Default Level**: `INFO`
- **Debug Mode**: `DEBUG` (enabled via command-line argument)
- **Production**: `WARNING` or `ERROR` as appropriate
### Output Destination
All logs should be written to **standard output (stdout)** to ensure compatibility with containerized environments and log aggregation systems.
## Implementation Guidelines
### 1. Logger Initialization
Each module should create its own logger using the module's `__name__`:
```python
import logging
logger = logging.getLogger(__name__)
```
### 2. Centralized Configuration
The logging configuration should be centralized in `async_processor.py` (or a dedicated logging configuration module) since it's inherited by much of the codebase:
```python
import logging
import argparse
def setup_logging(log_level='INFO'):
"""Configure logging for the entire application"""
logging.basicConfig(
level=getattr(logging, log_level.upper()),
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
handlers=[logging.StreamHandler()]
)
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument(
'--log-level',
default='INFO',
choices=['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'],
help='Set the logging level (default: INFO)'
)
return parser.parse_args()
# In main execution
if __name__ == '__main__':
args = parse_args()
setup_logging(args.log_level)
```
### 3. Logging Best Practices
#### Log Levels Usage
- **DEBUG**: Detailed information for diagnosing problems (variable values, function entry/exit)
- **INFO**: General informational messages (service started, configuration loaded, processing milestones)
- **WARNING**: Warning messages for potentially harmful situations (deprecated features, recoverable errors)
- **ERROR**: Error messages for serious problems (failed operations, exceptions)
- **CRITICAL**: Critical messages for system failures requiring immediate attention
#### Message Format
```python
# Good - includes context
logger.info(f"Processing document: {doc_id}, size: {doc_size} bytes")
logger.error(f"Failed to connect to database: {error}", exc_info=True)
# Avoid - lacks context
logger.info("Processing document")
logger.error("Connection failed")
```
#### Performance Considerations
```python
# Use lazy formatting for expensive operations
logger.debug("Expensive operation result: %s", expensive_function())
# Check log level for very expensive debug operations
if logger.isEnabledFor(logging.DEBUG):
debug_data = compute_expensive_debug_info()
logger.debug(f"Debug data: {debug_data}")
```
### 4. Structured Logging
For complex data, use structured logging:
```python
logger.info("Request processed", extra={
'request_id': request_id,
'duration_ms': duration,
'status_code': status_code,
'user_id': user_id
})
```
### 5. Exception Logging
Always include stack traces for exceptions:
```python
try:
process_data()
except Exception as e:
logger.error(f"Failed to process data: {e}", exc_info=True)
raise
```
### 6. Async Logging Considerations
For async code, ensure thread-safe logging:
```python
import asyncio
import logging
async def async_operation():
logger = logging.getLogger(__name__)
logger.info(f"Starting async operation in task: {asyncio.current_task().get_name()}")
```
## Environment Variables
Support environment-based configuration as a fallback:
```python
import os
log_level = os.environ.get('TRUSTGRAPH_LOG_LEVEL', 'INFO')
```
## Testing
During tests, consider using a different logging configuration:
```python
# In test setup
logging.getLogger().setLevel(logging.WARNING) # Reduce noise during tests
```
## Monitoring Integration
Ensure log format is compatible with monitoring tools:
- Include timestamps in ISO format
- Use consistent field names
- Include correlation IDs where applicable
- Structure logs for easy parsing (JSON format for production)
## Security Considerations
- Never log sensitive information (passwords, API keys, personal data)
- Sanitize user input before logging
- Use placeholders for sensitive fields: `user_id=****1234`
## Migration Path
For existing code using print statements:
1. Replace `print()` with appropriate logger calls
2. Choose appropriate log levels based on message importance
3. Add context to make logs more useful
4. Test logging output at different levels

91
docs/tech-specs/SCHEMA_REFACTORING_PROPOSAL.md Normal file
View file

@ -0,0 +1,91 @@
# Schema Directory Refactoring Proposal
## Current Issues
1. **Flat structure** - All schemas in one directory makes it hard to understand relationships
2. **Mixed concerns** - Core types, domain objects, and API contracts all mixed together
3. **Unclear naming** - Files like "object.py", "types.py", "topic.py" don't clearly indicate their purpose
4. **No clear layering** - Can't easily see what depends on what
## Proposed Structure
```
trustgraph-base/trustgraph/schema/
├── __init__.py
├── core/ # Core primitive types used everywhere
│ ├── __init__.py
│ ├── primitives.py # Error, Value, Triple, Field, RowSchema
│ ├── metadata.py # Metadata record
│ └── topic.py # Topic utilities
│
├── knowledge/ # Knowledge domain models and extraction
│ ├── __init__.py
│ ├── graph.py # EntityContext, EntityEmbeddings, Triples
│ ├── document.py # Document, TextDocument, Chunk
│ ├── knowledge.py # Knowledge extraction types
│ ├── embeddings.py # All embedding-related types (moved from multiple files)
│ └── nlp.py # Definition, Topic, Relationship, Fact types
│
└── services/ # Service request/response contracts
├── __init__.py
├── llm.py # TextCompletion, Embeddings, Tool requests/responses
├── retrieval.py # GraphRAG, DocumentRAG queries/responses
├── query.py # GraphEmbeddingsRequest/Response, DocumentEmbeddingsRequest/Response
├── agent.py # Agent requests/responses
├── flow.py # Flow requests/responses
├── prompt.py # Prompt service requests/responses
├── config.py # Configuration service
├── library.py # Librarian service
└── lookup.py # Lookup service
```
## Key Changes
1. **Hierarchical organization** - Clear separation between core types, knowledge models, and service contracts
2. **Better naming**:
- `types.py` → `core/primitives.py` (clearer purpose)
- `object.py` → Split between appropriate files based on actual content
- `documents.py` → `knowledge/document.py` (singular, consistent)
- `models.py` → `services/llm.py` (clearer what kind of models)
- `prompt.py` → Split: service parts to `services/prompt.py`, data types to `knowledge/nlp.py`
3. **Logical grouping**:
- All embedding types consolidated in `knowledge/embeddings.py`
- All LLM-related service contracts in `services/llm.py`
- Clear separation of request/response pairs in services directory
- Knowledge extraction types grouped with other knowledge domain models
4. **Dependency clarity**:
- Core types have no dependencies
- Knowledge models depend only on core
- Service contracts can depend on both core and knowledge models
## Migration Benefits
1. **Easier navigation** - Developers can quickly find what they need
2. **Better modularity** - Clear boundaries between different concerns
3. **Simpler imports** - More intuitive import paths
4. **Future-proof** - Easy to add new knowledge types or services without cluttering
## Example Import Changes
```python
# Before
from trustgraph.schema import Error, Triple, GraphEmbeddings, TextCompletionRequest
# After
from trustgraph.schema.core import Error, Triple
from trustgraph.schema.knowledge import GraphEmbeddings
from trustgraph.schema.services import TextCompletionRequest
```
## Implementation Notes
1. Keep backward compatibility by maintaining imports in root `__init__.py`
2. Move files gradually, updating imports as needed
3. Consider adding a `legacy.py` that imports everything for transition period
4. Update documentation to reflect new structure
<function_calls>
<invoke name="TodoWrite">
<parameter name="todos">[{"id": "1", "content": "Examine current schema directory structure", "status": "completed", "priority": "high"}, {"id": "2", "content": "Analyze schema files and their purposes", "status": "completed", "priority": "high"}, {"id": "3", "content": "Propose improved naming and structure", "status": "completed", "priority": "high"}]

253
docs/tech-specs/STRUCTURED_DATA.md Normal file
View file

@ -0,0 +1,253 @@
# Structured Data Technical Specification
## Overview
This specification describes the integration of TrustGraph with structured data flows, enabling the system to work with data that can be represented as rows in tables or objects in object stores. The integration supports four primary use cases:
1. **Unstructured to Structured Extraction**: Read unstructured data sources, identify and extract object structures, and store them in a tabular format
2. **Structured Data Ingestion**: Load data that is already in structured formats directly into the structured store alongside extracted data
3. **Natural Language Querying**: Convert natural language questions into structured queries to extract matching data from the store
4. **Direct Structured Querying**: Execute structured queries directly against the data store for precise data retrieval
## Goals
- **Unified Data Access**: Provide a single interface for accessing both structured and unstructured data within TrustGraph
- **Seamless Integration**: Enable smooth interoperability between TrustGraph's graph-based knowledge representation and traditional structured data formats
- **Flexible Extraction**: Support automatic extraction of structured data from various unstructured sources (documents, text, etc.)
- **Query Versatility**: Allow users to query data using both natural language and structured query languages
- **Data Consistency**: Maintain data integrity and consistency across different data representations
- **Performance Optimization**: Ensure efficient storage and retrieval of structured data at scale
- **Schema Flexibility**: Support both schema-on-write and schema-on-read approaches to accommodate diverse data sources
- **Backwards Compatibility**: Preserve existing TrustGraph functionality while adding structured data capabilities
## Background
TrustGraph currently excels at processing unstructured data and building knowledge graphs from diverse sources. However, many enterprise use cases involve data that is inherently structured - customer records, transaction logs, inventory databases, and other tabular datasets. These structured datasets often need to be analyzed alongside unstructured content to provide comprehensive insights.
Current limitations include:
- No native support for ingesting pre-structured data formats (CSV, JSON arrays, database exports)
- Inability to preserve the inherent structure when extracting tabular data from documents
- Lack of efficient querying mechanisms for structured data patterns
- Missing bridge between SQL-like queries and TrustGraph's graph queries
This specification addresses these gaps by introducing a structured data layer that complements TrustGraph's existing capabilities. By supporting structured data natively, TrustGraph can:
- Serve as a unified platform for both structured and unstructured data analysis
- Enable hybrid queries that span both graph relationships and tabular data
- Provide familiar interfaces for users accustomed to working with structured data
- Unlock new use cases in data integration and business intelligence
## Technical Design
### Architecture
The structured data integration requires the following technical components:
1. **NLP-to-Structured-Query Service**
- Converts natural language questions into structured queries
- Supports multiple query language targets (initially SQL-like syntax)
- Integrates with existing TrustGraph NLP capabilities
Module: trustgraph-flow/trustgraph/query/nlp_query/cassandra
2. **Configuration Schema Support** ✅ **[COMPLETE]**
- Extended configuration system to store structured data schemas
- Support for defining table structures, field types, and relationships
- Schema versioning and migration capabilities
3. **Object Extraction Module** ✅ **[COMPLETE]**
- Enhanced knowledge extractor flow integration
- Identifies and extracts structured objects from unstructured sources
- Maintains provenance and confidence scores
- Registers a config handler (example: trustgraph-flow/trustgraph/prompt/template/service.py) to receive config data and decode schema information
- Receives objects and decodes them to ExtractedObject objects for delivery on the Pulsar queue
- NOTE: There's existing code at `trustgraph-flow/trustgraph/extract/object/row/`. This was a previous attempt and will need to be majorly refactored as it doesn't conform to current APIs. Use it if it's useful, start from scratch if not.
- Requires a command-line interface: `kg-extract-objects`
Module: trustgraph-flow/trustgraph/extract/kg/objects/
4. **Structured Store Writer Module** ✅ **[COMPLETE]**
- Receives objects in ExtractedObject format from Pulsar queues
- Initial implementation targeting Apache Cassandra as the structured data store
- Handles dynamic table creation based on schemas encountered
- Manages schema-to-Cassandra table mapping and data transformation
- Provides batch and streaming write operations for performance optimization
- No Pulsar outputs - this is a terminal service in the data flow
**Schema Handling**:
- Monitors incoming ExtractedObject messages for schema references
- When a new schema is encountered for the first time, automatically creates the corresponding Cassandra table
- Maintains a cache of known schemas to avoid redundant table creation attempts
- Should consider whether to receive schema definitions directly or rely on schema names in ExtractedObject messages
**Cassandra Table Mapping**:
- Keyspace is named after the `user` field from ExtractedObject's Metadata
- Table is named after the `schema_name` field from ExtractedObject
- Collection from Metadata becomes part of the partition key to ensure:
- Natural data distribution across Cassandra nodes
- Efficient queries within a specific collection
- Logical isolation between different data imports/sources
- Primary key structure: `PRIMARY KEY ((collection, <schema_primary_key_fields>), <clustering_keys>)`
- Collection is always the first component of the partition key
- Schema-defined primary key fields follow as part of the composite partition key
- This requires queries to specify the collection, ensuring predictable performance
- Field definitions map to Cassandra columns with type conversions:
- `string` → `text`
- `integer` → `int` or `bigint` based on size hint
- `float` → `float` or `double` based on precision needs
- `boolean` → `boolean`
- `timestamp` → `timestamp`
- `enum` → `text` with application-level validation
- Indexed fields create Cassandra secondary indexes (excluding fields already in the primary key)
- Required fields are enforced at the application level (Cassandra doesn't support NOT NULL)
**Object Storage**:
- Extracts values from ExtractedObject.values map
- Performs type conversion and validation before insertion
- Handles missing optional fields gracefully
- Maintains metadata about object provenance (source document, confidence scores)
- Supports idempotent writes to handle message replay scenarios
**Implementation Notes**:
- Existing code at `trustgraph-flow/trustgraph/storage/objects/cassandra/` is outdated and doesn't comply with current APIs
- Should reference `trustgraph-flow/trustgraph/storage/triples/cassandra` as an example of a working storage processor
- Needs evaluation of existing code for any reusable components before deciding to refactor or rewrite
Module: trustgraph-flow/trustgraph/storage/objects/cassandra
5. **Structured Query Service**
- Accepts structured queries in defined formats
- Executes queries against the structured store
- Returns objects matching query criteria
- Supports pagination and result filtering
Module: trustgraph-flow/trustgraph/query/objects/cassandra
6. **Agent Tool Integration**
- New tool class for agent frameworks
- Enables agents to query structured data stores
- Provides natural language and structured query interfaces
- Integrates with existing agent decision-making processes
7. **Structured Data Ingestion Service**
- Accepts structured data in multiple formats (JSON, CSV, XML)
- Parses and validates incoming data against defined schemas
- Converts data into normalized object streams
- Emits objects to appropriate message queues for processing
- Supports bulk uploads and streaming ingestion
Module: trustgraph-flow/trustgraph/decoding/structured
8. **Object Embedding Service**
- Generates vector embeddings for structured objects
- Enables semantic search across structured data
- Supports hybrid search combining structured queries with semantic similarity
- Integrates with existing vector stores
Module: trustgraph-flow/trustgraph/embeddings/object_embeddings/qdrant
### Data Models
#### Schema Storage Mechanism
Schemas are stored in TrustGraph's configuration system using the following structure:
- **Type**: `schema` (fixed value for all structured data schemas)
- **Key**: The unique name/identifier of the schema (e.g., `customer_records`, `transaction_log`)
- **Value**: JSON schema definition containing the structure
Example configuration entry:
```
Type: schema
Key: customer_records
Value: {
"name": "customer_records",
"description": "Customer information table",
"fields": [
{
"name": "customer_id",
"type": "string",
"primary_key": true
},
{
"name": "name",
"type": "string",
"required": true
},
{
"name": "email",
"type": "string",
"required": true
},
{
"name": "registration_date",
"type": "timestamp"
},
{
"name": "status",
"type": "string",
"enum": ["active", "inactive", "suspended"]
}
],
"indexes": ["email", "registration_date"]
}
```
This approach allows:
- Dynamic schema definition without code changes
- Easy schema updates and versioning
- Consistent integration with existing TrustGraph configuration management
- Support for multiple schemas within a single deployment
### APIs
New APIs:
- Pulsar schemas for above types
- Pulsar interfaces in new flows
- Need a means to specify schema types in flows so that flows know which
schema types to load
- APIs added to gateway and rev-gateway
Modified APIs:
- Knowledge extraction endpoints - Add structured object output option
- Agent endpoints - Add structured data tool support
### Implementation Details
Following existing conventions - these are just new processing modules.
Everything is in the trustgraph-flow packages except for schema items
in trustgraph-base.
Need some UI work in the Workbench to be able to demo / pilot this
capability.
## Security Considerations
No extra considerations.
## Performance Considerations
Some questions around using Cassandra queries and indexes so that queries
don't slow down.
## Testing Strategy
Use existing test strategy, will build unit, contract and integration tests.
## Migration Plan
None.
## Timeline
Not specified.
## Open Questions
- Can this be made to work with other store types? We're aiming to use
interfaces which make modules which work with one store applicable to
other stores.
## References
n/a.

139
docs/tech-specs/STRUCTURED_DATA_SCHEMAS.md Normal file
View file

@ -0,0 +1,139 @@
# Structured Data Pulsar Schema Changes
## Overview
Based on the STRUCTURED_DATA.md specification, this document proposes the necessary Pulsar schema additions and modifications to support structured data capabilities in TrustGraph.
## Required Schema Changes
### 1. Core Schema Enhancements
#### Enhanced Field Definition
The existing `Field` class in `core/primitives.py` needs additional properties:
```python
class Field(Record):
name = String()
type = String() # int, string, long, bool, float, double, timestamp
size = Integer()
primary = Boolean()
description = String()
# NEW FIELDS:
required = Boolean() # Whether field is required
enum_values = Array(String()) # For enum type fields
indexed = Boolean() # Whether field should be indexed
```
### 2. New Knowledge Schemas
#### 2.1 Structured Data Submission
New file: `knowledge/structured.py`
```python
from pulsar.schema import Record, String, Bytes, Map
from ..core.metadata import Metadata
class StructuredDataSubmission(Record):
metadata = Metadata()
format = String() # "json", "csv", "xml"
schema_name = String() # Reference to schema in config
data = Bytes() # Raw data to ingest
options = Map(String()) # Format-specific options
```
### 3. New Service Schemas
#### 3.1 NLP to Structured Query Service
New file: `services/nlp_query.py`
```python
from pulsar.schema import Record, String, Array, Map, Integer, Double
from ..core.primitives import Error
class NLPToStructuredQueryRequest(Record):
natural_language_query = String()
max_results = Integer()
context_hints = Map(String()) # Optional context for query generation
class NLPToStructuredQueryResponse(Record):
error = Error()
graphql_query = String() # Generated GraphQL query
variables = Map(String()) # GraphQL variables if any
detected_schemas = Array(String()) # Which schemas the query targets
confidence = Double()
```
#### 3.2 Structured Query Service
New file: `services/structured_query.py`
```python
from pulsar.schema import Record, String, Map, Array
from ..core.primitives import Error
class StructuredQueryRequest(Record):
query = String() # GraphQL query
variables = Map(String()) # GraphQL variables
operation_name = String() # Optional operation name for multi-operation documents
class StructuredQueryResponse(Record):
error = Error()
data = String() # JSON-encoded GraphQL response data
errors = Array(String()) # GraphQL errors if any
```
#### 2.2 Object Extraction Output
New file: `knowledge/object.py`
```python
from pulsar.schema import Record, String, Map, Double
from ..core.metadata import Metadata
class ExtractedObject(Record):
metadata = Metadata()
schema_name = String() # Which schema this object belongs to
values = Map(String()) # Field name -> value
confidence = Double()
source_span = String() # Text span where object was found
```
### 4. Enhanced Knowledge Schemas
#### 4.1 Object Embeddings Enhancement
Update `knowledge/embeddings.py` to support structured object embeddings better:
```python
class StructuredObjectEmbedding(Record):
metadata = Metadata()
vectors = Array(Array(Double()))
schema_name = String()
object_id = String() # Primary key value
field_embeddings = Map(Array(Double())) # Per-field embeddings
```
## Integration Points
### Flow Integration
The schemas will be used by new flow modules:
- `trustgraph-flow/trustgraph/decoding/structured` - Uses StructuredDataSubmission
- `trustgraph-flow/trustgraph/query/nlp_query/cassandra` - Uses NLP query schemas
- `trustgraph-flow/trustgraph/query/objects/cassandra` - Uses structured query schemas
- `trustgraph-flow/trustgraph/extract/object/row/` - Consumes Chunk, produces ExtractedObject
- `trustgraph-flow/trustgraph/storage/objects/cassandra` - Uses Rows schema
- `trustgraph-flow/trustgraph/embeddings/object_embeddings/qdrant` - Uses object embedding schemas
## Implementation Notes
1. **Schema Versioning**: Consider adding a `version` field to RowSchema for future migration support
2. **Type System**: The `Field.type` should support all Cassandra native types
3. **Batch Operations**: Most services should support both single and batch operations
4. **Error Handling**: Consistent error reporting across all new services
5. **Backwards Compatibility**: Existing schemas remain unchanged except for minor Field enhancements
## Next Steps
1. Implement schema files in the new structure
2. Update existing services to recognize new schema types
3. Implement flow modules that use these schemas
4. Add gateway/rev-gateway endpoints for new services
5. Create unit tests for schema validation