From c96200caa63e9e0671c791cf45be26f25ccddb82 Mon Sep 17 00:00:00 2001
From: Cyber MacGeddon <cybermaggedon@gmail.com>
Date: Thu, 3 Jul 2025 12:26:01 +0100
Subject: [PATCH] More CLI docs

---
 docs/cli/tg-show-processor-state.md | 196 +++++++++++++++++++
 docs/cli/tg-show-token-rate.md      | 246 ++++++++++++++++++++++++
 docs/cli/tg-show-tools.md           | 283 ++++++++++++++++++++++++++++
 3 files changed, 725 insertions(+)
 create mode 100644 docs/cli/tg-show-processor-state.md
 create mode 100644 docs/cli/tg-show-token-rate.md
 create mode 100644 docs/cli/tg-show-tools.md

diff --git a/docs/cli/tg-show-processor-state.md b/docs/cli/tg-show-processor-state.md
new file mode 100644
index 00000000..ab6017c7
--- /dev/null
+++ b/docs/cli/tg-show-processor-state.md
@@ -0,0 +1,196 @@
+# tg-show-processor-state
+
+## Synopsis
+
+```
+tg-show-processor-state [OPTIONS]
+```
+
+## Description
+
+The `tg-show-processor-state` command displays the current state of TrustGraph processors by querying the metrics endpoint. It retrieves processor information from the Prometheus metrics API and displays active processors with visual status indicators.
+
+This command is useful for:
+- Monitoring processor health and availability
+- Verifying that processors are running correctly
+- Troubleshooting processor connectivity issues
+- Getting a quick overview of active TrustGraph components
+
+## Options
+
+- `-m, --metrics-url URL`
+  - Metrics endpoint URL to query for processor information
+  - Default: `http://localhost:8088/api/metrics`
+  - Should point to a Prometheus-compatible metrics endpoint
+
+- `-h, --help`
+  - Show help message and exit
+
+## Examples
+
+### Basic Usage
+
+Display processor states using the default metrics URL:
+```bash
+tg-show-processor-state
+```
+
+### Custom Metrics URL
+
+Query processor states from a different metrics endpoint:
+```bash
+tg-show-processor-state -m http://metrics.example.com:8088/api/metrics
+```
+
+### Remote Monitoring
+
+Monitor processors on a remote TrustGraph instance:
+```bash
+tg-show-processor-state --metrics-url http://10.0.1.100:8088/api/metrics
+```
+
+## Output Format
+
+The command displays processor information in a table format:
+```
+  processor_name                 💚
+  another_processor              💚
+  third_processor                💚
+```
+
+Each line shows:
+- Processor name (left-aligned, 30 characters wide)
+- Status indicator (💚 for active processors)
+
+## Advanced Usage
+
+### Monitoring Script
+
+Create a monitoring script to periodically check processor states:
+```bash
+#!/bin/bash
+while true; do
+    echo "=== Processor State Check ===" 
+    date
+    tg-show-processor-state
+    echo
+    sleep 30
+done
+```
+
+### Health Check Integration
+
+Use in health check scripts:
+```bash
+#!/bin/bash
+output=$(tg-show-processor-state 2>&1)
+if [ $? -eq 0 ]; then
+    echo "Processors are running"
+    echo "$output"
+else
+    echo "Error checking processor state: $output"
+    exit 1
+fi
+```
+
+### Multiple Environment Monitoring
+
+Monitor processors across different environments:
+```bash
+#!/bin/bash
+for env in dev staging prod; do
+    echo "=== $env Environment ==="
+    tg-show-processor-state -m "http://${env}-metrics:8088/api/metrics"
+    echo
+done
+```
+
+## Error Handling
+
+The command handles various error conditions:
+
+- **Connection errors**: If the metrics endpoint is unavailable
+- **Invalid JSON**: If the metrics response is malformed
+- **Missing data**: If the expected processor_info metric is not found
+- **HTTP errors**: If the metrics endpoint returns an error status
+
+Common error scenarios:
+```bash
+# Metrics endpoint not available
+tg-show-processor-state -m http://invalid-host:8088/api/metrics
+# Output: Exception: [Connection error details]
+
+# Invalid URL format
+tg-show-processor-state -m "not-a-url"
+# Output: Exception: [URL parsing error]
+```
+
+## Integration with Other Commands
+
+### With Flow Monitoring
+
+Combine with flow state monitoring:
+```bash
+echo "=== Processor States ==="
+tg-show-processor-state
+echo
+echo "=== Flow States ==="
+tg-show-flow-state
+```
+
+### With Configuration Display
+
+Check processors and current configuration:
+```bash
+echo "=== Active Processors ==="
+tg-show-processor-state
+echo
+echo "=== Current Configuration ==="
+tg-show-config
+```
+
+## Best Practices
+
+1. **Regular Monitoring**: Include in regular health check routines
+2. **Error Handling**: Always check command exit status in scripts
+3. **Logging**: Capture output for historical analysis
+4. **Alerting**: Set up alerts based on processor availability
+5. **Documentation**: Keep track of expected processors for each environment
+
+## Troubleshooting
+
+### No Processors Shown
+
+If no processors are displayed:
+1. Verify the metrics endpoint is accessible
+2. Check that TrustGraph processors are running
+3. Ensure processors are properly configured to export metrics
+4. Verify the metrics URL is correct
+
+### Connection Issues
+
+For connection problems:
+1. Test network connectivity to the metrics endpoint
+2. Verify the metrics service is running
+3. Check firewall rules and network policies
+4. Ensure the correct port is being used
+
+### Metrics Format Issues
+
+If the command fails with JSON parsing errors:
+1. Verify the metrics endpoint returns Prometheus-compatible data
+2. Check that the `processor_info` metric exists
+3. Ensure the metrics service is properly configured
+
+## Related Commands
+
+- [`tg-show-flow-state`](tg-show-flow-state.md) - Display flow processor states
+- [`tg-show-config`](tg-show-config.md) - Show TrustGraph configuration
+- [`tg-show-token-costs`](tg-show-token-costs.md) - Display token usage costs
+- [`tg-show-library-processing`](tg-show-library-processing.md) - Show library processing status
+
+## See Also
+
+- TrustGraph Processor Documentation
+- Prometheus Metrics Configuration
+- TrustGraph Monitoring Guide
\ No newline at end of file
diff --git a/docs/cli/tg-show-token-rate.md b/docs/cli/tg-show-token-rate.md
new file mode 100644
index 00000000..99cd1193
--- /dev/null
+++ b/docs/cli/tg-show-token-rate.md
@@ -0,0 +1,246 @@
+# tg-show-token-rate
+
+## Synopsis
+
+```
+tg-show-token-rate [OPTIONS]
+```
+
+## Description
+
+The `tg-show-token-rate` command displays a live stream of token usage rates from TrustGraph processors. It monitors both input and output tokens, showing instantaneous rates and cumulative averages over time. This command is essential for monitoring LLM token consumption and understanding processing throughput.
+
+The command queries the metrics endpoint for token usage data and displays:
+- Input token rates (tokens per second)
+- Output token rates (tokens per second) 
+- Total token rates (combined input + output)
+
+All rates are calculated as averages since the command started running.
+
+## Options
+
+- `-m, --metrics-url URL`
+  - Metrics endpoint URL to query for token information
+  - Default: `http://localhost:8088/api/metrics`
+  - Should point to a Prometheus-compatible metrics endpoint
+
+- `-p, --period SECONDS`
+  - Sampling period in seconds between measurements
+  - Default: `1`
+  - Controls how frequently token rates are updated
+
+- `-n, --number-samples COUNT`
+  - Number of samples to collect before stopping
+  - Default: `100`
+  - Set to a large value for continuous monitoring
+
+- `-h, --help`
+  - Show help message and exit
+
+## Examples
+
+### Basic Usage
+
+Monitor token rates with default settings (1-second intervals, 100 samples):
+```bash
+tg-show-token-rate
+```
+
+### Custom Sampling Period
+
+Monitor token rates with 5-second intervals:
+```bash
+tg-show-token-rate --period 5
+```
+
+### Continuous Monitoring
+
+Monitor token rates continuously (1000 samples):
+```bash
+tg-show-token-rate -n 1000
+```
+
+### Remote Monitoring
+
+Monitor token rates from a remote TrustGraph instance:
+```bash
+tg-show-token-rate -m http://10.0.1.100:8088/api/metrics
+```
+
+### High-Frequency Monitoring
+
+Monitor token rates with sub-second precision:
+```bash
+tg-show-token-rate --period 0.5 --number-samples 200
+```
+
+## Output Format
+
+The command displays a table with continuously updated token rates:
+```
+     Input     Output      Total
+     -----     ------      -----
+      12.3       8.7       21.0
+      15.2      10.1       25.3
+      18.7      12.4       31.1
+      ...
+```
+
+Each row shows:
+- **Input**: Average input tokens per second since monitoring started
+- **Output**: Average output tokens per second since monitoring started  
+- **Total**: Combined input + output tokens per second
+
+## Advanced Usage
+
+### Token Rate Analysis
+
+Create a script to analyze token usage patterns:
+```bash
+#!/bin/bash
+echo "Starting token rate analysis..."
+tg-show-token-rate --period 2 --number-samples 60 > token_rates.txt
+echo "Analysis complete. Data saved to token_rates.txt"
+```
+
+### Performance Monitoring
+
+Monitor token rates during load testing:
+```bash
+#!/bin/bash
+echo "Starting load test monitoring..."
+tg-show-token-rate --period 1 --number-samples 300 | tee load_test_tokens.log
+```
+
+### Alert on High Token Usage
+
+Create an alert script for excessive token consumption:
+```bash
+#!/bin/bash
+tg-show-token-rate -n 10 -p 5 | tail -n 1 | awk '{
+    if ($3 > 100) {
+        print "WARNING: High token rate detected:", $3, "tokens/sec"
+        exit 1
+    }
+}'
+```
+
+### Cost Estimation
+
+Estimate token costs during processing:
+```bash
+#!/bin/bash
+echo "Monitoring token usage for cost estimation..."
+tg-show-token-rate --period 10 --number-samples 36 | \
+awk 'NR>2 {total+=$3} END {print "Average tokens/sec:", total/NR-2}'
+```
+
+## Error Handling
+
+The command handles various error conditions:
+
+- **Connection errors**: If the metrics endpoint is unavailable
+- **Invalid JSON**: If the metrics response is malformed
+- **Missing metrics**: If token metrics are not found
+- **Network timeouts**: If requests to the metrics endpoint time out
+
+Common error scenarios:
+```bash
+# Metrics endpoint not available
+tg-show-token-rate -m http://invalid-host:8088/api/metrics
+# Output: Exception: [Connection error details]
+
+# Invalid period value
+tg-show-token-rate --period 0
+# Output: Exception: [Invalid period error]
+```
+
+## Integration with Other Commands
+
+### With Cost Monitoring
+
+Combine with token cost analysis:
+```bash
+echo "=== Token Rates ==="
+tg-show-token-rate -n 5 -p 2
+echo
+echo "=== Token Costs ==="
+tg-show-token-costs
+```
+
+### With Processor State
+
+Monitor tokens alongside processor health:
+```bash
+echo "=== Processor States ==="
+tg-show-processor-state
+echo
+echo "=== Token Rates ==="
+tg-show-token-rate -n 10 -p 1
+```
+
+### With Flow Monitoring
+
+Track token usage per flow:
+```bash
+#!/bin/bash
+echo "=== Active Flows ==="
+tg-show-flows
+echo
+echo "=== Token Usage ==="
+tg-show-token-rate -n 20 -p 3
+```
+
+## Best Practices
+
+1. **Baseline Monitoring**: Establish baseline token rates for normal operation
+2. **Alert Thresholds**: Set up alerts for unusually high token consumption
+3. **Cost Tracking**: Monitor token rates to estimate operational costs
+4. **Load Testing**: Use during load testing to understand capacity limits
+5. **Historical Analysis**: Save token rate data for trend analysis
+
+## Troubleshooting
+
+### No Token Data
+
+If no token rates are displayed:
+1. Verify that TrustGraph processors are actively processing requests
+2. Check that token metrics are being exported properly
+3. Ensure the metrics endpoint is accessible
+4. Verify that LLM services are receiving requests
+
+### Inconsistent Rates
+
+For inconsistent or erratic token rates:
+1. Check for network issues affecting metrics collection
+2. Verify that the sampling period is appropriate for your workload
+3. Ensure multiple processors aren't conflicting
+4. Check system resources (CPU, memory) on the TrustGraph instance
+
+### High Token Rates
+
+If token rates are unexpectedly high:
+1. Investigate the types of queries being processed
+2. Check for inefficient prompts or large document processing
+3. Verify that caching is working properly
+4. Consider if the workload justifies the token usage
+
+## Performance Considerations
+
+- **Sampling Frequency**: Higher frequencies provide more granular data but consume more resources
+- **Network Latency**: Consider network latency when setting sampling periods
+- **Metrics Storage**: Long monitoring sessions generate significant data
+- **Resource Usage**: The command itself uses minimal resources
+
+## Related Commands
+
+- [`tg-show-token-costs`](tg-show-token-costs.md) - Display token usage costs
+- [`tg-show-processor-state`](tg-show-processor-state.md) - Show processor states
+- [`tg-show-flow-state`](tg-show-flow-state.md) - Display flow processor states
+- [`tg-show-config`](tg-show-config.md) - Show TrustGraph configuration
+
+## See Also
+
+- TrustGraph Token Management Documentation
+- Prometheus Metrics Configuration
+- LLM Cost Optimization Guide
\ No newline at end of file
diff --git a/docs/cli/tg-show-tools.md b/docs/cli/tg-show-tools.md
new file mode 100644
index 00000000..9abaca2e
--- /dev/null
+++ b/docs/cli/tg-show-tools.md
@@ -0,0 +1,283 @@
+# tg-show-tools
+
+## Synopsis
+
+```
+tg-show-tools [OPTIONS]
+```
+
+## Description
+
+The `tg-show-tools` command displays the current agent tool configuration from TrustGraph. It retrieves and presents detailed information about all available tools that agents can use, including their descriptions, arguments, and parameter types.
+
+This command is useful for:
+- Understanding available agent tools and their capabilities
+- Debugging agent tool configuration issues
+- Documenting the current tool set
+- Verifying tool definitions and argument specifications
+
+The command queries the TrustGraph API to fetch the tool index and individual tool definitions, then presents them in a formatted table for easy reading.
+
+## Options
+
+- `-u, --api-url URL`
+  - TrustGraph API URL to query for tool configuration
+  - Default: `http://localhost:8088/` (or `TRUSTGRAPH_URL` environment variable)
+  - Should point to a running TrustGraph API instance
+
+- `-h, --help`
+  - Show help message and exit
+
+## Examples
+
+### Basic Usage
+
+Display all available agent tools using the default API URL:
+```bash
+tg-show-tools
+```
+
+### Custom API URL
+
+Display tools from a specific TrustGraph instance:
+```bash
+tg-show-tools -u http://trustgraph.example.com:8088/
+```
+
+### Remote Instance
+
+Query tools from a remote TrustGraph deployment:
+```bash
+tg-show-tools --api-url http://10.0.1.100:8088/
+```
+
+### Using Environment Variable
+
+Set the API URL via environment variable:
+```bash
+export TRUSTGRAPH_URL=http://production.trustgraph.com:8088/
+tg-show-tools
+```
+
+## Output Format
+
+The command displays each tool in a detailed table format:
+```
+web-search:
++-------------+----------------------------------------------------------------------+
+| id          | web-search                                                           |
++-------------+----------------------------------------------------------------------+
+| name        | Web Search                                                           |
++-------------+----------------------------------------------------------------------+
+| description | Search the web for information using a search engine                |
++-------------+----------------------------------------------------------------------+
+| arg 0       | query: string                                                        |
+|             | The search query to execute                                          |
++-------------+----------------------------------------------------------------------+
+| arg 1       | max_results: integer                                                 |
+|             | Maximum number of search results to return                           |
++-------------+----------------------------------------------------------------------+
+
+file-read:
++-------------+----------------------------------------------------------------------+
+| id          | file-read                                                            |
++-------------+----------------------------------------------------------------------+
+| name        | File Reader                                                          |
++-------------+----------------------------------------------------------------------+
+| description | Read contents of a file from the filesystem                         |
++-------------+----------------------------------------------------------------------+
+| arg 0       | path: string                                                         |
+|             | Path to the file to read                                             |
++-------------+----------------------------------------------------------------------+
+```
+
+For each tool, the output includes:
+- **id**: Unique identifier for the tool
+- **name**: Human-readable name of the tool
+- **description**: Detailed description of what the tool does
+- **arg N**: Arguments the tool accepts, with name, type, and description
+
+## Advanced Usage
+
+### Tool Inventory
+
+Create a complete inventory of available tools:
+```bash
+#!/bin/bash
+echo "=== TrustGraph Agent Tools Inventory ==="
+echo "Generated on: $(date)"
+echo
+tg-show-tools > tools_inventory.txt
+echo "Inventory saved to tools_inventory.txt"
+```
+
+### Tool Comparison
+
+Compare tools across different environments:
+```bash
+#!/bin/bash
+echo "=== Development Tools ==="
+tg-show-tools -u http://dev.trustgraph.com:8088/ > dev_tools.txt
+echo
+echo "=== Production Tools ==="
+tg-show-tools -u http://prod.trustgraph.com:8088/ > prod_tools.txt
+echo
+diff dev_tools.txt prod_tools.txt
+```
+
+### Tool Documentation
+
+Generate documentation for agent tools:
+```bash
+#!/bin/bash
+echo "# Available Agent Tools" > AGENT_TOOLS.md
+echo "" >> AGENT_TOOLS.md
+echo "Generated on: $(date)" >> AGENT_TOOLS.md
+echo "" >> AGENT_TOOLS.md
+tg-show-tools >> AGENT_TOOLS.md
+```
+
+### Tool Configuration Validation
+
+Validate tool configuration after updates:
+```bash
+#!/bin/bash
+echo "Validating tool configuration..."
+if tg-show-tools > /dev/null 2>&1; then
+    echo "✓ Tool configuration is valid"
+    tool_count=$(tg-show-tools | grep -c "^[a-zA-Z].*:$")
+    echo "✓ Found $tool_count tools"
+else
+    echo "✗ Tool configuration validation failed"
+    exit 1
+fi
+```
+
+## Error Handling
+
+The command handles various error conditions:
+
+- **API connection errors**: If the TrustGraph API is unavailable
+- **Authentication errors**: If API access is denied
+- **Invalid configuration**: If tool configuration is malformed
+- **Network timeouts**: If API requests time out
+
+Common error scenarios:
+```bash
+# API not available
+tg-show-tools -u http://invalid-host:8088/
+# Output: Exception: [Connection error details]
+
+# Invalid API URL
+tg-show-tools --api-url "not-a-url"
+# Output: Exception: [URL parsing error]
+
+# Configuration not found
+# Output: Exception: [Configuration retrieval error]
+```
+
+## Integration with Other Commands
+
+### With Agent Configuration
+
+Display tools alongside agent configuration:
+```bash
+echo "=== Agent Tools ==="
+tg-show-tools
+echo
+echo "=== Agent Configuration ==="
+tg-show-config
+```
+
+### With Flow Analysis
+
+Understand tools used in flows:
+```bash
+echo "=== Available Tools ==="
+tg-show-tools
+echo
+echo "=== Active Flows ==="
+tg-show-flows
+```
+
+### With Prompt Analysis
+
+Analyze tool usage in prompts:
+```bash
+echo "=== Agent Tools ==="
+tg-show-tools | grep -E "^[a-zA-Z].*:$"
+echo
+echo "=== Available Prompts ==="
+tg-show-prompts
+```
+
+## Best Practices
+
+1. **Regular Documentation**: Keep tool documentation updated
+2. **Version Control**: Track tool configuration changes
+3. **Testing**: Test tool functionality after configuration changes
+4. **Security**: Review tool permissions and capabilities
+5. **Monitoring**: Monitor tool usage and performance
+
+## Troubleshooting
+
+### No Tools Displayed
+
+If no tools are shown:
+1. Verify the TrustGraph API is running and accessible
+2. Check that tool configuration has been properly loaded
+3. Ensure the API URL is correct
+4. Verify network connectivity
+
+### Incomplete Tool Information
+
+If tool information is missing or incomplete:
+1. Check the tool configuration files
+2. Verify the tool index is properly maintained
+3. Ensure tool definitions are valid JSON
+4. Check for configuration loading errors
+
+### Tool Configuration Errors
+
+If tools are not working as expected:
+1. Validate tool definitions against the schema
+2. Check for missing or invalid arguments
+3. Verify tool implementation is available
+4. Review agent logs for tool execution errors
+
+## Tool Management
+
+### Adding New Tools
+
+After adding new tools to the system:
+```bash
+# Verify the new tool appears
+tg-show-tools | grep "new-tool-name"
+
+# Test the tool configuration
+tg-show-tools > current_tools.txt
+```
+
+### Removing Tools
+
+After removing tools:
+```bash
+# Verify the tool is no longer listed
+tg-show-tools | grep -v "removed-tool-name"
+
+# Update tool documentation
+tg-show-tools > updated_tools.txt
+```
+
+## Related Commands
+
+- [`tg-show-config`](tg-show-config.md) - Show TrustGraph configuration
+- [`tg-show-prompts`](tg-show-prompts.md) - Display available prompts
+- [`tg-show-flows`](tg-show-flows.md) - Show active flows
+- [`tg-invoke-agent`](tg-invoke-agent.md) - Invoke agent with tools
+
+## See Also
+
+- TrustGraph Agent Documentation
+- Tool Configuration Guide
+- Agent API Reference
\ No newline at end of file