More CLI

2026-07-21 11:11:03 +02:00 · 2025-07-03 12:12:04 +01:00 · 2025-07-03 12:12:04 +01:00 · e75f8ca96d
commit e75f8ca96d
parent 1e462887b7
2 changed files with 1127 additions and 0 deletions
--- a/docs/cli/tg-save-doc-embeds.md
+++ b/docs/cli/tg-save-doc-embeds.md
@ -0,0 +1,609 @@
+# tg-save-doc-embeds
+
+Saves document embeddings from TrustGraph processing streams to MessagePack format files.
+
+## Synopsis
+
+```bash
+tg-save-doc-embeds -o OUTPUT_FILE [options]
+```
+
+## Description
+
+The `tg-save-doc-embeds` command connects to TrustGraph's document embeddings export stream and saves the embeddings to a file in MessagePack format. This is useful for creating backups of document embeddings, exporting data for analysis, or preparing data for migration between systems.
+
+The command should typically be started before document processing begins to capture all embeddings as they are generated.
+
+## Options
+
+### Required Arguments
+
+- `-o, --output-file FILE`: Output file for saved embeddings
+
+### Optional Arguments
+
+- `-u, --url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_API` or `http://localhost:8088/`)
+- `-f, --flow-id ID`: Flow instance ID to monitor (default: `default`)
+- `--format FORMAT`: Output format - `msgpack` or `json` (default: `msgpack`)
+- `--user USER`: Filter by user ID (default: no filter)
+- `--collection COLLECTION`: Filter by collection ID (default: no filter)
+
+## Examples
+
+### Basic Document Embeddings Export
+```bash
+tg-save-doc-embeds -o document-embeddings.msgpack
+```
+
+### Export from Specific Flow
+```bash
+tg-save-doc-embeds \
+  -o research-embeddings.msgpack \
+  -f "research-processing-flow"
+```
+
+### Filter by User and Collection
+```bash
+tg-save-doc-embeds \
+  -o filtered-embeddings.msgpack \
+  --user "research-team" \
+  --collection "research-docs"
+```
+
+### Export to JSON Format
+```bash
+tg-save-doc-embeds \
+  -o embeddings.json \
+  --format json
+```
+
+### Production Backup
+```bash
+tg-save-doc-embeds \
+  -o "backup-$(date +%Y%m%d-%H%M%S).msgpack" \
+  -u https://production-api.company.com/ \
+  -f "production-flow"
+```
+
+## Output Format
+
+### MessagePack Structure
+Document embeddings are saved as MessagePack records:
+
+```json
+["de", {
+  "m": {
+    "i": "document-id",
+    "m": [{"metadata": "objects"}],
+    "u": "user-id",
+    "c": "collection-id"
+  },
+  "c": [{
+    "c": "text chunk content",
+    "v": [0.1, 0.2, 0.3, ...]
+  }]
+}]
+```
+
+### Components
+- **Record Type**: `"de"` indicates document embeddings
+- **Metadata** (`m`): Document information and context
+- **Chunks** (`c`): Text chunks with their vector embeddings
+
+## Use Cases
+
+### Backup Creation
+```bash
+# Create regular backups of document embeddings
+create_embeddings_backup() {
+  local backup_dir="embeddings-backups"
+  local timestamp=$(date +%Y%m%d_%H%M%S)
+  local backup_file="$backup_dir/embeddings-$timestamp.msgpack"
+  
+  mkdir -p "$backup_dir"
+  
+  echo "Creating embeddings backup: $backup_file"
+  
+  # Start backup process
+  tg-save-doc-embeds -o "$backup_file" &
+  save_pid=$!
+  
+  echo "Backup process started (PID: $save_pid)"
+  echo "To stop: kill $save_pid"
+  echo "Backup file: $backup_file"
+  
+  # Optionally wait for a specific duration
+  # sleep 3600  # Run for 1 hour
+  # kill $save_pid
+}
+
+# Create backup
+create_embeddings_backup
+```
+
+### Data Migration Preparation
+```bash
+# Prepare embeddings for migration
+prepare_migration_data() {
+  local source_env="$1"
+  local collection="$2"
+  local migration_file="migration-$(date +%Y%m%d).msgpack"
+  
+  echo "Preparing migration data from: $source_env"
+  echo "Collection: $collection"
+  
+  # Export embeddings from source
+  tg-save-doc-embeds \
+    -o "$migration_file" \
+    -u "http://$source_env:8088/" \
+    --collection "$collection" &
+  
+  export_pid=$!
+  
+  # Let it run for specified time to capture data
+  echo "Capturing embeddings for migration..."
+  echo "Process PID: $export_pid"
+  
+  # In practice, you'd run this for the duration needed
+  # sleep 1800  # 30 minutes
+  # kill $export_pid
+  
+  echo "Migration data will be saved to: $migration_file"
+}
+
+# Prepare migration from dev to production
+prepare_migration_data "dev-server" "processed-docs"
+```
+
+### Continuous Export
+```bash
+# Continuous embeddings export with rotation
+continuous_export() {
+  local output_dir="continuous-exports"
+  local rotation_hours=24
+  local file_prefix="embeddings"
+  
+  mkdir -p "$output_dir"
+  
+  while true; do
+    timestamp=$(date +%Y%m%d_%H%M%S)
+    output_file="$output_dir/${file_prefix}-${timestamp}.msgpack"
+    
+    echo "Starting export to: $output_file"
+    
+    # Start export for specified duration
+    timeout ${rotation_hours}h tg-save-doc-embeds -o "$output_file"
+    
+    # Compress completed file
+    gzip "$output_file"
+    
+    echo "Export completed and compressed: ${output_file}.gz"
+    
+    # Optional: clean up old files
+    find "$output_dir" -name "*.msgpack.gz" -mtime +30 -delete
+    
+    # Brief pause before next rotation
+    sleep 60
+  done
+}
+
+# Start continuous export (run in background)
+continuous_export &
+```
+
+### Analysis and Research
+```bash
+# Export embeddings for research analysis
+export_for_research() {
+  local research_topic="$1"
+  local output_file="research-${research_topic}-$(date +%Y%m%d).msgpack"
+  
+  echo "Exporting embeddings for research: $research_topic"
+  
+  # Start export with filtering
+  tg-save-doc-embeds \
+    -o "$output_file" \
+    --collection "$research_topic" &
+  
+  export_pid=$!
+  
+  echo "Research export started (PID: $export_pid)"
+  echo "Output: $output_file"
+  
+  # Create analysis script
+  cat > "analyze-${research_topic}.sh" << EOF
+#!/bin/bash
+# Analysis script for $research_topic embeddings
+
+echo "Analyzing $research_topic embeddings..."
+
+# Basic statistics
+echo "=== Basic Statistics ==="
+tg-dump-msgpack -i "$output_file" --summary
+
+# Detailed analysis
+echo "=== Detailed Analysis ==="
+tg-dump-msgpack -i "$output_file" | head -10
+
+echo "Analysis complete for $research_topic"
+EOF
+  
+  chmod +x "analyze-${research_topic}.sh"
+  echo "Analysis script created: analyze-${research_topic}.sh"
+}
+
+# Export for different research topics
+export_for_research "cybersecurity"
+export_for_research "climate-change"
+```
+
+## Advanced Usage
+
+### Selective Export
+```bash
+# Export embeddings with multiple filters
+selective_export() {
+  local users=("user1" "user2" "user3")
+  local collections=("docs1" "docs2")
+  
+  for user in "${users[@]}"; do
+    for collection in "${collections[@]}"; do
+      output_file="embeddings-${user}-${collection}.msgpack"
+      
+      echo "Exporting for user: $user, collection: $collection"
+      
+      tg-save-doc-embeds \
+        -o "$output_file" \
+        --user "$user" \
+        --collection "$collection" &
+      
+      # Store PID for later management
+      echo $! > "${output_file}.pid"
+    done
+  done
+  
+  echo "All selective exports started"
+}
+```
+
+### Monitoring and Statistics
+```bash
+# Monitor export progress with statistics
+monitor_export() {
+  local output_file="$1"
+  local pid_file="${output_file}.pid"
+  
+  if [ ! -f "$pid_file" ]; then
+    echo "PID file not found: $pid_file"
+    return 1
+  fi
+  
+  local export_pid=$(cat "$pid_file")
+  
+  echo "Monitoring export (PID: $export_pid)..."
+  echo "Output file: $output_file"
+  
+  while kill -0 "$export_pid" 2>/dev/null; do
+    if [ -f "$output_file" ]; then
+      file_size=$(stat -c%s "$output_file" 2>/dev/null || echo "0")
+      human_size=$(numfmt --to=iec-i --suffix=B "$file_size")
+      
+      # Try to count embeddings
+      embedding_count=$(tg-dump-msgpack -i "$output_file" 2>/dev/null | grep -c '^\["de"' || echo "0")
+      
+      echo "File size: $human_size, Embeddings: $embedding_count"
+    else
+      echo "Output file not yet created..."
+    fi
+    
+    sleep 30
+  done
+  
+  echo "Export process completed"
+  rm "$pid_file"
+}
+
+# Start export and monitor
+tg-save-doc-embeds -o "monitored-export.msgpack" &
+echo $! > "monitored-export.msgpack.pid"
+monitor_export "monitored-export.msgpack"
+```
+
+### Export Validation
+```bash
+# Validate exported embeddings
+validate_export() {
+  local export_file="$1"
+  
+  echo "Validating export file: $export_file"
+  
+  # Check file exists and has content
+  if [ ! -s "$export_file" ]; then
+    echo "✗ Export file is empty or missing"
+    return 1
+  fi
+  
+  # Check MessagePack format
+  if tg-dump-msgpack -i "$export_file" --summary > /dev/null 2>&1; then
+    echo "✓ Valid MessagePack format"
+  else
+    echo "✗ Invalid MessagePack format"
+    return 1
+  fi
+  
+  # Check for document embeddings
+  embedding_count=$(tg-dump-msgpack -i "$export_file" | grep -c '^\["de"' || echo "0")
+  
+  if [ "$embedding_count" -gt 0 ]; then
+    echo "✓ Contains $embedding_count document embeddings"
+  else
+    echo "✗ No document embeddings found"
+    return 1
+  fi
+  
+  # Get vector dimension information
+  summary=$(tg-dump-msgpack -i "$export_file" --summary)
+  if echo "$summary" | grep -q "Vector dimension:"; then
+    dimension=$(echo "$summary" | grep "Vector dimension:" | awk '{print $3}')
+    echo "✓ Vector dimension: $dimension"
+  else
+    echo "⚠ Could not determine vector dimension"
+  fi
+  
+  echo "Validation completed successfully"
+}
+```
+
+### Export Scheduling
+```bash
+# Scheduled export with cron-like functionality
+schedule_export() {
+  local schedule="$1"  # e.g., "daily", "hourly", "weekly"
+  local output_prefix="$2"
+  
+  case "$schedule" in
+    "hourly")
+      interval=3600
+      ;;
+    "daily")
+      interval=86400
+      ;;
+    "weekly")
+      interval=604800
+      ;;
+    *)
+      echo "Invalid schedule: $schedule"
+      return 1
+      ;;
+  esac
+  
+  echo "Starting $schedule exports with prefix: $output_prefix"
+  
+  while true; do
+    timestamp=$(date +%Y%m%d_%H%M%S)
+    output_file="${output_prefix}-${timestamp}.msgpack"
+    
+    echo "Starting scheduled export: $output_file"
+    
+    # Run export for the scheduled interval
+    timeout ${interval}s tg-save-doc-embeds -o "$output_file"
+    
+    # Validate and compress
+    if validate_export "$output_file"; then
+      gzip "$output_file"
+      echo "✓ Export completed and compressed: ${output_file}.gz"
+    else
+      echo "✗ Export validation failed: $output_file"
+      mv "$output_file" "${output_file}.failed"
+    fi
+    
+    # Brief pause before next cycle
+    sleep 60
+  done
+}
+
+# Start daily scheduled exports
+schedule_export "daily" "daily-embeddings" &
+```
+
+## Performance Considerations
+
+### Memory Management
+```bash
+# Monitor memory usage during export
+monitor_memory_export() {
+  local output_file="$1"
+  
+  # Start export
+  tg-save-doc-embeds -o "$output_file" &
+  export_pid=$!
+  
+  echo "Monitoring memory usage for export (PID: $export_pid)..."
+  
+  while kill -0 "$export_pid" 2>/dev/null; do
+    memory_usage=$(ps -p "$export_pid" -o rss= 2>/dev/null | awk '{print $1/1024}')
+    
+    if [ -n "$memory_usage" ]; then
+      echo "Memory usage: ${memory_usage}MB"
+    fi
+    
+    sleep 10
+  done
+  
+  echo "Export completed"
+}
+```
+
+### Network Optimization
+```bash
+# Optimize for network conditions
+network_optimized_export() {
+  local output_file="$1"
+  local api_url="$2"
+  
+  echo "Starting network-optimized export..."
+  
+  # Use compression and buffering
+  tg-save-doc-embeds \
+    -o "$output_file" \
+    -u "$api_url" \
+    --format msgpack &  # MessagePack is more compact than JSON
+  
+  export_pid=$!
+  
+  # Monitor network usage
+  echo "Monitoring export (PID: $export_pid)..."
+  
+  while kill -0 "$export_pid" 2>/dev/null; do
+    # Monitor network connections
+    connections=$(netstat -an | grep ":8088" | wc -l)
+    echo "Active connections: $connections"
+    sleep 30
+  done
+}
+```
+
+## Error Handling
+
+### Connection Issues
+```bash
+Exception: WebSocket connection failed
+```
+**Solution**: Check API URL and ensure TrustGraph WebSocket service is running.
+
+### Disk Space Issues
+```bash
+Exception: No space left on device
+```
+**Solution**: Free up disk space or use a different output location.
+
+### Permission Errors
+```bash
+Exception: Permission denied
+```
+**Solution**: Check write permissions for the output file location.
+
+### Memory Issues
+```bash
+MemoryError: Unable to allocate memory
+```
+**Solution**: Monitor memory usage and consider using smaller export windows.
+
+## Integration with Other Commands
+
+### Complete Backup Workflow
+```bash
+# Complete backup and restore workflow
+backup_restore_workflow() {
+  local backup_file="embeddings-backup.msgpack"
+  
+  echo "=== Backup Phase ==="
+  
+  # Create backup
+  tg-save-doc-embeds -o "$backup_file" &
+  backup_pid=$!
+  
+  # Let it run for a while
+  sleep 300  # 5 minutes
+  kill $backup_pid
+  
+  echo "Backup created: $backup_file"
+  
+  # Validate backup
+  validate_export "$backup_file"
+  
+  echo "=== Restore Phase ==="
+  
+  # Restore from backup (to different collection)
+  tg-load-doc-embeds -i "$backup_file" --collection "restored"
+  
+  echo "Backup and restore workflow completed"
+}
+```
+
+### Analysis Pipeline
+```bash
+# Export and analyze embeddings
+export_analyze_pipeline() {
+  local topic="$1"
+  local export_file="analysis-${topic}.msgpack"
+  
+  echo "Starting export and analysis pipeline for: $topic"
+  
+  # Export embeddings
+  tg-save-doc-embeds \
+    -o "$export_file" \
+    --collection "$topic" &
+  
+  export_pid=$!
+  
+  # Run for analysis duration
+  sleep 600  # 10 minutes
+  kill $export_pid
+  
+  # Analyze exported data
+  echo "Analyzing exported embeddings..."
+  tg-dump-msgpack -i "$export_file" --summary
+  
+  # Count embeddings by user
+  echo "Embeddings by user:"
+  tg-dump-msgpack -i "$export_file" | \
+    jq -r '.[1].m.u' | \
+    sort | uniq -c
+  
+  echo "Analysis pipeline completed"
+}
+```
+
+## Environment Variables
+
+- `TRUSTGRAPH_API`: Default API URL
+
+## Related Commands
+
+- [`tg-load-doc-embeds`](tg-load-doc-embeds.md) - Load document embeddings from files
+- [`tg-dump-msgpack`](tg-dump-msgpack.md) - Analyze MessagePack files
+- [`tg-show-flows`](tg-show-flows.md) - List available flows for monitoring
+
+## API Integration
+
+This command uses TrustGraph's WebSocket API for document embeddings export, specifically the `/api/v1/flow/{flow-id}/export/document-embeddings` endpoint.
+
+## Best Practices
+
+1. **Start Early**: Begin export before processing starts to capture all data
+2. **Monitoring**: Monitor export progress and file sizes
+3. **Validation**: Always validate exported files
+4. **Compression**: Use compression for long-term storage
+5. **Rotation**: Implement file rotation for continuous exports
+6. **Backup**: Keep multiple backup copies in different locations
+7. **Documentation**: Document export schedules and procedures
+
+## Troubleshooting
+
+### No Data Captured
+```bash
+# Check if processing is generating embeddings
+tg-show-flows | grep processing
+
+# Verify WebSocket connection
+netstat -an | grep :8088
+```
+
+### Large File Issues
+```bash
+# Monitor file growth
+watch -n 5 'ls -lh *.msgpack'
+
+# Check available disk space
+df -h
+```
+
+### Process Management
+```bash
+# List running export processes
+ps aux | grep tg-save-doc-embeds
+
+# Kill stuck processes
+pkill -f tg-save-doc-embeds
+```
--- a/docs/cli/tg-show-flow-state.md
+++ b/docs/cli/tg-show-flow-state.md
@ -0,0 +1,518 @@
+# tg-show-flow-state
+
+Displays the processor states for a specific flow and its associated flow class.
+
+## Synopsis
+
+```bash
+tg-show-flow-state [options]
+```
+
+## Description
+
+The `tg-show-flow-state` command shows the current state of processors within a specific TrustGraph flow instance and its corresponding flow class. It queries the metrics system to determine which processing components are running and displays their status with visual indicators.
+
+This command is essential for monitoring flow health and debugging processing issues.
+
+## Options
+
+### Optional Arguments
+
+- `-f, --flow-id ID`: Flow instance ID to examine (default: `default`)
+- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
+- `-m, --metrics-url URL`: Metrics API URL (default: `http://localhost:8088/api/metrics`)
+
+## Examples
+
+### Check Default Flow State
+```bash
+tg-show-flow-state
+```
+
+### Check Specific Flow
+```bash
+tg-show-flow-state -f "production-flow"
+```
+
+### Use Custom Metrics URL
+```bash
+tg-show-flow-state \
+  -f "research-flow" \
+  -m "http://metrics-server:8088/api/metrics"
+```
+
+### Check Flow in Different Environment
+```bash
+tg-show-flow-state \
+  -f "staging-flow" \
+  -u "http://staging:8088/" \
+  -m "http://staging:8088/api/metrics"
+```
+
+## Output Format
+
+The command displays processor states for both the flow instance and its flow class:
+
+```
+Flow production-flow
+- pdf-processor                💚
+- text-extractor              💚
+- embeddings-generator        💚
+- knowledge-builder           ❌
+- document-indexer            💚
+
+Class document-processing-v2
+- base-pdf-processor          💚
+- base-text-extractor         💚
+- base-embeddings-generator   💚
+- base-knowledge-builder      💚
+- base-document-indexer       💚
+```
+
+### Status Indicators
+- **💚 (Green Heart)**: Processor is running and healthy
+- **❌ (Red X)**: Processor is not running or unhealthy
+
+### Information Displayed
+- **Flow Section**: Shows the state of processors in the specific flow instance
+- **Class Section**: Shows the state of processors in the flow class template
+- **Processor Names**: Individual processing components within the flow
+
+## Use Cases
+
+### Flow Health Monitoring
+```bash
+# Monitor flow health continuously
+monitor_flow_health() {
+  local flow_id="$1"
+  local interval="${2:-30}"  # Default 30 seconds
+  
+  echo "Monitoring flow health: $flow_id"
+  echo "Refresh interval: ${interval}s"
+  echo "Press Ctrl+C to stop"
+  
+  while true; do
+    clear
+    echo "Flow Health Monitor - $(date)"
+    echo "=============================="
+    
+    tg-show-flow-state -f "$flow_id"
+    
+    sleep "$interval"
+  done
+}
+
+# Monitor production flow
+monitor_flow_health "production-flow" 15
+```
+
+### Debugging Processing Issues
+```bash
+# Comprehensive flow debugging
+debug_flow_issues() {
+  local flow_id="$1"
+  
+  echo "Debugging flow: $flow_id"
+  echo "======================="
+  
+  # Check flow state
+  echo "1. Processor States:"
+  tg-show-flow-state -f "$flow_id"
+  
+  # Check flow configuration
+  echo -e "\n2. Flow Configuration:"
+  tg-show-flows | grep "$flow_id"
+  
+  # Check active processing
+  echo -e "\n3. Active Processing:"
+  tg-show-flows | grep -i processing
+  
+  # Check system resources
+  echo -e "\n4. System Resources:"
+  free -h
+  df -h
+  
+  echo -e "\nDebugging complete for: $flow_id"
+}
+
+# Debug specific flow
+debug_flow_issues "problematic-flow"
+```
+
+### Multi-Flow Status Dashboard
+```bash
+# Create status dashboard for multiple flows
+create_flow_dashboard() {
+  local flows=("$@")
+  
+  echo "TrustGraph Flow Dashboard - $(date)"
+  echo "==================================="
+  
+  for flow in "${flows[@]}"; do
+    echo -e "\n=== Flow: $flow ==="
+    tg-show-flow-state -f "$flow" 2>/dev/null || echo "Flow not found or inaccessible"
+  done
+  
+  echo -e "\n=== Summary ==="
+  echo "Total flows monitored: ${#flows[@]}"
+  echo "Dashboard generated: $(date)"
+}
+
+# Monitor multiple flows
+flows=("production-flow" "research-flow" "development-flow")
+create_flow_dashboard "${flows[@]}"
+```
+
+### Automated Health Checks
+```bash
+# Automated health check with alerts
+health_check_with_alerts() {
+  local flow_id="$1"
+  local alert_email="$2"
+  
+  echo "Performing health check for: $flow_id"
+  
+  # Capture flow state
+  flow_state=$(tg-show-flow-state -f "$flow_id" 2>&1)
+  
+  if [ $? -ne 0 ]; then
+    echo "ERROR: Failed to get flow state"
+    # Send alert email if configured
+    if [ -n "$alert_email" ]; then
+      echo "Flow $flow_id is not responding" | mail -s "TrustGraph Alert" "$alert_email"
+    fi
+    return 1
+  fi
+  
+  # Check for failed processors
+  failed_count=$(echo "$flow_state" | grep -c "❌")
+  
+  if [ "$failed_count" -gt 0 ]; then
+    echo "WARNING: $failed_count processors are not running"
+    echo "$flow_state"
+    
+    # Send alert if configured
+    if [ -n "$alert_email" ]; then
+      echo -e "Flow $flow_id has $failed_count failed processors:\n\n$flow_state" | \
+        mail -s "TrustGraph Health Alert" "$alert_email"
+    fi
+    return 1
+  else
+    echo "✓ All processors are running normally"
+    return 0
+  fi
+}
+
+# Run health check
+health_check_with_alerts "production-flow" "admin@company.com"
+```
+
+## Advanced Usage
+
+### Flow State Comparison
+```bash
+# Compare flow states between environments
+compare_flow_states() {
+  local flow_id="$1"
+  local env1_url="$2"
+  local env2_url="$3"
+  
+  echo "Comparing flow state: $flow_id"
+  echo "Environment 1: $env1_url"
+  echo "Environment 2: $env2_url"
+  echo "================================"
+  
+  # Get states from both environments
+  echo "Environment 1 State:"
+  tg-show-flow-state -f "$flow_id" -u "$env1_url" -m "$env1_url/api/metrics"
+  
+  echo -e "\nEnvironment 2 State:"
+  tg-show-flow-state -f "$flow_id" -u "$env2_url" -m "$env2_url/api/metrics"
+  
+  echo -e "\nComparison complete"
+}
+
+# Compare production vs staging
+compare_flow_states "main-flow" "http://prod:8088" "http://staging:8088"
+```
+
+### Historical State Tracking
+```bash
+# Track flow state over time
+track_flow_state_history() {
+  local flow_id="$1"
+  local log_file="flow_state_history.log"
+  local interval="${2:-60}"  # Default 1 minute
+  
+  echo "Starting flow state tracking: $flow_id"
+  echo "Log file: $log_file"
+  echo "Interval: ${interval}s"
+  
+  while true; do
+    timestamp=$(date '+%Y-%m-%d %H:%M:%S')
+    
+    # Get current state
+    state_output=$(tg-show-flow-state -f "$flow_id" 2>&1)
+    
+    if [ $? -eq 0 ]; then
+      # Count healthy and failed processors
+      healthy_count=$(echo "$state_output" | grep -c "💚")
+      failed_count=$(echo "$state_output" | grep -c "❌")
+      
+      # Log summary
+      echo "$timestamp,$flow_id,$healthy_count,$failed_count" >> "$log_file"
+      
+      # If there are failures, log details
+      if [ "$failed_count" -gt 0 ]; then
+        echo "$timestamp - FAILURES DETECTED in $flow_id:" >> "${log_file}.detailed"
+        echo "$state_output" >> "${log_file}.detailed"
+        echo "---" >> "${log_file}.detailed"
+      fi
+    else
+      echo "$timestamp,$flow_id,ERROR,ERROR" >> "$log_file"
+    fi
+    
+    sleep "$interval"
+  done
+}
+
+# Start tracking (run in background)
+track_flow_state_history "production-flow" 30 &
+```
+
+### State-Based Actions
+```bash
+# Perform actions based on flow state
+state_based_actions() {
+  local flow_id="$1"
+  
+  echo "Checking flow state for automated actions: $flow_id"
+  
+  # Get current state
+  state_output=$(tg-show-flow-state -f "$flow_id")
+  
+  if [ $? -ne 0 ]; then
+    echo "ERROR: Cannot get flow state"
+    return 1
+  fi
+  
+  # Check specific processors
+  if echo "$state_output" | grep -q "pdf-processor.*❌"; then
+    echo "PDF processor is down - attempting restart..."
+    # Restart specific processor (this would need additional commands)
+    # restart_processor "$flow_id" "pdf-processor"
+  fi
+  
+  if echo "$state_output" | grep -q "embeddings-generator.*❌"; then
+    echo "Embeddings generator is down - checking dependencies..."
+    # Check GPU availability, memory, etc.
+    nvidia-smi 2>/dev/null || echo "GPU not available"
+  fi
+  
+  # Count total failures
+  failed_count=$(echo "$state_output" | grep -c "❌")
+  
+  if [ "$failed_count" -gt 3 ]; then
+    echo "CRITICAL: More than 3 processors failed - considering flow restart"
+    # This would trigger more serious recovery actions
+  fi
+}
+```
+
+### Performance Correlation
+```bash
+# Correlate flow state with performance metrics
+correlate_state_performance() {
+  local flow_id="$1"
+  local metrics_url="$2"
+  
+  echo "Correlating flow state with performance for: $flow_id"
+  
+  # Get flow state
+  state_output=$(tg-show-flow-state -f "$flow_id" -m "$metrics_url")
+  healthy_count=$(echo "$state_output" | grep -c "💚")
+  failed_count=$(echo "$state_output" | grep -c "❌")
+  
+  echo "Processors - Healthy: $healthy_count, Failed: $failed_count"
+  
+  # Get performance metrics (this would need additional API calls)
+  # throughput=$(get_flow_throughput "$flow_id" "$metrics_url")
+  # latency=$(get_flow_latency "$flow_id" "$metrics_url")
+  
+  # echo "Performance - Throughput: ${throughput}/min, Latency: ${latency}ms"
+  
+  # Calculate health ratio
+  total_processors=$((healthy_count + failed_count))
+  if [ "$total_processors" -gt 0 ]; then
+    health_ratio=$(echo "scale=2; $healthy_count * 100 / $total_processors" | bc)
+    echo "Health ratio: ${health_ratio}%"
+  fi
+}
+```
+
+## Integration with Monitoring Systems
+
+### Prometheus Integration
+```bash
+# Export flow state metrics to Prometheus format
+export_prometheus_metrics() {
+  local flow_id="$1"
+  local metrics_file="flow_state_metrics.prom"
+  
+  # Get flow state
+  state_output=$(tg-show-flow-state -f "$flow_id")
+  
+  # Count states
+  healthy_count=$(echo "$state_output" | grep -c "💚")
+  failed_count=$(echo "$state_output" | grep -c "❌")
+  
+  # Generate Prometheus metrics
+  cat > "$metrics_file" << EOF
+# HELP trustgraph_flow_processors_healthy Number of healthy processors in flow
+# TYPE trustgraph_flow_processors_healthy gauge
+trustgraph_flow_processors_healthy{flow_id="$flow_id"} $healthy_count
+
+# HELP trustgraph_flow_processors_failed Number of failed processors in flow
+# TYPE trustgraph_flow_processors_failed gauge
+trustgraph_flow_processors_failed{flow_id="$flow_id"} $failed_count
+
+# HELP trustgraph_flow_health_ratio Ratio of healthy processors
+# TYPE trustgraph_flow_health_ratio gauge
+EOF
+  
+  total=$((healthy_count + failed_count))
+  if [ "$total" -gt 0 ]; then
+    ratio=$(echo "scale=4; $healthy_count / $total" | bc)
+    echo "trustgraph_flow_health_ratio{flow_id=\"$flow_id\"} $ratio" >> "$metrics_file"
+  fi
+  
+  echo "Prometheus metrics exported to: $metrics_file"
+}
+```
+
+### Grafana Dashboard Data
+```bash
+# Generate data for Grafana dashboard
+generate_grafana_data() {
+  local flows=("$@")
+  local output_file="grafana_flow_data.json"
+  
+  echo "Generating Grafana dashboard data..."
+  
+  echo "{" > "$output_file"
+  echo "  \"flows\": [" >> "$output_file"
+  
+  for i in "${!flows[@]}"; do
+    flow="${flows[$i]}"
+    
+    # Get flow state
+    state_output=$(tg-show-flow-state -f "$flow" 2>/dev/null)
+    
+    if [ $? -eq 0 ]; then
+      healthy=$(echo "$state_output" | grep -c "💚")
+      failed=$(echo "$state_output" | grep -c "❌")
+    else
+      healthy=0
+      failed=0
+    fi
+    
+    echo "    {" >> "$output_file"
+    echo "      \"flow_id\": \"$flow\"," >> "$output_file"
+    echo "      \"healthy_processors\": $healthy," >> "$output_file"
+    echo "      \"failed_processors\": $failed," >> "$output_file"
+    echo "      \"timestamp\": \"$(date -Iseconds)\"" >> "$output_file"
+    
+    if [ $i -lt $((${#flows[@]} - 1)) ]; then
+      echo "    }," >> "$output_file"
+    else
+      echo "    }" >> "$output_file"
+    fi
+  done
+  
+  echo "  ]" >> "$output_file"
+  echo "}" >> "$output_file"
+  
+  echo "Grafana data generated: $output_file"
+}
+```
+
+## Error Handling
+
+### Flow Not Found
+```bash
+Exception: Flow 'nonexistent-flow' not found
+```
+**Solution**: Verify the flow ID exists with `tg-show-flows`.
+
+### Metrics API Unavailable
+```bash
+Exception: Connection refused to metrics API
+```
+**Solution**: Check metrics URL and ensure metrics service is running.
+
+### Permission Issues
+```bash
+Exception: Access denied to metrics
+```
+**Solution**: Verify permissions for accessing metrics and flow information.
+
+### Invalid Flow State
+```bash
+Exception: Unable to parse flow state
+```
+**Solution**: Check if the flow is properly initialized and processors are configured.
+
+## Environment Variables
+
+- `TRUSTGRAPH_URL`: Default API URL
+
+## Related Commands
+
+- [`tg-show-flows`](tg-show-flows.md) - List all flows
+- [`tg-show-processor-state`](tg-show-processor-state.md) - Show all processor states
+- [`tg-start-flow`](tg-start-flow.md) - Start flow instances
+- [`tg-stop-flow`](tg-stop-flow.md) - Stop flow instances
+
+## API Integration
+
+This command integrates with:
+- TrustGraph Flow API for flow information
+- Prometheus/Metrics API for processor state information
+
+## Best Practices
+
+1. **Regular Monitoring**: Check flow states regularly in production
+2. **Automated Alerts**: Set up automated health checks with alerting
+3. **Historical Tracking**: Maintain historical flow state data
+4. **Integration**: Integrate with monitoring systems like Prometheus/Grafana
+5. **Documentation**: Document expected processor configurations
+6. **Correlation**: Correlate flow state with performance metrics
+7. **Recovery Procedures**: Develop automated recovery procedures for common failures
+
+## Troubleshooting
+
+### No Processors Shown
+```bash
+# Check if flow exists
+tg-show-flows | grep "flow-id"
+
+# Verify metrics service
+curl -s http://localhost:8088/api/metrics/query?query=processor_info
+```
+
+### Inconsistent States
+```bash
+# Check metrics service health
+curl -s http://localhost:8088/api/metrics/health
+
+# Restart metrics collection if needed
+```
+
+### Connection Issues
+```bash
+# Test API connectivity
+curl -s http://localhost:8088/api/v1/flows
+
+# Test metrics connectivity  
+curl -s http://localhost:8088/api/metrics/query?query=up
+```