diff --git a/docs/cli/tg-save-doc-embeds.md b/docs/cli/tg-save-doc-embeds.md new file mode 100644 index 00000000..cdbd7882 --- /dev/null +++ b/docs/cli/tg-save-doc-embeds.md @@ -0,0 +1,609 @@ +# tg-save-doc-embeds + +Saves document embeddings from TrustGraph processing streams to MessagePack format files. + +## Synopsis + +```bash +tg-save-doc-embeds -o OUTPUT_FILE [options] +``` + +## Description + +The `tg-save-doc-embeds` command connects to TrustGraph's document embeddings export stream and saves the embeddings to a file in MessagePack format. This is useful for creating backups of document embeddings, exporting data for analysis, or preparing data for migration between systems. + +The command should typically be started before document processing begins to capture all embeddings as they are generated. + +## Options + +### Required Arguments + +- `-o, --output-file FILE`: Output file for saved embeddings + +### Optional Arguments + +- `-u, --url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_API` or `http://localhost:8088/`) +- `-f, --flow-id ID`: Flow instance ID to monitor (default: `default`) +- `--format FORMAT`: Output format - `msgpack` or `json` (default: `msgpack`) +- `--user USER`: Filter by user ID (default: no filter) +- `--collection COLLECTION`: Filter by collection ID (default: no filter) + +## Examples + +### Basic Document Embeddings Export +```bash +tg-save-doc-embeds -o document-embeddings.msgpack +``` + +### Export from Specific Flow +```bash +tg-save-doc-embeds \ + -o research-embeddings.msgpack \ + -f "research-processing-flow" +``` + +### Filter by User and Collection +```bash +tg-save-doc-embeds \ + -o filtered-embeddings.msgpack \ + --user "research-team" \ + --collection "research-docs" +``` + +### Export to JSON Format +```bash +tg-save-doc-embeds \ + -o embeddings.json \ + --format json +``` + +### Production Backup +```bash +tg-save-doc-embeds \ + -o "backup-$(date +%Y%m%d-%H%M%S).msgpack" \ + -u https://production-api.company.com/ \ + -f "production-flow" +``` + +## Output Format + +### MessagePack Structure +Document embeddings are saved as MessagePack records: + +```json +["de", { + "m": { + "i": "document-id", + "m": [{"metadata": "objects"}], + "u": "user-id", + "c": "collection-id" + }, + "c": [{ + "c": "text chunk content", + "v": [0.1, 0.2, 0.3, ...] + }] +}] +``` + +### Components +- **Record Type**: `"de"` indicates document embeddings +- **Metadata** (`m`): Document information and context +- **Chunks** (`c`): Text chunks with their vector embeddings + +## Use Cases + +### Backup Creation +```bash +# Create regular backups of document embeddings +create_embeddings_backup() { + local backup_dir="embeddings-backups" + local timestamp=$(date +%Y%m%d_%H%M%S) + local backup_file="$backup_dir/embeddings-$timestamp.msgpack" + + mkdir -p "$backup_dir" + + echo "Creating embeddings backup: $backup_file" + + # Start backup process + tg-save-doc-embeds -o "$backup_file" & + save_pid=$! + + echo "Backup process started (PID: $save_pid)" + echo "To stop: kill $save_pid" + echo "Backup file: $backup_file" + + # Optionally wait for a specific duration + # sleep 3600 # Run for 1 hour + # kill $save_pid +} + +# Create backup +create_embeddings_backup +``` + +### Data Migration Preparation +```bash +# Prepare embeddings for migration +prepare_migration_data() { + local source_env="$1" + local collection="$2" + local migration_file="migration-$(date +%Y%m%d).msgpack" + + echo "Preparing migration data from: $source_env" + echo "Collection: $collection" + + # Export embeddings from source + tg-save-doc-embeds \ + -o "$migration_file" \ + -u "http://$source_env:8088/" \ + --collection "$collection" & + + export_pid=$! + + # Let it run for specified time to capture data + echo "Capturing embeddings for migration..." + echo "Process PID: $export_pid" + + # In practice, you'd run this for the duration needed + # sleep 1800 # 30 minutes + # kill $export_pid + + echo "Migration data will be saved to: $migration_file" +} + +# Prepare migration from dev to production +prepare_migration_data "dev-server" "processed-docs" +``` + +### Continuous Export +```bash +# Continuous embeddings export with rotation +continuous_export() { + local output_dir="continuous-exports" + local rotation_hours=24 + local file_prefix="embeddings" + + mkdir -p "$output_dir" + + while true; do + timestamp=$(date +%Y%m%d_%H%M%S) + output_file="$output_dir/${file_prefix}-${timestamp}.msgpack" + + echo "Starting export to: $output_file" + + # Start export for specified duration + timeout ${rotation_hours}h tg-save-doc-embeds -o "$output_file" + + # Compress completed file + gzip "$output_file" + + echo "Export completed and compressed: ${output_file}.gz" + + # Optional: clean up old files + find "$output_dir" -name "*.msgpack.gz" -mtime +30 -delete + + # Brief pause before next rotation + sleep 60 + done +} + +# Start continuous export (run in background) +continuous_export & +``` + +### Analysis and Research +```bash +# Export embeddings for research analysis +export_for_research() { + local research_topic="$1" + local output_file="research-${research_topic}-$(date +%Y%m%d).msgpack" + + echo "Exporting embeddings for research: $research_topic" + + # Start export with filtering + tg-save-doc-embeds \ + -o "$output_file" \ + --collection "$research_topic" & + + export_pid=$! + + echo "Research export started (PID: $export_pid)" + echo "Output: $output_file" + + # Create analysis script + cat > "analyze-${research_topic}.sh" << EOF +#!/bin/bash +# Analysis script for $research_topic embeddings + +echo "Analyzing $research_topic embeddings..." + +# Basic statistics +echo "=== Basic Statistics ===" +tg-dump-msgpack -i "$output_file" --summary + +# Detailed analysis +echo "=== Detailed Analysis ===" +tg-dump-msgpack -i "$output_file" | head -10 + +echo "Analysis complete for $research_topic" +EOF + + chmod +x "analyze-${research_topic}.sh" + echo "Analysis script created: analyze-${research_topic}.sh" +} + +# Export for different research topics +export_for_research "cybersecurity" +export_for_research "climate-change" +``` + +## Advanced Usage + +### Selective Export +```bash +# Export embeddings with multiple filters +selective_export() { + local users=("user1" "user2" "user3") + local collections=("docs1" "docs2") + + for user in "${users[@]}"; do + for collection in "${collections[@]}"; do + output_file="embeddings-${user}-${collection}.msgpack" + + echo "Exporting for user: $user, collection: $collection" + + tg-save-doc-embeds \ + -o "$output_file" \ + --user "$user" \ + --collection "$collection" & + + # Store PID for later management + echo $! > "${output_file}.pid" + done + done + + echo "All selective exports started" +} +``` + +### Monitoring and Statistics +```bash +# Monitor export progress with statistics +monitor_export() { + local output_file="$1" + local pid_file="${output_file}.pid" + + if [ ! -f "$pid_file" ]; then + echo "PID file not found: $pid_file" + return 1 + fi + + local export_pid=$(cat "$pid_file") + + echo "Monitoring export (PID: $export_pid)..." + echo "Output file: $output_file" + + while kill -0 "$export_pid" 2>/dev/null; do + if [ -f "$output_file" ]; then + file_size=$(stat -c%s "$output_file" 2>/dev/null || echo "0") + human_size=$(numfmt --to=iec-i --suffix=B "$file_size") + + # Try to count embeddings + embedding_count=$(tg-dump-msgpack -i "$output_file" 2>/dev/null | grep -c '^\["de"' || echo "0") + + echo "File size: $human_size, Embeddings: $embedding_count" + else + echo "Output file not yet created..." + fi + + sleep 30 + done + + echo "Export process completed" + rm "$pid_file" +} + +# Start export and monitor +tg-save-doc-embeds -o "monitored-export.msgpack" & +echo $! > "monitored-export.msgpack.pid" +monitor_export "monitored-export.msgpack" +``` + +### Export Validation +```bash +# Validate exported embeddings +validate_export() { + local export_file="$1" + + echo "Validating export file: $export_file" + + # Check file exists and has content + if [ ! -s "$export_file" ]; then + echo "✗ Export file is empty or missing" + return 1 + fi + + # Check MessagePack format + if tg-dump-msgpack -i "$export_file" --summary > /dev/null 2>&1; then + echo "✓ Valid MessagePack format" + else + echo "✗ Invalid MessagePack format" + return 1 + fi + + # Check for document embeddings + embedding_count=$(tg-dump-msgpack -i "$export_file" | grep -c '^\["de"' || echo "0") + + if [ "$embedding_count" -gt 0 ]; then + echo "✓ Contains $embedding_count document embeddings" + else + echo "✗ No document embeddings found" + return 1 + fi + + # Get vector dimension information + summary=$(tg-dump-msgpack -i "$export_file" --summary) + if echo "$summary" | grep -q "Vector dimension:"; then + dimension=$(echo "$summary" | grep "Vector dimension:" | awk '{print $3}') + echo "✓ Vector dimension: $dimension" + else + echo "⚠ Could not determine vector dimension" + fi + + echo "Validation completed successfully" +} +``` + +### Export Scheduling +```bash +# Scheduled export with cron-like functionality +schedule_export() { + local schedule="$1" # e.g., "daily", "hourly", "weekly" + local output_prefix="$2" + + case "$schedule" in + "hourly") + interval=3600 + ;; + "daily") + interval=86400 + ;; + "weekly") + interval=604800 + ;; + *) + echo "Invalid schedule: $schedule" + return 1 + ;; + esac + + echo "Starting $schedule exports with prefix: $output_prefix" + + while true; do + timestamp=$(date +%Y%m%d_%H%M%S) + output_file="${output_prefix}-${timestamp}.msgpack" + + echo "Starting scheduled export: $output_file" + + # Run export for the scheduled interval + timeout ${interval}s tg-save-doc-embeds -o "$output_file" + + # Validate and compress + if validate_export "$output_file"; then + gzip "$output_file" + echo "✓ Export completed and compressed: ${output_file}.gz" + else + echo "✗ Export validation failed: $output_file" + mv "$output_file" "${output_file}.failed" + fi + + # Brief pause before next cycle + sleep 60 + done +} + +# Start daily scheduled exports +schedule_export "daily" "daily-embeddings" & +``` + +## Performance Considerations + +### Memory Management +```bash +# Monitor memory usage during export +monitor_memory_export() { + local output_file="$1" + + # Start export + tg-save-doc-embeds -o "$output_file" & + export_pid=$! + + echo "Monitoring memory usage for export (PID: $export_pid)..." + + while kill -0 "$export_pid" 2>/dev/null; do + memory_usage=$(ps -p "$export_pid" -o rss= 2>/dev/null | awk '{print $1/1024}') + + if [ -n "$memory_usage" ]; then + echo "Memory usage: ${memory_usage}MB" + fi + + sleep 10 + done + + echo "Export completed" +} +``` + +### Network Optimization +```bash +# Optimize for network conditions +network_optimized_export() { + local output_file="$1" + local api_url="$2" + + echo "Starting network-optimized export..." + + # Use compression and buffering + tg-save-doc-embeds \ + -o "$output_file" \ + -u "$api_url" \ + --format msgpack & # MessagePack is more compact than JSON + + export_pid=$! + + # Monitor network usage + echo "Monitoring export (PID: $export_pid)..." + + while kill -0 "$export_pid" 2>/dev/null; do + # Monitor network connections + connections=$(netstat -an | grep ":8088" | wc -l) + echo "Active connections: $connections" + sleep 30 + done +} +``` + +## Error Handling + +### Connection Issues +```bash +Exception: WebSocket connection failed +``` +**Solution**: Check API URL and ensure TrustGraph WebSocket service is running. + +### Disk Space Issues +```bash +Exception: No space left on device +``` +**Solution**: Free up disk space or use a different output location. + +### Permission Errors +```bash +Exception: Permission denied +``` +**Solution**: Check write permissions for the output file location. + +### Memory Issues +```bash +MemoryError: Unable to allocate memory +``` +**Solution**: Monitor memory usage and consider using smaller export windows. + +## Integration with Other Commands + +### Complete Backup Workflow +```bash +# Complete backup and restore workflow +backup_restore_workflow() { + local backup_file="embeddings-backup.msgpack" + + echo "=== Backup Phase ===" + + # Create backup + tg-save-doc-embeds -o "$backup_file" & + backup_pid=$! + + # Let it run for a while + sleep 300 # 5 minutes + kill $backup_pid + + echo "Backup created: $backup_file" + + # Validate backup + validate_export "$backup_file" + + echo "=== Restore Phase ===" + + # Restore from backup (to different collection) + tg-load-doc-embeds -i "$backup_file" --collection "restored" + + echo "Backup and restore workflow completed" +} +``` + +### Analysis Pipeline +```bash +# Export and analyze embeddings +export_analyze_pipeline() { + local topic="$1" + local export_file="analysis-${topic}.msgpack" + + echo "Starting export and analysis pipeline for: $topic" + + # Export embeddings + tg-save-doc-embeds \ + -o "$export_file" \ + --collection "$topic" & + + export_pid=$! + + # Run for analysis duration + sleep 600 # 10 minutes + kill $export_pid + + # Analyze exported data + echo "Analyzing exported embeddings..." + tg-dump-msgpack -i "$export_file" --summary + + # Count embeddings by user + echo "Embeddings by user:" + tg-dump-msgpack -i "$export_file" | \ + jq -r '.[1].m.u' | \ + sort | uniq -c + + echo "Analysis pipeline completed" +} +``` + +## Environment Variables + +- `TRUSTGRAPH_API`: Default API URL + +## Related Commands + +- [`tg-load-doc-embeds`](tg-load-doc-embeds.md) - Load document embeddings from files +- [`tg-dump-msgpack`](tg-dump-msgpack.md) - Analyze MessagePack files +- [`tg-show-flows`](tg-show-flows.md) - List available flows for monitoring + +## API Integration + +This command uses TrustGraph's WebSocket API for document embeddings export, specifically the `/api/v1/flow/{flow-id}/export/document-embeddings` endpoint. + +## Best Practices + +1. **Start Early**: Begin export before processing starts to capture all data +2. **Monitoring**: Monitor export progress and file sizes +3. **Validation**: Always validate exported files +4. **Compression**: Use compression for long-term storage +5. **Rotation**: Implement file rotation for continuous exports +6. **Backup**: Keep multiple backup copies in different locations +7. **Documentation**: Document export schedules and procedures + +## Troubleshooting + +### No Data Captured +```bash +# Check if processing is generating embeddings +tg-show-flows | grep processing + +# Verify WebSocket connection +netstat -an | grep :8088 +``` + +### Large File Issues +```bash +# Monitor file growth +watch -n 5 'ls -lh *.msgpack' + +# Check available disk space +df -h +``` + +### Process Management +```bash +# List running export processes +ps aux | grep tg-save-doc-embeds + +# Kill stuck processes +pkill -f tg-save-doc-embeds +``` \ No newline at end of file diff --git a/docs/cli/tg-show-flow-state.md b/docs/cli/tg-show-flow-state.md new file mode 100644 index 00000000..d0741522 --- /dev/null +++ b/docs/cli/tg-show-flow-state.md @@ -0,0 +1,518 @@ +# tg-show-flow-state + +Displays the processor states for a specific flow and its associated flow class. + +## Synopsis + +```bash +tg-show-flow-state [options] +``` + +## Description + +The `tg-show-flow-state` command shows the current state of processors within a specific TrustGraph flow instance and its corresponding flow class. It queries the metrics system to determine which processing components are running and displays their status with visual indicators. + +This command is essential for monitoring flow health and debugging processing issues. + +## Options + +### Optional Arguments + +- `-f, --flow-id ID`: Flow instance ID to examine (default: `default`) +- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`) +- `-m, --metrics-url URL`: Metrics API URL (default: `http://localhost:8088/api/metrics`) + +## Examples + +### Check Default Flow State +```bash +tg-show-flow-state +``` + +### Check Specific Flow +```bash +tg-show-flow-state -f "production-flow" +``` + +### Use Custom Metrics URL +```bash +tg-show-flow-state \ + -f "research-flow" \ + -m "http://metrics-server:8088/api/metrics" +``` + +### Check Flow in Different Environment +```bash +tg-show-flow-state \ + -f "staging-flow" \ + -u "http://staging:8088/" \ + -m "http://staging:8088/api/metrics" +``` + +## Output Format + +The command displays processor states for both the flow instance and its flow class: + +``` +Flow production-flow +- pdf-processor 💚 +- text-extractor 💚 +- embeddings-generator 💚 +- knowledge-builder ❌ +- document-indexer 💚 + +Class document-processing-v2 +- base-pdf-processor 💚 +- base-text-extractor 💚 +- base-embeddings-generator 💚 +- base-knowledge-builder 💚 +- base-document-indexer 💚 +``` + +### Status Indicators +- **💚 (Green Heart)**: Processor is running and healthy +- **❌ (Red X)**: Processor is not running or unhealthy + +### Information Displayed +- **Flow Section**: Shows the state of processors in the specific flow instance +- **Class Section**: Shows the state of processors in the flow class template +- **Processor Names**: Individual processing components within the flow + +## Use Cases + +### Flow Health Monitoring +```bash +# Monitor flow health continuously +monitor_flow_health() { + local flow_id="$1" + local interval="${2:-30}" # Default 30 seconds + + echo "Monitoring flow health: $flow_id" + echo "Refresh interval: ${interval}s" + echo "Press Ctrl+C to stop" + + while true; do + clear + echo "Flow Health Monitor - $(date)" + echo "==============================" + + tg-show-flow-state -f "$flow_id" + + sleep "$interval" + done +} + +# Monitor production flow +monitor_flow_health "production-flow" 15 +``` + +### Debugging Processing Issues +```bash +# Comprehensive flow debugging +debug_flow_issues() { + local flow_id="$1" + + echo "Debugging flow: $flow_id" + echo "=======================" + + # Check flow state + echo "1. Processor States:" + tg-show-flow-state -f "$flow_id" + + # Check flow configuration + echo -e "\n2. Flow Configuration:" + tg-show-flows | grep "$flow_id" + + # Check active processing + echo -e "\n3. Active Processing:" + tg-show-flows | grep -i processing + + # Check system resources + echo -e "\n4. System Resources:" + free -h + df -h + + echo -e "\nDebugging complete for: $flow_id" +} + +# Debug specific flow +debug_flow_issues "problematic-flow" +``` + +### Multi-Flow Status Dashboard +```bash +# Create status dashboard for multiple flows +create_flow_dashboard() { + local flows=("$@") + + echo "TrustGraph Flow Dashboard - $(date)" + echo "===================================" + + for flow in "${flows[@]}"; do + echo -e "\n=== Flow: $flow ===" + tg-show-flow-state -f "$flow" 2>/dev/null || echo "Flow not found or inaccessible" + done + + echo -e "\n=== Summary ===" + echo "Total flows monitored: ${#flows[@]}" + echo "Dashboard generated: $(date)" +} + +# Monitor multiple flows +flows=("production-flow" "research-flow" "development-flow") +create_flow_dashboard "${flows[@]}" +``` + +### Automated Health Checks +```bash +# Automated health check with alerts +health_check_with_alerts() { + local flow_id="$1" + local alert_email="$2" + + echo "Performing health check for: $flow_id" + + # Capture flow state + flow_state=$(tg-show-flow-state -f "$flow_id" 2>&1) + + if [ $? -ne 0 ]; then + echo "ERROR: Failed to get flow state" + # Send alert email if configured + if [ -n "$alert_email" ]; then + echo "Flow $flow_id is not responding" | mail -s "TrustGraph Alert" "$alert_email" + fi + return 1 + fi + + # Check for failed processors + failed_count=$(echo "$flow_state" | grep -c "❌") + + if [ "$failed_count" -gt 0 ]; then + echo "WARNING: $failed_count processors are not running" + echo "$flow_state" + + # Send alert if configured + if [ -n "$alert_email" ]; then + echo -e "Flow $flow_id has $failed_count failed processors:\n\n$flow_state" | \ + mail -s "TrustGraph Health Alert" "$alert_email" + fi + return 1 + else + echo "✓ All processors are running normally" + return 0 + fi +} + +# Run health check +health_check_with_alerts "production-flow" "admin@company.com" +``` + +## Advanced Usage + +### Flow State Comparison +```bash +# Compare flow states between environments +compare_flow_states() { + local flow_id="$1" + local env1_url="$2" + local env2_url="$3" + + echo "Comparing flow state: $flow_id" + echo "Environment 1: $env1_url" + echo "Environment 2: $env2_url" + echo "================================" + + # Get states from both environments + echo "Environment 1 State:" + tg-show-flow-state -f "$flow_id" -u "$env1_url" -m "$env1_url/api/metrics" + + echo -e "\nEnvironment 2 State:" + tg-show-flow-state -f "$flow_id" -u "$env2_url" -m "$env2_url/api/metrics" + + echo -e "\nComparison complete" +} + +# Compare production vs staging +compare_flow_states "main-flow" "http://prod:8088" "http://staging:8088" +``` + +### Historical State Tracking +```bash +# Track flow state over time +track_flow_state_history() { + local flow_id="$1" + local log_file="flow_state_history.log" + local interval="${2:-60}" # Default 1 minute + + echo "Starting flow state tracking: $flow_id" + echo "Log file: $log_file" + echo "Interval: ${interval}s" + + while true; do + timestamp=$(date '+%Y-%m-%d %H:%M:%S') + + # Get current state + state_output=$(tg-show-flow-state -f "$flow_id" 2>&1) + + if [ $? -eq 0 ]; then + # Count healthy and failed processors + healthy_count=$(echo "$state_output" | grep -c "💚") + failed_count=$(echo "$state_output" | grep -c "❌") + + # Log summary + echo "$timestamp,$flow_id,$healthy_count,$failed_count" >> "$log_file" + + # If there are failures, log details + if [ "$failed_count" -gt 0 ]; then + echo "$timestamp - FAILURES DETECTED in $flow_id:" >> "${log_file}.detailed" + echo "$state_output" >> "${log_file}.detailed" + echo "---" >> "${log_file}.detailed" + fi + else + echo "$timestamp,$flow_id,ERROR,ERROR" >> "$log_file" + fi + + sleep "$interval" + done +} + +# Start tracking (run in background) +track_flow_state_history "production-flow" 30 & +``` + +### State-Based Actions +```bash +# Perform actions based on flow state +state_based_actions() { + local flow_id="$1" + + echo "Checking flow state for automated actions: $flow_id" + + # Get current state + state_output=$(tg-show-flow-state -f "$flow_id") + + if [ $? -ne 0 ]; then + echo "ERROR: Cannot get flow state" + return 1 + fi + + # Check specific processors + if echo "$state_output" | grep -q "pdf-processor.*❌"; then + echo "PDF processor is down - attempting restart..." + # Restart specific processor (this would need additional commands) + # restart_processor "$flow_id" "pdf-processor" + fi + + if echo "$state_output" | grep -q "embeddings-generator.*❌"; then + echo "Embeddings generator is down - checking dependencies..." + # Check GPU availability, memory, etc. + nvidia-smi 2>/dev/null || echo "GPU not available" + fi + + # Count total failures + failed_count=$(echo "$state_output" | grep -c "❌") + + if [ "$failed_count" -gt 3 ]; then + echo "CRITICAL: More than 3 processors failed - considering flow restart" + # This would trigger more serious recovery actions + fi +} +``` + +### Performance Correlation +```bash +# Correlate flow state with performance metrics +correlate_state_performance() { + local flow_id="$1" + local metrics_url="$2" + + echo "Correlating flow state with performance for: $flow_id" + + # Get flow state + state_output=$(tg-show-flow-state -f "$flow_id" -m "$metrics_url") + healthy_count=$(echo "$state_output" | grep -c "💚") + failed_count=$(echo "$state_output" | grep -c "❌") + + echo "Processors - Healthy: $healthy_count, Failed: $failed_count" + + # Get performance metrics (this would need additional API calls) + # throughput=$(get_flow_throughput "$flow_id" "$metrics_url") + # latency=$(get_flow_latency "$flow_id" "$metrics_url") + + # echo "Performance - Throughput: ${throughput}/min, Latency: ${latency}ms" + + # Calculate health ratio + total_processors=$((healthy_count + failed_count)) + if [ "$total_processors" -gt 0 ]; then + health_ratio=$(echo "scale=2; $healthy_count * 100 / $total_processors" | bc) + echo "Health ratio: ${health_ratio}%" + fi +} +``` + +## Integration with Monitoring Systems + +### Prometheus Integration +```bash +# Export flow state metrics to Prometheus format +export_prometheus_metrics() { + local flow_id="$1" + local metrics_file="flow_state_metrics.prom" + + # Get flow state + state_output=$(tg-show-flow-state -f "$flow_id") + + # Count states + healthy_count=$(echo "$state_output" | grep -c "💚") + failed_count=$(echo "$state_output" | grep -c "❌") + + # Generate Prometheus metrics + cat > "$metrics_file" << EOF +# HELP trustgraph_flow_processors_healthy Number of healthy processors in flow +# TYPE trustgraph_flow_processors_healthy gauge +trustgraph_flow_processors_healthy{flow_id="$flow_id"} $healthy_count + +# HELP trustgraph_flow_processors_failed Number of failed processors in flow +# TYPE trustgraph_flow_processors_failed gauge +trustgraph_flow_processors_failed{flow_id="$flow_id"} $failed_count + +# HELP trustgraph_flow_health_ratio Ratio of healthy processors +# TYPE trustgraph_flow_health_ratio gauge +EOF + + total=$((healthy_count + failed_count)) + if [ "$total" -gt 0 ]; then + ratio=$(echo "scale=4; $healthy_count / $total" | bc) + echo "trustgraph_flow_health_ratio{flow_id=\"$flow_id\"} $ratio" >> "$metrics_file" + fi + + echo "Prometheus metrics exported to: $metrics_file" +} +``` + +### Grafana Dashboard Data +```bash +# Generate data for Grafana dashboard +generate_grafana_data() { + local flows=("$@") + local output_file="grafana_flow_data.json" + + echo "Generating Grafana dashboard data..." + + echo "{" > "$output_file" + echo " \"flows\": [" >> "$output_file" + + for i in "${!flows[@]}"; do + flow="${flows[$i]}" + + # Get flow state + state_output=$(tg-show-flow-state -f "$flow" 2>/dev/null) + + if [ $? -eq 0 ]; then + healthy=$(echo "$state_output" | grep -c "💚") + failed=$(echo "$state_output" | grep -c "❌") + else + healthy=0 + failed=0 + fi + + echo " {" >> "$output_file" + echo " \"flow_id\": \"$flow\"," >> "$output_file" + echo " \"healthy_processors\": $healthy," >> "$output_file" + echo " \"failed_processors\": $failed," >> "$output_file" + echo " \"timestamp\": \"$(date -Iseconds)\"" >> "$output_file" + + if [ $i -lt $((${#flows[@]} - 1)) ]; then + echo " }," >> "$output_file" + else + echo " }" >> "$output_file" + fi + done + + echo " ]" >> "$output_file" + echo "}" >> "$output_file" + + echo "Grafana data generated: $output_file" +} +``` + +## Error Handling + +### Flow Not Found +```bash +Exception: Flow 'nonexistent-flow' not found +``` +**Solution**: Verify the flow ID exists with `tg-show-flows`. + +### Metrics API Unavailable +```bash +Exception: Connection refused to metrics API +``` +**Solution**: Check metrics URL and ensure metrics service is running. + +### Permission Issues +```bash +Exception: Access denied to metrics +``` +**Solution**: Verify permissions for accessing metrics and flow information. + +### Invalid Flow State +```bash +Exception: Unable to parse flow state +``` +**Solution**: Check if the flow is properly initialized and processors are configured. + +## Environment Variables + +- `TRUSTGRAPH_URL`: Default API URL + +## Related Commands + +- [`tg-show-flows`](tg-show-flows.md) - List all flows +- [`tg-show-processor-state`](tg-show-processor-state.md) - Show all processor states +- [`tg-start-flow`](tg-start-flow.md) - Start flow instances +- [`tg-stop-flow`](tg-stop-flow.md) - Stop flow instances + +## API Integration + +This command integrates with: +- TrustGraph Flow API for flow information +- Prometheus/Metrics API for processor state information + +## Best Practices + +1. **Regular Monitoring**: Check flow states regularly in production +2. **Automated Alerts**: Set up automated health checks with alerting +3. **Historical Tracking**: Maintain historical flow state data +4. **Integration**: Integrate with monitoring systems like Prometheus/Grafana +5. **Documentation**: Document expected processor configurations +6. **Correlation**: Correlate flow state with performance metrics +7. **Recovery Procedures**: Develop automated recovery procedures for common failures + +## Troubleshooting + +### No Processors Shown +```bash +# Check if flow exists +tg-show-flows | grep "flow-id" + +# Verify metrics service +curl -s http://localhost:8088/api/metrics/query?query=processor_info +``` + +### Inconsistent States +```bash +# Check metrics service health +curl -s http://localhost:8088/api/metrics/health + +# Restart metrics collection if needed +``` + +### Connection Issues +```bash +# Test API connectivity +curl -s http://localhost:8088/api/v1/flows + +# Test metrics connectivity +curl -s http://localhost:8088/api/metrics/query?query=up +``` \ No newline at end of file