Update docs

docs/cli/tg-dump-msgpack.md

@@ -0,0 +1,489 @@
+# tg-dump-msgpack
+
+Reads and analyzes knowledge core files in MessagePack format for diagnostic purposes.
+
+## Synopsis
+
+```bash
+tg-dump-msgpack -i INPUT_FILE [options]
+```
+
+## Description
+
+The `tg-dump-msgpack` command is a diagnostic utility that reads knowledge core files stored in MessagePack format and outputs their contents in JSON format or provides a summary analysis. This tool is primarily used for debugging, data inspection, and understanding the structure of knowledge cores.
+
+MessagePack is a binary serialization format that TrustGraph uses for efficient storage and transfer of knowledge graph data.
+
+## Options
+
+### Required Arguments
+
+- `-i, --input-file FILE`: Input MessagePack file to read
+
+### Optional Arguments
+
+- `-s, --summary`: Show a summary analysis of the file contents
+- `-r, --records`: Dump individual records in JSON format (default behavior)
+
+## Examples
+
+### Dump Records as JSON
+```bash
+tg-dump-msgpack -i knowledge-core.msgpack
+```
+
+### Show Summary Analysis
+```bash
+tg-dump-msgpack -i knowledge-core.msgpack --summary
+```
+
+### Save Output to File
+```bash
+tg-dump-msgpack -i knowledge-core.msgpack > analysis.json
+```
+
+### Analyze Multiple Files
+```bash
+for file in *.msgpack; do
+  echo "=== $file ==="
+  tg-dump-msgpack -i "$file" --summary
+  echo
+done
+```
+
+## Output Formats
+
+### Record Output (Default)
+With `-r` or `--records` (default behavior), the command outputs each record as a separate JSON object:
+
+```json
+["t", {"m": {"m": [{"s": {"v": "uri1"}, "p": {"v": "predicate"}, "o": {"v": "object"}}]}}]
+["ge", {"v": [[0.1, 0.2, 0.3, ...]]}]
+["de", {"metadata": {...}, "chunks": [...]}]
+```
+
+### Summary Output
+With `-s` or `--summary`, the command provides an analytical overview:
+
+```
+Vector dimension: 384
+- NASA Challenger Report
+- Technical Documentation
+- Safety Engineering Guidelines
+```
+
+## Record Types
+
+MessagePack files may contain different types of records:
+
+### Triple Records ("t")
+RDF triples representing knowledge graph relationships:
+```json
+["t", {
+  "m": {
+    "m": [{
+      "s": {"v": "http://example.org/subject"},
+      "p": {"v": "http://example.org/predicate"}, 
+      "o": {"v": "object value"}
+    }]
+  }
+}]
+```
+
+### Graph Embeddings ("ge") 
+Vector embeddings for graph entities:
+```json
+["ge", {
+  "v": [[0.1, 0.2, 0.3, 0.4, ...]]
+}]
+```
+
+### Document Embeddings ("de")
+Document chunk embeddings with metadata:
+```json
+["de", {
+  "metadata": {
+    "id": "doc-123",
+    "user": "trustgraph",
+    "collection": "default"
+  },
+  "chunks": [{
+    "chunk": "text content",
+    "vectors": [0.1, 0.2, 0.3, ...]
+  }]
+}]
+```
+
+## Use Cases
+
+### Data Inspection
+```bash
+# Quick peek at file structure
+tg-dump-msgpack -i mystery-core.msgpack --summary
+
+# Detailed record analysis
+tg-dump-msgpack -i knowledge-core.msgpack | head -20
+```
+
+### Debugging Knowledge Cores
+```bash
+# Check if file contains expected data types
+tg-dump-msgpack -i core.msgpack | grep -o '^\["[^"]*"' | sort | uniq -c
+
+# Find specific entities
+tg-dump-msgpack -i core.msgpack | grep "NASA"
+
+# Check vector dimensions
+tg-dump-msgpack -i core.msgpack --summary | grep "Vector dimension"
+```
+
+### Quality Assurance
+```bash
+# Validate file completeness
+validate_msgpack() {
+  local file="$1"
+  
+  echo "Validating: $file"
+  
+  # Check file exists and is readable
+  if [ ! -r "$file" ]; then
+    echo "Error: Cannot read file $file"
+    return 1
+  fi
+  
+  # Get summary
+  summary=$(tg-dump-msgpack -i "$file" --summary 2>/dev/null)
+  
+  if [ $? -ne 0 ]; then
+    echo "Error: Failed to read MessagePack file"
+    return 1
+  fi
+  
+  # Check for vector dimension (indicates embeddings present)
+  if echo "$summary" | grep -q "Vector dimension:"; then
+    dim=$(echo "$summary" | grep "Vector dimension:" | awk '{print $3}')
+    echo "✓ Contains embeddings (dimension: $dim)"
+  else
+    echo "⚠ No embeddings found"
+  fi
+  
+  # Count labels (indicates entities present)
+  label_count=$(echo "$summary" | grep "^-" | wc -l)
+  echo "✓ Found $label_count labeled entities"
+  
+  return 0
+}
+
+# Validate multiple files
+for file in cores/*.msgpack; do
+  validate_msgpack "$file"
+done
+```
+
+### Data Migration
+```bash
+# Convert MessagePack to JSON for processing
+convert_to_json() {
+  local input="$1"
+  local output="$2"
+  
+  echo "Converting $input to $output..."
+  tg-dump-msgpack -i "$input" > "$output"
+  
+  # Add array wrapper for valid JSON array
+  sed -i '1i[' "$output"
+  sed -i '$a]' "$output"
+  sed -i 's/$/,/' "$output"
+  sed -i '$s/,$//' "$output"
+  
+  echo "Conversion complete"
+}
+
+convert_to_json "knowledge.msgpack" "knowledge.json"
+```
+
+### Analysis and Reporting
+```bash
+# Generate comprehensive analysis report
+analyze_msgpack() {
+  local file="$1"
+  local report_file="${file%.msgpack}_analysis.txt"
+  
+  echo "MessagePack Analysis Report" > "$report_file"
+  echo "File: $file" >> "$report_file"
+  echo "Generated: $(date)" >> "$report_file"
+  echo "=============================" >> "$report_file"
+  echo "" >> "$report_file"
+  
+  # Summary information
+  echo "Summary:" >> "$report_file"
+  tg-dump-msgpack -i "$file" --summary >> "$report_file"
+  echo "" >> "$report_file"
+  
+  # Record type analysis
+  echo "Record Type Distribution:" >> "$report_file"
+  tg-dump-msgpack -i "$file" | \
+    grep -o '^\["[^"]*"' | \
+    sort | uniq -c | \
+    awk '{print "  " $2 ": " $1 " records"}' >> "$report_file"
+  echo "" >> "$report_file"
+  
+  # File statistics
+  file_size=$(stat -c%s "$file")
+  echo "File Statistics:" >> "$report_file"
+  echo "  Size: $file_size bytes" >> "$report_file"
+  echo "  Size (human): $(numfmt --to=iec-i --suffix=B $file_size)" >> "$report_file"
+  
+  echo "Analysis saved to: $report_file"
+}
+
+# Analyze all MessagePack files
+for file in *.msgpack; do
+  analyze_msgpack "$file"
+done
+```
+
+### Comparative Analysis
+```bash
+# Compare two knowledge cores
+compare_msgpack() {
+  local file1="$1"
+  local file2="$2"
+  
+  echo "Comparing MessagePack files:"
+  echo "File 1: $file1"
+  echo "File 2: $file2"
+  echo "=========================="
+  
+  # Compare summaries
+  echo "Summary comparison:"
+  echo "File 1:"
+  tg-dump-msgpack -i "$file1" --summary | sed 's/^/  /'
+  echo ""
+  echo "File 2:"
+  tg-dump-msgpack -i "$file2" --summary | sed 's/^/  /'
+  echo ""
+  
+  # Compare record counts
+  echo "Record type comparison:"
+  echo "File 1:"
+  tg-dump-msgpack -i "$file1" | \
+    grep -o '^\["[^"]*"' | \
+    sort | uniq -c | \
+    awk '{print "  " $2 ": " $1}' | \
+    sort
+  
+  echo "File 2:"
+  tg-dump-msgpack -i "$file2" | \
+    grep -o '^\["[^"]*"' | \
+    sort | uniq -c | \
+    awk '{print "  " $2 ": " $1}' | \
+    sort
+}
+
+compare_msgpack "core1.msgpack" "core2.msgpack"
+```
+
+## Advanced Usage
+
+### Large File Processing
+```bash
+# Process large files in chunks
+process_large_msgpack() {
+  local file="$1"
+  local chunk_size=1000
+  
+  echo "Processing large file: $file"
+  
+  # Count total records first
+  total_records=$(tg-dump-msgpack -i "$file" | wc -l)
+  echo "Total records: $total_records"
+  
+  # Process in chunks
+  tg-dump-msgpack -i "$file" | \
+    split -l $chunk_size - "chunk_"
+  
+  echo "Split into chunks of $chunk_size records each"
+  
+  # Process each chunk
+  for chunk in chunk_*; do
+    echo "Processing $chunk..."
+    # Add your processing logic here
+    wc -l "$chunk"
+  done
+  
+  # Clean up
+  rm chunk_*
+}
+```
+
+### Data Extraction
+```bash
+# Extract specific data types
+extract_triples() {
+  local file="$1"
+  local output="triples.json"
+  
+  echo "Extracting triples from $file..."
+  tg-dump-msgpack -i "$file" | \
+    grep '^\["t"' > "$output"
+  
+  echo "Triples saved to: $output"
+}
+
+extract_embeddings() {
+  local file="$1"
+  local output="embeddings.json"
+  
+  echo "Extracting embeddings from $file..."
+  tg-dump-msgpack -i "$file" | \
+    grep -E '^\["(ge|de)"' > "$output"
+  
+  echo "Embeddings saved to: $output"
+}
+
+# Extract all data types
+extract_triples "knowledge.msgpack"
+extract_embeddings "knowledge.msgpack"
+```
+
+### Integration with Other Tools
+```bash
+# Convert MessagePack to formats for other tools
+msgpack_to_turtle() {
+  local input="$1"
+  local output="$2"
+  
+  echo "Converting MessagePack to Turtle format..."
+  
+  # Extract triples and convert to Turtle
+  tg-dump-msgpack -i "$input" | \
+    grep '^\["t"' | \
+    jq -r '.[1].m.m[] | 
+      "<" + .s.v + "> <" + .p.v + "> " + 
+      (if .o.e then "<" + .o.v + ">" else "\"" + .o.v + "\"" end) + " ."' \
+    > "$output"
+  
+  echo "Turtle format saved to: $output"
+}
+
+msgpack_to_turtle "knowledge.msgpack" "knowledge.ttl"
+```
+
+## Error Handling
+
+### File Not Found
+```bash
+Exception: [Errno 2] No such file or directory: 'missing.msgpack'
+```
+**Solution**: Check file path and ensure the file exists.
+
+### Invalid MessagePack Format
+```bash
+Exception: Unpack failed
+```
+**Solution**: Verify the file is a valid MessagePack file and not corrupted.
+
+### Memory Issues with Large Files
+```bash
+MemoryError: Unable to allocate memory
+```
+**Solution**: Process large files in chunks or use streaming approaches.
+
+### Permission Errors
+```bash
+Exception: [Errno 13] Permission denied
+```
+**Solution**: Check file permissions and ensure read access.
+
+## Performance Considerations
+
+### File Size Optimization
+```bash
+# Check file compression efficiency
+check_compression() {
+  local file="$1"
+  
+  original_size=$(stat -c%s "$file")
+  
+  # Test compression
+  gzip -c "$file" > "${file}.gz"
+  compressed_size=$(stat -c%s "${file}.gz")
+  
+  ratio=$(echo "scale=2; $compressed_size * 100 / $original_size" | bc)
+  
+  echo "Original: $(numfmt --to=iec-i --suffix=B $original_size)"
+  echo "Compressed: $(numfmt --to=iec-i --suffix=B $compressed_size)"
+  echo "Compression ratio: ${ratio}%"
+  
+  rm "${file}.gz"
+}
+```
+
+### Processing Speed
+```bash
+# Time processing operations
+time_msgpack_ops() {
+  local file="$1"
+  
+  echo "Timing MessagePack operations for: $file"
+  
+  # Time summary generation
+  echo "Summary generation:"
+  time tg-dump-msgpack -i "$file" --summary > /dev/null
+  
+  # Time full dump
+  echo "Full record dump:"
+  time tg-dump-msgpack -i "$file" > /dev/null
+}
+```
+
+## Related Commands
+
+- [`tg-get-kg-core`](tg-get-kg-core.md) - Export knowledge cores to MessagePack
+- [`tg-load-kg-core`](tg-load-kg-core.md) - Load MessagePack knowledge cores
+- [`tg-save-doc-embeds`](tg-save-doc-embeds.md) - Save document embeddings to MessagePack
+
+## Best Practices
+
+1. **File Validation**: Always validate MessagePack files before processing
+2. **Memory Management**: Be cautious with large files to avoid memory issues
+3. **Backup**: Keep backups of original MessagePack files before analysis
+4. **Incremental Processing**: Process large files incrementally when possible
+5. **Documentation**: Document the structure and content of your MessagePack files
+6. **Version Control**: Track changes in MessagePack file formats over time
+
+## Troubleshooting
+
+### Corrupted Files
+```bash
+# Test file integrity
+if tg-dump-msgpack -i "test.msgpack" --summary > /dev/null 2>&1; then
+  echo "File appears valid"
+else
+  echo "File may be corrupted"
+fi
+```
+
+### Empty or Incomplete Files
+```bash
+# Check for empty files
+if [ ! -s "test.msgpack" ]; then
+  echo "File is empty"
+fi
+
+# Check record count
+record_count=$(tg-dump-msgpack -i "test.msgpack" 2>/dev/null | wc -l)
+echo "Records found: $record_count"
+```
+
+### Format Issues
+```bash
+# Validate JSON output
+tg-dump-msgpack -i "test.msgpack" | head -1 | jq . > /dev/null
+if [ $? -eq 0 ]; then
+  echo "JSON output is valid"
+else
+  echo "JSON output may be malformed"
+fi
+```

docs/cli/tg-init-pulsar-manager.md

@@ -0,0 +1,452 @@
+# tg-init-pulsar-manager
+
+Initializes Pulsar Manager with default superuser credentials for TrustGraph.
+
+## Synopsis
+
+```bash
+tg-init-pulsar-manager
+```
+
+## Description
+
+The `tg-init-pulsar-manager` command is a setup utility that creates a default superuser account in Pulsar Manager. This is typically run once during initial TrustGraph deployment to establish administrative access to the Pulsar message queue management interface.
+
+The command configures a superuser with predefined credentials that can be used to access the Pulsar Manager web interface for monitoring and managing Pulsar topics, namespaces, and tenants.
+
+## Default Configuration
+
+The command creates a superuser with these default credentials:
+
+- **Username**: `admin`
+- **Password**: `apachepulsar`
+- **Description**: `test`
+- **Email**: `username@test.org`
+
+## Prerequisites
+
+### Pulsar Manager Service
+Pulsar Manager must be running and accessible at `http://localhost:7750` before running this command.
+
+### Network Connectivity
+The command requires network access to the Pulsar Manager API endpoint.
+
+## Examples
+
+### Basic Initialization
+```bash
+tg-init-pulsar-manager
+```
+
+### Verify Initialization
+```bash
+# Run the initialization
+tg-init-pulsar-manager
+
+# Check if Pulsar Manager is accessible
+curl -s http://localhost:7750/pulsar-manager/ | grep -q "Pulsar Manager"
+echo "Pulsar Manager status: $?"
+```
+
+### Integration with Setup Scripts
+```bash
+#!/bin/bash
+# setup-trustgraph.sh
+
+echo "Setting up TrustGraph infrastructure..."
+
+# Wait for Pulsar Manager to be ready
+echo "Waiting for Pulsar Manager..."
+while ! curl -s http://localhost:7750/pulsar-manager/ > /dev/null; do
+  echo "  Waiting for Pulsar Manager to start..."
+  sleep 5
+done
+
+# Initialize Pulsar Manager
+echo "Initializing Pulsar Manager..."
+tg-init-pulsar-manager
+
+if [ $? -eq 0 ]; then
+  echo "✓ Pulsar Manager initialized successfully"
+  echo "  You can access it at: http://localhost:7750/pulsar-manager/"
+  echo "  Username: admin"
+  echo "  Password: apachepulsar"
+else
+  echo "✗ Failed to initialize Pulsar Manager"
+  exit 1
+fi
+```
+
+## What It Does
+
+The command performs the following operations:
+
+1. **Retrieves CSRF Token**: Gets a CSRF token from Pulsar Manager for secure API access
+2. **Creates Superuser**: Makes an authenticated API call to create the superuser account
+3. **Sets Permissions**: Configures the user with administrative privileges
+
+### HTTP Operations
+```bash
+# Equivalent manual operations:
+CSRF_TOKEN=$(curl http://localhost:7750/pulsar-manager/csrf-token)
+
+curl \
+    -H "X-XSRF-TOKEN: $CSRF_TOKEN" \
+    -H "Cookie: XSRF-TOKEN=$CSRF_TOKEN;" \
+    -H 'Content-Type: application/json' \
+    -X PUT \
+    http://localhost:7750/pulsar-manager/users/superuser \
+    -d '{"name": "admin", "password": "apachepulsar", "description": "test", "email": "username@test.org"}'
+```
+
+## Use Cases
+
+### Initial Deployment
+```bash
+# Part of TrustGraph deployment sequence
+deploy_trustgraph() {
+  echo "Deploying TrustGraph..."
+  
+  # Start services
+  docker-compose up -d pulsar pulsar-manager
+  
+  # Wait for services
+  wait_for_service "http://localhost:7750/pulsar-manager/" "Pulsar Manager"
+  wait_for_service "http://localhost:8080/admin/v2/clusters" "Pulsar"
+  
+  # Initialize Pulsar Manager
+  echo "Initializing Pulsar Manager..."
+  tg-init-pulsar-manager
+  
+  # Initialize TrustGraph
+  echo "Initializing TrustGraph..."
+  tg-init-trustgraph
+  
+  echo "Deployment complete!"
+}
+```
+
+### Development Environment Setup
+```bash
+# Development setup script
+setup_dev_environment() {
+  echo "Setting up development environment..."
+  
+  # Start local services
+  docker-compose -f docker-compose.dev.yml up -d
+  
+  # Wait for readiness
+  echo "Waiting for services to start..."
+  sleep 30
+  
+  # Initialize components
+  tg-init-pulsar-manager
+  tg-init-trustgraph
+  
+  echo "Development environment ready!"
+  echo "Pulsar Manager: http://localhost:7750/pulsar-manager/"
+  echo "Credentials: admin / apachepulsar"
+}
+```
+
+### CI/CD Integration
+```bash
+# Integration testing setup
+setup_test_environment() {
+  local timeout=300  # 5 minutes
+  local elapsed=0
+  
+  echo "Setting up test environment..."
+  
+  # Start services
+  docker-compose up -d --wait
+  
+  # Wait for Pulsar Manager
+  while ! curl -s http://localhost:7750/pulsar-manager/ > /dev/null; do
+    if [ $elapsed -ge $timeout ]; then
+      echo "Timeout waiting for Pulsar Manager"
+      return 1
+    fi
+    sleep 5
+    elapsed=$((elapsed + 5))
+  done
+  
+  # Initialize
+  if tg-init-pulsar-manager; then
+    echo "✓ Test environment ready"
+  else
+    echo "✗ Failed to initialize test environment"
+    return 1
+  fi
+}
+```
+
+## Docker Integration
+
+### Docker Compose Setup
+```yaml
+# docker-compose.yml
+version: '3.8'
+
+services:
+  pulsar:
+    image: apachepulsar/pulsar:latest
+    ports:
+      - "6650:6650"
+      - "8080:8080"
+    command: bin/pulsar standalone
+    
+  pulsar-manager:
+    image: apachepulsar/pulsar-manager:latest
+    ports:
+      - "7750:7750"
+    depends_on:
+      - pulsar
+    environment:
+      SPRING_CONFIGURATION_FILE: /pulsar-manager/pulsar-manager/application.properties
+
+  trustgraph-init:
+    image: trustgraph/cli:latest
+    depends_on:
+      - pulsar-manager
+    command: >
+      sh -c "
+        sleep 30 &&
+        tg-init-pulsar-manager &&
+        tg-init-trustgraph
+      "
+```
+
+### Kubernetes Setup
+```yaml
+# k8s-init-job.yaml
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: trustgraph-init
+spec:
+  template:
+    spec:
+      containers:
+      - name: init
+        image: trustgraph/cli:latest
+        command:
+        - sh
+        - -c
+        - |
+          echo "Waiting for Pulsar Manager..."
+          while ! curl -s http://pulsar-manager:7750/pulsar-manager/; do
+            sleep 5
+          done
+          
+          echo "Initializing Pulsar Manager..."
+          tg-init-pulsar-manager
+          
+          echo "Initializing TrustGraph..."
+          tg-init-trustgraph
+        env:
+        - name: PULSAR_MANAGER_URL
+          value: "http://pulsar-manager:7750"
+      restartPolicy: Never
+```
+
+## Error Handling
+
+### Connection Refused
+```bash
+curl: (7) Failed to connect to localhost port 7750: Connection refused
+```
+**Solution**: Ensure Pulsar Manager is running and accessible on port 7750.
+
+### CSRF Token Issues
+```bash
+curl: (22) The requested URL returned error: 403 Forbidden
+```
+**Solution**: The CSRF token mechanism may have changed. Check Pulsar Manager API documentation.
+
+### User Already Exists
+```bash
+HTTP 409 Conflict - User already exists
+```
+**Solution**: This is expected on subsequent runs. The superuser is already created.
+
+### Network Issues
+```bash
+curl: (28) Operation timed out
+```
+**Solution**: Check network connectivity and firewall settings.
+
+## Security Considerations
+
+### Default Credentials
+The command uses default credentials that should be changed in production:
+
+```bash
+# After initialization, change the password via Pulsar Manager UI
+# Or use the API to update credentials
+change_admin_password() {
+  local new_password="$1"
+  
+  # Login to get session
+  session=$(curl -s -c cookies.txt \
+    -d "username=admin&password=apachepulsar" \
+    http://localhost:7750/pulsar-manager/login)
+  
+  # Update password
+  curl -s -b cookies.txt \
+    -H "Content-Type: application/json" \
+    -X PUT \
+    -d "{\"password\": \"$new_password\"}" \
+    http://localhost:7750/pulsar-manager/users/admin
+  
+  rm cookies.txt
+}
+```
+
+### Access Control
+```bash
+# Restrict access to Pulsar Manager in production
+configure_security() {
+  echo "Configuring Pulsar Manager security..."
+  
+  # Change default password
+  change_admin_password "$(openssl rand -base64 32)"
+  
+  # Configure firewall rules (example)
+  # iptables -A INPUT -p tcp --dport 7750 -s 10.0.0.0/8 -j ACCEPT
+  # iptables -A INPUT -p tcp --dport 7750 -j DROP
+  
+  echo "Security configuration complete"
+}
+```
+
+## Advanced Usage
+
+### Custom Configuration
+```bash
+# Create custom initialization script
+create_custom_init() {
+  cat > custom-pulsar-manager-init.sh << 'EOF'
+#!/bin/bash
+
+PULSAR_MANAGER_URL=${PULSAR_MANAGER_URL:-http://localhost:7750}
+ADMIN_USER=${ADMIN_USER:-admin}
+ADMIN_PASS=${ADMIN_PASS:-$(openssl rand -base64 16)}
+ADMIN_EMAIL=${ADMIN_EMAIL:-admin@example.com}
+
+echo "Initializing Pulsar Manager at: $PULSAR_MANAGER_URL"
+
+# Get CSRF token
+CSRF_TOKEN=$(curl -s "$PULSAR_MANAGER_URL/pulsar-manager/csrf-token")
+
+if [ -z "$CSRF_TOKEN" ]; then
+  echo "Failed to get CSRF token"
+  exit 1
+fi
+
+# Create superuser
+response=$(curl -s -w "%{http_code}" \
+  -H "X-XSRF-TOKEN: $CSRF_TOKEN" \
+  -H "Cookie: XSRF-TOKEN=$CSRF_TOKEN;" \
+  -H 'Content-Type: application/json' \
+  -X PUT \
+  "$PULSAR_MANAGER_URL/pulsar-manager/users/superuser" \
+  -d "{\"name\": \"$ADMIN_USER\", \"password\": \"$ADMIN_PASS\", \"description\": \"Admin user\", \"email\": \"$ADMIN_EMAIL\"}")
+
+http_code="${response: -3}"
+
+if [ "$http_code" = "200" ] || [ "$http_code" = "409" ]; then
+  echo "Pulsar Manager initialized successfully"
+  echo "Username: $ADMIN_USER"
+  echo "Password: $ADMIN_PASS"
+else
+  echo "Failed to initialize Pulsar Manager (HTTP $http_code)"
+  exit 1
+fi
+EOF
+
+  chmod +x custom-pulsar-manager-init.sh
+}
+```
+
+### Health Checks
+```bash
+# Health check script
+check_pulsar_manager() {
+  local max_attempts=30
+  local attempt=1
+  
+  echo "Checking Pulsar Manager health..."
+  
+  while [ $attempt -le $max_attempts ]; do
+    if curl -s http://localhost:7750/pulsar-manager/ > /dev/null; then
+      echo "✓ Pulsar Manager is healthy"
+      return 0
+    fi
+    
+    echo "Attempt $attempt/$max_attempts - Pulsar Manager not ready"
+    sleep 5
+    attempt=$((attempt + 1))
+  done
+  
+  echo "✗ Pulsar Manager health check failed"
+  return 1
+}
+
+# Use in deployment scripts
+if check_pulsar_manager; then
+  tg-init-pulsar-manager
+else
+  echo "Cannot initialize Pulsar Manager - service not healthy"
+  exit 1
+fi
+```
+
+## Related Commands
+
+- [`tg-init-trustgraph`](tg-init-trustgraph.md) - Initialize TrustGraph with Pulsar configuration
+- [`tg-show-config`](tg-show-config.md) - Display current TrustGraph configuration
+
+## Integration Points
+
+### Pulsar Manager UI
+After initialization, access the web interface at:
+- **URL**: `http://localhost:7750/pulsar-manager/`
+- **Username**: `admin`
+- **Password**: `apachepulsar`
+
+### TrustGraph Integration
+This command is typically run before `tg-init-trustgraph` as part of the complete TrustGraph setup process.
+
+## Best Practices
+
+1. **Run Once**: Only run during initial setup - subsequent runs are harmless but unnecessary
+2. **Change Defaults**: Change default credentials in production environments
+3. **Network Security**: Restrict access to Pulsar Manager in production
+4. **Health Checks**: Always verify Pulsar Manager is running before initialization
+5. **Automation**: Include in deployment automation scripts
+6. **Documentation**: Document custom credentials for operations teams
+
+## Troubleshooting
+
+### Service Not Ready
+```bash
+# Check if Pulsar Manager is running
+docker ps | grep pulsar-manager
+netstat -tlnp | grep 7750
+```
+
+### Port Conflicts
+```bash
+# Check if port 7750 is in use
+lsof -i :7750
+```
+
+### Docker Issues
+```bash
+# Check Pulsar Manager logs
+docker logs pulsar-manager
+
+# Restart if needed
+docker restart pulsar-manager
+```

docs/cli/tg-init-trustgraph.md

@@ -0,0 +1,523 @@
+# tg-init-trustgraph
+
+Initializes Pulsar with TrustGraph tenant, namespaces, and configuration settings.
+
+## Synopsis
+
+```bash
+tg-init-trustgraph [options]
+```
+
+## Description
+
+The `tg-init-trustgraph` command initializes the Apache Pulsar messaging system with the required tenant, namespaces, policies, and configuration needed for TrustGraph operation. This is a foundational setup command that must be run before TrustGraph can operate properly.
+
+The command creates the necessary Pulsar infrastructure and optionally loads initial configuration data into the system.
+
+## Options
+
+### Optional Arguments
+
+- `-p, --pulsar-admin-url URL`: Pulsar admin URL (default: `http://pulsar:8080`)
+- `--pulsar-host HOST`: Pulsar host for client connections (default: `pulsar://pulsar:6650`)
+- `--pulsar-api-key KEY`: Pulsar API key for authentication
+- `-c, --config CONFIG`: Initial configuration JSON to load
+- `-t, --tenant TENANT`: Tenant name (default: `tg`)
+
+## Examples
+
+### Basic Initialization
+```bash
+tg-init-trustgraph
+```
+
+### Custom Pulsar Configuration
+```bash
+tg-init-trustgraph \
+  --pulsar-admin-url http://localhost:8080 \
+  --pulsar-host pulsar://localhost:6650
+```
+
+### With Initial Configuration
+```bash
+tg-init-trustgraph \
+  --config '{"prompt": {"system": "You are a helpful AI assistant"}}'
+```
+
+### Custom Tenant
+```bash
+tg-init-trustgraph --tenant production-tg
+```
+
+### Production Setup
+```bash
+tg-init-trustgraph \
+  --pulsar-admin-url http://pulsar-cluster:8080 \
+  --pulsar-host pulsar://pulsar-cluster:6650 \
+  --pulsar-api-key "your-api-key" \
+  --tenant production \
+  --config "$(cat production-config.json)"
+```
+
+## What It Creates
+
+### Tenant Structure
+The command creates a TrustGraph tenant with the following namespaces:
+
+#### Flow Namespace (`tg/flow`)
+- **Purpose**: Processing workflows and flow definitions
+- **Retention**: Default retention policies
+
+#### Request Namespace (`tg/request`)
+- **Purpose**: Incoming API requests and commands
+- **Retention**: Default retention policies
+
+#### Response Namespace (`tg/response`)
+- **Purpose**: API responses and results
+- **Retention**: 3 minutes, unlimited size
+- **Subscription Expiration**: 30 minutes
+
+#### Config Namespace (`tg/config`)
+- **Purpose**: System configuration and settings
+- **Retention**: 10MB size limit, unlimited time
+- **Subscription Expiration**: 5 minutes
+
+### Configuration Loading
+
+If a configuration is provided, the command also:
+1. Connects to the configuration service
+2. Loads the provided configuration data
+3. Ensures configuration versioning is maintained
+
+## Configuration Format
+
+The configuration should be provided as JSON with this structure:
+
+```json
+{
+  "prompt": {
+    "system": "System prompt text",
+    "template-index": ["template1", "template2"],
+    "template.template1": {
+      "id": "template1",
+      "prompt": "Template text with {{variables}}",
+      "response-type": "text"
+    }
+  },
+  "token-costs": {
+    "gpt-4": {
+      "input_price": 0.00003,
+      "output_price": 0.00006
+    }
+  },
+  "agent": {
+    "tool-index": ["tool1"],
+    "tool.tool1": {
+      "id": "tool1",
+      "name": "Example Tool",
+      "description": "Tool description",
+      "arguments": []
+    }
+  }
+}
+```
+
+## Use Cases
+
+### Initial Deployment
+```bash
+# Complete TrustGraph initialization sequence
+initialize_trustgraph() {
+  echo "Initializing TrustGraph infrastructure..."
+  
+  # Wait for Pulsar to be ready
+  wait_for_pulsar
+  
+  # Initialize Pulsar Manager (if using)
+  tg-init-pulsar-manager
+  
+  # Initialize TrustGraph
+  tg-init-trustgraph \
+    --config "$(cat initial-config.json)"
+  
+  echo "TrustGraph initialization complete!"
+}
+
+wait_for_pulsar() {
+  local timeout=300
+  local elapsed=0
+  
+  while ! curl -s http://pulsar:8080/admin/v2/clusters > /dev/null; do
+    if [ $elapsed -ge $timeout ]; then
+      echo "Timeout waiting for Pulsar"
+      exit 1
+    fi
+    echo "Waiting for Pulsar..."
+    sleep 5
+    elapsed=$((elapsed + 5))
+  done
+}
+```
+
+### Environment-Specific Setup
+```bash
+# Development environment
+setup_dev() {
+  tg-init-trustgraph \
+    --pulsar-admin-url http://localhost:8080 \
+    --pulsar-host pulsar://localhost:6650 \
+    --tenant dev \
+    --config "$(cat dev-config.json)"
+}
+
+# Staging environment
+setup_staging() {
+  tg-init-trustgraph \
+    --pulsar-admin-url http://staging-pulsar:8080 \
+    --pulsar-host pulsar://staging-pulsar:6650 \
+    --tenant staging \
+    --config "$(cat staging-config.json)"
+}
+
+# Production environment
+setup_production() {
+  tg-init-trustgraph \
+    --pulsar-admin-url http://prod-pulsar:8080 \
+    --pulsar-host pulsar://prod-pulsar:6650 \
+    --pulsar-api-key "$PULSAR_API_KEY" \
+    --tenant production \
+    --config "$(cat production-config.json)"
+}
+```
+
+### Configuration Management
+```bash
+# Load different configurations
+load_ai_config() {
+  local config='{
+    "prompt": {
+      "system": "You are an AI assistant specialized in data analysis.",
+      "template-index": ["analyze", "summarize"],
+      "template.analyze": {
+        "id": "analyze",
+        "prompt": "Analyze this data: {{data}}",
+        "response-type": "json"
+      }
+    },
+    "token-costs": {
+      "gpt-4": {"input_price": 0.00003, "output_price": 0.00006},
+      "claude-3-sonnet": {"input_price": 0.000003, "output_price": 0.000015}
+    }
+  }'
+  
+  tg-init-trustgraph --config "$config"
+}
+
+load_research_config() {
+  local config='{
+    "prompt": {
+      "system": "You are a research assistant focused on academic literature.",
+      "template-index": ["research", "citation"],
+      "template.research": {
+        "id": "research",
+        "prompt": "Research question: {{question}}\nContext: {{context}}",
+        "response-type": "text"
+      }
+    }
+  }'
+  
+  tg-init-trustgraph --config "$config"
+}
+```
+
+## Advanced Usage
+
+### Cluster Setup
+```bash
+# Multi-cluster initialization
+setup_cluster() {
+  local clusters=("cluster1:8080" "cluster2:8080" "cluster3:8080")
+  
+  for cluster in "${clusters[@]}"; do
+    echo "Initializing cluster: $cluster"
+    
+    tg-init-trustgraph \
+      --pulsar-admin-url "http://$cluster" \
+      --pulsar-host "pulsar://${cluster%:*}:6650" \
+      --tenant "cluster-$(echo $cluster | cut -d: -f1)" \
+      --config "$(cat cluster-config.json)"
+  done
+}
+```
+
+### Configuration Migration
+```bash
+# Migrate configuration between environments
+migrate_config() {
+  local source_env="$1"
+  local target_env="$2"
+  
+  echo "Migrating configuration from $source_env to $target_env"
+  
+  # Export existing configuration (would need a tg-export-config command)
+  # For now, assume we have the config in a file
+  
+  tg-init-trustgraph \
+    --pulsar-admin-url "http://$target_env:8080" \
+    --pulsar-host "pulsar://$target_env:6650" \
+    --config "$(cat ${source_env}-config.json)"
+}
+```
+
+### Validation and Testing
+```bash
+# Validate initialization
+validate_initialization() {
+  local tenant="${1:-tg}"
+  local admin_url="${2:-http://pulsar:8080}"
+  
+  echo "Validating TrustGraph initialization..."
+  
+  # Check tenant exists
+  if curl -s "$admin_url/admin/v2/tenants/$tenant" > /dev/null; then
+    echo "✓ Tenant '$tenant' exists"
+  else
+    echo "✗ Tenant '$tenant' missing"
+    return 1
+  fi
+  
+  # Check namespaces
+  local namespaces=("flow" "request" "response" "config")
+  for ns in "${namespaces[@]}"; do
+    if curl -s "$admin_url/admin/v2/namespaces/$tenant/$ns" > /dev/null; then
+      echo "✓ Namespace '$tenant/$ns' exists"
+    else
+      echo "✗ Namespace '$tenant/$ns' missing"
+      return 1
+    fi
+  done
+  
+  echo "✓ TrustGraph initialization validated"
+}
+
+# Test configuration loading
+test_config_loading() {
+  local test_config='{
+    "test": {
+      "value": "test-value",
+      "timestamp": "'$(date -Iseconds)'"
+    }
+  }'
+  
+  echo "Testing configuration loading..."
+  
+  if tg-init-trustgraph --config "$test_config"; then
+    echo "✓ Configuration loading successful"
+  else
+    echo "✗ Configuration loading failed"
+    return 1
+  fi
+}
+```
+
+### Retry Logic and Error Handling
+```bash
+# Robust initialization with retry
+robust_init() {
+  local max_attempts=5
+  local attempt=1
+  local delay=10
+  
+  while [ $attempt -le $max_attempts ]; do
+    echo "Initialization attempt $attempt of $max_attempts..."
+    
+    if tg-init-trustgraph "$@"; then
+      echo "✓ Initialization successful on attempt $attempt"
+      return 0
+    else
+      echo "✗ Attempt $attempt failed"
+      
+      if [ $attempt -lt $max_attempts ]; then
+        echo "Waiting ${delay}s before retry..."
+        sleep $delay
+        delay=$((delay * 2))  # Exponential backoff
+      fi
+    fi
+    
+    attempt=$((attempt + 1))
+  done
+  
+  echo "✗ All initialization attempts failed"
+  return 1
+}
+```
+
+## Docker Integration
+
+### Docker Compose
+```yaml
+version: '3.8'
+
+services:
+  pulsar:
+    image: apachepulsar/pulsar:latest
+    ports:
+      - "6650:6650"
+      - "8080:8080"
+    command: bin/pulsar standalone
+    
+  trustgraph-init:
+    image: trustgraph/cli:latest
+    depends_on:
+      - pulsar
+    volumes:
+      - ./config.json:/config.json:ro
+    command: >
+      sh -c "
+        sleep 30 &&
+        tg-init-trustgraph --config '$$(cat /config.json)'
+      "
+    environment:
+      - TRUSTGRAPH_PULSAR_ADMIN_URL=http://pulsar:8080
+      - TRUSTGRAPH_PULSAR_HOST=pulsar://pulsar:6650
+```
+
+### Kubernetes Init Container
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: trustgraph-config
+data:
+  config.json: |
+    {
+      "prompt": {
+        "system": "You are a helpful AI assistant."
+      }
+    }
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: trustgraph-init
+spec:
+  template:
+    spec:
+      initContainers:
+      - name: wait-for-pulsar
+        image: busybox
+        command:
+        - sh
+        - -c
+        - |
+          until nc -z pulsar 8080; do
+            echo "Waiting for Pulsar..."
+            sleep 5
+          done
+      containers:
+      - name: init
+        image: trustgraph/cli:latest
+        command:
+        - tg-init-trustgraph
+        - --pulsar-admin-url=http://pulsar:8080
+        - --pulsar-host=pulsar://pulsar:6650
+        - --config=$(cat /config/config.json)
+        volumeMounts:
+        - name: config
+          mountPath: /config
+      volumes:
+      - name: config
+        configMap:
+          name: trustgraph-config
+      restartPolicy: Never
+```
+
+## Error Handling
+
+### Connection Issues
+```bash
+Exception: Connection refused
+```
+**Solution**: Verify Pulsar is running and accessible at the specified admin URL.
+
+### Authentication Errors
+```bash
+Exception: 401 Unauthorized
+```
+**Solution**: Check Pulsar API key if authentication is enabled.
+
+### Tenant Creation Failures
+```bash
+Exception: Tenant creation failed
+```
+**Solution**: Verify admin permissions and cluster configuration.
+
+### Configuration Loading Errors
+```bash
+Exception: Invalid JSON configuration
+```
+**Solution**: Validate JSON syntax and structure.
+
+## Security Considerations
+
+### API Key Management
+```bash
+# Use environment variables for sensitive data
+export PULSAR_API_KEY="your-secure-api-key"
+tg-init-trustgraph --pulsar-api-key "$PULSAR_API_KEY"
+
+# Or use a secure file
+tg-init-trustgraph --pulsar-api-key "$(cat /secure/pulsar-key.txt)"
+```
+
+### Network Security
+```bash
+# Use TLS for production
+tg-init-trustgraph \
+  --pulsar-admin-url https://secure-pulsar:8443 \
+  --pulsar-host pulsar+ssl://secure-pulsar:6651
+```
+
+## Related Commands
+
+- [`tg-init-pulsar-manager`](tg-init-pulsar-manager.md) - Initialize Pulsar Manager
+- [`tg-show-config`](tg-show-config.md) - Display current configuration
+- [`tg-set-prompt`](tg-set-prompt.md) - Configure individual prompts
+
+## Best Practices
+
+1. **Run Once**: Typically run once per environment during initial setup
+2. **Idempotent**: Safe to run multiple times - existing resources are preserved
+3. **Configuration**: Always load initial configuration during setup
+4. **Validation**: Verify initialization success with validation scripts
+5. **Environment Variables**: Use environment variables for sensitive configuration
+6. **Retry Logic**: Implement retry logic for robust deployments
+7. **Monitoring**: Monitor namespace and topic creation for issues
+
+## Troubleshooting
+
+### Pulsar Not Ready
+```bash
+# Check Pulsar health
+curl http://pulsar:8080/admin/v2/clusters
+
+# Check Pulsar logs
+docker logs pulsar
+```
+
+### Permission Issues
+```bash
+# Verify Pulsar admin access
+curl http://pulsar:8080/admin/v2/tenants
+
+# Check API key validity if using authentication
+```
+
+### Configuration Validation
+```bash
+# Validate JSON configuration
+echo "$CONFIG" | jq .
+
+# Test configuration loading separately
+tg-init-trustgraph --config '{"test": "value"}'
+```

docs/cli/tg-load-doc-embeds.md

@@ -0,0 +1,568 @@
+# tg-load-doc-embeds
+
+Loads document embeddings from MessagePack format into TrustGraph processing pipelines.
+
+## Synopsis
+
+```bash
+tg-load-doc-embeds -i INPUT_FILE [options]
+```
+
+## Description
+
+The `tg-load-doc-embeds` command loads document embeddings from MessagePack files into a running TrustGraph system. This is typically used to restore previously saved document embeddings or to load embeddings generated by external systems.
+
+The command reads document embedding data in MessagePack format and streams it to TrustGraph's document embeddings import API via WebSocket connections.
+
+## Options
+
+### Required Arguments
+
+- `-i, --input-file FILE`: Input MessagePack file containing document embeddings
+
+### Optional Arguments
+
+- `-u, --url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_API` or `http://localhost:8088/`)
+- `-f, --flow-id ID`: Flow instance ID to use (default: `default`)
+- `--format FORMAT`: Input format - `msgpack` or `json` (default: `msgpack`)
+- `--user USER`: Override user ID from input data
+- `--collection COLLECTION`: Override collection ID from input data
+
+## Examples
+
+### Basic Loading
+```bash
+tg-load-doc-embeds -i document-embeddings.msgpack
+```
+
+### Load with Custom Flow
+```bash
+tg-load-doc-embeds \
+  -i embeddings.msgpack \
+  -f "document-processing-flow"
+```
+
+### Override User and Collection
+```bash
+tg-load-doc-embeds \
+  -i embeddings.msgpack \
+  --user "research-team" \
+  --collection "research-docs"
+```
+
+### Load from JSON Format
+```bash
+tg-load-doc-embeds \
+  -i embeddings.json \
+  --format json
+```
+
+### Production Loading
+```bash
+tg-load-doc-embeds \
+  -i production-embeddings.msgpack \
+  -u https://trustgraph-api.company.com/ \
+  -f "production-flow" \
+  --user "system" \
+  --collection "production-docs"
+```
+
+## Input Data Format
+
+### MessagePack Structure
+Document embeddings are stored as MessagePack records with this structure:
+
+```json
+["de", {
+  "m": {
+    "i": "document-id",
+    "m": [{"metadata": "objects"}],
+    "u": "user-id", 
+    "c": "collection-id"
+  },
+  "c": [{
+    "c": "text chunk content",
+    "v": [0.1, 0.2, 0.3, ...]
+  }]
+}]
+```
+
+### Components
+- **Document Metadata** (`m`):
+  - `i`: Document ID
+  - `m`: Document metadata objects
+  - `u`: User ID
+  - `c`: Collection ID
+- **Chunks** (`c`): Array of text chunks with embeddings:
+  - `c`: Text content of the chunk
+  - `v`: Vector embedding array
+
+## Use Cases
+
+### Backup Restoration
+```bash
+# Restore document embeddings from backup
+restore_embeddings() {
+  local backup_file="$1"
+  local target_collection="$2"
+  
+  echo "Restoring document embeddings from: $backup_file"
+  
+  if [ ! -f "$backup_file" ]; then
+    echo "Backup file not found: $backup_file"
+    return 1
+  fi
+  
+  # Verify backup file
+  if tg-dump-msgpack -i "$backup_file" --summary | grep -q "Vector dimension:"; then
+    echo "✓ Backup file contains embeddings"
+  else
+    echo "✗ Backup file does not contain valid embeddings"
+    return 1
+  fi
+  
+  # Load embeddings
+  tg-load-doc-embeds \
+    -i "$backup_file" \
+    --collection "$target_collection"
+  
+  echo "Embedding restoration complete"
+}
+
+# Restore from backup
+restore_embeddings "backup-20231215.msgpack" "restored-docs"
+```
+
+### Data Migration
+```bash
+# Migrate embeddings between environments
+migrate_embeddings() {
+  local source_file="$1"
+  local target_env="$2"
+  local target_user="$3"
+  
+  echo "Migrating embeddings to: $target_env"
+  
+  # Load to target environment
+  tg-load-doc-embeds \
+    -i "$source_file" \
+    -u "https://$target_env/api/" \
+    --user "$target_user" \
+    --collection "migrated-docs"
+  
+  echo "Migration complete"
+}
+
+# Migrate to production
+migrate_embeddings "dev-embeddings.msgpack" "prod.company.com" "migration-user"
+```
+
+### Batch Processing
+```bash
+# Load multiple embedding files
+batch_load_embeddings() {
+  local input_dir="$1"
+  local collection="$2"
+  
+  echo "Batch loading embeddings from: $input_dir"
+  
+  for file in "$input_dir"/*.msgpack; do
+    if [ -f "$file" ]; then
+      echo "Loading: $(basename "$file")"
+      
+      tg-load-doc-embeds \
+        -i "$file" \
+        --collection "$collection"
+      
+      if [ $? -eq 0 ]; then
+        echo "✓ Loaded: $(basename "$file")"
+      else
+        echo "✗ Failed: $(basename "$file")"
+      fi
+    fi
+  done
+  
+  echo "Batch loading complete"
+}
+
+# Load all embeddings
+batch_load_embeddings "embeddings/" "batch-processed"
+```
+
+### Incremental Loading
+```bash
+# Load new embeddings incrementally
+incremental_load() {
+  local embeddings_dir="$1"
+  local processed_log="processed_embeddings.log"
+  
+  # Create log if it doesn't exist
+  touch "$processed_log"
+  
+  for file in "$embeddings_dir"/*.msgpack; do
+    if [ -f "$file" ]; then
+      # Check if already processed
+      if grep -q "$(basename "$file")" "$processed_log"; then
+        echo "Skipping already processed: $(basename "$file")"
+        continue
+      fi
+      
+      echo "Processing new file: $(basename "$file")"
+      
+      if tg-load-doc-embeds -i "$file"; then
+        echo "$(date): $(basename "$file")" >> "$processed_log"
+        echo "✓ Processed: $(basename "$file")"
+      else
+        echo "✗ Failed: $(basename "$file")"
+      fi
+    fi
+  done
+}
+
+# Run incremental loading
+incremental_load "embeddings/"
+```
+
+## Advanced Usage
+
+### Parallel Loading
+```bash
+# Load multiple files in parallel
+parallel_load_embeddings() {
+  local files=("$@")
+  local max_parallel=3
+  local current_jobs=0
+  
+  for file in "${files[@]}"; do
+    # Wait if max parallel jobs reached
+    while [ $current_jobs -ge $max_parallel ]; do
+      wait -n  # Wait for any job to complete
+      current_jobs=$((current_jobs - 1))
+    done
+    
+    # Start loading in background
+    (
+      echo "Loading: $file"
+      tg-load-doc-embeds -i "$file"
+      echo "Completed: $file"
+    ) &
+    
+    current_jobs=$((current_jobs + 1))
+  done
+  
+  # Wait for all remaining jobs
+  wait
+  echo "All parallel loading completed"
+}
+
+# Load files in parallel
+embedding_files=(embeddings1.msgpack embeddings2.msgpack embeddings3.msgpack)
+parallel_load_embeddings "${embedding_files[@]}"
+```
+
+### Validation and Loading
+```bash
+# Validate before loading
+validate_and_load() {
+  local file="$1"
+  local collection="$2"
+  
+  echo "Validating embedding file: $file"
+  
+  # Check file exists and is readable
+  if [ ! -r "$file" ]; then
+    echo "Error: Cannot read file $file"
+    return 1
+  fi
+  
+  # Validate MessagePack structure
+  if ! tg-dump-msgpack -i "$file" --summary > /dev/null 2>&1; then
+    echo "Error: Invalid MessagePack format"
+    return 1
+  fi
+  
+  # Check for document embeddings
+  if ! tg-dump-msgpack -i "$file" | grep -q '^\["de"'; then
+    echo "Error: No document embeddings found"
+    return 1
+  fi
+  
+  # Get embedding statistics
+  summary=$(tg-dump-msgpack -i "$file" --summary)
+  vector_dim=$(echo "$summary" | grep "Vector dimension:" | awk '{print $3}')
+  
+  if [ -n "$vector_dim" ]; then
+    echo "✓ Found embeddings with dimension: $vector_dim"
+  else
+    echo "Warning: Could not determine vector dimension"
+  fi
+  
+  # Load embeddings
+  echo "Loading validated embeddings..."
+  tg-load-doc-embeds -i "$file" --collection "$collection"
+  
+  echo "Loading complete"
+}
+
+# Validate and load
+validate_and_load "embeddings.msgpack" "validated-docs"
+```
+
+### Progress Monitoring
+```bash
+# Monitor loading progress
+monitor_loading() {
+  local file="$1"
+  local log_file="loading_progress.log"
+  
+  # Start loading in background
+  tg-load-doc-embeds -i "$file" > "$log_file" 2>&1 &
+  local load_pid=$!
+  
+  echo "Monitoring loading progress (PID: $load_pid)..."
+  
+  # Monitor progress
+  while kill -0 $load_pid 2>/dev/null; do
+    if [ -f "$log_file" ]; then
+      # Extract progress from log
+      embeddings_count=$(grep -o "Document embeddings:.*[0-9]" "$log_file" | tail -1 | awk '{print $3}')
+      if [ -n "$embeddings_count" ]; then
+        echo "Progress: $embeddings_count embeddings loaded"
+      fi
+    fi
+    sleep 5
+  done
+  
+  # Check final status
+  wait $load_pid
+  if [ $? -eq 0 ]; then
+    echo "✓ Loading completed successfully"
+  else
+    echo "✗ Loading failed"
+    cat "$log_file"
+  fi
+  
+  rm "$log_file"
+}
+
+# Monitor loading
+monitor_loading "large-embeddings.msgpack"
+```
+
+### Data Transformation
+```bash
+# Transform embeddings during loading
+transform_and_load() {
+  local input_file="$1"
+  local output_file="transformed-$(basename "$input_file")"
+  local new_user="$2"
+  local new_collection="$3"
+  
+  echo "Transforming embeddings: user=$new_user, collection=$new_collection"
+  
+  # This would require a transformation script
+  # For now, we'll show the concept
+  
+  # Load with override parameters
+  tg-load-doc-embeds \
+    -i "$input_file" \
+    --user "$new_user" \
+    --collection "$new_collection"
+  
+  echo "Transformation and loading complete"
+}
+
+# Transform during loading
+transform_and_load "original.msgpack" "new-user" "new-collection"
+```
+
+## Performance Optimization
+
+### Memory Management
+```bash
+# Monitor memory usage during loading
+monitor_memory_usage() {
+  local file="$1"
+  
+  echo "Starting memory-monitored loading..."
+  
+  # Start loading in background
+  tg-load-doc-embeds -i "$file" &
+  local load_pid=$!
+  
+  # Monitor memory usage
+  while kill -0 $load_pid 2>/dev/null; do
+    memory_usage=$(ps -p $load_pid -o rss= 2>/dev/null | awk '{print $1/1024}')
+    if [ -n "$memory_usage" ]; then
+      echo "Memory usage: ${memory_usage}MB"
+    fi
+    sleep 10
+  done
+  
+  wait $load_pid
+  echo "Loading completed"
+}
+```
+
+### Chunked Loading
+```bash
+# Load large files in chunks
+chunked_load() {
+  local large_file="$1"
+  local chunk_size=1000  # Records per chunk
+  
+  echo "Loading large file in chunks: $large_file"
+  
+  # Split the MessagePack file (this would need special tooling)
+  # For demonstration, assuming we have pre-split files
+  
+  for chunk in "${large_file%.msgpack}"_chunk_*.msgpack; do
+    if [ -f "$chunk" ]; then
+      echo "Loading chunk: $(basename "$chunk")"
+      tg-load-doc-embeds -i "$chunk"
+      
+      # Add delay between chunks to reduce system load
+      sleep 2
+    fi
+  done
+  
+  echo "Chunked loading complete"
+}
+```
+
+## Error Handling
+
+### File Not Found
+```bash
+Exception: [Errno 2] No such file or directory
+```
+**Solution**: Verify file path and ensure the MessagePack file exists.
+
+### Invalid Format
+```bash
+Exception: Unpack failed
+```
+**Solution**: Verify the file is a valid MessagePack file with document embeddings.
+
+### WebSocket Connection Issues
+```bash
+Exception: Connection failed
+```
+**Solution**: Check API URL and ensure TrustGraph is running with WebSocket support.
+
+### Memory Errors
+```bash
+MemoryError: Unable to allocate memory
+```
+**Solution**: Process large files in smaller chunks or increase available memory.
+
+### Flow Not Found
+```bash
+Exception: Flow not found
+```
+**Solution**: Verify the flow ID exists with `tg-show-flows`.
+
+## Integration with Other Commands
+
+### Complete Workflow
+```bash
+# Complete document processing workflow
+process_documents_workflow() {
+  local pdf_dir="$1"
+  local embeddings_file="embeddings.msgpack"
+  
+  echo "Starting complete document workflow..."
+  
+  # 1. Load PDFs
+  for pdf in "$pdf_dir"/*.pdf; do
+    tg-load-pdf "$pdf"
+  done
+  
+  # 2. Wait for processing
+  sleep 30
+  
+  # 3. Save embeddings
+  tg-save-doc-embeds -o "$embeddings_file"
+  
+  # 4. Process embeddings (example: load to different collection)
+  tg-load-doc-embeds -i "$embeddings_file" --collection "processed-docs"
+  
+  echo "Complete workflow finished"
+}
+```
+
+### Backup and Restore
+```bash
+# Complete backup and restore cycle
+backup_restore_cycle() {
+  local backup_file="embeddings-backup.msgpack"
+  
+  echo "Creating embeddings backup..."
+  tg-save-doc-embeds -o "$backup_file"
+  
+  echo "Simulating data loss..."
+  # (In real scenario, this might be system failure)
+  
+  echo "Restoring from backup..."
+  tg-load-doc-embeds -i "$backup_file" --collection "restored"
+  
+  echo "Backup/restore cycle complete"
+}
+```
+
+## Environment Variables
+
+- `TRUSTGRAPH_API`: Default API URL
+
+## Related Commands
+
+- [`tg-save-doc-embeds`](tg-save-doc-embeds.md) - Save document embeddings to MessagePack
+- [`tg-dump-msgpack`](tg-dump-msgpack.md) - Analyze MessagePack files
+- [`tg-load-pdf`](tg-load-pdf.md) - Load PDF documents for processing
+- [`tg-show-flows`](tg-show-flows.md) - List available flows
+
+## API Integration
+
+This command uses TrustGraph's WebSocket API for document embeddings import, specifically the `/api/v1/flow/{flow-id}/import/document-embeddings` endpoint.
+
+## Best Practices
+
+1. **Validation**: Always validate MessagePack files before loading
+2. **Backups**: Keep backups of original embedding files
+3. **Monitoring**: Monitor memory usage and loading progress
+4. **Chunking**: Process large files in manageable chunks
+5. **Error Handling**: Implement robust error handling and retry logic
+6. **Documentation**: Document the source and format of embedding files
+7. **Testing**: Test loading procedures in non-production environments
+
+## Troubleshooting
+
+### Loading Stalls
+```bash
+# Check WebSocket connection
+netstat -an | grep :8088
+
+# Check system resources
+free -h
+df -h
+```
+
+### Incomplete Loading
+```bash
+# Compare input vs loaded data
+input_count=$(tg-dump-msgpack -i input.msgpack | grep '^\["de"' | wc -l)
+echo "Input embeddings: $input_count"
+
+# Check loaded data (would need query command)
+# loaded_count=$(tg-query-embeddings --count)
+# echo "Loaded embeddings: $loaded_count"
+```
+
+### Performance Issues
+```bash
+# Monitor network usage
+iftop
+
+# Check TrustGraph service logs
+docker logs trustgraph-service
+```

docs/cli/tg-load-sample-documents.md

@@ -0,0 +1,567 @@
+# tg-load-sample-documents
+
+Loads predefined sample documents into TrustGraph library for testing and demonstration purposes.
+
+## Synopsis
+
+```bash
+tg-load-sample-documents [options]
+```
+
+## Description
+
+The `tg-load-sample-documents` command loads a curated set of sample documents into TrustGraph's document library. These documents include academic papers, government reports, and reference materials that demonstrate TrustGraph's capabilities and provide data for testing and evaluation.
+
+The command downloads documents from public sources and adds them to the library with comprehensive metadata including RDF triples for semantic relationships.
+
+## Options
+
+### Optional Arguments
+
+- `-u, --url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
+- `-U, --user USER`: User ID for document ownership (default: `trustgraph`)
+
+## Examples
+
+### Basic Loading
+```bash
+tg-load-sample-documents
+```
+
+### Load with Custom User
+```bash
+tg-load-sample-documents -U "demo-user"
+```
+
+### Load to Custom Environment
+```bash
+tg-load-sample-documents -u http://demo.trustgraph.ai:8088/
+```
+
+## Sample Documents
+
+The command loads the following sample documents:
+
+### 1. NASA Challenger Report
+- **Title**: Report of the Presidential Commission on the Space Shuttle Challenger Accident, Volume 1
+- **Topics**: Safety engineering, space shuttle, NASA
+- **Format**: PDF
+- **Source**: NASA Technical Reports Server
+- **Use Case**: Demonstrates technical document processing and safety analysis
+
+### 2. Old Icelandic Dictionary
+- **Title**: A Concise Dictionary of Old Icelandic
+- **Topics**: Language, linguistics, Old Norse, grammar
+- **Format**: PDF
+- **Publication**: 1910, Clarendon Press
+- **Use Case**: Historical document processing and linguistic analysis
+
+### 3. US Intelligence Threat Assessment
+- **Title**: Annual Threat Assessment of the U.S. Intelligence Community - March 2025
+- **Topics**: National security, cyberthreats, geopolitics
+- **Format**: PDF
+- **Source**: Director of National Intelligence
+- **Use Case**: Current affairs analysis and security research
+
+### 4. Intelligence and State Policy
+- **Title**: The Role of Intelligence and State Policies in International Security
+- **Topics**: Intelligence, international security, state policy
+- **Format**: PDF (sample)
+- **Publication**: Cambridge Scholars Publishing, 2021
+- **Use Case**: Academic research and policy analysis
+
+### 5. Globalization and Intelligence
+- **Title**: Beyond the Vigilant State: Globalisation and Intelligence
+- **Topics**: Intelligence, globalization, security studies
+- **Format**: PDF
+- **Author**: Richard J. Aldrich
+- **Use Case**: Academic paper analysis and research
+
+## Use Cases
+
+### Demo Environment Setup
+```bash
+# Set up demonstration environment
+setup_demo_environment() {
+  echo "Setting up TrustGraph demo environment..."
+  
+  # Initialize system
+  tg-init-trustgraph
+  
+  # Load sample documents
+  echo "Loading sample documents..."
+  tg-load-sample-documents -U "demo"
+  
+  # Wait for processing
+  echo "Waiting for document processing..."
+  sleep 60
+  
+  # Start document processing
+  echo "Starting document processing..."
+  tg-show-library-documents -U "demo" | \
+    grep "| id" | \
+    awk '{print $3}' | \
+    while read doc_id; do
+      proc_id="demo_proc_$(date +%s)_${doc_id}"
+      tg-start-library-processing -d "$doc_id" --id "$proc_id" -U "demo"
+    done
+  
+  echo "Demo environment ready!"
+  echo "Try: tg-invoke-document-rag -q 'What caused the Challenger accident?' -U demo"
+}
+```
+
+### Testing Data Pipeline
+```bash
+# Test complete document processing pipeline
+test_document_pipeline() {
+  echo "Testing document processing pipeline..."
+  
+  # Load sample documents
+  tg-load-sample-documents -U "test"
+  
+  # List loaded documents
+  echo "Loaded documents:"
+  tg-show-library-documents -U "test"
+  
+  # Start processing for each document
+  tg-show-library-documents -U "test" | \
+    grep "| id" | \
+    awk '{print $3}' | \
+    while read doc_id; do
+      echo "Processing document: $doc_id"
+      proc_id="test_$(date +%s)_${doc_id}"
+      tg-start-library-processing -d "$doc_id" --id "$proc_id" -U "test"
+    done
+  
+  # Wait for processing
+  echo "Processing documents... (this may take several minutes)"
+  sleep 300
+  
+  # Test document queries
+  echo "Testing document queries..."
+  
+  test_queries=(
+    "What is the Challenger accident?"
+    "What is Old Icelandic?"
+    "What are the main cybersecurity threats?"
+    "What is intelligence policy?"
+  )
+  
+  for query in "${test_queries[@]}"; do
+    echo "Query: $query"
+    tg-invoke-document-rag -q "$query" -U "test" | head -5
+    echo "---"
+  done
+  
+  echo "Pipeline test complete!"
+}
+```
+
+### Educational Environment
+```bash
+# Set up educational/training environment
+setup_educational_environment() {
+  local class_name="$1"
+  
+  echo "Setting up educational environment for: $class_name"
+  
+  # Create user for the class
+  class_user=$(echo "$class_name" | tr '[:upper:]' '[:lower:]' | tr ' ' '-')
+  
+  # Load sample documents for the class
+  tg-load-sample-documents -U "$class_user"
+  
+  # Process documents
+  echo "Processing documents for educational use..."
+  tg-show-library-documents -U "$class_user" | \
+    grep "| id" | \
+    awk '{print $3}' | \
+    while read doc_id; do
+      proc_id="edu_$(date +%s)_${doc_id}"
+      tg-start-library-processing \
+        -d "$doc_id" \
+        --id "$proc_id" \
+        -U "$class_user" \
+        --collection "education"
+    done
+  
+  echo "Educational environment ready for: $class_name"
+  echo "User: $class_user"
+  echo "Collection: education"
+}
+
+# Set up for different classes
+setup_educational_environment "AI Research Methods"
+setup_educational_environment "Security Studies"
+```
+
+### Benchmarking and Performance Testing
+```bash
+# Benchmark document processing performance
+benchmark_processing() {
+  echo "Starting document processing benchmark..."
+  
+  # Load sample documents
+  start_time=$(date +%s)
+  tg-load-sample-documents -U "benchmark"
+  load_time=$(date +%s)
+  
+  echo "Document loading time: $((load_time - start_time))s"
+  
+  # Count documents
+  doc_count=$(tg-show-library-documents -U "benchmark" | grep -c "| id")
+  echo "Documents loaded: $doc_count"
+  
+  # Start processing
+  processing_ids=()
+  tg-show-library-documents -U "benchmark" | \
+    grep "| id" | \
+    awk '{print $3}' | \
+    while read doc_id; do
+      proc_id="bench_$(date +%s)_${doc_id}"
+      processing_ids+=("$proc_id")
+      tg-start-library-processing -d "$doc_id" --id "$proc_id" -U "benchmark"
+    done
+  
+  processing_start=$(date +%s)
+  
+  # Monitor processing completion
+  echo "Monitoring processing completion..."
+  while true; do
+    active_processing=$(tg-show-flows | grep -c "bench_" || echo "0")
+    
+    if [ "$active_processing" -eq 0 ]; then
+      break
+    fi
+    
+    echo "Active processing jobs: $active_processing"
+    sleep 30
+  done
+  
+  processing_end=$(date +%s)
+  
+  echo "Processing completion time: $((processing_end - processing_start))s"
+  echo "Total benchmark time: $((processing_end - start_time))s"
+  
+  # Test query performance
+  echo "Testing query performance..."
+  query_start=$(date +%s)
+  
+  for i in {1..10}; do
+    tg-invoke-document-rag \
+      -q "What are the main topics in these documents?" \
+      -U "benchmark" > /dev/null
+  done
+  
+  query_end=$(date +%s)
+  echo "Average query time: $(echo "scale=2; ($query_end - $query_start) / 10" | bc)s"
+}
+```
+
+## Advanced Usage
+
+### Selective Document Loading
+```bash
+# Load only specific types of documents
+load_by_category() {
+  local category="$1"
+  
+  case "$category" in
+    "government")
+      echo "Loading government documents..."
+      # This would require modifying the script to load selectively
+      # For now, we load all and filter by tags later
+      tg-load-sample-documents -U "gov-docs"
+      ;;
+    "academic")
+      echo "Loading academic documents..."
+      tg-load-sample-documents -U "academic-docs"
+      ;;
+    "historical")
+      echo "Loading historical documents..."
+      tg-load-sample-documents -U "historical-docs"
+      ;;
+    *)
+      echo "Loading all sample documents..."
+      tg-load-sample-documents
+      ;;
+  esac
+}
+
+# Load by category
+load_by_category "government"
+load_by_category "academic"
+```
+
+### Multi-Environment Loading
+```bash
+# Load sample documents to multiple environments
+multi_environment_setup() {
+  local environments=("dev" "staging" "demo")
+  
+  for env in "${environments[@]}"; do
+    echo "Setting up $env environment..."
+    
+    tg-load-sample-documents \
+      -u "http://$env.trustgraph.company.com:8088/" \
+      -U "sample-data"
+    
+    echo "✓ $env environment loaded"
+  done
+  
+  echo "All environments loaded with sample documents"
+}
+```
+
+### Custom Document Sets
+```bash
+# Create custom document loading scripts based on the sample
+create_custom_loader() {
+  local domain="$1"
+  
+  cat > "load-${domain}-documents.py" << 'EOF'
+#!/usr/bin/env python3
+"""
+Custom document loader for specific domain
+Based on tg-load-sample-documents
+"""
+
+import argparse
+import os
+from trustgraph.api import Api
+
+# Define your own document set here
+documents = [
+    {
+        "id": "https://example.com/doc/custom-1",
+        "title": "Custom Document 1",
+        "url": "https://example.com/docs/custom1.pdf",
+        # Add your document definitions...
+    }
+]
+
+# Rest of the implementation similar to tg-load-sample-documents
+EOF
+  
+  echo "Custom loader created: load-${domain}-documents.py"
+}
+
+# Create custom loaders for different domains
+create_custom_loader "medical"
+create_custom_loader "legal"
+create_custom_loader "technical"
+```
+
+## Document Analysis
+
+### Content Analysis
+```bash
+# Analyze loaded sample documents
+analyze_sample_documents() {
+  echo "Analyzing sample documents..."
+  
+  # Get document statistics
+  total_docs=$(tg-show-library-documents | grep -c "| id")
+  echo "Total documents: $total_docs"
+  
+  # Analyze by type
+  echo "Document types:"
+  tg-show-library-documents | \
+    grep "| kind" | \
+    awk '{print $3}' | \
+    sort | uniq -c
+  
+  # Analyze tags
+  echo "Popular tags:"
+  tg-show-library-documents | \
+    grep "| tags" | \
+    sed 's/.*| tags.*| \(.*\) |.*/\1/' | \
+    tr ',' '\n' | \
+    sed 's/^ *//;s/ *$//' | \
+    sort | uniq -c | sort -nr | head -10
+  
+  # Document sizes (would need additional API)
+  echo "Document analysis complete"
+}
+```
+
+### Query Testing
+```bash
+# Test sample documents with various queries
+test_sample_queries() {
+  echo "Testing sample document queries..."
+  
+  # Define test queries for different domains
+  queries=(
+    "What caused the Challenger space shuttle accident?"
+    "What is Old Norse language?"
+    "What are current cybersecurity threats?"
+    "How does globalization affect intelligence services?"
+    "What are the main security challenges in international relations?"
+  )
+  
+  for query in "${queries[@]}"; do
+    echo "Testing query: $query"
+    echo "===================="
+    
+    result=$(tg-invoke-document-rag -q "$query" 2>/dev/null)
+    
+    if [ $? -eq 0 ]; then
+      echo "$result" | head -3
+      echo "✓ Query successful"
+    else
+      echo "✗ Query failed"
+    fi
+    
+    echo ""
+  done
+}
+```
+
+## Error Handling
+
+### Network Issues
+```bash
+Exception: Connection failed during download
+```
+**Solution**: Check internet connectivity and retry. Documents are cached locally after first download.
+
+### Insufficient Storage
+```bash
+Exception: No space left on device
+```
+**Solution**: Free up disk space. Sample documents total approximately 50-100MB.
+
+### API Connection Issues
+```bash
+Exception: Connection refused
+```
+**Solution**: Verify TrustGraph API is running and accessible.
+
+### Processing Failures
+```bash
+Exception: Document processing failed
+```
+**Solution**: Check TrustGraph service logs and ensure all components are running.
+
+## Monitoring and Validation
+
+### Loading Progress
+```bash
+# Monitor sample document loading
+monitor_sample_loading() {
+  echo "Starting sample document loading with monitoring..."
+  
+  # Start loading in background
+  tg-load-sample-documents &
+  load_pid=$!
+  
+  # Monitor progress
+  while kill -0 $load_pid 2>/dev/null; do
+    doc_count=$(tg-show-library-documents 2>/dev/null | grep -c "| id" || echo "0")
+    echo "Documents loaded so far: $doc_count"
+    sleep 10
+  done
+  
+  wait $load_pid
+  
+  if [ $? -eq 0 ]; then
+    final_count=$(tg-show-library-documents | grep -c "| id")
+    echo "✓ Loading completed successfully"
+    echo "Total documents loaded: $final_count"
+  else
+    echo "✗ Loading failed"
+  fi
+}
+```
+
+### Validation
+```bash
+# Validate sample document loading
+validate_sample_loading() {
+  echo "Validating sample document loading..."
+  
+  # Expected document count (based on current sample set)
+  expected_docs=5
+  
+  # Check actual count
+  actual_docs=$(tg-show-library-documents | grep -c "| id")
+  
+  if [ "$actual_docs" -eq "$expected_docs" ]; then
+    echo "✓ Document count correct: $actual_docs"
+  else
+    echo "⚠ Document count mismatch: expected $expected_docs, got $actual_docs"
+  fi
+  
+  # Check for expected documents
+  expected_titles=(
+    "Challenger"
+    "Icelandic"
+    "Intelligence"
+    "Threat Assessment"
+    "Vigilant State"
+  )
+  
+  for title in "${expected_titles[@]}"; do
+    if tg-show-library-documents | grep -q "$title"; then
+      echo "✓ Found document containing: $title"
+    else
+      echo "✗ Missing document containing: $title"
+    fi
+  done
+  
+  echo "Validation complete"
+}
+```
+
+## Environment Variables
+
+- `TRUSTGRAPH_URL`: Default API URL
+
+## Related Commands
+
+- [`tg-show-library-documents`](tg-show-library-documents.md) - List loaded documents
+- [`tg-start-library-processing`](tg-start-library-processing.md) - Process loaded documents
+- [`tg-invoke-document-rag`](tg-invoke-document-rag.md) - Query processed documents
+- [`tg-load-pdf`](tg-load-pdf.md) - Load individual PDF documents
+
+## API Integration
+
+This command uses the [Library API](../apis/api-librarian.md) to add sample documents to TrustGraph's document repository.
+
+## Best Practices
+
+1. **Demo Preparation**: Use for setting up demonstration environments
+2. **Testing**: Ideal for testing document processing pipelines
+3. **Education**: Excellent for training and educational purposes
+4. **Development**: Use in development environments for consistent test data
+5. **Benchmarking**: Suitable for performance testing and optimization
+6. **Documentation**: Great for documenting TrustGraph capabilities
+
+## Troubleshooting
+
+### Download Failures
+```bash
+# Check document URLs are accessible
+curl -I "https://ntrs.nasa.gov/api/citations/19860015255/downloads/19860015255.pdf"
+
+# Check local cache
+ls -la doc-cache/
+```
+
+### Processing Issues
+```bash
+# Check document processing status
+tg-show-library-processing
+
+# Verify documents are in library
+tg-show-library-documents | grep -E "(Challenger|Icelandic|Intelligence)"
+```
+
+### Performance Problems
+```bash
+# Monitor system resources during loading
+top
+df -h
+```