More commands

This commit is contained in:
Cyber MacGeddon 2025-07-03 11:14:37 +01:00
parent 3014b11ce9
commit 312375fa07
12 changed files with 4769 additions and 0 deletions

View file

@ -0,0 +1,330 @@
# tg-delete-flow-class
Permanently deletes a flow class definition from TrustGraph.
## Synopsis
```bash
tg-delete-flow-class -n CLASS_NAME [options]
```
## Description
The `tg-delete-flow-class` command permanently removes a flow class definition from TrustGraph. This operation cannot be undone, so use with caution.
**⚠️ Warning**: Deleting a flow class that has active flow instances may cause those instances to become unusable. Always check for active flows before deletion.
## Options
### Required Arguments
- `-n, --class-name CLASS_NAME`: Name of the flow class to delete
### Optional Arguments
- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
## Examples
### Delete a Flow Class
```bash
tg-delete-flow-class -n "old-test-flow"
```
### Delete with Custom API URL
```bash
tg-delete-flow-class -n "deprecated-flow" -u http://staging:8088/
```
### Safe Deletion Workflow
```bash
# 1. Check if flow class exists
tg-show-flow-classes | grep "target-flow"
# 2. Backup the flow class first
tg-get-flow-class -n "target-flow" > backup-target-flow.json
# 3. Check for active flow instances
tg-show-flows | grep "target-flow"
# 4. Delete the flow class
tg-delete-flow-class -n "target-flow"
# 5. Verify deletion
tg-show-flow-classes | grep "target-flow" || echo "Flow class deleted successfully"
```
## Prerequisites
### Flow Class Must Exist
Verify the flow class exists before attempting deletion:
```bash
# List all flow classes
tg-show-flow-classes
# Check specific flow class
tg-show-flow-classes | grep "target-class"
```
### Check for Active Flow Instances
Before deleting a flow class, check if any flow instances are using it:
```bash
# List all active flows
tg-show-flows
# Look for instances using the flow class
tg-show-flows | grep "target-class"
```
## Error Handling
### Flow Class Not Found
```bash
Exception: Flow class 'nonexistent-class' not found
```
**Solution**: Verify the flow class exists with `tg-show-flow-classes`.
### Connection Errors
```bash
Exception: Connection refused
```
**Solution**: Check the API URL and ensure TrustGraph is running.
### Permission Errors
```bash
Exception: Access denied to delete flow class
```
**Solution**: Verify user permissions for flow class management.
### Active Flow Instances
```bash
Exception: Cannot delete flow class with active instances
```
**Solution**: Stop all flow instances using this class before deletion.
## Use Cases
### Cleanup Development Classes
```bash
# Delete test and development flow classes
test_classes=("test-flow-v1" "dev-experiment" "prototype-flow")
for class in "${test_classes[@]}"; do
echo "Deleting $class..."
tg-delete-flow-class -n "$class"
done
```
### Migration Cleanup
```bash
# After migrating to new flow classes, remove old ones
old_classes=("legacy-flow" "deprecated-processor" "old-pipeline")
for class in "${old_classes[@]}"; do
# Backup first
tg-get-flow-class -n "$class" > "backup-$class.json" 2>/dev/null
# Delete
tg-delete-flow-class -n "$class"
echo "Deleted $class"
done
```
### Conditional Deletion
```bash
# Delete flow class only if no active instances exist
flow_class="target-flow"
active_instances=$(tg-show-flows | grep "$flow_class" | wc -l)
if [ $active_instances -eq 0 ]; then
echo "No active instances found, deleting flow class..."
tg-delete-flow-class -n "$flow_class"
else
echo "Warning: $active_instances active instances found. Cannot delete."
tg-show-flows | grep "$flow_class"
fi
```
## Safety Considerations
### Always Backup First
```bash
# Create backup before deletion
flow_class="important-flow"
backup_dir="flow-class-backups/$(date +%Y%m%d-%H%M%S)"
mkdir -p "$backup_dir"
echo "Backing up flow class: $flow_class"
tg-get-flow-class -n "$flow_class" > "$backup_dir/$flow_class.json"
if [ $? -eq 0 ]; then
echo "Backup created: $backup_dir/$flow_class.json"
echo "Proceeding with deletion..."
tg-delete-flow-class -n "$flow_class"
else
echo "Backup failed. Aborting deletion."
exit 1
fi
```
### Verification Script
```bash
#!/bin/bash
# safe-delete-flow-class.sh
flow_class="$1"
if [ -z "$flow_class" ]; then
echo "Usage: $0 <flow-class-name>"
exit 1
fi
echo "Safety checks for deleting flow class: $flow_class"
# Check if flow class exists
if ! tg-show-flow-classes | grep -q "$flow_class"; then
echo "ERROR: Flow class '$flow_class' not found"
exit 1
fi
# Check for active instances
active_count=$(tg-show-flows | grep "$flow_class" | wc -l)
if [ $active_count -gt 0 ]; then
echo "ERROR: Found $active_count active instances using this flow class"
echo "Active instances:"
tg-show-flows | grep "$flow_class"
exit 1
fi
# Create backup
backup_file="backup-$flow_class-$(date +%Y%m%d-%H%M%S).json"
echo "Creating backup: $backup_file"
tg-get-flow-class -n "$flow_class" > "$backup_file"
if [ $? -ne 0 ]; then
echo "ERROR: Failed to create backup"
exit 1
fi
# Confirm deletion
echo "Ready to delete flow class: $flow_class"
echo "Backup saved as: $backup_file"
read -p "Are you sure you want to delete this flow class? (y/N): " confirm
if [ "$confirm" = "y" ] || [ "$confirm" = "Y" ]; then
echo "Deleting flow class..."
tg-delete-flow-class -n "$flow_class"
# Verify deletion
if ! tg-show-flow-classes | grep -q "$flow_class"; then
echo "Flow class deleted successfully"
else
echo "ERROR: Flow class still exists after deletion"
exit 1
fi
else
echo "Deletion cancelled"
rm "$backup_file"
fi
```
## Integration with Other Commands
### Complete Flow Class Lifecycle
```bash
# 1. List existing flow classes
tg-show-flow-classes
# 2. Get flow class details
tg-get-flow-class -n "target-flow"
# 3. Check for active instances
tg-show-flows | grep "target-flow"
# 4. Stop active instances if needed
tg-stop-flow -i "instance-id"
# 5. Create backup
tg-get-flow-class -n "target-flow" > backup.json
# 6. Delete flow class
tg-delete-flow-class -n "target-flow"
# 7. Verify deletion
tg-show-flow-classes | grep "target-flow"
```
### Bulk Deletion with Validation
```bash
# Delete multiple flow classes safely
classes_to_delete=("old-flow1" "old-flow2" "test-flow")
for class in "${classes_to_delete[@]}"; do
echo "Processing $class..."
# Check if exists
if ! tg-show-flow-classes | grep -q "$class"; then
echo " $class not found, skipping"
continue
fi
# Check for active instances
if tg-show-flows | grep -q "$class"; then
echo " $class has active instances, skipping"
continue
fi
# Backup and delete
tg-get-flow-class -n "$class" > "backup-$class.json"
tg-delete-flow-class -n "$class"
echo " $class deleted"
done
```
## Environment Variables
- `TRUSTGRAPH_URL`: Default API URL
## Related Commands
- [`tg-show-flow-classes`](tg-show-flow-classes.md) - List available flow classes
- [`tg-get-flow-class`](tg-get-flow-class.md) - Retrieve flow class definitions
- [`tg-put-flow-class`](tg-put-flow-class.md) - Create/update flow class definitions
- [`tg-show-flows`](tg-show-flows.md) - List active flow instances
- [`tg-stop-flow`](tg-stop-flow.md) - Stop flow instances
## API Integration
This command uses the [Flow API](../apis/api-flow.md) with the `delete-class` operation to remove flow class definitions.
## Best Practices
1. **Always Backup**: Create backups before deletion
2. **Check Dependencies**: Verify no active flow instances exist
3. **Confirmation**: Use interactive confirmation for important deletions
4. **Logging**: Log deletion operations for audit trails
5. **Permissions**: Ensure appropriate access controls for deletion operations
6. **Testing**: Test deletion procedures in non-production environments first
## Troubleshooting
### Command Succeeds but Class Still Exists
```bash
# Check if deletion actually occurred
tg-show-flow-classes | grep "deleted-class"
# Verify API connectivity
tg-show-flow-classes > /dev/null && echo "API accessible"
```
### Permissions Issues
```bash
# Verify user has deletion permissions
# Contact system administrator if access denied
```
### Network Connectivity
```bash
# Test API connectivity
curl -s "$TRUSTGRAPH_URL/api/v1/flow/classes" > /dev/null
echo "API response: $?"
```

View file

@ -0,0 +1,312 @@
# tg-delete-kg-core
Permanently removes a knowledge core from the TrustGraph system.
## Synopsis
```bash
tg-delete-kg-core --id CORE_ID [options]
```
## Description
The `tg-delete-kg-core` command permanently removes a stored knowledge core from the TrustGraph system. This operation is irreversible and will delete all RDF triples, graph embeddings, and metadata associated with the specified knowledge core.
**Warning**: This operation permanently deletes data. Ensure you have backups if the knowledge core might be needed in the future.
## Options
### Required Arguments
- `--id, --identifier CORE_ID`: Identifier of the knowledge core to delete
### Optional Arguments
- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
- `-U, --user USER`: User identifier (default: `trustgraph`)
## Examples
### Delete Specific Knowledge Core
```bash
tg-delete-kg-core --id "old-research-data"
```
### Delete with Specific User
```bash
tg-delete-kg-core --id "test-knowledge" -U developer
```
### Using Custom API URL
```bash
tg-delete-kg-core --id "obsolete-core" -u http://production:8088/
```
## Prerequisites
### Knowledge Core Must Exist
Verify the knowledge core exists before deletion:
```bash
# Check available knowledge cores
tg-show-kg-cores
# Ensure the core exists
tg-show-kg-cores | grep "target-core-id"
```
### Backup Important Data
Create backups before deletion:
```bash
# Export knowledge core before deletion
tg-get-kg-core --id "important-core" -o backup.msgpack
# Then proceed with deletion
tg-delete-kg-core --id "important-core"
```
## Safety Considerations
### Unload from Flows First
Unload the knowledge core from any active flows:
```bash
# Check which flows might be using the core
tg-show-flows
# Unload from active flows
tg-unload-kg-core --id "target-core" --flow-id "active-flow"
# Then delete the core
tg-delete-kg-core --id "target-core"
```
### Verify Dependencies
Check if other systems depend on the knowledge core:
```bash
# Search for references in flow configurations
tg-show-config | grep "target-core"
# Check processing history
tg-show-library-processing | grep "target-core"
```
## Deletion Process
1. **Validation**: Verifies knowledge core exists and user has permission
2. **Dependency Check**: Ensures core is not actively loaded in flows
3. **Data Removal**: Permanently deletes RDF triples and graph embeddings
4. **Metadata Cleanup**: Removes all associated metadata and references
5. **Index Updates**: Updates system indexes to reflect deletion
## Output
Successful deletion typically produces no output:
```bash
# Delete core (no output expected on success)
tg-delete-kg-core --id "test-core"
# Verify deletion
tg-show-kg-cores | grep "test-core"
# Should return no results
```
## Error Handling
### Knowledge Core Not Found
```bash
Exception: Knowledge core 'invalid-core' not found
```
**Solution**: Check available cores with `tg-show-kg-cores` and verify the core ID.
### Permission Denied
```bash
Exception: Access denied to knowledge core
```
**Solution**: Verify user permissions and ownership of the knowledge core.
### Core In Use
```bash
Exception: Knowledge core is currently loaded in active flows
```
**Solution**: Unload the core from all flows before deletion using `tg-unload-kg-core`.
### Connection Errors
```bash
Exception: Connection refused
```
**Solution**: Check the API URL and ensure TrustGraph is running.
## Deletion Verification
### Confirm Deletion
```bash
# Verify core no longer exists
tg-show-kg-cores | grep "deleted-core-id"
# Should return no results if successfully deleted
echo $? # Should be 1 (not found)
```
### Check Flow Impact
```bash
# Verify flows are not affected
tg-show-flows
# Test that queries still work for remaining knowledge
tg-invoke-graph-rag -q "test query" -f remaining-flow
```
## Environment Variables
- `TRUSTGRAPH_URL`: Default API URL
## Related Commands
- [`tg-show-kg-cores`](tg-show-kg-cores.md) - List available knowledge cores
- [`tg-get-kg-core`](tg-get-kg-core.md) - Export knowledge core for backup
- [`tg-unload-kg-core`](tg-unload-kg-core.md) - Unload core from flows
- [`tg-put-kg-core`](tg-put-kg-core.md) - Store new knowledge cores
## API Integration
This command uses the [Knowledge API](../apis/api-knowledge.md) with the `delete-kg-core` operation to permanently remove knowledge cores.
## Use Cases
### Development Cleanup
```bash
# Remove test knowledge cores
tg-delete-kg-core --id "test-data-v1" -U developer
tg-delete-kg-core --id "experimental-core" -U developer
```
### Version Management
```bash
# Remove obsolete versions after upgrading
tg-get-kg-core --id "knowledge-v1" -o backup-v1.msgpack
tg-delete-kg-core --id "knowledge-v1"
# Keep only knowledge-v2
```
### Storage Cleanup
```bash
# Clean up unused knowledge cores
for core in $(tg-show-kg-cores | grep "temp-"); do
echo "Deleting temporary core: $core"
tg-delete-kg-core --id "$core"
done
```
### Error Recovery
```bash
# Remove corrupted knowledge cores
tg-delete-kg-core --id "corrupted-core-2024"
tg-put-kg-core --id "restored-core-2024" -i restored-backup.msgpack
```
## Safe Deletion Workflow
### Standard Procedure
```bash
# 1. Backup the knowledge core
tg-get-kg-core --id "target-core" -o "backup-$(date +%Y%m%d).msgpack"
# 2. Unload from active flows
tg-unload-kg-core --id "target-core" --flow-id "production-flow"
# 3. Verify no dependencies
tg-show-config | grep "target-core"
# 4. Perform deletion
tg-delete-kg-core --id "target-core"
# 5. Verify deletion
tg-show-kg-cores | grep "target-core"
```
### Bulk Deletion
```bash
# Delete multiple cores safely
cores_to_delete=("old-core-1" "old-core-2" "test-core")
for core in "${cores_to_delete[@]}"; do
echo "Processing $core..."
# Backup
tg-get-kg-core --id "$core" -o "backup-$core-$(date +%Y%m%d).msgpack"
# Delete
tg-delete-kg-core --id "$core"
# Verify
if tg-show-kg-cores | grep -q "$core"; then
echo "ERROR: $core still exists after deletion"
else
echo "SUCCESS: $core deleted"
fi
done
```
## Best Practices
1. **Always Backup**: Export knowledge cores before deletion
2. **Check Dependencies**: Verify no flows are using the core
3. **Staged Deletion**: Delete test/development cores before production
4. **Verification**: Confirm deletion completed successfully
5. **Documentation**: Record why cores were deleted for audit purposes
6. **Access Control**: Ensure only authorized users can delete cores
## Recovery Options
### If Accidentally Deleted
```bash
# Restore from backup if available
tg-put-kg-core --id "restored-core" -i backup.msgpack
# Reload into flows if needed
tg-load-kg-core --id "restored-core" --flow-id "production-flow"
```
### Audit Trail
```bash
# Keep records of deletions
echo "$(date): Deleted knowledge core 'old-core' - reason: obsolete version" >> deletion-log.txt
```
## System Impact
### Storage Recovery
- Disk space is freed immediately
- Database indexes are updated
- System performance may improve
### Service Continuity
- Running flows continue to operate
- Other knowledge cores remain unaffected
- New knowledge cores can use the same ID
## Troubleshooting
### Deletion Fails
```bash
# Check if core is loaded in flows
tg-show-flows | grep -A 10 "knowledge"
# Force unload if necessary
tg-unload-kg-core --id "stuck-core" --flow-id "problem-flow"
# Retry deletion
tg-delete-kg-core --id "stuck-core"
```
### Partial Deletion
```bash
# If core still appears in listings
tg-show-kg-cores | grep "partially-deleted"
# Contact system administrator if deletion appears incomplete
```

View file

@ -0,0 +1,344 @@
# tg-get-flow-class
Retrieves and displays a flow class definition in JSON format.
## Synopsis
```bash
tg-get-flow-class -n CLASS_NAME [options]
```
## Description
The `tg-get-flow-class` command retrieves a stored flow class definition from TrustGraph and displays it in formatted JSON. This is useful for examining flow class configurations, creating backups, or preparing to modify existing flow classes.
The output can be saved to files for version control, documentation, or as input for creating new flow classes with `tg-put-flow-class`.
## Options
### Required Arguments
- `-n, --class-name CLASS_NAME`: Name of the flow class to retrieve
### Optional Arguments
- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
## Examples
### Display Flow Class Definition
```bash
tg-get-flow-class -n "document-processing"
```
### Save Flow Class to File
```bash
tg-get-flow-class -n "production-flow" > production-flow-backup.json
```
### Compare Flow Classes
```bash
# Get multiple flow classes for comparison
tg-get-flow-class -n "dev-flow" > dev-flow.json
tg-get-flow-class -n "prod-flow" > prod-flow.json
diff dev-flow.json prod-flow.json
```
### Using Custom API URL
```bash
tg-get-flow-class -n "remote-flow" -u http://production:8088/
```
## Output Format
The command outputs the flow class definition in formatted JSON:
```json
{
"description": "Document processing and analysis flow",
"interfaces": {
"agent": {
"request": "non-persistent://tg/request/agent:doc-proc",
"response": "non-persistent://tg/response/agent:doc-proc"
},
"document-rag": {
"request": "non-persistent://tg/request/document-rag:doc-proc",
"response": "non-persistent://tg/response/document-rag:doc-proc"
},
"text-load": "persistent://tg/flow/text-document-load:doc-proc",
"document-load": "persistent://tg/flow/document-load:doc-proc",
"triples-store": "persistent://tg/flow/triples-store:doc-proc"
},
"tags": ["production", "document-processing"]
}
```
### Key Components
#### Description
Human-readable description of the flow class purpose and capabilities.
#### Interfaces
Service definitions showing:
- **Request/Response Services**: Services with both request and response queues
- **Fire-and-Forget Services**: Services with only input queues
#### Tags (Optional)
Categorization tags for organizing flow classes.
## Prerequisites
### Flow Class Must Exist
Verify the flow class exists before retrieval:
```bash
# Check available flow classes
tg-show-flow-classes
# Look for specific class
tg-show-flow-classes | grep "target-class"
```
## Error Handling
### Flow Class Not Found
```bash
Exception: Flow class 'invalid-class' not found
```
**Solution**: Check available classes with `tg-show-flow-classes` and verify the class name.
### Connection Errors
```bash
Exception: Connection refused
```
**Solution**: Check the API URL and ensure TrustGraph is running.
### Permission Errors
```bash
Exception: Access denied to flow class
```
**Solution**: Verify user permissions for accessing flow class definitions.
## Use Cases
### Configuration Backup
```bash
# Backup all flow classes
mkdir -p flow-class-backups/$(date +%Y%m%d)
tg-show-flow-classes | awk '{print $1}' | while read class; do
if [ "$class" != "flow" ]; then # Skip header
tg-get-flow-class -n "$class" > "flow-class-backups/$(date +%Y%m%d)/$class.json"
fi
done
```
### Flow Class Migration
```bash
# Export from source environment
tg-get-flow-class -n "production-flow" -u http://source:8088/ > prod-flow.json
# Import to target environment
tg-put-flow-class -n "production-flow" -c "$(cat prod-flow.json)" -u http://target:8088/
```
### Template Creation
```bash
# Get existing flow class as template
tg-get-flow-class -n "base-flow" > template.json
# Modify template and create new class
sed 's/base-flow/new-flow/g' template.json > new-flow.json
tg-put-flow-class -n "custom-flow" -c "$(cat new-flow.json)"
```
### Configuration Analysis
```bash
# Analyze flow class configurations
tg-get-flow-class -n "complex-flow" | jq '.interfaces | keys'
tg-get-flow-class -n "complex-flow" | jq '.interfaces | length'
```
### Version Control Integration
```bash
# Store flow classes in git
mkdir -p flow-classes
tg-get-flow-class -n "main-flow" > flow-classes/main-flow.json
git add flow-classes/main-flow.json
git commit -m "Update main-flow configuration"
```
## JSON Processing
### Extract Specific Information
```bash
# Get only interface names
tg-get-flow-class -n "my-flow" | jq -r '.interfaces | keys[]'
# Get only description
tg-get-flow-class -n "my-flow" | jq -r '.description'
# Get request queues
tg-get-flow-class -n "my-flow" | jq -r '.interfaces | to_entries[] | select(.value.request) | .value.request'
```
### Validate Configuration
```bash
# Validate JSON structure
tg-get-flow-class -n "my-flow" | jq . > /dev/null && echo "Valid JSON" || echo "Invalid JSON"
# Check required fields
config=$(tg-get-flow-class -n "my-flow")
echo "$config" | jq -e '.description' > /dev/null || echo "Missing description"
echo "$config" | jq -e '.interfaces' > /dev/null || echo "Missing interfaces"
```
## Integration with Other Commands
### Flow Class Lifecycle
```bash
# 1. Examine existing flow class
tg-get-flow-class -n "old-flow"
# 2. Save backup
tg-get-flow-class -n "old-flow" > old-flow-backup.json
# 3. Modify configuration
cp old-flow-backup.json new-flow.json
# Edit new-flow.json as needed
# 4. Upload new version
tg-put-flow-class -n "updated-flow" -c "$(cat new-flow.json)"
# 5. Test new flow class
tg-start-flow -n "updated-flow" -i "test-instance" -d "Testing updated flow"
```
### Bulk Operations
```bash
# Process multiple flow classes
flow_classes=("flow1" "flow2" "flow3")
for class in "${flow_classes[@]}"; do
echo "Processing $class..."
tg-get-flow-class -n "$class" > "backup-$class.json"
# Modify configuration
sed 's/old-pattern/new-pattern/g' "backup-$class.json" > "updated-$class.json"
# Upload updated version
tg-put-flow-class -n "$class" -c "$(cat updated-$class.json)"
done
```
## Environment Variables
- `TRUSTGRAPH_URL`: Default API URL
## Related Commands
- [`tg-put-flow-class`](tg-put-flow-class.md) - Upload/update flow class definitions
- [`tg-show-flow-classes`](tg-show-flow-classes.md) - List available flow classes
- [`tg-delete-flow-class`](tg-delete-flow-class.md) - Remove flow class definitions
- [`tg-start-flow`](tg-start-flow.md) - Create flow instances from classes
## API Integration
This command uses the [Flow API](../apis/api-flow.md) with the `get-class` operation to retrieve flow class definitions.
## Advanced Usage
### Configuration Diff
```bash
# Compare flow class versions
tg-get-flow-class -n "flow-v1" > v1.json
tg-get-flow-class -n "flow-v2" > v2.json
diff -u v1.json v2.json
```
### Extract Queue Information
```bash
# Get all queue names from flow class
tg-get-flow-class -n "my-flow" | jq -r '
.interfaces |
to_entries[] |
if .value | type == "object" then
.value.request, .value.response
else
.value
end
' | sort | uniq
```
### Configuration Validation Script
```bash
#!/bin/bash
# validate-flow-class.sh
flow_class="$1"
if [ -z "$flow_class" ]; then
echo "Usage: $0 <flow-class-name>"
exit 1
fi
echo "Validating flow class: $flow_class"
# Get configuration
config=$(tg-get-flow-class -n "$flow_class" 2>/dev/null)
if [ $? -ne 0 ]; then
echo "ERROR: Flow class not found"
exit 1
fi
# Validate JSON
echo "$config" | jq . > /dev/null
if [ $? -ne 0 ]; then
echo "ERROR: Invalid JSON structure"
exit 1
fi
# Check required fields
desc=$(echo "$config" | jq -r '.description // empty')
if [ -z "$desc" ]; then
echo "WARNING: Missing description"
fi
interfaces=$(echo "$config" | jq -r '.interfaces // empty')
if [ -z "$interfaces" ] || [ "$interfaces" = "null" ]; then
echo "ERROR: Missing interfaces"
exit 1
fi
echo "Flow class validation passed"
```
## Best Practices
1. **Regular Backups**: Save flow class definitions before modifications
2. **Version Control**: Store configurations in version control systems
3. **Documentation**: Include meaningful descriptions in flow classes
4. **Validation**: Validate JSON structure before using configurations
5. **Template Management**: Use existing classes as templates for new ones
6. **Change Tracking**: Document changes when updating flow classes
## Troubleshooting
### Empty Output
```bash
# If command returns empty output
tg-get-flow-class -n "my-flow"
# Check if flow class exists
tg-show-flow-classes | grep "my-flow"
```
### Invalid JSON Output
```bash
# If output appears corrupted
tg-get-flow-class -n "my-flow" | jq .
# Should show parsing error if JSON is invalid
```
### Permission Issues
```bash
# If access denied errors occur
# Verify authentication and user permissions
# Contact system administrator if needed
```

365
docs/cli/tg-get-kg-core.md Normal file
View file

@ -0,0 +1,365 @@
# tg-get-kg-core
Exports a knowledge core from TrustGraph to a MessagePack file.
## Synopsis
```bash
tg-get-kg-core --id CORE_ID -o OUTPUT_FILE [options]
```
## Description
The `tg-get-kg-core` command retrieves a stored knowledge core from TrustGraph and exports it to a MessagePack format file. This allows you to backup knowledge cores, transfer them between systems, or examine their contents offline.
The exported file contains both RDF triples and graph embeddings in a compact binary format that can later be imported using `tg-put-kg-core`.
## Options
### Required Arguments
- `--id, --identifier CORE_ID`: Identifier of the knowledge core to export
- `-o, --output OUTPUT_FILE`: Path for the output MessagePack file
### Optional Arguments
- `-u, --url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `ws://localhost:8088/`)
- `-U, --user USER`: User identifier (default: `trustgraph`)
## Examples
### Basic Knowledge Core Export
```bash
tg-get-kg-core --id "research-knowledge" -o research-backup.msgpack
```
### Export with Specific User
```bash
tg-get-kg-core \
--id "medical-knowledge" \
-o medical-backup.msgpack \
-U medical-team
```
### Export with Timestamped Filename
```bash
tg-get-kg-core \
--id "production-core" \
-o "production-backup-$(date +%Y%m%d-%H%M%S).msgpack"
```
### Using Custom API URL
```bash
tg-get-kg-core \
--id "remote-core" \
-o remote-backup.msgpack \
-u ws://production:8088/
```
## Prerequisites
### Knowledge Core Must Exist
Verify the knowledge core exists:
```bash
# Check available knowledge cores
tg-show-kg-cores
# Verify specific core exists
tg-show-kg-cores | grep "target-core-id"
```
### Output Directory Must Be Writable
Ensure the output directory exists and is writable:
```bash
# Create backup directory if needed
mkdir -p backups
# Export to backup directory
tg-get-kg-core --id "my-core" -o backups/my-core-backup.msgpack
```
## Export Process
1. **Connection**: Establishes WebSocket connection to Knowledge API
2. **Request**: Sends get-kg-core request with core ID and user
3. **Streaming**: Receives data in chunks via WebSocket
4. **Processing**: Converts response data to MessagePack format
5. **Writing**: Writes binary data to output file
6. **Summary**: Reports statistics on exported data
## Output Format
The exported MessagePack file contains structured data with two types of messages:
### Triple Messages (`"t"`)
Contains RDF triples (facts and relationships):
```python
("t", {
"m": { # metadata
"i": "core-id",
"m": [], # metadata triples
"u": "user",
"c": "collection"
},
"t": [ # triples array
{
"s": {"value": "subject", "is_uri": true},
"p": {"value": "predicate", "is_uri": true},
"o": {"value": "object", "is_uri": false}
}
]
})
```
### Graph Embedding Messages (`"ge"`)
Contains vector embeddings for entities:
```python
("ge", {
"m": { # metadata
"i": "core-id",
"m": [], # metadata triples
"u": "user",
"c": "collection"
},
"e": [ # entities array
{
"e": {"value": "entity", "is_uri": true},
"v": [[0.1, 0.2, 0.3]] # vectors
}
]
})
```
## Output Statistics
The command reports the number of messages exported:
```bash
Got: 150 triple, 75 GE messages.
```
Where:
- **triple**: Number of RDF triple message chunks exported
- **GE**: Number of graph embedding message chunks exported
## Error Handling
### Knowledge Core Not Found
```bash
Exception: Knowledge core 'invalid-core' not found
```
**Solution**: Check available cores with `tg-show-kg-cores` and verify the core ID.
### Permission Denied
```bash
Exception: Access denied to knowledge core
```
**Solution**: Verify user permissions for the specified knowledge core.
### File Permission Errors
```bash
Exception: Permission denied: output.msgpack
```
**Solution**: Check write permissions for the output directory and filename.
### Connection Errors
```bash
Exception: Connection refused
```
**Solution**: Check the API URL and ensure TrustGraph is running.
### Disk Space Errors
```bash
Exception: No space left on device
```
**Solution**: Free up disk space or use a different output location.
## File Management
### Backup Organization
```bash
# Create organized backup structure
mkdir -p backups/{daily,weekly,monthly}
# Daily backup
tg-get-kg-core --id "prod-core" -o "backups/daily/prod-$(date +%Y%m%d).msgpack"
# Weekly backup
tg-get-kg-core --id "prod-core" -o "backups/weekly/prod-week-$(date +%V).msgpack"
```
### Compression
```bash
# Export and compress for storage
tg-get-kg-core --id "large-core" -o large-core.msgpack
gzip large-core.msgpack
# Results in large-core.msgpack.gz
```
## File Verification
### Check File Size
```bash
# Export and verify
tg-get-kg-core --id "my-core" -o my-core.msgpack
ls -lh my-core.msgpack
# Typical sizes: small cores (KB-MB), large cores (MB-GB)
```
### Validate Export
```bash
# Test the exported file by importing to different ID
tg-put-kg-core --id "test-import" -i my-core.msgpack
tg-show-kg-cores | grep "test-import"
```
## Environment Variables
- `TRUSTGRAPH_URL`: Default API URL (automatically converted to WebSocket format)
## Related Commands
- [`tg-put-kg-core`](tg-put-kg-core.md) - Import knowledge core from MessagePack file
- [`tg-show-kg-cores`](tg-show-kg-cores.md) - List available knowledge cores
- [`tg-delete-kg-core`](tg-delete-kg-core.md) - Delete knowledge cores
- [`tg-dump-msgpack`](tg-dump-msgpack.md) - Examine MessagePack file contents
## API Integration
This command uses the [Knowledge API](../apis/api-knowledge.md) via WebSocket connection with `get-kg-core` operations to retrieve knowledge data.
## Use Cases
### Regular Backups
```bash
#!/bin/bash
# Daily backup script
cores=("production-core" "research-core" "customer-data")
backup_dir="backups/$(date +%Y%m%d)"
mkdir -p "$backup_dir"
for core in "${cores[@]}"; do
echo "Backing up $core..."
tg-get-kg-core --id "$core" -o "$backup_dir/$core.msgpack"
done
```
### Migration Between Environments
```bash
# Export from development
tg-get-kg-core --id "dev-knowledge" -o dev-export.msgpack
# Import to staging
tg-put-kg-core --id "staging-knowledge" -i dev-export.msgpack
```
### Knowledge Core Versioning
```bash
# Create versioned backups
version="v$(date +%Y%m%d)"
tg-get-kg-core --id "main-knowledge" -o "knowledge-$version.msgpack"
# Tag with git or other version control
git add "knowledge-$version.msgpack"
git commit -m "Knowledge core backup $version"
```
### Data Analysis
```bash
# Export for offline analysis
tg-get-kg-core --id "analytics-data" -o analytics.msgpack
# Process with custom tools
python analyze_knowledge.py analytics.msgpack
```
### Disaster Recovery
```bash
# Create comprehensive backup
cores=$(tg-show-kg-cores)
backup_date=$(date +%Y%m%d-%H%M%S)
backup_dir="disaster-recovery-$backup_date"
mkdir -p "$backup_dir"
for core in $cores; do
echo "Backing up $core..."
tg-get-kg-core --id "$core" -o "$backup_dir/$core.msgpack"
done
# Create checksum file
cd "$backup_dir"
sha256sum *.msgpack > checksums.sha256
```
## Automated Backup Strategies
### Cron Job Setup
```bash
# Add to crontab for daily backups at 2 AM
# 0 2 * * * /path/to/backup-script.sh
#!/bin/bash
# backup-script.sh
BACKUP_DIR="/backups/$(date +%Y%m%d)"
mkdir -p "$BACKUP_DIR"
# Backup all cores
tg-show-kg-cores | while read core; do
tg-get-kg-core --id "$core" -o "$BACKUP_DIR/$core.msgpack"
done
# Cleanup old backups (keep 30 days)
find /backups -type d -mtime +30 -exec rm -rf {} \;
```
### Incremental Backups
```bash
# Compare with previous backup
current_cores=$(tg-show-kg-cores | sort)
previous_cores=$(cat last-backup-cores.txt 2>/dev/null | sort)
# Only backup changed cores
comm -13 <(echo "$previous_cores") <(echo "$current_cores") | while read core; do
tg-get-kg-core --id "$core" -o "incremental/$core.msgpack"
done
echo "$current_cores" > last-backup-cores.txt
```
## Best Practices
1. **Regular Backups**: Schedule automated backups of important knowledge cores
2. **Organized Storage**: Use dated directories and consistent naming
3. **Verification**: Test backup files periodically by importing them
4. **Compression**: Compress large backup files to save storage
5. **Access Control**: Secure backup files with appropriate permissions
6. **Documentation**: Document what each knowledge core contains
7. **Retention Policy**: Implement backup retention policies
## Troubleshooting
### Large File Exports
```bash
# For very large knowledge cores
# Monitor progress and disk space
df -h . # Check available space
tg-get-kg-core --id "huge-core" -o huge-core.msgpack &
watch -n 5 'ls -lh huge-core.msgpack' # Monitor file growth
```
### Network Timeouts
```bash
# If export times out, try smaller cores or check network
# Split large cores if possible, or increase timeout settings
```
### Corrupted Exports
```bash
# Verify file integrity
file my-core.msgpack # Should show "data"
python -c "import msgpack; msgpack.unpack(open('my-core.msgpack', 'rb'))"
```

View file

@ -0,0 +1,494 @@
# tg-graph-to-turtle
Exports knowledge graph data to Turtle (TTL) format for backup, analysis, or migration.
## Synopsis
```bash
tg-graph-to-turtle [options]
```
## Description
The `tg-graph-to-turtle` command connects to TrustGraph's triple query service and exports all graph triples in Turtle format. This is useful for creating backups, analyzing graph structure, migrating data, or integrating with external RDF tools.
The command queries up to 10,000 triples and outputs them in standard Turtle format to stdout, while also saving to an `output.ttl` file.
## Options
### Optional Arguments
- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
- `-f, --flow-id ID`: Flow instance ID to use (default: `default`)
- `-U, --user USER`: User ID for data scope (default: `trustgraph`)
- `-C, --collection COLLECTION`: Collection to export (default: `default`)
## Examples
### Basic Export
```bash
tg-graph-to-turtle
```
### Export to File
```bash
tg-graph-to-turtle > knowledge-graph.ttl
```
### Export Specific Collection
```bash
tg-graph-to-turtle -C "research-data" > research-graph.ttl
```
### Export with Custom Flow
```bash
tg-graph-to-turtle -f "production-flow" -U "admin" > production-graph.ttl
```
## Output Format
The command generates Turtle format with proper RDF syntax:
```turtle
@prefix ns1: <http://example.org/> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
ns1:Person rdf:type rdfs:Class .
ns1:john rdf:type ns1:Person ;
ns1:name "John Doe" ;
ns1:age "30" .
ns1:jane rdf:type ns1:Person ;
ns1:name "Jane Smith" ;
ns1:department "Engineering" .
```
### Output Destinations
1. **stdout**: Standard output for piping or display
2. **output.ttl**: Automatically created file in current directory
## Use Cases
### Data Backup
```bash
# Create timestamped backups
timestamp=$(date +%Y%m%d_%H%M%S)
tg-graph-to-turtle > "backup_${timestamp}.ttl"
# Backup specific collections
collections=("research" "products" "customers")
for collection in "${collections[@]}"; do
tg-graph-to-turtle -C "$collection" > "backup_${collection}_${timestamp}.ttl"
done
```
### Data Migration
```bash
# Export from source environment
tg-graph-to-turtle -u "http://source:8088/" > source-data.ttl
# Import to target environment
tg-load-turtle -i "migration-$(date +%Y%m%d)" \
-u "ws://target:8088/" \
source-data.ttl
```
### Graph Analysis
```bash
# Export for analysis
tg-graph-to-turtle > analysis-data.ttl
# Analyze with external tools
rapper -i turtle -o ntriples analysis-data.ttl | wc -l # Count triples
grep -c "rdf:type" analysis-data.ttl # Count type assertions
```
### Integration with External Tools
```bash
# Export for Apache Jena
tg-graph-to-turtle > jena-input.ttl
tdb2.tdbloader --loc=tdb-database jena-input.ttl
# Export for Virtuoso
tg-graph-to-turtle > virtuoso-data.ttl
isql-v -U dba -P password < load-script.sql
```
## Advanced Usage
### Incremental Exports
```bash
# Export with timestamps for incremental backups
last_export_file="last_export_timestamp.txt"
current_time=$(date +%Y%m%d_%H%M%S)
if [ -f "$last_export_file" ]; then
last_export=$(cat "$last_export_file")
echo "Last export: $last_export"
fi
echo "Current export: $current_time"
tg-graph-to-turtle > "incremental_${current_time}.ttl"
echo "$current_time" > "$last_export_file"
```
### Multi-Collection Export
```bash
# Export all collections to separate files
export_all_collections() {
local output_dir="graph_exports_$(date +%Y%m%d)"
mkdir -p "$output_dir"
echo "Exporting all collections to $output_dir"
# Get list of collections (this would need to be implemented)
# For now, use known collections
collections=("default" "research" "products" "documents")
for collection in "${collections[@]}"; do
echo "Exporting collection: $collection"
tg-graph-to-turtle -C "$collection" > "$output_dir/${collection}.ttl"
# Verify export
if [ -s "$output_dir/${collection}.ttl" ]; then
triple_count=$(grep -c "\." "$output_dir/${collection}.ttl")
echo " Exported $triple_count triples"
else
echo " No data exported"
fi
done
}
export_all_collections
```
### Filtered Export
```bash
# Export specific types of triples
export_filtered() {
local filter_type="$1"
local output_file="$2"
echo "Exporting $filter_type triples to $output_file"
# Export all data first
tg-graph-to-turtle > temp_full_export.ttl
# Filter based on type
case "$filter_type" in
"classes")
grep "rdf:type.*Class" temp_full_export.ttl > "$output_file"
;;
"instances")
grep -v "rdf:type.*Class" temp_full_export.ttl > "$output_file"
;;
"properties")
grep "rdf:type.*Property" temp_full_export.ttl > "$output_file"
;;
*)
echo "Unknown filter type: $filter_type"
return 1
;;
esac
rm temp_full_export.ttl
}
# Usage
export_filtered "classes" "schema-classes.ttl"
export_filtered "instances" "instance-data.ttl"
```
### Compression and Packaging
```bash
# Export and compress
export_compressed() {
local collection="$1"
local timestamp=$(date +%Y%m%d_%H%M%S)
local filename="${collection}_${timestamp}"
echo "Exporting and compressing collection: $collection"
# Export to temporary file
tg-graph-to-turtle -C "$collection" > "${filename}.ttl"
# Compress
gzip "${filename}.ttl"
# Create metadata
cat > "${filename}.meta" << EOF
Collection: $collection
Export Date: $(date)
Compressed Size: $(stat -c%s "${filename}.ttl.gz") bytes
MD5: $(md5sum "${filename}.ttl.gz" | cut -d' ' -f1)
EOF
echo "Export complete: ${filename}.ttl.gz"
}
# Export multiple collections compressed
collections=("research" "products" "customers")
for collection in "${collections[@]}"; do
export_compressed "$collection"
done
```
### Validation and Quality Checks
```bash
# Export with validation
export_with_validation() {
local output_file="$1"
echo "Exporting with validation to $output_file"
# Export
tg-graph-to-turtle > "$output_file"
# Validate Turtle syntax
if rapper -q -i turtle "$output_file" > /dev/null 2>&1; then
echo "✓ Valid Turtle syntax"
else
echo "✗ Invalid Turtle syntax"
return 1
fi
# Count triples
triple_count=$(rapper -i turtle -c "$output_file" 2>/dev/null)
echo "Total triples: $triple_count"
# Check for common issues
if grep -q "^@prefix" "$output_file"; then
echo "✓ Prefixes found"
else
echo "⚠ No prefixes found"
fi
# Check for URIs with spaces (malformed)
malformed_uris=$(grep -c " " "$output_file" || echo "0")
if [ "$malformed_uris" -gt 0 ]; then
echo "⚠ Found $malformed_uris lines with spaces (potential malformed URIs)"
fi
}
# Validate export
export_with_validation "validated-export.ttl"
```
## Performance Optimization
### Streaming Export
```bash
# Handle large datasets with streaming
stream_export() {
local collection="$1"
local chunk_size="$2"
local output_prefix="$3"
echo "Streaming export of collection: $collection"
# Export to temporary file
tg-graph-to-turtle -C "$collection" > temp_export.ttl
# Split into chunks
split -l "$chunk_size" temp_export.ttl "${output_prefix}_"
# Add .ttl extension and validate each chunk
for chunk in ${output_prefix}_*; do
mv "$chunk" "$chunk.ttl"
# Validate chunk
if rapper -q -i turtle "$chunk.ttl" > /dev/null 2>&1; then
echo "✓ Valid chunk: $chunk.ttl"
else
echo "✗ Invalid chunk: $chunk.ttl"
fi
done
rm temp_export.ttl
}
# Stream large collection
stream_export "large-collection" 1000 "chunk"
```
### Parallel Processing
```bash
# Export multiple collections in parallel
parallel_export() {
local collections=("$@")
local timestamp=$(date +%Y%m%d_%H%M%S)
echo "Exporting ${#collections[@]} collections in parallel"
for collection in "${collections[@]}"; do
(
echo "Exporting $collection..."
tg-graph-to-turtle -C "$collection" > "${collection}_${timestamp}.ttl"
echo "✓ Completed: $collection"
) &
done
wait
echo "All exports completed"
}
# Export collections in parallel
parallel_export "research" "products" "customers" "documents"
```
## Integration Scripts
### Automated Backup System
```bash
#!/bin/bash
# automated-backup.sh
backup_dir="graph_backups"
retention_days=30
echo "Starting automated graph backup..."
# Create backup directory
mkdir -p "$backup_dir"
# Export with timestamp
timestamp=$(date +%Y%m%d_%H%M%S)
backup_file="$backup_dir/graph_backup_${timestamp}.ttl"
echo "Exporting to: $backup_file"
tg-graph-to-turtle > "$backup_file"
# Compress
gzip "$backup_file"
echo "Compressed: ${backup_file}.gz"
# Clean old backups
find "$backup_dir" -name "*.ttl.gz" -mtime +$retention_days -delete
echo "Cleaned backups older than $retention_days days"
# Verify backup
if [ -f "${backup_file}.gz" ]; then
size=$(stat -c%s "${backup_file}.gz")
echo "Backup completed: ${size} bytes"
else
echo "Backup failed!"
exit 1
fi
```
### Data Sync Script
```bash
#!/bin/bash
# sync-graphs.sh
source_url="$1"
target_url="$2"
collection="$3"
if [ -z "$source_url" ] || [ -z "$target_url" ] || [ -z "$collection" ]; then
echo "Usage: $0 <source-url> <target-url> <collection>"
exit 1
fi
echo "Syncing collection '$collection' from $source_url to $target_url"
# Export from source
temp_file="sync_temp_$(date +%s).ttl"
tg-graph-to-turtle -u "$source_url" -C "$collection" > "$temp_file"
# Validate export
if [ ! -s "$temp_file" ]; then
echo "No data exported from source"
exit 1
fi
# Load to target
doc_id="sync-$(date +%Y%m%d-%H%M%S)"
if tg-load-turtle -i "$doc_id" -u "$target_url" -C "$collection" "$temp_file"; then
echo "Sync completed successfully"
else
echo "Sync failed"
exit 1
fi
# Cleanup
rm "$temp_file"
```
## Error Handling
### Connection Issues
```bash
Exception: Connection refused
```
**Solution**: Check API URL and ensure TrustGraph is running.
### Flow Not Found
```bash
Exception: Flow instance not found
```
**Solution**: Verify flow ID with `tg-show-flows`.
### Permission Errors
```bash
Exception: Access denied
```
**Solution**: Check user permissions for the specified collection.
### Empty Output
```bash
# No triples exported
```
**Solution**: Verify collection contains data and user has access.
## Environment Variables
- `TRUSTGRAPH_URL`: Default API URL
## Related Commands
- [`tg-load-turtle`](tg-load-turtle.md) - Import Turtle files
- [`tg-triples-query`](tg-triples-query.md) - Query graph triples
- [`tg-show-flows`](tg-show-flows.md) - List available flows
- [`tg-get-kg-core`](tg-get-kg-core.md) - Export knowledge cores
## API Integration
This command uses the [Triples Query API](../apis/api-triples-query.md) to retrieve graph data and convert it to Turtle format.
## Best Practices
1. **Regular Backups**: Schedule regular exports for data protection
2. **Validation**: Always validate exported Turtle files
3. **Compression**: Compress large exports for storage efficiency
4. **Monitoring**: Track export sizes and success rates
5. **Documentation**: Document export procedures and retention policies
6. **Security**: Ensure sensitive data is properly protected in exports
7. **Version Control**: Consider versioning exported schemas
## Troubleshooting
### Large Dataset Issues
```bash
# Check query limits
grep -c "\." output.ttl # Count exported triples
# Default limit is 10,000 triples
# For larger datasets, consider using tg-get-kg-core
tg-get-kg-core -n "collection-name" > large-export.msgpack
```
### Malformed URIs
```bash
# Check for URIs with spaces
grep " " output.ttl | head -5
# Clean URIs if needed
sed 's/ /%20/g' output.ttl > cleaned-output.ttl
```
### Memory Issues
```bash
# Monitor memory usage during export
free -h
# Consider splitting exports for large datasets
```

View file

@ -0,0 +1,438 @@
# tg-invoke-document-rag
Invokes the DocumentRAG service to answer questions using document context and retrieval-augmented generation.
## Synopsis
```bash
tg-invoke-document-rag -q QUESTION [options]
```
## Description
The `tg-invoke-document-rag` command uses TrustGraph's DocumentRAG service to answer questions by retrieving relevant document context and generating responses using large language models. This implements a Retrieval-Augmented Generation (RAG) approach that grounds AI responses in your document corpus.
The service searches through indexed documents to find relevant context, then uses that context to generate accurate, source-backed answers to questions.
## Options
### Required Arguments
- `-q, --question QUESTION`: The question to answer
### Optional Arguments
- `-u, --url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
- `-f, --flow-id ID`: Flow instance ID to use (default: `default`)
- `-U, --user USER`: User ID for context isolation (default: `trustgraph`)
- `-C, --collection COLLECTION`: Document collection to search (default: `default`)
- `-d, --doc-limit LIMIT`: Maximum number of documents to retrieve (default: `10`)
## Examples
### Basic Question Answering
```bash
tg-invoke-document-rag -q "What is the company's return policy?"
```
### Question with Custom Parameters
```bash
tg-invoke-document-rag \
-q "How do I configure SSL certificates?" \
-f "production-docs" \
-U "admin" \
-C "technical-docs" \
-d 5
```
### Complex Technical Questions
```bash
tg-invoke-document-rag \
-q "What are the performance benchmarks for the new API endpoints?" \
-f "api-docs" \
-C "performance-reports"
```
### Multi-domain Questions
```bash
# Legal documents
tg-invoke-document-rag -q "What are the privacy policy requirements?" -C "legal-docs"
# Technical documentation
tg-invoke-document-rag -q "How do I troubleshoot connection timeouts?" -C "tech-docs"
# Marketing materials
tg-invoke-document-rag -q "What are our key product differentiators?" -C "marketing"
```
## Output Format
The command returns a structured response with:
```json
{
"question": "What is the company's return policy?",
"answer": "Based on the company policy documents, customers can return items within 30 days of purchase for a full refund. Items must be in original condition with receipt. Digital products are non-refundable except in cases of technical defects.",
"sources": [
{
"document": "customer-service-policy.pdf",
"relevance": 0.92,
"section": "Returns and Refunds"
},
{
"document": "terms-of-service.pdf",
"relevance": 0.85,
"section": "Customer Rights"
}
],
"confidence": 0.89
}
```
## Use Cases
### Customer Support
```bash
# Answer common customer questions
tg-invoke-document-rag -q "How do I reset my password?" -C "support-docs"
# Product information queries
tg-invoke-document-rag -q "What are the system requirements?" -C "product-specs"
# Troubleshooting assistance
tg-invoke-document-rag -q "Why is my upload failing?" -C "troubleshooting"
```
### Technical Documentation
```bash
# API documentation queries
tg-invoke-document-rag -q "How do I authenticate with the REST API?" -C "api-docs"
# Configuration questions
tg-invoke-document-rag -q "What are the required environment variables?" -C "config-docs"
# Architecture information
tg-invoke-document-rag -q "How does the caching system work?" -C "architecture"
```
### Research and Analysis
```bash
# Research queries
tg-invoke-document-rag -q "What are the latest industry trends?" -C "research-reports"
# Compliance questions
tg-invoke-document-rag -q "What are the GDPR requirements?" -C "compliance-docs"
# Best practices
tg-invoke-document-rag -q "What are the security best practices?" -C "security-guidelines"
```
### Interactive Q&A Sessions
```bash
# Batch questions for analysis
questions=(
"What is our market share?"
"How do we compare to competitors?"
"What are the growth projections?"
)
for question in "${questions[@]}"; do
echo "Question: $question"
tg-invoke-document-rag -q "$question" -C "business-reports"
echo "---"
done
```
## Document Context and Retrieval
### Document Limit Tuning
```bash
# Few documents for focused answers
tg-invoke-document-rag -q "What is the API rate limit?" -d 3
# Many documents for comprehensive analysis
tg-invoke-document-rag -q "What are all the security measures?" -d 20
```
### Collection-Specific Queries
```bash
# Target specific document collections
tg-invoke-document-rag -q "What is the deployment process?" -C "devops-docs"
tg-invoke-document-rag -q "What are the testing standards?" -C "qa-docs"
tg-invoke-document-rag -q "What is the coding style guide?" -C "dev-standards"
```
### User Context Isolation
```bash
# Department-specific contexts
tg-invoke-document-rag -q "What is the budget allocation?" -U "finance" -C "finance-docs"
tg-invoke-document-rag -q "What are the hiring requirements?" -U "hr" -C "hr-docs"
```
## Error Handling
### Question Required
```bash
Exception: Question is required
```
**Solution**: Provide a question with the `-q` option.
### Flow Not Found
```bash
Exception: Flow instance 'nonexistent-flow' not found
```
**Solution**: Verify the flow ID exists with `tg-show-flows`.
### Collection Not Found
```bash
Exception: Collection 'invalid-collection' not found
```
**Solution**: Check available collections with document library commands.
### No Documents Found
```bash
Exception: No relevant documents found for query
```
**Solution**: Verify documents are indexed and collection contains relevant content.
### API Connection Issues
```bash
Exception: Connection refused
```
**Solution**: Check API URL and ensure TrustGraph services are running.
## Advanced Usage
### Batch Processing
```bash
# Process questions from file
while IFS= read -r question; do
if [ -n "$question" ]; then
echo "Processing: $question"
tg-invoke-document-rag -q "$question" -C "knowledge-base" > "answer-$(date +%s).json"
fi
done < questions.txt
```
### Question Analysis Pipeline
```bash
#!/bin/bash
# analyze-questions.sh
questions_file="$1"
collection="$2"
if [ -z "$questions_file" ] || [ -z "$collection" ]; then
echo "Usage: $0 <questions-file> <collection>"
exit 1
fi
echo "Question Analysis Report - $(date)"
echo "Collection: $collection"
echo "=================================="
question_num=1
while IFS= read -r question; do
if [ -n "$question" ]; then
echo -e "\n$question_num. $question"
echo "$(printf '=%.0s' {1..50})"
# Get answer
answer=$(tg-invoke-document-rag -q "$question" -C "$collection" 2>/dev/null)
if [ $? -eq 0 ]; then
echo "$answer" | jq -r '.answer' 2>/dev/null || echo "$answer"
# Extract sources if available
sources=$(echo "$answer" | jq -r '.sources[]?.document' 2>/dev/null)
if [ -n "$sources" ]; then
echo -e "\nSources:"
echo "$sources" | sed 's/^/ - /'
fi
else
echo "ERROR: Could not get answer"
fi
question_num=$((question_num + 1))
fi
done < "$questions_file"
```
### Quality Assessment
```bash
# Assess answer quality with multiple document limits
question="What are the security protocols?"
collection="security-docs"
echo "Answer Quality Assessment"
echo "Question: $question"
echo "========================"
for limit in 3 5 10 15 20; do
echo -e "\nDocument limit: $limit"
echo "$(printf '-%.0s' {1..30})"
answer=$(tg-invoke-document-rag -q "$question" -C "$collection" -d $limit 2>/dev/null)
if [ $? -eq 0 ]; then
# Get answer length and source count
answer_length=$(echo "$answer" | jq -r '.answer' 2>/dev/null | wc -c)
source_count=$(echo "$answer" | jq -r '.sources | length' 2>/dev/null)
confidence=$(echo "$answer" | jq -r '.confidence' 2>/dev/null)
echo "Answer length: $answer_length characters"
echo "Source count: $source_count"
echo "Confidence: $confidence"
else
echo "ERROR: Failed to get answer"
fi
done
```
### Interactive Q&A Interface
```bash
#!/bin/bash
# interactive-rag.sh
collection="${1:-default}"
flow_id="${2:-default}"
echo "Interactive Document RAG Interface"
echo "Collection: $collection"
echo "Flow ID: $flow_id"
echo "Type 'quit' to exit"
echo "=================================="
while true; do
echo -n "Question: "
read -r question
if [ "$question" = "quit" ]; then
break
fi
if [ -n "$question" ]; then
echo "Thinking..."
answer=$(tg-invoke-document-rag -q "$question" -C "$collection" -f "$flow_id" 2>/dev/null)
if [ $? -eq 0 ]; then
echo "Answer:"
echo "$answer" | jq -r '.answer' 2>/dev/null || echo "$answer"
# Show sources if available
sources=$(echo "$answer" | jq -r '.sources[]?.document' 2>/dev/null)
if [ -n "$sources" ]; then
echo -e "\nSources:"
echo "$sources" | sed 's/^/ - /'
fi
else
echo "Sorry, I couldn't answer that question."
fi
echo -e "\n$(printf '=%.0s' {1..50})"
fi
done
echo "Goodbye!"
```
## Performance Optimization
### Document Limit Optimization
```bash
# Test different document limits for performance
question="What is the system architecture?"
collection="tech-docs"
for limit in 3 5 10 15 20; do
echo "Testing document limit: $limit"
start_time=$(date +%s%N)
tg-invoke-document-rag -q "$question" -C "$collection" -d $limit > /dev/null 2>&1
end_time=$(date +%s%N)
duration=$(( (end_time - start_time) / 1000000 )) # Convert to milliseconds
echo " Duration: ${duration}ms"
done
```
### Caching Strategy
```bash
# Cache frequently asked questions
cache_dir="rag-cache"
mkdir -p "$cache_dir"
ask_question() {
local question="$1"
local collection="$2"
local cache_key=$(echo "$question-$collection" | md5sum | cut -d' ' -f1)
local cache_file="$cache_dir/$cache_key.json"
if [ -f "$cache_file" ]; then
echo "Cache hit for: $question"
cat "$cache_file"
else
echo "Cache miss, querying: $question"
tg-invoke-document-rag -q "$question" -C "$collection" | tee "$cache_file"
fi
}
# Use cached queries
ask_question "What is the API documentation?" "tech-docs"
ask_question "What are the system requirements?" "spec-docs"
```
## Environment Variables
- `TRUSTGRAPH_URL`: Default API URL
## Related Commands
- [`tg-load-pdf`](tg-load-pdf.md) - Load PDF documents for RAG
- [`tg-show-library-documents`](tg-show-library-documents.md) - List available documents
- [`tg-invoke-prompt`](tg-invoke-prompt.md) - Direct prompt invocation without RAG
- [`tg-start-flow`](tg-start-flow.md) - Start flows for document processing
- [`tg-show-flows`](tg-show-flows.md) - List active flow instances
## API Integration
This command uses the [DocumentRAG API](../apis/api-document-rag.md) to perform retrieval-augmented generation using the document corpus.
## Best Practices
1. **Question Formulation**: Use specific, well-formed questions for better results
2. **Collection Organization**: Organize documents into logical collections
3. **Document Limits**: Balance accuracy with performance using appropriate document limits
4. **User Context**: Use user isolation for sensitive or department-specific queries
5. **Source Verification**: Always check source documents for critical information
6. **Caching**: Implement caching for frequently asked questions
7. **Quality Assessment**: Regularly evaluate answer quality and adjust parameters
## Troubleshooting
### Poor Answer Quality
```bash
# Try different document limits
tg-invoke-document-rag -q "your question" -d 5 # Fewer documents
tg-invoke-document-rag -q "your question" -d 15 # More documents
# Check document collection
tg-show-library-documents -C "your-collection"
```
### Slow Response Times
```bash
# Reduce document limit
tg-invoke-document-rag -q "your question" -d 3
# Check flow performance
tg-show-flows | grep "document-rag"
```
### Missing Context
```bash
# Verify documents are indexed
tg-show-library-documents -C "your-collection"
# Check if collection exists
tg-show-library-documents | grep "your-collection"
```

View file

@ -0,0 +1,430 @@
# tg-invoke-prompt
Invokes the LLM prompt service using predefined prompt templates with variable substitution.
## Synopsis
```bash
tg-invoke-prompt [options] template-id [variable=value ...]
```
## Description
The `tg-invoke-prompt` command invokes TrustGraph's LLM prompt service using predefined prompt templates. Templates contain placeholder variables in the format `{{variable}}` that are replaced with values provided on the command line.
This provides a structured way to interact with language models using consistent, reusable prompt templates for specific tasks like question answering, text extraction, analysis, and more.
## Options
### Required Arguments
- `template-id`: Prompt template identifier (e.g., `question`, `extract-definitions`, `summarize`)
### Optional Arguments
- `-u, --url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
- `-f, --flow-id ID`: Flow instance ID to use (default: `default`)
- `variable=value`: Template variable assignments (can be specified multiple times)
## Examples
### Basic Question Answering
```bash
tg-invoke-prompt question text="What is artificial intelligence?" context="AI research field"
```
### Extract Definitions
```bash
tg-invoke-prompt extract-definitions \
document="Machine learning is a subset of artificial intelligence..." \
terms="machine learning,neural networks"
```
### Text Summarization
```bash
tg-invoke-prompt summarize \
text="$(cat large-document.txt)" \
max_length="200" \
style="technical"
```
### Custom Flow and Variables
```bash
tg-invoke-prompt analysis \
-f "research-flow" \
data="$(cat research-data.json)" \
focus="trends" \
output_format="markdown"
```
## Variable Substitution
Templates use `{{variable}}` placeholders that are replaced with command-line values:
### Simple Variables
```bash
tg-invoke-prompt greeting name="Alice" time="morning"
# Template: "Good {{time}}, {{name}}!"
# Result: "Good morning, Alice!"
```
### Complex Variables
```bash
tg-invoke-prompt analyze \
dataset="$(cat data.csv)" \
columns="name,age,salary" \
analysis_type="statistical_summary"
```
### Multi-line Variables
```bash
tg-invoke-prompt review \
code="$(cat app.py)" \
checklist="security,performance,maintainability" \
severity="high"
```
## Common Template Types
### Question Answering
```bash
# Direct question
tg-invoke-prompt question \
text="What is the capital of France?" \
context="geography"
# Contextual question
tg-invoke-prompt question \
text="How does this work?" \
context="$(cat technical-manual.txt)"
```
### Text Processing
```bash
# Extract key information
tg-invoke-prompt extract-key-points \
document="$(cat meeting-notes.txt)" \
format="bullet_points"
# Text classification
tg-invoke-prompt classify \
text="Customer is very unhappy with service" \
categories="positive,negative,neutral"
```
### Code Analysis
```bash
# Code review
tg-invoke-prompt code-review \
code="$(cat script.py)" \
language="python" \
focus="security,performance"
# Bug analysis
tg-invoke-prompt debug \
code="$(cat buggy-code.js)" \
error="TypeError: Cannot read property 'length' of undefined"
```
### Data Analysis
```bash
# Data insights
tg-invoke-prompt data-analysis \
data="$(cat sales-data.json)" \
metrics="revenue,growth,trends" \
period="quarterly"
```
## Template Management
### List Available Templates
```bash
# Show available prompt templates
tg-show-prompts
```
### Create Custom Templates
```bash
# Define a new template
tg-set-prompt analysis-template \
"Analyze the following {{data_type}}: {{data}}. Focus on {{focus_areas}}. Output format: {{format}}"
```
### Template Variables
Common template variables:
- `{{text}}` - Input text to process
- `{{context}}` - Additional context information
- `{{format}}` - Output format specification
- `{{language}}` - Programming language for code analysis
- `{{style}}` - Writing or analysis style
- `{{length}}` - Length constraints for output
## Output Formats
### String Response
```bash
tg-invoke-prompt summarize text="Long document..." max_length="100"
# Output: "This document discusses..."
```
### JSON Response
```bash
tg-invoke-prompt extract-structured data="Name: John, Age: 30, City: NYC"
# Output:
# {
# "name": "John",
# "age": 30,
# "city": "NYC"
# }
```
## Error Handling
### Missing Template
```bash
Exception: Template 'nonexistent-template' not found
```
**Solution**: Check available templates with `tg-show-prompts`.
### Missing Variables
```bash
Exception: Template variable 'required_var' not provided
```
**Solution**: Provide all required variables as `variable=value` arguments.
### Malformed Variables
```bash
Exception: Malformed variable: invalid-format
```
**Solution**: Use `variable=value` format for all variable assignments.
### Flow Not Found
```bash
Exception: Flow instance 'invalid-flow' not found
```
**Solution**: Verify flow ID exists with `tg-show-flows`.
## Advanced Usage
### File Input Processing
```bash
# Process multiple files
for file in *.txt; do
echo "Processing $file..."
tg-invoke-prompt summarize \
text="$(cat "$file")" \
filename="$file" \
max_length="150"
done
```
### Batch Processing
```bash
# Process data in batches
while IFS= read -r line; do
tg-invoke-prompt classify \
text="$line" \
categories="spam,ham,promotional" \
confidence_threshold="0.8"
done < input-data.txt
```
### Pipeline Processing
```bash
# Chain multiple prompts
initial_analysis=$(tg-invoke-prompt analyze data="$(cat raw-data.json)")
summary=$(tg-invoke-prompt summarize text="$initial_analysis" style="executive")
echo "$summary"
```
### Interactive Processing
```bash
#!/bin/bash
# interactive-prompt.sh
template="$1"
if [ -z "$template" ]; then
echo "Usage: $0 <template-id>"
exit 1
fi
echo "Interactive prompt using template: $template"
echo "Enter variables (var=value), empty line to execute:"
variables=()
while true; do
read -p "> " input
if [ -z "$input" ]; then
break
fi
variables+=("$input")
done
echo "Executing prompt..."
tg-invoke-prompt "$template" "${variables[@]}"
```
### Configuration-Driven Processing
```bash
# Use configuration file for prompts
config_file="prompt-config.json"
template=$(jq -r '.template' "$config_file")
variables=$(jq -r '.variables | to_entries[] | "\(.key)=\(.value)"' "$config_file")
tg-invoke-prompt "$template" $variables
```
## Performance Optimization
### Caching Results
```bash
# Cache prompt results
cache_dir="prompt-cache"
mkdir -p "$cache_dir"
invoke_with_cache() {
local template="$1"
shift
local args="$@"
local cache_key=$(echo "$template-$args" | md5sum | cut -d' ' -f1)
local cache_file="$cache_dir/$cache_key.txt"
if [ -f "$cache_file" ]; then
echo "Cache hit"
cat "$cache_file"
else
echo "Cache miss, invoking prompt..."
tg-invoke-prompt "$template" "$@" | tee "$cache_file"
fi
}
```
### Parallel Processing
```bash
# Process multiple items in parallel
input_files=(file1.txt file2.txt file3.txt)
for file in "${input_files[@]}"; do
(
echo "Processing $file..."
tg-invoke-prompt analyze \
text="$(cat "$file")" \
filename="$file" > "result-$file.json"
) &
done
wait
```
## Use Cases
### Document Processing
```bash
# Extract metadata from documents
tg-invoke-prompt extract-metadata \
document="$(cat document.pdf)" \
fields="title,author,date,keywords"
# Generate document summaries
tg-invoke-prompt summarize \
text="$(cat report.txt)" \
audience="executives" \
key_points="5"
```
### Code Analysis
```bash
# Security analysis
tg-invoke-prompt security-review \
code="$(cat webapp.py)" \
framework="flask" \
focus="injection,authentication"
# Performance optimization suggestions
tg-invoke-prompt optimize \
code="$(cat slow-function.js)" \
language="javascript" \
target="performance"
```
### Data Analysis
```bash
# Generate insights from data
tg-invoke-prompt insights \
data="$(cat metrics.json)" \
timeframe="monthly" \
focus="trends,anomalies"
# Create data visualizations
tg-invoke-prompt visualize \
data="$(cat sales-data.csv)" \
chart_type="line" \
metrics="revenue,growth"
```
### Content Generation
```bash
# Generate marketing copy
tg-invoke-prompt marketing \
product="AI Assistant" \
audience="developers" \
tone="professional,friendly"
# Create technical documentation
tg-invoke-prompt document \
code="$(cat api.py)" \
format="markdown" \
sections="overview,examples,parameters"
```
## Environment Variables
- `TRUSTGRAPH_URL`: Default API URL
## Related Commands
- [`tg-show-prompts`](tg-show-prompts.md) - List available prompt templates
- [`tg-set-prompt`](tg-set-prompt.md) - Create/update prompt templates
- [`tg-invoke-document-rag`](tg-invoke-document-rag.md) - Document-based question answering
- [`tg-show-flows`](tg-show-flows.md) - List available flow instances
## API Integration
This command uses the prompt service API to process templates and generate responses using configured language models.
## Best Practices
1. **Template Reuse**: Create reusable templates for common tasks
2. **Variable Validation**: Validate required variables before execution
3. **Error Handling**: Implement proper error handling for production use
4. **Caching**: Cache results for repeated operations
5. **Documentation**: Document custom templates and their expected variables
6. **Security**: Avoid embedding sensitive data in templates
7. **Performance**: Use appropriate flow instances for different workloads
## Troubleshooting
### Template Not Found
```bash
# Check available templates
tg-show-prompts
# Verify template name spelling
tg-show-prompts | grep "template-name"
```
### Variable Errors
```bash
# Check template definition for required variables
tg-show-prompts | grep -A 10 "template-name"
# Validate variable format
echo "variable=value" | grep "="
```
### Flow Issues
```bash
# Check flow status
tg-show-flows | grep "flow-id"
# Verify flow has prompt service
tg-get-flow-class -n "flow-class" | jq '.interfaces.prompt'
```

480
docs/cli/tg-load-pdf.md Normal file
View file

@ -0,0 +1,480 @@
# tg-load-pdf
Loads PDF documents into TrustGraph for processing and analysis.
## Synopsis
```bash
tg-load-pdf [options] file1.pdf [file2.pdf ...]
```
## Description
The `tg-load-pdf` command loads PDF documents into TrustGraph by directing them to the PDF decoder service. The command extracts content, generates document metadata, and makes the documents available for processing by other TrustGraph services.
Each PDF is assigned a unique identifier based on its content hash, and comprehensive metadata can be attached including copyright information, publication details, and keywords.
**Note**: Consider using `tg-add-library-document` followed by `tg-start-library-processing` for more comprehensive document management.
## Options
### Required Arguments
- `files`: One or more PDF files to load
### Optional Arguments
- `-u, --url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
- `-f, --flow-id ID`: Flow instance ID to use (default: `default`)
- `-U, --user USER`: User ID for document ownership (default: `trustgraph`)
- `-C, --collection COLLECTION`: Collection to assign document (default: `default`)
### Document Metadata
- `--name NAME`: Document name/title
- `--description DESCRIPTION`: Document description
- `--identifier ID`: Custom document identifier
- `--document-url URL`: Source URL for the document
- `--keyword KEYWORD`: Document keywords (can be specified multiple times)
### Copyright Information
- `--copyright-notice NOTICE`: Copyright notice text
- `--copyright-holder HOLDER`: Copyright holder name
- `--copyright-year YEAR`: Copyright year
- `--license LICENSE`: Copyright license
### Publication Details
- `--publication-organization ORG`: Publishing organization
- `--publication-description DESC`: Publication description
- `--publication-date DATE`: Publication date
## Examples
### Basic PDF Loading
```bash
tg-load-pdf document.pdf
```
### Multiple Files
```bash
tg-load-pdf report1.pdf report2.pdf manual.pdf
```
### With Basic Metadata
```bash
tg-load-pdf \
--name "Technical Manual" \
--description "System administration guide" \
--keyword "technical" --keyword "manual" \
technical-manual.pdf
```
### Complete Metadata
```bash
tg-load-pdf \
--name "Annual Report 2023" \
--description "Company annual financial report" \
--copyright-holder "Acme Corporation" \
--copyright-year "2023" \
--license "All Rights Reserved" \
--publication-organization "Acme Corporation" \
--publication-date "2023-12-31" \
--keyword "financial" --keyword "annual" --keyword "report" \
annual-report-2023.pdf
```
### Custom Flow and Collection
```bash
tg-load-pdf \
-f "document-processing-flow" \
-U "finance-team" \
-C "financial-documents" \
--name "Budget Analysis" \
budget-2024.pdf
```
## Document Processing
### Content Extraction
The PDF loader:
1. Calculates SHA256 hash for unique document ID
2. Extracts text content from PDF
3. Preserves document structure and formatting metadata
4. Generates searchable text index
### Metadata Generation
Document metadata includes:
- **Document ID**: SHA256 hash-based unique identifier
- **Content Hash**: For duplicate detection
- **File Information**: Size, format, creation date
- **Custom Metadata**: User-provided attributes
### Integration with Processing Pipeline
```bash
# Load PDF and start processing
tg-load-pdf research-paper.pdf --name "AI Research Paper"
# Check processing status
tg-show-flows | grep "document-processing"
# Query loaded content
tg-invoke-document-rag -q "What is the main conclusion?" -C "default"
```
## Error Handling
### File Not Found
```bash
Exception: [Errno 2] No such file or directory: 'missing.pdf'
```
**Solution**: Verify file path and ensure PDF exists.
### Invalid PDF Format
```bash
Exception: PDF parsing failed: Invalid PDF structure
```
**Solution**: Verify PDF is not corrupted and is a valid PDF file.
### Permission Errors
```bash
Exception: [Errno 13] Permission denied: 'protected.pdf'
```
**Solution**: Check file permissions and ensure read access.
### Flow Not Found
```bash
Exception: Flow instance 'invalid-flow' not found
```
**Solution**: Verify flow ID exists with `tg-show-flows`.
### API Connection Issues
```bash
Exception: Connection refused
```
**Solution**: Check API URL and ensure TrustGraph is running.
## Advanced Usage
### Batch Processing
```bash
# Process all PDFs in directory
for pdf in *.pdf; do
echo "Loading $pdf..."
tg-load-pdf \
--name "$(basename "$pdf" .pdf)" \
--collection "research-papers" \
"$pdf"
done
```
### Organized Loading
```bash
# Load with structured metadata
categories=("technical" "financial" "legal")
for category in "${categories[@]}"; do
for pdf in "$category"/*.pdf; do
if [ -f "$pdf" ]; then
tg-load-pdf \
--collection "$category-documents" \
--keyword "$category" \
--name "$(basename "$pdf" .pdf)" \
"$pdf"
fi
done
done
```
### CSV-Driven Loading
```bash
# Load PDFs with metadata from CSV
# Format: filename,title,description,keywords
while IFS=',' read -r filename title description keywords; do
if [ -f "$filename" ]; then
echo "Loading $filename..."
# Convert comma-separated keywords to multiple --keyword args
keyword_args=""
IFS='|' read -ra KEYWORDS <<< "$keywords"
for kw in "${KEYWORDS[@]}"; do
keyword_args="$keyword_args --keyword \"$kw\""
done
eval "tg-load-pdf \
--name \"$title\" \
--description \"$description\" \
$keyword_args \
\"$filename\""
fi
done < documents.csv
```
### Publication Processing
```bash
# Load academic papers with publication details
load_academic_paper() {
local file="$1"
local title="$2"
local authors="$3"
local journal="$4"
local year="$5"
tg-load-pdf \
--name "$title" \
--description "Academic paper: $title" \
--copyright-holder "$authors" \
--copyright-year "$year" \
--publication-organization "$journal" \
--publication-date "$year-01-01" \
--keyword "academic" --keyword "research" \
"$file"
}
# Usage
load_academic_paper "ai-paper.pdf" "AI in Healthcare" "Smith et al." "AI Journal" "2023"
```
## Monitoring and Validation
### Load Status Checking
```bash
# Check document loading progress
check_load_status() {
local file="$1"
local expected_name="$2"
echo "Checking load status for: $file"
# Check if document appears in library
if tg-show-library-documents | grep -q "$expected_name"; then
echo "✓ Document loaded successfully"
else
echo "✗ Document not found in library"
return 1
fi
}
# Monitor batch loading
for pdf in *.pdf; do
name=$(basename "$pdf" .pdf)
check_load_status "$pdf" "$name"
done
```
### Content Verification
```bash
# Verify PDF content is accessible
verify_pdf_content() {
local pdf_name="$1"
local test_query="$2"
echo "Verifying content for: $pdf_name"
# Try to query the document
result=$(tg-invoke-document-rag -q "$test_query" -C "default" 2>/dev/null)
if [ $? -eq 0 ] && [ -n "$result" ]; then
echo "✓ Content accessible via RAG"
else
echo "✗ Content not accessible"
return 1
fi
}
# Verify loaded documents
verify_pdf_content "Technical Manual" "What is the installation process?"
```
## Performance Optimization
### Parallel Loading
```bash
# Load multiple PDFs in parallel
pdf_files=(document1.pdf document2.pdf document3.pdf)
for pdf in "${pdf_files[@]}"; do
(
echo "Loading $pdf in background..."
tg-load-pdf \
--name "$(basename "$pdf" .pdf)" \
--collection "batch-$(date +%Y%m%d)" \
"$pdf"
) &
done
wait
echo "All PDFs loaded"
```
### Size-Based Processing
```bash
# Process files based on size
for pdf in *.pdf; do
size=$(stat -c%s "$pdf")
if [ $size -lt 10485760 ]; then # < 10MB
echo "Processing small file: $pdf"
tg-load-pdf --collection "small-docs" "$pdf"
else
echo "Processing large file: $pdf"
tg-load-pdf --collection "large-docs" "$pdf"
fi
done
```
## Document Organization
### Collection Management
```bash
# Organize by document type
organize_by_type() {
local pdf="$1"
local filename=$(basename "$pdf" .pdf)
case "$filename" in
*manual*|*guide*) collection="manuals" ;;
*report*|*analysis*) collection="reports" ;;
*spec*|*specification*) collection="specifications" ;;
*legal*|*contract*) collection="legal" ;;
*) collection="general" ;;
esac
tg-load-pdf \
--collection "$collection" \
--name "$filename" \
"$pdf"
}
# Process all PDFs
for pdf in *.pdf; do
organize_by_type "$pdf"
done
```
### Metadata Standardization
```bash
# Apply consistent metadata standards
standardize_metadata() {
local pdf="$1"
local dept="$2"
local year="$3"
local name=$(basename "$pdf" .pdf)
local collection="$dept-$(date +%Y)"
tg-load-pdf \
--name "$name" \
--description "$dept document from $year" \
--copyright-holder "Company Name" \
--copyright-year "$year" \
--collection "$collection" \
--keyword "$dept" --keyword "$year" \
"$pdf"
}
# Usage
standardize_metadata "finance-report.pdf" "finance" "2023"
```
## Integration with Other Services
### Library Integration
```bash
# Alternative approach using library services
load_via_library() {
local pdf="$1"
local name="$2"
# Add to library first
tg-add-library-document \
--name "$name" \
--file "$pdf" \
--collection "documents"
# Start processing
tg-start-library-processing \
--collection "documents"
}
```
### Workflow Integration
```bash
# Complete document workflow
process_document_workflow() {
local pdf="$1"
local name="$2"
echo "Starting document workflow for: $name"
# 1. Load PDF
tg-load-pdf --name "$name" "$pdf"
# 2. Wait for processing
sleep 5
# 3. Verify availability
if tg-show-library-documents | grep -q "$name"; then
echo "Document available in library"
# 4. Test RAG functionality
tg-invoke-document-rag -q "What is this document about?"
# 5. Extract key information
tg-invoke-prompt extract-key-points \
text="Document: $name" \
format="bullet_points"
else
echo "Document processing failed"
fi
}
```
## Environment Variables
- `TRUSTGRAPH_URL`: Default API URL
## Related Commands
- [`tg-add-library-document`](tg-add-library-document.md) - Add documents to library
- [`tg-start-library-processing`](tg-start-library-processing.md) - Process library documents
- [`tg-show-library-documents`](tg-show-library-documents.md) - List library documents
- [`tg-invoke-document-rag`](tg-invoke-document-rag.md) - Query document content
- [`tg-show-flows`](tg-show-flows.md) - Monitor processing flows
## API Integration
This command uses the document loading API to process PDF files and make them available for text extraction, search, and analysis.
## Best Practices
1. **Metadata Completeness**: Provide comprehensive metadata for better organization
2. **Collection Organization**: Use logical collections for document categorization
3. **Error Handling**: Implement robust error handling for batch operations
4. **Performance**: Consider file sizes and processing capacity
5. **Monitoring**: Verify successful loading and processing
6. **Security**: Ensure sensitive documents are properly protected
7. **Backup**: Maintain backups of source PDFs
## Troubleshooting
### PDF Processing Issues
```bash
# Check PDF validity
file document.pdf
pdfinfo document.pdf
# Try alternative PDF processors
qpdf --check document.pdf
```
### Memory Issues
```bash
# For large PDFs, monitor memory usage
free -h
# Consider processing large files separately
```
### Content Extraction Problems
```bash
# Verify PDF contains extractable text
pdftotext document.pdf test-output.txt
cat test-output.txt | head -20
```

505
docs/cli/tg-load-turtle.md Normal file
View file

@ -0,0 +1,505 @@
# tg-load-turtle
Loads RDF triples from Turtle files into the TrustGraph knowledge graph.
## Synopsis
```bash
tg-load-turtle -i DOCUMENT_ID [options] file1.ttl [file2.ttl ...]
```
## Description
The `tg-load-turtle` command loads RDF triples from Turtle (TTL) format files into TrustGraph's knowledge graph. It parses Turtle files, converts them to TrustGraph's internal triple format, and imports them using WebSocket connections for efficient batch processing.
The command supports retry logic and automatic reconnection to handle network interruptions during large data imports.
## Options
### Required Arguments
- `-i, --document-id ID`: Document ID to associate with the triples
- `files`: One or more Turtle files to load
### Optional Arguments
- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `ws://localhost:8088/`)
- `-f, --flow-id ID`: Flow instance ID to use (default: `default`)
- `-U, --user USER`: User ID for triple ownership (default: `trustgraph`)
- `-C, --collection COLLECTION`: Collection to assign triples (default: `default`)
## Examples
### Basic Turtle Loading
```bash
tg-load-turtle -i "doc123" knowledge-base.ttl
```
### Multiple Files
```bash
tg-load-turtle -i "ontology-v1" \
schema.ttl \
instances.ttl \
relationships.ttl
```
### Custom Flow and Collection
```bash
tg-load-turtle \
-i "research-data" \
-f "knowledge-import-flow" \
-U "research-team" \
-C "research-kg" \
research-triples.ttl
```
### Load with Custom API URL
```bash
tg-load-turtle \
-i "production-data" \
-u "ws://production:8088/" \
production-ontology.ttl
```
## Turtle Format Support
### Basic Triples
```turtle
@prefix ex: <http://example.org/> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
ex:Person rdf:type rdfs:Class .
ex:john rdf:type ex:Person .
ex:john ex:name "John Doe" .
ex:john ex:age "30"^^xsd:integer .
```
### Complex Structures
```turtle
@prefix org: <http://example.org/organization/> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
org:TechCorp rdf:type foaf:Organization ;
foaf:name "Technology Corporation" ;
org:hasEmployee org:john, org:jane ;
org:foundedYear "2010"^^xsd:gYear .
org:john foaf:name "John Smith" ;
foaf:mbox <mailto:john@techcorp.com> ;
org:position "Software Engineer" .
```
### Ontology Loading
```turtle
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix dc: <http://purl.org/dc/elements/1.1/> .
<http://example.org/ontology> rdf:type owl:Ontology ;
dc:title "Example Ontology" ;
dc:creator "Knowledge Team" .
ex:Vehicle rdf:type owl:Class ;
rdfs:label "Vehicle" ;
rdfs:comment "A means of transportation" .
ex:Car rdfs:subClassOf ex:Vehicle .
ex:Truck rdfs:subClassOf ex:Vehicle .
```
## Data Processing
### Triple Conversion
The loader converts Turtle triples to TrustGraph format:
- **URIs**: Converted to URI references with `is_uri=true`
- **Literals**: Converted to literal values with `is_uri=false`
- **Datatypes**: Preserved in literal values
### Batch Processing
- Triples are sent individually via WebSocket
- Each triple includes document metadata
- Automatic retry on connection failures
- Progress tracking for large files
### Error Handling
- Invalid Turtle syntax causes parsing errors
- Network interruptions trigger automatic retry
- Malformed triples are skipped with warnings
## Use Cases
### Ontology Import
```bash
# Load domain ontology
tg-load-turtle -i "healthcare-ontology" \
-C "ontologies" \
healthcare-schema.ttl
# Load instance data
tg-load-turtle -i "patient-data" \
-C "healthcare-data" \
patient-records.ttl
```
### Knowledge Base Migration
```bash
# Migrate from external knowledge base
tg-load-turtle -i "migration-$(date +%Y%m%d)" \
-C "migrated-data" \
exported-knowledge.ttl
```
### Research Data Loading
```bash
# Load research datasets
datasets=("publications" "authors" "citations")
for dataset in "${datasets[@]}"; do
tg-load-turtle -i "research-$dataset" \
-C "research-data" \
"$dataset.ttl"
done
```
### Structured Data Import
```bash
# Load structured data from various sources
tg-load-turtle -i "products" -C "catalog" product-catalog.ttl
tg-load-turtle -i "customers" -C "crm" customer-data.ttl
tg-load-turtle -i "orders" -C "transactions" order-history.ttl
```
## Advanced Usage
### Batch Processing Multiple Files
```bash
# Process all Turtle files in directory
for ttl in *.ttl; do
doc_id=$(basename "$ttl" .ttl)
echo "Loading $ttl as document $doc_id..."
tg-load-turtle -i "$doc_id" \
-C "bulk-import-$(date +%Y%m%d)" \
"$ttl"
done
```
### Parallel Loading
```bash
# Load multiple files in parallel
ttl_files=(schema.ttl instances.ttl relationships.ttl)
for ttl in "${ttl_files[@]}"; do
(
doc_id=$(basename "$ttl" .ttl)
echo "Loading $ttl in background..."
tg-load-turtle -i "parallel-$doc_id" \
-C "parallel-import" \
"$ttl"
) &
done
wait
echo "All files loaded"
```
### Size-Based Processing
```bash
# Handle large files differently
for ttl in *.ttl; do
size=$(stat -c%s "$ttl")
doc_id=$(basename "$ttl" .ttl)
if [ $size -lt 10485760 ]; then # < 10MB
echo "Processing small file: $ttl"
tg-load-turtle -i "$doc_id" -C "small-files" "$ttl"
else
echo "Processing large file: $ttl"
# Use dedicated collection for large files
tg-load-turtle -i "$doc_id" -C "large-files" "$ttl"
fi
done
```
### Validation and Loading
```bash
# Validate before loading
validate_and_load() {
local ttl_file="$1"
local doc_id="$2"
echo "Validating $ttl_file..."
# Check Turtle syntax
if rapper -q -i turtle "$ttl_file" > /dev/null 2>&1; then
echo "✓ Valid Turtle syntax"
# Count triples
triple_count=$(rapper -i turtle -c "$ttl_file" 2>/dev/null)
echo " Triples: $triple_count"
# Load if valid
echo "Loading $ttl_file..."
tg-load-turtle -i "$doc_id" -C "validated-data" "$ttl_file"
else
echo "✗ Invalid Turtle syntax in $ttl_file"
return 1
fi
}
# Validate and load all files
for ttl in *.ttl; do
doc_id=$(basename "$ttl" .ttl)
validate_and_load "$ttl" "$doc_id"
done
```
## Error Handling
### Invalid Turtle Syntax
```bash
Exception: Turtle parsing failed
```
**Solution**: Validate Turtle syntax with tools like `rapper` or `rdflib`.
### Document ID Required
```bash
Exception: Document ID is required
```
**Solution**: Provide document ID with `-i` option.
### WebSocket Connection Issues
```bash
Exception: WebSocket connection failed
```
**Solution**: Check API URL and ensure TrustGraph WebSocket service is running.
### File Not Found
```bash
Exception: [Errno 2] No such file or directory
```
**Solution**: Verify file paths and ensure Turtle files exist.
### Flow Not Found
```bash
Exception: Flow instance not found
```
**Solution**: Verify flow ID with `tg-show-flows`.
## Monitoring and Verification
### Load Progress Tracking
```bash
# Monitor loading progress
monitor_load() {
local ttl_file="$1"
local doc_id="$2"
echo "Starting load: $ttl_file"
start_time=$(date +%s)
tg-load-turtle -i "$doc_id" -C "monitored" "$ttl_file"
end_time=$(date +%s)
duration=$((end_time - start_time))
echo "Load completed in ${duration}s"
# Verify data is accessible
if tg-triples-query -s "http://example.org/test" > /dev/null 2>&1; then
echo "✓ Data accessible via query"
else
echo "✗ Data not accessible"
fi
}
```
### Data Verification
```bash
# Verify loaded triples
verify_triples() {
local collection="$1"
local expected_count="$2"
echo "Verifying triples in collection: $collection"
# Query for triples
actual_count=$(tg-triples-query -C "$collection" | wc -l)
if [ "$actual_count" -ge "$expected_count" ]; then
echo "✓ Expected triples found ($actual_count >= $expected_count)"
else
echo "✗ Missing triples ($actual_count < $expected_count)"
return 1
fi
}
```
### Content Analysis
```bash
# Analyze loaded content
analyze_turtle_content() {
local ttl_file="$1"
echo "Analyzing content: $ttl_file"
# Extract prefixes
echo "Prefixes:"
grep "^@prefix" "$ttl_file" | head -5
# Count statements
statement_count=$(grep -c "\." "$ttl_file")
echo "Statements: $statement_count"
# Extract subjects
echo "Sample subjects:"
grep -o "^[^[:space:]]*" "$ttl_file" | grep -v "^@" | sort | uniq | head -5
}
```
## Performance Optimization
### Connection Pooling
```bash
# Reuse WebSocket connections for multiple files
load_batch_optimized() {
local collection="$1"
shift
local files=("$@")
echo "Loading ${#files[@]} files to collection: $collection"
# Process files in batches to reuse connections
for ((i=0; i<${#files[@]}; i+=5)); do
batch=("${files[@]:$i:5}")
echo "Processing batch $((i/5 + 1))..."
for ttl in "${batch[@]}"; do
doc_id=$(basename "$ttl" .ttl)
tg-load-turtle -i "$doc_id" -C "$collection" "$ttl" &
done
wait
done
}
```
### Memory Management
```bash
# Handle large files with memory monitoring
load_with_memory_check() {
local ttl_file="$1"
local doc_id="$2"
# Check available memory
available=$(free -m | awk 'NR==2{print $7}')
if [ "$available" -lt 1000 ]; then
echo "Warning: Low memory ($available MB). Consider splitting file."
fi
# Monitor memory during load
tg-load-turtle -i "$doc_id" -C "memory-monitored" "$ttl_file" &
load_pid=$!
while kill -0 $load_pid 2>/dev/null; do
memory_usage=$(ps -p $load_pid -o rss= | awk '{print $1/1024}')
echo "Memory usage: ${memory_usage}MB"
sleep 5
done
}
```
## Data Preparation
### Turtle File Preparation
```bash
# Clean and prepare Turtle files
prepare_turtle() {
local input_file="$1"
local output_file="$2"
echo "Preparing $input_file -> $output_file"
# Remove comments and empty lines
sed '/^#/d; /^$/d' "$input_file" > "$output_file"
# Validate output
if rapper -q -i turtle "$output_file" > /dev/null 2>&1; then
echo "✓ Prepared file is valid"
else
echo "✗ Prepared file is invalid"
return 1
fi
}
```
### Data Splitting
```bash
# Split large Turtle files
split_turtle() {
local input_file="$1"
local lines_per_file="$2"
echo "Splitting $input_file into chunks of $lines_per_file lines"
# Split file
split -l "$lines_per_file" "$input_file" "$(basename "$input_file" .ttl)_part_"
# Add .ttl extension to parts
for part in $(basename "$input_file" .ttl)_part_*; do
mv "$part" "$part.ttl"
done
}
```
## Environment Variables
- `TRUSTGRAPH_URL`: Default API URL (WebSocket format)
## Related Commands
- [`tg-triples-query`](tg-triples-query.md) - Query loaded triples
- [`tg-graph-to-turtle`](tg-graph-to-turtle.md) - Export graph to Turtle format
- [`tg-show-flows`](tg-show-flows.md) - Monitor processing flows
- [`tg-load-pdf`](tg-load-pdf.md) - Load document content
## API Integration
This command uses TrustGraph's WebSocket-based triple import API for efficient batch loading of RDF data.
## Best Practices
1. **Validation**: Always validate Turtle syntax before loading
2. **Document IDs**: Use meaningful, unique document identifiers
3. **Collections**: Organize triples into logical collections
4. **Error Handling**: Implement retry logic for network issues
5. **Performance**: Consider file sizes and system resources
6. **Monitoring**: Track loading progress and verify results
7. **Backup**: Maintain backups of source Turtle files
## Troubleshooting
### WebSocket Connection Issues
```bash
# Test WebSocket connectivity
wscat -c ws://localhost:8088/api/v1/flow/default/import/triples
# Check WebSocket service status
tg-show-flows | grep -i websocket
```
### Parsing Errors
```bash
# Validate Turtle syntax
rapper -i turtle -q file.ttl
# Check for common issues
grep -n "^[[:space:]]*@prefix" file.ttl # Check prefixes
grep -n "\.$" file.ttl | head -5 # Check statement terminators
```
### Memory Issues
```bash
# Monitor memory usage
free -h
ps aux | grep tg-load-turtle
# Split large files if needed
split -l 10000 large-file.ttl chunk_
```

View file

@ -0,0 +1,406 @@
# tg-put-flow-class
Uploads or updates a flow class definition in TrustGraph.
## Synopsis
```bash
tg-put-flow-class -n CLASS_NAME -c CONFIG_JSON [options]
```
## Description
The `tg-put-flow-class` command creates or updates a flow class definition in TrustGraph. Flow classes are templates that define processing pipeline configurations, service interfaces, and resource requirements. These classes are used by `tg-start-flow` to create running flow instances.
Flow classes define the structure and capabilities of processing flows, including which services are available and how they connect to Pulsar queues.
## Options
### Required Arguments
- `-n, --class-name CLASS_NAME`: Name for the flow class
- `-c, --config CONFIG_JSON`: Flow class configuration as raw JSON string
### Optional Arguments
- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
## Examples
### Basic Flow Class Creation
```bash
tg-put-flow-class \
-n "simple-processing" \
-c '{"description": "Simple text processing flow", "interfaces": {"text-completion": {"request": "non-persistent://tg/request/text-completion:simple", "response": "non-persistent://tg/response/text-completion:simple"}}}'
```
### Document Processing Flow Class
```bash
tg-put-flow-class \
-n "document-analysis" \
-c '{
"description": "Document analysis and RAG processing",
"interfaces": {
"document-rag": {
"request": "non-persistent://tg/request/document-rag:doc-analysis",
"response": "non-persistent://tg/response/document-rag:doc-analysis"
},
"text-load": "persistent://tg/flow/text-document-load:doc-analysis",
"document-load": "persistent://tg/flow/document-load:doc-analysis"
}
}'
```
### Loading from File
```bash
# Create configuration file
cat > research-flow.json << 'EOF'
{
"description": "Research analysis flow with multiple AI services",
"interfaces": {
"agent": {
"request": "non-persistent://tg/request/agent:research",
"response": "non-persistent://tg/response/agent:research"
},
"graph-rag": {
"request": "non-persistent://tg/request/graph-rag:research",
"response": "non-persistent://tg/response/graph-rag:research"
},
"document-rag": {
"request": "non-persistent://tg/request/document-rag:research",
"response": "non-persistent://tg/response/document-rag:research"
},
"embeddings": {
"request": "non-persistent://tg/request/embeddings:research",
"response": "non-persistent://tg/response/embeddings:research"
},
"text-load": "persistent://tg/flow/text-document-load:research",
"triples-store": "persistent://tg/flow/triples-store:research"
}
}
EOF
# Upload the flow class
tg-put-flow-class -n "research-analysis" -c "$(cat research-flow.json)"
```
### Update Existing Flow Class
```bash
# Modify existing flow class by adding new service
tg-put-flow-class \
-n "existing-flow" \
-c '{
"description": "Updated flow with new capabilities",
"interfaces": {
"text-completion": {
"request": "non-persistent://tg/request/text-completion:updated",
"response": "non-persistent://tg/response/text-completion:updated"
},
"prompt": {
"request": "non-persistent://tg/request/prompt:updated",
"response": "non-persistent://tg/response/prompt:updated"
}
}
}'
```
## Flow Class Configuration Format
### Required Fields
#### Description
```json
{
"description": "Human-readable description of the flow class"
}
```
#### Interfaces
```json
{
"interfaces": {
"service-name": "queue-definition-or-object"
}
}
```
### Interface Types
#### Request/Response Services
Services that accept requests and return responses:
```json
{
"service-name": {
"request": "pulsar-queue-url",
"response": "pulsar-queue-url"
}
}
```
Examples:
- `agent`
- `graph-rag`
- `document-rag`
- `text-completion`
- `prompt`
- `embeddings`
- `graph-embeddings`
- `triples`
#### Fire-and-Forget Services
Services that accept data without returning responses:
```json
{
"service-name": "pulsar-queue-url"
}
```
Examples:
- `text-load`
- `document-load`
- `triples-store`
- `graph-embeddings-store`
- `document-embeddings-store`
- `entity-contexts-load`
### Queue Naming Conventions
#### Request/Response Queues
```
non-persistent://tg/request/{service}:{flow-identifier}
non-persistent://tg/response/{service}:{flow-identifier}
```
#### Fire-and-Forget Queues
```
persistent://tg/flow/{service}:{flow-identifier}
```
## Complete Example
### Comprehensive Flow Class
```bash
tg-put-flow-class \
-n "full-processing-pipeline" \
-c '{
"description": "Complete document processing and analysis pipeline",
"interfaces": {
"agent": {
"request": "non-persistent://tg/request/agent:full-pipeline",
"response": "non-persistent://tg/response/agent:full-pipeline"
},
"graph-rag": {
"request": "non-persistent://tg/request/graph-rag:full-pipeline",
"response": "non-persistent://tg/response/graph-rag:full-pipeline"
},
"document-rag": {
"request": "non-persistent://tg/request/document-rag:full-pipeline",
"response": "non-persistent://tg/response/document-rag:full-pipeline"
},
"text-completion": {
"request": "non-persistent://tg/request/text-completion:full-pipeline",
"response": "non-persistent://tg/response/text-completion:full-pipeline"
},
"prompt": {
"request": "non-persistent://tg/request/prompt:full-pipeline",
"response": "non-persistent://tg/response/prompt:full-pipeline"
},
"embeddings": {
"request": "non-persistent://tg/request/embeddings:full-pipeline",
"response": "non-persistent://tg/response/embeddings:full-pipeline"
},
"graph-embeddings": {
"request": "non-persistent://tg/request/graph-embeddings:full-pipeline",
"response": "non-persistent://tg/response/graph-embeddings:full-pipeline"
},
"triples": {
"request": "non-persistent://tg/request/triples:full-pipeline",
"response": "non-persistent://tg/response/triples:full-pipeline"
},
"text-load": "persistent://tg/flow/text-document-load:full-pipeline",
"document-load": "persistent://tg/flow/document-load:full-pipeline",
"triples-store": "persistent://tg/flow/triples-store:full-pipeline",
"graph-embeddings-store": "persistent://tg/flow/graph-embeddings-store:full-pipeline",
"document-embeddings-store": "persistent://tg/flow/document-embeddings-store:full-pipeline",
"entity-contexts-load": "persistent://tg/flow/entity-contexts-load:full-pipeline"
}
}'
```
## Output
Successful upload typically produces no output:
```bash
# Upload flow class (no output expected)
tg-put-flow-class -n "my-flow" -c '{"description": "test", "interfaces": {}}'
# Verify upload
tg-show-flow-classes | grep "my-flow"
```
## Error Handling
### Invalid JSON Format
```bash
Exception: Invalid JSON in config parameter
```
**Solution**: Validate JSON syntax using tools like `jq` or online JSON validators.
### Missing Required Fields
```bash
Exception: Missing required field 'description'
```
**Solution**: Ensure configuration includes all required fields (description, interfaces).
### Invalid Queue Names
```bash
Exception: Invalid queue URL format
```
**Solution**: Verify queue URLs follow the correct Pulsar format with proper tenant/namespace.
### Connection Errors
```bash
Exception: Connection refused
```
**Solution**: Check the API URL and ensure TrustGraph is running.
## Validation
### JSON Syntax Check
```bash
# Validate JSON before uploading
config='{"description": "test flow", "interfaces": {}}'
echo "$config" | jq . > /dev/null && echo "Valid JSON" || echo "Invalid JSON"
```
### Flow Class Verification
```bash
# After uploading, verify the flow class exists
tg-show-flow-classes | grep "my-flow-class"
# Get the flow class definition to verify content
tg-get-flow-class -n "my-flow-class"
```
## Flow Class Lifecycle
### Development Workflow
```bash
# 1. Create flow class
tg-put-flow-class -n "dev-flow" -c "$dev_config"
# 2. Test with flow instance
tg-start-flow -n "dev-flow" -i "test-instance" -d "Testing"
# 3. Update flow class as needed
tg-put-flow-class -n "dev-flow" -c "$updated_config"
# 4. Restart flow instance with updates
tg-stop-flow -i "test-instance"
tg-start-flow -n "dev-flow" -i "test-instance" -d "Testing updated"
```
## Environment Variables
- `TRUSTGRAPH_URL`: Default API URL
## Related Commands
- [`tg-get-flow-class`](tg-get-flow-class.md) - Retrieve flow class definitions
- [`tg-show-flow-classes`](tg-show-flow-classes.md) - List available flow classes
- [`tg-delete-flow-class`](tg-delete-flow-class.md) - Remove flow class definitions
- [`tg-start-flow`](tg-start-flow.md) - Create flow instances from classes
## API Integration
This command uses the [Flow API](../apis/api-flow.md) with the `put-class` operation to store flow class definitions.
## Use Cases
### Custom Processing Pipelines
```bash
# Create specialized medical analysis flow
tg-put-flow-class -n "medical-nlp" -c "$medical_config"
```
### Development Environments
```bash
# Create lightweight development flow
tg-put-flow-class -n "dev-minimal" -c "$minimal_config"
```
### Production Deployments
```bash
# Create robust production flow with all services
tg-put-flow-class -n "production-full" -c "$production_config"
```
### Domain-Specific Workflows
```bash
# Create legal document analysis flow
tg-put-flow-class -n "legal-analysis" -c "$legal_config"
```
## Best Practices
1. **Descriptive Names**: Use clear, descriptive flow class names
2. **Comprehensive Descriptions**: Include detailed descriptions of flow capabilities
3. **Consistent Naming**: Follow consistent queue naming conventions
4. **Version Control**: Store flow class configurations in version control
5. **Testing**: Test flow classes thoroughly before production use
6. **Documentation**: Document flow class purposes and requirements
## Template Examples
### Minimal Flow Class
```json
{
"description": "Minimal text processing flow",
"interfaces": {
"text-completion": {
"request": "non-persistent://tg/request/text-completion:minimal",
"response": "non-persistent://tg/response/text-completion:minimal"
}
}
}
```
### RAG-Focused Flow Class
```json
{
"description": "Retrieval Augmented Generation flow",
"interfaces": {
"graph-rag": {
"request": "non-persistent://tg/request/graph-rag:rag-flow",
"response": "non-persistent://tg/response/graph-rag:rag-flow"
},
"document-rag": {
"request": "non-persistent://tg/request/document-rag:rag-flow",
"response": "non-persistent://tg/response/document-rag:rag-flow"
},
"embeddings": {
"request": "non-persistent://tg/request/embeddings:rag-flow",
"response": "non-persistent://tg/response/embeddings:rag-flow"
}
}
}
```
### Document Processing Flow Class
```json
{
"description": "Document ingestion and processing flow",
"interfaces": {
"text-load": "persistent://tg/flow/text-document-load:doc-proc",
"document-load": "persistent://tg/flow/document-load:doc-proc",
"triples-store": "persistent://tg/flow/triples-store:doc-proc",
"embeddings": {
"request": "non-persistent://tg/request/embeddings:doc-proc",
"response": "non-persistent://tg/response/embeddings:doc-proc"
}
}
}
```

View file

@ -0,0 +1,330 @@
# tg-show-flow-classes
Lists all defined flow classes in TrustGraph with their descriptions and tags.
## Synopsis
```bash
tg-show-flow-classes [options]
```
## Description
The `tg-show-flow-classes` command displays a formatted table of all flow class definitions currently stored in TrustGraph. Each flow class is shown with its name, description, and associated tags.
Flow classes are templates that define the structure and services available for creating flow instances. This command helps you understand what flow classes are available for use.
## Options
### Optional Arguments
- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
## Examples
### List All Flow Classes
```bash
tg-show-flow-classes
```
Output:
```
+-----------------+----------------------------------+----------------------+
| flow class | description | tags |
+-----------------+----------------------------------+----------------------+
| document-proc | Document processing pipeline | production, nlp |
| data-analysis | Data analysis and visualization | analytics, dev |
| web-scraper | Web content extraction flow | scraping, batch |
| chat-assistant | Conversational AI assistant | ai, interactive |
+-----------------+----------------------------------+----------------------+
```
### Using Custom API URL
```bash
tg-show-flow-classes -u http://production:8088/
```
### Filter Flow Classes
```bash
# Show only production-tagged flow classes
tg-show-flow-classes | grep "production"
# Count total flow classes
tg-show-flow-classes | grep -c "^|"
# Show flow classes with specific patterns
tg-show-flow-classes | grep -E "(document|text|nlp)"
```
## Output Format
The command displays results in a formatted table with columns:
- **flow class**: The unique name/identifier of the flow class
- **description**: Human-readable description of the flow class purpose
- **tags**: Comma-separated list of categorization tags
### Empty Results
If no flow classes exist:
```
No flows.
```
## Use Cases
### Flow Class Discovery
```bash
# Find available flow classes for document processing
tg-show-flow-classes | grep -i document
# List all AI-related flow classes
tg-show-flow-classes | grep -i "ai\|nlp\|chat\|assistant"
# Find development vs production flow classes
tg-show-flow-classes | grep -E "(dev|test|staging)"
tg-show-flow-classes | grep "production"
```
### Flow Class Management
```bash
# Get list of flow class names for scripting
tg-show-flow-classes | awk 'NR>3 && /^\|/ {gsub(/[| ]/, "", $2); print $2}' | grep -v "^$"
# Check if specific flow class exists
if tg-show-flow-classes | grep -q "target-flow"; then
echo "Flow class 'target-flow' exists"
else
echo "Flow class 'target-flow' not found"
fi
```
### Environment Comparison
```bash
# Compare flow classes between environments
echo "Development environment:"
tg-show-flow-classes -u http://dev:8088/
echo "Production environment:"
tg-show-flow-classes -u http://prod:8088/
```
### Reporting and Documentation
```bash
# Generate flow class inventory report
echo "Flow Class Inventory - $(date)" > flow-inventory.txt
echo "=====================================" >> flow-inventory.txt
tg-show-flow-classes >> flow-inventory.txt
# Create CSV export
echo "flow_class,description,tags" > flow-classes.csv
tg-show-flow-classes | awk 'NR>3 && /^\|/ {
gsub(/^\| */, "", $0); gsub(/ *\|$/, "", $0);
gsub(/ *\| */, ",", $0); print $0
}' >> flow-classes.csv
```
## Error Handling
### Connection Errors
```bash
Exception: Connection refused
```
**Solution**: Check the API URL and ensure TrustGraph is running.
### Permission Errors
```bash
Exception: Access denied to list flow classes
```
**Solution**: Verify user permissions for reading flow class definitions.
### Network Timeouts
```bash
Exception: Request timeout
```
**Solution**: Check network connectivity and API server status.
## Integration with Other Commands
### Flow Class Lifecycle
```bash
# 1. List available flow classes
tg-show-flow-classes
# 2. Get details of specific flow class
tg-get-flow-class -n "interesting-flow"
# 3. Start flow instance from class
tg-start-flow -n "interesting-flow" -i "my-instance"
# 4. Monitor flow instance
tg-show-flows | grep "my-instance"
```
### Bulk Operations
```bash
# Process all flow classes
tg-show-flow-classes | awk 'NR>3 && /^\|/ {gsub(/[| ]/, "", $2); if($2) print $2}' | \
while read class_name; do
if [ -n "$class_name" ]; then
echo "Processing flow class: $class_name"
tg-get-flow-class -n "$class_name" > "backup-$class_name.json"
fi
done
```
### Automated Validation
```bash
# Check flow class health
echo "Validating flow classes..."
tg-show-flow-classes | awk 'NR>3 && /^\|/ {gsub(/[| ]/, "", $2); if($2) print $2}' | \
while read class_name; do
if [ -n "$class_name" ]; then
echo -n "Checking $class_name... "
if tg-get-flow-class -n "$class_name" > /dev/null 2>&1; then
echo "OK"
else
echo "ERROR"
fi
fi
done
```
## Advanced Usage
### Flow Class Analysis
```bash
# Analyze flow class distribution by tags
tg-show-flow-classes | awk 'NR>3 && /^\|/ {
# Extract tags column
split($0, parts, "|");
tags = parts[4];
gsub(/^ *| *$/, "", tags);
if (tags) {
split(tags, tag_array, ",");
for (i in tag_array) {
gsub(/^ *| *$/, "", tag_array[i]);
if (tag_array[i]) print tag_array[i];
}
}
}' | sort | uniq -c | sort -nr
```
### Environment Synchronization
```bash
# Sync flow classes between environments
echo "Synchronizing flow classes from dev to staging..."
# Get list from development
dev_classes=$(tg-show-flow-classes -u http://dev:8088/ | \
awk 'NR>3 && /^\|/ {gsub(/[| ]/, "", $2); if($2) print $2}')
# Check each class in staging
for class in $dev_classes; do
if tg-show-flow-classes -u http://staging:8088/ | grep -q "$class"; then
echo "$class: Already exists in staging"
else
echo "$class: Missing in staging - needs sync"
# Get from dev and put to staging
tg-get-flow-class -n "$class" -u http://dev:8088/ > temp-class.json
tg-put-flow-class -n "$class" -c "$(cat temp-class.json)" -u http://staging:8088/
rm temp-class.json
fi
done
```
### Monitoring Script
```bash
#!/bin/bash
# monitor-flow-classes.sh
api_url="${1:-http://localhost:8088/}"
echo "Flow Class Monitoring Report - $(date)"
echo "API URL: $api_url"
echo "----------------------------------------"
# Total count
total=$(tg-show-flow-classes -u "$api_url" | grep -c "^|" 2>/dev/null || echo "0")
echo "Total flow classes: $((total - 3))" # Subtract header rows
# Tag analysis
echo -e "\nTag distribution:"
tg-show-flow-classes -u "$api_url" | awk 'NR>3 && /^\|/ {
split($0, parts, "|");
tags = parts[4];
gsub(/^ *| *$/, "", tags);
if (tags) {
split(tags, tag_array, ",");
for (i in tag_array) {
gsub(/^ *| *$/, "", tag_array[i]);
if (tag_array[i]) print tag_array[i];
}
}
}' | sort | uniq -c | sort -nr
# Health check
echo -e "\nHealth check:"
healthy=0
unhealthy=0
tg-show-flow-classes -u "$api_url" | awk 'NR>3 && /^\|/ {gsub(/[| ]/, "", $2); if($2) print $2}' | \
while read class_name; do
if [ -n "$class_name" ]; then
if tg-get-flow-class -n "$class_name" -u "$api_url" > /dev/null 2>&1; then
healthy=$((healthy + 1))
else
unhealthy=$((unhealthy + 1))
echo " ERROR: $class_name"
fi
fi
done
echo "Healthy: $healthy, Unhealthy: $unhealthy"
```
## Environment Variables
- `TRUSTGRAPH_URL`: Default API URL
## Related Commands
- [`tg-get-flow-class`](tg-get-flow-class.md) - Retrieve specific flow class definitions
- [`tg-put-flow-class`](tg-put-flow-class.md) - Create/update flow class definitions
- [`tg-delete-flow-class`](tg-delete-flow-class.md) - Delete flow class definitions
- [`tg-start-flow`](tg-start-flow.md) - Create flow instances from classes
- [`tg-show-flows`](tg-show-flows.md) - List active flow instances
## API Integration
This command uses the [Flow API](../apis/api-flow.md) with the `list-classes` operation to retrieve flow class listings.
## Best Practices
1. **Regular Inventory**: Periodically review available flow classes
2. **Documentation**: Ensure flow classes have meaningful descriptions
3. **Tagging**: Use consistent tagging for better organization
4. **Cleanup**: Remove unused or deprecated flow classes
5. **Monitoring**: Include flow class health checks in monitoring
6. **Environment Parity**: Keep flow classes synchronized across environments
## Troubleshooting
### No Output
```bash
# If command returns no output, check API connectivity
tg-show-flow-classes -u http://localhost:8088/
# Verify TrustGraph is running and accessible
```
### Formatting Issues
```bash
# If table formatting is broken, check terminal width
export COLUMNS=120
tg-show-flow-classes
```
### Missing Flow Classes
```bash
# If expected flow classes are missing, verify:
# 1. Correct API URL
# 2. Database connectivity
# 3. Flow class definitions are properly stored
```

View file

@ -0,0 +1,335 @@
# tg-unload-kg-core
Removes a knowledge core from an active flow without deleting the stored core.
## Synopsis
```bash
tg-unload-kg-core --id CORE_ID [options]
```
## Description
The `tg-unload-kg-core` command removes a previously loaded knowledge core from an active processing flow, making that knowledge unavailable for queries and processing within that specific flow. The knowledge core remains stored in the system and can be loaded again later or into different flows.
This is useful for managing flow memory usage, switching knowledge contexts, or temporarily removing knowledge without permanent deletion.
## Options
### Required Arguments
- `--id, --identifier CORE_ID`: Identifier of the knowledge core to unload
### Optional Arguments
- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
- `-U, --user USER`: User identifier (default: `trustgraph`)
- `-f, --flow-id FLOW`: Flow ID to unload knowledge from (default: `default`)
## Examples
### Unload from Default Flow
```bash
tg-unload-kg-core --id "research-knowledge"
```
### Unload from Specific Flow
```bash
tg-unload-kg-core \
--id "medical-knowledge" \
--flow-id "medical-analysis" \
-U medical-team
```
### Unload Multiple Cores
```bash
# Unload several knowledge cores from a flow
tg-unload-kg-core --id "core-1" --flow-id "analysis-flow"
tg-unload-kg-core --id "core-2" --flow-id "analysis-flow"
tg-unload-kg-core --id "core-3" --flow-id "analysis-flow"
```
### Using Custom API URL
```bash
tg-unload-kg-core \
--id "production-knowledge" \
--flow-id "prod-flow" \
-u http://production:8088/
```
## Prerequisites
### Knowledge Core Must Be Loaded
The knowledge core must currently be loaded in the specified flow:
```bash
# Check what's loaded by querying the flow
tg-show-graph -f target-flow | head -10
# If no output, core may not be loaded
```
### Flow Must Be Running
The target flow must be active:
```bash
# Check running flows
tg-show-flows
# Verify the target flow exists
tg-show-flows | grep "target-flow"
```
## Unloading Process
1. **Validation**: Verifies knowledge core is loaded in the specified flow
2. **Query Termination**: Stops any ongoing queries using the knowledge
3. **Index Cleanup**: Removes knowledge indexes from flow context
4. **Memory Release**: Frees memory allocated to the knowledge core
5. **Service Update**: Updates flow services to reflect knowledge unavailability
## Effects of Unloading
### Knowledge Becomes Unavailable
After unloading, the knowledge is no longer accessible through the flow:
```bash
# Before unloading - knowledge available
tg-invoke-graph-rag -q "What knowledge is loaded?" -f my-flow
# Unload the knowledge
tg-unload-kg-core --id "my-knowledge" --flow-id "my-flow"
# After unloading - reduced knowledge available
tg-invoke-graph-rag -q "What knowledge is loaded?" -f my-flow
```
### Memory Recovery
- RAM used by knowledge indexes is freed
- Flow performance may improve
- Other knowledge cores in the flow remain unaffected
### Core Preservation
- Knowledge core remains stored in the system
- Can be reloaded later
- Available for loading into other flows
## Output
Successful unloading typically produces no output:
```bash
# Unload core (no output expected)
tg-unload-kg-core --id "test-core" --flow-id "test-flow"
# Verify unloading by checking available knowledge
tg-show-graph -f test-flow | wc -l
# Should show fewer triples if core was successfully unloaded
```
## Error Handling
### Knowledge Core Not Loaded
```bash
Exception: Knowledge core 'my-core' not loaded in flow 'my-flow'
```
**Solution**: Verify the core is actually loaded using `tg-show-graph` or load it first with `tg-load-kg-core`.
### Flow Not Found
```bash
Exception: Flow 'invalid-flow' not found
```
**Solution**: Check running flows with `tg-show-flows` and verify the flow ID.
### Permission Errors
```bash
Exception: Access denied to unload knowledge core
```
**Solution**: Verify user permissions for the knowledge core and flow.
### Connection Errors
```bash
Exception: Connection refused
```
**Solution**: Check the API URL and ensure TrustGraph is running.
## Verification
### Check Knowledge Reduction
```bash
# Count triples before unloading
before=$(tg-show-graph -f my-flow | wc -l)
# Unload knowledge
tg-unload-kg-core --id "my-core" --flow-id "my-flow"
# Count triples after unloading
after=$(tg-show-graph -f my-flow | wc -l)
echo "Triples before: $before, after: $after"
```
### Test Query Impact
```bash
# Test queries before and after unloading
tg-invoke-graph-rag -q "test query" -f my-flow
# Should work with loaded knowledge
tg-unload-kg-core --id "relevant-core" --flow-id "my-flow"
tg-invoke-graph-rag -q "test query" -f my-flow
# May return different results or "no relevant knowledge found"
```
## Use Cases
### Memory Management
```bash
# Free up memory by unloading unused knowledge
tg-unload-kg-core --id "large-historical-data" --flow-id "analysis-flow"
# Load more relevant knowledge
tg-load-kg-core --id "current-data" --flow-id "analysis-flow"
```
### Context Switching
```bash
# Switch from medical to legal knowledge context
tg-unload-kg-core --id "medical-knowledge" --flow-id "analysis-flow"
tg-load-kg-core --id "legal-knowledge" --flow-id "analysis-flow"
```
### Selective Knowledge Loading
```bash
# Load only specific knowledge for focused analysis
tg-unload-kg-core --id "general-knowledge" --flow-id "specialized-flow"
tg-load-kg-core --id "domain-specific" --flow-id "specialized-flow"
```
### Testing and Development
```bash
# Test flow behavior with different knowledge sets
tg-unload-kg-core --id "production-data" --flow-id "test-flow"
tg-load-kg-core --id "test-data" --flow-id "test-flow"
# Run tests
./run-knowledge-tests.sh
# Restore production knowledge
tg-unload-kg-core --id "test-data" --flow-id "test-flow"
tg-load-kg-core --id "production-data" --flow-id "test-flow"
```
### Flow Maintenance
```bash
# Prepare flow for maintenance by unloading all knowledge
cores=$(tg-show-kg-cores)
for core in $cores; do
tg-unload-kg-core --id "$core" --flow-id "maintenance-flow" 2>/dev/null || true
done
# Perform maintenance
./flow-maintenance.sh
# Reload required knowledge
tg-load-kg-core --id "essential-core" --flow-id "maintenance-flow"
```
## Knowledge Management Workflow
### Dynamic Knowledge Loading
```bash
# Function to switch knowledge contexts
switch_knowledge_context() {
local flow_id=$1
local old_core=$2
local new_core=$3
echo "Switching from $old_core to $new_core in $flow_id"
# Unload old knowledge
tg-unload-kg-core --id "$old_core" --flow-id "$flow_id"
# Load new knowledge
tg-load-kg-core --id "$new_core" --flow-id "$flow_id"
echo "Context switch completed"
}
# Usage
switch_knowledge_context "analysis-flow" "old-data" "new-data"
```
### Bulk Knowledge Management
```bash
# Unload all knowledge from a flow
unload_all_knowledge() {
local flow_id=$1
# Get list of potentially loaded cores
tg-show-kg-cores | while read core; do
echo "Attempting to unload $core from $flow_id"
tg-unload-kg-core --id "$core" --flow-id "$flow_id" 2>/dev/null || true
done
echo "All knowledge unloaded from $flow_id"
}
# Usage
unload_all_knowledge "cleanup-flow"
```
## Environment Variables
- `TRUSTGRAPH_URL`: Default API URL
## Related Commands
- [`tg-load-kg-core`](tg-load-kg-core.md) - Load knowledge core into flow
- [`tg-show-kg-cores`](tg-show-kg-cores.md) - List available knowledge cores
- [`tg-show-graph`](tg-show-graph.md) - View currently loaded knowledge
- [`tg-show-flows`](tg-show-flows.md) - List active flows
## API Integration
This command uses the [Knowledge API](../apis/api-knowledge.md) with the `unload-kg-core` operation to remove knowledge from active flows.
## Best Practices
1. **Memory Monitoring**: Monitor flow memory usage when loading/unloading knowledge
2. **Graceful Unloading**: Ensure no critical queries are running before unloading
3. **Documentation**: Document which knowledge cores are needed for each flow
4. **Testing**: Test flow behavior after unloading knowledge
5. **Backup Strategy**: Keep knowledge cores stored even when not loaded
6. **Performance Optimization**: Unload unused knowledge to improve performance
## Troubleshooting
### Knowledge Still Appears in Queries
```bash
# If knowledge still appears after unloading
# Check if multiple cores contain similar data
tg-show-graph -f my-flow | grep "expected-removed-entity"
# Verify all relevant cores were unloaded
```
### Memory Not Released
```bash
# If memory usage doesn't decrease after unloading
# Check system memory usage
free -h
# Contact system administrator if memory leak suspected
```
### Query Performance Issues
```bash
# If queries become slow after unloading
# May need to reload essential knowledge
tg-load-kg-core --id "essential-core" --flow-id "slow-flow"
# Or restart the flow
tg-stop-flow -i "slow-flow"
tg-start-flow -n "flow-class" -i "slow-flow" -d "Restarted flow"
```