mirror of
https://github.com/trustgraph-ai/trustgraph.git
synced 2026-07-22 19:51:02 +02:00
More commands
This commit is contained in:
parent
3014b11ce9
commit
312375fa07
12 changed files with 4769 additions and 0 deletions
330
docs/cli/tg-delete-flow-class.md
Normal file
330
docs/cli/tg-delete-flow-class.md
Normal file
|
|
@ -0,0 +1,330 @@
|
||||||
|
# tg-delete-flow-class
|
||||||
|
|
||||||
|
Permanently deletes a flow class definition from TrustGraph.
|
||||||
|
|
||||||
|
## Synopsis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tg-delete-flow-class -n CLASS_NAME [options]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Description
|
||||||
|
|
||||||
|
The `tg-delete-flow-class` command permanently removes a flow class definition from TrustGraph. This operation cannot be undone, so use with caution.
|
||||||
|
|
||||||
|
**⚠️ Warning**: Deleting a flow class that has active flow instances may cause those instances to become unusable. Always check for active flows before deletion.
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
### Required Arguments
|
||||||
|
|
||||||
|
- `-n, --class-name CLASS_NAME`: Name of the flow class to delete
|
||||||
|
|
||||||
|
### Optional Arguments
|
||||||
|
|
||||||
|
- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
### Delete a Flow Class
|
||||||
|
```bash
|
||||||
|
tg-delete-flow-class -n "old-test-flow"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Delete with Custom API URL
|
||||||
|
```bash
|
||||||
|
tg-delete-flow-class -n "deprecated-flow" -u http://staging:8088/
|
||||||
|
```
|
||||||
|
|
||||||
|
### Safe Deletion Workflow
|
||||||
|
```bash
|
||||||
|
# 1. Check if flow class exists
|
||||||
|
tg-show-flow-classes | grep "target-flow"
|
||||||
|
|
||||||
|
# 2. Backup the flow class first
|
||||||
|
tg-get-flow-class -n "target-flow" > backup-target-flow.json
|
||||||
|
|
||||||
|
# 3. Check for active flow instances
|
||||||
|
tg-show-flows | grep "target-flow"
|
||||||
|
|
||||||
|
# 4. Delete the flow class
|
||||||
|
tg-delete-flow-class -n "target-flow"
|
||||||
|
|
||||||
|
# 5. Verify deletion
|
||||||
|
tg-show-flow-classes | grep "target-flow" || echo "Flow class deleted successfully"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
### Flow Class Must Exist
|
||||||
|
Verify the flow class exists before attempting deletion:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# List all flow classes
|
||||||
|
tg-show-flow-classes
|
||||||
|
|
||||||
|
# Check specific flow class
|
||||||
|
tg-show-flow-classes | grep "target-class"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Check for Active Flow Instances
|
||||||
|
Before deleting a flow class, check if any flow instances are using it:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# List all active flows
|
||||||
|
tg-show-flows
|
||||||
|
|
||||||
|
# Look for instances using the flow class
|
||||||
|
tg-show-flows | grep "target-class"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
### Flow Class Not Found
|
||||||
|
```bash
|
||||||
|
Exception: Flow class 'nonexistent-class' not found
|
||||||
|
```
|
||||||
|
**Solution**: Verify the flow class exists with `tg-show-flow-classes`.
|
||||||
|
|
||||||
|
### Connection Errors
|
||||||
|
```bash
|
||||||
|
Exception: Connection refused
|
||||||
|
```
|
||||||
|
**Solution**: Check the API URL and ensure TrustGraph is running.
|
||||||
|
|
||||||
|
### Permission Errors
|
||||||
|
```bash
|
||||||
|
Exception: Access denied to delete flow class
|
||||||
|
```
|
||||||
|
**Solution**: Verify user permissions for flow class management.
|
||||||
|
|
||||||
|
### Active Flow Instances
|
||||||
|
```bash
|
||||||
|
Exception: Cannot delete flow class with active instances
|
||||||
|
```
|
||||||
|
**Solution**: Stop all flow instances using this class before deletion.
|
||||||
|
|
||||||
|
## Use Cases
|
||||||
|
|
||||||
|
### Cleanup Development Classes
|
||||||
|
```bash
|
||||||
|
# Delete test and development flow classes
|
||||||
|
test_classes=("test-flow-v1" "dev-experiment" "prototype-flow")
|
||||||
|
for class in "${test_classes[@]}"; do
|
||||||
|
echo "Deleting $class..."
|
||||||
|
tg-delete-flow-class -n "$class"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Migration Cleanup
|
||||||
|
```bash
|
||||||
|
# After migrating to new flow classes, remove old ones
|
||||||
|
old_classes=("legacy-flow" "deprecated-processor" "old-pipeline")
|
||||||
|
for class in "${old_classes[@]}"; do
|
||||||
|
# Backup first
|
||||||
|
tg-get-flow-class -n "$class" > "backup-$class.json" 2>/dev/null
|
||||||
|
|
||||||
|
# Delete
|
||||||
|
tg-delete-flow-class -n "$class"
|
||||||
|
echo "Deleted $class"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Conditional Deletion
|
||||||
|
```bash
|
||||||
|
# Delete flow class only if no active instances exist
|
||||||
|
flow_class="target-flow"
|
||||||
|
active_instances=$(tg-show-flows | grep "$flow_class" | wc -l)
|
||||||
|
|
||||||
|
if [ $active_instances -eq 0 ]; then
|
||||||
|
echo "No active instances found, deleting flow class..."
|
||||||
|
tg-delete-flow-class -n "$flow_class"
|
||||||
|
else
|
||||||
|
echo "Warning: $active_instances active instances found. Cannot delete."
|
||||||
|
tg-show-flows | grep "$flow_class"
|
||||||
|
fi
|
||||||
|
```
|
||||||
|
|
||||||
|
## Safety Considerations
|
||||||
|
|
||||||
|
### Always Backup First
|
||||||
|
```bash
|
||||||
|
# Create backup before deletion
|
||||||
|
flow_class="important-flow"
|
||||||
|
backup_dir="flow-class-backups/$(date +%Y%m%d-%H%M%S)"
|
||||||
|
mkdir -p "$backup_dir"
|
||||||
|
|
||||||
|
echo "Backing up flow class: $flow_class"
|
||||||
|
tg-get-flow-class -n "$flow_class" > "$backup_dir/$flow_class.json"
|
||||||
|
|
||||||
|
if [ $? -eq 0 ]; then
|
||||||
|
echo "Backup created: $backup_dir/$flow_class.json"
|
||||||
|
echo "Proceeding with deletion..."
|
||||||
|
tg-delete-flow-class -n "$flow_class"
|
||||||
|
else
|
||||||
|
echo "Backup failed. Aborting deletion."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verification Script
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
# safe-delete-flow-class.sh
|
||||||
|
flow_class="$1"
|
||||||
|
|
||||||
|
if [ -z "$flow_class" ]; then
|
||||||
|
echo "Usage: $0 <flow-class-name>"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Safety checks for deleting flow class: $flow_class"
|
||||||
|
|
||||||
|
# Check if flow class exists
|
||||||
|
if ! tg-show-flow-classes | grep -q "$flow_class"; then
|
||||||
|
echo "ERROR: Flow class '$flow_class' not found"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check for active instances
|
||||||
|
active_count=$(tg-show-flows | grep "$flow_class" | wc -l)
|
||||||
|
if [ $active_count -gt 0 ]; then
|
||||||
|
echo "ERROR: Found $active_count active instances using this flow class"
|
||||||
|
echo "Active instances:"
|
||||||
|
tg-show-flows | grep "$flow_class"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Create backup
|
||||||
|
backup_file="backup-$flow_class-$(date +%Y%m%d-%H%M%S).json"
|
||||||
|
echo "Creating backup: $backup_file"
|
||||||
|
tg-get-flow-class -n "$flow_class" > "$backup_file"
|
||||||
|
|
||||||
|
if [ $? -ne 0 ]; then
|
||||||
|
echo "ERROR: Failed to create backup"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Confirm deletion
|
||||||
|
echo "Ready to delete flow class: $flow_class"
|
||||||
|
echo "Backup saved as: $backup_file"
|
||||||
|
read -p "Are you sure you want to delete this flow class? (y/N): " confirm
|
||||||
|
|
||||||
|
if [ "$confirm" = "y" ] || [ "$confirm" = "Y" ]; then
|
||||||
|
echo "Deleting flow class..."
|
||||||
|
tg-delete-flow-class -n "$flow_class"
|
||||||
|
|
||||||
|
# Verify deletion
|
||||||
|
if ! tg-show-flow-classes | grep -q "$flow_class"; then
|
||||||
|
echo "Flow class deleted successfully"
|
||||||
|
else
|
||||||
|
echo "ERROR: Flow class still exists after deletion"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo "Deletion cancelled"
|
||||||
|
rm "$backup_file"
|
||||||
|
fi
|
||||||
|
```
|
||||||
|
|
||||||
|
## Integration with Other Commands
|
||||||
|
|
||||||
|
### Complete Flow Class Lifecycle
|
||||||
|
```bash
|
||||||
|
# 1. List existing flow classes
|
||||||
|
tg-show-flow-classes
|
||||||
|
|
||||||
|
# 2. Get flow class details
|
||||||
|
tg-get-flow-class -n "target-flow"
|
||||||
|
|
||||||
|
# 3. Check for active instances
|
||||||
|
tg-show-flows | grep "target-flow"
|
||||||
|
|
||||||
|
# 4. Stop active instances if needed
|
||||||
|
tg-stop-flow -i "instance-id"
|
||||||
|
|
||||||
|
# 5. Create backup
|
||||||
|
tg-get-flow-class -n "target-flow" > backup.json
|
||||||
|
|
||||||
|
# 6. Delete flow class
|
||||||
|
tg-delete-flow-class -n "target-flow"
|
||||||
|
|
||||||
|
# 7. Verify deletion
|
||||||
|
tg-show-flow-classes | grep "target-flow"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Bulk Deletion with Validation
|
||||||
|
```bash
|
||||||
|
# Delete multiple flow classes safely
|
||||||
|
classes_to_delete=("old-flow1" "old-flow2" "test-flow")
|
||||||
|
|
||||||
|
for class in "${classes_to_delete[@]}"; do
|
||||||
|
echo "Processing $class..."
|
||||||
|
|
||||||
|
# Check if exists
|
||||||
|
if ! tg-show-flow-classes | grep -q "$class"; then
|
||||||
|
echo " $class not found, skipping"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check for active instances
|
||||||
|
if tg-show-flows | grep -q "$class"; then
|
||||||
|
echo " $class has active instances, skipping"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Backup and delete
|
||||||
|
tg-get-flow-class -n "$class" > "backup-$class.json"
|
||||||
|
tg-delete-flow-class -n "$class"
|
||||||
|
echo " $class deleted"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
- `TRUSTGRAPH_URL`: Default API URL
|
||||||
|
|
||||||
|
## Related Commands
|
||||||
|
|
||||||
|
- [`tg-show-flow-classes`](tg-show-flow-classes.md) - List available flow classes
|
||||||
|
- [`tg-get-flow-class`](tg-get-flow-class.md) - Retrieve flow class definitions
|
||||||
|
- [`tg-put-flow-class`](tg-put-flow-class.md) - Create/update flow class definitions
|
||||||
|
- [`tg-show-flows`](tg-show-flows.md) - List active flow instances
|
||||||
|
- [`tg-stop-flow`](tg-stop-flow.md) - Stop flow instances
|
||||||
|
|
||||||
|
## API Integration
|
||||||
|
|
||||||
|
This command uses the [Flow API](../apis/api-flow.md) with the `delete-class` operation to remove flow class definitions.
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
1. **Always Backup**: Create backups before deletion
|
||||||
|
2. **Check Dependencies**: Verify no active flow instances exist
|
||||||
|
3. **Confirmation**: Use interactive confirmation for important deletions
|
||||||
|
4. **Logging**: Log deletion operations for audit trails
|
||||||
|
5. **Permissions**: Ensure appropriate access controls for deletion operations
|
||||||
|
6. **Testing**: Test deletion procedures in non-production environments first
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Command Succeeds but Class Still Exists
|
||||||
|
```bash
|
||||||
|
# Check if deletion actually occurred
|
||||||
|
tg-show-flow-classes | grep "deleted-class"
|
||||||
|
|
||||||
|
# Verify API connectivity
|
||||||
|
tg-show-flow-classes > /dev/null && echo "API accessible"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Permissions Issues
|
||||||
|
```bash
|
||||||
|
# Verify user has deletion permissions
|
||||||
|
# Contact system administrator if access denied
|
||||||
|
```
|
||||||
|
|
||||||
|
### Network Connectivity
|
||||||
|
```bash
|
||||||
|
# Test API connectivity
|
||||||
|
curl -s "$TRUSTGRAPH_URL/api/v1/flow/classes" > /dev/null
|
||||||
|
echo "API response: $?"
|
||||||
|
```
|
||||||
312
docs/cli/tg-delete-kg-core.md
Normal file
312
docs/cli/tg-delete-kg-core.md
Normal file
|
|
@ -0,0 +1,312 @@
|
||||||
|
# tg-delete-kg-core
|
||||||
|
|
||||||
|
Permanently removes a knowledge core from the TrustGraph system.
|
||||||
|
|
||||||
|
## Synopsis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tg-delete-kg-core --id CORE_ID [options]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Description
|
||||||
|
|
||||||
|
The `tg-delete-kg-core` command permanently removes a stored knowledge core from the TrustGraph system. This operation is irreversible and will delete all RDF triples, graph embeddings, and metadata associated with the specified knowledge core.
|
||||||
|
|
||||||
|
**Warning**: This operation permanently deletes data. Ensure you have backups if the knowledge core might be needed in the future.
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
### Required Arguments
|
||||||
|
|
||||||
|
- `--id, --identifier CORE_ID`: Identifier of the knowledge core to delete
|
||||||
|
|
||||||
|
### Optional Arguments
|
||||||
|
|
||||||
|
- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
|
||||||
|
- `-U, --user USER`: User identifier (default: `trustgraph`)
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
### Delete Specific Knowledge Core
|
||||||
|
```bash
|
||||||
|
tg-delete-kg-core --id "old-research-data"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Delete with Specific User
|
||||||
|
```bash
|
||||||
|
tg-delete-kg-core --id "test-knowledge" -U developer
|
||||||
|
```
|
||||||
|
|
||||||
|
### Using Custom API URL
|
||||||
|
```bash
|
||||||
|
tg-delete-kg-core --id "obsolete-core" -u http://production:8088/
|
||||||
|
```
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
### Knowledge Core Must Exist
|
||||||
|
Verify the knowledge core exists before deletion:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check available knowledge cores
|
||||||
|
tg-show-kg-cores
|
||||||
|
|
||||||
|
# Ensure the core exists
|
||||||
|
tg-show-kg-cores | grep "target-core-id"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Backup Important Data
|
||||||
|
Create backups before deletion:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Export knowledge core before deletion
|
||||||
|
tg-get-kg-core --id "important-core" -o backup.msgpack
|
||||||
|
|
||||||
|
# Then proceed with deletion
|
||||||
|
tg-delete-kg-core --id "important-core"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Safety Considerations
|
||||||
|
|
||||||
|
### Unload from Flows First
|
||||||
|
Unload the knowledge core from any active flows:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check which flows might be using the core
|
||||||
|
tg-show-flows
|
||||||
|
|
||||||
|
# Unload from active flows
|
||||||
|
tg-unload-kg-core --id "target-core" --flow-id "active-flow"
|
||||||
|
|
||||||
|
# Then delete the core
|
||||||
|
tg-delete-kg-core --id "target-core"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verify Dependencies
|
||||||
|
Check if other systems depend on the knowledge core:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Search for references in flow configurations
|
||||||
|
tg-show-config | grep "target-core"
|
||||||
|
|
||||||
|
# Check processing history
|
||||||
|
tg-show-library-processing | grep "target-core"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deletion Process
|
||||||
|
|
||||||
|
1. **Validation**: Verifies knowledge core exists and user has permission
|
||||||
|
2. **Dependency Check**: Ensures core is not actively loaded in flows
|
||||||
|
3. **Data Removal**: Permanently deletes RDF triples and graph embeddings
|
||||||
|
4. **Metadata Cleanup**: Removes all associated metadata and references
|
||||||
|
5. **Index Updates**: Updates system indexes to reflect deletion
|
||||||
|
|
||||||
|
## Output
|
||||||
|
|
||||||
|
Successful deletion typically produces no output:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Delete core (no output expected on success)
|
||||||
|
tg-delete-kg-core --id "test-core"
|
||||||
|
|
||||||
|
# Verify deletion
|
||||||
|
tg-show-kg-cores | grep "test-core"
|
||||||
|
# Should return no results
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
### Knowledge Core Not Found
|
||||||
|
```bash
|
||||||
|
Exception: Knowledge core 'invalid-core' not found
|
||||||
|
```
|
||||||
|
**Solution**: Check available cores with `tg-show-kg-cores` and verify the core ID.
|
||||||
|
|
||||||
|
### Permission Denied
|
||||||
|
```bash
|
||||||
|
Exception: Access denied to knowledge core
|
||||||
|
```
|
||||||
|
**Solution**: Verify user permissions and ownership of the knowledge core.
|
||||||
|
|
||||||
|
### Core In Use
|
||||||
|
```bash
|
||||||
|
Exception: Knowledge core is currently loaded in active flows
|
||||||
|
```
|
||||||
|
**Solution**: Unload the core from all flows before deletion using `tg-unload-kg-core`.
|
||||||
|
|
||||||
|
### Connection Errors
|
||||||
|
```bash
|
||||||
|
Exception: Connection refused
|
||||||
|
```
|
||||||
|
**Solution**: Check the API URL and ensure TrustGraph is running.
|
||||||
|
|
||||||
|
## Deletion Verification
|
||||||
|
|
||||||
|
### Confirm Deletion
|
||||||
|
```bash
|
||||||
|
# Verify core no longer exists
|
||||||
|
tg-show-kg-cores | grep "deleted-core-id"
|
||||||
|
|
||||||
|
# Should return no results if successfully deleted
|
||||||
|
echo $? # Should be 1 (not found)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Check Flow Impact
|
||||||
|
```bash
|
||||||
|
# Verify flows are not affected
|
||||||
|
tg-show-flows
|
||||||
|
|
||||||
|
# Test that queries still work for remaining knowledge
|
||||||
|
tg-invoke-graph-rag -q "test query" -f remaining-flow
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
- `TRUSTGRAPH_URL`: Default API URL
|
||||||
|
|
||||||
|
## Related Commands
|
||||||
|
|
||||||
|
- [`tg-show-kg-cores`](tg-show-kg-cores.md) - List available knowledge cores
|
||||||
|
- [`tg-get-kg-core`](tg-get-kg-core.md) - Export knowledge core for backup
|
||||||
|
- [`tg-unload-kg-core`](tg-unload-kg-core.md) - Unload core from flows
|
||||||
|
- [`tg-put-kg-core`](tg-put-kg-core.md) - Store new knowledge cores
|
||||||
|
|
||||||
|
## API Integration
|
||||||
|
|
||||||
|
This command uses the [Knowledge API](../apis/api-knowledge.md) with the `delete-kg-core` operation to permanently remove knowledge cores.
|
||||||
|
|
||||||
|
## Use Cases
|
||||||
|
|
||||||
|
### Development Cleanup
|
||||||
|
```bash
|
||||||
|
# Remove test knowledge cores
|
||||||
|
tg-delete-kg-core --id "test-data-v1" -U developer
|
||||||
|
tg-delete-kg-core --id "experimental-core" -U developer
|
||||||
|
```
|
||||||
|
|
||||||
|
### Version Management
|
||||||
|
```bash
|
||||||
|
# Remove obsolete versions after upgrading
|
||||||
|
tg-get-kg-core --id "knowledge-v1" -o backup-v1.msgpack
|
||||||
|
tg-delete-kg-core --id "knowledge-v1"
|
||||||
|
# Keep only knowledge-v2
|
||||||
|
```
|
||||||
|
|
||||||
|
### Storage Cleanup
|
||||||
|
```bash
|
||||||
|
# Clean up unused knowledge cores
|
||||||
|
for core in $(tg-show-kg-cores | grep "temp-"); do
|
||||||
|
echo "Deleting temporary core: $core"
|
||||||
|
tg-delete-kg-core --id "$core"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Error Recovery
|
||||||
|
```bash
|
||||||
|
# Remove corrupted knowledge cores
|
||||||
|
tg-delete-kg-core --id "corrupted-core-2024"
|
||||||
|
tg-put-kg-core --id "restored-core-2024" -i restored-backup.msgpack
|
||||||
|
```
|
||||||
|
|
||||||
|
## Safe Deletion Workflow
|
||||||
|
|
||||||
|
### Standard Procedure
|
||||||
|
```bash
|
||||||
|
# 1. Backup the knowledge core
|
||||||
|
tg-get-kg-core --id "target-core" -o "backup-$(date +%Y%m%d).msgpack"
|
||||||
|
|
||||||
|
# 2. Unload from active flows
|
||||||
|
tg-unload-kg-core --id "target-core" --flow-id "production-flow"
|
||||||
|
|
||||||
|
# 3. Verify no dependencies
|
||||||
|
tg-show-config | grep "target-core"
|
||||||
|
|
||||||
|
# 4. Perform deletion
|
||||||
|
tg-delete-kg-core --id "target-core"
|
||||||
|
|
||||||
|
# 5. Verify deletion
|
||||||
|
tg-show-kg-cores | grep "target-core"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Bulk Deletion
|
||||||
|
```bash
|
||||||
|
# Delete multiple cores safely
|
||||||
|
cores_to_delete=("old-core-1" "old-core-2" "test-core")
|
||||||
|
|
||||||
|
for core in "${cores_to_delete[@]}"; do
|
||||||
|
echo "Processing $core..."
|
||||||
|
|
||||||
|
# Backup
|
||||||
|
tg-get-kg-core --id "$core" -o "backup-$core-$(date +%Y%m%d).msgpack"
|
||||||
|
|
||||||
|
# Delete
|
||||||
|
tg-delete-kg-core --id "$core"
|
||||||
|
|
||||||
|
# Verify
|
||||||
|
if tg-show-kg-cores | grep -q "$core"; then
|
||||||
|
echo "ERROR: $core still exists after deletion"
|
||||||
|
else
|
||||||
|
echo "SUCCESS: $core deleted"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
1. **Always Backup**: Export knowledge cores before deletion
|
||||||
|
2. **Check Dependencies**: Verify no flows are using the core
|
||||||
|
3. **Staged Deletion**: Delete test/development cores before production
|
||||||
|
4. **Verification**: Confirm deletion completed successfully
|
||||||
|
5. **Documentation**: Record why cores were deleted for audit purposes
|
||||||
|
6. **Access Control**: Ensure only authorized users can delete cores
|
||||||
|
|
||||||
|
## Recovery Options
|
||||||
|
|
||||||
|
### If Accidentally Deleted
|
||||||
|
```bash
|
||||||
|
# Restore from backup if available
|
||||||
|
tg-put-kg-core --id "restored-core" -i backup.msgpack
|
||||||
|
|
||||||
|
# Reload into flows if needed
|
||||||
|
tg-load-kg-core --id "restored-core" --flow-id "production-flow"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Audit Trail
|
||||||
|
```bash
|
||||||
|
# Keep records of deletions
|
||||||
|
echo "$(date): Deleted knowledge core 'old-core' - reason: obsolete version" >> deletion-log.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
## System Impact
|
||||||
|
|
||||||
|
### Storage Recovery
|
||||||
|
- Disk space is freed immediately
|
||||||
|
- Database indexes are updated
|
||||||
|
- System performance may improve
|
||||||
|
|
||||||
|
### Service Continuity
|
||||||
|
- Running flows continue to operate
|
||||||
|
- Other knowledge cores remain unaffected
|
||||||
|
- New knowledge cores can use the same ID
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Deletion Fails
|
||||||
|
```bash
|
||||||
|
# Check if core is loaded in flows
|
||||||
|
tg-show-flows | grep -A 10 "knowledge"
|
||||||
|
|
||||||
|
# Force unload if necessary
|
||||||
|
tg-unload-kg-core --id "stuck-core" --flow-id "problem-flow"
|
||||||
|
|
||||||
|
# Retry deletion
|
||||||
|
tg-delete-kg-core --id "stuck-core"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Partial Deletion
|
||||||
|
```bash
|
||||||
|
# If core still appears in listings
|
||||||
|
tg-show-kg-cores | grep "partially-deleted"
|
||||||
|
|
||||||
|
# Contact system administrator if deletion appears incomplete
|
||||||
|
```
|
||||||
344
docs/cli/tg-get-flow-class.md
Normal file
344
docs/cli/tg-get-flow-class.md
Normal file
|
|
@ -0,0 +1,344 @@
|
||||||
|
# tg-get-flow-class
|
||||||
|
|
||||||
|
Retrieves and displays a flow class definition in JSON format.
|
||||||
|
|
||||||
|
## Synopsis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tg-get-flow-class -n CLASS_NAME [options]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Description
|
||||||
|
|
||||||
|
The `tg-get-flow-class` command retrieves a stored flow class definition from TrustGraph and displays it in formatted JSON. This is useful for examining flow class configurations, creating backups, or preparing to modify existing flow classes.
|
||||||
|
|
||||||
|
The output can be saved to files for version control, documentation, or as input for creating new flow classes with `tg-put-flow-class`.
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
### Required Arguments
|
||||||
|
|
||||||
|
- `-n, --class-name CLASS_NAME`: Name of the flow class to retrieve
|
||||||
|
|
||||||
|
### Optional Arguments
|
||||||
|
|
||||||
|
- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
### Display Flow Class Definition
|
||||||
|
```bash
|
||||||
|
tg-get-flow-class -n "document-processing"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Save Flow Class to File
|
||||||
|
```bash
|
||||||
|
tg-get-flow-class -n "production-flow" > production-flow-backup.json
|
||||||
|
```
|
||||||
|
|
||||||
|
### Compare Flow Classes
|
||||||
|
```bash
|
||||||
|
# Get multiple flow classes for comparison
|
||||||
|
tg-get-flow-class -n "dev-flow" > dev-flow.json
|
||||||
|
tg-get-flow-class -n "prod-flow" > prod-flow.json
|
||||||
|
diff dev-flow.json prod-flow.json
|
||||||
|
```
|
||||||
|
|
||||||
|
### Using Custom API URL
|
||||||
|
```bash
|
||||||
|
tg-get-flow-class -n "remote-flow" -u http://production:8088/
|
||||||
|
```
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
The command outputs the flow class definition in formatted JSON:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"description": "Document processing and analysis flow",
|
||||||
|
"interfaces": {
|
||||||
|
"agent": {
|
||||||
|
"request": "non-persistent://tg/request/agent:doc-proc",
|
||||||
|
"response": "non-persistent://tg/response/agent:doc-proc"
|
||||||
|
},
|
||||||
|
"document-rag": {
|
||||||
|
"request": "non-persistent://tg/request/document-rag:doc-proc",
|
||||||
|
"response": "non-persistent://tg/response/document-rag:doc-proc"
|
||||||
|
},
|
||||||
|
"text-load": "persistent://tg/flow/text-document-load:doc-proc",
|
||||||
|
"document-load": "persistent://tg/flow/document-load:doc-proc",
|
||||||
|
"triples-store": "persistent://tg/flow/triples-store:doc-proc"
|
||||||
|
},
|
||||||
|
"tags": ["production", "document-processing"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Key Components
|
||||||
|
|
||||||
|
#### Description
|
||||||
|
Human-readable description of the flow class purpose and capabilities.
|
||||||
|
|
||||||
|
#### Interfaces
|
||||||
|
Service definitions showing:
|
||||||
|
- **Request/Response Services**: Services with both request and response queues
|
||||||
|
- **Fire-and-Forget Services**: Services with only input queues
|
||||||
|
|
||||||
|
#### Tags (Optional)
|
||||||
|
Categorization tags for organizing flow classes.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
### Flow Class Must Exist
|
||||||
|
Verify the flow class exists before retrieval:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check available flow classes
|
||||||
|
tg-show-flow-classes
|
||||||
|
|
||||||
|
# Look for specific class
|
||||||
|
tg-show-flow-classes | grep "target-class"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
### Flow Class Not Found
|
||||||
|
```bash
|
||||||
|
Exception: Flow class 'invalid-class' not found
|
||||||
|
```
|
||||||
|
**Solution**: Check available classes with `tg-show-flow-classes` and verify the class name.
|
||||||
|
|
||||||
|
### Connection Errors
|
||||||
|
```bash
|
||||||
|
Exception: Connection refused
|
||||||
|
```
|
||||||
|
**Solution**: Check the API URL and ensure TrustGraph is running.
|
||||||
|
|
||||||
|
### Permission Errors
|
||||||
|
```bash
|
||||||
|
Exception: Access denied to flow class
|
||||||
|
```
|
||||||
|
**Solution**: Verify user permissions for accessing flow class definitions.
|
||||||
|
|
||||||
|
## Use Cases
|
||||||
|
|
||||||
|
### Configuration Backup
|
||||||
|
```bash
|
||||||
|
# Backup all flow classes
|
||||||
|
mkdir -p flow-class-backups/$(date +%Y%m%d)
|
||||||
|
tg-show-flow-classes | awk '{print $1}' | while read class; do
|
||||||
|
if [ "$class" != "flow" ]; then # Skip header
|
||||||
|
tg-get-flow-class -n "$class" > "flow-class-backups/$(date +%Y%m%d)/$class.json"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Flow Class Migration
|
||||||
|
```bash
|
||||||
|
# Export from source environment
|
||||||
|
tg-get-flow-class -n "production-flow" -u http://source:8088/ > prod-flow.json
|
||||||
|
|
||||||
|
# Import to target environment
|
||||||
|
tg-put-flow-class -n "production-flow" -c "$(cat prod-flow.json)" -u http://target:8088/
|
||||||
|
```
|
||||||
|
|
||||||
|
### Template Creation
|
||||||
|
```bash
|
||||||
|
# Get existing flow class as template
|
||||||
|
tg-get-flow-class -n "base-flow" > template.json
|
||||||
|
|
||||||
|
# Modify template and create new class
|
||||||
|
sed 's/base-flow/new-flow/g' template.json > new-flow.json
|
||||||
|
tg-put-flow-class -n "custom-flow" -c "$(cat new-flow.json)"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configuration Analysis
|
||||||
|
```bash
|
||||||
|
# Analyze flow class configurations
|
||||||
|
tg-get-flow-class -n "complex-flow" | jq '.interfaces | keys'
|
||||||
|
tg-get-flow-class -n "complex-flow" | jq '.interfaces | length'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Version Control Integration
|
||||||
|
```bash
|
||||||
|
# Store flow classes in git
|
||||||
|
mkdir -p flow-classes
|
||||||
|
tg-get-flow-class -n "main-flow" > flow-classes/main-flow.json
|
||||||
|
git add flow-classes/main-flow.json
|
||||||
|
git commit -m "Update main-flow configuration"
|
||||||
|
```
|
||||||
|
|
||||||
|
## JSON Processing
|
||||||
|
|
||||||
|
### Extract Specific Information
|
||||||
|
```bash
|
||||||
|
# Get only interface names
|
||||||
|
tg-get-flow-class -n "my-flow" | jq -r '.interfaces | keys[]'
|
||||||
|
|
||||||
|
# Get only description
|
||||||
|
tg-get-flow-class -n "my-flow" | jq -r '.description'
|
||||||
|
|
||||||
|
# Get request queues
|
||||||
|
tg-get-flow-class -n "my-flow" | jq -r '.interfaces | to_entries[] | select(.value.request) | .value.request'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Validate Configuration
|
||||||
|
```bash
|
||||||
|
# Validate JSON structure
|
||||||
|
tg-get-flow-class -n "my-flow" | jq . > /dev/null && echo "Valid JSON" || echo "Invalid JSON"
|
||||||
|
|
||||||
|
# Check required fields
|
||||||
|
config=$(tg-get-flow-class -n "my-flow")
|
||||||
|
echo "$config" | jq -e '.description' > /dev/null || echo "Missing description"
|
||||||
|
echo "$config" | jq -e '.interfaces' > /dev/null || echo "Missing interfaces"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Integration with Other Commands
|
||||||
|
|
||||||
|
### Flow Class Lifecycle
|
||||||
|
```bash
|
||||||
|
# 1. Examine existing flow class
|
||||||
|
tg-get-flow-class -n "old-flow"
|
||||||
|
|
||||||
|
# 2. Save backup
|
||||||
|
tg-get-flow-class -n "old-flow" > old-flow-backup.json
|
||||||
|
|
||||||
|
# 3. Modify configuration
|
||||||
|
cp old-flow-backup.json new-flow.json
|
||||||
|
# Edit new-flow.json as needed
|
||||||
|
|
||||||
|
# 4. Upload new version
|
||||||
|
tg-put-flow-class -n "updated-flow" -c "$(cat new-flow.json)"
|
||||||
|
|
||||||
|
# 5. Test new flow class
|
||||||
|
tg-start-flow -n "updated-flow" -i "test-instance" -d "Testing updated flow"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Bulk Operations
|
||||||
|
```bash
|
||||||
|
# Process multiple flow classes
|
||||||
|
flow_classes=("flow1" "flow2" "flow3")
|
||||||
|
for class in "${flow_classes[@]}"; do
|
||||||
|
echo "Processing $class..."
|
||||||
|
tg-get-flow-class -n "$class" > "backup-$class.json"
|
||||||
|
|
||||||
|
# Modify configuration
|
||||||
|
sed 's/old-pattern/new-pattern/g' "backup-$class.json" > "updated-$class.json"
|
||||||
|
|
||||||
|
# Upload updated version
|
||||||
|
tg-put-flow-class -n "$class" -c "$(cat updated-$class.json)"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
- `TRUSTGRAPH_URL`: Default API URL
|
||||||
|
|
||||||
|
## Related Commands
|
||||||
|
|
||||||
|
- [`tg-put-flow-class`](tg-put-flow-class.md) - Upload/update flow class definitions
|
||||||
|
- [`tg-show-flow-classes`](tg-show-flow-classes.md) - List available flow classes
|
||||||
|
- [`tg-delete-flow-class`](tg-delete-flow-class.md) - Remove flow class definitions
|
||||||
|
- [`tg-start-flow`](tg-start-flow.md) - Create flow instances from classes
|
||||||
|
|
||||||
|
## API Integration
|
||||||
|
|
||||||
|
This command uses the [Flow API](../apis/api-flow.md) with the `get-class` operation to retrieve flow class definitions.
|
||||||
|
|
||||||
|
## Advanced Usage
|
||||||
|
|
||||||
|
### Configuration Diff
|
||||||
|
```bash
|
||||||
|
# Compare flow class versions
|
||||||
|
tg-get-flow-class -n "flow-v1" > v1.json
|
||||||
|
tg-get-flow-class -n "flow-v2" > v2.json
|
||||||
|
diff -u v1.json v2.json
|
||||||
|
```
|
||||||
|
|
||||||
|
### Extract Queue Information
|
||||||
|
```bash
|
||||||
|
# Get all queue names from flow class
|
||||||
|
tg-get-flow-class -n "my-flow" | jq -r '
|
||||||
|
.interfaces |
|
||||||
|
to_entries[] |
|
||||||
|
if .value | type == "object" then
|
||||||
|
.value.request, .value.response
|
||||||
|
else
|
||||||
|
.value
|
||||||
|
end
|
||||||
|
' | sort | uniq
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configuration Validation Script
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
# validate-flow-class.sh
|
||||||
|
flow_class="$1"
|
||||||
|
|
||||||
|
if [ -z "$flow_class" ]; then
|
||||||
|
echo "Usage: $0 <flow-class-name>"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Validating flow class: $flow_class"
|
||||||
|
|
||||||
|
# Get configuration
|
||||||
|
config=$(tg-get-flow-class -n "$flow_class" 2>/dev/null)
|
||||||
|
if [ $? -ne 0 ]; then
|
||||||
|
echo "ERROR: Flow class not found"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Validate JSON
|
||||||
|
echo "$config" | jq . > /dev/null
|
||||||
|
if [ $? -ne 0 ]; then
|
||||||
|
echo "ERROR: Invalid JSON structure"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check required fields
|
||||||
|
desc=$(echo "$config" | jq -r '.description // empty')
|
||||||
|
if [ -z "$desc" ]; then
|
||||||
|
echo "WARNING: Missing description"
|
||||||
|
fi
|
||||||
|
|
||||||
|
interfaces=$(echo "$config" | jq -r '.interfaces // empty')
|
||||||
|
if [ -z "$interfaces" ] || [ "$interfaces" = "null" ]; then
|
||||||
|
echo "ERROR: Missing interfaces"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Flow class validation passed"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
1. **Regular Backups**: Save flow class definitions before modifications
|
||||||
|
2. **Version Control**: Store configurations in version control systems
|
||||||
|
3. **Documentation**: Include meaningful descriptions in flow classes
|
||||||
|
4. **Validation**: Validate JSON structure before using configurations
|
||||||
|
5. **Template Management**: Use existing classes as templates for new ones
|
||||||
|
6. **Change Tracking**: Document changes when updating flow classes
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Empty Output
|
||||||
|
```bash
|
||||||
|
# If command returns empty output
|
||||||
|
tg-get-flow-class -n "my-flow"
|
||||||
|
# Check if flow class exists
|
||||||
|
tg-show-flow-classes | grep "my-flow"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Invalid JSON Output
|
||||||
|
```bash
|
||||||
|
# If output appears corrupted
|
||||||
|
tg-get-flow-class -n "my-flow" | jq .
|
||||||
|
# Should show parsing error if JSON is invalid
|
||||||
|
```
|
||||||
|
|
||||||
|
### Permission Issues
|
||||||
|
```bash
|
||||||
|
# If access denied errors occur
|
||||||
|
# Verify authentication and user permissions
|
||||||
|
# Contact system administrator if needed
|
||||||
|
```
|
||||||
365
docs/cli/tg-get-kg-core.md
Normal file
365
docs/cli/tg-get-kg-core.md
Normal file
|
|
@ -0,0 +1,365 @@
|
||||||
|
# tg-get-kg-core
|
||||||
|
|
||||||
|
Exports a knowledge core from TrustGraph to a MessagePack file.
|
||||||
|
|
||||||
|
## Synopsis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tg-get-kg-core --id CORE_ID -o OUTPUT_FILE [options]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Description
|
||||||
|
|
||||||
|
The `tg-get-kg-core` command retrieves a stored knowledge core from TrustGraph and exports it to a MessagePack format file. This allows you to backup knowledge cores, transfer them between systems, or examine their contents offline.
|
||||||
|
|
||||||
|
The exported file contains both RDF triples and graph embeddings in a compact binary format that can later be imported using `tg-put-kg-core`.
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
### Required Arguments
|
||||||
|
|
||||||
|
- `--id, --identifier CORE_ID`: Identifier of the knowledge core to export
|
||||||
|
- `-o, --output OUTPUT_FILE`: Path for the output MessagePack file
|
||||||
|
|
||||||
|
### Optional Arguments
|
||||||
|
|
||||||
|
- `-u, --url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `ws://localhost:8088/`)
|
||||||
|
- `-U, --user USER`: User identifier (default: `trustgraph`)
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
### Basic Knowledge Core Export
|
||||||
|
```bash
|
||||||
|
tg-get-kg-core --id "research-knowledge" -o research-backup.msgpack
|
||||||
|
```
|
||||||
|
|
||||||
|
### Export with Specific User
|
||||||
|
```bash
|
||||||
|
tg-get-kg-core \
|
||||||
|
--id "medical-knowledge" \
|
||||||
|
-o medical-backup.msgpack \
|
||||||
|
-U medical-team
|
||||||
|
```
|
||||||
|
|
||||||
|
### Export with Timestamped Filename
|
||||||
|
```bash
|
||||||
|
tg-get-kg-core \
|
||||||
|
--id "production-core" \
|
||||||
|
-o "production-backup-$(date +%Y%m%d-%H%M%S).msgpack"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Using Custom API URL
|
||||||
|
```bash
|
||||||
|
tg-get-kg-core \
|
||||||
|
--id "remote-core" \
|
||||||
|
-o remote-backup.msgpack \
|
||||||
|
-u ws://production:8088/
|
||||||
|
```
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
### Knowledge Core Must Exist
|
||||||
|
Verify the knowledge core exists:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check available knowledge cores
|
||||||
|
tg-show-kg-cores
|
||||||
|
|
||||||
|
# Verify specific core exists
|
||||||
|
tg-show-kg-cores | grep "target-core-id"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Output Directory Must Be Writable
|
||||||
|
Ensure the output directory exists and is writable:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Create backup directory if needed
|
||||||
|
mkdir -p backups
|
||||||
|
|
||||||
|
# Export to backup directory
|
||||||
|
tg-get-kg-core --id "my-core" -o backups/my-core-backup.msgpack
|
||||||
|
```
|
||||||
|
|
||||||
|
## Export Process
|
||||||
|
|
||||||
|
1. **Connection**: Establishes WebSocket connection to Knowledge API
|
||||||
|
2. **Request**: Sends get-kg-core request with core ID and user
|
||||||
|
3. **Streaming**: Receives data in chunks via WebSocket
|
||||||
|
4. **Processing**: Converts response data to MessagePack format
|
||||||
|
5. **Writing**: Writes binary data to output file
|
||||||
|
6. **Summary**: Reports statistics on exported data
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
The exported MessagePack file contains structured data with two types of messages:
|
||||||
|
|
||||||
|
### Triple Messages (`"t"`)
|
||||||
|
Contains RDF triples (facts and relationships):
|
||||||
|
```python
|
||||||
|
("t", {
|
||||||
|
"m": { # metadata
|
||||||
|
"i": "core-id",
|
||||||
|
"m": [], # metadata triples
|
||||||
|
"u": "user",
|
||||||
|
"c": "collection"
|
||||||
|
},
|
||||||
|
"t": [ # triples array
|
||||||
|
{
|
||||||
|
"s": {"value": "subject", "is_uri": true},
|
||||||
|
"p": {"value": "predicate", "is_uri": true},
|
||||||
|
"o": {"value": "object", "is_uri": false}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
### Graph Embedding Messages (`"ge"`)
|
||||||
|
Contains vector embeddings for entities:
|
||||||
|
```python
|
||||||
|
("ge", {
|
||||||
|
"m": { # metadata
|
||||||
|
"i": "core-id",
|
||||||
|
"m": [], # metadata triples
|
||||||
|
"u": "user",
|
||||||
|
"c": "collection"
|
||||||
|
},
|
||||||
|
"e": [ # entities array
|
||||||
|
{
|
||||||
|
"e": {"value": "entity", "is_uri": true},
|
||||||
|
"v": [[0.1, 0.2, 0.3]] # vectors
|
||||||
|
}
|
||||||
|
]
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
## Output Statistics
|
||||||
|
|
||||||
|
The command reports the number of messages exported:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
Got: 150 triple, 75 GE messages.
|
||||||
|
```
|
||||||
|
|
||||||
|
Where:
|
||||||
|
- **triple**: Number of RDF triple message chunks exported
|
||||||
|
- **GE**: Number of graph embedding message chunks exported
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
### Knowledge Core Not Found
|
||||||
|
```bash
|
||||||
|
Exception: Knowledge core 'invalid-core' not found
|
||||||
|
```
|
||||||
|
**Solution**: Check available cores with `tg-show-kg-cores` and verify the core ID.
|
||||||
|
|
||||||
|
### Permission Denied
|
||||||
|
```bash
|
||||||
|
Exception: Access denied to knowledge core
|
||||||
|
```
|
||||||
|
**Solution**: Verify user permissions for the specified knowledge core.
|
||||||
|
|
||||||
|
### File Permission Errors
|
||||||
|
```bash
|
||||||
|
Exception: Permission denied: output.msgpack
|
||||||
|
```
|
||||||
|
**Solution**: Check write permissions for the output directory and filename.
|
||||||
|
|
||||||
|
### Connection Errors
|
||||||
|
```bash
|
||||||
|
Exception: Connection refused
|
||||||
|
```
|
||||||
|
**Solution**: Check the API URL and ensure TrustGraph is running.
|
||||||
|
|
||||||
|
### Disk Space Errors
|
||||||
|
```bash
|
||||||
|
Exception: No space left on device
|
||||||
|
```
|
||||||
|
**Solution**: Free up disk space or use a different output location.
|
||||||
|
|
||||||
|
## File Management
|
||||||
|
|
||||||
|
### Backup Organization
|
||||||
|
```bash
|
||||||
|
# Create organized backup structure
|
||||||
|
mkdir -p backups/{daily,weekly,monthly}
|
||||||
|
|
||||||
|
# Daily backup
|
||||||
|
tg-get-kg-core --id "prod-core" -o "backups/daily/prod-$(date +%Y%m%d).msgpack"
|
||||||
|
|
||||||
|
# Weekly backup
|
||||||
|
tg-get-kg-core --id "prod-core" -o "backups/weekly/prod-week-$(date +%V).msgpack"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Compression
|
||||||
|
```bash
|
||||||
|
# Export and compress for storage
|
||||||
|
tg-get-kg-core --id "large-core" -o large-core.msgpack
|
||||||
|
gzip large-core.msgpack
|
||||||
|
|
||||||
|
# Results in large-core.msgpack.gz
|
||||||
|
```
|
||||||
|
|
||||||
|
## File Verification
|
||||||
|
|
||||||
|
### Check File Size
|
||||||
|
```bash
|
||||||
|
# Export and verify
|
||||||
|
tg-get-kg-core --id "my-core" -o my-core.msgpack
|
||||||
|
ls -lh my-core.msgpack
|
||||||
|
|
||||||
|
# Typical sizes: small cores (KB-MB), large cores (MB-GB)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Validate Export
|
||||||
|
```bash
|
||||||
|
# Test the exported file by importing to different ID
|
||||||
|
tg-put-kg-core --id "test-import" -i my-core.msgpack
|
||||||
|
tg-show-kg-cores | grep "test-import"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
- `TRUSTGRAPH_URL`: Default API URL (automatically converted to WebSocket format)
|
||||||
|
|
||||||
|
## Related Commands
|
||||||
|
|
||||||
|
- [`tg-put-kg-core`](tg-put-kg-core.md) - Import knowledge core from MessagePack file
|
||||||
|
- [`tg-show-kg-cores`](tg-show-kg-cores.md) - List available knowledge cores
|
||||||
|
- [`tg-delete-kg-core`](tg-delete-kg-core.md) - Delete knowledge cores
|
||||||
|
- [`tg-dump-msgpack`](tg-dump-msgpack.md) - Examine MessagePack file contents
|
||||||
|
|
||||||
|
## API Integration
|
||||||
|
|
||||||
|
This command uses the [Knowledge API](../apis/api-knowledge.md) via WebSocket connection with `get-kg-core` operations to retrieve knowledge data.
|
||||||
|
|
||||||
|
## Use Cases
|
||||||
|
|
||||||
|
### Regular Backups
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
# Daily backup script
|
||||||
|
cores=("production-core" "research-core" "customer-data")
|
||||||
|
backup_dir="backups/$(date +%Y%m%d)"
|
||||||
|
mkdir -p "$backup_dir"
|
||||||
|
|
||||||
|
for core in "${cores[@]}"; do
|
||||||
|
echo "Backing up $core..."
|
||||||
|
tg-get-kg-core --id "$core" -o "$backup_dir/$core.msgpack"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Migration Between Environments
|
||||||
|
```bash
|
||||||
|
# Export from development
|
||||||
|
tg-get-kg-core --id "dev-knowledge" -o dev-export.msgpack
|
||||||
|
|
||||||
|
# Import to staging
|
||||||
|
tg-put-kg-core --id "staging-knowledge" -i dev-export.msgpack
|
||||||
|
```
|
||||||
|
|
||||||
|
### Knowledge Core Versioning
|
||||||
|
```bash
|
||||||
|
# Create versioned backups
|
||||||
|
version="v$(date +%Y%m%d)"
|
||||||
|
tg-get-kg-core --id "main-knowledge" -o "knowledge-$version.msgpack"
|
||||||
|
|
||||||
|
# Tag with git or other version control
|
||||||
|
git add "knowledge-$version.msgpack"
|
||||||
|
git commit -m "Knowledge core backup $version"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data Analysis
|
||||||
|
```bash
|
||||||
|
# Export for offline analysis
|
||||||
|
tg-get-kg-core --id "analytics-data" -o analytics.msgpack
|
||||||
|
|
||||||
|
# Process with custom tools
|
||||||
|
python analyze_knowledge.py analytics.msgpack
|
||||||
|
```
|
||||||
|
|
||||||
|
### Disaster Recovery
|
||||||
|
```bash
|
||||||
|
# Create comprehensive backup
|
||||||
|
cores=$(tg-show-kg-cores)
|
||||||
|
backup_date=$(date +%Y%m%d-%H%M%S)
|
||||||
|
backup_dir="disaster-recovery-$backup_date"
|
||||||
|
mkdir -p "$backup_dir"
|
||||||
|
|
||||||
|
for core in $cores; do
|
||||||
|
echo "Backing up $core..."
|
||||||
|
tg-get-kg-core --id "$core" -o "$backup_dir/$core.msgpack"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Create checksum file
|
||||||
|
cd "$backup_dir"
|
||||||
|
sha256sum *.msgpack > checksums.sha256
|
||||||
|
```
|
||||||
|
|
||||||
|
## Automated Backup Strategies
|
||||||
|
|
||||||
|
### Cron Job Setup
|
||||||
|
```bash
|
||||||
|
# Add to crontab for daily backups at 2 AM
|
||||||
|
# 0 2 * * * /path/to/backup-script.sh
|
||||||
|
|
||||||
|
#!/bin/bash
|
||||||
|
# backup-script.sh
|
||||||
|
BACKUP_DIR="/backups/$(date +%Y%m%d)"
|
||||||
|
mkdir -p "$BACKUP_DIR"
|
||||||
|
|
||||||
|
# Backup all cores
|
||||||
|
tg-show-kg-cores | while read core; do
|
||||||
|
tg-get-kg-core --id "$core" -o "$BACKUP_DIR/$core.msgpack"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Cleanup old backups (keep 30 days)
|
||||||
|
find /backups -type d -mtime +30 -exec rm -rf {} \;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Incremental Backups
|
||||||
|
```bash
|
||||||
|
# Compare with previous backup
|
||||||
|
current_cores=$(tg-show-kg-cores | sort)
|
||||||
|
previous_cores=$(cat last-backup-cores.txt 2>/dev/null | sort)
|
||||||
|
|
||||||
|
# Only backup changed cores
|
||||||
|
comm -13 <(echo "$previous_cores") <(echo "$current_cores") | while read core; do
|
||||||
|
tg-get-kg-core --id "$core" -o "incremental/$core.msgpack"
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "$current_cores" > last-backup-cores.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
1. **Regular Backups**: Schedule automated backups of important knowledge cores
|
||||||
|
2. **Organized Storage**: Use dated directories and consistent naming
|
||||||
|
3. **Verification**: Test backup files periodically by importing them
|
||||||
|
4. **Compression**: Compress large backup files to save storage
|
||||||
|
5. **Access Control**: Secure backup files with appropriate permissions
|
||||||
|
6. **Documentation**: Document what each knowledge core contains
|
||||||
|
7. **Retention Policy**: Implement backup retention policies
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Large File Exports
|
||||||
|
```bash
|
||||||
|
# For very large knowledge cores
|
||||||
|
# Monitor progress and disk space
|
||||||
|
df -h . # Check available space
|
||||||
|
tg-get-kg-core --id "huge-core" -o huge-core.msgpack &
|
||||||
|
watch -n 5 'ls -lh huge-core.msgpack' # Monitor file growth
|
||||||
|
```
|
||||||
|
|
||||||
|
### Network Timeouts
|
||||||
|
```bash
|
||||||
|
# If export times out, try smaller cores or check network
|
||||||
|
# Split large cores if possible, or increase timeout settings
|
||||||
|
```
|
||||||
|
|
||||||
|
### Corrupted Exports
|
||||||
|
```bash
|
||||||
|
# Verify file integrity
|
||||||
|
file my-core.msgpack # Should show "data"
|
||||||
|
python -c "import msgpack; msgpack.unpack(open('my-core.msgpack', 'rb'))"
|
||||||
|
```
|
||||||
494
docs/cli/tg-graph-to-turtle.md
Normal file
494
docs/cli/tg-graph-to-turtle.md
Normal file
|
|
@ -0,0 +1,494 @@
|
||||||
|
# tg-graph-to-turtle
|
||||||
|
|
||||||
|
Exports knowledge graph data to Turtle (TTL) format for backup, analysis, or migration.
|
||||||
|
|
||||||
|
## Synopsis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tg-graph-to-turtle [options]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Description
|
||||||
|
|
||||||
|
The `tg-graph-to-turtle` command connects to TrustGraph's triple query service and exports all graph triples in Turtle format. This is useful for creating backups, analyzing graph structure, migrating data, or integrating with external RDF tools.
|
||||||
|
|
||||||
|
The command queries up to 10,000 triples and outputs them in standard Turtle format to stdout, while also saving to an `output.ttl` file.
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
### Optional Arguments
|
||||||
|
|
||||||
|
- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
|
||||||
|
- `-f, --flow-id ID`: Flow instance ID to use (default: `default`)
|
||||||
|
- `-U, --user USER`: User ID for data scope (default: `trustgraph`)
|
||||||
|
- `-C, --collection COLLECTION`: Collection to export (default: `default`)
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
### Basic Export
|
||||||
|
```bash
|
||||||
|
tg-graph-to-turtle
|
||||||
|
```
|
||||||
|
|
||||||
|
### Export to File
|
||||||
|
```bash
|
||||||
|
tg-graph-to-turtle > knowledge-graph.ttl
|
||||||
|
```
|
||||||
|
|
||||||
|
### Export Specific Collection
|
||||||
|
```bash
|
||||||
|
tg-graph-to-turtle -C "research-data" > research-graph.ttl
|
||||||
|
```
|
||||||
|
|
||||||
|
### Export with Custom Flow
|
||||||
|
```bash
|
||||||
|
tg-graph-to-turtle -f "production-flow" -U "admin" > production-graph.ttl
|
||||||
|
```
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
The command generates Turtle format with proper RDF syntax:
|
||||||
|
|
||||||
|
```turtle
|
||||||
|
@prefix ns1: <http://example.org/> .
|
||||||
|
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
|
||||||
|
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
|
||||||
|
|
||||||
|
ns1:Person rdf:type rdfs:Class .
|
||||||
|
ns1:john rdf:type ns1:Person ;
|
||||||
|
ns1:name "John Doe" ;
|
||||||
|
ns1:age "30" .
|
||||||
|
ns1:jane rdf:type ns1:Person ;
|
||||||
|
ns1:name "Jane Smith" ;
|
||||||
|
ns1:department "Engineering" .
|
||||||
|
```
|
||||||
|
|
||||||
|
### Output Destinations
|
||||||
|
|
||||||
|
1. **stdout**: Standard output for piping or display
|
||||||
|
2. **output.ttl**: Automatically created file in current directory
|
||||||
|
|
||||||
|
## Use Cases
|
||||||
|
|
||||||
|
### Data Backup
|
||||||
|
```bash
|
||||||
|
# Create timestamped backups
|
||||||
|
timestamp=$(date +%Y%m%d_%H%M%S)
|
||||||
|
tg-graph-to-turtle > "backup_${timestamp}.ttl"
|
||||||
|
|
||||||
|
# Backup specific collections
|
||||||
|
collections=("research" "products" "customers")
|
||||||
|
for collection in "${collections[@]}"; do
|
||||||
|
tg-graph-to-turtle -C "$collection" > "backup_${collection}_${timestamp}.ttl"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data Migration
|
||||||
|
```bash
|
||||||
|
# Export from source environment
|
||||||
|
tg-graph-to-turtle -u "http://source:8088/" > source-data.ttl
|
||||||
|
|
||||||
|
# Import to target environment
|
||||||
|
tg-load-turtle -i "migration-$(date +%Y%m%d)" \
|
||||||
|
-u "ws://target:8088/" \
|
||||||
|
source-data.ttl
|
||||||
|
```
|
||||||
|
|
||||||
|
### Graph Analysis
|
||||||
|
```bash
|
||||||
|
# Export for analysis
|
||||||
|
tg-graph-to-turtle > analysis-data.ttl
|
||||||
|
|
||||||
|
# Analyze with external tools
|
||||||
|
rapper -i turtle -o ntriples analysis-data.ttl | wc -l # Count triples
|
||||||
|
grep -c "rdf:type" analysis-data.ttl # Count type assertions
|
||||||
|
```
|
||||||
|
|
||||||
|
### Integration with External Tools
|
||||||
|
```bash
|
||||||
|
# Export for Apache Jena
|
||||||
|
tg-graph-to-turtle > jena-input.ttl
|
||||||
|
tdb2.tdbloader --loc=tdb-database jena-input.ttl
|
||||||
|
|
||||||
|
# Export for Virtuoso
|
||||||
|
tg-graph-to-turtle > virtuoso-data.ttl
|
||||||
|
isql-v -U dba -P password < load-script.sql
|
||||||
|
```
|
||||||
|
|
||||||
|
## Advanced Usage
|
||||||
|
|
||||||
|
### Incremental Exports
|
||||||
|
```bash
|
||||||
|
# Export with timestamps for incremental backups
|
||||||
|
last_export_file="last_export_timestamp.txt"
|
||||||
|
current_time=$(date +%Y%m%d_%H%M%S)
|
||||||
|
|
||||||
|
if [ -f "$last_export_file" ]; then
|
||||||
|
last_export=$(cat "$last_export_file")
|
||||||
|
echo "Last export: $last_export"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Current export: $current_time"
|
||||||
|
tg-graph-to-turtle > "incremental_${current_time}.ttl"
|
||||||
|
echo "$current_time" > "$last_export_file"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Multi-Collection Export
|
||||||
|
```bash
|
||||||
|
# Export all collections to separate files
|
||||||
|
export_all_collections() {
|
||||||
|
local output_dir="graph_exports_$(date +%Y%m%d)"
|
||||||
|
mkdir -p "$output_dir"
|
||||||
|
|
||||||
|
echo "Exporting all collections to $output_dir"
|
||||||
|
|
||||||
|
# Get list of collections (this would need to be implemented)
|
||||||
|
# For now, use known collections
|
||||||
|
collections=("default" "research" "products" "documents")
|
||||||
|
|
||||||
|
for collection in "${collections[@]}"; do
|
||||||
|
echo "Exporting collection: $collection"
|
||||||
|
tg-graph-to-turtle -C "$collection" > "$output_dir/${collection}.ttl"
|
||||||
|
|
||||||
|
# Verify export
|
||||||
|
if [ -s "$output_dir/${collection}.ttl" ]; then
|
||||||
|
triple_count=$(grep -c "\." "$output_dir/${collection}.ttl")
|
||||||
|
echo " Exported $triple_count triples"
|
||||||
|
else
|
||||||
|
echo " No data exported"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
export_all_collections
|
||||||
|
```
|
||||||
|
|
||||||
|
### Filtered Export
|
||||||
|
```bash
|
||||||
|
# Export specific types of triples
|
||||||
|
export_filtered() {
|
||||||
|
local filter_type="$1"
|
||||||
|
local output_file="$2"
|
||||||
|
|
||||||
|
echo "Exporting $filter_type triples to $output_file"
|
||||||
|
|
||||||
|
# Export all data first
|
||||||
|
tg-graph-to-turtle > temp_full_export.ttl
|
||||||
|
|
||||||
|
# Filter based on type
|
||||||
|
case "$filter_type" in
|
||||||
|
"classes")
|
||||||
|
grep "rdf:type.*Class" temp_full_export.ttl > "$output_file"
|
||||||
|
;;
|
||||||
|
"instances")
|
||||||
|
grep -v "rdf:type.*Class" temp_full_export.ttl > "$output_file"
|
||||||
|
;;
|
||||||
|
"properties")
|
||||||
|
grep "rdf:type.*Property" temp_full_export.ttl > "$output_file"
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "Unknown filter type: $filter_type"
|
||||||
|
return 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
rm temp_full_export.ttl
|
||||||
|
}
|
||||||
|
|
||||||
|
# Usage
|
||||||
|
export_filtered "classes" "schema-classes.ttl"
|
||||||
|
export_filtered "instances" "instance-data.ttl"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Compression and Packaging
|
||||||
|
```bash
|
||||||
|
# Export and compress
|
||||||
|
export_compressed() {
|
||||||
|
local collection="$1"
|
||||||
|
local timestamp=$(date +%Y%m%d_%H%M%S)
|
||||||
|
local filename="${collection}_${timestamp}"
|
||||||
|
|
||||||
|
echo "Exporting and compressing collection: $collection"
|
||||||
|
|
||||||
|
# Export to temporary file
|
||||||
|
tg-graph-to-turtle -C "$collection" > "${filename}.ttl"
|
||||||
|
|
||||||
|
# Compress
|
||||||
|
gzip "${filename}.ttl"
|
||||||
|
|
||||||
|
# Create metadata
|
||||||
|
cat > "${filename}.meta" << EOF
|
||||||
|
Collection: $collection
|
||||||
|
Export Date: $(date)
|
||||||
|
Compressed Size: $(stat -c%s "${filename}.ttl.gz") bytes
|
||||||
|
MD5: $(md5sum "${filename}.ttl.gz" | cut -d' ' -f1)
|
||||||
|
EOF
|
||||||
|
|
||||||
|
echo "Export complete: ${filename}.ttl.gz"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Export multiple collections compressed
|
||||||
|
collections=("research" "products" "customers")
|
||||||
|
for collection in "${collections[@]}"; do
|
||||||
|
export_compressed "$collection"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Validation and Quality Checks
|
||||||
|
```bash
|
||||||
|
# Export with validation
|
||||||
|
export_with_validation() {
|
||||||
|
local output_file="$1"
|
||||||
|
|
||||||
|
echo "Exporting with validation to $output_file"
|
||||||
|
|
||||||
|
# Export
|
||||||
|
tg-graph-to-turtle > "$output_file"
|
||||||
|
|
||||||
|
# Validate Turtle syntax
|
||||||
|
if rapper -q -i turtle "$output_file" > /dev/null 2>&1; then
|
||||||
|
echo "✓ Valid Turtle syntax"
|
||||||
|
else
|
||||||
|
echo "✗ Invalid Turtle syntax"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Count triples
|
||||||
|
triple_count=$(rapper -i turtle -c "$output_file" 2>/dev/null)
|
||||||
|
echo "Total triples: $triple_count"
|
||||||
|
|
||||||
|
# Check for common issues
|
||||||
|
if grep -q "^@prefix" "$output_file"; then
|
||||||
|
echo "✓ Prefixes found"
|
||||||
|
else
|
||||||
|
echo "⚠ No prefixes found"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check for URIs with spaces (malformed)
|
||||||
|
malformed_uris=$(grep -c " " "$output_file" || echo "0")
|
||||||
|
if [ "$malformed_uris" -gt 0 ]; then
|
||||||
|
echo "⚠ Found $malformed_uris lines with spaces (potential malformed URIs)"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Validate export
|
||||||
|
export_with_validation "validated-export.ttl"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Performance Optimization
|
||||||
|
|
||||||
|
### Streaming Export
|
||||||
|
```bash
|
||||||
|
# Handle large datasets with streaming
|
||||||
|
stream_export() {
|
||||||
|
local collection="$1"
|
||||||
|
local chunk_size="$2"
|
||||||
|
local output_prefix="$3"
|
||||||
|
|
||||||
|
echo "Streaming export of collection: $collection"
|
||||||
|
|
||||||
|
# Export to temporary file
|
||||||
|
tg-graph-to-turtle -C "$collection" > temp_export.ttl
|
||||||
|
|
||||||
|
# Split into chunks
|
||||||
|
split -l "$chunk_size" temp_export.ttl "${output_prefix}_"
|
||||||
|
|
||||||
|
# Add .ttl extension and validate each chunk
|
||||||
|
for chunk in ${output_prefix}_*; do
|
||||||
|
mv "$chunk" "$chunk.ttl"
|
||||||
|
|
||||||
|
# Validate chunk
|
||||||
|
if rapper -q -i turtle "$chunk.ttl" > /dev/null 2>&1; then
|
||||||
|
echo "✓ Valid chunk: $chunk.ttl"
|
||||||
|
else
|
||||||
|
echo "✗ Invalid chunk: $chunk.ttl"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
rm temp_export.ttl
|
||||||
|
}
|
||||||
|
|
||||||
|
# Stream large collection
|
||||||
|
stream_export "large-collection" 1000 "chunk"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Parallel Processing
|
||||||
|
```bash
|
||||||
|
# Export multiple collections in parallel
|
||||||
|
parallel_export() {
|
||||||
|
local collections=("$@")
|
||||||
|
local timestamp=$(date +%Y%m%d_%H%M%S)
|
||||||
|
|
||||||
|
echo "Exporting ${#collections[@]} collections in parallel"
|
||||||
|
|
||||||
|
for collection in "${collections[@]}"; do
|
||||||
|
(
|
||||||
|
echo "Exporting $collection..."
|
||||||
|
tg-graph-to-turtle -C "$collection" > "${collection}_${timestamp}.ttl"
|
||||||
|
echo "✓ Completed: $collection"
|
||||||
|
) &
|
||||||
|
done
|
||||||
|
|
||||||
|
wait
|
||||||
|
echo "All exports completed"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Export collections in parallel
|
||||||
|
parallel_export "research" "products" "customers" "documents"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Integration Scripts
|
||||||
|
|
||||||
|
### Automated Backup System
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
# automated-backup.sh
|
||||||
|
backup_dir="graph_backups"
|
||||||
|
retention_days=30
|
||||||
|
|
||||||
|
echo "Starting automated graph backup..."
|
||||||
|
|
||||||
|
# Create backup directory
|
||||||
|
mkdir -p "$backup_dir"
|
||||||
|
|
||||||
|
# Export with timestamp
|
||||||
|
timestamp=$(date +%Y%m%d_%H%M%S)
|
||||||
|
backup_file="$backup_dir/graph_backup_${timestamp}.ttl"
|
||||||
|
|
||||||
|
echo "Exporting to: $backup_file"
|
||||||
|
tg-graph-to-turtle > "$backup_file"
|
||||||
|
|
||||||
|
# Compress
|
||||||
|
gzip "$backup_file"
|
||||||
|
echo "Compressed: ${backup_file}.gz"
|
||||||
|
|
||||||
|
# Clean old backups
|
||||||
|
find "$backup_dir" -name "*.ttl.gz" -mtime +$retention_days -delete
|
||||||
|
echo "Cleaned backups older than $retention_days days"
|
||||||
|
|
||||||
|
# Verify backup
|
||||||
|
if [ -f "${backup_file}.gz" ]; then
|
||||||
|
size=$(stat -c%s "${backup_file}.gz")
|
||||||
|
echo "Backup completed: ${size} bytes"
|
||||||
|
else
|
||||||
|
echo "Backup failed!"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data Sync Script
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
# sync-graphs.sh
|
||||||
|
source_url="$1"
|
||||||
|
target_url="$2"
|
||||||
|
collection="$3"
|
||||||
|
|
||||||
|
if [ -z "$source_url" ] || [ -z "$target_url" ] || [ -z "$collection" ]; then
|
||||||
|
echo "Usage: $0 <source-url> <target-url> <collection>"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Syncing collection '$collection' from $source_url to $target_url"
|
||||||
|
|
||||||
|
# Export from source
|
||||||
|
temp_file="sync_temp_$(date +%s).ttl"
|
||||||
|
tg-graph-to-turtle -u "$source_url" -C "$collection" > "$temp_file"
|
||||||
|
|
||||||
|
# Validate export
|
||||||
|
if [ ! -s "$temp_file" ]; then
|
||||||
|
echo "No data exported from source"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Load to target
|
||||||
|
doc_id="sync-$(date +%Y%m%d-%H%M%S)"
|
||||||
|
if tg-load-turtle -i "$doc_id" -u "$target_url" -C "$collection" "$temp_file"; then
|
||||||
|
echo "Sync completed successfully"
|
||||||
|
else
|
||||||
|
echo "Sync failed"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Cleanup
|
||||||
|
rm "$temp_file"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
### Connection Issues
|
||||||
|
```bash
|
||||||
|
Exception: Connection refused
|
||||||
|
```
|
||||||
|
**Solution**: Check API URL and ensure TrustGraph is running.
|
||||||
|
|
||||||
|
### Flow Not Found
|
||||||
|
```bash
|
||||||
|
Exception: Flow instance not found
|
||||||
|
```
|
||||||
|
**Solution**: Verify flow ID with `tg-show-flows`.
|
||||||
|
|
||||||
|
### Permission Errors
|
||||||
|
```bash
|
||||||
|
Exception: Access denied
|
||||||
|
```
|
||||||
|
**Solution**: Check user permissions for the specified collection.
|
||||||
|
|
||||||
|
### Empty Output
|
||||||
|
```bash
|
||||||
|
# No triples exported
|
||||||
|
```
|
||||||
|
**Solution**: Verify collection contains data and user has access.
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
- `TRUSTGRAPH_URL`: Default API URL
|
||||||
|
|
||||||
|
## Related Commands
|
||||||
|
|
||||||
|
- [`tg-load-turtle`](tg-load-turtle.md) - Import Turtle files
|
||||||
|
- [`tg-triples-query`](tg-triples-query.md) - Query graph triples
|
||||||
|
- [`tg-show-flows`](tg-show-flows.md) - List available flows
|
||||||
|
- [`tg-get-kg-core`](tg-get-kg-core.md) - Export knowledge cores
|
||||||
|
|
||||||
|
## API Integration
|
||||||
|
|
||||||
|
This command uses the [Triples Query API](../apis/api-triples-query.md) to retrieve graph data and convert it to Turtle format.
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
1. **Regular Backups**: Schedule regular exports for data protection
|
||||||
|
2. **Validation**: Always validate exported Turtle files
|
||||||
|
3. **Compression**: Compress large exports for storage efficiency
|
||||||
|
4. **Monitoring**: Track export sizes and success rates
|
||||||
|
5. **Documentation**: Document export procedures and retention policies
|
||||||
|
6. **Security**: Ensure sensitive data is properly protected in exports
|
||||||
|
7. **Version Control**: Consider versioning exported schemas
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Large Dataset Issues
|
||||||
|
```bash
|
||||||
|
# Check query limits
|
||||||
|
grep -c "\." output.ttl # Count exported triples
|
||||||
|
# Default limit is 10,000 triples
|
||||||
|
|
||||||
|
# For larger datasets, consider using tg-get-kg-core
|
||||||
|
tg-get-kg-core -n "collection-name" > large-export.msgpack
|
||||||
|
```
|
||||||
|
|
||||||
|
### Malformed URIs
|
||||||
|
```bash
|
||||||
|
# Check for URIs with spaces
|
||||||
|
grep " " output.ttl | head -5
|
||||||
|
|
||||||
|
# Clean URIs if needed
|
||||||
|
sed 's/ /%20/g' output.ttl > cleaned-output.ttl
|
||||||
|
```
|
||||||
|
|
||||||
|
### Memory Issues
|
||||||
|
```bash
|
||||||
|
# Monitor memory usage during export
|
||||||
|
free -h
|
||||||
|
# Consider splitting exports for large datasets
|
||||||
|
```
|
||||||
438
docs/cli/tg-invoke-document-rag.md
Normal file
438
docs/cli/tg-invoke-document-rag.md
Normal file
|
|
@ -0,0 +1,438 @@
|
||||||
|
# tg-invoke-document-rag
|
||||||
|
|
||||||
|
Invokes the DocumentRAG service to answer questions using document context and retrieval-augmented generation.
|
||||||
|
|
||||||
|
## Synopsis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tg-invoke-document-rag -q QUESTION [options]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Description
|
||||||
|
|
||||||
|
The `tg-invoke-document-rag` command uses TrustGraph's DocumentRAG service to answer questions by retrieving relevant document context and generating responses using large language models. This implements a Retrieval-Augmented Generation (RAG) approach that grounds AI responses in your document corpus.
|
||||||
|
|
||||||
|
The service searches through indexed documents to find relevant context, then uses that context to generate accurate, source-backed answers to questions.
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
### Required Arguments
|
||||||
|
|
||||||
|
- `-q, --question QUESTION`: The question to answer
|
||||||
|
|
||||||
|
### Optional Arguments
|
||||||
|
|
||||||
|
- `-u, --url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
|
||||||
|
- `-f, --flow-id ID`: Flow instance ID to use (default: `default`)
|
||||||
|
- `-U, --user USER`: User ID for context isolation (default: `trustgraph`)
|
||||||
|
- `-C, --collection COLLECTION`: Document collection to search (default: `default`)
|
||||||
|
- `-d, --doc-limit LIMIT`: Maximum number of documents to retrieve (default: `10`)
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
### Basic Question Answering
|
||||||
|
```bash
|
||||||
|
tg-invoke-document-rag -q "What is the company's return policy?"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Question with Custom Parameters
|
||||||
|
```bash
|
||||||
|
tg-invoke-document-rag \
|
||||||
|
-q "How do I configure SSL certificates?" \
|
||||||
|
-f "production-docs" \
|
||||||
|
-U "admin" \
|
||||||
|
-C "technical-docs" \
|
||||||
|
-d 5
|
||||||
|
```
|
||||||
|
|
||||||
|
### Complex Technical Questions
|
||||||
|
```bash
|
||||||
|
tg-invoke-document-rag \
|
||||||
|
-q "What are the performance benchmarks for the new API endpoints?" \
|
||||||
|
-f "api-docs" \
|
||||||
|
-C "performance-reports"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Multi-domain Questions
|
||||||
|
```bash
|
||||||
|
# Legal documents
|
||||||
|
tg-invoke-document-rag -q "What are the privacy policy requirements?" -C "legal-docs"
|
||||||
|
|
||||||
|
# Technical documentation
|
||||||
|
tg-invoke-document-rag -q "How do I troubleshoot connection timeouts?" -C "tech-docs"
|
||||||
|
|
||||||
|
# Marketing materials
|
||||||
|
tg-invoke-document-rag -q "What are our key product differentiators?" -C "marketing"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
The command returns a structured response with:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"question": "What is the company's return policy?",
|
||||||
|
"answer": "Based on the company policy documents, customers can return items within 30 days of purchase for a full refund. Items must be in original condition with receipt. Digital products are non-refundable except in cases of technical defects.",
|
||||||
|
"sources": [
|
||||||
|
{
|
||||||
|
"document": "customer-service-policy.pdf",
|
||||||
|
"relevance": 0.92,
|
||||||
|
"section": "Returns and Refunds"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"document": "terms-of-service.pdf",
|
||||||
|
"relevance": 0.85,
|
||||||
|
"section": "Customer Rights"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"confidence": 0.89
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Use Cases
|
||||||
|
|
||||||
|
### Customer Support
|
||||||
|
```bash
|
||||||
|
# Answer common customer questions
|
||||||
|
tg-invoke-document-rag -q "How do I reset my password?" -C "support-docs"
|
||||||
|
|
||||||
|
# Product information queries
|
||||||
|
tg-invoke-document-rag -q "What are the system requirements?" -C "product-specs"
|
||||||
|
|
||||||
|
# Troubleshooting assistance
|
||||||
|
tg-invoke-document-rag -q "Why is my upload failing?" -C "troubleshooting"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Technical Documentation
|
||||||
|
```bash
|
||||||
|
# API documentation queries
|
||||||
|
tg-invoke-document-rag -q "How do I authenticate with the REST API?" -C "api-docs"
|
||||||
|
|
||||||
|
# Configuration questions
|
||||||
|
tg-invoke-document-rag -q "What are the required environment variables?" -C "config-docs"
|
||||||
|
|
||||||
|
# Architecture information
|
||||||
|
tg-invoke-document-rag -q "How does the caching system work?" -C "architecture"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Research and Analysis
|
||||||
|
```bash
|
||||||
|
# Research queries
|
||||||
|
tg-invoke-document-rag -q "What are the latest industry trends?" -C "research-reports"
|
||||||
|
|
||||||
|
# Compliance questions
|
||||||
|
tg-invoke-document-rag -q "What are the GDPR requirements?" -C "compliance-docs"
|
||||||
|
|
||||||
|
# Best practices
|
||||||
|
tg-invoke-document-rag -q "What are the security best practices?" -C "security-guidelines"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Interactive Q&A Sessions
|
||||||
|
```bash
|
||||||
|
# Batch questions for analysis
|
||||||
|
questions=(
|
||||||
|
"What is our market share?"
|
||||||
|
"How do we compare to competitors?"
|
||||||
|
"What are the growth projections?"
|
||||||
|
)
|
||||||
|
|
||||||
|
for question in "${questions[@]}"; do
|
||||||
|
echo "Question: $question"
|
||||||
|
tg-invoke-document-rag -q "$question" -C "business-reports"
|
||||||
|
echo "---"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
## Document Context and Retrieval
|
||||||
|
|
||||||
|
### Document Limit Tuning
|
||||||
|
```bash
|
||||||
|
# Few documents for focused answers
|
||||||
|
tg-invoke-document-rag -q "What is the API rate limit?" -d 3
|
||||||
|
|
||||||
|
# Many documents for comprehensive analysis
|
||||||
|
tg-invoke-document-rag -q "What are all the security measures?" -d 20
|
||||||
|
```
|
||||||
|
|
||||||
|
### Collection-Specific Queries
|
||||||
|
```bash
|
||||||
|
# Target specific document collections
|
||||||
|
tg-invoke-document-rag -q "What is the deployment process?" -C "devops-docs"
|
||||||
|
tg-invoke-document-rag -q "What are the testing standards?" -C "qa-docs"
|
||||||
|
tg-invoke-document-rag -q "What is the coding style guide?" -C "dev-standards"
|
||||||
|
```
|
||||||
|
|
||||||
|
### User Context Isolation
|
||||||
|
```bash
|
||||||
|
# Department-specific contexts
|
||||||
|
tg-invoke-document-rag -q "What is the budget allocation?" -U "finance" -C "finance-docs"
|
||||||
|
tg-invoke-document-rag -q "What are the hiring requirements?" -U "hr" -C "hr-docs"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
### Question Required
|
||||||
|
```bash
|
||||||
|
Exception: Question is required
|
||||||
|
```
|
||||||
|
**Solution**: Provide a question with the `-q` option.
|
||||||
|
|
||||||
|
### Flow Not Found
|
||||||
|
```bash
|
||||||
|
Exception: Flow instance 'nonexistent-flow' not found
|
||||||
|
```
|
||||||
|
**Solution**: Verify the flow ID exists with `tg-show-flows`.
|
||||||
|
|
||||||
|
### Collection Not Found
|
||||||
|
```bash
|
||||||
|
Exception: Collection 'invalid-collection' not found
|
||||||
|
```
|
||||||
|
**Solution**: Check available collections with document library commands.
|
||||||
|
|
||||||
|
### No Documents Found
|
||||||
|
```bash
|
||||||
|
Exception: No relevant documents found for query
|
||||||
|
```
|
||||||
|
**Solution**: Verify documents are indexed and collection contains relevant content.
|
||||||
|
|
||||||
|
### API Connection Issues
|
||||||
|
```bash
|
||||||
|
Exception: Connection refused
|
||||||
|
```
|
||||||
|
**Solution**: Check API URL and ensure TrustGraph services are running.
|
||||||
|
|
||||||
|
## Advanced Usage
|
||||||
|
|
||||||
|
### Batch Processing
|
||||||
|
```bash
|
||||||
|
# Process questions from file
|
||||||
|
while IFS= read -r question; do
|
||||||
|
if [ -n "$question" ]; then
|
||||||
|
echo "Processing: $question"
|
||||||
|
tg-invoke-document-rag -q "$question" -C "knowledge-base" > "answer-$(date +%s).json"
|
||||||
|
fi
|
||||||
|
done < questions.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
### Question Analysis Pipeline
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
# analyze-questions.sh
|
||||||
|
questions_file="$1"
|
||||||
|
collection="$2"
|
||||||
|
|
||||||
|
if [ -z "$questions_file" ] || [ -z "$collection" ]; then
|
||||||
|
echo "Usage: $0 <questions-file> <collection>"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Question Analysis Report - $(date)"
|
||||||
|
echo "Collection: $collection"
|
||||||
|
echo "=================================="
|
||||||
|
|
||||||
|
question_num=1
|
||||||
|
while IFS= read -r question; do
|
||||||
|
if [ -n "$question" ]; then
|
||||||
|
echo -e "\n$question_num. $question"
|
||||||
|
echo "$(printf '=%.0s' {1..50})"
|
||||||
|
|
||||||
|
# Get answer
|
||||||
|
answer=$(tg-invoke-document-rag -q "$question" -C "$collection" 2>/dev/null)
|
||||||
|
|
||||||
|
if [ $? -eq 0 ]; then
|
||||||
|
echo "$answer" | jq -r '.answer' 2>/dev/null || echo "$answer"
|
||||||
|
|
||||||
|
# Extract sources if available
|
||||||
|
sources=$(echo "$answer" | jq -r '.sources[]?.document' 2>/dev/null)
|
||||||
|
if [ -n "$sources" ]; then
|
||||||
|
echo -e "\nSources:"
|
||||||
|
echo "$sources" | sed 's/^/ - /'
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo "ERROR: Could not get answer"
|
||||||
|
fi
|
||||||
|
|
||||||
|
question_num=$((question_num + 1))
|
||||||
|
fi
|
||||||
|
done < "$questions_file"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Quality Assessment
|
||||||
|
```bash
|
||||||
|
# Assess answer quality with multiple document limits
|
||||||
|
question="What are the security protocols?"
|
||||||
|
collection="security-docs"
|
||||||
|
|
||||||
|
echo "Answer Quality Assessment"
|
||||||
|
echo "Question: $question"
|
||||||
|
echo "========================"
|
||||||
|
|
||||||
|
for limit in 3 5 10 15 20; do
|
||||||
|
echo -e "\nDocument limit: $limit"
|
||||||
|
echo "$(printf '-%.0s' {1..30})"
|
||||||
|
|
||||||
|
answer=$(tg-invoke-document-rag -q "$question" -C "$collection" -d $limit 2>/dev/null)
|
||||||
|
|
||||||
|
if [ $? -eq 0 ]; then
|
||||||
|
# Get answer length and source count
|
||||||
|
answer_length=$(echo "$answer" | jq -r '.answer' 2>/dev/null | wc -c)
|
||||||
|
source_count=$(echo "$answer" | jq -r '.sources | length' 2>/dev/null)
|
||||||
|
confidence=$(echo "$answer" | jq -r '.confidence' 2>/dev/null)
|
||||||
|
|
||||||
|
echo "Answer length: $answer_length characters"
|
||||||
|
echo "Source count: $source_count"
|
||||||
|
echo "Confidence: $confidence"
|
||||||
|
else
|
||||||
|
echo "ERROR: Failed to get answer"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Interactive Q&A Interface
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
# interactive-rag.sh
|
||||||
|
collection="${1:-default}"
|
||||||
|
flow_id="${2:-default}"
|
||||||
|
|
||||||
|
echo "Interactive Document RAG Interface"
|
||||||
|
echo "Collection: $collection"
|
||||||
|
echo "Flow ID: $flow_id"
|
||||||
|
echo "Type 'quit' to exit"
|
||||||
|
echo "=================================="
|
||||||
|
|
||||||
|
while true; do
|
||||||
|
echo -n "Question: "
|
||||||
|
read -r question
|
||||||
|
|
||||||
|
if [ "$question" = "quit" ]; then
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ -n "$question" ]; then
|
||||||
|
echo "Thinking..."
|
||||||
|
answer=$(tg-invoke-document-rag -q "$question" -C "$collection" -f "$flow_id" 2>/dev/null)
|
||||||
|
|
||||||
|
if [ $? -eq 0 ]; then
|
||||||
|
echo "Answer:"
|
||||||
|
echo "$answer" | jq -r '.answer' 2>/dev/null || echo "$answer"
|
||||||
|
|
||||||
|
# Show sources if available
|
||||||
|
sources=$(echo "$answer" | jq -r '.sources[]?.document' 2>/dev/null)
|
||||||
|
if [ -n "$sources" ]; then
|
||||||
|
echo -e "\nSources:"
|
||||||
|
echo "$sources" | sed 's/^/ - /'
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo "Sorry, I couldn't answer that question."
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo -e "\n$(printf '=%.0s' {1..50})"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "Goodbye!"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Performance Optimization
|
||||||
|
|
||||||
|
### Document Limit Optimization
|
||||||
|
```bash
|
||||||
|
# Test different document limits for performance
|
||||||
|
question="What is the system architecture?"
|
||||||
|
collection="tech-docs"
|
||||||
|
|
||||||
|
for limit in 3 5 10 15 20; do
|
||||||
|
echo "Testing document limit: $limit"
|
||||||
|
start_time=$(date +%s%N)
|
||||||
|
|
||||||
|
tg-invoke-document-rag -q "$question" -C "$collection" -d $limit > /dev/null 2>&1
|
||||||
|
|
||||||
|
end_time=$(date +%s%N)
|
||||||
|
duration=$(( (end_time - start_time) / 1000000 )) # Convert to milliseconds
|
||||||
|
|
||||||
|
echo " Duration: ${duration}ms"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Caching Strategy
|
||||||
|
```bash
|
||||||
|
# Cache frequently asked questions
|
||||||
|
cache_dir="rag-cache"
|
||||||
|
mkdir -p "$cache_dir"
|
||||||
|
|
||||||
|
ask_question() {
|
||||||
|
local question="$1"
|
||||||
|
local collection="$2"
|
||||||
|
local cache_key=$(echo "$question-$collection" | md5sum | cut -d' ' -f1)
|
||||||
|
local cache_file="$cache_dir/$cache_key.json"
|
||||||
|
|
||||||
|
if [ -f "$cache_file" ]; then
|
||||||
|
echo "Cache hit for: $question"
|
||||||
|
cat "$cache_file"
|
||||||
|
else
|
||||||
|
echo "Cache miss, querying: $question"
|
||||||
|
tg-invoke-document-rag -q "$question" -C "$collection" | tee "$cache_file"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Use cached queries
|
||||||
|
ask_question "What is the API documentation?" "tech-docs"
|
||||||
|
ask_question "What are the system requirements?" "spec-docs"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
- `TRUSTGRAPH_URL`: Default API URL
|
||||||
|
|
||||||
|
## Related Commands
|
||||||
|
|
||||||
|
- [`tg-load-pdf`](tg-load-pdf.md) - Load PDF documents for RAG
|
||||||
|
- [`tg-show-library-documents`](tg-show-library-documents.md) - List available documents
|
||||||
|
- [`tg-invoke-prompt`](tg-invoke-prompt.md) - Direct prompt invocation without RAG
|
||||||
|
- [`tg-start-flow`](tg-start-flow.md) - Start flows for document processing
|
||||||
|
- [`tg-show-flows`](tg-show-flows.md) - List active flow instances
|
||||||
|
|
||||||
|
## API Integration
|
||||||
|
|
||||||
|
This command uses the [DocumentRAG API](../apis/api-document-rag.md) to perform retrieval-augmented generation using the document corpus.
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
1. **Question Formulation**: Use specific, well-formed questions for better results
|
||||||
|
2. **Collection Organization**: Organize documents into logical collections
|
||||||
|
3. **Document Limits**: Balance accuracy with performance using appropriate document limits
|
||||||
|
4. **User Context**: Use user isolation for sensitive or department-specific queries
|
||||||
|
5. **Source Verification**: Always check source documents for critical information
|
||||||
|
6. **Caching**: Implement caching for frequently asked questions
|
||||||
|
7. **Quality Assessment**: Regularly evaluate answer quality and adjust parameters
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Poor Answer Quality
|
||||||
|
```bash
|
||||||
|
# Try different document limits
|
||||||
|
tg-invoke-document-rag -q "your question" -d 5 # Fewer documents
|
||||||
|
tg-invoke-document-rag -q "your question" -d 15 # More documents
|
||||||
|
|
||||||
|
# Check document collection
|
||||||
|
tg-show-library-documents -C "your-collection"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Slow Response Times
|
||||||
|
```bash
|
||||||
|
# Reduce document limit
|
||||||
|
tg-invoke-document-rag -q "your question" -d 3
|
||||||
|
|
||||||
|
# Check flow performance
|
||||||
|
tg-show-flows | grep "document-rag"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Missing Context
|
||||||
|
```bash
|
||||||
|
# Verify documents are indexed
|
||||||
|
tg-show-library-documents -C "your-collection"
|
||||||
|
|
||||||
|
# Check if collection exists
|
||||||
|
tg-show-library-documents | grep "your-collection"
|
||||||
|
```
|
||||||
430
docs/cli/tg-invoke-prompt.md
Normal file
430
docs/cli/tg-invoke-prompt.md
Normal file
|
|
@ -0,0 +1,430 @@
|
||||||
|
# tg-invoke-prompt
|
||||||
|
|
||||||
|
Invokes the LLM prompt service using predefined prompt templates with variable substitution.
|
||||||
|
|
||||||
|
## Synopsis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tg-invoke-prompt [options] template-id [variable=value ...]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Description
|
||||||
|
|
||||||
|
The `tg-invoke-prompt` command invokes TrustGraph's LLM prompt service using predefined prompt templates. Templates contain placeholder variables in the format `{{variable}}` that are replaced with values provided on the command line.
|
||||||
|
|
||||||
|
This provides a structured way to interact with language models using consistent, reusable prompt templates for specific tasks like question answering, text extraction, analysis, and more.
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
### Required Arguments
|
||||||
|
|
||||||
|
- `template-id`: Prompt template identifier (e.g., `question`, `extract-definitions`, `summarize`)
|
||||||
|
|
||||||
|
### Optional Arguments
|
||||||
|
|
||||||
|
- `-u, --url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
|
||||||
|
- `-f, --flow-id ID`: Flow instance ID to use (default: `default`)
|
||||||
|
- `variable=value`: Template variable assignments (can be specified multiple times)
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
### Basic Question Answering
|
||||||
|
```bash
|
||||||
|
tg-invoke-prompt question text="What is artificial intelligence?" context="AI research field"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Extract Definitions
|
||||||
|
```bash
|
||||||
|
tg-invoke-prompt extract-definitions \
|
||||||
|
document="Machine learning is a subset of artificial intelligence..." \
|
||||||
|
terms="machine learning,neural networks"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Text Summarization
|
||||||
|
```bash
|
||||||
|
tg-invoke-prompt summarize \
|
||||||
|
text="$(cat large-document.txt)" \
|
||||||
|
max_length="200" \
|
||||||
|
style="technical"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Custom Flow and Variables
|
||||||
|
```bash
|
||||||
|
tg-invoke-prompt analysis \
|
||||||
|
-f "research-flow" \
|
||||||
|
data="$(cat research-data.json)" \
|
||||||
|
focus="trends" \
|
||||||
|
output_format="markdown"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Variable Substitution
|
||||||
|
|
||||||
|
Templates use `{{variable}}` placeholders that are replaced with command-line values:
|
||||||
|
|
||||||
|
### Simple Variables
|
||||||
|
```bash
|
||||||
|
tg-invoke-prompt greeting name="Alice" time="morning"
|
||||||
|
# Template: "Good {{time}}, {{name}}!"
|
||||||
|
# Result: "Good morning, Alice!"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Complex Variables
|
||||||
|
```bash
|
||||||
|
tg-invoke-prompt analyze \
|
||||||
|
dataset="$(cat data.csv)" \
|
||||||
|
columns="name,age,salary" \
|
||||||
|
analysis_type="statistical_summary"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Multi-line Variables
|
||||||
|
```bash
|
||||||
|
tg-invoke-prompt review \
|
||||||
|
code="$(cat app.py)" \
|
||||||
|
checklist="security,performance,maintainability" \
|
||||||
|
severity="high"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common Template Types
|
||||||
|
|
||||||
|
### Question Answering
|
||||||
|
```bash
|
||||||
|
# Direct question
|
||||||
|
tg-invoke-prompt question \
|
||||||
|
text="What is the capital of France?" \
|
||||||
|
context="geography"
|
||||||
|
|
||||||
|
# Contextual question
|
||||||
|
tg-invoke-prompt question \
|
||||||
|
text="How does this work?" \
|
||||||
|
context="$(cat technical-manual.txt)"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Text Processing
|
||||||
|
```bash
|
||||||
|
# Extract key information
|
||||||
|
tg-invoke-prompt extract-key-points \
|
||||||
|
document="$(cat meeting-notes.txt)" \
|
||||||
|
format="bullet_points"
|
||||||
|
|
||||||
|
# Text classification
|
||||||
|
tg-invoke-prompt classify \
|
||||||
|
text="Customer is very unhappy with service" \
|
||||||
|
categories="positive,negative,neutral"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Code Analysis
|
||||||
|
```bash
|
||||||
|
# Code review
|
||||||
|
tg-invoke-prompt code-review \
|
||||||
|
code="$(cat script.py)" \
|
||||||
|
language="python" \
|
||||||
|
focus="security,performance"
|
||||||
|
|
||||||
|
# Bug analysis
|
||||||
|
tg-invoke-prompt debug \
|
||||||
|
code="$(cat buggy-code.js)" \
|
||||||
|
error="TypeError: Cannot read property 'length' of undefined"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data Analysis
|
||||||
|
```bash
|
||||||
|
# Data insights
|
||||||
|
tg-invoke-prompt data-analysis \
|
||||||
|
data="$(cat sales-data.json)" \
|
||||||
|
metrics="revenue,growth,trends" \
|
||||||
|
period="quarterly"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Template Management
|
||||||
|
|
||||||
|
### List Available Templates
|
||||||
|
```bash
|
||||||
|
# Show available prompt templates
|
||||||
|
tg-show-prompts
|
||||||
|
```
|
||||||
|
|
||||||
|
### Create Custom Templates
|
||||||
|
```bash
|
||||||
|
# Define a new template
|
||||||
|
tg-set-prompt analysis-template \
|
||||||
|
"Analyze the following {{data_type}}: {{data}}. Focus on {{focus_areas}}. Output format: {{format}}"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Template Variables
|
||||||
|
Common template variables:
|
||||||
|
- `{{text}}` - Input text to process
|
||||||
|
- `{{context}}` - Additional context information
|
||||||
|
- `{{format}}` - Output format specification
|
||||||
|
- `{{language}}` - Programming language for code analysis
|
||||||
|
- `{{style}}` - Writing or analysis style
|
||||||
|
- `{{length}}` - Length constraints for output
|
||||||
|
|
||||||
|
## Output Formats
|
||||||
|
|
||||||
|
### String Response
|
||||||
|
```bash
|
||||||
|
tg-invoke-prompt summarize text="Long document..." max_length="100"
|
||||||
|
# Output: "This document discusses..."
|
||||||
|
```
|
||||||
|
|
||||||
|
### JSON Response
|
||||||
|
```bash
|
||||||
|
tg-invoke-prompt extract-structured data="Name: John, Age: 30, City: NYC"
|
||||||
|
# Output:
|
||||||
|
# {
|
||||||
|
# "name": "John",
|
||||||
|
# "age": 30,
|
||||||
|
# "city": "NYC"
|
||||||
|
# }
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
### Missing Template
|
||||||
|
```bash
|
||||||
|
Exception: Template 'nonexistent-template' not found
|
||||||
|
```
|
||||||
|
**Solution**: Check available templates with `tg-show-prompts`.
|
||||||
|
|
||||||
|
### Missing Variables
|
||||||
|
```bash
|
||||||
|
Exception: Template variable 'required_var' not provided
|
||||||
|
```
|
||||||
|
**Solution**: Provide all required variables as `variable=value` arguments.
|
||||||
|
|
||||||
|
### Malformed Variables
|
||||||
|
```bash
|
||||||
|
Exception: Malformed variable: invalid-format
|
||||||
|
```
|
||||||
|
**Solution**: Use `variable=value` format for all variable assignments.
|
||||||
|
|
||||||
|
### Flow Not Found
|
||||||
|
```bash
|
||||||
|
Exception: Flow instance 'invalid-flow' not found
|
||||||
|
```
|
||||||
|
**Solution**: Verify flow ID exists with `tg-show-flows`.
|
||||||
|
|
||||||
|
## Advanced Usage
|
||||||
|
|
||||||
|
### File Input Processing
|
||||||
|
```bash
|
||||||
|
# Process multiple files
|
||||||
|
for file in *.txt; do
|
||||||
|
echo "Processing $file..."
|
||||||
|
tg-invoke-prompt summarize \
|
||||||
|
text="$(cat "$file")" \
|
||||||
|
filename="$file" \
|
||||||
|
max_length="150"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Batch Processing
|
||||||
|
```bash
|
||||||
|
# Process data in batches
|
||||||
|
while IFS= read -r line; do
|
||||||
|
tg-invoke-prompt classify \
|
||||||
|
text="$line" \
|
||||||
|
categories="spam,ham,promotional" \
|
||||||
|
confidence_threshold="0.8"
|
||||||
|
done < input-data.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pipeline Processing
|
||||||
|
```bash
|
||||||
|
# Chain multiple prompts
|
||||||
|
initial_analysis=$(tg-invoke-prompt analyze data="$(cat raw-data.json)")
|
||||||
|
summary=$(tg-invoke-prompt summarize text="$initial_analysis" style="executive")
|
||||||
|
echo "$summary"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Interactive Processing
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
# interactive-prompt.sh
|
||||||
|
template="$1"
|
||||||
|
|
||||||
|
if [ -z "$template" ]; then
|
||||||
|
echo "Usage: $0 <template-id>"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Interactive prompt using template: $template"
|
||||||
|
echo "Enter variables (var=value), empty line to execute:"
|
||||||
|
|
||||||
|
variables=()
|
||||||
|
while true; do
|
||||||
|
read -p "> " input
|
||||||
|
if [ -z "$input" ]; then
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
variables+=("$input")
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "Executing prompt..."
|
||||||
|
tg-invoke-prompt "$template" "${variables[@]}"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configuration-Driven Processing
|
||||||
|
```bash
|
||||||
|
# Use configuration file for prompts
|
||||||
|
config_file="prompt-config.json"
|
||||||
|
template=$(jq -r '.template' "$config_file")
|
||||||
|
variables=$(jq -r '.variables | to_entries[] | "\(.key)=\(.value)"' "$config_file")
|
||||||
|
|
||||||
|
tg-invoke-prompt "$template" $variables
|
||||||
|
```
|
||||||
|
|
||||||
|
## Performance Optimization
|
||||||
|
|
||||||
|
### Caching Results
|
||||||
|
```bash
|
||||||
|
# Cache prompt results
|
||||||
|
cache_dir="prompt-cache"
|
||||||
|
mkdir -p "$cache_dir"
|
||||||
|
|
||||||
|
invoke_with_cache() {
|
||||||
|
local template="$1"
|
||||||
|
shift
|
||||||
|
local args="$@"
|
||||||
|
local cache_key=$(echo "$template-$args" | md5sum | cut -d' ' -f1)
|
||||||
|
local cache_file="$cache_dir/$cache_key.txt"
|
||||||
|
|
||||||
|
if [ -f "$cache_file" ]; then
|
||||||
|
echo "Cache hit"
|
||||||
|
cat "$cache_file"
|
||||||
|
else
|
||||||
|
echo "Cache miss, invoking prompt..."
|
||||||
|
tg-invoke-prompt "$template" "$@" | tee "$cache_file"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Parallel Processing
|
||||||
|
```bash
|
||||||
|
# Process multiple items in parallel
|
||||||
|
input_files=(file1.txt file2.txt file3.txt)
|
||||||
|
for file in "${input_files[@]}"; do
|
||||||
|
(
|
||||||
|
echo "Processing $file..."
|
||||||
|
tg-invoke-prompt analyze \
|
||||||
|
text="$(cat "$file")" \
|
||||||
|
filename="$file" > "result-$file.json"
|
||||||
|
) &
|
||||||
|
done
|
||||||
|
wait
|
||||||
|
```
|
||||||
|
|
||||||
|
## Use Cases
|
||||||
|
|
||||||
|
### Document Processing
|
||||||
|
```bash
|
||||||
|
# Extract metadata from documents
|
||||||
|
tg-invoke-prompt extract-metadata \
|
||||||
|
document="$(cat document.pdf)" \
|
||||||
|
fields="title,author,date,keywords"
|
||||||
|
|
||||||
|
# Generate document summaries
|
||||||
|
tg-invoke-prompt summarize \
|
||||||
|
text="$(cat report.txt)" \
|
||||||
|
audience="executives" \
|
||||||
|
key_points="5"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Code Analysis
|
||||||
|
```bash
|
||||||
|
# Security analysis
|
||||||
|
tg-invoke-prompt security-review \
|
||||||
|
code="$(cat webapp.py)" \
|
||||||
|
framework="flask" \
|
||||||
|
focus="injection,authentication"
|
||||||
|
|
||||||
|
# Performance optimization suggestions
|
||||||
|
tg-invoke-prompt optimize \
|
||||||
|
code="$(cat slow-function.js)" \
|
||||||
|
language="javascript" \
|
||||||
|
target="performance"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data Analysis
|
||||||
|
```bash
|
||||||
|
# Generate insights from data
|
||||||
|
tg-invoke-prompt insights \
|
||||||
|
data="$(cat metrics.json)" \
|
||||||
|
timeframe="monthly" \
|
||||||
|
focus="trends,anomalies"
|
||||||
|
|
||||||
|
# Create data visualizations
|
||||||
|
tg-invoke-prompt visualize \
|
||||||
|
data="$(cat sales-data.csv)" \
|
||||||
|
chart_type="line" \
|
||||||
|
metrics="revenue,growth"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Content Generation
|
||||||
|
```bash
|
||||||
|
# Generate marketing copy
|
||||||
|
tg-invoke-prompt marketing \
|
||||||
|
product="AI Assistant" \
|
||||||
|
audience="developers" \
|
||||||
|
tone="professional,friendly"
|
||||||
|
|
||||||
|
# Create technical documentation
|
||||||
|
tg-invoke-prompt document \
|
||||||
|
code="$(cat api.py)" \
|
||||||
|
format="markdown" \
|
||||||
|
sections="overview,examples,parameters"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
- `TRUSTGRAPH_URL`: Default API URL
|
||||||
|
|
||||||
|
## Related Commands
|
||||||
|
|
||||||
|
- [`tg-show-prompts`](tg-show-prompts.md) - List available prompt templates
|
||||||
|
- [`tg-set-prompt`](tg-set-prompt.md) - Create/update prompt templates
|
||||||
|
- [`tg-invoke-document-rag`](tg-invoke-document-rag.md) - Document-based question answering
|
||||||
|
- [`tg-show-flows`](tg-show-flows.md) - List available flow instances
|
||||||
|
|
||||||
|
## API Integration
|
||||||
|
|
||||||
|
This command uses the prompt service API to process templates and generate responses using configured language models.
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
1. **Template Reuse**: Create reusable templates for common tasks
|
||||||
|
2. **Variable Validation**: Validate required variables before execution
|
||||||
|
3. **Error Handling**: Implement proper error handling for production use
|
||||||
|
4. **Caching**: Cache results for repeated operations
|
||||||
|
5. **Documentation**: Document custom templates and their expected variables
|
||||||
|
6. **Security**: Avoid embedding sensitive data in templates
|
||||||
|
7. **Performance**: Use appropriate flow instances for different workloads
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Template Not Found
|
||||||
|
```bash
|
||||||
|
# Check available templates
|
||||||
|
tg-show-prompts
|
||||||
|
|
||||||
|
# Verify template name spelling
|
||||||
|
tg-show-prompts | grep "template-name"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Variable Errors
|
||||||
|
```bash
|
||||||
|
# Check template definition for required variables
|
||||||
|
tg-show-prompts | grep -A 10 "template-name"
|
||||||
|
|
||||||
|
# Validate variable format
|
||||||
|
echo "variable=value" | grep "="
|
||||||
|
```
|
||||||
|
|
||||||
|
### Flow Issues
|
||||||
|
```bash
|
||||||
|
# Check flow status
|
||||||
|
tg-show-flows | grep "flow-id"
|
||||||
|
|
||||||
|
# Verify flow has prompt service
|
||||||
|
tg-get-flow-class -n "flow-class" | jq '.interfaces.prompt'
|
||||||
|
```
|
||||||
480
docs/cli/tg-load-pdf.md
Normal file
480
docs/cli/tg-load-pdf.md
Normal file
|
|
@ -0,0 +1,480 @@
|
||||||
|
# tg-load-pdf
|
||||||
|
|
||||||
|
Loads PDF documents into TrustGraph for processing and analysis.
|
||||||
|
|
||||||
|
## Synopsis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tg-load-pdf [options] file1.pdf [file2.pdf ...]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Description
|
||||||
|
|
||||||
|
The `tg-load-pdf` command loads PDF documents into TrustGraph by directing them to the PDF decoder service. The command extracts content, generates document metadata, and makes the documents available for processing by other TrustGraph services.
|
||||||
|
|
||||||
|
Each PDF is assigned a unique identifier based on its content hash, and comprehensive metadata can be attached including copyright information, publication details, and keywords.
|
||||||
|
|
||||||
|
**Note**: Consider using `tg-add-library-document` followed by `tg-start-library-processing` for more comprehensive document management.
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
### Required Arguments
|
||||||
|
|
||||||
|
- `files`: One or more PDF files to load
|
||||||
|
|
||||||
|
### Optional Arguments
|
||||||
|
|
||||||
|
- `-u, --url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
|
||||||
|
- `-f, --flow-id ID`: Flow instance ID to use (default: `default`)
|
||||||
|
- `-U, --user USER`: User ID for document ownership (default: `trustgraph`)
|
||||||
|
- `-C, --collection COLLECTION`: Collection to assign document (default: `default`)
|
||||||
|
|
||||||
|
### Document Metadata
|
||||||
|
|
||||||
|
- `--name NAME`: Document name/title
|
||||||
|
- `--description DESCRIPTION`: Document description
|
||||||
|
- `--identifier ID`: Custom document identifier
|
||||||
|
- `--document-url URL`: Source URL for the document
|
||||||
|
- `--keyword KEYWORD`: Document keywords (can be specified multiple times)
|
||||||
|
|
||||||
|
### Copyright Information
|
||||||
|
|
||||||
|
- `--copyright-notice NOTICE`: Copyright notice text
|
||||||
|
- `--copyright-holder HOLDER`: Copyright holder name
|
||||||
|
- `--copyright-year YEAR`: Copyright year
|
||||||
|
- `--license LICENSE`: Copyright license
|
||||||
|
|
||||||
|
### Publication Details
|
||||||
|
|
||||||
|
- `--publication-organization ORG`: Publishing organization
|
||||||
|
- `--publication-description DESC`: Publication description
|
||||||
|
- `--publication-date DATE`: Publication date
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
### Basic PDF Loading
|
||||||
|
```bash
|
||||||
|
tg-load-pdf document.pdf
|
||||||
|
```
|
||||||
|
|
||||||
|
### Multiple Files
|
||||||
|
```bash
|
||||||
|
tg-load-pdf report1.pdf report2.pdf manual.pdf
|
||||||
|
```
|
||||||
|
|
||||||
|
### With Basic Metadata
|
||||||
|
```bash
|
||||||
|
tg-load-pdf \
|
||||||
|
--name "Technical Manual" \
|
||||||
|
--description "System administration guide" \
|
||||||
|
--keyword "technical" --keyword "manual" \
|
||||||
|
technical-manual.pdf
|
||||||
|
```
|
||||||
|
|
||||||
|
### Complete Metadata
|
||||||
|
```bash
|
||||||
|
tg-load-pdf \
|
||||||
|
--name "Annual Report 2023" \
|
||||||
|
--description "Company annual financial report" \
|
||||||
|
--copyright-holder "Acme Corporation" \
|
||||||
|
--copyright-year "2023" \
|
||||||
|
--license "All Rights Reserved" \
|
||||||
|
--publication-organization "Acme Corporation" \
|
||||||
|
--publication-date "2023-12-31" \
|
||||||
|
--keyword "financial" --keyword "annual" --keyword "report" \
|
||||||
|
annual-report-2023.pdf
|
||||||
|
```
|
||||||
|
|
||||||
|
### Custom Flow and Collection
|
||||||
|
```bash
|
||||||
|
tg-load-pdf \
|
||||||
|
-f "document-processing-flow" \
|
||||||
|
-U "finance-team" \
|
||||||
|
-C "financial-documents" \
|
||||||
|
--name "Budget Analysis" \
|
||||||
|
budget-2024.pdf
|
||||||
|
```
|
||||||
|
|
||||||
|
## Document Processing
|
||||||
|
|
||||||
|
### Content Extraction
|
||||||
|
The PDF loader:
|
||||||
|
1. Calculates SHA256 hash for unique document ID
|
||||||
|
2. Extracts text content from PDF
|
||||||
|
3. Preserves document structure and formatting metadata
|
||||||
|
4. Generates searchable text index
|
||||||
|
|
||||||
|
### Metadata Generation
|
||||||
|
Document metadata includes:
|
||||||
|
- **Document ID**: SHA256 hash-based unique identifier
|
||||||
|
- **Content Hash**: For duplicate detection
|
||||||
|
- **File Information**: Size, format, creation date
|
||||||
|
- **Custom Metadata**: User-provided attributes
|
||||||
|
|
||||||
|
### Integration with Processing Pipeline
|
||||||
|
```bash
|
||||||
|
# Load PDF and start processing
|
||||||
|
tg-load-pdf research-paper.pdf --name "AI Research Paper"
|
||||||
|
|
||||||
|
# Check processing status
|
||||||
|
tg-show-flows | grep "document-processing"
|
||||||
|
|
||||||
|
# Query loaded content
|
||||||
|
tg-invoke-document-rag -q "What is the main conclusion?" -C "default"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
### File Not Found
|
||||||
|
```bash
|
||||||
|
Exception: [Errno 2] No such file or directory: 'missing.pdf'
|
||||||
|
```
|
||||||
|
**Solution**: Verify file path and ensure PDF exists.
|
||||||
|
|
||||||
|
### Invalid PDF Format
|
||||||
|
```bash
|
||||||
|
Exception: PDF parsing failed: Invalid PDF structure
|
||||||
|
```
|
||||||
|
**Solution**: Verify PDF is not corrupted and is a valid PDF file.
|
||||||
|
|
||||||
|
### Permission Errors
|
||||||
|
```bash
|
||||||
|
Exception: [Errno 13] Permission denied: 'protected.pdf'
|
||||||
|
```
|
||||||
|
**Solution**: Check file permissions and ensure read access.
|
||||||
|
|
||||||
|
### Flow Not Found
|
||||||
|
```bash
|
||||||
|
Exception: Flow instance 'invalid-flow' not found
|
||||||
|
```
|
||||||
|
**Solution**: Verify flow ID exists with `tg-show-flows`.
|
||||||
|
|
||||||
|
### API Connection Issues
|
||||||
|
```bash
|
||||||
|
Exception: Connection refused
|
||||||
|
```
|
||||||
|
**Solution**: Check API URL and ensure TrustGraph is running.
|
||||||
|
|
||||||
|
## Advanced Usage
|
||||||
|
|
||||||
|
### Batch Processing
|
||||||
|
```bash
|
||||||
|
# Process all PDFs in directory
|
||||||
|
for pdf in *.pdf; do
|
||||||
|
echo "Loading $pdf..."
|
||||||
|
tg-load-pdf \
|
||||||
|
--name "$(basename "$pdf" .pdf)" \
|
||||||
|
--collection "research-papers" \
|
||||||
|
"$pdf"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Organized Loading
|
||||||
|
```bash
|
||||||
|
# Load with structured metadata
|
||||||
|
categories=("technical" "financial" "legal")
|
||||||
|
for category in "${categories[@]}"; do
|
||||||
|
for pdf in "$category"/*.pdf; do
|
||||||
|
if [ -f "$pdf" ]; then
|
||||||
|
tg-load-pdf \
|
||||||
|
--collection "$category-documents" \
|
||||||
|
--keyword "$category" \
|
||||||
|
--name "$(basename "$pdf" .pdf)" \
|
||||||
|
"$pdf"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### CSV-Driven Loading
|
||||||
|
```bash
|
||||||
|
# Load PDFs with metadata from CSV
|
||||||
|
# Format: filename,title,description,keywords
|
||||||
|
while IFS=',' read -r filename title description keywords; do
|
||||||
|
if [ -f "$filename" ]; then
|
||||||
|
echo "Loading $filename..."
|
||||||
|
|
||||||
|
# Convert comma-separated keywords to multiple --keyword args
|
||||||
|
keyword_args=""
|
||||||
|
IFS='|' read -ra KEYWORDS <<< "$keywords"
|
||||||
|
for kw in "${KEYWORDS[@]}"; do
|
||||||
|
keyword_args="$keyword_args --keyword \"$kw\""
|
||||||
|
done
|
||||||
|
|
||||||
|
eval "tg-load-pdf \
|
||||||
|
--name \"$title\" \
|
||||||
|
--description \"$description\" \
|
||||||
|
$keyword_args \
|
||||||
|
\"$filename\""
|
||||||
|
fi
|
||||||
|
done < documents.csv
|
||||||
|
```
|
||||||
|
|
||||||
|
### Publication Processing
|
||||||
|
```bash
|
||||||
|
# Load academic papers with publication details
|
||||||
|
load_academic_paper() {
|
||||||
|
local file="$1"
|
||||||
|
local title="$2"
|
||||||
|
local authors="$3"
|
||||||
|
local journal="$4"
|
||||||
|
local year="$5"
|
||||||
|
|
||||||
|
tg-load-pdf \
|
||||||
|
--name "$title" \
|
||||||
|
--description "Academic paper: $title" \
|
||||||
|
--copyright-holder "$authors" \
|
||||||
|
--copyright-year "$year" \
|
||||||
|
--publication-organization "$journal" \
|
||||||
|
--publication-date "$year-01-01" \
|
||||||
|
--keyword "academic" --keyword "research" \
|
||||||
|
"$file"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Usage
|
||||||
|
load_academic_paper "ai-paper.pdf" "AI in Healthcare" "Smith et al." "AI Journal" "2023"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Monitoring and Validation
|
||||||
|
|
||||||
|
### Load Status Checking
|
||||||
|
```bash
|
||||||
|
# Check document loading progress
|
||||||
|
check_load_status() {
|
||||||
|
local file="$1"
|
||||||
|
local expected_name="$2"
|
||||||
|
|
||||||
|
echo "Checking load status for: $file"
|
||||||
|
|
||||||
|
# Check if document appears in library
|
||||||
|
if tg-show-library-documents | grep -q "$expected_name"; then
|
||||||
|
echo "✓ Document loaded successfully"
|
||||||
|
else
|
||||||
|
echo "✗ Document not found in library"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Monitor batch loading
|
||||||
|
for pdf in *.pdf; do
|
||||||
|
name=$(basename "$pdf" .pdf)
|
||||||
|
check_load_status "$pdf" "$name"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Content Verification
|
||||||
|
```bash
|
||||||
|
# Verify PDF content is accessible
|
||||||
|
verify_pdf_content() {
|
||||||
|
local pdf_name="$1"
|
||||||
|
local test_query="$2"
|
||||||
|
|
||||||
|
echo "Verifying content for: $pdf_name"
|
||||||
|
|
||||||
|
# Try to query the document
|
||||||
|
result=$(tg-invoke-document-rag -q "$test_query" -C "default" 2>/dev/null)
|
||||||
|
|
||||||
|
if [ $? -eq 0 ] && [ -n "$result" ]; then
|
||||||
|
echo "✓ Content accessible via RAG"
|
||||||
|
else
|
||||||
|
echo "✗ Content not accessible"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Verify loaded documents
|
||||||
|
verify_pdf_content "Technical Manual" "What is the installation process?"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Performance Optimization
|
||||||
|
|
||||||
|
### Parallel Loading
|
||||||
|
```bash
|
||||||
|
# Load multiple PDFs in parallel
|
||||||
|
pdf_files=(document1.pdf document2.pdf document3.pdf)
|
||||||
|
for pdf in "${pdf_files[@]}"; do
|
||||||
|
(
|
||||||
|
echo "Loading $pdf in background..."
|
||||||
|
tg-load-pdf \
|
||||||
|
--name "$(basename "$pdf" .pdf)" \
|
||||||
|
--collection "batch-$(date +%Y%m%d)" \
|
||||||
|
"$pdf"
|
||||||
|
) &
|
||||||
|
done
|
||||||
|
wait
|
||||||
|
echo "All PDFs loaded"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Size-Based Processing
|
||||||
|
```bash
|
||||||
|
# Process files based on size
|
||||||
|
for pdf in *.pdf; do
|
||||||
|
size=$(stat -c%s "$pdf")
|
||||||
|
if [ $size -lt 10485760 ]; then # < 10MB
|
||||||
|
echo "Processing small file: $pdf"
|
||||||
|
tg-load-pdf --collection "small-docs" "$pdf"
|
||||||
|
else
|
||||||
|
echo "Processing large file: $pdf"
|
||||||
|
tg-load-pdf --collection "large-docs" "$pdf"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
## Document Organization
|
||||||
|
|
||||||
|
### Collection Management
|
||||||
|
```bash
|
||||||
|
# Organize by document type
|
||||||
|
organize_by_type() {
|
||||||
|
local pdf="$1"
|
||||||
|
local filename=$(basename "$pdf" .pdf)
|
||||||
|
|
||||||
|
case "$filename" in
|
||||||
|
*manual*|*guide*) collection="manuals" ;;
|
||||||
|
*report*|*analysis*) collection="reports" ;;
|
||||||
|
*spec*|*specification*) collection="specifications" ;;
|
||||||
|
*legal*|*contract*) collection="legal" ;;
|
||||||
|
*) collection="general" ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
tg-load-pdf \
|
||||||
|
--collection "$collection" \
|
||||||
|
--name "$filename" \
|
||||||
|
"$pdf"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Process all PDFs
|
||||||
|
for pdf in *.pdf; do
|
||||||
|
organize_by_type "$pdf"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Metadata Standardization
|
||||||
|
```bash
|
||||||
|
# Apply consistent metadata standards
|
||||||
|
standardize_metadata() {
|
||||||
|
local pdf="$1"
|
||||||
|
local dept="$2"
|
||||||
|
local year="$3"
|
||||||
|
|
||||||
|
local name=$(basename "$pdf" .pdf)
|
||||||
|
local collection="$dept-$(date +%Y)"
|
||||||
|
|
||||||
|
tg-load-pdf \
|
||||||
|
--name "$name" \
|
||||||
|
--description "$dept document from $year" \
|
||||||
|
--copyright-holder "Company Name" \
|
||||||
|
--copyright-year "$year" \
|
||||||
|
--collection "$collection" \
|
||||||
|
--keyword "$dept" --keyword "$year" \
|
||||||
|
"$pdf"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Usage
|
||||||
|
standardize_metadata "finance-report.pdf" "finance" "2023"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Integration with Other Services
|
||||||
|
|
||||||
|
### Library Integration
|
||||||
|
```bash
|
||||||
|
# Alternative approach using library services
|
||||||
|
load_via_library() {
|
||||||
|
local pdf="$1"
|
||||||
|
local name="$2"
|
||||||
|
|
||||||
|
# Add to library first
|
||||||
|
tg-add-library-document \
|
||||||
|
--name "$name" \
|
||||||
|
--file "$pdf" \
|
||||||
|
--collection "documents"
|
||||||
|
|
||||||
|
# Start processing
|
||||||
|
tg-start-library-processing \
|
||||||
|
--collection "documents"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Workflow Integration
|
||||||
|
```bash
|
||||||
|
# Complete document workflow
|
||||||
|
process_document_workflow() {
|
||||||
|
local pdf="$1"
|
||||||
|
local name="$2"
|
||||||
|
|
||||||
|
echo "Starting document workflow for: $name"
|
||||||
|
|
||||||
|
# 1. Load PDF
|
||||||
|
tg-load-pdf --name "$name" "$pdf"
|
||||||
|
|
||||||
|
# 2. Wait for processing
|
||||||
|
sleep 5
|
||||||
|
|
||||||
|
# 3. Verify availability
|
||||||
|
if tg-show-library-documents | grep -q "$name"; then
|
||||||
|
echo "Document available in library"
|
||||||
|
|
||||||
|
# 4. Test RAG functionality
|
||||||
|
tg-invoke-document-rag -q "What is this document about?"
|
||||||
|
|
||||||
|
# 5. Extract key information
|
||||||
|
tg-invoke-prompt extract-key-points \
|
||||||
|
text="Document: $name" \
|
||||||
|
format="bullet_points"
|
||||||
|
else
|
||||||
|
echo "Document processing failed"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
- `TRUSTGRAPH_URL`: Default API URL
|
||||||
|
|
||||||
|
## Related Commands
|
||||||
|
|
||||||
|
- [`tg-add-library-document`](tg-add-library-document.md) - Add documents to library
|
||||||
|
- [`tg-start-library-processing`](tg-start-library-processing.md) - Process library documents
|
||||||
|
- [`tg-show-library-documents`](tg-show-library-documents.md) - List library documents
|
||||||
|
- [`tg-invoke-document-rag`](tg-invoke-document-rag.md) - Query document content
|
||||||
|
- [`tg-show-flows`](tg-show-flows.md) - Monitor processing flows
|
||||||
|
|
||||||
|
## API Integration
|
||||||
|
|
||||||
|
This command uses the document loading API to process PDF files and make them available for text extraction, search, and analysis.
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
1. **Metadata Completeness**: Provide comprehensive metadata for better organization
|
||||||
|
2. **Collection Organization**: Use logical collections for document categorization
|
||||||
|
3. **Error Handling**: Implement robust error handling for batch operations
|
||||||
|
4. **Performance**: Consider file sizes and processing capacity
|
||||||
|
5. **Monitoring**: Verify successful loading and processing
|
||||||
|
6. **Security**: Ensure sensitive documents are properly protected
|
||||||
|
7. **Backup**: Maintain backups of source PDFs
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### PDF Processing Issues
|
||||||
|
```bash
|
||||||
|
# Check PDF validity
|
||||||
|
file document.pdf
|
||||||
|
pdfinfo document.pdf
|
||||||
|
|
||||||
|
# Try alternative PDF processors
|
||||||
|
qpdf --check document.pdf
|
||||||
|
```
|
||||||
|
|
||||||
|
### Memory Issues
|
||||||
|
```bash
|
||||||
|
# For large PDFs, monitor memory usage
|
||||||
|
free -h
|
||||||
|
# Consider processing large files separately
|
||||||
|
```
|
||||||
|
|
||||||
|
### Content Extraction Problems
|
||||||
|
```bash
|
||||||
|
# Verify PDF contains extractable text
|
||||||
|
pdftotext document.pdf test-output.txt
|
||||||
|
cat test-output.txt | head -20
|
||||||
|
```
|
||||||
505
docs/cli/tg-load-turtle.md
Normal file
505
docs/cli/tg-load-turtle.md
Normal file
|
|
@ -0,0 +1,505 @@
|
||||||
|
# tg-load-turtle
|
||||||
|
|
||||||
|
Loads RDF triples from Turtle files into the TrustGraph knowledge graph.
|
||||||
|
|
||||||
|
## Synopsis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tg-load-turtle -i DOCUMENT_ID [options] file1.ttl [file2.ttl ...]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Description
|
||||||
|
|
||||||
|
The `tg-load-turtle` command loads RDF triples from Turtle (TTL) format files into TrustGraph's knowledge graph. It parses Turtle files, converts them to TrustGraph's internal triple format, and imports them using WebSocket connections for efficient batch processing.
|
||||||
|
|
||||||
|
The command supports retry logic and automatic reconnection to handle network interruptions during large data imports.
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
### Required Arguments
|
||||||
|
|
||||||
|
- `-i, --document-id ID`: Document ID to associate with the triples
|
||||||
|
- `files`: One or more Turtle files to load
|
||||||
|
|
||||||
|
### Optional Arguments
|
||||||
|
|
||||||
|
- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `ws://localhost:8088/`)
|
||||||
|
- `-f, --flow-id ID`: Flow instance ID to use (default: `default`)
|
||||||
|
- `-U, --user USER`: User ID for triple ownership (default: `trustgraph`)
|
||||||
|
- `-C, --collection COLLECTION`: Collection to assign triples (default: `default`)
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
### Basic Turtle Loading
|
||||||
|
```bash
|
||||||
|
tg-load-turtle -i "doc123" knowledge-base.ttl
|
||||||
|
```
|
||||||
|
|
||||||
|
### Multiple Files
|
||||||
|
```bash
|
||||||
|
tg-load-turtle -i "ontology-v1" \
|
||||||
|
schema.ttl \
|
||||||
|
instances.ttl \
|
||||||
|
relationships.ttl
|
||||||
|
```
|
||||||
|
|
||||||
|
### Custom Flow and Collection
|
||||||
|
```bash
|
||||||
|
tg-load-turtle \
|
||||||
|
-i "research-data" \
|
||||||
|
-f "knowledge-import-flow" \
|
||||||
|
-U "research-team" \
|
||||||
|
-C "research-kg" \
|
||||||
|
research-triples.ttl
|
||||||
|
```
|
||||||
|
|
||||||
|
### Load with Custom API URL
|
||||||
|
```bash
|
||||||
|
tg-load-turtle \
|
||||||
|
-i "production-data" \
|
||||||
|
-u "ws://production:8088/" \
|
||||||
|
production-ontology.ttl
|
||||||
|
```
|
||||||
|
|
||||||
|
## Turtle Format Support
|
||||||
|
|
||||||
|
### Basic Triples
|
||||||
|
```turtle
|
||||||
|
@prefix ex: <http://example.org/> .
|
||||||
|
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
|
||||||
|
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
|
||||||
|
|
||||||
|
ex:Person rdf:type rdfs:Class .
|
||||||
|
ex:john rdf:type ex:Person .
|
||||||
|
ex:john ex:name "John Doe" .
|
||||||
|
ex:john ex:age "30"^^xsd:integer .
|
||||||
|
```
|
||||||
|
|
||||||
|
### Complex Structures
|
||||||
|
```turtle
|
||||||
|
@prefix org: <http://example.org/organization/> .
|
||||||
|
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
|
||||||
|
|
||||||
|
org:TechCorp rdf:type foaf:Organization ;
|
||||||
|
foaf:name "Technology Corporation" ;
|
||||||
|
org:hasEmployee org:john, org:jane ;
|
||||||
|
org:foundedYear "2010"^^xsd:gYear .
|
||||||
|
|
||||||
|
org:john foaf:name "John Smith" ;
|
||||||
|
foaf:mbox <mailto:john@techcorp.com> ;
|
||||||
|
org:position "Software Engineer" .
|
||||||
|
```
|
||||||
|
|
||||||
|
### Ontology Loading
|
||||||
|
```turtle
|
||||||
|
@prefix owl: <http://www.w3.org/2002/07/owl#> .
|
||||||
|
@prefix dc: <http://purl.org/dc/elements/1.1/> .
|
||||||
|
|
||||||
|
<http://example.org/ontology> rdf:type owl:Ontology ;
|
||||||
|
dc:title "Example Ontology" ;
|
||||||
|
dc:creator "Knowledge Team" .
|
||||||
|
|
||||||
|
ex:Vehicle rdf:type owl:Class ;
|
||||||
|
rdfs:label "Vehicle" ;
|
||||||
|
rdfs:comment "A means of transportation" .
|
||||||
|
|
||||||
|
ex:Car rdfs:subClassOf ex:Vehicle .
|
||||||
|
ex:Truck rdfs:subClassOf ex:Vehicle .
|
||||||
|
```
|
||||||
|
|
||||||
|
## Data Processing
|
||||||
|
|
||||||
|
### Triple Conversion
|
||||||
|
The loader converts Turtle triples to TrustGraph format:
|
||||||
|
- **URIs**: Converted to URI references with `is_uri=true`
|
||||||
|
- **Literals**: Converted to literal values with `is_uri=false`
|
||||||
|
- **Datatypes**: Preserved in literal values
|
||||||
|
|
||||||
|
### Batch Processing
|
||||||
|
- Triples are sent individually via WebSocket
|
||||||
|
- Each triple includes document metadata
|
||||||
|
- Automatic retry on connection failures
|
||||||
|
- Progress tracking for large files
|
||||||
|
|
||||||
|
### Error Handling
|
||||||
|
- Invalid Turtle syntax causes parsing errors
|
||||||
|
- Network interruptions trigger automatic retry
|
||||||
|
- Malformed triples are skipped with warnings
|
||||||
|
|
||||||
|
## Use Cases
|
||||||
|
|
||||||
|
### Ontology Import
|
||||||
|
```bash
|
||||||
|
# Load domain ontology
|
||||||
|
tg-load-turtle -i "healthcare-ontology" \
|
||||||
|
-C "ontologies" \
|
||||||
|
healthcare-schema.ttl
|
||||||
|
|
||||||
|
# Load instance data
|
||||||
|
tg-load-turtle -i "patient-data" \
|
||||||
|
-C "healthcare-data" \
|
||||||
|
patient-records.ttl
|
||||||
|
```
|
||||||
|
|
||||||
|
### Knowledge Base Migration
|
||||||
|
```bash
|
||||||
|
# Migrate from external knowledge base
|
||||||
|
tg-load-turtle -i "migration-$(date +%Y%m%d)" \
|
||||||
|
-C "migrated-data" \
|
||||||
|
exported-knowledge.ttl
|
||||||
|
```
|
||||||
|
|
||||||
|
### Research Data Loading
|
||||||
|
```bash
|
||||||
|
# Load research datasets
|
||||||
|
datasets=("publications" "authors" "citations")
|
||||||
|
for dataset in "${datasets[@]}"; do
|
||||||
|
tg-load-turtle -i "research-$dataset" \
|
||||||
|
-C "research-data" \
|
||||||
|
"$dataset.ttl"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Structured Data Import
|
||||||
|
```bash
|
||||||
|
# Load structured data from various sources
|
||||||
|
tg-load-turtle -i "products" -C "catalog" product-catalog.ttl
|
||||||
|
tg-load-turtle -i "customers" -C "crm" customer-data.ttl
|
||||||
|
tg-load-turtle -i "orders" -C "transactions" order-history.ttl
|
||||||
|
```
|
||||||
|
|
||||||
|
## Advanced Usage
|
||||||
|
|
||||||
|
### Batch Processing Multiple Files
|
||||||
|
```bash
|
||||||
|
# Process all Turtle files in directory
|
||||||
|
for ttl in *.ttl; do
|
||||||
|
doc_id=$(basename "$ttl" .ttl)
|
||||||
|
echo "Loading $ttl as document $doc_id..."
|
||||||
|
|
||||||
|
tg-load-turtle -i "$doc_id" \
|
||||||
|
-C "bulk-import-$(date +%Y%m%d)" \
|
||||||
|
"$ttl"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Parallel Loading
|
||||||
|
```bash
|
||||||
|
# Load multiple files in parallel
|
||||||
|
ttl_files=(schema.ttl instances.ttl relationships.ttl)
|
||||||
|
for ttl in "${ttl_files[@]}"; do
|
||||||
|
(
|
||||||
|
doc_id=$(basename "$ttl" .ttl)
|
||||||
|
echo "Loading $ttl in background..."
|
||||||
|
tg-load-turtle -i "parallel-$doc_id" \
|
||||||
|
-C "parallel-import" \
|
||||||
|
"$ttl"
|
||||||
|
) &
|
||||||
|
done
|
||||||
|
wait
|
||||||
|
echo "All files loaded"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Size-Based Processing
|
||||||
|
```bash
|
||||||
|
# Handle large files differently
|
||||||
|
for ttl in *.ttl; do
|
||||||
|
size=$(stat -c%s "$ttl")
|
||||||
|
doc_id=$(basename "$ttl" .ttl)
|
||||||
|
|
||||||
|
if [ $size -lt 10485760 ]; then # < 10MB
|
||||||
|
echo "Processing small file: $ttl"
|
||||||
|
tg-load-turtle -i "$doc_id" -C "small-files" "$ttl"
|
||||||
|
else
|
||||||
|
echo "Processing large file: $ttl"
|
||||||
|
# Use dedicated collection for large files
|
||||||
|
tg-load-turtle -i "$doc_id" -C "large-files" "$ttl"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Validation and Loading
|
||||||
|
```bash
|
||||||
|
# Validate before loading
|
||||||
|
validate_and_load() {
|
||||||
|
local ttl_file="$1"
|
||||||
|
local doc_id="$2"
|
||||||
|
|
||||||
|
echo "Validating $ttl_file..."
|
||||||
|
|
||||||
|
# Check Turtle syntax
|
||||||
|
if rapper -q -i turtle "$ttl_file" > /dev/null 2>&1; then
|
||||||
|
echo "✓ Valid Turtle syntax"
|
||||||
|
|
||||||
|
# Count triples
|
||||||
|
triple_count=$(rapper -i turtle -c "$ttl_file" 2>/dev/null)
|
||||||
|
echo " Triples: $triple_count"
|
||||||
|
|
||||||
|
# Load if valid
|
||||||
|
echo "Loading $ttl_file..."
|
||||||
|
tg-load-turtle -i "$doc_id" -C "validated-data" "$ttl_file"
|
||||||
|
else
|
||||||
|
echo "✗ Invalid Turtle syntax in $ttl_file"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Validate and load all files
|
||||||
|
for ttl in *.ttl; do
|
||||||
|
doc_id=$(basename "$ttl" .ttl)
|
||||||
|
validate_and_load "$ttl" "$doc_id"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
### Invalid Turtle Syntax
|
||||||
|
```bash
|
||||||
|
Exception: Turtle parsing failed
|
||||||
|
```
|
||||||
|
**Solution**: Validate Turtle syntax with tools like `rapper` or `rdflib`.
|
||||||
|
|
||||||
|
### Document ID Required
|
||||||
|
```bash
|
||||||
|
Exception: Document ID is required
|
||||||
|
```
|
||||||
|
**Solution**: Provide document ID with `-i` option.
|
||||||
|
|
||||||
|
### WebSocket Connection Issues
|
||||||
|
```bash
|
||||||
|
Exception: WebSocket connection failed
|
||||||
|
```
|
||||||
|
**Solution**: Check API URL and ensure TrustGraph WebSocket service is running.
|
||||||
|
|
||||||
|
### File Not Found
|
||||||
|
```bash
|
||||||
|
Exception: [Errno 2] No such file or directory
|
||||||
|
```
|
||||||
|
**Solution**: Verify file paths and ensure Turtle files exist.
|
||||||
|
|
||||||
|
### Flow Not Found
|
||||||
|
```bash
|
||||||
|
Exception: Flow instance not found
|
||||||
|
```
|
||||||
|
**Solution**: Verify flow ID with `tg-show-flows`.
|
||||||
|
|
||||||
|
## Monitoring and Verification
|
||||||
|
|
||||||
|
### Load Progress Tracking
|
||||||
|
```bash
|
||||||
|
# Monitor loading progress
|
||||||
|
monitor_load() {
|
||||||
|
local ttl_file="$1"
|
||||||
|
local doc_id="$2"
|
||||||
|
|
||||||
|
echo "Starting load: $ttl_file"
|
||||||
|
start_time=$(date +%s)
|
||||||
|
|
||||||
|
tg-load-turtle -i "$doc_id" -C "monitored" "$ttl_file"
|
||||||
|
|
||||||
|
end_time=$(date +%s)
|
||||||
|
duration=$((end_time - start_time))
|
||||||
|
|
||||||
|
echo "Load completed in ${duration}s"
|
||||||
|
|
||||||
|
# Verify data is accessible
|
||||||
|
if tg-triples-query -s "http://example.org/test" > /dev/null 2>&1; then
|
||||||
|
echo "✓ Data accessible via query"
|
||||||
|
else
|
||||||
|
echo "✗ Data not accessible"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data Verification
|
||||||
|
```bash
|
||||||
|
# Verify loaded triples
|
||||||
|
verify_triples() {
|
||||||
|
local collection="$1"
|
||||||
|
local expected_count="$2"
|
||||||
|
|
||||||
|
echo "Verifying triples in collection: $collection"
|
||||||
|
|
||||||
|
# Query for triples
|
||||||
|
actual_count=$(tg-triples-query -C "$collection" | wc -l)
|
||||||
|
|
||||||
|
if [ "$actual_count" -ge "$expected_count" ]; then
|
||||||
|
echo "✓ Expected triples found ($actual_count >= $expected_count)"
|
||||||
|
else
|
||||||
|
echo "✗ Missing triples ($actual_count < $expected_count)"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Content Analysis
|
||||||
|
```bash
|
||||||
|
# Analyze loaded content
|
||||||
|
analyze_turtle_content() {
|
||||||
|
local ttl_file="$1"
|
||||||
|
|
||||||
|
echo "Analyzing content: $ttl_file"
|
||||||
|
|
||||||
|
# Extract prefixes
|
||||||
|
echo "Prefixes:"
|
||||||
|
grep "^@prefix" "$ttl_file" | head -5
|
||||||
|
|
||||||
|
# Count statements
|
||||||
|
statement_count=$(grep -c "\." "$ttl_file")
|
||||||
|
echo "Statements: $statement_count"
|
||||||
|
|
||||||
|
# Extract subjects
|
||||||
|
echo "Sample subjects:"
|
||||||
|
grep -o "^[^[:space:]]*" "$ttl_file" | grep -v "^@" | sort | uniq | head -5
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Performance Optimization
|
||||||
|
|
||||||
|
### Connection Pooling
|
||||||
|
```bash
|
||||||
|
# Reuse WebSocket connections for multiple files
|
||||||
|
load_batch_optimized() {
|
||||||
|
local collection="$1"
|
||||||
|
shift
|
||||||
|
local files=("$@")
|
||||||
|
|
||||||
|
echo "Loading ${#files[@]} files to collection: $collection"
|
||||||
|
|
||||||
|
# Process files in batches to reuse connections
|
||||||
|
for ((i=0; i<${#files[@]}; i+=5)); do
|
||||||
|
batch=("${files[@]:$i:5}")
|
||||||
|
|
||||||
|
echo "Processing batch $((i/5 + 1))..."
|
||||||
|
for ttl in "${batch[@]}"; do
|
||||||
|
doc_id=$(basename "$ttl" .ttl)
|
||||||
|
tg-load-turtle -i "$doc_id" -C "$collection" "$ttl" &
|
||||||
|
done
|
||||||
|
wait
|
||||||
|
done
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Memory Management
|
||||||
|
```bash
|
||||||
|
# Handle large files with memory monitoring
|
||||||
|
load_with_memory_check() {
|
||||||
|
local ttl_file="$1"
|
||||||
|
local doc_id="$2"
|
||||||
|
|
||||||
|
# Check available memory
|
||||||
|
available=$(free -m | awk 'NR==2{print $7}')
|
||||||
|
if [ "$available" -lt 1000 ]; then
|
||||||
|
echo "Warning: Low memory ($available MB). Consider splitting file."
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Monitor memory during load
|
||||||
|
tg-load-turtle -i "$doc_id" -C "memory-monitored" "$ttl_file" &
|
||||||
|
load_pid=$!
|
||||||
|
|
||||||
|
while kill -0 $load_pid 2>/dev/null; do
|
||||||
|
memory_usage=$(ps -p $load_pid -o rss= | awk '{print $1/1024}')
|
||||||
|
echo "Memory usage: ${memory_usage}MB"
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Data Preparation
|
||||||
|
|
||||||
|
### Turtle File Preparation
|
||||||
|
```bash
|
||||||
|
# Clean and prepare Turtle files
|
||||||
|
prepare_turtle() {
|
||||||
|
local input_file="$1"
|
||||||
|
local output_file="$2"
|
||||||
|
|
||||||
|
echo "Preparing $input_file -> $output_file"
|
||||||
|
|
||||||
|
# Remove comments and empty lines
|
||||||
|
sed '/^#/d; /^$/d' "$input_file" > "$output_file"
|
||||||
|
|
||||||
|
# Validate output
|
||||||
|
if rapper -q -i turtle "$output_file" > /dev/null 2>&1; then
|
||||||
|
echo "✓ Prepared file is valid"
|
||||||
|
else
|
||||||
|
echo "✗ Prepared file is invalid"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data Splitting
|
||||||
|
```bash
|
||||||
|
# Split large Turtle files
|
||||||
|
split_turtle() {
|
||||||
|
local input_file="$1"
|
||||||
|
local lines_per_file="$2"
|
||||||
|
|
||||||
|
echo "Splitting $input_file into chunks of $lines_per_file lines"
|
||||||
|
|
||||||
|
# Split file
|
||||||
|
split -l "$lines_per_file" "$input_file" "$(basename "$input_file" .ttl)_part_"
|
||||||
|
|
||||||
|
# Add .ttl extension to parts
|
||||||
|
for part in $(basename "$input_file" .ttl)_part_*; do
|
||||||
|
mv "$part" "$part.ttl"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
- `TRUSTGRAPH_URL`: Default API URL (WebSocket format)
|
||||||
|
|
||||||
|
## Related Commands
|
||||||
|
|
||||||
|
- [`tg-triples-query`](tg-triples-query.md) - Query loaded triples
|
||||||
|
- [`tg-graph-to-turtle`](tg-graph-to-turtle.md) - Export graph to Turtle format
|
||||||
|
- [`tg-show-flows`](tg-show-flows.md) - Monitor processing flows
|
||||||
|
- [`tg-load-pdf`](tg-load-pdf.md) - Load document content
|
||||||
|
|
||||||
|
## API Integration
|
||||||
|
|
||||||
|
This command uses TrustGraph's WebSocket-based triple import API for efficient batch loading of RDF data.
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
1. **Validation**: Always validate Turtle syntax before loading
|
||||||
|
2. **Document IDs**: Use meaningful, unique document identifiers
|
||||||
|
3. **Collections**: Organize triples into logical collections
|
||||||
|
4. **Error Handling**: Implement retry logic for network issues
|
||||||
|
5. **Performance**: Consider file sizes and system resources
|
||||||
|
6. **Monitoring**: Track loading progress and verify results
|
||||||
|
7. **Backup**: Maintain backups of source Turtle files
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### WebSocket Connection Issues
|
||||||
|
```bash
|
||||||
|
# Test WebSocket connectivity
|
||||||
|
wscat -c ws://localhost:8088/api/v1/flow/default/import/triples
|
||||||
|
|
||||||
|
# Check WebSocket service status
|
||||||
|
tg-show-flows | grep -i websocket
|
||||||
|
```
|
||||||
|
|
||||||
|
### Parsing Errors
|
||||||
|
```bash
|
||||||
|
# Validate Turtle syntax
|
||||||
|
rapper -i turtle -q file.ttl
|
||||||
|
|
||||||
|
# Check for common issues
|
||||||
|
grep -n "^[[:space:]]*@prefix" file.ttl # Check prefixes
|
||||||
|
grep -n "\.$" file.ttl | head -5 # Check statement terminators
|
||||||
|
```
|
||||||
|
|
||||||
|
### Memory Issues
|
||||||
|
```bash
|
||||||
|
# Monitor memory usage
|
||||||
|
free -h
|
||||||
|
ps aux | grep tg-load-turtle
|
||||||
|
|
||||||
|
# Split large files if needed
|
||||||
|
split -l 10000 large-file.ttl chunk_
|
||||||
|
```
|
||||||
406
docs/cli/tg-put-flow-class.md
Normal file
406
docs/cli/tg-put-flow-class.md
Normal file
|
|
@ -0,0 +1,406 @@
|
||||||
|
# tg-put-flow-class
|
||||||
|
|
||||||
|
Uploads or updates a flow class definition in TrustGraph.
|
||||||
|
|
||||||
|
## Synopsis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tg-put-flow-class -n CLASS_NAME -c CONFIG_JSON [options]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Description
|
||||||
|
|
||||||
|
The `tg-put-flow-class` command creates or updates a flow class definition in TrustGraph. Flow classes are templates that define processing pipeline configurations, service interfaces, and resource requirements. These classes are used by `tg-start-flow` to create running flow instances.
|
||||||
|
|
||||||
|
Flow classes define the structure and capabilities of processing flows, including which services are available and how they connect to Pulsar queues.
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
### Required Arguments
|
||||||
|
|
||||||
|
- `-n, --class-name CLASS_NAME`: Name for the flow class
|
||||||
|
- `-c, --config CONFIG_JSON`: Flow class configuration as raw JSON string
|
||||||
|
|
||||||
|
### Optional Arguments
|
||||||
|
|
||||||
|
- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
### Basic Flow Class Creation
|
||||||
|
```bash
|
||||||
|
tg-put-flow-class \
|
||||||
|
-n "simple-processing" \
|
||||||
|
-c '{"description": "Simple text processing flow", "interfaces": {"text-completion": {"request": "non-persistent://tg/request/text-completion:simple", "response": "non-persistent://tg/response/text-completion:simple"}}}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Document Processing Flow Class
|
||||||
|
```bash
|
||||||
|
tg-put-flow-class \
|
||||||
|
-n "document-analysis" \
|
||||||
|
-c '{
|
||||||
|
"description": "Document analysis and RAG processing",
|
||||||
|
"interfaces": {
|
||||||
|
"document-rag": {
|
||||||
|
"request": "non-persistent://tg/request/document-rag:doc-analysis",
|
||||||
|
"response": "non-persistent://tg/response/document-rag:doc-analysis"
|
||||||
|
},
|
||||||
|
"text-load": "persistent://tg/flow/text-document-load:doc-analysis",
|
||||||
|
"document-load": "persistent://tg/flow/document-load:doc-analysis"
|
||||||
|
}
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Loading from File
|
||||||
|
```bash
|
||||||
|
# Create configuration file
|
||||||
|
cat > research-flow.json << 'EOF'
|
||||||
|
{
|
||||||
|
"description": "Research analysis flow with multiple AI services",
|
||||||
|
"interfaces": {
|
||||||
|
"agent": {
|
||||||
|
"request": "non-persistent://tg/request/agent:research",
|
||||||
|
"response": "non-persistent://tg/response/agent:research"
|
||||||
|
},
|
||||||
|
"graph-rag": {
|
||||||
|
"request": "non-persistent://tg/request/graph-rag:research",
|
||||||
|
"response": "non-persistent://tg/response/graph-rag:research"
|
||||||
|
},
|
||||||
|
"document-rag": {
|
||||||
|
"request": "non-persistent://tg/request/document-rag:research",
|
||||||
|
"response": "non-persistent://tg/response/document-rag:research"
|
||||||
|
},
|
||||||
|
"embeddings": {
|
||||||
|
"request": "non-persistent://tg/request/embeddings:research",
|
||||||
|
"response": "non-persistent://tg/response/embeddings:research"
|
||||||
|
},
|
||||||
|
"text-load": "persistent://tg/flow/text-document-load:research",
|
||||||
|
"triples-store": "persistent://tg/flow/triples-store:research"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# Upload the flow class
|
||||||
|
tg-put-flow-class -n "research-analysis" -c "$(cat research-flow.json)"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Update Existing Flow Class
|
||||||
|
```bash
|
||||||
|
# Modify existing flow class by adding new service
|
||||||
|
tg-put-flow-class \
|
||||||
|
-n "existing-flow" \
|
||||||
|
-c '{
|
||||||
|
"description": "Updated flow with new capabilities",
|
||||||
|
"interfaces": {
|
||||||
|
"text-completion": {
|
||||||
|
"request": "non-persistent://tg/request/text-completion:updated",
|
||||||
|
"response": "non-persistent://tg/response/text-completion:updated"
|
||||||
|
},
|
||||||
|
"prompt": {
|
||||||
|
"request": "non-persistent://tg/request/prompt:updated",
|
||||||
|
"response": "non-persistent://tg/response/prompt:updated"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Flow Class Configuration Format
|
||||||
|
|
||||||
|
### Required Fields
|
||||||
|
|
||||||
|
#### Description
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"description": "Human-readable description of the flow class"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Interfaces
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"interfaces": {
|
||||||
|
"service-name": "queue-definition-or-object"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Interface Types
|
||||||
|
|
||||||
|
#### Request/Response Services
|
||||||
|
Services that accept requests and return responses:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"service-name": {
|
||||||
|
"request": "pulsar-queue-url",
|
||||||
|
"response": "pulsar-queue-url"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
- `agent`
|
||||||
|
- `graph-rag`
|
||||||
|
- `document-rag`
|
||||||
|
- `text-completion`
|
||||||
|
- `prompt`
|
||||||
|
- `embeddings`
|
||||||
|
- `graph-embeddings`
|
||||||
|
- `triples`
|
||||||
|
|
||||||
|
#### Fire-and-Forget Services
|
||||||
|
Services that accept data without returning responses:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"service-name": "pulsar-queue-url"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
- `text-load`
|
||||||
|
- `document-load`
|
||||||
|
- `triples-store`
|
||||||
|
- `graph-embeddings-store`
|
||||||
|
- `document-embeddings-store`
|
||||||
|
- `entity-contexts-load`
|
||||||
|
|
||||||
|
### Queue Naming Conventions
|
||||||
|
|
||||||
|
#### Request/Response Queues
|
||||||
|
```
|
||||||
|
non-persistent://tg/request/{service}:{flow-identifier}
|
||||||
|
non-persistent://tg/response/{service}:{flow-identifier}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Fire-and-Forget Queues
|
||||||
|
```
|
||||||
|
persistent://tg/flow/{service}:{flow-identifier}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Complete Example
|
||||||
|
|
||||||
|
### Comprehensive Flow Class
|
||||||
|
```bash
|
||||||
|
tg-put-flow-class \
|
||||||
|
-n "full-processing-pipeline" \
|
||||||
|
-c '{
|
||||||
|
"description": "Complete document processing and analysis pipeline",
|
||||||
|
"interfaces": {
|
||||||
|
"agent": {
|
||||||
|
"request": "non-persistent://tg/request/agent:full-pipeline",
|
||||||
|
"response": "non-persistent://tg/response/agent:full-pipeline"
|
||||||
|
},
|
||||||
|
"graph-rag": {
|
||||||
|
"request": "non-persistent://tg/request/graph-rag:full-pipeline",
|
||||||
|
"response": "non-persistent://tg/response/graph-rag:full-pipeline"
|
||||||
|
},
|
||||||
|
"document-rag": {
|
||||||
|
"request": "non-persistent://tg/request/document-rag:full-pipeline",
|
||||||
|
"response": "non-persistent://tg/response/document-rag:full-pipeline"
|
||||||
|
},
|
||||||
|
"text-completion": {
|
||||||
|
"request": "non-persistent://tg/request/text-completion:full-pipeline",
|
||||||
|
"response": "non-persistent://tg/response/text-completion:full-pipeline"
|
||||||
|
},
|
||||||
|
"prompt": {
|
||||||
|
"request": "non-persistent://tg/request/prompt:full-pipeline",
|
||||||
|
"response": "non-persistent://tg/response/prompt:full-pipeline"
|
||||||
|
},
|
||||||
|
"embeddings": {
|
||||||
|
"request": "non-persistent://tg/request/embeddings:full-pipeline",
|
||||||
|
"response": "non-persistent://tg/response/embeddings:full-pipeline"
|
||||||
|
},
|
||||||
|
"graph-embeddings": {
|
||||||
|
"request": "non-persistent://tg/request/graph-embeddings:full-pipeline",
|
||||||
|
"response": "non-persistent://tg/response/graph-embeddings:full-pipeline"
|
||||||
|
},
|
||||||
|
"triples": {
|
||||||
|
"request": "non-persistent://tg/request/triples:full-pipeline",
|
||||||
|
"response": "non-persistent://tg/response/triples:full-pipeline"
|
||||||
|
},
|
||||||
|
"text-load": "persistent://tg/flow/text-document-load:full-pipeline",
|
||||||
|
"document-load": "persistent://tg/flow/document-load:full-pipeline",
|
||||||
|
"triples-store": "persistent://tg/flow/triples-store:full-pipeline",
|
||||||
|
"graph-embeddings-store": "persistent://tg/flow/graph-embeddings-store:full-pipeline",
|
||||||
|
"document-embeddings-store": "persistent://tg/flow/document-embeddings-store:full-pipeline",
|
||||||
|
"entity-contexts-load": "persistent://tg/flow/entity-contexts-load:full-pipeline"
|
||||||
|
}
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Output
|
||||||
|
|
||||||
|
Successful upload typically produces no output:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Upload flow class (no output expected)
|
||||||
|
tg-put-flow-class -n "my-flow" -c '{"description": "test", "interfaces": {}}'
|
||||||
|
|
||||||
|
# Verify upload
|
||||||
|
tg-show-flow-classes | grep "my-flow"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
### Invalid JSON Format
|
||||||
|
```bash
|
||||||
|
Exception: Invalid JSON in config parameter
|
||||||
|
```
|
||||||
|
**Solution**: Validate JSON syntax using tools like `jq` or online JSON validators.
|
||||||
|
|
||||||
|
### Missing Required Fields
|
||||||
|
```bash
|
||||||
|
Exception: Missing required field 'description'
|
||||||
|
```
|
||||||
|
**Solution**: Ensure configuration includes all required fields (description, interfaces).
|
||||||
|
|
||||||
|
### Invalid Queue Names
|
||||||
|
```bash
|
||||||
|
Exception: Invalid queue URL format
|
||||||
|
```
|
||||||
|
**Solution**: Verify queue URLs follow the correct Pulsar format with proper tenant/namespace.
|
||||||
|
|
||||||
|
### Connection Errors
|
||||||
|
```bash
|
||||||
|
Exception: Connection refused
|
||||||
|
```
|
||||||
|
**Solution**: Check the API URL and ensure TrustGraph is running.
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
### JSON Syntax Check
|
||||||
|
```bash
|
||||||
|
# Validate JSON before uploading
|
||||||
|
config='{"description": "test flow", "interfaces": {}}'
|
||||||
|
echo "$config" | jq . > /dev/null && echo "Valid JSON" || echo "Invalid JSON"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Flow Class Verification
|
||||||
|
```bash
|
||||||
|
# After uploading, verify the flow class exists
|
||||||
|
tg-show-flow-classes | grep "my-flow-class"
|
||||||
|
|
||||||
|
# Get the flow class definition to verify content
|
||||||
|
tg-get-flow-class -n "my-flow-class"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Flow Class Lifecycle
|
||||||
|
|
||||||
|
### Development Workflow
|
||||||
|
```bash
|
||||||
|
# 1. Create flow class
|
||||||
|
tg-put-flow-class -n "dev-flow" -c "$dev_config"
|
||||||
|
|
||||||
|
# 2. Test with flow instance
|
||||||
|
tg-start-flow -n "dev-flow" -i "test-instance" -d "Testing"
|
||||||
|
|
||||||
|
# 3. Update flow class as needed
|
||||||
|
tg-put-flow-class -n "dev-flow" -c "$updated_config"
|
||||||
|
|
||||||
|
# 4. Restart flow instance with updates
|
||||||
|
tg-stop-flow -i "test-instance"
|
||||||
|
tg-start-flow -n "dev-flow" -i "test-instance" -d "Testing updated"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
- `TRUSTGRAPH_URL`: Default API URL
|
||||||
|
|
||||||
|
## Related Commands
|
||||||
|
|
||||||
|
- [`tg-get-flow-class`](tg-get-flow-class.md) - Retrieve flow class definitions
|
||||||
|
- [`tg-show-flow-classes`](tg-show-flow-classes.md) - List available flow classes
|
||||||
|
- [`tg-delete-flow-class`](tg-delete-flow-class.md) - Remove flow class definitions
|
||||||
|
- [`tg-start-flow`](tg-start-flow.md) - Create flow instances from classes
|
||||||
|
|
||||||
|
## API Integration
|
||||||
|
|
||||||
|
This command uses the [Flow API](../apis/api-flow.md) with the `put-class` operation to store flow class definitions.
|
||||||
|
|
||||||
|
## Use Cases
|
||||||
|
|
||||||
|
### Custom Processing Pipelines
|
||||||
|
```bash
|
||||||
|
# Create specialized medical analysis flow
|
||||||
|
tg-put-flow-class -n "medical-nlp" -c "$medical_config"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Development Environments
|
||||||
|
```bash
|
||||||
|
# Create lightweight development flow
|
||||||
|
tg-put-flow-class -n "dev-minimal" -c "$minimal_config"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Production Deployments
|
||||||
|
```bash
|
||||||
|
# Create robust production flow with all services
|
||||||
|
tg-put-flow-class -n "production-full" -c "$production_config"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Domain-Specific Workflows
|
||||||
|
```bash
|
||||||
|
# Create legal document analysis flow
|
||||||
|
tg-put-flow-class -n "legal-analysis" -c "$legal_config"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
1. **Descriptive Names**: Use clear, descriptive flow class names
|
||||||
|
2. **Comprehensive Descriptions**: Include detailed descriptions of flow capabilities
|
||||||
|
3. **Consistent Naming**: Follow consistent queue naming conventions
|
||||||
|
4. **Version Control**: Store flow class configurations in version control
|
||||||
|
5. **Testing**: Test flow classes thoroughly before production use
|
||||||
|
6. **Documentation**: Document flow class purposes and requirements
|
||||||
|
|
||||||
|
## Template Examples
|
||||||
|
|
||||||
|
### Minimal Flow Class
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"description": "Minimal text processing flow",
|
||||||
|
"interfaces": {
|
||||||
|
"text-completion": {
|
||||||
|
"request": "non-persistent://tg/request/text-completion:minimal",
|
||||||
|
"response": "non-persistent://tg/response/text-completion:minimal"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### RAG-Focused Flow Class
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"description": "Retrieval Augmented Generation flow",
|
||||||
|
"interfaces": {
|
||||||
|
"graph-rag": {
|
||||||
|
"request": "non-persistent://tg/request/graph-rag:rag-flow",
|
||||||
|
"response": "non-persistent://tg/response/graph-rag:rag-flow"
|
||||||
|
},
|
||||||
|
"document-rag": {
|
||||||
|
"request": "non-persistent://tg/request/document-rag:rag-flow",
|
||||||
|
"response": "non-persistent://tg/response/document-rag:rag-flow"
|
||||||
|
},
|
||||||
|
"embeddings": {
|
||||||
|
"request": "non-persistent://tg/request/embeddings:rag-flow",
|
||||||
|
"response": "non-persistent://tg/response/embeddings:rag-flow"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Document Processing Flow Class
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"description": "Document ingestion and processing flow",
|
||||||
|
"interfaces": {
|
||||||
|
"text-load": "persistent://tg/flow/text-document-load:doc-proc",
|
||||||
|
"document-load": "persistent://tg/flow/document-load:doc-proc",
|
||||||
|
"triples-store": "persistent://tg/flow/triples-store:doc-proc",
|
||||||
|
"embeddings": {
|
||||||
|
"request": "non-persistent://tg/request/embeddings:doc-proc",
|
||||||
|
"response": "non-persistent://tg/response/embeddings:doc-proc"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
330
docs/cli/tg-show-flow-classes.md
Normal file
330
docs/cli/tg-show-flow-classes.md
Normal file
|
|
@ -0,0 +1,330 @@
|
||||||
|
# tg-show-flow-classes
|
||||||
|
|
||||||
|
Lists all defined flow classes in TrustGraph with their descriptions and tags.
|
||||||
|
|
||||||
|
## Synopsis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tg-show-flow-classes [options]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Description
|
||||||
|
|
||||||
|
The `tg-show-flow-classes` command displays a formatted table of all flow class definitions currently stored in TrustGraph. Each flow class is shown with its name, description, and associated tags.
|
||||||
|
|
||||||
|
Flow classes are templates that define the structure and services available for creating flow instances. This command helps you understand what flow classes are available for use.
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
### Optional Arguments
|
||||||
|
|
||||||
|
- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
### List All Flow Classes
|
||||||
|
```bash
|
||||||
|
tg-show-flow-classes
|
||||||
|
```
|
||||||
|
|
||||||
|
Output:
|
||||||
|
```
|
||||||
|
+-----------------+----------------------------------+----------------------+
|
||||||
|
| flow class | description | tags |
|
||||||
|
+-----------------+----------------------------------+----------------------+
|
||||||
|
| document-proc | Document processing pipeline | production, nlp |
|
||||||
|
| data-analysis | Data analysis and visualization | analytics, dev |
|
||||||
|
| web-scraper | Web content extraction flow | scraping, batch |
|
||||||
|
| chat-assistant | Conversational AI assistant | ai, interactive |
|
||||||
|
+-----------------+----------------------------------+----------------------+
|
||||||
|
```
|
||||||
|
|
||||||
|
### Using Custom API URL
|
||||||
|
```bash
|
||||||
|
tg-show-flow-classes -u http://production:8088/
|
||||||
|
```
|
||||||
|
|
||||||
|
### Filter Flow Classes
|
||||||
|
```bash
|
||||||
|
# Show only production-tagged flow classes
|
||||||
|
tg-show-flow-classes | grep "production"
|
||||||
|
|
||||||
|
# Count total flow classes
|
||||||
|
tg-show-flow-classes | grep -c "^|"
|
||||||
|
|
||||||
|
# Show flow classes with specific patterns
|
||||||
|
tg-show-flow-classes | grep -E "(document|text|nlp)"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
The command displays results in a formatted table with columns:
|
||||||
|
|
||||||
|
- **flow class**: The unique name/identifier of the flow class
|
||||||
|
- **description**: Human-readable description of the flow class purpose
|
||||||
|
- **tags**: Comma-separated list of categorization tags
|
||||||
|
|
||||||
|
### Empty Results
|
||||||
|
If no flow classes exist:
|
||||||
|
```
|
||||||
|
No flows.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Use Cases
|
||||||
|
|
||||||
|
### Flow Class Discovery
|
||||||
|
```bash
|
||||||
|
# Find available flow classes for document processing
|
||||||
|
tg-show-flow-classes | grep -i document
|
||||||
|
|
||||||
|
# List all AI-related flow classes
|
||||||
|
tg-show-flow-classes | grep -i "ai\|nlp\|chat\|assistant"
|
||||||
|
|
||||||
|
# Find development vs production flow classes
|
||||||
|
tg-show-flow-classes | grep -E "(dev|test|staging)"
|
||||||
|
tg-show-flow-classes | grep "production"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Flow Class Management
|
||||||
|
```bash
|
||||||
|
# Get list of flow class names for scripting
|
||||||
|
tg-show-flow-classes | awk 'NR>3 && /^\|/ {gsub(/[| ]/, "", $2); print $2}' | grep -v "^$"
|
||||||
|
|
||||||
|
# Check if specific flow class exists
|
||||||
|
if tg-show-flow-classes | grep -q "target-flow"; then
|
||||||
|
echo "Flow class 'target-flow' exists"
|
||||||
|
else
|
||||||
|
echo "Flow class 'target-flow' not found"
|
||||||
|
fi
|
||||||
|
```
|
||||||
|
|
||||||
|
### Environment Comparison
|
||||||
|
```bash
|
||||||
|
# Compare flow classes between environments
|
||||||
|
echo "Development environment:"
|
||||||
|
tg-show-flow-classes -u http://dev:8088/
|
||||||
|
|
||||||
|
echo "Production environment:"
|
||||||
|
tg-show-flow-classes -u http://prod:8088/
|
||||||
|
```
|
||||||
|
|
||||||
|
### Reporting and Documentation
|
||||||
|
```bash
|
||||||
|
# Generate flow class inventory report
|
||||||
|
echo "Flow Class Inventory - $(date)" > flow-inventory.txt
|
||||||
|
echo "=====================================" >> flow-inventory.txt
|
||||||
|
tg-show-flow-classes >> flow-inventory.txt
|
||||||
|
|
||||||
|
# Create CSV export
|
||||||
|
echo "flow_class,description,tags" > flow-classes.csv
|
||||||
|
tg-show-flow-classes | awk 'NR>3 && /^\|/ {
|
||||||
|
gsub(/^\| */, "", $0); gsub(/ *\|$/, "", $0);
|
||||||
|
gsub(/ *\| */, ",", $0); print $0
|
||||||
|
}' >> flow-classes.csv
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
### Connection Errors
|
||||||
|
```bash
|
||||||
|
Exception: Connection refused
|
||||||
|
```
|
||||||
|
**Solution**: Check the API URL and ensure TrustGraph is running.
|
||||||
|
|
||||||
|
### Permission Errors
|
||||||
|
```bash
|
||||||
|
Exception: Access denied to list flow classes
|
||||||
|
```
|
||||||
|
**Solution**: Verify user permissions for reading flow class definitions.
|
||||||
|
|
||||||
|
### Network Timeouts
|
||||||
|
```bash
|
||||||
|
Exception: Request timeout
|
||||||
|
```
|
||||||
|
**Solution**: Check network connectivity and API server status.
|
||||||
|
|
||||||
|
## Integration with Other Commands
|
||||||
|
|
||||||
|
### Flow Class Lifecycle
|
||||||
|
```bash
|
||||||
|
# 1. List available flow classes
|
||||||
|
tg-show-flow-classes
|
||||||
|
|
||||||
|
# 2. Get details of specific flow class
|
||||||
|
tg-get-flow-class -n "interesting-flow"
|
||||||
|
|
||||||
|
# 3. Start flow instance from class
|
||||||
|
tg-start-flow -n "interesting-flow" -i "my-instance"
|
||||||
|
|
||||||
|
# 4. Monitor flow instance
|
||||||
|
tg-show-flows | grep "my-instance"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Bulk Operations
|
||||||
|
```bash
|
||||||
|
# Process all flow classes
|
||||||
|
tg-show-flow-classes | awk 'NR>3 && /^\|/ {gsub(/[| ]/, "", $2); if($2) print $2}' | \
|
||||||
|
while read class_name; do
|
||||||
|
if [ -n "$class_name" ]; then
|
||||||
|
echo "Processing flow class: $class_name"
|
||||||
|
tg-get-flow-class -n "$class_name" > "backup-$class_name.json"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Automated Validation
|
||||||
|
```bash
|
||||||
|
# Check flow class health
|
||||||
|
echo "Validating flow classes..."
|
||||||
|
tg-show-flow-classes | awk 'NR>3 && /^\|/ {gsub(/[| ]/, "", $2); if($2) print $2}' | \
|
||||||
|
while read class_name; do
|
||||||
|
if [ -n "$class_name" ]; then
|
||||||
|
echo -n "Checking $class_name... "
|
||||||
|
if tg-get-flow-class -n "$class_name" > /dev/null 2>&1; then
|
||||||
|
echo "OK"
|
||||||
|
else
|
||||||
|
echo "ERROR"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
## Advanced Usage
|
||||||
|
|
||||||
|
### Flow Class Analysis
|
||||||
|
```bash
|
||||||
|
# Analyze flow class distribution by tags
|
||||||
|
tg-show-flow-classes | awk 'NR>3 && /^\|/ {
|
||||||
|
# Extract tags column
|
||||||
|
split($0, parts, "|");
|
||||||
|
tags = parts[4];
|
||||||
|
gsub(/^ *| *$/, "", tags);
|
||||||
|
if (tags) {
|
||||||
|
split(tags, tag_array, ",");
|
||||||
|
for (i in tag_array) {
|
||||||
|
gsub(/^ *| *$/, "", tag_array[i]);
|
||||||
|
if (tag_array[i]) print tag_array[i];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}' | sort | uniq -c | sort -nr
|
||||||
|
```
|
||||||
|
|
||||||
|
### Environment Synchronization
|
||||||
|
```bash
|
||||||
|
# Sync flow classes between environments
|
||||||
|
echo "Synchronizing flow classes from dev to staging..."
|
||||||
|
|
||||||
|
# Get list from development
|
||||||
|
dev_classes=$(tg-show-flow-classes -u http://dev:8088/ | \
|
||||||
|
awk 'NR>3 && /^\|/ {gsub(/[| ]/, "", $2); if($2) print $2}')
|
||||||
|
|
||||||
|
# Check each class in staging
|
||||||
|
for class in $dev_classes; do
|
||||||
|
if tg-show-flow-classes -u http://staging:8088/ | grep -q "$class"; then
|
||||||
|
echo "$class: Already exists in staging"
|
||||||
|
else
|
||||||
|
echo "$class: Missing in staging - needs sync"
|
||||||
|
# Get from dev and put to staging
|
||||||
|
tg-get-flow-class -n "$class" -u http://dev:8088/ > temp-class.json
|
||||||
|
tg-put-flow-class -n "$class" -c "$(cat temp-class.json)" -u http://staging:8088/
|
||||||
|
rm temp-class.json
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Monitoring Script
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
# monitor-flow-classes.sh
|
||||||
|
api_url="${1:-http://localhost:8088/}"
|
||||||
|
|
||||||
|
echo "Flow Class Monitoring Report - $(date)"
|
||||||
|
echo "API URL: $api_url"
|
||||||
|
echo "----------------------------------------"
|
||||||
|
|
||||||
|
# Total count
|
||||||
|
total=$(tg-show-flow-classes -u "$api_url" | grep -c "^|" 2>/dev/null || echo "0")
|
||||||
|
echo "Total flow classes: $((total - 3))" # Subtract header rows
|
||||||
|
|
||||||
|
# Tag analysis
|
||||||
|
echo -e "\nTag distribution:"
|
||||||
|
tg-show-flow-classes -u "$api_url" | awk 'NR>3 && /^\|/ {
|
||||||
|
split($0, parts, "|");
|
||||||
|
tags = parts[4];
|
||||||
|
gsub(/^ *| *$/, "", tags);
|
||||||
|
if (tags) {
|
||||||
|
split(tags, tag_array, ",");
|
||||||
|
for (i in tag_array) {
|
||||||
|
gsub(/^ *| *$/, "", tag_array[i]);
|
||||||
|
if (tag_array[i]) print tag_array[i];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}' | sort | uniq -c | sort -nr
|
||||||
|
|
||||||
|
# Health check
|
||||||
|
echo -e "\nHealth check:"
|
||||||
|
healthy=0
|
||||||
|
unhealthy=0
|
||||||
|
tg-show-flow-classes -u "$api_url" | awk 'NR>3 && /^\|/ {gsub(/[| ]/, "", $2); if($2) print $2}' | \
|
||||||
|
while read class_name; do
|
||||||
|
if [ -n "$class_name" ]; then
|
||||||
|
if tg-get-flow-class -n "$class_name" -u "$api_url" > /dev/null 2>&1; then
|
||||||
|
healthy=$((healthy + 1))
|
||||||
|
else
|
||||||
|
unhealthy=$((unhealthy + 1))
|
||||||
|
echo " ERROR: $class_name"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "Healthy: $healthy, Unhealthy: $unhealthy"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
- `TRUSTGRAPH_URL`: Default API URL
|
||||||
|
|
||||||
|
## Related Commands
|
||||||
|
|
||||||
|
- [`tg-get-flow-class`](tg-get-flow-class.md) - Retrieve specific flow class definitions
|
||||||
|
- [`tg-put-flow-class`](tg-put-flow-class.md) - Create/update flow class definitions
|
||||||
|
- [`tg-delete-flow-class`](tg-delete-flow-class.md) - Delete flow class definitions
|
||||||
|
- [`tg-start-flow`](tg-start-flow.md) - Create flow instances from classes
|
||||||
|
- [`tg-show-flows`](tg-show-flows.md) - List active flow instances
|
||||||
|
|
||||||
|
## API Integration
|
||||||
|
|
||||||
|
This command uses the [Flow API](../apis/api-flow.md) with the `list-classes` operation to retrieve flow class listings.
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
1. **Regular Inventory**: Periodically review available flow classes
|
||||||
|
2. **Documentation**: Ensure flow classes have meaningful descriptions
|
||||||
|
3. **Tagging**: Use consistent tagging for better organization
|
||||||
|
4. **Cleanup**: Remove unused or deprecated flow classes
|
||||||
|
5. **Monitoring**: Include flow class health checks in monitoring
|
||||||
|
6. **Environment Parity**: Keep flow classes synchronized across environments
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### No Output
|
||||||
|
```bash
|
||||||
|
# If command returns no output, check API connectivity
|
||||||
|
tg-show-flow-classes -u http://localhost:8088/
|
||||||
|
# Verify TrustGraph is running and accessible
|
||||||
|
```
|
||||||
|
|
||||||
|
### Formatting Issues
|
||||||
|
```bash
|
||||||
|
# If table formatting is broken, check terminal width
|
||||||
|
export COLUMNS=120
|
||||||
|
tg-show-flow-classes
|
||||||
|
```
|
||||||
|
|
||||||
|
### Missing Flow Classes
|
||||||
|
```bash
|
||||||
|
# If expected flow classes are missing, verify:
|
||||||
|
# 1. Correct API URL
|
||||||
|
# 2. Database connectivity
|
||||||
|
# 3. Flow class definitions are properly stored
|
||||||
|
```
|
||||||
335
docs/cli/tg-unload-kg-core.md
Normal file
335
docs/cli/tg-unload-kg-core.md
Normal file
|
|
@ -0,0 +1,335 @@
|
||||||
|
# tg-unload-kg-core
|
||||||
|
|
||||||
|
Removes a knowledge core from an active flow without deleting the stored core.
|
||||||
|
|
||||||
|
## Synopsis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tg-unload-kg-core --id CORE_ID [options]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Description
|
||||||
|
|
||||||
|
The `tg-unload-kg-core` command removes a previously loaded knowledge core from an active processing flow, making that knowledge unavailable for queries and processing within that specific flow. The knowledge core remains stored in the system and can be loaded again later or into different flows.
|
||||||
|
|
||||||
|
This is useful for managing flow memory usage, switching knowledge contexts, or temporarily removing knowledge without permanent deletion.
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
### Required Arguments
|
||||||
|
|
||||||
|
- `--id, --identifier CORE_ID`: Identifier of the knowledge core to unload
|
||||||
|
|
||||||
|
### Optional Arguments
|
||||||
|
|
||||||
|
- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`)
|
||||||
|
- `-U, --user USER`: User identifier (default: `trustgraph`)
|
||||||
|
- `-f, --flow-id FLOW`: Flow ID to unload knowledge from (default: `default`)
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
### Unload from Default Flow
|
||||||
|
```bash
|
||||||
|
tg-unload-kg-core --id "research-knowledge"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Unload from Specific Flow
|
||||||
|
```bash
|
||||||
|
tg-unload-kg-core \
|
||||||
|
--id "medical-knowledge" \
|
||||||
|
--flow-id "medical-analysis" \
|
||||||
|
-U medical-team
|
||||||
|
```
|
||||||
|
|
||||||
|
### Unload Multiple Cores
|
||||||
|
```bash
|
||||||
|
# Unload several knowledge cores from a flow
|
||||||
|
tg-unload-kg-core --id "core-1" --flow-id "analysis-flow"
|
||||||
|
tg-unload-kg-core --id "core-2" --flow-id "analysis-flow"
|
||||||
|
tg-unload-kg-core --id "core-3" --flow-id "analysis-flow"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Using Custom API URL
|
||||||
|
```bash
|
||||||
|
tg-unload-kg-core \
|
||||||
|
--id "production-knowledge" \
|
||||||
|
--flow-id "prod-flow" \
|
||||||
|
-u http://production:8088/
|
||||||
|
```
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
### Knowledge Core Must Be Loaded
|
||||||
|
The knowledge core must currently be loaded in the specified flow:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check what's loaded by querying the flow
|
||||||
|
tg-show-graph -f target-flow | head -10
|
||||||
|
|
||||||
|
# If no output, core may not be loaded
|
||||||
|
```
|
||||||
|
|
||||||
|
### Flow Must Be Running
|
||||||
|
The target flow must be active:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check running flows
|
||||||
|
tg-show-flows
|
||||||
|
|
||||||
|
# Verify the target flow exists
|
||||||
|
tg-show-flows | grep "target-flow"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Unloading Process
|
||||||
|
|
||||||
|
1. **Validation**: Verifies knowledge core is loaded in the specified flow
|
||||||
|
2. **Query Termination**: Stops any ongoing queries using the knowledge
|
||||||
|
3. **Index Cleanup**: Removes knowledge indexes from flow context
|
||||||
|
4. **Memory Release**: Frees memory allocated to the knowledge core
|
||||||
|
5. **Service Update**: Updates flow services to reflect knowledge unavailability
|
||||||
|
|
||||||
|
## Effects of Unloading
|
||||||
|
|
||||||
|
### Knowledge Becomes Unavailable
|
||||||
|
After unloading, the knowledge is no longer accessible through the flow:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Before unloading - knowledge available
|
||||||
|
tg-invoke-graph-rag -q "What knowledge is loaded?" -f my-flow
|
||||||
|
|
||||||
|
# Unload the knowledge
|
||||||
|
tg-unload-kg-core --id "my-knowledge" --flow-id "my-flow"
|
||||||
|
|
||||||
|
# After unloading - reduced knowledge available
|
||||||
|
tg-invoke-graph-rag -q "What knowledge is loaded?" -f my-flow
|
||||||
|
```
|
||||||
|
|
||||||
|
### Memory Recovery
|
||||||
|
- RAM used by knowledge indexes is freed
|
||||||
|
- Flow performance may improve
|
||||||
|
- Other knowledge cores in the flow remain unaffected
|
||||||
|
|
||||||
|
### Core Preservation
|
||||||
|
- Knowledge core remains stored in the system
|
||||||
|
- Can be reloaded later
|
||||||
|
- Available for loading into other flows
|
||||||
|
|
||||||
|
## Output
|
||||||
|
|
||||||
|
Successful unloading typically produces no output:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Unload core (no output expected)
|
||||||
|
tg-unload-kg-core --id "test-core" --flow-id "test-flow"
|
||||||
|
|
||||||
|
# Verify unloading by checking available knowledge
|
||||||
|
tg-show-graph -f test-flow | wc -l
|
||||||
|
# Should show fewer triples if core was successfully unloaded
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
### Knowledge Core Not Loaded
|
||||||
|
```bash
|
||||||
|
Exception: Knowledge core 'my-core' not loaded in flow 'my-flow'
|
||||||
|
```
|
||||||
|
**Solution**: Verify the core is actually loaded using `tg-show-graph` or load it first with `tg-load-kg-core`.
|
||||||
|
|
||||||
|
### Flow Not Found
|
||||||
|
```bash
|
||||||
|
Exception: Flow 'invalid-flow' not found
|
||||||
|
```
|
||||||
|
**Solution**: Check running flows with `tg-show-flows` and verify the flow ID.
|
||||||
|
|
||||||
|
### Permission Errors
|
||||||
|
```bash
|
||||||
|
Exception: Access denied to unload knowledge core
|
||||||
|
```
|
||||||
|
**Solution**: Verify user permissions for the knowledge core and flow.
|
||||||
|
|
||||||
|
### Connection Errors
|
||||||
|
```bash
|
||||||
|
Exception: Connection refused
|
||||||
|
```
|
||||||
|
**Solution**: Check the API URL and ensure TrustGraph is running.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
### Check Knowledge Reduction
|
||||||
|
```bash
|
||||||
|
# Count triples before unloading
|
||||||
|
before=$(tg-show-graph -f my-flow | wc -l)
|
||||||
|
|
||||||
|
# Unload knowledge
|
||||||
|
tg-unload-kg-core --id "my-core" --flow-id "my-flow"
|
||||||
|
|
||||||
|
# Count triples after unloading
|
||||||
|
after=$(tg-show-graph -f my-flow | wc -l)
|
||||||
|
|
||||||
|
echo "Triples before: $before, after: $after"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Test Query Impact
|
||||||
|
```bash
|
||||||
|
# Test queries before and after unloading
|
||||||
|
tg-invoke-graph-rag -q "test query" -f my-flow
|
||||||
|
|
||||||
|
# Should work with loaded knowledge
|
||||||
|
tg-unload-kg-core --id "relevant-core" --flow-id "my-flow"
|
||||||
|
|
||||||
|
tg-invoke-graph-rag -q "test query" -f my-flow
|
||||||
|
# May return different results or "no relevant knowledge found"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Use Cases
|
||||||
|
|
||||||
|
### Memory Management
|
||||||
|
```bash
|
||||||
|
# Free up memory by unloading unused knowledge
|
||||||
|
tg-unload-kg-core --id "large-historical-data" --flow-id "analysis-flow"
|
||||||
|
|
||||||
|
# Load more relevant knowledge
|
||||||
|
tg-load-kg-core --id "current-data" --flow-id "analysis-flow"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Context Switching
|
||||||
|
```bash
|
||||||
|
# Switch from medical to legal knowledge context
|
||||||
|
tg-unload-kg-core --id "medical-knowledge" --flow-id "analysis-flow"
|
||||||
|
tg-load-kg-core --id "legal-knowledge" --flow-id "analysis-flow"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Selective Knowledge Loading
|
||||||
|
```bash
|
||||||
|
# Load only specific knowledge for focused analysis
|
||||||
|
tg-unload-kg-core --id "general-knowledge" --flow-id "specialized-flow"
|
||||||
|
tg-load-kg-core --id "domain-specific" --flow-id "specialized-flow"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Testing and Development
|
||||||
|
```bash
|
||||||
|
# Test flow behavior with different knowledge sets
|
||||||
|
tg-unload-kg-core --id "production-data" --flow-id "test-flow"
|
||||||
|
tg-load-kg-core --id "test-data" --flow-id "test-flow"
|
||||||
|
|
||||||
|
# Run tests
|
||||||
|
./run-knowledge-tests.sh
|
||||||
|
|
||||||
|
# Restore production knowledge
|
||||||
|
tg-unload-kg-core --id "test-data" --flow-id "test-flow"
|
||||||
|
tg-load-kg-core --id "production-data" --flow-id "test-flow"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Flow Maintenance
|
||||||
|
```bash
|
||||||
|
# Prepare flow for maintenance by unloading all knowledge
|
||||||
|
cores=$(tg-show-kg-cores)
|
||||||
|
for core in $cores; do
|
||||||
|
tg-unload-kg-core --id "$core" --flow-id "maintenance-flow" 2>/dev/null || true
|
||||||
|
done
|
||||||
|
|
||||||
|
# Perform maintenance
|
||||||
|
./flow-maintenance.sh
|
||||||
|
|
||||||
|
# Reload required knowledge
|
||||||
|
tg-load-kg-core --id "essential-core" --flow-id "maintenance-flow"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Knowledge Management Workflow
|
||||||
|
|
||||||
|
### Dynamic Knowledge Loading
|
||||||
|
```bash
|
||||||
|
# Function to switch knowledge contexts
|
||||||
|
switch_knowledge_context() {
|
||||||
|
local flow_id=$1
|
||||||
|
local old_core=$2
|
||||||
|
local new_core=$3
|
||||||
|
|
||||||
|
echo "Switching from $old_core to $new_core in $flow_id"
|
||||||
|
|
||||||
|
# Unload old knowledge
|
||||||
|
tg-unload-kg-core --id "$old_core" --flow-id "$flow_id"
|
||||||
|
|
||||||
|
# Load new knowledge
|
||||||
|
tg-load-kg-core --id "$new_core" --flow-id "$flow_id"
|
||||||
|
|
||||||
|
echo "Context switch completed"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Usage
|
||||||
|
switch_knowledge_context "analysis-flow" "old-data" "new-data"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Bulk Knowledge Management
|
||||||
|
```bash
|
||||||
|
# Unload all knowledge from a flow
|
||||||
|
unload_all_knowledge() {
|
||||||
|
local flow_id=$1
|
||||||
|
|
||||||
|
# Get list of potentially loaded cores
|
||||||
|
tg-show-kg-cores | while read core; do
|
||||||
|
echo "Attempting to unload $core from $flow_id"
|
||||||
|
tg-unload-kg-core --id "$core" --flow-id "$flow_id" 2>/dev/null || true
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "All knowledge unloaded from $flow_id"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Usage
|
||||||
|
unload_all_knowledge "cleanup-flow"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
- `TRUSTGRAPH_URL`: Default API URL
|
||||||
|
|
||||||
|
## Related Commands
|
||||||
|
|
||||||
|
- [`tg-load-kg-core`](tg-load-kg-core.md) - Load knowledge core into flow
|
||||||
|
- [`tg-show-kg-cores`](tg-show-kg-cores.md) - List available knowledge cores
|
||||||
|
- [`tg-show-graph`](tg-show-graph.md) - View currently loaded knowledge
|
||||||
|
- [`tg-show-flows`](tg-show-flows.md) - List active flows
|
||||||
|
|
||||||
|
## API Integration
|
||||||
|
|
||||||
|
This command uses the [Knowledge API](../apis/api-knowledge.md) with the `unload-kg-core` operation to remove knowledge from active flows.
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
1. **Memory Monitoring**: Monitor flow memory usage when loading/unloading knowledge
|
||||||
|
2. **Graceful Unloading**: Ensure no critical queries are running before unloading
|
||||||
|
3. **Documentation**: Document which knowledge cores are needed for each flow
|
||||||
|
4. **Testing**: Test flow behavior after unloading knowledge
|
||||||
|
5. **Backup Strategy**: Keep knowledge cores stored even when not loaded
|
||||||
|
6. **Performance Optimization**: Unload unused knowledge to improve performance
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Knowledge Still Appears in Queries
|
||||||
|
```bash
|
||||||
|
# If knowledge still appears after unloading
|
||||||
|
# Check if multiple cores contain similar data
|
||||||
|
tg-show-graph -f my-flow | grep "expected-removed-entity"
|
||||||
|
|
||||||
|
# Verify all relevant cores were unloaded
|
||||||
|
```
|
||||||
|
|
||||||
|
### Memory Not Released
|
||||||
|
```bash
|
||||||
|
# If memory usage doesn't decrease after unloading
|
||||||
|
# Check system memory usage
|
||||||
|
free -h
|
||||||
|
|
||||||
|
# Contact system administrator if memory leak suspected
|
||||||
|
```
|
||||||
|
|
||||||
|
### Query Performance Issues
|
||||||
|
```bash
|
||||||
|
# If queries become slow after unloading
|
||||||
|
# May need to reload essential knowledge
|
||||||
|
tg-load-kg-core --id "essential-core" --flow-id "slow-flow"
|
||||||
|
|
||||||
|
# Or restart the flow
|
||||||
|
tg-stop-flow -i "slow-flow"
|
||||||
|
tg-start-flow -n "flow-class" -i "slow-flow" -d "Restarted flow"
|
||||||
|
```
|
||||||
Loading…
Add table
Add a link
Reference in a new issue