mirror of
https://github.com/trustgraph-ai/trustgraph.git
synced 2026-04-25 00:16:23 +02:00
Update docs for API/CLI changes in 1.0 (#421)
* Update some API basics for the 0.23/1.0 API change
This commit is contained in:
cybermaggedon
GitHub
parent
f907ea7db8
commit
44bdd29f51
69 changed files with 19981 additions and 407 deletions
523
docs/cli/tg-init-trustgraph.md
Normal file
523
docs/cli/tg-init-trustgraph.md
Normal file
|
|
@ -0,0 +1,523 @@
|
|||
# tg-init-trustgraph
|
||||
|
||||
Initializes Pulsar with TrustGraph tenant, namespaces, and configuration settings.
|
||||
|
||||
## Synopsis
|
||||
|
||||
```bash
|
||||
tg-init-trustgraph [options]
|
||||
```
|
||||
|
||||
## Description
|
||||
|
||||
The `tg-init-trustgraph` command initializes the Apache Pulsar messaging system with the required tenant, namespaces, policies, and configuration needed for TrustGraph operation. This is a foundational setup command that must be run before TrustGraph can operate properly.
|
||||
|
||||
The command creates the necessary Pulsar infrastructure and optionally loads initial configuration data into the system.
|
||||
|
||||
## Options
|
||||
|
||||
### Optional Arguments
|
||||
|
||||
- `-p, --pulsar-admin-url URL`: Pulsar admin URL (default: `http://pulsar:8080`)
|
||||
- `--pulsar-host HOST`: Pulsar host for client connections (default: `pulsar://pulsar:6650`)
|
||||
- `--pulsar-api-key KEY`: Pulsar API key for authentication
|
||||
- `-c, --config CONFIG`: Initial configuration JSON to load
|
||||
- `-t, --tenant TENANT`: Tenant name (default: `tg`)
|
||||
|
||||
## Examples
|
||||
|
||||
### Basic Initialization
|
||||
```bash
|
||||
tg-init-trustgraph
|
||||
```
|
||||
|
||||
### Custom Pulsar Configuration
|
||||
```bash
|
||||
tg-init-trustgraph \
|
||||
--pulsar-admin-url http://localhost:8080 \
|
||||
--pulsar-host pulsar://localhost:6650
|
||||
```
|
||||
|
||||
### With Initial Configuration
|
||||
```bash
|
||||
tg-init-trustgraph \
|
||||
--config '{"prompt": {"system": "You are a helpful AI assistant"}}'
|
||||
```
|
||||
|
||||
### Custom Tenant
|
||||
```bash
|
||||
tg-init-trustgraph --tenant production-tg
|
||||
```
|
||||
|
||||
### Production Setup
|
||||
```bash
|
||||
tg-init-trustgraph \
|
||||
--pulsar-admin-url http://pulsar-cluster:8080 \
|
||||
--pulsar-host pulsar://pulsar-cluster:6650 \
|
||||
--pulsar-api-key "your-api-key" \
|
||||
--tenant production \
|
||||
--config "$(cat production-config.json)"
|
||||
```
|
||||
|
||||
## What It Creates
|
||||
|
||||
### Tenant Structure
|
||||
The command creates a TrustGraph tenant with the following namespaces:
|
||||
|
||||
#### Flow Namespace (`tg/flow`)
|
||||
- **Purpose**: Processing workflows and flow definitions
|
||||
- **Retention**: Default retention policies
|
||||
|
||||
#### Request Namespace (`tg/request`)
|
||||
- **Purpose**: Incoming API requests and commands
|
||||
- **Retention**: Default retention policies
|
||||
|
||||
#### Response Namespace (`tg/response`)
|
||||
- **Purpose**: API responses and results
|
||||
- **Retention**: 3 minutes, unlimited size
|
||||
- **Subscription Expiration**: 30 minutes
|
||||
|
||||
#### Config Namespace (`tg/config`)
|
||||
- **Purpose**: System configuration and settings
|
||||
- **Retention**: 10MB size limit, unlimited time
|
||||
- **Subscription Expiration**: 5 minutes
|
||||
|
||||
### Configuration Loading
|
||||
|
||||
If a configuration is provided, the command also:
|
||||
1. Connects to the configuration service
|
||||
2. Loads the provided configuration data
|
||||
3. Ensures configuration versioning is maintained
|
||||
|
||||
## Configuration Format
|
||||
|
||||
The configuration should be provided as JSON with this structure:
|
||||
|
||||
```json
|
||||
{
|
||||
"prompt": {
|
||||
"system": "System prompt text",
|
||||
"template-index": ["template1", "template2"],
|
||||
"template.template1": {
|
||||
"id": "template1",
|
||||
"prompt": "Template text with {{variables}}",
|
||||
"response-type": "text"
|
||||
}
|
||||
},
|
||||
"token-costs": {
|
||||
"gpt-4": {
|
||||
"input_price": 0.00003,
|
||||
"output_price": 0.00006
|
||||
}
|
||||
},
|
||||
"agent": {
|
||||
"tool-index": ["tool1"],
|
||||
"tool.tool1": {
|
||||
"id": "tool1",
|
||||
"name": "Example Tool",
|
||||
"description": "Tool description",
|
||||
"arguments": []
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
|
||||
### Initial Deployment
|
||||
```bash
|
||||
# Complete TrustGraph initialization sequence
|
||||
initialize_trustgraph() {
|
||||
echo "Initializing TrustGraph infrastructure..."
|
||||
|
||||
# Wait for Pulsar to be ready
|
||||
wait_for_pulsar
|
||||
|
||||
# Initialize Pulsar Manager (if using)
|
||||
tg-init-pulsar-manager
|
||||
|
||||
# Initialize TrustGraph
|
||||
tg-init-trustgraph \
|
||||
--config "$(cat initial-config.json)"
|
||||
|
||||
echo "TrustGraph initialization complete!"
|
||||
}
|
||||
|
||||
wait_for_pulsar() {
|
||||
local timeout=300
|
||||
local elapsed=0
|
||||
|
||||
while ! curl -s http://pulsar:8080/admin/v2/clusters > /dev/null; do
|
||||
if [ $elapsed -ge $timeout ]; then
|
||||
echo "Timeout waiting for Pulsar"
|
||||
exit 1
|
||||
fi
|
||||
echo "Waiting for Pulsar..."
|
||||
sleep 5
|
||||
elapsed=$((elapsed + 5))
|
||||
done
|
||||
}
|
||||
```
|
||||
|
||||
### Environment-Specific Setup
|
||||
```bash
|
||||
# Development environment
|
||||
setup_dev() {
|
||||
tg-init-trustgraph \
|
||||
--pulsar-admin-url http://localhost:8080 \
|
||||
--pulsar-host pulsar://localhost:6650 \
|
||||
--tenant dev \
|
||||
--config "$(cat dev-config.json)"
|
||||
}
|
||||
|
||||
# Staging environment
|
||||
setup_staging() {
|
||||
tg-init-trustgraph \
|
||||
--pulsar-admin-url http://staging-pulsar:8080 \
|
||||
--pulsar-host pulsar://staging-pulsar:6650 \
|
||||
--tenant staging \
|
||||
--config "$(cat staging-config.json)"
|
||||
}
|
||||
|
||||
# Production environment
|
||||
setup_production() {
|
||||
tg-init-trustgraph \
|
||||
--pulsar-admin-url http://prod-pulsar:8080 \
|
||||
--pulsar-host pulsar://prod-pulsar:6650 \
|
||||
--pulsar-api-key "$PULSAR_API_KEY" \
|
||||
--tenant production \
|
||||
--config "$(cat production-config.json)"
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration Management
|
||||
```bash
|
||||
# Load different configurations
|
||||
load_ai_config() {
|
||||
local config='{
|
||||
"prompt": {
|
||||
"system": "You are an AI assistant specialized in data analysis.",
|
||||
"template-index": ["analyze", "summarize"],
|
||||
"template.analyze": {
|
||||
"id": "analyze",
|
||||
"prompt": "Analyze this data: {{data}}",
|
||||
"response-type": "json"
|
||||
}
|
||||
},
|
||||
"token-costs": {
|
||||
"gpt-4": {"input_price": 0.00003, "output_price": 0.00006},
|
||||
"claude-3-sonnet": {"input_price": 0.000003, "output_price": 0.000015}
|
||||
}
|
||||
}'
|
||||
|
||||
tg-init-trustgraph --config "$config"
|
||||
}
|
||||
|
||||
load_research_config() {
|
||||
local config='{
|
||||
"prompt": {
|
||||
"system": "You are a research assistant focused on academic literature.",
|
||||
"template-index": ["research", "citation"],
|
||||
"template.research": {
|
||||
"id": "research",
|
||||
"prompt": "Research question: {{question}}\nContext: {{context}}",
|
||||
"response-type": "text"
|
||||
}
|
||||
}
|
||||
}'
|
||||
|
||||
tg-init-trustgraph --config "$config"
|
||||
}
|
||||
```
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Cluster Setup
|
||||
```bash
|
||||
# Multi-cluster initialization
|
||||
setup_cluster() {
|
||||
local clusters=("cluster1:8080" "cluster2:8080" "cluster3:8080")
|
||||
|
||||
for cluster in "${clusters[@]}"; do
|
||||
echo "Initializing cluster: $cluster"
|
||||
|
||||
tg-init-trustgraph \
|
||||
--pulsar-admin-url "http://$cluster" \
|
||||
--pulsar-host "pulsar://${cluster%:*}:6650" \
|
||||
--tenant "cluster-$(echo $cluster | cut -d: -f1)" \
|
||||
--config "$(cat cluster-config.json)"
|
||||
done
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration Migration
|
||||
```bash
|
||||
# Migrate configuration between environments
|
||||
migrate_config() {
|
||||
local source_env="$1"
|
||||
local target_env="$2"
|
||||
|
||||
echo "Migrating configuration from $source_env to $target_env"
|
||||
|
||||
# Export existing configuration (would need a tg-export-config command)
|
||||
# For now, assume we have the config in a file
|
||||
|
||||
tg-init-trustgraph \
|
||||
--pulsar-admin-url "http://$target_env:8080" \
|
||||
--pulsar-host "pulsar://$target_env:6650" \
|
||||
--config "$(cat ${source_env}-config.json)"
|
||||
}
|
||||
```
|
||||
|
||||
### Validation and Testing
|
||||
```bash
|
||||
# Validate initialization
|
||||
validate_initialization() {
|
||||
local tenant="${1:-tg}"
|
||||
local admin_url="${2:-http://pulsar:8080}"
|
||||
|
||||
echo "Validating TrustGraph initialization..."
|
||||
|
||||
# Check tenant exists
|
||||
if curl -s "$admin_url/admin/v2/tenants/$tenant" > /dev/null; then
|
||||
echo "✓ Tenant '$tenant' exists"
|
||||
else
|
||||
echo "✗ Tenant '$tenant' missing"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Check namespaces
|
||||
local namespaces=("flow" "request" "response" "config")
|
||||
for ns in "${namespaces[@]}"; do
|
||||
if curl -s "$admin_url/admin/v2/namespaces/$tenant/$ns" > /dev/null; then
|
||||
echo "✓ Namespace '$tenant/$ns' exists"
|
||||
else
|
||||
echo "✗ Namespace '$tenant/$ns' missing"
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
|
||||
echo "✓ TrustGraph initialization validated"
|
||||
}
|
||||
|
||||
# Test configuration loading
|
||||
test_config_loading() {
|
||||
local test_config='{
|
||||
"test": {
|
||||
"value": "test-value",
|
||||
"timestamp": "'$(date -Iseconds)'"
|
||||
}
|
||||
}'
|
||||
|
||||
echo "Testing configuration loading..."
|
||||
|
||||
if tg-init-trustgraph --config "$test_config"; then
|
||||
echo "✓ Configuration loading successful"
|
||||
else
|
||||
echo "✗ Configuration loading failed"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
```
|
||||
|
||||
### Retry Logic and Error Handling
|
||||
```bash
|
||||
# Robust initialization with retry
|
||||
robust_init() {
|
||||
local max_attempts=5
|
||||
local attempt=1
|
||||
local delay=10
|
||||
|
||||
while [ $attempt -le $max_attempts ]; do
|
||||
echo "Initialization attempt $attempt of $max_attempts..."
|
||||
|
||||
if tg-init-trustgraph "$@"; then
|
||||
echo "✓ Initialization successful on attempt $attempt"
|
||||
return 0
|
||||
else
|
||||
echo "✗ Attempt $attempt failed"
|
||||
|
||||
if [ $attempt -lt $max_attempts ]; then
|
||||
echo "Waiting ${delay}s before retry..."
|
||||
sleep $delay
|
||||
delay=$((delay * 2)) # Exponential backoff
|
||||
fi
|
||||
fi
|
||||
|
||||
attempt=$((attempt + 1))
|
||||
done
|
||||
|
||||
echo "✗ All initialization attempts failed"
|
||||
return 1
|
||||
}
|
||||
```
|
||||
|
||||
## Docker Integration
|
||||
|
||||
### Docker Compose
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
pulsar:
|
||||
image: apachepulsar/pulsar:latest
|
||||
ports:
|
||||
- "6650:6650"
|
||||
- "8080:8080"
|
||||
command: bin/pulsar standalone
|
||||
|
||||
trustgraph-init:
|
||||
image: trustgraph/cli:latest
|
||||
depends_on:
|
||||
- pulsar
|
||||
volumes:
|
||||
- ./config.json:/config.json:ro
|
||||
command: >
|
||||
sh -c "
|
||||
sleep 30 &&
|
||||
tg-init-trustgraph --config '$$(cat /config.json)'
|
||||
"
|
||||
environment:
|
||||
- TRUSTGRAPH_PULSAR_ADMIN_URL=http://pulsar:8080
|
||||
- TRUSTGRAPH_PULSAR_HOST=pulsar://pulsar:6650
|
||||
```
|
||||
|
||||
### Kubernetes Init Container
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: trustgraph-config
|
||||
data:
|
||||
config.json: |
|
||||
{
|
||||
"prompt": {
|
||||
"system": "You are a helpful AI assistant."
|
||||
}
|
||||
}
|
||||
---
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: trustgraph-init
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
initContainers:
|
||||
- name: wait-for-pulsar
|
||||
image: busybox
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
until nc -z pulsar 8080; do
|
||||
echo "Waiting for Pulsar..."
|
||||
sleep 5
|
||||
done
|
||||
containers:
|
||||
- name: init
|
||||
image: trustgraph/cli:latest
|
||||
command:
|
||||
- tg-init-trustgraph
|
||||
- --pulsar-admin-url=http://pulsar:8080
|
||||
- --pulsar-host=pulsar://pulsar:6650
|
||||
- --config=$(cat /config/config.json)
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: trustgraph-config
|
||||
restartPolicy: Never
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Connection Issues
|
||||
```bash
|
||||
Exception: Connection refused
|
||||
```
|
||||
**Solution**: Verify Pulsar is running and accessible at the specified admin URL.
|
||||
|
||||
### Authentication Errors
|
||||
```bash
|
||||
Exception: 401 Unauthorized
|
||||
```
|
||||
**Solution**: Check Pulsar API key if authentication is enabled.
|
||||
|
||||
### Tenant Creation Failures
|
||||
```bash
|
||||
Exception: Tenant creation failed
|
||||
```
|
||||
**Solution**: Verify admin permissions and cluster configuration.
|
||||
|
||||
### Configuration Loading Errors
|
||||
```bash
|
||||
Exception: Invalid JSON configuration
|
||||
```
|
||||
**Solution**: Validate JSON syntax and structure.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### API Key Management
|
||||
```bash
|
||||
# Use environment variables for sensitive data
|
||||
export PULSAR_API_KEY="your-secure-api-key"
|
||||
tg-init-trustgraph --pulsar-api-key "$PULSAR_API_KEY"
|
||||
|
||||
# Or use a secure file
|
||||
tg-init-trustgraph --pulsar-api-key "$(cat /secure/pulsar-key.txt)"
|
||||
```
|
||||
|
||||
### Network Security
|
||||
```bash
|
||||
# Use TLS for production
|
||||
tg-init-trustgraph \
|
||||
--pulsar-admin-url https://secure-pulsar:8443 \
|
||||
--pulsar-host pulsar+ssl://secure-pulsar:6651
|
||||
```
|
||||
|
||||
## Related Commands
|
||||
|
||||
- [`tg-init-pulsar-manager`](tg-init-pulsar-manager.md) - Initialize Pulsar Manager
|
||||
- [`tg-show-config`](tg-show-config.md) - Display current configuration
|
||||
- [`tg-set-prompt`](tg-set-prompt.md) - Configure individual prompts
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Run Once**: Typically run once per environment during initial setup
|
||||
2. **Idempotent**: Safe to run multiple times - existing resources are preserved
|
||||
3. **Configuration**: Always load initial configuration during setup
|
||||
4. **Validation**: Verify initialization success with validation scripts
|
||||
5. **Environment Variables**: Use environment variables for sensitive configuration
|
||||
6. **Retry Logic**: Implement retry logic for robust deployments
|
||||
7. **Monitoring**: Monitor namespace and topic creation for issues
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Pulsar Not Ready
|
||||
```bash
|
||||
# Check Pulsar health
|
||||
curl http://pulsar:8080/admin/v2/clusters
|
||||
|
||||
# Check Pulsar logs
|
||||
docker logs pulsar
|
||||
```
|
||||
|
||||
### Permission Issues
|
||||
```bash
|
||||
# Verify Pulsar admin access
|
||||
curl http://pulsar:8080/admin/v2/tenants
|
||||
|
||||
# Check API key validity if using authentication
|
||||
```
|
||||
|
||||
### Configuration Validation
|
||||
```bash
|
||||
# Validate JSON configuration
|
||||
echo "$CONFIG" | jq .
|
||||
|
||||
# Test configuration loading separately
|
||||
tg-init-trustgraph --config '{"test": "value"}'
|
||||
```
|
||||
Loading…
Add table
Add a link
Reference in a new issue