diff --git a/docs/cli/README.md b/docs/cli/README.md new file mode 100644 index 00000000..95fc83ff --- /dev/null +++ b/docs/cli/README.md @@ -0,0 +1,166 @@ +# TrustGraph CLI Documentation + +The TrustGraph Command Line Interface (CLI) provides comprehensive command-line access to all TrustGraph services. These tools wrap the REST and WebSocket APIs to provide convenient, scriptable access to TrustGraph functionality. + +## Installation + +The CLI tools are installed as part of the `trustgraph-cli` package: + +```bash +pip install trustgraph-cli +``` + +## Global Options + +Most CLI commands support these common options: + +- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`) +- `-U, --user USER`: User identifier (default: `trustgraph`) +- `-C, --collection COLLECTION`: Collection identifier (default: `default`) +- `-f, --flow-id FLOW`: Flow identifier (default: `default`) + +## Command Categories + +### System Administration & Configuration + +**System Setup:** +- [`tg-init-trustgraph`](tg-init-trustgraph.md) - Initialize Pulsar with TrustGraph configuration +- [`tg-init-pulsar-manager`](tg-init-pulsar-manager.md) - Initialize Pulsar manager setup +- [`tg-show-config`](tg-show-config.md) - Display current system configuration + +**Token Management:** +- [`tg-set-token-costs`](tg-set-token-costs.md) - Configure model token costs +- [`tg-show-token-costs`](tg-show-token-costs.md) - Display token cost configuration +- [`tg-show-token-rate`](tg-show-token-rate.md) - Show token usage rates + +### Flow Management + +**Flow Operations:** +- [`tg-start-flow`](tg-start-flow.md) - Start a processing flow +- [`tg-stop-flow`](tg-stop-flow.md) - Stop a running flow +- [`tg-show-flows`](tg-show-flows.md) - List all configured flows +- [`tg-show-flow-state`](tg-show-flow-state.md) - Show current flow states + +**Flow Class Management:** +- [`tg-put-flow-class`](tg-put-flow-class.md) - Upload/update flow class definition +- [`tg-get-flow-class`](tg-get-flow-class.md) - Retrieve flow class definition +- [`tg-delete-flow-class`](tg-delete-flow-class.md) - Remove flow class definition +- [`tg-show-flow-classes`](tg-show-flow-classes.md) - List available flow classes + +### Knowledge Graph Management + +**Knowledge Core Operations:** +- [`tg-load-kg-core`](tg-load-kg-core.md) - Load knowledge core into processing +- [`tg-put-kg-core`](tg-put-kg-core.md) - Store knowledge core in system +- [`tg-get-kg-core`](tg-get-kg-core.md) - Retrieve knowledge core +- [`tg-delete-kg-core`](tg-delete-kg-core.md) - Remove knowledge core +- [`tg-unload-kg-core`](tg-unload-kg-core.md) - Unload knowledge core from processing +- [`tg-show-kg-cores`](tg-show-kg-cores.md) - List available knowledge cores + +**Graph Data Operations:** +- [`tg-show-graph`](tg-show-graph.md) - Display graph triples/edges +- [`tg-graph-to-turtle`](tg-graph-to-turtle.md) - Export graph to Turtle format +- [`tg-load-turtle`](tg-load-turtle.md) - Import RDF triples from Turtle files + +### Document Processing & Library Management + +**Document Loading:** +- [`tg-load-pdf`](tg-load-pdf.md) - Load PDF documents into processing +- [`tg-load-text`](tg-load-text.md) - Load text documents into processing +- [`tg-load-sample-documents`](tg-load-sample-documents.md) - Load sample documents for testing + +**Library Management:** +- [`tg-add-library-document`](tg-add-library-document.md) - Add documents to library +- [`tg-remove-library-document`](tg-remove-library-document.md) - Remove documents from library +- [`tg-show-library-documents`](tg-show-library-documents.md) - List library documents +- [`tg-start-library-processing`](tg-start-library-processing.md) - Start library document processing +- [`tg-stop-library-processing`](tg-stop-library-processing.md) - Stop library document processing +- [`tg-show-library-processing`](tg-show-library-processing.md) - Show library processing status + +**Document Embeddings:** +- [`tg-load-doc-embeds`](tg-load-doc-embeds.md) - Load document embeddings +- [`tg-save-doc-embeds`](tg-save-doc-embeds.md) - Save document embeddings + +### AI Services & Agent Interaction + +**Query & Interaction:** +- [`tg-invoke-agent`](tg-invoke-agent.md) - Interactive agent Q&A via WebSocket +- [`tg-invoke-llm`](tg-invoke-llm.md) - Direct LLM text completion +- [`tg-invoke-prompt`](tg-invoke-prompt.md) - Use configured prompt templates +- [`tg-invoke-document-rag`](tg-invoke-document-rag.md) - Document-based RAG queries +- [`tg-invoke-graph-rag`](tg-invoke-graph-rag.md) - Graph-based RAG queries + +**Tool & Prompt Management:** +- [`tg-show-tools`](tg-show-tools.md) - List available agent tools +- [`tg-set-prompt`](tg-set-prompt.md) - Configure prompt templates +- [`tg-show-prompts`](tg-show-prompts.md) - List configured prompts + +### System Monitoring & Debugging + +**System Status:** +- [`tg-show-processor-state`](tg-show-processor-state.md) - Show processing component states + +**Debugging:** +- [`tg-dump-msgpack`](tg-dump-msgpack.md) - Dump MessagePack data for debugging + +## Quick Start Examples + +### Basic Document Processing +```bash +# Start a flow +tg-start-flow --flow-id my-flow --class-name document-processing + +# Load a document +tg-load-text --flow-id my-flow --text "Your document content" --title "Test Document" + +# Query the knowledge +tg-invoke-graph-rag --flow-id my-flow --query "What is the document about?" +``` + +### Knowledge Management +```bash +# List available knowledge cores +tg-show-kg-cores + +# Load a knowledge core into a flow +tg-load-kg-core --flow-id my-flow --kg-core-id my-knowledge + +# Query the knowledge graph +tg-show-graph --limit 100 +``` + +### Flow Management +```bash +# Show available flow classes +tg-show-flow-classes + +# Show running flows +tg-show-flows + +# Stop a flow +tg-stop-flow --flow-id my-flow +``` + +## Environment Variables + +- `TRUSTGRAPH_URL`: Default API URL for all commands +- `TRUSTGRAPH_USER`: Default user identifier +- `TRUSTGRAPH_COLLECTION`: Default collection identifier + +## Authentication + +CLI commands inherit authentication from the environment or API configuration. See the main TrustGraph documentation for authentication setup. + +## Error Handling + +All CLI commands provide: +- Consistent error reporting +- Exit codes (0 for success, non-zero for errors) +- Detailed error messages for troubleshooting +- Retry logic for network operations where appropriate + +## Related Documentation + +- [TrustGraph API Documentation](../apis/README.md) +- [TrustGraph WebSocket Guide](../apis/websocket.md) +- [TrustGraph Pulsar Guide](../apis/pulsar.md) \ No newline at end of file diff --git a/docs/cli/tg-invoke-agent.md b/docs/cli/tg-invoke-agent.md new file mode 100644 index 00000000..e3423fe1 --- /dev/null +++ b/docs/cli/tg-invoke-agent.md @@ -0,0 +1,163 @@ +# tg-invoke-agent + +Uses the agent service to answer a question via interactive WebSocket connection. + +## Synopsis + +```bash +tg-invoke-agent -q "your question" [options] +``` + +## Description + +The `tg-invoke-agent` command provides an interactive interface to TrustGraph's agent service. It connects via WebSocket to submit questions and receive real-time responses, including the agent's thinking process and observations when verbose mode is enabled. + +The agent uses available tools and knowledge sources to answer questions, providing a conversational AI interface to your TrustGraph knowledge base. + +## Options + +### Required Arguments + +- `-q, --question QUESTION`: The question to ask the agent + +### Optional Arguments + +- `-u, --url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `ws://localhost:8088/`) +- `-f, --flow-id FLOW`: Flow ID to use (default: `default`) +- `-U, --user USER`: User identifier (default: `trustgraph`) +- `-C, --collection COLLECTION`: Collection identifier (default: `default`) +- `-l, --plan PLAN`: Agent plan specification (optional) +- `-s, --state STATE`: Agent initial state (optional) +- `-v, --verbose`: Output agent's thinking process and observations + +## Examples + +### Basic Question +```bash +tg-invoke-agent -q "What is machine learning?" +``` + +### Verbose Output with Thinking Process +```bash +tg-invoke-agent -q "Explain the benefits of neural networks" -v +``` + +### Using Specific Flow +```bash +tg-invoke-agent -q "What documents are available?" -f research-flow +``` + +### With Custom User and Collection +```bash +tg-invoke-agent -q "Show me recent papers" -U alice -C research-papers +``` + +### Using Custom API URL +```bash +tg-invoke-agent -q "What is AI?" -u ws://production:8088/ +``` + +## Output Format + +### Standard Output +The agent provides direct answers to your questions: + +``` +AI stands for Artificial Intelligence, which refers to computer systems that can perform tasks typically requiring human intelligence. +``` + +### Verbose Output +With `-v` flag, you see the agent's thinking process: + +``` +❓ What is machine learning? + +🤔 I need to provide a comprehensive explanation of machine learning, including its definition, key concepts, and applications. + +💡 Let me search for information about machine learning in the knowledge base. + +Machine learning is a subset of artificial intelligence that enables computers to learn and improve automatically from experience without being explicitly programmed... +``` + +The emoji indicators represent: +- ❓ Your question +- 🤔 Agent's thinking/reasoning +- 💡 Agent's observations from tools/searches + +## Error Handling + +Common errors and solutions: + +### Connection Errors +```bash +Exception: Connection refused +``` +**Solution**: Verify the API URL and ensure TrustGraph is running. + +### Flow Not Found +```bash +Exception: Invalid flow +``` +**Solution**: Check that the specified flow exists and is running using `tg-show-flows`. + +### Authentication Errors +```bash +Exception: Unauthorized +``` +**Solution**: Verify your authentication credentials and permissions. + +## Environment Variables + +- `TRUSTGRAPH_URL`: Default API URL (converted to WebSocket URL automatically) + +## Related Commands + +- [`tg-invoke-graph-rag`](tg-invoke-graph-rag.md) - Graph-based retrieval augmented generation +- [`tg-invoke-document-rag`](tg-invoke-document-rag.md) - Document-based retrieval augmented generation +- [`tg-invoke-llm`](tg-invoke-llm.md) - Direct LLM text completion +- [`tg-show-tools`](tg-show-tools.md) - List available agent tools +- [`tg-show-flows`](tg-show-flows.md) - List available flows + +## Technical Details + +### WebSocket Communication +The command uses WebSocket protocol for real-time communication with the agent service. The URL is automatically converted from HTTP to WebSocket format. + +### Message Format +Messages are exchanged in JSON format: + +**Request:** +```json +{ + "id": "unique-message-id", + "service": "agent", + "flow": "flow-id", + "request": { + "question": "your question" + } +} +``` + +**Response:** +```json +{ + "id": "unique-message-id", + "response": { + "thought": "agent thinking", + "observation": "agent observation", + "answer": "final answer" + }, + "complete": true +} +``` + +### API Integration +This command uses the [Agent API](../apis/api-agent.md) via WebSocket connection for real-time interaction. + +## Use Cases + +- **Interactive Q&A**: Ask questions about your knowledge base +- **Research Assistance**: Get help analyzing documents and data +- **Knowledge Discovery**: Explore connections in your data +- **Troubleshooting**: Get help with technical issues using verbose mode +- **Educational**: Learn about topics in your knowledge base \ No newline at end of file diff --git a/docs/cli/tg-load-text.md b/docs/cli/tg-load-text.md new file mode 100644 index 00000000..765cb80a --- /dev/null +++ b/docs/cli/tg-load-text.md @@ -0,0 +1,211 @@ +# tg-load-text + +Loads text documents into TrustGraph processing pipelines with rich metadata support. + +## Synopsis + +```bash +tg-load-text [options] file1 [file2 ...] +``` + +## Description + +The `tg-load-text` command loads text documents into TrustGraph for processing. It creates a SHA256 hash-based document ID and supports comprehensive metadata including copyright information, publication details, and keywords. + +**Note**: Consider using `tg-add-library-document` followed by `tg-start-library-processing` for better document management and processing control. + +## Options + +### Connection & Flow +- `-u, --url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`) +- `-f, --flow-id FLOW`: Flow ID for processing (default: `default`) +- `-U, --user USER`: User identifier (default: `trustgraph`) +- `-C, --collection COLLECTION`: Collection identifier (default: `default`) + +### Document Metadata +- `--name NAME`: Document name/title +- `--description DESCRIPTION`: Document description +- `--document-url URL`: Document source URL + +### Copyright Information +- `--copyright-notice NOTICE`: Copyright notice text +- `--copyright-holder HOLDER`: Copyright holder name +- `--copyright-year YEAR`: Copyright year +- `--license LICENSE`: Copyright license + +### Publication Information +- `--publication-organization ORG`: Publishing organization +- `--publication-description DESC`: Publication description +- `--publication-date DATE`: Publication date + +### Keywords +- `--keyword KEYWORD [KEYWORD ...]`: Document keywords (can specify multiple) + +## Arguments + +- `file1 [file2 ...]`: One or more text files to load + +## Examples + +### Basic Document Loading +```bash +tg-load-text document.txt +``` + +### Loading with Metadata +```bash +tg-load-text \ + --name "Research Paper on AI" \ + --description "Comprehensive study of machine learning algorithms" \ + --keyword "AI" "machine learning" "research" \ + research-paper.txt +``` + +### Complete Metadata Example +```bash +tg-load-text \ + --name "TrustGraph Documentation" \ + --description "Complete user guide for TrustGraph system" \ + --copyright-holder "TrustGraph Project" \ + --copyright-year "2024" \ + --license "MIT" \ + --publication-organization "TrustGraph Foundation" \ + --publication-date "2024-01-15" \ + --keyword "documentation" "guide" "tutorial" \ + --flow-id research-flow \ + trustgraph-guide.txt +``` + +### Multiple Files +```bash +tg-load-text chapter1.txt chapter2.txt chapter3.txt +``` + +### Custom Flow and Collection +```bash +tg-load-text \ + --flow-id medical-research \ + --user researcher \ + --collection medical-papers \ + medical-study.txt +``` + +## Output + +For each file processed, the command outputs: + +### Success +``` +document.txt: Loaded successfully. +``` + +### Failure +``` +document.txt: Failed: Connection refused +``` + +## Document Processing + +1. **File Reading**: Reads the text file content +2. **Hash Generation**: Creates SHA256 hash for unique document ID +3. **URI Creation**: Converts hash to document URI format +4. **Metadata Assembly**: Combines all metadata into RDF triples +5. **API Submission**: Sends to TrustGraph via Text Load API + +## Document ID Generation + +Documents are assigned IDs based on their content hash: +- SHA256 hash of file content +- Converted to TrustGraph document URI format +- Example: `http://trustgraph.ai/d/abc123...` + +## Metadata Format + +The metadata is stored as RDF triples including: + +### Standard Properties +- `dc:title`: Document name +- `dc:description`: Document description +- `dc:creator`: Copyright holder +- `dc:date`: Publication date +- `dc:rights`: Copyright notice +- `dc:license`: License information + +### Keywords +- `dc:subject`: Each keyword as separate triple + +### Organization Information +- `foaf:Organization`: Publication organization details + +## Error Handling + +### File Errors +```bash +document.txt: Failed: No such file or directory +``` +**Solution**: Verify the file path exists and is readable. + +### Connection Errors +```bash +document.txt: Failed: Connection refused +``` +**Solution**: Check the API URL and ensure TrustGraph is running. + +### Flow Errors +```bash +document.txt: Failed: Invalid flow +``` +**Solution**: Verify the flow exists and is running using `tg-show-flows`. + +## Environment Variables + +- `TRUSTGRAPH_URL`: Default API URL + +## Related Commands + +- [`tg-add-library-document`](tg-add-library-document.md) - Add documents to library (recommended) +- [`tg-load-pdf`](tg-load-pdf.md) - Load PDF documents +- [`tg-show-library-documents`](tg-show-library-documents.md) - List loaded documents +- [`tg-start-library-processing`](tg-start-library-processing.md) - Start document processing + +## API Integration + +This command uses the [Text Load API](../apis/api-text-load.md) to submit documents for processing. The text content is base64-encoded for transmission. + +## Use Cases + +### Academic Research +```bash +tg-load-text \ + --name "Climate Change Impact Study" \ + --publication-organization "University Research Center" \ + --keyword "climate" "research" "environment" \ + climate-study.txt +``` + +### Corporate Documentation +```bash +tg-load-text \ + --name "Product Manual" \ + --copyright-holder "Acme Corp" \ + --license "Proprietary" \ + --keyword "manual" "product" "guide" \ + product-manual.txt +``` + +### Technical Documentation +```bash +tg-load-text \ + --name "API Reference" \ + --description "Complete API documentation" \ + --keyword "API" "reference" "technical" \ + api-docs.txt +``` + +## Best Practices + +1. **Use Descriptive Names**: Provide clear document names and descriptions +2. **Add Keywords**: Include relevant keywords for better searchability +3. **Complete Metadata**: Fill in copyright and publication information +4. **Batch Processing**: Load multiple related files together +5. **Use Collections**: Organize documents by topic or project using collections \ No newline at end of file diff --git a/docs/cli/tg-show-config.md b/docs/cli/tg-show-config.md new file mode 100644 index 00000000..2fa3c64c --- /dev/null +++ b/docs/cli/tg-show-config.md @@ -0,0 +1,170 @@ +# tg-show-config + +Displays the current TrustGraph system configuration. + +## Synopsis + +```bash +tg-show-config [options] +``` + +## Description + +The `tg-show-config` command retrieves and displays the complete TrustGraph system configuration in JSON format. This includes flow definitions, service configurations, and other system settings stored in the configuration service. + +This is particularly useful for: +- Understanding the current system setup +- Debugging configuration issues +- Finding queue names for Pulsar integration +- Verifying flow definitions and interfaces + +## Options + +- `-u, --api-url URL`: TrustGraph API URL (default: `$TRUSTGRAPH_URL` or `http://localhost:8088/`) + +## Examples + +### Display Complete Configuration +```bash +tg-show-config +``` + +### Using Custom API URL +```bash +tg-show-config -u http://production:8088/ +``` + +## Output Format + +The command outputs the configuration version followed by the complete configuration in JSON format: + +``` +Version: 42 +{ + "flows": { + "default": { + "class-name": "document-rag+graph-rag", + "description": "Default processing flow", + "interfaces": { + "agent": { + "request": "non-persistent://tg/request/agent:default", + "response": "non-persistent://tg/response/agent:default" + }, + "graph-rag": { + "request": "non-persistent://tg/request/graph-rag:document-rag+graph-rag", + "response": "non-persistent://tg/response/graph-rag:document-rag+graph-rag" + }, + "text-load": "persistent://tg/flow/text-document-load:default", + ... + } + } + }, + "prompts": { + "system": "You are a helpful AI assistant...", + "graph-rag": "Answer the question using the provided context..." + }, + "token-costs": { + "gpt-4": { + "prompt": 0.03, + "completion": 0.06 + } + }, + ... +} +``` + +## Configuration Sections + +### Flow Definitions +Flow configurations showing: +- **class-name**: The flow class being used +- **description**: Human-readable flow description +- **interfaces**: Pulsar queue names for each service + +### Prompt Templates +System and service-specific prompt templates used by AI services. + +### Token Costs +Model pricing information for cost tracking and billing. + +### Service Settings +Various service-specific configuration parameters. + +## Finding Queue Names + +The configuration output is essential for discovering Pulsar queue names: + +### Flow-Hosted Services +Look in the `flows` section under `interfaces`: + +```json +"graph-rag": { + "request": "non-persistent://tg/request/graph-rag:document-rag+graph-rag", + "response": "non-persistent://tg/response/graph-rag:document-rag+graph-rag" +} +``` + +### Fire-and-Forget Services +Some services only have input queues: + +```json +"text-load": "persistent://tg/flow/text-document-load:default" +``` + +## Error Handling + +### Connection Errors +```bash +Exception: Connection refused +``` +**Solution**: Verify the API URL and ensure TrustGraph is running. + +### Authentication Errors +```bash +Exception: Unauthorized +``` +**Solution**: Check authentication credentials and permissions. + +## Environment Variables + +- `TRUSTGRAPH_URL`: Default API URL + +## Related Commands + +- [`tg-put-flow-class`](tg-put-flow-class.md) - Update flow class definitions +- [`tg-show-flows`](tg-show-flows.md) - List active flows +- [`tg-set-prompt`](tg-set-prompt.md) - Configure prompt templates +- [`tg-set-token-costs`](tg-set-token-costs.md) - Configure token costs + +## API Integration + +This command uses the [Config API](../apis/api-config.md) with the `config` operation to retrieve the complete system configuration. + +**API Call:** +```json +{ + "operation": "config" +} +``` + +## Use Cases + +### Development and Debugging +- Verify flow configurations are correct +- Check that services have proper queue assignments +- Debug configuration-related issues + +### System Administration +- Monitor configuration changes over time +- Document current system setup +- Prepare for system migrations + +### Integration Development +- Discover Pulsar queue names for direct integration +- Understand service interfaces and capabilities +- Verify API endpoint configurations + +### Troubleshooting +- Check if flows are properly configured +- Verify prompt templates are set correctly +- Confirm token cost configurations \ No newline at end of file