diff --git a/docs/tech-specs/structured-diag-service.md b/docs/tech-specs/structured-diag-service.md new file mode 100644 index 00000000..1eab9df2 --- /dev/null +++ b/docs/tech-specs/structured-diag-service.md @@ -0,0 +1,273 @@ +# Structured Data Diagnostic Service Technical Specification + +## Overview + +This specification describes a new invokable service for diagnosing and analyzing structured data within TrustGraph. The service extracts functionality from the existing `tg-load-structured-data` command-line tool and exposes it as a request/response service, enabling programmatic access to data type detection and descriptor generation capabilities. + +The service supports three primary operations: + +1. **Data Type Detection**: Analyze a data sample to determine its format (CSV, JSON, or XML) +2. **Descriptor Generation**: Generate a TrustGraph structured data descriptor for a given data sample and type +3. **Combined Diagnosis**: Perform both type detection and descriptor generation in sequence + +## Goals + +- **Modularize Data Analysis**: Extract data diagnosis logic from CLI into reusable service components +- **Enable Programmatic Access**: Provide API-based access to data analysis capabilities +- **Support Multiple Data Formats**: Handle CSV, JSON, and XML data formats consistently +- **Generate Accurate Descriptors**: Produce structured data descriptors that accurately map source data to TrustGraph schemas +- **Maintain Backward Compatibility**: Ensure existing CLI functionality continues to work +- **Enable Service Composition**: Allow other services to leverage data diagnosis capabilities +- **Improve Testability**: Separate business logic from CLI interface for better testing +- **Support Streaming Analysis**: Enable analysis of data samples without loading entire files + +## Background + +Currently, the `tg-load-structured-data` command provides comprehensive functionality for analyzing structured data and generating descriptors. However, this functionality is tightly coupled to the CLI interface, limiting its reusability. + +Current limitations include: +- Data diagnosis logic embedded in CLI code +- No programmatic access to type detection and descriptor generation +- Difficult to integrate diagnosis capabilities into other services +- Limited ability to compose data analysis workflows + +This specification addresses these gaps by creating a dedicated service for structured data diagnosis. By exposing these capabilities as a service, TrustGraph can: +- Enable other services to analyze data programmatically +- Support more complex data processing pipelines +- Facilitate integration with external systems +- Improve maintainability through separation of concerns + +## Technical Design + +### Architecture + +The structured data diagnostic service requires the following technical components: + +1. **Diagnostic Service Processor** + - Handles incoming diagnosis requests + - Orchestrates type detection and descriptor generation + - Returns structured responses with diagnosis results + + Module: `trustgraph-flow/trustgraph/diagnosis/structured_data/service.py` + +2. **Data Type Detector** + - Uses algorithmic detection to identify data format (CSV, JSON, XML) + - Analyzes data structure, delimiters, and syntax patterns + - Returns detected format and confidence scores + + Module: `trustgraph-flow/trustgraph/diagnosis/structured_data/type_detector.py` + +3. **Descriptor Generator** + - Uses prompt service to generate descriptors + - Invokes format-specific prompts (diagnose-csv, diagnose-json, diagnose-xml) + - Maps data fields to TrustGraph schema fields through prompt responses + + Module: `trustgraph-flow/trustgraph/diagnosis/structured_data/descriptor_generator.py` + +### Data Models + +#### StructuredDataDiagnosisRequest + +Request message for structured data diagnosis operations: + +```python +class StructuredDataDiagnosisRequest: + operation: str # "detect-type", "generate-descriptor", or "diagnose" + sample: str # Data sample to analyze (text content) + type: Optional[str] # Data type (csv, json, xml) - required for generate-descriptor + schema_name: Optional[str] # Target schema name for descriptor generation + options: Dict[str, Any] # Additional options (e.g., delimiter for CSV) +``` + +#### StructuredDataDiagnosisResponse + +Response message containing diagnosis results: + +```python +class StructuredDataDiagnosisResponse: + operation: str # The operation that was performed + detected_type: Optional[str] # Detected data type (for detect-type/diagnose) + confidence: Optional[float] # Confidence score for type detection + descriptor: Optional[Dict] # Generated descriptor (for generate-descriptor/diagnose) + error: Optional[str] # Error message if operation failed + metadata: Dict[str, Any] # Additional metadata (e.g., field count, sample records) +``` + +#### Descriptor Structure + +The generated descriptor follows the existing structured data descriptor format: + +```json +{ + "format": { + "type": "csv", + "encoding": "utf-8", + "options": { + "delimiter": ",", + "has_header": true + } + }, + "mappings": [ + { + "source_field": "customer_id", + "target_field": "id", + "transforms": [ + {"type": "trim"} + ] + } + ], + "output": { + "schema_name": "customer", + "options": { + "batch_size": 1000, + "confidence": 0.9 + } + } +} +``` + +### Service Interface + +The service will expose the following operations through the request/response pattern: + +1. **Type Detection Operation** + - Input: Data sample + - Processing: Analyze data structure using algorithmic detection + - Output: Detected type with confidence score + +2. **Descriptor Generation Operation** + - Input: Data sample, type, target schema name + - Processing: + - Call prompt service with format-specific prompt ID (diagnose-csv, diagnose-json, or diagnose-xml) + - Pass data sample and available schemas to prompt + - Receive generated descriptor from prompt response + - Output: Structured data descriptor + +3. **Combined Diagnosis Operation** + - Input: Data sample, optional schema name + - Processing: + - Use algorithmic detection to identify format first + - Select appropriate format-specific prompt based on detected type + - Call prompt service to generate descriptor + - Output: Both detected type and descriptor + +### Implementation Details + +The service will follow TrustGraph service conventions: + +1. **Service Registration** + - Register as `structured-diag` service type + - Use standard request/response topics + - Implement FlowProcessor base class + - Register PromptClientSpec for prompt service interaction + +2. **Configuration Management** + - Access schema configurations via config service + - Cache schemas for performance + - Handle configuration updates dynamically + +3. **Prompt Integration** + - Use existing prompt service infrastructure + - Call prompt service with format-specific prompt IDs: + - `diagnose-csv`: For CSV data analysis + - `diagnose-json`: For JSON data analysis + - `diagnose-xml`: For XML data analysis + - Prompts are configured in prompt config, not hard-coded in service + - Pass schemas and data samples as prompt variables + - Parse prompt responses to extract descriptors + +4. **Error Handling** + - Validate input data samples + - Provide descriptive error messages + - Handle malformed data gracefully + - Handle prompt service failures + +5. **Data Sampling** + - Process configurable sample sizes + - Handle incomplete records appropriately + - Maintain sampling consistency + +### API Integration + +The service will integrate with existing TrustGraph APIs: + +Modified Components: +- `tg-load-structured-data` CLI - Refactored to use the new service for diagnosis operations +- Flow API - Extended to support structured data diagnosis requests + +New Service Endpoints: +- `/api/v1/flow/{flow}/diagnose/structured-data` - WebSocket endpoint for diagnosis requests +- `/api/v1/diagnose/structured-data` - REST endpoint for synchronous diagnosis + +### Message Flow + +``` +Client → Gateway → Structured Diag Service → Config Service (for schemas) + ↓ + Type Detector (algorithmic) + ↓ + Prompt Service (diagnose-csv/json/xml) + ↓ + Descriptor Generator (parses prompt response) + ↓ +Client ← Gateway ← Structured Diag Service (response) +``` + +## Security Considerations + +- Input validation to prevent injection attacks +- Size limits on data samples to prevent DoS +- Sanitization of generated descriptors +- Access control through existing TrustGraph authentication + +## Performance Considerations + +- Cache schema definitions to reduce config service calls +- Limit sample sizes to maintain responsive performance +- Use streaming processing for large data samples +- Implement timeout mechanisms for long-running analyses + +## Testing Strategy + +1. **Unit Tests** + - Type detection for various data formats + - Descriptor generation accuracy + - Error handling scenarios + +2. **Integration Tests** + - Service request/response flow + - Schema retrieval and caching + - CLI integration + +3. **Performance Tests** + - Large sample processing + - Concurrent request handling + - Memory usage under load + +## Migration Plan + +1. **Phase 1**: Implement service with core functionality +2. **Phase 2**: Refactor CLI to use service (maintain backward compatibility) +3. **Phase 3**: Add REST API endpoints +4. **Phase 4**: Deprecate embedded CLI logic (with notice period) + +## Timeline + +- Week 1-2: Implement core service and type detection +- Week 3-4: Add descriptor generation and integration +- Week 5: Testing and documentation +- Week 6: CLI refactoring and migration + +## Open Questions + +- Should the service support additional data formats (e.g., Parquet, Avro)? +- What should be the maximum sample size for analysis? +- Should diagnosis results be cached for repeated requests? +- How should the service handle multi-schema scenarios? +- Should the prompt IDs be configurable parameters for the service? + +## References + +- [Structured Data Descriptor Specification](structured-data-descriptor.md) +- [Structured Data Loading Documentation](structured-data.md) +- `tg-load-structured-data` implementation: `trustgraph-cli/trustgraph/cli/load_structured_data.py` \ No newline at end of file