trustgraph/docs/tech-specs/structured-diag-service.md

# Structured Data Diagnostic Service Technical Specification

## Overview

This specification describes a new invokable service for diagnosing and analyzing structured data within TrustGraph. The service extracts functionality from the existing `tg-load-structured-data` command-line tool and exposes it as a request/response service, enabling programmatic access to data type detection and descriptor generation capabilities.

The service supports three primary operations:

1. **Data Type Detection**: Analyze a data sample to determine its format (CSV, JSON, or XML)
2. **Descriptor Generation**: Generate a TrustGraph structured data descriptor for a given data sample and type
3. **Combined Diagnosis**: Perform both type detection and descriptor generation in sequence

## Goals

- **Modularize Data Analysis**: Extract data diagnosis logic from CLI into reusable service components
- **Enable Programmatic Access**: Provide API-based access to data analysis capabilities
- **Support Multiple Data Formats**: Handle CSV, JSON, and XML data formats consistently
- **Generate Accurate Descriptors**: Produce structured data descriptors that accurately map source data to TrustGraph schemas
- **Maintain Backward Compatibility**: Ensure existing CLI functionality continues to work
- **Enable Service Composition**: Allow other services to leverage data diagnosis capabilities
- **Improve Testability**: Separate business logic from CLI interface for better testing
- **Support Streaming Analysis**: Enable analysis of data samples without loading entire files

## Background

Currently, the `tg-load-structured-data` command provides comprehensive functionality for analyzing structured data and generating descriptors. However, this functionality is tightly coupled to the CLI interface, limiting its reusability.

Current limitations include:
- Data diagnosis logic embedded in CLI code
- No programmatic access to type detection and descriptor generation
- Difficult to integrate diagnosis capabilities into other services
- Limited ability to compose data analysis workflows

This specification addresses these gaps by creating a dedicated service for structured data diagnosis. By exposing these capabilities as a service, TrustGraph can:
- Enable other services to analyze data programmatically
- Support more complex data processing pipelines
- Facilitate integration with external systems
- Improve maintainability through separation of concerns

## Technical Design

### Architecture

The structured data diagnostic service requires the following technical components:

1. **Diagnostic Service Processor**
   - Handles incoming diagnosis requests
   - Orchestrates type detection and descriptor generation
   - Returns structured responses with diagnosis results

   Module: `trustgraph-flow/trustgraph/diagnosis/structured_data/service.py`

2. **Data Type Detector**
   - Uses algorithmic detection to identify data format (CSV, JSON, XML)
   - Analyzes data structure, delimiters, and syntax patterns
   - Returns detected format and confidence scores

   Module: `trustgraph-flow/trustgraph/diagnosis/structured_data/type_detector.py`

3. **Descriptor Generator**
   - Uses prompt service to generate descriptors
   - Invokes format-specific prompts (diagnose-csv, diagnose-json, diagnose-xml)
   - Maps data fields to TrustGraph schema fields through prompt responses

   Module: `trustgraph-flow/trustgraph/diagnosis/structured_data/descriptor_generator.py`

### Data Models

#### StructuredDataDiagnosisRequest

Request message for structured data diagnosis operations:

```python
class StructuredDataDiagnosisRequest:
    operation: str  # "detect-type", "generate-descriptor", or "diagnose"
    sample: str     # Data sample to analyze (text content)
    type: Optional[str]  # Data type (csv, json, xml) - required for generate-descriptor
    schema_name: Optional[str]  # Target schema name for descriptor generation
    options: Dict[str, Any]  # Additional options (e.g., delimiter for CSV)
```

#### StructuredDataDiagnosisResponse

Response message containing diagnosis results:

```python
class StructuredDataDiagnosisResponse:
    operation: str  # The operation that was performed
    detected_type: Optional[str]  # Detected data type (for detect-type/diagnose)
    confidence: Optional[float]  # Confidence score for type detection
    descriptor: Optional[Dict]  # Generated descriptor (for generate-descriptor/diagnose)
    error: Optional[str]  # Error message if operation failed
    metadata: Dict[str, Any]  # Additional metadata (e.g., field count, sample records)
```

#### Descriptor Structure

The generated descriptor follows the existing structured data descriptor format:

```json
{
  "format": {
    "type": "csv",
    "encoding": "utf-8",
    "options": {
      "delimiter": ",",
      "has_header": true
    }
  },
  "mappings": [
    {
      "source_field": "customer_id",
      "target_field": "id",
      "transforms": [
        {"type": "trim"}
      ]
    }
  ],
  "output": {
    "schema_name": "customer",
    "options": {
      "batch_size": 1000,
      "confidence": 0.9
    }
  }
}
```

### Service Interface

The service will expose the following operations through the request/response pattern:

1. **Type Detection Operation**
   - Input: Data sample
   - Processing: Analyze data structure using algorithmic detection
   - Output: Detected type with confidence score

2. **Descriptor Generation Operation**
   - Input: Data sample, type, target schema name
   - Processing:
     - Call prompt service with format-specific prompt ID (diagnose-csv, diagnose-json, or diagnose-xml)
     - Pass data sample and available schemas to prompt
     - Receive generated descriptor from prompt response
   - Output: Structured data descriptor

3. **Combined Diagnosis Operation**
   - Input: Data sample, optional schema name
   - Processing:
     - Use algorithmic detection to identify format first
     - Select appropriate format-specific prompt based on detected type
     - Call prompt service to generate descriptor
   - Output: Both detected type and descriptor

### Implementation Details

The service will follow TrustGraph service conventions:

1. **Service Registration**
   - Register as `structured-diag` service type
   - Use standard request/response topics
   - Implement FlowProcessor base class
   - Register PromptClientSpec for prompt service interaction

2. **Configuration Management**
   - Access schema configurations via config service
   - Cache schemas for performance
   - Handle configuration updates dynamically

3. **Prompt Integration**
   - Use existing prompt service infrastructure
   - Call prompt service with format-specific prompt IDs:
     - `diagnose-csv`: For CSV data analysis
     - `diagnose-json`: For JSON data analysis
     - `diagnose-xml`: For XML data analysis
   - Prompts are configured in prompt config, not hard-coded in service
   - Pass schemas and data samples as prompt variables
   - Parse prompt responses to extract descriptors

4. **Error Handling**
   - Validate input data samples
   - Provide descriptive error messages
   - Handle malformed data gracefully
   - Handle prompt service failures

5. **Data Sampling**
   - Process configurable sample sizes
   - Handle incomplete records appropriately
   - Maintain sampling consistency

### API Integration

The service will integrate with existing TrustGraph APIs:

Modified Components:
- `tg-load-structured-data` CLI - Refactored to use the new service for diagnosis operations
- Flow API - Extended to support structured data diagnosis requests

New Service Endpoints:
- `/api/v1/flow/{flow}/diagnose/structured-data` - WebSocket endpoint for diagnosis requests
- `/api/v1/diagnose/structured-data` - REST endpoint for synchronous diagnosis

### Message Flow

```
Client → Gateway → Structured Diag Service → Config Service (for schemas)
                                           ↓
                                    Type Detector (algorithmic)
                                           ↓
                                    Prompt Service (diagnose-csv/json/xml)
                                           ↓
                                 Descriptor Generator (parses prompt response)
                                           ↓
Client ← Gateway ← Structured Diag Service (response)
```

## Security Considerations

- Input validation to prevent injection attacks
- Size limits on data samples to prevent DoS
- Sanitization of generated descriptors
- Access control through existing TrustGraph authentication

## Performance Considerations

- Cache schema definitions to reduce config service calls
- Limit sample sizes to maintain responsive performance
- Use streaming processing for large data samples
- Implement timeout mechanisms for long-running analyses

## Testing Strategy

1. **Unit Tests**
   - Type detection for various data formats
   - Descriptor generation accuracy
   - Error handling scenarios

2. **Integration Tests**
   - Service request/response flow
   - Schema retrieval and caching
   - CLI integration

3. **Performance Tests**
   - Large sample processing
   - Concurrent request handling
   - Memory usage under load

## Migration Plan

1. **Phase 1**: Implement service with core functionality
2. **Phase 2**: Refactor CLI to use service (maintain backward compatibility)
3. **Phase 3**: Add REST API endpoints
4. **Phase 4**: Deprecate embedded CLI logic (with notice period)

## Timeline

- Week 1-2: Implement core service and type detection
- Week 3-4: Add descriptor generation and integration
- Week 5: Testing and documentation
- Week 6: CLI refactoring and migration

## Open Questions

- Should the service support additional data formats (e.g., Parquet, Avro)?
- What should be the maximum sample size for analysis?
- Should diagnosis results be cached for repeated requests?
- How should the service handle multi-schema scenarios?
- Should the prompt IDs be configurable parameters for the service?

## References

- [Structured Data Descriptor Specification](structured-data-descriptor.md)
- [Structured Data Loading Documentation](structured-data.md)
- `tg-load-structured-data` implementation: `trustgraph-cli/trustgraph/cli/load_structured_data.py`