mirror of
https://github.com/trustgraph-ai/trustgraph.git
synced 2026-04-25 08:26:21 +02:00
Structure data diagnosis service (#518)
* Import flow tech spec * Structured diag service * Plumbed into API gateway * Type detector * Diag service * Added entry point
This commit is contained in:
cybermaggedon
GitHub
parent
d73af56690
commit
3d783f4bd4
13 changed files with 1201 additions and 3 deletions
273
docs/tech-specs/structured-diag-service.md
Normal file
273
docs/tech-specs/structured-diag-service.md
Normal file
|
|
@ -0,0 +1,273 @@
|
|||
# Structured Data Diagnostic Service Technical Specification
|
||||
|
||||
## Overview
|
||||
|
||||
This specification describes a new invokable service for diagnosing and analyzing structured data within TrustGraph. The service extracts functionality from the existing `tg-load-structured-data` command-line tool and exposes it as a request/response service, enabling programmatic access to data type detection and descriptor generation capabilities.
|
||||
|
||||
The service supports three primary operations:
|
||||
|
||||
1. **Data Type Detection**: Analyze a data sample to determine its format (CSV, JSON, or XML)
|
||||
2. **Descriptor Generation**: Generate a TrustGraph structured data descriptor for a given data sample and type
|
||||
3. **Combined Diagnosis**: Perform both type detection and descriptor generation in sequence
|
||||
|
||||
## Goals
|
||||
|
||||
- **Modularize Data Analysis**: Extract data diagnosis logic from CLI into reusable service components
|
||||
- **Enable Programmatic Access**: Provide API-based access to data analysis capabilities
|
||||
- **Support Multiple Data Formats**: Handle CSV, JSON, and XML data formats consistently
|
||||
- **Generate Accurate Descriptors**: Produce structured data descriptors that accurately map source data to TrustGraph schemas
|
||||
- **Maintain Backward Compatibility**: Ensure existing CLI functionality continues to work
|
||||
- **Enable Service Composition**: Allow other services to leverage data diagnosis capabilities
|
||||
- **Improve Testability**: Separate business logic from CLI interface for better testing
|
||||
- **Support Streaming Analysis**: Enable analysis of data samples without loading entire files
|
||||
|
||||
## Background
|
||||
|
||||
Currently, the `tg-load-structured-data` command provides comprehensive functionality for analyzing structured data and generating descriptors. However, this functionality is tightly coupled to the CLI interface, limiting its reusability.
|
||||
|
||||
Current limitations include:
|
||||
- Data diagnosis logic embedded in CLI code
|
||||
- No programmatic access to type detection and descriptor generation
|
||||
- Difficult to integrate diagnosis capabilities into other services
|
||||
- Limited ability to compose data analysis workflows
|
||||
|
||||
This specification addresses these gaps by creating a dedicated service for structured data diagnosis. By exposing these capabilities as a service, TrustGraph can:
|
||||
- Enable other services to analyze data programmatically
|
||||
- Support more complex data processing pipelines
|
||||
- Facilitate integration with external systems
|
||||
- Improve maintainability through separation of concerns
|
||||
|
||||
## Technical Design
|
||||
|
||||
### Architecture
|
||||
|
||||
The structured data diagnostic service requires the following technical components:
|
||||
|
||||
1. **Diagnostic Service Processor**
|
||||
- Handles incoming diagnosis requests
|
||||
- Orchestrates type detection and descriptor generation
|
||||
- Returns structured responses with diagnosis results
|
||||
|
||||
Module: `trustgraph-flow/trustgraph/diagnosis/structured_data/service.py`
|
||||
|
||||
2. **Data Type Detector**
|
||||
- Uses algorithmic detection to identify data format (CSV, JSON, XML)
|
||||
- Analyzes data structure, delimiters, and syntax patterns
|
||||
- Returns detected format and confidence scores
|
||||
|
||||
Module: `trustgraph-flow/trustgraph/diagnosis/structured_data/type_detector.py`
|
||||
|
||||
3. **Descriptor Generator**
|
||||
- Uses prompt service to generate descriptors
|
||||
- Invokes format-specific prompts (diagnose-csv, diagnose-json, diagnose-xml)
|
||||
- Maps data fields to TrustGraph schema fields through prompt responses
|
||||
|
||||
Module: `trustgraph-flow/trustgraph/diagnosis/structured_data/descriptor_generator.py`
|
||||
|
||||
### Data Models
|
||||
|
||||
#### StructuredDataDiagnosisRequest
|
||||
|
||||
Request message for structured data diagnosis operations:
|
||||
|
||||
```python
|
||||
class StructuredDataDiagnosisRequest:
|
||||
operation: str # "detect-type", "generate-descriptor", or "diagnose"
|
||||
sample: str # Data sample to analyze (text content)
|
||||
type: Optional[str] # Data type (csv, json, xml) - required for generate-descriptor
|
||||
schema_name: Optional[str] # Target schema name for descriptor generation
|
||||
options: Dict[str, Any] # Additional options (e.g., delimiter for CSV)
|
||||
```
|
||||
|
||||
#### StructuredDataDiagnosisResponse
|
||||
|
||||
Response message containing diagnosis results:
|
||||
|
||||
```python
|
||||
class StructuredDataDiagnosisResponse:
|
||||
operation: str # The operation that was performed
|
||||
detected_type: Optional[str] # Detected data type (for detect-type/diagnose)
|
||||
confidence: Optional[float] # Confidence score for type detection
|
||||
descriptor: Optional[Dict] # Generated descriptor (for generate-descriptor/diagnose)
|
||||
error: Optional[str] # Error message if operation failed
|
||||
metadata: Dict[str, Any] # Additional metadata (e.g., field count, sample records)
|
||||
```
|
||||
|
||||
#### Descriptor Structure
|
||||
|
||||
The generated descriptor follows the existing structured data descriptor format:
|
||||
|
||||
```json
|
||||
{
|
||||
"format": {
|
||||
"type": "csv",
|
||||
"encoding": "utf-8",
|
||||
"options": {
|
||||
"delimiter": ",",
|
||||
"has_header": true
|
||||
}
|
||||
},
|
||||
"mappings": [
|
||||
{
|
||||
"source_field": "customer_id",
|
||||
"target_field": "id",
|
||||
"transforms": [
|
||||
{"type": "trim"}
|
||||
]
|
||||
}
|
||||
],
|
||||
"output": {
|
||||
"schema_name": "customer",
|
||||
"options": {
|
||||
"batch_size": 1000,
|
||||
"confidence": 0.9
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Service Interface
|
||||
|
||||
The service will expose the following operations through the request/response pattern:
|
||||
|
||||
1. **Type Detection Operation**
|
||||
- Input: Data sample
|
||||
- Processing: Analyze data structure using algorithmic detection
|
||||
- Output: Detected type with confidence score
|
||||
|
||||
2. **Descriptor Generation Operation**
|
||||
- Input: Data sample, type, target schema name
|
||||
- Processing:
|
||||
- Call prompt service with format-specific prompt ID (diagnose-csv, diagnose-json, or diagnose-xml)
|
||||
- Pass data sample and available schemas to prompt
|
||||
- Receive generated descriptor from prompt response
|
||||
- Output: Structured data descriptor
|
||||
|
||||
3. **Combined Diagnosis Operation**
|
||||
- Input: Data sample, optional schema name
|
||||
- Processing:
|
||||
- Use algorithmic detection to identify format first
|
||||
- Select appropriate format-specific prompt based on detected type
|
||||
- Call prompt service to generate descriptor
|
||||
- Output: Both detected type and descriptor
|
||||
|
||||
### Implementation Details
|
||||
|
||||
The service will follow TrustGraph service conventions:
|
||||
|
||||
1. **Service Registration**
|
||||
- Register as `structured-diag` service type
|
||||
- Use standard request/response topics
|
||||
- Implement FlowProcessor base class
|
||||
- Register PromptClientSpec for prompt service interaction
|
||||
|
||||
2. **Configuration Management**
|
||||
- Access schema configurations via config service
|
||||
- Cache schemas for performance
|
||||
- Handle configuration updates dynamically
|
||||
|
||||
3. **Prompt Integration**
|
||||
- Use existing prompt service infrastructure
|
||||
- Call prompt service with format-specific prompt IDs:
|
||||
- `diagnose-csv`: For CSV data analysis
|
||||
- `diagnose-json`: For JSON data analysis
|
||||
- `diagnose-xml`: For XML data analysis
|
||||
- Prompts are configured in prompt config, not hard-coded in service
|
||||
- Pass schemas and data samples as prompt variables
|
||||
- Parse prompt responses to extract descriptors
|
||||
|
||||
4. **Error Handling**
|
||||
- Validate input data samples
|
||||
- Provide descriptive error messages
|
||||
- Handle malformed data gracefully
|
||||
- Handle prompt service failures
|
||||
|
||||
5. **Data Sampling**
|
||||
- Process configurable sample sizes
|
||||
- Handle incomplete records appropriately
|
||||
- Maintain sampling consistency
|
||||
|
||||
### API Integration
|
||||
|
||||
The service will integrate with existing TrustGraph APIs:
|
||||
|
||||
Modified Components:
|
||||
- `tg-load-structured-data` CLI - Refactored to use the new service for diagnosis operations
|
||||
- Flow API - Extended to support structured data diagnosis requests
|
||||
|
||||
New Service Endpoints:
|
||||
- `/api/v1/flow/{flow}/diagnose/structured-data` - WebSocket endpoint for diagnosis requests
|
||||
- `/api/v1/diagnose/structured-data` - REST endpoint for synchronous diagnosis
|
||||
|
||||
### Message Flow
|
||||
|
||||
```
|
||||
Client → Gateway → Structured Diag Service → Config Service (for schemas)
|
||||
↓
|
||||
Type Detector (algorithmic)
|
||||
↓
|
||||
Prompt Service (diagnose-csv/json/xml)
|
||||
↓
|
||||
Descriptor Generator (parses prompt response)
|
||||
↓
|
||||
Client ← Gateway ← Structured Diag Service (response)
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Input validation to prevent injection attacks
|
||||
- Size limits on data samples to prevent DoS
|
||||
- Sanitization of generated descriptors
|
||||
- Access control through existing TrustGraph authentication
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
- Cache schema definitions to reduce config service calls
|
||||
- Limit sample sizes to maintain responsive performance
|
||||
- Use streaming processing for large data samples
|
||||
- Implement timeout mechanisms for long-running analyses
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
1. **Unit Tests**
|
||||
- Type detection for various data formats
|
||||
- Descriptor generation accuracy
|
||||
- Error handling scenarios
|
||||
|
||||
2. **Integration Tests**
|
||||
- Service request/response flow
|
||||
- Schema retrieval and caching
|
||||
- CLI integration
|
||||
|
||||
3. **Performance Tests**
|
||||
- Large sample processing
|
||||
- Concurrent request handling
|
||||
- Memory usage under load
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. **Phase 1**: Implement service with core functionality
|
||||
2. **Phase 2**: Refactor CLI to use service (maintain backward compatibility)
|
||||
3. **Phase 3**: Add REST API endpoints
|
||||
4. **Phase 4**: Deprecate embedded CLI logic (with notice period)
|
||||
|
||||
## Timeline
|
||||
|
||||
- Week 1-2: Implement core service and type detection
|
||||
- Week 3-4: Add descriptor generation and integration
|
||||
- Week 5: Testing and documentation
|
||||
- Week 6: CLI refactoring and migration
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Should the service support additional data formats (e.g., Parquet, Avro)?
|
||||
- What should be the maximum sample size for analysis?
|
||||
- Should diagnosis results be cached for repeated requests?
|
||||
- How should the service handle multi-schema scenarios?
|
||||
- Should the prompt IDs be configurable parameters for the service?
|
||||
|
||||
## References
|
||||
|
||||
- [Structured Data Descriptor Specification](structured-data-descriptor.md)
|
||||
- [Structured Data Loading Documentation](structured-data.md)
|
||||
- `tg-load-structured-data` implementation: `trustgraph-cli/trustgraph/cli/load_structured_data.py`
|
||||
Loading…
Add table
Add a link
Reference in a new issue