mirror of
https://github.com/trustgraph-ai/trustgraph.git
synced 2026-04-29 10:26:21 +02:00
Structure data diagnosis service (#518)
* Import flow tech spec * Structured diag service * Plumbed into API gateway * Type detector * Diag service * Added entry point
This commit is contained in:
cybermaggedon
GitHub
parent
d73af56690
commit
3d783f4bd4
13 changed files with 1201 additions and 3 deletions
156
docs/tech-specs/flow-class-definition.md
Normal file
156
docs/tech-specs/flow-class-definition.md
Normal file
|
|
@ -0,0 +1,156 @@
|
|||
# Flow Class Definition Specification
|
||||
|
||||
## Overview
|
||||
|
||||
A flow class defines a complete dataflow pattern template in the TrustGraph system. When instantiated, it creates an interconnected network of processors that handle data ingestion, processing, storage, and querying as a unified system.
|
||||
|
||||
## Structure
|
||||
|
||||
A flow class definition consists of four main sections:
|
||||
|
||||
### 1. Class Section
|
||||
Defines shared service processors that are instantiated once per flow class. These processors handle requests from all flow instances of this class.
|
||||
|
||||
```json
|
||||
"class": {
|
||||
"service-name:{class}": {
|
||||
"request": "queue-pattern:{class}",
|
||||
"response": "queue-pattern:{class}"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Characteristics:**
|
||||
- Shared across all flow instances of the same class
|
||||
- Typically expensive or stateless services (LLMs, embedding models)
|
||||
- Use `{class}` template variable for queue naming
|
||||
- Examples: `embeddings:{class}`, `text-completion:{class}`, `graph-rag:{class}`
|
||||
|
||||
### 2. Flow Section
|
||||
Defines flow-specific processors that are instantiated for each individual flow instance. Each flow gets its own isolated set of these processors.
|
||||
|
||||
```json
|
||||
"flow": {
|
||||
"processor-name:{id}": {
|
||||
"input": "queue-pattern:{id}",
|
||||
"output": "queue-pattern:{id}"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Characteristics:**
|
||||
- Unique instance per flow
|
||||
- Handle flow-specific data and state
|
||||
- Use `{id}` template variable for queue naming
|
||||
- Examples: `chunker:{id}`, `pdf-decoder:{id}`, `kg-extract-relationships:{id}`
|
||||
|
||||
### 3. Interfaces Section
|
||||
Defines the entry points and interaction contracts for the flow. These form the API surface for external systems and internal component communication.
|
||||
|
||||
Interfaces can take two forms:
|
||||
|
||||
**Fire-and-Forget Pattern** (single queue):
|
||||
```json
|
||||
"interfaces": {
|
||||
"document-load": "persistent://tg/flow/document-load:{id}",
|
||||
"triples-store": "persistent://tg/flow/triples-store:{id}"
|
||||
}
|
||||
```
|
||||
|
||||
**Request/Response Pattern** (object with request/response fields):
|
||||
```json
|
||||
"interfaces": {
|
||||
"embeddings": {
|
||||
"request": "non-persistent://tg/request/embeddings:{class}",
|
||||
"response": "non-persistent://tg/response/embeddings:{class}"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Types of Interfaces:**
|
||||
- **Entry Points**: Where external systems inject data (`document-load`, `agent`)
|
||||
- **Service Interfaces**: Request/response patterns for services (`embeddings`, `text-completion`)
|
||||
- **Data Interfaces**: Fire-and-forget data flow connection points (`triples-store`, `entity-contexts-load`)
|
||||
|
||||
### 4. Metadata
|
||||
Additional information about the flow class:
|
||||
|
||||
```json
|
||||
"description": "Human-readable description",
|
||||
"tags": ["capability-1", "capability-2"]
|
||||
```
|
||||
|
||||
## Template Variables
|
||||
|
||||
### {id}
|
||||
- Replaced with the unique flow instance identifier
|
||||
- Creates isolated resources for each flow
|
||||
- Example: `flow-123`, `customer-A-flow`
|
||||
|
||||
### {class}
|
||||
- Replaced with the flow class name
|
||||
- Creates shared resources across flows of the same class
|
||||
- Example: `standard-rag`, `enterprise-rag`
|
||||
|
||||
## Queue Patterns (Pulsar)
|
||||
|
||||
Flow classes use Apache Pulsar for messaging. Queue names follow the Pulsar format:
|
||||
```
|
||||
<persistence>://<tenant>/<namespace>/<topic>
|
||||
```
|
||||
|
||||
### Components:
|
||||
- **persistence**: `persistent` or `non-persistent` (Pulsar persistence mode)
|
||||
- **tenant**: `tg` for TrustGraph-supplied flow class definitions
|
||||
- **namespace**: Indicates the messaging pattern
|
||||
- `flow`: Fire-and-forget services
|
||||
- `request`: Request portion of request/response services
|
||||
- `response`: Response portion of request/response services
|
||||
- **topic**: The specific queue/topic name with template variables
|
||||
|
||||
### Persistent Queues
|
||||
- Pattern: `persistent://tg/flow/<topic>:{id}`
|
||||
- Used for fire-and-forget services and durable data flow
|
||||
- Data persists in Pulsar storage across restarts
|
||||
- Example: `persistent://tg/flow/chunk-load:{id}`
|
||||
|
||||
### Non-Persistent Queues
|
||||
- Pattern: `non-persistent://tg/request/<topic>:{class}` or `non-persistent://tg/response/<topic>:{class}`
|
||||
- Used for request/response messaging patterns
|
||||
- Ephemeral, not persisted to disk by Pulsar
|
||||
- Lower latency, suitable for RPC-style communication
|
||||
- Example: `non-persistent://tg/request/embeddings:{class}`
|
||||
|
||||
## Dataflow Architecture
|
||||
|
||||
The flow class creates a unified dataflow where:
|
||||
|
||||
1. **Document Processing Pipeline**: Flows from ingestion through transformation to storage
|
||||
2. **Query Services**: Integrated processors that query the same data stores and services
|
||||
3. **Shared Services**: Centralized processors that all flows can utilize
|
||||
4. **Storage Writers**: Persist processed data to appropriate stores
|
||||
|
||||
All processors (both `{id}` and `{class}`) work together as a cohesive dataflow graph, not as separate systems.
|
||||
|
||||
## Example Flow Instantiation
|
||||
|
||||
Given:
|
||||
- Flow Instance ID: `customer-A-flow`
|
||||
- Flow Class: `standard-rag`
|
||||
|
||||
Template expansions:
|
||||
- `persistent://tg/flow/chunk-load:{id}` → `persistent://tg/flow/chunk-load:customer-A-flow`
|
||||
- `non-persistent://tg/request/embeddings:{class}` → `non-persistent://tg/request/embeddings:standard-rag`
|
||||
|
||||
This creates:
|
||||
- Isolated document processing pipeline for `customer-A-flow`
|
||||
- Shared embedding service for all `standard-rag` flows
|
||||
- Complete dataflow from document ingestion through querying
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **Resource Efficiency**: Expensive services are shared across flows
|
||||
2. **Flow Isolation**: Each flow has its own data processing pipeline
|
||||
3. **Scalability**: Can instantiate multiple flows from the same template
|
||||
4. **Modularity**: Clear separation between shared and flow-specific components
|
||||
5. **Unified Architecture**: Query and processing are part of the same dataflow
|
||||
273
docs/tech-specs/structured-diag-service.md
Normal file
273
docs/tech-specs/structured-diag-service.md
Normal file
|
|
@ -0,0 +1,273 @@
|
|||
# Structured Data Diagnostic Service Technical Specification
|
||||
|
||||
## Overview
|
||||
|
||||
This specification describes a new invokable service for diagnosing and analyzing structured data within TrustGraph. The service extracts functionality from the existing `tg-load-structured-data` command-line tool and exposes it as a request/response service, enabling programmatic access to data type detection and descriptor generation capabilities.
|
||||
|
||||
The service supports three primary operations:
|
||||
|
||||
1. **Data Type Detection**: Analyze a data sample to determine its format (CSV, JSON, or XML)
|
||||
2. **Descriptor Generation**: Generate a TrustGraph structured data descriptor for a given data sample and type
|
||||
3. **Combined Diagnosis**: Perform both type detection and descriptor generation in sequence
|
||||
|
||||
## Goals
|
||||
|
||||
- **Modularize Data Analysis**: Extract data diagnosis logic from CLI into reusable service components
|
||||
- **Enable Programmatic Access**: Provide API-based access to data analysis capabilities
|
||||
- **Support Multiple Data Formats**: Handle CSV, JSON, and XML data formats consistently
|
||||
- **Generate Accurate Descriptors**: Produce structured data descriptors that accurately map source data to TrustGraph schemas
|
||||
- **Maintain Backward Compatibility**: Ensure existing CLI functionality continues to work
|
||||
- **Enable Service Composition**: Allow other services to leverage data diagnosis capabilities
|
||||
- **Improve Testability**: Separate business logic from CLI interface for better testing
|
||||
- **Support Streaming Analysis**: Enable analysis of data samples without loading entire files
|
||||
|
||||
## Background
|
||||
|
||||
Currently, the `tg-load-structured-data` command provides comprehensive functionality for analyzing structured data and generating descriptors. However, this functionality is tightly coupled to the CLI interface, limiting its reusability.
|
||||
|
||||
Current limitations include:
|
||||
- Data diagnosis logic embedded in CLI code
|
||||
- No programmatic access to type detection and descriptor generation
|
||||
- Difficult to integrate diagnosis capabilities into other services
|
||||
- Limited ability to compose data analysis workflows
|
||||
|
||||
This specification addresses these gaps by creating a dedicated service for structured data diagnosis. By exposing these capabilities as a service, TrustGraph can:
|
||||
- Enable other services to analyze data programmatically
|
||||
- Support more complex data processing pipelines
|
||||
- Facilitate integration with external systems
|
||||
- Improve maintainability through separation of concerns
|
||||
|
||||
## Technical Design
|
||||
|
||||
### Architecture
|
||||
|
||||
The structured data diagnostic service requires the following technical components:
|
||||
|
||||
1. **Diagnostic Service Processor**
|
||||
- Handles incoming diagnosis requests
|
||||
- Orchestrates type detection and descriptor generation
|
||||
- Returns structured responses with diagnosis results
|
||||
|
||||
Module: `trustgraph-flow/trustgraph/diagnosis/structured_data/service.py`
|
||||
|
||||
2. **Data Type Detector**
|
||||
- Uses algorithmic detection to identify data format (CSV, JSON, XML)
|
||||
- Analyzes data structure, delimiters, and syntax patterns
|
||||
- Returns detected format and confidence scores
|
||||
|
||||
Module: `trustgraph-flow/trustgraph/diagnosis/structured_data/type_detector.py`
|
||||
|
||||
3. **Descriptor Generator**
|
||||
- Uses prompt service to generate descriptors
|
||||
- Invokes format-specific prompts (diagnose-csv, diagnose-json, diagnose-xml)
|
||||
- Maps data fields to TrustGraph schema fields through prompt responses
|
||||
|
||||
Module: `trustgraph-flow/trustgraph/diagnosis/structured_data/descriptor_generator.py`
|
||||
|
||||
### Data Models
|
||||
|
||||
#### StructuredDataDiagnosisRequest
|
||||
|
||||
Request message for structured data diagnosis operations:
|
||||
|
||||
```python
|
||||
class StructuredDataDiagnosisRequest:
|
||||
operation: str # "detect-type", "generate-descriptor", or "diagnose"
|
||||
sample: str # Data sample to analyze (text content)
|
||||
type: Optional[str] # Data type (csv, json, xml) - required for generate-descriptor
|
||||
schema_name: Optional[str] # Target schema name for descriptor generation
|
||||
options: Dict[str, Any] # Additional options (e.g., delimiter for CSV)
|
||||
```
|
||||
|
||||
#### StructuredDataDiagnosisResponse
|
||||
|
||||
Response message containing diagnosis results:
|
||||
|
||||
```python
|
||||
class StructuredDataDiagnosisResponse:
|
||||
operation: str # The operation that was performed
|
||||
detected_type: Optional[str] # Detected data type (for detect-type/diagnose)
|
||||
confidence: Optional[float] # Confidence score for type detection
|
||||
descriptor: Optional[Dict] # Generated descriptor (for generate-descriptor/diagnose)
|
||||
error: Optional[str] # Error message if operation failed
|
||||
metadata: Dict[str, Any] # Additional metadata (e.g., field count, sample records)
|
||||
```
|
||||
|
||||
#### Descriptor Structure
|
||||
|
||||
The generated descriptor follows the existing structured data descriptor format:
|
||||
|
||||
```json
|
||||
{
|
||||
"format": {
|
||||
"type": "csv",
|
||||
"encoding": "utf-8",
|
||||
"options": {
|
||||
"delimiter": ",",
|
||||
"has_header": true
|
||||
}
|
||||
},
|
||||
"mappings": [
|
||||
{
|
||||
"source_field": "customer_id",
|
||||
"target_field": "id",
|
||||
"transforms": [
|
||||
{"type": "trim"}
|
||||
]
|
||||
}
|
||||
],
|
||||
"output": {
|
||||
"schema_name": "customer",
|
||||
"options": {
|
||||
"batch_size": 1000,
|
||||
"confidence": 0.9
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Service Interface
|
||||
|
||||
The service will expose the following operations through the request/response pattern:
|
||||
|
||||
1. **Type Detection Operation**
|
||||
- Input: Data sample
|
||||
- Processing: Analyze data structure using algorithmic detection
|
||||
- Output: Detected type with confidence score
|
||||
|
||||
2. **Descriptor Generation Operation**
|
||||
- Input: Data sample, type, target schema name
|
||||
- Processing:
|
||||
- Call prompt service with format-specific prompt ID (diagnose-csv, diagnose-json, or diagnose-xml)
|
||||
- Pass data sample and available schemas to prompt
|
||||
- Receive generated descriptor from prompt response
|
||||
- Output: Structured data descriptor
|
||||
|
||||
3. **Combined Diagnosis Operation**
|
||||
- Input: Data sample, optional schema name
|
||||
- Processing:
|
||||
- Use algorithmic detection to identify format first
|
||||
- Select appropriate format-specific prompt based on detected type
|
||||
- Call prompt service to generate descriptor
|
||||
- Output: Both detected type and descriptor
|
||||
|
||||
### Implementation Details
|
||||
|
||||
The service will follow TrustGraph service conventions:
|
||||
|
||||
1. **Service Registration**
|
||||
- Register as `structured-diag` service type
|
||||
- Use standard request/response topics
|
||||
- Implement FlowProcessor base class
|
||||
- Register PromptClientSpec for prompt service interaction
|
||||
|
||||
2. **Configuration Management**
|
||||
- Access schema configurations via config service
|
||||
- Cache schemas for performance
|
||||
- Handle configuration updates dynamically
|
||||
|
||||
3. **Prompt Integration**
|
||||
- Use existing prompt service infrastructure
|
||||
- Call prompt service with format-specific prompt IDs:
|
||||
- `diagnose-csv`: For CSV data analysis
|
||||
- `diagnose-json`: For JSON data analysis
|
||||
- `diagnose-xml`: For XML data analysis
|
||||
- Prompts are configured in prompt config, not hard-coded in service
|
||||
- Pass schemas and data samples as prompt variables
|
||||
- Parse prompt responses to extract descriptors
|
||||
|
||||
4. **Error Handling**
|
||||
- Validate input data samples
|
||||
- Provide descriptive error messages
|
||||
- Handle malformed data gracefully
|
||||
- Handle prompt service failures
|
||||
|
||||
5. **Data Sampling**
|
||||
- Process configurable sample sizes
|
||||
- Handle incomplete records appropriately
|
||||
- Maintain sampling consistency
|
||||
|
||||
### API Integration
|
||||
|
||||
The service will integrate with existing TrustGraph APIs:
|
||||
|
||||
Modified Components:
|
||||
- `tg-load-structured-data` CLI - Refactored to use the new service for diagnosis operations
|
||||
- Flow API - Extended to support structured data diagnosis requests
|
||||
|
||||
New Service Endpoints:
|
||||
- `/api/v1/flow/{flow}/diagnose/structured-data` - WebSocket endpoint for diagnosis requests
|
||||
- `/api/v1/diagnose/structured-data` - REST endpoint for synchronous diagnosis
|
||||
|
||||
### Message Flow
|
||||
|
||||
```
|
||||
Client → Gateway → Structured Diag Service → Config Service (for schemas)
|
||||
↓
|
||||
Type Detector (algorithmic)
|
||||
↓
|
||||
Prompt Service (diagnose-csv/json/xml)
|
||||
↓
|
||||
Descriptor Generator (parses prompt response)
|
||||
↓
|
||||
Client ← Gateway ← Structured Diag Service (response)
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Input validation to prevent injection attacks
|
||||
- Size limits on data samples to prevent DoS
|
||||
- Sanitization of generated descriptors
|
||||
- Access control through existing TrustGraph authentication
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
- Cache schema definitions to reduce config service calls
|
||||
- Limit sample sizes to maintain responsive performance
|
||||
- Use streaming processing for large data samples
|
||||
- Implement timeout mechanisms for long-running analyses
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
1. **Unit Tests**
|
||||
- Type detection for various data formats
|
||||
- Descriptor generation accuracy
|
||||
- Error handling scenarios
|
||||
|
||||
2. **Integration Tests**
|
||||
- Service request/response flow
|
||||
- Schema retrieval and caching
|
||||
- CLI integration
|
||||
|
||||
3. **Performance Tests**
|
||||
- Large sample processing
|
||||
- Concurrent request handling
|
||||
- Memory usage under load
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. **Phase 1**: Implement service with core functionality
|
||||
2. **Phase 2**: Refactor CLI to use service (maintain backward compatibility)
|
||||
3. **Phase 3**: Add REST API endpoints
|
||||
4. **Phase 4**: Deprecate embedded CLI logic (with notice period)
|
||||
|
||||
## Timeline
|
||||
|
||||
- Week 1-2: Implement core service and type detection
|
||||
- Week 3-4: Add descriptor generation and integration
|
||||
- Week 5: Testing and documentation
|
||||
- Week 6: CLI refactoring and migration
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Should the service support additional data formats (e.g., Parquet, Avro)?
|
||||
- What should be the maximum sample size for analysis?
|
||||
- Should diagnosis results be cached for repeated requests?
|
||||
- How should the service handle multi-schema scenarios?
|
||||
- Should the prompt IDs be configurable parameters for the service?
|
||||
|
||||
## References
|
||||
|
||||
- [Structured Data Descriptor Specification](structured-data-descriptor.md)
|
||||
- [Structured Data Loading Documentation](structured-data.md)
|
||||
- `tg-load-structured-data` implementation: `trustgraph-cli/trustgraph/cli/load_structured_data.py`
|
||||
Loading…
Add table
Add a link
Reference in a new issue