mirror of
https://github.com/trustgraph-ai/trustgraph.git
synced 2026-07-24 20:51:02 +02:00
Structured data loader framework
This commit is contained in:
Cyber MacGeddon
parent
257a7951a7
commit
1a095569b8
3 changed files with 842 additions and 0 deletions
559
docs/tech-specs/structured-data-descriptor.md
Normal file
559
docs/tech-specs/structured-data-descriptor.md
Normal file
|
|
@ -0,0 +1,559 @@
|
||||||
|
# Structured Data Descriptor Specification
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
The Structured Data Descriptor is a JSON-based configuration language that describes how to parse, transform, and import structured data into TrustGraph. It provides a declarative approach to data ingestion, supporting multiple input formats and complex transformation pipelines without requiring custom code.
|
||||||
|
|
||||||
|
## Core Concepts
|
||||||
|
|
||||||
|
### 1. Format Definition
|
||||||
|
Describes the input file type and parsing options. Determines which parser to use and how to interpret the source data.
|
||||||
|
|
||||||
|
### 2. Field Mappings
|
||||||
|
Maps source paths to target fields with transformations. Defines how data flows from input sources to output schema fields.
|
||||||
|
|
||||||
|
### 3. Transform Pipeline
|
||||||
|
Chain of data transformations that can be applied to field values, including:
|
||||||
|
- Data cleaning (trim, normalize)
|
||||||
|
- Format conversion (date parsing, type casting)
|
||||||
|
- Calculations (arithmetic, string manipulation)
|
||||||
|
- Lookups (reference tables, substitutions)
|
||||||
|
|
||||||
|
### 4. Validation Rules
|
||||||
|
Data quality checks applied to ensure data integrity:
|
||||||
|
- Type validation
|
||||||
|
- Range checks
|
||||||
|
- Pattern matching (regex)
|
||||||
|
- Required field validation
|
||||||
|
- Custom validation logic
|
||||||
|
|
||||||
|
### 5. Global Settings
|
||||||
|
Configuration that applies across the entire import process:
|
||||||
|
- Lookup tables for data enrichment
|
||||||
|
- Global variables and constants
|
||||||
|
- Output format specifications
|
||||||
|
- Error handling policies
|
||||||
|
|
||||||
|
## Implementation Strategy
|
||||||
|
|
||||||
|
The importer implementation follows this pipeline:
|
||||||
|
|
||||||
|
1. **Parse Configuration** - Load and validate the JSON descriptor
|
||||||
|
2. **Initialize Parser** - Load appropriate parser (CSV, XML, JSON, etc.) based on `format.type`
|
||||||
|
3. **Apply Preprocessing** - Execute global filters and transformations
|
||||||
|
4. **Process Records** - For each input record:
|
||||||
|
- Extract data using source paths (JSONPath, XPath, column names)
|
||||||
|
- Apply field-level transforms in sequence
|
||||||
|
- Validate results against defined rules
|
||||||
|
- Apply default values for missing data
|
||||||
|
5. **Apply Postprocessing** - Execute deduplication, aggregation, etc.
|
||||||
|
6. **Generate Output** - Produce data in specified target format
|
||||||
|
|
||||||
|
## Path Expression Support
|
||||||
|
|
||||||
|
Different input formats use appropriate path expression languages:
|
||||||
|
|
||||||
|
- **CSV**: Column names or indices (`"column_name"` or `"[2]"`)
|
||||||
|
- **JSON**: JSONPath syntax (`"$.user.profile.email"`)
|
||||||
|
- **XML**: XPath expressions (`"//product[@id='123']/price"`)
|
||||||
|
- **Fixed-width**: Field names from field definitions
|
||||||
|
|
||||||
|
## Benefits
|
||||||
|
|
||||||
|
- **Single Codebase** - One importer handles multiple input formats
|
||||||
|
- **User-Friendly** - Non-technical users can create configurations
|
||||||
|
- **Reusable** - Configurations can be shared and versioned
|
||||||
|
- **Flexible** - Complex transformations without custom coding
|
||||||
|
- **Robust** - Built-in validation and comprehensive error handling
|
||||||
|
- **Maintainable** - Declarative approach reduces implementation complexity
|
||||||
|
|
||||||
|
## Language Specification
|
||||||
|
|
||||||
|
The Structured Data Descriptor uses a JSON configuration format with the following top-level structure:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"version": "1.0",
|
||||||
|
"metadata": {
|
||||||
|
"name": "Configuration Name",
|
||||||
|
"description": "Description of what this config does",
|
||||||
|
"author": "Author Name",
|
||||||
|
"created": "2024-01-01T00:00:00Z"
|
||||||
|
},
|
||||||
|
"format": { ... },
|
||||||
|
"globals": { ... },
|
||||||
|
"preprocessing": [ ... ],
|
||||||
|
"mappings": [ ... ],
|
||||||
|
"postprocessing": [ ... ],
|
||||||
|
"output": { ... }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Format Definition
|
||||||
|
|
||||||
|
Describes the input data format and parsing options:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"format": {
|
||||||
|
"type": "csv|json|xml|fixed-width|excel|parquet",
|
||||||
|
"encoding": "utf-8",
|
||||||
|
"options": {
|
||||||
|
// Format-specific options
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### CSV Format Options
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"format": {
|
||||||
|
"type": "csv",
|
||||||
|
"options": {
|
||||||
|
"delimiter": ",",
|
||||||
|
"quote_char": "\"",
|
||||||
|
"escape_char": "\\",
|
||||||
|
"skip_rows": 1,
|
||||||
|
"has_header": true,
|
||||||
|
"null_values": ["", "NULL", "null", "N/A"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JSON Format Options
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"format": {
|
||||||
|
"type": "json",
|
||||||
|
"options": {
|
||||||
|
"root_path": "$.data",
|
||||||
|
"array_mode": "records|single",
|
||||||
|
"flatten": false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### XML Format Options
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"format": {
|
||||||
|
"type": "xml",
|
||||||
|
"options": {
|
||||||
|
"root_element": "//records/record",
|
||||||
|
"namespaces": {
|
||||||
|
"ns": "http://example.com/namespace"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Global Settings
|
||||||
|
|
||||||
|
Define lookup tables, variables, and global configuration:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"globals": {
|
||||||
|
"variables": {
|
||||||
|
"current_date": "2024-01-01",
|
||||||
|
"batch_id": "BATCH_001",
|
||||||
|
"default_confidence": 0.8
|
||||||
|
},
|
||||||
|
"lookup_tables": {
|
||||||
|
"country_codes": {
|
||||||
|
"US": "United States",
|
||||||
|
"UK": "United Kingdom",
|
||||||
|
"CA": "Canada"
|
||||||
|
},
|
||||||
|
"status_mapping": {
|
||||||
|
"1": "active",
|
||||||
|
"0": "inactive"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"constants": {
|
||||||
|
"source_system": "legacy_crm",
|
||||||
|
"import_type": "full"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Field Mappings
|
||||||
|
|
||||||
|
Define how source data maps to target fields with transformations:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"mappings": [
|
||||||
|
{
|
||||||
|
"target_field": "person_name",
|
||||||
|
"source": "$.name",
|
||||||
|
"transforms": [
|
||||||
|
{"type": "trim"},
|
||||||
|
{"type": "title_case"},
|
||||||
|
{"type": "required"}
|
||||||
|
],
|
||||||
|
"validation": [
|
||||||
|
{"type": "min_length", "value": 2},
|
||||||
|
{"type": "max_length", "value": 100},
|
||||||
|
{"type": "pattern", "value": "^[A-Za-z\\s]+$"}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"target_field": "age",
|
||||||
|
"source": "$.age",
|
||||||
|
"transforms": [
|
||||||
|
{"type": "to_int"},
|
||||||
|
{"type": "default", "value": 0}
|
||||||
|
],
|
||||||
|
"validation": [
|
||||||
|
{"type": "range", "min": 0, "max": 150}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"target_field": "country",
|
||||||
|
"source": "$.country_code",
|
||||||
|
"transforms": [
|
||||||
|
{"type": "lookup", "table": "country_codes"},
|
||||||
|
{"type": "default", "value": "Unknown"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Transform Types
|
||||||
|
|
||||||
|
Available transformation functions:
|
||||||
|
|
||||||
|
#### String Transforms
|
||||||
|
```json
|
||||||
|
{"type": "trim"},
|
||||||
|
{"type": "upper"},
|
||||||
|
{"type": "lower"},
|
||||||
|
{"type": "title_case"},
|
||||||
|
{"type": "replace", "pattern": "old", "replacement": "new"},
|
||||||
|
{"type": "regex_replace", "pattern": "\\d+", "replacement": "XXX"},
|
||||||
|
{"type": "substring", "start": 0, "end": 10},
|
||||||
|
{"type": "pad_left", "length": 10, "char": "0"}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Type Conversions
|
||||||
|
```json
|
||||||
|
{"type": "to_string"},
|
||||||
|
{"type": "to_int"},
|
||||||
|
{"type": "to_float"},
|
||||||
|
{"type": "to_bool"},
|
||||||
|
{"type": "to_date", "format": "YYYY-MM-DD"},
|
||||||
|
{"type": "parse_json"}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Data Operations
|
||||||
|
```json
|
||||||
|
{"type": "default", "value": "default_value"},
|
||||||
|
{"type": "lookup", "table": "table_name"},
|
||||||
|
{"type": "concat", "values": ["field1", " - ", "field2"]},
|
||||||
|
{"type": "calculate", "expression": "${field1} + ${field2}"},
|
||||||
|
{"type": "conditional", "condition": "${age} > 18", "true_value": "adult", "false_value": "minor"}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Validation Rules
|
||||||
|
|
||||||
|
Data quality checks with configurable error handling:
|
||||||
|
|
||||||
|
#### Basic Validations
|
||||||
|
```json
|
||||||
|
{"type": "required"},
|
||||||
|
{"type": "not_null"},
|
||||||
|
{"type": "min_length", "value": 5},
|
||||||
|
{"type": "max_length", "value": 100},
|
||||||
|
{"type": "range", "min": 0, "max": 1000},
|
||||||
|
{"type": "pattern", "value": "^[A-Z]{2,3}$"},
|
||||||
|
{"type": "in_list", "values": ["active", "inactive", "pending"]}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Custom Validations
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"type": "custom",
|
||||||
|
"expression": "${age} >= 18 && ${country} == 'US'",
|
||||||
|
"message": "Must be 18+ and in US"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "cross_field",
|
||||||
|
"fields": ["start_date", "end_date"],
|
||||||
|
"expression": "${start_date} < ${end_date}",
|
||||||
|
"message": "Start date must be before end date"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Preprocessing and Postprocessing
|
||||||
|
|
||||||
|
Global operations applied before/after field mapping:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"preprocessing": [
|
||||||
|
{
|
||||||
|
"type": "filter",
|
||||||
|
"condition": "${status} != 'deleted'"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "sort",
|
||||||
|
"field": "created_date",
|
||||||
|
"order": "asc"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"postprocessing": [
|
||||||
|
{
|
||||||
|
"type": "deduplicate",
|
||||||
|
"key_fields": ["email", "phone"]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "aggregate",
|
||||||
|
"group_by": ["country"],
|
||||||
|
"functions": {
|
||||||
|
"total_count": {"type": "count"},
|
||||||
|
"avg_age": {"type": "avg", "field": "age"}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Output Configuration
|
||||||
|
|
||||||
|
Define how processed data should be output:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"output": {
|
||||||
|
"format": "trustgraph-objects",
|
||||||
|
"schema_name": "person",
|
||||||
|
"options": {
|
||||||
|
"batch_size": 1000,
|
||||||
|
"confidence": 0.9,
|
||||||
|
"source_span_field": "raw_text",
|
||||||
|
"metadata": {
|
||||||
|
"source": "crm_import",
|
||||||
|
"version": "1.0"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"error_handling": {
|
||||||
|
"on_validation_error": "skip|fail|log",
|
||||||
|
"on_transform_error": "skip|fail|default",
|
||||||
|
"max_errors": 100,
|
||||||
|
"error_output": "errors.json"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Complete Example
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"version": "1.0",
|
||||||
|
"metadata": {
|
||||||
|
"name": "Customer Import from CRM CSV",
|
||||||
|
"description": "Imports customer data from legacy CRM system",
|
||||||
|
"author": "Data Team",
|
||||||
|
"created": "2024-01-01T00:00:00Z"
|
||||||
|
},
|
||||||
|
"format": {
|
||||||
|
"type": "csv",
|
||||||
|
"encoding": "utf-8",
|
||||||
|
"options": {
|
||||||
|
"delimiter": ",",
|
||||||
|
"has_header": true,
|
||||||
|
"skip_rows": 1
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"globals": {
|
||||||
|
"variables": {
|
||||||
|
"import_date": "2024-01-01",
|
||||||
|
"default_confidence": 0.85
|
||||||
|
},
|
||||||
|
"lookup_tables": {
|
||||||
|
"country_codes": {
|
||||||
|
"US": "United States",
|
||||||
|
"CA": "Canada",
|
||||||
|
"UK": "United Kingdom"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"preprocessing": [
|
||||||
|
{
|
||||||
|
"type": "filter",
|
||||||
|
"condition": "${status} == 'active'"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"mappings": [
|
||||||
|
{
|
||||||
|
"target_field": "full_name",
|
||||||
|
"source": "customer_name",
|
||||||
|
"transforms": [
|
||||||
|
{"type": "trim"},
|
||||||
|
{"type": "title_case"}
|
||||||
|
],
|
||||||
|
"validation": [
|
||||||
|
{"type": "required"},
|
||||||
|
{"type": "min_length", "value": 2}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"target_field": "email",
|
||||||
|
"source": "email_address",
|
||||||
|
"transforms": [
|
||||||
|
{"type": "trim"},
|
||||||
|
{"type": "lower"}
|
||||||
|
],
|
||||||
|
"validation": [
|
||||||
|
{"type": "pattern", "value": "^[\\w.-]+@[\\w.-]+\\.[a-zA-Z]{2,}$"}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"target_field": "age",
|
||||||
|
"source": "age",
|
||||||
|
"transforms": [
|
||||||
|
{"type": "to_int"},
|
||||||
|
{"type": "default", "value": 0}
|
||||||
|
],
|
||||||
|
"validation": [
|
||||||
|
{"type": "range", "min": 0, "max": 120}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"target_field": "country",
|
||||||
|
"source": "country_code",
|
||||||
|
"transforms": [
|
||||||
|
{"type": "lookup", "table": "country_codes"},
|
||||||
|
{"type": "default", "value": "Unknown"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"output": {
|
||||||
|
"format": "trustgraph-objects",
|
||||||
|
"schema_name": "customer",
|
||||||
|
"options": {
|
||||||
|
"confidence": "${default_confidence}",
|
||||||
|
"batch_size": 500
|
||||||
|
},
|
||||||
|
"error_handling": {
|
||||||
|
"on_validation_error": "log",
|
||||||
|
"max_errors": 50
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## LLM Prompt for Descriptor Generation
|
||||||
|
|
||||||
|
The following prompt can be used to have an LLM analyze sample data and generate a descriptor configuration:
|
||||||
|
|
||||||
|
```
|
||||||
|
I need you to analyze the provided data sample and create a Structured Data Descriptor configuration in JSON format.
|
||||||
|
|
||||||
|
The descriptor should follow this specification:
|
||||||
|
- version: "1.0"
|
||||||
|
- metadata: Configuration name, description, author, and creation date
|
||||||
|
- format: Input format type and parsing options
|
||||||
|
- globals: Variables, lookup tables, and constants
|
||||||
|
- preprocessing: Filters and transformations applied before mapping
|
||||||
|
- mappings: Field-by-field mapping from source to target with transformations and validations
|
||||||
|
- postprocessing: Operations like deduplication or aggregation
|
||||||
|
- output: Target format and error handling configuration
|
||||||
|
|
||||||
|
ANALYZE THE DATA:
|
||||||
|
1. Identify the format (CSV, JSON, XML, etc.)
|
||||||
|
2. Detect delimiters, encodings, and structure
|
||||||
|
3. Find data types for each field
|
||||||
|
4. Identify patterns and constraints
|
||||||
|
5. Look for fields that need cleaning or transformation
|
||||||
|
6. Find relationships between fields
|
||||||
|
7. Identify lookup opportunities (codes that map to values)
|
||||||
|
8. Detect required vs optional fields
|
||||||
|
|
||||||
|
CREATE THE DESCRIPTOR:
|
||||||
|
For each field in the sample data:
|
||||||
|
- Map it to an appropriate target field name
|
||||||
|
- Add necessary transformations (trim, case conversion, type casting)
|
||||||
|
- Include appropriate validations (required, patterns, ranges)
|
||||||
|
- Set defaults for missing values
|
||||||
|
|
||||||
|
Include preprocessing if needed:
|
||||||
|
- Filters to exclude invalid records
|
||||||
|
- Sorting requirements
|
||||||
|
|
||||||
|
Include postprocessing if beneficial:
|
||||||
|
- Deduplication on key fields
|
||||||
|
- Aggregation for summary data
|
||||||
|
|
||||||
|
Configure output for TrustGraph:
|
||||||
|
- format: "trustgraph-objects"
|
||||||
|
- schema_name: Based on the data entity type
|
||||||
|
- Appropriate error handling
|
||||||
|
|
||||||
|
DATA SAMPLE:
|
||||||
|
[Insert data sample here]
|
||||||
|
|
||||||
|
ADDITIONAL CONTEXT (optional):
|
||||||
|
- Target schema name: [if known]
|
||||||
|
- Business rules: [any specific requirements]
|
||||||
|
- Data quality issues to address: [known problems]
|
||||||
|
|
||||||
|
Generate a complete, valid Structured Data Descriptor configuration that will properly import this data into TrustGraph. Include comments explaining key decisions.
|
||||||
|
```
|
||||||
|
|
||||||
|
### Example Usage Prompt
|
||||||
|
|
||||||
|
```
|
||||||
|
I need you to analyze the provided data sample and create a Structured Data Descriptor configuration in JSON format.
|
||||||
|
|
||||||
|
[Standard instructions from above...]
|
||||||
|
|
||||||
|
DATA SAMPLE:
|
||||||
|
```csv
|
||||||
|
CustomerID,Name,Email,Age,Country,Status,JoinDate,TotalPurchases
|
||||||
|
1001,"Smith, John",john.smith@email.com,35,US,1,2023-01-15,5420.50
|
||||||
|
1002,"doe, jane",JANE.DOE@GMAIL.COM,28,CA,1,2023-03-22,3200.00
|
||||||
|
1003,"Bob Johnson",bob@,62,UK,0,2022-11-01,0
|
||||||
|
1004,"Alice Chen","alice.chen@company.org",41,US,1,2023-06-10,8900.25
|
||||||
|
1005,,invalid-email,25,XX,1,2024-01-01,100
|
||||||
|
```
|
||||||
|
|
||||||
|
ADDITIONAL CONTEXT:
|
||||||
|
- Target schema name: customer
|
||||||
|
- Business rules: Email should be valid and lowercase, names should be title case
|
||||||
|
- Data quality issues: Some emails are invalid, some names are missing, country codes need mapping
|
||||||
|
```
|
||||||
|
|
||||||
|
### Prompt for Analyzing Existing Data Without Sample
|
||||||
|
|
||||||
|
```
|
||||||
|
I need you to help me create a Structured Data Descriptor configuration for importing [data type] data.
|
||||||
|
|
||||||
|
The source data has these characteristics:
|
||||||
|
- Format: [CSV/JSON/XML/etc]
|
||||||
|
- Fields: [list the fields]
|
||||||
|
- Data quality issues: [describe any known issues]
|
||||||
|
- Volume: [approximate number of records]
|
||||||
|
|
||||||
|
Requirements:
|
||||||
|
- [List any specific transformation needs]
|
||||||
|
- [List any validation requirements]
|
||||||
|
- [List any business rules]
|
||||||
|
|
||||||
|
Please generate a Structured Data Descriptor configuration that will:
|
||||||
|
1. Parse the input format correctly
|
||||||
|
2. Clean and standardize the data
|
||||||
|
3. Validate according to the requirements
|
||||||
|
4. Handle errors gracefully
|
||||||
|
5. Output in TrustGraph ExtractedObject format
|
||||||
|
|
||||||
|
Focus on making the configuration robust and reusable.
|
||||||
|
```
|
||||||
|
|
@ -54,6 +54,7 @@ tg-load-sample-documents = "trustgraph.cli.load_sample_documents:main"
|
||||||
tg-load-text = "trustgraph.cli.load_text:main"
|
tg-load-text = "trustgraph.cli.load_text:main"
|
||||||
tg-load-turtle = "trustgraph.cli.load_turtle:main"
|
tg-load-turtle = "trustgraph.cli.load_turtle:main"
|
||||||
tg-load-knowledge = "trustgraph.cli.load_knowledge:main"
|
tg-load-knowledge = "trustgraph.cli.load_knowledge:main"
|
||||||
|
tg-load-structured-data = "trustgraph.cli.load_structured_data:main"
|
||||||
tg-put-flow-class = "trustgraph.cli.put_flow_class:main"
|
tg-put-flow-class = "trustgraph.cli.put_flow_class:main"
|
||||||
tg-put-kg-core = "trustgraph.cli.put_kg_core:main"
|
tg-put-kg-core = "trustgraph.cli.put_kg_core:main"
|
||||||
tg-remove-library-document = "trustgraph.cli.remove_library_document:main"
|
tg-remove-library-document = "trustgraph.cli.remove_library_document:main"
|
||||||
|
|
|
||||||
282
trustgraph-cli/trustgraph/cli/load_structured_data.py
Normal file
282
trustgraph-cli/trustgraph/cli/load_structured_data.py
Normal file
|
|
@ -0,0 +1,282 @@
|
||||||
|
"""
|
||||||
|
Load structured data into TrustGraph using a descriptor configuration.
|
||||||
|
|
||||||
|
This utility can:
|
||||||
|
1. Analyze data samples to suggest appropriate schemas
|
||||||
|
2. Generate descriptor configurations from data samples
|
||||||
|
3. Parse and transform data using descriptor configurations
|
||||||
|
4. Import processed data into TrustGraph
|
||||||
|
|
||||||
|
The tool supports running all steps automatically or individual steps for
|
||||||
|
validation and debugging. The descriptor language allows for complex
|
||||||
|
transformations, validations, and mappings without requiring custom code.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
|
||||||
|
# Module logger
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
default_url = os.getenv("TRUSTGRAPH_URL", 'http://localhost:8088/')
|
||||||
|
|
||||||
|
|
||||||
|
def load_structured_data(
|
||||||
|
api_url: str,
|
||||||
|
input_file: str,
|
||||||
|
descriptor_file: str = None,
|
||||||
|
suggest_schema: bool = False,
|
||||||
|
generate_descriptor: bool = False,
|
||||||
|
parse_only: bool = False,
|
||||||
|
output_file: str = None,
|
||||||
|
sample_size: int = 100,
|
||||||
|
sample_chars: int = 500,
|
||||||
|
schema_name: str = None,
|
||||||
|
dry_run: bool = False,
|
||||||
|
verbose: bool = False
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Load structured data using a descriptor configuration.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
api_url: TrustGraph API URL
|
||||||
|
input_file: Path to input data file
|
||||||
|
descriptor_file: Path to JSON descriptor configuration
|
||||||
|
suggest_schema: Analyze data and suggest matching schemas
|
||||||
|
generate_descriptor: Generate descriptor from data sample
|
||||||
|
parse_only: Parse data but don't import to TrustGraph
|
||||||
|
output_file: Path to write output (descriptor/parsed data)
|
||||||
|
sample_size: Number of records to sample for analysis
|
||||||
|
sample_chars: Maximum characters to read for sampling
|
||||||
|
schema_name: Target schema name for generation
|
||||||
|
dry_run: If True, validate but don't import data
|
||||||
|
verbose: Enable verbose logging
|
||||||
|
"""
|
||||||
|
if verbose:
|
||||||
|
logging.basicConfig(level=logging.DEBUG)
|
||||||
|
else:
|
||||||
|
logging.basicConfig(level=logging.INFO)
|
||||||
|
|
||||||
|
# Determine operation mode
|
||||||
|
if suggest_schema:
|
||||||
|
logger.info(f"Analyzing {input_file} to suggest schemas...")
|
||||||
|
logger.info(f"Sample size: {sample_size} records")
|
||||||
|
logger.info(f"Sample chars: {sample_chars} characters")
|
||||||
|
# TODO: Implement schema suggestion
|
||||||
|
print(f"Would analyze {input_file} and suggest matching schemas")
|
||||||
|
print(f"Using sample of {sample_size} records, max {sample_chars} characters")
|
||||||
|
|
||||||
|
elif generate_descriptor:
|
||||||
|
logger.info(f"Generating descriptor from {input_file}...")
|
||||||
|
logger.info(f"Sample size: {sample_size} records")
|
||||||
|
logger.info(f"Sample chars: {sample_chars} characters")
|
||||||
|
if schema_name:
|
||||||
|
logger.info(f"Target schema: {schema_name}")
|
||||||
|
# TODO: Implement descriptor generation
|
||||||
|
print(f"Would generate descriptor from {input_file}")
|
||||||
|
print(f"Using sample of {sample_size} records, max {sample_chars} characters")
|
||||||
|
if output_file:
|
||||||
|
print(f"Would save descriptor to {output_file}")
|
||||||
|
|
||||||
|
elif parse_only:
|
||||||
|
if not descriptor_file:
|
||||||
|
raise ValueError("--descriptor is required when using --parse-only")
|
||||||
|
logger.info(f"Parsing {input_file} with descriptor {descriptor_file}...")
|
||||||
|
# TODO: Implement parsing
|
||||||
|
print(f"Would parse {input_file} using {descriptor_file}")
|
||||||
|
if output_file:
|
||||||
|
print(f"Would save parsed data to {output_file}")
|
||||||
|
|
||||||
|
else:
|
||||||
|
# Full pipeline: parse and import
|
||||||
|
if not descriptor_file:
|
||||||
|
# Auto-generate descriptor if not provided
|
||||||
|
logger.info("No descriptor provided, auto-generating...")
|
||||||
|
# TODO: Generate descriptor
|
||||||
|
print(f"Would auto-generate descriptor from {input_file}")
|
||||||
|
|
||||||
|
logger.info(f"Processing {input_file} for import...")
|
||||||
|
# TODO: Implement full pipeline
|
||||||
|
print(f"Would process and import data from {input_file}")
|
||||||
|
if dry_run:
|
||||||
|
print("Dry run mode - no data will be imported")
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Main entry point for the CLI."""
|
||||||
|
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
prog='tg-load-structured-data',
|
||||||
|
description=__doc__,
|
||||||
|
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||||
|
epilog="""
|
||||||
|
Examples:
|
||||||
|
# Step 1: Analyze data and suggest matching schemas
|
||||||
|
%(prog)s --input customers.csv --suggest-schema
|
||||||
|
|
||||||
|
# Step 2: Generate descriptor configuration from data sample
|
||||||
|
%(prog)s --input customers.csv --generate-descriptor --schema-name customer --output descriptor.json
|
||||||
|
|
||||||
|
# Generate descriptor with custom sampling (more data for better analysis)
|
||||||
|
%(prog)s --input large_dataset.csv --generate-descriptor --schema-name product --sample-chars 100000 --sample-size 500
|
||||||
|
|
||||||
|
# Step 3: Parse data and review output without importing
|
||||||
|
%(prog)s --input customers.csv --descriptor descriptor.json --parse-only --output parsed.json
|
||||||
|
|
||||||
|
# Step 4: Import data to TrustGraph using descriptor
|
||||||
|
%(prog)s --input customers.csv --descriptor descriptor.json
|
||||||
|
|
||||||
|
# All-in-one: Auto-generate descriptor and import (for simple cases)
|
||||||
|
%(prog)s --input customers.csv --schema-name customer
|
||||||
|
|
||||||
|
# Dry run to validate without importing
|
||||||
|
%(prog)s --input customers.csv --descriptor descriptor.json --dry-run
|
||||||
|
|
||||||
|
Use Cases:
|
||||||
|
--suggest-schema : Diagnose which TrustGraph schemas might match your data
|
||||||
|
--generate-descriptor: Create/review the structured data language configuration
|
||||||
|
--parse-only : Validate that parsed data looks correct before import
|
||||||
|
(no mode flags) : Full pipeline - parse and import to TrustGraph
|
||||||
|
|
||||||
|
For more information on the descriptor format, see:
|
||||||
|
docs/tech-specs/structured-data-descriptor.md
|
||||||
|
""".strip()
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
'-u', '--api-url',
|
||||||
|
default=default_url,
|
||||||
|
help=f'TrustGraph API URL (default: {default_url})'
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
'-i', '--input',
|
||||||
|
required=True,
|
||||||
|
help='Path to input data file to process'
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
'-d', '--descriptor',
|
||||||
|
help='Path to JSON descriptor configuration file (required for full import and parse-only)'
|
||||||
|
)
|
||||||
|
|
||||||
|
# Operation modes (mutually exclusive)
|
||||||
|
mode_group = parser.add_mutually_exclusive_group()
|
||||||
|
mode_group.add_argument(
|
||||||
|
'--suggest-schema',
|
||||||
|
action='store_true',
|
||||||
|
help='Analyze data sample and suggest matching TrustGraph schemas'
|
||||||
|
)
|
||||||
|
mode_group.add_argument(
|
||||||
|
'--generate-descriptor',
|
||||||
|
action='store_true',
|
||||||
|
help='Generate descriptor configuration from data sample'
|
||||||
|
)
|
||||||
|
mode_group.add_argument(
|
||||||
|
'--parse-only',
|
||||||
|
action='store_true',
|
||||||
|
help='Parse data using descriptor but don\'t import to TrustGraph'
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
'-o', '--output',
|
||||||
|
help='Output file path (for generated descriptors or parsed data)'
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
'--sample-size',
|
||||||
|
type=int,
|
||||||
|
default=100,
|
||||||
|
help='Number of records to sample for analysis/generation (default: 100)'
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
'--sample-chars',
|
||||||
|
type=int,
|
||||||
|
default=500,
|
||||||
|
help='Maximum characters to read from data file for sampling (default: 500)'
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
'--schema-name',
|
||||||
|
help='Target schema name for descriptor generation'
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
'--dry-run',
|
||||||
|
action='store_true',
|
||||||
|
help='Validate configuration and data without importing (full pipeline only)'
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
'-v', '--verbose',
|
||||||
|
action='store_true',
|
||||||
|
help='Enable verbose output for debugging'
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
'--batch-size',
|
||||||
|
type=int,
|
||||||
|
default=1000,
|
||||||
|
help='Number of records to process in each batch (default: 1000)'
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
'--max-errors',
|
||||||
|
type=int,
|
||||||
|
default=100,
|
||||||
|
help='Maximum number of errors before stopping (default: 100)'
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
'--error-file',
|
||||||
|
help='Path to write error records (optional)'
|
||||||
|
)
|
||||||
|
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
# Validate argument combinations
|
||||||
|
if args.parse_only and not args.descriptor:
|
||||||
|
print("Error: --descriptor is required when using --parse-only", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
if not any([args.suggest_schema, args.generate_descriptor, args.parse_only]) and not args.descriptor:
|
||||||
|
# Full pipeline mode without descriptor - schema_name should be provided
|
||||||
|
if not args.schema_name:
|
||||||
|
print("Error: --descriptor or --schema-name is required for full import", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
try:
|
||||||
|
load_structured_data(
|
||||||
|
api_url=args.api_url,
|
||||||
|
input_file=args.input,
|
||||||
|
descriptor_file=args.descriptor,
|
||||||
|
suggest_schema=args.suggest_schema,
|
||||||
|
generate_descriptor=args.generate_descriptor,
|
||||||
|
parse_only=args.parse_only,
|
||||||
|
output_file=args.output,
|
||||||
|
sample_size=args.sample_size,
|
||||||
|
sample_chars=args.sample_chars,
|
||||||
|
schema_name=args.schema_name,
|
||||||
|
dry_run=args.dry_run,
|
||||||
|
verbose=args.verbose
|
||||||
|
)
|
||||||
|
except FileNotFoundError as e:
|
||||||
|
print(f"Error: File not found - {e}", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
print(f"Error: Invalid JSON in descriptor - {e}", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Error: {e}", file=sys.stderr)
|
||||||
|
if args.verbose:
|
||||||
|
import traceback
|
||||||
|
traceback.print_exc()
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Loading…
Add table
Add a link
Reference in a new issue