apunkt/trustgraph
1
0
Fork 0
mirror of https://github.com/trustgraph-ai/trustgraph.git synced 2026-04-25 00:16:23 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Flow class -> flow blueprint

Browse source
This commit is contained in:
Cyber MacGeddon 2026-01-14 12:18:23 +00:00
parent ef576b9ad0
commit 15c3363a40
3 changed files with 58 additions and 58 deletions
Download patch file Download diff file Expand all files Collapse all files

44
docs/apis/api-flow.md
View file

@ -1,6 +1,6 @@
# TrustGraph Flow API
This API provides workflow management for TrustGraph components. It manages flow classes
This API provides workflow management for TrustGraph components. It manages flow blueprintes
(workflow templates) and flow instances (active running workflows) that orchestrate
complex data processing pipelines.
@ -10,26 +10,26 @@ complex data processing pipelines.
The request contains the following fields:
- `operation`: The operation to perform (see operations below)
- `class_name`: Flow class name (for class operations and start-flow)
- `class_definition`: Flow class definition JSON (for put-class)
- `blueprint_name`: Flow blueprint name (for class operations and start-flow)
- `class_definition`: Flow blueprint definition JSON (for put-class)
- `description`: Flow description (for start-flow)
- `flow_id`: Flow instance ID (for flow instance operations)
### Response
The response contains the following fields:
- `class_names`: Array of flow class names (returned by list-classes)
- `blueprint_names`: Array of flow blueprint names (returned by list-classes)
- `flow_ids`: Array of active flow IDs (returned by list-flows)
- `class_definition`: Flow class definition JSON (returned by get-class)
- `class_definition`: Flow blueprint definition JSON (returned by get-class)
- `flow`: Flow instance JSON (returned by get-flow)
- `description`: Flow description (returned by get-flow)
- `error`: Error information if operation fails
## Operations
### Flow Class Operations
### Flow Blueprint Operations
#### LIST-CLASSES - List All Flow Classes
#### LIST-CLASSES - List All Flow Blueprintes
Request:
```json
@ -41,17 +41,17 @@ Request:
Response:
```json
{
"class_names": ["pdf-processor", "text-analyzer", "knowledge-extractor"]
"blueprint_names": ["pdf-processor", "text-analyzer", "knowledge-extractor"]
}
```
#### GET-CLASS - Get Flow Class Definition
#### GET-CLASS - Get Flow Blueprint Definition
Request:
```json
{
"operation": "get-class",
"class_name": "pdf-processor"
"blueprint_name": "pdf-processor"
}
```
@ -62,13 +62,13 @@ Response:
}
```
#### PUT-CLASS - Create/Update Flow Class
#### PUT-CLASS - Create/Update Flow Blueprint
Request:
```json
{
"operation": "put-class",
"class_name": "pdf-processor",
"blueprint_name": "pdf-processor",
"class_definition": "{\"interfaces\": {\"text-completion\": {\"request\": \"persistent://tg/request/text-completion\", \"response\": \"persistent://tg/response/text-completion\"}}, \"description\": \"PDF processing workflow\"}"
}
```
@ -78,13 +78,13 @@ Response:
{}
```
#### DELETE-CLASS - Remove Flow Class
#### DELETE-CLASS - Remove Flow Blueprint
Request:
```json
{
"operation": "delete-class",
"class_name": "pdf-processor"
"blueprint_name": "pdf-processor"
}
```
@ -135,7 +135,7 @@ Request:
```json
{
"operation": "start-flow",
"class_name": "pdf-processor",
"blueprint_name": "pdf-processor",
"flow_id": "flow-123",
"description": "Processing document batch 1"
}
@ -186,7 +186,7 @@ Response:
{
"id": "unique-request-id",
"response": {
"class_names": ["pdf-processor", "text-analyzer"]
"blueprint_names": ["pdf-processor", "text-analyzer"]
},
"complete": true
}
@ -264,11 +264,11 @@ from trustgraph.api.flow import FlowClient
client = FlowClient()
# List all flow classes
classes = await client.list_classes()
# List all flow blueprintes
classes = await client.list_blueprints()
# Get a flow class definition
definition = await client.get_class("pdf-processor")
# Get a flow blueprint definition
definition = await client.get_blueprint("pdf-processor")
# Start a flow instance
await client.start_flow("pdf-processor", "flow-123", "Processing batch 1")
@ -286,8 +286,8 @@ result = await flow.mcp_tool("file-reader", {"path": "/path/to/file.txt"})
## Features
- **Flow Classes**: Templates that define workflow structure and interfaces
- **Flow Instances**: Active running workflows based on flow classes
- **Flow Blueprintes**: Templates that define workflow structure and interfaces
- **Flow Instances**: Active running workflows based on flow blueprintes
- **Dynamic Management**: Flows can be started/stopped dynamically
- **Template Processing**: Uses template replacement for customizing flow instances
- **Integration**: Works with TrustGraph ecosystem for data processing pipelines

20
docs/tech-specs/flow-class-definition.md
View file

@ -1,15 +1,15 @@
# Flow Class Definition Specification
# Flow Blueprint Definition Specification
## Overview
A flow class defines a complete dataflow pattern template in the TrustGraph system. When instantiated, it creates an interconnected network of processors that handle data ingestion, processing, storage, and querying as a unified system.
A flow blueprint defines a complete dataflow pattern template in the TrustGraph system. When instantiated, it creates an interconnected network of processors that handle data ingestion, processing, storage, and querying as a unified system.
## Structure
A flow class definition consists of five main sections:
A flow blueprint definition consists of five main sections:
### 1. Class Section
Defines shared service processors that are instantiated once per flow class. These processors handle requests from all flow instances of this class.
Defines shared service processors that are instantiated once per flow blueprint. These processors handle requests from all flow instances of this class.
```json
"class": {
@ -100,7 +100,7 @@ Maps flow-specific parameter names to centrally-stored parameter definitions:
- Reduces duplication of parameter schemas
### 5. Metadata
Additional information about the flow class:
Additional information about the flow blueprint:
```json
"description": "Human-readable description",
@ -117,7 +117,7 @@ Additional information about the flow class:
- Example: `flow-123`, `customer-A-flow`
#### {class}
- Replaced with the flow class name
- Replaced with the flow blueprint name
- Creates shared resources across flows of the same class
- Example: `standard-rag`, `enterprise-rag`
@ -203,14 +203,14 @@ Parameter names in settings correspond to keys in the flow's `parameters` sectio
## Queue Patterns (Pulsar)
Flow classes use Apache Pulsar for messaging. Queue names follow the Pulsar format:
Flow blueprintes use Apache Pulsar for messaging. Queue names follow the Pulsar format:
```
<persistence>://<tenant>/<namespace>/<topic>
```
### Components:
- **persistence**: `persistent` or `non-persistent` (Pulsar persistence mode)
- **tenant**: `tg` for TrustGraph-supplied flow class definitions
- **tenant**: `tg` for TrustGraph-supplied flow blueprint definitions
- **namespace**: Indicates the messaging pattern
- `flow`: Fire-and-forget services
- `request`: Request portion of request/response services
@ -232,7 +232,7 @@ Flow classes use Apache Pulsar for messaging. Queue names follow the Pulsar form
## Dataflow Architecture
The flow class creates a unified dataflow where:
The flow blueprint creates a unified dataflow where:
1. **Document Processing Pipeline**: Flows from ingestion through transformation to storage
2. **Query Services**: Integrated processors that query the same data stores and services
@ -245,7 +245,7 @@ All processors (both `{id}` and `{class}`) work together as a cohesive dataflow
Given:
- Flow Instance ID: `customer-A-flow`
- Flow Class: `standard-rag`
- Flow Blueprint: `standard-rag`
- Flow parameter mappings:
- `"model": "llm-model"`
- `"temp": "temperature"`

52
docs/tech-specs/flow-configurable-parameters.md
View file

@ -1,8 +1,8 @@
# Flow Class Configurable Parameters Technical Specification
# Flow Blueprint Configurable Parameters Technical Specification
## Overview
This specification describes the implementation of configurable parameters for flow classes in TrustGraph. Parameters enable users to customize processor parameters at flow launch time by providing values that replace parameter placeholders in the flow class definition.
This specification describes the implementation of configurable parameters for flow blueprintes in TrustGraph. Parameters enable users to customize processor parameters at flow launch time by providing values that replace parameter placeholders in the flow blueprint definition.
Parameters work through template variable substitution in processor parameters, similar to how `{id}` and `{class}` variables work, but with user-provided values.
@ -21,25 +21,25 @@ The integration supports four primary use cases:
- **Template Substitution**: Seamlessly replace parameter placeholders in processor parameters
- **UI Integration**: Enable parameter input through both API and UI interfaces
- **Type Safety**: Ensure parameter types match expected processor parameter types
- **Documentation**: Self-documenting parameter schemas within flow class definitions
- **Backward Compatibility**: Maintain compatibility with existing flow classes that don't use parameters
- **Documentation**: Self-documenting parameter schemas within flow blueprint definitions
- **Backward Compatibility**: Maintain compatibility with existing flow blueprintes that don't use parameters
## Background
Flow classes in TrustGraph now support processor parameters that can contain either fixed values or parameter placeholders. This creates an opportunity for runtime customization.
Flow blueprintes in TrustGraph now support processor parameters that can contain either fixed values or parameter placeholders. This creates an opportunity for runtime customization.
Current processor parameters support:
- Fixed values: `"model": "gemma3:12b"`
- Parameter placeholders: `"model": "gemma3:{model-size}"`
This specification defines how parameters are:
- Declared in flow class definitions
- Declared in flow blueprint definitions
- Validated when flows are launched
- Substituted in processor parameters
- Exposed through APIs and UI
By leveraging parameterized processor parameters, TrustGraph can:
- Reduce flow class duplication by using parameters for variations
- Reduce flow blueprint duplication by using parameters for variations
- Enable users to tune processor behavior without modifying definitions
- Support environment-specific configurations through parameter values
- Maintain type safety through parameter schema validation
@ -51,7 +51,7 @@ By leveraging parameterized processor parameters, TrustGraph can:
The configurable parameters system requires the following technical components:
1. **Parameter Schema Definition**
- JSON Schema-based parameter definitions within flow class metadata
- JSON Schema-based parameter definitions within flow blueprint metadata
- Type definitions including string, number, boolean, enum, and object types
- Validation rules including min/max values, patterns, and required fields
@ -92,7 +92,7 @@ The configurable parameters system requires the following technical components:
#### Parameter Definitions (Stored in Schema/Config)
Parameter definitions are stored centrally in the schema and config system with type "parameter-types":
Parameter definitions are stored centrally in the schema and config system with type "parameter-type":
```json
{
@ -146,9 +146,9 @@ Parameter definitions are stored centrally in the schema and config system with
}
```
#### Flow Class with Parameter References
#### Flow Blueprint with Parameter References
Flow classes define parameter metadata with type references, descriptions, and ordering:
Flow blueprintes define parameter metadata with type references, descriptions, and ordering:
```json
{
@ -225,7 +225,7 @@ The `parameters` section maps flow-specific parameter names (keys) to parameter
- `controlled-by` (optional): Name of another parameter that controls this parameter's value when in simple mode. When specified, this parameter inherits its value from the controlling parameter unless explicitly overridden
This approach allows:
- Reusable parameter type definitions across multiple flow classes
- Reusable parameter type definitions across multiple flow blueprintes
- Centralized parameter type management and validation
- Flow-specific parameter descriptions and ordering
- Enhanced UI experience with descriptive parameter forms
@ -253,7 +253,7 @@ The flow launch API accepts parameters using the flow's parameter names:
Note: In this example, `llm-rag-model` is not explicitly provided but will inherit the value "claude-3" from `llm-model` due to its `controlled-by` relationship. Similarly, `chunk-overlap` could inherit a calculated value based on `chunk-size`.
The system will:
1. Extract parameter metadata from flow class definition
1. Extract parameter metadata from flow blueprint definition
2. Map flow parameter names to their type definitions (e.g., `llm-model` → `llm-model` type)
3. Resolve controlled-by relationships (e.g., `llm-rag-model` inherits from `llm-model`)
4. Validate user-provided and inherited values against the parameter type definitions
@ -265,14 +265,14 @@ The system will:
When a flow is started, the system performs the following parameter resolution steps:
1. **Flow Class Loading**: Load flow class definition and extract parameter metadata
2. **Metadata Extraction**: Extract `type`, `description`, `order`, `advanced`, and `controlled-by` for each parameter defined in the flow class's `parameters` section
3. **Type Definition Lookup**: For each parameter in the flow class:
1. **Flow Blueprint Loading**: Load flow blueprint definition and extract parameter metadata
2. **Metadata Extraction**: Extract `type`, `description`, `order`, `advanced`, and `controlled-by` for each parameter defined in the flow blueprint's `parameters` section
3. **Type Definition Lookup**: For each parameter in the flow blueprint:
- Retrieve the parameter type definition from schema/config store using the `type` field
- The type definitions are stored with type "parameter-types" in the config system
- The type definitions are stored with type "parameter-type" in the config system
- Each type definition contains the parameter's schema, default value, and validation rules
4. **Default Value Resolution**:
- For each parameter defined in the flow class:
- For each parameter defined in the flow blueprint:
- Check if the user provided a value for this parameter
- If no user value provided, use the `default` value from the parameter type definition
- Build a complete parameter map containing both user-provided and default values
@ -361,7 +361,7 @@ The flow configuration service (`trustgraph-flow/trustgraph/config/service/flow.
Resolve parameters by merging user-provided values with defaults.
Args:
flow_class: The flow class definition dict
flow_class: The flow blueprint definition dict
user_params: User-provided parameters dict
Returns:
@ -370,20 +370,20 @@ The flow configuration service (`trustgraph-flow/trustgraph/config/service/flow.
```
This function should:
- Extract parameter metadata from the flow class's `parameters` section
- Extract parameter metadata from the flow blueprint's `parameters` section
- For each parameter, fetch its type definition from config store
- Apply defaults for any parameters not provided by the user
- Handle `controlled-by` inheritance relationships
- Return the complete parameter set
2. **Modified `handle_start_flow` Method**
- Call `resolve_parameters` after loading the flow class
- Call `resolve_parameters` after loading the flow blueprint
- Use the complete resolved parameter set for template substitution
- Store the complete parameter set (not just user-provided) with the flow
- Validate that all required parameters have values
3. **Parameter Type Fetching**
- Parameter type definitions are stored in config with type "parameter-types"
- Parameter type definitions are stored in config with type "parameter-type"
- Each type definition contains schema, default value, and validation rules
- Cache frequently-used parameter types to reduce config lookups
@ -400,12 +400,12 @@ The flow configuration service (`trustgraph-flow/trustgraph/config/service/flow.
4. **Library CLI Commands**
- CLI commands that start flows need parameter support:
- Accept parameter values via command-line flags or configuration files
- Validate parameters against flow class definitions before submission
- Validate parameters against flow blueprint definitions before submission
- Support parameter file input (JSON/YAML) for complex parameter sets
- CLI commands that show flows need to display parameter information:
- Show parameter values used when the flow was started
- Display available parameters for a flow class
- Display available parameters for a flow blueprint
- Show parameter validation schemas and defaults
#### Processor Base Class Integration
@ -456,7 +456,7 @@ Substituted in processor: "0.7" (string)
## Migration Plan
1. The system should continue to support flow classes with no parameters
1. The system should continue to support flow blueprintes with no parameters
declared.
2. The system should continue to support flows no parameters specified:
This works for flows with no parameters, and flows with parameters
@ -482,4 +482,4 @@ A: Just string substitution to remove strange injections and edge-cases.
## References
- JSON Schema Specification: https://json-schema.org/
- Flow Class Definition Spec: docs/tech-specs/flow-class-definition.md
- Flow Blueprint Definition Spec: docs/tech-specs/flow-class-definition.md