Flow configurable parameters (#532)

* Fix pyproject.toml missing requests dep

* parameters is now parameter-types

* Update flow parameters tech spec for recent changes (no impact on this repo)
This commit is contained in:
cybermaggedon 2025-09-25 19:11:40 +01:00 committed by GitHub
parent 7a3bfad826
commit aa8e422e8c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 48 additions and 23 deletions

View file

@ -80,8 +80,10 @@ The configurable parameters system requires the following technical components:
Module: trustgraph-flow/trustgraph/flow/launcher.py Module: trustgraph-flow/trustgraph/flow/launcher.py
5. **UI Parameter Forms** 5. **UI Parameter Forms**
- Dynamic form generation from parameter schemas - Dynamic form generation from flow parameter metadata
- Input validation and error display - Ordered parameter display using `order` field
- Descriptive parameter labels using `description` field
- Input validation against parameter type definitions
- Parameter presets and templates - Parameter presets and templates
Module: trustgraph-ui/components/flow-parameters/ Module: trustgraph-ui/components/flow-parameters/
@ -90,7 +92,7 @@ The configurable parameters system requires the following technical components:
#### Parameter Definitions (Stored in Schema/Config) #### Parameter Definitions (Stored in Schema/Config)
Parameter definitions are stored centrally in the schema and config system with type "parameters": Parameter definitions are stored centrally in the schema and config system with type "parameter-types":
```json ```json
{ {
@ -129,16 +131,32 @@ Parameter definitions are stored centrally in the schema and config system with
#### Flow Class with Parameter References #### Flow Class with Parameter References
Flow classes reference parameter definitions by name: Flow classes define parameter metadata with type references, descriptions, and ordering:
```json ```json
{ {
"flow_class": "document-analysis", "flow_class": "document-analysis",
"parameters": { "parameters": {
"model": "llm-model", "model": {
"size": "model-size", "type": "llm-model",
"temp": "temperature", "description": "LLM model to use for document analysis",
"chunk": "chunk-size" "order": 1
},
"size": {
"type": "model-size",
"description": "Model size variant to use",
"order": 2
},
"temp": {
"type": "temperature",
"description": "Generation temperature for creativity control",
"order": 3
},
"chunk": {
"type": "chunk-size",
"description": "Document chunk size for processing",
"order": 4
}
}, },
"class": { "class": {
"text-completion:{class}": { "text-completion:{class}": {
@ -163,15 +181,18 @@ Flow classes reference parameter definitions by name:
} }
``` ```
The `parameters` section maps flow-specific parameter names (keys) to centrally-defined parameter definitions (values). The `parameters` section maps flow-specific parameter names (keys) to parameter metadata objects containing:
``` - `type`: Reference to centrally-defined parameter definition (e.g., "llm-model")
- `description`: Human-readable description for UI display
- `order`: Display order for parameter forms (lower numbers appear first)
This approach allows: This approach allows:
- Reusable parameter definitions across multiple flow classes - Reusable parameter type definitions across multiple flow classes
- Centralized parameter management and updates - Centralized parameter type management and validation
- Reduced duplication in flow class definitions - Flow-specific parameter descriptions and ordering
- Enhanced UI experience with descriptive parameter forms
- Consistent parameter validation across flows - Consistent parameter validation across flows
- Easy addition of new standard parameters - Easy addition of new standard parameter types
#### Flow Launch Request #### Flow Launch Request
@ -191,20 +212,23 @@ The flow launch API accepts parameters using the flow's parameter names:
``` ```
The system will: The system will:
1. Map flow parameter names to their definitions (e.g., `model``llm-model`) 1. Extract parameter metadata from flow class definition
2. Validate values against the parameter definitions 2. Map flow parameter names to their type definitions (e.g., `model``llm-model` type)
3. Substitute values into processor parameters during flow instantiation 3. Validate user-provided values against the parameter type definitions
4. Substitute validated values into processor parameters during flow instantiation
### Implementation Details ### Implementation Details
#### Parameter Resolution Process #### Parameter Resolution Process
1. **Flow Class Loading**: Load flow class and extract parameter references 1. **Flow Class Loading**: Load flow class and extract parameter metadata
2. **Definition Lookup**: Retrieve parameter definitions from schema/config store 2. **Metadata Extraction**: Extract `type`, `description`, and `order` for each parameter
3. **Validation**: Validate user-provided parameters against definitions 3. **Type Definition Lookup**: Retrieve parameter type definitions from schema/config store using `type` field
4. **Default Application**: Apply default values for missing parameters 4. **UI Form Generation**: Use `description` and `order` fields to create ordered parameter forms
5. **Template Substitution**: Replace parameter placeholders in processor parameters 5. **Validation**: Validate user-provided parameters against type definitions
6. **Processor Instantiation**: Create processors with substituted parameters 6. **Default Application**: Apply default values for missing parameters from type definitions
7. **Template Substitution**: Replace parameter placeholders in processor parameters with validated values
8. **Processor Instantiation**: Create processors with substituted parameters
#### Pulsar Integration #### Pulsar Integration

View file

@ -12,6 +12,7 @@ requires-python = ">=3.8"
dependencies = [ dependencies = [
"pulsar-client", "pulsar-client",
"prometheus-client", "prometheus-client",
"requests",
] ]
classifiers = [ classifiers = [
"Programming Language :: Python :: 3", "Programming Language :: Python :: 3",