mirror of
https://github.com/trustgraph-ai/trustgraph.git
synced 2026-07-03 15:01:00 +02:00
Squashed 'ai-context/trustgraph-templates/' content from commit 42a5fd1b
git-subtree-dir: ai-context/trustgraph-templates git-subtree-split: 42a5fd1b678f32be378062e30451e2052ccb95dd
This commit is contained in:
commit
74cc8a4685
1216 changed files with 116347 additions and 0 deletions
244
docs/garage-deployment.md
Normal file
244
docs/garage-deployment.md
Normal file
|
|
@ -0,0 +1,244 @@
|
|||
# Garage S3-Compatible Object Storage
|
||||
|
||||
## Overview
|
||||
|
||||
Garage is a lightweight, self-hosted S3-compatible object storage system used as an alternative to Ceph for storing documents and other objects in the TrustGraph system. It provides a simpler deployment model while maintaining S3 API compatibility.
|
||||
|
||||
## Features
|
||||
|
||||
- **S3-Compatible API**: Full compatibility with AWS S3 SDK and CLI tools
|
||||
- **Lightweight**: Minimal resource requirements compared to Ceph
|
||||
- **Simple Deployment**: Single container with no complex dependencies
|
||||
- **Distributed**: Supports multi-node clusters (though configured for single-node by default)
|
||||
|
||||
## Architecture
|
||||
|
||||
### Components
|
||||
|
||||
1. **Garage Daemon** (`garage`)
|
||||
- Main storage service container
|
||||
- Provides S3 API on port 3900
|
||||
- Admin API on port 3903
|
||||
- RPC communication on port 3901
|
||||
|
||||
2. **Init Container** (`garage-init`)
|
||||
- Runs once to initialize the cluster
|
||||
- Configures cluster layout
|
||||
- Creates S3 access credentials
|
||||
- Uses remote RPC to communicate with daemon (no shared volumes)
|
||||
|
||||
### Storage
|
||||
|
||||
Garage uses two separate volumes:
|
||||
|
||||
- **Metadata Volume** (`garage-meta`): Stores cluster metadata and LMDB database
|
||||
- **Data Volume** (`garage-data`): Stores actual object data
|
||||
|
||||
## Configuration Parameters
|
||||
|
||||
All configuration is done via Jsonnet parameters with the `garage-` prefix:
|
||||
|
||||
### S3 Credentials
|
||||
|
||||
```jsonnet
|
||||
"garage-access-key":: "GK000000000000000000000001",
|
||||
"garage-secret-key":: "b171f00be9be4c32c734f4c05fe64c527a8ab5eb823b376cfa8c2531f70fc427",
|
||||
```
|
||||
|
||||
**Format Requirements:**
|
||||
- **Access Key ID**: Must start with `GK` followed by exactly 24 hex characters (0-9, a-f)
|
||||
- **Secret Key**: Must be exactly 64 hex characters
|
||||
|
||||
**Generate Secure Credentials:**
|
||||
|
||||
```bash
|
||||
# Generate Access Key ID (GK + 24 hex chars)
|
||||
echo "GK$(openssl rand -hex 12)"
|
||||
|
||||
# Generate Secret Key (64 hex chars)
|
||||
openssl rand -hex 32
|
||||
```
|
||||
|
||||
### Cluster Configuration
|
||||
|
||||
```jsonnet
|
||||
"garage-rpc-secret":: "bbba746a9e289bad64a9e7a36a4299dac8d6e0b8cc2a6c2937fe756df4492008",
|
||||
"garage-admin-token":: "batts-rockhearted-unpartially",
|
||||
"garage-region":: "garage",
|
||||
"garage-replication-factor":: "1",
|
||||
```
|
||||
|
||||
- **rpc-secret**: 64 hex characters for node-to-node RPC authentication
|
||||
- **admin-token**: Bearer token for Admin API access
|
||||
- **region**: S3 region name
|
||||
- **replication-factor**: Number of data replicas (set to 1 for single-node, 3+ for production)
|
||||
|
||||
### Storage Volumes
|
||||
|
||||
```jsonnet
|
||||
"garage-meta-size":: "5G",
|
||||
"garage-data-size":: "100G",
|
||||
```
|
||||
|
||||
Both values can be overridden using the `.with()` function:
|
||||
|
||||
```jsonnet
|
||||
.with("meta-size", "10G")
|
||||
.with("data-size", "500G")
|
||||
```
|
||||
|
||||
## Integration with Librarian
|
||||
|
||||
The librarian component automatically connects to Garage for object storage:
|
||||
|
||||
```jsonnet
|
||||
"--object-store-endpoint", url.object_store,
|
||||
"--object-store-access-key", $["garage-access-key"],
|
||||
"--object-store-secret-key", $["garage-secret-key"],
|
||||
```
|
||||
|
||||
The object store endpoint is defined in `values/url.jsonnet`:
|
||||
|
||||
```jsonnet
|
||||
object_store: "http://garage:3900",
|
||||
```
|
||||
|
||||
## Testing Garage with S3 Clients
|
||||
|
||||
### Using AWS CLI
|
||||
|
||||
```bash
|
||||
# Set credentials
|
||||
export AWS_ACCESS_KEY_ID="GK000000000000000000000001"
|
||||
export AWS_SECRET_ACCESS_KEY="b171f00be9be4c32c734f4c05fe64c527a8ab5eb823b376cfa8c2531f70fc427"
|
||||
export AWS_ENDPOINT_URL="http://localhost:3900"
|
||||
export AWS_DEFAULT_REGION="garage"
|
||||
|
||||
# Create a bucket
|
||||
aws s3 mb s3://test-bucket
|
||||
|
||||
# Upload a file
|
||||
echo "Hello from Garage!" > test.txt
|
||||
aws s3 cp test.txt s3://test-bucket/
|
||||
|
||||
# List files
|
||||
aws s3 ls s3://test-bucket/
|
||||
|
||||
# Download a file
|
||||
aws s3 cp s3://test-bucket/test.txt downloaded.txt
|
||||
|
||||
# Verify
|
||||
cat downloaded.txt
|
||||
```
|
||||
|
||||
### Using s3cmd
|
||||
|
||||
Create configuration file `~/.s3cfg`:
|
||||
|
||||
```ini
|
||||
[default]
|
||||
access_key = GK000000000000000000000001
|
||||
secret_key = b171f00be9be4c32c734f4c05fe64c527a8ab5eb823b376cfa8c2531f70fc427
|
||||
host_base = localhost:3900
|
||||
host_bucket = localhost:3900
|
||||
use_https = False
|
||||
```
|
||||
|
||||
Then use s3cmd:
|
||||
|
||||
```bash
|
||||
# List buckets
|
||||
s3cmd ls
|
||||
|
||||
# Create bucket
|
||||
s3cmd mb s3://my-bucket
|
||||
|
||||
# Upload file
|
||||
s3cmd put file.txt s3://my-bucket/
|
||||
|
||||
# Download file
|
||||
s3cmd get s3://my-bucket/file.txt
|
||||
```
|
||||
|
||||
## Deployment Details
|
||||
|
||||
### Initialization Process
|
||||
|
||||
The init container performs these steps:
|
||||
|
||||
1. **Wait for Garage daemon** - Polls `/health` endpoint until daemon is ready
|
||||
2. **Get Node ID** - Queries `/v2/GetNodeInfo?node=self` via Admin API
|
||||
3. **Configure Layout** - Assigns node to cluster with specified capacity via RPC
|
||||
4. **Apply Layout** - Activates the cluster layout
|
||||
5. **Import Credentials** - Creates S3 access key with provided credentials
|
||||
6. **Grant Permissions** - Enables bucket creation for the key
|
||||
|
||||
All operations are **idempotent** - the init container can be restarted safely and will skip already-configured items.
|
||||
|
||||
### Network Architecture
|
||||
|
||||
- **S3 API**: Port 3900 (HTTP)
|
||||
- **RPC**: Port 3901 (internal cluster communication)
|
||||
- **Web UI**: Port 3902 (optional web interface)
|
||||
- **Admin API**: Port 3903 (cluster management)
|
||||
- **K2V API**: Port 3904 (key-value store)
|
||||
|
||||
### No Shared Volumes
|
||||
|
||||
The init container communicates with the Garage daemon entirely over the network:
|
||||
|
||||
- Admin API (HTTP) for status queries
|
||||
- RPC (via garage CLI `-h` and `-s` flags) for cluster management
|
||||
|
||||
This design works across all orchestrators (Kubernetes, Docker Compose) without requiring shared volume mounts.
|
||||
|
||||
## Version Information
|
||||
|
||||
Current deployment uses **Garage v2.1.0**
|
||||
|
||||
- Image: `docker.io/dxflrs/garage:v2.1.0`
|
||||
- Documentation: https://garagehq.deuxfleurs.fr/documentation/
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Init Container Fails
|
||||
|
||||
Check logs: `podman logs <container-name>`
|
||||
|
||||
Common issues:
|
||||
- **403 Forbidden**: Check `garage-admin-token` is correct
|
||||
- **Invalid layout version**: Cluster already initialized, init will retry
|
||||
- **Node ID null**: Admin API not responding, check daemon logs
|
||||
|
||||
### S3 Access Denied
|
||||
|
||||
Verify credentials format:
|
||||
- Access Key ID must start with `GK` + 24 hex chars
|
||||
- Secret Key must be 64 hex chars
|
||||
- Credentials must match what was imported during init
|
||||
|
||||
### Check Garage Status
|
||||
|
||||
```bash
|
||||
# Query cluster status
|
||||
curl -H "Authorization: Bearer <admin-token>" \
|
||||
http://localhost:3903/v2/GetClusterStatus
|
||||
|
||||
# Check health
|
||||
curl http://localhost:3903/health
|
||||
```
|
||||
|
||||
## Production Considerations
|
||||
|
||||
1. **Generate Secure Credentials**: Never use default credentials in production
|
||||
2. **Set Replication Factor**: Use 3+ for redundancy in multi-node clusters
|
||||
3. **Increase Storage Size**: Adjust `garage-data-size` based on expected usage
|
||||
4. **Secure Admin Token**: Use a strong random token for `garage-admin-token`
|
||||
5. **Monitor Storage**: Watch disk usage on data volume
|
||||
6. **Backup Metadata**: The metadata volume contains critical cluster state
|
||||
|
||||
## References
|
||||
|
||||
- [Garage Documentation](https://garagehq.deuxfleurs.fr/documentation/)
|
||||
- [Garage Admin API v2](https://garagehq.deuxfleurs.fr/api/garage-admin-v2.json)
|
||||
- [Quick Start Guide](https://garagehq.deuxfleurs.fr/documentation/quick-start/)
|
||||
416
docs/tech-specs/tests.md
Normal file
416
docs/tech-specs/tests.md
Normal file
|
|
@ -0,0 +1,416 @@
|
|||
# Test Specification
|
||||
|
||||
## Test Categories
|
||||
|
||||
### Unit Tests
|
||||
Test Python modules in isolation:
|
||||
- **Generator** - Jsonnet template processing, import callbacks
|
||||
- **Packager** - Zip file creation, configuration assembly
|
||||
- **API** - Template listing, version resolution
|
||||
- **CLI** - Argument parsing, error handling, exit codes
|
||||
|
||||
### Integration Tests
|
||||
Test full CLI workflow:
|
||||
- Template compilation across version/platform/config matrix
|
||||
- Output file generation (TrustGraph config + platform resources)
|
||||
- Error propagation and reporting
|
||||
|
||||
### Validation Tests
|
||||
Verify correctness of generated outputs:
|
||||
- Syntax validation (JSON/YAML parsing)
|
||||
- Schema validation (structure compliance)
|
||||
- Semantic validation (cross-references, consistency)
|
||||
- Regression testing (golden files)
|
||||
|
||||
## Test Matrix
|
||||
|
||||
**Dimensions:**
|
||||
- Versions: 1.6, 1.7, 1.8
|
||||
- Platforms: docker-compose, podman-compose, minikube-k8s, gcp-k8s, aks-k8s, eks-k8s, scw-k8s, ovh-k8s
|
||||
- Configs: minimal.json, complex-rag.json, multi-service.json, cloud-aws.json
|
||||
|
||||
**Total combinations:** 3 versions × 8 platforms × 4 configs = 96 combinations per output type = 192 tests
|
||||
|
||||
## Validation Approaches
|
||||
|
||||
### Syntax Validation
|
||||
- **JSON**: Parse with `json.loads()`, check no exceptions
|
||||
- **YAML**: Parse with `yaml.safe_load()`, check no exceptions
|
||||
- **Docker Compose**: Validate with `docker-compose config`
|
||||
- **Kubernetes**: Validate with `kubectl apply --dry-run=client`
|
||||
|
||||
### Schema Validation
|
||||
- **TrustGraph Config**: Define JSON schema, validate with `jsonschema`
|
||||
- Required fields: services, modules, parameters
|
||||
- Type checking for all configuration values
|
||||
- Enum validation for fixed sets (llm providers, platforms)
|
||||
|
||||
- **Kubernetes**: Check required fields
|
||||
- apiVersion, kind, metadata present
|
||||
- metadata.name, metadata.namespace defined
|
||||
- spec structure matches resource kind
|
||||
|
||||
- **Docker Compose**: Check required fields
|
||||
- services defined with image/build
|
||||
- Valid port mappings
|
||||
- Valid volume definitions
|
||||
|
||||
### Semantic Validation
|
||||
|
||||
#### Kubernetes Resources
|
||||
- **Label/Selector matching**: Deployment selectors match pod labels
|
||||
- **Volume references**: volumeMounts reference defined volumes
|
||||
- **Service targeting**: Service selectors match deployment labels
|
||||
- **Port consistency**: containerPort matches service targetPort
|
||||
- **ConfigMap/Secret references**: Referenced resources exist in manifest
|
||||
|
||||
#### Docker Compose
|
||||
- **Service dependencies**: depends_on references valid services
|
||||
- **Volume references**: Volume names in bind mounts are defined
|
||||
- **Network references**: Networks used by services are defined
|
||||
- **Port conflicts**: No duplicate host port bindings
|
||||
- **Environment variable references**: ${VAR} expansions are resolvable
|
||||
|
||||
#### TrustGraph Config
|
||||
- **Service references**: Configured services reference valid modules
|
||||
- **Parameter validation**: Module parameters match schema
|
||||
- **Storage consistency**: Graph/object/vector stores configured correctly
|
||||
- **LLM configuration**: Valid model IDs, API configurations
|
||||
|
||||
### Golden File Testing
|
||||
Store reference outputs for each test case:
|
||||
- **Location**: `tests/golden/{version}/{platform}/{config}/`
|
||||
- **Files**:
|
||||
- `tg-config.json` - Reference TrustGraph configuration
|
||||
- `resources.yaml` - Reference platform resources
|
||||
- **Comparison**: Use `pytest-golden` for automatic diff generation
|
||||
- **Updates**: Explicit flag to regenerate golden files when intentional changes occur
|
||||
|
||||
## Test Cases
|
||||
|
||||
### Compilation Tests
|
||||
For each (version, platform, config) combination:
|
||||
1. Run `tg-build-deployment -t {version} -p {platform} -i {config} -O`
|
||||
2. Assert exit code = 0
|
||||
3. Assert stdout contains valid JSON
|
||||
4. Parse and validate TrustGraph config structure
|
||||
|
||||
5. Run `tg-build-deployment -t {version} -p {platform} -i {config} -R`
|
||||
6. Assert exit code = 0
|
||||
7. Assert stdout contains valid YAML
|
||||
8. Parse and validate resource manifest structure
|
||||
|
||||
### Error Handling Tests
|
||||
- **Invalid config**: Malformed JSON input → exit code 1, error to stderr
|
||||
- **Missing file**: Non-existent config file → exit code 1, error to stderr
|
||||
- **Invalid template**: Non-existent version → exit code 1, error to stderr
|
||||
- **Invalid platform**: Non-existent platform → exit code 1, error to stderr
|
||||
- **Template errors**: Jsonnet compilation errors → exit code 1, error to stderr
|
||||
|
||||
### CLI Interface Tests
|
||||
- **Argument parsing**: Valid/invalid argument combinations
|
||||
- **Help output**: `-h` flag displays usage
|
||||
- **Version display**: Version flag shows package version
|
||||
- **Output modes**: `-O` and `-R` flags produce correct output types
|
||||
- **Default values**: Missing optional args use documented defaults
|
||||
|
||||
### Module Unit Tests
|
||||
|
||||
#### Generator
|
||||
- `process(config)` - Valid jsonnet → parsed JSON
|
||||
- `process(config)` - Invalid jsonnet → raises exception
|
||||
- Import callback mechanism works correctly
|
||||
- Template loading from package resources
|
||||
|
||||
#### Packager
|
||||
- `write(config, output)` - Creates valid zip file
|
||||
- `write_tg_config(config)` - Outputs TrustGraph config to stdout
|
||||
- `write_resources(config)` - Outputs platform resources to stdout
|
||||
- Template version resolution (--latest, --latest-stable)
|
||||
- Platform-specific template selection
|
||||
|
||||
## Test Execution Methods
|
||||
|
||||
### Direct Function Call (Primary Method)
|
||||
Most tests call the Python entry point function directly rather than invoking the subprocess:
|
||||
|
||||
```python
|
||||
from trustgraph_configurator import run
|
||||
import sys
|
||||
import json
|
||||
|
||||
def test_basic_compilation(monkeypatch, capsys):
|
||||
"""Test compilation by calling run() directly"""
|
||||
# Mock sys.argv with CLI arguments
|
||||
monkeypatch.setattr(sys, 'argv', [
|
||||
'tg-build-deployment',
|
||||
'-t', '1.8',
|
||||
'-p', 'docker-compose',
|
||||
'-i', 'tests/configs/minimal.json',
|
||||
'-O'
|
||||
])
|
||||
|
||||
# Call the entry point directly
|
||||
run.run()
|
||||
|
||||
# Capture and validate output
|
||||
captured = capsys.readouterr()
|
||||
config = json.loads(captured.out)
|
||||
assert 'services' in config
|
||||
```
|
||||
|
||||
**Advantages:**
|
||||
- **Fast**: No subprocess overhead (100x+ faster)
|
||||
- **Easy stdout/stderr capture**: Use pytest's `capsys` fixture
|
||||
- **Easy mocking**: Use `monkeypatch` for arguments, environment, file system
|
||||
- **Better debugging**: Direct code path, breakpoints work naturally
|
||||
- **Exit code testing**: Catch `SystemExit` exception to verify exit codes
|
||||
|
||||
```python
|
||||
def test_error_handling(monkeypatch):
|
||||
"""Test that errors exit with code 1"""
|
||||
monkeypatch.setattr(sys, 'argv', [
|
||||
'tg-build-deployment',
|
||||
'-i', 'nonexistent.json'
|
||||
])
|
||||
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
run.run()
|
||||
|
||||
assert exc_info.value.code == 1
|
||||
```
|
||||
|
||||
### Subprocess Invocation (Smoke Tests)
|
||||
A small number of tests (1-2) should invoke the actual CLI executable to verify installation:
|
||||
|
||||
```python
|
||||
import subprocess
|
||||
|
||||
def test_cli_executable_installed():
|
||||
"""Verify the installed CLI entry point works"""
|
||||
result = subprocess.run(
|
||||
['tg-build-deployment', '--help'],
|
||||
capture_output=True,
|
||||
text=True
|
||||
)
|
||||
assert result.returncode == 0
|
||||
assert 'usage:' in result.stdout
|
||||
|
||||
def test_cli_version_command():
|
||||
"""Verify version command works from CLI"""
|
||||
result = subprocess.run(
|
||||
['tg-build-deployment', '--version'],
|
||||
capture_output=True,
|
||||
text=True
|
||||
)
|
||||
assert result.returncode == 0
|
||||
```
|
||||
|
||||
**Purpose:**
|
||||
- Verify `pyproject.toml` entry point configuration is correct
|
||||
- Verify CLI is accessible in PATH after installation
|
||||
- End-to-end smoke test
|
||||
|
||||
**Limitations:**
|
||||
- Slower (subprocess overhead)
|
||||
- Harder to mock/patch
|
||||
- Less detailed error information
|
||||
|
||||
### Fixture for Direct Execution
|
||||
Create a reusable fixture for calling configurator:
|
||||
|
||||
```python
|
||||
# tests/conftest.py
|
||||
import pytest
|
||||
import sys
|
||||
from io import StringIO
|
||||
|
||||
@pytest.fixture
|
||||
def run_configurator(monkeypatch, capsys):
|
||||
"""Fixture to run configurator with given arguments"""
|
||||
def _run(args):
|
||||
"""
|
||||
Run configurator with args list.
|
||||
Returns (stdout, stderr, exit_code)
|
||||
"""
|
||||
from trustgraph_configurator import run
|
||||
|
||||
monkeypatch.setattr(sys, 'argv', ['tg-build-deployment'] + args)
|
||||
|
||||
exit_code = 0
|
||||
try:
|
||||
run.run()
|
||||
except SystemExit as e:
|
||||
exit_code = e.code or 0
|
||||
|
||||
captured = capsys.readouterr()
|
||||
return captured.out, captured.err, exit_code
|
||||
|
||||
return _run
|
||||
```
|
||||
|
||||
**Usage:**
|
||||
```python
|
||||
def test_with_fixture(run_configurator):
|
||||
stdout, stderr, code = run_configurator([
|
||||
'-t', '1.8',
|
||||
'-p', 'docker-compose',
|
||||
'-i', 'tests/configs/minimal.json',
|
||||
'-O'
|
||||
])
|
||||
assert code == 0
|
||||
assert json.loads(stdout)
|
||||
```
|
||||
|
||||
## Test Infrastructure
|
||||
|
||||
### Pytest Configuration
|
||||
```toml
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
python_files = ["test_*.py"]
|
||||
python_classes = ["Test*"]
|
||||
python_functions = ["test_*"]
|
||||
addopts = [
|
||||
"-v",
|
||||
"--strict-markers",
|
||||
"--cov=trustgraph_configurator",
|
||||
"--cov-report=term-missing",
|
||||
"--cov-report=html",
|
||||
]
|
||||
markers = [
|
||||
"unit: Unit tests",
|
||||
"integration: Integration tests",
|
||||
"validation: Output validation tests",
|
||||
"slow: Slow-running tests",
|
||||
]
|
||||
```
|
||||
|
||||
### Fixtures (`tests/conftest.py`)
|
||||
- `test_config_dir` - Path to tests/configs/
|
||||
- `test_configs` - Dict of loaded test configurations
|
||||
- `temp_output_dir` - Temporary directory for test outputs
|
||||
- `run_configurator` - Function to execute configurator CLI
|
||||
- `golden_dir` - Path to golden file directory for test case
|
||||
|
||||
### Parametrization
|
||||
Use `pytest.mark.parametrize` for matrix testing:
|
||||
```python
|
||||
@pytest.mark.parametrize("version", ["1.6", "1.7", "1.8"])
|
||||
@pytest.mark.parametrize("platform", ["docker-compose", "minikube-k8s", ...])
|
||||
@pytest.mark.parametrize("config", ["minimal.json", "complex-rag.json", ...])
|
||||
def test_compilation(version, platform, config, run_configurator):
|
||||
...
|
||||
```
|
||||
|
||||
### Parallel Execution
|
||||
Use pytest-xdist for parallel test execution:
|
||||
```bash
|
||||
pytest -n auto # Use all CPU cores
|
||||
```
|
||||
|
||||
## Test File Organization
|
||||
|
||||
```
|
||||
tests/
|
||||
├── conftest.py # Shared fixtures
|
||||
├── unit/
|
||||
│ ├── test_generator.py
|
||||
│ ├── test_packager.py
|
||||
│ ├── test_api.py
|
||||
│ └── test_run.py
|
||||
├── integration/
|
||||
│ ├── test_compilation.py # Template compilation matrix
|
||||
│ ├── test_cli.py # CLI interface tests
|
||||
│ └── test_errors.py # Error handling tests
|
||||
├── validation/
|
||||
│ ├── test_syntax.py # Syntax validation
|
||||
│ ├── test_schema.py # Schema validation
|
||||
│ ├── test_semantics_k8s.py
|
||||
│ ├── test_semantics_docker.py
|
||||
│ └── test_semantics_tg.py
|
||||
├── configs/ # Test input configs (existing)
|
||||
├── schemas/ # JSON schemas for validation
|
||||
│ ├── trustgraph-config.schema.json
|
||||
│ ├── kubernetes-deployment.schema.json
|
||||
│ └── docker-compose.schema.json
|
||||
├── golden/ # Reference outputs
|
||||
│ └── {version}/{platform}/{config}/
|
||||
│ ├── tg-config.json
|
||||
│ └── resources.yaml
|
||||
└── validators/ # Validation helper modules
|
||||
├── kubernetes.py
|
||||
├── docker_compose.py
|
||||
└── trustgraph.py
|
||||
```
|
||||
|
||||
## Development Dependencies
|
||||
|
||||
Add to pyproject.toml:
|
||||
```toml
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=7.0",
|
||||
"pytest-xdist>=3.0", # Parallel execution
|
||||
"pytest-cov>=4.0", # Coverage reporting
|
||||
"pytest-golden>=0.2", # Golden file testing
|
||||
"jsonschema>=4.0", # Schema validation
|
||||
"pyyaml>=6.0", # Already in main deps
|
||||
]
|
||||
```
|
||||
|
||||
Install for development:
|
||||
```bash
|
||||
pip install -e .[dev]
|
||||
```
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
Update `.github/workflows/pull-request.yaml`:
|
||||
```yaml
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python3 -m venv env
|
||||
. env/bin/activate
|
||||
pip install -e .[dev]
|
||||
|
||||
- name: Run tests
|
||||
run: |
|
||||
. env/bin/activate
|
||||
pytest -n auto --cov --cov-report=xml
|
||||
|
||||
- name: Upload coverage
|
||||
uses: codecov/codecov-action@v3
|
||||
with:
|
||||
file: ./coverage.xml
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
```bash
|
||||
# All tests
|
||||
pytest
|
||||
|
||||
# Specific category
|
||||
pytest tests/unit/
|
||||
pytest tests/integration/
|
||||
pytest -m validation
|
||||
|
||||
# Specific test file
|
||||
pytest tests/unit/test_generator.py
|
||||
|
||||
# Parallel execution
|
||||
pytest -n auto
|
||||
|
||||
# With coverage
|
||||
pytest --cov=trustgraph_configurator --cov-report=html
|
||||
|
||||
# Update golden files
|
||||
pytest --update-golden
|
||||
|
||||
# Verbose output
|
||||
pytest -v
|
||||
|
||||
# Stop on first failure
|
||||
pytest -x
|
||||
```
|
||||
188
docs/templates-structure.md
Normal file
188
docs/templates-structure.md
Normal file
|
|
@ -0,0 +1,188 @@
|
|||
# How TrustGraph Templates Are Structured
|
||||
|
||||
## Overview
|
||||
|
||||
The TrustGraph template system is a Jsonnet-based configuration framework that generates deployment configurations for multiple platforms (Docker Compose, Kubernetes, etc.) from a single JSON configuration file. The system uses a component-based architecture with abstraction layers for different deployment targets.
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
trustgraph_configurator/
|
||||
├── packager.py # Main entry point
|
||||
├── generator.py # Jsonnet processor wrapper
|
||||
├── templates/
|
||||
│ └── 1.3/ # Template version
|
||||
│ ├── components.jsonnet # Component registry
|
||||
│ ├── config-to-docker-compose.jsonnet # Docker Compose generator
|
||||
│ ├── config-to-tg-configuration.jsonnet # TrustGraph config extractor
|
||||
│ ├── components/ # Component definitions
|
||||
│ ├── engine/ # Platform-specific engines
|
||||
│ ├── util/ # Utility functions
|
||||
│ ├── prompts/ # LLM prompt templates
|
||||
│ └── values/ # Shared configuration values
|
||||
└── resources/
|
||||
└── 1.3/ # Static resource files
|
||||
├── grafana/ # Grafana dashboards/configs
|
||||
└── prometheus/ # Prometheus configs
|
||||
```
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### 1. Components
|
||||
|
||||
Components are the building blocks of the system. Each component:
|
||||
- Defines configuration parameters with defaults
|
||||
- Implements a `create` function that generates platform-specific resources
|
||||
- Can compose with other components through Jsonnet's object composition (`+`)
|
||||
- Lives in `components/` directory
|
||||
|
||||
Example component structure (simplified `ollama.jsonnet`):
|
||||
```jsonnet
|
||||
{
|
||||
// Parameter with default value
|
||||
"ollama-model":: "gemma2:9b",
|
||||
|
||||
// Service definition
|
||||
"text-completion" +: {
|
||||
create:: function(engine)
|
||||
// Use engine abstraction to create resources
|
||||
local container = engine.container("text-completion")
|
||||
.with_image(...)
|
||||
.with_command([...]);
|
||||
|
||||
engine.resources([container, ...])
|
||||
},
|
||||
|
||||
// Custom parameter setter
|
||||
with:: function(key, value)
|
||||
self + { ["ollama-" + key]:: value }
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Engines
|
||||
|
||||
Engines provide platform-specific implementations for resource creation. Each engine implements:
|
||||
- `container()` - Create container definitions
|
||||
- `service()` - Create service definitions
|
||||
- `volume()` - Create volume definitions
|
||||
- `resources()` - Aggregate resources for output
|
||||
|
||||
The engine abstraction allows components to be platform-agnostic. Available engines:
|
||||
- `docker-compose.jsonnet` - Docker Compose format
|
||||
- `noop.jsonnet` - No-op engine for configuration extraction only
|
||||
- Kubernetes engines (various cloud providers)
|
||||
|
||||
### 3. Configuration Flow
|
||||
|
||||
```
|
||||
config.json → decode → patterns → engine.create() → resources → output
|
||||
```
|
||||
|
||||
1. **Input**: `config.json` contains a list of components to enable:
|
||||
```json
|
||||
[
|
||||
{"name": "trustgraph-base", "parameters": {}},
|
||||
{"name": "ollama", "parameters": {"model": "mixtral"}}
|
||||
]
|
||||
```
|
||||
|
||||
2. **Decode**: The `decode-config.jsonnet` utility:
|
||||
- Loads each component from the registry
|
||||
- Applies parameters using the `with_params` function
|
||||
- Merges all components into a single "patterns" object
|
||||
|
||||
3. **Engine Processing**: Platform-specific files like `config-to-docker-compose.jsonnet`:
|
||||
- Call each component's `create(engine)` function
|
||||
- Fold results into final resource structure
|
||||
- Output platform-specific configuration
|
||||
|
||||
### 4. Configuration Extraction
|
||||
|
||||
The `config-to-tg-configuration.jsonnet` file extracts the TrustGraph runtime configuration from components. This includes:
|
||||
- Prompt templates
|
||||
- Flow definitions
|
||||
- Model token costs
|
||||
- Agent tools
|
||||
- MCP server configurations
|
||||
|
||||
This configuration is embedded into the deployment separately from the infrastructure resources.
|
||||
|
||||
## Processing Pipeline
|
||||
|
||||
### Entry Point: `packager.py`
|
||||
|
||||
The Packager class orchestrates the entire process:
|
||||
|
||||
1. **Template Selection**: Determines version and template based on user input
|
||||
2. **Resource Generation**:
|
||||
- For Docker Compose: Calls `config-to-docker-compose.jsonnet`
|
||||
- For Kubernetes: Calls `config-to-<platform>-k8s.jsonnet`
|
||||
3. **Configuration Generation**: Calls `config-to-tg-configuration.jsonnet` for runtime config
|
||||
4. **Packaging**: Creates ZIP file with all generated files and static resources
|
||||
|
||||
### Jsonnet Processing: `generator.py`
|
||||
|
||||
Simple wrapper around the Jsonnet library that:
|
||||
- Evaluates Jsonnet templates
|
||||
- Provides custom import callback for file resolution
|
||||
- Returns parsed JSON output
|
||||
|
||||
## Component Composition
|
||||
|
||||
Components can be composed in several ways:
|
||||
|
||||
### 1. Base Component Extension
|
||||
Many components extend `trustgraph-base` which provides core services:
|
||||
```jsonnet
|
||||
local trustgraph = import "components/trustgraph.jsonnet";
|
||||
// Inherits all trustgraph services
|
||||
{} + trustgraph + myCustomizations
|
||||
```
|
||||
|
||||
### 2. Field Merging
|
||||
Components use `+:` to merge with existing fields:
|
||||
```jsonnet
|
||||
"text-completion" +: {
|
||||
// Adds to existing text-completion definition
|
||||
create:: function(engine) ...
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Parameter Injection
|
||||
The `with` pattern allows runtime parameter injection:
|
||||
```jsonnet
|
||||
with:: function(key, value)
|
||||
self + { [key]:: value }
|
||||
```
|
||||
|
||||
## Hidden Fields and Configuration
|
||||
|
||||
Jsonnet's `::` operator creates hidden fields that aren't included in JSON output by default. The template system uses this for:
|
||||
- Default values that can be overridden
|
||||
- Internal helper functions
|
||||
- Configuration that needs special extraction
|
||||
|
||||
For example, `trustgraph-base` has a hidden `configuration::` field containing runtime config that's extracted separately by `config-to-tg-configuration.jsonnet`.
|
||||
|
||||
## Platform Abstraction
|
||||
|
||||
The engine pattern provides clean separation between:
|
||||
- **Component logic** - What services/containers to create
|
||||
- **Platform specifics** - How to represent them (Docker Compose YAML, K8s manifests, etc.)
|
||||
|
||||
This allows the same component definitions to generate configurations for multiple platforms without modification.
|
||||
|
||||
## Static Resources
|
||||
|
||||
Files in `resources/` are copied directly to the output package. These include:
|
||||
- Grafana dashboard definitions
|
||||
- Prometheus configuration
|
||||
- Other platform-specific configs that don't need templating
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Component Independence**: Components should be self-contained and not depend on specific ordering
|
||||
2. **Parameter Namespacing**: Use prefixes (e.g., `ollama-model`) to avoid conflicts
|
||||
3. **Hidden Fields for Defaults**: Use `::` for overridable defaults
|
||||
4. **Engine Abstraction**: Always use engine methods rather than creating platform-specific structures directly
|
||||
5. **Composition Over Inheritance**: Use Jsonnet's object composition (`+`) rather than complex inheritance hierarchies
|
||||
Loading…
Add table
Add a link
Reference in a new issue