Squashed 'ai-context/trustgraph-templates/' content from commit 42a5fd1b

git-subtree-dir: ai-context/trustgraph-templates
git-subtree-split: 42a5fd1b678f32be378062e30451e2052ccb95dd
This commit is contained in:
elpresidank 2026-04-05 21:09:49 -05:00
commit 74cc8a4685
1216 changed files with 116347 additions and 0 deletions

244
docs/garage-deployment.md Normal file
View file

@ -0,0 +1,244 @@
# Garage S3-Compatible Object Storage
## Overview
Garage is a lightweight, self-hosted S3-compatible object storage system used as an alternative to Ceph for storing documents and other objects in the TrustGraph system. It provides a simpler deployment model while maintaining S3 API compatibility.
## Features
- **S3-Compatible API**: Full compatibility with AWS S3 SDK and CLI tools
- **Lightweight**: Minimal resource requirements compared to Ceph
- **Simple Deployment**: Single container with no complex dependencies
- **Distributed**: Supports multi-node clusters (though configured for single-node by default)
## Architecture
### Components
1. **Garage Daemon** (`garage`)
- Main storage service container
- Provides S3 API on port 3900
- Admin API on port 3903
- RPC communication on port 3901
2. **Init Container** (`garage-init`)
- Runs once to initialize the cluster
- Configures cluster layout
- Creates S3 access credentials
- Uses remote RPC to communicate with daemon (no shared volumes)
### Storage
Garage uses two separate volumes:
- **Metadata Volume** (`garage-meta`): Stores cluster metadata and LMDB database
- **Data Volume** (`garage-data`): Stores actual object data
## Configuration Parameters
All configuration is done via Jsonnet parameters with the `garage-` prefix:
### S3 Credentials
```jsonnet
"garage-access-key":: "GK000000000000000000000001",
"garage-secret-key":: "b171f00be9be4c32c734f4c05fe64c527a8ab5eb823b376cfa8c2531f70fc427",
```
**Format Requirements:**
- **Access Key ID**: Must start with `GK` followed by exactly 24 hex characters (0-9, a-f)
- **Secret Key**: Must be exactly 64 hex characters
**Generate Secure Credentials:**
```bash
# Generate Access Key ID (GK + 24 hex chars)
echo "GK$(openssl rand -hex 12)"
# Generate Secret Key (64 hex chars)
openssl rand -hex 32
```
### Cluster Configuration
```jsonnet
"garage-rpc-secret":: "bbba746a9e289bad64a9e7a36a4299dac8d6e0b8cc2a6c2937fe756df4492008",
"garage-admin-token":: "batts-rockhearted-unpartially",
"garage-region":: "garage",
"garage-replication-factor":: "1",
```
- **rpc-secret**: 64 hex characters for node-to-node RPC authentication
- **admin-token**: Bearer token for Admin API access
- **region**: S3 region name
- **replication-factor**: Number of data replicas (set to 1 for single-node, 3+ for production)
### Storage Volumes
```jsonnet
"garage-meta-size":: "5G",
"garage-data-size":: "100G",
```
Both values can be overridden using the `.with()` function:
```jsonnet
.with("meta-size", "10G")
.with("data-size", "500G")
```
## Integration with Librarian
The librarian component automatically connects to Garage for object storage:
```jsonnet
"--object-store-endpoint", url.object_store,
"--object-store-access-key", $["garage-access-key"],
"--object-store-secret-key", $["garage-secret-key"],
```
The object store endpoint is defined in `values/url.jsonnet`:
```jsonnet
object_store: "http://garage:3900",
```
## Testing Garage with S3 Clients
### Using AWS CLI
```bash
# Set credentials
export AWS_ACCESS_KEY_ID="GK000000000000000000000001"
export AWS_SECRET_ACCESS_KEY="b171f00be9be4c32c734f4c05fe64c527a8ab5eb823b376cfa8c2531f70fc427"
export AWS_ENDPOINT_URL="http://localhost:3900"
export AWS_DEFAULT_REGION="garage"
# Create a bucket
aws s3 mb s3://test-bucket
# Upload a file
echo "Hello from Garage!" > test.txt
aws s3 cp test.txt s3://test-bucket/
# List files
aws s3 ls s3://test-bucket/
# Download a file
aws s3 cp s3://test-bucket/test.txt downloaded.txt
# Verify
cat downloaded.txt
```
### Using s3cmd
Create configuration file `~/.s3cfg`:
```ini
[default]
access_key = GK000000000000000000000001
secret_key = b171f00be9be4c32c734f4c05fe64c527a8ab5eb823b376cfa8c2531f70fc427
host_base = localhost:3900
host_bucket = localhost:3900
use_https = False
```
Then use s3cmd:
```bash
# List buckets
s3cmd ls
# Create bucket
s3cmd mb s3://my-bucket
# Upload file
s3cmd put file.txt s3://my-bucket/
# Download file
s3cmd get s3://my-bucket/file.txt
```
## Deployment Details
### Initialization Process
The init container performs these steps:
1. **Wait for Garage daemon** - Polls `/health` endpoint until daemon is ready
2. **Get Node ID** - Queries `/v2/GetNodeInfo?node=self` via Admin API
3. **Configure Layout** - Assigns node to cluster with specified capacity via RPC
4. **Apply Layout** - Activates the cluster layout
5. **Import Credentials** - Creates S3 access key with provided credentials
6. **Grant Permissions** - Enables bucket creation for the key
All operations are **idempotent** - the init container can be restarted safely and will skip already-configured items.
### Network Architecture
- **S3 API**: Port 3900 (HTTP)
- **RPC**: Port 3901 (internal cluster communication)
- **Web UI**: Port 3902 (optional web interface)
- **Admin API**: Port 3903 (cluster management)
- **K2V API**: Port 3904 (key-value store)
### No Shared Volumes
The init container communicates with the Garage daemon entirely over the network:
- Admin API (HTTP) for status queries
- RPC (via garage CLI `-h` and `-s` flags) for cluster management
This design works across all orchestrators (Kubernetes, Docker Compose) without requiring shared volume mounts.
## Version Information
Current deployment uses **Garage v2.1.0**
- Image: `docker.io/dxflrs/garage:v2.1.0`
- Documentation: https://garagehq.deuxfleurs.fr/documentation/
## Troubleshooting
### Init Container Fails
Check logs: `podman logs <container-name>`
Common issues:
- **403 Forbidden**: Check `garage-admin-token` is correct
- **Invalid layout version**: Cluster already initialized, init will retry
- **Node ID null**: Admin API not responding, check daemon logs
### S3 Access Denied
Verify credentials format:
- Access Key ID must start with `GK` + 24 hex chars
- Secret Key must be 64 hex chars
- Credentials must match what was imported during init
### Check Garage Status
```bash
# Query cluster status
curl -H "Authorization: Bearer <admin-token>" \
http://localhost:3903/v2/GetClusterStatus
# Check health
curl http://localhost:3903/health
```
## Production Considerations
1. **Generate Secure Credentials**: Never use default credentials in production
2. **Set Replication Factor**: Use 3+ for redundancy in multi-node clusters
3. **Increase Storage Size**: Adjust `garage-data-size` based on expected usage
4. **Secure Admin Token**: Use a strong random token for `garage-admin-token`
5. **Monitor Storage**: Watch disk usage on data volume
6. **Backup Metadata**: The metadata volume contains critical cluster state
## References
- [Garage Documentation](https://garagehq.deuxfleurs.fr/documentation/)
- [Garage Admin API v2](https://garagehq.deuxfleurs.fr/api/garage-admin-v2.json)
- [Quick Start Guide](https://garagehq.deuxfleurs.fr/documentation/quick-start/)

416
docs/tech-specs/tests.md Normal file
View file

@ -0,0 +1,416 @@
# Test Specification
## Test Categories
### Unit Tests
Test Python modules in isolation:
- **Generator** - Jsonnet template processing, import callbacks
- **Packager** - Zip file creation, configuration assembly
- **API** - Template listing, version resolution
- **CLI** - Argument parsing, error handling, exit codes
### Integration Tests
Test full CLI workflow:
- Template compilation across version/platform/config matrix
- Output file generation (TrustGraph config + platform resources)
- Error propagation and reporting
### Validation Tests
Verify correctness of generated outputs:
- Syntax validation (JSON/YAML parsing)
- Schema validation (structure compliance)
- Semantic validation (cross-references, consistency)
- Regression testing (golden files)
## Test Matrix
**Dimensions:**
- Versions: 1.6, 1.7, 1.8
- Platforms: docker-compose, podman-compose, minikube-k8s, gcp-k8s, aks-k8s, eks-k8s, scw-k8s, ovh-k8s
- Configs: minimal.json, complex-rag.json, multi-service.json, cloud-aws.json
**Total combinations:** 3 versions × 8 platforms × 4 configs = 96 combinations per output type = 192 tests
## Validation Approaches
### Syntax Validation
- **JSON**: Parse with `json.loads()`, check no exceptions
- **YAML**: Parse with `yaml.safe_load()`, check no exceptions
- **Docker Compose**: Validate with `docker-compose config`
- **Kubernetes**: Validate with `kubectl apply --dry-run=client`
### Schema Validation
- **TrustGraph Config**: Define JSON schema, validate with `jsonschema`
- Required fields: services, modules, parameters
- Type checking for all configuration values
- Enum validation for fixed sets (llm providers, platforms)
- **Kubernetes**: Check required fields
- apiVersion, kind, metadata present
- metadata.name, metadata.namespace defined
- spec structure matches resource kind
- **Docker Compose**: Check required fields
- services defined with image/build
- Valid port mappings
- Valid volume definitions
### Semantic Validation
#### Kubernetes Resources
- **Label/Selector matching**: Deployment selectors match pod labels
- **Volume references**: volumeMounts reference defined volumes
- **Service targeting**: Service selectors match deployment labels
- **Port consistency**: containerPort matches service targetPort
- **ConfigMap/Secret references**: Referenced resources exist in manifest
#### Docker Compose
- **Service dependencies**: depends_on references valid services
- **Volume references**: Volume names in bind mounts are defined
- **Network references**: Networks used by services are defined
- **Port conflicts**: No duplicate host port bindings
- **Environment variable references**: ${VAR} expansions are resolvable
#### TrustGraph Config
- **Service references**: Configured services reference valid modules
- **Parameter validation**: Module parameters match schema
- **Storage consistency**: Graph/object/vector stores configured correctly
- **LLM configuration**: Valid model IDs, API configurations
### Golden File Testing
Store reference outputs for each test case:
- **Location**: `tests/golden/{version}/{platform}/{config}/`
- **Files**:
- `tg-config.json` - Reference TrustGraph configuration
- `resources.yaml` - Reference platform resources
- **Comparison**: Use `pytest-golden` for automatic diff generation
- **Updates**: Explicit flag to regenerate golden files when intentional changes occur
## Test Cases
### Compilation Tests
For each (version, platform, config) combination:
1. Run `tg-build-deployment -t {version} -p {platform} -i {config} -O`
2. Assert exit code = 0
3. Assert stdout contains valid JSON
4. Parse and validate TrustGraph config structure
5. Run `tg-build-deployment -t {version} -p {platform} -i {config} -R`
6. Assert exit code = 0
7. Assert stdout contains valid YAML
8. Parse and validate resource manifest structure
### Error Handling Tests
- **Invalid config**: Malformed JSON input → exit code 1, error to stderr
- **Missing file**: Non-existent config file → exit code 1, error to stderr
- **Invalid template**: Non-existent version → exit code 1, error to stderr
- **Invalid platform**: Non-existent platform → exit code 1, error to stderr
- **Template errors**: Jsonnet compilation errors → exit code 1, error to stderr
### CLI Interface Tests
- **Argument parsing**: Valid/invalid argument combinations
- **Help output**: `-h` flag displays usage
- **Version display**: Version flag shows package version
- **Output modes**: `-O` and `-R` flags produce correct output types
- **Default values**: Missing optional args use documented defaults
### Module Unit Tests
#### Generator
- `process(config)` - Valid jsonnet → parsed JSON
- `process(config)` - Invalid jsonnet → raises exception
- Import callback mechanism works correctly
- Template loading from package resources
#### Packager
- `write(config, output)` - Creates valid zip file
- `write_tg_config(config)` - Outputs TrustGraph config to stdout
- `write_resources(config)` - Outputs platform resources to stdout
- Template version resolution (--latest, --latest-stable)
- Platform-specific template selection
## Test Execution Methods
### Direct Function Call (Primary Method)
Most tests call the Python entry point function directly rather than invoking the subprocess:
```python
from trustgraph_configurator import run
import sys
import json
def test_basic_compilation(monkeypatch, capsys):
"""Test compilation by calling run() directly"""
# Mock sys.argv with CLI arguments
monkeypatch.setattr(sys, 'argv', [
'tg-build-deployment',
'-t', '1.8',
'-p', 'docker-compose',
'-i', 'tests/configs/minimal.json',
'-O'
])
# Call the entry point directly
run.run()
# Capture and validate output
captured = capsys.readouterr()
config = json.loads(captured.out)
assert 'services' in config
```
**Advantages:**
- **Fast**: No subprocess overhead (100x+ faster)
- **Easy stdout/stderr capture**: Use pytest's `capsys` fixture
- **Easy mocking**: Use `monkeypatch` for arguments, environment, file system
- **Better debugging**: Direct code path, breakpoints work naturally
- **Exit code testing**: Catch `SystemExit` exception to verify exit codes
```python
def test_error_handling(monkeypatch):
"""Test that errors exit with code 1"""
monkeypatch.setattr(sys, 'argv', [
'tg-build-deployment',
'-i', 'nonexistent.json'
])
with pytest.raises(SystemExit) as exc_info:
run.run()
assert exc_info.value.code == 1
```
### Subprocess Invocation (Smoke Tests)
A small number of tests (1-2) should invoke the actual CLI executable to verify installation:
```python
import subprocess
def test_cli_executable_installed():
"""Verify the installed CLI entry point works"""
result = subprocess.run(
['tg-build-deployment', '--help'],
capture_output=True,
text=True
)
assert result.returncode == 0
assert 'usage:' in result.stdout
def test_cli_version_command():
"""Verify version command works from CLI"""
result = subprocess.run(
['tg-build-deployment', '--version'],
capture_output=True,
text=True
)
assert result.returncode == 0
```
**Purpose:**
- Verify `pyproject.toml` entry point configuration is correct
- Verify CLI is accessible in PATH after installation
- End-to-end smoke test
**Limitations:**
- Slower (subprocess overhead)
- Harder to mock/patch
- Less detailed error information
### Fixture for Direct Execution
Create a reusable fixture for calling configurator:
```python
# tests/conftest.py
import pytest
import sys
from io import StringIO
@pytest.fixture
def run_configurator(monkeypatch, capsys):
"""Fixture to run configurator with given arguments"""
def _run(args):
"""
Run configurator with args list.
Returns (stdout, stderr, exit_code)
"""
from trustgraph_configurator import run
monkeypatch.setattr(sys, 'argv', ['tg-build-deployment'] + args)
exit_code = 0
try:
run.run()
except SystemExit as e:
exit_code = e.code or 0
captured = capsys.readouterr()
return captured.out, captured.err, exit_code
return _run
```
**Usage:**
```python
def test_with_fixture(run_configurator):
stdout, stderr, code = run_configurator([
'-t', '1.8',
'-p', 'docker-compose',
'-i', 'tests/configs/minimal.json',
'-O'
])
assert code == 0
assert json.loads(stdout)
```
## Test Infrastructure
### Pytest Configuration
```toml
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
addopts = [
"-v",
"--strict-markers",
"--cov=trustgraph_configurator",
"--cov-report=term-missing",
"--cov-report=html",
]
markers = [
"unit: Unit tests",
"integration: Integration tests",
"validation: Output validation tests",
"slow: Slow-running tests",
]
```
### Fixtures (`tests/conftest.py`)
- `test_config_dir` - Path to tests/configs/
- `test_configs` - Dict of loaded test configurations
- `temp_output_dir` - Temporary directory for test outputs
- `run_configurator` - Function to execute configurator CLI
- `golden_dir` - Path to golden file directory for test case
### Parametrization
Use `pytest.mark.parametrize` for matrix testing:
```python
@pytest.mark.parametrize("version", ["1.6", "1.7", "1.8"])
@pytest.mark.parametrize("platform", ["docker-compose", "minikube-k8s", ...])
@pytest.mark.parametrize("config", ["minimal.json", "complex-rag.json", ...])
def test_compilation(version, platform, config, run_configurator):
...
```
### Parallel Execution
Use pytest-xdist for parallel test execution:
```bash
pytest -n auto # Use all CPU cores
```
## Test File Organization
```
tests/
├── conftest.py # Shared fixtures
├── unit/
│ ├── test_generator.py
│ ├── test_packager.py
│ ├── test_api.py
│ └── test_run.py
├── integration/
│ ├── test_compilation.py # Template compilation matrix
│ ├── test_cli.py # CLI interface tests
│ └── test_errors.py # Error handling tests
├── validation/
│ ├── test_syntax.py # Syntax validation
│ ├── test_schema.py # Schema validation
│ ├── test_semantics_k8s.py
│ ├── test_semantics_docker.py
│ └── test_semantics_tg.py
├── configs/ # Test input configs (existing)
├── schemas/ # JSON schemas for validation
│ ├── trustgraph-config.schema.json
│ ├── kubernetes-deployment.schema.json
│ └── docker-compose.schema.json
├── golden/ # Reference outputs
│ └── {version}/{platform}/{config}/
│ ├── tg-config.json
│ └── resources.yaml
└── validators/ # Validation helper modules
├── kubernetes.py
├── docker_compose.py
└── trustgraph.py
```
## Development Dependencies
Add to pyproject.toml:
```toml
[project.optional-dependencies]
dev = [
"pytest>=7.0",
"pytest-xdist>=3.0", # Parallel execution
"pytest-cov>=4.0", # Coverage reporting
"pytest-golden>=0.2", # Golden file testing
"jsonschema>=4.0", # Schema validation
"pyyaml>=6.0", # Already in main deps
]
```
Install for development:
```bash
pip install -e .[dev]
```
## CI/CD Integration
Update `.github/workflows/pull-request.yaml`:
```yaml
- name: Install dependencies
run: |
python3 -m venv env
. env/bin/activate
pip install -e .[dev]
- name: Run tests
run: |
. env/bin/activate
pytest -n auto --cov --cov-report=xml
- name: Upload coverage
uses: codecov/codecov-action@v3
with:
file: ./coverage.xml
```
## Running Tests
```bash
# All tests
pytest
# Specific category
pytest tests/unit/
pytest tests/integration/
pytest -m validation
# Specific test file
pytest tests/unit/test_generator.py
# Parallel execution
pytest -n auto
# With coverage
pytest --cov=trustgraph_configurator --cov-report=html
# Update golden files
pytest --update-golden
# Verbose output
pytest -v
# Stop on first failure
pytest -x
```

188
docs/templates-structure.md Normal file
View file

@ -0,0 +1,188 @@
# How TrustGraph Templates Are Structured
## Overview
The TrustGraph template system is a Jsonnet-based configuration framework that generates deployment configurations for multiple platforms (Docker Compose, Kubernetes, etc.) from a single JSON configuration file. The system uses a component-based architecture with abstraction layers for different deployment targets.
## Directory Structure
```
trustgraph_configurator/
├── packager.py # Main entry point
├── generator.py # Jsonnet processor wrapper
├── templates/
│ └── 1.3/ # Template version
│ ├── components.jsonnet # Component registry
│ ├── config-to-docker-compose.jsonnet # Docker Compose generator
│ ├── config-to-tg-configuration.jsonnet # TrustGraph config extractor
│ ├── components/ # Component definitions
│ ├── engine/ # Platform-specific engines
│ ├── util/ # Utility functions
│ ├── prompts/ # LLM prompt templates
│ └── values/ # Shared configuration values
└── resources/
└── 1.3/ # Static resource files
├── grafana/ # Grafana dashboards/configs
└── prometheus/ # Prometheus configs
```
## Core Concepts
### 1. Components
Components are the building blocks of the system. Each component:
- Defines configuration parameters with defaults
- Implements a `create` function that generates platform-specific resources
- Can compose with other components through Jsonnet's object composition (`+`)
- Lives in `components/` directory
Example component structure (simplified `ollama.jsonnet`):
```jsonnet
{
// Parameter with default value
"ollama-model":: "gemma2:9b",
// Service definition
"text-completion" +: {
create:: function(engine)
// Use engine abstraction to create resources
local container = engine.container("text-completion")
.with_image(...)
.with_command([...]);
engine.resources([container, ...])
},
// Custom parameter setter
with:: function(key, value)
self + { ["ollama-" + key]:: value }
}
```
### 2. Engines
Engines provide platform-specific implementations for resource creation. Each engine implements:
- `container()` - Create container definitions
- `service()` - Create service definitions
- `volume()` - Create volume definitions
- `resources()` - Aggregate resources for output
The engine abstraction allows components to be platform-agnostic. Available engines:
- `docker-compose.jsonnet` - Docker Compose format
- `noop.jsonnet` - No-op engine for configuration extraction only
- Kubernetes engines (various cloud providers)
### 3. Configuration Flow
```
config.json → decode → patterns → engine.create() → resources → output
```
1. **Input**: `config.json` contains a list of components to enable:
```json
[
{"name": "trustgraph-base", "parameters": {}},
{"name": "ollama", "parameters": {"model": "mixtral"}}
]
```
2. **Decode**: The `decode-config.jsonnet` utility:
- Loads each component from the registry
- Applies parameters using the `with_params` function
- Merges all components into a single "patterns" object
3. **Engine Processing**: Platform-specific files like `config-to-docker-compose.jsonnet`:
- Call each component's `create(engine)` function
- Fold results into final resource structure
- Output platform-specific configuration
### 4. Configuration Extraction
The `config-to-tg-configuration.jsonnet` file extracts the TrustGraph runtime configuration from components. This includes:
- Prompt templates
- Flow definitions
- Model token costs
- Agent tools
- MCP server configurations
This configuration is embedded into the deployment separately from the infrastructure resources.
## Processing Pipeline
### Entry Point: `packager.py`
The Packager class orchestrates the entire process:
1. **Template Selection**: Determines version and template based on user input
2. **Resource Generation**:
- For Docker Compose: Calls `config-to-docker-compose.jsonnet`
- For Kubernetes: Calls `config-to-<platform>-k8s.jsonnet`
3. **Configuration Generation**: Calls `config-to-tg-configuration.jsonnet` for runtime config
4. **Packaging**: Creates ZIP file with all generated files and static resources
### Jsonnet Processing: `generator.py`
Simple wrapper around the Jsonnet library that:
- Evaluates Jsonnet templates
- Provides custom import callback for file resolution
- Returns parsed JSON output
## Component Composition
Components can be composed in several ways:
### 1. Base Component Extension
Many components extend `trustgraph-base` which provides core services:
```jsonnet
local trustgraph = import "components/trustgraph.jsonnet";
// Inherits all trustgraph services
{} + trustgraph + myCustomizations
```
### 2. Field Merging
Components use `+:` to merge with existing fields:
```jsonnet
"text-completion" +: {
// Adds to existing text-completion definition
create:: function(engine) ...
}
```
### 3. Parameter Injection
The `with` pattern allows runtime parameter injection:
```jsonnet
with:: function(key, value)
self + { [key]:: value }
```
## Hidden Fields and Configuration
Jsonnet's `::` operator creates hidden fields that aren't included in JSON output by default. The template system uses this for:
- Default values that can be overridden
- Internal helper functions
- Configuration that needs special extraction
For example, `trustgraph-base` has a hidden `configuration::` field containing runtime config that's extracted separately by `config-to-tg-configuration.jsonnet`.
## Platform Abstraction
The engine pattern provides clean separation between:
- **Component logic** - What services/containers to create
- **Platform specifics** - How to represent them (Docker Compose YAML, K8s manifests, etc.)
This allows the same component definitions to generate configurations for multiple platforms without modification.
## Static Resources
Files in `resources/` are copied directly to the output package. These include:
- Grafana dashboard definitions
- Prometheus configuration
- Other platform-specific configs that don't need templating
## Best Practices
1. **Component Independence**: Components should be self-contained and not depend on specific ordering
2. **Parameter Namespacing**: Use prefixes (e.g., `ollama-model`) to avoid conflicts
3. **Hidden Fields for Defaults**: Use `::` for overridable defaults
4. **Engine Abstraction**: Always use engine methods rather than creating platform-specific structures directly
5. **Composition Over Inheritance**: Use Jsonnet's object composition (`+`) rather than complex inheritance hierarchies