trustgraph/ai-context/trustgraph-templates/docs/tech-specs/tests.md

# Test Specification

## Test Categories

### Unit Tests
Test Python modules in isolation:
- **Generator** - Jsonnet template processing, import callbacks
- **Packager** - Zip file creation, configuration assembly
- **API** - Template listing, version resolution
- **CLI** - Argument parsing, error handling, exit codes

### Integration Tests
Test full CLI workflow:
- Template compilation across version/platform/config matrix
- Output file generation (TrustGraph config + platform resources)
- Error propagation and reporting

### Validation Tests
Verify correctness of generated outputs:
- Syntax validation (JSON/YAML parsing)
- Schema validation (structure compliance)
- Semantic validation (cross-references, consistency)
- Regression testing (golden files)

## Test Matrix

**Dimensions:**
- Versions: 1.6, 1.7, 1.8
- Platforms: docker-compose, podman-compose, minikube-k8s, gcp-k8s, aks-k8s, eks-k8s, scw-k8s, ovh-k8s
- Configs: minimal.json, complex-rag.json, multi-service.json, cloud-aws.json

**Total combinations:** 3 versions × 8 platforms × 4 configs = 96 combinations per output type = 192 tests

## Validation Approaches

### Syntax Validation
- **JSON**: Parse with `json.loads()`, check no exceptions
- **YAML**: Parse with `yaml.safe_load()`, check no exceptions
- **Docker Compose**: Validate with `docker-compose config`
- **Kubernetes**: Validate with `kubectl apply --dry-run=client`

### Schema Validation
- **TrustGraph Config**: Define JSON schema, validate with `jsonschema`
  - Required fields: services, modules, parameters
  - Type checking for all configuration values
  - Enum validation for fixed sets (llm providers, platforms)

- **Kubernetes**: Check required fields
  - apiVersion, kind, metadata present
  - metadata.name, metadata.namespace defined
  - spec structure matches resource kind

- **Docker Compose**: Check required fields
  - services defined with image/build
  - Valid port mappings
  - Valid volume definitions

### Semantic Validation

#### Kubernetes Resources
- **Label/Selector matching**: Deployment selectors match pod labels
- **Volume references**: volumeMounts reference defined volumes
- **Service targeting**: Service selectors match deployment labels
- **Port consistency**: containerPort matches service targetPort
- **ConfigMap/Secret references**: Referenced resources exist in manifest

#### Docker Compose
- **Service dependencies**: depends_on references valid services
- **Volume references**: Volume names in bind mounts are defined
- **Network references**: Networks used by services are defined
- **Port conflicts**: No duplicate host port bindings
- **Environment variable references**: ${VAR} expansions are resolvable

#### TrustGraph Config
- **Service references**: Configured services reference valid modules
- **Parameter validation**: Module parameters match schema
- **Storage consistency**: Graph/object/vector stores configured correctly
- **LLM configuration**: Valid model IDs, API configurations

### Golden File Testing
Store reference outputs for each test case:
- **Location**: `tests/golden/{version}/{platform}/{config}/`
- **Files**:
  - `tg-config.json` - Reference TrustGraph configuration
  - `resources.yaml` - Reference platform resources
- **Comparison**: Use `pytest-golden` for automatic diff generation
- **Updates**: Explicit flag to regenerate golden files when intentional changes occur

## Test Cases

### Compilation Tests
For each (version, platform, config) combination:
1. Run `tg-build-deployment -t {version} -p {platform} -i {config} -O`
2. Assert exit code = 0
3. Assert stdout contains valid JSON
4. Parse and validate TrustGraph config structure

5. Run `tg-build-deployment -t {version} -p {platform} -i {config} -R`
6. Assert exit code = 0
7. Assert stdout contains valid YAML
8. Parse and validate resource manifest structure

### Error Handling Tests
- **Invalid config**: Malformed JSON input → exit code 1, error to stderr
- **Missing file**: Non-existent config file → exit code 1, error to stderr
- **Invalid template**: Non-existent version → exit code 1, error to stderr
- **Invalid platform**: Non-existent platform → exit code 1, error to stderr
- **Template errors**: Jsonnet compilation errors → exit code 1, error to stderr

### CLI Interface Tests
- **Argument parsing**: Valid/invalid argument combinations
- **Help output**: `-h` flag displays usage
- **Version display**: Version flag shows package version
- **Output modes**: `-O` and `-R` flags produce correct output types
- **Default values**: Missing optional args use documented defaults

### Module Unit Tests

#### Generator
- `process(config)` - Valid jsonnet → parsed JSON
- `process(config)` - Invalid jsonnet → raises exception
- Import callback mechanism works correctly
- Template loading from package resources

#### Packager
- `write(config, output)` - Creates valid zip file
- `write_tg_config(config)` - Outputs TrustGraph config to stdout
- `write_resources(config)` - Outputs platform resources to stdout
- Template version resolution (--latest, --latest-stable)
- Platform-specific template selection

## Test Execution Methods

### Direct Function Call (Primary Method)
Most tests call the Python entry point function directly rather than invoking the subprocess:

```python
from trustgraph_configurator import run
import sys
import json

def test_basic_compilation(monkeypatch, capsys):
    """Test compilation by calling run() directly"""
    # Mock sys.argv with CLI arguments
    monkeypatch.setattr(sys, 'argv', [
        'tg-build-deployment',
        '-t', '1.8',
        '-p', 'docker-compose',
        '-i', 'tests/configs/minimal.json',
        '-O'
    ])

    # Call the entry point directly
    run.run()

    # Capture and validate output
    captured = capsys.readouterr()
    config = json.loads(captured.out)
    assert 'services' in config
```

**Advantages:**
- **Fast**: No subprocess overhead (100x+ faster)
- **Easy stdout/stderr capture**: Use pytest's `capsys` fixture
- **Easy mocking**: Use `monkeypatch` for arguments, environment, file system
- **Better debugging**: Direct code path, breakpoints work naturally
- **Exit code testing**: Catch `SystemExit` exception to verify exit codes

```python
def test_error_handling(monkeypatch):
    """Test that errors exit with code 1"""
    monkeypatch.setattr(sys, 'argv', [
        'tg-build-deployment',
        '-i', 'nonexistent.json'
    ])

    with pytest.raises(SystemExit) as exc_info:
        run.run()

    assert exc_info.value.code == 1
```

### Subprocess Invocation (Smoke Tests)
A small number of tests (1-2) should invoke the actual CLI executable to verify installation:

```python
import subprocess

def test_cli_executable_installed():
    """Verify the installed CLI entry point works"""
    result = subprocess.run(
        ['tg-build-deployment', '--help'],
        capture_output=True,
        text=True
    )
    assert result.returncode == 0
    assert 'usage:' in result.stdout

def test_cli_version_command():
    """Verify version command works from CLI"""
    result = subprocess.run(
        ['tg-build-deployment', '--version'],
        capture_output=True,
        text=True
    )
    assert result.returncode == 0
```

**Purpose:**
- Verify `pyproject.toml` entry point configuration is correct
- Verify CLI is accessible in PATH after installation
- End-to-end smoke test

**Limitations:**
- Slower (subprocess overhead)
- Harder to mock/patch
- Less detailed error information

### Fixture for Direct Execution
Create a reusable fixture for calling configurator:

```python
# tests/conftest.py
import pytest
import sys
from io import StringIO

@pytest.fixture
def run_configurator(monkeypatch, capsys):
    """Fixture to run configurator with given arguments"""
    def _run(args):
        """
        Run configurator with args list.
        Returns (stdout, stderr, exit_code)
        """
        from trustgraph_configurator import run

        monkeypatch.setattr(sys, 'argv', ['tg-build-deployment'] + args)

        exit_code = 0
        try:
            run.run()
        except SystemExit as e:
            exit_code = e.code or 0

        captured = capsys.readouterr()
        return captured.out, captured.err, exit_code

    return _run
```

**Usage:**
```python
def test_with_fixture(run_configurator):
    stdout, stderr, code = run_configurator([
        '-t', '1.8',
        '-p', 'docker-compose',
        '-i', 'tests/configs/minimal.json',
        '-O'
    ])
    assert code == 0
    assert json.loads(stdout)
```

## Test Infrastructure

### Pytest Configuration
```toml
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
addopts = [
    "-v",
    "--strict-markers",
    "--cov=trustgraph_configurator",
    "--cov-report=term-missing",
    "--cov-report=html",
]
markers = [
    "unit: Unit tests",
    "integration: Integration tests",
    "validation: Output validation tests",
    "slow: Slow-running tests",
]
```

### Fixtures (`tests/conftest.py`)
- `test_config_dir` - Path to tests/configs/
- `test_configs` - Dict of loaded test configurations
- `temp_output_dir` - Temporary directory for test outputs
- `run_configurator` - Function to execute configurator CLI
- `golden_dir` - Path to golden file directory for test case

### Parametrization
Use `pytest.mark.parametrize` for matrix testing:
```python
@pytest.mark.parametrize("version", ["1.6", "1.7", "1.8"])
@pytest.mark.parametrize("platform", ["docker-compose", "minikube-k8s", ...])
@pytest.mark.parametrize("config", ["minimal.json", "complex-rag.json", ...])
def test_compilation(version, platform, config, run_configurator):
    ...
```

### Parallel Execution
Use pytest-xdist for parallel test execution:
```bash
pytest -n auto  # Use all CPU cores
```

## Test File Organization

```
tests/
├── conftest.py              # Shared fixtures
├── unit/
│   ├── test_generator.py
│   ├── test_packager.py
│   ├── test_api.py
│   └── test_run.py
├── integration/
│   ├── test_compilation.py  # Template compilation matrix
│   ├── test_cli.py          # CLI interface tests
│   └── test_errors.py       # Error handling tests
├── validation/
│   ├── test_syntax.py       # Syntax validation
│   ├── test_schema.py       # Schema validation
│   ├── test_semantics_k8s.py
│   ├── test_semantics_docker.py
│   └── test_semantics_tg.py
├── configs/                 # Test input configs (existing)
├── schemas/                 # JSON schemas for validation
│   ├── trustgraph-config.schema.json
│   ├── kubernetes-deployment.schema.json
│   └── docker-compose.schema.json
├── golden/                  # Reference outputs
│   └── {version}/{platform}/{config}/
│       ├── tg-config.json
│       └── resources.yaml
└── validators/              # Validation helper modules
    ├── kubernetes.py
    ├── docker_compose.py
    └── trustgraph.py
```

## Development Dependencies

Add to pyproject.toml:
```toml
[project.optional-dependencies]
dev = [
    "pytest>=7.0",
    "pytest-xdist>=3.0",      # Parallel execution
    "pytest-cov>=4.0",         # Coverage reporting
    "pytest-golden>=0.2",      # Golden file testing
    "jsonschema>=4.0",         # Schema validation
    "pyyaml>=6.0",             # Already in main deps
]
```

Install for development:
```bash
pip install -e .[dev]
```

## CI/CD Integration

Update `.github/workflows/pull-request.yaml`:
```yaml
- name: Install dependencies
  run: |
    python3 -m venv env
    . env/bin/activate
    pip install -e .[dev]

- name: Run tests
  run: |
    . env/bin/activate
    pytest -n auto --cov --cov-report=xml

- name: Upload coverage
  uses: codecov/codecov-action@v3
  with:
    file: ./coverage.xml
```

## Running Tests

```bash
# All tests
pytest

# Specific category
pytest tests/unit/
pytest tests/integration/
pytest -m validation

# Specific test file
pytest tests/unit/test_generator.py

# Parallel execution
pytest -n auto

# With coverage
pytest --cov=trustgraph_configurator --cov-report=html

# Update golden files
pytest --update-golden

# Verbose output
pytest -v

# Stop on first failure
pytest -x
```