plano/TEST_COVERAGE_ANALYSIS.md

# Test Coverage Analysis

**Date:** 2026-02-18

## Executive Summary

The Plano codebase has **~370 automated tests**: ~297 Rust unit tests, ~65 Python tests (29 CLI + 50 E2E + 4 archgw integration), 10 Hurl/REST manual test files, and zero JS/TS tests. Coverage is strong in the LLM translation layer (hermesllm) and behavioral signals (brightstaff/signals), moderate in state management and configuration, and weak in the `llm_gateway` WASM plugin and several Python CLI modules.

**Note:** The `prompt_gateway` crate is deprecated and excluded from recommendations.

Below is a detailed breakdown by component with prioritized improvement recommendations.

---

## 1. Rust Crates (`crates/`)

### Current State

| Crate | Tests | Files With Tests | Status |
|-------|-------|------------------|--------|
| hermesllm | 148 | 21 | Good — broad coverage of provider translation |
| brightstaff | 126 | 11 | Good — signals/state/routing well tested; handler endpoints less so |
| common | 36 | 10 | Moderate — core utilities covered; some gaps |
| prompt_gateway | 4 | 2 | Deprecated — not prioritized for new tests |
| llm_gateway | 0 | 0 | None — WASM filter completely untested |
| **Total** | **~314** | **44** | |

### Well-Tested Areas

- **hermesllm provider translation (148 tests):** Request/response transforms for all providers (OpenAI, Anthropic, Bedrock, Gemini, Mistral) are thoroughly tested. Streaming response parsing (20 tests), endpoint resolution (11 tests), request generation (16 tests), and cross-provider format conversion (~45 tests) are solid.
- **hermesllm streaming buffers (12 tests):** SSE chunk processor (6 tests), Anthropic streaming buffer (3 tests), Responses API streaming buffer (2 tests), and passthrough buffer (1 test) have coverage.
- **brightstaff signals/analyzer (48 tests):** Character n-gram similarity, token cosine similarity, layered matching, frustration/escalation/positive-feedback detection are thoroughly tested.
- **brightstaff state management (26 tests):** In-memory state (16 tests) and PostgreSQL persistence (10 tests) have good unit test coverage.
- **brightstaff function calling (17 tests):** Tool extraction, JSON fixing, hallucination detection, and tool call verification are well covered.
- **brightstaff routing models (17 tests):** Orchestrator model v1 (9 tests) and router model v1 (8 tests) are tested.
- **brightstaff pipeline processor (5 tests):** Has basic test coverage (4 tokio::test + 1 sync test).
- **brightstaff agent selector (5 tests):** Listener lookup and agent map creation are tested.
- **brightstaff response handler (5 tests):** Response transformation has tests.
- **common rate limiting (8 tests):** Rate limit logic with token quotas and header-based selectors is tested.
- **common OpenAI API (9 tests):** Chat completion parsing and request conversions covered.

### Gaps and Recommendations

#### Gap 1: `llm_gateway` crate — 0 tests (1,399 LOC)

This WASM filter handles all LLM request/response processing and streaming. `stream_context.rs` (~1,000 lines) manages streaming chunk assembly and response forwarding with zero coverage.

**Recommendation:** Extract core logic from the WASM host context into pure, testable functions. Test streaming chunk reassembly, header manipulation, error response construction, and the filter lifecycle. Consider a thin WASM shim over well-tested logic modules.

#### ~~Gap 2: `prompt_gateway` crate~~ — DEPRECATED (skipped)

The `prompt_gateway` crate is deprecated. Investing in new tests for this crate is not recommended.

#### Gap 2: brightstaff handler endpoints — limited coverage

Several handler modules have no unit tests:
- `handlers/llm.rs` (553 LOC) — LLM chat handler
- `handlers/agent_chat_completions.rs` (418 LOC) — Multi-agent orchestration
- `handlers/router_chat.rs` (159 LOC) — Router endpoint
- `handlers/utils.rs` (288 LOC) — Handler utilities

The pipeline_processor has only 5 tests for 834 LOC — basic flow is covered but error paths and edge cases are not.

**Recommendation:** Add tests for error paths in `pipeline_processor.rs` (malformed requests, downstream failures, timeout handling). Add handler-level tests for `llm.rs` and `agent_chat_completions.rs` using `mockito` (already a dev dependency) to mock HTTP backends.

#### Gap 4: hermesllm streaming *transforms* — 0 tests

While the streaming *buffers* (SSE parser, Anthropic buffer, etc.) have tests, the streaming *transform* modules that convert between formats during streaming are untested:
- `transforms/response_streaming/to_openai_streaming.rs`
- `transforms/response_streaming/to_anthropic_streaming.rs`

Also untested: `apis/streaming_shapes/amazon_bedrock_binary_frame.rs` (AWS Event Stream binary decoding) and `apis/streaming_shapes/chat_completions_streaming_buffer.rs`.

**Recommendation:** Add tests for the streaming transform modules. The Bedrock binary frame decoder is particularly important — it parses a proprietary binary protocol and failures here are hard to diagnose in production.

#### Gap 5: common utility modules — no tests

Several `common` modules lack tests:
- `routing.rs` — Provider routing logic
- `errors.rs` — Error types (ClientError, ServerError)
- `http.rs` — HTTP utilities and CallArgs
- `stats.rs` — Metrics traits
- `api/prompt_guard.rs` — Prompt guard types
- `api/zero_shot.rs` — Zero-shot classification types

**Recommendation:** Add tests for `routing.rs` (routing decisions), `http.rs` (CallArgs construction, URL handling), and `prompt_guard.rs` (guard rule evaluation). The error/stats/consts modules are mostly type definitions and don't need extensive testing.

#### Gap 6: brightstaff state — edge cases

The state backends have solid basic coverage (26 tests total), but lack tests for:
- Concurrent access patterns
- State expiration/eviction
- Connection failure recovery (PostgreSQL)
- Large conversation histories

**Recommendation:** Add tokio::test cases for concurrent read/write scenarios in the memory backend and connection pool behavior in the PostgreSQL backend.

---

## 2. Python CLI (`cli/`)

### Current State

| Test File | Tests | Modules Covered |
|-----------|-------|-----------------|
| test_config_generator.py | 11 (5 functions + 6 parametrized) | config_generator, utils |
| test_version_check.py | 18 (4 classes, 18 methods) | versioning |
| test_init.py | 4 | init_cmd |
| test_trace_cmd.py | 2 | trace_cmd (minimal) |
| **Total** | **35 executions** | **5 of 13 modules** |

### Well-Tested Areas

- **versioning.py (18 tests):** Version parsing, comparison, PyPI fetching, network error handling, and environment variable overrides are thoroughly tested across 4 test classes.
- **config_generator.py (11 tests):** Happy-path config validation, schema validation errors (6 parametrized cases), and legacy format conversion are covered.
- **init_cmd.py (4 tests):** Clean init, template init, overwrite protection, and force overwrite are tested.

### Gaps and Recommendations

#### Gap 7: `main.py` — 0 tests (441 LOC)

The CLI entry point defines all Click commands (`up`, `down`, `build`, `logs`, `cli_agent`, `generate_prompt_targets`). None have tests. The `up` command has complex logic for port conflict detection, API key validation, and container orchestration.

**Recommendation:** Add tests using Click's `CliRunner`. Test `planoai up` with mocked Docker calls (validate argument handling, port conflict error messages, API key resolution). Test `planoai down` and `planoai build` for basic argument handling and error paths.

#### Gap 8: `targets.py` — 0 tests (365 LOC)

AST-based Python code parser that extracts prompt targets from Flask/FastAPI routes and Pydantic models. This is complex parsing logic prone to edge cases with decorators, type annotations, and docstrings.

**Recommendation:** Create test fixtures with sample Flask/FastAPI app files and verify extracted prompt targets. Test edge cases: nested decorators, complex type hints (Optional, Union, List[dict]), missing docstrings, and unsupported patterns.

#### Gap 9: `core.py` and `docker_cli.py` — 0 tests (377 LOC combined)

Container lifecycle management and Docker subprocess wrappers are untested.

**Recommendation:** Mock `subprocess.run` / `subprocess.Popen` and test the health check retry loop, container state transitions, and error handling. A shared `conftest.py` with Docker mock fixtures would benefit multiple test files.

#### Gap 10: `trace_cmd.py` — 2 tests for 993 LOC

Only gRPC bind error handling is tested. Trace collection, OTEL span processing, and trace analysis logic (the bulk of the module) are untested.

**Recommendation:** Add tests for trace data parsing and the analysis/summarization logic. Mock gRPC server interactions for collection tests.

---

## 3. JavaScript/TypeScript (`apps/`, `packages/`)

### Current State

**Zero test files. No test framework configured.** The codebase has 70+ TypeScript/React source files across two Next.js apps and shared packages. Quality tooling is limited to type checking and Biome linting.

### Recommendations

#### Gap 11: No test infrastructure

**Recommendation:** Set up Vitest in the Turbo workspace. Add `@testing-library/react` for component testing. Priority candidates:
- `apps/www/src/utils/asciiBuilder.ts` (425 lines of pure utility functions — ideal for unit tests)
- `packages/ui/src/` (shared UI components reused across apps)

**Note:** These are marketing websites, not the core proxy. Prioritize this lower than Rust and Python testing.

---

## 4. E2E and Integration Tests (`tests/`)

### Current State

| Suite | Tests | Coverage |
|-------|-------|----------|
| tests/e2e/test_prompt_gateway.py | 12 | Prompt routing, guardrails, cross-provider SDK compatibility *(deprecated path)* |
| tests/e2e/test_model_alias_routing.py | 19 | Model aliases, format translation, streaming, error handling |
| tests/e2e/test_openai_responses_api_client.py | 17 | Responses API across all providers (passthrough, chat completions, Bedrock, Anthropic) |
| tests/e2e/test_openai_responses_api_client_with_state.py | 2 | Multi-turn conversation state (memory backend) |
| tests/archgw/test_prompt_gateway.py | 3 | Prompt gateway with mock HTTP server *(deprecated path)* |
| tests/archgw/test_llm_gateway.py | 1 | LLM gateway with provider hints |
| **Total** | **54** | |

**Additional manual tests:** 3 Hurl files and 6 REST files for exploratory/manual testing.

### Well-Tested Areas

- **Cross-provider format translation:** OpenAI client → Claude model, Anthropic client → OpenAI model, etc. covered via model alias routing tests.
- **OpenAI Responses API:** Comprehensive coverage across all 4 providers in both streaming and non-streaming modes, with and without tools.
- **Prompt gateway routing:** Intent matching, parameter gathering, default targets, and jailbreak detection tested end-to-end.
- **Error handling basics:** 400 errors with invalid aliases, nonexistent aliases, and unsupported parameters.
- **archgw mock-server tests:** 404 and 500 upstream error handling tested with `pytest_httpserver`.

### Gaps and Recommendations

#### Gap 12: Error and failure scenarios underrepresented

Only a few tests cover error paths. Missing scenarios:
- Upstream provider timeouts
- 5xx errors from LLM providers during streaming
- Malformed/incomplete streaming responses
- Rate limiting behavior end-to-end
- Invalid or expired API keys

**Recommendation:** Add E2E error scenario tests. Use a mock upstream that returns errors/timeouts to test resilience behavior without depending on real provider availability.

#### Gap 13: Bedrock tests unreliable

Several AWS Bedrock tests are marked as skipped/unreliable, reducing coverage of this provider path.

**Recommendation:** Add a mock Bedrock endpoint (or use the archgw mock server pattern from `tests/archgw/`) that returns Bedrock-formatted responses including the binary event stream format. This would make Bedrock tests deterministic.

#### Gap 14: PostgreSQL state storage not E2E tested

State management E2E tests only use the memory backend. PostgreSQL is the production persistence backend.

**Recommendation:** Add a PostgreSQL container to the E2E Docker Compose setup and add tests for multi-turn state persistence, session retrieval, and cleanup.

#### Gap 15: No concurrent request / load tests

There are no tests for behavior under concurrent requests or verifying proper resource cleanup.

**Recommendation:** Add parallel request tests using `pytest-xdist` (already in dependencies) or `asyncio.gather`. Test for race conditions in state writes and resource cleanup.

#### Gap 16: No configuration validation E2E tests

Invalid configs, missing required fields, and misconfigured providers are not tested end-to-end.

**Recommendation:** Add tests that pass intentionally invalid configs to `planoai up` and verify the error messages and exit behavior.

---

## Priority Summary

| Priority | Area | Gap | Recommendation |
|----------|------|-----|----------------|
| **P0** | Rust: llm_gateway | 0 tests, 1,399 LOC | Extract logic from WASM, add unit tests (#1) |
| **P0** | Rust: handler endpoints | llm.rs, agent_chat_completions.rs untested | Add handler-level tests with mockito (#2) |
| **P1** | Rust: streaming transforms | to_openai_streaming, to_anthropic_streaming, bedrock binary | Add streaming transform unit tests (#3) |
| **P1** | Rust: common utilities | routing.rs, http.rs, prompt_guard.rs | Add tests for routing decisions and HTTP utils (#4) |
| **P1** | Python: main.py | 0 tests, 441 LOC | Test CLI commands with CliRunner (#6) |
| **P1** | Python: targets.py | 0 tests, 365 LOC | Test AST parsing with sample app fixtures (#7) |
| **P1** | E2E: error scenarios | Few error path tests | Add timeout/5xx/rate-limit E2E tests (#11) |
| **P2** | Rust: state edge cases | No concurrent/expiration tests | Add async edge case tests (#5) |
| **P2** | Python: core.py/docker_cli.py | 0 tests, 377 LOC | Mock subprocess, test lifecycle (#8) |
| **P2** | Python: trace_cmd.py | 2 tests for 993 LOC | Test trace processing logic (#9) |
| **P2** | E2E: Bedrock | Tests skipped as unreliable | Use mock Bedrock endpoint (#12) |
| **P2** | E2E: PostgreSQL state | Only memory backend tested | Add PG to Docker Compose (#13) |
| **P3** | JS/TS | 0 tests, no framework | Set up Vitest, test asciiBuilder.ts (#10) |
| **P3** | E2E: concurrency | No parallel request tests | Add concurrent request tests (#14) |
| **P3** | E2E: config validation | No invalid config tests | Test error handling for bad configs (#15) |
| ~~skip~~ | ~~Rust: prompt_gateway~~ | ~~4 tests, 1,717 LOC~~ | ~~Deprecated — do not invest in new tests~~ |