2026-03-06 23:33:21 +08:00
# V2 Research Assistant — Flakestorm v2 Example
A **working** HTTP agent and v2.0 config that demonstrates all three V2 pillars: **Environment Chaos** , **Behavioral Contracts** , and **Replay-Based Regression** .
## Prerequisites
- Python 3.10+
2026-03-07 00:31:26 +08:00
- **Ollama** running with a model (e.g. `ollama pull gemma3:1b` then `ollama run gemma3:1b` ). The agent calls Ollama to generate real LLM responses; Flakestorm uses the same Ollama for mutation generation.
- Dependencies: `pip install -r requirements.txt` (fastapi, uvicorn, pydantic, httpx)
2026-03-06 23:33:21 +08:00
## 1. Start the agent
From the project root or this directory:
```bash
cd examples/v2_research_agent
uvicorn agent:app --host 0.0.0.0 --port 8790
```
Or: `python agent.py` (uses port 8790 by default).
Verify: `curl -X POST http://localhost:8790/invoke -H "Content-Type: application/json" -d "{\"input\": \"Hello\"}"`
## 2. Run Flakestorm v2 commands
From the **project root** (so `flakestorm` and config paths resolve):
```bash
# Mutation testing only (v1 style)
flakestorm run -c examples/v2_research_agent/flakestorm.yaml
# With chaos (tool/LLM faults)
flakestorm run -c examples/v2_research_agent/flakestorm.yaml --chaos
# Chaos only (no mutations, golden prompts under chaos)
flakestorm run -c examples/v2_research_agent/flakestorm.yaml --chaos-only
# Built-in chaos profile
flakestorm run -c examples/v2_research_agent/flakestorm.yaml --chaos-profile api_outage
# Behavioral contract ×
flakestorm contract run -c examples/v2_research_agent/flakestorm.yaml
# Contract score only (CI gate)
flakestorm contract score -c examples/v2_research_agent/flakestorm.yaml
# Replay regression (one session)
flakestorm replay run examples/v2_research_agent/replays/incident_001.yaml -c examples/v2_research_agent/flakestorm.yaml
# Export failures from a report as replay files
flakestorm replay export --from-report reports/report.json -o examples/v2_research_agent/replays/
# Full CI run (mutation + contract + chaos + replay, overall weighted score)
flakestorm ci -c examples/v2_research_agent/flakestorm.yaml --min-score 0.5
2026-03-12 20:05:51 +08:00
# CI with reports: summary + detailed phase reports (mutation, contract, chaos, replay)
flakestorm ci -c examples/v2_research_agent/flakestorm.yaml -o ./reports --min-score 0.5
2026-03-06 23:33:21 +08:00
```
## 3. What this example demonstrates
| Feature | Config / usage |
|--------|-----------------|
| **Chaos** | `chaos.tool_faults` (503 with probability), `chaos.llm_faults` (truncated); `--chaos` , `--chaos-profile` |
| **Contract** | `contract` with invariants (always-cite-source, completes, max-latency) and `chaos_matrix` (no-chaos, api-outage) |
| **Replay** | `replays.sessions` with `file: replays/incident_001.yaml` ; contract resolved by name "Research Agent Contract" |
| **Scoring** | `scoring` weights (mutation 20%, chaos 35%, contract 35%, replay 10%); used in `flakestorm ci` |
| **Reset** | `agent.reset_endpoint: http://localhost:8790/reset` for contract matrix isolation |
2026-03-12 20:05:51 +08:00
| **Reproducibility** | Set `advanced.seed` (e.g. `42` ) for deterministic chaos and mutation generation; same config → same scores. |
2026-03-06 23:33:21 +08:00
## 4. Config layout (v2.0)
- `version: "2.0"`
- `agent` + `reset_endpoint`
- `chaos` (tool_faults, llm_faults)
- `contract` (invariants, chaos_matrix)
- `replays.sessions` (file reference)
- `scoring` (weights)
2026-03-07 00:31:26 +08:00
The agent calls **Ollama** (same model as in `flakestorm.yaml` : `gemma3:1b` by default). Set `OLLAMA_BASE_URL` or `OLLAMA_MODEL` if your Ollama runs elsewhere or uses a different model. The agent is stateless except for a call counter; `/reset` clears it so contract cells stay isolated.
