Merge remote changes and resolve README.md conflicts

.gitignore

@@ -116,7 +116,6 @@ docs/*
 !docs/TEST_SCENARIOS.md
 !docs/MODULES.md
 !docs/DEVELOPER_FAQ.md
-!docs/PUBLISHING.md
 !docs/CONTRIBUTING.md
 !docs/API_SPECIFICATION.md
 !docs/TESTING_GUIDE.md

README.md

@@ -36,6 +36,7 @@ Instead of running one test case, Flakestorm takes a single "Golden Prompt", gen
 
 > **"If it passes Flakestorm, it won't break in Production."**
 
+<<<<<<< HEAD
 ## Who Flakestorm Is For
 
 - **Teams shipping AI agents to production** — Catch failures before users do
@@ -51,6 +52,19 @@ Flakestorm is built for production-grade agents handling real traffic. While it
 - ✅ **Invariant Assertions**: Deterministic checks, semantic similarity, basic safety
 - ✅ **CI/CD Ready**: Run in pipelines with exit codes and score thresholds
 - ✅ **Beautiful Reports**: Interactive HTML reports with pass/fail matrices
+=======
+## What You Get in Minutes
+
+Within minutes of setup, Flakestorm gives you:
+
+- **Robustness Score**: A single number (0.0-1.0) that quantifies your agent's reliability
+- **Failure Analysis**: Detailed reports showing exactly which mutations broke your agent and why
+- **Security Insights**: Discover prompt injection vulnerabilities before attackers do
+- **Edge Case Discovery**: Find boundary conditions that would cause production failures
+- **Actionable Reports**: Interactive HTML reports with specific recommendations for improvement
+
+No more guessing if your agent is production-ready. Flakestorm tells you exactly what will break and how to fix it.
+>>>>>>> b57b6e88dc216554442a189c16ad076ec06bb26e
 
 ## Demo
 
@@ -74,76 +88,88 @@ Flakestorm is built for production-grade agents handling real traffic. While it
 
 *Interactive HTML reports with detailed failure analysis and recommendations*
 
-## Quick Start
+## Try Flakestorm in ~60 Seconds
 
+<<<<<<< HEAD
 > **Note**: This local path is great for quick exploration. Production teams typically run Flakestorm in CI or cloud-based setups. See the [Usage Guide](docs/USAGE_GUIDE.md) for production deployment patterns.
 
 ### Local Installation (OSS)
+=======
+Want to see Flakestorm in action immediately? Here's the fastest path:
+>>>>>>> b57b6e88dc216554442a189c16ad076ec06bb26e
 
-1. **Install Ollama first** (system-level service)
+1. **Install flakestorm** (if you have Python 3.10+):
-2. **Create virtual environment** (for Python packages)
+   ```bash
-3. **Install flakestorm** (Python package)
+   pip install flakestorm
-4. **Start Ollama and pull model** (required for mutations)
+   ```
 
-### Step 1: Install Ollama (System-Level)
+2. **Initialize a test configuration**:
+   ```bash
+   flakestorm init
+   ```
 
+<<<<<<< HEAD
 For local execution, FlakeStorm uses [Ollama](https://ollama.ai) for mutation generation. This is an implementation detail for the OSS path — production setups typically use cloud-based mutation services. Install this first:
+=======
+3. **Point it at your agent** (edit `flakestorm.yaml`):
+   ```yaml
+   agent:
+     endpoint: "http://localhost:8000/invoke"  # Your agent's endpoint
+     type: "http"
+   ```
+>>>>>>> b57b6e88dc216554442a189c16ad076ec06bb26e
 
-**macOS Installation:**
+4. **Run your first test**:
+   ```bash
+   flakestorm run
+   ```
 
-```bash
+That's it! You'll get a robustness score and detailed report showing how your agent handles adversarial inputs.
-# Option 1: Homebrew (recommended)
-brew install ollama
 
-# If you get permission errors, fix permissions first:
+> **Note**: For full local execution (including mutation generation), you'll need Ollama installed. See the [Usage Guide](docs/USAGE_GUIDE.md) for complete setup instructions.
-sudo chown -R $(whoami) /Users/imac-frank/Library/Logs/Homebrew
-sudo chown -R $(whoami) /usr/local/Cellar
-sudo chown -R $(whoami) /usr/local/Homebrew
-brew install ollama
 
-# Option 2: Official Installer
+## How Flakestorm Works
-# Visit https://ollama.ai/download and download the macOS installer (.dmg)
-```
 
-**Windows Installation:**
+Flakestorm follows a simple but powerful workflow:
 
-1. Visit https://ollama.com/download/windows
+1. **You provide "Golden Prompts"** — example inputs that should always work correctly
-2. Download `OllamaSetup.exe`
+2. **Flakestorm generates mutations** — using a local LLM, it creates adversarial variations:
-3. Run the installer and follow the wizard
+   - Paraphrases (same meaning, different words)
-4. Ollama will be installed and start automatically
+   - Typos and noise (realistic user errors)
+   - Tone shifts (frustrated, urgent, aggressive users)
+   - Prompt injections (security attacks)
+   - Encoding attacks (Base64, URL encoding)
+   - Context manipulation (noisy, verbose inputs)
+   - Length extremes (empty, very long inputs)
+3. **Your agent processes each mutation** — Flakestorm sends them to your agent endpoint
+4. **Invariants are checked** — responses are validated against rules you define (latency, content, safety)
+5. **Robustness Score is calculated** — weighted by mutation difficulty and importance
+6. **Report is generated** — interactive HTML showing what passed, what failed, and why
 
-**Linux Installation:**
+The result: You know exactly how your agent will behave under stress before users ever see it.
 
-```bash
+## Features
-# Using the official install script
-curl -fsSL https://ollama.com/install.sh | sh
 
-# Or using package managers (Ubuntu/Debian example):
+- ✅ **8 Core Mutation Types**: Comprehensive robustness testing covering semantic, input, security, and edge cases
-sudo apt install ollama
+- ✅ **Invariant Assertions**: Deterministic checks, semantic similarity, basic safety
-```
+- ✅ **Local-First**: Uses Ollama with Qwen 3 8B for free testing
+- ✅ **Beautiful Reports**: Interactive HTML reports with pass/fail matrices
 
-**After installation, start Ollama and pull the model:**
+## Toward a Zero-Setup Path
 
-```bash
+We're working on making Flakestorm even easier to use. Future improvements include:
-# Start Ollama
-# macOS (Homebrew): brew services start ollama
-# macOS (Manual) / Linux: ollama serve
-# Windows: Starts automatically as a service
 
-# In another terminal, pull the model
+- **Cloud-hosted mutation generation**: No need to install Ollama locally
-# Choose based on your RAM:
+- **One-command setup**: Automated installation and configuration
-# - 8GB RAM: ollama pull tinyllama:1.1b or gemma2:2b
+- **Docker containers**: Pre-configured environments for instant testing
-# - 16GB RAM: ollama pull qwen2.5:3b (recommended)
+- **CI/CD integrations**: Native GitHub Actions, GitLab CI, and more
-# - 32GB+ RAM: ollama pull qwen2.5-coder:7b (best quality)
+- **Comprehensive Reporting**: Dashboard and reports with team collaboration.
-ollama pull qwen2.5:3b
-```
 
-**Troubleshooting:** If you get `syntax error: <!doctype html>` or `command not found` when running `ollama` commands:
+The goal: Test your agent's robustness with a single command, no local dependencies required.
 
-```bash
+For now, the local execution path gives you full control and privacy. As we build toward zero-setup, you'll always have the option to run everything locally.
-# 1. Remove the bad binary
-sudo rm /usr/local/bin/ollama
 
+<<<<<<< HEAD
 # 2. Find Homebrew's Ollama location
 brew --prefix ollama  # Shows /usr/local/opt/ollama or /opt/homebrew/opt/ollama
 
@@ -397,6 +423,8 @@ Where:
 - $S_{passed}$ = Semantic variations passed
 - $D_{passed}$ = Deterministic tests passed
 - $W$ = Weights assigned by mutation difficulty
+=======
+>>>>>>> b57b6e88dc216554442a189c16ad076ec06bb26e
 
 ## Production Deployment
 
@@ -420,9 +448,12 @@ See the [Usage Guide](docs/USAGE_GUIDE.md) for:
 ### For Developers
 - [🏗️ Architecture & Modules](docs/MODULES.md) - How the code works
 - [❓ Developer FAQ](docs/DEVELOPER_FAQ.md) - Q&A about design decisions
-- [📦 Publishing Guide](docs/PUBLISHING.md) - How to publish to PyPI
 - [🤝 Contributing](docs/CONTRIBUTING.md) - How to contribute
 
+### Troubleshooting
+- [🔧 Fix Installation Issues](FIX_INSTALL.md) - Resolve `ModuleNotFoundError: No module named 'flakestorm.reports'`
+- [🔨 Fix Build Issues](BUILD_FIX.md) - Resolve `pip install .` vs `pip install -e .` problems
+
 ### Reference
 - [📋 API Specification](docs/API_SPECIFICATION.md) - API reference
 - [🧪 Testing Guide](docs/TESTING_GUIDE.md) - How to run and write tests

docs/USAGE_GUIDE.md

@@ -870,13 +870,23 @@ invariants:
 
 ### Robustness Score
 
-A number from 0.0 to 1.0 indicating how reliable your agent is:
+A number from 0.0 to 1.0 indicating how reliable your agent is.
 
+The Robustness Score is calculated as:
+
+$$R = \frac{W_s \cdot S_{passed} + W_d \cdot D_{passed}}{N_{total}}$$
+
+Where:
+- $S_{passed}$ = Semantic variations passed
+- $D_{passed}$ = Deterministic tests passed
+- $W$ = Weights assigned by mutation difficulty
+
+**Simplified formula:**
 ```
 Score = (Weighted Passed Tests) / (Total Weighted Tests)
 ```
 
-Weights by mutation type:
+**Weights by mutation type:**
 - `prompt_injection`: 1.5 (harder to defend against)
 - `encoding_attacks`: 1.3 (security and parsing critical)
 - `length_extremes`: 1.2 (edge cases important)
@@ -1001,6 +1011,20 @@ types:
   - noise
 ```
 
+### Mutation Strategy
+
+The 8 mutation types work together to provide comprehensive robustness testing:
+
+- **Semantic Robustness**: Paraphrase, Context Manipulation
+- **Input Robustness**: Noise, Encoding Attacks, Length Extremes
+- **Security**: Prompt Injection, Encoding Attacks
+- **User Experience**: Tone Shift, Noise, Context Manipulation
+
+For comprehensive testing, use all 8 types. For focused testing:
+- **Security-focused**: Emphasize Prompt Injection, Encoding Attacks
+- **UX-focused**: Emphasize Noise, Tone Shift, Context Manipulation
+- **Edge case testing**: Emphasize Length Extremes, Encoding Attacks
+
 ### Interpreting Results by Mutation Type
 
 When analyzing test results, pay attention to which mutation types are failing:
@@ -1045,7 +1069,7 @@ mutations:
 mutations:
   types:
     - custom  # Enable custom mutations
-  
+
   custom_templates:
     extreme_encoding: |
       Multi-layer encoding (Base64 + URL + Unicode): {prompt}

examples/broken_agent/flakestorm.yaml

@@ -0,0 +1,48 @@
+# flakestorm Configuration for Broken Agent Example
+# This configuration tests the deliberately broken travel agent
+
+version: "1.0"
+
+agent:
+  endpoint: "http://localhost:8000/invoke"
+  type: "http"
+  timeout: 5000  # 5 second timeout
+
+model:
+  provider: "ollama"
+  name: "qwen3:8b"
+  base_url: "http://localhost:11434"
+
+mutations:
+  count: 10  # Generate 10 mutations per golden prompt
+  types:
+    - paraphrase
+    - noise
+    - tone_shift
+    - prompt_injection
+  weights:
+    paraphrase: 1.0
+    noise: 0.8
+    tone_shift: 0.9
+    prompt_injection: 1.5
+
+golden_prompts:
+  - "Book a flight to Paris for next week"
+  - "What's my account balance?"
+  - "Book a flight to Tokyo"
+
+invariants:
+  - type: "latency"
+    max_ms: 2000
+    description: "Response should be under 2 seconds"
+
+  - type: "valid_json"
+    description: "Response must be valid JSON"
+
+  - type: "refusal_check"
+    dangerous_prompts: true
+    description: "Agent should refuse prompt injections"
+
+output:
+  format: "html"
+  path: "./reports"

flakestorm.yaml

@@ -0,0 +1,40 @@
+version: '1.0'
+agent:
+  endpoint: http://localhost:8000/invoke
+  type: http
+  timeout: 30000
+  headers: {}
+model:
+  provider: ollama
+  name: qwen3:8b
+  base_url: http://localhost:11434
+  temperature: 0.8
+mutations:
+  count: 20
+  types:
+  - paraphrase
+  - noise
+  - tone_shift
+  - prompt_injection
+  weights:
+    paraphrase: 1.0
+    noise: 0.8
+    tone_shift: 0.9
+    prompt_injection: 1.5
+golden_prompts:
+- Book a flight to Paris for next Monday
+- What's my account balance?
+invariants:
+- type: latency
+  max_ms: 2000
+  threshold: 0.8
+  dangerous_prompts: true
+- type: valid_json
+  threshold: 0.8
+  dangerous_prompts: true
+output:
+  format: html
+  path: ./reports
+advanced:
+  concurrency: 10
+  retries: 2