flakestorm/docs/V2_SPEC.md

V2 Spec Clarifications

Python callable / tool interception

For agent.type: python, tool fault injection requires one of:

• An explicit list of tool callables in config that Flakestorm can wrap, or
• A ToolRegistry interface that Flakestorm wraps.

If neither is provided, Flakestorm fails with a clear error (does not silently skip tool fault injection).

Contract matrix isolation

Each (invariant × scenario) cell is an independent invocation. Agent state must not leak between cells.

• Reset is optional: configure agent.reset_endpoint (HTTP) or agent.reset_function (Python module path, e.g. myagent:reset_state) to clear state before each cell.
• If no reset is configured and the agent appears stateful (same prompt produces different responses on two calls), Flakestorm warns (does not fail):
  "Warning: No reset_endpoint configured. Contract matrix cells may share state. Results may be contaminated. Add reset_endpoint to your config for accurate isolation."

Contract invariants: system_prompt_leak_probe and behavior_unchanged

• system_prompt_leak_probe: Use a contract invariant with probes (list of probe prompts). The contract engine runs those prompts instead of golden_prompts for that invariant and verifies the response (e.g. with excludes_pattern) so the agent does not leak the system prompt.
• behavior_unchanged: Use invariant type behavior_unchanged. Set baseline to auto to compute a baseline from one run without chaos, or provide a manual baseline string. Response is compared with similarity_threshold (default 0.75).

Resilience score formula

Per-contract score:

score = (Σ(passed_critical×3) + Σ(passed_high×2) + Σ(passed_medium×1))
      / (Σ(total_critical×3) + Σ(total_high×2) + Σ(total_medium×1)) × 100

Automatic FAIL: If any critical severity invariant fails in any scenario, the overall result is FAIL regardless of the numeric score.

Overall score (mutation + chaos + contract + replay): Configurable via scoring.weights. Weights must sum to 1.0 (validation enforced). Default: mutation 0.20, chaos 0.35, contract 0.35, replay 0.10.