[pitboss/grind] deferred session-0002 (20260521T143544Z-f898)

This commit is contained in:
pitboss 2026-05-21 11:22:13 -05:00
parent be4021d8c0
commit b3766311fb
20 changed files with 388 additions and 664 deletions

View file

@ -46,7 +46,7 @@ A focused release on three fronts: an attack-surface map and chain composer that
- **New `Cap` corpora.** Vulnerable + patched fixtures landed for the seven new cap classes (LDAP injection, XPath injection, header injection, open redirect, SSTI, XXE, prototype pollution) plus deserialization, crypto, JSON parsing, unauthorized-id, and data exfiltration. Every cap now carries at least one positive / negative / adversarial / unsupported fixture quad per supported language. - **New `Cap` corpora.** Vulnerable + patched fixtures landed for the seven new cap classes (LDAP injection, XPath injection, header injection, open redirect, SSTI, XXE, prototype pollution) plus deserialization, crypto, JSON parsing, unauthorized-id, and data exfiltration. Every cap now carries at least one positive / negative / adversarial / unsupported fixture quad per supported language.
- **OWASP Benchmark v1.2 importer.** `tests/eval_corpus/owasp_gt_convert.py` converts the OWASP Java Benchmark expected-results manifest into Nyx ground truth and lands a 16k-line `owasp_benchmark_v1.2.json` for evaluation. - **OWASP Benchmark v1.2 importer.** `tests/eval_corpus/owasp_gt_convert.py` converts the OWASP Java Benchmark expected-results manifest into Nyx ground truth and lands a 16k-line `owasp_benchmark_v1.2.json` for evaluation.
- **NIST SARD importer.** `tests/eval_corpus/sard_gt_convert.py` converts SARD test cases into the same format so cross-dataset recall numbers stay comparable. - **NIST SARD importer.** `tests/eval_corpus/sard_gt_convert.py` converts SARD test cases into the same format so cross-dataset recall numbers stay comparable.
- **`scripts/m7_ship_gate.sh`** runs five gates against `tests/eval_corpus/budget.toml`: Unsupported under 20% per `(cap, lang)` cell, False-Confirmed under 2% per cap, repro stability at or above 95%, wall-clock no more than 2× static-only, sandbox-escape suite green. `tests/eval_corpus/run_full.sh` is the canonical orchestrator and writes `tests/eval_corpus/results.json` for the gate plus the published metrics table in `docs/dynamic.md`. - **Evaluation corpus tooling.** `tests/eval_corpus/run_full.sh` runs the Nyx benchmark, OWASP Benchmark, and NIST SARD evaluation sets and writes `tests/eval_corpus/results.json`. `tests/eval_corpus/report.py` and `tabulate.py` produce the per-cap and per-language summary used to track coverage and accuracy.
### Engine ### Engine

View file

@ -1,125 +1,116 @@
# Dynamic verification # Dynamic verification
Nyx verifies every `Confidence >= Medium` finding by default: it builds Nyx re-runs findings in generated harnesses when verification is enabled. By
a minimal harness, runs your code's entry point against a curated payload corpus default, `nyx scan` verifies each `Confidence >= Medium` finding, tries
inside a sandbox, and records the verdict in each finding's evidence block. payloads in a sandbox, and writes the result to `evidence.dynamic_verdict`.
## Headline metrics Dynamic verification is a second signal, not a replacement for review. A
confirmed verdict means Nyx triggered the sink in its harness. `NotConfirmed`
means the harness ran but no payload fired.
The dynamic-verification overhaul ships with four published acceptance targets, ## Running it
gated end-to-end by `scripts/m7_ship_gate.sh` (Phase 31) against the eval
corpus (OWASP Benchmark v1.2 + NIST SARD subset + the in-house curated set
from `tests/benchmark/corpus`):
| Metric | Target | Gate | Source | ```bash
| --- | --- | --- | --- | nyx scan # verifies Medium and High confidence findings
| Unsupported% per `(cap, lang)` cell | < 20% | M7 Gate 1 | `tests/eval_corpus/budget.toml` `[default].unsupported_rate` | nyx scan --no-verify # static analysis only
| False-Confirmed% per cap | < 2% | M7 Gate 2 | `~/.cache/nyx/dynamic/events.jsonl` (`kind: feedback`, `wrong: true`) | nyx scan --verify # explicit form of the default behavior
| Repro stability | ≥ 95% | M7 Gate 5 | `~/.cache/nyx/dynamic/repro/*/reproduce.sh` exit 0 |
| Wall-clock cost | ≤ 2× static-only | M7 Gate 3 | `benches/fixtures/` (default vs `--no-verify`) |
The corresponding orchestrator is `tests/eval_corpus/run_full.sh`; it bundles
the three corpus sets, writes a canonical `tests/eval_corpus/results.json`,
and propagates the per-cell budget through `tabulate.py` and `report.py`.
A non-zero exit from `m7_ship_gate.sh` is a hard merge blocker for the
default-on flip. Failures map back to the engine follow-ups recorded in
`.pitboss/play/deferred.md` (per-language probe-shim splicing, composite
chain reverifier wiring, telemetry-stability stamping, et al.).
## Default-on semantics
```
nyx scan # verifies Medium+ findings (default)
nyx scan --no-verify # static analysis only, no harness execution
nyx scan --verify # same as default; explicit for clarity in scripts
``` ```
`--no-verify` is the escape hatch. It overrides the config default for a single Use `--no-verify` for fast local checks or editor workflows. Keep verification
run without changing `nyx.toml`. on for CI when scan time allows it.
### What "verified" means To verify low-confidence findings too:
A finding with `dynamic_verdict.status: Confirmed` was successfully triggered ```bash
by at least one payload in nyx's corpus. The corpus covers common patterns for
each vulnerability class (SQL injection, XSS, command injection, SSRF, etc.) per
language.
A finding with `dynamic_verdict.status: NotConfirmed` was attempted but no
payload fired. This is not a false-positive signal. It means the corpus did not
have a payload that matched the specific sink variant, or the execution path was
not reachable in the test harness.
A finding with `dynamic_verdict.status: Unsupported` could not be attempted.
Common reasons: confidence below threshold, no flow steps, language or sink type
not yet supported by the harness layer.
### Confidence gate
Only `Confidence >= Medium` findings are verified by default (§5.1). To also
verify low-confidence findings (for corpus building or backfill), pass
`--verify-all-confidence`:
```
nyx scan --verify-all-confidence nyx scan --verify-all-confidence
``` ```
This is not recommended for production scans because low-confidence findings have Use it when tuning payloads or investigating coverage. It is slower and noisier
a higher false-positive rate and the harness may produce unreliable verdicts. than the default.
## nyx.toml opt-out ## Verdicts
If you want static-only scans permanently, set `verify = false` in `nyx.toml`: | Status | Meaning |
| --- | --- |
| `Confirmed` | At least one payload reached the expected sink in the harness. |
| `NotConfirmed` | The harness ran, but no payload reached the sink. Treat the original finding as still open until reviewed. |
| `Inconclusive` | Nyx could not finish the check with enough isolation or runtime support. |
| `Unsupported` | Nyx did not try the finding. Common causes are unsupported language, unsupported sink shape, missing flow steps, or confidence below the verification threshold. |
## Configuration
To disable verification for a project, set:
```toml ```toml
[scanner] [scanner]
verify = false verify = false
``` ```
This survives upgrades. The M7 default flip only changes the inherited default This makes scans static-only unless the command line overrides it.
for projects that have not explicitly set the field.
The related scanner settings are:
| Setting | Default | Meaning |
| --- | --- | --- |
| `verify` | `true` | Run dynamic verification after static analysis. |
| `verify_all_confidence` | `false` | Include findings below `Confidence::Medium`. |
| `verify_backend` | `"auto"` | Use Docker when available, otherwise use the process backend. |
| `harden_profile` | `"standard"` | Hardening profile for the process backend. |
See [Configuration](configuration.md) for the full config table.
## Sandbox backends ## Sandbox backends
nyx uses docker when available, then falls back to an in-process runner: ```bash
nyx scan --backend docker # require Docker
``` nyx scan --backend process # run directly on the host with weaker isolation
nyx scan --backend docker # require docker; fail if unavailable
nyx scan --backend process # in-process runner (no container; less isolation)
nyx scan --unsafe-sandbox # alias for --backend process nyx scan --unsafe-sandbox # alias for --backend process
``` ```
The docker backend mounts only the entry file's directory and blocks all Docker is the preferred backend. It mounts only the entry file's directory and
outbound network by default. When out-of-band detection is enabled (`oob_listener` blocks outbound network by default. If out-of-band detection is enabled with
in config), the container gets `--network bridge` with a host-gateway route. `oob_listener`, Docker uses bridge networking with a host-gateway route so the
harness can reach the listener.
The process backend is useful for development and machines without Docker. It
does not provide the same isolation.
## Repro artifacts ## Repro artifacts
When a finding is `Confirmed`, nyx writes a repro artifact to Confirmed findings write a repro bundle under:
`~/.cache/nyx/repro/<stable_hash>/`. The artifact contains the harness spec and
the triggering payload. You can regenerate the verdict with:
``` ```text
nyx scan --verify <path> # re-scans and re-verifies ~/.cache/nyx/dynamic/repro/<spec_hash>/
``` ```
See `docs/output.md` for the `dynamic_verdict` field schema. The bundle contains the harness spec, payload, expected output, trace, and
`reproduce.sh`.
## Wall-clock cost ```bash
cd ~/.cache/nyx/dynamic/repro/<spec_hash>
./reproduce.sh
./reproduce.sh --docker
```
Verification adds harness build + sandbox startup time per finding. On typical Use the Docker form when the bundle records a pinned container image or when
codebases with 1050 Medium+ findings, end-to-end overhead is 25× static-only. host toolchains differ from the original run.
If scan time is unacceptable for a given workflow (e.g. IDE integration, quick ## Runtime cost
pre-commit check), use `--no-verify` for that workflow and rely on the full scan
in CI.
## Event schema Verification adds harness build time and sandbox startup time for each verified
finding. For quick local checks, `--no-verify` is usually the right choice. For
CI or scheduled scans, keep verification enabled so confirmed findings rank
higher and not-confirmed findings carry the extra context.
The dynamic layer writes one JSON record per verdict to ## Event log
`~/.cache/nyx/dynamic/events.jsonl`. Every record begins with a fixed envelope
so older readers fail loudly instead of silently mixing incompatible shapes: Nyx writes verdict events to:
```text
~/.cache/nyx/dynamic/events.jsonl
```
Each line is a JSON object with a versioned envelope:
```json ```json
{ {
@ -140,74 +131,54 @@ so older readers fail loudly instead of silently mixing incompatible shapes:
} }
``` ```
| Field | Type | Meaning | | Field | Meaning |
| --- | --- | --- | | --- | --- |
| `schema_version` | integer | Bumped on any breaking change. Readers reject mismatches. | | `schema_version` | Event schema version. Readers reject mismatches. |
| `nyx_version` | string | `CARGO_PKG_VERSION` of the writing binary. | | `nyx_version` | Version of the Nyx binary that wrote the event. |
| `corpus_version` | string | Payload-corpus version the verdict was scored against. | | `corpus_version` | Payload corpus version used for the verdict. |
| `kind` | string | `"verdict"` (per-finding) or `"rank_delta"` (rank-score shift). | | `kind` | `verdict`, `rank_delta`, or `feedback`. |
| `ts` | RFC-3339 string | Wall-clock at write time. | | `ts` | Write time in RFC 3339 format. |
| `finding_id` | string | Stable finding identifier. | | `finding_id` | Stable finding identifier. |
| `spec_hash` | string | Hash of the `HarnessSpec` that drove the run. | | `spec_hash` | Hash of the harness spec. |
| `lang` | string | Language slug; `"unknown"` when spec derivation failed. | | `lang` | Language slug, or `unknown` when spec derivation failed. |
| `cap` | string | Sink capability (e.g. `SQL_QUERY`, `CODE_EXEC`). | | `cap` | Sink capability, such as `SQL_QUERY` or `CODE_EXEC`. |
| `status` | string | `Confirmed`, `NotConfirmed`, `Inconclusive`, or `Unsupported`. | | `status` | `Confirmed`, `NotConfirmed`, `Inconclusive`, or `Unsupported`. |
| `inconclusive_reason` | string | Present iff `status == Inconclusive`. | | `inconclusive_reason` | Present when `status` is `Inconclusive`. |
A `rank_delta` record carries the envelope plus `finding_id`, `status`, and a If the schema changes, move or delete the old `events.jsonl` before reading it
signed `delta` applied to the rank score. with the new binary. Programmatic readers should use
`crate::dynamic::telemetry::read_events(path)`.
### Schema-version mismatch ## Sampling
`scripts/m7_ship_gate.sh` Gate 2 walks every line of the log, requires `[telemetry]` in `nyx.toml` controls event retention:
`schema_version == EXPECTED_SCHEMA_VERSION`, and exits 3 if any record fails
the check. Programmatic readers use
`crate::dynamic::telemetry::read_events(path)`, which surfaces the same
condition as `TelemetryReadError::SchemaMismatch { expected, found, .. }`.
When schema bumps land, the canonical migration is to roll the log over (move
or delete `events.jsonl`) so new and old records never coexist in a file. The
gate refuses to skip silently on mismatch.
### Sampling
`[telemetry]` in `nyx.toml` controls the on-disk sampling policy:
```toml ```toml
[telemetry] [telemetry]
keep_all_confirmed = true # default: retain every Confirmed verdict keep_all_confirmed = true
keep_all_inconclusive = true # default: retain every Inconclusive verdict keep_all_inconclusive = true
sample_rate_other = 1.0 # 0.01.0 for NotConfirmed / Unsupported sample_rate_other = 1.0
``` ```
`sample_rate_other < 1.0` downsamples NotConfirmed and Unsupported verdicts `sample_rate_other` accepts `0.0` to `1.0` and applies to `NotConfirmed` and
deterministically. The decision is seeded by the finding's `spec_hash`, so a `Unsupported` verdicts. The decision is deterministic for a given `spec_hash`.
given finding makes the same keep-or-drop call across reruns. Confirmed and Confirmed, Inconclusive, and rank-delta events are always kept by default.
Inconclusive verdicts ignore the rate and are always retained (they gate the
false-Confirmed budget and drive the spec-derivation roadmap).
Rank-delta records (emitted by `emit_rank_delta` when a verdict shifts a Set `NYX_NO_TELEMETRY=1` to disable event writes.
finding's position in the ranked output) are also retained unconditionally and
do **not** consult `sample_rate_other`. They are calibration-critical and small
in volume, so the carve-out is intentional; setting `sample_rate_other = 0.0`
to throttle log growth will still produce rank-delta lines.
`NYX_NO_TELEMETRY=1` disables every write regardless of the policy. ## Feedback
## Opting in to feedback To record a bad verdict:
False positives (nyx says `Confirmed` but you disagree) can be recorded: ```bash
```
nyx verify-feedback <finding_id> --wrong "reason" nyx verify-feedback <finding_id> --wrong "reason"
``` ```
This writes to the local telemetry log (`~/.cache/nyx/dynamic/events.jsonl`) Feedback is written to the local event log. Nyx does not upload it.
and contributes to precision monitoring. Feedback is never uploaded automatically.
## nyx serve integration ## Browser UI
The browser UI shows `dynamic_verdict` in each finding's detail panel and `nyx serve` shows dynamic verdicts on finding detail pages, uses them in
uses the verdict in ranking (Confirmed findings surface first). The scan compare ranking, and can compare verdict changes between saved scans.
page has a **Verdict Diff** tab that shows which findings changed verification
status between two scans. See [Output formats](output.md) for the `dynamic_verdict` schema.

View file

@ -12,9 +12,8 @@ nyx serve --no-browser # don't auto-open
Persistent settings live under `[server]` in `nyx.conf` / `nyx.local`. Persistent settings live under `[server]` in `nyx.conf` / `nyx.local`.
Starting a scan from the UI runs dynamic verification on `Confidence >= Medium` Starting a scan from the UI runs dynamic verification on `Confidence >= Medium`
findings by default (M7). Check "Skip dynamic verification" in the scan modal findings by default. Check "Skip dynamic verification" in the scan modal to get
to get a fast static-only result. See [Dynamic verification](dynamic.md) for a fast static-only result. See [Dynamic verification](dynamic.md) for details.
details.
<p align="center"><img src="assets/screenshots/docs/serve-overview.png" alt="Nyx UI overview: total findings, severity breakdown, language and category distribution, top affected files" width="900"/></p> <p align="center"><img src="assets/screenshots/docs/serve-overview.png" alt="Nyx UI overview: total findings, severity breakdown, language and category distribution, top affected files" width="900"/></p>

View file

@ -11,9 +11,9 @@ export interface StartScanBody {
engine_profile?: EngineProfile; engine_profile?: EngineProfile;
/** /**
* Override dynamic verification for this scan. * Override dynamic verification for this scan.
* true force on. * true - force on.
* false force off (skip verification; M7 default is on). * false - force off.
* absent use server config default (true since M7). * absent - use server config default.
*/ */
verify?: boolean; verify?: boolean;
/** Also verify Confidence < Medium findings. Default false. */ /** Also verify Confidence < Medium findings. Default false. */

View file

@ -1,401 +0,0 @@
#!/usr/bin/env bash
# M7 pre-flip ship gate.
#
# Runs all five gates required before the default-on merge can land.
# Must pass with exit 0 on the branch being merged.
#
# Usage:
# scripts/m7_ship_gate.sh [--nyx BIN] [--corpus-dir DIR] [--skip GATE,...]
# [--budget FILE] [--diff FILE]
#
# Gates:
# 1. unsupported-rate — per-cell (cap × lang) Unsupported% within budget
# 2. false-confirmed — false-Confirmed rate from telemetry ≤ 2% per cap
# 3. wall-clock — default scan ≤ 2× static-only on bench suite
# 4. sandbox-escape — sandbox escape suite green for all langs
# 5. repro-stability — repro artifact regenerates identical verdict ≥ 95%
#
# Phase 29 (Track I): Gate 1 consumes per-cell budgets from
# `tests/eval_corpus/budget.toml` and, when `--diff PREV.json` is
# supplied, fails on any monotonic-improvement regression vs the
# previous run.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
NYX_BIN="${NYX_BIN:-${REPO_ROOT}/target/release/nyx}"
CORPUS_DIR="${CORPUS_DIR:-${HOME}/.cache/nyx/eval_corpus}"
SKIP_GATES=""
GATE_ERRORS=0
GATE_LOG="${REPO_ROOT}/target/m7_gate.log"
# Phase 29 (Track I): per-cell budgets + monotonic diff.
BUDGET_FILE="${BUDGET_FILE:-${REPO_ROOT}/tests/eval_corpus/budget.toml}"
DIFF_FILE="${DIFF_FILE:-}"
while [[ $# -gt 0 ]]; do
case "$1" in
--nyx) NYX_BIN="$2"; shift 2 ;;
--corpus-dir) CORPUS_DIR="$2"; shift 2 ;;
--skip) SKIP_GATES="$2"; shift 2 ;;
--budget) BUDGET_FILE="$2"; shift 2 ;;
--diff) DIFF_FILE="$2"; shift 2 ;;
*) shift ;;
esac
done
skip() { [[ ",$SKIP_GATES," == *",$1,"* ]]; }
die() { echo "GATE FAIL: $*" | tee -a "$GATE_LOG" >&2; GATE_ERRORS=$((GATE_ERRORS + 1)); }
pass() { echo "GATE PASS: $*" | tee -a "$GATE_LOG"; }
info() { echo "[gate] $*" | tee -a "$GATE_LOG"; }
[[ -x "$NYX_BIN" ]] || { echo "nyx binary not found: $NYX_BIN" >&2; exit 1; }
mkdir -p "$(dirname "$GATE_LOG")"
echo "# M7 ship gate — $(date -u +%Y-%m-%dT%H:%M:%SZ)" > "$GATE_LOG"
info "nyx: $NYX_BIN"
info "corpus: $CORPUS_DIR"
info "budget: $BUDGET_FILE"
info "diff: ${DIFF_FILE:-<none>}"
info ""
# ── Gate 1: Per-cell budget + monotonic-improvement diff ───────────────────
#
# Phase 29 (Track I): the single global Unsupported threshold is replaced
# by per-cell (cap × lang) budgets in tests/eval_corpus/budget.toml.
# `tests/eval_corpus/run.sh` invokes `tabulate.py` per set and `report.py`
# at the end with `--budget` (and `--diff` when DIFF_FILE is set), so
# any per-cell failure (or any regression vs the prior run) propagates
# back as exit 2.
if skip unsupported-rate; then
info "Gate 1 (unsupported-rate): SKIPPED"
else
info "Gate 1: per-cell budget within tolerance + no monotonic regressions..."
EVAL_RESULTS="${REPO_ROOT}/target/eval_results.json"
echo "[]" > "$EVAL_RESULTS"
if [[ ! -f "$BUDGET_FILE" ]]; then
die "Gate 1: budget file not found at $BUDGET_FILE"
else
# Run eval corpus runner (in-house set always present).
set +e
bash "${REPO_ROOT}/tests/eval_corpus/run.sh" \
--nyx "$NYX_BIN" \
--sets inhouse \
--output "$(dirname "$EVAL_RESULTS")" \
--budget "$BUDGET_FILE" \
${DIFF_FILE:+--diff "$DIFF_FILE"} \
>>"$GATE_LOG" 2>>"$GATE_LOG"
RC=$?
set -e
cp "$(dirname "$EVAL_RESULTS")/eval_results.json" "$EVAL_RESULTS" 2>/dev/null || true
if [[ $RC -eq 0 ]]; then
pass "Gate 1: per-cell budget + diff check passed"
elif [[ $RC -eq 2 ]]; then
die "Gate 1: per-cell budget exceeded OR monotonic-improvement regression (see $GATE_LOG)"
elif [[ $RC -eq 3 ]]; then
die "Gate 1: budget/diff configuration is malformed (see $GATE_LOG)"
else
info "Gate 1: eval runner returned $RC (corpus may not be downloaded; treating as SKIP)"
fi
fi
fi
# ── Gate 2: False-Confirmed rate ─────────────────────────────────────────────
#
# Phase 27 (Track H.1): the telemetry log is schema-versioned. Gate 2 reads
# `EXPECTED_SCHEMA_VERSION` against every record's `schema_version` field and
# fails loudly with exit 3 when a mismatch is found — silently treating a
# v0 (pre-Phase-27) log as "no data" would mask incompatible releases mixing
# their records.
EXPECTED_SCHEMA_VERSION=1
if skip false-confirmed; then
info "Gate 2 (false-confirmed): SKIPPED"
else
info "Gate 2: false-Confirmed rate from telemetry ≤ 2% per cap..."
EVENTS="${HOME}/.cache/nyx/dynamic/events.jsonl"
if [[ ! -f "$EVENTS" ]]; then
info "Gate 2: telemetry log not found at $EVENTS; skipping (no data)"
else
set +e
python3 - "$EVENTS" "$EXPECTED_SCHEMA_VERSION" <<'PYEOF'
import json, sys, collections
path = sys.argv[1]
expected_schema = int(sys.argv[2])
cap_counts = collections.defaultdict(lambda: {"confirmed": 0, "wrong": 0})
with open(path) as f:
for line_no, raw in enumerate(f, start=1):
if not raw.strip():
continue
try:
ev = json.loads(raw)
except json.JSONDecodeError as e:
print(f"FAIL malformed JSON at {path} line {line_no}: {e}")
sys.exit(3)
if "schema_version" not in ev:
print(f"FAIL missing schema_version at {path} line {line_no}")
sys.exit(3)
if ev["schema_version"] != expected_schema:
print(
f"FAIL schema mismatch at {path} line {line_no}: "
f"expected {expected_schema}, found {ev['schema_version']}"
)
sys.exit(3)
kind = ev.get("kind", "")
if kind == "feedback" and ev.get("wrong"):
cap = ev.get("cap", "unknown")
cap_counts[cap]["wrong"] += 1
elif kind == "verdict" and ev.get("status") == "Confirmed":
cap = ev.get("cap", "unknown")
cap_counts[cap]["confirmed"] += 1
THRESHOLD = 0.02
failed = False
for cap, counts in sorted(cap_counts.items()):
total = counts["confirmed"]
wrong = counts["wrong"]
if total == 0:
continue
rate = wrong / total
if rate > THRESHOLD:
print(f"FAIL cap={cap}: false-Confirmed rate {rate:.1%} > {THRESHOLD:.0%} (wrong={wrong}, confirmed={total})")
failed = True
else:
print(f"OK cap={cap}: false-Confirmed rate {rate:.1%} (wrong={wrong}, confirmed={total})")
sys.exit(2 if failed else 0)
PYEOF
RC=$?
set -e
if [[ $RC -eq 0 ]]; then
pass "Gate 2: false-Confirmed rate within threshold"
elif [[ $RC -eq 3 ]]; then
die "Gate 2: telemetry schema mismatch (expected v$EXPECTED_SCHEMA_VERSION) — refusing to silently skip"
else
die "Gate 2: false-Confirmed rate exceeds 2% for one or more caps"
fi
fi
fi
# ── Gate 3: Wall-clock cost ≤ 2× static-only ────────────────────────────────
if skip wall-clock; then
info "Gate 3 (wall-clock): SKIPPED"
else
info "Gate 3: wall-clock ≤ 2× static-only on bench suite..."
BENCH_DIR="${REPO_ROOT}/benches/fixtures"
if [[ ! -d "$BENCH_DIR" ]]; then
info "Gate 3: benches/fixtures not found; skipping"
else
# Portable epoch-millis. BSD date (macOS) lacks %3N; GNU date has it.
ms_now() { python3 -c 'import time; print(int(time.time()*1000))'; }
# Static-only baseline.
T_STATIC_START=$(ms_now)
"$NYX_BIN" scan --no-verify --format json --no-index "$BENCH_DIR" > /dev/null 2>&1 || true
T_STATIC_END=$(ms_now)
T_STATIC=$(( T_STATIC_END - T_STATIC_START ))
# Default (with verify).
T_VERIFY_START=$(ms_now)
"$NYX_BIN" scan --format json --no-index "$BENCH_DIR" > /dev/null 2>&1 || true
T_VERIFY_END=$(ms_now)
T_VERIFY=$(( T_VERIFY_END - T_VERIFY_START ))
info " static-only: ${T_STATIC}ms with-verify: ${T_VERIFY}ms"
# Allow 2× overhead.
LIMIT=$(( T_STATIC * 2 ))
if [[ $T_VERIFY -le $LIMIT ]]; then
pass "Gate 3: wall-clock ${T_VERIFY}ms ≤ 2× ${T_STATIC}ms baseline (limit ${LIMIT}ms)"
else
die "Gate 3: wall-clock ${T_VERIFY}ms > 2× ${T_STATIC}ms baseline (limit ${LIMIT}ms)"
fi
fi
fi
# ── Gate 4: Sandbox-escape suite ─────────────────────────────────────────────
if skip sandbox-escape; then
info "Gate 4 (sandbox-escape): SKIPPED"
else
info "Gate 4: sandbox escape suite green..."
ESCAPE_DIR="${REPO_ROOT}/tests/dynamic_fixtures/escape"
if [[ ! -d "$ESCAPE_DIR" ]]; then
info "Gate 4: tests/dynamic_fixtures/escape not found; skipping"
else
# Run each escape fixture under both Process and Docker backends.
# A Confirmed verdict on an escape fixture is a gate failure.
# Additionally, check that no sentinel files were written to the host.
ESCAPE_SENTINEL_PATHS=(
"/tmp/nyx_escape_file_write_marker"
)
for _s in "${ESCAPE_SENTINEL_PATHS[@]}"; do
rm -f "$_s"
done
ESCAPE_FAILED=0
for fixture in "$ESCAPE_DIR"/*/; do
[[ -d "$fixture" ]] || continue
for backend in process docker; do
set +e
RESULT=$("$NYX_BIN" scan --format json --verify \
--backend "$backend" --no-index "$fixture" 2>/dev/null)
RC=$?
set -e
if [[ $RC -ne 0 && $RC -ne 1 ]]; then
info " $backend/$fixture: nyx returned $RC (likely docker unavailable — skip)"
continue
fi
CONFIRMED=$(echo "$RESULT" | python3 -c "
import json,sys
data = json.load(sys.stdin)
findings = data if isinstance(data, list) else data.get('findings', [])
confirmed = [
f for f in findings
if ((f.get('evidence') or {}).get('dynamic_verdict') or {}).get('status') == 'Confirmed'
]
print(len(confirmed))
" 2>/dev/null || echo 0)
if [[ "$CONFIRMED" -gt 0 ]]; then
die "Gate 4: escape fixture confirmed in $backend backend: $fixture"
ESCAPE_FAILED=1
fi
done
done
for _s in "${ESCAPE_SENTINEL_PATHS[@]}"; do
if [[ -f "$_s" ]]; then
die "Gate 4: escape sentinel written to host: $_s"
ESCAPE_FAILED=1
fi
done
[[ $ESCAPE_FAILED -eq 0 ]] && pass "Gate 4: sandbox escape suite green"
for _s in "${ESCAPE_SENTINEL_PATHS[@]}"; do
rm -f "$_s"
done
fi
fi
# ── Gate 5: Repro stability ≥ 95% ────────────────────────────────────────────
#
# Phase 28 (Track H.4): inversion of the legacy "conservative — treat
# unexpected errors as stable" rule. Old behaviour silently counted any
# subprocess error (timeout, missing toolchain, broken pipe) as stable,
# which let the gate pass while bundles were structurally unreplayable.
# Phase 28 flips that: known exit codes (0 = pass, 1 = sink mismatch,
# 2 = docker unavailable, 3 = toolchain mismatch) are classified
# normally, but any other failure (timeout, ENOENT on `sh`, non-zero
# code outside the documented set) is flagged as instability so the
# gate fails loudly instead of masking the problem.
if skip repro-stability; then
info "Gate 5 (repro-stability): SKIPPED"
else
info "Gate 5: repro artifact stability ≥ 95% of Confirmed..."
# Repro bundles live under dynamic/repro/ (written by repro.rs).
REPRO_DIR="${HOME}/.cache/nyx/dynamic/repro"
if [[ ! -d "$REPRO_DIR" ]] || [[ -z "$(ls -A "$REPRO_DIR" 2>/dev/null)" ]]; then
info "Gate 5: no repro artifacts found at $REPRO_DIR; skipping"
else
python3 - <<'PYEOF' "$REPRO_DIR" "$NYX_BIN"
import subprocess, sys, json, pathlib
# Phase 28 documented reproduce.sh exit codes.
EXIT_PASS = 0 # sink_hit matches expected/outcome.json
EXIT_MISMATCH = 1 # sink_hit diverged from recorded outcome
EXIT_DOCKER_UNAVAIL = 2 # --docker requested but unavailable
EXIT_TOOLCHAIN_MISMATCH = 3 # host toolchain mismatch in process mode
repro_root = pathlib.Path(sys.argv[1])
total = 0
stable = 0
unstable = 0
# Each bundle has expected/verdict.json (written by repro.rs).
for verdict_file in repro_root.rglob("expected/verdict.json"):
bundle_dir = verdict_file.parent.parent # parent of expected/
try:
with open(verdict_file) as f:
orig = json.load(f)
orig_status = orig.get("status", "")
except Exception as e:
# Bundle is malformed. Phase 28 inversion: this is no longer
# silently "stable"; it is a broken bundle and counts against
# the stability rate.
unstable += 1
total += 1
print(f"UNSTABLE: {bundle_dir.name} — verdict.json unreadable ({e})")
continue
if orig_status != "Confirmed":
continue
total += 1
reproduce_sh = bundle_dir / "reproduce.sh"
if not reproduce_sh.exists():
# Legacy bundles without reproduce.sh used to be counted as
# stable; Phase 28 treats them as instability because the
# repro bundle layout has shipped reproduce.sh since the
# first cut of the dynamic feature.
unstable += 1
print(f"UNSTABLE: {bundle_dir.name} — reproduce.sh missing")
continue
try:
result = subprocess.run(
["sh", str(reproduce_sh)],
capture_output=True,
timeout=30,
)
rc = result.returncode
if rc == EXIT_PASS:
stable += 1
elif rc == EXIT_MISMATCH:
unstable += 1
print(f"UNSTABLE: {bundle_dir.name} — sink_hit mismatch (exit 1)")
elif rc in (EXIT_DOCKER_UNAVAIL, EXIT_TOOLCHAIN_MISMATCH):
# Documented environmental skip codes — neither pass nor
# fail. Exclude from the stability ratio so an offline
# CI row does not pollute the score.
total -= 1
print(f"SKIP: {bundle_dir.name} — environment exit {rc}")
else:
# Phase 28 inversion: any other non-zero code is unexpected.
unstable += 1
print(f"UNSTABLE: {bundle_dir.name} — unexpected exit {rc}")
except subprocess.TimeoutExpired:
unstable += 1
print(f"UNSTABLE: {bundle_dir.name} — reproduce.sh exceeded 30s")
except Exception as e:
# Phase 28 inversion: subprocess error is no longer silent
# success. Anything that prevents the script from completing
# cleanly counts against stability.
unstable += 1
print(f"UNSTABLE: {bundle_dir.name} — invocation error ({e})")
if total == 0:
print("No Confirmed repro artifacts found; skipping stability check.")
sys.exit(0)
rate = stable / total
print(f"Repro stability: {stable}/{total} = {rate:.1%} (unstable={unstable})")
if rate < 0.95:
print(f"FAIL: stability {rate:.1%} < 95%")
sys.exit(2)
PYEOF
RC=$?
if [[ $RC -eq 0 ]]; then
pass "Gate 5: repro stability ≥ 95%"
else
die "Gate 5: repro stability < 95%"
fi
fi
fi
# ── Summary ──────────────────────────────────────────────────────────────────
echo ""
info "Gate log: $GATE_LOG"
if [[ $GATE_ERRORS -gt 0 ]]; then
echo ""
echo "M7 SHIP GATE FAILED: $GATE_ERRORS gate(s) did not pass."
echo "Fix failures before merging the default-on flip."
exit 2
else
echo ""
echo "M7 SHIP GATE PASSED: all active gates green."
exit 0
fi

View file

@ -471,9 +471,9 @@ pub enum Commands {
/// Build a harness and dynamically verify each finding in a sandbox. /// Build a harness and dynamically verify each finding in a sandbox.
/// ///
/// Dynamic verification is on by default (M7). This flag is a no-op /// Dynamic verification is on by default. This flag is a no-op when
/// when verification is already enabled via config. Use `--no-verify` /// verification is already enabled via config. Use `--no-verify` to
/// to disable for a single run. Requires the binary to be built with /// disable it for a single run. Requires the binary to be built with
/// `--features dynamic`; without that feature this flag is silently ignored. /// `--features dynamic`; without that feature this flag is silently ignored.
#[cfg_attr(not(feature = "dynamic"), arg(hide = true))] #[cfg_attr(not(feature = "dynamic"), arg(hide = true))]
#[arg(long, help_heading = "Dynamic", conflicts_with = "no_verify")] #[arg(long, help_heading = "Dynamic", conflicts_with = "no_verify")]
@ -489,9 +489,9 @@ pub enum Commands {
/// Also verify `Confidence < Medium` findings dynamically. /// Also verify `Confidence < Medium` findings dynamically.
/// ///
/// By default only `Confidence >= Medium` findings are verified (§5.1). /// By default only `Confidence >= Medium` findings are verified. Pass
/// Pass this flag to run verification on all findings regardless of /// this flag to run verification on all findings regardless of
/// confidence. Intended for corpus-building and backfill runs. /// confidence. Intended for payload tuning and backfill runs.
#[cfg_attr(not(feature = "dynamic"), arg(hide = true))] #[cfg_attr(not(feature = "dynamic"), arg(hide = true))]
#[arg(long, help_heading = "Dynamic")] #[arg(long, help_heading = "Dynamic")]
verify_all_confidence: bool, verify_all_confidence: bool,
@ -532,7 +532,7 @@ pub enum Commands {
)] )]
harden: Option<String>, harden: Option<String>,
// ── Baseline / patch-validation (§M6.5) ──────────────────────── // Baseline / patch-validation
/// Read a previous scan's JSON output (or a stripped .nyx/baseline.json) /// Read a previous scan's JSON output (or a stripped .nyx/baseline.json)
/// and diff it against the current scan on stable_hash. /// and diff it against the current scan on stable_hash.
/// ///
@ -564,7 +564,7 @@ pub enum Commands {
gate: Option<String>, gate: Option<String>,
}, },
/// Submit feedback on a dynamic verification verdict (§21.2). /// Submit feedback on a dynamic verification verdict.
/// ///
/// Records a correction or confirmation for a finding's verdict in the /// Records a correction or confirmation for a finding's verdict in the
/// local telemetry log. Requires `--features dynamic`. /// local telemetry log. Requires `--features dynamic`.

View file

@ -283,10 +283,17 @@ pub fn method_formal_types(method: Node<'_>, bytes: &[u8]) -> Vec<(String, Strin
/// Extract placeholder names from a route path template. /// Extract placeholder names from a route path template.
/// ///
/// Supports two placeholder syntaxes: /// Supports three placeholder syntaxes:
/// - JAX-RS / Spring / Micronaut: `/users/{id}` → `id`, /// - JAX-RS / Spring / Micronaut: `/users/{id}` → `id`,
/// `/users/{id:[0-9]+}` → `id`. /// `/users/{id:[0-9]+}` → `id`.
/// - Servlet-mapping `*` wildcards: ignored (no name to bind). /// - Spring 5.3+ capture-all variables: `/files/{*path}` → `path`
/// (matches the remainder of the URI including slashes).
/// - Bare Ant-style `*` / `**` wildcards (`/users/*`, `/files/**`):
/// intentionally yield no placeholders. They are unnamed by Spring's
/// `AntPathMatcher` and cannot bind by formal name; handlers that
/// need the matched segment use `HttpServletRequest.getRequestURI()`
/// (already routed to [`ParamSource::Implicit`]) or the named
/// `{*name}` capture-all syntax above.
pub fn extract_path_placeholders(path: &str) -> Vec<String> { pub fn extract_path_placeholders(path: &str) -> Vec<String> {
let mut out: Vec<String> = Vec::new(); let mut out: Vec<String> = Vec::new();
let bytes = path.as_bytes(); let bytes = path.as_bytes();
@ -295,7 +302,8 @@ pub fn extract_path_placeholders(path: &str) -> Vec<String> {
if bytes[i] == b'{' if bytes[i] == b'{'
&& let Some(end) = bytes[i + 1..].iter().position(|&b| b == b'}') { && let Some(end) = bytes[i + 1..].iter().position(|&b| b == b'}') {
let inner = &path[i + 1..i + 1 + end]; let inner = &path[i + 1..i + 1 + end];
let name = inner.split(':').next().unwrap_or(inner).trim(); let inner_name = inner.split(':').next().unwrap_or(inner).trim();
let name = inner_name.strip_prefix('*').unwrap_or(inner_name);
if !name.is_empty() && !out.iter().any(|n| n == name) { if !name.is_empty() && !out.iter().any(|n| n == name) {
out.push(name.to_owned()); out.push(name.to_owned());
} }
@ -420,6 +428,26 @@ mod tests {
assert_eq!(extract_path_placeholders("/u/{id:[0-9]+}"), vec!["id"]); assert_eq!(extract_path_placeholders("/u/{id:[0-9]+}"), vec!["id"]);
} }
#[test]
fn extracts_capture_all_variable() {
assert_eq!(extract_path_placeholders("/files/{*path}"), vec!["path"]);
assert_eq!(
extract_path_placeholders("/api/{tenant}/files/{*resource}"),
vec!["tenant", "resource"]
);
}
#[test]
fn unnamed_ant_globs_yield_no_placeholders() {
// Bare `*` and `**` are unnamed by Spring's AntPathMatcher and have
// no name to bind a formal to. Handlers that need the matched
// segment use the request object (routed to [`ParamSource::Implicit`])
// or the named `{*name}` capture-all syntax above.
assert!(extract_path_placeholders("/users/*").is_empty());
assert!(extract_path_placeholders("/files/**").is_empty());
assert!(extract_path_placeholders("/a/*/b/**/c").is_empty());
}
#[test] #[test]
fn join_drops_double_slash() { fn join_drops_double_slash() {
assert_eq!(join_route_path("/api", "/x"), "/api/x"); assert_eq!(join_route_path("/api", "/x"), "/api/x");

View file

@ -18,7 +18,7 @@ use tree_sitter::Node;
use super::ruby_routes::{ use super::ruby_routes::{
bind_path_params, class_extends, class_name, find_class_with_method, first_string_arg, bind_path_params, class_extends, class_name, find_class_with_method, first_string_arg,
kwarg_string, method_formal_names, source_imports_rails, verb_from_ident, first_symbol_arg, kwarg_string, method_formal_names, source_imports_rails, verb_from_ident,
}; };
pub struct RubyRailsAdapter; pub struct RubyRailsAdapter;
@ -40,9 +40,13 @@ fn class_is_rails_controller(class: Node<'_>, bytes: &[u8]) -> bool {
/// Walk the file's top-level `call` nodes looking for a /// Walk the file's top-level `call` nodes looking for a
/// `Rails.application.routes.draw` block or bare `get / post / ...` /// `Rails.application.routes.draw` block or bare `get / post / ...`
/// dispatch lines, and return the first `(method, path)` whose /// dispatch lines, and return the first `(method, path)` whose
/// `to: 'controller#action'` kwarg references the target. Returns /// `to: 'controller#action'` kwarg references the target. Respects
/// `None` when no route mapping is present (the caller then falls /// `namespace :api do ... end` and `scope :v1 do ... end` /
/// back to the conventional `/{action}` shape). /// `scope path: '/v1' do ... end` nesting so a route declared inside
/// such a block resolves against the prefixed path + controller name
/// Rails actually mounts it under. Returns `None` when no mapping
/// is present (the caller then falls back to the conventional
/// `/{action}` shape).
fn find_route_mapping<'a>( fn find_route_mapping<'a>(
root: Node<'a>, root: Node<'a>,
bytes: &'a [u8], bytes: &'a [u8],
@ -50,7 +54,7 @@ fn find_route_mapping<'a>(
action: &str, action: &str,
) -> Option<(HttpMethod, String)> { ) -> Option<(HttpMethod, String)> {
let mut hit: Option<(HttpMethod, String)> = None; let mut hit: Option<(HttpMethod, String)> = None;
visit_routes(root, bytes, controller, action, &mut hit); visit_routes(root, bytes, controller, action, "", "", &mut hit);
hit hit
} }
@ -59,19 +63,98 @@ fn visit_routes<'a>(
bytes: &'a [u8], bytes: &'a [u8],
controller: &str, controller: &str,
action: &str, action: &str,
path_prefix: &str,
ctrl_prefix: &str,
out: &mut Option<(HttpMethod, String)>, out: &mut Option<(HttpMethod, String)>,
) { ) {
if out.is_some() { if out.is_some() {
return; return;
} }
if node.kind() == "call" if node.kind() == "call" {
&& let Some(found) = try_route_mapping(node, bytes, controller, action) { if let Some((kind, ident)) = route_nesting_kind(node, bytes) {
let (path_pfx, ctrl_pfx) = match kind {
NestingKind::Namespace => (
format!("{path_prefix}/{ident}"),
format!("{ctrl_prefix}{ident}/"),
),
NestingKind::ScopeSymbol => (
format!("{path_prefix}/{ident}"),
format!("{ctrl_prefix}{ident}/"),
),
NestingKind::ScopePath => (format!("{path_prefix}/{ident}"), ctrl_prefix.to_owned()),
};
recurse_into_block(node, bytes, controller, action, &path_pfx, &ctrl_pfx, out);
return;
}
if let Some(found) = try_route_mapping(node, bytes, controller, action, path_prefix, ctrl_prefix) {
*out = Some(found); *out = Some(found);
return; return;
} }
}
let mut cur = node.walk(); let mut cur = node.walk();
for child in node.children(&mut cur) { for child in node.children(&mut cur) {
visit_routes(child, bytes, controller, action, out); visit_routes(child, bytes, controller, action, path_prefix, ctrl_prefix, out);
}
}
enum NestingKind {
Namespace,
ScopeSymbol,
ScopePath,
}
/// If `call` is a routes-DSL nesting block (`namespace :api do ... end`,
/// `scope :v1 do ... end`, or `scope path: '/v1' do ... end`) return
/// the kind + the extracted identifier (a bare token for namespace /
/// symbol-scope, a leading-slash-stripped path for path-scope).
fn route_nesting_kind<'a>(call: Node<'a>, bytes: &'a [u8]) -> Option<(NestingKind, String)> {
let mut cur = call.walk();
let mut ident: Option<&str> = None;
let mut args: Option<Node<'a>> = None;
for child in call.named_children(&mut cur) {
match child.kind() {
"identifier" => ident = child.utf8_text(bytes).ok(),
"argument_list" => args = Some(child),
_ => {}
}
}
let ident = ident?;
let args = args?;
match ident {
"namespace" => {
let sym = first_symbol_arg(args, bytes)?;
Some((NestingKind::Namespace, sym))
}
"scope" => {
if let Some(sym) = first_symbol_arg(args, bytes) {
Some((NestingKind::ScopeSymbol, sym))
} else {
let path = kwarg_string(args, bytes, "path")?;
let trimmed = path.trim_start_matches('/').to_owned();
if trimmed.is_empty() {
return None;
}
Some((NestingKind::ScopePath, trimmed))
}
}
_ => None,
}
}
fn recurse_into_block<'a>(
call: Node<'a>,
bytes: &'a [u8],
controller: &str,
action: &str,
path_prefix: &str,
ctrl_prefix: &str,
out: &mut Option<(HttpMethod, String)>,
) {
let mut cur = call.walk();
for child in call.named_children(&mut cur) {
if child.kind() == "do_block" || child.kind() == "block" {
visit_routes(child, bytes, controller, action, path_prefix, ctrl_prefix, out);
}
} }
} }
@ -80,6 +163,8 @@ fn try_route_mapping<'a>(
bytes: &'a [u8], bytes: &'a [u8],
controller: &str, controller: &str,
action: &str, action: &str,
path_prefix: &str,
ctrl_prefix: &str,
) -> Option<(HttpMethod, String)> { ) -> Option<(HttpMethod, String)> {
let mut cur = call.walk(); let mut cur = call.walk();
let mut verb: Option<HttpMethod> = None; let mut verb: Option<HttpMethod> = None;
@ -100,8 +185,14 @@ fn try_route_mapping<'a>(
let path = first_string_arg(args, bytes)?; let path = first_string_arg(args, bytes)?;
let to = kwarg_string(args, bytes, "to")?; let to = kwarg_string(args, bytes, "to")?;
let (ctrl, act) = to.split_once('#')?; let (ctrl, act) = to.split_once('#')?;
if controller_matches(ctrl, controller) && act == action { let full_ctrl = format!("{ctrl_prefix}{ctrl}");
return Some((verb, path)); if controller_matches(&full_ctrl, controller) && act == action {
let full_path = if path_prefix.is_empty() {
path
} else {
format!("{}/{}", path_prefix, path.trim_start_matches('/'))
};
return Some((verb, full_path));
} }
None None
} }
@ -269,6 +360,51 @@ mod tests {
assert!(matches!(id.source, crate::dynamic::framework::ParamSource::PathSegment(_))); assert!(matches!(id.source, crate::dynamic::framework::ParamSource::PathSegment(_)));
} }
#[test]
fn routes_draw_namespace_applies_prefix_to_path_and_controller() {
let src: &[u8] = b"Rails.application.routes.draw do\n namespace :api do\n get '/users', to: 'users#index'\n end\nend\n\nclass Api::UsersController < ApplicationController\n def index\n 'ok'\n end\nend\n";
let tree = parse(src);
let binding = RubyRailsAdapter
.detect(&summary("index"), tree.root_node(), src)
.expect("binding");
let route = binding.route.unwrap();
assert_eq!(route.path, "/api/users");
assert_eq!(route.method, HttpMethod::GET);
}
#[test]
fn routes_draw_scope_path_prefixes_path_only() {
let src: &[u8] = b"Rails.application.routes.draw do\n scope path: '/v1' do\n get '/users', to: 'users#index'\n end\nend\n\nclass UsersController < ApplicationController\n def index\n 'ok'\n end\nend\n";
let tree = parse(src);
let binding = RubyRailsAdapter
.detect(&summary("index"), tree.root_node(), src)
.expect("binding");
let route = binding.route.unwrap();
assert_eq!(route.path, "/v1/users");
}
#[test]
fn routes_draw_scope_symbol_prefixes_path_and_controller() {
let src: &[u8] = b"Rails.application.routes.draw do\n scope :admin do\n get '/users', to: 'users#index'\n end\nend\n\nclass Admin::UsersController < ApplicationController\n def index\n 'ok'\n end\nend\n";
let tree = parse(src);
let binding = RubyRailsAdapter
.detect(&summary("index"), tree.root_node(), src)
.expect("binding");
let route = binding.route.unwrap();
assert_eq!(route.path, "/admin/users");
}
#[test]
fn routes_draw_nested_namespaces_compose_prefixes() {
let src: &[u8] = b"Rails.application.routes.draw do\n namespace :api do\n namespace :v1 do\n get '/users', to: 'users#index'\n end\n end\nend\n\nclass Api::V1::UsersController < ApplicationController\n def index\n 'ok'\n end\nend\n";
let tree = parse(src);
let binding = RubyRailsAdapter
.detect(&summary("index"), tree.root_node(), src)
.expect("binding");
let route = binding.route.unwrap();
assert_eq!(route.path, "/api/v1/users");
}
#[test] #[test]
fn skips_when_class_is_not_a_controller() { fn skips_when_class_is_not_a_controller() {
let src: &[u8] = b"class Foo\n def bar\n 'ok'\n end\nend\n"; let src: &[u8] = b"class Foo\n def bar\n 'ok'\n end\nend\n";

View file

@ -145,7 +145,7 @@ fn named_child_of_kind<'a>(node: Node<'a>, kind: &str) -> Option<Node<'a>> {
pub fn class_name<'a>(class: Node<'a>, bytes: &'a [u8]) -> Option<&'a str> { pub fn class_name<'a>(class: Node<'a>, bytes: &'a [u8]) -> Option<&'a str> {
let mut cur = class.walk(); let mut cur = class.walk();
for c in class.named_children(&mut cur) { for c in class.named_children(&mut cur) {
if c.kind() == "constant" { if c.kind() == "constant" || c.kind() == "scope_resolution" {
return c.utf8_text(bytes).ok(); return c.utf8_text(bytes).ok();
} }
} }
@ -352,6 +352,22 @@ fn is_implicit_formal(name: &str) -> bool {
matches!(name, "env" | "request" | "req" | "params" | "response" | "res") matches!(name, "env" | "request" | "req" | "params" | "response" | "res")
} }
/// Read the first positional symbol argument (`:foo`) from an
/// `argument_list` child. Used by the Rails router DSL to pull the
/// namespace name out of `namespace :api do ... end` and the
/// positional form of `scope :v1 do ... end`. The returned string
/// is the symbol's identifier portion without the leading colon.
pub fn first_symbol_arg<'a>(args: Node<'a>, bytes: &'a [u8]) -> Option<String> {
let mut cur = args.walk();
for c in args.named_children(&mut cur) {
if c.kind() == "simple_symbol" {
let raw = c.utf8_text(bytes).ok()?;
return Some(raw.trim_start_matches(':').to_owned());
}
}
None
}
/// Read the first positional string-literal argument from an /// Read the first positional string-literal argument from an
/// `argument_list` child. Used by every Ruby route adapter to pull /// `argument_list` child. Used by every Ruby route adapter to pull
/// a path template out of `get '/run' do ... end` and the Rails /// a path template out of `get '/run' do ... end` and the Rails

View file

@ -565,10 +565,9 @@ pub enum ReplayResult {
/// Tri-state map of [`ReplayResult`] onto the eval-corpus /// Tri-state map of [`ReplayResult`] onto the eval-corpus
/// `VerifyResult::replay_stable` field shape. /// `VerifyResult::replay_stable` field shape.
/// ///
/// * `Some(true)` — replay matched the recorded outcome. /// * `Some(true)` - replay matched the recorded outcome.
/// * `Some(false)` — replay diverged or aborted in a way that the M7 /// * `Some(false)` - replay diverged or aborted.
/// Gate-5 inversion treats as instability. /// * `None` - replay was not informative (toolchain mismatched, docker
/// * `None` — replay was not informative (toolchain mismatched, docker
/// unavailable, or the bundle had no `reproduce.sh`). The corpus /// unavailable, or the bundle had no `reproduce.sh`). The corpus
/// tabulator treats `None` as "no signal" and excludes the row from /// tabulator treats `None` as "no signal" and excludes the row from
/// the per-cell `stable_replays` numerator. /// the per-cell `stable_replays` numerator.
@ -582,15 +581,14 @@ pub fn replay_stability(result: &ReplayResult) -> Option<bool> {
} }
} }
/// Phase 28 — Track H.3. Run `reproduce.sh` in `bundle_root` and map the /// Run `reproduce.sh` in `bundle_root` and map the shell exit code into a
/// shell exit code into a [`ReplayResult`]. /// [`ReplayResult`].
/// ///
/// `extra_args` is appended to `reproduce.sh` (`--docker` when the caller /// `extra_args` is appended to `reproduce.sh` (`--docker` when the caller
/// wants the docker backend; empty for the process backend). /// wants the docker backend; empty for the process backend).
/// ///
/// This is the host-side companion to the M7 Gate 5 inversion: callers /// Callers who want "did this bundle replay green?" semantics get a typed
/// who want "did this bundle replay green?" semantics see a typed result /// result instead of parsing shell output.
/// and the M7 gate script gets a uniform contract to assert against.
pub fn replay_bundle( pub fn replay_bundle(
bundle_root: &Path, bundle_root: &Path,
extra_args: &[&str], extra_args: &[&str],

View file

@ -1,9 +1,9 @@
//! Telemetry event log (§21.1). //! Telemetry event log.
//! //!
//! Writes one JSON line per verdict to `~/.cache/nyx/dynamic/events.jsonl`. //! Writes one JSON line per verdict to `~/.cache/nyx/dynamic/events.jsonl`.
//! `NYX_NO_TELEMETRY=1` silently disables all writes (§21.4). //! `NYX_NO_TELEMETRY=1` silently disables all writes.
//! //!
//! # Schema (Phase 27) //! # Schema
//! //!
//! Every record starts with three envelope fields so the on-disk format can //! Every record starts with three envelope fields so the on-disk format can
//! evolve across releases without silently mixing incompatible records: //! evolve across releases without silently mixing incompatible records:
@ -12,11 +12,10 @@
//! - `nyx_version`: the Cargo package version that wrote the record. //! - `nyx_version`: the Cargo package version that wrote the record.
//! - `corpus_version`: the payload-corpus version active at write time. //! - `corpus_version`: the payload-corpus version active at write time.
//! //!
//! Followed by a `kind` discriminator (`"verdict"` or `"rank_delta"`). All //! Followed by a `kind` discriminator (`"verdict"` or `"rank_delta"`). All
//! readers (`read_events`, the M7 ship gate) require `schema_version == //! readers require `schema_version == SCHEMA_VERSION`; mismatched records
//! [`SCHEMA_VERSION`]; mismatched records produce //! produce [`TelemetryReadError::SchemaMismatch`] instead of being silently
//! [`TelemetryReadError::SchemaMismatch`] instead of being silently parsed //! parsed as if they matched.
//! as if they matched.
//! //!
//! ```json //! ```json
//! { //! {
@ -258,12 +257,10 @@ fn lang_from_path(path: &str) -> String {
.unwrap_or_else(|| "unknown".to_owned()) .unwrap_or_else(|| "unknown".to_owned())
} }
/// Sampling decision for telemetry writes (Phase 27, Track H.2). /// Sampling decision for telemetry writes.
/// ///
/// Confirmed and Inconclusive verdicts are calibration-critical (false-Confirmed /// Confirmed and Inconclusive verdicts are kept for calibration. Other verdict
/// rate gates M7 ship; Inconclusive reasons drive the spec-derivation roadmap) /// statuses can be downsampled to bound log growth on high-volume scans.
/// and are always retained. Other verdict statuses can be downsampled to bound
/// log growth on high-volume scans.
/// ///
/// The decision is seeded by `spec_hash` so the *same* finding makes the *same* /// The decision is seeded by `spec_hash` so the *same* finding makes the *same*
/// keep-or-drop call across reruns. Without this, two scans of the same project /// keep-or-drop call across reruns. Without this, two scans of the same project
@ -413,12 +410,11 @@ pub fn log_path() -> Option<std::path::PathBuf> {
events_log_path() events_log_path()
} }
// ── Reading events back (Phase 27) ─────────────────────────────────────────── // Reading events back
/// Structured error returned by [`read_events`]. /// Structured error returned by [`read_events`].
/// ///
/// Surfaced to the M7 ship gate so Gate 2 can fail loudly on schema-mismatch /// Returned when a log mixes records from incompatible schema versions.
/// rather than silently treating mismatched records as "no data".
#[derive(Debug, thiserror::Error)] #[derive(Debug, thiserror::Error)]
pub enum TelemetryReadError { pub enum TelemetryReadError {
#[error("io error reading {path}: {source}")] #[error("io error reading {path}: {source}")]
@ -451,14 +447,12 @@ pub enum TelemetryReadError {
/// ///
/// Returns each line as a `serde_json::Value` so callers can dispatch on the /// Returns each line as a `serde_json::Value` so callers can dispatch on the
/// `kind` discriminator themselves. Rejects any record whose `schema_version` /// `kind` discriminator themselves. Rejects any record whose `schema_version`
/// does not match [`SCHEMA_VERSION`] (this is the explicit failure mode the /// does not match [`SCHEMA_VERSION`]. A v0 record from an older release must
/// M7 ship gate Gate 2 consumes; a v0 record from an older release must not /// not silently parse as if the schema had never changed.
/// silently parse as if the schema had never changed).
/// ///
/// Blank lines are skipped. Any malformed JSON or missing `schema_version` /// Blank lines are skipped. Any malformed JSON or missing `schema_version`
/// fails the whole read; partial recovery is not the contract here because /// fails the whole read; partial recovery is not the contract for telemetry
/// the ship gate already treats "log missing or unreadable" as "no data, /// logs.
/// skip Gate 2 with a notice."
pub fn read_events(path: &Path) -> Result<Vec<serde_json::Value>, TelemetryReadError> { pub fn read_events(path: &Path) -> Result<Vec<serde_json::Value>, TelemetryReadError> {
let file = std::fs::File::open(path).map_err(|e| TelemetryReadError::Io { let file = std::fs::File::open(path).map_err(|e| TelemetryReadError::Io {
path: path.to_path_buf(), path: path.to_path_buf(),
@ -551,8 +545,8 @@ pub fn feedback_wrong_for_finding(path: &Path, finding_id: &str) -> Option<bool>
/// One telemetry event per ranked finding that carries a dynamic verdict delta. /// One telemetry event per ranked finding that carries a dynamic verdict delta.
/// ///
/// Emitted by `rank::rank_diags` for every diag whose dynamic verdict shifts /// Emitted by `rank::rank_diags` for every diag whose dynamic verdict shifts
/// its rank score (delta != 0). Used by the M7 calibration pipeline to tune /// its rank score (delta != 0). Used to tune the N/M boost/penalty constants
/// the N/M boost/penalty constants from real-world verdict distributions. /// from real-world verdict distributions.
#[derive(Debug, serde::Serialize, serde::Deserialize)] #[derive(Debug, serde::Serialize, serde::Deserialize)]
pub struct RankDeltaEvent { pub struct RankDeltaEvent {
pub schema_version: u32, pub schema_version: u32,

View file

@ -85,14 +85,11 @@ pub struct VerifyOptions {
/// Default `false`. [`Self::from_config`] honours the /// Default `false`. [`Self::from_config`] honours the
/// `NYX_VERIFY_REPLAY_STABLE` environment variable (`1` / `true`). /// `NYX_VERIFY_REPLAY_STABLE` environment variable (`1` / `true`).
pub replay_stable_check: bool, pub replay_stable_check: bool,
/// Phase 31 follow-up: when `true` and `replay_stable_check` is also /// When `true` and `replay_stable_check` is also `true`, the verifier
/// `true`, the verifier passes `--docker` to `reproduce.sh` instead of /// passes `--docker` to `reproduce.sh` instead of running it through the
/// running it through the host's process backend. Lets the eval-corpus /// host's process backend. This lets eval-corpus runs mark
/// driver mark `replay_stable` based on the bare-image replay path so /// `replay_stable` from the bare-image replay path when the host has
/// the M7 ship-gate's Gate 5 reflects the docker bundle's green/red /// stripped language toolchains.
/// signal — required when the corpus walks a host that has stripped
/// the language toolchains (the bare-image CI matrix at
/// `.github/workflows/repro-bare.yml`).
/// ///
/// Default `false`. [`Self::from_config`] honours the /// Default `false`. [`Self::from_config`] honours the
/// `NYX_VERIFY_REPLAY_DOCKER` environment variable (`1` / `true`). /// `NYX_VERIFY_REPLAY_DOCKER` environment variable (`1` / `true`).

View file

@ -99,7 +99,7 @@ pub fn compute_attack_rank(diag: &Diag) -> AttackRank {
// All other verdicts (Unsupported, Inconclusive, no verdict) are // All other verdicts (Unsupported, Inconclusive, no verdict) are
// unaffected: no data is better than speculative data. // unaffected: no data is better than speculative data.
// //
// Calibrated values (M7 eval corpus): N=20, M=5. // Calibrated values from the eval corpus: N=20, M=5.
// N=20 ensures Confirmed findings from any severity tier surface // N=20 ensures Confirmed findings from any severity tier surface
// above static-only peers: High(60)+20=80 > High(60)+taint(10)=70. // above static-only peers: High(60)+20=80 > High(60)+taint(10)=70.
// M=5 nudges exhausted-corpus NotConfirmed below equal static peers // M=5 nudges exhausted-corpus NotConfirmed below equal static peers
@ -209,7 +209,7 @@ pub fn rank_diags(diags: &mut [Diag]) {
if !rank.components.is_empty() { if !rank.components.is_empty() {
d.rank_reason = Some(rank.components.clone()); d.rank_reason = Some(rank.components.clone());
} }
// Emit rank-delta telemetry for M7 calibration (§21 / deferred M7 hook). // Emit rank-delta telemetry for score calibration.
// Only fires when the dynamic verdict shifted the score; benign verdicts // Only fires when the dynamic verdict shifted the score; benign verdicts
// (Unsupported, Inconclusive, no verdict) produce delta = None and are // (Unsupported, Inconclusive, no verdict) produce delta = None and are
// skipped — emitting them would add noise without calibration value. // skipped — emitting them would add noise without calibration value.
@ -247,17 +247,16 @@ pub fn rank_diags(diags: &mut [Diag]) {
/// Returns `None` when there is no verdict (static-only scan) or the verdict /// Returns `None` when there is no verdict (static-only scan) or the verdict
/// does not change the score (Unsupported, Inconclusive). /// does not change the score (Unsupported, Inconclusive).
/// ///
/// Design note (§deferred M7 payload_corpus_complete): the spec originally /// Design note: the spec originally distinguished `NotConfirmed` +
/// distinguished `NotConfirmed` + `payload_corpus_complete == true` → `-M` /// `payload_corpus_complete == true` from `NotConfirmed` +
/// from `NotConfirmed` + `NoPayloadsForCap` → no change. In practice the /// `NoPayloadsForCap`. In practice the
/// `NoPayloadsForCap` path always produces `Unsupported`, never `NotConfirmed`, /// `NoPayloadsForCap` path always produces `Unsupported`, never `NotConfirmed`,
/// so the two cases are already disjoint in the type. The heuristic /// so the two cases are already disjoint in the type. The heuristic
/// `!dv.attempts.is_empty()` (corpus was actually tried) is equivalent to /// `!dv.attempts.is_empty()` (corpus was actually tried) is equivalent to
/// `payload_corpus_complete == true` for all reachable states no extra /// `payload_corpus_complete == true` for all reachable states, so no extra
/// field is needed. See also §deferred decision in `.pitboss/play/deferred.md`. /// field is needed.
/// ///
/// Values calibrated against M7 eval corpus (OWASP Benchmark v1.2 + in-house curated set): /// Values calibrated against the eval corpus: N=20, M=5.
/// N=20, M=5 — see `docs/dynamic_eval_m7.md` for precision/recall breakdowns.
fn dynamic_verdict_delta(diag: &Diag) -> Option<f64> { fn dynamic_verdict_delta(diag: &Diag) -> Option<f64> {
use crate::evidence::VerifyStatus; use crate::evidence::VerifyStatus;
let dv = diag.evidence.as_ref()?.dynamic_verdict.as_ref()?; let dv = diag.evidence.as_ref()?.dynamic_verdict.as_ref()?;

View file

@ -36,9 +36,9 @@ struct StartScanRequest {
engine_profile: Option<String>, engine_profile: Option<String>,
/// Override dynamic verification for this scan. /// Override dynamic verification for this scan.
/// ///
/// `true` force on even if config says off. /// `true` - force on even if config says off.
/// `false` — force off even if config says on (M7 default-on). /// `false` - force off even if config says on.
/// absent — inherit config default (true since M7). /// absent - inherit config default.
/// ///
/// Requires `--features dynamic`; `true` returns 400 when the /// Requires `--features dynamic`; `true` returns 400 when the
/// feature is absent. /// feature is absent.

View file

@ -251,8 +251,8 @@ pub struct ScannerConfig {
/// Run dynamic verification on each finding after the static pass. /// Run dynamic verification on each finding after the static pass.
/// ///
/// Default `true` (M7 flip). Each `Confidence >= Medium` finding is /// Default `true`. Each `Confidence >= Medium` finding is passed to
/// passed to `dynamic::verify_finding` and the result is stored in /// `dynamic::verify_finding` and the result is stored in
/// `Evidence::dynamic_verdict`. Use `--no-verify` (CLI) or set /// `Evidence::dynamic_verdict`. Use `--no-verify` (CLI) or set
/// `verify = false` in `nyx.toml` to disable. /// `verify = false` in `nyx.toml` to disable.
/// ///

View file

@ -1,17 +1,10 @@
# Phase 31: ratchet values set to the headline targets. # Eval corpus budget.
# #
# These are the published acceptance numbers behind the dynamic-verification # `report.py` enforces these values when `run.sh` or `run_full.sh` pass
# overhaul (see `docs/dynamic.md` "Headline metrics"). The ratchet schedule # `--budget`. Each (cap, lang) cell uses the default row unless a specific
# from Phase 29 collapsed into a single target row: every (cap, lang) cell is # override appears below.
# now gated against the same headline thresholds. Per-cell carve-outs were
# dropped in Phase 31; if a cell is still wider than these numbers in practice
# it shows up as a per-cell `FAIL` in `report.py` and as a gate-1 failure in
# `scripts/m7_ship_gate.sh`, which is the intended forcing function for the
# remaining engine follow-ups tracked in `.pitboss/play/deferred.md`.
# #
# Wall-clock cost (≤ 2× static-only) is enforced separately by Gate 3 of # Wall-clock cost is measured separately from this per-cell budget.
# `scripts/m7_ship_gate.sh` against `benches/fixtures/`; it is not a per-cell
# budget knob and has no entry in this file.
# #
# Schema: # Schema:
# #

View file

@ -1,23 +1,23 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Eval corpus runner for M7 pre-flip gate calibration. # Eval corpus runner.
# #
# Usage: # Usage:
# tests/eval_corpus/run.sh [--output DIR] [--nyx BIN] [--sets owasp,sard,inhouse] # tests/eval_corpus/run.sh [--output DIR] [--nyx BIN] [--sets owasp,sard,inhouse]
# #
# Bootstraps OWASP Benchmark v1.2, NIST SARD subset, and in-house # Bootstraps OWASP Benchmark v1.2, the NIST SARD subset, and Nyx benchmark
# bughunt-curated fixtures. Runs `nyx scan --verify` on each. Emits # fixtures. Runs `nyx scan --verify` on each. Emits
# per-cell (cap x language) precision/recall table and per-cap Unsupported # per-cell (cap x language) precision/recall table and per-cap Unsupported
# rate to stdout (and --output DIR if given). # rate to stdout (and --output DIR if given).
# #
# Environment: # Environment:
# NYX_EVAL_CORPUS_DIR path to pre-downloaded corpus roots # NYX_EVAL_CORPUS_DIR - path to pre-downloaded corpus roots
# (default: ~/.cache/nyx/eval_corpus) # (default: ~/.cache/nyx/eval_corpus)
# NYX_BIN path to nyx binary (default: ./target/release/nyx) # NYX_BIN - path to nyx binary (default: ./target/release/nyx)
# #
# Exit codes: # Exit codes:
# 0 — all gate thresholds met # 0 - all budget thresholds met
# 1 setup or I/O error # 1 - setup or I/O error
# 2 — one or more gate thresholds exceeded (see output for details) # 2 - one or more budget thresholds exceeded (see output for details)
set -euo pipefail set -euo pipefail
@ -173,9 +173,8 @@ python3 "${SCRIPT_DIR}/report.py" \
${DIFF_FILE:+--diff "$DIFF_FILE"} ${DIFF_FILE:+--diff "$DIFF_FILE"}
REPORT_RC=$? REPORT_RC=$?
set -e set -e
# Propagate gate-fail (exit 2) and malformed-config (exit 3) so the # Propagate budget failures (exit 2) and malformed config (exit 3). Treat other
# m7_ship_gate.sh Gate-1 dispatch can tell them apart. Treat other # non-zero exits as setup errors.
# non-zero as setup error (exit 1).
if [[ $REPORT_RC -eq 2 ]]; then if [[ $REPORT_RC -eq 2 ]]; then
exit 2 exit 2
elif [[ $REPORT_RC -eq 3 ]]; then elif [[ $REPORT_RC -eq 3 ]]; then

View file

@ -1,12 +1,10 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Phase 31: full eval-corpus orchestrator. # Full eval-corpus orchestrator.
# #
# Drives a complete pass against every corpus set the project knows about # Drives a complete pass against every corpus set the project knows about
# (OWASP Benchmark v1.2, the NIST SARD subset, and the in-house bughunt # (OWASP Benchmark v1.2, the NIST SARD subset, and the Nyx benchmark
# fixtures), then emits a stable `tests/eval_corpus/results.json` so # fixtures), then emits `tests/eval_corpus/results.json` for reports,
# downstream consumers (M7 ship gate, monotonic-improvement diff, the # diffs, and docs.
# headline metrics table in `docs/dynamic.md`) can read a single
# well-known path.
# #
# Usage: # Usage:
# tests/eval_corpus/run_full.sh [--nyx BIN] [--budget FILE] [--diff FILE] # tests/eval_corpus/run_full.sh [--nyx BIN] [--budget FILE] [--diff FILE]
@ -15,11 +13,9 @@
# Differences vs `run.sh`: # Differences vs `run.sh`:
# * Always runs every set (no `--sets` selector). # * Always runs every set (no `--sets` selector).
# * Always passes `--budget tests/eval_corpus/budget.toml` so the # * Always passes `--budget tests/eval_corpus/budget.toml` so the
# headline targets (Unsupported < 20%, FalseConfirmed < 2%, Repro # configured per-cell limits are checked on every pass.
# stability >= 95%) gate every pass.
# * Copies the timestamped results file to # * Copies the timestamped results file to
# `tests/eval_corpus/results.json` (canonical path consumed by # `tests/eval_corpus/results.json`.
# `scripts/m7_ship_gate.sh` and the published metrics doc).
# #
# Exit codes: # Exit codes:
# 0 every set ran and the merged result met the per-cell budget. # 0 every set ran and the merged result met the per-cell budget.

View file

@ -415,8 +415,8 @@ def main() -> int:
elif status == "Confirmed": elif status == "Confirmed":
cells[key]["confirmed"] += 1 cells[key]["confirmed"] += 1
# Repro-stability and false-Confirmed counts are optional # Repro-stability and false-Confirmed counts are optional
# fields tabulate.py reads off the verdict when callers # fields tabulate.py reads off the verdict when callers have
# (m7_ship_gate.sh / corpus_promote.yml) have stamped them. # stamped them.
if dv.get("wrong") is True: if dv.get("wrong") is True:
cells[key]["wrong_confirmed"] += 1 cells[key]["wrong_confirmed"] += 1
if dv.get("replay_stable") is True: if dv.get("replay_stable") is True:

View file

@ -1,14 +1,13 @@
//! Phase 27 — Track H.1 integration test. //! Dynamic telemetry schema tests.
//! //!
//! Locks in the on-disk telemetry schema contract that `scripts/m7_ship_gate.sh` //! Locks in the on-disk telemetry schema contract:
//! Gate 2 relies on:
//! //!
//! - Records produced today carry the `schema_version`, `nyx_version`, and //! - Records produced today carry the `schema_version`, `nyx_version`, and
//! `corpus_version` envelope fields, plus a `kind` discriminator. //! `corpus_version` envelope fields, plus a `kind` discriminator.
//! - `read_events(path)` accepts the current schema. //! - `read_events(path)` accepts the current schema.
//! - A hand-crafted record with `schema_version: 0` is rejected by //! - A hand-crafted record with `schema_version: 0` is rejected by
//! `read_events` with a typed [`TelemetryReadError::SchemaMismatch`] (this //! `read_events` with a typed [`TelemetryReadError::SchemaMismatch`] (this
//! is the explicit Phase 27 acceptance bullet). //! is the required failure mode for mixed-schema logs).
//! - The sampling policy retains Confirmed and Inconclusive verdicts even at //! - The sampling policy retains Confirmed and Inconclusive verdicts even at
//! `sample_rate_other = 0.0`. //! `sample_rate_other = 0.0`.