feat(eval-corpus): add Track R.2 polyglot corpora (RailsGoat, DVWA, DVPWA, gosec, RustSec) with curated manifests, negative controls, and CI validation

tests/eval_corpus/ground_truth/README.md

@@ -69,3 +69,38 @@ known vulns) is the meaningful metric; precision vs this partial ground
 truth is informational. Gate 7 publishes per-cap precision/recall/confirmed
 report-only by default (`NYX_JSTS_FLOOR_CAPS` empty), matching the OWASP
 gate.
+
+## Polyglot real corpora (Ruby/PHP/Python/Go/Rust — Track R.2)
+
+Phase 29 wires the remaining language families into the same machinery, one
+corpus per family, each with a curated `*.manifest.toml` → committed `*.json`:
+
+- `railsgoat.{manifest.toml,json}` — OWASP RailsGoat (Rails, `.rb`).
+- `dvwa.{manifest.toml,json}` — Damn Vulnerable Web Application (PHP). DVWA
+  ships graded source variants (`source/{low,impossible}.php`), so this is
+  the one Track R corpus besides OWASP with real vuln/benign **pairs**
+  (`low.php` = vuln, `impossible.php` = benign control) — precision is
+  meaningful here, not just informational.
+- `dvpwa.{manifest.toml,json}` — Damn Vulnerable Python Web App (aiohttp,
+  `.py`). Its parameterized DAO siblings are benign controls for the one
+  `%`-formatted SQL sink.
+- `gosec.{manifest.toml,json}` — the gosec Go SAST tool repo; the scannable,
+  `// want`-annotated sample under `goanalysis/testdata` is the curated
+  ground truth (gosec's string-embedded rule samples are not scannable, so
+  they are deliberately unlabelled).
+- `rustsec.{manifest.toml,json}` — RustSec advisory-db, a **negative
+  control**. advisory-db ships advisory metadata, not vulnerable `.rs`
+  source, so its committed ground truth is empty (`[]`) by construction. The
+  manifest sets `negative_control = true` (mutually exclusive with
+  `[[entry]]` tables); `manifest_gt_convert.py` emits the empty JSON and the
+  row asserts the Rust scan/verify path runs at scale within wall-clock and
+  Confirms nothing there (any Confirmed Rust finding is a false confirm).
+
+These are converted, validated and asserted-in-sync exactly like NodeGoat /
+Juice Shop (the `polyglot` job in `.github/workflows/eval.yml`). Because each
+corpus targets a single language, Gate 8 scopes tabulation to that language
+(`tabulate.py --lang`) so the vendored third-party JavaScript these Ruby /
+Python apps bundle does not pollute their per-cap metrics. Gate 8 publishes
+per-cap precision/recall/confirmed report-only by default
+(`NYX_POLYGLOT_FLOOR_CAPS` empty), matching the OWASP and JS/TS gates. See
+`tests/eval_corpus/budget.toml` for the per-(cap,lang) gate policy.

tests/eval_corpus/ground_truth/dvpwa.json

@@ -0,0 +1,38 @@
+[
+  {
+    "path": "sqli/dao/course.py",
+    "line": 0,
+    "cap": "sqli",
+    "vuln": false
+  },
+  {
+    "path": "sqli/dao/mark.py",
+    "line": 0,
+    "cap": "sqli",
+    "vuln": false
+  },
+  {
+    "path": "sqli/dao/review.py",
+    "line": 0,
+    "cap": "sqli",
+    "vuln": false
+  },
+  {
+    "path": "sqli/dao/student.py",
+    "line": 0,
+    "cap": "sqli",
+    "vuln": true
+  },
+  {
+    "path": "sqli/dao/user.py",
+    "line": 0,
+    "cap": "crypto",
+    "vuln": true
+  },
+  {
+    "path": "sqli/views.py",
+    "line": 0,
+    "cap": "auth",
+    "vuln": true
+  }
+]

tests/eval_corpus/ground_truth/dvpwa.manifest.toml

@@ -0,0 +1,70 @@
+# DVPWA (Damn Vulnerable Python Web Application) — curated ground-truth
+# manifest (Phase 29, Track R.2).
+#
+# DVPWA is an intentionally-vulnerable aiohttp app whose headline flaw is
+# SQL injection (the package is literally named `sqli`).  It ships no
+# machine-readable per-file labels, so this manifest IS the authoritative
+# source.  Its DAO layer is convenient: one method builds a query with
+# Python `%` string-formatting (the injectable sink) while its siblings use
+# proper parameterized `cur.execute(q, params)` — so the parameterized DAOs
+# serve as genuine benign controls (vuln = false) for the sqli cell, making
+# precision there meaningful, not just informational.
+#
+# tests/eval_corpus/manifest_gt_convert.py turns this into the committed
+# ground_truth/dvpwa.json.  CI regenerates it against a fresh clone of the
+# pinned ref and asserts byte-equality; the converter HARD-ERRORS on any
+# path that no longer exists, so a corpus bump that moves a DAO fails the
+# job loudly rather than silently dropping recall.
+#
+# `cap` is a nyx cap label (tabulate.py), aligned to how nyx classifies each
+# sink (the request-scoped ownership lookups in views.py surface as `auth`).
+# `path` is relative to the DVPWA clone root, POSIX separators.  Lang is
+# inferred from the extension (.py -> python).  See
+# tests/eval_corpus/budget.toml for the gate policy on these cells.
+
+corpus = "dvpwa"
+upstream = "https://github.com/anxolerd/dvpwa"
+# DVPWA publishes no release tags; the eval job pins the default branch via
+# the CI cache key (clone HEAD a1d8f89fac2e57093189853c6527c2b01fc1d9c1).
+# The sqli/ package layout has been stable; re-validate if the cache key is
+# bumped.
+pinned_ref = "master"
+
+# ── SQL injection (sqli) — one injectable sink + parameterized controls ──────
+[[entry]]
+path = "sqli/dao/student.py"
+cap = "sqli"
+vuln = true
+note = "Student.create builds the INSERT with Python `%` formatting (\"... VALUES ('%(name)s')\" % {'name': name}) on the request-supplied student name, then cur.execute(q) — SQL injection."
+
+[[entry]]
+path = "sqli/dao/course.py"
+cap = "sqli"
+vuln = false
+note = "benign control: every Course query uses parameterized cur.execute(q, params) / VALUES (%(title)s, %(description)s) — not injectable."
+
+[[entry]]
+path = "sqli/dao/review.py"
+cap = "sqli"
+vuln = false
+note = "benign control: Review.create / get_for_course bind via cur.execute(q, params) with %(course_id)s / %s placeholders — parameterized."
+
+[[entry]]
+path = "sqli/dao/mark.py"
+cap = "sqli"
+vuln = false
+note = "benign control: Mark.create / get_for_student bind via parameterized cur.execute(q, params) — not injectable."
+
+# ── Weak crypto (crypto) ─────────────────────────────────────────────────────
+[[entry]]
+path = "sqli/dao/user.py"
+cap = "crypto"
+vuln = true
+note = "User.check_password compares against md5(password).hexdigest() — unsalted MD5 for credential storage (weak cryptography)."
+
+# ── Broken access control (auth) ─────────────────────────────────────────────
+[[entry]]
+path = "sqli/views.py"
+cap = "auth"
+vuln = true
+note = "request handlers resolve the acting user from a client-controlled session id and act on objects without an ownership/authorization check — broken access control."

tests/eval_corpus/ground_truth/dvwa.json

@@ -0,0 +1,50 @@
+[
+  {
+    "path": "vulnerabilities/exec/source/impossible.php",
+    "line": 0,
+    "cap": "cmdi",
+    "vuln": false
+  },
+  {
+    "path": "vulnerabilities/exec/source/low.php",
+    "line": 0,
+    "cap": "cmdi",
+    "vuln": true
+  },
+  {
+    "path": "vulnerabilities/open_redirect/source/impossible.php",
+    "line": 0,
+    "cap": "header_injection",
+    "vuln": false
+  },
+  {
+    "path": "vulnerabilities/open_redirect/source/impossible.php",
+    "line": 0,
+    "cap": "redirect",
+    "vuln": false
+  },
+  {
+    "path": "vulnerabilities/open_redirect/source/low.php",
+    "line": 0,
+    "cap": "header_injection",
+    "vuln": true
+  },
+  {
+    "path": "vulnerabilities/open_redirect/source/low.php",
+    "line": 0,
+    "cap": "redirect",
+    "vuln": true
+  },
+  {
+    "path": "vulnerabilities/sqli/source/impossible.php",
+    "line": 0,
+    "cap": "sqli",
+    "vuln": false
+  },
+  {
+    "path": "vulnerabilities/sqli/source/low.php",
+    "line": 0,
+    "cap": "sqli",
+    "vuln": true
+  }
+]

tests/eval_corpus/ground_truth/dvwa.manifest.toml

@@ -0,0 +1,84 @@
+# DVWA (Damn Vulnerable Web Application) — curated ground-truth manifest
+# (Phase 29, Track R.2).
+#
+# DVWA is an intentionally-vulnerable PHP app.  Unlike the other Track R
+# apps it ships its vulnerabilities as graded source variants under
+# vulnerabilities/<module>/source/{low,medium,high,impossible}.php, where
+# `low.php` is the textbook-vulnerable handler and `impossible.php` is the
+# hardened, secure rewrite of the SAME sink.  That gives DVWA real
+# vuln/benign PAIRS (low = vuln, impossible = benign control) the way OWASP
+# Benchmark does — so precision against this manifest is meaningful, not
+# just informational: a Confirmed finding on an `impossible.php` control is
+# a genuine false confirm.
+#
+# tests/eval_corpus/manifest_gt_convert.py turns this into the committed
+# ground_truth/dvwa.json.  CI regenerates it against a fresh clone of the
+# pinned tag and asserts byte-equality; the converter HARD-ERRORS on any
+# path that no longer exists, so a DVWA bump that restructures a module
+# fails loudly rather than silently dropping recall.  Re-pin `pinned_ref`
+# and re-validate the paths together.
+#
+# `cap` is a nyx cap label (tabulate.py), aligned to how nyx classifies the
+# sink.  `path` is relative to the DVWA clone root, POSIX separators.  Lang
+# is inferred from the extension (.php -> php).  See
+# tests/eval_corpus/budget.toml for the gate policy on these cells.
+
+corpus = "dvwa"
+upstream = "https://github.com/digininja/DVWA"
+# Pinned to release tag 2.5 (clone HEAD
+# a96943dc1f52f390ee5df72144660636c4b7dd06).  The
+# vulnerabilities/<module>/source/{low,impossible}.php layout has been stable
+# for years; re-validate if the tag is bumped.
+pinned_ref = "2.5"
+
+# ── SQL injection (sqli) ─────────────────────────────────────────────────────
+[[entry]]
+path = "vulnerabilities/sqli/source/low.php"
+cap = "sqli"
+vuln = true
+note = "id = $_REQUEST['id'] is concatenated straight into \"... WHERE user_id = '$id'\" and run via mysqli_query — classic SQL injection."
+
+[[entry]]
+path = "vulnerabilities/sqli/source/impossible.php"
+cap = "sqli"
+vuln = false
+note = "benign control: same query via PDO prepare + bindParam(:id, PDO::PARAM_INT) with is_numeric/intval validation — parameterized, not injectable."
+
+# ── OS command injection (cmdi) ──────────────────────────────────────────────
+[[entry]]
+path = "vulnerabilities/exec/source/low.php"
+cap = "cmdi"
+vuln = true
+note = "target = $_REQUEST['ip'] is concatenated into shell_exec('ping -c 4 ' . $target) with no validation — OS command injection."
+
+[[entry]]
+path = "vulnerabilities/exec/source/impossible.php"
+cap = "cmdi"
+vuln = false
+note = "benign control: the IP is split into 4 octets and each is_numeric-checked before being reassembled and passed to shell_exec — not injectable."
+
+# ── Open redirect (redirect) ─────────────────────────────────────────────────
+[[entry]]
+path = "vulnerabilities/open_redirect/source/low.php"
+cap = "redirect"
+vuln = true
+note = "header('location: ' . $_GET['redirect']) forwards to an unvalidated user-supplied URL — open redirect."
+
+[[entry]]
+path = "vulnerabilities/open_redirect/source/impossible.php"
+cap = "redirect"
+vuln = false
+note = "benign control: redirect target is chosen by an integer switch on is_numeric($_GET['redirect']) — no user-controlled URL reaches the Location header."
+
+# ── CRLF / HTTP header injection (header_injection) ──────────────────────────
+[[entry]]
+path = "vulnerabilities/open_redirect/source/low.php"
+cap = "header_injection"
+vuln = true
+note = "the same unvalidated $_GET['redirect'] flows into a raw header() call, so CRLF in the value splits/injects response headers — HTTP header injection."
+
+[[entry]]
+path = "vulnerabilities/open_redirect/source/impossible.php"
+cap = "header_injection"
+vuln = false
+note = "benign control: only a fixed, integer-selected target string reaches header() — no user bytes, so no CRLF injection."

tests/eval_corpus/ground_truth/gosec.json

@@ -0,0 +1,14 @@
+[
+  {
+    "path": "goanalysis/testdata/src/a/basic_output.go",
+    "line": 0,
+    "cap": "cmdi",
+    "vuln": true
+  },
+  {
+    "path": "goanalysis/testdata/src/a/basic_output.go",
+    "line": 0,
+    "cap": "crypto",
+    "vuln": true
+  }
+]

tests/eval_corpus/ground_truth/gosec.manifest.toml

@@ -0,0 +1,42 @@
+# gosec — curated Go ground-truth manifest (Phase 29, Track R.2).
+#
+# gosec is the Go SAST tool; its repo doubles as the de-facto Go security
+# corpus.  Most of gosec's rule samples live as Go source embedded in
+# backtick string literals inside testutils/g*_samples.go — those are NOT
+# scannable by a taint analyzer (the vulnerable code is string data, not
+# real AST), so they are deliberately NOT labelled here.  gosec also ships a
+# small set of REAL, compilable sample programs under goanalysis/testdata
+# that carry the tool's OWN inline `// want 'GNNN ...'` expectations — that
+# is the authoritative, scannable ground truth this manifest pins.
+#
+# Because the eval scans the whole gosec checkout (the tool's own source
+# included), unlabelled findings are expected and are NOT false positives —
+# precision against this manifest is informational, recall on the curated
+# samples is the meaningful floor (same policy as the all-vulnerable apps;
+# see tests/eval_corpus/budget.toml).
+#
+# tests/eval_corpus/manifest_gt_convert.py turns this into the committed
+# ground_truth/gosec.json.  CI regenerates it against a fresh clone of the
+# pinned tag and asserts byte-equality; the converter HARD-ERRORS on any
+# path that no longer exists, so a gosec bump that moves the testdata fails
+# the job loudly.  `cap` is a nyx cap label (tabulate.py); `path` is relative
+# to the gosec clone root, POSIX separators; lang is inferred (.go -> go).
+
+corpus = "gosec"
+upstream = "https://github.com/securego/gosec"
+# Pinned to release tag v2.26.1 (clone HEAD
+# 4a3bd8af174872c778439083ded7adbf3747e770).  goanalysis/testdata/src/a/ has
+# been stable; re-validate if the tag is bumped.
+pinned_ref = "v2.26.1"
+
+[[entry]]
+path = "goanalysis/testdata/src/a/basic_output.go"
+cap = "cmdi"
+vuln = true
+note = "VulnerableFunction runs exec.Command(\"sh\", \"-c\", getUserInput()) — subprocess launched with a non-constant argument (gosec's own `// want G204 [CWE-78]` expectation)."
+
+[[entry]]
+path = "goanalysis/testdata/src/a/basic_output.go"
+cap = "crypto"
+vuln = true
+note = "VulnerableFunction imports crypto/md5 and calls md5.New() — weak cryptographic primitive (gosec's own `// want G401/G501` expectations)."

tests/eval_corpus/ground_truth/railsgoat.json

@@ -0,0 +1,56 @@
+[
+  {
+    "path": "app/controllers/admin_controller.rb",
+    "line": 0,
+    "cap": "auth",
+    "vuln": true
+  },
+  {
+    "path": "app/controllers/benefit_forms_controller.rb",
+    "line": 0,
+    "cap": "deserialize",
+    "vuln": true
+  },
+  {
+    "path": "app/controllers/benefit_forms_controller.rb",
+    "line": 0,
+    "cap": "path_traversal",
+    "vuln": true
+  },
+  {
+    "path": "app/controllers/messages_controller.rb",
+    "line": 0,
+    "cap": "auth",
+    "vuln": true
+  },
+  {
+    "path": "app/controllers/password_resets_controller.rb",
+    "line": 0,
+    "cap": "crypto",
+    "vuln": true
+  },
+  {
+    "path": "app/controllers/password_resets_controller.rb",
+    "line": 0,
+    "cap": "deserialize",
+    "vuln": true
+  },
+  {
+    "path": "app/controllers/sessions_controller.rb",
+    "line": 0,
+    "cap": "redirect",
+    "vuln": true
+  },
+  {
+    "path": "app/controllers/users_controller.rb",
+    "line": 0,
+    "cap": "auth",
+    "vuln": true
+  },
+  {
+    "path": "app/models/user.rb",
+    "line": 0,
+    "cap": "crypto",
+    "vuln": true
+  }
+]

tests/eval_corpus/ground_truth/railsgoat.manifest.toml

@@ -0,0 +1,88 @@
+# OWASP RailsGoat — curated vuln ground-truth manifest (Phase 29, Track R.2).
+#
+# RailsGoat is an intentionally-vulnerable Ruby on Rails app that maps the
+# OWASP Top 10 to concrete controllers/models.  Like NodeGoat / Juice Shop
+# (Phase 28) it ships no machine-readable per-file vuln labels, so this
+# manifest IS the authoritative source: one [[entry]] per known-vulnerable
+# location, curated from the project's own tutorial walk-throughs, each with
+# a `note` citing why.
+#
+# tests/eval_corpus/manifest_gt_convert.py turns this into the committed
+# ground_truth/railsgoat.json.  CI regenerates it against a fresh clone of
+# the pinned tag and asserts byte-equality, and the converter HARD-ERRORS on
+# any path that no longer exists in the corpus, so a RailsGoat bump that
+# moves a controller fails the eval job loudly rather than silently dropping
+# recall.  Update `pinned_ref` + the paths together when re-pinning.
+#
+# `cap` is a nyx cap label (tabulate.py); it is aligned with how nyx
+# classifies the sink in each file (e.g. a missing ownership check on a
+# direct-object lookup surfaces as `auth`, not `unauthorized_id`), so recall
+# (did nyx catch the canonical vuln) is meaningful.  `path` is relative to
+# the RailsGoat clone root, POSIX separators.  Lang is inferred from the
+# extension (.rb -> ruby).  All `vuln = true`: RailsGoat is all-vulnerable,
+# so there is no benign-control file to pair against — precision vs this
+# manifest is informational (an unlabelled finding may be a real uncurated
+# vuln), while recall is the meaningful floor.  See
+# tests/eval_corpus/budget.toml for how the gate treats these cells.
+
+corpus = "railsgoat"
+upstream = "https://github.com/OWASP/railsgoat"
+# Pinned to the stable Rails 5 release tag (clone HEAD
+# 0766ca80bf2d94acbde1dd4aaf7baf9b86afe4eb).  The app/controllers + app/models
+# layout below has been stable across this tag; re-validate the paths if the
+# ref is bumped.
+pinned_ref = "rails.5.0.0"
+
+[[entry]]
+path = "app/controllers/users_controller.rb"
+cap = "auth"
+vuln = true
+note = "update looks up the account with User.where(\"id = '#{params[:user][:id]}'\") and mass-assigns user_params (params.require(:user).permit!) with no ownership check — broken access control / mass-assignment privilege escalation (OWASP A4/A5)."
+
+[[entry]]
+path = "app/controllers/messages_controller.rb"
+cap = "auth"
+vuln = true
+note = "show / destroy fetch Message.where(id: params[:id]) with no check that the message belongs to current_user — insecure direct object reference (OWASP A4 broken access control)."
+
+[[entry]]
+path = "app/controllers/admin_controller.rb"
+cap = "auth"
+vuln = true
+note = "administrative actions are gated by a bypassable admin_param check (params[:admin_id] != \"1\"); update_user / delete_user act on any admin_id — broken access control / privilege escalation (OWASP A5)."
+
+[[entry]]
+path = "app/models/user.rb"
+cap = "crypto"
+vuln = true
+note = "passwords are hashed with Digest::MD5.hexdigest (hash_password / authenticate) — unsalted weak hash for credential storage (OWASP A2 cryptographic failure)."
+
+[[entry]]
+path = "app/controllers/password_resets_controller.rb"
+cap = "crypto"
+vuln = true
+note = "generate_token derives the reset token as Digest::MD5.hexdigest(email) — a predictable, forgeable password-reset token (weak cryptography)."
+
+[[entry]]
+path = "app/controllers/password_resets_controller.rb"
+cap = "deserialize"
+vuln = true
+note = "reset_password runs Marshal.load(Base64.decode64(params[:user])) on attacker-controlled input — insecure deserialization leading to RCE (OWASP A8)."
+
+[[entry]]
+path = "app/controllers/sessions_controller.rb"
+cap = "redirect"
+vuln = true
+note = "create redirects to params[:url] with no allow-list (path = params[:url] then redirect_to path) — open redirect (OWASP unvalidated redirects)."
+
+[[entry]]
+path = "app/controllers/benefit_forms_controller.rb"
+cap = "path_traversal"
+vuln = true
+note = "download builds send_file from a user-controlled params[:name] path with no containment — arbitrary file read / path traversal."
+
+[[entry]]
+path = "app/controllers/benefit_forms_controller.rb"
+cap = "deserialize"
+vuln = true
+note = "download calls params[:type].constantize.new(path), constantizing a user-supplied class name — unsafe reflection / object injection."

tests/eval_corpus/ground_truth/rustsec.json

@@ -0,0 +1 @@
+[]

tests/eval_corpus/ground_truth/rustsec.manifest.toml

@@ -0,0 +1,37 @@
+# RustSec advisory-db — Rust negative-control corpus (Phase 29, Track R.2).
+#
+# The plan's Rust real-corpus row is the RustSec advisory database.  Unlike
+# RailsGoat / DVWA / DVPWA / gosec, advisory-db ships advisory METADATA
+# (TOML + Markdown under crates/<crate>/RUSTSEC-*.md), not vulnerable Rust
+# SOURCE.  A static scan of it therefore contains zero `.rs` files and nyx
+# correctly produces zero findings — so there are no source-level vuln
+# positives to label, and no canonical scannable "RustGoat" exists to
+# substitute without fabricating paths (which the CI byte-equality + path
+# existence guards would reject outright).
+#
+# advisory-db is still worth pinning and scanning as a NEGATIVE CONTROL for
+# the Rust language path:
+#   * it exercises the Rust scan + verify pipeline (Phase 23 Rust build
+#     pool) end to end on a large real-world tree (thousands of files) and
+#     asserts it stays within the wall-clock budget without crashing, and
+#   * it is an over-confirmation guard: nyx must Confirm NOTHING on a corpus
+#     with no real source vulns.  Any Confirmed finding here is provably a
+#     false confirm and trips the per-cell false_confirmed_rate budget
+#     (tests/eval_corpus/budget.toml) — a genuine regression sentinel if a
+#     future change makes nyx treat advisory text as scannable code.
+#
+# `negative_control = true` tells manifest_gt_convert.py to emit an empty
+# `[]` ground truth.  It is mutually exclusive with `[[entry]]` tables, so a
+# real Rust vuln can never be silently hidden behind the flag.  When a
+# scannable advisory-backed Rust corpus (a vulnerable crate pinned at its
+# affected version with a source-level taint sink) is curated, drop the flag
+# and add [[entry]] tables here exactly as the other Track R.2 manifests do.
+
+corpus = "rustsec"
+upstream = "https://github.com/rustsec/advisory-db"
+# advisory-db publishes no release tags; the eval job pins the default
+# branch via the CI cache key (clone HEAD
+# eaf48e749baa3d5e27d304107d8abf175fd756bb).
+pinned_ref = "main"
+
+negative_control = true