apunkt/nyx
1
0
Fork 0
mirror of https://github.com/elicpeter/nyx.git synced 2026-06-06 19:35:13 +02:00
Code Issues Projects Releases Packages Wiki Activity
nyx/docs/recall-validation.md

237 lines
11 KiB
Markdown
Raw Blame History

# Recall validation runbook
The recall-validation harness freezes a finding-shape baseline against
real-world OSS targets so future engine work can prove "actually lifts
recall on real code", not just "tests pass". This runbook covers
re-running the validation against a fresh OSS release.
## Targets
| Target | Clone URL | Recall items exercised |
|-------------------|--------------------------------------------|------------------------|
| `cal_com` | https://github.com/calcom/cal.com | 1, 5, 6, 7 |
| `vercel_commerce` | https://github.com/vercel/commerce | 1, 4, 7 |
| `shadcn_examples` | https://github.com/shadcn-ui/ui | 4, 7 |
| `blitz_apps` | https://github.com/blitz-js/blitz | 1, 3, 6 |
Item numbering is from `.pitboss/RECALL_GAPS.md`.
## Files
| File | Role |
|-----------------------------------------------|-----------------------------------------|
| `scripts/validate_recall.sh` | runner (capture + diff modes) |
| `tests/recall_targets/<target>.json` | per-target baseline |
| `tests/recall_gaps.rs::validate_real_world_targets` | schema-validity test (`#[ignore]`)|
| `tests/recall_gaps_baseline.json` | corpus regression baseline |
Baselines live next to the harness rather than under `.pitboss/`:
pitboss implementer agents are forbidden to write under `.pitboss/`,
so the baseline files were placed beside the test that consumes them.
## Baseline schema
```json
{
"_doc": "...",
"target": "cal_com",
"clone_url": "https://github.com/calcom/cal.com",
"exercises_recall_items": [1, 5, 6, 7],
"captured_against": "real-scan @ <sha>",
"captured_on": "YYYY-MM-DD",
"pinned_commit": "<sha>",
"findings": [
{
"rule_id": "taint-unsanitised-flow",
"path_suffix": "packages/...",
"line": 130,
"severity": "High",
"verdict": "TP" | "FP" | "needs_review",
"note": "..."
}
]
}
```
The diff key is `(rule_id, path_suffix, line)`. The `verdict` field
must be one of `TP`, `FP`, or `needs_review`; unknown verdicts are
rejected by the schema test.
## Usage
### Diff a fresh scan against the frozen baseline
```bash
scripts/validate_recall.sh cal_com /path/to/cal.com
```
Output is a JSON object `{ added, removed, unchanged, *_total }`
keyed by `rule_id`. Use this to spot intentional recall lift
(`added`) and regressions (`removed`).
### Refresh the baseline after an intentional recall lift
```bash
scripts/validate_recall.sh cal_com /path/to/cal.com --capture
```
This overwrites `tests/recall_targets/cal_com.json` with the current
scan output. Every finding is re-marked `verdict: "needs_review"`;
hand-label `TP`/`FP` afterwards as you triage.
### Schema-validity check
```bash
cargo test --release --test recall_gaps -- --ignored validate_real_world_targets
```
Loads each per-target JSON, asserts the required keys exist, and
asserts every finding carries a valid verdict label.
## Refresh procedure
1. Clone or pull the target repo into `~/oss/<target>` (or wherever).
2. Build nyx: `cargo build --release`.
3. Run the diff in plain mode to see what changed:
`scripts/validate_recall.sh <target> ~/oss/<target>`.
4. If the lift is intentional, recapture:
`scripts/validate_recall.sh <target> ~/oss/<target> --capture`.
5. Spot-check a handful of new findings. Open the file at
`path_suffix:line` and confirm the source-to-sink flow is real.
Hand-label them `TP`/`FP`.
6. Commit the updated `tests/recall_targets/<target>.json`.
## Known captured baselines (2026-05-08)
| Target | Pinned commit | Findings | TP | FP | needs_review |
|-------------------|---------------|----------|----|----|--------------|
| `cal_com` | `d278d6c9` | 662 | 0 | 4 | 658 |
| `vercel_commerce` | unknown | 0 (placeholder) | | | |
| `shadcn_examples` | unknown | 0 (placeholder) | | | |
| `blitz_apps` | unknown | 0 (placeholder) | | | |
The `cal_com` capture used commit `d278d6c9bc535bf3f2c6ba0607654f78dd74d6ee`
(`refactor: remove dead insights references (#29029)`). The 4 `FP`
labels are `ts.crypto.math_random` hits inside `apps/web/playwright/`
test fixtures, which are not a security context.
The other three targets ship as placeholders (empty `findings`).
Nobody has cloned them locally yet. Run `validate_recall.sh
<target> <clone> --capture` to populate. The schema test still passes
because `[]` is a valid `findings` array with zero entries to check.
## Perf baseline
The frozen JS-target perf snapshot lives in
`tests/recall_targets/perf_after.txt`. Compare against the
`captured_against` snapshot in `tests/recall_gaps_baseline.json`
(`corpus_finding_lines.findings_total` = 1121, captured at master
`ea82ea98`). The acceptance bar: scanner throughput on the existing
`tests/fixtures/` corpus must regress by no more than 15%. Future
recall work uses the same corpus and the same record file to measure
its own perf delta.
## Cross-language runbook
The JS-target baselines above only cover JS/TS. Cross-language
baselines mirror that work against real-world non-JS targets so
multi-language engine changes can be measured against actual code,
not just synthetic fixtures. Per-lang baselines live under
`tests/recall_targets/xlang/<lang>/<target>.json` and the runner
accepts a `--lang` flag to select the target set.
### Cross-language targets
| Lang | Target | Clone URL | Pinned commit (capture) | Findings | Notes |
|--------|--------------|----------------------------------------------|-------------------------|----------|-------|
| php | phpmyadmin | https://github.com/phpmyadmin/phpmyadmin | `ddf4e993` | 119 | DBA UI; XSS / `php.deser` / `cfg-unguarded-sink` heavy. |
| php | joomla | https://github.com/joomla/joomla-cms | `7e8527d0` | 83 | CMS; `php.deser.unserialize` and `php.path.include_variable` clusters. |
| php | drupal | https://github.com/drupal/drupal | `92aa759e` | 635 | CMS / DI container; `cfg-unguarded-sink` (198) and `taint-prototype-pollution` (121) dominant. |
| php | nextcloud | https://github.com/nextcloud/server | `5c0fe4c3` | 262 | File-sync platform; `cfg-resource-leak` / `state-resource-leak` heavy. |
| java | openmrs | https://github.com/openmrs/openmrs-core | `f9c76db2` | 273 | Hibernate-heavy; JPA Criteria fix from `project_realrepo_openmrs.md` already applied. |
| python | airflow | https://github.com/apache/airflow | `3d42610a` | 892 | Scheduler / DAG runner; `cfg-unguarded-sink` (252) and `taint-unsanitised-flow` (179) lead. |
| python | flask | https://github.com/pallets/flask | placeholder | 0 | Smaller-surface Python framework; capture deferred. |
| go | gin | https://github.com/gin-gonic/gin | `d3ffc998` | 20 | HTTP framework test corpus; `taint-header-injection` and TLS skip-verify in tests. |
| rust | axum | https://github.com/tokio-rs/axum | placeholder | 0 | Not cloned in pitboss sandbox at capture time; populate locally. |
| ruby | rails | https://github.com/rails/rails | placeholder | 0 | Capture against the `actionpack/` subtree once cloned. |
Captures dated `2026-05-09` (UTC). Counts are deduplicated tuples
`(rule_id, path_suffix, line)`. Duplicate raw findings collapse on
the diff key, so the schema-test count and diff-mode `unchanged_total`
may differ from the `findings | length` total by a handful of
duplicate sites. The diff key is what matters for regression
detection.
### Per-lang TP/FP splits
Every captured finding ships with `verdict: "needs_review"` from
`--capture`. Hand-triage is bounded but pending; none of the cross-
language captures are sweep-labelled yet. Use the per-lang dominant
rule_id clusters above as the priority queue:
- **PHP**: `cfg-unguarded-sink` and `taint-prototype-pollution` are
the FP-dominant clusters across drupal / nextcloud / phpmyadmin
(CMS routing + JS object construction). `php.deser.unserialize` is
the highest-value TP cluster on joomla (17) and drupal (83). See
`project_realrepo_joomla.md` 2026-05-03 for the magic-method
passthrough fix that already filters one shape.
- **Java**: `taint-unsanitised-flow` (61) and `state-resource-leak`
(60) are openmrs's leading clusters. The JPA Criteria-API fix
already absorbed the `cfg-unguarded-sink` cluster (216 to 24);
remaining Hibernate / Spring resource-management FPs are the next
triage target.
- **Python**: `cfg-unguarded-sink` (252) on airflow is dominated by
Airflow's scheduler / DB plumbing; `py.auth.token_override_*`
(83) and `py.auth.missing_ownership_check` (61) are the auth-rule
noise typical of an admin/operator codebase.
- **Go**: gin's 20 findings are mostly test-corpus artifacts
(`gin_test.go`, `routes_test.go`); 4 of 4 `go.transport.insecure_skip_verify`
hits are inside `gin*_test.go` and are legitimate test setup.
- **Rust / Ruby**: placeholder. Capture once a local clone exists.
### `--lang` runner usage
```bash
# diff mode (default)
scripts/validate_recall.sh --lang php drupal /Users/me/oss/drupal
scripts/validate_recall.sh --lang java openmrs /Users/me/oss/openmrs
# capture / refresh
scripts/validate_recall.sh --lang go gin /Users/me/oss/gin --capture
```
Output is the same `{ added, removed, unchanged, *_total }` JSON shape
as the JS-target diff. The diff key is `(rule_id, path_suffix, line)`.
### Cross-language refresh procedure
1. Clone or update the target into `~/oss/<target>` (or wherever).
2. Build nyx: `cargo build --release`.
3. Diff vs the frozen baseline:
`scripts/validate_recall.sh --lang <lang> <target> ~/oss/<target>`.
4. If the lift is intentional, recapture with `--capture`.
5. Spot-check new findings; hand-label `TP`/`FP`.
6. Commit the updated `tests/recall_targets/xlang/<lang>/<target>.json`.
### Sandbox-capture caveat
Pitboss implementer agents run sandboxed without network egress, so
target repos that are not already present under `~/oss/` ship as
placeholders (`pinned_commit: "unknown"`, `findings: []`). The
current cross-language baselines cover php / java / python / go
(every target whose repo was already cloned locally) and ship
placeholders for `rust/axum`, `ruby/rails`, and `python/flask`. The
schema test in `validate_real_world_targets` passes against
placeholders because `[]` is a valid `findings` array.
## What lives where (quick reference)
- Targets list and recall-item mapping in this file.
- Per-target JS findings under `tests/recall_targets/<target>.json`.
- Per-target cross-lang findings under `tests/recall_targets/xlang/<lang>/<target>.json`.
- Diff/capture runner at `scripts/validate_recall.sh` (accepts `--lang`).
- Schema-validity test at `tests/recall_gaps.rs::validate_real_world_targets`.
- Corpus regression baseline at `tests/recall_gaps_baseline.json`.
- Perf records at `tests/recall_targets/perf_after.txt` (JS-target
snapshot) and `tests/recall_targets/perf_after_xlang.txt`
(cross-language delta).
Reference in a new issue View git blame Copy permalink