mirror of
https://github.com/elicpeter/nyx.git
synced 2026-06-09 19:45:13 +02:00
* feat: Introduce function-scoped variable interning for state analysis with new tests and fixtures * feat: Add Phase 26 symbolic execution enhancements with bitwise operator support, abstract interpretation refinements, and new taint analysis tests * feat: Refine state analysis to handle factory-pattern resource returns with mixed-path tests and leak detection enhancements * feat: Add Phase 27 debug views with symbolic execution, abstract interpretation, SSA, and call graph viewers; integrate with debug layout and styles * feat: Add Phase 31 type-qualified symbolic resolution with receiver-based callee disambiguation and testing * feat: Extend symbolic execution with state iteration, enhanced debug views, and debounced input handling * feat: Add Phase 13 resource and auth pattern extensions with new tests and fixtures * feat: Introduce CFG debug graph renderer with compact mode, toolbar, and DAG layout integration * feat: Add Phase 28 encoding and decoding transform modeling with structural symex enhancements and new taint analysis tests * feat: Extend abstract interpretation with type facts and constant value tracking in debug views and server logic * feat: Add linear path handling and witness extraction to symbolic execution with Phase 28 transform mismatch detection * feat: Refine Go auth and sanitizer handling with enhanced rules, state updates, and benchmark improvements * feat: Enable auth-state analysis by default and update relevant tests in benchmark config * test: Update state_tests to reflect default enablement of auth-state analysis and add auth suppression test * docs: update CHANGELOG.md * feat: Introduce per-index taint tracking in `HeapState` with `HeapSlot`, overflow handling, and revised SSA transfers * feat: Introduce C/C++ language labels and refine heap state tracking in SSA transfers * feat: Implement per-index array slot tracking in symbolic heap with overflow collapse * feat: Add implicit definition handling for uninitialized declarations in SSA value allocation * feat: Refactor function parameters and constants for improved clarity and maintainability * refactor: Reorder module imports and improve formatting for consistency * refactor: Fix formatting erorrs * refactor: Fix clippy warnings * refactor: Fix fmt warnings (again) * chore: Update dependencies and improve feature configuration * Add comprehensive tests for undertested modules (#36) (COPILOT) * Add comprehensive tests for undertested modules Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> Agent-Logs-Url: https://github.com/elicpeter/nyx/sessions/f3fc877e-f386-49ba-9793-fc93d3805083 * Add comprehensive tests for ext, project, walk, and errors modules Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> Agent-Logs-Url: https://github.com/elicpeter/nyx/sessions/f3fc877e-f386-49ba-9793-fc93d3805083 --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> * chore: Update dependencies and improve feature configuration * fix: formatting errors in new tests * chore: Update license list in about.toml * chore: made functions input inline * chore: updated cfg graph to take up the full page * chore: add Prettier configuration and update code formatting * Add frontend test suite with Vitest (111 tests) (#37) * Add Vitest test suite for frontend - 111 tests across utils, components, hooks, and graph utilities Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> Agent-Logs-Url: https://github.com/elicpeter/nyx/sessions/7cf0dba2-ecff-4740-ba4d-92717e74a0b7 * ci: add frontend test step to CI workflow Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> Agent-Logs-Url: https://github.com/elicpeter/nyx/sessions/5bc0ac9f-0a32-4d03-9cb7-7a15aea53fca --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> * chore: simplify array initialization in test files for consistency * ran typecheck * feat: add AnalysisWorkspace component and integrate it into CfgViewerPage * feat: update routing in AppLayout and improve empty state message in ExplorerPage * feat: enhance scan progress tracking with additional metrics and stages * feat: update license information and add license check script * feat: implement cross-file symbolic execution with callee body persistence * feat: replace dagre graphs with Graphology + ELK + Sigma for more advanced call stack and cfg rendering * feat: ensure CFG function view is scoped to the selected function, preventing bleed into sibling functions * feat: enhance resource tracking with proxy method summaries and improve finding extraction * feat: add terminal function exit detection for accurate resource leak analysis * feat: add warnings for loops and functions without bodies to improve error recovery * feat: update lambda expression handling to ensure proper function classification and control flow * feat: remove bounded formatting/string ops and add JSON.parse sanitizer for improved data handling * feat: add inline return taint analysis and regression tests for improved security checks * feat: add engine version management and migration handling for database schema updates * feat: enhance first_call_ident to skip nested function bodies and add regression tests * feat: enhance callee name resolution with two-segment normalization and disambiguation * feat: add cross-file context flags and debug assertions for taint analysis * feat: refactor taint analysis structure to unify context handling and improve clarity * feat: enhance dead code elimination to preserve Sink, Source, and Sanitizer labels with new tests * docs: updated CHANGELOG.md * fmt: formatting fixes * fix: fixed frontend formatting and lint warnings * fix: optimized ci * fix: optimized ci * Add comprehensive multi-file test coverage to Nyx (#38) * Initial checklist for multi-file test suite expansion Agent-Logs-Url: https://github.com/elicpeter/nyx/sessions/e550cb88-9767-4442-94d4-101bf5bb0e23 Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> * Add 12 new multi-file test fixtures with TP/TN/near-miss coverage Agent-Logs-Url: https://github.com/elicpeter/nyx/sessions/e550cb88-9767-4442-94d4-101bf5bb0e23 Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> * deleted root repo * rebuilt to test for regressions --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> Co-authored-by: elipeter <elicpeter@gmail.com> * feat: enhance import alias resolution and taint tracking * feat: implement security hardening with CSRF protection and path validation * feat: add support for import alias bindings in Python, PHP, and Rust * feat: enhance CFG analysis modes and improve code readability * feat: add detection for parameterized SQL queries to enhance security * feat: add safe internal redirect handling and enhance session destroy validation * feat: implement security improvements by addressing vulnerabilities in execAsync, session management, and file downloads * feat: enhance taint detection by adding support for inline source member expressions in call arguments * feat: implement pre-emission of Source nodes for inline source member expressions in call arguments * feat: add support for Throw statement in control flow and error handling * feat: add debug and echo endpoints with potential information leakage * feat: implement internal redirect suppression and enhance taint detection * feat: implement module alias tracking for dynamic dispatch in JS/TS * feat: add authorization analysis module with Express support * feat: add authorization analysis module with Express support * feat: add tests for admin guard requirements and clean checks in authorization analysis * feat: integrate Koa and Fastify frameworks into authorization analysis * feat: add Flask and Django support to authorization analysis module * feat: add support for Rails and Sinatra frameworks in authorization analysis * feat: add support for Axum, ActixWeb, and Rocket frameworks in authorization analysis * feat: add support for ActixWeb, Axum, and Rocket frameworks in authorization analysis * feat: add support for Rails and Sinatra in authorization analysis * chore: add .DS_Store to .gitignore * refactor: simplify conditional checks and improve readability in multiple files * refactor: update usage of Option methods for improved clarity and consistency * refactor: improve code readability by simplifying conditional checks and formatting * refactor: improve code formatting and readability by simplifying conditional checks * refactor: simplify conditional checks and improve readability in multiple files * refactor: simplify conditional checks in axum.rs for improved readability * feat: add CodeQL analysis configuration for enhanced security scanning * test: add comprehensive tests for `src/output.rs` SARIF builder (#39) * chore: start test coverage improvement work Agent-Logs-Url: https://github.com/elicpeter/nyx/sessions/cd7ff398-134e-4728-a5e7-0353a0744423 Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> * test: add comprehensive tests for src/output.rs SARIF builder Agent-Logs-Url: https://github.com/elicpeter/nyx/sessions/cd7ff398-134e-4728-a5e7-0353a0744423 Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> * refactor: improve code formatting and readability in output.rs --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> Co-authored-by: elipeter <elicpeter@gmail.com> * refactor: improve code formatting and readability in output.rs * Potential fix for code scanning alert no. 210: Uncontrolled data used in path expression Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Potential fix for code scanning alert no. 211: Uncontrolled data used in path expression Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * refactor: enhance triage file path handling with improved error management and validation * refactor: updated func summaries for richer detail * refactor: update SSA summary extraction to use canonical FuncKey for distinct entries * refactor: enhance callee metadata structure to support arity, receiver, and qualifier for better overload resolution * refactor: add support for keyword arguments in function calls and enhance receiver extraction for method-style calls * refactor: implement new Flask routes for safe and unsafe shell command execution * refactor: separate receiver handling in SSA operations and enhance taint propagation * refactor: improve arity handling by using arg_uses for positional argument count and enhance witness scoring for tainted arguments * refactor: implement auth decorator extraction and classification for multiple languages * refactor: enhance Rust module path resolution and use map handling for cross-file disambiguation * refactor: introduce CalleeQuery struct for structured callee resolution and enhance resolver logic * refactor: implement same-file identity collision handling for `runTask` to ensure correct resolver behavior * refactor: standardize default struct initialization across multiple files * feat: add scripts for formatting checks and auto-fixes with test summaries * refactor: simplify character splitting and enhance namespace qualifier handling * refactor: improve documentation clarity and enhance code readability in resolver logic * refactor: replace default struct initialization with explicit field assignments for clarity * feat: enhance anonymous function naming by deriving context-based bindings * refactor: streamline match expressions for improved readability and performance * refactor: streamline match expressions for improved readability and performance * refactor: replace loop with while let for improved clarity and performance * feat: add SSA constant propagation support to analysis context for improved accuracy * feat: add SSA constant propagation support to analysis context for improved accuracy * feat: implement shell metacharacter validation and bounded-length checks in Rust analysis * feat: add static map analysis for command injection suppression and type safety * refactor: simplify match statements and reduce line breaks for improved readability * feat(summary): phase 1/5 SinkSite data model for primary sink-location attribution Introduce SinkSite (file_rel, line, col, snippet, cap) carrying the primary sink source-location through function summaries. Swap SsaFuncSummary.param_to_sink and FuncSummary.param_to_sink from a coarse Cap map to a deduped SmallVec<[SinkSite; 1]> per parameter, with a backward-compatible cap_sites() helper and serde defaults so pre-phase-1 on-disk rows continue to deserialise cleanly. Extraction: SinkSiteLocator bundles the tree/bytes/file_rel needed by extract_ssa_func_summary; ParsedFile::extract_ssa_artifacts wires the locator in for the persisted pass-1 path, while pass-2 intra-file transient summaries fall back to cap-only sites (behavior unchanged). Merge: GlobalSummaries::insert now unions sink sites with (file_rel, line, col, cap) dedup via shared union_param_sink_sites helper. Database: JSON-serialised summary columns carry the new shape automatically; no schema change needed. Phase 2 will consume SinkSite in build_taint_diag() to overwrite the caller-site Finding.line with the callee's sink line when resolved via summary. Phase 1 keeps behavior unchanged: scanning tests/benchmark/corpus/rust/cmdi/cmdi_indirect.rs still produces the same (wrong) line 10 finding. Adds round-trip tests covering SinkSite solo, SsaFuncSummary with sink sites, legacy-JSON default handling for both summary types, and merge dedup. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(taint): phase 2/5 thread SinkSite into SsaTaintEvent and Finding Plumb Phase 1's SinkSite through the event pipeline into Findings, no output change yet. SsaTaintEvent gains `primary_sink_site: Option<SinkSite>`; when the main or callback sink-emission path has non-empty `param_to_sink_sites`, filter to sites whose `(line != 0) && (cap ∩ sink_caps != ∅)` and emit one event per distinct site — the multi-primary collapse keeps each downstream Finding single-primary. Resolution: ResolvedSummary and SinkInfo gain mirror `param_to_sink_sites` fields, populated from `SsaFuncSummary.param_to_sink` (SSA + callback paths) and `FuncSummary.param_to_sink` (global paths). Label, local-summary, and interop resolution paths leave the field empty — they only ever had cap-level info to begin with. Finding: new `primary_location: Option<SinkLocation>` with `file_rel/line/col`. `ssa_events_to_findings` maps `event.primary_sink_site` → `Finding.primary_location`, filtering cap-only sites (`line == 0`) to `None` so the (0,0) sentinel never leaks to formatters. Dedup key extended with the primary location so multi-site events aren't collapsed back together. Invariants (debug_assert!): * every SinkSite reaching emission has `line != 0 && cap ∩ sink_caps != ∅` — enforced by the pick_primary_sink_sites* filters; * every populated Finding.primary_location has `line != 0` AND non-empty `file_rel` — the cap-only → None translation upstream guarantees this. Deliberately independent of `uses_summary`: that flag tracks whether the *taint chain* used a summary, whereas primary attribution requires only that the *sink* itself was summary-resolved. A local source reaching a cross-file sink produces `uses_summary=false` alongside a populated primary_location — documented on Finding.primary_location, covered by `cross_file_sink_finding_carries_primary_location`. build_taint_diag, SARIF/JSON/explanation formatters, and the benchmark scorer remain untouched: finding.line still comes from `cfg_graph[finding.sink]`, so cmdi_indirect.rs still reports line 10 and the benchmark's rs-cmdi-003 row still shows FN in the LOC column. Tests: `cross_file_sink_finding_carries_primary_location` (proves plumbing via a synthetic FuncSummary carrying a SinkSite at 42:5) and `cross_file_sink_cap_only_site_leaves_primary_location_none` (regression guard against cap-only sites surfacing). All 1566 lib tests + integration tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(output): phase 3/5 consume primary sink location in diag + SARIF When a finding's primary_location (populated in phase 2 from a callee summary's SinkSite) names the dangerous instruction inside a callee body, attribute the diagnostic line to that location instead of the caller's call site. The call site is demoted to a Call step in flow_steps, and a synthetic Sink step at the primary location is appended so analysts still see the full trace. Changes: - Add scan_root parameter to build_taint_diag so file_rel can be resolved back to an absolute path via a shared resolve_file_rel helper. Empty file_rel (single-file scans where namespace == "") resolves to the file under analysis. - Extend SinkLocation with snippet, carried from the upstream SinkSite so the formatter needs no second file read. - Relax the ssa_events_to_findings debug_assert to allow empty file_rel, which is valid when scan root equals the file itself. - SARIF: emit data-flow as codeFlows[0].threadFlows[0].locations[]; locations[0] already reflects the primary sink position via the updated diag line/col. Acceptance: scan on tests/benchmark/corpus/rust/cmdi/cmdi_indirect.rs now reports line 5 (Command::new) as the primary sink, with the call site at line 10 visible in flow_steps. Two expect.json fixtures updated (must_match line_range widened): - javascript/taint/context_sensitive_call: 12-14 -> 7-14 (line 8 is the real sink inside run()). - rust/cfg/closure_async: 10-10 -> 10-11 (line 11 is Command::new inside the closure). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(bench): phase 4/5 validate primary sink attribution across corpus Extend the benchmark scorer and ground truth to lock in phase 3's primary-location behavior, and add fixtures that exercise the new capability end-to-end. Scorer (tests/benchmark_test.rs): - Add optional `expected_call_site_lines: Option<Vec<[usize; 2]>>` on Case. When present, score_location_level additionally requires at least one flow_step in the finding's evidence trace to fall within ±2 of the call-site range. When absent, the check is skipped — fully forward-compatible with existing fixtures. - Retain ±2 tolerance on expected_sink_lines (compared against the now-primary Diag.line post-phase-3). Ground truth edits: - rs-cmdi-cross-001: expected_sink_lines [8,8] -> [9,9]. Line 8 is the transform::wrap call site (a cross-file propagator, not a sink); line 9 is Command::new, the real sink. The ±2 tolerance happened to mask this stale attribution but it was semantically wrong — phase 4 is the right time to correct it. Also adds expected_call_site_lines [8,8] so the new field is exercised on an existing cross-file case. - rs-cmdi-003: adds expected_call_site_lines [10,10] (run_cmd call). This fixture's sink (Command::new inside run_cmd at line 5) was the motivating case for phases 1-3; adding the call-site assertion guards against regression to caller-line attribution. New fixtures: - rust/cmdi/cmdi_indirect_multisink.rs (rs-cmdi-009): helper run_both takes two tainted params and invokes two Command sinks on consecutive lines. Locks in that primary line lands inside the helper (lines 5-6), not at the caller (line 12). Notes document that SinkSite is currently one-per-callee so both findings today collapse onto the first sink; expected_sink_lines=[5,6] and expected_call_site_lines=[12,12] stay valid either way. - python/cmdi/cross_indirect_sink/{app.py,helper.py} (py-cmdi-cross- 004): sink os.system lives in helper.py (cross-file), caller in app.py reads env source and calls run_cmd. Verifies phase 3's cross-file primary attribution: Diag.path = helper.py, Diag.line = 5, with app.py:7 recorded in flow_steps as a Call step. Acceptance: - `cargo test --test benchmark_test -- --ignored --nocapture` passes. - rs-cmdi-003 is TP/TP/TP (the target flip FN->TP at LOC). All pre-existing TP/TP/TP fixtures remain TP/TP/TP; 2 new fixtures are TP/TP/TP. - Aggregate rule-level: TP=158 FP=10 FN=1 TN=97, P=0.940 R=0.994 F1=0.966 on the 266-case corpus (was TP=156 FP=10 FN=1 TN=97 on 264 pre-phase-4, delta is the +2 new cases both resolving TP). - Full `cargo test` green (1566 lib tests + all integration tests). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(taint): phase 5/5 lock Finding.primary_location contract via regression test Add a regression test in src/taint/ssa_transfer.rs that wires up a synthetic SsaFuncSummary with a SinkSite at other.rs:42:10 and drives the three emission stages (pick_primary_sink_sites → emit_ssa_taint_events → ssa_events_to_findings) against a minimal caller SSA body. Asserts the resulting Finding.primary_location is exactly that triple. The existing integration tests in src/taint/tests.rs cover the coarse FuncSummary path end-to-end through analyse_file. This test locks in the lower-level SSA-side plumbing so a future refactor that silently drops the site between pick → emit → findings fails here rather than only at the benchmark layer. Also refreshes tests/benchmark/results/latest.json (timestamp only; rs-cmdi-003 remains TP/TP/TP and the aggregate P/R/F1 are unchanged from phase 4). Closes the primary sink-location attribution feature (phases 1-5/5): * Phase 1 — SinkSite data model on summaries. * Phase 2 — SinkSite threaded into SsaTaintEvent and Finding. * Phase 3 — diag + SARIF consume primary_location. * Phase 4 — benchmark validates primary_call_site_lines across corpus. * Phase 5 — regression test locks the event→finding contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: clean up formatting and improve readability in multiple files * refactor: simplify type definition for deduplication key in findings * test(harness): add must_not_match expectation for FP regression guards Extends ExpectedFinding with must_not_match field that asserts a diagnostic must NOT fire — presence is a hard failure. Non-consuming scan so it coexists with must_match entries on the same rule_id. Adds forbidden_violations accumulator and updates summary line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(regression): update expectations to ensure must_not_match for various taint and resource leak rules * feat: implement auto-seeding for JS/TS handler parameters to enhance taint tracking * feat: update switch statement handling to improve control flow analysis * feat: implement promisify alias handling for JS/TS to enhance taint tracking * feat: enhance taint tracking by refining expectation handling and adding mode filtering * feat: refine SQL handling in stream processing and enhance auto-seeding for handler parameters * feat: update taint tracking rules to enforce full mode matching and improve flow analysis * feat: enhance Ruby subshell handling to improve taint tracking and flow analysis * feat: update xss_response expectations to refine taint flow analysis and enhance regression guarding * feat: refine framework detection and update expectation handling for Echo and Sinatra * feat: implement max_count for taint tracking expectations and deduplicate findings * feat: add strict_unexpected handling for taint-unsanitised-flow in expectation files * feat: enhance deduplication of taint-unsanitised-flow findings by collapsing based on line and severity * feat: add strict_unexpected handling for taint-unsanitised-flow in multiple expectation files * feat: add structural invariant checks for SSA bodies * feat: ensure deterministic phi emission order using BTreeSet * feat: enhance handling of terminators to ensure authoritative flow through successor edges * feat: enhance Goto terminator handling to ensure all successors are marked executable * feat: refactor code for improved readability and organization * feat: simplify predicate checks and enhance readability in SSA handling * feat: implement per-file parse timeout and enhance file size handling * feat: migrate analysis engine toggles from environment variables to configuration file * feat: remove unnecessary whitespace in hostile_input_tests.rs * feat: remove unnecessary whitespace in hostile_input_tests.rs * feat: update dependencies and enhance documentation on language maturity * feat: enhance security headers and improve request body limits * feat: implement sink capability bits for deduplication and enhance evidence tagging * feat: implement dynamic activation handling for gated sinks and enhance validation logic * feat: enhance configuration documentation and clarify inline analysis cache behavior * feat: implement panic recovery during analysis to continue scans past errors * feat: add expectations configuration for taint analysis and performance metrics * feat: enhance error handling and logging during file reading and mutex locking * feat: add cross-file body loading tests and plumbing for CF-1 phase * feat: implement cross-file k=1 context-sensitive inline taint analysis with new tests and fixtures * feat: implement indexed-scan parity in cross-file inline analysis with new dropdown and copy functionality * feat: enhance classification span handling in CFG and AST for improved source attribution * feat: add new Express routes for handling user input and telemetry data * feat: implement ternary expression handling in CFG with diamond structure for JS/TS * feat: implement Phase CF-3 abstract-domain transfer channels in summaries * feat: add support for string-prefix transfer in cross-file calls and update tests * docs: reduce RESULTS.md doc size * feat: implement Phase CF-4 per-return-path summary decomposition with tests * feat: update parameter handling in pass1 and refactor SsaFuncSummary initialization * feat: implement Phase CF-5 for cross-file SCC joint fixed-point convergence with new flags and tests * feat: implement Phase CF-6 with parameter-granularity points-to summaries and associated tests * refactor: update comments and documentation for clarity and consistency * style: format code for consistency and readability * refactor: simplify verdict handling and improve edge checking logic * refactor: optimize path and identifier collection by avoiding unnecessary cloning * chore: update Cargo.toml for Rust version 1.85 and add ignored files; modify CHANGELOG and README for clarity on state analysis defaults * refactor: update documentation and improve clarity in configuration files * refactor: update documentation and improve clarity in configuration files * feat: add JS/TS pass-2 convergence tests and expectations configuration * feat: add Phase 5 regression tests for inline cache origin attribution and update related logic * feat: implement Phase 7 deduplication and alternative path linking for taint findings * feat: implement structural DFS index for anonymous functions and update naming conventions * feat: add Phase 8 regression tests for container-element taint in JS and Python * feat: add engine-depth profiles and explain-engine option for CLI * feat: update expectations and add new README fixtures for multi-file scan regression * feat: implement Phase 11 callback-alias and factory patterns with regression tests * feat: implement Terminator::Switch for multi-way dispatch and add regression tests * feat: add real-CVE benchmark fixtures for CVE-2023-48022, CVE-2019-14939, and CVE-2023-26159 with corresponding patched variants * refactor: extract cfg and ssa_transfer to submodules * refactor: cargo fmt * refactor: remove unnecessary blank line in cfg_tests.rs * refactor: remove unnecessary planning file * chore: update Rust version to 1.88 and bump dependencies in Cargo files * feat: enhance triage UI with new layout and controls, update README for clarity * feat: enhance triage UI with new layout and controls, update README for clarity * chore: remove outdated section from README for version 0.5.0 * docs: improve clarity and consistency in README content * chore: add "GPL-3.0-or-later" to license options in about.toml * chore: update license handling in about.toml and check-licenses.mjs * style: format code for improved readability in TriagePage component * style: format code for improved readability in TriagePage component * chore: enhance license handling and improve body_id scoping in seed lookup * feat: introduce owner and parent body IDs for enhanced seed scoping * feat: implement direction-aware engine provenance with new CLI flag for strict CI gating * feat: add Undef SSA operation for improved control-flow handling * style: improve code formatting for consistency and readability in multiple files * feat: add 16-function chain SCC across multiple files for enhanced analysis * style: simplify code formatting for improved readability in multiple files * fix: update CapHitReason default implementation and improve README clarity * docs: enhance README with detailed explanations of taint analysis and limitations * docs: refine README for clarity and consistency in taint analysis section * style: improve code formatting for better readability in NewScanModal and scans * fix: update cargo-about command to use --offline for deterministic license generation * fix: update cargo-about command to use --offline for deterministic license generation * ci: add step to prime cargo registry cache for deterministic license generation * feat: add support for non-sink collections in authorization analysis * feat: enhance authorization checks with row-level ownership equality and binding tracking * feat: implement self-scoped user handling and enhance ownership checks * refactor: simplify assertions and formatting in authorization analysis tests * fix: normalize line endings in THIRDPARTY-LICENSES.html generation and update README with AI disclosure * docs: update AI disclosure section for clarity and conciseness * feat: add AI Contribution Policy and update contributing guidelines for AI assistance disclosure * feat: enhance authorization analysis with SSA-derived variable type classification * feat: implement auth_finding_to_diag function for enhanced security diagnostics * feat: add args_value_refs to CallSite struct for enhanced argument tracking * feat: add args_value_refs to CallSite struct for enhanced argument tracking * feat: add direction-aware engine provenance with LossDirection classification and new CLI flag * feat: simplify strip_cap_from_call_args call by removing unnecessary line breaks * feat: enhance error message handling in cli_validation_tests for better Windows compatibility * feat: optimize release profile settings in Cargo.toml and update CodeQL configuration * feat: enhance release build process with SBOM generation and SLSA provenance * feat: update actions/checkout and actions/setup-node to v6, enhance CLI options, and improve auth-check summaries * feat: introduce PathFact handling for path safety checks and rejection logic * feat: introduce PathFact handling for path safety checks and rejection logic * feat: update benchmark data and enhance path sanitization logic with new safety checks * feat: document AI assistance in frontend UI development and human review process * feat: add return path facts for enhanced path safety checks and update documentation * chore: update release date for version 0.5.0 in CHANGELOG.md * chore: clean up ci.yml by removing outdated comments and clarifying steps * feat: implement cross-language path sanitizers and validators for enhanced security * feat: enhance SSA value usage tracking by including block terminators and improve path safety checks * feat: enhance switch statement handling by adding per-case path constraints and support for exclusive cases * refactor: simplify conditional formatting and improve code readability in executor and lower modules * feat: add vulnerable examples for various languages demonstrating authentication and sanitization issues * feat: enhance actor context recognition for self-actor identifiers and add support for global non-sink receivers * feat: enhance actor context recognition for self-actor identifiers and add support for global non-sink receivers * feat: add transform classifiers for Java, Go, and Ruby with corresponding tests * refactor: clarify comments on reassign-to-constant idiom and sink behavior in guards.rs --------- Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
813 lines
25 KiB
Rust
813 lines
25 KiB
Rust
//! Nyx Benchmark Evaluation Framework
|
|
//!
|
|
//! Run with: `cargo test benchmark_evaluation -- --ignored --nocapture`
|
|
//!
|
|
//! Filter with env vars:
|
|
//! NYX_BENCH_LANG=python
|
|
//! NYX_BENCH_CLASS=sqli
|
|
//! NYX_BENCH_CASE=js-sqli-001
|
|
//! NYX_BENCH_POSITIVE_ONLY=1
|
|
//! NYX_BENCH_NEGATIVE_ONLY=1
|
|
//! NYX_BENCH_TAG=express
|
|
|
|
mod common;
|
|
|
|
use common::test_config;
|
|
use nyx_scanner::commands::scan::Diag;
|
|
use nyx_scanner::evidence::Confidence;
|
|
use nyx_scanner::patterns::FindingCategory;
|
|
use nyx_scanner::utils::config::AnalysisMode;
|
|
use serde::{Deserialize, Serialize};
|
|
use std::collections::BTreeMap;
|
|
use std::path::{Path, PathBuf};
|
|
|
|
// ── Ground-truth schema ──────────────────────────────────────────────
|
|
|
|
#[derive(Deserialize)]
|
|
struct GroundTruth {
|
|
#[allow(dead_code)]
|
|
schema_version: String,
|
|
#[allow(dead_code)]
|
|
metadata: Metadata,
|
|
cases: Vec<Case>,
|
|
}
|
|
|
|
#[derive(Deserialize)]
|
|
struct Metadata {
|
|
#[allow(dead_code)]
|
|
description: String,
|
|
#[allow(dead_code)]
|
|
created: String,
|
|
#[allow(dead_code)]
|
|
corpus_size: usize,
|
|
}
|
|
|
|
#[derive(Deserialize)]
|
|
struct Case {
|
|
case_id: String,
|
|
file: String,
|
|
language: String,
|
|
is_vulnerable: bool,
|
|
vuln_class: String,
|
|
#[allow(dead_code)]
|
|
cwe: String,
|
|
#[allow(dead_code)]
|
|
provenance: String,
|
|
#[allow(dead_code)]
|
|
equivalence_tier: String,
|
|
#[allow(dead_code)]
|
|
match_mode: String,
|
|
expected_rule_ids: Vec<String>,
|
|
allowed_alternative_rule_ids: Vec<String>,
|
|
forbidden_rule_ids: Vec<String>,
|
|
#[allow(dead_code)]
|
|
expected_severity: Option<String>,
|
|
#[allow(dead_code)]
|
|
expected_category: Option<String>,
|
|
expected_sink_lines: Option<Vec<[usize; 2]>>,
|
|
#[allow(dead_code)]
|
|
expected_source_lines: Option<Vec<[usize; 2]>>,
|
|
/// Optional: line ranges where the *call site* leading to the sink
|
|
/// appears. When present, at least one `flow_step` in the finding's
|
|
/// trace must fall within ±2 of one of these ranges. When absent,
|
|
/// the check is skipped (forward-compatible with existing fixtures).
|
|
#[serde(default)]
|
|
expected_call_site_lines: Option<Vec<[usize; 2]>>,
|
|
#[allow(dead_code)]
|
|
tags: Vec<String>,
|
|
#[serde(default)]
|
|
disabled: bool,
|
|
#[allow(dead_code)]
|
|
notes: String,
|
|
}
|
|
|
|
// ── Result types ─────────────────────────────────────────────────────
|
|
|
|
#[derive(Clone, Serialize)]
|
|
enum Outcome {
|
|
TP,
|
|
FP,
|
|
FN,
|
|
TN,
|
|
}
|
|
|
|
impl std::fmt::Display for Outcome {
|
|
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
|
match self {
|
|
Outcome::TP => write!(f, "TP"),
|
|
Outcome::FP => write!(f, "FP"),
|
|
Outcome::FN => write!(f, "FN"),
|
|
Outcome::TN => write!(f, "TN"),
|
|
}
|
|
}
|
|
}
|
|
|
|
#[derive(Serialize)]
|
|
struct CaseOutcome {
|
|
case_id: String,
|
|
file: String,
|
|
language: String,
|
|
vuln_class: String,
|
|
is_vulnerable: bool,
|
|
outcome_file_level: Outcome,
|
|
outcome_rule_level: Outcome,
|
|
outcome_location_level: Option<Outcome>,
|
|
matched_rule_ids: Vec<String>,
|
|
unexpected_rule_ids: Vec<String>,
|
|
all_finding_ids: Vec<String>,
|
|
security_finding_count: usize,
|
|
non_security_finding_count: usize,
|
|
#[serde(skip)]
|
|
diags: Vec<Diag>,
|
|
}
|
|
|
|
#[derive(Serialize)]
|
|
struct Metrics {
|
|
tp: usize,
|
|
fp: usize,
|
|
fn_: usize,
|
|
tn: usize,
|
|
precision: f64,
|
|
recall: f64,
|
|
f1: f64,
|
|
}
|
|
|
|
impl Metrics {
|
|
fn compute(tp: usize, fp: usize, fn_: usize, tn: usize) -> Self {
|
|
let precision = if tp + fp == 0 {
|
|
1.0
|
|
} else {
|
|
tp as f64 / (tp + fp) as f64
|
|
};
|
|
let recall = if tp + fn_ == 0 {
|
|
1.0
|
|
} else {
|
|
tp as f64 / (tp + fn_) as f64
|
|
};
|
|
let f1 = if precision + recall == 0.0 {
|
|
0.0
|
|
} else {
|
|
2.0 * precision * recall / (precision + recall)
|
|
};
|
|
Metrics {
|
|
tp,
|
|
fp,
|
|
fn_,
|
|
tn,
|
|
precision,
|
|
recall,
|
|
f1,
|
|
}
|
|
}
|
|
}
|
|
|
|
#[derive(Serialize)]
|
|
struct ScannerConfig {
|
|
analysis_mode: String,
|
|
taint_enabled: bool,
|
|
ast_patterns_enabled: bool,
|
|
state_analysis_enabled: bool,
|
|
worker_threads: usize,
|
|
}
|
|
|
|
#[derive(Serialize)]
|
|
struct BenchmarkResults {
|
|
benchmark_version: String,
|
|
timestamp: String,
|
|
scanner_version: String,
|
|
scanner_config: ScannerConfig,
|
|
ground_truth_hash: String,
|
|
corpus_size: usize,
|
|
cases_run: usize,
|
|
cases_skipped: usize,
|
|
outcomes: Vec<CaseOutcome>,
|
|
aggregate_file_level: Metrics,
|
|
aggregate_rule_level: Metrics,
|
|
by_language: BTreeMap<String, Metrics>,
|
|
by_vuln_class: BTreeMap<String, Metrics>,
|
|
by_confidence: BTreeMap<String, Metrics>,
|
|
}
|
|
|
|
// ── Scanning ─────────────────────────────────────────────────────────
|
|
|
|
fn scan_corpus_file(corpus_root: &Path, relative_path: &str) -> Vec<Diag> {
|
|
// `cve_corpus/*` cases live in a sibling of `corpus/` — see
|
|
// `tests/benchmark/cve_corpus/`.
|
|
let source = if relative_path.starts_with("cve_corpus/") {
|
|
corpus_root
|
|
.parent()
|
|
.expect("corpus_root has a parent")
|
|
.join(relative_path)
|
|
} else {
|
|
corpus_root.join(relative_path)
|
|
};
|
|
assert!(
|
|
source.exists(),
|
|
"Corpus path not found: {}",
|
|
source.display()
|
|
);
|
|
|
|
let tmp = tempfile::TempDir::with_prefix("nyx_bench_").expect("tempdir");
|
|
|
|
if source.is_dir() {
|
|
copy_dir_recursive(&source, tmp.path());
|
|
} else {
|
|
let dest = tmp.path().join(source.file_name().unwrap());
|
|
std::fs::copy(&source, &dest).expect("copy corpus file");
|
|
}
|
|
|
|
let cfg = test_config(AnalysisMode::Full);
|
|
let mut diags =
|
|
nyx_scanner::scan_no_index(tmp.path(), &cfg).expect("scan_no_index should succeed");
|
|
|
|
// Normalize paths to filename only.
|
|
for d in &mut diags {
|
|
if let Some(fname) = Path::new(&d.path).file_name() {
|
|
d.path = fname.to_string_lossy().to_string();
|
|
}
|
|
}
|
|
|
|
// Sort deterministically.
|
|
diags.sort_by(|a, b| {
|
|
a.path
|
|
.cmp(&b.path)
|
|
.then(a.line.cmp(&b.line))
|
|
.then(a.id.cmp(&b.id))
|
|
.then(a.col.cmp(&b.col))
|
|
});
|
|
|
|
diags
|
|
}
|
|
|
|
fn copy_dir_recursive(src: &Path, dst: &Path) {
|
|
for entry in std::fs::read_dir(src).expect("read corpus dir") {
|
|
let entry = entry.expect("dir entry");
|
|
let dest_path = dst.join(entry.file_name());
|
|
if entry.file_type().expect("file type").is_dir() {
|
|
std::fs::create_dir_all(&dest_path).expect("create subdir");
|
|
copy_dir_recursive(&entry.path(), &dest_path);
|
|
} else {
|
|
std::fs::copy(entry.path(), &dest_path).expect("copy file");
|
|
}
|
|
}
|
|
}
|
|
|
|
// ── Scoring helpers ──────────────────────────────────────────────────
|
|
|
|
fn is_security(d: &Diag) -> bool {
|
|
d.category == FindingCategory::Security
|
|
}
|
|
|
|
fn rule_matches(finding_id: &str, expected_id: &str) -> bool {
|
|
if finding_id == expected_id {
|
|
return true;
|
|
}
|
|
// Substring fallback: the expected id is contained in the finding id.
|
|
finding_id.contains(expected_id)
|
|
}
|
|
|
|
fn score_file_level(case: &Case, diags: &[Diag]) -> Outcome {
|
|
let has_security = diags.iter().any(is_security);
|
|
match (case.is_vulnerable, has_security) {
|
|
(true, true) => Outcome::TP,
|
|
(true, false) => Outcome::FN,
|
|
(false, true) => Outcome::FP,
|
|
(false, false) => Outcome::TN,
|
|
}
|
|
}
|
|
|
|
fn score_rule_level(case: &Case, diags: &[Diag]) -> (Outcome, Vec<String>, Vec<String>) {
|
|
let security_diags: Vec<&Diag> = diags.iter().filter(|d| is_security(d)).collect();
|
|
|
|
// Check forbidden rules.
|
|
if case.is_vulnerable {
|
|
for d in &security_diags {
|
|
for forbidden in &case.forbidden_rule_ids {
|
|
if rule_matches(&d.id, forbidden) {
|
|
// Wrong-reason detection counts as FP.
|
|
let matched = vec![];
|
|
let unexpected = security_diags.iter().map(|d| d.id.clone()).collect();
|
|
return (Outcome::FP, matched, unexpected);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
if !case.is_vulnerable {
|
|
if security_diags.is_empty() {
|
|
return (Outcome::TN, vec![], vec![]);
|
|
} else {
|
|
let unexpected = security_diags.iter().map(|d| d.id.clone()).collect();
|
|
return (Outcome::FP, vec![], unexpected);
|
|
}
|
|
}
|
|
|
|
// Positive case: check expected + alternative rule matches.
|
|
let all_acceptable: Vec<&str> = case
|
|
.expected_rule_ids
|
|
.iter()
|
|
.chain(case.allowed_alternative_rule_ids.iter())
|
|
.map(|s| s.as_str())
|
|
.collect();
|
|
|
|
let mut matched = Vec::new();
|
|
let mut unexpected = Vec::new();
|
|
|
|
for d in &security_diags {
|
|
let is_expected = all_acceptable.iter().any(|exp| rule_matches(&d.id, exp));
|
|
if is_expected {
|
|
matched.push(d.id.clone());
|
|
} else {
|
|
unexpected.push(d.id.clone());
|
|
}
|
|
}
|
|
|
|
if matched.is_empty() {
|
|
(Outcome::FN, matched, unexpected)
|
|
} else {
|
|
(Outcome::TP, matched, unexpected)
|
|
}
|
|
}
|
|
|
|
fn score_location_level(
|
|
case: &Case,
|
|
diags: &[Diag],
|
|
matched_rule_ids: &[String],
|
|
) -> Option<Outcome> {
|
|
let expected_sinks = case.expected_sink_lines.as_ref()?;
|
|
if expected_sinks.is_empty() {
|
|
return None;
|
|
}
|
|
|
|
if !case.is_vulnerable {
|
|
let has_security = diags.iter().any(is_security);
|
|
return Some(if has_security {
|
|
Outcome::FP
|
|
} else {
|
|
Outcome::TN
|
|
});
|
|
}
|
|
|
|
// Check if any matched finding has a line within tolerance of expected sinks.
|
|
let all_acceptable: Vec<&str> = case
|
|
.expected_rule_ids
|
|
.iter()
|
|
.chain(case.allowed_alternative_rule_ids.iter())
|
|
.map(|s| s.as_str())
|
|
.collect();
|
|
|
|
let security_diags: Vec<&Diag> = diags.iter().filter(|d| is_security(d)).collect();
|
|
|
|
if matched_rule_ids.is_empty() {
|
|
return Some(Outcome::FN);
|
|
}
|
|
|
|
for d in &security_diags {
|
|
let is_expected = all_acceptable.iter().any(|exp| rule_matches(&d.id, exp));
|
|
if !is_expected {
|
|
continue;
|
|
}
|
|
let primary_ok = expected_sinks.iter().any(|r| {
|
|
let lo = r[0].saturating_sub(2);
|
|
let hi = r[1] + 2;
|
|
d.line >= lo && d.line <= hi
|
|
});
|
|
if !primary_ok {
|
|
continue;
|
|
}
|
|
// Optional: require at least one flow_step to fall within the
|
|
// caller's call-site range. Only active when the fixture opts in.
|
|
if let Some(call_ranges) = case.expected_call_site_lines.as_ref()
|
|
&& !call_ranges.is_empty()
|
|
{
|
|
let steps = d
|
|
.evidence
|
|
.as_ref()
|
|
.map(|e| e.flow_steps.as_slice())
|
|
.unwrap_or(&[]);
|
|
let call_ok = steps.iter().any(|s| {
|
|
call_ranges.iter().any(|r| {
|
|
let lo = r[0].saturating_sub(2);
|
|
let hi = r[1] + 2;
|
|
(s.line as usize) >= lo && (s.line as usize) <= hi
|
|
})
|
|
});
|
|
if !call_ok {
|
|
continue;
|
|
}
|
|
}
|
|
return Some(Outcome::TP);
|
|
}
|
|
|
|
// Rule matched but location didn't.
|
|
Some(Outcome::FN)
|
|
}
|
|
|
|
// ── Filtering ────────────────────────────────────────────────────────
|
|
|
|
fn should_run(case: &Case) -> bool {
|
|
if case.disabled {
|
|
return false;
|
|
}
|
|
|
|
if let Ok(lang) = std::env::var("NYX_BENCH_LANG")
|
|
&& case.language != lang
|
|
{
|
|
return false;
|
|
}
|
|
if let Ok(class) = std::env::var("NYX_BENCH_CLASS")
|
|
&& case.vuln_class != class
|
|
{
|
|
return false;
|
|
}
|
|
if let Ok(id) = std::env::var("NYX_BENCH_CASE")
|
|
&& case.case_id != id
|
|
{
|
|
return false;
|
|
}
|
|
if std::env::var("NYX_BENCH_POSITIVE_ONLY").is_ok() && !case.is_vulnerable {
|
|
return false;
|
|
}
|
|
if std::env::var("NYX_BENCH_NEGATIVE_ONLY").is_ok() && case.is_vulnerable {
|
|
return false;
|
|
}
|
|
if let Ok(tag) = std::env::var("NYX_BENCH_TAG")
|
|
&& !case.tags.iter().any(|t| t == &tag)
|
|
{
|
|
return false;
|
|
}
|
|
|
|
true
|
|
}
|
|
|
|
// ── Aggregation ──────────────────────────────────────────────────────
|
|
|
|
fn aggregate(outcomes: &[CaseOutcome], level: &str) -> Metrics {
|
|
let (mut tp, mut fp, mut fn_, mut tn) = (0, 0, 0, 0);
|
|
for o in outcomes {
|
|
let outcome = match level {
|
|
"file" => &o.outcome_file_level,
|
|
"rule" => &o.outcome_rule_level,
|
|
_ => &o.outcome_file_level,
|
|
};
|
|
match outcome {
|
|
Outcome::TP => tp += 1,
|
|
Outcome::FP => fp += 1,
|
|
Outcome::FN => fn_ += 1,
|
|
Outcome::TN => tn += 1,
|
|
}
|
|
}
|
|
Metrics::compute(tp, fp, fn_, tn)
|
|
}
|
|
|
|
fn aggregate_by_key(
|
|
outcomes: &[CaseOutcome],
|
|
key_fn: impl Fn(&CaseOutcome) -> &str,
|
|
) -> BTreeMap<String, Metrics> {
|
|
let mut groups: BTreeMap<String, Vec<&CaseOutcome>> = BTreeMap::new();
|
|
for o in outcomes {
|
|
groups.entry(key_fn(o).to_string()).or_default().push(o);
|
|
}
|
|
groups
|
|
.into_iter()
|
|
.map(|(k, cases)| {
|
|
let (mut tp, mut fp, mut fn_, mut tn) = (0, 0, 0, 0);
|
|
for o in &cases {
|
|
match &o.outcome_rule_level {
|
|
Outcome::TP => tp += 1,
|
|
Outcome::FP => fp += 1,
|
|
Outcome::FN => fn_ += 1,
|
|
Outcome::TN => tn += 1,
|
|
}
|
|
}
|
|
(k, Metrics::compute(tp, fp, fn_, tn))
|
|
})
|
|
.collect()
|
|
}
|
|
|
|
// ── Printing ─────────────────────────────────────────────────────────
|
|
|
|
fn print_case_table(outcomes: &[CaseOutcome]) {
|
|
println!(
|
|
"\n{:<25} {:<40} {:<6} {:<6} {:<6} {:<4} {:<4}",
|
|
"CASE_ID", "FILE", "FILE", "RULE", "LOC", "SEC", "OTH"
|
|
);
|
|
println!("{}", "-".repeat(100));
|
|
for o in outcomes {
|
|
let loc = match &o.outcome_location_level {
|
|
Some(out) => format!("{}", out),
|
|
None => "-".to_string(),
|
|
};
|
|
println!(
|
|
"{:<25} {:<40} {:<6} {:<6} {:<6} {:<4} {:<4}",
|
|
o.case_id,
|
|
truncate(&o.file, 39),
|
|
o.outcome_file_level,
|
|
o.outcome_rule_level,
|
|
loc,
|
|
o.security_finding_count,
|
|
o.non_security_finding_count,
|
|
);
|
|
}
|
|
}
|
|
|
|
fn print_metrics_table(label: &str, metrics: &Metrics) {
|
|
println!(
|
|
" {:<20} TP={:<4} FP={:<4} FN={:<4} TN={:<4} P={:.3} R={:.3} F1={:.3}",
|
|
label,
|
|
metrics.tp,
|
|
metrics.fp,
|
|
metrics.fn_,
|
|
metrics.tn,
|
|
metrics.precision,
|
|
metrics.recall,
|
|
metrics.f1
|
|
);
|
|
}
|
|
|
|
fn print_map_table(title: &str, map: &BTreeMap<String, Metrics>) {
|
|
println!("\n {}:", title);
|
|
for (k, m) in map {
|
|
print_metrics_table(k, m);
|
|
}
|
|
}
|
|
|
|
fn truncate(s: &str, max: usize) -> String {
|
|
if s.len() <= max {
|
|
s.to_string()
|
|
} else {
|
|
format!("...{}", &s[s.len() - max + 3..])
|
|
}
|
|
}
|
|
|
|
// ── Main test ────────────────────────────────────────────────────────
|
|
|
|
#[test]
|
|
#[ignore]
|
|
fn benchmark_evaluation() {
|
|
let benchmark_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/benchmark");
|
|
let corpus_root = benchmark_dir.join("corpus");
|
|
let gt_path = benchmark_dir.join("ground_truth.json");
|
|
let results_dir = benchmark_dir.join("results");
|
|
|
|
// Load ground truth.
|
|
let gt_bytes = std::fs::read(>_path).expect("read ground_truth.json");
|
|
let gt: GroundTruth = serde_json::from_slice(>_bytes).expect("parse ground_truth.json");
|
|
|
|
// Compute ground truth hash for provenance.
|
|
let gt_hash = format!("sha256:{}", sha256_hex(>_bytes));
|
|
|
|
// Filter cases.
|
|
let mut cases_skipped = 0usize;
|
|
let cases_to_run: Vec<&Case> = gt
|
|
.cases
|
|
.iter()
|
|
.filter(|c| {
|
|
if !should_run(c) {
|
|
cases_skipped += 1;
|
|
false
|
|
} else {
|
|
true
|
|
}
|
|
})
|
|
.collect();
|
|
|
|
println!("\n=== Nyx Benchmark Evaluation ===");
|
|
println!(
|
|
"Corpus: {} total, {} to run, {} skipped",
|
|
gt.cases.len(),
|
|
cases_to_run.len(),
|
|
cases_skipped
|
|
);
|
|
|
|
// Run each case.
|
|
let mut outcomes: Vec<CaseOutcome> = Vec::with_capacity(cases_to_run.len());
|
|
|
|
for case in &cases_to_run {
|
|
let diags = scan_corpus_file(&corpus_root, &case.file);
|
|
|
|
let security_count = diags.iter().filter(|d| is_security(d)).count();
|
|
let non_security_count = diags.len() - security_count;
|
|
let all_finding_ids: Vec<String> = diags.iter().map(|d| d.id.clone()).collect();
|
|
|
|
let file_outcome = score_file_level(case, &diags);
|
|
let (rule_outcome, matched, unexpected) = score_rule_level(case, &diags);
|
|
let loc_outcome = score_location_level(case, &diags, &matched);
|
|
|
|
outcomes.push(CaseOutcome {
|
|
case_id: case.case_id.clone(),
|
|
file: case.file.clone(),
|
|
language: case.language.clone(),
|
|
vuln_class: case.vuln_class.clone(),
|
|
is_vulnerable: case.is_vulnerable,
|
|
outcome_file_level: file_outcome,
|
|
outcome_rule_level: rule_outcome,
|
|
outcome_location_level: loc_outcome,
|
|
matched_rule_ids: matched,
|
|
unexpected_rule_ids: unexpected,
|
|
all_finding_ids,
|
|
security_finding_count: security_count,
|
|
non_security_finding_count: non_security_count,
|
|
diags,
|
|
});
|
|
}
|
|
|
|
// Sort outcomes by case_id for deterministic output.
|
|
outcomes.sort_by(|a, b| a.case_id.cmp(&b.case_id));
|
|
|
|
// Print per-case table.
|
|
print_case_table(&outcomes);
|
|
|
|
// Compute aggregates.
|
|
let agg_file = aggregate(&outcomes, "file");
|
|
let agg_rule = aggregate(&outcomes, "rule");
|
|
let by_language = aggregate_by_key(&outcomes, |o| &o.language);
|
|
let by_class = aggregate_by_key(&outcomes, |o| &o.vuln_class);
|
|
|
|
// Score at each confidence threshold.
|
|
let by_confidence = score_by_confidence(&outcomes, &cases_to_run);
|
|
|
|
// Print summary.
|
|
println!("\n=== Aggregate Metrics ===");
|
|
print_metrics_table("File-level", &agg_file);
|
|
print_metrics_table("Rule-level", &agg_rule);
|
|
print_map_table("By language (rule-level)", &by_language);
|
|
print_map_table("By vuln class (rule-level)", &by_class);
|
|
print_map_table("By confidence threshold (rule-level)", &by_confidence);
|
|
|
|
// Write results JSON.
|
|
std::fs::create_dir_all(&results_dir).ok();
|
|
let results = BenchmarkResults {
|
|
benchmark_version: "1.0".to_string(),
|
|
timestamp: chrono_now(),
|
|
scanner_version: env!("CARGO_PKG_VERSION").to_string(),
|
|
scanner_config: ScannerConfig {
|
|
analysis_mode: "Full".to_string(),
|
|
taint_enabled: true,
|
|
ast_patterns_enabled: true,
|
|
state_analysis_enabled: true,
|
|
worker_threads: 1,
|
|
},
|
|
ground_truth_hash: gt_hash,
|
|
corpus_size: gt.cases.len(),
|
|
cases_run: outcomes.len(),
|
|
cases_skipped,
|
|
outcomes,
|
|
aggregate_file_level: agg_file,
|
|
aggregate_rule_level: agg_rule,
|
|
by_language,
|
|
by_vuln_class: by_class,
|
|
by_confidence,
|
|
};
|
|
|
|
let results_path = results_dir.join("latest.json");
|
|
let json = serde_json::to_string_pretty(&results).expect("serialize results");
|
|
std::fs::write(&results_path, &json).expect("write results/latest.json");
|
|
|
|
println!("\nResults written to: {}", results_path.display());
|
|
println!("=== Benchmark complete ===\n");
|
|
|
|
// ── Regression thresholds (current baseline minus ~5pp) ────────
|
|
// Baseline (2026-04-24, post per-return-path PathFact landing):
|
|
// Rule-level P=0.947 R=0.994 F1=0.970 on the 316-case corpus
|
|
// (TP=179 FP=10 FN=1 TN=125). Adds 8 cases from the path-sanitiser
|
|
// FP cluster (rs-safe-014, rs-safe-016, CVE-2018-20997 +
|
|
// CVE-2022-36113 + CVE-2024-24576 patched/vulnerable pairs). Rust
|
|
// language F1 stayed at 1.000.
|
|
//
|
|
// Floors sit ~5pp below that baseline: a single-case flip is ~0.3pp
|
|
// on this corpus, so 5pp is generous enough to absorb honest
|
|
// FP↔TN trades while still catching a real regression in a
|
|
// vulnerability class. When you land a durable, measurable
|
|
// improvement, tighten these floors — do not relax them to paper
|
|
// over a regression.
|
|
let rule = &results.aggregate_rule_level;
|
|
assert!(
|
|
rule.precision >= 0.897,
|
|
"Rule-level precision {:.3} fell below threshold 0.897 (baseline 0.947)",
|
|
rule.precision,
|
|
);
|
|
assert!(
|
|
rule.recall >= 0.944,
|
|
"Rule-level recall {:.3} fell below threshold 0.944 (baseline 0.994)",
|
|
rule.recall,
|
|
);
|
|
assert!(
|
|
rule.f1 >= 0.920,
|
|
"Rule-level F1 {:.3} fell below threshold 0.920 (baseline 0.970)",
|
|
rule.f1,
|
|
);
|
|
}
|
|
|
|
// ── Confidence-threshold scoring ─────────────────────────────────────
|
|
|
|
fn score_by_confidence(outcomes: &[CaseOutcome], cases: &[&Case]) -> BTreeMap<String, Metrics> {
|
|
let mut result = BTreeMap::new();
|
|
for (label, min_conf) in [
|
|
(">=Low", Confidence::Low),
|
|
(">=Medium", Confidence::Medium),
|
|
(">=High", Confidence::High),
|
|
] {
|
|
let (mut tp, mut fp, mut fn_, mut tn) = (0, 0, 0, 0);
|
|
for (outcome, case) in outcomes.iter().zip(cases.iter()) {
|
|
let filtered: Vec<&Diag> = outcome
|
|
.diags
|
|
.iter()
|
|
.filter(|d| d.confidence.is_some_and(|c| c >= min_conf))
|
|
.collect();
|
|
let (rule_outcome, _, _) = score_rule_level_with_diags(case, &filtered);
|
|
match rule_outcome {
|
|
Outcome::TP => tp += 1,
|
|
Outcome::FP => fp += 1,
|
|
Outcome::FN => fn_ += 1,
|
|
Outcome::TN => tn += 1,
|
|
}
|
|
}
|
|
result.insert(label.to_string(), Metrics::compute(tp, fp, fn_, tn));
|
|
}
|
|
result
|
|
}
|
|
|
|
fn score_rule_level_with_diags(
|
|
case: &Case,
|
|
security_diags: &[&Diag],
|
|
) -> (Outcome, Vec<String>, Vec<String>) {
|
|
if case.is_vulnerable {
|
|
for d in security_diags {
|
|
if !is_security(d) {
|
|
continue;
|
|
}
|
|
for forbidden in &case.forbidden_rule_ids {
|
|
if rule_matches(&d.id, forbidden) {
|
|
let unexpected = security_diags
|
|
.iter()
|
|
.filter(|d| is_security(d))
|
|
.map(|d| d.id.clone())
|
|
.collect();
|
|
return (Outcome::FP, vec![], unexpected);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
let sec_diags: Vec<&&Diag> = security_diags.iter().filter(|d| is_security(d)).collect();
|
|
|
|
if !case.is_vulnerable {
|
|
if sec_diags.is_empty() {
|
|
return (Outcome::TN, vec![], vec![]);
|
|
} else {
|
|
let unexpected = sec_diags.iter().map(|d| d.id.clone()).collect();
|
|
return (Outcome::FP, vec![], unexpected);
|
|
}
|
|
}
|
|
|
|
let all_acceptable: Vec<&str> = case
|
|
.expected_rule_ids
|
|
.iter()
|
|
.chain(case.allowed_alternative_rule_ids.iter())
|
|
.map(|s| s.as_str())
|
|
.collect();
|
|
|
|
let mut matched = Vec::new();
|
|
let mut unexpected = Vec::new();
|
|
for d in &sec_diags {
|
|
let is_expected = all_acceptable.iter().any(|exp| rule_matches(&d.id, exp));
|
|
if is_expected {
|
|
matched.push(d.id.clone());
|
|
} else {
|
|
unexpected.push(d.id.clone());
|
|
}
|
|
}
|
|
|
|
if matched.is_empty() {
|
|
(Outcome::FN, matched, unexpected)
|
|
} else {
|
|
(Outcome::TP, matched, unexpected)
|
|
}
|
|
}
|
|
|
|
// ── Utilities ────────────────────────────────────────────────────────
|
|
|
|
fn sha256_hex(data: &[u8]) -> String {
|
|
use std::io::Write;
|
|
// Simple SHA-256 via command — avoids adding a crypto dependency.
|
|
let mut child = std::process::Command::new("shasum")
|
|
.args(["-a", "256"])
|
|
.stdin(std::process::Stdio::piped())
|
|
.stdout(std::process::Stdio::piped())
|
|
.spawn()
|
|
.expect("shasum");
|
|
child.stdin.as_mut().unwrap().write_all(data).unwrap();
|
|
let out = child.wait_with_output().unwrap();
|
|
let s = String::from_utf8(out.stdout).unwrap();
|
|
s.split_whitespace().next().unwrap_or("unknown").to_string()
|
|
}
|
|
|
|
fn chrono_now() -> String {
|
|
// ISO 8601 timestamp without chrono dependency.
|
|
let out = std::process::Command::new("date")
|
|
.args(["-u", "+%Y-%m-%dT%H:%M:%SZ"])
|
|
.output()
|
|
.expect("date");
|
|
String::from_utf8(out.stdout).unwrap().trim().to_string()
|
|
}
|