mirror of
https://github.com/elicpeter/nyx.git
synced 2026-06-09 19:45:13 +02:00
* feat: Introduce function-scoped variable interning for state analysis with new tests and fixtures * feat: Add Phase 26 symbolic execution enhancements with bitwise operator support, abstract interpretation refinements, and new taint analysis tests * feat: Refine state analysis to handle factory-pattern resource returns with mixed-path tests and leak detection enhancements * feat: Add Phase 27 debug views with symbolic execution, abstract interpretation, SSA, and call graph viewers; integrate with debug layout and styles * feat: Add Phase 31 type-qualified symbolic resolution with receiver-based callee disambiguation and testing * feat: Extend symbolic execution with state iteration, enhanced debug views, and debounced input handling * feat: Add Phase 13 resource and auth pattern extensions with new tests and fixtures * feat: Introduce CFG debug graph renderer with compact mode, toolbar, and DAG layout integration * feat: Add Phase 28 encoding and decoding transform modeling with structural symex enhancements and new taint analysis tests * feat: Extend abstract interpretation with type facts and constant value tracking in debug views and server logic * feat: Add linear path handling and witness extraction to symbolic execution with Phase 28 transform mismatch detection * feat: Refine Go auth and sanitizer handling with enhanced rules, state updates, and benchmark improvements * feat: Enable auth-state analysis by default and update relevant tests in benchmark config * test: Update state_tests to reflect default enablement of auth-state analysis and add auth suppression test * docs: update CHANGELOG.md * feat: Introduce per-index taint tracking in `HeapState` with `HeapSlot`, overflow handling, and revised SSA transfers * feat: Introduce C/C++ language labels and refine heap state tracking in SSA transfers * feat: Implement per-index array slot tracking in symbolic heap with overflow collapse * feat: Add implicit definition handling for uninitialized declarations in SSA value allocation * feat: Refactor function parameters and constants for improved clarity and maintainability * refactor: Reorder module imports and improve formatting for consistency * refactor: Fix formatting erorrs * refactor: Fix clippy warnings * refactor: Fix fmt warnings (again) * chore: Update dependencies and improve feature configuration * Add comprehensive tests for undertested modules (#36) (COPILOT) * Add comprehensive tests for undertested modules Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> Agent-Logs-Url: https://github.com/elicpeter/nyx/sessions/f3fc877e-f386-49ba-9793-fc93d3805083 * Add comprehensive tests for ext, project, walk, and errors modules Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> Agent-Logs-Url: https://github.com/elicpeter/nyx/sessions/f3fc877e-f386-49ba-9793-fc93d3805083 --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> * chore: Update dependencies and improve feature configuration * fix: formatting errors in new tests * chore: Update license list in about.toml * chore: made functions input inline * chore: updated cfg graph to take up the full page * chore: add Prettier configuration and update code formatting * Add frontend test suite with Vitest (111 tests) (#37) * Add Vitest test suite for frontend - 111 tests across utils, components, hooks, and graph utilities Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> Agent-Logs-Url: https://github.com/elicpeter/nyx/sessions/7cf0dba2-ecff-4740-ba4d-92717e74a0b7 * ci: add frontend test step to CI workflow Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> Agent-Logs-Url: https://github.com/elicpeter/nyx/sessions/5bc0ac9f-0a32-4d03-9cb7-7a15aea53fca --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> * chore: simplify array initialization in test files for consistency * ran typecheck * feat: add AnalysisWorkspace component and integrate it into CfgViewerPage * feat: update routing in AppLayout and improve empty state message in ExplorerPage * feat: enhance scan progress tracking with additional metrics and stages * feat: update license information and add license check script * feat: implement cross-file symbolic execution with callee body persistence * feat: replace dagre graphs with Graphology + ELK + Sigma for more advanced call stack and cfg rendering * feat: ensure CFG function view is scoped to the selected function, preventing bleed into sibling functions * feat: enhance resource tracking with proxy method summaries and improve finding extraction * feat: add terminal function exit detection for accurate resource leak analysis * feat: add warnings for loops and functions without bodies to improve error recovery * feat: update lambda expression handling to ensure proper function classification and control flow * feat: remove bounded formatting/string ops and add JSON.parse sanitizer for improved data handling * feat: add inline return taint analysis and regression tests for improved security checks * feat: add engine version management and migration handling for database schema updates * feat: enhance first_call_ident to skip nested function bodies and add regression tests * feat: enhance callee name resolution with two-segment normalization and disambiguation * feat: add cross-file context flags and debug assertions for taint analysis * feat: refactor taint analysis structure to unify context handling and improve clarity * feat: enhance dead code elimination to preserve Sink, Source, and Sanitizer labels with new tests * docs: updated CHANGELOG.md * fmt: formatting fixes * fix: fixed frontend formatting and lint warnings * fix: optimized ci * fix: optimized ci * Add comprehensive multi-file test coverage to Nyx (#38) * Initial checklist for multi-file test suite expansion Agent-Logs-Url: https://github.com/elicpeter/nyx/sessions/e550cb88-9767-4442-94d4-101bf5bb0e23 Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> * Add 12 new multi-file test fixtures with TP/TN/near-miss coverage Agent-Logs-Url: https://github.com/elicpeter/nyx/sessions/e550cb88-9767-4442-94d4-101bf5bb0e23 Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> * deleted root repo * rebuilt to test for regressions --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> Co-authored-by: elipeter <elicpeter@gmail.com> * feat: enhance import alias resolution and taint tracking * feat: implement security hardening with CSRF protection and path validation * feat: add support for import alias bindings in Python, PHP, and Rust * feat: enhance CFG analysis modes and improve code readability * feat: add detection for parameterized SQL queries to enhance security * feat: add safe internal redirect handling and enhance session destroy validation * feat: implement security improvements by addressing vulnerabilities in execAsync, session management, and file downloads * feat: enhance taint detection by adding support for inline source member expressions in call arguments * feat: implement pre-emission of Source nodes for inline source member expressions in call arguments * feat: add support for Throw statement in control flow and error handling * feat: add debug and echo endpoints with potential information leakage * feat: implement internal redirect suppression and enhance taint detection * feat: implement module alias tracking for dynamic dispatch in JS/TS * feat: add authorization analysis module with Express support * feat: add authorization analysis module with Express support * feat: add tests for admin guard requirements and clean checks in authorization analysis * feat: integrate Koa and Fastify frameworks into authorization analysis * feat: add Flask and Django support to authorization analysis module * feat: add support for Rails and Sinatra frameworks in authorization analysis * feat: add support for Axum, ActixWeb, and Rocket frameworks in authorization analysis * feat: add support for ActixWeb, Axum, and Rocket frameworks in authorization analysis * feat: add support for Rails and Sinatra in authorization analysis * chore: add .DS_Store to .gitignore * refactor: simplify conditional checks and improve readability in multiple files * refactor: update usage of Option methods for improved clarity and consistency * refactor: improve code readability by simplifying conditional checks and formatting * refactor: improve code formatting and readability by simplifying conditional checks * refactor: simplify conditional checks and improve readability in multiple files * refactor: simplify conditional checks in axum.rs for improved readability * feat: add CodeQL analysis configuration for enhanced security scanning * test: add comprehensive tests for `src/output.rs` SARIF builder (#39) * chore: start test coverage improvement work Agent-Logs-Url: https://github.com/elicpeter/nyx/sessions/cd7ff398-134e-4728-a5e7-0353a0744423 Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> * test: add comprehensive tests for src/output.rs SARIF builder Agent-Logs-Url: https://github.com/elicpeter/nyx/sessions/cd7ff398-134e-4728-a5e7-0353a0744423 Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> * refactor: improve code formatting and readability in output.rs --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: elicpeter <54954007+elicpeter@users.noreply.github.com> Co-authored-by: elipeter <elicpeter@gmail.com> * refactor: improve code formatting and readability in output.rs * Potential fix for code scanning alert no. 210: Uncontrolled data used in path expression Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Potential fix for code scanning alert no. 211: Uncontrolled data used in path expression Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * refactor: enhance triage file path handling with improved error management and validation * refactor: updated func summaries for richer detail * refactor: update SSA summary extraction to use canonical FuncKey for distinct entries * refactor: enhance callee metadata structure to support arity, receiver, and qualifier for better overload resolution * refactor: add support for keyword arguments in function calls and enhance receiver extraction for method-style calls * refactor: implement new Flask routes for safe and unsafe shell command execution * refactor: separate receiver handling in SSA operations and enhance taint propagation * refactor: improve arity handling by using arg_uses for positional argument count and enhance witness scoring for tainted arguments * refactor: implement auth decorator extraction and classification for multiple languages * refactor: enhance Rust module path resolution and use map handling for cross-file disambiguation * refactor: introduce CalleeQuery struct for structured callee resolution and enhance resolver logic * refactor: implement same-file identity collision handling for `runTask` to ensure correct resolver behavior * refactor: standardize default struct initialization across multiple files * feat: add scripts for formatting checks and auto-fixes with test summaries * refactor: simplify character splitting and enhance namespace qualifier handling * refactor: improve documentation clarity and enhance code readability in resolver logic * refactor: replace default struct initialization with explicit field assignments for clarity * feat: enhance anonymous function naming by deriving context-based bindings * refactor: streamline match expressions for improved readability and performance * refactor: streamline match expressions for improved readability and performance * refactor: replace loop with while let for improved clarity and performance * feat: add SSA constant propagation support to analysis context for improved accuracy * feat: add SSA constant propagation support to analysis context for improved accuracy * feat: implement shell metacharacter validation and bounded-length checks in Rust analysis * feat: add static map analysis for command injection suppression and type safety * refactor: simplify match statements and reduce line breaks for improved readability * feat(summary): phase 1/5 SinkSite data model for primary sink-location attribution Introduce SinkSite (file_rel, line, col, snippet, cap) carrying the primary sink source-location through function summaries. Swap SsaFuncSummary.param_to_sink and FuncSummary.param_to_sink from a coarse Cap map to a deduped SmallVec<[SinkSite; 1]> per parameter, with a backward-compatible cap_sites() helper and serde defaults so pre-phase-1 on-disk rows continue to deserialise cleanly. Extraction: SinkSiteLocator bundles the tree/bytes/file_rel needed by extract_ssa_func_summary; ParsedFile::extract_ssa_artifacts wires the locator in for the persisted pass-1 path, while pass-2 intra-file transient summaries fall back to cap-only sites (behavior unchanged). Merge: GlobalSummaries::insert now unions sink sites with (file_rel, line, col, cap) dedup via shared union_param_sink_sites helper. Database: JSON-serialised summary columns carry the new shape automatically; no schema change needed. Phase 2 will consume SinkSite in build_taint_diag() to overwrite the caller-site Finding.line with the callee's sink line when resolved via summary. Phase 1 keeps behavior unchanged: scanning tests/benchmark/corpus/rust/cmdi/cmdi_indirect.rs still produces the same (wrong) line 10 finding. Adds round-trip tests covering SinkSite solo, SsaFuncSummary with sink sites, legacy-JSON default handling for both summary types, and merge dedup. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(taint): phase 2/5 thread SinkSite into SsaTaintEvent and Finding Plumb Phase 1's SinkSite through the event pipeline into Findings, no output change yet. SsaTaintEvent gains `primary_sink_site: Option<SinkSite>`; when the main or callback sink-emission path has non-empty `param_to_sink_sites`, filter to sites whose `(line != 0) && (cap ∩ sink_caps != ∅)` and emit one event per distinct site — the multi-primary collapse keeps each downstream Finding single-primary. Resolution: ResolvedSummary and SinkInfo gain mirror `param_to_sink_sites` fields, populated from `SsaFuncSummary.param_to_sink` (SSA + callback paths) and `FuncSummary.param_to_sink` (global paths). Label, local-summary, and interop resolution paths leave the field empty — they only ever had cap-level info to begin with. Finding: new `primary_location: Option<SinkLocation>` with `file_rel/line/col`. `ssa_events_to_findings` maps `event.primary_sink_site` → `Finding.primary_location`, filtering cap-only sites (`line == 0`) to `None` so the (0,0) sentinel never leaks to formatters. Dedup key extended with the primary location so multi-site events aren't collapsed back together. Invariants (debug_assert!): * every SinkSite reaching emission has `line != 0 && cap ∩ sink_caps != ∅` — enforced by the pick_primary_sink_sites* filters; * every populated Finding.primary_location has `line != 0` AND non-empty `file_rel` — the cap-only → None translation upstream guarantees this. Deliberately independent of `uses_summary`: that flag tracks whether the *taint chain* used a summary, whereas primary attribution requires only that the *sink* itself was summary-resolved. A local source reaching a cross-file sink produces `uses_summary=false` alongside a populated primary_location — documented on Finding.primary_location, covered by `cross_file_sink_finding_carries_primary_location`. build_taint_diag, SARIF/JSON/explanation formatters, and the benchmark scorer remain untouched: finding.line still comes from `cfg_graph[finding.sink]`, so cmdi_indirect.rs still reports line 10 and the benchmark's rs-cmdi-003 row still shows FN in the LOC column. Tests: `cross_file_sink_finding_carries_primary_location` (proves plumbing via a synthetic FuncSummary carrying a SinkSite at 42:5) and `cross_file_sink_cap_only_site_leaves_primary_location_none` (regression guard against cap-only sites surfacing). All 1566 lib tests + integration tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(output): phase 3/5 consume primary sink location in diag + SARIF When a finding's primary_location (populated in phase 2 from a callee summary's SinkSite) names the dangerous instruction inside a callee body, attribute the diagnostic line to that location instead of the caller's call site. The call site is demoted to a Call step in flow_steps, and a synthetic Sink step at the primary location is appended so analysts still see the full trace. Changes: - Add scan_root parameter to build_taint_diag so file_rel can be resolved back to an absolute path via a shared resolve_file_rel helper. Empty file_rel (single-file scans where namespace == "") resolves to the file under analysis. - Extend SinkLocation with snippet, carried from the upstream SinkSite so the formatter needs no second file read. - Relax the ssa_events_to_findings debug_assert to allow empty file_rel, which is valid when scan root equals the file itself. - SARIF: emit data-flow as codeFlows[0].threadFlows[0].locations[]; locations[0] already reflects the primary sink position via the updated diag line/col. Acceptance: scan on tests/benchmark/corpus/rust/cmdi/cmdi_indirect.rs now reports line 5 (Command::new) as the primary sink, with the call site at line 10 visible in flow_steps. Two expect.json fixtures updated (must_match line_range widened): - javascript/taint/context_sensitive_call: 12-14 -> 7-14 (line 8 is the real sink inside run()). - rust/cfg/closure_async: 10-10 -> 10-11 (line 11 is Command::new inside the closure). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(bench): phase 4/5 validate primary sink attribution across corpus Extend the benchmark scorer and ground truth to lock in phase 3's primary-location behavior, and add fixtures that exercise the new capability end-to-end. Scorer (tests/benchmark_test.rs): - Add optional `expected_call_site_lines: Option<Vec<[usize; 2]>>` on Case. When present, score_location_level additionally requires at least one flow_step in the finding's evidence trace to fall within ±2 of the call-site range. When absent, the check is skipped — fully forward-compatible with existing fixtures. - Retain ±2 tolerance on expected_sink_lines (compared against the now-primary Diag.line post-phase-3). Ground truth edits: - rs-cmdi-cross-001: expected_sink_lines [8,8] -> [9,9]. Line 8 is the transform::wrap call site (a cross-file propagator, not a sink); line 9 is Command::new, the real sink. The ±2 tolerance happened to mask this stale attribution but it was semantically wrong — phase 4 is the right time to correct it. Also adds expected_call_site_lines [8,8] so the new field is exercised on an existing cross-file case. - rs-cmdi-003: adds expected_call_site_lines [10,10] (run_cmd call). This fixture's sink (Command::new inside run_cmd at line 5) was the motivating case for phases 1-3; adding the call-site assertion guards against regression to caller-line attribution. New fixtures: - rust/cmdi/cmdi_indirect_multisink.rs (rs-cmdi-009): helper run_both takes two tainted params and invokes two Command sinks on consecutive lines. Locks in that primary line lands inside the helper (lines 5-6), not at the caller (line 12). Notes document that SinkSite is currently one-per-callee so both findings today collapse onto the first sink; expected_sink_lines=[5,6] and expected_call_site_lines=[12,12] stay valid either way. - python/cmdi/cross_indirect_sink/{app.py,helper.py} (py-cmdi-cross- 004): sink os.system lives in helper.py (cross-file), caller in app.py reads env source and calls run_cmd. Verifies phase 3's cross-file primary attribution: Diag.path = helper.py, Diag.line = 5, with app.py:7 recorded in flow_steps as a Call step. Acceptance: - `cargo test --test benchmark_test -- --ignored --nocapture` passes. - rs-cmdi-003 is TP/TP/TP (the target flip FN->TP at LOC). All pre-existing TP/TP/TP fixtures remain TP/TP/TP; 2 new fixtures are TP/TP/TP. - Aggregate rule-level: TP=158 FP=10 FN=1 TN=97, P=0.940 R=0.994 F1=0.966 on the 266-case corpus (was TP=156 FP=10 FN=1 TN=97 on 264 pre-phase-4, delta is the +2 new cases both resolving TP). - Full `cargo test` green (1566 lib tests + all integration tests). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(taint): phase 5/5 lock Finding.primary_location contract via regression test Add a regression test in src/taint/ssa_transfer.rs that wires up a synthetic SsaFuncSummary with a SinkSite at other.rs:42:10 and drives the three emission stages (pick_primary_sink_sites → emit_ssa_taint_events → ssa_events_to_findings) against a minimal caller SSA body. Asserts the resulting Finding.primary_location is exactly that triple. The existing integration tests in src/taint/tests.rs cover the coarse FuncSummary path end-to-end through analyse_file. This test locks in the lower-level SSA-side plumbing so a future refactor that silently drops the site between pick → emit → findings fails here rather than only at the benchmark layer. Also refreshes tests/benchmark/results/latest.json (timestamp only; rs-cmdi-003 remains TP/TP/TP and the aggregate P/R/F1 are unchanged from phase 4). Closes the primary sink-location attribution feature (phases 1-5/5): * Phase 1 — SinkSite data model on summaries. * Phase 2 — SinkSite threaded into SsaTaintEvent and Finding. * Phase 3 — diag + SARIF consume primary_location. * Phase 4 — benchmark validates primary_call_site_lines across corpus. * Phase 5 — regression test locks the event→finding contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: clean up formatting and improve readability in multiple files * refactor: simplify type definition for deduplication key in findings * test(harness): add must_not_match expectation for FP regression guards Extends ExpectedFinding with must_not_match field that asserts a diagnostic must NOT fire — presence is a hard failure. Non-consuming scan so it coexists with must_match entries on the same rule_id. Adds forbidden_violations accumulator and updates summary line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(regression): update expectations to ensure must_not_match for various taint and resource leak rules * feat: implement auto-seeding for JS/TS handler parameters to enhance taint tracking * feat: update switch statement handling to improve control flow analysis * feat: implement promisify alias handling for JS/TS to enhance taint tracking * feat: enhance taint tracking by refining expectation handling and adding mode filtering * feat: refine SQL handling in stream processing and enhance auto-seeding for handler parameters * feat: update taint tracking rules to enforce full mode matching and improve flow analysis * feat: enhance Ruby subshell handling to improve taint tracking and flow analysis * feat: update xss_response expectations to refine taint flow analysis and enhance regression guarding * feat: refine framework detection and update expectation handling for Echo and Sinatra * feat: implement max_count for taint tracking expectations and deduplicate findings * feat: add strict_unexpected handling for taint-unsanitised-flow in expectation files * feat: enhance deduplication of taint-unsanitised-flow findings by collapsing based on line and severity * feat: add strict_unexpected handling for taint-unsanitised-flow in multiple expectation files * feat: add structural invariant checks for SSA bodies * feat: ensure deterministic phi emission order using BTreeSet * feat: enhance handling of terminators to ensure authoritative flow through successor edges * feat: enhance Goto terminator handling to ensure all successors are marked executable * feat: refactor code for improved readability and organization * feat: simplify predicate checks and enhance readability in SSA handling * feat: implement per-file parse timeout and enhance file size handling * feat: migrate analysis engine toggles from environment variables to configuration file * feat: remove unnecessary whitespace in hostile_input_tests.rs * feat: remove unnecessary whitespace in hostile_input_tests.rs * feat: update dependencies and enhance documentation on language maturity * feat: enhance security headers and improve request body limits * feat: implement sink capability bits for deduplication and enhance evidence tagging * feat: implement dynamic activation handling for gated sinks and enhance validation logic * feat: enhance configuration documentation and clarify inline analysis cache behavior * feat: implement panic recovery during analysis to continue scans past errors * feat: add expectations configuration for taint analysis and performance metrics * feat: enhance error handling and logging during file reading and mutex locking * feat: add cross-file body loading tests and plumbing for CF-1 phase * feat: implement cross-file k=1 context-sensitive inline taint analysis with new tests and fixtures * feat: implement indexed-scan parity in cross-file inline analysis with new dropdown and copy functionality * feat: enhance classification span handling in CFG and AST for improved source attribution * feat: add new Express routes for handling user input and telemetry data * feat: implement ternary expression handling in CFG with diamond structure for JS/TS * feat: implement Phase CF-3 abstract-domain transfer channels in summaries * feat: add support for string-prefix transfer in cross-file calls and update tests * docs: reduce RESULTS.md doc size * feat: implement Phase CF-4 per-return-path summary decomposition with tests * feat: update parameter handling in pass1 and refactor SsaFuncSummary initialization * feat: implement Phase CF-5 for cross-file SCC joint fixed-point convergence with new flags and tests * feat: implement Phase CF-6 with parameter-granularity points-to summaries and associated tests * refactor: update comments and documentation for clarity and consistency * style: format code for consistency and readability * refactor: simplify verdict handling and improve edge checking logic * refactor: optimize path and identifier collection by avoiding unnecessary cloning * chore: update Cargo.toml for Rust version 1.85 and add ignored files; modify CHANGELOG and README for clarity on state analysis defaults * refactor: update documentation and improve clarity in configuration files * refactor: update documentation and improve clarity in configuration files * feat: add JS/TS pass-2 convergence tests and expectations configuration * feat: add Phase 5 regression tests for inline cache origin attribution and update related logic * feat: implement Phase 7 deduplication and alternative path linking for taint findings * feat: implement structural DFS index for anonymous functions and update naming conventions * feat: add Phase 8 regression tests for container-element taint in JS and Python * feat: add engine-depth profiles and explain-engine option for CLI * feat: update expectations and add new README fixtures for multi-file scan regression * feat: implement Phase 11 callback-alias and factory patterns with regression tests * feat: implement Terminator::Switch for multi-way dispatch and add regression tests * feat: add real-CVE benchmark fixtures for CVE-2023-48022, CVE-2019-14939, and CVE-2023-26159 with corresponding patched variants * refactor: extract cfg and ssa_transfer to submodules * refactor: cargo fmt * refactor: remove unnecessary blank line in cfg_tests.rs * refactor: remove unnecessary planning file * chore: update Rust version to 1.88 and bump dependencies in Cargo files * feat: enhance triage UI with new layout and controls, update README for clarity * feat: enhance triage UI with new layout and controls, update README for clarity * chore: remove outdated section from README for version 0.5.0 * docs: improve clarity and consistency in README content * chore: add "GPL-3.0-or-later" to license options in about.toml * chore: update license handling in about.toml and check-licenses.mjs * style: format code for improved readability in TriagePage component * style: format code for improved readability in TriagePage component * chore: enhance license handling and improve body_id scoping in seed lookup * feat: introduce owner and parent body IDs for enhanced seed scoping * feat: implement direction-aware engine provenance with new CLI flag for strict CI gating * feat: add Undef SSA operation for improved control-flow handling * style: improve code formatting for consistency and readability in multiple files * feat: add 16-function chain SCC across multiple files for enhanced analysis * style: simplify code formatting for improved readability in multiple files * fix: update CapHitReason default implementation and improve README clarity * docs: enhance README with detailed explanations of taint analysis and limitations * docs: refine README for clarity and consistency in taint analysis section * style: improve code formatting for better readability in NewScanModal and scans * fix: update cargo-about command to use --offline for deterministic license generation * fix: update cargo-about command to use --offline for deterministic license generation * ci: add step to prime cargo registry cache for deterministic license generation * feat: add support for non-sink collections in authorization analysis * feat: enhance authorization checks with row-level ownership equality and binding tracking * feat: implement self-scoped user handling and enhance ownership checks * refactor: simplify assertions and formatting in authorization analysis tests * fix: normalize line endings in THIRDPARTY-LICENSES.html generation and update README with AI disclosure * docs: update AI disclosure section for clarity and conciseness * feat: add AI Contribution Policy and update contributing guidelines for AI assistance disclosure * feat: enhance authorization analysis with SSA-derived variable type classification * feat: implement auth_finding_to_diag function for enhanced security diagnostics * feat: add args_value_refs to CallSite struct for enhanced argument tracking * feat: add args_value_refs to CallSite struct for enhanced argument tracking * feat: add direction-aware engine provenance with LossDirection classification and new CLI flag * feat: simplify strip_cap_from_call_args call by removing unnecessary line breaks * feat: enhance error message handling in cli_validation_tests for better Windows compatibility * feat: optimize release profile settings in Cargo.toml and update CodeQL configuration * feat: enhance release build process with SBOM generation and SLSA provenance * feat: update actions/checkout and actions/setup-node to v6, enhance CLI options, and improve auth-check summaries * feat: introduce PathFact handling for path safety checks and rejection logic * feat: introduce PathFact handling for path safety checks and rejection logic * feat: update benchmark data and enhance path sanitization logic with new safety checks * feat: document AI assistance in frontend UI development and human review process * feat: add return path facts for enhanced path safety checks and update documentation * chore: update release date for version 0.5.0 in CHANGELOG.md * chore: clean up ci.yml by removing outdated comments and clarifying steps * feat: implement cross-language path sanitizers and validators for enhanced security * feat: enhance SSA value usage tracking by including block terminators and improve path safety checks * feat: enhance switch statement handling by adding per-case path constraints and support for exclusive cases * refactor: simplify conditional formatting and improve code readability in executor and lower modules * feat: add vulnerable examples for various languages demonstrating authentication and sanitization issues * feat: enhance actor context recognition for self-actor identifiers and add support for global non-sink receivers * feat: enhance actor context recognition for self-actor identifiers and add support for global non-sink receivers * feat: add transform classifiers for Java, Go, and Ruby with corresponding tests * refactor: clarify comments on reassign-to-constant idiom and sink behavior in guards.rs --------- Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
780 lines
29 KiB
Rust
780 lines
29 KiB
Rust
//! Real-world vulnerability fixture test suite.
|
||
//!
|
||
//! Scans realistic code snippets (20–120 lines) across all 10 supported languages
|
||
//! and compares findings against `.expect.json` expectation files.
|
||
//!
|
||
//! # Environment Variables
|
||
//!
|
||
//! - `NYX_TEST_LANG=python` — run only fixtures for one language
|
||
//! - `NYX_TEST_FIXTURE=cmdi_subprocess` — run only fixtures whose name contains this string
|
||
//! - `NYX_TEST_VERBOSE=1` — print full diff details for every fixture
|
||
//! - `NYX_TEST_CATEGORY=taint` — run only one category (taint/cfg/state/mixed)
|
||
//!
|
||
//! # Known-failure handling
|
||
//!
|
||
//! Expectations with `"must_match": false` are tracked but do not cause test failure.
|
||
//! A summary of soft misses is always printed at the end.
|
||
|
||
mod common;
|
||
|
||
use common::test_config;
|
||
use nyx_scanner::commands::scan::Diag;
|
||
use nyx_scanner::utils::config::AnalysisMode;
|
||
use serde::Deserialize;
|
||
use std::collections::BTreeMap;
|
||
use std::path::{Path, PathBuf};
|
||
use std::sync::OnceLock;
|
||
|
||
// ── Expectation schema ───────────────────────────────────────────────────────
|
||
|
||
#[derive(Debug, Clone, Deserialize)]
|
||
#[allow(dead_code)]
|
||
struct RealWorldExpectations {
|
||
/// Human description of what this fixture tests.
|
||
#[serde(default)]
|
||
description: String,
|
||
/// Tags for coverage matrix (e.g. ["taint", "cmdi", "express"]).
|
||
#[serde(default)]
|
||
tags: Vec<String>,
|
||
/// Which analysis modes this fixture targets.
|
||
#[serde(default = "default_modes")]
|
||
modes: Vec<String>,
|
||
/// Rule-id prefixes whose unexpected findings promote from
|
||
/// informational to hard failure. A diag is "unexpected" if it is
|
||
/// not consumed by any `expected` entry; if its `id` starts with any
|
||
/// prefix here, the suite fails. Default empty → all extras remain
|
||
/// informational (pre-existing behavior).
|
||
///
|
||
/// Use this to lock in precision for fixtures whose expected set is
|
||
/// exhaustive for a given rule family. Typical value:
|
||
/// `["taint-unsanitised-flow"]` — any extra taint flow is a
|
||
/// precision regression. AST-pattern families (`*.code_exec.*`,
|
||
/// `*.quality.*`) are intentionally excluded by default since they
|
||
/// fire syntactically and bystander triggers aren't precision drift.
|
||
#[serde(default)]
|
||
strict_unexpected: Vec<String>,
|
||
/// Expected findings.
|
||
expected: Vec<ExpectedFinding>,
|
||
}
|
||
|
||
fn default_modes() -> Vec<String> {
|
||
vec!["full".to_string()]
|
||
}
|
||
|
||
#[derive(Debug, Clone, Deserialize)]
|
||
struct ExpectedFinding {
|
||
/// Rule ID substring to match (e.g. "taint-" or "js.xss.innerhtml").
|
||
rule_id: String,
|
||
/// Severity (optional, not checked if absent).
|
||
#[serde(default)]
|
||
severity: Option<String>,
|
||
/// If true, missing this finding is a hard failure. If false, it's a soft miss.
|
||
#[serde(default = "default_must_match")]
|
||
must_match: bool,
|
||
/// If true, presence of a matching finding is a hard failure (regression guard).
|
||
/// Overrides `must_match`. Useful for locking in FP suppressions — sanitizer
|
||
/// wrappers, gated sinks, field-aware absence, Layer-B suppressions, etc.
|
||
#[serde(default)]
|
||
must_not_match: bool,
|
||
/// Line number or range [start, end] where finding should appear.
|
||
#[serde(default)]
|
||
line_range: Option<(usize, usize)>,
|
||
/// Substrings that must appear in message or evidence fields.
|
||
#[serde(default)]
|
||
evidence_contains: Vec<String>,
|
||
/// Human explanation of this expectation.
|
||
#[serde(default)]
|
||
notes: String,
|
||
/// Optional per-expectation mode filter. When absent, the expectation
|
||
/// applies in every mode listed at the fixture level. When present,
|
||
/// only the listed modes evaluate this expectation — useful when a
|
||
/// finding is mode-specific (e.g. a taint flow only resolves in `full`
|
||
/// mode while the fixture also runs in `ast` mode for AST-pattern
|
||
/// coverage).
|
||
#[serde(default)]
|
||
modes: Option<Vec<String>>,
|
||
/// Upper bound on matching diags. When set, the count of diags that
|
||
/// match this expectation's filters (rule_id / severity / line_range /
|
||
/// evidence_contains) must not exceed this value. Composes with
|
||
/// `must_match: true` — a `must_match: true, max_count: 1` expectation
|
||
/// means "exactly one matching finding must exist". Mutually exclusive
|
||
/// with `must_not_match: true`; the combination is rejected at parse
|
||
/// time.
|
||
#[serde(default)]
|
||
max_count: Option<usize>,
|
||
}
|
||
|
||
fn default_must_match() -> bool {
|
||
true
|
||
}
|
||
|
||
// ── Fixture discovery ────────────────────────────────────────────────────────
|
||
|
||
#[derive(Debug, Clone)]
|
||
struct Fixture {
|
||
/// Language slug (rust, c, cpp, java, go, php, python, ruby, typescript, javascript).
|
||
lang: String,
|
||
/// Category (taint, cfg, state, mixed).
|
||
category: String,
|
||
/// Fixture name (stem of source file).
|
||
name: String,
|
||
/// Path to the source fixture file.
|
||
source_path: PathBuf,
|
||
/// Parsed expectations.
|
||
expectations: RealWorldExpectations,
|
||
}
|
||
|
||
fn discover_fixtures() -> Vec<Fixture> {
|
||
let base = Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/real_world");
|
||
let mut fixtures = Vec::new();
|
||
|
||
let langs = [
|
||
"rust",
|
||
"c",
|
||
"cpp",
|
||
"java",
|
||
"go",
|
||
"php",
|
||
"python",
|
||
"ruby",
|
||
"typescript",
|
||
"javascript",
|
||
];
|
||
let categories = ["taint", "cfg", "state", "mixed"];
|
||
|
||
for lang in &langs {
|
||
for category in &categories {
|
||
let dir = base.join(lang).join(category);
|
||
if !dir.is_dir() {
|
||
continue;
|
||
}
|
||
|
||
// Find all .expect.json files, derive source file from them.
|
||
let Ok(entries) = std::fs::read_dir(&dir) else {
|
||
continue;
|
||
};
|
||
for entry in entries.flatten() {
|
||
let path = entry.path();
|
||
let fname = path.file_name().unwrap().to_string_lossy().to_string();
|
||
if !fname.ends_with(".expect.json") {
|
||
continue;
|
||
}
|
||
|
||
let stem = fname.trim_end_matches(".expect.json");
|
||
|
||
// Find the corresponding source file (any extension).
|
||
let source_path = find_source_file(&dir, stem);
|
||
let Some(source_path) = source_path else {
|
||
eprintln!(
|
||
"WARN: no source file for {}/{}/{}/{}",
|
||
lang, category, stem, fname
|
||
);
|
||
continue;
|
||
};
|
||
|
||
let expect_content = std::fs::read_to_string(&path).unwrap_or_else(|e| {
|
||
panic!("Failed to read {}: {e}", path.display());
|
||
});
|
||
let expectations: RealWorldExpectations = serde_json::from_str(&expect_content)
|
||
.unwrap_or_else(|e| {
|
||
panic!("Failed to parse {}: {e}", path.display());
|
||
});
|
||
|
||
fixtures.push(Fixture {
|
||
lang: lang.to_string(),
|
||
category: category.to_string(),
|
||
name: stem.to_string(),
|
||
source_path,
|
||
expectations,
|
||
});
|
||
}
|
||
}
|
||
}
|
||
|
||
// Sort for deterministic ordering.
|
||
fixtures.sort_by(|a, b| {
|
||
a.lang
|
||
.cmp(&b.lang)
|
||
.then(a.category.cmp(&b.category))
|
||
.then(a.name.cmp(&b.name))
|
||
});
|
||
|
||
fixtures
|
||
}
|
||
|
||
fn find_source_file(dir: &Path, stem: &str) -> Option<PathBuf> {
|
||
let extensions = [
|
||
"rs", "c", "cpp", "cc", "cxx", "java", "go", "php", "py", "rb", "ts", "tsx", "js", "jsx",
|
||
];
|
||
for ext in &extensions {
|
||
let candidate = dir.join(format!("{stem}.{ext}"));
|
||
if candidate.exists() {
|
||
return Some(candidate);
|
||
}
|
||
}
|
||
None
|
||
}
|
||
|
||
// ── Scanning ─────────────────────────────────────────────────────────────────
|
||
|
||
fn scan_fixture(fixture: &Fixture, mode: AnalysisMode) -> Vec<Diag> {
|
||
// We scan the parent directory containing just this fixture file.
|
||
// To isolate, we copy the fixture to a temp dir.
|
||
let tmp = tempfile::TempDir::with_prefix("nyx_rw_test_").expect("tempdir");
|
||
let dest = tmp.path().join(fixture.source_path.file_name().unwrap());
|
||
std::fs::copy(&fixture.source_path, &dest).expect("copy fixture");
|
||
|
||
let cfg = test_config(mode);
|
||
let mut diags =
|
||
nyx_scanner::scan_no_index(tmp.path(), &cfg).expect("scan_no_index should succeed");
|
||
|
||
// Normalize paths to just the filename for comparison.
|
||
for d in &mut diags {
|
||
if let Some(fname) = Path::new(&d.path).file_name() {
|
||
d.path = fname.to_string_lossy().to_string();
|
||
}
|
||
}
|
||
|
||
// Sort deterministically.
|
||
diags.sort_by(|a, b| {
|
||
a.path
|
||
.cmp(&b.path)
|
||
.then(a.line.cmp(&b.line))
|
||
.then(a.id.cmp(&b.id))
|
||
.then(a.col.cmp(&b.col))
|
||
});
|
||
|
||
diags
|
||
}
|
||
|
||
// ── Matching ─────────────────────────────────────────────────────────────────
|
||
|
||
#[derive(Debug)]
|
||
struct MatchResult {
|
||
hard_misses: Vec<(ExpectedFinding, String)>,
|
||
soft_misses: Vec<(ExpectedFinding, String)>,
|
||
forbidden_violations: Vec<(ExpectedFinding, Diag)>,
|
||
count_violations: Vec<(ExpectedFinding, usize)>,
|
||
unexpected: Vec<Diag>,
|
||
/// Subset of `unexpected` whose rule-id matched a `strict_unexpected`
|
||
/// prefix for this fixture — these cause hard failure.
|
||
strict_unexpected: Vec<Diag>,
|
||
matched: usize,
|
||
}
|
||
|
||
fn diag_matches_expectation(d: &Diag, exp: &ExpectedFinding, fixture_file: &str) -> bool {
|
||
if !d.id.contains(&exp.rule_id) {
|
||
return false;
|
||
}
|
||
if !d.path.contains(fixture_file) && fixture_file != d.path {
|
||
return false;
|
||
}
|
||
if let Some(ref sev) = exp.severity
|
||
&& d.severity.as_db_str() != sev.to_uppercase()
|
||
{
|
||
return false;
|
||
}
|
||
if let Some((start, end)) = exp.line_range
|
||
&& (d.line < start || d.line > end)
|
||
{
|
||
return false;
|
||
}
|
||
for substr in &exp.evidence_contains {
|
||
let msg = d.message.as_deref().unwrap_or("");
|
||
let ev_text = if let Some(ev) = &d.evidence {
|
||
let mut parts = Vec::new();
|
||
if let Some(src) = &ev.source {
|
||
parts.push(format!(
|
||
"source: {}",
|
||
src.snippet.as_deref().unwrap_or(&src.kind)
|
||
));
|
||
}
|
||
if let Some(snk) = &ev.sink {
|
||
parts.push(format!(
|
||
"sink: {}",
|
||
snk.snippet.as_deref().unwrap_or(&snk.kind)
|
||
));
|
||
}
|
||
for note in &ev.notes {
|
||
parts.push(note.clone());
|
||
}
|
||
if let Some(ref sym) = ev.symbolic
|
||
&& let Some(ref w) = sym.witness
|
||
{
|
||
parts.push(w.clone());
|
||
}
|
||
parts.join(" ")
|
||
} else {
|
||
String::new()
|
||
};
|
||
let combined = format!("{msg} {ev_text}");
|
||
if !combined.to_lowercase().contains(&substr.to_lowercase()) {
|
||
return false;
|
||
}
|
||
}
|
||
true
|
||
}
|
||
|
||
fn match_expectations(
|
||
diags: &[Diag],
|
||
expectations: &[ExpectedFinding],
|
||
fixture_file: &str,
|
||
active_mode: &str,
|
||
strict_prefixes: &[String],
|
||
) -> MatchResult {
|
||
let mut hard_misses = Vec::new();
|
||
let mut soft_misses = Vec::new();
|
||
let mut forbidden_violations = Vec::new();
|
||
let mut count_violations: Vec<(ExpectedFinding, usize)> = Vec::new();
|
||
let mut matched_indices: Vec<bool> = vec![false; diags.len()];
|
||
let mut matched = 0;
|
||
|
||
for exp in expectations {
|
||
if let Some(ref m) = exp.modes
|
||
&& !m.iter().any(|s| s.eq_ignore_ascii_case(active_mode))
|
||
{
|
||
continue;
|
||
}
|
||
|
||
// must_not_match wins over any other assertion combo.
|
||
if exp.must_not_match {
|
||
// Forbidden-finding assertion: non-consuming scan for any matching diag.
|
||
// Presence = hard failure (regression guard).
|
||
for d in diags {
|
||
if diag_matches_expectation(d, exp, fixture_file) {
|
||
forbidden_violations.push((exp.clone(), d.clone()));
|
||
}
|
||
}
|
||
continue;
|
||
}
|
||
|
||
// When max_count is set, count all diags matching the expectation's
|
||
// filter (regardless of prior consumption) and validate against the
|
||
// cap. Then consume the first unmatched matching diag like the
|
||
// normal path so later expectations still see the rest.
|
||
if let Some(cap) = exp.max_count {
|
||
let total_matches = diags
|
||
.iter()
|
||
.filter(|d| diag_matches_expectation(d, exp, fixture_file))
|
||
.count();
|
||
if total_matches > cap {
|
||
count_violations.push((exp.clone(), total_matches));
|
||
}
|
||
|
||
let mut found_idx: Option<usize> = None;
|
||
for (i, d) in diags.iter().enumerate() {
|
||
if matched_indices[i] {
|
||
continue;
|
||
}
|
||
if diag_matches_expectation(d, exp, fixture_file) {
|
||
found_idx = Some(i);
|
||
break;
|
||
}
|
||
}
|
||
if let Some(i) = found_idx {
|
||
matched_indices[i] = true;
|
||
matched += 1;
|
||
} else if exp.must_match {
|
||
hard_misses.push((
|
||
exp.clone(),
|
||
format!(
|
||
"rule_id='{}' severity={:?} line_range={:?} max_count={}",
|
||
exp.rule_id, exp.severity, exp.line_range, cap
|
||
),
|
||
));
|
||
}
|
||
continue;
|
||
}
|
||
|
||
let mut found_idx: Option<usize> = None;
|
||
for (i, d) in diags.iter().enumerate() {
|
||
if matched_indices[i] {
|
||
continue;
|
||
}
|
||
if diag_matches_expectation(d, exp, fixture_file) {
|
||
found_idx = Some(i);
|
||
break;
|
||
}
|
||
}
|
||
|
||
if let Some(i) = found_idx {
|
||
matched_indices[i] = true;
|
||
matched += 1;
|
||
} else {
|
||
let reason = format!(
|
||
"rule_id='{}' severity={:?} line_range={:?}",
|
||
exp.rule_id, exp.severity, exp.line_range
|
||
);
|
||
if exp.must_match {
|
||
hard_misses.push((exp.clone(), reason));
|
||
} else {
|
||
soft_misses.push((exp.clone(), reason));
|
||
}
|
||
}
|
||
}
|
||
|
||
// Unexpected = diags not matched by any expectation. Informational by
|
||
// default; promoted to hard-failure if the fixture's `strict_unexpected`
|
||
// list contains a prefix of the diag's rule-id.
|
||
let unexpected: Vec<Diag> = diags
|
||
.iter()
|
||
.enumerate()
|
||
.filter(|(i, _)| !matched_indices[*i])
|
||
.map(|(_, d)| d.clone())
|
||
.collect();
|
||
|
||
let strict_unexpected: Vec<Diag> = if strict_prefixes.is_empty() {
|
||
Vec::new()
|
||
} else {
|
||
unexpected
|
||
.iter()
|
||
.filter(|d| strict_prefixes.iter().any(|p| d.id.starts_with(p)))
|
||
.cloned()
|
||
.collect()
|
||
};
|
||
|
||
MatchResult {
|
||
hard_misses,
|
||
soft_misses,
|
||
forbidden_violations,
|
||
count_violations,
|
||
unexpected,
|
||
strict_unexpected,
|
||
matched,
|
||
}
|
||
}
|
||
|
||
// ── Mode resolution ──────────────────────────────────────────────────────────
|
||
|
||
fn resolve_mode(mode_str: &str) -> AnalysisMode {
|
||
match mode_str.to_lowercase().as_str() {
|
||
"ast" => AnalysisMode::Ast,
|
||
"cfg" => AnalysisMode::Cfg,
|
||
"taint" => AnalysisMode::Taint,
|
||
"full" => AnalysisMode::Full,
|
||
_ => AnalysisMode::Full,
|
||
}
|
||
}
|
||
|
||
// ── Coverage matrix ──────────────────────────────────────────────────────────
|
||
|
||
fn print_coverage_matrix(fixtures: &[Fixture]) {
|
||
let mut matrix: BTreeMap<String, BTreeMap<String, usize>> = BTreeMap::new();
|
||
let mut tag_counts: BTreeMap<String, usize> = BTreeMap::new();
|
||
|
||
for f in fixtures {
|
||
*matrix
|
||
.entry(f.lang.clone())
|
||
.or_default()
|
||
.entry(f.category.clone())
|
||
.or_default() += 1;
|
||
for tag in &f.expectations.tags {
|
||
*tag_counts.entry(tag.clone()).or_default() += 1;
|
||
}
|
||
}
|
||
|
||
eprintln!("\n╔══════════════════════════════════════════════════════════╗");
|
||
eprintln!("║ REAL-WORLD TEST COVERAGE MATRIX ║");
|
||
eprintln!("╠══════════════╦════════╦══════╦════════╦════════╦════════╣");
|
||
eprintln!("║ Language ║ Taint ║ CFG ║ State ║ Mixed ║ Total ║");
|
||
eprintln!("╠══════════════╬════════╬══════╬════════╬════════╬════════╣");
|
||
|
||
let mut grand_total = 0;
|
||
for (lang, cats) in &matrix {
|
||
let t = cats.get("taint").unwrap_or(&0);
|
||
let c = cats.get("cfg").unwrap_or(&0);
|
||
let s = cats.get("state").unwrap_or(&0);
|
||
let m = cats.get("mixed").unwrap_or(&0);
|
||
let total = t + c + s + m;
|
||
grand_total += total;
|
||
eprintln!(
|
||
"║ {:<12} ║ {:>4} ║ {:>3} ║ {:>4} ║ {:>4} ║ {:>4} ║",
|
||
lang, t, c, s, m, total
|
||
);
|
||
}
|
||
eprintln!("╠══════════════╬════════╬══════╬════════╬════════╬════════╣");
|
||
eprintln!(
|
||
"║ TOTAL ║ ║ ║ ║ ║ {:>4} ║",
|
||
grand_total
|
||
);
|
||
eprintln!("╚══════════════╩════════╩══════╩════════╩════════╩════════╝");
|
||
|
||
if !tag_counts.is_empty() {
|
||
eprintln!("\nTag distribution:");
|
||
for (tag, count) in &tag_counts {
|
||
eprintln!(" {tag}: {count}");
|
||
}
|
||
}
|
||
}
|
||
|
||
// ── Main test ────────────────────────────────────────────────────────────────
|
||
|
||
static ALL_FIXTURES: OnceLock<Vec<Fixture>> = OnceLock::new();
|
||
|
||
fn get_fixtures() -> &'static [Fixture] {
|
||
ALL_FIXTURES.get_or_init(discover_fixtures)
|
||
}
|
||
|
||
fn should_run(fixture: &Fixture) -> bool {
|
||
if let Ok(lang) = std::env::var("NYX_TEST_LANG")
|
||
&& !fixture.lang.eq_ignore_ascii_case(&lang)
|
||
{
|
||
return false;
|
||
}
|
||
if let Ok(name) = std::env::var("NYX_TEST_FIXTURE")
|
||
&& !fixture.name.contains(&name)
|
||
{
|
||
return false;
|
||
}
|
||
if let Ok(cat) = std::env::var("NYX_TEST_CATEGORY")
|
||
&& !fixture.category.eq_ignore_ascii_case(&cat)
|
||
{
|
||
return false;
|
||
}
|
||
true
|
||
}
|
||
|
||
fn is_verbose() -> bool {
|
||
std::env::var("NYX_TEST_VERBOSE").is_ok()
|
||
}
|
||
|
||
#[test]
|
||
fn real_world_fixture_suite() {
|
||
let fixtures = get_fixtures();
|
||
let verbose = is_verbose();
|
||
|
||
let active: Vec<&Fixture> = fixtures.iter().filter(|f| should_run(f)).collect();
|
||
|
||
if active.is_empty() {
|
||
eprintln!(
|
||
"No fixtures matched filters. Total available: {}",
|
||
fixtures.len()
|
||
);
|
||
print_coverage_matrix(fixtures);
|
||
return;
|
||
}
|
||
|
||
eprintln!(
|
||
"\nRunning {} real-world fixtures (of {} total)\n",
|
||
active.len(),
|
||
fixtures.len()
|
||
);
|
||
|
||
let mut total_hard_fails = 0;
|
||
let mut total_soft_misses = 0;
|
||
let mut total_forbidden = 0;
|
||
let mut total_count_violations = 0;
|
||
let mut total_strict_unexpected = 0;
|
||
let mut total_matched = 0;
|
||
let mut total_unexpected = 0;
|
||
let mut failure_details: Vec<String> = Vec::new();
|
||
let mut soft_miss_details: Vec<String> = Vec::new();
|
||
let mut forbidden_details: Vec<String> = Vec::new();
|
||
let mut count_violation_details: Vec<String> = Vec::new();
|
||
let mut strict_unexpected_details: Vec<String> = Vec::new();
|
||
|
||
// Reject (must_not_match && max_count) at parse time. `must_not_match`
|
||
// overrides `must_match` in `match_expectations` (pre-existing semantics),
|
||
// but there is no sensible interpretation of "forbidden, up to N of".
|
||
for fixture in &active {
|
||
let fixture_label = format!("{}/{}/{}", fixture.lang, fixture.category, fixture.name);
|
||
for (idx, exp) in fixture.expectations.expected.iter().enumerate() {
|
||
if exp.must_not_match && exp.max_count.is_some() {
|
||
panic!(
|
||
"{}: expectation[{}] rule_id='{}' has both must_not_match and max_count set — these are mutually exclusive",
|
||
fixture_label, idx, exp.rule_id
|
||
);
|
||
}
|
||
}
|
||
}
|
||
|
||
for fixture in &active {
|
||
let fixture_label = format!("{}/{}/{}", fixture.lang, fixture.category, fixture.name);
|
||
|
||
for mode_str in &fixture.expectations.modes {
|
||
let mode = resolve_mode(mode_str);
|
||
let diags = scan_fixture(fixture, mode);
|
||
let fixture_file = fixture
|
||
.source_path
|
||
.file_name()
|
||
.unwrap()
|
||
.to_string_lossy()
|
||
.to_string();
|
||
|
||
let result = match_expectations(
|
||
&diags,
|
||
&fixture.expectations.expected,
|
||
&fixture_file,
|
||
mode_str,
|
||
&fixture.expectations.strict_unexpected,
|
||
);
|
||
|
||
total_matched += result.matched;
|
||
total_unexpected += result.unexpected.len();
|
||
|
||
if !result.hard_misses.is_empty() {
|
||
let mut msg = format!("FAIL {fixture_label} [{mode_str}]:");
|
||
for (exp, reason) in &result.hard_misses {
|
||
msg.push_str(&format!(
|
||
"\n MISSING (must_match): {} — {}",
|
||
reason, exp.notes
|
||
));
|
||
}
|
||
failure_details.push(msg);
|
||
total_hard_fails += result.hard_misses.len();
|
||
}
|
||
|
||
if !result.forbidden_violations.is_empty() {
|
||
let mut msg = format!("FORB {fixture_label} [{mode_str}]:");
|
||
for (exp, diag) in &result.forbidden_violations {
|
||
msg.push_str(&format!(
|
||
"\n FORBIDDEN (must_not_match): {}:{} [{}] {} matched rule_id='{}' — {}",
|
||
diag.path,
|
||
diag.line,
|
||
diag.severity.as_db_str(),
|
||
diag.id,
|
||
exp.rule_id,
|
||
exp.notes
|
||
));
|
||
}
|
||
forbidden_details.push(msg);
|
||
total_forbidden += result.forbidden_violations.len();
|
||
}
|
||
|
||
if !result.strict_unexpected.is_empty() {
|
||
let prefixes = fixture.expectations.strict_unexpected.join(",");
|
||
let mut msg =
|
||
format!("STRICT {fixture_label} [{mode_str}] (prefixes=[{prefixes}]):");
|
||
for d in &result.strict_unexpected {
|
||
msg.push_str(&format!(
|
||
"\n STRICT unexpected: {}:{} [{}] {} — not consumed by any expectation",
|
||
d.path,
|
||
d.line,
|
||
d.severity.as_db_str(),
|
||
d.id
|
||
));
|
||
}
|
||
strict_unexpected_details.push(msg);
|
||
total_strict_unexpected += result.strict_unexpected.len();
|
||
}
|
||
|
||
if !result.count_violations.is_empty() {
|
||
let mut msg = format!("COUNT {fixture_label} [{mode_str}]:");
|
||
for (exp, count) in &result.count_violations {
|
||
msg.push_str(&format!(
|
||
"\n COUNT violation: rule_id='{}' severity={:?} line_range={:?} — {} matches exceed max_count={} — {}",
|
||
exp.rule_id,
|
||
exp.severity,
|
||
exp.line_range,
|
||
count,
|
||
exp.max_count.unwrap_or(0),
|
||
exp.notes
|
||
));
|
||
}
|
||
count_violation_details.push(msg);
|
||
total_count_violations += result.count_violations.len();
|
||
}
|
||
|
||
if !result.soft_misses.is_empty() {
|
||
let mut msg = format!("SOFT {fixture_label} [{mode_str}]:");
|
||
for (exp, reason) in &result.soft_misses {
|
||
msg.push_str(&format!("\n soft miss: {} — {}", reason, exp.notes));
|
||
}
|
||
soft_miss_details.push(msg);
|
||
total_soft_misses += result.soft_misses.len();
|
||
}
|
||
|
||
if verbose {
|
||
eprintln!(
|
||
" {fixture_label} [{mode_str}]: {} matched, {} hard misses, {} forbidden, {} count violations, {} strict unexpected, {} soft misses, {} unexpected",
|
||
result.matched,
|
||
result.hard_misses.len(),
|
||
result.forbidden_violations.len(),
|
||
result.count_violations.len(),
|
||
result.strict_unexpected.len(),
|
||
result.soft_misses.len(),
|
||
result.unexpected.len()
|
||
);
|
||
if !result.unexpected.is_empty() {
|
||
for d in &result.unexpected {
|
||
eprintln!(
|
||
" EXTRA: {}:{} [{}] {}",
|
||
d.path,
|
||
d.line,
|
||
d.severity.as_db_str(),
|
||
d.id
|
||
);
|
||
}
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
// Print coverage matrix.
|
||
print_coverage_matrix(fixtures);
|
||
|
||
// Print summary.
|
||
eprintln!("\n────────────────────────────────────────────────────");
|
||
eprintln!(
|
||
"RESULTS: {} matched, {} hard failures, {} forbidden violations, {} count violations, {} strict unexpected, {} soft misses, {} unexpected",
|
||
total_matched,
|
||
total_hard_fails,
|
||
total_forbidden,
|
||
total_count_violations,
|
||
total_strict_unexpected,
|
||
total_soft_misses,
|
||
total_unexpected
|
||
);
|
||
eprintln!("────────────────────────────────────────────────────");
|
||
|
||
if !failure_details.is_empty() {
|
||
eprintln!("\n=== HARD FAILURES (must_match=true) ===");
|
||
for msg in &failure_details {
|
||
eprintln!("{msg}");
|
||
}
|
||
}
|
||
|
||
if !forbidden_details.is_empty() {
|
||
eprintln!("\n=== FORBIDDEN VIOLATIONS (must_not_match=true) ===");
|
||
for msg in &forbidden_details {
|
||
eprintln!("{msg}");
|
||
}
|
||
}
|
||
|
||
if !count_violation_details.is_empty() {
|
||
eprintln!("\n=== COUNT VIOLATIONS (max_count exceeded) ===");
|
||
for msg in &count_violation_details {
|
||
eprintln!("{msg}");
|
||
}
|
||
}
|
||
|
||
if !strict_unexpected_details.is_empty() {
|
||
eprintln!(
|
||
"\n=== STRICT UNEXPECTED (unconsumed diag matched fixture's strict_unexpected prefix) ==="
|
||
);
|
||
for msg in &strict_unexpected_details {
|
||
eprintln!("{msg}");
|
||
}
|
||
}
|
||
|
||
if !soft_miss_details.is_empty() {
|
||
eprintln!("\n=== SOFT MISSES (must_match=false, informational) ===");
|
||
for msg in &soft_miss_details {
|
||
eprintln!("{msg}");
|
||
}
|
||
}
|
||
|
||
// Hard failures, forbidden violations, count violations, and strict
|
||
// unexpected findings all cause failure. Soft misses and unexpected diags
|
||
// outside the strict_unexpected prefix set remain informational.
|
||
assert_eq!(
|
||
total_hard_fails + total_forbidden + total_count_violations + total_strict_unexpected,
|
||
0,
|
||
"{total_hard_fails} expected findings not found (must_match=true); \
|
||
{total_forbidden} forbidden findings present (must_not_match=true); \
|
||
{total_count_violations} count violations (max_count exceeded); \
|
||
{total_strict_unexpected} strict-unexpected diags (unconsumed finding matched a \
|
||
fixture's strict_unexpected prefix). \
|
||
Run with NYX_TEST_VERBOSE=1 for details."
|
||
);
|
||
}
|