mirror of
https://github.com/feder-cr/invisible_playwright.git
synced 2026-06-10 08:45:13 +02:00
5dac302938
The 138 @pytest.mark.e2e tests were doubly inactive: deselected by addopts AND skipped without a cached binary — and 3 of the 6 per-file firefox_binary fixtures silently ignored INVPW_BINARY_PATH, so they'd test whatever was cached even when you pointed the suite elsewhere (a false-confidence trap). - Centralize firefox_binary into conftest.py (env INVPW_BINARY_PATH → cache → skip); delete the 6 duplicates. Unify test_webrtc_realness onto the same env. - scripts/run_e2e.py: one command that runs ALL e2e against a chosen binary, with reruns so an under-load interaction flake (dblclick/hover pass 3/3 in isolation) self-heals while a real break fails every attempt. The webrtc e2e fake a TCP-only SOCKS locally, so the suite is offline. This is the MANDATORY pre-release browser gate (local — hosted runners are too interaction-flaky). - Running the suite against firefox-9 surfaced a real gap: `invisible_playwright fetch --force` was unrecognized (the subparser took no args) though the e2e test + docstring expect it. Implement it: drop the cached version dir, refetch. - Add pytest-rerunfailures + playwright to the dev extras. Baseline against firefox-9: 136 passed, 1 skipped (linux_only on win host), 1 was the --force gap now fixed.
67 lines
2.7 KiB
Python
67 lines
2.7 KiB
Python
#!/usr/bin/env python3
|
|
"""Run the FULL e2e suite (every test that opens the browser) against a binary.
|
|
|
|
The 138 ``@pytest.mark.e2e`` tests are excluded from the default `pytest` run
|
|
(`addopts = -m 'not slow and not e2e'`) because they need a real Firefox binary
|
|
and a display, and they skip themselves when no binary is available. That makes
|
|
them easy to forget — and "we can't afford for something to not work". This is
|
|
the gate that runs them all, deliberately, against a chosen binary.
|
|
|
|
It is the MANDATORY pre-release e2e gate: run it green against the freshly-built
|
|
release binary BEFORE un-drafting a firefox-N (alongside the fppro + WebRTC
|
|
realness gates). It is NOT in the public CI drive-gate — the hosted runners are
|
|
content-process unstable under a heavy headless interaction sequence (see
|
|
70-known-bugs / 60-ci-release-pipeline); this runs locally on reliable hardware.
|
|
|
|
Flake-resilience: under full-suite load a couple of interaction tests (dblclick,
|
|
hover/mouseenter) can flake even though they pass 3/3 in isolation, so failures
|
|
are reran up to twice on the known transient signatures. A genuinely broken
|
|
binary fails all attempts. The webrtc e2e fake a TCP-only SOCKS locally (no
|
|
proxy/secrets), so the whole suite is offline.
|
|
|
|
Usage:
|
|
python scripts/run_e2e.py <firefox-binary>
|
|
python scripts/run_e2e.py # uses $INVPW_BINARY_PATH
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import subprocess
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
_RERUN_SIGNATURES = "Timeout|context was destroyed|was detached|not visible|because of a navigation|TargetClosed"
|
|
|
|
|
|
def main() -> int:
|
|
binary = sys.argv[1] if len(sys.argv) > 1 else os.environ.get("INVPW_BINARY_PATH")
|
|
if not binary:
|
|
print("usage: run_e2e.py <firefox-binary> (or set INVPW_BINARY_PATH)", file=sys.stderr)
|
|
return 2
|
|
if not Path(binary).exists():
|
|
print(f"ERROR: binary not found: {binary}", file=sys.stderr)
|
|
return 2
|
|
|
|
env = dict(os.environ)
|
|
# One setting drives the whole suite: conftest's firefox_binary fixture and
|
|
# the webrtc e2e both resolve from these.
|
|
env["INVPW_BINARY_PATH"] = binary
|
|
env["STEALTHFOX_E2E_BINARY"] = binary
|
|
|
|
repo = Path(__file__).resolve().parent.parent
|
|
cmd = [
|
|
sys.executable, "-m", "pytest",
|
|
"-m", "e2e",
|
|
"-o", "addopts=", # override the default 'not e2e' deselection
|
|
"--reruns", "2", "--reruns-delay", "1",
|
|
"--only-rerun", _RERUN_SIGNATURES,
|
|
"-p", "no:cacheprovider",
|
|
"-q", "--tb=short",
|
|
] + sys.argv[2:]
|
|
print(f"[run_e2e] binary={binary}")
|
|
print(f"[run_e2e] {' '.join(cmd)}")
|
|
return subprocess.run(cmd, cwd=repo, env=env).returncode
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|