mirror of
https://github.com/feder-cr/invisible_playwright.git
synced 2026-06-16 09:05:13 +02:00
test: activate the full e2e (browser-driving) suite + add fetch --force
The 138 @pytest.mark.e2e tests were doubly inactive: deselected by addopts AND skipped without a cached binary — and 3 of the 6 per-file firefox_binary fixtures silently ignored INVPW_BINARY_PATH, so they'd test whatever was cached even when you pointed the suite elsewhere (a false-confidence trap). - Centralize firefox_binary into conftest.py (env INVPW_BINARY_PATH → cache → skip); delete the 6 duplicates. Unify test_webrtc_realness onto the same env. - scripts/run_e2e.py: one command that runs ALL e2e against a chosen binary, with reruns so an under-load interaction flake (dblclick/hover pass 3/3 in isolation) self-heals while a real break fails every attempt. The webrtc e2e fake a TCP-only SOCKS locally, so the suite is offline. This is the MANDATORY pre-release browser gate (local — hosted runners are too interaction-flaky). - Running the suite against firefox-9 surfaced a real gap: `invisible_playwright fetch --force` was unrecognized (the subparser took no args) though the e2e test + docstring expect it. Implement it: drop the cached version dir, refetch. - Add pytest-rerunfailures + playwright to the dev extras. Baseline against firefox-9: 136 passed, 1 skipped (linux_only on win host), 1 was the --force gap now fixed.
This commit is contained in:
parent
67b5e7cd5e
commit
5dac302938
11 changed files with 120 additions and 135 deletions
67
scripts/run_e2e.py
Normal file
67
scripts/run_e2e.py
Normal file
|
|
@ -0,0 +1,67 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Run the FULL e2e suite (every test that opens the browser) against a binary.
|
||||
|
||||
The 138 ``@pytest.mark.e2e`` tests are excluded from the default `pytest` run
|
||||
(`addopts = -m 'not slow and not e2e'`) because they need a real Firefox binary
|
||||
and a display, and they skip themselves when no binary is available. That makes
|
||||
them easy to forget — and "we can't afford for something to not work". This is
|
||||
the gate that runs them all, deliberately, against a chosen binary.
|
||||
|
||||
It is the MANDATORY pre-release e2e gate: run it green against the freshly-built
|
||||
release binary BEFORE un-drafting a firefox-N (alongside the fppro + WebRTC
|
||||
realness gates). It is NOT in the public CI drive-gate — the hosted runners are
|
||||
content-process unstable under a heavy headless interaction sequence (see
|
||||
70-known-bugs / 60-ci-release-pipeline); this runs locally on reliable hardware.
|
||||
|
||||
Flake-resilience: under full-suite load a couple of interaction tests (dblclick,
|
||||
hover/mouseenter) can flake even though they pass 3/3 in isolation, so failures
|
||||
are reran up to twice on the known transient signatures. A genuinely broken
|
||||
binary fails all attempts. The webrtc e2e fake a TCP-only SOCKS locally (no
|
||||
proxy/secrets), so the whole suite is offline.
|
||||
|
||||
Usage:
|
||||
python scripts/run_e2e.py <firefox-binary>
|
||||
python scripts/run_e2e.py # uses $INVPW_BINARY_PATH
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
_RERUN_SIGNATURES = "Timeout|context was destroyed|was detached|not visible|because of a navigation|TargetClosed"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
binary = sys.argv[1] if len(sys.argv) > 1 else os.environ.get("INVPW_BINARY_PATH")
|
||||
if not binary:
|
||||
print("usage: run_e2e.py <firefox-binary> (or set INVPW_BINARY_PATH)", file=sys.stderr)
|
||||
return 2
|
||||
if not Path(binary).exists():
|
||||
print(f"ERROR: binary not found: {binary}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
env = dict(os.environ)
|
||||
# One setting drives the whole suite: conftest's firefox_binary fixture and
|
||||
# the webrtc e2e both resolve from these.
|
||||
env["INVPW_BINARY_PATH"] = binary
|
||||
env["STEALTHFOX_E2E_BINARY"] = binary
|
||||
|
||||
repo = Path(__file__).resolve().parent.parent
|
||||
cmd = [
|
||||
sys.executable, "-m", "pytest",
|
||||
"-m", "e2e",
|
||||
"-o", "addopts=", # override the default 'not e2e' deselection
|
||||
"--reruns", "2", "--reruns-delay", "1",
|
||||
"--only-rerun", _RERUN_SIGNATURES,
|
||||
"-p", "no:cacheprovider",
|
||||
"-q", "--tb=short",
|
||||
] + sys.argv[2:]
|
||||
print(f"[run_e2e] binary={binary}")
|
||||
print(f"[run_e2e] {' '.join(cmd)}")
|
||||
return subprocess.run(cmd, cwd=repo, env=env).returncode
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Loading…
Add table
Add a link
Reference in a new issue