test: activate the full e2e (browser-driving) suite + add fetch --force

The 138 @pytest.mark.e2e tests were doubly inactive: deselected by addopts AND
skipped without a cached binary — and 3 of the 6 per-file firefox_binary
fixtures silently ignored INVPW_BINARY_PATH, so they'd test whatever was cached
even when you pointed the suite elsewhere (a false-confidence trap).

- Centralize firefox_binary into conftest.py (env INVPW_BINARY_PATH → cache →
  skip); delete the 6 duplicates. Unify test_webrtc_realness onto the same env.
- scripts/run_e2e.py: one command that runs ALL e2e against a chosen binary,
  with reruns so an under-load interaction flake (dblclick/hover pass 3/3 in
  isolation) self-heals while a real break fails every attempt. The webrtc e2e
  fake a TCP-only SOCKS locally, so the suite is offline. This is the MANDATORY
  pre-release browser gate (local — hosted runners are too interaction-flaky).
- Running the suite against firefox-9 surfaced a real gap: `invisible_playwright
  fetch --force` was unrecognized (the subparser took no args) though the e2e
  test + docstring expect it. Implement it: drop the cached version dir, refetch.
- Add pytest-rerunfailures + playwright to the dev extras.

Baseline against firefox-9: 136 passed, 1 skipped (linux_only on win host),
1 was the --force gap now fixed.
This commit is contained in:
feder-cr 2026-06-09 15:40:02 +02:00
parent 67b5e7cd5e
commit 5dac302938
11 changed files with 120 additions and 135 deletions

67
scripts/run_e2e.py Normal file
View file

@ -0,0 +1,67 @@
#!/usr/bin/env python3
"""Run the FULL e2e suite (every test that opens the browser) against a binary.
The 138 ``@pytest.mark.e2e`` tests are excluded from the default `pytest` run
(`addopts = -m 'not slow and not e2e'`) because they need a real Firefox binary
and a display, and they skip themselves when no binary is available. That makes
them easy to forget and "we can't afford for something to not work". This is
the gate that runs them all, deliberately, against a chosen binary.
It is the MANDATORY pre-release e2e gate: run it green against the freshly-built
release binary BEFORE un-drafting a firefox-N (alongside the fppro + WebRTC
realness gates). It is NOT in the public CI drive-gate the hosted runners are
content-process unstable under a heavy headless interaction sequence (see
70-known-bugs / 60-ci-release-pipeline); this runs locally on reliable hardware.
Flake-resilience: under full-suite load a couple of interaction tests (dblclick,
hover/mouseenter) can flake even though they pass 3/3 in isolation, so failures
are reran up to twice on the known transient signatures. A genuinely broken
binary fails all attempts. The webrtc e2e fake a TCP-only SOCKS locally (no
proxy/secrets), so the whole suite is offline.
Usage:
python scripts/run_e2e.py <firefox-binary>
python scripts/run_e2e.py # uses $INVPW_BINARY_PATH
"""
from __future__ import annotations
import os
import subprocess
import sys
from pathlib import Path
_RERUN_SIGNATURES = "Timeout|context was destroyed|was detached|not visible|because of a navigation|TargetClosed"
def main() -> int:
binary = sys.argv[1] if len(sys.argv) > 1 else os.environ.get("INVPW_BINARY_PATH")
if not binary:
print("usage: run_e2e.py <firefox-binary> (or set INVPW_BINARY_PATH)", file=sys.stderr)
return 2
if not Path(binary).exists():
print(f"ERROR: binary not found: {binary}", file=sys.stderr)
return 2
env = dict(os.environ)
# One setting drives the whole suite: conftest's firefox_binary fixture and
# the webrtc e2e both resolve from these.
env["INVPW_BINARY_PATH"] = binary
env["STEALTHFOX_E2E_BINARY"] = binary
repo = Path(__file__).resolve().parent.parent
cmd = [
sys.executable, "-m", "pytest",
"-m", "e2e",
"-o", "addopts=", # override the default 'not e2e' deselection
"--reruns", "2", "--reruns-delay", "1",
"--only-rerun", _RERUN_SIGNATURES,
"-p", "no:cacheprovider",
"-q", "--tb=short",
] + sys.argv[2:]
print(f"[run_e2e] binary={binary}")
print(f"[run_e2e] {' '.join(cmd)}")
return subprocess.run(cmd, cwd=repo, env=env).returncode
if __name__ == "__main__":
sys.exit(main())