SurfSense/plans/backend/03f-undetectability-testing.md

Phase 3f — Undetectability & extraction test harness (manual scorecard)

Part of Phase 3 — WebURL Crawler & Crawl Billing. See 00-umbrella-plan.md. Status: ✅ IMPLEMENTED + first baseline run (ci_mvp, 2026-06-30). Harness lives under the proprietary boundary at surfsense_backend/app/proprietary/web_crawler/testbench/ (package: core.py scorecard/closure-cell helper, suite_stealth.py Suite S, suite_extraction.py Suite E, __main__.py CLI, README.md runbook) — it's the moat's measurement tool, so it carries the app/proprietary/LICENSE, not Apache-2. Run: python -m app.proprietary.web_crawler.testbench --suite all [--proxy URL] [--headed] [--no-screenshots]. Suite S builds the StealthyFetcher tier from the shipped build_stealthy_kwargs(get_stealth_config()) (no drift); Suite E drives the real crawl_url against ToS-safe sandboxes with deterministic assertions. Writes timestamped scorecard-*.json/.md + diffs the last run; the whole results/ tree is gitignored (run-local: scorecards, latest.json, screenshots, dumps). Captcha forced OFF (unaided score). Not pytest-collected (live internet + proxy). Manual-only — no CI/automated gating (resolved decision); it's a dev-run scorecard, not a build gate. First baseline: see "First baseline results" below.

Post-implementation notes (reconciled to shipped code): (1) the 03a/03e "FetchStrategy seam" never shipped — levers are a centralized kwargs builder, reflected below; (2) every detection site is now auto-graded from its real DOM verdict — the initial "INFO + screenshot, read manually" fallback was replaced after a first run by per-site parsers written against actual DOM dumps (reCAPTCHA-v3 reads the server "score" JSON, CreepJS the headless-% + boolean spoof tells incl. hasHeadlessWorkerUA, incolumitas the fpscanner FAIL keys + is_datacenter, fingerprint-scan the Bot Risk Score, FingerprintJS the block message, BrowserScan the Normal/Abnormal count, iphey the masthead verdict, Cloudflare the bypass line). INFO is now reserved for purely informational rows (TLS JA3/JA4, exit IP) + the manual browserleaks links; screenshots are still captured as a backstop. Each site is one entry in suite_stealth.py, so tightening a parser is a one-function change. Depends on 03a (the Scrapling tier ladder + CrawlOutcome), 03b (proxy provider), 03e (the stealth levers being tested), and reuses the page_action + closure-cell mechanism shared with 03d.

Convention note. This is dev/operator tooling, not a product code path, so it's untouched by the Phase 1–2 rename (no search_space_id/workspace_id concern). Citations to the gitignored reference checkouts (references/CloakBrowser/, references/Scrapling/) are pinned to what's on disk; locate code by symbol/grep if lines drift.

Objective

A manual, repeatable scorecard that answers one question: how undetectable (and how correct) is our Universal WebURL Crawler right now? It drives the real Scrapling tiers against the industry's standard detection + sandbox sites, parses each site's verdict, and prints a pass/fail + numeric scorecard. Its real job is to quantify the free-stack ceiling (03e) over time so we know — with evidence — when fingerprint maintenance stops being worth it and we should flip the deferred paid-unblocker tier.

It is explicitly two axes, two suites (kept separate so each scales independently):

• Suite S — Stealth / anti-bot (is the crawler detected?): browser-tier fingerprint/bot tests, HTTP/TLS-tier fingerprint, proxy/leak verification.
• Suite E — Extraction correctness (does the crawler get the content right?): scraping sandboxes for the HTTP vs JS (DynamicFetcher) tiers + Trafilatura quality.

How CloakBrowser tests (the pattern we copy)

CloakBrowser's suite is a driven-browser verdict-scraper, not unit tests (references/CloakBrowser/examples/stealth_test.py, run via bin/cloaktest). The repeated shape:

1. Launch a real page with proxy + geoip, toggleable headed/headless (stealth_test.py:231 launch(headless=…, proxy=…, geoip=True)).
2. page.goto(site, wait_until="networkidle") then sleep — scores compute async (Castle.js ~20 s fingerprint_scan_test.py:31; CreepJS ~30 s :96; reCAPTCHA polls up to 30 s stealth_test.py:156–164).
3. page.evaluate(js) to parse the verdict from the rendered DOM (sannysoft table stealth_test.py:32–45) or from internal JS objects (window.Fingerprint.headless fingerprint_scan_test.py:117–127).
4. Apply an explicit pass threshold + screenshot (stealth_test.py:169–217).

Their shipped bars (we adopt as aspirational targets, see Scorecard): sannysoft 0 fails; bot.incolumitas ≤ {WEBDRIVER, connectionRTT} known-FPs; browserscan 0 Abnormal; deviceandbrowserinfo isBot=false; FingerprintJS demo not blocked; reCAPTCHA v3 ≥0.7 (they hit 0.9); CreepJS headless ≤30% / stealth ≤30% (fingerprint_scan_test.py:112–114,166–171).

The bridge to our Scrapling crawler

We are not a single browser; we're the 03a tier ladder. The harness therefore tests per tier, and extracts verdicts two ways:

• DOM/JSON-rendered verdicts → just StealthyFetcher.fetch(url, …) (or Fetcher.get for JSON endpoints) and parse the returned post-JS page with Scrapling's selector (load_dom is on by default — references/Scrapling/scrapling/fetchers/stealth_chrome.py:43). Covers sannysoft, incolumitas, deviceandbrowserinfo, the reCAPTCHA score text, and every JSON endpoint (tls.peet.ws/api/all, httpbin/headers).
• Internal JS-object verdicts → a page_action that runs page.evaluate() and stashes the result into a closure cell, because Scrapling discards page_action's return value (sync _stealth.py:260, async :536). This is the exact same page_action+closure-cell plumbing as 03d's token injector — building the harness de-risks 03d (and vice-versa); factor it once. (As-built: CreepJS/fingerprint-scan turned out to render their verdicts into the visible DOM, so their parsers read the post-JS page text directly; the closure-cell page_action path is retained as the mechanism for any future window.*-only verdict + for the screenshot backstop.)

Suite S — Stealth / anti-bot

S1. Browser tier (StealthyFetcher — the "undetectable" tier)

All rows are auto-graded from the post-JS DOM text (no manual screenshot read); the Extraction column names the actual marker each parser keys on.

Site	Signal	Extraction	Aspirational bar
bot.sannysoft.com	webdriver/chrome/plugins/UA leaks	class="failed" cell count	0 fails
bot.incolumitas.com	30+ checks incl. behavioral + IP class	"<key>":"FAIL" keys, is_datacenter	0 fpscanner FAIL
browserscan.net/bot-detection	WebDriver/CDP/Navigator	Normal/Abnormal count	0 Abnormal
deviceandbrowserinfo.com/are_you_a_bot	fingerprint + behavioral	"isBot": JSON	isBot=false
abrahamjuliot.github.io/creepjs	fingerprint consistency/lies, headless%	N% headless + boolean tells (hasHeadlessWorkerUA…)	headless ≤30%, no tells
fingerprint-scan.com	Castle.js bot-risk + headless signals	Bot Risk Score: N/100	risk < 50
demo.fingerprint.com/web-scraping	behavioral block (FingerprintJS Pro Smart Signals)	block message (access denied / tampering detected)	not blocked
recaptcha-demo.appspot.com/...v3-request-scores.php	Google server-verified human score	server verify "score": JSON	score ≥0.7
www.scrapingcourse.com/cloudflare-challenge	Cloudflare challenge (only row exercising solve_cloudflare)	you bypassed the cloudflare challenge line	bypassed
iphey.com	cross-layer fingerprint+IP+geo coherence	masthead Your Digital Identity Looks <verdict> (async, ~25 s settle)	Trustworthy

S2. Per-property fingerprint detail (manual/debug — validates 03e levers directly)

• browserleaks.com/canvas, /webgl, /fonts — confirms 03e hide_canvas + font packages.
• browserleaks.com/webrtc + DNS leak — confirms 03e block_webrtc (no real-IP leak through the proxy).

S3. HTTP/TLS tier (AsyncFetcher / Fetcher — curl_cffi impersonation)

Our HTTP tier is curl_cffi-based and impersonates a real Chrome TLS stack (references/Scrapling/scrapling/fetchers/requests.py:29; engines/static.py:6–9,36–47) only when impersonate= is set — which 03a now ships (app/proprietary/web_crawler/connector.py, the AsyncFetcher.get call passes impersonate="chrome"). This row therefore validates the shipped parity rather than driving a fix: confirm the static tier's JA3/JA4 matches a real Chrome (the 03e §2b lever). If you ever need a before/after, temporarily drop impersonate= to reproduce the curl_cffi-default (red) baseline.

• tls.peet.ws/api/all (+ /api/clean) — JSON JA3/JA4/Akamai-HTTP2/PeetPrint; diff against a real-Chrome baseline.
• httpbin.co/headers (or httpbingo) — header set/order/UA sanity.

Recommendation (resolved): TLS parity is a first-class axis but an informational threshold, not a hard gate. We record JA3/JA4 and flag drift from the Chrome baseline, but don't "fail" on it — curl_cffi impersonation is strong yet JA-hashes shift across versions, and this is a manual scorecard anyway. Treat a mismatch as a tuning signal (pick a closer impersonate profile), not a regression.

S4. Proxy / leak verification

• httpbin.org/ip — exit IP == the proxy endpoint actually used (capture-once seam from 03b/03e, not a re-rotated get_proxy_url()).
• WebRTC/DNS (S2) — no real-IP leak.

Suite E — Extraction correctness (separate axis)

Purpose-built, ToS-safe sandboxes — validates the HTTP vs DynamicFetcher (JS) tiers and Trafilatura output, independent of stealth:

• books.toscrape.com — static catalog + pagination (HTTP tier; baseline extraction).
• quotes.toscrape.com — has /js (JS-rendered), /js-delayed (?delay=), scroll (infinite), login (CSRF) variants → exercises the DynamicFetcher JS tier + wait_selector/network_idle.
• scrapethissite.com — mixed structures for extraction robustness.

Assertion style: known expected values (e.g. first book title, quote count per page) so extraction regressions are caught deterministically.

First baseline results (2026-06-30, headless, rotating residential proxy)

First real run of the free stack (patchright-Chromium + curl_cffi impersonate="chrome" + anonymous_proxies rotating residential), captcha solving OFF, Slice-A levers only (CRAWL_GEOIP_MATCH_ENABLED=false). Suite S: 6 PASS / 4 FAIL across the 10 detection sites (the other 6 rows are informational: TLS, exit IP, 4 browserleaks manual links).

Site	Verdict	Observed
sannysoft	✅ PASS	0 failed cells
deviceandbrowserinfo	✅ PASS	isBot=false (incl. isPlaywright=false, isAutomatedWithCDP=false)
reCAPTCHA v3	✅ PASS	server score 0.9
BrowserScan	✅ PASS	Test Results: Normal, 0 Abnormal (WebDriver/CDP/Navigator)
fingerprint-scan	✅ PASS	Bot Risk 35/100 (site flags >50)
cloudflare_challenge	✅ PASS	turnstile solved → "you bypassed the Cloudflare challenge"
CreepJS	❌ FAIL	headless 33%, hasHeadlessWorkerUA: true (worker UA leaks HeadlessChrome)
incolumitas	❌ FAIL	only legacy fpscanner WEBDRIVER: FAIL; all modern tests OK
FingerprintJS Pro	❌ FAIL	"anti-detect tampering detected, access denied"
iphey	❌ FAIL	verdict Unreliable

Reading the failures (maps directly to the moat roadmap):

• FingerprintJS Pro = the commercial bar (the documented free-stack ceiling). Needs WebGL/GPU + deeper patches (03e Slice B/C) or the deferred paid-unblocker (03e §8). Hardest.
• CreepJS hasHeadlessWorkerUA = the Web Worker navigator.userAgent still reports HeadlessChrome (main-thread UA is clean). Known patchright leak, plausibly fixable (worker UA override) → concrete 03e Slice-B candidate.
• iphey "Unreliable" = almost certainly geoip incoherence (browser tz/locale default America/Los_Angeles vs the rotating proxy's exit geo, because CRAWL_GEOIP_MATCH_ENABLED is default-off in Slice A). iphey is now a live regression test for the 03e geoip-coherence lever — flipping CRAWL_GEOIP_MATCH_ENABLED is the expected fix to validate.
• incolumitas = a single 2017-era fpscanner WEBDRIVER check that even some real browsers trip; modern checks (webdriverPresent, SELENIUM_DRIVER, HEADCHR_*, CDP) all OK. Lowest priority.

Proxy-quality note: incolumitas's IP classifier returned is_datacenter: true on one rotation and false on another — the anonymous_proxies rotating pool mixes datacenter and residential exits, a real undetectability variable independent of our code (input to future proxy-provider evaluation).

Scorecard & thresholds (resolved)

• Adopt CloakBrowser's bars as aspirational targets (above), but the harness's primary output is our actual measured numbers recorded as the baseline — a scorecard.md/JSON snapshot per run (date, tier, proxy on/off, headed/headless, per-site result), written to the gitignored results/ tree. Subsequent runs diff against the last on-disk baseline so we see drift (ours improving, or a WAF tightening).
• Each row reports: site, tier, verdict, numeric (where applicable), PASS/FAIL vs aspirational bar, and screenshot path.
• A run is summarized as passed/total per suite (like stealth_test.py:314–317), never blocking anything.

Harness design

• Lives under the proprietary boundary at surfsense_backend/app/proprietary/web_crawler/testbench/ (moat measurement tooling — not Apache-2; not collected by the normal pytest run, since it hits the live internet + needs proxies). It is moved here as a coherent package because it can't be cleanly partially moved (a proprietary Suite S would otherwise back-import generic scaffolding from scripts/, a forbidden app→scripts direction). A thin CLI mirrors bin/cloaktest: python -m app.proprietary.web_crawler.testbench [--proxy URL] [--headed] [--suite S|E|all] [--no-screenshots].
• Reuse split (important — crawl_url is not a drop-in here):
  • Suite E drives the real crawl_url end-to-end — extraction correctness is the production path (auto-ladder + Trafilatura markdown is exactly what we want to assert).
  • Suite S does not use crawl_url. Two reasons: (1) crawl_url auto-ladders and stops at the first SUCCESS, so a detection site might be answered by the cheap HTTP tier when we mean to grade the StealthyFetcher tier; (2) crawl_url returns Trafilatura markdown (_build_result), but verdict parsing needs the raw post-JS DOM (and window.* objects need a live page). So Suite S drives the individual Scrapling fetchers directly, per tier.
  • Avoid test-vs-prod drift: Suite S must construct the StealthyFetcher tier from the same centralized stealth-config builder 03e shipped — build_stealthy_kwargs(get_stealth_config()) in app/proprietary/web_crawler/stealth.py (the single source of truth for block_webrtc/hide_canvas/google_search/dns_over_https + geoip locale/timezone_id) — not a hand-rolled kwargs set, otherwise the scorecard grades a browser we don't ship. The static (HTTP) tier's impersonate="chrome" anchor stays hardcoded in the connector (03a §2b); real_chrome/headed/persistent-profile are Slice B/C and not yet in the builder, so the harness exercises today's Slice-A browser.
• Outputs: console summary + screenshots + the scorecard snapshot.
• Runs with the app-wide proxy provider (03b) and 03e levers on, so the scorecard reflects production fetch behavior — except captcha solving, which Suite S forces OFF (CAPTCHA_SOLVING_ENABLED=false): we measure the unaided stealth/score, and it avoids the 03d injector firing paid solves against the reCAPTCHA-demo / FingerprintJS rows. Captcha solving is exercised separately (functional 03d test), not in the stealth scorecard.

The ceiling-decision loop (why this matters)

The scorecard is the evidence base for the moat strategy: re-run it (a) on a cadence and (b) whenever crawls start failing in the wild. When the hard rows (FingerprintJS demo, reCAPTCHA-v3 ≥0.7, and any DataDome/Kasada targets) degrade and 03e levers can't recover them, that's the documented trigger to flip the deferred paid-unblocker tier (03e §8). Without this harness, that decision is guesswork.

Config / dependencies

• No new production dependencies — reuses Scrapling (already pinned) + the crawler. Optionally a tiny dev helper for the scorecard diff (or just stdlib json).
• Needs residential proxy creds (the 03b env) to be meaningful; document a "no-proxy" mode that still runs but is expected to fail the harder rows (datacenter IP).
• A results dir — entirely gitignored (run-local scorecards, latest.json, screenshots, dumps); kept tracked only via its .gitignore.

Work items

1. Shared page_action+closure-cell helper (factored with 03d) for JS-object verdict extraction.
2. Suite S runners (S1 browser, S3 TLS, S4 proxy-leak) with per-site parse + aspirational thresholds; S2 as manual debug links.
3. Suite E runners against the toscrape/scrapethissite sandboxes with deterministic assertions.
4. CLI + scorecard (snapshot writer + diff vs last baseline) mirroring bin/cloaktest ergonomics.
5. Docs: a short runbook (how to run, read the scorecard, interpret the ceiling trigger).

Risks / trade-offs

• Flaky/rate-limited/changing sites. Detection sites move DOM and tighten over time; the harness must tolerate parse misses (report ERROR, not crash — stealth_test.py:298–300) and is manual precisely so flakiness never blocks development.
• Proxy required for realism. Without residential egress the hard rows fail by design (datacenter IP); document this so a red scorecard isn't misread.
• ToS. These are public detection/sandbox sites intended for this purpose; keep volume low and don't hammer.
• Not a guarantee. Passing sannysoft/CreepJS ≠ beating DataDome/Kasada; the scorecard's value is trend + ceiling visibility, not a green checkmark.

Out of scope (hand-offs)

• CI automation — deferred (resolved: manual-only now). If revisited, it needs tolerant thresholds + proxy creds + nightly cadence; the scorecard JSON is designed to make that easy later.
• The levers themselves (03e), captcha solving (03d), proxy provider (03b), billing (03c) — this plan only measures them.
• The deferred paid-unblocker integration (03e §8) — the harness defines its trigger, not its build.