feat: completed init mvp of phase 3

plans/backend/00-umbrella-plan.md

@@ -84,7 +84,7 @@ The Universal WebURL Crawler is the flagship Type-1 data source (**the moat**).
 - **`03c-crawl-billing.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `17bdb0682`)* — Charge crawl credits at **$1 / 1000 successful requests = 1000 micro-USD per successful crawl** (config-driven via `WEB_CRAWL_MICROS_PER_SUCCESS`, retunable with no code change), drawn from the existing credit wallet (`credit_micros_balance`), gated by a new `WEB_CRAWL_CREDIT_BILLING_ENABLED` flag (off for self-hosted). Two surfaces: **connector/pipeline crawls** billed to the **workspace owner** via a dedicated `WebCrawlCreditService` (mirrors `EtlCreditService`'s gate → `check_credits` → `charge_credits`, **not** `billable_call`); **chat scrapes** fold their crawl cost into the chat turn's existing bill (turn accumulator). No DB migration (uses the existing free-form `web_crawl` usage_type).
 - **`03e-stealth-hardening.md`** — The in-house "undetectable" layer on top of Scrapling's default patchright-Chromium stealth. **Geoip coherence** (match browser `locale`/`timezone_id` to the proxy's exit geo), **fingerprint flags** (`hide_canvas`/`block_webrtc`), **persistent per-domain profiles** (`user_data_dir`), **headed execution under Xvfb**, **real fonts** in the worker image, and **DIY behavioral humanization** via `page_action` (the Chromium engine has no built-in `humanize`). Adds a **block classifier** (label Cloudflare/DataDome/Kasada/captcha/empty from the response) + **per-domain strategy memory** (Redis, no migration) so the ladder learns the known-good tier per domain. Defines — but does **not** build — the **deferred paid-unblocker `FetchStrategy`** (ZenRows/ScrapFly/Bright Data) as the config-flagged escape hatch, with its own (later) billing. Honest ceiling: defeats Cloudflare + the moderate long tail, **not** top-tier behavioral fingerprinting (DataDome/Kasada/reCAPTCHA-Enterprise) — that's the deferred tier.
 - **`03d-captcha-solving.md`** *(✅ IMPLEMENTED — `ci_mvp`; `captchatools` page_action + per-attempt `web_crawl_captcha` billing, off by default)* — Covers the captcha types Scrapling does **not** (reCAPTCHA v2/v3, hCaptcha, image) via `captchatools`, **opt-in + off by default**. `captchatools` is **itself** the provider registry (`new_harvester(solving_site=…)` across capmonster/2captcha/anticaptcha/capsolver/captchaai), so we do **not** rebuild a provider hierarchy — our layer is thin: config resolution + a StealthyFetcher `page_action` that detects the sitekey, harvests a token (egressing from the **same** proxy IP as the crawl), and injects it. Scrapling already handles Cloudflare Turnstile (`03a`), and `page_action` runs **after** `solve_cloudflare`, so the tiers compose. The **billing asymmetry** is now **resolved**: a **separate per-attempt** `web_crawl_captcha` unit (solvers charge per attempt regardless of crawl success), attached via a `WebCrawlCreditService` seam in `03c`. `ErrNoBalance` stops solving (no retry loop → avoids IP bans). Requires a paid solver account.
-- **`03f-undetectability-testing.md`** *(MANUAL-only — no CI gating; sequenced last)* — A **manual scorecard harness** that drives the real Scrapling tiers against the industry-standard detection + sandbox sites (modeled on CloakBrowser's `bin/cloaktest` suite) to **quantify the free-stack ceiling** over time. Two labeled axes: **Suite S (stealth/anti-bot)** — browser-tier (`bot.sannysoft`, `bot.incolumitas`, CreepJS, `deviceandbrowserinfo`, FingerprintJS demo, reCAPTCHA-v3 score, `fingerprint-scan`/Castle.js, `browserscan`), HTTP/TLS-tier JA3/JA4 parity (`tls.peet.ws`, **informational** not a gate), and proxy/leak checks (`httpbin/ip`, WebRTC/DNS); **Suite E (extraction correctness)** — toscrape/scrapethissite sandboxes for the HTTP vs JS (DynamicFetcher) tiers. Reuses `03d`'s `page_action`+closure-cell for JS-object verdicts. Adopts CloakBrowser's bars as **aspirational** (sannysoft 0 fails, CreepJS ≤30%, reCAPTCHA ≥0.7) while recording **our actual numbers as the committed baseline**; the scorecard is the documented **trigger** for flipping `03e`'s deferred paid-unblocker tier.
+- **`03f-undetectability-testing.md`** *(✅ IMPLEMENTED + first baseline — `ci_mvp`, MANUAL-only, no CI gating; lives under `app/proprietary/web_crawler/testbench/`)* — A **manual scorecard harness** that drives the real Scrapling tiers against the industry-standard detection + sandbox sites (modeled on CloakBrowser's `bin/cloaktest` suite) to **quantify the free-stack ceiling** over time. Two labeled axes: **Suite S (stealth/anti-bot)** — browser-tier (`bot.sannysoft`, `bot.incolumitas`, CreepJS, `deviceandbrowserinfo`, FingerprintJS demo, reCAPTCHA-v3 score, `fingerprint-scan`/Castle.js, `browserscan`, **`cloudflare-challenge`** exercising `solve_cloudflare`, **`iphey`** geoip-coherence), HTTP/TLS-tier JA3/JA4 parity (`tls.peet.ws`, **informational** not a gate), and proxy/leak checks (`httpbin/ip`, WebRTC/DNS); **Suite E (extraction correctness)** — toscrape/scrapethissite sandboxes for the HTTP vs JS (DynamicFetcher) tiers. **Every detection site is auto-graded from its real DOM verdict** (parsers written against captured dumps; `INFO` reserved for TLS/IP/manual rows). Reuses `03d`'s `page_action`+closure-cell mechanism. Adopts CloakBrowser's bars as **aspirational** (sannysoft 0 fails, CreepJS ≤30%, reCAPTCHA ≥0.7) while recording **our actual numbers as the baseline** (the whole `results/` tree is **gitignored**, run-local). **First baseline (2026-06-30, headless, rotating residential, captcha OFF): Suite S 6 PASS / 4 FAIL** — PASS sannysoft, deviceandbrowserinfo, reCAPTCHA-v3 (0.9), BrowserScan, fingerprint-scan (35/100), cloudflare-challenge; FAIL CreepJS (`hasHeadlessWorkerUA` worker-UA leak → `03e` Slice-B candidate), FingerprintJS Pro (commercial ceiling), iphey ("Unreliable" → expected geoip-coherence fix), incolumitas (legacy `fpscanner WEBDRIVER` only). The scorecard is the documented **trigger** for flipping `03e`'s deferred paid-unblocker tier.
 
 ### Phase 4 — Connector two-type restructure (backend) [`subplans: 04a–04b`]
 
@@ -165,6 +165,7 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta
 - Authenticated/logged-in scraping: **OUT OF SCOPE this MVP** (public data only). Sticky/static proxies + credential management are deferred and paired with the future platform actors (`03b` static-proxy hand-off + Phase 8).
 - Phase 3 stealth-hardening subplan `03e` ADDED: geoip locale/tz coherence, `hide_canvas`/`block_webrtc`, persistent per-domain profiles, headed+Xvfb, fonts, DIY humanization (`page_action`; Chromium engine has no built-in `humanize`), a block classifier, and per-domain strategy memory (Redis, no migration).
 - Phase 3 test-harness subplan `03f` ADDED: **manual-only** (no CI/automated gating now) undetectability + extraction scorecard, modeled on CloakBrowser's `bin/cloaktest`. Two labeled axes (Suite S stealth + Suite E extraction) so they scale independently. Drives the **real** Scrapling tiers (browser + curl_cffi HTTP/TLS), reuses `03d`'s `page_action`+closure-cell for JS-object verdicts. **TLS JA3/JA4 parity = informational axis, not a hard gate.** Adopt CloakBrowser bars as aspirational; record our actual free-stack numbers as the committed baseline. The scorecard is the documented evidence/trigger for flipping `03e`'s deferred paid-unblocker tier.
+- Licensing placement of 03e/03f code (decided during 03f impl, applying the §boundary test): the **stealth kwargs builder + geoip coherence** (`03e` bypass tuning) live **proprietary** at `app/proprietary/web_crawler/stealth.py`; the **block classifier** (passive telemetry, public markers) stays **Apache-2** at `app/utils/crawl/classifier.py` (direct analog of the captcha split: proprietary `captcha.py` logic + Apache-2 `app/utils/captcha/` config). The **03f scorecard harness** moved whole to `app/proprietary/web_crawler/testbench/` (run `python -m app.proprietary.web_crawler.testbench`) — it's the moat's measurement tool and can't be cleanly half-moved (a proprietary Suite S would back-import generic scaffolding from `scripts/`, a forbidden app→scripts direction). The `scripts/e2e_phase3_crawl_billing.py` billing e2e stays in `scripts/` (Apache-2) since it exercises billing, not the stealth moat.
 - Roadmap: WebURL Crawler & Crawl Billing inserted as the new Phase 3; connector two-type → Phase 4; pipelines → Phases 5/6/7.
 - Phase 5 pipelines data model: two new tables `pipelines` (mutable) + `pipeline_runs` (append-only), modeled on `automations`/`automation_runs`; ORM lives in `db.py` next to connectors/folders. `connector_id` nullable (NULL = Phase-7 Uploads), eligibility enforced at create via 04a's `is_pipeline_eligible`. Schedule = `schedule_cron` + `schedule_timezone` (default UTC) + `next_scheduled_at` (cron, matching automations). `pipeline_runs` pre-includes `charged_micros`/`crawls_*`/`result_blob_key` so Phase 6 needs no extra migration. Both tables published to Zero **full-row** (like folders/connectors). Routes reuse `CONNECTORS_*` permissions. Phase 5 ships the data model + API surface only; the `/run` endpoint enqueues a Phase-6 task stub.
 - Phase 7 uploads-as-pipeline: a **singleton "Uploads" pipeline** per workspace (`connector_id NULL`, `save_to_kb=true`), lazily get-or-created (race-safe via a partial unique index `ON pipelines(workspace_id) WHERE connector_id IS NULL`). Each `fileupload`/`folder-upload` request writes a **terminal audit `PipelineRun(trigger=upload, status=succeeded, documents_indexed=<accepted count>)`** — uploads are **route-recorded, not engine-executed** (Phase 6 fails NULL-connector runs by design; existing upload code stays the executor). Best-effort via an **inner** try/except (never 5xx the upload — the route's outer handler would otherwise 500 an already-committed upload). No crawl billing (uploads aren't crawls; `charged_micros` NULL). Per-file ETL truth stays on `Document.status`; accurate roll-up needs the deferred `documents.pipeline_run_id` provenance. Connector `save_to_kb` default stays `False` (opt-in for connectors, mandatory for uploads). Phase 7 also **guards Phase-5's generic CRUD** against the system Uploads pipeline: `POST /pipelines` rejects `connector_id=None` (supersedes Phase 5's permissive create), `/run` and schedule-`PUT` reject NULL-connector pipelines, and Phase 6's scheduler `_claim_due` filters `connector_id IS NOT NULL` as a backstop (so the Uploads pipeline can never be manually-run or scheduled into perpetually-failing runs). See `07-upload-pipeline-kb.md`.
@@ -179,9 +180,9 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta
 | 3 | `03a-crawler-core.md` | **IMPLEMENTED** (`ci_mvp` @ `5c36cd3`) — Firecrawl removed, Scrapling-only 3-tier `CrawlOutcome`, crawler relocated to `app/proprietary/web_crawler/` |
 | 3 | `03b-proxy-expansion.md` | **IMPLEMENTED** (`ci_mvp` @ `6226012`) — `CustomProxyProvider` (BYO single/pool) + registry + bounded rotation-retry |
 | 3 | `03c-crawl-billing.md` | **IMPLEMENTED** (`ci_mvp` @ `17bdb0682`) — `WebCrawlCreditService` (config-driven price) + indexer wiring + chat-turn fold; functional e2e green |
-| 3 | `03e-stealth-hardening.md` | drafted |
+| 3 | `03e-stealth-hardening.md` | **Slice A IMPLEMENTED** (`ci_mvp`) — stealth kwargs builder + geoip coherence (**proprietary** `app/proprietary/web_crawler/stealth.py`) + additive block classifier (Apache-2 `app/utils/crawl/`; `CrawlOutcome.block_type`, incl. static-tier 4xx) + Xvfb/fonts in image; Slices B/C deferred (WebGL spoof, humanization, persistent profiles, strategy memory, paid-unblocker) |
 | 3 | `03d-captcha-solving.md` | **IMPLEMENTED** (`ci_mvp`) — `captchatools` page_action (proprietary) + Apache-2 config + per-attempt `web_crawl_captcha` billing; off by default |
-| 3 | `03f-undetectability-testing.md` | drafted (manual scorecard — sequenced last) |
+| 3 | `03f-undetectability-testing.md` | **IMPLEMENTED** (`ci_mvp`) — manual scorecard under the **proprietary boundary** at `app/proprietary/web_crawler/testbench/` (`python -m app.proprietary.web_crawler.testbench`); Suite S (stealth, shipped builder) + Suite E (extraction via real `crawl_url`) + scorecard JSON/MD baseline diff |
 | 4 | `04a-connector-category.md` | drafted |
 | 4 | `04b-source-discovery.md` | drafted |
 | 5 | `05-pipelines-model.md` | drafted |

plans/backend/03e-stealth-hardening.md

@@ -1,34 +1,52 @@
 # Phase 3e — Stealth hardening (the in-house "undetectable" moat)
 
 > Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
-> Depends on `03a` (Scrapling StealthyFetcher tier + `FetchStrategy` seam + `CrawlOutcome`) and `03b` (app-wide proxy provider). Precedes `03d` (captcha) in the escalation order — hardening **avoids** challenges; captcha solving is the paid last resort for the ones we can't avoid. Touches `03c` only via the deferred paid-unblocker tier (its own billing, decided if/when built).
+> Depends on `03a` (Scrapling StealthyFetcher tier + `CrawlOutcome`) and `03b` (app-wide proxy provider). Precedes `03d` (captcha) in the escalation order — hardening **avoids** challenges; captcha solving is the paid last resort for the ones we can't avoid. Touches `03c` only via the deferred paid-unblocker tier (its own billing, decided if/when built).
+
+> **Architecture reconciliation (read before building).** Earlier drafts of this plan referenced a `FetchStrategy` seam / strategy-object contract `(url, ctx) -> CrawlOutcome`. **That seam does not exist in shipped code.** `03a`–`03d` shipped a *hardcoded 3-tier ladder* inside `WebCrawlerConnector.crawl_url` (`app/proprietary/web_crawler/connector.py`): plain methods `_crawl_with_async_fetcher` / `_crawl_with_dynamic` / `_crawl_with_stealthy(_sync)` wrapped by `_run_tier_with_proxy_retry`, every tier returning the shared `CrawlOutcome`. `03e` therefore implements levers **not** as strategy classes but as: (1) a **centralized per-tier kwargs builder** (`app/proprietary/web_crawler/stealth.py` — **proprietary**, since it's bypass-specific tuning; the generic block classifier stays Apache-2 in `app/utils/crawl/`) that turns config → `StealthyFetcher`/`AsyncFetcher` kwargs, imported by both the connector and `03f`'s harness so there's no test-vs-prod drift; (2) small **helpers around the existing ladder** (block classifier, strategy memory); and (3) the deferred paid tier as a config-gated **extra ladder step**, not a `FetchStrategy`. The captcha precedent holds: generic config/plumbing → `app/utils/`; actual bypass logic (WebGL spoof JS, humanize choreography) → `app/proprietary/web_crawler/`.
 
 > **Implementation note.** Same convention as `03a`–`03d`: Phases 1–2 are **SHIPPED** (live code says `workspace_id`/`Workspace` — substitute for any old `search_space_*`/`SearchSpace` citations and grep the new name); citations also predate `03a`'s `crawl_url` refactor; locate code by **symbol/grep**, not absolute lines. Scrapling references point at the on-disk (gitignored) `references/Scrapling/` checkout pinned to `scrapling[fetchers]>=0.4.9` (`pyproject.toml:91`).
 
+> **Build status.** Sliced for risk (see "Build slicing" below). **Slice A (SHIPPED in this pass):** centralized stealth kwargs builder + fingerprint flags + geoip-from-proxy-location + block classifier (additive `CrawlOutcome.block_type`) + fonts/Xvfb packages in the image. **Slice B / C (deferred):** headed-Xvfb runtime path, humanization/WebGL `init_script` assets, persistent profiles, strategy memory, paid-unblocker stub. Effectiveness of every lever is graded in `03f` (manual), not unit-tested here.
+>
+> **First baseline evidence (`03f`, 2026-06-30 — headless, rotating residential, captcha OFF, Slice-A only with `CRAWL_GEOIP_MATCH_ENABLED=false`).** Suite S = **6 PASS / 4 FAIL** (full table in `03f`). The 4 fails map cleanly onto this plan's deferred levers and confirm the predicted ceiling — they are **not** Slice-A regressions:
+> - **iphey "Unreliable"** → the predicted geoip-incoherence tell (browser tz/locale vs proxy exit geo). It's now the live regression test for **§1 geoip coherence** — flipping `CRAWL_GEOIP_MATCH_ENABLED` is the expected fix to validate next.
+> - **CreepJS `hasHeadlessWorkerUA: true`** (headless 33%) → the Web **Worker** `navigator.userAgent` still leaks `HeadlessChrome` (main-thread UA is clean). A concrete, plausibly-fixable **Slice-B** target (worker UA override via `init_script`), beyond today's Slice-A flags.
+> - **FingerprintJS Pro "tampering detected, access denied"** → the documented **commercial/device-fingerprint wall** (§2c WebGL/GPU gap). Needs Slice B/C or the §8 paid tier; expected to stay red on the free stack.
+> - **incolumitas** → a single legacy `fpscanner WEBDRIVER` check (all modern checks pass); lowest priority. Its IP classifier also flip-flopped `is_datacenter` true/false across rotations, exposing that the `anonymous_proxies` pool mixes datacenter+residential exits (a proxy-provider variable, not a code issue).
+
 ## Objective
 
-Push the Universal WebURL Crawler as far toward "undetectable" as the **free / in-house stack** allows, so we hold a scraping moat for the next ~4–6 months **without** a third-party unblocker (ZenRows/ScrapFly/Bright Data) or a source-patched browser (CloakBrowser — rejected on licensing). Everything here is **runtime/config-level** stealth layered on Scrapling's patchright-Chromium (`03a` engine note): geoip fingerprint coherence, persistent profiles, headed execution, real fonts, and behavioral humanization — plus a **block classifier** + **per-domain strategy memory** so the ladder learns and the cheap tiers get skipped once a domain's working strategy is known. The paid-unblocker tier is defined here as a **deferred** `FetchStrategy` (the explicit escape hatch for when in-house maintenance gets too costly), not built now.
+Push the Universal WebURL Crawler as far toward "undetectable" as the **free / in-house stack** allows, so we hold a scraping moat for the next ~4–6 months **without** a third-party unblocker (ZenRows/ScrapFly/Bright Data) or a source-patched browser (CloakBrowser — rejected on licensing). Everything here is **runtime/config-level** stealth layered on Scrapling's patchright-Chromium (`03a` engine note): geoip fingerprint coherence, persistent profiles, headed execution, real fonts, and behavioral humanization — plus a **block classifier** + **per-domain strategy memory** so the ladder learns and the cheap tiers get skipped once a domain's working strategy is known. The paid-unblocker tier is defined here as a **deferred config-gated ladder step** (the explicit escape hatch for when in-house maintenance gets too costly), not built now.
 
 ## The realistic ceiling (be honest with downstream devs)
 
-Scrapling already ships strong **runtime** stealth by default: `DEFAULT_ARGS + STEALTH_ARGS` (`references/Scrapling/scrapling/engines/constants.py:24–99`, incl. `--disable-blink-features=AutomationControlled` `:94`), a persistent context (`_browsers/_stealth.py:91–93,366–368`), and `navigator.webdriver` masking via patchright. With `03b` residential proxies + this plan's coherence/humanization on top, the crawler reliably handles **Cloudflare** (via `03a` `solve_cloudflare`) and the long tail of **moderate** anti-bot.
+Scrapling already ships strong **runtime** stealth by default: `DEFAULT_ARGS + STEALTH_ARGS` (incl. `--disable-blink-features=AutomationControlled`), a persistent context, and `navigator.webdriver` masking via **patchright**. Be precise about what patchright actually is: it patches the **Playwright driver** to remove *automation leaks* — it never calls `Runtime.enable`, disables `Console.enable`, and reaches closed shadow roots (verified: [patchright-python](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright-python), [Scrappey](https://scrappey.com/qa/web-scraping-apis/what-is-patchright)). It hides **"I'm automated," not "this is a different machine."** It does **not** spoof canvas, WebGL, audio, screen, or GPU. With `03b` residential proxies + this plan's coherence/humanization on top, the crawler reliably handles **Cloudflare** (via `03a` `solve_cloudflare`) and the long tail of **moderate** anti-bot — which is most CI targets.
 
-What the free stack does **not** reliably beat: **DataDome, Kasada, reCAPTCHA-Enterprise**, and other behavioral/device-fingerprint systems. Those generally require C++ source-level browser patches (CloakBrowser's 58-patch approach — out on licensing) or a commercial unblocker. For **competitive-intelligence targets** these top-tier defenses are the exception, not the rule, so the in-house moat is a sound 4–6 month bet — with the deferred paid tier (§8) as the pre-wired escape hatch. **Do not promise "beats everything."**
+**The device-fingerprint wall (the WebGL/GPU gap — see §2c).** Our worker has no GPU, so Chromium's WebGL renderer reports `Google SwiftShader` / `Mesa llvmpipe` — a string anti-bot vendors treat as ">95% bot" ([ipasis](https://ipasis.com/blog/webgl-fingerprinting-bot-detection), [Scrappey](https://scrappey.com/qa/anti-bot/what-is-webgl-fingerprinting)). Scrapling exposes only `allow_webgl` (on/off — and *off* is itself a tell), **no renderer spoof**; CloakBrowser fixes this at the C++ level (`--fingerprint-gpu-renderer`) and we cannot. Concretely, **without a real GPU or a source-patched binary we cannot**:
 
-## Levers (all behind the `03a` `FetchStrategy` seam — callers stay outcome-only)
+1. **Pass pure device-fingerprint bot checks** — FingerprintJS bot-detection, CreepJS / BrowserScan "is this a real device." CloakBrowser passes these (BrowserScan 4/4); our SwiftShader renderer fails them.
+2. **Reach human-level reCAPTCHA v3 *scores*** (~0.7–0.9) **on our own browser** — we land ~0.1–0.5. (Note: `03d`'s **paid solver sidesteps this for token challenges** — the token is minted on the vendor's real-browser infra, so its score isn't ours. The gap only bites where the *site itself* reads our live v3 score.)
+3. **Max out the device-fingerprint component** of DataDome / Kasada / Akamai — one of several reasons we plateau below managed APIs.
+
+**Numbers (2026 benchmarks — use these in any user-facing copy).** Free stack **+ residential**: Basic (CF) ~99%; Medium (CF Pro / PerimeterX) ~90%; **Enterprise (DataDome / Kasada / Akamai) ~60–75% initially, decaying toward ~60% within 24h** as their ML clusters the fingerprint (managed APIs hold ~89%) ([scrapewise DataDome 2026](https://scrapewise.ai/blogs/bypass-datadome-web-scraping-2026)). So the honest line is **"partial and decaying, not SLA-grade,"** — neither "can't beat" nor "beats everything." For CI targets the top-tier defenses are the exception, so the in-house moat is a sound 4–6 month bet, with the deferred paid tier (§8) as the **evidence-driven** escape hatch (`03f`'s scorecard is the trigger). **Do not promise "beats everything."**
+
+## Levers (wired via the centralized kwargs builder + ladder helpers — callers stay outcome-only)
 
 ### 1. Geoip fingerprint coherence (match the browser to the proxy exit IP)
 
 A residential IP in Berlin behind an `en-US`/`America/New_York` browser is an instant tell. Make the fingerprint cohere with the proxy's exit geo:
 
-- Resolve the proxy exit IP → country/locale/timezone (lightweight geoip; e.g. a bundled MaxmindLite/`geoip2` DB or the proxy provider's location knob `RESIDENTIAL_PROXY_LOCATION` from `03b`).
+- **Primary source (Slice A, no network call): the configured `RESIDENTIAL_PROXY_LOCATION` (`03b`, `config:1048`).** Map that string (best-effort: ISO-3166 alpha-2 like `us`/`de`, or a common country name) → a representative `(locale, timezone_id)`. Coarse country granularity is fine (per "Risks": wrong-but-coherent beats default-mismatched); unknown/empty → skip (leave Scrapling's system default). **No exit-IP resolution** — avoids a per-crawl network round-trip + failure mode. (A `geoip2`/MaxmindLite exit-IP path is a later refinement, not MVP.)
-- Pass the matched values into StealthyFetcher: `locale=` (drives `navigator.language` + `Accept-Language` — `stealth_chrome.py:34–35`) and `timezone_id=` (`:36`). These flow into the Playwright context (`_browsers/_base.py:440–441`).
+- Pass the matched values into StealthyFetcher via the kwargs builder: `locale=` (drives `navigator.language` + `Accept-Language`) and `timezone_id=` (confirmed real kwargs on `StealthyFetcher.fetch`). These flow into the Playwright context.
-- This needs the crawl's **chosen proxy endpoint** surfaced into the strategy (the same seam `03d` needs for IP-bound solves) — capture it once in `03a`'s strategy context, don't re-call `get_proxy_url()` (which rotates on a pool-backed provider, `03b`).
+- The stealthy tier **already captures the chosen proxy endpoint once** (`_crawl_with_stealthy_sync`: `proxy = get_proxy_url()`, the same value handed to `03d`'s solver for IP-coherence) — reuse that captured `proxy`/location, **don't** re-call `get_proxy_url()` (which rotates on a pool-backed provider, `03b`).
+- **Caveat (be honest):** Scrapling applies `locale`/`timezone_id` via Playwright **CDP emulation**, which advanced systems can flag *as* emulation — CloakBrowser sets these as *binary flags* precisely to avoid "detectable CDP emulation" (`references/CloakBrowser/README.md:309`). Still strictly better than a mismatched default (wrong-but-coherent beats `UTC`+`en-US` behind a Berlin IP), but it's a weaker variant than a source-patched browser, not parity.
 
 ### 2. Fingerprint flags Scrapling already exposes (turn them on)
 
-- `hide_canvas=True` — random noise on canvas ops to defeat canvas fingerprinting (`stealth_chrome.py:40`).
+- `hide_canvas` — Scrapling adds **random** noise to canvas ops (verified in the installed `StealthyFetcher.fetch` docstring). **Not "free safe":** random-per-call noise is itself a signal on FingerprintJS/CreepJS-class detectors — an *unstable* canvas hash is as loud as `navigator.webdriver=true` ([dev.to](https://dev.to/tanwydd/how-your-canvas-fingerprint-gets-you-caught-and-why-random-noise-makes-it-worse-3ba1)); CloakBrowser's own FPJS fix is `--fingerprint-noise=false`. **Default OFF; gate per-domain and validate in `03f`.** Harmless against Cloudflare/moderate, potentially harmful against device-fingerprint graders. (Canvas stealth that actually *helps* must be **deterministic per persistent profile**, not random — out of scope here.)
-- `block_webrtc=True` — forces WebRTC to respect the proxy, preventing the **real local IP leak** that unmasks proxied browsers (`:41`).
+- `block_webrtc=True` — forces WebRTC to respect the proxy, preventing the **real local IP leak** that unmasks proxied browsers (`:41`). **This one is genuinely cheap + safe — keep default TRUE.**
+- `dns_over_https` — Scrapling can route DNS via Cloudflare DoH to stop the **DNS leak** that unmasks proxied browsers (same anti-leak class as `block_webrtc`; confirmed kwarg on `StealthyFetcher.fetch`). **Trade-off: it adds a DNS round-trip, a (small) latency cost — so per the "no speed regression" constraint it ships *off by default* (`CRAWL_DNS_OVER_HTTPS`, default FALSE), pre-wired for operators who prefer leak-safety over the marginal latency.** (Idea sourced from the Camoufox-based FlareSolverr alternatives `references/Byparr-main` / `references/trawl-dev`, which treat DNS/WebRTC leak coherence as first-class.)
 - `google_search=True` (default) — sets a Google referer so the first hit looks like organic arrival (`:45`); override per-need via `extra_headers` (`:46`).
 - `additional_args=` — last-priority Playwright context overrides for anything not surfaced as a first-class param (`:51`).
 
@@ -36,9 +54,21 @@ A residential IP in Berlin behind an `en-US`/`America/New_York` browser is an in
 
 The cheap static tier (`03a` tier 1) is the **first** thing every crawl hits. `03a` **already ships `impersonate="chrome"`** on it (`app/proprietary/web_crawler/connector.py`, the `AsyncFetcher.get` call) — Scrapling's static engine selects a matching curl_cffi browser profile (`references/Scrapling/scrapling/engines/static.py:36–47`), so the HTTP tier's **JA3/JA4/HTTP-2** already matches a real Chrome and coheres with the browser tiers' UA. `03e`'s remaining work here is **not** to add it, but to (a) fold the `impersonate` profile into the centralized per-tier kwargs builder (below) so it stays the single source of truth, and (b) keep the chosen profile coherent with the proxy exit / UA. `03f §S3` validates the shipped parity against `tls.peet.ws`.
 
+### 2c. WebGL / GPU renderer (the biggest free-stack gap — partial mitigation only)
+
+This is the "device-fingerprint wall" from the ceiling section. On a GPU-less worker the WebGL renderer string is `Google SwiftShader` / `ANGLE (Mesa, llvmpipe…)` — an instant tell on any detector that reads `WEBGL_debug_renderer_info`. Scrapling gives us **no renderer knob** (only `allow_webgl` on/off, and *off* is a tell). Three options, increasing strength + cost:
+
+1. **DIY JS spoof (optional, OFF by default — Slice B).** An `init_script` overrides `getParameter` for `UNMASKED_VENDOR_WEBGL` / `UNMASKED_RENDERER_WEBGL` to a believable, **platform-coherent** value (e.g. an Intel/ANGLE renderer that matches the spoofed Windows UA), held **consistent per persistent profile** (§3). **Impl note:** Scrapling's `init_script` is **an absolute path to a `.js` file**, *not* an inline string (confirmed in `StealthyFetcher.fetch` docstring) — so this ships as a **bundled `.js` asset** in `app/proprietary/web_crawler/` (next to `03d`'s captcha logic) passed by absolute path; the kwargs builder resolves the path when `CRAWL_WEBGL_SPOOF_ENABLED`. **Must** also patch `Function.prototype.toString` so the override still returns `[native code]` — otherwise *the spoof itself is the signal* (exactly the `toString()` probe Kasada catalogs, [web-scraping-guide](https://web-scraping-guide.com/)). This defeats **string-based** WebGL checks (the common case) but **not** render-output pixel-hash or GPU-perf probes (FingerprintJS-grade) — a software rasterizer's actual draw output still differs and can't be faked in JS. It is the brittle config-level approach patchright/CloakBrowser explicitly warn rots; treat it as a **cheap, test-gated win, not the moat**.
+2. **Real GPU passthrough (hosted workers only).** `--device /dev/dri` + ANGLE/EGL so WebGL reports a *genuine* renderer with *real* render output, defeating even output/perf probes. Best technical fix; costs GPU instances + image complexity ([livekit/egress GPU notes](https://stackoverflow.com/questions/78985698)). A hosted-tier escalation, off for self-hosted.
+3. **Defer to the paid unblocker (§8).** Route the residual device-fingerprint-gated domains (flagged by the §7 classifier as in-house-unreachable) to the external unblocker. The pragmatic moat-bounding fallback.
+
+**Decision for this MVP:** ship option 1 as an opt-in flag (`CRAWL_WEBGL_SPOOF_ENABLED`, default FALSE), name option 2 as the hosted escalation, and lean on §8 for the rest. Quantify all three in `03f`.
+
 ### 3. Persistent per-domain profiles (look like a returning human)
 
-Scrapling defaults to a **temporary** user-data dir (fresh = suspicious). Use `user_data_dir=` (`stealth_chrome.py:48`; `launch_persistent_context` `_stealth.py:93,366–368`) to keep a **persistent profile per domain** (or per domain+proxy-geo), so cookies/localStorage/site-trust carry across crawls and the browser presents as a returning visitor rather than a brand-new incognito session. Store profiles under a configured dir (`shared_tmp`-style volume so API + worker share them).
+Scrapling defaults to a **temporary** user-data dir (fresh = suspicious). Use `user_data_dir=` (confirmed real kwarg; `launch_persistent_context`) to keep a **persistent profile per domain** (or per domain+proxy-geo), so cookies/localStorage/site-trust carry across crawls and the browser presents as a returning visitor rather than a brand-new incognito session. Store profiles under a configured dir (`shared_tmp`-style volume so API + worker share them).
+
+- **Concurrency hazard (Slice C design item — why this is deferred, not a one-liner):** Chromium **hard-locks** a `user_data_dir`; two crawls touching the same profile dir at once crash (`SingletonLock`). Our crawler runs concurrently (multiple connectors/chats across workers), so a naive per-domain dir collides. The fix needs one of: (a) an async lock keyed by profile dir so same-domain crawls serialize (adds latency, can deadlock under load), (b) per-`(domain, worker, pid)` dirs + a periodic prune (loses some cross-run trust, no contention), or (c) persistent only when the strategy memory (§7) says a domain *needs* it, temp otherwise. Decide in Slice C alongside §7; **Slice A keeps Scrapling's temp default.**
 
 ### 4. Headed execution under Xvfb (defeat headless tells)
 
@@ -46,28 +76,30 @@ Scrapling defaults to a **temporary** user-data dir (fresh = suspicious). Use `u
 
 ### 5. Real fonts (canvas/emoji hash realism)
 
-A minimal container has almost no fonts, making canvas/emoji fingerprint hashes obviously synthetic. Install real font packages (DejaVu, Liberation, Noto incl. CJK + emoji) in the worker image so font-enumeration + canvas hashes resemble a real desktop. `Dockerfile` change only (`apt-get install fonts-*`).
+A minimal container has almost no fonts, making canvas/emoji fingerprint hashes obviously synthetic. Install real font packages so font-enumeration + canvas/emoji hashes resemble a real desktop. Use the set CloakBrowser ships as proven against Kasada/Akamai emoji-canvas hashing (`README.md:737–739`): `fonts-noto-color-emoji`, `fonts-unifont`, `fonts-ipafont-gothic` (CJK), `fonts-wqy-zenhei` (CJK), `fonts-tlwg-loma-otf` — plus `fonts-dejavu`/`fonts-liberation` for Latin coverage. `Dockerfile` change only (`apt-get install fonts-*`).
 
 ### 6. Behavioral humanization (DIY — Scrapling has no `humanize` for the Chromium engine)
 
 Unlike Camoufox, the patchright-Chromium StealthyFetcher exposes **no built-in `humanize`** (verified: zero matches in the StealthyFetcher param set). So humanization is custom, injected via the existing hooks:
 
-- `page_action=` (runs after navigation, `stealth_chrome.py:30`; engine `_stealth.py:258–262`) — randomized mouse moves/curves, scrolls, hover-before-click, and small think-time delays before extraction. This is the **same hook `03d` uses** for token injection, so the two compose (humanize → optional captcha solve).
+- `page_action=` (runs after navigation) — randomized mouse moves/curves, scrolls, hover-before-click, and small think-time delays before extraction. This is the **same hook `03d` uses** for token injection — `_crawl_with_stealthy_sync` currently builds exactly **one** `page_action` (the captcha injector), so Slice B must **compose** two callables (humanize → optional captcha solve) into the single `page_action` kwarg, not overwrite it.
-- `init_script=` (JS executed on page creation, `:33`) — early shims for any residual JS tells not covered by patchright.
+- `init_script=` (an **absolute path to a `.js` file** executed on page creation — *not* inline JS, see §2c) — early shims for any residual JS tells not covered by patchright (and the §2c WebGL spoof, if enabled).
 - Tunable `wait`/`network_idle` (`:29,27`) so dwell time isn't robotically constant.
+- **Concrete rules (from CloakBrowser's reCAPTCHA notes, `README.md:1280–1314` — encode these):** use native `time.sleep`, **never** `page.wait_for_timeout()` (it emits CDP traffic detectors flag); prefer `page.type()` over `fill()` for any input (real keystrokes vs. value-set); dwell **15s+** before triggering score-based checks; space repeat hits to score-gated domains **30s+**; keep a **stable identity per domain** (persistent profile §3 + consistent spoofed values §2c) so a score-based system sees a *returning* device, not a fresh one each visit.
 
 ### 7. Block classifier + per-domain strategy memory (the "learning ladder")
 
 Two small in-house pieces make the ladder smart instead of brute-force:
 
-- **Block classifier** — inspect each `Response` (status + body/cookie markers) and label the outcome: `OK` / `CLOUDFLARE` / `CAPTCHA_RECAPTCHA` / `CAPTCHA_HCAPTCHA` / `DATADOME` / `KASADA` / `RATE_LIMITED` / `EMPTY`. (Markers: `"Just a moment"` + `cf-mitigated` → Cloudflare; `datadome` cookie/script → DataDome; `g-recaptcha`/`h-captcha` sitekeys → captcha; etc.) The label drives escalation (which tier/lever next), routes captcha types to `03d`, and feeds the memory below. It also makes `CrawlOutcome.status` decisions principled instead of "empty == fail".
+- **Block classifier (Slice A — additive, pure, unit-testable).** Inspect the fetched page (`status` + body markers; cookies/headers are not threaded through `_build_result` today, so MVP is **status + body-marker based**) and label the outcome: `OK` / `CLOUDFLARE` / `CAPTCHA_RECAPTCHA` / `CAPTCHA_HCAPTCHA` / `DATADOME` / `KASADA` / `RATE_LIMITED` / `EMPTY` / `UNKNOWN`. **Concrete markers — adopt the set proven in `references/trawl-dev/packages/tiers/src/detect.ts` (a near-identical classifier):** Cloudflare/DDoS-Guard → `cf-mitigated`-style title strings `"just a moment"` / `"checking your browser"` / `"enable javascript and cookies to continue"` / `"verify you are human"`, DOM ids `challenge-running` / `cf-challenge-running` / `turnstile-wrapper`, and `ddos-guard.net`; Turnstile → `class="cf-turnstile"` / `challenges.cloudflare.com/turnstile`; hCaptcha → `class="h-captcha"` / `hcaptcha.com/1/api`; reCAPTCHA → `class="g-recaptcha"` / `google.com/recaptcha` / `recaptcha.net`; DataDome → `datadome` script/cookie; **status `202`/`403`/`429` as a bot-gate** (`202` is an IMDb-style pre-gate, per `detect.ts:isBlocked`). **Invariant: the classifier is purely *additive* — it attaches `CrawlOutcome.block_type` and feeds telemetry/routing; it MUST NOT change *when* `SUCCESS` is returned, because `03c` bills on `status == SUCCESS` (`CrawlOutcome` docstring).** Wiring mirrors `03d`'s proven `captcha_state` pattern: a per-call `block_state` dict mutated in `_build_result` (which has `raw_html` + `status`) and stamped onto every `CrawlOutcome` return path in `crawl_url`.
-- **Per-domain strategy memory** — cache the **strategy that last succeeded per domain** (e.g. Redis key `crawl:strategy:{domain}` with TTL, reusing the existing Redis the indexer already uses for `indexing_locks`). Next crawl of that domain starts at the known-good tier/lever set, skipping cheaper tiers that always fail there — fewer requests, lower latency, less proxy/captcha spend. No DB migration (Redis, best-effort, self-healing on miss).
+- **Per-domain strategy memory (Slice C — needs a new dependency + control-flow change).** Cache the **strategy that last succeeded per domain** (Redis key `crawl:strategy:{domain}` with TTL). Next crawl starts at the known-good tier, skipping cheaper tiers that always fail there. **Why deferred:** `WebCrawlerConnector()` is currently constructed with **no args** and is Redis/DB-free; this requires injecting a cache client (best-effort, no-op when absent) **and** letting `crawl_url` skip ladder tiers — a control-flow change to the tuned ladder. No DB migration (Redis, best-effort, self-healing on miss), but it is a deliberate design item, not a flag flip.
+- **Solved-session cache (Slice C refinement — speed-positive, the strongest version of the memory above).** Beyond remembering *which tier* won, cache the **solved session itself** — the `cf_clearance`/`__cf_bm` cookies + the UA that cleared the challenge — in Redis (`crawl:session:{domain}` with TTL), and **replay** it on the next crawl via Scrapling's `cookies=` input, so a returning hit **skips the expensive solve entirely** (the pattern `references/trawl-dev` Tier 2 uses for its ~500ms repeat path: `browser/src/session.ts` `session:{domain}` save/load/invalidate; `tiers/src/tier2.ts` replays cookies, and on a re-challenge marks `session-expired` → invalidate → escalate). **Net effect is faster, not slower** (it removes a solve), so it fits the "no speed regression" bar. **Caveat:** `cf_clearance` is **IP+UA-bound**, so it pays off most behind **sticky/static proxies** (the `03b` static-proxy future) and the cache key should include proxy-geo; under rotating residential the replayed cookie may not match the new exit IP (best-effort, self-healing via the `session-expired` invalidate).
 
 ## 8. Deferred: paid-unblocker tier (the escape hatch, NOT built now)
 
 The moat strategy is explicit: **maintain in-house bypass for ~4–6 months, then move hostile targets to a paid unblocker if demand/maintenance justifies it.** That switch is **evidence-driven, not a guess** — `03f`'s manual scorecard quantifies the free-stack ceiling over time and is the documented trigger for flipping this tier. Pre-wire the seam so the switch is a config flip, not a refactor:
 
-- Define (but do not implement) a `PaidUnblockerStrategy` that satisfies the `03a` `FetchStrategy` contract `(url, ctx) -> CrawlOutcome`, appended **last** in the ladder and active only when an env flag + API key are set.
+- Define (but do not implement) a paid-unblocker tier — a `_crawl_with_paid_unblocker(url) -> dict | None` method returning the same shape as the existing tiers, appended **last** in `crawl_url`'s ladder (after the stealthy tier) and active only when an env flag + API key are set. (Same plain-method shape as today's tiers — there is no `FetchStrategy` contract to satisfy.)
 - It would call an external unblocker (ZenRows/ScrapFly/Bright Data Web Unlocker) for the residual `DATADOME`/`KASADA`/`reCAPTCHA-Enterprise` domains the block classifier flags as unreachable in-house.
 - **Its own billing** (cost-plus pass-through, decided at build time) — separate from `03c`'s flat crawl unit and `03d`'s per-solve unit, because unblocker pricing is per-request and provider-specific.
 - Until built, those domains simply return non-`SUCCESS` (free under `03c`). This keeps the umbrella's "WebURL Crawler is the moat" honest while bounding our maintenance risk.
@@ -76,8 +108,10 @@ The moat strategy is explicit: **maintain in-house bypass for ~4–6 months, the
 
 Add (all default OFF / conservative; next to the `03b`/`03c` knobs in `config/__init__.py` + `.env.example`):
 
-- `CRAWL_GEOIP_MATCH_ENABLED` (default FALSE) + optional geoip DB path.
+- `CRAWL_GEOIP_MATCH_ENABLED` (default FALSE; Slice A — maps `RESIDENTIAL_PROXY_LOCATION` → `locale`/`timezone_id`, no exit-IP lookup).
-- `CRAWL_HIDE_CANVAS` / `CRAWL_BLOCK_WEBRTC` (default TRUE — cheap, safe).
+- `CRAWL_BLOCK_WEBRTC` (default TRUE — cheap, safe; Slice A), `CRAWL_HIDE_CANVAS` (default **FALSE** — random canvas noise can itself be a tell; opt-in + `03f`-validated, see §2; Slice A), and `CRAWL_GOOGLE_SEARCH_REFERER` (default TRUE — Scrapling's `google_search`; Slice A).
+- `CRAWL_DNS_OVER_HTTPS` (default **FALSE** — anti DNS-leak, but adds a DNS round-trip; default-off to honor the "no speed regression" bar, pre-wired for leak-safety-first operators; §2; Slice A).
+- `CRAWL_WEBGL_SPOOF_ENABLED` (default FALSE) + optional `CRAWL_WEBGL_VENDOR` / `CRAWL_WEBGL_RENDERER` override strings (§2c; `toString`-safe JS spoof, defeats string-checks only).
 - `CRAWL_PERSISTENT_PROFILES_DIR` (unset → Scrapling's temp default; set → per-domain profiles).
 - `CRAWL_HEADED_XVFB_ENABLED` (default FALSE; requires Xvfb in the image).
 - `CRAWL_HUMANIZE_ENABLED` (default TRUE) + dwell/jitter bounds.
@@ -86,33 +120,47 @@ Add (all default OFF / conservative; next to the `03b`/`03c` knobs in `config/__
 
 ## Docker changes (`surfsense_backend/Dockerfile`)
 
-- Install **Xvfb** + a font set (`fonts-dejavu`, `fonts-liberation`, `fonts-noto`, `fonts-noto-cjk`, `fonts-noto-color-emoji`) in the worker image. (Note from `03a`: also drop the stale "+ Camoufox" comment near `:112`; `scrapling install` only fetches Chromium.)
+- Install **Xvfb** + the proven font set (`fonts-noto-color-emoji`, `fonts-unifont`, `fonts-ipafont-gothic`, `fonts-wqy-zenhei`, `fonts-tlwg-loma-otf`, `fonts-dejavu`, `fonts-liberation`; §5) in the worker image. (Note from `03a`: also drop the stale "+ Camoufox" comment near `:112`; `scrapling install` only fetches Chromium.)
 - Headed runs need the browser launched under `xvfb-run` (or an Xvfb display in the worker entrypoint), gated by `CRAWL_HEADED_XVFB_ENABLED`.
 
+## Build slicing (risk-staged; `03f` validates effectiveness)
+
+The levers split by risk/testability. **Slice A is built now** (prod-safe, config-gated, defaults preserve today's behavior, wiring + classifier unit-tested). **B/C are deferred** (need infra or architectural design; mostly only verifiable in `03f`).
+
+- **Slice A (SHIPPED):** centralized kwargs builder → fingerprint flags (`block_webrtc` ON, `hide_canvas` OFF, `google_search` ON, `dns_over_https` OFF); geoip coherence from `RESIDENTIAL_PROXY_LOCATION`; fonts + Xvfb **packages** in the Dockerfile; **additive** block classifier (`CrawlOutcome.block_type`, classified in `_build_result` **and** the static tier's `>=400` early-return so the first/cheapest-tier bot-gate isn't lost). **Licensing split:** the stealth kwargs builder is **proprietary** (`app/proprietary/web_crawler/stealth.py` — bypass tuning); the block classifier stays **Apache-2** (`app/utils/crawl/classifier.py` — generic passive telemetry from public markers). **`extra_headers`/`additional_args` were intentionally NOT wired** — they're last-priority override hatches with no clean env representation (dict-in-env) and no concrete need yet; add them when a real use appears rather than speculative config (YAGNI).
+- **Slice B (next):** headed-Xvfb **runtime path** (`headless=False` + entrypoint), humanization `page_action` (composed with `03d`'s injector), WebGL-spoof `init_script` `.js` asset (proprietary).
+- **Slice C (defer):** per-domain persistent profiles (concurrency design, §3), per-domain strategy memory + tier-skipping (`crawl_url` control-flow change + cache injection, §7), **solved-session cache** (`cf_clearance` replay via `cookies=`, §7 — speed-positive, best with sticky proxies), **warm browser pool** (a hosted-scale latency win — `references/trawl-dev` `browser/src/pool.ts` keeps N warm instances with an acquire-timeout→429; Scrapling's `StealthySession` already has page pooling), paid-unblocker config stub (§8). GPU passthrough stays infra/hosted-only (§2c).
+
 ## Work items
 
-1. **Geoip coherence** — resolve proxy exit geo → `locale`/`timezone_id`; thread the crawl's chosen endpoint into the strategy context (shared seam with `03d`).
+**Slice A (this pass):**
-2. **Fingerprint flags** — wire `hide_canvas`/`block_webrtc`/`google_search`/`extra_headers`/`additional_args` from config into the StealthyFetcher tier. The AsyncFetcher (HTTP) tier **already carries `impersonate="chrome"`** (shipped in `03a`, §2b) — **fold it into** the builder below, don't re-add it. **Centralize this into a single per-tier kwargs builder** (one function that returns the StealthyFetcher / AsyncFetcher kwargs from config) — the crawler *and* `03f`'s harness both import it, so the scorecard grades the exact browser we ship (no test-vs-prod drift).
+1. **Config + `.env.example`** — add the Slice-A knobs (below); geoip reuses `RESIDENTIAL_PROXY_LOCATION`.
-3. **Persistent profiles** — per-domain `user_data_dir` under `CRAWL_PERSISTENT_PROFILES_DIR` (shared volume).
+2. **`app/proprietary/web_crawler/stealth.py`** (**proprietary** — bypass tuning) — `StealthConfig` + `get_stealth_config()` snapshot; coarse `RESIDENTIAL_PROXY_LOCATION → (locale, timezone_id)` map; `build_stealthy_kwargs(cfg)` returning the config-derived `StealthyFetcher` kwargs. Single source of truth imported by the connector **and** `03f`'s harness (no test-vs-prod drift). The AsyncFetcher tier **already carries `impersonate="chrome"`** (shipped `03a`, §2b) — leave it; document it as the static-tier coherence anchor.
-4. **Headed + Xvfb** — `headless=False` path gated by flag; Xvfb + fonts in the worker image.
+3. **`app/utils/crawl/classifier.py`** (Apache-2) — `BlockType` enum + `classify_block(status, html) -> BlockType` (pure, status + body-marker based).
-5. **Humanization** — a `page_action` humanizer (mouse/scroll/dwell) composing with `03d`'s injector; optional `init_script` shims.
+4. **Connector wiring** — merge `build_stealthy_kwargs(...)` into `_crawl_with_stealthy_sync`'s kwargs (existing `headless`/`network_idle`/`block_ads`/`solve_cloudflare`/`proxy`/captcha `page_action` preserved; lever keys never collide); add `block_type` to `CrawlOutcome`; thread a per-call `block_state` dict (classify in `_build_result` **and** the static tier's `>=400` early-return path, stamp on every `crawl_url` return) — **additive only, SUCCESS predicate unchanged**.
-6. **Block classifier** — `classify(response) -> BlockType`, used by the ladder + `CrawlOutcome.status` + `03d` routing.
+5. **Docker** — Xvfb + the proven font set in the worker image; drop the stale "+ Camoufox" comment.
-7. **Per-domain strategy memory** — Redis read/write around the ladder; start at known-good tier; self-heal on miss.
+6. **Tests + lints** — unit-test the builder (geoip map, flag wiring, snapshot) + classifier (each marker); update connector tests for `block_type`; run suite.
-8. **Paid-unblocker seam** — define the deferred `FetchStrategy` + config flag only (no provider integration).
+
-9. **Instrumentation** — log `(domain, block_type, winning_strategy, attempts, latency)` for tuning the ladder.
+**Slice B / C (deferred — tracked above):** WebGL-spoof `init_script`, humanization composer, headed-Xvfb runtime, persistent profiles, strategy memory + tier-skip, paid-unblocker stub, richer `(domain, block_type, winning_strategy, attempts, latency)` instrumentation.
-10. **Config + Docker + tests**.
 
 ## Risks / trade-offs
 
 - **Arms race / maintenance.** Fingerprint bypasses rot as WAFs update; this is exactly the cost the deferred paid tier (§8) hedges. Instrumentation (work item 9) plus **`03f`'s scorecard** are what tell us when in-house upkeep stops being worth it.
+- **WebGL/GPU is a structural gap, not a bug (§2c).** The free stack cannot present a real consumer GPU; the JS spoof defeats string checks but not render-output/perf probes, and is itself brittle (rots on Chrome updates, must stay `toString`-clean). Plan for device-fingerprint-grade targets to need GPU-passthrough workers or the §8 paid tier — don't expect parity with CloakBrowser here.
 - **Headed/Xvfb cost.** Headful browsers use more CPU/RAM than headless; gate per-flag and only escalate to headed when the block classifier says cheaper tiers fail for a domain (per-domain memory keeps it from being the default).
 - **Profile growth.** Persistent profiles accumulate disk; add a size/TTL cap and periodic prune.
 - **Geoip accuracy.** A coarse country→locale map is fine; over-fitting per-city tz isn't worth it. Wrong-but-coherent beats default-mismatched.
 - **No silver bullet.** Reiterate the ceiling (§"realistic ceiling") in any user-facing copy: the crawler is "best-effort undetectable," not guaranteed.
 
+## Prior art evaluated — why not Camoufox (decision: skip for now)
+
+`references/Byparr-main` and `references/trawl-dev` are FlareSolverr alternatives built on **Camoufox** — an **MPL-2.0**, open-source, **C++/Juggler-patched Firefox** that natively closes our biggest gaps: hardware-accurate **WebGL** renderer/vendor spoofing (bundled `webgl_data.db`, §2c), **`geoip=True`** locale/tz/geo coherence from the proxy IP (§1), **`humanize=True`** mouse, plus font/WebRTC/audio/screen spoofing — all at the engine level (no detectable JS shims). It is effectively the CloakBrowser-class capability *without* CloakBrowser's licensing problem.
+
+**Decision (this MVP): do not adopt Camoufox.** Rationale: Scrapling **dropped** Camoufox in favor of patchright-Chromium for its `StealthyFetcher` ([scrapling stealthy docs](https://scrapling.readthedocs.io/en/latest/fetching/stealthy.html); confirmed in source — `references/Scrapling/scrapling/engines/_browsers/_stealth.py` imports `patchright`), so wiring Camoufox would mean a **separate proprietary fetch tier** (camoufox+playwright, outside Scrapling), a Firefox engine (a few Chrome-only targets won't fit), a heavy extra binary, and tracking an actively-but-experimentally-developed project. **Revisit trigger:** if `03f`'s scorecard shows the patchright tier's device-fingerprint ceiling (§2c) is blocking real CI targets, a Camoufox tier is the prime free escalation to evaluate **before** the §8 paid unblocker.
+
 ## Out of scope (hand-offs)
 
-- Cloudflare solving (`03a`), reCAPTCHA/hCaptcha solving + its per-solve billing (`03d`), proxy rotation (`03b`), flat crawl billing (`03c`).
+- Cloudflare solving (`03a`), reCAPTCHA/hCaptcha solving + its per-solve billing (`03d`), proxy rotation (`03b`), flat crawl billing (`03c`). **Forward note for `03d` (free solver fallback):** the Camoufox-based references solve reCAPTCHA-v2 / Turnstile **free** in-browser (audio-STT — `references/trawl-dev/packages/tiers/src/solvers/{stt,turnstile}.ts`) instead of a paid solver. `03d` shipped with paid `captchatools` as primary; a free STT path is a possible *fallback-before-paid* later, but audio STT is brittle/Google-rate-limited, so paid stays primary (and this doesn't change `03e`).
 - **Logged-in / account-based bypass** (sticky/static proxies + credential management) — deferred to the platform-actor work (umbrella Phase 8 + `03b` static-proxy hand-off). Public data only this MVP.
 - Building the paid-unblocker provider integration — deferred (§8 leaves only the seam + flag).
 - **Measuring** undetectability (the scorecard that grades these levers) → `03f` (manual harness). This plan *builds* the levers; `03f` *tests* them.

plans/backend/03f-undetectability-testing.md

@@ -1,7 +1,9 @@
 # Phase 3f — Undetectability & extraction test harness (manual scorecard)
 
 > Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
-> **Status: ACTIVE, sequenced last in Phase 3** (after `03a`–`03e` ship — the harness measures what those built). **Manual-only** — no CI/automated gating for now (resolved decision); it's a dev-run scorecard, not a build gate. Depends on `03a` (the Scrapling tiers + `FetchStrategy`/`CrawlOutcome`), `03b` (proxy provider), `03e` (the stealth levers being tested), and reuses the `page_action` + closure-cell mechanism shared with `03d`.
+> **Status: ✅ IMPLEMENTED + first baseline run (`ci_mvp`, 2026-06-30).** Harness lives under the **proprietary boundary** at `surfsense_backend/app/proprietary/web_crawler/testbench/` (package: `core.py` scorecard/closure-cell helper, `suite_stealth.py` Suite S, `suite_extraction.py` Suite E, `__main__.py` CLI, `README.md` runbook) — it's the moat's measurement tool, so it carries the `app/proprietary/LICENSE`, not Apache-2. Run: `python -m app.proprietary.web_crawler.testbench --suite all [--proxy URL] [--headed] [--no-screenshots]`. Suite S builds the StealthyFetcher tier from the **shipped** `build_stealthy_kwargs(get_stealth_config())` (no drift); Suite E drives the real `crawl_url` against ToS-safe sandboxes with deterministic assertions. Writes timestamped `scorecard-*.json`/`.md` + diffs the last run; the **whole `results/` tree is gitignored** (run-local: scorecards, `latest.json`, screenshots, dumps). Captcha forced OFF (unaided score). **Not** pytest-collected (live internet + proxy). **Manual-only** — no CI/automated gating (resolved decision); it's a dev-run scorecard, not a build gate. **First baseline: see "First baseline results" below.**
+>
+> **Post-implementation notes (reconciled to shipped code):** (1) the `03a`/`03e` "`FetchStrategy` seam" never shipped — levers are a centralized kwargs builder, reflected below; (2) **every detection site is now auto-graded from its real DOM verdict** — the initial "INFO + screenshot, read manually" fallback was replaced after a first run by per-site parsers written against actual DOM dumps (reCAPTCHA-v3 reads the server `"score"` JSON, CreepJS the headless-% + boolean spoof tells incl. `hasHeadlessWorkerUA`, incolumitas the fpscanner FAIL keys + `is_datacenter`, fingerprint-scan the `Bot Risk Score`, FingerprintJS the block message, BrowserScan the Normal/Abnormal count, iphey the masthead verdict, Cloudflare the bypass line). `INFO` is now reserved for purely informational rows (TLS JA3/JA4, exit IP) + the manual browserleaks links; screenshots are still captured as a backstop. Each site is one entry in `suite_stealth.py`, so tightening a parser is a one-function change. Depends on `03a` (the Scrapling tier ladder + `CrawlOutcome`), `03b` (proxy provider), `03e` (the stealth levers being tested), and reuses the `page_action` + closure-cell mechanism shared with `03d`.
 
 > **Convention note.** This is **dev/operator tooling**, not a product code path, so it's untouched by the Phase 1–2 rename (no `search_space_id`/`workspace_id` concern). Citations to the gitignored reference checkouts (`references/CloakBrowser/`, `references/Scrapling/`) are pinned to what's on disk; locate code by **symbol/grep** if lines drift.
 
@@ -30,22 +32,26 @@ Their shipped bars (we adopt as **aspirational targets**, see Scorecard): sannys
 We are **not** a single browser; we're the `03a` tier ladder. The harness therefore tests **per tier**, and extracts verdicts two ways:
 
 - **DOM/JSON-rendered verdicts** → just `StealthyFetcher.fetch(url, …)` (or `Fetcher.get` for JSON endpoints) and parse the **returned post-JS page** with Scrapling's selector (`load_dom` is on by default — `references/Scrapling/scrapling/fetchers/stealth_chrome.py:43`). Covers sannysoft, incolumitas, deviceandbrowserinfo, the reCAPTCHA score text, and every JSON endpoint (`tls.peet.ws/api/all`, `httpbin/headers`).
-- **Internal JS-object verdicts** (CreepJS `window.Fingerprint`, Castle.js score node) → a **`page_action`** that runs `page.evaluate()` and stashes the result into a **closure cell**, because Scrapling **discards `page_action`'s return value** (sync `_stealth.py:260`, async `:536`). **This is the exact same `page_action`+closure-cell plumbing as `03d`'s token injector** — building the harness de-risks `03d` (and vice-versa); factor it once.
+- **Internal JS-object verdicts** → a **`page_action`** that runs `page.evaluate()` and stashes the result into a **closure cell**, because Scrapling **discards `page_action`'s return value** (sync `_stealth.py:260`, async `:536`). **This is the exact same `page_action`+closure-cell plumbing as `03d`'s token injector** — building the harness de-risks `03d` (and vice-versa); factor it once. *(As-built: CreepJS/fingerprint-scan turned out to render their verdicts into the visible DOM, so their parsers read the post-JS page text directly; the closure-cell `page_action` path is retained as the mechanism for any future `window.*`-only verdict + for the screenshot backstop.)*
 
 ## Suite S — Stealth / anti-bot
 
 ### S1. Browser tier (StealthyFetcher — the "undetectable" tier)
 
+All rows are **auto-graded** from the post-JS DOM text (no manual screenshot read); the `Extraction` column names the actual marker each parser keys on.
+
 | Site | Signal | Extraction | Aspirational bar |
 |---|---|---|---|
-| `bot.sannysoft.com` | webdriver/chrome/plugins/UA leaks | DOM table | 0 fails |
+| `bot.sannysoft.com` | webdriver/chrome/plugins/UA leaks | `class="failed"` cell count | 0 fails |
-| `bot.incolumitas.com` | 30+ checks incl. behavioral | JSON-in-body | ≤ `{WEBDRIVER, connectionRTT}` |
+| `bot.incolumitas.com` | 30+ checks incl. behavioral + IP class | `"<key>":"FAIL"` keys, `is_datacenter` | 0 fpscanner FAIL |
-| `browserscan.net/bot-detection` | WebDriver/CDP/Navigator | DOM text | 0 Abnormal |
+| `browserscan.net/bot-detection` | WebDriver/CDP/Navigator | `Normal`/`Abnormal` count | 0 Abnormal |
-| `deviceandbrowserinfo.com/are_you_a_bot` | fingerprint + behavioral | JSON `isBot` | `isBot=false` |
+| `deviceandbrowserinfo.com/are_you_a_bot` | fingerprint + behavioral | `"isBot":` JSON | `isBot=false` |
-| `abrahamjuliot.github.io/creepjs` | fingerprint **consistency/lies**, headless% | `window.Fingerprint` (page_action) | headless ≤30%, stealth ≤30% |
+| `abrahamjuliot.github.io/creepjs` | fingerprint **consistency/lies**, headless% | `N% headless` + boolean tells (`hasHeadlessWorkerUA`…) | headless ≤30%, no tells |
-| `fingerprint-scan.com` | Castle.js bot-risk + headless signals | DOM node + evaluate | low risk; headless signals 0 |
+| `fingerprint-scan.com` | Castle.js bot-risk + headless signals | `Bot Risk Score: N/100` | risk < 50 |
-| `demo.fingerprint.com/web-scraping` | **behavioral block** (click→blocked vs flights) | page_action click + DOM | not blocked |
+| `demo.fingerprint.com/web-scraping` | **behavioral block** (FingerprintJS Pro Smart Signals) | block message (`access denied` / `tampering detected`) | not blocked |
-| `recaptcha-demo.appspot.com/...v3-request-scores.php` | Google server-verified human score | `wait_for_function` + regex (`recaptcha_score.py:22–28`) | score ≥0.7 |
+| `recaptcha-demo.appspot.com/...v3-request-scores.php` | Google server-verified human score | server verify `"score":` JSON | score ≥0.7 |
+| `www.scrapingcourse.com/cloudflare-challenge` | **Cloudflare challenge** (only row exercising `solve_cloudflare`) | `you bypassed the cloudflare challenge` line | bypassed |
+| `iphey.com` | cross-layer fingerprint+IP+geo coherence | masthead `Your Digital Identity Looks <verdict>` (async, ~25 s settle) | `Trustworthy` |
 
 ### S2. Per-property fingerprint detail (manual/debug — validates `03e` levers directly)
 
@@ -76,31 +82,56 @@ Purpose-built, ToS-safe sandboxes — validates the HTTP vs **DynamicFetcher (JS
 
 Assertion style: known expected values (e.g. first book title, quote count per page) so extraction regressions are caught deterministically.
 
+## First baseline results (2026-06-30, headless, rotating residential proxy)
+
+First real run of the free stack (patchright-Chromium + curl_cffi `impersonate="chrome"` + `anonymous_proxies` rotating residential), captcha solving OFF, Slice-A levers only (`CRAWL_GEOIP_MATCH_ENABLED=false`). **Suite S: 6 PASS / 4 FAIL across the 10 detection sites** (the other 6 rows are informational: TLS, exit IP, 4 browserleaks manual links).
+
+| Site | Verdict | Observed |
+|---|---|---|
+| sannysoft | ✅ PASS | 0 failed cells |
+| deviceandbrowserinfo | ✅ PASS | `isBot=false` (incl. `isPlaywright=false`, `isAutomatedWithCDP=false`) |
+| reCAPTCHA v3 | ✅ PASS | server score **0.9** |
+| BrowserScan | ✅ PASS | Test Results: Normal, 0 Abnormal (WebDriver/CDP/Navigator) |
+| fingerprint-scan | ✅ PASS | Bot Risk **35/100** (site flags >50) |
+| cloudflare_challenge | ✅ PASS | turnstile **solved** → "you bypassed the Cloudflare challenge" |
+| **CreepJS** | ❌ FAIL | headless **33%**, **`hasHeadlessWorkerUA: true`** (worker UA leaks `HeadlessChrome`) |
+| **incolumitas** | ❌ FAIL | only legacy `fpscanner WEBDRIVER: FAIL`; all modern tests OK |
+| **FingerprintJS Pro** | ❌ FAIL | "anti-detect tampering detected, access denied" |
+| **iphey** | ❌ FAIL | verdict **Unreliable** |
+
+**Reading the failures (maps directly to the moat roadmap):**
+- **FingerprintJS Pro** = the commercial bar (the documented free-stack ceiling). Needs WebGL/GPU + deeper patches (`03e` Slice B/C) or the deferred paid-unblocker (`03e §8`). Hardest.
+- **CreepJS `hasHeadlessWorkerUA`** = the **Web Worker** `navigator.userAgent` still reports `HeadlessChrome` (main-thread UA is clean). Known patchright leak, **plausibly fixable** (worker UA override) → concrete `03e` Slice-B candidate.
+- **iphey "Unreliable"** = almost certainly **geoip incoherence** (browser tz/locale default `America/Los_Angeles` vs the rotating proxy's exit geo, because `CRAWL_GEOIP_MATCH_ENABLED` is default-off in Slice A). iphey is now a **live regression test for the `03e` geoip-coherence lever** — flipping `CRAWL_GEOIP_MATCH_ENABLED` is the expected fix to validate.
+- **incolumitas** = a single 2017-era `fpscanner WEBDRIVER` check that even some real browsers trip; modern checks (webdriverPresent, SELENIUM_DRIVER, HEADCHR_*, CDP) all OK. Lowest priority.
+
+**Proxy-quality note:** incolumitas's IP classifier returned `is_datacenter: true` on one rotation and `false` on another — the `anonymous_proxies` rotating pool **mixes datacenter and residential exits**, a real undetectability variable independent of our code (input to future proxy-provider evaluation).
+
 ## Scorecard & thresholds (resolved)
 
-- **Adopt CloakBrowser's bars as aspirational targets** (above), but the harness's primary output is **our actual measured numbers recorded as the baseline** — a committed `scorecard.md`/JSON snapshot per run (date, tier, proxy on/off, headed/headless, per-site result). Subsequent runs diff against the last baseline so we see drift (ours improving, or a WAF tightening).
+- **Adopt CloakBrowser's bars as aspirational targets** (above), but the harness's primary output is **our actual measured numbers recorded as the baseline** — a `scorecard.md`/JSON snapshot per run (date, tier, proxy on/off, headed/headless, per-site result), written to the **gitignored** `results/` tree. Subsequent runs diff against the last on-disk baseline so we see drift (ours improving, or a WAF tightening).
 - Each row reports: site, tier, verdict, numeric (where applicable), PASS/FAIL vs aspirational bar, and screenshot path.
 - A run is summarized as `passed/total` per suite (like `stealth_test.py:314–317`), **never blocking** anything.
 
 ## Harness design
 
-- Lives under dev tooling (e.g. `surfsense_backend/scripts/crawler_testbench/` or `tests/manual/` — not collected by the normal pytest run, since it hits the live internet + needs proxies). A thin CLI mirrors `bin/cloaktest`: `python -m ... [--proxy URL] [--headed] [--headless] [--suite S|E|all] [--no-screenshots]`.
+- Lives under the **proprietary boundary** at `surfsense_backend/app/proprietary/web_crawler/testbench/` (moat measurement tooling — not Apache-2; not collected by the normal pytest run, since it hits the live internet + needs proxies). It is moved here as a coherent package because it can't be cleanly *partially* moved (a proprietary Suite S would otherwise back-import generic scaffolding from `scripts/`, a forbidden app→scripts direction). A thin CLI mirrors `bin/cloaktest`: `python -m app.proprietary.web_crawler.testbench [--proxy URL] [--headed] [--suite S|E|all] [--no-screenshots]`.
 - **Reuse split (important — `crawl_url` is *not* a drop-in here):**
   - **Suite E** drives the **real `crawl_url`** end-to-end — extraction correctness *is* the production path (auto-ladder + Trafilatura markdown is exactly what we want to assert).
   - **Suite S** does **not** use `crawl_url`. Two reasons: (1) `crawl_url` auto-ladders and stops at the first `SUCCESS`, so a detection site might be answered by the cheap HTTP tier when we mean to grade the **StealthyFetcher** tier; (2) `crawl_url` returns Trafilatura **markdown** (`_build_result`), but verdict parsing needs the **raw post-JS DOM** (and `window.*` objects need a live page). So Suite S drives the **individual Scrapling fetchers directly**, per tier.
-  - **Avoid test-vs-prod drift:** Suite S must construct each fetcher from the **same centralized stealth-config builder** `03e` introduces (the single source of truth for `locale`/`timezone_id`/`hide_canvas`/`block_webrtc`/`impersonate`/profile/headed), **not** a hand-rolled kwargs set — otherwise the scorecard grades a browser we don't ship. (`03e` work item: expose that builder so both the crawler and this harness import it.)
+  - **Avoid test-vs-prod drift:** Suite S must construct the StealthyFetcher tier from the **same centralized stealth-config builder** `03e` shipped — `build_stealthy_kwargs(get_stealth_config())` in `app/proprietary/web_crawler/stealth.py` (the single source of truth for `block_webrtc`/`hide_canvas`/`google_search`/`dns_over_https` + geoip `locale`/`timezone_id`) — **not** a hand-rolled kwargs set, otherwise the scorecard grades a browser we don't ship. The static (HTTP) tier's `impersonate="chrome"` anchor stays hardcoded in the connector (`03a` §2b); `real_chrome`/headed/persistent-profile are Slice B/C and **not** yet in the builder, so the harness exercises today's Slice-A browser.
 - Outputs: console summary + screenshots + the `scorecard` snapshot.
 - Runs with the app-wide proxy provider (`03b`) and `03e` levers on, so the scorecard reflects production fetch behavior — **except captcha solving, which Suite S forces OFF** (`CAPTCHA_SOLVING_ENABLED=false`): we measure the *unaided* stealth/score, and it avoids the `03d` injector firing paid solves against the reCAPTCHA-demo / FingerprintJS rows. Captcha solving is exercised separately (functional `03d` test), not in the stealth scorecard.
 
 ## The ceiling-decision loop (why this matters)
 
-The scorecard is the evidence base for the moat strategy: re-run it (a) on a cadence and (b) whenever crawls start failing in the wild. When the **hard** rows (FingerprintJS demo, reCAPTCHA-v3 ≥0.7, and any DataDome/Kasada targets) degrade and `03e` levers can't recover them, that's the documented trigger to flip the **deferred paid-unblocker `FetchStrategy`** (`03e §8`). Without this harness, that decision is guesswork.
+The scorecard is the evidence base for the moat strategy: re-run it (a) on a cadence and (b) whenever crawls start failing in the wild. When the **hard** rows (FingerprintJS demo, reCAPTCHA-v3 ≥0.7, and any DataDome/Kasada targets) degrade and `03e` levers can't recover them, that's the documented trigger to flip the **deferred paid-unblocker tier** (`03e §8`). Without this harness, that decision is guesswork.
 
 ## Config / dependencies
 
 - **No new production dependencies** — reuses Scrapling (already pinned) + the crawler. Optionally a tiny dev helper for the scorecard diff (or just stdlib `json`).
 - Needs **residential proxy creds** (the `03b` env) to be meaningful; document a "no-proxy" mode that still runs but is expected to fail the harder rows (datacenter IP).
-- A results dir (gitignored screenshots; committed `scorecard` snapshots).
+- A results dir — **entirely gitignored** (run-local scorecards, `latest.json`, screenshots, dumps); kept tracked only via its `.gitignore`.
 
 ## Work items
 

surfsense_backend/.env.example

@@ -369,6 +369,22 @@ TURNSTILE_SECRET_KEY=
 # CAPTCHA_V3_MIN_SCORE=0.7
 # CAPTCHA_V3_ACTION=verify
 
+# =====================================================================
+# Stealth hardening (Phase 3e, Slice A) — runtime/config-level levers on the
+# stealth browser tier. Consumed by app/proprietary/web_crawler/stealth.py.
+# Defaults add no crawl-speed regression and preserve today's behavior.
+# Match browser locale/timezone to RESIDENTIAL_PROXY_LOCATION (no exit-IP lookup).
+# CRAWL_GEOIP_MATCH_ENABLED=FALSE
+# WebRTC respects the proxy (anti local-IP leak). Cheap + safe.
+# CRAWL_BLOCK_WEBRTC=TRUE
+# Random canvas noise (an unstable canvas hash is itself a tell) — opt-in only.
+# CRAWL_HIDE_CANVAS=FALSE
+# Send a Google referer so the first hit looks like organic arrival.
+# CRAWL_GOOGLE_SEARCH_REFERER=TRUE
+# Route DNS via Cloudflare DoH (anti DNS-leak). Adds a DNS round-trip => off by
+# default to avoid any speed regression; enable for leak-safety-first setups.
+# CRAWL_DNS_OVER_HTTPS=FALSE
+
 # File Parser Service
 ETL_SERVICE=UNSTRUCTURED or LLAMACLOUD or DOCLING
 UNSTRUCTURED_API_KEY=Tpu3P0U8iy

surfsense_backend/Dockerfile

@@ -30,6 +30,20 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     libxrender1 \
     dos2unix \
     git \
+    # ── Phase 3e stealth hardening ──────────────────────────────────────────
+    # Xvfb: virtual framebuffer so the stealth browser can run headful
+    # (headless=False) without a real display — many WAFs flag headless Chromium.
+    # Gated at runtime by CRAWL_HEADED_XVFB_ENABLED (Slice B); installed now so
+    # the image is ready. Real font packages make canvas/emoji/font-enumeration
+    # fingerprints resemble a real desktop (set proven against Kasada/Akamai).
+    xvfb \
+    fonts-noto-color-emoji \
+    fonts-unifont \
+    fonts-ipafont-gothic \
+    fonts-wqy-zenhei \
+    fonts-tlwg-loma-otf \
+    fonts-dejavu \
+    fonts-liberation \
     && rm -rf /var/lib/apt/lists/*
 
 RUN which ffmpeg && ffmpeg -version

surfsense_backend/app/config/__init__.py

@@ -1086,6 +1086,38 @@ class Config:
     CAPTCHA_V3_MIN_SCORE = float(os.getenv("CAPTCHA_V3_MIN_SCORE", "0.7"))
     CAPTCHA_V3_ACTION = os.getenv("CAPTCHA_V3_ACTION", "verify")
 
+    # =====================================================================
+    # Phase 3e — Stealth hardening (Slice A): runtime/config-level levers
+    # layered on Scrapling's patchright-Chromium StealthyFetcher tier. All are
+    # consumed by the centralized kwargs builder in
+    # app/proprietary/web_crawler/stealth.py (proprietary — bypass tuning), which
+    # is the single source of truth imported by the crawler AND the 03f harness
+    # (no test-vs-prod drift). Defaults preserve today's behavior /
+    # introduce no crawl-speed regression. See plans/backend/03e-stealth-hardening.md.
+    # =====================================================================
+    # Map the configured proxy region (RESIDENTIAL_PROXY_LOCATION) -> browser
+    # locale/timezone so the fingerprint coheres with the proxy exit geo. No
+    # exit-IP lookup (zero added latency); unknown/empty region => skip.
+    CRAWL_GEOIP_MATCH_ENABLED = (
+        os.getenv("CRAWL_GEOIP_MATCH_ENABLED", "FALSE").upper() == "TRUE"
+    )
+    # Force WebRTC to respect the proxy (prevents real-local-IP leak). Cheap +
+    # safe => default TRUE.
+    CRAWL_BLOCK_WEBRTC = os.getenv("CRAWL_BLOCK_WEBRTC", "TRUE").upper() == "TRUE"
+    # Random canvas noise. An UNSTABLE canvas hash is itself a fingerprint tell,
+    # so default FALSE (opt-in + 03f-validated). See 03e §2.
+    CRAWL_HIDE_CANVAS = os.getenv("CRAWL_HIDE_CANVAS", "FALSE").upper() == "TRUE"
+    # Set a Google referer so the first hit looks like organic arrival.
+    CRAWL_GOOGLE_SEARCH_REFERER = (
+        os.getenv("CRAWL_GOOGLE_SEARCH_REFERER", "TRUE").upper() == "TRUE"
+    )
+    # Route DNS via Cloudflare DoH (anti DNS-leak behind proxies). Adds a DNS
+    # round-trip => default FALSE to honor the "no speed regression" bar; flip on
+    # when leak-safety outweighs the marginal latency.
+    CRAWL_DNS_OVER_HTTPS = (
+        os.getenv("CRAWL_DNS_OVER_HTTPS", "FALSE").upper() == "TRUE"
+    )
+
     # Litellm TTS Configuration
     TTS_SERVICE = os.getenv("TTS_SERVICE")
     TTS_SERVICE_API_BASE = os.getenv("TTS_SERVICE_API_BASE")

surfsense_backend/app/proprietary/web_crawler/connector.py

@@ -34,7 +34,12 @@ from scrapling.engines.toolbelt import is_proxy_error
 from scrapling.fetchers import AsyncFetcher, DynamicFetcher, StealthyFetcher
 
 from app.proprietary.web_crawler.captcha import build_captcha_page_action
+from app.proprietary.web_crawler.stealth import (
+    build_stealthy_kwargs,
+    get_stealth_config,
+)
 from app.utils.captcha import captcha_enabled, get_captcha_config
+from app.utils.crawl import BlockType, classify_block
 from app.utils.proxy import get_proxy_url, is_pool_backed
 
 logger = logging.getLogger(__name__)
@@ -64,6 +69,11 @@ class CrawlOutcome:
     them off the outcome regardless of crawl SUCCESS (the solver charges per
     *attempt*). They are populated only by the StealthyFetcher tier when captcha
     solving is enabled; every other path leaves the defaults (0 / False).
+
+    Phase 3e ``block_type`` is purely *additive* telemetry: the block classifier
+    labels the last fetched page (Cloudflare / captcha / DataDome / rate-limited
+    / ...) for tuning + future escalation routing. It does NOT influence the
+    billable ``SUCCESS`` predicate above.
     """
 
     status: CrawlOutcomeStatus
@@ -72,6 +82,7 @@ class CrawlOutcome:
     tier: str | None = None
     captcha_attempts: int = 0
     captcha_solved: bool = False
+    block_type: BlockType = BlockType.UNKNOWN
 
 
 class WebCrawlerConnector:
@@ -102,11 +113,17 @@ class WebCrawlerConnector:
         # billing can read it regardless of crawl SUCCESS. Per-call (not on
         # ``self``) => safe under concurrent ``crawl_url`` calls.
         captcha_state: dict[str, Any] = {"attempts": 0, "solved": False}
+        # Per-call block-classifier telemetry (03e). ``_build_result`` (the one
+        # place with raw_html + status) classifies each fetched page into here;
+        # crawl_url stamps it onto every outcome. Additive only — never gates
+        # SUCCESS. Per-call (not on ``self``) => concurrency-safe.
+        block_state: dict[str, Any] = {"block_type": BlockType.UNKNOWN}
         try:
             if not validators.url(url):
                 return CrawlOutcome(
                     status=CrawlOutcomeStatus.FAILED,
                     error=f"Invalid URL: {url}",
+                    block_type=block_state["block_type"],
                 )
 
             errors: list[str] = []
@@ -120,7 +137,8 @@ class WebCrawlerConnector:
             try:
                 logger.info(f"[webcrawler] Using Scrapling AsyncFetcher for: {url}")
                 result = await self._run_tier_with_proxy_retry(
-                    "scrapling-static", lambda: self._crawl_with_async_fetcher(url)
+                    "scrapling-static",
+                    lambda: self._crawl_with_async_fetcher(url, block_state),
                 )
                 if result:
                     self._log_tier_outcome(
@@ -131,6 +149,7 @@ class WebCrawlerConnector:
                         status=CrawlOutcomeStatus.SUCCESS,
                         result=result,
                         tier="scrapling-static",
+                        block_type=block_state["block_type"],
                     )
                 reached_without_content = True
                 errors.append("Scrapling static: empty extraction")
@@ -146,7 +165,8 @@ class WebCrawlerConnector:
             try:
                 logger.info(f"[webcrawler] Using Scrapling DynamicFetcher for: {url}")
                 result = await self._run_tier_with_proxy_retry(
-                    "scrapling-dynamic", lambda: self._crawl_with_dynamic(url)
+                    "scrapling-dynamic",
+                    lambda: self._crawl_with_dynamic(url, block_state),
                 )
                 if result:
                     self._log_tier_outcome(
@@ -157,6 +177,7 @@ class WebCrawlerConnector:
                         status=CrawlOutcomeStatus.SUCCESS,
                         result=result,
                         tier="scrapling-dynamic",
+                        block_type=block_state["block_type"],
                     )
                 reached_without_content = True
                 errors.append("Scrapling dynamic: empty extraction")
@@ -181,7 +202,9 @@ class WebCrawlerConnector:
                 logger.info(f"[webcrawler] Using Scrapling StealthyFetcher for: {url}")
                 result = await self._run_tier_with_proxy_retry(
                     "scrapling-stealthy",
-                    lambda: self._crawl_with_stealthy(url, captcha_state),
+                    lambda: self._crawl_with_stealthy(
+                        url, captcha_state, block_state
+                    ),
                 )
                 if result:
                     self._log_tier_outcome(
@@ -194,6 +217,7 @@ class WebCrawlerConnector:
                         tier="scrapling-stealthy",
                         captcha_attempts=captcha_state["attempts"],
                         captcha_solved=captcha_state["solved"],
+                        block_type=block_state["block_type"],
                     )
                 reached_without_content = True
                 errors.append("Scrapling stealthy: empty extraction")
@@ -219,12 +243,14 @@ class WebCrawlerConnector:
                     error=f"No content extracted for {url}. {'; '.join(errors)}",
                     captcha_attempts=captcha_state["attempts"],
                     captcha_solved=captcha_state["solved"],
+                    block_type=block_state["block_type"],
                 )
             return CrawlOutcome(
                 status=CrawlOutcomeStatus.FAILED,
                 error=f"All crawl methods failed for {url}. {'; '.join(errors)}",
                 captcha_attempts=captcha_state["attempts"],
                 captcha_solved=captcha_state["solved"],
+                block_type=block_state["block_type"],
             )
 
         except Exception as e:
@@ -234,6 +260,7 @@ class WebCrawlerConnector:
                 error=f"Error crawling URL {url}: {e!s}",
                 captcha_attempts=captcha_state["attempts"],
                 captcha_solved=captcha_state["solved"],
+                block_type=block_state["block_type"],
             )
 
     async def _run_tier_with_proxy_retry(
@@ -307,7 +334,9 @@ class WebCrawlerConnector:
             total_ms,
         )
 
-    async def _crawl_with_async_fetcher(self, url: str) -> dict[str, Any] | None:
+    async def _crawl_with_async_fetcher(
+        self, url: str, block_state: dict[str, Any] | None = None
+    ) -> dict[str, Any] | None:
         """
         Crawl URL using Scrapling's AsyncFetcher (static HTTP) + Trafilatura.
 
@@ -331,6 +360,13 @@ class WebCrawlerConnector:
 
         status = getattr(page, "status", None)
         if status is not None and status >= 400:
+            # 03e: classify here too — this early return skips _build_result, and
+            # the static tier is the first/cheapest hit, so the 403/429 bot-gate
+            # (the most common block signal) would otherwise never be labeled.
+            if block_state is not None:
+                block_state["block_type"] = classify_block(
+                    status, getattr(page, "html_content", None)
+                )
             logger.info(
                 "%s tier=scrapling-static url=%s fetch_ms=%.1f status=%s outcome=http_error",
                 _PERF,
@@ -347,18 +383,25 @@ class WebCrawlerConnector:
             allow_raw_fallback=False,
             fetch_ms=fetch_ms,
             status=status,
+            block_state=block_state,
         )
 
-    async def _crawl_with_dynamic(self, url: str) -> dict[str, Any] | None:
+    async def _crawl_with_dynamic(
+        self, url: str, block_state: dict[str, Any] | None = None
+    ) -> dict[str, Any] | None:
         """
         Crawl URL using Scrapling's DynamicFetcher (full browser) + Trafilatura.
 
         Runs the sync fetch in a worker thread so it works on any event loop,
         including Windows ``SelectorEventLoop`` which cannot spawn subprocesses.
         """
-        return await asyncio.to_thread(self._crawl_with_dynamic_sync, url)
+        return await asyncio.to_thread(
+            self._crawl_with_dynamic_sync, url, block_state
+        )
 
-    def _crawl_with_dynamic_sync(self, url: str) -> dict[str, Any] | None:
+    def _crawl_with_dynamic_sync(
+        self, url: str, block_state: dict[str, Any] | None = None
+    ) -> dict[str, Any] | None:
         """Synchronous DynamicFetcher crawl executed in a worker thread."""
         fetch_start = time.perf_counter()
         page = DynamicFetcher.fetch(
@@ -376,10 +419,14 @@ class WebCrawlerConnector:
             allow_raw_fallback=False,
             fetch_ms=fetch_ms,
             status=getattr(page, "status", None),
+            block_state=block_state,
         )
 
     async def _crawl_with_stealthy(
-        self, url: str, captcha_state: dict[str, Any] | None = None
+        self,
+        url: str,
+        captcha_state: dict[str, Any] | None = None,
+        block_state: dict[str, Any] | None = None,
     ) -> dict[str, Any] | None:
         """
         Crawl URL using Scrapling's StealthyFetcher (patchright-Chromium) + Trafilatura.
@@ -390,13 +437,18 @@ class WebCrawlerConnector:
 
         ``captcha_state`` (03d) is mutated in place by the captcha page_action
         (attempts/solved) so ``crawl_url`` can surface it on the outcome.
+        ``block_state`` (03e) is populated by ``_build_result`` with the block
+        classification of the fetched page.
         """
         return await asyncio.to_thread(
-            self._crawl_with_stealthy_sync, url, captcha_state
+            self._crawl_with_stealthy_sync, url, captcha_state, block_state
         )
 
     def _crawl_with_stealthy_sync(
-        self, url: str, captcha_state: dict[str, Any] | None = None
+        self,
+        url: str,
+        captcha_state: dict[str, Any] | None = None,
+        block_state: dict[str, Any] | None = None,
     ) -> dict[str, Any] | None:
         """Synchronous StealthyFetcher crawl executed in a worker thread."""
         fetch_start = time.perf_counter()
@@ -425,6 +477,11 @@ class WebCrawlerConnector:
             "solve_cloudflare": True,
             "proxy": proxy,
         }
+        # 03e Slice A: merge config-driven stealth levers (block_webrtc,
+        # hide_canvas, google_search, dns_over_https, geoip locale/timezone).
+        # Keys never collide with the core kwargs above; defaults preserve
+        # today's behavior and add no crawl-speed regression.
+        fetch_kwargs.update(build_stealthy_kwargs(get_stealth_config()))
         if page_action is not None:
             fetch_kwargs["page_action"] = page_action
         page = StealthyFetcher.fetch(url, **fetch_kwargs)
@@ -436,6 +493,7 @@ class WebCrawlerConnector:
             allow_raw_fallback=True,
             fetch_ms=fetch_ms,
             status=getattr(page, "status", None),
+            block_state=block_state,
         )
 
     def _build_result(
@@ -447,6 +505,7 @@ class WebCrawlerConnector:
         allow_raw_fallback: bool,
         fetch_ms: float | None = None,
         status: int | None = None,
+        block_state: dict[str, Any] | None = None,
     ) -> dict[str, Any] | None:
         """
         Extract markdown + metadata from raw HTML using Trafilatura.
@@ -465,6 +524,12 @@ class WebCrawlerConnector:
         Returns:
             Result dict (content/metadata/crawler_type) or ``None``.
         """
+        # 03e: classify the fetched page (additive telemetry/routing only — never
+        # gates SUCCESS). Done before the early returns so EMPTY/no-extraction
+        # pages still get labeled. Last tier to fetch wins in block_state.
+        if block_state is not None:
+            block_state["block_type"] = classify_block(status, raw_html)
+
         html_len = len(raw_html) if raw_html else 0
 
         if not raw_html or len(raw_html.strip()) == 0:

surfsense_backend/app/proprietary/web_crawler/stealth.py

@@ -0,0 +1,216 @@
+# SurfSense proprietary crawler engine.
+#
+# This module is part of the ``app.proprietary`` package and is licensed
+# SEPARATELY from the Apache-2.0 project root. See ``app/proprietary/LICENSE``.
+# Do not relicense or redistribute this file under Apache-2.0.
+"""Stealth-hardening config + the centralized per-tier kwargs builder (Phase 3e).
+
+This is **bypass-specific tuning** (hence proprietary): it decides the
+anti-detection lever set the StealthyFetcher tier runs with (WebRTC/canvas/DNS
+handling, organic-referer, and proxy-geo-coherent ``locale``/``timezone_id``).
+It is the **single source of truth** that turns config -> ``StealthyFetcher``
+keyword arguments, imported by both the crawler (``connector.py``) and the 03f
+manual harness, so the scorecard grades the exact browser we ship (no
+test-vs-prod drift).
+
+The generic, vendor-agnostic block *classifier* (passive telemetry, public
+markers) stays Apache-2.0 in ``app/utils/crawl/``. The further bypass logic
+(WebGL spoof JS, humanize choreography) is deferred to later 03e slices and will
+also live here.
+
+Defaults preserve today's behavior and add **no crawl-speed regression**:
+``dns_over_https`` (the only lever with a latency cost) is off unless explicitly
+enabled, and geoip resolution is a pure in-process dict lookup (no exit-IP call).
+"""
+
+from dataclasses import dataclass
+from typing import Any
+
+from app.config import config
+
+# Coarse region -> (locale, IANA timezone) map. Country granularity only: per the
+# 03e "Geoip accuracy" risk, wrong-but-coherent beats a default mismatch and
+# per-city precision isn't worth it. Keyed by ISO-3166 alpha-2; common full names
+# are aliased below. Unknown/empty => (None, None) => leave Scrapling's default.
+_REGION_TO_LOCALE_TZ: dict[str, tuple[str, str]] = {
+    "us": ("en-US", "America/New_York"),
+    "ca": ("en-CA", "America/Toronto"),
+    "gb": ("en-GB", "Europe/London"),
+    "ie": ("en-IE", "Europe/Dublin"),
+    "au": ("en-AU", "Australia/Sydney"),
+    "nz": ("en-NZ", "Pacific/Auckland"),
+    "de": ("de-DE", "Europe/Berlin"),
+    "fr": ("fr-FR", "Europe/Paris"),
+    "es": ("es-ES", "Europe/Madrid"),
+    "it": ("it-IT", "Europe/Rome"),
+    "nl": ("nl-NL", "Europe/Amsterdam"),
+    "be": ("nl-BE", "Europe/Brussels"),
+    "ch": ("de-CH", "Europe/Zurich"),
+    "at": ("de-AT", "Europe/Vienna"),
+    "se": ("sv-SE", "Europe/Stockholm"),
+    "no": ("nb-NO", "Europe/Oslo"),
+    "dk": ("da-DK", "Europe/Copenhagen"),
+    "fi": ("fi-FI", "Europe/Helsinki"),
+    "pl": ("pl-PL", "Europe/Warsaw"),
+    "pt": ("pt-PT", "Europe/Lisbon"),
+    "ru": ("ru-RU", "Europe/Moscow"),
+    "ua": ("uk-UA", "Europe/Kyiv"),
+    "tr": ("tr-TR", "Europe/Istanbul"),
+    "in": ("en-IN", "Asia/Kolkata"),
+    "jp": ("ja-JP", "Asia/Tokyo"),
+    "kr": ("ko-KR", "Asia/Seoul"),
+    "cn": ("zh-CN", "Asia/Shanghai"),
+    "hk": ("zh-HK", "Asia/Hong_Kong"),
+    "tw": ("zh-TW", "Asia/Taipei"),
+    "sg": ("en-SG", "Asia/Singapore"),
+    "id": ("id-ID", "Asia/Jakarta"),
+    "ph": ("en-PH", "Asia/Manila"),
+    "th": ("th-TH", "Asia/Bangkok"),
+    "vn": ("vi-VN", "Asia/Ho_Chi_Minh"),
+    "ae": ("ar-AE", "Asia/Dubai"),
+    "il": ("he-IL", "Asia/Jerusalem"),
+    "za": ("en-ZA", "Africa/Johannesburg"),
+    "br": ("pt-BR", "America/Sao_Paulo"),
+    "mx": ("es-MX", "America/Mexico_City"),
+    "ar": ("es-AR", "America/Argentina/Buenos_Aires"),
+    "cl": ("es-CL", "America/Santiago"),
+}
+
+# Full-name / synonym aliases -> alpha-2 key above.
+_REGION_ALIASES: dict[str, str] = {
+    "usa": "us",
+    "united states": "us",
+    "united states of america": "us",
+    "america": "us",
+    "uk": "gb",
+    "united kingdom": "gb",
+    "great britain": "gb",
+    "england": "gb",
+    "canada": "ca",
+    "australia": "au",
+    "germany": "de",
+    "deutschland": "de",
+    "france": "fr",
+    "spain": "es",
+    "italy": "it",
+    "netherlands": "nl",
+    "holland": "nl",
+    "sweden": "se",
+    "norway": "no",
+    "denmark": "dk",
+    "finland": "fi",
+    "poland": "pl",
+    "portugal": "pt",
+    "russia": "ru",
+    "ukraine": "ua",
+    "turkey": "tr",
+    "india": "in",
+    "japan": "jp",
+    "korea": "kr",
+    "south korea": "kr",
+    "china": "cn",
+    "hong kong": "hk",
+    "taiwan": "tw",
+    "singapore": "sg",
+    "indonesia": "id",
+    "philippines": "ph",
+    "thailand": "th",
+    "vietnam": "vn",
+    "israel": "il",
+    "south africa": "za",
+    "brazil": "br",
+    "brasil": "br",
+    "mexico": "mx",
+    "argentina": "ar",
+    "chile": "cl",
+}
+
+
+def location_to_locale_timezone(
+    location: str | None,
+) -> tuple[str | None, str | None]:
+    """Map a free-form proxy region string -> (locale, timezone_id).
+
+    ``location`` is the vendor-specific ``RESIDENTIAL_PROXY_LOCATION`` value
+    (``03b``). Best-effort: matches an ISO-3166 alpha-2 code (e.g. ``"us"``) or a
+    common country name (e.g. ``"Germany"``); only the leading token is honored,
+    so vendor strings like ``"us:nyc"`` or ``"de region"`` still resolve. Returns
+    ``(None, None)`` for empty/unknown input (caller then leaves the browser
+    default).
+    """
+    if not location:
+        return (None, None)
+
+    normalized = location.strip().lower()
+    if not normalized:
+        return (None, None)
+
+    # Direct alpha-2 hit.
+    if normalized in _REGION_TO_LOCALE_TZ:
+        return _REGION_TO_LOCALE_TZ[normalized]
+
+    # Full-name / synonym hit.
+    if normalized in _REGION_ALIASES:
+        return _REGION_TO_LOCALE_TZ[_REGION_ALIASES[normalized]]
+
+    # Leading token (handles "us:nyc", "de-rotating", "united states (east)").
+    head = normalized.replace("-", " ").replace("_", " ").replace(":", " ").split()
+    if head:
+        token = head[0]
+        if token in _REGION_TO_LOCALE_TZ:
+            return _REGION_TO_LOCALE_TZ[token]
+        if token in _REGION_ALIASES:
+            return _REGION_TO_LOCALE_TZ[_REGION_ALIASES[token]]
+
+    return (None, None)
+
+
+@dataclass(frozen=True)
+class StealthConfig:
+    """Immutable snapshot of the Phase 3e Slice-A stealth levers for one crawl."""
+
+    geoip_match_enabled: bool
+    proxy_location: str
+    block_webrtc: bool
+    hide_canvas: bool
+    google_search: bool
+    dns_over_https: bool
+
+
+def get_stealth_config() -> StealthConfig:
+    """Build a :class:`StealthConfig` from the current process config/env."""
+    return StealthConfig(
+        geoip_match_enabled=config.CRAWL_GEOIP_MATCH_ENABLED,
+        proxy_location=config.RESIDENTIAL_PROXY_LOCATION or "",
+        block_webrtc=config.CRAWL_BLOCK_WEBRTC,
+        hide_canvas=config.CRAWL_HIDE_CANVAS,
+        google_search=config.CRAWL_GOOGLE_SEARCH_REFERER,
+        dns_over_https=config.CRAWL_DNS_OVER_HTTPS,
+    )
+
+
+def build_stealthy_kwargs(cfg: StealthConfig) -> dict[str, Any]:
+    """Return the config-derived ``StealthyFetcher.fetch`` keyword arguments.
+
+    Only the Slice-A stealth levers are returned; the caller merges these into
+    the tier's own kwargs (``headless``/``network_idle``/``block_ads``/
+    ``solve_cloudflare``/``proxy``/captcha ``page_action``), which never collide
+    with the keys here. ``locale``/``timezone_id`` are added only when geoip
+    matching is enabled *and* the proxy region resolves — otherwise the browser
+    keeps its system default.
+    """
+    kwargs: dict[str, Any] = {
+        "block_webrtc": cfg.block_webrtc,
+        "hide_canvas": cfg.hide_canvas,
+        "google_search": cfg.google_search,
+        "dns_over_https": cfg.dns_over_https,
+    }
+
+    if cfg.geoip_match_enabled:
+        locale, timezone_id = location_to_locale_timezone(cfg.proxy_location)
+        if locale:
+            kwargs["locale"] = locale
+        if timezone_id:
+            kwargs["timezone_id"] = timezone_id
+
+    return kwargs

surfsense_backend/app/proprietary/web_crawler/testbench/README.md

@@ -0,0 +1,83 @@
+# Crawler testbench — manual undetectability & extraction scorecard (Phase 3f)
+
+A **manual, repeatable scorecard** for the Universal WebURL Crawler. It answers
+one question with evidence: *how undetectable (and how correct) is the crawler
+right now?* — so we know when the free-stack ceiling (`03e`) is reached and the
+deferred paid-unblocker tier is worth flipping.
+
+This is **dev/operator tooling**, not a product code path and **not** part of the
+pytest suite (it hits the live internet and needs proxy creds). See
+`plans/backend/03f-undetectability-testing.md` for the full design.
+
+## What it measures (two axes, two suites)
+
+- **Suite S — stealth / anti-bot:** drives the real `StealthyFetcher` tier
+  (built from the **same** `app/proprietary/web_crawler/stealth.py` builder the
+  crawler ships, so no test-vs-prod drift) against the standard bot-detection
+  sites — sannysoft, deviceandbrowserinfo, reCAPTCHA v3, CreepJS, BrowserScan,
+  incolumitas, fingerprint-scan, FingerprintJS Pro, a **Cloudflare-challenge
+  canary** (the only row that exercises `solve_cloudflare`), and iphey — plus the
+  HTTP-tier TLS fingerprint (`tls.peet.ws`), exit-IP echo, and manual
+  per-property links (browserleaks). Every detection site is **auto-graded** from
+  its real DOM verdict (no screenshot reading required).
+- **Suite E — extraction correctness:** drives the real `crawl_url` ladder
+  against ToS-safe scraping sandboxes (`toscrape`, `scrapethissite`) and asserts
+  known content appears in the extracted markdown.
+
+## Run it
+
+From the backend directory (`surfsense_backend/`):
+
+```bash
+uv run python -m app.proprietary.web_crawler.testbench --suite all
+# or
+.\.venv\Scripts\python.exe -m app.proprietary.web_crawler.testbench --suite all
+```
+
+Flags:
+
+| Flag | Meaning |
+|---|---|
+| `--suite S\|E\|all` | Which suite(s) to run (default `all`). |
+| `--proxy URL` | Override the proxy for Suite S (default: the app proxy provider from `.env`). |
+| `--headed` | Run the browser tier headful (`headless=False`). |
+| `--no-screenshots` | Skip per-site screenshots. |
+
+Captcha solving is **forced OFF** every run — Suite S measures the *unaided*
+stealth score.
+
+## Reading the output
+
+Each row prints `PASS / FAIL / ERR / INFO` with the tier and (where one exists) a
+numeric. The harness is **tolerant**: a parse miss is `ERROR`, never a crash.
+Detection sites are auto-graded from their DOM verdict; `INFO` is reserved for
+purely informational rows (TLS fingerprint, exit IP) and the manual
+per-property links (browserleaks canvas/webgl/fonts/webrtc). Screenshots are
+still captured for every browser row as a backstop.
+
+Outputs land in `results/`, which is **entirely gitignored** (run-local):
+
+- `scorecard-<ts>.json` / `.md` — timestamped scorecard + readable report.
+- `latest.json` — convenience pointer.
+- `screenshots/` — per-site full-page captures.
+
+Every run prints **drift vs the last baseline** (the previous on-disk scorecard)
+so you can see the trend (our stealth improving, or a WAF tightening).
+
+## The aspirational bars (from CloakBrowser, recorded as targets)
+
+`sannysoft` 0 failed cells · `deviceandbrowserinfo` isBot=false ·
+reCAPTCHA v3 `>= 0.7` · CreepJS headless `<= 30%`, no spoof tells · FingerprintJS
+demo not blocked · Cloudflare challenge bypassed · iphey `Trustworthy`. We
+**record our actual numbers as the baseline** — these are targets, not build
+gates (the run never blocks anything).
+
+## Caveats
+
+- **Proxy required for realism.** Without residential egress the hard rows fail
+  by design (datacenter IP); a red scorecard from no-proxy is expected.
+- **Sites change.** Detection sites move their DOM; if an auto-parser starts
+  returning `INFO`/`ERROR`, read the screenshot and (optionally) tighten the
+  parser — each site is one entry in `suite_stealth.py`.
+- **Not a guarantee.** Passing sannysoft/CreepJS ≠ beating DataDome/Kasada; the
+  value is *trend + ceiling visibility*, not a green checkmark.

surfsense_backend/app/proprietary/web_crawler/testbench/__init__.py

@@ -0,0 +1,25 @@
+# SurfSense proprietary crawler engine.
+#
+# This module is part of the ``app.proprietary`` package and is licensed
+# SEPARATELY from the Apache-2.0 project root. See ``app/proprietary/LICENSE``.
+# Do not relicense or redistribute this file under Apache-2.0.
+"""Phase 3f — manual undetectability & extraction test harness (dev tooling).
+
+A repeatable, **manual** scorecard that grades the Universal WebURL Crawler on two
+axes:
+
+- **Suite S (stealth / anti-bot):** drives the real Scrapling tiers against the
+  industry-standard bot-detection + fingerprint sites and parses each verdict.
+- **Suite E (extraction correctness):** drives the real ``crawl_url`` ladder
+  against ToS-safe scraping sandboxes and asserts known values.
+
+It is **not** collected by pytest (it hits the live internet and needs proxy
+creds). Run it from the backend directory:
+
+    uv run python -m app.proprietary.web_crawler.testbench --suite all
+    # or: .\\.venv\\Scripts\\python.exe -m app.proprietary.web_crawler.testbench --suite all
+
+See ``README.md`` for the runbook and ``plans/backend/03f-undetectability-testing.md``
+for the design. This package is dev/operator tooling, untouched by the product
+rename, and only *measures* what 03a–03e built.
+"""

surfsense_backend/app/proprietary/web_crawler/testbench/__main__.py

@@ -0,0 +1,141 @@
+# SurfSense proprietary crawler engine.
+#
+# This module is part of the ``app.proprietary`` package and is licensed
+# SEPARATELY from the Apache-2.0 project root. See ``app/proprietary/LICENSE``.
+# Do not relicense or redistribute this file under Apache-2.0.
+"""CLI for the 03f manual scorecard. Run from the backend directory:
+
+    uv run python -m app.proprietary.web_crawler.testbench --suite all
+    uv run python -m app.proprietary.web_crawler.testbench --suite S --headed
+    uv run python -m app.proprietary.web_crawler.testbench --suite E --no-screenshots
+
+Mirrors CloakBrowser's ``bin/cloaktest`` ergonomics. Writes a timestamped
+scorecard (JSON + markdown) under ``results/`` and diffs against the last
+baseline. Captcha solving is forced OFF so the scorecard measures the *unaided*
+stealth ceiling (03f §Harness).
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import os
+import sys
+from pathlib import Path
+
+from dotenv import load_dotenv
+
+# --- bootstrap: load .env + put backend root on sys.path BEFORE importing app.* ---
+# __file__ = app/proprietary/web_crawler/testbench/__main__.py -> backend root is 4 up.
+_BACKEND_ROOT = Path(__file__).resolve().parents[4]
+if str(_BACKEND_ROOT) not in sys.path:
+    sys.path.insert(0, str(_BACKEND_ROOT))
+for _candidate in (_BACKEND_ROOT / ".env", _BACKEND_ROOT.parent / ".env"):
+    if _candidate.exists():
+        load_dotenv(_candidate)
+        break
+
+# Force captcha solving OFF: Suite S measures the unaided score and we never want
+# the 03d injector firing paid solves against the reCAPTCHA-demo row. Must be set
+# before app.config is imported (config snapshots env at import).
+os.environ["CAPTCHA_SOLVING_ENABLED"] = "false"
+
+
+def _parse_args(argv: list[str]) -> argparse.Namespace:
+    p = argparse.ArgumentParser(
+        prog="python -m app.proprietary.web_crawler.testbench",
+        description="Manual undetectability & extraction scorecard (Phase 3f).",
+    )
+    p.add_argument(
+        "--suite",
+        choices=["S", "E", "all"],
+        default="all",
+        help="S=stealth/anti-bot, E=extraction correctness, all=both.",
+    )
+    p.add_argument(
+        "--proxy",
+        default=None,
+        help="Override proxy URL for Suite S (default: the app proxy provider).",
+    )
+    p.add_argument(
+        "--headed",
+        action="store_true",
+        help="Run the browser tier headful (headless=False).",
+    )
+    p.add_argument(
+        "--no-screenshots",
+        action="store_true",
+        help="Skip per-site screenshots (faster, no results/screenshots writes).",
+    )
+    return p.parse_args(argv)
+
+
+async def _amain(args: argparse.Namespace) -> int:
+    # Imports happen here (after bootstrap + env override) so app.config sees the
+    # forced CAPTCHA_SOLVING_ENABLED=false and the resolved .env.
+    from app.utils.proxy import get_proxy_url
+
+    from .core import (
+        RunMeta,
+        diff_against_baseline,
+        load_last_baseline,
+        render_console,
+        scrapling_version,
+        write_scorecard,
+    )
+
+    proxy = args.proxy or get_proxy_url()
+    screenshots = not args.no_screenshots
+    results = []
+
+    print(
+        f"== crawler scorecard == suite={args.suite} "
+        f"headed={args.headed} proxy={'set' if proxy else 'NONE'} "
+        f"screenshots={screenshots}"
+    )
+    if not proxy:
+        print(
+            "  WARNING: no proxy configured — the hard anti-bot rows are expected "
+            "to fail from a datacenter IP (see README)."
+        )
+
+    baseline = load_last_baseline()
+
+    if args.suite in ("S", "all"):
+        from .suite_stealth import run_suite_s
+
+        results += await run_suite_s(
+            proxy=proxy, headed=args.headed, screenshots=screenshots
+        )
+
+    if args.suite in ("E", "all"):
+        from .suite_extraction import run_suite_e
+
+        results += await run_suite_e()
+
+    render_console(results)
+
+    meta = RunMeta.now(
+        suites=args.suite,
+        proxy=proxy,
+        headed=args.headed,
+        scrapling_version=scrapling_version(),
+    )
+    json_path, md_path = write_scorecard(results, meta)
+
+    print("\n--- drift vs last baseline ---")
+    for line in diff_against_baseline(results, baseline):
+        print(f"  {line}")
+
+    print(f"\nscorecard JSON: {json_path}")
+    print(f"scorecard MD:   {md_path}")
+    return 0
+
+
+def main() -> None:
+    args = _parse_args(sys.argv[1:])
+    raise SystemExit(asyncio.run(_amain(args)))
+
+
+if __name__ == "__main__":
+    main()

surfsense_backend/app/proprietary/web_crawler/testbench/core.py

@@ -0,0 +1,325 @@
+# SurfSense proprietary crawler engine.
+#
+# This module is part of the ``app.proprietary`` package and is licensed
+# SEPARATELY from the Apache-2.0 project root. See ``app/proprietary/LICENSE``.
+# Do not relicense or redistribute this file under Apache-2.0.
+"""Shared primitives for the 03f scorecard: result model, the closure-cell
+``page_action`` verdict-extractor (the 03d-shared mechanism), and the scorecard
+snapshot writer / baseline-differ / console renderer.
+
+Stdlib-only on purpose (the plan's "no new prod dependency" bar). Everything here
+is tolerant: a parse miss yields an ``ERROR`` row, never a crash — detection sites
+change their DOM and the harness is manual.
+"""
+
+from __future__ import annotations
+
+import json
+from collections.abc import Callable
+from dataclasses import asdict, dataclass
+from datetime import datetime, timezone
+from enum import Enum
+from pathlib import Path
+from typing import Any
+from urllib.parse import urlsplit
+
+# Results live next to the harness; screenshots are gitignored, scorecard JSON is
+# committed so runs diff against the last baseline (see README + plan §Scorecard).
+_PKG_DIR = Path(__file__).resolve().parent
+RESULTS_DIR = _PKG_DIR / "results"
+SCREENSHOTS_DIR = RESULTS_DIR / "screenshots"
+
+
+class CheckStatus(str, Enum):
+    """Outcome of a single scorecard row."""
+
+    PASS = "PASS"  # met the aspirational bar
+    FAIL = "FAIL"  # ran, did not meet the bar
+    ERROR = "ERROR"  # could not run / parse (never fatal to the run)
+    INFO = "INFO"  # recorded, no pass/fail bar (e.g. TLS JA3, manual links)
+    SKIP = "SKIP"  # intentionally not run this invocation
+
+
+@dataclass
+class CheckResult:
+    """One row of the scorecard."""
+
+    suite: str  # "S" (stealth) | "E" (extraction)
+    name: str  # site / check key
+    tier: str  # scrapling-static | scrapling-stealthy | crawl_url | n/a
+    status: CheckStatus
+    bar: str  # human-readable aspirational threshold
+    detail: str = ""  # short human summary of what was observed
+    numeric: float | None = None  # comparable metric where one exists
+    screenshot: str | None = None  # path, when captured
+
+    def to_row(self) -> dict[str, Any]:
+        d = asdict(self)
+        d["status"] = self.status.value
+        return d
+
+
+@dataclass
+class RunMeta:
+    """Provenance for a scorecard snapshot so baselines are comparable."""
+
+    timestamp: str
+    suites: str
+    proxy: str  # masked
+    headed: bool
+    scrapling_version: str
+    captcha_disabled: bool = True
+    notes: str = ""
+
+    @staticmethod
+    def now(
+        *, suites: str, proxy: str | None, headed: bool, scrapling_version: str
+    ) -> RunMeta:
+        return RunMeta(
+            timestamp=datetime.now(timezone.utc).isoformat(timespec="seconds"),
+            suites=suites,
+            proxy=mask_proxy(proxy),
+            headed=headed,
+            scrapling_version=scrapling_version,
+        )
+
+
+# --- closure-cell page_action (the mechanism shared with 03d) ----------------
+
+
+@dataclass
+class EvalCell:
+    """A mutable cell a ``page_action`` writes its ``page.evaluate`` result into.
+
+    Scrapling discards a ``page_action``'s return value, so the only way to get a
+    JS-object verdict (CreepJS ``window.Fingerprint``, a Castle.js score node) out
+    of the browser is to stash it in a closure variable. This is the exact same
+    plumbing 03d's captcha-token injector uses — factored once here.
+    """
+
+    value: Any = None
+    error: str | None = None
+    captured: bool = False
+
+
+def make_page_action(
+    *,
+    evaluate_js: str | None = None,
+    screenshot_path: str | None = None,
+    pre_wait_ms: int = 0,
+) -> tuple[Callable[[Any], Any], EvalCell]:
+    """Build a sync ``page_action`` + the :class:`EvalCell` it writes into.
+
+    The action (in priority order) optionally sleeps ``pre_wait_ms`` (lets async
+    scores settle), optionally evaluates ``evaluate_js`` into the cell, and
+    optionally writes a full-page screenshot. Each step is independently guarded
+    so one failure (e.g. a site without the JS object) never aborts the fetch.
+    Returns the page unchanged (Scrapling re-reads its DOM afterwards).
+    """
+    cell = EvalCell()
+
+    def _action(page: Any) -> Any:
+        if pre_wait_ms > 0:
+            try:
+                page.wait_for_timeout(pre_wait_ms)
+            except Exception:
+                pass
+        if evaluate_js is not None:
+            try:
+                cell.value = page.evaluate(evaluate_js)
+                cell.captured = True
+            except Exception as exc:  # noqa: BLE001 - tolerant by design
+                cell.error = f"{type(exc).__name__}: {exc}"
+        if screenshot_path is not None:
+            try:
+                # The screenshot dir must exist *now* (the page action runs mid-suite,
+                # before the end-of-run ``ensure_dirs``); Scrapling/patchright will not
+                # create it for us, so a missing dir silently drops every screenshot.
+                Path(screenshot_path).parent.mkdir(parents=True, exist_ok=True)
+                page.screenshot(path=screenshot_path, full_page=True)
+            except Exception as exc:  # noqa: BLE001 - tolerant, but surface the cause
+                if cell.error is None:
+                    cell.error = f"screenshot: {type(exc).__name__}: {exc}"
+        return page
+
+    return _action, cell
+
+
+# --- helpers -----------------------------------------------------------------
+
+
+def mask_proxy(url: str | None) -> str:
+    """Mask credentials in a proxy URL for logs/snapshots."""
+    if not url:
+        return "<none>"
+    try:
+        p = urlsplit(url)
+        host = p.hostname or "?"
+        port = f":{p.port}" if p.port else ""
+        return f"{p.scheme}://***@{host}{port}"
+    except Exception:
+        return "<set>"
+
+
+def ensure_dirs() -> None:
+    RESULTS_DIR.mkdir(parents=True, exist_ok=True)
+    SCREENSHOTS_DIR.mkdir(parents=True, exist_ok=True)
+
+
+def scrapling_version() -> str:
+    try:
+        import scrapling  # noqa: PLC0415
+
+        return getattr(scrapling, "__version__", "unknown")
+    except Exception:
+        return "unavailable"
+
+
+# --- scorecard I/O -----------------------------------------------------------
+
+
+def write_scorecard(
+    results: list[CheckResult], meta: RunMeta
+) -> tuple[Path, Path]:
+    """Write the JSON snapshot (committed, baseline) + a readable markdown report.
+
+    The JSON filename is timestamped so prior runs are kept as the baseline trail;
+    ``latest.json`` is overwritten as the convenience pointer used by the differ.
+    """
+    ensure_dirs()
+    stamp = meta.timestamp.replace(":", "").replace("-", "")
+    payload = {
+        "meta": asdict(meta),
+        "summary": summarize(results),
+        "results": [r.to_row() for r in results],
+    }
+    json_path = RESULTS_DIR / f"scorecard-{stamp}.json"
+    json_path.write_text(json.dumps(payload, indent=2), encoding="utf-8")
+    (RESULTS_DIR / "latest.json").write_text(
+        json.dumps(payload, indent=2), encoding="utf-8"
+    )
+    md_path = RESULTS_DIR / f"scorecard-{stamp}.md"
+    md_path.write_text(_render_markdown(results, meta), encoding="utf-8")
+    return json_path, md_path
+
+
+def load_last_baseline() -> dict[str, Any] | None:
+    """Load the most recent prior scorecard JSON (excluding ``latest.json``)."""
+    if not RESULTS_DIR.exists():
+        return None
+    snaps = sorted(RESULTS_DIR.glob("scorecard-*.json"))
+    if not snaps:
+        return None
+    try:
+        return json.loads(snaps[-1].read_text(encoding="utf-8"))
+    except Exception:
+        return None
+
+
+def summarize(results: list[CheckResult]) -> dict[str, dict[str, int]]:
+    """Per-suite ``{passed, failed, error, info, total}`` counts."""
+    out: dict[str, dict[str, int]] = {}
+    for r in results:
+        s = out.setdefault(
+            r.suite,
+            {"passed": 0, "failed": 0, "error": 0, "info": 0, "skip": 0, "total": 0},
+        )
+        s["total"] += 1
+        key = {
+            CheckStatus.PASS: "passed",
+            CheckStatus.FAIL: "failed",
+            CheckStatus.ERROR: "error",
+            CheckStatus.INFO: "info",
+            CheckStatus.SKIP: "skip",
+        }[r.status]
+        s[key] += 1
+    return out
+
+
+def diff_against_baseline(
+    results: list[CheckResult], baseline: dict[str, Any] | None
+) -> list[str]:
+    """Human-readable drift lines comparing this run to the last baseline."""
+    if not baseline:
+        return ["(no prior baseline — this run becomes the baseline)"]
+    prior = {(r["suite"], r["name"]): r for r in baseline.get("results", [])}
+    lines: list[str] = []
+    for r in results:
+        was = prior.get((r.suite, r.name))
+        if was is None:
+            lines.append(f"+ NEW  [{r.suite}] {r.name}: {r.status.value}")
+            continue
+        if was["status"] != r.status.value:
+            lines.append(
+                f"~ {r.name}: status {was['status']} -> {r.status.value}"
+            )
+        old_n, new_n = was.get("numeric"), r.numeric
+        if old_n is not None and new_n is not None and abs(old_n - new_n) > 1e-9:
+            lines.append(f"~ {r.name}: numeric {old_n} -> {new_n}")
+    seen = {(r.suite, r.name) for r in results}
+    for key, was in prior.items():
+        if key not in seen:
+            lines.append(f"- GONE [{key[0]}] {key[1]} (was {was['status']})")
+    return lines or ["(no changes vs last baseline)"]
+
+
+# --- rendering ---------------------------------------------------------------
+
+_ICON = {
+    CheckStatus.PASS: "PASS",
+    CheckStatus.FAIL: "FAIL",
+    CheckStatus.ERROR: "ERR ",
+    CheckStatus.INFO: "INFO",
+    CheckStatus.SKIP: "SKIP",
+}
+
+
+def render_console(results: list[CheckResult]) -> None:
+    """Print the grouped scorecard + per-suite totals to stdout."""
+    by_suite: dict[str, list[CheckResult]] = {}
+    for r in results:
+        by_suite.setdefault(r.suite, []).append(r)
+
+    for suite in sorted(by_suite):
+        label = "Suite S - stealth/anti-bot" if suite == "S" else (
+            "Suite E - extraction" if suite == "E" else f"Suite {suite}"
+        )
+        print(f"\n=== {label} ===")
+        for r in by_suite[suite]:
+            num = f" [{r.numeric:g}]" if r.numeric is not None else ""
+            print(f"  {_ICON[r.status]}  {r.name:<34} {r.tier:<18}{num}")
+            if r.detail:
+                print(f"         {r.detail}")
+
+    print("\n--- summary ---")
+    for suite, s in summarize(results).items():
+        print(
+            f"  Suite {suite}: {s['passed']}/{s['total']} passed "
+            f"(fail={s['failed']} err={s['error']} info={s['info']})"
+        )
+
+
+def _render_markdown(results: list[CheckResult], meta: RunMeta) -> str:
+    lines = [
+        f"# Crawler scorecard — {meta.timestamp}",
+        "",
+        f"- proxy: `{meta.proxy}`  headed: `{meta.headed}`  "
+        f"captcha-disabled: `{meta.captcha_disabled}`",
+        f"- scrapling: `{meta.scrapling_version}`  suites: `{meta.suites}`",
+        "",
+        "| Suite | Check | Tier | Status | Numeric | Detail |",
+        "|---|---|---|---|---|---|",
+    ]
+    for r in results:
+        num = "" if r.numeric is None else f"{r.numeric:g}"
+        detail = r.detail.replace("|", "\\|")
+        lines.append(
+            f"| {r.suite} | {r.name} | {r.tier} | {r.status.value} | {num} | {detail} |"
+        )
+    lines.append("")
+    for suite, s in summarize(results).items():
+        lines.append(
+            f"- **Suite {suite}**: {s['passed']}/{s['total']} passed "
+            f"(fail={s['failed']}, err={s['error']}, info={s['info']})"
+        )
+    return "\n".join(lines) + "\n"

surfsense_backend/app/proprietary/web_crawler/testbench/results/.gitignore

@@ -0,0 +1,5 @@
+# Everything under results/ is run-local output (timestamped scorecards,
+# latest.json pointer, screenshots, ad-hoc dumps). None of it is committed —
+# the dir is kept tracked only via this file so the harness has a write target.
+*
+!.gitignore

surfsense_backend/app/proprietary/web_crawler/testbench/suite_extraction.py

@@ -0,0 +1,123 @@
+# SurfSense proprietary crawler engine.
+#
+# This module is part of the ``app.proprietary`` package and is licensed
+# SEPARATELY from the Apache-2.0 project root. See ``app/proprietary/LICENSE``.
+# Do not relicense or redistribute this file under Apache-2.0.
+"""Suite E — extraction correctness (separate axis from stealth).
+
+Unlike Suite S, this drives the **real ``crawl_url`` ladder end-to-end** — the
+auto-tier + Trafilatura markdown path is exactly the production behavior we want
+to assert. Targets are purpose-built, ToS-safe scraping sandboxes with known
+content, so extraction regressions are caught deterministically (a missing
+expected string => FAIL). The ``/js`` cases exercise the DynamicFetcher (browser)
+tier; the static catalogs exercise the HTTP tier.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+from app.proprietary.web_crawler import CrawlOutcomeStatus, WebCrawlerConnector
+
+from .core import CheckResult, CheckStatus
+
+
+@dataclass
+class ExtractionCase:
+    """A sandbox URL + strings that must appear in the extracted markdown."""
+
+    name: str
+    url: str
+    must_contain: list[str] = field(default_factory=list)
+
+
+_CASES: list[ExtractionCase] = [
+    ExtractionCase(
+        name="books_static",
+        url="https://books.toscrape.com/",
+        must_contain=["A Light in the Attic", "Tipping the Velvet"],
+    ),
+    ExtractionCase(
+        name="quotes_static",
+        url="https://quotes.toscrape.com/",
+        must_contain=["Albert Einstein", "world as we have created it"],
+    ),
+    ExtractionCase(
+        name="quotes_js_rendered",
+        url="https://quotes.toscrape.com/js/",
+        must_contain=["Albert Einstein", "world as we have created it"],
+    ),
+    ExtractionCase(
+        name="scrapethissite_simple",
+        url="https://www.scrapethissite.com/pages/simple/",
+        must_contain=["Andorra", "Afghanistan"],
+    ),
+]
+
+
+def _content_of(outcome) -> str:
+    result = getattr(outcome, "result", None) or {}
+    if isinstance(result, dict):
+        return str(result.get("content") or "")
+    return ""
+
+
+async def _run_case(
+    connector: WebCrawlerConnector, case: ExtractionCase
+) -> CheckResult:
+    bar = f"SUCCESS + contains {len(case.must_contain)} marker(s)"
+    try:
+        outcome = await connector.crawl_url(case.url)
+    except Exception as exc:  # noqa: BLE001 - never crash the run
+        return CheckResult(
+            suite="E",
+            name=case.name,
+            tier="crawl_url",
+            status=CheckStatus.ERROR,
+            bar=bar,
+            detail=f"{type(exc).__name__}: {exc}",
+        )
+
+    tier = getattr(outcome, "tier", None) or "crawl_url"
+    if outcome.status is not CrawlOutcomeStatus.SUCCESS:
+        return CheckResult(
+            suite="E",
+            name=case.name,
+            tier=tier,
+            status=CheckStatus.FAIL,
+            bar=bar,
+            detail=f"status={outcome.status.value}: {outcome.error or ''}"[:160],
+        )
+
+    content = _content_of(outcome)
+    lowered = content.lower()
+    missing = [m for m in case.must_contain if m.lower() not in lowered]
+    if missing:
+        return CheckResult(
+            suite="E",
+            name=case.name,
+            tier=tier,
+            status=CheckStatus.FAIL,
+            bar=bar,
+            detail=f"missing {missing} (len={len(content)})",
+            numeric=float(len(content)),
+        )
+    return CheckResult(
+        suite="E",
+        name=case.name,
+        tier=tier,
+        status=CheckStatus.PASS,
+        bar=bar,
+        detail=f"all markers present (tier={tier}, len={len(content)})",
+        numeric=float(len(content)),
+    )
+
+
+async def run_suite_e() -> list[CheckResult]:
+    """Run the extraction-correctness cases against the live sandboxes."""
+    connector = WebCrawlerConnector()
+    results: list[CheckResult] = []
+    for case in _CASES:
+        print(f"  [E] {case.name} -> {case.url}")
+        results.append(await _run_case(connector, case))
+    return results

surfsense_backend/app/proprietary/web_crawler/testbench/suite_stealth.py

@@ -0,0 +1,509 @@
+# SurfSense proprietary crawler engine.
+#
+# This module is part of the ``app.proprietary`` package and is licensed
+# SEPARATELY from the Apache-2.0 project root. See ``app/proprietary/LICENSE``.
+# Do not relicense or redistribute this file under Apache-2.0.
+"""Suite S — stealth / anti-bot scorecard.
+
+Drives the **browser tier** (``StealthyFetcher``) against the standard
+bot-detection sites and the **HTTP tier** (``AsyncFetcher``) against the TLS /
+proxy-leak JSON endpoints. Critically, the browser tier is built from the **same
+centralized stealth builder the crawler ships** (``build_stealthy_kwargs`` /
+``get_stealth_config`` in ``app/proprietary/web_crawler/stealth.py``) so the scorecard grades
+the exact browser we run in production — no test-vs-prod drift (03f §Harness).
+
+Verdict extraction is best-effort and tolerant: machine-readable signals
+(reCAPTCHA score text, are-you-a-bot text, sannysoft failed-cell count, TLS/proxy
+JSON) get a real PASS/FAIL/numeric; DOM-heavy fingerprint pages are recorded as
+INFO with a screenshot + captured text for the operator to read. Tightening any
+parser later is a one-function change (the spec list is the extension point).
+"""
+
+from __future__ import annotations
+
+import asyncio
+import re
+from collections.abc import Callable
+from dataclasses import dataclass
+from typing import Any
+
+from scrapling.fetchers import AsyncFetcher, StealthyFetcher
+
+from app.proprietary.web_crawler.stealth import (
+    build_stealthy_kwargs,
+    get_stealth_config,
+)
+
+from .core import (
+    SCREENSHOTS_DIR,
+    CheckResult,
+    CheckStatus,
+    EvalCell,
+    make_page_action,
+)
+
+# Parser: (page, eval_cell) -> (status, numeric, detail).
+Parser = Callable[[Any, EvalCell], "tuple[CheckStatus, float | None, str]"]
+
+
+@dataclass
+class StealthSite:
+    """One browser-tier detection site + how to read its verdict."""
+
+    name: str
+    url: str
+    bar: str
+    parse: Parser
+    settle_ms: int = 0  # extra in-page wait for async-computed scores
+    evaluate_js: str | None = None  # for window.* / JS-object verdicts
+
+
+def _text(page: Any, limit: int = 4000) -> str:
+    """Best-effort visible text of a fetched page (short, for fallback details)."""
+    return _full_text(page, limit)
+
+
+def _full_text(page: Any, limit: int = 200_000) -> str:
+    """Untruncated visible text — detector verdicts live deep in long pages."""
+    try:
+        txt = page.get_all_text()
+        if txt:
+            return str(txt)[:limit]
+    except Exception:
+        pass
+    try:
+        return str(page.html_content or "")[:limit]
+    except Exception:
+        return ""
+
+
+# --- per-site parsers (all written against real DOM dumps, no screenshots) ---
+
+
+def _parse_sannysoft(page: Any, _cell: EvalCell):
+    """Count failed result cells (JS sets class='failed' on red rows)."""
+    html = ""
+    try:
+        html = page.html_content or ""
+    except Exception:
+        pass
+    fails = len(re.findall(r'class="[^"]*\bfailed\b[^"]*"', html))
+    if not html:
+        return (CheckStatus.ERROR, None, "no html returned")
+    status = CheckStatus.PASS if fails == 0 else CheckStatus.FAIL
+    return (status, float(fails), f"{fails} failed cell(s) (bar: 0)")
+
+
+# deviceandbrowserinfo renders a JSON verdict block: {"isBot": false, ...}.
+_ISBOT_RE = re.compile(r'"isbot"\s*:\s*(true|false)', re.IGNORECASE)
+
+
+def _parse_areyouabot(page: Any, _cell: EvalCell):
+    txt = _full_text(page)
+    m = _ISBOT_RE.search(txt)
+    if m:
+        is_bot = m.group(1).lower() == "true"
+        return (
+            CheckStatus.FAIL if is_bot else CheckStatus.PASS,
+            1.0 if is_bot else 0.0,
+            f"isBot={is_bot}",
+        )
+    low = txt.lower()
+    if "you are human" in low or "not a bot" in low:
+        return (CheckStatus.PASS, 0.0, "verdict: human")
+    if "you are a bot" in low:
+        return (CheckStatus.FAIL, 1.0, "verdict: BOT")
+    return (CheckStatus.INFO, None, f"unparsed: {low[:120]!r}")
+
+
+# The reCAPTCHA demo echoes the server verify response: {"success":true,"score":0.9}.
+_SCORE_JSON_RE = re.compile(r'"score"\s*:\s*([0-9]*\.?[0-9]+)')
+
+
+def _parse_recaptcha_v3(page: Any, _cell: EvalCell):
+    txt = _full_text(page)
+    m = _SCORE_JSON_RE.search(txt)
+    if not m:  # fall back to looser prose match
+        m = re.search(r"score[^0-9]{0,12}([01](?:\.\d+)?)", txt, re.IGNORECASE)
+    if not m:
+        return (CheckStatus.INFO, None, f"no score parsed: {txt[:120]!r}")
+    score = float(m.group(1))
+    status = CheckStatus.PASS if score >= 0.7 else CheckStatus.FAIL
+    return (status, score, f"reCAPTCHA v3 score={score} (bar: >=0.7)")
+
+
+# CreepJS prints category percentages ("33% headless") + boolean tells inline.
+_CREEPJS_TELLS = (
+    "webDriverIsOn",
+    "hasHeadlessUA",
+    "hasHeadlessWorkerUA",
+    "hasBadWebGL",
+    "hasSwiftShader",
+)
+
+
+def _parse_creepjs(page: Any, _cell: EvalCell):
+    """Grade CreepJS by its headless-similarity % + the boolean spoof tells."""
+    txt = _full_text(page)
+
+    def _pct(label: str) -> int | None:
+        m = re.search(r"(\d+)%\s*" + label, txt, re.IGNORECASE)
+        return int(m.group(1)) if m else None
+
+    headless = _pct(r"headless")
+    stealth = _pct(r"stealth")
+    tells = [
+        flag
+        for flag in _CREEPJS_TELLS
+        if re.search(re.escape(flag) + r"\s*:\s*true", txt, re.IGNORECASE)
+    ]
+    if headless is None and not tells:
+        return (CheckStatus.INFO, None, f"unparsed: {txt[:120]!r}")
+    bad = bool(tells) or (headless is not None and headless > 30)
+    detail = f"headless={headless}% stealth={stealth}% tells={tells or 'none'}"
+    return (
+        CheckStatus.FAIL if bad else CheckStatus.PASS,
+        float(headless) if headless is not None else None,
+        detail,
+    )
+
+
+# incolumitas emits JSON test blocks ("WEBDRIVER":"FAIL") + an IP classification.
+_FAIL_KEY_RE = re.compile(r'"([A-Za-z_]+)"\s*:\s*"FAIL"')
+_DC_RE = re.compile(r'"is_datacenter"\s*:\s*(true|false)', re.IGNORECASE)
+_BEHAV_RE = re.compile(r"Your Behavioral Score:\s*([0-9.]+)", re.IGNORECASE)
+
+
+def _parse_incolumitas(page: Any, _cell: EvalCell):
+    txt = _full_text(page)
+    fails = sorted(set(_FAIL_KEY_RE.findall(txt)))
+    dc = _DC_RE.search(txt)
+    datacenter = dc.group(1) if dc else "?"
+    behav = _BEHAV_RE.search(txt)
+    bscore = behav.group(1) if behav else "n/a (no synthetic input)"
+    if not fails and dc is None:
+        return (CheckStatus.INFO, None, f"unparsed: {txt[:120]!r}")
+    detail = (
+        f"fpscanner FAIL={fails or 'none'} datacenter={datacenter} "
+        f"behavioral={bscore}"
+    )
+    return (
+        CheckStatus.PASS if not fails else CheckStatus.FAIL,
+        float(len(fails)),
+        detail,
+    )
+
+
+# fingerprint-scan.com: "Bot Risk Score: 35/100"; the site flags >50 as bot.
+_RISK_RE = re.compile(r"Bot Risk Score:\s*(\d+)\s*/\s*100", re.IGNORECASE)
+
+
+def _parse_fingerprint_scan(page: Any, _cell: EvalCell):
+    txt = _full_text(page)
+    m = _RISK_RE.search(txt)
+    if not m:
+        return (CheckStatus.INFO, None, f"no risk score: {txt[:120]!r}")
+    score = int(m.group(1))
+    status = CheckStatus.PASS if score < 50 else CheckStatus.FAIL
+    return (status, float(score), f"bot risk {score}/100 (bar: <50)")
+
+
+def _parse_fingerprintjs(page: Any, _cell: EvalCell):
+    """FingerprintJS Pro Smart Signals: a block message means we were detected."""
+    low = _full_text(page).lower()
+    if "access denied" in low or "tampering detected" in low:
+        return (CheckStatus.FAIL, 1.0, "blocked: anti-detect tampering / access denied")
+    if "search for today's flights" in low or "flight" in low:
+        return (CheckStatus.PASS, 0.0, "not blocked (flight results served)")
+    return (CheckStatus.INFO, None, f"unparsed: {low[:120]!r}")
+
+
+def _parse_browserscan(page: Any, _cell: EvalCell):
+    """BrowserScan grades each probe Normal/Abnormal; any Abnormal = detected."""
+    txt = _full_text(page)
+    abnormal = len(re.findall(r"\bAbnormal\b", txt))
+    if abnormal == 0 and "Test Results:" in txt and "Normal" in txt:
+        return (CheckStatus.PASS, 0.0, "Test Results: Normal (0 Abnormal)")
+    if abnormal > 0:
+        return (CheckStatus.FAIL, float(abnormal), f"{abnormal} Abnormal signal(s)")
+    return (CheckStatus.INFO, None, f"unparsed: {txt[:120]!r}")
+
+
+# The scrapingcourse CF canary prints an explicit success line once solved; a
+# failure leaves the interstitial ("Just a moment...") in the DOM. This is the
+# ONLY site here that exercises solve_cloudflare (the others present no challenge).
+_CF_PASS = "you bypassed the cloudflare challenge"
+_CF_BLOCK = ("just a moment", "attention required", "verify you are human")
+
+
+def _parse_cloudflare(page: Any, _cell: EvalCell):
+    low = _full_text(page).lower()
+    if _CF_PASS in low:
+        return (CheckStatus.PASS, 0.0, "bypassed Cloudflare challenge")
+    if any(m in low for m in _CF_BLOCK):
+        return (CheckStatus.FAIL, 1.0, "stuck on Cloudflare interstitial")
+    return (CheckStatus.INFO, None, f"unparsed: {low[:120]!r}")
+
+
+# iphey renders a masthead verdict "Your Digital Identity Looks <Trustworthy|
+# Unreliable|Suspicious>" after an async fingerprint+IP correlation (slow → big
+# settle). Only "Trustworthy" is a clean pass.
+_IPHEY_RE = re.compile(r"Your Digital Identity Looks\s+([A-Za-z]+)", re.IGNORECASE)
+
+
+def _parse_iphey(page: Any, _cell: EvalCell):
+    txt = _full_text(page)
+    m = _IPHEY_RE.search(txt)
+    if not m:
+        return (CheckStatus.INFO, None, f"verdict not loaded: {txt[:120]!r}")
+    verdict = m.group(1)
+    ok = verdict.lower() == "trustworthy"
+    return (
+        CheckStatus.PASS if ok else CheckStatus.FAIL,
+        0.0 if ok else 1.0,
+        f"iphey verdict: {verdict}",
+    )
+
+
+_BROWSER_SITES: list[StealthSite] = [
+    StealthSite(
+        name="sannysoft",
+        url="https://bot.sannysoft.com/",
+        bar="0 failed cells",
+        parse=_parse_sannysoft,
+        settle_ms=3000,
+    ),
+    StealthSite(
+        name="deviceandbrowserinfo",
+        url="https://deviceandbrowserinfo.com/are_you_a_bot",
+        bar="isBot=false",
+        parse=_parse_areyouabot,
+        settle_ms=3000,
+    ),
+    StealthSite(
+        name="recaptcha_v3_score",
+        url="https://recaptcha-demo.appspot.com/recaptcha-v3-request-scores.php",
+        bar="score >= 0.7",
+        parse=_parse_recaptcha_v3,
+        # v3 runs grecaptcha.execute then round-trips a server verify; too-short a
+        # wait reads the page before the {"score":..} JSON lands (false 0.0).
+        settle_ms=12000,
+    ),
+    StealthSite(
+        name="creepjs",
+        url="https://abrahamjuliot.github.io/creepjs/",
+        bar="headless <=30%, no spoof tells",
+        parse=_parse_creepjs,
+        settle_ms=30000,
+    ),
+    StealthSite(
+        name="browserscan",
+        url="https://www.browserscan.net/bot-detection",
+        bar="0 Abnormal",
+        parse=_parse_browserscan,
+        settle_ms=8000,
+    ),
+    StealthSite(
+        name="incolumitas",
+        url="https://bot.incolumitas.com/",
+        bar="0 fpscanner FAIL",
+        parse=_parse_incolumitas,
+        settle_ms=12000,
+    ),
+    StealthSite(
+        name="fingerprint_scan",
+        url="https://fingerprint-scan.com/",
+        bar="bot risk < 50/100",
+        parse=_parse_fingerprint_scan,
+        settle_ms=20000,
+    ),
+    StealthSite(
+        name="fingerprintjs_demo",
+        url="https://demo.fingerprint.com/web-scraping",
+        bar="not blocked",
+        parse=_parse_fingerprintjs,
+        settle_ms=6000,
+    ),
+    StealthSite(
+        name="cloudflare_challenge",
+        url="https://www.scrapingcourse.com/cloudflare-challenge",
+        bar="bypass CF challenge (exercises solve_cloudflare)",
+        parse=_parse_cloudflare,
+        settle_ms=15000,
+    ),
+    StealthSite(
+        name="iphey",
+        url="https://iphey.com/",
+        bar="verdict Trustworthy",
+        parse=_parse_iphey,
+        # iphey's verdict loads via a slow async correlation; <~20s reads the
+        # "Temporary value" placeholder (false INFO).
+        settle_ms=25000,
+    ),
+]
+
+# S2 — per-property fingerprint pages: too visual to auto-grade; emit as manual
+# links so the operator can confirm the 03e levers (canvas/webgl/fonts/webrtc).
+_MANUAL_LINKS: list[tuple[str, str]] = [
+    ("browserleaks_canvas", "https://browserleaks.com/canvas"),
+    ("browserleaks_webgl", "https://browserleaks.com/webgl"),
+    ("browserleaks_fonts", "https://browserleaks.com/fonts"),
+    ("browserleaks_webrtc", "https://browserleaks.com/webrtc"),
+]
+
+
+def _run_browser_site(
+    site: StealthSite, *, proxy: str | None, headed: bool, screenshots: bool
+) -> CheckResult:
+    shot = str(SCREENSHOTS_DIR / f"S_{site.name}.png") if screenshots else None
+    action, cell = make_page_action(
+        evaluate_js=site.evaluate_js,
+        screenshot_path=shot,
+        pre_wait_ms=site.settle_ms,
+    )
+    kwargs: dict[str, Any] = {
+        "headless": not headed,
+        "network_idle": True,
+        "block_ads": True,
+        "solve_cloudflare": True,
+        "proxy": proxy,
+        "timeout": 120000,
+        "page_action": action,
+    }
+    # Single source of truth — the exact levers the production crawler ships.
+    kwargs.update(build_stealthy_kwargs(get_stealth_config()))
+
+    try:
+        page = StealthyFetcher.fetch(site.url, **kwargs)
+    except Exception as exc:  # noqa: BLE001 - never crash the run
+        return CheckResult(
+            suite="S",
+            name=site.name,
+            tier="scrapling-stealthy",
+            status=CheckStatus.ERROR,
+            bar=site.bar,
+            detail=f"fetch failed: {type(exc).__name__}: {exc}",
+            screenshot=shot,
+        )
+
+    try:
+        status, numeric, detail = site.parse(page, cell)
+    except Exception as exc:  # noqa: BLE001
+        status, numeric, detail = (
+            CheckStatus.ERROR,
+            None,
+            f"parse failed: {type(exc).__name__}: {exc}",
+        )
+    return CheckResult(
+        suite="S",
+        name=site.name,
+        tier="scrapling-stealthy",
+        status=status,
+        bar=site.bar,
+        detail=detail,
+        numeric=numeric,
+        screenshot=shot,
+    )
+
+
+async def _run_tls(proxy: str | None) -> CheckResult:
+    """S3 — JA3/JA4/PeetPrint of the impersonated static (curl_cffi) tier."""
+    try:
+        page = await AsyncFetcher.get(
+            "https://tls.peet.ws/api/all",
+            stealthy_headers=True,
+            impersonate="chrome",
+            proxy=proxy,
+            timeout=30,
+        )
+        data = page.json()
+        tls = data.get("tls", {}) if isinstance(data, dict) else {}
+        ja3 = tls.get("ja3_hash") or tls.get("ja3")
+        ja4 = tls.get("ja4")
+        peet = tls.get("peetprint_hash")
+        return CheckResult(
+            suite="S",
+            name="tls_fingerprint",
+            tier="scrapling-static",
+            status=CheckStatus.INFO,
+            bar="informational (diff vs real Chrome)",
+            detail=f"ja3={ja3} ja4={ja4} peet={peet}",
+        )
+    except Exception as exc:  # noqa: BLE001
+        return CheckResult(
+            suite="S",
+            name="tls_fingerprint",
+            tier="scrapling-static",
+            status=CheckStatus.ERROR,
+            bar="informational",
+            detail=f"{type(exc).__name__}: {exc}",
+        )
+
+
+async def _run_proxy_leak(proxy: str | None) -> CheckResult:
+    """S4 — record the exit IP seen by an echo endpoint (proves egress path)."""
+    try:
+        page = await AsyncFetcher.get(
+            "https://api.ipify.org?format=json",
+            impersonate="chrome",
+            proxy=proxy,
+            timeout=30,
+        )
+        data = page.json()
+        ip = data.get("ip") if isinstance(data, dict) else None
+        note = "via proxy" if proxy else "DIRECT (no proxy — datacenter IP)"
+        return CheckResult(
+            suite="S",
+            name="exit_ip",
+            tier="scrapling-static",
+            status=CheckStatus.INFO,
+            bar="not your real/datacenter IP",
+            detail=f"exit_ip={ip} ({note})",
+        )
+    except Exception as exc:  # noqa: BLE001
+        return CheckResult(
+            suite="S",
+            name="exit_ip",
+            tier="scrapling-static",
+            status=CheckStatus.ERROR,
+            bar="n/a",
+            detail=f"{type(exc).__name__}: {exc}",
+        )
+
+
+async def run_suite_s(
+    *, proxy: str | None, headed: bool, screenshots: bool
+) -> list[CheckResult]:
+    """Run the full stealth suite (browser tier sites + TLS + proxy + manual links)."""
+    results: list[CheckResult] = []
+
+    # Browser sites are sync (spin up a real browser) → offload per-site so one
+    # slow score doesn't stall the loop and the event loop stays clean.
+    for site in _BROWSER_SITES:
+        print(f"  [S] {site.name} ... (settle {site.settle_ms}ms)")
+        res = await asyncio.to_thread(
+            _run_browser_site,
+            site,
+            proxy=proxy,
+            headed=headed,
+            screenshots=screenshots,
+        )
+        results.append(res)
+
+    print("  [S] tls_fingerprint + exit_ip ...")
+    results.append(await _run_tls(proxy))
+    results.append(await _run_proxy_leak(proxy))
+
+    for name, url in _MANUAL_LINKS:
+        results.append(
+            CheckResult(
+                suite="S",
+                name=name,
+                tier="manual",
+                status=CheckStatus.INFO,
+                bar="open in a real headed run + compare",
+                detail=url,
+            )
+        )
+
+    return results

surfsense_backend/app/utils/crawl/__init__.py

@@ -0,0 +1,18 @@
+"""App-wide crawler block classification (Apache-2.0, generic).
+
+Phase 3e (Slice A). Mirrors ``app/utils/proxy`` and ``app/utils/captcha``: this
+package holds only the **generic, vendor-agnostic** glue — here, a pure block
+classifier (passive telemetry from public anti-bot markers). It is consumed by
+the separately licensed proprietary crawler to label ``CrawlOutcome.block_type``.
+
+The **bypass-specific tuning** (the stealth kwargs builder / geoip coherence, and
+the deferred WebGL spoof + humanize choreography) is NOT generic and lives under
+the proprietary boundary in ``app/proprietary/web_crawler/`` (``stealth.py``).
+"""
+
+from app.utils.crawl.classifier import BlockType, classify_block
+
+__all__ = [
+    "BlockType",
+    "classify_block",
+]

surfsense_backend/app/utils/crawl/classifier.py

@@ -0,0 +1,94 @@
+"""Block classifier — labels a fetched page so the ladder can learn (Phase 3e).
+
+Pure, additive, status + body-marker based. It attaches
+:class:`~app.proprietary.web_crawler.connector.CrawlOutcome.block_type` for
+telemetry / future escalation routing (per-domain memory, `03d` captcha routing)
+and **never** changes *when* a crawl is ``SUCCESS`` — `03c` bills on
+``status == SUCCESS``, so this stays read-only metadata.
+
+Markers mirror the proven set in the Camoufox-based FlareSolverr alternative
+``references/trawl-dev/packages/tiers/src/detect.ts`` (plus DataDome/Kasada for
+our enterprise targets). Regexes are compiled once at import (hot-path hygiene).
+"""
+
+import re
+from enum import Enum
+
+
+class BlockType(str, Enum):
+    """Coarse label for what a fetched page represents."""
+
+    OK = "ok"  # usable content, no challenge detected
+    CLOUDFLARE = "cloudflare"  # CF interstitial / Turnstile / DDoS-Guard
+    CAPTCHA_RECAPTCHA = "captcha_recaptcha"
+    CAPTCHA_HCAPTCHA = "captcha_hcaptcha"
+    DATADOME = "datadome"
+    KASADA = "kasada"
+    RATE_LIMITED = "rate_limited"
+    EMPTY = "empty"  # fetched but no HTML body
+    UNKNOWN = "unknown"  # blocking-ish status with no recognized marker
+
+
+# --- compiled marker patterns (hoisted; see Vercel rule 7.9) ---
+_CLOUDFLARE = re.compile(
+    r"just a moment"
+    r"|checking your browser"
+    r"|enable javascript and cookies to continue"
+    r"|verify you are human"
+    r"|cf-mitigated"
+    r"|id=\"(?:cf-)?challenge-running\""
+    r"|id=\"turnstile-wrapper\""
+    r"|ddos-guard",
+    re.IGNORECASE,
+)
+_TURNSTILE = re.compile(
+    r"class=\"cf-turnstile\""
+    r"|challenges\.cloudflare\.com/turnstile"
+    r"|cdn-cgi/challenge-platform[^\"']*turnstile",
+    re.IGNORECASE,
+)
+_HCAPTCHA = re.compile(r"class=\"h-captcha\"|hcaptcha\.com/1/api", re.IGNORECASE)
+_RECAPTCHA = re.compile(
+    r"class=\"g-recaptcha\"|google\.com/recaptcha|recaptcha\.net/recaptcha",
+    re.IGNORECASE,
+)
+_DATADOME = re.compile(r"datadome|geo\.captcha-delivery\.com", re.IGNORECASE)
+_KASADA = re.compile(r"kpsdk|x-kpsdk|kasada", re.IGNORECASE)
+
+# Statuses some CDNs use as a bot-gate before the real response (detect.ts:isBlocked).
+_BOT_GATE_STATUSES = frozenset({202, 403})
+
+
+def classify_block(status: int | None, html: str | None) -> BlockType:
+    """Label a fetched page from its HTTP status and HTML body.
+
+    Precedence: explicit rate-limit status, then specific anti-bot/captcha body
+    markers, then a generic bot-gate status with no marker, else ``OK`` (or
+    ``EMPTY`` when there is no body). Marker checks run before the generic
+    bot-gate so a ``403`` Cloudflare challenge is labeled ``CLOUDFLARE``, not
+    ``UNKNOWN``.
+    """
+    if status == 429:
+        return BlockType.RATE_LIMITED
+
+    if not html or not html.strip():
+        # No body: distinguish a blocking-ish status from a genuinely empty 200.
+        if status in _BOT_GATE_STATUSES:
+            return BlockType.UNKNOWN
+        return BlockType.EMPTY
+
+    if _CLOUDFLARE.search(html) or _TURNSTILE.search(html):
+        return BlockType.CLOUDFLARE
+    if _DATADOME.search(html):
+        return BlockType.DATADOME
+    if _KASADA.search(html):
+        return BlockType.KASADA
+    if _HCAPTCHA.search(html):
+        return BlockType.CAPTCHA_HCAPTCHA
+    if _RECAPTCHA.search(html):
+        return BlockType.CAPTCHA_RECAPTCHA
+
+    if status in _BOT_GATE_STATUSES:
+        return BlockType.UNKNOWN
+
+    return BlockType.OK

surfsense_backend/tests/unit/proprietary/web_crawler/test_connector.py

@@ -12,6 +12,7 @@ from app.proprietary.web_crawler import (
     WebCrawlerConnector,
 )
 from app.proprietary.web_crawler import connector as connector_module
+from app.utils.crawl import BlockType
 
 pytestmark = pytest.mark.unit
 
@@ -40,10 +41,10 @@ async def test_static_tier_success_short_circuits(
     crawler = WebCrawlerConnector()
     later_calls: list[str] = []
 
-    async def _static(_url: str) -> dict:
+    async def _static(_url: str, *_args) -> dict:
         return _result("scrapling-static")
 
-    async def _record_dynamic(_url: str) -> None:
+    async def _record_dynamic(_url: str, *_args) -> None:
         later_calls.append("dynamic")
         return None
 
@@ -70,10 +71,10 @@ async def test_escalates_to_dynamic_on_static_miss(
     """Static empty extraction escalates to the dynamic tier."""
     crawler = WebCrawlerConnector()
 
-    async def _empty(_url: str) -> None:
+    async def _empty(_url: str, *_args) -> None:
         return None
 
-    async def _dynamic(_url: str) -> dict:
+    async def _dynamic(_url: str, *_args) -> dict:
         return _result("scrapling-dynamic")
 
     monkeypatch.setattr(crawler, "_crawl_with_async_fetcher", _empty)
@@ -126,7 +127,7 @@ async def test_proxy_error_rotates_once_when_pool_backed(
     crawler = WebCrawlerConnector()
     calls = {"n": 0}
 
-    async def _flaky(_url: str) -> dict:
+    async def _flaky(_url: str, *_args) -> dict:
         calls["n"] += 1
         if calls["n"] == 1:
             raise RuntimeError("connection refused by upstream proxy")
@@ -149,7 +150,7 @@ async def test_proxy_error_no_retry_when_single_endpoint(
     crawler = WebCrawlerConnector()
     static_calls = {"n": 0}
 
-    async def _proxy_err(_url: str) -> None:
+    async def _proxy_err(_url: str, *_args) -> None:
         static_calls["n"] += 1
         raise RuntimeError("connection refused by upstream proxy")
 
@@ -173,7 +174,7 @@ async def test_captcha_defaults_zero_on_non_stealthy_success(
     """03d: a static success never touches captcha → fields stay 0/False."""
     crawler = WebCrawlerConnector()
 
-    async def _static(_url: str) -> dict:
+    async def _static(_url: str, *_args) -> dict:
         return _result("scrapling-static")
 
     monkeypatch.setattr(crawler, "_crawl_with_async_fetcher", _static)
@@ -194,7 +195,7 @@ async def test_captcha_state_surfaced_onto_outcome(
     async def _empty(_url: str, *_args) -> None:
         return None
 
-    async def _stealthy(_url: str, captcha_state: dict) -> dict:
+    async def _stealthy(_url: str, captcha_state: dict, *_args) -> dict:
         # Simulate the page_action having solved a captcha mid-fetch.
         captcha_state["attempts"] = 2
         captcha_state["solved"] = True
@@ -221,7 +222,9 @@ async def test_captcha_attempts_surface_even_when_crawl_fails(
     async def _empty(_url: str, *_args) -> None:
         return None
 
-    async def _stealthy_attempt_then_empty(_url: str, captcha_state: dict) -> None:
+    async def _stealthy_attempt_then_empty(
+        _url: str, captcha_state: dict, *_args
+    ) -> None:
         captcha_state["attempts"] = 1  # solve attempted but crawl still empty
         return None
 
@@ -245,7 +248,7 @@ async def test_non_proxy_error_never_retries(
     crawler = WebCrawlerConnector()
     calls = {"n": 0}
 
-    async def _boom(_url: str) -> None:
+    async def _boom(_url: str, *_args) -> None:
         calls["n"] += 1
         raise RuntimeError("totally unrelated failure")
 
@@ -261,3 +264,104 @@ async def test_non_proxy_error_never_retries(
 
     assert calls["n"] == 1  # not retried (not a proxy error)
     assert outcome.status is CrawlOutcomeStatus.EMPTY
+
+
+async def test_invalid_url_block_type_defaults_unknown() -> None:
+    """03e: a pre-fetch FAILED carries the default UNKNOWN block_type."""
+    outcome = await WebCrawlerConnector().crawl_url("not a url")
+
+    assert outcome.status is CrawlOutcomeStatus.FAILED
+    assert outcome.block_type is BlockType.UNKNOWN
+
+
+async def test_block_type_surfaced_onto_outcome(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """03e: a tier's block classification (in block_state) is stamped onto the
+    outcome — additive only, never gating SUCCESS."""
+    crawler = WebCrawlerConnector()
+
+    async def _empty(_url: str, *_args) -> None:
+        return None
+
+    async def _stealthy_blocked(
+        _url: str, _captcha_state: dict, block_state: dict
+    ) -> None:
+        # Simulate _build_result having classified a Cloudflare interstitial.
+        block_state["block_type"] = BlockType.CLOUDFLARE
+        return None
+
+    monkeypatch.setattr(crawler, "_crawl_with_async_fetcher", _empty)
+    monkeypatch.setattr(crawler, "_crawl_with_dynamic", _empty)
+    monkeypatch.setattr(crawler, "_crawl_with_stealthy", _stealthy_blocked)
+
+    outcome = await crawler.crawl_url("https://example.com")
+
+    assert outcome.status is CrawlOutcomeStatus.EMPTY
+    assert outcome.block_type is BlockType.CLOUDFLARE
+
+
+def test_build_result_classifies_into_block_state() -> None:
+    """03e: _build_result labels the fetched page into the passed block_state."""
+    crawler = WebCrawlerConnector()
+    block_state: dict = {"block_type": BlockType.UNKNOWN}
+
+    cf_html = "<html><head><title>Just a moment...</title></head></html>"
+    result = crawler._build_result(
+        cf_html,
+        "https://example.com",
+        "scrapling-stealthy",
+        allow_raw_fallback=False,
+        status=403,
+        block_state=block_state,
+    )
+
+    # Cloudflare interstitial: no real content extracted (None) but classified.
+    assert result is None
+    assert block_state["block_type"] is BlockType.CLOUDFLARE
+
+
+async def test_static_4xx_is_classified(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """03e: a static-tier 4xx bot-gate is classified before the early return
+    (otherwise the cheapest/first tier's block signal would be lost)."""
+    crawler = WebCrawlerConnector()
+
+    class _Page:
+        status = 403
+        html_content = "<title>Just a moment...</title>"
+
+    class _AsyncFetcher:
+        @staticmethod
+        async def get(*_a, **_k):
+            return _Page()
+
+    monkeypatch.setattr(connector_module, "AsyncFetcher", _AsyncFetcher)
+    monkeypatch.setattr(connector_module, "get_proxy_url", lambda: None)
+
+    block_state: dict = {"block_type": BlockType.UNKNOWN}
+    result = await crawler._crawl_with_async_fetcher(
+        "https://example.com", block_state
+    )
+
+    assert result is None  # 4xx => fall through to next tier
+    assert block_state["block_type"] is BlockType.CLOUDFLARE
+
+
+def test_build_result_ok_on_real_content() -> None:
+    """03e: a normal 200 page with content classifies OK."""
+    crawler = WebCrawlerConnector()
+    block_state: dict = {"block_type": BlockType.UNKNOWN}
+
+    html = "<html><body><article>" + ("Real content. " * 40) + "</article></body></html>"
+    crawler._build_result(
+        html,
+        "https://example.com",
+        "scrapling-static",
+        allow_raw_fallback=False,
+        status=200,
+        block_state=block_state,
+    )
+
+    assert block_state["block_type"] is BlockType.OK

surfsense_backend/tests/unit/proprietary/web_crawler/test_stealth.py

@@ -0,0 +1,113 @@
+"""Unit tests for the Phase 3e stealth kwargs builder (proprietary boundary)."""
+
+import pytest
+
+from app.config import config
+from app.proprietary.web_crawler.stealth import (
+    build_stealthy_kwargs,
+    get_stealth_config,
+    location_to_locale_timezone,
+)
+
+pytestmark = pytest.mark.unit
+
+
+class TestLocationToLocaleTimezone:
+    def test_alpha2_code(self):
+        assert location_to_locale_timezone("us") == ("en-US", "America/New_York")
+        assert location_to_locale_timezone("de") == ("de-DE", "Europe/Berlin")
+
+    def test_case_and_whitespace_insensitive(self):
+        assert location_to_locale_timezone("  US  ") == (
+            "en-US",
+            "America/New_York",
+        )
+
+    def test_full_country_name_alias(self):
+        assert location_to_locale_timezone("Germany") == (
+            "de-DE",
+            "Europe/Berlin",
+        )
+        assert location_to_locale_timezone("united kingdom") == (
+            "en-GB",
+            "Europe/London",
+        )
+
+    def test_leading_token_of_vendor_string(self):
+        # Vendor strings like "us:nyc" / "de-rotating" still resolve on the head.
+        assert location_to_locale_timezone("us:nyc") == (
+            "en-US",
+            "America/New_York",
+        )
+        assert location_to_locale_timezone("de-rotating") == (
+            "de-DE",
+            "Europe/Berlin",
+        )
+
+    def test_empty_and_unknown_return_none(self):
+        assert location_to_locale_timezone("") == (None, None)
+        assert location_to_locale_timezone(None) == (None, None)
+        assert location_to_locale_timezone("atlantis") == (None, None)
+
+
+class TestBuildStealthyKwargs:
+    def test_defaults_have_no_geoip_keys(self, monkeypatch):
+        # geoip off => locale/timezone_id absent (browser keeps system default).
+        monkeypatch.setattr(config, "CRAWL_GEOIP_MATCH_ENABLED", False)
+        monkeypatch.setattr(config, "CRAWL_BLOCK_WEBRTC", True)
+        monkeypatch.setattr(config, "CRAWL_HIDE_CANVAS", False)
+        monkeypatch.setattr(config, "CRAWL_GOOGLE_SEARCH_REFERER", True)
+        monkeypatch.setattr(config, "CRAWL_DNS_OVER_HTTPS", False)
+        monkeypatch.setattr(config, "RESIDENTIAL_PROXY_LOCATION", "us")
+
+        kwargs = build_stealthy_kwargs(get_stealth_config())
+
+        assert kwargs == {
+            "block_webrtc": True,
+            "hide_canvas": False,
+            "google_search": True,
+            "dns_over_https": False,
+        }
+        assert "locale" not in kwargs
+        assert "timezone_id" not in kwargs
+
+    def test_flags_reflect_config(self, monkeypatch):
+        monkeypatch.setattr(config, "CRAWL_GEOIP_MATCH_ENABLED", False)
+        monkeypatch.setattr(config, "CRAWL_BLOCK_WEBRTC", False)
+        monkeypatch.setattr(config, "CRAWL_HIDE_CANVAS", True)
+        monkeypatch.setattr(config, "CRAWL_GOOGLE_SEARCH_REFERER", False)
+        monkeypatch.setattr(config, "CRAWL_DNS_OVER_HTTPS", True)
+        monkeypatch.setattr(config, "RESIDENTIAL_PROXY_LOCATION", "")
+
+        kwargs = build_stealthy_kwargs(get_stealth_config())
+
+        assert kwargs["block_webrtc"] is False
+        assert kwargs["hide_canvas"] is True
+        assert kwargs["google_search"] is False
+        assert kwargs["dns_over_https"] is True
+
+    def test_geoip_on_adds_locale_timezone(self, monkeypatch):
+        monkeypatch.setattr(config, "CRAWL_GEOIP_MATCH_ENABLED", True)
+        monkeypatch.setattr(config, "CRAWL_BLOCK_WEBRTC", True)
+        monkeypatch.setattr(config, "CRAWL_HIDE_CANVAS", False)
+        monkeypatch.setattr(config, "CRAWL_GOOGLE_SEARCH_REFERER", True)
+        monkeypatch.setattr(config, "CRAWL_DNS_OVER_HTTPS", False)
+        monkeypatch.setattr(config, "RESIDENTIAL_PROXY_LOCATION", "de")
+
+        kwargs = build_stealthy_kwargs(get_stealth_config())
+
+        assert kwargs["locale"] == "de-DE"
+        assert kwargs["timezone_id"] == "Europe/Berlin"
+
+    def test_geoip_on_but_unknown_location_skips(self, monkeypatch):
+        monkeypatch.setattr(config, "CRAWL_GEOIP_MATCH_ENABLED", True)
+        monkeypatch.setattr(config, "CRAWL_BLOCK_WEBRTC", True)
+        monkeypatch.setattr(config, "CRAWL_HIDE_CANVAS", False)
+        monkeypatch.setattr(config, "CRAWL_GOOGLE_SEARCH_REFERER", True)
+        monkeypatch.setattr(config, "CRAWL_DNS_OVER_HTTPS", False)
+        monkeypatch.setattr(config, "RESIDENTIAL_PROXY_LOCATION", "atlantis")
+
+        kwargs = build_stealthy_kwargs(get_stealth_config())
+
+        assert "locale" not in kwargs
+        assert "timezone_id" not in kwargs

surfsense_backend/tests/unit/utils/test_crawl_classifier.py

@@ -0,0 +1,75 @@
+"""Unit tests for the Phase 3e block classifier (Apache-2 layer)."""
+
+import pytest
+
+from app.utils.crawl import BlockType, classify_block
+
+pytestmark = pytest.mark.unit
+
+
+class TestClassifyBlock:
+    def test_ok_on_plain_content(self):
+        assert classify_block(200, "<html><body>hello</body></html>") is BlockType.OK
+
+    def test_none_body_is_empty(self):
+        assert classify_block(200, None) is BlockType.EMPTY
+        assert classify_block(200, "   ") is BlockType.EMPTY
+
+    def test_rate_limited_takes_precedence(self):
+        # 429 wins even if a challenge marker is present in the body.
+        assert (
+            classify_block(429, '<div class="cf-turnstile"></div>')
+            is BlockType.RATE_LIMITED
+        )
+
+    @pytest.mark.parametrize(
+        "html",
+        [
+            "<title>Just a moment...</title>",
+            "<h1>Checking your browser before accessing</h1>",
+            "please enable javascript and cookies to continue",
+            "verify you are human",
+            '<div id="challenge-running"></div>',
+            '<div id="turnstile-wrapper"></div>',
+            '<div class="cf-turnstile"></div>',
+            "blocked by ddos-guard.net",
+        ],
+    )
+    def test_cloudflare_markers(self, html):
+        assert classify_block(403, html) is BlockType.CLOUDFLARE
+
+    def test_hcaptcha(self):
+        assert (
+            classify_block(200, '<div class="h-captcha"></div>')
+            is BlockType.CAPTCHA_HCAPTCHA
+        )
+
+    def test_recaptcha(self):
+        assert (
+            classify_block(200, '<div class="g-recaptcha"></div>')
+            is BlockType.CAPTCHA_RECAPTCHA
+        )
+
+    def test_datadome(self):
+        assert (
+            classify_block(403, "var dd={'host':'geo.captcha-delivery.com'}")
+            is BlockType.DATADOME
+        )
+
+    def test_kasada(self):
+        assert classify_block(200, "window.KPSDK.configure()") is BlockType.KASADA
+
+    def test_bot_gate_status_without_marker_is_unknown(self):
+        # 202/403 with no recognized challenge marker => generic blocked-ish.
+        assert classify_block(202, "<html><body>x</body></html>") is BlockType.UNKNOWN
+        assert classify_block(403, "<html><body>x</body></html>") is BlockType.UNKNOWN
+
+    def test_bot_gate_status_empty_body_is_unknown(self):
+        assert classify_block(403, None) is BlockType.UNKNOWN
+
+    def test_cloudflare_marker_beats_generic_403(self):
+        # A 403 Cloudflare page is CLOUDFLARE, not UNKNOWN.
+        assert (
+            classify_block(403, "<title>Just a moment...</title>")
+            is BlockType.CLOUDFLARE
+        )