SurfSense/plans/backend/03e-stealth-hardening.md

# Phase 3e — Stealth hardening (the in-house "undetectable" moat)

> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
> Depends on `03a` (Scrapling StealthyFetcher tier + `CrawlOutcome`) and `03b` (app-wide proxy provider). Precedes `03d` (captcha) in the escalation order — hardening **avoids** challenges; captcha solving is the paid last resort for the ones we can't avoid. Touches `03c` only via the deferred paid-unblocker tier (its own billing, decided if/when built).

> **Architecture reconciliation (read before building).** Earlier drafts of this plan referenced a `FetchStrategy` seam / strategy-object contract `(url, ctx) -> CrawlOutcome`. **That seam does not exist in shipped code.** `03a`–`03d` shipped a *hardcoded 3-tier ladder* inside `WebCrawlerConnector.crawl_url` (`app/proprietary/web_crawler/connector.py`): plain methods `_crawl_with_async_fetcher` / `_crawl_with_dynamic` / `_crawl_with_stealthy(_sync)` wrapped by `_run_tier_with_proxy_retry`, every tier returning the shared `CrawlOutcome`. `03e` therefore implements levers **not** as strategy classes but as: (1) a **centralized per-tier kwargs builder** (`app/proprietary/web_crawler/stealth.py` — **proprietary**, since it's bypass-specific tuning; the generic block classifier stays Apache-2 in `app/utils/crawl/`) that turns config → `StealthyFetcher`/`AsyncFetcher` kwargs, imported by both the connector and `03f`'s harness so there's no test-vs-prod drift; (2) small **helpers around the existing ladder** (block classifier, strategy memory); and (3) the deferred paid tier as a config-gated **extra ladder step**, not a `FetchStrategy`. The captcha precedent holds: generic config/plumbing → `app/utils/`; actual bypass logic (WebGL spoof JS, humanize choreography) → `app/proprietary/web_crawler/`.

> **Implementation note.** Same convention as `03a`–`03d`: Phases 1–2 are **SHIPPED** (live code says `workspace_id`/`Workspace` — substitute for any old `search_space_*`/`SearchSpace` citations and grep the new name); citations also predate `03a`'s `crawl_url` refactor; locate code by **symbol/grep**, not absolute lines. Scrapling references point at the on-disk (gitignored) `references/Scrapling/` checkout pinned to `scrapling[fetchers]>=0.4.9` (`pyproject.toml:91`).

> **Build status.** Sliced for risk (see "Build slicing" below). **Slice A (SHIPPED in this pass):** centralized stealth kwargs builder + fingerprint flags + geoip-from-proxy-location + block classifier (additive `CrawlOutcome.block_type`) + fonts/Xvfb packages in the image. **Slice B / C (deferred):** headed-Xvfb runtime path, humanization/WebGL `init_script` assets, persistent profiles, strategy memory, paid-unblocker stub. Effectiveness of every lever is graded in `03f` (manual), not unit-tested here.
>
> **First baseline evidence (`03f`, 2026-06-30 — headless, rotating residential, captcha OFF, Slice-A only with `CRAWL_GEOIP_MATCH_ENABLED=false`).** Suite S = **6 PASS / 4 FAIL** (full table in `03f`). The 4 fails map cleanly onto this plan's deferred levers and confirm the predicted ceiling — they are **not** Slice-A regressions:
> - **iphey "Unreliable"** → the predicted geoip-incoherence tell (browser tz/locale vs proxy exit geo). It's now the live regression test for **§1 geoip coherence** — flipping `CRAWL_GEOIP_MATCH_ENABLED` is the expected fix to validate next.
> - **CreepJS `hasHeadlessWorkerUA: true`** (headless 33%) → the Web **Worker** `navigator.userAgent` still leaks `HeadlessChrome` (main-thread UA is clean). A concrete, plausibly-fixable **Slice-B** target (worker UA override via `init_script`), beyond today's Slice-A flags.
> - **FingerprintJS Pro "tampering detected, access denied"** → the documented **commercial/device-fingerprint wall** (§2c WebGL/GPU gap). Needs Slice B/C or the §8 paid tier; expected to stay red on the free stack.
> - **incolumitas** → a single legacy `fpscanner WEBDRIVER` check (all modern checks pass); lowest priority. Its IP classifier also flip-flopped `is_datacenter` true/false across rotations, exposing that the `anonymous_proxies` pool mixes datacenter+residential exits (a proxy-provider variable, not a code issue).

## Objective

Push the Universal WebURL Crawler as far toward "undetectable" as the **free / in-house stack** allows, so we hold a scraping moat for the next ~4–6 months **without** a third-party unblocker (ZenRows/ScrapFly/Bright Data) or a source-patched browser (CloakBrowser — rejected on licensing). Everything here is **runtime/config-level** stealth layered on Scrapling's patchright-Chromium (`03a` engine note): geoip fingerprint coherence, persistent profiles, headed execution, real fonts, and behavioral humanization — plus a **block classifier** + **per-domain strategy memory** so the ladder learns and the cheap tiers get skipped once a domain's working strategy is known. The paid-unblocker tier is defined here as a **deferred config-gated ladder step** (the explicit escape hatch for when in-house maintenance gets too costly), not built now.

## The realistic ceiling (be honest with downstream devs)

Scrapling already ships strong **runtime** stealth by default: `DEFAULT_ARGS + STEALTH_ARGS` (incl. `--disable-blink-features=AutomationControlled`), a persistent context, and `navigator.webdriver` masking via **patchright**. Be precise about what patchright actually is: it patches the **Playwright driver** to remove *automation leaks* — it never calls `Runtime.enable`, disables `Console.enable`, and reaches closed shadow roots (verified: [patchright-python](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright-python), [Scrappey](https://scrappey.com/qa/web-scraping-apis/what-is-patchright)). It hides **"I'm automated," not "this is a different machine."** It does **not** spoof canvas, WebGL, audio, screen, or GPU. With `03b` residential proxies + this plan's coherence/humanization on top, the crawler reliably handles **Cloudflare** (via `03a` `solve_cloudflare`) and the long tail of **moderate** anti-bot — which is most CI targets.

**The device-fingerprint wall (the WebGL/GPU gap — see §2c).** Our worker has no GPU, so Chromium's WebGL renderer reports `Google SwiftShader` / `Mesa llvmpipe` — a string anti-bot vendors treat as ">95% bot" ([ipasis](https://ipasis.com/blog/webgl-fingerprinting-bot-detection), [Scrappey](https://scrappey.com/qa/anti-bot/what-is-webgl-fingerprinting)). Scrapling exposes only `allow_webgl` (on/off — and *off* is itself a tell), **no renderer spoof**; CloakBrowser fixes this at the C++ level (`--fingerprint-gpu-renderer`) and we cannot. Concretely, **without a real GPU or a source-patched binary we cannot**:

1. **Pass pure device-fingerprint bot checks** — FingerprintJS bot-detection, CreepJS / BrowserScan "is this a real device." CloakBrowser passes these (BrowserScan 4/4); our SwiftShader renderer fails them.
2. **Reach human-level reCAPTCHA v3 *scores*** (~0.7–0.9) **on our own browser** — we land ~0.1–0.5. (Note: `03d`'s **paid solver sidesteps this for token challenges** — the token is minted on the vendor's real-browser infra, so its score isn't ours. The gap only bites where the *site itself* reads our live v3 score.)
3. **Max out the device-fingerprint component** of DataDome / Kasada / Akamai — one of several reasons we plateau below managed APIs.

**Numbers (2026 benchmarks — use these in any user-facing copy).** Free stack **+ residential**: Basic (CF) ~99%; Medium (CF Pro / PerimeterX) ~90%; **Enterprise (DataDome / Kasada / Akamai) ~60–75% initially, decaying toward ~60% within 24h** as their ML clusters the fingerprint (managed APIs hold ~89%) ([scrapewise DataDome 2026](https://scrapewise.ai/blogs/bypass-datadome-web-scraping-2026)). So the honest line is **"partial and decaying, not SLA-grade,"** — neither "can't beat" nor "beats everything." For CI targets the top-tier defenses are the exception, so the in-house moat is a sound 4–6 month bet, with the deferred paid tier (§8) as the **evidence-driven** escape hatch (`03f`'s scorecard is the trigger). **Do not promise "beats everything."**

## Levers (wired via the centralized kwargs builder + ladder helpers — callers stay outcome-only)

### 1. Geoip fingerprint coherence (match the browser to the proxy exit IP)

A residential IP in Berlin behind an `en-US`/`America/New_York` browser is an instant tell. Make the fingerprint cohere with the proxy's exit geo:

- **Primary source (Slice A, no network call): the configured `RESIDENTIAL_PROXY_LOCATION` (`03b`, `config:1048`).** Map that string (best-effort: ISO-3166 alpha-2 like `us`/`de`, or a common country name) → a representative `(locale, timezone_id)`. Coarse country granularity is fine (per "Risks": wrong-but-coherent beats default-mismatched); unknown/empty → skip (leave Scrapling's system default). **No exit-IP resolution** — avoids a per-crawl network round-trip + failure mode. (A `geoip2`/MaxmindLite exit-IP path is a later refinement, not MVP.)
- Pass the matched values into StealthyFetcher via the kwargs builder: `locale=` (drives `navigator.language` + `Accept-Language`) and `timezone_id=` (confirmed real kwargs on `StealthyFetcher.fetch`). These flow into the Playwright context.
- The stealthy tier **already captures the chosen proxy endpoint once** (`_crawl_with_stealthy_sync`: `proxy = get_proxy_url()`, the same value handed to `03d`'s solver for IP-coherence) — reuse that captured `proxy`/location, **don't** re-call `get_proxy_url()` (which rotates on a pool-backed provider, `03b`).
- **Caveat (be honest):** Scrapling applies `locale`/`timezone_id` via Playwright **CDP emulation**, which advanced systems can flag *as* emulation — CloakBrowser sets these as *binary flags* precisely to avoid "detectable CDP emulation" (`references/CloakBrowser/README.md:309`). Still strictly better than a mismatched default (wrong-but-coherent beats `UTC`+`en-US` behind a Berlin IP), but it's a weaker variant than a source-patched browser, not parity.

### 2. Fingerprint flags Scrapling already exposes (turn them on)

- `hide_canvas` — Scrapling adds **random** noise to canvas ops (verified in the installed `StealthyFetcher.fetch` docstring). **Not "free safe":** random-per-call noise is itself a signal on FingerprintJS/CreepJS-class detectors — an *unstable* canvas hash is as loud as `navigator.webdriver=true` ([dev.to](https://dev.to/tanwydd/how-your-canvas-fingerprint-gets-you-caught-and-why-random-noise-makes-it-worse-3ba1)); CloakBrowser's own FPJS fix is `--fingerprint-noise=false`. **Default OFF; gate per-domain and validate in `03f`.** Harmless against Cloudflare/moderate, potentially harmful against device-fingerprint graders. (Canvas stealth that actually *helps* must be **deterministic per persistent profile**, not random — out of scope here.)
- `block_webrtc=True` — forces WebRTC to respect the proxy, preventing the **real local IP leak** that unmasks proxied browsers (`:41`). **This one is genuinely cheap + safe — keep default TRUE.**
- `dns_over_https` — Scrapling can route DNS via Cloudflare DoH to stop the **DNS leak** that unmasks proxied browsers (same anti-leak class as `block_webrtc`; confirmed kwarg on `StealthyFetcher.fetch`). **Trade-off: it adds a DNS round-trip, a (small) latency cost — so per the "no speed regression" constraint it ships *off by default* (`CRAWL_DNS_OVER_HTTPS`, default FALSE), pre-wired for operators who prefer leak-safety over the marginal latency.** (Idea sourced from the Camoufox-based FlareSolverr alternatives `references/Byparr-main` / `references/trawl-dev`, which treat DNS/WebRTC leak coherence as first-class.)
- `google_search=True` (default) — sets a Google referer so the first hit looks like organic arrival (`:45`); override per-need via `extra_headers` (`:46`).
- `additional_args=` — last-priority Playwright context overrides for anything not surfaced as a first-class param (`:51`).

### 2b. HTTP-tier TLS fingerprint (the AsyncFetcher tier — `impersonate`)

The cheap static tier (`03a` tier 1) is the **first** thing every crawl hits. `03a` **already ships `impersonate="chrome"`** on it (`app/proprietary/web_crawler/connector.py`, the `AsyncFetcher.get` call) — Scrapling's static engine selects a matching curl_cffi browser profile (`references/Scrapling/scrapling/engines/static.py:36–47`), so the HTTP tier's **JA3/JA4/HTTP-2** already matches a real Chrome and coheres with the browser tiers' UA. `03e`'s remaining work here is **not** to add it, but to (a) fold the `impersonate` profile into the centralized per-tier kwargs builder (below) so it stays the single source of truth, and (b) keep the chosen profile coherent with the proxy exit / UA. `03f §S3` validates the shipped parity against `tls.peet.ws`.

### 2c. WebGL / GPU renderer (the biggest free-stack gap — partial mitigation only)

This is the "device-fingerprint wall" from the ceiling section. On a GPU-less worker the WebGL renderer string is `Google SwiftShader` / `ANGLE (Mesa, llvmpipe…)` — an instant tell on any detector that reads `WEBGL_debug_renderer_info`. Scrapling gives us **no renderer knob** (only `allow_webgl` on/off, and *off* is a tell). Three options, increasing strength + cost:

1. **DIY JS spoof (optional, OFF by default — Slice B).** An `init_script` overrides `getParameter` for `UNMASKED_VENDOR_WEBGL` / `UNMASKED_RENDERER_WEBGL` to a believable, **platform-coherent** value (e.g. an Intel/ANGLE renderer that matches the spoofed Windows UA), held **consistent per persistent profile** (§3). **Impl note:** Scrapling's `init_script` is **an absolute path to a `.js` file**, *not* an inline string (confirmed in `StealthyFetcher.fetch` docstring) — so this ships as a **bundled `.js` asset** in `app/proprietary/web_crawler/` (next to `03d`'s captcha logic) passed by absolute path; the kwargs builder resolves the path when `CRAWL_WEBGL_SPOOF_ENABLED`. **Must** also patch `Function.prototype.toString` so the override still returns `[native code]` — otherwise *the spoof itself is the signal* (exactly the `toString()` probe Kasada catalogs, [web-scraping-guide](https://web-scraping-guide.com/)). This defeats **string-based** WebGL checks (the common case) but **not** render-output pixel-hash or GPU-perf probes (FingerprintJS-grade) — a software rasterizer's actual draw output still differs and can't be faked in JS. It is the brittle config-level approach patchright/CloakBrowser explicitly warn rots; treat it as a **cheap, test-gated win, not the moat**.
2. **Real GPU passthrough (hosted workers only).** `--device /dev/dri` + ANGLE/EGL so WebGL reports a *genuine* renderer with *real* render output, defeating even output/perf probes. Best technical fix; costs GPU instances + image complexity ([livekit/egress GPU notes](https://stackoverflow.com/questions/78985698)). A hosted-tier escalation, off for self-hosted.
3. **Defer to the paid unblocker (§8).** Route the residual device-fingerprint-gated domains (flagged by the §7 classifier as in-house-unreachable) to the external unblocker. The pragmatic moat-bounding fallback.

**Decision for this MVP:** ship option 1 as an opt-in flag (`CRAWL_WEBGL_SPOOF_ENABLED`, default FALSE), name option 2 as the hosted escalation, and lean on §8 for the rest. Quantify all three in `03f`.

### 3. Persistent per-domain profiles (look like a returning human)

Scrapling defaults to a **temporary** user-data dir (fresh = suspicious). Use `user_data_dir=` (confirmed real kwarg; `launch_persistent_context`) to keep a **persistent profile per domain** (or per domain+proxy-geo), so cookies/localStorage/site-trust carry across crawls and the browser presents as a returning visitor rather than a brand-new incognito session. Store profiles under a configured dir (`shared_tmp`-style volume so API + worker share them).

- **Concurrency hazard (Slice C design item — why this is deferred, not a one-liner):** Chromium **hard-locks** a `user_data_dir`; two crawls touching the same profile dir at once crash (`SingletonLock`). Our crawler runs concurrently (multiple connectors/chats across workers), so a naive per-domain dir collides. The fix needs one of: (a) an async lock keyed by profile dir so same-domain crawls serialize (adds latency, can deadlock under load), (b) per-`(domain, worker, pid)` dirs + a periodic prune (loses some cross-run trust, no contention), or (c) persistent only when the strategy memory (§7) says a domain *needs* it, temp otherwise. Decide in Slice C alongside §7; **Slice A keeps Scrapling's temp default.**

### 4. Headed execution under Xvfb (defeat headless tells)

`headless` defaults to hidden (`stealth_chrome.py:19/71`). Many WAFs flag headless Chromium. Run **headful** (`headless=False`) inside a virtual framebuffer (**Xvfb**) in the Docker worker so the browser is "visible" to itself but needs no real display. Gate behind a config flag (off by default for self-hosted, on for hosted workers that have Xvfb installed).

### 5. Real fonts (canvas/emoji hash realism)

A minimal container has almost no fonts, making canvas/emoji fingerprint hashes obviously synthetic. Install real font packages so font-enumeration + canvas/emoji hashes resemble a real desktop. Use the set CloakBrowser ships as proven against Kasada/Akamai emoji-canvas hashing (`README.md:737–739`): `fonts-noto-color-emoji`, `fonts-unifont`, `fonts-ipafont-gothic` (CJK), `fonts-wqy-zenhei` (CJK), `fonts-tlwg-loma-otf` — plus `fonts-dejavu`/`fonts-liberation` for Latin coverage. `Dockerfile` change only (`apt-get install fonts-*`).

### 6. Behavioral humanization (DIY — Scrapling has no `humanize` for the Chromium engine)

Unlike Camoufox, the patchright-Chromium StealthyFetcher exposes **no built-in `humanize`** (verified: zero matches in the StealthyFetcher param set). So humanization is custom, injected via the existing hooks:

- `page_action=` (runs after navigation) — randomized mouse moves/curves, scrolls, hover-before-click, and small think-time delays before extraction. This is the **same hook `03d` uses** for token injection — `_crawl_with_stealthy_sync` currently builds exactly **one** `page_action` (the captcha injector), so Slice B must **compose** two callables (humanize → optional captcha solve) into the single `page_action` kwarg, not overwrite it.
- `init_script=` (an **absolute path to a `.js` file** executed on page creation — *not* inline JS, see §2c) — early shims for any residual JS tells not covered by patchright (and the §2c WebGL spoof, if enabled).
- Tunable `wait`/`network_idle` (`:29,27`) so dwell time isn't robotically constant.
- **Concrete rules (from CloakBrowser's reCAPTCHA notes, `README.md:1280–1314` — encode these):** use native `time.sleep`, **never** `page.wait_for_timeout()` (it emits CDP traffic detectors flag); prefer `page.type()` over `fill()` for any input (real keystrokes vs. value-set); dwell **15s+** before triggering score-based checks; space repeat hits to score-gated domains **30s+**; keep a **stable identity per domain** (persistent profile §3 + consistent spoofed values §2c) so a score-based system sees a *returning* device, not a fresh one each visit.

### 7. Block classifier + per-domain strategy memory (the "learning ladder")

Two small in-house pieces make the ladder smart instead of brute-force:

- **Block classifier (Slice A — additive, pure, unit-testable).** Inspect the fetched page (`status` + body markers; cookies/headers are not threaded through `_build_result` today, so MVP is **status + body-marker based**) and label the outcome: `OK` / `CLOUDFLARE` / `CAPTCHA_RECAPTCHA` / `CAPTCHA_HCAPTCHA` / `DATADOME` / `KASADA` / `RATE_LIMITED` / `EMPTY` / `UNKNOWN`. **Concrete markers — adopt the set proven in `references/trawl-dev/packages/tiers/src/detect.ts` (a near-identical classifier):** Cloudflare/DDoS-Guard → `cf-mitigated`-style title strings `"just a moment"` / `"checking your browser"` / `"enable javascript and cookies to continue"` / `"verify you are human"`, DOM ids `challenge-running` / `cf-challenge-running` / `turnstile-wrapper`, and `ddos-guard.net`; Turnstile → `class="cf-turnstile"` / `challenges.cloudflare.com/turnstile`; hCaptcha → `class="h-captcha"` / `hcaptcha.com/1/api`; reCAPTCHA → `class="g-recaptcha"` / `google.com/recaptcha` / `recaptcha.net`; DataDome → `datadome` script/cookie; **status `202`/`403`/`429` as a bot-gate** (`202` is an IMDb-style pre-gate, per `detect.ts:isBlocked`). **Invariant: the classifier is purely *additive* — it attaches `CrawlOutcome.block_type` and feeds telemetry/routing; it MUST NOT change *when* `SUCCESS` is returned, because `03c` bills on `status == SUCCESS` (`CrawlOutcome` docstring).** Wiring mirrors `03d`'s proven `captcha_state` pattern: a per-call `block_state` dict mutated in `_build_result` (which has `raw_html` + `status`) and stamped onto every `CrawlOutcome` return path in `crawl_url`.
- **Per-domain strategy memory (Slice C — needs a new dependency + control-flow change).** Cache the **strategy that last succeeded per domain** (Redis key `crawl:strategy:{domain}` with TTL). Next crawl starts at the known-good tier, skipping cheaper tiers that always fail there. **Why deferred:** `WebCrawlerConnector()` is currently constructed with **no args** and is Redis/DB-free; this requires injecting a cache client (best-effort, no-op when absent) **and** letting `crawl_url` skip ladder tiers — a control-flow change to the tuned ladder. No DB migration (Redis, best-effort, self-healing on miss), but it is a deliberate design item, not a flag flip.
- **Solved-session cache (Slice C refinement — speed-positive, the strongest version of the memory above).** Beyond remembering *which tier* won, cache the **solved session itself** — the `cf_clearance`/`__cf_bm` cookies + the UA that cleared the challenge — in Redis (`crawl:session:{domain}` with TTL), and **replay** it on the next crawl via Scrapling's `cookies=` input, so a returning hit **skips the expensive solve entirely** (the pattern `references/trawl-dev` Tier 2 uses for its ~500ms repeat path: `browser/src/session.ts` `session:{domain}` save/load/invalidate; `tiers/src/tier2.ts` replays cookies, and on a re-challenge marks `session-expired` → invalidate → escalate). **Net effect is faster, not slower** (it removes a solve), so it fits the "no speed regression" bar. **Caveat:** `cf_clearance` is **IP+UA-bound**, so it pays off most behind **sticky/static proxies** (the `03b` static-proxy future) and the cache key should include proxy-geo; under rotating residential the replayed cookie may not match the new exit IP (best-effort, self-healing via the `session-expired` invalidate).

## 8. Deferred: paid-unblocker tier (the escape hatch, NOT built now)

The moat strategy is explicit: **maintain in-house bypass for ~4–6 months, then move hostile targets to a paid unblocker if demand/maintenance justifies it.** That switch is **evidence-driven, not a guess** — `03f`'s manual scorecard quantifies the free-stack ceiling over time and is the documented trigger for flipping this tier. Pre-wire the seam so the switch is a config flip, not a refactor:

- Define (but do not implement) a paid-unblocker tier — a `_crawl_with_paid_unblocker(url) -> dict | None` method returning the same shape as the existing tiers, appended **last** in `crawl_url`'s ladder (after the stealthy tier) and active only when an env flag + API key are set. (Same plain-method shape as today's tiers — there is no `FetchStrategy` contract to satisfy.)
- It would call an external unblocker (ZenRows/ScrapFly/Bright Data Web Unlocker) for the residual `DATADOME`/`KASADA`/`reCAPTCHA-Enterprise` domains the block classifier flags as unreachable in-house.
- **Its own billing** (cost-plus pass-through, decided at build time) — separate from `03c`'s flat crawl unit and `03d`'s per-solve unit, because unblocker pricing is per-request and provider-specific.
- Until built, those domains simply return non-`SUCCESS` (free under `03c`). This keeps the umbrella's "WebURL Crawler is the moat" honest while bounding our maintenance risk.

## Config / env changes

Add (all default OFF / conservative; next to the `03b`/`03c` knobs in `config/__init__.py` + `.env.example`):

- `CRAWL_GEOIP_MATCH_ENABLED` (default FALSE; Slice A — maps `RESIDENTIAL_PROXY_LOCATION` → `locale`/`timezone_id`, no exit-IP lookup).
- `CRAWL_BLOCK_WEBRTC` (default TRUE — cheap, safe; Slice A), `CRAWL_HIDE_CANVAS` (default **FALSE** — random canvas noise can itself be a tell; opt-in + `03f`-validated, see §2; Slice A), and `CRAWL_GOOGLE_SEARCH_REFERER` (default TRUE — Scrapling's `google_search`; Slice A).
- `CRAWL_DNS_OVER_HTTPS` (default **FALSE** — anti DNS-leak, but adds a DNS round-trip; default-off to honor the "no speed regression" bar, pre-wired for leak-safety-first operators; §2; Slice A).
- `CRAWL_WEBGL_SPOOF_ENABLED` (default FALSE) + optional `CRAWL_WEBGL_VENDOR` / `CRAWL_WEBGL_RENDERER` override strings (§2c; `toString`-safe JS spoof, defeats string-checks only).
- `CRAWL_PERSISTENT_PROFILES_DIR` (unset → Scrapling's temp default; set → per-domain profiles).
- `CRAWL_HEADED_XVFB_ENABLED` (default FALSE; requires Xvfb in the image).
- `CRAWL_HUMANIZE_ENABLED` (default TRUE) + dwell/jitter bounds.
- `CRAWL_STRATEGY_MEMORY_TTL_S` (default e.g. `86400`; 0 → disabled).
- `CRAWL_PAID_UNBLOCKER_ENABLED` (default FALSE) + provider/key (deferred tier).

## Docker changes (`surfsense_backend/Dockerfile`)

- Install **Xvfb** + the proven font set (`fonts-noto-color-emoji`, `fonts-unifont`, `fonts-ipafont-gothic`, `fonts-wqy-zenhei`, `fonts-tlwg-loma-otf`, `fonts-dejavu`, `fonts-liberation`; §5) in the worker image. (Note from `03a`: also drop the stale "+ Camoufox" comment near `:112`; `scrapling install` only fetches Chromium.)
- Headed runs need the browser launched under `xvfb-run` (or an Xvfb display in the worker entrypoint), gated by `CRAWL_HEADED_XVFB_ENABLED`.

## Build slicing (risk-staged; `03f` validates effectiveness)

The levers split by risk/testability. **Slice A is built now** (prod-safe, config-gated, defaults preserve today's behavior, wiring + classifier unit-tested). **B/C are deferred** (need infra or architectural design; mostly only verifiable in `03f`).

- **Slice A (SHIPPED):** centralized kwargs builder → fingerprint flags (`block_webrtc` ON, `hide_canvas` OFF, `google_search` ON, `dns_over_https` OFF); geoip coherence from `RESIDENTIAL_PROXY_LOCATION`; fonts + Xvfb **packages** in the Dockerfile; **additive** block classifier (`CrawlOutcome.block_type`, classified in `_build_result` **and** the static tier's `>=400` early-return so the first/cheapest-tier bot-gate isn't lost). **Licensing split:** the stealth kwargs builder is **proprietary** (`app/proprietary/web_crawler/stealth.py` — bypass tuning); the block classifier stays **Apache-2** (`app/utils/crawl/classifier.py` — generic passive telemetry from public markers). **`extra_headers`/`additional_args` were intentionally NOT wired** — they're last-priority override hatches with no clean env representation (dict-in-env) and no concrete need yet; add them when a real use appears rather than speculative config (YAGNI).
- **Slice B (next):** headed-Xvfb **runtime path** (`headless=False` + entrypoint), humanization `page_action` (composed with `03d`'s injector), WebGL-spoof `init_script` `.js` asset (proprietary).
- **Slice C (defer):** per-domain persistent profiles (concurrency design, §3), per-domain strategy memory + tier-skipping (`crawl_url` control-flow change + cache injection, §7), **solved-session cache** (`cf_clearance` replay via `cookies=`, §7 — speed-positive, best with sticky proxies), **warm browser pool** (a hosted-scale latency win — `references/trawl-dev` `browser/src/pool.ts` keeps N warm instances with an acquire-timeout→429; Scrapling's `StealthySession` already has page pooling), paid-unblocker config stub (§8). GPU passthrough stays infra/hosted-only (§2c).

## Work items

**Slice A (this pass):**
1. **Config + `.env.example`** — add the Slice-A knobs (below); geoip reuses `RESIDENTIAL_PROXY_LOCATION`.
2. **`app/proprietary/web_crawler/stealth.py`** (**proprietary** — bypass tuning) — `StealthConfig` + `get_stealth_config()` snapshot; coarse `RESIDENTIAL_PROXY_LOCATION → (locale, timezone_id)` map; `build_stealthy_kwargs(cfg)` returning the config-derived `StealthyFetcher` kwargs. Single source of truth imported by the connector **and** `03f`'s harness (no test-vs-prod drift). The AsyncFetcher tier **already carries `impersonate="chrome"`** (shipped `03a`, §2b) — leave it; document it as the static-tier coherence anchor.
3. **`app/utils/crawl/classifier.py`** (Apache-2) — `BlockType` enum + `classify_block(status, html) -> BlockType` (pure, status + body-marker based).
4. **Connector wiring** — merge `build_stealthy_kwargs(...)` into `_crawl_with_stealthy_sync`'s kwargs (existing `headless`/`network_idle`/`block_ads`/`solve_cloudflare`/`proxy`/captcha `page_action` preserved; lever keys never collide); add `block_type` to `CrawlOutcome`; thread a per-call `block_state` dict (classify in `_build_result` **and** the static tier's `>=400` early-return path, stamp on every `crawl_url` return) — **additive only, SUCCESS predicate unchanged**.
5. **Docker** — Xvfb + the proven font set in the worker image; drop the stale "+ Camoufox" comment.
6. **Tests + lints** — unit-test the builder (geoip map, flag wiring, snapshot) + classifier (each marker); update connector tests for `block_type`; run suite.

**Slice B / C (deferred — tracked above):** WebGL-spoof `init_script`, humanization composer, headed-Xvfb runtime, persistent profiles, strategy memory + tier-skip, paid-unblocker stub, richer `(domain, block_type, winning_strategy, attempts, latency)` instrumentation.

## Risks / trade-offs

- **Arms race / maintenance.** Fingerprint bypasses rot as WAFs update; this is exactly the cost the deferred paid tier (§8) hedges. Instrumentation (work item 9) plus **`03f`'s scorecard** are what tell us when in-house upkeep stops being worth it.
- **WebGL/GPU is a structural gap, not a bug (§2c).** The free stack cannot present a real consumer GPU; the JS spoof defeats string checks but not render-output/perf probes, and is itself brittle (rots on Chrome updates, must stay `toString`-clean). Plan for device-fingerprint-grade targets to need GPU-passthrough workers or the §8 paid tier — don't expect parity with CloakBrowser here.
- **Headed/Xvfb cost.** Headful browsers use more CPU/RAM than headless; gate per-flag and only escalate to headed when the block classifier says cheaper tiers fail for a domain (per-domain memory keeps it from being the default).
- **Profile growth.** Persistent profiles accumulate disk; add a size/TTL cap and periodic prune.
- **Geoip accuracy.** A coarse country→locale map is fine; over-fitting per-city tz isn't worth it. Wrong-but-coherent beats default-mismatched.
- **No silver bullet.** Reiterate the ceiling (§"realistic ceiling") in any user-facing copy: the crawler is "best-effort undetectable," not guaranteed.

## Prior art evaluated — why not Camoufox (decision: skip for now)

`references/Byparr-main` and `references/trawl-dev` are FlareSolverr alternatives built on **Camoufox** — an **MPL-2.0**, open-source, **C++/Juggler-patched Firefox** that natively closes our biggest gaps: hardware-accurate **WebGL** renderer/vendor spoofing (bundled `webgl_data.db`, §2c), **`geoip=True`** locale/tz/geo coherence from the proxy IP (§1), **`humanize=True`** mouse, plus font/WebRTC/audio/screen spoofing — all at the engine level (no detectable JS shims). It is effectively the CloakBrowser-class capability *without* CloakBrowser's licensing problem.

**Decision (this MVP): do not adopt Camoufox.** Rationale: Scrapling **dropped** Camoufox in favor of patchright-Chromium for its `StealthyFetcher` ([scrapling stealthy docs](https://scrapling.readthedocs.io/en/latest/fetching/stealthy.html); confirmed in source — `references/Scrapling/scrapling/engines/_browsers/_stealth.py` imports `patchright`), so wiring Camoufox would mean a **separate proprietary fetch tier** (camoufox+playwright, outside Scrapling), a Firefox engine (a few Chrome-only targets won't fit), a heavy extra binary, and tracking an actively-but-experimentally-developed project. **Revisit trigger:** if `03f`'s scorecard shows the patchright tier's device-fingerprint ceiling (§2c) is blocking real CI targets, a Camoufox tier is the prime free escalation to evaluate **before** the §8 paid unblocker.

## Out of scope (hand-offs)

- Cloudflare solving (`03a`), reCAPTCHA/hCaptcha solving + its per-solve billing (`03d`), proxy rotation (`03b`), flat crawl billing (`03c`). **Forward note for `03d` (free solver fallback):** the Camoufox-based references solve reCAPTCHA-v2 / Turnstile **free** in-browser (audio-STT — `references/trawl-dev/packages/tiers/src/solvers/{stt,turnstile}.ts`) instead of a paid solver. `03d` shipped with paid `captchatools` as primary; a free STT path is a possible *fallback-before-paid* later, but audio STT is brittle/Google-rate-limited, so paid stays primary (and this doesn't change `03e`).
- **Logged-in / account-based bypass** (sticky/static proxies + credential management) — deferred to the platform-actor work (umbrella Phase 8 + `03b` static-proxy hand-off). Public data only this MVP.
- Building the paid-unblocker provider integration — deferred (§8 leaves only the seam + flag).
- **Measuring** undetectability (the scorecard that grades these levers) → `03f` (manual harness). This plan *builds* the levers; `03f` *tests* them.
- Platform-specific structured extractors (Google Maps, LinkedIn public, …) — these sit **on top** of this hardened fetch core as Phase-8 actors; this plan only delivers the core they depend on.