SurfSense/plans/backend/03e-stealth-hardening.md

Phase 3e — Stealth hardening (the in-house "undetectable" moat)

Part of Phase 3 — WebURL Crawler & Crawl Billing. See 00-umbrella-plan.md. Depends on 03a (Scrapling StealthyFetcher tier + CrawlOutcome) and 03b (app-wide proxy provider). Precedes 03d (captcha) in the escalation order — hardening avoids challenges; captcha solving is the paid last resort for the ones we can't avoid. Touches 03c only via the deferred paid-unblocker tier (its own billing, decided if/when built).

Architecture reconciliation (read before building). Earlier drafts of this plan referenced a FetchStrategy seam / strategy-object contract (url, ctx) -> CrawlOutcome. That seam does not exist in shipped code. 03a–03d shipped a hardcoded 3-tier ladder inside WebCrawlerConnector.crawl_url (app/proprietary/web_crawler/connector.py): plain methods _crawl_with_async_fetcher / _crawl_with_dynamic / _crawl_with_stealthy(_sync) wrapped by _run_tier_with_proxy_retry, every tier returning the shared CrawlOutcome. 03e therefore implements levers not as strategy classes but as: (1) a centralized per-tier kwargs builder (app/proprietary/web_crawler/stealth.py — proprietary, since it's bypass-specific tuning; the generic block classifier stays Apache-2 in app/utils/crawl/) that turns config → StealthyFetcher/AsyncFetcher kwargs, imported by both the connector and 03f's harness so there's no test-vs-prod drift; (2) small helpers around the existing ladder (block classifier, strategy memory); and (3) the deferred paid tier as a config-gated extra ladder step, not a FetchStrategy. The captcha precedent holds: generic config/plumbing → app/utils/; actual bypass logic (WebGL spoof JS, humanize choreography) → app/proprietary/web_crawler/.

Implementation note. Same convention as 03a–03d: Phases 1–2 are SHIPPED (live code says workspace_id/Workspace — substitute for any old search_space_*/SearchSpace citations and grep the new name); citations also predate 03a's crawl_url refactor; locate code by symbol/grep, not absolute lines. Scrapling references point at the on-disk (gitignored) references/Scrapling/ checkout pinned to scrapling[fetchers]>=0.4.9 (pyproject.toml:91).

Build status. Sliced for risk (see "Build slicing" below). Slice A (SHIPPED in this pass): centralized stealth kwargs builder + fingerprint flags + geoip-from-proxy-location + block classifier (additive CrawlOutcome.block_type) + fonts/Xvfb packages in the image. Slice B / C (deferred): headed-Xvfb runtime path, humanization/WebGL init_script assets, persistent profiles, strategy memory, paid-unblocker stub. Effectiveness of every lever is graded in 03f (manual), not unit-tested here.

First baseline evidence (03f, 2026-06-30 — headless, rotating residential, captcha OFF, Slice-A only with CRAWL_GEOIP_MATCH_ENABLED=false). Suite S = 6 PASS / 4 FAIL (full table in 03f). The 4 fails map cleanly onto this plan's deferred levers and confirm the predicted ceiling — they are not Slice-A regressions:

• iphey "Unreliable" → the predicted geoip-incoherence tell (browser tz/locale vs proxy exit geo). It's now the live regression test for §1 geoip coherence — flipping CRAWL_GEOIP_MATCH_ENABLED is the expected fix to validate next.
• CreepJS hasHeadlessWorkerUA: true (headless 33%) → the Web Worker navigator.userAgent still leaks HeadlessChrome (main-thread UA is clean). A concrete, plausibly-fixable Slice-B target (worker UA override via init_script), beyond today's Slice-A flags.
• FingerprintJS Pro "tampering detected, access denied" → the documented commercial/device-fingerprint wall (§2c WebGL/GPU gap). Needs Slice B/C or the §8 paid tier; expected to stay red on the free stack.
• incolumitas → a single legacy fpscanner WEBDRIVER check (all modern checks pass); lowest priority. Its IP classifier also flip-flopped is_datacenter true/false across rotations, exposing that the anonymous_proxies pool mixes datacenter+residential exits (a proxy-provider variable, not a code issue).

Objective

Push the Universal WebURL Crawler as far toward "undetectable" as the free / in-house stack allows, so we hold a scraping moat for the next ~4–6 months without a third-party unblocker (ZenRows/ScrapFly/Bright Data) or a source-patched browser (CloakBrowser — rejected on licensing). Everything here is runtime/config-level stealth layered on Scrapling's patchright-Chromium (03a engine note): geoip fingerprint coherence, persistent profiles, headed execution, real fonts, and behavioral humanization — plus a block classifier + per-domain strategy memory so the ladder learns and the cheap tiers get skipped once a domain's working strategy is known. The paid-unblocker tier is defined here as a deferred config-gated ladder step (the explicit escape hatch for when in-house maintenance gets too costly), not built now.

The realistic ceiling (be honest with downstream devs)

Scrapling already ships strong runtime stealth by default: DEFAULT_ARGS + STEALTH_ARGS (incl. --disable-blink-features=AutomationControlled), a persistent context, and navigator.webdriver masking via patchright. Be precise about what patchright actually is: it patches the Playwright driver to remove automation leaks — it never calls Runtime.enable, disables Console.enable, and reaches closed shadow roots (verified: patchright-python, Scrappey). It hides "I'm automated," not "this is a different machine." It does not spoof canvas, WebGL, audio, screen, or GPU. With 03b residential proxies + this plan's coherence/humanization on top, the crawler reliably handles Cloudflare (via 03a solve_cloudflare) and the long tail of moderate anti-bot — which is most CI targets.

The device-fingerprint wall (the WebGL/GPU gap — see §2c). Our worker has no GPU, so Chromium's WebGL renderer reports Google SwiftShader / Mesa llvmpipe — a string anti-bot vendors treat as ">95% bot" (ipasis, Scrappey). Scrapling exposes only allow_webgl (on/off — and off is itself a tell), no renderer spoof; CloakBrowser fixes this at the C++ level (--fingerprint-gpu-renderer) and we cannot. Concretely, without a real GPU or a source-patched binary we cannot:

1. Pass pure device-fingerprint bot checks — FingerprintJS bot-detection, CreepJS / BrowserScan "is this a real device." CloakBrowser passes these (BrowserScan 4/4); our SwiftShader renderer fails them.
2. Reach human-level reCAPTCHA v3 scores (~0.7–0.9) on our own browser — we land ~0.1–0.5. (Note: 03d's paid solver sidesteps this for token challenges — the token is minted on the vendor's real-browser infra, so its score isn't ours. The gap only bites where the site itself reads our live v3 score.)
3. Max out the device-fingerprint component of DataDome / Kasada / Akamai — one of several reasons we plateau below managed APIs.

Numbers (2026 benchmarks — use these in any user-facing copy). Free stack + residential: Basic (CF) ~99%; Medium (CF Pro / PerimeterX) ~90%; Enterprise (DataDome / Kasada / Akamai) ~60–75% initially, decaying toward ~60% within 24h as their ML clusters the fingerprint (managed APIs hold ~89%) (scrapewise DataDome 2026). So the honest line is "partial and decaying, not SLA-grade," — neither "can't beat" nor "beats everything." For CI targets the top-tier defenses are the exception, so the in-house moat is a sound 4–6 month bet, with the deferred paid tier (§8) as the evidence-driven escape hatch (03f's scorecard is the trigger). Do not promise "beats everything."

Levers (wired via the centralized kwargs builder + ladder helpers — callers stay outcome-only)

1. Geoip fingerprint coherence (match the browser to the proxy exit IP)

A residential IP in Berlin behind an en-US/America/New_York browser is an instant tell. Make the fingerprint cohere with the proxy's exit geo:

• Primary source (Slice A, no network call): the configured RESIDENTIAL_PROXY_LOCATION (03b, config:1048). Map that string (best-effort: ISO-3166 alpha-2 like us/de, or a common country name) → a representative (locale, timezone_id). Coarse country granularity is fine (per "Risks": wrong-but-coherent beats default-mismatched); unknown/empty → skip (leave Scrapling's system default). No exit-IP resolution — avoids a per-crawl network round-trip + failure mode. (A geoip2/MaxmindLite exit-IP path is a later refinement, not MVP.)
• Pass the matched values into StealthyFetcher via the kwargs builder: locale= (drives navigator.language + Accept-Language) and timezone_id= (confirmed real kwargs on StealthyFetcher.fetch). These flow into the Playwright context.
• The stealthy tier already captures the chosen proxy endpoint once (_crawl_with_stealthy_sync: proxy = get_proxy_url(), the same value handed to 03d's solver for IP-coherence) — reuse that captured proxy/location, don't re-call get_proxy_url() (which rotates on a pool-backed provider, 03b).
• Caveat (be honest): Scrapling applies locale/timezone_id via Playwright CDP emulation, which advanced systems can flag as emulation — CloakBrowser sets these as binary flags precisely to avoid "detectable CDP emulation" (references/CloakBrowser/README.md:309). Still strictly better than a mismatched default (wrong-but-coherent beats UTC+en-US behind a Berlin IP), but it's a weaker variant than a source-patched browser, not parity.

2. Fingerprint flags Scrapling already exposes (turn them on)

• hide_canvas — Scrapling adds random noise to canvas ops (verified in the installed StealthyFetcher.fetch docstring). Not "free safe": random-per-call noise is itself a signal on FingerprintJS/CreepJS-class detectors — an unstable canvas hash is as loud as navigator.webdriver=true (dev.to); CloakBrowser's own FPJS fix is --fingerprint-noise=false. Default OFF; gate per-domain and validate in 03f. Harmless against Cloudflare/moderate, potentially harmful against device-fingerprint graders. (Canvas stealth that actually helps must be deterministic per persistent profile, not random — out of scope here.)
• block_webrtc=True — forces WebRTC to respect the proxy, preventing the real local IP leak that unmasks proxied browsers (:41). This one is genuinely cheap + safe — keep default TRUE.
• dns_over_https — Scrapling can route DNS via Cloudflare DoH to stop the DNS leak that unmasks proxied browsers (same anti-leak class as block_webrtc; confirmed kwarg on StealthyFetcher.fetch). Trade-off: it adds a DNS round-trip, a (small) latency cost — so per the "no speed regression" constraint it ships off by default (CRAWL_DNS_OVER_HTTPS, default FALSE), pre-wired for operators who prefer leak-safety over the marginal latency. (Idea sourced from the Camoufox-based FlareSolverr alternatives references/Byparr-main / references/trawl-dev, which treat DNS/WebRTC leak coherence as first-class.)
• google_search=True (default) — sets a Google referer so the first hit looks like organic arrival (:45); override per-need via extra_headers (:46).
• additional_args= — last-priority Playwright context overrides for anything not surfaced as a first-class param (:51).

2b. HTTP-tier TLS fingerprint (the AsyncFetcher tier — impersonate)

The cheap static tier (03a tier 1) is the first thing every crawl hits. 03a already ships impersonate="chrome" on it (app/proprietary/web_crawler/connector.py, the AsyncFetcher.get call) — Scrapling's static engine selects a matching curl_cffi browser profile (references/Scrapling/scrapling/engines/static.py:36–47), so the HTTP tier's JA3/JA4/HTTP-2 already matches a real Chrome and coheres with the browser tiers' UA. 03e's remaining work here is not to add it, but to (a) fold the impersonate profile into the centralized per-tier kwargs builder (below) so it stays the single source of truth, and (b) keep the chosen profile coherent with the proxy exit / UA. 03f §S3 validates the shipped parity against tls.peet.ws.

2c. WebGL / GPU renderer (the biggest free-stack gap — partial mitigation only)

This is the "device-fingerprint wall" from the ceiling section. On a GPU-less worker the WebGL renderer string is Google SwiftShader / ANGLE (Mesa, llvmpipe…) — an instant tell on any detector that reads WEBGL_debug_renderer_info. Scrapling gives us no renderer knob (only allow_webgl on/off, and off is a tell). Three options, increasing strength + cost:

1. DIY JS spoof (optional, OFF by default — Slice B). An init_script overrides getParameter for UNMASKED_VENDOR_WEBGL / UNMASKED_RENDERER_WEBGL to a believable, platform-coherent value (e.g. an Intel/ANGLE renderer that matches the spoofed Windows UA), held consistent per persistent profile (§3). Impl note: Scrapling's init_script is an absolute path to a .js file, not an inline string (confirmed in StealthyFetcher.fetch docstring) — so this ships as a bundled .js asset in app/proprietary/web_crawler/ (next to 03d's captcha logic) passed by absolute path; the kwargs builder resolves the path when CRAWL_WEBGL_SPOOF_ENABLED. Must also patch Function.prototype.toString so the override still returns [native code] — otherwise the spoof itself is the signal (exactly the toString() probe Kasada catalogs, web-scraping-guide). This defeats string-based WebGL checks (the common case) but not render-output pixel-hash or GPU-perf probes (FingerprintJS-grade) — a software rasterizer's actual draw output still differs and can't be faked in JS. It is the brittle config-level approach patchright/CloakBrowser explicitly warn rots; treat it as a cheap, test-gated win, not the moat.
2. Real GPU passthrough (hosted workers only). --device /dev/dri + ANGLE/EGL so WebGL reports a genuine renderer with real render output, defeating even output/perf probes. Best technical fix; costs GPU instances + image complexity (livekit/egress GPU notes). A hosted-tier escalation, off for self-hosted.
3. Defer to the paid unblocker (§8). Route the residual device-fingerprint-gated domains (flagged by the §7 classifier as in-house-unreachable) to the external unblocker. The pragmatic moat-bounding fallback.

Decision for this MVP: ship option 1 as an opt-in flag (CRAWL_WEBGL_SPOOF_ENABLED, default FALSE), name option 2 as the hosted escalation, and lean on §8 for the rest. Quantify all three in 03f.

3. Persistent per-domain profiles (look like a returning human)

Scrapling defaults to a temporary user-data dir (fresh = suspicious). Use user_data_dir= (confirmed real kwarg; launch_persistent_context) to keep a persistent profile per domain (or per domain+proxy-geo), so cookies/localStorage/site-trust carry across crawls and the browser presents as a returning visitor rather than a brand-new incognito session. Store profiles under a configured dir (shared_tmp-style volume so API + worker share them).

• Concurrency hazard (Slice C design item — why this is deferred, not a one-liner): Chromium hard-locks a user_data_dir; two crawls touching the same profile dir at once crash (SingletonLock). Our crawler runs concurrently (multiple connectors/chats across workers), so a naive per-domain dir collides. The fix needs one of: (a) an async lock keyed by profile dir so same-domain crawls serialize (adds latency, can deadlock under load), (b) per-(domain, worker, pid) dirs + a periodic prune (loses some cross-run trust, no contention), or (c) persistent only when the strategy memory (§7) says a domain needs it, temp otherwise. Decide in Slice C alongside §7; Slice A keeps Scrapling's temp default.

4. Headed execution under Xvfb (defeat headless tells)

headless defaults to hidden (stealth_chrome.py:19/71). Many WAFs flag headless Chromium. Run headful (headless=False) inside a virtual framebuffer (Xvfb) in the Docker worker so the browser is "visible" to itself but needs no real display. Gate behind a config flag (off by default for self-hosted, on for hosted workers that have Xvfb installed).

5. Real fonts (canvas/emoji hash realism)

A minimal container has almost no fonts, making canvas/emoji fingerprint hashes obviously synthetic. Install real font packages so font-enumeration + canvas/emoji hashes resemble a real desktop. Use the set CloakBrowser ships as proven against Kasada/Akamai emoji-canvas hashing (README.md:737–739): fonts-noto-color-emoji, fonts-unifont, fonts-ipafont-gothic (CJK), fonts-wqy-zenhei (CJK), fonts-tlwg-loma-otf — plus fonts-dejavu/fonts-liberation for Latin coverage. Dockerfile change only (apt-get install fonts-*).

6. Behavioral humanization (DIY — Scrapling has no humanize for the Chromium engine)

Unlike Camoufox, the patchright-Chromium StealthyFetcher exposes no built-in humanize (verified: zero matches in the StealthyFetcher param set). So humanization is custom, injected via the existing hooks:

• page_action= (runs after navigation) — randomized mouse moves/curves, scrolls, hover-before-click, and small think-time delays before extraction. This is the same hook 03d uses for token injection — _crawl_with_stealthy_sync currently builds exactly one page_action (the captcha injector), so Slice B must compose two callables (humanize → optional captcha solve) into the single page_action kwarg, not overwrite it.
• init_script= (an absolute path to a .js file executed on page creation — not inline JS, see §2c) — early shims for any residual JS tells not covered by patchright (and the §2c WebGL spoof, if enabled).
• Tunable wait/network_idle (:29,27) so dwell time isn't robotically constant.
• Concrete rules (from CloakBrowser's reCAPTCHA notes, README.md:1280–1314 — encode these): use native time.sleep, never page.wait_for_timeout() (it emits CDP traffic detectors flag); prefer page.type() over fill() for any input (real keystrokes vs. value-set); dwell 15s+ before triggering score-based checks; space repeat hits to score-gated domains 30s+; keep a stable identity per domain (persistent profile §3 + consistent spoofed values §2c) so a score-based system sees a returning device, not a fresh one each visit.

7. Block classifier + per-domain strategy memory (the "learning ladder")

Two small in-house pieces make the ladder smart instead of brute-force:

• Block classifier (Slice A — additive, pure, unit-testable). Inspect the fetched page (status + body markers; cookies/headers are not threaded through _build_result today, so MVP is status + body-marker based) and label the outcome: OK / CLOUDFLARE / CAPTCHA_RECAPTCHA / CAPTCHA_HCAPTCHA / DATADOME / KASADA / RATE_LIMITED / EMPTY / UNKNOWN. Concrete markers — adopt the set proven in references/trawl-dev/packages/tiers/src/detect.ts (a near-identical classifier): Cloudflare/DDoS-Guard → cf-mitigated-style title strings "just a moment" / "checking your browser" / "enable javascript and cookies to continue" / "verify you are human", DOM ids challenge-running / cf-challenge-running / turnstile-wrapper, and ddos-guard.net; Turnstile → class="cf-turnstile" / challenges.cloudflare.com/turnstile; hCaptcha → class="h-captcha" / hcaptcha.com/1/api; reCAPTCHA → class="g-recaptcha" / google.com/recaptcha / recaptcha.net; DataDome → datadome script/cookie; status 202/403/429 as a bot-gate (202 is an IMDb-style pre-gate, per detect.ts:isBlocked). Invariant: the classifier is purely additive — it attaches CrawlOutcome.block_type and feeds telemetry/routing; it MUST NOT change when SUCCESS is returned, because 03c bills on status == SUCCESS (CrawlOutcome docstring). Wiring mirrors 03d's proven captcha_state pattern: a per-call block_state dict mutated in _build_result (which has raw_html + status) and stamped onto every CrawlOutcome return path in crawl_url.
• Per-domain strategy memory (Slice C — needs a new dependency + control-flow change). Cache the strategy that last succeeded per domain (Redis key crawl:strategy:{domain} with TTL). Next crawl starts at the known-good tier, skipping cheaper tiers that always fail there. Why deferred: WebCrawlerConnector() is currently constructed with no args and is Redis/DB-free; this requires injecting a cache client (best-effort, no-op when absent) and letting crawl_url skip ladder tiers — a control-flow change to the tuned ladder. No DB migration (Redis, best-effort, self-healing on miss), but it is a deliberate design item, not a flag flip.
• Solved-session cache (Slice C refinement — speed-positive, the strongest version of the memory above). Beyond remembering which tier won, cache the solved session itself — the cf_clearance/__cf_bm cookies + the UA that cleared the challenge — in Redis (crawl:session:{domain} with TTL), and replay it on the next crawl via Scrapling's cookies= input, so a returning hit skips the expensive solve entirely (the pattern references/trawl-dev Tier 2 uses for its ~500ms repeat path: browser/src/session.ts session:{domain} save/load/invalidate; tiers/src/tier2.ts replays cookies, and on a re-challenge marks session-expired → invalidate → escalate). Net effect is faster, not slower (it removes a solve), so it fits the "no speed regression" bar. Caveat: cf_clearance is IP+UA-bound, so it pays off most behind sticky/static proxies (the 03b static-proxy future) and the cache key should include proxy-geo; under rotating residential the replayed cookie may not match the new exit IP (best-effort, self-healing via the session-expired invalidate).

8. Deferred: paid-unblocker tier (the escape hatch, NOT built now)

The moat strategy is explicit: maintain in-house bypass for ~4–6 months, then move hostile targets to a paid unblocker if demand/maintenance justifies it. That switch is evidence-driven, not a guess — 03f's manual scorecard quantifies the free-stack ceiling over time and is the documented trigger for flipping this tier. Pre-wire the seam so the switch is a config flip, not a refactor:

• Define (but do not implement) a paid-unblocker tier — a _crawl_with_paid_unblocker(url) -> dict | None method returning the same shape as the existing tiers, appended last in crawl_url's ladder (after the stealthy tier) and active only when an env flag + API key are set. (Same plain-method shape as today's tiers — there is no FetchStrategy contract to satisfy.)
• It would call an external unblocker (ZenRows/ScrapFly/Bright Data Web Unlocker) for the residual DATADOME/KASADA/reCAPTCHA-Enterprise domains the block classifier flags as unreachable in-house.
• Its own billing (cost-plus pass-through, decided at build time) — separate from 03c's flat crawl unit and 03d's per-solve unit, because unblocker pricing is per-request and provider-specific.
• Until built, those domains simply return non-SUCCESS (free under 03c). This keeps the umbrella's "WebURL Crawler is the moat" honest while bounding our maintenance risk.

Config / env changes

Add (all default OFF / conservative; next to the 03b/03c knobs in config/__init__.py + .env.example):

• CRAWL_GEOIP_MATCH_ENABLED (default FALSE; Slice A — maps RESIDENTIAL_PROXY_LOCATION → locale/timezone_id, no exit-IP lookup).
• CRAWL_BLOCK_WEBRTC (default TRUE — cheap, safe; Slice A), CRAWL_HIDE_CANVAS (default FALSE — random canvas noise can itself be a tell; opt-in + 03f-validated, see §2; Slice A), and CRAWL_GOOGLE_SEARCH_REFERER (default TRUE — Scrapling's google_search; Slice A).
• CRAWL_DNS_OVER_HTTPS (default FALSE — anti DNS-leak, but adds a DNS round-trip; default-off to honor the "no speed regression" bar, pre-wired for leak-safety-first operators; §2; Slice A).
• CRAWL_WEBGL_SPOOF_ENABLED (default FALSE) + optional CRAWL_WEBGL_VENDOR / CRAWL_WEBGL_RENDERER override strings (§2c; toString-safe JS spoof, defeats string-checks only).
• CRAWL_PERSISTENT_PROFILES_DIR (unset → Scrapling's temp default; set → per-domain profiles).
• CRAWL_HEADED_XVFB_ENABLED (default FALSE; requires Xvfb in the image).
• CRAWL_HUMANIZE_ENABLED (default TRUE) + dwell/jitter bounds.
• CRAWL_STRATEGY_MEMORY_TTL_S (default e.g. 86400; 0 → disabled).
• CRAWL_PAID_UNBLOCKER_ENABLED (default FALSE) + provider/key (deferred tier).

Docker changes (surfsense_backend/Dockerfile)

• Install Xvfb + the proven font set (fonts-noto-color-emoji, fonts-unifont, fonts-ipafont-gothic, fonts-wqy-zenhei, fonts-tlwg-loma-otf, fonts-dejavu, fonts-liberation; §5) in the worker image. (Note from 03a: also drop the stale "+ Camoufox" comment near :112; scrapling install only fetches Chromium.)
• Headed runs need the browser launched under xvfb-run (or an Xvfb display in the worker entrypoint), gated by CRAWL_HEADED_XVFB_ENABLED.

Build slicing (risk-staged; 03f validates effectiveness)

The levers split by risk/testability. Slice A is built now (prod-safe, config-gated, defaults preserve today's behavior, wiring + classifier unit-tested). B/C are deferred (need infra or architectural design; mostly only verifiable in 03f).

• Slice A (SHIPPED): centralized kwargs builder → fingerprint flags (block_webrtc ON, hide_canvas OFF, google_search ON, dns_over_https OFF); geoip coherence from RESIDENTIAL_PROXY_LOCATION; fonts + Xvfb packages in the Dockerfile; additive block classifier (CrawlOutcome.block_type, classified in _build_result and the static tier's >=400 early-return so the first/cheapest-tier bot-gate isn't lost). Licensing split: the stealth kwargs builder is proprietary (app/proprietary/web_crawler/stealth.py — bypass tuning); the block classifier stays Apache-2 (app/utils/crawl/classifier.py — generic passive telemetry from public markers). extra_headers/additional_args were intentionally NOT wired — they're last-priority override hatches with no clean env representation (dict-in-env) and no concrete need yet; add them when a real use appears rather than speculative config (YAGNI).
• Slice B (next): headed-Xvfb runtime path (headless=False + entrypoint), humanization page_action (composed with 03d's injector), WebGL-spoof init_script .js asset (proprietary).
• Slice C (defer): per-domain persistent profiles (concurrency design, §3), per-domain strategy memory + tier-skipping (crawl_url control-flow change + cache injection, §7), solved-session cache (cf_clearance replay via cookies=, §7 — speed-positive, best with sticky proxies), warm browser pool (a hosted-scale latency win — references/trawl-dev browser/src/pool.ts keeps N warm instances with an acquire-timeout→429; Scrapling's StealthySession already has page pooling), paid-unblocker config stub (§8). GPU passthrough stays infra/hosted-only (§2c).

Work items

Slice A (this pass):

1. Config + .env.example — add the Slice-A knobs (below); geoip reuses RESIDENTIAL_PROXY_LOCATION.
2. app/proprietary/web_crawler/stealth.py (proprietary — bypass tuning) — StealthConfig + get_stealth_config() snapshot; coarse RESIDENTIAL_PROXY_LOCATION → (locale, timezone_id) map; build_stealthy_kwargs(cfg) returning the config-derived StealthyFetcher kwargs. Single source of truth imported by the connector and 03f's harness (no test-vs-prod drift). The AsyncFetcher tier already carries impersonate="chrome" (shipped 03a, §2b) — leave it; document it as the static-tier coherence anchor.
3. app/utils/crawl/classifier.py (Apache-2) — BlockType enum + classify_block(status, html) -> BlockType (pure, status + body-marker based).
4. Connector wiring — merge build_stealthy_kwargs(...) into _crawl_with_stealthy_sync's kwargs (existing headless/network_idle/block_ads/solve_cloudflare/proxy/captcha page_action preserved; lever keys never collide); add block_type to CrawlOutcome; thread a per-call block_state dict (classify in _build_result and the static tier's >=400 early-return path, stamp on every crawl_url return) — additive only, SUCCESS predicate unchanged.
5. Docker — Xvfb + the proven font set in the worker image; drop the stale "+ Camoufox" comment.
6. Tests + lints — unit-test the builder (geoip map, flag wiring, snapshot) + classifier (each marker); update connector tests for block_type; run suite.

Slice B / C (deferred — tracked above): WebGL-spoof init_script, humanization composer, headed-Xvfb runtime, persistent profiles, strategy memory + tier-skip, paid-unblocker stub, richer (domain, block_type, winning_strategy, attempts, latency) instrumentation.

Risks / trade-offs

• Arms race / maintenance. Fingerprint bypasses rot as WAFs update; this is exactly the cost the deferred paid tier (§8) hedges. Instrumentation (work item 9) plus 03f's scorecard are what tell us when in-house upkeep stops being worth it.
• WebGL/GPU is a structural gap, not a bug (§2c). The free stack cannot present a real consumer GPU; the JS spoof defeats string checks but not render-output/perf probes, and is itself brittle (rots on Chrome updates, must stay toString-clean). Plan for device-fingerprint-grade targets to need GPU-passthrough workers or the §8 paid tier — don't expect parity with CloakBrowser here.
• Headed/Xvfb cost. Headful browsers use more CPU/RAM than headless; gate per-flag and only escalate to headed when the block classifier says cheaper tiers fail for a domain (per-domain memory keeps it from being the default).
• Profile growth. Persistent profiles accumulate disk; add a size/TTL cap and periodic prune.
• Geoip accuracy. A coarse country→locale map is fine; over-fitting per-city tz isn't worth it. Wrong-but-coherent beats default-mismatched.
• No silver bullet. Reiterate the ceiling (§"realistic ceiling") in any user-facing copy: the crawler is "best-effort undetectable," not guaranteed.

Prior art evaluated — why not Camoufox (decision: skip for now)

references/Byparr-main and references/trawl-dev are FlareSolverr alternatives built on Camoufox — an MPL-2.0, open-source, C++/Juggler-patched Firefox that natively closes our biggest gaps: hardware-accurate WebGL renderer/vendor spoofing (bundled webgl_data.db, §2c), geoip=True locale/tz/geo coherence from the proxy IP (§1), humanize=True mouse, plus font/WebRTC/audio/screen spoofing — all at the engine level (no detectable JS shims). It is effectively the CloakBrowser-class capability without CloakBrowser's licensing problem.

Decision (this MVP): do not adopt Camoufox. Rationale: Scrapling dropped Camoufox in favor of patchright-Chromium for its StealthyFetcher (scrapling stealthy docs; confirmed in source — references/Scrapling/scrapling/engines/_browsers/_stealth.py imports patchright), so wiring Camoufox would mean a separate proprietary fetch tier (camoufox+playwright, outside Scrapling), a Firefox engine (a few Chrome-only targets won't fit), a heavy extra binary, and tracking an actively-but-experimentally-developed project. Revisit trigger: if 03f's scorecard shows the patchright tier's device-fingerprint ceiling (§2c) is blocking real CI targets, a Camoufox tier is the prime free escalation to evaluate before the §8 paid unblocker.

Out of scope (hand-offs)

• Cloudflare solving (03a), reCAPTCHA/hCaptcha solving + its per-solve billing (03d), proxy rotation (03b), flat crawl billing (03c). Forward note for 03d (free solver fallback): the Camoufox-based references solve reCAPTCHA-v2 / Turnstile free in-browser (audio-STT — references/trawl-dev/packages/tiers/src/solvers/{stt,turnstile}.ts) instead of a paid solver. 03d shipped with paid captchatools as primary; a free STT path is a possible fallback-before-paid later, but audio STT is brittle/Google-rate-limited, so paid stays primary (and this doesn't change 03e).
• Logged-in / account-based bypass (sticky/static proxies + credential management) — deferred to the platform-actor work (umbrella Phase 8 + 03b static-proxy hand-off). Public data only this MVP.
• Building the paid-unblocker provider integration — deferred (§8 leaves only the seam + flag).
• Measuring undetectability (the scorecard that grades these levers) → 03f (manual harness). This plan builds the levers; 03f tests them.
• Platform-specific structured extractors (Google Maps, LinkedIn public, …) — these sit on top of this hardened fetch core as Phase-8 actors; this plan only delivers the core they depend on.