diff --git a/plans/backend/00-umbrella-plan.md b/plans/backend/00-umbrella-plan.md index 1cc1bb8b8..e6dd99889 100644 --- a/plans/backend/00-umbrella-plan.md +++ b/plans/backend/00-umbrella-plan.md @@ -84,7 +84,7 @@ The Universal WebURL Crawler is the flagship Type-1 data source (**the moat**). - **`03c-crawl-billing.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `17bdb0682`)* — Charge crawl credits at **$1 / 1000 successful requests = 1000 micro-USD per successful crawl** (config-driven via `WEB_CRAWL_MICROS_PER_SUCCESS`, retunable with no code change), drawn from the existing credit wallet (`credit_micros_balance`), gated by a new `WEB_CRAWL_CREDIT_BILLING_ENABLED` flag (off for self-hosted). Two surfaces: **connector/pipeline crawls** billed to the **workspace owner** via a dedicated `WebCrawlCreditService` (mirrors `EtlCreditService`'s gate → `check_credits` → `charge_credits`, **not** `billable_call`); **chat scrapes** fold their crawl cost into the chat turn's existing bill (turn accumulator). No DB migration (uses the existing free-form `web_crawl` usage_type). - **`03e-stealth-hardening.md`** — The in-house "undetectable" layer on top of Scrapling's default patchright-Chromium stealth. **Geoip coherence** (match browser `locale`/`timezone_id` to the proxy's exit geo), **fingerprint flags** (`hide_canvas`/`block_webrtc`), **persistent per-domain profiles** (`user_data_dir`), **headed execution under Xvfb**, **real fonts** in the worker image, and **DIY behavioral humanization** via `page_action` (the Chromium engine has no built-in `humanize`). Adds a **block classifier** (label Cloudflare/DataDome/Kasada/captcha/empty from the response) + **per-domain strategy memory** (Redis, no migration) so the ladder learns the known-good tier per domain. Defines — but does **not** build — the **deferred paid-unblocker `FetchStrategy`** (ZenRows/ScrapFly/Bright Data) as the config-flagged escape hatch, with its own (later) billing. Honest ceiling: defeats Cloudflare + the moderate long tail, **not** top-tier behavioral fingerprinting (DataDome/Kasada/reCAPTCHA-Enterprise) — that's the deferred tier. - **`03d-captcha-solving.md`** *(✅ IMPLEMENTED — `ci_mvp`; `captchatools` page_action + per-attempt `web_crawl_captcha` billing, off by default)* — Covers the captcha types Scrapling does **not** (reCAPTCHA v2/v3, hCaptcha, image) via `captchatools`, **opt-in + off by default**. `captchatools` is **itself** the provider registry (`new_harvester(solving_site=…)` across capmonster/2captcha/anticaptcha/capsolver/captchaai), so we do **not** rebuild a provider hierarchy — our layer is thin: config resolution + a StealthyFetcher `page_action` that detects the sitekey, harvests a token (egressing from the **same** proxy IP as the crawl), and injects it. Scrapling already handles Cloudflare Turnstile (`03a`), and `page_action` runs **after** `solve_cloudflare`, so the tiers compose. The **billing asymmetry** is now **resolved**: a **separate per-attempt** `web_crawl_captcha` unit (solvers charge per attempt regardless of crawl success), attached via a `WebCrawlCreditService` seam in `03c`. `ErrNoBalance` stops solving (no retry loop → avoids IP bans). Requires a paid solver account. -- **`03f-undetectability-testing.md`** *(MANUAL-only — no CI gating; sequenced last)* — A **manual scorecard harness** that drives the real Scrapling tiers against the industry-standard detection + sandbox sites (modeled on CloakBrowser's `bin/cloaktest` suite) to **quantify the free-stack ceiling** over time. Two labeled axes: **Suite S (stealth/anti-bot)** — browser-tier (`bot.sannysoft`, `bot.incolumitas`, CreepJS, `deviceandbrowserinfo`, FingerprintJS demo, reCAPTCHA-v3 score, `fingerprint-scan`/Castle.js, `browserscan`), HTTP/TLS-tier JA3/JA4 parity (`tls.peet.ws`, **informational** not a gate), and proxy/leak checks (`httpbin/ip`, WebRTC/DNS); **Suite E (extraction correctness)** — toscrape/scrapethissite sandboxes for the HTTP vs JS (DynamicFetcher) tiers. Reuses `03d`'s `page_action`+closure-cell for JS-object verdicts. Adopts CloakBrowser's bars as **aspirational** (sannysoft 0 fails, CreepJS ≤30%, reCAPTCHA ≥0.7) while recording **our actual numbers as the committed baseline**; the scorecard is the documented **trigger** for flipping `03e`'s deferred paid-unblocker tier. +- **`03f-undetectability-testing.md`** *(✅ IMPLEMENTED + first baseline — `ci_mvp`, MANUAL-only, no CI gating; lives under `app/proprietary/web_crawler/testbench/`)* — A **manual scorecard harness** that drives the real Scrapling tiers against the industry-standard detection + sandbox sites (modeled on CloakBrowser's `bin/cloaktest` suite) to **quantify the free-stack ceiling** over time. Two labeled axes: **Suite S (stealth/anti-bot)** — browser-tier (`bot.sannysoft`, `bot.incolumitas`, CreepJS, `deviceandbrowserinfo`, FingerprintJS demo, reCAPTCHA-v3 score, `fingerprint-scan`/Castle.js, `browserscan`, **`cloudflare-challenge`** exercising `solve_cloudflare`, **`iphey`** geoip-coherence), HTTP/TLS-tier JA3/JA4 parity (`tls.peet.ws`, **informational** not a gate), and proxy/leak checks (`httpbin/ip`, WebRTC/DNS); **Suite E (extraction correctness)** — toscrape/scrapethissite sandboxes for the HTTP vs JS (DynamicFetcher) tiers. **Every detection site is auto-graded from its real DOM verdict** (parsers written against captured dumps; `INFO` reserved for TLS/IP/manual rows). Reuses `03d`'s `page_action`+closure-cell mechanism. Adopts CloakBrowser's bars as **aspirational** (sannysoft 0 fails, CreepJS ≤30%, reCAPTCHA ≥0.7) while recording **our actual numbers as the baseline** (the whole `results/` tree is **gitignored**, run-local). **First baseline (2026-06-30, headless, rotating residential, captcha OFF): Suite S 6 PASS / 4 FAIL** — PASS sannysoft, deviceandbrowserinfo, reCAPTCHA-v3 (0.9), BrowserScan, fingerprint-scan (35/100), cloudflare-challenge; FAIL CreepJS (`hasHeadlessWorkerUA` worker-UA leak → `03e` Slice-B candidate), FingerprintJS Pro (commercial ceiling), iphey ("Unreliable" → expected geoip-coherence fix), incolumitas (legacy `fpscanner WEBDRIVER` only). The scorecard is the documented **trigger** for flipping `03e`'s deferred paid-unblocker tier. ### Phase 4 — Connector two-type restructure (backend) [`subplans: 04a–04b`] @@ -165,6 +165,7 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta - Authenticated/logged-in scraping: **OUT OF SCOPE this MVP** (public data only). Sticky/static proxies + credential management are deferred and paired with the future platform actors (`03b` static-proxy hand-off + Phase 8). - Phase 3 stealth-hardening subplan `03e` ADDED: geoip locale/tz coherence, `hide_canvas`/`block_webrtc`, persistent per-domain profiles, headed+Xvfb, fonts, DIY humanization (`page_action`; Chromium engine has no built-in `humanize`), a block classifier, and per-domain strategy memory (Redis, no migration). - Phase 3 test-harness subplan `03f` ADDED: **manual-only** (no CI/automated gating now) undetectability + extraction scorecard, modeled on CloakBrowser's `bin/cloaktest`. Two labeled axes (Suite S stealth + Suite E extraction) so they scale independently. Drives the **real** Scrapling tiers (browser + curl_cffi HTTP/TLS), reuses `03d`'s `page_action`+closure-cell for JS-object verdicts. **TLS JA3/JA4 parity = informational axis, not a hard gate.** Adopt CloakBrowser bars as aspirational; record our actual free-stack numbers as the committed baseline. The scorecard is the documented evidence/trigger for flipping `03e`'s deferred paid-unblocker tier. +- Licensing placement of 03e/03f code (decided during 03f impl, applying the §boundary test): the **stealth kwargs builder + geoip coherence** (`03e` bypass tuning) live **proprietary** at `app/proprietary/web_crawler/stealth.py`; the **block classifier** (passive telemetry, public markers) stays **Apache-2** at `app/utils/crawl/classifier.py` (direct analog of the captcha split: proprietary `captcha.py` logic + Apache-2 `app/utils/captcha/` config). The **03f scorecard harness** moved whole to `app/proprietary/web_crawler/testbench/` (run `python -m app.proprietary.web_crawler.testbench`) — it's the moat's measurement tool and can't be cleanly half-moved (a proprietary Suite S would back-import generic scaffolding from `scripts/`, a forbidden app→scripts direction). The `scripts/e2e_phase3_crawl_billing.py` billing e2e stays in `scripts/` (Apache-2) since it exercises billing, not the stealth moat. - Roadmap: WebURL Crawler & Crawl Billing inserted as the new Phase 3; connector two-type → Phase 4; pipelines → Phases 5/6/7. - Phase 5 pipelines data model: two new tables `pipelines` (mutable) + `pipeline_runs` (append-only), modeled on `automations`/`automation_runs`; ORM lives in `db.py` next to connectors/folders. `connector_id` nullable (NULL = Phase-7 Uploads), eligibility enforced at create via 04a's `is_pipeline_eligible`. Schedule = `schedule_cron` + `schedule_timezone` (default UTC) + `next_scheduled_at` (cron, matching automations). `pipeline_runs` pre-includes `charged_micros`/`crawls_*`/`result_blob_key` so Phase 6 needs no extra migration. Both tables published to Zero **full-row** (like folders/connectors). Routes reuse `CONNECTORS_*` permissions. Phase 5 ships the data model + API surface only; the `/run` endpoint enqueues a Phase-6 task stub. - Phase 7 uploads-as-pipeline: a **singleton "Uploads" pipeline** per workspace (`connector_id NULL`, `save_to_kb=true`), lazily get-or-created (race-safe via a partial unique index `ON pipelines(workspace_id) WHERE connector_id IS NULL`). Each `fileupload`/`folder-upload` request writes a **terminal audit `PipelineRun(trigger=upload, status=succeeded, documents_indexed=)`** — uploads are **route-recorded, not engine-executed** (Phase 6 fails NULL-connector runs by design; existing upload code stays the executor). Best-effort via an **inner** try/except (never 5xx the upload — the route's outer handler would otherwise 500 an already-committed upload). No crawl billing (uploads aren't crawls; `charged_micros` NULL). Per-file ETL truth stays on `Document.status`; accurate roll-up needs the deferred `documents.pipeline_run_id` provenance. Connector `save_to_kb` default stays `False` (opt-in for connectors, mandatory for uploads). Phase 7 also **guards Phase-5's generic CRUD** against the system Uploads pipeline: `POST /pipelines` rejects `connector_id=None` (supersedes Phase 5's permissive create), `/run` and schedule-`PUT` reject NULL-connector pipelines, and Phase 6's scheduler `_claim_due` filters `connector_id IS NOT NULL` as a backstop (so the Uploads pipeline can never be manually-run or scheduled into perpetually-failing runs). See `07-upload-pipeline-kb.md`. @@ -179,9 +180,9 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta | 3 | `03a-crawler-core.md` | **IMPLEMENTED** (`ci_mvp` @ `5c36cd3`) — Firecrawl removed, Scrapling-only 3-tier `CrawlOutcome`, crawler relocated to `app/proprietary/web_crawler/` | | 3 | `03b-proxy-expansion.md` | **IMPLEMENTED** (`ci_mvp` @ `6226012`) — `CustomProxyProvider` (BYO single/pool) + registry + bounded rotation-retry | | 3 | `03c-crawl-billing.md` | **IMPLEMENTED** (`ci_mvp` @ `17bdb0682`) — `WebCrawlCreditService` (config-driven price) + indexer wiring + chat-turn fold; functional e2e green | -| 3 | `03e-stealth-hardening.md` | drafted | +| 3 | `03e-stealth-hardening.md` | **Slice A IMPLEMENTED** (`ci_mvp`) — stealth kwargs builder + geoip coherence (**proprietary** `app/proprietary/web_crawler/stealth.py`) + additive block classifier (Apache-2 `app/utils/crawl/`; `CrawlOutcome.block_type`, incl. static-tier 4xx) + Xvfb/fonts in image; Slices B/C deferred (WebGL spoof, humanization, persistent profiles, strategy memory, paid-unblocker) | | 3 | `03d-captcha-solving.md` | **IMPLEMENTED** (`ci_mvp`) — `captchatools` page_action (proprietary) + Apache-2 config + per-attempt `web_crawl_captcha` billing; off by default | -| 3 | `03f-undetectability-testing.md` | drafted (manual scorecard — sequenced last) | +| 3 | `03f-undetectability-testing.md` | **IMPLEMENTED** (`ci_mvp`) — manual scorecard under the **proprietary boundary** at `app/proprietary/web_crawler/testbench/` (`python -m app.proprietary.web_crawler.testbench`); Suite S (stealth, shipped builder) + Suite E (extraction via real `crawl_url`) + scorecard JSON/MD baseline diff | | 4 | `04a-connector-category.md` | drafted | | 4 | `04b-source-discovery.md` | drafted | | 5 | `05-pipelines-model.md` | drafted | diff --git a/plans/backend/03e-stealth-hardening.md b/plans/backend/03e-stealth-hardening.md index 09227fe48..0d16efd6b 100644 --- a/plans/backend/03e-stealth-hardening.md +++ b/plans/backend/03e-stealth-hardening.md @@ -1,34 +1,52 @@ # Phase 3e — Stealth hardening (the in-house "undetectable" moat) > Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`. -> Depends on `03a` (Scrapling StealthyFetcher tier + `FetchStrategy` seam + `CrawlOutcome`) and `03b` (app-wide proxy provider). Precedes `03d` (captcha) in the escalation order — hardening **avoids** challenges; captcha solving is the paid last resort for the ones we can't avoid. Touches `03c` only via the deferred paid-unblocker tier (its own billing, decided if/when built). +> Depends on `03a` (Scrapling StealthyFetcher tier + `CrawlOutcome`) and `03b` (app-wide proxy provider). Precedes `03d` (captcha) in the escalation order — hardening **avoids** challenges; captcha solving is the paid last resort for the ones we can't avoid. Touches `03c` only via the deferred paid-unblocker tier (its own billing, decided if/when built). + +> **Architecture reconciliation (read before building).** Earlier drafts of this plan referenced a `FetchStrategy` seam / strategy-object contract `(url, ctx) -> CrawlOutcome`. **That seam does not exist in shipped code.** `03a`–`03d` shipped a *hardcoded 3-tier ladder* inside `WebCrawlerConnector.crawl_url` (`app/proprietary/web_crawler/connector.py`): plain methods `_crawl_with_async_fetcher` / `_crawl_with_dynamic` / `_crawl_with_stealthy(_sync)` wrapped by `_run_tier_with_proxy_retry`, every tier returning the shared `CrawlOutcome`. `03e` therefore implements levers **not** as strategy classes but as: (1) a **centralized per-tier kwargs builder** (`app/proprietary/web_crawler/stealth.py` — **proprietary**, since it's bypass-specific tuning; the generic block classifier stays Apache-2 in `app/utils/crawl/`) that turns config → `StealthyFetcher`/`AsyncFetcher` kwargs, imported by both the connector and `03f`'s harness so there's no test-vs-prod drift; (2) small **helpers around the existing ladder** (block classifier, strategy memory); and (3) the deferred paid tier as a config-gated **extra ladder step**, not a `FetchStrategy`. The captcha precedent holds: generic config/plumbing → `app/utils/`; actual bypass logic (WebGL spoof JS, humanize choreography) → `app/proprietary/web_crawler/`. > **Implementation note.** Same convention as `03a`–`03d`: Phases 1–2 are **SHIPPED** (live code says `workspace_id`/`Workspace` — substitute for any old `search_space_*`/`SearchSpace` citations and grep the new name); citations also predate `03a`'s `crawl_url` refactor; locate code by **symbol/grep**, not absolute lines. Scrapling references point at the on-disk (gitignored) `references/Scrapling/` checkout pinned to `scrapling[fetchers]>=0.4.9` (`pyproject.toml:91`). +> **Build status.** Sliced for risk (see "Build slicing" below). **Slice A (SHIPPED in this pass):** centralized stealth kwargs builder + fingerprint flags + geoip-from-proxy-location + block classifier (additive `CrawlOutcome.block_type`) + fonts/Xvfb packages in the image. **Slice B / C (deferred):** headed-Xvfb runtime path, humanization/WebGL `init_script` assets, persistent profiles, strategy memory, paid-unblocker stub. Effectiveness of every lever is graded in `03f` (manual), not unit-tested here. +> +> **First baseline evidence (`03f`, 2026-06-30 — headless, rotating residential, captcha OFF, Slice-A only with `CRAWL_GEOIP_MATCH_ENABLED=false`).** Suite S = **6 PASS / 4 FAIL** (full table in `03f`). The 4 fails map cleanly onto this plan's deferred levers and confirm the predicted ceiling — they are **not** Slice-A regressions: +> - **iphey "Unreliable"** → the predicted geoip-incoherence tell (browser tz/locale vs proxy exit geo). It's now the live regression test for **§1 geoip coherence** — flipping `CRAWL_GEOIP_MATCH_ENABLED` is the expected fix to validate next. +> - **CreepJS `hasHeadlessWorkerUA: true`** (headless 33%) → the Web **Worker** `navigator.userAgent` still leaks `HeadlessChrome` (main-thread UA is clean). A concrete, plausibly-fixable **Slice-B** target (worker UA override via `init_script`), beyond today's Slice-A flags. +> - **FingerprintJS Pro "tampering detected, access denied"** → the documented **commercial/device-fingerprint wall** (§2c WebGL/GPU gap). Needs Slice B/C or the §8 paid tier; expected to stay red on the free stack. +> - **incolumitas** → a single legacy `fpscanner WEBDRIVER` check (all modern checks pass); lowest priority. Its IP classifier also flip-flopped `is_datacenter` true/false across rotations, exposing that the `anonymous_proxies` pool mixes datacenter+residential exits (a proxy-provider variable, not a code issue). + ## Objective -Push the Universal WebURL Crawler as far toward "undetectable" as the **free / in-house stack** allows, so we hold a scraping moat for the next ~4–6 months **without** a third-party unblocker (ZenRows/ScrapFly/Bright Data) or a source-patched browser (CloakBrowser — rejected on licensing). Everything here is **runtime/config-level** stealth layered on Scrapling's patchright-Chromium (`03a` engine note): geoip fingerprint coherence, persistent profiles, headed execution, real fonts, and behavioral humanization — plus a **block classifier** + **per-domain strategy memory** so the ladder learns and the cheap tiers get skipped once a domain's working strategy is known. The paid-unblocker tier is defined here as a **deferred** `FetchStrategy` (the explicit escape hatch for when in-house maintenance gets too costly), not built now. +Push the Universal WebURL Crawler as far toward "undetectable" as the **free / in-house stack** allows, so we hold a scraping moat for the next ~4–6 months **without** a third-party unblocker (ZenRows/ScrapFly/Bright Data) or a source-patched browser (CloakBrowser — rejected on licensing). Everything here is **runtime/config-level** stealth layered on Scrapling's patchright-Chromium (`03a` engine note): geoip fingerprint coherence, persistent profiles, headed execution, real fonts, and behavioral humanization — plus a **block classifier** + **per-domain strategy memory** so the ladder learns and the cheap tiers get skipped once a domain's working strategy is known. The paid-unblocker tier is defined here as a **deferred config-gated ladder step** (the explicit escape hatch for when in-house maintenance gets too costly), not built now. ## The realistic ceiling (be honest with downstream devs) -Scrapling already ships strong **runtime** stealth by default: `DEFAULT_ARGS + STEALTH_ARGS` (`references/Scrapling/scrapling/engines/constants.py:24–99`, incl. `--disable-blink-features=AutomationControlled` `:94`), a persistent context (`_browsers/_stealth.py:91–93,366–368`), and `navigator.webdriver` masking via patchright. With `03b` residential proxies + this plan's coherence/humanization on top, the crawler reliably handles **Cloudflare** (via `03a` `solve_cloudflare`) and the long tail of **moderate** anti-bot. +Scrapling already ships strong **runtime** stealth by default: `DEFAULT_ARGS + STEALTH_ARGS` (incl. `--disable-blink-features=AutomationControlled`), a persistent context, and `navigator.webdriver` masking via **patchright**. Be precise about what patchright actually is: it patches the **Playwright driver** to remove *automation leaks* — it never calls `Runtime.enable`, disables `Console.enable`, and reaches closed shadow roots (verified: [patchright-python](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright-python), [Scrappey](https://scrappey.com/qa/web-scraping-apis/what-is-patchright)). It hides **"I'm automated," not "this is a different machine."** It does **not** spoof canvas, WebGL, audio, screen, or GPU. With `03b` residential proxies + this plan's coherence/humanization on top, the crawler reliably handles **Cloudflare** (via `03a` `solve_cloudflare`) and the long tail of **moderate** anti-bot — which is most CI targets. -What the free stack does **not** reliably beat: **DataDome, Kasada, reCAPTCHA-Enterprise**, and other behavioral/device-fingerprint systems. Those generally require C++ source-level browser patches (CloakBrowser's 58-patch approach — out on licensing) or a commercial unblocker. For **competitive-intelligence targets** these top-tier defenses are the exception, not the rule, so the in-house moat is a sound 4–6 month bet — with the deferred paid tier (§8) as the pre-wired escape hatch. **Do not promise "beats everything."** +**The device-fingerprint wall (the WebGL/GPU gap — see §2c).** Our worker has no GPU, so Chromium's WebGL renderer reports `Google SwiftShader` / `Mesa llvmpipe` — a string anti-bot vendors treat as ">95% bot" ([ipasis](https://ipasis.com/blog/webgl-fingerprinting-bot-detection), [Scrappey](https://scrappey.com/qa/anti-bot/what-is-webgl-fingerprinting)). Scrapling exposes only `allow_webgl` (on/off — and *off* is itself a tell), **no renderer spoof**; CloakBrowser fixes this at the C++ level (`--fingerprint-gpu-renderer`) and we cannot. Concretely, **without a real GPU or a source-patched binary we cannot**: -## Levers (all behind the `03a` `FetchStrategy` seam — callers stay outcome-only) +1. **Pass pure device-fingerprint bot checks** — FingerprintJS bot-detection, CreepJS / BrowserScan "is this a real device." CloakBrowser passes these (BrowserScan 4/4); our SwiftShader renderer fails them. +2. **Reach human-level reCAPTCHA v3 *scores*** (~0.7–0.9) **on our own browser** — we land ~0.1–0.5. (Note: `03d`'s **paid solver sidesteps this for token challenges** — the token is minted on the vendor's real-browser infra, so its score isn't ours. The gap only bites where the *site itself* reads our live v3 score.) +3. **Max out the device-fingerprint component** of DataDome / Kasada / Akamai — one of several reasons we plateau below managed APIs. + +**Numbers (2026 benchmarks — use these in any user-facing copy).** Free stack **+ residential**: Basic (CF) ~99%; Medium (CF Pro / PerimeterX) ~90%; **Enterprise (DataDome / Kasada / Akamai) ~60–75% initially, decaying toward ~60% within 24h** as their ML clusters the fingerprint (managed APIs hold ~89%) ([scrapewise DataDome 2026](https://scrapewise.ai/blogs/bypass-datadome-web-scraping-2026)). So the honest line is **"partial and decaying, not SLA-grade,"** — neither "can't beat" nor "beats everything." For CI targets the top-tier defenses are the exception, so the in-house moat is a sound 4–6 month bet, with the deferred paid tier (§8) as the **evidence-driven** escape hatch (`03f`'s scorecard is the trigger). **Do not promise "beats everything."** + +## Levers (wired via the centralized kwargs builder + ladder helpers — callers stay outcome-only) ### 1. Geoip fingerprint coherence (match the browser to the proxy exit IP) A residential IP in Berlin behind an `en-US`/`America/New_York` browser is an instant tell. Make the fingerprint cohere with the proxy's exit geo: -- Resolve the proxy exit IP → country/locale/timezone (lightweight geoip; e.g. a bundled MaxmindLite/`geoip2` DB or the proxy provider's location knob `RESIDENTIAL_PROXY_LOCATION` from `03b`). -- Pass the matched values into StealthyFetcher: `locale=` (drives `navigator.language` + `Accept-Language` — `stealth_chrome.py:34–35`) and `timezone_id=` (`:36`). These flow into the Playwright context (`_browsers/_base.py:440–441`). -- This needs the crawl's **chosen proxy endpoint** surfaced into the strategy (the same seam `03d` needs for IP-bound solves) — capture it once in `03a`'s strategy context, don't re-call `get_proxy_url()` (which rotates on a pool-backed provider, `03b`). +- **Primary source (Slice A, no network call): the configured `RESIDENTIAL_PROXY_LOCATION` (`03b`, `config:1048`).** Map that string (best-effort: ISO-3166 alpha-2 like `us`/`de`, or a common country name) → a representative `(locale, timezone_id)`. Coarse country granularity is fine (per "Risks": wrong-but-coherent beats default-mismatched); unknown/empty → skip (leave Scrapling's system default). **No exit-IP resolution** — avoids a per-crawl network round-trip + failure mode. (A `geoip2`/MaxmindLite exit-IP path is a later refinement, not MVP.) +- Pass the matched values into StealthyFetcher via the kwargs builder: `locale=` (drives `navigator.language` + `Accept-Language`) and `timezone_id=` (confirmed real kwargs on `StealthyFetcher.fetch`). These flow into the Playwright context. +- The stealthy tier **already captures the chosen proxy endpoint once** (`_crawl_with_stealthy_sync`: `proxy = get_proxy_url()`, the same value handed to `03d`'s solver for IP-coherence) — reuse that captured `proxy`/location, **don't** re-call `get_proxy_url()` (which rotates on a pool-backed provider, `03b`). +- **Caveat (be honest):** Scrapling applies `locale`/`timezone_id` via Playwright **CDP emulation**, which advanced systems can flag *as* emulation — CloakBrowser sets these as *binary flags* precisely to avoid "detectable CDP emulation" (`references/CloakBrowser/README.md:309`). Still strictly better than a mismatched default (wrong-but-coherent beats `UTC`+`en-US` behind a Berlin IP), but it's a weaker variant than a source-patched browser, not parity. ### 2. Fingerprint flags Scrapling already exposes (turn them on) -- `hide_canvas=True` — random noise on canvas ops to defeat canvas fingerprinting (`stealth_chrome.py:40`). -- `block_webrtc=True` — forces WebRTC to respect the proxy, preventing the **real local IP leak** that unmasks proxied browsers (`:41`). +- `hide_canvas` — Scrapling adds **random** noise to canvas ops (verified in the installed `StealthyFetcher.fetch` docstring). **Not "free safe":** random-per-call noise is itself a signal on FingerprintJS/CreepJS-class detectors — an *unstable* canvas hash is as loud as `navigator.webdriver=true` ([dev.to](https://dev.to/tanwydd/how-your-canvas-fingerprint-gets-you-caught-and-why-random-noise-makes-it-worse-3ba1)); CloakBrowser's own FPJS fix is `--fingerprint-noise=false`. **Default OFF; gate per-domain and validate in `03f`.** Harmless against Cloudflare/moderate, potentially harmful against device-fingerprint graders. (Canvas stealth that actually *helps* must be **deterministic per persistent profile**, not random — out of scope here.) +- `block_webrtc=True` — forces WebRTC to respect the proxy, preventing the **real local IP leak** that unmasks proxied browsers (`:41`). **This one is genuinely cheap + safe — keep default TRUE.** +- `dns_over_https` — Scrapling can route DNS via Cloudflare DoH to stop the **DNS leak** that unmasks proxied browsers (same anti-leak class as `block_webrtc`; confirmed kwarg on `StealthyFetcher.fetch`). **Trade-off: it adds a DNS round-trip, a (small) latency cost — so per the "no speed regression" constraint it ships *off by default* (`CRAWL_DNS_OVER_HTTPS`, default FALSE), pre-wired for operators who prefer leak-safety over the marginal latency.** (Idea sourced from the Camoufox-based FlareSolverr alternatives `references/Byparr-main` / `references/trawl-dev`, which treat DNS/WebRTC leak coherence as first-class.) - `google_search=True` (default) — sets a Google referer so the first hit looks like organic arrival (`:45`); override per-need via `extra_headers` (`:46`). - `additional_args=` — last-priority Playwright context overrides for anything not surfaced as a first-class param (`:51`). @@ -36,9 +54,21 @@ A residential IP in Berlin behind an `en-US`/`America/New_York` browser is an in The cheap static tier (`03a` tier 1) is the **first** thing every crawl hits. `03a` **already ships `impersonate="chrome"`** on it (`app/proprietary/web_crawler/connector.py`, the `AsyncFetcher.get` call) — Scrapling's static engine selects a matching curl_cffi browser profile (`references/Scrapling/scrapling/engines/static.py:36–47`), so the HTTP tier's **JA3/JA4/HTTP-2** already matches a real Chrome and coheres with the browser tiers' UA. `03e`'s remaining work here is **not** to add it, but to (a) fold the `impersonate` profile into the centralized per-tier kwargs builder (below) so it stays the single source of truth, and (b) keep the chosen profile coherent with the proxy exit / UA. `03f §S3` validates the shipped parity against `tls.peet.ws`. +### 2c. WebGL / GPU renderer (the biggest free-stack gap — partial mitigation only) + +This is the "device-fingerprint wall" from the ceiling section. On a GPU-less worker the WebGL renderer string is `Google SwiftShader` / `ANGLE (Mesa, llvmpipe…)` — an instant tell on any detector that reads `WEBGL_debug_renderer_info`. Scrapling gives us **no renderer knob** (only `allow_webgl` on/off, and *off* is a tell). Three options, increasing strength + cost: + +1. **DIY JS spoof (optional, OFF by default — Slice B).** An `init_script` overrides `getParameter` for `UNMASKED_VENDOR_WEBGL` / `UNMASKED_RENDERER_WEBGL` to a believable, **platform-coherent** value (e.g. an Intel/ANGLE renderer that matches the spoofed Windows UA), held **consistent per persistent profile** (§3). **Impl note:** Scrapling's `init_script` is **an absolute path to a `.js` file**, *not* an inline string (confirmed in `StealthyFetcher.fetch` docstring) — so this ships as a **bundled `.js` asset** in `app/proprietary/web_crawler/` (next to `03d`'s captcha logic) passed by absolute path; the kwargs builder resolves the path when `CRAWL_WEBGL_SPOOF_ENABLED`. **Must** also patch `Function.prototype.toString` so the override still returns `[native code]` — otherwise *the spoof itself is the signal* (exactly the `toString()` probe Kasada catalogs, [web-scraping-guide](https://web-scraping-guide.com/)). This defeats **string-based** WebGL checks (the common case) but **not** render-output pixel-hash or GPU-perf probes (FingerprintJS-grade) — a software rasterizer's actual draw output still differs and can't be faked in JS. It is the brittle config-level approach patchright/CloakBrowser explicitly warn rots; treat it as a **cheap, test-gated win, not the moat**. +2. **Real GPU passthrough (hosted workers only).** `--device /dev/dri` + ANGLE/EGL so WebGL reports a *genuine* renderer with *real* render output, defeating even output/perf probes. Best technical fix; costs GPU instances + image complexity ([livekit/egress GPU notes](https://stackoverflow.com/questions/78985698)). A hosted-tier escalation, off for self-hosted. +3. **Defer to the paid unblocker (§8).** Route the residual device-fingerprint-gated domains (flagged by the §7 classifier as in-house-unreachable) to the external unblocker. The pragmatic moat-bounding fallback. + +**Decision for this MVP:** ship option 1 as an opt-in flag (`CRAWL_WEBGL_SPOOF_ENABLED`, default FALSE), name option 2 as the hosted escalation, and lean on §8 for the rest. Quantify all three in `03f`. + ### 3. Persistent per-domain profiles (look like a returning human) -Scrapling defaults to a **temporary** user-data dir (fresh = suspicious). Use `user_data_dir=` (`stealth_chrome.py:48`; `launch_persistent_context` `_stealth.py:93,366–368`) to keep a **persistent profile per domain** (or per domain+proxy-geo), so cookies/localStorage/site-trust carry across crawls and the browser presents as a returning visitor rather than a brand-new incognito session. Store profiles under a configured dir (`shared_tmp`-style volume so API + worker share them). +Scrapling defaults to a **temporary** user-data dir (fresh = suspicious). Use `user_data_dir=` (confirmed real kwarg; `launch_persistent_context`) to keep a **persistent profile per domain** (or per domain+proxy-geo), so cookies/localStorage/site-trust carry across crawls and the browser presents as a returning visitor rather than a brand-new incognito session. Store profiles under a configured dir (`shared_tmp`-style volume so API + worker share them). + +- **Concurrency hazard (Slice C design item — why this is deferred, not a one-liner):** Chromium **hard-locks** a `user_data_dir`; two crawls touching the same profile dir at once crash (`SingletonLock`). Our crawler runs concurrently (multiple connectors/chats across workers), so a naive per-domain dir collides. The fix needs one of: (a) an async lock keyed by profile dir so same-domain crawls serialize (adds latency, can deadlock under load), (b) per-`(domain, worker, pid)` dirs + a periodic prune (loses some cross-run trust, no contention), or (c) persistent only when the strategy memory (§7) says a domain *needs* it, temp otherwise. Decide in Slice C alongside §7; **Slice A keeps Scrapling's temp default.** ### 4. Headed execution under Xvfb (defeat headless tells) @@ -46,28 +76,30 @@ Scrapling defaults to a **temporary** user-data dir (fresh = suspicious). Use `u ### 5. Real fonts (canvas/emoji hash realism) -A minimal container has almost no fonts, making canvas/emoji fingerprint hashes obviously synthetic. Install real font packages (DejaVu, Liberation, Noto incl. CJK + emoji) in the worker image so font-enumeration + canvas hashes resemble a real desktop. `Dockerfile` change only (`apt-get install fonts-*`). +A minimal container has almost no fonts, making canvas/emoji fingerprint hashes obviously synthetic. Install real font packages so font-enumeration + canvas/emoji hashes resemble a real desktop. Use the set CloakBrowser ships as proven against Kasada/Akamai emoji-canvas hashing (`README.md:737–739`): `fonts-noto-color-emoji`, `fonts-unifont`, `fonts-ipafont-gothic` (CJK), `fonts-wqy-zenhei` (CJK), `fonts-tlwg-loma-otf` — plus `fonts-dejavu`/`fonts-liberation` for Latin coverage. `Dockerfile` change only (`apt-get install fonts-*`). ### 6. Behavioral humanization (DIY — Scrapling has no `humanize` for the Chromium engine) Unlike Camoufox, the patchright-Chromium StealthyFetcher exposes **no built-in `humanize`** (verified: zero matches in the StealthyFetcher param set). So humanization is custom, injected via the existing hooks: -- `page_action=` (runs after navigation, `stealth_chrome.py:30`; engine `_stealth.py:258–262`) — randomized mouse moves/curves, scrolls, hover-before-click, and small think-time delays before extraction. This is the **same hook `03d` uses** for token injection, so the two compose (humanize → optional captcha solve). -- `init_script=` (JS executed on page creation, `:33`) — early shims for any residual JS tells not covered by patchright. +- `page_action=` (runs after navigation) — randomized mouse moves/curves, scrolls, hover-before-click, and small think-time delays before extraction. This is the **same hook `03d` uses** for token injection — `_crawl_with_stealthy_sync` currently builds exactly **one** `page_action` (the captcha injector), so Slice B must **compose** two callables (humanize → optional captcha solve) into the single `page_action` kwarg, not overwrite it. +- `init_script=` (an **absolute path to a `.js` file** executed on page creation — *not* inline JS, see §2c) — early shims for any residual JS tells not covered by patchright (and the §2c WebGL spoof, if enabled). - Tunable `wait`/`network_idle` (`:29,27`) so dwell time isn't robotically constant. +- **Concrete rules (from CloakBrowser's reCAPTCHA notes, `README.md:1280–1314` — encode these):** use native `time.sleep`, **never** `page.wait_for_timeout()` (it emits CDP traffic detectors flag); prefer `page.type()` over `fill()` for any input (real keystrokes vs. value-set); dwell **15s+** before triggering score-based checks; space repeat hits to score-gated domains **30s+**; keep a **stable identity per domain** (persistent profile §3 + consistent spoofed values §2c) so a score-based system sees a *returning* device, not a fresh one each visit. ### 7. Block classifier + per-domain strategy memory (the "learning ladder") Two small in-house pieces make the ladder smart instead of brute-force: -- **Block classifier** — inspect each `Response` (status + body/cookie markers) and label the outcome: `OK` / `CLOUDFLARE` / `CAPTCHA_RECAPTCHA` / `CAPTCHA_HCAPTCHA` / `DATADOME` / `KASADA` / `RATE_LIMITED` / `EMPTY`. (Markers: `"Just a moment"` + `cf-mitigated` → Cloudflare; `datadome` cookie/script → DataDome; `g-recaptcha`/`h-captcha` sitekeys → captcha; etc.) The label drives escalation (which tier/lever next), routes captcha types to `03d`, and feeds the memory below. It also makes `CrawlOutcome.status` decisions principled instead of "empty == fail". -- **Per-domain strategy memory** — cache the **strategy that last succeeded per domain** (e.g. Redis key `crawl:strategy:{domain}` with TTL, reusing the existing Redis the indexer already uses for `indexing_locks`). Next crawl of that domain starts at the known-good tier/lever set, skipping cheaper tiers that always fail there — fewer requests, lower latency, less proxy/captcha spend. No DB migration (Redis, best-effort, self-healing on miss). +- **Block classifier (Slice A — additive, pure, unit-testable).** Inspect the fetched page (`status` + body markers; cookies/headers are not threaded through `_build_result` today, so MVP is **status + body-marker based**) and label the outcome: `OK` / `CLOUDFLARE` / `CAPTCHA_RECAPTCHA` / `CAPTCHA_HCAPTCHA` / `DATADOME` / `KASADA` / `RATE_LIMITED` / `EMPTY` / `UNKNOWN`. **Concrete markers — adopt the set proven in `references/trawl-dev/packages/tiers/src/detect.ts` (a near-identical classifier):** Cloudflare/DDoS-Guard → `cf-mitigated`-style title strings `"just a moment"` / `"checking your browser"` / `"enable javascript and cookies to continue"` / `"verify you are human"`, DOM ids `challenge-running` / `cf-challenge-running` / `turnstile-wrapper`, and `ddos-guard.net`; Turnstile → `class="cf-turnstile"` / `challenges.cloudflare.com/turnstile`; hCaptcha → `class="h-captcha"` / `hcaptcha.com/1/api`; reCAPTCHA → `class="g-recaptcha"` / `google.com/recaptcha` / `recaptcha.net`; DataDome → `datadome` script/cookie; **status `202`/`403`/`429` as a bot-gate** (`202` is an IMDb-style pre-gate, per `detect.ts:isBlocked`). **Invariant: the classifier is purely *additive* — it attaches `CrawlOutcome.block_type` and feeds telemetry/routing; it MUST NOT change *when* `SUCCESS` is returned, because `03c` bills on `status == SUCCESS` (`CrawlOutcome` docstring).** Wiring mirrors `03d`'s proven `captcha_state` pattern: a per-call `block_state` dict mutated in `_build_result` (which has `raw_html` + `status`) and stamped onto every `CrawlOutcome` return path in `crawl_url`. +- **Per-domain strategy memory (Slice C — needs a new dependency + control-flow change).** Cache the **strategy that last succeeded per domain** (Redis key `crawl:strategy:{domain}` with TTL). Next crawl starts at the known-good tier, skipping cheaper tiers that always fail there. **Why deferred:** `WebCrawlerConnector()` is currently constructed with **no args** and is Redis/DB-free; this requires injecting a cache client (best-effort, no-op when absent) **and** letting `crawl_url` skip ladder tiers — a control-flow change to the tuned ladder. No DB migration (Redis, best-effort, self-healing on miss), but it is a deliberate design item, not a flag flip. +- **Solved-session cache (Slice C refinement — speed-positive, the strongest version of the memory above).** Beyond remembering *which tier* won, cache the **solved session itself** — the `cf_clearance`/`__cf_bm` cookies + the UA that cleared the challenge — in Redis (`crawl:session:{domain}` with TTL), and **replay** it on the next crawl via Scrapling's `cookies=` input, so a returning hit **skips the expensive solve entirely** (the pattern `references/trawl-dev` Tier 2 uses for its ~500ms repeat path: `browser/src/session.ts` `session:{domain}` save/load/invalidate; `tiers/src/tier2.ts` replays cookies, and on a re-challenge marks `session-expired` → invalidate → escalate). **Net effect is faster, not slower** (it removes a solve), so it fits the "no speed regression" bar. **Caveat:** `cf_clearance` is **IP+UA-bound**, so it pays off most behind **sticky/static proxies** (the `03b` static-proxy future) and the cache key should include proxy-geo; under rotating residential the replayed cookie may not match the new exit IP (best-effort, self-healing via the `session-expired` invalidate). ## 8. Deferred: paid-unblocker tier (the escape hatch, NOT built now) The moat strategy is explicit: **maintain in-house bypass for ~4–6 months, then move hostile targets to a paid unblocker if demand/maintenance justifies it.** That switch is **evidence-driven, not a guess** — `03f`'s manual scorecard quantifies the free-stack ceiling over time and is the documented trigger for flipping this tier. Pre-wire the seam so the switch is a config flip, not a refactor: -- Define (but do not implement) a `PaidUnblockerStrategy` that satisfies the `03a` `FetchStrategy` contract `(url, ctx) -> CrawlOutcome`, appended **last** in the ladder and active only when an env flag + API key are set. +- Define (but do not implement) a paid-unblocker tier — a `_crawl_with_paid_unblocker(url) -> dict | None` method returning the same shape as the existing tiers, appended **last** in `crawl_url`'s ladder (after the stealthy tier) and active only when an env flag + API key are set. (Same plain-method shape as today's tiers — there is no `FetchStrategy` contract to satisfy.) - It would call an external unblocker (ZenRows/ScrapFly/Bright Data Web Unlocker) for the residual `DATADOME`/`KASADA`/`reCAPTCHA-Enterprise` domains the block classifier flags as unreachable in-house. - **Its own billing** (cost-plus pass-through, decided at build time) — separate from `03c`'s flat crawl unit and `03d`'s per-solve unit, because unblocker pricing is per-request and provider-specific. - Until built, those domains simply return non-`SUCCESS` (free under `03c`). This keeps the umbrella's "WebURL Crawler is the moat" honest while bounding our maintenance risk. @@ -76,8 +108,10 @@ The moat strategy is explicit: **maintain in-house bypass for ~4–6 months, the Add (all default OFF / conservative; next to the `03b`/`03c` knobs in `config/__init__.py` + `.env.example`): -- `CRAWL_GEOIP_MATCH_ENABLED` (default FALSE) + optional geoip DB path. -- `CRAWL_HIDE_CANVAS` / `CRAWL_BLOCK_WEBRTC` (default TRUE — cheap, safe). +- `CRAWL_GEOIP_MATCH_ENABLED` (default FALSE; Slice A — maps `RESIDENTIAL_PROXY_LOCATION` → `locale`/`timezone_id`, no exit-IP lookup). +- `CRAWL_BLOCK_WEBRTC` (default TRUE — cheap, safe; Slice A), `CRAWL_HIDE_CANVAS` (default **FALSE** — random canvas noise can itself be a tell; opt-in + `03f`-validated, see §2; Slice A), and `CRAWL_GOOGLE_SEARCH_REFERER` (default TRUE — Scrapling's `google_search`; Slice A). +- `CRAWL_DNS_OVER_HTTPS` (default **FALSE** — anti DNS-leak, but adds a DNS round-trip; default-off to honor the "no speed regression" bar, pre-wired for leak-safety-first operators; §2; Slice A). +- `CRAWL_WEBGL_SPOOF_ENABLED` (default FALSE) + optional `CRAWL_WEBGL_VENDOR` / `CRAWL_WEBGL_RENDERER` override strings (§2c; `toString`-safe JS spoof, defeats string-checks only). - `CRAWL_PERSISTENT_PROFILES_DIR` (unset → Scrapling's temp default; set → per-domain profiles). - `CRAWL_HEADED_XVFB_ENABLED` (default FALSE; requires Xvfb in the image). - `CRAWL_HUMANIZE_ENABLED` (default TRUE) + dwell/jitter bounds. @@ -86,33 +120,47 @@ Add (all default OFF / conservative; next to the `03b`/`03c` knobs in `config/__ ## Docker changes (`surfsense_backend/Dockerfile`) -- Install **Xvfb** + a font set (`fonts-dejavu`, `fonts-liberation`, `fonts-noto`, `fonts-noto-cjk`, `fonts-noto-color-emoji`) in the worker image. (Note from `03a`: also drop the stale "+ Camoufox" comment near `:112`; `scrapling install` only fetches Chromium.) +- Install **Xvfb** + the proven font set (`fonts-noto-color-emoji`, `fonts-unifont`, `fonts-ipafont-gothic`, `fonts-wqy-zenhei`, `fonts-tlwg-loma-otf`, `fonts-dejavu`, `fonts-liberation`; §5) in the worker image. (Note from `03a`: also drop the stale "+ Camoufox" comment near `:112`; `scrapling install` only fetches Chromium.) - Headed runs need the browser launched under `xvfb-run` (or an Xvfb display in the worker entrypoint), gated by `CRAWL_HEADED_XVFB_ENABLED`. +## Build slicing (risk-staged; `03f` validates effectiveness) + +The levers split by risk/testability. **Slice A is built now** (prod-safe, config-gated, defaults preserve today's behavior, wiring + classifier unit-tested). **B/C are deferred** (need infra or architectural design; mostly only verifiable in `03f`). + +- **Slice A (SHIPPED):** centralized kwargs builder → fingerprint flags (`block_webrtc` ON, `hide_canvas` OFF, `google_search` ON, `dns_over_https` OFF); geoip coherence from `RESIDENTIAL_PROXY_LOCATION`; fonts + Xvfb **packages** in the Dockerfile; **additive** block classifier (`CrawlOutcome.block_type`, classified in `_build_result` **and** the static tier's `>=400` early-return so the first/cheapest-tier bot-gate isn't lost). **Licensing split:** the stealth kwargs builder is **proprietary** (`app/proprietary/web_crawler/stealth.py` — bypass tuning); the block classifier stays **Apache-2** (`app/utils/crawl/classifier.py` — generic passive telemetry from public markers). **`extra_headers`/`additional_args` were intentionally NOT wired** — they're last-priority override hatches with no clean env representation (dict-in-env) and no concrete need yet; add them when a real use appears rather than speculative config (YAGNI). +- **Slice B (next):** headed-Xvfb **runtime path** (`headless=False` + entrypoint), humanization `page_action` (composed with `03d`'s injector), WebGL-spoof `init_script` `.js` asset (proprietary). +- **Slice C (defer):** per-domain persistent profiles (concurrency design, §3), per-domain strategy memory + tier-skipping (`crawl_url` control-flow change + cache injection, §7), **solved-session cache** (`cf_clearance` replay via `cookies=`, §7 — speed-positive, best with sticky proxies), **warm browser pool** (a hosted-scale latency win — `references/trawl-dev` `browser/src/pool.ts` keeps N warm instances with an acquire-timeout→429; Scrapling's `StealthySession` already has page pooling), paid-unblocker config stub (§8). GPU passthrough stays infra/hosted-only (§2c). + ## Work items -1. **Geoip coherence** — resolve proxy exit geo → `locale`/`timezone_id`; thread the crawl's chosen endpoint into the strategy context (shared seam with `03d`). -2. **Fingerprint flags** — wire `hide_canvas`/`block_webrtc`/`google_search`/`extra_headers`/`additional_args` from config into the StealthyFetcher tier. The AsyncFetcher (HTTP) tier **already carries `impersonate="chrome"`** (shipped in `03a`, §2b) — **fold it into** the builder below, don't re-add it. **Centralize this into a single per-tier kwargs builder** (one function that returns the StealthyFetcher / AsyncFetcher kwargs from config) — the crawler *and* `03f`'s harness both import it, so the scorecard grades the exact browser we ship (no test-vs-prod drift). -3. **Persistent profiles** — per-domain `user_data_dir` under `CRAWL_PERSISTENT_PROFILES_DIR` (shared volume). -4. **Headed + Xvfb** — `headless=False` path gated by flag; Xvfb + fonts in the worker image. -5. **Humanization** — a `page_action` humanizer (mouse/scroll/dwell) composing with `03d`'s injector; optional `init_script` shims. -6. **Block classifier** — `classify(response) -> BlockType`, used by the ladder + `CrawlOutcome.status` + `03d` routing. -7. **Per-domain strategy memory** — Redis read/write around the ladder; start at known-good tier; self-heal on miss. -8. **Paid-unblocker seam** — define the deferred `FetchStrategy` + config flag only (no provider integration). -9. **Instrumentation** — log `(domain, block_type, winning_strategy, attempts, latency)` for tuning the ladder. -10. **Config + Docker + tests**. +**Slice A (this pass):** +1. **Config + `.env.example`** — add the Slice-A knobs (below); geoip reuses `RESIDENTIAL_PROXY_LOCATION`. +2. **`app/proprietary/web_crawler/stealth.py`** (**proprietary** — bypass tuning) — `StealthConfig` + `get_stealth_config()` snapshot; coarse `RESIDENTIAL_PROXY_LOCATION → (locale, timezone_id)` map; `build_stealthy_kwargs(cfg)` returning the config-derived `StealthyFetcher` kwargs. Single source of truth imported by the connector **and** `03f`'s harness (no test-vs-prod drift). The AsyncFetcher tier **already carries `impersonate="chrome"`** (shipped `03a`, §2b) — leave it; document it as the static-tier coherence anchor. +3. **`app/utils/crawl/classifier.py`** (Apache-2) — `BlockType` enum + `classify_block(status, html) -> BlockType` (pure, status + body-marker based). +4. **Connector wiring** — merge `build_stealthy_kwargs(...)` into `_crawl_with_stealthy_sync`'s kwargs (existing `headless`/`network_idle`/`block_ads`/`solve_cloudflare`/`proxy`/captcha `page_action` preserved; lever keys never collide); add `block_type` to `CrawlOutcome`; thread a per-call `block_state` dict (classify in `_build_result` **and** the static tier's `>=400` early-return path, stamp on every `crawl_url` return) — **additive only, SUCCESS predicate unchanged**. +5. **Docker** — Xvfb + the proven font set in the worker image; drop the stale "+ Camoufox" comment. +6. **Tests + lints** — unit-test the builder (geoip map, flag wiring, snapshot) + classifier (each marker); update connector tests for `block_type`; run suite. + +**Slice B / C (deferred — tracked above):** WebGL-spoof `init_script`, humanization composer, headed-Xvfb runtime, persistent profiles, strategy memory + tier-skip, paid-unblocker stub, richer `(domain, block_type, winning_strategy, attempts, latency)` instrumentation. ## Risks / trade-offs - **Arms race / maintenance.** Fingerprint bypasses rot as WAFs update; this is exactly the cost the deferred paid tier (§8) hedges. Instrumentation (work item 9) plus **`03f`'s scorecard** are what tell us when in-house upkeep stops being worth it. +- **WebGL/GPU is a structural gap, not a bug (§2c).** The free stack cannot present a real consumer GPU; the JS spoof defeats string checks but not render-output/perf probes, and is itself brittle (rots on Chrome updates, must stay `toString`-clean). Plan for device-fingerprint-grade targets to need GPU-passthrough workers or the §8 paid tier — don't expect parity with CloakBrowser here. - **Headed/Xvfb cost.** Headful browsers use more CPU/RAM than headless; gate per-flag and only escalate to headed when the block classifier says cheaper tiers fail for a domain (per-domain memory keeps it from being the default). - **Profile growth.** Persistent profiles accumulate disk; add a size/TTL cap and periodic prune. - **Geoip accuracy.** A coarse country→locale map is fine; over-fitting per-city tz isn't worth it. Wrong-but-coherent beats default-mismatched. - **No silver bullet.** Reiterate the ceiling (§"realistic ceiling") in any user-facing copy: the crawler is "best-effort undetectable," not guaranteed. +## Prior art evaluated — why not Camoufox (decision: skip for now) + +`references/Byparr-main` and `references/trawl-dev` are FlareSolverr alternatives built on **Camoufox** — an **MPL-2.0**, open-source, **C++/Juggler-patched Firefox** that natively closes our biggest gaps: hardware-accurate **WebGL** renderer/vendor spoofing (bundled `webgl_data.db`, §2c), **`geoip=True`** locale/tz/geo coherence from the proxy IP (§1), **`humanize=True`** mouse, plus font/WebRTC/audio/screen spoofing — all at the engine level (no detectable JS shims). It is effectively the CloakBrowser-class capability *without* CloakBrowser's licensing problem. + +**Decision (this MVP): do not adopt Camoufox.** Rationale: Scrapling **dropped** Camoufox in favor of patchright-Chromium for its `StealthyFetcher` ([scrapling stealthy docs](https://scrapling.readthedocs.io/en/latest/fetching/stealthy.html); confirmed in source — `references/Scrapling/scrapling/engines/_browsers/_stealth.py` imports `patchright`), so wiring Camoufox would mean a **separate proprietary fetch tier** (camoufox+playwright, outside Scrapling), a Firefox engine (a few Chrome-only targets won't fit), a heavy extra binary, and tracking an actively-but-experimentally-developed project. **Revisit trigger:** if `03f`'s scorecard shows the patchright tier's device-fingerprint ceiling (§2c) is blocking real CI targets, a Camoufox tier is the prime free escalation to evaluate **before** the §8 paid unblocker. + ## Out of scope (hand-offs) -- Cloudflare solving (`03a`), reCAPTCHA/hCaptcha solving + its per-solve billing (`03d`), proxy rotation (`03b`), flat crawl billing (`03c`). +- Cloudflare solving (`03a`), reCAPTCHA/hCaptcha solving + its per-solve billing (`03d`), proxy rotation (`03b`), flat crawl billing (`03c`). **Forward note for `03d` (free solver fallback):** the Camoufox-based references solve reCAPTCHA-v2 / Turnstile **free** in-browser (audio-STT — `references/trawl-dev/packages/tiers/src/solvers/{stt,turnstile}.ts`) instead of a paid solver. `03d` shipped with paid `captchatools` as primary; a free STT path is a possible *fallback-before-paid* later, but audio STT is brittle/Google-rate-limited, so paid stays primary (and this doesn't change `03e`). - **Logged-in / account-based bypass** (sticky/static proxies + credential management) — deferred to the platform-actor work (umbrella Phase 8 + `03b` static-proxy hand-off). Public data only this MVP. - Building the paid-unblocker provider integration — deferred (§8 leaves only the seam + flag). - **Measuring** undetectability (the scorecard that grades these levers) → `03f` (manual harness). This plan *builds* the levers; `03f` *tests* them. diff --git a/plans/backend/03f-undetectability-testing.md b/plans/backend/03f-undetectability-testing.md index 02d7c28c4..dc5af1380 100644 --- a/plans/backend/03f-undetectability-testing.md +++ b/plans/backend/03f-undetectability-testing.md @@ -1,7 +1,9 @@ # Phase 3f — Undetectability & extraction test harness (manual scorecard) > Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`. -> **Status: ACTIVE, sequenced last in Phase 3** (after `03a`–`03e` ship — the harness measures what those built). **Manual-only** — no CI/automated gating for now (resolved decision); it's a dev-run scorecard, not a build gate. Depends on `03a` (the Scrapling tiers + `FetchStrategy`/`CrawlOutcome`), `03b` (proxy provider), `03e` (the stealth levers being tested), and reuses the `page_action` + closure-cell mechanism shared with `03d`. +> **Status: ✅ IMPLEMENTED + first baseline run (`ci_mvp`, 2026-06-30).** Harness lives under the **proprietary boundary** at `surfsense_backend/app/proprietary/web_crawler/testbench/` (package: `core.py` scorecard/closure-cell helper, `suite_stealth.py` Suite S, `suite_extraction.py` Suite E, `__main__.py` CLI, `README.md` runbook) — it's the moat's measurement tool, so it carries the `app/proprietary/LICENSE`, not Apache-2. Run: `python -m app.proprietary.web_crawler.testbench --suite all [--proxy URL] [--headed] [--no-screenshots]`. Suite S builds the StealthyFetcher tier from the **shipped** `build_stealthy_kwargs(get_stealth_config())` (no drift); Suite E drives the real `crawl_url` against ToS-safe sandboxes with deterministic assertions. Writes timestamped `scorecard-*.json`/`.md` + diffs the last run; the **whole `results/` tree is gitignored** (run-local: scorecards, `latest.json`, screenshots, dumps). Captcha forced OFF (unaided score). **Not** pytest-collected (live internet + proxy). **Manual-only** — no CI/automated gating (resolved decision); it's a dev-run scorecard, not a build gate. **First baseline: see "First baseline results" below.** +> +> **Post-implementation notes (reconciled to shipped code):** (1) the `03a`/`03e` "`FetchStrategy` seam" never shipped — levers are a centralized kwargs builder, reflected below; (2) **every detection site is now auto-graded from its real DOM verdict** — the initial "INFO + screenshot, read manually" fallback was replaced after a first run by per-site parsers written against actual DOM dumps (reCAPTCHA-v3 reads the server `"score"` JSON, CreepJS the headless-% + boolean spoof tells incl. `hasHeadlessWorkerUA`, incolumitas the fpscanner FAIL keys + `is_datacenter`, fingerprint-scan the `Bot Risk Score`, FingerprintJS the block message, BrowserScan the Normal/Abnormal count, iphey the masthead verdict, Cloudflare the bypass line). `INFO` is now reserved for purely informational rows (TLS JA3/JA4, exit IP) + the manual browserleaks links; screenshots are still captured as a backstop. Each site is one entry in `suite_stealth.py`, so tightening a parser is a one-function change. Depends on `03a` (the Scrapling tier ladder + `CrawlOutcome`), `03b` (proxy provider), `03e` (the stealth levers being tested), and reuses the `page_action` + closure-cell mechanism shared with `03d`. > **Convention note.** This is **dev/operator tooling**, not a product code path, so it's untouched by the Phase 1–2 rename (no `search_space_id`/`workspace_id` concern). Citations to the gitignored reference checkouts (`references/CloakBrowser/`, `references/Scrapling/`) are pinned to what's on disk; locate code by **symbol/grep** if lines drift. @@ -30,22 +32,26 @@ Their shipped bars (we adopt as **aspirational targets**, see Scorecard): sannys We are **not** a single browser; we're the `03a` tier ladder. The harness therefore tests **per tier**, and extracts verdicts two ways: - **DOM/JSON-rendered verdicts** → just `StealthyFetcher.fetch(url, …)` (or `Fetcher.get` for JSON endpoints) and parse the **returned post-JS page** with Scrapling's selector (`load_dom` is on by default — `references/Scrapling/scrapling/fetchers/stealth_chrome.py:43`). Covers sannysoft, incolumitas, deviceandbrowserinfo, the reCAPTCHA score text, and every JSON endpoint (`tls.peet.ws/api/all`, `httpbin/headers`). -- **Internal JS-object verdicts** (CreepJS `window.Fingerprint`, Castle.js score node) → a **`page_action`** that runs `page.evaluate()` and stashes the result into a **closure cell**, because Scrapling **discards `page_action`'s return value** (sync `_stealth.py:260`, async `:536`). **This is the exact same `page_action`+closure-cell plumbing as `03d`'s token injector** — building the harness de-risks `03d` (and vice-versa); factor it once. +- **Internal JS-object verdicts** → a **`page_action`** that runs `page.evaluate()` and stashes the result into a **closure cell**, because Scrapling **discards `page_action`'s return value** (sync `_stealth.py:260`, async `:536`). **This is the exact same `page_action`+closure-cell plumbing as `03d`'s token injector** — building the harness de-risks `03d` (and vice-versa); factor it once. *(As-built: CreepJS/fingerprint-scan turned out to render their verdicts into the visible DOM, so their parsers read the post-JS page text directly; the closure-cell `page_action` path is retained as the mechanism for any future `window.*`-only verdict + for the screenshot backstop.)* ## Suite S — Stealth / anti-bot ### S1. Browser tier (StealthyFetcher — the "undetectable" tier) +All rows are **auto-graded** from the post-JS DOM text (no manual screenshot read); the `Extraction` column names the actual marker each parser keys on. + | Site | Signal | Extraction | Aspirational bar | |---|---|---|---| -| `bot.sannysoft.com` | webdriver/chrome/plugins/UA leaks | DOM table | 0 fails | -| `bot.incolumitas.com` | 30+ checks incl. behavioral | JSON-in-body | ≤ `{WEBDRIVER, connectionRTT}` | -| `browserscan.net/bot-detection` | WebDriver/CDP/Navigator | DOM text | 0 Abnormal | -| `deviceandbrowserinfo.com/are_you_a_bot` | fingerprint + behavioral | JSON `isBot` | `isBot=false` | -| `abrahamjuliot.github.io/creepjs` | fingerprint **consistency/lies**, headless% | `window.Fingerprint` (page_action) | headless ≤30%, stealth ≤30% | -| `fingerprint-scan.com` | Castle.js bot-risk + headless signals | DOM node + evaluate | low risk; headless signals 0 | -| `demo.fingerprint.com/web-scraping` | **behavioral block** (click→blocked vs flights) | page_action click + DOM | not blocked | -| `recaptcha-demo.appspot.com/...v3-request-scores.php` | Google server-verified human score | `wait_for_function` + regex (`recaptcha_score.py:22–28`) | score ≥0.7 | +| `bot.sannysoft.com` | webdriver/chrome/plugins/UA leaks | `class="failed"` cell count | 0 fails | +| `bot.incolumitas.com` | 30+ checks incl. behavioral + IP class | `"":"FAIL"` keys, `is_datacenter` | 0 fpscanner FAIL | +| `browserscan.net/bot-detection` | WebDriver/CDP/Navigator | `Normal`/`Abnormal` count | 0 Abnormal | +| `deviceandbrowserinfo.com/are_you_a_bot` | fingerprint + behavioral | `"isBot":` JSON | `isBot=false` | +| `abrahamjuliot.github.io/creepjs` | fingerprint **consistency/lies**, headless% | `N% headless` + boolean tells (`hasHeadlessWorkerUA`…) | headless ≤30%, no tells | +| `fingerprint-scan.com` | Castle.js bot-risk + headless signals | `Bot Risk Score: N/100` | risk < 50 | +| `demo.fingerprint.com/web-scraping` | **behavioral block** (FingerprintJS Pro Smart Signals) | block message (`access denied` / `tampering detected`) | not blocked | +| `recaptcha-demo.appspot.com/...v3-request-scores.php` | Google server-verified human score | server verify `"score":` JSON | score ≥0.7 | +| `www.scrapingcourse.com/cloudflare-challenge` | **Cloudflare challenge** (only row exercising `solve_cloudflare`) | `you bypassed the cloudflare challenge` line | bypassed | +| `iphey.com` | cross-layer fingerprint+IP+geo coherence | masthead `Your Digital Identity Looks ` (async, ~25 s settle) | `Trustworthy` | ### S2. Per-property fingerprint detail (manual/debug — validates `03e` levers directly) @@ -76,31 +82,56 @@ Purpose-built, ToS-safe sandboxes — validates the HTTP vs **DynamicFetcher (JS Assertion style: known expected values (e.g. first book title, quote count per page) so extraction regressions are caught deterministically. +## First baseline results (2026-06-30, headless, rotating residential proxy) + +First real run of the free stack (patchright-Chromium + curl_cffi `impersonate="chrome"` + `anonymous_proxies` rotating residential), captcha solving OFF, Slice-A levers only (`CRAWL_GEOIP_MATCH_ENABLED=false`). **Suite S: 6 PASS / 4 FAIL across the 10 detection sites** (the other 6 rows are informational: TLS, exit IP, 4 browserleaks manual links). + +| Site | Verdict | Observed | +|---|---|---| +| sannysoft | ✅ PASS | 0 failed cells | +| deviceandbrowserinfo | ✅ PASS | `isBot=false` (incl. `isPlaywright=false`, `isAutomatedWithCDP=false`) | +| reCAPTCHA v3 | ✅ PASS | server score **0.9** | +| BrowserScan | ✅ PASS | Test Results: Normal, 0 Abnormal (WebDriver/CDP/Navigator) | +| fingerprint-scan | ✅ PASS | Bot Risk **35/100** (site flags >50) | +| cloudflare_challenge | ✅ PASS | turnstile **solved** → "you bypassed the Cloudflare challenge" | +| **CreepJS** | ❌ FAIL | headless **33%**, **`hasHeadlessWorkerUA: true`** (worker UA leaks `HeadlessChrome`) | +| **incolumitas** | ❌ FAIL | only legacy `fpscanner WEBDRIVER: FAIL`; all modern tests OK | +| **FingerprintJS Pro** | ❌ FAIL | "anti-detect tampering detected, access denied" | +| **iphey** | ❌ FAIL | verdict **Unreliable** | + +**Reading the failures (maps directly to the moat roadmap):** +- **FingerprintJS Pro** = the commercial bar (the documented free-stack ceiling). Needs WebGL/GPU + deeper patches (`03e` Slice B/C) or the deferred paid-unblocker (`03e §8`). Hardest. +- **CreepJS `hasHeadlessWorkerUA`** = the **Web Worker** `navigator.userAgent` still reports `HeadlessChrome` (main-thread UA is clean). Known patchright leak, **plausibly fixable** (worker UA override) → concrete `03e` Slice-B candidate. +- **iphey "Unreliable"** = almost certainly **geoip incoherence** (browser tz/locale default `America/Los_Angeles` vs the rotating proxy's exit geo, because `CRAWL_GEOIP_MATCH_ENABLED` is default-off in Slice A). iphey is now a **live regression test for the `03e` geoip-coherence lever** — flipping `CRAWL_GEOIP_MATCH_ENABLED` is the expected fix to validate. +- **incolumitas** = a single 2017-era `fpscanner WEBDRIVER` check that even some real browsers trip; modern checks (webdriverPresent, SELENIUM_DRIVER, HEADCHR_*, CDP) all OK. Lowest priority. + +**Proxy-quality note:** incolumitas's IP classifier returned `is_datacenter: true` on one rotation and `false` on another — the `anonymous_proxies` rotating pool **mixes datacenter and residential exits**, a real undetectability variable independent of our code (input to future proxy-provider evaluation). + ## Scorecard & thresholds (resolved) -- **Adopt CloakBrowser's bars as aspirational targets** (above), but the harness's primary output is **our actual measured numbers recorded as the baseline** — a committed `scorecard.md`/JSON snapshot per run (date, tier, proxy on/off, headed/headless, per-site result). Subsequent runs diff against the last baseline so we see drift (ours improving, or a WAF tightening). +- **Adopt CloakBrowser's bars as aspirational targets** (above), but the harness's primary output is **our actual measured numbers recorded as the baseline** — a `scorecard.md`/JSON snapshot per run (date, tier, proxy on/off, headed/headless, per-site result), written to the **gitignored** `results/` tree. Subsequent runs diff against the last on-disk baseline so we see drift (ours improving, or a WAF tightening). - Each row reports: site, tier, verdict, numeric (where applicable), PASS/FAIL vs aspirational bar, and screenshot path. - A run is summarized as `passed/total` per suite (like `stealth_test.py:314–317`), **never blocking** anything. ## Harness design -- Lives under dev tooling (e.g. `surfsense_backend/scripts/crawler_testbench/` or `tests/manual/` — not collected by the normal pytest run, since it hits the live internet + needs proxies). A thin CLI mirrors `bin/cloaktest`: `python -m ... [--proxy URL] [--headed] [--headless] [--suite S|E|all] [--no-screenshots]`. +- Lives under the **proprietary boundary** at `surfsense_backend/app/proprietary/web_crawler/testbench/` (moat measurement tooling — not Apache-2; not collected by the normal pytest run, since it hits the live internet + needs proxies). It is moved here as a coherent package because it can't be cleanly *partially* moved (a proprietary Suite S would otherwise back-import generic scaffolding from `scripts/`, a forbidden app→scripts direction). A thin CLI mirrors `bin/cloaktest`: `python -m app.proprietary.web_crawler.testbench [--proxy URL] [--headed] [--suite S|E|all] [--no-screenshots]`. - **Reuse split (important — `crawl_url` is *not* a drop-in here):** - **Suite E** drives the **real `crawl_url`** end-to-end — extraction correctness *is* the production path (auto-ladder + Trafilatura markdown is exactly what we want to assert). - **Suite S** does **not** use `crawl_url`. Two reasons: (1) `crawl_url` auto-ladders and stops at the first `SUCCESS`, so a detection site might be answered by the cheap HTTP tier when we mean to grade the **StealthyFetcher** tier; (2) `crawl_url` returns Trafilatura **markdown** (`_build_result`), but verdict parsing needs the **raw post-JS DOM** (and `window.*` objects need a live page). So Suite S drives the **individual Scrapling fetchers directly**, per tier. - - **Avoid test-vs-prod drift:** Suite S must construct each fetcher from the **same centralized stealth-config builder** `03e` introduces (the single source of truth for `locale`/`timezone_id`/`hide_canvas`/`block_webrtc`/`impersonate`/profile/headed), **not** a hand-rolled kwargs set — otherwise the scorecard grades a browser we don't ship. (`03e` work item: expose that builder so both the crawler and this harness import it.) + - **Avoid test-vs-prod drift:** Suite S must construct the StealthyFetcher tier from the **same centralized stealth-config builder** `03e` shipped — `build_stealthy_kwargs(get_stealth_config())` in `app/proprietary/web_crawler/stealth.py` (the single source of truth for `block_webrtc`/`hide_canvas`/`google_search`/`dns_over_https` + geoip `locale`/`timezone_id`) — **not** a hand-rolled kwargs set, otherwise the scorecard grades a browser we don't ship. The static (HTTP) tier's `impersonate="chrome"` anchor stays hardcoded in the connector (`03a` §2b); `real_chrome`/headed/persistent-profile are Slice B/C and **not** yet in the builder, so the harness exercises today's Slice-A browser. - Outputs: console summary + screenshots + the `scorecard` snapshot. - Runs with the app-wide proxy provider (`03b`) and `03e` levers on, so the scorecard reflects production fetch behavior — **except captcha solving, which Suite S forces OFF** (`CAPTCHA_SOLVING_ENABLED=false`): we measure the *unaided* stealth/score, and it avoids the `03d` injector firing paid solves against the reCAPTCHA-demo / FingerprintJS rows. Captcha solving is exercised separately (functional `03d` test), not in the stealth scorecard. ## The ceiling-decision loop (why this matters) -The scorecard is the evidence base for the moat strategy: re-run it (a) on a cadence and (b) whenever crawls start failing in the wild. When the **hard** rows (FingerprintJS demo, reCAPTCHA-v3 ≥0.7, and any DataDome/Kasada targets) degrade and `03e` levers can't recover them, that's the documented trigger to flip the **deferred paid-unblocker `FetchStrategy`** (`03e §8`). Without this harness, that decision is guesswork. +The scorecard is the evidence base for the moat strategy: re-run it (a) on a cadence and (b) whenever crawls start failing in the wild. When the **hard** rows (FingerprintJS demo, reCAPTCHA-v3 ≥0.7, and any DataDome/Kasada targets) degrade and `03e` levers can't recover them, that's the documented trigger to flip the **deferred paid-unblocker tier** (`03e §8`). Without this harness, that decision is guesswork. ## Config / dependencies - **No new production dependencies** — reuses Scrapling (already pinned) + the crawler. Optionally a tiny dev helper for the scorecard diff (or just stdlib `json`). - Needs **residential proxy creds** (the `03b` env) to be meaningful; document a "no-proxy" mode that still runs but is expected to fail the harder rows (datacenter IP). -- A results dir (gitignored screenshots; committed `scorecard` snapshots). +- A results dir — **entirely gitignored** (run-local scorecards, `latest.json`, screenshots, dumps); kept tracked only via its `.gitignore`. ## Work items diff --git a/surfsense_backend/.env.example b/surfsense_backend/.env.example index 66e0bc49a..c08a696a3 100644 --- a/surfsense_backend/.env.example +++ b/surfsense_backend/.env.example @@ -369,6 +369,22 @@ TURNSTILE_SECRET_KEY= # CAPTCHA_V3_MIN_SCORE=0.7 # CAPTCHA_V3_ACTION=verify +# ===================================================================== +# Stealth hardening (Phase 3e, Slice A) — runtime/config-level levers on the +# stealth browser tier. Consumed by app/proprietary/web_crawler/stealth.py. +# Defaults add no crawl-speed regression and preserve today's behavior. +# Match browser locale/timezone to RESIDENTIAL_PROXY_LOCATION (no exit-IP lookup). +# CRAWL_GEOIP_MATCH_ENABLED=FALSE +# WebRTC respects the proxy (anti local-IP leak). Cheap + safe. +# CRAWL_BLOCK_WEBRTC=TRUE +# Random canvas noise (an unstable canvas hash is itself a tell) — opt-in only. +# CRAWL_HIDE_CANVAS=FALSE +# Send a Google referer so the first hit looks like organic arrival. +# CRAWL_GOOGLE_SEARCH_REFERER=TRUE +# Route DNS via Cloudflare DoH (anti DNS-leak). Adds a DNS round-trip => off by +# default to avoid any speed regression; enable for leak-safety-first setups. +# CRAWL_DNS_OVER_HTTPS=FALSE + # File Parser Service ETL_SERVICE=UNSTRUCTURED or LLAMACLOUD or DOCLING UNSTRUCTURED_API_KEY=Tpu3P0U8iy diff --git a/surfsense_backend/Dockerfile b/surfsense_backend/Dockerfile index 25e186259..fffe2e45d 100644 --- a/surfsense_backend/Dockerfile +++ b/surfsense_backend/Dockerfile @@ -30,6 +30,20 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ libxrender1 \ dos2unix \ git \ + # ── Phase 3e stealth hardening ────────────────────────────────────────── + # Xvfb: virtual framebuffer so the stealth browser can run headful + # (headless=False) without a real display — many WAFs flag headless Chromium. + # Gated at runtime by CRAWL_HEADED_XVFB_ENABLED (Slice B); installed now so + # the image is ready. Real font packages make canvas/emoji/font-enumeration + # fingerprints resemble a real desktop (set proven against Kasada/Akamai). + xvfb \ + fonts-noto-color-emoji \ + fonts-unifont \ + fonts-ipafont-gothic \ + fonts-wqy-zenhei \ + fonts-tlwg-loma-otf \ + fonts-dejavu \ + fonts-liberation \ && rm -rf /var/lib/apt/lists/* RUN which ffmpeg && ffmpeg -version diff --git a/surfsense_backend/app/config/__init__.py b/surfsense_backend/app/config/__init__.py index 7840c2016..0065faa9a 100644 --- a/surfsense_backend/app/config/__init__.py +++ b/surfsense_backend/app/config/__init__.py @@ -1086,6 +1086,38 @@ class Config: CAPTCHA_V3_MIN_SCORE = float(os.getenv("CAPTCHA_V3_MIN_SCORE", "0.7")) CAPTCHA_V3_ACTION = os.getenv("CAPTCHA_V3_ACTION", "verify") + # ===================================================================== + # Phase 3e — Stealth hardening (Slice A): runtime/config-level levers + # layered on Scrapling's patchright-Chromium StealthyFetcher tier. All are + # consumed by the centralized kwargs builder in + # app/proprietary/web_crawler/stealth.py (proprietary — bypass tuning), which + # is the single source of truth imported by the crawler AND the 03f harness + # (no test-vs-prod drift). Defaults preserve today's behavior / + # introduce no crawl-speed regression. See plans/backend/03e-stealth-hardening.md. + # ===================================================================== + # Map the configured proxy region (RESIDENTIAL_PROXY_LOCATION) -> browser + # locale/timezone so the fingerprint coheres with the proxy exit geo. No + # exit-IP lookup (zero added latency); unknown/empty region => skip. + CRAWL_GEOIP_MATCH_ENABLED = ( + os.getenv("CRAWL_GEOIP_MATCH_ENABLED", "FALSE").upper() == "TRUE" + ) + # Force WebRTC to respect the proxy (prevents real-local-IP leak). Cheap + + # safe => default TRUE. + CRAWL_BLOCK_WEBRTC = os.getenv("CRAWL_BLOCK_WEBRTC", "TRUE").upper() == "TRUE" + # Random canvas noise. An UNSTABLE canvas hash is itself a fingerprint tell, + # so default FALSE (opt-in + 03f-validated). See 03e §2. + CRAWL_HIDE_CANVAS = os.getenv("CRAWL_HIDE_CANVAS", "FALSE").upper() == "TRUE" + # Set a Google referer so the first hit looks like organic arrival. + CRAWL_GOOGLE_SEARCH_REFERER = ( + os.getenv("CRAWL_GOOGLE_SEARCH_REFERER", "TRUE").upper() == "TRUE" + ) + # Route DNS via Cloudflare DoH (anti DNS-leak behind proxies). Adds a DNS + # round-trip => default FALSE to honor the "no speed regression" bar; flip on + # when leak-safety outweighs the marginal latency. + CRAWL_DNS_OVER_HTTPS = ( + os.getenv("CRAWL_DNS_OVER_HTTPS", "FALSE").upper() == "TRUE" + ) + # Litellm TTS Configuration TTS_SERVICE = os.getenv("TTS_SERVICE") TTS_SERVICE_API_BASE = os.getenv("TTS_SERVICE_API_BASE") diff --git a/surfsense_backend/app/proprietary/web_crawler/connector.py b/surfsense_backend/app/proprietary/web_crawler/connector.py index e5eeb1494..7604bdc73 100644 --- a/surfsense_backend/app/proprietary/web_crawler/connector.py +++ b/surfsense_backend/app/proprietary/web_crawler/connector.py @@ -34,7 +34,12 @@ from scrapling.engines.toolbelt import is_proxy_error from scrapling.fetchers import AsyncFetcher, DynamicFetcher, StealthyFetcher from app.proprietary.web_crawler.captcha import build_captcha_page_action +from app.proprietary.web_crawler.stealth import ( + build_stealthy_kwargs, + get_stealth_config, +) from app.utils.captcha import captcha_enabled, get_captcha_config +from app.utils.crawl import BlockType, classify_block from app.utils.proxy import get_proxy_url, is_pool_backed logger = logging.getLogger(__name__) @@ -64,6 +69,11 @@ class CrawlOutcome: them off the outcome regardless of crawl SUCCESS (the solver charges per *attempt*). They are populated only by the StealthyFetcher tier when captcha solving is enabled; every other path leaves the defaults (0 / False). + + Phase 3e ``block_type`` is purely *additive* telemetry: the block classifier + labels the last fetched page (Cloudflare / captcha / DataDome / rate-limited + / ...) for tuning + future escalation routing. It does NOT influence the + billable ``SUCCESS`` predicate above. """ status: CrawlOutcomeStatus @@ -72,6 +82,7 @@ class CrawlOutcome: tier: str | None = None captcha_attempts: int = 0 captcha_solved: bool = False + block_type: BlockType = BlockType.UNKNOWN class WebCrawlerConnector: @@ -102,11 +113,17 @@ class WebCrawlerConnector: # billing can read it regardless of crawl SUCCESS. Per-call (not on # ``self``) => safe under concurrent ``crawl_url`` calls. captcha_state: dict[str, Any] = {"attempts": 0, "solved": False} + # Per-call block-classifier telemetry (03e). ``_build_result`` (the one + # place with raw_html + status) classifies each fetched page into here; + # crawl_url stamps it onto every outcome. Additive only — never gates + # SUCCESS. Per-call (not on ``self``) => concurrency-safe. + block_state: dict[str, Any] = {"block_type": BlockType.UNKNOWN} try: if not validators.url(url): return CrawlOutcome( status=CrawlOutcomeStatus.FAILED, error=f"Invalid URL: {url}", + block_type=block_state["block_type"], ) errors: list[str] = [] @@ -120,7 +137,8 @@ class WebCrawlerConnector: try: logger.info(f"[webcrawler] Using Scrapling AsyncFetcher for: {url}") result = await self._run_tier_with_proxy_retry( - "scrapling-static", lambda: self._crawl_with_async_fetcher(url) + "scrapling-static", + lambda: self._crawl_with_async_fetcher(url, block_state), ) if result: self._log_tier_outcome( @@ -131,6 +149,7 @@ class WebCrawlerConnector: status=CrawlOutcomeStatus.SUCCESS, result=result, tier="scrapling-static", + block_type=block_state["block_type"], ) reached_without_content = True errors.append("Scrapling static: empty extraction") @@ -146,7 +165,8 @@ class WebCrawlerConnector: try: logger.info(f"[webcrawler] Using Scrapling DynamicFetcher for: {url}") result = await self._run_tier_with_proxy_retry( - "scrapling-dynamic", lambda: self._crawl_with_dynamic(url) + "scrapling-dynamic", + lambda: self._crawl_with_dynamic(url, block_state), ) if result: self._log_tier_outcome( @@ -157,6 +177,7 @@ class WebCrawlerConnector: status=CrawlOutcomeStatus.SUCCESS, result=result, tier="scrapling-dynamic", + block_type=block_state["block_type"], ) reached_without_content = True errors.append("Scrapling dynamic: empty extraction") @@ -181,7 +202,9 @@ class WebCrawlerConnector: logger.info(f"[webcrawler] Using Scrapling StealthyFetcher for: {url}") result = await self._run_tier_with_proxy_retry( "scrapling-stealthy", - lambda: self._crawl_with_stealthy(url, captcha_state), + lambda: self._crawl_with_stealthy( + url, captcha_state, block_state + ), ) if result: self._log_tier_outcome( @@ -194,6 +217,7 @@ class WebCrawlerConnector: tier="scrapling-stealthy", captcha_attempts=captcha_state["attempts"], captcha_solved=captcha_state["solved"], + block_type=block_state["block_type"], ) reached_without_content = True errors.append("Scrapling stealthy: empty extraction") @@ -219,12 +243,14 @@ class WebCrawlerConnector: error=f"No content extracted for {url}. {'; '.join(errors)}", captcha_attempts=captcha_state["attempts"], captcha_solved=captcha_state["solved"], + block_type=block_state["block_type"], ) return CrawlOutcome( status=CrawlOutcomeStatus.FAILED, error=f"All crawl methods failed for {url}. {'; '.join(errors)}", captcha_attempts=captcha_state["attempts"], captcha_solved=captcha_state["solved"], + block_type=block_state["block_type"], ) except Exception as e: @@ -234,6 +260,7 @@ class WebCrawlerConnector: error=f"Error crawling URL {url}: {e!s}", captcha_attempts=captcha_state["attempts"], captcha_solved=captcha_state["solved"], + block_type=block_state["block_type"], ) async def _run_tier_with_proxy_retry( @@ -307,7 +334,9 @@ class WebCrawlerConnector: total_ms, ) - async def _crawl_with_async_fetcher(self, url: str) -> dict[str, Any] | None: + async def _crawl_with_async_fetcher( + self, url: str, block_state: dict[str, Any] | None = None + ) -> dict[str, Any] | None: """ Crawl URL using Scrapling's AsyncFetcher (static HTTP) + Trafilatura. @@ -331,6 +360,13 @@ class WebCrawlerConnector: status = getattr(page, "status", None) if status is not None and status >= 400: + # 03e: classify here too — this early return skips _build_result, and + # the static tier is the first/cheapest hit, so the 403/429 bot-gate + # (the most common block signal) would otherwise never be labeled. + if block_state is not None: + block_state["block_type"] = classify_block( + status, getattr(page, "html_content", None) + ) logger.info( "%s tier=scrapling-static url=%s fetch_ms=%.1f status=%s outcome=http_error", _PERF, @@ -347,18 +383,25 @@ class WebCrawlerConnector: allow_raw_fallback=False, fetch_ms=fetch_ms, status=status, + block_state=block_state, ) - async def _crawl_with_dynamic(self, url: str) -> dict[str, Any] | None: + async def _crawl_with_dynamic( + self, url: str, block_state: dict[str, Any] | None = None + ) -> dict[str, Any] | None: """ Crawl URL using Scrapling's DynamicFetcher (full browser) + Trafilatura. Runs the sync fetch in a worker thread so it works on any event loop, including Windows ``SelectorEventLoop`` which cannot spawn subprocesses. """ - return await asyncio.to_thread(self._crawl_with_dynamic_sync, url) + return await asyncio.to_thread( + self._crawl_with_dynamic_sync, url, block_state + ) - def _crawl_with_dynamic_sync(self, url: str) -> dict[str, Any] | None: + def _crawl_with_dynamic_sync( + self, url: str, block_state: dict[str, Any] | None = None + ) -> dict[str, Any] | None: """Synchronous DynamicFetcher crawl executed in a worker thread.""" fetch_start = time.perf_counter() page = DynamicFetcher.fetch( @@ -376,10 +419,14 @@ class WebCrawlerConnector: allow_raw_fallback=False, fetch_ms=fetch_ms, status=getattr(page, "status", None), + block_state=block_state, ) async def _crawl_with_stealthy( - self, url: str, captcha_state: dict[str, Any] | None = None + self, + url: str, + captcha_state: dict[str, Any] | None = None, + block_state: dict[str, Any] | None = None, ) -> dict[str, Any] | None: """ Crawl URL using Scrapling's StealthyFetcher (patchright-Chromium) + Trafilatura. @@ -390,13 +437,18 @@ class WebCrawlerConnector: ``captcha_state`` (03d) is mutated in place by the captcha page_action (attempts/solved) so ``crawl_url`` can surface it on the outcome. + ``block_state`` (03e) is populated by ``_build_result`` with the block + classification of the fetched page. """ return await asyncio.to_thread( - self._crawl_with_stealthy_sync, url, captcha_state + self._crawl_with_stealthy_sync, url, captcha_state, block_state ) def _crawl_with_stealthy_sync( - self, url: str, captcha_state: dict[str, Any] | None = None + self, + url: str, + captcha_state: dict[str, Any] | None = None, + block_state: dict[str, Any] | None = None, ) -> dict[str, Any] | None: """Synchronous StealthyFetcher crawl executed in a worker thread.""" fetch_start = time.perf_counter() @@ -425,6 +477,11 @@ class WebCrawlerConnector: "solve_cloudflare": True, "proxy": proxy, } + # 03e Slice A: merge config-driven stealth levers (block_webrtc, + # hide_canvas, google_search, dns_over_https, geoip locale/timezone). + # Keys never collide with the core kwargs above; defaults preserve + # today's behavior and add no crawl-speed regression. + fetch_kwargs.update(build_stealthy_kwargs(get_stealth_config())) if page_action is not None: fetch_kwargs["page_action"] = page_action page = StealthyFetcher.fetch(url, **fetch_kwargs) @@ -436,6 +493,7 @@ class WebCrawlerConnector: allow_raw_fallback=True, fetch_ms=fetch_ms, status=getattr(page, "status", None), + block_state=block_state, ) def _build_result( @@ -447,6 +505,7 @@ class WebCrawlerConnector: allow_raw_fallback: bool, fetch_ms: float | None = None, status: int | None = None, + block_state: dict[str, Any] | None = None, ) -> dict[str, Any] | None: """ Extract markdown + metadata from raw HTML using Trafilatura. @@ -465,6 +524,12 @@ class WebCrawlerConnector: Returns: Result dict (content/metadata/crawler_type) or ``None``. """ + # 03e: classify the fetched page (additive telemetry/routing only — never + # gates SUCCESS). Done before the early returns so EMPTY/no-extraction + # pages still get labeled. Last tier to fetch wins in block_state. + if block_state is not None: + block_state["block_type"] = classify_block(status, raw_html) + html_len = len(raw_html) if raw_html else 0 if not raw_html or len(raw_html.strip()) == 0: diff --git a/surfsense_backend/app/proprietary/web_crawler/stealth.py b/surfsense_backend/app/proprietary/web_crawler/stealth.py new file mode 100644 index 000000000..25b48d69b --- /dev/null +++ b/surfsense_backend/app/proprietary/web_crawler/stealth.py @@ -0,0 +1,216 @@ +# SurfSense proprietary crawler engine. +# +# This module is part of the ``app.proprietary`` package and is licensed +# SEPARATELY from the Apache-2.0 project root. See ``app/proprietary/LICENSE``. +# Do not relicense or redistribute this file under Apache-2.0. +"""Stealth-hardening config + the centralized per-tier kwargs builder (Phase 3e). + +This is **bypass-specific tuning** (hence proprietary): it decides the +anti-detection lever set the StealthyFetcher tier runs with (WebRTC/canvas/DNS +handling, organic-referer, and proxy-geo-coherent ``locale``/``timezone_id``). +It is the **single source of truth** that turns config -> ``StealthyFetcher`` +keyword arguments, imported by both the crawler (``connector.py``) and the 03f +manual harness, so the scorecard grades the exact browser we ship (no +test-vs-prod drift). + +The generic, vendor-agnostic block *classifier* (passive telemetry, public +markers) stays Apache-2.0 in ``app/utils/crawl/``. The further bypass logic +(WebGL spoof JS, humanize choreography) is deferred to later 03e slices and will +also live here. + +Defaults preserve today's behavior and add **no crawl-speed regression**: +``dns_over_https`` (the only lever with a latency cost) is off unless explicitly +enabled, and geoip resolution is a pure in-process dict lookup (no exit-IP call). +""" + +from dataclasses import dataclass +from typing import Any + +from app.config import config + +# Coarse region -> (locale, IANA timezone) map. Country granularity only: per the +# 03e "Geoip accuracy" risk, wrong-but-coherent beats a default mismatch and +# per-city precision isn't worth it. Keyed by ISO-3166 alpha-2; common full names +# are aliased below. Unknown/empty => (None, None) => leave Scrapling's default. +_REGION_TO_LOCALE_TZ: dict[str, tuple[str, str]] = { + "us": ("en-US", "America/New_York"), + "ca": ("en-CA", "America/Toronto"), + "gb": ("en-GB", "Europe/London"), + "ie": ("en-IE", "Europe/Dublin"), + "au": ("en-AU", "Australia/Sydney"), + "nz": ("en-NZ", "Pacific/Auckland"), + "de": ("de-DE", "Europe/Berlin"), + "fr": ("fr-FR", "Europe/Paris"), + "es": ("es-ES", "Europe/Madrid"), + "it": ("it-IT", "Europe/Rome"), + "nl": ("nl-NL", "Europe/Amsterdam"), + "be": ("nl-BE", "Europe/Brussels"), + "ch": ("de-CH", "Europe/Zurich"), + "at": ("de-AT", "Europe/Vienna"), + "se": ("sv-SE", "Europe/Stockholm"), + "no": ("nb-NO", "Europe/Oslo"), + "dk": ("da-DK", "Europe/Copenhagen"), + "fi": ("fi-FI", "Europe/Helsinki"), + "pl": ("pl-PL", "Europe/Warsaw"), + "pt": ("pt-PT", "Europe/Lisbon"), + "ru": ("ru-RU", "Europe/Moscow"), + "ua": ("uk-UA", "Europe/Kyiv"), + "tr": ("tr-TR", "Europe/Istanbul"), + "in": ("en-IN", "Asia/Kolkata"), + "jp": ("ja-JP", "Asia/Tokyo"), + "kr": ("ko-KR", "Asia/Seoul"), + "cn": ("zh-CN", "Asia/Shanghai"), + "hk": ("zh-HK", "Asia/Hong_Kong"), + "tw": ("zh-TW", "Asia/Taipei"), + "sg": ("en-SG", "Asia/Singapore"), + "id": ("id-ID", "Asia/Jakarta"), + "ph": ("en-PH", "Asia/Manila"), + "th": ("th-TH", "Asia/Bangkok"), + "vn": ("vi-VN", "Asia/Ho_Chi_Minh"), + "ae": ("ar-AE", "Asia/Dubai"), + "il": ("he-IL", "Asia/Jerusalem"), + "za": ("en-ZA", "Africa/Johannesburg"), + "br": ("pt-BR", "America/Sao_Paulo"), + "mx": ("es-MX", "America/Mexico_City"), + "ar": ("es-AR", "America/Argentina/Buenos_Aires"), + "cl": ("es-CL", "America/Santiago"), +} + +# Full-name / synonym aliases -> alpha-2 key above. +_REGION_ALIASES: dict[str, str] = { + "usa": "us", + "united states": "us", + "united states of america": "us", + "america": "us", + "uk": "gb", + "united kingdom": "gb", + "great britain": "gb", + "england": "gb", + "canada": "ca", + "australia": "au", + "germany": "de", + "deutschland": "de", + "france": "fr", + "spain": "es", + "italy": "it", + "netherlands": "nl", + "holland": "nl", + "sweden": "se", + "norway": "no", + "denmark": "dk", + "finland": "fi", + "poland": "pl", + "portugal": "pt", + "russia": "ru", + "ukraine": "ua", + "turkey": "tr", + "india": "in", + "japan": "jp", + "korea": "kr", + "south korea": "kr", + "china": "cn", + "hong kong": "hk", + "taiwan": "tw", + "singapore": "sg", + "indonesia": "id", + "philippines": "ph", + "thailand": "th", + "vietnam": "vn", + "israel": "il", + "south africa": "za", + "brazil": "br", + "brasil": "br", + "mexico": "mx", + "argentina": "ar", + "chile": "cl", +} + + +def location_to_locale_timezone( + location: str | None, +) -> tuple[str | None, str | None]: + """Map a free-form proxy region string -> (locale, timezone_id). + + ``location`` is the vendor-specific ``RESIDENTIAL_PROXY_LOCATION`` value + (``03b``). Best-effort: matches an ISO-3166 alpha-2 code (e.g. ``"us"``) or a + common country name (e.g. ``"Germany"``); only the leading token is honored, + so vendor strings like ``"us:nyc"`` or ``"de region"`` still resolve. Returns + ``(None, None)`` for empty/unknown input (caller then leaves the browser + default). + """ + if not location: + return (None, None) + + normalized = location.strip().lower() + if not normalized: + return (None, None) + + # Direct alpha-2 hit. + if normalized in _REGION_TO_LOCALE_TZ: + return _REGION_TO_LOCALE_TZ[normalized] + + # Full-name / synonym hit. + if normalized in _REGION_ALIASES: + return _REGION_TO_LOCALE_TZ[_REGION_ALIASES[normalized]] + + # Leading token (handles "us:nyc", "de-rotating", "united states (east)"). + head = normalized.replace("-", " ").replace("_", " ").replace(":", " ").split() + if head: + token = head[0] + if token in _REGION_TO_LOCALE_TZ: + return _REGION_TO_LOCALE_TZ[token] + if token in _REGION_ALIASES: + return _REGION_TO_LOCALE_TZ[_REGION_ALIASES[token]] + + return (None, None) + + +@dataclass(frozen=True) +class StealthConfig: + """Immutable snapshot of the Phase 3e Slice-A stealth levers for one crawl.""" + + geoip_match_enabled: bool + proxy_location: str + block_webrtc: bool + hide_canvas: bool + google_search: bool + dns_over_https: bool + + +def get_stealth_config() -> StealthConfig: + """Build a :class:`StealthConfig` from the current process config/env.""" + return StealthConfig( + geoip_match_enabled=config.CRAWL_GEOIP_MATCH_ENABLED, + proxy_location=config.RESIDENTIAL_PROXY_LOCATION or "", + block_webrtc=config.CRAWL_BLOCK_WEBRTC, + hide_canvas=config.CRAWL_HIDE_CANVAS, + google_search=config.CRAWL_GOOGLE_SEARCH_REFERER, + dns_over_https=config.CRAWL_DNS_OVER_HTTPS, + ) + + +def build_stealthy_kwargs(cfg: StealthConfig) -> dict[str, Any]: + """Return the config-derived ``StealthyFetcher.fetch`` keyword arguments. + + Only the Slice-A stealth levers are returned; the caller merges these into + the tier's own kwargs (``headless``/``network_idle``/``block_ads``/ + ``solve_cloudflare``/``proxy``/captcha ``page_action``), which never collide + with the keys here. ``locale``/``timezone_id`` are added only when geoip + matching is enabled *and* the proxy region resolves — otherwise the browser + keeps its system default. + """ + kwargs: dict[str, Any] = { + "block_webrtc": cfg.block_webrtc, + "hide_canvas": cfg.hide_canvas, + "google_search": cfg.google_search, + "dns_over_https": cfg.dns_over_https, + } + + if cfg.geoip_match_enabled: + locale, timezone_id = location_to_locale_timezone(cfg.proxy_location) + if locale: + kwargs["locale"] = locale + if timezone_id: + kwargs["timezone_id"] = timezone_id + + return kwargs diff --git a/surfsense_backend/app/proprietary/web_crawler/testbench/README.md b/surfsense_backend/app/proprietary/web_crawler/testbench/README.md new file mode 100644 index 000000000..e0bb4cc71 --- /dev/null +++ b/surfsense_backend/app/proprietary/web_crawler/testbench/README.md @@ -0,0 +1,83 @@ +# Crawler testbench — manual undetectability & extraction scorecard (Phase 3f) + +A **manual, repeatable scorecard** for the Universal WebURL Crawler. It answers +one question with evidence: *how undetectable (and how correct) is the crawler +right now?* — so we know when the free-stack ceiling (`03e`) is reached and the +deferred paid-unblocker tier is worth flipping. + +This is **dev/operator tooling**, not a product code path and **not** part of the +pytest suite (it hits the live internet and needs proxy creds). See +`plans/backend/03f-undetectability-testing.md` for the full design. + +## What it measures (two axes, two suites) + +- **Suite S — stealth / anti-bot:** drives the real `StealthyFetcher` tier + (built from the **same** `app/proprietary/web_crawler/stealth.py` builder the + crawler ships, so no test-vs-prod drift) against the standard bot-detection + sites — sannysoft, deviceandbrowserinfo, reCAPTCHA v3, CreepJS, BrowserScan, + incolumitas, fingerprint-scan, FingerprintJS Pro, a **Cloudflare-challenge + canary** (the only row that exercises `solve_cloudflare`), and iphey — plus the + HTTP-tier TLS fingerprint (`tls.peet.ws`), exit-IP echo, and manual + per-property links (browserleaks). Every detection site is **auto-graded** from + its real DOM verdict (no screenshot reading required). +- **Suite E — extraction correctness:** drives the real `crawl_url` ladder + against ToS-safe scraping sandboxes (`toscrape`, `scrapethissite`) and asserts + known content appears in the extracted markdown. + +## Run it + +From the backend directory (`surfsense_backend/`): + +```bash +uv run python -m app.proprietary.web_crawler.testbench --suite all +# or +.\.venv\Scripts\python.exe -m app.proprietary.web_crawler.testbench --suite all +``` + +Flags: + +| Flag | Meaning | +|---|---| +| `--suite S\|E\|all` | Which suite(s) to run (default `all`). | +| `--proxy URL` | Override the proxy for Suite S (default: the app proxy provider from `.env`). | +| `--headed` | Run the browser tier headful (`headless=False`). | +| `--no-screenshots` | Skip per-site screenshots. | + +Captcha solving is **forced OFF** every run — Suite S measures the *unaided* +stealth score. + +## Reading the output + +Each row prints `PASS / FAIL / ERR / INFO` with the tier and (where one exists) a +numeric. The harness is **tolerant**: a parse miss is `ERROR`, never a crash. +Detection sites are auto-graded from their DOM verdict; `INFO` is reserved for +purely informational rows (TLS fingerprint, exit IP) and the manual +per-property links (browserleaks canvas/webgl/fonts/webrtc). Screenshots are +still captured for every browser row as a backstop. + +Outputs land in `results/`, which is **entirely gitignored** (run-local): + +- `scorecard-.json` / `.md` — timestamped scorecard + readable report. +- `latest.json` — convenience pointer. +- `screenshots/` — per-site full-page captures. + +Every run prints **drift vs the last baseline** (the previous on-disk scorecard) +so you can see the trend (our stealth improving, or a WAF tightening). + +## The aspirational bars (from CloakBrowser, recorded as targets) + +`sannysoft` 0 failed cells · `deviceandbrowserinfo` isBot=false · +reCAPTCHA v3 `>= 0.7` · CreepJS headless `<= 30%`, no spoof tells · FingerprintJS +demo not blocked · Cloudflare challenge bypassed · iphey `Trustworthy`. We +**record our actual numbers as the baseline** — these are targets, not build +gates (the run never blocks anything). + +## Caveats + +- **Proxy required for realism.** Without residential egress the hard rows fail + by design (datacenter IP); a red scorecard from no-proxy is expected. +- **Sites change.** Detection sites move their DOM; if an auto-parser starts + returning `INFO`/`ERROR`, read the screenshot and (optionally) tighten the + parser — each site is one entry in `suite_stealth.py`. +- **Not a guarantee.** Passing sannysoft/CreepJS ≠ beating DataDome/Kasada; the + value is *trend + ceiling visibility*, not a green checkmark. diff --git a/surfsense_backend/app/proprietary/web_crawler/testbench/__init__.py b/surfsense_backend/app/proprietary/web_crawler/testbench/__init__.py new file mode 100644 index 000000000..40c2edae1 --- /dev/null +++ b/surfsense_backend/app/proprietary/web_crawler/testbench/__init__.py @@ -0,0 +1,25 @@ +# SurfSense proprietary crawler engine. +# +# This module is part of the ``app.proprietary`` package and is licensed +# SEPARATELY from the Apache-2.0 project root. See ``app/proprietary/LICENSE``. +# Do not relicense or redistribute this file under Apache-2.0. +"""Phase 3f — manual undetectability & extraction test harness (dev tooling). + +A repeatable, **manual** scorecard that grades the Universal WebURL Crawler on two +axes: + +- **Suite S (stealth / anti-bot):** drives the real Scrapling tiers against the + industry-standard bot-detection + fingerprint sites and parses each verdict. +- **Suite E (extraction correctness):** drives the real ``crawl_url`` ladder + against ToS-safe scraping sandboxes and asserts known values. + +It is **not** collected by pytest (it hits the live internet and needs proxy +creds). Run it from the backend directory: + + uv run python -m app.proprietary.web_crawler.testbench --suite all + # or: .\\.venv\\Scripts\\python.exe -m app.proprietary.web_crawler.testbench --suite all + +See ``README.md`` for the runbook and ``plans/backend/03f-undetectability-testing.md`` +for the design. This package is dev/operator tooling, untouched by the product +rename, and only *measures* what 03a–03e built. +""" diff --git a/surfsense_backend/app/proprietary/web_crawler/testbench/__main__.py b/surfsense_backend/app/proprietary/web_crawler/testbench/__main__.py new file mode 100644 index 000000000..823648ecb --- /dev/null +++ b/surfsense_backend/app/proprietary/web_crawler/testbench/__main__.py @@ -0,0 +1,141 @@ +# SurfSense proprietary crawler engine. +# +# This module is part of the ``app.proprietary`` package and is licensed +# SEPARATELY from the Apache-2.0 project root. See ``app/proprietary/LICENSE``. +# Do not relicense or redistribute this file under Apache-2.0. +"""CLI for the 03f manual scorecard. Run from the backend directory: + + uv run python -m app.proprietary.web_crawler.testbench --suite all + uv run python -m app.proprietary.web_crawler.testbench --suite S --headed + uv run python -m app.proprietary.web_crawler.testbench --suite E --no-screenshots + +Mirrors CloakBrowser's ``bin/cloaktest`` ergonomics. Writes a timestamped +scorecard (JSON + markdown) under ``results/`` and diffs against the last +baseline. Captcha solving is forced OFF so the scorecard measures the *unaided* +stealth ceiling (03f §Harness). +""" + +from __future__ import annotations + +import argparse +import asyncio +import os +import sys +from pathlib import Path + +from dotenv import load_dotenv + +# --- bootstrap: load .env + put backend root on sys.path BEFORE importing app.* --- +# __file__ = app/proprietary/web_crawler/testbench/__main__.py -> backend root is 4 up. +_BACKEND_ROOT = Path(__file__).resolve().parents[4] +if str(_BACKEND_ROOT) not in sys.path: + sys.path.insert(0, str(_BACKEND_ROOT)) +for _candidate in (_BACKEND_ROOT / ".env", _BACKEND_ROOT.parent / ".env"): + if _candidate.exists(): + load_dotenv(_candidate) + break + +# Force captcha solving OFF: Suite S measures the unaided score and we never want +# the 03d injector firing paid solves against the reCAPTCHA-demo row. Must be set +# before app.config is imported (config snapshots env at import). +os.environ["CAPTCHA_SOLVING_ENABLED"] = "false" + + +def _parse_args(argv: list[str]) -> argparse.Namespace: + p = argparse.ArgumentParser( + prog="python -m app.proprietary.web_crawler.testbench", + description="Manual undetectability & extraction scorecard (Phase 3f).", + ) + p.add_argument( + "--suite", + choices=["S", "E", "all"], + default="all", + help="S=stealth/anti-bot, E=extraction correctness, all=both.", + ) + p.add_argument( + "--proxy", + default=None, + help="Override proxy URL for Suite S (default: the app proxy provider).", + ) + p.add_argument( + "--headed", + action="store_true", + help="Run the browser tier headful (headless=False).", + ) + p.add_argument( + "--no-screenshots", + action="store_true", + help="Skip per-site screenshots (faster, no results/screenshots writes).", + ) + return p.parse_args(argv) + + +async def _amain(args: argparse.Namespace) -> int: + # Imports happen here (after bootstrap + env override) so app.config sees the + # forced CAPTCHA_SOLVING_ENABLED=false and the resolved .env. + from app.utils.proxy import get_proxy_url + + from .core import ( + RunMeta, + diff_against_baseline, + load_last_baseline, + render_console, + scrapling_version, + write_scorecard, + ) + + proxy = args.proxy or get_proxy_url() + screenshots = not args.no_screenshots + results = [] + + print( + f"== crawler scorecard == suite={args.suite} " + f"headed={args.headed} proxy={'set' if proxy else 'NONE'} " + f"screenshots={screenshots}" + ) + if not proxy: + print( + " WARNING: no proxy configured — the hard anti-bot rows are expected " + "to fail from a datacenter IP (see README)." + ) + + baseline = load_last_baseline() + + if args.suite in ("S", "all"): + from .suite_stealth import run_suite_s + + results += await run_suite_s( + proxy=proxy, headed=args.headed, screenshots=screenshots + ) + + if args.suite in ("E", "all"): + from .suite_extraction import run_suite_e + + results += await run_suite_e() + + render_console(results) + + meta = RunMeta.now( + suites=args.suite, + proxy=proxy, + headed=args.headed, + scrapling_version=scrapling_version(), + ) + json_path, md_path = write_scorecard(results, meta) + + print("\n--- drift vs last baseline ---") + for line in diff_against_baseline(results, baseline): + print(f" {line}") + + print(f"\nscorecard JSON: {json_path}") + print(f"scorecard MD: {md_path}") + return 0 + + +def main() -> None: + args = _parse_args(sys.argv[1:]) + raise SystemExit(asyncio.run(_amain(args))) + + +if __name__ == "__main__": + main() diff --git a/surfsense_backend/app/proprietary/web_crawler/testbench/core.py b/surfsense_backend/app/proprietary/web_crawler/testbench/core.py new file mode 100644 index 000000000..e201b74d8 --- /dev/null +++ b/surfsense_backend/app/proprietary/web_crawler/testbench/core.py @@ -0,0 +1,325 @@ +# SurfSense proprietary crawler engine. +# +# This module is part of the ``app.proprietary`` package and is licensed +# SEPARATELY from the Apache-2.0 project root. See ``app/proprietary/LICENSE``. +# Do not relicense or redistribute this file under Apache-2.0. +"""Shared primitives for the 03f scorecard: result model, the closure-cell +``page_action`` verdict-extractor (the 03d-shared mechanism), and the scorecard +snapshot writer / baseline-differ / console renderer. + +Stdlib-only on purpose (the plan's "no new prod dependency" bar). Everything here +is tolerant: a parse miss yields an ``ERROR`` row, never a crash — detection sites +change their DOM and the harness is manual. +""" + +from __future__ import annotations + +import json +from collections.abc import Callable +from dataclasses import asdict, dataclass +from datetime import datetime, timezone +from enum import Enum +from pathlib import Path +from typing import Any +from urllib.parse import urlsplit + +# Results live next to the harness; screenshots are gitignored, scorecard JSON is +# committed so runs diff against the last baseline (see README + plan §Scorecard). +_PKG_DIR = Path(__file__).resolve().parent +RESULTS_DIR = _PKG_DIR / "results" +SCREENSHOTS_DIR = RESULTS_DIR / "screenshots" + + +class CheckStatus(str, Enum): + """Outcome of a single scorecard row.""" + + PASS = "PASS" # met the aspirational bar + FAIL = "FAIL" # ran, did not meet the bar + ERROR = "ERROR" # could not run / parse (never fatal to the run) + INFO = "INFO" # recorded, no pass/fail bar (e.g. TLS JA3, manual links) + SKIP = "SKIP" # intentionally not run this invocation + + +@dataclass +class CheckResult: + """One row of the scorecard.""" + + suite: str # "S" (stealth) | "E" (extraction) + name: str # site / check key + tier: str # scrapling-static | scrapling-stealthy | crawl_url | n/a + status: CheckStatus + bar: str # human-readable aspirational threshold + detail: str = "" # short human summary of what was observed + numeric: float | None = None # comparable metric where one exists + screenshot: str | None = None # path, when captured + + def to_row(self) -> dict[str, Any]: + d = asdict(self) + d["status"] = self.status.value + return d + + +@dataclass +class RunMeta: + """Provenance for a scorecard snapshot so baselines are comparable.""" + + timestamp: str + suites: str + proxy: str # masked + headed: bool + scrapling_version: str + captcha_disabled: bool = True + notes: str = "" + + @staticmethod + def now( + *, suites: str, proxy: str | None, headed: bool, scrapling_version: str + ) -> RunMeta: + return RunMeta( + timestamp=datetime.now(timezone.utc).isoformat(timespec="seconds"), + suites=suites, + proxy=mask_proxy(proxy), + headed=headed, + scrapling_version=scrapling_version, + ) + + +# --- closure-cell page_action (the mechanism shared with 03d) ---------------- + + +@dataclass +class EvalCell: + """A mutable cell a ``page_action`` writes its ``page.evaluate`` result into. + + Scrapling discards a ``page_action``'s return value, so the only way to get a + JS-object verdict (CreepJS ``window.Fingerprint``, a Castle.js score node) out + of the browser is to stash it in a closure variable. This is the exact same + plumbing 03d's captcha-token injector uses — factored once here. + """ + + value: Any = None + error: str | None = None + captured: bool = False + + +def make_page_action( + *, + evaluate_js: str | None = None, + screenshot_path: str | None = None, + pre_wait_ms: int = 0, +) -> tuple[Callable[[Any], Any], EvalCell]: + """Build a sync ``page_action`` + the :class:`EvalCell` it writes into. + + The action (in priority order) optionally sleeps ``pre_wait_ms`` (lets async + scores settle), optionally evaluates ``evaluate_js`` into the cell, and + optionally writes a full-page screenshot. Each step is independently guarded + so one failure (e.g. a site without the JS object) never aborts the fetch. + Returns the page unchanged (Scrapling re-reads its DOM afterwards). + """ + cell = EvalCell() + + def _action(page: Any) -> Any: + if pre_wait_ms > 0: + try: + page.wait_for_timeout(pre_wait_ms) + except Exception: + pass + if evaluate_js is not None: + try: + cell.value = page.evaluate(evaluate_js) + cell.captured = True + except Exception as exc: # noqa: BLE001 - tolerant by design + cell.error = f"{type(exc).__name__}: {exc}" + if screenshot_path is not None: + try: + # The screenshot dir must exist *now* (the page action runs mid-suite, + # before the end-of-run ``ensure_dirs``); Scrapling/patchright will not + # create it for us, so a missing dir silently drops every screenshot. + Path(screenshot_path).parent.mkdir(parents=True, exist_ok=True) + page.screenshot(path=screenshot_path, full_page=True) + except Exception as exc: # noqa: BLE001 - tolerant, but surface the cause + if cell.error is None: + cell.error = f"screenshot: {type(exc).__name__}: {exc}" + return page + + return _action, cell + + +# --- helpers ----------------------------------------------------------------- + + +def mask_proxy(url: str | None) -> str: + """Mask credentials in a proxy URL for logs/snapshots.""" + if not url: + return "" + try: + p = urlsplit(url) + host = p.hostname or "?" + port = f":{p.port}" if p.port else "" + return f"{p.scheme}://***@{host}{port}" + except Exception: + return "" + + +def ensure_dirs() -> None: + RESULTS_DIR.mkdir(parents=True, exist_ok=True) + SCREENSHOTS_DIR.mkdir(parents=True, exist_ok=True) + + +def scrapling_version() -> str: + try: + import scrapling # noqa: PLC0415 + + return getattr(scrapling, "__version__", "unknown") + except Exception: + return "unavailable" + + +# --- scorecard I/O ----------------------------------------------------------- + + +def write_scorecard( + results: list[CheckResult], meta: RunMeta +) -> tuple[Path, Path]: + """Write the JSON snapshot (committed, baseline) + a readable markdown report. + + The JSON filename is timestamped so prior runs are kept as the baseline trail; + ``latest.json`` is overwritten as the convenience pointer used by the differ. + """ + ensure_dirs() + stamp = meta.timestamp.replace(":", "").replace("-", "") + payload = { + "meta": asdict(meta), + "summary": summarize(results), + "results": [r.to_row() for r in results], + } + json_path = RESULTS_DIR / f"scorecard-{stamp}.json" + json_path.write_text(json.dumps(payload, indent=2), encoding="utf-8") + (RESULTS_DIR / "latest.json").write_text( + json.dumps(payload, indent=2), encoding="utf-8" + ) + md_path = RESULTS_DIR / f"scorecard-{stamp}.md" + md_path.write_text(_render_markdown(results, meta), encoding="utf-8") + return json_path, md_path + + +def load_last_baseline() -> dict[str, Any] | None: + """Load the most recent prior scorecard JSON (excluding ``latest.json``).""" + if not RESULTS_DIR.exists(): + return None + snaps = sorted(RESULTS_DIR.glob("scorecard-*.json")) + if not snaps: + return None + try: + return json.loads(snaps[-1].read_text(encoding="utf-8")) + except Exception: + return None + + +def summarize(results: list[CheckResult]) -> dict[str, dict[str, int]]: + """Per-suite ``{passed, failed, error, info, total}`` counts.""" + out: dict[str, dict[str, int]] = {} + for r in results: + s = out.setdefault( + r.suite, + {"passed": 0, "failed": 0, "error": 0, "info": 0, "skip": 0, "total": 0}, + ) + s["total"] += 1 + key = { + CheckStatus.PASS: "passed", + CheckStatus.FAIL: "failed", + CheckStatus.ERROR: "error", + CheckStatus.INFO: "info", + CheckStatus.SKIP: "skip", + }[r.status] + s[key] += 1 + return out + + +def diff_against_baseline( + results: list[CheckResult], baseline: dict[str, Any] | None +) -> list[str]: + """Human-readable drift lines comparing this run to the last baseline.""" + if not baseline: + return ["(no prior baseline — this run becomes the baseline)"] + prior = {(r["suite"], r["name"]): r for r in baseline.get("results", [])} + lines: list[str] = [] + for r in results: + was = prior.get((r.suite, r.name)) + if was is None: + lines.append(f"+ NEW [{r.suite}] {r.name}: {r.status.value}") + continue + if was["status"] != r.status.value: + lines.append( + f"~ {r.name}: status {was['status']} -> {r.status.value}" + ) + old_n, new_n = was.get("numeric"), r.numeric + if old_n is not None and new_n is not None and abs(old_n - new_n) > 1e-9: + lines.append(f"~ {r.name}: numeric {old_n} -> {new_n}") + seen = {(r.suite, r.name) for r in results} + for key, was in prior.items(): + if key not in seen: + lines.append(f"- GONE [{key[0]}] {key[1]} (was {was['status']})") + return lines or ["(no changes vs last baseline)"] + + +# --- rendering --------------------------------------------------------------- + +_ICON = { + CheckStatus.PASS: "PASS", + CheckStatus.FAIL: "FAIL", + CheckStatus.ERROR: "ERR ", + CheckStatus.INFO: "INFO", + CheckStatus.SKIP: "SKIP", +} + + +def render_console(results: list[CheckResult]) -> None: + """Print the grouped scorecard + per-suite totals to stdout.""" + by_suite: dict[str, list[CheckResult]] = {} + for r in results: + by_suite.setdefault(r.suite, []).append(r) + + for suite in sorted(by_suite): + label = "Suite S - stealth/anti-bot" if suite == "S" else ( + "Suite E - extraction" if suite == "E" else f"Suite {suite}" + ) + print(f"\n=== {label} ===") + for r in by_suite[suite]: + num = f" [{r.numeric:g}]" if r.numeric is not None else "" + print(f" {_ICON[r.status]} {r.name:<34} {r.tier:<18}{num}") + if r.detail: + print(f" {r.detail}") + + print("\n--- summary ---") + for suite, s in summarize(results).items(): + print( + f" Suite {suite}: {s['passed']}/{s['total']} passed " + f"(fail={s['failed']} err={s['error']} info={s['info']})" + ) + + +def _render_markdown(results: list[CheckResult], meta: RunMeta) -> str: + lines = [ + f"# Crawler scorecard — {meta.timestamp}", + "", + f"- proxy: `{meta.proxy}` headed: `{meta.headed}` " + f"captcha-disabled: `{meta.captcha_disabled}`", + f"- scrapling: `{meta.scrapling_version}` suites: `{meta.suites}`", + "", + "| Suite | Check | Tier | Status | Numeric | Detail |", + "|---|---|---|---|---|---|", + ] + for r in results: + num = "" if r.numeric is None else f"{r.numeric:g}" + detail = r.detail.replace("|", "\\|") + lines.append( + f"| {r.suite} | {r.name} | {r.tier} | {r.status.value} | {num} | {detail} |" + ) + lines.append("") + for suite, s in summarize(results).items(): + lines.append( + f"- **Suite {suite}**: {s['passed']}/{s['total']} passed " + f"(fail={s['failed']}, err={s['error']}, info={s['info']})" + ) + return "\n".join(lines) + "\n" diff --git a/surfsense_backend/app/proprietary/web_crawler/testbench/results/.gitignore b/surfsense_backend/app/proprietary/web_crawler/testbench/results/.gitignore new file mode 100644 index 000000000..bdd535a38 --- /dev/null +++ b/surfsense_backend/app/proprietary/web_crawler/testbench/results/.gitignore @@ -0,0 +1,5 @@ +# Everything under results/ is run-local output (timestamped scorecards, +# latest.json pointer, screenshots, ad-hoc dumps). None of it is committed — +# the dir is kept tracked only via this file so the harness has a write target. +* +!.gitignore diff --git a/surfsense_backend/app/proprietary/web_crawler/testbench/suite_extraction.py b/surfsense_backend/app/proprietary/web_crawler/testbench/suite_extraction.py new file mode 100644 index 000000000..40fc1a223 --- /dev/null +++ b/surfsense_backend/app/proprietary/web_crawler/testbench/suite_extraction.py @@ -0,0 +1,123 @@ +# SurfSense proprietary crawler engine. +# +# This module is part of the ``app.proprietary`` package and is licensed +# SEPARATELY from the Apache-2.0 project root. See ``app/proprietary/LICENSE``. +# Do not relicense or redistribute this file under Apache-2.0. +"""Suite E — extraction correctness (separate axis from stealth). + +Unlike Suite S, this drives the **real ``crawl_url`` ladder end-to-end** — the +auto-tier + Trafilatura markdown path is exactly the production behavior we want +to assert. Targets are purpose-built, ToS-safe scraping sandboxes with known +content, so extraction regressions are caught deterministically (a missing +expected string => FAIL). The ``/js`` cases exercise the DynamicFetcher (browser) +tier; the static catalogs exercise the HTTP tier. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field + +from app.proprietary.web_crawler import CrawlOutcomeStatus, WebCrawlerConnector + +from .core import CheckResult, CheckStatus + + +@dataclass +class ExtractionCase: + """A sandbox URL + strings that must appear in the extracted markdown.""" + + name: str + url: str + must_contain: list[str] = field(default_factory=list) + + +_CASES: list[ExtractionCase] = [ + ExtractionCase( + name="books_static", + url="https://books.toscrape.com/", + must_contain=["A Light in the Attic", "Tipping the Velvet"], + ), + ExtractionCase( + name="quotes_static", + url="https://quotes.toscrape.com/", + must_contain=["Albert Einstein", "world as we have created it"], + ), + ExtractionCase( + name="quotes_js_rendered", + url="https://quotes.toscrape.com/js/", + must_contain=["Albert Einstein", "world as we have created it"], + ), + ExtractionCase( + name="scrapethissite_simple", + url="https://www.scrapethissite.com/pages/simple/", + must_contain=["Andorra", "Afghanistan"], + ), +] + + +def _content_of(outcome) -> str: + result = getattr(outcome, "result", None) or {} + if isinstance(result, dict): + return str(result.get("content") or "") + return "" + + +async def _run_case( + connector: WebCrawlerConnector, case: ExtractionCase +) -> CheckResult: + bar = f"SUCCESS + contains {len(case.must_contain)} marker(s)" + try: + outcome = await connector.crawl_url(case.url) + except Exception as exc: # noqa: BLE001 - never crash the run + return CheckResult( + suite="E", + name=case.name, + tier="crawl_url", + status=CheckStatus.ERROR, + bar=bar, + detail=f"{type(exc).__name__}: {exc}", + ) + + tier = getattr(outcome, "tier", None) or "crawl_url" + if outcome.status is not CrawlOutcomeStatus.SUCCESS: + return CheckResult( + suite="E", + name=case.name, + tier=tier, + status=CheckStatus.FAIL, + bar=bar, + detail=f"status={outcome.status.value}: {outcome.error or ''}"[:160], + ) + + content = _content_of(outcome) + lowered = content.lower() + missing = [m for m in case.must_contain if m.lower() not in lowered] + if missing: + return CheckResult( + suite="E", + name=case.name, + tier=tier, + status=CheckStatus.FAIL, + bar=bar, + detail=f"missing {missing} (len={len(content)})", + numeric=float(len(content)), + ) + return CheckResult( + suite="E", + name=case.name, + tier=tier, + status=CheckStatus.PASS, + bar=bar, + detail=f"all markers present (tier={tier}, len={len(content)})", + numeric=float(len(content)), + ) + + +async def run_suite_e() -> list[CheckResult]: + """Run the extraction-correctness cases against the live sandboxes.""" + connector = WebCrawlerConnector() + results: list[CheckResult] = [] + for case in _CASES: + print(f" [E] {case.name} -> {case.url}") + results.append(await _run_case(connector, case)) + return results diff --git a/surfsense_backend/app/proprietary/web_crawler/testbench/suite_stealth.py b/surfsense_backend/app/proprietary/web_crawler/testbench/suite_stealth.py new file mode 100644 index 000000000..83b2ce023 --- /dev/null +++ b/surfsense_backend/app/proprietary/web_crawler/testbench/suite_stealth.py @@ -0,0 +1,509 @@ +# SurfSense proprietary crawler engine. +# +# This module is part of the ``app.proprietary`` package and is licensed +# SEPARATELY from the Apache-2.0 project root. See ``app/proprietary/LICENSE``. +# Do not relicense or redistribute this file under Apache-2.0. +"""Suite S — stealth / anti-bot scorecard. + +Drives the **browser tier** (``StealthyFetcher``) against the standard +bot-detection sites and the **HTTP tier** (``AsyncFetcher``) against the TLS / +proxy-leak JSON endpoints. Critically, the browser tier is built from the **same +centralized stealth builder the crawler ships** (``build_stealthy_kwargs`` / +``get_stealth_config`` in ``app/proprietary/web_crawler/stealth.py``) so the scorecard grades +the exact browser we run in production — no test-vs-prod drift (03f §Harness). + +Verdict extraction is best-effort and tolerant: machine-readable signals +(reCAPTCHA score text, are-you-a-bot text, sannysoft failed-cell count, TLS/proxy +JSON) get a real PASS/FAIL/numeric; DOM-heavy fingerprint pages are recorded as +INFO with a screenshot + captured text for the operator to read. Tightening any +parser later is a one-function change (the spec list is the extension point). +""" + +from __future__ import annotations + +import asyncio +import re +from collections.abc import Callable +from dataclasses import dataclass +from typing import Any + +from scrapling.fetchers import AsyncFetcher, StealthyFetcher + +from app.proprietary.web_crawler.stealth import ( + build_stealthy_kwargs, + get_stealth_config, +) + +from .core import ( + SCREENSHOTS_DIR, + CheckResult, + CheckStatus, + EvalCell, + make_page_action, +) + +# Parser: (page, eval_cell) -> (status, numeric, detail). +Parser = Callable[[Any, EvalCell], "tuple[CheckStatus, float | None, str]"] + + +@dataclass +class StealthSite: + """One browser-tier detection site + how to read its verdict.""" + + name: str + url: str + bar: str + parse: Parser + settle_ms: int = 0 # extra in-page wait for async-computed scores + evaluate_js: str | None = None # for window.* / JS-object verdicts + + +def _text(page: Any, limit: int = 4000) -> str: + """Best-effort visible text of a fetched page (short, for fallback details).""" + return _full_text(page, limit) + + +def _full_text(page: Any, limit: int = 200_000) -> str: + """Untruncated visible text — detector verdicts live deep in long pages.""" + try: + txt = page.get_all_text() + if txt: + return str(txt)[:limit] + except Exception: + pass + try: + return str(page.html_content or "")[:limit] + except Exception: + return "" + + +# --- per-site parsers (all written against real DOM dumps, no screenshots) --- + + +def _parse_sannysoft(page: Any, _cell: EvalCell): + """Count failed result cells (JS sets class='failed' on red rows).""" + html = "" + try: + html = page.html_content or "" + except Exception: + pass + fails = len(re.findall(r'class="[^"]*\bfailed\b[^"]*"', html)) + if not html: + return (CheckStatus.ERROR, None, "no html returned") + status = CheckStatus.PASS if fails == 0 else CheckStatus.FAIL + return (status, float(fails), f"{fails} failed cell(s) (bar: 0)") + + +# deviceandbrowserinfo renders a JSON verdict block: {"isBot": false, ...}. +_ISBOT_RE = re.compile(r'"isbot"\s*:\s*(true|false)', re.IGNORECASE) + + +def _parse_areyouabot(page: Any, _cell: EvalCell): + txt = _full_text(page) + m = _ISBOT_RE.search(txt) + if m: + is_bot = m.group(1).lower() == "true" + return ( + CheckStatus.FAIL if is_bot else CheckStatus.PASS, + 1.0 if is_bot else 0.0, + f"isBot={is_bot}", + ) + low = txt.lower() + if "you are human" in low or "not a bot" in low: + return (CheckStatus.PASS, 0.0, "verdict: human") + if "you are a bot" in low: + return (CheckStatus.FAIL, 1.0, "verdict: BOT") + return (CheckStatus.INFO, None, f"unparsed: {low[:120]!r}") + + +# The reCAPTCHA demo echoes the server verify response: {"success":true,"score":0.9}. +_SCORE_JSON_RE = re.compile(r'"score"\s*:\s*([0-9]*\.?[0-9]+)') + + +def _parse_recaptcha_v3(page: Any, _cell: EvalCell): + txt = _full_text(page) + m = _SCORE_JSON_RE.search(txt) + if not m: # fall back to looser prose match + m = re.search(r"score[^0-9]{0,12}([01](?:\.\d+)?)", txt, re.IGNORECASE) + if not m: + return (CheckStatus.INFO, None, f"no score parsed: {txt[:120]!r}") + score = float(m.group(1)) + status = CheckStatus.PASS if score >= 0.7 else CheckStatus.FAIL + return (status, score, f"reCAPTCHA v3 score={score} (bar: >=0.7)") + + +# CreepJS prints category percentages ("33% headless") + boolean tells inline. +_CREEPJS_TELLS = ( + "webDriverIsOn", + "hasHeadlessUA", + "hasHeadlessWorkerUA", + "hasBadWebGL", + "hasSwiftShader", +) + + +def _parse_creepjs(page: Any, _cell: EvalCell): + """Grade CreepJS by its headless-similarity % + the boolean spoof tells.""" + txt = _full_text(page) + + def _pct(label: str) -> int | None: + m = re.search(r"(\d+)%\s*" + label, txt, re.IGNORECASE) + return int(m.group(1)) if m else None + + headless = _pct(r"headless") + stealth = _pct(r"stealth") + tells = [ + flag + for flag in _CREEPJS_TELLS + if re.search(re.escape(flag) + r"\s*:\s*true", txt, re.IGNORECASE) + ] + if headless is None and not tells: + return (CheckStatus.INFO, None, f"unparsed: {txt[:120]!r}") + bad = bool(tells) or (headless is not None and headless > 30) + detail = f"headless={headless}% stealth={stealth}% tells={tells or 'none'}" + return ( + CheckStatus.FAIL if bad else CheckStatus.PASS, + float(headless) if headless is not None else None, + detail, + ) + + +# incolumitas emits JSON test blocks ("WEBDRIVER":"FAIL") + an IP classification. +_FAIL_KEY_RE = re.compile(r'"([A-Za-z_]+)"\s*:\s*"FAIL"') +_DC_RE = re.compile(r'"is_datacenter"\s*:\s*(true|false)', re.IGNORECASE) +_BEHAV_RE = re.compile(r"Your Behavioral Score:\s*([0-9.]+)", re.IGNORECASE) + + +def _parse_incolumitas(page: Any, _cell: EvalCell): + txt = _full_text(page) + fails = sorted(set(_FAIL_KEY_RE.findall(txt))) + dc = _DC_RE.search(txt) + datacenter = dc.group(1) if dc else "?" + behav = _BEHAV_RE.search(txt) + bscore = behav.group(1) if behav else "n/a (no synthetic input)" + if not fails and dc is None: + return (CheckStatus.INFO, None, f"unparsed: {txt[:120]!r}") + detail = ( + f"fpscanner FAIL={fails or 'none'} datacenter={datacenter} " + f"behavioral={bscore}" + ) + return ( + CheckStatus.PASS if not fails else CheckStatus.FAIL, + float(len(fails)), + detail, + ) + + +# fingerprint-scan.com: "Bot Risk Score: 35/100"; the site flags >50 as bot. +_RISK_RE = re.compile(r"Bot Risk Score:\s*(\d+)\s*/\s*100", re.IGNORECASE) + + +def _parse_fingerprint_scan(page: Any, _cell: EvalCell): + txt = _full_text(page) + m = _RISK_RE.search(txt) + if not m: + return (CheckStatus.INFO, None, f"no risk score: {txt[:120]!r}") + score = int(m.group(1)) + status = CheckStatus.PASS if score < 50 else CheckStatus.FAIL + return (status, float(score), f"bot risk {score}/100 (bar: <50)") + + +def _parse_fingerprintjs(page: Any, _cell: EvalCell): + """FingerprintJS Pro Smart Signals: a block message means we were detected.""" + low = _full_text(page).lower() + if "access denied" in low or "tampering detected" in low: + return (CheckStatus.FAIL, 1.0, "blocked: anti-detect tampering / access denied") + if "search for today's flights" in low or "flight" in low: + return (CheckStatus.PASS, 0.0, "not blocked (flight results served)") + return (CheckStatus.INFO, None, f"unparsed: {low[:120]!r}") + + +def _parse_browserscan(page: Any, _cell: EvalCell): + """BrowserScan grades each probe Normal/Abnormal; any Abnormal = detected.""" + txt = _full_text(page) + abnormal = len(re.findall(r"\bAbnormal\b", txt)) + if abnormal == 0 and "Test Results:" in txt and "Normal" in txt: + return (CheckStatus.PASS, 0.0, "Test Results: Normal (0 Abnormal)") + if abnormal > 0: + return (CheckStatus.FAIL, float(abnormal), f"{abnormal} Abnormal signal(s)") + return (CheckStatus.INFO, None, f"unparsed: {txt[:120]!r}") + + +# The scrapingcourse CF canary prints an explicit success line once solved; a +# failure leaves the interstitial ("Just a moment...") in the DOM. This is the +# ONLY site here that exercises solve_cloudflare (the others present no challenge). +_CF_PASS = "you bypassed the cloudflare challenge" +_CF_BLOCK = ("just a moment", "attention required", "verify you are human") + + +def _parse_cloudflare(page: Any, _cell: EvalCell): + low = _full_text(page).lower() + if _CF_PASS in low: + return (CheckStatus.PASS, 0.0, "bypassed Cloudflare challenge") + if any(m in low for m in _CF_BLOCK): + return (CheckStatus.FAIL, 1.0, "stuck on Cloudflare interstitial") + return (CheckStatus.INFO, None, f"unparsed: {low[:120]!r}") + + +# iphey renders a masthead verdict "Your Digital Identity Looks " after an async fingerprint+IP correlation (slow → big +# settle). Only "Trustworthy" is a clean pass. +_IPHEY_RE = re.compile(r"Your Digital Identity Looks\s+([A-Za-z]+)", re.IGNORECASE) + + +def _parse_iphey(page: Any, _cell: EvalCell): + txt = _full_text(page) + m = _IPHEY_RE.search(txt) + if not m: + return (CheckStatus.INFO, None, f"verdict not loaded: {txt[:120]!r}") + verdict = m.group(1) + ok = verdict.lower() == "trustworthy" + return ( + CheckStatus.PASS if ok else CheckStatus.FAIL, + 0.0 if ok else 1.0, + f"iphey verdict: {verdict}", + ) + + +_BROWSER_SITES: list[StealthSite] = [ + StealthSite( + name="sannysoft", + url="https://bot.sannysoft.com/", + bar="0 failed cells", + parse=_parse_sannysoft, + settle_ms=3000, + ), + StealthSite( + name="deviceandbrowserinfo", + url="https://deviceandbrowserinfo.com/are_you_a_bot", + bar="isBot=false", + parse=_parse_areyouabot, + settle_ms=3000, + ), + StealthSite( + name="recaptcha_v3_score", + url="https://recaptcha-demo.appspot.com/recaptcha-v3-request-scores.php", + bar="score >= 0.7", + parse=_parse_recaptcha_v3, + # v3 runs grecaptcha.execute then round-trips a server verify; too-short a + # wait reads the page before the {"score":..} JSON lands (false 0.0). + settle_ms=12000, + ), + StealthSite( + name="creepjs", + url="https://abrahamjuliot.github.io/creepjs/", + bar="headless <=30%, no spoof tells", + parse=_parse_creepjs, + settle_ms=30000, + ), + StealthSite( + name="browserscan", + url="https://www.browserscan.net/bot-detection", + bar="0 Abnormal", + parse=_parse_browserscan, + settle_ms=8000, + ), + StealthSite( + name="incolumitas", + url="https://bot.incolumitas.com/", + bar="0 fpscanner FAIL", + parse=_parse_incolumitas, + settle_ms=12000, + ), + StealthSite( + name="fingerprint_scan", + url="https://fingerprint-scan.com/", + bar="bot risk < 50/100", + parse=_parse_fingerprint_scan, + settle_ms=20000, + ), + StealthSite( + name="fingerprintjs_demo", + url="https://demo.fingerprint.com/web-scraping", + bar="not blocked", + parse=_parse_fingerprintjs, + settle_ms=6000, + ), + StealthSite( + name="cloudflare_challenge", + url="https://www.scrapingcourse.com/cloudflare-challenge", + bar="bypass CF challenge (exercises solve_cloudflare)", + parse=_parse_cloudflare, + settle_ms=15000, + ), + StealthSite( + name="iphey", + url="https://iphey.com/", + bar="verdict Trustworthy", + parse=_parse_iphey, + # iphey's verdict loads via a slow async correlation; <~20s reads the + # "Temporary value" placeholder (false INFO). + settle_ms=25000, + ), +] + +# S2 — per-property fingerprint pages: too visual to auto-grade; emit as manual +# links so the operator can confirm the 03e levers (canvas/webgl/fonts/webrtc). +_MANUAL_LINKS: list[tuple[str, str]] = [ + ("browserleaks_canvas", "https://browserleaks.com/canvas"), + ("browserleaks_webgl", "https://browserleaks.com/webgl"), + ("browserleaks_fonts", "https://browserleaks.com/fonts"), + ("browserleaks_webrtc", "https://browserleaks.com/webrtc"), +] + + +def _run_browser_site( + site: StealthSite, *, proxy: str | None, headed: bool, screenshots: bool +) -> CheckResult: + shot = str(SCREENSHOTS_DIR / f"S_{site.name}.png") if screenshots else None + action, cell = make_page_action( + evaluate_js=site.evaluate_js, + screenshot_path=shot, + pre_wait_ms=site.settle_ms, + ) + kwargs: dict[str, Any] = { + "headless": not headed, + "network_idle": True, + "block_ads": True, + "solve_cloudflare": True, + "proxy": proxy, + "timeout": 120000, + "page_action": action, + } + # Single source of truth — the exact levers the production crawler ships. + kwargs.update(build_stealthy_kwargs(get_stealth_config())) + + try: + page = StealthyFetcher.fetch(site.url, **kwargs) + except Exception as exc: # noqa: BLE001 - never crash the run + return CheckResult( + suite="S", + name=site.name, + tier="scrapling-stealthy", + status=CheckStatus.ERROR, + bar=site.bar, + detail=f"fetch failed: {type(exc).__name__}: {exc}", + screenshot=shot, + ) + + try: + status, numeric, detail = site.parse(page, cell) + except Exception as exc: # noqa: BLE001 + status, numeric, detail = ( + CheckStatus.ERROR, + None, + f"parse failed: {type(exc).__name__}: {exc}", + ) + return CheckResult( + suite="S", + name=site.name, + tier="scrapling-stealthy", + status=status, + bar=site.bar, + detail=detail, + numeric=numeric, + screenshot=shot, + ) + + +async def _run_tls(proxy: str | None) -> CheckResult: + """S3 — JA3/JA4/PeetPrint of the impersonated static (curl_cffi) tier.""" + try: + page = await AsyncFetcher.get( + "https://tls.peet.ws/api/all", + stealthy_headers=True, + impersonate="chrome", + proxy=proxy, + timeout=30, + ) + data = page.json() + tls = data.get("tls", {}) if isinstance(data, dict) else {} + ja3 = tls.get("ja3_hash") or tls.get("ja3") + ja4 = tls.get("ja4") + peet = tls.get("peetprint_hash") + return CheckResult( + suite="S", + name="tls_fingerprint", + tier="scrapling-static", + status=CheckStatus.INFO, + bar="informational (diff vs real Chrome)", + detail=f"ja3={ja3} ja4={ja4} peet={peet}", + ) + except Exception as exc: # noqa: BLE001 + return CheckResult( + suite="S", + name="tls_fingerprint", + tier="scrapling-static", + status=CheckStatus.ERROR, + bar="informational", + detail=f"{type(exc).__name__}: {exc}", + ) + + +async def _run_proxy_leak(proxy: str | None) -> CheckResult: + """S4 — record the exit IP seen by an echo endpoint (proves egress path).""" + try: + page = await AsyncFetcher.get( + "https://api.ipify.org?format=json", + impersonate="chrome", + proxy=proxy, + timeout=30, + ) + data = page.json() + ip = data.get("ip") if isinstance(data, dict) else None + note = "via proxy" if proxy else "DIRECT (no proxy — datacenter IP)" + return CheckResult( + suite="S", + name="exit_ip", + tier="scrapling-static", + status=CheckStatus.INFO, + bar="not your real/datacenter IP", + detail=f"exit_ip={ip} ({note})", + ) + except Exception as exc: # noqa: BLE001 + return CheckResult( + suite="S", + name="exit_ip", + tier="scrapling-static", + status=CheckStatus.ERROR, + bar="n/a", + detail=f"{type(exc).__name__}: {exc}", + ) + + +async def run_suite_s( + *, proxy: str | None, headed: bool, screenshots: bool +) -> list[CheckResult]: + """Run the full stealth suite (browser tier sites + TLS + proxy + manual links).""" + results: list[CheckResult] = [] + + # Browser sites are sync (spin up a real browser) → offload per-site so one + # slow score doesn't stall the loop and the event loop stays clean. + for site in _BROWSER_SITES: + print(f" [S] {site.name} ... (settle {site.settle_ms}ms)") + res = await asyncio.to_thread( + _run_browser_site, + site, + proxy=proxy, + headed=headed, + screenshots=screenshots, + ) + results.append(res) + + print(" [S] tls_fingerprint + exit_ip ...") + results.append(await _run_tls(proxy)) + results.append(await _run_proxy_leak(proxy)) + + for name, url in _MANUAL_LINKS: + results.append( + CheckResult( + suite="S", + name=name, + tier="manual", + status=CheckStatus.INFO, + bar="open in a real headed run + compare", + detail=url, + ) + ) + + return results diff --git a/surfsense_backend/app/utils/crawl/__init__.py b/surfsense_backend/app/utils/crawl/__init__.py new file mode 100644 index 000000000..66a38ce4f --- /dev/null +++ b/surfsense_backend/app/utils/crawl/__init__.py @@ -0,0 +1,18 @@ +"""App-wide crawler block classification (Apache-2.0, generic). + +Phase 3e (Slice A). Mirrors ``app/utils/proxy`` and ``app/utils/captcha``: this +package holds only the **generic, vendor-agnostic** glue — here, a pure block +classifier (passive telemetry from public anti-bot markers). It is consumed by +the separately licensed proprietary crawler to label ``CrawlOutcome.block_type``. + +The **bypass-specific tuning** (the stealth kwargs builder / geoip coherence, and +the deferred WebGL spoof + humanize choreography) is NOT generic and lives under +the proprietary boundary in ``app/proprietary/web_crawler/`` (``stealth.py``). +""" + +from app.utils.crawl.classifier import BlockType, classify_block + +__all__ = [ + "BlockType", + "classify_block", +] diff --git a/surfsense_backend/app/utils/crawl/classifier.py b/surfsense_backend/app/utils/crawl/classifier.py new file mode 100644 index 000000000..8b3ba8e98 --- /dev/null +++ b/surfsense_backend/app/utils/crawl/classifier.py @@ -0,0 +1,94 @@ +"""Block classifier — labels a fetched page so the ladder can learn (Phase 3e). + +Pure, additive, status + body-marker based. It attaches +:class:`~app.proprietary.web_crawler.connector.CrawlOutcome.block_type` for +telemetry / future escalation routing (per-domain memory, `03d` captcha routing) +and **never** changes *when* a crawl is ``SUCCESS`` — `03c` bills on +``status == SUCCESS``, so this stays read-only metadata. + +Markers mirror the proven set in the Camoufox-based FlareSolverr alternative +``references/trawl-dev/packages/tiers/src/detect.ts`` (plus DataDome/Kasada for +our enterprise targets). Regexes are compiled once at import (hot-path hygiene). +""" + +import re +from enum import Enum + + +class BlockType(str, Enum): + """Coarse label for what a fetched page represents.""" + + OK = "ok" # usable content, no challenge detected + CLOUDFLARE = "cloudflare" # CF interstitial / Turnstile / DDoS-Guard + CAPTCHA_RECAPTCHA = "captcha_recaptcha" + CAPTCHA_HCAPTCHA = "captcha_hcaptcha" + DATADOME = "datadome" + KASADA = "kasada" + RATE_LIMITED = "rate_limited" + EMPTY = "empty" # fetched but no HTML body + UNKNOWN = "unknown" # blocking-ish status with no recognized marker + + +# --- compiled marker patterns (hoisted; see Vercel rule 7.9) --- +_CLOUDFLARE = re.compile( + r"just a moment" + r"|checking your browser" + r"|enable javascript and cookies to continue" + r"|verify you are human" + r"|cf-mitigated" + r"|id=\"(?:cf-)?challenge-running\"" + r"|id=\"turnstile-wrapper\"" + r"|ddos-guard", + re.IGNORECASE, +) +_TURNSTILE = re.compile( + r"class=\"cf-turnstile\"" + r"|challenges\.cloudflare\.com/turnstile" + r"|cdn-cgi/challenge-platform[^\"']*turnstile", + re.IGNORECASE, +) +_HCAPTCHA = re.compile(r"class=\"h-captcha\"|hcaptcha\.com/1/api", re.IGNORECASE) +_RECAPTCHA = re.compile( + r"class=\"g-recaptcha\"|google\.com/recaptcha|recaptcha\.net/recaptcha", + re.IGNORECASE, +) +_DATADOME = re.compile(r"datadome|geo\.captcha-delivery\.com", re.IGNORECASE) +_KASADA = re.compile(r"kpsdk|x-kpsdk|kasada", re.IGNORECASE) + +# Statuses some CDNs use as a bot-gate before the real response (detect.ts:isBlocked). +_BOT_GATE_STATUSES = frozenset({202, 403}) + + +def classify_block(status: int | None, html: str | None) -> BlockType: + """Label a fetched page from its HTTP status and HTML body. + + Precedence: explicit rate-limit status, then specific anti-bot/captcha body + markers, then a generic bot-gate status with no marker, else ``OK`` (or + ``EMPTY`` when there is no body). Marker checks run before the generic + bot-gate so a ``403`` Cloudflare challenge is labeled ``CLOUDFLARE``, not + ``UNKNOWN``. + """ + if status == 429: + return BlockType.RATE_LIMITED + + if not html or not html.strip(): + # No body: distinguish a blocking-ish status from a genuinely empty 200. + if status in _BOT_GATE_STATUSES: + return BlockType.UNKNOWN + return BlockType.EMPTY + + if _CLOUDFLARE.search(html) or _TURNSTILE.search(html): + return BlockType.CLOUDFLARE + if _DATADOME.search(html): + return BlockType.DATADOME + if _KASADA.search(html): + return BlockType.KASADA + if _HCAPTCHA.search(html): + return BlockType.CAPTCHA_HCAPTCHA + if _RECAPTCHA.search(html): + return BlockType.CAPTCHA_RECAPTCHA + + if status in _BOT_GATE_STATUSES: + return BlockType.UNKNOWN + + return BlockType.OK diff --git a/surfsense_backend/tests/unit/proprietary/web_crawler/test_connector.py b/surfsense_backend/tests/unit/proprietary/web_crawler/test_connector.py index dbd389634..2c749b2cd 100644 --- a/surfsense_backend/tests/unit/proprietary/web_crawler/test_connector.py +++ b/surfsense_backend/tests/unit/proprietary/web_crawler/test_connector.py @@ -12,6 +12,7 @@ from app.proprietary.web_crawler import ( WebCrawlerConnector, ) from app.proprietary.web_crawler import connector as connector_module +from app.utils.crawl import BlockType pytestmark = pytest.mark.unit @@ -40,10 +41,10 @@ async def test_static_tier_success_short_circuits( crawler = WebCrawlerConnector() later_calls: list[str] = [] - async def _static(_url: str) -> dict: + async def _static(_url: str, *_args) -> dict: return _result("scrapling-static") - async def _record_dynamic(_url: str) -> None: + async def _record_dynamic(_url: str, *_args) -> None: later_calls.append("dynamic") return None @@ -70,10 +71,10 @@ async def test_escalates_to_dynamic_on_static_miss( """Static empty extraction escalates to the dynamic tier.""" crawler = WebCrawlerConnector() - async def _empty(_url: str) -> None: + async def _empty(_url: str, *_args) -> None: return None - async def _dynamic(_url: str) -> dict: + async def _dynamic(_url: str, *_args) -> dict: return _result("scrapling-dynamic") monkeypatch.setattr(crawler, "_crawl_with_async_fetcher", _empty) @@ -126,7 +127,7 @@ async def test_proxy_error_rotates_once_when_pool_backed( crawler = WebCrawlerConnector() calls = {"n": 0} - async def _flaky(_url: str) -> dict: + async def _flaky(_url: str, *_args) -> dict: calls["n"] += 1 if calls["n"] == 1: raise RuntimeError("connection refused by upstream proxy") @@ -149,7 +150,7 @@ async def test_proxy_error_no_retry_when_single_endpoint( crawler = WebCrawlerConnector() static_calls = {"n": 0} - async def _proxy_err(_url: str) -> None: + async def _proxy_err(_url: str, *_args) -> None: static_calls["n"] += 1 raise RuntimeError("connection refused by upstream proxy") @@ -173,7 +174,7 @@ async def test_captcha_defaults_zero_on_non_stealthy_success( """03d: a static success never touches captcha → fields stay 0/False.""" crawler = WebCrawlerConnector() - async def _static(_url: str) -> dict: + async def _static(_url: str, *_args) -> dict: return _result("scrapling-static") monkeypatch.setattr(crawler, "_crawl_with_async_fetcher", _static) @@ -194,7 +195,7 @@ async def test_captcha_state_surfaced_onto_outcome( async def _empty(_url: str, *_args) -> None: return None - async def _stealthy(_url: str, captcha_state: dict) -> dict: + async def _stealthy(_url: str, captcha_state: dict, *_args) -> dict: # Simulate the page_action having solved a captcha mid-fetch. captcha_state["attempts"] = 2 captcha_state["solved"] = True @@ -221,7 +222,9 @@ async def test_captcha_attempts_surface_even_when_crawl_fails( async def _empty(_url: str, *_args) -> None: return None - async def _stealthy_attempt_then_empty(_url: str, captcha_state: dict) -> None: + async def _stealthy_attempt_then_empty( + _url: str, captcha_state: dict, *_args + ) -> None: captcha_state["attempts"] = 1 # solve attempted but crawl still empty return None @@ -245,7 +248,7 @@ async def test_non_proxy_error_never_retries( crawler = WebCrawlerConnector() calls = {"n": 0} - async def _boom(_url: str) -> None: + async def _boom(_url: str, *_args) -> None: calls["n"] += 1 raise RuntimeError("totally unrelated failure") @@ -261,3 +264,104 @@ async def test_non_proxy_error_never_retries( assert calls["n"] == 1 # not retried (not a proxy error) assert outcome.status is CrawlOutcomeStatus.EMPTY + + +async def test_invalid_url_block_type_defaults_unknown() -> None: + """03e: a pre-fetch FAILED carries the default UNKNOWN block_type.""" + outcome = await WebCrawlerConnector().crawl_url("not a url") + + assert outcome.status is CrawlOutcomeStatus.FAILED + assert outcome.block_type is BlockType.UNKNOWN + + +async def test_block_type_surfaced_onto_outcome( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """03e: a tier's block classification (in block_state) is stamped onto the + outcome — additive only, never gating SUCCESS.""" + crawler = WebCrawlerConnector() + + async def _empty(_url: str, *_args) -> None: + return None + + async def _stealthy_blocked( + _url: str, _captcha_state: dict, block_state: dict + ) -> None: + # Simulate _build_result having classified a Cloudflare interstitial. + block_state["block_type"] = BlockType.CLOUDFLARE + return None + + monkeypatch.setattr(crawler, "_crawl_with_async_fetcher", _empty) + monkeypatch.setattr(crawler, "_crawl_with_dynamic", _empty) + monkeypatch.setattr(crawler, "_crawl_with_stealthy", _stealthy_blocked) + + outcome = await crawler.crawl_url("https://example.com") + + assert outcome.status is CrawlOutcomeStatus.EMPTY + assert outcome.block_type is BlockType.CLOUDFLARE + + +def test_build_result_classifies_into_block_state() -> None: + """03e: _build_result labels the fetched page into the passed block_state.""" + crawler = WebCrawlerConnector() + block_state: dict = {"block_type": BlockType.UNKNOWN} + + cf_html = "Just a moment..." + result = crawler._build_result( + cf_html, + "https://example.com", + "scrapling-stealthy", + allow_raw_fallback=False, + status=403, + block_state=block_state, + ) + + # Cloudflare interstitial: no real content extracted (None) but classified. + assert result is None + assert block_state["block_type"] is BlockType.CLOUDFLARE + + +async def test_static_4xx_is_classified( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """03e: a static-tier 4xx bot-gate is classified before the early return + (otherwise the cheapest/first tier's block signal would be lost).""" + crawler = WebCrawlerConnector() + + class _Page: + status = 403 + html_content = "Just a moment..." + + class _AsyncFetcher: + @staticmethod + async def get(*_a, **_k): + return _Page() + + monkeypatch.setattr(connector_module, "AsyncFetcher", _AsyncFetcher) + monkeypatch.setattr(connector_module, "get_proxy_url", lambda: None) + + block_state: dict = {"block_type": BlockType.UNKNOWN} + result = await crawler._crawl_with_async_fetcher( + "https://example.com", block_state + ) + + assert result is None # 4xx => fall through to next tier + assert block_state["block_type"] is BlockType.CLOUDFLARE + + +def test_build_result_ok_on_real_content() -> None: + """03e: a normal 200 page with content classifies OK.""" + crawler = WebCrawlerConnector() + block_state: dict = {"block_type": BlockType.UNKNOWN} + + html = "
" + ("Real content. " * 40) + "
" + crawler._build_result( + html, + "https://example.com", + "scrapling-static", + allow_raw_fallback=False, + status=200, + block_state=block_state, + ) + + assert block_state["block_type"] is BlockType.OK diff --git a/surfsense_backend/tests/unit/proprietary/web_crawler/test_stealth.py b/surfsense_backend/tests/unit/proprietary/web_crawler/test_stealth.py new file mode 100644 index 000000000..951fcd614 --- /dev/null +++ b/surfsense_backend/tests/unit/proprietary/web_crawler/test_stealth.py @@ -0,0 +1,113 @@ +"""Unit tests for the Phase 3e stealth kwargs builder (proprietary boundary).""" + +import pytest + +from app.config import config +from app.proprietary.web_crawler.stealth import ( + build_stealthy_kwargs, + get_stealth_config, + location_to_locale_timezone, +) + +pytestmark = pytest.mark.unit + + +class TestLocationToLocaleTimezone: + def test_alpha2_code(self): + assert location_to_locale_timezone("us") == ("en-US", "America/New_York") + assert location_to_locale_timezone("de") == ("de-DE", "Europe/Berlin") + + def test_case_and_whitespace_insensitive(self): + assert location_to_locale_timezone(" US ") == ( + "en-US", + "America/New_York", + ) + + def test_full_country_name_alias(self): + assert location_to_locale_timezone("Germany") == ( + "de-DE", + "Europe/Berlin", + ) + assert location_to_locale_timezone("united kingdom") == ( + "en-GB", + "Europe/London", + ) + + def test_leading_token_of_vendor_string(self): + # Vendor strings like "us:nyc" / "de-rotating" still resolve on the head. + assert location_to_locale_timezone("us:nyc") == ( + "en-US", + "America/New_York", + ) + assert location_to_locale_timezone("de-rotating") == ( + "de-DE", + "Europe/Berlin", + ) + + def test_empty_and_unknown_return_none(self): + assert location_to_locale_timezone("") == (None, None) + assert location_to_locale_timezone(None) == (None, None) + assert location_to_locale_timezone("atlantis") == (None, None) + + +class TestBuildStealthyKwargs: + def test_defaults_have_no_geoip_keys(self, monkeypatch): + # geoip off => locale/timezone_id absent (browser keeps system default). + monkeypatch.setattr(config, "CRAWL_GEOIP_MATCH_ENABLED", False) + monkeypatch.setattr(config, "CRAWL_BLOCK_WEBRTC", True) + monkeypatch.setattr(config, "CRAWL_HIDE_CANVAS", False) + monkeypatch.setattr(config, "CRAWL_GOOGLE_SEARCH_REFERER", True) + monkeypatch.setattr(config, "CRAWL_DNS_OVER_HTTPS", False) + monkeypatch.setattr(config, "RESIDENTIAL_PROXY_LOCATION", "us") + + kwargs = build_stealthy_kwargs(get_stealth_config()) + + assert kwargs == { + "block_webrtc": True, + "hide_canvas": False, + "google_search": True, + "dns_over_https": False, + } + assert "locale" not in kwargs + assert "timezone_id" not in kwargs + + def test_flags_reflect_config(self, monkeypatch): + monkeypatch.setattr(config, "CRAWL_GEOIP_MATCH_ENABLED", False) + monkeypatch.setattr(config, "CRAWL_BLOCK_WEBRTC", False) + monkeypatch.setattr(config, "CRAWL_HIDE_CANVAS", True) + monkeypatch.setattr(config, "CRAWL_GOOGLE_SEARCH_REFERER", False) + monkeypatch.setattr(config, "CRAWL_DNS_OVER_HTTPS", True) + monkeypatch.setattr(config, "RESIDENTIAL_PROXY_LOCATION", "") + + kwargs = build_stealthy_kwargs(get_stealth_config()) + + assert kwargs["block_webrtc"] is False + assert kwargs["hide_canvas"] is True + assert kwargs["google_search"] is False + assert kwargs["dns_over_https"] is True + + def test_geoip_on_adds_locale_timezone(self, monkeypatch): + monkeypatch.setattr(config, "CRAWL_GEOIP_MATCH_ENABLED", True) + monkeypatch.setattr(config, "CRAWL_BLOCK_WEBRTC", True) + monkeypatch.setattr(config, "CRAWL_HIDE_CANVAS", False) + monkeypatch.setattr(config, "CRAWL_GOOGLE_SEARCH_REFERER", True) + monkeypatch.setattr(config, "CRAWL_DNS_OVER_HTTPS", False) + monkeypatch.setattr(config, "RESIDENTIAL_PROXY_LOCATION", "de") + + kwargs = build_stealthy_kwargs(get_stealth_config()) + + assert kwargs["locale"] == "de-DE" + assert kwargs["timezone_id"] == "Europe/Berlin" + + def test_geoip_on_but_unknown_location_skips(self, monkeypatch): + monkeypatch.setattr(config, "CRAWL_GEOIP_MATCH_ENABLED", True) + monkeypatch.setattr(config, "CRAWL_BLOCK_WEBRTC", True) + monkeypatch.setattr(config, "CRAWL_HIDE_CANVAS", False) + monkeypatch.setattr(config, "CRAWL_GOOGLE_SEARCH_REFERER", True) + monkeypatch.setattr(config, "CRAWL_DNS_OVER_HTTPS", False) + monkeypatch.setattr(config, "RESIDENTIAL_PROXY_LOCATION", "atlantis") + + kwargs = build_stealthy_kwargs(get_stealth_config()) + + assert "locale" not in kwargs + assert "timezone_id" not in kwargs diff --git a/surfsense_backend/tests/unit/utils/test_crawl_classifier.py b/surfsense_backend/tests/unit/utils/test_crawl_classifier.py new file mode 100644 index 000000000..c595686c9 --- /dev/null +++ b/surfsense_backend/tests/unit/utils/test_crawl_classifier.py @@ -0,0 +1,75 @@ +"""Unit tests for the Phase 3e block classifier (Apache-2 layer).""" + +import pytest + +from app.utils.crawl import BlockType, classify_block + +pytestmark = pytest.mark.unit + + +class TestClassifyBlock: + def test_ok_on_plain_content(self): + assert classify_block(200, "hello") is BlockType.OK + + def test_none_body_is_empty(self): + assert classify_block(200, None) is BlockType.EMPTY + assert classify_block(200, " ") is BlockType.EMPTY + + def test_rate_limited_takes_precedence(self): + # 429 wins even if a challenge marker is present in the body. + assert ( + classify_block(429, '
') + is BlockType.RATE_LIMITED + ) + + @pytest.mark.parametrize( + "html", + [ + "Just a moment...", + "

Checking your browser before accessing

", + "please enable javascript and cookies to continue", + "verify you are human", + '
', + '
', + '
', + "blocked by ddos-guard.net", + ], + ) + def test_cloudflare_markers(self, html): + assert classify_block(403, html) is BlockType.CLOUDFLARE + + def test_hcaptcha(self): + assert ( + classify_block(200, '
') + is BlockType.CAPTCHA_HCAPTCHA + ) + + def test_recaptcha(self): + assert ( + classify_block(200, '
') + is BlockType.CAPTCHA_RECAPTCHA + ) + + def test_datadome(self): + assert ( + classify_block(403, "var dd={'host':'geo.captcha-delivery.com'}") + is BlockType.DATADOME + ) + + def test_kasada(self): + assert classify_block(200, "window.KPSDK.configure()") is BlockType.KASADA + + def test_bot_gate_status_without_marker_is_unknown(self): + # 202/403 with no recognized challenge marker => generic blocked-ish. + assert classify_block(202, "x") is BlockType.UNKNOWN + assert classify_block(403, "x") is BlockType.UNKNOWN + + def test_bot_gate_status_empty_body_is_unknown(self): + assert classify_block(403, None) is BlockType.UNKNOWN + + def test_cloudflare_marker_beats_generic_403(self): + # A 403 Cloudflare page is CLOUDFLARE, not UNKNOWN. + assert ( + classify_block(403, "Just a moment...") + is BlockType.CLOUDFLARE + )