apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-22 23:31:12 +02:00
Code Issues Projects Releases Packages Wiki Activity

feat(crawler): captcha solving + per-attempt billing (Phase 3d)

Browse source 
Wire captchatools as the StealthyFetcher-tier page_action to detect, harvest
(egressing from the crawl's own proxy IP), inject, and submit reCAPTCHA v2/v3
and hCaptcha tokens. Opt-in and off by default (zero attempts, zero cost).

Licensing split:
- Apache-2 app/utils/captcha/ holds the generic, vendor-agnostic config
  (CaptchaConfig + captcha_enabled() = flag AND key present).
- Proprietary app/proprietary/web_crawler/captcha.py holds the bypass logic
  (detect/harvest/inject) plus a process-wide solver latch that halts solving
  on unrecoverable errors (no balance / bad key).

Crawler: CrawlOutcome gains captcha_attempts/captcha_solved, surfaced via a
per-call captcha_state dict threaded crawl_url -> _crawl_with_stealthy(_sync)
and stamped onto every stealth terminal outcome. The stealth tier captures the
proxy once and reuses it for both the fetch and the solver (IP-coherence).

Billing: WebCrawlCreditService gains captcha_billing_enabled,
captcha_solves_to_micros, charge_captcha, and a generic check_balance, sharing
a single _apply_debit path. The indexer accumulates attempts (even on failed
crawls), runs a combined crawl+captcha pre-flight, and posts a per-attempt
owner charge as usage_type="web_crawl_captcha". The captcha worst-case is only
reserved when solving is actually enabled, so a solving-off deployment is never
blocked for captcha that can never run. Both chat scrape tools fold attempts
into the current turn before the success/fail branch.

Fully config-driven prices; no migration. New unit tests cover the config,
factory (detection/latch/timeout/cap), credit service, indexer wiring, and the
chat fold.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
DESKTOP-RTLN3BA\$punk 2026-06-29 22:58:21 -07:00
parent aad366ba13
commit bdacea8b6e
19 changed files with 1494 additions and 51 deletions
Download patch file Download diff file Expand all files Collapse all files

326
surfsense_backend/app/proprietary/web_crawler/captcha.py Normal file
View file

@ -0,0 +1,326 @@
# SurfSense proprietary crawler engine.
#
# This module is part of the ``app.proprietary`` package and is licensed
# SEPARATELY from the Apache-2.0 project root. See ``app/proprietary/LICENSE``.
# Do not relicense or redistribute this file under Apache-2.0.
"""Captcha detection + token-injection ``page_action`` (Phase 3d).
This is the **bypass logic** (hence proprietary); the generic, vendor-agnostic
config lives in the Apache-2.0 ``app/utils/captcha/`` package. ``captchatools``
is the multi-vendor solver registry — we only detect the challenge, harvest a
token egressing from **the crawl's own proxy IP** (token IP-binding), inject it,
and submit.
Why a closure cell: Scrapling runs ``page_action`` after navigation but
**swallows its exceptions and discards its return value** (see
``references/Scrapling`` engine ``_stealth.py``). So the only way to surface
"did we solve / how many attempts" back to the crawler (for per-attempt billing
and the retry cap) is to mutate a caller-owned ``state`` dict.
Why a process latch: the solver README warns that hammering it while out of
balance can get the IP **temporarily banned**, so ``ErrNoBalance`` /
``ErrWrongAPIKey`` latch solving OFF for the rest of the process (config/balance
won't change without a restart). ``reset_solver_latch()`` clears it for tests.
"""
import logging
import re
from collections.abc import Callable
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeout
from typing import Any
from urllib.parse import urlsplit
from app.utils.captcha import CaptchaConfig
logger = logging.getLogger(__name__)
_CAPTCHA_LOG = "[webcrawler][captcha]"
# Process-wide kill switch tripped on unrecoverable solver errors (no balance /
# bad key). Module-level by design: it must persist across the per-URL
# ``page_action`` instances created during one crawl run.
_solver_latched = False
def reset_solver_latch() -> None:
"""Clear the process solver latch (test seam / explicit re-enable)."""
global _solver_latched
_solver_latched = False
def solver_latched() -> bool:
return _solver_latched
def _latch_solver(reason: str) -> None:
global _solver_latched
_solver_latched = True
logger.warning("%s solver latched OFF for this process: %s", _CAPTCHA_LOG, reason)
# --- detection -------------------------------------------------------------
# (challenge_type, css_selector_for_widget). Order = detection priority.
_WIDGET_SELECTORS = (
("v2", ".g-recaptcha[data-sitekey]"),
("hcaptcha", ".h-captcha[data-sitekey]"),
)
_IFRAME_PATTERNS = (
("v2", re.compile(r"google\.com/recaptcha", re.I)),
("hcaptcha", re.compile(r"hcaptcha\.com", re.I)),
)
_SITEKEY_IN_SRC = re.compile(r"[?&]k=([\w-]+)")
_RENDER_SITEKEY = re.compile(r"render=([\w-]+)")
def detect_challenge(page: Any, cfg: CaptchaConfig) -> tuple[str, str] | None:
"""Return ``(challenge_type, sitekey)`` if a solvable challenge is present.
``challenge_type`` is one of ``v2`` / ``v3`` / ``hcaptcha``. Returns
``None`` when nothing solvable is found (the crawler then proceeds to
``wait_selector`` / extraction unchanged). Detection is defensive: any DOM
access raising is treated as "not found" rather than aborting the fetch.
"""
try:
for ctype, selector in _WIDGET_SELECTORS:
el = page.query_selector(selector)
if el is not None:
sitekey = el.get_attribute("data-sitekey")
if sitekey:
return ctype, sitekey
# Iframe fallback: widget rendered inside an iframe whose src carries
# the sitekey as ``?k=...``.
for frame_el in page.query_selector_all("iframe[src]") or []:
src = frame_el.get_attribute("src") or ""
for ctype, pat in _IFRAME_PATTERNS:
if pat.search(src):
m = _SITEKEY_IN_SRC.search(src)
if m:
return ctype, m.group(1)
# reCAPTCHA v3: no widget; sitekey rides the api.js ``?render=<key>``.
if cfg.captcha_type_default == "v3":
for script_el in page.query_selector_all("script[src]") or []:
src = script_el.get_attribute("src") or ""
if "recaptcha" in src.lower():
m = _RENDER_SITEKEY.search(src)
if m and m.group(1) != "explicit":
return "v3", m.group(1)
except Exception as exc: # noqa: BLE001
logger.debug("%s detection error (treated as no challenge): %s", _CAPTCHA_LOG, exc)
return None
# --- proxy / harvest -------------------------------------------------------
def proxy_url_to_captchatools(proxy_url: str | None) -> str | None:
"""Reformat ``http://user:pass@host:port`` -> ``host:port:user:pass``.
captchatools wants the colon-delimited form. Returns ``None`` for a missing
or unparseable proxy so the solver harvests proxyless (the token may then be
IP-mismatched, but that's better than crashing the fetch).
"""
if not proxy_url:
return None
try:
p = urlsplit(proxy_url)
if not p.hostname or not p.port:
return None
if p.username and p.password:
return f"{p.hostname}:{p.port}:{p.username}:{p.password}"
return f"{p.hostname}:{p.port}"
except Exception: # noqa: BLE001
return None
def _harvest_token(
cfg: CaptchaConfig,
challenge_type: str,
sitekey: str,
page_url: str,
proxy: str | None,
user_agent: str | None,
) -> str | None:
"""Harvest a token from the solver. Raises on solver errors so the caller
can latch on the unrecoverable ones; returns ``None`` on an empty token.
"""
import captchatools
harvester = captchatools.new_harvester(
api_key=cfg.api_key,
solving_site=cfg.solving_site,
sitekey=sitekey,
captcha_url=page_url,
captcha_type=challenge_type,
min_score=cfg.v3_min_score,
action=cfg.v3_action,
)
token = harvester.get_token(
proxy=proxy,
proxy_type="HTTP",
user_agent=user_agent,
)
return token or None
# --- injection -------------------------------------------------------------
# Sets the response token into the right textarea(s) and fires the JS callbacks
# registered by the widget, then submits the closest form. Best-effort: returns
# nothing; success is judged by the crawler's post-fetch extraction.
_INJECT_JS = r"""
(args) => {
const { token, ctype } = args;
const names = ctype === 'hcaptcha'
? ['h-captcha-response', 'g-recaptcha-response']
: ['g-recaptcha-response'];
for (const name of names) {
let ta = document.querySelector('textarea[name="' + name + '"]')
|| document.getElementById(name);
if (!ta) {
ta = document.createElement('textarea');
ta.name = name; ta.id = name;
ta.style.display = 'none';
document.body.appendChild(ta);
}
ta.value = token; ta.innerHTML = token;
ta.dispatchEvent(new Event('input', { bubbles: true }));
ta.dispatchEvent(new Event('change', { bubbles: true }));
}
// Fire grecaptcha client callbacks when present (v2 invisible / explicit).
try {
const cfg = window.___grecaptcha_cfg;
if (cfg && cfg.clients) {
for (const id in cfg.clients) {
const client = cfg.clients[id];
for (const k in client) {
const o = client[k];
if (o && typeof o === 'object') {
for (const kk in o) {
const cb = o[kk];
if (cb && cb.callback && typeof cb.callback === 'function') {
try { cb.callback(token); } catch (e) {}
}
}
}
}
}
}
} catch (e) {}
// Submit the form containing the response field, if any.
const field = document.querySelector('textarea[name="g-recaptcha-response"], textarea[name="h-captcha-response"]');
const form = field ? field.closest('form') : document.querySelector('form');
if (form) { try { form.requestSubmit ? form.requestSubmit() : form.submit(); } catch (e) {} }
}
"""
def _inject_and_submit(page: Any, challenge_type: str, token: str) -> None:
try:
page.evaluate(_INJECT_JS, {"token": token, "ctype": challenge_type})
try:
page.wait_for_load_state("networkidle", timeout=15000)
except Exception: # noqa: BLE001
pass
except Exception as exc: # noqa: BLE001
logger.warning("%s injection error: %s", _CAPTCHA_LOG, exc)
# --- factory ---------------------------------------------------------------
def build_captcha_page_action(
state: dict[str, Any],
proxy_url: str | None,
cfg: CaptchaConfig,
) -> Callable[[Any], Any] | None:
"""Build the sync ``page_action`` for ``StealthyFetcher.fetch``.
``state`` is mutated in place: ``attempts`` (int) and ``solved`` (bool). The
crawler reads it back after the fetch to bill per attempt and enforce the
per-URL cap. Returns ``None`` when solving is disabled/latched so the caller
can skip wiring it entirely (keeping the stealth tier unchanged).
"""
if not cfg.enabled or solver_latched():
return None
captcha_proxy = proxy_url_to_captchatools(proxy_url)
def page_action(page: Any) -> Any:
# Hard cap: never exceed the per-URL budget even across proxy-retry
# re-runs that share this ``state``.
if state.get("attempts", 0) >= cfg.max_attempts_per_url:
return page
if solver_latched():
return page
detected = detect_challenge(page, cfg)
if detected is None:
return page # nothing to solve; let extraction proceed
challenge_type, sitekey = detected
page_url = getattr(page, "url", "") or ""
try:
user_agent = page.evaluate("() => navigator.userAgent")
except Exception: # noqa: BLE001
user_agent = None
# This counts as an attempt the moment we call the (paid) solver.
state["attempts"] = state.get("attempts", 0) + 1
logger.info(
"%s solving type=%s site=%s attempt=%d",
_CAPTCHA_LOG,
challenge_type,
sitekey[:12],
state["attempts"],
)
# Run the (blocking) solver in a worker so the timeout can actually
# ABANDON a slow solve. NOTE: do NOT use ``with ThreadPoolExecutor`` —
# its ``__exit__`` joins the worker (``wait=True``), which would block
# for the full solve and defeat the timeout. Shut down non-blocking.
executor = ThreadPoolExecutor(max_workers=1)
future = executor.submit(
_harvest_token,
cfg,
challenge_type,
sitekey,
page_url,
captcha_proxy,
user_agent,
)
try:
token = future.result(timeout=cfg.timeout_s)
except FuturesTimeout:
logger.warning("%s solve timed out after %ss", _CAPTCHA_LOG, cfg.timeout_s)
return page
except Exception as exc: # noqa: BLE001
if _is_unrecoverable(exc):
_latch_solver(repr(exc))
else:
logger.warning("%s solve error: %s", _CAPTCHA_LOG, exc)
return page
finally:
executor.shutdown(wait=False, cancel_futures=True)
if not token:
return page
_inject_and_submit(page, challenge_type, token)
state["solved"] = True
logger.info("%s solved type=%s site=%s", _CAPTCHA_LOG, challenge_type, sitekey[:12])
return page
return page_action
def _is_unrecoverable(exc: Exception) -> bool:
"""True for solver errors that must latch solving off (no balance / bad key).
Matched by class name to avoid a hard import of ``captchatools.exceptions``
at module load (the dep is optional / lazily imported).
"""
name = type(exc).__name__.lower()
return "nobalance" in name or "wrongapikey" in name or "apikey" in name

79
surfsense_backend/app/proprietary/web_crawler/connector.py
View file

@ -33,6 +33,8 @@ import validators
from scrapling.engines.toolbelt import is_proxy_error
from scrapling.fetchers import AsyncFetcher, DynamicFetcher, StealthyFetcher
from app.proprietary.web_crawler.captcha import build_captcha_page_action
from app.utils.captcha import captcha_enabled, get_captcha_config
from app.utils.proxy import get_proxy_url, is_pool_backed
logger = logging.getLogger(__name__)
@ -56,14 +58,20 @@ class CrawlOutcome:
The **billable success predicate is single-sourced**:
``status == CrawlOutcomeStatus.SUCCESS`` (Phase 3c meters on it). Picking a
dataclass over a tuple lets later subplans append fields without breaking
callers (03d adds ``captcha_attempts``/``captcha_solved`` for per-attempt
billing; 03e's block classifier can attach a ``block_type``).
callers (03e's block classifier can attach a ``block_type``).
Phase 3d captcha fields are surfaced here so per-attempt billing can read
them off the outcome regardless of crawl SUCCESS (the solver charges per
*attempt*). They are populated only by the StealthyFetcher tier when captcha
solving is enabled; every other path leaves the defaults (0 / False).
"""
status: CrawlOutcomeStatus
result: dict[str, Any] | None = None
error: str | None = None
tier: str | None = None
captcha_attempts: int = 0
captcha_solved: bool = False
class WebCrawlerConnector:
@ -89,6 +97,11 @@ class WebCrawlerConnector:
- crawler_type: Identifier of the tier that produced the content
"""
total_start = time.perf_counter()
# Per-call captcha telemetry (03d). Mutated only by the StealthyFetcher
# tier's page_action; stamped onto the returned outcome so per-attempt
# billing can read it regardless of crawl SUCCESS. Per-call (not on
# ``self``) => safe under concurrent ``crawl_url`` calls.
captcha_state: dict[str, Any] = {"attempts": 0, "solved": False}
try:
if not validators.url(url):
return CrawlOutcome(
@ -167,7 +180,8 @@ class WebCrawlerConnector:
try:
logger.info(f"[webcrawler] Using Scrapling StealthyFetcher for: {url}")
result = await self._run_tier_with_proxy_retry(
"scrapling-stealthy", lambda: self._crawl_with_stealthy(url)
"scrapling-stealthy",
lambda: self._crawl_with_stealthy(url, captcha_state),
)
if result:
self._log_tier_outcome(
@ -178,6 +192,8 @@ class WebCrawlerConnector:
status=CrawlOutcomeStatus.SUCCESS,
result=result,
tier="scrapling-stealthy",
captcha_attempts=captcha_state["attempts"],
captcha_solved=captcha_state["solved"],
)
reached_without_content = True
errors.append("Scrapling stealthy: empty extraction")
@ -201,10 +217,14 @@ class WebCrawlerConnector:
return CrawlOutcome(
status=CrawlOutcomeStatus.EMPTY,
error=f"No content extracted for {url}. {'; '.join(errors)}",
captcha_attempts=captcha_state["attempts"],
captcha_solved=captcha_state["solved"],
)
return CrawlOutcome(
status=CrawlOutcomeStatus.FAILED,
error=f"All crawl methods failed for {url}. {'; '.join(errors)}",
captcha_attempts=captcha_state["attempts"],
captcha_solved=captcha_state["solved"],
)
except Exception as e:
@ -212,6 +232,8 @@ class WebCrawlerConnector:
return CrawlOutcome(
status=CrawlOutcomeStatus.FAILED,
error=f"Error crawling URL {url}: {e!s}",
captcha_attempts=captcha_state["attempts"],
captcha_solved=captcha_state["solved"],
)
async def _run_tier_with_proxy_retry(
@ -356,29 +378,56 @@ class WebCrawlerConnector:
status=getattr(page, "status", None),
)
async def _crawl_with_stealthy(self, url: str) -> dict[str, Any] | None:
async def _crawl_with_stealthy(
self, url: str, captcha_state: dict[str, Any] | None = None
) -> dict[str, Any] | None:
"""
Crawl URL using Scrapling's StealthyFetcher (patchright-Chromium) + Trafilatura.
Last-resort tier with anti-bot features. Runs the sync fetch in a worker
thread for the same event-loop-safety reasons as DynamicFetcher. Falls
back to the raw HTML when Trafilatura extraction is empty.
"""
return await asyncio.to_thread(self._crawl_with_stealthy_sync, url)
def _crawl_with_stealthy_sync(self, url: str) -> dict[str, Any] | None:
``captcha_state`` (03d) is mutated in place by the captcha page_action
(attempts/solved) so ``crawl_url`` can surface it on the outcome.
"""
return await asyncio.to_thread(
self._crawl_with_stealthy_sync, url, captcha_state
)
def _crawl_with_stealthy_sync(
self, url: str, captcha_state: dict[str, Any] | None = None
) -> dict[str, Any] | None:
"""Synchronous StealthyFetcher crawl executed in a worker thread."""
fetch_start = time.perf_counter()
# Capture the proxy endpoint ONCE so the captcha solver egresses from the
# SAME IP as this fetch (tokens are IP-bound). Re-calling get_proxy_url()
# inside the page_action would rotate a pool-backed provider to a
# different IP and invalidate the token (03d proxy-coherence caveat).
proxy = get_proxy_url()
# Build the captcha page_action only when solving is enabled (and not
# process-latched). ``None`` => stealth tier behaves exactly as before.
page_action = None
if captcha_state is not None and captcha_enabled():
page_action = build_captcha_page_action(
captcha_state, proxy, get_captcha_config()
)
# ``solve_cloudflare=True`` runs the full Turnstile/Interstitial challenge
# loop; scoped to this last-resort tier only (it spins up the browser).
page = StealthyFetcher.fetch(
url,
headless=True,
network_idle=True,
block_ads=True,
solve_cloudflare=True,
proxy=get_proxy_url(),
)
# Scrapling runs solve_cloudflare BEFORE page_action, so Cloudflare is
# cleared first, then our captcha injector runs.
fetch_kwargs: dict[str, Any] = {
"headless": True,
"network_idle": True,
"block_ads": True,
"solve_cloudflare": True,
"proxy": proxy,
}
if page_action is not None:
fetch_kwargs["page_action"] = page_action
page = StealthyFetcher.fetch(url, **fetch_kwargs)
fetch_ms = (time.perf_counter() - fetch_start) * 1000
return self._build_result(
page.html_content,