feat(crawler): captcha solving + per-attempt billing (Phase 3d)

Wire captchatools as the StealthyFetcher-tier page_action to detect, harvest
(egressing from the crawl's own proxy IP), inject, and submit reCAPTCHA v2/v3
and hCaptcha tokens. Opt-in and off by default (zero attempts, zero cost).

Licensing split:
- Apache-2 app/utils/captcha/ holds the generic, vendor-agnostic config
  (CaptchaConfig + captcha_enabled() = flag AND key present).
- Proprietary app/proprietary/web_crawler/captcha.py holds the bypass logic
  (detect/harvest/inject) plus a process-wide solver latch that halts solving
  on unrecoverable errors (no balance / bad key).

Crawler: CrawlOutcome gains captcha_attempts/captcha_solved, surfaced via a
per-call captcha_state dict threaded crawl_url -> _crawl_with_stealthy(_sync)
and stamped onto every stealth terminal outcome. The stealth tier captures the
proxy once and reuses it for both the fetch and the solver (IP-coherence).

Billing: WebCrawlCreditService gains captcha_billing_enabled,
captcha_solves_to_micros, charge_captcha, and a generic check_balance, sharing
a single _apply_debit path. The indexer accumulates attempts (even on failed
crawls), runs a combined crawl+captcha pre-flight, and posts a per-attempt
owner charge as usage_type="web_crawl_captcha". The captcha worst-case is only
reserved when solving is actually enabled, so a solving-off deployment is never
blocked for captcha that can never run. Both chat scrape tools fold attempts
into the current turn before the success/fail branch.

Fully config-driven prices; no migration. New unit tests cover the config,
factory (detection/latch/timeout/cap), credit service, indexer wiring, and the
chat fold.

Co-authored-by: Cursor <cursoragent@cursor.com>

surfsense_backend/app/utils/captcha/__init__.py

@@ -0,0 +1,22 @@
+"""App-wide captcha-solver configuration (Apache-2.0, generic glue).
+
+This package holds only the **generic, vendor-agnostic** config resolution for
+captcha solving — mirroring ``app/utils/proxy/`` (which stays Apache-2.0 even
+though the crawler that consumes it is proprietary). The actual bypass logic
+(challenge detection + token injection ``page_action``) lives in the separately
+licensed ``app/proprietary/web_crawler/captcha.py``.
+
+``captchatools`` is itself the multi-vendor registry (capmonster / 2captcha /
+anticaptcha / capsolver / captchaai), so there is no provider hierarchy here —
+just one env-driven :class:`CaptchaConfig` and a cheap ``captcha_enabled()``
+gate callers check before constructing anything (mirrors
+``WebCrawlCreditService.billing_enabled()``).
+"""
+
+from app.utils.captcha.config import (
+    CaptchaConfig,
+    captcha_enabled,
+    get_captcha_config,
+)
+
+__all__ = ["CaptchaConfig", "captcha_enabled", "get_captcha_config"]

surfsense_backend/app/utils/captcha/config.py

@@ -0,0 +1,54 @@
+"""Env-resolved captcha-solver config (one app-wide config; no per-connector).
+
+Resolved from :data:`app.config.config` only, mirroring ``03b``'s single
+``PROXY_PROVIDER`` model. Off by default: ``captcha_enabled()`` is ``False``
+unless both ``CAPTCHA_SOLVING_ENABLED=TRUE`` and an API key are present, so a
+misconfigured deployment (flag on, key missing) never attempts a paid solve.
+"""
+
+from dataclasses import dataclass
+
+from app.config import config
+
+
+@dataclass(frozen=True)
+class CaptchaConfig:
+    """Immutable snapshot of the captcha-solver settings for one crawl."""
+
+    enabled: bool
+    solving_site: str
+    api_key: str | None
+    max_attempts_per_url: int
+    timeout_s: int
+    captcha_type_default: str
+    v3_min_score: float
+    v3_action: str
+
+
+def get_captcha_config() -> CaptchaConfig:
+    """Build a :class:`CaptchaConfig` from the current process config/env."""
+    api_key = config.CAPTCHA_SOLVER_API_KEY
+    flag_on = config.CAPTCHA_SOLVING_ENABLED
+    return CaptchaConfig(
+        # Effective-enabled requires a key too: a flag with no key would only
+        # produce ``ErrWrongAPIKey`` per attempt (and still risk an IP ban from
+        # the solver), so treat key-less config as disabled.
+        enabled=bool(flag_on and api_key),
+        solving_site=config.CAPTCHA_SOLVER_PROVIDER,
+        api_key=api_key,
+        max_attempts_per_url=config.CAPTCHA_MAX_ATTEMPTS_PER_URL,
+        timeout_s=config.CAPTCHA_SOLVE_TIMEOUT_S,
+        captcha_type_default=config.CAPTCHA_TYPE_DEFAULT,
+        v3_min_score=config.CAPTCHA_V3_MIN_SCORE,
+        v3_action=config.CAPTCHA_V3_ACTION,
+    )
+
+
+def captcha_enabled() -> bool:
+    """Cheap gate: is captcha solving effectively configured (flag + key)?
+
+    Callers check this before building the ``page_action`` or capturing the
+    crawl's proxy endpoint for the solver, so the stealth tier stays unchanged
+    when solving is off.
+    """
+    return bool(config.CAPTCHA_SOLVING_ENABLED and config.CAPTCHA_SOLVER_API_KEY)