feat(crawler): captcha solving + per-attempt billing (Phase 3d)

Wire captchatools as the StealthyFetcher-tier page_action to detect, harvest
(egressing from the crawl's own proxy IP), inject, and submit reCAPTCHA v2/v3
and hCaptcha tokens. Opt-in and off by default (zero attempts, zero cost).

Licensing split:
- Apache-2 app/utils/captcha/ holds the generic, vendor-agnostic config
  (CaptchaConfig + captcha_enabled() = flag AND key present).
- Proprietary app/proprietary/web_crawler/captcha.py holds the bypass logic
  (detect/harvest/inject) plus a process-wide solver latch that halts solving
  on unrecoverable errors (no balance / bad key).

Crawler: CrawlOutcome gains captcha_attempts/captcha_solved, surfaced via a
per-call captcha_state dict threaded crawl_url -> _crawl_with_stealthy(_sync)
and stamped onto every stealth terminal outcome. The stealth tier captures the
proxy once and reuses it for both the fetch and the solver (IP-coherence).

Billing: WebCrawlCreditService gains captcha_billing_enabled,
captcha_solves_to_micros, charge_captcha, and a generic check_balance, sharing
a single _apply_debit path. The indexer accumulates attempts (even on failed
crawls), runs a combined crawl+captcha pre-flight, and posts a per-attempt
owner charge as usage_type="web_crawl_captcha". The captcha worst-case is only
reserved when solving is actually enabled, so a solving-off deployment is never
blocked for captcha that can never run. Both chat scrape tools fold attempts
into the current turn before the success/fail branch.

Fully config-driven prices; no migration. New unit tests cover the config,
factory (detection/latch/timeout/cap), credit service, indexer wiring, and the
chat fold.

Co-authored-by: Cursor <cursoragent@cursor.com>

surfsense_backend/app/services/web_crawl_credit_service.py

@@ -51,15 +51,31 @@ class WebCrawlCreditService:
         """
         return int(successes) * config.WEB_CRAWL_MICROS_PER_SUCCESS
 
-    async def get_available_micros(self, user_id: str | UUID) -> int | None:
-        """Return spendable credit in micro-USD (``balance - reserved``).
+    @staticmethod
+    def captcha_billing_enabled() -> bool:
+        """Phase 3d: whether captcha *solves* are metered.
 
-        Returns ``None`` when crawl billing is disabled, which callers treat as
-        "unlimited" (no blocking, no charge).
+        Independent of crawl billing: a deployment may bill solves without
+        billing crawls (or vice-versa). Off by default.
         """
-        if not config.WEB_CRAWL_CREDIT_BILLING_ENABLED:
-            return None
+        return config.WEB_CRAWL_CAPTCHA_BILLING_ENABLED
 
+    @staticmethod
+    def captcha_solves_to_micros(attempts: int) -> int:
+        """Convert a captcha *attempt* count to USD micro-credits.
+
+        Reads ``config.WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE`` (single source of
+        truth). Charged per attempt — not per success — because the solver
+        vendor bills every attempt regardless of crawl outcome.
+        """
+        return int(attempts) * config.WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE
+
+    async def _spendable_micros(self, user_id: str | UUID) -> int:
+        """Raw ``balance - reserved`` read, **ungated** by any billing flag.
+
+        Used by :meth:`check_balance` for combined (crawl + captcha) pre-flight,
+        where the relevant gate is decided by the caller, not by a single flag.
+        """
         from app.db import User
 
         result = await self.session.execute(
@@ -74,6 +90,39 @@ class WebCrawlCreditService:
         balance, reserved = row
         return balance - reserved
 
+    async def get_available_micros(self, user_id: str | UUID) -> int | None:
+        """Return spendable credit in micro-USD (``balance - reserved``).
+
+        Returns ``None`` when crawl billing is disabled, which callers treat as
+        "unlimited" (no blocking, no charge).
+        """
+        if not config.WEB_CRAWL_CREDIT_BILLING_ENABLED:
+            return None
+        return await self._spendable_micros(user_id)
+
+    async def check_balance(self, user_id: str | UUID, required_micros: int) -> None:
+        """Raise :class:`InsufficientCreditsError` if the wallet can't cover
+        ``required_micros`` (a combined crawl + worst-case captcha estimate).
+
+        Generic and **ungated** — the caller computes ``required_micros`` from
+        whichever billers are enabled and only calls this when at least one is.
+        No-op for a non-positive requirement.
+        """
+        if required_micros <= 0:
+            return
+        available = await self._spendable_micros(user_id)
+        if required_micros > available:
+            raise InsufficientCreditsError(
+                message=(
+                    "This run would exceed your available credit. "
+                    f"Available: ${available / 1_000_000:.2f}, "
+                    f"estimated need: ${required_micros / 1_000_000:.2f}. "
+                    "Add more credits to continue."
+                ),
+                balance_micros=available,
+                required_micros=required_micros,
+            )
+
     async def check_credits(
         self, user_id: str | UUID, estimated_successes: int = 1
     ) -> None:
@@ -104,20 +153,14 @@ class WebCrawlCreditService:
                 required_micros=required,
             )
 
-    async def charge_credits(
-        self, user_id: str | UUID, successes: int
-    ) -> int | None:
-        """Debit the wallet for ``successes`` successful crawls.
+    async def _apply_debit(self, user_id: str | UUID, cost_micros: int) -> int | None:
+        """Debit ``cost_micros`` from the wallet and commit (shared by all
+        charge paths). Flushes any audit row the caller staged before this.
 
-        Commits the session (mirroring ``EtlCreditService.charge_credits``),
-        which also flushes any audit row staged by the caller before this call.
-        No-op when billing is disabled or ``successes <= 0``.
-
-        Returns the new balance in micros, or ``None`` when nothing was charged.
+        Mirrors ``EtlCreditService.charge_credits``' commit-then-refresh +
+        best-effort auto-reload. No-op for a non-positive cost.
         """
-        if not config.WEB_CRAWL_CREDIT_BILLING_ENABLED:
-            return None
-        if successes <= 0:
+        if cost_micros <= 0:
             return None
 
         from app.db import User
@@ -127,8 +170,7 @@ class WebCrawlCreditService:
         if not user:
             raise ValueError(f"User with ID {user_id} not found")
 
-        cost = self.successes_to_micros(successes)
-        user.credit_micros_balance -= cost
+        user.credit_micros_balance -= cost_micros
         await self.session.commit()
         await self.session.refresh(user)
 
@@ -141,3 +183,32 @@ class WebCrawlCreditService:
             pass
 
         return user.credit_micros_balance
+
+    async def charge_credits(
+        self, user_id: str | UUID, successes: int
+    ) -> int | None:
+        """Debit the wallet for ``successes`` successful crawls.
+
+        No-op when crawl billing is disabled or ``successes <= 0``. Returns the
+        new balance in micros, or ``None`` when nothing was charged.
+        """
+        if not config.WEB_CRAWL_CREDIT_BILLING_ENABLED:
+            return None
+        if successes <= 0:
+            return None
+        return await self._apply_debit(user_id, self.successes_to_micros(successes))
+
+    async def charge_captcha(
+        self, user_id: str | UUID, attempts: int
+    ) -> int | None:
+        """Debit the wallet for ``attempts`` captcha solves (Phase 3d).
+
+        Per-attempt (not per-success): the solver charges for every attempt even
+        when the crawl ultimately fails. No-op when captcha billing is disabled
+        or ``attempts <= 0``.
+        """
+        if not config.WEB_CRAWL_CAPTCHA_BILLING_ENABLED:
+            return None
+        if attempts <= 0:
+            return None
+        return await self._apply_debit(user_id, self.captcha_solves_to_micros(attempts))