feat(crawler): captcha solving + per-attempt billing (Phase 3d)

Wire captchatools as the StealthyFetcher-tier page_action to detect, harvest
(egressing from the crawl's own proxy IP), inject, and submit reCAPTCHA v2/v3
and hCaptcha tokens. Opt-in and off by default (zero attempts, zero cost).

Licensing split:
- Apache-2 app/utils/captcha/ holds the generic, vendor-agnostic config
  (CaptchaConfig + captcha_enabled() = flag AND key present).
- Proprietary app/proprietary/web_crawler/captcha.py holds the bypass logic
  (detect/harvest/inject) plus a process-wide solver latch that halts solving
  on unrecoverable errors (no balance / bad key).

Crawler: CrawlOutcome gains captcha_attempts/captcha_solved, surfaced via a
per-call captcha_state dict threaded crawl_url -> _crawl_with_stealthy(_sync)
and stamped onto every stealth terminal outcome. The stealth tier captures the
proxy once and reuses it for both the fetch and the solver (IP-coherence).

Billing: WebCrawlCreditService gains captcha_billing_enabled,
captcha_solves_to_micros, charge_captcha, and a generic check_balance, sharing
a single _apply_debit path. The indexer accumulates attempts (even on failed
crawls), runs a combined crawl+captcha pre-flight, and posts a per-attempt
owner charge as usage_type="web_crawl_captcha". The captcha worst-case is only
reserved when solving is actually enabled, so a solving-off deployment is never
blocked for captcha that can never run. Both chat scrape tools fold attempts
into the current turn before the success/fail branch.

Fully config-driven prices; no migration. New unit tests cover the config,
factory (detection/latch/timeout/cap), credit service, indexer wiring, and the
chat fold.

Co-authored-by: Cursor <cursoragent@cursor.com>

surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/tools/scrape_webpage.py

@@ -55,6 +55,36 @@ def _bill_successful_scrape() -> None:
     )
 
 
+def _bill_captcha_attempts(outcome) -> None:
+    """Fold captcha solve attempts (Phase 3d) into the current chat turn's bill.
+
+    Per *attempt*, not per success: a solve that didn't rescue the crawl still
+    cost real solver money, so this runs before the success/failure branch.
+    Mirrors :func:`_bill_successful_scrape` (rides the turn accumulator, settles
+    at ``finalize_credit``; record-only on free/anonymous turns). No-op when
+    captcha billing is off, there were no attempts, or no turn is active.
+    """
+    from app.services.token_tracking_service import get_current_accumulator
+    from app.services.web_crawl_credit_service import WebCrawlCreditService
+
+    if not WebCrawlCreditService.captcha_billing_enabled():
+        return
+    attempts = getattr(outcome, "captcha_attempts", 0) or 0
+    if attempts <= 0:
+        return
+    acc = get_current_accumulator()
+    if acc is None:
+        return
+    acc.add(
+        model="web_crawl_captcha",
+        prompt_tokens=0,
+        completion_tokens=0,
+        total_tokens=0,
+        cost_micros=WebCrawlCreditService.captcha_solves_to_micros(attempts),
+        call_kind="web_crawl_captcha",
+    )
+
+
 def extract_domain(url: str) -> str:
     """Extract the domain from a URL."""
     try:
@@ -258,6 +288,9 @@ def create_scrape_webpage_tool():
             connector = WebCrawlerConnector()
             outcome = await connector.crawl_url(url)
 
+            # 03d: bill any captcha attempts (even if the crawl ultimately failed).
+            _bill_captcha_attempts(outcome)
+
             if outcome.status is not CrawlOutcomeStatus.SUCCESS or not outcome.result:
                 return {
                     "id": scrape_id,

surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/tools/scrape_webpage.py

@@ -49,6 +49,36 @@ def _bill_successful_scrape() -> None:
     )
 
 
+def _bill_captcha_attempts(outcome) -> None:
+    """Fold captcha solve attempts (Phase 3d) into the current chat turn's bill.
+
+    Per *attempt*, not per success: a solve that didn't rescue the crawl still
+    cost real solver money, so this runs before the success/failure branch.
+    Mirrors :func:`_bill_successful_scrape` (rides the turn accumulator, settles
+    at ``finalize_credit``; record-only on free/anonymous turns). No-op when
+    captcha billing is off, there were no attempts, or no turn is active.
+    """
+    from app.services.token_tracking_service import get_current_accumulator
+    from app.services.web_crawl_credit_service import WebCrawlCreditService
+
+    if not WebCrawlCreditService.captcha_billing_enabled():
+        return
+    attempts = getattr(outcome, "captcha_attempts", 0) or 0
+    if attempts <= 0:
+        return
+    acc = get_current_accumulator()
+    if acc is None:
+        return
+    acc.add(
+        model="web_crawl_captcha",
+        prompt_tokens=0,
+        completion_tokens=0,
+        total_tokens=0,
+        cost_micros=WebCrawlCreditService.captcha_solves_to_micros(attempts),
+        call_kind="web_crawl_captcha",
+    )
+
+
 def extract_domain(url: str) -> str:
     """Extract the domain from a URL."""
     try:
@@ -252,6 +282,9 @@ def create_scrape_webpage_tool():
             connector = WebCrawlerConnector()
             outcome = await connector.crawl_url(url)
 
+            # 03d: bill any captcha attempts (even if the crawl ultimately failed).
+            _bill_captcha_attempts(outcome)
+
             if outcome.status is not CrawlOutcomeStatus.SUCCESS or not outcome.result:
                 return {
                     "id": scrape_id,