feat(crawler): captcha solving + per-attempt billing (Phase 3d)

Wire captchatools as the StealthyFetcher-tier page_action to detect, harvest
(egressing from the crawl's own proxy IP), inject, and submit reCAPTCHA v2/v3
and hCaptcha tokens. Opt-in and off by default (zero attempts, zero cost).

Licensing split:
- Apache-2 app/utils/captcha/ holds the generic, vendor-agnostic config
  (CaptchaConfig + captcha_enabled() = flag AND key present).
- Proprietary app/proprietary/web_crawler/captcha.py holds the bypass logic
  (detect/harvest/inject) plus a process-wide solver latch that halts solving
  on unrecoverable errors (no balance / bad key).

Crawler: CrawlOutcome gains captcha_attempts/captcha_solved, surfaced via a
per-call captcha_state dict threaded crawl_url -> _crawl_with_stealthy(_sync)
and stamped onto every stealth terminal outcome. The stealth tier captures the
proxy once and reuses it for both the fetch and the solver (IP-coherence).

Billing: WebCrawlCreditService gains captcha_billing_enabled,
captcha_solves_to_micros, charge_captcha, and a generic check_balance, sharing
a single _apply_debit path. The indexer accumulates attempts (even on failed
crawls), runs a combined crawl+captcha pre-flight, and posts a per-attempt
owner charge as usage_type="web_crawl_captcha". The captcha worst-case is only
reserved when solving is actually enabled, so a solving-off deployment is never
blocked for captcha that can never run. Both chat scrape tools fold attempts
into the current turn before the success/fail branch.

Fully config-driven prices; no migration. New unit tests cover the config,
factory (detection/latch/timeout/cap), credit service, indexer wiring, and the
chat fold.

Co-authored-by: Cursor <cursoragent@cursor.com>

surfsense_backend/.env.example

@@ -266,6 +266,15 @@ MICROS_PER_PAGE=1000
 # WEB_CRAWL_CREDIT_BILLING_ENABLED=FALSE
 # WEB_CRAWL_MICROS_PER_SUCCESS=1000
 
+# Phase 3d: bill captcha solves as a SEPARATE per-attempt unit (the solver
+# charges per attempt regardless of crawl success). Independent of the crawl
+# flag above. Price config-driven (no hardcoded rate):
+#   WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE = round(USD_per_1000_solves * 1_000)
+#   3000 == $3 / 1000 solves (default) | 5000 == $5/1000
+# Set with margin over your solver vendor's per-attempt price.
+# WEB_CRAWL_CAPTCHA_BILLING_ENABLED=FALSE
+# WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE=3000
+
 # Low-balance warning threshold (micro-USD), surfaced to the UI. Default $0.50.
 CREDIT_LOW_BALANCE_WARNING_MICROS=500000
 
@@ -340,6 +349,26 @@ TURNSTILE_SECRET_KEY=
 # CUSTOM_PROXY_URL=http://user:pass@host:port
 # CUSTOM_PROXY_URLS=http://user:pass@host1:port,http://user:pass@host2:port
 
+# =====================================================================
+# Captcha solving (Phase 3d) — LAST-resort bypass tier via captchatools.
+# Only fires on the stealth browser tier when a sitekey is detected AND
+# CAPTCHA_SOLVING_ENABLED=TRUE. Cloudflare Turnstile is already solved free
+# in-framework (no config needed). Off by default => zero attempts, zero cost.
+# NOTE: automated solving may violate a target site's ToS — opt-in, public
+# data only (no logged-in bypass). captchatools is itself the vendor registry.
+# CAPTCHA_SOLVING_ENABLED=FALSE
+# solving_site: capmonster | 2captcha | anticaptcha | capsolver | captchaai
+# CAPTCHA_SOLVER_PROVIDER=capsolver
+# CAPTCHA_SOLVER_API_KEY=
+# Per-URL solve cap (bounds solver spend on a hostile page).
+# CAPTCHA_MAX_ATTEMPTS_PER_URL=1
+# CAPTCHA_SOLVE_TIMEOUT_S=120
+# Default type when detection is ambiguous: v2 | v3 | hcaptcha
+# CAPTCHA_TYPE_DEFAULT=v2
+# reCAPTCHA v3 tuning (only used for v3 challenges).
+# CAPTCHA_V3_MIN_SCORE=0.7
+# CAPTCHA_V3_ACTION=verify
+
 # File Parser Service
 ETL_SERVICE=UNSTRUCTURED or LLAMACLOUD or DOCLING
 UNSTRUCTURED_API_KEY=Tpu3P0U8iy