feat(crawler): captcha solving + per-attempt billing (Phase 3d)

Wire captchatools as the StealthyFetcher-tier page_action to detect, harvest
(egressing from the crawl's own proxy IP), inject, and submit reCAPTCHA v2/v3
and hCaptcha tokens. Opt-in and off by default (zero attempts, zero cost).

Licensing split:
- Apache-2 app/utils/captcha/ holds the generic, vendor-agnostic config
  (CaptchaConfig + captcha_enabled() = flag AND key present).
- Proprietary app/proprietary/web_crawler/captcha.py holds the bypass logic
  (detect/harvest/inject) plus a process-wide solver latch that halts solving
  on unrecoverable errors (no balance / bad key).

Crawler: CrawlOutcome gains captcha_attempts/captcha_solved, surfaced via a
per-call captcha_state dict threaded crawl_url -> _crawl_with_stealthy(_sync)
and stamped onto every stealth terminal outcome. The stealth tier captures the
proxy once and reuses it for both the fetch and the solver (IP-coherence).

Billing: WebCrawlCreditService gains captcha_billing_enabled,
captcha_solves_to_micros, charge_captcha, and a generic check_balance, sharing
a single _apply_debit path. The indexer accumulates attempts (even on failed
crawls), runs a combined crawl+captcha pre-flight, and posts a per-attempt
owner charge as usage_type="web_crawl_captcha". The captcha worst-case is only
reserved when solving is actually enabled, so a solving-off deployment is never
blocked for captcha that can never run. Both chat scrape tools fold attempts
into the current turn before the success/fail branch.

Fully config-driven prices; no migration. New unit tests cover the config,
factory (detection/latch/timeout/cap), credit service, indexer wiring, and the
chat fold.

Co-authored-by: Cursor <cursoragent@cursor.com>

surfsense_backend/tests/unit/utils/test_captcha_config.py

@@ -0,0 +1,47 @@
+"""Unit tests for the generic captcha config gate (Phase 3d, Apache-2 layer)."""
+
+import pytest
+
+from app.config import config
+from app.utils.captcha import captcha_enabled, get_captcha_config
+
+pytestmark = pytest.mark.unit
+
+
+class TestCaptchaEnabled:
+    def test_off_by_default(self, monkeypatch):
+        monkeypatch.setattr(config, "CAPTCHA_SOLVING_ENABLED", False)
+        monkeypatch.setattr(config, "CAPTCHA_SOLVER_API_KEY", "key")
+        assert captcha_enabled() is False
+
+    def test_flag_on_but_no_key_is_disabled(self, monkeypatch):
+        monkeypatch.setattr(config, "CAPTCHA_SOLVING_ENABLED", True)
+        monkeypatch.setattr(config, "CAPTCHA_SOLVER_API_KEY", None)
+        assert captcha_enabled() is False
+
+    def test_flag_on_with_key_is_enabled(self, monkeypatch):
+        monkeypatch.setattr(config, "CAPTCHA_SOLVING_ENABLED", True)
+        monkeypatch.setattr(config, "CAPTCHA_SOLVER_API_KEY", "key")
+        assert captcha_enabled() is True
+
+
+class TestGetCaptchaConfig:
+    def test_snapshot_reflects_config(self, monkeypatch):
+        monkeypatch.setattr(config, "CAPTCHA_SOLVING_ENABLED", True)
+        monkeypatch.setattr(config, "CAPTCHA_SOLVER_API_KEY", "key")
+        monkeypatch.setattr(config, "CAPTCHA_SOLVER_PROVIDER", "2captcha")
+        monkeypatch.setattr(config, "CAPTCHA_MAX_ATTEMPTS_PER_URL", 3)
+        monkeypatch.setattr(config, "CAPTCHA_SOLVE_TIMEOUT_S", 90)
+        monkeypatch.setattr(config, "CAPTCHA_TYPE_DEFAULT", "hcaptcha")
+
+        cfg = get_captcha_config()
+        assert cfg.enabled is True
+        assert cfg.solving_site == "2captcha"
+        assert cfg.max_attempts_per_url == 3
+        assert cfg.timeout_s == 90
+        assert cfg.captcha_type_default == "hcaptcha"
+
+    def test_keyless_config_is_not_enabled(self, monkeypatch):
+        monkeypatch.setattr(config, "CAPTCHA_SOLVING_ENABLED", True)
+        monkeypatch.setattr(config, "CAPTCHA_SOLVER_API_KEY", None)
+        assert get_captcha_config().enabled is False