feat(crawler): captcha solving + per-attempt billing (Phase 3d)

Wire captchatools as the StealthyFetcher-tier page_action to detect, harvest
(egressing from the crawl's own proxy IP), inject, and submit reCAPTCHA v2/v3
and hCaptcha tokens. Opt-in and off by default (zero attempts, zero cost).

Licensing split:
- Apache-2 app/utils/captcha/ holds the generic, vendor-agnostic config
  (CaptchaConfig + captcha_enabled() = flag AND key present).
- Proprietary app/proprietary/web_crawler/captcha.py holds the bypass logic
  (detect/harvest/inject) plus a process-wide solver latch that halts solving
  on unrecoverable errors (no balance / bad key).

Crawler: CrawlOutcome gains captcha_attempts/captcha_solved, surfaced via a
per-call captcha_state dict threaded crawl_url -> _crawl_with_stealthy(_sync)
and stamped onto every stealth terminal outcome. The stealth tier captures the
proxy once and reuses it for both the fetch and the solver (IP-coherence).

Billing: WebCrawlCreditService gains captcha_billing_enabled,
captcha_solves_to_micros, charge_captcha, and a generic check_balance, sharing
a single _apply_debit path. The indexer accumulates attempts (even on failed
crawls), runs a combined crawl+captcha pre-flight, and posts a per-attempt
owner charge as usage_type="web_crawl_captcha". The captcha worst-case is only
reserved when solving is actually enabled, so a solving-off deployment is never
blocked for captcha that can never run. Both chat scrape tools fold attempts
into the current turn before the success/fail branch.

Fully config-driven prices; no migration. New unit tests cover the config,
factory (detection/latch/timeout/cap), credit service, indexer wiring, and the
chat fold.

Co-authored-by: Cursor <cursoragent@cursor.com>

surfsense_backend/tests/unit/services/test_web_crawl_credit_service.py

@@ -213,3 +213,112 @@ class TestChatScrapeFold:
         # Run in a fresh context so the turn ContextVar resolves to its default
         # (None) regardless of other tests — must not raise.
         contextvars.Context().run(self._import_helper())
+
+
+# ===================================================================
+# F) Captcha per-attempt unit (Phase 3d)
+# ===================================================================
+
+
+@pytest.fixture
+def _enable_captcha_billing(monkeypatch):
+    monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_BILLING_ENABLED", True)
+    monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE", 3000)
+
+
+class TestCaptchaStatics:
+    def test_billing_enabled_reads_flag(self, monkeypatch):
+        monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_BILLING_ENABLED", True)
+        assert WebCrawlCreditService.captcha_billing_enabled() is True
+        monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_BILLING_ENABLED", False)
+        assert WebCrawlCreditService.captcha_billing_enabled() is False
+
+    def test_solves_to_micros_is_config_driven(self, monkeypatch):
+        monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE", 3000)
+        assert WebCrawlCreditService.captcha_solves_to_micros(1) == 3000
+        assert WebCrawlCreditService.captcha_solves_to_micros(1000) == 3_000_000
+        monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE", 5000)
+        assert WebCrawlCreditService.captcha_solves_to_micros(2) == 10_000
+
+
+class TestChargeCaptcha:
+    async def test_debits_per_attempt(self, _enable_captcha_billing):
+        session, user = _make_session(balance_micros=100_000)
+        svc = WebCrawlCreditService(session)
+        new_balance = await svc.charge_captcha(_USER_ID, attempts=2)
+        assert user.credit_micros_balance == 100_000 - 2 * 3000
+        assert new_balance == 94_000
+        session.commit.assert_awaited()
+
+    async def test_zero_attempts_is_noop(self, _enable_captcha_billing):
+        session, user = _make_session(balance_micros=100_000)
+        svc = WebCrawlCreditService(session)
+        assert await svc.charge_captcha(_USER_ID, attempts=0) is None
+        assert user.credit_micros_balance == 100_000
+        session.commit.assert_not_awaited()
+
+    async def test_disabled_is_noop(self, monkeypatch):
+        monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_BILLING_ENABLED", False)
+        session, user = _make_session(balance_micros=100_000)
+        svc = WebCrawlCreditService(session)
+        assert await svc.charge_captcha(_USER_ID, attempts=5) is None
+        assert user.credit_micros_balance == 100_000
+        session.execute.assert_not_called()
+
+
+class TestCheckBalance:
+    """Combined (crawl + captcha) pre-flight primitive — ungated."""
+
+    async def test_passes_when_affordable(self):
+        session, _user = _make_session(balance_micros=10_000)
+        await WebCrawlCreditService(session).check_balance(_USER_ID, 5_000)
+
+    async def test_raises_when_short(self):
+        session, _user = _make_session(balance_micros=3_000)
+        with pytest.raises(InsufficientCreditsError):
+            await WebCrawlCreditService(session).check_balance(_USER_ID, 5_000)
+
+    async def test_zero_required_is_noop(self):
+        session, _user = _make_session(balance_micros=0)
+        await WebCrawlCreditService(session).check_balance(_USER_ID, 0)
+        session.execute.assert_not_called()
+
+
+class TestCaptchaChatFold:
+    """The scrape tools fold captcha attempts into the live turn accumulator."""
+
+    def _import_helper(self):
+        from app.agents.chat.multi_agent_chat.main_agent.tools.scrape_webpage import (
+            _bill_captcha_attempts,
+        )
+
+        return _bill_captcha_attempts
+
+    def _outcome(self, attempts):
+        o = MagicMock()
+        o.captcha_attempts = attempts
+        return o
+
+    def test_folds_attempts_when_enabled(self, _enable_captcha_billing):
+        from app.services.token_tracking_service import start_turn
+
+        acc = start_turn()
+        self._import_helper()(self._outcome(2))
+        assert acc.total_cost_micros == 6000
+        assert len(acc.calls) == 1
+        assert acc.calls[0].call_kind == "web_crawl_captcha"
+
+    def test_noop_when_zero_attempts(self, _enable_captcha_billing):
+        from app.services.token_tracking_service import start_turn
+
+        acc = start_turn()
+        self._import_helper()(self._outcome(0))
+        assert acc.calls == []
+
+    def test_noop_when_billing_disabled(self, monkeypatch):
+        from app.services.token_tracking_service import start_turn
+
+        monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_BILLING_ENABLED", False)
+        acc = start_turn()
+        self._import_helper()(self._outcome(3))
+        assert acc.calls == []