feat(crawler): captcha solving + per-attempt billing (Phase 3d)

Wire captchatools as the StealthyFetcher-tier page_action to detect, harvest
(egressing from the crawl's own proxy IP), inject, and submit reCAPTCHA v2/v3
and hCaptcha tokens. Opt-in and off by default (zero attempts, zero cost).

Licensing split:
- Apache-2 app/utils/captcha/ holds the generic, vendor-agnostic config
  (CaptchaConfig + captcha_enabled() = flag AND key present).
- Proprietary app/proprietary/web_crawler/captcha.py holds the bypass logic
  (detect/harvest/inject) plus a process-wide solver latch that halts solving
  on unrecoverable errors (no balance / bad key).

Crawler: CrawlOutcome gains captcha_attempts/captcha_solved, surfaced via a
per-call captcha_state dict threaded crawl_url -> _crawl_with_stealthy(_sync)
and stamped onto every stealth terminal outcome. The stealth tier captures the
proxy once and reuses it for both the fetch and the solver (IP-coherence).

Billing: WebCrawlCreditService gains captcha_billing_enabled,
captcha_solves_to_micros, charge_captcha, and a generic check_balance, sharing
a single _apply_debit path. The indexer accumulates attempts (even on failed
crawls), runs a combined crawl+captcha pre-flight, and posts a per-attempt
owner charge as usage_type="web_crawl_captcha". The captcha worst-case is only
reserved when solving is actually enabled, so a solving-off deployment is never
blocked for captcha that can never run. Both chat scrape tools fold attempts
into the current turn before the success/fail branch.

Fully config-driven prices; no migration. New unit tests cover the config,
factory (detection/latch/timeout/cap), credit service, indexer wiring, and the
chat fold.

Co-authored-by: Cursor <cursoragent@cursor.com>

plans/backend/03d-captcha-solving.md

@@ -1,8 +1,13 @@
 # Phase 3d — Captcha solving (reCAPTCHA / hCaptcha / image)
 
 > Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
-> **Status: ACTIVE (sequenced last in Phase 3).** No longer deferred — captcha bypass is now an explicit goal of the "undetectable, captcha-bypassing universal WebURL crawler." Build **after** `03a` (StealthyFetcher tier + `FetchStrategy` seam + `CrawlOutcome`), `03b` (app-wide proxy provider, accessed via the zero-arg `get_proxy_url()`), `03c` (crawl wallet), and `03e` (stealth-hardening — solving is the *last* resort once humanization + fingerprint coherence have minimized challenges).
-> **Decision recap (this pass):** Cloudflare stays in-framework (free, `03a`); reCAPTCHA/hCaptcha/image use a **paid** solver via `captchatools`, **opt-in + off by default**, and are billed as a **separate per-attempt unit** (option (a) below — now decided, not deferred). Logged-in/account-gated captcha flows are **out of scope** (no authenticated scraping in this MVP — see umbrella Phase 8 / `03b` static-proxy hand-off).
+> **Status: ✅ IMPLEMENTED (`ci_mvp`, captcha solving + per-attempt billing).** `captchatools==1.5.0` (MIT, pure-Python, single `requests` dep) wired as the StealthyFetcher-tier `page_action`. Generic config in Apache-2 `app/utils/captcha/` (`CaptchaConfig` + `captcha_enabled()` = flag **and** key); bypass logic (detect/harvest/inject + process latch) in proprietary `app/proprietary/web_crawler/captcha.py`. `CrawlOutcome` gained `captcha_attempts`/`captcha_solved`; the stealth tier captures the proxy **once** and reuses it for the solver (IP-coherence). Billing: `WebCrawlCreditService.captcha_solves_to_micros` + `charge_captcha` + generic `check_balance`; indexer accumulates attempts (even on failed crawls), combined crawl+captcha pre-flight, per-attempt owner charge with `usage_type="web_crawl_captcha"`; both chat scrape tools fold attempts into the turn before the success/fail branch. **No migration.** Off by default (zero attempts, zero cost). 40+ new unit tests pass.
+> **Implementation deltas from the original draft (the draft predated the real 03a code):**
+> - **There is no `FetchStrategy` class/seam** in the shipped 03a crawler — tiers are plain methods (`_crawl_with_async_fetcher`/`_crawl_with_dynamic`/`_crawl_with_stealthy`) that call `get_proxy_url()` inline. The "surface the chosen endpoint" seam is implemented concretely by capturing `proxy = get_proxy_url()` **once** in `_crawl_with_stealthy_sync` and passing it to both `StealthyFetcher.fetch(proxy=…)` and the factory (no re-rotation).
+> - **Attempts reach `CrawlOutcome` via a per-call `captcha_state` dict** threaded `crawl_url → _crawl_with_stealthy(_sync)` and stamped onto every stealth terminal outcome (success/empty/failed). Per-call (not on `self`) → concurrency-safe.
+> - **Solver timeout uses a non-context-managed `ThreadPoolExecutor`** + `shutdown(wait=False, cancel_futures=True)`; a `with` block would join the worker and defeat the timeout.
+> - **Combined pre-flight** lives on the indexer using the new generic `check_balance(required_micros)` (crawl successes + worst-case `len(urls) × MAX_ATTEMPTS` captcha), not a captcha-specific check method. The captcha worst-case is only reserved when **both** `captcha_billing_enabled()` **and** `captcha_enabled()` (solving actually on) — with solving off, attempts can never happen, so reserving would wrongly block a run for captcha that will never run (regression-tested).
+> **Decision recap (this pass):** Cloudflare stays in-framework (free, `03a`); reCAPTCHA/hCaptcha/image use a **paid** solver via `captchatools`, **opt-in + off by default**, and are billed as a **separate per-attempt unit** (option (a) below). Logged-in/account-gated captcha flows are **out of scope** (no authenticated scraping in this MVP — see umbrella Phase 8 / `03b` static-proxy hand-off).
 
 ## Why this is the last tier, not the first