feat(crawler): captcha solving + per-attempt billing (Phase 3d)
Wire captchatools as the StealthyFetcher-tier page_action to detect, harvest
(egressing from the crawl's own proxy IP), inject, and submit reCAPTCHA v2/v3
and hCaptcha tokens. Opt-in and off by default (zero attempts, zero cost).
Licensing split:
- Apache-2 app/utils/captcha/ holds the generic, vendor-agnostic config
(CaptchaConfig + captcha_enabled() = flag AND key present).
- Proprietary app/proprietary/web_crawler/captcha.py holds the bypass logic
(detect/harvest/inject) plus a process-wide solver latch that halts solving
on unrecoverable errors (no balance / bad key).
Crawler: CrawlOutcome gains captcha_attempts/captcha_solved, surfaced via a
per-call captcha_state dict threaded crawl_url -> _crawl_with_stealthy(_sync)
and stamped onto every stealth terminal outcome. The stealth tier captures the
proxy once and reuses it for both the fetch and the solver (IP-coherence).
Billing: WebCrawlCreditService gains captcha_billing_enabled,
captcha_solves_to_micros, charge_captcha, and a generic check_balance, sharing
a single _apply_debit path. The indexer accumulates attempts (even on failed
crawls), runs a combined crawl+captcha pre-flight, and posts a per-attempt
owner charge as usage_type="web_crawl_captcha". The captcha worst-case is only
reserved when solving is actually enabled, so a solving-off deployment is never
blocked for captcha that can never run. Both chat scrape tools fold attempts
into the current turn before the success/fail branch.
Fully config-driven prices; no migration. New unit tests cover the config,
factory (detection/latch/timeout/cap), credit service, indexer wiring, and the
chat fold.
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-29 22:58:21 -07:00
|
|
|
"""App-wide captcha-solver configuration (Apache-2.0, generic glue).
|
|
|
|
|
|
2026-07-17 03:27:43 -07:00
|
|
|
This package holds the **generic, vendor-agnostic** config resolution for
|
|
|
|
|
captcha solving (:class:`CaptchaConfig`) plus the in-house vendor API clients
|
|
|
|
|
(:mod:`app.utils.captcha.solvers`) — mirroring ``app/utils/proxy/`` (which stays
|
|
|
|
|
Apache-2.0 even though the crawler that consumes it is proprietary). The actual
|
|
|
|
|
bypass logic (challenge detection + token injection ``page_action``) lives in
|
|
|
|
|
the separately licensed ``app/proprietary/web_crawler/captcha.py`` and
|
|
|
|
|
``app/proprietary/platforms/google_search/captcha.py``.
|
feat(crawler): captcha solving + per-attempt billing (Phase 3d)
Wire captchatools as the StealthyFetcher-tier page_action to detect, harvest
(egressing from the crawl's own proxy IP), inject, and submit reCAPTCHA v2/v3
and hCaptcha tokens. Opt-in and off by default (zero attempts, zero cost).
Licensing split:
- Apache-2 app/utils/captcha/ holds the generic, vendor-agnostic config
(CaptchaConfig + captcha_enabled() = flag AND key present).
- Proprietary app/proprietary/web_crawler/captcha.py holds the bypass logic
(detect/harvest/inject) plus a process-wide solver latch that halts solving
on unrecoverable errors (no balance / bad key).
Crawler: CrawlOutcome gains captcha_attempts/captcha_solved, surfaced via a
per-call captcha_state dict threaded crawl_url -> _crawl_with_stealthy(_sync)
and stamped onto every stealth terminal outcome. The stealth tier captures the
proxy once and reuses it for both the fetch and the solver (IP-coherence).
Billing: WebCrawlCreditService gains captcha_billing_enabled,
captcha_solves_to_micros, charge_captcha, and a generic check_balance, sharing
a single _apply_debit path. The indexer accumulates attempts (even on failed
crawls), runs a combined crawl+captcha pre-flight, and posts a per-attempt
owner charge as usage_type="web_crawl_captcha". The captcha worst-case is only
reserved when solving is actually enabled, so a solving-off deployment is never
blocked for captcha that can never run. Both chat scrape tools fold attempts
into the current turn before the success/fail branch.
Fully config-driven prices; no migration. New unit tests cover the config,
factory (detection/latch/timeout/cap), credit service, indexer wiring, and the
chat fold.
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-29 22:58:21 -07:00
|
|
|
|
2026-07-17 03:27:43 -07:00
|
|
|
``solvers.solve()`` dispatches on ``CAPTCHA_SOLVER_PROVIDER``; only 2captcha is
|
|
|
|
|
wired today, more vendors are added progressively. ``captcha_enabled()`` is the
|
|
|
|
|
cheap gate callers check before constructing anything (mirrors
|
feat(crawler): captcha solving + per-attempt billing (Phase 3d)
Wire captchatools as the StealthyFetcher-tier page_action to detect, harvest
(egressing from the crawl's own proxy IP), inject, and submit reCAPTCHA v2/v3
and hCaptcha tokens. Opt-in and off by default (zero attempts, zero cost).
Licensing split:
- Apache-2 app/utils/captcha/ holds the generic, vendor-agnostic config
(CaptchaConfig + captcha_enabled() = flag AND key present).
- Proprietary app/proprietary/web_crawler/captcha.py holds the bypass logic
(detect/harvest/inject) plus a process-wide solver latch that halts solving
on unrecoverable errors (no balance / bad key).
Crawler: CrawlOutcome gains captcha_attempts/captcha_solved, surfaced via a
per-call captcha_state dict threaded crawl_url -> _crawl_with_stealthy(_sync)
and stamped onto every stealth terminal outcome. The stealth tier captures the
proxy once and reuses it for both the fetch and the solver (IP-coherence).
Billing: WebCrawlCreditService gains captcha_billing_enabled,
captcha_solves_to_micros, charge_captcha, and a generic check_balance, sharing
a single _apply_debit path. The indexer accumulates attempts (even on failed
crawls), runs a combined crawl+captcha pre-flight, and posts a per-attempt
owner charge as usage_type="web_crawl_captcha". The captcha worst-case is only
reserved when solving is actually enabled, so a solving-off deployment is never
blocked for captcha that can never run. Both chat scrape tools fold attempts
into the current turn before the success/fail branch.
Fully config-driven prices; no migration. New unit tests cover the config,
factory (detection/latch/timeout/cap), credit service, indexer wiring, and the
chat fold.
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-29 22:58:21 -07:00
|
|
|
``WebCrawlCreditService.billing_enabled()``).
|
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
from app.utils.captcha.config import (
|
|
|
|
|
CaptchaConfig,
|
|
|
|
|
captcha_enabled,
|
|
|
|
|
get_captcha_config,
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
__all__ = ["CaptchaConfig", "captcha_enabled", "get_captcha_config"]
|