mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-06 22:12:12 +02:00
bdacea8b6e
Wire captchatools as the StealthyFetcher-tier page_action to detect, harvest (egressing from the crawl's own proxy IP), inject, and submit reCAPTCHA v2/v3 and hCaptcha tokens. Opt-in and off by default (zero attempts, zero cost). Licensing split: - Apache-2 app/utils/captcha/ holds the generic, vendor-agnostic config (CaptchaConfig + captcha_enabled() = flag AND key present). - Proprietary app/proprietary/web_crawler/captcha.py holds the bypass logic (detect/harvest/inject) plus a process-wide solver latch that halts solving on unrecoverable errors (no balance / bad key). Crawler: CrawlOutcome gains captcha_attempts/captcha_solved, surfaced via a per-call captcha_state dict threaded crawl_url -> _crawl_with_stealthy(_sync) and stamped onto every stealth terminal outcome. The stealth tier captures the proxy once and reuses it for both the fetch and the solver (IP-coherence). Billing: WebCrawlCreditService gains captcha_billing_enabled, captcha_solves_to_micros, charge_captcha, and a generic check_balance, sharing a single _apply_debit path. The indexer accumulates attempts (even on failed crawls), runs a combined crawl+captcha pre-flight, and posts a per-attempt owner charge as usage_type="web_crawl_captcha". The captcha worst-case is only reserved when solving is actually enabled, so a solving-off deployment is never blocked for captcha that can never run. Both chat scrape tools fold attempts into the current turn before the success/fail branch. Fully config-driven prices; no migration. New unit tests cover the config, factory (detection/latch/timeout/cap), credit service, indexer wiring, and the chat fold. Co-authored-by: Cursor <cursoragent@cursor.com>
22 lines
959 B
Python
22 lines
959 B
Python
"""App-wide captcha-solver configuration (Apache-2.0, generic glue).
|
|
|
|
This package holds only the **generic, vendor-agnostic** config resolution for
|
|
captcha solving — mirroring ``app/utils/proxy/`` (which stays Apache-2.0 even
|
|
though the crawler that consumes it is proprietary). The actual bypass logic
|
|
(challenge detection + token injection ``page_action``) lives in the separately
|
|
licensed ``app/proprietary/web_crawler/captcha.py``.
|
|
|
|
``captchatools`` is itself the multi-vendor registry (capmonster / 2captcha /
|
|
anticaptcha / capsolver / captchaai), so there is no provider hierarchy here —
|
|
just one env-driven :class:`CaptchaConfig` and a cheap ``captcha_enabled()``
|
|
gate callers check before constructing anything (mirrors
|
|
``WebCrawlCreditService.billing_enabled()``).
|
|
"""
|
|
|
|
from app.utils.captcha.config import (
|
|
CaptchaConfig,
|
|
captcha_enabled,
|
|
get_captcha_config,
|
|
)
|
|
|
|
__all__ = ["CaptchaConfig", "captcha_enabled", "get_captcha_config"]
|