mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-06 22:12:12 +02:00
bdacea8b6e
Wire captchatools as the StealthyFetcher-tier page_action to detect, harvest (egressing from the crawl's own proxy IP), inject, and submit reCAPTCHA v2/v3 and hCaptcha tokens. Opt-in and off by default (zero attempts, zero cost). Licensing split: - Apache-2 app/utils/captcha/ holds the generic, vendor-agnostic config (CaptchaConfig + captcha_enabled() = flag AND key present). - Proprietary app/proprietary/web_crawler/captcha.py holds the bypass logic (detect/harvest/inject) plus a process-wide solver latch that halts solving on unrecoverable errors (no balance / bad key). Crawler: CrawlOutcome gains captcha_attempts/captcha_solved, surfaced via a per-call captcha_state dict threaded crawl_url -> _crawl_with_stealthy(_sync) and stamped onto every stealth terminal outcome. The stealth tier captures the proxy once and reuses it for both the fetch and the solver (IP-coherence). Billing: WebCrawlCreditService gains captcha_billing_enabled, captcha_solves_to_micros, charge_captcha, and a generic check_balance, sharing a single _apply_debit path. The indexer accumulates attempts (even on failed crawls), runs a combined crawl+captcha pre-flight, and posts a per-attempt owner charge as usage_type="web_crawl_captcha". The captcha worst-case is only reserved when solving is actually enabled, so a solving-off deployment is never blocked for captcha that can never run. Both chat scrape tools fold attempts into the current turn before the success/fail branch. Fully config-driven prices; no migration. New unit tests cover the config, factory (detection/latch/timeout/cap), credit service, indexer wiring, and the chat fold. Co-authored-by: Cursor <cursoragent@cursor.com>
54 lines
1.9 KiB
Python
54 lines
1.9 KiB
Python
"""Env-resolved captcha-solver config (one app-wide config; no per-connector).
|
|
|
|
Resolved from :data:`app.config.config` only, mirroring ``03b``'s single
|
|
``PROXY_PROVIDER`` model. Off by default: ``captcha_enabled()`` is ``False``
|
|
unless both ``CAPTCHA_SOLVING_ENABLED=TRUE`` and an API key are present, so a
|
|
misconfigured deployment (flag on, key missing) never attempts a paid solve.
|
|
"""
|
|
|
|
from dataclasses import dataclass
|
|
|
|
from app.config import config
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class CaptchaConfig:
|
|
"""Immutable snapshot of the captcha-solver settings for one crawl."""
|
|
|
|
enabled: bool
|
|
solving_site: str
|
|
api_key: str | None
|
|
max_attempts_per_url: int
|
|
timeout_s: int
|
|
captcha_type_default: str
|
|
v3_min_score: float
|
|
v3_action: str
|
|
|
|
|
|
def get_captcha_config() -> CaptchaConfig:
|
|
"""Build a :class:`CaptchaConfig` from the current process config/env."""
|
|
api_key = config.CAPTCHA_SOLVER_API_KEY
|
|
flag_on = config.CAPTCHA_SOLVING_ENABLED
|
|
return CaptchaConfig(
|
|
# Effective-enabled requires a key too: a flag with no key would only
|
|
# produce ``ErrWrongAPIKey`` per attempt (and still risk an IP ban from
|
|
# the solver), so treat key-less config as disabled.
|
|
enabled=bool(flag_on and api_key),
|
|
solving_site=config.CAPTCHA_SOLVER_PROVIDER,
|
|
api_key=api_key,
|
|
max_attempts_per_url=config.CAPTCHA_MAX_ATTEMPTS_PER_URL,
|
|
timeout_s=config.CAPTCHA_SOLVE_TIMEOUT_S,
|
|
captcha_type_default=config.CAPTCHA_TYPE_DEFAULT,
|
|
v3_min_score=config.CAPTCHA_V3_MIN_SCORE,
|
|
v3_action=config.CAPTCHA_V3_ACTION,
|
|
)
|
|
|
|
|
|
def captcha_enabled() -> bool:
|
|
"""Cheap gate: is captcha solving effectively configured (flag + key)?
|
|
|
|
Callers check this before building the ``page_action`` or capturing the
|
|
crawl's proxy endpoint for the solver, so the stealth tier stays unchanged
|
|
when solving is off.
|
|
"""
|
|
return bool(config.CAPTCHA_SOLVING_ENABLED and config.CAPTCHA_SOLVER_API_KEY)
|