mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-06 22:12:12 +02:00
bdacea8b6e
Wire captchatools as the StealthyFetcher-tier page_action to detect, harvest (egressing from the crawl's own proxy IP), inject, and submit reCAPTCHA v2/v3 and hCaptcha tokens. Opt-in and off by default (zero attempts, zero cost). Licensing split: - Apache-2 app/utils/captcha/ holds the generic, vendor-agnostic config (CaptchaConfig + captcha_enabled() = flag AND key present). - Proprietary app/proprietary/web_crawler/captcha.py holds the bypass logic (detect/harvest/inject) plus a process-wide solver latch that halts solving on unrecoverable errors (no balance / bad key). Crawler: CrawlOutcome gains captcha_attempts/captcha_solved, surfaced via a per-call captcha_state dict threaded crawl_url -> _crawl_with_stealthy(_sync) and stamped onto every stealth terminal outcome. The stealth tier captures the proxy once and reuses it for both the fetch and the solver (IP-coherence). Billing: WebCrawlCreditService gains captcha_billing_enabled, captcha_solves_to_micros, charge_captcha, and a generic check_balance, sharing a single _apply_debit path. The indexer accumulates attempts (even on failed crawls), runs a combined crawl+captcha pre-flight, and posts a per-attempt owner charge as usage_type="web_crawl_captcha". The captcha worst-case is only reserved when solving is actually enabled, so a solving-off deployment is never blocked for captcha that can never run. Both chat scrape tools fold attempts into the current turn before the success/fail branch. Fully config-driven prices; no migration. New unit tests cover the config, factory (detection/latch/timeout/cap), credit service, indexer wiring, and the chat fold. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|---|---|---|
| .. | ||
| streaming | ||
| __init__.py | ||
| test_agent_billing_resolver.py | ||
| test_ai_file_sort_service.py | ||
| test_ai_sort_task_dedupe.py | ||
| test_auto_model_pin_service.py | ||
| test_auto_pin_image_aware.py | ||
| test_billable_call.py | ||
| test_docling_image_support.py | ||
| test_folder_hierarchy.py | ||
| test_image_gen_api_base_defense.py | ||
| test_llm_router_pool_filter.py | ||
| test_memory_service.py | ||
| test_model_connections.py | ||
| test_openrouter_integration_service.py | ||
| test_openrouter_legacy_config.py | ||
| test_or_health_enrichment.py | ||
| test_pricing_registration.py | ||
| test_provider_capabilities.py | ||
| test_quality_score.py | ||
| test_quota_checked_vision_llm.py | ||
| test_revert_filesystem_tools.py | ||
| test_revert_service.py | ||
| test_supports_image_input.py | ||
| test_token_quota_service_cost.py | ||
| test_web_crawl_credit_service.py | ||