apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-20 23:21:06 +02:00
Code Issues Projects Releases Packages Wiki Activity

refactor: enhance Google search scraping capabilities with warm sticky-IP pool and improved captcha handling

Browse source
This commit is contained in:
DESKTOP-RTLN3BA\$punk 2026-07-17 03:27:43 -07:00
parent 4b97e9a54e
commit 68d1832f10
20 changed files with 1900 additions and 175 deletions
Download patch file Download diff file Expand all files Collapse all files

35
surfsense_backend/.env.example
View file

@ -374,15 +374,38 @@ TURNSTILE_SECRET_KEY=
# PROXY_URL=http://user:pass@host:port
# PROXY_URLS=http://user:pass@host1:port,http://user:pass@host2:port
# --- Google Search scraper: scale / warm sticky-IP pool -----------------------
# Per-process render concurrency AND the throughput lever: ceiling =
# MAX_CONCURRENT_PAGES / warm-render-secs (~4/14s ≈ 17 SERP/min). Renders are
# resource-light (disable_resources), so one Chromium holds many text contexts —
# raise this to trade RAM/CPU for throughput before adding processes
# (16 → ~69/min/process ⇒ ~7 processes cover 500/min). Validate with
# scripts/scale_google_search.py.
# GOOGLE_SEARCH_MAX_CONCURRENT_PAGES=4
# Warm sticky-IP pool: concurrent renders spread across this many solved IPs so
# no single IP is hammered (Google re-walls a hot IP). Steady-state paid solves
# ≈ pool size, not request count. Size for the fleet: fleet per-IP load =
# IP_MAX_CONCURRENCY × processes must stay gentle. Solved IPs are shared across
# workers via Redis (REDIS_APP_URL) so the fleet solves each IP once.
# GOOGLE_SEARCH_WARM_POOL_TARGET=8
# GOOGLE_SEARCH_IP_MAX_CONCURRENCY=2
# How long a shared exemption stays reusable before a re-solve (Google's cookie
# outlives this; conservative TTL just re-solves idle IPs occasionally).
# GOOGLE_SEARCH_EXEMPTION_TTL_S=3600
# Per-fetch budget before giving up under saturation (a cold solve is ~45s).
# GOOGLE_SEARCH_FETCH_DEADLINE_S=180
# =====================================================================
# Captcha solving (Phase 3d) — LAST-resort bypass tier via captchatools.
# Only fires on the stealth browser tier when a sitekey is detected AND
# CAPTCHA_SOLVING_ENABLED=TRUE. Cloudflare Turnstile is already solved free
# in-framework (no config needed). Off by default => zero attempts, zero cost.
# Captcha solving (Phase 3d) — LAST-resort bypass tier (in-house solver seam,
# app/utils/captcha/solvers.py). Only fires on the stealth browser tier when a
# sitekey is detected AND CAPTCHA_SOLVING_ENABLED=TRUE. Also unlocks Google's
# reCAPTCHA-Enterprise /sorry wall for the SERP scraper. Cloudflare Turnstile is
# already solved free in-framework. Off by default => zero attempts, zero cost.
# NOTE: automated solving may violate a target site's ToS — opt-in, public
# data only (no logged-in bypass). captchatools is itself the vendor registry.
# data only (no logged-in bypass).
# CAPTCHA_SOLVING_ENABLED=FALSE
# solving_site: capmonster | 2captcha | anticaptcha | capsolver | captchaai
# Provider: "capsolver" (AI-native, fastest on reCAPTCHA-Enterprise) or
# "2captcha" have in-house clients today (more added over time).
# CAPTCHA_SOLVER_PROVIDER=capsolver
# CAPTCHA_SOLVER_API_KEY=
# Per-URL solve cap (bounds solver spend on a hostile page).