apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-26 23:51:14 +02:00
Code Issues Projects Releases Packages Wiki Activity

Merge remote-tracking branch 'upstream/dev' into feature-indeed-jobs-scraper

Browse source 
# Conflicts:
#	README.es.md
#	README.hi.md
#	README.md
#	README.pt-BR.md
#	README.zh-CN.md
#	surfsense_backend/tests/unit/capabilities/google_maps/test_registry.py
#	surfsense_backend/tests/unit/capabilities/reddit/test_registry.py
#	surfsense_backend/tests/unit/capabilities/youtube/test_registry.py
#	surfsense_mcp/mcp_server/features/scrapers/__init__.py
#	surfsense_web/content/docs/connectors/index.mdx
#	surfsense_web/content/docs/connectors/native/index.mdx
#	surfsense_web/content/docs/connectors/native/meta.json
#	surfsense_web/content/docs/how-to/mcp-server.mdx
#	surfsense_web/lib/connectors-marketing/index.ts
#	surfsense_web/lib/playground/catalog.ts
This commit is contained in:
CREDO23 2026-07-18 22:11:30 +02:00
commit 91aa265afb
259 changed files with 13867 additions and 1883 deletions
Download patch file Download diff file Expand all files Collapse all files

48
surfsense_backend/.env.example
View file

@ -277,8 +277,8 @@ MICROS_PER_PAGE=1000
# WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE=3000
# Debit the credit wallet per *item returned* by the platform-native scrapers
# (Reddit, Google Search, Google Maps, YouTube). Default FALSE keeps scraping
# effectively free for self-hosted/OSS installs; hosted deployments set TRUE.
# (Reddit, Google Search, Google Maps, Amazon, YouTube). Default FALSE keeps
# scraping effectively free for self-hosted/OSS installs; hosted set TRUE.
# Each rate is micro-USD per item, fully config-driven (no hardcoded rate):
# <KEY> = round(USD_per_1000_items * 1_000)
# 3500 == $3.50/1000 | 5000 == $5/1000 | 2000 == $2/1000
@ -290,6 +290,7 @@ MICROS_PER_PAGE=1000
# GOOGLE_SEARCH_MICROS_PER_SERP=5500
# GOOGLE_MAPS_MICROS_PER_PLACE=3500
# GOOGLE_MAPS_MICROS_PER_REVIEW=1500
# AMAZON_MICROS_PER_PRODUCT=3500
# YOUTUBE_MICROS_PER_VIDEO=2500
# YOUTUBE_MICROS_PER_COMMENT=1500
# INSTAGRAM_SCRAPE_MICROS_PER_ITEM=3500
@ -373,16 +374,47 @@ TURNSTILE_SECRET_KEY=
# (cyclic); server-side-rotating gateways ignore it.
# PROXY_URL=http://user:pass@host:port
# PROXY_URLS=http://user:pass@host1:port,http://user:pass@host2:port
#
# NOTE (dataimpulse): its default *worldwide* pool (a PROXY_URL with no
# "__cr.<country>" suffix) hands out exit IPs some targets hard-block — Reddit
# 403s every one, TikTok withholds its ttwid cookie — so those scrapers
# pin/rotate exit countries on their own (no config needed). Other scrapers
# (e.g. Google SERP) deliberately prefer the worldwide pool, so leave PROXY_URL
# unpinned unless you have a reason; pin a country only for your whole install by
# adding a "__cr.<country>" suffix above.
# --- Google Search scraper: scale / warm sticky-IP pool -----------------------
# Per-process render concurrency AND the throughput lever: ceiling =
# MAX_CONCURRENT_PAGES / warm-render-secs (~4/14s ≈ 17 SERP/min). Renders are
# resource-light (disable_resources), so one Chromium holds many text contexts —
# raise this to trade RAM/CPU for throughput before adding processes
# (16 → ~69/min/process ⇒ ~7 processes cover 500/min). Validate with
# scripts/scale_google_search.py.
# GOOGLE_SEARCH_MAX_CONCURRENT_PAGES=4
# Warm sticky-IP pool: concurrent renders spread across this many solved IPs so
# no single IP is hammered (Google re-walls a hot IP). Steady-state paid solves
# ≈ pool size, not request count. Size for the fleet: fleet per-IP load =
# IP_MAX_CONCURRENCY × processes must stay gentle. Solved IPs are shared across
# workers via Redis (REDIS_APP_URL) so the fleet solves each IP once.
# GOOGLE_SEARCH_WARM_POOL_TARGET=8
# GOOGLE_SEARCH_IP_MAX_CONCURRENCY=2
# How long a shared exemption stays reusable before a re-solve (Google's cookie
# outlives this; conservative TTL just re-solves idle IPs occasionally).
# GOOGLE_SEARCH_EXEMPTION_TTL_S=3600
# Per-fetch budget before giving up under saturation (a cold solve is ~45s).
# GOOGLE_SEARCH_FETCH_DEADLINE_S=180
# =====================================================================
# Captcha solving (Phase 3d) — LAST-resort bypass tier via captchatools.
# Only fires on the stealth browser tier when a sitekey is detected AND
# CAPTCHA_SOLVING_ENABLED=TRUE. Cloudflare Turnstile is already solved free
# in-framework (no config needed). Off by default => zero attempts, zero cost.
# Captcha solving (Phase 3d) — LAST-resort bypass tier (in-house solver seam,
# app/utils/captcha/solvers.py). Only fires on the stealth browser tier when a
# sitekey is detected AND CAPTCHA_SOLVING_ENABLED=TRUE. Also unlocks Google's
# reCAPTCHA-Enterprise /sorry wall for the SERP scraper. Cloudflare Turnstile is
# already solved free in-framework. Off by default => zero attempts, zero cost.
# NOTE: automated solving may violate a target site's ToS — opt-in, public
# data only (no logged-in bypass). captchatools is itself the vendor registry.
# data only (no logged-in bypass).
# CAPTCHA_SOLVING_ENABLED=FALSE
# solving_site: capmonster | 2captcha | anticaptcha | capsolver | captchaai
# Provider: "capsolver" (AI-native, fastest on reCAPTCHA-Enterprise) or
# "2captcha" have in-house clients today (more added over time).
# CAPTCHA_SOLVER_PROVIDER=capsolver
# CAPTCHA_SOLVER_API_KEY=
# Per-URL solve cap (bounds solver spend on a hostile page).