apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-20 23:21:06 +02:00
Code Issues Projects Releases Packages Wiki Activity

refactor: implement cross-country proxy rotation for Reddit and TikTok scrapers, enhancing resilience against IP blocks and improving search query handling

Browse source
This commit is contained in:
DESKTOP-RTLN3BA\$punk 2026-07-17 16:30:26 -07:00
parent c0ebb62fb2
commit 8a5d37a4db
10 changed files with 213 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

8
surfsense_backend/.env.example
View file

@ -373,6 +373,14 @@ TURNSTILE_SECRET_KEY=
# (cyclic); server-side-rotating gateways ignore it.
# PROXY_URL=http://user:pass@host:port
# PROXY_URLS=http://user:pass@host1:port,http://user:pass@host2:port
#
# NOTE (dataimpulse): its default *worldwide* pool (a PROXY_URL with no
# "__cr.<country>" suffix) hands out exit IPs some targets hard-block — Reddit
# 403s every one, TikTok withholds its ttwid cookie — so those scrapers
# pin/rotate exit countries on their own (no config needed). Other scrapers
# (e.g. Google SERP) deliberately prefer the worldwide pool, so leave PROXY_URL
# unpinned unless you have a reason; pin a country only for your whole install by
# adding a "__cr.<country>" suffix above.
# --- Google Search scraper: scale / warm sticky-IP pool -----------------------
# Per-process render concurrency AND the throughput lever: ceiling =