apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-06 22:12:12 +02:00
Code Issues Projects Releases Packages Wiki Activity

feat(03b): add BYO custom proxy provider + bounded crawler rotation-retry

Browse source 
Add CustomProxyProvider (single endpoint or rotating pool via Scrapling ProxyRotator), registered as 'custom' alongside anonymous_proxies and selectable via PROXY_PROVIDER. Adds is_pool_backed to the ProxyProvider ABC + a zero-arg package helper. The web crawler does a bounded one-per-tier is_proxy_error rotation-retry gated on is_pool_backed() (single-endpoint providers no-op). Config/.env.example gain CUSTOM_PROXY_URL(S). Zero-arg getter contract unchanged for all consumers. Documents the proprietary boundary test (generic proxy infra stays Apache-2). Tests: provider, registry, crawler rotation (16).

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
DESKTOP-RTLN3BA\$punk 2026-06-29 21:03:34 -07:00
parent 5c36cd3071
commit 62260125f7
12 changed files with 358 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

6
surfsense_backend/.env.example
View file

@ -325,6 +325,12 @@ TURNSTILE_SECRET_KEY=
# RESIDENTIAL_PROXY_LOCATION=
# RESIDENTIAL_PROXY_TYPE=1
# Custom (BYO) proxy provider. Activate with PROXY_PROVIDER=custom.
# Provide a single endpoint and/or a comma-separated pool; a pool of >1 rotates
# client-side (cyclic). Each value is a full http://user:pass@host:port URL.
# CUSTOM_PROXY_URL=http://user:pass@host:port
# CUSTOM_PROXY_URLS=http://user:pass@host1:port,http://user:pass@host2:port
# File Parser Service
ETL_SERVICE=UNSTRUCTURED or LLAMACLOUD or DOCLING
UNSTRUCTURED_API_KEY=Tpu3P0U8iy