SurfSense/surfsense_backend/app/utils/proxy
DESKTOP-RTLN3BA\$punk 62260125f7 feat(03b): add BYO custom proxy provider + bounded crawler rotation-retry
Add CustomProxyProvider (single endpoint or rotating pool via Scrapling ProxyRotator), registered as 'custom' alongside anonymous_proxies and selectable via PROXY_PROVIDER. Adds is_pool_backed to the ProxyProvider ABC + a zero-arg package helper. The web crawler does a bounded one-per-tier is_proxy_error rotation-retry gated on is_pool_backed() (single-endpoint providers no-op). Config/.env.example gain CUSTOM_PROXY_URL(S). Zero-arg getter contract unchanged for all consumers. Documents the proprietary boundary test (generic proxy infra stays Apache-2). Tests: provider, registry, crawler rotation (16).

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-29 21:03:34 -07:00
..
providers feat(03b): add BYO custom proxy provider + bounded crawler rotation-retry 2026-06-29 21:03:34 -07:00
__init__.py feat(03b): add BYO custom proxy provider + bounded crawler rotation-retry 2026-06-29 21:03:34 -07:00
base.py feat(03b): add BYO custom proxy provider + bounded crawler rotation-retry 2026-06-29 21:03:34 -07:00
registry.py feat(03b): add BYO custom proxy provider + bounded crawler rotation-retry 2026-06-29 21:03:34 -07:00