SurfSense/plans/backend/03b-proxy-expansion.md

Phase 3b — Proxy provider expansion + rotation

Part of Phase 3 — WebURL Crawler & Crawl Billing. See 00-umbrella-plan.md. Depends on 03a-crawler-core.md (Scrapling-only crawler). Siblings: 03c-crawl-billing.md, 03e-stealth-hardening.md, 03d-captcha-solving.md.

Implementation note. Same convention as 03a: Phases 1–2 are SHIPPED, so the live code already says workspace_id/Workspace — substitute for the old search_space_*/SearchSpace names in citations below and grep the new name. Crucially, 03a is now IMPLEMENTED: the crawler moved to app/proprietary/web_crawler/connector.py (non-Apache-2 boundary) with Firecrawl removed and crawl_url refactored — the three Scrapling tiers now pass proxy=get_proxy_url() at 267/309/342 (not the pre-refactor 287/329/359). Locate code by symbol/grep against the new path, not absolute lines.

Objective

Let the WebURL Crawler (and every other proxy consumer) run behind either the existing anonymous_proxies gateway or a new BYO CustomProxyProvider, chosen by a single, app-wide Config.PROXY_PROVIDER. When the active provider is backed by a pool of endpoints, rotate across them client-side with a bounded retry. No branded vendors and no per-connector/per-crawl provider divergence (resolved decisions): the whole app uses one provider.

The provider abstraction already exists and is clean; the real gaps are just (1) only anonymous_proxies is registered (no BYO option), and (2) there is no client-side rotation/retry when a provider exposes multiple endpoints. The "process-global, env-only" shape is exactly what we want for a single-provider app, so we keep it.

Current state (cited)

A clean but process-global abstraction

app/utils/proxy/ is already a "subclass + register" provider package:

• base.py — ProxyProvider ABC: get_proxy_url() (canonical http://user:pass@host:port for Scrapling/curl_cffi), get_playwright_proxy(), get_requests_proxies() (built from the URL by default).
• registry.py — _PROVIDERS dict keyed by provider name; get_active_provider() resolves Config.PROXY_PROVIDER and caches a single instance process-wide (_active_provider, lines 23, 26–44). Only AnonymousProxiesProvider is registered (lines 17–19).
• __init__.py — zero-arg module helpers get_proxy_url() / get_playwright_proxy() / get_requests_proxies() + a get_residential_proxy_url() back-compat alias, all delegating to get_active_provider().
• providers/anonymous_proxies.py — the one vendor. Note it's a server-side rotating gateway: host is rotating.dnsproxifier.com:PORT and the rotation type/location are encoded into a base64 "password" dict (lines 23–51). So per-request IP rotation already happens upstream; the client sends one static endpoint.
• app/utils/proxy_config.py — a thin back-compat shim re-exporting the package (kept for old import paths).

Config knobs: Config.PROXY_PROVIDER (default "anonymous_proxies") + RESIDENTIAL_PROXY_{USERNAME,PASSWORD,HOSTNAME,LOCATION,TYPE} (config/__init__.py:983–992; documented commented-out in .env.example:300–309).

Who consumes the proxy (blast radius)

The zero-arg getters are used in more than just the crawler, so any signature change must stay backward-compatible:

Consumer	File:line	Uses
WebURL crawler (all 3 Scrapling tiers)	proprietary/web_crawler/connector.py (267, 309, 342)	get_proxy_url()
YouTube transcript route	routes/youtube_routes.py (78, 119)	get_proxy_url()
YouTube processor (indexing)	tasks/document_processors/youtube_processor.py (223, 229)	get_requests_proxies(), get_proxy_url()
Chat scrape tools (YouTube branch)	main_agent/tools/scrape_webpage.py (80, 93), research/tools/scrape_webpage.py (74, 87)	get_requests_proxies(), get_proxy_url()

Implication: the whole app shares one global provider, so every consumer keeps calling the zero-arg getters unchanged. The only behavior change is (a) which provider class those getters resolve to (selected by Config.PROXY_PROVIDER) and (b) optional client-side rotation when that provider is pool-backed. No caller passes a key; no signatures change.

Shape note (verified): get_playwright_proxy() has no consumers anywhere — only the ABC/impl/exports define it. All three crawler tiers and YouTube's AsyncFetcher.get consume the string form (get_proxy_url); browser fetchers accept a string proxy (references/Scrapling/.../stealth_chrome.py:47 — "it can be a string or a dictionary"). The dict form (get_requests_proxies) is consumed only by YouTube-transcript fetches (youtube_processor.py:223, plus the chat tools' YouTube branch main_agent/tools/scrape_webpage.py:80 and research/tools/scrape_webpage.py:74) — none of which are crawler tiers. Conclusion: rotation only needs the string shape; the playwright dict can be ignored (or left as-is on providers) rather than threaded through rotation.

Scrapling's rotation primitive

ProxyRotator(proxies: list, strategy=cyclic_rotation) is a thread-safe cycler over a static list of proxy URLs/dicts (get_proxy(), proxy_rotation.py:88–92) plus is_proxy_error(exc) (:27–30) that matches proxy failure strings (net::err_proxy, connection refused, …). It does not manage credentials or sessions — it just hands out the next endpoint. So it's only useful when we have a pool of distinct endpoints; it adds nothing for a single server-side-rotating gateway like anonymous_proxies.

Both are publicly importable: from scrapling.engines.toolbelt import ProxyRotator, is_proxy_error (re-exported in engines/toolbelt/__init__.py:1–3). The thread-safe Lock (:71,90) makes ProxyRotator safe to call from the browser tiers that 03a runs via asyncio.to_thread. Version is guaranteed: the pinned floor is scrapling[fetchers]>=0.4.9 (pyproject.toml:91) and these APIs exist as of 0.4.9 (scrapling/__init__.py:2).

Selection scope (resolved)

The user expects a single proxy provider across the entire app, so 03b's selection scope is simply the global, env-configured provider — no per-connector or per-crawl override is built now. That is both the fastest path and, with one thin seam, the most scalable later:

• All resolution stays behind today's get_active_provider() (env-selected, process-cached). One provider, app-wide.
• "Scalable in future": if per-pipeline proxying is ever wanted (Phase 5+), it layers on as an optional argument to a resolver (resolve_proxy(override=pipeline.config)) with zero changes to existing call sites — but it is explicitly not implemented in 03b (YAGNI for a single-provider app).

Target design

1. Registry: register the BYO provider; keep single-provider selection

The existing get_active_provider() (env-selected via Config.PROXY_PROVIDER, cached process-wide in _active_provider — registry.py:23,26–44) is exactly the single-provider model we want — keep it. The only change is to register CustomProxyProvider so PROXY_PROVIDER can select it:

• _PROVIDERS["anonymous_proxies"] = AnonymousProxiesProvider (existing) + _PROVIDERS["custom"] = CustomProxyProvider (new).
• get_active_provider() resolves the one active provider from env and caches it — no keyed multi-provider coexistence (get_provider(key) is intentionally NOT added; YAGNI for a single-provider app).
• Unknown PROXY_PROVIDER → existing warn-and-fallback path (registry.py:35–41) is unchanged.

2. Provider config source: env only

Credentials come from env, full stop (single global provider):

• anonymous_proxies → existing RESIDENTIAL_PROXY_*. Unchanged.
• custom → new env knobs (one URL or a pool): CUSTOM_PROXY_URLS (comma-separated) and/or CUSTOM_PROXY_URL.

No per-connector proxy keys, and no connector-config validator changes (that was the per-connector path, now dropped).

3. New provider: CustomProxyProvider (BYO) — the only addition

• Accepts a raw proxy URL or a list of URLs from env. Covers "bring your own proxy / our own pool" with no vendor-specific auth assumptions.
• Implements just get_proxy_url(); the base derives get_requests_proxies() from it (base.py:37–46). When configured with a pool, get_proxy_url() returns the next endpoint from an internal ProxyRotator (see §4) — so rotation is transparent to every caller of the zero-arg getter.
• No branded vendors (resolved): Webshare/BrightData/Smartproxy/etc. are not shipped. A user who wants any specific vendor points CustomProxyProvider at that vendor's endpoint(s) via CUSTOM_PROXY_URLS.

4. Rotation + retry (only when the active provider is pool-backed)

• If CustomProxyProvider is configured with multiple URLs, it wraps them in Scrapling's ProxyRotator (cyclic default, thread-safe Lock — proxy_rotation.py:71,88–92) and returns the next endpoint per get_proxy_url() call. Because 03a's browser tiers run under asyncio.to_thread, the rotator's lock keeps this safe.
• The crawler adds a bounded client-side retry: on a tier failure where is_proxy_error(exc) is true (proxy_rotation.py:27–30), it re-reads get_proxy_url() (next endpoint) and retries that tier once before falling through. One rotation-retry per tier — no unbounded fan-out on billable crawls.
• Single-endpoint providers (anonymous_proxies, already server-side rotating; or custom with one URL) return the same endpoint every call, so the retry is a harmless no-op for them.

5. Crawler stays on the zero-arg getter (no injection needed)

Because there is one global provider and rotation lives inside it, the crawler does not need a proxy injected per crawl:

• All three Scrapling tiers keep calling get_proxy_url() (which yields the rotating value when pool-backed). The only crawler edit is the bounded is_proxy_error retry from §4.
• No change to YouTube/chat-tool consumers — same zero-arg getters, same global provider.
• Future seam (NOT built): if per-pipeline proxying is ever needed, resolve a provider from pipeline.config and pass its get_proxy_url into the crawl — an additive optional arg, no change to today's call sites.

Config / env changes

• config/__init__.py:983–992 + .env.example:300–309: add CUSTOM_PROXY_URLS (comma-separated pool) and/or CUSTOM_PROXY_URL for the custom provider; keep all existing RESIDENTIAL_PROXY_* working unchanged. PROXY_PROVIDER now accepts "custom" in addition to "anonymous_proxies".
• No connector-config validator changes (the per-connector proxy path is dropped — single global provider).

Work items

1. CustomProxyProvider: BYO single-URL or pool-of-URLs provider (reads CUSTOM_PROXY_URL(S)); internal ProxyRotator when pool-backed; register as "custom" in _PROVIDERS.
2. Rotation retry: add a bounded (one-per-tier) is_proxy_error retry to the crawler's Scrapling tiers; single-endpoint providers no-op.
3. Config + docs: CUSTOM_PROXY_URL(S) in Config + .env.example; document PROXY_PROVIDER="custom".
4. Tests: CustomProxyProvider single-URL vs pool; ProxyRotator cyclic order; get_active_provider() resolves "custom" and still warn-falls-back on unknown; crawler rotates once on is_proxy_error then falls through; anonymous_proxies + YouTube path unchanged (same endpoint each call).

Risks / trade-offs

• Backward compatibility of the getters. The zero-arg get_proxy_url() contract is consumed in 4+ places; the design keeps it completely intact — no signature changes, no per-caller keys. Only the cached active provider's implementation changes when PROXY_PROVIDER="custom".
• Over-rotation cost. Rotation-retry is bounded to one extra attempt per tier so a billable crawl can't silently multiply upstream proxy usage.
• Single provider is a deliberate constraint. One global provider app-wide (resolved). Per-pipeline/per-connector selection is intentionally deferred behind a no-op seam (§5); not built now.
• Pool rotation under to_thread. ProxyRotator's Lock makes the rotating get_proxy_url() safe to call from the browser tiers 03a offloads via asyncio.to_thread.

Resolved decisions

• Branded vendors → NONE. Ship CustomProxyProvider (BYO) only; no Webshare/BrightData/Smartproxy/etc. subclasses. A user who wants a specific vendor points CUSTOM_PROXY_URLS at it.
• Selection scope → single global provider, app-wide. No per-connector/per-crawl override is built; resolution is env-only via Config.PROXY_PROVIDER. A future per-pipeline override is left as a no-op seam (§5) so it stays "scalable + fast to add later."
• Client-side rotation → built, but only active for a pool-backed CustomProxyProvider. anonymous_proxies (server-side rotating) and single-URL custom configs skip it automatically. Rotation lives inside the provider so it's transparent to all callers.

Out of scope (hand-offs)

• Per-pipeline / per-connector proxy selection → deferred (Phases 5–7 if ever needed); §5 leaves a no-op seam, nothing is wired now.
• Branded-vendor provider subclasses → not planned (use CustomProxyProvider).
• Static / sticky-session proxies (future). A later capability will add static proxy support — sticky IPs held for the duration of a session — most likely paired with authenticated/account-based scraping to bypass logged-in platforms (the deferred platform connectors: LinkedIn, Instagram, etc.). This is a different axis from the rotating pool here: rotation maximizes IP diversity, whereas account bypass needs IP stability so a session/cookie stays bound to one IP. It is additive to this design — a new ProxyProvider (or a "sticky" mode/flag on CustomProxyProvider) registered under a new PROXY_PROVIDER key, with no change to the zero-arg getter contract — and stays consistent with the single-provider model (the active provider would be the static one when that workflow is selected). Build it alongside the platform connectors, not in Phase 3.
• Crawl credit metering (proxy cost is absorbed into the flat $1 / 1000 successful price, not metered separately) → 03c.
• Captcha solving → 03d.
• Geoip fingerprint coherence (matching browser locale/timezone_id to the proxy's exit geo) → 03e. 03b only owns selecting the proxy; 03e consumes the chosen endpoint's geo. The provider's RESIDENTIAL_PROXY_LOCATION (config/__init__.py) is one input; resolving the actual exit IP's geo is 03e's job.
• Surfacing the crawl's chosen endpoint into the strategy. Both 03d (IP-bound captcha solves) and 03e (geoip + sticky reuse) need the exact endpoint a crawl used, not a fresh get_proxy_url() call (which rotates on a pool-backed CustomProxyProvider). The capture-once seam lives in 03a's FetchStrategy context; 03b only guarantees the rotating getter and notes the consumers.