mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-06 22:12:12 +02:00
62260125f7
Add CustomProxyProvider (single endpoint or rotating pool via Scrapling ProxyRotator), registered as 'custom' alongside anonymous_proxies and selectable via PROXY_PROVIDER. Adds is_pool_backed to the ProxyProvider ABC + a zero-arg package helper. The web crawler does a bounded one-per-tier is_proxy_error rotation-retry gated on is_pool_backed() (single-endpoint providers no-op). Config/.env.example gain CUSTOM_PROXY_URL(S). Zero-arg getter contract unchanged for all consumers. Documents the proprietary boundary test (generic proxy infra stays Apache-2). Tests: provider, registry, crawler rotation (16). Co-authored-by: Cursor <cursoragent@cursor.com>
40 lines
1.8 KiB
Markdown
40 lines
1.8 KiB
Markdown
# `app.proprietary` — non-Apache-2 license boundary
|
|
|
|
Everything in this directory tree is licensed **separately** from the rest of
|
|
SurfSense (which is Apache-2.0). See [`LICENSE`](./LICENSE).
|
|
|
|
## Why this exists
|
|
|
|
This package holds the product moat:
|
|
|
|
- the in-house **undetectable web crawler** (Scrapling tiers + stealth/captcha
|
|
hardening), and
|
|
- (future) **platform-specific actors** that scrape/extract structured data from
|
|
individual platforms.
|
|
|
|
Keeping it in one clearly-named directory makes the license boundary
|
|
unambiguous: a single rule — *everything under `app/proprietary/**` is not
|
|
Apache-2.0* — instead of per-file headers scattered across the tree.
|
|
|
|
## Layout
|
|
|
|
- `web_crawler/` — the Scrapling-based crawler engine. Public API:
|
|
`WebCrawlerConnector`, `CrawlOutcome`, `CrawlOutcomeStatus`
|
|
(`from app.proprietary.web_crawler import ...`).
|
|
- `platforms/` — (future, Phase 8) platform-specific actors; scaffolded/empty.
|
|
|
|
## Rules
|
|
|
|
- **Do not** add Apache-2.0-intended code here.
|
|
- Apache-2.0 code elsewhere **may import from** this package (the indexer and the
|
|
chat `scrape_webpage` tools do); that does not move them under this license.
|
|
- Depend only on the public API exported from each subpackage's `__init__`, not
|
|
on internal modules, so the boundary stays clean and swappable.
|
|
- **Boundary test:** put code here only if it is used *exclusively* by the moat.
|
|
Generic infrastructure that Apache-2 features also depend on stays Apache-2
|
|
even when the crawler uses it too. Example: `app/utils/proxy/` (provider
|
|
abstraction, registry, `CustomProxyProvider` + rotation — a thin wrapper over
|
|
Scrapling's public `ProxyRotator`) is shared with the YouTube/transcript and
|
|
chat features, so it stays Apache-2; only the crawl-ladder-coupled
|
|
rotation-retry orchestration (`web_crawler/connector.py::_run_tier_with_proxy_retry`)
|
|
lives here.
|