mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-08 22:22:17 +02:00
80927a2872
Add per-item, per-platform billing for the platform-native connectors (Reddit, Google Search, Google Maps places/reviews, YouTube videos/comments) through the capability gate/charge seam. Rates are config-driven with a shared wallet-credit module (wallet_credit) and a dedicated PlatformScrapeCreditService; agent and REST capability runs now record cost_micros. Google Maps scrape dual-meters places and attached reviews. Remove the main-agent scrape_webpage tool now that the web.crawl capability covers single-page (maxCrawlDepth=0) and site crawling. The main agent now reaches crawling via task(web_crawler, ...). Update prompts, tool catalog, receipts, skills, proprietary docs, and tests; drop the obsolete chat-turn crawl fold path. Co-authored-by: Cursor <cursoragent@cursor.com>
44 lines
2.1 KiB
Markdown
44 lines
2.1 KiB
Markdown
# `app.proprietary` — non-Apache-2 license boundary
|
|
|
|
Everything in this directory tree is licensed **separately** from the rest of
|
|
SurfSense (which is Apache-2.0), under the **Business Source License 1.1** —
|
|
see [`LICENSE`](./LICENSE). In short: source-available; production use is
|
|
permitted (including self-hosting the whole app) *except* offering it to third
|
|
parties as a commercial product or hosted/managed service; each released
|
|
version converts to Apache-2.0 four years after its release.
|
|
|
|
## Why this exists
|
|
|
|
This package holds the product moat:
|
|
|
|
- the in-house **undetectable web crawler** (Scrapling tiers + stealth/captcha
|
|
hardening), and
|
|
- (future) **platform-specific actors** that scrape/extract structured data from
|
|
individual platforms.
|
|
|
|
Keeping it in one clearly-named directory makes the license boundary
|
|
unambiguous: a single rule — *everything under `app/proprietary/**` is not
|
|
Apache-2.0* — instead of per-file headers scattered across the tree.
|
|
|
|
## Layout
|
|
|
|
- `web_crawler/` — the Scrapling-based crawler engine. Public API:
|
|
`WebCrawlerConnector`, `CrawlOutcome`, `CrawlOutcomeStatus`
|
|
(`from app.proprietary.web_crawler import ...`).
|
|
- `platforms/` — (future, Phase 8) platform-specific actors; scaffolded/empty.
|
|
|
|
## Rules
|
|
|
|
- **Do not** add Apache-2.0-intended code here.
|
|
- Apache-2.0 code elsewhere **may import from** this package (the indexer and the
|
|
`web.crawl` capability do); that does not move them under this license.
|
|
- Depend only on the public API exported from each subpackage's `__init__`, not
|
|
on internal modules, so the boundary stays clean and swappable.
|
|
- **Boundary test:** put code here only if it is used *exclusively* by the moat.
|
|
Generic infrastructure that Apache-2 features also depend on stays Apache-2
|
|
even when the crawler uses it too. Example: `app/utils/proxy/` (provider
|
|
abstraction, registry, `CustomProxyProvider` + rotation — a thin wrapper over
|
|
Scrapling's public `ProxyRotator`) is shared with the YouTube/transcript and
|
|
chat features, so it stays Apache-2; only the crawl-ladder-coupled
|
|
rotation-retry orchestration (`web_crawler/connector.py::_run_tier_with_proxy_retry`)
|
|
lives here.
|