apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-08 22:22:17 +02:00
Code Issues Projects Releases Packages Wiki Activity
SurfSense/surfsense_backend/app/proprietary/README.md
DESKTOP-RTLN3BA\$punk 80927a2872 feat(billing): meter platform scrapers per item; consolidate web scraping onto web.crawl 
Add per-item, per-platform billing for the platform-native connectors (Reddit, Google Search, Google Maps places/reviews, YouTube videos/comments) through the capability gate/charge seam. Rates are config-driven with a shared wallet-credit module (wallet_credit) and a dedicated PlatformScrapeCreditService; agent and REST capability runs now record cost_micros. Google Maps scrape dual-meters places and attached reviews.

Remove the main-agent scrape_webpage tool now that the web.crawl capability covers single-page (maxCrawlDepth=0) and site crawling. The main agent now reaches crawling via task(web_crawler, ...). Update prompts, tool catalog, receipts, skills, proprietary docs, and tests; drop the obsolete chat-turn crawl fold path.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-05 17:08:01 -07:00

2.1 KiB
Raw Blame History

app.proprietary — non-Apache-2 license boundary

Everything in this directory tree is licensed separately from the rest of SurfSense (which is Apache-2.0), under the Business Source License 1.1 — see LICENSE. In short: source-available; production use is permitted (including self-hosting the whole app) except offering it to third parties as a commercial product or hosted/managed service; each released version converts to Apache-2.0 four years after its release.

Why this exists

This package holds the product moat:

  • the in-house undetectable web crawler (Scrapling tiers + stealth/captcha hardening), and
  • (future) platform-specific actors that scrape/extract structured data from individual platforms.

Keeping it in one clearly-named directory makes the license boundary unambiguous: a single rule — everything under app/proprietary/** is not Apache-2.0 — instead of per-file headers scattered across the tree.

Layout

  • web_crawler/ — the Scrapling-based crawler engine. Public API: WebCrawlerConnector, CrawlOutcome, CrawlOutcomeStatus (from app.proprietary.web_crawler import ...).
  • platforms/ — (future, Phase 8) platform-specific actors; scaffolded/empty.

Rules

  • Do not add Apache-2.0-intended code here.
  • Apache-2.0 code elsewhere may import from this package (the indexer and the web.crawl capability do); that does not move them under this license.
  • Depend only on the public API exported from each subpackage's __init__, not on internal modules, so the boundary stays clean and swappable.
  • Boundary test: put code here only if it is used exclusively by the moat. Generic infrastructure that Apache-2 features also depend on stays Apache-2 even when the crawler uses it too. Example: app/utils/proxy/ (provider abstraction, registry, CustomProxyProvider + rotation — a thin wrapper over Scrapling's public ProxyRotator) is shared with the YouTube/transcript and chat features, so it stays Apache-2; only the crawl-ladder-coupled rotation-retry orchestration (web_crawler/connector.py::_run_tier_with_proxy_retry) lives here.