apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-06 22:12:12 +02:00
Code Issues Projects Releases Packages Wiki Activity
SurfSense/surfsense_backend/app/proprietary
History
DESKTOP-RTLN3BA\$punk bdacea8b6e feat(crawler): captcha solving + per-attempt billing (Phase 3d) 
Wire captchatools as the StealthyFetcher-tier page_action to detect, harvest
(egressing from the crawl's own proxy IP), inject, and submit reCAPTCHA v2/v3
and hCaptcha tokens. Opt-in and off by default (zero attempts, zero cost).

Licensing split:
- Apache-2 app/utils/captcha/ holds the generic, vendor-agnostic config
  (CaptchaConfig + captcha_enabled() = flag AND key present).
- Proprietary app/proprietary/web_crawler/captcha.py holds the bypass logic
  (detect/harvest/inject) plus a process-wide solver latch that halts solving
  on unrecoverable errors (no balance / bad key).

Crawler: CrawlOutcome gains captcha_attempts/captcha_solved, surfaced via a
per-call captcha_state dict threaded crawl_url -> _crawl_with_stealthy(_sync)
and stamped onto every stealth terminal outcome. The stealth tier captures the
proxy once and reuses it for both the fetch and the solver (IP-coherence).

Billing: WebCrawlCreditService gains captcha_billing_enabled,
captcha_solves_to_micros, charge_captcha, and a generic check_balance, sharing
a single _apply_debit path. The indexer accumulates attempts (even on failed
crawls), runs a combined crawl+captcha pre-flight, and posts a per-attempt
owner charge as usage_type="web_crawl_captcha". The captcha worst-case is only
reserved when solving is actually enabled, so a solving-off deployment is never
blocked for captcha that can never run. Both chat scrape tools fold attempts
into the current turn before the success/fail branch.

Fully config-driven prices; no migration. New unit tests cover the config,
factory (detection/latch/timeout/cap), credit service, indexer wiring, and the
chat fold.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-29 22:58:21 -07:00
..
platforms feat(03a): implement Scrapling-only web crawler and remove Firecrawl integration 2026-06-29 20:37:56 -07:00
web_crawler feat(crawler): captcha solving + per-attempt billing (Phase 3d) 2026-06-29 22:58:21 -07:00
__init__.py feat(03a): implement Scrapling-only web crawler and remove Firecrawl integration 2026-06-29 20:37:56 -07:00
LICENSE feat(03a): implement Scrapling-only web crawler and remove Firecrawl integration 2026-06-29 20:37:56 -07:00
README.md feat(03b): add BYO custom proxy provider + bounded crawler rotation-retry 2026-06-29 21:03:34 -07:00

README.md

app.proprietary — non-Apache-2 license boundary

Everything in this directory tree is licensed separately from the rest of SurfSense (which is Apache-2.0). See LICENSE.

Why this exists

This package holds the product moat:

  • the in-house undetectable web crawler (Scrapling tiers + stealth/captcha hardening), and
  • (future) platform-specific actors that scrape/extract structured data from individual platforms.

Keeping it in one clearly-named directory makes the license boundary unambiguous: a single rule — everything under app/proprietary/** is not Apache-2.0 — instead of per-file headers scattered across the tree.

Layout

  • web_crawler/ — the Scrapling-based crawler engine. Public API: WebCrawlerConnector, CrawlOutcome, CrawlOutcomeStatus (from app.proprietary.web_crawler import ...).
  • platforms/ — (future, Phase 8) platform-specific actors; scaffolded/empty.

Rules

  • Do not add Apache-2.0-intended code here.
  • Apache-2.0 code elsewhere may import from this package (the indexer and the chat scrape_webpage tools do); that does not move them under this license.
  • Depend only on the public API exported from each subpackage's __init__, not on internal modules, so the boundary stays clean and swappable.
  • Boundary test: put code here only if it is used exclusively by the moat. Generic infrastructure that Apache-2 features also depend on stays Apache-2 even when the crawler uses it too. Example: app/utils/proxy/ (provider abstraction, registry, CustomProxyProvider + rotation — a thin wrapper over Scrapling's public ProxyRotator) is shared with the YouTube/transcript and chat features, so it stays Apache-2; only the crawl-ladder-coupled rotation-retry orchestration (web_crawler/connector.py::_run_tier_with_proxy_retry) lives here.