apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-06 22:12:12 +02:00
Code Issues Projects Releases Packages Wiki Activity
SurfSense/surfsense_backend/app/proprietary
History
DESKTOP-RTLN3BA\$punk 5c36cd3071 feat(03a): implement Scrapling-only web crawler and remove Firecrawl integration 
- Standardized the web crawler to use Scrapling exclusively, removing Firecrawl entirely.
- Updated the crawler's location to `app/proprietary/web_crawler/connector.py` under a non-Apache-2 license boundary.
- Refactored the `WebCrawlerConnector` to eliminate the Firecrawl API key dependency, simplifying the interface for crawling URLs.
- Adjusted related components to accommodate the new structure and ensure successful crawl outcomes are properly handled.
- Updated documentation to reflect these changes and the new implementation status.
2026-06-29 20:37:56 -07:00
..
platforms feat(03a): implement Scrapling-only web crawler and remove Firecrawl integration 2026-06-29 20:37:56 -07:00
web_crawler feat(03a): implement Scrapling-only web crawler and remove Firecrawl integration 2026-06-29 20:37:56 -07:00
__init__.py feat(03a): implement Scrapling-only web crawler and remove Firecrawl integration 2026-06-29 20:37:56 -07:00
LICENSE feat(03a): implement Scrapling-only web crawler and remove Firecrawl integration 2026-06-29 20:37:56 -07:00
README.md feat(03a): implement Scrapling-only web crawler and remove Firecrawl integration 2026-06-29 20:37:56 -07:00

README.md

app.proprietary — non-Apache-2 license boundary

Everything in this directory tree is licensed separately from the rest of SurfSense (which is Apache-2.0). See LICENSE.

Why this exists

This package holds the product moat:

  • the in-house undetectable web crawler (Scrapling tiers + stealth/captcha hardening), and
  • (future) platform-specific actors that scrape/extract structured data from individual platforms.

Keeping it in one clearly-named directory makes the license boundary unambiguous: a single rule — everything under app/proprietary/** is not Apache-2.0 — instead of per-file headers scattered across the tree.

Layout

  • web_crawler/ — the Scrapling-based crawler engine. Public API: WebCrawlerConnector, CrawlOutcome, CrawlOutcomeStatus (from app.proprietary.web_crawler import ...).
  • platforms/ — (future, Phase 8) platform-specific actors; scaffolded/empty.

Rules

  • Do not add Apache-2.0-intended code here.
  • Apache-2.0 code elsewhere may import from this package (the indexer and the chat scrape_webpage tools do); that does not move them under this license.
  • Depend only on the public API exported from each subpackage's __init__, not on internal modules, so the boundary stays clean and swappable.