mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-06 22:12:12 +02:00
5c36cd3071
- Standardized the web crawler to use Scrapling exclusively, removing Firecrawl entirely. - Updated the crawler's location to `app/proprietary/web_crawler/connector.py` under a non-Apache-2 license boundary. - Refactored the `WebCrawlerConnector` to eliminate the Firecrawl API key dependency, simplifying the interface for crawling URLs. - Adjusted related components to accommodate the new structure and ensure successful crawl outcomes are properly handled. - Updated documentation to reflect these changes and the new implementation status. |
||
|---|---|---|
| .. | ||
| 00-umbrella-plan.md | ||
| 01-rename-db.md | ||
| 02-rename-backend.md | ||
| 03a-crawler-core.md | ||
| 03b-proxy-expansion.md | ||
| 03c-crawl-billing.md | ||
| 03d-captcha-solving.md | ||
| 03e-stealth-hardening.md | ||
| 03f-undetectability-testing.md | ||
| 04a-connector-category.md | ||
| 04b-source-discovery.md | ||
| 05-pipelines-model.md | ||
| 06-pipelines-exec.md | ||
| 07-upload-pipeline-kb.md | ||