apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-06 22:12:12 +02:00
Code Issues Projects Releases Packages Wiki Activity

feat(03a): implement Scrapling-only web crawler and remove Firecrawl integration

Browse source 
- Standardized the web crawler to use Scrapling exclusively, removing Firecrawl entirely.
- Updated the crawler's location to `app/proprietary/web_crawler/connector.py` under a non-Apache-2 license boundary.
- Refactored the `WebCrawlerConnector` to eliminate the Firecrawl API key dependency, simplifying the interface for crawling URLs.
- Adjusted related components to accommodate the new structure and ensure successful crawl outcomes are properly handled.
- Updated documentation to reflect these changes and the new implementation status.
This commit is contained in:
DESKTOP-RTLN3BA\$punk 2026-06-29 20:37:56 -07:00
parent 34ab23f5d8
commit 5c36cd3071
32 changed files with 506 additions and 257 deletions
Download patch file Download diff file Expand all files Collapse all files

2
surfsense_backend/Dockerfile
View file

@ -109,7 +109,7 @@ RUN --mount=type=secret,id=HF_TOKEN \
HF_TOKEN="$(cat /run/secrets/HF_TOKEN 2>/dev/null || true)" \
python -c "from chonkie import AutoEmbeddings; AutoEmbeddings.get_embeddings('${EMBEDDING_MODEL}')"
# Install Scrapling's browser engines (patchright Chromium + Camoufox).
# Install Scrapling's browser engines (patchright Chromium).
# Scrapling pulls playwright/patchright via the `fetchers` extra; `scrapling install`
# downloads the matching browser binaries used by DynamicFetcher/StealthyFetcher.
RUN scrapling install