SurfSense/surfsense_backend/app/utils/crawl
DESKTOP-RTLN3BA\$punk c600a2920b feat(crawler): harden web crawler and agent tooling for real-world research tasks
Crawler engine: escalate thin JS-shell pages past static fetch, repair
currency-lossy extractions, emit categorized link records with anchor
provenance, and decode percent-encoded mailto:/tel: contacts; site crawls
reuse the connector ladder via Scrapling's spider engine with URL pattern
filters. Agent layer: read_run gains char_offset paging, search_run gains
match excerpts, new export_run turns stored runs into CSV workspace docs;
reddit search fair-shares the item budget across queries and dedupes
cross-query hits. Subagent prompts and routing teach crawl-after-search,
full-run coverage before summarizing, and executing own-tool next steps
instead of returning partial.
2026-07-05 03:51:16 -07:00
..
__init__.py feat(crawler): harden web crawler and agent tooling for real-world research tasks 2026-07-05 03:51:16 -07:00
classifier.py feat(native-connector): added google maps places & reviews scrapers 2026-07-02 21:58:24 -07:00
contacts.py feat(crawler): harden web crawler and agent tooling for real-world research tasks 2026-07-05 03:51:16 -07:00