mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-08 22:22:17 +02:00
c600a2920b
Crawler engine: escalate thin JS-shell pages past static fetch, repair currency-lossy extractions, emit categorized link records with anchor provenance, and decode percent-encoded mailto:/tel: contacts; site crawls reuse the connector ladder via Scrapling's spider engine with URL pattern filters. Agent layer: read_run gains char_offset paging, search_run gains match excerpts, new export_run turns stored runs into CSV workspace docs; reddit search fair-shares the item budget across queries and dedupes cross-query hits. Subagent prompts and routing teach crawl-after-search, full-run coverage before summarizing, and executing own-tool next steps instead of returning partial.
22 lines
895 B
Python
22 lines
895 B
Python
"""App-wide crawler block classification (Apache-2.0, generic).
|
|
|
|
Phase 3e (Slice A). Mirrors ``app/utils/proxy`` and ``app/utils/captcha``: this
|
|
package holds only the **generic, vendor-agnostic** glue — here, a pure block
|
|
classifier (passive telemetry from public anti-bot markers). It is consumed by
|
|
the separately licensed proprietary crawler to label ``CrawlOutcome.block_type``.
|
|
|
|
The **bypass-specific tuning** (the stealth kwargs builder / geoip coherence, and
|
|
the deferred WebGL spoof + humanize choreography) is NOT generic and lives under
|
|
the proprietary boundary in ``app/proprietary/web_crawler/`` (``stealth.py``).
|
|
"""
|
|
|
|
from app.utils.crawl.classifier import BlockType, classify_block
|
|
from app.utils.crawl.contacts import Contacts, extract_contacts, is_social_host
|
|
|
|
__all__ = [
|
|
"BlockType",
|
|
"Contacts",
|
|
"classify_block",
|
|
"extract_contacts",
|
|
"is_social_host",
|
|
]
|