apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-06 22:12:12 +02:00
Code Issues Projects Releases Packages Wiki Activity

feat: completed init mvp of phase 3

Browse source
This commit is contained in:
DESKTOP-RTLN3BA\$punk 2026-06-30 03:02:40 -07:00
parent bdacea8b6e
commit 15fd0b08d6
20 changed files with 2107 additions and 69 deletions
Download patch file Download diff file Expand all files Collapse all files

18
surfsense_backend/app/utils/crawl/__init__.py Normal file
View file

@ -0,0 +1,18 @@
"""App-wide crawler block classification (Apache-2.0, generic).
Phase 3e (Slice A). Mirrors ``app/utils/proxy`` and ``app/utils/captcha``: this
package holds only the **generic, vendor-agnostic** glue — here, a pure block
classifier (passive telemetry from public anti-bot markers). It is consumed by
the separately licensed proprietary crawler to label ``CrawlOutcome.block_type``.
The **bypass-specific tuning** (the stealth kwargs builder / geoip coherence, and
the deferred WebGL spoof + humanize choreography) is NOT generic and lives under
the proprietary boundary in ``app/proprietary/web_crawler/`` (``stealth.py``).
"""
from app.utils.crawl.classifier import BlockType, classify_block
__all__ = [
"BlockType",
"classify_block",
]