SurfSense/surfsense_backend/tests/unit
DESKTOP-RTLN3BA\$punk 80927a2872 feat(billing): meter platform scrapers per item; consolidate web scraping onto web.crawl
Add per-item, per-platform billing for the platform-native connectors (Reddit, Google Search, Google Maps places/reviews, YouTube videos/comments) through the capability gate/charge seam. Rates are config-driven with a shared wallet-credit module (wallet_credit) and a dedicated PlatformScrapeCreditService; agent and REST capability runs now record cost_micros. Google Maps scrape dual-meters places and attached reviews.

Remove the main-agent scrape_webpage tool now that the web.crawl capability covers single-page (maxCrawlDepth=0) and site crawling. The main agent now reaches crawling via task(web_crawler, ...). Update prompts, tool catalog, receipts, skills, proprietary docs, and tests; drop the obsolete chat-turn crawl fold path.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-05 17:08:01 -07:00
..
adapters test: bootstrap pytest environment for backend 2026-02-24 18:19:56 +02:00
agents feat(billing): meter platform scrapers per item; consolidate web scraping onto web.crawl 2026-07-05 17:08:01 -07:00
automations Merge upstream/ci_mvp (google maps scrapers) into feature-ci_phase-4-7 2026-07-03 12:37:12 +02:00
capabilities feat(billing): meter platform scrapers per item; consolidate web scraping onto web.crawl 2026-07-05 17:08:01 -07:00
connector_indexers refactor(connectors): remove legacy web-crawler KB indexer paths (keep enums) 2026-07-03 11:53:33 +02:00
connectors chore: ran linting 2026-04-07 05:55:39 +05:30
db feat: improved agent streaming 2026-04-29 07:20:31 -07:00
e2e_fakes test(backend): enhance Drive file filtering and add unit tests for _drive_list_files 2026-05-06 17:58:58 +05:30
etl_pipeline test(etl-cache): cover over-budget eviction selection 2026-06-12 11:50:52 +02:00
event_bus test: rename SearchSpace -> Workspace across tests + fixtures (Phase 2 Wave F) 2026-06-26 18:36:46 +02:00
gateway test: rename SearchSpace -> Workspace across tests + fixtures (Phase 2 Wave F) 2026-06-26 18:36:46 +02:00
google_unification chore: ran linting 2026-05-09 05:28:09 +05:30
indexing_pipeline feat(native-connector): added google maps places & reviews scrapers 2026-07-02 21:58:24 -07:00
middleware test: rename SearchSpace -> Workspace across tests + fixtures (Phase 2 Wave F) 2026-06-26 18:36:46 +02:00
notifications test: rename SearchSpace -> Workspace across tests + fixtures (Phase 2 Wave F) 2026-06-26 18:36:46 +02:00
observability feat(agents): consolidate connectors under mcp_discovery; route web search through google_search 2026-07-04 21:06:04 -07:00
platforms feat(crawler): harden web crawler and agent tooling for real-world research tasks 2026-07-05 03:51:16 -07:00
podcasts test: rename SearchSpace -> Workspace across tests + fixtures (Phase 2 Wave F) 2026-06-26 18:36:46 +02:00
proprietary feat(crawler): harden web crawler and agent tooling for real-world research tasks 2026-07-05 03:51:16 -07:00
routes test: rename SearchSpace -> Workspace across tests + fixtures (Phase 2 Wave F) 2026-06-26 18:36:46 +02:00
services feat(billing): meter platform scrapers per item; consolidate web scraping onto web.crawl 2026-07-05 17:08:01 -07:00
tasks feat(agents): consolidate connectors under mcp_discovery; route web search through google_search 2026-07-04 21:06:04 -07:00
utils feat(crawler): harden web crawler and agent tooling for real-world research tasks 2026-07-05 03:51:16 -07:00
__init__.py test: bootstrap pytest environment for backend 2026-02-24 18:19:56 +02:00
test_error_contract.py chore: linting 2026-04-22 01:05:31 -07:00
test_obsidian_plugin_indexer.py chore: linting 2026-04-27 14:04:50 -07:00
test_pat_fail_closed_static.py feat(native-connector): added google maps places & reviews scrapers 2026-07-02 21:58:24 -07:00
test_zero_authz_static.py fix(authz):add zero context authorization checks 2026-06-23 12:55:25 +05:30