SurfSense/surfsense_backend/scripts
DESKTOP-RTLN3BA\$punk 80927a2872 feat(billing): meter platform scrapers per item; consolidate web scraping onto web.crawl
Add per-item, per-platform billing for the platform-native connectors (Reddit, Google Search, Google Maps places/reviews, YouTube videos/comments) through the capability gate/charge seam. Rates are config-driven with a shared wallet-credit module (wallet_credit) and a dedicated PlatformScrapeCreditService; agent and REST capability runs now record cost_micros. Google Maps scrape dual-meters places and attached reviews.

Remove the main-agent scrape_webpage tool now that the web.crawl capability covers single-page (maxCrawlDepth=0) and site crawling. The main agent now reaches crawling via task(web_crawler, ...). Update prompts, tool catalog, receipts, skills, proprietary docs, and tests; drop the obsolete chat-turn crawl fold path.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-05 17:08:01 -07:00
..
docker feat(docker): add ZERO_AUTO_RESET configuration for improved replication safety 2026-06-06 14:21:14 +05:30
whatsapp-bridge feat(gateway): enhance WhatsApp bridge with pairing timeout and health check integration 2026-05-29 13:37:45 +05:30
create_sandbox_snapshot.py feat: added ai file sorting 2026-04-14 01:43:30 -07:00
e2e_google_maps_deep.py Merge upstream/ci_mvp (google maps scrapers) into feature-ci_phase-4-7 2026-07-03 12:37:12 +02:00
e2e_google_maps_scraper.py Merge upstream/ci_mvp (google maps scrapers) into feature-ci_phase-4-7 2026-07-03 12:37:12 +02:00
e2e_google_search.py feat(native-connector): added google search results scraper 2026-07-03 20:51:45 -07:00
e2e_phase3_crawl_billing.py feat(billing): meter platform scrapers per item; consolidate web scraping onto web.crawl 2026-07-05 17:08:01 -07:00
e2e_reddit_scraper.py feat(reddit): implement Reddit scraping subagent and associated capabilities 2026-07-04 17:31:11 -07:00
e2e_youtube_scraper.py Merge upstream/ci_mvp (google maps scrapers) into feature-ci_phase-4-7 2026-07-03 12:37:12 +02:00
register_webhook.py chore: linting 2026-06-09 00:42:26 -07:00
revoke_refresh_tokens_cutover.py fix(auth):enforce session auth cutover 2026-06-24 03:55:39 +05:30
verify_chat_image_capability.py refactor(provider-configuration): standardize provider parameter naming across various modules and improve quota error handling in tests 2026-06-13 14:23:32 +05:30