apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-08 22:22:17 +02:00
Code Issues Projects Releases Packages Wiki Activity

feat(billing): meter platform scrapers per item; consolidate web scraping onto web.crawl

Browse source 
Add per-item, per-platform billing for the platform-native connectors (Reddit, Google Search, Google Maps places/reviews, YouTube videos/comments) through the capability gate/charge seam. Rates are config-driven with a shared wallet-credit module (wallet_credit) and a dedicated PlatformScrapeCreditService; agent and REST capability runs now record cost_micros. Google Maps scrape dual-meters places and attached reviews.

Remove the main-agent scrape_webpage tool now that the web.crawl capability covers single-page (maxCrawlDepth=0) and site crawling. The main agent now reaches crawling via task(web_crawler, ...). Update prompts, tool catalog, receipts, skills, proprietary docs, and tests; drop the obsolete chat-turn crawl fold path.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
DESKTOP-RTLN3BA\$punk 2026-07-05 17:08:01 -07:00
parent b8285a0b72
commit 80927a2872
48 changed files with 724 additions and 766 deletions
Download patch file Download diff file Expand all files Collapse all files

21
surfsense_backend/.env.example
View file

@ -256,10 +256,10 @@ MICROS_PER_PAGE=1000
# crawling effectively free for self-hosted/OSS installs; hosted sets TRUE.
# Price is fully config-driven (the only source of truth, no hardcoded rate):
# WEB_CRAWL_MICROS_PER_SUCCESS = round(USD_per_1000_crawls * 1_000)
# 1000 == $1 / 1000 crawls (default) | 2000 == $2/1000 | 500 == $0.50/1000
# 2000 == $2 / 1000 crawls (default) | 1000 == $1/1000 | 500 == $0.50/1000
# Retune anytime with just an env change + restart (no code/migration).
# WEB_CRAWL_CREDIT_BILLING_ENABLED=FALSE
# WEB_CRAWL_MICROS_PER_SUCCESS=1000
# WEB_CRAWL_MICROS_PER_SUCCESS=2000
# Phase 3d: bill captcha solves as a SEPARATE per-attempt unit (the solver
# charges per attempt regardless of crawl success). Independent of the crawl
@ -270,6 +270,23 @@ MICROS_PER_PAGE=1000
# WEB_CRAWL_CAPTCHA_BILLING_ENABLED=FALSE
# WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE=3000
# Debit the credit wallet per *item returned* by the platform-native scrapers
# (Reddit, Google Search, Google Maps, YouTube). Default FALSE keeps scraping
# effectively free for self-hosted/OSS installs; hosted deployments set TRUE.
# Each rate is micro-USD per item, fully config-driven (no hardcoded rate):
# <KEY> = round(USD_per_1000_items * 1_000)
# 3500 == $3.50/1000 | 5000 == $5/1000 | 2000 == $2/1000
# Defaults sit at/above Apify's first-party actor rates (Jul 2026); justified
# because we charge no subscription tiers, no per-run start fees, and no
# separate proxy/compute/storage billing. Retune anytime via env + restart.
# PLATFORM_SCRAPE_BILLING_ENABLED=FALSE
# REDDIT_SCRAPE_MICROS_PER_ITEM=3500
# GOOGLE_SEARCH_MICROS_PER_SERP=5500
# GOOGLE_MAPS_MICROS_PER_PLACE=5000
# GOOGLE_MAPS_MICROS_PER_REVIEW=2000
# YOUTUBE_MICROS_PER_VIDEO=3500
# YOUTUBE_MICROS_PER_COMMENT=3500
# Low-balance warning threshold (micro-USD), surfaced to the UI. Default $0.50.
CREDIT_LOW_BALANCE_WARNING_MICROS=500000