SurfSense/surfsense_backend
DESKTOP-RTLN3BA\$punk 80927a2872 feat(billing): meter platform scrapers per item; consolidate web scraping onto web.crawl
Add per-item, per-platform billing for the platform-native connectors (Reddit, Google Search, Google Maps places/reviews, YouTube videos/comments) through the capability gate/charge seam. Rates are config-driven with a shared wallet-credit module (wallet_credit) and a dedicated PlatformScrapeCreditService; agent and REST capability runs now record cost_micros. Google Maps scrape dual-meters places and attached reviews.

Remove the main-agent scrape_webpage tool now that the web.crawl capability covers single-page (maxCrawlDepth=0) and site crawling. The main agent now reaches crawling via task(web_crawler, ...). Update prompts, tool catalog, receipts, skills, proprietary docs, and tests; drop the obsolete chat-turn crawl fold path.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-05 17:08:01 -07:00
..
alembic feat(runs): introduce Run and ToolOutputSpill models for logging scraper invocations 2026-07-04 22:55:55 -07:00
app feat(billing): meter platform scrapers per item; consolidate web scraping onto web.crawl 2026-07-05 17:08:01 -07:00
scripts feat(billing): meter platform scrapers per item; consolidate web scraping onto web.crawl 2026-07-05 17:08:01 -07:00
tests feat(billing): meter platform scrapers per item; consolidate web scraping onto web.crawl 2026-07-05 17:08:01 -07:00
.dockerignore chore(backend): exclude tests/ from production Docker image 2026-05-06 17:16:22 +05:30
.env.example feat(billing): meter platform scrapers per item; consolidate web scraping onto web.crawl 2026-07-05 17:08:01 -07:00
.gitignore fix(gitignore): anchor data/ rule; track podcast voice catalogs 2026-06-12 00:06:37 +02:00
.python-version feat: SurfSense v0.0.6 init 2025-03-14 18:53:14 -07:00
alembic.ini add github connector, add alembic for db migrations, fix bug updating connectors 2025-04-13 13:56:22 -07:00
celery_worker.py fix: celery_app path and gmail indexing 2025-10-21 21:11:41 -07:00
Dockerfile feat: completed init mvp of phase 3 2026-06-30 03:02:40 -07:00
main.py feat(observability): add OpenTelemetry process bootstrap 2026-05-21 23:01:54 +05:30
pyproject.toml feat(agents): consolidate connectors under mcp_discovery; route web search through google_search 2026-07-04 21:06:04 -07:00
uv.lock feat(crawler): harden web crawler and agent tooling for real-world research tasks 2026-07-05 03:51:16 -07:00