apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-06 22:12:12 +02:00
Code Issues Projects Releases Packages Wiki Activity

feat(03c): bill credits per successful web crawl

Browse source 
Add a config-driven crawl meter (1 USD / 1000 successes by default, retunable
via WEB_CRAWL_MICROS_PER_SUCCESS) mirroring EtlCreditService:

- WebCrawlCreditService (gate -> pre-check -> post-charge) on the unified wallet
- webcrawler indexer: pre-flight credit block + post-charge of the workspace
  owner per success, with a web_crawl TokenUsage audit row
- chat scrape_webpage (main + research): fold each successful scrape into the
  active turn's token accumulator so it settles with the chat turn
- WEB_CRAWL_CREDIT_BILLING_ENABLED / WEB_CRAWL_MICROS_PER_SUCCESS config + env
- unit tests (service) + integration tests (indexer wiring)
- scripts/e2e_phase3_crawl_billing.py: manual functional e2e (3a/3b/3c)

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
DESKTOP-RTLN3BA\$punk 2026-06-29 21:40:53 -07:00
parent a517eb7d2d
commit 17bdb06825
9 changed files with 1041 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

9
surfsense_backend/.env.example
View file

@ -257,6 +257,15 @@ DEFAULT_CREDIT_MICROS_BALANCE=5000000
ETL_CREDIT_BILLING_ENABLED=FALSE
MICROS_PER_PAGE=1000
# Debit the credit wallet per *successful* web crawl. Default FALSE keeps
# crawling effectively free for self-hosted/OSS installs; hosted sets TRUE.
# Price is fully config-driven (the only source of truth, no hardcoded rate):
# WEB_CRAWL_MICROS_PER_SUCCESS = round(USD_per_1000_crawls * 1_000)
# 1000 == $1 / 1000 crawls (default) | 2000 == $2/1000 | 500 == $0.50/1000
# Retune anytime with just an env change + restart (no code/migration).
# WEB_CRAWL_CREDIT_BILLING_ENABLED=FALSE
# WEB_CRAWL_MICROS_PER_SUCCESS=1000
# Low-balance warning threshold (micro-USD), surfaced to the UI. Default $0.50.
CREDIT_LOW_BALANCE_WARNING_MICROS=500000