Add per-item, per-platform billing for the platform-native connectors (Reddit, Google Search, Google Maps places/reviews, YouTube videos/comments) through the capability gate/charge seam. Rates are config-driven with a shared wallet-credit module (wallet_credit) and a dedicated PlatformScrapeCreditService; agent and REST capability runs now record cost_micros. Google Maps scrape dual-meters places and attached reviews.
Remove the main-agent scrape_webpage tool now that the web.crawl capability covers single-page (maxCrawlDepth=0) and site crawling. The main agent now reaches crawling via task(web_crawler, ...). Update prompts, tool catalog, receipts, skills, proprietary docs, and tests; drop the obsolete chat-turn crawl fold path.
Co-authored-by: Cursor <cursoragent@cursor.com>
MCP consolidation:
- Route all MCP-capable connectors (Slack, Jira, Linear, ClickUp, Airtable,
Notion, Confluence, interim Gmail/Calendar, custom MCP) through a single
`mcp_discovery` subagent. Drive/OneDrive/Dropbox stay native to enrich the KB.
- Deprecate Discord/Teams/Luma: no viable official MCP server.
Google-only web search:
- Remove the main-agent `web_search` tool and the SearXNG platform service;
all public web search now flows through the `google_search` subagent via task().
- Deprecate the Tavily/SearXNG/Linkup/Baidu search connectors (HTTP 410 on
create, "Deprecated" badge); guide heavy users to the custom MCP connector.
- Remove web search from anonymous chat (pure Q&A).
- Tear SearXNG out of docker compose + install scripts; drop tavily-python
and linkup-sdk deps and their config/env vars.
Fix:
- metrics._package_version() now swallows any metadata lookup failure. A
malformed editable-install distribution with no `Version` field raised
KeyError deep in importlib.metadata, and since it runs on every
record_subagent_invoke_duration call it was crashing every task()
delegation. Verified end-to-end against live GPT-5.4.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Updated proxy configuration in `.env.example` files to use `PROXY_URL` and `PROXY_URLS` instead of `CUSTOM_PROXY_URL` and `CUSTOM_PROXY_URLS`.
- Introduced `DataImpulseProvider` for proxy management, replacing the deprecated `AnonymousProxiesProvider`.
- Enhanced documentation to reflect changes in proxy setup and usage.
- Adjusted related code in the proxy registry and configuration files to support the new provider structure.
Resolve conflicts against the new native google-maps actor + repo-wide
ruff-format pass:
- Keep legacy webcrawler KB indexer + its test deleted (modify/delete).
- test_validators: keep WEBCRAWLER case removed (validator gone).
- test_fetch_resilience: keep platforms.youtube import path (our reorg).
- Relocate google_maps actor + tests scrapers/ -> platforms/ to match the
reorg convention (youtube already there); rewrite imports + fixture paths.
- Add missing __init__.py across the capabilities/ test subtree so duplicate
test basenames get unique module paths under importlib mode.
Note: google_maps fixture-backed tests error on ci_mvp too (fixtures/*.json
never committed upstream) - pre-existing, out of scope here.
Wire captchatools as the StealthyFetcher-tier page_action to detect, harvest
(egressing from the crawl's own proxy IP), inject, and submit reCAPTCHA v2/v3
and hCaptcha tokens. Opt-in and off by default (zero attempts, zero cost).
Licensing split:
- Apache-2 app/utils/captcha/ holds the generic, vendor-agnostic config
(CaptchaConfig + captcha_enabled() = flag AND key present).
- Proprietary app/proprietary/web_crawler/captcha.py holds the bypass logic
(detect/harvest/inject) plus a process-wide solver latch that halts solving
on unrecoverable errors (no balance / bad key).
Crawler: CrawlOutcome gains captcha_attempts/captcha_solved, surfaced via a
per-call captcha_state dict threaded crawl_url -> _crawl_with_stealthy(_sync)
and stamped onto every stealth terminal outcome. The stealth tier captures the
proxy once and reuses it for both the fetch and the solver (IP-coherence).
Billing: WebCrawlCreditService gains captcha_billing_enabled,
captcha_solves_to_micros, charge_captcha, and a generic check_balance, sharing
a single _apply_debit path. The indexer accumulates attempts (even on failed
crawls), runs a combined crawl+captcha pre-flight, and posts a per-attempt
owner charge as usage_type="web_crawl_captcha". The captcha worst-case is only
reserved when solving is actually enabled, so a solving-off deployment is never
blocked for captcha that can never run. Both chat scrape tools fold attempts
into the current turn before the success/fail branch.
Fully config-driven prices; no migration. New unit tests cover the config,
factory (detection/latch/timeout/cap), credit service, indexer wiring, and the
chat fold.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add a config-driven crawl meter (1 USD / 1000 successes by default, retunable
via WEB_CRAWL_MICROS_PER_SUCCESS) mirroring EtlCreditService:
- WebCrawlCreditService (gate -> pre-check -> post-charge) on the unified wallet
- webcrawler indexer: pre-flight credit block + post-charge of the workspace
owner per success, with a web_crawl TokenUsage audit row
- chat scrape_webpage (main + research): fold each successful scrape into the
active turn's token accumulator so it settles with the chat turn
- WEB_CRAWL_CREDIT_BILLING_ENABLED / WEB_CRAWL_MICROS_PER_SUCCESS config + env
- unit tests (service) + integration tests (indexer wiring)
- scripts/e2e_phase3_crawl_billing.py: manual functional e2e (3a/3b/3c)
Co-authored-by: Cursor <cursoragent@cursor.com>
Add CustomProxyProvider (single endpoint or rotating pool via Scrapling ProxyRotator), registered as 'custom' alongside anonymous_proxies and selectable via PROXY_PROVIDER. Adds is_pool_backed to the ProxyProvider ABC + a zero-arg package helper. The web crawler does a bounded one-per-tier is_proxy_error rotation-retry gated on is_pool_backed() (single-endpoint providers no-op). Config/.env.example gain CUSTOM_PROXY_URL(S). Zero-arg getter contract unchanged for all consumers. Documents the proprietary boundary test (generic proxy infra stays Apache-2). Tests: provider, registry, crawler rotation (16).
Co-authored-by: Cursor <cursoragent@cursor.com>
- Increased ACCESS_TOKEN_LIFETIME_SECONDS from 30 minutes to 60 minutes for improved session duration.
- Introduced getRequestHeadersWithCurrentDesktopAuth function to streamline authorization header management across fetch requests in the chat page.
- Changed default AUTH_TYPE in backend configuration to "LOCAL".
- Updated frontend environment configuration to reflect the new default for packaged clients.
- Adjusted runtime authentication resolution to use "LOCAL" as the fallback value.
- Introduced a new endpoint to check the existence of a global LLM configuration file.
- Updated the frontend to utilize this status, affecting onboarding flow and user experience.
- Added necessary atoms and types for managing global LLM config status in the application state.
- Refactored navigation to ensure proper routing based on the global config status.
The long-running ingestion/podcast/video tasks run on a separate Celery
engine (NullPool), so the web engine's idle_in_transaction_session_timeout
did not cover them — which is exactly where the original 11h zombie
(INSERT INTO chunks) came from. Apply the same protection to the Celery
engine with a generous 60-minute default so a worker that hangs/crashes
mid-transaction can't hold locks on documents/chunks indefinitely, while
never reaping a legitimate per-document embed window.
- config + .env.example: DB_CELERY_IDLE_IN_TX_TIMEOUT_MS (default 3600000).
Co-authored-by: Cursor <cursoragent@cursor.com>
A single abandoned "idle in transaction" session held locks on the
documents table, which blocked the non-concurrent CREATE INDEX (hnsw)
run inside the FastAPI lifespan. Each API restart queued another
CREATE INDEX behind an advisory lock, leaving the server stuck at
"Waiting for application startup." indefinitely and freezing ingestion
writes.
Changes:
- setup_indexes(): build every index with CREATE INDEX CONCURRENTLY
(non-blocking ShareUpdateExclusiveLock) under a per-session
lock_timeout, and make each statement non-fatal so a contended/slow
build is retried next boot instead of wedging startup. Drop leftover
invalid indexes before rebuilding.
- create_db_and_tables(): apply lock_timeout to extension/create_all
DDL and gate the whole bootstrap behind DB_BOOTSTRAP_ON_STARTUP.
- engine: set idle_in_transaction_session_timeout (asyncpg) so an
abandoned transaction is reaped automatically.
- config + .env.example: DB_BOOTSTRAP_ON_STARTUP, DB_DDL_LOCK_TIMEOUT_MS,
DB_IDLE_IN_TX_TIMEOUT_MS.
Co-authored-by: Cursor <cursoragent@cursor.com>
surfsense.indexing.reconcile.chunks counts reused/embedded/deleted chunks per
re-index. CHUNK_RECONCILE_ENABLED (default on) falls back to delete-all +
full re-embed if the diff path ever misbehaves.
The cached payload is the indexing pipeline's embeddings (markdown is
chunked then embedded), so "embedding cache" names the expensive output
directly and removes the "index" ambiguity (DB index vs vector index vs
indexing phase). Renames the service, settings, eligibility, eviction
task, metrics, config flags (INDEX_CACHE_* -> EMBEDDING_CACHE_*), object
prefix, and the table (index_cache_embedding_sets -> embedding_cache_sets)
with its constraint and indexes. Migration 161 renamed accordingly.
- Updated environment variables and - configurations for credit purchases via Stripe, replacing legacy page pack system.
- Introduced auto-reload feature for credit top-ups and modified database models to track credit transactions.
- Updated notification system to handle insufficient credits and auto-reload failures.
- Adjusted API routes and schemas to reflect changes in credit management.
- Replaced Playwright with Scrapling's fetchers in the web crawling and YouTube processing modules for improved performance and flexibility.
- Updated proxy configuration to support dynamic proxy selection via environment variables.
- Enhanced logging to track performance metrics during web scraping operations.
- Refactored related modules to utilize the new proxy utilities and streamline the scraping process.
- Consolidated Redis configuration by introducing a single `REDIS_URL` variable for Celery broker, result backend, and app cache.
- Removed deprecated variables related to Firecrawl and Stripe token limits from `.env.example` files.
- Updated documentation to reflect changes in environment variable usage for improved clarity and maintainability.
- Replaced environment variable usage with a centralized configuration system in multiple modules, including `celery_app`, `agent_cache_store`, `sandbox`, `file_storage`, and `connector_service`.
- Enhanced maintainability and readability by sourcing configuration values from the `config` module instead of directly from environment variables.
- Updated relevant settings to ensure consistent access to configuration values across the application.
- Added a global switch `GATEWAY_ENABLED` to control the activation of all messaging gateway channels (Telegram, WhatsApp, Slack, Discord).
- Updated relevant routes and workers to check the `GATEWAY_ENABLED` flag, returning 404 for HTTP routes when disabled.
- Enhanced documentation in the `.env.example` file to reflect the new configuration option.
