Commit graph

142 commits

Author SHA1 Message Date
DESKTOP-RTLN3BA\$punk
80927a2872 feat(billing): meter platform scrapers per item; consolidate web scraping onto web.crawl
Add per-item, per-platform billing for the platform-native connectors (Reddit, Google Search, Google Maps places/reviews, YouTube videos/comments) through the capability gate/charge seam. Rates are config-driven with a shared wallet-credit module (wallet_credit) and a dedicated PlatformScrapeCreditService; agent and REST capability runs now record cost_micros. Google Maps scrape dual-meters places and attached reviews.

Remove the main-agent scrape_webpage tool now that the web.crawl capability covers single-page (maxCrawlDepth=0) and site crawling. The main agent now reaches crawling via task(web_crawler, ...). Update prompts, tool catalog, receipts, skills, proprietary docs, and tests; drop the obsolete chat-turn crawl fold path.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-05 17:08:01 -07:00
DESKTOP-RTLN3BA\$punk
ab747e7a49 feat(agents): consolidate connectors under mcp_discovery; route web search through google_search
MCP consolidation:
- Route all MCP-capable connectors (Slack, Jira, Linear, ClickUp, Airtable,
  Notion, Confluence, interim Gmail/Calendar, custom MCP) through a single
  `mcp_discovery` subagent. Drive/OneDrive/Dropbox stay native to enrich the KB.
- Deprecate Discord/Teams/Luma: no viable official MCP server.

Google-only web search:
- Remove the main-agent `web_search` tool and the SearXNG platform service;
  all public web search now flows through the `google_search` subagent via task().
- Deprecate the Tavily/SearXNG/Linkup/Baidu search connectors (HTTP 410 on
  create, "Deprecated" badge); guide heavy users to the custom MCP connector.
- Remove web search from anonymous chat (pure Q&A).
- Tear SearXNG out of docker compose + install scripts; drop tavily-python
  and linkup-sdk deps and their config/env vars.

Fix:
- metrics._package_version() now swallows any metadata lookup failure. A
  malformed editable-install distribution with no `Version` field raised
  KeyError deep in importlib.metadata, and since it runs on every
  record_subagent_invoke_duration call it was crashing every task()
  delegation. Verified end-to-end against live GPT-5.4.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-04 21:06:04 -07:00
DESKTOP-RTLN3BA\$punk
701f888b9e feat(proxy): refactor proxy configuration and add DataImpulse provider
- Updated proxy configuration in `.env.example` files to use `PROXY_URL` and `PROXY_URLS` instead of `CUSTOM_PROXY_URL` and `CUSTOM_PROXY_URLS`.
- Introduced `DataImpulseProvider` for proxy management, replacing the deprecated `AnonymousProxiesProvider`.
- Enhanced documentation to reflect changes in proxy setup and usage.
- Adjusted related code in the proxy registry and configuration files to support the new provider structure.
2026-07-04 17:18:40 -07:00
CREDO23
88a94611c9 chore(docker): add isolated deps compose for watch e2e 2026-07-02 17:54:31 +02:00
DESKTOP-RTLN3BA\$punk
2e4959f947 fix(docker): add reverse proxy for zero sync auth context in Caddyfile 2026-06-26 12:24:12 -07:00
DESKTOP-RTLN3BA\$punk
b4e8aa0a1c feat(docker): add proxy directory and Caddyfile to installation script 2026-06-26 12:00:50 -07:00
Rohan Verma
94fdb8a113
Merge pull request #1539 from CREDO23/improve-chat-agent-context-and-citations
[FEAT] Unified [n] citation registry for KB + web, pull-based retrieval
2026-06-25 13:34:52 -07:00
CREDO23
2beafbdec8 agent: retire eager KB priority/planner path and its dead flags
The pull-based KB design (on-demand search_knowledge_base tool + pre-injected
workspace tree) fully replaced the old eager retrieval path. Remove its last
remnants:

- Delete KnowledgePriorityMiddleware (knowledge_search.py) and its tests.
- Drop the kb_priority state field + reducer default; trim
  KbContextProjectionMiddleware to project only workspace_tree_text.
- Remove the now-dead feature flags enable_kb_priority_preinjection and
  enable_kb_planner_runnable across backend (flags, route schema, tests,
  env examples) and frontend (settings toggle, zod schema).
- Scrub <priority_documents> and stale KnowledgePriorityMiddleware references
  from prompts, docstrings, and the ADR.

No functional change: nothing wrote kb_priority and neither flag gated live
behavior after the cutover. Full backend suite green (pre-existing unrelated
failures aside).
2026-06-25 18:37:14 +02:00
Anish Sarkar
33f353bd20 fix(docker):wire auth service configuration 2026-06-23 12:48:44 +05:30
Anish Sarkar
4d49d9311c fix(config):add auth environment defaults 2026-06-23 12:48:23 +05:30
Anish Sarkar
83fdef2180 chore(dependencies): update @rocicorp/zero to version 1.6.0 in docker-compose files and package.json 2026-06-22 02:33:18 +05:30
DESKTOP-RTLN3BA\$punk
0fe650fd8e Merge commit '7ce409c580' into dev 2026-06-16 22:48:14 -07:00
Anish Sarkar
9b7e278114 refactor(config): update GATEWAY_ENABLED variable to FALSE and adjust related configurations for improved messaging gateway handling 2026-06-16 23:49:26 +05:30
CREDO23
a7be41d50a fix(docker): share persistent object_store volume in dev 2026-06-16 20:09:08 +02:00
CREDO23
fc045d200d fix(docker): share persistent object_store volume across services 2026-06-16 20:09:08 +02:00
Anish Sarkar
66659ee8d3 feat(docker): update Caddyfile to include specific reverse proxy routes for frontend auth and backend user profiles 2026-06-16 04:36:26 +05:30
Anish Sarkar
34a49df8a3 docs: document runtime env vars and same-origin proxy access 2026-06-16 02:12:11 +05:30
Anish Sarkar
1313ec2109 fix(docker): disable response buffering for backend streaming 2026-06-16 02:11:32 +05:30
Anish Sarkar
cd5e0abe53 refactor(docker): drop build-time NEXT_PUBLIC injection for runtime env 2026-06-16 02:11:13 +05:30
Anish Sarkar
cb1565c8f3 docs(docker): update .env.example for improved proxy configuration and clarity 2026-06-15 19:42:37 +05:30
Anish Sarkar
2c6cf8d795 refactor(docker): update installer output for proxy access 2026-06-15 11:04:02 +05:30
Anish Sarkar
2373014943 feat(docker): route production stack through bundled proxy 2026-06-15 11:03:33 +05:30
Anish Sarkar
1f835d2818 feat(docker): add caddy reverse proxy assets 2026-06-15 11:03:12 +05:30
Anish Sarkar
25541e8d3b chore: remove commented-out FLOWER_PORT from .env.example 2026-06-14 20:23:43 +05:30
DESKTOP-RTLN3BA\$punk
a7407502d3 feat(refactor): refactor payment system to implement unified credit wallet.
- Updated environment variables and - configurations for credit purchases via Stripe, replacing legacy page pack system.
- Introduced auto-reload feature for credit top-ups and modified database models to track credit transactions.
- Updated notification system to handle insufficient credits and auto-reload failures.
- Adjusted API routes and schemas to reflect changes in credit management.
2026-06-10 16:49:03 -07:00
DESKTOP-RTLN3BA\$punk
a024b03fb0 fix(install.ps1): add manual validation for Variant parameter to prevent errors during script execution 2026-06-10 00:57:41 -07:00
DESKTOP-RTLN3BA\$punk
41a93ca8fb refactor(env): streamline Redis configuration and remove deprecated variables
- Consolidated Redis configuration by introducing a single `REDIS_URL` variable for Celery broker, result backend, and app cache.
- Removed deprecated variables related to Firecrawl and Stripe token limits from `.env.example` files.
- Updated documentation to reflect changes in environment variable usage for improved clarity and maintainability.
2026-06-08 14:37:44 -07:00
Anish Sarkar
4e00f24a03 feat(docker): add ZERO_AUTO_RESET configuration for improved replication safety
- Introduced the ZERO_AUTO_RESET environment variable to enable automatic reset of the SQLite replica in case of replication halts.
- Updated Docker Compose files to include ZERO_AUTO_RESET in service configurations.
- Enhanced documentation to clarify the purpose and usage of the new variable.
2026-06-06 14:21:14 +05:30
Anish Sarkar
19fabaf011 feat(docker): enhance banner display to installation scripts 2026-06-06 01:49:41 +05:30
Anish Sarkar
4bfa04ed57 feat(docker): add interactive Watchtower update preference to installation scripts 2026-06-06 01:45:27 +05:30
Anish Sarkar
c5afce3873 feat(docker): add GPU support and enhance installation scripts
- Introduced a new docker-compose.gpu.yml file to define GPU resource reservations for backend services.
- Updated .env.example to include GPU-related environment variables and usage instructions.
- Enhanced install.ps1 and install.sh scripts to support GPU variant selection and validation for GPU count.
- Improved error handling and user feedback for invalid GPU configurations.
2026-06-06 01:15:04 +05:30
Anish Sarkar
7e4077d67a chore(docker): remove commented-out flower service from Docker Compose files 2026-06-06 00:45:56 +05:30
Anish Sarkar
c1ee5fabf9 feat(docker): enhance multi-architecture and CUDA support in Docker workflows
- Added support for multiple image variants (CPU, CUDA 12.8, CUDA 12.6) in the Docker build workflow.
- Updated Docker Compose configuration to utilize the new SURFSENSE_VARIANT environment variable for selecting image variants.
- Enhanced documentation to clarify usage of backend image variants and their corresponding environment variable settings.
2026-06-05 23:39:42 +05:30
Anish Sarkar
54fc2d4619 chore(docker): comment out whatsapp-bridge service in docker-compose.yml 2026-06-05 02:24:14 +05:30
Anish Sarkar
3c82200e89 chore(docker): comment out otel-collector service in docker-compose.yml 2026-06-04 22:29:39 +05:30
CREDO23
a4f3af7ed8 chore(dev): document storage env vars and add azurite service 2026-06-02 16:10:50 +02:00
Anish Sarkar
ebddf4506a feat(messaging): introduce comprehensive setup docs for Telegram, WhatsApp, Slack, and Discord messaging channels 2026-06-02 00:25:49 +05:30
Anish Sarkar
20994671bc feat(gateway): add configuration options for Telegram, WhatsApp, Slack, and Discord gateways 2026-06-02 00:03:26 +05:30
Anish Sarkar
799a83239f feat(gateway): add Slack and Telegram gateway configuration and enablement checks 2026-06-01 23:58:28 +05:30
Anish Sarkar
9143cb6ad1 feat(gateway): update WhatsApp bridge configuration and expose port 9929 2026-05-29 11:39:11 +05:30
Anish Sarkar
51bf2a8361 feat(gateway): wire WhatsApp bridge runtime 2026-05-29 10:20:25 +05:30
Anish Sarkar
4c8d47617d feat(env): add SURFSENSE_ENV variable for deployment environment and update observability resource attributes 2026-05-23 02:13:24 +05:30
Anish Sarkar
df698e0216 feat(observability): integrate OpenTelemetry collector and configuration for enhanced telemetry 2026-05-23 00:17:23 +05:30
Anish Sarkar
60049936e3 chore(dev): add local OpenTelemetry backend configuration 2026-05-21 23:00:56 +05:30
DESKTOP-RTLN3BA\$punk
b285293b4e fix: docker one click setup 2026-05-20 01:25:07 -07:00
Anish Sarkar
af1d2fa430 Merge remote-tracking branch 'upstream/dev' into fix/zero-cache-stale-replica-1355 2026-05-16 19:30:09 +05:30
Anish Sarkar
ce4531461c chore: update zero-cache image and dependencies to version 1.4.0 2026-05-12 15:24:19 +05:30
Anish Sarkar
315329f344 chore: update E2E tests workflow to capture logs on cancellation and add shared volume for backend services 2026-05-12 01:35:33 +05:30
Anish Sarkar
b247ff37df chore: implement test-only token mint endpoint and update E2E test authentication flow 2026-05-11 19:48:18 +05:30
Anish Sarkar
83e40c5aea chore: update Docker configuration to include pnpm workspace and refine dependency management 2026-05-11 15:31:24 +05:30