Commit graph

7292 commits

Author SHA1 Message Date
DESKTOP-RTLN3BA\$punk
ab747e7a49 feat(agents): consolidate connectors under mcp_discovery; route web search through google_search
MCP consolidation:
- Route all MCP-capable connectors (Slack, Jira, Linear, ClickUp, Airtable,
  Notion, Confluence, interim Gmail/Calendar, custom MCP) through a single
  `mcp_discovery` subagent. Drive/OneDrive/Dropbox stay native to enrich the KB.
- Deprecate Discord/Teams/Luma: no viable official MCP server.

Google-only web search:
- Remove the main-agent `web_search` tool and the SearXNG platform service;
  all public web search now flows through the `google_search` subagent via task().
- Deprecate the Tavily/SearXNG/Linkup/Baidu search connectors (HTTP 410 on
  create, "Deprecated" badge); guide heavy users to the custom MCP connector.
- Remove web search from anonymous chat (pure Q&A).
- Tear SearXNG out of docker compose + install scripts; drop tavily-python
  and linkup-sdk deps and their config/env vars.

Fix:
- metrics._package_version() now swallows any metadata lookup failure. A
  malformed editable-install distribution with no `Version` field raised
  KeyError deep in importlib.metadata, and since it runs on every
  record_subagent_invoke_duration call it was crashing every task()
  delegation. Verified end-to-end against live GPT-5.4.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-04 21:06:04 -07:00
DESKTOP-RTLN3BA\$punk
ff2e5f390f feat(reddit): implement Reddit scraping subagent and associated capabilities
- Added a new `reddit` subagent to scrape structured data from Reddit posts, comments, and users.
- Introduced `reddit.scrape` capability for fetching data using URLs and search queries.
- Implemented tools for scraping and parsing Reddit data, including handling pagination and rate limits.
- Created input/output models for the Reddit scraper to define request and response structures.
- Added documentation for the new Reddit scraping functionality and its usage.
- Integrated the Reddit subagent into the existing multi-agent chat framework.
2026-07-04 17:31:11 -07:00
DESKTOP-RTLN3BA\$punk
701f888b9e feat(proxy): refactor proxy configuration and add DataImpulse provider
- Updated proxy configuration in `.env.example` files to use `PROXY_URL` and `PROXY_URLS` instead of `CUSTOM_PROXY_URL` and `CUSTOM_PROXY_URLS`.
- Introduced `DataImpulseProvider` for proxy management, replacing the deprecated `AnonymousProxiesProvider`.
- Enhanced documentation to reflect changes in proxy setup and usage.
- Adjusted related code in the proxy registry and configuration files to support the new provider structure.
2026-07-04 17:18:40 -07:00
Rohan Verma
203215adc0
Merge pull request #1572 from CREDO23/add-google-search-verb
[Feat] CI : Add google_search.scrape verb + subagent
2026-07-04 15:41:39 -07:00
Rohan Verma
07ace7ff7d
Merge pull request #1571 from AnishSarkar22/feat/reddit-scraping
feat(scrapers): add anonymous Reddit scraper
2026-07-04 15:37:53 -07:00
CREDO23
371e33e02c feat(google_search): map subagent connector requirement 2026-07-04 20:49:59 +02:00
CREDO23
16db4e9e4a feat(google_search): register subagent builder 2026-07-04 20:49:59 +02:00
CREDO23
0d22b20023 feat(google_search): add subagent system prompt 2026-07-04 20:49:59 +02:00
CREDO23
725e21d262 feat(google_search): add subagent description 2026-07-04 20:49:59 +02:00
CREDO23
21bd1aec0e feat(google_search): add subagent builder 2026-07-04 20:49:59 +02:00
CREDO23
87869d0979 feat(google_search): add subagent tools 2026-07-04 20:49:58 +02:00
CREDO23
a6332e23e9 feat(google_search): wire capability into REST door 2026-07-04 20:49:58 +02:00
CREDO23
b4ad8c5f4c feat(google_search): register scrape capability 2026-07-04 20:49:58 +02:00
CREDO23
fcf1c6e052 feat(google_search): add scrape executor 2026-07-04 20:49:58 +02:00
CREDO23
b568dbefda feat(google_search): add scrape verb schemas 2026-07-04 20:49:58 +02:00
Anish Sarkar
82141d8047 chore: fix docs 2026-07-04 17:45:43 +05:30
Anish Sarkar
0a83bf6192 test(native-connector): cover reddit schemas and url resolver 2026-07-04 17:28:21 +05:30
Anish Sarkar
0ef02c43cc test(native-connector): cover reddit parser mappings 2026-07-04 17:28:17 +05:30
Anish Sarkar
1b1e612e20 test(native-connector): cover reddit fetch resilience 2026-07-04 17:28:13 +05:30
Anish Sarkar
4d950ddcb5 test(native-connector): add reddit post fixture 2026-07-04 17:28:08 +05:30
Anish Sarkar
672f79d620 test(native-connector): add reddit listing fixture 2026-07-04 17:28:02 +05:30
Anish Sarkar
83f52b17b1 test(native-connector): add reddit comment fixture 2026-07-04 17:27:56 +05:30
Anish Sarkar
f14bf1f06d test(native-connector): add reddit scraper test package 2026-07-04 17:27:53 +05:30
Anish Sarkar
975fdad0ef test(native-connector): add live reddit scraper e2e check 2026-07-04 17:27:47 +05:30
Anish Sarkar
3956045221 feat(native-connector): classify reddit scrape urls 2026-07-04 17:27:44 +05:30
Anish Sarkar
eaa8239e36 feat(native-connector): add reddit scraper orchestrator 2026-07-04 17:27:38 +05:30
Anish Sarkar
dd43ab8505 feat(native-connector): add reddit scraper schemas 2026-07-04 17:27:33 +05:30
Anish Sarkar
0745f433ab feat(native-connector): add reddit json parsers 2026-07-04 17:27:22 +05:30
Anish Sarkar
7151465e12 feat(native-connector): add anonymous reddit fetch layer 2026-07-04 17:27:13 +05:30
Anish Sarkar
1115d647cb feat(native-connector): expose reddit scraper public API 2026-07-04 17:26:58 +05:30
Anish Sarkar
54eab56b25 docs(native-connector): document anonymous reddit scraper design 2026-07-04 17:26:12 +05:30
DESKTOP-RTLN3BA\$punk
0d83916cc5 feat(native-connector): added google search results scraper 2026-07-03 20:51:45 -07:00
Rohan Verma
de04ef7111
Merge pull request #1567 from CREDO23/feature-ci_phase-4-7
[Feat] CI: unified web.crawl + per-platform subagents; retire legacy connectors
2026-07-03 14:55:07 -07:00
CREDO23
dd61728111 refactor(subagents): drop automation/monitoring framing from prompts (keep in-chat comparison) 2026-07-03 18:36:48 +02:00
CREDO23
7691e95e36 refactor(capabilities): drop Apify references from web/youtube docstrings 2026-07-03 17:57:13 +02:00
CREDO23
b5d221c19c fix(capabilities): map any executor fault to 502 at the REST door 2026-07-03 17:57:07 +02:00
CREDO23
0844dd581f refactor(routes): drop duplicate /google-maps/* routes (use capability doors) 2026-07-03 17:56:59 +02:00
CREDO23
2c76ef1f57 feat(subagents): add google_maps builtin subagent 2026-07-03 17:56:53 +02:00
CREDO23
f31823f765 feat(google_maps): add scrape + reviews capability verbs 2026-07-03 17:56:43 +02:00
CREDO23
d5d673384f Merge upstream/ci_mvp (google maps scrapers) into feature-ci_phase-4-7
Resolve conflicts against the new native google-maps actor + repo-wide
ruff-format pass:
- Keep legacy webcrawler KB indexer + its test deleted (modify/delete).
- test_validators: keep WEBCRAWLER case removed (validator gone).
- test_fetch_resilience: keep platforms.youtube import path (our reorg).
- Relocate google_maps actor + tests scrapers/ -> platforms/ to match the
  reorg convention (youtube already there); rewrite imports + fixture paths.
- Add missing __init__.py across the capabilities/ test subtree so duplicate
  test basenames get unique module paths under importlib mode.

Note: google_maps fixture-backed tests error on ci_mvp too (fixtures/*.json
never committed upstream) - pre-existing, out of scope here.
2026-07-03 12:37:12 +02:00
CREDO23
817ec0e9f4 refactor(youtube): remove legacy KB ingestion; use actor get_youtube_video_id 2026-07-03 12:00:22 +02:00
CREDO23
ab6be6cbda refactor(connectors): remove legacy web-crawler KB indexer paths (keep enums) 2026-07-03 11:53:33 +02:00
CREDO23
e3ed3b2be3 refactor(subagents): remove dormant research subagent 2026-07-03 11:43:26 +02:00
CREDO23
62cb0efb44 refactor(youtube): drop duplicate POST /youtube/scrape+comments routes (use capability doors) 2026-07-03 11:38:56 +02:00
CREDO23
c66b2f0e0e refactor(automations): remove phase-6 chat watch (watch service/routes + chat_message action) 2026-07-03 11:20:22 +02:00
CREDO23
5da624399d refactor(subagents): split scraping into web_crawler + youtube builtins 2026-07-03 11:20:09 +02:00
CREDO23
3312a442f3 refactor(proprietary): move scrapers/ -> platforms/ (one subpackage per platform) 2026-07-03 11:01:58 +02:00
CREDO23
9fe9c5b71d feat(capabilities): replace web.scrape/web.discover with unified web.crawl
web.crawl scrapes a single URL (maxCrawlDepth=0) or spiders a whole site,
backed by the proprietary site_crawler engine. Rewires the scraping subagent
tools and capability tests onto the new verb.
2026-07-03 10:51:05 +02:00
CREDO23
f82fae3973 feat(web-crawler): add site_crawler spider + url_policy over the fetch engine 2026-07-03 10:50:41 +02:00
DESKTOP-RTLN3BA\$punk
7185079bd6 feat(native-connector): added google maps places & reviews scrapers 2026-07-02 21:58:24 -07:00