Commit graph

929 commits

Author SHA1 Message Date
CREDO23
dd5d267056 docs: instruct agents to cite scraper runs 2026-07-22 19:34:56 +02:00
CREDO23
95b4afddeb feat: declare citation channel for subagents 2026-07-22 19:34:56 +02:00
CREDO23
81f2d694bd feat: map run citation to run_<uuid> payload 2026-07-22 19:34:56 +02:00
CREDO23
609adea006 feat: add RUN citation source type 2026-07-22 19:34:56 +02:00
CREDO23
ca317c4686 Merge remote-tracking branch 'upstream/dev' into feature-walmart-scraper
# Conflicts:
#	docker/.env.example
#	surfsense_backend/app/capabilities/core/billing.py
#	surfsense_backend/app/capabilities/core/types.py
#	surfsense_backend/app/config/__init__.py
#	surfsense_mcp/mcp_server/features/scrapers/__init__.py
#	surfsense_web/content/docs/connectors/index.mdx
#	surfsense_web/content/docs/connectors/native/index.mdx
#	surfsense_web/content/docs/how-to/mcp-server.mdx
2026-07-21 02:42:41 +02:00
Rohan Verma
2eed9b2ac8
Merge pull request #1613 from Sanjays2402/fix/nonblocking-audio-write
fix: avoid blocking audio writes
2026-07-19 16:17:05 -07:00
CREDO23
56e8fa13e6 feat(walmart): route product-review questions to marketplace specialists 2026-07-19 08:45:27 +02:00
CREDO23
cabc5b5c86 feat(walmart): register subagent in roster and connector map 2026-07-19 08:45:27 +02:00
CREDO23
f6a8a2a566 feat(walmart): add chat subagent with scrape and reviews tools 2026-07-19 08:45:27 +02:00
Sanjay Santhanam
5cc05d90f0 fix: avoid blocking audio writes
TTS responses were written synchronously from the async presentation pipeline, blocking concurrent requests. Offload byte writes to a worker thread and add a regression test that keeps the event loop responsive.
2026-07-18 17:46:31 -07:00
CREDO23
91aa265afb Merge remote-tracking branch 'upstream/dev' into feature-indeed-jobs-scraper
# Conflicts:
#	README.es.md
#	README.hi.md
#	README.md
#	README.pt-BR.md
#	README.zh-CN.md
#	surfsense_backend/tests/unit/capabilities/google_maps/test_registry.py
#	surfsense_backend/tests/unit/capabilities/reddit/test_registry.py
#	surfsense_backend/tests/unit/capabilities/youtube/test_registry.py
#	surfsense_mcp/mcp_server/features/scrapers/__init__.py
#	surfsense_web/content/docs/connectors/index.mdx
#	surfsense_web/content/docs/connectors/native/index.mdx
#	surfsense_web/content/docs/connectors/native/meta.json
#	surfsense_web/content/docs/how-to/mcp-server.mdx
#	surfsense_web/lib/connectors-marketing/index.ts
#	surfsense_web/lib/playground/catalog.ts
2026-07-18 22:11:30 +02:00
DESKTOP-RTLN3BA\$punk
bd7647b314 refactor: update README files and system prompts to include Amazon as a data source for AI agents, enhancing the platform's capabilities for live web research 2026-07-17 16:44:11 -07:00
DESKTOP-RTLN3BA\$punk
7bc02ce55b refactor: update references from competitive intelligence to open web research across various components and documentation 2026-07-16 19:41:16 -07:00
DESKTOP-RTLN3BA\$punk
1917d76e10 refactor: update Google search system prompt and add Amazon route prefix 2026-07-16 17:39:08 -07:00
Rohan Verma
d1d839a491
Merge pull request #1604 from AnishSarkar22/feat/amazon-scraper
feat(amazon): add public multi-marketplace product scraping capability
2026-07-15 16:29:03 -07:00
CREDO23
c485fe4c43 fix: scrape indeed first page only, steer subagent to one query 2026-07-15 20:54:37 +02:00
CREDO23
3a9b0405c3 feat: add indeed to required-connector map 2026-07-15 18:46:55 +02:00
CREDO23
238d2767c5 feat: register indeed subagent in registry 2026-07-15 18:46:55 +02:00
CREDO23
e193867e77 feat: indeed subagent builder 2026-07-15 18:46:55 +02:00
CREDO23
e0cc816f67 feat: indeed subagent system prompt 2026-07-15 18:46:55 +02:00
CREDO23
723eb8c6cb feat: indeed subagent routing description 2026-07-15 18:46:55 +02:00
CREDO23
ada83efedf feat: wire indeed.scrape into subagent tools 2026-07-15 18:46:55 +02:00
CREDO23
7cbd8345f9 feat: indeed subagent tools package 2026-07-15 18:46:55 +02:00
CREDO23
2ade6bb903 feat: indeed subagent package 2026-07-15 18:46:55 +02:00
Anish Sarkar
69848b4c71 feat(amazon): register amazon subagent 2026-07-15 10:24:45 +05:30
Anish Sarkar
6888f0b8bb feat(amazon): add subagent builder 2026-07-15 10:24:40 +05:30
Anish Sarkar
5a34435160 docs(amazon): add subagent description and system prompt 2026-07-15 10:24:34 +05:30
Anish Sarkar
03cd99194b feat(amazon): add subagent capability tools 2026-07-15 10:24:22 +05:30
Anish Sarkar
2d837ea18c Merge remote-tracking branch 'upstream/dev' into fix/onboarding 2026-07-14 10:18:03 +05:30
DESKTOP-RTLN3BA\$punk
2b018c4474 refactor: streamline TikTok and Instagram scraping logic by removing search_queries and enhancing documentation for clarity 2026-07-13 17:11:25 -07:00
DESKTOP-RTLN3BA\$punk
1131da5ed7 feat: bumped version to 0.0.32 2026-07-13 16:29:39 -07:00
Anish Sarkar
91531286b2 fix: update terminology from "language model" to "chat model" across various components 2026-07-14 01:09:00 +05:30
Anish Sarkar
e38ca19b18 Merge remote-tracking branch 'upstream/dev' into feat/instagram-scraper 2026-07-11 04:31:11 +05:30
Anish Sarkar
c001f4b16e feat(subagent): update Instagram subagent system prompt 2026-07-11 03:25:25 +05:30
Anish Sarkar
9e9dd8a124 feat(subagent): update Instagram subagent prompts and tools 2026-07-11 02:21:02 +05:30
CREDO23
1911b78379 fix(tiktok-agent): drop keyword search from the routing description's video sources 2026-07-10 17:40:04 +02:00
CREDO23
cd82f861de fix(tiktok-agent): stop steering the subagent to keyword search for videos
The playbook listed search_queries as a video source in three places, then contradicted itself. Replace with action-oriented guidance: videos via hashtags/URLs, accounts via user_search. Drop 'login-walled' (the agent has no login path).
2026-07-10 17:40:04 +02:00
CREDO23
6b15d610d9 feat(tiktok): surface comments, user_search, trending across all product surfaces
Brings the three newer verbs to parity with tiktok.scrape everywhere it was
wired: MCP tools (+ selfcheck manifest), the chat subagent prompt/description,
the playground catalog, the native docs page, and the SEO marketing page. Also
adds live e2e stages for comments, user search, and trending. Docs/FAQ now
state the real contract: profile metadata is reliable while its video list can
be withheld, and keyword video search is walled (use user search for accounts).
2026-07-09 19:09:00 +02:00
CREDO23
67b5472b9f feat(tiktok): add tiktok.trending verb for the Explore feed
The Explore feed (/api/explore/item_list) is a global trending-video feed
served to anonymous sessions, and it returns the same itemStruct shape as the
other listings — so the verb reuses parse_video, the listing flow, the
TikTokVideoItem output, and the per-video billing meter wholesale. Adds a
browser-capture marker + fetch_trending, a synthetic-target orchestrator entry,
and the tiktok.trending capability, surfaced on the chat subagent.
2026-07-09 18:52:56 +02:00
CREDO23
7723b5b8b6 feat(tiktok): add tiktok.comments verb for public comment scraping
Comments load over a signed /api/comment/list XHR that TikTok serves to
anonymous sessions once the comments panel is opened (unlike profile-video and
general-search feeds), so this is a reliable verb. Given video URLs it returns
CommentItems (text, author, likes, reply counts; replies carry repliesToId),
deduped per video, capped, and degraded to an ErrorItem for empty/withheld
videos or a bad_url ErrorItem for non-video inputs.

Generalizes the browser capture over a pluggable interaction step so the
comments flow (open panel, scroll the panel to paginate) reuses the same
warm+capture scaffolding as listing/user-search. Billed per comment on a new
TIKTOK_COMMENT meter (TIKTOK_MICROS_PER_COMMENT, matching the per-comment
market), surfaced on the chat subagent alongside tiktok.scrape/user_search.
2026-07-09 18:35:26 +02:00
CREDO23
192b6dc31a feat(tiktok): add tiktok.user_search verb for account discovery
Video/general search is login-walled for anonymous sessions, but the Users
tab (/api/search/user) returns public account records without a redirect, so
this exposes the one reliably-unblocked search path. A keyword yields
TikTokProfileItems (name, followers, bio, verification), deduped per query,
capped, and degraded to an ErrorItem when a query is empty/withheld.

Reuses the browser capture (generalized over XHR markers + extractor) and the
shared profile item shape. Billed per account on a new TIKTOK_USER meter
(TIKTOK_MICROS_PER_USER), surfaced on the chat subagent alongside tiktok.scrape.
2026-07-09 18:00:40 +02:00
Anish Sarkar
98eab8c4b5 feat(chat): add Instagram builtin subagent 2026-07-09 16:02:05 +05:30
Anish Sarkar
8486a4f914 feat(constants): add Instagram to SUBAGENT_TO_REQUIRED_CONNECTOR_MAP for enhanced multi-agent chat capabilities 2026-07-09 14:08:23 +05:30
CREDO23
5f9472b122 feat(tiktok): advertise tiktok across agent prompts, MCP instructions, docs 2026-07-08 20:30:38 +02:00
CREDO23
ed1c3a1f3d feat(tiktok): expose tiktok.scrape on MCP, chat subagent, playground 2026-07-08 20:20:20 +02:00
DESKTOP-RTLN3BA\$punk
b477d33cba Merge commit 'fa7075fde6' into dev 2026-07-07 00:39:58 -07:00
DESKTOP-RTLN3BA\$punk
1c9ab207ef chore: bumped version to 0.0.31 2026-07-06 21:43:15 -07:00
DESKTOP-RTLN3BA\$punk
1fd58752a3 feat: update environment variables and enhance scraping capabilities
- Adjusted Google Maps and YouTube micro pricing in the .env.example file for better cost management.
- Introduced new environment variables for captcha solving and stealth browser hardening to improve scraping resilience.
- Removed outdated smoke test for scraper API endpoints to streamline testing.
- Enhanced anonymous chat agent's system prompt to clarify capabilities and suggest account creation for advanced features.
- Updated Reddit fetch logic to prioritize new session handling and improve resilience against IP-related issues.
- Added compacting functionality for scraper results to optimize data handling and presentation.
- Improved workspace and document management tools with clearer descriptions and enhanced functionality.
- Introduced new UI components for agent setup guidance in the web application.
2026-07-06 20:27:36 -07:00
DESKTOP-RTLN3BA\$punk
271a21aee6 feat: docs and ui tweaks 2026-07-06 19:26:35 -07:00
DESKTOP-RTLN3BA\$punk
50f2d095aa feat: api playground, other tweaks 2026-07-06 16:45:04 -07:00