Commit graph

19 commits

Author SHA1 Message Date
CREDO23
ca317c4686 Merge remote-tracking branch 'upstream/dev' into feature-walmart-scraper
# Conflicts:
#	docker/.env.example
#	surfsense_backend/app/capabilities/core/billing.py
#	surfsense_backend/app/capabilities/core/types.py
#	surfsense_backend/app/config/__init__.py
#	surfsense_mcp/mcp_server/features/scrapers/__init__.py
#	surfsense_web/content/docs/connectors/index.mdx
#	surfsense_web/content/docs/connectors/native/index.mdx
#	surfsense_web/content/docs/how-to/mcp-server.mdx
2026-07-21 02:42:41 +02:00
CREDO23
6c98bd33a8 docs(walmart): add connector docs and list in indexes 2026-07-19 09:01:11 +02:00
CREDO23
91aa265afb Merge remote-tracking branch 'upstream/dev' into feature-indeed-jobs-scraper
# Conflicts:
#	README.es.md
#	README.hi.md
#	README.md
#	README.pt-BR.md
#	README.zh-CN.md
#	surfsense_backend/tests/unit/capabilities/google_maps/test_registry.py
#	surfsense_backend/tests/unit/capabilities/reddit/test_registry.py
#	surfsense_backend/tests/unit/capabilities/youtube/test_registry.py
#	surfsense_mcp/mcp_server/features/scrapers/__init__.py
#	surfsense_web/content/docs/connectors/index.mdx
#	surfsense_web/content/docs/connectors/native/index.mdx
#	surfsense_web/content/docs/connectors/native/meta.json
#	surfsense_web/content/docs/how-to/mcp-server.mdx
#	surfsense_web/lib/connectors-marketing/index.ts
#	surfsense_web/lib/playground/catalog.ts
2026-07-18 22:11:30 +02:00
DESKTOP-RTLN3BA\$punk
bd7647b314 refactor: update README files and system prompts to include Amazon as a data source for AI agents, enhancing the platform's capabilities for live web research 2026-07-17 16:44:11 -07:00
DESKTOP-RTLN3BA\$punk
7bc02ce55b refactor: update references from competitive intelligence to open web research across various components and documentation 2026-07-16 19:41:16 -07:00
CREDO23
5b0982f80b docs: add indeed native connector page and listings 2026-07-15 18:57:15 +02:00
Anish Sarkar
92055846d1 docs(web): document amazon eu marketplace support 2026-07-15 15:32:58 +05:30
Anish Sarkar
441f671ebc feat(amazon): add documentation for Amazon scraper API endpoint and input parameters 2026-07-15 10:14:15 +05:30
DESKTOP-RTLN3BA\$punk
2b018c4474 refactor: streamline TikTok and Instagram scraping logic by removing search_queries and enhancing documentation for clarity 2026-07-13 17:11:25 -07:00
DESKTOP-RTLN3BA\$punk
e8b3692b54 chore: linting 2026-07-13 16:30:37 -07:00
Anish Sarkar
e38ca19b18 Merge remote-tracking branch 'upstream/dev' into feat/instagram-scraper 2026-07-11 04:31:11 +05:30
Anish Sarkar
42f9f35ec7 docs(web): document new media fields and search_type aliasing 2026-07-11 03:44:45 +05:30
Anish Sarkar
b75ab81261 docs(web): refresh Instagram connector docs 2026-07-11 03:25:30 +05:30
Anish Sarkar
69a8433910 docs(web): refresh Instagram connector and MCP docs 2026-07-11 02:21:05 +05:30
CREDO23
6b15d610d9 feat(tiktok): surface comments, user_search, trending across all product surfaces
Brings the three newer verbs to parity with tiktok.scrape everywhere it was
wired: MCP tools (+ selfcheck manifest), the chat subagent prompt/description,
the playground catalog, the native docs page, and the SEO marketing page. Also
adds live e2e stages for comments, user search, and trending. Docs/FAQ now
state the real contract: profile metadata is reliable while its video list can
be withheld, and keyword video search is walled (use user search for accounts).
2026-07-09 19:09:00 +02:00
Anish Sarkar
f5aad487ac docs(web): document Instagram native connector 2026-07-09 16:07:42 +05:30
CREDO23
ea6befdf59 docs(tiktok): native connector page + nav card 2026-07-08 20:21:29 +02:00
DESKTOP-RTLN3BA\$punk
1fd58752a3 feat: update environment variables and enhance scraping capabilities
- Adjusted Google Maps and YouTube micro pricing in the .env.example file for better cost management.
- Introduced new environment variables for captcha solving and stealth browser hardening to improve scraping resilience.
- Removed outdated smoke test for scraper API endpoints to streamline testing.
- Enhanced anonymous chat agent's system prompt to clarify capabilities and suggest account creation for advanced features.
- Updated Reddit fetch logic to prioritize new session handling and improve resilience against IP-related issues.
- Added compacting functionality for scraper results to optimize data handling and presentation.
- Improved workspace and document management tools with clearer descriptions and enhanced functionality.
- Introduced new UI components for agent setup guidance in the web application.
2026-07-06 20:27:36 -07:00
DESKTOP-RTLN3BA\$punk
271a21aee6 feat: docs and ui tweaks 2026-07-06 19:26:35 -07:00