feat: update environment variables and enhance scraping capabilities

- Adjusted Google Maps and YouTube micro pricing in the .env.example file for better cost management.
- Introduced new environment variables for captcha solving and stealth browser hardening to improve scraping resilience.
- Removed outdated smoke test for scraper API endpoints to streamline testing.
- Enhanced anonymous chat agent's system prompt to clarify capabilities and suggest account creation for advanced features.
- Updated Reddit fetch logic to prioritize new session handling and improve resilience against IP-related issues.
- Added compacting functionality for scraper results to optimize data handling and presentation.
- Improved workspace and document management tools with clearer descriptions and enhanced functionality.
- Introduced new UI components for agent setup guidance in the web application.
This commit is contained in:
DESKTOP-RTLN3BA\$punk 2026-07-06 20:27:36 -07:00
parent 271a21aee6
commit 1fd58752a3
24 changed files with 1326 additions and 320 deletions

View file

@ -35,13 +35,14 @@ Native connectors are SurfSense's own scraper APIs — built into the platform,
/>
</Cards>
## Three ways to use them
## Four ways to use them
Every scraper is available through the same three doors:
Every scraper is available through the same four doors:
1. **In chat** — the AI agent uses these scrapers as tools automatically. Ask "what is r/selfhosted saying about SurfSense?" and the agent runs the Reddit scraper for you.
2. **API Playground** — open **API Playground** in your workspace sidebar, pick a scraper, fill in the form, and run it interactively. Great for exploring what a scraper returns before writing code.
3. **REST API** — call the scrapers from your own code. Each one is a single `POST`:
3. **MCP server** — hand every scraper to Claude Code, Codex, OpenCode, Cursor, or any MCP client as native tools. See the [MCP server guide](/docs/how-to/mcp-server).
4. **REST API** — call the scrapers from your own code. Each one is a single `POST`:
```bash
POST /api/v1/workspaces/{workspace_id}/scrapers/{platform}/{verb}