feat: docs and ui tweaks

surfsense_web/content/docs/connectors/native/google-maps.mdx

@@ -0,0 +1,64 @@
+---
+title: Google Maps
+description: Scrape public Google Maps places and reviews
+---
+
+The Google Maps scraper has two verbs: **scrape** for place data and **reviews** for a place's review feed.
+
+## Scrape places
+
+```bash
+POST /api/v1/workspaces/{workspace_id}/scrapers/google_maps/scrape
+```
+
+Give it search queries (optionally scoped by location), Google Maps URLs, or place IDs. Returns structured place items — name, address, category, phone, website, rating, review count, coordinates, and opening hours.
+
+At least one of `search_queries`, `urls`, or `place_ids` is required.
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `search_queries` | — | Search terms, e.g. `"coffee shops"` (max 20 sources per call, combined with URLs and place IDs) |
+| `urls` | — | Place page (`/maps/place/...`) or search-results URLs |
+| `place_ids` | — | Known Google place IDs (`ChIJ...`) |
+| `location` | — | Scope for search queries, e.g. `"New York, USA"` |
+| `max_places` | `10` | Max places per search query (max 1000) |
+| `include_details` | `false` | Also fetch each place's detail page: opening hours, popular times, extra contact info (slower) |
+| `max_reviews` | `0` | Reviews to attach per place |
+| `max_images` | `0` | Images to attach per place |
+| `language` | `en` | Result language code |
+
+```bash
+curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/google_maps/scrape" \
+  -H "Authorization: Bearer $SURFSENSE_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "search_queries": ["specialty coffee"],
+    "location": "Amsterdam, Netherlands",
+    "max_places": 20,
+    "include_details": true
+  }'
+```
+
+This verb is dual-metered: billed per returned place, plus per attached review when `max_reviews > 0`.
+
+## Fetch reviews
+
+```bash
+POST /api/v1/workspaces/{workspace_id}/scrapers/google_maps/reviews
+```
+
+Give it place URLs or place IDs; returns review items with author, text, star rating, like count, owner response, and timestamps.
+
+At least one of `urls` or `place_ids` is required.
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `urls` / `place_ids` | — | Up to 20 places per call |
+| `max_reviews` | `20` | Max reviews per place (max 100,000) |
+| `sort_by` | `newest` | `newest`, `mostRelevant`, `highestRanking`, or `lowestRanking` |
+| `start_date` | — | Only reviews on/after this ISO date |
+| `language` | `en` | Review language code |
+
+Billing is per returned review.
+
+For the full input and output JSON schemas and generated code snippets in your language, open **API Playground → Google Maps** in your workspace.

surfsense_web/content/docs/connectors/native/google-search.mdx

@@ -0,0 +1,39 @@
+---
+title: Google Search
+description: Search Google and get structured SERP results
+---
+
+The Google Search scraper runs searches and returns structured SERP data: organic results (title, URL, description), related queries, people-also-ask questions, and any AI overview.
+
+## Endpoint
+
+```bash
+POST /api/v1/workspaces/{workspace_id}/scrapers/google_search/scrape
+```
+
+## Inputs
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `queries` | required | 1–20 search terms or full Google Search URLs — terms are searched, URLs are scraped as-is |
+| `max_pages_per_query` | `1` | Result pages to fetch per query (max 10) |
+| `country_code` | — | Two-letter country to search from, e.g. `us`, `fr` |
+| `language_code` | — | Result language code (blank = Google default) |
+| `site` | — | Restrict results to a single domain, e.g. `example.com` |
+
+## Example
+
+```bash
+curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/google_search/scrape" \
+  -H "Authorization: Bearer $SURFSENSE_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "queries": ["open source notebooklm alternative"],
+    "max_pages_per_query": 2,
+    "country_code": "us"
+  }'
+```
+
+The response is `{ "items": [...] }` — one item per fetched SERP page. Billing is per SERP page.
+
+For the full input and output JSON schemas and generated code snippets in your language, open **API Playground → Google Search → Scrape** in your workspace.

surfsense_web/content/docs/connectors/native/index.mdx

@@ -0,0 +1,91 @@
+---
+title: Native Connectors
+description: SurfSense's built-in scraper APIs for Reddit, YouTube, Google Maps, Google Search, and the web
+---
+
+import { Card, Cards } from 'fumadocs-ui/components/card';
+
+Native connectors are SurfSense's own scraper APIs — built into the platform, no third-party account or OAuth app required. They pull structured, public data from the platforms below and return clean JSON.
+
+<Cards>
+  <Card
+    title="Reddit"
+    description="Posts, comments, subreddits, and users — by URL or search"
+    href="/docs/connectors/native/reddit"
+  />
+  <Card
+    title="YouTube"
+    description="Videos, channels, playlists, subtitles, and comments"
+    href="/docs/connectors/native/youtube"
+  />
+  <Card
+    title="Google Maps"
+    description="Places with details, ratings, photos, and reviews"
+    href="/docs/connectors/native/google-maps"
+  />
+  <Card
+    title="Google Search"
+    description="Structured SERPs: organic results, people-also-ask, AI overviews"
+    href="/docs/connectors/native/google-search"
+  />
+  <Card
+    title="Web Crawl"
+    description="Scrape any page or spider a whole site into clean markdown"
+    href="/docs/connectors/native/web-crawl"
+  />
+</Cards>
+
+## Three ways to use them
+
+Every scraper is available through the same three doors:
+
+1. **In chat** — the AI agent uses these scrapers as tools automatically. Ask "what is r/selfhosted saying about SurfSense?" and the agent runs the Reddit scraper for you.
+2. **API Playground** — open **API Playground** in your workspace sidebar, pick a scraper, fill in the form, and run it interactively. Great for exploring what a scraper returns before writing code.
+3. **REST API** — call the scrapers from your own code. Each one is a single `POST`:
+
+```bash
+POST /api/v1/workspaces/{workspace_id}/scrapers/{platform}/{verb}
+Authorization: Bearer <your-api-key>
+```
+
+The playground's **API reference** section on every scraper page generates ready-to-paste snippets (cURL, Python, JavaScript, Go, and more) with your workspace ID already filled in, plus the full input and output JSON schemas.
+
+## API keys
+
+To call the REST API you need two things, both under **API Playground → API Keys**:
+
+1. Toggle **API key access** on for the workspace.
+2. Create a personal API key and send it as an `Authorization: Bearer` header.
+
+## Sync and async runs
+
+By default a `POST` blocks until the scrape finishes and returns the results. For long scrapes, append `?mode=async` — you get a `202` with a `run_id` immediately, then:
+
+- **Stream progress**: `GET .../scrapers/runs/{run_id}/events` (Server-Sent Events, ends with a `run.finished` event)
+- **Fetch the result**: `GET .../scrapers/runs/{run_id}`
+- **Cancel**: `POST .../scrapers/runs/{run_id}/cancel`
+
+## Avoiding blocks with a proxy (self-hosted)
+
+Scrapers make real requests to the target platforms, and heavy use from a single server IP will eventually get rate-limited or blocked. If you self-host, we recommend routing scraper traffic through a proxy — any HTTP proxy or rotating residential/datacenter gateway you already use works; SurfSense doesn't require a specific vendor.
+
+Set it up in your backend `.env` and restart:
+
+```bash
+# A single endpoint (user:pass@host:port), used for all scraper traffic
+PROXY_URL=http://username:password@proxy.example.com:8080
+
+# Or a comma-separated pool that SurfSense rotates through per request.
+# Gateways that rotate server-side just need the single PROXY_URL above.
+PROXY_URLS=http://user:pass@host1:port,http://user:pass@host2:port
+```
+
+Leave both unset and requests go out directly from your server's IP — fine for light use. See the proxy section of `.env.example` for the full details, including `PROXY_PROVIDER` if your vendor has a built-in integration.
+
+<Callout type="info">
+On SurfSense Cloud, proxying is already handled — nothing to configure.
+</Callout>
+
+## Runs and pricing
+
+Every run — from chat, the playground, or the API — is recorded under **API Playground → Runs** with its input, output, duration, and cost. Scrapers are metered per item returned (per post, video, comment, place, review, SERP page, or crawled page); the current rate is shown on each scraper's playground card, and pricing is returned by the capabilities API (`GET .../scrapers/capabilities`).

surfsense_web/content/docs/connectors/native/meta.json

@@ -0,0 +1,5 @@
+{
+	"title": "Native Connectors",
+	"pages": ["reddit", "youtube", "google-maps", "google-search", "web-crawl"],
+	"defaultOpen": false
+}

surfsense_web/content/docs/connectors/native/reddit.mdx

@@ -0,0 +1,49 @@
+---
+title: Reddit
+description: Scrape public Reddit posts, comments, subreddits, and users
+---
+
+The Reddit scraper pulls structured public data from Reddit. Give it URLs (a post, a subreddit, a user page, or a search URL) and/or search terms, and it returns posts (title, body, score, comment count, subreddit, author), their comment trees, and community/user metadata.
+
+## Endpoint
+
+```bash
+POST /api/v1/workspaces/{workspace_id}/scrapers/reddit/scrape
+```
+
+## Inputs
+
+At least one of `urls`, `search_queries`, or `community` is required.
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `urls` | — | Reddit URLs: a post, `/r/<subreddit>`, `/user/<name>`, or a search URL (max 20 sources per call, combined with queries) |
+| `search_queries` | — | Search terms to run on Reddit |
+| `community` | — | Subreddit name (without `r/`) to scope searches to; with no queries, its listing is scraped |
+| `sort` | `new` | `relevance`, `hot`, `top`, `new`, `rising`, or `comments` |
+| `time_filter` | — | Window for `top`/`controversial`: `hour`, `day`, `week`, `month`, `year`, `all` |
+| `max_items` | `10` | Max total items returned across all sources (hard cap 100) |
+| `max_posts` | `10` | Max posts per subreddit/user/search target |
+| `max_comments` | `10` | Max comments per post (`0` = none) |
+| `skip_comments` | `false` | Skip comment trees entirely (faster) |
+| `include_nsfw` | `true` | Include over-18 posts |
+| `post_date_limit` / `comment_date_limit` | — | ISO dates for incremental scrapes: only return newer content |
+
+## Example
+
+```bash
+curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/reddit/scrape" \
+  -H "Authorization: Bearer $SURFSENSE_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "search_queries": ["self-hosted AI research agent"],
+    "community": "selfhosted",
+    "sort": "top",
+    "time_filter": "month",
+    "max_items": 25
+  }'
+```
+
+The response is `{ "items": [...] }` — one item per post, comment, community, or user. Billing is per returned item.
+
+For the full input and output JSON schemas and generated code snippets in your language, open **API Playground → Reddit → Scrape** in your workspace.

surfsense_web/content/docs/connectors/native/web-crawl.mdx

@@ -0,0 +1,58 @@
+---
+title: Web Crawl
+description: Scrape any page or crawl a whole site into clean markdown
+---
+
+The web crawler fetches one or more pages — or spiders a whole site — and returns each page as clean markdown, plus metadata, every link with its anchor text, and harvested contact signals (emails, phone numbers, social profiles). JS-rendered pages are loaded in a real browser and auto-scrolled, so lazy-loaded listings are captured too.
+
+## Endpoint
+
+```bash
+POST /api/v1/workspaces/{workspace_id}/scrapers/web/crawl
+```
+
+## Inputs
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `startUrls` | required | 1–20 seed URLs |
+| `maxCrawlDepth` | `0` | Link-hops to follow from each seed. `0` = fetch only the seeds; `1` = also the pages they link to; up to 5. The spider stays on the seed's site |
+| `maxCrawlPages` | `10` | Total pages fetched per call, seeds included (max 200) |
+| `maxLength` | `50000` | Max characters of markdown kept per page |
+| `includeUrlPatterns` | — | Regexes a discovered link must match to be followed (empty = follow every same-site link) |
+| `excludeUrlPatterns` | — | Regexes that exclude links from being followed (wins over include) |
+
+## Example
+
+Scrape a single page:
+
+```bash
+curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/web/crawl" \
+  -H "Authorization: Bearer $SURFSENSE_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{ "startUrls": ["https://example.com/pricing"] }'
+```
+
+Crawl a site's blog, two hops deep:
+
+```bash
+curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/web/crawl" \
+  -H "Authorization: Bearer $SURFSENSE_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "startUrls": ["https://example.com/blog"],
+    "maxCrawlDepth": 2,
+    "maxCrawlPages": 50,
+    "includeUrlPatterns": ["/blog/"]
+  }'
+```
+
+## Output
+
+One item per fetched page (in crawl order) with its markdown, metadata (title, description), crawl provenance (depth, referrer), links classified as internal/external/social/email/tel, and page-level contacts. The response also includes a site-wide `contacts` summary that deduplicates every email, phone, and social profile found — `siteWide: true` marks header/footer values (the company's own contacts) versus page-local finds like individual team members.
+
+Contact details often live on about/contact/privacy pages, so crawl with `maxCrawlDepth >= 1` to surface them — useful for lead generation and competitive intelligence.
+
+Billing is per successfully fetched page.
+
+For the full input and output JSON schemas and generated code snippets in your language, open **API Playground → Web → Crawl** in your workspace.

surfsense_web/content/docs/connectors/native/youtube.mdx

@@ -0,0 +1,54 @@
+---
+title: YouTube
+description: Scrape public YouTube videos, channels, playlists, and comments
+---
+
+The YouTube scraper has two verbs: **scrape** for video metadata (and optionally subtitles) and **comments** for a video's comment threads.
+
+## Scrape videos
+
+```bash
+POST /api/v1/workspaces/{workspace_id}/scrapers/youtube/scrape
+```
+
+Give it YouTube URLs (video, channel `/@handle`, playlist, shorts, or hashtag pages) and/or search queries. Returns structured video items — title, views, likes, publish date, channel info, description, and optionally subtitles.
+
+At least one of `urls` or `search_queries` is required.
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `urls` | — | Video, channel, playlist, shorts, or hashtag URLs (max 20 sources per call, combined with queries) |
+| `search_queries` | — | Search terms; each returns up to `max_results` videos |
+| `max_results` | `10` | Max items per source and per content type (a channel's videos, shorts, and streams are capped independently; max 1000) |
+| `download_subtitles` | `false` | Also fetch each video's subtitle track (slower) |
+| `subtitles_language` | `en` | Subtitle language code |
+
+```bash
+curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/youtube/scrape" \
+  -H "Authorization: Bearer $SURFSENSE_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "urls": ["https://www.youtube.com/@surfsense"],
+    "max_results": 20
+  }'
+```
+
+Billing is per returned video/short/stream.
+
+## Fetch comments
+
+```bash
+POST /api/v1/workspaces/{workspace_id}/scrapers/youtube/comments
+```
+
+Give it video URLs; returns comment items with author, text, like count, reply relationships, and timestamps — useful for gauging sentiment on specific videos.
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `urls` | required | 1–20 video URLs |
+| `max_comments` | `20` | Max items per video, counting top-level comments and replies (max 100,000) |
+| `sort_by` | `NEWEST_FIRST` | `TOP_COMMENTS` or `NEWEST_FIRST` |
+
+Billing is per returned comment or reply.
+
+For the full input and output JSON schemas and generated code snippets in your language, open **API Playground → YouTube** in your workspace.