mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-16 23:01:06 +02:00
Merge remote-tracking branch 'upstream/dev' into feat/instagram-scraper
This commit is contained in:
Anish Sarkar
commit
e38ca19b18
119 changed files with 5948 additions and 36 deletions
|
|
@ -12,7 +12,7 @@ Connectors bring data into SurfSense — either as searchable knowledge or as li
|
|||
<Card
|
||||
icon={<Zap />}
|
||||
title="Native Connectors"
|
||||
description="SurfSense's built-in scraper APIs: Reddit, YouTube, Instagram, Google Maps, Google Search, and Web Crawl — usable in chat, the API Playground, or via REST"
|
||||
description="SurfSense's built-in scraper APIs: Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search, and Web Crawl — usable in chat, the API Playground, or via REST"
|
||||
href="/docs/connectors/native"
|
||||
/>
|
||||
<Card
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
title: Native Connectors
|
||||
description: SurfSense's built-in scraper APIs for Reddit, YouTube, Instagram, Google Maps, Google Search, and the web
|
||||
description: SurfSense's built-in scraper APIs for Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search, and the web
|
||||
---
|
||||
|
||||
import { Card, Cards } from 'fumadocs-ui/components/card';
|
||||
|
|
@ -23,6 +23,11 @@ Native connectors are SurfSense's own scraper APIs — built into the platform,
|
|||
description="Posts, reels, comments, and profile, hashtag, and place details"
|
||||
href="/docs/connectors/native/instagram"
|
||||
/>
|
||||
<Card
|
||||
title="TikTok"
|
||||
description="Videos by URL, profile, hashtag, or search"
|
||||
href="/docs/connectors/native/tiktok"
|
||||
/>
|
||||
<Card
|
||||
title="Google Maps"
|
||||
description="Places with details, ratings, photos, and reviews"
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
{
|
||||
"title": "Native Connectors",
|
||||
"pages": ["reddit", "youtube", "instagram", "google-maps", "google-search", "web-crawl"],
|
||||
"pages": ["reddit", "youtube", "instagram", "tiktok", "google-maps", "google-search", "web-crawl"],
|
||||
"defaultOpen": false
|
||||
}
|
||||
|
|
|
|||
97
surfsense_web/content/docs/connectors/native/tiktok.mdx
Normal file
97
surfsense_web/content/docs/connectors/native/tiktok.mdx
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
---
|
||||
title: TikTok
|
||||
description: Scrape public TikTok videos, comments, accounts, and trending feeds
|
||||
---
|
||||
|
||||
The TikTok connector pulls structured public data from TikTok across four verbs: **scrape** (videos), **comments**, **user search** (accounts), and **trending**. Every verb returns `{ "items": [...] }` and is billed per returned item; surfaced errors (an `errorCode` field, e.g. a withheld feed) are never charged.
|
||||
|
||||
## Scrape videos
|
||||
|
||||
```bash
|
||||
POST /api/v1/workspaces/{workspace_id}/scrapers/tiktok/scrape
|
||||
```
|
||||
|
||||
Give it URLs (a video, a profile, a hashtag, or a search page) and/or profiles, hashtags, or search terms; returns videos (caption, author, play/like/comment/share counts, music, hashtags, timestamps, and the web URL). At least one of `urls`, `profiles`, `hashtags`, or `search_queries` is required.
|
||||
|
||||
| Field | Default | Description |
|
||||
|-------|---------|-------------|
|
||||
| `urls` | — | TikTok URLs: a video, a profile (`/@<user>`), a hashtag (`/tag/<name>`), or a search URL (max 20 sources per call) |
|
||||
| `profiles` | — | Profile usernames, with or without a leading `@` |
|
||||
| `hashtags` | — | Hashtag names, without the `#` |
|
||||
| `search_queries` | — | Search terms to run on TikTok |
|
||||
| `results_per_page` | `10` | Max videos per profile/hashtag/search target |
|
||||
| `max_items` | `10` | Max total videos returned across all sources (hard cap 100) |
|
||||
|
||||
```bash
|
||||
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/tiktok/scrape" \
|
||||
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{ "hashtags": ["food"], "max_items": 25 }'
|
||||
```
|
||||
|
||||
<Callout type="info">
|
||||
Video and hashtag targets are the reliable video paths. A `profiles` target returns the account's **metadata** (name, followers, bio, verification) reliably, but TikTok often withholds its **video list** from automated clients — so a profile can return metadata with no videos. Keyword **video** search is login-walled and returns a surfaced error; to find accounts by keyword use **user search** below.
|
||||
</Callout>
|
||||
|
||||
## Comments
|
||||
|
||||
```bash
|
||||
POST /api/v1/workspaces/{workspace_id}/scrapers/tiktok/comments
|
||||
```
|
||||
|
||||
Given TikTok video URLs, returns each video's public comment thread: comment text, author, like count, and reply count (replies carry `repliesToId`, the parent comment id).
|
||||
|
||||
| Field | Default | Description |
|
||||
|-------|---------|-------------|
|
||||
| `video_urls` | — | TikTok video URLs (`/@<user>/video/<id>`), max 20 per call |
|
||||
| `comments_per_video` | `20` | Max comments per video |
|
||||
| `max_items` | `20` | Max total comments returned (hard cap 100) |
|
||||
|
||||
```bash
|
||||
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/tiktok/comments" \
|
||||
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{ "video_urls": ["https://www.tiktok.com/@nasa/video/123"], "max_items": 50 }'
|
||||
```
|
||||
|
||||
## User search
|
||||
|
||||
```bash
|
||||
POST /api/v1/workspaces/{workspace_id}/scrapers/tiktok/user_search
|
||||
```
|
||||
|
||||
Finds public TikTok accounts by keyword — the reliable account-discovery path. Returns matching profiles with name, followers, bio, and verification.
|
||||
|
||||
| Field | Default | Description |
|
||||
|-------|---------|-------------|
|
||||
| `queries` | — | Keywords to find accounts by, e.g. `["nasa", "cooking"]` (max 20) |
|
||||
| `results_per_query` | `10` | Max accounts per query |
|
||||
| `max_items` | `10` | Max total accounts returned (hard cap 100) |
|
||||
|
||||
```bash
|
||||
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/tiktok/user_search" \
|
||||
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{ "queries": ["space agency"], "max_items": 20 }'
|
||||
```
|
||||
|
||||
## Trending
|
||||
|
||||
```bash
|
||||
POST /api/v1/workspaces/{workspace_id}/scrapers/tiktok/trending
|
||||
```
|
||||
|
||||
Returns the current trending videos from TikTok's Explore feed — no input needed beyond how many to return. Items use the same video shape as **scrape** and bill on the same per-video meter.
|
||||
|
||||
| Field | Default | Description |
|
||||
|-------|---------|-------------|
|
||||
| `max_items` | `20` | Max trending videos returned (hard cap 100) |
|
||||
|
||||
```bash
|
||||
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/tiktok/trending" \
|
||||
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{ "max_items": 30 }'
|
||||
```
|
||||
|
||||
For the full input and output JSON schemas and generated code snippets in your language, open **API Playground → TikTok** in your workspace.
|
||||
|
|
@ -7,7 +7,7 @@ import { Tab, Tabs } from 'fumadocs-ui/components/tabs';
|
|||
|
||||
# SurfSense MCP Server
|
||||
|
||||
The SurfSense MCP server exposes your workspace to any [Model Context Protocol](https://modelcontextprotocol.io/) client. Your agent gets 21 native, typed tools: every scraper (Reddit, YouTube, Instagram, Google Maps, Google Search, web crawl), full knowledge-base access (search, read, add, upload, update, delete), and a workspace selector.
|
||||
The SurfSense MCP server exposes your workspace to any [Model Context Protocol](https://modelcontextprotocol.io/) client. Your agent gets 21 native, typed tools: every scraper (Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search, web crawl), full knowledge-base access (search, read, add, upload, update, delete), and a workspace selector.
|
||||
|
||||
Connect it two ways: the **hosted** server at `https://mcp.surfsense.com/mcp` (nothing to install — just an API key), or run it yourself over **stdio** against any SurfSense backend, cloud or self-hosted.
|
||||
|
||||
|
|
@ -264,7 +264,7 @@ For self-host (stdio), all settings are environment variables passed by the clie
|
|||
| Group | Tools |
|
||||
|-------|-------|
|
||||
| Workspaces | `surfsense_list_workspaces`, `surfsense_select_workspace` |
|
||||
| Scrapers | `surfsense_reddit_scrape`, `surfsense_youtube_scrape`, `surfsense_youtube_comments`, `surfsense_instagram_scrape`, `surfsense_instagram_details`, `surfsense_google_maps_scrape`, `surfsense_google_maps_reviews`, `surfsense_google_search`, `surfsense_web_crawl`, `surfsense_list_scraper_runs`, `surfsense_get_scraper_run` |
|
||||
| Scrapers | `surfsense_reddit_scrape`, `surfsense_youtube_scrape`, `surfsense_youtube_comments`, `surfsense_instagram_scrape`, `surfsense_instagram_details`, `surfsense_tiktok_scrape`, `surfsense_google_maps_scrape`, `surfsense_google_maps_reviews`, `surfsense_google_search`, `surfsense_web_crawl`, `surfsense_list_scraper_runs`, `surfsense_get_scraper_run` |
|
||||
| Knowledge base | `surfsense_search_knowledge_base`, `surfsense_list_documents`, `surfsense_get_document`, `surfsense_add_document`, `surfsense_upload_file`, `surfsense_update_document`, `surfsense_delete_document` |
|
||||
|
||||
Usage is billed exactly like the REST API — scraper tools are metered per returned item, and every call is recorded under **API Playground → Runs**.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue