mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-16 23:01:06 +02:00
feat(tiktok): surface comments, user_search, trending across all product surfaces
Brings the three newer verbs to parity with tiktok.scrape everywhere it was wired: MCP tools (+ selfcheck manifest), the chat subagent prompt/description, the playground catalog, the native docs page, and the SEO marketing page. Also adds live e2e stages for comments, user search, and trending. Docs/FAQ now state the real contract: profile metadata is reliable while its video list can be withheld, and keyword video search is walled (use user search for accounts).
This commit is contained in:
CREDO23
parent
67b5472b9f
commit
6b15d610d9
8 changed files with 294 additions and 33 deletions
|
|
@ -1,19 +1,17 @@
|
|||
---
|
||||
title: TikTok
|
||||
description: Scrape public TikTok videos by URL, profile, hashtag, or search
|
||||
description: Scrape public TikTok videos, comments, accounts, and trending feeds
|
||||
---
|
||||
|
||||
The TikTok scraper pulls structured public data from TikTok. Give it URLs (a video, a profile, a hashtag, or a search page) and/or profiles, hashtags, or search terms, and it returns videos (caption, author, play/like/comment/share counts, music, hashtags, timestamps, and the web URL).
|
||||
The TikTok connector pulls structured public data from TikTok across four verbs: **scrape** (videos), **comments**, **user search** (accounts), and **trending**. Every verb returns `{ "items": [...] }` and is billed per returned item; surfaced errors (an `errorCode` field, e.g. a withheld feed) are never charged.
|
||||
|
||||
## Endpoint
|
||||
## Scrape videos
|
||||
|
||||
```bash
|
||||
POST /api/v1/workspaces/{workspace_id}/scrapers/tiktok/scrape
|
||||
```
|
||||
|
||||
## Inputs
|
||||
|
||||
At least one of `urls`, `profiles`, `hashtags`, or `search_queries` is required.
|
||||
Give it URLs (a video, a profile, a hashtag, or a search page) and/or profiles, hashtags, or search terms; returns videos (caption, author, play/like/comment/share counts, music, hashtags, timestamps, and the web URL). At least one of `urls`, `profiles`, `hashtags`, or `search_queries` is required.
|
||||
|
||||
| Field | Default | Description |
|
||||
|-------|---------|-------------|
|
||||
|
|
@ -24,22 +22,76 @@ At least one of `urls`, `profiles`, `hashtags`, or `search_queries` is required.
|
|||
| `results_per_page` | `10` | Max videos per profile/hashtag/search target |
|
||||
| `max_items` | `10` | Max total videos returned across all sources (hard cap 100) |
|
||||
|
||||
## Example
|
||||
|
||||
```bash
|
||||
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/tiktok/scrape" \
|
||||
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"hashtags": ["food"],
|
||||
"max_items": 25
|
||||
}'
|
||||
-d '{ "hashtags": ["food"], "max_items": 25 }'
|
||||
```
|
||||
|
||||
The response is `{ "items": [...] }` — one item per video. Billing is per returned item.
|
||||
|
||||
<Callout type="info">
|
||||
Video, hashtag, and search targets are the reliable paths. TikTok restricts the profile video endpoint for automated clients, so a `profiles` target can return no items even when the account is public — prefer hashtags, search, or a direct video URL.
|
||||
Video and hashtag targets are the reliable video paths. A `profiles` target returns the account's **metadata** (name, followers, bio, verification) reliably, but TikTok often withholds its **video list** from automated clients — so a profile can return metadata with no videos. Keyword **video** search is login-walled and returns a surfaced error; to find accounts by keyword use **user search** below.
|
||||
</Callout>
|
||||
|
||||
For the full input and output JSON schemas and generated code snippets in your language, open **API Playground → TikTok → Scrape** in your workspace.
|
||||
## Comments
|
||||
|
||||
```bash
|
||||
POST /api/v1/workspaces/{workspace_id}/scrapers/tiktok/comments
|
||||
```
|
||||
|
||||
Given TikTok video URLs, returns each video's public comment thread: comment text, author, like count, and reply count (replies carry `repliesToId`, the parent comment id).
|
||||
|
||||
| Field | Default | Description |
|
||||
|-------|---------|-------------|
|
||||
| `video_urls` | — | TikTok video URLs (`/@<user>/video/<id>`), max 20 per call |
|
||||
| `comments_per_video` | `20` | Max comments per video |
|
||||
| `max_items` | `20` | Max total comments returned (hard cap 100) |
|
||||
|
||||
```bash
|
||||
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/tiktok/comments" \
|
||||
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{ "video_urls": ["https://www.tiktok.com/@nasa/video/123"], "max_items": 50 }'
|
||||
```
|
||||
|
||||
## User search
|
||||
|
||||
```bash
|
||||
POST /api/v1/workspaces/{workspace_id}/scrapers/tiktok/user_search
|
||||
```
|
||||
|
||||
Finds public TikTok accounts by keyword — the reliable account-discovery path. Returns matching profiles with name, followers, bio, and verification.
|
||||
|
||||
| Field | Default | Description |
|
||||
|-------|---------|-------------|
|
||||
| `queries` | — | Keywords to find accounts by, e.g. `["nasa", "cooking"]` (max 20) |
|
||||
| `results_per_query` | `10` | Max accounts per query |
|
||||
| `max_items` | `10` | Max total accounts returned (hard cap 100) |
|
||||
|
||||
```bash
|
||||
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/tiktok/user_search" \
|
||||
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{ "queries": ["space agency"], "max_items": 20 }'
|
||||
```
|
||||
|
||||
## Trending
|
||||
|
||||
```bash
|
||||
POST /api/v1/workspaces/{workspace_id}/scrapers/tiktok/trending
|
||||
```
|
||||
|
||||
Returns the current trending videos from TikTok's Explore feed — no input needed beyond how many to return. Items use the same video shape as **scrape** and bill on the same per-video meter.
|
||||
|
||||
| Field | Default | Description |
|
||||
|-------|---------|-------------|
|
||||
| `max_items` | `20` | Max trending videos returned (hard cap 100) |
|
||||
|
||||
```bash
|
||||
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/tiktok/trending" \
|
||||
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{ "max_items": 30 }'
|
||||
```
|
||||
|
||||
For the full input and output JSON schemas and generated code snippets in your language, open **API Playground → TikTok** in your workspace.
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ export const tiktok: ConnectorPageContent = {
|
|||
|
||||
metaTitle: "TikTok Scraper API for Trend and Creator Research | SurfSense",
|
||||
metaDescription:
|
||||
"Scrape public TikTok videos by hashtag, search, or URL with the SurfSense TikTok Scraper API. No approval process or research-API gatekeeping, plus a free tier. Start now.",
|
||||
"Scrape public TikTok videos, comments, accounts, and trending feeds by hashtag, search, or URL with the SurfSense TikTok Scraper API. No approval process or research-API gatekeeping, plus a free tier. Start now.",
|
||||
keywords: [
|
||||
"tiktok scraper",
|
||||
"tiktok scraper api",
|
||||
|
|
@ -17,6 +17,9 @@ export const tiktok: ConnectorPageContent = {
|
|||
"scrape tiktok",
|
||||
"tiktok data api",
|
||||
"tiktok hashtag scraper",
|
||||
"tiktok comments scraper",
|
||||
"tiktok trending scraper",
|
||||
"tiktok user search",
|
||||
"tiktok trend tracking",
|
||||
"tiktok mcp",
|
||||
"social listening",
|
||||
|
|
@ -52,7 +55,7 @@ export const tiktok: ConnectorPageContent = {
|
|||
},
|
||||
|
||||
extractIntro:
|
||||
"Every call returns structured video items. Point the API at a hashtag, a search query, a creator profile, or a specific video URL.",
|
||||
"Every call returns structured items. Scrape videos from a hashtag, search query, creator profile, or video URL — or switch verbs to pull a video's comments, discover accounts by keyword, or fetch the current trending feed.",
|
||||
extractFields: [
|
||||
{
|
||||
label: "Videos",
|
||||
|
|
@ -101,7 +104,7 @@ export const tiktok: ConnectorPageContent = {
|
|||
{
|
||||
title: "Campaign and sentiment tracking",
|
||||
description:
|
||||
"Measure how a launch or branded hashtag spreads across TikTok — video count, reach, and engagement over time — and report the momentum, not a vanity view count.",
|
||||
"Measure how a launch or branded hashtag spreads across TikTok — video count, reach, and engagement over time — then pull the comments on top videos to read how the audience actually reacts, not just a vanity view count.",
|
||||
},
|
||||
],
|
||||
|
||||
|
|
@ -134,7 +137,7 @@ export const tiktok: ConnectorPageContent = {
|
|||
{
|
||||
feature: "Agent-ready",
|
||||
official: "No; you build the harness yourself",
|
||||
surfsense: "MCP server exposes tiktok.scrape as a native tool",
|
||||
surfsense: "MCP server exposes scrape, comments, user search, and trending as native tools",
|
||||
},
|
||||
],
|
||||
},
|
||||
|
|
@ -262,7 +265,12 @@ export const tiktok: ConnectorPageContent = {
|
|||
{
|
||||
question: "Can I scrape a specific creator's videos?",
|
||||
answer:
|
||||
"You can pass profiles or a profile URL, but TikTok restricts its profile video endpoint for automated clients, so a profile target can return no videos even for a public account. For reliable results, scrape by hashtag, by search query, or by a direct video URL.",
|
||||
"Pass a profile or profile URL and you always get the account's metadata — name, followers, bio, verification. TikTok often withholds the profile video list from automated clients, so that list can come back empty even for a public account; for reliable video results, scrape by hashtag, by search query, or by a direct video URL.",
|
||||
},
|
||||
{
|
||||
question: "What TikTok data can I scrape?",
|
||||
answer:
|
||||
"Four verbs: scrape (videos by hashtag, search, profile, or URL), comments (a video's public comment thread), user search (find accounts by keyword — the reliable discovery path, since keyword video search is login-walled), and trending (the current Explore feed). Each returns structured items and is billed per item returned.",
|
||||
},
|
||||
],
|
||||
|
||||
|
|
|
|||
|
|
@ -53,7 +53,16 @@ export const PLAYGROUND_PLATFORMS: PlaygroundPlatform[] = [
|
|||
id: "tiktok",
|
||||
label: "TikTok",
|
||||
icon: TikTokIcon,
|
||||
verbs: [{ name: "tiktok.scrape", verb: "scrape", label: "Scrape" }],
|
||||
verbs: [
|
||||
{ name: "tiktok.scrape", verb: "scrape", label: "Scrape" },
|
||||
{ name: "tiktok.comments", verb: "comments", label: "Comments" },
|
||||
{
|
||||
name: "tiktok.user_search",
|
||||
verb: "user_search",
|
||||
label: "User Search",
|
||||
},
|
||||
{ name: "tiktok.trending", verb: "trending", label: "Trending" },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: "google_maps",
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue