feat(tiktok): surface comments, user_search, trending across all product surfaces

Brings the three newer verbs to parity with tiktok.scrape everywhere it was
wired: MCP tools (+ selfcheck manifest), the chat subagent prompt/description,
the playground catalog, the native docs page, and the SEO marketing page. Also
adds live e2e stages for comments, user search, and trending. Docs/FAQ now
state the real contract: profile metadata is reliable while its video list can
be withheld, and keyword video search is walled (use user search for accounts).
This commit is contained in:
CREDO23 2026-07-09 19:09:00 +02:00
parent 67b5472b9f
commit 6b15d610d9
8 changed files with 294 additions and 33 deletions

View file

@ -1,2 +1,2 @@
TikTok specialist: pulls structured public TikTok data — videos (caption/text, author, play/like/comment/share counts, music, hashtags, timestamps, web URL) from a hashtag feed, a search query, a creator profile, or a known video URL. Also compares fresh TikTok results against earlier findings in this chat.
Use whenever the task is to find what is trending or being said on TikTok about a topic, gather a creator's or hashtag's videos, or scrape a specific video URL. Triggers include "search TikTok for X", "trending TikTok videos about X", "videos with #X", and "scrape this TikTok video". Not for general web pages (use the web crawling specialist), Google results (use the Google Search specialist), Reddit (use the Reddit specialist), or YouTube (use the YouTube specialist).
TikTok specialist: pulls structured public TikTok data — videos (caption/text, author, play/like/comment/share counts, music, hashtags, timestamps, web URL) from a hashtag feed, a search query, a creator profile, or a known video URL; a video's public comment thread; accounts found by keyword; and the current Explore trending-video feed. Also compares fresh TikTok results against earlier findings in this chat.
Use whenever the task is to find what is trending or being said on TikTok about a topic, gather a creator's or hashtag's videos, read a video's comments, discover accounts by keyword, or scrape a specific video URL. Triggers include "search TikTok for X", "what's trending on TikTok", "videos with #X", "comments on this TikTok", "find TikTok accounts about X", and "scrape this TikTok video". Not for general web pages (use the web crawling specialist), Google results (use the Google Search specialist), Reddit (use the Reddit specialist), or YouTube (use the YouTube specialist).

View file

@ -6,17 +6,23 @@ Answer the delegated question from live TikTok data gathered with your verb, com
</goal>
<available_tools>
- `tiktok_scrape`
- `tiktok_scrape` — videos from urls / profiles / hashtags / search_queries
- `tiktok_comments` — a video's comment thread, from `video_urls`
- `tiktok_user_search` — find accounts by keyword, from `queries`
- `tiktok_trending` — the current Explore trending-video feed
- `read_run` / `search_run` (free readers for stored scrape output)
</available_tools>
<playbook>
- Finding videos on a topic: call `tiktok_scrape` with `hashtags` (no leading '#') and/or `search_queries`.
- Scraping a specific video, profile, hashtag, or search page: pass its TikTok URL in `urls`.
- Profiles: a creator's `profiles` feed can come back empty — TikTok restricts the profile video endpoint. Prefer `hashtags`, `search_queries`, or a direct video URL, and treat an empty profile result as a known limit, not a failure to retry endlessly.
- Controlling volume: use `max_items` for the total cap and `results_per_page` per target.
- Requested counts: `max_items` defaults to only 10 — when the task asks for N videos, set `max_items` and `results_per_page` above N. A call that caps below the target can never satisfy it.
- Batch multiple hashtags or search terms into one call rather than many single-term calls.
- Profiles: a creator's `profiles` feed returns the account's metadata (name, followers, bio, verification) reliably, but its video list is often withheld by TikTok — treat an empty video list as a known limit, not a failure to retry endlessly. Prefer `hashtags`, `search_queries`, or a direct video URL for videos.
- Comments on a video: call `tiktok_comments` with the video URL(s) in `video_urls`.
- Finding accounts (not videos): call `tiktok_user_search` with `queries` — this is the reliable path for account discovery (keyword *video* search is login-walled).
- "What's trending now": call `tiktok_trending` (no query needed); set `max_items` for how many.
- Controlling volume: use `max_items` for the total cap and `results_per_page` per target (per-verb equivalents: `comments_per_video`, `results_per_query`).
- Requested counts: `max_items` defaults low — when the task asks for N items, set `max_items` (and the per-target count) above N. A call that caps below the target can never satisfy it.
- Batch multiple hashtags, search terms, queries, or video URLs into one call rather than many single-item calls.
<include snippet="run_reader"/>
- Comparison requests: pull the current results, compare against prior values already in this conversation's earlier tool results, and report concrete deltas (added, removed, count changes).
</playbook>
@ -59,6 +65,6 @@ Return **only** one JSON object (no markdown/prose):
}
<include snippet="output_contract_base"/>
Route-specific rules:
- `evidence.findings`: one entry per distinct video or delta — a single sentence each; do not paste raw payloads. Max 10 entries, unless the delegated task asks for N items: then return up to N (each backed by a real scraped result, never padded).
- `evidence.findings`: one entry per distinct result (video, comment, or account) or delta — a single sentence each; do not paste raw payloads. Max 10 entries, unless the delegated task asks for N items: then return up to N (each backed by a real scraped result, never padded).
- `evidence.sources`: one TikTok URL per finding when applicable, same cap as findings. List each URL once.
</output_contract>

View file

@ -15,6 +15,12 @@ What it exercises (everything REAL — live network, live proxy, live browser):
Stage 5 full scrape_tiktok() pipeline on a hashtag.
Stage 6 search via the full pipeline: same graceful-degrade contract as
profile (results feed doesn't load for anonymous sessions).
Stage 7 comments on a real video URL (served anonymously once the panel
opens): real comments OR a single honest ErrorItem.
Stage 8 user search: the account-discovery XHR that DOES serve anonymous
headless sessions asserts real account records come back.
Stage 9 trending: the Explore feed of trending videos asserts real,
normalized video items come back.
On success it writes raw itemStructs under tests/fixtures/tiktok/ so the parser
suites can pin against real-shaped data without network.
@ -199,6 +205,61 @@ async def stage_search_listing() -> tuple[bool, list[dict[str, Any]]]:
return ok, items
async def stage_comments(video_url: str) -> tuple[bool, list[dict[str, Any]]]:
_hr("STAGE 7 — comments graceful-degrade")
print(f" target: {video_url}")
from app.proprietary.platforms.tiktok import scrape_tiktok_comments
# Comments load over a signed /api/comment/list XHR that TikTok serves to
# anonymous sessions once the panel opens. Pass if real comments come back
# OR a graceful ErrorItem (video has none / disabled / withheld).
items = await scrape_tiktok_comments(
[video_url], per_video=_COUNT, limit=_COUNT
)
has_comment = any(it.get("id") and not it.get("errorCode") for it in items)
has_error = any(it.get("errorCode") == "no_comments" for it in items)
ok = _check(
"comments yield records or a graceful ErrorItem (never silent empty)",
has_comment or has_error,
f"{len(items)} item(s); comment={has_comment} error={has_error}",
)
return ok, items
async def stage_user_search() -> tuple[bool, list[dict[str, Any]]]:
_hr(f"STAGE 8 — user search (browser): {_PROFILE!r}")
from app.proprietary.platforms.tiktok import search_tiktok_users
# Unlike keyword *video* search, the account-search XHR serves anonymous
# headless sessions — so this asserts real records, not just degradation.
items = await search_tiktok_users([_PROFILE], per_query=_COUNT, limit=_COUNT)
real = [it for it in items if not it.get("errorCode")]
ok = _check(
"user search returns account records",
bool(real) and bool(real[0].get("uniqueId") or real[0].get("name")),
f"{len(items)} item(s); accounts={len(real)}",
)
if real:
print(f" sample: @{real[0].get('uniqueId') or real[0].get('name')}")
return ok, items
async def stage_trending() -> tuple[bool, list[dict[str, Any]]]:
_hr("STAGE 9 — trending (browser): Explore feed")
from app.proprietary.platforms.tiktok import scrape_tiktok_trending
items = await scrape_tiktok_trending(count=_COUNT)
real = [it for it in items if not it.get("errorCode")]
ok = _check(
"trending returns normalized video items",
bool(real) and bool(real[0].get("id")) and bool(real[0].get("webVideoUrl")),
f"{len(items)} item(s); videos={len(real)}",
)
if real:
print(f" sample: {real[0].get('webVideoUrl')}{real[0].get('text', '')[:60]!r}")
return ok, items
async def main() -> int:
print("TikTok scraper functional e2e — live network + proxy + browser")
results: dict[str, bool] = {}
@ -214,8 +275,10 @@ async def main() -> int:
if video_url:
ok_video, _ = await stage_blob_video(video_url)
results["Stage 3 blob video"] = ok_video
ok_comments, _ = await stage_comments(video_url)
results["Stage 7 comments"] = ok_comments
else:
print("\n [SKIP] Stage 3 — no captured struct to build a video URL")
print("\n [SKIP] Stage 3/7 — no captured struct to build a video URL")
ok_search, _ = await stage_search_listing()
results["Stage 6 search listing"] = ok_search
@ -224,6 +287,11 @@ async def main() -> int:
results["Stage 2 profile listing"] = ok_profile
results["Stage 5 pipeline"] = await stage_pipeline()
ok_users, _ = await stage_user_search()
results["Stage 8 user search"] = ok_users
ok_trending, _ = await stage_trending()
results["Stage 9 trending"] = ok_trending
_hr("SUMMARY")
for name, ok in results.items():
print(f" {'PASS' if ok else 'FAIL/SKIP'}{name}")

View file

@ -1,4 +1,4 @@
"""TikTok scraper tool."""
"""TikTok scraper tools: scrape (videos), comments, user search, and trending."""
from __future__ import annotations
@ -17,7 +17,7 @@ from ..capability import run_scraper
def register(
mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext
) -> None:
"""Register the TikTok tool."""
"""Register the TikTok tools."""
@mcp.tool(
name="surfsense_tiktok_scrape",
@ -86,3 +86,118 @@ def register(
workspace=workspace,
response_format=response_format,
)
@mcp.tool(
name="surfsense_tiktok_comments",
title="Scrape TikTok comments",
annotations=SCRAPE,
structured_output=False,
)
async def tiktok_comments(
video_urls: Annotated[
list[str],
Field(
description="TikTok video URLs "
"('https://www.tiktok.com/@user/video/123') to pull comments from."
),
],
comments_per_video: Annotated[
int, Field(ge=1, description="Max comments to return per video.")
] = 20,
max_items: Annotated[
int, Field(ge=1, description="Maximum comments to return in total.")
] = 20,
workspace: WorkspaceParam = None,
response_format: ResponseFormatParam = "markdown",
) -> str:
"""Scrape the public comments of TikTok videos.
Returns each comment's text, author, like count, and reply count (replies
carry the parent comment id). Example: video_urls=['https://www.tiktok.com/
@nasa/video/123'], max_items=50.
"""
return await run_scraper(
client,
context,
platform="tiktok",
verb="comments",
payload={
"video_urls": video_urls,
"comments_per_video": comments_per_video,
"max_items": max_items,
},
workspace=workspace,
response_format=response_format,
)
@mcp.tool(
name="surfsense_tiktok_user_search",
title="Search TikTok accounts",
annotations=SCRAPE,
structured_output=False,
)
async def tiktok_user_search(
queries: Annotated[
list[str],
Field(
description="Keywords to find TikTok accounts by, e.g. "
"['nasa', 'cooking']."
),
],
results_per_query: Annotated[
int, Field(ge=1, description="Max accounts to return per query.")
] = 10,
max_items: Annotated[
int, Field(ge=1, description="Maximum accounts to return in total.")
] = 10,
workspace: WorkspaceParam = None,
response_format: ResponseFormatParam = "markdown",
) -> str:
"""Find public TikTok accounts by keyword.
Returns matching profiles with name, followers, bio, and verification
the reliable account-discovery path (video search is login-walled).
Example: queries=['space agency'], max_items=20.
"""
return await run_scraper(
client,
context,
platform="tiktok",
verb="user_search",
payload={
"queries": queries,
"results_per_query": results_per_query,
"max_items": max_items,
},
workspace=workspace,
response_format=response_format,
)
@mcp.tool(
name="surfsense_tiktok_trending",
title="Get trending TikTok videos",
annotations=SCRAPE,
structured_output=False,
)
async def tiktok_trending(
max_items: Annotated[
int,
Field(ge=1, description="Max trending videos to return from Explore."),
] = 20,
workspace: WorkspaceParam = None,
response_format: ResponseFormatParam = "markdown",
) -> str:
"""Get the current trending TikTok videos from the Explore feed.
No input needed beyond how many to return; each video comes with caption,
author, stats, music, and its web URL. Example: max_items=30.
"""
return await run_scraper(
client,
context,
platform="tiktok",
verb="trending",
payload={"max_items": max_items},
workspace=workspace,
response_format=response_format,
)

View file

@ -24,6 +24,9 @@ EXPECTED_TOOLS = {
"surfsense_youtube_scrape",
"surfsense_youtube_comments",
"surfsense_tiktok_scrape",
"surfsense_tiktok_comments",
"surfsense_tiktok_user_search",
"surfsense_tiktok_trending",
"surfsense_google_maps_scrape",
"surfsense_google_maps_reviews",
"surfsense_list_scraper_runs",

View file

@ -1,19 +1,17 @@
---
title: TikTok
description: Scrape public TikTok videos by URL, profile, hashtag, or search
description: Scrape public TikTok videos, comments, accounts, and trending feeds
---
The TikTok scraper pulls structured public data from TikTok. Give it URLs (a video, a profile, a hashtag, or a search page) and/or profiles, hashtags, or search terms, and it returns videos (caption, author, play/like/comment/share counts, music, hashtags, timestamps, and the web URL).
The TikTok connector pulls structured public data from TikTok across four verbs: **scrape** (videos), **comments**, **user search** (accounts), and **trending**. Every verb returns `{ "items": [...] }` and is billed per returned item; surfaced errors (an `errorCode` field, e.g. a withheld feed) are never charged.
## Endpoint
## Scrape videos
```bash
POST /api/v1/workspaces/{workspace_id}/scrapers/tiktok/scrape
```
## Inputs
At least one of `urls`, `profiles`, `hashtags`, or `search_queries` is required.
Give it URLs (a video, a profile, a hashtag, or a search page) and/or profiles, hashtags, or search terms; returns videos (caption, author, play/like/comment/share counts, music, hashtags, timestamps, and the web URL). At least one of `urls`, `profiles`, `hashtags`, or `search_queries` is required.
| Field | Default | Description |
|-------|---------|-------------|
@ -24,22 +22,76 @@ At least one of `urls`, `profiles`, `hashtags`, or `search_queries` is required.
| `results_per_page` | `10` | Max videos per profile/hashtag/search target |
| `max_items` | `10` | Max total videos returned across all sources (hard cap 100) |
## Example
```bash
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/tiktok/scrape" \
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"hashtags": ["food"],
"max_items": 25
}'
-d '{ "hashtags": ["food"], "max_items": 25 }'
```
The response is `{ "items": [...] }` — one item per video. Billing is per returned item.
<Callout type="info">
Video, hashtag, and search targets are the reliable paths. TikTok restricts the profile video endpoint for automated clients, so a `profiles` target can return no items even when the account is public — prefer hashtags, search, or a direct video URL.
Video and hashtag targets are the reliable video paths. A `profiles` target returns the account's **metadata** (name, followers, bio, verification) reliably, but TikTok often withholds its **video list** from automated clients — so a profile can return metadata with no videos. Keyword **video** search is login-walled and returns a surfaced error; to find accounts by keyword use **user search** below.
</Callout>
For the full input and output JSON schemas and generated code snippets in your language, open **API Playground → TikTok → Scrape** in your workspace.
## Comments
```bash
POST /api/v1/workspaces/{workspace_id}/scrapers/tiktok/comments
```
Given TikTok video URLs, returns each video's public comment thread: comment text, author, like count, and reply count (replies carry `repliesToId`, the parent comment id).
| Field | Default | Description |
|-------|---------|-------------|
| `video_urls` | — | TikTok video URLs (`/@<user>/video/<id>`), max 20 per call |
| `comments_per_video` | `20` | Max comments per video |
| `max_items` | `20` | Max total comments returned (hard cap 100) |
```bash
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/tiktok/comments" \
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "video_urls": ["https://www.tiktok.com/@nasa/video/123"], "max_items": 50 }'
```
## User search
```bash
POST /api/v1/workspaces/{workspace_id}/scrapers/tiktok/user_search
```
Finds public TikTok accounts by keyword — the reliable account-discovery path. Returns matching profiles with name, followers, bio, and verification.
| Field | Default | Description |
|-------|---------|-------------|
| `queries` | — | Keywords to find accounts by, e.g. `["nasa", "cooking"]` (max 20) |
| `results_per_query` | `10` | Max accounts per query |
| `max_items` | `10` | Max total accounts returned (hard cap 100) |
```bash
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/tiktok/user_search" \
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "queries": ["space agency"], "max_items": 20 }'
```
## Trending
```bash
POST /api/v1/workspaces/{workspace_id}/scrapers/tiktok/trending
```
Returns the current trending videos from TikTok's Explore feed — no input needed beyond how many to return. Items use the same video shape as **scrape** and bill on the same per-video meter.
| Field | Default | Description |
|-------|---------|-------------|
| `max_items` | `20` | Max trending videos returned (hard cap 100) |
```bash
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/tiktok/trending" \
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "max_items": 30 }'
```
For the full input and output JSON schemas and generated code snippets in your language, open **API Playground → TikTok** in your workspace.

View file

@ -8,7 +8,7 @@ export const tiktok: ConnectorPageContent = {
metaTitle: "TikTok Scraper API for Trend and Creator Research | SurfSense",
metaDescription:
"Scrape public TikTok videos by hashtag, search, or URL with the SurfSense TikTok Scraper API. No approval process or research-API gatekeeping, plus a free tier. Start now.",
"Scrape public TikTok videos, comments, accounts, and trending feeds by hashtag, search, or URL with the SurfSense TikTok Scraper API. No approval process or research-API gatekeeping, plus a free tier. Start now.",
keywords: [
"tiktok scraper",
"tiktok scraper api",
@ -17,6 +17,9 @@ export const tiktok: ConnectorPageContent = {
"scrape tiktok",
"tiktok data api",
"tiktok hashtag scraper",
"tiktok comments scraper",
"tiktok trending scraper",
"tiktok user search",
"tiktok trend tracking",
"tiktok mcp",
"social listening",
@ -52,7 +55,7 @@ export const tiktok: ConnectorPageContent = {
},
extractIntro:
"Every call returns structured video items. Point the API at a hashtag, a search query, a creator profile, or a specific video URL.",
"Every call returns structured items. Scrape videos from a hashtag, search query, creator profile, or video URL — or switch verbs to pull a video's comments, discover accounts by keyword, or fetch the current trending feed.",
extractFields: [
{
label: "Videos",
@ -101,7 +104,7 @@ export const tiktok: ConnectorPageContent = {
{
title: "Campaign and sentiment tracking",
description:
"Measure how a launch or branded hashtag spreads across TikTok — video count, reach, and engagement over time — and report the momentum, not a vanity view count.",
"Measure how a launch or branded hashtag spreads across TikTok — video count, reach, and engagement over time — then pull the comments on top videos to read how the audience actually reacts, not just a vanity view count.",
},
],
@ -134,7 +137,7 @@ export const tiktok: ConnectorPageContent = {
{
feature: "Agent-ready",
official: "No; you build the harness yourself",
surfsense: "MCP server exposes tiktok.scrape as a native tool",
surfsense: "MCP server exposes scrape, comments, user search, and trending as native tools",
},
],
},
@ -262,7 +265,12 @@ export const tiktok: ConnectorPageContent = {
{
question: "Can I scrape a specific creator's videos?",
answer:
"You can pass profiles or a profile URL, but TikTok restricts its profile video endpoint for automated clients, so a profile target can return no videos even for a public account. For reliable results, scrape by hashtag, by search query, or by a direct video URL.",
"Pass a profile or profile URL and you always get the account's metadata — name, followers, bio, verification. TikTok often withholds the profile video list from automated clients, so that list can come back empty even for a public account; for reliable video results, scrape by hashtag, by search query, or by a direct video URL.",
},
{
question: "What TikTok data can I scrape?",
answer:
"Four verbs: scrape (videos by hashtag, search, profile, or URL), comments (a video's public comment thread), user search (find accounts by keyword — the reliable discovery path, since keyword video search is login-walled), and trending (the current Explore feed). Each returns structured items and is billed per item returned.",
},
],

View file

@ -53,7 +53,16 @@ export const PLAYGROUND_PLATFORMS: PlaygroundPlatform[] = [
id: "tiktok",
label: "TikTok",
icon: TikTokIcon,
verbs: [{ name: "tiktok.scrape", verb: "scrape", label: "Scrape" }],
verbs: [
{ name: "tiktok.scrape", verb: "scrape", label: "Scrape" },
{ name: "tiktok.comments", verb: "comments", label: "Comments" },
{
name: "tiktok.user_search",
verb: "user_search",
label: "User Search",
},
{ name: "tiktok.trending", verb: "trending", label: "Trending" },
],
},
{
id: "google_maps",