mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-16 23:01:06 +02:00
feat(tiktok): expose tiktok.scrape on MCP, chat subagent, playground
This commit is contained in:
CREDO23
parent
2943d8b23c
commit
ed1c3a1f3d
15 changed files with 248 additions and 2 deletions
|
|
@ -36,6 +36,7 @@ SUBAGENT_TO_REQUIRED_CONNECTOR_MAP: dict[str, frozenset[str]] = {
|
|||
"google_maps": frozenset(),
|
||||
"google_search": frozenset(),
|
||||
"reddit": frozenset(),
|
||||
"tiktok": frozenset(),
|
||||
"mcp_discovery": frozenset(
|
||||
{
|
||||
"SLACK_CONNECTOR",
|
||||
|
|
|
|||
|
|
@ -0,0 +1 @@
|
|||
"""``tiktok`` builtin subagent: structured public TikTok videos and listings."""
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
"""``tiktok`` route: ``SurfSenseSubagentSpec`` builder for deepagents."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.language_models import BaseChatModel
|
||||
from langchain_core.tools import BaseTool
|
||||
|
||||
from app.agents.chat.multi_agent_chat.subagents.shared.md_file_reader import (
|
||||
read_md_file,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.subagents.shared.spec import SurfSenseSubagentSpec
|
||||
from app.agents.chat.multi_agent_chat.subagents.shared.subagent_builder import (
|
||||
pack_subagent,
|
||||
)
|
||||
|
||||
from .tools.index import NAME, RULESET, load_tools
|
||||
|
||||
|
||||
def build_subagent(
|
||||
*,
|
||||
dependencies: dict[str, Any],
|
||||
model: BaseChatModel | None = None,
|
||||
middleware_stack: dict[str, Any] | None = None,
|
||||
mcp_tools: list[BaseTool] | None = None,
|
||||
) -> SurfSenseSubagentSpec:
|
||||
tools = [*load_tools(dependencies=dependencies), *(mcp_tools or [])]
|
||||
description = (
|
||||
read_md_file(__package__, "description").strip()
|
||||
or "Pulls structured data from public TikTok videos, hashtags, and searches."
|
||||
)
|
||||
system_prompt = read_md_file(__package__, "system_prompt").strip()
|
||||
return pack_subagent(
|
||||
name=NAME,
|
||||
description=description,
|
||||
system_prompt=system_prompt,
|
||||
tools=tools,
|
||||
ruleset=RULESET,
|
||||
dependencies=dependencies,
|
||||
model=model,
|
||||
middleware_stack=middleware_stack,
|
||||
)
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
TikTok specialist: pulls structured public TikTok data — videos (caption/text, author, play/like/comment/share counts, music, hashtags, timestamps, web URL) from a hashtag feed, a search query, a creator profile, or a known video URL. Also compares fresh TikTok results against earlier findings in this chat.
|
||||
Use whenever the task is to find what is trending or being said on TikTok about a topic, gather a creator's or hashtag's videos, or scrape a specific video URL. Triggers include "search TikTok for X", "trending TikTok videos about X", "videos with #X", and "scrape this TikTok video". Not for general web pages (use the web crawling specialist), Google results (use the Google Search specialist), Reddit (use the Reddit specialist), or YouTube (use the YouTube specialist).
|
||||
|
|
@ -0,0 +1,64 @@
|
|||
You are the SurfSense TikTok sub-agent.
|
||||
You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis.
|
||||
|
||||
<goal>
|
||||
Answer the delegated question from live TikTok data gathered with your verb, comparing against earlier results already in this conversation when the task calls for it.
|
||||
</goal>
|
||||
|
||||
<available_tools>
|
||||
- `tiktok_scrape`
|
||||
- `read_run` / `search_run` (free readers for stored scrape output)
|
||||
</available_tools>
|
||||
|
||||
<playbook>
|
||||
- Finding videos on a topic: call `tiktok_scrape` with `hashtags` (no leading '#') and/or `search_queries`.
|
||||
- Scraping a specific video, profile, hashtag, or search page: pass its TikTok URL in `urls`.
|
||||
- Profiles: a creator's `profiles` feed can come back empty — TikTok restricts the profile video endpoint. Prefer `hashtags`, `search_queries`, or a direct video URL, and treat an empty profile result as a known limit, not a failure to retry endlessly.
|
||||
- Controlling volume: use `max_items` for the total cap and `results_per_page` per target.
|
||||
- Requested counts: `max_items` defaults to only 10 — when the task asks for N videos, set `max_items` and `results_per_page` above N. A call that caps below the target can never satisfy it.
|
||||
- Batch multiple hashtags or search terms into one call rather than many single-term calls.
|
||||
<include snippet="run_reader"/>
|
||||
- Comparison requests: pull the current results, compare against prior values already in this conversation's earlier tool results, and report concrete deltas (added, removed, count changes).
|
||||
</playbook>
|
||||
|
||||
<tool_policy>
|
||||
- Use only tools in `<available_tools>`.
|
||||
- Report only results present in the tool output. Never invent captions, URLs, authors, or counts.
|
||||
</tool_policy>
|
||||
|
||||
<out_of_scope>
|
||||
- Do not read arbitrary web pages — that belongs to the web crawling specialist.
|
||||
- Do not generate deliverables or perform connector mutations; return findings for the supervisor to act on.
|
||||
- Reddit belongs to the Reddit specialist; YouTube belongs to the YouTube specialist; Google results belong to the Google Search specialist.
|
||||
</out_of_scope>
|
||||
|
||||
<safety>
|
||||
- Report uncertainty explicitly when evidence is incomplete or conflicting.
|
||||
- Never present unverified claims as facts.
|
||||
</safety>
|
||||
|
||||
<failure_policy>
|
||||
- Underspecified request — no usable hashtag, query, or URL — return `status=blocked` with the missing fields.
|
||||
- Tool failure: return `status=error` with a concise recovery `next_step`.
|
||||
- No useful evidence: return `status=blocked` with a narrower query or the scope you still need.
|
||||
</failure_policy>
|
||||
|
||||
<output_contract>
|
||||
Return **only** one JSON object (no markdown/prose):
|
||||
{
|
||||
"status": "success" | "partial" | "blocked" | "error",
|
||||
"action_summary": string,
|
||||
"evidence": {
|
||||
"findings": string[],
|
||||
"sources": string[],
|
||||
"confidence": "high" | "medium" | "low"
|
||||
},
|
||||
"next_step": string | null,
|
||||
"missing_fields": string[] | null,
|
||||
"assumptions": string[] | null
|
||||
}
|
||||
<include snippet="output_contract_base"/>
|
||||
Route-specific rules:
|
||||
- `evidence.findings`: one entry per distinct video or delta — a single sentence each; do not paste raw payloads. Max 10 entries, unless the delegated task asks for N items: then return up to N (each backed by a real scraped result, never padded).
|
||||
- `evidence.sources`: one TikTok URL per finding when applicable, same cap as findings. List each URL once.
|
||||
</output_contract>
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
"""``tiktok`` sub-agent tools: the TikTok scrape capability verb."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.tools import BaseTool
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.permissions import Ruleset
|
||||
from app.capabilities.core.access.agent import build_capability_tools
|
||||
from app.capabilities.tiktok.scrape.definition import TIKTOK_SCRAPE
|
||||
|
||||
NAME = "tiktok"
|
||||
|
||||
RULESET = Ruleset(origin=NAME, rules=[])
|
||||
|
||||
_CI_VERBS = [TIKTOK_SCRAPE]
|
||||
|
||||
|
||||
def load_tools(
|
||||
*, dependencies: dict[str, Any] | None = None, **kwargs: Any
|
||||
) -> list[BaseTool]:
|
||||
d = {**(dependencies or {}), **kwargs}
|
||||
return build_capability_tools(
|
||||
workspace_id=d.get("workspace_id"),
|
||||
capabilities=_CI_VERBS,
|
||||
)
|
||||
|
|
@ -33,6 +33,9 @@ from app.agents.chat.multi_agent_chat.subagents.builtins.memory.agent import (
|
|||
from app.agents.chat.multi_agent_chat.subagents.builtins.reddit.agent import (
|
||||
build_subagent as build_reddit_subagent,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.subagents.builtins.tiktok.agent import (
|
||||
build_subagent as build_tiktok_subagent,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.subagents.builtins.web_crawler.agent import (
|
||||
build_subagent as build_web_crawler_subagent,
|
||||
)
|
||||
|
|
@ -84,6 +87,7 @@ SUBAGENT_BUILDERS_BY_NAME: dict[str, SubagentBuilder] = {
|
|||
"memory": build_memory_subagent,
|
||||
"onedrive": build_onedrive_subagent,
|
||||
"reddit": build_reddit_subagent,
|
||||
"tiktok": build_tiktok_subagent,
|
||||
"web_crawler": build_web_crawler_subagent,
|
||||
"youtube": build_youtube_subagent,
|
||||
}
|
||||
|
|
|
|||
|
|
@ -37,6 +37,7 @@ _EXPECTED_SUBAGENTS = frozenset(
|
|||
"memory",
|
||||
"onedrive",
|
||||
"reddit",
|
||||
"tiktok",
|
||||
"web_crawler",
|
||||
"youtube",
|
||||
}
|
||||
|
|
|
|||
|
|
@ -13,9 +13,9 @@ from mcp.server.fastmcp import FastMCP
|
|||
from ...core.client import SurfSenseClient
|
||||
from ...core.workspace_context import WorkspaceContext
|
||||
from . import run_history
|
||||
from .platforms import google_maps, google_search, reddit, web, youtube
|
||||
from .platforms import google_maps, google_search, reddit, tiktok, web, youtube
|
||||
|
||||
_REGISTRARS = (web, google_search, reddit, youtube, google_maps, run_history)
|
||||
_REGISTRARS = (web, google_search, reddit, youtube, tiktok, google_maps, run_history)
|
||||
|
||||
|
||||
def register(
|
||||
|
|
|
|||
|
|
@ -0,0 +1,88 @@
|
|||
"""TikTok scraper tool."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Annotated
|
||||
|
||||
from mcp.server.fastmcp import FastMCP
|
||||
from pydantic import Field
|
||||
|
||||
from ....core.client import SurfSenseClient
|
||||
from ....core.rendering import ResponseFormatParam
|
||||
from ....core.workspace_context import WorkspaceContext, WorkspaceParam
|
||||
from ..annotations import SCRAPE
|
||||
from ..capability import run_scraper
|
||||
|
||||
|
||||
def register(
|
||||
mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext
|
||||
) -> None:
|
||||
"""Register the TikTok tool."""
|
||||
|
||||
@mcp.tool(
|
||||
name="surfsense_tiktok_scrape",
|
||||
title="Search or scrape TikTok",
|
||||
annotations=SCRAPE,
|
||||
structured_output=False,
|
||||
)
|
||||
async def tiktok_scrape(
|
||||
urls: Annotated[
|
||||
list[str] | None,
|
||||
Field(
|
||||
description="TikTok URLs: a video, a profile "
|
||||
"('https://www.tiktok.com/@nasa'), a hashtag "
|
||||
"('https://www.tiktok.com/tag/food'), or a search URL. Provide "
|
||||
"urls OR profiles/hashtags/search_queries."
|
||||
),
|
||||
] = None,
|
||||
profiles: Annotated[
|
||||
list[str] | None,
|
||||
Field(
|
||||
description="Profile usernames to scrape, with or without a "
|
||||
"leading '@', e.g. ['nasa']."
|
||||
),
|
||||
] = None,
|
||||
hashtags: Annotated[
|
||||
list[str] | None,
|
||||
Field(
|
||||
description="Hashtag names to scrape, without the '#', e.g. "
|
||||
"['food']."
|
||||
),
|
||||
] = None,
|
||||
search_queries: Annotated[
|
||||
list[str] | None,
|
||||
Field(description="Terms to search TikTok for, e.g. ['cooking']."),
|
||||
] = None,
|
||||
results_per_page: Annotated[
|
||||
int,
|
||||
Field(ge=1, description="Max videos per profile/hashtag/search target."),
|
||||
] = 10,
|
||||
max_items: Annotated[
|
||||
int, Field(ge=1, description="Maximum videos to return in total.")
|
||||
] = 10,
|
||||
workspace: WorkspaceParam = None,
|
||||
response_format: ResponseFormatParam = "markdown",
|
||||
) -> str:
|
||||
"""Search or scrape public TikTok videos.
|
||||
|
||||
Use this for ANY TikTok research — a creator's videos, a hashtag feed,
|
||||
a search, or a specific video URL — instead of a generic web search.
|
||||
Returns videos with text, author, stats, music, and the web URL.
|
||||
Example: hashtags=['food'], max_items=20.
|
||||
"""
|
||||
return await run_scraper(
|
||||
client,
|
||||
context,
|
||||
platform="tiktok",
|
||||
verb="scrape",
|
||||
payload={
|
||||
"urls": urls,
|
||||
"profiles": profiles,
|
||||
"hashtags": hashtags,
|
||||
"search_queries": search_queries,
|
||||
"results_per_page": results_per_page,
|
||||
"max_items": max_items,
|
||||
},
|
||||
workspace=workspace,
|
||||
response_format=response_format,
|
||||
)
|
||||
|
|
@ -23,6 +23,7 @@ EXPECTED_TOOLS = {
|
|||
"surfsense_reddit_scrape",
|
||||
"surfsense_youtube_scrape",
|
||||
"surfsense_youtube_comments",
|
||||
"surfsense_tiktok_scrape",
|
||||
"surfsense_google_maps_scrape",
|
||||
"surfsense_google_maps_reviews",
|
||||
"surfsense_list_scraper_runs",
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ import {
|
|||
GoogleMapsIcon,
|
||||
GoogleSearchIcon,
|
||||
RedditIcon,
|
||||
TikTokIcon,
|
||||
WebIcon,
|
||||
YouTubeIcon,
|
||||
} from "./platform-icons";
|
||||
|
|
@ -48,6 +49,12 @@ export const PLAYGROUND_PLATFORMS: PlaygroundPlatform[] = [
|
|||
{ name: "youtube.comments", verb: "comments", label: "Comments" },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: "tiktok",
|
||||
label: "TikTok",
|
||||
icon: TikTokIcon,
|
||||
verbs: [{ name: "tiktok.scrape", verb: "scrape", label: "Scrape" }],
|
||||
},
|
||||
{
|
||||
id: "google_maps",
|
||||
label: "Google Maps",
|
||||
|
|
|
|||
|
|
@ -24,6 +24,7 @@ function brandIcon(src: string, alt: string) {
|
|||
|
||||
export const RedditIcon = brandIcon("/connectors/reddit.svg", "Reddit");
|
||||
export const YouTubeIcon = brandIcon("/connectors/youtube.svg", "YouTube");
|
||||
export const TikTokIcon = brandIcon("/connectors/tiktok.svg", "TikTok");
|
||||
export const GoogleMapsIcon = brandIcon("/connectors/google-maps.svg", "Google Maps");
|
||||
export const GoogleSearchIcon = brandIcon("/connectors/google-search.svg", "Google Search");
|
||||
export const WebIcon = brandIcon("/connectors/web.svg", "Web");
|
||||
|
|
|
|||
6
surfsense_web/public/connectors/tiktok.svg
Normal file
6
surfsense_web/public/connectors/tiktok.svg
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 48 48" width="48" height="48">
|
||||
<path fill="#212121" fill-rule="evenodd" d="M10.904,6h26.191C39.804,6,42,8.196,42,10.909v26.191C42,39.804,39.804,42,37.096,42H10.904C8.196,42,6,39.804,6,37.091V10.909C6,8.196,8.196,6,10.904,6z" clip-rule="evenodd"/>
|
||||
<path fill="#ec407a" fill-rule="evenodd" d="M29.208,20.607c1.576,1.126,3.507,1.788,5.592,1.788v-4.011c-0.395,0-0.788-0.041-1.174-0.123v3.157c-2.085,0-4.015-0.663-5.592-1.788v8.184c0,4.094-3.321,7.413-7.417,7.413c-1.528,0-2.949-0.462-4.129-1.254c1.347,1.376,3.225,2.23,5.303,2.23c4.096,0,7.417-3.319,7.417-7.413L29.208,20.607L29.208,20.607z M30.657,16.561c-0.805-0.879-1.334-2.016-1.449-3.273v-0.516h-1.113C28.375,14.369,29.331,15.734,30.657,16.561L30.657,16.561z M19.079,30.832c-0.45-0.59-0.693-1.311-0.692-2.053c0-1.873,1.519-3.391,3.393-3.391c0.349,0,0.696,0.053,1.029,0.159v-4.1c-0.389-0.053-0.781-0.076-1.174-0.068v3.191c-0.333-0.106-0.68-0.16-1.03-0.159c-1.874,0-3.392,1.518-3.392,3.392C17.213,29.156,17.972,30.31,19.079,30.832z" clip-rule="evenodd"/>
|
||||
<path fill="#fff" fill-rule="evenodd" d="M28.034,19.63c1.576,1.126,3.507,1.788,5.592,1.788v-3.157c-1.164-0.248-2.194-0.856-2.969-1.701c-1.326-0.827-2.282-2.191-2.561-3.789h-2.923v16.018c-0.007,1.867-1.523,3.379-3.393,3.379c-1.102,0-2.081-0.525-2.701-1.338c-1.107-0.522-1.866-1.678-1.866-3.011c0-1.874,1.518-3.392,3.392-3.392c0.359,0,0.705,0.056,1.03,0.159v-3.191c-4.024,0.083-7.26,3.369-7.26,7.411c0,2.018,0.806,3.847,2.114,5.183c1.18,0.792,2.601,1.254,4.129,1.254c4.096,0,7.417-3.319,7.417-7.413L28.034,19.63z" clip-rule="evenodd"/>
|
||||
<path fill="#81ecec" fill-rule="evenodd" d="M33.626,18.261v-0.854c-1.05,0.002-2.078-0.292-2.969-0.848C31.445,17.423,32.483,18.018,33.626,18.261z M28.095,12.771c-0.027-0.153-0.047-0.306-0.061-0.461v-0.516h-4.036v16.019c-0.006,1.867-1.522,3.379-3.393,3.379c-0.549,0-1.068-0.13-1.527-0.362c0.62,0.813,1.599,1.338,2.701,1.338c1.87,0,3.386-1.512,3.393-3.379V12.771H28.095z M21.35,21.657v-0.909c-0.337-0.046-0.677-0.069-1.017-0.069c-4.097,0-7.418,3.319-7.418,7.413c0,2.567,1.305,4.829,3.289,6.159c-1.308-1.336-2.114-3.165-2.114-5.183C14.09,25.026,17.326,21.74,21.35,21.657z"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.1 KiB |
Loading…
Add table
Add a link
Reference in a new issue