apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-16 23:01:06 +02:00
Code Issues Projects Releases Packages Wiki Activity

Merge pull request #1602 from MODSetter/dev

Browse source 
Release 0.0.32: TikTok & Instagram scrapers, resumable chat streaming, and CI platform polish
This commit is contained in:
Rohan Verma 2026-07-13 17:17:05 -07:00 • committed by GitHub
commit 3bb5c5e01f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
327 changed files with 16052 additions and 3483 deletions
Download patch file Download diff file Expand all files Collapse all files

15
.github/workflows/desktop-release.yml vendored
View file

@ -75,6 +75,21 @@ jobs:
echo "Windows signing: skipped"
fi
- name: Verify Apple notarization credentials
if: matrix.os == 'macos-latest'
shell: bash
# Fails in seconds with the same 401 notarytool would throw after the
# full build, instead of wasting ~15 min of build time on bad secrets.
run: |
xcrun notarytool history \
--apple-id "$APPLE_ID" \
--password "$APPLE_APP_SPECIFIC_PASSWORD" \
--team-id "$APPLE_TEAM_ID" > /dev/null
env:
APPLE_ID: ${{ secrets.APPLE_ID }}
APPLE_APP_SPECIFIC_PASSWORD: ${{ secrets.APPLE_APP_SPECIFIC_PASSWORD }}
APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }}
- name: Setup pnpm
uses: pnpm/action-setup@v5

2
CONTRIBUTING.md
View file

@ -79,7 +79,7 @@ We follow a **branch protection model** to keep `main` stable:
```
3. **Choose your setup method**:
- **Docker Setup**: Follow the [Docker Setup Guide](./DOCKER_SETUP.md)
- **Docker Setup**: Follow the [Building from Source (Contributors)](https://www.surfsense.com/docs/docker-installation#building-from-source-contributors) section of the Docker Installation guide
- **Manual Setup**: Follow the [Installation Guide](https://www.surfsense.com/docs/)
4. **Configure services**:

8
README.es.md
View file

@ -20,9 +20,9 @@
<a href="https://trendshift.io/repositories/13606" target="_blank"><img src="https://trendshift.io/api/badge/repositories/13606" alt="MODSetter%2FSurfSense | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</div>
# SurfSense: Dale inteligencia competitiva a tus agentes de IA
# SurfSense: NotebookLM para investigación de inteligencia competitiva
SurfSense es la **plataforma de inteligencia competitiva de código abierto para agentes de IA**. Tus agentes monitorean a la competencia, siguen los rankings y escuchan a tu mercado con datos en vivo de **Reddit, YouTube, Google Maps, Google Search y la web abierta**, a través de una única **API REST** o un **servidor MCP**. Agentes programados y activados por eventos convierten lo que encuentran en informes y alertas, y una base de conocimiento integrada mantiene cada hallazgo disponible para búsqueda con citas.
SurfSense es la **plataforma de inteligencia competitiva de código abierto para agentes de IA**, como NotebookLM pero con conectores de scraping en vivo. Tus agentes monitorean a la competencia, siguen los rankings y escuchan a tu mercado con datos en vivo de **Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search y la web abierta**, a través de una única **API REST** o un **servidor MCP**. Agentes programados y activados por eventos convierten lo que encuentran en informes y alertas, y una base de conocimiento integrada mantiene cada hallazgo disponible para búsqueda con citas.
> [!NOTE]
> **📢 Una nota para nuestros usuarios de la alternativa a NotebookLM**
@ -103,6 +103,8 @@ Las automatizaciones ejecutan turnos completos de agente según un horario o en
|---|---|---|
| **Reddit** | Publicaciones, comentarios y flujos de subreddits sin los límites de tasa de la API oficial | [Reddit Scraper API](https://www.surfsense.com/reddit) |
| **YouTube** | Videos, transcripciones e hilos de comentarios para escuchar a tu marca y productos | [YouTube Scraper API](https://www.surfsense.com/youtube) |
| **Instagram** | Perfiles, publicaciones y reels públicos sin la Graph API | [Instagram Scraper API](https://www.surfsense.com/instagram) |
| **TikTok** | Videos, comentarios, hashtags y perfiles sin aprobación de la Research API | [TikTok Scraper API](https://www.surfsense.com/tiktok) |
| **Google Maps** | Lugares, calificaciones y reseñas para investigar competidores locales y prospectos | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
| **Google Search** | SERPs en vivo para seguimiento de posiciones y monitoreo de mercado | [Google Search API](https://www.surfsense.com/google-search) |
| **Web Crawl** (rastreo web) | Cualquier página de la web abierta como contenido limpio y estructurado | [Web Crawling API](https://www.surfsense.com/web-crawl) |
@ -244,7 +246,7 @@ https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
| Característica | Google NotebookLM | SurfSense |
|---------|-------------------|-----------|
| **Datos de mercado en vivo para agentes** | No | Conectores de Reddit, YouTube, Google Maps, Google Search y rastreo web vía API REST y MCP |
| **Datos de mercado en vivo para agentes** | No | Conectores de Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search y rastreo web vía API REST y MCP |
| **Servidor MCP** | No | Cada conector expuesto como herramienta nativa de agente, más servidores MCP propios con aplicaciones OAuth de un clic |
| **Fuentes por Notebook** | 50 (gratis) a 600 (Ultra, $249.99/mes) | Ilimitadas |
| **Número de Notebooks** | 100 (gratis) a 500 (niveles de pago) | Ilimitado |

8
README.hi.md
View file

@ -20,9 +20,9 @@
<a href="https://trendshift.io/repositories/13606" target="_blank"><img src="https://trendshift.io/api/badge/repositories/13606" alt="MODSetter%2FSurfSense | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</div>
# SurfSense: अपने AI एजेंट्स को दें कॉम्पिटिटिव इंटेलिजेंस
# SurfSense: कॉम्पिटिटिव इंटेलिजेंस रिसर्च के लिए NotebookLM
SurfSense **AI एजेंट्स के लिए ओपन सोर्स कॉम्पिटिटिव इंटेलिजेंस प्लेटफ़ॉर्म** है। आपके एजेंट प्रतिस्पर्धियों पर नज़र रखते हैं, रैंकिंग ट्रैक करते हैं, और **Reddit, YouTube, Google Maps, Google Search और ओपन वेब** से लाइव डेटा के साथ आपके बाज़ार की बात सुनते हैं, वह भी एक ही **REST API** या **MCP सर्वर** के ज़रिए। शेड्यूल्ड और इवेंट-ट्रिगर्ड एजेंट अपनी खोजों को ब्रीफ़ और अलर्ट में बदलते हैं, और एक बिल्ट-इन नॉलेज बेस हर खोज को साइटेशन के साथ खोजने योग्य बनाए रखता है।
SurfSense **AI एजेंट्स के लिए ओपन सोर्स कॉम्पिटिटिव इंटेलिजेंस प्लेटफ़ॉर्म** है, बिलकुल NotebookLM जैसा, पर लाइव स्क्रैपिंग कनेक्टर्स के साथ। आपके एजेंट प्रतिस्पर्धियों पर नज़र रखते हैं, रैंकिंग ट्रैक करते हैं, और **Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search और ओपन वेब** से लाइव डेटा के साथ आपके बाज़ार की बात सुनते हैं, वह भी एक ही **REST API** या **MCP सर्वर** के ज़रिए। शेड्यूल्ड और इवेंट-ट्रिगर्ड एजेंट अपनी खोजों को ब्रीफ़ और अलर्ट में बदलते हैं, और एक बिल्ट-इन नॉलेज बेस हर खोज को साइटेशन के साथ खोजने योग्य बनाए रखता है।
> [!NOTE]
> **📢 हमारे NotebookLM-विकल्प उपयोगकर्ताओं के लिए एक सूचना**
@ -103,6 +103,8 @@ SurfSense **AI एजेंट्स के लिए ओपन सोर्स
|---|---|---|
| **Reddit** | आधिकारिक API की रेट लिमिट के बिना पोस्ट, कमेंट और सबरेडिट स्ट्रीम | [Reddit Scraper API](https://www.surfsense.com/reddit) |
| **YouTube** | ब्रांड और प्रोडक्ट लिसनिंग के लिए वीडियो, ट्रांसक्रिप्ट और कमेंट थ्रेड | [YouTube Scraper API](https://www.surfsense.com/youtube) |
| **Instagram** | Graph API के बिना सार्वजनिक प्रोफ़ाइल, पोस्ट और रील्स | [Instagram Scraper API](https://www.surfsense.com/instagram) |
| **TikTok** | Research API अप्रूवल के बिना वीडियो, कमेंट, हैशटैग और प्रोफ़ाइल | [TikTok Scraper API](https://www.surfsense.com/tiktok) |
| **Google Maps** | स्थानीय प्रतिस्पर्धी और लीड रिसर्च के लिए स्थान, रेटिंग और रिव्यू | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
| **Google Search** | रैंक ट्रैकिंग और मार्केट मॉनिटरिंग के लिए लाइव SERP | [Google Search API](https://www.surfsense.com/google-search) |
| **Web Crawl** | ओपन वेब का कोई भी पेज साफ़-सुथरे, स्ट्रक्चर्ड कंटेंट के रूप में | [Web Crawling API](https://www.surfsense.com/web-crawl) |
@ -244,7 +246,7 @@ https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
| फ़ीचर | Google NotebookLM | SurfSense |
|---------|-------------------|-----------|
| **एजेंट्स के लिए लाइव मार्केट डेटा** | नहीं | REST API और MCP के ज़रिए Reddit, YouTube, Google Maps, Google Search और वेब क्रॉल कनेक्टर |
| **एजेंट्स के लिए लाइव मार्केट डेटा** | नहीं | REST API और MCP के ज़रिए Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search और वेब क्रॉल कनेक्टर |
| **MCP सर्वर** | नहीं | हर कनेक्टर नेटिव एजेंट टूल के रूप में उपलब्ध, साथ ही वन-क्लिक OAuth ऐप्स के साथ अपने MCP सर्वर लाने की सुविधा |
| **प्रति नोटबुक स्रोत** | 50 (Free) से 600 (Ultra, $249.99/माह) | असीमित |
| **नोटबुक की संख्या** | 100 (Free) से 500 (सशुल्क टियर) | असीमित |

8
README.md
View file

@ -20,9 +20,9 @@
<a href="https://trendshift.io/repositories/13606" target="_blank"><img src="https://trendshift.io/api/badge/repositories/13606" alt="MODSetter%2FSurfSense | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</div>
# SurfSense: Give Your AI Agents Competitive Intelligence
# SurfSense: NotebookLM for Competitive Intelligence Research
SurfSense is the **open-source competitive intelligence platform for AI agents**. Your agents monitor competitors, track rankings, and listen to your market with live data from **Reddit, YouTube, Google Maps, Google Search, and the open web**, through one **REST API** or **MCP server**. Scheduled and event-triggered agents turn what they find into briefs and alerts, and a built-in knowledge base keeps every finding searchable with citations.
SurfSense is the **open-source competitive intelligence platform for AI agents**, like NotebookLM but with live scraping connectors. Your agents monitor competitors, track rankings, and listen to your market with live data from **Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search, and the open web**, through one **REST API** or **MCP server**. Scheduled and event-triggered agents turn what they find into briefs and alerts, and a built-in knowledge base keeps every finding searchable with citations.
> [!NOTE]
> **📢 A note for our NotebookLM-alternative users**
@ -103,6 +103,8 @@ Automations run full agent turns on a schedule or in response to events, then wr
|---|---|---|
| **Reddit** | Posts, comments, and subreddit streams without the official API's rate limits | [Reddit Scraper API](https://www.surfsense.com/reddit) |
| **YouTube** | Videos, transcripts, and comment threads for brand and product listening | [YouTube Scraper API](https://www.surfsense.com/youtube) |
| **Instagram** | Public profiles, posts, and reels without the Graph API | [Instagram Scraper API](https://www.surfsense.com/instagram) |
| **TikTok** | Videos, comments, hashtags, and profiles without Research API approval | [TikTok Scraper API](https://www.surfsense.com/tiktok) |
| **Google Maps** | Places, ratings, and reviews for local competitor and lead research | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
| **Google Search** | Live SERPs for rank tracking and market monitoring | [Google Search API](https://www.surfsense.com/google-search) |
| **Web Crawl** | Any page on the open web as clean, structured content | [Web Crawling API](https://www.surfsense.com/web-crawl) |
@ -243,7 +245,7 @@ Still comparing us as a NotebookLM alternative? Here is the honest breakdown.
| Feature | Google NotebookLM | SurfSense |
|---------|-------------------|-----------|
| **Live market data for agents** | No | Reddit, YouTube, Google Maps, Google Search, and web crawl connectors via REST API and MCP |
| **Live market data for agents** | No | Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search, and web crawl connectors via REST API and MCP |
| **MCP server** | No | Every connector exposed as a native agent tool, plus bring-your-own MCP servers with one-click OAuth apps |
| **Sources per Notebook** | 50 (Free) to 600 (Ultra, $249.99/mo) | Unlimited |
| **Number of Notebooks** | 100 (Free) to 500 (paid tiers) | Unlimited |

8
README.pt-BR.md
View file

@ -20,9 +20,9 @@
<a href="https://trendshift.io/repositories/13606" target="_blank"><img src="https://trendshift.io/api/badge/repositories/13606" alt="MODSetter%2FSurfSense | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</div>
# SurfSense: Dê Inteligência Competitiva aos Seus Agentes de IA
# SurfSense: NotebookLM para Pesquisa de Inteligência Competitiva
O SurfSense é a **plataforma open source de inteligência competitiva para agentes de IA**. Seus agentes monitoram concorrentes, acompanham rankings e escutam o seu mercado com dados ao vivo do **Reddit, YouTube, Google Maps, Google Search e da web aberta**, por meio de uma única **API REST** ou de um **servidor MCP**. Agentes agendados ou acionados por eventos transformam o que encontram em relatórios e alertas, e uma base de conhecimento integrada mantém cada descoberta pesquisável, com citações.
O SurfSense é a **plataforma open source de inteligência competitiva para agentes de IA**, como o NotebookLM, mas com conectores de scraping ao vivo. Seus agentes monitoram concorrentes, acompanham rankings e escutam o seu mercado com dados ao vivo do **Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search e da web aberta**, por meio de uma única **API REST** ou de um **servidor MCP**. Agentes agendados ou acionados por eventos transformam o que encontram em relatórios e alertas, e uma base de conhecimento integrada mantém cada descoberta pesquisável, com citações.
> [!NOTE]
> **📢 Um recado para nossos usuários que buscavam uma alternativa ao NotebookLM**
@ -103,6 +103,8 @@ As automações executam turnos completos de agente de forma agendada ou em resp
|---|---|---|
| **Reddit** | Posts, comentários e fluxos de subreddits sem os limites de requisição da API oficial | [Reddit Scraper API](https://www.surfsense.com/reddit) |
| **YouTube** | Vídeos, transcrições e threads de comentários para monitoramento de marca e produto | [YouTube Scraper API](https://www.surfsense.com/youtube) |
| **Instagram** | Perfis, posts e reels públicos sem a Graph API | [Instagram Scraper API](https://www.surfsense.com/instagram) |
| **TikTok** | Vídeos, comentários, hashtags e perfis sem aprovação da Research API | [TikTok Scraper API](https://www.surfsense.com/tiktok) |
| **Google Maps** | Estabelecimentos, avaliações e reviews para pesquisa local de concorrentes e leads | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
| **Google Search** | SERPs ao vivo para acompanhamento de rankings e monitoramento de mercado | [Google Search API](https://www.surfsense.com/google-search) |
| **Web Crawl** (rastreamento web) | Qualquer página da web aberta como conteúdo limpo e estruturado | [Web Crawling API](https://www.surfsense.com/web-crawl) |
@ -244,7 +246,7 @@ Ainda nos comparando como alternativa ao NotebookLM? Aqui está o comparativo ho
| Recurso | Google NotebookLM | SurfSense |
|---------|-------------------|-----------|
| **Dados de mercado ao vivo para agentes** | Não | Conectores de Reddit, YouTube, Google Maps, Google Search e rastreamento web via API REST e MCP |
| **Dados de mercado ao vivo para agentes** | Não | Conectores de Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search e rastreamento web via API REST e MCP |
| **Servidor MCP** | Não | Cada conector exposto como ferramenta nativa de agente, além de servidores MCP próprios com apps OAuth em um clique |
| **Fontes por Notebook** | 50 (gratuito) a 600 (Ultra, US$ 249,99/mês) | Ilimitadas |
| **Número de Notebooks** | 100 (gratuito) a 500 (planos pagos) | Ilimitado |

8
README.zh-CN.md
View file

@ -20,9 +20,9 @@
<a href="https://trendshift.io/repositories/13606" target="_blank"><img src="https://trendshift.io/api/badge/repositories/13606" alt="MODSetter%2FSurfSense | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</div>
# SurfSense：为你的 AI 智能体注入竞争情报能力
# SurfSense：面向竞争情报研究的 NotebookLM
SurfSense 是**面向 AI 智能体的开源竞争情报平台**。你的智能体可以通过一个 **REST API** 或 **MCP 服务器**，利用来自 **Reddit、YouTube、Google Maps、Google Search 和开放网络**的实时数据，监控竞争对手、追踪排名、倾听市场动态。定时和事件触发的智能体会把发现的内容转化为简报和预警，内置的知识库则让每一条发现都可搜索、可引用。
SurfSense 是**面向 AI 智能体的开源竞争情报平台**，就像 NotebookLM，但配备了实时抓取连接器。你的智能体可以通过一个 **REST API** 或 **MCP 服务器**，利用来自 **Reddit、YouTube、Instagram、TikTok、Google Maps、Google Search 和开放网络**的实时数据，监控竞争对手、追踪排名、倾听市场动态。定时和事件触发的智能体会把发现的内容转化为简报和预警，内置的知识库则让每一条发现都可搜索、可引用。
> [!NOTE]
> **📢 致我们的 NotebookLM 替代品用户**
@ -103,6 +103,8 @@ SurfSense 是**面向 AI 智能体的开源竞争情报平台**。你的智能
|---|---|---|
| **Reddit** | 帖子、评论和子版块信息流，不受官方 API 速率限制 | [Reddit Scraper API](https://www.surfsense.com/reddit) |
| **YouTube** | 视频、字幕转录和评论串，用于品牌和产品舆情监听 | [YouTube Scraper API](https://www.surfsense.com/youtube) |
| **Instagram** | 公开主页、帖子和 Reels，无需 Graph API | [Instagram Scraper API](https://www.surfsense.com/instagram) |
| **TikTok** | 视频、评论、话题标签和主页，无需 Research API 审批 | [TikTok Scraper API](https://www.surfsense.com/tiktok) |
| **Google Maps** | 地点、评分和评论，用于本地竞争对手和潜在客户调研 | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
| **Google Search** | 实时搜索结果页，用于排名追踪和市场监控 | [Google Search API](https://www.surfsense.com/google-search) |
| **Web Crawl** | 把开放网络上的任意页面转为干净、结构化的内容 | [Web Crawling API](https://www.surfsense.com/web-crawl) |
@ -244,7 +246,7 @@ https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
| 功能 | Google NotebookLM | SurfSense |
|---------|-------------------|-----------|
| **面向智能体的实时市场数据** | 无 | 通过 REST API 和 MCP 提供 Reddit、YouTube、Google Maps、Google Search 和网页爬取连接器 |
| **面向智能体的实时市场数据** | 无 | 通过 REST API 和 MCP 提供 Reddit、YouTube、Instagram、TikTok、Google Maps、Google Search 和网页爬取连接器 |
| **MCP 服务器** | 无 | 每个连接器都作为原生智能体工具暴露，还可自带 MCP 服务器并使用一键 OAuth 应用 |
| **每个笔记本的来源数** | 50 个（免费版）至 600 个（Ultra 版，249.99 美元/月） | 无限制 |
| **笔记本数量** | 100 个（免费版）至 500 个（付费档位） | 无限制 |

2
VERSION
View file

@ -1 +1 @@
0.0.31
0.0.32

8
docker/.env.example
View file

@ -48,7 +48,12 @@ ETL_SERVICE=DOCLING
# Local: sentence-transformers/all-MiniLM-L6-v2
# OpenAI: openai://text-embedding-ada-002 (set OPENAI_API_KEY below)
# Cohere: cohere://embed-english-light-v3.0 (set COHERE_API_KEY below)
# Ollama or OpenAI-compatible embedding endpoint:
# EMBEDDING_MODEL=litellm://ollama/nomic-embed-text
# EMBEDDING_BASE_URL=http://host.docker.internal:11434
EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2
# EMBEDDING_BASE_URL=
# OLLAMA_EMBEDDING_BASE_URL=
# ------------------------------------------------------------------------------
# How You Access SurfSense
@ -449,6 +454,9 @@ SURFSENSE_ENABLE_DOOM_LOOP=true
# GOOGLE_MAPS_MICROS_PER_REVIEW=1500
# YOUTUBE_MICROS_PER_VIDEO=2500
# YOUTUBE_MICROS_PER_COMMENT=1500
# TIKTOK_MICROS_PER_VIDEO=3500
# TIKTOK_MICROS_PER_USER=2500
# TIKTOK_MICROS_PER_COMMENT=1500
# Safety ceiling on per-call premium reservation, in micro-USD ($1.00 default).
# QUOTA_MAX_RESERVE_MICROS=1000000

21
surfsense_backend/.env.example
View file

@ -120,8 +120,10 @@ STRIPE_RECONCILIATION_BATCH_SIZE=100
# BACKEND_URL=https://api.yourdomain.com
# Auth
AUTH_TYPE=GOOGLE or LOCAL
REGISTRATION_ENABLED=TRUE or FALSE
# AUTH_TYPE: GOOGLE or LOCAL
AUTH_TYPE=LOCAL
# REGISTRATION_ENABLED: TRUE or FALSE
REGISTRATION_ENABLED=TRUE
# For Google Auth Only
GOOGLE_OAUTH_CLIENT_ID=924507538m
GOOGLE_OAUTH_CLIENT_SECRET=GOCSV
@ -209,6 +211,10 @@ COMPOSIO_REDIRECT_URI=http://localhost:8000/api/v1/auth/composio/connector/callb
# # Get Cohere embeddings
# embeddings = AutoEmbeddings.get_embeddings("cohere://embed-english-light-v3.0", api_key="...")
EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2
# Optional: use a separate endpoint for Chonkie/LiteLLM embedding models, for
# example EMBEDDING_MODEL=litellm://ollama/nomic-embed-text.
# EMBEDDING_BASE_URL=http://host.docker.internal:11434
# OLLAMA_EMBEDDING_BASE_URL=http://host.docker.internal:11434
# Rerankers Config
RERANKERS_ENABLED=TRUE or FALSE(Default: FALSE)
@ -286,6 +292,14 @@ MICROS_PER_PAGE=1000
# GOOGLE_MAPS_MICROS_PER_REVIEW=1500
# YOUTUBE_MICROS_PER_VIDEO=2500
# YOUTUBE_MICROS_PER_COMMENT=1500
# INSTAGRAM_SCRAPE_MICROS_PER_ITEM=3500
# INSTAGRAM_SCRAPE_MICROS_PER_COMMENT=1500
# TIKTOK_MICROS_PER_VIDEO=3500
# TIKTOK_MICROS_PER_USER=2500
# TIKTOK_MICROS_PER_COMMENT=1500
# Browser-listing retries when a feed is empty (profile feed is withheld from
# flagged IPs; each retry draws a fresh rotating exit IP). Set to 1 for a static IP.
# TIKTOK_LISTING_MAX_ATTEMPTS=3
# Low-balance warning threshold (micro-USD), surfaced to the UI. Default $0.50.
CREDIT_LOW_BALANCE_WARNING_MICROS=500000
@ -394,6 +408,9 @@ TURNSTILE_SECRET_KEY=
# Route DNS via Cloudflare DoH (anti DNS-leak). Adds a DNS round-trip => off by
# default to avoid any speed regression; enable for leak-safety-first setups.
# CRAWL_DNS_OVER_HTTPS=FALSE
# Run the browser headful on Xvfb — required for TikTok's profile video feed
# (empty to headless Chromium). Entrypoint starts Xvfb when TRUE.
# CRAWL_HEADED_XVFB_ENABLED=FALSE
# File Parser Service
ETL_SERVICE=UNSTRUCTURED or LLAMACLOUD or DOCLING

6
surfsense_backend/.gitignore vendored
View file

@ -15,4 +15,8 @@ celerybeat-schedule.*
celerybeat-schedule.dir
celerybeat-schedule.bak
/app/config/global_llm_config.yaml
app/templates/_generated/
app/templates/_generated/
/tests/unit/platforms/instagram/fixtures/post.json
/tests/unit/platforms/instagram/fixtures/profile.json
/tests/unit/platforms/instagram/fixtures/tagged.json

8
surfsense_backend/Dockerfile
View file

@ -31,11 +31,9 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
dos2unix \
git \
# ── Phase 3e stealth hardening ──────────────────────────────────────────
# Xvfb: virtual framebuffer so the stealth browser can run headful
# (headless=False) without a real display — many WAFs flag headless Chromium.
# Gated at runtime by CRAWL_HEADED_XVFB_ENABLED (Slice B); installed now so
# the image is ready. Real font packages make canvas/emoji/font-enumeration
# fingerprints resemble a real desktop (set proven against Kasada/Akamai).
# Xvfb: virtual display so the browser can run headful without real hardware
# (entrypoint starts it when CRAWL_HEADED_XVFB_ENABLED; used by TikTok's profile
# feed). Real fonts make canvas/emoji/font fingerprints look like a real desktop.
xvfb \
fonts-noto-color-emoji \
fonts-unifont \

2
surfsense_backend/app/agents/chat/multi_agent_chat/constants.py
View file

@ -36,6 +36,8 @@ SUBAGENT_TO_REQUIRED_CONNECTOR_MAP: dict[str, frozenset[str]] = {
"google_maps": frozenset(),
"google_search": frozenset(),
"reddit": frozenset(),
"instagram": frozenset(),
"tiktok": frozenset(),
"mcp_discovery": frozenset(
{
"SLACK_CONNECTOR",

6
surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/system_prompt/prompts/identity/private.md
View file

@ -6,9 +6,9 @@ reviews are moving, and what is being said across the open web — and to put
that intelligence to work alongside their own knowledge base.
You do this by dispatching **specialist subagents** via the `task` tool:
- **Live market data** — Reddit, YouTube, Google Maps, Google Search, and the
web crawler return structured, current platform data (posts, comments,
transcripts, reviews, SERPs, full page content).
- **Live market data** — Reddit, YouTube, TikTok, Google Maps, Google Search,
and the web crawler return structured, current platform data (posts,
comments, transcripts, videos, reviews, SERPs, full page content).
- **The user's own context** — their knowledge base, connected apps, and
persistent memory.
- **Deliverables** — reports, podcasts, and presentations built from what the

6
surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/system_prompt/prompts/identity/team.md
View file

@ -6,9 +6,9 @@ reviews are moving, and what is being said across the open web — and to put
that intelligence to work alongside the team's shared knowledge base.
You do this by dispatching **specialist subagents** via the `task` tool:
- **Live market data** — Reddit, YouTube, Google Maps, Google Search, and the
web crawler return structured, current platform data (posts, comments,
transcripts, reviews, SERPs, full page content).
- **Live market data** — Reddit, YouTube, TikTok, Google Maps, Google Search,
and the web crawler return structured, current platform data (posts,
comments, transcripts, videos, reviews, SERPs, full page content).
- **The team's own context** — its shared knowledge base, connected apps, and
persistent team memory.
- **Deliverables** — reports, podcasts, and presentations built from what the

5
surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/system_prompt/prompts/kb_first.md
View file

@ -1,8 +1,9 @@
<knowledge_base_first>
CRITICAL — ground factual answers in what you actually receive this turn:
- **live platform data** via the market specialists —
`task(reddit, ...)`, `task(youtube, ...)`, `task(google_maps, ...)`,
`task(google_search, ...)`, `task(web_crawler, ...)`. Anything about
`task(reddit, ...)`, `task(youtube, ...)`, `task(tiktok, ...)`,
`task(google_maps, ...)`, `task(google_search, ...)`,
`task(web_crawler, ...)`. Anything about
competitors, markets, rankings, reviews, or audience sentiment is answered
from what these return **this turn**, never from your training data: your
general knowledge of companies, prices, and rankings is stale by definition,

1
surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/system_prompt/prompts/routing.md
View file

@ -31,6 +31,7 @@ bounded fan-out (≤20 sites) the user already requested.
about a brand, product, or topic is answered from the platform where they
say it — `task(reddit, …)` for community discussion and threads,
`task(youtube, …)` for video content, transcripts, and comment sections,
`task(tiktok, …)` for short-form video trends by hashtag or search,
`task(google_maps, …)` for customer reviews of physical businesses. Web
search only finds articles *about* the conversation; the platform
specialists return the conversation itself, structured and current. For

1
surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/instagram/__init__.py Normal file
View file

@ -0,0 +1 @@
"""``instagram`` builtin subagent: structured Instagram posts, comments, and details."""

43
surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/instagram/agent.py Normal file
View file

@ -0,0 +1,43 @@
"""``instagram`` route: ``SurfSenseSubagentSpec`` builder for deepagents."""
from __future__ import annotations
from typing import Any
from langchain_core.language_models import BaseChatModel
from langchain_core.tools import BaseTool
from app.agents.chat.multi_agent_chat.subagents.shared.md_file_reader import (
read_md_file,
)
from app.agents.chat.multi_agent_chat.subagents.shared.spec import SurfSenseSubagentSpec
from app.agents.chat.multi_agent_chat.subagents.shared.subagent_builder import (
pack_subagent,
)
from .tools.index import NAME, RULESET, load_tools
def build_subagent(
*,
dependencies: dict[str, Any],
model: BaseChatModel | None = None,
middleware_stack: dict[str, Any] | None = None,
mcp_tools: list[BaseTool] | None = None,
) -> SurfSenseSubagentSpec:
tools = [*load_tools(dependencies=dependencies), *(mcp_tools or [])]
description = (
read_md_file(__package__, "description").strip()
or "Pulls structured data from Instagram posts, reels, comments, and profiles."
)
system_prompt = read_md_file(__package__, "system_prompt").strip()
return pack_subagent(
name=NAME,
description=description,
system_prompt=system_prompt,
tools=tools,
ruleset=RULESET,
dependencies=dependencies,
model=model,
middleware_stack=middleware_stack,
)

2
surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/instagram/description.md Normal file
View file

@ -0,0 +1,2 @@
Instagram specialist: pulls structured data from public Instagram — posts and reels (caption, likes, comments count, media URLs, owner, timestamp) and profile details (follower and post counts, bio). Finds public profiles by search and compares fresh Instagram data against earlier findings in this chat. Anonymous-only: hashtag/place feeds and comment threads are login-walled and unavailable.
Use whenever the task involves public Instagram content or an instagram.com profile/post/reel link. Triggers include "get this Instagram profile/post/reel", "find the Instagram profile for X", "how many followers/likes", and comparisons against earlier Instagram results in this chat. Not for general web pages (use the web crawling specialist for non-Instagram URLs).

65
surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/instagram/system_prompt.md Normal file
View file

@ -0,0 +1,65 @@
You are the SurfSense Instagram sub-agent.
You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis.
<goal>
Answer the delegated question from live Instagram data gathered with your verbs, comparing against earlier results already in this conversation when the task calls for it.
</goal>
<available_tools>
- `instagram_scrape`
- `instagram_details`
- `read_run` / `search_run` (free readers for stored scrape output)
</available_tools>
<playbook>
- Known profile/post/reel links: call `instagram_scrape` with the links in `urls` (use `result_type` to pick posts or reels). Hashtag/place URLs are unsupported (login-walled).
- Finding a profile on a topic: call `instagram_scrape` with `search_queries` (resolved to public profiles via Google; `search_type` is profile-only). Google-backed discovery is slow (~30-60s per query), so start with **at most 3** distinct queries per task and only add more if the first round returns nothing significant — never batch many phrasing variants of the same intent.
- Profile metadata (follower counts, bio, post count): call `instagram_details`.
- Batch multiple URLs (or queries) into one call rather than many single-item calls.
<include snippet="run_reader"/>
- Comparison requests: pull the current values, compare against prior values already in this conversation's earlier tool results, and report concrete deltas (added, removed, old -> new).
</playbook>
<tool_policy>
- Use only tools in `<available_tools>`.
- An item whose `status` is not `success` returned no data — report it unavailable, never invent it.
- Anonymous Instagram access can be rate-limited or blocked; if a verb returns no data, report it unavailable and suggest a narrower retry rather than fabricating.
- Report only deltas you can point to in the evidence. Never fabricate facts, counts, quotes, or URLs.
</tool_policy>
<out_of_scope>
- Do not generate deliverables or perform connector mutations; return findings for the supervisor to act on.
- Non-Instagram web pages belong to the web crawling specialist, not here.
</out_of_scope>
<safety>
- Report uncertainty explicitly when evidence is incomplete or conflicting.
- Never present unverified claims as facts.
</safety>
<failure_policy>
- Underspecified request — no usable URL or search query — return `status=blocked` with the missing fields.
- Tool failure or access block: return `status=error` with a concise recovery `next_step`.
- No useful evidence: return `status=blocked` with a narrower query or the URLs you still need.
</failure_policy>
<output_contract>
Return **only** one JSON object (no markdown/prose):
{
"status": "success" | "partial" | "blocked" | "error",
"action_summary": string,
"evidence": {
"findings": string[],
"sources": string[],
"confidence": "high" | "medium" | "low"
},
"next_step": string | null,
"missing_fields": string[] | null,
"assumptions": string[] | null
}
<include snippet="output_contract_base"/>
Route-specific rules:
- `evidence.findings`: max 10 entries, each a single sentence stating one distinct fact or delta. Do not paste raw payloads.
- `evidence.sources`: max 10 URLs, one per finding when applicable. List each URL once.
</output_contract>
</output>

0
surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/instagram/tools/__init__.py Normal file
View file

28
surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/instagram/tools/index.py Normal file
View file

@ -0,0 +1,28 @@
"""``instagram`` sub-agent tools: the Instagram capability verbs."""
from __future__ import annotations
from typing import Any
from langchain_core.tools import BaseTool
from app.agents.chat.multi_agent_chat.shared.permissions import Ruleset
from app.capabilities.core.access.agent import build_capability_tools
from app.capabilities.instagram.details.definition import INSTAGRAM_DETAILS
from app.capabilities.instagram.scrape.definition import INSTAGRAM_SCRAPE
NAME = "instagram"
RULESET = Ruleset(origin=NAME, rules=[])
_CI_VERBS = [INSTAGRAM_SCRAPE, INSTAGRAM_DETAILS]
def load_tools(
*, dependencies: dict[str, Any] | None = None, **kwargs: Any
) -> list[BaseTool]:
d = {**(dependencies or {}), **kwargs}
return build_capability_tools(
workspace_id=d.get("workspace_id"),
capabilities=_CI_VERBS,
)

1
surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/tiktok/__init__.py Normal file
View file

@ -0,0 +1 @@
"""``tiktok`` builtin subagent: structured public TikTok videos and listings."""

43
surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/tiktok/agent.py Normal file
View file

@ -0,0 +1,43 @@
"""``tiktok`` route: ``SurfSenseSubagentSpec`` builder for deepagents."""
from __future__ import annotations
from typing import Any
from langchain_core.language_models import BaseChatModel
from langchain_core.tools import BaseTool
from app.agents.chat.multi_agent_chat.subagents.shared.md_file_reader import (
read_md_file,
)
from app.agents.chat.multi_agent_chat.subagents.shared.spec import SurfSenseSubagentSpec
from app.agents.chat.multi_agent_chat.subagents.shared.subagent_builder import (
pack_subagent,
)
from .tools.index import NAME, RULESET, load_tools
def build_subagent(
*,
dependencies: dict[str, Any],
model: BaseChatModel | None = None,
middleware_stack: dict[str, Any] | None = None,
mcp_tools: list[BaseTool] | None = None,
) -> SurfSenseSubagentSpec:
tools = [*load_tools(dependencies=dependencies), *(mcp_tools or [])]
description = (
read_md_file(__package__, "description").strip()
or "Pulls structured data from public TikTok videos, hashtags, and searches."
)
system_prompt = read_md_file(__package__, "system_prompt").strip()
return pack_subagent(
name=NAME,
description=description,
system_prompt=system_prompt,
tools=tools,
ruleset=RULESET,
dependencies=dependencies,
model=model,
middleware_stack=middleware_stack,
)

2
surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/tiktok/description.md Normal file
View file

@ -0,0 +1,2 @@
TikTok specialist: pulls structured public TikTok data — videos (caption/text, author, play/like/comment/share counts, music, hashtags, timestamps, web URL) from a hashtag feed, a creator profile, or a known video URL; a video's public comment thread; accounts found by keyword; and the current Explore trending-video feed. Also compares fresh TikTok results against earlier findings in this chat.
Use whenever the task is to find what is trending or being said on TikTok about a topic, gather a creator's or hashtag's videos, read a video's comments, discover accounts by keyword, or scrape a specific video URL. Triggers include "search TikTok for X", "what's trending on TikTok", "videos with #X", "comments on this TikTok", "find TikTok accounts about X", and "scrape this TikTok video". Not for general web pages (use the web crawling specialist), Google results (use the Google Search specialist), Reddit (use the Reddit specialist), or YouTube (use the YouTube specialist).

70
surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/tiktok/system_prompt.md Normal file
View file

@ -0,0 +1,70 @@
You are the SurfSense TikTok sub-agent.
You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis.
<goal>
Answer the delegated question from live TikTok data gathered with your verb, comparing against earlier results already in this conversation when the task calls for it.
</goal>
<available_tools>
- `tiktok_scrape` — videos from a hashtag, a profile, or a TikTok URL
- `tiktok_comments` — a video's comment thread, from `video_urls`
- `tiktok_user_search` — find accounts by keyword, from `queries`
- `tiktok_trending` — the current Explore trending-video feed
- `read_run` / `search_run` (free readers for stored scrape output)
</available_tools>
<playbook>
- Finding videos on a topic: call `tiktok_scrape` with `hashtags` (no leading '#'), or pass a TikTok URL in `urls`. There is no keyword-video search — use hashtags or a video URL.
- Scraping a specific video, profile, hashtag, or search page: pass its TikTok URL in `urls`.
- Profiles: a creator's `profiles` feed returns the account's metadata (name, followers, bio, verification) reliably, but its video list is often withheld by TikTok — treat an empty video list as a known limit, not a failure to retry endlessly. Prefer `hashtags` or a direct video URL for videos.
- Comments on a video: call `tiktok_comments` with the video URL(s) in `video_urls`.
- Finding accounts by keyword: call `tiktok_user_search` with `queries` — that is the path for accounts.
- "What's trending now": call `tiktok_trending` (no query needed); set `max_items` for how many.
- Controlling volume: use `max_items` for the total cap and `results_per_page` per target (per-verb equivalents: `comments_per_video`, `results_per_query`).
- Requested counts: `max_items` defaults low — when the task asks for N items, set `max_items` (and the per-target count) above N. A call that caps below the target can never satisfy it.
- Batch multiple hashtags, queries, or video URLs into one call rather than many single-item calls.
<include snippet="run_reader"/>
- Comparison requests: pull the current results, compare against prior values already in this conversation's earlier tool results, and report concrete deltas (added, removed, count changes).
</playbook>
<tool_policy>
- Use only tools in `<available_tools>`.
- Report only results present in the tool output. Never invent captions, URLs, authors, or counts.
</tool_policy>
<out_of_scope>
- Do not read arbitrary web pages — that belongs to the web crawling specialist.
- Do not generate deliverables or perform connector mutations; return findings for the supervisor to act on.
- Reddit belongs to the Reddit specialist; YouTube belongs to the YouTube specialist; Google results belong to the Google Search specialist.
</out_of_scope>
<safety>
- Report uncertainty explicitly when evidence is incomplete or conflicting.
- Never present unverified claims as facts.
</safety>
<failure_policy>
- Underspecified request — no usable hashtag, query, or URL — return `status=blocked` with the missing fields.
- Tool failure: return `status=error` with a concise recovery `next_step`.
- No useful evidence: return `status=blocked` with a narrower query or the scope you still need.
</failure_policy>
<output_contract>
Return **only** one JSON object (no markdown/prose):
{
"status": "success" | "partial" | "blocked" | "error",
"action_summary": string,
"evidence": {
"findings": string[],
"sources": string[],
"confidence": "high" | "medium" | "low"
},
"next_step": string | null,
"missing_fields": string[] | null,
"assumptions": string[] | null
}
<include snippet="output_contract_base"/>
Route-specific rules:
- `evidence.findings`: one entry per distinct result (video, comment, or account) or delta — a single sentence each; do not paste raw payloads. Max 10 entries, unless the delegated task asks for N items: then return up to N (each backed by a real scraped result, never padded).
- `evidence.sources`: one TikTok URL per finding when applicable, same cap as findings. List each URL once.
</output_contract>

0
surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/tiktok/tools/__init__.py Normal file
View file

30
surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/tiktok/tools/index.py Normal file
View file

@ -0,0 +1,30 @@
"""``tiktok`` sub-agent tools: scrape, comments, user-search, and trending verbs."""
from __future__ import annotations
from typing import Any
from langchain_core.tools import BaseTool
from app.agents.chat.multi_agent_chat.shared.permissions import Ruleset
from app.capabilities.core.access.agent import build_capability_tools
from app.capabilities.tiktok.comments.definition import TIKTOK_COMMENTS
from app.capabilities.tiktok.scrape.definition import TIKTOK_SCRAPE
from app.capabilities.tiktok.trending.definition import TIKTOK_TRENDING
from app.capabilities.tiktok.user_search.definition import TIKTOK_USER_SEARCH
NAME = "tiktok"
RULESET = Ruleset(origin=NAME, rules=[])
_CI_VERBS = [TIKTOK_SCRAPE, TIKTOK_COMMENTS, TIKTOK_USER_SEARCH, TIKTOK_TRENDING]
def load_tools(
*, dependencies: dict[str, Any] | None = None, **kwargs: Any
) -> list[BaseTool]:
d = {**(dependencies or {}), **kwargs}
return build_capability_tools(
workspace_id=d.get("workspace_id"),
capabilities=_CI_VERBS,
)

8
surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py
View file

@ -21,6 +21,9 @@ from app.agents.chat.multi_agent_chat.subagents.builtins.google_maps.agent impor
from app.agents.chat.multi_agent_chat.subagents.builtins.google_search.agent import (
build_subagent as build_google_search_subagent,
)
from app.agents.chat.multi_agent_chat.subagents.builtins.instagram.agent import (
build_subagent as build_instagram_subagent,
)
from app.agents.chat.multi_agent_chat.subagents.builtins.knowledge_base.agent import (
build_subagent as build_knowledge_base_subagent,
)
@ -33,6 +36,9 @@ from app.agents.chat.multi_agent_chat.subagents.builtins.memory.agent import (
from app.agents.chat.multi_agent_chat.subagents.builtins.reddit.agent import (
build_subagent as build_reddit_subagent,
)
from app.agents.chat.multi_agent_chat.subagents.builtins.tiktok.agent import (
build_subagent as build_tiktok_subagent,
)
from app.agents.chat.multi_agent_chat.subagents.builtins.web_crawler.agent import (
build_subagent as build_web_crawler_subagent,
)
@ -79,11 +85,13 @@ SUBAGENT_BUILDERS_BY_NAME: dict[str, SubagentBuilder] = {
"google_drive": build_google_drive_subagent,
"google_maps": build_google_maps_subagent,
"google_search": build_google_search_subagent,
"instagram": build_instagram_subagent,
"knowledge_base": build_knowledge_base_subagent,
"mcp_discovery": build_mcp_discovery_subagent,
"memory": build_memory_subagent,
"onedrive": build_onedrive_subagent,
"reddit": build_reddit_subagent,
"tiktok": build_tiktok_subagent,
"web_crawler": build_web_crawler_subagent,
"youtube": build_youtube_subagent,
}

7
surfsense_backend/app/agents/video_presentation/graph.py
View file

@ -24,9 +24,10 @@ def build_graph():
workflow.add_edge("create_presentation_slides", "create_slide_audio")
workflow.add_edge("create_presentation_slides", "assign_slide_themes")
# Fan-in: scene code generation waits for both audio and themes.
workflow.add_edge("create_slide_audio", "generate_slide_scene_codes")
workflow.add_edge("assign_slide_themes", "generate_slide_scene_codes")
# Fan-in: wait for BOTH audio and themes (barrier).
workflow.add_edge(
["create_slide_audio", "assign_slide_themes"], "generate_slide_scene_codes"
)
workflow.add_edge("generate_slide_scene_codes", "__end__")

10
surfsense_backend/app/capabilities/core/billing.py
View file

@ -35,6 +35,11 @@ _PLATFORM_RATE_KEYS: dict[BillingUnit, str] = {
BillingUnit.GOOGLE_MAPS_REVIEW: "GOOGLE_MAPS_MICROS_PER_REVIEW",
BillingUnit.YOUTUBE_VIDEO: "YOUTUBE_MICROS_PER_VIDEO",
BillingUnit.YOUTUBE_COMMENT: "YOUTUBE_MICROS_PER_COMMENT",
BillingUnit.INSTAGRAM_ITEM: "INSTAGRAM_SCRAPE_MICROS_PER_ITEM",
BillingUnit.INSTAGRAM_COMMENT: "INSTAGRAM_SCRAPE_MICROS_PER_COMMENT",
BillingUnit.TIKTOK_VIDEO: "TIKTOK_MICROS_PER_VIDEO",
BillingUnit.TIKTOK_USER: "TIKTOK_MICROS_PER_USER",
BillingUnit.TIKTOK_COMMENT: "TIKTOK_MICROS_PER_COMMENT",
}
@ -51,6 +56,11 @@ _UNIT_NOUNS: dict[BillingUnit, str] = {
BillingUnit.GOOGLE_MAPS_REVIEW: "review",
BillingUnit.YOUTUBE_VIDEO: "video",
BillingUnit.YOUTUBE_COMMENT: "comment",
BillingUnit.INSTAGRAM_ITEM: "item",
BillingUnit.INSTAGRAM_COMMENT: "comment",
BillingUnit.TIKTOK_VIDEO: "video",
BillingUnit.TIKTOK_USER: "profile",
BillingUnit.TIKTOK_COMMENT: "comment",
}

5
surfsense_backend/app/capabilities/core/types.py
View file

@ -25,6 +25,11 @@ class BillingUnit(StrEnum):
GOOGLE_MAPS_REVIEW = "google_maps_review"
YOUTUBE_VIDEO = "youtube_video"
YOUTUBE_COMMENT = "youtube_comment"
INSTAGRAM_ITEM = "instagram_item"
INSTAGRAM_COMMENT = "instagram_comment"
TIKTOK_VIDEO = "tiktok_video"
TIKTOK_USER = "tiktok_user"
TIKTOK_COMMENT = "tiktok_comment"
class BillableInput(Protocol):

6
surfsense_backend/app/capabilities/instagram/__init__.py Normal file
View file

@ -0,0 +1,6 @@
"""``instagram.*`` namespace: platform-native Instagram data verbs."""
from __future__ import annotations
from app.capabilities.instagram.details import definition as _details # noqa: F401
from app.capabilities.instagram.scrape import definition as _scrape # noqa: F401

3
surfsense_backend/app/capabilities/instagram/details/__init__.py Normal file
View file

@ -0,0 +1,3 @@
"""``instagram.details`` verb: profile/hashtag/place URLs or search → metadata."""
from __future__ import annotations

23
surfsense_backend/app/capabilities/instagram/details/definition.py Normal file
View file

@ -0,0 +1,23 @@
"""``instagram.details`` capability registration (billed per item; see config
``INSTAGRAM_SCRAPE_MICROS_PER_ITEM``)."""
from __future__ import annotations
from app.capabilities.core import BillingUnit, Capability, register_capability
from app.capabilities.instagram.details.executor import build_details_executor
from app.capabilities.instagram.details.schemas import DetailsInput, DetailsOutput
INSTAGRAM_DETAILS = Capability(
name="instagram.details",
description=(
"Fetch Instagram profile, hashtag, or place metadata by URL or discovery "
"search. Each item carries a detailKind discriminator."
),
input_schema=DetailsInput,
output_schema=DetailsOutput,
executor=build_details_executor(),
billing_unit=BillingUnit.INSTAGRAM_ITEM,
docs_url="/docs/connectors/native/instagram",
)
register_capability(INSTAGRAM_DETAILS)

50
surfsense_backend/app/capabilities/instagram/details/executor.py Normal file
View file

@ -0,0 +1,50 @@
"""``instagram.details`` executor: verb input → scraper → detail items."""
from __future__ import annotations
from collections.abc import Awaitable, Callable
from app.capabilities.core import Executor
from app.capabilities.core.progress import emit_progress
from app.capabilities.instagram.details.schemas import DetailsInput, DetailsOutput
from app.exceptions import ForbiddenError
from app.proprietary.platforms.instagram import (
InstagramAccessBlockedError,
InstagramScrapeInput,
scrape_instagram,
)
ScrapeFn = Callable[..., Awaitable[list[dict]]]
def build_details_executor(scrape_fn: ScrapeFn | None = None) -> Executor:
"""Bind the executor to a scraper fn (defaults to the proprietary actor)."""
scrape_fn = scrape_fn or scrape_instagram
async def execute(payload: DetailsInput) -> DetailsOutput:
actor_input = InstagramScrapeInput(
resultsType="details",
directUrls=payload.urls,
search=",".join(payload.search_queries),
searchType=payload.search_type,
searchLimit=payload.search_limit,
)
emit_progress(
"starting",
"Resolving Instagram detail targets",
total=payload.max_items,
unit="item",
)
try:
items = await scrape_fn(actor_input, limit=payload.max_items)
except InstagramAccessBlockedError as exc:
raise ForbiddenError(
f"Instagram refused anonymous access: {exc}",
code="INSTAGRAM_ACCESS_BLOCKED",
) from exc
emit_progress(
"done", f"Scraped {len(items)} item(s)", current=len(items), unit="item"
)
return DetailsOutput(items=items)
return execute

77
surfsense_backend/app/capabilities/instagram/details/schemas.py Normal file
View file

@ -0,0 +1,77 @@
"""``instagram.details`` I/O contracts.
A lean surface over ``InstagramScrapeInput`` (``resultsType="details"``). Each
output item is a profile (``detailKind="profile"``, a SurfSense addition; every
other field mirrors the actor). Hashtag/place details are login-walled and
therefore unsupported.
"""
from __future__ import annotations
from typing import Literal
from pydantic import BaseModel, Field, model_validator
from app.capabilities.instagram.scrape.schemas import (
MAX_INSTAGRAM_ITEMS,
MAX_INSTAGRAM_SOURCES,
)
from app.proprietary.platforms.instagram import InstagramProfile
InstagramDetailItem = InstagramProfile
class DetailsInput(BaseModel):
urls: list[str] = Field(
default_factory=list,
max_length=MAX_INSTAGRAM_SOURCES,
description=(
"Profile URLs or bare profile IDs. Provide these OR search_queries."
),
)
search_queries: list[str] = Field(
default_factory=list,
max_length=MAX_INSTAGRAM_SOURCES,
description="Discovery keywords resolved to profiles. Provide these OR urls.",
)
search_type: Literal["profile", "user"] = Field(
default="profile",
description="Discovery kind (profile-only; hashtag/place are login-walled).",
)
search_limit: int = Field(
default=10,
ge=1,
le=MAX_INSTAGRAM_ITEMS,
description="Max discovered entities per query.",
)
max_items: int = Field(
default=10,
ge=1,
le=MAX_INSTAGRAM_ITEMS,
description="Max total detail items to return.",
)
@model_validator(mode="after")
def _exactly_one_source(self) -> DetailsInput:
if not self.urls and not self.search_queries:
raise ValueError("Provide at least one of 'urls' or 'search_queries'.")
if self.urls and self.search_queries:
raise ValueError(
"Provide 'urls' OR 'search_queries', not both (they cannot be combined)."
)
return self
@property
def estimated_units(self) -> int:
return self.max_items
class DetailsOutput(BaseModel):
items: list[InstagramDetailItem] = Field(
default_factory=list,
description="One profile detail item per resolved profile.",
)
@property
def billable_units(self) -> int:
return len(self.items)

3
surfsense_backend/app/capabilities/instagram/scrape/__init__.py Normal file
View file

@ -0,0 +1,3 @@
"""``instagram.scrape`` verb: Instagram URLs / search terms → posts, reels."""
from __future__ import annotations

23
surfsense_backend/app/capabilities/instagram/scrape/definition.py Normal file
View file

@ -0,0 +1,23 @@
"""``instagram.scrape`` capability registration (billed per item; see config
``INSTAGRAM_SCRAPE_MICROS_PER_ITEM``)."""
from __future__ import annotations
from app.capabilities.core import BillingUnit, Capability, register_capability
from app.capabilities.instagram.scrape.executor import build_scrape_executor
from app.capabilities.instagram.scrape.schemas import ScrapeInput, ScrapeOutput
INSTAGRAM_SCRAPE = Capability(
name="instagram.scrape",
description=(
"Scrape public Instagram posts or reels from profile/post/reel URLs, "
"or discover public profiles via search queries."
),
input_schema=ScrapeInput,
output_schema=ScrapeOutput,
executor=build_scrape_executor(),
billing_unit=BillingUnit.INSTAGRAM_ITEM,
docs_url="/docs/connectors/native/instagram",
)
register_capability(INSTAGRAM_SCRAPE)

57
surfsense_backend/app/capabilities/instagram/scrape/executor.py Normal file
View file

@ -0,0 +1,57 @@
"""``instagram.scrape`` executor: verb input → scraper → media items."""
from __future__ import annotations
from collections.abc import Awaitable, Callable
from app.capabilities.core import Executor
from app.capabilities.core.progress import emit_progress
from app.capabilities.instagram.scrape.schemas import ScrapeInput, ScrapeOutput
from app.exceptions import ForbiddenError
from app.proprietary.platforms.instagram import (
InstagramAccessBlockedError,
InstagramScrapeInput,
scrape_instagram,
)
ScrapeFn = Callable[..., Awaitable[list[dict]]]
def build_scrape_executor(scrape_fn: ScrapeFn | None = None) -> Executor:
"""Bind the executor to a scraper fn (defaults to the proprietary actor)."""
scrape_fn = scrape_fn or scrape_instagram
async def execute(payload: ScrapeInput) -> ScrapeOutput:
actor_input = InstagramScrapeInput(
resultsType=payload.result_type,
directUrls=payload.urls,
search=",".join(payload.search_queries),
searchType=payload.search_type,
resultsLimit=payload.max_per_target,
onlyPostsNewerThan=payload.newer_than,
skipPinnedPosts=payload.skip_pinned_posts,
addParentData=payload.add_parent_data,
)
emit_progress(
"starting",
"Resolving Instagram targets",
total=payload.max_items,
unit="item",
)
try:
items = await scrape_fn(actor_input, limit=payload.max_items)
except InstagramAccessBlockedError as exc:
# Anonymous-only scraper; a hard block can't be retried with creds.
raise ForbiddenError(
"Instagram requires a login for this request and SurfSense scrapes "
"anonymously. Provide a profile URL or handle via directUrls; "
"keyword/hashtag search needs an account and is unavailable. "
f"Details: {exc}",
code="INSTAGRAM_ACCESS_BLOCKED",
) from exc
emit_progress(
"done", f"Scraped {len(items)} item(s)", current=len(items), unit="item"
)
return ScrapeOutput(items=items)
return execute

103
surfsense_backend/app/capabilities/instagram/scrape/schemas.py Normal file
View file

@ -0,0 +1,103 @@
"""``instagram.scrape`` I/O contracts.
A lean, agent-friendly surface over ``InstagramScrapeInput``
(``app/proprietary/platforms/instagram``). The executor maps this to the full
scraper input; the scraper's ``InstagramMediaItem`` is reused verbatim as the
output element.
"""
from __future__ import annotations
from typing import Literal
from pydantic import BaseModel, Field, model_validator
from app.proprietary.platforms.instagram import InstagramMediaItem
MAX_INSTAGRAM_SOURCES = 20
"""Per-call cap on urls + search_queries: bounds a synchronous request's fan-out."""
MAX_INSTAGRAM_ITEMS = 100
"""Hard ceiling on items returned per call, regardless of the per-target caps."""
class ScrapeInput(BaseModel):
urls: list[str] = Field(
default_factory=list,
max_length=MAX_INSTAGRAM_SOURCES,
description=(
"Instagram URLs or bare profile IDs: profile, post (/p/), or reel "
"(/reel/). Hashtag/place URLs are unsupported (login-walled). "
"Provide these OR search_queries (never both)."
),
)
search_queries: list[str] = Field(
default_factory=list,
max_length=MAX_INSTAGRAM_SOURCES,
description=(
"Discovery keywords resolved to profiles via Google (IG's keyword "
"search is login-walled). Provide these OR urls (never both)."
),
)
search_type: Literal["profile", "user"] = Field(
default="profile",
description="Discovery kind (profile-only; hashtag/place are login-walled).",
)
result_type: Literal["posts", "reels"] = Field(
default="posts",
description="Which feed to return: 'posts' or 'reels'.",
)
newer_than: str | None = Field(
default=None,
description=(
"Only return posts newer than this: YYYY-MM-DD, ISO timestamp, or "
"relative ('1 day', '2 months'); UTC."
),
)
skip_pinned_posts: bool = Field(
default=False,
description="Exclude pinned posts (posts mode).",
)
max_per_target: int = Field(
default=10,
ge=1,
description="Max results per URL or per discovered target.",
)
max_items: int = Field(
default=10,
ge=1,
le=MAX_INSTAGRAM_ITEMS,
description="Max total items to return across all sources.",
)
add_parent_data: bool = Field(
default=False,
description="Attach a dataSource block to each feed item.",
)
@model_validator(mode="after")
def _exactly_one_source(self) -> ScrapeInput:
if not self.urls and not self.search_queries:
raise ValueError("Provide at least one of 'urls' or 'search_queries'.")
if self.urls and self.search_queries:
raise ValueError(
"Provide 'urls' OR 'search_queries', not both (they cannot be combined)."
)
return self
@property
def estimated_units(self) -> int:
"""Worst-case billable items for the pre-flight gate: ``max_items`` is a
hard cross-source ceiling (le=100), so no call can exceed it."""
return self.max_items
class ScrapeOutput(BaseModel):
items: list[InstagramMediaItem] = Field(
default_factory=list,
description="One media item per result (post/reel/mention), in emission order.",
)
@property
def billable_units(self) -> int:
"""One returned item = one billable unit."""
return len(self.items)

8
surfsense_backend/app/capabilities/tiktok/__init__.py Normal file
View file

@ -0,0 +1,8 @@
"""``tiktok.*`` namespace: platform-native TikTok data verbs."""
from __future__ import annotations
from app.capabilities.tiktok.comments import definition as _comments # noqa: F401
from app.capabilities.tiktok.scrape import definition as _scrape # noqa: F401
from app.capabilities.tiktok.trending import definition as _trending # noqa: F401
from app.capabilities.tiktok.user_search import definition as _user_search # noqa: F401

3
surfsense_backend/app/capabilities/tiktok/comments/__init__.py Normal file
View file

@ -0,0 +1,3 @@
"""``tiktok.comments``: scrape the public comment thread of TikTok videos."""
from __future__ import annotations

23
surfsense_backend/app/capabilities/tiktok/comments/definition.py Normal file
View file

@ -0,0 +1,23 @@
"""``tiktok.comments`` capability registration (billed per comment; see config
``TIKTOK_MICROS_PER_COMMENT``)."""
from __future__ import annotations
from app.capabilities.core import BillingUnit, Capability, register_capability
from app.capabilities.tiktok.comments.executor import build_comments_executor
from app.capabilities.tiktok.comments.schemas import CommentsInput, CommentsOutput
TIKTOK_COMMENTS = Capability(
name="tiktok.comments",
description=(
"Scrape the public comments of TikTok videos. Provide video URLs; "
"returns comment text, author, likes, and reply counts."
),
input_schema=CommentsInput,
output_schema=CommentsOutput,
executor=build_comments_executor(),
billing_unit=BillingUnit.TIKTOK_COMMENT,
docs_url="/docs/connectors/native/tiktok",
)
register_capability(TIKTOK_COMMENTS)

36
surfsense_backend/app/capabilities/tiktok/comments/executor.py Normal file
View file

@ -0,0 +1,36 @@
"""``tiktok.comments`` executor: video URLs -> scraper -> TikTok comment items."""
from __future__ import annotations
from collections.abc import Awaitable, Callable
from app.capabilities.core import Executor
from app.capabilities.core.progress import emit_progress
from app.capabilities.tiktok.comments.schemas import CommentsInput, CommentsOutput
from app.proprietary.platforms.tiktok import scrape_tiktok_comments
CommentsFn = Callable[..., Awaitable[list[dict]]]
def build_comments_executor(comments_fn: CommentsFn | None = None) -> Executor:
"""Bind the executor to a comments fn (defaults to the proprietary actor)."""
comments_fn = comments_fn or scrape_tiktok_comments
async def execute(payload: CommentsInput) -> CommentsOutput:
emit_progress(
"starting",
"Scraping TikTok comments",
total=payload.max_items,
unit="item",
)
items = await comments_fn(
payload.video_urls,
per_video=payload.comments_per_video,
limit=payload.max_items,
)
emit_progress(
"done", f"Scraped {len(items)} comment(s)", current=len(items), unit="item"
)
return CommentsOutput(items=items)
return execute

56
surfsense_backend/app/capabilities/tiktok/comments/schemas.py Normal file
View file

@ -0,0 +1,56 @@
"""``tiktok.comments`` I/O contracts.
URL-only: given TikTok video URLs, return each video's public comment thread.
Unlike profile-video/general-search feeds, ``/api/comment/list`` is served to
anonymous sessions once the comments panel opens, so this verb is reliable. Each
result is a :class:`CommentItem` (top-level comments; replies carry ``repliesToId``).
"""
from __future__ import annotations
from pydantic import BaseModel, Field
from app.capabilities.tiktok.scrape.schemas import (
MAX_TIKTOK_ITEMS,
MAX_TIKTOK_SOURCES,
)
from app.proprietary.platforms.tiktok import CommentItem
class CommentsInput(BaseModel):
video_urls: list[str] = Field(
min_length=1,
max_length=MAX_TIKTOK_SOURCES,
description="TikTok video URLs (/@<user>/video/<id>) to pull comments from.",
)
comments_per_video: int = Field(
default=20,
ge=1,
le=MAX_TIKTOK_ITEMS,
description="Max comments to return per video.",
)
max_items: int = Field(
default=20,
ge=1,
le=MAX_TIKTOK_ITEMS,
description="Max total comments to return across all videos.",
)
@property
def estimated_units(self) -> int:
"""Worst-case billable comments for the pre-flight gate: ``max_items`` is a
hard cross-video ceiling (le=100), so no call can exceed it."""
return self.max_items
class CommentsOutput(BaseModel):
items: list[CommentItem] = Field(
default_factory=list,
description="One item per comment returned, in emission order.",
)
@property
def billable_units(self) -> int:
"""One returned comment = one billable unit; ErrorItems (``errorCode`` set,
for bad URLs or empty/withheld videos) are surfaced but never charged."""
return sum(1 for item in self.items if not getattr(item, "errorCode", None))

1
surfsense_backend/app/capabilities/tiktok/scrape/__init__.py Normal file
View file

@ -0,0 +1 @@
"""``tiktok.scrape``: public TikTok videos over the browser-driven scraper."""

23
surfsense_backend/app/capabilities/tiktok/scrape/definition.py Normal file
View file

@ -0,0 +1,23 @@
"""``tiktok.scrape`` capability registration (billed per video; see config
``TIKTOK_MICROS_PER_VIDEO``)."""
from __future__ import annotations
from app.capabilities.core import BillingUnit, Capability, register_capability
from app.capabilities.tiktok.scrape.executor import build_scrape_executor
from app.capabilities.tiktok.scrape.schemas import ScrapeInput, ScrapeOutput
TIKTOK_SCRAPE = Capability(
name="tiktok.scrape",
description=(
"Scrape public TikTok videos. Use urls, profiles, or hashtags. To find "
"accounts by keyword, use tiktok.user_search."
),
input_schema=ScrapeInput,
output_schema=ScrapeOutput,
executor=build_scrape_executor(),
billing_unit=BillingUnit.TIKTOK_VIDEO,
docs_url="/docs/connectors/native/tiktok",
)
register_capability(TIKTOK_SCRAPE)

47
surfsense_backend/app/capabilities/tiktok/scrape/executor.py Normal file
View file

@ -0,0 +1,47 @@
"""``tiktok.scrape`` executor: verb input → scraper → TikTok video items."""
from __future__ import annotations
from collections.abc import Awaitable, Callable
from app.capabilities.core import Executor
from app.capabilities.core.progress import emit_progress
from app.capabilities.tiktok.scrape.schemas import ScrapeInput, ScrapeOutput
from app.exceptions import ForbiddenError
from app.proprietary.platforms.tiktok import (
TikTokAccessBlockedError,
TikTokScrapeInput,
scrape_tiktok,
)
ScrapeFn = Callable[..., Awaitable[list[dict]]]
def build_scrape_executor(scrape_fn: ScrapeFn | None = None) -> Executor:
"""Bind the executor to a scraper fn (defaults to the proprietary actor)."""
scrape_fn = scrape_fn or scrape_tiktok
async def execute(payload: ScrapeInput) -> ScrapeOutput:
actor_input = TikTokScrapeInput(
startUrls=[{"url": url} for url in payload.urls],
profiles=payload.profiles,
hashtags=payload.hashtags,
resultsPerPage=payload.results_per_page,
)
emit_progress(
"starting", "Resolving TikTok targets", total=payload.max_items, unit="item"
)
try:
items = await scrape_fn(actor_input, limit=payload.max_items)
except TikTokAccessBlockedError as exc:
# Anonymous-only scraper; a hard block can't be retried with creds.
raise ForbiddenError(
f"TikTok refused anonymous access: {exc}",
code="TIKTOK_ACCESS_BLOCKED",
) from exc
emit_progress(
"done", f"Scraped {len(items)} item(s)", current=len(items), unit="item"
)
return ScrapeOutput(items=items)
return execute

82
surfsense_backend/app/capabilities/tiktok/scrape/schemas.py Normal file
View file

@ -0,0 +1,82 @@
"""``tiktok.scrape`` I/O contracts.
A lean, agent-friendly surface over ``TikTokScrapeInput``
(``app/proprietary/platforms/tiktok``). The executor maps this to the full
scraper input; the scraper's ``TikTokVideoItem`` is reused verbatim as the
output element. Any TikTok URL kind (video, profile, hashtag, search) goes in
``urls``; ``profiles``/``hashtags`` are typed shortcuts. Keyword search is not a
video source here — use ``tiktok.user_search`` to find accounts by keyword.
"""
from __future__ import annotations
from pydantic import BaseModel, Field, model_validator
from app.proprietary.platforms.tiktok import TikTokVideoItem
MAX_TIKTOK_SOURCES = 20
"""Per-call cap on each source list: bounds a synchronous request's fan-out."""
MAX_TIKTOK_ITEMS = 100
"""Hard ceiling on items returned per call, regardless of the per-target count."""
class ScrapeInput(BaseModel):
urls: list[str] = Field(
default_factory=list,
max_length=MAX_TIKTOK_SOURCES,
description=(
"TikTok URLs to scrape: a video, a profile (/@<user>), a hashtag "
"(/tag/<name>), or a search URL. Provide these OR profiles/hashtags "
"(at least one source is required)."
),
)
profiles: list[str] = Field(
default_factory=list,
max_length=MAX_TIKTOK_SOURCES,
description="Profile usernames (with or without a leading '@').",
)
hashtags: list[str] = Field(
default_factory=list,
max_length=MAX_TIKTOK_SOURCES,
description="Hashtag names to scrape, without the leading '#'.",
)
results_per_page: int = Field(
default=10,
ge=1,
le=MAX_TIKTOK_ITEMS,
description="Max videos to pull per profile/hashtag target.",
)
max_items: int = Field(
default=10,
ge=1,
le=MAX_TIKTOK_ITEMS,
description="Max total items to return across all sources.",
)
@model_validator(mode="after")
def _require_a_source(self) -> ScrapeInput:
if not any((self.urls, self.profiles, self.hashtags)):
raise ValueError(
"Provide at least one of 'urls', 'profiles', or 'hashtags'."
)
return self
@property
def estimated_units(self) -> int:
"""Worst-case billable items for the pre-flight gate: ``max_items`` is a
hard cross-source ceiling (le=100), so no call can exceed it."""
return self.max_items
class ScrapeOutput(BaseModel):
items: list[TikTokVideoItem] = Field(
default_factory=list,
description="One item per video returned, in emission order.",
)
@property
def billable_units(self) -> int:
"""One returned video = one billable unit; ErrorItems (``errorCode`` set,
for blocked/empty targets) are surfaced but never charged."""
return sum(1 for item in self.items if not getattr(item, "errorCode", None))

3
surfsense_backend/app/capabilities/tiktok/trending/__init__.py Normal file
View file

@ -0,0 +1,3 @@
"""``tiktok.trending``: pull the current trending videos from the Explore feed."""
from __future__ import annotations

23
surfsense_backend/app/capabilities/tiktok/trending/definition.py Normal file
View file

@ -0,0 +1,23 @@
"""``tiktok.trending`` capability registration (billed per video on the shared
``TIKTOK_MICROS_PER_VIDEO`` meter)."""
from __future__ import annotations
from app.capabilities.core import BillingUnit, Capability, register_capability
from app.capabilities.tiktok.trending.executor import build_trending_executor
from app.capabilities.tiktok.trending.schemas import TrendingInput, TrendingOutput
TIKTOK_TRENDING = Capability(
name="tiktok.trending",
description=(
"Get the current trending TikTok videos from the Explore feed. No input "
"needed beyond how many to return."
),
input_schema=TrendingInput,
output_schema=TrendingOutput,
executor=build_trending_executor(),
billing_unit=BillingUnit.TIKTOK_VIDEO,
docs_url="/docs/connectors/native/tiktok",
)
register_capability(TIKTOK_TRENDING)

32
surfsense_backend/app/capabilities/tiktok/trending/executor.py Normal file
View file

@ -0,0 +1,32 @@
"""``tiktok.trending`` executor: Explore feed -> scraper -> TikTok video items."""
from __future__ import annotations
from collections.abc import Awaitable, Callable
from app.capabilities.core import Executor
from app.capabilities.core.progress import emit_progress
from app.capabilities.tiktok.trending.schemas import TrendingInput, TrendingOutput
from app.proprietary.platforms.tiktok import scrape_tiktok_trending
TrendingFn = Callable[..., Awaitable[list[dict]]]
def build_trending_executor(trending_fn: TrendingFn | None = None) -> Executor:
"""Bind the executor to a trending fn (defaults to the proprietary actor)."""
trending_fn = trending_fn or scrape_tiktok_trending
async def execute(payload: TrendingInput) -> TrendingOutput:
emit_progress(
"starting",
"Fetching TikTok trending videos",
total=payload.max_items,
unit="item",
)
items = await trending_fn(count=payload.max_items)
emit_progress(
"done", f"Fetched {len(items)} video(s)", current=len(items), unit="item"
)
return TrendingOutput(items=items)
return execute

41
surfsense_backend/app/capabilities/tiktok/trending/schemas.py Normal file
View file

@ -0,0 +1,41 @@
"""``tiktok.trending`` I/O contracts.
The Explore feed (``/api/explore/item_list``) is a single global feed of trending
videos, served to anonymous sessions. No source input is needed — just how many
to return. Each result reuses :class:`TikTokVideoItem`, so trending videos bill on
the same per-video meter as ``tiktok.scrape``.
"""
from __future__ import annotations
from pydantic import BaseModel, Field
from app.capabilities.tiktok.scrape.schemas import MAX_TIKTOK_ITEMS
from app.proprietary.platforms.tiktok import TikTokVideoItem
class TrendingInput(BaseModel):
max_items: int = Field(
default=20,
ge=1,
le=MAX_TIKTOK_ITEMS,
description="Max trending videos to return from the Explore feed.",
)
@property
def estimated_units(self) -> int:
"""Worst-case billable videos for the pre-flight gate (le=100 ceiling)."""
return self.max_items
class TrendingOutput(BaseModel):
items: list[TikTokVideoItem] = Field(
default_factory=list,
description="One item per trending video returned, in feed order.",
)
@property
def billable_units(self) -> int:
"""One returned video = one billable unit; an ErrorItem (``errorCode`` set,
for an empty/withheld feed) is surfaced but never charged."""
return sum(1 for item in self.items if not getattr(item, "errorCode", None))

3
surfsense_backend/app/capabilities/tiktok/user_search/__init__.py Normal file
View file

@ -0,0 +1,3 @@
"""``tiktok.user_search``: find public TikTok accounts by keyword."""
from __future__ import annotations

26
surfsense_backend/app/capabilities/tiktok/user_search/definition.py Normal file
View file

@ -0,0 +1,26 @@
"""``tiktok.user_search`` capability registration (billed per account; see config
``TIKTOK_MICROS_PER_USER``)."""
from __future__ import annotations
from app.capabilities.core import BillingUnit, Capability, register_capability
from app.capabilities.tiktok.user_search.executor import build_user_search_executor
from app.capabilities.tiktok.user_search.schemas import (
UserSearchInput,
UserSearchOutput,
)
TIKTOK_USER_SEARCH = Capability(
name="tiktok.user_search",
description=(
"Find public TikTok accounts by keyword. Returns profile metadata "
"(name, followers, bio, verification) per matching account."
),
input_schema=UserSearchInput,
output_schema=UserSearchOutput,
executor=build_user_search_executor(),
billing_unit=BillingUnit.TIKTOK_USER,
docs_url="/docs/connectors/native/tiktok",
)
register_capability(TIKTOK_USER_SEARCH)

39
surfsense_backend/app/capabilities/tiktok/user_search/executor.py Normal file
View file

@ -0,0 +1,39 @@
"""``tiktok.user_search`` executor: queries -> scraper -> TikTok profile items."""
from __future__ import annotations
from collections.abc import Awaitable, Callable
from app.capabilities.core import Executor
from app.capabilities.core.progress import emit_progress
from app.capabilities.tiktok.user_search.schemas import (
UserSearchInput,
UserSearchOutput,
)
from app.proprietary.platforms.tiktok import search_tiktok_users
SearchFn = Callable[..., Awaitable[list[dict]]]
def build_user_search_executor(search_fn: SearchFn | None = None) -> Executor:
"""Bind the executor to a search fn (defaults to the proprietary actor)."""
search_fn = search_fn or search_tiktok_users
async def execute(payload: UserSearchInput) -> UserSearchOutput:
emit_progress(
"starting",
"Searching TikTok accounts",
total=payload.max_items,
unit="item",
)
items = await search_fn(
payload.queries,
per_query=payload.results_per_query,
limit=payload.max_items,
)
emit_progress(
"done", f"Found {len(items)} account(s)", current=len(items), unit="item"
)
return UserSearchOutput(items=items)
return execute

56
surfsense_backend/app/capabilities/tiktok/user_search/schemas.py Normal file
View file

@ -0,0 +1,56 @@
"""``tiktok.user_search`` I/O contracts.
Account discovery over ``TikTok``'s Users tab. Where video/general search is
login-walled for anonymous sessions, ``/api/search/user`` returns public account
records, so this verb exposes the one reliably-unblocked search path. Each result
is a :class:`TikTokProfileItem` (the same shape the profile verb emits).
"""
from __future__ import annotations
from pydantic import BaseModel, Field
from app.capabilities.tiktok.scrape.schemas import (
MAX_TIKTOK_ITEMS,
MAX_TIKTOK_SOURCES,
)
from app.proprietary.platforms.tiktok import TikTokProfileItem
class UserSearchInput(BaseModel):
queries: list[str] = Field(
min_length=1,
max_length=MAX_TIKTOK_SOURCES,
description="Keywords to search for TikTok accounts (e.g. names, brands).",
)
results_per_query: int = Field(
default=10,
ge=1,
le=MAX_TIKTOK_ITEMS,
description="Max accounts to return per query.",
)
max_items: int = Field(
default=10,
ge=1,
le=MAX_TIKTOK_ITEMS,
description="Max total accounts to return across all queries.",
)
@property
def estimated_units(self) -> int:
"""Worst-case billable accounts for the pre-flight gate: ``max_items`` is a
hard cross-query ceiling (le=100), so no call can exceed it."""
return self.max_items
class UserSearchOutput(BaseModel):
items: list[TikTokProfileItem] = Field(
default_factory=list,
description="One item per account found, in emission order.",
)
@property
def billable_units(self) -> int:
"""One returned account = one billable unit; ErrorItems (``errorCode`` set,
for empty/withheld queries) are surfaced but never charged."""
return sum(1 for item in self.items if not getattr(item, "errorCode", None))

39
surfsense_backend/app/config/__init__.py
View file

@ -9,6 +9,11 @@ from chonkie import AutoEmbeddings, CodeChunker, RecursiveChunker
from dotenv import load_dotenv
from rerankers import Reranker
from app.config.embedding_settings import (
build_embedding_kwargs,
resolve_embedding_base_url,
)
# Get the base directory of the project
BASE_DIR = Path(__file__).resolve().parent.parent.parent
@ -714,6 +719,26 @@ class Config:
# Kept separate from the video rate so comments can be re-tuned toward the
# cheaper per-comment market ($0.40-2.00/1k) without touching video pricing.
YOUTUBE_MICROS_PER_COMMENT = int(os.getenv("YOUTUBE_MICROS_PER_COMMENT", "1500"))
INSTAGRAM_SCRAPE_MICROS_PER_ITEM = int(
os.getenv("INSTAGRAM_SCRAPE_MICROS_PER_ITEM", "3500")
)
# Kept separate from the item rate so comments can be re-tuned toward the
# cheaper per-comment market without touching post/reel pricing.
INSTAGRAM_SCRAPE_MICROS_PER_COMMENT = int(
os.getenv("INSTAGRAM_SCRAPE_MICROS_PER_COMMENT", "1500")
)
# Browser-driven listings make TikTok heavier per item than the API-backed
# video meter, so it sits a touch above YouTube's video rate.
TIKTOK_MICROS_PER_VIDEO = int(os.getenv("TIKTOK_MICROS_PER_VIDEO", "3500"))
# User search returns lighter account records (name/followers/bio), priced
# below the video meter to mirror the cheaper account-discovery market.
TIKTOK_MICROS_PER_USER = int(os.getenv("TIKTOK_MICROS_PER_USER", "2500"))
# Comments are the cheapest per-item TikTok data, matching the per-comment
# market (and YouTube's comment meter).
TIKTOK_MICROS_PER_COMMENT = int(os.getenv("TIKTOK_MICROS_PER_COMMENT", "1500"))
# Retry an empty listing draw on a fresh rotating IP. Set to 1 for a static
# proxy, where every retry re-hits the same exit.
TIKTOK_LISTING_MAX_ATTEMPTS = int(os.getenv("TIKTOK_LISTING_MAX_ATTEMPTS", "3"))
# Low-balance WARNING threshold (micro-USD). Surfaced by the quota service
# so the UI can nudge the user to top up / enable auto-reload. $0.50.
@ -937,16 +962,13 @@ class Config:
# Chonkie Configuration | Edit this to your needs
EMBEDDING_MODEL = os.getenv("EMBEDDING_MODEL")
EMBEDDING_BASE_URL = resolve_embedding_base_url()
# Azure OpenAI credentials from environment variables
AZURE_OPENAI_ENDPOINT = os.getenv("AZURE_OPENAI_ENDPOINT")
AZURE_OPENAI_API_KEY = os.getenv("AZURE_OPENAI_API_KEY")
# Pass Azure credentials to embeddings when using Azure OpenAI
embedding_kwargs = {}
if AZURE_OPENAI_ENDPOINT:
embedding_kwargs["azure_endpoint"] = AZURE_OPENAI_ENDPOINT
if AZURE_OPENAI_API_KEY:
embedding_kwargs["azure_api_key"] = AZURE_OPENAI_API_KEY
# Pass provider-specific settings to embeddings when supported.
embedding_kwargs = build_embedding_kwargs(embedding_model=EMBEDDING_MODEL)
embedding_model_instance = AutoEmbeddings.get_embeddings(
EMBEDDING_MODEL,
@ -1136,6 +1158,11 @@ class Config:
# round-trip => default FALSE to honor the "no speed regression" bar; flip on
# when leak-safety outweighs the marginal latency.
CRAWL_DNS_OVER_HTTPS = os.getenv("CRAWL_DNS_OVER_HTTPS", "FALSE").upper() == "TRUE"
# Promises an Xvfb display so the browser can run headful (TikTok's profile
# feed is empty to headless Chromium). Off keeps every browser headless.
CRAWL_HEADED_XVFB_ENABLED = (
os.getenv("CRAWL_HEADED_XVFB_ENABLED", "FALSE").upper() == "TRUE"
)
# Litellm TTS Configuration
TTS_SERVICE = os.getenv("TTS_SERVICE")

48
surfsense_backend/app/config/embedding_settings.py Normal file
View file

@ -0,0 +1,48 @@
import os
from collections.abc import Mapping
EMBEDDING_BASE_URL_ENV = "EMBEDDING_BASE_URL"
OLLAMA_EMBEDDING_BASE_URL_ENV = "OLLAMA_EMBEDDING_BASE_URL"
def _clean_env_value(value: str | None) -> str | None:
if value is None:
return None
stripped = value.strip()
return stripped or None
def resolve_embedding_base_url(environ: Mapping[str, str] | None = None) -> str | None:
"""Return the configured embedding endpoint, if any."""
environ = os.environ if environ is None else environ
return _clean_env_value(environ.get(EMBEDDING_BASE_URL_ENV)) or _clean_env_value(
environ.get(OLLAMA_EMBEDDING_BASE_URL_ENV)
)
def _supports_embedding_api_base(embedding_model: str | None) -> bool:
return (embedding_model or "").startswith("litellm://")
def build_embedding_kwargs(
environ: Mapping[str, str] | None = None,
*,
embedding_model: str | None = None,
) -> dict[str, str]:
"""Build keyword arguments for Chonkie's embedding provider."""
environ = os.environ if environ is None else environ
embedding_kwargs: dict[str, str] = {}
embedding_base_url = resolve_embedding_base_url(environ)
if embedding_base_url and _supports_embedding_api_base(embedding_model):
embedding_kwargs["api_base"] = embedding_base_url
azure_openai_endpoint = _clean_env_value(environ.get("AZURE_OPENAI_ENDPOINT"))
azure_openai_api_key = _clean_env_value(environ.get("AZURE_OPENAI_API_KEY"))
if azure_openai_endpoint:
embedding_kwargs["azure_endpoint"] = azure_openai_endpoint
if azure_openai_api_key:
embedding_kwargs["azure_api_key"] = azure_openai_api_key
return embedding_kwargs

150
surfsense_backend/app/proprietary/platforms/instagram/README.md Normal file
View file

@ -0,0 +1,150 @@
# Instagram scraper (anonymous)
Platform-native Instagram scraper. **Anonymous-only** and browser-free: every
flow stays on the cheap HTTP tier (`app.utils.proxy` + `scrapling`), and profile
discovery reuses the `google_search` platform (see below). It exposes a stable
public API whose input/output surface mirrors the public Instagram scraper actor
spec (same `resultsType` / `directUrls` / camelCase field names, additive
`extra="allow"` parity), so callers written against that surface work unchanged.
It is **not** wired into ingestion or Celery — the capability layer under
`app/capabilities/instagram/` turns these primitives into REST + agent + MCP
surfaces.
## Approach
Instagram's public web app exposes anonymous, logged-out data once a session
carries an anonymous `csrftoken` + `mid` cookie pair and the `x-ig-app-id` web
header:
> Warm an anonymous session with one plain GET to `www.instagram.com/` (mints
> `csrftoken` + `mid`), then GET the web endpoints through that same
> Chrome-impersonated, sticky-IP session. Rotate the residential IP + re-warm on
> a login wall (401/403), back off on 429.
Surfaces used:
| Flow | Surface | Extractor |
|---|---|---|
| profile / details | `api/v1/users/web_profile_info/?username=…` (JSON) | `parse_profile` |
| profile feed (posts/reels) | the media embedded in the same profile JSON | `parse_media` |
| single post / reel | `/p/<shortcode>/` (embedded mobile-v1 `PolarisMedia` JSON, og-meta fallback) | `parse_post` |
| profile discovery | Google `site:instagram.com <query>` | `resolve_url` |
All of these are richer than the core fields: the feed node and the single-post
relay blob both carry carousel children (`images`/`childPosts`), tagged users,
coauthor producers, location, product type, and pin state; `web_profile_info`
also carries related profiles. Comment **content** stays login-walled — only the
anonymous comment **count** (`commentsCount`) is exposed, so `firstComment` /
`latestComments` are intentionally absent from the item schema.
**Why anonymous-only is a hard constraint.** Live logged-out probes show that
Instagram walls the interesting endpoints for anyone without a `sessionid`
account cookie: `api/v1/tags/web_info/`, `api/v1/locations/web_info/`, the
comment thread API (`?__a=1`), and `web/search/topsearch/` all **302 to
`/accounts/login/`**. We cannot log in (see below), so hashtag feeds, place
feeds, comment scraping, and IG's native keyword search were **removed** — they
can only ever return a login wall. What survives is what a logged-out browser can
actually read: a profile's web info + its embedded recent media, and a public
post/reel page's embedded metadata.
## Anonymous only — no authentication, ever
No login, no `sessionid` account cookie, no app password. The only cookies held
are the anonymous `csrftoken` / `mid` minted by the warm-up. There is **no**
authenticate option in the input surface or the fetch layer, by design. A
persistent block after IP rotation surfaces as `InstagramAccessBlockedError`
(mirrors Reddit's `RedditAccessBlockedError`) rather than a silent empty result,
so the capability layer can map it to a `403 INSTAGRAM_ACCESS_BLOCKED`.
## Module map
| File | Responsibility |
|---|---|
| `__init__.py` | Public exports: `InstagramScrapeInput`, item models, `iter_instagram`, `scrape_instagram`, `InstagramAccessBlockedError`. |
| `schemas.py` | `InstagramScrapeInput` (`extra="allow"`, no auth fields) + optional-field item models (`InstagramMediaItem`, `InstagramProfile`) each with `to_output()`. |
| `fetch.py` | The core. Rotate-on-block sticky `_RotatingSession` + `_current_session` ContextVar + `warm_session` (csrftoken/mid) + `fetch_json` (JSON) / `fetch_html` (HTML) sharing one resilient `_fetch(path, params, extract)` loop. |
| `url_resolver.py` | Classify an Instagram URL → `profile`/`post`/`reel`; non-Instagram (and hashtag/place) → `None`. Strips `_u/`, `profilecard/`; story → profile. |
| `parsers.py` | Pure mapping (`parse_media`, `parse_profile`, `parse_post` [relay `PolarisMedia` JSON, og-meta fallback], `_edges`). I/O-free. |
| `scraper.py` | Orchestrator: `_media_flow`/`_details_flow`/`_discover` (+ `_discover_via_google`), `_targets`, `fan_out`, `iter_instagram`, `scrape_instagram`. |
## How it works
1. `iter_instagram` resolves `directUrls` (or runs a discovery `search` per the
comma-split queries) into targets and fans them out on a pool of warm proxy
sessions (`fan_out`, 8-way). Each worker opens one sticky-IP session and warms
`csrftoken`/`mid` once, reusing it across the sequential targets it pulls.
2. `resultsType` selects the flow: `posts`/`reels` → media items,
`details` → profile metadata. Media items de-dupe by `id` across targets.
- A **profile** target → `web_profile_info` JSON → `parse_media` over the
embedded recent-media edges (feed) or `parse_profile` (details).
- A **post/reel** target → `fetch_html("p/<code>/")` → `parse_post`, which
reads the embedded mobile-v1 `PolarisMedia` JSON (full fidelity) and falls
back to Open Graph meta only if that blob is absent. Numeric-ID post URLs are
skipped (the page keys on the shortCode).
3. `fetch_json` / `fetch_html` warm the session on first use, rotate the IP +
re-warm on 401/403, back off on 429, return `None` on 404, and raise
`InstagramAccessBlockedError` on a `/accounts/login/` redirect.
4. Parsers map raw web JSON/HTML to flat dicts; the orchestrator stamps
`scrapedAt` and applies `resultsLimit` / `onlyPostsNewerThan` as request-time
policy.
## Profile discovery (Google-backed)
Instagram's native keyword search is login-walled, so `_discover` resolves a
query that is a valid handle directly (`"messi"` → `instagram.com/messi/`) and
routes any other query (e.g. `"national geographic"`) through
`_discover_via_google`, which calls the `google_search` platform with
`site:instagram.com`, classifies each organic URL with `resolve_url`, keeps the
**profile** hits (discovery is profile-only), de-dupes, and caps at `searchLimit`.
Caveats:
- **Coupling**: Instagram depends on the `google_search` platform. The dependency
is one-directional and lives behind `_discover_via_google` so it stays testable.
- **Quality**: results reflect Google's index/ranking of `instagram.com`, not
IG's own relevance. This is discovery, not search parity.
## Observed limits & calibration caveats
- Anonymous web JSON/HTML is rate-limited per IP; the sticky-session pool keeps
each IP's request rate modest but a hot pool will still hit login walls — that's
the `InstagramAccessBlockedError` path, not a bug.
- `likesCount` is frequently withheld on anonymous responses (surfaces as `-1` or
absent upstream); treat it as best-effort.
- **Single-post extraction** reads the mobile-v1 `PolarisMedia` object embedded in
the public `/p/` document (og-meta is a lossy fallback). If Instagram strips both
for a given post (private, taken down, or a login interstitial), `parse_post`
returns `None` — an honest empty, never a fabricated item. ponytail: the
embedded-blob shape can drift; a live probe that dumps the raw HTML pins it (see
Testing) and any change is contained to `_find_media` / `parse_post`.
- The `$3.50 / 1k items` default meter assumes the proxy-bytes-per-item measured
on the reference targets; re-measure with the scale harness before high-volume use.
## Testing
- Offline unit tests: `tests/unit/platforms/instagram/` — `test_skeleton.py`
(schema + URL resolver), `test_parsers.py` (mapping incl. `parse_post`
relay-JSON/og shapes; fixture-pinned tests skip when the fixture is absent),
`test_discovery.py` (Google-backed profile discovery with a fake `scrape_serps`),
`test_fetch_resilience.py` (warm / rotate / backoff loop + fan-out with fake
sessions, no network), `test_budget.py` (fair-share caps + de-dup).
- Stress / accuracy harness (live, needs network + residential proxy):
`scripts/stress/stress_instagram_scraper.py` — `--mode live-discovery` (profile
discovery accuracy), `--mode probe-post` (dumps a real anonymous `/p/` payload
to `fixtures/post.json` and shows what `parse_post` extracted), `--mode
probe-mentions` (settles that the tagged/`mentions` feed is login-walled), and
`--mode accuracy` (field coverage across the profile + single-post flows).
```bash
cd surfsense_backend
uv run pytest tests/unit/platforms/instagram/
# Live single-post probe: confirms /p/ is anonymously extractable + pins the shape
uv run python scripts/stress/stress_instagram_scraper.py --mode probe-post \
--post https://www.instagram.com/p/<shortcode>/
```
## TODO / out of scope (v1)
- Deep feed pagination past the first web page of profile media (GraphQL cursor
doc-id).
- Sticky-IP provider parity (same `__sid` caveat as the Reddit sibling).

18
surfsense_backend/app/proprietary/platforms/instagram/__init__.py Normal file
View file

@ -0,0 +1,18 @@
"""Platform-native Instagram scraper (anonymous, no browser)."""
from .fetch import InstagramAccessBlockedError
from .schemas import (
InstagramMediaItem,
InstagramProfile,
InstagramScrapeInput,
)
from .scraper import iter_instagram, scrape_instagram
__all__ = [
"InstagramAccessBlockedError",
"InstagramMediaItem",
"InstagramProfile",
"InstagramScrapeInput",
"iter_instagram",
"scrape_instagram",
]

403
surfsense_backend/app/proprietary/platforms/instagram/fetch.py Normal file
View file

@ -0,0 +1,403 @@
"""Proxy-aware fetch seam for the Instagram scraper (no browser).
All network I/O flows through :func:`fetch_json` and always egresses through the
residential proxy (a direct hit would expose and risk-block the server IP).
Instagram's public web app exposes anonymous JSON endpoints that a logged-out
browser calls, guarded by the ``X-IG-App-ID`` web app id and a warmed
``csrftoken``/``mid`` cookie pair:
warm one anonymous session (plain GET to ``www.instagram.com/`` mints
``csrftoken``/``mid``), then GET the ``api/v1/*/web_info`` /
``web_profile_info`` endpoints through that same Chrome-impersonated,
sticky-IP session with the ``X-IG-App-ID`` header.
This is a direct port of ``../reddit/fetch.py``'s rotate-on-block sticky-session
pattern (``_RotatingSession`` + ``_current_session`` ContextVar +
``open_proxy_holder``/``bind_proxy_holder``/``proxy_session``), with an
Instagram-specific :func:`warm_session` and header set.
Honest ceiling: anonymous Instagram access is the most hostile of our platforms.
Login walls appear as 401/403 and rotate the exit IP; 429 backs off on the same
IP. Observed per-IP/session limits are documented in ``README.md``; the safe
``_FANOUT_CONCURRENCY`` is deliberately low. ponytail: the pacing/rotation
constants are calibrated to residential exits and may need retuning per pool —
watch for 401/403/429 log spam and adjust.
"""
from __future__ import annotations
import asyncio
import json
import logging
import random
import time
from collections.abc import Callable
from contextlib import asynccontextmanager, suppress
from contextvars import ContextVar
from datetime import UTC, datetime
from typing import Any
from urllib.parse import urlencode
from scrapling.fetchers import AsyncFetcher, FetcherSession
from app.utils.proxy import get_proxy_url
logger = logging.getLogger(__name__)
class InstagramAccessBlockedError(RuntimeError):
"""Raised when every rotated IP is refused anonymous access.
This is the Instagram login-wall branch: after warming and rotating, the
exit IPs still 401/403. We are anonymous-only and cannot log in, so instead
of silently returning nothing we surface it as a clear error (mirrors
reddit's ``RedditAccessBlockedError`` / google_maps' ``SignInRequiredError``).
The executor turns it into a 403 for REST callers.
"""
# Per-flow proxy session, set by ``bind_proxy_holder`` around one continuation
# chain. Reusing one keep-alive connection pins a single sticky exit IP so the
# warmed ``csrftoken``/``mid`` cookies (bound to that IP) stay valid across the
# warm-up + every subsequent web-endpoint fetch. A ContextVar keeps each
# concurrent fan-out flow on its own session/IP without threading a param
# through every call.
_current_session: ContextVar[_RotatingSession | None] = ContextVar(
"instagram_proxy_session", default=None
)
# 401/403 => this IP hit the login wall; rotate to a fresh one and re-warm.
# 429 => rate limited; back off on the SAME IP (rotating wouldn't help and burns
# the pool).
_ROTATE_STATUSES = frozenset({401, 403})
_BACKOFF_STATUS = 429
_MAX_ROTATIONS = 3
_MAX_BACKOFFS = 4
_BACKOFF_BASE_S = 5.0
# Instagram 429s hard on bursts. Pace each sticky session so a fast IP can't
# burst past the per-IP threshold. ponytail: 1.5s is tuned to residential exits;
# a pool with a stricter per-IP cap may need it raised — watch for 429 log spam.
_MIN_INTERVAL_S = 1.5
_PACE_JITTER_S = 0.5
# A healthy fetch lands in ~1-2s; cap at 15s so a dead sticky IP costs one
# bounded wait, then the timeout falls into the generic exception branch of
# fetch_json and rotates to a fresh IP — same treatment as a 403.
_REQUEST_TIMEOUT_S = 15.0
# The anonymous web app id every logged-out instagram.com XHR carries. Without
# it the api/v1/*/web_info and web_profile_info endpoints 403 outright.
_IG_APP_ID = "936619743392459"
_HEADERS = {
"Accept-Language": "en-US,en;q=0.9",
"X-IG-App-ID": _IG_APP_ID,
"X-Requested-With": "XMLHttpRequest",
"Referer": "https://www.instagram.com/",
}
# A plain GET to the home page mints the anonymous csrftoken/mid cookie pair.
_WARM_URL = "https://www.instagram.com/"
_BASE = "https://www.instagram.com"
_CSRF_COOKIE = "csrftoken"
# Soft login wall: instead of a 401/403, IG answers an api/v1/* request with a
# 302 to /accounts/login/ that the impersonated client follows to a 200 login
# page. The status is 200 but the body is login HTML, so this evades the
# status-code rotate check — detect it via the response's final URL and treat
# it exactly like a 403.
_LOGIN_PATH = "/accounts/login"
def now_iso() -> str:
"""UTC timestamp in the millisecond ISO shape used by scraper output."""
return datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"
def _response_cookie_names(page: Any) -> set[str]:
"""Cookie names set by a response (best-effort across scrapling shapes)."""
cookies = getattr(page, "cookies", None)
if isinstance(cookies, dict):
return set(cookies.keys())
return set()
def _parse_json(page: Any) -> Any | None:
"""Parse a scrapling response body into JSON, or ``None``.
Prefers ``page.json()``; falls back to ``json.loads`` on the raw body when
the impersonated response hands back text.
"""
fn = getattr(page, "json", None)
if callable(fn):
with suppress(Exception):
return fn()
for attr in ("body", "text"):
val = getattr(page, attr, None)
if isinstance(val, bytes):
val = val.decode("utf-8", "replace")
if isinstance(val, str) and val.strip():
with suppress(Exception):
return json.loads(val)
return None
return None
def _page_text(page: Any) -> str | None:
"""Best-effort HTML/text body of a scrapling response, or ``None``."""
for attr in ("text", "body", "content"):
val = getattr(page, attr, None)
if callable(val):
with suppress(Exception):
val = val()
if isinstance(val, bytes):
val = val.decode("utf-8", "replace")
if isinstance(val, str) and val.strip():
return val
return None
def _is_login_redirect(page: Any) -> bool:
"""True if IG redirected this request to the anonymous login wall.
A soft block: the final URL lands on ``/accounts/login/`` (served 200), so
the status check never fires. Best-effort — returns ``False`` when the
response exposes no URL.
"""
final = getattr(page, "url", None)
return isinstance(final, str) and _LOGIN_PATH in final
def _build_url(path: str, params: dict[str, Any] | None) -> str:
"""Absolute URL for an instagram.com path (accepts already-absolute URLs)."""
base = path if path.startswith("http") else f"{_BASE}/{path.strip('/')}/"
if not params:
return base
qs = urlencode({k: v for k, v in params.items() if v is not None})
sep = "&" if "?" in base else "?"
return f"{base}{sep}{qs}" if qs else base
class _RotatingSession:
"""Owns one live ``FetcherSession`` (sticky IP) and can swap it for a fresh one.
``rotate()`` closes the current keep-alive connection and opens a new one, so
the rotating gateway hands out a different residential exit IP. Because the
warmed cookies bind to the exit IP, ``rotate()`` also drops the warmed state
— the next fetch re-warms on the new IP. Used sequentially within a single
flow (never shared across concurrent tasks), so no locking is needed.
``session`` is ``None`` only when no proxy is configured.
"""
def __init__(self) -> None:
self._cm: Any | None = None
self.session: Any | None = None
self.rotations = 0
self.warmed = False
self._last_at = 0.0
async def _open(self) -> None:
proxy = get_proxy_url()
self.warmed = False
if proxy is None:
self._cm = self.session = None
return
self._cm = FetcherSession(
proxy=proxy,
stealthy_headers=True,
impersonate="chrome",
timeout=_REQUEST_TIMEOUT_S,
)
self.session = await self._cm.__aenter__()
async def close(self) -> None:
if self._cm is not None:
with suppress(Exception): # best-effort teardown
await self._cm.__aexit__(None, None, None)
self._cm = self.session = None
async def rotate(self) -> Any | None:
"""Drop the current IP and connect through a fresh one. Returns new session."""
await self.close()
self.rotations += 1
await self._open()
logger.info("[instagram] rotated proxy session (rotation #%d)", self.rotations)
return self.session
async def pace(self) -> None:
"""Sleep to hold this sticky IP under Instagram's per-IP rate threshold."""
wait = _MIN_INTERVAL_S - (time.monotonic() - self._last_at)
if wait > 0:
await asyncio.sleep(wait + random.uniform(0, _PACE_JITTER_S))
self._last_at = time.monotonic()
async def open_proxy_holder() -> _RotatingSession:
"""Open a warm rotate-on-block session holder (caller owns ``close()``)."""
holder = _RotatingSession()
await holder._open()
return holder
@asynccontextmanager
async def bind_proxy_holder(holder: _RotatingSession):
"""Route this task's fetches through ``holder`` for the enclosed block.
Does NOT close the holder — enables pooling warm sessions across sequential
jobs so each job skips the proxy handshake AND the cookie warm-up.
"""
token = _current_session.set(holder)
try:
yield holder
finally:
_current_session.reset(token)
@asynccontextmanager
async def proxy_session():
"""Open one reused, rotate-on-block proxy session for a continuation chain."""
holder = await open_proxy_holder()
try:
async with bind_proxy_holder(holder):
yield holder
finally:
await holder.close()
async def warm_session(session: Any) -> bool:
"""Mint anonymous ``csrftoken``/``mid`` cookies on a freshly opened session.
Returns ``True`` when a ``csrftoken`` was issued (the session can now reach
the web endpoints), else ``False`` (caller rotates the IP and retries).
Takes an already-open ``session`` (never constructs one) so tests can drive
warm/rotate deterministically with a fake session, exactly like the reddit
sibling's fetch-resilience tests.
"""
seen: set[str] = set()
with suppress(Exception):
page = await session.get(_WARM_URL, headers=_HEADERS)
seen |= _response_cookie_names(page)
return _CSRF_COOKIE in seen
async def _get_page(session: Any, url: str) -> Any:
"""GET through the warmed sticky session, or a one-shot proxied fetch."""
if session is not None:
return await session.get(url, headers=_HEADERS)
return await AsyncFetcher.get(
url,
headers=_HEADERS,
proxy=get_proxy_url(),
stealthy_headers=True,
timeout=_REQUEST_TIMEOUT_S,
)
async def fetch_json(path: str, params: dict[str, Any] | None = None) -> Any | None:
"""GET an Instagram web endpoint through a warmed session; parse JSON.
Returns parsed JSON (dict or list), or ``None`` on 404 / non-block failure.
See :func:`_fetch` for the warm/rotate/backoff resilience contract.
"""
return await _fetch(path, params, _parse_json)
async def fetch_html(path: str, params: dict[str, Any] | None = None) -> str | None:
"""GET an Instagram web page through a warmed session; return its HTML text.
Same warm/rotate/backoff resilience as :func:`fetch_json` (a login-wall
redirect still raises :class:`InstagramAccessBlockedError`), but hands back
the raw HTML body for the pages that embed their data in the document
(``/p/<shortcode>/`` embedded PolarisMedia JSON / og-meta) instead of a JSON
XHR endpoint.
"""
return await _fetch(path, params, _page_text)
async def _fetch(
path: str,
params: dict[str, Any] | None,
extract: Callable[[Any], Any | None],
) -> Any | None:
"""GET an Instagram web endpoint through a warmed HTTP session.
Applies ``extract`` to the 200 response (JSON parse or HTML text); returns
``None`` on 404 / non-block failure. Warms cookies once per session; rotates
the residential IP and re-warms on 401/403; backs off on 429. Raises
:class:`InstagramAccessBlockedError` only when every rotated IP refuses
anonymous access (the login-wall branch, which we cannot satisfy).
"""
holder = _current_session.get()
if holder is None:
# No bound session (e.g. a direct call outside fan_out): open a
# short-lived warmed session for this one fetch, then tear it down.
async with proxy_session():
return await _fetch(path, params, extract)
url = _build_url(path, params)
attempt = 0
backoffs = 0
while True:
session = holder.session
try:
if session is not None and not holder.warmed:
warmed_ok = await warm_session(session)
holder.warmed = True # attempted; don't re-warm this IP
if not warmed_ok:
if attempt < _MAX_ROTATIONS:
attempt += 1
await holder.rotate()
continue
raise InstagramAccessBlockedError(
f"could not warm session after {attempt} IP rotations for {path}"
)
await holder.pace()
page = await _get_page(session, url)
status = page.status
# Endpoint-level login wall (302 -> /accounts/login/, served as 200
# login HTML): fail fast, do NOT rotate. Unlike the per-IP 401/403
# below — which recovers on a fresh exit IP, so it still rotates —
# every rotated IP hits this same wall (observed live), so rotating
# only burns the pool and re-warms for an unwinnable block. Raising
# (vs returning None) keeps a blocked target distinguishable from an
# empty one; fan_out swallows it per-target for partial results.
if _is_login_redirect(page):
raise InstagramAccessBlockedError(
f"Instagram login wall on {path} (endpoint requires auth)"
)
if status == 200:
return extract(page)
if status == 404:
return None
if status == _BACKOFF_STATUS and backoffs < _MAX_BACKOFFS:
backoffs += 1
delay = _BACKOFF_BASE_S * (2 ** (backoffs - 1))
logger.warning("[instagram] 429 on %s; backing off %.1fs", path, delay)
await asyncio.sleep(delay + random.uniform(0, 1))
continue
if status in _ROTATE_STATUSES:
# Bare 401/403: a per-IP block that a fresh exit IP recovers, so
# rotate and re-warm. (The endpoint-level auth wall is caught by
# the login-redirect branch above and fails fast without rotating.)
if attempt < _MAX_ROTATIONS:
attempt += 1
await holder.rotate()
continue
raise InstagramAccessBlockedError(
f"Instagram refused {path} on {attempt} rotated IPs ({status})"
)
logger.warning("[instagram] GET %s returned %s", path, status)
return None
except InstagramAccessBlockedError:
raise
except Exception as e:
logger.warning("[instagram] GET %s failed: %s", path, e)
if attempt < _MAX_ROTATIONS:
attempt += 1
await holder.rotate()
continue
return None

567
surfsense_backend/app/proprietary/platforms/instagram/parsers.py Normal file
View file

@ -0,0 +1,567 @@
"""Pure JSON -> item mapping for the Instagram scraper.
Framework-agnostic and I/O-free so it can be unit-tested against captured
fixtures. Every function takes a raw Instagram web-JSON node and returns a plain
dict shaped like the public actor spec — no network, no proxy, no ``scrapedAt``
stamp (the orchestrator adds provenance so these stay deterministic under
fixture tests).
Instagram's web JSON nests media under GraphQL-style ``edge_*`` containers
(``edge_media_to_caption``, ``edge_liked_by``, ...) with ``taken_at_timestamp``
epoch seconds. These parsers flatten that into the actor's camelCase item shape.
Fields the anonymous endpoints don't expose are left unset (``None``/``[]``) so
parity is additive.
"""
from __future__ import annotations
import html
import json
import re
from datetime import UTC, datetime
from typing import Any
_BASE = "https://www.instagram.com"
_HASHTAG_RE = re.compile(r"#(\w+)")
# Instagram handles are letters/digits/period/underscore but never start or end
# with a period, so anchor both ends to alphanumerics/underscore — otherwise
# trailing sentence punctuation ("@hulu.") leaks into the handle.
_MENTION_RE = re.compile(r"@([A-Za-z0-9_](?:[A-Za-z0-9_.]*[A-Za-z0-9_])?)")
_TYPE_MAP = {
"GraphImage": "Image",
"GraphVideo": "Video",
"GraphSidecar": "Sidecar",
"XDTGraphImage": "Image",
"XDTGraphVideo": "Video",
"XDTGraphSidecar": "Sidecar",
}
# Mobile v1 ``media_type``: 1 = image, 2 = video, 8 = carousel/sidecar. Used by
# the single-post relay parser (the embedded PolarisMedia blob uses this int, not
# the GraphQL ``__typename`` the profile feed uses).
_MEDIA_TYPE = {1: "Image", 2: "Video", 8: "Sidecar"}
def _int(value: Any) -> int | None:
"""Coerce to int, or ``None`` (never coerces bools)."""
if isinstance(value, bool):
return None
if isinstance(value, int):
return value
if isinstance(value, float):
return int(value)
return None
def _utc_from_sec(value: Any) -> str | None:
"""Epoch seconds -> millisecond ISO string, or ``None``."""
if not isinstance(value, int | float) or isinstance(value, bool):
return None
dt = datetime.fromtimestamp(float(value), tz=UTC)
return dt.strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"
def _edge_count(node: dict[str, Any], key: str) -> int | None:
"""``node[key].count`` for a GraphQL ``edge_*`` container."""
container = node.get(key)
if isinstance(container, dict):
return _int(container.get("count"))
return None
def _edges(container: Any) -> list[dict[str, Any]]:
"""``container.edges[].node`` list for a GraphQL ``edge_*`` container."""
if not isinstance(container, dict):
return []
out = []
for edge in container.get("edges") or []:
node = edge.get("node") if isinstance(edge, dict) else None
if not isinstance(node, dict):
continue
out.append(node)
return out
def _caption_text(node: dict[str, Any]) -> str | None:
"""First caption edge's text (web feed) or a flat ``caption`` fallback."""
edges = _edges(node.get("edge_media_to_caption"))
if edges:
text = edges[0].get("text")
if isinstance(text, str):
return text
cap = node.get("caption")
if isinstance(cap, dict):
cap = cap.get("text")
if isinstance(cap, str):
return cap
return None
def _likes_count(node: dict[str, Any]) -> int | None:
"""Like count. ``-1`` (creator hid it) is passed through, never coerced."""
for key in ("edge_liked_by", "edge_media_preview_like"):
count = _edge_count(node, key)
if count is not None:
return count
return _int(node.get("like_count"))
def _shortcode(node: dict[str, Any]) -> str | None:
code = node.get("shortcode") or node.get("code")
if isinstance(code, str):
return code
return None
def _user_ref(user: Any) -> dict[str, Any] | None:
"""A trimmed public-user dict (tagged users / coauthor producers), or None.
Normalizes the two anonymous dialects: the profile feed nests the handle
under ``edge_media_to_tagged_user...node.user`` / ``coauthor_producers`` while
the single-post relay blob uses ``usertags.in[].user`` — both carry the same
scalar user fields, so this trims them to one shape.
"""
if not isinstance(user, dict):
return None
ref = {
"username": user.get("username"),
"fullName": user.get("full_name"),
"id": user.get("id") or user.get("pk"),
"isVerified": user.get("is_verified"),
"profilePicUrl": user.get("profile_pic_url"),
}
return ref if ref["username"] or ref["id"] else None
def _iv2_url(iv2: Any) -> str | None:
"""First candidate URL from a mobile ``image_versions2`` container, or None."""
if isinstance(iv2, dict):
cands = iv2.get("candidates")
if isinstance(cands, list) and cands and isinstance(cands[0], dict):
url = cands[0].get("url")
return url if isinstance(url, str) else None
return None
def _location_ref(loc: Any) -> tuple[str | None, str | None]:
"""``(name, id)`` from a location node, or ``(None, None)``."""
if isinstance(loc, dict):
lid = loc.get("id") or loc.get("pk")
return loc.get("name"), (str(lid) if lid is not None else None)
return None, None
def _feed_child(node: dict[str, Any]) -> dict[str, Any]:
"""Map a profile-feed ``edge_sidecar_to_children`` child to a childPost dict."""
dims = node.get("dimensions") if isinstance(node.get("dimensions"), dict) else {}
is_video = bool(node.get("is_video"))
return {
"id": node.get("id"),
"shortCode": node.get("shortcode"),
"type": "Video" if is_video else "Image",
"displayUrl": node.get("display_url"),
"videoUrl": node.get("video_url") if is_video else None,
"alt": node.get("accessibility_caption"),
"dimensionsHeight": _int(dims.get("height")),
"dimensionsWidth": _int(dims.get("width")),
}
def _relay_child(node: dict[str, Any]) -> dict[str, Any]:
"""Map a single-post relay ``carousel_media`` child to a childPost dict."""
mt = node.get("media_type")
vv = node.get("video_versions")
video_url = (
vv[0].get("url")
if isinstance(vv, list) and vv and isinstance(vv[0], dict)
else None
)
is_video = mt == 2 or bool(video_url)
return {
"id": node.get("id"),
"shortCode": node.get("code"),
"type": _MEDIA_TYPE.get(mt) or ("Video" if is_video else "Image"),
"displayUrl": _iv2_url(node.get("image_versions2")) or node.get("display_uri"),
"videoUrl": video_url,
"alt": node.get("accessibility_caption"),
"dimensionsHeight": _int(node.get("original_height")),
"dimensionsWidth": _int(node.get("original_width")),
}
def parse_media(node: dict[str, Any]) -> dict[str, Any]:
"""Map a raw timeline/feed media node to a flat media item dict."""
code = _shortcode(node)
caption = _caption_text(node)
typename = node.get("__typename")
owner = node.get("owner") if isinstance(node.get("owner"), dict) else {}
dims = node.get("dimensions") if isinstance(node.get("dimensions"), dict) else {}
is_video = bool(node.get("is_video"))
children = _edges(node.get("edge_sidecar_to_children"))
tagged = [
ref
for n in _edges(node.get("edge_media_to_tagged_user"))
if (ref := _user_ref(n.get("user")))
]
coauthors = [
ref for c in (node.get("coauthor_producers") or []) if (ref := _user_ref(c))
]
loc_name, loc_id = _location_ref(node.get("location"))
return {
"id": node.get("id"),
"type": _TYPE_MAP.get(typename) or ("Video" if is_video else "Image"),
"shortCode": code,
"caption": caption,
"hashtags": _HASHTAG_RE.findall(caption) if caption else [],
"mentions": _MENTION_RE.findall(caption) if caption else [],
"url": f"{_BASE}/p/{code}/" if code else None,
"commentsCount": _edge_count(node, "edge_media_to_comment")
or _int(node.get("comment_count")),
"dimensionsHeight": _int(dims.get("height")),
"dimensionsWidth": _int(dims.get("width")),
"displayUrl": node.get("display_url"),
"images": [c.get("display_url") for c in children if c.get("display_url")],
"childPosts": [_feed_child(c) for c in children],
"videoUrl": node.get("video_url") if is_video else None,
"alt": node.get("accessibility_caption"),
"likesCount": _likes_count(node),
"videoViewCount": _int(node.get("video_view_count")) if is_video else None,
"videoDuration": node.get("video_duration") if is_video else None,
"timestamp": _utc_from_sec(node.get("taken_at_timestamp")),
"ownerUsername": owner.get("username"),
"ownerId": owner.get("id") or node.get("owner_id"),
"ownerFullName": owner.get("full_name"),
"isPinned": bool(node.get("pinned_for_users")),
"productType": node.get("product_type"),
"paidPartnership": node.get("is_paid_partnership"),
"taggedUsers": tagged,
"coauthorProducers": coauthors,
"musicInfo": node.get("clips_music_attribution_info"),
"locationName": loc_name,
"locationId": loc_id,
"isCommentsDisabled": node.get("comments_disabled"),
}
def parse_profile(user: dict[str, Any]) -> dict[str, Any]:
"""Map a raw ``web_profile_info`` ``data.user`` to a flat profile item dict."""
username = user.get("username")
latest = [parse_media(n) for n in _edges(user.get("edge_owner_to_timeline_media"))]
related = [
ref for n in _edges(user.get("edge_related_profiles")) if (ref := _user_ref(n))
]
return {
"detailKind": "profile",
"id": user.get("id"),
"username": username,
"url": f"{_BASE}/{username}/" if username else None,
"fullName": user.get("full_name"),
"biography": user.get("biography"),
"externalUrl": user.get("external_url"),
"followersCount": _edge_count(user, "edge_followed_by"),
"followsCount": _edge_count(user, "edge_follow"),
"postsCount": _edge_count(user, "edge_owner_to_timeline_media"),
"highlightReelCount": _int(user.get("highlight_reel_count")),
"igtvVideoCount": _edge_count(user, "edge_felix_video_timeline"),
"isBusinessAccount": user.get("is_business_account"),
"businessCategoryName": user.get("business_category_name"),
"private": user.get("is_private"),
"verified": user.get("is_verified"),
"profilePicUrl": user.get("profile_pic_url"),
"profilePicUrlHD": user.get("profile_pic_url_hd"),
"relatedProfiles": related,
"latestPosts": latest,
}
# Anonymous single-post extraction (/p/<shortcode>/, /reel/<shortcode>/) --------
#
# Instagram serves logged-out visitors the post's full metadata inside the
# document itself, not via a JSON XHR (the ``?__a=1`` API 404s / login-walls for
# anonymous callers). Two anonymous surfaces carry it, in order of fidelity:
# 1. An inline ``<script type="application/json">`` block embedding the mobile
# v1 ``PolarisMedia`` object (pk, taken_at, media_type, like/comment counts,
# caption, carousel_media, usertags, coauthor_producers, location, ...). This
# is the same shape the app's private API returns and is the real source.
# 2. Open Graph ``<meta property="og:*">`` — a lossy fallback (og:description
# packs "N likes, M comments - author on DATE: caption"). Used only if the
# relay blob is somehow absent (e.g. a layout change).
# Live logged-out probes show NO ld+json on these pages, so it isn't parsed.
# ponytail: pinned to these two surfaces — if the relay shape drifts, update
# ``_find_media``; the wiring in scraper._media_flow stays the same.
_APP_JSON_RE = re.compile(
r'<script type="application/json"[^>]*>(.*?)</script>', re.DOTALL
)
_OG_RE = re.compile(r'<meta\s+property="og:([^"]+)"\s+content="([^"]*)"', re.IGNORECASE)
# og tags are the fallback source (used only when the relay blob is absent). They
# follow a fixed English shape because the fetch layer pins Accept-Language en-US:
# og:description = "{likes} likes, {comments} comments - {username} on {Month D, YYYY}: "{caption}""
# og:title = "{fullName} on Instagram: "{caption}""
# Each field is matched independently and guarded so an unrecognised shape (hidden
# likes, a non-English locale that slipped the header, a format change) degrades
# to None rather than crashing or polluting the caption.
_OG_COUNTS_RE = re.compile(
r"([\d.,]+)\s+likes?,\s*([\d.,]+)\s+comments?", re.IGNORECASE
)
# The username sits after the counts' " - " and before " on {date}:"; the date is
# anchored to the English "Month D, YYYY" so a caption containing " on " or ":"
# can't be mistaken for the prefix.
_OG_OWNER_DATE_RE = re.compile(
r"-\s+([^-\n]+?)\s+on\s+([A-Z][a-z]+ \d{1,2}, \d{4}):", re.DOTALL
)
# og:title is the cleaner caption source (no counts/date prefix): the caption is
# everything after "<fullName> on Instagram: ".
_OG_TITLE_RE = re.compile(r"^(.+?)\s+on Instagram:\s*(.*)$", re.DOTALL)
# The numeric media id (pk) rides in the App Link deep-link meta tags
# (al:ios:url / al:android:url = "instagram://media?id=<pk>") on anonymous pages,
# even though the og:* tags omit it.
_MEDIA_ID_RE = re.compile(r"instagram://media\?id=(\d+)")
def _og_date_to_iso(value: str) -> str | None:
"""``"July 9, 2026"`` -> ``"2026-07-09"`` (date-only; og carries no time)."""
try:
return (
datetime.strptime(value, "%B %d, %Y").replace(tzinfo=UTC).date().isoformat()
)
except ValueError:
return None
def _clean_caption(raw: str) -> str | None:
"""HTML-unescape and unwrap the surrounding quotes off an og caption preview."""
return html.unescape(raw).strip().strip('"').strip() or None
def _parse_og_meta(og: dict[str, str]) -> dict[str, Any]:
"""Lift post fields out of Instagram's Open Graph tags (see module notes above).
Caption + full name come from ``og:title``; counts + username + date from
``og:description``. Every field is optional and independently guarded, so a
shape we don't recognise yields a partial (or empty) dict instead of raising.
"""
out: dict[str, Any] = {}
desc = og.get("description", "")
title = og.get("title", "")
counts = _OG_COUNTS_RE.search(desc)
if counts:
out["likes"] = _html_int(counts.group(1))
out["comments"] = _html_int(counts.group(2))
owner_date = _OG_OWNER_DATE_RE.search(desc)
if owner_date:
out["ownerUsername"] = owner_date.group(1).strip().lstrip("@") or None
out["timestamp"] = _og_date_to_iso(owner_date.group(2))
tm = _OG_TITLE_RE.match(title)
if tm:
out["ownerFullName"] = tm.group(1).strip() or None
out["caption"] = _clean_caption(tm.group(2))
elif owner_date:
# No usable og:title: fall back to the caption after og:description's
# date prefix — still clean (the counts/username/date are stripped).
out["caption"] = _clean_caption(desc[owner_date.end() :])
return out
def _html_int(value: Any) -> int | None:
"""Coerce a string/number (``"1,234"``) to int, or ``None``."""
if isinstance(value, bool):
return None
if isinstance(value, int | float):
return int(value)
if isinstance(value, str):
digits = value.replace(",", "").strip()
if digits.isdigit():
return int(digits)
return None
def _og_tags(html: str) -> dict[str, str]:
"""Map ``og:<key>`` -> content for the post document."""
return {k.lower(): v for k, v in _OG_RE.findall(html)}
def _find_media(root: Any, shortcode: str | None) -> dict[str, Any] | None:
"""Depth-first search a JSON tree for the post's mobile-v1 media object.
Matches on ``code == shortcode`` (so a carousel *child* or a related post
can't be picked instead of the target) plus ``taken_at`` and an id, which
together uniquely identify the top-level ``PolarisMedia`` node.
"""
stack = [root]
while stack:
cur = stack.pop()
if isinstance(cur, dict):
if (
cur.get("taken_at") is not None
and ("pk" in cur or "id" in cur)
and (shortcode is None or cur.get("code") == shortcode)
):
return cur
stack.extend(cur.values())
elif isinstance(cur, list):
stack.extend(cur)
return None
def _relay_media(html: str, shortcode: str | None) -> dict[str, Any] | None:
"""Locate the embedded ``PolarisMedia`` object for this post, or ``None``.
The logged-out media payload is inlined as one of ~40 ``application/json``
script blocks. We only ``json.loads`` blocks that mention ``taken_at`` (and
the shortcode when known) so a single post fetch doesn't parse every blob.
"""
for raw in _APP_JSON_RE.findall(html):
if "taken_at" not in raw:
continue
if shortcode and shortcode not in raw:
continue
try:
data = json.loads(raw)
except (ValueError, TypeError):
continue
media = _find_media(data, shortcode)
if media is not None:
return media
return None
def _media_from_relay(
media: dict[str, Any], *, url: str, shortcode: str | None
) -> dict[str, Any]:
"""Map an embedded mobile-v1 ``PolarisMedia`` object to a flat media item.
Same output shape as :func:`parse_media` (so it flows through
``InstagramMediaItem`` unchanged), sourced from the relay dialect
(``user``/``taken_at``/``usertags.in``/``carousel_media``/flat counts).
"""
mt = media.get("media_type")
cap = media.get("caption")
caption = (
cap.get("text")
if isinstance(cap, dict)
else (cap if isinstance(cap, str) else None)
)
carousel = media.get("carousel_media")
carousel = (
[c for c in carousel if isinstance(c, dict)]
if isinstance(carousel, list)
else []
)
vv = media.get("video_versions")
video_url = (
vv[0].get("url")
if isinstance(vv, list) and vv and isinstance(vv[0], dict)
else None
)
is_video = mt == 2 or bool(video_url)
owner = media.get("user") if isinstance(media.get("user"), dict) else {}
tagged = [
ref
for t in ((media.get("usertags") or {}).get("in") or [])
if isinstance(t, dict) and (ref := _user_ref(t.get("user")))
]
coauthors = [
ref for c in (media.get("coauthor_producers") or []) if (ref := _user_ref(c))
]
loc_name, loc_id = _location_ref(media.get("location"))
# The relay ``id`` is ``POLARIS_<pk>``; strip the prefix so single-post ids
# match the numeric pk that og-fallback + the al:ios meta also yield.
ident = media.get("id")
if isinstance(ident, str) and ident.startswith("POLARIS_"):
ident = ident[len("POLARIS_") :]
pk = media.get("pk")
media_id = ident or (str(pk) if pk is not None else None)
return {
"id": media_id,
"type": _MEDIA_TYPE.get(mt) or ("Video" if is_video else "Image"),
"shortCode": media.get("code") or shortcode,
"caption": caption,
"hashtags": list(dict.fromkeys(_HASHTAG_RE.findall(caption)))
if caption
else [],
"mentions": list(dict.fromkeys(_MENTION_RE.findall(caption)))
if caption
else [],
"url": url,
"commentsCount": _int(media.get("comment_count")),
"dimensionsHeight": _int(media.get("original_height")),
"dimensionsWidth": _int(media.get("original_width")),
"displayUrl": _iv2_url(media.get("image_versions2"))
or media.get("display_uri"),
"images": [
u
for c in carousel
if (u := _iv2_url(c.get("image_versions2")) or c.get("display_uri"))
],
"childPosts": [_relay_child(c) for c in carousel],
"videoUrl": video_url,
"alt": media.get("accessibility_caption"),
"likesCount": _int(media.get("like_count")),
"videoViewCount": _int(media.get("view_count") or media.get("play_count"))
if is_video
else None,
"videoDuration": media.get("video_duration") if is_video else None,
"timestamp": _utc_from_sec(media.get("taken_at")),
"ownerUsername": owner.get("username"),
"ownerId": owner.get("id") or owner.get("pk"),
"ownerFullName": owner.get("full_name"),
"productType": media.get("product_type"),
"taggedUsers": tagged,
"coauthorProducers": coauthors,
"locationName": loc_name,
"locationId": loc_id,
}
def parse_post(
html: str | None, *, url: str, shortcode: str | None = None
) -> dict[str, Any] | None:
"""Map an anonymous ``/p/<code>/`` (or ``/reel/``) HTML page to a media dict.
Prefers the embedded mobile-v1 ``PolarisMedia`` relay JSON (full fidelity),
falling back to the lossy Open Graph meta tags only if that blob is absent.
Returns a dict shaped like :func:`parse_media` (so it flows through
``InstagramMediaItem`` unchanged), or ``None`` when the document carries
neither surface (e.g. a login interstitial slipped past the fetch-layer
redirect check — the caller treats ``None`` as "nothing to emit", never a
silent success).
"""
if not isinstance(html, str) or not html.strip():
return None
media = _relay_media(html, shortcode)
if media is not None:
return _media_from_relay(media, url=url, shortcode=shortcode)
# Fallback: no embedded relay blob -> Open Graph meta only.
og = _og_tags(html)
if not og:
return None
og_meta = _parse_og_meta(og)
caption = og_meta.get("caption")
video_url = og.get("video")
is_video = bool(video_url) or og.get("type") == "video.other"
id_match = _MEDIA_ID_RE.search(html)
return {
"id": id_match.group(1) if id_match else None,
"type": "Video" if is_video else "Image",
"shortCode": shortcode,
"caption": caption,
"hashtags": list(dict.fromkeys(_HASHTAG_RE.findall(caption)))
if caption
else [],
"mentions": list(dict.fromkeys(_MENTION_RE.findall(caption)))
if caption
else [],
"url": url,
"commentsCount": og_meta.get("comments"),
"displayUrl": og.get("image"),
"videoUrl": video_url if is_video else None,
"likesCount": og_meta.get("likes"),
"timestamp": og_meta.get("timestamp"),
"ownerUsername": og_meta.get("ownerUsername"),
"ownerFullName": og_meta.get("ownerFullName"),
}

136
surfsense_backend/app/proprietary/platforms/instagram/schemas.py Normal file
View file

@ -0,0 +1,136 @@
# ruff: noqa: N815
"""Input/output models for the Instagram scraper.
The models mirror the public Instagram scraper actor spec so the endpoint can be
a drop-in: the input accepts the full documented surface, and every output field
is emitted (``None``/``[]`` when the anonymous web endpoints cannot source it
yet) so the contract expands additively — the same rule the Google Search and
YouTube models follow.
**Anonymous only.** There is deliberately **no** authentication field on the
input (no username/password/token/cookie/``login*``) — the scraper holds only
Instagram's anonymous web-session cookies (``csrftoken``/``mid``) and can never
log in. Anything auth-shaped a caller sends lands in ``extra`` and is ignored.
"""
from __future__ import annotations
from typing import Any, Literal
from pydantic import BaseModel, ConfigDict, Field
InstagramResultsType = Literal["posts", "details", "reels"]
# Kept as distinct values for actor-spec parity, but under anonymous Google-backed
# discovery ``user`` and ``profile`` are aliases: both resolve to profile targets
# (IG's own hashtag/place/keyword search is login-walled).
InstagramSearchType = Literal["profile", "user"]
class InstagramScrapeInput(BaseModel):
"""Instagram scraper input surface (anonymous, no auth fields).
Field names mirror the public actor spec verbatim. ``resultsLimit`` /
``searchLimit`` are collector policy applied by :func:`scrape_instagram`,
NOT ceilings baked into the streaming flows. Fields the acquisition layer
doesn't source yet are still accepted via ``extra="allow"`` for parity.
"""
model_config = ConfigDict(extra="allow")
resultsType: InstagramResultsType = "posts"
directUrls: list[str] = Field(default_factory=list)
resultsLimit: int | None = Field(default=None, ge=1)
onlyPostsNewerThan: str | None = None
search: str | None = None
searchType: InstagramSearchType = "profile"
searchLimit: int | None = Field(default=None, ge=1, le=250)
addParentData: bool = False
skipPinnedPosts: bool = False
addProfileStatistics: bool = False
class _ItemBase(BaseModel):
"""Common error / provenance fields carried on every output item.
Errors surface as item-level fields (never exceptions) so a partial run
still returns the items it could source, mirroring the actor's shape.
"""
model_config = ConfigDict(extra="allow")
inputUrl: str | None = None
error: str | None = None
errorDescription: str | None = None
requestErrorMessages: list[str] = Field(default_factory=list)
def to_output(self) -> dict[str, Any]:
"""Serialize to the flat output dict shape (keeps extras)."""
return self.model_dump(exclude_none=False)
class InstagramMediaItem(_ItemBase):
"""A post or reel. One flat schema per the actor FAQ.
``firstComment``/``latestComments`` are intentionally absent: comment
*content* is login-walled (only the anonymous comment *count* is exposed, as
``commentsCount``), so this scraper can never source them.
"""
id: str | None = None
type: Literal["Image", "Video", "Sidecar"] | None = None
shortCode: str | None = None
caption: str | None = None
hashtags: list[str] = Field(default_factory=list)
mentions: list[str] = Field(default_factory=list)
url: str | None = None
commentsCount: int | None = None
dimensionsHeight: int | None = None
dimensionsWidth: int | None = None
displayUrl: str | None = None
images: list[str] = Field(default_factory=list)
videoUrl: str | None = None
alt: str | None = None
likesCount: int | None = None
videoViewCount: int | None = None
videoPlayCount: int | None = None
reshareCount: int | None = None
timestamp: str | None = None
childPosts: list[dict[str, Any]] = Field(default_factory=list)
ownerUsername: str | None = None
ownerId: str | None = None
ownerFullName: str | None = None
isPinned: bool | None = None
productType: str | None = None
videoDuration: float | None = None
paidPartnership: bool | None = None
taggedUsers: list[dict[str, Any]] = Field(default_factory=list)
musicInfo: dict[str, Any] | None = None
coauthorProducers: list[dict[str, Any]] = Field(default_factory=list)
locationName: str | None = None
locationId: str | None = None
isCommentsDisabled: bool | None = None
dataSource: dict[str, Any] | None = None
class InstagramProfile(_ItemBase):
"""A profile detail item (``detailKind = "profile"``)."""
detailKind: Literal["profile"] = "profile"
id: str | None = None
username: str | None = None
url: str | None = None
fullName: str | None = None
biography: str | None = None
externalUrl: str | None = None
followersCount: int | None = None
followsCount: int | None = None
postsCount: int | None = None
highlightReelCount: int | None = None
igtvVideoCount: int | None = None
isBusinessAccount: bool | None = None
businessCategoryName: str | None = None
private: bool | None = None
verified: bool | None = None
profilePicUrl: str | None = None
profilePicUrlHD: str | None = None
relatedProfiles: list[dict[str, Any]] = Field(default_factory=list)
latestPosts: list[dict[str, Any]] = Field(default_factory=list)
statistics: dict[str, Any] | None = None

429
surfsense_backend/app/proprietary/platforms/instagram/scraper.py Normal file
View file

@ -0,0 +1,429 @@
"""Orchestrator for the Instagram scraper.
The core is the async generator :func:`iter_instagram` (unbounded);
:func:`scrape_instagram` is a thin collector with a caller-supplied ``limit``
guard. Any cap is caller policy, never baked into flow logic.
Independent targets (one per ``directUrl`` / discovered entity) fan out
concurrently on a pool of warm sessions (sticky IPs); each target's own paging
stays sequential. ``fan_out`` is ported from ``../reddit/scraper.py`` but bound
to *this* module's proxy holders so every worker warms its own session once and
reuses it.
Anonymous-only. Every surface here is reachable without a login: profile web
info, the media embedded in a profile page, single-post/reel pages, and
Google-backed handle discovery. Login-walled surfaces (hashtag/place feeds,
comment threads, IG's native keyword search) are deliberately absent.
Flows are selected by ``resultsType``:
- ``posts`` / ``reels`` -> media items (profile feed, or a single
``/p/``/``/reel/`` page, or discovery search)
- ``details`` -> profile metadata (by URL or discovery search)
ponytail: deep feed pagination (past the first web page of media) needs the
GraphQL cursor endpoint whose doc-id drifts; v1 emits the first page and stops.
The upgrade path is a ``_paginate_feed`` helper in this file plus a doc-id in
``fetch.py`` — contained to these two files, per the acquisition-seam rule.
"""
from __future__ import annotations
import asyncio
import logging
import re
from collections.abc import AsyncIterator
from contextlib import aclosing
from datetime import UTC, datetime, timedelta
from typing import Any
from app.proprietary.platforms.google_search.schemas import GoogleSearchScrapeInput
from app.proprietary.platforms.google_search.scraper import scrape_serps
from .fetch import (
InstagramAccessBlockedError,
bind_proxy_holder,
fetch_html,
fetch_json,
now_iso,
open_proxy_holder,
)
from .parsers import parse_media, parse_post, parse_profile
from .schemas import InstagramScrapeInput
from .url_resolver import ResolvedUrl, resolve_url
logger = logging.getLogger(__name__)
__all__ = [
"InstagramAccessBlockedError",
"iter_instagram",
"scrape_instagram",
]
# Independent jobs run concurrently on a pool of warm proxy sessions. Anonymous
# Instagram is the most hostile platform, so this stays low to avoid burning the
# residential pool with parallel login walls.
_FANOUT_CONCURRENCY = 8
_PROFILE_PATH = "api/v1/users/web_profile_info/"
# Instagram usernames: 1-30 chars of letters/digits/period/underscore. Used to
# treat a profile/user discovery query as a direct (anonymous) handle lookup.
_HANDLE_RE = re.compile(r"[A-Za-z0-9._]{1,30}\Z")
def _parse_newer_than(value: str | None) -> datetime | None:
"""Parse ``onlyPostsNewerThan`` (ISO, YYYY-MM-DD, or relative) to UTC.
Relative forms: ``"<n> <unit>"`` where unit is minute/hour/day/week/month/
year (singular or plural). Anything unparseable returns ``None`` (no filter).
"""
if not value:
return None
text = value.strip().lower()
parts = text.split()
if len(parts) == 2 and parts[0].isdigit():
n = int(parts[0])
unit = parts[1].rstrip("s")
days = {
"minute": n / 1440,
"hour": n / 24,
"day": n,
"week": n * 7,
"month": n * 30,
"year": n * 365,
}.get(unit)
if days is None:
return None
return datetime.now(UTC) - timedelta(days=days)
try:
dt = datetime.fromisoformat(value.replace("Z", "+00:00"))
if dt.tzinfo:
return dt
return dt.replace(tzinfo=UTC)
except ValueError:
return None
def _is_after(timestamp: str | None, cutoff: datetime | None) -> bool:
"""True if the item ``timestamp`` (ISO) is at/after the cutoff (or no cutoff)."""
if cutoff is None:
return True
if not timestamp:
return True
try:
dt = datetime.fromisoformat(timestamp.replace("Z", "+00:00"))
return dt >= cutoff
except ValueError:
return True
async def fan_out(
jobs: list[AsyncIterator[dict[str, Any]]], *, concurrency: int = _FANOUT_CONCURRENCY
) -> AsyncIterator[dict[str, Any]]:
"""Stream items from independent async-iterator jobs via a warm worker pool.
Each worker opens ONE proxy session and reuses it across the sequential jobs
it pulls, so only the first job per worker pays the proxy handshake + the
cookie warm-up. Partial results (matches the reddit sibling): one blocked or
failed target yields nothing rather than aborting the batch — Instagram is
an aggregation, not an atomic transaction, so 4/5 good targets beat 0/5. But
if EVERY target was refused (zero items AND a hard block seen), the whole run
couldn't reach anonymous data, so we surface ``InstagramAccessBlockedError``
(-> 403) instead of a misleading empty success. Workers are cancelled and
their sessions closed if the consumer stops early.
"""
if not jobs:
return
job_queue: asyncio.Queue[AsyncIterator[dict[str, Any]]] = asyncio.Queue()
for job in jobs:
job_queue.put_nowait(job)
results: asyncio.Queue[list[dict[str, Any]]] = asyncio.Queue()
blocked = False # set if any target hit a hard login/auth wall
async def worker() -> None:
nonlocal blocked
holder = None
try:
holder = await open_proxy_holder()
except Exception as e: # no session: jobs still run via one-shot fetches
logger.warning("[instagram] proxy session open failed: %s", e)
try:
while True:
try:
job = job_queue.get_nowait()
except asyncio.QueueEmpty:
return
items: list[dict[str, Any]] = []
try:
if holder is not None:
async with bind_proxy_holder(holder):
items = [item async for item in job]
else:
items = [item async for item in job]
except InstagramAccessBlockedError as e:
# Partial results: a blocked target must not kill the batch.
# Record it so a fully-blocked run can still surface the 403.
blocked = True
logger.warning("[instagram] target blocked: %s", e)
except Exception as e: # one bad target must not kill the run
logger.warning("[instagram] fan-out job failed: %s", e)
await results.put(items)
finally:
if holder is not None:
await holder.close()
tasks = [asyncio.create_task(worker()) for _ in range(min(concurrency, len(jobs)))]
emitted = 0
try:
for _ in range(len(jobs)):
for item in await results.get():
emitted += 1
yield item
finally:
for task in tasks:
if not task.done():
task.cancel()
await asyncio.gather(*tasks, return_exceptions=True)
# Reached only on natural exhaustion (an early-stop close raises GeneratorExit
# inside the loop and skips this). Nothing came back AND a wall was hit ->
# the run was fully refused, so fail loud rather than return empty.
if emitted == 0 and blocked:
raise InstagramAccessBlockedError(
"Instagram refused anonymous access to every target"
)
def _emit(partial: dict[str, Any], *, input_url: str | None) -> dict[str, Any]:
"""Stamp provenance and serialize (parsers return plain dicts)."""
out = {**partial, "scrapedAt": now_iso()}
if input_url is not None:
out.setdefault("inputUrl", input_url)
return out
async def _profile_user(username: str) -> dict[str, Any] | None:
"""Fetch a profile's ``data.user`` node, or ``None``."""
data = await fetch_json(_PROFILE_PATH, {"username": username})
if isinstance(data, dict):
user = (
data.get("data", {}).get("user")
if isinstance(data.get("data"), dict)
else None
)
if isinstance(user, dict):
return user
return None
return None
def _media_matches(item: dict[str, Any], result_type: str) -> bool:
"""Filter a media item by feed type. ``reels`` keeps clips/videos only."""
if result_type == "reels":
return item.get("type") == "Video" or item.get("productType") == "clips"
return True
async def _media_flow(
resolved: ResolvedUrl,
*,
input_model: InstagramScrapeInput,
cutoff: datetime | None,
per_target: int,
) -> AsyncIterator[dict[str, Any]]:
"""Emit media items for a profile feed, or a single ``/p/``/``/reel/`` page."""
from .parsers import _edges
result_type = input_model.resultsType
if resolved.kind == "profile":
user = await _profile_user(resolved.value)
if user is None:
return
nodes = _edges(user.get("edge_owner_to_timeline_media"))
emitted = 0
for node in nodes:
item = parse_media(node)
if input_model.skipPinnedPosts and item.get("isPinned"):
continue
if not _media_matches(item, result_type):
continue
if not _is_after(item.get("timestamp"), cutoff):
continue
yield _emit(item, input_url=resolved.url)
emitted += 1
if emitted >= per_target:
return
return
if resolved.kind in ("post", "reel"):
# Single-post extraction: the anonymous ``?__a=1`` JSON API 404s/login-
# walls, but the public /p/<code>/ document embeds the mobile-v1
# PolarisMedia JSON (og-meta fallback), which parse_post reads. Numeric-ID
# URLs can't be keyed this way (the page needs the shortCode), so they're
# skipped upstream.
if resolved.numeric_post_id:
return
html = await fetch_html(f"p/{resolved.value}/")
item = parse_post(html, url=resolved.url, shortcode=resolved.value)
if item is None:
return
if not _media_matches(item, result_type):
return
if not _is_after(item.get("timestamp"), cutoff):
return
yield _emit(item, input_url=resolved.url)
return
async def _details_flow(
resolved: ResolvedUrl, *, input_model: InstagramScrapeInput
) -> AsyncIterator[dict[str, Any]]:
"""Emit one profile detail item for a URL (anonymous web_profile_info)."""
if resolved.kind == "profile":
user = await _profile_user(resolved.value)
if user is not None:
yield _emit(parse_profile(user), input_url=resolved.url)
def _kind_matches(resolved: ResolvedUrl, search_type: str) -> bool:
"""True if a resolved IG URL is the kind the discovery query asked for.
Discovery is profile-only now (hashtag/place feeds are login-walled), so
every supported ``search_type`` maps to a profile target.
"""
return resolved.kind == "profile"
async def _discover_via_google(
query: str, *, search_type: str, limit: int
) -> list[ResolvedUrl]:
"""Discover IG profile targets via Google ``site:instagram.com`` (anonymous).
Instagram's own keyword search is login-walled, so we reuse the existing
``google_search`` platform, classify each organic URL with ``resolve_url``,
keep the profile hits, de-dup, and cap at ``limit``.
Quality caveat: results reflect Google's index/ranking of instagram.com, not
IG's own relevance. This is discovery, not search parity (see README).
"""
serps = await scrape_serps(
GoogleSearchScrapeInput(
queries=query, site="instagram.com", maxPagesPerQuery=1
),
limit=1,
)
resolved: list[ResolvedUrl] = []
seen: set[tuple[str, str]] = set()
for serp in serps:
for org in serp.get("organicResults") or []:
url = org.get("url", "") if isinstance(org, dict) else ""
r = resolve_url(url)
if r is None or not _kind_matches(r, search_type):
continue
key = (r.kind, r.value)
if key in seen:
continue
seen.add(key)
resolved.append(r)
if len(resolved) >= limit:
return resolved
return resolved
async def _discover(query: str, *, search_type: str, limit: int) -> list[ResolvedUrl]:
"""Resolve a discovery query into profile targets - anonymously.
A query that is a valid handle resolves directly against the anonymous
profile endpoint ("messi" -> instagram.com/messi/). A non-handle query (e.g.
"national geographic") goes through Google ``site:instagram.com`` since IG's
native keyword search is login-walled.
"""
handle = query.strip().lstrip("@")
if _HANDLE_RE.match(handle):
url = f"https://www.instagram.com/{handle}/"
return [ResolvedUrl("profile", handle, url)][:limit]
return await _discover_via_google(query, search_type=search_type, limit=limit)
def _resolve_inputs(input_model: InstagramScrapeInput) -> list[ResolvedUrl]:
"""Resolve ``directUrls`` (URLs take priority over ``search``)."""
resolved: list[ResolvedUrl] = []
for url in input_model.directUrls:
r = resolve_url(url)
if r is None:
logger.warning("[instagram] unrecognized URL: %s", url)
continue
resolved.append(r)
return resolved
async def _targets(input_model: InstagramScrapeInput) -> list[ResolvedUrl]:
"""The resolved targets for this run: direct URLs, else discovery search."""
if input_model.directUrls:
return _resolve_inputs(input_model)
if not input_model.search:
return []
limit = input_model.searchLimit or 10
queries = [q.strip() for q in input_model.search.split(",") if q.strip()]
targets: list[ResolvedUrl] = []
for query in queries:
targets.extend(
await _discover(query, search_type=input_model.searchType, limit=limit)
)
return targets
async def iter_instagram(
input_model: InstagramScrapeInput,
) -> AsyncIterator[dict[str, Any]]:
"""Yield flat Instagram items. ``directUrls`` override ``search``.
Independent targets fan out concurrently; each target's paging stays
sequential. De-dupes media by ``id`` across targets.
"""
targets = await _targets(input_model)
if not targets:
return
result_type = input_model.resultsType
cutoff = _parse_newer_than(input_model.onlyPostsNewerThan)
per_target = input_model.resultsLimit or 10
if result_type == "details":
jobs = [_details_flow(r, input_model=input_model) for r in targets]
async with aclosing(fan_out(jobs)) as stream:
async for item in stream:
yield item
return
# posts / reels -> media feeds, de-duped by id across targets.
jobs = [
_media_flow(r, input_model=input_model, cutoff=cutoff, per_target=per_target)
for r in targets
]
seen: set[str] = set()
async with aclosing(fan_out(jobs)) as stream:
async for item in stream:
item_id = item.get("id")
if isinstance(item_id, str):
if item_id in seen:
continue
seen.add(item_id)
yield item
async def scrape_instagram(
input_model: InstagramScrapeInput, *, limit: int | None = None
) -> list[dict[str, Any]]:
"""Collect :func:`iter_instagram` into a list, honoring an optional ``limit``.
``limit`` is a request-time policy guard, NOT a ceiling in the streaming
core.
"""
from app.capabilities.core.progress import emit_progress
results: list[dict[str, Any]] = []
async with aclosing(iter_instagram(input_model)) as stream:
async for item in stream:
results.append(item)
emit_progress("scraping", current=len(results), total=limit, unit="item")
if limit is not None and len(results) >= limit:
break
return results

85
surfsense_backend/app/proprietary/platforms/instagram/url_resolver.py Normal file
View file

@ -0,0 +1,85 @@
"""Classify and normalize an Instagram URL into a scrape job.
Covers the anonymously-scrapable ``directUrls`` shapes: a profile, a post
(``/p/``), and a reel (``/reel/``), plus bare profile IDs. Hashtag and place
URLs are deliberately unsupported — their feeds are login-walled for anonymous
callers (use Google-backed discovery + single-post extraction instead).
Non-Instagram hosts resolve to ``None`` so the orchestrator can skip them.
Mirrors the frozen ``ResolvedUrl`` dataclass shape of ``../reddit/url_resolver.py``.
Normalization rules (from the reference spec):
- ``_u/`` and ``/profilecard/`` segments are stripped.
- Story URLs (``/stories/<user>/...``) reduce to the profile.
- Numeric post-ID URLs cannot be single-post-extracted anonymously (the HTML
page keys on the shortCode), so they resolve with ``numeric_post_id`` set and
the media flow skips them.
- ``share/`` links are unsupported (they need a network redirect to resolve to a
canonical post/profile URL); pass the resolved ``/p/`` or profile URL instead.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Literal
from urllib.parse import urlparse
ResolvedKind = Literal["profile", "post", "reel"]
_INSTAGRAM_HOSTS = frozenset({"m.instagram.com", "www.instagram.com", "instagram.com"})
_STRIP_SEGMENTS = frozenset({"_u", "profilecard"})
_RESERVED = frozenset(
{"p", "s", "tv", "reel", "reels", "share", "explore", "stories", "accounts"}
)
@dataclass(frozen=True)
class ResolvedUrl:
"""A classified Instagram target: the kind, its key, and the source URL."""
kind: ResolvedKind
value: str
url: str
numeric_post_id: bool = False
def _is_instagram_host(hostname: str | None) -> bool:
if not hostname:
return False
return hostname.lower() in _INSTAGRAM_HOSTS
def _segments(url: str) -> list[str]:
parsed = urlparse(url)
if not _is_instagram_host(parsed.hostname):
return []
if not parsed.path:
return []
return [s for s in parsed.path.split("/") if s and s not in _STRIP_SEGMENTS]
def resolve_url(url: str) -> ResolvedUrl | None:
"""Classify an Instagram URL into a scrape job, or ``None`` if unrecognized.
A bare token with no ``http`` prefix and no ``/`` is treated as a profile ID
(the reference accepts bare profile handles alongside full URLs).
"""
if "instagram.com" not in url.lower():
token = url.strip().lstrip("@")
if token and "/" not in token and "." not in token:
return ResolvedUrl("profile", token, f"https://www.instagram.com/{token}/")
segments = _segments(url)
if not segments:
return None
head = segments[0]
if head == "p" and len(segments) >= 2:
code = segments[1]
return ResolvedUrl("post", code, url, numeric_post_id=code.isdigit())
if head in ("reel", "reels") and len(segments) >= 2:
code = segments[1]
return ResolvedUrl("reel", code, url, numeric_post_id=code.isdigit())
if head == "stories" and len(segments) >= 2:
user = segments[1]
return ResolvedUrl("profile", user, f"https://www.instagram.com/{user}/")
if head not in _RESERVED:
return ResolvedUrl("profile", head, url)
return None

35
surfsense_backend/app/proprietary/platforms/tiktok/__init__.py Normal file
View file

@ -0,0 +1,35 @@
"""Anonymous, blob-first TikTok scraper (public interface).
The capability layer depends only on the names re-exported here: the input
schema, the collector/generator, the video item shape, and the hard-block error.
"""
from __future__ import annotations
from .orchestrator import (
iter_tiktok,
scrape_tiktok,
scrape_tiktok_comments,
scrape_tiktok_trending,
search_tiktok_users,
)
from .schemas import (
CommentItem,
TikTokProfileItem,
TikTokScrapeInput,
TikTokVideoItem,
)
from .session import TikTokAccessBlockedError
__all__ = [
"CommentItem",
"TikTokAccessBlockedError",
"TikTokProfileItem",
"TikTokScrapeInput",
"TikTokVideoItem",
"iter_tiktok",
"scrape_tiktok",
"scrape_tiktok_comments",
"scrape_tiktok_trending",
"search_tiktok_users",
]

25
surfsense_backend/app/proprietary/platforms/tiktok/extraction/__init__.py Normal file
View file

@ -0,0 +1,25 @@
"""Turn raw TikTok page/API payloads into normalized items."""
from __future__ import annotations
from .author import parse_author, parse_profile
from .comments import comments_from_response, parse_comment
from .hydration import extract_rehydration_data
from .item_list import items_from_response
from .scopes import user_info, video_item_struct
from .user_search import parse_search_user, users_from_response
from .video import parse_video
__all__ = [
"comments_from_response",
"extract_rehydration_data",
"items_from_response",
"parse_author",
"parse_comment",
"parse_profile",
"parse_search_user",
"parse_video",
"user_info",
"users_from_response",
"video_item_struct",
]

38
surfsense_backend/app/proprietary/platforms/tiktok/extraction/author.py Normal file
View file

@ -0,0 +1,38 @@
"""Normalize TikTok author/profile payloads into an ``authorMeta`` dict."""
from __future__ import annotations
from typing import Any
_PROFILE_URL = "https://www.tiktok.com/@{username}"
def build_author_meta(author: dict[str, Any], stats: dict[str, Any]) -> dict[str, Any]:
"""Map an author object + its stats to the ``authorMeta`` output shape."""
username = author.get("uniqueId")
return {
"id": author.get("id"),
"name": username,
"nickName": author.get("nickname"),
"profileUrl": _PROFILE_URL.format(username=username) if username else None,
"verified": author.get("verified"),
"signature": author.get("signature"),
"avatar": author.get("avatarLarger") or author.get("avatarMedium"),
"privateAccount": author.get("privateAccount"),
"fans": stats.get("followerCount"),
"following": stats.get("followingCount"),
"heart": stats.get("heartCount"),
"video": stats.get("videoCount"),
}
def parse_author(user_info: dict[str, Any]) -> dict[str, Any]:
"""Map a ``webapp.user-detail`` ``userInfo`` (``{user, stats}``) to authorMeta."""
return build_author_meta(user_info.get("user") or {}, user_info.get("stats") or {})
def parse_profile(user_info: dict[str, Any]) -> dict[str, Any]:
"""Map a ``userInfo`` to a standalone :class:`TikTokProfileItem` output dict."""
from ..schemas.items import TikTokProfileItem
return TikTokProfileItem(**parse_author(user_info)).to_output()

55
surfsense_backend/app/proprietary/platforms/tiktok/extraction/comments.py Normal file
View file

@ -0,0 +1,55 @@
"""Parse a captured ``/api/comment/list`` response into comment items.
The comment API returns ``{"comments": [...]}`` where each entry uses the
mobile-API snake_case shape (``cid``, ``digg_count``, ``reply_comment_total``,
``create_time``, and a nested ``user`` with ``uid``/``unique_id``/``nickname``/
``avatar_thumb``). ``reply_id != "0"`` marks a reply to a parent comment.
"""
from __future__ import annotations
from typing import Any
from .timestamps import epoch_to_iso
def comments_from_response(body: Any) -> list[dict[str, Any]]:
"""Return the raw comment records carried by one API response, or ``[]``."""
if not isinstance(body, dict):
return []
comments = body.get("comments")
if not isinstance(comments, list):
return []
return [c for c in comments if isinstance(c, dict)]
def _avatar(user: dict[str, Any]) -> str | None:
thumb = user.get("avatar_thumb")
if isinstance(thumb, dict):
urls = thumb.get("url_list")
if isinstance(urls, list) and urls:
return urls[0]
return None
def parse_comment(raw: dict[str, Any], video_url: str) -> dict[str, Any]:
"""Map a raw comment record to a :class:`CommentItem` output dict."""
from ..schemas.items import CommentItem
user = raw.get("user") if isinstance(raw.get("user"), dict) else {}
reply_id = raw.get("reply_id")
create_time = raw.get("create_time")
return CommentItem(
id=raw.get("cid"),
text=raw.get("text"),
videoWebUrl=video_url,
diggCount=raw.get("digg_count"),
replyCommentTotal=raw.get("reply_comment_total"),
createTime=create_time,
createTimeISO=epoch_to_iso(create_time),
uid=user.get("uid"),
uniqueId=user.get("unique_id"),
nickName=user.get("nickname"),
avatar=_avatar(user),
repliesToId=reply_id if reply_id and reply_id != "0" else None,
).to_output()

28
surfsense_backend/app/proprietary/platforms/tiktok/extraction/hydration.py Normal file
View file

@ -0,0 +1,28 @@
"""Extract the ``__UNIVERSAL_DATA_FOR_REHYDRATION__`` JSON embedded in page HTML.
TikTok server-renders the first page of data into a single script tag; parsing
it yields page-one items (video/profile/hashtag) without any signed API call.
"""
from __future__ import annotations
import json
import re
from typing import Any
_BLOB_RE = re.compile(
r'<script[^>]*id="__UNIVERSAL_DATA_FOR_REHYDRATION__"[^>]*>(.*?)</script>',
re.DOTALL,
)
def extract_rehydration_data(html: str) -> dict[str, Any] | None:
"""Return the parsed rehydration blob, or ``None`` if absent/unparseable."""
match = _BLOB_RE.search(html)
if not match:
return None
try:
data = json.loads(match.group(1))
except (json.JSONDecodeError, ValueError):
return None
return data if isinstance(data, dict) else None

30
surfsense_backend/app/proprietary/platforms/tiktok/extraction/item_list.py Normal file
View file

@ -0,0 +1,30 @@
"""Item structs from a captured ``item_list`` / search API response body.
Profile and hashtag listings return ``{"itemList": [...]}``; search returns
``{"data": [{"item": {...}}]}``. Both element shapes are the same itemStruct
:func:`parse_video` already consumes.
"""
from __future__ import annotations
from typing import Any
def items_from_response(body: Any) -> list[dict[str, Any]]:
"""Return the itemStructs carried by one API response, or ``[]``."""
if not isinstance(body, dict):
return []
item_list = body.get("itemList")
if isinstance(item_list, list):
return [i for i in item_list if isinstance(i, dict)]
data = body.get("data")
if isinstance(data, list):
return [
entry["item"]
for entry in data
if isinstance(entry, dict) and isinstance(entry.get("item"), dict)
]
return []

30
surfsense_backend/app/proprietary/platforms/tiktok/extraction/scopes.py Normal file
View file

@ -0,0 +1,30 @@
"""Navigate the rehydration blob to the scopes the flows consume."""
from __future__ import annotations
from typing import Any
_DEFAULT = "__DEFAULT_SCOPE__"
def _scope(data: dict[str, Any], name: str) -> dict[str, Any] | None:
scope = (data.get(_DEFAULT) or {}).get(name)
return scope if isinstance(scope, dict) else None
def video_item_struct(data: dict[str, Any]) -> dict[str, Any] | None:
"""The ``itemStruct`` of a video-detail page, or ``None``."""
scope = _scope(data, "webapp.video-detail")
if not scope:
return None
item = (scope.get("itemInfo") or {}).get("itemStruct")
return item if isinstance(item, dict) else None
def user_info(data: dict[str, Any]) -> dict[str, Any] | None:
"""The ``userInfo`` (``{user, stats}``) of a profile page, or ``None``."""
scope = _scope(data, "webapp.user-detail")
if not scope:
return None
info = scope.get("userInfo")
return info if isinstance(info, dict) else None

18
surfsense_backend/app/proprietary/platforms/tiktok/extraction/timestamps.py Normal file
View file

@ -0,0 +1,18 @@
"""Millisecond-ISO timestamps matching the actor's output shape."""
from __future__ import annotations
from datetime import UTC, datetime
def epoch_to_iso(seconds: int | str | None) -> str | None:
"""Convert a Unix-seconds timestamp to ``YYYY-MM-DDTHH:MM:SS.000Z``."""
if not seconds:
return None
stamp = datetime.fromtimestamp(int(seconds), tz=UTC)
return stamp.strftime("%Y-%m-%dT%H:%M:%S.000Z")
def now_iso() -> str:
"""Current UTC time in the millisecond-ISO output shape."""
return datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"

56
surfsense_backend/app/proprietary/platforms/tiktok/extraction/user_search.py Normal file
View file

@ -0,0 +1,56 @@
"""Parse the ``/api/search/user`` response into profile items.
User search returns ``{"user_list": [{"user_info": {...}}, ...]}`` where each
``user_info`` uses the mobile-API snake_case shape (``uid``, ``unique_id``,
``follower_count``, ``total_favorited``, ``avatar_thumb.url_list``) — distinct
from the camelCase ``webapp.user-detail`` blob the profile flow reads, so it gets
its own mapping into the shared :class:`TikTokProfileItem` output contract.
"""
from __future__ import annotations
from typing import Any
_PROFILE_URL = "https://www.tiktok.com/@{username}"
def users_from_response(body: Any) -> list[dict[str, Any]]:
"""Return the ``user_info`` objects carried by one search response, or ``[]``."""
if not isinstance(body, dict):
return []
user_list = body.get("user_list")
if not isinstance(user_list, list):
return []
return [
entry["user_info"]
for entry in user_list
if isinstance(entry, dict) and isinstance(entry.get("user_info"), dict)
]
def _avatar(user_info: dict[str, Any]) -> str | None:
thumb = user_info.get("avatar_thumb")
if isinstance(thumb, dict):
urls = thumb.get("url_list")
if isinstance(urls, list) and urls:
return urls[0]
return None
def parse_search_user(user_info: dict[str, Any]) -> dict[str, Any]:
"""Map a search ``user_info`` to a :class:`TikTokProfileItem` output dict."""
from ..schemas.items import TikTokProfileItem
username = user_info.get("unique_id")
return TikTokProfileItem(
id=user_info.get("uid"),
name=username,
nickName=user_info.get("nickname"),
profileUrl=_PROFILE_URL.format(username=username) if username else None,
verified=bool(user_info.get("enterprise_verify_reason")),
signature=user_info.get("signature"),
avatar=_avatar(user_info),
fans=user_info.get("follower_count"),
heart=user_info.get("total_favorited"),
secUid=user_info.get("sec_uid"),
).to_output()

73
surfsense_backend/app/proprietary/platforms/tiktok/extraction/video.py Normal file
View file

@ -0,0 +1,73 @@
"""Normalize a raw TikTok ``itemStruct`` into a :class:`TikTokVideoItem` dict."""
from __future__ import annotations
from typing import Any
from ..schemas.items import TikTokVideoItem
from .author import build_author_meta
from .timestamps import epoch_to_iso
_VIDEO_URL = "https://www.tiktok.com/@{username}/video/{video_id}"
def _music_meta(music: dict[str, Any]) -> dict[str, Any]:
return {
"musicId": music.get("id"),
"musicName": music.get("title"),
"musicAuthor": music.get("authorName"),
"musicOriginal": music.get("original"),
"playUrl": music.get("playUrl"),
}
def _video_meta(video: dict[str, Any]) -> dict[str, Any]:
return {
"height": video.get("height"),
"width": video.get("width"),
"duration": video.get("duration"),
"coverUrl": video.get("cover"),
"format": video.get("format"),
"definition": video.get("definition"),
}
def _hashtags(challenges: list[dict[str, Any]]) -> list[dict[str, Any]]:
return [
{"id": c.get("id"), "name": c.get("title")}
for c in challenges
if isinstance(c, dict)
]
def parse_video(item: dict[str, Any]) -> dict[str, Any]:
"""Map an ``itemStruct`` to the flat output contract, filling known fields."""
author = item.get("author") or {}
author_stats = item.get("authorStats") or {}
stats = item.get("stats") or {}
username = author.get("uniqueId")
video_id = item.get("id")
web_url = (
_VIDEO_URL.format(username=username, video_id=video_id)
if username and video_id
else None
)
create_time = item.get("createTime")
return TikTokVideoItem(
id=video_id,
text=item.get("desc"),
createTime=create_time,
createTimeISO=epoch_to_iso(create_time),
authorMeta=build_author_meta(author, author_stats),
musicMeta=_music_meta(item.get("music") or {}),
videoMeta=_video_meta(item.get("video") or {}),
webVideoUrl=web_url,
diggCount=stats.get("diggCount"),
shareCount=stats.get("shareCount"),
playCount=stats.get("playCount"),
collectCount=stats.get("collectCount"),
commentCount=stats.get("commentCount"),
hashtags=_hashtags(item.get("challenges") or []),
).to_output()

19
surfsense_backend/app/proprietary/platforms/tiktok/flows/__init__.py Normal file
View file

@ -0,0 +1,19 @@
"""Per-target scrape flows: resolved target -> normalized items."""
from __future__ import annotations
from collections.abc import AsyncIterator, Awaitable, Callable
FetchFn = Callable[[str], Awaitable[str | None]]
"""Fetch a page's HTML by URL (blob-first video flow)."""
FetchListingFn = Callable[[str, int], Awaitable[list[dict]]]
"""Load a listing page and return up to ``count`` captured itemStructs."""
FetchUsersFn = Callable[[str, int], Awaitable[list[dict]]]
"""Load a user-search page and return up to ``count`` captured ``user_info`` records."""
FetchCommentsFn = Callable[[str, int], Awaitable[list[dict]]]
"""Load a video page and return up to ``count`` captured raw comment records."""
FlowResult = AsyncIterator[dict]

52
surfsense_backend/app/proprietary/platforms/tiktok/flows/comments.py Normal file
View file

@ -0,0 +1,52 @@
"""Comments flow: a video URL -> its public comment thread.
Comments load over a signed ``/api/comment/list`` XHR that TikTok *does* serve to
anonymous sessions once the comments panel is opened (unlike profile-video/search
feeds). Records are deduped by comment id, capped, and — when a video has none or
withholds them — degraded to one ErrorItem, mirroring the listing flow.
"""
from __future__ import annotations
from collections.abc import AsyncIterator
from typing import Any
from ..extraction import parse_comment
from ..extraction.timestamps import now_iso
from ..schemas import ErrorItem
from ..targets.types import TikTokTarget
from . import FetchCommentsFn
_EMPTY_MESSAGE = (
"No comments returned. The video may have none, comments disabled, or TikTok "
"withheld them from anonymous access."
)
async def iter_comments(
target: TikTokTarget, *, cap: int, fetch_comments: FetchCommentsFn
) -> AsyncIterator[dict[str, Any]]:
if cap <= 0:
return
seen: set[str] = set()
emitted = 0
for raw in await fetch_comments(target.url, cap):
out = parse_comment(raw, target.url)
cid = out.get("id")
if cid is not None:
if cid in seen:
continue
seen.add(cid)
out["scrapedAt"] = now_iso()
yield out
emitted += 1
if emitted >= cap:
return
if emitted == 0:
yield ErrorItem(
url=target.url,
input=target.value,
error=_EMPTY_MESSAGE,
errorCode="no_comments",
scrapedAt=now_iso(),
).to_output()

55
surfsense_backend/app/proprietary/platforms/tiktok/flows/listing.py Normal file
View file

@ -0,0 +1,55 @@
"""Listing flow shared by profile, hashtag, and search targets.
The browser seam returns raw itemStructs captured from the signed ``item_list``
XHRs; this maps each to the output contract, drops duplicate video ids, and
stops at the per-target ``cap``.
"""
from __future__ import annotations
from collections.abc import AsyncIterator
from typing import Any
from ..extraction import parse_video
from ..extraction.timestamps import now_iso
from ..schemas import ErrorItem
from ..targets.types import TikTokTarget
from . import FetchListingFn
# Profile and search feeds are trust-gated: an anonymous headless session gets an
# empty body (profile) or no results XHR (search), while hashtag feeds load. We
# can't tell "genuinely empty" from "blocked" here, so a zero-item listing emits
# one honest ErrorItem instead of vanishing silently.
_EMPTY_LISTING_MESSAGE = (
"No videos returned. The target may be empty/private/removed, or TikTok "
"withheld this feed from anonymous access (common for profiles and search)."
)
async def iter_listing(
target: TikTokTarget, *, cap: int, fetch_listing: FetchListingFn
) -> AsyncIterator[dict[str, Any]]:
if cap <= 0:
return
seen: set[str] = set()
emitted = 0
for item in await fetch_listing(target.url, cap):
out = parse_video(item)
video_id = out.get("id")
if video_id is not None:
if video_id in seen:
continue
seen.add(video_id)
out["scrapedAt"] = now_iso()
yield out
emitted += 1
if emitted >= cap:
return
if emitted == 0:
yield ErrorItem(
url=target.url,
input=target.value,
error=_EMPTY_LISTING_MESSAGE,
errorCode="no_items",
scrapedAt=now_iso(),
).to_output()

37
surfsense_backend/app/proprietary/platforms/tiktok/flows/profile.py Normal file
View file

@ -0,0 +1,37 @@
"""Profile flow: reliable blob metadata first, then the (gated) video listing.
A profile's account data (name, followers, bio, verification) lives in the page's
rehydration blob and loads over plain HTTP without a signed request, so we emit it
first and always. The video listing needs a signed ``item_list`` XHR that TikTok
withholds from anonymous sessions, so it is best-effort: it streams videos when it
loads and degrades to an ErrorItem (via :func:`iter_listing`) when withheld. The
metadata item therefore survives even when the videos are blocked.
"""
from __future__ import annotations
from collections.abc import AsyncIterator
from typing import Any
from ..extraction import extract_rehydration_data, parse_profile, user_info
from ..extraction.timestamps import now_iso
from ..targets.types import TikTokTarget
from . import FetchFn, FetchListingFn
from .listing import iter_listing
async def iter_profile(
target: TikTokTarget,
*,
cap: int,
fetch: FetchFn,
fetch_listing: FetchListingFn,
) -> AsyncIterator[dict[str, Any]]:
html = await fetch(target.url)
info = user_info(extract_rehydration_data(html) or {}) if html else None
if info:
item = parse_profile(info)
item["scrapedAt"] = now_iso()
yield item
async for out in iter_listing(target, cap=cap, fetch_listing=fetch_listing):
yield out

54
surfsense_backend/app/proprietary/platforms/tiktok/flows/user_search.py Normal file
View file

@ -0,0 +1,54 @@
"""User-search flow: keyword -> public account records.
Unlike video/general search (login-walled for anonymous sessions), the Users tab
hits ``/api/search/user`` and returns account records without a redirect. Each
query's results are deduped by uid, capped, and — when a query returns nothing —
degraded to one ErrorItem, mirroring the listing flow.
"""
from __future__ import annotations
from collections.abc import AsyncIterator
from typing import Any
from urllib.parse import quote
from ..extraction import parse_search_user
from ..extraction.timestamps import now_iso
from ..schemas import ErrorItem
from . import FetchUsersFn
_USER_SEARCH_URL = "https://www.tiktok.com/search/user?q={query}"
_EMPTY_MESSAGE = (
"No accounts returned for this query. It may have no matches, or TikTok "
"withheld the results from anonymous access."
)
async def iter_user_search(
query: str, *, cap: int, fetch_users: FetchUsersFn
) -> AsyncIterator[dict[str, Any]]:
if cap <= 0:
return
url = _USER_SEARCH_URL.format(query=quote(query))
seen: set[str] = set()
emitted = 0
for user_info in await fetch_users(url, cap):
out = parse_search_user(user_info)
uid = out.get("id")
if uid is not None:
if uid in seen:
continue
seen.add(uid)
out["scrapedAt"] = now_iso()
yield out
emitted += 1
if emitted >= cap:
return
if emitted == 0:
yield ErrorItem(
url=url,
input=query,
error=_EMPTY_MESSAGE,
errorCode="no_users",
scrapedAt=now_iso(),
).to_output()

28
surfsense_backend/app/proprietary/platforms/tiktok/flows/video.py Normal file
View file

@ -0,0 +1,28 @@
"""Video-URL flow: fetch a post page, read its rehydration blob, emit one item."""
from __future__ import annotations
from collections.abc import AsyncIterator
from typing import Any
from ..extraction import extract_rehydration_data, parse_video, video_item_struct
from ..extraction.timestamps import now_iso
from ..targets.types import TikTokTarget
from . import FetchFn
async def iter_video(
target: TikTokTarget, *, fetch: FetchFn
) -> AsyncIterator[dict[str, Any]]:
html = await fetch(target.url)
if not html:
return
data = extract_rehydration_data(html)
if not data:
return
item = video_item_struct(data)
if item is None:
return
out = parse_video(item)
out["scrapedAt"] = now_iso()
yield out

196
surfsense_backend/app/proprietary/platforms/tiktok/orchestrator.py Normal file
View file

@ -0,0 +1,196 @@
"""Resolve a :class:`TikTokScrapeInput` into targets and stream their items.
Targets run sequentially on one warm sticky IP; ``limit`` is collector policy
applied by :func:`scrape_tiktok`, never baked into a flow. Each kind routes to
its flow via :func:`_dispatch`: video URLs read the rehydration blob over HTTP,
listings capture signed item_list XHRs through the stealth browser.
"""
from __future__ import annotations
from collections.abc import AsyncIterator
from typing import Any
from .extraction.timestamps import now_iso
from .flows import FetchCommentsFn, FetchFn, FetchListingFn, FetchUsersFn
from .flows.comments import iter_comments
from .flows.listing import iter_listing
from .flows.profile import iter_profile
from .flows.user_search import iter_user_search
from .flows.video import iter_video
from .schemas import ErrorItem, TikTokScrapeInput
from .session import (
fetch_comments,
fetch_html,
fetch_item_list,
fetch_trending,
fetch_user_search,
)
from .targets import resolve_target
from .targets.types import TikTokTarget
_PROFILE_URL = "https://www.tiktok.com/@{name}"
_HASHTAG_URL = "https://www.tiktok.com/tag/{tag}"
_EXPLORE_URL = "https://www.tiktok.com/explore"
def _resolve_targets(input_model: TikTokScrapeInput) -> list[TikTokTarget]:
"""Build the target list from the URL/profile/hashtag sources.
A raw ``tiktok.com/search?...`` URL passed explicitly in
``startUrls``/``postURLs`` still resolves here and keeps its native listing
routing; there is no keyword-search shortcut.
"""
targets: list[TikTokTarget] = []
for entry in input_model.startUrls:
resolved = resolve_target(entry.url)
if resolved is not None:
targets.append(resolved)
for url in input_model.postURLs:
resolved = resolve_target(url)
if resolved is not None:
targets.append(resolved)
for profile in input_model.profiles:
name = profile.lstrip("@")
targets.append(TikTokTarget("profile", name, _PROFILE_URL.format(name=name)))
for tag in input_model.hashtags:
targets.append(TikTokTarget("hashtag", tag, _HASHTAG_URL.format(tag=tag)))
return targets
def _dispatch(
target: TikTokTarget,
*,
cap: int,
fetch: FetchFn,
fetch_listing: FetchListingFn,
) -> AsyncIterator[dict[str, Any]]:
if target.kind == "video":
return iter_video(target, fetch=fetch)
if target.kind == "profile":
return iter_profile(target, cap=cap, fetch=fetch, fetch_listing=fetch_listing)
return iter_listing(target, cap=cap, fetch_listing=fetch_listing)
async def iter_tiktok(
input_model: TikTokScrapeInput,
*,
fetch: FetchFn = fetch_html,
fetch_listing: FetchListingFn = fetch_item_list,
) -> AsyncIterator[dict[str, Any]]:
"""Yield normalized items for every resolved target, in order.
The video flow's ``fetch_html`` opens its own warmed proxy session per call
when none is bound; the listing flow drives its own browser. Neither binds a
ContextVar across these ``yield``s, so the generator stays context-safe.
"""
cap = input_model.resultsPerPage
for target in _resolve_targets(input_model):
async for item in _dispatch(
target, cap=cap, fetch=fetch, fetch_listing=fetch_listing
):
yield item
async def scrape_tiktok(
input_model: TikTokScrapeInput,
*,
limit: int | None = None,
fetch: FetchFn = fetch_html,
fetch_listing: FetchListingFn = fetch_item_list,
) -> list[dict[str, Any]]:
"""Collect :func:`iter_tiktok` into a list, honoring an optional ``limit``."""
from app.capabilities.core.progress import emit_progress
results: list[dict[str, Any]] = []
async for item in iter_tiktok(
input_model, fetch=fetch, fetch_listing=fetch_listing
):
results.append(item)
emit_progress("scraping", current=len(results), total=limit, unit="item")
if limit is not None and len(results) >= limit:
break
return results
async def search_tiktok_users(
queries: list[str],
*,
per_query: int,
limit: int | None = None,
fetch_users: FetchUsersFn = fetch_user_search,
) -> list[dict[str, Any]]:
"""Collect user-search account records across queries, honoring ``limit``."""
from app.capabilities.core.progress import emit_progress
results: list[dict[str, Any]] = []
for query in queries:
async for item in iter_user_search(
query, cap=per_query, fetch_users=fetch_users
):
results.append(item)
emit_progress("searching", current=len(results), total=limit, unit="item")
if limit is not None and len(results) >= limit:
return results
return results
async def scrape_tiktok_trending(
*,
count: int,
fetch_trending_fn: FetchListingFn = fetch_trending,
) -> list[dict[str, Any]]:
"""Collect up to ``count`` trending videos from the Explore feed.
A single global feed, so it reuses the listing flow (parse/dedupe/cap/empty-
ErrorItem) over a synthetic target — no user input to resolve.
"""
from app.capabilities.core.progress import emit_progress
target = TikTokTarget(kind="trending", value="explore", url=_EXPLORE_URL)
results: list[dict[str, Any]] = []
async for item in iter_listing(target, cap=count, fetch_listing=fetch_trending_fn):
results.append(item)
emit_progress("scraping", current=len(results), total=count, unit="item")
return results
async def scrape_tiktok_comments(
video_urls: list[str],
*,
per_video: int,
limit: int | None = None,
fetch_comments_fn: FetchCommentsFn = fetch_comments,
) -> list[dict[str, Any]]:
"""Collect comments across video URLs, honoring ``limit``.
A non-video URL yields one ``bad_url`` ErrorItem (never a silent drop) so the
caller can tell "wrong input" from "video has no comments".
"""
from app.capabilities.core.progress import emit_progress
results: list[dict[str, Any]] = []
for url in video_urls:
target = resolve_target(url)
if target is None or target.kind != "video":
results.append(
ErrorItem(
url=url,
input=url,
error="Not a TikTok video URL.",
errorCode="bad_url",
scrapedAt=now_iso(),
).to_output()
)
emit_progress("scraping", current=len(results), total=limit, unit="item")
if limit is not None and len(results) >= limit:
return results
continue
async for item in iter_comments(
target, cap=per_video, fetch_comments=fetch_comments_fn
):
results.append(item)
emit_progress("scraping", current=len(results), total=limit, unit="item")
if limit is not None and len(results) >= limit:
return results
return results

26
surfsense_backend/app/proprietary/platforms/tiktok/schemas/__init__.py Normal file
View file

@ -0,0 +1,26 @@
"""Apify-compatible input and output contracts for the TikTok scraper."""
from __future__ import annotations
from .input import StartUrl, TikTokScrapeInput
from .items import (
AuthorMeta,
CommentItem,
ErrorItem,
MusicMeta,
TikTokProfileItem,
TikTokVideoItem,
VideoMeta,
)
__all__ = [
"AuthorMeta",
"CommentItem",
"ErrorItem",
"MusicMeta",
"StartUrl",
"TikTokProfileItem",
"TikTokScrapeInput",
"TikTokVideoItem",
"VideoMeta",
]

59
surfsense_backend/app/proprietary/platforms/tiktok/schemas/input.py Normal file
View file

@ -0,0 +1,59 @@
# ruff: noqa: N815 - field names mirror the public camelCase TikTok/Apify API
"""Input surface for the TikTok scraper, shaped to the Clockworks actor.
Anonymous only: no auth-shaped field exists here. ``extra="allow"`` keeps the
contract in parity with the actor for fields the scraper does not read.
``resultsPerPage`` is a per-target count; the cross-target ceiling is caller
policy applied by the collector, never baked into a flow.
"""
from __future__ import annotations
from typing import Literal
from pydantic import BaseModel, ConfigDict, Field
ProfileSorting = Literal["latest", "popular", "oldest"]
ProfileSection = Literal["videos", "reposts"]
SearchSection = Literal["", "/video", "/user"]
class StartUrl(BaseModel):
"""A single direct URL entry (``{"url": ...}``; extra keys ignored)."""
model_config = ConfigDict(extra="allow")
url: str
class TikTokScrapeInput(BaseModel):
model_config = ConfigDict(extra="allow")
# Discovery
startUrls: list[StartUrl] = Field(default_factory=list)
hashtags: list[str] = Field(default_factory=list)
profiles: list[str] = Field(default_factory=list)
searchQueries: list[str] = Field(default_factory=list)
postURLs: list[str] = Field(default_factory=list)
# Per-target count
resultsPerPage: int = Field(default=1, ge=1)
# Profile options
profileScrapeSections: list[ProfileSection] = Field(
default_factory=lambda: ["videos"]
)
profileSorting: ProfileSorting = "latest"
excludePinnedPosts: bool = False
# Search options
searchSection: SearchSection = ""
maxProfilesPerQuery: int = Field(default=10, ge=1)
# Incremental filters (ISO date or relative "<n> days" per the actor)
oldestPostDateUnified: str | None = None
newestPostDate: str | None = None
# Proxy
proxyCountryCode: str = "None"

147
surfsense_backend/app/proprietary/platforms/tiktok/schemas/items.py Normal file
View file

@ -0,0 +1,147 @@
# ruff: noqa: N815 - field names mirror the public camelCase TikTok/Apify API
"""Output items keyed to the Clockworks TikTok actor's shape.
Every model is open (``extra="allow"``) and defaults unsourced fields to
``None``/``[]`` so the MVP can populate a reliable subset and expand the
contract additively without breaking consumers.
"""
from __future__ import annotations
from typing import Any
from pydantic import BaseModel, ConfigDict, Field
class AuthorMeta(BaseModel):
model_config = ConfigDict(extra="allow")
id: str | None = None
name: str | None = None
nickName: str | None = None
profileUrl: str | None = None
verified: bool | None = None
signature: str | None = None
avatar: str | None = None
privateAccount: bool | None = None
fans: int | None = None
following: int | None = None
heart: int | None = None
video: int | None = None
class TikTokProfileItem(AuthorMeta):
"""A profile's public metadata, read from the page's rehydration blob.
Emitted even when the video listing is withheld from anonymous access, so a
blocked profile still yields its account data (name, followers, bio,
verification) instead of only an ErrorItem. Distinguishable from a video item
by the absence of ``webVideoUrl``/``text``.
"""
scrapedAt: str | None = None
def to_output(self) -> dict[str, Any]:
return self.model_dump(exclude_none=False)
class MusicMeta(BaseModel):
model_config = ConfigDict(extra="allow")
musicId: str | None = None
musicName: str | None = None
musicAuthor: str | None = None
musicOriginal: bool | None = None
playUrl: str | None = None
class VideoMeta(BaseModel):
model_config = ConfigDict(extra="allow")
height: int | None = None
width: int | None = None
duration: int | None = None
coverUrl: str | None = None
format: str | None = None
definition: str | None = None
class TikTokVideoItem(BaseModel):
"""A single scraped post (video or slideshow)."""
model_config = ConfigDict(extra="allow")
id: str | None = None
text: str | None = None
textLanguage: str | None = None
createTime: int | None = None
createTimeISO: str | None = None
isAd: bool | None = None
authorMeta: AuthorMeta = Field(default_factory=AuthorMeta)
musicMeta: MusicMeta = Field(default_factory=MusicMeta)
videoMeta: VideoMeta = Field(default_factory=VideoMeta)
webVideoUrl: str | None = None
mediaUrls: list[str] = Field(default_factory=list)
diggCount: int | None = None
shareCount: int | None = None
playCount: int | None = None
collectCount: int | None = None
commentCount: int | None = None
hashtags: list[dict[str, Any]] = Field(default_factory=list)
mentions: list[str] = Field(default_factory=list)
isSlideshow: bool | None = None
isPinned: bool | None = None
isSponsored: bool | None = None
scrapedAt: str | None = None
def to_output(self) -> dict[str, Any]:
return self.model_dump(exclude_none=False)
class CommentItem(BaseModel):
"""A single comment or reply under a post."""
model_config = ConfigDict(extra="allow")
id: str | None = None
text: str | None = None
videoWebUrl: str | None = None
diggCount: int | None = None
replyCommentTotal: int | None = None
createTime: int | None = None
createTimeISO: str | None = None
uid: str | None = None
uniqueId: str | None = None
nickName: str | None = None
avatar: str | None = None
repliesToId: str | None = None
scrapedAt: str | None = None
def to_output(self) -> dict[str, Any]:
return self.model_dump(exclude_none=False)
class ErrorItem(BaseModel):
"""Per-input failure, distinguished from normal items by ``errorCode``.
Mirrors the actor's convention so a private/deleted/empty target surfaces
as an item instead of silently vanishing from the results.
"""
model_config = ConfigDict(extra="allow")
url: str | None = None
input: str | None = None
error: str | None = None
errorCode: str | None = None
def to_output(self) -> dict[str, Any]:
return self.model_dump(exclude_none=False)

25
surfsense_backend/app/proprietary/platforms/tiktok/session/__init__.py Normal file
View file

@ -0,0 +1,25 @@
"""Cookie-warmed, rotate-on-block proxy session and page-fetch seam."""
from __future__ import annotations
from .client import fetch_html
from .errors import TikTokAccessBlockedError
from .listing import (
fetch_comments,
fetch_item_list,
fetch_trending,
fetch_user_search,
)
from .proxy import bind_proxy_holder, open_proxy_holder, proxy_session
__all__ = [
"TikTokAccessBlockedError",
"bind_proxy_holder",
"fetch_comments",
"fetch_html",
"fetch_item_list",
"fetch_trending",
"fetch_user_search",
"open_proxy_holder",
"proxy_session",
]

130
surfsense_backend/app/proprietary/platforms/tiktok/session/client.py Normal file
View file

@ -0,0 +1,130 @@
"""GET TikTok page HTML through a cookie-warmed, sticky-IP proxy session.
The warm-up mints TikTok's anonymous device cookie (``ttwid``) on the first
homepage hit; the target page then server-renders its rehydration blob. Rotates
the residential IP and re-warms on 403, backs off on 429, and raises
:class:`TikTokAccessBlockedError` only when every rotated IP refuses access.
"""
from __future__ import annotations
import asyncio
import logging
import random
from contextlib import suppress
from typing import Any
from scrapling.fetchers import AsyncFetcher
from app.utils.proxy import get_proxy_url
from .errors import TikTokAccessBlockedError
from .proxy import _REQUEST_TIMEOUT_S, _current_session, proxy_session
logger = logging.getLogger(__name__)
# 403 => IP blocked; rotate and re-warm. 429 => rate limited; back off same IP.
_ROTATE_STATUS = 403
_BACKOFF_STATUS = 429
_MAX_ROTATIONS = 3
_MAX_BACKOFFS = 4
_BACKOFF_BASE_S = 5.0
_HOME_URL = "https://www.tiktok.com/"
_TTWID_COOKIE = "ttwid"
_HEADERS = {"Accept-Language": "en-US,en;q=0.9"}
def _response_cookie_names(page: Any) -> set[str]:
cookies = getattr(page, "cookies", None)
return set(cookies.keys()) if isinstance(cookies, dict) else set()
def _page_html(page: Any) -> str | None:
for attr in ("text", "body"):
val = getattr(page, attr, None)
if isinstance(val, bytes):
val = val.decode("utf-8", "replace")
if isinstance(val, str) and val.strip():
return val
return None
async def warm_session(session: Any) -> bool:
"""Mint an anonymous ``ttwid`` cookie; ``True`` if the session can now fetch."""
with suppress(Exception):
page = await session.get(_HOME_URL, headers=_HEADERS)
if _TTWID_COOKIE in _response_cookie_names(page):
return True
return False
async def _get_page(session: Any, url: str) -> Any:
if session is not None:
return await session.get(url, headers=_HEADERS)
return await AsyncFetcher.get(
url,
headers=_HEADERS,
proxy=get_proxy_url(),
stealthy_headers=True,
timeout=_REQUEST_TIMEOUT_S,
)
async def fetch_html(url: str) -> str | None:
"""Return page HTML, or ``None`` on 404 / non-block failure."""
holder = _current_session.get()
if holder is None:
async with proxy_session():
return await fetch_html(url)
attempt = 0
backoffs = 0
while True:
session = holder.session
try:
if session is not None and not holder.warmed:
warmed_ok = await warm_session(session)
holder.warmed = True
if not warmed_ok:
if attempt < _MAX_ROTATIONS:
attempt += 1
await holder.rotate()
continue
raise TikTokAccessBlockedError(
f"could not warm session after {attempt} IP rotations: {url}"
)
await holder.pace()
page = await _get_page(session, url)
status = page.status
if status == 200:
return _page_html(page)
if status == 404:
return None
if status == _BACKOFF_STATUS and backoffs < _MAX_BACKOFFS:
backoffs += 1
delay = _BACKOFF_BASE_S * (2 ** (backoffs - 1))
logger.warning("[tiktok] 429 on %s; backing off %.1fs", url, delay)
await asyncio.sleep(delay + random.uniform(0, 1))
continue
if status == _ROTATE_STATUS and attempt < _MAX_ROTATIONS:
attempt += 1
await holder.rotate()
continue
if status == _ROTATE_STATUS:
raise TikTokAccessBlockedError(
f"TikTok refused {url} on {attempt} rotated IPs (403)"
)
logger.warning("[tiktok] GET %s returned %s", url, status)
return None
except TikTokAccessBlockedError:
raise
except Exception as e:
logger.warning("[tiktok] GET %s failed: %s", url, e)
if attempt < _MAX_ROTATIONS:
attempt += 1
await holder.rotate()
continue
return None

10
surfsense_backend/app/proprietary/platforms/tiktok/session/errors.py Normal file
View file

@ -0,0 +1,10 @@
"""Fetch-seam errors surfaced to the capability layer."""
from __future__ import annotations
class TikTokAccessBlockedError(RuntimeError):
"""Raised when every rotated IP is refused anonymous access.
Distinguishes a hard block from an empty result; the route maps it to 403.
"""

314
surfsense_backend/app/proprietary/platforms/tiktok/session/listing.py Normal file
View file

@ -0,0 +1,314 @@
"""Browser-driven listing fetch: let TikTok sign its own ``item_list`` XHRs.
Profile/hashtag/search listings need signed requests (``X-Gnarly``) whose
algorithm rev's monthly and reads a browser canvas fingerprint. Rather than port
and chase that signer, we load the page in the stealth browser we already run
(patchright-Chromium, via the web-crawler tier) and capture the itemStruct JSON
the page's own scripts fetch while scrolling. The browser is the client, so it
signs correctly for whatever version TikTok ships.
The pure response-shape parsing lives in :func:`items_from_response`; this module
is the untested browser-I/O glue (covered by the e2e smoke, not unit tests).
Needs a residential proxy; datacenter IPs get empty bodies and 429s. The profile
feed returns an empty 200 to headless sessions, so :func:`fetch_item_list` goes
headful only when ``CRAWL_HEADED_XVFB_ENABLED`` promises an Xvfb display — else it
stays headless and degrades to an ``ErrorItem`` instead of crashing on launch.
"""
from __future__ import annotations
import asyncio
import logging
from collections.abc import Callable
from typing import Any
from scrapling.fetchers import StealthyFetcher
from app.config import config
from app.proprietary.web_crawler.stealth import (
build_stealthy_kwargs,
get_stealth_config,
)
from app.utils.proxy import get_proxy_url
from ..extraction import (
comments_from_response,
items_from_response,
users_from_response,
)
logger = logging.getLogger(__name__)
ExtractFn = Callable[[Any], list[dict[str, Any]]]
# Drives the page after navigation to trigger/paginate the target XHRs, filling
# ``collected`` until it reaches ``target_count`` (or the interaction gives up).
InteractFn = Callable[[Any, list[dict[str, Any]], int], None]
# XHR paths that carry itemStructs for the three listing kinds.
_ITEM_LIST_MARKERS = (
"/api/post/item_list",
"/api/challenge/item_list",
"/api/search/",
)
# The user-search XHR carries account records (user_list), not itemStructs.
_USER_SEARCH_MARKERS = ("/api/search/user",)
# The Explore feed's trending videos arrive as ordinary itemStructs.
_EXPLORE_MARKERS = ("/api/explore/item_list",)
# The comment feed fires only after the comments panel is opened.
_COMMENT_MARKERS = ("/api/comment/list",)
_COMMENT_ICON_SELECTORS = (
'[data-e2e="comment-icon"]',
'[data-e2e="browse-comment"]',
)
# The comment icon hydrates a beat after DOM-ready; wait for it before clicking.
_COMMENT_ICON_WAIT_MS = 8000
# First comment page lands shortly after the click — don't declare "empty" early.
_COMMENT_FIRST_PAGE_MS = 3500
_HOME_URL = "https://www.tiktok.com/"
_MSTOKEN_COOKIE = "msToken"
# Bounded scroll: a dead page can't loop forever, and a live one stops early
# once enough items are captured.
_SCROLL_MAX_ROUNDS = 20
_SCROLL_SETTLE_MS = 1500
# Warm-up poll for the anonymous msToken cookie the item_list API requires.
_WARM_POLLS = 8
_WARM_POLL_MS = 500
def _has_mstoken(page: Any) -> bool:
try:
return any(c.get("name") == _MSTOKEN_COOKIE for c in page.context.cookies())
except Exception:
return False
def _dismiss_login_modal(page: Any) -> None:
"""Close the login modal that blocks scrolling; Escape as fallback.
Only the modal's own close button, never a generic dialog button (avoids
clicking "Log in").
"""
try:
closed = page.evaluate(
"""() => {
const btn = document.querySelector('[data-e2e="modal-close-inner-button"]');
if (btn) { btn.click(); return true; }
return false;
}"""
)
if not closed:
page.keyboard.press("Escape")
except Exception:
pass
def _scroll_page(page: Any, collected: list[dict[str, Any]], target_count: int) -> None:
"""Page down a listing feed until enough items are captured or it stops growing."""
last_height = 0
for _ in range(_SCROLL_MAX_ROUNDS):
if len(collected) >= target_count:
break
_dismiss_login_modal(page)
page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
page.wait_for_timeout(_SCROLL_SETTLE_MS)
height = page.evaluate("document.body.scrollHeight")
if not height or height <= last_height:
break
last_height = height
def _open_comments(page: Any) -> None:
"""Click the comment icon so the first ``/api/comment/list`` XHR fires.
The icon must be present and interactive first (the SPA hydrates it a beat
after DOM-ready), so we wait for it, then fall back to a JS click if the
normal click is intercepted (cookie banner / overlay).
"""
for selector in _COMMENT_ICON_SELECTORS:
try:
page.wait_for_selector(selector, timeout=_COMMENT_ICON_WAIT_MS)
except Exception:
continue
try:
page.click(selector, timeout=_COMMENT_ICON_WAIT_MS)
return
except Exception:
try:
page.eval_on_selector(selector, "el => el.click()")
return
except Exception:
continue
def _scroll_comments(
page: Any, collected: list[dict[str, Any]], target_count: int
) -> None:
"""Open the comments panel, then scroll its last comment into view to paginate.
Comment XHRs fire only after the panel is opened, and paging must scroll the
panel (not the page, which would advance the video feed), so we anchor on the
last ``comment-level-1`` element. ponytail: naive scroll-to-last paging,
bounded by ``_SCROLL_MAX_ROUNDS``; upgrade to container-height tracking if
deep threads under-fetch.
"""
_open_comments(page)
# The panel's first page lands a beat after the click; give it room before
# we decide there are no comments to page through.
page.wait_for_timeout(_COMMENT_FIRST_PAGE_MS)
for _ in range(_SCROLL_MAX_ROUNDS):
if len(collected) >= target_count:
break
moved = page.evaluate(
"""() => {
const items = document.querySelectorAll('[data-e2e="comment-level-1"]');
if (!items.length) return false;
items[items.length - 1].scrollIntoView({block: 'end'});
return true;
}"""
)
page.wait_for_timeout(_SCROLL_SETTLE_MS)
if not moved:
break
def _build_page_action(
collected: list[dict[str, Any]],
url: str,
target_count: int,
markers: tuple[str, ...],
extract: ExtractFn,
interact: InteractFn,
):
"""A sync ``page_action`` that warms the session then captures matching XHRs.
A cold context returns an empty body, so we first mint the anonymous
``msToken`` (homepage hit), then navigate to the target with the listener
already attached so page-one fires into it; ``interact`` pages the rest.
``markers``/``extract`` select which XHRs to keep and how to unwrap them.
"""
def _on_response(response: Any) -> None:
response_url = getattr(response, "url", "")
if not any(marker in response_url for marker in markers):
return
try:
body = response.json()
except Exception:
# An empty 200 (TikTok soft-block) or a body evicted before read.
return
collected.extend(extract(body))
def _warm(page: Any) -> None:
if _has_mstoken(page):
return
page.goto(_HOME_URL, wait_until="domcontentloaded")
for _ in range(_WARM_POLLS):
page.wait_for_timeout(_WARM_POLL_MS)
if _has_mstoken(page):
break
def page_action(page: Any) -> Any:
page.on("response", _on_response)
try:
_warm(page)
# Navigate (back) to the target with the listener attached and a
# token in hand, so the page-one XHR fires into the capture.
page.goto(url, wait_until="domcontentloaded")
page.wait_for_timeout(_SCROLL_SETTLE_MS)
interact(page, collected, target_count)
except Exception as exc:
logger.debug("[tiktok] capture interaction aborted: %s", exc)
return page
return page_action
def _fetch_sync(
url: str,
target_count: int,
markers: tuple[str, ...],
extract: ExtractFn,
interact: InteractFn,
*,
headless: bool = True,
) -> list[dict[str, Any]]:
collected: list[dict[str, Any]] = []
kwargs = build_stealthy_kwargs(get_stealth_config())
StealthyFetcher.fetch(
url,
headless=headless,
network_idle=False,
proxy=get_proxy_url(),
page_action=_build_page_action(
collected, url, target_count, markers, extract, interact
),
**kwargs,
)
return collected[:target_count]
async def fetch_item_list(page_url: str, target_count: int) -> list[dict[str, Any]]:
"""Return up to ``target_count`` itemStructs from a listing page's XHRs.
Headful when ``CRAWL_HEADED_XVFB_ENABLED`` promises a display (the profile feed
is empty to headless sessions); headless otherwise so launch never fails.
Retries an empty draw up to ``TIKTOK_LISTING_MAX_ATTEMPTS`` for a fresh exit IP.
"""
headless = not config.CRAWL_HEADED_XVFB_ENABLED
attempts = max(1, config.TIKTOK_LISTING_MAX_ATTEMPTS)
for attempt in range(1, attempts + 1):
items = await asyncio.to_thread(
_fetch_sync,
page_url,
target_count,
_ITEM_LIST_MARKERS,
items_from_response,
_scroll_page,
headless=headless,
)
if items or attempt == attempts:
return items
logger.info(
"[tiktok] empty item_list for %s (attempt %d/%d); retrying on a fresh exit IP",
page_url,
attempt,
attempts,
)
return []
async def fetch_user_search(page_url: str, target_count: int) -> list[dict[str, Any]]:
"""Return up to ``target_count`` ``user_info`` records from a user-search page."""
return await asyncio.to_thread(
_fetch_sync,
page_url,
target_count,
_USER_SEARCH_MARKERS,
users_from_response,
_scroll_page,
)
async def fetch_comments(page_url: str, target_count: int) -> list[dict[str, Any]]:
"""Return up to ``target_count`` raw comment records from a video page's XHRs."""
return await asyncio.to_thread(
_fetch_sync,
page_url,
target_count,
_COMMENT_MARKERS,
comments_from_response,
_scroll_comments,
)
async def fetch_trending(page_url: str, target_count: int) -> list[dict[str, Any]]:
"""Return up to ``target_count`` trending itemStructs from the Explore feed."""
return await asyncio.to_thread(
_fetch_sync,
page_url,
target_count,
_EXPLORE_MARKERS,
items_from_response,
_scroll_page,
)

113
surfsense_backend/app/proprietary/platforms/tiktok/session/proxy.py Normal file
View file

@ -0,0 +1,113 @@
"""Rotate-on-block sticky proxy session, bound per-flow via a ContextVar.
Reusing one keep-alive connection pins a single residential exit IP so the
warmed cookie jar (``ttwid``/``msToken``, bound to that IP) stays valid across
the warm-up and every subsequent fetch. Ported from the Reddit sibling; the
TikTok-specific warm-up lives in :mod:`client`.
"""
from __future__ import annotations
import asyncio
import logging
import random
import time
from contextlib import asynccontextmanager, suppress
from contextvars import ContextVar
from typing import Any
from scrapling.fetchers import FetcherSession
from app.utils.proxy import get_proxy_url
logger = logging.getLogger(__name__)
# Pace each sticky IP so a fast exit can't burst past TikTok's per-IP threshold.
_MIN_INTERVAL_S = 0.5
_PACE_JITTER_S = 0.25
# A healthy fetch lands in ~1-2s; cap a dead IP at one bounded wait before it
# falls through to a rotation.
_REQUEST_TIMEOUT_S = 15.0
_current_session: ContextVar[_RotatingSession | None] = ContextVar(
"tiktok_proxy_session", default=None
)
class _RotatingSession:
"""Owns one live ``FetcherSession`` (sticky IP); ``rotate()`` swaps the IP.
Used sequentially within a single flow (never shared across concurrent
tasks), so no locking is needed. ``session`` is ``None`` only when no proxy
is configured.
"""
def __init__(self) -> None:
self._cm: Any | None = None
self.session: Any | None = None
self.rotations = 0
self.warmed = False
self._last_at = 0.0
async def _open(self) -> None:
proxy = get_proxy_url()
self.warmed = False
if proxy is None:
self._cm = self.session = None
return
self._cm = FetcherSession(
proxy=proxy,
stealthy_headers=True,
impersonate="chrome",
timeout=_REQUEST_TIMEOUT_S,
)
self.session = await self._cm.__aenter__()
async def close(self) -> None:
if self._cm is not None:
with suppress(Exception):
await self._cm.__aexit__(None, None, None)
self._cm = self.session = None
async def rotate(self) -> Any | None:
"""Drop the current IP and connect through a fresh one."""
await self.close()
self.rotations += 1
await self._open()
logger.info("[tiktok] rotated proxy session (rotation #%d)", self.rotations)
return self.session
async def pace(self) -> None:
"""Sleep to hold this sticky IP under TikTok's per-IP rate threshold."""
wait = _MIN_INTERVAL_S - (time.monotonic() - self._last_at)
if wait > 0:
await asyncio.sleep(wait + random.uniform(0, _PACE_JITTER_S))
self._last_at = time.monotonic()
async def open_proxy_holder() -> _RotatingSession:
"""Open a warm rotate-on-block session holder (caller owns ``close()``)."""
holder = _RotatingSession()
await holder._open()
return holder
@asynccontextmanager
async def bind_proxy_holder(holder: _RotatingSession):
"""Route this task's fetches through ``holder`` for the enclosed block."""
token = _current_session.set(holder)
try:
yield holder
finally:
_current_session.reset(token)
@asynccontextmanager
async def proxy_session():
"""Open one reused, rotate-on-block proxy session for a continuation chain."""
holder = await open_proxy_holder()
try:
async with bind_proxy_holder(holder):
yield holder
finally:
await holder.close()

8
surfsense_backend/app/proprietary/platforms/tiktok/targets/__init__.py Normal file
View file

@ -0,0 +1,8 @@
"""TikTok URL classification into scrape targets."""
from __future__ import annotations
from .resolver import resolve_target
from .types import TargetKind, TikTokTarget
__all__ = ["TargetKind", "TikTokTarget", "resolve_target"]

50
surfsense_backend/app/proprietary/platforms/tiktok/targets/resolver.py Normal file
View file

@ -0,0 +1,50 @@
"""Classify a TikTok URL into a :class:`TikTokTarget`, or ``None``."""
from __future__ import annotations
from urllib.parse import parse_qs, unquote, urlparse
from .types import SearchSection, TikTokTarget
_TIKTOK_HOSTS = frozenset({"tiktok.com", "www.tiktok.com", "m.tiktok.com"})
_SEARCH_SECTIONS: frozenset[SearchSection] = frozenset({"video", "user"})
def _is_tiktok_host(hostname: str | None) -> bool:
return bool(hostname) and hostname.lower() in _TIKTOK_HOSTS
def resolve_target(url: str) -> TikTokTarget | None:
parsed = urlparse(url)
if not _is_tiktok_host(parsed.hostname):
return None
segments = [s for s in (parsed.path or "").split("/") if s]
if not segments:
return None
# Profile / video live under /@username[...].
if segments[0].startswith("@"):
username = segments[0][1:]
if not username:
return None
if len(segments) >= 3 and segments[1] == "video" and segments[2]:
return TikTokTarget("video", segments[2], url, username=username)
return TikTokTarget("profile", username, url)
if segments[0] == "tag" and len(segments) >= 2 and segments[1]:
return TikTokTarget("hashtag", unquote(segments[1]), url)
if segments[0] == "search":
query = parse_qs(parsed.query).get("q", [None])[0]
if not query:
return None
section = segments[1] if len(segments) >= 2 else None
return TikTokTarget(
"search",
unquote(query),
url,
section=section if section in _SEARCH_SECTIONS else None,
)
return None

25
surfsense_backend/app/proprietary/platforms/tiktok/targets/types.py Normal file
View file

@ -0,0 +1,25 @@
"""Scrape-target value object produced by URL classification."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Literal
TargetKind = Literal["video", "profile", "hashtag", "search", "trending"]
SearchSection = Literal["video", "user"]
@dataclass(frozen=True, slots=True)
class TikTokTarget:
"""One classified scrape target.
``value`` holds the kind-specific identifier: video id, username, hashtag
name, or search query. ``username`` is set for videos (needed to build the
canonical post URL). ``section`` narrows a search to videos or users.
"""
kind: TargetKind
value: str
url: str
username: str | None = None
section: SearchSection | None = None

2
surfsense_backend/app/routes/__init__.py
View file

@ -3,7 +3,9 @@ from fastapi import APIRouter, Depends
# Import verb namespaces for their registration side effects before the door builds.
import app.capabilities.google_maps
import app.capabilities.google_search
import app.capabilities.instagram
import app.capabilities.reddit
import app.capabilities.tiktok
import app.capabilities.web
import app.capabilities.youtube # noqa: F401
from app.automations.api import router as automations_router

15
surfsense_backend/app/services/model_connection_service.py
View file

@ -15,6 +15,7 @@ from app.db import Connection, Model, ModelSource
from app.services.model_resolver import ensure_v1, to_litellm
from app.services.openrouter_model_normalizer import normalize_openrouter_models
from app.services.provider_registry import Transport, provider_label, spec_for
from app.services.requesty_model_normalizer import normalize_requesty_models
logger = logging.getLogger(__name__)
@ -148,7 +149,7 @@ async def verify_connection(conn: Connection) -> VerifyResult:
if spec.transport == Transport.OLLAMA and base_url:
url = f"{base_url.rstrip('/')}/api/version"
elif spec.discovery in {"openai_models", "openrouter"} and base_url:
elif spec.discovery in {"openai_models", "openrouter", "requesty"} and base_url:
url = f"{ensure_v1(base_url)}/models"
elif spec.discovery == "anthropic_models" and base_url:
url = f"{base_url.rstrip('/')}/models"
@ -363,6 +364,16 @@ async def _openrouter_models(conn: Connection) -> list[dict[str, Any]]:
return normalize_openrouter_models(response.json().get("data", []))
async def _requesty_models(conn: Connection) -> list[dict[str, Any]]:
base_url = _base_url_or_default(conn) or "https://router.requesty.ai/v1"
async with httpx.AsyncClient(timeout=DISCOVERY_TIMEOUT_SECONDS) as client:
response = await client.get(
f"{ensure_v1(base_url)}/models", headers=_auth_headers(conn)
)
response.raise_for_status()
return normalize_requesty_models(response.json().get("data", []))
def _litellm_static_models(conn: Connection) -> list[dict[str, Any]]:
provider = conn.provider
prefix = spec_for(provider).litellm_prefix or provider
@ -446,6 +457,8 @@ async def discover_models(conn: Connection) -> list[dict[str, Any]]:
results = await _ollama_tags_then_show(conn)
elif spec.discovery == "openrouter":
results = await _openrouter_models(conn)
elif spec.discovery == "requesty":
results = await _requesty_models(conn)
elif spec.discovery == "anthropic_models":
results = await _discover_anthropic_models(conn)
elif spec.discovery == "openai_models":

19
surfsense_backend/app/services/provider_registry.py
View file

@ -23,6 +23,7 @@ DiscoveryKind = Literal[
"anthropic_models",
"bedrock_models",
"openrouter",
"requesty",
"static",
"none",
]
@ -78,6 +79,15 @@ REGISTRY: dict[str, ProviderSpec] = {
"bearer",
"OpenRouter",
),
"requesty": ProviderSpec(
Transport.OPENAI_COMPATIBLE,
"openai",
"requesty",
"https://router.requesty.ai/v1",
False,
"bearer",
"Requesty",
),
"openai_compatible": ProviderSpec(
Transport.OPENAI_COMPATIBLE,
"openai",
@ -87,6 +97,15 @@ REGISTRY: dict[str, ProviderSpec] = {
"bearer",
"OpenAI-compatible provider",
),
"openai_compatible_raw": ProviderSpec(
Transport.NATIVE,
"openai",
"none",
None,
True,
"bearer",
"OpenAI-compatible raw endpoint",
),
"lm_studio": ProviderSpec(
Transport.OPENAI_COMPATIBLE,
"openai",

1
surfsense_backend/app/services/quality_score.py
View file

@ -123,6 +123,7 @@ PROVIDER_PRESTIGE_YAML: dict[str, int] = {
"perplexity": 28,
"bedrock": 28,
"openrouter": 25,
"requesty": 25,
"ollama_chat": 12,
"custom": 12,
}

Some files were not shown because too many files have changed in this diff Show more