mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-26 23:51:14 +02:00
Merge pull request #1628 from MODSetter/dev
Release v0.0.35: Indeed & Walmart scrapers, Amazon EU marketplaces, OKF knowledge base export, run citations, Fix Image Attachments
This commit is contained in:
Rohan Verma
GitHub
commit
3b319ce44d
330 changed files with 13525 additions and 2863 deletions
|
|
@ -22,7 +22,7 @@
|
|||
|
||||
# SurfSense: La alternativa de código abierto a NotebookLM para la investigación de la web abierta
|
||||
|
||||
SurfSense es la **alternativa de código abierto a NotebookLM para agentes de IA**, una plataforma de investigación de la web abierta con conectores de datos en vivo. Tus agentes investigan la web en vivo con datos estructurados de **Reddit, YouTube, Instagram, TikTok, Amazon, Google Maps, Google Search y cualquier página de la web abierta**, a través de una única **API REST** o un **servidor MCP**. Agentes programados y activados por eventos convierten lo que encuentran en informes y alertas, y una base de conocimiento integrada mantiene cada hallazgo disponible para búsqueda con citas.
|
||||
SurfSense es la **alternativa de código abierto a NotebookLM para agentes de IA**, una plataforma de investigación de la web abierta con conectores de datos en vivo. Tus agentes investigan la web en vivo con datos estructurados de **Reddit, YouTube, Instagram, TikTok, Amazon, Google Maps, Google Search, Indeed y cualquier página de la web abierta**, a través de una única **API REST** o un **servidor MCP**. Agentes programados y activados por eventos convierten lo que encuentran en informes y alertas, y una base de conocimiento integrada mantiene cada hallazgo disponible para búsqueda con citas.
|
||||
|
||||
> [!NOTE]
|
||||
> **📢 Una nota para nuestros usuarios de la alternativa a NotebookLM**
|
||||
|
|
@ -60,6 +60,7 @@ Pregúntale a cualquier agente capaz "¿qué está diciendo Reddit sobre este pr
|
|||
| **TikTok** | Videos, comentarios, hashtags y perfiles sin aprobación de la Research API | [TikTok Scraper API](https://www.surfsense.com/tiktok) |
|
||||
| **Google Maps** | Lugares, calificaciones y reseñas para investigar negocios locales | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
|
||||
| **Google Search** | SERPs en vivo para investigación y monitoreo de búsquedas | [Google Search API](https://www.surfsense.com/google-search) |
|
||||
| **Indeed** | Ofertas de empleo públicas con salarios y descripciones completas, por búsqueda o empresa | [Indeed Scraper API](https://www.surfsense.com/indeed) |
|
||||
| **Amazon** | Datos públicos de productos: precios, calificaciones, ofertas, vendedores y rankings de más vendidos | [Amazon Product API](https://www.surfsense.com/amazon) |
|
||||
| **Web Crawl** (rastreo web) | Cualquier página de la web abierta como contenido limpio y estructurado | [Web Crawling API](https://www.surfsense.com/web-crawl) |
|
||||
| **Conectores MCP externos** | Conecta cualquier servidor MCP a tus agentes, con OAuth de un clic para Notion, Slack, Jira y más | [External MCP Connectors](https://www.surfsense.com/external-mcp-connectors) |
|
||||
|
|
@ -217,7 +218,7 @@ SurfSense es el único producto de código abierto que combina un espacio de tra
|
|||
|
||||
| Característica | Google NotebookLM | SurfSense |
|
||||
|---------|-------------------|-----------|
|
||||
| **Datos web en vivo para agentes** | No | Conectores de Reddit, YouTube, Instagram, TikTok, Amazon, Google Maps, Google Search y rastreo web vía API REST y MCP |
|
||||
| **Datos web en vivo para agentes** | No | Conectores de Reddit, YouTube, Instagram, TikTok, Amazon, Google Maps, Google Search, Indeed y rastreo web vía API REST y MCP |
|
||||
| **Servidor MCP** | No | Cada conector expuesto como herramienta nativa de agente, más servidores MCP propios con aplicaciones OAuth de un clic |
|
||||
| **Fuentes por Notebook** | 50 (gratis) a 600 (Ultra, $249.99/mes) | Ilimitadas |
|
||||
| **Número de Notebooks** | 100 (gratis) a 500 (niveles de pago) | Ilimitado |
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@
|
|||
|
||||
# SurfSense: ओपन वेब रिसर्च के लिए ओपन सोर्स NotebookLM विकल्प
|
||||
|
||||
SurfSense **AI एजेंट्स के लिए ओपन सोर्स NotebookLM विकल्प** है, लाइव डेटा कनेक्टर्स के साथ एक ओपन वेब रिसर्च प्लेटफ़ॉर्म। आपके एजेंट **Reddit, YouTube, Instagram, TikTok, Amazon, Google Maps, Google Search और ओपन वेब के किसी भी पेज** से स्ट्रक्चर्ड डेटा के साथ लाइव वेब पर रिसर्च करते हैं, वह भी एक ही **REST API** या **MCP सर्वर** के ज़रिए। शेड्यूल्ड और इवेंट-ट्रिगर्ड एजेंट अपनी खोजों को ब्रीफ़ और अलर्ट में बदलते हैं, और एक बिल्ट-इन नॉलेज बेस हर खोज को साइटेशन के साथ खोजने योग्य बनाए रखता है।
|
||||
SurfSense **AI एजेंट्स के लिए ओपन सोर्स NotebookLM विकल्प** है, लाइव डेटा कनेक्टर्स के साथ एक ओपन वेब रिसर्च प्लेटफ़ॉर्म। आपके एजेंट **Reddit, YouTube, Instagram, TikTok, Amazon, Google Maps, Google Search, Indeed और ओपन वेब के किसी भी पेज** से स्ट्रक्चर्ड डेटा के साथ लाइव वेब पर रिसर्च करते हैं, वह भी एक ही **REST API** या **MCP सर्वर** के ज़रिए। शेड्यूल्ड और इवेंट-ट्रिगर्ड एजेंट अपनी खोजों को ब्रीफ़ और अलर्ट में बदलते हैं, और एक बिल्ट-इन नॉलेज बेस हर खोज को साइटेशन के साथ खोजने योग्य बनाए रखता है।
|
||||
|
||||
> [!NOTE]
|
||||
> **📢 हमारे NotebookLM-विकल्प उपयोगकर्ताओं के लिए एक सूचना**
|
||||
|
|
@ -60,6 +60,7 @@ SurfSense **AI एजेंट्स के लिए ओपन सोर्स
|
|||
| **TikTok** | Research API अप्रूवल के बिना वीडियो, कमेंट, हैशटैग और प्रोफ़ाइल | [TikTok Scraper API](https://www.surfsense.com/tiktok) |
|
||||
| **Google Maps** | स्थानीय बिज़नेस रिसर्च के लिए स्थान, रेटिंग और रिव्यू | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
|
||||
| **Google Search** | सर्च रिसर्च और मॉनिटरिंग के लिए लाइव SERP | [Google Search API](https://www.surfsense.com/google-search) |
|
||||
| **Indeed** | सार्वजनिक नौकरी लिस्टिंग, सैलरी और पूरे विवरण के साथ, सर्च या कंपनी के अनुसार | [Indeed Scraper API](https://www.surfsense.com/indeed) |
|
||||
| **Amazon** | सार्वजनिक प्रोडक्ट डेटा: कीमतें, रेटिंग, ऑफ़र, विक्रेता और बेस्ट-सेलर रैंक | [Amazon Product API](https://www.surfsense.com/amazon) |
|
||||
| **Web Crawl** | ओपन वेब का कोई भी पेज साफ़-सुथरे, स्ट्रक्चर्ड कंटेंट के रूप में | [Web Crawling API](https://www.surfsense.com/web-crawl) |
|
||||
| **External MCP Connectors** | कोई भी MCP सर्वर अपने एजेंट्स से जोड़ें, Notion, Slack, Jira और अन्य के लिए वन-क्लिक OAuth के साथ | [External MCP Connectors](https://www.surfsense.com/external-mcp-connectors) |
|
||||
|
|
@ -217,7 +218,7 @@ SurfSense एकमात्र ओपन सोर्स प्रोडक्
|
|||
|
||||
| फ़ीचर | Google NotebookLM | SurfSense |
|
||||
|---------|-------------------|-----------|
|
||||
| **एजेंट्स के लिए लाइव वेब डेटा** | नहीं | REST API और MCP के ज़रिए Reddit, YouTube, Instagram, TikTok, Amazon, Google Maps, Google Search और वेब क्रॉल कनेक्टर |
|
||||
| **एजेंट्स के लिए लाइव वेब डेटा** | नहीं | REST API और MCP के ज़रिए Reddit, YouTube, Instagram, TikTok, Amazon, Google Maps, Google Search, Indeed और वेब क्रॉल कनेक्टर |
|
||||
| **MCP सर्वर** | नहीं | हर कनेक्टर नेटिव एजेंट टूल के रूप में उपलब्ध, साथ ही वन-क्लिक OAuth ऐप्स के साथ अपने MCP सर्वर लाने की सुविधा |
|
||||
| **प्रति नोटबुक स्रोत** | 50 (Free) से 600 (Ultra, $249.99/माह) | असीमित |
|
||||
| **नोटबुक की संख्या** | 100 (Free) से 500 (सशुल्क टियर) | असीमित |
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@
|
|||
|
||||
# SurfSense: The Open-Source NotebookLM Alternative for Open Web Research
|
||||
|
||||
SurfSense is the **open-source NotebookLM alternative for AI agents**, an open web research platform with live data connectors. Your agents research the live web with structured data from **Reddit, YouTube, Instagram, TikTok, Amazon, Google Maps, Google Search, and any page on the open web**, through one **REST API** or **MCP server**. Scheduled and event-triggered agents turn what they find into briefs and alerts, and a built-in knowledge base keeps every finding searchable with citations.
|
||||
SurfSense is the **open-source NotebookLM alternative for AI agents**, an open web research platform with live data connectors. Your agents research the live web with structured data from **Reddit, YouTube, Instagram, TikTok, Amazon, Google Maps, Google Search, Indeed, and any page on the open web**, through one **REST API** or **MCP server**. Scheduled and event-triggered agents turn what they find into briefs and alerts, and a built-in knowledge base keeps every finding searchable with citations.
|
||||
|
||||
> [!NOTE]
|
||||
> **📢 A note for our NotebookLM-alternative users**
|
||||
|
|
@ -60,6 +60,7 @@ Ask any capable agent "what is Reddit saying about this product since launch?" o
|
|||
| **TikTok** | Videos, comments, hashtags, and profiles without Research API approval | [TikTok Scraper API](https://www.surfsense.com/tiktok) |
|
||||
| **Google Maps** | Places, ratings, and reviews for local business research | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
|
||||
| **Google Search** | Live SERPs for search research and monitoring | [Google Search API](https://www.surfsense.com/google-search) |
|
||||
| **Indeed** | Public job postings with salaries and full descriptions, by search or company | [Indeed Scraper API](https://www.surfsense.com/indeed) |
|
||||
| **Amazon** | Public product data: prices, ratings, offers, sellers, and best-seller ranks | [Amazon Product API](https://www.surfsense.com/amazon) |
|
||||
| **Web Crawl** | Any page on the open web as clean, structured content | [Web Crawling API](https://www.surfsense.com/web-crawl) |
|
||||
| **External MCP Connectors** | Bring any MCP server to your agents, with one-click OAuth for Notion, Slack, Jira, and more | [External MCP Connectors](https://www.surfsense.com/external-mcp-connectors) |
|
||||
|
|
@ -217,7 +218,7 @@ Still comparing us as a NotebookLM alternative? Here is the honest breakdown.
|
|||
|
||||
| Feature | Google NotebookLM | SurfSense |
|
||||
|---------|-------------------|-----------|
|
||||
| **Live web data for agents** | No | Reddit, YouTube, Instagram, TikTok, Amazon, Google Maps, Google Search, and web crawl connectors via REST API and MCP |
|
||||
| **Live web data for agents** | No | Reddit, YouTube, Instagram, TikTok, Amazon, Google Maps, Google Search, Indeed, and web crawl connectors via REST API and MCP |
|
||||
| **MCP server** | No | Every connector exposed as a native agent tool, plus bring-your-own MCP servers with one-click OAuth apps |
|
||||
| **Sources per Notebook** | 50 (Free) to 600 (Ultra, $249.99/mo) | Unlimited |
|
||||
| **Number of Notebooks** | 100 (Free) to 500 (paid tiers) | Unlimited |
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@
|
|||
|
||||
# SurfSense: A Alternativa Open Source ao NotebookLM para Pesquisa na Web Aberta
|
||||
|
||||
O SurfSense é a **alternativa open source ao NotebookLM para agentes de IA**, uma plataforma de pesquisa na web aberta com conectores de dados ao vivo. Seus agentes pesquisam a web ao vivo com dados estruturados do **Reddit, YouTube, Instagram, TikTok, Amazon, Google Maps, Google Search e de qualquer página da web aberta**, por meio de uma única **API REST** ou de um **servidor MCP**. Agentes agendados ou acionados por eventos transformam o que encontram em relatórios e alertas, e uma base de conhecimento integrada mantém cada descoberta pesquisável, com citações.
|
||||
O SurfSense é a **alternativa open source ao NotebookLM para agentes de IA**, uma plataforma de pesquisa na web aberta com conectores de dados ao vivo. Seus agentes pesquisam a web ao vivo com dados estruturados do **Reddit, YouTube, Instagram, TikTok, Amazon, Google Maps, Google Search, Indeed e de qualquer página da web aberta**, por meio de uma única **API REST** ou de um **servidor MCP**. Agentes agendados ou acionados por eventos transformam o que encontram em relatórios e alertas, e uma base de conhecimento integrada mantém cada descoberta pesquisável, com citações.
|
||||
|
||||
> [!NOTE]
|
||||
> **📢 Um recado para nossos usuários que buscavam uma alternativa ao NotebookLM**
|
||||
|
|
@ -60,6 +60,7 @@ Pergunte a qualquer agente capaz "o que o Reddit está dizendo sobre este produt
|
|||
| **TikTok** | Vídeos, comentários, hashtags e perfis sem aprovação da Research API | [TikTok Scraper API](https://www.surfsense.com/tiktok) |
|
||||
| **Google Maps** | Estabelecimentos, avaliações e reviews para pesquisa de negócios locais | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
|
||||
| **Google Search** | SERPs ao vivo para pesquisa e monitoramento de buscas | [Google Search API](https://www.surfsense.com/google-search) |
|
||||
| **Indeed** | Vagas públicas com salários e descrições completas, por busca ou empresa | [Indeed Scraper API](https://www.surfsense.com/indeed) |
|
||||
| **Amazon** | Dados públicos de produtos: preços, avaliações, ofertas, vendedores e rankings de mais vendidos | [Amazon Product API](https://www.surfsense.com/amazon) |
|
||||
| **Web Crawl** (rastreamento web) | Qualquer página da web aberta como conteúdo limpo e estruturado | [Web Crawling API](https://www.surfsense.com/web-crawl) |
|
||||
| **Conectores MCP externos** | Traga qualquer servidor MCP para seus agentes, com OAuth em um clique para Notion, Slack, Jira e outros | [External MCP Connectors](https://www.surfsense.com/external-mcp-connectors) |
|
||||
|
|
@ -217,7 +218,7 @@ Ainda nos comparando como alternativa ao NotebookLM? Aqui está o comparativo ho
|
|||
|
||||
| Recurso | Google NotebookLM | SurfSense |
|
||||
|---------|-------------------|-----------|
|
||||
| **Dados da web ao vivo para agentes** | Não | Conectores de Reddit, YouTube, Instagram, TikTok, Amazon, Google Maps, Google Search e rastreamento web via API REST e MCP |
|
||||
| **Dados da web ao vivo para agentes** | Não | Conectores de Reddit, YouTube, Instagram, TikTok, Amazon, Google Maps, Google Search, Indeed e rastreamento web via API REST e MCP |
|
||||
| **Servidor MCP** | Não | Cada conector exposto como ferramenta nativa de agente, além de servidores MCP próprios com apps OAuth em um clique |
|
||||
| **Fontes por Notebook** | 50 (gratuito) a 600 (Ultra, US$ 249,99/mês) | Ilimitadas |
|
||||
| **Número de Notebooks** | 100 (gratuito) a 500 (planos pagos) | Ilimitado |
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@
|
|||
|
||||
# SurfSense：面向开放网络研究的开源 NotebookLM 替代品
|
||||
|
||||
SurfSense 是**面向 AI 智能体的开源 NotebookLM 替代品**，一个配备实时数据连接器的开放网络研究平台。你的智能体可以通过一个 **REST API** 或 **MCP 服务器**，利用来自 **Reddit、YouTube、Instagram、TikTok、Amazon、Google Maps、Google Search 以及开放网络上任意页面**的结构化数据研究实时网络。定时和事件触发的智能体会把发现的内容转化为简报和预警，内置的知识库则让每一条发现都可搜索、可引用。
|
||||
SurfSense 是**面向 AI 智能体的开源 NotebookLM 替代品**，一个配备实时数据连接器的开放网络研究平台。你的智能体可以通过一个 **REST API** 或 **MCP 服务器**，利用来自 **Reddit、YouTube、Instagram、TikTok、Amazon、Google Maps、Google Search、Indeed 以及开放网络上任意页面**的结构化数据研究实时网络。定时和事件触发的智能体会把发现的内容转化为简报和预警，内置的知识库则让每一条发现都可搜索、可引用。
|
||||
|
||||
> [!NOTE]
|
||||
> **📢 致我们的 NotebookLM 替代品用户**
|
||||
|
|
@ -60,6 +60,7 @@ SurfSense 是**面向 AI 智能体的开源 NotebookLM 替代品**，一个配
|
|||
| **TikTok** | 视频、评论、话题标签和主页，无需 Research API 审批 | [TikTok Scraper API](https://www.surfsense.com/tiktok) |
|
||||
| **Google Maps** | 地点、评分和评论，用于本地商户研究 | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
|
||||
| **Google Search** | 实时搜索结果页，用于搜索研究和监控 | [Google Search API](https://www.surfsense.com/google-search) |
|
||||
| **Indeed** | 公开职位信息，含薪资与完整职位描述，按搜索或公司抓取 | [Indeed Scraper API](https://www.surfsense.com/indeed) |
|
||||
| **Amazon** | 公开商品数据：价格、评分、报价、卖家和畅销榜排名 | [Amazon Product API](https://www.surfsense.com/amazon) |
|
||||
| **Web Crawl** | 把开放网络上的任意页面转为干净、结构化的内容 | [Web Crawling API](https://www.surfsense.com/web-crawl) |
|
||||
| **外部 MCP 连接器** | 将任意 MCP 服务器接入你的智能体，Notion、Slack、Jira 等支持一键 OAuth | [External MCP Connectors](https://www.surfsense.com/external-mcp-connectors) |
|
||||
|
|
@ -217,7 +218,7 @@ SurfSense 是唯一一款把面向人的 NotebookLM 式研究工作区与面向
|
|||
|
||||
| 功能 | Google NotebookLM | SurfSense |
|
||||
|---------|-------------------|-----------|
|
||||
| **面向智能体的实时网络数据** | 无 | 通过 REST API 和 MCP 提供 Reddit、YouTube、Instagram、TikTok、Amazon、Google Maps、Google Search 和网页爬取连接器 |
|
||||
| **面向智能体的实时网络数据** | 无 | 通过 REST API 和 MCP 提供 Reddit、YouTube、Instagram、TikTok、Amazon、Google Maps、Google Search、Indeed 和网页爬取连接器 |
|
||||
| **MCP 服务器** | 无 | 每个连接器都作为原生智能体工具暴露，还可自带 MCP 服务器并使用一键 OAuth 应用 |
|
||||
| **每个笔记本的来源数** | 50 个（免费版）至 600 个（Ultra 版，249.99 美元/月） | 无限制 |
|
||||
| **笔记本数量** | 100 个（免费版）至 500 个（付费档位） | 无限制 |
|
||||
|
|
|
|||
2
VERSION
2
VERSION
|
|
@ -1 +1 @@
|
|||
0.0.34
|
||||
0.0.35
|
||||
|
|
|
|||
|
|
@ -392,6 +392,15 @@ STT_SERVICE=local/base
|
|||
# OTEL_HTTP_PORT=4318
|
||||
# OTEL_HEALTH_PORT=13133
|
||||
|
||||
# PostHog product analytics (server-side). Opt-in like OTel: leave
|
||||
# POSTHOG_API_KEY unset for zero telemetry. Passed to backend/worker/beat via
|
||||
# env_file, so no compose changes are needed. Use the SAME project key as the
|
||||
# frontend's NEXT_PUBLIC_POSTHOG_KEY so server events merge onto the persons the
|
||||
# web app already identifies by user id.
|
||||
# POSTHOG_API_KEY=phc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
||||
# POSTHOG_HOST=https://us.i.posthog.com
|
||||
# POSTHOG_AI_PRIVACY_MODE=true # false ships LLM prompt/completion bodies to PostHog
|
||||
|
||||
# ------------------------------------------------------------------------------
|
||||
# Advanced (optional)
|
||||
# ------------------------------------------------------------------------------
|
||||
|
|
@ -441,7 +450,7 @@ SURFSENSE_ENABLE_DOOM_LOOP=true
|
|||
# WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE=3000
|
||||
|
||||
# Debit the credit wallet per *item returned* by the platform-native scrapers
|
||||
# (Reddit, Google Search, Google Maps, Amazon, YouTube). Default FALSE keeps scraping
|
||||
# (Reddit, Google Search, Google Maps, Amazon, Walmart, YouTube). Default FALSE keeps scraping
|
||||
# effectively free for self-hosted installs. Each rate is micro-USD per item,
|
||||
# config-driven: <KEY> = round(USD_per_1000_items * 1_000). Defaults sit
|
||||
# at/above Apify's first-party actor rates (we charge no subscription tiers,
|
||||
|
|
@ -458,6 +467,9 @@ SURFSENSE_ENABLE_DOOM_LOOP=true
|
|||
# TIKTOK_MICROS_PER_VIDEO=3500
|
||||
# TIKTOK_MICROS_PER_USER=2500
|
||||
# TIKTOK_MICROS_PER_COMMENT=1500
|
||||
# INDEED_SCRAPE_MICROS_PER_JOB=3500
|
||||
# WALMART_MICROS_PER_PRODUCT=3500
|
||||
# WALMART_MICROS_PER_REVIEW=1500
|
||||
|
||||
# Safety ceiling on per-call premium reservation, in micro-USD ($1.00 default).
|
||||
# QUOTA_MAX_RESERVE_MICROS=1000000
|
||||
|
|
|
|||
|
|
@ -277,8 +277,8 @@ MICROS_PER_PAGE=1000
|
|||
# WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE=3000
|
||||
|
||||
# Debit the credit wallet per *item returned* by the platform-native scrapers
|
||||
# (Reddit, Google Search, Google Maps, Amazon, YouTube). Default FALSE keeps
|
||||
# scraping effectively free for self-hosted/OSS installs; hosted set TRUE.
|
||||
# (Reddit, Google Search, Google Maps, Amazon, Walmart, YouTube). Default FALSE
|
||||
# keeps scraping effectively free for self-hosted/OSS installs; hosted set TRUE.
|
||||
# Each rate is micro-USD per item, fully config-driven (no hardcoded rate):
|
||||
# <KEY> = round(USD_per_1000_items * 1_000)
|
||||
# 3500 == $3.50/1000 | 5000 == $5/1000 | 2000 == $2/1000
|
||||
|
|
@ -298,9 +298,15 @@ MICROS_PER_PAGE=1000
|
|||
# TIKTOK_MICROS_PER_VIDEO=3500
|
||||
# TIKTOK_MICROS_PER_USER=2500
|
||||
# TIKTOK_MICROS_PER_COMMENT=1500
|
||||
# INDEED_SCRAPE_MICROS_PER_JOB=3500
|
||||
# Browser-listing retries when a feed is empty (profile feed is withheld from
|
||||
# flagged IPs; each retry draws a fresh rotating exit IP). Set to 1 for a static IP.
|
||||
# TIKTOK_LISTING_MAX_ATTEMPTS=3
|
||||
# Walmart products (server-rendered JSON behind residential proxies) priced like
|
||||
# Amazon; reviews are 10/page (many light requests) so priced on the per-review
|
||||
# market like Google Maps reviews.
|
||||
# WALMART_MICROS_PER_PRODUCT=3500
|
||||
# WALMART_MICROS_PER_REVIEW=1500
|
||||
|
||||
# Low-balance warning threshold (micro-USD), surfaced to the UI. Default $0.50.
|
||||
CREDIT_LOW_BALANCE_WARNING_MICROS=500000
|
||||
|
|
@ -549,6 +555,14 @@ LANGSMITH_PROJECT=surfsense
|
|||
# OTEL_METRIC_EXPORT_INTERVAL=300000 # ms; 5 minutes
|
||||
# OTEL_SDK_DISABLED=true # emergency kill-switch
|
||||
|
||||
# Observability - PostHog product analytics (server-side)
|
||||
# Opt-in like OTel: leave POSTHOG_API_KEY unset for zero telemetry. Use the
|
||||
# SAME project key as the frontend's NEXT_PUBLIC_POSTHOG_KEY so server events
|
||||
# merge onto the persons the web app already identifies by user id.
|
||||
# POSTHOG_API_KEY=phc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
||||
# POSTHOG_HOST=https://us.i.posthog.com
|
||||
# POSTHOG_AI_PRIVACY_MODE=true # false ships LLM prompt/completion bodies to PostHog
|
||||
|
||||
# Skills + subagents
|
||||
# SURFSENSE_ENABLE_SKILLS=false
|
||||
# SURFSENSE_ENABLE_SPECIALIZED_SUBAGENTS=false
|
||||
|
|
|
|||
|
|
@ -36,9 +36,11 @@ SUBAGENT_TO_REQUIRED_CONNECTOR_MAP: dict[str, frozenset[str]] = {
|
|||
"youtube": frozenset(),
|
||||
"google_maps": frozenset(),
|
||||
"google_search": frozenset(),
|
||||
"indeed": frozenset(),
|
||||
"reddit": frozenset(),
|
||||
"instagram": frozenset(),
|
||||
"tiktok": frozenset(),
|
||||
"walmart": frozenset(),
|
||||
"mcp_discovery": frozenset(
|
||||
{
|
||||
"SLACK_CONNECTOR",
|
||||
|
|
|
|||
|
|
@ -1,8 +1,9 @@
|
|||
<citations>
|
||||
Cite with one token: the bracket label `[n]`. Every citable result —
|
||||
prose from a `task` knowledge_base/research specialist (including the
|
||||
knowledge_base specialist's `[n]`-labelled workspace findings) — already
|
||||
carries `[n]` labels on a single shared count.
|
||||
knowledge_base specialist's `[n]`-labelled workspace findings) and
|
||||
scraper specialists' run-backed findings — already carries `[n]` labels
|
||||
on a single shared count.
|
||||
Those labels are the only citation you write; the server resolves each one
|
||||
back to its source after the turn.
|
||||
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ are changing, and what is being published across the open web — and to put
|
|||
that research to work alongside their own knowledge base.
|
||||
|
||||
You do this by dispatching **specialist subagents** via the `task` tool:
|
||||
- **Live web data** — Reddit, YouTube, Instagram, TikTok, Amazon, Google
|
||||
- **Live web data** — Reddit, YouTube, Instagram, TikTok, Amazon, Walmart, Google
|
||||
Maps, Google Search, and the web crawler return structured, current
|
||||
platform data (posts, comments, transcripts, videos, products, reviews,
|
||||
SERPs, full page content).
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ are changing, and what is being published across the open web — and to put
|
|||
that research to work alongside the team's shared knowledge base.
|
||||
|
||||
You do this by dispatching **specialist subagents** via the `task` tool:
|
||||
- **Live web data** — Reddit, YouTube, Instagram, TikTok, Amazon, Google
|
||||
- **Live web data** — Reddit, YouTube, Instagram, TikTok, Amazon, Walmart, Google
|
||||
Maps, Google Search, and the web crawler return structured, current
|
||||
platform data (posts, comments, transcripts, videos, products, reviews,
|
||||
SERPs, full page content).
|
||||
|
|
|
|||
|
|
@ -2,7 +2,8 @@
|
|||
CRITICAL — ground factual answers in what you actually receive this turn:
|
||||
- **live platform data** via the market specialists —
|
||||
`task(reddit, ...)`, `task(youtube, ...)`, `task(instagram, ...)`,
|
||||
`task(tiktok, ...)`, `task(amazon, ...)`, `task(google_maps, ...)`,
|
||||
`task(tiktok, ...)`, `task(amazon, ...)`, `task(walmart, ...)`,
|
||||
`task(google_maps, ...)`,
|
||||
`task(google_search, ...)`, `task(web_crawler, ...)`. Anything about
|
||||
competitors, markets, rankings, reviews, or audience sentiment is answered
|
||||
from what these return **this turn**, never from your training data: your
|
||||
|
|
|
|||
|
|
@ -32,7 +32,9 @@ about a brand, product, or topic is answered from the platform where they
|
|||
say it — `task(reddit, …)` for community discussion and threads,
|
||||
`task(youtube, …)` for video content, transcripts, and comment sections,
|
||||
`task(tiktok, …)` for short-form video trends by hashtag or search,
|
||||
`task(google_maps, …)` for customer reviews of physical businesses. Web
|
||||
`task(google_maps, …)` for customer reviews of physical businesses,
|
||||
`task(amazon, …)` / `task(walmart, …)` for product ratings and customer
|
||||
reviews of retail products (Walmart pages the full review history). Web
|
||||
search only finds articles *about* the conversation; the platform
|
||||
specialists return the conversation itself, structured and current. For
|
||||
competitive questions ("what are people saying about X", "how is Y
|
||||
|
|
|
|||
|
|
@ -1,10 +1,11 @@
|
|||
"""Map a registered citation to the frontend ``[citation:<payload>]`` payload.
|
||||
|
||||
The citation renderer understands a chunk id (``42``), a negative chunk id for
|
||||
anonymous uploads (``-3``), and a URL. This is the seam that turns a server-side
|
||||
source into one the renderer can resolve; it grows as more source kinds become
|
||||
renderable. Kinds with no renderable form yet return ``None`` so the marker is
|
||||
dropped rather than emitted broken.
|
||||
anonymous uploads (``-3``), a URL, and a scraper-run handle (``run_<uuid>``).
|
||||
This is the seam that turns a server-side source into one the renderer can
|
||||
resolve; it grows as more source kinds become renderable. Kinds with no
|
||||
renderable form yet return ``None`` so the marker is dropped rather than
|
||||
emitted broken.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -22,6 +23,9 @@ def to_frontend_payload(entry: CitationEntry) -> str | None:
|
|||
case CitationSourceType.WEB_RESULT:
|
||||
url = locator.get("url")
|
||||
return url or None
|
||||
case CitationSourceType.RUN:
|
||||
run_id = locator.get("run_id")
|
||||
return str(run_id) if run_id else None
|
||||
case _:
|
||||
# Connector items and chat turns have no client-side renderer yet
|
||||
# (the frontend resolves only chunk ids and URLs), so they stay
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@ class CitationSourceType(StrEnum):
|
|||
WEB_RESULT = "web_result"
|
||||
CHAT_TURN = "chat_turn"
|
||||
ANON_CHUNK = "anon_chunk"
|
||||
RUN = "run"
|
||||
|
||||
|
||||
class CitationEntry(BaseModel):
|
||||
|
|
|
|||
|
|
@ -0,0 +1 @@
|
|||
"""``indeed`` builtin subagent: structured Indeed job postings."""
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
"""``indeed`` route: ``SurfSenseSubagentSpec`` builder for deepagents."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.language_models import BaseChatModel
|
||||
from langchain_core.tools import BaseTool
|
||||
|
||||
from app.agents.chat.multi_agent_chat.subagents.shared.md_file_reader import (
|
||||
read_md_file,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.subagents.shared.spec import SurfSenseSubagentSpec
|
||||
from app.agents.chat.multi_agent_chat.subagents.shared.subagent_builder import (
|
||||
pack_subagent,
|
||||
)
|
||||
|
||||
from .tools.index import NAME, RULESET, load_tools
|
||||
|
||||
|
||||
def build_subagent(
|
||||
*,
|
||||
dependencies: dict[str, Any],
|
||||
model: BaseChatModel | None = None,
|
||||
middleware_stack: dict[str, Any] | None = None,
|
||||
mcp_tools: list[BaseTool] | None = None,
|
||||
) -> SurfSenseSubagentSpec:
|
||||
tools = [*load_tools(dependencies=dependencies), *(mcp_tools or [])]
|
||||
description = (
|
||||
read_md_file(__package__, "description").strip()
|
||||
or "Pulls structured job postings from Indeed search and company pages."
|
||||
)
|
||||
system_prompt = read_md_file(__package__, "system_prompt").strip()
|
||||
return pack_subagent(
|
||||
name=NAME,
|
||||
description=description,
|
||||
system_prompt=system_prompt,
|
||||
tools=tools,
|
||||
ruleset=RULESET,
|
||||
dependencies=dependencies,
|
||||
model=model,
|
||||
middleware_stack=middleware_stack,
|
||||
)
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
Indeed jobs specialist: pulls structured job postings — title, company, location, salary (range, currency, period), job types, benefits, remote/hybrid flag, posting age, apply URL, and the full job description. Discovers jobs by search query (with country, location, radius, job type, experience level, remote/hybrid, and date-posted filters), or scrapes a known Indeed search, company jobs, or single job URL as-is. Optionally fetches each job's detail page for the full description.
|
||||
Use whenever the task is to find or gather job listings from Indeed — openings for a role, hiring at a company, salaries for a title in a location, or remote roles in a field. Triggers include "find jobs for X", "who is hiring X", "data analyst jobs in Y", "remote X roles", and scraping a specific Indeed URL. Not for general web pages (use the web crawling specialist), Google results (use the Google Search specialist), or other job boards.
|
||||
|
|
@ -0,0 +1,65 @@
|
|||
You are the SurfSense Indeed sub-agent.
|
||||
You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis.
|
||||
|
||||
<goal>
|
||||
Answer the delegated question from live Indeed job data gathered with your verb, comparing against earlier results already in this conversation when the task calls for it.
|
||||
</goal>
|
||||
|
||||
<available_tools>
|
||||
- `indeed_scrape`
|
||||
- `read_run` / `search_run` (free readers for stored scrape output)
|
||||
</available_tools>
|
||||
|
||||
<playbook>
|
||||
- Finding jobs for a role: call `indeed_scrape` with `search_queries`; narrow with `location`, `country`, `job_type`, `level`, `remote`, `radius`, and `from_days`.
|
||||
- Scraping a specific Indeed URL: pass a search, company jobs, or single job URL in `urls`.
|
||||
- Full descriptions: set `scrape_job_details=true` to fetch each job's detail page (slower: one extra load per job). Leave it false when the listing snippet is enough.
|
||||
- Cost model: Indeed is latency-bound. A cold session spends minutes solving Cloudflare, and each query returns only its first page (~15 jobs) — anonymous pagination is gated, so `max_items` above ~15/query buys nothing. Every extra phrasing adds wait, not depth.
|
||||
- Default to ONE focused query. Only add phrasings when the role genuinely needs variety, and then put at most 2–3 in a SINGLE call's `search_queries` (they reuse one warmed session); never make separate `indeed_scrape` calls, which each pay the cold-start cost.
|
||||
- Prefer returning the first page's on-topic hits promptly over exhaustive coverage. If a single call under-delivers against a large requested N, return `status=partial` noting the ~one-page-per-query ceiling — do not chase it with more calls.
|
||||
<include snippet="run_reader"/>
|
||||
- Comparison requests: pull the current results, compare against prior values already in this conversation's earlier tool results, and report concrete deltas (added, removed, salary/rank changes).
|
||||
</playbook>
|
||||
|
||||
<tool_policy>
|
||||
- Use only tools in `<available_tools>`.
|
||||
- Report only results present in the tool output. Never invent titles, companies, salaries, locations, or description text.
|
||||
</tool_policy>
|
||||
|
||||
<out_of_scope>
|
||||
- Do not read arbitrary web pages — that belongs to the web crawling specialist.
|
||||
- Do not generate deliverables or perform connector mutations; return findings for the supervisor to act on.
|
||||
- Google results belong to the Google Search specialist; other job boards are out of scope.
|
||||
</out_of_scope>
|
||||
|
||||
<safety>
|
||||
- Report uncertainty explicitly when evidence is incomplete or conflicting.
|
||||
- Never present unverified claims as facts.
|
||||
</safety>
|
||||
|
||||
<failure_policy>
|
||||
- Underspecified request — no usable query or URL — return `status=blocked` with the missing fields.
|
||||
- Tool failure: return `status=error` with a concise recovery `next_step`.
|
||||
- No useful evidence: return `status=blocked` with a narrower query or the scope you still need.
|
||||
</failure_policy>
|
||||
|
||||
<output_contract>
|
||||
Return **only** one JSON object (no markdown/prose):
|
||||
{
|
||||
"status": "success" | "partial" | "blocked" | "error",
|
||||
"action_summary": string,
|
||||
"evidence": {
|
||||
"findings": string[],
|
||||
"sources": string[],
|
||||
"confidence": "high" | "medium" | "low"
|
||||
},
|
||||
"next_step": string | null,
|
||||
"missing_fields": string[] | null,
|
||||
"assumptions": string[] | null
|
||||
}
|
||||
<include snippet="output_contract_base"/>
|
||||
Route-specific rules:
|
||||
- `evidence.findings`: one entry per distinct job or delta — a single sentence each; do not paste raw payloads. Max 10 entries, unless the delegated task asks for N items: then return up to N (each backed by a real scraped result, never padded).
|
||||
- `evidence.sources`: one Indeed job URL per finding when applicable, same cap as findings. List each URL once.
|
||||
</output_contract>
|
||||
</output>
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
"""``indeed`` sub-agent tools: the Indeed scrape capability verb."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.tools import BaseTool
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.permissions import Ruleset
|
||||
from app.capabilities.core.access.agent import build_capability_tools
|
||||
from app.capabilities.indeed.scrape.definition import INDEED_SCRAPE
|
||||
|
||||
NAME = "indeed"
|
||||
|
||||
RULESET = Ruleset(origin=NAME, rules=[])
|
||||
|
||||
_CI_VERBS = [INDEED_SCRAPE]
|
||||
|
||||
|
||||
def load_tools(
|
||||
*, dependencies: dict[str, Any] | None = None, **kwargs: Any
|
||||
) -> list[BaseTool]:
|
||||
d = {**(dependencies or {}), **kwargs}
|
||||
return build_capability_tools(
|
||||
workspace_id=d.get("workspace_id"),
|
||||
capabilities=_CI_VERBS,
|
||||
)
|
||||
|
|
@ -0,0 +1 @@
|
|||
"""``walmart`` builtin subagent: structured public Walmart product data and reviews."""
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
"""``walmart`` route: ``SurfSenseSubagentSpec`` builder for deepagents."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.language_models import BaseChatModel
|
||||
from langchain_core.tools import BaseTool
|
||||
|
||||
from app.agents.chat.multi_agent_chat.subagents.shared.md_file_reader import (
|
||||
read_md_file,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.subagents.shared.spec import SurfSenseSubagentSpec
|
||||
from app.agents.chat.multi_agent_chat.subagents.shared.subagent_builder import (
|
||||
pack_subagent,
|
||||
)
|
||||
|
||||
from .tools.index import NAME, RULESET, load_tools
|
||||
|
||||
|
||||
def build_subagent(
|
||||
*,
|
||||
dependencies: dict[str, Any],
|
||||
model: BaseChatModel | None = None,
|
||||
middleware_stack: dict[str, Any] | None = None,
|
||||
mcp_tools: list[BaseTool] | None = None,
|
||||
) -> SurfSenseSubagentSpec:
|
||||
tools = [*load_tools(dependencies=dependencies), *(mcp_tools or [])]
|
||||
description = (
|
||||
read_md_file(__package__, "description").strip()
|
||||
or "Scrapes public Walmart product data and reviews for a URL or search term."
|
||||
)
|
||||
system_prompt = read_md_file(__package__, "system_prompt").strip()
|
||||
return pack_subagent(
|
||||
name=NAME,
|
||||
description=description,
|
||||
system_prompt=system_prompt,
|
||||
tools=tools,
|
||||
ruleset=RULESET,
|
||||
dependencies=dependencies,
|
||||
model=model,
|
||||
middleware_stack=middleware_stack,
|
||||
)
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
Walmart product specialist: scrapes public Walmart listings and returns structured product data — title, item id (usItemId), brand, price and list price, star rating and review count, availability, images, features, seller, and product variants — plus deep paginated customer reviews (rating, text, author, verified-purchase flag, images, and seller responses). Works from a search term (e.g. "air fryer") or from Walmart product (/ip/...), search, category, or browse URLs, and can pull many reviews per product by item id or URL. Only public, anonymous US Walmart data — no login or seller account.
|
||||
Use it for product research, price tracking, catalog enrichment by item id, and review mining. Triggers include "find X on Walmart", "Walmart price of X", "reviews for this Walmart product", "compare these Walmart products", and "look up this Walmart URL/item id". Not for general web search (use the Google Search specialist), reading an arbitrary non-Walmart page (use the web crawling specialist), or other marketplaces.
|
||||
|
|
@ -0,0 +1,68 @@
|
|||
You are the SurfSense Walmart sub-agent.
|
||||
You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis.
|
||||
|
||||
<goal>
|
||||
Answer the delegated question from public Walmart product data and reviews gathered with your verbs, comparing against earlier results already in this conversation when the task calls for it.
|
||||
</goal>
|
||||
|
||||
<available_tools>
|
||||
- `walmart_scrape` — product details and search/category listings
|
||||
- `walmart_reviews` — deep paginated reviews for a product
|
||||
- `read_run` / `search_run` (free readers for stored scrape output)
|
||||
</available_tools>
|
||||
|
||||
<playbook>
|
||||
- Discovering products: call `walmart_scrape` with `search_terms` (e.g. ["air fryer"]).
|
||||
- Specific products: pass Walmart product URLs (/ip/...) or search/category/browse URLs in `urls`.
|
||||
- Faster listings: set `include_details=false` to return card-only results without opening each product page.
|
||||
- Sampled reviews: `walmart_scrape` returns a small on-page review sample by default (`include_reviews_sample=true`); disable it when reviews are irrelevant.
|
||||
- Deep review mining: use `walmart_reviews` with product `urls` or numeric `item_ids` (usItemId); raise `max_reviews` and set `sort_by` (most-recent, most-helpful, rating-high, rating-low) as the task needs. Reviews are billed per review, so keep `max_reviews` to what the task actually requires.
|
||||
- Batch multiple URLs or search terms into one call rather than many single-source calls.
|
||||
<include snippet="run_reader"/>
|
||||
- Comparison requests: pull the current products, compare against prior values already in this conversation's earlier tool results, and report concrete deltas (price up/down, rating change, stock changes).
|
||||
</playbook>
|
||||
|
||||
<tool_policy>
|
||||
- Use only tools in `<available_tools>`.
|
||||
- Report only results present in the tool output. Never invent titles, item ids, prices, ratings, or reviews.
|
||||
- `walmart_scrape`: provide at least one of `urls` or `search_terms`.
|
||||
- `walmart_reviews`: provide at least one of `urls` or `item_ids`.
|
||||
</tool_policy>
|
||||
|
||||
<out_of_scope>
|
||||
- Do not perform general web search — that is the Google Search specialist's job.
|
||||
- Do not read or extract an arbitrary non-Walmart page — return the URL for the web crawling specialist.
|
||||
- Do not generate deliverables or perform connector mutations; return findings for the supervisor to act on.
|
||||
- Only public, anonymous Walmart data — never anything behind a login or seller account.
|
||||
</out_of_scope>
|
||||
|
||||
<safety>
|
||||
- Report uncertainty explicitly when evidence is incomplete or conflicting.
|
||||
- Never present unverified claims as facts.
|
||||
</safety>
|
||||
|
||||
<failure_policy>
|
||||
- Underspecified request — no usable search term, URL, or item id — return `status=blocked` with the missing fields.
|
||||
- Tool failure: return `status=error` with a concise recovery `next_step`.
|
||||
- No useful evidence: return `status=blocked` with a narrower query or the scope you still need.
|
||||
</failure_policy>
|
||||
|
||||
<output_contract>
|
||||
Return **only** one JSON object (no markdown/prose):
|
||||
{
|
||||
"status": "success" | "partial" | "blocked" | "error",
|
||||
"action_summary": string,
|
||||
"evidence": {
|
||||
"findings": string[],
|
||||
"sources": string[],
|
||||
"confidence": "high" | "medium" | "low"
|
||||
},
|
||||
"next_step": string | null,
|
||||
"missing_fields": string[] | null,
|
||||
"assumptions": string[] | null
|
||||
}
|
||||
<include snippet="output_contract_base"/>
|
||||
Route-specific rules:
|
||||
- `evidence.findings`: max 10 entries, each a single sentence stating one distinct product, review theme, or delta. Do not paste raw payloads.
|
||||
- `evidence.sources`: max 10 URLs, one per finding when applicable. List each URL once.
|
||||
</output_contract>
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
"""``walmart`` sub-agent tools: the Walmart product scrape and reviews verbs."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.tools import BaseTool
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.permissions import Ruleset
|
||||
from app.capabilities.core.access.agent import build_capability_tools
|
||||
from app.capabilities.walmart.reviews.definition import WALMART_REVIEWS
|
||||
from app.capabilities.walmart.scrape.definition import WALMART_SCRAPE
|
||||
|
||||
NAME = "walmart"
|
||||
|
||||
RULESET = Ruleset(origin=NAME, rules=[])
|
||||
|
||||
_CI_VERBS = [WALMART_SCRAPE, WALMART_REVIEWS]
|
||||
|
||||
|
||||
def load_tools(
|
||||
*, dependencies: dict[str, Any] | None = None, **kwargs: Any
|
||||
) -> list[BaseTool]:
|
||||
d = {**(dependencies or {}), **kwargs}
|
||||
return build_capability_tools(
|
||||
workspace_id=d.get("workspace_id"),
|
||||
capabilities=_CI_VERBS,
|
||||
)
|
||||
|
|
@ -15,6 +15,9 @@ from __future__ import annotations
|
|||
from typing import Any
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.feature_flags import AgentFeatureFlags
|
||||
from app.agents.chat.multi_agent_chat.shared.middleware.citation_state import (
|
||||
build_citation_state_mw,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.shared.middleware.resilience import (
|
||||
ResilienceMiddlewares,
|
||||
)
|
||||
|
|
@ -45,6 +48,8 @@ def build_subagent_middleware_stack(
|
|||
return {
|
||||
"todos": build_todos_mw(),
|
||||
"permission": permission,
|
||||
# Declares the citation_registry channel so run [n]s merge up from tools.
|
||||
"citation": build_citation_state_mw(),
|
||||
"retry": resilience.retry,
|
||||
"fallback": resilience.fallback,
|
||||
"model_call_limit": resilience.model_call_limit,
|
||||
|
|
|
|||
|
|
@ -24,6 +24,9 @@ from app.agents.chat.multi_agent_chat.subagents.builtins.google_maps.agent impor
|
|||
from app.agents.chat.multi_agent_chat.subagents.builtins.google_search.agent import (
|
||||
build_subagent as build_google_search_subagent,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.subagents.builtins.indeed.agent import (
|
||||
build_subagent as build_indeed_subagent,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.subagents.builtins.instagram.agent import (
|
||||
build_subagent as build_instagram_subagent,
|
||||
)
|
||||
|
|
@ -42,6 +45,9 @@ from app.agents.chat.multi_agent_chat.subagents.builtins.reddit.agent import (
|
|||
from app.agents.chat.multi_agent_chat.subagents.builtins.tiktok.agent import (
|
||||
build_subagent as build_tiktok_subagent,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.subagents.builtins.walmart.agent import (
|
||||
build_subagent as build_walmart_subagent,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.subagents.builtins.web_crawler.agent import (
|
||||
build_subagent as build_web_crawler_subagent,
|
||||
)
|
||||
|
|
@ -89,6 +95,7 @@ SUBAGENT_BUILDERS_BY_NAME: dict[str, SubagentBuilder] = {
|
|||
"google_drive": build_google_drive_subagent,
|
||||
"google_maps": build_google_maps_subagent,
|
||||
"google_search": build_google_search_subagent,
|
||||
"indeed": build_indeed_subagent,
|
||||
"instagram": build_instagram_subagent,
|
||||
"knowledge_base": build_knowledge_base_subagent,
|
||||
"mcp_discovery": build_mcp_discovery_subagent,
|
||||
|
|
@ -96,6 +103,7 @@ SUBAGENT_BUILDERS_BY_NAME: dict[str, SubagentBuilder] = {
|
|||
"onedrive": build_onedrive_subagent,
|
||||
"reddit": build_reddit_subagent,
|
||||
"tiktok": build_tiktok_subagent,
|
||||
"walmart": build_walmart_subagent,
|
||||
"web_crawler": build_web_crawler_subagent,
|
||||
"youtube": build_youtube_subagent,
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,3 +5,4 @@ Rules (universal):
|
|||
- `status=blocked` due to missing required inputs -> `missing_fields` must be non-null.
|
||||
- `assumptions`: any inferences you made about the user's intent; `null` when no inferences were needed.
|
||||
- The `evidence` object's fields are documented in your route-specific `<output_contract>` above; never invent fields the tool did not return.
|
||||
- When a finding is drawn from a scraper run, append that run's `[n]` (the tool result states `Cite this scraper run as [n]`) to the finding text so the citation survives into the final answer. Copy the label exactly; never invent one.
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@ from app.config import config as app_config
|
|||
from app.services.kokoro_tts_service import get_kokoro_tts_service
|
||||
from app.services.llm_service import get_agent_llm
|
||||
from app.utils.content_utils import extract_text_content, strip_markdown_fences
|
||||
from app.utils.file_io import write_bytes
|
||||
|
||||
from .configuration import Configuration
|
||||
from .prompts import (
|
||||
|
|
@ -137,8 +138,7 @@ async def create_slide_audio(state: State, config: RunnableConfig) -> dict[str,
|
|||
kwargs["api_base"] = app_config.TTS_SERVICE_API_BASE
|
||||
|
||||
response = await aspeech(**kwargs)
|
||||
with open(chunk_path, "wb") as f:
|
||||
f.write(response.content)
|
||||
await write_bytes(chunk_path, response.content)
|
||||
|
||||
return chunk_path
|
||||
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ from collections import defaultdict
|
|||
from contextlib import asynccontextmanager
|
||||
from datetime import UTC, datetime
|
||||
from threading import Lock
|
||||
from typing import Any
|
||||
|
||||
import redis
|
||||
from fastapi import Depends, FastAPI, HTTPException, Request, Response, status
|
||||
|
|
@ -50,7 +51,7 @@ from app.gateway.inbox_worker import (
|
|||
start_gateway_inbox_worker,
|
||||
stop_gateway_inbox_worker,
|
||||
)
|
||||
from app.observability import metrics as ot_metrics
|
||||
from app.observability import analytics as ph_analytics, metrics as ot_metrics
|
||||
from app.observability.bootstrap import init_otel, shutdown_otel
|
||||
from app.rate_limiter import get_real_client_ip, limiter
|
||||
from app.routes import router as crud_router
|
||||
|
|
@ -94,17 +95,21 @@ def _build_error_response(
|
|||
code: str = "INTERNAL_ERROR",
|
||||
request_id: str = "",
|
||||
extra_headers: dict[str, str] | None = None,
|
||||
fields: list[dict[str, Any]] | None = None,
|
||||
) -> JSONResponse:
|
||||
"""Build the standardized error envelope (new ``error`` + legacy ``detail``)."""
|
||||
error: dict[str, Any] = {
|
||||
"code": code,
|
||||
"message": message,
|
||||
"status": status_code,
|
||||
"request_id": request_id,
|
||||
"timestamp": datetime.now(UTC).isoformat(),
|
||||
"report_url": ISSUES_URL,
|
||||
}
|
||||
if fields:
|
||||
error["fields"] = fields
|
||||
body = {
|
||||
"error": {
|
||||
"code": code,
|
||||
"message": message,
|
||||
"status": status_code,
|
||||
"request_id": request_id,
|
||||
"timestamp": datetime.now(UTC).isoformat(),
|
||||
"report_url": ISSUES_URL,
|
||||
},
|
||||
"error": error,
|
||||
"detail": message,
|
||||
}
|
||||
headers = {"X-Request-ID": request_id}
|
||||
|
|
@ -222,16 +227,35 @@ def _http_exception_handler(request: Request, exc: HTTPException) -> JSONRespons
|
|||
def _validation_error_handler(
|
||||
request: Request, exc: RequestValidationError
|
||||
) -> JSONResponse:
|
||||
"""Return 422 with field-level detail in the standard envelope."""
|
||||
"""Return 422 with field-level detail in the standard envelope.
|
||||
|
||||
``error.fields`` carries each failure's location path and message so clients
|
||||
can attach errors to the offending input; ``message`` is the flat summary.
|
||||
"""
|
||||
rid = _get_request_id(request)
|
||||
fields = []
|
||||
for err in exc.errors():
|
||||
loc = " -> ".join(str(part) for part in err.get("loc", []))
|
||||
fields.append(f"{loc}: {err.get('msg', 'invalid')}")
|
||||
message = (
|
||||
f"Validation failed: {'; '.join(fields)}" if fields else "Validation failed."
|
||||
fields = [
|
||||
{
|
||||
"loc": [str(part) for part in err.get("loc", ())],
|
||||
# Drop pydantic's "Value error, " prefix so messages read for humans.
|
||||
"msg": str(err.get("msg", "invalid")).removeprefix("Value error, "),
|
||||
}
|
||||
for err in exc.errors()
|
||||
]
|
||||
|
||||
def _segment(field: dict[str, Any]) -> str:
|
||||
# Drop the "body" request root so model-level errors read as a plain
|
||||
# sentence and field errors read as "field -> sub", not "body -> field".
|
||||
path = field["loc"]
|
||||
if path and path[0] == "body":
|
||||
path = path[1:]
|
||||
loc = " -> ".join(path)
|
||||
return f"{loc}: {field['msg']}" if loc else field["msg"]
|
||||
|
||||
summary = "; ".join(_segment(f) for f in fields)
|
||||
message = f"Validation failed: {summary}" if fields else "Validation failed."
|
||||
return _build_error_response(
|
||||
422, message, code="VALIDATION_ERROR", request_id=rid, fields=fields
|
||||
)
|
||||
return _build_error_response(422, message, code="VALIDATION_ERROR", request_id=rid)
|
||||
|
||||
|
||||
def _unhandled_exception_handler(request: Request, exc: Exception) -> JSONResponse:
|
||||
|
|
@ -690,6 +714,7 @@ async def lifespan(app: FastAPI):
|
|||
await stop_gateway_inbox_worker()
|
||||
_stop_openrouter_background_refresh()
|
||||
await close_checkpointer()
|
||||
ph_analytics.shutdown()
|
||||
shutdown_otel()
|
||||
|
||||
|
||||
|
|
@ -796,6 +821,52 @@ class RequestPerfMiddleware(BaseHTTPMiddleware):
|
|||
|
||||
app.add_middleware(RequestPerfMiddleware)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# PAT / MCP API attribution middleware
|
||||
# ---------------------------------------------------------------------------
|
||||
# Emits a PostHog ``pat_api_request`` event for any request authenticated by a
|
||||
# Personal Access Token, so "documents added via MCP", "searches via MCP" etc.
|
||||
# are queryable without instrumenting each route. Relies on ``get_auth_context``
|
||||
# stashing the resolved principal on ``request.state.auth_context``; requests
|
||||
# that never resolve a PAT principal are silently skipped. No-op when PostHog
|
||||
# is unconfigured.
|
||||
|
||||
|
||||
class PatApiAnalyticsMiddleware(BaseHTTPMiddleware):
|
||||
"""Capture PAT-authenticated API usage (incl. MCP) after each response."""
|
||||
|
||||
async def dispatch(
|
||||
self, request: StarletteRequest, call_next: RequestResponseEndpoint
|
||||
) -> StarletteResponse:
|
||||
response = await call_next(request)
|
||||
with contextlib.suppress(Exception):
|
||||
ctx = getattr(request.state, "auth_context", None)
|
||||
if ctx is not None and ctx.method == "pat" and ph_analytics.is_enabled():
|
||||
# Use the route *template* (e.g. /documents/{id}) to keep the
|
||||
# ``route`` property low-cardinality; fall back to the raw path.
|
||||
route = request.scope.get("route")
|
||||
route_path = getattr(route, "path", None) or request.url.path
|
||||
client = (
|
||||
"mcp"
|
||||
if request.headers.get("X-SurfSense-Client") == "mcp"
|
||||
else "pat_script"
|
||||
)
|
||||
ph_analytics.capture_for(
|
||||
ctx,
|
||||
"pat_api_request",
|
||||
{
|
||||
"route": route_path,
|
||||
"method": request.method,
|
||||
"status_code": response.status_code,
|
||||
"client": client,
|
||||
},
|
||||
)
|
||||
return response
|
||||
|
||||
|
||||
app.add_middleware(PatApiAnalyticsMiddleware)
|
||||
|
||||
# Add SlowAPI middleware for automatic rate limiting
|
||||
# Uses Starlette BaseHTTPMiddleware (not the raw ASGI variant) to avoid
|
||||
# corrupting StreamingResponse — SlowAPIASGIMiddleware re-sends
|
||||
|
|
|
|||
|
|
@ -15,11 +15,36 @@ from app.automations.schemas.definition.envelope import (
|
|||
)
|
||||
from app.automations.schemas.definition.plan_step import PlanStep
|
||||
from app.automations.templating import build_run_context
|
||||
from app.observability import analytics as ph_analytics
|
||||
|
||||
from . import repository
|
||||
from .step import execute_step
|
||||
|
||||
|
||||
def _capture_run_outcome(run: AutomationRun, status: str) -> None:
|
||||
"""Emit ``automation_run_completed`` — headless runs the frontend never sees.
|
||||
|
||||
No-op when PostHog is unconfigured or the owning user is unknown.
|
||||
"""
|
||||
automation = run.automation
|
||||
creator_id = getattr(automation, "created_by_user_id", None)
|
||||
if not creator_id:
|
||||
return
|
||||
workspace_id = getattr(automation, "workspace_id", None)
|
||||
ph_analytics.capture(
|
||||
"automation_run_completed",
|
||||
distinct_id=str(creator_id),
|
||||
properties={
|
||||
"automation_id": run.automation_id,
|
||||
"run_id": run.id,
|
||||
"workspace_id": workspace_id,
|
||||
"status": status,
|
||||
"trigger_type": run.trigger.type.value if run.trigger else None,
|
||||
},
|
||||
groups={"workspace": str(workspace_id)} if workspace_id is not None else None,
|
||||
)
|
||||
|
||||
|
||||
async def execute_run(session: AsyncSession, run_id: int) -> None:
|
||||
"""Load run ``run_id`` and execute its snapshot plan to a terminal state."""
|
||||
run = await repository.load_run(session, run_id)
|
||||
|
|
@ -41,6 +66,7 @@ async def execute_run(session: AsyncSession, run_id: int) -> None:
|
|||
},
|
||||
)
|
||||
await session.commit()
|
||||
_capture_run_outcome(run, "failed")
|
||||
return
|
||||
|
||||
await repository.mark_running(session, run)
|
||||
|
|
@ -66,6 +92,7 @@ async def execute_run(session: AsyncSession, run_id: int) -> None:
|
|||
await _run_on_failure(session, run, definition)
|
||||
await repository.mark_failed(session, run, result.get("error"))
|
||||
await session.commit()
|
||||
_capture_run_outcome(run, "failed")
|
||||
return
|
||||
|
||||
if result["status"] == "succeeded":
|
||||
|
|
@ -73,6 +100,7 @@ async def execute_run(session: AsyncSession, run_id: int) -> None:
|
|||
|
||||
await repository.mark_succeeded(session, run)
|
||||
await session.commit()
|
||||
_capture_run_outcome(run, "succeeded")
|
||||
|
||||
|
||||
async def _run_on_failure(
|
||||
|
|
|
|||
|
|
@ -29,6 +29,7 @@ from app.automations.services.model_policy import (
|
|||
from app.automations.triggers import get_trigger
|
||||
from app.automations.triggers.builtin.schedule import compute_next_fire_at
|
||||
from app.db import Permission, Workspace, get_async_session
|
||||
from app.observability import analytics as ph_analytics
|
||||
from app.users import get_auth_context
|
||||
from app.utils.rbac import check_permission
|
||||
|
||||
|
|
@ -75,6 +76,18 @@ class AutomationService:
|
|||
|
||||
self.session.add(automation)
|
||||
await self.session.commit()
|
||||
|
||||
# Authoritative creation (migrated from automations-mutation.atoms.ts).
|
||||
ph_analytics.capture_for(
|
||||
self.auth,
|
||||
"automation_created",
|
||||
{
|
||||
"automation_id": automation.id,
|
||||
"workspace_id": automation.workspace_id,
|
||||
"trigger_count": len(payload.triggers),
|
||||
},
|
||||
groups={"workspace": str(automation.workspace_id)},
|
||||
)
|
||||
return await self._get_with_triggers_or_raise(automation.id)
|
||||
|
||||
async def list(
|
||||
|
|
@ -150,6 +163,31 @@ class AutomationService:
|
|||
automation.version += 1
|
||||
|
||||
await self.session.commit()
|
||||
|
||||
# Migrated from automations-mutation.atoms.ts: a status-only change is a
|
||||
# distinct event; other field edits are ``automation_updated``.
|
||||
if "status" in data:
|
||||
ph_analytics.capture_for(
|
||||
self.auth,
|
||||
"automation_status_changed",
|
||||
{
|
||||
"automation_id": automation.id,
|
||||
"workspace_id": automation.workspace_id,
|
||||
"next_status": str(data["status"]),
|
||||
},
|
||||
groups={"workspace": str(automation.workspace_id)},
|
||||
)
|
||||
if any(k in data for k in ("name", "description", "definition")):
|
||||
ph_analytics.capture_for(
|
||||
self.auth,
|
||||
"automation_updated",
|
||||
{
|
||||
"automation_id": automation.id,
|
||||
"workspace_id": automation.workspace_id,
|
||||
"has_definition_change": "definition" in data,
|
||||
},
|
||||
groups={"workspace": str(automation.workspace_id)},
|
||||
)
|
||||
return await self._get_with_triggers_or_raise(automation_id)
|
||||
|
||||
async def delete(self, automation_id: int) -> None:
|
||||
|
|
@ -158,9 +196,18 @@ class AutomationService:
|
|||
await self._authorize(
|
||||
automation.workspace_id, Permission.AUTOMATIONS_DELETE.value
|
||||
)
|
||||
workspace_id = automation.workspace_id
|
||||
await self.session.delete(automation)
|
||||
await self.session.commit()
|
||||
|
||||
# Authoritative deletion (migrated from automations-mutation.atoms.ts).
|
||||
ph_analytics.capture_for(
|
||||
self.auth,
|
||||
"automation_deleted",
|
||||
{"automation_id": automation_id, "workspace_id": workspace_id},
|
||||
groups={"workspace": str(workspace_id)},
|
||||
)
|
||||
|
||||
async def _get_or_raise(self, automation_id: int) -> Automation:
|
||||
automation = await self.session.get(Automation, automation_id)
|
||||
if automation is None:
|
||||
|
|
|
|||
|
|
@ -16,6 +16,7 @@ from app.automations.schemas.api import TriggerCreate, TriggerUpdate
|
|||
from app.automations.triggers import get_trigger
|
||||
from app.automations.triggers.builtin.schedule import compute_next_fire_at
|
||||
from app.db import Permission, get_async_session
|
||||
from app.observability import analytics as ph_analytics
|
||||
from app.users import get_auth_context
|
||||
from app.utils.rbac import check_permission
|
||||
|
||||
|
|
@ -48,6 +49,18 @@ class TriggerService:
|
|||
self.session.add(trigger)
|
||||
await self.session.commit()
|
||||
await self.session.refresh(trigger)
|
||||
|
||||
# Migrated from automations-mutation.atoms.ts.
|
||||
ph_analytics.capture_for(
|
||||
self.auth,
|
||||
"automation_trigger_added",
|
||||
{
|
||||
"automation_id": automation_id,
|
||||
"trigger_id": trigger.id,
|
||||
"trigger_type": getattr(trigger.type, "value", str(trigger.type)),
|
||||
"enabled": trigger.enabled,
|
||||
},
|
||||
)
|
||||
return trigger
|
||||
|
||||
async def update(
|
||||
|
|
@ -82,6 +95,26 @@ class TriggerService:
|
|||
|
||||
await self.session.commit()
|
||||
await self.session.refresh(trigger)
|
||||
|
||||
# Migrated from automations-mutation.atoms.ts. ``change`` mirrors the
|
||||
# frontend's coarse categorisation.
|
||||
_change = (
|
||||
"enabled"
|
||||
if "enabled" in data and "params" not in data
|
||||
else "params"
|
||||
if "params" in data
|
||||
else "other"
|
||||
)
|
||||
ph_analytics.capture_for(
|
||||
self.auth,
|
||||
"automation_trigger_updated",
|
||||
{
|
||||
"automation_id": automation_id,
|
||||
"trigger_id": trigger_id,
|
||||
"change": _change,
|
||||
"enabled": trigger.enabled,
|
||||
},
|
||||
)
|
||||
return trigger
|
||||
|
||||
async def remove(self, *, automation_id: int, trigger_id: int) -> None:
|
||||
|
|
@ -92,6 +125,13 @@ class TriggerService:
|
|||
await self.session.delete(trigger)
|
||||
await self.session.commit()
|
||||
|
||||
# Migrated from automations-mutation.atoms.ts.
|
||||
ph_analytics.capture_for(
|
||||
self.auth,
|
||||
"automation_trigger_removed",
|
||||
{"automation_id": automation_id, "trigger_id": trigger_id},
|
||||
)
|
||||
|
||||
async def _authorize_automation(
|
||||
self, automation_id: int, permission: str
|
||||
) -> Automation:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
"""Scraper capability registry — typed, stateless verbs. See plans/backend/04-capabilities.md."""
|
||||
"""Scraper capability registry — typed, stateless verbs."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ from __future__ import annotations
|
|||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
from app.capabilities.core.validation import HttpUrlStr
|
||||
from app.proprietary.platforms.amazon import ProductItem
|
||||
|
||||
MAX_AMAZON_SOURCES = 20
|
||||
|
|
@ -13,7 +14,7 @@ MAX_AMAZON_RESULTS = 1000
|
|||
class ScrapeInput(BaseModel):
|
||||
"""Agent-facing controls for public product discovery and enrichment."""
|
||||
|
||||
urls: list[str] = Field(default_factory=list, max_length=MAX_AMAZON_SOURCES)
|
||||
urls: list[HttpUrlStr] = Field(default_factory=list, max_length=MAX_AMAZON_SOURCES)
|
||||
search_terms: list[str] = Field(default_factory=list, max_length=MAX_AMAZON_SOURCES)
|
||||
max_items: int = Field(default=10, ge=1, le=100)
|
||||
domain: str = Field(
|
||||
|
|
|
|||
|
|
@ -11,9 +11,13 @@ subagent can follow a truncation reference without extra wiring.
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
|
||||
from langchain.tools import ToolRuntime
|
||||
from langchain_core.messages import ToolMessage
|
||||
from langchain_core.tools import BaseTool, StructuredTool
|
||||
from langgraph.types import Command
|
||||
|
||||
from app.capabilities.core.billing import charge_capability, gate_capability
|
||||
from app.capabilities.core.progress import progress_scope
|
||||
|
|
@ -64,7 +68,7 @@ def _capability_tool(capability: Capability, workspace_id: int) -> BaseTool:
|
|||
executor = capability.executor
|
||||
name = capability.name
|
||||
|
||||
async def _run(**kwargs: object) -> dict | str:
|
||||
async def _run(runtime: ToolRuntime, **kwargs: object) -> dict | str | Command:
|
||||
payload = input_model(**kwargs)
|
||||
input_dump = payload.model_dump(exclude_none=True)
|
||||
thread_id = _current_thread_id()
|
||||
|
|
@ -119,13 +123,42 @@ def _capability_tool(capability: Capability, workspace_id: int) -> BaseTool:
|
|||
progress=reporter.coarse,
|
||||
)
|
||||
|
||||
# No stored run to cite: keep the legacy return shape, no citation.
|
||||
if run_id is None:
|
||||
if serialized.char_count <= RUN_OUTPUT_CHAR_CAP:
|
||||
return output.model_dump(exclude_none=True)
|
||||
return _build_preview(serialized, run_id)
|
||||
|
||||
run_external_id = f"run_{run_id}"
|
||||
if serialized.char_count <= RUN_OUTPUT_CHAR_CAP:
|
||||
dump = output.model_dump(exclude_none=True)
|
||||
if run_id is not None:
|
||||
dump["run_id"] = f"run_{run_id}"
|
||||
return dump
|
||||
dump["run_id"] = run_external_id
|
||||
content = json.dumps(dump, ensure_ascii=False, default=str)
|
||||
else:
|
||||
content = _build_preview(serialized, run_id)
|
||||
|
||||
return _build_preview(serialized, run_id)
|
||||
# Deferred import: the citation spine imports from here; lazy avoids a cycle.
|
||||
from app.agents.chat.multi_agent_chat.shared.citations import load_registry
|
||||
from app.capabilities.core.access.run_citation import attach_run_citation
|
||||
|
||||
registry = load_registry(getattr(runtime, "state", None))
|
||||
_, label = attach_run_citation(
|
||||
registry, run_external_id=run_external_id, capability=name
|
||||
)
|
||||
return Command(
|
||||
update={
|
||||
"messages": [
|
||||
ToolMessage(
|
||||
content=content + label,
|
||||
tool_call_id=runtime.tool_call_id,
|
||||
)
|
||||
],
|
||||
"citation_registry": registry,
|
||||
}
|
||||
)
|
||||
|
||||
# Un-stringify for StructuredTool's signature-based runtime injection.
|
||||
_run.__annotations__["runtime"] = ToolRuntime
|
||||
|
||||
return StructuredTool.from_function(
|
||||
coroutine=_run,
|
||||
|
|
|
|||
|
|
@ -45,6 +45,7 @@ from app.capabilities.core.store import all_capabilities
|
|||
from app.capabilities.core.types import Capability, CapabilityContext
|
||||
from app.db import Run, async_session_maker, get_async_session
|
||||
from app.exceptions import ExternalServiceError, SurfSenseError
|
||||
from app.observability import analytics as ph_analytics
|
||||
from app.services.web_crawl_credit_service import InsufficientCreditsError
|
||||
from app.users import get_auth_context
|
||||
from app.utils.rbac import check_workspace_access
|
||||
|
|
@ -107,6 +108,41 @@ def _origin_for(auth: AuthContext) -> str:
|
|||
return "ui" if getattr(auth, "method", None) == "session" else "api"
|
||||
|
||||
|
||||
def _capture_scraper_run(
|
||||
*,
|
||||
capability: str,
|
||||
status: str,
|
||||
user_id,
|
||||
origin: str,
|
||||
duration_ms: int | None = None,
|
||||
item_count: int | None = None,
|
||||
cost_micros: int | None = None,
|
||||
) -> None:
|
||||
"""Emit ``scraper_run_completed`` — scrapers are the highest-value MCP surface.
|
||||
|
||||
Covers both sync and async runs; the async background task never flows
|
||||
through the PAT middleware, so this is the only place its outcome is
|
||||
captured. No-op when PostHog is unconfigured.
|
||||
"""
|
||||
if not ph_analytics.is_enabled() or not user_id:
|
||||
return
|
||||
platform, _, verb = capability.partition(".")
|
||||
ph_analytics.capture(
|
||||
"scraper_run_completed",
|
||||
distinct_id=str(user_id),
|
||||
properties={
|
||||
"platform": platform,
|
||||
"verb": verb,
|
||||
"capability": capability,
|
||||
"status": status,
|
||||
"origin": origin,
|
||||
"duration_ms": duration_ms,
|
||||
"item_count": item_count,
|
||||
"cost_micros": cost_micros,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _now_ms() -> int:
|
||||
return int(time.time() * 1000)
|
||||
|
||||
|
|
@ -185,6 +221,8 @@ async def _execute_async_run(
|
|||
unit,
|
||||
executor,
|
||||
payload,
|
||||
user_id=None,
|
||||
origin: str = "api",
|
||||
) -> None:
|
||||
"""Run a scrape in the background: stream progress, charge, finalize the row.
|
||||
|
||||
|
|
@ -218,6 +256,13 @@ async def _execute_async_run(
|
|||
progress=reporter.coarse,
|
||||
)
|
||||
_publish_finished(run_id, "error", error=str(exc))
|
||||
_capture_scraper_run(
|
||||
capability=capability,
|
||||
status="error",
|
||||
user_id=user_id,
|
||||
origin=origin,
|
||||
duration_ms=int((time.perf_counter() - started) * 1000),
|
||||
)
|
||||
return
|
||||
except Exception:
|
||||
logger.exception("async run %s failed with an upstream error", run_id)
|
||||
|
|
@ -229,6 +274,13 @@ async def _execute_async_run(
|
|||
progress=reporter.coarse,
|
||||
)
|
||||
_publish_finished(run_id, "error", error="upstream error")
|
||||
_capture_scraper_run(
|
||||
capability=capability,
|
||||
status="error",
|
||||
user_id=user_id,
|
||||
origin=origin,
|
||||
duration_ms=int((time.perf_counter() - started) * 1000),
|
||||
)
|
||||
return
|
||||
|
||||
duration_ms = int((time.perf_counter() - started) * 1000)
|
||||
|
|
@ -252,6 +304,15 @@ async def _execute_async_run(
|
|||
progress=reporter.coarse,
|
||||
)
|
||||
_publish_finished(run_id, "success", item_count=serialized.item_count)
|
||||
_capture_scraper_run(
|
||||
capability=capability,
|
||||
status="success",
|
||||
user_id=user_id,
|
||||
origin=origin,
|
||||
duration_ms=duration_ms,
|
||||
item_count=serialized.item_count,
|
||||
cost_micros=cost_micros,
|
||||
)
|
||||
|
||||
|
||||
async def _finalize_async(
|
||||
|
|
@ -358,6 +419,8 @@ def _register_verb(router: APIRouter, capability: Capability) -> None:
|
|||
unit=unit,
|
||||
executor=executor,
|
||||
payload=payload,
|
||||
user_id=user_id,
|
||||
origin=origin,
|
||||
)
|
||||
)
|
||||
run_event_bus.register_task(run_id, task)
|
||||
|
|
@ -373,6 +436,7 @@ def _register_verb(router: APIRouter, capability: Capability) -> None:
|
|||
try:
|
||||
output = await executor(payload)
|
||||
except (SurfSenseError, HTTPException) as exc:
|
||||
_sync_err_duration = int((time.perf_counter() - started) * 1000)
|
||||
await _record_rest_run(
|
||||
workspace_id=workspace_id,
|
||||
capability=name,
|
||||
|
|
@ -381,11 +445,19 @@ def _register_verb(router: APIRouter, capability: Capability) -> None:
|
|||
input=input_dump,
|
||||
user_id=user_id,
|
||||
error=str(exc),
|
||||
duration_ms=int((time.perf_counter() - started) * 1000),
|
||||
duration_ms=_sync_err_duration,
|
||||
progress=reporter.coarse,
|
||||
)
|
||||
_capture_scraper_run(
|
||||
capability=name,
|
||||
status="error",
|
||||
user_id=user_id,
|
||||
origin=origin,
|
||||
duration_ms=_sync_err_duration,
|
||||
)
|
||||
raise
|
||||
except Exception as exc:
|
||||
_sync_err_duration = int((time.perf_counter() - started) * 1000)
|
||||
await _record_rest_run(
|
||||
workspace_id=workspace_id,
|
||||
capability=name,
|
||||
|
|
@ -394,9 +466,16 @@ def _register_verb(router: APIRouter, capability: Capability) -> None:
|
|||
input=input_dump,
|
||||
user_id=user_id,
|
||||
error=str(exc),
|
||||
duration_ms=int((time.perf_counter() - started) * 1000),
|
||||
duration_ms=_sync_err_duration,
|
||||
progress=reporter.coarse,
|
||||
)
|
||||
_capture_scraper_run(
|
||||
capability=name,
|
||||
status="error",
|
||||
user_id=user_id,
|
||||
origin=origin,
|
||||
duration_ms=_sync_err_duration,
|
||||
)
|
||||
raise ExternalServiceError(
|
||||
f"The '{name}' capability failed due to an upstream error.",
|
||||
code="CAPABILITY_UPSTREAM_ERROR",
|
||||
|
|
@ -418,6 +497,15 @@ def _register_verb(router: APIRouter, capability: Capability) -> None:
|
|||
cost_micros=cost_micros,
|
||||
progress=reporter.coarse,
|
||||
)
|
||||
_capture_scraper_run(
|
||||
capability=name,
|
||||
status="success",
|
||||
user_id=user_id,
|
||||
origin=origin,
|
||||
duration_ms=duration_ms,
|
||||
item_count=serialized.item_count,
|
||||
cost_micros=cost_micros,
|
||||
)
|
||||
if run_id is not None:
|
||||
response.headers["X-Run-Id"] = f"run_{run_id}"
|
||||
return output
|
||||
|
|
|
|||
|
|
@ -0,0 +1,23 @@
|
|||
"""Register a recorded scraper run as a citable ``[n]``."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.citations import (
|
||||
CitationRegistry,
|
||||
CitationSourceType,
|
||||
)
|
||||
|
||||
|
||||
def attach_run_citation(
|
||||
registry: CitationRegistry,
|
||||
*,
|
||||
run_external_id: str,
|
||||
capability: str,
|
||||
) -> tuple[int, str]:
|
||||
"""Register the ``run_<uuid>`` handle; return its ``[n]`` and the label line."""
|
||||
n = registry.register(
|
||||
CitationSourceType.RUN,
|
||||
{"run_id": run_external_id},
|
||||
{"capability": capability},
|
||||
)
|
||||
return n, f"\n\nCite this scraper run as [{n}] after any claim drawn from its data."
|
||||
|
|
@ -41,6 +41,9 @@ _PLATFORM_RATE_KEYS: dict[BillingUnit, str] = {
|
|||
BillingUnit.TIKTOK_VIDEO: "TIKTOK_MICROS_PER_VIDEO",
|
||||
BillingUnit.TIKTOK_USER: "TIKTOK_MICROS_PER_USER",
|
||||
BillingUnit.TIKTOK_COMMENT: "TIKTOK_MICROS_PER_COMMENT",
|
||||
BillingUnit.INDEED_JOB: "INDEED_SCRAPE_MICROS_PER_JOB",
|
||||
BillingUnit.WALMART_PRODUCT: "WALMART_MICROS_PER_PRODUCT",
|
||||
BillingUnit.WALMART_REVIEW: "WALMART_MICROS_PER_REVIEW",
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -63,6 +66,9 @@ _UNIT_NOUNS: dict[BillingUnit, str] = {
|
|||
BillingUnit.TIKTOK_VIDEO: "video",
|
||||
BillingUnit.TIKTOK_USER: "profile",
|
||||
BillingUnit.TIKTOK_COMMENT: "comment",
|
||||
BillingUnit.INDEED_JOB: "job",
|
||||
BillingUnit.WALMART_PRODUCT: "product",
|
||||
BillingUnit.WALMART_REVIEW: "review",
|
||||
}
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -31,6 +31,9 @@ class BillingUnit(StrEnum):
|
|||
TIKTOK_VIDEO = "tiktok_video"
|
||||
TIKTOK_USER = "tiktok_user"
|
||||
TIKTOK_COMMENT = "tiktok_comment"
|
||||
INDEED_JOB = "indeed_job"
|
||||
WALMART_PRODUCT = "walmart_product"
|
||||
WALMART_REVIEW = "walmart_review"
|
||||
|
||||
|
||||
class BillableInput(Protocol):
|
||||
|
|
|
|||
24
surfsense_backend/app/capabilities/core/validation.py
Normal file
24
surfsense_backend/app/capabilities/core/validation.py
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
"""Shared Pydantic field types for capability I/O schemas."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Annotated
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
import validators
|
||||
from pydantic import AfterValidator
|
||||
from pydantic_core import PydanticCustomError
|
||||
|
||||
_HTTP_SCHEMES = frozenset({"http", "https"})
|
||||
|
||||
|
||||
def _validate_http_url(value: str) -> str:
|
||||
"""Accept only well-formed http(s) URLs, returned trimmed and unchanged."""
|
||||
url = value.strip()
|
||||
if not validators.url(url) or urlsplit(url).scheme.lower() not in _HTTP_SCHEMES:
|
||||
raise PydanticCustomError("http_url", "must be a valid http(s) URL")
|
||||
return url
|
||||
|
||||
|
||||
HttpUrlStr = Annotated[str, AfterValidator(_validate_http_url)]
|
||||
"""A request URL validated as http(s) and kept as ``str`` (no normalization)."""
|
||||
|
|
@ -10,6 +10,7 @@ from typing import Literal
|
|||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
from app.capabilities.core.validation import HttpUrlStr
|
||||
from app.proprietary.platforms.google_maps import ReviewItem
|
||||
|
||||
MAX_MAPS_REVIEW_SOURCES = 20
|
||||
|
|
@ -17,7 +18,7 @@ MAX_MAPS_REVIEW_SOURCES = 20
|
|||
|
||||
|
||||
class ReviewsInput(BaseModel):
|
||||
urls: list[str] = Field(
|
||||
urls: list[HttpUrlStr] = Field(
|
||||
default_factory=list,
|
||||
max_length=MAX_MAPS_REVIEW_SOURCES,
|
||||
description=(
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ from __future__ import annotations
|
|||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
from app.capabilities.core.validation import HttpUrlStr
|
||||
from app.proprietary.platforms.google_maps import PlaceItem
|
||||
|
||||
MAX_MAPS_SOURCES = 20
|
||||
|
|
@ -26,7 +27,7 @@ class ScrapeInput(BaseModel):
|
|||
"(at least one is required). Pair with location to scope a search."
|
||||
),
|
||||
)
|
||||
urls: list[str] = Field(
|
||||
urls: list[HttpUrlStr] = Field(
|
||||
default_factory=list,
|
||||
max_length=MAX_MAPS_SOURCES,
|
||||
description=(
|
||||
|
|
|
|||
5
surfsense_backend/app/capabilities/indeed/__init__.py
Normal file
5
surfsense_backend/app/capabilities/indeed/__init__.py
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
"""``indeed.*`` namespace: platform-native Indeed data verbs."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.capabilities.indeed.scrape import definition as _scrape # noqa: F401
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
"""``indeed.scrape`` verb: Indeed search / company URLs → job postings."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
"""``indeed.scrape`` capability registration (billed per job; see config
|
||||
``INDEED_SCRAPE_MICROS_PER_JOB``)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.capabilities.core import BillingUnit, Capability, register_capability
|
||||
from app.capabilities.indeed.scrape.executor import build_scrape_executor
|
||||
from app.capabilities.indeed.scrape.schemas import ScrapeInput, ScrapeOutput
|
||||
|
||||
INDEED_SCRAPE = Capability(
|
||||
name="indeed.scrape",
|
||||
description=(
|
||||
"Scrape public Indeed job postings, including title, company, location, "
|
||||
"salary, and description. Use urls or search_queries."
|
||||
),
|
||||
input_schema=ScrapeInput,
|
||||
output_schema=ScrapeOutput,
|
||||
executor=build_scrape_executor(),
|
||||
billing_unit=BillingUnit.INDEED_JOB,
|
||||
docs_url="/docs/connectors/native/indeed",
|
||||
)
|
||||
|
||||
register_capability(INDEED_SCRAPE)
|
||||
56
surfsense_backend/app/capabilities/indeed/scrape/executor.py
Normal file
56
surfsense_backend/app/capabilities/indeed/scrape/executor.py
Normal file
|
|
@ -0,0 +1,56 @@
|
|||
"""``indeed.scrape`` executor: verb input → scraper → Indeed job items."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Awaitable, Callable
|
||||
|
||||
from app.capabilities.core import Executor
|
||||
from app.capabilities.core.progress import emit_progress
|
||||
from app.capabilities.indeed.scrape.schemas import ScrapeInput, ScrapeOutput
|
||||
from app.exceptions import ForbiddenError
|
||||
from app.proprietary.platforms.indeed_jobs import (
|
||||
IndeedAccessBlockedError,
|
||||
IndeedScrapeInput,
|
||||
scrape_indeed,
|
||||
)
|
||||
|
||||
ScrapeFn = Callable[..., Awaitable[list[dict]]]
|
||||
|
||||
|
||||
def build_scrape_executor(scrape_fn: ScrapeFn | None = None) -> Executor:
|
||||
"""Bind the executor to a scraper fn (defaults to the proprietary actor)."""
|
||||
scrape_fn = scrape_fn or scrape_indeed
|
||||
|
||||
async def execute(payload: ScrapeInput) -> ScrapeOutput:
|
||||
actor_input = IndeedScrapeInput(
|
||||
startUrls=[{"url": url} for url in payload.urls],
|
||||
queries=payload.search_queries,
|
||||
country=payload.country,
|
||||
location=payload.location,
|
||||
radius=payload.radius,
|
||||
jobType=payload.job_type,
|
||||
level=payload.level,
|
||||
remote=payload.remote,
|
||||
fromDays=payload.from_days,
|
||||
sort=payload.sort,
|
||||
scrapeJobDetails=payload.scrape_job_details,
|
||||
maxItems=payload.max_items,
|
||||
maxItemsPerQuery=payload.max_items_per_query,
|
||||
)
|
||||
emit_progress(
|
||||
"starting", "Resolving Indeed targets", total=payload.max_items, unit="job"
|
||||
)
|
||||
try:
|
||||
items = await scrape_fn(actor_input, limit=payload.max_items)
|
||||
except IndeedAccessBlockedError as exc:
|
||||
# Anonymous-only scraper; a hard block can't be retried with creds.
|
||||
raise ForbiddenError(
|
||||
f"Indeed refused anonymous access: {exc}",
|
||||
code="INDEED_ACCESS_BLOCKED",
|
||||
) from exc
|
||||
emit_progress(
|
||||
"done", f"Scraped {len(items)} job(s)", current=len(items), unit="job"
|
||||
)
|
||||
return ScrapeOutput(items=items)
|
||||
|
||||
return execute
|
||||
120
surfsense_backend/app/capabilities/indeed/scrape/schemas.py
Normal file
120
surfsense_backend/app/capabilities/indeed/scrape/schemas.py
Normal file
|
|
@ -0,0 +1,120 @@
|
|||
"""``indeed.scrape`` I/O contracts.
|
||||
|
||||
A lean, agent-friendly surface over ``IndeedScrapeInput``
|
||||
(``app/proprietary/platforms/indeed_jobs``). The executor maps this to the full
|
||||
scraper input; the scraper's ``IndeedItem`` is reused verbatim as the output
|
||||
element.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
from app.capabilities.core.validation import HttpUrlStr
|
||||
from app.proprietary.platforms.indeed_jobs import IndeedItem
|
||||
from app.proprietary.platforms.indeed_jobs.schemas import (
|
||||
IndeedJobType,
|
||||
IndeedLevel,
|
||||
IndeedRemote,
|
||||
IndeedSort,
|
||||
)
|
||||
|
||||
MAX_INDEED_SOURCES = 20
|
||||
"""Per-call cap on urls + search_queries: bounds a synchronous request's fan-out."""
|
||||
|
||||
MAX_INDEED_ITEMS = 100
|
||||
"""Hard ceiling on jobs returned per call, regardless of the per-query caps."""
|
||||
|
||||
|
||||
class ScrapeInput(BaseModel):
|
||||
urls: list[HttpUrlStr] = Field(
|
||||
default_factory=list,
|
||||
max_length=MAX_INDEED_SOURCES,
|
||||
description=(
|
||||
"Indeed URLs to scrape: a search page (/jobs?q=&l=) or a company "
|
||||
"jobs page (/cmp/<slug>/jobs). Provide these OR search_queries "
|
||||
"(at least one source is required)."
|
||||
),
|
||||
)
|
||||
search_queries: list[str] = Field(
|
||||
default_factory=list,
|
||||
max_length=MAX_INDEED_SOURCES,
|
||||
description=(
|
||||
"Job search terms; each returns up to max_items_per_query results, "
|
||||
"shaped by country/location/job_type/etc."
|
||||
),
|
||||
)
|
||||
country: str = Field(
|
||||
default="us",
|
||||
description="Country code selecting the Indeed domain, e.g. 'us', 'gb', 'de'.",
|
||||
)
|
||||
location: str | None = Field(
|
||||
default=None,
|
||||
description="Where to search, e.g. 'Remote', 'New York, NY'.",
|
||||
)
|
||||
radius: int | None = Field(
|
||||
default=None,
|
||||
description="Search radius in miles/km around location.",
|
||||
)
|
||||
job_type: IndeedJobType | None = Field(
|
||||
default=None,
|
||||
description="Employment type filter: fulltime, parttime, contract, etc.",
|
||||
)
|
||||
level: IndeedLevel | None = Field(
|
||||
default=None,
|
||||
description="Experience level filter: entry_level, mid_level, senior_level.",
|
||||
)
|
||||
remote: IndeedRemote | None = Field(
|
||||
default=None,
|
||||
description="Work model filter: remote or hybrid.",
|
||||
)
|
||||
from_days: int | None = Field(
|
||||
default=None,
|
||||
description="Only return jobs posted within the last N days.",
|
||||
)
|
||||
sort: IndeedSort = Field(
|
||||
default="relevance",
|
||||
description="Result ordering: relevance or date.",
|
||||
)
|
||||
scrape_job_details: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Fetch each job's detail page for the full description (slower: one "
|
||||
"extra page load per job)."
|
||||
),
|
||||
)
|
||||
max_items: int = Field(
|
||||
default=25,
|
||||
ge=1,
|
||||
le=MAX_INDEED_ITEMS,
|
||||
description="Max total jobs to return across all sources.",
|
||||
)
|
||||
max_items_per_query: int = Field(
|
||||
default=25,
|
||||
ge=0,
|
||||
description="Max jobs to pull per search/company target.",
|
||||
)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _require_a_source(self) -> ScrapeInput:
|
||||
if not self.urls and not self.search_queries:
|
||||
raise ValueError("Provide at least one of 'urls' or 'search_queries'.")
|
||||
return self
|
||||
|
||||
@property
|
||||
def estimated_units(self) -> int:
|
||||
"""Worst-case billable jobs for the pre-flight gate: ``max_items`` is a
|
||||
hard cross-source ceiling (le=100), so no call can exceed it."""
|
||||
return self.max_items
|
||||
|
||||
|
||||
class ScrapeOutput(BaseModel):
|
||||
items: list[IndeedItem] = Field(
|
||||
default_factory=list,
|
||||
description="One item per job posting, in emission order.",
|
||||
)
|
||||
|
||||
@property
|
||||
def billable_units(self) -> int:
|
||||
"""One returned job = one billable unit."""
|
||||
return len(self.items)
|
||||
|
|
@ -10,6 +10,7 @@ from __future__ import annotations
|
|||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
from app.capabilities.core.validation import HttpUrlStr
|
||||
from app.proprietary.platforms.reddit import RedditItem
|
||||
from app.proprietary.platforms.reddit.schemas import RedditSort, RedditTime
|
||||
|
||||
|
|
@ -21,7 +22,7 @@ MAX_REDDIT_ITEMS = 100
|
|||
|
||||
|
||||
class ScrapeInput(BaseModel):
|
||||
urls: list[str] = Field(
|
||||
urls: list[HttpUrlStr] = Field(
|
||||
default_factory=list,
|
||||
max_length=MAX_REDDIT_SOURCES,
|
||||
description=(
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ from __future__ import annotations
|
|||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.capabilities.core.validation import HttpUrlStr
|
||||
from app.capabilities.tiktok.scrape.schemas import (
|
||||
MAX_TIKTOK_ITEMS,
|
||||
MAX_TIKTOK_SOURCES,
|
||||
|
|
@ -18,7 +19,7 @@ from app.proprietary.platforms.tiktok import CommentItem
|
|||
|
||||
|
||||
class CommentsInput(BaseModel):
|
||||
video_urls: list[str] = Field(
|
||||
video_urls: list[HttpUrlStr] = Field(
|
||||
min_length=1,
|
||||
max_length=MAX_TIKTOK_SOURCES,
|
||||
description="TikTok video URLs (/@<user>/video/<id>) to pull comments from.",
|
||||
|
|
|
|||
|
|
@ -12,6 +12,7 @@ from __future__ import annotations
|
|||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
from app.capabilities.core.validation import HttpUrlStr
|
||||
from app.proprietary.platforms.tiktok import TikTokVideoItem
|
||||
|
||||
MAX_TIKTOK_SOURCES = 20
|
||||
|
|
@ -22,7 +23,7 @@ MAX_TIKTOK_ITEMS = 100
|
|||
|
||||
|
||||
class ScrapeInput(BaseModel):
|
||||
urls: list[str] = Field(
|
||||
urls: list[HttpUrlStr] = Field(
|
||||
default_factory=list,
|
||||
max_length=MAX_TIKTOK_SOURCES,
|
||||
description=(
|
||||
|
|
|
|||
6
surfsense_backend/app/capabilities/walmart/__init__.py
Normal file
6
surfsense_backend/app/capabilities/walmart/__init__.py
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
"""``walmart.*`` namespace: platform-native Walmart data verbs."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.capabilities.walmart.reviews import definition as _reviews # noqa: F401
|
||||
from app.capabilities.walmart.scrape import definition as _scrape # noqa: F401
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
"""Walmart deep review scraping capability."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
"""``walmart.reviews`` capability registration (billed per review; see config
|
||||
``WALMART_MICROS_PER_REVIEW``)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.capabilities.core import BillingUnit, Capability, register_capability
|
||||
from app.capabilities.walmart.reviews.executor import build_reviews_executor
|
||||
from app.capabilities.walmart.reviews.schemas import ReviewsInput, ReviewsOutput
|
||||
|
||||
WALMART_REVIEWS = Capability(
|
||||
name="walmart.reviews",
|
||||
description=(
|
||||
"Fetch deep paginated public Walmart product reviews with ratings, text, "
|
||||
"authors, verified-purchase flags, images, and seller responses. Use "
|
||||
"product urls or item ids."
|
||||
),
|
||||
input_schema=ReviewsInput,
|
||||
output_schema=ReviewsOutput,
|
||||
executor=build_reviews_executor(),
|
||||
billing_unit=BillingUnit.WALMART_REVIEW,
|
||||
docs_url="/docs/connectors/native/walmart",
|
||||
)
|
||||
|
||||
register_capability(WALMART_REVIEWS)
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
"""``walmart.reviews`` executor: verb input → scraper → review items."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Awaitable, Callable
|
||||
|
||||
from app.capabilities.core import Executor
|
||||
from app.capabilities.core.progress import emit_progress
|
||||
from app.capabilities.walmart.reviews.schemas import ReviewsInput, ReviewsOutput
|
||||
from app.proprietary.platforms.walmart import WalmartReviewsInput, scrape_reviews
|
||||
|
||||
ReviewsFn = Callable[..., Awaitable[list[dict]]]
|
||||
|
||||
|
||||
def build_reviews_executor(scrape_fn: ReviewsFn | None = None) -> Executor:
|
||||
"""Bind the capability input mapping to a replaceable reviews scraper function."""
|
||||
scrape_fn = scrape_fn or scrape_reviews
|
||||
|
||||
async def execute(payload: ReviewsInput) -> ReviewsOutput:
|
||||
input_model = WalmartReviewsInput(
|
||||
itemIds=payload.sources(),
|
||||
maxReviews=payload.max_reviews,
|
||||
sort=payload.sort_by,
|
||||
)
|
||||
emit_progress(
|
||||
"starting",
|
||||
"Fetching Walmart reviews",
|
||||
total=payload.estimated_units,
|
||||
unit="review",
|
||||
)
|
||||
items = await scrape_fn(input_model, limit=payload.estimated_units)
|
||||
emit_progress(
|
||||
"done",
|
||||
f"Scraped {sum('error' not in item for item in items)} review(s)",
|
||||
current=len(items),
|
||||
unit="review",
|
||||
)
|
||||
return ReviewsOutput(items=items)
|
||||
|
||||
return execute
|
||||
|
|
@ -0,0 +1,73 @@
|
|||
"""``walmart.reviews`` I/O contracts.
|
||||
|
||||
A lean surface over ``WalmartReviewsInput``; the scraper's ``ReviewItem`` is
|
||||
reused verbatim as the output element. Accepts product URLs or bare item ids —
|
||||
both resolve to a ``usItemId`` the reviews page is keyed on.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
from app.capabilities.core.validation import HttpUrlStr
|
||||
from app.proprietary.platforms.walmart import ReviewItem
|
||||
|
||||
MAX_WALMART_REVIEW_SOURCES = 20
|
||||
|
||||
|
||||
class ReviewsInput(BaseModel):
|
||||
urls: list[HttpUrlStr] = Field(
|
||||
default_factory=list,
|
||||
max_length=MAX_WALMART_REVIEW_SOURCES,
|
||||
description=(
|
||||
"Walmart product URLs (/ip/...) or reviews URLs to fetch reviews for. "
|
||||
"Provide these OR item_ids (at least one is required)."
|
||||
),
|
||||
)
|
||||
item_ids: list[str] = Field(
|
||||
default_factory=list,
|
||||
max_length=MAX_WALMART_REVIEW_SOURCES,
|
||||
description="Walmart numeric item ids (usItemId) to fetch reviews for.",
|
||||
)
|
||||
max_reviews: int = Field(
|
||||
default=200,
|
||||
ge=1,
|
||||
le=5000,
|
||||
description="Max reviews to return per product (10 per page).",
|
||||
)
|
||||
sort_by: Literal["most-recent", "most-helpful", "rating-high", "rating-low"] = (
|
||||
Field(default="most-recent", description="Review ordering.")
|
||||
)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _require_a_source(self) -> ReviewsInput:
|
||||
if not (self.urls or self.item_ids):
|
||||
raise ValueError("Provide at least one of 'urls' or 'item_ids'.")
|
||||
return self
|
||||
|
||||
def sources(self) -> list[str]:
|
||||
"""URLs and item ids merged; the scraper resolves each to a usItemId."""
|
||||
return [*self.urls, *self.item_ids]
|
||||
|
||||
@property
|
||||
def estimated_units(self) -> int:
|
||||
"""Worst-case billable reviews: up to ``max_reviews`` per source."""
|
||||
return (len(self.urls) + len(self.item_ids)) * self.max_reviews
|
||||
|
||||
|
||||
class ReviewsOutput(BaseModel):
|
||||
items: list[ReviewItem] = Field(
|
||||
default_factory=list,
|
||||
description="One item per review, in the scraper's emission order.",
|
||||
)
|
||||
|
||||
@property
|
||||
def billable_units(self) -> int:
|
||||
"""One returned review = one billable unit; error items are not billed."""
|
||||
return sum(
|
||||
1
|
||||
for item in self.items
|
||||
if not (item.model_extra and item.model_extra.get("error"))
|
||||
)
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
"""Walmart product scraping capability."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
"""Registration for the ``walmart.scrape`` capability."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.capabilities.core import BillingUnit, Capability, register_capability
|
||||
from app.capabilities.walmart.scrape.executor import build_scrape_executor
|
||||
from app.capabilities.walmart.scrape.schemas import ScrapeInput, ScrapeOutput
|
||||
|
||||
WALMART_SCRAPE = Capability(
|
||||
name="walmart.scrape",
|
||||
description=(
|
||||
"Scrape public Walmart product details, search/category listings, "
|
||||
"prices, sellers, variants, availability, and a sample of on-page reviews."
|
||||
),
|
||||
input_schema=ScrapeInput,
|
||||
output_schema=ScrapeOutput,
|
||||
executor=build_scrape_executor(),
|
||||
billing_unit=BillingUnit.WALMART_PRODUCT,
|
||||
docs_url="/docs/connectors/native/walmart",
|
||||
)
|
||||
|
||||
register_capability(WALMART_SCRAPE)
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
"""Executor for the ``walmart.scrape`` capability."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Awaitable, Callable
|
||||
|
||||
from app.capabilities.core import Executor
|
||||
from app.capabilities.core.progress import emit_progress
|
||||
from app.capabilities.walmart.scrape.schemas import (
|
||||
MAX_WALMART_RESULTS,
|
||||
ScrapeInput,
|
||||
ScrapeOutput,
|
||||
)
|
||||
from app.proprietary.platforms.walmart import WalmartScrapeInput, scrape_products
|
||||
|
||||
ScrapeFn = Callable[..., Awaitable[list[dict]]]
|
||||
|
||||
|
||||
def build_scrape_executor(scrape_fn: ScrapeFn | None = None) -> Executor:
|
||||
"""Bind the capability input mapping to a replaceable scraper function."""
|
||||
scrape_fn = scrape_fn or scrape_products
|
||||
|
||||
async def execute(payload: ScrapeInput) -> ScrapeOutput:
|
||||
input_model = WalmartScrapeInput(
|
||||
startUrls=payload.start_urls(),
|
||||
maxItemsPerStartUrl=payload.max_items,
|
||||
includeDetails=payload.include_details,
|
||||
includeReviewsSample=payload.include_reviews_sample,
|
||||
)
|
||||
emit_progress(
|
||||
"starting",
|
||||
"Scraping Walmart products",
|
||||
total=payload.estimated_units,
|
||||
unit="product",
|
||||
)
|
||||
items = await scrape_fn(input_model, limit=MAX_WALMART_RESULTS)
|
||||
emit_progress(
|
||||
"done",
|
||||
f"Scraped {sum('error' not in item for item in items)} product(s)",
|
||||
current=len(items),
|
||||
total=payload.estimated_units,
|
||||
unit="product",
|
||||
)
|
||||
return ScrapeOutput(items=items)
|
||||
|
||||
return execute
|
||||
65
surfsense_backend/app/capabilities/walmart/scrape/schemas.py
Normal file
65
surfsense_backend/app/capabilities/walmart/scrape/schemas.py
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
"""Input and output contracts for ``walmart.scrape``."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from urllib.parse import quote_plus
|
||||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
from app.capabilities.core.validation import HttpUrlStr
|
||||
from app.proprietary.platforms.walmart import ProductItem
|
||||
|
||||
MAX_WALMART_SOURCES = 20
|
||||
MAX_WALMART_RESULTS = 1000
|
||||
|
||||
|
||||
class ScrapeInput(BaseModel):
|
||||
"""Agent-facing controls for public Walmart product discovery and enrichment."""
|
||||
|
||||
urls: list[HttpUrlStr] = Field(default_factory=list, max_length=MAX_WALMART_SOURCES)
|
||||
search_terms: list[str] = Field(
|
||||
default_factory=list, max_length=MAX_WALMART_SOURCES
|
||||
)
|
||||
max_items: int = Field(default=10, ge=1, le=100)
|
||||
include_details: bool = True
|
||||
include_reviews_sample: bool = True
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _require_source(self) -> ScrapeInput:
|
||||
if not (self.urls or self.search_terms):
|
||||
raise ValueError("Provide at least one URL or search term.")
|
||||
if len(self.urls) + len(self.search_terms) > MAX_WALMART_SOURCES:
|
||||
raise ValueError(
|
||||
f"Provide no more than {MAX_WALMART_SOURCES} combined sources."
|
||||
)
|
||||
return self
|
||||
|
||||
def start_urls(self) -> list[str]:
|
||||
"""Direct URLs plus a search URL synthesized per search term."""
|
||||
searches = [
|
||||
f"https://www.walmart.com/search?q={quote_plus(term)}"
|
||||
for term in self.search_terms
|
||||
]
|
||||
return [*self.urls, *searches]
|
||||
|
||||
@property
|
||||
def estimated_units(self) -> int:
|
||||
"""Worst-case returned products within the hard per-run ceiling."""
|
||||
search_products = len(self.search_terms) * self.max_items
|
||||
direct_products = len(self.urls) * self.max_items
|
||||
return min(search_products + direct_products, MAX_WALMART_RESULTS)
|
||||
|
||||
|
||||
class ScrapeOutput(BaseModel):
|
||||
"""Products and structured per-input errors in emission order."""
|
||||
|
||||
items: list[ProductItem] = Field(default_factory=list)
|
||||
|
||||
@property
|
||||
def billable_units(self) -> int:
|
||||
"""Count successful products; error items are never billed."""
|
||||
return sum(
|
||||
1
|
||||
for item in self.items
|
||||
if not (item.model_extra and item.model_extra.get("error"))
|
||||
)
|
||||
|
|
@ -18,6 +18,8 @@ from typing import Literal
|
|||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.capabilities.core.validation import HttpUrlStr
|
||||
|
||||
MAX_START_URLS = 20
|
||||
"""Per-call cap on seed URLs: bounds a synchronous request's fan-out (05)."""
|
||||
|
||||
|
|
@ -29,7 +31,7 @@ MAX_CRAWL_PAGES = 200
|
|||
|
||||
|
||||
class CrawlInput(BaseModel):
|
||||
startUrls: list[str] = Field(
|
||||
startUrls: list[HttpUrlStr] = Field(
|
||||
min_length=1,
|
||||
max_length=MAX_START_URLS,
|
||||
description=(
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ from typing import Literal
|
|||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.capabilities.core.validation import HttpUrlStr
|
||||
from app.proprietary.platforms.youtube import CommentItem
|
||||
|
||||
MAX_COMMENT_VIDEOS = 20
|
||||
|
|
@ -17,7 +18,7 @@ MAX_COMMENT_VIDEOS = 20
|
|||
|
||||
|
||||
class CommentsInput(BaseModel):
|
||||
urls: list[str] = Field(
|
||||
urls: list[HttpUrlStr] = Field(
|
||||
min_length=1,
|
||||
max_length=MAX_COMMENT_VIDEOS,
|
||||
description="YouTube video URLs to fetch comments (and replies) for (1-20).",
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ from __future__ import annotations
|
|||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
from app.capabilities.core.validation import HttpUrlStr
|
||||
from app.proprietary.platforms.youtube import VideoItem
|
||||
|
||||
MAX_YOUTUBE_SOURCES = 20
|
||||
|
|
@ -17,7 +18,7 @@ MAX_YOUTUBE_SOURCES = 20
|
|||
|
||||
|
||||
class ScrapeInput(BaseModel):
|
||||
urls: list[str] = Field(
|
||||
urls: list[HttpUrlStr] = Field(
|
||||
default_factory=list,
|
||||
max_length=MAX_YOUTUBE_SOURCES,
|
||||
description=(
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ from celery.signals import (
|
|||
task_postrun,
|
||||
task_prerun,
|
||||
worker_process_init,
|
||||
worker_process_shutdown,
|
||||
)
|
||||
from dotenv import load_dotenv
|
||||
|
||||
|
|
@ -123,6 +124,18 @@ def init_worker(**kwargs):
|
|||
initialize_image_gen_router()
|
||||
|
||||
|
||||
@worker_process_shutdown.connect
|
||||
def shutdown_worker(**kwargs):
|
||||
"""Flush queued PostHog events before a Celery worker process exits.
|
||||
|
||||
The analytics client init is lazy (fork-safe), so there is nothing to
|
||||
start here — only a flush to avoid dropping events captured by tasks.
|
||||
"""
|
||||
from app.observability import analytics as ph_analytics
|
||||
|
||||
ph_analytics.shutdown()
|
||||
|
||||
|
||||
# Celery configuration, sourced from the central Config singleton
|
||||
CELERY_BROKER_URL = config.CELERY_BROKER_URL
|
||||
CELERY_RESULT_BACKEND = config.CELERY_RESULT_BACKEND
|
||||
|
|
|
|||
|
|
@ -736,6 +736,17 @@ class Config:
|
|||
# Comments are the cheapest per-item TikTok data, matching the per-comment
|
||||
# market (and YouTube's comment meter).
|
||||
TIKTOK_MICROS_PER_COMMENT = int(os.getenv("TIKTOK_MICROS_PER_COMMENT", "1500"))
|
||||
# Warmed-browser listings put Indeed on par with the other browser-driven
|
||||
# scrapers (Reddit, Instagram) rather than the cheaper API-backed meters.
|
||||
INDEED_SCRAPE_MICROS_PER_JOB = int(
|
||||
os.getenv("INDEED_SCRAPE_MICROS_PER_JOB", "3500")
|
||||
)
|
||||
# Walmart products come from server-rendered JSON behind residential proxies,
|
||||
# priced alongside Amazon's per-product meter.
|
||||
WALMART_MICROS_PER_PRODUCT = int(os.getenv("WALMART_MICROS_PER_PRODUCT", "3500"))
|
||||
# Reviews are 10 per page (many light requests per product), priced on the
|
||||
# cheaper per-review market like the Google Maps review meter.
|
||||
WALMART_MICROS_PER_REVIEW = int(os.getenv("WALMART_MICROS_PER_REVIEW", "1500"))
|
||||
# Retry an empty listing draw on a fresh rotating IP. Set to 1 for a static
|
||||
# proxy, where every retry re-hits the same exit.
|
||||
TIKTOK_LISTING_MAX_ATTEMPTS = int(os.getenv("TIKTOK_LISTING_MAX_ATTEMPTS", "3"))
|
||||
|
|
@ -850,6 +861,9 @@ class Config:
|
|||
# Auth
|
||||
AUTH_TYPE = os.getenv("AUTH_TYPE", "LOCAL")
|
||||
REGISTRATION_ENABLED = os.getenv("REGISTRATION_ENABLED", "TRUE").upper() == "TRUE"
|
||||
# Max workspaces a user may own. The frontend reads this through the
|
||||
# workspace limits route; do not duplicate this value client-side.
|
||||
MAX_WORKSPACES_PER_USER = int(os.getenv("MAX_WORKSPACES_PER_USER", "400"))
|
||||
|
||||
# Google OAuth
|
||||
GOOGLE_OAUTH_CLIENT_ID = os.getenv("GOOGLE_OAUTH_CLIENT_ID")
|
||||
|
|
@ -1165,6 +1179,19 @@ class Config:
|
|||
os.getenv("CRAWL_HEADED_XVFB_ENABLED", "FALSE").upper() == "TRUE"
|
||||
)
|
||||
|
||||
# PostHog server-side product analytics (opt-in, mirrors the OTel pattern:
|
||||
# no key set => the analytics wrapper is a silent no-op). Use the SAME
|
||||
# project key as the frontend's NEXT_PUBLIC_POSTHOG_KEY so server events
|
||||
# merge onto the persons the web app already identifies by user id.
|
||||
POSTHOG_API_KEY = os.getenv("POSTHOG_API_KEY")
|
||||
POSTHOG_HOST = os.getenv("POSTHOG_HOST", "https://us.i.posthog.com")
|
||||
# When true (default), the LLM-analytics LangChain handler suppresses
|
||||
# prompt/completion bodies ($ai_input / $ai_output_choices) and captures
|
||||
# only metrics — chat content includes users' private documents.
|
||||
POSTHOG_AI_PRIVACY_MODE = (
|
||||
os.getenv("POSTHOG_AI_PRIVACY_MODE", "TRUE").upper() == "TRUE"
|
||||
)
|
||||
|
||||
# Litellm TTS Configuration
|
||||
TTS_SERVICE = os.getenv("TTS_SERVICE")
|
||||
TTS_SERVICE_API_BASE = os.getenv("TTS_SERVICE_API_BASE")
|
||||
|
|
|
|||
|
|
@ -6,4 +6,4 @@ wrapper is a no-op when OTEL is not configured, so importing it from
|
|||
performance-critical paths is safe.
|
||||
"""
|
||||
|
||||
__all__ = ["bootstrap", "metrics", "otel"]
|
||||
__all__ = ["analytics", "bootstrap", "metrics", "otel"]
|
||||
|
|
|
|||
202
surfsense_backend/app/observability/analytics.py
Normal file
202
surfsense_backend/app/observability/analytics.py
Normal file
|
|
@ -0,0 +1,202 @@
|
|||
"""Server-side PostHog product analytics for SurfSense.
|
||||
|
||||
Opt-in, mirroring the OpenTelemetry bootstrap contract: when
|
||||
``POSTHOG_API_KEY`` is unset every function here is a silent no-op, so it is
|
||||
safe to call from hot paths (including async request handlers) and from
|
||||
self-hosted installs that never configure telemetry.
|
||||
|
||||
Design notes:
|
||||
- The underlying ``posthog`` client enqueues events onto a background
|
||||
consumer thread, so ``capture()`` is a non-blocking queue append; the only
|
||||
network I/O happens off-thread. ``shutdown()`` flushes and joins that thread
|
||||
and MUST run before a process exits or queued events are lost.
|
||||
- The client is created lazily on first use, never at import time. This keeps
|
||||
it fork-safe under Celery's prefork pool: a client (and its consumer thread)
|
||||
created in the parent would not survive ``fork()``, so each worker process
|
||||
builds its own on first capture.
|
||||
- ``distinct_id`` is always ``str(user.id)`` so server events merge onto the
|
||||
same PostHog persons the web frontend identifies (see
|
||||
``surfsense_web/components/providers/PostHogIdentify.tsx``).
|
||||
- Every event passes ``disable_geoip=True``; without it PostHog would resolve
|
||||
the *server's* IP and overwrite each person's real (client-derived) location.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import threading
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from app.config import config
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.auth.context import AuthContext
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_client: Any | None = None
|
||||
_init_attempted = False
|
||||
_lock = threading.Lock()
|
||||
|
||||
# Stamped on every backend event so client-observed (frontend) and
|
||||
# server-truth events are always distinguishable in PostHog.
|
||||
_SOURCE = "backend"
|
||||
|
||||
|
||||
def _get_client() -> Any | None:
|
||||
"""Return the process-local PostHog client, or ``None`` when disabled.
|
||||
|
||||
Lazy + fork-safe: built on first use inside whichever process (web worker
|
||||
or Celery worker) calls it, never at import time.
|
||||
"""
|
||||
global _client, _init_attempted
|
||||
|
||||
if _init_attempted:
|
||||
return _client
|
||||
|
||||
with _lock:
|
||||
if _init_attempted:
|
||||
return _client
|
||||
_init_attempted = True
|
||||
|
||||
api_key = config.POSTHOG_API_KEY
|
||||
if not api_key:
|
||||
# ponytail: opt-in like OTel — no key means telemetry is off, not
|
||||
# a misconfiguration. Stay silent so self-hosters see no noise.
|
||||
return None
|
||||
|
||||
try:
|
||||
from posthog import Posthog
|
||||
|
||||
_client = Posthog(
|
||||
project_api_key=api_key,
|
||||
host=config.POSTHOG_HOST,
|
||||
)
|
||||
except Exception:
|
||||
logger.warning("PostHog analytics init failed; disabling", exc_info=True)
|
||||
_client = None
|
||||
|
||||
return _client
|
||||
|
||||
|
||||
def is_enabled() -> bool:
|
||||
"""True when a PostHog client is configured and available."""
|
||||
return _get_client() is not None
|
||||
|
||||
|
||||
def get_client() -> Any | None:
|
||||
"""Raw PostHog client for integrations that need it (e.g. the LLM handler)."""
|
||||
return _get_client()
|
||||
|
||||
|
||||
def _client_label(auth: AuthContext) -> str:
|
||||
"""Best-effort ``client`` property derived from the auth principal.
|
||||
|
||||
``session`` can't be split into web vs desktop from auth alone, so callers
|
||||
that know better may override ``client`` in ``properties``.
|
||||
"""
|
||||
if auth.method == "system":
|
||||
return auth.source or "system"
|
||||
if auth.method == "pat":
|
||||
return "pat"
|
||||
return "web"
|
||||
|
||||
|
||||
def capture(
|
||||
event: str,
|
||||
*,
|
||||
distinct_id: str,
|
||||
properties: dict[str, Any] | None = None,
|
||||
groups: dict[str, str] | None = None,
|
||||
) -> None:
|
||||
"""Capture a product event. No-op (and never raises) when disabled.
|
||||
|
||||
Wrapped in try/except like the frontend ``safeCapture`` — analytics must
|
||||
never break a request. ``posthog`` v6 signature is ``capture(event,
|
||||
distinct_id=..., properties=...)`` (event first, distinct_id a kwarg).
|
||||
"""
|
||||
client = _get_client()
|
||||
if client is None:
|
||||
return
|
||||
|
||||
try:
|
||||
props = {"source": _SOURCE, **(properties or {})}
|
||||
client.capture(
|
||||
event,
|
||||
distinct_id=distinct_id,
|
||||
properties=props,
|
||||
groups=groups,
|
||||
disable_geoip=True,
|
||||
)
|
||||
except Exception:
|
||||
logger.debug("PostHog capture failed for %s", event, exc_info=True)
|
||||
|
||||
|
||||
def capture_for(
|
||||
auth: AuthContext,
|
||||
event: str,
|
||||
properties: dict[str, Any] | None = None,
|
||||
groups: dict[str, str] | None = None,
|
||||
) -> None:
|
||||
"""Capture an event attributed to an ``AuthContext`` principal.
|
||||
|
||||
Derives ``distinct_id`` from the user id and stamps ``auth_method`` and a
|
||||
best-effort ``client`` so events are attributable to their surface
|
||||
(web/desktop/pat/gateway/automation).
|
||||
"""
|
||||
if _get_client() is None:
|
||||
return
|
||||
|
||||
props = {
|
||||
"auth_method": auth.method,
|
||||
"client": _client_label(auth),
|
||||
**(properties or {}),
|
||||
}
|
||||
capture(
|
||||
event,
|
||||
distinct_id=str(auth.user.id),
|
||||
properties=props,
|
||||
groups=groups,
|
||||
)
|
||||
|
||||
|
||||
def group_identify(
|
||||
group_type: str,
|
||||
group_key: str,
|
||||
properties: dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
"""Upsert group properties (e.g. per-workspace metadata). No-op when disabled."""
|
||||
client = _get_client()
|
||||
if client is None:
|
||||
return
|
||||
|
||||
try:
|
||||
client.group_identify(
|
||||
group_type=group_type,
|
||||
group_key=group_key,
|
||||
properties=properties or {},
|
||||
)
|
||||
except Exception:
|
||||
logger.debug("PostHog group_identify failed for %s", group_type, exc_info=True)
|
||||
|
||||
|
||||
def shutdown() -> None:
|
||||
"""Flush queued events and stop the consumer thread. Safe to call always."""
|
||||
global _client
|
||||
client = _client
|
||||
if client is None:
|
||||
return
|
||||
try:
|
||||
client.shutdown()
|
||||
except Exception:
|
||||
logger.debug("PostHog shutdown failed", exc_info=True)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"capture",
|
||||
"capture_for",
|
||||
"get_client",
|
||||
"group_identify",
|
||||
"is_enabled",
|
||||
"shutdown",
|
||||
]
|
||||
|
|
@ -11,7 +11,10 @@ import logging
|
|||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.celery_app import celery_app
|
||||
from app.observability import analytics as ph_analytics
|
||||
from app.podcasts.persistence import PodcastRepository
|
||||
from app.podcasts.rendering import PodcastRenderer
|
||||
from app.podcasts.service import (
|
||||
|
|
@ -76,6 +79,30 @@ async def _render_audio(podcast_id: int) -> dict:
|
|||
podcast, storage_backend=backend_name, storage_key=key
|
||||
)
|
||||
await session.commit()
|
||||
|
||||
# Credit-consuming deliverable; the frontend never confirms the
|
||||
# render finished. Owner (workspace.user_id) resolved lazily so
|
||||
# disabled installs pay nothing for the extra query.
|
||||
if ph_analytics.is_enabled():
|
||||
# Local import: app.db <-> app.podcasts.persistence have a
|
||||
# module-init cycle; deferring keeps this task importable.
|
||||
from app.db import Workspace
|
||||
|
||||
owner_id = await session.scalar(
|
||||
select(Workspace.user_id).where(
|
||||
Workspace.id == podcast.workspace_id
|
||||
)
|
||||
)
|
||||
if owner_id:
|
||||
ph_analytics.capture(
|
||||
"podcast_generated",
|
||||
distinct_id=str(owner_id),
|
||||
properties={
|
||||
"workspace_id": podcast.workspace_id,
|
||||
"podcast_id": podcast_id,
|
||||
},
|
||||
groups={"workspace": str(podcast.workspace_id)},
|
||||
)
|
||||
except InvalidTransitionError:
|
||||
# A user back-out won the race (e.g. the regeneration was
|
||||
# reverted): drop the stale render and leave the row alone.
|
||||
|
|
|
|||
|
|
@ -35,7 +35,6 @@ import contextlib
|
|||
import logging
|
||||
import os
|
||||
import random
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from urllib.parse import urlsplit, urlunsplit
|
||||
|
|
@ -46,6 +45,7 @@ from app.proprietary.platforms.google_search import (
|
|||
captcha as _captcha,
|
||||
pool_store as _store,
|
||||
)
|
||||
from app.utils.browser_loop import in_browser_loop as _in_browser_loop
|
||||
from app.utils.captcha import captcha_enabled, get_captcha_config
|
||||
from app.utils.proxy import get_proxy_url
|
||||
|
||||
|
|
@ -393,41 +393,11 @@ _MOBILE_UA = "Mozilla/5.0 (Android 14; Mobile; rv:132.0) Gecko/132.0 Firefox/132
|
|||
_MOBILE_VIEWPORT = {"width": 412, "height": 915}
|
||||
|
||||
|
||||
# patchright launches Chromium via asyncio.create_subprocess_exec, which the
|
||||
# server's main loop cannot do on Windows (main.py pins a SelectorEventLoop
|
||||
# for psycopg; Selector loops raise NotImplementedError on subprocess_exec).
|
||||
# All browser work therefore runs on ONE dedicated background loop that is
|
||||
# explicitly subprocess-capable; callers await it across threads. This also
|
||||
# keeps the persistent AsyncStealthySession (and its async page_action) intact
|
||||
# — the sync-fetcher-in-a-thread pattern the other scrapers use would tear
|
||||
# down the browser on every fetch.
|
||||
_browser_loop: asyncio.AbstractEventLoop | None = None
|
||||
_browser_loop_guard = threading.Lock()
|
||||
|
||||
|
||||
def _get_browser_loop() -> asyncio.AbstractEventLoop:
|
||||
"""The lazily-started, process-wide event loop the browser lives on."""
|
||||
global _browser_loop
|
||||
with _browser_loop_guard:
|
||||
if _browser_loop is None:
|
||||
loop = (
|
||||
asyncio.ProactorEventLoop()
|
||||
if sys.platform == "win32"
|
||||
else asyncio.new_event_loop()
|
||||
)
|
||||
threading.Thread(
|
||||
target=loop.run_forever, name="google-search-browser", daemon=True
|
||||
).start()
|
||||
_browser_loop = loop
|
||||
return _browser_loop
|
||||
|
||||
|
||||
async def _in_browser_loop(coro):
|
||||
"""Run ``coro`` on the browser loop and await its result from this loop."""
|
||||
return await asyncio.wrap_future(
|
||||
asyncio.run_coroutine_threadsafe(coro, _get_browser_loop())
|
||||
)
|
||||
|
||||
# All browser work runs on the shared subprocess-capable loop (Windows: the
|
||||
# server's SelectorEventLoop cannot spawn Chromium; see app.utils.browser_loop).
|
||||
# This also keeps the persistent AsyncStealthySession (and its async
|
||||
# page_action) intact — the sync-fetcher-in-a-thread pattern would tear down
|
||||
# the browser on every fetch.
|
||||
|
||||
# One live browser per layout (desktop / mobile — the UA and viewport are
|
||||
# session-level context options). Launching Chromium costs ~5 s, so it's paid
|
||||
|
|
|
|||
|
|
@ -0,0 +1,13 @@
|
|||
"""Platform-native Indeed jobs scraper (anonymous, warmed browser session)."""
|
||||
|
||||
from .fetch import IndeedAccessBlockedError
|
||||
from .schemas import IndeedItem, IndeedScrapeInput
|
||||
from .scraper import iter_indeed, scrape_indeed
|
||||
|
||||
__all__ = [
|
||||
"IndeedAccessBlockedError",
|
||||
"IndeedItem",
|
||||
"IndeedScrapeInput",
|
||||
"iter_indeed",
|
||||
"scrape_indeed",
|
||||
]
|
||||
200
surfsense_backend/app/proprietary/platforms/indeed_jobs/fetch.py
Normal file
200
surfsense_backend/app/proprietary/platforms/indeed_jobs/fetch.py
Normal file
|
|
@ -0,0 +1,200 @@
|
|||
"""Browser-session fetch seam for the Indeed scraper.
|
||||
|
||||
Indeed fronts its origin with Cloudflare plus an anonymous-bot check that bounces
|
||||
cold sessions to ``secure.indeed.com/auth``. The working recipe: a persistent
|
||||
camoufox session that solves Cloudflare, warms on the domain home page, then
|
||||
navigates to ``/jobs`` in the same context so the clearance carries.
|
||||
|
||||
:class:`IndeedSession` warms per domain once, retries a blocked page on a fresh
|
||||
residential IP, and caps each navigation with a hard timeout so a stuck solve
|
||||
can't stall a run. All egress is through the residential proxy.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from collections.abc import Awaitable, Callable
|
||||
from contextlib import asynccontextmanager, suppress
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any, Protocol
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from app.utils.browser_loop import in_browser_loop
|
||||
from app.utils.proxy import get_proxy_url
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class IndeedAccessBlockedError(RuntimeError):
|
||||
"""Every rotated IP was bounced to Indeed's security wall."""
|
||||
|
||||
|
||||
# Per navigation; a stuck Cloudflare solve otherwise hangs the whole run.
|
||||
_PAGE_TIMEOUT_S = 75.0
|
||||
# Browser-internal timeout; kept above the page timeout so ours fires first.
|
||||
_SESSION_TIMEOUT_MS = 90_000
|
||||
_MAX_ROTATIONS = 3
|
||||
|
||||
# Markers of a Cloudflare / security-check interstitial served instead of jobs.
|
||||
_BLOCK_MARKERS = (
|
||||
"secure.indeed.com",
|
||||
"bot-detection",
|
||||
"security check",
|
||||
"challenge-platform",
|
||||
"just a moment",
|
||||
"verify you are human",
|
||||
"hcaptcha",
|
||||
)
|
||||
|
||||
|
||||
def now_iso() -> str:
|
||||
"""UTC timestamp in the millisecond ISO shape used by scraper output."""
|
||||
return datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"
|
||||
|
||||
|
||||
class _Session(Protocol):
|
||||
"""Minimal browser-session surface used here (real or fake)."""
|
||||
|
||||
async def start(self) -> Any: ...
|
||||
async def fetch(self, url: str, **kwargs: Any) -> Any: ...
|
||||
async def close(self) -> Any: ...
|
||||
|
||||
|
||||
def _default_session_factory() -> _Session:
|
||||
"""Build a proxied, Cloudflare-solving camoufox session.
|
||||
|
||||
``disable_resources`` skips images/fonts/media; job data is inline in the
|
||||
document, so this only trims bandwidth.
|
||||
"""
|
||||
from scrapling.fetchers import AsyncStealthySession
|
||||
|
||||
return AsyncStealthySession(
|
||||
headless=True,
|
||||
solve_cloudflare=True,
|
||||
network_idle=True,
|
||||
block_webrtc=True,
|
||||
disable_resources=True,
|
||||
timeout=_SESSION_TIMEOUT_MS,
|
||||
proxy=get_proxy_url(),
|
||||
)
|
||||
|
||||
|
||||
def _html(page: Any) -> str:
|
||||
"""Best-effort HTML body across scrapling response shapes."""
|
||||
for attr in ("html_content", "body", "text"):
|
||||
val = getattr(page, attr, None)
|
||||
if isinstance(val, bytes):
|
||||
val = val.decode("utf-8", "replace")
|
||||
if isinstance(val, str) and val:
|
||||
return val
|
||||
return ""
|
||||
|
||||
|
||||
def _looks_blocked(html: str, final_url: str) -> bool:
|
||||
"""Whether a response is an interstitial rather than a real page."""
|
||||
if not html:
|
||||
return True
|
||||
haystack = (final_url + " " + html[:6000]).lower()
|
||||
return any(marker in haystack for marker in _BLOCK_MARKERS)
|
||||
|
||||
|
||||
class IndeedSession:
|
||||
"""One warmed browser session that rotates its exit IP when blocked."""
|
||||
|
||||
def __init__(
|
||||
self, session_factory: Callable[[], _Session] = _default_session_factory
|
||||
) -> None:
|
||||
self._factory = session_factory
|
||||
self._session: _Session | None = None
|
||||
self._warmed: set[str] = set()
|
||||
self.rotations = 0
|
||||
|
||||
# The session's whole lifecycle (build, start, fetch, close) is marshalled
|
||||
# onto the shared browser loop: patchright can't spawn Chromium from the
|
||||
# server's Windows SelectorEventLoop (see app.utils.browser_loop), and its
|
||||
# internals are bound to the loop they started on.
|
||||
|
||||
async def start(self) -> None:
|
||||
async def _build_and_start() -> _Session:
|
||||
session = self._factory()
|
||||
await session.start()
|
||||
return session
|
||||
|
||||
self._session = await in_browser_loop(_build_and_start())
|
||||
|
||||
async def close(self) -> None:
|
||||
if self._session is not None:
|
||||
with suppress(Exception):
|
||||
await in_browser_loop(self._session.close())
|
||||
self._session = None
|
||||
self._warmed.clear()
|
||||
|
||||
async def _rotate(self) -> None:
|
||||
"""Drop the session for a fresh exit IP; clears warmed domains."""
|
||||
await self.close()
|
||||
self.rotations += 1
|
||||
await self.start()
|
||||
logger.info("[indeed] rotated session (rotation #%d)", self.rotations)
|
||||
|
||||
async def _timed_fetch(self, url: str, **kwargs: Any) -> Any:
|
||||
assert self._session is not None
|
||||
coro: Awaitable[Any] = self._session.fetch(url, **kwargs)
|
||||
# wait_for runs on the browser loop too, so its timeout task lives on
|
||||
# the same loop as the fetch it cancels.
|
||||
return await in_browser_loop(asyncio.wait_for(coro, timeout=_PAGE_TIMEOUT_S))
|
||||
|
||||
async def _ensure_warm(self, domain: str) -> None:
|
||||
"""Land on the domain home with a Google referer before scraping it."""
|
||||
if domain in self._warmed:
|
||||
return
|
||||
with suppress(Exception):
|
||||
await self._timed_fetch(f"https://{domain}/", google_search=True)
|
||||
self._warmed.add(domain)
|
||||
|
||||
async def fetch_html(self, url: str, *, max_rotations: int | None = None) -> str:
|
||||
"""Return a search/company/job page's HTML through the warmed session.
|
||||
|
||||
Rotates the IP and re-warms on a security-wall bounce or timeout; raises
|
||||
:class:`IndeedAccessBlockedError` once rotations are exhausted. ``max_rotations``
|
||||
overrides the default budget: pass ``0`` to fail fast on a systematically
|
||||
gated page (e.g. anonymous pagination) instead of burning rotations on a
|
||||
block no fresh IP will clear.
|
||||
"""
|
||||
if self._session is None:
|
||||
await self.start()
|
||||
budget = _MAX_ROTATIONS if max_rotations is None else max_rotations
|
||||
domain = urlparse(url).hostname or "www.indeed.com"
|
||||
attempt = 0
|
||||
while True:
|
||||
try:
|
||||
await self._ensure_warm(domain)
|
||||
page = await self._timed_fetch(url)
|
||||
html = _html(page)
|
||||
if not _looks_blocked(html, str(getattr(page, "url", "") or "")):
|
||||
return html
|
||||
logger.info("[indeed] blocked on %s", url)
|
||||
except TimeoutError:
|
||||
logger.warning("[indeed] fetch timed out on %s", url)
|
||||
except Exception as e:
|
||||
logger.warning("[indeed] fetch failed on %s: %s", url, e)
|
||||
|
||||
if attempt >= budget:
|
||||
raise IndeedAccessBlockedError(
|
||||
f"Indeed refused {url} after {attempt + 1} attempt(s)"
|
||||
)
|
||||
attempt += 1
|
||||
await self._rotate()
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def open_session(
|
||||
session_factory: Callable[[], _Session] = _default_session_factory,
|
||||
):
|
||||
"""Open an :class:`IndeedSession` and guarantee teardown."""
|
||||
session = IndeedSession(session_factory)
|
||||
await session.start()
|
||||
try:
|
||||
yield session
|
||||
finally:
|
||||
await session.close()
|
||||
|
|
@ -0,0 +1,327 @@
|
|||
"""Pure HTML/JSON -> item mapping for the Indeed scraper.
|
||||
|
||||
I/O-free and deterministic so it can be unit-tested against captured fixtures;
|
||||
the orchestrator stamps ``scrapedAt``.
|
||||
|
||||
Indeed embeds job data as a JS assignment::
|
||||
|
||||
window.mosaic.providerData["mosaic-provider-jobcards"]={"metaData":{...}};
|
||||
|
||||
The literal recurs hundreds of times in the page, so :func:`extract_jobcards_blob`
|
||||
anchors on the assignment (``...]=``) and brace-matches the balanced object.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from html import unescape
|
||||
from re import sub as _re_sub
|
||||
from typing import Any
|
||||
|
||||
_DEFAULT_BASE = "https://www.indeed.com"
|
||||
|
||||
_JOBCARDS_ANCHOR = 'window.mosaic.providerData["mosaic-provider-jobcards"]='
|
||||
|
||||
# A /viewjob page carries the posting model in ``window._rootProps`` (JSON,
|
||||
# under ``preloadedVJData``); older pages inlined it as ``window._initialData``.
|
||||
_ROOT_PROPS_ANCHOR = "window._rootProps"
|
||||
_ROOT_PROPS_KEY = "preloadedVJData"
|
||||
_INITIAL_DATA_ANCHOR = "window._initialData"
|
||||
|
||||
# Indeed's extractedSalary.type -> our SalaryPeriod.
|
||||
_SALARY_PERIODS = {
|
||||
"HOURLY": "hour",
|
||||
"DAILY": "day",
|
||||
"WEEKLY": "week",
|
||||
"MONTHLY": "month",
|
||||
"YEARLY": "year",
|
||||
}
|
||||
|
||||
|
||||
def _brace_match(text: str, start: int) -> str | None:
|
||||
"""Return the balanced ``{...}``/``[...]`` blob at ``text[start]``, quote-aware."""
|
||||
open_ch = text[start] if start < len(text) else ""
|
||||
close_ch = {"[": "]", "{": "}"}.get(open_ch)
|
||||
if close_ch is None:
|
||||
return None
|
||||
depth = 0
|
||||
i = start
|
||||
n = len(text)
|
||||
while i < n:
|
||||
ch = text[i]
|
||||
if ch == open_ch:
|
||||
depth += 1
|
||||
elif ch == close_ch:
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
return text[start : i + 1]
|
||||
elif ch == '"':
|
||||
i += 1
|
||||
while i < n and text[i] != '"':
|
||||
if text[i] == "\\":
|
||||
i += 1
|
||||
i += 1
|
||||
i += 1
|
||||
return None
|
||||
|
||||
|
||||
def extract_jobcards_blob(html: str) -> dict | None:
|
||||
"""Decode the ``mosaic-provider-jobcards`` assignment, or ``None`` if absent."""
|
||||
import json
|
||||
|
||||
idx = html.find(_JOBCARDS_ANCHOR)
|
||||
if idx == -1:
|
||||
return None
|
||||
brace = html.find("{", idx + len(_JOBCARDS_ANCHOR))
|
||||
if brace == -1:
|
||||
return None
|
||||
blob = _brace_match(html, brace)
|
||||
if not blob:
|
||||
return None
|
||||
try:
|
||||
data = json.loads(blob)
|
||||
except ValueError:
|
||||
return None
|
||||
return data if isinstance(data, dict) else None
|
||||
|
||||
|
||||
def _decode_assignment(html: str, anchor: str) -> dict | None:
|
||||
"""Decode the balanced JSON object assigned after ``anchor``, or ``None``."""
|
||||
import json
|
||||
|
||||
idx = html.find(anchor)
|
||||
if idx == -1:
|
||||
return None
|
||||
brace = html.find("{", idx + len(anchor))
|
||||
if brace == -1:
|
||||
return None
|
||||
blob = _brace_match(html, brace)
|
||||
if not blob:
|
||||
return None
|
||||
try:
|
||||
data = json.loads(blob)
|
||||
except ValueError:
|
||||
return None
|
||||
return data if isinstance(data, dict) else None
|
||||
|
||||
|
||||
def extract_initial_data(html: str) -> dict | None:
|
||||
"""Return a /viewjob posting model rooted at ``jobInfoWrapperModel``.
|
||||
|
||||
Prefers ``window._rootProps`` (JSON) unwrapped at ``preloadedVJData``; falls
|
||||
back to a legacy inline ``window._initialData`` blob. ``window._initialData``
|
||||
is now a JS object literal that references other globals, so it is not JSON
|
||||
and is skipped when the JSON parse fails.
|
||||
"""
|
||||
root = _decode_assignment(html, _ROOT_PROPS_ANCHOR)
|
||||
if isinstance(root, dict):
|
||||
vj = root.get(_ROOT_PROPS_KEY)
|
||||
if isinstance(vj, dict) and vj.get("jobInfoWrapperModel"):
|
||||
return vj
|
||||
legacy = _decode_assignment(html, _INITIAL_DATA_ANCHOR)
|
||||
if isinstance(legacy, dict) and legacy.get("jobInfoWrapperModel"):
|
||||
return legacy
|
||||
return None
|
||||
|
||||
|
||||
def job_results(blob: dict | None) -> list[dict[str, Any]]:
|
||||
"""Return the raw job records from a decoded blob."""
|
||||
if not isinstance(blob, dict):
|
||||
return []
|
||||
results = (
|
||||
blob.get("metaData", {}).get("mosaicProviderJobCardsModel", {}).get("results")
|
||||
)
|
||||
if not isinstance(results, list):
|
||||
return []
|
||||
return [r for r in results if isinstance(r, dict)]
|
||||
|
||||
|
||||
def _utc_from_ms(value: Any) -> str | None:
|
||||
"""Epoch milliseconds -> millisecond ISO string."""
|
||||
if isinstance(value, bool) or not isinstance(value, int | float):
|
||||
return None
|
||||
dt = datetime.fromtimestamp(float(value) / 1000.0, tz=UTC)
|
||||
return dt.strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"
|
||||
|
||||
|
||||
def _int(value: Any) -> int | None:
|
||||
"""Coerce to int, dropping bools."""
|
||||
if isinstance(value, bool):
|
||||
return None
|
||||
if isinstance(value, int):
|
||||
return value
|
||||
if isinstance(value, float):
|
||||
return int(value)
|
||||
return None
|
||||
|
||||
|
||||
def _abs_url(path: Any, base_url: str) -> str | None:
|
||||
"""Resolve an Indeed-relative path against ``base_url``; keep absolute URLs."""
|
||||
if not isinstance(path, str) or not path:
|
||||
return None
|
||||
if path.startswith("http"):
|
||||
return path
|
||||
return f"{base_url}{path if path.startswith('/') else '/' + path}"
|
||||
|
||||
|
||||
def _clean_snippet(snippet: Any) -> str | None:
|
||||
"""Strip tags and decode entities into plain text."""
|
||||
if not isinstance(snippet, str) or not snippet:
|
||||
return None
|
||||
text = _re_sub(r"<[^>]+>", " ", snippet)
|
||||
text = unescape(text)
|
||||
return _re_sub(r"\s+", " ", text).strip() or None
|
||||
|
||||
|
||||
def _taxonomy(raw: dict[str, Any]) -> dict[str, list[str]]:
|
||||
"""Flatten ``taxonomyAttributes`` into ``{group label: [attribute labels]}``."""
|
||||
out: dict[str, list[str]] = {}
|
||||
for group in raw.get("taxonomyAttributes") or []:
|
||||
if not isinstance(group, dict):
|
||||
continue
|
||||
label = group.get("label")
|
||||
attrs = group.get("attributes")
|
||||
if isinstance(label, str) and isinstance(attrs, list):
|
||||
out[label] = [
|
||||
a["label"]
|
||||
for a in attrs
|
||||
if isinstance(a, dict) and isinstance(a.get("label"), str)
|
||||
]
|
||||
return out
|
||||
|
||||
|
||||
def _job_types(raw: dict[str, Any], taxo: dict[str, list[str]]) -> list[str]:
|
||||
"""Job types from ``jobTypes`` then the taxonomy, deduped and order-stable."""
|
||||
seen: dict[str, None] = {}
|
||||
for jt in raw.get("jobTypes") or []:
|
||||
if isinstance(jt, str):
|
||||
seen.setdefault(jt, None)
|
||||
for label in ("job-types", "job-types-cc"):
|
||||
for jt in taxo.get(label, []):
|
||||
seen.setdefault(jt, None)
|
||||
return list(seen)
|
||||
|
||||
|
||||
def _salary(raw: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Flatten salary from ``salarySnippet`` (text) + ``extractedSalary`` (bounds)."""
|
||||
snippet = raw.get("salarySnippet") or {}
|
||||
extracted = raw.get("extractedSalary") or {}
|
||||
estimated = raw.get("estimatedSalary") or {}
|
||||
source = extracted or estimated
|
||||
text = snippet.get("text") if isinstance(snippet, dict) else None
|
||||
return {
|
||||
"salaryText": text if isinstance(text, str) else None,
|
||||
"salaryMin": source.get("min") if isinstance(source, dict) else None,
|
||||
"salaryMax": source.get("max") if isinstance(source, dict) else None,
|
||||
"currency": snippet.get("currency") if isinstance(snippet, dict) else None,
|
||||
"period": _SALARY_PERIODS.get(
|
||||
source.get("type") if isinstance(source, dict) else None
|
||||
),
|
||||
"isEstimated": bool(estimated) and not extracted,
|
||||
}
|
||||
|
||||
|
||||
def _is_remote(raw: dict[str, Any], taxo: dict[str, list[str]]) -> bool:
|
||||
"""Resolve remote/hybrid across Indeed's several signals."""
|
||||
if raw.get("remoteLocation") is True:
|
||||
return True
|
||||
if isinstance(raw.get("remoteWorkModel"), dict):
|
||||
return True
|
||||
return bool(taxo.get("remote"))
|
||||
|
||||
|
||||
def parse_job(raw: dict[str, Any], *, base_url: str = _DEFAULT_BASE) -> dict[str, Any]:
|
||||
"""Map one raw ``results[]`` record to a flat item dict.
|
||||
|
||||
``base_url`` is the country domain the record came from, so job and company
|
||||
URLs resolve to the right host.
|
||||
"""
|
||||
taxo = _taxonomy(raw)
|
||||
job_key = raw.get("jobkey")
|
||||
remote_model = raw.get("remoteWorkModel")
|
||||
return {
|
||||
"jobKey": job_key if isinstance(job_key, str) else None,
|
||||
"title": raw.get("displayTitle") or raw.get("title"),
|
||||
"jobUrl": f"{base_url}/viewjob?jk={job_key}" if job_key else None,
|
||||
"applyUrl": raw.get("thirdPartyApplyUrl") or None,
|
||||
"company": raw.get("company") or raw.get("truncatedCompany"),
|
||||
"companyUrl": _abs_url(raw.get("companyOverviewLink"), base_url),
|
||||
"companyRating": raw.get("companyRating"),
|
||||
"companyReviewCount": _int(raw.get("companyReviewCount")),
|
||||
"formattedLocation": raw.get("formattedLocation"),
|
||||
"city": raw.get("jobLocationCity"),
|
||||
"state": raw.get("jobLocationState"),
|
||||
"postalCode": raw.get("jobLocationPostal"),
|
||||
"country": raw.get("country"),
|
||||
"isRemote": _is_remote(raw, taxo),
|
||||
"remoteType": remote_model.get("type")
|
||||
if isinstance(remote_model, dict)
|
||||
else None,
|
||||
"jobTypes": _job_types(raw, taxo),
|
||||
"salary": _salary(raw),
|
||||
"benefits": taxo.get("benefits", []),
|
||||
"descriptionText": _clean_snippet(raw.get("snippet")),
|
||||
"descriptionHtml": None,
|
||||
"sponsored": raw.get("sponsored"),
|
||||
"isNew": raw.get("newJob"),
|
||||
"urgentlyHiring": raw.get("urgentlyHiring"),
|
||||
"expired": raw.get("expired"),
|
||||
"indeedApplyEnabled": raw.get("indeedApplyEnabled"),
|
||||
"age": raw.get("formattedRelativeTime"),
|
||||
"datePublished": _utc_from_ms(raw.get("pubDate")),
|
||||
"createdAt": _utc_from_ms(raw.get("createDate")),
|
||||
}
|
||||
|
||||
|
||||
def _detail_salary(hdr: dict[str, Any]) -> dict[str, Any] | None:
|
||||
"""Salary from the detail header's flat ``salaryMin/Max/Currency/Type`` fields."""
|
||||
smin = hdr.get("salaryMin")
|
||||
smax = hdr.get("salaryMax")
|
||||
if smin is None and smax is None:
|
||||
return None
|
||||
return {
|
||||
"salaryText": None,
|
||||
"salaryMin": smin,
|
||||
"salaryMax": smax,
|
||||
"currency": hdr.get("salaryCurrency"),
|
||||
"period": _SALARY_PERIODS.get(hdr.get("salaryType")),
|
||||
"isEstimated": False,
|
||||
}
|
||||
|
||||
|
||||
def parse_job_detail(html: str, *, base_url: str = _DEFAULT_BASE) -> dict[str, Any]:
|
||||
"""Map a /viewjob page to enrichment fields (empty dict if not a job page).
|
||||
|
||||
Returns only fields the detail page actually carries, so the caller can merge
|
||||
it onto a listing item without clobbering known values with blanks. The full
|
||||
description (``sanitizedJobDescription``) is the field listings never have.
|
||||
"""
|
||||
data = extract_initial_data(html)
|
||||
if not isinstance(data, dict):
|
||||
return {}
|
||||
jim = (data.get("jobInfoWrapperModel") or {}).get("jobInfoModel") or {}
|
||||
if not isinstance(jim, dict):
|
||||
return {}
|
||||
hdr = jim.get("jobInfoHeaderModel")
|
||||
hdr = hdr if isinstance(hdr, dict) else {}
|
||||
taxo = _taxonomy(hdr)
|
||||
desc_html = jim.get("sanitizedJobDescription")
|
||||
desc_html = desc_html if isinstance(desc_html, str) and desc_html else None
|
||||
remote_model = hdr.get("remoteWorkModel")
|
||||
out: dict[str, Any] = {
|
||||
"descriptionHtml": desc_html,
|
||||
"descriptionText": _clean_snippet(desc_html),
|
||||
"title": hdr.get("jobTitle"),
|
||||
"company": hdr.get("companyName"),
|
||||
"companyUrl": _abs_url(hdr.get("companyOverviewLink"), base_url),
|
||||
"formattedLocation": hdr.get("formattedLocation") or data.get("jobLocation"),
|
||||
"remoteType": remote_model.get("type")
|
||||
if isinstance(remote_model, dict)
|
||||
else None,
|
||||
"jobTypes": _job_types(hdr, taxo),
|
||||
"benefits": taxo.get("benefits", []),
|
||||
"salary": _detail_salary(hdr),
|
||||
}
|
||||
if _is_remote(hdr, taxo):
|
||||
out["isRemote"] = True
|
||||
return {k: v for k, v in out.items() if v not in (None, [], {})}
|
||||
|
|
@ -0,0 +1,122 @@
|
|||
# ruff: noqa: N815 - field names intentionally use the public camelCase API
|
||||
"""Input/output models for the Indeed scraper.
|
||||
|
||||
Anonymous scraper: there is no auth field. Fields absent from a listing (full
|
||||
description, benefits) stay ``None``/``[]`` until a detail fetch fills them.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
IndeedSort = Literal["relevance", "date"]
|
||||
IndeedJobType = Literal[
|
||||
"fulltime",
|
||||
"parttime",
|
||||
"contract",
|
||||
"internship",
|
||||
"temporary",
|
||||
"permanent",
|
||||
"seasonal",
|
||||
"freelance",
|
||||
]
|
||||
IndeedLevel = Literal["entry_level", "mid_level", "senior_level"]
|
||||
IndeedRemote = Literal["remote", "hybrid"]
|
||||
SalaryPeriod = Literal["hour", "day", "week", "month", "year"]
|
||||
|
||||
|
||||
class StartUrl(BaseModel):
|
||||
"""A direct URL entry; extra keys ignored."""
|
||||
|
||||
model_config = ConfigDict(extra="allow")
|
||||
|
||||
url: str
|
||||
|
||||
|
||||
class IndeedScrapeInput(BaseModel):
|
||||
"""Indeed scraper input. Caps are collector policy, enforced by ``scrape_indeed``."""
|
||||
|
||||
model_config = ConfigDict(extra="allow")
|
||||
|
||||
# Discovery: direct URLs and/or search queries.
|
||||
startUrls: list[StartUrl] = Field(default_factory=list)
|
||||
queries: list[str] = Field(default_factory=list)
|
||||
|
||||
# Search parameters applied to ``queries``.
|
||||
country: str = "us"
|
||||
location: str | None = None
|
||||
radius: int | None = None
|
||||
jobType: IndeedJobType | None = None
|
||||
level: IndeedLevel | None = None
|
||||
remote: IndeedRemote | None = None
|
||||
fromDays: int | None = None
|
||||
sort: IndeedSort = "relevance"
|
||||
|
||||
# Fetch each job's detail page for the full description.
|
||||
scrapeJobDetails: bool = False
|
||||
|
||||
maxItems: int = Field(default=25, ge=0)
|
||||
maxItemsPerQuery: int = Field(default=25, ge=0)
|
||||
|
||||
|
||||
class Salary(BaseModel):
|
||||
"""Salary block; fields are ``None`` when Indeed omits pay."""
|
||||
|
||||
model_config = ConfigDict(extra="allow")
|
||||
|
||||
salaryText: str | None = None
|
||||
salaryMin: float | None = None
|
||||
salaryMax: float | None = None
|
||||
currency: str | None = None
|
||||
period: SalaryPeriod | None = None
|
||||
isEstimated: bool | None = None
|
||||
|
||||
|
||||
class IndeedItem(BaseModel):
|
||||
"""One job posting. ``extra="allow"`` keeps the contract additive."""
|
||||
|
||||
model_config = ConfigDict(extra="allow")
|
||||
|
||||
jobKey: str | None = None
|
||||
title: str | None = None
|
||||
jobUrl: str | None = None
|
||||
applyUrl: str | None = None
|
||||
|
||||
company: str | None = None
|
||||
companyUrl: str | None = None
|
||||
companyRating: float | None = None
|
||||
companyReviewCount: int | None = None
|
||||
|
||||
formattedLocation: str | None = None
|
||||
city: str | None = None
|
||||
state: str | None = None
|
||||
postalCode: str | None = None
|
||||
country: str | None = None
|
||||
isRemote: bool | None = None
|
||||
remoteType: str | None = None
|
||||
|
||||
jobTypes: list[str] = Field(default_factory=list)
|
||||
salary: Salary = Field(default_factory=Salary)
|
||||
benefits: list[str] = Field(default_factory=list)
|
||||
|
||||
descriptionText: str | None = None
|
||||
descriptionHtml: str | None = None
|
||||
|
||||
sponsored: bool | None = None
|
||||
isNew: bool | None = None
|
||||
urgentlyHiring: bool | None = None
|
||||
expired: bool | None = None
|
||||
indeedApplyEnabled: bool | None = None
|
||||
|
||||
age: str | None = None
|
||||
datePublished: str | None = None
|
||||
createdAt: str | None = None
|
||||
scrapedAt: str | None = None
|
||||
|
||||
source: str = "indeed"
|
||||
|
||||
def to_output(self) -> dict[str, Any]:
|
||||
"""Serialize to the flat output dict, keeping extras."""
|
||||
return self.model_dump(exclude_none=False)
|
||||
|
|
@ -0,0 +1,178 @@
|
|||
"""Orchestrator for the Indeed scraper.
|
||||
|
||||
:func:`iter_indeed` streams deduped job items from one warmed session; each
|
||||
search/company target contributes its first page. :func:`scrape_indeed` collects
|
||||
the stream under a caller ``limit``. Targets run sequentially to reuse the session.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import AsyncIterator
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from .fetch import IndeedSession, now_iso, open_session
|
||||
from .parsers import (
|
||||
extract_jobcards_blob,
|
||||
job_results,
|
||||
parse_job,
|
||||
parse_job_detail,
|
||||
)
|
||||
from .schemas import IndeedItem, IndeedScrapeInput
|
||||
from .url_resolver import build_search_url, resolve_url
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
__all__ = ["iter_indeed", "scrape_indeed"]
|
||||
|
||||
|
||||
def _emit(partial: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Stamp ``scrapedAt`` and normalize through the output model."""
|
||||
return IndeedItem(**{**partial, "scrapedAt": now_iso()}).to_output()
|
||||
|
||||
|
||||
async def _search_items(
|
||||
session: IndeedSession, url: str, *, domain: str, max_items: int
|
||||
) -> AsyncIterator[dict[str, Any]]:
|
||||
"""Yield deduped job cards from one search/company page.
|
||||
|
||||
ponytail: caps a query at its first page (~15 jobs) — anonymous Indeed gates
|
||||
``start>=10``; deeper depth needs an authenticated session or Indeed's API.
|
||||
"""
|
||||
if max_items <= 0:
|
||||
return
|
||||
base_url = f"https://{domain}"
|
||||
html = await session.fetch_html(url)
|
||||
seen: set[str] = set()
|
||||
emitted = 0
|
||||
for raw in job_results(extract_jobcards_blob(html)):
|
||||
item = parse_job(raw, base_url=base_url)
|
||||
job_key = item.get("jobKey")
|
||||
if isinstance(job_key, str):
|
||||
if job_key in seen:
|
||||
continue
|
||||
seen.add(job_key)
|
||||
yield _emit(item)
|
||||
emitted += 1
|
||||
if emitted >= max_items:
|
||||
return
|
||||
|
||||
|
||||
def _targets(input_model: IndeedScrapeInput) -> list[tuple[str, str, str]]:
|
||||
"""Resolve inputs to ``(kind, url, domain)`` targets.
|
||||
|
||||
``startUrls`` take precedence over ``queries``. ``kind`` is ``search`` for
|
||||
query-built and search/company URLs, or ``job`` for a ``/viewjob`` URL.
|
||||
"""
|
||||
if input_model.startUrls:
|
||||
out: list[tuple[str, str, str]] = []
|
||||
for entry in input_model.startUrls:
|
||||
resolved = resolve_url(entry.url)
|
||||
if resolved is None:
|
||||
logger.warning("[indeed] skipping unrecognized URL: %s", entry.url)
|
||||
continue
|
||||
kind = "job" if resolved.kind == "job" else "search"
|
||||
out.append((kind, resolved.url, resolved.domain))
|
||||
return out
|
||||
|
||||
domain = None
|
||||
urls: list[tuple[str, str, str]] = []
|
||||
for query in input_model.queries:
|
||||
url = build_search_url(
|
||||
query,
|
||||
country=input_model.country,
|
||||
location=input_model.location,
|
||||
radius=input_model.radius,
|
||||
job_type=input_model.jobType,
|
||||
level=input_model.level,
|
||||
remote=input_model.remote,
|
||||
from_days=input_model.fromDays,
|
||||
sort=input_model.sort,
|
||||
)
|
||||
domain = domain or urlparse(url).hostname or "www.indeed.com"
|
||||
urls.append(("search", url, domain))
|
||||
return urls
|
||||
|
||||
|
||||
async def _enrich(session: IndeedSession, item: dict[str, Any], base_url: str) -> None:
|
||||
"""Merge a job's /viewjob detail (full description, etc.) onto ``item`` in place.
|
||||
|
||||
Best-effort: a blocked or malformed detail page leaves the listing fields as-is
|
||||
rather than failing the run.
|
||||
"""
|
||||
job_url = item.get("jobUrl")
|
||||
if not isinstance(job_url, str):
|
||||
return
|
||||
try:
|
||||
# Fail fast: enrichment is best-effort, so a gated detail page must not
|
||||
# rotate IPs and eat the run's time budget for one job's description.
|
||||
html = await session.fetch_html(job_url, max_rotations=0)
|
||||
detail = parse_job_detail(html, base_url=base_url)
|
||||
except Exception as exc:
|
||||
logger.warning("[indeed] detail fetch failed for %s: %s", job_url, exc)
|
||||
return
|
||||
item.update(detail)
|
||||
|
||||
|
||||
async def _job_item(
|
||||
session: IndeedSession, url: str, base_url: str
|
||||
) -> dict[str, Any] | None:
|
||||
"""Scrape a single /viewjob URL into an item from its detail page alone."""
|
||||
detail = parse_job_detail(await session.fetch_html(url), base_url=base_url)
|
||||
if not detail:
|
||||
return None
|
||||
return _emit({"jobUrl": url, "source": "indeed", **detail})
|
||||
|
||||
|
||||
async def iter_indeed(
|
||||
input_model: IndeedScrapeInput, session: IndeedSession
|
||||
) -> AsyncIterator[dict[str, Any]]:
|
||||
"""Stream flat job items for every target, deduped by ``jobKey`` across all."""
|
||||
global_seen: set[str] = set()
|
||||
for kind, url, domain in _targets(input_model):
|
||||
base_url = f"https://{domain}"
|
||||
if kind == "job":
|
||||
item = await _job_item(session, url, base_url)
|
||||
if item is not None:
|
||||
yield item
|
||||
continue
|
||||
async for item in _search_items(
|
||||
session, url, domain=domain, max_items=input_model.maxItemsPerQuery
|
||||
):
|
||||
job_key = item.get("jobKey")
|
||||
if isinstance(job_key, str):
|
||||
if job_key in global_seen:
|
||||
continue
|
||||
global_seen.add(job_key)
|
||||
if input_model.scrapeJobDetails:
|
||||
await _enrich(session, item, base_url)
|
||||
yield item
|
||||
|
||||
|
||||
async def scrape_indeed(
|
||||
input_model: IndeedScrapeInput,
|
||||
*,
|
||||
limit: int | None = None,
|
||||
session: IndeedSession | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Collect :func:`iter_indeed` into a list under an optional ``limit``.
|
||||
|
||||
Opens a warmed session when one is not supplied. ``limit`` is a request-time
|
||||
guard, not a ceiling baked into the stream.
|
||||
"""
|
||||
from app.capabilities.core.progress import emit_progress
|
||||
|
||||
async def _collect(sess: IndeedSession) -> list[dict[str, Any]]:
|
||||
results: list[dict[str, Any]] = []
|
||||
async for item in iter_indeed(input_model, sess):
|
||||
results.append(item)
|
||||
emit_progress("scraping", current=len(results), total=limit, unit="item")
|
||||
if limit is not None and len(results) >= limit:
|
||||
break
|
||||
return results
|
||||
|
||||
if session is not None:
|
||||
return await _collect(session)
|
||||
async with open_session() as sess:
|
||||
return await _collect(sess)
|
||||
|
|
@ -0,0 +1,124 @@
|
|||
"""Classify Indeed URLs and build search URLs.
|
||||
|
||||
Recognizes search pages (``/jobs?q=&l=``), company pages (``/cmp/<slug>/jobs``),
|
||||
and single jobs (``/viewjob?jk=``); other hosts resolve to ``None``. Also owns
|
||||
the country->domain map so classification and URL building share one source.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Literal
|
||||
from urllib.parse import parse_qs, urlencode, urlparse
|
||||
|
||||
ResolvedKind = Literal["search", "company", "job"]
|
||||
|
||||
# Locale subdomains that deviate from the ISO code; others map to <cc>.indeed.com.
|
||||
_DOMAIN_OVERRIDES = {"us": "www", "gb": "uk"}
|
||||
|
||||
_JT_VALUES = frozenset(
|
||||
{
|
||||
"fulltime",
|
||||
"parttime",
|
||||
"contract",
|
||||
"internship",
|
||||
"temporary",
|
||||
"permanent",
|
||||
"seasonal",
|
||||
"freelance",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ResolvedUrl:
|
||||
kind: ResolvedKind
|
||||
value: str # search query, company slug, or job key
|
||||
url: str
|
||||
domain: str
|
||||
location: str | None = None
|
||||
params: dict[str, str] = field(default_factory=dict)
|
||||
|
||||
|
||||
def _is_indeed_host(hostname: str | None) -> bool:
|
||||
if not hostname:
|
||||
return False
|
||||
h = hostname.lower()
|
||||
return h == "indeed.com" or h.endswith(".indeed.com")
|
||||
|
||||
|
||||
def country_domain(country: str) -> str:
|
||||
"""Country code -> Indeed host, e.g. ``us`` -> ``www.indeed.com``."""
|
||||
cc = (country or "us").strip().lower()
|
||||
return f"{_DOMAIN_OVERRIDES.get(cc, cc)}.indeed.com"
|
||||
|
||||
|
||||
def resolve_url(url: str) -> ResolvedUrl | None:
|
||||
"""Classify an Indeed URL into a scrape job, or ``None`` if unrecognized."""
|
||||
parsed = urlparse(url)
|
||||
if not _is_indeed_host(parsed.hostname):
|
||||
return None
|
||||
domain = parsed.hostname or "www.indeed.com"
|
||||
path = (parsed.path or "").rstrip("/")
|
||||
query = parse_qs(parsed.query)
|
||||
segments = [s for s in path.split("/") if s]
|
||||
|
||||
# /viewjob?jk=<key>
|
||||
if path.endswith("/viewjob") or segments[:1] == ["viewjob"]:
|
||||
jk = query.get("jk", [None])[0]
|
||||
return ResolvedUrl("job", jk, url, domain) if jk else None
|
||||
|
||||
# /cmp/<slug>/jobs
|
||||
if segments[:1] == ["cmp"] and "jobs" in segments and len(segments) >= 2:
|
||||
return ResolvedUrl("company", segments[1], url, domain)
|
||||
|
||||
# /jobs?q=&l=
|
||||
if path.endswith("/jobs") or segments[-1:] == ["jobs"]:
|
||||
q = query.get("q", [""])[0]
|
||||
loc = query.get("l", [None])[0]
|
||||
extra = {
|
||||
k: v[0]
|
||||
for k, v in query.items()
|
||||
if k in ("radius", "sort", "fromage", "jt", "explvl") and v
|
||||
}
|
||||
return ResolvedUrl("search", q, url, domain, location=loc, params=extra)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def build_search_url(
|
||||
query: str,
|
||||
*,
|
||||
country: str = "us",
|
||||
location: str | None = None,
|
||||
radius: int | None = None,
|
||||
job_type: str | None = None,
|
||||
level: str | None = None,
|
||||
remote: str | None = None,
|
||||
from_days: int | None = None,
|
||||
sort: str = "relevance",
|
||||
start: int = 0,
|
||||
) -> str:
|
||||
"""Build an Indeed ``/jobs`` search URL.
|
||||
|
||||
Remote/hybrid is passed as a query keyword; Indeed's structured ``sc``
|
||||
attribute codes rotate and aren't stable to hardcode.
|
||||
"""
|
||||
domain = country_domain(country)
|
||||
q = f"{query} {remote}".strip() if remote else query
|
||||
params: dict[str, str] = {"q": q}
|
||||
if location:
|
||||
params["l"] = location
|
||||
if radius is not None:
|
||||
params["radius"] = str(radius)
|
||||
if job_type in _JT_VALUES:
|
||||
params["jt"] = job_type # type: ignore[assignment]
|
||||
if level:
|
||||
params["explvl"] = level
|
||||
if from_days is not None:
|
||||
params["fromage"] = str(from_days)
|
||||
if sort == "date":
|
||||
params["sort"] = "date"
|
||||
if start:
|
||||
params["start"] = str(start)
|
||||
return f"https://{domain}/jobs?{urlencode(params)}"
|
||||
|
|
@ -43,6 +43,7 @@ from urllib.parse import urlencode
|
|||
|
||||
from scrapling.fetchers import AsyncFetcher, AsyncStealthySession, FetcherSession
|
||||
|
||||
from app.utils.browser_loop import in_browser_loop
|
||||
from app.utils.proxy import get_proxy_url, get_sticky_proxy_url
|
||||
|
||||
# Shared cross-country rotation walk (also used by the TikTok sibling). Kept under
|
||||
|
|
@ -331,18 +332,24 @@ async def warm_session(proxy: str | None) -> dict[str, str] | None:
|
|||
"""
|
||||
if proxy is None:
|
||||
return None
|
||||
|
||||
# Runs on the shared browser loop: patchright can't spawn Chromium from the
|
||||
# server's Windows SelectorEventLoop (see app.utils.browser_loop).
|
||||
async def _warm() -> dict[str, str] | None:
|
||||
async with AsyncStealthySession(
|
||||
headless=True,
|
||||
google_search=True,
|
||||
network_idle=False,
|
||||
proxy=proxy,
|
||||
timeout=_WARM_TIMEOUT_MS,
|
||||
) as sess:
|
||||
page = await sess.fetch(_WARM_HTML_URL)
|
||||
jar = _browser_cookie_jar(page)
|
||||
return jar if _LOID_COOKIE in jar else None
|
||||
|
||||
async with _warm_slots:
|
||||
try:
|
||||
async with AsyncStealthySession(
|
||||
headless=True,
|
||||
google_search=True,
|
||||
network_idle=False,
|
||||
proxy=proxy,
|
||||
timeout=_WARM_TIMEOUT_MS,
|
||||
) as sess:
|
||||
page = await sess.fetch(_WARM_HTML_URL)
|
||||
jar = _browser_cookie_jar(page)
|
||||
return jar if _LOID_COOKIE in jar else None
|
||||
return await in_browser_loop(_warm())
|
||||
except Exception as e: # a browser crash must not abort the flow
|
||||
logger.warning("[reddit] browser warm failed: %s", e)
|
||||
return None
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ from __future__ import annotations
|
|||
|
||||
from typing import Any, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
RedditSort = Literal["relevance", "hot", "top", "new", "rising", "comments"]
|
||||
RedditTime = Literal["all", "hour", "day", "week", "month", "year"]
|
||||
|
|
@ -44,6 +44,23 @@ class RedditScrapeInput(BaseModel):
|
|||
searches: list[str] = Field(default_factory=list)
|
||||
searchCommunityName: str | None = None
|
||||
|
||||
@field_validator("searchCommunityName", mode="after")
|
||||
@classmethod
|
||||
def _normalize_community(cls, v: str | None) -> str | None:
|
||||
"""Normalize the bare subreddit name at the trust boundary.
|
||||
|
||||
Both the in-subreddit search path and the community-only listing path
|
||||
interpolate this straight into ``r/{name}/...``, so a pasted ``r/python``
|
||||
or ``/r/python/`` would otherwise become ``r/r/python`` and 404. Strip a
|
||||
leading ``r/`` and surrounding slashes/whitespace; empty coerces to None.
|
||||
"""
|
||||
if v is None:
|
||||
return None
|
||||
name = v.strip().strip("/")
|
||||
if name[:2].lower() == "r/":
|
||||
name = name[2:].strip("/")
|
||||
return name or None
|
||||
|
||||
# Sort / filter
|
||||
sort: RedditSort = "new"
|
||||
time: RedditTime | None = None
|
||||
|
|
|
|||
|
|
@ -407,6 +407,18 @@ async def iter_reddit(
|
|||
yield item
|
||||
return
|
||||
|
||||
# Community-only: a bare subreddit name with no urls/searches. The capability
|
||||
# schema promises "with no search_queries, its listing is scraped"; without
|
||||
# this the searches loop below builds zero jobs and yields nothing. Reuse the
|
||||
# same flow the URL path dispatches for /r/<sub>. (searches present => the
|
||||
# name stays a search scope, handled below.)
|
||||
if not input_model.searches and input_model.searchCommunityName:
|
||||
async for item in _subreddit_flow(
|
||||
input_model.searchCommunityName, input_model=input_model
|
||||
):
|
||||
yield item
|
||||
return
|
||||
|
||||
# Fair-share the item budget across queries: with a shared cap, the
|
||||
# first-finishing (often broadest/noisiest) search would fill the whole
|
||||
# collector limit and starve the precise queries.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,52 @@
|
|||
# Walmart Scraper
|
||||
|
||||
Two verbs read Walmart's public, anonymous pages: `walmart.scrape` (products +
|
||||
search/category/browse listings, per-product billing) and `walmart.reviews`
|
||||
(deep paginated reviews, per-review billing).
|
||||
|
||||
## Scope
|
||||
|
||||
The scraper reads public pages available to anonymous visitors — no login, no
|
||||
account cookies. Data is extracted from the Next.js `__NEXT_DATA__` JSON blob
|
||||
embedded in each page (with an `__APP_DATA__` fallback), not from the rendered
|
||||
DOM, because Walmart obfuscates CSS classes and A/B-tests layout constantly.
|
||||
|
||||
`walmart.scrape` returns a free sample of on-page reviews (rating distribution,
|
||||
aspects, top reviews) under `reviewsSample`. `walmart.reviews` fetches the full
|
||||
review history from the public `/reviews/product/{usItemId}` page, which
|
||||
robots.txt permits (unlike `/search`).
|
||||
|
||||
## Architecture
|
||||
|
||||
- `schemas.py` defines the stable input, product, review, and error models.
|
||||
- `url_resolver.py` classifies product (`/ip/`) vs listing (`/search`, `/cp/`,
|
||||
`/browse/`) URLs and extracts the numeric `usItemId`.
|
||||
- `next_data.py` extracts and navigates the hidden Next.js JSON state.
|
||||
- `fetch.py` owns proxy-aware HTTP access (US-pinned), block detection, and
|
||||
retries.
|
||||
- `parsers.py` contains pure, defensive JSON parsers.
|
||||
- `scraper.py` coordinates discovery, enrichment, pagination, concurrency,
|
||||
limits, and in-stream error items.
|
||||
|
||||
## Anti-bot
|
||||
|
||||
Walmart runs Akamai (edge/TLS) + PerimeterX/HUMAN (behavioral JS). Requests go
|
||||
through US residential proxies with TLS-impersonated headers; blocked responses
|
||||
(body markers, `412`/`429`/`503`, or the `200`-OK CAPTCHA body) rotate to a
|
||||
fresh proxy exit.
|
||||
|
||||
Known ceilings and upgrade paths (see `fetch.py` / `scraper.py` `ponytail:`
|
||||
notes): reviews page at 10/page; search capped at Walmart's 25-page limit;
|
||||
session warming (seed `_px3`/`_pxhd` on a sticky exit) is the next lever if
|
||||
block rates on the SSR pages climb, and the `/orchestra/*` GraphQL API is
|
||||
deliberately avoided (rotating persisted-query hashes make it brittle).
|
||||
|
||||
## Verification
|
||||
|
||||
Offline fixtures cover the parsers and both flows:
|
||||
|
||||
uv run pytest tests/unit/platforms/walmart/
|
||||
|
||||
A manual live check (requires network + residential proxy):
|
||||
|
||||
uv run python scripts/e2e_walmart_scraper.py
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
"""Platform-native Walmart scraper (products, listings, reviews)."""
|
||||
|
||||
from .schemas import (
|
||||
ErrorItem,
|
||||
ProductItem,
|
||||
ReviewItem,
|
||||
WalmartReviewsInput,
|
||||
WalmartScrapeInput,
|
||||
)
|
||||
from .scraper import (
|
||||
iter_products,
|
||||
iter_reviews,
|
||||
scrape_products,
|
||||
scrape_reviews,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"ErrorItem",
|
||||
"ProductItem",
|
||||
"ReviewItem",
|
||||
"WalmartReviewsInput",
|
||||
"WalmartScrapeInput",
|
||||
"iter_products",
|
||||
"iter_reviews",
|
||||
"scrape_products",
|
||||
"scrape_reviews",
|
||||
]
|
||||
171
surfsense_backend/app/proprietary/platforms/walmart/fetch.py
Normal file
171
surfsense_backend/app/proprietary/platforms/walmart/fetch.py
Normal file
|
|
@ -0,0 +1,171 @@
|
|||
"""Network access for public Walmart pages.
|
||||
|
||||
Mirrors the Amazon fetch layer: every request goes through the configured
|
||||
residential proxy (pinned to the US, since Walmart geo-locks inventory), and any
|
||||
response that looks like an anti-bot interstitial is retried on a fresh proxy
|
||||
exit. Ordinary HTTP failures are returned to the caller for domain-specific
|
||||
handling.
|
||||
|
||||
Walmart runs Akamai (edge/TLS) + PerimeterX/HUMAN (behavioral JS). Two Walmart
|
||||
specifics differ from Amazon:
|
||||
|
||||
* Walmart serves CAPTCHA with an HTTP ``200`` body ("Robot or human?"), so block
|
||||
detection scans the body, never the status alone.
|
||||
* ``412`` is PerimeterX's rejection code and is treated as blocked → rotate.
|
||||
|
||||
``ponytail:`` MVP hits only the server-rendered ``__NEXT_DATA__`` pages, which
|
||||
TLS impersonation + residential proxies clear without seeding PerimeterX cookies.
|
||||
If block rates on those pages climb, the upgrade path is a warmed sticky session
|
||||
(seed ``_px3``/``_pxhd``/``ACID`` from a homepage fetch, reuse exit + cookies) —
|
||||
the same shape as Amazon's ``get_location_session``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from collections.abc import Awaitable, Callable
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from scrapling.fetchers import AsyncFetcher
|
||||
|
||||
from app.utils.proxy import get_geo_proxy_url, get_sticky_proxy_url
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_MAX_IP_ATTEMPTS = 6
|
||||
_REQUEST_TIMEOUT_S = 30
|
||||
_HEADERS = {
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
}
|
||||
_BLOCK_MARKERS = (
|
||||
"robot or human",
|
||||
"px-captcha",
|
||||
"/blocked",
|
||||
"verify you are a human",
|
||||
"access to this page has been denied",
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FetchResult:
|
||||
"""The response details needed by scraper flows."""
|
||||
|
||||
status: int
|
||||
html: str
|
||||
url: str
|
||||
cookies: dict[str, str]
|
||||
headers: dict[str, str] = field(default_factory=dict)
|
||||
|
||||
|
||||
async def gather_bounded[T](
|
||||
factories: list[Callable[[], Awaitable[T]]], *, concurrency: int
|
||||
) -> list[T]:
|
||||
"""Run async factories concurrently while preserving input order."""
|
||||
if not factories:
|
||||
return []
|
||||
semaphore = asyncio.Semaphore(max(1, concurrency))
|
||||
|
||||
async def run(factory: Callable[[], Awaitable[T]]) -> T:
|
||||
async with semaphore:
|
||||
return await factory()
|
||||
|
||||
return await asyncio.gather(*(run(factory) for factory in factories))
|
||||
|
||||
|
||||
def is_blocked(
|
||||
html: str | None, status: int, headers: dict[str, str] | None = None
|
||||
) -> bool:
|
||||
"""Return whether a response is a Walmart anti-bot interstitial.
|
||||
|
||||
``412`` is PerimeterX's rejection; ``429``/``503`` are throttles. Walmart also
|
||||
serves CAPTCHA with a ``200`` body, so the body is scanned regardless of
|
||||
status.
|
||||
"""
|
||||
if status in {412, 429, 503}:
|
||||
return True
|
||||
text = (html or "")[:200_000].lower()
|
||||
return any(marker in text for marker in _BLOCK_MARKERS)
|
||||
|
||||
|
||||
def _response_url(page: Any, fallback: str) -> str:
|
||||
value = getattr(page, "url", None)
|
||||
return str(value) if value else fallback
|
||||
|
||||
|
||||
def _response_cookies(page: Any) -> dict[str, str]:
|
||||
cookies = getattr(page, "cookies", None)
|
||||
return dict(cookies) if isinstance(cookies, dict) else {}
|
||||
|
||||
|
||||
def _response_headers(page: Any) -> dict[str, str]:
|
||||
headers = getattr(page, "headers", None)
|
||||
return dict(headers) if isinstance(headers, dict) else {}
|
||||
|
||||
|
||||
def _selected_proxy(
|
||||
proxy: str | None, country: str, attempt: int, url: str
|
||||
) -> str | None:
|
||||
if proxy is not None:
|
||||
return proxy
|
||||
if attempt > 1:
|
||||
session_id = f"walmart-{attempt}-{abs(hash((url, time.time_ns()))):x}"
|
||||
return get_sticky_proxy_url(session_id, country)
|
||||
return get_geo_proxy_url(country)
|
||||
|
||||
|
||||
async def fetch_page(
|
||||
url: str,
|
||||
*,
|
||||
cookies: dict[str, str] | None = None,
|
||||
proxy: str | None = None,
|
||||
country: str = "us",
|
||||
rotate_on_block: bool = True,
|
||||
) -> FetchResult | None:
|
||||
"""Fetch a page and retry blocked responses with fresh proxy exits."""
|
||||
attempts = _MAX_IP_ATTEMPTS if rotate_on_block else 1
|
||||
for attempt in range(1, attempts + 1):
|
||||
selected_proxy = _selected_proxy(proxy, country, attempt, url)
|
||||
started = time.perf_counter()
|
||||
try:
|
||||
page = await AsyncFetcher.get(
|
||||
url,
|
||||
headers={**_HEADERS},
|
||||
cookies=cookies or {},
|
||||
proxy=selected_proxy,
|
||||
stealthy_headers=True,
|
||||
timeout=_REQUEST_TIMEOUT_S,
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning("Walmart request failed for %s: %s", url, exc)
|
||||
if proxy is not None:
|
||||
return None
|
||||
continue
|
||||
|
||||
status = int(getattr(page, "status", 0) or 0)
|
||||
html = getattr(page, "html_content", None) or ""
|
||||
response_headers = _response_headers(page)
|
||||
logger.info(
|
||||
"[walmart][perf] status=%s attempt=%s fetch_ms=%.1f url=%s",
|
||||
status,
|
||||
attempt,
|
||||
(time.perf_counter() - started) * 1000,
|
||||
url,
|
||||
)
|
||||
if rotate_on_block and is_blocked(html, status, response_headers):
|
||||
logger.info(
|
||||
"Walmart blocked proxy attempt %s/%s for %s", attempt, attempts, url
|
||||
)
|
||||
continue
|
||||
return FetchResult(
|
||||
status=status,
|
||||
html=html,
|
||||
url=_response_url(page, url),
|
||||
cookies=_response_cookies(page),
|
||||
headers=response_headers,
|
||||
)
|
||||
logger.warning("Walmart exhausted %s proxy attempts for %s", attempts, url)
|
||||
return None
|
||||
|
|
@ -0,0 +1,69 @@
|
|||
"""Extract Walmart's hidden Next.js JSON state from page HTML.
|
||||
|
||||
Every serious Walmart scraper (Scrapfly, Oxylabs, ScrapeOps, Apify) reads the
|
||||
``<script id="__NEXT_DATA__">`` JSON blob rather than parsing the rendered DOM,
|
||||
because Walmart obfuscates CSS classes and A/B-tests layout constantly. The JSON
|
||||
shape is far more stable across redesigns.
|
||||
|
||||
``__APP_DATA__`` is the documented fallback anchor Walmart occasionally ships
|
||||
instead during layout experiments.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_NEXT_DATA_RE = re.compile(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', re.DOTALL)
|
||||
_APP_DATA_RE = re.compile(r'<script id="__APP_DATA__"[^>]*>(.*?)</script>', re.DOTALL)
|
||||
|
||||
|
||||
def extract_next_data(html: str | None) -> dict[str, Any] | None:
|
||||
"""Return the parsed Next.js state object, or ``None`` when absent/invalid.
|
||||
|
||||
Tries ``__NEXT_DATA__`` first, then the ``__APP_DATA__`` fallback so a single
|
||||
Walmart layout experiment does not blank the whole extractor.
|
||||
"""
|
||||
if not html:
|
||||
return None
|
||||
for pattern in (_NEXT_DATA_RE, _APP_DATA_RE):
|
||||
match = pattern.search(html)
|
||||
if not match:
|
||||
continue
|
||||
try:
|
||||
data = json.loads(match.group(1))
|
||||
except (ValueError, TypeError):
|
||||
logger.warning("Walmart hidden JSON present but did not parse")
|
||||
continue
|
||||
if isinstance(data, dict):
|
||||
return data
|
||||
return None
|
||||
|
||||
|
||||
def dig(obj: Any, *keys: str | int) -> Any:
|
||||
"""Walk nested dict/list keys, returning ``None`` on any miss.
|
||||
|
||||
Tolerates the layout drift between Walmart's ``initialData`` variants without
|
||||
a cascade of ``if key in ...`` guards at every call site.
|
||||
"""
|
||||
current = obj
|
||||
for key in keys:
|
||||
if isinstance(key, int):
|
||||
if not isinstance(current, list) or not -len(current) <= key < len(current):
|
||||
return None
|
||||
current = current[key]
|
||||
else:
|
||||
if not isinstance(current, dict) or key not in current:
|
||||
return None
|
||||
current = current[key]
|
||||
return current
|
||||
|
||||
|
||||
def initial_data(next_data: dict[str, Any]) -> dict[str, Any] | None:
|
||||
"""The ``props.pageProps.initialData`` node shared by every page type."""
|
||||
node = dig(next_data, "props", "pageProps", "initialData")
|
||||
return node if isinstance(node, dict) else None
|
||||
248
surfsense_backend/app/proprietary/platforms/walmart/parsers.py
Normal file
248
surfsense_backend/app/proprietary/platforms/walmart/parsers.py
Normal file
|
|
@ -0,0 +1,248 @@
|
|||
"""Pure parsers for Walmart's hidden ``__NEXT_DATA__`` JSON.
|
||||
|
||||
Product, listing, and review data all live in the same Next.js state tree; these
|
||||
functions navigate it defensively (via :func:`.next_data.dig`) and normalize the
|
||||
fields into the stable output shape. Missing sections yield ``None``/``[]`` so an
|
||||
isolated schema change never discards an otherwise usable record.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from urllib.parse import urljoin
|
||||
|
||||
from .next_data import dig, initial_data
|
||||
|
||||
_WALMART_ORIGIN = "https://www.walmart.com"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Shared helpers. #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def _price(node: Any) -> dict[str, Any] | None:
|
||||
"""Normalize a Walmart price node into ``{value, currency}``."""
|
||||
if not isinstance(node, dict):
|
||||
return None
|
||||
value = node.get("price")
|
||||
currency = node.get("currencyUnit")
|
||||
if value is None and currency is None:
|
||||
return None
|
||||
return {"value": value, "currency": currency}
|
||||
|
||||
|
||||
def _seller(raw: dict[str, Any]) -> dict[str, Any] | None:
|
||||
name = raw.get("sellerName") or raw.get("sellerDisplayName")
|
||||
seller_id = raw.get("sellerId")
|
||||
if not name and not seller_id:
|
||||
return None
|
||||
is_walmart = bool(name) and "walmart" in name.lower()
|
||||
return {
|
||||
"id": seller_id,
|
||||
"name": name,
|
||||
"type": "WALMART" if is_walmart else "MARKETPLACE",
|
||||
}
|
||||
|
||||
|
||||
def _absolute(url: str | None) -> str | None:
|
||||
return urljoin(_WALMART_ORIGIN, url) if url else None
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Listing / search cards. #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def _listing_card(raw: dict[str, Any]) -> dict[str, Any] | None:
|
||||
"""Normalize one search/category result item into a product card."""
|
||||
item_id = raw.get("usItemId") or raw.get("id")
|
||||
name = raw.get("name")
|
||||
if not item_id or not name:
|
||||
return None
|
||||
price_info = raw.get("priceInfo") or {}
|
||||
availability = raw.get("availabilityStatusV2") or {}
|
||||
image = raw.get("imageInfo") or {}
|
||||
return {
|
||||
"usItemId": str(item_id),
|
||||
"name": name,
|
||||
"brand": raw.get("brand"),
|
||||
"url": _absolute(raw.get("canonicalUrl")),
|
||||
"price": _price(price_info.get("currentPrice"))
|
||||
or ({"value": raw.get("price")} if raw.get("price") is not None else None),
|
||||
"listPrice": _price(price_info.get("wasPrice")),
|
||||
"stars": raw.get("averageRating"),
|
||||
"reviewsCount": raw.get("numberOfReviews"),
|
||||
"seller": _seller(raw),
|
||||
"availabilityStatus": availability.get("value")
|
||||
or ("OUT_OF_STOCK" if raw.get("isOutOfStock") else None),
|
||||
"inStock": (availability.get("value") == "IN_STOCK")
|
||||
if availability.get("value")
|
||||
else (raw.get("isOutOfStock") is False if "isOutOfStock" in raw else None),
|
||||
"thumbnailImage": image.get("thumbnailUrl") or raw.get("image"),
|
||||
"sponsored": raw.get("isSponsoredFlag"),
|
||||
}
|
||||
|
||||
|
||||
def parse_listing_page(next_data: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
"""Extract normalized product cards from a search/category/browse page."""
|
||||
data = initial_data(next_data)
|
||||
if data is None:
|
||||
return []
|
||||
stacks = dig(data, "searchResult", "itemStacks")
|
||||
if not isinstance(stacks, list):
|
||||
return []
|
||||
cards: list[dict[str, Any]] = []
|
||||
seen: set[str] = set()
|
||||
for stack in stacks:
|
||||
for raw in (stack or {}).get("items") or []:
|
||||
if not isinstance(raw, dict) or raw.get("__typename") not in (
|
||||
None,
|
||||
"Product",
|
||||
):
|
||||
continue
|
||||
card = _listing_card(raw)
|
||||
if card and card["usItemId"] not in seen:
|
||||
seen.add(card["usItemId"])
|
||||
cards.append(card)
|
||||
return cards
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Product detail. #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def _images(image_info: dict[str, Any]) -> list[str]:
|
||||
images: list[str] = []
|
||||
for entry in image_info.get("allImages") or []:
|
||||
url = entry.get("url") if isinstance(entry, dict) else None
|
||||
if url and url not in images:
|
||||
images.append(url)
|
||||
return images
|
||||
|
||||
|
||||
def _breadcrumbs(product: dict[str, Any]) -> list[str]:
|
||||
"""Category breadcrumb names, root→leaf.
|
||||
|
||||
Walmart ships ``category.path`` as a list of ``{name, url}`` nodes, e.g.
|
||||
``[{"name": "Home Improvement"}, ..., {"name": "Air Conditioners"}]`` — not
|
||||
a string.
|
||||
"""
|
||||
path = dig(product, "category", "path")
|
||||
if not isinstance(path, list):
|
||||
return []
|
||||
return [
|
||||
node["name"] for node in path if isinstance(node, dict) and node.get("name")
|
||||
]
|
||||
|
||||
|
||||
def _reviews_sample(
|
||||
reviews: dict[str, Any] | None, limit: int = 10
|
||||
) -> dict[str, Any] | None:
|
||||
if not isinstance(reviews, dict):
|
||||
return None
|
||||
customer = reviews.get("customerReviews") or []
|
||||
return {
|
||||
"averageOverallRating": reviews.get("averageOverallRating"),
|
||||
"totalReviewCount": reviews.get("totalReviewCount"),
|
||||
"aspects": reviews.get("aspects") or [],
|
||||
"topReviews": [
|
||||
normalize_review(r) for r in customer[:limit] if isinstance(r, dict)
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def parse_product(
|
||||
next_data: dict[str, Any], *, url: str, include_reviews_sample: bool = True
|
||||
) -> dict[str, Any]:
|
||||
"""Extract normalized product fields from a product detail page."""
|
||||
data = initial_data(next_data)
|
||||
product = dig(data, "data", "product") if data else None
|
||||
if not isinstance(product, dict):
|
||||
return {}
|
||||
idml = dig(data, "data", "idml") if data else None
|
||||
reviews = dig(data, "data", "reviews") if data else None
|
||||
price_info = product.get("priceInfo") or {}
|
||||
image_info = product.get("imageInfo") or {}
|
||||
availability = product.get("availabilityStatus")
|
||||
breadcrumbs = _breadcrumbs(product)
|
||||
|
||||
fields: dict[str, Any] = {
|
||||
"usItemId": str(product.get("usItemId") or product.get("id") or "") or None,
|
||||
"name": product.get("name"),
|
||||
"brand": product.get("brand"),
|
||||
"url": url,
|
||||
"price": _price(price_info.get("currentPrice")),
|
||||
"listPrice": _price(price_info.get("wasPrice")),
|
||||
"currency": dig(price_info, "currentPrice", "currencyUnit"),
|
||||
"availabilityStatus": availability,
|
||||
"inStock": availability == "IN_STOCK" if availability else None,
|
||||
"stars": product.get("averageRating"),
|
||||
"reviewsCount": product.get("numberOfReviews"),
|
||||
"seller": _seller(product),
|
||||
"manufacturerName": product.get("manufacturerName"),
|
||||
"shortDescription": product.get("shortDescription"),
|
||||
"longDescription": (idml or {}).get("longDescription")
|
||||
if isinstance(idml, dict)
|
||||
else None,
|
||||
"thumbnailImage": image_info.get("thumbnailUrl"),
|
||||
"images": _images(image_info),
|
||||
"category": breadcrumbs[-1] if breadcrumbs else None,
|
||||
"breadCrumbs": breadcrumbs,
|
||||
"variants": product.get("variantCriteria") or [],
|
||||
}
|
||||
if include_reviews_sample:
|
||||
fields["reviewsSample"] = _reviews_sample(reviews)
|
||||
return {key: value for key, value in fields.items() if value is not None}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Reviews (deep pagination). #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def normalize_review(raw: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Normalize one Walmart ``customerReviews`` record into a ``ReviewItem`` dict."""
|
||||
badges = raw.get("badges") or []
|
||||
verified = any(
|
||||
isinstance(b, dict) and b.get("id") == "VerifiedPurchaser" for b in badges
|
||||
)
|
||||
photos = raw.get("photos") or raw.get("media") or []
|
||||
images: list[str] = []
|
||||
for photo in photos:
|
||||
if not isinstance(photo, dict):
|
||||
continue
|
||||
url = photo.get("normalUrl") or dig(photo, "sizes", "normal", "url")
|
||||
if url and url not in images:
|
||||
images.append(url)
|
||||
responses = raw.get("clientResponses") or []
|
||||
seller_response = None
|
||||
if responses and isinstance(responses[0], dict):
|
||||
seller_response = responses[0].get("response") or responses[0].get(
|
||||
"responseText"
|
||||
)
|
||||
return {
|
||||
"reviewId": raw.get("reviewId"),
|
||||
"rating": raw.get("rating"),
|
||||
"title": raw.get("reviewTitle"),
|
||||
"text": raw.get("reviewText"),
|
||||
"submissionTime": raw.get("reviewSubmissionTime"),
|
||||
"author": raw.get("userNickname"),
|
||||
"verifiedPurchase": verified,
|
||||
"positiveFeedback": raw.get("positiveFeedback"),
|
||||
"negativeFeedback": raw.get("negativeFeedback"),
|
||||
"images": images,
|
||||
"syndicated": bool(raw.get("syndicationSource")),
|
||||
"sellerResponse": seller_response,
|
||||
}
|
||||
|
||||
|
||||
def parse_reviews_page(next_data: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
"""Extract normalized reviews from one ``/reviews/product/{id}`` page."""
|
||||
data = initial_data(next_data)
|
||||
reviews = dig(data, "data", "reviews") if data else None
|
||||
customer = reviews.get("customerReviews") if isinstance(reviews, dict) else None
|
||||
if not isinstance(customer, list):
|
||||
return []
|
||||
return [normalize_review(r) for r in customer if isinstance(r, dict)]
|
||||
188
surfsense_backend/app/proprietary/platforms/walmart/schemas.py
Normal file
188
surfsense_backend/app/proprietary/platforms/walmart/schemas.py
Normal file
|
|
@ -0,0 +1,188 @@
|
|||
# ruff: noqa: N815
|
||||
"""Input/output models for the public Walmart scraper.
|
||||
|
||||
Two verbs share this module: ``walmart.scrape`` (products + listings) and
|
||||
``walmart.reviews`` (deep paginated reviews). Walmart is a Next.js app that
|
||||
ships its data as JSON in a ``<script id="__NEXT_DATA__">`` tag, so the parsers
|
||||
read structured JSON rather than the rendered DOM — more robust against the
|
||||
constant CSS/layout A/B testing on walmart.com.
|
||||
|
||||
Outputs use ``extra="allow"`` on purpose (same as the Amazon scraper): the wire
|
||||
shape can grow without breaking existing consumers. Only the small, stable,
|
||||
always-populated nested objects are typed; volatile sections stay loose.
|
||||
|
||||
Public, anonymous data only: there is no auth/login field. Deep review
|
||||
pagination uses the public ``/reviews/product/{id}`` page, which robots.txt
|
||||
permits (unlike ``/search``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Typed nested objects (the stable, always-populated ones). #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
class Price(BaseModel):
|
||||
"""A monetary amount as Walmart renders it (``price``, ``listPrice``)."""
|
||||
|
||||
model_config = ConfigDict(extra="allow")
|
||||
|
||||
value: float | None = None
|
||||
currency: str | None = None
|
||||
|
||||
|
||||
class Seller(BaseModel):
|
||||
"""The offer seller. ``type`` distinguishes Walmart 1P from 3P marketplace."""
|
||||
|
||||
model_config = ConfigDict(extra="allow")
|
||||
|
||||
id: str | None = None
|
||||
name: str | None = None
|
||||
type: Literal["WALMART", "MARKETPLACE"] | None = None
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Input surfaces. #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
class WalmartScrapeInput(BaseModel):
|
||||
"""Full input surface for ``walmart.scrape``.
|
||||
|
||||
``startUrls`` mixes product (``/ip/``), search (``/search``), category
|
||||
(``/cp/``), and browse (``/browse/``) URLs; each is classified and dispatched
|
||||
by :mod:`.url_resolver`. ``extra="allow"`` keeps a verbatim payload valid.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(extra="allow")
|
||||
|
||||
startUrls: list[str] = Field(min_length=1)
|
||||
maxItemsPerStartUrl: int | None = Field(default=None, ge=0)
|
||||
maxSearchPagesPerStartUrl: int = Field(default=25, ge=1)
|
||||
includeDetails: bool = True
|
||||
includeReviewsSample: bool = True
|
||||
country: str = "us"
|
||||
|
||||
|
||||
class WalmartReviewsInput(BaseModel):
|
||||
"""Full input surface for ``walmart.reviews`` (deep paginated reviews)."""
|
||||
|
||||
model_config = ConfigDict(extra="allow")
|
||||
|
||||
itemIds: list[str] = Field(min_length=1)
|
||||
maxReviews: int = Field(default=200, ge=1)
|
||||
sort: Literal["most-recent", "most-helpful", "rating-high", "rating-low"] = (
|
||||
"most-recent"
|
||||
)
|
||||
country: str = "us"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Output items. #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
class ProductItem(BaseModel):
|
||||
"""``walmart.scrape`` output item (one per product / listing card).
|
||||
|
||||
Unsourced fields default to ``None``/``[]``; ``extra="allow"`` keeps the
|
||||
contract open so later milestones add fields without breaking consumers.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(extra="allow", populate_by_name=True)
|
||||
|
||||
# Identity / core
|
||||
usItemId: str | None = None
|
||||
name: str | None = None
|
||||
brand: str | None = None
|
||||
url: str | None = None
|
||||
price: Price | None = None
|
||||
listPrice: Price | None = None
|
||||
currency: str | None = None
|
||||
availabilityStatus: str | None = None
|
||||
inStock: bool | None = None
|
||||
|
||||
# Ratings / social
|
||||
stars: float | None = None
|
||||
reviewsCount: int | None = None
|
||||
|
||||
# Commerce
|
||||
seller: Seller | None = None
|
||||
manufacturerName: str | None = None
|
||||
sponsored: bool | None = None
|
||||
|
||||
# Content (detail pages only)
|
||||
shortDescription: str | None = None
|
||||
longDescription: str | None = None
|
||||
thumbnailImage: str | None = None
|
||||
images: list[str] = Field(default_factory=list)
|
||||
breadCrumbs: list[str] = Field(default_factory=list)
|
||||
category: str | None = None
|
||||
specifications: dict | None = None
|
||||
variants: list[dict] = Field(default_factory=list)
|
||||
fulfillment: dict | None = None
|
||||
|
||||
# Embedded free review sample (detail pages only)
|
||||
reviewsSample: dict | None = None
|
||||
|
||||
# Provenance / meta
|
||||
input: str | None = None
|
||||
|
||||
def to_output(self) -> dict[str, Any]:
|
||||
"""Serialize to the flat dict output shape (keeps extras)."""
|
||||
return self.model_dump(by_alias=True, exclude_none=False)
|
||||
|
||||
|
||||
class ReviewItem(BaseModel):
|
||||
"""``walmart.reviews`` output item (one per customer review)."""
|
||||
|
||||
model_config = ConfigDict(extra="allow")
|
||||
|
||||
reviewId: str | None = None
|
||||
rating: float | None = None
|
||||
title: str | None = None
|
||||
text: str | None = None
|
||||
submissionTime: str | None = None
|
||||
author: str | None = None
|
||||
verifiedPurchase: bool | None = None
|
||||
positiveFeedback: int | None = None
|
||||
negativeFeedback: int | None = None
|
||||
images: list[str] = Field(default_factory=list)
|
||||
syndicated: bool | None = None
|
||||
sellerResponse: str | None = None
|
||||
usItemId: str | None = None
|
||||
input: str | None = None
|
||||
|
||||
def to_output(self) -> dict[str, Any]:
|
||||
return self.model_dump(exclude_none=False)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Error item (the failure model — pushed into the stream, not raised). #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
ErrorCode = Literal[
|
||||
"invalid_url",
|
||||
"product_not_found",
|
||||
"no_results_found",
|
||||
"reviews_not_found",
|
||||
]
|
||||
|
||||
|
||||
class ErrorItem(BaseModel):
|
||||
"""A per-input failure emitted into the dataset instead of a normal item.
|
||||
|
||||
Consumers tell error items apart from products/reviews by the ``error`` key.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(extra="allow")
|
||||
|
||||
error: ErrorCode
|
||||
errorDescription: str | None = None
|
||||
input: str | None = None
|
||||
url: str | None = None
|
||||
288
surfsense_backend/app/proprietary/platforms/walmart/scraper.py
Normal file
288
surfsense_backend/app/proprietary/platforms/walmart/scraper.py
Normal file
|
|
@ -0,0 +1,288 @@
|
|||
"""Orchestrate public Walmart product discovery, enrichment, and reviews.
|
||||
|
||||
Two streaming cores: :func:`iter_products` dispatches each start URL to a product
|
||||
or listing flow, and :func:`iter_reviews` paginates the public reviews page per
|
||||
item id. Network and parsing concerns stay isolated in their own modules so
|
||||
markup changes and retry policy can be tested independently.
|
||||
|
||||
The failure model is in-stream error items (dicts with an ``error`` key), never
|
||||
exceptions — identical to the Amazon scraper.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import AsyncIterator
|
||||
from typing import Any
|
||||
from urllib.parse import parse_qsl, urlencode, urlparse, urlunparse
|
||||
|
||||
from .fetch import fetch_page, gather_bounded
|
||||
from .next_data import extract_next_data
|
||||
from .parsers import parse_listing_page, parse_product, parse_reviews_page
|
||||
from .schemas import (
|
||||
ErrorItem,
|
||||
ProductItem,
|
||||
ReviewItem,
|
||||
WalmartReviewsInput,
|
||||
WalmartScrapeInput,
|
||||
)
|
||||
from .url_resolver import ResolvedUrl, extract_item_id, resolve_url
|
||||
|
||||
__all__ = ["iter_products", "iter_reviews", "scrape_products", "scrape_reviews"]
|
||||
|
||||
_DETAIL_CONCURRENCY = 6
|
||||
_SEARCH_PAGE_LIMIT = 25 # Walmart caps result pagination at 25 pages.
|
||||
_REVIEWS_PAGE_LIMIT = 500 # 10 reviews/page → 5000-review safety ceiling.
|
||||
_DEFAULT_ITEMS_PER_START_URL = 40
|
||||
|
||||
# Friendly sort → Walmart's ``sort`` query value on the reviews page.
|
||||
_REVIEW_SORT = {
|
||||
"most-recent": "submission-desc",
|
||||
"most-helpful": "helpful",
|
||||
"rating-high": "rating-desc",
|
||||
"rating-low": "rating-asc",
|
||||
}
|
||||
|
||||
|
||||
def _error(
|
||||
code: str, description: str, *, input_url: str | None, url: str | None = None
|
||||
) -> dict[str, Any]:
|
||||
return ErrorItem(
|
||||
error=code,
|
||||
errorDescription=description,
|
||||
input=input_url,
|
||||
url=url or input_url,
|
||||
).model_dump()
|
||||
|
||||
|
||||
def _page_url(url: str, page: int) -> str:
|
||||
parsed = urlparse(url)
|
||||
query = dict(parse_qsl(parsed.query, keep_blank_values=True))
|
||||
query["page"] = str(page)
|
||||
return urlunparse(parsed._replace(query=urlencode(query)))
|
||||
|
||||
|
||||
def _product_url(item_id: str) -> str:
|
||||
return f"https://www.walmart.com/ip/{item_id}"
|
||||
|
||||
|
||||
def _reviews_url(item_id: str, page: int, sort: str) -> str:
|
||||
query = urlencode(
|
||||
{"page": page, "sort": sort, "entryPoint": "viewAllReviewsBottom"}
|
||||
)
|
||||
return f"https://www.walmart.com/reviews/product/{item_id}?{query}"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Product / listing flows. #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
async def _product_flow(
|
||||
resolved: ResolvedUrl, input_model: WalmartScrapeInput
|
||||
) -> AsyncIterator[dict[str, Any]]:
|
||||
"""Fetch and parse one product detail page."""
|
||||
url = _product_url(resolved.item_id) if resolved.item_id else resolved.url
|
||||
response = await fetch_page(url, country=input_model.country)
|
||||
if response is None:
|
||||
yield _error(
|
||||
"product_not_found",
|
||||
"The product page could not be loaded after retrying proxy exits.",
|
||||
input_url=resolved.url,
|
||||
)
|
||||
return
|
||||
if response.status in {404, 410}:
|
||||
yield _error(
|
||||
"product_not_found",
|
||||
"The product page was not found.",
|
||||
input_url=resolved.url,
|
||||
)
|
||||
return
|
||||
|
||||
next_data = extract_next_data(response.html)
|
||||
fields = (
|
||||
parse_product(
|
||||
next_data,
|
||||
url=response.url,
|
||||
include_reviews_sample=input_model.includeReviewsSample,
|
||||
)
|
||||
if next_data
|
||||
else {}
|
||||
)
|
||||
if not fields.get("name"):
|
||||
yield _error(
|
||||
"product_not_found",
|
||||
"The response did not contain a recognizable product.",
|
||||
input_url=resolved.url,
|
||||
url=response.url,
|
||||
)
|
||||
return
|
||||
fields["input"] = resolved.url
|
||||
yield ProductItem(**fields).to_output()
|
||||
|
||||
|
||||
async def _listing_flow(
|
||||
resolved: ResolvedUrl, input_model: WalmartScrapeInput
|
||||
) -> AsyncIterator[dict[str, Any]]:
|
||||
"""Page through a search/category/browse listing and optionally enrich."""
|
||||
cap = (
|
||||
input_model.maxItemsPerStartUrl
|
||||
if input_model.maxItemsPerStartUrl is not None
|
||||
else _DEFAULT_ITEMS_PER_START_URL
|
||||
)
|
||||
max_pages = min(input_model.maxSearchPagesPerStartUrl, _SEARCH_PAGE_LIMIT)
|
||||
seen: set[str] = set()
|
||||
emitted = 0
|
||||
for page in range(1, max_pages + 1):
|
||||
response = await fetch_page(
|
||||
_page_url(resolved.url, page), country=input_model.country
|
||||
)
|
||||
next_data = extract_next_data(response.html) if response else None
|
||||
cards = parse_listing_page(next_data) if next_data else []
|
||||
cards = [c for c in cards if c["usItemId"] not in seen]
|
||||
for card in cards:
|
||||
seen.add(card["usItemId"])
|
||||
if not cards:
|
||||
if page == 1:
|
||||
yield _error(
|
||||
"no_results_found",
|
||||
"The listing page did not contain any products.",
|
||||
input_url=resolved.url,
|
||||
)
|
||||
return
|
||||
cards = cards[: max(0, cap - emitted)]
|
||||
|
||||
if not input_model.includeDetails:
|
||||
for card in cards:
|
||||
card["input"] = resolved.url
|
||||
yield ProductItem(**card).to_output()
|
||||
emitted += 1
|
||||
else:
|
||||
|
||||
async def load_card(card: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
product = resolve_url(card["url"]) if card.get("url") else None
|
||||
if product is None or product.item_id is None:
|
||||
card["input"] = resolved.url
|
||||
return [ProductItem(**card).to_output()]
|
||||
return [item async for item in _product_flow(product, input_model)]
|
||||
|
||||
for batch in await gather_bounded(
|
||||
[lambda card=card: load_card(card) for card in cards],
|
||||
concurrency=_DETAIL_CONCURRENCY,
|
||||
):
|
||||
for item in batch:
|
||||
if "error" not in item and emitted >= cap:
|
||||
return
|
||||
yield item
|
||||
if "error" not in item:
|
||||
emitted += 1
|
||||
if emitted >= cap:
|
||||
return
|
||||
|
||||
|
||||
_FLOWS = {"product": _product_flow, "listing": _listing_flow}
|
||||
|
||||
|
||||
async def iter_products(
|
||||
input_model: WalmartScrapeInput,
|
||||
) -> AsyncIterator[dict[str, Any]]:
|
||||
"""Yield product items for every start URL.
|
||||
|
||||
Each URL is classified and dispatched to its per-kind flow. An unrecognized
|
||||
URL yields an ``invalid_url`` error item.
|
||||
"""
|
||||
for url in input_model.startUrls:
|
||||
resolved = resolve_url(url)
|
||||
if resolved is None:
|
||||
yield _error(
|
||||
"invalid_url",
|
||||
"Start URL was malformed or not a recognized Walmart URL.",
|
||||
input_url=url,
|
||||
)
|
||||
continue
|
||||
async for item in _FLOWS[resolved.kind](resolved, input_model):
|
||||
yield item
|
||||
|
||||
|
||||
async def scrape_products(
|
||||
input_model: WalmartScrapeInput, *, limit: int | None = None
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Collect :func:`iter_products` into a list, honoring an optional ``limit``."""
|
||||
from app.capabilities.core.progress import emit_progress
|
||||
|
||||
results: list[dict[str, Any]] = []
|
||||
async for item in iter_products(input_model):
|
||||
results.append(item)
|
||||
emit_progress("scraping", current=len(results), total=limit, unit="product")
|
||||
if limit is not None and len(results) >= limit:
|
||||
break
|
||||
return results
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Reviews flow (deep pagination). #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
async def _reviews_flow(
|
||||
item_id: str, input_model: WalmartReviewsInput
|
||||
) -> AsyncIterator[dict[str, Any]]:
|
||||
"""Paginate the public reviews page until empty or ``maxReviews`` reached.
|
||||
|
||||
Walmart does not expose a reliable total up front, so the loop stops on the
|
||||
first empty page rather than trusting a page count.
|
||||
"""
|
||||
sort = _REVIEW_SORT[input_model.sort]
|
||||
emitted = 0
|
||||
for page in range(1, _REVIEWS_PAGE_LIMIT + 1):
|
||||
response = await fetch_page(
|
||||
_reviews_url(item_id, page, sort), country=input_model.country
|
||||
)
|
||||
next_data = extract_next_data(response.html) if response else None
|
||||
reviews = parse_reviews_page(next_data) if next_data else []
|
||||
if not reviews:
|
||||
if page == 1 and emitted == 0:
|
||||
yield _error(
|
||||
"reviews_not_found",
|
||||
"The product has no public reviews or could not be loaded.",
|
||||
input_url=item_id,
|
||||
)
|
||||
return
|
||||
for raw in reviews:
|
||||
raw["usItemId"] = item_id
|
||||
raw["input"] = item_id
|
||||
yield ReviewItem(**raw).to_output()
|
||||
emitted += 1
|
||||
if emitted >= input_model.maxReviews:
|
||||
return
|
||||
|
||||
|
||||
async def iter_reviews(
|
||||
input_model: WalmartReviewsInput,
|
||||
) -> AsyncIterator[dict[str, Any]]:
|
||||
"""Yield review items for every requested item id (or resolvable URL)."""
|
||||
for raw_id in input_model.itemIds:
|
||||
item_id = extract_item_id(raw_id)
|
||||
if item_id is None:
|
||||
yield _error(
|
||||
"invalid_url",
|
||||
"Could not extract a Walmart item id from the input.",
|
||||
input_url=raw_id,
|
||||
)
|
||||
continue
|
||||
async for item in _reviews_flow(item_id, input_model):
|
||||
yield item
|
||||
|
||||
|
||||
async def scrape_reviews(
|
||||
input_model: WalmartReviewsInput, *, limit: int | None = None
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Collect :func:`iter_reviews` into a list, honoring an optional ``limit``."""
|
||||
from app.capabilities.core.progress import emit_progress
|
||||
|
||||
results: list[dict[str, Any]] = []
|
||||
async for item in iter_reviews(input_model):
|
||||
results.append(item)
|
||||
emit_progress("scraping", current=len(results), total=limit, unit="review")
|
||||
if limit is not None and len(results) >= limit:
|
||||
break
|
||||
return results
|
||||
|
|
@ -0,0 +1,75 @@
|
|||
"""Classify a Walmart start URL and extract its identifiers.
|
||||
|
||||
Covers the shapes accepted on input: product pages (``/ip/{slug}/{id}`` or
|
||||
``/ip/{id}``), search (``/search?q=``), category (``/cp/{slug}/{id}``), and
|
||||
browse (``/browse/...``) pages. Category and browse render the same
|
||||
``searchResult`` JSON as search, so they share the ``listing`` kind and parser.
|
||||
|
||||
Pure, no I/O. ``item_id`` is Walmart's ``usItemId`` — the numeric id the review
|
||||
page and detail page are both keyed on.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from typing import Literal
|
||||
from urllib.parse import parse_qs, urlparse
|
||||
|
||||
ResolvedKind = Literal["product", "listing"]
|
||||
|
||||
# usItemId: the trailing numeric segment of an /ip/ or /reviews/product/ path,
|
||||
# or a bare numeric id passed directly.
|
||||
_ITEM_ID_PATH_RE = re.compile(r"/(?:ip|reviews/product)/(?:[^/?#]+/)*(\d{4,})")
|
||||
_BARE_ID_RE = re.compile(r"^\d{4,}$")
|
||||
|
||||
_LISTING_QUERY_KEYS = ("q", "cat_id", "browse")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ResolvedUrl:
|
||||
kind: ResolvedKind
|
||||
url: str
|
||||
item_id: str | None = None
|
||||
domain: str | None = None
|
||||
|
||||
|
||||
def extract_item_id(url: str) -> str | None:
|
||||
"""Pull the numeric ``usItemId`` out of a product/review URL or bare id."""
|
||||
candidate = url.strip()
|
||||
if _BARE_ID_RE.match(candidate):
|
||||
return candidate
|
||||
match = _ITEM_ID_PATH_RE.search(candidate)
|
||||
return match.group(1) if match else None
|
||||
|
||||
|
||||
def resolve_url(url: str) -> ResolvedUrl | None:
|
||||
"""Classify a Walmart start URL into a scrape job, or ``None`` if unrecognized.
|
||||
|
||||
A bare numeric id resolves to a product (its ``usItemId``) so callers can
|
||||
pass item ids directly without constructing a full URL.
|
||||
"""
|
||||
if _BARE_ID_RE.match(url.strip()):
|
||||
return ResolvedUrl("product", url.strip(), item_id=url.strip())
|
||||
|
||||
parsed = urlparse(url)
|
||||
host = (parsed.hostname or "").lower()
|
||||
path = parsed.path or ""
|
||||
|
||||
if "walmart." not in host:
|
||||
return None
|
||||
|
||||
if path.startswith("/ip/"):
|
||||
return ResolvedUrl("product", url, item_id=extract_item_id(path), domain=host)
|
||||
|
||||
query = parse_qs(parsed.query)
|
||||
is_listing = (
|
||||
path.startswith("/search")
|
||||
or path.startswith("/cp/")
|
||||
or path.startswith("/browse/")
|
||||
or any(key in query for key in _LISTING_QUERY_KEYS)
|
||||
)
|
||||
if is_listing:
|
||||
return ResolvedUrl("listing", url, domain=host)
|
||||
|
||||
return None
|
||||
|
|
@ -15,6 +15,9 @@ ResolvedKind = Literal["video", "channel", "playlist", "hashtag", "search"]
|
|||
|
||||
_PLAYLIST_ID_RE = re.compile(r"[?&]list=([\w-]+)")
|
||||
|
||||
_YOUTUBE_HOSTS = frozenset({"www.youtube.com", "youtube.com", "m.youtube.com"})
|
||||
"""Hosts whose page paths (/@, /channel, /playlist, ...) are trusted as YouTube."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ResolvedUrl:
|
||||
|
|
@ -29,7 +32,7 @@ def get_youtube_video_id(url: str) -> str | None:
|
|||
hostname = parsed.hostname or ""
|
||||
if hostname == "youtu.be":
|
||||
return parsed.path[1:] or None
|
||||
if hostname in ("www.youtube.com", "youtube.com", "m.youtube.com"):
|
||||
if hostname in _YOUTUBE_HOSTS:
|
||||
if parsed.path == "/watch":
|
||||
return parse_qs(parsed.query).get("v", [None])[0]
|
||||
for prefix in ("/embed/", "/v/", "/shorts/"):
|
||||
|
|
@ -41,17 +44,21 @@ def get_youtube_video_id(url: str) -> str | None:
|
|||
def resolve_url(url: str) -> ResolvedUrl | None:
|
||||
"""Classify a YouTube URL into a scrape job, or ``None`` if unrecognized."""
|
||||
parsed = urlparse(url)
|
||||
hostname = (parsed.hostname or "").lower()
|
||||
path = parsed.path or ""
|
||||
|
||||
# Shorts are videos with their own path.
|
||||
if "/shorts/" in path:
|
||||
vid = path.split("/shorts/")[1].split("/")[0]
|
||||
return ResolvedUrl("video", vid, url) if vid else None
|
||||
|
||||
# Videos: watch / youtu.be / embed / shorts. get_youtube_video_id validates
|
||||
# the host itself, so this also covers youtu.be short links.
|
||||
video_id = get_youtube_video_id(url)
|
||||
if video_id:
|
||||
return ResolvedUrl("video", video_id, url)
|
||||
|
||||
# Every remaining shape is a youtube.com page keyed off its path. Require a
|
||||
# YouTube host so a well-formed non-YouTube URL (e.g. https://evil.com/@x)
|
||||
# is rejected instead of misclassified as a channel by its path alone.
|
||||
if hostname not in _YOUTUBE_HOSTS:
|
||||
return None
|
||||
|
||||
# Playlist (either a /playlist page or any URL carrying ?list=).
|
||||
playlist_match = _PLAYLIST_ID_RE.search(url)
|
||||
if path.startswith("/playlist") and playlist_match:
|
||||
|
|
|
|||
|
|
@ -4,9 +4,11 @@ from fastapi import APIRouter, Depends
|
|||
import app.capabilities.amazon
|
||||
import app.capabilities.google_maps
|
||||
import app.capabilities.google_search
|
||||
import app.capabilities.indeed
|
||||
import app.capabilities.instagram
|
||||
import app.capabilities.reddit
|
||||
import app.capabilities.tiktok
|
||||
import app.capabilities.walmart
|
||||
import app.capabilities.web
|
||||
import app.capabilities.youtube # noqa: F401
|
||||
from app.automations.api import router as automations_router
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import logging
|
||||
import secrets
|
||||
import uuid
|
||||
|
|
@ -17,8 +18,12 @@ from app.etl_pipeline.file_classifier import (
|
|||
DIRECT_CONVERT_EXTENSIONS,
|
||||
PLAINTEXT_EXTENSIONS,
|
||||
)
|
||||
from app.observability import analytics as ph_analytics
|
||||
from app.rate_limiter import limiter
|
||||
from app.tasks.chat.streaming.errors.classifier import classify_stream_exception
|
||||
from app.tasks.chat.streaming.flows.shared.analytics import (
|
||||
build_llm_callback_handler,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
|
@ -354,6 +359,7 @@ async def stream_anonymous_chat(
|
|||
|
||||
accumulator = start_turn()
|
||||
streaming_service = VercelStreamingService()
|
||||
anon_outcome = "success"
|
||||
|
||||
try:
|
||||
async with shielded_async_session():
|
||||
|
|
@ -394,6 +400,21 @@ async def stream_anonymous_chat(
|
|||
"recursion_limit": 40,
|
||||
}
|
||||
|
||||
# PostHog LLM analytics for the free tier — model spend per
|
||||
# model is the highest-value cost insight. distinct_id is the
|
||||
# anon session id (not joined to any registered person).
|
||||
_anon_llm_handler = build_llm_callback_handler(
|
||||
distinct_id=session_id,
|
||||
trace_id=anon_thread_id,
|
||||
properties={
|
||||
"client": "anonymous",
|
||||
"model_slug": body.model_slug,
|
||||
"$ai_session_id": session_id,
|
||||
},
|
||||
)
|
||||
if _anon_llm_handler is not None:
|
||||
langgraph_config["callbacks"] = [_anon_llm_handler]
|
||||
|
||||
yield streaming_service.format_message_start()
|
||||
yield streaming_service.format_start_step()
|
||||
|
||||
|
|
@ -465,6 +486,7 @@ async def stream_anonymous_chat(
|
|||
|
||||
except Exception as e:
|
||||
logger.exception("Anonymous chat stream error")
|
||||
anon_outcome = "error"
|
||||
await TokenQuotaService.anon_release(session_key, ip_key, request_id)
|
||||
_, error_code, _, _, user_message, extra = classify_stream_exception(
|
||||
e,
|
||||
|
|
@ -479,6 +501,26 @@ async def stream_anonymous_chat(
|
|||
finally:
|
||||
await TokenQuotaService.anon_release_stream_slot(client_ip)
|
||||
|
||||
# Server-truth free-tier volume/model/outcome/token-burn. distinct_id
|
||||
# is the anon session id — deliberately NOT joined to the frontend's
|
||||
# PostHog anonymous id (the point is server truth, not funnel merge).
|
||||
with contextlib.suppress(Exception):
|
||||
if ph_analytics.is_enabled():
|
||||
ph_analytics.capture(
|
||||
"anon_chat_turn_completed",
|
||||
distinct_id=session_id,
|
||||
properties={
|
||||
"client": "anonymous",
|
||||
"outcome": anon_outcome,
|
||||
"model_slug": body.model_slug,
|
||||
"model_name": model_cfg.get("model_name"),
|
||||
"total_tokens": accumulator.grand_total,
|
||||
"prompt_tokens": accumulator.total_prompt_tokens,
|
||||
"completion_tokens": accumulator.total_completion_tokens,
|
||||
"cost_micros": accumulator.total_cost_micros,
|
||||
},
|
||||
)
|
||||
|
||||
return StreamingResponse(
|
||||
_generate(),
|
||||
media_type="text/event-stream",
|
||||
|
|
|
|||
|
|
@ -1,7 +1,8 @@
|
|||
# Force asyncio to use standard event loop before unstructured imports
|
||||
import asyncio
|
||||
|
||||
from fastapi import APIRouter, Depends, Form, HTTPException, Query, UploadFile
|
||||
from fastapi import APIRouter, Depends, Form, HTTPException, Query, Request, UploadFile
|
||||
from fastapi.responses import PlainTextResponse
|
||||
from pydantic import BaseModel as PydanticBaseModel, Field
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.future import select
|
||||
|
|
@ -34,6 +35,8 @@ from app.schemas import (
|
|||
FolderRead,
|
||||
PaginatedResponse,
|
||||
)
|
||||
from app.services.export_service import resolve_document_markdown
|
||||
from app.services.okf import document_to_concept
|
||||
from app.services.task_dispatcher import TaskDispatcher, get_task_dispatcher
|
||||
from app.users import get_auth_context
|
||||
from app.utils.rbac import check_permission
|
||||
|
|
@ -1255,12 +1258,14 @@ async def get_document_chunks_paginated(
|
|||
@router.get("/documents/{document_id}", response_model=DocumentRead)
|
||||
async def read_document(
|
||||
document_id: int,
|
||||
request: Request,
|
||||
session: AsyncSession = Depends(get_async_session),
|
||||
auth: AuthContext = Depends(get_auth_context),
|
||||
):
|
||||
"""
|
||||
Get a specific document by ID.
|
||||
Requires DOCUMENTS_READ permission for the workspace.
|
||||
"""Get a document by ID.
|
||||
|
||||
``Accept: text/markdown`` returns the OKF concept; otherwise the JSON record.
|
||||
Requires DOCUMENTS_READ permission.
|
||||
"""
|
||||
try:
|
||||
result = await session.execute(
|
||||
|
|
@ -1282,6 +1287,16 @@ async def read_document(
|
|||
"You don't have permission to read documents in this workspace",
|
||||
)
|
||||
|
||||
# ponytail: substring match, not RFC 7231 q-values (OKF is the only non-JSON view).
|
||||
if "text/markdown" in request.headers.get("accept", ""):
|
||||
body = (
|
||||
await resolve_document_markdown(session, document)
|
||||
or document.content
|
||||
or ""
|
||||
)
|
||||
concept = document_to_concept(document, body=body)
|
||||
return PlainTextResponse(concept, media_type="text/markdown")
|
||||
|
||||
raw_content = document.content or ""
|
||||
return DocumentRead(
|
||||
id=document.id,
|
||||
|
|
|
|||
|
|
@ -26,6 +26,7 @@ from app.db import (
|
|||
WorkspaceMembership,
|
||||
get_async_session,
|
||||
)
|
||||
from app.observability import analytics as ph_analytics
|
||||
from app.schemas import (
|
||||
ImageGenerationCreate,
|
||||
ImageGenerationListRead,
|
||||
|
|
@ -327,6 +328,20 @@ async def create_image_generation(
|
|||
|
||||
await session.commit()
|
||||
await session.refresh(db_image_gen)
|
||||
|
||||
# Credit-consuming deliverable; the frontend never confirms
|
||||
# completion. ``error_message`` set => provider call failed.
|
||||
ph_analytics.capture_for(
|
||||
auth,
|
||||
"image_generated",
|
||||
{
|
||||
"workspace_id": data.workspace_id,
|
||||
"model": data.model,
|
||||
"n": data.n,
|
||||
"status": "error" if db_image_gen.error_message else "success",
|
||||
},
|
||||
groups={"workspace": str(data.workspace_id)},
|
||||
)
|
||||
return db_image_gen
|
||||
|
||||
except HTTPException:
|
||||
|
|
|
|||
|
|
@ -15,6 +15,7 @@ from app.db import (
|
|||
UserIncentiveTask,
|
||||
get_async_session,
|
||||
)
|
||||
from app.observability import analytics as ph_analytics
|
||||
from app.schemas.incentive_tasks import (
|
||||
CompleteTaskResponse,
|
||||
IncentiveTaskInfo,
|
||||
|
|
@ -125,6 +126,20 @@ async def complete_task(
|
|||
await session.commit()
|
||||
await session.refresh(user)
|
||||
|
||||
# Authoritative reward grant (migrated from earn-credits-content.tsx).
|
||||
# Placed after the already-completed early-return so retries never
|
||||
# double-count.
|
||||
ph_analytics.capture_for(
|
||||
auth,
|
||||
"incentive_task_completed",
|
||||
{
|
||||
"task_type": task_type.value
|
||||
if hasattr(task_type, "value")
|
||||
else str(task_type),
|
||||
"credit_micros_rewarded": credit_micros_reward,
|
||||
},
|
||||
)
|
||||
|
||||
return CompleteTaskResponse(
|
||||
success=True,
|
||||
message=f"Task completed! You earned ${credit_micros_reward / 1_000_000:.2f} of credit.",
|
||||
|
|
|
|||
|
|
@ -24,7 +24,6 @@ from app.schemas import (
|
|||
ConnectionRead,
|
||||
ConnectionUpdate,
|
||||
LlmSetupStatusRead,
|
||||
ModelCreate,
|
||||
ModelPreviewRead,
|
||||
ModelProviderRead,
|
||||
ModelRead,
|
||||
|
|
@ -619,49 +618,6 @@ async def discover_connection_models(
|
|||
return [_model_read(model) for model in conn.models]
|
||||
|
||||
|
||||
@router.post("/model-connections/{connection_id}/models", response_model=ModelRead)
|
||||
async def add_manual_model(
|
||||
connection_id: int,
|
||||
data: ModelCreate,
|
||||
session: AsyncSession = Depends(get_async_session),
|
||||
auth: AuthContext = Depends(get_auth_context),
|
||||
):
|
||||
conn = await _load_connection(session, connection_id)
|
||||
await _assert_connection_access(
|
||||
session, auth, conn, Permission.LLM_CONFIGS_UPDATE.value
|
||||
)
|
||||
|
||||
model_id = data.model_id.strip()
|
||||
if not model_id:
|
||||
raise HTTPException(status_code=400, detail="model_id is required")
|
||||
if any(existing.model_id == model_id for existing in conn.models):
|
||||
raise HTTPException(
|
||||
status_code=400, detail="Model already exists on this connection"
|
||||
)
|
||||
|
||||
capabilities = derive_capabilities(conn, model_id)
|
||||
model = Model(
|
||||
connection_id=conn.id,
|
||||
model_id=model_id,
|
||||
display_name=data.display_name or None,
|
||||
source=ModelSource.MANUAL,
|
||||
capabilities_override={},
|
||||
enabled=True,
|
||||
catalog={},
|
||||
)
|
||||
_apply_model_facts(model, capabilities)
|
||||
session.add(model)
|
||||
await session.commit()
|
||||
await session.refresh(model)
|
||||
conn = await _load_connection(session, connection_id)
|
||||
await _default_unset_roles(session, conn, list(conn.models))
|
||||
if conn.workspace_id is not None:
|
||||
await _clear_invalid_roles(session, conn.workspace_id)
|
||||
await session.commit()
|
||||
await session.refresh(model)
|
||||
return _model_read(model)
|
||||
|
||||
|
||||
@router.patch(
|
||||
"/model-connections/{connection_id}/models", response_model=list[ModelRead]
|
||||
)
|
||||
|
|
|
|||
|
|
@ -51,6 +51,7 @@ from app.db import (
|
|||
get_async_session,
|
||||
shielded_async_session,
|
||||
)
|
||||
from app.observability import analytics as ph_analytics
|
||||
from app.schemas.new_chat import (
|
||||
AgentToolInfo,
|
||||
CancelActiveTurnResponse,
|
||||
|
|
@ -823,6 +824,15 @@ async def create_thread(
|
|||
session.add(db_thread)
|
||||
await session.commit()
|
||||
await session.refresh(db_thread)
|
||||
|
||||
# Authoritative thread creation (migrated from stream-engine/engine.ts).
|
||||
# get_auth_context => covers PAT/MCP callers the frontend never sees.
|
||||
ph_analytics.capture_for(
|
||||
auth,
|
||||
"chat_created",
|
||||
{"workspace_id": db_thread.workspace_id, "chat_id": db_thread.id},
|
||||
groups={"workspace": str(db_thread.workspace_id)},
|
||||
)
|
||||
return db_thread
|
||||
|
||||
except HTTPException:
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ from sqlalchemy.future import select
|
|||
from app.auth.context import AuthContext
|
||||
from app.config import config
|
||||
from app.db import PersonalAccessToken, get_async_session
|
||||
from app.observability import analytics as ph_analytics
|
||||
from app.schemas.pat import PATCreate, PATCreated, PATRead
|
||||
from app.users import require_session_context
|
||||
from app.utils.pat import generate_pat, hash_pat, token_prefix
|
||||
|
|
@ -57,6 +58,13 @@ async def create_personal_access_token(
|
|||
await session.commit()
|
||||
await session.refresh(pat)
|
||||
|
||||
# Leading indicator of MCP / programmatic-API adoption.
|
||||
ph_analytics.capture_for(
|
||||
auth,
|
||||
"pat_created",
|
||||
{"pat_id": pat.id, "has_expiry": pat.expires_at is not None},
|
||||
)
|
||||
|
||||
return PATCreated(
|
||||
id=pat.id,
|
||||
label=pat.label,
|
||||
|
|
@ -102,3 +110,5 @@ async def delete_personal_access_token(
|
|||
)
|
||||
)
|
||||
await session.commit()
|
||||
|
||||
ph_analytics.capture_for(auth, "pat_revoked", {"pat_id": pat_id})
|
||||
|
|
|
|||
|
|
@ -13,6 +13,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
|||
|
||||
from app.auth.context import AuthContext
|
||||
from app.db import get_async_session
|
||||
from app.observability import analytics as ph_analytics
|
||||
from app.schemas.new_chat import (
|
||||
CloneResponse,
|
||||
PublicChatResponse,
|
||||
|
|
@ -56,7 +57,20 @@ async def clone_public_chat(
|
|||
Creates thread and copies messages.
|
||||
Requires authentication.
|
||||
"""
|
||||
return await clone_from_snapshot(session, share_token, user)
|
||||
result = await clone_from_snapshot(session, share_token, user)
|
||||
|
||||
# Share-link conversion — only observable server-side.
|
||||
ph_analytics.capture_for(
|
||||
auth,
|
||||
"public_chat_cloned",
|
||||
{
|
||||
"workspace_id": result.workspace_id,
|
||||
"chat_id": result.thread_id,
|
||||
},
|
||||
groups={"workspace": str(result.workspace_id)},
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
@router.get("/{share_token}/podcasts/{podcast_id}")
|
||||
|
|
|
|||
|
|
@ -28,6 +28,7 @@ from app.db import (
|
|||
WorkspaceRole,
|
||||
get_async_session,
|
||||
)
|
||||
from app.observability import analytics as ph_analytics
|
||||
from app.schemas import (
|
||||
InviteAcceptRequest,
|
||||
InviteAcceptResponse,
|
||||
|
|
@ -782,6 +783,19 @@ async def create_invite(
|
|||
)
|
||||
db_invite = result.scalars().first()
|
||||
|
||||
# Authoritative invite creation (migrated from team-content.tsx).
|
||||
ph_analytics.capture_for(
|
||||
auth,
|
||||
"workspace_invite_sent",
|
||||
{
|
||||
"workspace_id": workspace_id,
|
||||
"role_name": db_invite.role.name if db_invite.role else None,
|
||||
"has_expiry": db_invite.expires_at is not None,
|
||||
"has_max_uses": db_invite.max_uses is not None,
|
||||
},
|
||||
groups={"workspace": str(workspace_id)},
|
||||
)
|
||||
|
||||
return db_invite
|
||||
|
||||
except HTTPException:
|
||||
|
|
@ -1091,6 +1105,16 @@ async def accept_invite(
|
|||
role_name = invite.role.name if invite.role else "Default"
|
||||
workspace_name = invite.workspace.name if invite.workspace else ""
|
||||
|
||||
# Authoritative join (migrated from app/invite/[invite_code]/page.tsx,
|
||||
# which fired both events). workspace_name dropped — user content.
|
||||
for _evt in ("workspace_invite_accepted", "workspace_user_added"):
|
||||
ph_analytics.capture_for(
|
||||
auth,
|
||||
_evt,
|
||||
{"workspace_id": invite.workspace_id, "role_name": role_name},
|
||||
groups={"workspace": str(invite.workspace_id)},
|
||||
)
|
||||
|
||||
return InviteAcceptResponse(
|
||||
message="Successfully joined the workspace",
|
||||
workspace_id=invite.workspace_id,
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue