apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-08 22:22:17 +02:00
Code Issues Projects Releases Packages Wiki Activity

Merge remote-tracking branch 'upstream/ci_mvp' into feat/ci-ui-changes

Browse source
This commit is contained in:
Anish Sarkar 2026-07-06 13:25:51 +05:30
commit fd8d1273cd
188 changed files with 2163 additions and 3644 deletions
Download patch file Download diff file Expand all files Collapse all files

371
README.es.md
View file

@ -1,4 +1,4 @@
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="readme_banner" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="SurfSense, la plataforma de inteligencia competitiva de código abierto para agentes de IA" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
@ -20,289 +20,224 @@
<a href="https://trendshift.io/repositories/13606" target="_blank"><img src="https://trendshift.io/api/badge/repositories/13606" alt="MODSetter%2FSurfSense | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</div>
# SurfSense
# SurfSense: Dale inteligencia competitiva a tus agentes de IA
NotebookLM es una de las mejores y más útiles plataformas de IA que existen, pero una vez que comienzas a usarla regularmente también sientes sus limitaciones dejando algo que desear.
SurfSense es la **plataforma de inteligencia competitiva de código abierto para agentes de IA**. Tus agentes monitorean a la competencia, siguen los rankings y escuchan a tu mercado con datos en vivo de **Reddit, YouTube, Google Maps, Google Search y la web abierta**, a través de una única **API REST** o un **servidor MCP**. Agentes programados y activados por eventos convierten lo que encuentran en informes y alertas, y una base de conocimiento integrada mantiene cada hallazgo disponible para búsqueda con citas.
1. Hay límites en la cantidad de fuentes que puedes agregar en un notebook.
2. Hay límites en la cantidad de notebooks que puedes tener.
3. No puedes tener fuentes que excedan 500,000 palabras y más de 200MB.
4. Estás bloqueado con los servicios de Google (LLMs, modelos de uso, etc.) sin opción de configurarlos.
5. Fuentes de datos externas e integraciones de servicios limitadas.
6. El agente de NotebookLM está específicamente optimizado solo para estudiar e investigar, pero puedes hacer mucho más con los datos de origen.
7. Falta de soporte multijugador.
> [!NOTE]
> **📢 Una nota para nuestros usuarios de la alternativa a NotebookLM**
>
> Durante los últimos meses construimos SurfSense como el mejor agente de investigación general para tu propio conocimiento, y ese capítulo nos dio una comunidad de la que estamos genuinamente orgullosos. Herramientas agénticas como Claude, OpenCode, Hermes y OpenClaw ya han demostrado que los agentes son el futuro, y la investigación general se está convirtiendo en algo que todo agente capaz hace de fábrica. Lo que a los agentes todavía les falta son **datos de mercado en vivo y los flujos de trabajo a su alrededor**, así que ahí es donde estamos dirigiendo toda nuestra energía: convertirnos en la plataforma definitiva de agentes de inteligencia competitiva de código abierto.
>
> **Nada de lo que dependes va a desaparecer.** Tu base de conocimiento, el chat con citas, los informes, los podcasts, las presentaciones, las automatizaciones y los chats colaborativos siguen funcionando, y el autoalojamiento sigue siendo gratuito y de código abierto. Lee el anuncio completo en [nuestro changelog](https://www.surfsense.com/changelog).
...y más.
## Por qué los agentes necesitan SurfSense
**SurfSense está específicamente hecho para resolver estos problemas.** SurfSense te permite:
Pregúntale a cualquier agente capaz "¿cuánto están cobrando los competidores esta semana?" o "¿qué está diciendo Reddit sobre nosotros desde el lanzamiento?" y no tiene ningún lugar confiable donde buscar. Las APIs oficiales de las plataformas tienen límites de tasa, precios pensados para empresas grandes o directamente no existen, y la infraestructura de scraping es frágil. SurfSense cierra esa brecha:
- **Controla Tu Flujo de Datos** - Mantén tus datos privados y seguros.
- **Sin Límites de Datos** - Agrega una cantidad ilimitada de fuentes y notebooks.
- **Sin Dependencia de Proveedores** - Configura cualquier modelo LLM, de imagen, TTS y STT.
- **25+ Fuentes de Datos Externas** - Agrega tus fuentes desde Google Drive, OneDrive, Dropbox, Notion y muchos otros servicios externos.
- **Soporte Multijugador en Tiempo Real** - Trabaja fácilmente con los miembros de tu equipo en un notebook compartido.
- **Automatizaciones y Agentes de IA** - Ejecuta agentes de IA según una programación o actívalos en el momento en que un documento llega a una carpeta, y luego escribe los resultados de vuelta en Notion, Slack, Linear y Drive. Crea automatizaciones sin código solo describiéndolas en el chat.
- **Aplicación de Escritorio** - Obtén asistencia de IA en cualquier aplicación con Quick Assist, General Assist, Screenshot Assist y sincronización de carpetas locales.
- **Conectores nativos por plataforma**, cada uno un endpoint REST tipado que devuelve JSON estructurado. Sin ruleta de límites de tasa, sin parsear HTML.
- **Un servidor MCP** que expone cada conector como una herramienta nativa (`reddit.scrape`, `google_search.scrape` y más) para Claude, Cursor o cualquier framework de agentes.
- **Un arnés de agentes**, no solo datos en bruto: reintentos, salida estructurada y medición de créditos vienen integrados, así que los agentes pasan de una pregunta a un informe sin que tú construyas la infraestructura.
- **Código abierto y autoalojable**, para que tu investigación competitiva se quede en tu propia infraestructura.
...y más por venir.
## Conectores de datos en vivo
| Conector | Qué obtienen tus agentes | Más información |
|---|---|---|
| **Reddit** | Publicaciones, comentarios y flujos de subreddits sin los límites de tasa de la API oficial | [Reddit Scraper API](https://www.surfsense.com/reddit) |
| **YouTube** | Videos, transcripciones e hilos de comentarios para escuchar a tu marca y productos | [YouTube Scraper API](https://www.surfsense.com/youtube) |
| **Google Maps** | Lugares, calificaciones y reseñas para investigar competidores locales y prospectos | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
| **Google Search** | SERPs en vivo para seguimiento de posiciones y monitoreo de mercado | [Google Search API](https://www.surfsense.com/google-search) |
| **Web Crawl** (rastreo web) | Cualquier página de la web abierta como contenido limpio y estructurado | [Web Crawling API](https://www.surfsense.com/web-crawl) |
| **MCP Connector** | Conecta cualquier servidor MCP a tus agentes, con OAuth de un clic para Notion, Slack, Jira y más | [MCP Connector](https://www.surfsense.com/mcp-connector) |
La facturación es de pago por uso: los conectores facturan por elemento realmente devuelto, los rastreos por página obtenida con éxito, y las llamadas fallidas nunca se facturan. Las instalaciones autoalojadas funcionan con la facturación desactivada. Consulta los [precios](https://www.surfsense.com/pricing).
## Ejemplo de Agente de Video
## Inicio rápido
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
### Llama a un conector desde código
Cada conector es un endpoint REST que puedes llamar desde cualquier lenguaje con tu clave de API de SurfSense:
```bash
curl -X POST "$SURFSENSE_API_URL/workspaces/$WORKSPACE_ID/scrapers/reddit/scrape" \
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"search_queries": ["your brand"],
"community": "webscraping",
"sort": "top",
"time_filter": "week"
}'
```
## Ejemplo de Agente de Podcast
Cada [página de conector](https://www.surfsense.com/connectors) tiene ejemplos listos para copiar y pegar en Python, JavaScript, Go, PHP, Ruby, Java y C#.
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
### Dale las herramientas a tus agentes vía MCP
Agrega el servidor MCP de SurfSense a Claude, Cursor o tu propio framework de agentes:
## Cómo usar SurfSense
```json
{
"mcpServers": {
"surfsense": {
"url": "https://mcp.surfsense.com",
"headers": { "Authorization": "Bearer ${SURFSENSE_API_KEY}" }
}
}
}
```
### Cloud
Tu agente ahora puede llamar a cada conector como una herramienta nativa.
1. Ve a [surfsense.com](https://www.surfsense.com) e inicia sesión.
### Usa la nube
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/LoginFlowGif.gif" alt="Login" /></p>
Ve a [surfsense.com](https://www.surfsense.com), inicia sesión y pídele al agente datos de mercado en vivo en lenguaje natural. Las cuentas nuevas comienzan con $5 de crédito gratuito y sin suscripción.
2. Conecta tus conectores y sincroniza. Activa la sincronización periódica para mantenerlos actualizados.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="Conectores" /></p>
### Autoalójalo gratis
3. Mientras se indexan los datos de los conectores, sube documentos.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="Subir Documentos" /></p>
4. Una vez que todo esté indexado, pregunta lo que quieras (Casos de uso):
**Aplicación de Escritorio** (extras nativos, además de todo lo de abajo, no un conjunto aparte)
- General Assist: abre SurfSense al instante desde cualquier aplicación con un atajo global.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/general_assist.gif" alt="General Assist" /></p>
- Quick Assist: selecciona texto en cualquier lugar y pide a la IA que lo explique, reescriba o actúe sobre él.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
- Screenshot Assist: captura cualquier región de tu pantalla y pregunta a la IA sobre lo que contiene.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/screenshot_assist.gif" alt="Screenshot Assist" /></p>
- Watch Local Folder: sincroniza automáticamente una carpeta local con tu base de conocimiento. Ideal para bóvedas de Obsidian.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/folder_watch.gif" alt="Watch Local Folder" /></p>
**Estudio de Entregables**
- AI Report Generator: genera informes de investigación con citas y expórtalos a PDF, DOCX, HTML, LaTeX, EPUB, ODT o texto plano.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="AI Report Generator" /></p>
- AI Podcast Generator: convierte cualquier documento o carpeta en un pódcast de IA con dos presentadores en menos de 20 segundos.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/PodcastGenGif.gif" alt="AI Podcast Generator" /></p>
- AI Presentation & Video Maker: crea presentaciones editables y videos narrados a partir de tus fuentes.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/video_gen_gif.gif" alt="AI Presentation and Video Maker" /></p>
- AI Image Generator: genera imágenes de alta calidad directamente desde tus chats y documentos.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ImageGenGif.gif" alt="AI Image Generator" /></p>
- AI Resume Builder: adapta tu currículum existente a cualquier descripción de empleo y supera el ATS.
Prueba indicaciones como estas:
- "Adapta mi currículum a esta descripción de empleo para superar el ATS y conseguir una entrevista."
- "Optimiza mi currículum para ATS haciendo coincidir las palabras clave de esta oferta."
- "Reescribe los puntos de mi currículum para resaltar las habilidades que pide este puesto."
- "Compara mi currículum con esta descripción de empleo y enumera las carencias a corregir."
- "Escribe una carta de presentación a juego con mi currículum y esta descripción de empleo."
**Búsqueda y Chat**
- Chat With Your PDFs & Docs: haz preguntas sobre todos tus archivos y obtén respuestas con citas en línea.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="Chat With Your PDFs and Docs" /></p>
- AI Search With Citations: búsqueda híbrida semántica y por palabras clave en toda tu base de conocimiento.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BSNCGif.gif" alt="AI Search With Citations" /></p>
- Collaborative AI Chat: trabaja en conversaciones de IA con tu equipo en tiempo real.
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Collaborative AI Chat" /></p>
- Comments & Mentions: comenta y menciona a tus compañeros en cualquier mensaje de IA.
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Comments and Mentions" /></p>
**Conectores e Integraciones**
- Connect & Sync Your Tools: sincroniza Notion, Slack, Google Drive, Gmail, GitHub, Linear y más de 25 fuentes en un único corpus consultable.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="Connect and Sync Your Tools" /></p>
- Chat With Uploaded Files: sube PDFs, documentos de Office, imágenes y audio. Consultables al instante.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="Chat With Uploaded Files" /></p>
- Connector Write-Back: deja que el agente publique los resultados de vuelta en Notion, Slack, Linear y Drive.
Prueba indicaciones como estas:
- "Publica este resumen de investigación en mi espacio de Notion."
- "Envía estos elementos de acción de la reunión a nuestro canal de Slack."
- "Crea un ticket de Jira a partir de este informe de error."
- "Abre una incidencia en Linear a partir de esta solicitud de función."
- "Guarda este informe generado en Google Drive como un documento."
- Obsidian & Knowledge Base Sync: mantén tu bóveda de Obsidian y tu base de conocimiento personal sincronizadas.
**Automatizaciones**
- Scheduled AI Workflows: ejecuta un agente según una programación: resúmenes diarios, boletines semanales, informes recurrentes.
Prueba indicaciones como estas:
- "Envíame cada mañana un resumen diario de los nuevos documentos en mi base de conocimiento."
- "Genera un informe de estado semanal a partir de mi Slack y Gmail cada viernes."
- "Ejecuta un informe mensual de análisis de la competencia y guárdalo en mi espacio de trabajo."
- "Resume mi actividad de GitHub y Linear en una actualización diaria de standup."
- "Crea un informe de investigación semanal recurrente sobre los temas que sigo."
- Event-Triggered Automations: lanza un agente en el momento en que un documento llega a una carpeta y publica el resultado en tus herramientas.
Prueba indicaciones como estas:
- "Cuando llegue un PDF a mi carpeta de Investigación, genera un resumen de IA con citas."
- "Cuando se añadan nuevas notas de reunión, conviértelas en actas con elementos de acción."
- "Cuando se suba una factura, extrae el proveedor, el total y la fecha de vencimiento en una tabla."
- "Cuando entre un contrato en mi carpeta Legal, señala los términos clave y las fechas de renovación."
- "Cuando se añada un currículum a Candidatos, evalúalo frente a la descripción del empleo."
- Chat-Built Automations: describe una automatización en lenguaje sencillo y SurfSense la crea por ti.
Prueba indicaciones como estas:
- "Crea un agente de IA que me envíe cada mañana un resumen de las nuevas páginas de Notion."
- "Crea una automatización sin código que publique un resumen de investigación semanal en Slack."
- "Configura un tomador de notas con IA que convierta las nuevas notas de reunión en actas."
- "Crea un flujo que extraiga los elementos de acción de las notas de reunión y asigne responsables."
- "Automatiza un resumen diario por correo a partir de mi Gmail y Google Drive."
### Auto-Hospedado
Ejecuta SurfSense en tu propia infraestructura para control total de datos y privacidad.
Ejecuta toda la plataforma, los conectores, los agentes, las automatizaciones y el servidor MCP en tu propia infraestructura. Las instalaciones autoalojadas vienen con la facturación desactivada, así que el scraping, el rastreo y las ejecuciones de agentes solo están limitados por tu hardware y las claves de modelos que aportes.
**Requisitos previos:** [Docker Desktop](https://www.docker.com/products/docker-desktop/) debe estar instalado y en ejecución.
#### Para usuarios de Linux/MacOS:
Para Linux/macOS:
```bash
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash
```
#### Para usuarios de Windows:
Para Windows:
```powershell
```bash
irm https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.ps1 | iex
```
El script de instalación configura [Watchtower](https://github.com/nicholas-fedor/watchtower) automáticamente para actualizaciones diarias. Para omitirlo, agrega la bandera `--no-watchtower`.
El script de instalación configura [Watchtower](https://github.com/nicholas-fedor/watchtower) automáticamente para actualizaciones automáticas diarias. Para omitirlo, agrega la opción `--no-watchtower`. Para Docker Compose, instalación manual y otras opciones de despliegue, consulta la [documentación](https://www.surfsense.com/docs/).
Para Docker Compose, instalación manual y otras opciones de despliegue, consulta la [documentación](https://www.surfsense.com/docs/).
## Flujos de trabajo de inteligencia competitiva
### Aplicación de Escritorio
Las automatizaciones ejecutan turnos completos de agente según un horario o en respuesta a eventos, y luego escriben los resultados en Notion, Slack, Linear y Jira. Describe el flujo de trabajo en lenguaje natural y SurfSense lo construye, sin necesidad de código. Prueba con instrucciones como:
SurfSense también ofrece una aplicación de escritorio que lleva la asistencia de IA a cada aplicación en tu computadora. Descárgala desde la [última versión](https://github.com/MODSetter/SurfSense/releases/latest).
- "Vigila las páginas de precios de nuestros 3 principales competidores y avísame en Slack cuando cambie un plan o un precio."
- "Rastrea cada mención de nuestra marca en Reddit y YouTube y envíame un resumen diario."
- "Monitorea nuestro ranking en Google para nuestras 10 palabras clave principales y señala las caídas semana a semana."
- "Extrae las nuevas reseñas de Google Maps de nuestras ubicaciones y las de nuestros competidores cada lunes."
- "Ejecuta un informe mensual de análisis de la competencia y guárdalo en mi espacio de trabajo."
La aplicación de escritorio incluye estas potentes funciones:
- **General Assist** — Lanza SurfSense al instante desde cualquier aplicación con un atajo global.
- **Quick Assist** — Selecciona texto en cualquier lugar, luego pide a la IA que lo explique, reescriba o actúe sobre él.
- **Screenshot Assist** — Selecciona una región de tu pantalla y adjúntala al chat para que las respuestas se basen en tu base de conocimiento.
- **Watch Local Folder** — Vigila una carpeta local y sincroniza automáticamente los cambios de archivos con tu base de conocimiento. **Pro tip:** Apúntalo a tu bóveda de Obsidian para mantener tus notas buscables en SurfSense.
## Todo lo demás que viene incluido
Todas las funciones operan contra tu espacio de búsqueda elegido, por lo que tus respuestas siempre están basadas en tus propios datos.
El espacio de trabajo de investigación que convirtió a SurfSense en la alternativa de código abierto líder a NotebookLM sigue aquí, y todo lo que tus agentes recopilan aterriza en él.
### Cómo Colaborar en Tiempo Real (Beta)
**Base de conocimiento**
1. Ve a la página de Gestión de Miembros y crea una invitación.
- Sube PDFs, documentos de Office, imágenes y audio, o sincroniza **Google Drive, OneDrive y Dropbox**. Más de 50 formatos de archivo compatibles.
- Búsqueda híbrida semántica y de texto completo con respuestas citadas al estilo Perplexity.
- La organización de archivos con IA clasifica los documentos automáticamente por fuente, fecha y tema.
<p align="center"><img src="https://github.com/user-attachments/assets/40ed7683-5aa6-48a0-a3df-00575528c392" alt="Invitar Miembros" /></p>
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="Chatea con tus PDFs y documentos" /></p>
2. El compañero de equipo se une y ese SearchSpace se comparte.
**Estudio de entregables**
<p align="center"><img src="https://github.com/user-attachments/assets/ea4e1057-4d2b-4fd2-9ca0-cd19286a285e" alt="Flujo de Unión por Invitación" /></p>
- Generador de informes con IA con exportación a PDF, DOCX, HTML, LaTeX, EPUB, ODT o texto plano.
- Podcasts de IA con dos presentadores a partir de cualquier documento o carpeta en menos de 20 segundos.
- Presentaciones de diapositivas editables, resúmenes en video narrados y generación de imágenes con IA.
3. Haz el chat compartido.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="Generador de informes con IA" /></p>
<p align="center"><img src="https://github.com/user-attachments/assets/17b93904-0888-4c3a-ac12-51a24a8ea26a" alt="Hacer Chat Compartido" /></p>
**Colaboración en equipo**
4. Tu equipo ahora puede chatear en tiempo real.
- Chats de IA colaborativos en tiempo real con comentarios y menciones.
- RBAC con roles de Propietario, Administrador, Editor y Visualizador.
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Chat en Tiempo Real" /></p>
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Chat de IA colaborativo" /></p>
5. Agrega comentarios para etiquetar a compañeros de equipo.
**Aplicación de escritorio**
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Comentarios en Tiempo Real" /></p>
Asistencia de IA nativa en todas las aplicaciones de tu computadora. Descárgala desde la [última versión](https://github.com/MODSetter/SurfSense/releases/latest).
- **General Assist**: abre SurfSense desde cualquier aplicación con un atajo global.
- **Quick Assist**: selecciona texto en cualquier lugar y pídele a la IA que lo explique, lo reescriba o actúe sobre él.
- **Screenshot Assist**: captura cualquier región de tu pantalla y pregúntale a la IA sobre ella.
- **Watch Local Folder** (vigilar carpeta local): sincroniza automáticamente una carpeta local con tu base de conocimiento. Apúntala a tu bóveda de Obsidian para mantener tus notas disponibles para búsqueda.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
**Sin dependencia de un proveedor**
- Más de 100 LLMs vía la especificación de OpenAI y LiteLLM, incluidos GPT-5.5, Claude Sonnet 5 y Gemini 3.1 Pro.
- Más de 6,000 modelos de embeddings y todos los rerankers principales.
- Soporte completo para LLMs locales y privados (vLLM, Ollama), para que tus datos sigan siendo tuyos.
## Muestra del agente de video
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
## Muestra del agente de podcast
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
## Cómo colaborar en tiempo real (Beta)
1. Ve a la página de administración de miembros y crea una invitación.
<p align="center"><img src="https://github.com/user-attachments/assets/40ed7683-5aa6-48a0-a3df-00575528c392" alt="Invitar miembros" /></p>
2. Un compañero de equipo se une y ese espacio de trabajo se vuelve compartido.
<p align="center"><img src="https://github.com/user-attachments/assets/ea4e1057-4d2b-4fd2-9ca0-cd19286a285e" alt="Flujo de invitación y unión" /></p>
3. Haz que un chat sea compartido y trabajen en él juntos en tiempo real, con comentarios para etiquetar a compañeros.
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Comentarios en tiempo real" /></p>
## SurfSense vs Google NotebookLM
¿Todavía nos comparas como una alternativa a NotebookLM? Aquí tienes el desglose honesto.
| Característica | Google NotebookLM | SurfSense |
|---------|-------------------|-----------|
| **Fuentes por Notebook** | 50 (Gratis) a 600 (Ultra, $249.99/mes) | Ilimitadas |
| **Número de Notebooks** | 100 (Gratis) a 500 (planes de pago) | Ilimitados |
| **Límite de Tamaño de Fuente** | 500,000 palabras / 200MB por fuente | Sin límite |
| **Precios** | Nivel gratuito disponible; Pro $19.99/mes, Ultra $249.99/mes | Gratuito y de código abierto, auto-hospedable en tu propia infra |
| **Soporte de LLM** | Solo Google Gemini | 100+ LLMs vía OpenAI spec y LiteLLM |
| **Modelos de Embeddings** | Solo Google | 6,000+ modelos de embeddings, todos los principales rerankers |
| **LLMs Locales / Privados** | No disponible | Soporte completo (vLLM, Ollama) - tus datos son tuyos |
| **Auto-Hospedable** | No | Sí - Docker en un solo comando o Docker Compose completo |
| **Código Abierto** | No | Sí |
| **Conectores Externos** | Google Drive, YouTube, sitios web | 27+ conectores - Motores de búsqueda, Google Drive, OneDrive, Dropbox, Slack, Teams, Jira, Notion, GitHub, Discord y [más](#fuentes-externas) |
| **Soporte de Formatos de Archivo** | PDFs, Docs, Slides, Sheets, CSV, Word, EPUB, imágenes, URLs web, YouTube | 50+ formatos - documentos, imágenes, videos vía LlamaCloud, Unstructured o Docling (local) |
| **Búsqueda** | Búsqueda semántica | Búsqueda Híbrida - Semántica + Texto completo con Índices Jerárquicos y Reciprocal Rank Fusion |
| **Respuestas con Citas** | Sí | Sí - Respuestas citadas al estilo Perplexity |
| **Arquitectura de Agentes** | No | Sí - impulsado por [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) con planificación, subagentes y acceso al sistema de archivos |
| **Multijugador en Tiempo Real** | Notebooks compartidos con roles de Visor/Editor (sin chat en tiempo real) | RBAC con roles de Propietario / Admin / Editor / Visor, chat en tiempo real e hilos de comentarios |
| **Generación de Videos** | Resúmenes en video cinemáticos vía Veo 3 (solo Ultra) | Disponible (NotebookLM es mejor aquí, mejorando activamente) |
| **Generación de Presentaciones** | Diapositivas más atractivas pero no editables | Crea presentaciones editables basadas en diapositivas |
| **Generación de Podcasts** | Resúmenes de audio con hosts e idiomas personalizables | Disponible con múltiples proveedores TTS (NotebookLM es mejor aquí, mejorando activamente) |
| **Automatizaciones y Agentes de IA** | No | Flujos de trabajo de IA programados, disparadores por eventos en documentos nuevos y automatizaciones sin código creadas por chat con escritura de vuelta a Notion, Slack, Linear y Jira |
| **Aplicación de Escritorio** | No | Aplicación nativa con General Assist, Quick Assist, Screenshot Assist y sincronización de carpetas locales |
| **Extensión de Navegador** | No | Extensión multi-navegador para guardar cualquier página web, incluyendo páginas protegidas por autenticación |
| **Datos de mercado en vivo para agentes** | No | Conectores de Reddit, YouTube, Google Maps, Google Search y rastreo web vía API REST y MCP |
| **Servidor MCP** | No | Cada conector expuesto como herramienta nativa de agente, más servidores MCP propios con aplicaciones OAuth de un clic |
| **Fuentes por Notebook** | 50 (gratis) a 600 (Ultra, $249.99/mes) | Ilimitadas |
| **Número de Notebooks** | 100 (gratis) a 500 (niveles de pago) | Ilimitado |
| **Límite de tamaño por fuente** | 500,000 palabras / 200MB por fuente | Sin límite |
| **Precios** | Nivel gratuito; Pro $19.99/mes, Ultra $249.99/mes | Gratuito y de código abierto para autoalojar; la nube es de pago por uso con $5 de crédito gratuito |
| **Soporte de LLM** | Solo Google Gemini | Más de 100 LLMs vía la especificación de OpenAI y LiteLLM |
| **Modelos de embeddings** | Solo Google | Más de 6,000 modelos de embeddings, todos los rerankers principales |
| **LLMs locales / privados** | No disponible | Soporte completo (vLLM, Ollama), tus datos siguen siendo tuyos |
| **Autoalojable** | No | Sí, con un solo comando de Docker o Docker Compose completo |
| **Código abierto** | No | Sí |
| **Fuentes de la base de conocimiento** | Google Drive, YouTube, sitios web | Subida de archivos, Google Drive, OneDrive, Dropbox, sincronización de carpetas locales y páginas rastreadas |
| **Formatos de archivo compatibles** | PDFs, Docs, Slides, Sheets, CSV, Word, EPUB, imágenes, URLs web, YouTube | Más de 50 formatos: documentos, imágenes, videos vía LlamaCloud, Unstructured o Docling (local) |
| **Búsqueda** | Búsqueda semántica | Híbrida semántica + texto completo con índices jerárquicos y fusión de rangos recíprocos |
| **Respuestas con citas** | Sí | Sí, respuestas citadas al estilo Perplexity |
| **Arquitectura agéntica** | No | Sí, impulsada por [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) con planificación, subagentes y acceso al sistema de archivos |
| **Automatizaciones y agentes de IA** | No | Flujos de trabajo programados, activadores por eventos y automatizaciones sin código creadas por chat con escritura en Notion, Slack, Linear y Jira |
| **Multijugador en tiempo real** | Notebooks compartidos con roles de Visualizador/Editor (sin chat en tiempo real) | RBAC con roles de Propietario / Administrador / Editor / Visualizador, chat en tiempo real e hilos de comentarios |
| **Generación de video** | Resúmenes de video cinematográficos vía Veo 3 (solo Ultra) | Disponible (NotebookLM es mejor aquí, en mejora activa) |
| **Generación de presentaciones** | Diapositivas más atractivas pero no editables | Presentaciones editables basadas en diapositivas |
| **Generación de podcasts** | Resúmenes de audio con presentadores e idiomas personalizables | Disponible con múltiples proveedores de TTS (NotebookLM es mejor aquí, en mejora activa) |
| **Aplicación de escritorio** | No | Aplicación nativa con General Assist, Quick Assist, Screenshot Assist y sincronización de carpetas locales |
<details>
<summary><b>Lista completa de Fuentes Externas</b></summary>
<a id="fuentes-externas"></a>
## Solicitudes de funciones y futuro
Motores de Búsqueda (SearXNG, Tavily, LinkUp, Baidu Search) · Google Drive · OneDrive · Dropbox · Slack · Microsoft Teams · Linear · Jira · ClickUp · Confluence · BookStack · Notion · Gmail · Videos de YouTube · GitHub · Discord · Airtable · Google Calendar · Luma · Circleback · Elasticsearch · Obsidian, y más por venir.
</details>
## SOLICITUDES DE FUNCIONES Y FUTURO
**SurfSense está en desarrollo activo.** Aunque aún no está listo para producción, puedes ayudarnos a acelerar el proceso.
**SurfSense está en desarrollo activo.** Aunque todavía no está listo para producción, puedes ayudarnos a acelerar el proceso.
¡Únete al [Discord de SurfSense](https://discord.gg/ejRNvftDp9) y ayuda a dar forma al futuro de SurfSense!
## Hoja de Ruta
## Hoja de ruta
¡Mantente al día con nuestro progreso de desarrollo y próximas funcionalidades!
Consulta nuestra hoja de ruta pública y contribuye con tus ideas o comentarios:
Mantente al día con nuestro progreso de desarrollo y las próximas funciones. Consulta nuestra hoja de ruta pública y aporta tus ideas o comentarios:
**Discusión de la Hoja de Ruta:** [SurfSense 2026 Roadmap](https://github.com/MODSetter/SurfSense/discussions/565)
**Discusión de la hoja de ruta:** [SurfSense 2026 Roadmap](https://github.com/MODSetter/SurfSense/discussions/565)
**Tablero Kanban:** [SurfSense Project Board](https://github.com/users/MODSetter/projects/3)
## Contribuye
## Contribuir
Todas las contribuciones son bienvenidas, desde estrellas y reportes de bugs hasta mejoras del backend. Consulta [CONTRIBUTING.md](CONTRIBUTING.md) para comenzar.
Todas las contribuciones son bienvenidas, desde estrellas y reportes de errores hasta mejoras del backend. Consulta [CONTRIBUTING.md](CONTRIBUTING.md) para comenzar.
Gracias a todos nuestros Surfers:
@ -310,7 +245,7 @@ Gracias a todos nuestros Surfers:
<img src="https://contrib.rocks/image?repo=MODSetter/SurfSense" />
</a>
## Historial de Stars
## Historial de estrellas
<a href="https://www.star-history.com/#MODSetter/SurfSense&Date">
<picture>

375
README.hi.md
View file

@ -1,4 +1,4 @@
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="readme_banner" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="SurfSense, AI एजेंट्स के लिए ओपन सोर्स कॉम्पिटिटिव इंटेलिजेंस प्लेटफ़ॉर्म" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
@ -20,297 +20,232 @@
<a href="https://trendshift.io/repositories/13606" target="_blank"><img src="https://trendshift.io/api/badge/repositories/13606" alt="MODSetter%2FSurfSense | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</div>
# SurfSense
# SurfSense: अपने AI एजेंट्स को दें कॉम्पिटिटिव इंटेलिजेंस
NotebookLM वहाँ उपलब्ध सबसे अच्छे और सबसे उपयोगी AI प्लेटफ़ॉर्म में से एक है, लेकिन जब आप इसे नियमित रूप से उपयोग करना शुरू करते हैं तो आप इसकी सीमाओं को भी महसूस करते हैं जो कुछ और की चाह छोड़ती हैं।
SurfSense **AI एजेंट्स के लिए ओपन सोर्स कॉम्पिटिटिव इंटेलिजेंस प्लेटफ़ॉर्म** है। आपके एजेंट प्रतिस्पर्धियों पर नज़र रखते हैं, रैंकिंग ट्रैक करते हैं, और **Reddit, YouTube, Google Maps, Google Search और ओपन वेब** से लाइव डेटा के साथ आपके बाज़ार की बात सुनते हैं, वह भी एक ही **REST API** या **MCP सर्वर** के ज़रिए। शेड्यूल्ड और इवेंट-ट्रिगर्ड एजेंट अपनी खोजों को ब्रीफ़ और अलर्ट में बदलते हैं, और एक बिल्ट-इन नॉलेज बेस हर खोज को साइटेशन के साथ खोजने योग्य बनाए रखता है।
1. एक notebook में जोड़े जा सकने वाले स्रोतों की मात्रा पर सीमाएं हैं।
2. आपके पास कितने notebooks हो सकते हैं इस पर सीमाएं हैं।
3. आपके पास ऐसे स्रोत नहीं हो सकते जो 500,000 शब्दों और 200MB से अधिक हों।
4. आप Google सेवाओं (LLMs, उपयोग मॉडल, आदि) में बंद हैं और उन्हें कॉन्फ़िगर करने का कोई विकल्प नहीं है।
5. सीमित बाहरी डेटा स्रोत और सेवा एकीकरण।
6. NotebookLM एजेंट विशेष रूप से केवल अध्ययन और शोध के लिए अनुकूलित है, लेकिन आप स्रोत डेटा के साथ और भी बहुत कुछ कर सकते हैं।
7. मल्टीप्लेयर सपोर्ट की कमी।
> [!NOTE]
> **📢 हमारे NotebookLM-विकल्प उपयोगकर्ताओं के लिए एक सूचना**
>
> पिछले कुछ महीनों में हमने SurfSense को आपके अपने ज्ञान के लिए सबसे बेहतरीन जनरल रिसर्च एजेंट के रूप में बनाया, और उस अध्याय ने हमें एक ऐसा समुदाय दिया जिस पर हमें सचमुच गर्व है। Claude, OpenCode, Hermes और OpenClaw जैसे एजेंटिक टूल्स ने अब साबित कर दिया है कि एजेंट ही भविष्य हैं, और जनरल रिसर्च अब हर सक्षम एजेंट की एक बुनियादी क्षमता बनती जा रही है। एजेंट्स के पास अब भी जिस चीज़ की कमी है वह है **लाइव मार्केट डेटा और उसके इर्द-गिर्द के वर्कफ़्लो**, इसलिए हम अपनी पूरी ऊर्जा वहीं लगा रहे हैं: निश्चित रूप से सबसे बेहतरीन ओपन सोर्स कॉम्पिटिटिव इंटेलिजेंस एजेंट प्लेटफ़ॉर्म बनना।
>
> **आप जिस भी चीज़ पर निर्भर हैं, वह कहीं नहीं जा रही।** आपका नॉलेज बेस, साइटेशन वाली चैट, रिपोर्ट, पॉडकास्ट, प्रेज़ेंटेशन, ऑटोमेशन और सहयोगी चैट, सब पहले की तरह काम करते रहेंगे, और सेल्फ-होस्टिंग मुफ़्त और ओपन सोर्स बनी रहेगी। पूरी घोषणा [हमारे changelog](https://www.surfsense.com/changelog) पर पढ़ें।
...और भी बहुत कुछ।
## एजेंट्स को SurfSense की ज़रूरत क्यों है
**SurfSense विशेष रूप से इन समस्याओं को हल करने के लिए बनाया गया है।** SurfSense आपको सक्षम बनाता है:
किसी भी सक्षम एजेंट से पूछिए "इस हफ़्ते प्रतिस्पर्धी क्या दाम ले रहे हैं?" या "लॉन्च के बाद से Reddit पर हमारे बारे में क्या कहा जा रहा है?" और उसके पास देखने के लिए कोई भरोसेमंद जगह नहीं होती। आधिकारिक प्लेटफ़ॉर्म API या तो रेट-लिमिटेड हैं, एंटरप्राइज़ के हिसाब से महंगे हैं, या हैं ही नहीं, और स्क्रैपिंग का ढांचा नाज़ुक होता है। SurfSense इसी कमी को पूरा करता है:
- **अपने डेटा प्रवाह को नियंत्रित करें** - अपने डेटा को निजी और सुरक्षित रखें।
- **कोई डेटा सीमा नहीं** - असीमित मात्रा में स्रोत और notebooks जोड़ें।
- **कोई विक्रेता लॉक-इन नहीं** - किसी भी LLM, इमेज, TTS और STT मॉडल को कॉन्फ़िगर करें।
- **25+ बाहरी डेटा स्रोत** - Google Drive, OneDrive, Dropbox, Notion और कई अन्य बाहरी सेवाओं से अपने स्रोत जोड़ें।
- **रीयल-टाइम मल्टीप्लेयर सपोर्ट** - एक साझा notebook में अपनी टीम के सदस्यों के साथ आसानी से काम करें।
- **AI ऑटोमेशन और एजेंट** - AI एजेंट को शेड्यूल पर चलाएं या जैसे ही कोई दस्तावेज़ किसी फ़ोल्डर में आए उसे ट्रिगर करें, फिर परिणाम वापस Notion, Slack, Linear और Drive में लिखें। चैट में बस वर्णन करके बिना-कोड ऑटोमेशन बनाएं।
- **डेस्कटॉप ऐप** - Quick Assist, General Assist, Screenshot Assist और लोकल फ़ोल्डर सिंक के साथ किसी भी एप्लिकेशन में AI सहायता प्राप्त करें।
- **प्लेटफ़ॉर्म-नेटिव कनेक्टर**, हर एक टाइप्ड REST एंडपॉइंट है जो स्ट्रक्चर्ड JSON लौटाता है। न रेट-लिमिट का जुआ, न HTML पार्सिंग।
- **एक MCP सर्वर** जो हर कनेक्टर को नेटिव टूल के रूप में (`reddit.scrape`, `google_search.scrape` और अन्य) Claude, Cursor या किसी भी एजेंट फ़्रेमवर्क को उपलब्ध कराता है।
- **एक एजेंट हार्नेस**, सिर्फ़ कच्चा डेटा नहीं: रीट्राई, स्ट्रक्चर्ड आउटपुट और क्रेडिट मीटरिंग बिल्ट-इन हैं, ताकि एजेंट सवाल से सीधे ब्रीफ़ तक पहुंच सकें और आपको ढांचा खुद न बनाना पड़े।
- **ओपन सोर्स और सेल्फ-होस्ट करने योग्य**, ताकि आपकी प्रतिस्पर्धी रिसर्च आपके अपने इन्फ्रास्ट्रक्चर पर ही रहे।
...और भी बहुत कुछ आने वाला है।
## लाइव डेटा कनेक्टर
| कनेक्टर | आपके एजेंट्स को क्या मिलता है | और जानें |
|---|---|---|
| **Reddit** | आधिकारिक API की रेट लिमिट के बिना पोस्ट, कमेंट और सबरेडिट स्ट्रीम | [Reddit Scraper API](https://www.surfsense.com/reddit) |
| **YouTube** | ब्रांड और प्रोडक्ट लिसनिंग के लिए वीडियो, ट्रांसक्रिप्ट और कमेंट थ्रेड | [YouTube Scraper API](https://www.surfsense.com/youtube) |
| **Google Maps** | स्थानीय प्रतिस्पर्धी और लीड रिसर्च के लिए स्थान, रेटिंग और रिव्यू | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
| **Google Search** | रैंक ट्रैकिंग और मार्केट मॉनिटरिंग के लिए लाइव SERP | [Google Search API](https://www.surfsense.com/google-search) |
| **Web Crawl** | ओपन वेब का कोई भी पेज साफ़-सुथरे, स्ट्रक्चर्ड कंटेंट के रूप में | [Web Crawling API](https://www.surfsense.com/web-crawl) |
| **MCP Connector** | कोई भी MCP सर्वर अपने एजेंट्स से जोड़ें, Notion, Slack, Jira और अन्य के लिए वन-क्लिक OAuth के साथ | [MCP Connector](https://www.surfsense.com/mcp-connector) |
बिलिंग पे-एज़-यू-गो है: कनेक्टर सिर्फ़ वास्तव में लौटाए गए हर आइटम पर बिल करते हैं, क्रॉल सफलतापूर्वक फ़ेच किए गए हर पेज पर, और असफल कॉल कभी बिल नहीं होतीं। सेल्फ-होस्टेड इंस्टॉल बिलिंग बंद रखकर चलते हैं। देखें [pricing](https://www.surfsense.com/pricing)।
## वीडियो एजेंट नमूना
## क्विक स्टार्ट
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
### कोड से कनेक्टर कॉल करें
हर कनेक्टर एक REST एंडपॉइंट है जिसे आप अपनी SurfSense API कुंजी के साथ किसी भी भाषा से कॉल कर सकते हैं:
```bash
curl -X POST "$SURFSENSE_API_URL/workspaces/$WORKSPACE_ID/scrapers/reddit/scrape" \
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"search_queries": ["your brand"],
"community": "webscraping",
"sort": "top",
"time_filter": "week"
}'
```
## पॉडकास्ट एजेंट नमूना
हर [कनेक्टर पेज](https://www.surfsense.com/connectors) पर Python, JavaScript, Go, PHP, Ruby, Java और C# में कॉपी-पेस्ट उदाहरण मौजूद हैं।
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
### MCP के ज़रिए ये टूल्स अपने एजेंट्स को दें
SurfSense MCP सर्वर को Claude, Cursor या अपने एजेंट फ़्रेमवर्क में जोड़ें:
## SurfSense का उपयोग कैसे करें
```json
{
"mcpServers": {
"surfsense": {
"url": "https://mcp.surfsense.com",
"headers": { "Authorization": "Bearer ${SURFSENSE_API_KEY}" }
}
}
}
```
### Cloud
अब आपका एजेंट हर कनेक्टर को नेटिव टूल के रूप में कॉल कर सकता है।
1. [surfsense.com](https://www.surfsense.com) पर जाएं और लॉगिन करें।
### क्लाउड इस्तेमाल करें
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/LoginFlowGif.gif" alt="लॉगिन" /></p>
[surfsense.com](https://www.surfsense.com) पर जाएं, लॉग इन करें, और एजेंट से सीधी-सादी भाषा में लाइव मार्केट डेटा मांगें। नए अकाउंट $5 के मुफ़्त क्रेडिट के साथ शुरू होते हैं, बिना किसी सब्सक्रिप्शन के।
2. अपने कनेक्टर जोड़ें और सिंक करें। कनेक्टर्स को अपडेट रखने के लिए आवधिक सिंकिंग सक्षम करें।
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="कनेक्टर्स" /></p>
### मुफ़्त में सेल्फ-होस्ट करें
3. जब तक कनेक्टर्स का डेटा इंडेक्स हो रहा है, दस्तावेज़ अपलोड करें।
पूरा प्लेटफ़ॉर्म, कनेक्टर, एजेंट, ऑटोमेशन और MCP सर्वर अपने इन्फ्रास्ट्रक्चर पर चलाएं। सेल्फ-होस्टेड इंस्टॉल बिलिंग बंद के साथ आते हैं, इसलिए स्क्रैपिंग, क्रॉलिंग और एजेंट रन की सीमा सिर्फ़ आपके हार्डवेयर और आपके द्वारा लाई गई मॉडल कुंजियों पर निर्भर करती है।
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="दस्तावेज़ अपलोड करें" /></p>
**आवश्यकताएं:** [Docker Desktop](https://www.docker.com/products/docker-desktop/) इंस्टॉल होना और चल रहा होना चाहिए।
4. सब कुछ इंडेक्स हो जाने के बाद, कुछ भी पूछें (उपयोग के मामले):
**डेस्कटॉप ऐप** (नीचे दी गई सभी सुविधाओं के अलावा नेटिव एक्स्ट्रा, कोई अलग सेट नहीं)
- General Assist: किसी भी ऐप्लिकेशन से ग्लोबल शॉर्टकट के ज़रिए SurfSense तुरंत खोलें।
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/general_assist.gif" alt="General Assist" /></p>
- Quick Assist: कहीं भी टेक्स्ट चुनें और AI से उसे समझाने, दोबारा लिखने या उस पर कार्रवाई करने को कहें।
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
- Screenshot Assist: अपनी स्क्रीन का कोई भी हिस्सा कैप्चर करें और AI से उसमें मौजूद चीज़ों के बारे में पूछें।
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/screenshot_assist.gif" alt="Screenshot Assist" /></p>
- Watch Local Folder: किसी लोकल फ़ोल्डर को अपने नॉलेज बेस के साथ अपने-आप सिंक करें। Obsidian vaults के लिए बढ़िया।
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/folder_watch.gif" alt="Watch Local Folder" /></p>
**डिलीवरेबल स्टूडियो**
- AI Report Generator: उद्धरण सहित रिसर्च रिपोर्ट बनाएं और PDF, DOCX, HTML, LaTeX, EPUB, ODT या सादे टेक्स्ट में एक्सपोर्ट करें।
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="AI Report Generator" /></p>
- AI Podcast Generator: किसी भी दस्तावेज़ या फ़ोल्डर को 20 सेकंड से भी कम में दो-होस्ट वाले AI पॉडकास्ट में बदलें।
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/PodcastGenGif.gif" alt="AI Podcast Generator" /></p>
- AI Presentation & Video Maker: अपने स्रोतों से एडिट करने योग्य स्लाइड डेक और नैरेटेड वीडियो बनाएं।
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/video_gen_gif.gif" alt="AI Presentation and Video Maker" /></p>
- AI Image Generator: अपनी चैट और दस्तावेज़ों से सीधे उच्च-गुणवत्ता वाली इमेज बनाएं।
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ImageGenGif.gif" alt="AI Image Generator" /></p>
- AI Resume Builder: अपने मौजूदा रिज़्यूमे को किसी भी जॉब डिस्क्रिप्शन के अनुसार ढालें और ATS को पार करें।
इस तरह के प्रॉम्प्ट आज़माएं:
- "मेरे रिज़्यूमे को इस जॉब डिस्क्रिप्शन के अनुसार ढालें ताकि वह ATS पार करे और इंटरव्यू दिलाए।"
- "इस जॉब पोस्टिंग के कीवर्ड्स से मिलान करके मेरे रिज़्यूमे को ATS के लिए ऑप्टिमाइज़ करें।"
- "इस भूमिका के लिए ज़रूरी स्किल्स को उभारने के लिए मेरे रिज़्यूमे के बुलेट पॉइंट फिर से लिखें।"
- "मेरे रिज़्यूमे की तुलना इस जॉब डिस्क्रिप्शन से करें और सुधारने योग्य कमियों की सूची दें।"
- "मेरे रिज़्यूमे और इस जॉब डिस्क्रिप्शन से मेल खाता एक कवर लेटर लिखें।"
**सर्च और चैट**
- Chat With Your PDFs & Docs: अपनी सभी फ़ाइलों पर सवाल पूछें और इनलाइन उद्धरणों के साथ जवाब पाएं।
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="Chat With Your PDFs and Docs" /></p>
- AI Search With Citations: अपने पूरे नॉलेज बेस में हाइब्रिड सेमांटिक और कीवर्ड सर्च।
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BSNCGif.gif" alt="AI Search With Citations" /></p>
- Collaborative AI Chat: अपनी टीम के साथ रियल टाइम में AI बातचीत पर काम करें।
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Collaborative AI Chat" /></p>
- Comments & Mentions: किसी भी AI संदेश पर टिप्पणी करें और टीम के साथियों को टैग करें।
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Comments and Mentions" /></p>
**कनेक्टर्स और इंटीग्रेशन**
- Connect & Sync Your Tools: Notion, Slack, Google Drive, Gmail, GitHub, Linear और 25+ स्रोतों को एक खोजने योग्य कॉर्पस में सिंक करें।
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="Connect and Sync Your Tools" /></p>
- Chat With Uploaded Files: PDF, Office दस्तावेज़, इमेज और ऑडियो अपलोड करें। तुरंत खोजने योग्य।
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="Chat With Uploaded Files" /></p>
- Connector Write-Back: एजेंट को परिणाम वापस Notion, Slack, Linear और Drive में पोस्ट करने दें।
इस तरह के प्रॉम्प्ट आज़माएं:
- "इस रिसर्च सारांश को मेरे Notion वर्कस्पेस में पोस्ट करें।"
- "इन मीटिंग एक्शन आइटम्स को हमारे टीम Slack चैनल पर भेजें।"
- "इस बग रिपोर्ट से एक Jira टिकट बनाएं।"
- "इस फ़ीचर अनुरोध से Linear में एक इश्यू खोलें।"
- "इस जनरेट की गई रिपोर्ट को Google Drive में एक डॉक के रूप में सेव करें।"
- Obsidian & Knowledge Base Sync: अपने Obsidian vault और व्यक्तिगत नॉलेज बेस को सिंक रखें।
**ऑटोमेशन**
- Scheduled AI Workflows: किसी एजेंट को शेड्यूल पर चलाएं: रोज़ाना ब्रीफ़, साप्ताहिक डाइजेस्ट, आवर्ती रिपोर्ट।
इस तरह के प्रॉम्प्ट आज़माएं:
- "हर सुबह मेरे नॉलेज बेस में जुड़े नए दस्तावेज़ों का रोज़ाना ब्रीफ़ मुझे ईमेल करें।"
- "हर शुक्रवार मेरे Slack और Gmail से एक साप्ताहिक स्टेटस रिपोर्ट बनाएं।"
- "एक मासिक प्रतिस्पर्धी विश्लेषण रिपोर्ट चलाएं और उसे मेरे वर्कस्पेस में सेव करें।"
- "मेरी GitHub और Linear गतिविधि को एक रोज़ाना standup अपडेट में सारांशित करें।"
- "मैं जिन विषयों को ट्रैक करता हूं उन पर एक आवर्ती साप्ताहिक रिसर्च रिपोर्ट बनाएं।"
- Event-Triggered Automations: जैसे ही कोई दस्तावेज़ किसी फ़ोल्डर में आता है, एजेंट को चलाएं और परिणाम अपने टूल में पोस्ट करें।
इस तरह के प्रॉम्प्ट आज़माएं:
- "जब मेरे Research फ़ोल्डर में कोई PDF आए, तो उद्धरण सहित एक AI सारांश बनाएं।"
- "जब नई मीटिंग नोट्स जुड़ें, तो उन्हें एक्शन आइटम्स के साथ मीटिंग मिनट्स में बदलें।"
- "जब कोई इनवॉइस अपलोड हो, तो विक्रेता, कुल राशि और देय तिथि को एक तालिका में निकालें।"
- "जब मेरे Legal फ़ोल्डर में कोई अनुबंध आए, तो मुख्य शर्तों और नवीनीकरण तिथियों को चिह्नित करें।"
- "जब Candidates में कोई रिज़्यूमे जुड़े, तो उसे जॉब डिस्क्रिप्शन के विरुद्ध स्क्रीन करें।"
- Chat-Built Automations: सरल भाषा में किसी ऑटोमेशन का वर्णन करें और SurfSense उसे आपके लिए बना देगा।
इस तरह के प्रॉम्प्ट आज़माएं:
- "एक AI एजेंट बनाएं जो हर सुबह नई Notion पेजों का सारांश मुझे ईमेल करे।"
- "एक नो-कोड ऑटोमेशन बनाएं जो हर सप्ताह एक रिसर्च डाइजेस्ट Slack पर पोस्ट करे।"
- "एक AI नोट-टेकर सेट करें जो नई मीटिंग नोट्स को मिनट्स में बदल दे।"
- "एक वर्कफ़्लो बनाएं जो मीटिंग नोट्स से एक्शन आइटम्स निकाले और ज़िम्मेदार सौंपे।"
- "मेरे Gmail और Google Drive से एक रोज़ाना ईमेल ब्रीफ़ को ऑटोमेट करें।"
### सेल्फ-होस्टेड
पूर्ण डेटा नियंत्रण और गोपनीयता के लिए SurfSense को अपने स्वयं के बुनियादी ढांचे पर चलाएं।
**आवश्यकताएँ:** [Docker Desktop](https://www.docker.com/products/docker-desktop/) इंस्टॉल और चालू होना चाहिए।
#### Linux/MacOS उपयोगकर्ताओं के लिए:
Linux/macOS के लिए:
```bash
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash
```
#### Windows उपयोगकर्ताओं के लिए:
Windows के लिए:
```powershell
```bash
irm https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.ps1 | iex
```
इंस्टॉल स्क्रिप्ट दैनिक ऑटो-अपडेट के लिए स्वचालित रूप से [Watchtower](https://github.com/nicholas-fedor/watchtower) सेटअप करती है। इसे छोड़ने के लिए, `--no-watchtower` फ्लैग जोड़ें।
इंस्टॉल स्क्रिप्ट दैनिक ऑटो-अपडेट के लिए [Watchtower](https://github.com/nicholas-fedor/watchtower) अपने आप सेट कर देती है। इसे छोड़ने के लिए `--no-watchtower` फ़्लैग जोड़ें। Docker Compose, मैनुअल इंस्टॉलेशन और अन्य डिप्लॉयमेंट विकल्पों के लिए [docs](https://www.surfsense.com/docs/) देखें।
Docker Compose, मैनुअल इंस्टॉलेशन और अन्य डिप्लॉयमेंट विकल्पों के लिए, [डॉक्स](https://www.surfsense.com/docs/) देखें।
## कॉम्पिटिटिव इंटेलिजेंस वर्कफ़्लो
### डेस्कटॉप ऐप
ऑटोमेशन शेड्यूल पर या इवेंट के जवाब में पूरे एजेंट टर्न चलाते हैं, फिर नतीजे Notion, Slack, Linear और Jira में वापस लिख देते हैं। वर्कफ़्लो को सीधी-सादी भाषा में बताइए और SurfSense उसे बना देगा, कोई कोड नहीं चाहिए। ये प्रॉम्प्ट आज़माएं:
SurfSense एक डेस्कटॉप ऐप भी प्रदान करता है जो आपके कंप्यूटर पर हर एप्लिकेशन में AI सहायता लाता है। इसे [नवीनतम रिलीज़](https://github.com/MODSetter/SurfSense/releases/latest) से डाउनलोड करें।
- "हमारे टॉप 3 प्रतिस्पर्धियों के प्राइसिंग पेज पर नज़र रखो और जब कोई प्लान या कीमत बदले तो मुझे Slack में अलर्ट करो।"
- "Reddit और YouTube पर हमारे ब्रांड के हर उल्लेख को ट्रैक करो और मुझे रोज़ाना डाइजेस्ट भेजो।"
- "हमारे टॉप 10 कीवर्ड के लिए हमारी Google रैंकिंग मॉनिटर करो और हफ़्ते-दर-हफ़्ते गिरावट को फ़्लैग करो।"
- "हर सोमवार हमारी लोकेशनों और हमारे प्रतिस्पर्धियों की नई Google Maps रिव्यू खींचो।"
- "हर महीने एक प्रतिस्पर्धी विश्लेषण रिपोर्ट चलाओ और उसे मेरे वर्कस्पेस में सेव करो।"
डेस्कटॉप ऐप में ये शक्तिशाली सुविधाएं शामिल हैं:
- **General Assist** — एक ग्लोबल शॉर्टकट से किसी भी एप्लिकेशन से तुरंत SurfSense लॉन्च करें।
- **Quick Assist** — कहीं भी टेक्स्ट चुनें, फिर AI से समझाने, फिर से लिखने या उस पर कार्रवाई करने को कहें।
- **Screenshot Assist** — स्क्रीन पर एक क्षेत्र चुनें और उसे चैट में जोड़ें, ताकि उत्तर आपकी नॉलेज बेस पर आधारित रहें।
- **Watch Local Folder** — एक लोकल फ़ोल्डर को वॉच करें और फ़ाइल परिवर्तनों को स्वचालित रूप से अपनी नॉलेज बेस में सिंक करें। **Pro tip:** इसे अपने Obsidian vault पर पॉइंट करें ताकि आपके नोट्स SurfSense में सर्च करने योग्य रहें।
## बॉक्स में बाकी सब कुछ
सभी सुविधाएं आपके चुने हुए सर्च स्पेस पर काम करती हैं, ताकि आपके उत्तर हमेशा आपके अपने डेटा पर आधारित हों।
जिस रिसर्च वर्कस्पेस ने SurfSense को अग्रणी ओपन सोर्स NotebookLM विकल्प बनाया, वह अब भी यहीं है, और आपके एजेंट जो कुछ भी इकट्ठा करते हैं वह सब इसी में पहुंचता है।
### रीयल-टाइम सहयोग कैसे करें (बीटा)
**नॉलेज बेस**
1. सदस्य प्रबंधन पेज पर जाएं और एक आमंत्रण बनाएं।
- PDF, Office दस्तावेज़, इमेज और ऑडियो अपलोड करें, या **Google Drive, OneDrive और Dropbox** सिंक करें। 50+ फ़ाइल फ़ॉर्मैट समर्थित हैं।
- हाइब्रिड सिमेंटिक और फ़ुल-टेक्स्ट सर्च, Perplexity-शैली के साइटेड जवाबों के साथ।
- AI फ़ाइल सॉर्टिंग दस्तावेज़ों को स्रोत, तारीख़ और विषय के अनुसार अपने आप व्यवस्थित करती है।
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="अपनी PDF और दस्तावेज़ों से चैट करें" /></p>
**डिलीवरेबल स्टूडियो**
- AI रिपोर्ट जनरेटर, PDF, DOCX, HTML, LaTeX, EPUB, ODT या सादे टेक्स्ट में एक्सपोर्ट के साथ।
- किसी भी दस्तावेज़ या फ़ोल्डर से 20 सेकंड से कम में दो-होस्ट वाले AI पॉडकास्ट।
- संपादन योग्य स्लाइड डेक, नैरेटेड वीडियो ओवरव्यू और AI इमेज जनरेशन।
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="AI रिपोर्ट जनरेटर" /></p>
**टीम सहयोग**
- कमेंट और मेंशन के साथ रीयल-टाइम सहयोगी AI चैट।
- Owner, Admin, Editor और Viewer भूमिकाओं के साथ RBAC।
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="सहयोगी AI चैट" /></p>
**डेस्कटॉप ऐप**
आपके कंप्यूटर के हर एप्लिकेशन में नेटिव AI सहायता। [latest release](https://github.com/MODSetter/SurfSense/releases/latest) से डाउनलोड करें।
- **General Assist**: ग्लोबल शॉर्टकट से किसी भी ऐप से SurfSense लॉन्च करें।
- **Quick Assist**: कहीं भी टेक्स्ट चुनें, फिर AI से उसे समझाने, फिर से लिखने या उस पर कार्रवाई करने को कहें।
- **Screenshot Assist**: अपनी स्क्रीन का कोई भी हिस्सा कैप्चर करें और AI से उसके बारे में पूछें।
- **Watch Local Folder**: किसी लोकल फ़ोल्डर को अपने नॉलेज बेस से ऑटो-सिंक करें। इसे अपने Obsidian वॉल्ट पर पॉइंट करें ताकि आपके नोट्स खोजने योग्य बने रहें।
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
**कोई वेंडर लॉक-इन नहीं**
- OpenAI स्पेक और LiteLLM के ज़रिए 100+ LLM, जिनमें GPT-5.5, Claude Sonnet 5 और Gemini 3.1 Pro शामिल हैं।
- 6,000+ एम्बेडिंग मॉडल और सभी प्रमुख रीरैंकर।
- पूर्ण लोकल और प्राइवेट LLM समर्थन (vLLM, Ollama), ताकि आपका डेटा आपका ही रहे।
## वीडियो एजेंट नमूना
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
## पॉडकास्ट एजेंट नमूना
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
## रीयल टाइम में सहयोग कैसे करें (बीटा)
1. Manage Members पेज पर जाएं और एक इनवाइट बनाएं।
<p align="center"><img src="https://github.com/user-attachments/assets/40ed7683-5aa6-48a0-a3df-00575528c392" alt="सदस्यों को आमंत्रित करें" /></p>
2. टीममेट जुड़ता है और वह SearchSpace साझा हो जाता है।
2. कोई साथी जुड़ता है और वह वर्कस्पेस साझा हो जाता है।
<p align="center"><img src="https://github.com/user-attachments/assets/ea4e1057-4d2b-4fd2-9ca0-cd19286a285e" alt="आमंत्रण स्वीकार प्रवाह" /></p>
<p align="center"><img src="https://github.com/user-attachments/assets/ea4e1057-4d2b-4fd2-9ca0-cd19286a285e" alt="इनवाइट जॉइन फ़्लो" /></p>
3. चैट को साझा करें।
3. किसी चैट को साझा करें और उसमें रीयल टाइम में साथ काम करें, साथियों को टैग करने के लिए कमेंट के साथ।
<p align="center"><img src="https://github.com/user-attachments/assets/17b93904-0888-4c3a-ac12-51a24a8ea26a" alt="चैट साझा करें" /></p>
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="रीयलटाइम कमेंट" /></p>
4. आपकी टीम अब रीयल-टाइम में चैट कर सकती है।
## SurfSense बनाम Google NotebookLM
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="रीयल-टाइम चैट" /></p>
अब भी हमें NotebookLM विकल्प के तौर पर तौल रहे हैं? यह रहा ईमानदार तुलनात्मक ब्यौरा।
5. टीममेट्स को टैग करने के लिए कमेंट जोड़ें।
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="रीयल-टाइम कमेंट्स" /></p>
## SurfSense vs Google NotebookLM
| विशेषता | Google NotebookLM | SurfSense |
| फ़ीचर | Google NotebookLM | SurfSense |
|---------|-------------------|-----------|
| **प्रति Notebook स्रोत** | 50 (मुफ़्त) से 600 (Ultra, $249.99/माह) | असीमित |
| **Notebooks की संख्या** | 100 (मुफ़्त) से 500 (सशुल्क योजनाएं) | असीमित |
| **एजेंट्स के लिए लाइव मार्केट डेटा** | नहीं | REST API और MCP के ज़रिए Reddit, YouTube, Google Maps, Google Search और वेब क्रॉल कनेक्टर |
| **MCP सर्वर** | नहीं | हर कनेक्टर नेटिव एजेंट टूल के रूप में उपलब्ध, साथ ही वन-क्लिक OAuth ऐप्स के साथ अपने MCP सर्वर लाने की सुविधा |
| **प्रति नोटबुक स्रोत** | 50 (Free) से 600 (Ultra, $249.99/माह) | असीमित |
| **नोटबुक की संख्या** | 100 (Free) से 500 (सशुल्क टियर) | असीमित |
| **स्रोत आकार सीमा** | 500,000 शब्द / 200MB प्रति स्रोत | कोई सीमा नहीं |
| **मूल्य निर्धारण** | मुफ़्त स्तर उपलब्ध; Pro $19.99/माह, Ultra $249.99/माह | मुफ़्त और ओपन सोर्स, अपनी इंफ्रा पर सेल्फ-होस्ट करें |
| **LLM सपोर्ट** | केवल Google Gemini | 100+ LLMs OpenAI spec और LiteLLM के माध्यम से |
| **एम्बेडिंग मॉडल** | केवल Google | 6,000+ एम्बेडिंग मॉडल, सभी प्रमुख रीरैंकर्स |
| **लोकल / प्राइवेट LLMs** | उपलब्ध नहीं | पूर्ण सपोर्ट (vLLM, Ollama) - आपका डेटा आपका रहता है |
| **सेल्फ-होस्ट करने योग्य** | नहीं | हाँ - Docker एक कमांड या पूर्ण Docker Compose |
| **ओपन सोर्स** | नहीं | हाँ |
| **बाहरी कनेक्टर्स** | Google Drive, YouTube, वेबसाइटें | 27+ कनेक्टर्स - सर्च इंजन, Google Drive, OneDrive, Dropbox, Slack, Teams, Jira, Notion, GitHub, Discord और [अधिक](#बाहरी-स्रोत) |
| **फ़ाइल फ़ॉर्मेट सपोर्ट** | PDFs, Docs, Slides, Sheets, CSV, Word, EPUB, इमेज, वेब URLs, YouTube | 50+ फ़ॉर्मेट - दस्तावेज़, इमेज, वीडियो LlamaCloud, Unstructured या Docling (लोकल) के माध्यम से |
| **सर्च** | सिमैंटिक सर्च | हाइब्रिड सर्च - हायरार्किकल इंडाइसेस और Reciprocal Rank Fusion के साथ सिमैंटिक + फुल टेक्स्ट |
| **उद्धृत उत्तर** | हाँ | हाँ - Perplexity शैली के उद्धृत उत्तर |
| **एजेंट आर्किटेक्चर** | नहीं | हाँ - [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) द्वारा संचालित, योजना, सब-एजेंट और फ़ाइल सिस्टम एक्सेस |
| **रीयल-टाइम मल्टीप्लेयर** | दर्शक/संपादक भूमिकाओं के साथ साझा notebooks (कोई रीयल-टाइम चैट नहीं) | मालिक / एडमिन / संपादक / दर्शक भूमिकाओं के साथ RBAC, रीयल-टाइम चैट और कमेंट थ्रेड |
| **वीडियो जनरेशन** | Veo 3 के माध्यम से सिनेमैटिक वीडियो ओवरव्यू (केवल Ultra) | उपलब्ध (NotebookLM यहाँ बेहतर है, सक्रिय रूप से सुधार हो रहा है) |
| **प्रेजेंटेशन जनरेशन** | बेहतर दिखने वाली स्लाइड्स लेकिन संपादन योग्य नहीं | संपादन योग्य, स्लाइड आधारित प्रेजेंटेशन बनाएं |
| **पॉडकास्ट जनरेशन** | कस्टमाइज़ेबल होस्ट और भाषाओं के साथ ऑडियो ओवरव्यू | कई TTS प्रदाताओं के साथ उपलब्ध (NotebookLM यहाँ बेहतर है, सक्रिय रूप से सुधार हो रहा है) |
| **AI ऑटोमेशन और एजेंट** | नहीं | शेड्यूल किए गए AI वर्कफ़्लो, नए दस्तावेज़ों पर इवेंट ट्रिगर, और चैट से बने बिना-कोड ऑटोमेशन, Notion, Slack, Linear और Jira में कनेक्टर राइट-बैक के साथ |
| **मूल्य निर्धारण** | Free टियर; Pro $19.99/माह, Ultra $249.99/माह | सेल्फ-होस्ट के लिए मुफ़्त और ओपन सोर्स; क्लाउड पे-एज़-यू-गो है, $5 मुफ़्त क्रेडिट के साथ |
| **LLM समर्थन** | केवल Google Gemini | OpenAI स्पेक और LiteLLM के ज़रिए 100+ LLM |
| **एम्बेडिंग मॉडल** | केवल Google | 6,000+ एम्बेडिंग मॉडल, सभी प्रमुख रीरैंकर |
| **लोकल / प्राइवेट LLM** | उपलब्ध नहीं | पूर्ण समर्थन (vLLM, Ollama), आपका डेटा आपका ही रहता है |
| **सेल्फ-होस्ट करने योग्य** | नहीं | हां, Docker वन-लाइनर या पूर्ण Docker Compose |
| **ओपन सोर्स** | नहीं | हां |
| **नॉलेज बेस स्रोत** | Google Drive, YouTube, वेबसाइट | फ़ाइल अपलोड, Google Drive, OneDrive, Dropbox, लोकल फ़ोल्डर सिंक और क्रॉल किए गए पेज |
| **फ़ाइल फ़ॉर्मैट समर्थन** | PDF, Docs, Slides, Sheets, CSV, Word, EPUB, इमेज, वेब URL, YouTube | 50+ फ़ॉर्मैट: LlamaCloud, Unstructured या Docling (लोकल) के ज़रिए दस्तावेज़, इमेज, वीडियो |
| **सर्च** | सिमेंटिक सर्च | हाइरार्किकल इंडेक्स और रेसिप्रोकल रैंक फ़्यूज़न के साथ हाइब्रिड सिमेंटिक + फ़ुल-टेक्स्ट |
| **साइटेड जवाब** | हां | हां, Perplexity-शैली के साइटेड जवाब |
| **एजेंटिक आर्किटेक्चर** | नहीं | हां, [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) द्वारा संचालित, प्लानिंग, सबएजेंट और फ़ाइल सिस्टम एक्सेस के साथ |
| **AI ऑटोमेशन और एजेंट** | नहीं | शेड्यूल्ड वर्कफ़्लो, इवेंट ट्रिगर और चैट से बने नो-कोड ऑटोमेशन, Notion, Slack, Linear और Jira में राइट-बैक के साथ |
| **रीयल-टाइम मल्टीप्लेयर** | Viewer/Editor भूमिकाओं के साथ साझा नोटबुक (रीयल-टाइम चैट नहीं) | Owner / Admin / Editor / Viewer भूमिकाओं के साथ RBAC, रीयल-टाइम चैट और कमेंट थ्रेड |
| **वीडियो जनरेशन** | Veo 3 के ज़रिए सिनेमैटिक वीडियो ओवरव्यू (केवल Ultra) | उपलब्ध (यहां NotebookLM बेहतर है, सक्रिय रूप से सुधार जारी) |
| **प्रेज़ेंटेशन जनरेशन** | दिखने में बेहतर स्लाइड, लेकिन संपादन योग्य नहीं | संपादन योग्य, स्लाइड-आधारित प्रेज़ेंटेशन |
| **पॉडकास्ट जनरेशन** | कस्टमाइज़ करने योग्य होस्ट और भाषाओं के साथ ऑडियो ओवरव्यू | कई TTS प्रोवाइडर के साथ उपलब्ध (यहां NotebookLM बेहतर है, सक्रिय रूप से सुधार जारी) |
| **डेस्कटॉप ऐप** | नहीं | General Assist, Quick Assist, Screenshot Assist और लोकल फ़ोल्डर सिंक के साथ नेटिव ऐप |
| **ब्राउज़र एक्सटेंशन** | नहीं | किसी भी वेबपेज को सहेजने के लिए क्रॉस-ब्राउज़र एक्सटेंशन, प्रमाणीकरण सुरक्षित पेज सहित |
<details>
<summary><b>बाहरी स्रोतों की पूरी सूची</b></summary>
<a id="बाहरी-स्रोत"></a>
सर्च इंजन (SearXNG, Tavily, LinkUp, Baidu Search) · Google Drive · OneDrive · Dropbox · Slack · Microsoft Teams · Linear · Jira · ClickUp · Confluence · BookStack · Notion · Gmail · YouTube वीडियो · GitHub · Discord · Airtable · Google Calendar · Luma · Circleback · Elasticsearch · Obsidian, और भी बहुत कुछ आने वाला है।
</details>
## फ़ीचर अनुरोध और भविष्य
**SurfSense सक्रिय रूप से विकसित किया जा रहा है।** हालांकि यह अभी प्रोडक्शन-रेडी नहीं है, आप इस प्रक्रिया को तेज़ करने में हमारी मदद कर सकते हैं।
**SurfSense सक्रिय रूप से विकसित किया जा रहा है।** हालांकि यह अभी प्रोडक्शन-रेडी नहीं है, आप प्रक्रिया को तेज़ करने में हमारी मदद कर सकते हैं।
[SurfSense Discord](https://discord.gg/ejRNvftDp9) में शामिल हों और SurfSense के भविष्य को आकार देने में मदद करें!
[SurfSense Discord](https://discord.gg/ejRNvftDp9) से जुड़ें और SurfSense का भविष्य गढ़ने में मदद करें!
## रोडमैप
हमारे विकास की प्रगति और आने वाली सुविधाओं से अपडेट रहें!
हमारा सार्वजनिक रोडमैप देखें और अपने विचार या फ़ीडबैक दें:
हमारी विकास प्रगति और आने वाले फ़ीचर्स से अपडेट रहें। हमारा सार्वजनिक रोडमैप देखें और अपने विचार या फ़ीडबैक साझा करें:
**रोडमैप चर्चा:** [SurfSense 2026 Roadmap](https://github.com/MODSetter/SurfSense/discussions/565)
**कानबन बोर्ड:** [SurfSense Project Board](https://github.com/users/MODSetter/projects/3)
**कानबान बोर्ड:** [SurfSense Project Board](https://github.com/users/MODSetter/projects/3)
## योगदान करें
सभी योगदान स्वागत योग्य हैं, स्टार और बग रिपोर्ट से लेकर बैकएंड सुधार तक। शुरू करने के लिए [CONTRIBUTING.md](CONTRIBUTING.md) देखें।
हर तरह का योगदान स्वागत योग्य है, स्टार और बग रिपोर्ट से लेकर बैकएंड सुधार तक। शुरू करने के लिए [CONTRIBUTING.md](CONTRIBUTING.md) देखें।
हमारे सभी Surfers को धन्यवाद:
हमारे सभी Surfers का धन्यवाद:
<a href="https://github.com/MODSetter/SurfSense/graphs/contributors">
<img src="https://contrib.rocks/image?repo=MODSetter/SurfSense" />
</a>
## Star इतिहास
## स्टार हिस्ट्री
<a href="https://www.star-history.com/#MODSetter/SurfSense&Date">
<picture>

339
README.md
View file

@ -1,4 +1,4 @@
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="readme_banner" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="SurfSense, the open-source competitive intelligence platform for AI agents" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
@ -20,271 +20,206 @@
<a href="https://trendshift.io/repositories/13606" target="_blank"><img src="https://trendshift.io/api/badge/repositories/13606" alt="MODSetter%2FSurfSense | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</div>
# SurfSense
# SurfSense: Give Your AI Agents Competitive Intelligence
NotebookLM is one of the best and most useful AI platforms out there, but once you start using it regularly you also feel its limitations leaving something to be desired more.
SurfSense is the **open-source competitive intelligence platform for AI agents**. Your agents monitor competitors, track rankings, and listen to your market with live data from **Reddit, YouTube, Google Maps, Google Search, and the open web**, through one **REST API** or **MCP server**. Scheduled and event-triggered agents turn what they find into briefs and alerts, and a built-in knowledge base keeps every finding searchable with citations.
1. There are limits on the amount of sources you can add in a notebook.
2. There are limits on the number of notebooks you can have.
3. You cannot have sources that exceed 500,000 words and are more than 200MB.
4. You are vendor locked in to Google services (LLMs, usage models, etc.) with no option to configure them.
5. Limited external data sources and service integrations.
6. NotebookLM Agent is specifically optimised for just studying and researching, but you can do so much more with the source data.
7. Lack of multiplayer support.
> [!NOTE]
> **📢 A note for our NotebookLM-alternative users**
>
> For the past couple of months we built SurfSense as the best general research agent for your own knowledge, and that chapter earned us a community we are genuinely proud of. Agentic tools like Claude, OpenCode, Hermes, and OpenClaw have now proven that agents are the future, and general research is becoming something every capable agent does out of the box. What agents still lack is **live market data and the workflows around it**, so that is where we are pointing all of our energy: becoming the definitive open-source competitive intelligence agent platform.
>
> **Nothing you rely on is going away.** Your knowledge base, chat with citations, reports, podcasts, presentations, automations, and collaborative chats all keep working, and self-hosting stays free and open source. Read the full announcement on [our changelog](https://www.surfsense.com/changelog).
...and more.
## Why agents need SurfSense
**SurfSense is specifically made to solve these problems.** SurfSense empowers you to:
Ask any capable agent "what are competitors charging this week?" or "what is Reddit saying about us since the launch?" and it has nowhere trustworthy to look. Official platform APIs are rate-limited, priced for enterprises, or missing entirely, and scraping plumbing is brittle. SurfSense closes that gap:
- **Control Your Data Flow** - Keep your data private and secure.
- **No Data Limits** - Add an unlimited amount of sources and notebooks.
- **No Vendor Lock-in** - Configure any LLM, image, TTS, and STT models to use.
- **25+ External Data Sources** - Add your sources from Google Drive, OneDrive, Dropbox, Notion, and many other external services.
- **Real-Time Multiplayer Support** - Work easily with your team members in a shared notebook.
- **AI Automations & Agents** - Run AI agents on a schedule or trigger them the moment a document lands in a folder, then write results back to Notion, Slack, Linear, and Drive. Build no-code automations just by describing them in chat.
- **Desktop App** - Get AI assistance in any application with Quick Assist, General Assist, Screenshot Assist, and local folder sync.
- **Platform-native connectors**, each a typed REST endpoint returning structured JSON. No rate-limit roulette, no HTML parsing.
- **An MCP server** that exposes every connector as a native tool (`reddit.scrape`, `google_search.scrape`, and more) to Claude, Cursor, or any agent framework.
- **An agent harness**, not just raw data: retries, structured output, and credit metering are built in, so agents go from a question to a brief without you building the plumbing.
- **Open source and self-hostable**, so your competitive research stays on your own infrastructure.
...and more to come.
## Live data connectors
| Connector | What your agents get | Learn more |
|---|---|---|
| **Reddit** | Posts, comments, and subreddit streams without the official API's rate limits | [Reddit Scraper API](https://www.surfsense.com/reddit) |
| **YouTube** | Videos, transcripts, and comment threads for brand and product listening | [YouTube Scraper API](https://www.surfsense.com/youtube) |
| **Google Maps** | Places, ratings, and reviews for local competitor and lead research | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
| **Google Search** | Live SERPs for rank tracking and market monitoring | [Google Search API](https://www.surfsense.com/google-search) |
| **Web Crawl** | Any page on the open web as clean, structured content | [Web Crawling API](https://www.surfsense.com/web-crawl) |
| **MCP Connector** | Bring any MCP server to your agents, with one-click OAuth for Notion, Slack, Jira, and more | [MCP Connector](https://www.surfsense.com/mcp-connector) |
Billing is pay as you go: connectors bill per item actually returned, crawls per page successfully fetched, and failed calls are never billed. Self-hosted installs run with billing off. See [pricing](https://www.surfsense.com/pricing).
## Video Agent Sample
## Quick start
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
### Call a connector from code
Every connector is a REST endpoint you can call from any language with your SurfSense API key:
```bash
curl -X POST "$SURFSENSE_API_URL/workspaces/$WORKSPACE_ID/scrapers/reddit/scrape" \
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"search_queries": ["your brand"],
"community": "webscraping",
"sort": "top",
"time_filter": "week"
}'
```
## Podcast Agent Sample
Each [connector page](https://www.surfsense.com/connectors) has copy-paste examples in Python, JavaScript, Go, PHP, Ruby, Java, and C#.
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
### Give the tools to your agents over MCP
Add the SurfSense MCP server to Claude, Cursor, or your own agent framework:
## How to Use SurfSense
```json
{
"mcpServers": {
"surfsense": {
"url": "https://mcp.surfsense.com",
"headers": { "Authorization": "Bearer ${SURFSENSE_API_KEY}" }
}
}
}
```
### Cloud
Your agent can now call every connector as a native tool.
1. Go to [surfsense.com](https://www.surfsense.com) and login.
### Use the cloud
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/LoginFlowGif.gif" alt="Login" /></p>
Go to [surfsense.com](https://www.surfsense.com), log in, and ask the agent for live market data in plain English. New accounts start with $5 of free credit and no subscription.
2. Connect your connectors and sync. Enable periodic syncing to keep connectors synced.
### Self-host for free
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="Connectors" /></p>
3. Till connectors data index, upload Documents.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="Upload Documents" /></p>
4. Once everything is indexed, Ask Away (Use Cases):
**Desktop App** (native extras on top of everything below, not a separate feature set)
- General Assist: launch SurfSense instantly from any application with a global shortcut.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/general_assist.gif" alt="General Assist" /></p>
- Quick Assist: select text anywhere, then ask AI to explain, rewrite, or act on it.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
- Screenshot Assist: capture any region of your screen and ask AI about what's in it.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/screenshot_assist.gif" alt="Screenshot Assist" /></p>
- Watch Local Folder: auto-sync a local folder to your knowledge base. Great for Obsidian vaults.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/folder_watch.gif" alt="Watch Local Folder" /></p>
**Deliverable Studio**
- AI Report Generator: generate cited research reports and export to PDF, DOCX, HTML, LaTeX, EPUB, ODT, or plain text.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="AI Report Generator" /></p>
- AI Podcast Generator: turn any document or folder into a two-host AI podcast in under 20 seconds.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/PodcastGenGif.gif" alt="AI Podcast Generator" /></p>
- AI Presentation & Video Maker: create editable slide decks and narrated video overviews from your sources.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/video_gen_gif.gif" alt="AI Presentation and Video Maker" /></p>
- AI Image Generator: generate high-quality images straight from your chats and documents.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ImageGenGif.gif" alt="AI Image Generator" /></p>
- AI Resume Builder: tailor your existing resume to any job description and beat the ATS.
Try prompts like these:
- "Tailor my resume to this job description so it gets past ATS and lands an interview."
- "Optimize my resume for ATS by matching the keywords in this job posting."
- "Rewrite my resume bullet points to highlight the skills this role is asking for."
- "Compare my resume against this job description and list the gaps to fix."
- "Write a matching cover letter from my resume and this job description."
**Search & Chat**
- Chat With Your PDFs & Docs: ask questions across all your files and get answers with inline citations.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="Chat With Your PDFs and Docs" /></p>
- AI Search With Citations: hybrid semantic and keyword search across your entire knowledge base.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BSNCGif.gif" alt="AI Search With Citations" /></p>
- Collaborative AI Chat: work on AI conversations with your team in real time.
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Collaborative AI Chat" /></p>
- Comments & Mentions: comment and tag teammates on any AI message.
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Comments and Mentions" /></p>
**Connectors & Integrations**
- Connect & Sync Your Tools: sync Notion, Slack, Google Drive, Gmail, GitHub, Linear and 25+ sources into one searchable corpus.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="Connect and Sync Your Tools" /></p>
- Chat With Uploaded Files: drop in PDFs, Office docs, images and audio. Instantly searchable.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="Chat With Uploaded Files" /></p>
- Connector Write-Back: let the agent post results back to Notion, Slack, Linear and Drive.
Try prompts like these:
- "Post this research summary to my Notion workspace."
- "Send these meeting action items to our team Slack channel."
- "Create a Jira ticket from this bug report."
- "Open a Linear issue from this feature request."
- "Save this generated report to Google Drive as a doc."
- Obsidian & Knowledge Base Sync: keep your Obsidian vault and personal knowledge base in sync.
**Automations**
- Scheduled AI Workflows: run an agent on a schedule: daily briefs, weekly digests, recurring reports.
Try prompts like these:
- "Email me a daily brief of new documents in my knowledge base every morning."
- "Generate a weekly status report from my Slack and Gmail every Friday."
- "Run a monthly competitor analysis report and save it to my workspace."
- "Summarize my GitHub and Linear activity into a daily standup update."
- "Create a recurring weekly research report on the topics I track."
- Event-Triggered Automations: fire an agent the moment a document lands in a folder, then post the result to your tools.
Try prompts like these:
- "When a PDF lands in my Research folder, generate a cited AI summary."
- "When new meeting notes are added, turn them into meeting minutes with action items."
- "When an invoice is uploaded, extract the vendor, total, and due date into a table."
- "When a contract enters my Legal folder, flag key terms and renewal dates."
- "When a resume is added to Candidates, screen it against the job description."
- Chat-Built Automations: describe an automation in plain English and SurfSense builds it for you.
Try prompts like these:
- "Build an AI agent that emails me a summary of new Notion pages each morning."
- "Create a no-code automation that posts a weekly research digest to Slack."
- "Set up an AI note taker that turns new meeting notes into minutes."
- "Make a workflow that extracts action items from meeting notes and assigns owners."
- "Automate a daily email brief from my Gmail and Google Drive."
### Self Hosted
Run SurfSense on your own infrastructure for full data control and privacy.
Run the entire platform, connectors, agents, automations, and the MCP server on your own infrastructure. Self-hosted installs ship with billing off, so scraping, crawling, and agent runs are limited only by your hardware and the model keys you bring.
**Prerequisites:** [Docker Desktop](https://www.docker.com/products/docker-desktop/) must be installed and running.
#### For Linux/MacOS users:
For Linux/macOS:
```bash
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash
```
#### For Windows users:
For Windows:
```bash
irm https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.ps1 | iex
```
The install script sets up [Watchtower](https://github.com/nicholas-fedor/watchtower) automatically for daily auto-updates. To skip it, add the `--no-watchtower` flag.
The install script sets up [Watchtower](https://github.com/nicholas-fedor/watchtower) automatically for daily auto-updates. To skip it, add the `--no-watchtower` flag. For Docker Compose, manual installation, and other deployment options, see the [docs](https://www.surfsense.com/docs/).
For Docker Compose, manual installation, and other deployment options, see the [docs](https://www.surfsense.com/docs/).
## Competitive intelligence workflows
### Desktop App
Automations run full agent turns on a schedule or in response to events, then write results back to Notion, Slack, Linear, and Jira. Describe the workflow in plain English and SurfSense builds it, no code needed. Try prompts like:
SurfSense also ships a desktop app that brings AI assistance to every application on your computer. Download it from the [latest release](https://github.com/MODSetter/SurfSense/releases/latest).
- "Watch our top 3 competitors' pricing pages and alert me in Slack when a plan or price changes."
- "Track every mention of our brand on Reddit and YouTube and send me a daily digest."
- "Monitor our Google ranking for our top 10 keywords and flag drops week over week."
- "Pull new Google Maps reviews for our locations and our competitors' every Monday."
- "Run a monthly competitor analysis report and save it to my workspace."
The desktop app includes these powerful features:
## Everything else in the box
- **General Assist** — Launch SurfSense instantly from any application with a global shortcut.
- **Quick Assist** — Select text anywhere, then ask AI to explain, rewrite, or act on it.
- **Screenshot Assist** — Select a region on your screen and attach it to chat so answers stay grounded in your knowledge base.
- **Watch Local Folder** — Watch a local folder and automatically sync file changes to your knowledge base. **Pro tip:** Point it at your Obsidian vault to keep your notes searchable in SurfSense.
The research workspace that made SurfSense the leading open-source NotebookLM alternative is still here, and everything your agents gather lands in it.
All features operate against your chosen search space, so your answers are always grounded in your own data.
**Knowledge base**
### How to Realtime Collaborate (Beta)
- Upload PDFs, Office docs, images, and audio, or sync **Google Drive, OneDrive, and Dropbox**. 50+ file formats supported.
- Hybrid semantic and full-text search with cited, Perplexity-style answers.
- AI file sorting auto-organizes documents by source, date, and topic.
1. Go to Manage Members page and create an invite.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="Chat With Your PDFs and Docs" /></p>
**Deliverable studio**
- AI report generator with export to PDF, DOCX, HTML, LaTeX, EPUB, ODT, or plain text.
- Two-host AI podcasts from any document or folder in under 20 seconds.
- Editable slide decks, narrated video overviews, and AI image generation.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="AI Report Generator" /></p>
**Team collaboration**
- Real-time collaborative AI chats with comments and mentions.
- RBAC with Owner, Admin, Editor, and Viewer roles.
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Collaborative AI Chat" /></p>
**Desktop app**
Native AI assistance in every application on your computer. Download from the [latest release](https://github.com/MODSetter/SurfSense/releases/latest).
- **General Assist**: launch SurfSense from any app with a global shortcut.
- **Quick Assist**: select text anywhere, then ask AI to explain, rewrite, or act on it.
- **Screenshot Assist**: capture any region of your screen and ask AI about it.
- **Watch Local Folder**: auto-sync a local folder to your knowledge base. Point it at your Obsidian vault to keep your notes searchable.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
**No vendor lock-in**
- 100+ LLMs via the OpenAI spec and LiteLLM, including GPT-5.5, Claude Sonnet 5, and Gemini 3.1 Pro.
- 6,000+ embedding models and all major rerankers.
- Full local and private LLM support (vLLM, Ollama), so your data stays yours.
## Video Agent Sample
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
## Podcast Agent Sample
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
## How to collaborate in real time (Beta)
1. Go to the Manage Members page and create an invite.
<p align="center"><img src="https://github.com/user-attachments/assets/40ed7683-5aa6-48a0-a3df-00575528c392" alt="Invite Members" /></p>
2. Teammate joins and that SearchSpace becomes shared.
2. A teammate joins and that workspace becomes shared.
<p align="center"><img src="https://github.com/user-attachments/assets/ea4e1057-4d2b-4fd2-9ca0-cd19286a285e" alt="Invite Join Flow" /></p>
3. Make chat shared.
<p align="center"><img src="https://github.com/user-attachments/assets/17b93904-0888-4c3a-ac12-51a24a8ea26a" alt="Make Chat Shared" /></p>
4. Your team can now chat in realtime.
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Realtime Chat" /></p>
5. Add comment to tag teammates.
3. Make a chat shared and work in it together in real time, with comments to tag teammates.
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Realtime Comments" /></p>
## SurfSense vs Google NotebookLM
Still comparing us as a NotebookLM alternative? Here is the honest breakdown.
| Feature | Google NotebookLM | SurfSense |
|---------|-------------------|-----------|
| **Live market data for agents** | No | Reddit, YouTube, Google Maps, Google Search, and web crawl connectors via REST API and MCP |
| **MCP server** | No | Every connector exposed as a native agent tool, plus bring-your-own MCP servers with one-click OAuth apps |
| **Sources per Notebook** | 50 (Free) to 600 (Ultra, $249.99/mo) | Unlimited |
| **Number of Notebooks** | 100 (Free) to 500 (paid tiers) | Unlimited |
| **Source Size Limit** | 500,000 words / 200MB per source | No limit |
| **Pricing** | Free tier available; Pro $19.99/mo, Ultra $249.99/mo | Free and open source, self-host on your own infra |
| **Pricing** | Free tier; Pro $19.99/mo, Ultra $249.99/mo | Free and open source to self-host; cloud is pay as you go with $5 free credit |
| **LLM Support** | Google Gemini only | 100+ LLMs via OpenAI spec & LiteLLM |
| **Embedding Models** | Google only | 6,000+ embedding models, all major rerankers |
| **Local / Private LLMs** | Not available | Full support (vLLM, Ollama) - your data stays yours |
| **Self Hostable** | No | Yes - Docker one-liner or full Docker Compose |
| **Local / Private LLMs** | Not available | Full support (vLLM, Ollama), your data stays yours |
| **Self Hostable** | No | Yes, Docker one-liner or full Docker Compose |
| **Open Source** | No | Yes |
| **External Connectors** | Google Drive, YouTube, websites | 27+ connectors - Search Engines, Google Drive, OneDrive, Dropbox, Slack, Teams, Jira, Notion, GitHub, Discord & [more](#external-sources) |
| **File Format Support** | PDFs, Docs, Slides, Sheets, CSV, Word, EPUB, images, web URLs, YouTube | 50+ formats - documents, images, videos via LlamaCloud, Unstructured, or Docling (local) |
| **Search** | Semantic search | Hybrid Search - Semantic + Full Text with Hierarchical Indices & Reciprocal Rank Fusion |
| **Cited Answers** | Yes | Yes - Perplexity-style cited responses |
| **Agentic Architecture** | No | Yes - powered by [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) with planning, subagents, and file system access |
| **Knowledge Base Sources** | Google Drive, YouTube, websites | File uploads, Google Drive, OneDrive, Dropbox, local folder sync, and crawled pages |
| **File Format Support** | PDFs, Docs, Slides, Sheets, CSV, Word, EPUB, images, web URLs, YouTube | 50+ formats: documents, images, videos via LlamaCloud, Unstructured, or Docling (local) |
| **Search** | Semantic search | Hybrid semantic + full-text with hierarchical indices & reciprocal rank fusion |
| **Cited Answers** | Yes | Yes, Perplexity-style cited responses |
| **Agentic Architecture** | No | Yes, powered by [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) with planning, subagents, and file system access |
| **AI Automations & Agents** | No | Scheduled workflows, event triggers, and chat-built no-code automations with write-back to Notion, Slack, Linear & Jira |
| **Real-Time Multiplayer** | Shared notebooks with Viewer/Editor roles (no real-time chat) | RBAC with Owner / Admin / Editor / Viewer roles, real-time chat & comment threads |
| **Video Generation** | Cinematic Video Overviews via Veo 3 (Ultra only) | Available (NotebookLM is better here, actively improving) |
| **Presentation Generation** | Better looking slides but not editable | Create editable, slide-based presentations |
| **Presentation Generation** | Better looking slides but not editable | Editable, slide-based presentations |
| **Podcast Generation** | Audio Overviews with customizable hosts and languages | Available with multiple TTS providers (NotebookLM is better here, actively improving) |
| **AI Automations & Agents** | No | Scheduled AI workflows, event triggers on new documents, and chat-built no-code automations with connector write-back to Notion, Slack, Linear & Jira |
| **Desktop App** | No | Native app with General Assist, Quick Assist, Screenshot Assist, and local folder sync |
| **Browser Extension** | No | Cross-browser extension to save any webpage, including auth-protected pages |
<details>
<summary><b>Full list of External Sources</b></summary>
<a id="external-sources"></a>
Search Engines (SearXNG, Tavily, LinkUp, Baidu Search) · Google Drive · OneDrive · Dropbox · Slack · Microsoft Teams · Linear · Jira · ClickUp · Confluence · BookStack · Notion · Gmail · YouTube Videos · GitHub · Discord · Airtable · Google Calendar · Luma · Circleback · Elasticsearch · Obsidian, and more to come.
</details>
## FEATURE REQUESTS AND FUTURE
## Feature requests and future
**SurfSense is actively being developed.** While it's not yet production-ready, you can help us speed up the process.
@ -292,14 +227,12 @@ Join the [SurfSense Discord](https://discord.gg/ejRNvftDp9) and help shape the f
## Roadmap
Stay up to date with our development progress and upcoming features!
Check out our public roadmap and contribute your ideas or feedback:
Stay up to date with our development progress and upcoming features. Check out our public roadmap and contribute your ideas or feedback:
**Roadmap Discussion:** [SurfSense 2026 Roadmap](https://github.com/MODSetter/SurfSense/discussions/565)
**Kanban Board:** [SurfSense Project Board](https://github.com/users/MODSetter/projects/3)
## Contribute
All contributions welcome, from stars and bug reports to backend improvements. See [CONTRIBUTING.md](CONTRIBUTING.md) to get started.

373
README.pt-BR.md
View file

@ -1,4 +1,4 @@
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="readme_banner" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="SurfSense, a plataforma open source de inteligência competitiva para agentes de IA" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
@ -20,289 +20,224 @@
<a href="https://trendshift.io/repositories/13606" target="_blank"><img src="https://trendshift.io/api/badge/repositories/13606" alt="MODSetter%2FSurfSense | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</div>
# SurfSense
# SurfSense: Dê Inteligência Competitiva aos Seus Agentes de IA
O NotebookLM é uma das melhores e mais úteis plataformas de IA disponíveis, mas quando você começa a usá-lo regularmente também sente suas limitações deixando algo a desejar.
O SurfSense é a **plataforma open source de inteligência competitiva para agentes de IA**. Seus agentes monitoram concorrentes, acompanham rankings e escutam o seu mercado com dados ao vivo do **Reddit, YouTube, Google Maps, Google Search e da web aberta**, por meio de uma única **API REST** ou de um **servidor MCP**. Agentes agendados ou acionados por eventos transformam o que encontram em relatórios e alertas, e uma base de conhecimento integrada mantém cada descoberta pesquisável, com citações.
1. Há limites na quantidade de fontes que você pode adicionar em um notebook.
2. Há limites no número de notebooks que você pode ter.
3. Você não pode ter fontes que excedam 500.000 palavras e mais de 200MB.
4. Você fica preso aos serviços do Google (LLMs, modelos de uso, etc.) sem opção de configurá-los.
5. Fontes de dados externas e integrações de serviços limitadas.
6. O agente do NotebookLM é especificamente otimizado apenas para estudar e pesquisar, mas você pode fazer muito mais com os dados de origem.
7. Falta de suporte multiplayer.
> [!NOTE]
> **📢 Um recado para nossos usuários que buscavam uma alternativa ao NotebookLM**
>
> Nos últimos meses, construímos o SurfSense como o melhor agente de pesquisa geral para o seu próprio conhecimento, e esse capítulo nos rendeu uma comunidade da qual temos muito orgulho. Ferramentas agênticas como Claude, OpenCode, Hermes e OpenClaw já provaram que os agentes são o futuro, e a pesquisa geral está se tornando algo que todo agente capaz faz nativamente. O que ainda falta aos agentes são **dados de mercado ao vivo e os fluxos de trabalho em torno deles**, então é para lá que estamos direcionando toda a nossa energia: nos tornar a plataforma open source definitiva de agentes de inteligência competitiva.
>
> **Nada do que você usa vai deixar de existir.** Sua base de conhecimento, o chat com citações, os relatórios, os podcasts, as apresentações, as automações e os chats colaborativos continuam funcionando, e a auto-hospedagem segue gratuita e open source. Leia o anúncio completo no [nosso changelog](https://www.surfsense.com/changelog).
...e mais.
## Por que os agentes precisam do SurfSense
**O SurfSense foi feito especificamente para resolver esses problemas.** O SurfSense permite que você:
Pergunte a qualquer agente capaz "quanto os concorrentes estão cobrando esta semana?" ou "o que o Reddit está dizendo sobre nós desde o lançamento?" e ele não terá nenhum lugar confiável para procurar. As APIs oficiais das plataformas têm limites de requisições, preços voltados para empresas ou simplesmente não existem, e a infraestrutura de scraping é frágil. O SurfSense fecha essa lacuna:
- **Controle Seu Fluxo de Dados** - Mantenha seus dados privados e seguros.
- **Sem Limites de Dados** - Adicione uma quantidade ilimitada de fontes e notebooks.
- **Sem Dependência de Fornecedor** - Configure qualquer modelo LLM, de imagem, TTS e STT.
- **25+ Fontes de Dados Externas** - Adicione suas fontes do Google Drive, OneDrive, Dropbox, Notion e muitos outros serviços externos.
- **Suporte Multiplayer em Tempo Real** - Trabalhe facilmente com os membros da sua equipe em um notebook compartilhado.
- **Automações e Agentes de IA** - Execute agentes de IA em uma programação ou dispare-os no momento em que um documento chega a uma pasta, e escreva os resultados de volta no Notion, Slack, Linear e Drive. Crie automações sem código apenas descrevendo-as no chat.
- **Aplicativo Desktop** - Obtenha assistência de IA em qualquer aplicativo com Quick Assist, General Assist, Screenshot Assist e sincronização de pastas locais.
- **Conectores nativos de cada plataforma**, cada um sendo um endpoint REST tipado que retorna JSON estruturado. Sem roleta de limites de requisição, sem parsing de HTML.
- **Um servidor MCP** que expõe cada conector como uma ferramenta nativa (`reddit.scrape`, `google_search.scrape` e outras) para o Claude, o Cursor ou qualquer framework de agentes.
- **Um harness de agentes**, não apenas dados brutos: novas tentativas, saída estruturada e medição de créditos já vêm prontos, então os agentes vão de uma pergunta a um relatório sem que você precise construir a infraestrutura.
- **Open source e auto-hospedável**, para que sua pesquisa competitiva permaneça na sua própria infraestrutura.
...e mais por vir.
## Conectores de dados ao vivo
| Conector | O que seus agentes recebem | Saiba mais |
|---|---|---|
| **Reddit** | Posts, comentários e fluxos de subreddits sem os limites de requisição da API oficial | [Reddit Scraper API](https://www.surfsense.com/reddit) |
| **YouTube** | Vídeos, transcrições e threads de comentários para monitoramento de marca e produto | [YouTube Scraper API](https://www.surfsense.com/youtube) |
| **Google Maps** | Estabelecimentos, avaliações e reviews para pesquisa local de concorrentes e leads | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
| **Google Search** | SERPs ao vivo para acompanhamento de rankings e monitoramento de mercado | [Google Search API](https://www.surfsense.com/google-search) |
| **Web Crawl** (rastreamento web) | Qualquer página da web aberta como conteúdo limpo e estruturado | [Web Crawling API](https://www.surfsense.com/web-crawl) |
| **Conector MCP** | Traga qualquer servidor MCP para seus agentes, com OAuth em um clique para Notion, Slack, Jira e outros | [MCP Connector](https://www.surfsense.com/mcp-connector) |
A cobrança é por uso: os conectores cobram por item efetivamente retornado, os rastreamentos por página obtida com sucesso, e chamadas com falha nunca são cobradas. Instalações auto-hospedadas rodam com a cobrança desativada. Veja os [preços](https://www.surfsense.com/pricing).
## Exemplo de Agente de Vídeo
## Início rápido
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
### Chame um conector a partir do código
Cada conector é um endpoint REST que você pode chamar de qualquer linguagem com a sua chave de API do SurfSense:
```bash
curl -X POST "$SURFSENSE_API_URL/workspaces/$WORKSPACE_ID/scrapers/reddit/scrape" \
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"search_queries": ["your brand"],
"community": "webscraping",
"sort": "top",
"time_filter": "week"
}'
```
## Exemplo de Agente de Podcast
Cada [página de conector](https://www.surfsense.com/connectors) tem exemplos prontos para copiar e colar em Python, JavaScript, Go, PHP, Ruby, Java e C#.
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
### Entregue as ferramentas aos seus agentes via MCP
Adicione o servidor MCP do SurfSense ao Claude, ao Cursor ou ao seu próprio framework de agentes:
## Como Usar o SurfSense
```json
{
"mcpServers": {
"surfsense": {
"url": "https://mcp.surfsense.com",
"headers": { "Authorization": "Bearer ${SURFSENSE_API_KEY}" }
}
}
}
```
### Cloud
Seu agente agora pode chamar cada conector como uma ferramenta nativa.
1. Acesse [surfsense.com](https://www.surfsense.com) e faça login.
### Use a nuvem
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/LoginFlowGif.gif" alt="Login" /></p>
Acesse [surfsense.com](https://www.surfsense.com), faça login e peça ao agente dados de mercado ao vivo em linguagem natural. Contas novas começam com US$ 5 de crédito gratuito e sem assinatura.
2. Conecte seus conectores e sincronize. Ative a sincronização periódica para manter os conectores atualizados.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="Conectores" /></p>
### Auto-hospede gratuitamente
3. Enquanto os dados dos conectores são indexados, faça upload de documentos.
Rode a plataforma inteira, conectores, agentes, automações e o servidor MCP na sua própria infraestrutura. Instalações auto-hospedadas vêm com a cobrança desativada, então scraping, rastreamento e execuções de agentes são limitados apenas pelo seu hardware e pelas chaves de modelo que você trouxer.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="Upload de Documentos" /></p>
**Pré-requisitos:** o [Docker Desktop](https://www.docker.com/products/docker-desktop/) precisa estar instalado e em execução.
4. Quando tudo estiver indexado, pergunte o que quiser (Casos de uso):
**Aplicativo Desktop** (extras nativos, além de tudo o que está abaixo, não um conjunto separado)
- General Assist: abra o SurfSense instantaneamente de qualquer aplicativo com um atalho global.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/general_assist.gif" alt="General Assist" /></p>
- Quick Assist: selecione um texto em qualquer lugar e peça à IA para explicar, reescrever ou agir sobre ele.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
- Screenshot Assist: capture qualquer região da tela e pergunte à IA sobre o que está nela.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/screenshot_assist.gif" alt="Screenshot Assist" /></p>
- Watch Local Folder: sincronize automaticamente uma pasta local com sua base de conhecimento. Ótimo para cofres do Obsidian.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/folder_watch.gif" alt="Watch Local Folder" /></p>
**Estúdio de Entregáveis**
- AI Report Generator: gere relatórios de pesquisa com citações e exporte para PDF, DOCX, HTML, LaTeX, EPUB, ODT ou texto simples.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="AI Report Generator" /></p>
- AI Podcast Generator: transforme qualquer documento ou pasta em um podcast de IA com dois apresentadores em menos de 20 segundos.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/PodcastGenGif.gif" alt="AI Podcast Generator" /></p>
- AI Presentation & Video Maker: crie apresentações editáveis e vídeos narrados a partir das suas fontes.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/video_gen_gif.gif" alt="AI Presentation and Video Maker" /></p>
- AI Image Generator: gere imagens de alta qualidade diretamente das suas conversas e documentos.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ImageGenGif.gif" alt="AI Image Generator" /></p>
- AI Resume Builder: adapte seu currículo atual a qualquer descrição de vaga e supere o ATS.
Experimente prompts como estes:
- "Adapte meu currículo a esta descrição de vaga para passar pelo ATS e conseguir uma entrevista."
- "Otimize meu currículo para o ATS combinando as palavras-chave desta vaga."
- "Reescreva os tópicos do meu currículo para destacar as habilidades que esta vaga exige."
- "Compare meu currículo com esta descrição de vaga e liste as lacunas a corrigir."
- "Escreva uma carta de apresentação combinando com meu currículo e esta descrição de vaga."
**Busca e Chat**
- Chat With Your PDFs & Docs: faça perguntas sobre todos os seus arquivos e receba respostas com citações inline.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="Chat With Your PDFs and Docs" /></p>
- AI Search With Citations: busca híbrida semântica e por palavra-chave em toda a sua base de conhecimento.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BSNCGif.gif" alt="AI Search With Citations" /></p>
- Collaborative AI Chat: trabalhe em conversas de IA com sua equipe em tempo real.
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Collaborative AI Chat" /></p>
- Comments & Mentions: comente e marque colegas em qualquer mensagem de IA.
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Comments and Mentions" /></p>
**Conectores e Integrações**
- Connect & Sync Your Tools: sincronize Notion, Slack, Google Drive, Gmail, GitHub, Linear e mais de 25 fontes em um único acervo pesquisável.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="Connect and Sync Your Tools" /></p>
- Chat With Uploaded Files: envie PDFs, documentos do Office, imagens e áudio. Pesquisáveis instantaneamente.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="Chat With Uploaded Files" /></p>
- Connector Write-Back: deixe o agente publicar os resultados de volta no Notion, Slack, Linear e Drive.
Experimente prompts como estes:
- "Publique este resumo de pesquisa no meu espaço do Notion."
- "Envie estes itens de ação da reunião para o nosso canal do Slack."
- "Crie um ticket no Jira a partir deste relatório de bug."
- "Abra uma issue no Linear a partir desta solicitação de funcionalidade."
- "Salve este relatório gerado no Google Drive como um documento."
- Obsidian & Knowledge Base Sync: mantenha seu cofre do Obsidian e sua base de conhecimento pessoal sincronizados.
**Automações**
- Scheduled AI Workflows: execute um agente em uma programação: resumos diários, boletins semanais, relatórios recorrentes.
Experimente prompts como estes:
- "Envie-me todas as manhãs um resumo diário dos novos documentos na minha base de conhecimento."
- "Gere um relatório de status semanal a partir do meu Slack e Gmail toda sexta-feira."
- "Execute um relatório mensal de análise da concorrência e salve-o no meu espaço de trabalho."
- "Resuma minha atividade no GitHub e Linear em uma atualização diária de standup."
- "Crie um relatório de pesquisa semanal recorrente sobre os temas que acompanho."
- Event-Triggered Automations: dispare um agente no momento em que um documento chega a uma pasta e publique o resultado nas suas ferramentas.
Experimente prompts como estes:
- "Quando um PDF chegar à minha pasta de Pesquisa, gere um resumo com IA e citações."
- "Quando novas notas de reunião forem adicionadas, transforme-as em atas com itens de ação."
- "Quando uma fatura for enviada, extraia o fornecedor, o total e a data de vencimento em uma tabela."
- "Quando um contrato entrar na minha pasta Jurídica, sinalize os termos-chave e as datas de renovação."
- "Quando um currículo for adicionado a Candidatos, avalie-o em relação à descrição da vaga."
- Chat-Built Automations: descreva uma automação em linguagem simples e o SurfSense a cria para você.
Experimente prompts como estes:
- "Crie um agente de IA que me envie todas as manhãs um resumo das novas páginas do Notion."
- "Crie uma automação sem código que publique um resumo de pesquisa semanal no Slack."
- "Configure um anotador com IA que transforme as novas notas de reunião em atas."
- "Crie um fluxo que extraia os itens de ação das notas de reunião e atribua responsáveis."
- "Automatize um resumo diário por e-mail a partir do meu Gmail e Google Drive."
### Auto-Hospedado
Execute o SurfSense na sua própria infraestrutura para controle total de dados e privacidade.
**Pré-requisitos:** [Docker Desktop](https://www.docker.com/products/docker-desktop/) deve estar instalado e em execução.
#### Para usuários de Linux/MacOS:
Para Linux/macOS:
```bash
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash
```
#### Para usuários do Windows:
Para Windows:
```powershell
```bash
irm https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.ps1 | iex
```
O script de instalação configura o [Watchtower](https://github.com/nicholas-fedor/watchtower) automaticamente para atualizações diárias. Para pular, adicione a flag `--no-watchtower`.
O script de instalação configura o [Watchtower](https://github.com/nicholas-fedor/watchtower) automaticamente para atualizações automáticas diárias. Para pular essa etapa, adicione a flag `--no-watchtower`. Para Docker Compose, instalação manual e outras opções de implantação, consulte a [documentação](https://www.surfsense.com/docs/).
Para Docker Compose, instalação manual e outras opções de implantação, consulte a [documentação](https://www.surfsense.com/docs/).
## Fluxos de trabalho de inteligência competitiva
### Aplicativo Desktop
As automações executam turnos completos de agente de forma agendada ou em resposta a eventos, e depois gravam os resultados no Notion, Slack, Linear e Jira. Descreva o fluxo de trabalho em linguagem natural e o SurfSense o constrói, sem precisar de código. Experimente prompts como:
O SurfSense também oferece um aplicativo desktop que traz assistência de IA para cada aplicativo no seu computador. Baixe-o na [última versão](https://github.com/MODSetter/SurfSense/releases/latest).
- "Monitore as páginas de preços dos nossos 3 principais concorrentes e me avise no Slack quando um plano ou preço mudar."
- "Acompanhe cada menção à nossa marca no Reddit e no YouTube e me envie um resumo diário."
- "Monitore nosso ranking no Google para nossas 10 principais palavras-chave e sinalize quedas semana a semana."
- "Colete os novos reviews do Google Maps das nossas unidades e das dos concorrentes toda segunda-feira."
- "Execute um relatório mensal de análise de concorrentes e salve no meu workspace."
O aplicativo desktop inclui estes recursos poderosos:
- **General Assist** — Abra o SurfSense instantaneamente de qualquer aplicativo com um atalho global.
- **Quick Assist** — Selecione texto em qualquer lugar, depois peça à IA para explicar, reescrever ou agir sobre ele.
- **Screenshot Assist** — Selecione uma região da tela e anexe ao chat para respostas fundamentadas na sua base de conhecimento.
- **Watch Local Folder** — Monitore uma pasta local e sincronize automaticamente as alterações de arquivos com sua base de conhecimento. **Pro tip:** Aponte para seu cofre do Obsidian para manter suas notas pesquisáveis no SurfSense.
## Tudo o mais que vem na caixa
Todos os recursos operam no espaço de busca escolhido, para que suas respostas sejam sempre baseadas nos seus próprios dados.
O workspace de pesquisa que fez do SurfSense a principal alternativa open source ao NotebookLM continua aqui, e tudo o que seus agentes coletam chega até ele.
### Como Colaborar em Tempo Real (Beta)
**Base de conhecimento**
1. Acesse a página de Gerenciar Membros e crie um convite.
- Envie PDFs, documentos do Office, imagens e áudio, ou sincronize **Google Drive, OneDrive e Dropbox**. Mais de 50 formatos de arquivo suportados.
- Busca híbrida semântica e de texto completo, com respostas citadas no estilo Perplexity.
- Organização de arquivos por IA que classifica automaticamente os documentos por origem, data e tópico.
<p align="center"><img src="https://github.com/user-attachments/assets/40ed7683-5aa6-48a0-a3df-00575528c392" alt="Convidar Membros" /></p>
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="Converse com seus PDFs e documentos" /></p>
2. O colega aceita e aquele SearchSpace se torna compartilhado.
**Estúdio de entregáveis**
<p align="center"><img src="https://github.com/user-attachments/assets/ea4e1057-4d2b-4fd2-9ca0-cd19286a285e" alt="Fluxo de Entrada por Convite" /></p>
- Gerador de relatórios com IA, com exportação para PDF, DOCX, HTML, LaTeX, EPUB, ODT ou texto simples.
- Podcasts de IA com dois apresentadores a partir de qualquer documento ou pasta em menos de 20 segundos.
- Apresentações de slides editáveis, resumos em vídeo narrados e geração de imagens por IA.
3. Torne o chat compartilhado.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="Gerador de Relatórios com IA" /></p>
<p align="center"><img src="https://github.com/user-attachments/assets/17b93904-0888-4c3a-ac12-51a24a8ea26a" alt="Tornar Chat Compartilhado" /></p>
**Colaboração em equipe**
4. Sua equipe agora pode conversar em tempo real.
- Chats de IA colaborativos em tempo real, com comentários e menções.
- RBAC com os papéis de Owner, Admin, Editor e Viewer.
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Chat em Tempo Real" /></p>
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Chat de IA colaborativo" /></p>
5. Adicione comentários para marcar colegas de equipe.
**Aplicativo desktop**
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Comentários em Tempo Real" /></p>
Assistência de IA nativa em todos os aplicativos do seu computador. Baixe na [versão mais recente](https://github.com/MODSetter/SurfSense/releases/latest).
- **General Assist**: abra o SurfSense a partir de qualquer aplicativo com um atalho global.
- **Quick Assist**: selecione texto em qualquer lugar e peça à IA para explicar, reescrever ou agir sobre ele.
- **Screenshot Assist**: capture qualquer região da tela e faça perguntas à IA sobre ela.
- **Watch Local Folder** (monitorar pasta local): sincronize automaticamente uma pasta local com a sua base de conhecimento. Aponte para o seu cofre do Obsidian para manter suas notas pesquisáveis.
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
**Sem dependência de fornecedor**
- Mais de 100 LLMs via especificação da OpenAI e LiteLLM, incluindo GPT-5.5, Claude Sonnet 5 e Gemini 3.1 Pro.
- Mais de 6.000 modelos de embedding e todos os principais rerankers.
- Suporte completo a LLMs locais e privados (vLLM, Ollama), para que seus dados continuem sendo seus.
## Exemplo do Agente de Vídeo
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
## Exemplo do Agente de Podcast
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
## Como colaborar em tempo real (Beta)
1. Vá até a página Manage Members e crie um convite.
<p align="center"><img src="https://github.com/user-attachments/assets/40ed7683-5aa6-48a0-a3df-00575528c392" alt="Convidar membros" /></p>
2. Um colega entra e aquele workspace passa a ser compartilhado.
<p align="center"><img src="https://github.com/user-attachments/assets/ea4e1057-4d2b-4fd2-9ca0-cd19286a285e" alt="Fluxo de entrada por convite" /></p>
3. Torne um chat compartilhado e trabalhem nele juntos em tempo real, com comentários para marcar colegas.
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Comentários em tempo real" /></p>
## SurfSense vs Google NotebookLM
Ainda nos comparando como alternativa ao NotebookLM? Aqui está o comparativo honesto.
| Recurso | Google NotebookLM | SurfSense |
|---------|-------------------|-----------|
| **Fontes por Notebook** | 50 (Grátis) a 600 (Ultra, $249.99/mês) | Ilimitadas |
| **Número de Notebooks** | 100 (Grátis) a 500 (planos pagos) | Ilimitados |
| **Limite de Tamanho da Fonte** | 500.000 palavras / 200MB por fonte | Sem limite |
| **Preços** | Nível gratuito disponível; Pro $19.99/mês, Ultra $249.99/mês | Gratuito e de código aberto, auto-hospedável na sua própria infra |
| **Suporte a LLM** | Apenas Google Gemini | 100+ LLMs via OpenAI spec e LiteLLM |
| **Modelos de Embeddings** | Apenas Google | 6.000+ modelos de embeddings, todos os principais rerankers |
| **LLMs Locais / Privados** | Não disponível | Suporte completo (vLLM, Ollama) - seus dados ficam com você |
| **Auto-Hospedável** | Não | Sim - Docker em um único comando ou Docker Compose completo |
| **Código Aberto** | Não | Sim |
| **Conectores Externos** | Google Drive, YouTube, sites | 27+ conectores - Mecanismos de busca, Google Drive, OneDrive, Dropbox, Slack, Teams, Jira, Notion, GitHub, Discord e [mais](#fontes-externas) |
| **Suporte a Formatos de Arquivo** | PDFs, Docs, Slides, Sheets, CSV, Word, EPUB, imagens, URLs web, YouTube | 50+ formatos - documentos, imagens, vídeos via LlamaCloud, Unstructured ou Docling (local) |
| **Busca** | Busca semântica | Busca Híbrida - Semântica + Texto completo com Índices Hierárquicos e Reciprocal Rank Fusion |
| **Respostas com Citações** | Sim | Sim - Respostas citadas no estilo Perplexity |
| **Arquitetura de Agentes** | Não | Sim - alimentado por [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) com planejamento, subagentes e acesso ao sistema de arquivos |
| **Multiplayer em Tempo Real** | Notebooks compartilhados com papéis de Visualizador/Editor (sem chat em tempo real) | RBAC com papéis de Proprietário / Admin / Editor / Visualizador, chat em tempo real e threads de comentários |
| **Geração de Vídeos** | Visões gerais cinemáticas via Veo 3 (apenas Ultra) | Disponível (NotebookLM é melhor aqui, melhorando ativamente) |
| **Geração de Apresentações** | Slides mais bonitos mas não editáveis | Cria apresentações editáveis baseadas em slides |
| **Geração de Podcasts** | Visões gerais em áudio com hosts e idiomas personalizáveis | Disponível com múltiplos provedores TTS (NotebookLM é melhor aqui, melhorando ativamente) |
| **Automações e Agentes de IA** | Não | Fluxos de trabalho de IA agendados, gatilhos por eventos em novos documentos e automações sem código criadas por chat com escrita de volta no Notion, Slack, Linear e Jira |
| **Aplicativo Desktop** | Não | Aplicativo nativo com General Assist, Quick Assist, Screenshot Assist e sincronização de pastas locais |
| **Extensão de Navegador** | Não | Extensão multi-navegador para salvar qualquer página web, incluindo páginas protegidas por autenticação |
<details>
<summary><b>Lista completa de Fontes Externas</b></summary>
<a id="fontes-externas"></a>
Mecanismos de Busca (SearXNG, Tavily, LinkUp, Baidu Search) · Google Drive · OneDrive · Dropbox · Slack · Microsoft Teams · Linear · Jira · ClickUp · Confluence · BookStack · Notion · Gmail · Vídeos do YouTube · GitHub · Discord · Airtable · Google Calendar · Luma · Circleback · Elasticsearch · Obsidian, e mais por vir.
</details>
## SOLICITAÇÕES DE FUNCIONALIDADES E FUTURO
| **Dados de mercado ao vivo para agentes** | Não | Conectores de Reddit, YouTube, Google Maps, Google Search e rastreamento web via API REST e MCP |
| **Servidor MCP** | Não | Cada conector exposto como ferramenta nativa de agente, além de servidores MCP próprios com apps OAuth em um clique |
| **Fontes por Notebook** | 50 (gratuito) a 600 (Ultra, US$ 249,99/mês) | Ilimitadas |
| **Número de Notebooks** | 100 (gratuito) a 500 (planos pagos) | Ilimitado |
| **Limite de tamanho por fonte** | 500.000 palavras / 200 MB por fonte | Sem limite |
| **Preços** | Plano gratuito; Pro US$ 19,99/mês, Ultra US$ 249,99/mês | Gratuito e open source para auto-hospedar; a nuvem é paga por uso, com US$ 5 de crédito gratuito |
| **Suporte a LLMs** | Apenas Google Gemini | Mais de 100 LLMs via especificação da OpenAI e LiteLLM |
| **Modelos de embedding** | Apenas Google | Mais de 6.000 modelos de embedding, todos os principais rerankers |
| **LLMs locais / privados** | Não disponível | Suporte completo (vLLM, Ollama), seus dados continuam sendo seus |
| **Auto-hospedável** | Não | Sim, com uma linha de Docker ou Docker Compose completo |
| **Open source** | Não | Sim |
| **Fontes da base de conhecimento** | Google Drive, YouTube, sites | Upload de arquivos, Google Drive, OneDrive, Dropbox, sincronização de pasta local e páginas rastreadas |
| **Formatos de arquivo suportados** | PDFs, Docs, Slides, Sheets, CSV, Word, EPUB, imagens, URLs, YouTube | Mais de 50 formatos: documentos, imagens, vídeos via LlamaCloud, Unstructured ou Docling (local) |
| **Busca** | Busca semântica | Híbrida semântica + texto completo, com índices hierárquicos e fusão de rankings recíprocos |
| **Respostas com citações** | Sim | Sim, respostas citadas no estilo Perplexity |
| **Arquitetura agêntica** | Não | Sim, com [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview), incluindo planejamento, subagentes e acesso ao sistema de arquivos |
| **Automações e agentes de IA** | Não | Fluxos agendados, gatilhos por eventos e automações sem código criadas via chat, com gravação no Notion, Slack, Linear e Jira |
| **Multiplayer em tempo real** | Notebooks compartilhados com papéis Viewer/Editor (sem chat em tempo real) | RBAC com papéis Owner / Admin / Editor / Viewer, chat em tempo real e threads de comentários |
| **Geração de vídeo** | Video Overviews cinematográficos via Veo 3 (apenas Ultra) | Disponível (o NotebookLM é melhor aqui, em melhoria contínua) |
| **Geração de apresentações** | Slides mais bonitos, porém não editáveis | Apresentações editáveis baseadas em slides |
| **Geração de podcasts** | Audio Overviews com apresentadores e idiomas personalizáveis | Disponível com vários provedores de TTS (o NotebookLM é melhor aqui, em melhoria contínua) |
| **Aplicativo desktop** | Não | App nativo com General Assist, Quick Assist, Screenshot Assist e sincronização de pasta local |
## Pedidos de recursos e futuro
**O SurfSense está em desenvolvimento ativo.** Embora ainda não esteja pronto para produção, você pode nos ajudar a acelerar o processo.
Junte-se ao [Discord do SurfSense](https://discord.gg/ejRNvftDp9) e ajude a moldar o futuro do SurfSense!
Entre no [Discord do SurfSense](https://discord.gg/ejRNvftDp9) e ajude a moldar o futuro do SurfSense!
## Roadmap
Fique atualizado com nosso progresso de desenvolvimento e próximas funcionalidades!
Confira nosso roadmap público e contribua com suas ideias ou feedback:
Fique por dentro do nosso progresso de desenvolvimento e dos próximos recursos. Confira nosso roadmap público e contribua com suas ideias ou feedback:
**Discussão do Roadmap:** [SurfSense 2026 Roadmap](https://github.com/MODSetter/SurfSense/discussions/565)
**Discussão do roadmap:** [SurfSense 2026 Roadmap](https://github.com/MODSetter/SurfSense/discussions/565)
**Quadro Kanban:** [SurfSense Project Board](https://github.com/users/MODSetter/projects/3)
## Contribua
## Contribuir
Todas as contribuições são bem-vindas, desde estrelas e relatórios de bugs até melhorias no backend. Consulte [CONTRIBUTING.md](CONTRIBUTING.md) para começar.
Todas as contribuições são bem-vindas, de estrelas e relatos de bugs a melhorias no backend. Veja o [CONTRIBUTING.md](CONTRIBUTING.md) para começar.
Obrigado a todos os nossos Surfers:
@ -310,13 +245,13 @@ Obrigado a todos os nossos Surfers:
<img src="https://contrib.rocks/image?repo=MODSetter/SurfSense" />
</a>
## Histórico de Stars
## Histórico de estrelas
<a href="https://www.star-history.com/#MODSetter/SurfSense&Date">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=MODSetter/SurfSense&type=Date&theme=dark" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=MODSetter/SurfSense&type=Date" />
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=MODSetter/SurfSense&type=Date" />
<img alt="Gráfico do histórico de estrelas" src="https://api.star-history.com/svg?repos=MODSetter/SurfSense&type=Date" />
</picture>
</a>

375
README.zh-CN.md
View file

@ -1,4 +1,4 @@
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="readme_banner" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
<a href="https://www.surfsense.com/"><img width="1584" height="396" alt="SurfSense，面向 AI 智能体的开源竞争情报平台" src="https://github.com/user-attachments/assets/9361ef58-1753-4b6e-b275-5020d8847261" /></a>
@ -20,291 +20,226 @@
<a href="https://trendshift.io/repositories/13606" target="_blank"><img src="https://trendshift.io/api/badge/repositories/13606" alt="MODSetter%2FSurfSense | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</div>
# SurfSense
# SurfSense：为你的 AI 智能体注入竞争情报能力
NotebookLM 是目前最好、最实用的 AI 平台之一，但当你开始经常使用它时，你也会感受到它的局限性，总觉得还有不足之处。
SurfSense 是**面向 AI 智能体的开源竞争情报平台**。你的智能体可以通过一个 **REST API** 或 **MCP 服务器**，利用来自 **Reddit、YouTube、Google Maps、Google Search 和开放网络**的实时数据，监控竞争对手、追踪排名、倾听市场动态。定时和事件触发的智能体会把发现的内容转化为简报和预警，内置的知识库则让每一条发现都可搜索、可引用。
1. 一个笔记本中可以添加的来源数量有限制。
2. 可以拥有的笔记本数量有限制。
3. 来源不能超过 500,000 个单词和 200MB。
4. 你被锁定在 Google 服务中（LLM、使用模型等），没有配置选项。
5. 有限的外部数据源和服务集成。
6. NotebookLM 代理专门针对学习和研究进行了优化，但你可以用源数据做更多事情。
7. 缺乏多人协作支持。
> [!NOTE]
> **📢 致我们的 NotebookLM 替代品用户**
>
> 在过去几个月里，我们把 SurfSense 打造成了针对个人知识的最佳通用研究智能体，这段旅程为我们赢得了一个令我们由衷自豪的社区。如今，Claude、OpenCode、Hermes、OpenClaw 等智能体工具已经证明智能体就是未来，通用研究正在成为每个有能力的智能体开箱即用的功能。而智能体仍然缺少的是**实时市场数据以及围绕它的工作流**，所以这正是我们全力投入的方向：成为标杆级的开源竞争情报智能体平台。
>
> **你所依赖的一切功能都不会消失。**你的知识库、带引用的对话、报告、播客、演示文稿、自动化以及协作聊天都会继续可用，自托管也依然免费且开源。完整公告请阅读[我们的更新日志](https://www.surfsense.com/changelog)。
...还有更多。
## 为什么智能体需要 SurfSense
**SurfSense 正是为了解决这些问题而生。** SurfSense 赋予你：
问任何一个有能力的智能体"竞争对手这周的定价是多少？"或者"自发布以来 Reddit 上对我们的评价如何？"，它都找不到可信赖的数据来源。官方平台 API 要么有速率限制，要么按企业级定价，要么根本不存在，而自建抓取管线又非常脆弱。SurfSense 正是为填补这一空白而生：
- **控制你的数据流** - 保持数据私密和安全。
- **无数据限制** - 添加无限数量的来源和笔记本。
- **无供应商锁定** - 配置任何 LLM、图像、TTS 和 STT 模型。
- **25+ 外部数据源** - 从 Google Drive、OneDrive、Dropbox、Notion 和许多其他外部服务添加你的来源。
- **实时多人协作支持** - 在共享笔记本中轻松与团队成员协作。
- **AI 自动化与智能体** - 按计划运行 AI 智能体，或在文档进入文件夹的那一刻触发它们，然后将结果回写到 Notion、Slack、Linear 和 Drive。只需在聊天中描述即可创建无代码自动化。
- **桌面应用** - 通过 Quick Assist、General Assist、Screenshot Assist 和本地文件夹同步在任何应用程序中获得 AI 助手。
- **平台原生连接器**，每个都是返回结构化 JSON 的强类型 REST 端点。不用赌速率限制，也不用解析 HTML。
- **一个 MCP 服务器**，把每个连接器都作为原生工具（`reddit.scrape`、`google_search.scrape` 等）暴露给 Claude、Cursor 或任何智能体框架。
- **一套智能体运行框架**，而不只是原始数据：重试、结构化输出和额度计量都已内置，智能体可以从一个问题直达一份简报，无需你自己搭建管线。
- **开源且可自托管**，你的竞争研究数据始终留在你自己的基础设施上。
...更多功能即将推出。
## 实时数据连接器
| 连接器 | 你的智能体能获得什么 | 了解更多 |
|---|---|---|
| **Reddit** | 帖子、评论和子版块信息流，不受官方 API 速率限制 | [Reddit Scraper API](https://www.surfsense.com/reddit) |
| **YouTube** | 视频、字幕转录和评论串，用于品牌和产品舆情监听 | [YouTube Scraper API](https://www.surfsense.com/youtube) |
| **Google Maps** | 地点、评分和评论，用于本地竞争对手和潜在客户调研 | [Google Maps Scraper API](https://www.surfsense.com/google-maps) |
| **Google Search** | 实时搜索结果页，用于排名追踪和市场监控 | [Google Search API](https://www.surfsense.com/google-search) |
| **Web Crawl** | 把开放网络上的任意页面转为干净、结构化的内容 | [Web Crawling API](https://www.surfsense.com/web-crawl) |
| **MCP Connector** | 将任意 MCP 服务器接入你的智能体，Notion、Slack、Jira 等支持一键 OAuth | [MCP Connector](https://www.surfsense.com/mcp-connector) |
计费采用按量付费：连接器按实际返回的条目计费，爬取按成功抓取的页面计费，失败的调用永不计费。自托管部署默认关闭计费。详见[定价](https://www.surfsense.com/pricing)。
## 视频代理示例
## 快速开始
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
### 在代码中调用连接器
每个连接器都是一个 REST 端点，你可以用任何语言、凭借你的 SurfSense API 密钥来调用：
```bash
curl -X POST "$SURFSENSE_API_URL/workspaces/$WORKSPACE_ID/scrapers/reddit/scrape" \
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"search_queries": ["your brand"],
"community": "webscraping",
"sort": "top",
"time_filter": "week"
}'
```
## 播客代理示例
每个[连接器页面](https://www.surfsense.com/connectors)都提供 Python、JavaScript、Go、PHP、Ruby、Java 和 C# 的可直接复制粘贴的示例。
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
### 通过 MCP 把工具交给你的智能体
把 SurfSense MCP 服务器添加到 Claude、Cursor 或你自己的智能体框架：
## 如何使用 SurfSense
```json
{
"mcpServers": {
"surfsense": {
"url": "https://mcp.surfsense.com",
"headers": { "Authorization": "Bearer ${SURFSENSE_API_KEY}" }
}
}
}
```
### Cloud
现在，你的智能体就可以把每个连接器当作原生工具来调用。
1. 访问 [surfsense.com](https://www.surfsense.com) 并登录。
### 使用云端服务
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/LoginFlowGif.gif" alt="登录" /></p>
访问 [surfsense.com](https://www.surfsense.com)，登录后用自然语言向智能体索取实时市场数据。新账户自带 5 美元免费额度，无需订阅。
2. 连接您的连接器并同步。启用定期同步以保持连接器数据更新。
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="连接器" /></p>
### 免费自托管
3. 在连接器数据索引期间，上传文档。
在你自己的基础设施上运行整个平台，包括连接器、智能体、自动化和 MCP 服务器。自托管部署默认关闭计费，抓取、爬取和智能体运行只受你的硬件和你自带的模型密钥限制。
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="上传文档" /></p>
**前置条件：**必须已安装并运行 [Docker Desktop](https://www.docker.com/products/docker-desktop/)。
4. 一切索引完成后，尽管提问（使用场景）：
**桌面应用**（在以下所有功能之外的原生附加功能，并非独立的功能集）
- General Assist：通过全局快捷键，从任意应用中即刻打开 SurfSense。
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/general_assist.gif" alt="General Assist" /></p>
- Quick Assist：在任意位置选中文本，让 AI 解释、改写或对其执行操作。
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
- Screenshot Assist：截取屏幕上任意区域，并就其中内容向 AI 提问。
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/screenshot_assist.gif" alt="Screenshot Assist" /></p>
- Watch Local Folder：将本地文件夹自动同步到你的知识库。非常适合 Obsidian 库。
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/folder_watch.gif" alt="Watch Local Folder" /></p>
**成果工作室**
- AI Report Generator：生成带引用的研究报告，并导出为 PDF、DOCX、HTML、LaTeX、EPUB、ODT 或纯文本。
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="AI Report Generator" /></p>
- AI Podcast Generator：在 20 秒内将任意文档或文件夹转换为双主持人 AI 播客。
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/PodcastGenGif.gif" alt="AI Podcast Generator" /></p>
- AI Presentation & Video Maker：根据你的资料创建可编辑的幻灯片和带旁白的视频概览。
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/video_gen_gif.gif" alt="AI Presentation and Video Maker" /></p>
- AI Image Generator：直接从你的聊天和文档生成高质量图像。
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ImageGenGif.gif" alt="AI Image Generator" /></p>
- AI Resume Builder：根据任意职位描述定制你现有的简历，顺利通过 ATS。
可以试试这样的提示：
- “根据这份职位描述定制我的简历，让它通过 ATS 并赢得面试。”
- “匹配这份招聘启事中的关键词，为 ATS 优化我的简历。”
- “重写我的简历要点，突出这个岗位所需要的技能。”
- “将我的简历与这份职位描述对比，列出需要改进的差距。”
- “根据我的简历和这份职位描述，写一封相匹配的求职信。”
**搜索与聊天**
- Chat With Your PDFs & Docs：跨所有文件提问，并获得带内联引用的答案。
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="Chat With Your PDFs and Docs" /></p>
- AI Search With Citations：在整个知识库中进行语义与关键词的混合搜索。
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BSNCGif.gif" alt="AI Search With Citations" /></p>
- Collaborative AI Chat：与团队实时协作处理 AI 对话。
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="Collaborative AI Chat" /></p>
- Comments & Mentions：在任意 AI 消息上评论并 @ 你的队友。
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="Comments and Mentions" /></p>
**连接器与集成**
- Connect & Sync Your Tools：将 Notion、Slack、Google Drive、Gmail、GitHub、Linear 等 25+ 数据源同步为一个可搜索的语料库。
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ConnectorFlowGif.gif" alt="Connect and Sync Your Tools" /></p>
- Chat With Uploaded Files：上传 PDF、Office 文档、图像和音频。即刻可搜索。
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/DocUploadGif.gif" alt="Chat With Uploaded Files" /></p>
- Connector Write-Back：让智能体将结果回写到 Notion、Slack、Linear 和 Drive。
可以试试这样的提示：
- “把这份研究摘要发布到我的 Notion 工作区。”
- “把这些会议行动项发送到我们的团队 Slack 频道。”
- “根据这份缺陷报告创建一个 Jira 工单。”
- “根据这个功能需求在 Linear 中创建一个 issue。”
- “把这份生成的报告作为文档保存到 Google Drive。”
- Obsidian & Knowledge Base Sync：让你的 Obsidian 库与个人知识库保持同步。
**自动化**
- Scheduled AI Workflows：按计划运行智能体：每日简报、每周摘要、周期性报告。
可以试试这样的提示：
- “每天早上把我知识库中新增文档的每日简报发邮件给我。”
- “每周五根据我的 Slack 和 Gmail 生成一份每周状态报告。”
- “每月运行一次竞争对手分析报告并保存到我的工作区。”
- “把我的 GitHub 和 Linear 活动汇总成一份每日站会更新。”
- “针对我关注的主题创建一份周期性的每周研究报告。”
- Event-Triggered Automations：在文档进入文件夹的那一刻触发智能体，并将结果发布到你的工具中。
可以试试这样的提示：
- “当一个 PDF 进入我的 Research 文件夹时，生成一份带引用的 AI 摘要。”
- “当新增会议记录时，把它整理成带行动项的会议纪要。”
- “当上传发票时，把供应商、总额和到期日提取到一张表格中。”
- “当一份合同进入我的 Legal 文件夹时，标记关键条款和续约日期。”
- “当一份简历加入 Candidates 时，根据职位描述对其进行筛选。”
- Chat-Built Automations：用通俗的语言描述一个自动化，SurfSense 就会为你构建它。
可以试试这样的提示：
- “创建一个 AI 智能体，每天早上把新增 Notion 页面的摘要发邮件给我。”
- “创建一个无代码自动化，每周把研究摘要发布到 Slack。”
- “设置一个 AI 笔记助手，把新增会议记录整理成纪要。”
- “创建一个工作流，从会议记录中提取行动项并指派负责人。”
- “自动化一份来自我的 Gmail 和 Google Drive 的每日邮件简报。”
### 自托管
在您自己的基础设施上运行 SurfSense，实现完全的数据控制和隐私保护。
**前置条件：** 需要安装并运行 [Docker Desktop](https://www.docker.com/products/docker-desktop/)。
#### Linux/MacOS 用户：
Linux/macOS：
```bash
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash
```
#### Windows 用户：
Windows：
```powershell
```bash
irm https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.ps1 | iex
```
安装脚本会自动配置 [Watchtower](https://github.com/nicholas-fedor/watchtower) 以实现每日自动更新。如需跳过，请添加 `--no-watchtower` 参数。
安装脚本会自动配置 [Watchtower](https://github.com/nicholas-fedor/watchtower) 以实现每日自动更新。如需跳过，请加上 `--no-watchtower` 参数。关于 Docker Compose、手动安装及其他部署方式，请参阅[文档](https://www.surfsense.com/docs/)。
如需 Docker Compose、手动安装及其他部署方式，请查看[文档](https://www.surfsense.com/docs/)。
## 竞争情报工作流
### 桌面应用
自动化功能可以按计划或响应事件运行完整的智能体回合，然后把结果写回 Notion、Slack、Linear 和 Jira。用自然语言描述工作流，SurfSense 就会替你构建，无需写代码。可以试试这些提示语：
SurfSense 还提供桌面应用，将 AI 助手带到您计算机上的每个应用程序中。从[最新版本](https://github.com/MODSetter/SurfSense/releases/latest)下载。
- "盯住我们前 3 名竞争对手的定价页面，一旦套餐或价格变动就在 Slack 上提醒我。"
- "追踪 Reddit 和 YouTube 上对我们品牌的每一次提及，每天给我发一份摘要。"
- "监控我们前 10 个关键词的 Google 排名，按周标记下滑情况。"
- "每周一拉取我们门店以及竞争对手门店的最新 Google Maps 评论。"
- "每月生成一份竞争对手分析报告并保存到我的工作区。"
桌面应用包含以下强大功能：
- **General Assist** — 通过全局快捷键从任何应用程序即时启动 SurfSense。
- **Quick Assist** — 在任何位置选中文本，然后让 AI 解释、改写或对其执行操作。
- **Screenshot Assist** — 在屏幕上框选区域并附加到聊天，让回复基于您的知识库。
- **Watch Local Folder** — 监视本地文件夹，自动将文件更改同步到您的知识库。**Pro tip：** 将其指向您的 Obsidian vault，让笔记在 SurfSense 中随时可搜索。
## 开箱即用的其他能力
所有功能均基于您选择的搜索空间运行，确保回答始终以您自己的数据为依据。
让 SurfSense 成为领先的开源 NotebookLM 替代品的那套研究工作区依然都在，而且你的智能体收集到的一切都会汇入其中。
### 如何实时协作（Beta）
**知识库**
1. 前往成员管理页面并创建邀请。
- 上传 PDF、Office 文档、图片和音频，或同步 **Google Drive、OneDrive 和 Dropbox**。支持 50 多种文件格式。
- 混合语义与全文搜索，提供 Perplexity 风格的带引用回答。
- AI 文件整理功能按来源、日期和主题自动归类文档。
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/BQnaGif_compressed.gif" alt="与你的 PDF 和文档对话" /></p>
**成果工作室**
- AI 报告生成器，可导出为 PDF、DOCX、HTML、LaTeX、EPUB、ODT 或纯文本。
- 20 秒内基于任意文档或文件夹生成双主持人 AI 播客。
- 可编辑的幻灯片、带旁白的视频概览以及 AI 图像生成。
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/ReportGenGif_compressed.gif" alt="AI 报告生成器" /></p>
**团队协作**
- 支持评论和提及的实时协作 AI 聊天。
- 基于角色的访问控制（RBAC），提供所有者、管理员、编辑者和查看者角色。
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="协作 AI 聊天" /></p>
**桌面应用**
在电脑上的每个应用中获得原生 AI 辅助。从[最新版本](https://github.com/MODSetter/SurfSense/releases/latest)下载。
- **General Assist**：通过全局快捷键在任意应用中唤起 SurfSense。
- **Quick Assist**：在任意位置选中文本，让 AI 解释、改写或据此执行操作。
- **Screenshot Assist**：截取屏幕上任意区域，向 AI 提问。
- **Watch Local Folder**：把本地文件夹自动同步到知识库。将它指向你的 Obsidian 仓库，让笔记随时可搜索。
<p align="center"><img src="surfsense_web/public/homepage/hero_tutorial/quick_assist.gif" alt="Quick Assist" /></p>
**无供应商锁定**
- 通过 OpenAI 规范和 LiteLLM 支持 100 多种 LLM，包括 GPT-5.5、Claude Sonnet 5 和 Gemini 3.1 Pro。
- 支持 6,000 多种嵌入模型和所有主流重排序器。
- 完整支持本地和私有 LLM（vLLM、Ollama），你的数据始终属于你。
## 视频智能体示例
https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a
## 播客智能体示例
https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7
## 如何进行实时协作（Beta）
1. 进入成员管理页面并创建邀请。
<p align="center"><img src="https://github.com/user-attachments/assets/40ed7683-5aa6-48a0-a3df-00575528c392" alt="邀请成员" /></p>
2. 队友加入后，该 SearchSpace 变为共享。
2. 队友加入后，该工作区即变为共享。
<p align="center"><img src="https://github.com/user-attachments/assets/ea4e1057-4d2b-4fd2-9ca0-cd19286a285e" alt="邀请加入流程" /></p>
3. 将聊天设为共享。
<p align="center"><img src="https://github.com/user-attachments/assets/17b93904-0888-4c3a-ac12-51a24a8ea26a" alt="设为共享聊天" /></p>
4. 您的团队现在可以实时聊天。
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeChatGif.gif" alt="实时聊天" /></p>
5. 添加评论以标记队友。
3. 将聊天设为共享，即可与团队实时协作，并通过评论标记队友。
<p align="center"><img src="surfsense_web/public/homepage/hero_realtime/RealTimeCommentsFlow.gif" alt="实时评论" /></p>
## SurfSense vs Google NotebookLM
## SurfSense 对比 Google NotebookLM
还在把我们当作 NotebookLM 替代品来比较？这里是坦诚的对比。
| 功能 | Google NotebookLM | SurfSense |
|---------|-------------------|-----------|
| **每个笔记本的来源数** | 50（免费）到 600（Ultra，$249.99/月） | 无限制 |
| **笔记本数量** | 100（免费）到 500（付费方案） | 无限制 |
| **来源大小限制** | 500,000 词 / 200MB 每个来源 | 无限制 |
| **定价** | 免费版可用；Pro $19.99/月，Ultra $249.99/月 | 免费开源，在自己的基础设施上自托管 |
| **LLM 支持** | 仅 Google Gemini | 100+ LLM，通过 OpenAI spec 和 LiteLLM |
| **嵌入模型** | 仅 Google | 6,000+ 嵌入模型，所有主流重排序器 |
| **本地 / 私有 LLM** | 不可用 | 完整支持（vLLM、Ollama）- 您的数据由您掌控 |
| **可自托管** | 否 | 是 - Docker 一行命令或完整 Docker Compose |
| **面向智能体的实时市场数据** | 无 | 通过 REST API 和 MCP 提供 Reddit、YouTube、Google Maps、Google Search 和网页爬取连接器 |
| **MCP 服务器** | 无 | 每个连接器都作为原生智能体工具暴露，还可自带 MCP 服务器并使用一键 OAuth 应用 |
| **每个笔记本的来源数** | 50 个（免费版）至 600 个（Ultra 版，249.99 美元/月） | 无限制 |
| **笔记本数量** | 100 个（免费版）至 500 个（付费档位） | 无限制 |
| **来源大小限制** | 每个来源 50 万字 / 200MB | 无限制 |
| **定价** | 免费档；Pro 19.99 美元/月，Ultra 249.99 美元/月 | 自托管免费且开源；云端按量付费，附赠 5 美元免费额度 |
| **LLM 支持** | 仅 Google Gemini | 通过 OpenAI 规范和 LiteLLM 支持 100 多种 LLM |
| **嵌入模型** | 仅 Google | 6,000 多种嵌入模型，所有主流重排序器 |
| **本地 / 私有 LLM** | 不支持 | 完整支持（vLLM、Ollama），你的数据始终属于你 |
| **可自托管** | 否 | 是，Docker 一行命令或完整 Docker Compose |
| **开源** | 否 | 是 |
| **外部连接器** | Google Drive、YouTube、网站 | 27+ 连接器 - 搜索引擎、Google Drive、OneDrive、Dropbox、Slack、Teams、Jira、Notion、GitHub、Discord 等[更多](#外部数据源) |
| **文件格式支持** | PDF、Docs、Slides、Sheets、CSV、Word、EPUB、图像、网页 URL、YouTube | 50+ 格式 - 文档、图像、视频，通过 LlamaCloud、Unstructured 或 Docling（本地） |
| **搜索** | 语义搜索 | 混合搜索 - 语义 + 全文搜索，结合层次化索引和倒数排名融合 |
| **引用回答** | 是 | 是 - Perplexity 风格的引用回答 |
| **代理架构** | 否 | 是 - 基于 [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) 构建，支持规划、子代理和文件系统访问 |
| **实时多人协作** | 共享笔记本，支持查看者/编辑者角色（无实时聊天） | RBAC 角色控制（所有者/管理员/编辑者/查看者），实时聊天和评论线程 |
| **视频生成** | 通过 Veo 3 的电影级视频概览（仅 Ultra） | 可用（NotebookLM 在此方面更好，正在积极改进） |
| **演示文稿生成** | 更美观的幻灯片但不可编辑 | 创建可编辑的幻灯片式演示文稿 |
| **播客生成** | 可自定义主持人和语言的音频概览 | 可用，支持多种 TTS 提供商（NotebookLM 在此方面更好，正在积极改进） |
| **AI 自动化与智能体** | 否 | 定时 AI 工作流、新文档的事件触发，以及通过聊天构建的无代码自动化，支持回写到 Notion、Slack、Linear 和 Jira |
| **桌面应用** | 否 | 原生应用，包含 General Assist、Quick Assist、Screenshot Assist 和本地文件夹同步 |
| **浏览器扩展** | 否 | 跨浏览器扩展，保存任何网页，包括需要身份验证的页面 |
<details>
<summary><b>外部数据源完整列表</b></summary>
<a id="外部数据源"></a>
搜索引擎（SearXNG、Tavily、LinkUp、Baidu Search）· Google Drive · OneDrive · Dropbox · Slack · Microsoft Teams · Linear · Jira · ClickUp · Confluence · BookStack · Notion · Gmail · YouTube 视频 · GitHub · Discord · Airtable · Google Calendar · Luma · Circleback · Elasticsearch · Obsidian，更多即将推出。
</details>
| **知识库来源** | Google Drive、YouTube、网站 | 文件上传、Google Drive、OneDrive、Dropbox、本地文件夹同步以及爬取的网页 |
| **文件格式支持** | PDF、Docs、Slides、Sheets、CSV、Word、EPUB、图片、网页 URL、YouTube | 50 多种格式：文档、图片、视频，通过 LlamaCloud、Unstructured 或 Docling（本地）解析 |
| **搜索** | 语义搜索 | 混合语义 + 全文搜索，带分层索引和倒数排名融合 |
| **带引用的回答** | 有 | 有，Perplexity 风格的引用回答 |
| **智能体架构** | 无 | 有，由 [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) 驱动，具备规划、子智能体和文件系统访问能力 |
| **AI 自动化与智能体** | 无 | 定时工作流、事件触发以及通过聊天构建的无代码自动化，可写回 Notion、Slack、Linear 和 Jira |
| **实时多人协作** | 共享笔记本，仅有查看者/编辑者角色（无实时聊天） | RBAC 提供所有者 / 管理员 / 编辑者 / 查看者角色，支持实时聊天和评论串 |
| **视频生成** | 通过 Veo 3 生成电影级视频概览（仅 Ultra 版） | 已提供（此项 NotebookLM 更强，我们正在积极改进） |
| **演示文稿生成** | 幻灯片更美观但不可编辑 | 可编辑的幻灯片式演示文稿 |
| **播客生成** | 音频概览，支持自定义主持人和语言 | 已提供，支持多种 TTS 服务商（此项 NotebookLM 更强，我们正在积极改进） |
| **桌面应用** | 无 | 原生应用，包含 General Assist、Quick Assist、Screenshot Assist 和本地文件夹同步 |
## 功能请求与未来规划
**SurfSense 正在积极开发中。**虽然它尚未达到生产就绪状态，但你可以帮助我们加快进度。
**SurfSense 正在积极开发中。** 虽然它还未达到生产就绪状态，但您可以帮助我们加快进度。
加入 [SurfSense Discord](https://discord.gg/ejRNvftDp9) 一起塑造 SurfSense 的未来！
加入 [SurfSense Discord](https://discord.gg/ejRNvftDp9)，一起塑造 SurfSense 的未来！
## 路线图
随时了解我们的开发进度和即将推出的功能！
查看我们的公开路线图并贡献您的想法或反馈：
随时了解我们的开发进度和即将推出的功能。查看我们的公开路线图，贡献你的想法或反馈：
**路线图讨论：** [SurfSense 2026 Roadmap](https://github.com/MODSetter/SurfSense/discussions/565)
**路线图讨论：**[SurfSense 2026 Roadmap](https://github.com/MODSetter/SurfSense/discussions/565)
**看板：** [SurfSense Project Board](https://github.com/users/MODSetter/projects/3)
**看板：**[SurfSense Project Board](https://github.com/users/MODSetter/projects/3)
## 参与贡献
## 贡献
欢迎一切形式的贡献，从点星标、报告缺陷到后端改进。请参阅 [CONTRIBUTING.md](CONTRIBUTING.md) 开始参与。
欢迎所有贡献，从 Star 和 Bug 报告到后端改进。请参阅 [CONTRIBUTING.md](CONTRIBUTING.md) 开始贡献。
感谢所有 Surfers:
感谢所有 Surfer：
<a href="https://github.com/MODSetter/SurfSense/graphs/contributors">
<img src="https://contrib.rocks/image?repo=MODSetter/SurfSense" />

257
plans/backend/00-umbrella-plan.md
View file

@ -1,257 +0,0 @@
# CI Pivot MVP — Umbrella Plan
> Master roadmap for the Competitive Intelligence pivot. Each phase becomes its own subplan saved in this folder (`plans/backend/`).
This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shipped), then the WebURL crawler moat (Phase 3, shipped), then the CI product itself as **four revamped phases** — **Phase 4 Capabilities** ([`04-capabilities.md`](04-capabilities.md)), **Phase 5 Access** ([`05-access.md`](05-access.md)), **Phase 6 Ongoing-Automation** ([`06-ongoing-automation.md`](06-ongoing-automation.md), design deferred), **Phase 7 Orchestration** ([`07-orchestration.md`](07-orchestration.md)). **NOTE:** the original "connectors → Pipelines" plan (old Phases 4–7) was superseded on 2026-06-30, and the stateful **Tracker + Timeline + Triggers** design was superseded on **2026-07-01** (scraper-APIs-first, stateless, chat-native) — see the Architecture corrections below.
> SCOPE: This umbrella currently covers the BACKEND only (`surfsense_backend`). Frontend (`surfsense_web`) and client apps (desktop, Obsidian, browser extension) will get their own umbrella/subplans LATER, once the backend is fully working as expected. Frontend-facing decisions (URL segment, TS types, i18n copy) are recorded below where relevant but are out of scope for the active phases.
## Positioning
Premium, open-source, self-hostable **scraper APIs** (Bright Data / Firecrawl / Scrapfly-class) that solve the hard parts — bypass, proxies, stealth — and return **cleaned, AI-ready data** for Google Maps + general web. A chat agent uses those APIs to answer competitive-intelligence questions and, for standing needs, keeps watching over time.
## ⚠️ Architecture pivot (2026-07-01) — scraper-APIs-first, stateless, chat-native
> **This supersedes the stateful Tracker/Timeline/Triggers design (the 2026-06-30 revamp's Phases 5–6).** The canonical Phase 4–7 subplans (`04`–`07`, below) are re-cast to four phases:
>
> - **The product = scraper capabilities**, not a CI engine. `web.*` + `maps.*` typed verbs return cleaned, AI-ready data, exposed identically through **REST + API key**, an **MCP server**, and **chat**, all generated from one capability registry. These endpoints are the revenue driver (premium, OSS, self-hostable).
> - **Stateless.** No Timeline (3-store delta), no `Tracker`, no diff/materiality engine, no stored `entity_changes`. Memory = the **chat history**; the agent reasons over prior tool outputs in context to report "what changed."
> - **Direct calls.** Capabilities return their result directly — no job store, no `completed | pending + job_id` envelope, no `deliverable_wait` polling. Async results reach the client via **write-then-sync / stream** (existing SSE or a Zero-published table).
> - **Automation = a persistent ongoing chat** that periodically re-invokes verbs. This replaces the `Triggers` subsystem; the periodic mechanism is **design-deferred** (Phase 6).
> - **KB stays input-only**; crawled data is never indexed.
>
> **Phase mapping:** **04 Capabilities** · **05 Access** · **06 Ongoing-Automation** · **07 Orchestration** (`scraping` subagent). The subplans `05a-timeline`/`05b-intelligence`/`06-triggers` are **removed** (history in git); `04a`/`04b` are folded into `04`/`05`.
## ⚠️ Architecture correction (2026-06-30) — Pipelines dropped; automations + input-only KB adopted
> **Historical record.** The pipelines-drop + input-only-KB decisions below still hold. The stateful Tracker/Timeline/Triggers framing in its "Net effect" bullets was itself superseded by the **2026-07-01 pivot** above.
> **This supersedes the original Pipelines paradigm (Phases 5–7).** During Phase 4 discussion we concluded that the *"sync data into the KB, then operate over it"* model is wrong for competitive intelligence. Verified against `references/opencode` — a coding agent that pulls context **live** via `read`/`grep`/`webfetch`/`websearch` tools (`packages/opencode/src/tool/`) and persists **sessions**, never a scraped corpus; its `websearch` tool even live-crawls at query time (`tool/websearch.ts`). The corrected model:
>
> - **Knowledge Base = input-only** — the user's personal files/context that *enriches* the agent. Filled by file uploads + file connectors (Google Drive / Dropbox / OneDrive). **Nothing scrapes into it.** Its job is to be searchable *user-provided* context, not a dump of fetched pages.
> - **Web search + the WebURL crawler = platform-native agent tools** (already `web_search` + `scrape_webpage` on the main agent, `main_agent/tools/index.py`), always available; later exposed to developers via a platform **API key**. They are **NOT connectors and NOT pipelines**.
> - **Connectors = MCP tools** (Type-2) **+ file/KB-input connectors** (the only Type-1 that survives). All branded natives → MCP.
> - **Automations are the scheduling + run-history substrate.** An automation run already invokes the *full chat agent* (with the crawler/search tools) on a **cron/event** trigger and persists **run history** (`automation_runs`) — exactly the recurring-fetch + monitoring loop Pipelines were reinventing. You store the agent's *insight per run*, not raw snapshots.
> - **`WEBCRAWLER_CONNECTOR` is retired** — the crawler is a native tool, not a connector/data-source.
>
> **Net effect on this umbrella:** the old **Pipelines** stack (old Phases 5–7) is dropped and **Phases 4–7 are re-cast** by the canonical subplans `04`–`07` (this framing was itself superseded by the 2026-07-01 pivot above):
> - **Phase 4 — Capabilities & Access** (`revamp/04a` + `04b`): turn the crawler + search into **typed, callable verbs** (MVP = `web.scrape` + `web.discover`; per-platform scrapers like `maps.*`/`linkedin.*` are *later, uncommitted* drop-ins) over a single **capability registry**, exposed identically through **chat / REST+API-key / MCP** doors. Ships **Product A** (stateless utility) — revenue day one. *The old `04a-connector-category` is demoted to backward-compat hygiene; the old `04b-source-discovery` is absorbed as the `web.discover` verb.*
> - **Phase 5 — Intelligence & Timeline** (`revamp/05a` + `05b`): the `Tracker` primitive + a **3-store delta Timeline** (`tracked_entities`/`entity_current_state`/`entity_changes`, "store deltas, no change → no row"). This is the durable CI state and **the moat** — it replaces "pipelines" as the *standing concern*. Ships the **Product B** engine.
> - **Phase 6 — Triggers** (`revamp/06`): decide **when** a Tracker refreshes. `refresh(tracker)` is trigger-agnostic; recurrence is a **CI action on the existing automations** (reuse its schedule selector + `AutomationRun` — no new scheduler, no new run table). Alert **delivery is separate** (via `app/notifications/`; automations have no delivery path).
> - **Phase 7 — Orchestration** (`revamp/07`): a net-new `scraping` subagent (intent routing one-shot-vs-standing, verb chains, Tracker crafting, decision-grounded answering) so the whole product is reachable in plain language.
>
> **KB stays input-only; `WEBCRAWLER_CONNECTOR` retired; crawler/search are native tools (later a platform API key).** The three correction decisions land in their real homes: **bill crawls at the capability executor** (`revamp/04a`, works uniformly across chat/automation/REST/MCP/cron — see 03c below), **diffable run history = the Timeline delta store** (`revamp/05a`, *not* `automation_runs.output`), and **"run now" = a Trigger adapter** (`revamp/06`). Phase 8 platform scrapers re-slot as a family of **individual native endpoints** — each one a capability verb → an agent tool + a dev API-key REST endpoint — added incrementally (**none, incl. Google Maps, committed for MVP**; they're examples of the pattern), not pipeline executors.
## Target architecture (scraper-APIs-first, stateless)
> Full end-to-end flow diagrams live in [`00b-pipeline-diagrams.md`](00b-pipeline-diagrams.md). Summary below.
```mermaid
flowchart TD
ACQ[["Acquisition (Phase 3, shipped): WebURL crawler + proxy/stealth/captcha + 03c billing · + Maps actor (new)"]]
REG["Capability registry (P4): web.scrape · web.discover · maps.search/place/reviews — typed verbs, cleaned AI-ready output, bill at executor"]
ACQ --> REG
REG --> DOORS["Access doors (P5): REST + API key · MCP server · chat tools — generated from the registry"]
DOORS --> AGENT["scraping subagent (P7): intent routing · verb chains · 'what changed' from chat history"]
KB[("Knowledge Base — input-only: user files/context (uploads + Drive/Dropbox/OneDrive)")] --> AGENT
CONN["Connectors = MCP tools (BYO) + file/KB-input"] --> AGENT
AGENT -->|one-shot| ANS["plain-language answer (nothing persists)"]
AGENT -.->|standing need| ONG["Ongoing-Automation (P6): persistent chat re-invokes verbs; memory = chat history"]
ONG -.-> REG
APIKEY["Platform API key"] --> DOORS
```
<details>
<summary>Original (superseded) Pipelines-centric architecture</summary>
```mermaid
flowchart TD
WS[WorkSpace] --> CONN[Connectors]
WS --> PIPE[Pipelines]
WS --> KB[(Knowledge Base: documents + chunks)]
CONN --> T1[Type 1: Data Sources - pull]
CONN --> T2[Type 2: MCP Tools - act]
T1 --> WEB[Universal WebURL Crawler - functional]
T1 --> PLAT[Platform connectors - coming soon]
T1 --> UP[File Upload]
PIPE --> RUN[PipelineRun history]
PIPE -->|"save_to_kb + destination folder"| KB
RUN -->|"manual or cron"| T1
T2 --> CHAT[Chat / Automations]
CHAT --> DELIV[Deliverables: audio/video/report/image]
RUN -->|"read-only context"| CHAT
```
</details>
## Decisions locked
- Full rename SearchSpace -> WorkSpace across DB, API, URLs, code, satellite apps.
- Canonical names (proposed defaults): DB table `workspaces`, column `workspace_id`, RBAC tables `workspace_roles` / `workspace_memberships` / `workspace_invites`, API base `/workspaces` (consolidating today's `/searchspaces` vs `/search-spaces` split), URL segment `[workspace_id]`, settings folder `workspace-settings`, TS type `Workspace`.
- **Capabilities, not connectors, are the core of the product.** The crawler + search + Maps actor become typed **verbs** (`web.scrape`, `web.discover`, `maps.search/place/reviews`) in a single **capability registry** that generates the REST/MCP/chat doors identically (`04`/`05`). A capability is *call → cleaned data → bill*; no `SearchSourceConnector` row, no KB write, no schedule. The **old connector two-type taxonomy is demoted to backward-compat hygiene** (kept so existing rows/KB docs stay searchable) — the one genuinely useful piece, the **`MCP_CONNECTOR` (BYO MCP) routing-gap fix**, moves into `05` (consume-user-MCP). Surviving connectors: **file/KB-input** (Google Drive native+Composio, OneDrive, Dropbox, uploads → the input-only KB) + **BYO `MCP_CONNECTOR`** (act in chat). **`WEBCRAWLER_CONNECTOR` RETIRED** (native tool); branded natives stay `MIGRATING` (off); Obsidian/Circleback `DISABLED`. Artifacts stay in the existing `deliverables` system.
- Web search APIs (SearXNG, Linkup, Baidu) are repurposed as the **`web.discover` capability** (a platform-level search provider set): given a topic/competitor, suggest candidate URLs the agent can feed to `web.scrape`. Not a standalone connector type, does not index data. NOTE: Tavily and Serper are being REMOVED from the search infra and are not part of this set.
- **Stateless — the chat history is the memory.** No Timeline, no `Tracker`, no diff/materiality engine, no stored `entity_changes`. "What changed" is the agent reasoning over prior tool outputs already in context (`07`). Standing needs are served by a **persistent ongoing chat** that periodically re-invokes verbs (`06`, design deferred).
- **Bill at the capability executor.** Each verb declares a `billing_unit`; the executor charges the workspace owner once per billable success via the billing service (`WebCrawlCreditService`, 03c, first provider) — so chat, ongoing automation, REST, and MCP all meter **uniformly**. `maps.*` registers a new per-place / per-page unit.
- Obsidian and Circleback (push/webhook sources) are DISABLED for the MVP.
- MCP-availability audit complete: BookStack (community MCP servers), Elasticsearch (official Elastic Agent Builder MCP), and Luma (community MCP servers) all have MCP available, so none are `DISABLED` — they're tagged `MIGRATING` (turned off for MVP like the other branded natives, pending the post-MVP MCP re-point).
- ~~`Pipeline` and `PipelineRun` are new first-class tables.~~ **SUPERSEDED** — no Pipelines, and (2026-07-01) no Tracker/Timeline either. Standing needs are a **persistent ongoing chat** re-invoking verbs (`06`, deferred); no new scheduler, no new run/state table. Uploads simply populate the input-only KB (existing upload code, no pipeline wrapper).
- ~~The chat agent gets read-only access to pipeline run history~~ → the agent answers "what changed?" by **reasoning over the chat history** (prior tool outputs already in context, `07`), not by re-deriving from run logs or a stored timeline.
- **Web search + WebURL crawler are platform-native** (agent tools now; developer platform-API-key access later). The KB is **input-only**.
- Deferred (post-MVP): platform scraper implementations (**Phase 8 — Platform actors**, public-data-only first; built in-house on the Phase-3 fetch core) surfaced as **native tools / API endpoints**, public pay-as-you-go API over the crawler/search, public MCP server exposing the KB. Logged-in/account-based scraping deferred beyond that.
## Platform connector research list (deferred build, MVP = "coming soon")
- LinkedIn — people profiles (discovery by keyword/company), company info, job listings.
- Amazon — product (ASIN), search (keyword), pricing; reviews secondary.
- Google — Web Search (organic SERP), AI Overviews, Maps/Local (discover by location).
- Instagram — profiles first, then posts; discover profiles by username/keyword.
- Zillow / Redfin — full property listings (discover by search URL/filters); Zillow price history.
- Walmart — product, search; zipcode-localized pricing premium variant.
- eBay — search by keyword/category; price-comparison/resale feeds.
- Crunchbase — company info, search by keyword (B2B lead-gen / investor research).
- TikTok / YouTube — profiles/channels, posts/videos; discover by keyword/hashtag; TikTok Shop.
- Indeed / Glassdoor — job listings (discover by keyword in location), company reviews.
## Backend phases (active — this umbrella)
### Phase 1 — Rename foundation (DB) [`subplan: 01-rename-db.md`]
> **✅ SHIPPED** (2026-06-27) · branch `feat/rename-searchspace-to-workspace` · PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546) (merged to `ci_mvp` via [#1562](https://github.com/MODSetter/SurfSense/pull/1562)). Migration `170` (chains `169`, current head) does the physical rename + Zero-publication reconcile. Shipped atomically with Phase 2. As-built record + re-runnable verification live in `01-rename-db.md`. **Deploy caveat:** zero-cache replica reset (`ZERO_AUTO_RESET`) required; from-scratch `alembic upgrade head` stays pre-existing-broken (rev 23 conflict — separate baseline-squash task), only the `169→170` path is verified.
- Alembic migration: rename `searchspaces` -> `workspaces`; rename `search_space_id` -> `workspace_id` on ~20 child tables; rename RBAC tables and their FKs; rename indexes/constraints (`uq_searchspace_*`, `idx_documents_search_space_id`, etc.); update Rocicorp Zero publication column lists (backend-owned `publication` definition; frontend Zero schema rename happens in the later frontend umbrella).
- Decide transition strategy: hard cutover (simplest for MVP) vs temporary API aliases for clients.
- Key files: `surfsense_backend/app/db.py`, `surfsense_backend/alembic/versions/` (new migration).
### Phase 2 — Rename backend (code + API) [`subplan: 02-rename-backend.md`]
> **✅ SHIPPED** (2026-06-27) · branch `feat/rename-searchspace-to-workspace` · PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546) (merged to `ci_mvp` via [#1562](https://github.com/MODSetter/SurfSense/pull/1562)). Symbolic rename across `app/` + `tests/` (Phase-1 shim dropped), API consolidated to `/workspaces` (legacy `/searchspaces` · `/search-spaces` · `/search-space` all retired/404). Verified (ground-truth `git grep`, 2026-06-29): every residual `search_space`/`SEARCH_SPACE` is a documented carve-out — enum values (`ConnectionScope`/`ChatVisibility.SEARCH_SPACE`), the `'SEARCH_SPACE'` CHECK literal (now paired with `workspace_id`), Celery wire names (`delete_search_space_background`, `ai_sort_search_space`), OTel key `search_space.id`, and the `SEARCH_SPACE_FORBIDDEN` error code; `alembic/versions/` untouched except `168`+`170`. Suite: `3016 passed, 1 skipped`. As-built record in `02-rename-backend.md`. **Clients are intentionally broken** until the frontend/satellite umbrellas land (hard cutover).
- Rename models/schemas/services/routes/agents/tasks identifiers: `SearchSpace*` -> `Workspace*`, `search_space_id` -> `workspace_id`.
- Consolidate API to `/workspaces` and fix the `/searchspaces` vs `/search-spaces` inconsistency.
- High-touch files: `routes/search_spaces_routes.py`, `routes/rbac_routes.py`, `utils/rbac.py` (`check_search_space_access`), `schemas/search_space.py`, plus `search_space_id` threading through agents/Redis keys/storage paths (`documents/{id}/...`).
### Phase 3 — WebURL Crawler & Crawl Billing (backend) [`subplans: 03a–03f`]
The Universal WebURL Crawler is the flagship Type-1 data source (**the moat**). This phase hardens it into an **in-house, best-effort "undetectable, captcha-bypassing" crawler** on a single framework (Scrapling): it standardizes the fetch layer, generalizes proxy support, introduces pay-as-you-go crawl credits, adds **stealth hardening** (geoip coherence, persistent profiles, headed/Xvfb, fonts, humanization, a block classifier + per-domain strategy memory), and adds **opt-in captcha solving**. All tiers plug in behind a single `FetchStrategy` seam returning `CrawlOutcome`, so callers never depend on *how* a page was fetched — that seam is what lets the moat grow (and lets a **deferred paid-unblocker tier** drop in later by config). **Strategy decision (recorded in the log):** CloakBrowser is rejected on licensing and external unblocker APIs are deferred; we hold an in-house bypass moat for ~4–6 months, then move hostile targets to a paid tier if demand/maintenance justifies it. **Logged-in/account-based bypass is out of scope** (public data only this MVP). It is broken into focused subplans:
> **Sequencing within Phase 3 (critical path vs hardening).** Only **`03a` + `03b` + `03c`** are on the **MVP critical path**. The crawler (`03a` `CrawlOutcome`) + proxy provider (`03b`) + `03c` `WebCrawlCreditService` billing seam are consumed by the **capability verbs** (`04`) that back every chat / automation / REST / MCP path (the crawler is a platform-native tool — see Architecture pivot; the old "Phases 4–7 pipelines consume this" framing is superseded). **`03d` (captcha), `03e` (stealth hardening), and `03f` (test harness) are hardening/measurement that nothing downstream imports** — they tune the *same* seam behind the *same* contract, so they can land **in parallel with, or after, Phase 4** without blocking the pivot. Recommended build order: `03a → 03b → 03c` (then `04` Capabilities → `05` Access → `07` Orchestration), with `03e → 03d → 03f` slotted in whenever crawler robustness is prioritized.
- **`03a-crawler-core.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `5c36cd3`; crawler moved to `app/proprietary/web_crawler/`, `impersonate="chrome"` + `solve_cloudflare=True` shipped)* — Standardize the fetch layer on Scrapling. **Remove Firecrawl entirely** (no other frameworks). Define crisp per-URL success/empty/failure semantics, keep Trafilatura extraction, and expose a single billable "successful crawl" signal (one unit per URL that yields usable content, regardless of how many internal fallback tiers ran).
- **`03b-proxy-expansion.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `6226012`)* — Add a BYO `CustomProxyProvider` (the only new provider — **no branded vendors**) alongside `anonymous_proxies`, selectable via a **single, app-wide** `Config.PROXY_PROVIDER`. Add bounded client-side rotation+retry via Scrapling's `ProxyRotator`/`is_proxy_error` **only** when the active provider is pool-backed (`CUSTOM_PROXY_URLS`); single-endpoint providers (incl. `anonymous_proxies`) stay the default and no-op the retry. **No per-connector/per-crawl selection** (one provider app-wide); a per-capability override is left as a no-op seam (`04`), not built.
- **`03c-crawl-billing.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `17bdb0682`)* — Charge crawl credits at **$1 / 1000 successful requests = 1000 micro-USD per successful crawl** (config-driven via `WEB_CRAWL_MICROS_PER_SUCCESS`, retunable with no code change), drawn from the existing credit wallet (`credit_micros_balance`), gated by a new `WEB_CRAWL_CREDIT_BILLING_ENABLED` flag (off for self-hosted). Billing surfaces (**reframed** by the Architecture pivot + revamp — see log): the charge moves to the **capability executor** (`04`), so **every** caller meters uniformly — chat, ongoing automation, REST, and MCP — billing the **workspace owner** via `WebCrawlCreditService`. The interactive **chat turn accumulator** becomes an *optional presentation fold* (so chat still shows the crawl line on the turn bill), **not** the charging mechanism. The original **connector/pipeline-indexer** billing branch (`webcrawler_indexer`) is now **vestigial** (WEBCRAWLER connector retired, no pipelines) — kept dormant, not deleted. The `WebCrawlCreditService` (mirrors `EtlCreditService`'s gate → `check_credits` → `charge_credits`) is unchanged. No DB migration (uses the existing free-form `web_crawl` usage_type).
- **`03e-stealth-hardening.md`** — The in-house "undetectable" layer on top of Scrapling's default patchright-Chromium stealth. **Geoip coherence** (match browser `locale`/`timezone_id` to the proxy's exit geo), **fingerprint flags** (`hide_canvas`/`block_webrtc`), **persistent per-domain profiles** (`user_data_dir`), **headed execution under Xvfb**, **real fonts** in the worker image, and **DIY behavioral humanization** via `page_action` (the Chromium engine has no built-in `humanize`). Adds a **block classifier** (label Cloudflare/DataDome/Kasada/captcha/empty from the response) + **per-domain strategy memory** (Redis, no migration) so the ladder learns the known-good tier per domain. Defines — but does **not** build — the **deferred paid-unblocker `FetchStrategy`** (ZenRows/ScrapFly/Bright Data) as the config-flagged escape hatch, with its own (later) billing. Honest ceiling: defeats Cloudflare + the moderate long tail, **not** top-tier behavioral fingerprinting (DataDome/Kasada/reCAPTCHA-Enterprise) — that's the deferred tier.
- **`03d-captcha-solving.md`** *(✅ IMPLEMENTED — `ci_mvp`; `captchatools` page_action + per-attempt `web_crawl_captcha` billing, off by default)* — Covers the captcha types Scrapling does **not** (reCAPTCHA v2/v3, hCaptcha, image) via `captchatools`, **opt-in + off by default**. `captchatools` is **itself** the provider registry (`new_harvester(solving_site=…)` across capmonster/2captcha/anticaptcha/capsolver/captchaai), so we do **not** rebuild a provider hierarchy — our layer is thin: config resolution + a StealthyFetcher `page_action` that detects the sitekey, harvests a token (egressing from the **same** proxy IP as the crawl), and injects it. Scrapling already handles Cloudflare Turnstile (`03a`), and `page_action` runs **after** `solve_cloudflare`, so the tiers compose. The **billing asymmetry** is now **resolved**: a **separate per-attempt** `web_crawl_captcha` unit (solvers charge per attempt regardless of crawl success), attached via a `WebCrawlCreditService` seam in `03c`. `ErrNoBalance` stops solving (no retry loop → avoids IP bans). Requires a paid solver account.
- **`03f-undetectability-testing.md`** *(✅ IMPLEMENTED + first baseline — `ci_mvp`, MANUAL-only, no CI gating; lives under `app/proprietary/web_crawler/testbench/`)* — A **manual scorecard harness** that drives the real Scrapling tiers against the industry-standard detection + sandbox sites (modeled on CloakBrowser's `bin/cloaktest` suite) to **quantify the free-stack ceiling** over time. Two labeled axes: **Suite S (stealth/anti-bot)** — browser-tier (`bot.sannysoft`, `bot.incolumitas`, CreepJS, `deviceandbrowserinfo`, FingerprintJS demo, reCAPTCHA-v3 score, `fingerprint-scan`/Castle.js, `browserscan`, **`cloudflare-challenge`** exercising `solve_cloudflare`, **`iphey`** geoip-coherence), HTTP/TLS-tier JA3/JA4 parity (`tls.peet.ws`, **informational** not a gate), and proxy/leak checks (`httpbin/ip`, WebRTC/DNS); **Suite E (extraction correctness)** — toscrape/scrapethissite sandboxes for the HTTP vs JS (DynamicFetcher) tiers. **Every detection site is auto-graded from its real DOM verdict** (parsers written against captured dumps; `INFO` reserved for TLS/IP/manual rows). Reuses `03d`'s `page_action`+closure-cell mechanism. Adopts CloakBrowser's bars as **aspirational** (sannysoft 0 fails, CreepJS ≤30%, reCAPTCHA ≥0.7) while recording **our actual numbers as the baseline** (the whole `results/` tree is **gitignored**, run-local). **First baseline (2026-06-30, headless, rotating residential, captcha OFF): Suite S 6 PASS / 4 FAIL** — PASS sannysoft, deviceandbrowserinfo, reCAPTCHA-v3 (0.9), BrowserScan, fingerprint-scan (35/100), cloudflare-challenge; FAIL CreepJS (`hasHeadlessWorkerUA` worker-UA leak → `03e` Slice-B candidate), FingerprintJS Pro (commercial ceiling), iphey ("Unreliable" → expected geoip-coherence fix), incolumitas (legacy `fpscanner WEBDRIVER` only). The scorecard is the documented **trigger** for flipping `03e`'s deferred paid-unblocker tier.
### Phase 4 — Capabilities (backend) [`subplan: 04-capabilities.md`]
> **Canonical subplan:** [`04-capabilities.md`](04-capabilities.md). **Build first** — every later phase calls the registry.
- Turn Acquisition (crawler + search + Maps actor) into a **small set of typed, callable verbs** over a single **capability registry** — `web.scrape(urls[])`, `web.discover(query, top_k)`, `maps.search/place/reviews`. Each verb is a **direct async call that returns cleaned, AI-ready data** (no job store, no envelope, no polling). `maps.*` returns typed structured objects, not markdown. Each declares a **`billing_unit`**; charging is delegated to the billing service (03c first provider) at the executor so every door meters uniformly. New Apache-2 package `app/capabilities/`; Maps extractor stays proprietary.
**Locked model (MVP):**
- **Capabilities replace connectors as data sources.** `web.*` / `maps.*` are *call → cleaned data → bill* verbs — no `SearchSourceConnector` row, no KB write, no schedule.
- **Surviving connectors = file/KB-input + BYO MCP.** File/KB-input (Google Drive native+Composio, OneDrive, Dropbox, uploads) put the user's personal files into the **input-only** KB. BYO `MCP_CONNECTOR` (act in chat) is the only functional Type-2; its routing-gap fix lands in `05`. **`WEBCRAWLER_CONNECTOR` RETIRED.** The old connector two-type taxonomy (`04a-connector-category`) is **demoted to backward-compat hygiene**.
- **Deprecated (`MIGRATING`, off for MVP):** every branded native — indexers (Notion, GitHub, Confluence, BookStack, Elasticsearch) + act-only (Slack, Teams, Linear, Jira, ClickUp, Airtable, Discord, Gmail, Google Calendar, Luma, + Composio Gmail/Calendar). Existing rows + already-indexed KB docs stay **searchable**; new-create/`/index`/periodic/subagents off. Real MCP migration is post-MVP.
- **Disabled (`DISABLED`):** Obsidian (plugin push) and Circleback (meeting webhook).
- Web search: **drop all 5 search-API `connector_type` values** (`SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API`); survivors (SearXNG/Linkup/Baidu) become the `web.discover` provider set keyed by **platform env** (Linkup/Baidu keys move off per-connector `config`). Destructive migration deletes the 5 types' rows. (Absorbs the old `04b-source-discovery`.)
- The **Google Maps actor is net-new**, proprietary, built as a separate effort; `maps.*` are contracts against it. Other "coming soon" platforms (LinkedIn, Amazon, Instagram, Zillow/Redfin, Walmart, eBay, Crunchbase, TikTok, Indeed/Glassdoor) re-slot as later namespaces, not Type-1 KB connectors.
- Frontend connector/capability UI restructure is DEFERRED (frontend umbrella).
### Phase 5 — Access (backend) [`subplan: 05-access.md`]
> **Canonical subplan:** [`05-access.md`](05-access.md). **Build after `04`.** Together `04 + 05` ship the scraper-API product — revenue day one.
- Expose the registry through **three generated doors** — **REST + API key** (public day one), an **MCP server** (fast-follow), and **chat tools** — each the same thin adapter (`parse → validate → authn/authz → 03c meter-gate → same executor → serialize → cleaned data`). The executor returns directly; there is **no envelope, no `deliverable_wait` polling**. Async results (e.g. the `06` ongoing mode) reach the client via **write-then-sync / stream** (existing SSE or a Zero-published table).
- **Natural language is the human surface** (raw verbs only on REST/MCP); the chat door classifies **one-shot** vs **keep-watching** and hands keep-watching to `06`. Two MCP directions kept distinct: **serve** our verbs as an MCP server (new door) vs **consume** the user's BYO `MCP_CONNECTOR` (the old-`04a` routing-gap fix lands here).
### Phase 6 — Ongoing-Automation (backend) [`subplan: 06-ongoing-automation.md`]
> **Canonical subplan:** [`06-ongoing-automation.md`](06-ongoing-automation.md) — **shipped.** A chat watch is an `Automation` bound to the current chat (`schedule` + `chat_message`); the durable checkpointer is the memory. **Depends on `04`** (the verbs it re-invokes) and `05` (the chat surface + delivery channel).
- Support **"keep watching"** with no stateful storage: a **persistent, ongoing chat** where the agent periodically re-invokes scraper verbs and drops results into the session; the agent reports "what's new" by reading the chat history. Open questions (resolved together): periodicity driver, delivery channel (SSE vs Zero-published table), context-window limit, loop owner, stop/cost controls.
### Phase 7 — Orchestration (backend) [`subplan: 07-orchestration.md`]
> **Canonical subplan:** [`07-orchestration.md`](07-orchestration.md) — the human-facing brain. **Shipped for the web verbs** (atop `04`/`05`/`06`). We plug into the shipped multi-agent runtime; we don't rebuild it.
- Ship the builtin **`scraping`** subagent (peer to `research`/`deliverables`): **intent routing** (one-shot vs keep-watching, block on ambiguous cadence), **verb composition** (`web.discover → web.scrape`; `maps.*` chains deferred until the Maps actor exists), and **"what changed" from the chat history** (re-invoke a verb, compare against prior tool outputs already in context — no timeline, no diff store).
- Toolset: registry-backed capability verbs + `start_watch`/`stop_watch`/`refresh_watch` bound to the current chat (`06`). Tools follow the shipped `scrape_webpage` shape (executor + door + 03c billing), **direct-return** (no `deliverable_wait`).
### (Future) Platform API-key + public MCP
Platform **API-key** access to the capability verbs + a public **MCP server** exposing the input-only KB — post-MVP (see Phase 8 + Deferred).
### Phase 8 — Platform actors (FUTURE — post-MVP, public data only) [`subplan: TBD`]
NOT planned in this umbrella; recorded so the Phase-3 architecture stays aimed at it. Google Maps is the **committed MVP actor** (a `maps.*` namespace, Phase 4). Beyond it, more **platform scrapers** — a family of **individual scraping endpoints** (LinkedIn public profiles/companies, Amazon products, etc. — see "Platform connector research list"; **examples, none committed**) — layer **on top** of the hardened fetch core once it's solid. **Each is simply another capability verb** (`04`), so registering one automatically yields (a) an **agent tool** and (b) a **dev-callable REST endpoint behind the platform API key** — same executor, same billing, no new machinery. Each is a per-platform structured extractor built in-house "Apify-style" on the `03a` crawler core (proxies + `03e` hardening + `03d` captcha). Billed per call via `WebCrawlCreditService`.
- **Public data only** at first — discovery/extraction of publicly visible pages. **Logged-in/account-based bypass is explicitly deferred** beyond Phase 8's first cut; it needs sticky/static proxies + credential management (`03b` static-proxy hand-off) and is the higher-risk, later workstream.
- The **deferred paid-unblocker tier** (`03e §8`) is the fallback for any platform whose anti-bot exceeds the in-house ceiling.
## Deferred — Frontend & client phases (separate umbrella, planned LATER)
These are recorded for continuity but are NOT planned in this umbrella. They start once the backend phases above are working.
- Frontend rename + i18n: route segment `[search_space_id]` -> `[workspace_id]`, `search-space-settings/` -> `workspace-settings/`, TS types, api services, Jotai atoms, components, cache keys, and "Workspace" copy across 5 locales (`messages/{en,zh,es,pt,hi}.json`), plus frontend Zero schema rename.
- Satellite/client apps + docs rename: `surfsense_desktop`, `surfsense_obsidian`, `surfsense_browser_extension`, `surfsense_evals`, README/docs.
- Connector two-type UI: restructure `connector-popup` and `connector-constants.ts` into the two labeled types.
- Scraper-API + chat + positioning UI (**replaces the superseded "Pipelines UI"**): the developer surface for the scraper endpoints (API keys, usage/billing), the chat experience (one-shot answers + "keep watching" ongoing chats), `web.discover` source-suggestion UX, input-only KB framing (uploads + file connectors), MCP connector management, "coming soon" platform cards.
## Open items to confirm during subplanning
- ~~Rename transition: hard cutover vs temporary API aliases~~ RESOLVED: HARD CUTOVER (see resolved log + 02-rename-backend.md). The frontend is rebuilt against the corrected backend in its own umbrella; backend is verified via tests/OpenAPI, not the old UI.
- ~~Whether existing connector periodic-indexing config is migrated into Pipelines or coexists during MVP.~~ **MOOT** (Architecture correction — no pipelines). Connector periodic indexing stays as-is for file sources; there is no pipeline scheduler to coexist with.
- ~~Chat agent run-history access: tool vs middleware injection vs both (default: tool).~~ **MOOT** (Architecture correction — no pipeline runs). The agent's recurring work is automations, whose history is already persisted + queryable (`automation_runs`).
- ~~Type-2 MCP migration depth~~ RESOLVED (Phase 4): branded natives are tagged `MIGRATING` and turned OFF for MVP (not re-pointed to MCP yet); only the generic `MCP_CONNECTOR` is a functional Type-2. Real MCP re-pointing is post-MVP.
- **Revamp open items (decide during subplanning):** (1) the **Ongoing-Automation mechanism** (`06`, deferred) — periodicity driver, delivery channel (SSE vs Zero-published table), context-window limit, stop/cost controls; (2) how large `web.scrape` / `maps.reviews` responses are bounded/streamed (`04`); (3) is `web.discover` metered or free (`04`); (4) MCP-serve auth depth — OAuth 2.1 vs bearer (`05`); (5) public REST rate-limiting / abuse posture (`05`).
## Resolved decisions log
- **PIVOT (2026-07-01) — scraper-APIs-first, stateless, chat-native; Tracker/Timeline/Triggers dropped.** The product is the **scraper capabilities** (`web.*` + `maps.*` typed verbs → cleaned, AI-ready data), exposed as **REST + API key / MCP / chat** generated from one registry — premium, OSS, self-hostable, the revenue driver. **Stateless:** no Timeline (3-store delta), no `Tracker`, no diff/materiality engine, no stored `entity_changes`; memory = the **chat history** (the agent reasons over prior tool outputs for "what changed"). **Direct calls:** capabilities return their result directly — no job store, no `completed | pending + job_id` envelope, no `deliverable_wait` polling; async delivery is **write-then-sync / stream**. **Automation = a persistent ongoing chat** re-invoking verbs (replaces `Triggers`; mechanism **design-deferred**). KB stays **input-only**. Phase re-cast: **04 Capabilities · 05 Access · 06 Ongoing-Automation (deferred) · 07 Orchestration** (`scraping`). Subplans `05a-timeline`/`05b-intelligence`/`06-triggers` **removed** (git history); `04a`/`04b` folded into `04`/`05`. This **supersedes** the 2026-06-30 revamp's Tracker/Timeline/Triggers decisions below.
- **REVAMP ADOPTED as canonical (2026-06-30) — Phases 4–7 re-cast; `revamp phases 4-7/` is the source of truth.** A principal-engineer review found the flat `04a–07` files and my earlier "Phase 5′" framing had been overtaken by a more complete engineer draft (then in a `revamp phases 4-7/` folder, since flattened into `04`–`07`). We adopt the revamp: **Phase 4 = Capabilities & Access** (typed verbs + generated doors; old `04a-connector-category` demoted to hygiene, old `04b-source-discovery` absorbed as `web.discover`), **Phase 5 = Intelligence & Timeline** (the `Tracker` + 3-store delta Timeline — the moat), **Phase 6 = Triggers** (reuse automations via a CI action), **Phase 7 = Orchestration** (`scraping`). The three architecture-correction decisions land in their real homes and one is revised: (1) **bill at the capability executor** — a code-verified gap review confirmed `run_agent_task` (automation `agent_task`) sets up **no turn accumulator**, so today `scrape_webpage` bills nothing under automations; billing moves to the executor so chat/automation/REST/MCP/cron meter uniformly (chat turn accumulator becomes optional presentation). (2) **"diffable run history" = the Timeline delta store** (`revamp/05a`), **superseding** the earlier "structured `automation_runs.output`" decision. (3) **"run now" = a Trigger adapter** (`revamp/06`). The flat `04a/04b/05/06/07` files are retained with redirect banners; **`revamp/` filenames win** on any number collision. Build order: `04a → 04b → 05a → 05b → 06 → 07`.
- **ARCHITECTURE CORRECTION (2026-06-30) — Pipelines dropped; automations + input-only KB adopted.** The "sync into KB → operate over it" paradigm was judged wrong for CI. Verified against `references/opencode` (live tool-fetched context + persisted sessions, no scraped corpus; `tool/websearch.ts` live-crawls at query time). Decisions: (1) **KB is input-only** — user's personal files/context (uploads + Drive/Dropbox/OneDrive); nothing scrapes into it. (2) **Web search + WebURL crawler are platform-native tools** (already `web_search`/`scrape_webpage`), later a developer platform-API-key surface — NOT connectors, NOT pipelines. (3) **Connectors = MCP tools + file/KB-input connectors**; all branded natives → MCP. (4) **Automations are the scheduling + run-history substrate** (an `agent_task` run already invokes the full chat agent with the crawler/search tools on cron/event and persists `automation_runs`). (5) **`WEBCRAWLER_CONNECTOR` RETIRED.** Consequences: **Phases 5/6/7 SUPERSEDED** (docs kept with banners); **Phase 4a reframed** (drop `is_pipeline_eligible`; Type-1 = file/KB-input only; WEBCRAWLER retired); **Phase 4b stands**; new **Phase 5′** automation-enhancement workstream (bill automation-run crawls; structured `run.output`; wire "run now"); **Phase 8** actors re-slot as native tools/API. `03c` connector-indexer billing branch becomes **vestigial** (billing lives on chat-turn / automation-run / API paths).
- ~~Web search APIs (SearXNG/Linkup/Baidu): repurposed as source-discovery helper for the WebURL Crawler (suggest URLs for pipelines)~~ — still repurposed as a source-discovery helper (04b), but "for pipelines" → "for the user / the native crawler tool" (pipelines dropped); not a standalone connector type.
- Tavily and Serper: REMOVED from the search infra. They are dropped as search providers entirely (not repurposed). Phase 4's source-discovery endpoint must build only on the remaining providers (SearXNG, Linkup, Baidu).
- Obsidian + Circleback: disabled for MVP.
- MCP-availability audit: BookStack, Elasticsearch, Luma all have MCP available -> eligible for Type-2 (so deprecated as `MIGRATING`, not `DISABLED`). For MVP they are turned off pending the post-MVP MCP re-point, like the other branded natives.
- Phase 4 connector taxonomy: modeled as a STATIC code registry (`connector_type` -> category/availability), NOT a DB column — no migration in 04a; `is_indexable` is KEPT (orthogonal). Only the generic `MCP_CONNECTOR` is a functional Type-2 for MVP; all branded natives (indexers + act-only, incl. Composio Gmail/Calendar) are `MIGRATING` (new-create blocked, `/index`+periodic+their subagents off, existing KB docs stay searchable). The `MCP_CONNECTOR` subagent routing-map gap (`constants.py`) is fixed in 04a.
- Phase 4 search APIs: all 5 enum values dropped (`SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API`) in 04b. Survivors (SearXNG/Linkup/Baidu) become PLATFORM providers keyed by env (Linkup/Baidu keys move from per-connector `config` to env — app-wide, not per-workspace). 04b carries a destructive migration deleting the 5 connector types' rows.
- Phase 4 structure: split into 04a (taxonomy/gating/MCP-fix, no migration) and 04b (search repurposing + source-discovery endpoint, with migration); intended order 04a -> 04b (both orders safe).
- Rename transition policy: HARD CUTOVER of the external API (paths + JSON field names) in Phase 2 — no backward-compat aliases. Rationale: the frontend is (re)built against the corrected backend later, so there is no old client to keep alive; backend correctness is verified via the test suite + OpenAPI rather than the existing UI.
- Crawler code location & licensing boundary (decided during 03a impl): the WebURL crawler engine — and future Phase-8 platform actors — live under `surfsense_backend/app/proprietary/`, a **non-Apache-2 license boundary** (its own `LICENSE`, currently an all-rights-reserved placeholder; the repo root stays Apache-2). 03a's `WebCrawlerConnector` / `CrawlOutcome` / `CrawlOutcomeStatus` moved to `app/proprietary/web_crawler/` (public API re-exported from its `__init__`); the 3 Apache-2 callers (webcrawler indexer + both chat `scrape_webpage` tools) import `from app.proprietary.web_crawler import ...`. Rule: everything under `app/proprietary/**` is non-Apache-2; Apache-2 code may import *from* it but not move *into* it. Rationale: keep the moat under a clearly-bounded, swappable license.
- Proxy code placement (decided during 03b): the proxy provider package (`app/utils/proxy/` — base/registry/`anonymous_proxies`/`CustomProxyProvider`/rotation) **stays Apache-2 shared infra**, NOT proprietary. Rationale: it's consumed by Apache-2 features unrelated to the moat (YouTube transcript route + indexer, chat tools' YouTube branch), and `CustomProxyProvider` is a thin wrapper over Scrapling's *public* `ProxyRotator`. Only the crawl-ladder-coupled rotation-retry (`app/proprietary/web_crawler/connector.py::_run_tier_with_proxy_retry`) lives under the boundary. **Boundary test:** code goes in `app/proprietary/` only if used *exclusively* by the moat (applies to 03e's geoip/sticky-proxy hardening too — the bypass-specific tuning is proprietary; the generic provider plumbing is not).
- WebURL Crawler framework: STANDARDIZE on Scrapling; **remove Firecrawl entirely** (no other scraping frameworks now or planned). Scrapling's `StealthyFetcher` (patchright-Chromium as of 0.4.9 — **not** Camoufox) handles Cloudflare; `03e` stealth-hardening minimizes challenges; captcha-tools (`03d`) covers the rest. All fetch tiers sit behind a `FetchStrategy` seam returning `CrawlOutcome` (callers never depend on the tier).
- Crawl billing: reuse the existing credit wallet (`credit_micros_balance`) with a new `web_crawl` usage_type. Price: **$1 / 1000 successful requests** (1000 micro-USD per success). Connector/pipeline crawls bill the **workspace owner**; chat scrapes fold their crawl cost into the already-billed chat turn. Gated by `WEB_CRAWL_CREDIT_BILLING_ENABLED` (off for self-hosted); no DB migration required.
- Billable unit: one unit per URL that returns usable extracted content, regardless of how many internal fallback tiers were attempted (not per HTTP fetch, not per URL-processed).
- Captcha solving (captcha-tools): **ACTIVE** (no longer deferred) — sequenced last in Phase 3 (`03d`), **after** `03e` hardening, **opt-in + off by default**. Cloudflare stays in-framework (`03a`); reCAPTCHA/hCaptcha/image use `captchatools`. **Billing asymmetry RESOLVED → option (a): a separate per-attempt `web_crawl_captcha` unit** (`WEB_CRAWL_CAPTCHA_*` knobs on `WebCrawlCreditService`), since solvers charge per attempt regardless of crawl success. `ErrNoBalance` halts solving (no retry-loop IP bans).
- Crawler stealth strategy (the moat): **CloakBrowser REJECTED** (source-patched Chromium binary requires an OEM/SaaS license incompatible with our model). **External unblocker APIs DEFERRED** (ZenRows/ScrapFly/Bright Data) — pre-wired as a config-flagged `FetchStrategy` (`03e §8`) but not built. Plan: maintain an **in-house bypass moat for ~4–6 months** (Scrapling stealth + residential proxies + `03e` hardening + `03d` captcha), then move hostile/top-tier-fingerprinted targets (DataDome/Kasada/reCAPTCHA-Enterprise) to a paid tier if demand/maintenance justifies it. Realistic ceiling acknowledged: in-house beats Cloudflare + the moderate long tail, not top-tier behavioral fingerprinting.
- Authenticated/logged-in scraping: **OUT OF SCOPE this MVP** (public data only). Sticky/static proxies + credential management are deferred and paired with the future platform actors (`03b` static-proxy hand-off + Phase 8).
- Phase 3 stealth-hardening subplan `03e` ADDED: geoip locale/tz coherence, `hide_canvas`/`block_webrtc`, persistent per-domain profiles, headed+Xvfb, fonts, DIY humanization (`page_action`; Chromium engine has no built-in `humanize`), a block classifier, and per-domain strategy memory (Redis, no migration).
- Phase 3 test-harness subplan `03f` ADDED: **manual-only** (no CI/automated gating now) undetectability + extraction scorecard, modeled on CloakBrowser's `bin/cloaktest`. Two labeled axes (Suite S stealth + Suite E extraction) so they scale independently. Drives the **real** Scrapling tiers (browser + curl_cffi HTTP/TLS), reuses `03d`'s `page_action`+closure-cell for JS-object verdicts. **TLS JA3/JA4 parity = informational axis, not a hard gate.** Adopt CloakBrowser bars as aspirational; record our actual free-stack numbers as the committed baseline. The scorecard is the documented evidence/trigger for flipping `03e`'s deferred paid-unblocker tier.
- Licensing placement of 03e/03f code (decided during 03f impl, applying the §boundary test): the **stealth kwargs builder + geoip coherence** (`03e` bypass tuning) live **proprietary** at `app/proprietary/web_crawler/stealth.py`; the **block classifier** (passive telemetry, public markers) stays **Apache-2** at `app/utils/crawl/classifier.py` (direct analog of the captcha split: proprietary `captcha.py` logic + Apache-2 `app/utils/captcha/` config). The **03f scorecard harness** moved whole to `app/proprietary/web_crawler/testbench/` (run `python -m app.proprietary.web_crawler.testbench`) — it's the moat's measurement tool and can't be cleanly half-moved (a proprietary Suite S would back-import generic scaffolding from `scripts/`, a forbidden app→scripts direction). The `scripts/e2e_phase3_crawl_billing.py` billing e2e stays in `scripts/` (Apache-2) since it exercises billing, not the stealth moat.
- Roadmap: WebURL Crawler & Crawl Billing inserted as the new Phase 3; connector two-type → Phase 4; pipelines → Phases 5/6/7.
- **[SUPERSEDED — Architecture correction 2026-06-30; no pipelines]** Phase 5 pipelines data model: two new tables `pipelines` (mutable) + `pipeline_runs` (append-only), modeled on `automations`/`automation_runs`; ORM lives in `db.py` next to connectors/folders. `connector_id` nullable (NULL = Phase-7 Uploads), eligibility enforced at create via 04a's `is_pipeline_eligible`. Schedule = `schedule_cron` + `schedule_timezone` (default UTC) + `next_scheduled_at` (cron, matching automations). `pipeline_runs` pre-includes `charged_micros`/`crawls_*`/`result_blob_key` so Phase 6 needs no extra migration. Both tables published to Zero **full-row** (like folders/connectors). Routes reuse `CONNECTORS_*` permissions. Phase 5 ships the data model + API surface only; the `/run` endpoint enqueues a Phase-6 task stub.
- **[SUPERSEDED — Architecture correction 2026-06-30; uploads just populate the input-only KB]** Phase 7 uploads-as-pipeline: a **singleton "Uploads" pipeline** per workspace (`connector_id NULL`, `save_to_kb=true`), lazily get-or-created (race-safe via a partial unique index `ON pipelines(workspace_id) WHERE connector_id IS NULL`). Each `fileupload`/`folder-upload` request writes a **terminal audit `PipelineRun(trigger=upload, status=succeeded, documents_indexed=<accepted count>)`** — uploads are **route-recorded, not engine-executed** (Phase 6 fails NULL-connector runs by design; existing upload code stays the executor). Best-effort via an **inner** try/except (never 5xx the upload — the route's outer handler would otherwise 500 an already-committed upload). No crawl billing (uploads aren't crawls; `charged_micros` NULL). Per-file ETL truth stays on `Document.status`; accurate roll-up needs the deferred `documents.pipeline_run_id` provenance. Connector `save_to_kb` default stays `False` (opt-in for connectors, mandatory for uploads). Phase 7 also **guards Phase-5's generic CRUD** against the system Uploads pipeline: `POST /pipelines` rejects `connector_id=None` (supersedes Phase 5's permissive create), `/run` and schedule-`PUT` reject NULL-connector pipelines, and Phase 6's scheduler `_claim_due` filters `connector_id IS NOT NULL` as a backstop (so the Uploads pipeline can never be manually-run or scheduled into perpetually-failing runs). See `07-upload-pipeline-kb.md`.
- **[SUPERSEDED — Architecture correction 2026-06-30; automations are the run engine]** Phase 6 pipeline execution: run engine mirrors **automations** (thin Celery `run_pipeline(run_id)` → `execute_pipeline_run`; PENDING-gated, idempotent terminal no-op; `pending→running→succeeded/failed` with timing/counts/error). MVP executor = **WebURL crawler only** (other types fail cleanly). `save_to_kb=true` reuses `index_crawled_urls` extended with a `folder_id` param (lands in the destination folder); `save_to_kb=false` runs a **fetch-only** loop and persists one JSON blob via `file_storage` (`result_blob_key`). **Crawl billing is owned by the run engine** for the pipeline path (pre-check on `len(urls)` + charge `crawls_succeeded` + idempotent `charged_micros`), calling the crawler with a new `bill=False` seam (the connector `/index`+periodic paths keep `03c`'s in-indexer `bill=True`) — so non-KB runs are billed identically. Scheduler = a `pipeline_schedule_select` Beat tick modeled on the automations cron **selector** (cron + `FOR UPDATE SKIP LOCKED` + self-heal, using the existing `croniter` util), plus the **de-dup guard** (a pipeline over a connector disables that connector's `periodic_indexing_enabled`). Chat context = the `get_pipeline_runs` tool. Carries a small additive `05` amendment: a `schedule_timezone` column (cron util needs a tz). See `06-pipelines-exec.md`.
## Subplan index (backend)
| Phase | Subplan file | Status |
|-------|--------------|--------|
| 1 | `01-rename-db.md` | **SHIPPED** (2026-06-27, PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546)/[#1562](https://github.com/MODSetter/SurfSense/pull/1562); migration `170`) |
| 2 | `02-rename-backend.md` | **SHIPPED** (2026-06-27, PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546)/[#1562](https://github.com/MODSetter/SurfSense/pull/1562)) |
| 3 | `03a-crawler-core.md` | **IMPLEMENTED** (`ci_mvp` @ `5c36cd3`) — Firecrawl removed, Scrapling-only 3-tier `CrawlOutcome`, crawler relocated to `app/proprietary/web_crawler/` |
| 3 | `03b-proxy-expansion.md` | **IMPLEMENTED** (`ci_mvp` @ `6226012`) — `CustomProxyProvider` (BYO single/pool) + registry + bounded rotation-retry |
| 3 | `03c-crawl-billing.md` | **IMPLEMENTED** (`ci_mvp` @ `17bdb0682`) — `WebCrawlCreditService` (config-driven price) + indexer wiring + chat-turn fold; functional e2e green |
| 3 | `03e-stealth-hardening.md` | **Slice A IMPLEMENTED** (`ci_mvp`) — stealth kwargs builder + geoip coherence (**proprietary** `app/proprietary/web_crawler/stealth.py`) + additive block classifier (Apache-2 `app/utils/crawl/`; `CrawlOutcome.block_type`, incl. static-tier 4xx) + Xvfb/fonts in image; Slices B/C deferred (WebGL spoof, humanization, persistent profiles, strategy memory, paid-unblocker) |
| 3 | `03d-captcha-solving.md` | **IMPLEMENTED** (`ci_mvp`) — `captchatools` page_action (proprietary) + Apache-2 config + per-attempt `web_crawl_captcha` billing; off by default |
| 3 | `03f-undetectability-testing.md` | **IMPLEMENTED** (`ci_mvp`) — manual scorecard under the **proprietary boundary** at `app/proprietary/web_crawler/testbench/` (`python -m app.proprietary.web_crawler.testbench`); Suite S (stealth, shipped builder) + Suite E (extraction via real `crawl_url`) + scorecard JSON/MD baseline diff |
| — | `00b-pipeline-diagrams.md` | end-to-end flow diagrams (companion to this umbrella) |
| 4 | `04-capabilities.md` | **IMPLEMENTED (web verbs)** — capability registry + `web.scrape`/`web.discover` → cleaned data, bill at executor (top_k bounds, max_length truncation, per-attempt captcha metering). `maps.*` deferred (Maps actor) |
| 5 | `05-access.md` | **IMPLEMENTED (REST + chat doors)** — generated REST/API-key + chat tools from the registry. MCP door + BYO-MCP routing fix deferred (MCP server) |
| 6 | `06-ongoing-automation.md` | **IMPLEMENTED** — chat-native "keep watching": `Automation` bound to the chat (`schedule` + `chat_message`), worker-safe durable checkpointer, watch tools + REST controls |
| 7 | `07-orchestration.md` | **IMPLEMENTED (web verbs)** — `scraping` subagent; intent routing, verb composition, "what changed" from chat history |
Frontend & client subplans will be added under a separate umbrella later (see "Deferred — Frontend & client phases").

109
plans/backend/00b-pipeline-diagrams.md
View file

@ -1,109 +0,0 @@
# Pipeline diagrams — end-to-end (scraper-APIs-first, stateless)
> Visual companion to `00-umbrella-plan.md`.
> Phase refs: `04` Capabilities · `05` Access · `06` Ongoing-Automation (design deferred) · `07` Orchestration.
## 1. The shape — scraper APIs + a chat agent (no state)
```mermaid
flowchart LR
subgraph FIXED["FIXED · Phases 1-3 (shipped)"]
ACQ["Acquisition<br/>crawler · proxy · stealth · captcha"]
MET["Metering · 03c"]
ID["Identity / API keys"]
end
subgraph SCOPE["OUR SCOPE · Phase 04 -> 07"]
CAP["04 Capabilities<br/>web.* · maps.*<br/>cleaned, AI-ready output"]
ACC["05 Access<br/>REST · MCP · chat doors"]
ORC["07 Orchestration<br/>scraping subagent"]
ONG["06 Ongoing-Automation<br/>(keep-watching)"]
end
ACQ --> CAP
MET --> CAP
CAP --> ACC
ID --> ACC
ACC --> ORC
ORC -. "standing need" .-> ONG
ONG -. "re-invokes verbs" .-> CAP
classDef a fill:#22314f,stroke:#5b7fbf,color:#e6edf7;
classDef b fill:#1f3a2e,stroke:#4f9d76,color:#e6f7ee;
classDef d fill:#3a2f1f,stroke:#bf975b,color:#f7efe6;
class CAP,ACC,ORC a;
class ACQ,MET,ID b;
class ONG d;
```
Memory for "what changed" = the chat history.
## 2. Doors — one registry, three surfaces (generated)
```mermaid
flowchart TD
REG["04 Capability registry<br/>web.scrape · web.discover · maps.search/place/reviews"]
REG --> R["REST + API keys<br/>(public · OSS self-host · revenue)"]
REG --> M["MCP server<br/>(external agents · fast-follow)"]
REG --> C["Chat tools<br/>(in-app agent, 07)"]
R --> EX["same executor · direct return · 03c billing"]
M --> EX
C --> EX
```
## 3. One-shot scrape (the core path) — direct call, no polling
```mermaid
sequenceDiagram
autonumber
participant U as User / Dev
participant D as Door 05 (REST · MCP · chat)
participant EX as Verb executor 04
participant ACQ as Acquisition / Maps actor
participant BILL as Billing (03c)
U->>D: intent (chat) OR raw verb call (REST/MCP)
D->>D: validate input_schema · authn/authz · meter-gate
D->>EX: call executor (same for all doors)
EX->>ACQ: crawl_url / maps / search
ACQ-->>EX: cleaned, AI-ready data
EX->>BILL: charge billing_unit per success
EX-->>D: results (returned directly)
D-->>U: plain-language answer (chat) / JSON (REST/MCP)
Note over U,BILL: Stateless — nothing persists.
```
## 4. Intent fork (in the agent, 07)
```mermaid
flowchart TD
U(["User speaks in natural language"]) --> R{"Intent router<br/>(scraping prompt, 07)"}
R -->|"find / compare / what is / right now"| A["ONE-SHOT<br/>compose verbs, answer (section 3)"]
R -->|"watch / track / notify me / over time"| B["KEEP-WATCHING<br/>start_watch -> Ongoing-Automation (06)"]
R -->|ambiguous cadence| Q["Block: ask for the missing cadence / timezone"]
Q -->|once| A
Q -->|keep watching| B
```
## 5. "What changed" — chat history is the memory
```mermaid
flowchart LR
P["Prior tool outputs<br/>(already in chat context)"] --> AG["scraping subagent"]
NEW["Fresh verb call (04)"] --> AG
AG --> ANS["'Here's what's new since last time'<br/>(reasoned from chat history)"]
```
## 6. Ongoing-Automation (06) — intent only, mechanism deferred
```mermaid
flowchart LR
W["'Keep watching X'"] --> LOOP["Persistent ongoing chat<br/>periodically re-invokes verbs (04)"]
LOOP --> OUT["results delivered into the session<br/>(write-then-sync / stream)"]
OUT --> MEM["chat history = memory<br/>agent reports what's new"]
LOOP -.->|OPEN| Q["periodicity driver? · delivery channel? · context-window limit?"]
classDef d fill:#3a2f1f,stroke:#bf975b,color:#f7efe6;
class W,LOOP,OUT,MEM d;
```
> Section 6 is a **placeholder** — the periodic mechanism is designed separately (see `06-ongoing-automation.md`).

213
plans/backend/01-rename-db.md
View file

@ -1,213 +0,0 @@
# Subplan 01 — Rename foundation (DB)
Part of [00-umbrella-plan.md](00-umbrella-plan.md), Phase 1. Backend only (`surfsense_backend`).
> **Status: SHIPPED** · as of 2026-06-27 · branch `feat/rename-searchspace-to-workspace` · PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546)
> Last commit of this phase: `49d3001fb` (DB rename through migration 170). Phase 1 + Phase 2 shipped as one atomic PR.
> The sections below are the **original design/rationale**; the as-built state + how to re-verify live in the [Implementation record](#implementation-record-as-built) at the bottom. Ground truth for files/commits is the PR, not this doc.
## Goal
Physically rename the `SearchSpace` schema to `WorkSpace` in PostgreSQL via a single Alembic migration, and update the ORM's physical mapping + the Zero publication so the app keeps booting and running — WITHOUT yet touching the ~250 backend files that reference the `search_space_id` Python attribute (that symbolic rename is Phase 2).
Canonical target names:
- Table `searchspaces` -> `workspaces`
- Tables `search_space_roles` / `search_space_memberships` / `search_space_invites` -> `workspace_roles` / `workspace_memberships` / `workspace_invites`
- Column `search_space_id` -> `workspace_id` (all child tables); `external_chat_accounts.owner_search_space_id` -> `owner_workspace_id`
- Named constraints/indexes containing `searchspace` -> `workspace` (see inventory)
## The central coupling decision (read first)
A physical column rename instantly breaks every line of code that reads the old name. Today that is large:
- `search_space_id` / `searchspaces` appear in ~250 backend files outside `db.py` (Grep over `surfsense_backend/app`, excluding `db.py`). Almost all are Python ORM attribute reads like `Document.search_space_id` or `.search_space_id`.
To keep Phase 1 self-contained and low-risk, decouple the PHYSICAL rename from the SYMBOLIC rename using an explicit column-name mapping shim:
- Keep the Python attribute name `search_space_id`, but bind it to the new physical column:
- Today: `search_space_id = Column(Integer, ForeignKey("searchspaces.id", ondelete="CASCADE"), ...)` ([surfsense_backend/app/db.py](surfsense_backend/app/db.py) line 1382 etc.)
- Phase 1: `search_space_id = Column("workspace_id", Integer, ForeignKey("workspaces.id", ondelete="CASCADE"), ...)`
- Result: the physical DB is fully renamed, the ORM maps the unchanged Python attribute to the new column, and the ~250 callers keep working untouched.
- Phase 2 then renames the Python attribute (`search_space_id` -> `workspace_id`), the class (`SearchSpace` -> `Workspace`), schemas, routes/API, and drops the explicit `"workspace_id"` first-arg shim.
CONFIRMED: ship the shim approach. (Alternative big-bang rename rejected as too risky.)
## Current-state inventory (all citations from code)
### The table itself
- `class SearchSpace(... __tablename__ = "searchspaces")` — [surfsense_backend/app/db.py](surfsense_backend/app/db.py) line 1688.
### Child tables with a `search_space_id` FK to `searchspaces.id`
All declared `ForeignKey("searchspaces.id", ondelete="CASCADE")` unless noted. Locations in [surfsense_backend/app/db.py](surfsense_backend/app/db.py):
- `new_chat_threads` (605), `external_chat_bindings` (897), `token_usage` (1113), `folders` (1318), `documents` (1382), `video_presentations` (1506), `reports` (1533), `connections` (1561, nullable), `image_generations` (1672), `search_source_connectors` (1878), `logs` (1905), `search_space_roles` (2045), `search_space_memberships` (2076), `search_space_invites` (2118), `prompts` (2176), `agent_action_log` (2495), `document_revisions` (2566), `folder_revisions` (2607), `agent_permission_rules` (2643).
- `external_chat_accounts.owner_search_space_id` (785-786, nullable) — note the distinct column name.
In other model modules:
- `document_files` — [surfsense_backend/app/file_storage/persistence/models.py](surfsense_backend/app/file_storage/persistence/models.py) line 32.
- `automations` — [surfsense_backend/app/automations/persistence/models/automation.py](surfsense_backend/app/automations/persistence/models/automation.py) line 27.
- `notifications` — [surfsense_backend/app/notifications/persistence/models.py](surfsense_backend/app/notifications/persistence/models.py) line 50.
- `podcasts` — [surfsense_backend/app/podcasts/persistence/models.py](surfsense_backend/app/podcasts/persistence/models.py) line 71.
### Named constraints / indexes that embed the old name
- `uq_searchspace_user_connector_type_name` on `search_source_connectors` — [db.py](surfsense_backend/app/db.py) line 1831.
- `uq_searchspace_role_name` on `search_space_roles` — line 2032.
- `uq_user_searchspace_membership` on `search_space_memberships` — line 2069.
- `idx_documents_search_space_id` and `idx_documents_search_space_updated` (raw `CREATE INDEX CONCURRENTLY` DDL in a module-level list) — lines 2821-2830.
- `ix_external_chat_bindings_search_space_state` — line 978-979.
These reference `search_space_id` by COLUMN but the index/constraint NAME does not embed "searchspace" (definitions auto-follow a column rename; names can stay):
- `ix_notifications_user_space_created` — [notifications/persistence/models.py](surfsense_backend/app/notifications/persistence/models.py) line 36-41.
- `uq_agent_permission_rules_scope` — [db.py](surfsense_backend/app/db.py) line 2680.
### CHECK constraint referencing the column AND a scope literal
- `ck_connections_scope_owner` on `connections` — [db.py](surfsense_backend/app/db.py) lines 1578-1584:
- `"(scope = 'GLOBAL' AND search_space_id IS NULL ...) OR (scope = 'SEARCH_SPACE' AND search_space_id IS NOT NULL ...) OR (scope = 'USER' ...)"`.
- The physical CHECK expression auto-follows a column rename (Postgres tracks by attnum), but the model's `CheckConstraint(...)` text must be updated to `workspace_id` for create_all/autogenerate consistency.
- The scope string literal `'SEARCH_SPACE'` is a `ConnectionScope` value, not a column — see Confirmed decisions.
### Zero publication (single source of truth)
- [surfsense_backend/app/zero_publication.py](surfsense_backend/app/zero_publication.py) is "the single source of truth for `zero_publication`" (module docstring, lines 1-11). Publication changes "should update `ZERO_PUBLICATION` and call `apply_publication()` from a migration" (lines 3-5).
- `search_space_id` appears in **four** published column lists (a recent `main` merge added the `automations`/`new_chat_threads` entries — previously only `documents`/`podcasts`):
- `DOCUMENT_COLS` line 31.
- `AUTOMATION_COLS` line 57 (`["id", "search_space_id"]`).
- `NEW_CHAT_THREAD_COLS` line 62 (`["id", "search_space_id"]`).
- `PODCAST_COLS` line 76.
- (`AUTOMATION_RUN_COLS` lines 44-53 does **not** contain the column — `automation_runs` is scoped via `automation_id`, so it auto-follows and needs no edit.)
- These are the only **four** published tables with column lists that include the column (`ZERO_PUBLICATION` map, lines 81-94; other entries are `None` = all columns and auto-follow). All four (`documents`/`automations`/`new_chat_threads`/`podcasts`) columns must be updated to `workspace_id`, then `apply_publication(op.get_bind())` re-run.
- Reconcile pattern to copy: [surfsense_backend/alembic/versions/155_reconcile_zero_publication.py](surfsense_backend/alembic/versions/155_reconcile_zero_publication.py) (calls `apply_publication(op.get_bind())`, no-op downgrade).
## Migration design
### Conventions (from the repo)
- Revision ids are plain integers; the new migration chains after the **then-current head** — verify with `alembic heads` at implementation time, since the head advances as other PRs merge. As of the latest `main` sync the head is `169` ([surfsense_backend/alembic/versions/169_migrate_google_oauth_account_ids_to_sub.py](surfsense_backend/alembic/versions/169_migrate_google_oauth_account_ids_to_sub.py); chain `166→167→168→169`), so the new file would be `170_rename_searchspace_to_workspace.py` with `revision="170"`, `down_revision="169"`. (Note: Phase 4b and Phase 5 each add a migration too; whichever lands first takes the next integer — chain by actual head, not by these illustrative numbers.)
- Migrations run one-per-transaction under an advisory lock — [surfsense_backend/alembic/env.py](surfsense_backend/alembic/env.py) lines 77, 80-95. So all renames below land atomically.
- Raw `op.execute("ALTER TABLE ...")` is the house style (e.g. [165_add_chunk_position.py](surfsense_backend/alembic/versions/165_add_chunk_position.py) lines 39-49).
### upgrade() steps (order matters)
1. Rename tables:
- `ALTER TABLE searchspaces RENAME TO workspaces;`
- `ALTER TABLE search_space_roles RENAME TO workspace_roles;`
- `ALTER TABLE search_space_memberships RENAME TO workspace_memberships;`
- `ALTER TABLE search_space_invites RENAME TO workspace_invites;`
- FKs and PKs follow the table automatically (referenced by OID). Sequence `searchspaces_id_seq` and `*_pkey` keep old names — CONFIRMED: also `ALTER SEQUENCE searchspaces_id_seq RENAME TO workspaces_id_seq;` and `ALTER INDEX searchspaces_pkey RENAME TO workspaces_pkey;` (plus the RBAC tables' `*_id_seq` / `*_pkey`) for a clean final schema.
2. Rename columns on every child table (use `ALTER TABLE <t> RENAME COLUMN search_space_id TO workspace_id;` for the full list in the inventory; plus `external_chat_accounts.owner_search_space_id -> owner_workspace_id`).
- Metadata-only, fast even on large tables (same property the repo relies on in [165](surfsense_backend/alembic/versions/165_add_chunk_position.py) lines 36-41).
- ORDERING NOTE: the RBAC tables were renamed in step 1, so their column rename must target the NEW table names (`workspace_roles` / `workspace_memberships` / `workspace_invites`), not the old ones.
3. Rename the named constraints/indexes. Use defensive/idempotent forms because some objects are created at runtime, not by migrations (see review finding 4):
- Constraints (no `IF EXISTS` for `RENAME CONSTRAINT`; wrap each in a guarded `DO` block that checks `pg_constraint`):
- `uq_searchspace_user_connector_type_name` -> `uq_workspace_user_connector_type_name` (on `search_source_connectors`).
- `uq_searchspace_role_name` -> `uq_workspace_role_name` (on `workspace_roles`).
- `uq_user_searchspace_membership` -> `uq_user_workspace_membership` (on `workspace_memberships`).
- Indexes (these two are created by the runtime `setup_indexes()` startup routine, NOT a migration, so they may or may not exist depending on `DB_BOOTSTRAP_ON_STARTUP`; use `IF EXISTS`):
- `ALTER INDEX IF EXISTS idx_documents_search_space_id RENAME TO idx_documents_workspace_id;`
- `ALTER INDEX IF EXISTS idx_documents_search_space_updated RENAME TO idx_documents_workspace_updated;`
- `ALTER INDEX IF EXISTS ix_external_chat_bindings_search_space_state RENAME TO ix_external_chat_bindings_workspace_state;`
4. Zero publication. DECISION: we will recreate the Zero replication (reset the zero-cache replica) as part of this deploy, so consumer recovery is handled out-of-band and finding 3 is de-risked. Follow the repo's established publication patterns — do NOT use raw `DROP`/`CREATE PUBLICATION` (forbidden by [116_create_zero_publication.py](surfsense_backend/alembic/versions/116_create_zero_publication.py) lines 8-17; reintroduces bug #1355). Sequence inside the migration:
- (Safe ordering) Neutralize the column-list dependency surgically so the RENAME is unconditionally permitted: `ALTER PUBLICATION zero_publication DROP TABLE documents, automations, new_chat_threads, podcasts;` (all **four** column-list tables). This removes ONLY their column-list dependency and is far safer than re-emitting the whole member set via `SET TABLE` (which is a full replacement and would drop any table accidentally omitted from the hand-written list, including the quoted `"user"` table). `DROP TABLE` from a publication is within the blessed ALTER-PUBLICATION family — it is NOT the forbidden raw `DROP/CREATE PUBLICATION`.
- Rename the columns (steps 1-2).
- Update canonical `DOCUMENT_COLS`/`AUTOMATION_COLS`/`NEW_CHAT_THREAD_COLS`/`PODCAST_COLS` to `workspace_id` in [zero_publication.py](surfsense_backend/app/zero_publication.py) (31, 57, 62, 76), then call `apply_publication(op.get_bind())` (blessed plain `ALTER ... SET TABLE`, as used by [155_reconcile_zero_publication.py](surfsense_backend/alembic/versions/155_reconcile_zero_publication.py)) to re-add all four dropped tables with the narrowed `workspace_id` column lists and reconcile the full member set in one shot.
- Reference template for forcing a schema-change event (only if event triggers are unavailable): the `COMMENT ON PUBLICATION` bookend trio in [143_force_zero_publication_resync.py](surfsense_backend/alembic/versions/143_force_zero_publication_resync.py) lines 104-137. With event triggers installed (current setup), `apply_publication` is sufficient.
- Replica recreate (operational, out-of-band): reset the `surfsense-zero-cache` replica (delete the volume / rely on `ZERO_AUTO_RESET=true`) so zero-cache does a fresh initial sync from the corrected publication. This is the "recreate replication" step and the primary consumer-recovery mechanism.
- INTERLOCK FOOTGUN (review finding 2): if any of the canonical `DOCUMENT_COLS`/`AUTOMATION_COLS`/`NEW_CHAT_THREAD_COLS`/`PODCAST_COLS` are NOT updated to `workspace_id` before `apply_publication` runs, `_format_table_entry` ([zero_publication.py](surfsense_backend/app/zero_publication.py) lines 148-149) silently DROPS the mismatched table(s) from the publication (no error). Gate with `--verify` (see Verification).
5. No DDL needed for the `ck_connections_scope_owner` CHECK: the physical expression auto-follows the column rename (Postgres tracks the column by attnum). The model's `CheckConstraint(...)` text is updated separately as an ORM edit (see ORM edits) so create_all/autogenerate stay consistent.
### downgrade()
Provide a full reverse (rename `workspaces` -> `searchspaces`, columns back, constraints/indexes back, then restore the OLD publication shape). Note the repo sometimes uses no-op downgrades for publication-only migrations ([155](surfsense_backend/alembic/versions/155_reconcile_zero_publication.py) lines 22-23), but a structural rename should be reversible.
CRITICAL (review finding 7 — do NOT call `apply_publication()` in downgrade): `apply_publication()` reads the LIVE canonical in `zero_publication.py`, which after this change is `workspace_id`. In a downgrade the column is back to `search_space_id`, so `apply_publication` would find `workspace_id` missing and silently DROP the four column-list tables (`documents`/`automations`/`new_chat_threads`/`podcasts`) from the publication — the finding-2 footgun, self-inflicted. Instead, the downgrade must restore the old shape with HARDCODED `search_space_id` column lists (for all four tables) via a plain `ALTER PUBLICATION ... SET TABLE`, mirroring the constants-style approach in [143_force_zero_publication_resync.py](surfsense_backend/alembic/versions/143_force_zero_publication_resync.py) (which embeds literal `DOCUMENT_COLS` rather than importing the live module). Use the same DROP-then-SET neutralize sequence in reverse.
COHERENCE (review finding 6): even with a correct downgrade SQL, `downgrade()` only produces a consistent RUNNING system if the PRIOR code revision is redeployed alongside it — under the shim the model maps the attribute to physical `workspace_id`, which no longer exists after a pure DB downgrade. Document rollback = revert code + schema together as one operation.
## ORM / source edits in this phase (mapping only)
These are required so the ORM matches the renamed physical schema; they are NOT the broad symbolic rename. Explicitly OUT of scope for Phase 1: relationship attribute names (`User.search_spaces`, `back_populates="search_space"`, etc.) and class names (`SearchSpace`, `SearchSpaceRole`, ...). They stay as-is here and are renamed in Phase 2 — leaving them untouched is what keeps the ~250 callers green.
1. [surfsense_backend/app/db.py](surfsense_backend/app/db.py):
- `__tablename__` for `SearchSpace` (1688), `SearchSpaceRole` (2027), `SearchSpaceMembership` (2064), `SearchSpaceInvite` (2113) -> new table names.
- Every `ForeignKey("searchspaces.id", ...)` -> `ForeignKey("workspaces.id", ...)`; `ForeignKey("search_space_roles.id"...)` -> `workspace_roles.id`; `ForeignKey("search_space_invites.id"...)` -> `workspace_invites.id`.
- Apply the explicit-name shim on each `search_space_id` column: `Column("workspace_id", ...)`; and `owner_search_space_id = Column("owner_workspace_id", ...)`.
- Update ONLY the `name=` kwarg of constraints/indexes (1831, 2032, 2069, and the Index at 978-979). CRITICAL (review finding 1): do NOT change the inner column-reference strings (`"search_space_id"` at 1827, 2030, 2068, 2674, 979) — under the shim the column's `.key` is still `search_space_id`, and declarative `__table_args__` resolves these strings against `Table.c` by `.key`. Changing them to `"workspace_id"` raises a config-time error and the app won't boot.
- Update the runtime raw index DDL strings + names in `_INDEX_DEFINITIONS` (~2821-2831; +1 line vs the pre-merge cite after `main`'s `RefreshToken` edit — locate by grep): both the index name and the physical column (`search_space_id` -> `workspace_id`). These must match the migration's renamed indexes and ship in the same release; `setup_indexes()` ([db.py](surfsense_backend/app/db.py) ~line 2856) re-creates them under the new name on next boot when `DB_BOOTSTRAP_ON_STARTUP` is true.
- Update `ck_connections_scope_owner` text (1579-1582) `search_space_id` -> `workspace_id`.
- Update `ix_external_chat_bindings_search_space_state` name + the surrounding Index (978-979).
2. Module models — `__tablename__` unaffected, but apply the column shim + FK target string:
- [file_storage/persistence/models.py](surfsense_backend/app/file_storage/persistence/models.py) (32), [automations/persistence/models/automation.py](surfsense_backend/app/automations/persistence/models/automation.py) (27), [notifications/persistence/models.py](surfsense_backend/app/notifications/persistence/models.py) (50, plus the index at 36-41), [podcasts/persistence/models.py](surfsense_backend/app/podcasts/persistence/models.py) (71).
3. [surfsense_backend/app/zero_publication.py](surfsense_backend/app/zero_publication.py): `DOCUMENT_COLS` (31), `AUTOMATION_COLS` (57), `NEW_CHAT_THREAD_COLS` (62), and `PODCAST_COLS` (76) `search_space_id` -> `workspace_id`. (The `_expected_columns` special-casing on line 118 keys off table names `{"documents","user","podcasts"}`, not the column, so no change there — and `automations`/`new_chat_threads` aren't in that `_0_version` allowlist anyway.)
4. Runtime raw-SQL audit (defensive): grep `surfsense_backend/app` for any `text(...)` / hardcoded `"searchspaces"` or `search_space_id` strings that execute at runtime (not Python attribute access) and fix them here, since the shim only covers ORM attribute access. Expectation is few; the ~250-file footprint is overwhelmingly attribute access handled by the shim.
## Confirmed decisions
- Strategy: SHIM approach (physical DB rename now; Python attribute kept via `Column("workspace_id", ...)`; symbolic attribute/class/API rename deferred to Phase 2).
- Scope literal `'SEARCH_SPACE'` (the `ConnectionScope` value in `ck_connections_scope_owner`): KEEP the string value as-is; only fix the `search_space_id` column reference in the CHECK. No enum change, no data UPDATE.
- Cosmetic names: RENAME auto-named sequences/PK indexes (`searchspaces_id_seq` -> `workspaces_id_seq`, `searchspaces_pkey` -> `workspaces_pkey`, and the RBAC tables' `*_id_seq`/`*_pkey`) for a clean final schema.
- Transition: HARD CUTOVER at the DB layer (no backward-compat old column/view). Client apps hit the API (Phase 2), not the DB, so the DB rename does not affect them. (Phase 2 RESOLVED: hard cutover, no API aliases — see [02-rename-backend.md](02-rename-backend.md).)
- Zero: RECREATE THE REPLICATION (reset the zero-cache replica / `ZERO_AUTO_RESET`) on deploy for consumer recovery. Publication mutated only via the blessed `apply_publication` / `ALTER ... SET TABLE` path (never raw DROP/CREATE PUBLICATION, per migration 116). See migration step 4.
## Release coupling (review finding 5)
The Alembic migration, the `db.py` edits (shim + constraint `name=` + `_INDEX_DEFINITIONS` + `ck_connections_scope_owner` text), the 4 module-model edits, and `zero_publication.py` MUST ship as ONE atomic release. There is no safe intermediate state where some are updated and others are not.
## Verification & rollout
- HARD GATE: `python -m app.zero_publication --verify` (CLI at [zero_publication.py](surfsense_backend/app/zero_publication.py) ~lines 266-269; `verify_publication` defined at ~203) must report verified, in CI and immediately post-migrate. This is the guard against the interlock footgun (finding 2).
- Boot the API + a Celery worker; run a smoke chat + a document upload to exercise `documents.workspace_id` and the publication; confirm Zero clients still sync `documents`/`podcasts`.
- AUTOGENERATE DRIFT CHECK (strong, cheap): after the change, `alembic revision --autogenerate` must produce an EMPTY diff. A non-empty diff means the ORM and migrated DB disagree — i.e. a missed `ForeignKey("searchspaces.id")` string, a constraint `name=` mismatch, or a forgotten `Column("workspace_id", ...)` shim. This single check catches most subtle misses.
- Confirm `alembic upgrade head` then `alembic downgrade -1` then `upgrade head` round-trips on a staging copy. NOTE: this validates SQL reversibility only — it is NOT an app-health check, because new code cannot run against the downgraded schema (finding 6). The downgraded intermediate state must show `documents`/`podcasts` still present in the publication with `search_space_id` lists (proves the hardcoded downgrade publication shape from finding 7 works).
- Run the full backend test suite; the shim keeps attribute-based tests green — watch specifically for fixtures/raw SQL that hardcode the table name `searchspaces`.
- Pre-flight check: confirm no DB views/materialized views depend on the renamed objects (they auto-follow a rename, but verify so a later `pg_dump` diff holds no surprises).
- Raw-SQL audit done-check: grep `surfsense_backend/app` for `text(...)` or string literals containing `searchspaces` / `search_space_id` that execute at runtime; current audit found none beyond ORM attribute/dict-key usage (which the shim covers).
## Risks (with review findings)
- RENAME COLUMN under an active Zero publication column list (finding 3) — de-risked: step 4 neutralizes the column-list dependency (publish documents/podcasts as ALL columns) before the rename, and the zero-cache replica is recreated on deploy, so consumers re-sync cleanly.
- Silent publication drop if canonical col lists aren't updated (finding 2) — mitigated by the `--verify` hard gate.
- Zero consumers recover via the planned replica recreate (reset `surfsense-zero-cache` / `ZERO_AUTO_RESET=true`), giving a fresh initial sync from the corrected publication; no raw DROP/CREATE PUBLICATION (bug #1355).
- Constraint/index column-ref strings accidentally changed under the shim (finding 1) — config-time boot failure; mitigated by the explicit "only change name=" instruction.
- Runtime-created indexes may not exist at migrate time (finding 4) — mitigated by `IF EXISTS` renames + guarded `DO` blocks for constraints.
- Missing a `ForeignKey("searchspaces.id")` string leaves SQLAlchemy metadata pointing at a non-existent table — fails fast at mapper config (`NoReferencedTableError`); the inventory list above is the checklist.
- Partial/non-atomic release (finding 5) or code-less downgrade (finding 6) — mitigated by shipping all edits in one release and treating rollback as code+schema together.
- `downgrade()` calling `apply_publication()` against the live (new) canonical (finding 7) — would silently re-drop `documents`/`podcasts`; mitigated by hardcoding the old `search_space_id` publication shape in `downgrade()` (143-style), never importing the live module.
## Out of scope (later phases)
- Phase 2: rename Python attribute `search_space_id` -> `workspace_id`, class `SearchSpace` -> `Workspace`, Pydantic schemas, API routes/paths (`/searchspaces` + `/search-spaces` -> `/workspaces`), Redis keys, storage path segments; drop the explicit `Column("workspace_id", ...)` shim.
- Frontend Zero schema, route segment, i18n — deferred frontend umbrella.
---
## Implementation record (as-built)
### Deviations from the plan
- **Migration 168 idempotency fixed up-front**: the plan flagged from-scratch alembic as pre-existing-broken; we made 168 idempotent so the rename starts from a clean head, but did **not** take on the full baseline-squash. From-scratch `alembic upgrade head` therefore stays pre-existing-broken (rev 23 conflict); only the existing-DB `169 -> 170` path is in scope/verified.
- **Cosmetic sequence/PK renames applied** (the plan's confirmed decision): `searchspaces_id_seq -> workspaces_id_seq`, `searchspaces_pkey -> workspaces_pkey`, plus the RBAC tables' `*_id_seq` / `*_pkey`, for a clean final schema.
- Everything else implemented exactly as designed: shim via `Column("workspace_id", ...)`; publication mutated **only** through the blessed `apply_publication` path (no raw DROP/CREATE PUBLICATION, per migration 116); `__table_args__` inner column-reference strings deliberately **left** as `search_space_id` (flipped in Phase 2).
### Carve-outs as-shipped (Phase 1)
- `'SEARCH_SPACE'` CHECK literal in `ck_connections_scope_owner` kept; only the `search_space_id` column reference flipped to `workspace_id`. No enum/data migration.
### Verify current state (re-runnable)
Each line is a command + the last captured result. Re-run to confirm the *current* truth instead of trusting the snapshot. Schema gates assume a DB at rev `170` (locally `surfsense_oldshape`) with `AUTH_TYPE=LOCAL` (so the env-gated `oauth_account` table isn't in the ORM metadata).
- **Schema drift (ORM ↔ physical @170)** —
`AUTH_TYPE=LOCAL DATABASE_URL=postgresql+asyncpg://…@localhost:5432/surfsense_oldshape uv run alembic check`
→ last (2026-06-27): `No new upgrade operations detected.` (With `AUTH_TYPE=GOOGLE` the only delta is the env-gated `oauth_account` table — unrelated to the rename.)
- **Publication columns** —
`rg -n 'workspace_id|search_space_id' app/zero_publication.py`
→ last (2026-06-27): all four lists = `workspace_id`, zero `search_space_id`.
Validated during the build run (2026-06-26), **not** reproducible on a `create_all`-built snapshot (it has no `zero_publication` object): `python -m app.zero_publication --verify` → verified (interlock-footgun guard, finding 2); `alembic upgrade 170 → downgrade -1 → upgrade 170` round-trip (SQL reversibility; downgrade restores the `search_space_id` publication shape via hardcoded lists, per finding 7).

188
plans/backend/02-rename-backend.md
View file

@ -1,188 +0,0 @@
# Subplan 02 — Rename backend (code + API)
Part of [00-umbrella-plan.md](00-umbrella-plan.md), Phase 2. Backend only (`surfsense_backend`).
> **Status: SHIPPED** · as of 2026-06-27 · branch `feat/rename-searchspace-to-workspace` · PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546)
> Last commit of this phase: `902b3374e` (backend code/API rename, Waves A–F + carve-outs). Phase 1 + Phase 2 shipped as one atomic PR.
> The sections below are the **original design/rationale**; the as-built state + how to re-verify live in the [Implementation record](#implementation-record-as-built) at the bottom. Ground truth for files/commits is the PR, not this doc.
## Goal
Remove the Phase 1 shim and complete the SYMBOLIC rename `SearchSpace -> Workspace` / `search_space_id -> workspace_id` across the ~150 backend files, then consolidate the three live URL spellings (`/searchspaces`, `/search-spaces`, `/search-space`) onto a single canonical `/workspaces`. After this phase the physical DB (Phase 1) and the Python/API surface speak the same name.
Precondition: [01-rename-db.md](01-rename-db.md) is merged and live. Phase 1 left the Python attribute as `search_space_id` mapped to the physical column via `Column("workspace_id", ...)`; Phase 2 flips the attribute itself and drops that explicit-name shim.
## Surface area (cited; counts are current-tree, will shift after Phase 1)
Measured over `surfsense_backend/app` (Grep, case-insensitive `search.?space`):
- `search_space` — ~150 files, ~2,800 line matches (dominated by ORM attribute access).
- `SearchSpace` (class/relationship target) — ~38 files, ~450 matches.
- Heaviest files: [services/connector_service.py](surfsense_backend/app/services/connector_service.py) (158), [routes/search_source_connectors_routes.py](surfsense_backend/app/routes/search_source_connectors_routes.py) (138), [routes/rbac_routes.py](surfsense_backend/app/routes/rbac_routes.py) (142), [routes/model_connections_routes.py](surfsense_backend/app/routes/model_connections_routes.py) (132), [routes/new_chat_routes.py](surfsense_backend/app/routes/new_chat_routes.py) (133), [routes/documents_routes.py](surfsense_backend/app/routes/documents_routes.py) (121), [db.py](surfsense_backend/app/db.py) (119).
- Plus ~120 test files under `surfsense_backend/tests` mirror these patterns (rename in the same release).
This is mostly mechanical (an automated symbol rename handles the bulk), but a fixed set of STRING-LITERAL contracts and EXTERNAL contracts are NOT safe to blind-replace — they get explicit decisions in this plan (see "String-literal & external contracts").
## Transition policy: HARD CUTOVER (confirmed)
The umbrella defers all frontend/client work, and the frontend will be (re)built against the corrected backend in its own umbrella rather than kept alive in lockstep. So this phase HARD-CUTS the external API — no backward-compat aliases.
- URL paths: rename `/api/v1/searchspaces...`, `/api/v1/search-spaces/...`, `/api/v1/...search-space...` outright to the canonical `/api/v1/workspaces...` ([routes/search_spaces_routes.py](surfsense_backend/app/routes/search_spaces_routes.py) 73-373; mount at [app.py](surfsense_backend/app/app.py) 993, `crud_router` under `/api/v1`).
- JSON field names: rename `search_space_id` -> `workspace_id` in request/response bodies outright (e.g. [schemas/new_chat.py](surfsense_backend/app/schemas/new_chat.py), [schemas/rbac_schemas.py](surfsense_backend/app/schemas/rbac_schemas.py), [schemas/documents.py](surfsense_backend/app/schemas/documents.py)). No `populate_by_name` alias plumbing.
CONSEQUENCE (accepted): the existing deployed frontend breaks against the renamed API until its umbrella lands. Backend correctness is therefore verified INDEPENDENTLY of the old UI — via the test suite, OpenAPI/`/docs`, and direct API calls — not by clicking through the current frontend. (Alias/dual-serve was considered and rejected: it adds plumbing to keep a frontend alive that is being redesigned anyway.)
## Rename waves (order matters)
### Wave A — ORM core in [db.py](surfsense_backend/app/db.py)
1. Classes: `SearchSpace` (1688), `SearchSpaceRole` (2021/2027), `SearchSpaceMembership` (2058/2064), `SearchSpaceInvite` (2107/2113) -> `Workspace`, `WorkspaceRole`, `WorkspaceMembership`, `WorkspaceInvite`.
2. Drop the Phase 1 shim on every column: `search_space_id = Column("workspace_id", Integer, ForeignKey("workspaces.id", ...))` -> `workspace_id = Column(Integer, ForeignKey("workspaces.id", ...))` (the explicit `"workspace_id"` first arg is now redundant because attribute name == column name). Same for `owner_search_space_id` -> `owner_workspace_id` (785-787).
3. INVERSE OF PHASE-1 FINDING 1: now that the attribute `.key` becomes `workspace_id`, the `__table_args__` inner column-reference strings MUST flip to `"workspace_id"` to match — `UniqueConstraint("search_space_id", ...)` -> `UniqueConstraint("workspace_id", ...)` at 1826-1832 (`uq_workspace_user_connector_type_name`), 2029-2033 (`uq_workspace_role_name`), 2066-2070 (`uq_user_workspace_membership`), and the `Index(... "search_space_id" ...)` at 978-979. (Phase 1 deliberately left these as `search_space_id`; Phase 2 completes them. Getting this wrong = config-time boot failure, same failure mode as Phase 1 finding 1, just mirrored.)
4. Relationships (~35 pairs): rename the attribute names and both ends of `back_populates`, and the target-class strings:
- Child side: `search_space = relationship("SearchSpace", back_populates="...")` -> `workspace = relationship("Workspace", back_populates="...")` (e.g. Folder 1340, Document 1421, Connection 948).
- Hub side on `Workspace`: `back_populates="search_space"` -> `"workspace"` for folders (1726-1728), documents, threads, podcasts, video_presentations, reports, image_generations, logs, notifications, connectors, connections, automations, roles, memberships, invites.
- User side (both auth branches ~2205 and ~2337): `search_spaces = relationship("SearchSpace", ...)` -> `workspaces = relationship("Workspace", ...)`; `search_space_memberships` -> `workspace_memberships`; invites likewise.
- External chat: `owner_search_space = relationship("SearchSpace", foreign_keys=[owner_search_space_id])` (819-820) -> `owner_workspace = relationship("Workspace", foreign_keys=[owner_workspace_id])`.
5. CHECK text on `connections`: `ck_connections_scope_owner` (1578-1581) `search_space_id` -> `workspace_id` in the SQL string. (Phase 1 already did this if shipped; re-confirm. The scope literal `'SEARCH_SPACE'` stays — see enum carve-out.)
6. Runtime index DDL in `_INDEX_DEFINITIONS` (2820-2830): already renamed to `workspace_id`/`idx_documents_workspace_id` in Phase 1 — confirm no `search_space` remains.
### Wave B — satellite ORM modules (mirror Wave A; drop shim, flip strings)
- [automations/persistence/models/automation.py](surfsense_backend/app/automations/persistence/models/automation.py) (27-29 column, 68 relationship).
- [file_storage/persistence/models.py](surfsense_backend/app/file_storage/persistence/models.py) (32-34).
- [notifications/persistence/models.py](surfsense_backend/app/notifications/persistence/models.py) (50-52 column, 72 relationship, plus the index `ix_notifications_user_space_created` 36-41 — definition auto-follows; rename only if you want the name clean).
- [podcasts/persistence/models.py](surfsense_backend/app/podcasts/persistence/models.py) (71-72).
### Wave C — Pydantic schemas
- Dedicated module [schemas/search_space.py](surfsense_backend/app/schemas/search_space.py): classes `SearchSpaceBase/Create/Update/Read/WithStats` -> `Workspace*`. Rename file to `schemas/workspace.py`. Update re-exports in [schemas/__init__.py](surfsense_backend/app/schemas/__init__.py) (106-111, and `UserSearchSpaceAccess` ~87).
- Field `search_space_id` -> `workspace_id` across: new_chat, rbac_schemas (+ `search_space_name`, `UserSearchSpaceAccess`), documents, folders, image_generation, model_connections, search_source_connector, logs, stripe, prompts, chat_comments, video_presentations, reports, obsidian_plugin, podcasts/api/schemas, automations schemas, notifications/api/schemas.
- Under hard cutover the serialized JSON key changes outright to `workspace_id` (no alias). Update fixtures/contract tests accordingly.
### Wave D — services / utils / tasks / agents / gateway / event_bus (the mechanical bulk)
Pure attribute/param/function symbol rename `search_space_id -> workspace_id`, `search_space -> workspace`, plus function renames:
- RBAC helpers [utils/rbac.py](surfsense_backend/app/utils/rbac.py): `check_search_space_access` (129), `is_search_space_owner` (160), `get_search_space_with_access_check` (180) -> `*_workspace_*`.
- Validator [utils/validators.py](surfsense_backend/app/utils/validators.py): `validate_search_space_id` (16).
- Retrievers [retriever/documents_hybrid_search.py](surfsense_backend/app/retriever/documents_hybrid_search.py), [retriever/chunks_hybrid_search.py](surfsense_backend/app/retriever/chunks_hybrid_search.py) (~26 each, filter clauses).
- Connector service [services/connector_service.py](surfsense_backend/app/services/connector_service.py) (ctor param + `self.search_space_id`, + the in-process caches keyed by id — see literals).
- Indexers under `app/tasks/connector_indexers/*`, `indexing_pipeline/*`, `services/*/kb_sync_service.py`.
- Agents: dep-dict KEYS like `"search_space_id": d["search_space_id"]` (e.g. [agents/.../subagents/connectors/google_drive/tools/index.py](surfsense_backend/app/agents/chat/multi_agent_chat/subagents/connectors/google_drive/tools/index.py) 28; `main_agent/runtime/factory.py` 137); class `SearchSpaceSkillsBackend` (`.../main_agent/skills/backends.py` 184).
- Gateway: [gateway/inbox_processor.py](surfsense_backend/app/gateway/inbox_processor.py), [gateway/agent_invoke.py](surfsense_backend/app/gateway/agent_invoke.py), [gateway/auth_invariant.py](surfsense_backend/app/gateway/auth_invariant.py).
- Event bus: `Event.search_space_id` field ([event_bus/event.py](surfsense_backend/app/event_bus/event.py) 36), `publish(... search_space_id=...)` ([event_bus/bus.py](surfsense_backend/app/event_bus/bus.py) 43-49), payload-dict key in [event_bus/events/document_entered_folder.py](surfsense_backend/app/event_bus/events/document_entered_folder.py) 74-76 — see literals decision.
### Wave E — routes: rename handlers + consolidate the three URL spellings
Canonical: `/workspaces` (umbrella decision). Current spellings to retire:
- `/searchspaces` (no hyphen): [search_spaces_routes.py](surfsense_backend/app/routes/search_spaces_routes.py) (73-373), [rbac_routes.py](surfsense_backend/app/routes/rbac_routes.py), [agent_permissions_route.py](surfsense_backend/app/routes/agent_permissions_route.py), [team_memory_routes.py](surfsense_backend/app/routes/team_memory_routes.py).
- `/search-spaces` (hyphen, plural): [editor_routes.py](surfsense_backend/app/routes/editor_routes.py), [notes_routes.py](surfsense_backend/app/routes/notes_routes.py), [export_routes.py](surfsense_backend/app/routes/export_routes.py), [model_connections_routes.py](surfsense_backend/app/routes/model_connections_routes.py) (`/model-roles`).
- `/search-space` (hyphen, singular): [logs_routes.py](surfsense_backend/app/routes/logs_routes.py) 252, [gateway_webhook_routes.py](surfsense_backend/app/routes/gateway_webhook_routes.py) 992/1022, webhook path `{search_space_id}` in [circleback_webhook_route.py](surfsense_backend/app/routes/circleback_webhook_route.py) 215/315.
- Rename the route files (`search_spaces_routes.py` -> `workspaces_routes.py`) and handler fns (`create_search_space` etc.) and the import/include in [routes/__init__.py](surfsense_backend/app/routes/__init__.py) (63, 73).
Mechanism: routers use `APIRouter()` with the path fully spelled in each decorator (no per-router prefix), mounted under `/api/v1` at [app.py](surfsense_backend/app/app.py) 993. So consolidation = edit each decorator string to `/workspaces/{workspace_id}/...` outright (hard cutover; no alias routers).
### Wave F — tests + fixtures
Rename `surfsense_backend/tests/**`; watch for fixtures that hardcode the table name `searchspaces`/JSON key `search_space_id` (those depend on the transition policy for API tests).
## String-literal & external contracts (explicit decisions — NOT blind-replace)
These do not move with a symbol rename; each is decided here.
1. Enum VALUES `'SEARCH_SPACE'` — `ConnectionScope.SEARCH_SPACE` ([db.py](surfsense_backend/app/db.py) 204-207, stored in `connections.scope` via `SQLAlchemyEnum`, line 1558) and `ChatVisibility.SEARCH_SPACE` (510-520, stored in `new_chat_threads.visibility`, 596-597). DECISION: KEEP the enum value strings as-is. They are persisted in Postgres and exposed in JSON; renaming the value needs a data migration + PG enum-type alter for zero benefit. This matches Phase 1's decision to keep the `'SEARCH_SPACE'` CHECK literal. (Optionally rename only the Python member to `WORKSPACE` while keeping `= "SEARCH_SPACE"` — deferred; not worth the churn for MVP.)
2. Celery task `name=` strings — `"delete_search_space_background"` ([tasks/celery_tasks/document_tasks.py](surfsense_backend/app/tasks/celery_tasks/document_tasks.py) 206), `"ai_sort_search_space"` (1546). These are the WIRE NAME between producer and worker; tasks dispatch via `.delay()`/`send_task` (e.g. event trigger `send_task(TASK_NAME, ...)` in [automations/triggers/builtin/event/source.py](surfsense_backend/app/automations/triggers/builtin/event/source.py) 19). DECISION: KEEP the `name=` strings unchanged; freely rename the Python function symbols (`ai_sort_search_space_task` etc.). A rolling deploy with renamed wire names would orphan in-flight messages. (If a cosmetic rename is wanted later, do it with a dual-register + queue-drain, out of scope here.)
3. Redis key literals — `surfsense:spawn_paused:{search_space_id}` (`tasks/.../spawn_paused.py`) and `ai_sort:search_space:{search_space_id}:lock` ([document_tasks.py](surfsense_backend/app/tasks/celery_tasks/document_tasks.py) 1542). DECISION: rename literals to `workspace` (these hold short-lived locks / an ops toggle). Accept that any in-flight lock / paused-flag resets at deploy (locks are seconds-long; paused-flag is an ops action). UPDATE the runbook in `.env.example` (the `redis-cli SET surfsense:spawn_paused:<id>` doc, ~503-504).
4. Event payload key `search_space_id` — `Event` is `model_dump`ed and sent to Celery for automation event triggers, and trigger filters read the key. DECISION: rename the field to `workspace_id` (internal, and triggers are re-evaluated continuously); accept transient loss of any event enqueued across the deploy boundary (fire-and-forget). Drain the event queue during the maintenance window to be safe.
5. OpenTelemetry attribute `search_space.id` + metric label ([observability/otel.py](surfsense_backend/app/observability/otel.py) 263-264/305-306, [observability/metrics.py](surfsense_backend/app/observability/metrics.py) 537-542). DECISION: KEEP the OTel/metric KEY `search_space.id` for now (dashboards/alerts depend on it); rename only the Python params. Schedule the observability-key rename as a separate, announced change. (Carve-out to avoid silently breaking alerting.)
6. Notification dedup/operation IDs embedding `{search_space_id}` (e.g. `doc_..._{search_space_id}_...`, `insufficient_credits_{search_space_id}_...`) and frontend deep-link strings like `/dashboard/{search_space_id}/buy-more`. DECISION: the ID is a numeric value, not the literal word — leave format strings as-is functionally; the embedded VALUE is unchanged. The `/dashboard/{id}/...` deep link points at a FRONTEND route still named `[search_space_id]` (deferred umbrella) — KEEP it until the frontend segment renames, else links 404.
7. Storage path builders — `documents/{search_space_id}/...` ([file_storage/keys.py](surfsense_backend/app/file_storage/keys.py) 20-26) and `podcasts/{search_space_id}/...` ([podcasts/storage.py](surfsense_backend/app/podcasts/storage.py) 22-25). The path segment is the numeric ID; the literal word `search_space` is NOT in stored object keys. DECISION: rename the param only; NO blob migration needed; existing objects keep resolving.
8. `SearchSourceConnector` — contains "search" but is the connectors table, a different concept (the word "search" here is unrelated to `SearchSpace`). OUT OF SCOPE: this rename does NOT touch it, and Phase 4 does not rename the class either — Phase 4 adds a Type-1/Type-2 taxonomy via a static `connector_type`→(category, availability) registry (no new column) and KEEPS `is_indexable` (`db.py:1868`). Leave `SearchSourceConnector` as-is.
9. Historical Alembic migrations (`surfsense_backend/alembic/versions/*`) — ~20 files embed `searchspaces` / `search_space_id` as raw-SQL string literals (e.g. `23_associate_connectors_with_search_spaces.py`, `41_backfill_rbac_for_existing_searchspaces.py`, `40_move_llm_preferences_to_searchspace.py`). DECISION: NEVER rewrite these. They are an immutable replay log that intentionally references the schema as it existed at that revision; rewriting them corrupts history and breaks a clean `alembic upgrade` from zero. Verified safe: no migration imports the ORM classes (only `from app.db import Base` in `env.py` / `0_initial_schema.py`), so the class rename does not touch them. Phase 1's rename migration (the next integer after the live head — `170` at time of writing, since `main` advanced the head to `169`, but chain by actual `alembic heads`) is the single transition point; migrations after it use the new names. The scripted rename scope is `app/` + `tests/` ONLY.
10. LangGraph persisted state channel key `search_space_id` — `input_state = {..., "search_space_id": search_space_id, ...}` ([tasks/chat/streaming/flows/new_chat/input_state.py](surfsense_backend/app/tasks/chat/streaming/flows/new_chat/input_state.py) 131) is a CHECKPOINTED state channel. The resume_chat flow reads persisted state (`agent.aget_state({"configurable": {"thread_id": ...}})`, [flows/resume_chat/resume_routing.py](surfsense_backend/app/tasks/chat/streaming/flows/resume_chat/resume_routing.py) 50) and HITL interrupts (`surfsense_resume_value`, `HumanReview`) can sit pending across a deploy. DECISION: rename the channel key to `workspace_id` for consistency, and ACCEPT that any chat thread paused at a HITL interrupt BEFORE the cutover must be restarted after (the old checkpoint carries `search_space_id`, the new graph reads `workspace_id`). Operationally: drain/resolve pending interrupts before deploy if practical. (Alternative — keep the channel key as a carve-out — rejected: leaves a lone `search_space_id` in otherwise-renamed agent state for a transient, conversation-scoped value. The `configurable` keys are `thread_id` / `surfsense_resume_value`, not `search_space_id`, so only this state channel is affected.)
11. User-facing default literal — `users.py` seeds the default workspace `name="My Search Space"` (line 158, persisted + shown to users) and logs "Created default search space" (207). DECISION: rename the default name to "My Workspace" (backend-created seed value, not caught by a symbol rename); log strings are cosmetic. Also update the Redis-key assertion in `tests/unit/services/test_ai_sort_task_dedupe.py:14` when literal 3 is renamed.
## Execution approach
- Automate the safe bulk: a scripted symbol rename (IDE rename / `ast-grep`-style) for `search_space_id -> workspace_id`, `search_space -> workspace`, `SearchSpace -> Workspace`. SCOPE STRICTLY to `app/` + `tests/` — NEVER `alembic/versions/` (carve-out 9: immutable replay log). EXCLUDE the other carve-outs above (enum values, Celery `name=`, OTel keys, frontend deep-links, default-name literal). Run per-wave, not all at once, so review is tractable.
- Then hand-apply the carve-outs and the routing consolidation.
- One atomic release (same coupling rule as Phase 1): ORM + schemas + routes + literals ship together; there is no half-renamed steady state.
## Verification & rollout
- `alembic revision --autogenerate` must produce an EMPTY diff: confirms the renamed ORM (classes/attrs/constraint inner-strings) still matches the Phase 1 physical schema. A diff means a missed attribute flip or a stale `__table_args__` string.
- Boot API + Celery worker; verify via OpenAPI/`/docs` + direct API calls (NOT the old frontend): create workspace, chat, document upload, an RBAC invite/membership op, an automation event trigger, and a podcast — exercising the renamed relationships (`workspace`, `workspaces`), tasks, and event payloads.
- Hard-cutover contract check: confirm `/workspaces...` resolves and the three legacy spellings now 404; confirm bodies use `workspace_id` only.
- Full backend test suite green; grep `surfsense_backend/app` for residual `search_space`/`SearchSpace` and confirm every remaining hit is an intentional carve-out (enum value, Celery `name=`, OTel key, frontend deep-link).
- Confirm `alembic/versions/*` was NOT modified by the rename (git diff shows zero changes under that path) — guard against carve-out 9.
- Resume check (carve-out 10): start a chat, trigger a HITL interrupt, deploy/rename, then verify NEW interrupts resume on `workspace_id`; accept that pre-cutover pending interrupts are restarted. Ideally drain pending interrupts before deploy.
- Observability check: `search_space.id` OTel attribute still emitted (carve-out 5) so dashboards keep working.
## Risks
- Stale `__table_args__` inner string after attribute flip (Wave A.3) — config-time boot failure; mirror of Phase 1 finding 1. Mitigated by the autogenerate empty-diff gate + boot smoke test.
- Relationship rename misses one `back_populates` end — mapper-config error at startup; the pair list in Wave A.4 is the checklist.
- Hard cutover breaks the existing frontend in the interim (accepted) — mitigated by verifying the backend via tests/OpenAPI/API calls, and by (re)building the frontend against the new contract in its umbrella. Do not rely on the old UI to smoke-test during this window.
- Renaming a Celery `name=` or the event payload key without a drain — orphaned/dropped in-flight messages; mitigated by the carve-out (keep task names) + queue-drain for events.
- Enum value migration scope creep — explicitly OUT (carve-out 1); keep `'SEARCH_SPACE'` values.
- Scripted rename corrupting historical Alembic migrations (carve-out 9) — would break `alembic upgrade` from zero / replay history; mitigated by scoping the rename to `app/`+`tests/` and the git-diff guard on `alembic/versions/`.
- Renaming the persisted LangGraph state channel key (carve-out 10) — breaks resume of threads paused at a HITL interrupt across the cutover; mitigated by draining interrupts pre-deploy and accepting restart of any stragglers (state is conversation-scoped/ephemeral).
- Sheer size (~150 app + ~120 test files) — mitigated by per-wave scripted rename + the empty-diff and residual-grep gates.
## Out of scope (later phases / umbrella)
- Frontend route segment `[search_space_id] -> [workspace_id]`, `search-space-settings/`, TS types, atoms, i18n, frontend Zero schema — deferred frontend umbrella.
- Satellite apps (desktop, Obsidian, browser extension, evals) + docs — deferred.
- Connector `category` discriminator and `SearchSourceConnector` handling — Phase 4 ([04-capabilities.md](04-capabilities.md)).
- Enum VALUE migration and observability-key rename — deliberately deferred announced changes.
---
## Implementation record (as-built)
Whole-PR footprint (vs `ci_mvp`): **452 files, +5317/-4781** — 325 under `app/`, 124 under `tests/`, and **exactly 2** `alembic/versions/` files (`168` idempotency + `170` rename), confirming the immutable-replay-log carve-out (#9) held.
The plan's Waves A–F were executed in order, as described above; what follows records only what diverged from that design and the evidence that it landed cleanly.
### Deviations from the plan
- **Bulk rename executed via scoped Python codemods** (multiple passes), strictly limited to `app/` + `tests/` (never `alembic/versions/`), instead of IDE/`ast-grep`. Same scope rule and same carve-out exclusions as designed.
- **camelCase `searchSpaceId` -> `workspaceId`**: a camelCase payload key (in `app/agents/chat/multi_agent_chat/main_agent/middleware/kb_persistence/middleware.py`) was renamed too — decided internal/transient, so it follows the hard cutover. The plan enumerated snake/Pascal/Title/hyphen variants; this camelCase variant was caught in a follow-up codemod pass (residual `searchSpaceId` in `app/` now = 0).
- All other carve-outs honored exactly as decided (below).
### Carve-outs as-shipped (verified by residual grep)
KEPT (intentional):
- Enum VALUES `'SEARCH_SPACE'` (`ConnectionScope`, `ChatVisibility`) — value migration deferred (carve-out 1).
- Celery wire task names `"delete_search_space_background"`, `"ai_sort_search_space"` (carve-out 2). Python functions renamed (`ai_sort_workspace_task`, `delete_workspace_task`); producer and worker agree because dispatch is via `.delay()` on the task object (no hardcoded `send_task("ai_sort_search_space")`).
- OTel/metric key `search_space.id` (carve-out 5) — 3 sites (`observability/otel.py` x2, `observability/metrics.py`); dashboards/alerts depend on it.
- API error code `"SEARCH_SPACE_FORBIDDEN"` (external contract string).
- `SearchSourceConnector` class/table (carve-out 8, unrelated taxonomy).
- Historical `alembic/versions/*` (carve-out 9, immutable replay log).
RENAMED (accepted transient deploy cost):
- Redis keys `surfsense:spawn_paused:{workspace_id}`, `ai_sort:workspace:{workspace_id}:lock` (carve-out 3) + `.env.example` runbook.
- Event payload key (carve-out 4) and LangGraph state-channel key (carve-out 10) -> `workspace_id`; default seed name -> `"My Workspace"` (carve-out 11).
### Verify current state (re-runnable)
Each line is a command + the last captured result. Re-run to confirm the *current* truth instead of trusting the snapshot. Run from `surfsense_backend/`.
- **Full backend suite** — `uv run pytest tests/unit tests/integration`
→ last (2026-06-27): `3016 passed, 1 skipped` in ~101s. The skip is a Windows-only event-loop test (`tests/unit/tasks/test_celery_async_runner.py:302`).
- **Schema drift @170** — `AUTH_TYPE=LOCAL DATABASE_URL=…surfsense_oldshape uv run alembic check`
→ last (2026-06-27): `No new upgrade operations detected.`
- **Route hard-cutover** — load `app.app:app` and inspect `{r.path for r in app.routes}`
→ last (2026-06-27): legacy `/searchspaces` `/search-spaces` `/search-space` = 0 each; canonical `/workspaces` = 26 (of 278 routes).
- **Residual symbols** — `rg -ni 'search.?space' app`
→ last (2026-06-27): 52 hits, all carve-outs (enum `SEARCH_SPACE` values dominate; 3× OTel `search_space.id`; 2× Celery wire names; 1× `SEARCH_SPACE_FORBIDDEN`). No stray attribute/relationship/route residue.
- **`alembic/versions/` immutability** — `git diff --stat ci_mvp..HEAD -- surfsense_backend/alembic/versions`
→ last (2026-06-27): only `168` + `170` (guard against carve-out 9).
### Known caveats at deploy (team-acknowledged)
- **Zero-cache replica reset** required on deploy (`ZERO_AUTO_RESET`) — consequence of the schema rename.
- **From-scratch `alembic upgrade head` stays pre-existing-broken** (rev 23 conflict); the existing-DB `169 -> 170` path is verified. Separate baseline-squash task.
- **Planned-downtime deploy**: drain Celery/event queues and restart workers before cutover (renamed event payload / LangGraph state-channel / Redis keys). In-flight state across the boundary is discarded by design — acceptable given the downtime window.
- **Live Celery worker not exercised end-to-end** (the broker is mocked in tests). The task code paths are covered by unit/integration tests; a real worker + Redis smoke is the one verification not yet run.
- **Clients are intentionally broken** by the hard cutover until the frontend/satellite umbrellas land — backend correctness is verified via the suite + OpenAPI/route introspection + direct API calls, not the old UI.

145
plans/backend/03a-crawler-core.md
View file

@ -1,145 +0,0 @@
# Phase 3a — WebURL Crawler core (Scrapling-only) + success semantics
> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
> Sibling subplans: `03b-proxy-expansion.md`, `03c-crawl-billing.md`, `03e-stealth-hardening.md`, `03d-captcha-solving.md`, `03f-undetectability-testing.md` (manual scorecard).
> **Implementation note (applies to all Phase-3 plans).** Phases 1–2 are **SHIPPED** (2026-06-27) — `SearchSpace`→`Workspace` and `search_space_id`→`workspace_id` are renamed everywhere, so the **live code already says `workspace_*`**. Where citations below use the **old** `search_space_*`/`SearchSpace` names (written pre-rename), substitute the `workspace_*` equivalent and **grep the new name** (grepping the old name now returns nothing). Apply every edit by **symbol/grep** (e.g. `firecrawl_api_key`, `crawl_url`, `FIRECRAWL_API_KEY`), **not** by the absolute line numbers cited here — the rename (and `03a`'s own Firecrawl removal) shifted them.
> **Status: ✅ IMPLEMENTED (`ci_mvp` @ `5c36cd3`).** All 5 work items done: Firecrawl removed repo-wide, `crawl_url` is the 3-tier Scrapling ladder returning `CrawlOutcome` (`impersonate="chrome"` on the static tier + `solve_cloudflare=True` on the stealthy tier), `crawls_succeeded` added to the indexer (positional return unchanged — surfaced in task metadata), both `scrape_webpage` tools updated, unit tests added/passing. Crawler relocated to `app/proprietary/web_crawler/connector.py` under the non-Apache-2 boundary (see umbrella decisions log + `app/proprietary/README.md`).
## Objective
Make the Universal WebURL Crawler a **single-framework (Scrapling) component** with deterministic per-URL outcome semantics and a clean, explicit **"successful crawl" signal** that Phase 3c bills on.
Two hard requirements from the decisions log:
1. **Remove Firecrawl entirely.** No other scraping framework now or planned. Scrapling's `StealthyFetcher` can bypass Cloudflare Turnstile when invoked with `solve_cloudflare=True` (see tier design below); captcha-tools (`03d`) covers reCAPTCHA/hCaptcha, and `03e` adds the broader stealth-hardening that makes the crawler "undetectable" as far as free tooling reaches.
2. **One billable unit = one URL that yields usable extracted content**, regardless of how many internal fallback tiers ran. `03a` must expose that signal; `03c` meters it.
This subplan does NOT touch proxy rotation (→ `03b`), credit metering (→ `03c`), or captcha (→ `03d`).
## Current state (cited)
### The crawler
> **Post-implementation note (location moved).** 03a is implemented. The crawler now lives at `app/proprietary/web_crawler/connector.py` (public API re-exported from `app/proprietary/web_crawler/__init__.py`), under the non-Apache-2 license boundary `app/proprietary/` (see `app/proprietary/README.md` + `LICENSE`). Importers use `from app.proprietary.web_crawler import WebCrawlerConnector, CrawlOutcomeStatus`. The file:line references below describe the *original* `app/connectors/webcrawler_connector.py` surface that 03a refactored away (kept for historical context).
`app/connectors/webcrawler_connector.py` — `WebCrawlerConnector.crawl_url()` is a 4-tier fallback ladder:
1. **Firecrawl** (premium, if `firecrawl_api_key` set) — `_crawl_with_firecrawl()` (lines 89–100, 223–272), imports `from firecrawl import AsyncFirecrawlApp` (line 22).
2. Scrapling `AsyncFetcher` (static HTTP, curl_cffi) — `_crawl_with_async_fetcher()` (lines 274–310).
3. Scrapling `DynamicFetcher` (browser, run in a thread) — `_crawl_with_dynamic()` (lines 312–339).
4. Scrapling `StealthyFetcher` (patchright-Chromium anti-bot, run in a thread) — `_crawl_with_stealthy()` (lines 341–369).
> **Engine note (do not say "Camoufox").** As of the pinned `scrapling[fetchers]>=0.4.9` (`pyproject.toml:91`), `StealthyFetcher` is **"completely stealthy built on top of Chromium"** (`references/Scrapling/scrapling/fetchers/stealth_chrome.py:8`) driven by **patchright** (`references/Scrapling/scrapling/engines/_browsers/_stealth.py:8–9`), **not** Camoufox — Scrapling removed Camoufox (zero matches in the 0.4.9 tree; `uv.lock` carries `patchright`, no `camoufox`). The default Playwright `channel` is `"chromium"` (patchright), or `"chrome"` when `real_chrome=True` (`_browsers/_base.py:469`). Stealth = the compiled-in flag set `DEFAULT_ARGS + STEALTH_ARGS` (`engines/constants.py:24–99`, incl. `--disable-blink-features=AutomationControlled` `:94`) + a persistent context by default (`_stealth.py:90–93`) — this is **runtime/config-level** stealth, which sets the realistic ceiling (see `03e`).
>
> **Scrub the in-code "Camoufox" mentions too.** The connector still carries stale Camoufox wording in its own docstrings — the module header tier list (`webcrawler_connector.py:10–12`) and `_crawl_with_stealthy`'s docstring (`:343` "StealthyFetcher (Camoufox)"). Fix these in the 03a refactor so the code matches reality (patchright-Chromium).
Extraction is Trafilatura HTML→markdown in `_build_result()` (lines 371–469). Every Scrapling tier passes `proxy=get_proxy_url()` (lines 287, 329, 359). `crawl_url()` returns a tuple `(result_dict | None, error | None)`; `result_dict` has `content` / `metadata` / `crawler_type`.
### The two call sites
- **Type-1 indexer (billable path):** `app/tasks/connector_indexers/webcrawler_indexer.py` reads `FIRECRAWL_API_KEY` from connector config (line 115), builds `WebCrawlerConnector(firecrawl_api_key=api_key)` (line 139), and calls `crawler.crawl_url(url)` per URL (line 297). It already tracks `documents_indexed` / `documents_updated` / `documents_skipped` / `documents_failed` / `duplicate_content_count` (lines 167–171), but **none of these is a clean "crawl succeeded" count** — duplicates and unchanged docs are bucketed as skipped even though a successful fetch happened.
- **Chat scrape tool (ad-hoc):** `main_agent/tools/scrape_webpage.py` (line 232) and `subagents/builtins/research/tools/scrape_webpage.py` (line 226) build `WebCrawlerConnector(firecrawl_api_key=...)` and call `crawl_url(url, formats=["markdown"])`. The `formats` arg is Firecrawl-only (declared at `crawl_url` signature line 59; docstring line 72: "only for Firecrawl").
### Firecrawl threading (the cross-cutting surface)
Firecrawl's API key is plumbed end-to-end and must be removed everywhere:
| Layer | File:line | What to remove |
|-------|-----------|----------------|
| Crawler | `app/connectors/webcrawler_connector.py` | `firecrawl` import (22), `firecrawl_api_key`/`use_firecrawl` ctor (36–46), `set_api_key()` (48–56), tier-1 block (89–100), `_crawl_with_firecrawl()` (223–272), `formats` param + Firecrawl mentions in docstrings |
| Dependency | `pyproject.toml:45` | `"firecrawl-py>=4.9.0"` (+ regenerate `uv.lock`). Scrapling stays (`pyproject.toml:91` → `"scrapling[fetchers]>=0.4.9"`) |
| Connector config | `app/utils/validators.py` | `FIRECRAWL_API_KEY` from `WEBCRAWLER_CONNECTOR.optional`/`validators` (573–580); delete `validate_firecrawl_api_key_format()` (472–478) |
| Indexer | `app/tasks/connector_indexers/webcrawler_indexer.py` | `api_key = connector.config.get("FIRECRAWL_API_KEY")` (115), `use_firecrawl` log field (135), pass-through to ctor (139) |
| Chat setup | `app/tasks/chat/streaming/flows/shared/pre_stream_setup.py` | `setup_connector_and_firecrawl()` returns `firecrawl_api_key` (1–30) — collapse to connector-service-only |
| Chat orchestrators | `new_chat/orchestrator.py` (86, 378, 412, 665), `resume_chat/orchestrator.py` (65, 317, 347, 483) | `firecrawl_api_key` threading |
| Automations | `automations/actions/builtin/agent_task/dependencies.py` (19, 34, 85, 103), `.../invoke.py` (174) | `firecrawl_api_key` dep field |
| Main agent | `main_agent/runtime/factory.py` (71, 142), `main_agent/tools/registry.py` (36) | `firecrawl_api_key` param/wiring |
| Scrape tools | `main_agent/tools/scrape_webpage.py` (170, 232), `research/tools/scrape_webpage.py` (164, 226), `research/tools/index.py` (28) | `firecrawl_api_key` factory arg + ctor arg |
| Tests | `tests/unit/automations/actions/builtin/agent_task/test_dependencies.py` (44, 81) | `firecrawl_api_key == "fc-key"` assertion + fake |
> **Not a global env var.** `FIRECRAWL_API_KEY` is **not** in `.env.example` and there is no `Config.FIRECRAWL_API_KEY`. It lives only inside each WebCrawler connector's `config` JSON — the validator reads it from that dict (`validators.py:474`). So removal is code-only; **no env/docs change**. Existing `WEBCRAWLER_CONNECTOR` rows may still carry a now-dead `FIRECRAWL_API_KEY` key in `config` — harmless (it's simply ignored), optionally scrubbed by a tiny data migration if we want clean rows.
>
> **Signature change, not just deletions.** Removing `firecrawl_api_key` mutates the `RuntimeDeps`-style dataclass (`agent_task/dependencies.py:34`) and the agent runtime factory/tool factories — a coordinated signature change across the chat + automations call graph, so land it atomically.
### Runtime/deps already in place
- `Dockerfile:112–115` runs `RUN scrapling install`; the `scrapling[fetchers]` extra pulls playwright/patchright. **No new install step needed** once Firecrawl is gone. *(Accuracy fix: the Dockerfile comment says "patchright Chromium + Camoufox", but `scrapling install` in 0.4.9 only fetches Chromium — `references/Scrapling/scrapling/cli.py:122,131` runs `playwright install chromium` + `install-deps chromium`, no Camoufox. Drop the stale "+ Camoufox" wording from the comment when touching this file. `03e` may add `install-deps` extras for fonts/Xvfb.)*
- Proxy is read via `app/utils/proxy/get_proxy_url()` (`__init__.py:13`), backed by the `PROXY_PROVIDER` registry (`config/__init__.py:983`). `03a` leaves this single-URL model untouched (rotation is `03b`).
## Target design
### Tier ladder (Scrapling-only)
`crawl_url()` becomes a 3-tier ladder, preserving the existing thread-offload + `NotImplementedError` handling for the browser tiers (Windows `SelectorEventLoop` cannot spawn subprocesses — lines 134–141, 161–168):
1. `AsyncFetcher.get(...)` — fast static HTTP. **TLS gap to close:** the current call passes `stealthy_headers=True` but **not** `impersonate` (`webcrawler_connector.py:284–289`), so its TLS ClientHello is curl_cffi's *default* JA3 — trivially bot-flagged and incoherent with the browser tiers' UA. Add an `impersonate="chrome"` profile here (Scrapling's static engine accepts it — `references/Scrapling/scrapling/engines/static.py:36–47`). Tracked as a lever in `03e §2b` and validated by `03f §S3`.
2. `DynamicFetcher.fetch(...)` — full browser (via `asyncio.to_thread`).
3. `StealthyFetcher.fetch(...)` — patchright-Chromium anti-bot, last resort. Enable Cloudflare solving here by passing **`solve_cloudflare=True`** — a documented `StealthyFetcher.fetch` kwarg ("Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response", `references/Scrapling/scrapling/fetchers/stealth_chrome.py:38`; it's a `StealthSession` TypedDict key passed via `**kwargs`). The current stealthy call (connector lines 354–360) passes `headless`/`network_idle`/`block_ads`/`proxy` but **not** `solve_cloudflare`, so this is a real behavior add. (Note: `solve_cloudflare` runs the full browser challenge loop, so it's correctly scoped to the last-resort tier only.)
Trafilatura extraction (`_build_result`) and `format_to_structured_document()` are unchanged.
### Explicit outcome model
Replace the implicit `(dict|None, str|None)` contract with an explicit outcome so callers (indexer, chat tool, and `03c` metering) agree on what "success" means. Proposed:
```python
class CrawlOutcomeStatus(str, Enum):
SUCCESS = "success" # a tier returned usable extracted content
EMPTY = "empty" # fetched, but no usable content after ALL tiers
FAILED = "failed" # invalid URL or every tier errored
```
`crawl_url()` returns a small dataclass `CrawlOutcome(status, result, error, tier)` — **commit to the dataclass** (not a tuple): `03c` keys billing off `status == SUCCESS`, and Phase 6's fetch-only path (`06-pipelines-exec.md`) consumes `outcome.status` / `outcome.result` / `outcome.error` as attributes, so a tuple form would break that consumer. The **billable success predicate is single-sourced**: `status == CrawlOutcomeStatus.SUCCESS`. Picking a dataclass (over a tuple) also leaves room for later subplans to **append fields without breaking callers** — `03d` adds `captcha_attempts` / `captcha_solved` for per-attempt billing, and `03e`'s block classifier can attach a `block_type`. (This is distinct from the indexer's positional return, which must stay 2-tuple — see the wrapper note below.)
| Outcome | When | Billable (`03c`)? | Document status (indexer) |
|---------|------|-------------------|---------------------------|
| `SUCCESS` | a tier extracted usable content (`_build_result` returned a dict) | **Yes — 1 unit** | `ready` (or unchanged/duplicate, see note) |
| `EMPTY` | every tier was reached but none produced usable extracted content (the static tier also treats HTTP ≥ 400 as a miss and falls through — connector lines 292–301; the browser tiers attempt extraction regardless of status) | No | `failed("No content extracted")` |
| `FAILED` | invalid URL, or all tiers raised | No | `failed(<error>)` |
**Billing-policy note for `03c`:** success is the *crawl* succeeding (we fetched + extracted), independent of downstream KB dedupe. The indexer currently marks unchanged content as `skipped` (`webcrawler_indexer.py:341–347`) and cross-connector duplicates as `failed` (`:350–369`) — those still represent a **successful crawl** and should bill. `03c` must count `CrawlOutcomeStatus.SUCCESS`, not `documents_indexed`. Flagging here; final call lives in `03c`.
### Extensibility seam (the tier ladder is a strategy chain)
The 3-tier ladder is the first instance of a deliberate **`FetchStrategy` seam**: an *ordered list of strategies*, each `(url, ctx) -> CrawlOutcome`, tried in order until one returns `SUCCESS`. `crawl_url()` owns the chain; **every caller depends only on `CrawlOutcome`, never on which strategy produced it** (the indexer, the chat tool, and `03c` metering already do — keep that invariant sacred). This is what lets the moat grow without rework:
- `03d` (captcha) attaches by escalating to the StealthyFetcher strategy with a `page_action` token-injector — a *parameterization* of the last tier, not a new caller contract.
- `03e` (stealth-hardening) tunes/adds strategies (humanize, headed/Xvfb, persistent profiles, fingerprint flags) **behind the same return type**.
- A future **paid-unblocker tier** (deferred — see `03e`) is just one more strategy appended last, flippable by config; no caller changes.
- Future **platform actors** (Phase 8) reuse the same fetch strategies under their own structured extractors.
MVP scope here is only the 3 Scrapling tiers + the `CrawlOutcome` contract; the seam is a design constraint (keep tiers pluggable + callers outcome-only), **not** a call for a heavyweight Strategy framework now.
### Success counter for the indexer
Add an explicit `crawls_succeeded` counter in `index_crawled_urls()` incremented whenever `crawl_url` returns `SUCCESS` (right after line 297's call, before the dedupe/unchanged branches), and surface it in the task-success metadata (lines 455–466). `03c` meters against this **in-function** counter (it charges inside the indexer — the count does not need to escape via the return).
> **Do NOT widen the positional return tuple.** The shared `_run_indexing_with_notifications` wrapper unpacks every indexer's return **by length** (`search_source_connectors_routes.py:1499–1507`: `if len(result) == 3: a,b,c = result else: a,b = result`) — a 3-tuple would mislabel `crawls_succeeded` as `documents_skipped`, a 4-tuple would raise `ValueError`. Keep the existing `(total_processed, error)` shape. **Phase 6** (`06-pipelines-exec.md`) later exposes `crawls_succeeded`/`documents_indexed`/`crawls_attempted` to the pipeline run engine via an optional `stats` **out-param** (plus `folder_id`/`urls`/`bill`), not via the return — so `03a` only needs the counter + metadata here.
### Chat scrape tool
Drop the Firecrawl-only `formats=["markdown"]` arg (markdown is already the Trafilatura default). The tool keeps returning its asset dict; map the new outcome onto the existing `error` / `content` shape (lines 235–269) with no behavioral change for the agent.
## Work items
1. **Rip out Firecrawl** across the surface table above (crawler, dep, validators, indexer, chat plumbing, automations, tests) — code-only, no env/docs change.
2. **Refactor `crawl_url`** to the 3-tier Scrapling ladder + explicit `CrawlOutcome`; enable `solve_cloudflare` on the stealthy tier.
3. **Add `crawls_succeeded`** counting in `webcrawler_indexer.py` + expose in **task metadata only** (03c bills off the in-function counter). **Do not change the positional return tuple** (the shared wrapper unpacks by length; Phase 6 adds a `stats` out-param).
4. **Update both `scrape_webpage` tools** to drop `firecrawl_api_key` + `formats`.
5. **Tests:** unit tests for `crawl_url` outcomes (mock Scrapling fetchers → SUCCESS/EMPTY/FAILED); update `test_dependencies.py`; assert the indexer's `crawls_succeeded` count.
## Risks / trade-offs
- **Loss of a managed fallback.** Firecrawl was a hosted last resort for hostile anti-bot sites. Mitigation: the in-house stack — `StealthyFetcher` + Cloudflare solving (this plan), `03b` proxy rotation, `03e` stealth-hardening (humanize/headed/profiles/fonts + block-classifier), `03d` captcha solving — plus a **deferred paid-unblocker tier** behind the seam for the hostile residual. Realistic ceiling: this defeats Cloudflare + the long tail of moderate anti-bot, but runtime-level (patchright) stealth does **not** reliably beat top-tier fingerprinting (DataDome/Kasada/reCAPTCHA-Enterprise) — that's the deferred paid tier's job (`03e`). Acceptable per the decisions log (single-framework intent + in-house moat).
- **Browser tiers on dev/Windows.** `DynamicFetcher`/`StealthyFetcher` need subprocess support; the existing `to_thread` + `NotImplementedError` guards (lines 134–141, 161–168) are preserved so static-only crawling still works in `uvicorn --reload`.
- **`uv.lock` churn.** Removing `firecrawl-py` requires a lockfile regen + image rebuild; no new runtime deps are added.
## Out of scope (hand-offs)
- Proxy provider expansion + rotation → `03b`.
- Crawl credit metering on `CrawlOutcomeStatus.SUCCESS` → `03c`.
- reCAPTCHA/hCaptcha solving via captcha-tools → `03d` (**now active**, sequenced after `03e`). Cloudflare Turnstile stays in-framework (Scrapling).
- Stealth-hardening (humanize, headed/Xvfb, persistent profiles, fonts, geoip locale/tz, block-classifier + per-domain strategy memory) and the deferred paid-unblocker tier → `03e`.
- Whether ad-hoc **chat** scrapes are billed (vs only pipeline crawls) → decided in `03c`.

130
plans/backend/03b-proxy-expansion.md
View file

@ -1,130 +0,0 @@
# Phase 3b — Proxy provider expansion + rotation
> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
> Depends on `03a-crawler-core.md` (Scrapling-only crawler). Siblings: `03c-crawl-billing.md`, `03e-stealth-hardening.md`, `03d-captcha-solving.md`.
> **Implementation note.** Same convention as `03a`: Phases 1–2 are **SHIPPED**, so the live code already says `workspace_id`/`Workspace` — substitute for the old `search_space_*`/`SearchSpace` names in citations below and grep the new name. Crucially, `03a` is **now IMPLEMENTED**: the crawler moved to **`app/proprietary/web_crawler/connector.py`** (non-Apache-2 boundary) with Firecrawl removed and `crawl_url` refactored — the three Scrapling tiers now pass `proxy=get_proxy_url()` at **267/309/342** (not the pre-refactor 287/329/359). Locate code by **symbol/grep** against the new path, not absolute lines.
> **Status: ✅ IMPLEMENTED (`ci_mvp` @ `6226012`).** `CustomProxyProvider` added (single URL or rotating pool via Scrapling `ProxyRotator`) and registered as `"custom"`; `Config.CUSTOM_PROXY_URL`/`CUSTOM_PROXY_URLS` + `.env.example` docs; `is_pool_backed` added to the `ProxyProvider` ABC (default `False`) + a zero-arg `is_pool_backed()` package helper. The crawler does a **bounded, one-per-tier** `is_proxy_error` rotation-retry gated on `is_pool_backed()` — single-endpoint providers (incl. server-side-rotating `anonymous_proxies`) no-op. The zero-arg getter contract is unchanged for all consumers (crawler/YouTube/chat tools). Tests: provider (single/pool/dedupe/empty/playwright), registry (custom/anonymous/unknown-fallback), crawler rotation (retry / no-retry / non-proxy) — **16 passing**.
## Objective
Let the WebURL Crawler (and every other proxy consumer) run behind **either** the existing `anonymous_proxies` gateway **or** a new BYO `CustomProxyProvider`, chosen by a **single, app-wide** `Config.PROXY_PROVIDER`. When the active provider is backed by a **pool of endpoints**, rotate across them client-side with a bounded retry. **No branded vendors and no per-connector/per-crawl provider divergence** (resolved decisions): the whole app uses one provider.
The provider abstraction already exists and is clean; the real gaps are just (1) only `anonymous_proxies` is registered (no BYO option), and (2) there is **no client-side rotation/retry** when a provider exposes multiple endpoints. The "process-global, env-only" shape is exactly what we want for a single-provider app, so we keep it.
## Current state (cited)
### A clean but process-global abstraction
`app/utils/proxy/` is already a "subclass + register" provider package:
- `base.py` — `ProxyProvider` ABC: `get_proxy_url()` (canonical `http://user:pass@host:port` for Scrapling/curl_cffi), `get_playwright_proxy()`, `get_requests_proxies()` (built from the URL by default).
- `registry.py` — `_PROVIDERS` dict keyed by provider `name`; `get_active_provider()` resolves `Config.PROXY_PROVIDER` and **caches a single instance process-wide** (`_active_provider`, lines 23, 26–44). Only `AnonymousProxiesProvider` is registered (lines 17–19).
- `__init__.py` — zero-arg module helpers `get_proxy_url()` / `get_playwright_proxy()` / `get_requests_proxies()` + a `get_residential_proxy_url()` back-compat alias, all delegating to `get_active_provider()`.
- `providers/anonymous_proxies.py` — the one vendor. Note it's a **server-side rotating gateway**: host is `rotating.dnsproxifier.com:PORT` and the rotation `type`/location are encoded into a base64 "password" dict (lines 23–51). So per-request IP rotation already happens **upstream**; the client sends one static endpoint.
- `app/utils/proxy_config.py` — a thin back-compat shim re-exporting the package (kept for old import paths).
Config knobs: `Config.PROXY_PROVIDER` (default `"anonymous_proxies"`) + `RESIDENTIAL_PROXY_{USERNAME,PASSWORD,HOSTNAME,LOCATION,TYPE}` (`config/__init__.py:983–992`; documented commented-out in `.env.example:300–309`).
### Who consumes the proxy (blast radius)
The zero-arg getters are used in **more than just the crawler**, so any signature change must stay backward-compatible:
| Consumer | File:line | Uses |
|----------|-----------|------|
| WebURL crawler (all 3 Scrapling tiers) | `proprietary/web_crawler/connector.py` (267, 309, 342) | `get_proxy_url()` |
| YouTube transcript route | `routes/youtube_routes.py` (78, 119) | `get_proxy_url()` |
| YouTube processor (indexing) | `tasks/document_processors/youtube_processor.py` (223, 229) | `get_requests_proxies()`, `get_proxy_url()` |
| Chat scrape tools (YouTube branch) | `main_agent/tools/scrape_webpage.py` (80, 93), `research/tools/scrape_webpage.py` (74, 87) | `get_requests_proxies()`, `get_proxy_url()` |
**Implication:** the whole app shares **one** global provider, so every consumer keeps calling the zero-arg getters **unchanged**. The only behavior change is (a) *which* provider class those getters resolve to (selected by `Config.PROXY_PROVIDER`) and (b) optional client-side rotation when that provider is pool-backed. No caller passes a key; no signatures change.
**Shape note (verified):** `get_playwright_proxy()` has **no consumers** anywhere — only the ABC/impl/exports define it. All three crawler tiers and YouTube's `AsyncFetcher.get` consume the **string** form (`get_proxy_url`); browser fetchers accept a string proxy (`references/Scrapling/.../stealth_chrome.py:47` — "it can be a string or a dictionary"). The dict form (`get_requests_proxies`) is consumed only by YouTube-transcript fetches (`youtube_processor.py:223`, plus the chat tools' YouTube branch `main_agent/tools/scrape_webpage.py:80` and `research/tools/scrape_webpage.py:74`) — none of which are crawler tiers. **Conclusion:** rotation only needs the **string** shape; the playwright dict can be ignored (or left as-is on providers) rather than threaded through rotation.
### Scrapling's rotation primitive
`ProxyRotator(proxies: list, strategy=cyclic_rotation)` is a thread-safe cycler over a **static list** of proxy URLs/dicts (`get_proxy()`, `proxy_rotation.py:88–92`) plus `is_proxy_error(exc)` (`:27–30`) that matches proxy failure strings (`net::err_proxy`, `connection refused`, …). It does **not** manage credentials or sessions — it just hands out the next endpoint. So it's only useful when we have a **pool of distinct endpoints**; it adds nothing for a single server-side-rotating gateway like `anonymous_proxies`.
Both are **publicly importable**: `from scrapling.engines.toolbelt import ProxyRotator, is_proxy_error` (re-exported in `engines/toolbelt/__init__.py:1–3`). The thread-safe `Lock` (`:71,90`) makes `ProxyRotator` safe to call from the browser tiers that `03a` runs via `asyncio.to_thread`. Version is guaranteed: the pinned floor is `scrapling[fetchers]>=0.4.9` (`pyproject.toml:91`) and these APIs exist as of `0.4.9` (`scrapling/__init__.py:2`).
## Selection scope (resolved)
The user expects a **single proxy provider across the entire app**, so 03b's selection scope is simply the **global, env-configured provider** — no per-connector or per-crawl override is built now. That is both the fastest path and, with one thin seam, the most scalable later:
- All resolution stays behind today's `get_active_provider()` (env-selected, process-cached). One provider, app-wide.
- "Scalable in future": if per-capability / per-Tracker proxying is ever wanted (revamp Phase 4a/5+), it layers on as an **optional argument** to a resolver (`resolve_proxy(override=...)`) with **zero** changes to existing call sites — but it is explicitly **not** implemented in 03b (YAGNI for a single-provider app).
## Target design
### 1. Registry: register the BYO provider; keep single-provider selection
The existing `get_active_provider()` (env-selected via `Config.PROXY_PROVIDER`, cached process-wide in `_active_provider` — `registry.py:23,26–44`) is exactly the single-provider model we want — keep it. The only change is to **register `CustomProxyProvider`** so `PROXY_PROVIDER` can select it:
- `_PROVIDERS["anonymous_proxies"] = AnonymousProxiesProvider` (existing) + `_PROVIDERS["custom"] = CustomProxyProvider` (new).
- `get_active_provider()` resolves the **one** active provider from env and caches it — **no keyed multi-provider coexistence** (`get_provider(key)` is intentionally NOT added; YAGNI for a single-provider app).
- Unknown `PROXY_PROVIDER` → existing warn-and-fallback path (`registry.py:35–41`) is unchanged.
### 2. Provider config source: env only
Credentials come from **env**, full stop (single global provider):
- `anonymous_proxies` → existing `RESIDENTIAL_PROXY_*`. Unchanged.
- `custom` → new env knobs (one URL or a pool): `CUSTOM_PROXY_URLS` (comma-separated) and/or `CUSTOM_PROXY_URL`.
No per-connector proxy keys, and **no** connector-config validator changes (that was the per-connector path, now dropped).
### 3. New provider: `CustomProxyProvider` (BYO) — the only addition
- Accepts a raw proxy URL **or** a list of URLs from env. Covers "bring your own proxy / our own pool" with no vendor-specific auth assumptions.
- Implements just `get_proxy_url()`; the base derives `get_requests_proxies()` from it (`base.py:37–46`). When configured with a **pool**, `get_proxy_url()` returns the *next* endpoint from an internal `ProxyRotator` (see §4) — so rotation is transparent to every caller of the zero-arg getter.
- **No branded vendors** (resolved): Webshare/BrightData/Smartproxy/etc. are not shipped. A user who wants any specific vendor points `CustomProxyProvider` at that vendor's endpoint(s) via `CUSTOM_PROXY_URLS`.
### 4. Rotation + retry (only when the active provider is pool-backed)
- If `CustomProxyProvider` is configured with **multiple** URLs, it wraps them in Scrapling's `ProxyRotator` (cyclic default, thread-safe `Lock` — `proxy_rotation.py:71,88–92`) and returns the next endpoint per `get_proxy_url()` call. Because `03a`'s browser tiers run under `asyncio.to_thread`, the rotator's lock keeps this safe.
- The crawler adds a **bounded** client-side retry: on a tier failure where `is_proxy_error(exc)` is true (`proxy_rotation.py:27–30`), it re-reads `get_proxy_url()` (next endpoint) and retries **that tier once** before falling through. One rotation-retry per tier — no unbounded fan-out on billable crawls.
- Single-endpoint providers (`anonymous_proxies`, already server-side rotating; or `custom` with one URL) return the same endpoint every call, so the retry is a harmless no-op for them.
### 5. Crawler stays on the zero-arg getter (no injection needed)
Because there is one global provider and rotation lives **inside** it, the crawler does **not** need a proxy injected per crawl:
- All three Scrapling tiers keep calling `get_proxy_url()` (which yields the rotating value when pool-backed). The only crawler edit is the bounded `is_proxy_error` retry from §4.
- **No change** to YouTube/chat-tool consumers — same zero-arg getters, same global provider.
- Future seam (NOT built): if per-capability / per-Tracker proxying is ever needed, resolve a provider from the capability/Tracker config and pass its `get_proxy_url` into the crawl — an additive optional arg, no change to today's call sites.
## Config / env changes
- `config/__init__.py:983–992` + `.env.example:300–309`: add `CUSTOM_PROXY_URLS` (comma-separated pool) and/or `CUSTOM_PROXY_URL` for the `custom` provider; keep all existing `RESIDENTIAL_PROXY_*` working unchanged. `PROXY_PROVIDER` now accepts `"custom"` in addition to `"anonymous_proxies"`.
- **No** connector-config validator changes (the per-connector proxy path is dropped — single global provider).
## Work items
1. **`CustomProxyProvider`**: BYO single-URL or pool-of-URLs provider (reads `CUSTOM_PROXY_URL(S)`); internal `ProxyRotator` when pool-backed; register as `"custom"` in `_PROVIDERS`.
2. **Rotation retry**: add a bounded (one-per-tier) `is_proxy_error` retry to the crawler's Scrapling tiers; single-endpoint providers no-op.
3. **Config + docs**: `CUSTOM_PROXY_URL(S)` in `Config` + `.env.example`; document `PROXY_PROVIDER="custom"`.
4. **Tests**: `CustomProxyProvider` single-URL vs pool; `ProxyRotator` cyclic order; `get_active_provider()` resolves `"custom"` and still warn-falls-back on unknown; crawler rotates once on `is_proxy_error` then falls through; `anonymous_proxies` + YouTube path unchanged (same endpoint each call).
## Risks / trade-offs
- **Backward compatibility of the getters.** The zero-arg `get_proxy_url()` contract is consumed in 4+ places; the design keeps it **completely intact** — no signature changes, no per-caller keys. Only the cached active provider's *implementation* changes when `PROXY_PROVIDER="custom"`.
- **Over-rotation cost.** Rotation-retry is bounded to one extra attempt per tier so a billable crawl can't silently multiply upstream proxy usage.
- **Single provider is a deliberate constraint.** One global provider app-wide (resolved). Per-capability/per-Tracker/per-connector selection is intentionally deferred behind a no-op seam (§5); not built now.
- **Pool rotation under `to_thread`.** `ProxyRotator`'s `Lock` makes the rotating `get_proxy_url()` safe to call from the browser tiers `03a` offloads via `asyncio.to_thread`.
## Resolved decisions
- **Branded vendors → NONE.** Ship `CustomProxyProvider` (BYO) only; no Webshare/BrightData/Smartproxy/etc. subclasses. A user who wants a specific vendor points `CUSTOM_PROXY_URLS` at it.
- **Selection scope → single global provider, app-wide.** No per-connector/per-crawl override is built; resolution is env-only via `Config.PROXY_PROVIDER`. A future per-capability/per-Tracker override is left as a no-op seam (§5) so it stays "scalable + fast to add later."
- **Client-side rotation → built, but only active for a pool-backed `CustomProxyProvider`.** `anonymous_proxies` (server-side rotating) and single-URL custom configs skip it automatically. Rotation lives inside the provider so it's transparent to all callers.
## Out of scope (hand-offs)
- Per-**capability** / per-Tracker / per-connector proxy selection → deferred (revamp Phase 4a/5 *if ever needed*); §5 leaves a no-op seam, nothing is wired now.
- Branded-vendor provider subclasses → not planned (use `CustomProxyProvider`).
- **Static / sticky-session proxies (future).** A later capability will add **static proxy** support — sticky IPs held for the duration of a session — most likely paired with **authenticated/account-based scraping** to bypass logged-in platforms (the deferred platform connectors: LinkedIn, Instagram, etc.). This is a *different axis* from the rotating pool here: rotation maximizes IP diversity, whereas account bypass needs IP **stability** so a session/cookie stays bound to one IP. It is additive to this design — a new `ProxyProvider` (or a "sticky" mode/flag on `CustomProxyProvider`) registered under a new `PROXY_PROVIDER` key, with no change to the zero-arg getter contract — and stays consistent with the single-provider model (the active provider would be the static one when that workflow is selected). Build it alongside the platform connectors, not in Phase 3.
- Crawl credit metering (proxy cost is absorbed into the flat `$1 / 1000 successful` price, **not** metered separately) → `03c`.
- Captcha solving → `03d`.
- **Geoip fingerprint coherence** (matching browser `locale`/`timezone_id` to the proxy's exit geo) → `03e`. 03b only owns *selecting* the proxy; `03e` consumes the chosen endpoint's geo. The provider's `RESIDENTIAL_PROXY_LOCATION` (`config/__init__.py`) is one input; resolving the actual exit IP's geo is `03e`'s job.
- **Surfacing the crawl's chosen endpoint into the strategy.** Both `03d` (IP-bound captcha solves) and `03e` (geoip + sticky reuse) need the **exact** endpoint a crawl used, *not* a fresh `get_proxy_url()` call (which rotates on a pool-backed `CustomProxyProvider`). The capture-once seam lives in `03a`'s `FetchStrategy` context; 03b only guarantees the rotating getter and notes the consumers.

140
plans/backend/03c-crawl-billing.md
View file

@ -1,140 +0,0 @@
# Phase 3c — Crawl credit billing ($1 / 1000 successful requests)
> **Status: ✅ IMPLEMENTED (`ci_mvp` @ `17bdb0682`).** `WebCrawlCreditService` added (mirrors `EtlCreditService`: gate → `check_credits` → `charge_credits`, plus static `billing_enabled`/`successes_to_micros`). Price is fully config-driven (`WEB_CRAWL_MICROS_PER_SUCCESS`, default `1000` = $1/1000; `WEB_CRAWL_CREDIT_BILLING_ENABLED`, default FALSE) — no hardcoded rate. Indexer wires owner-resolution + pre-flight block + audit-then-charge; both `scrape_webpage` tools fold one success into the turn accumulator (`call_kind="web_crawl"`). 20 new unit tests pass (service, config-driven price, indexer wiring, chat fold); `connector_indexers` suite green (120). Functional e2e (3a/3b/3c) green via `scripts/e2e_phase3_crawl_billing.py` (live proxy egress flip, static-tier crawl, chat fold = 1000µ, indexer bills owner 2×1000µ not trigger). Captcha per-attempt unit (`03d`) seam left open. **No migration** (`web_crawl` rides the existing `TokenUsage.usage_type` String(50)).
> **⚠️ Billing surfaces reframed (2026-06-30 — Architecture correction + revamp adoption, `00-umbrella-plan.md`).** The implemented `WebCrawlCreditService` **stands unchanged**, but *where* it's called shifts, and the **charge moves to the capability executor** ([`04-capabilities.md`](04-capabilities.md)). Pipelines are dropped and `WEBCRAWLER_CONNECTOR` is retired, so the **connector/`webcrawler_indexer`** billing branch (described throughout this doc) is now **vestigial** (dormant, not deleted). The correct model: every `web.*`/`maps.*` verb **charges once per billable success inside its executor**, so all callers meter uniformly — chat, ongoing automation, REST, MCP. This closes a **code-verified gap**: `run_agent_task` (automation `agent_task`) establishes **no turn accumulator**, so the shipped `scrape_webpage` fold (no-op when `get_current_accumulator()` is `None`) bills **nothing** under automations today. The interactive **chat turn accumulator becomes an optional presentation fold** (still shows the crawl line on a chat turn), **not** the charging mechanism. Read "connector/pipeline crawls" below as historical context.
> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
> Depends on `03a-crawler-core.md` (the `CrawlOutcomeStatus.SUCCESS` signal + `crawls_succeeded` counter). Siblings: `03b-proxy-expansion.md`, `03e-stealth-hardening.md`, `03d-captcha-solving.md` (now active). `03d` extends this service with a **separate per-attempt captcha unit** (see §"Captcha billing seam" + `03d §4`).
## Objective
Charge **$1 per 1000 successful crawl requests** = **1000 micro-USD per successful crawl**, drawn from the existing unified credit wallet, **off by default for self-hosted/OSS** installs. The rate is **fully config-driven** — a single env/`Config` knob (`WEB_CRAWL_MICROS_PER_SUCCESS`) with **no hardcoded price anywhere** — so it can be retuned to any future rate with just an env change + restart, never a code edit (same convention as `MICROS_PER_PAGE`). $1/1000 is only the *default*. Two surfaces crawl, so both are metered:
- **Connector / pipeline crawls** (the `webcrawler_indexer` path) → billed to the **workspace owner** via a dedicated `WebCrawlCreditService` (check + charge).
- **Ad-hoc chat scrapes** (`scrape_webpage` tools) → the crawl cost is **folded into the chat turn's existing bill** (added to the turn accumulator), so it settles with the turn's normal premium finalize (`start_turn()` + `credit_finalize`). No second wallet hit. (Implies a second gate: only charged when the turn itself is a premium/billed turn — see §3.)
> **Migration impact: none.** This phase adds config flags + code only. `TokenUsage.usage_type` is already a free-form `String(50)`, so the new `web_crawl` value needs **no Alembic migration or schema change**. (Per the standing guidance, the migration-sensitive work lives in Phases 1 and 5+; 03c is purely additive.)
## The key realization (don't use `billable_call`)
My first instinct was to add a flat reserve/finalize path around `TokenQuotaService` because `billable_call` settles at the LLM token accumulator (`acc.total_cost_micros`, `billable_calls.py:367`) — which is 0 for a non-LLM crawl, so `billable_call` would finalize at nothing.
But there's already a **purpose-built precedent for per-unit, non-LLM wallet billing**: `app/services/etl_credit_service.py` (`EtlCreditService`). It deliberately **skips** the reserve/finalize dance and does a simple **gate → pre-check → post-charge**:
- `billing_enabled()` → `config.ETL_CREDIT_BILLING_ENABLED` (`etl_credit_service.py:44–46`); **every method is a no-op when disabled** — that's what keeps self-hosted "effectively free" (`:9–11`, `:59–60`, `:84–85`, `:115–116`).
- `pages_to_micros(pages, multiplier)` → `pages * multiplier * config.MICROS_PER_PAGE` (`:48–51`).
- `check_credits(user_id, estimated_pages)` → raises `InsufficientCreditsError` if `required > available` where `available = balance - reserved` (`:76–102`).
- `charge_credits(user_id, pages)` → `user.credit_micros_balance -= cost; commit; maybe_trigger_auto_reload(...)` (`:104–138`).
It's wired into the **sibling connector indexers** exactly the way we need: `google_drive_indexer.py` (`558,562,605`), `dropbox_indexer.py` (`426,504,593`), `local_folder_indexer.py` (`639,698,864`) — instantiate `EtlCreditService(session)`, `check_credits(...)` before processing, `charge_credits(...)` after.
**So 03c mirrors `EtlCreditService` with a crawl-specific unit and flag — no `billable_call`, no reserve/finalize.**
## Current state (cited)
- **Crawled URLs are billed nothing today.** `webcrawler_indexer.py` does not import `EtlCreditService` or touch the wallet (verified — it's absent from the file). The crawler extracts content directly via Trafilatura and bypasses the ETL page pipeline, so there's **no existing ETL charge** to double up with.
- **Wallet model** (`token_quota_service.py`): the balance lives on `User.credit_micros_balance` / `User.credit_micros_reserved`; spendable = `balance - reserved`. Direct debit is exactly what `EtlCreditService.charge_credits` does.
- **Audit trail**: `record_token_usage(session, *, usage_type, search_space_id, user_id, cost_micros=…, call_details=…, message_id=None, …)` (`token_tracking_service.py:517–568`) inserts a `TokenUsage` row (best-effort; caller commits). `TokenUsage.usage_type` is a `String(50)` indexed column explicitly intended for non-chat usage with `message_id` NULL (`db.py:1068–1116`, comment at `:1072–1084`: "indexing, image generation, podcasts — which keep message_id NULL").
- **Per-feature config knobs** are the convention: `MICROS_PER_PAGE` (`config/__init__.py:655`, default `1000`), `ETL_CREDIT_BILLING_ENABLED` (`:652–653`, default FALSE), `QUOTA_DEFAULT_IMAGE_RESERVE_MICROS` (`:729`), etc. `maybe_trigger_auto_reload` (`auto_reload_service`) is the shared low-balance top-up nudge.
> **Implementation note.** Same convention as `03a`/`03b`: Phases 1–2 are **SHIPPED**, so the live code already says `workspace_id`/`Workspace` — substitute for the old `search_space_*`/`SearchSpace` names in citations below and grep the new name (e.g. the `record_token_usage(search_space_id=…)` kwarg is now `workspace_id=…`; `_resolve_agent_billing_for_search_space`→`_resolve_agent_billing_for_workspace`; `select SearchSpace.user_id`→`select Workspace.user_id`). Locate code by **symbol/grep**, not the absolute line numbers cited here, since the rename and `03a`'s `crawl_url`/`scrape_webpage` refactor shift them.
## Target design
### 1. `WebCrawlCreditService` (mirror of `EtlCreditService`)
New `app/services/web_crawl_credit_service.py` — a dedicated service (independent flag + price; lowest risk, matches the per-feature-knob convention). The two `@staticmethod` helpers are the **shared primitives both surfaces reuse**:
- `billing_enabled()` → `config.WEB_CRAWL_CREDIT_BILLING_ENABLED` (new flag, default FALSE). *(static — also called by the chat tool)*
- `successes_to_micros(n)` → `n * config.WEB_CRAWL_MICROS_PER_SUCCESS` (new knob, default `1000` = $1/1000). *(static — also called by the chat tool)*
- `check_credits(owner_user_id, estimated_urls)` → no-op when disabled; else raise `InsufficientCreditsError` (reuse the one from `etl_credit_service`) if `successes_to_micros(estimated_urls) > available`.
- `charge_credits(owner_user_id, successes)` → no-op when disabled; else debit `credit_micros_balance -= successes_to_micros(successes)`, commit, `maybe_trigger_auto_reload`.
A separate `CreditMeterService` generalization is deliberately deferred until a third per-unit biller appears — exposing the price/flag as statics already lets the chat surface share the math without coupling the two flows.
**Captcha billing seam (for `03d`).** `03d` (now active) adds a **separate per-attempt** captcha unit to this same service — `captcha_solves_to_micros(n) = n * config.WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE`, gated by `WEB_CRAWL_CAPTCHA_BILLING_ENABLED` (both new knobs, default FALSE), with `usage_type="web_crawl_captcha"`. It is **independent** of crawl SUCCESS because solvers charge per *attempt* regardless of crawl outcome (the asymmetry in the §"Risks" proxy/captcha note). Build the `WebCrawlCreditService` statics so `03d` can add one more `*_to_micros` static + one charge path without restructuring — no need to implement the captcha methods in `03c`, just leave the shape open.
### 2. Wiring in `webcrawler_indexer.py`
1. **Resolve the owner.** Bill the **workspace owner** (the product concept), not the triggering `user_id`: `owner_user_id = (select SearchSpace.user_id where id == search_space_id)`. (Mirrors how `billable_calls._resolve_agent_billing_for_search_space` (func `:441`) reads `search_space.user_id` at `billable_calls.py:499` — but we only need the owner id, so a direct `select` is enough; no need to call that LLM-model-resolving helper.)
2. **Pre-flight gate** (after URL parse, before the crawl loop — i.e. the indexer's second pass that actually fetches URLs; "phase 2 of the 2-phase indexer", not roadmap Phase 2): `await svc.check_credits(owner_user_id, len(urls))`. On `InsufficientCreditsError`, `log_task_failure` with a clear "out of crawl credit" message and return — don't crawl. Because **successes ≤ len(urls)**, this upper-bound check means the wallet can never go negative from crawls (unlike ETL, where actual pages can exceed the estimate).
3. **Audit (add, don't commit)**: `record_token_usage(session, usage_type="web_crawl", search_space_id=search_space_id, user_id=owner_user_id, cost_micros=successes_to_micros(crawls_succeeded), call_details={"urls": len(urls), "successes": crawls_succeeded, "connector_id": connector_id}, message_id=None)` — `record_token_usage` only `session.add`s the row (`token_tracking_service.py:552`), it does not commit.
4. **Charge actuals**: `await svc.charge_credits(owner_user_id, crawls_succeeded)`. `charge_credits` debits the balance and **commits** (mirroring `EtlCreditService.charge_credits`), which flushes the step-3 audit row **and** the balance debit in one transaction.
Ordering matters: audit must be added **before** the charge, because the charge's commit is what persists both. Both run on the indexer's existing `session` (same as the ETL services), after the final per-URL document commit (`webcrawler_indexer.py:432`). If `crawls_succeeded == 0`, skip both (no-op).
### 3. Chat scrape crawls (folded into the chat turn's bill)
The chat `scrape_webpage` tools (`main_agent/tools/scrape_webpage.py:183`, plus the `research` sibling) crawl via `WebCrawlerConnector.crawl_url` (`main_agent/tools/scrape_webpage.py:232–233`) — the **same** call `03a` refactors. The chat hook therefore keys off `03a`'s `CrawlOutcomeStatus.SUCCESS` return (a hard dependency on 03a's new `crawl_url` signature).
**How the chat turn actually bills** (this is *not* `billable_call`): the chat orchestrators create the turn accumulator with `start_turn()` (`new_chat/orchestrator.py:185`, `resume_chat/orchestrator.py:147`, `anonymous_chat_routes.py:360`); a premium turn reserves up front and, at the end, `finalize_credit` debits `accumulator.total_cost_micros` via `credit_finalize` (`shared/premium_quota.py:83–104`). So adding to the accumulator before finalize is debited with the turn:
```python
from app.services.token_tracking_service import get_current_accumulator
from app.services.web_crawl_credit_service import WebCrawlCreditService
# after a SUCCESS crawl in the scrape tool
acc = get_current_accumulator() # token_tracking_service.py:212–213 (turn ContextVar)
if acc is not None and WebCrawlCreditService.billing_enabled():
acc.add( # :112–135 — appends a TokenCallRecord
model="web_crawl",
prompt_tokens=0, completion_tokens=0, total_tokens=0,
cost_micros=WebCrawlCreditService.successes_to_micros(1),
call_kind="web_crawl",
)
```
`accumulator.total_cost_micros` sums every call's `cost_micros` (`:174–186`), so the crawl micros ride along into the premium finalize — "included in the already-billed chat." `call_kind="web_crawl"` keeps it distinguishable in `per_message_summary` (`:137–159`).
**Why the ContextVar is reliably visible (not hand-waving):** the LiteLLM `TokenTrackingCallback` already populates *this same* `_turn_accumulator` from deep inside the graph's LLM calls — that's how chat billing works today. If the ContextVar set by `start_turn()` reaches the LLM callback, it reaches a `scrape_webpage` tool in the same graph. The tools are `async` and `await crawl_url` in-loop, so the context is intact (the only thing that would break propagation is offloading to a raw OS thread without `copy_context`, which this path doesn't do).
Two gates, both required, for a chat scrape to actually cost money:
- **`WEB_CRAWL_CREDIT_BILLING_ENABLED`** (self-hosted: off → adds nothing).
- **The turn is premium.** `needs_credit_quota` = `agent_config.is_premium` (`premium_quota.py:43–44`). Free / BYOK / **anonymous** (`anonymous_chat_routes.py:360`, no wallet) turns never reserve/finalize → the `web_crawl` cost is recorded in the accumulator breakdown but **not debited**. This is intentional: if the chat turn isn't billed, neither is its scrape.
Other notes:
- **No per-scrape pre-block.** The turn's up-front reservation is an LLM-only estimate (`reserve_credit`/`estimate_call_reserve_micros`, `premium_quota.py:65`); crawl micros added afterward aren't reserved, but `credit_finalize` debits actuals regardless and may push the balance slightly negative — same allow-exceed posture as ETL. Pre-blocking applies only to the indexer batch path (§2).
- **Accounting caveat.** Chat-scrape crawl spend lands on the turn's `usage_type="chat"` `TokenUsage` row (inside `cost_micros` / the `web_crawl` line of `model_breakdown`), **not** a `usage_type="web_crawl"` row. "Total crawl spend" analytics must sum the indexer's `web_crawl` rows **plus** the `web_crawl` call-kind inside chat rows.
### 4. What counts (from `03a`)
One unit per `CrawlOutcomeStatus.SUCCESS` — a URL that yielded usable extracted content — **regardless of internal fallback tiers** and **regardless of downstream KB dedupe** (unchanged/duplicate still crawled successfully). `EMPTY`/`FAILED` are free.
## Config / env changes
- `config/__init__.py` (next to `ETL_CREDIT_BILLING_ENABLED`/`MICROS_PER_PAGE`, `:649–655`):
- `WEB_CRAWL_CREDIT_BILLING_ENABLED = os.getenv(..., "FALSE").upper() == "TRUE"`
- `WEB_CRAWL_MICROS_PER_SUCCESS = int(os.getenv("WEB_CRAWL_MICROS_PER_SUCCESS", "1000"))` — **the single source of truth for crawl price.** No literal price may appear in the service or call sites; every charge/estimate reads this. Default `1000` = $1/1000 crawls.
- **Retuning formula (document in both `config` comment + `.env.example`):** `WEB_CRAWL_MICROS_PER_SUCCESS = round(USD_per_crawl * 1_000_000)`, equivalently `round(USD_per_1000_crawls * 1_000)`. Examples: $1/1000 → `1000`; $2/1000 → `2000`; $0.50/1000 → `500`; $5/1000 → `5000`. Change the env value and restart — no code/migration.
- `.env.example`: document both (commented), noting hosted = TRUE, self-hosted = FALSE, and the conversion formula above.
## Work items
1. **`WebCrawlCreditService`** mirroring `EtlCreditService` (gate + `check_credits` + `charge_credits` + static `successes_to_micros`/`billing_enabled`).
2. **Config knobs** + `.env.example` docs.
3. **Indexer wiring**: owner resolution, pre-flight `check_credits`, then at end-of-run (after `:432`) `record_token_usage(usage_type="web_crawl")` (add) **followed by** `charge_credits(crawls_succeeded)` (commits both); skip when `crawls_succeeded == 0`.
4. **Chat scrape wiring**: in both `scrape_webpage` tools, on SUCCESS fold `successes_to_micros(1)` into the turn accumulator (§3).
5. **Tests**: billing-disabled → both surfaces no-op (self-hosted); indexer enabled + sufficient → debits `successes * 1000` + one `web_crawl` `TokenUsage`; indexer enabled + insufficient → task fails pre-crawl, no debit; owner (not trigger user) billed; `EMPTY`/`FAILED` free; chat scrape on a **premium** turn → `acc.total_cost_micros` rises by `1000` per success and is debited at finalize; chat scrape on a **free/anonymous** turn → recorded in the accumulator but **not** debited.
## Risks / trade-offs
- **Celery retry re-billing.** A task retry re-crawls only the URLs not already `ready`/`pending`/`processing` (the 2-phase indexer skips those, `webcrawler_indexer.py:196–219,341–347`), so a retry charges only re-crawled successes. Acceptable for MVP; Phase 5's `PipelineRun` can persist `charged_micros` for stronger idempotency.
- **Same URL across connectors.** Each connector that crawls it performs a real request → each successful crawl bills, even though the 2nd is KB-deduped. This is intended per the `03a` billing-policy note.
- **Session coupling.** `charge_credits` commits the indexer session; placing it after the final doc commit avoids entangling a debit with mid-run document state.
- **Gating asymmetry (intended).** The **indexer** path bills whenever `WEB_CRAWL_CREDIT_BILLING_ENABLED` is on and the owner has credit — no "premium tier" requirement (matches ETL, which gates only on its own flag). The **chat** path additionally requires the turn to be premium, because it piggybacks the turn's reserve/finalize machinery. Same price, two different gate sets — call this out in the credit-status UI copy (frontend phase).
- **No cross-surface double charge.** A chat scrape calls `crawl_url` directly and never enters `webcrawler_indexer`, so a given crawl is billed by exactly one surface.
- **Proxy cost.** Absorbed into the flat $1/1000 (margin), **not** metered separately (see `03b`).
- **Captcha cost.** **Metered separately** as a per-attempt unit (`03d §4`, decided: option (a)) because the solver charges per attempt regardless of crawl success — it cannot ride the per-success flat rate. The `captcha_solves_to_micros` seam above is where it attaches.
## Resolved decisions (this pass)
- **Bill ad-hoc chat scrapes? → YES.** Chat scrape crawls are metered too, but folded into the chat turn's existing bill via the turn accumulator (§3) rather than a separate wallet debit.
- **Dedicated `WebCrawlCreditService` vs generalized meter? → dedicated**, with `billing_enabled`/`successes_to_micros` exposed as statics so the chat surface shares the price/flag. Generalize later only if a third per-unit biller appears.
- **Pre-flight strictness → pre-block** the indexer batch run on insufficient credit (`len(urls)` is a safe upper bound). Chat scrapes are governed by the turn's own reservation, not a per-scrape block.
## Out of scope (hand-offs)
- Per-**pipeline** run accounting / `charged_micros` persistence for idempotency → Phases 5–7. Specifically, **Phase 6 moves pipeline-run billing out of the indexer to the run level**: it adds a `bill: bool = True` param to this indexer wiring and calls `index_crawled_urls(bill=False)` from the pipeline engine, which then does its own `check_credits` + `charge_credits(crawls_succeeded)` and stamps `PipelineRun.charged_micros` (idempotency). So this phase's in-indexer billing keeps serving the **connector `/index` + periodic** paths; the "pipeline crawls" half of the objective is refined by `06`. Structure the §2 wiring so it's easy to gate behind that flag.
- Captcha-solver upstream cost pass-through → `03d` (now active; separate per-attempt `web_crawl_captcha` unit via the seam above).
- Surfacing crawl spend in the credit-status UI → frontend umbrella.

111
plans/backend/03d-captcha-solving.md
View file

@ -1,111 +0,0 @@
# Phase 3d — Captcha solving (reCAPTCHA / hCaptcha / image)
> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
> **Status: ✅ IMPLEMENTED (`ci_mvp`, captcha solving + per-attempt billing).** `captchatools==1.5.0` (MIT, pure-Python, single `requests` dep) wired as the StealthyFetcher-tier `page_action`. Generic config in Apache-2 `app/utils/captcha/` (`CaptchaConfig` + `captcha_enabled()` = flag **and** key); bypass logic (detect/harvest/inject + process latch) in proprietary `app/proprietary/web_crawler/captcha.py`. `CrawlOutcome` gained `captcha_attempts`/`captcha_solved`; the stealth tier captures the proxy **once** and reuses it for the solver (IP-coherence). Billing: `WebCrawlCreditService.captcha_solves_to_micros` + `charge_captcha` + generic `check_balance`; indexer accumulates attempts (even on failed crawls), combined crawl+captcha pre-flight, per-attempt owner charge with `usage_type="web_crawl_captcha"`; both chat scrape tools fold attempts into the turn before the success/fail branch. **No migration.** Off by default (zero attempts, zero cost). 40+ new unit tests pass.
> **Implementation deltas from the original draft (the draft predated the real 03a code):**
> - **There is no `FetchStrategy` class/seam** in the shipped 03a crawler — tiers are plain methods (`_crawl_with_async_fetcher`/`_crawl_with_dynamic`/`_crawl_with_stealthy`) that call `get_proxy_url()` inline. The "surface the chosen endpoint" seam is implemented concretely by capturing `proxy = get_proxy_url()` **once** in `_crawl_with_stealthy_sync` and passing it to both `StealthyFetcher.fetch(proxy=…)` and the factory (no re-rotation).
> - **Attempts reach `CrawlOutcome` via a per-call `captcha_state` dict** threaded `crawl_url → _crawl_with_stealthy(_sync)` and stamped onto every stealth terminal outcome (success/empty/failed). Per-call (not on `self`) → concurrency-safe.
> - **Solver timeout uses a non-context-managed `ThreadPoolExecutor`** + `shutdown(wait=False, cancel_futures=True)`; a `with` block would join the worker and defeat the timeout.
> - **Combined pre-flight** lives on the indexer using the new generic `check_balance(required_micros)` (crawl successes + worst-case `len(urls) × MAX_ATTEMPTS` captcha), not a captcha-specific check method. The captcha worst-case is only reserved when **both** `captcha_billing_enabled()` **and** `captcha_enabled()` (solving actually on) — with solving off, attempts can never happen, so reserving would wrongly block a run for captcha that will never run (regression-tested).
> **Decision recap (this pass):** Cloudflare stays in-framework (free, `03a`); reCAPTCHA/hCaptcha/image use a **paid** solver via `captchatools`, **opt-in + off by default**, and are billed as a **separate per-attempt unit** (option (a) below). Logged-in/account-gated captcha flows are **out of scope** (no authenticated scraping in this MVP — see umbrella Phase 8 / `03b` static-proxy hand-off).
## Why this is the last tier, not the first
A solve is the **most expensive and least reliable** move available (paid per attempt, 10–60 s latency, <100% success, ToS-sensitive). So the crawler must exhaust the cheaper, in-house levers first; captcha solving only fires when everything else has already failed:
1. `03e` humanization + fingerprint coherence (geoip locale/tz, persistent profile, headed/Xvfb) → **avoid** triggering a challenge at all.
2. `03a` `solve_cloudflare=True` on StealthyFetcher → free Cloudflare Turnstile/Interstitial.
3. **This subplan** → only the residual reCAPTCHA v2/v3 + hCaptcha + image challenges, and only when `CAPTCHA_SOLVING_ENABLED`.
## Scope boundary
| Challenge | Handled by | Where |
| --- | --- | --- |
| Cloudflare Turnstile / Interstitial | Scrapling `solve_cloudflare=True` | `03a`, StealthyFetcher tier |
| reCAPTCHA v2/v3, hCaptcha, image | **`captchatools`** (this subplan) | StealthyFetcher `page_action` |
| DataDome / Kasada / reCAPTCHA-**Enterprise** | **none in the free stack** | deferred paid-unblocker tier (`03e`) |
## Grounding (libraries verified)
- **`captchatools` is itself the provider registry.** `new_harvester(api_key, solving_site, sitekey, captcha_url, captcha_type="v2"|"v3"|"hcaptcha"|"image", invisible_captcha=…, min_score=…, action=…)` → `.get_token(proxy=…, proxy_type=…, user_agent=…, b64_img=…)` returns a **token string** (`references/Captcha-Tools/README.md:28–47`). `solving_site` ∈ {`capmonster`,`2captcha`,`anticaptcha`,`capsolver`,`captchaai`} (`:113–127`). Errors: `ErrNoBalance`, `ErrWrongAPIKey`, `ErrWrongSitekey`, `ErrIncorrectCapType`, `ErrNoHarvester` via `captchatools.exceptions` (`:136–155`). **Implication:** we do **not** rebuild a multi-provider class hierarchy (unlike `03b`'s proxy registry) — `captchatools` already dispatches across the 5 services. Our layer is thin: config resolution + page detection/injection glue + billing.
- **`captchatools` only harvests a token; it does not inject it.** The caller must drop the token into the page (`g-recaptcha-response` / `h-captcha-response` textarea, or invoke the JS callback) and submit.
- **Injection requires a live browser page**, so this only works on the **StealthyFetcher** tier. Scrapling exposes `page_action`: "a function that takes the `page` object, runs after navigation, and does the automation you need" (`references/Scrapling/scrapling/fetchers/stealth_chrome.py:30,82`; engine docstrings `_browsers/_stealth.py:50,191,326,466`). The HTTP (`AsyncFetcher`) and `DynamicFetcher` tiers cannot solve interactive captchas — a captcha hit there must **escalate to StealthyFetcher** (`FetchStrategy` seam from `03a`; the ladder already ends there).
- **Ordering is already correct in Scrapling.** `solve_cloudflare` runs **before** `page_action` (sync `_stealth.py:253–254` then `:258–260`; async `:529–530` then `:534–536`). So Cloudflare is cleared first, *then* our injector runs — exactly the tier order we want. `page_action` exceptions are caught + logged by Scrapling (`:262` / `:538`), so our factory must also return a clear signal (it can't rely on raising to abort the fetch).
## Target design
### 1. Config layer (`app/utils/captcha/`, mirroring `app/utils/proxy/` from `03b`)
- `CaptchaConfig = (enabled, solving_site, api_key, captcha_type_default, min_score, max_attempts, timeout_s)`, resolved from **env only** — one app-wide config, mirroring `03b`'s env-only single-provider model (`get_active_provider()`); **no per-connector config** (that path was dropped in `03b`).
- Env knobs (next to the proxy + crawl-billing knobs in `config/__init__.py`):
- `CAPTCHA_SOLVING_ENABLED` (default `FALSE`) — off ⇒ zero solve attempts and zero solver cost.
- `CAPTCHA_SOLVER_PROVIDER` (e.g. `capsolver`), `CAPTCHA_SOLVER_API_KEY`.
- `CAPTCHA_MAX_ATTEMPTS_PER_URL` (default `1`), `CAPTCHA_SOLVE_TIMEOUT_S` (default `120`).
- A `captcha_enabled()` static (mirrors `WebCrawlCreditService.billing_enabled()`) so callers gate cheaply before constructing anything.
### 2. Detection + injection `page_action` factory
Builds the **sync** callable passed to `StealthyFetcher.fetch(..., page_action=…)` (03a runs StealthyFetcher via `asyncio.to_thread`, so the sync engine path `_stealth.py:258–262` applies — the factory returns a plain `def(page): ...`, not a coroutine):
1. **Detect** the challenge + sitekey in the DOM:
- reCAPTCHA v2: `.g-recaptcha[data-sitekey]` (or iframe `src*="recaptcha"`).
- hCaptcha: `.h-captcha[data-sitekey]` (or iframe `src*="hcaptcha"`).
- reCAPTCHA v3: presence of `grecaptcha.execute` + a known `action` (config/site-mapped).
- If no sitekey found → **no-op return** (nothing to solve; let `wait_selector`/extraction proceed).
2. **Bind the proxy/UA** (the IP-coherence caveat below): harvest with the **exact endpoint used for this crawl tier** —
`new_harvester(solving_site, api_key, sitekey, captcha_url=page.url, captcha_type=…, min_score=…, action=…).get_token(proxy=<this crawl's endpoint, reformatted ip:port:user:pass>, proxy_type="HTTP", user_agent=<page UA>)` (`README.md:44–46,107–110`).
3. **Inject + submit**: set the response token into `g-recaptcha-response` / `h-captcha-response` (make textarea visible if needed), dispatch `input`/`change`, invoke the JS callback when present (`grecaptcha.getResponse` flows / `___grecaptcha_cfg` callbacks), then submit the form and `page.wait_for_load_state`. For v3 (no widget) the token is consumed by the site's own `execute()` flow.
4. **Signal outcome** via a mutable closure cell (since Scrapling swallows `page_action` exceptions at `:262` **and discards its return value** at `:260`/`:536`): record `solved: bool` + `attempts: int` into a captured `dict` the crawler reads after `fetch()` returns, so billing (§4) and the retry cap can act on it. **This `page_action`+closure-cell helper is shared with `03f`'s test harness** (which uses the same mechanism to read JS-object verdicts like CreepJS `window.Fingerprint`) — factor it once.
### 3. Crawler escalation (uses the `03a` `FetchStrategy` seam)
- Only the **StealthyFetcher** strategy attempts solving. A captcha detected on a lower tier (HTTP/DynamicFetcher) returns non-`SUCCESS`, and the ladder already escalates to StealthyFetcher (`03a`), which this time is constructed with the captcha `page_action` when `captcha_enabled()`.
- **Attempt cap:** at most `CAPTCHA_MAX_ATTEMPTS_PER_URL` solves per URL so one hostile page can't burn unbounded solver credit. The closure cell's `attempts` enforces it.
- No new caller *contract*: the strategy still returns `CrawlOutcome` (`03a` invariant). Captcha is a *parameterization* of the last tier, not a new return shape.
- **But billing needs the attempt count to escape the closure cell.** §4 charges per attempt on **both** the indexer and chat paths, so the count can't stay buried inside `page_action`'s closure — the crawler must copy it onto the outcome. Add **`captcha_attempts: int = 0`** (and `captcha_solved: bool = False`) as **dataclass fields on `CrawlOutcome`** (`03a`). This is safe: `03a`'s "don't widen the return" rule is about the *indexer's* positional `(total_processed, error)` tuple unpacked by length, **not** about `crawl_url`'s dataclass — adding fields to a dataclass breaks no consumer. The indexer sums `outcome.captcha_attempts`; the chat tool reads it off the same outcome.
### 4. Billing — separate per-attempt unit (option (a), DECIDED)
`03c` bills per **successful crawl** (`CrawlOutcomeStatus.SUCCESS`) and absorbs proxy cost into the flat $1/1000. **Captcha can't ride that model**: the solver charges **per attempt regardless of whether the crawl ultimately succeeds**, so a failed solve = real upstream cost with no billable crawl success. We therefore meter solves **independently**:
- New static on `WebCrawlCreditService` (defined in `03c`, knobs added here): `captcha_solves_to_micros(n) = n * config.WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE`.
- Each **attempt** (not each success) is charged when `CAPTCHA_SOLVING_ENABLED` **and** `WEB_CRAWL_CAPTCHA_BILLING_ENABLED`:
- **Indexer/pipeline path:** after the crawl loop, debit `captcha_solves_to_micros(total_attempts)` — where `total_attempts = Σ outcome.captcha_attempts` over the batch — from the **workspace owner** (same owner resolution as `03c §2`) and `record_token_usage(usage_type="web_crawl_captcha", …, cost_micros=…, call_details={"attempts": n, "solved": k})`. Added before the charge commit, same one-transaction pattern as `03c`.
- **Chat-scrape path:** fold `captcha_solves_to_micros(outcome.captcha_attempts)` into the turn accumulator with `call_kind="web_crawl_captcha"` (same mechanism as `03c §3`), so it settles with the premium turn; non-premium/anonymous turns record-but-don't-debit.
- **No pre-block on solves** (unlike the crawl batch): attempts are bounded by `CAPTCHA_MAX_ATTEMPTS_PER_URL × len(urls)`, an upper bound the indexer can optionally pre-check against the wallet if we want symmetry with `03c §2` (recommended: pre-check the combined crawl + worst-case captcha estimate so a run can't strand mid-batch on solver insolvency).
- New env (next to `03c`'s knobs): `WEB_CRAWL_CAPTCHA_BILLING_ENABLED` (default `FALSE`), `WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE` (default e.g. `3000` = $3/1000 attempts; type-dependent, set with margin over the solver's per-attempt price). **No DB migration** — `web_crawl_captcha` is just another free-form `TokenUsage.usage_type` value (`db.py` `usage_type String(50)`).
### 5. Dependency + Docker
- Add `captchatools` to `pyproject.toml` (build-time only). It's a thin HTTP client for the solver services — no browser binaries.
- No `Dockerfile` change beyond the dependency (browser already installed by `03a`/`03e`).
## Error handling (must not loop or silently burn credit)
- `ErrNoBalance` / `ErrWrongAPIKey` (`README.md:134,136–143`) → **stop solving for the rest of the run** (set a process/run flag), surface a clear "captcha solver out of balance / misconfigured" message via `log_task_failure`, and fall through to a non-`SUCCESS` `CrawlOutcome`. The README explicitly warns balance-exhausted loops can get the **IP temporarily banned** (`:132–134`) — never retry on `ErrNoBalance`.
- `ErrWrongSitekey` / `ErrIncorrectCapType` → log + skip this URL's solve (likely detection bug, not transient); count the attempt for billing only if the solver actually charged.
- Timeout (`CAPTCHA_SOLVE_TIMEOUT_S`) → abort the solve, count one attempt, return non-`SUCCESS`.
## Risks / considerations
- **Solver-tier only.** No captcha solving on the HTTP/DynamicFetcher tiers; must escalate to StealthyFetcher (slower, browser-backed).
- **Latency & flakiness.** Solves take 10–60 s and aren't guaranteed; the `CAPTCHA_MAX_ATTEMPTS_PER_URL` + `CAPTCHA_SOLVE_TIMEOUT_S` caps keep a single URL from burning unbounded solver credit.
- **Proxy coherence (rotating-pool caveat).** Many captchas IP-bind the token, so the solver must egress from the **same IP** as the crawl. Under `03b`'s single app-wide provider this is automatic for single-endpoint providers (`anonymous_proxies`, single-URL `custom`). But a **pool-backed `CustomProxyProvider`** returns the *next* endpoint on each zero-arg `get_proxy_url()` call — so the `page_action` must **reuse the endpoint actually used for this crawl tier** (the crawler captures it once and passes it into the factory), NOT call `get_proxy_url()` again (which would rotate to a different IP). This is the same "surface the chosen endpoint" seam `03e`/`03d` both rely on — build it once in `03a`'s strategy context.
- **Enterprise captchas are out of reach.** reCAPTCHA **Enterprise**, DataDome, and Kasada use behavioral + device signals the token-injection model doesn't satisfy reliably; those route to the deferred paid-unblocker tier (`03e`), not here.
- **Policy / ToS.** Automated captcha solving may violate a target site's terms; gate behind the explicit `CAPTCHA_SOLVING_ENABLED` flag and treat as an opt-in, owner-acknowledged capability. Public data only (no logged-in bypass).
## Work items
1. **`app/utils/captcha/`**: `CaptchaConfig` + env resolution + `captcha_enabled()` static.
2. **`page_action` factory**: detection (v2/v3/hcaptcha/image sitekey), harvest via `captchatools` with bound proxy/UA, inject+submit, outcome closure cell.
3. **Crawler wiring**: construct StealthyFetcher with the factory when `captcha_enabled()`; enforce per-URL attempt cap; surface the crawl's chosen proxy endpoint into the factory.
4. **Billing**: `WebCrawlCreditService.captcha_solves_to_micros` + the two knobs; debit per-attempt on the indexer path (owner) and fold into the turn on the chat path; `record_token_usage(usage_type="web_crawl_captcha")`.
5. **Config + docs**: all env knobs in `Config` + `.env.example` (commented, hosted=TRUE / self-hosted=FALSE), plus the `captchatools` dependency.
6. **Tests**: solving disabled → zero attempts, zero solver cost (self-hosted); v2 detected → harvest+inject+submit path invoked with the **crawl's** proxy endpoint (not a re-rotated one); `ErrNoBalance` → stops solving + no retry loop; attempt cap honored; billing enabled → per-attempt debit even when the crawl ultimately fails; chat path folds captcha micros into a premium turn and record-only on anonymous; `web_crawl_captcha` `TokenUsage` rows written.
## Out of scope
- Anything owned by `03a`/`03b`/`03c`/`03e`.
- Cloudflare Turnstile (already handled by Scrapling in `03a`).
- Logged-in / account-gated captcha flows (no authenticated scraping this MVP → umbrella Phase 8 + `03b` static-proxy hand-off).
- Enterprise/behavioral anti-bot (DataDome/Kasada/reCAPTCHA-Enterprise) → deferred paid-unblocker tier (`03e`).

167
plans/backend/03e-stealth-hardening.md
View file

@ -1,167 +0,0 @@
# Phase 3e — Stealth hardening (the in-house "undetectable" moat)
> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
> Depends on `03a` (Scrapling StealthyFetcher tier + `CrawlOutcome`) and `03b` (app-wide proxy provider). Precedes `03d` (captcha) in the escalation order — hardening **avoids** challenges; captcha solving is the paid last resort for the ones we can't avoid. Touches `03c` only via the deferred paid-unblocker tier (its own billing, decided if/when built).
> **Architecture reconciliation (read before building).** Earlier drafts of this plan referenced a `FetchStrategy` seam / strategy-object contract `(url, ctx) -> CrawlOutcome`. **That seam does not exist in shipped code.** `03a`–`03d` shipped a *hardcoded 3-tier ladder* inside `WebCrawlerConnector.crawl_url` (`app/proprietary/web_crawler/connector.py`): plain methods `_crawl_with_async_fetcher` / `_crawl_with_dynamic` / `_crawl_with_stealthy(_sync)` wrapped by `_run_tier_with_proxy_retry`, every tier returning the shared `CrawlOutcome`. `03e` therefore implements levers **not** as strategy classes but as: (1) a **centralized per-tier kwargs builder** (`app/proprietary/web_crawler/stealth.py` — **proprietary**, since it's bypass-specific tuning; the generic block classifier stays Apache-2 in `app/utils/crawl/`) that turns config → `StealthyFetcher`/`AsyncFetcher` kwargs, imported by both the connector and `03f`'s harness so there's no test-vs-prod drift; (2) small **helpers around the existing ladder** (block classifier, strategy memory); and (3) the deferred paid tier as a config-gated **extra ladder step**, not a `FetchStrategy`. The captcha precedent holds: generic config/plumbing → `app/utils/`; actual bypass logic (WebGL spoof JS, humanize choreography) → `app/proprietary/web_crawler/`.
> **Implementation note.** Same convention as `03a`–`03d`: Phases 1–2 are **SHIPPED** (live code says `workspace_id`/`Workspace` — substitute for any old `search_space_*`/`SearchSpace` citations and grep the new name); citations also predate `03a`'s `crawl_url` refactor; locate code by **symbol/grep**, not absolute lines. Scrapling references point at the on-disk (gitignored) `references/Scrapling/` checkout pinned to `scrapling[fetchers]>=0.4.9` (`pyproject.toml:91`).
> **Build status.** Sliced for risk (see "Build slicing" below). **Slice A (SHIPPED in this pass):** centralized stealth kwargs builder + fingerprint flags + geoip-from-proxy-location + block classifier (additive `CrawlOutcome.block_type`) + fonts/Xvfb packages in the image. **Slice B / C (deferred):** headed-Xvfb runtime path, humanization/WebGL `init_script` assets, persistent profiles, strategy memory, paid-unblocker stub. Effectiveness of every lever is graded in `03f` (manual), not unit-tested here.
>
> **First baseline evidence (`03f`, 2026-06-30 — headless, rotating residential, captcha OFF, Slice-A only with `CRAWL_GEOIP_MATCH_ENABLED=false`).** Suite S = **6 PASS / 4 FAIL** (full table in `03f`). The 4 fails map cleanly onto this plan's deferred levers and confirm the predicted ceiling — they are **not** Slice-A regressions:
> - **iphey "Unreliable"** → the predicted geoip-incoherence tell (browser tz/locale vs proxy exit geo). It's now the live regression test for **§1 geoip coherence** — flipping `CRAWL_GEOIP_MATCH_ENABLED` is the expected fix to validate next.
> - **CreepJS `hasHeadlessWorkerUA: true`** (headless 33%) → the Web **Worker** `navigator.userAgent` still leaks `HeadlessChrome` (main-thread UA is clean). A concrete, plausibly-fixable **Slice-B** target (worker UA override via `init_script`), beyond today's Slice-A flags.
> - **FingerprintJS Pro "tampering detected, access denied"** → the documented **commercial/device-fingerprint wall** (§2c WebGL/GPU gap). Needs Slice B/C or the §8 paid tier; expected to stay red on the free stack.
> - **incolumitas** → a single legacy `fpscanner WEBDRIVER` check (all modern checks pass); lowest priority. Its IP classifier also flip-flopped `is_datacenter` true/false across rotations, exposing that the `anonymous_proxies` pool mixes datacenter+residential exits (a proxy-provider variable, not a code issue).
## Objective
Push the Universal WebURL Crawler as far toward "undetectable" as the **free / in-house stack** allows, so we hold a scraping moat for the next ~4–6 months **without** a third-party unblocker (ZenRows/ScrapFly/Bright Data) or a source-patched browser (CloakBrowser — rejected on licensing). Everything here is **runtime/config-level** stealth layered on Scrapling's patchright-Chromium (`03a` engine note): geoip fingerprint coherence, persistent profiles, headed execution, real fonts, and behavioral humanization — plus a **block classifier** + **per-domain strategy memory** so the ladder learns and the cheap tiers get skipped once a domain's working strategy is known. The paid-unblocker tier is defined here as a **deferred config-gated ladder step** (the explicit escape hatch for when in-house maintenance gets too costly), not built now.
## The realistic ceiling (be honest with downstream devs)
Scrapling already ships strong **runtime** stealth by default: `DEFAULT_ARGS + STEALTH_ARGS` (incl. `--disable-blink-features=AutomationControlled`), a persistent context, and `navigator.webdriver` masking via **patchright**. Be precise about what patchright actually is: it patches the **Playwright driver** to remove *automation leaks* — it never calls `Runtime.enable`, disables `Console.enable`, and reaches closed shadow roots (verified: [patchright-python](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright-python), [Scrappey](https://scrappey.com/qa/web-scraping-apis/what-is-patchright)). It hides **"I'm automated," not "this is a different machine."** It does **not** spoof canvas, WebGL, audio, screen, or GPU. With `03b` residential proxies + this plan's coherence/humanization on top, the crawler reliably handles **Cloudflare** (via `03a` `solve_cloudflare`) and the long tail of **moderate** anti-bot — which is most CI targets.
**The device-fingerprint wall (the WebGL/GPU gap — see §2c).** Our worker has no GPU, so Chromium's WebGL renderer reports `Google SwiftShader` / `Mesa llvmpipe` — a string anti-bot vendors treat as ">95% bot" ([ipasis](https://ipasis.com/blog/webgl-fingerprinting-bot-detection), [Scrappey](https://scrappey.com/qa/anti-bot/what-is-webgl-fingerprinting)). Scrapling exposes only `allow_webgl` (on/off — and *off* is itself a tell), **no renderer spoof**; CloakBrowser fixes this at the C++ level (`--fingerprint-gpu-renderer`) and we cannot. Concretely, **without a real GPU or a source-patched binary we cannot**:
1. **Pass pure device-fingerprint bot checks** — FingerprintJS bot-detection, CreepJS / BrowserScan "is this a real device." CloakBrowser passes these (BrowserScan 4/4); our SwiftShader renderer fails them.
2. **Reach human-level reCAPTCHA v3 *scores*** (~0.7–0.9) **on our own browser** — we land ~0.1–0.5. (Note: `03d`'s **paid solver sidesteps this for token challenges** — the token is minted on the vendor's real-browser infra, so its score isn't ours. The gap only bites where the *site itself* reads our live v3 score.)
3. **Max out the device-fingerprint component** of DataDome / Kasada / Akamai — one of several reasons we plateau below managed APIs.
**Numbers (2026 benchmarks — use these in any user-facing copy).** Free stack **+ residential**: Basic (CF) ~99%; Medium (CF Pro / PerimeterX) ~90%; **Enterprise (DataDome / Kasada / Akamai) ~60–75% initially, decaying toward ~60% within 24h** as their ML clusters the fingerprint (managed APIs hold ~89%) ([scrapewise DataDome 2026](https://scrapewise.ai/blogs/bypass-datadome-web-scraping-2026)). So the honest line is **"partial and decaying, not SLA-grade,"** — neither "can't beat" nor "beats everything." For CI targets the top-tier defenses are the exception, so the in-house moat is a sound 4–6 month bet, with the deferred paid tier (§8) as the **evidence-driven** escape hatch (`03f`'s scorecard is the trigger). **Do not promise "beats everything."**
## Levers (wired via the centralized kwargs builder + ladder helpers — callers stay outcome-only)
### 1. Geoip fingerprint coherence (match the browser to the proxy exit IP)
A residential IP in Berlin behind an `en-US`/`America/New_York` browser is an instant tell. Make the fingerprint cohere with the proxy's exit geo:
- **Primary source (Slice A, no network call): the configured `RESIDENTIAL_PROXY_LOCATION` (`03b`, `config:1048`).** Map that string (best-effort: ISO-3166 alpha-2 like `us`/`de`, or a common country name) → a representative `(locale, timezone_id)`. Coarse country granularity is fine (per "Risks": wrong-but-coherent beats default-mismatched); unknown/empty → skip (leave Scrapling's system default). **No exit-IP resolution** — avoids a per-crawl network round-trip + failure mode. (A `geoip2`/MaxmindLite exit-IP path is a later refinement, not MVP.)
- Pass the matched values into StealthyFetcher via the kwargs builder: `locale=` (drives `navigator.language` + `Accept-Language`) and `timezone_id=` (confirmed real kwargs on `StealthyFetcher.fetch`). These flow into the Playwright context.
- The stealthy tier **already captures the chosen proxy endpoint once** (`_crawl_with_stealthy_sync`: `proxy = get_proxy_url()`, the same value handed to `03d`'s solver for IP-coherence) — reuse that captured `proxy`/location, **don't** re-call `get_proxy_url()` (which rotates on a pool-backed provider, `03b`).
- **Caveat (be honest):** Scrapling applies `locale`/`timezone_id` via Playwright **CDP emulation**, which advanced systems can flag *as* emulation — CloakBrowser sets these as *binary flags* precisely to avoid "detectable CDP emulation" (`references/CloakBrowser/README.md:309`). Still strictly better than a mismatched default (wrong-but-coherent beats `UTC`+`en-US` behind a Berlin IP), but it's a weaker variant than a source-patched browser, not parity.
### 2. Fingerprint flags Scrapling already exposes (turn them on)
- `hide_canvas` — Scrapling adds **random** noise to canvas ops (verified in the installed `StealthyFetcher.fetch` docstring). **Not "free safe":** random-per-call noise is itself a signal on FingerprintJS/CreepJS-class detectors — an *unstable* canvas hash is as loud as `navigator.webdriver=true` ([dev.to](https://dev.to/tanwydd/how-your-canvas-fingerprint-gets-you-caught-and-why-random-noise-makes-it-worse-3ba1)); CloakBrowser's own FPJS fix is `--fingerprint-noise=false`. **Default OFF; gate per-domain and validate in `03f`.** Harmless against Cloudflare/moderate, potentially harmful against device-fingerprint graders. (Canvas stealth that actually *helps* must be **deterministic per persistent profile**, not random — out of scope here.)
- `block_webrtc=True` — forces WebRTC to respect the proxy, preventing the **real local IP leak** that unmasks proxied browsers (`:41`). **This one is genuinely cheap + safe — keep default TRUE.**
- `dns_over_https` — Scrapling can route DNS via Cloudflare DoH to stop the **DNS leak** that unmasks proxied browsers (same anti-leak class as `block_webrtc`; confirmed kwarg on `StealthyFetcher.fetch`). **Trade-off: it adds a DNS round-trip, a (small) latency cost — so per the "no speed regression" constraint it ships *off by default* (`CRAWL_DNS_OVER_HTTPS`, default FALSE), pre-wired for operators who prefer leak-safety over the marginal latency.** (Idea sourced from the Camoufox-based FlareSolverr alternatives `references/Byparr-main` / `references/trawl-dev`, which treat DNS/WebRTC leak coherence as first-class.)
- `google_search=True` (default) — sets a Google referer so the first hit looks like organic arrival (`:45`); override per-need via `extra_headers` (`:46`).
- `additional_args=` — last-priority Playwright context overrides for anything not surfaced as a first-class param (`:51`).
### 2b. HTTP-tier TLS fingerprint (the AsyncFetcher tier — `impersonate`)
The cheap static tier (`03a` tier 1) is the **first** thing every crawl hits. `03a` **already ships `impersonate="chrome"`** on it (`app/proprietary/web_crawler/connector.py`, the `AsyncFetcher.get` call) — Scrapling's static engine selects a matching curl_cffi browser profile (`references/Scrapling/scrapling/engines/static.py:36–47`), so the HTTP tier's **JA3/JA4/HTTP-2** already matches a real Chrome and coheres with the browser tiers' UA. `03e`'s remaining work here is **not** to add it, but to (a) fold the `impersonate` profile into the centralized per-tier kwargs builder (below) so it stays the single source of truth, and (b) keep the chosen profile coherent with the proxy exit / UA. `03f §S3` validates the shipped parity against `tls.peet.ws`.
### 2c. WebGL / GPU renderer (the biggest free-stack gap — partial mitigation only)
This is the "device-fingerprint wall" from the ceiling section. On a GPU-less worker the WebGL renderer string is `Google SwiftShader` / `ANGLE (Mesa, llvmpipe…)` — an instant tell on any detector that reads `WEBGL_debug_renderer_info`. Scrapling gives us **no renderer knob** (only `allow_webgl` on/off, and *off* is a tell). Three options, increasing strength + cost:
1. **DIY JS spoof (optional, OFF by default — Slice B).** An `init_script` overrides `getParameter` for `UNMASKED_VENDOR_WEBGL` / `UNMASKED_RENDERER_WEBGL` to a believable, **platform-coherent** value (e.g. an Intel/ANGLE renderer that matches the spoofed Windows UA), held **consistent per persistent profile** (§3). **Impl note:** Scrapling's `init_script` is **an absolute path to a `.js` file**, *not* an inline string (confirmed in `StealthyFetcher.fetch` docstring) — so this ships as a **bundled `.js` asset** in `app/proprietary/web_crawler/` (next to `03d`'s captcha logic) passed by absolute path; the kwargs builder resolves the path when `CRAWL_WEBGL_SPOOF_ENABLED`. **Must** also patch `Function.prototype.toString` so the override still returns `[native code]` — otherwise *the spoof itself is the signal* (exactly the `toString()` probe Kasada catalogs, [web-scraping-guide](https://web-scraping-guide.com/)). This defeats **string-based** WebGL checks (the common case) but **not** render-output pixel-hash or GPU-perf probes (FingerprintJS-grade) — a software rasterizer's actual draw output still differs and can't be faked in JS. It is the brittle config-level approach patchright/CloakBrowser explicitly warn rots; treat it as a **cheap, test-gated win, not the moat**.
2. **Real GPU passthrough (hosted workers only).** `--device /dev/dri` + ANGLE/EGL so WebGL reports a *genuine* renderer with *real* render output, defeating even output/perf probes. Best technical fix; costs GPU instances + image complexity ([livekit/egress GPU notes](https://stackoverflow.com/questions/78985698)). A hosted-tier escalation, off for self-hosted.
3. **Defer to the paid unblocker (§8).** Route the residual device-fingerprint-gated domains (flagged by the §7 classifier as in-house-unreachable) to the external unblocker. The pragmatic moat-bounding fallback.
**Decision for this MVP:** ship option 1 as an opt-in flag (`CRAWL_WEBGL_SPOOF_ENABLED`, default FALSE), name option 2 as the hosted escalation, and lean on §8 for the rest. Quantify all three in `03f`.
### 3. Persistent per-domain profiles (look like a returning human)
Scrapling defaults to a **temporary** user-data dir (fresh = suspicious). Use `user_data_dir=` (confirmed real kwarg; `launch_persistent_context`) to keep a **persistent profile per domain** (or per domain+proxy-geo), so cookies/localStorage/site-trust carry across crawls and the browser presents as a returning visitor rather than a brand-new incognito session. Store profiles under a configured dir (`shared_tmp`-style volume so API + worker share them).
- **Concurrency hazard (Slice C design item — why this is deferred, not a one-liner):** Chromium **hard-locks** a `user_data_dir`; two crawls touching the same profile dir at once crash (`SingletonLock`). Our crawler runs concurrently (multiple connectors/chats across workers), so a naive per-domain dir collides. The fix needs one of: (a) an async lock keyed by profile dir so same-domain crawls serialize (adds latency, can deadlock under load), (b) per-`(domain, worker, pid)` dirs + a periodic prune (loses some cross-run trust, no contention), or (c) persistent only when the strategy memory (§7) says a domain *needs* it, temp otherwise. Decide in Slice C alongside §7; **Slice A keeps Scrapling's temp default.**
### 4. Headed execution under Xvfb (defeat headless tells)
`headless` defaults to hidden (`stealth_chrome.py:19/71`). Many WAFs flag headless Chromium. Run **headful** (`headless=False`) inside a virtual framebuffer (**Xvfb**) in the Docker worker so the browser is "visible" to itself but needs no real display. Gate behind a config flag (off by default for self-hosted, on for hosted workers that have Xvfb installed).
### 5. Real fonts (canvas/emoji hash realism)
A minimal container has almost no fonts, making canvas/emoji fingerprint hashes obviously synthetic. Install real font packages so font-enumeration + canvas/emoji hashes resemble a real desktop. Use the set CloakBrowser ships as proven against Kasada/Akamai emoji-canvas hashing (`README.md:737–739`): `fonts-noto-color-emoji`, `fonts-unifont`, `fonts-ipafont-gothic` (CJK), `fonts-wqy-zenhei` (CJK), `fonts-tlwg-loma-otf` — plus `fonts-dejavu`/`fonts-liberation` for Latin coverage. `Dockerfile` change only (`apt-get install fonts-*`).
### 6. Behavioral humanization (DIY — Scrapling has no `humanize` for the Chromium engine)
Unlike Camoufox, the patchright-Chromium StealthyFetcher exposes **no built-in `humanize`** (verified: zero matches in the StealthyFetcher param set). So humanization is custom, injected via the existing hooks:
- `page_action=` (runs after navigation) — randomized mouse moves/curves, scrolls, hover-before-click, and small think-time delays before extraction. This is the **same hook `03d` uses** for token injection — `_crawl_with_stealthy_sync` currently builds exactly **one** `page_action` (the captcha injector), so Slice B must **compose** two callables (humanize → optional captcha solve) into the single `page_action` kwarg, not overwrite it.
- `init_script=` (an **absolute path to a `.js` file** executed on page creation — *not* inline JS, see §2c) — early shims for any residual JS tells not covered by patchright (and the §2c WebGL spoof, if enabled).
- Tunable `wait`/`network_idle` (`:29,27`) so dwell time isn't robotically constant.
- **Concrete rules (from CloakBrowser's reCAPTCHA notes, `README.md:1280–1314` — encode these):** use native `time.sleep`, **never** `page.wait_for_timeout()` (it emits CDP traffic detectors flag); prefer `page.type()` over `fill()` for any input (real keystrokes vs. value-set); dwell **15s+** before triggering score-based checks; space repeat hits to score-gated domains **30s+**; keep a **stable identity per domain** (persistent profile §3 + consistent spoofed values §2c) so a score-based system sees a *returning* device, not a fresh one each visit.
### 7. Block classifier + per-domain strategy memory (the "learning ladder")
Two small in-house pieces make the ladder smart instead of brute-force:
- **Block classifier (Slice A — additive, pure, unit-testable).** Inspect the fetched page (`status` + body markers; cookies/headers are not threaded through `_build_result` today, so MVP is **status + body-marker based**) and label the outcome: `OK` / `CLOUDFLARE` / `CAPTCHA_RECAPTCHA` / `CAPTCHA_HCAPTCHA` / `DATADOME` / `KASADA` / `RATE_LIMITED` / `EMPTY` / `UNKNOWN`. **Concrete markers — adopt the set proven in `references/trawl-dev/packages/tiers/src/detect.ts` (a near-identical classifier):** Cloudflare/DDoS-Guard → `cf-mitigated`-style title strings `"just a moment"` / `"checking your browser"` / `"enable javascript and cookies to continue"` / `"verify you are human"`, DOM ids `challenge-running` / `cf-challenge-running` / `turnstile-wrapper`, and `ddos-guard.net`; Turnstile → `class="cf-turnstile"` / `challenges.cloudflare.com/turnstile`; hCaptcha → `class="h-captcha"` / `hcaptcha.com/1/api`; reCAPTCHA → `class="g-recaptcha"` / `google.com/recaptcha` / `recaptcha.net`; DataDome → `datadome` script/cookie; **status `202`/`403`/`429` as a bot-gate** (`202` is an IMDb-style pre-gate, per `detect.ts:isBlocked`). **Invariant: the classifier is purely *additive* — it attaches `CrawlOutcome.block_type` and feeds telemetry/routing; it MUST NOT change *when* `SUCCESS` is returned, because `03c` bills on `status == SUCCESS` (`CrawlOutcome` docstring).** Wiring mirrors `03d`'s proven `captcha_state` pattern: a per-call `block_state` dict mutated in `_build_result` (which has `raw_html` + `status`) and stamped onto every `CrawlOutcome` return path in `crawl_url`.
- **Per-domain strategy memory (Slice C — needs a new dependency + control-flow change).** Cache the **strategy that last succeeded per domain** (Redis key `crawl:strategy:{domain}` with TTL). Next crawl starts at the known-good tier, skipping cheaper tiers that always fail there. **Why deferred:** `WebCrawlerConnector()` is currently constructed with **no args** and is Redis/DB-free; this requires injecting a cache client (best-effort, no-op when absent) **and** letting `crawl_url` skip ladder tiers — a control-flow change to the tuned ladder. No DB migration (Redis, best-effort, self-healing on miss), but it is a deliberate design item, not a flag flip.
- **Solved-session cache (Slice C refinement — speed-positive, the strongest version of the memory above).** Beyond remembering *which tier* won, cache the **solved session itself** — the `cf_clearance`/`__cf_bm` cookies + the UA that cleared the challenge — in Redis (`crawl:session:{domain}` with TTL), and **replay** it on the next crawl via Scrapling's `cookies=` input, so a returning hit **skips the expensive solve entirely** (the pattern `references/trawl-dev` Tier 2 uses for its ~500ms repeat path: `browser/src/session.ts` `session:{domain}` save/load/invalidate; `tiers/src/tier2.ts` replays cookies, and on a re-challenge marks `session-expired` → invalidate → escalate). **Net effect is faster, not slower** (it removes a solve), so it fits the "no speed regression" bar. **Caveat:** `cf_clearance` is **IP+UA-bound**, so it pays off most behind **sticky/static proxies** (the `03b` static-proxy future) and the cache key should include proxy-geo; under rotating residential the replayed cookie may not match the new exit IP (best-effort, self-healing via the `session-expired` invalidate).
## 8. Deferred: paid-unblocker tier (the escape hatch, NOT built now)
The moat strategy is explicit: **maintain in-house bypass for ~4–6 months, then move hostile targets to a paid unblocker if demand/maintenance justifies it.** That switch is **evidence-driven, not a guess** — `03f`'s manual scorecard quantifies the free-stack ceiling over time and is the documented trigger for flipping this tier. Pre-wire the seam so the switch is a config flip, not a refactor:
- Define (but do not implement) a paid-unblocker tier — a `_crawl_with_paid_unblocker(url) -> dict | None` method returning the same shape as the existing tiers, appended **last** in `crawl_url`'s ladder (after the stealthy tier) and active only when an env flag + API key are set. (Same plain-method shape as today's tiers — there is no `FetchStrategy` contract to satisfy.)
- It would call an external unblocker (ZenRows/ScrapFly/Bright Data Web Unlocker) for the residual `DATADOME`/`KASADA`/`reCAPTCHA-Enterprise` domains the block classifier flags as unreachable in-house.
- **Its own billing** (cost-plus pass-through, decided at build time) — separate from `03c`'s flat crawl unit and `03d`'s per-solve unit, because unblocker pricing is per-request and provider-specific.
- Until built, those domains simply return non-`SUCCESS` (free under `03c`). This keeps the umbrella's "WebURL Crawler is the moat" honest while bounding our maintenance risk.
## Config / env changes
Add (all default OFF / conservative; next to the `03b`/`03c` knobs in `config/__init__.py` + `.env.example`):
- `CRAWL_GEOIP_MATCH_ENABLED` (default FALSE; Slice A — maps `RESIDENTIAL_PROXY_LOCATION` → `locale`/`timezone_id`, no exit-IP lookup).
- `CRAWL_BLOCK_WEBRTC` (default TRUE — cheap, safe; Slice A), `CRAWL_HIDE_CANVAS` (default **FALSE** — random canvas noise can itself be a tell; opt-in + `03f`-validated, see §2; Slice A), and `CRAWL_GOOGLE_SEARCH_REFERER` (default TRUE — Scrapling's `google_search`; Slice A).
- `CRAWL_DNS_OVER_HTTPS` (default **FALSE** — anti DNS-leak, but adds a DNS round-trip; default-off to honor the "no speed regression" bar, pre-wired for leak-safety-first operators; §2; Slice A).
- `CRAWL_WEBGL_SPOOF_ENABLED` (default FALSE) + optional `CRAWL_WEBGL_VENDOR` / `CRAWL_WEBGL_RENDERER` override strings (§2c; `toString`-safe JS spoof, defeats string-checks only).
- `CRAWL_PERSISTENT_PROFILES_DIR` (unset → Scrapling's temp default; set → per-domain profiles).
- `CRAWL_HEADED_XVFB_ENABLED` (default FALSE; requires Xvfb in the image).
- `CRAWL_HUMANIZE_ENABLED` (default TRUE) + dwell/jitter bounds.
- `CRAWL_STRATEGY_MEMORY_TTL_S` (default e.g. `86400`; 0 → disabled).
- `CRAWL_PAID_UNBLOCKER_ENABLED` (default FALSE) + provider/key (deferred tier).
## Docker changes (`surfsense_backend/Dockerfile`)
- Install **Xvfb** + the proven font set (`fonts-noto-color-emoji`, `fonts-unifont`, `fonts-ipafont-gothic`, `fonts-wqy-zenhei`, `fonts-tlwg-loma-otf`, `fonts-dejavu`, `fonts-liberation`; §5) in the worker image. (Note from `03a`: also drop the stale "+ Camoufox" comment near `:112`; `scrapling install` only fetches Chromium.)
- Headed runs need the browser launched under `xvfb-run` (or an Xvfb display in the worker entrypoint), gated by `CRAWL_HEADED_XVFB_ENABLED`.
## Build slicing (risk-staged; `03f` validates effectiveness)
The levers split by risk/testability. **Slice A is built now** (prod-safe, config-gated, defaults preserve today's behavior, wiring + classifier unit-tested). **B/C are deferred** (need infra or architectural design; mostly only verifiable in `03f`).
- **Slice A (SHIPPED):** centralized kwargs builder → fingerprint flags (`block_webrtc` ON, `hide_canvas` OFF, `google_search` ON, `dns_over_https` OFF); geoip coherence from `RESIDENTIAL_PROXY_LOCATION`; fonts + Xvfb **packages** in the Dockerfile; **additive** block classifier (`CrawlOutcome.block_type`, classified in `_build_result` **and** the static tier's `>=400` early-return so the first/cheapest-tier bot-gate isn't lost). **Licensing split:** the stealth kwargs builder is **proprietary** (`app/proprietary/web_crawler/stealth.py` — bypass tuning); the block classifier stays **Apache-2** (`app/utils/crawl/classifier.py` — generic passive telemetry from public markers). **`extra_headers`/`additional_args` were intentionally NOT wired** — they're last-priority override hatches with no clean env representation (dict-in-env) and no concrete need yet; add them when a real use appears rather than speculative config (YAGNI).
- **Slice B (next):** headed-Xvfb **runtime path** (`headless=False` + entrypoint), humanization `page_action` (composed with `03d`'s injector), WebGL-spoof `init_script` `.js` asset (proprietary).
- **Slice C (defer):** per-domain persistent profiles (concurrency design, §3), per-domain strategy memory + tier-skipping (`crawl_url` control-flow change + cache injection, §7), **solved-session cache** (`cf_clearance` replay via `cookies=`, §7 — speed-positive, best with sticky proxies), **warm browser pool** (a hosted-scale latency win — `references/trawl-dev` `browser/src/pool.ts` keeps N warm instances with an acquire-timeout→429; Scrapling's `StealthySession` already has page pooling), paid-unblocker config stub (§8). GPU passthrough stays infra/hosted-only (§2c).
## Work items
**Slice A (this pass):**
1. **Config + `.env.example`** — add the Slice-A knobs (below); geoip reuses `RESIDENTIAL_PROXY_LOCATION`.
2. **`app/proprietary/web_crawler/stealth.py`** (**proprietary** — bypass tuning) — `StealthConfig` + `get_stealth_config()` snapshot; coarse `RESIDENTIAL_PROXY_LOCATION → (locale, timezone_id)` map; `build_stealthy_kwargs(cfg)` returning the config-derived `StealthyFetcher` kwargs. Single source of truth imported by the connector **and** `03f`'s harness (no test-vs-prod drift). The AsyncFetcher tier **already carries `impersonate="chrome"`** (shipped `03a`, §2b) — leave it; document it as the static-tier coherence anchor.
3. **`app/utils/crawl/classifier.py`** (Apache-2) — `BlockType` enum + `classify_block(status, html) -> BlockType` (pure, status + body-marker based).
4. **Connector wiring** — merge `build_stealthy_kwargs(...)` into `_crawl_with_stealthy_sync`'s kwargs (existing `headless`/`network_idle`/`block_ads`/`solve_cloudflare`/`proxy`/captcha `page_action` preserved; lever keys never collide); add `block_type` to `CrawlOutcome`; thread a per-call `block_state` dict (classify in `_build_result` **and** the static tier's `>=400` early-return path, stamp on every `crawl_url` return) — **additive only, SUCCESS predicate unchanged**.
5. **Docker** — Xvfb + the proven font set in the worker image; drop the stale "+ Camoufox" comment.
6. **Tests + lints** — unit-test the builder (geoip map, flag wiring, snapshot) + classifier (each marker); update connector tests for `block_type`; run suite.
**Slice B / C (deferred — tracked above):** WebGL-spoof `init_script`, humanization composer, headed-Xvfb runtime, persistent profiles, strategy memory + tier-skip, paid-unblocker stub, richer `(domain, block_type, winning_strategy, attempts, latency)` instrumentation.
## Risks / trade-offs
- **Arms race / maintenance.** Fingerprint bypasses rot as WAFs update; this is exactly the cost the deferred paid tier (§8) hedges. Instrumentation (work item 9) plus **`03f`'s scorecard** are what tell us when in-house upkeep stops being worth it.
- **WebGL/GPU is a structural gap, not a bug (§2c).** The free stack cannot present a real consumer GPU; the JS spoof defeats string checks but not render-output/perf probes, and is itself brittle (rots on Chrome updates, must stay `toString`-clean). Plan for device-fingerprint-grade targets to need GPU-passthrough workers or the §8 paid tier — don't expect parity with CloakBrowser here.
- **Headed/Xvfb cost.** Headful browsers use more CPU/RAM than headless; gate per-flag and only escalate to headed when the block classifier says cheaper tiers fail for a domain (per-domain memory keeps it from being the default).
- **Profile growth.** Persistent profiles accumulate disk; add a size/TTL cap and periodic prune.
- **Geoip accuracy.** A coarse country→locale map is fine; over-fitting per-city tz isn't worth it. Wrong-but-coherent beats default-mismatched.
- **No silver bullet.** Reiterate the ceiling (§"realistic ceiling") in any user-facing copy: the crawler is "best-effort undetectable," not guaranteed.
## Prior art evaluated — why not Camoufox (decision: skip for now)
`references/Byparr-main` and `references/trawl-dev` are FlareSolverr alternatives built on **Camoufox** — an **MPL-2.0**, open-source, **C++/Juggler-patched Firefox** that natively closes our biggest gaps: hardware-accurate **WebGL** renderer/vendor spoofing (bundled `webgl_data.db`, §2c), **`geoip=True`** locale/tz/geo coherence from the proxy IP (§1), **`humanize=True`** mouse, plus font/WebRTC/audio/screen spoofing — all at the engine level (no detectable JS shims). It is effectively the CloakBrowser-class capability *without* CloakBrowser's licensing problem.
**Decision (this MVP): do not adopt Camoufox.** Rationale: Scrapling **dropped** Camoufox in favor of patchright-Chromium for its `StealthyFetcher` ([scrapling stealthy docs](https://scrapling.readthedocs.io/en/latest/fetching/stealthy.html); confirmed in source — `references/Scrapling/scrapling/engines/_browsers/_stealth.py` imports `patchright`), so wiring Camoufox would mean a **separate proprietary fetch tier** (camoufox+playwright, outside Scrapling), a Firefox engine (a few Chrome-only targets won't fit), a heavy extra binary, and tracking an actively-but-experimentally-developed project. **Revisit trigger:** if `03f`'s scorecard shows the patchright tier's device-fingerprint ceiling (§2c) is blocking real CI targets, a Camoufox tier is the prime free escalation to evaluate **before** the §8 paid unblocker.
## Out of scope (hand-offs)
- Cloudflare solving (`03a`), reCAPTCHA/hCaptcha solving + its per-solve billing (`03d`), proxy rotation (`03b`), flat crawl billing (`03c`). **Forward note for `03d` (free solver fallback):** the Camoufox-based references solve reCAPTCHA-v2 / Turnstile **free** in-browser (audio-STT — `references/trawl-dev/packages/tiers/src/solvers/{stt,turnstile}.ts`) instead of a paid solver. `03d` shipped with paid `captchatools` as primary; a free STT path is a possible *fallback-before-paid* later, but audio STT is brittle/Google-rate-limited, so paid stays primary (and this doesn't change `03e`).
- **Logged-in / account-based bypass** (sticky/static proxies + credential management) — deferred to the platform-actor work (umbrella Phase 8 + `03b` static-proxy hand-off). Public data only this MVP.
- Building the paid-unblocker provider integration — deferred (§8 leaves only the seam + flag).
- **Measuring** undetectability (the scorecard that grades these levers) → `03f` (manual harness). This plan *builds* the levers; `03f` *tests* them.
- Platform-specific structured extractors (Google Maps, LinkedIn public, …) — these sit **on top** of this hardened fetch core as Phase-8 actors; this plan only delivers the core they depend on.

155
plans/backend/03f-undetectability-testing.md
View file

@ -1,155 +0,0 @@
# Phase 3f — Undetectability & extraction test harness (manual scorecard)
> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
> **Status: ✅ IMPLEMENTED + first baseline run (`ci_mvp`, 2026-06-30).** Harness lives under the **proprietary boundary** at `surfsense_backend/app/proprietary/web_crawler/testbench/` (package: `core.py` scorecard/closure-cell helper, `suite_stealth.py` Suite S, `suite_extraction.py` Suite E, `__main__.py` CLI, `README.md` runbook) — it's the moat's measurement tool, so it carries the `app/proprietary/LICENSE`, not Apache-2. Run: `python -m app.proprietary.web_crawler.testbench --suite all [--proxy URL] [--headed] [--no-screenshots]`. Suite S builds the StealthyFetcher tier from the **shipped** `build_stealthy_kwargs(get_stealth_config())` (no drift); Suite E drives the real `crawl_url` against ToS-safe sandboxes with deterministic assertions. Writes timestamped `scorecard-*.json`/`.md` + diffs the last run; the **whole `results/` tree is gitignored** (run-local: scorecards, `latest.json`, screenshots, dumps). Captcha forced OFF (unaided score). **Not** pytest-collected (live internet + proxy). **Manual-only** — no CI/automated gating (resolved decision); it's a dev-run scorecard, not a build gate. **First baseline: see "First baseline results" below.**
>
> **Post-implementation notes (reconciled to shipped code):** (1) the `03a`/`03e` "`FetchStrategy` seam" never shipped — levers are a centralized kwargs builder, reflected below; (2) **every detection site is now auto-graded from its real DOM verdict** — the initial "INFO + screenshot, read manually" fallback was replaced after a first run by per-site parsers written against actual DOM dumps (reCAPTCHA-v3 reads the server `"score"` JSON, CreepJS the headless-% + boolean spoof tells incl. `hasHeadlessWorkerUA`, incolumitas the fpscanner FAIL keys + `is_datacenter`, fingerprint-scan the `Bot Risk Score`, FingerprintJS the block message, BrowserScan the Normal/Abnormal count, iphey the masthead verdict, Cloudflare the bypass line). `INFO` is now reserved for purely informational rows (TLS JA3/JA4, exit IP) + the manual browserleaks links; screenshots are still captured as a backstop. Each site is one entry in `suite_stealth.py`, so tightening a parser is a one-function change. Depends on `03a` (the Scrapling tier ladder + `CrawlOutcome`), `03b` (proxy provider), `03e` (the stealth levers being tested), and reuses the `page_action` + closure-cell mechanism shared with `03d`.
> **Convention note.** This is **dev/operator tooling**, not a product code path, so it's untouched by the Phase 1–2 rename (no `search_space_id`/`workspace_id` concern). Citations to the gitignored reference checkouts (`references/CloakBrowser/`, `references/Scrapling/`) are pinned to what's on disk; locate code by **symbol/grep** if lines drift.
## Objective
A **manual, repeatable scorecard** that answers one question: *how undetectable (and how correct) is our Universal WebURL Crawler right now?* It drives the real Scrapling tiers against the industry's standard detection + sandbox sites, parses each site's verdict, and prints a pass/fail + numeric scorecard. Its real job is to **quantify the free-stack ceiling** (`03e`) over time so we know — with evidence — when fingerprint maintenance stops being worth it and we should flip the deferred paid-unblocker tier.
It is explicitly **two axes, two suites** (kept separate so each scales independently):
- **Suite S — Stealth / anti-bot** (is the crawler detected?): browser-tier fingerprint/bot tests, HTTP/TLS-tier fingerprint, proxy/leak verification.
- **Suite E — Extraction correctness** (does the crawler get the content right?): scraping sandboxes for the HTTP vs JS (DynamicFetcher) tiers + Trafilatura quality.
## How CloakBrowser tests (the pattern we copy)
CloakBrowser's suite is a **driven-browser verdict-scraper**, not unit tests (`references/CloakBrowser/examples/stealth_test.py`, run via `bin/cloaktest`). The repeated shape:
1. Launch a real page with **proxy + geoip**, toggleable **headed/headless** (`stealth_test.py:231` `launch(headless=…, proxy=…, geoip=True)`).
2. `page.goto(site, wait_until="networkidle")` then **sleep** — scores compute async (Castle.js ~20 s `fingerprint_scan_test.py:31`; CreepJS ~30 s `:96`; reCAPTCHA polls up to 30 s `stealth_test.py:156–164`).
3. `page.evaluate(js)` to **parse the verdict** from the rendered DOM (sannysoft table `stealth_test.py:32–45`) or from internal JS objects (`window.Fingerprint.headless` `fingerprint_scan_test.py:117–127`).
4. Apply an explicit **pass threshold** + screenshot (`stealth_test.py:169–217`).
Their shipped bars (we adopt as **aspirational targets**, see Scorecard): sannysoft **0 fails**; `bot.incolumitas` ≤ `{WEBDRIVER, connectionRTT}` known-FPs; `browserscan` **0 Abnormal**; `deviceandbrowserinfo` `isBot=false`; FingerprintJS demo **not blocked**; reCAPTCHA v3 **≥0.7** (they hit 0.9); CreepJS **headless ≤30% / stealth ≤30%** (`fingerprint_scan_test.py:112–114,166–171`).
## The bridge to our Scrapling crawler
We are **not** a single browser; we're the `03a` tier ladder. The harness therefore tests **per tier**, and extracts verdicts two ways:
- **DOM/JSON-rendered verdicts** → just `StealthyFetcher.fetch(url, …)` (or `Fetcher.get` for JSON endpoints) and parse the **returned post-JS page** with Scrapling's selector (`load_dom` is on by default — `references/Scrapling/scrapling/fetchers/stealth_chrome.py:43`). Covers sannysoft, incolumitas, deviceandbrowserinfo, the reCAPTCHA score text, and every JSON endpoint (`tls.peet.ws/api/all`, `httpbin/headers`).
- **Internal JS-object verdicts** → a **`page_action`** that runs `page.evaluate()` and stashes the result into a **closure cell**, because Scrapling **discards `page_action`'s return value** (sync `_stealth.py:260`, async `:536`). **This is the exact same `page_action`+closure-cell plumbing as `03d`'s token injector** — building the harness de-risks `03d` (and vice-versa); factor it once. *(As-built: CreepJS/fingerprint-scan turned out to render their verdicts into the visible DOM, so their parsers read the post-JS page text directly; the closure-cell `page_action` path is retained as the mechanism for any future `window.*`-only verdict + for the screenshot backstop.)*
## Suite S — Stealth / anti-bot
### S1. Browser tier (StealthyFetcher — the "undetectable" tier)
All rows are **auto-graded** from the post-JS DOM text (no manual screenshot read); the `Extraction` column names the actual marker each parser keys on.
| Site | Signal | Extraction | Aspirational bar |
|---|---|---|---|
| `bot.sannysoft.com` | webdriver/chrome/plugins/UA leaks | `class="failed"` cell count | 0 fails |
| `bot.incolumitas.com` | 30+ checks incl. behavioral + IP class | `"<key>":"FAIL"` keys, `is_datacenter` | 0 fpscanner FAIL |
| `browserscan.net/bot-detection` | WebDriver/CDP/Navigator | `Normal`/`Abnormal` count | 0 Abnormal |
| `deviceandbrowserinfo.com/are_you_a_bot` | fingerprint + behavioral | `"isBot":` JSON | `isBot=false` |
| `abrahamjuliot.github.io/creepjs` | fingerprint **consistency/lies**, headless% | `N% headless` + boolean tells (`hasHeadlessWorkerUA`…) | headless ≤30%, no tells |
| `fingerprint-scan.com` | Castle.js bot-risk + headless signals | `Bot Risk Score: N/100` | risk < 50 |
| `demo.fingerprint.com/web-scraping` | **behavioral block** (FingerprintJS Pro Smart Signals) | block message (`access denied` / `tampering detected`) | not blocked |
| `recaptcha-demo.appspot.com/...v3-request-scores.php` | Google server-verified human score | server verify `"score":` JSON | score ≥0.7 |
| `www.scrapingcourse.com/cloudflare-challenge` | **Cloudflare challenge** (only row exercising `solve_cloudflare`) | `you bypassed the cloudflare challenge` line | bypassed |
| `iphey.com` | cross-layer fingerprint+IP+geo coherence | masthead `Your Digital Identity Looks <verdict>` (async, ~25 s settle) | `Trustworthy` |
### S2. Per-property fingerprint detail (manual/debug — validates `03e` levers directly)
- `browserleaks.com/canvas`, `/webgl`, `/fonts` — confirms `03e` `hide_canvas` + font packages.
- `browserleaks.com/webrtc` + DNS leak — confirms `03e` `block_webrtc` (no real-IP leak through the proxy).
### S3. HTTP/TLS tier (AsyncFetcher / `Fetcher` — curl_cffi impersonation)
Our HTTP tier is `curl_cffi`-based and impersonates a real Chrome TLS stack (`references/Scrapling/scrapling/fetchers/requests.py:29`; `engines/static.py:6–9,36–47`) **only when `impersonate=` is set** — which `03a` **now ships** (`app/proprietary/web_crawler/connector.py`, the `AsyncFetcher.get` call passes `impersonate="chrome"`). This row therefore **validates the shipped parity** rather than driving a fix: confirm the static tier's JA3/JA4 matches a real Chrome (the `03e §2b` lever). If you ever need a before/after, temporarily drop `impersonate=` to reproduce the curl_cffi-default (red) baseline.
- `tls.peet.ws/api/all` (+ `/api/clean`) — JSON **JA3/JA4/Akamai-HTTP2/PeetPrint**; diff against a real-Chrome baseline.
- `httpbin.co/headers` (or httpbingo) — header set/order/UA sanity.
**Recommendation (resolved): TLS parity is a first-class *axis* but an *informational threshold*, not a hard gate.** We record JA3/JA4 and flag drift from the Chrome baseline, but don't "fail" on it — curl_cffi impersonation is strong yet JA-hashes shift across versions, and this is a manual scorecard anyway. Treat a *mismatch* as a tuning signal (pick a closer `impersonate` profile), not a regression.
### S4. Proxy / leak verification
- `httpbin.org/ip` — exit IP == the proxy endpoint actually used (capture-once seam from `03b`/`03e`, not a re-rotated `get_proxy_url()`).
- WebRTC/DNS (S2) — no real-IP leak.
## Suite E — Extraction correctness (separate axis)
Purpose-built, ToS-safe sandboxes — validates the HTTP vs **DynamicFetcher (JS)** tiers and Trafilatura output, independent of stealth:
- `books.toscrape.com` — static catalog + pagination (HTTP tier; baseline extraction).
- `quotes.toscrape.com` — has **`/js`** (JS-rendered), **`/js-delayed`** (`?delay=`), **scroll** (infinite), **login** (CSRF) variants → exercises the DynamicFetcher JS tier + `wait_selector`/`network_idle`.
- `scrapethissite.com` — mixed structures for extraction robustness.
Assertion style: known expected values (e.g. first book title, quote count per page) so extraction regressions are caught deterministically.
## First baseline results (2026-06-30, headless, rotating residential proxy)
First real run of the free stack (patchright-Chromium + curl_cffi `impersonate="chrome"` + `anonymous_proxies` rotating residential), captcha solving OFF, Slice-A levers only (`CRAWL_GEOIP_MATCH_ENABLED=false`). **Suite S: 6 PASS / 4 FAIL across the 10 detection sites** (the other 6 rows are informational: TLS, exit IP, 4 browserleaks manual links).
| Site | Verdict | Observed |
|---|---|---|
| sannysoft | ✅ PASS | 0 failed cells |
| deviceandbrowserinfo | ✅ PASS | `isBot=false` (incl. `isPlaywright=false`, `isAutomatedWithCDP=false`) |
| reCAPTCHA v3 | ✅ PASS | server score **0.9** |
| BrowserScan | ✅ PASS | Test Results: Normal, 0 Abnormal (WebDriver/CDP/Navigator) |
| fingerprint-scan | ✅ PASS | Bot Risk **35/100** (site flags >50) |
| cloudflare_challenge | ✅ PASS | turnstile **solved** → "you bypassed the Cloudflare challenge" |
| **CreepJS** | ❌ FAIL | headless **33%**, **`hasHeadlessWorkerUA: true`** (worker UA leaks `HeadlessChrome`) |
| **incolumitas** | ❌ FAIL | only legacy `fpscanner WEBDRIVER: FAIL`; all modern tests OK |
| **FingerprintJS Pro** | ❌ FAIL | "anti-detect tampering detected, access denied" |
| **iphey** | ❌ FAIL | verdict **Unreliable** |
**Reading the failures (maps directly to the moat roadmap):**
- **FingerprintJS Pro** = the commercial bar (the documented free-stack ceiling). Needs WebGL/GPU + deeper patches (`03e` Slice B/C) or the deferred paid-unblocker (`03e §8`). Hardest.
- **CreepJS `hasHeadlessWorkerUA`** = the **Web Worker** `navigator.userAgent` still reports `HeadlessChrome` (main-thread UA is clean). Known patchright leak, **plausibly fixable** (worker UA override) → concrete `03e` Slice-B candidate.
- **iphey "Unreliable"** = almost certainly **geoip incoherence** (browser tz/locale default `America/Los_Angeles` vs the rotating proxy's exit geo, because `CRAWL_GEOIP_MATCH_ENABLED` is default-off in Slice A). iphey is now a **live regression test for the `03e` geoip-coherence lever** — flipping `CRAWL_GEOIP_MATCH_ENABLED` is the expected fix to validate.
- **incolumitas** = a single 2017-era `fpscanner WEBDRIVER` check that even some real browsers trip; modern checks (webdriverPresent, SELENIUM_DRIVER, HEADCHR_*, CDP) all OK. Lowest priority.
**Proxy-quality note:** incolumitas's IP classifier returned `is_datacenter: true` on one rotation and `false` on another — the `anonymous_proxies` rotating pool **mixes datacenter and residential exits**, a real undetectability variable independent of our code (input to future proxy-provider evaluation).
## Scorecard & thresholds (resolved)
- **Adopt CloakBrowser's bars as aspirational targets** (above), but the harness's primary output is **our actual measured numbers recorded as the baseline** — a `scorecard.md`/JSON snapshot per run (date, tier, proxy on/off, headed/headless, per-site result), written to the **gitignored** `results/` tree. Subsequent runs diff against the last on-disk baseline so we see drift (ours improving, or a WAF tightening).
- Each row reports: site, tier, verdict, numeric (where applicable), PASS/FAIL vs aspirational bar, and screenshot path.
- A run is summarized as `passed/total` per suite (like `stealth_test.py:314–317`), **never blocking** anything.
## Harness design
- Lives under the **proprietary boundary** at `surfsense_backend/app/proprietary/web_crawler/testbench/` (moat measurement tooling — not Apache-2; not collected by the normal pytest run, since it hits the live internet + needs proxies). It is moved here as a coherent package because it can't be cleanly *partially* moved (a proprietary Suite S would otherwise back-import generic scaffolding from `scripts/`, a forbidden app→scripts direction). A thin CLI mirrors `bin/cloaktest`: `python -m app.proprietary.web_crawler.testbench [--proxy URL] [--headed] [--suite S|E|all] [--no-screenshots]`.
- **Reuse split (important — `crawl_url` is *not* a drop-in here):**
- **Suite E** drives the **real `crawl_url`** end-to-end — extraction correctness *is* the production path (auto-ladder + Trafilatura markdown is exactly what we want to assert).
- **Suite S** does **not** use `crawl_url`. Two reasons: (1) `crawl_url` auto-ladders and stops at the first `SUCCESS`, so a detection site might be answered by the cheap HTTP tier when we mean to grade the **StealthyFetcher** tier; (2) `crawl_url` returns Trafilatura **markdown** (`_build_result`), but verdict parsing needs the **raw post-JS DOM** (and `window.*` objects need a live page). So Suite S drives the **individual Scrapling fetchers directly**, per tier.
- **Avoid test-vs-prod drift:** Suite S must construct the StealthyFetcher tier from the **same centralized stealth-config builder** `03e` shipped — `build_stealthy_kwargs(get_stealth_config())` in `app/proprietary/web_crawler/stealth.py` (the single source of truth for `block_webrtc`/`hide_canvas`/`google_search`/`dns_over_https` + geoip `locale`/`timezone_id`) — **not** a hand-rolled kwargs set, otherwise the scorecard grades a browser we don't ship. The static (HTTP) tier's `impersonate="chrome"` anchor stays hardcoded in the connector (`03a` §2b); `real_chrome`/headed/persistent-profile are Slice B/C and **not** yet in the builder, so the harness exercises today's Slice-A browser.
- Outputs: console summary + screenshots + the `scorecard` snapshot.
- Runs with the app-wide proxy provider (`03b`) and `03e` levers on, so the scorecard reflects production fetch behavior — **except captcha solving, which Suite S forces OFF** (`CAPTCHA_SOLVING_ENABLED=false`): we measure the *unaided* stealth/score, and it avoids the `03d` injector firing paid solves against the reCAPTCHA-demo / FingerprintJS rows. Captcha solving is exercised separately (functional `03d` test), not in the stealth scorecard.
## The ceiling-decision loop (why this matters)
The scorecard is the evidence base for the moat strategy: re-run it (a) on a cadence and (b) whenever crawls start failing in the wild. When the **hard** rows (FingerprintJS demo, reCAPTCHA-v3 ≥0.7, and any DataDome/Kasada targets) degrade and `03e` levers can't recover them, that's the documented trigger to flip the **deferred paid-unblocker tier** (`03e §8`). Without this harness, that decision is guesswork.
## Config / dependencies
- **No new production dependencies** — reuses Scrapling (already pinned) + the crawler. Optionally a tiny dev helper for the scorecard diff (or just stdlib `json`).
- Needs **residential proxy creds** (the `03b` env) to be meaningful; document a "no-proxy" mode that still runs but is expected to fail the harder rows (datacenter IP).
- A results dir — **entirely gitignored** (run-local scorecards, `latest.json`, screenshots, dumps); kept tracked only via its `.gitignore`.
## Work items
1. **Shared `page_action`+closure-cell helper** (factored with `03d`) for JS-object verdict extraction.
2. **Suite S runners** (S1 browser, S3 TLS, S4 proxy-leak) with per-site parse + aspirational thresholds; S2 as manual debug links.
3. **Suite E runners** against the toscrape/scrapethissite sandboxes with deterministic assertions.
4. **CLI + scorecard** (snapshot writer + diff vs last baseline) mirroring `bin/cloaktest` ergonomics.
5. **Docs**: a short runbook (how to run, read the scorecard, interpret the ceiling trigger).
## Risks / trade-offs
- **Flaky/rate-limited/changing sites.** Detection sites move DOM and tighten over time; the harness must tolerate parse misses (report `ERROR`, not crash — `stealth_test.py:298–300`) and is **manual** precisely so flakiness never blocks development.
- **Proxy required for realism.** Without residential egress the hard rows fail by design (datacenter IP); document this so a red scorecard isn't misread.
- **ToS.** These are public detection/sandbox sites intended for this purpose; keep volume low and don't hammer.
- **Not a guarantee.** Passing sannysoft/CreepJS ≠ beating DataDome/Kasada; the scorecard's value is *trend + ceiling visibility*, not a green checkmark.
## Out of scope (hand-offs)
- **CI automation** — deferred (resolved: manual-only now). If revisited, it needs tolerant thresholds + proxy creds + nightly cadence; the scorecard JSON is designed to make that easy later.
- The levers themselves (`03e`), captcha solving (`03d`), proxy provider (`03b`), billing (`03c`) — this plan only **measures** them.
- The deferred paid-unblocker integration (`03e §8`) — the harness defines its *trigger*, not its build.

89
plans/backend/04-capabilities.md
View file

@ -1,89 +0,0 @@
# Phase 4 — Capabilities (the scraper APIs)
> Build first; every other phase calls this registry. Sibling: `05-access.md` (the doors).
> Reuses Phases 3a/3b/3c (shipped): `WebCrawlerConnector.crawl_url -> CrawlOutcome`, the proxy/stealth/
> captcha tiers, and the `WebCrawlCreditService` billing seam. Locate code by symbol/grep.
## Objective
Expose the Acquisition engine as a small set of typed, stateless scraper verbs (`web.*`, `maps.*`) that
return cleaned, AI-ready structured data. One capability = one function you call and get data back from.
## Verb set (MVP)
Two namespaces: `web.*` (generic crawler) and `maps.*` (Google Maps actor). Future platforms slot in as
their own namespace.
| Verb | Input → Output (cleaned) | Executes over | Billing unit |
|------|--------------------------|---------------|--------------|
| `web.scrape(urls[])` | `[{url, status, content, metadata}]` | loop `crawl_url` | per success |
| `web.discover(query, top_k)` | `[{url, title, snippet, provider}]` | search providers | per search / free |
| `maps.search(query, location)` | `[place]` | Maps actor | per place |
| `maps.place(place_id\|url)` | `place` | Maps actor | 1 |
| `maps.reviews(place)` | `[review]` (paged) | Maps actor | per page |
`maps.*` returns typed structured objects (`{name, rating, review_count, hours, price_level, …}`).
`web.scrape` takes a URL array.
## Execution
Each verb is a direct async call that returns its cleaned result. Slow verbs (large `web.scrape` arrays,
`maps.reviews` paging) are bounded/streamed at the endpoint.
## Registry
One entry per verb, and the single source of truth the doors (`05`) and the agent (`07`) read from:
```
Capability {
name # dotted, e.g. "web.scrape", "maps.search"
input_schema # Pydantic
output_schema # Pydantic (cleaned, AI-ready)
executor # async fn; wraps Acquisition / Maps actor; returns data directly
billing_unit # how the billing service charges this call
}
```
Adding a verb once lights it up on every door.
## Billing
A verb declares a `billing_unit`; charging is delegated to the billing service (`03c` first provider).
- `web.scrape` → per `SUCCESS` (`web_crawl` unit).
- `maps.*` → new per-place / per-page unit registered with the billing service.
- captcha attempts → existing per-attempt `web_crawl_captcha` unit (`03d`).
- `web.discover` → per-search unit (or free).
## Location
New Apache-2 package `app/capabilities/` (registry, schemas, executors) — open-source and self-hostable.
Executors import the proprietary Acquisition engine and Maps extractor (`app/proprietary/…`).
## Work items
1. Registry package: `Capability` dataclass + registry, imported by `05` and `07`.
2. `web.scrape` executor: loop `crawl_url` over the URL array → per-URL cleaned rows; `web_crawl` unit.
3. `web.discover` executor: wrap the search-provider core (SearXNG/Linkup/Baidu, env-keyed); its unit.
4. `maps.*` contracts: input/output schemas + executor stubs against the Maps actor.
5. Output normalization: each executor returns AI-ready structured output.
6. Billing seam: honor `billing_unit` via the billing service.
## Tests
- `web.scrape([a,b,c])` returns one cleaned row per URL inline; partial failures don't fail the batch.
- Each `SUCCESS` bills one `web_crawl` unit; `EMPTY`/`FAILED` free; disabling the flag no-ops.
- A door and the agent hit the same executor for a verb.
- `web.discover` returns `{url,title,snippet,provider}`; self-disables when no provider key is set.
- `maps.*` returns typed structured objects against fixtures.
## Out of scope
- Doors → `05`. Keep-watching mode → `06`. Agent → `07`.
- Google Maps actor build (incl. sourcing legality) — net-new, separate effort; `maps.*` are contracts.
- Additional platform namespaces; recursive site crawl; a generic `web.extract` verb.
## Open questions
- `web.discover` metered vs free.
- How large `web.scrape` / `maps.reviews` responses are bounded/streamed.

85
plans/backend/05-access.md
View file

@ -1,85 +0,0 @@
# Phase 5 — Access / Surfaces (REST · MCP · chat doors)
> Build after `04`. Together `04 + 05` ship the scraper-API product.
> Reuses shipped infra: Identity/Tenancy, API keys, the chat agent + tool registry, the streaming layer,
> Metering (`03c`), and Rocicorp Zero reactive sync. Locate code by symbol/grep.
## Objective
Expose the `04` capability registry through three doors — REST + API keys, an MCP server, and chat tools —
each generated from the one registry so the I/O contract can't drift. Every door is the same thin adapter
with no business logic.
## Adapter shape (identical on every door)
```
parse input → validate against verb.input_schema → authn/authz → meter-gate (03c)
→ call the SAME executor (04) → serialize verb.output_schema → return cleaned data
```
The executor returns directly (`04`).
## The three doors
| Door | Who | Auth | Status |
|------|-----|------|--------|
| REST + API keys | external developers | existing API-key infra | public day one |
| MCP server | external agents (Cursor/ChatGPT/Claude) | OAuth 2.1 or bearer (at implementation) | fast-follow |
| Chat tools | the in-app agent (`07`) | existing session + workspace | seed exists (`scrape_webpage`) |
Routes are generated, so public REST on day one costs nothing extra.
## Chat surface
The in-app product is the conversation: the user describes a need in plain language and the agent (`07`)
picks and fills verbs. Raw typed verbs are exposed on REST/MCP for devs and external agents. The chat
door classifies intent and routes:
```
"find / compare / what is / pull / right now" → ONE-SHOT → call verbs, answer now
"watch / track / notify me / keep an eye / over time" → KEEP-WATCHING → hand to the ongoing mode (06)
```
## MCP — two directions
- **Serve MCP** (new door): our capabilities as a remote MCP server — Streamable-HTTP `/mcp`, stateless,
bounded/paginated outputs, read-only verbs (least privilege).
- **Consume MCP** (reuse): the BYO-`MCP_CONNECTOR` — the user's own external MCP tools inside our chat
agent. Fix its routing-map gap here.
## Result delivery
A chat tool call returns its data inline in the turn. Asynchronous results (e.g. the `06` ongoing mode
writing between turns) reach the client via write-then-sync / stream: the worker writes rows and the
client receives them over the existing SSE stream or a Zero-published table.
## Work items
1. Door generator: from the `04` registry emit (a) REST routes + models, (b) MCP tool schemas + handlers,
(c) chat tool defs + handlers.
2. REST surface: public routes + API-key auth (reuse) + `03c` meter-gate.
3. Chat tools: generalize `scrape_webpage` into the registry-backed set (direct-return).
4. Intent routing: classify one-shot vs keep-watching; hand keep-watching to `06`.
5. MCP server (fast-follow): Streamable-HTTP `/mcp`, least-privilege; auth depth at implementation.
6. Consume-MCP fix: repair the BYO-`MCP_CONNECTOR` routing-map gap.
## Tests
- A verb added to the registry appears on REST + MCP + chat with identical I/O.
- REST without a valid key → 401; an over-budget call → blocked by the `03c` gate; a success charges once.
- A chat scrape returns cleaned data inline in the same turn.
- "compare X and Y" → one-shot; "watch X weekly" → routed to `06`.
- A configured BYO MCP tool is reachable by the chat agent.
## Out of scope
- Verbs, executors, billing units → `04`.
- Keep-watching mechanism + its delivery channel → `06`.
- The CI subagent + its prompt → `07`.
- Legacy branded connectors (backward-compat hygiene, separate task).
## Open questions
- MCP auth depth (decide at implementation).
- Public REST rate-limiting / abuse posture.
- Async delivery channel for `06` outputs: SSE stream vs Zero-published messages table (settled with `06`).

65
plans/backend/06-ongoing-automation.md
View file

@ -1,65 +0,0 @@
# Phase 6 — Ongoing Automation (chat-native "keep watching")
> Depends on `04` (the verbs it re-invokes), `05` (the agent tools), and `07` (the `scraping` subagent).
> Reuses the existing chat + automations machinery; adds no parallel engine.
## Objective
Support "keep watching": a persistent chat where the agent periodically re-invokes scraper verbs and drops
results into the session. "What changed" is the agent reading its own prior turns from the durable
checkpoint — no diff store.
## Mechanism
A **chat watch is an `Automation` bound to the current chat** — nothing more. No dedicated thread, no
thread "kind", no schema change. Start it and the chat gains a `schedule` trigger that re-posts the
question on a cadence; stop it and the automation is deleted and the chat is a normal chat again. The
chat's own checkpoint is the memory.
- **`chat_message` action** — params `{ thread_id, message }`. Its handler drains
`stream_new_chat(user_query=message, chat_id=thread_id, …)` under `AuthContext.system(creator,
source="automation")` against the **current chat**. Auto-approve (system auth; CI verbs are read-only
and don't interrupt).
- **Durable memory + delivery** — `stream_new_chat` persists messages to `new_chat_messages` (already
Zero-synced to the UI) and advances the shared Postgres checkpointer keyed by `chat_id`. A scheduled run
has no SSE client; it runs server-side and is delivered via the persisted rows. "What changed" is the
agent reading the chat's own prior turns.
- **Worker-safe checkpointer** — the shared `AsyncPostgresSaver` pool binds connections to the loop that
opened them, but Celery uses a fresh loop per task (`PoolTimeout`). Dispose the checkpointer pool per
task in `run_async_celery_task`, mirroring `_dispose_shared_db_engine`, so a worker can use the *durable*
checkpointer (not `InMemorySaver`) that "what changed" requires.
- **`start_watch`** — a `scraping` subagent tool that binds a watch to the *current* chat: it distills
the recurring question + cadence and creates the automation (`schedule` + `chat_message(thread_id =
current chat)`).
- **"Is this chat watched?"** — derived: an active automation with a `chat_message` action targeting the
chat. No stored flag.
- **Controls** — run-now = trigger a run; stop = delete the automation (chat reverts to normal).
- **Concurrency** — the checkpointer is single-writer per thread; a tick skips if the prior turn on that
chat is still running (DB `ai_responding` flag).
## Work items
1. Durable checkpointer in workers: `_dispose_shared_checkpointer_pool` in `run_async_celery_task`
(before + after), mirroring the SQLAlchemy engine dispose. **[done]**
2. `chat_message` action: params + factory + handler (drains `stream_new_chat`); concurrency guard. **[done]**
3. Watch service: create (bind `schedule` + `chat_message` automation to a chat) / stop (delete) /
find-for-thread (is-watched) / run-now. **[done]**
4. `start_watch` tool on the `scraping` subagent (+ prompt line); binds to the current chat. **[done]**
5. Controls — chat tools (`stop_watch`, `refresh_watch`) + REST (`GET /automations/watches`,
`POST /automations/{id}/run`; stop = `DELETE /automations/{id}`). **[done]**
## Tests
- `run_async_celery_task` disposes the checkpointer pool before and after a task. **[done]**
- `chat_message` drains a turn on the given thread; skips when one is in-flight. **[done]**
- Watch service: create binds a `schedule` + `chat_message` automation to the chat; stop deletes it;
find-for-thread filters by plan; run-now launches the schedule trigger. **[done]**
- `start_watch` / `stop_watch` / `refresh_watch` tools act on the current chat from tool context. **[done]**
- Watch routes registered on the automations router. **[done]**
- Integration (running stack): two scheduled runs on one chat — run 2 sees run 1 in checkpoint history.
## Deferred / out of scope
- Zero delivery-cost optimization (signal-column + REST fetch vs full-content sync) — app-wide, separate.
- Server-side turns surviving browser navigation for *interactive* chat ("zombie streaming").
- `start_watch` cadence UX refinements. Verbs/doors → `04`/`05`; agent playbook → `07`.

76
plans/backend/07-orchestration.md
View file

@ -1,76 +0,0 @@
# Phase 7 — Orchestration / Conversation (the CI-expert subagent)
> **Status (2026-07-02): shipped for the web verbs.** The `scraping` subagent, its tools, prompt, and
> router blurb are live; `web.discover`/`web.scrape` compose and the "keep watching" handoff (`06`) is
> wired. `maps.*` composition is deferred until the Maps actor exists.
> Sits atop `04`/`05` (and `06` for the ongoing mode). The multi-agent chat runtime (deepagents, subagent
> dispatch, streaming, citation middleware) is shipped. Net-new here = one builtin subagent + its tools +
> its prompt. Locate code by symbol/grep.
## Objective
Ship the `scraping` subagent — a builtin CI/scraper-expert that delivers the product through
plain conversation: understand intent, compose scraper verbs (`04`), answer in plain language, and hand
standing needs to the ongoing "keep watching" mode (`06`). It reasons over chat history to report what's
new vs prior tool outputs already in context.
## The subagent
Builtin `subagents/builtins/scraping/`, peer to `research`/`deliverables`, packed via
`pack_subagent`. Files follow the existing pattern: `agent.py` (`build_subagent`), `tools/index.py`
(`NAME · RULESET · load_tools`), `description.md` (router one-liner), `system_prompt.md` (the playbook).
The main agent delegates CI-/scraping-flavored requests to it via `description.md`.
## The playbook (`system_prompt.md`)
1. **Intent routing** — one-shot ("compare/find/what is") → compose verbs & answer; standing concern
("watch/track/notify when/weekly") → hand to `06`; ambiguous → ask one question ("just once, or keep
watching?").
2. **Verb composition** — chains `web.discover → web.scrape` and `maps.search → maps.place → maps.reviews`;
infer URLs/queries/locations from context.
3. **"What changed"** — re-invoke the verb and compare against prior tool outputs in the chat history,
summarizing what's new.
## The toolset (`load_tools`)
| Tool | Wraps | Billing |
|------|-------|---------|
| capability verbs — `web.scrape`, `web.discover` shipped; `maps.search`/`maps.place`/`maps.reviews` deferred (Maps actor) | `04` executors (direct-return) | `03c` owner-wallet charge via `charge_capability` (same path as REST/MCP) |
| `start_watch` / `stop_watch` / `refresh_watch` | bind a watch to the current chat via `06` | per re-invocation |
Verbs reach the subagent through the chat door generator `access/chat.py::build_capability_tools` (`05`),
which turns registry verbs (`04`) into tools with owner-wallet billing. The `scraping` subagent owns these
CI-flavored calls; the main agent's `web_search`/`scrape_webpage` stay untouched.
## Boundaries
- The subagent drives the `04` verbs via tools; fetch/bypass/clean logic stays in Acquisition + `04`,
callable headless.
- Humans get the agent; REST/MCP callers call `04` verbs directly, skipping this subagent.
## Work items
1. **[done]** Subagent scaffold: `subagents/builtins/scraping/` (`agent.py` / `tools/index.py` /
`description.md` / `system_prompt.md`), packed via `pack_subagent`.
2. **[done]** CI playbook prompt: intent routing (block on ambiguous cadence), verb chains, "what changed" reasoning.
3. **[done]** Chat door generator: `access/chat.py::build_capability_tools` turns registry verbs (`04`) into tools (`05`).
4. **[done]** `start_watch` / `stop_watch` / `refresh_watch` seam to `06`, bound to the current chat.
5. **[done]** Router registration: `description.md` for main-agent delegation.
## Tests
- A CI-/scraping-flavored request routes to `scraping`; a non-CI request does not.
- "compare X and Y" composes verbs and answers in plain language, persisting nothing.
- "reviews for the top 3 coffee shops near me" chains `maps.search → maps.place → maps.reviews`.
- A follow-up re-invokes the verb and reports new items vs prior in-context outputs.
- "watch this weekly" routes to the ongoing mode, not a one-shot answer.
## Out of scope
- Verbs, doors → `04`/`05`. Ongoing/periodic mechanism → `06`.
- Richer multi-step playbooks, proactive suggestions, cross-topic synthesis.
- Frontend CI surfaces → frontend umbrella.
## Open questions
- None outstanding for the web verbs. `maps.*` composition reopens when the Maps actor lands.

23
surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/system_prompt/prompts/identity/private.md
View file

@ -1,8 +1,23 @@
<agent_identity>
You are **SurfSense's main agent**. Your job is to answer the user using their
knowledge base, lightweight web research, persistent memory, and **specialist
subagents** invoked via the `task` tool. You are an orchestrator — most
non-trivial work belongs on a specialist.
You are **SurfSense's main agent**, the orchestrator of an open-source
competitive intelligence platform. Users come to you to understand their
market: what competitors are doing, how audiences react, where rankings and
reviews are moving, and what is being said across the open web — and to put
that intelligence to work alongside their own knowledge base.
You do this by dispatching **specialist subagents** via the `task` tool:
- **Live market data** — Reddit, YouTube, Google Maps, Google Search, and the
web crawler return structured, current platform data (posts, comments,
transcripts, reviews, SERPs, full page content).
- **The user's own context** — their knowledge base, connected apps, and
persistent memory.
- **Deliverables** — reports, podcasts, and presentations built from what the
specialists find.
You are an orchestrator — most non-trivial work belongs on a specialist. Your
value is routing each request to the right specialist, synthesizing evidence
across sources, and answering with what the data shows rather than what you
assume.
Today (UTC): {resolved_today}
</agent_identity>

23
surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/system_prompt/prompts/identity/team.md
View file

@ -1,8 +1,23 @@
<agent_identity>
You are **SurfSense's main agent**. Your job is to answer the user using their
shared team knowledge base, lightweight web research, persistent memory, and
**specialist subagents** invoked via the `task` tool. You are an orchestrator
— most non-trivial work belongs on a specialist.
You are **SurfSense's main agent**, the orchestrator of an open-source
competitive intelligence platform. This team comes to you to understand its
market: what competitors are doing, how audiences react, where rankings and
reviews are moving, and what is being said across the open web — and to put
that intelligence to work alongside the team's shared knowledge base.
You do this by dispatching **specialist subagents** via the `task` tool:
- **Live market data** — Reddit, YouTube, Google Maps, Google Search, and the
web crawler return structured, current platform data (posts, comments,
transcripts, reviews, SERPs, full page content).
- **The team's own context** — its shared knowledge base, connected apps, and
persistent team memory.
- **Deliverables** — reports, podcasts, and presentations built from what the
specialists find.
You are an orchestrator — most non-trivial work belongs on a specialist. Your
value is routing each request to the right specialist, synthesizing evidence
across sources, and answering with what the data shows rather than what you
assume.
Today (UTC): {resolved_today}

9
surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/system_prompt/prompts/kb_first.md
View file

@ -1,5 +1,11 @@
<knowledge_base_first>
CRITICAL — ground factual answers in what you actually receive this turn:
- **live platform data** via the market specialists —
`task(reddit, ...)`, `task(youtube, ...)`, `task(google_maps, ...)`,
`task(google_search, ...)`, `task(web_crawler, ...)`. Anything about
competitors, markets, rankings, reviews, or audience sentiment is answered
from what these return **this turn**, never from your training data: your
general knowledge of companies, prices, and rankings is stale by definition,
- the user's knowledge base via `task(knowledge_base, ...)` (your PRIMARY
source for anything about their own uploaded files, documents, and notes —
the `<workspace_tree>` only lists what exists, so delegate to the specialist
@ -7,9 +13,6 @@ CRITICAL — ground factual answers in what you actually receive this turn:
- injected workspace context (see `<dynamic_context>`),
- the user's connected apps via `task(mcp_discovery, ...)` (Slack, Jira,
Notion, Gmail, Calendar, etc. — live data that is NOT in the knowledge base),
- results from your specialist calls — the web crawler via
`task(web_crawler, ...)` or the Google Search specialist via
`task(google_search, ...)`,
- or substantive summaries returned by a `task` specialist you invoked.
For questions about the user's own files and notes, dispatch

5
surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/system_prompt/prompts/reminder.md
View file

@ -1,4 +1,5 @@
<reminder>
Concise · KB-grounded · delegation-first · one `task` per turn · no direct
filesystem · persist memory when durable facts appear.
Concise · grounded in this turn's specialist data, never stale general
knowledge · delegation-first · no direct filesystem · persist memory when
durable facts appear.
</reminder>

25
surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/system_prompt/prompts/routing.md
View file

@ -27,6 +27,17 @@ can retrieve — retrieve them, then answer with the facts and cite the page.
Large results are fine: extract and return them, don't ask permission for
bounded fan-out (≤20 sites) the user already requested.
**Audience sentiment lives on the platforms.** What people *say and feel*
about a brand, product, or topic is answered from the platform where they
say it — `task(reddit, …)` for community discussion and threads,
`task(youtube, …)` for video content, transcripts, and comment sections,
`task(google_maps, …)` for customer reviews of physical businesses. Web
search only finds articles *about* the conversation; the platform
specialists return the conversation itself, structured and current. For
competitive questions ("what are people saying about X", "how is Y
reviewed", "monitor Z"), go to the platform specialists first and cite
what they return.
**Places go to Maps, the open web goes to Search.** Discovering physical
businesses or venues of a type in a geography ("clinics in X", "tutoring
centers near Y", lead lists of local businesses) is the Maps specialist's
@ -113,6 +124,20 @@ user: "What did Maya say about the Q2 roadmap in Slack last week?"
timestamp.")
</example>
<example>
user: "What are people saying about Cursor vs Windsurf lately?"
→ Audience sentiment — go to the platform, not web search. Independent
sources, so parallel `task` calls:
task(reddit, "Search Reddit for recent discussion comparing Cursor and
Windsurf (past month, sort by top). Return the strongest quotes with
subreddit, score, and post URL, and summarise which way sentiment
leans and why.")
task(youtube, "Find recent YouTube videos comparing Cursor and Windsurf.
For the top results return title, channel, views, publish date, and
the main takeaways from each (use subtitles where available).")
Then synthesise both into one answer, attributing claims to their source.
</example>
<example>
user: "What's the current USD/INR rate?"
→ Public web lookup — delegate to the Google Search specialist:

5
surfsense_backend/app/db.py
View file

@ -3037,6 +3037,11 @@ async def create_db_and_tables():
await conn.execute(text("CREATE EXTENSION IF NOT EXISTS vector"))
await conn.execute(text("CREATE EXTENSION IF NOT EXISTS pg_trgm"))
await conn.run_sync(Base.metadata.create_all)
# create_all never creates zero_publication (a migration-only
# artifact), and without it zero-cache crash-loops. Idempotent.
from app.zero_publication import ensure_publication
await conn.run_sync(ensure_publication)
await setup_indexes()

44
surfsense_backend/app/proprietary/README.md
View file

@ -1,44 +0,0 @@
# `app.proprietary` — non-Apache-2 license boundary
Everything in this directory tree is licensed **separately** from the rest of
SurfSense (which is Apache-2.0), under the **Business Source License 1.1** —
see [`LICENSE`](./LICENSE). In short: source-available; production use is
permitted (including self-hosting the whole app) *except* offering it to third
parties as a commercial product or hosted/managed service; each released
version converts to Apache-2.0 four years after its release.
## Why this exists
This package holds the product moat:
- the in-house **undetectable web crawler** (Scrapling tiers + stealth/captcha
hardening), and
- (future) **platform-specific actors** that scrape/extract structured data from
individual platforms.
Keeping it in one clearly-named directory makes the license boundary
unambiguous: a single rule — *everything under `app/proprietary/**` is not
Apache-2.0* — instead of per-file headers scattered across the tree.
## Layout
- `web_crawler/` — the Scrapling-based crawler engine. Public API:
`WebCrawlerConnector`, `CrawlOutcome`, `CrawlOutcomeStatus`
(`from app.proprietary.web_crawler import ...`).
- `platforms/` — (future, Phase 8) platform-specific actors; scaffolded/empty.
## Rules
- **Do not** add Apache-2.0-intended code here.
- Apache-2.0 code elsewhere **may import from** this package (the indexer and the
`web.crawl` capability do); that does not move them under this license.
- Depend only on the public API exported from each subpackage's `__init__`, not
on internal modules, so the boundary stays clean and swappable.
- **Boundary test:** put code here only if it is used *exclusively* by the moat.
Generic infrastructure that Apache-2 features also depend on stays Apache-2
even when the crawler uses it too. Example: `app/utils/proxy/` (provider
abstraction, registry, `CustomProxyProvider` + rotation — a thin wrapper over
Scrapling's public `ProxyRotator`) is shared with the YouTube/transcript and
chat features, so it stays Apache-2; only the crawl-ladder-coupled
rotation-retry orchestration (`web_crawler/connector.py::_run_tier_with_proxy_retry`)
lives here.

31
surfsense_backend/app/zero_publication.py
View file

@ -173,6 +173,37 @@ def apply_publication(conn: Connection) -> None:
conn.execute(text(build_set_table_sql(conn)))
def ensure_publication(conn: Connection) -> None:
"""Create ``zero_publication`` if missing, then reconcile if its shape drifted.
Startup-bootstrap counterpart of migration 116: databases created via
``Base.metadata.create_all`` (dev/test, ``DB_BOOTSTRAP_ON_STARTUP=TRUE``)
never run migrations, so without this zero-cache crash-loops on
``Unknown or invalid publications``. Idempotent: when the publication
already matches the canonical shape no DDL is emitted, so a normal boot
fires no event triggers and never disturbs a running zero-cache.
"""
exists = conn.execute(
text("SELECT 1 FROM pg_publication WHERE pubname = :name"),
{"name": PUBLICATION_NAME},
).fetchone()
if not exists:
# Seed with one table; the reconcile below sets the full canonical
# shape. CREATE PUBLICATION is safe here (unlike in migrations, see
# 116_create_zero_publication.py): the publication does not exist, so
# no zero-cache replica can be attached to it yet.
conn.execute(
text(
f"CREATE PUBLICATION {_quote_identifier(PUBLICATION_NAME)} "
"FOR TABLE notifications"
)
)
if verify_publication(conn):
conn.execute(text(build_set_table_sql(conn)))
def _actual_publication_shape(conn: Connection) -> dict[str, list[str] | None]:
rows = conn.execute(
text(

46
surfsense_backend/scripts/check_zero_publication_bootstrap.py Normal file
View file

@ -0,0 +1,46 @@
"""Self-check for ensure_publication on a create_all-bootstrapped scratch DB."""
import asyncio
import asyncpg
from sqlalchemy import text
from sqlalchemy.ext.asyncio import create_async_engine
SCRATCH_DB = "surfsense_zero_pub_check"
ADMIN_DSN = "postgresql://postgres:postgres@localhost:5432/postgres"
SCRATCH_URL = f"postgresql+asyncpg://postgres:postgres@localhost:5432/{SCRATCH_DB}"
async def main() -> None:
admin = await asyncpg.connect(ADMIN_DSN)
await admin.execute(f'DROP DATABASE IF EXISTS "{SCRATCH_DB}" WITH (FORCE)')
await admin.execute(f'CREATE DATABASE "{SCRATCH_DB}"')
await admin.close()
from app.db import Base
from app.zero_publication import ensure_publication, verify_publication
engine = create_async_engine(SCRATCH_URL)
try:
async with engine.begin() as conn:
await conn.execute(text("CREATE EXTENSION IF NOT EXISTS vector"))
await conn.execute(text("CREATE EXTENSION IF NOT EXISTS pg_trgm"))
await conn.run_sync(Base.metadata.create_all)
await conn.run_sync(ensure_publication)
mismatches = await conn.run_sync(verify_publication)
assert not mismatches, f"shape wrong after ensure: {mismatches}"
# Second call must be a no-op that leaves a verified shape.
await conn.run_sync(ensure_publication)
mismatches = await conn.run_sync(verify_publication)
assert not mismatches, f"shape wrong after re-ensure: {mismatches}"
finally:
await engine.dispose()
admin = await asyncpg.connect(ADMIN_DSN)
await admin.execute(f'DROP DATABASE IF EXISTS "{SCRATCH_DB}" WITH (FORCE)')
await admin.close()
print("OK: ensure_publication creates and verifies on a create_all DB, idempotently.")
asyncio.run(main())

2
surfsense_web/app/dashboard/[workspace_id]/automations/components/builder/automation-builder-form.tsx
View file

@ -156,7 +156,7 @@ export function AutomationBuilderForm({
if (mode === "edit" && automation) {
return { ...buildUpdatePayload(formForPayload), status: automation.status };
}
const { search_space_id: _ignored, ...rest } = buildCreatePayload(formForPayload, workspaceId);
const { workspace_id: _ignored, ...rest } = buildCreatePayload(formForPayload, workspaceId);
return rest;
}

2
surfsense_web/app/dashboard/[workspace_id]/automations/components/builder/automation-model-fields.tsx
View file

@ -51,7 +51,7 @@ export function AutomationModelFields({
errors,
}: AutomationModelFieldsProps) {
const { llm, image, vision, isLoading } = useAutomationEligibleModels();
const rolesHref = `/dashboard/${workspaceId}/search-space-settings/models`;
const rolesHref = `/dashboard/${workspaceId}/workspace-settings/models`;
return (
<div className="flex flex-col gap-4">

2
surfsense_web/app/dashboard/[workspace_id]/client-layout.tsx
View file

@ -13,7 +13,7 @@ import {
modelConnectionsAtom,
modelRolesAtom,
} from "@/atoms/model-connections/model-connections-query.atoms";
import { activeWorkspaceIdAtom } from "@/atoms/search-spaces/search-space-query.atoms";
import { activeWorkspaceIdAtom } from "@/atoms/workspaces/workspace-query.atoms";
import { DocumentUploadDialogProvider } from "@/components/assistant-ui/document-upload-popup";
import { LayoutDataProvider } from "@/components/layout";
import { OnboardingTour } from "@/components/onboarding-tour";

6
surfsense_web/app/dashboard/[workspace_id]/new-chat/[[...chat_id]]/page.tsx
View file

@ -754,8 +754,8 @@ export default function NewChatPage() {
syncChatTab({
chatId: thread.id,
title: thread.title,
chatUrl: `/dashboard/${thread.search_space_id ?? workspaceId}/new-chat/${thread.id}`,
searchSpaceId: thread.search_space_id ?? workspaceId,
chatUrl: `/dashboard/${thread.workspace_id ?? workspaceId}/new-chat/${thread.id}`,
searchSpaceId: thread.workspace_id ?? workspaceId,
visibility: thread.visibility,
hasComments: thread.has_comments ?? false,
});
@ -906,7 +906,7 @@ export default function NewChatPage() {
setCurrentThreadMetadata({
id: currentThread.id,
searchSpaceId: currentThread.search_space_id ?? workspaceId,
searchSpaceId: currentThread.workspace_id ?? workspaceId,
visibility,
hasComments: currentThread.has_comments ?? false,
});

20
surfsense_web/app/dashboard/[workspace_id]/team/team-content.tsx
View file

@ -139,7 +139,7 @@ export function TeamContent({ workspaceId }: TeamContentProps) {
const handleRevokeInvite = useCallback(
async (inviteId: number): Promise<boolean> => {
const request: DeleteInviteRequest = {
search_space_id: workspaceId,
workspace_id: workspaceId,
invite_id: inviteId,
};
await revokeInvite(request);
@ -151,7 +151,7 @@ export function TeamContent({ workspaceId }: TeamContentProps) {
const handleCreateInvite = useCallback(
async (inviteData: CreateInviteRequest["data"]) => {
const request: CreateInviteRequest = {
search_space_id: workspaceId,
workspace_id: workspaceId,
data: inviteData,
};
return await createInvite(request);
@ -162,7 +162,7 @@ export function TeamContent({ workspaceId }: TeamContentProps) {
const handleUpdateMember = useCallback(
async (membershipId: number, roleId: number | null): Promise<Membership> => {
const request: UpdateMembershipRequest = {
search_space_id: workspaceId,
workspace_id: workspaceId,
membership_id: membershipId,
data: { role_id: roleId },
};
@ -174,7 +174,7 @@ export function TeamContent({ workspaceId }: TeamContentProps) {
const handleRemoveMember = useCallback(
async (membershipId: number) => {
const request: DeleteMembershipRequest = {
search_space_id: workspaceId,
workspace_id: workspaceId,
membership_id: membershipId,
};
await deleteMember(request);
@ -185,13 +185,13 @@ export function TeamContent({ workspaceId }: TeamContentProps) {
const { data: roles = [], isLoading: rolesLoading } = useQuery({
queryKey: cacheKeys.roles.all(workspaceId.toString()),
queryFn: () => rolesApiService.getRoles({ search_space_id: workspaceId }),
queryFn: () => rolesApiService.getRoles({ workspace_id: workspaceId }),
enabled: !!workspaceId,
});
const { data: invites = [], isLoading: invitesLoading } = useQuery({
queryKey: cacheKeys.invites.all(workspaceId.toString()),
queryFn: () => invitesApiService.getInvites({ search_space_id: workspaceId }),
queryFn: () => invitesApiService.getInvites({ workspace_id: workspaceId }),
staleTime: 5 * 60 * 1000,
});
@ -581,7 +581,7 @@ function MemberRow({
<AlertDialogTitle>Remove member?</AlertDialogTitle>
<AlertDialogDescription>
This will remove <span className="font-medium">{member.user_email}</span>{" "}
from this search space. They will lose access to all resources.
from this workspace. They will lose access to all resources.
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
@ -599,7 +599,7 @@ function MemberRow({
<DropdownMenuSeparator className="bg-popover-border" />
<DropdownMenuItem
onClick={() =>
router.push(`/dashboard/${workspaceId}/search-space-settings/team-roles`)
router.push(`/dashboard/${workspaceId}/workspace-settings/team-roles`)
}
>
Manage Roles
@ -709,7 +709,7 @@ function CreateInviteDialog({
Invite Created!
</DialogTitle>
<DialogDescription>
Share this link to invite people to your search space.
Share this link to invite people to your workspace.
</DialogDescription>
</DialogHeader>
<div className="space-y-3 py-2 md:py-4">
@ -748,7 +748,7 @@ function CreateInviteDialog({
<DialogHeader>
<DialogTitle>Invite Members</DialogTitle>
<DialogDescription>
Create a link to invite people to this search space.
Create a link to invite people to this workspace.
</DialogDescription>
</DialogHeader>
<div className="space-y-3 py-2 md:py-4">

2
surfsense_web/app/dashboard/[workspace_id]/user-settings/components/AgentPermissionsContent.tsx
View file

@ -6,7 +6,7 @@ import { AlertTriangle, Info, ShieldCheck, Trash2 } from "lucide-react";
import { useCallback, useMemo, useState } from "react";
import { toast } from "sonner";
import { agentFlagsAtom } from "@/atoms/agent/agent-flags-query.atom";
import { activeWorkspaceIdAtom } from "@/atoms/search-spaces/search-space-query.atoms";
import { activeWorkspaceIdAtom } from "@/atoms/workspaces/workspace-query.atoms";
import { Alert, AlertDescription, AlertTitle } from "@/components/ui/alert";
import {
AlertDialog,

4
surfsense_web/app/dashboard/[workspace_id]/user-settings/components/DesktopContent.tsx
View file

@ -13,9 +13,9 @@ import {
import { Separator } from "@/components/ui/separator";
import { Skeleton } from "@/components/ui/skeleton";
import { Switch } from "@/components/ui/switch";
import type { SearchSpace } from "@/contracts/types/search-space.types";
import type { SearchSpace } from "@/contracts/types/workspace.types";
import { useElectronAPI } from "@/hooks/use-platform";
import { searchSpacesApiService } from "@/lib/apis/search-spaces-api.service";
import { searchSpacesApiService } from "@/lib/apis/workspaces-api.service";
export function DesktopContent() {
const api = useElectronAPI();

4
surfsense_web/app/dashboard/[workspace_id]/user-settings/components/MessagingChannelsContent.tsx
View file

@ -17,8 +17,8 @@ import {
} from "@/components/ui/select";
import { Separator } from "@/components/ui/separator";
import { Skeleton } from "@/components/ui/skeleton";
import type { SearchSpace } from "@/contracts/types/search-space.types";
import { searchSpacesApiService } from "@/lib/apis/search-spaces-api.service";
import type { SearchSpace } from "@/contracts/types/workspace.types";
import { searchSpacesApiService } from "@/lib/apis/workspaces-api.service";
import { authenticatedFetch } from "@/lib/auth-fetch";
import { buildBackendUrl } from "@/lib/env-config";
import { cn } from "@/lib/utils";

4
surfsense_web/app/dashboard/[workspace_id]/user-settings/layout.tsx
View file

@ -11,7 +11,5 @@ export default function UserSettingsLayout({
}) {
const { workspace_id } = use(params);
return (
<UserSettingsLayoutShell workspaceId={workspace_id}>{children}</UserSettingsLayoutShell>
);
return <UserSettingsLayoutShell workspaceId={workspace_id}>{children}</UserSettingsLayoutShell>;
}

0
surfsense_web/app/dashboard/[workspace_id]/search-space-settings/general/page.tsx → surfsense_web/app/dashboard/[workspace_id]/workspace-settings/general/page.tsx
View file

2
surfsense_web/app/dashboard/[workspace_id]/search-space-settings/layout-shell.tsx → surfsense_web/app/dashboard/[workspace_id]/workspace-settings/layout-shell.tsx
View file

@ -76,7 +76,7 @@ export function SearchSpaceSettingsLayoutShell({
const selectedLabel = navItems.find((item) => item.value === activeTab)?.label ?? t("title");
const hrefFor = (tab: SearchSpaceSettingsTab) =>
`/dashboard/${workspaceId}/search-space-settings/${tab}`;
`/dashboard/${workspaceId}/workspace-settings/${tab}`;
return (
<section className="flex h-full min-h-[min(680px,calc(100vh-5rem))] w-full select-none flex-col gap-6 md:pt-6 md:flex-row">

0
surfsense_web/app/dashboard/[workspace_id]/search-space-settings/layout.tsx → surfsense_web/app/dashboard/[workspace_id]/workspace-settings/layout.tsx
View file

0
surfsense_web/app/dashboard/[workspace_id]/search-space-settings/models/page.tsx → surfsense_web/app/dashboard/[workspace_id]/workspace-settings/models/page.tsx
View file

2
surfsense_web/app/dashboard/[workspace_id]/search-space-settings/page.tsx → surfsense_web/app/dashboard/[workspace_id]/workspace-settings/page.tsx
View file

@ -6,5 +6,5 @@ export default async function SearchSpaceSettingsPage({
params: Promise<{ workspace_id: string }>;
}) {
const { workspace_id } = await params;
redirect(`/dashboard/${workspace_id}/search-space-settings/general`);
redirect(`/dashboard/${workspace_id}/workspace-settings/general`);
}

0
surfsense_web/app/dashboard/[workspace_id]/search-space-settings/prompts/page.tsx → surfsense_web/app/dashboard/[workspace_id]/workspace-settings/prompts/page.tsx
View file

0
surfsense_web/app/dashboard/[workspace_id]/search-space-settings/public-links/page.tsx → surfsense_web/app/dashboard/[workspace_id]/workspace-settings/public-links/page.tsx
View file

0
surfsense_web/app/dashboard/[workspace_id]/search-space-settings/team-roles/page.tsx → surfsense_web/app/dashboard/[workspace_id]/workspace-settings/team-roles/page.tsx
View file

2
surfsense_web/app/dashboard/page.tsx
View file

@ -6,7 +6,7 @@ import { motion } from "motion/react";
import { useRouter } from "next/navigation";
import { useTranslations } from "next-intl";
import { useEffect, useState } from "react";
import { searchSpacesAtom } from "@/atoms/search-spaces/search-space-query.atoms";
import { searchSpacesAtom } from "@/atoms/workspaces/workspace-query.atoms";
import { CreateSearchSpaceDialog } from "@/components/layout";
import { Button } from "@/components/ui/button";
import { Card, CardDescription, CardFooter, CardHeader, CardTitle } from "@/components/ui/card";

2
surfsense_web/app/desktop/login/page.tsx
View file

@ -14,7 +14,7 @@ import { Separator } from "@/components/ui/separator";
import { ShortcutKbd } from "@/components/ui/shortcut-kbd";
import { Spinner } from "@/components/ui/spinner";
import { useElectronAPI } from "@/hooks/use-platform";
import { searchSpacesApiService } from "@/lib/apis/search-spaces-api.service";
import { searchSpacesApiService } from "@/lib/apis/workspaces-api.service";
import { getPostLoginRedirectPath } from "@/lib/auth-utils";
type ShortcutKey = "generalAssist" | "quickAsk" | "screenshotAssist";

26
surfsense_web/app/invite/[invite_code]/page.tsx
View file

@ -98,15 +98,11 @@ export default function InviteAcceptPage() {
// Track invite accepted and user added events
trackSearchSpaceInviteAccepted(
result.search_space_id,
result.search_space_name,
result.role_name
);
trackSearchSpaceUserAdded(
result.search_space_id,
result.search_space_name,
result.workspace_id,
result.workspace_name,
result.role_name
);
trackSearchSpaceUserAdded(result.workspace_id, result.workspace_name, result.role_name);
}
} catch (err: any) {
setError(err.message || "Failed to accept invite");
@ -117,7 +113,7 @@ export default function InviteAcceptPage() {
const handleDecline = () => {
// Track invite declined event
trackSearchSpaceInviteDeclined(inviteInfo?.search_space_name);
trackSearchSpaceInviteDeclined(inviteInfo?.workspace_name);
router.push("/dashboard");
};
@ -180,7 +176,7 @@ export default function InviteAcceptPage() {
</motion.div>
<CardTitle className="text-2xl">Welcome to the team!</CardTitle>
<CardDescription>
You've successfully joined {acceptedData.search_space_name}
You've successfully joined {acceptedData.workspace_name}
</CardDescription>
</CardHeader>
<CardContent className="space-y-4">
@ -190,7 +186,7 @@ export default function InviteAcceptPage() {
<Users className="h-5 w-5 text-primary" />
</div>
<div>
<p className="font-medium">{acceptedData.search_space_name}</p>
<p className="font-medium">{acceptedData.workspace_name}</p>
<p className="text-sm text-muted-foreground">Search Space</p>
</div>
</div>
@ -208,7 +204,7 @@ export default function InviteAcceptPage() {
<CardFooter>
<Button
className="w-full gap-2"
onClick={() => router.push(`/dashboard/${acceptedData.search_space_id}`)}
onClick={() => router.push(`/dashboard/${acceptedData.workspace_id}`)}
>
Go to Search Space
<ArrowRight className="h-4 w-4" />
@ -260,7 +256,7 @@ export default function InviteAcceptPage() {
</motion.div>
<CardTitle className="text-2xl">You're Invited!</CardTitle>
<CardDescription>
Sign in to join {inviteInfo?.search_space_name || "this search space"}
Sign in to join {inviteInfo?.workspace_name || "this search space"}
</CardDescription>
</CardHeader>
<CardContent className="space-y-4">
@ -270,7 +266,7 @@ export default function InviteAcceptPage() {
<Users className="h-5 w-5 text-primary" />
</div>
<div>
<p className="font-medium">{inviteInfo?.search_space_name}</p>
<p className="font-medium">{inviteInfo?.workspace_name}</p>
<p className="text-sm text-muted-foreground">Search Space</p>
</div>
</div>
@ -307,7 +303,7 @@ export default function InviteAcceptPage() {
</motion.div>
<CardTitle className="text-2xl">You're Invited!</CardTitle>
<CardDescription>
Accept this invite to join {inviteInfo?.search_space_name || "this search space"}
Accept this invite to join {inviteInfo?.workspace_name || "this search space"}
</CardDescription>
</CardHeader>
<CardContent className="space-y-4">
@ -317,7 +313,7 @@ export default function InviteAcceptPage() {
<Users className="h-5 w-5 text-primary" />
</div>
<div>
<p className="font-medium">{inviteInfo?.search_space_name}</p>
<p className="font-medium">{inviteInfo?.workspace_name}</p>
<p className="text-sm text-muted-foreground">Search Space</p>
</div>
</div>

10
surfsense_web/atoms/agent-tools/agent-tools.atoms.ts
View file

@ -2,7 +2,7 @@ import { atom } from "jotai";
import { atomWithQuery } from "jotai-tanstack-query";
import { agentToolsApiService } from "@/lib/apis/agent-tools-api.service";
import { cacheKeys } from "@/lib/query-client/cache-keys";
import { activeSearchSpaceIdAtom } from "../search-spaces/search-space-query.atoms";
import { activeWorkspaceIdAtom } from "../workspaces/workspace-query.atoms";
export const agentToolsAtom = atomWithQuery((_get) => ({
queryKey: cacheKeys.agentTools.all(),
@ -42,7 +42,7 @@ const hydratedForAtom = atom<string | null>(null);
*/
export const disabledToolsAtom = atom(
(get) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
const hydratedFor = get(hydratedForAtom);
if (searchSpaceId && hydratedFor !== searchSpaceId) {
return loadDisabledTools(searchSpaceId);
@ -50,7 +50,7 @@ export const disabledToolsAtom = atom(
return get(disabledToolsBaseAtom);
},
(get, set, update: string[] | ((prev: string[]) => string[])) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
const prev = get(disabledToolsBaseAtom);
const next = typeof update === "function" ? update(prev) : update;
set(disabledToolsBaseAtom, next);
@ -66,7 +66,7 @@ export const disabledToolsAtom = atom(
* Call this from a useEffect in a component that has access to the search space.
*/
export const hydrateDisabledToolsAtom = atom(null, (get, set) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
if (!searchSpaceId) return;
const stored = loadDisabledTools(searchSpaceId);
set(disabledToolsBaseAtom, stored);
@ -75,7 +75,7 @@ export const hydrateDisabledToolsAtom = atom(null, (get, set) => {
/** Toggle a single tool's enabled/disabled state */
export const toggleToolAtom = atom(null, (get, set, toolName: string) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
const current = get(disabledToolsBaseAtom);
const next = current.includes(toolName)
? current.filter((t) => t !== toolName)

14
surfsense_web/atoms/automations/automations-mutation.atoms.ts
View file

@ -49,10 +49,10 @@ export const createAutomationMutationAtom = atomWithMutation(() => ({
return automationsApiService.createAutomation(request);
},
onSuccess: (automation, variables) => {
invalidateList(variables.search_space_id);
invalidateList(variables.workspace_id);
toast.success("Automation created");
trackAutomationCreated({
search_space_id: variables.search_space_id,
workspace_id: variables.workspace_id,
automation_id: automation.id,
task_count: variables.definition.plan.length,
trigger_type: variables.triggers?.[0]?.type ?? "none",
@ -67,7 +67,7 @@ export const createAutomationMutationAtom = atomWithMutation(() => ({
console.error("Error creating automation:", error);
toast.error("Failed to create automation");
trackAutomationCreateFailed({
search_space_id: variables.search_space_id,
workspace_id: variables.workspace_id,
error: error.message,
});
},
@ -80,20 +80,20 @@ export const updateAutomationMutationAtom = atomWithMutation(() => ({
},
onSuccess: (automation, vars) => {
invalidateDetail(vars.automationId);
invalidateList(automation.search_space_id);
invalidateList(automation.workspace_id);
toast.success("Automation updated");
// A status-only patch (pause/resume/archive) is a distinct action from a
// definition/name edit, so split it into its own event.
if (vars.patch.status && !vars.patch.definition) {
trackAutomationStatusChanged({
automation_id: vars.automationId,
search_space_id: automation.search_space_id,
workspace_id: automation.workspace_id,
next_status: vars.patch.status,
});
} else {
trackAutomationUpdated({
automation_id: vars.automationId,
search_space_id: automation.search_space_id,
workspace_id: automation.workspace_id,
has_definition_change: !!vars.patch.definition,
has_name_change: vars.patch.name != null,
has_description_change: vars.patch.description !== undefined,
@ -123,7 +123,7 @@ export const deleteAutomationMutationAtom = atomWithMutation(() => ({
toast.success("Automation deleted");
trackAutomationDeleted({
automation_id: vars.automationId,
search_space_id: vars.searchSpaceId,
workspace_id: vars.searchSpaceId,
});
},
onError: (error: Error, vars) => {

6
surfsense_web/atoms/automations/automations-query.atoms.ts
View file

@ -1,5 +1,5 @@
import { atomWithQuery } from "jotai-tanstack-query";
import { activeSearchSpaceIdAtom } from "@/atoms/search-spaces/search-space-query.atoms";
import { activeWorkspaceIdAtom } from "@/atoms/workspaces/workspace-query.atoms";
import { automationsApiService } from "@/lib/apis/automations-api.service";
import { cacheKeys } from "@/lib/query-client/cache-keys";
@ -11,7 +11,7 @@ const DEFAULT_LIMIT = 50;
const DEFAULT_OFFSET = 0;
export const automationsListAtom = atomWithQuery((get) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
return {
queryKey: cacheKeys.automations.list(Number(searchSpaceId ?? 0), DEFAULT_LIMIT, DEFAULT_OFFSET),
@ -22,7 +22,7 @@ export const automationsListAtom = atomWithQuery((get) => {
return { items: [], total: 0 };
}
return automationsApiService.listAutomations({
search_space_id: Number(searchSpaceId),
workspace_id: Number(searchSpaceId),
limit: DEFAULT_LIMIT,
offset: DEFAULT_OFFSET,
});

10
surfsense_web/atoms/connectors/connector-mutation.atoms.ts
View file

@ -10,10 +10,10 @@ import type {
import { connectorsApiService } from "@/lib/apis/connectors-api.service";
import { cacheKeys } from "@/lib/query-client/cache-keys";
import { queryClient } from "@/lib/query-client/client";
import { activeSearchSpaceIdAtom } from "../search-spaces/search-space-query.atoms";
import { activeWorkspaceIdAtom } from "../workspaces/workspace-query.atoms";
export const createConnectorMutationAtom = atomWithMutation((get) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
return {
mutationKey: cacheKeys.connectors.all(searchSpaceId ?? ""),
@ -32,7 +32,7 @@ export const createConnectorMutationAtom = atomWithMutation((get) => {
});
export const updateConnectorMutationAtom = atomWithMutation((get) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
return {
mutationKey: cacheKeys.connectors.all(searchSpaceId ?? ""),
@ -54,7 +54,7 @@ export const updateConnectorMutationAtom = atomWithMutation((get) => {
});
export const deleteConnectorMutationAtom = atomWithMutation((get) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
return {
mutationKey: cacheKeys.connectors.all(searchSpaceId ?? ""),
@ -80,7 +80,7 @@ export const deleteConnectorMutationAtom = atomWithMutation((get) => {
});
export const indexConnectorMutationAtom = atomWithMutation((get) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
return {
mutationKey: cacheKeys.connectors.index(),

6
surfsense_web/atoms/connectors/connector-query.atoms.ts
View file

@ -1,10 +1,10 @@
import { atomWithQuery } from "jotai-tanstack-query";
import { connectorsApiService } from "@/lib/apis/connectors-api.service";
import { cacheKeys } from "@/lib/query-client/cache-keys";
import { activeSearchSpaceIdAtom } from "../search-spaces/search-space-query.atoms";
import { activeWorkspaceIdAtom } from "../workspaces/workspace-query.atoms";
export const connectorsAtom = atomWithQuery((get) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
return {
queryKey: cacheKeys.connectors.all(searchSpaceId!),
@ -13,7 +13,7 @@ export const connectorsAtom = atomWithQuery((get) => {
queryFn: async () => {
return connectorsApiService.getConnectors({
queryParams: {
search_space_id: searchSpaceId!,
workspace_id: searchSpaceId!,
},
});
},

10
surfsense_web/atoms/documents/document-mutation.atoms.ts
View file

@ -1,6 +1,6 @@
import { atomWithMutation } from "jotai-tanstack-query";
import { toast } from "sonner";
import { activeSearchSpaceIdAtom } from "@/atoms/search-spaces/search-space-query.atoms";
import { activeWorkspaceIdAtom } from "@/atoms/workspaces/workspace-query.atoms";
import type {
CreateDocumentRequest,
DeleteDocumentRequest,
@ -14,7 +14,7 @@ import { queryClient } from "@/lib/query-client/client";
import { globalDocumentsQueryParamsAtom } from "./ui.atoms";
export const createDocumentMutationAtom = atomWithMutation((get) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
const documentsQueryParams = get(globalDocumentsQueryParamsAtom);
return {
@ -34,7 +34,7 @@ export const createDocumentMutationAtom = atomWithMutation((get) => {
});
export const uploadDocumentMutationAtom = atomWithMutation((get) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
const documentsQueryParams = get(globalDocumentsQueryParamsAtom);
return {
@ -54,7 +54,7 @@ export const uploadDocumentMutationAtom = atomWithMutation((get) => {
});
export const updateDocumentMutationAtom = atomWithMutation((get) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
const documentsQueryParams = get(globalDocumentsQueryParamsAtom);
return {
@ -77,7 +77,7 @@ export const updateDocumentMutationAtom = atomWithMutation((get) => {
});
export const deleteDocumentMutationAtom = atomWithMutation((get) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
const documentsQueryParams = get(globalDocumentsQueryParamsAtom);
return {

6
surfsense_web/atoms/invites/invites-mutation.atoms.ts
View file

@ -20,7 +20,7 @@ export const createInviteMutationAtom = atomWithMutation(() => ({
},
onSuccess: (_, variables) => {
queryClient.invalidateQueries({
queryKey: cacheKeys.invites.all(variables.search_space_id.toString()),
queryKey: cacheKeys.invites.all(variables.workspace_id.toString()),
});
toast.success("Invite created successfully");
},
@ -40,7 +40,7 @@ export const updateInviteMutationAtom = atomWithMutation(() => ({
},
onSuccess: (_, variables) => {
queryClient.invalidateQueries({
queryKey: cacheKeys.invites.all(variables.search_space_id.toString()),
queryKey: cacheKeys.invites.all(variables.workspace_id.toString()),
});
toast.success("Invite updated successfully");
},
@ -60,7 +60,7 @@ export const deleteInviteMutationAtom = atomWithMutation(() => ({
},
onSuccess: (_, variables) => {
queryClient.invalidateQueries({
queryKey: cacheKeys.invites.all(variables.search_space_id.toString()),
queryKey: cacheKeys.invites.all(variables.workspace_id.toString()),
});
toast.success("Invite deleted successfully");
},

6
surfsense_web/atoms/invites/invites-query.atoms.ts
View file

@ -1,10 +1,10 @@
import { atomWithQuery } from "jotai-tanstack-query";
import { activeSearchSpaceIdAtom } from "@/atoms/search-spaces/search-space-query.atoms";
import { activeWorkspaceIdAtom } from "@/atoms/workspaces/workspace-query.atoms";
import { invitesApiService } from "@/lib/apis/invites-api.service";
import { cacheKeys } from "@/lib/query-client/cache-keys";
export const invitesAtom = atomWithQuery((get) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
return {
queryKey: cacheKeys.invites.all(searchSpaceId?.toString() ?? ""),
@ -15,7 +15,7 @@ export const invitesAtom = atomWithQuery((get) => {
return [];
}
return invitesApiService.getInvites({
search_space_id: Number(searchSpaceId),
workspace_id: Number(searchSpaceId),
});
},
};

8
surfsense_web/atoms/logs/log-mutation.atoms.ts
View file

@ -1,5 +1,5 @@
import { atomWithMutation } from "jotai-tanstack-query";
import { activeSearchSpaceIdAtom } from "@/atoms/search-spaces/search-space-query.atoms";
import { activeWorkspaceIdAtom } from "@/atoms/workspaces/workspace-query.atoms";
import type {
CreateLogRequest,
DeleteLogRequest,
@ -13,7 +13,7 @@ import { queryClient } from "@/lib/query-client/client";
* Create Log Mutation
*/
export const createLogMutationAtom = atomWithMutation((get) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
return {
mutationKey: cacheKeys.logs.list(searchSpaceId ?? undefined),
enabled: !!searchSpaceId,
@ -29,7 +29,7 @@ export const createLogMutationAtom = atomWithMutation((get) => {
* Update Log Mutation
*/
export const updateLogMutationAtom = atomWithMutation((get) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
return {
mutationKey: cacheKeys.logs.list(searchSpaceId ?? undefined),
enabled: !!searchSpaceId,
@ -45,7 +45,7 @@ export const updateLogMutationAtom = atomWithMutation((get) => {
* Delete Log Mutation
*/
export const deleteLogMutationAtom = atomWithMutation((get) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
return {
mutationKey: cacheKeys.logs.list(searchSpaceId ?? undefined),
enabled: !!searchSpaceId,

6
surfsense_web/atoms/members/members-mutation.atoms.ts
View file

@ -21,7 +21,7 @@ export const updateMemberMutationAtom = atomWithMutation(() => {
onSuccess: (_: UpdateMembershipResponse, request: UpdateMembershipRequest) => {
toast.success("Member updated successfully");
queryClient.invalidateQueries({
queryKey: cacheKeys.members.all(request.search_space_id.toString()),
queryKey: cacheKeys.members.all(request.workspace_id.toString()),
});
},
onError: () => {
@ -39,7 +39,7 @@ export const deleteMemberMutationAtom = atomWithMutation(() => {
onSuccess: (_: DeleteMembershipResponse, request: DeleteMembershipRequest) => {
toast.success("Member removed successfully");
queryClient.invalidateQueries({
queryKey: cacheKeys.members.all(request.search_space_id.toString()),
queryKey: cacheKeys.members.all(request.workspace_id.toString()),
});
},
onError: () => {
@ -57,7 +57,7 @@ export const leaveSearchSpaceMutationAtom = atomWithMutation(() => {
onSuccess: (_: LeaveSearchSpaceResponse, request: LeaveSearchSpaceRequest) => {
toast.success("Successfully left the search space");
queryClient.invalidateQueries({
queryKey: cacheKeys.members.all(request.search_space_id.toString()),
queryKey: cacheKeys.members.all(request.workspace_id.toString()),
});
},
onError: () => {

10
surfsense_web/atoms/members/members-query.atoms.ts
View file

@ -1,11 +1,11 @@
import { useAtomValue } from "jotai";
import { atomWithQuery } from "jotai-tanstack-query";
import { activeSearchSpaceIdAtom } from "@/atoms/search-spaces/search-space-query.atoms";
import { activeWorkspaceIdAtom } from "@/atoms/workspaces/workspace-query.atoms";
import { membersApiService } from "@/lib/apis/members-api.service";
import { cacheKeys } from "@/lib/query-client/cache-keys";
export const membersAtom = atomWithQuery((get) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
return {
queryKey: cacheKeys.members.all(searchSpaceId?.toString() ?? ""),
@ -17,14 +17,14 @@ export const membersAtom = atomWithQuery((get) => {
return [];
}
return membersApiService.getMembers({
search_space_id: Number(searchSpaceId),
workspace_id: Number(searchSpaceId),
});
},
};
});
export const myAccessAtom = atomWithQuery((get) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
return {
queryKey: cacheKeys.members.myAccess(searchSpaceId?.toString() ?? ""),
@ -35,7 +35,7 @@ export const myAccessAtom = atomWithQuery((get) => {
return null;
}
return membersApiService.getMyAccess({
search_space_id: Number(searchSpaceId),
workspace_id: Number(searchSpaceId),
});
},
};

24
surfsense_web/atoms/model-connections/model-connections-mutation.atoms.ts
View file

@ -16,7 +16,7 @@ import type {
import { modelConnectionsApiService } from "@/lib/apis/model-connections-api.service";
import { cacheKeys } from "@/lib/query-client/cache-keys";
import { queryClient } from "@/lib/query-client/client";
import { activeSearchSpaceIdAtom } from "../search-spaces/search-space-query.atoms";
import { activeWorkspaceIdAtom } from "../workspaces/workspace-query.atoms";
function invalidateModelConnections(searchSpaceId: number) {
queryClient.invalidateQueries({
@ -40,14 +40,14 @@ function upsertModelConnection(searchSpaceId: number, connection: ConnectionRead
}
export const createModelConnectionMutationAtom = atomWithMutation((get) => {
const searchSpaceId = Number(get(activeSearchSpaceIdAtom));
const searchSpaceId = Number(get(activeWorkspaceIdAtom));
return {
mutationKey: ["model-connections", "create"],
mutationFn: (request: ConnectionCreateRequest) =>
modelConnectionsApiService.createConnection(request),
onSuccess: (connection: ConnectionRead, request: ConnectionCreateRequest) => {
const resolvedSearchSpaceId = Number(
request.search_space_id ?? connection.search_space_id ?? searchSpaceId
request.workspace_id ?? connection.workspace_id ?? searchSpaceId
);
toast.success("Connection created");
if (resolvedSearchSpaceId > 0) {
@ -60,7 +60,7 @@ export const createModelConnectionMutationAtom = atomWithMutation((get) => {
});
export const updateModelConnectionMutationAtom = atomWithMutation((get) => {
const searchSpaceId = Number(get(activeSearchSpaceIdAtom));
const searchSpaceId = Number(get(activeWorkspaceIdAtom));
return {
mutationKey: ["model-connections", "update"],
mutationFn: ({ id, data }: { id: number; data: ConnectionUpdateRequest }) =>
@ -74,7 +74,7 @@ export const updateModelConnectionMutationAtom = atomWithMutation((get) => {
});
export const deleteModelConnectionMutationAtom = atomWithMutation((get) => {
const searchSpaceId = Number(get(activeSearchSpaceIdAtom));
const searchSpaceId = Number(get(activeWorkspaceIdAtom));
return {
mutationKey: ["model-connections", "delete"],
mutationFn: (id: number) => modelConnectionsApiService.deleteConnection(id),
@ -87,7 +87,7 @@ export const deleteModelConnectionMutationAtom = atomWithMutation((get) => {
});
export const verifyModelConnectionMutationAtom = atomWithMutation((get) => {
const searchSpaceId = Number(get(activeSearchSpaceIdAtom));
const searchSpaceId = Number(get(activeWorkspaceIdAtom));
return {
mutationKey: ["model-connections", "verify"],
mutationFn: (id: number) => modelConnectionsApiService.verifyConnection(id),
@ -110,7 +110,7 @@ export const verifyModelConnectionMutationAtom = atomWithMutation((get) => {
});
export const discoverConnectionModelsMutationAtom = atomWithMutation((get) => {
const searchSpaceId = Number(get(activeSearchSpaceIdAtom));
const searchSpaceId = Number(get(activeWorkspaceIdAtom));
return {
mutationKey: ["model-connections", "discover"],
mutationFn: (id: number) => modelConnectionsApiService.discoverModels(id),
@ -149,7 +149,7 @@ export const testPreviewModelMutationAtom = atomWithMutation(() => {
});
export const addManualModelMutationAtom = atomWithMutation((get) => {
const searchSpaceId = Number(get(activeSearchSpaceIdAtom));
const searchSpaceId = Number(get(activeWorkspaceIdAtom));
return {
mutationKey: ["models", "add-manual"],
mutationFn: ({ connectionId, data }: { connectionId: number; data: ModelCreateRequest }) =>
@ -163,7 +163,7 @@ export const addManualModelMutationAtom = atomWithMutation((get) => {
});
export const updateModelMutationAtom = atomWithMutation((get) => {
const searchSpaceId = Number(get(activeSearchSpaceIdAtom));
const searchSpaceId = Number(get(activeWorkspaceIdAtom));
return {
mutationKey: ["models", "update"],
mutationFn: ({ id, data }: { id: number; data: ModelUpdateRequest }) =>
@ -174,7 +174,7 @@ export const updateModelMutationAtom = atomWithMutation((get) => {
});
export const bulkUpdateModelsMutationAtom = atomWithMutation((get) => {
const searchSpaceId = Number(get(activeSearchSpaceIdAtom));
const searchSpaceId = Number(get(activeWorkspaceIdAtom));
return {
mutationKey: ["models", "bulk-update"],
mutationFn: ({ connectionId, data }: { connectionId: number; data: ModelsBulkUpdateRequest }) =>
@ -185,7 +185,7 @@ export const bulkUpdateModelsMutationAtom = atomWithMutation((get) => {
});
export const testModelMutationAtom = atomWithMutation((get) => {
const searchSpaceId = Number(get(activeSearchSpaceIdAtom));
const searchSpaceId = Number(get(activeWorkspaceIdAtom));
return {
mutationKey: ["models", "test"],
mutationFn: (id: number) => modelConnectionsApiService.testModel(id),
@ -199,7 +199,7 @@ export const testModelMutationAtom = atomWithMutation((get) => {
});
export const updateModelRolesMutationAtom = atomWithMutation((get) => {
const searchSpaceId = Number(get(activeSearchSpaceIdAtom));
const searchSpaceId = Number(get(activeWorkspaceIdAtom));
return {
mutationKey: ["model-roles", "update"],
mutationFn: (roles: ModelRoles) =>

6
surfsense_web/atoms/model-connections/model-connections-query.atoms.ts
View file

@ -2,7 +2,7 @@ import { atomWithQuery } from "jotai-tanstack-query";
import { modelConnectionsApiService } from "@/lib/apis/model-connections-api.service";
import { isAuthenticated } from "@/lib/auth-utils";
import { cacheKeys } from "@/lib/query-client/cache-keys";
import { activeSearchSpaceIdAtom } from "../search-spaces/search-space-query.atoms";
import { activeWorkspaceIdAtom } from "../workspaces/workspace-query.atoms";
export const globalModelConnectionsAtom = atomWithQuery(() => ({
queryKey: cacheKeys.modelConnections.global(),
@ -26,7 +26,7 @@ export const modelProvidersAtom = atomWithQuery(() => ({
}));
export const modelConnectionsAtom = atomWithQuery((get) => {
const searchSpaceId = Number(get(activeSearchSpaceIdAtom));
const searchSpaceId = Number(get(activeWorkspaceIdAtom));
return {
queryKey: cacheKeys.modelConnections.all(searchSpaceId),
enabled: !!searchSpaceId,
@ -36,7 +36,7 @@ export const modelConnectionsAtom = atomWithQuery((get) => {
});
export const modelRolesAtom = atomWithQuery((get) => {
const searchSpaceId = Number(get(activeSearchSpaceIdAtom));
const searchSpaceId = Number(get(activeWorkspaceIdAtom));
return {
queryKey: cacheKeys.modelConnections.roles(searchSpaceId),
enabled: !!searchSpaceId,

6
surfsense_web/atoms/public-chat-snapshots/public-chat-snapshots-query.atoms.ts
View file

@ -1,10 +1,10 @@
import { atomWithQuery } from "jotai-tanstack-query";
import { activeSearchSpaceIdAtom } from "@/atoms/search-spaces/search-space-query.atoms";
import { activeWorkspaceIdAtom } from "@/atoms/workspaces/workspace-query.atoms";
import { chatThreadsApiService } from "@/lib/apis/chat-threads-api.service";
import { cacheKeys } from "@/lib/query-client/cache-keys";
export const publicChatSnapshotsAtom = atomWithQuery((get) => {
const searchSpaceId = get(activeSearchSpaceIdAtom);
const searchSpaceId = get(activeWorkspaceIdAtom);
return {
queryKey: cacheKeys.publicChatSnapshots.bySearchSpace(Number(searchSpaceId) || 0),
@ -15,7 +15,7 @@ export const publicChatSnapshotsAtom = atomWithQuery((get) => {
return { snapshots: [] };
}
return chatThreadsApiService.listPublicChatSnapshotsForSearchSpace({
search_space_id: Number(searchSpaceId),
workspace_id: Number(searchSpaceId),
});
},
};

11
surfsense_web/atoms/roles/roles-mutation.atoms.ts
View file

@ -21,7 +21,7 @@ export const createRoleMutationAtom = atomWithMutation(() => {
onSuccess: (_: CreateRoleResponse, request: CreateRoleRequest) => {
toast.success("Role created successfully");
queryClient.invalidateQueries({
queryKey: cacheKeys.roles.all(request.search_space_id.toString()),
queryKey: cacheKeys.roles.all(request.workspace_id.toString()),
});
},
onError: () => {
@ -39,13 +39,10 @@ export const updateRoleMutationAtom = atomWithMutation(() => {
onSuccess: (_: UpdateRoleResponse, request: UpdateRoleRequest) => {
toast.success("Role updated successfully");
queryClient.invalidateQueries({
queryKey: cacheKeys.roles.all(request.search_space_id.toString()),
queryKey: cacheKeys.roles.all(request.workspace_id.toString()),
});
queryClient.invalidateQueries({
queryKey: cacheKeys.roles.byId(
request.search_space_id.toString(),
request.role_id.toString()
),
queryKey: cacheKeys.roles.byId(request.workspace_id.toString(), request.role_id.toString()),
});
},
onError: () => {
@ -63,7 +60,7 @@ export const deleteRoleMutationAtom = atomWithMutation(() => {
onSuccess: (_: DeleteRoleResponse, request: DeleteRoleRequest) => {
toast.success("Role deleted successfully");
queryClient.invalidateQueries({
queryKey: cacheKeys.roles.all(request.search_space_id.toString()),
queryKey: cacheKeys.roles.all(request.workspace_id.toString()),
});
},
onError: () => {

12
surfsense_web/atoms/search-spaces/search-space-mutation.atoms.ts → surfsense_web/atoms/workspaces/workspace-mutation.atoms.ts
View file

@ -5,11 +5,11 @@ import type {
DeleteSearchSpaceRequest,
UpdateSearchSpaceApiAccessRequest,
UpdateSearchSpaceRequest,
} from "@/contracts/types/search-space.types";
import { searchSpacesApiService } from "@/lib/apis/search-spaces-api.service";
} from "@/contracts/types/workspace.types";
import { searchSpacesApiService } from "@/lib/apis/workspaces-api.service";
import { cacheKeys } from "@/lib/query-client/cache-keys";
import { queryClient } from "@/lib/query-client/client";
import { activeSearchSpaceIdAtom } from "./search-space-query.atoms";
import { activeWorkspaceIdAtom } from "./workspace-query.atoms";
export const createSearchSpaceMutationAtom = atomWithMutation(() => {
return {
@ -28,7 +28,7 @@ export const createSearchSpaceMutationAtom = atomWithMutation(() => {
});
export const updateSearchSpaceMutationAtom = atomWithMutation((get) => {
const activeSearchSpaceId = get(activeSearchSpaceIdAtom);
const activeSearchSpaceId = get(activeWorkspaceIdAtom);
return {
mutationKey: ["update-search-space", activeSearchSpaceId],
@ -52,7 +52,7 @@ export const updateSearchSpaceMutationAtom = atomWithMutation((get) => {
});
export const updateSearchSpaceApiAccessMutationAtom = atomWithMutation((get) => {
const activeSearchSpaceId = get(activeSearchSpaceIdAtom);
const activeSearchSpaceId = get(activeWorkspaceIdAtom);
return {
mutationKey: ["update-search-space-api-access", activeSearchSpaceId],
@ -74,7 +74,7 @@ export const updateSearchSpaceApiAccessMutationAtom = atomWithMutation((get) =>
});
export const deleteSearchSpaceMutationAtom = atomWithMutation((get) => {
const activeSearchSpaceId = get(activeSearchSpaceIdAtom);
const activeSearchSpaceId = get(activeWorkspaceIdAtom);
return {
mutationKey: ["delete-search-space", activeSearchSpaceId],

7
surfsense_web/atoms/search-spaces/search-space-query.atoms.ts → surfsense_web/atoms/workspaces/workspace-query.atoms.ts
View file

@ -1,11 +1,10 @@
import { atom } from "jotai";
import { atomWithQuery } from "jotai-tanstack-query";
import type { GetSearchSpacesRequest } from "@/contracts/types/search-space.types";
import { searchSpacesApiService } from "@/lib/apis/search-spaces-api.service";
import type { GetSearchSpacesRequest } from "@/contracts/types/workspace.types";
import { searchSpacesApiService } from "@/lib/apis/workspaces-api.service";
import { cacheKeys } from "@/lib/query-client/cache-keys";
export const activeSearchSpaceIdAtom = atom<string | null>(null);
export const activeWorkspaceIdAtom = activeSearchSpaceIdAtom;
export const activeWorkspaceIdAtom = atom<string | null>(null);
export const searchSpacesQueryParamsAtom = atom<GetSearchSpacesRequest["queryParams"]>({
skip: 0,

33
surfsense_web/changelog/content/2026-07-05.mdx Normal file
View file

@ -0,0 +1,33 @@
---
date: "2026-07-05"
---
## SurfSense's Next Chapter: Competitive Intelligence for AI Agents
Happy Independence Day to everyone celebrating in the United States! We hope you had a wonderful Fourth of July. It felt like the right weekend for a declaration of our own.
Today we are pointing SurfSense at a single mission: **give your AI agents competitive intelligence.**
### How we got here
For the past couple of months we set out to build the best general research agent for your own knowledge. That chapter treated us well. As the open-source alternative to NotebookLM, SurfSense earned a community we are genuinely proud of: more than 15,000 GitHub stars, self-hosted deployments everywhere from homelabs to enterprises, and users who pushed the product further than we imagined.
That same period settled a bigger question. Agentic tools like Claude, OpenCode, Hermes, and OpenClaw have proven themselves. Agents that plan, use tools, and finish multi-step work on their own are no longer a demo, they are how work gets done. General research assistance is becoming something every capable agent does out of the box.
So we asked what these agents still lack. Not reasoning. Not tools. **Live market data and the workflows around it.** An agent asked "what are competitors charging this week?" or "what is Reddit saying about us since the launch?" has nowhere trustworthy to look. We want to be one of the first, and hopefully the definitive, open-source competitive intelligence agent platforms. This focus also lets our team pour all of our energy into the high-value workflows that genuinely need AI automation.
### What SurfSense is now
- **Platform-native connectors for live market data.** [Reddit](/reddit), [YouTube](/youtube), [Google Maps](/google-maps), [Google Search](/google-search), and the [open web](/web-crawl). Each one is a typed REST endpoint you can call from any language with your SurfSense API key.
- **An MCP server that makes every connector an agent tool.** Add SurfSense to Claude, Cursor, or your own agent framework. And with the [MCP connector](/mcp-connector), your SurfSense agents can use any MCP server too, with one-click OAuth for apps like Notion, Slack, and Jira.
- **An agent harness, not just raw data.** Retries, structured output, and credit metering are built in, so agents go from a question to a brief without you building the plumbing.
- **Automations for the workflows that matter.** Scheduled and event-triggered agents watch competitor pricing, track brand mentions, monitor rankings and reviews, then deliver briefs and alerts.
- **Simpler pricing.** Self-hosting stays free with no metering. Cloud is pay as you go: start with $5 of free credit, connectors bill per item returned, crawls per page fetched, models at provider cost. Failed calls are never billed. Details on the [pricing page](/pricing).
### To our existing users: nothing you rely on is going away
This is an addition, not a replacement. Your knowledge base, chat with citations, reports, podcasts, presentations, automations, and collaborative chats all keep working exactly as they do today. You can still upload files and sync Google Drive, OneDrive, and Dropbox. Self-hosting stays free and open source, and your existing deployments keep running without any action from you. What changes is where we invest: expect deeper connectors, sharper agent workflows, and broader market coverage.
If you are building agents, [grab an API key](/login) or add the MCP server, and start with the [connector catalog](/connectors). Tell us what you are building on [Discord](https://discord.gg/ejRNvftDp9) or [GitHub](https://github.com/MODSetter/SurfSense).
We think the next great competitive intelligence tool will not be a dashboard. It will be an agent that already knows your market. We are building the platform it runs on.

4
surfsense_web/components/assistant-ui/assistant-message.tsx
View file

@ -29,7 +29,7 @@ import {
globalModelConnectionsAtom,
modelConnectionsAtom,
} from "@/atoms/model-connections/model-connections-query.atoms";
import { activeSearchSpaceIdAtom } from "@/atoms/search-spaces/search-space-query.atoms";
import { activeWorkspaceIdAtom } from "@/atoms/workspaces/workspace-query.atoms";
import {
CitationMetadataProvider,
useAllCitationMetadata,
@ -491,7 +491,7 @@ export const AssistantMessage: FC = () => {
const commentPanelRef = useRef<HTMLDivElement>(null);
const commentTriggerRef = useRef<HTMLButtonElement>(null);
const messageId = useAuiState(({ message }) => message?.id);
const searchSpaceId = useAtomValue(activeSearchSpaceIdAtom);
const searchSpaceId = useAtomValue(activeWorkspaceIdAtom);
const dbMessageId = parseMessageId(messageId);
const commentsEnabled = useAtomValue(commentsEnabledAtom);

4
surfsense_web/components/assistant-ui/connector-popup.tsx
View file

@ -4,7 +4,7 @@ import { useAtomValue } from "jotai";
import { forwardRef, useEffect, useImperativeHandle, useMemo, useState } from "react";
import { createPortal } from "react-dom";
import { statusInboxItemsAtom } from "@/atoms/inbox/status-inbox.atom";
import { activeSearchSpaceIdAtom } from "@/atoms/search-spaces/search-space-query.atoms";
import { activeWorkspaceIdAtom } from "@/atoms/workspaces/workspace-query.atoms";
import { Dialog, DialogContent, DialogTitle } from "@/components/ui/dialog";
import { Tabs, TabsContent } from "@/components/ui/tabs";
import type { SearchSourceConnector } from "@/contracts/types/connector.types";
@ -36,7 +36,7 @@ interface ConnectorIndicatorProps {
export const ConnectorIndicator = forwardRef<ConnectorIndicatorHandle, ConnectorIndicatorProps>(
(_props, ref) => {
const searchSpaceId = useAtomValue(activeSearchSpaceIdAtom);
const searchSpaceId = useAtomValue(activeWorkspaceIdAtom);
// Real-time document type counts via Zero (updates instantly as docs are indexed)
const documentTypeCounts = useZeroDocumentTypeCounts(searchSpaceId);

8
surfsense_web/components/assistant-ui/connector-popup/connector-configs/components/circleback-config.tsx
View file

@ -18,7 +18,7 @@ export interface CirclebackConfigProps extends ConnectorConfigProps {
// Type-safe schema for webhook info response
const circlebackWebhookInfoSchema = z.object({
webhook_url: z.string(),
search_space_id: z.number(),
workspace_id: z.number(),
method: z.string(),
content_type: z.string(),
description: z.string(),
@ -40,12 +40,12 @@ export const CirclebackConfig: FC<CirclebackConfigProps> = ({ connector, onNameC
const controller = new AbortController();
const doFetch = async () => {
if (!connector.search_space_id) return;
if (!connector.workspace_id) return;
setIsLoading(true);
try {
const response = await authenticatedFetch(
buildBackendUrl(`/api/v1/webhooks/circleback/${connector.search_space_id}/info`),
buildBackendUrl(`/api/v1/webhooks/circleback/${connector.workspace_id}/info`),
{ signal: controller.signal }
);
if (controller.signal.aborted) return;
@ -70,7 +70,7 @@ export const CirclebackConfig: FC<CirclebackConfigProps> = ({ connector, onNameC
doFetch().catch(() => {});
return () => controller.abort();
}, [connector.search_space_id]);
}, [connector.workspace_id]);
const handleNameChange = (value: string) => {
setName(value);

4
surfsense_web/components/assistant-ui/connector-popup/connector-configs/views/connector-edit-view.tsx
View file

@ -4,7 +4,7 @@ import { useAtomValue } from "jotai";
import { ArrowLeft, Info, RefreshCw } from "lucide-react";
import { type FC, useCallback, useEffect, useMemo, useRef, useState } from "react";
import { toast } from "sonner";
import { activeSearchSpaceIdAtom } from "@/atoms/search-spaces/search-space-query.atoms";
import { activeWorkspaceIdAtom } from "@/atoms/workspaces/workspace-query.atoms";
import { Alert, AlertDescription } from "@/components/ui/alert";
import { Button } from "@/components/ui/button";
import { Spinner } from "@/components/ui/spinner";
@ -78,7 +78,7 @@ export const ConnectorEditView: FC<ConnectorEditViewProps> = ({
onConfigChange,
onNameChange,
}) => {
const searchSpaceIdAtom = useAtomValue(activeSearchSpaceIdAtom);
const searchSpaceIdAtom = useAtomValue(activeWorkspaceIdAtom);
const isAuthExpired = connector.config?.auth_expired === true;
const reauthEndpoint = getReauthEndpoint(connector);
const [reauthing, setReauthing] = useState(false);

26
surfsense_web/components/assistant-ui/connector-popup/hooks/use-connector-dialog.ts
View file

@ -10,7 +10,7 @@ import {
updateConnectorMutationAtom,
} from "@/atoms/connectors/connector-mutation.atoms";
import { connectorsAtom } from "@/atoms/connectors/connector-query.atoms";
import { activeSearchSpaceIdAtom } from "@/atoms/search-spaces/search-space-query.atoms";
import { activeWorkspaceIdAtom } from "@/atoms/workspaces/workspace-query.atoms";
import { EnumConnectorName } from "@/contracts/enums/connector";
import type { SearchSourceConnector } from "@/contracts/types/connector.types";
import { searchSourceConnector } from "@/contracts/types/connector.types";
@ -58,7 +58,7 @@ function clearOAuthResultCookie(): void {
}
export const useConnectorDialog = () => {
const searchSpaceId = useAtomValue(activeSearchSpaceIdAtom);
const searchSpaceId = useAtomValue(activeWorkspaceIdAtom);
const { data: allConnectors, refetch: refetchAllConnectors } = useAtomValue(connectorsAtom);
const { mutateAsync: indexConnector } = useAtomValue(indexConnectorMutationAtom);
const { mutateAsync: updateConnector } = useAtomValue(updateConnectorMutationAtom);
@ -165,7 +165,7 @@ export const useConnectorDialog = () => {
await indexConnector({
connector_id: connector.id,
queryParams: {
search_space_id: searchSpaceId,
workspace_id: searchSpaceId,
start_date: format(startDate, "yyyy-MM-dd"),
end_date: format(endDate, "yyyy-MM-dd"),
},
@ -418,7 +418,7 @@ export const useConnectorDialog = () => {
enable_vision_llm: false,
},
queryParams: {
search_space_id: searchSpaceId,
workspace_id: searchSpaceId,
},
});
@ -519,7 +519,7 @@ export const useConnectorDialog = () => {
enable_vision_llm: false,
},
queryParams: {
search_space_id: searchSpaceId,
workspace_id: searchSpaceId,
},
});
// Refetch connectors to get the new one
@ -612,7 +612,7 @@ export const useConnectorDialog = () => {
await indexConnector({
connector_id: connector.id,
queryParams: {
search_space_id: searchSpaceId,
workspace_id: searchSpaceId,
start_date: startDateStr,
end_date: endDateStr,
},
@ -847,7 +847,7 @@ export const useConnectorDialog = () => {
await indexConnector({
connector_id: indexingConfig.connectorId,
queryParams: {
search_space_id: searchSpaceId,
workspace_id: searchSpaceId,
},
body: {
folders: selectedFolders || [],
@ -870,14 +870,14 @@ export const useConnectorDialog = () => {
await indexConnector({
connector_id: indexingConfig.connectorId,
queryParams: {
search_space_id: searchSpaceId,
workspace_id: searchSpaceId,
},
});
} else {
await indexConnector({
connector_id: indexingConfig.connectorId,
queryParams: {
search_space_id: searchSpaceId,
workspace_id: searchSpaceId,
start_date: startDateStr,
end_date: endDateStr,
},
@ -1114,7 +1114,7 @@ export const useConnectorDialog = () => {
await indexConnector({
connector_id: editingConnector.id,
queryParams: {
search_space_id: searchSpaceId,
workspace_id: searchSpaceId,
},
body: {
folders: selectedFolders || [],
@ -1134,7 +1134,7 @@ export const useConnectorDialog = () => {
await indexConnector({
connector_id: editingConnector.id,
queryParams: {
search_space_id: searchSpaceId,
workspace_id: searchSpaceId,
},
});
indexingDescription = "Re-indexing started with updated configuration.";
@ -1143,7 +1143,7 @@ export const useConnectorDialog = () => {
await indexConnector({
connector_id: editingConnector.id,
queryParams: {
search_space_id: searchSpaceId,
workspace_id: searchSpaceId,
start_date: startDateStr,
end_date: endDateStr,
},
@ -1296,7 +1296,7 @@ export const useConnectorDialog = () => {
await indexConnector({
connector_id: connectorId,
queryParams: {
search_space_id: searchSpaceId,
workspace_id: searchSpaceId,
start_date: startDateStr,
end_date: endDateStr,
},

4
surfsense_web/components/assistant-ui/connector-popup/views/connector-accounts-list-view.tsx
View file

@ -4,7 +4,7 @@ import { useAtomValue } from "jotai";
import { ArrowLeft, Plus, RefreshCw, Server } from "lucide-react";
import { type FC, useCallback, useState } from "react";
import { toast } from "sonner";
import { activeSearchSpaceIdAtom } from "@/atoms/search-spaces/search-space-query.atoms";
import { activeWorkspaceIdAtom } from "@/atoms/workspaces/workspace-query.atoms";
import { Button } from "@/components/ui/button";
import { Spinner } from "@/components/ui/spinner";
import { EnumConnectorName } from "@/contracts/enums/connector";
@ -44,7 +44,7 @@ export const ConnectorAccountsListView: FC<ConnectorAccountsListViewProps> = ({
isConnecting = false,
addButtonText,
}) => {
const searchSpaceId = useAtomValue(activeSearchSpaceIdAtom);
const searchSpaceId = useAtomValue(activeWorkspaceIdAtom);
const [reauthingId, setReauthingId] = useState<number | null>(null);
const [confirmDisconnectId, setConfirmDisconnectId] = useState<number | null>(null);
const [disconnectingId, setDisconnectingId] = useState<number | null>(null);

2
surfsense_web/components/assistant-ui/connector-popup/views/youtube-crawler-view.tsx
View file

@ -165,7 +165,7 @@ export const YouTubeCrawlerView: FC<YouTubeCrawlerViewProps> = ({ searchSpaceId,
{
document_type: "YOUTUBE_VIDEO",
content: videoUrls,
search_space_id: parseInt(searchSpaceId, 10),
workspace_id: parseInt(searchSpaceId, 10),
},
{
onSuccess: () => {

4
surfsense_web/components/assistant-ui/document-upload-popup.tsx
View file

@ -10,7 +10,7 @@ import {
useRef,
useState,
} from "react";
import { activeSearchSpaceIdAtom } from "@/atoms/search-spaces/search-space-query.atoms";
import { activeWorkspaceIdAtom } from "@/atoms/workspaces/workspace-query.atoms";
import { DocumentUploadTab } from "@/components/sources/DocumentUploadTab";
import {
Dialog,
@ -90,7 +90,7 @@ const DocumentUploadPopupContent: FC<{
isOpen: boolean;
onOpenChange: (open: boolean) => void;
}> = ({ isOpen, onOpenChange }) => {
const searchSpaceId = useAtomValue(activeSearchSpaceIdAtom);
const searchSpaceId = useAtomValue(activeWorkspaceIdAtom);
if (!searchSpaceId) return null;

2
surfsense_web/components/assistant-ui/markdown-text.tsx
View file

@ -223,7 +223,7 @@ function FilePathLink({ path, className }: { path: string; className?: string })
if (!resolvedSearchSpaceId || !path.startsWith("/documents/")) return;
try {
const doc = await documentsApiService.getDocumentByVirtualPath({
search_space_id: resolvedSearchSpaceId,
workspace_id: resolvedSearchSpaceId,
virtual_path: path,
});
openEditorPanel({

2
surfsense_web/components/citation-panel/citation-panel.tsx
View file

@ -77,7 +77,7 @@ export const CitationPanelContent: FC<CitationPanelContentProps> = ({
if (!data) return;
openEditorPanel({
documentId: data.id,
searchSpaceId: data.search_space_id,
searchSpaceId: data.workspace_id,
title: data.title,
});
};

80
surfsense_web/components/connectors-marketing/connector-page.tsx
View file

@ -6,7 +6,7 @@ import { BreadcrumbNav } from "@/components/seo/breadcrumb-nav";
import { Badge } from "@/components/ui/badge";
import { Button } from "@/components/ui/button";
import { Separator } from "@/components/ui/separator";
import type { ConnectorPageContent } from "@/lib/connectors-marketing/types";
import type { ConnectorPageContent, SchemaField } from "@/lib/connectors-marketing/types";
import { AgentTranscript } from "./agent-transcript";
import { ApiMcpTabs } from "./api-mcp-tabs";
import { ConnectorFaq } from "./connector-faq";
@ -14,6 +14,46 @@ import { Reveal } from "./reveal";
const GITHUB_URL = "https://github.com/MODSetter/SurfSense";
function SchemaTable({ caption, fields }: { caption: string; fields: SchemaField[] }) {
return (
<div className="overflow-x-auto rounded-xl border bg-card">
<table className="w-full min-w-xl text-sm">
<caption className="sr-only">{caption}</caption>
<thead>
<tr className="border-b bg-muted/40 text-left">
<th className="p-4 font-medium">Field</th>
<th className="p-4 font-medium">Type</th>
<th className="p-4 font-medium">Description</th>
</tr>
</thead>
<tbody>
{fields.map((field) => (
<tr key={field.name} className="border-b align-top last:border-b-0">
<th scope="row" className="p-4 text-left">
<code className="font-mono text-[13px] font-semibold">{field.name}</code>
</th>
<td className="whitespace-nowrap p-4">
<code className="font-mono text-[13px] text-muted-foreground">{field.type}</code>
{field.required ? (
<span className="ml-2 rounded-full bg-brand/10 px-2 py-0.5 text-xs font-medium text-brand">
required
</span>
) : null}
{field.defaultValue !== undefined ? (
<div className="mt-1 text-xs text-muted-foreground">
default <code className="font-mono">{field.defaultValue}</code>
</div>
) : null}
</td>
<td className="p-4 text-muted-foreground leading-relaxed">{field.description}</td>
</tr>
))}
</tbody>
</table>
</div>
);
}
export function ConnectorPage({ content }: { content: ConnectorPageContent }) {
const Icon = content.icon;
const label = content.cardTitle ?? `${content.name} API`;
@ -136,6 +176,44 @@ export function ConnectorPage({ content }: { content: ConnectorPageContent }) {
</Reveal>
</MarketingSection>
{/* Request / response schema */}
<MarketingSection>
<Reveal>
<h2 className="text-2xl font-bold tracking-tight sm:text-3xl">
{content.name} API request and response schema
</h2>
<p className="mt-3 max-w-2xl text-muted-foreground leading-relaxed">
The exact contract behind{" "}
<code className="rounded bg-muted px-1.5 py-0.5 font-mono text-sm">
POST /workspaces/{"{workspace_id}"}/scrapers/{content.api.platform}/{content.api.verb}
</code>
. The same fields power the{" "}
<code className="rounded bg-muted px-1.5 py-0.5 font-mono text-sm">
{content.api.mcpTool}
</code>{" "}
MCP tool.
</p>
</Reveal>
<Reveal>
<h3 className="mt-8 text-lg font-semibold">Request parameters</h3>
<p className="mt-2 max-w-2xl text-sm text-muted-foreground leading-relaxed">
{content.schema.requestNote}
</p>
<div className="mt-4">
<SchemaTable caption="Request parameters" fields={content.schema.request} />
</div>
</Reveal>
<Reveal>
<h3 className="mt-10 text-lg font-semibold">Response fields</h3>
<p className="mt-2 max-w-2xl text-sm text-muted-foreground leading-relaxed">
{content.schema.responseNote}
</p>
<div className="mt-4">
<SchemaTable caption="Response fields" fields={content.schema.response} />
</div>
</Reveal>
</MarketingSection>
{/* Comparison */}
<MarketingSection>
<Reveal>

6
surfsense_web/components/editor-panel/editor-panel.tsx
View file

@ -276,7 +276,7 @@ export function EditorPanelContent({
}
const response = await authenticatedFetch(
buildBackendUrl(
`/api/v1/search-spaces/${searchSpaceId}/documents/${documentId}/editor-content`
`/api/v1/workspaces/${searchSpaceId}/documents/${documentId}/editor-content`
),
{ method: "GET" }
);
@ -412,7 +412,7 @@ export function EditorPanelContent({
throw new Error("Missing document context");
}
const response = await authenticatedFetch(
buildBackendUrl(`/api/v1/search-spaces/${searchSpaceId}/documents/${documentId}/save`),
buildBackendUrl(`/api/v1/workspaces/${searchSpaceId}/documents/${documentId}/save`),
{
method: "POST",
headers: { "Content-Type": "application/json" },
@ -520,7 +520,7 @@ export function EditorPanelContent({
try {
const response = await authenticatedFetch(
buildBackendUrl(
`/api/v1/search-spaces/${searchSpaceId}/documents/${documentId}/download-markdown`
`/api/v1/workspaces/${searchSpaceId}/documents/${documentId}/download-markdown`
),
{ method: "GET" }
);

9
surfsense_web/components/homepage/community-strip.tsx
View file

@ -1,4 +1,4 @@
import { IconBrandDiscord, IconBrandGithub } from "@tabler/icons-react";
import { IconBrandDiscord, IconBrandGithub, IconBrandReddit } from "@tabler/icons-react";
import { ArrowRight } from "lucide-react";
import Link from "next/link";
import { Reveal } from "@/components/connectors-marketing/reveal";
@ -7,6 +7,7 @@ import { Button } from "@/components/ui/button";
const GITHUB_URL = "https://github.com/MODSetter/SurfSense";
const DISCORD_URL = "https://discord.gg/ejRNvftDp9";
const REDDIT_URL = "https://www.reddit.com/r/SurfSense/";
/** Closing CTA doubling as the GitHub/community strip (brief section 7). */
export function CommunityStrip() {
@ -40,6 +41,12 @@ export function CommunityStrip() {
Join Discord
</Link>
</Button>
<Button asChild variant="ghost" size="lg">
<Link href={REDDIT_URL} target="_blank" rel="noopener noreferrer">
<IconBrandReddit className="size-4" />
r/SurfSense
</Link>
</Button>
</div>
</div>
</Reveal>

6
surfsense_web/components/homepage/footer-new.tsx
View file

@ -2,6 +2,7 @@ import {
IconBrandDiscord,
IconBrandGithub,
IconBrandLinkedin,
IconBrandReddit,
IconBrandTwitter,
} from "@tabler/icons-react";
import Link from "next/link";
@ -68,6 +69,11 @@ export function FooterNew() {
href: "https://discord.gg/ejRNvftDp9",
icon: IconBrandDiscord,
},
{
title: "Reddit",
href: "https://www.reddit.com/r/SurfSense/",
icon: IconBrandReddit,
},
];
const legals = [
{

13
surfsense_web/components/layout/providers/LayoutDataProvider.tsx
View file

@ -12,10 +12,11 @@ import { currentThreadAtom, resetCurrentThreadAtom } from "@/atoms/chat/current-
import { documentsSidebarOpenAtom } from "@/atoms/documents/ui.atoms";
import { statusInboxItemsAtom } from "@/atoms/inbox/status-inbox.atom";
import { announcementsDialogAtom } from "@/atoms/layout/dialogs.atom";
import { deleteSearchSpaceMutationAtom } from "@/atoms/search-spaces/search-space-mutation.atoms";
import { searchSpacesAtom } from "@/atoms/search-spaces/search-space-query.atoms";
import { rightPanelCollapsedAtom } from "@/atoms/layout/right-panel.atom";
import { removeChatTabAtom, syncChatTabAtom, type Tab } from "@/atoms/tabs/tabs.atom";
import { currentUserAtom } from "@/atoms/user/user-query.atoms";
import { deleteSearchSpaceMutationAtom } from "@/atoms/workspaces/workspace-mutation.atoms";
import { searchSpacesAtom } from "@/atoms/workspaces/workspace-query.atoms";
import { ActionLogDialog } from "@/components/agent-action-log/action-log-dialog";
import { AnnouncementSpotlight } from "@/components/announcements/AnnouncementSpotlight";
import { AnnouncementsDialog } from "@/components/announcements/AnnouncementsDialog";
@ -46,7 +47,7 @@ import { useInbox } from "@/hooks/use-inbox";
import { useIsMobile } from "@/hooks/use-mobile";
import { useArchiveThread, useDeleteThread, useRenameThread } from "@/hooks/use-thread-mutations";
import { notificationsApiService } from "@/lib/apis/notifications-api.service";
import { searchSpacesApiService } from "@/lib/apis/search-spaces-api.service";
import { searchSpacesApiService } from "@/lib/apis/workspaces-api.service";
import { getLoginPath, logout } from "@/lib/auth-utils";
import { fetchThreads } from "@/lib/chat/thread-persistence";
import { resetUser, trackLogout } from "@/lib/posthog/events";
@ -406,7 +407,7 @@ export function LayoutDataProvider({ searchSpaceId, children }: LayoutDataProvid
const handleSearchSpaceSettings = useCallback(
(space: SearchSpace) => {
router.push(`/dashboard/${space.id}/search-space-settings/general`);
router.push(`/dashboard/${space.id}/workspace-settings`);
},
[router]
);
@ -594,7 +595,7 @@ export function LayoutDataProvider({ searchSpaceId, children }: LayoutDataProvid
);
const handleSettings = useCallback(() => {
router.push(`/dashboard/${searchSpaceId}/search-space-settings/general`);
router.push(`/dashboard/${searchSpaceId}/workspace-settings`);
}, [router, searchSpaceId]);
const handleManageMembers = useCallback(() => {
@ -694,7 +695,7 @@ export function LayoutDataProvider({ searchSpaceId, children }: LayoutDataProvid
// Detect if we're on the chat page (needs overflow-hidden for chat's own scroll)
const isChatPage = pathname?.includes("/new-chat") ?? false;
const isUserSettingsPage = pathname?.includes("/user-settings") === true;
const isSearchSpaceSettingsPage = pathname?.includes("/search-space-settings") === true;
const isSearchSpaceSettingsPage = pathname?.includes("/workspace-settings") === true;
const isTeamPage = pathname?.endsWith("/team") === true;
const isAutomationsPage = pathname?.includes("/automations") === true;
const isAllChatsPage = pathname?.endsWith("/chats") === true;

2
surfsense_web/components/layout/ui/dialogs/CreateSearchSpaceDialog.tsx → surfsense_web/components/layout/ui/dialogs/CreateWorkspaceDialog.tsx
View file

@ -7,7 +7,7 @@ import { useTranslations } from "next-intl";
import { useState } from "react";
import { useForm } from "react-hook-form";
import * as z from "zod";
import { createSearchSpaceMutationAtom } from "@/atoms/search-spaces/search-space-mutation.atoms";
import { createSearchSpaceMutationAtom } from "@/atoms/workspaces/workspace-mutation.atoms";
import { Button } from "@/components/ui/button";
import {
Dialog,

2
surfsense_web/components/layout/ui/dialogs/index.ts
View file

@ -1 +1 @@
export { CreateSearchSpaceDialog } from "./CreateSearchSpaceDialog";
export { CreateSearchSpaceDialog } from "./CreateWorkspaceDialog";

6
surfsense_web/components/layout/ui/header/Header.tsx
View file

@ -3,8 +3,8 @@
import { useAtomValue } from "jotai";
import { usePathname } from "next/navigation";
import { currentThreadAtom } from "@/atoms/chat/current-thread.atom";
import { activeSearchSpaceIdAtom } from "@/atoms/search-spaces/search-space-query.atoms";
import { activeTabAtom } from "@/atoms/tabs/tabs.atom";
import { activeWorkspaceIdAtom } from "@/atoms/workspaces/workspace-query.atoms";
import { ActionLogButton } from "@/components/agent-action-log/action-log-button";
import { ChatShareButton } from "@/components/new-chat/chat-share-button";
import { ArtifactsToggleButton } from "@/features/chat-artifacts";
@ -16,7 +16,7 @@ interface HeaderProps {
export function Header({ mobileMenuTrigger }: HeaderProps) {
const pathname = usePathname();
const searchSpaceId = useAtomValue(activeSearchSpaceIdAtom);
const searchSpaceId = useAtomValue(activeWorkspaceIdAtom);
const activeTab = useAtomValue(activeTabAtom);
const isFreePage = pathname?.startsWith("/free") ?? false;
@ -56,7 +56,7 @@ export function Header({ mobileMenuTrigger }: HeaderProps) {
id: currentThreadState.id,
visibility: currentThreadState.visibility,
created_by_id: null,
search_space_id: currentThreadState.searchSpaceId,
workspace_id: currentThreadState.searchSpaceId,
title: "",
archived: false,
created_at: "",

2
surfsense_web/components/layout/ui/icon-rail/IconRail.tsx
View file

@ -7,7 +7,7 @@ import { Tooltip, TooltipContent, TooltipTrigger } from "@/components/ui/tooltip
import { cn } from "@/lib/utils";
import type { NavItem, SearchSpace, User } from "../../types/layout.types";
import { SidebarUserProfile } from "../sidebar/SidebarUserProfile";
import { SearchSpaceAvatar } from "./SearchSpaceAvatar";
import { SearchSpaceAvatar } from "./WorkspaceAvatar";
interface IconRailProps {
searchSpaces: SearchSpace[];

0
surfsense_web/components/layout/ui/icon-rail/SearchSpaceAvatar.tsx → surfsense_web/components/layout/ui/icon-rail/WorkspaceAvatar.tsx
View file

2
surfsense_web/components/layout/ui/icon-rail/index.ts
View file

@ -1,3 +1,3 @@
export { IconRail } from "./IconRail";
export { NavIcon } from "./NavIcon";
export { SearchSpaceAvatar } from "./SearchSpaceAvatar";
export { SearchSpaceAvatar } from "./WorkspaceAvatar";

80
surfsense_web/components/layout/ui/sidebar/DocumentsSidebar.tsx
View file

@ -8,10 +8,16 @@ import { useTranslations } from "next-intl";
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import { toast } from "sonner";
import { makeFolderMention, mentionedDocumentsAtom } from "@/atoms/chat/mentioned-documents.atom";
import { connectorDialogOpenAtom } from "@/atoms/connector-dialog/connector-dialog.atoms";
import { deleteDocumentMutationAtom } from "@/atoms/documents/document-mutation.atoms";
import { expandedFolderIdsAtom } from "@/atoms/documents/folder.atoms";
import { agentCreatedDocumentsAtom } from "@/atoms/documents/ui.atoms";
import { openEditorPanelAtom } from "@/atoms/editor/editor-panel.atom";
import {
folderWatchDialogOpenAtom,
folderWatchInitialFolderAtom,
} from "@/atoms/folder-sync/folder-sync.atoms";
import { searchSpacesAtom } from "@/atoms/workspaces/workspace-query.atoms";
import { CreateFolderDialog } from "@/components/documents/CreateFolderDialog";
import type { DocumentNodeDoc } from "@/components/documents/DocumentNode";
import { getDocumentTypeIcon } from "@/components/documents/DocumentTypeIcon";
@ -51,6 +57,7 @@ import type { DocumentTypeEnum } from "@/contracts/types/document.types";
import { useElectronAPI, usePlatform } from "@/hooks/use-platform";
import { documentsApiService } from "@/lib/apis/documents-api.service";
import { foldersApiService } from "@/lib/apis/folders-api.service";
import { searchSpacesApiService } from "@/lib/apis/workspaces-api.service";
import { authenticatedFetch } from "@/lib/auth-fetch";
import { getMentionDocKey } from "@/lib/chat/mention-doc-key";
import { getDocumentTypeLabel } from "@/lib/documents/document-type-labels";
@ -234,7 +241,8 @@ function AuthenticatedDocumentsSidebarBase({
const platformElectronAPI = useElectronAPI();
const electronAPI = desktopFeaturesEnabled ? platformElectronAPI : null;
const { etlService } = useRuntimeConfig();
const searchSpaceId = getWorkspaceIdNumber(params) ?? 0;
const workspaceId = getWorkspaceIdNumber(params) ?? 0;
const setConnectorDialogOpen = useSetAtom(connectorDialogOpenAtom);
const openEditorPanel = useSetAtom(openEditorPanelAtom);
const [activeTypes, setActiveTypes] = useState<DocumentTypeEnum[]>([]);
@ -248,7 +256,7 @@ function AuthenticatedDocumentsSidebarBase({
if (folders.length === 0) {
try {
const backendFolders = await documentsApiService.getWatchedFolders(searchSpaceId);
const backendFolders = await documentsApiService.getWatchedFolders(workspaceId);
for (const bf of backendFolders) {
const meta = bf.metadata as Record<string, unknown> | null;
if (!meta?.watched || !meta.folder_path) continue;
@ -256,7 +264,7 @@ function AuthenticatedDocumentsSidebarBase({
path: meta.folder_path as string,
name: bf.name,
rootFolderId: bf.id,
searchSpaceId: bf.search_space_id,
searchSpaceId: bf.workspace_id,
excludePatterns: (meta.exclude_patterns as string[]) ?? [],
fileExtensions: (meta.file_extensions as string[] | null) ?? null,
active: true,
@ -281,7 +289,7 @@ function AuthenticatedDocumentsSidebarBase({
.map((f: WatchedFolderEntry) => f.rootFolderId as number)
);
setWatchedFolderIds(ids);
}, [searchSpaceId, electronAPI]);
}, [workspaceId, electronAPI]);
useEffect(() => {
refreshWatchedIds();
@ -297,24 +305,28 @@ function AuthenticatedDocumentsSidebarBase({
// Folder state
const [expandedFolderMap, setExpandedFolderMap] = useAtom(expandedFolderIdsAtom);
const expandedIds = useMemo(
() => new Set(expandedFolderMap[searchSpaceId] ?? []),
[expandedFolderMap, searchSpaceId]
() => new Set(expandedFolderMap[workspaceId] ?? []),
[expandedFolderMap, workspaceId]
);
const toggleFolderExpand = useCallback(
(folderId: number) => {
setExpandedFolderMap((prev) => {
const current = new Set(prev[searchSpaceId] ?? []);
const current = new Set(prev[workspaceId] ?? []);
if (current.has(folderId)) current.delete(folderId);
else current.add(folderId);
return { ...prev, [searchSpaceId]: [...current] };
return { ...prev, [workspaceId]: [...current] };
});
},
[searchSpaceId, setExpandedFolderMap]
[workspaceId, setExpandedFolderMap]
);
// Zero queries for tree data
const [zeroFolders, zeroFoldersResult] = useQuery(queries.folders.bySpace({ searchSpaceId }));
const [zeroAllDocs, zeroAllDocsResult] = useQuery(queries.documents.bySpace({ searchSpaceId }));
const [zeroFolders, zeroFoldersResult] = useQuery(
queries.folders.bySpace({ searchSpaceId: workspaceId })
);
const [zeroAllDocs, zeroAllDocsResult] = useQuery(
queries.documents.bySpace({ searchSpaceId: workspaceId })
);
const [agentCreatedDocs, setAgentCreatedDocs] = useAtom(agentCreatedDocumentsAtom);
const treeFolders: FolderDisplay[] = useMemo(
@ -349,7 +361,7 @@ function AuthenticatedDocumentsSidebarBase({
const zeroIds = new Set(zeroDocs.map((d) => d.id));
const pendingAgentDocs = agentCreatedDocs
.filter((d) => d.searchSpaceId === searchSpaceId && !zeroIds.has(d.id))
.filter((d) => d.searchSpaceId === workspaceId && !zeroIds.has(d.id))
.map((d) => ({
id: d.id,
title: d.title,
@ -359,7 +371,7 @@ function AuthenticatedDocumentsSidebarBase({
}));
return [...pendingAgentDocs, ...zeroDocs];
}, [zeroAllDocs, agentCreatedDocs, searchSpaceId]);
}, [zeroAllDocs, agentCreatedDocs, workspaceId]);
// Prune agent-created docs once Zero has caught up
useEffect(() => {
@ -409,21 +421,21 @@ function AuthenticatedDocumentsSidebarBase({
await foldersApiService.createFolder({
name,
parent_id: createFolderParentId,
search_space_id: searchSpaceId,
workspace_id: workspaceId,
});
toast.success("Folder created");
if (createFolderParentId !== null) {
setExpandedFolderMap((prev) => {
const current = new Set(prev[searchSpaceId] ?? []);
const current = new Set(prev[workspaceId] ?? []);
current.add(createFolderParentId);
return { ...prev, [searchSpaceId]: [...current] };
return { ...prev, [workspaceId]: [...current] };
});
}
} catch (e: unknown) {
toast.error((e as Error)?.message || "Failed to create folder");
}
},
[createFolderParentId, searchSpaceId, setExpandedFolderMap]
[createFolderParentId, workspaceId, setExpandedFolderMap]
);
const handleRescanFolder = useCallback(
@ -444,7 +456,7 @@ function AuthenticatedDocumentsSidebarBase({
await uploadFolderScan({
folderPath: matched.path,
folderName: matched.name,
searchSpaceId,
searchSpaceId: workspaceId,
excludePatterns: matched.excludePatterns ?? DEFAULT_EXCLUDE_PATTERNS,
fileExtensions:
matched.fileExtensions ?? Array.from(getSupportedExtensionsSet(undefined, etlService)),
@ -455,7 +467,7 @@ function AuthenticatedDocumentsSidebarBase({
toast.error((err as Error)?.message || "Failed to re-scan folder");
}
},
[searchSpaceId, electronAPI, etlService]
[workspaceId, electronAPI, etlService]
);
const handleStopWatching = useCallback(
@ -577,7 +589,7 @@ function AuthenticatedDocumentsSidebarBase({
.trim()
.slice(0, 80) || "folder";
await doExport(
buildBackendUrl(`/api/v1/search-spaces/${searchSpaceId}/export`, {
buildBackendUrl(`/api/v1/workspaces/${workspaceId}/export`, {
folder_id: ctx.folder.id,
}),
`${safeName}.zip`
@ -590,7 +602,7 @@ function AuthenticatedDocumentsSidebarBase({
isExportingKBRef.current = false;
}
setExportWarningContext(null);
}, [exportWarningContext, searchSpaceId, doExport]);
}, [exportWarningContext, workspaceId, doExport]);
const getPendingCountInSubtree = useCallback(
(folderId: number): number => {
@ -631,7 +643,7 @@ function AuthenticatedDocumentsSidebarBase({
.trim()
.slice(0, 80) || "folder";
await doExport(
buildBackendUrl(`/api/v1/search-spaces/${searchSpaceId}/export`, {
buildBackendUrl(`/api/v1/workspaces/${workspaceId}/export`, {
folder_id: folder.id,
}),
`${safeName}.zip`
@ -644,7 +656,7 @@ function AuthenticatedDocumentsSidebarBase({
isExportingKBRef.current = false;
}
},
[searchSpaceId, getPendingCountInSubtree, doExport]
[workspaceId, getPendingCountInSubtree, doExport]
);
const handleExportDocument = useCallback(
@ -654,7 +666,7 @@ function AuthenticatedDocumentsSidebarBase({
const endpoint =
doc.document_type === "USER_MEMORY"
? buildBackendUrl("/api/v1/users/me/memory")
: buildBackendUrl(`/api/v1/workspaces/${searchSpaceId}/memory`);
: buildBackendUrl(`/api/v1/workspaces/${workspaceId}/memory`);
const response = await authenticatedFetch(endpoint, { method: "GET" });
if (!response.ok) {
const errorData = await response.json().catch(() => ({ detail: "Export failed" }));
@ -682,7 +694,7 @@ function AuthenticatedDocumentsSidebarBase({
try {
const response = await authenticatedFetch(
buildBackendUrl(`/api/v1/search-spaces/${searchSpaceId}/documents/${doc.id}/export`, {
buildBackendUrl(`/api/v1/workspaces/${workspaceId}/documents/${doc.id}/export`, {
format,
}),
{ method: "GET" }
@ -707,7 +719,7 @@ function AuthenticatedDocumentsSidebarBase({
toast.error(err instanceof Error ? err.message : `Export failed`);
}
},
[searchSpaceId]
[workspaceId]
);
const handleFolderPickerSelect = useCallback(
@ -830,7 +842,7 @@ function AuthenticatedDocumentsSidebarBase({
openEditorPanel({
kind: "memory",
memoryScope: "user",
searchSpaceId,
searchSpaceId: workspaceId,
title: doc.title,
});
return true;
@ -839,14 +851,14 @@ function AuthenticatedDocumentsSidebarBase({
openEditorPanel({
kind: "memory",
memoryScope: "team",
searchSpaceId,
searchSpaceId: workspaceId,
title: doc.title,
});
return true;
}
return false;
},
[openEditorPanel, searchSpaceId]
[openEditorPanel, workspaceId]
);
const handleResetMemoryDocument = useCallback(
@ -858,7 +870,7 @@ function AuthenticatedDocumentsSidebarBase({
const endpoint =
doc.document_type === "USER_MEMORY"
? buildBackendUrl("/api/v1/users/me/memory/reset")
: buildBackendUrl(`/api/v1/workspaces/${searchSpaceId}/memory/reset`);
: buildBackendUrl(`/api/v1/workspaces/${workspaceId}/memory/reset`);
try {
const response = await authenticatedFetch(endpoint, { method: "POST" });
if (!response.ok) {
@ -871,7 +883,7 @@ function AuthenticatedDocumentsSidebarBase({
toast.error((error as Error)?.message || `Failed to reset ${doc.title.toLowerCase()}`);
}
},
[openMemoryDocument, searchSpaceId]
[openMemoryDocument, workspaceId]
);
const typeCounts = useMemo(() => {
@ -1008,7 +1020,7 @@ function AuthenticatedDocumentsSidebarBase({
if (openMemoryDocument(doc)) return;
openEditorPanel({
documentId: doc.id,
searchSpaceId,
searchSpaceId: workspaceId,
title: doc.title,
});
}}
@ -1016,7 +1028,7 @@ function AuthenticatedDocumentsSidebarBase({
if (openMemoryDocument(doc)) return;
openEditorPanel({
documentId: doc.id,
searchSpaceId,
searchSpaceId: workspaceId,
title: doc.title,
});
}}
@ -1098,7 +1110,7 @@ function AuthenticatedDocumentsSidebarBase({
{deletableSelectedIds.length === 1
? "This document"
: `These ${deletableSelectedIds.length} documents`}{" "}
will be permanently deleted from your search space.
will be permanently deleted from your workspace.
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>

6
surfsense_web/components/layout/ui/sidebar/InboxSidebar.tsx
View file

@ -208,7 +208,7 @@ export function InboxSidebarContent({
queryFn: () =>
notificationsApiService.getNotifications({
queryParams: {
search_space_id: searchSpaceId ?? undefined,
workspace_id: searchSpaceId ?? undefined,
type: searchTypeFilter,
search: debouncedSearch.trim(),
limit: 50,
@ -364,7 +364,7 @@ export function InboxSidebarContent({
if (item.type === "new_mention") {
if (isNewMentionMetadata(item.metadata)) {
const searchSpaceId = item.search_space_id;
const searchSpaceId = item.workspace_id;
const threadId = item.metadata.thread_id;
const commentId = item.metadata.comment_id;
@ -382,7 +382,7 @@ export function InboxSidebarContent({
}
} else if (item.type === "comment_reply") {
if (isCommentReplyMetadata(item.metadata)) {
const searchSpaceId = item.search_space_id;
const searchSpaceId = item.workspace_id;
const threadId = item.metadata.thread_id;
const replyId = item.metadata.reply_id;

2
surfsense_web/components/layout/ui/sidebar/MobileSidebar.tsx
View file

@ -5,7 +5,7 @@ import { Button } from "@/components/ui/button";
import { ScrollArea } from "@/components/ui/scroll-area";
import { Sheet, SheetContent, SheetTitle } from "@/components/ui/sheet";
import type { ChatItem, NavItem, PageUsage, SearchSpace, User } from "../../types/layout.types";
import { SearchSpaceAvatar } from "../icon-rail/SearchSpaceAvatar";
import { SearchSpaceAvatar } from "../icon-rail/WorkspaceAvatar";
import { Sidebar } from "./Sidebar";
interface MobileSidebarProps {

6
surfsense_web/components/layout/ui/tabs/DocumentTabContent.tsx
View file

@ -104,7 +104,7 @@ export function DocumentTabContent({ documentId, searchSpaceId, title }: Documen
try {
const response = await authenticatedFetch(
buildBackendUrl(
`/api/v1/search-spaces/${searchSpaceId}/documents/${documentId}/editor-content`
`/api/v1/workspaces/${searchSpaceId}/documents/${documentId}/editor-content`
),
{ method: "GET" }
);
@ -154,7 +154,7 @@ export function DocumentTabContent({ documentId, searchSpaceId, title }: Documen
setSaving(true);
try {
const response = await authenticatedFetch(
buildBackendUrl(`/api/v1/search-spaces/${searchSpaceId}/documents/${documentId}/save`),
buildBackendUrl(`/api/v1/workspaces/${searchSpaceId}/documents/${documentId}/save`),
{
method: "POST",
headers: { "Content-Type": "application/json" },
@ -313,7 +313,7 @@ export function DocumentTabContent({ documentId, searchSpaceId, title }: Documen
try {
const response = await authenticatedFetch(
buildBackendUrl(
`/api/v1/search-spaces/${searchSpaceId}/documents/${documentId}/download-markdown`
`/api/v1/workspaces/${searchSpaceId}/documents/${documentId}/download-markdown`
),
{ method: "GET" }
);

6
surfsense_web/components/new-chat/chat-share-button.tsx
View file

@ -51,7 +51,7 @@ export function ChatShareButton({ thread, onVisibilityChange, className }: ChatS
// Use Jotai atom for visibility (single source of truth)
const currentThreadState = useAtomValue(currentThreadAtom);
const { mutateAsync: updateVisibility } = useUpdateThreadVisibility(thread?.search_space_id ?? 0);
const { mutateAsync: updateVisibility } = useUpdateThreadVisibility(thread?.workspace_id ?? 0);
// Snapshot creation mutation
const { mutateAsync: createSnapshot, isPending: isCreatingSnapshot } = useAtomValue(
@ -139,9 +139,7 @@ export function ChatShareButton({ thread, onVisibilityChange, className }: ChatS
variant="ghost"
size="icon"
onClick={() =>
router.push(
`/dashboard/${thread.search_space_id}/search-space-settings/public-links`
)
router.push(`/dashboard/${thread.workspace_id}/workspace-settings/public-links`)
}
className="size-8 bg-muted/50 hover:bg-accent hover:text-accent-foreground"
>

4
surfsense_web/components/new-chat/document-mention-picker.tsx
View file

@ -316,7 +316,7 @@ export const DocumentMentionPicker = forwardRef<
const titleSearchParams = useMemo(
() => ({
search_space_id: searchSpaceId,
workspace_id: searchSpaceId,
page: 0,
page_size: PAGE_SIZE,
...(isSearchValid ? { title: debouncedSearch.trim() } : {}),
@ -362,7 +362,7 @@ export const DocumentMentionPicker = forwardRef<
try {
const queryParams = {
search_space_id: searchSpaceId,
workspace_id: searchSpaceId,
page: nextPage,
page_size: PAGE_SIZE,
...(isSearchValid ? { title: debouncedSearch.trim() } : {}),

2
surfsense_web/components/new-chat/image-model-selector.tsx
View file

@ -146,7 +146,7 @@ export function ImageModelSelector({
function manageModelConnections() {
setOpen(false);
router.push(`/dashboard/${searchSpaceId}/search-space-settings/models`);
router.push(`/dashboard/${searchSpaceId}/workspace-settings/models`);
}
const handleScroll = useCallback((event: UIEvent<HTMLDivElement>) => {

2
surfsense_web/components/new-chat/model-selector.tsx
View file

@ -149,7 +149,7 @@ export function ModelSelector({
function manageModelConnections() {
setOpen(false);
router.push(`/dashboard/${searchSpaceId}/search-space-settings/models`);
router.push(`/dashboard/${searchSpaceId}/workspace-settings/models`);
}
const handleScroll = useCallback((event: UIEvent<HTMLDivElement>) => {

4
surfsense_web/components/onboarding-tour.tsx
View file

@ -7,8 +7,8 @@ import { useTheme } from "next-themes";
import { useCallback, useEffect, useRef, useState } from "react";
import { createPortal } from "react-dom";
import { connectorsAtom } from "@/atoms/connectors/connector-query.atoms";
import { activeSearchSpaceIdAtom } from "@/atoms/search-spaces/search-space-query.atoms";
import { currentUserAtom } from "@/atoms/user/user-query.atoms";
import { activeWorkspaceIdAtom } from "@/atoms/workspaces/workspace-query.atoms";
import { Button } from "@/components/ui/button";
import { useIsMobile } from "@/hooks/use-mobile";
import { useZeroDocumentTypeCounts } from "@/hooks/use-zero-document-type-counts";
@ -391,7 +391,7 @@ export function OnboardingTour() {
// Get user data
const { data: user } = useAtomValue(currentUserAtom);
const searchSpaceId = useAtomValue(activeSearchSpaceIdAtom);
const searchSpaceId = useAtomValue(activeWorkspaceIdAtom);
// Fetch threads data
const { data: threadsData } = useQuery({

2
surfsense_web/components/public-chat/public-chat-footer.tsx
View file

@ -28,7 +28,7 @@ export function PublicChatFooter({ shareToken }: PublicChatFooterProps) {
});
// Redirect to the new chat page with cloned content
router.push(`/dashboard/${response.search_space_id}/new-chat/${response.thread_id}`);
router.push(`/dashboard/${response.workspace_id}/new-chat/${response.thread_id}`);
} catch (error) {
const message = error instanceof Error ? error.message : "Failed to copy chat";
toast.error(message);

7
surfsense_web/components/seo/json-ld.tsx
View file

@ -20,7 +20,12 @@ export function OrganizationJsonLd() {
logo: "https://www.surfsense.com/logo.png",
description:
"SurfSense is an open-source competitive intelligence platform. AI agents monitor competitors, track rankings, and listen to your market through one API or MCP server.",
sameAs: ["https://github.com/MODSetter/SurfSense", "https://discord.gg/ejRNvftDp9"],
sameAs: [
"https://github.com/MODSetter/SurfSense",
"https://discord.gg/ejRNvftDp9",
"https://www.reddit.com/r/SurfSense/",
"https://www.linkedin.com/company/surfsense/",
],
contactPoint: {
"@type": "ContactPoint",
email: "rohan@surfsense.com",

2
surfsense_web/components/settings/auto-reload-settings.tsx
View file

@ -80,7 +80,7 @@ export function AutoReloadSettings() {
const setupMutation = useMutation({
mutationFn: () =>
stripeApiService.createAutoReloadSetupSession({ search_space_id: searchSpaceId }),
stripeApiService.createAutoReloadSetupSession({ workspace_id: searchSpaceId }),
onSuccess: (response) => {
window.location.assign(response.checkout_url);
},

2
surfsense_web/components/settings/buy-credits-content.tsx
View file

@ -177,7 +177,7 @@ export function BuyCreditsContent() {
<Button
className="w-full"
disabled={purchaseMutation.isPending}
onClick={() => purchaseMutation.mutate({ quantity, search_space_id: searchSpaceId })}
onClick={() => purchaseMutation.mutate({ quantity, workspace_id: searchSpaceId })}
>
{purchaseMutation.isPending ? (
<>

Some files were not shown because too many files have changed in this diff Show more