From 0944d9bb78f1d07b48b54a21d1c4e381b71ac69b Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 14:51:29 +0200 Subject: [PATCH 01/64] docs(plans): re-cast Phase 4-7 to stateless scraper-APIs, flatten subplans --- plans/backend/00-umbrella-plan.md | 124 ++++--- plans/backend/00b-pipeline-diagrams.md | 110 ++++++ plans/backend/02-rename-backend.md | 2 +- plans/backend/03c-crawl-billing.md | 2 +- plans/backend/04-capabilities.md | 89 +++++ plans/backend/04a-connector-category.md | 167 --------- plans/backend/04b-source-discovery.md | 120 ------ plans/backend/05-access.md | 85 +++++ plans/backend/05-pipelines-model.md | 321 ---------------- plans/backend/06-ongoing-automation.md | 25 ++ plans/backend/06-pipelines-exec.md | 350 ------------------ plans/backend/07-orchestration.md | 73 ++++ plans/backend/07-upload-pipeline-kb.md | 195 ---------- .../backend/revamp phases 4-7/00-overview.md | 125 ------- .../00b-pipeline-diagrams.md | 202 ---------- .../revamp phases 4-7/04a-capabilities.md | 198 ---------- plans/backend/revamp phases 4-7/04b-access.md | 136 ------- .../backend/revamp phases 4-7/05a-timeline.md | 125 ------- .../revamp phases 4-7/05b-intelligence.md | 191 ---------- .../backend/revamp phases 4-7/06-triggers.md | 131 ------- .../revamp phases 4-7/07-orchestration.md | 126 ------- 21 files changed, 447 insertions(+), 2450 deletions(-) create mode 100644 plans/backend/00b-pipeline-diagrams.md create mode 100644 plans/backend/04-capabilities.md delete mode 100644 plans/backend/04a-connector-category.md delete mode 100644 plans/backend/04b-source-discovery.md create mode 100644 plans/backend/05-access.md delete mode 100644 plans/backend/05-pipelines-model.md create mode 100644 plans/backend/06-ongoing-automation.md delete mode 100644 plans/backend/06-pipelines-exec.md create mode 100644 plans/backend/07-orchestration.md delete mode 100644 plans/backend/07-upload-pipeline-kb.md delete mode 100644 plans/backend/revamp phases 4-7/00-overview.md delete mode 100644 plans/backend/revamp phases 4-7/00b-pipeline-diagrams.md delete mode 100644 plans/backend/revamp phases 4-7/04a-capabilities.md delete mode 100644 plans/backend/revamp phases 4-7/04b-access.md delete mode 100644 plans/backend/revamp phases 4-7/05a-timeline.md delete mode 100644 plans/backend/revamp phases 4-7/05b-intelligence.md delete mode 100644 plans/backend/revamp phases 4-7/06-triggers.md delete mode 100644 plans/backend/revamp phases 4-7/07-orchestration.md diff --git a/plans/backend/00-umbrella-plan.md b/plans/backend/00-umbrella-plan.md index cd0373052..300cd9e68 100644 --- a/plans/backend/00-umbrella-plan.md +++ b/plans/backend/00-umbrella-plan.md @@ -2,16 +2,30 @@ > Master roadmap for the Competitive Intelligence pivot. Each phase becomes its own subplan saved in this folder (`plans/backend/`). -This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shipped), then the WebURL crawler moat (Phase 3, shipped), then the CI product itself as **four revamped phases** — **Phase 4 Capabilities & Access**, **Phase 5 Intelligence & Timeline**, **Phase 6 Triggers**, **Phase 7 Orchestration** (canonical subplans in [`revamp phases 4-7/`](revamp%20phases%204-7/00-overview.md)). **NOTE:** the original "connectors → Pipelines" plan (old Phases 4–7) was **superseded on 2026-06-30** — see the Architecture correction below. The earlier "Phase 5′ automation-enhancements" framing is also retired (folded into Phases 4/5/6 of the revamp). +This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shipped), then the WebURL crawler moat (Phase 3, shipped), then the CI product itself as **four revamped phases** — **Phase 4 Capabilities** ([`04-capabilities.md`](04-capabilities.md)), **Phase 5 Access** ([`05-access.md`](05-access.md)), **Phase 6 Ongoing-Automation** ([`06-ongoing-automation.md`](06-ongoing-automation.md), design deferred), **Phase 7 Orchestration** ([`07-orchestration.md`](07-orchestration.md)). **NOTE:** the original "connectors → Pipelines" plan (old Phases 4–7) was superseded on 2026-06-30, and the stateful **Tracker + Timeline + Triggers** design was superseded on **2026-07-01** (scraper-APIs-first, stateless, chat-native) — see the Architecture corrections below. > SCOPE: This umbrella currently covers the BACKEND only (`surfsense_backend`). Frontend (`surfsense_web`) and client apps (desktop, Obsidian, browser extension) will get their own umbrella/subplans LATER, once the backend is fully working as expected. Frontend-facing decisions (URL segment, TS types, i18n copy) are recorded below where relevant but are out of scope for the active phases. ## Positioning -"NotebookLM for Competitive Intelligence" — each WorkSpace acts as a workspace for setting up competitive-intelligence-optimised notebooks. +Premium, open-source, self-hostable **scraper APIs** (Bright Data / Firecrawl / Scrapfly-class) that solve the hard parts — bypass, proxies, stealth — and return **cleaned, AI-ready data** for Google Maps + general web. A chat agent uses those APIs to answer competitive-intelligence questions and, for standing needs, keeps watching over time. + +## ⚠️ Architecture pivot (2026-07-01) — scraper-APIs-first, stateless, chat-native + +> **This supersedes the stateful Tracker/Timeline/Triggers design (the 2026-06-30 revamp's Phases 5–6).** The canonical Phase 4–7 subplans (`04`–`07`, below) are re-cast to four phases: +> +> - **The product = scraper capabilities**, not a CI engine. `web.*` + `maps.*` typed verbs return cleaned, AI-ready data, exposed identically through **REST + API key**, an **MCP server**, and **chat**, all generated from one capability registry. These endpoints are the revenue driver (premium, OSS, self-hostable). +> - **Stateless.** No Timeline (3-store delta), no `Tracker`, no diff/materiality engine, no stored `entity_changes`. Memory = the **chat history**; the agent reasons over prior tool outputs in context to report "what changed." +> - **Direct calls.** Capabilities return their result directly — no job store, no `completed | pending + job_id` envelope, no `deliverable_wait` polling. Async results reach the client via **write-then-sync / stream** (existing SSE or a Zero-published table). +> - **Automation = a persistent ongoing chat** that periodically re-invokes verbs. This replaces the `Triggers` subsystem; the periodic mechanism is **design-deferred** (Phase 6). +> - **KB stays input-only**; crawled data is never indexed. +> +> **Phase mapping:** **04 Capabilities** · **05 Access** · **06 Ongoing-Automation** (deferred) · **07 Orchestration** (`intelligence_agent`). The subplans `05a-timeline`/`05b-intelligence`/`06-triggers` are **removed** (history in git); `04a`/`04b` are folded into `04`/`05`. ## ⚠️ Architecture correction (2026-06-30) — Pipelines dropped; automations + input-only KB adopted +> **Historical record.** The pipelines-drop + input-only-KB decisions below still hold. The stateful Tracker/Timeline/Triggers framing in its "Net effect" bullets was itself superseded by the **2026-07-01 pivot** above. + > **This supersedes the original Pipelines paradigm (Phases 5–7).** During Phase 4 discussion we concluded that the *"sync data into the KB, then operate over it"* model is wrong for competitive intelligence. Verified against `references/opencode` — a coding agent that pulls context **live** via `read`/`grep`/`webfetch`/`websearch` tools (`packages/opencode/src/tool/`) and persists **sessions**, never a scraped corpus; its `websearch` tool even live-crawls at query time (`tool/websearch.ts`). The corrected model: > > - **Knowledge Base = input-only** — the user's personal files/context that *enriches* the agent. Filled by file uploads + file connectors (Google Drive / Dropbox / OneDrive). **Nothing scrapes into it.** Its job is to be searchable *user-provided* context, not a dump of fetched pages. @@ -20,7 +34,7 @@ This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shi > - **Automations are the scheduling + run-history substrate.** An automation run already invokes the *full chat agent* (with the crawler/search tools) on a **cron/event** trigger and persists **run history** (`automation_runs`) — exactly the recurring-fetch + monitoring loop Pipelines were reinventing. You store the agent's *insight per run*, not raw snapshots. > - **`WEBCRAWLER_CONNECTOR` is retired** — the crawler is a native tool, not a connector/data-source. > -> **Net effect on this umbrella:** the old **Pipelines** stack (old Phases 5–7) is dropped and **Phases 4–7 are re-cast** by the canonical revamp in [`revamp phases 4-7/`](revamp%20phases%204-7/00-overview.md): +> **Net effect on this umbrella:** the old **Pipelines** stack (old Phases 5–7) is dropped and **Phases 4–7 are re-cast** by the canonical subplans `04`–`07` (this framing was itself superseded by the 2026-07-01 pivot above): > - **Phase 4 — Capabilities & Access** (`revamp/04a` + `04b`): turn the crawler + search into **typed, callable verbs** (MVP = `web.scrape` + `web.discover`; per-platform scrapers like `maps.*`/`linkedin.*` are *later, uncommitted* drop-ins) over a single **capability registry**, exposed identically through **chat / REST+API-key / MCP** doors. Ships **Product A** (stateless utility) — revenue day one. *The old `04a-connector-category` is demoted to backward-compat hygiene; the old `04b-source-discovery` is absorbed as the `web.discover` verb.* > - **Phase 5 — Intelligence & Timeline** (`revamp/05a` + `05b`): the `Tracker` primitive + a **3-store delta Timeline** (`tracked_entities`/`entity_current_state`/`entity_changes`, "store deltas, no change → no row"). This is the durable CI state and **the moat** — it replaces "pipelines" as the *standing concern*. Ships the **Product B** engine. > - **Phase 6 — Triggers** (`revamp/06`): decide **when** a Tracker refreshes. `refresh(tracker)` is trigger-agnostic; recurrence is a **CI action on the existing automations** (reuse its schedule selector + `AutomationRun` — no new scheduler, no new run table). Alert **delivery is separate** (via `app/notifications/`; automations have no delivery path). @@ -28,27 +42,23 @@ This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shi > > **KB stays input-only; `WEBCRAWLER_CONNECTOR` retired; crawler/search are native tools (later a platform API key).** The three correction decisions land in their real homes: **bill crawls at the capability executor** (`revamp/04a`, works uniformly across chat/automation/REST/MCP/cron — see 03c below), **diffable run history = the Timeline delta store** (`revamp/05a`, *not* `automation_runs.output`), and **"run now" = a Trigger adapter** (`revamp/06`). Phase 8 platform scrapers re-slot as a family of **individual native endpoints** — each one a capability verb → an agent tool + a dev API-key REST endpoint — added incrementally (**none, incl. Google Maps, committed for MVP**; they're examples of the pattern), not pipeline executors. -## Target architecture (corrected + revamped) +## Target architecture (scraper-APIs-first, stateless) -> Full stateless/stateful end-to-end flow diagrams live in [`revamp phases 4-7/00b-pipeline-diagrams.md`](revamp%20phases%204-7/00b-pipeline-diagrams.md). Summary below. +> Full end-to-end flow diagrams live in [`00b-pipeline-diagrams.md`](00b-pipeline-diagrams.md). Summary below. ```mermaid flowchart TD - ACQ[["Acquisition (Phase 3, shipped): WebURL crawler + proxy/stealth/captcha + 03c billing · + Maps actor (P8)"]] - REG["Capability registry (P4a): web.scrape · web.discover (MVP) · <platform>.* scrapers (later) — typed verbs, bill at executor"] + ACQ[["Acquisition (Phase 3, shipped): WebURL crawler + proxy/stealth/captcha + 03c billing · + Maps actor (new)"]] + REG["Capability registry (P4): web.scrape · web.discover · maps.search/place/reviews — typed verbs, cleaned AI-ready output, bill at executor"] ACQ --> REG - REG --> DOORS["Access doors (P4b): chat tools · REST+API key · MCP — generated from the registry"] - DOORS --> AGENT["intelligence_agent subagent (P7): intent routing · verb chains · Tracker crafting"] + REG --> DOORS["Access doors (P5): REST + API key · MCP server · chat tools — generated from the registry"] + DOORS --> AGENT["intelligence_agent subagent (P7): intent routing · verb chains · 'what changed' from chat history"] KB[("Knowledge Base — input-only: user files/context (uploads + Drive/Dropbox/OneDrive)")] --> AGENT CONN["Connectors = MCP tools (BYO) + file/KB-input"] --> AGENT - AGENT -->|one-shot / Product A| ANS["plain-language answer (nothing persists)"] - AGENT -->|standing concern / Product B| TR["Tracker (P5b): locked schema · refresh hot loop"] - REG --> TR - TR --> TL[("Timeline (P5a): tracked_entities · entity_current_state · entity_changes — delta moat")] - TRIG["Triggers (P6): manual · agent · external-cron · CI automation action"] --> TR - AUTO["Automations (existing): schedule selector + AutomationRun (recurrence + audit; NO delivery)"] --> TRIG - TL --> DELIVER["alerts on material change → app/notifications (Zero-synced)"] - APIKEY["Platform API key (future)"] --> DOORS + AGENT -->|one-shot| ANS["plain-language answer (nothing persists)"] + AGENT -.->|standing need| ONG["Ongoing-Automation (P6, DEFERRED): persistent chat re-invokes verbs; memory = chat history"] + ONG -.-> REG + APIKEY["Platform API key"] --> DOORS ```
@@ -78,14 +88,14 @@ flowchart TD - Full rename SearchSpace -> WorkSpace across DB, API, URLs, code, satellite apps. - Canonical names (proposed defaults): DB table `workspaces`, column `workspace_id`, RBAC tables `workspace_roles` / `workspace_memberships` / `workspace_invites`, API base `/workspaces` (consolidating today's `/searchspaces` vs `/search-spaces` split), URL segment `[workspace_id]`, settings folder `workspace-settings`, TS type `Workspace`. -- **Capabilities, not connectors, are the core of Phase 4.** The crawler + search become typed **verbs** (MVP `web.scrape` + `web.discover`; per-platform scrapers like `maps.*` are later, uncommitted drop-ins) in a single **capability registry** that generates the chat/REST/MCP doors identically (revamp `04a`/`04b`). A capability is *call → data → bill*; no `SearchSourceConnector` row, no KB write, no schedule. The **old connector two-type taxonomy is demoted to backward-compat hygiene** (kept so existing rows/KB docs stay searchable) — the one genuinely useful piece, the **`MCP_CONNECTOR` (BYO MCP) routing-gap fix**, moves into revamp `04b` (consume-user-MCP). Surviving connectors: **file/KB-input** (Google Drive native+Composio, OneDrive, Dropbox, uploads → the input-only KB) + **BYO `MCP_CONNECTOR`** (act in chat). **`WEBCRAWLER_CONNECTOR` RETIRED** (native tool); branded natives stay `MIGRATING` (off); Obsidian/Circleback `DISABLED`. Artifacts stay in the existing `deliverables` system. **~~pipeline-eligibility~~ removed.** -- Web search APIs (SearXNG, Linkup, Baidu) are repurposed as the **`web.discover` capability** (a platform-level search provider set): given a topic/competitor, suggest candidate URLs the agent (or a Tracker binding) can feed to `web.scrape`. Not a standalone connector type, does not index data. NOTE: Tavily and Serper are being REMOVED from the search infra and are not part of this set. -- **`Tracker` + Timeline are the standing-concern primitive** (replaces "pipeline"). A Tracker binds a capability + input and a locked, versioned `definition` (`field_schema`/`identity_rule`/`materiality`); its `refresh(tracker)` hot loop writes a **3-store delta Timeline** (`tracked_entities`/`entity_current_state`/`entity_changes`). **Store deltas, no change → no row.** The Timeline — not the KB, not `automation_runs` — is the only new durable CI state (revamp `05a`/`05b`). -- **Bill at the capability executor.** Each verb declares a `billing_unit`; the executor charges the workspace owner once per billable success via `WebCrawlCreditService` (03c) — so chat, automation/recurring, REST, MCP, and external-cron all meter **uniformly**. (Closes the confirmed gap where automation-run crawls billed nothing; see 03c + the Architecture correction.) +- **Capabilities, not connectors, are the core of the product.** The crawler + search + Maps actor become typed **verbs** (`web.scrape`, `web.discover`, `maps.search/place/reviews`) in a single **capability registry** that generates the REST/MCP/chat doors identically (`04`/`05`). A capability is *call → cleaned data → bill*; no `SearchSourceConnector` row, no KB write, no schedule. The **old connector two-type taxonomy is demoted to backward-compat hygiene** (kept so existing rows/KB docs stay searchable) — the one genuinely useful piece, the **`MCP_CONNECTOR` (BYO MCP) routing-gap fix**, moves into `05` (consume-user-MCP). Surviving connectors: **file/KB-input** (Google Drive native+Composio, OneDrive, Dropbox, uploads → the input-only KB) + **BYO `MCP_CONNECTOR`** (act in chat). **`WEBCRAWLER_CONNECTOR` RETIRED** (native tool); branded natives stay `MIGRATING` (off); Obsidian/Circleback `DISABLED`. Artifacts stay in the existing `deliverables` system. +- Web search APIs (SearXNG, Linkup, Baidu) are repurposed as the **`web.discover` capability** (a platform-level search provider set): given a topic/competitor, suggest candidate URLs the agent can feed to `web.scrape`. Not a standalone connector type, does not index data. NOTE: Tavily and Serper are being REMOVED from the search infra and are not part of this set. +- **Stateless — the chat history is the memory.** No Timeline, no `Tracker`, no diff/materiality engine, no stored `entity_changes`. "What changed" is the agent reasoning over prior tool outputs already in context (`07`). Standing needs are served by a **persistent ongoing chat** that periodically re-invokes verbs (`06`, design deferred). +- **Bill at the capability executor.** Each verb declares a `billing_unit`; the executor charges the workspace owner once per billable success via the billing service (`WebCrawlCreditService`, 03c, first provider) — so chat, ongoing automation, REST, and MCP all meter **uniformly**. `maps.*` registers a new per-place / per-page unit. - Obsidian and Circleback (push/webhook sources) are DISABLED for the MVP. - MCP-availability audit complete: BookStack (community MCP servers), Elasticsearch (official Elastic Agent Builder MCP), and Luma (community MCP servers) all have MCP available, so none are `DISABLED` — they're tagged `MIGRATING` (turned off for MVP like the other branded natives, pending the post-MVP MCP re-point). -- ~~`Pipeline` and `PipelineRun` are new first-class tables.~~ **SUPERSEDED** — no Pipelines. The standing-concern need is met by the **`Tracker` + Timeline** (revamp `05`), and **recurrence + audit reuse the existing automations** (schedule selector + `AutomationRun`) via a CI **action** (revamp `06`) — **no new scheduler, no new run table**. **Delivery does NOT reuse automations** (no such path; `automation_runs.output` is never written) — alerts wire to the separate `app/notifications/` system. Uploads simply populate the input-only KB (existing upload code, no pipeline wrapper). -- ~~The chat agent gets read-only access to pipeline run history~~ → the agent answers "what changed?" by reading the **Timeline** (`query_timeline`, revamp `05a`), not by re-deriving from run logs. Recurring execution audit rides `AutomationRun` (already persisted + queryable). No separate pipeline-run context tool. +- ~~`Pipeline` and `PipelineRun` are new first-class tables.~~ **SUPERSEDED** — no Pipelines, and (2026-07-01) no Tracker/Timeline either. Standing needs are a **persistent ongoing chat** re-invoking verbs (`06`, deferred); no new scheduler, no new run/state table. Uploads simply populate the input-only KB (existing upload code, no pipeline wrapper). +- ~~The chat agent gets read-only access to pipeline run history~~ → the agent answers "what changed?" by **reasoning over the chat history** (prior tool outputs already in context, `07`), not by re-deriving from run logs or a stored timeline. - **Web search + WebURL crawler are platform-native** (agent tools now; developer platform-API-key access later). The KB is **input-only**. - Deferred (post-MVP): platform scraper implementations (**Phase 8 — Platform actors**, public-data-only first; built in-house on the Phase-3 fetch core) surfaced as **native tools / API endpoints**, public pay-as-you-go API over the crawler/search, public MCP server exposing the KB. Logged-in/account-based scraping deferred beyond that. @@ -124,52 +134,50 @@ flowchart TD The Universal WebURL Crawler is the flagship Type-1 data source (**the moat**). This phase hardens it into an **in-house, best-effort "undetectable, captcha-bypassing" crawler** on a single framework (Scrapling): it standardizes the fetch layer, generalizes proxy support, introduces pay-as-you-go crawl credits, adds **stealth hardening** (geoip coherence, persistent profiles, headed/Xvfb, fonts, humanization, a block classifier + per-domain strategy memory), and adds **opt-in captcha solving**. All tiers plug in behind a single `FetchStrategy` seam returning `CrawlOutcome`, so callers never depend on *how* a page was fetched — that seam is what lets the moat grow (and lets a **deferred paid-unblocker tier** drop in later by config). **Strategy decision (recorded in the log):** CloakBrowser is rejected on licensing and external unblocker APIs are deferred; we hold an in-house bypass moat for ~4–6 months, then move hostile targets to a paid tier if demand/maintenance justifies it. **Logged-in/account-based bypass is out of scope** (public data only this MVP). It is broken into focused subplans: -> **Sequencing within Phase 3 (critical path vs hardening).** Only **`03a` + `03b` + `03c`** are on the **MVP critical path**. The crawler (`03a` `CrawlOutcome`) + proxy provider (`03b`) + `03c` `WebCrawlCreditService` billing seam are consumed by the **capability verbs** (revamp `04a`) that back every chat / automation / REST / MCP path (the crawler is a platform-native tool — see Architecture correction; the old "Phases 4–7 pipelines consume this" framing is superseded). **`03d` (captcha), `03e` (stealth hardening), and `03f` (test harness) are hardening/measurement that nothing downstream imports** — they tune the *same* seam behind the *same* contract, so they can land **in parallel with, or after, Phase 4** without blocking the pivot. Recommended build order: `03a → 03b → 03c` (then revamp `04a` Capabilities → `04b` Access → `05` Intelligence & Timeline), with `03e → 03d → 03f` slotted in whenever crawler robustness is prioritized. +> **Sequencing within Phase 3 (critical path vs hardening).** Only **`03a` + `03b` + `03c`** are on the **MVP critical path**. The crawler (`03a` `CrawlOutcome`) + proxy provider (`03b`) + `03c` `WebCrawlCreditService` billing seam are consumed by the **capability verbs** (`04`) that back every chat / automation / REST / MCP path (the crawler is a platform-native tool — see Architecture pivot; the old "Phases 4–7 pipelines consume this" framing is superseded). **`03d` (captcha), `03e` (stealth hardening), and `03f` (test harness) are hardening/measurement that nothing downstream imports** — they tune the *same* seam behind the *same* contract, so they can land **in parallel with, or after, Phase 4** without blocking the pivot. Recommended build order: `03a → 03b → 03c` (then `04` Capabilities → `05` Access → `07` Orchestration), with `03e → 03d → 03f` slotted in whenever crawler robustness is prioritized. - **`03a-crawler-core.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `5c36cd3`; crawler moved to `app/proprietary/web_crawler/`, `impersonate="chrome"` + `solve_cloudflare=True` shipped)* — Standardize the fetch layer on Scrapling. **Remove Firecrawl entirely** (no other frameworks). Define crisp per-URL success/empty/failure semantics, keep Trafilatura extraction, and expose a single billable "successful crawl" signal (one unit per URL that yields usable content, regardless of how many internal fallback tiers ran). -- **`03b-proxy-expansion.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `6226012`)* — Add a BYO `CustomProxyProvider` (the only new provider — **no branded vendors**) alongside `anonymous_proxies`, selectable via a **single, app-wide** `Config.PROXY_PROVIDER`. Add bounded client-side rotation+retry via Scrapling's `ProxyRotator`/`is_proxy_error` **only** when the active provider is pool-backed (`CUSTOM_PROXY_URLS`); single-endpoint providers (incl. `anonymous_proxies`) stay the default and no-op the retry. **No per-connector/per-crawl selection** (one provider app-wide); a per-capability/per-Tracker override is left as a no-op seam (revamp `04a`/`05`), not built. -- **`03c-crawl-billing.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `17bdb0682`)* — Charge crawl credits at **$1 / 1000 successful requests = 1000 micro-USD per successful crawl** (config-driven via `WEB_CRAWL_MICROS_PER_SUCCESS`, retunable with no code change), drawn from the existing credit wallet (`credit_micros_balance`), gated by a new `WEB_CRAWL_CREDIT_BILLING_ENABLED` flag (off for self-hosted). Billing surfaces (**reframed** by the Architecture correction + revamp — see log): the charge moves to the **capability executor** (revamp `04a`), so **every** caller meters uniformly — chat, **automation/recurring** (the confirmed gap: `run_agent_task` establishes no turn accumulator today, so `scrape_webpage` bills nothing under automations), REST, MCP, and external-cron — billing the **workspace owner** via `WebCrawlCreditService`. The interactive **chat turn accumulator** becomes an *optional presentation fold* (so chat still shows the crawl line on the turn bill), **not** the charging mechanism. The original **connector/pipeline-indexer** billing branch (`webcrawler_indexer`) is now **vestigial** (WEBCRAWLER connector retired, no pipelines) — kept dormant, not deleted. The `WebCrawlCreditService` (mirrors `EtlCreditService`'s gate → `check_credits` → `charge_credits`) is unchanged. No DB migration (uses the existing free-form `web_crawl` usage_type). +- **`03b-proxy-expansion.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `6226012`)* — Add a BYO `CustomProxyProvider` (the only new provider — **no branded vendors**) alongside `anonymous_proxies`, selectable via a **single, app-wide** `Config.PROXY_PROVIDER`. Add bounded client-side rotation+retry via Scrapling's `ProxyRotator`/`is_proxy_error` **only** when the active provider is pool-backed (`CUSTOM_PROXY_URLS`); single-endpoint providers (incl. `anonymous_proxies`) stay the default and no-op the retry. **No per-connector/per-crawl selection** (one provider app-wide); a per-capability override is left as a no-op seam (`04`), not built. +- **`03c-crawl-billing.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `17bdb0682`)* — Charge crawl credits at **$1 / 1000 successful requests = 1000 micro-USD per successful crawl** (config-driven via `WEB_CRAWL_MICROS_PER_SUCCESS`, retunable with no code change), drawn from the existing credit wallet (`credit_micros_balance`), gated by a new `WEB_CRAWL_CREDIT_BILLING_ENABLED` flag (off for self-hosted). Billing surfaces (**reframed** by the Architecture pivot + revamp — see log): the charge moves to the **capability executor** (`04`), so **every** caller meters uniformly — chat, ongoing automation, REST, and MCP — billing the **workspace owner** via `WebCrawlCreditService`. The interactive **chat turn accumulator** becomes an *optional presentation fold* (so chat still shows the crawl line on the turn bill), **not** the charging mechanism. The original **connector/pipeline-indexer** billing branch (`webcrawler_indexer`) is now **vestigial** (WEBCRAWLER connector retired, no pipelines) — kept dormant, not deleted. The `WebCrawlCreditService` (mirrors `EtlCreditService`'s gate → `check_credits` → `charge_credits`) is unchanged. No DB migration (uses the existing free-form `web_crawl` usage_type). - **`03e-stealth-hardening.md`** — The in-house "undetectable" layer on top of Scrapling's default patchright-Chromium stealth. **Geoip coherence** (match browser `locale`/`timezone_id` to the proxy's exit geo), **fingerprint flags** (`hide_canvas`/`block_webrtc`), **persistent per-domain profiles** (`user_data_dir`), **headed execution under Xvfb**, **real fonts** in the worker image, and **DIY behavioral humanization** via `page_action` (the Chromium engine has no built-in `humanize`). Adds a **block classifier** (label Cloudflare/DataDome/Kasada/captcha/empty from the response) + **per-domain strategy memory** (Redis, no migration) so the ladder learns the known-good tier per domain. Defines — but does **not** build — the **deferred paid-unblocker `FetchStrategy`** (ZenRows/ScrapFly/Bright Data) as the config-flagged escape hatch, with its own (later) billing. Honest ceiling: defeats Cloudflare + the moderate long tail, **not** top-tier behavioral fingerprinting (DataDome/Kasada/reCAPTCHA-Enterprise) — that's the deferred tier. - **`03d-captcha-solving.md`** *(✅ IMPLEMENTED — `ci_mvp`; `captchatools` page_action + per-attempt `web_crawl_captcha` billing, off by default)* — Covers the captcha types Scrapling does **not** (reCAPTCHA v2/v3, hCaptcha, image) via `captchatools`, **opt-in + off by default**. `captchatools` is **itself** the provider registry (`new_harvester(solving_site=…)` across capmonster/2captcha/anticaptcha/capsolver/captchaai), so we do **not** rebuild a provider hierarchy — our layer is thin: config resolution + a StealthyFetcher `page_action` that detects the sitekey, harvests a token (egressing from the **same** proxy IP as the crawl), and injects it. Scrapling already handles Cloudflare Turnstile (`03a`), and `page_action` runs **after** `solve_cloudflare`, so the tiers compose. The **billing asymmetry** is now **resolved**: a **separate per-attempt** `web_crawl_captcha` unit (solvers charge per attempt regardless of crawl success), attached via a `WebCrawlCreditService` seam in `03c`. `ErrNoBalance` stops solving (no retry loop → avoids IP bans). Requires a paid solver account. - **`03f-undetectability-testing.md`** *(✅ IMPLEMENTED + first baseline — `ci_mvp`, MANUAL-only, no CI gating; lives under `app/proprietary/web_crawler/testbench/`)* — A **manual scorecard harness** that drives the real Scrapling tiers against the industry-standard detection + sandbox sites (modeled on CloakBrowser's `bin/cloaktest` suite) to **quantify the free-stack ceiling** over time. Two labeled axes: **Suite S (stealth/anti-bot)** — browser-tier (`bot.sannysoft`, `bot.incolumitas`, CreepJS, `deviceandbrowserinfo`, FingerprintJS demo, reCAPTCHA-v3 score, `fingerprint-scan`/Castle.js, `browserscan`, **`cloudflare-challenge`** exercising `solve_cloudflare`, **`iphey`** geoip-coherence), HTTP/TLS-tier JA3/JA4 parity (`tls.peet.ws`, **informational** not a gate), and proxy/leak checks (`httpbin/ip`, WebRTC/DNS); **Suite E (extraction correctness)** — toscrape/scrapethissite sandboxes for the HTTP vs JS (DynamicFetcher) tiers. **Every detection site is auto-graded from its real DOM verdict** (parsers written against captured dumps; `INFO` reserved for TLS/IP/manual rows). Reuses `03d`'s `page_action`+closure-cell mechanism. Adopts CloakBrowser's bars as **aspirational** (sannysoft 0 fails, CreepJS ≤30%, reCAPTCHA ≥0.7) while recording **our actual numbers as the baseline** (the whole `results/` tree is **gitignored**, run-local). **First baseline (2026-06-30, headless, rotating residential, captcha OFF): Suite S 6 PASS / 4 FAIL** — PASS sannysoft, deviceandbrowserinfo, reCAPTCHA-v3 (0.9), BrowserScan, fingerprint-scan (35/100), cloudflare-challenge; FAIL CreepJS (`hasHeadlessWorkerUA` worker-UA leak → `03e` Slice-B candidate), FingerprintJS Pro (commercial ceiling), iphey ("Unreliable" → expected geoip-coherence fix), incolumitas (legacy `fpscanner WEBDRIVER` only). The scorecard is the documented **trigger** for flipping `03e`'s deferred paid-unblocker tier. -### Phase 4 — Capabilities & Access (backend) [`subplans: revamp/04a–04b`] +### Phase 4 — Capabilities (backend) [`subplan: 04-capabilities.md`] -> **Canonical subplans:** [`revamp phases 4-7/04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md) + [`04b-access.md`](revamp%20phases%204-7/04b-access.md). Together they ship **Product A** (stateless utility) — revenue day one. **Build `04a` first** — every later phase calls the registry. +> **Canonical subplan:** [`04-capabilities.md`](04-capabilities.md). **Build first** — every later phase calls the registry. -- **`revamp/04a-capabilities.md`** — Turn Acquisition (crawler + search + a future Maps actor) into a **small set of typed, callable verbs** over a single **capability registry**. MVP verbs = **only `web.scrape(urls[])`** (array-friendly, inline-or-job) **+ `web.discover(query, top_k)`** (SearXNG/Linkup/Baidu). **Platform-specific scrapers** (Maps/LinkedIn/Amazon/… — *examples, none committed*) are a family of **individual endpoints** added later: each is just another verb → agent tool + dev API-key REST endpoint, so `04a` only ships **contract stubs** for them, not executors. Uniform result **envelope** (`completed | pending + job_id`); slow calls use a **thin job record** + existing Celery (not a resurrected `pipeline_runs`). Each verb declares a **`billing_unit`**; charging is **delegated to the billing service** (03c is the first provider) and happens **at the executor** so every door meters uniformly. New Apache-2 package `app/capabilities/`; Maps extractor stays proprietary. -- **`revamp/04b-access.md`** — Expose the registry through **three generated doors** (order: chat → REST+API-key → MCP), each the same thin adapter (`parse → validate → authn/authz → 03c meter-gate → same executor → serialize → envelope`). **Natural language is the only human-facing surface** (users never name verbs); an **intent router** classifies one-shot (Product A) vs standing-concern (Product B). Slow chat verbs reuse the shipped `deliverable_wait` poll-until-terminal + live-card path. This is where the **BYO-`MCP_CONNECTOR` routing-gap fix** (from the old `04a`) lands (consume-user-MCP). +- Turn Acquisition (crawler + search + Maps actor) into a **small set of typed, callable verbs** over a single **capability registry** — `web.scrape(urls[])`, `web.discover(query, top_k)`, `maps.search/place/reviews`. Each verb is a **direct async call that returns cleaned, AI-ready data** (no job store, no envelope, no polling). `maps.*` returns typed structured objects, not markdown. Each declares a **`billing_unit`**; charging is delegated to the billing service (03c first provider) at the executor so every door meters uniformly. New Apache-2 package `app/capabilities/`; Maps extractor stays proprietary. **Locked model (MVP):** -- **Capabilities replace connectors as data sources.** `web.scrape` / `web.discover` (MVP) — and later per-platform scraper verbs — are *call → data → bill* verbs — no `SearchSourceConnector` row, no KB write, no schedule. -- **Surviving connectors = file/KB-input + BYO MCP.** File/KB-input (Google Drive native+Composio, OneDrive, Dropbox, uploads) put the user's personal files into the **input-only** KB. BYO `MCP_CONNECTOR` (act in chat) is the only functional Type-2; its routing-gap fix lands in `revamp/04b`. **`WEBCRAWLER_CONNECTOR` RETIRED.** The old connector two-type taxonomy (`04a-connector-category`) is **demoted to backward-compat hygiene**. +- **Capabilities replace connectors as data sources.** `web.*` / `maps.*` are *call → cleaned data → bill* verbs — no `SearchSourceConnector` row, no KB write, no schedule. +- **Surviving connectors = file/KB-input + BYO MCP.** File/KB-input (Google Drive native+Composio, OneDrive, Dropbox, uploads) put the user's personal files into the **input-only** KB. BYO `MCP_CONNECTOR` (act in chat) is the only functional Type-2; its routing-gap fix lands in `05`. **`WEBCRAWLER_CONNECTOR` RETIRED.** The old connector two-type taxonomy (`04a-connector-category`) is **demoted to backward-compat hygiene**. - **Deprecated (`MIGRATING`, off for MVP):** every branded native — indexers (Notion, GitHub, Confluence, BookStack, Elasticsearch) + act-only (Slack, Teams, Linear, Jira, ClickUp, Airtable, Discord, Gmail, Google Calendar, Luma, + Composio Gmail/Calendar). Existing rows + already-indexed KB docs stay **searchable**; new-create/`/index`/periodic/subagents off. Real MCP migration is post-MVP. - **Disabled (`DISABLED`):** Obsidian (plugin push) and Circleback (meeting webhook). - Web search: **drop all 5 search-API `connector_type` values** (`SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API`); survivors (SearXNG/Linkup/Baidu) become the `web.discover` provider set keyed by **platform env** (Linkup/Baidu keys move off per-connector `config`). Destructive migration deletes the 5 types' rows. (Absorbs the old `04b-source-discovery`.) -- The deferred "coming soon" platforms (LinkedIn, Amazon, Google, Instagram, Zillow/Redfin, Walmart, eBay, Crunchbase, TikTok, Indeed/Glassdoor) re-slot as **Phase 8 native verbs / API endpoints**, not Type-1 KB connectors. +- The **Google Maps actor is net-new**, proprietary, built as a separate effort; `maps.*` are contracts against it. Other "coming soon" platforms (LinkedIn, Amazon, Instagram, Zillow/Redfin, Walmart, eBay, Crunchbase, TikTok, Indeed/Glassdoor) re-slot as later namespaces, not Type-1 KB connectors. - Frontend connector/capability UI restructure is DEFERRED (frontend umbrella). -### Phase 5 — Intelligence & Timeline (backend) [`subplans: revamp/05a–05b`] +### Phase 5 — Access (backend) [`subplan: 05-access.md`] -> **Canonical subplans:** [`revamp phases 4-7/05a-timeline.md`](revamp%20phases%204-7/05a-timeline.md) + [`05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md). Ships the **Product B engine**. **Depends on `04a`** (the verbs the loop calls). **Build `05a` (tables) before `05b` (the loop that writes them).** This *replaces* the old Pipelines Phases 5–7 — the durable CI state, and **the moat**. +> **Canonical subplan:** [`05-access.md`](05-access.md). **Build after `04`.** Together `04 + 05` ship the scraper-API product — revenue day one. -- **`revamp/05a-timeline.md`** — The **3-store delta Timeline**: `tracked_entities` (stable identity, written once), `entity_current_state` (latest values + `content_hash` + `last_checked_at`, overwritten each run), `entity_changes` (append-only material deltas: `{field:{from,to}}`, `materiality`, `decided_by`, `source_ref`). **Store deltas, no change → no row** (storage ∝ rate of change). Explicitly **not the KB, not `automation_runs`, not a resurrected `pipeline_runs`.** New CI-owned Apache-2 tables + Alembic migration; write/read API for the `05b` loop and the read side. **This is where "diffable run history" actually lives** (superseding the old "structured `automation_runs.output`" idea). -- **`revamp/05b-intelligence.md`** — The `Tracker` primitive + the **hot loop** `refresh(tracker)`: crawl (call the bound verb) → **content-hash pre-check** (identical → stamp `last_checked_at`, stop, no LLM) → **fill** raw data to the locked `field_schema` (structured output; extras → `notable_signals`) → **diff** (code) → **judge — the materiality split** (deterministic numeric/clear rules in code for free; agent only on ambiguous) → **append** a change + overwrite current state, else only `last_checked_at`. Setup = an **agent-designed, human-locked, versioned schema** flow (sample-fetch → draft `definition` → review/lock). **No new run table:** recurring audit/idempotency ride `AutomationRun`, manual rides the chat job record; **billing idempotency is per-capability-call** + the hash pre-check. Optional **CI context folder** (user files uploaded in a CI chat go into the KB as normal, routed to a folder, and can feed the judge). New Apache-2 package `app/intelligence/`. +- Expose the registry through **three generated doors** — **REST + API key** (public day one), an **MCP server** (fast-follow), and **chat tools** — each the same thin adapter (`parse → validate → authn/authz → 03c meter-gate → same executor → serialize → cleaned data`). The executor returns directly; there is **no envelope, no `deliverable_wait` polling**. Async results (e.g. the `06` ongoing mode) reach the client via **write-then-sync / stream** (existing SSE or a Zero-published table). +- **Natural language is the human surface** (raw verbs only on REST/MCP); the chat door classifies **one-shot** vs **keep-watching** and hands keep-watching to `06`. Two MCP directions kept distinct: **serve** our verbs as an MCP server (new door) vs **consume** the user's BYO `MCP_CONNECTOR` (the old-`04a` routing-gap fix lands here). -### Phase 6 — Triggers (backend) [`subplan: revamp/06-triggers.md`] +### Phase 6 — Ongoing-Automation (backend) [`subplan: 06-ongoing-automation.md`] -> **Canonical subplan:** [`revamp phases 4-7/06-triggers.md`](revamp%20phases%204-7/06-triggers.md) — the thinnest phase. **Depends on `05b`** (`refresh(tracker)`). Decides **when** a Tracker refreshes; Intelligence never knows which trigger fired. +> **Canonical subplan:** [`06-ongoing-automation.md`](06-ongoing-automation.md) — **⚠️ design deferred.** The periodic mechanism is designed separately, after `04`/`05`/`07`. **Depends on `04`** (the verbs it re-invokes) and `05` (the chat surface + delivery channel). -- Adapters: **manual** (user "refresh now" via chat tool / REST), **agent** (the agent calls `refresh` as a tool), **external cron** (`POST /v1/trackers/{id}/refresh`, zero infra on us), and the **CI automation action** — recurrence reusing the **existing automations**: its hardened **schedule selector** (no new scheduler) + **`AutomationRun`** (no new run table). **Alert delivery is separate** — persist deltas to the Timeline, then emit an in-app notification via `app/notifications/` (automations have no delivery path; `run.output` is unwritten). Guard concurrency with a **per-Tracker lock** (the automation PENDING-gate is non-atomic). A single `refresh_tracker` action, **not** a new automation shape. -- **Decoupling preserved:** CI core (`refresh` + Timeline) runs with **zero** automations dependency (manual/agent/external-cron); automations is the **optional** in-app recurrence+alert adapter. This is the correct home for the old "wire a run-now trigger" item. +- Support **"keep watching"** with no stateful storage: a **persistent, ongoing chat** where the agent periodically re-invokes scraper verbs and drops results into the session; the agent reports "what's new" by reading the chat history. Open questions (resolved together): periodicity driver, delivery channel (SSE vs Zero-published table), context-window limit, loop owner, stop/cost controls. -### Phase 7 — Orchestration (backend) [`subplan: revamp/07-orchestration.md`] +### Phase 7 — Orchestration (backend) [`subplan: 07-orchestration.md`] -> **Canonical subplan:** [`revamp phases 4-7/07-orchestration.md`](revamp%20phases%204-7/07-orchestration.md) — the human-facing brain. **Build last** (atop `04`–`06`). We plug into the shipped multi-agent runtime; we don't rebuild it. +> **Canonical subplan:** [`07-orchestration.md`](07-orchestration.md) — the human-facing brain. **Build last** (atop `04`/`05`, and `06` for the ongoing mode). We plug into the shipped multi-agent runtime; we don't rebuild it. -- Ship the net-new builtin **`intelligence_agent`** subagent (peer to `research`/`deliverables`): **intent routing** (one-shot vs standing concern, one clarifying question when ambiguous), **verb composition** (`web.discover → web.scrape`, `maps.search → maps.place → maps.reviews`), **Tracker crafting** (the `05b` schema-design flow in chat), and **decision-grounded answering** (read the Timeline via `query_timeline`, not chat history). -- Toolset: registry-backed capability verbs (shared with `research`) + Tracker/Timeline tools (`craft_tracker`/`lock_tracker`/`refresh_tracker`/`query_timeline`/`list_trackers`); slow verbs reuse `deliverable_wait`. Tools follow the shipped `scrape_webpage` shape (executor + door + 03c billing). +- Ship the net-new builtin **`intelligence_agent`** subagent (peer to `research`/`deliverables`): **intent routing** (one-shot vs keep-watching, one clarifying question when ambiguous), **verb composition** (`web.discover → web.scrape`, `maps.search → maps.place → maps.reviews`), and **"what changed" from the chat history** (re-invoke a verb, compare against prior tool outputs already in context — no timeline, no diff store). +- Toolset: registry-backed capability verbs (shared with `research`) + a deferred `start_watch` handoff to `06`. Tools follow the shipped `scrape_webpage` shape (executor + door + 03c billing), **direct-return** (no `deliverable_wait`). ### (Future) Platform API-key + public MCP @@ -177,7 +185,7 @@ Platform **API-key** access to the capability verbs + a public **MCP server** ex ### Phase 8 — Platform actors (FUTURE — post-MVP, public data only) [`subplan: TBD`] -NOT planned in this umbrella; recorded so the Phase-3 architecture stays aimed at it. Once the hardened fetch core (Phases 3a–3e) is solid, **platform scrapers** — a family of **individual scraping endpoints** (Google Maps/Local, LinkedIn public profiles/companies, Amazon products, etc. — see "Platform connector research list"; **all examples, none committed**) — layer **on top** of it. **Each is simply another capability verb** (`revamp/04a`), so registering one automatically yields (a) an **agent tool** and (b) a **dev-callable REST endpoint behind the platform API key** — same executor, same billing, no new machinery. Each is a per-platform structured extractor built in-house "Apify-style" on the `03a` crawler core (proxies + `03e` hardening + `03d` captcha), callable by the chat agent, by automations, and by the future platform API key. Billed per call via `WebCrawlCreditService`. +NOT planned in this umbrella; recorded so the Phase-3 architecture stays aimed at it. Google Maps is the **committed MVP actor** (a `maps.*` namespace, Phase 4). Beyond it, more **platform scrapers** — a family of **individual scraping endpoints** (LinkedIn public profiles/companies, Amazon products, etc. — see "Platform connector research list"; **examples, none committed**) — layer **on top** of the hardened fetch core once it's solid. **Each is simply another capability verb** (`04`), so registering one automatically yields (a) an **agent tool** and (b) a **dev-callable REST endpoint behind the platform API key** — same executor, same billing, no new machinery. Each is a per-platform structured extractor built in-house "Apify-style" on the `03a` crawler core (proxies + `03e` hardening + `03d` captcha). Billed per call via `WebCrawlCreditService`. - **Public data only** at first — discovery/extraction of publicly visible pages. **Logged-in/account-based bypass is explicitly deferred** beyond Phase 8's first cut; it needs sticky/static proxies + credential management (`03b` static-proxy hand-off) and is the higher-risk, later workstream. - The **deferred paid-unblocker tier** (`03e §8`) is the fallback for any platform whose anti-bot exceeds the in-house ceiling. @@ -189,7 +197,7 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta - Frontend rename + i18n: route segment `[search_space_id]` -> `[workspace_id]`, `search-space-settings/` -> `workspace-settings/`, TS types, api services, Jotai atoms, components, cache keys, and "Workspace" copy across 5 locales (`messages/{en,zh,es,pt,hi}.json`), plus frontend Zero schema rename. - Satellite/client apps + docs rename: `surfsense_desktop`, `surfsense_obsidian`, `surfsense_browser_extension`, `surfsense_evals`, README/docs. - Connector two-type UI: restructure `connector-popup` and `connector-constants.ts` into the two labeled types. -- CI + automations + positioning UI (**replaces the superseded "Pipelines UI"**): the CI experience (Tracker crafting/review-&-lock, Timeline/change views, "refresh now"), automation create/list/run-history, `web.discover` source-suggestion UX, input-only KB framing (uploads + file connectors), MCP connector management, "coming soon" platform cards, "NotebookLM for Competitive Intelligence" copy. +- Scraper-API + chat + positioning UI (**replaces the superseded "Pipelines UI"**): the developer surface for the scraper endpoints (API keys, usage/billing), the chat experience (one-shot answers + "keep watching" ongoing chats), `web.discover` source-suggestion UX, input-only KB framing (uploads + file connectors), MCP connector management, "coming soon" platform cards. ## Open items to confirm during subplanning @@ -197,11 +205,12 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta - ~~Whether existing connector periodic-indexing config is migrated into Pipelines or coexists during MVP.~~ **MOOT** (Architecture correction — no pipelines). Connector periodic indexing stays as-is for file sources; there is no pipeline scheduler to coexist with. - ~~Chat agent run-history access: tool vs middleware injection vs both (default: tool).~~ **MOOT** (Architecture correction — no pipeline runs). The agent's recurring work is automations, whose history is already persisted + queryable (`automation_runs`). - ~~Type-2 MCP migration depth~~ RESOLVED (Phase 4): branded natives are tagged `MIGRATING` and turned OFF for MVP (not re-pointed to MCP yet); only the generic `MCP_CONNECTOR` is a functional Type-2. Real MCP re-pointing is post-MVP. -- **Revamp cross-phase forks (from `revamp/00-overview.md`, decide during subplanning):** (1) schema **review-&-lock** UX pre-frontend — pure-chat confirmation for MVP? (`05b`); (2) Timeline ORM home — `app/db.py` vs a dedicated `app/timeline/` (`05a`); (3) recurrence = CI **action** on automations (default) vs a thin CI automation **shape** (fallback) (`06`); (4) is `web.discover` metered or free (`04a`); (5) MCP-serve auth depth — OAuth 2.1 vs bearer (`04b`). +- **Revamp open items (decide during subplanning):** (1) the **Ongoing-Automation mechanism** (`06`, deferred) — periodicity driver, delivery channel (SSE vs Zero-published table), context-window limit, stop/cost controls; (2) how large `web.scrape` / `maps.reviews` responses are bounded/streamed (`04`); (3) is `web.discover` metered or free (`04`); (4) MCP-serve auth depth — OAuth 2.1 vs bearer (`05`); (5) public REST rate-limiting / abuse posture (`05`). ## Resolved decisions log -- **REVAMP ADOPTED as canonical (2026-06-30) — Phases 4–7 re-cast; `revamp phases 4-7/` is the source of truth.** A principal-engineer review found the flat `04a–07` files and my earlier "Phase 5′" framing had been overtaken by a more complete engineer draft in [`revamp phases 4-7/`](revamp%20phases%204-7/00-overview.md). We adopt the revamp: **Phase 4 = Capabilities & Access** (typed verbs + generated doors; old `04a-connector-category` demoted to hygiene, old `04b-source-discovery` absorbed as `web.discover`), **Phase 5 = Intelligence & Timeline** (the `Tracker` + 3-store delta Timeline — the moat), **Phase 6 = Triggers** (reuse automations via a CI action), **Phase 7 = Orchestration** (`intelligence_agent`). The three architecture-correction decisions land in their real homes and one is revised: (1) **bill at the capability executor** — a code-verified gap review confirmed `run_agent_task` (automation `agent_task`) sets up **no turn accumulator**, so today `scrape_webpage` bills nothing under automations; billing moves to the executor so chat/automation/REST/MCP/cron meter uniformly (chat turn accumulator becomes optional presentation). (2) **"diffable run history" = the Timeline delta store** (`revamp/05a`), **superseding** the earlier "structured `automation_runs.output`" decision. (3) **"run now" = a Trigger adapter** (`revamp/06`). The flat `04a/04b/05/06/07` files are retained with redirect banners; **`revamp/` filenames win** on any number collision. Build order: `04a → 04b → 05a → 05b → 06 → 07`. +- **PIVOT (2026-07-01) — scraper-APIs-first, stateless, chat-native; Tracker/Timeline/Triggers dropped.** The product is the **scraper capabilities** (`web.*` + `maps.*` typed verbs → cleaned, AI-ready data), exposed as **REST + API key / MCP / chat** generated from one registry — premium, OSS, self-hostable, the revenue driver. **Stateless:** no Timeline (3-store delta), no `Tracker`, no diff/materiality engine, no stored `entity_changes`; memory = the **chat history** (the agent reasons over prior tool outputs for "what changed"). **Direct calls:** capabilities return their result directly — no job store, no `completed | pending + job_id` envelope, no `deliverable_wait` polling; async delivery is **write-then-sync / stream**. **Automation = a persistent ongoing chat** re-invoking verbs (replaces `Triggers`; mechanism **design-deferred**). KB stays **input-only**. Phase re-cast: **04 Capabilities · 05 Access · 06 Ongoing-Automation (deferred) · 07 Orchestration** (`intelligence_agent`). Subplans `05a-timeline`/`05b-intelligence`/`06-triggers` **removed** (git history); `04a`/`04b` folded into `04`/`05`. This **supersedes** the 2026-06-30 revamp's Tracker/Timeline/Triggers decisions below. +- **REVAMP ADOPTED as canonical (2026-06-30) — Phases 4–7 re-cast; `revamp phases 4-7/` is the source of truth.** A principal-engineer review found the flat `04a–07` files and my earlier "Phase 5′" framing had been overtaken by a more complete engineer draft (then in a `revamp phases 4-7/` folder, since flattened into `04`–`07`). We adopt the revamp: **Phase 4 = Capabilities & Access** (typed verbs + generated doors; old `04a-connector-category` demoted to hygiene, old `04b-source-discovery` absorbed as `web.discover`), **Phase 5 = Intelligence & Timeline** (the `Tracker` + 3-store delta Timeline — the moat), **Phase 6 = Triggers** (reuse automations via a CI action), **Phase 7 = Orchestration** (`intelligence_agent`). The three architecture-correction decisions land in their real homes and one is revised: (1) **bill at the capability executor** — a code-verified gap review confirmed `run_agent_task` (automation `agent_task`) sets up **no turn accumulator**, so today `scrape_webpage` bills nothing under automations; billing moves to the executor so chat/automation/REST/MCP/cron meter uniformly (chat turn accumulator becomes optional presentation). (2) **"diffable run history" = the Timeline delta store** (`revamp/05a`), **superseding** the earlier "structured `automation_runs.output`" decision. (3) **"run now" = a Trigger adapter** (`revamp/06`). The flat `04a/04b/05/06/07` files are retained with redirect banners; **`revamp/` filenames win** on any number collision. Build order: `04a → 04b → 05a → 05b → 06 → 07`. - **ARCHITECTURE CORRECTION (2026-06-30) — Pipelines dropped; automations + input-only KB adopted.** The "sync into KB → operate over it" paradigm was judged wrong for CI. Verified against `references/opencode` (live tool-fetched context + persisted sessions, no scraped corpus; `tool/websearch.ts` live-crawls at query time). Decisions: (1) **KB is input-only** — user's personal files/context (uploads + Drive/Dropbox/OneDrive); nothing scrapes into it. (2) **Web search + WebURL crawler are platform-native tools** (already `web_search`/`scrape_webpage`), later a developer platform-API-key surface — NOT connectors, NOT pipelines. (3) **Connectors = MCP tools + file/KB-input connectors**; all branded natives → MCP. (4) **Automations are the scheduling + run-history substrate** (an `agent_task` run already invokes the full chat agent with the crawler/search tools on cron/event and persists `automation_runs`). (5) **`WEBCRAWLER_CONNECTOR` RETIRED.** Consequences: **Phases 5/6/7 SUPERSEDED** (docs kept with banners); **Phase 4a reframed** (drop `is_pipeline_eligible`; Type-1 = file/KB-input only; WEBCRAWLER retired); **Phase 4b stands**; new **Phase 5′** automation-enhancement workstream (bill automation-run crawls; structured `run.output`; wire "run now"); **Phase 8** actors re-slot as native tools/API. `03c` connector-indexer billing branch becomes **vestigial** (billing lives on chat-turn / automation-run / API paths). - ~~Web search APIs (SearXNG/Linkup/Baidu): repurposed as source-discovery helper for the WebURL Crawler (suggest URLs for pipelines)~~ — still repurposed as a source-discovery helper (04b), but "for pipelines" → "for the user / the native crawler tool" (pipelines dropped); not a standalone connector type. - Tavily and Serper: REMOVED from the search infra. They are dropped as search providers entirely (not repurposed). Phase 4's source-discovery endpoint must build only on the remaining providers (SearXNG, Linkup, Baidu). @@ -239,17 +248,10 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta | 3 | `03e-stealth-hardening.md` | **Slice A IMPLEMENTED** (`ci_mvp`) — stealth kwargs builder + geoip coherence (**proprietary** `app/proprietary/web_crawler/stealth.py`) + additive block classifier (Apache-2 `app/utils/crawl/`; `CrawlOutcome.block_type`, incl. static-tier 4xx) + Xvfb/fonts in image; Slices B/C deferred (WebGL spoof, humanization, persistent profiles, strategy memory, paid-unblocker) | | 3 | `03d-captcha-solving.md` | **IMPLEMENTED** (`ci_mvp`) — `captchatools` page_action (proprietary) + Apache-2 config + per-attempt `web_crawl_captcha` billing; off by default | | 3 | `03f-undetectability-testing.md` | **IMPLEMENTED** (`ci_mvp`) — manual scorecard under the **proprietary boundary** at `app/proprietary/web_crawler/testbench/` (`python -m app.proprietary.web_crawler.testbench`); Suite S (stealth, shipped builder) + Suite E (extraction via real `crawl_url`) + scorecard JSON/MD baseline diff | -| 4 | `revamp phases 4-7/04a-capabilities.md` | **CANONICAL** · drafted — capability registry + MVP verbs `web.scrape`/`web.discover` (per-platform scrapers = later drop-ins); bill at executor. **← build next** | -| 4 | `revamp phases 4-7/04b-access.md` | **CANONICAL** · drafted — generated chat/REST/MCP doors + intent router + BYO-MCP routing fix | -| 5 | `revamp phases 4-7/05a-timeline.md` | **CANONICAL** · drafted — 3-store delta Timeline (the moat) | -| 5 | `revamp phases 4-7/05b-intelligence.md` | **CANONICAL** · drafted — `Tracker` + refresh hot loop + materiality split | -| 6 | `revamp phases 4-7/06-triggers.md` | **CANONICAL** · drafted — refresh clock; recurrence = CI action on automations | -| 7 | `revamp phases 4-7/07-orchestration.md` | **CANONICAL** · drafted — `intelligence_agent` subagent | -| — | `revamp phases 4-7/00-overview.md` · `00b-pipeline-diagrams.md` | revamp map + reconciliation + end-to-end flow diagrams | -| 4 | `04a-connector-category.md` *(flat)* | ↪ **REDIRECT** — demoted to backward-compat hygiene; superseded by `revamp/04a`+`04b` | -| 4 | `04b-source-discovery.md` *(flat)* | ↪ **REDIRECT** — absorbed as the `web.discover` verb in `revamp/04a` | -| 5 | `05-pipelines-model.md` *(flat)* | ❌ **SUPERSEDED** — replaced by `revamp/05a`+`05b` (Tracker/Timeline) | -| 6 | `06-pipelines-exec.md` *(flat)* | ❌ **SUPERSEDED** — replaced by `revamp/05b` (hot loop) + `revamp/06` (triggers) | -| 7 | `07-upload-pipeline-kb.md` *(flat)* | ❌ **SUPERSEDED** — uploads populate the input-only KB (no pipeline wrapper) | +| — | `00b-pipeline-diagrams.md` | end-to-end flow diagrams (companion to this umbrella) | +| 4 | `04-capabilities.md` | **CANONICAL** · drafted — capability registry + verbs `web.scrape`/`web.discover`/`maps.*` → cleaned data; bill at executor. **← build next** | +| 5 | `05-access.md` | **CANONICAL** · drafted — generated REST/MCP/chat doors + intent router + BYO-MCP routing fix | +| 6 | `06-ongoing-automation.md` | **CANONICAL** · **design deferred** — chat-native "keep watching" (periodic re-invocation) | +| 7 | `07-orchestration.md` | **CANONICAL** · drafted — `intelligence_agent` subagent; "what changed" from chat history | Frontend & client subplans will be added under a separate umbrella later (see "Deferred — Frontend & client phases"). diff --git a/plans/backend/00b-pipeline-diagrams.md b/plans/backend/00b-pipeline-diagrams.md new file mode 100644 index 000000000..d2ffb8be7 --- /dev/null +++ b/plans/backend/00b-pipeline-diagrams.md @@ -0,0 +1,110 @@ +# Pipeline diagrams — end-to-end (scraper-APIs-first, stateless) + +> Visual companion to `00-umbrella-plan.md`. +> Phase refs: `04` Capabilities · `05` Access · `06` Ongoing-Automation (design deferred) · `07` Orchestration. + +## 1. The shape — scraper APIs + a chat agent (no state) + +```mermaid +flowchart LR + subgraph FIXED["FIXED · Phases 1-3 (shipped)"] + ACQ["Acquisition
crawler · proxy · stealth · captcha"] + MET["Metering · 03c"] + ID["Identity / API keys"] + end + + subgraph SCOPE["OUR SCOPE · Phase 04 -> 07"] + CAP["04 Capabilities
web.* · maps.*
cleaned, AI-ready output"] + ACC["05 Access
REST · MCP · chat doors"] + ORC["07 Orchestration
intelligence_agent"] + ONG["06 Ongoing-Automation
(keep-watching · DEFERRED)"] + end + + ACQ --> CAP + MET --> CAP + CAP --> ACC + ID --> ACC + ACC --> ORC + ORC -. "standing need" .-> ONG + ONG -. "re-invokes verbs" .-> CAP + + classDef a fill:#22314f,stroke:#5b7fbf,color:#e6edf7; + classDef b fill:#1f3a2e,stroke:#4f9d76,color:#e6f7ee; + classDef d fill:#3a2f1f,stroke:#bf975b,color:#f7efe6; + class CAP,ACC,ORC a; + class ACQ,MET,ID b; + class ONG d; +``` + +Memory for "what changed" = the chat history. + +## 2. Doors — one registry, three surfaces (generated) + +```mermaid +flowchart TD + REG["04 Capability registry
web.scrape · web.discover · maps.search/place/reviews"] + REG --> R["REST + API keys
(public · OSS self-host · revenue)"] + REG --> M["MCP server
(external agents · fast-follow)"] + REG --> C["Chat tools
(in-app agent, 07)"] + R --> EX["same executor · direct return · 03c billing"] + M --> EX + C --> EX +``` + +## 3. One-shot scrape (the core path) — direct call, no polling + +```mermaid +sequenceDiagram + autonumber + participant U as User / Dev + participant D as Door 05 (REST · MCP · chat) + participant EX as Verb executor 04 + participant ACQ as Acquisition / Maps actor + participant BILL as Billing (03c) + + U->>D: intent (chat) OR raw verb call (REST/MCP) + D->>D: validate input_schema · authn/authz · meter-gate + D->>EX: call executor (same for all doors) + EX->>ACQ: crawl_url / maps / search + ACQ-->>EX: cleaned, AI-ready data + EX->>BILL: charge billing_unit per success + EX-->>D: results (returned directly) + D-->>U: plain-language answer (chat) / JSON (REST/MCP) + Note over U,BILL: Stateless — nothing persists. +``` + +## 4. Intent fork (in the agent, 07) + +```mermaid +flowchart TD + U(["User speaks in natural language"]) --> R{"Intent router
(intelligence_agent prompt, 07)"} + R -->|"find / compare / what is / right now"| A["ONE-SHOT
compose verbs, answer (section 3)"] + R -->|"watch / track / notify me / over time"| B["KEEP-WATCHING
hand to Ongoing-Automation (06)"] + R -->|ambiguous| Q["Ask ONE question:
'just once, or keep watching?'"] + Q -->|once| A + Q -->|keep watching| B + B -.->|DESIGN DEFERRED| X["06 mechanism TBD
(periodicity · delivery · context limits)"] +``` + +## 5. "What changed" — chat history is the memory + +```mermaid +flowchart LR + P["Prior tool outputs
(already in chat context)"] --> AG["intelligence_agent"] + NEW["Fresh verb call (04)"] --> AG + AG --> ANS["'Here's what's new since last time'
(reasoned from chat history)"] +``` + +## 6. Ongoing-Automation (06) — intent only, mechanism deferred + +```mermaid +flowchart LR + W["'Keep watching X'"] --> LOOP["Persistent ongoing chat
periodically re-invokes verbs (04)"] + LOOP --> OUT["results delivered into the session
(write-then-sync / stream)"] + OUT --> MEM["chat history = memory
agent reports what's new"] + LOOP -.->|OPEN| Q["periodicity driver? · delivery channel? · context-window limit?"] + classDef d fill:#3a2f1f,stroke:#bf975b,color:#f7efe6; + class W,LOOP,OUT,MEM d; +``` + +> Section 6 is a **placeholder** — the periodic mechanism is designed separately (see `06-ongoing-automation.md`). diff --git a/plans/backend/02-rename-backend.md b/plans/backend/02-rename-backend.md index c2ec94869..cc56df3f0 100644 --- a/plans/backend/02-rename-backend.md +++ b/plans/backend/02-rename-backend.md @@ -133,7 +133,7 @@ These do not move with a symbol rename; each is decided here. - Frontend route segment `[search_space_id] -> [workspace_id]`, `search-space-settings/`, TS types, atoms, i18n, frontend Zero schema — deferred frontend umbrella. - Satellite apps (desktop, Obsidian, browser extension, evals) + docs — deferred. -- Connector `category` discriminator and `SearchSourceConnector` handling — Phase 4 ([04a-connector-category.md](04a-connector-category.md)). +- Connector `category` discriminator and `SearchSourceConnector` handling — Phase 4 ([04-capabilities.md](04-capabilities.md)). - Enum VALUE migration and observability-key rename — deliberately deferred announced changes. --- diff --git a/plans/backend/03c-crawl-billing.md b/plans/backend/03c-crawl-billing.md index 131ffc64a..ae3073934 100644 --- a/plans/backend/03c-crawl-billing.md +++ b/plans/backend/03c-crawl-billing.md @@ -2,7 +2,7 @@ > **Status: ✅ IMPLEMENTED (`ci_mvp` @ `17bdb0682`).** `WebCrawlCreditService` added (mirrors `EtlCreditService`: gate → `check_credits` → `charge_credits`, plus static `billing_enabled`/`successes_to_micros`). Price is fully config-driven (`WEB_CRAWL_MICROS_PER_SUCCESS`, default `1000` = $1/1000; `WEB_CRAWL_CREDIT_BILLING_ENABLED`, default FALSE) — no hardcoded rate. Indexer wires owner-resolution + pre-flight block + audit-then-charge; both `scrape_webpage` tools fold one success into the turn accumulator (`call_kind="web_crawl"`). 20 new unit tests pass (service, config-driven price, indexer wiring, chat fold); `connector_indexers` suite green (120). Functional e2e (3a/3b/3c) green via `scripts/e2e_phase3_crawl_billing.py` (live proxy egress flip, static-tier crawl, chat fold = 1000µ, indexer bills owner 2×1000µ not trigger). Captcha per-attempt unit (`03d`) seam left open. **No migration** (`web_crawl` rides the existing `TokenUsage.usage_type` String(50)). -> **⚠️ Billing surfaces reframed (2026-06-30 — Architecture correction + revamp adoption, `00-umbrella-plan.md`).** The implemented `WebCrawlCreditService` **stands unchanged**, but *where* it's called shifts, and the **charge moves to the capability executor** (revamp [`04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md)). Pipelines are dropped and `WEBCRAWLER_CONNECTOR` is retired, so the **connector/`webcrawler_indexer`** billing branch (described throughout this doc) is now **vestigial** (dormant, not deleted). The correct model: every `web.*`/`maps.*` verb **charges once per billable success inside its executor**, so all callers meter uniformly — chat, **automation/recurring**, REST, MCP, external-cron. This closes a **code-verified gap**: `run_agent_task` (automation `agent_task`) establishes **no turn accumulator**, so the shipped `scrape_webpage` fold (no-op when `get_current_accumulator()` is `None`) bills **nothing** under automations today. The interactive **chat turn accumulator becomes an optional presentation fold** (still shows the crawl line on a chat turn), **not** the charging mechanism. Read "connector/pipeline crawls" below as historical context. +> **⚠️ Billing surfaces reframed (2026-06-30 — Architecture correction + revamp adoption, `00-umbrella-plan.md`).** The implemented `WebCrawlCreditService` **stands unchanged**, but *where* it's called shifts, and the **charge moves to the capability executor** ([`04-capabilities.md`](04-capabilities.md)). Pipelines are dropped and `WEBCRAWLER_CONNECTOR` is retired, so the **connector/`webcrawler_indexer`** billing branch (described throughout this doc) is now **vestigial** (dormant, not deleted). The correct model: every `web.*`/`maps.*` verb **charges once per billable success inside its executor**, so all callers meter uniformly — chat, ongoing automation, REST, MCP. This closes a **code-verified gap**: `run_agent_task` (automation `agent_task`) establishes **no turn accumulator**, so the shipped `scrape_webpage` fold (no-op when `get_current_accumulator()` is `None`) bills **nothing** under automations today. The interactive **chat turn accumulator becomes an optional presentation fold** (still shows the crawl line on a chat turn), **not** the charging mechanism. Read "connector/pipeline crawls" below as historical context. > Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`. > Depends on `03a-crawler-core.md` (the `CrawlOutcomeStatus.SUCCESS` signal + `crawls_succeeded` counter). Siblings: `03b-proxy-expansion.md`, `03e-stealth-hardening.md`, `03d-captcha-solving.md` (now active). `03d` extends this service with a **separate per-attempt captcha unit** (see §"Captcha billing seam" + `03d §4`). diff --git a/plans/backend/04-capabilities.md b/plans/backend/04-capabilities.md new file mode 100644 index 000000000..0e47fe155 --- /dev/null +++ b/plans/backend/04-capabilities.md @@ -0,0 +1,89 @@ +# Phase 4 — Capabilities (the scraper APIs) + +> Build first; every other phase calls this registry. Sibling: `05-access.md` (the doors). +> Reuses Phases 3a/3b/3c (shipped): `WebCrawlerConnector.crawl_url -> CrawlOutcome`, the proxy/stealth/ +> captcha tiers, and the `WebCrawlCreditService` billing seam. Locate code by symbol/grep. + +## Objective + +Expose the Acquisition engine as a small set of typed, stateless scraper verbs (`web.*`, `maps.*`) that +return cleaned, AI-ready structured data. One capability = one function you call and get data back from. + +## Verb set (MVP) + +Two namespaces: `web.*` (generic crawler) and `maps.*` (Google Maps actor). Future platforms slot in as +their own namespace. + +| Verb | Input → Output (cleaned) | Executes over | Billing unit | +|------|--------------------------|---------------|--------------| +| `web.scrape(urls[])` | `[{url, status, content, metadata}]` | loop `crawl_url` | per success | +| `web.discover(query, top_k)` | `[{url, title, snippet, provider}]` | search providers | per search / free | +| `maps.search(query, location)` | `[place]` | Maps actor | per place | +| `maps.place(place_id\|url)` | `place` | Maps actor | 1 | +| `maps.reviews(place)` | `[review]` (paged) | Maps actor | per page | + +`maps.*` returns typed structured objects (`{name, rating, review_count, hours, price_level, …}`). +`web.scrape` takes a URL array. + +## Execution + +Each verb is a direct async call that returns its cleaned result. Slow verbs (large `web.scrape` arrays, +`maps.reviews` paging) are bounded/streamed at the endpoint. + +## Registry + +One entry per verb, and the single source of truth the doors (`05`) and the agent (`07`) read from: + +``` +Capability { + name # dotted, e.g. "web.scrape", "maps.search" + input_schema # Pydantic + output_schema # Pydantic (cleaned, AI-ready) + executor # async fn; wraps Acquisition / Maps actor; returns data directly + billing_unit # how the billing service charges this call +} +``` + +Adding a verb once lights it up on every door. + +## Billing + +A verb declares a `billing_unit`; charging is delegated to the billing service (`03c` first provider). + +- `web.scrape` → per `SUCCESS` (`web_crawl` unit). +- `maps.*` → new per-place / per-page unit registered with the billing service. +- captcha attempts → existing per-attempt `web_crawl_captcha` unit (`03d`). +- `web.discover` → per-search unit (or free). + +## Location + +New Apache-2 package `app/capabilities/` (registry, schemas, executors) — open-source and self-hostable. +Executors import the proprietary Acquisition engine and Maps extractor (`app/proprietary/…`). + +## Work items + +1. Registry package: `Capability` dataclass + registry, imported by `05` and `07`. +2. `web.scrape` executor: loop `crawl_url` over the URL array → per-URL cleaned rows; `web_crawl` unit. +3. `web.discover` executor: wrap the search-provider core (SearXNG/Linkup/Baidu, env-keyed); its unit. +4. `maps.*` contracts: input/output schemas + executor stubs against the Maps actor. +5. Output normalization: each executor returns AI-ready structured output. +6. Billing seam: honor `billing_unit` via the billing service. + +## Tests + +- `web.scrape([a,b,c])` returns one cleaned row per URL inline; partial failures don't fail the batch. +- Each `SUCCESS` bills one `web_crawl` unit; `EMPTY`/`FAILED` free; disabling the flag no-ops. +- A door and the agent hit the same executor for a verb. +- `web.discover` returns `{url,title,snippet,provider}`; self-disables when no provider key is set. +- `maps.*` returns typed structured objects against fixtures. + +## Out of scope + +- Doors → `05`. Keep-watching mode → `06`. Agent → `07`. +- Google Maps actor build (incl. sourcing legality) — net-new, separate effort; `maps.*` are contracts. +- Additional platform namespaces; recursive site crawl; a generic `web.extract` verb. + +## Open questions + +- `web.discover` metered vs free. +- How large `web.scrape` / `maps.reviews` responses are bounded/streamed. diff --git a/plans/backend/04a-connector-category.md b/plans/backend/04a-connector-category.md deleted file mode 100644 index 7ca37c2cc..000000000 --- a/plans/backend/04a-connector-category.md +++ /dev/null @@ -1,167 +0,0 @@ -# Phase 4a — Connector taxonomy (Type-1/Type-2) + availability gating + MCP routing fix - -> ## ↪ REDIRECT / DEMOTED (2026-06-30) — superseded by the revamp -> **The canonical Phase 4 is now [`revamp phases 4-7/04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md) + [`04b-access.md`](revamp%20phases%204-7/04b-access.md)** (typed capability verbs + generated doors — see `00-umbrella-plan.md`). This file's connector-taxonomy work is **demoted to backward-compat hygiene**: it is **not** the core of Phase 4. Read the revamp first. -> -> What survives from this doc, and where it goes: -> - **The one genuinely useful piece — the `MCP_CONNECTOR` (BYO MCP) routing-gap fix — moves to `revamp/04b`** (the "consume-user-MCP" work). That's the actionable carry-over. -> - **Keeping legacy branded natives `MIGRATING` (off) + existing indexed KB docs searchable, and Obsidian/Circleback `DISABLED`** — still true; a small hygiene task, no longer a headline phase. -> - **`is_pipeline_eligible` is dropped** (no pipelines); **Type-1 = file/KB-input only** (Drive/Dropbox/OneDrive + uploads → input-only KB); **`WEBCRAWLER_CONNECTOR` is RETIRED** (native `scrape_webpage` tool, now a `web.scrape` capability verb). -> -> The static-registry design below is retained for reference, but do not treat "connector two-type restructure" as the Phase 4 deliverable — the deliverable is the **capability registry** in `revamp/04a`. - -> Part of **Phase 4 — Connector two-type restructure (backend)**. See `00-umbrella-plan.md`. -> Sibling: `04b-source-discovery.md` (web-search repurposing). Precondition: Phases 1–2 (rename) live. - -> **Implementation note.** Phases 1–2 are **SHIPPED**, so the live code already says `workspace_id`/`Workspace` — substitute for the old `search_space_*`/`SearchSpace` names in citations below and grep the new name. Locate code by **symbol/grep**, not the absolute line numbers cited here, since the rename shifted them. - -## Objective - -Give connectors a first-class **Type-1 (Data Source) / Type-2 (MCP Tool)** taxonomy and an **availability** state, then gate creation, indexing, chat-subagent build, and (forward-compat) pipeline-eligibility off it — without a DB migration. Also fix the long-standing bug where generic `MCP_CONNECTOR` tools are discovered but dropped. - -The taxonomy is **fully determined by `connector_type`** (every Notion row is the same category), so it is modeled as a **static code registry**, not a per-row column (resolved decision — avoids a migration + backfill drift across ~20 connector-create sites). `is_indexable` is **kept** (it is an orthogonal per-row capability that still gates the real `/index` + periodic machinery — e.g. Notion is `is_indexable=True` yet is a Type-2 tool). - -## Locked model (MVP) - -| Bucket | `category` | `availability` | Members | Behaviour | -|--------|-----------|----------------|---------|-----------| -| Functional data sources | `DATA_SOURCE` | `AVAILABLE` | `WEBCRAWLER_CONNECTOR`, `GOOGLE_DRIVE_CONNECTOR`, `COMPOSIO_GOOGLE_DRIVE_CONNECTOR`, `ONEDRIVE_CONNECTOR`, `DROPBOX_CONNECTOR` | create OK, `/index`+periodic OK, **pipeline-eligible** | -| Functional tool | `MCP_TOOL` | `AVAILABLE` | `MCP_CONNECTOR` (generic BYO MCP server) | create OK, acts in chat, **no pipelines** | -| Deprecated branded (indexers) | `MCP_TOOL` | `MIGRATING` | `NOTION_CONNECTOR`, `GITHUB_CONNECTOR`, `CONFLUENCE_CONNECTOR`, `BOOKSTACK_CONNECTOR`, `ELASTICSEARCH_CONNECTOR` | block new create, **disable `/index`+periodic**; disable subagent **only for Notion/Confluence** (GitHub/BookStack/Elasticsearch have no subagent); keep existing rows + already-indexed KB docs searchable | -| Deprecated branded (act-only) | `MCP_TOOL` | `MIGRATING` | `SLACK_CONNECTOR`, `TEAMS_CONNECTOR`, `LINEAR_CONNECTOR`, `JIRA_CONNECTOR`, `CLICKUP_CONNECTOR`, `AIRTABLE_CONNECTOR`, `DISCORD_CONNECTOR`, `GOOGLE_GMAIL_CONNECTOR`, `GOOGLE_CALENDAR_CONNECTOR`, `LUMA_CONNECTOR`, `COMPOSIO_GMAIL_CONNECTOR`, `COMPOSIO_GOOGLE_CALENDAR_CONNECTOR` | block new create, **disable subagent** (no KB data to keep) | -| Disabled | `DATA_SOURCE` | `DISABLED` | `OBSIDIAN_CONNECTOR`, `CIRCLEBACK_CONNECTOR` | block new create + subagent; not MCP-bound (distinct from MIGRATING). Obsidian is self-hosted-only today | -| Removed → `04b` | n/a | n/a | `SERPER_API`, `TAVILY_API`, `SEARXNG_API`, `LINKUP_API`, `BAIDU_SEARCH_API` | enum values dropped + repurposed in `04b`. **04a treats them as `HIDDEN`** (excluded from taxonomy) so the registry is total until `04b` removes them | - -**"Migrating" UX:** existing `MIGRATING` rows remain in the DB and (for the indexers) their already-indexed documents stay searchable via the always-on `knowledge_base` subagent; only the connector's own management (re-index, periodic, live subagent) is turned off, with a "Migrating to MCP soon" status surfaced by the API. - -## Current state (cited) - -### The enum + model - -- `class SearchSourceConnectorType(StrEnum)` — `db.py:85–117` (all **30** values, incl. the 5 search APIs `:86–90`, `MCP_CONNECTOR:113`, Composio variants `:115–117`). The registry's totality test must cover all 30. -- `class SearchSourceConnector` — `db.py:1829`. Columns: `connector_type` (`:1867`), `is_indexable` (`:1868`, `Boolean default False`), periodic fields `periodic_indexing_enabled`/`indexing_frequency_minutes`/`next_scheduled_at` (`:1880–1882`). -- Schema: `schemas/search_source_connector.py` — `is_indexable: bool` (`:16`), validator "periodic only if indexable" (`:42–45`), `Update.is_indexable` (`:66`). - -### Where indexability is currently set (per connector, scattered) - -Create routes hardcode `is_indexable` per type: `True` for Notion (`notion_add_connector_route.py:409`), Confluence (`:376`), Google Drive (`:420`), Dropbox (`:373`), OneDrive (`:380`); `False` for Slack/Teams/Discord/Gmail/Calendar/Linear/Jira/ClickUp/Airtable/Luma (their `*_add_connector_route.py`), MCP (`search_source_connectors_routes.py:2696`). Composio uses `INDEXABLE_TOOLKITS` (`composio_routes.py:289,400`, `composio_service.py:99`). `oauth_connector_base.py:59,598` defaults `True`. **This scatter is exactly why category lives in a registry, not duplicated at each create site.** - -### The `/index` dispatch (the indexable set, today) - -`index_connector_content()` — `search_source_connectors_routes.py:717` — is an `if/elif` chain on `connector_type` that dispatches to a Celery task for: Notion (`:850`), GitHub (`:861`), Confluence (`:872`), BookStack (`:885`), Google Drive (`:898`), OneDrive (`:948`), Dropbox (`:995`), Elasticsearch (`~:1050`), WebCrawler (`:1058`). **Movers** (Notion/GitHub/Confluence/BookStack/Elasticsearch) lose this branch; the file-based + WebCrawler branches stay. - -### The chat subagent maps + the routing gap - -- `multi_agent_chat/constants.py` (the chat-package root `constants.py`, **not** under `subagents/`) — `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS` (`:5–24`, connector_type → subagent name; **note GitHub/BookStack/Elasticsearch have NO entry** — they are index-only, no chat subagent) and `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP` (`:26–44`, subagent → required tokens; `deliverables`/`knowledge_base` require `frozenset()` = always built). The required tokens are a **mix of connector types and doc types** (e.g. `notion`→`NOTION_CONNECTOR`, but `dropbox`→`DROPBOX_FILE`, `google_drive`→`GOOGLE_DRIVE_FILE`, `onedrive`→`ONEDRIVE_FILE`). -- Subagent exclusion: `subagents/registry.py` `get_subagents_to_exclude(available_connectors)` (`:136–152`) excludes a builder in **two** cases: (a) it is **absent from** `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP` (`required_tokens is None`, `:145–147`) → excluded; (b) it has non-empty required tokens that don't intersect the available set (`:150–151`) → excluded. An **empty** `frozenset()` (deliverables/knowledge_base) is always kept (`:148–149`). `build_subagents(..., exclude=...)` (`:182–220`) additionally hard-skips `memory`/`research` (`:195`) and the names in `exclude`, then calls each builder with `mcp_tools=mcp.get(name)` (`:209`). `SUBAGENT_BUILDERS_BY_NAME` (`:92–112`) has **no `mcp` builder** today. **Consequence for §5:** adding an `mcp` builder *without* a `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP["mcp"]` entry would hit case (a) and exclude it forever — so §5's map entry is mandatory, not optional. -- **CRITICAL — `available_connectors` is NOT raw connector types.** `factory.py:102–105` computes `connector_types = get_available_connectors(...)` then `available_connectors = map_connectors_to_searchable_types(connector_types)`. This **shared, already-mapped** list of searchable-type tokens feeds *multiple* consumers: `get_subagents_to_exclude` (`factory.py:253`, `main_agent/middleware/stack.py:202`), the on-demand KB-search tool (`subagents/builtins/knowledge_base/tools/search_knowledge_base.py:_search_types:53–63` does `types.update(available_connectors)` at `:61–62` — this is the post-`main`-merge replacement for the now-deleted `shared/middleware/knowledge_search.py`, which used to do the same in `_resolve_search_types`), `web_search`'s live-connector filter (`shared/tools/web_search.py`), and the `deliverables` `report` tool. **Filtering this list would break legacy KB-doc searchability and web search** — see Target §4. -- `map_connectors_to_searchable_types` (`connector_searchable_types.py:65–100`) maps each configured connector type to a searchable token; **`MCP_CONNECTOR` is absent from `_CONNECTOR_TYPE_TO_SEARCHABLE` (`:22–54`)**, so MCP connectors never produce a token in `available_connectors`. -- **The bug:** `subagents/mcp_tools/index.py` `partition_mcp_tools_by_connector` (`:55–99`) routes each MCP tool via `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS.get(connector_type)`; `MCP_CONNECTOR` has **no entry**, so it hits the `connector_agent is None` branch and is **skipped with a warning** (`:88–95`). Generic user MCP servers therefore contribute zero tools today. (The bucket key is the agent name, e.g. `"mcp"`, consumed by `build_subagents` via `mcp.get("mcp")`.) - -### Query-time searchable mapping (keeps legacy docs visible) - -`main_agent/runtime/connector_searchable_types.py` `_CONNECTOR_TYPE_TO_SEARCHABLE` (`:22–54`, unchanged by the `main` merge) maps connector types → searchable doc-types that scope the KB-search tool (via `available_connectors` → `search_knowledge_base.py:_search_types`). The MIGRATING indexers' entries (NOTION→`NOTION_CONNECTOR`, etc.) **stay** so already-indexed docs remain searchable. (WebCrawler→`CRAWLED_URL`, file connectors→`*_FILE` stay too.) - -## Target design - -### 1. The static registry (single source of truth) - -New `app/connectors/connector_registry.py` (or `app/utils/connector_registry.py`): - -```python -class ConnectorCategory(StrEnum): - DATA_SOURCE = "DATA_SOURCE" - MCP_TOOL = "MCP_TOOL" - -class ConnectorAvailability(StrEnum): - AVAILABLE = "AVAILABLE" # usable now - MIGRATING = "MIGRATING" # branded native, "moving to MCP soon" - DISABLED = "DISABLED" # off for MVP, not MCP-bound (Obsidian/Circleback) - HIDDEN = "HIDDEN" # not a real connector in the taxonomy (search APIs, until 04b removes them) - -# connector_type -> (category, availability). Total over SearchSourceConnectorType. -CONNECTOR_REGISTRY: dict[SearchSourceConnectorType, tuple[ConnectorCategory, ConnectorAvailability]] = { ... } - -def get_category(ct) -> ConnectorCategory | None -def get_availability(ct) -> ConnectorAvailability -def is_creatable(ct) -> bool # availability == AVAILABLE -def is_pipeline_eligible(ct) -> bool # category == DATA_SOURCE and availability == AVAILABLE -def is_indexable_type(ct) -> bool # the file/web data sources that own a Celery indexer -``` - -- **Totality guard (test):** assert every `SearchSourceConnectorType` member has a registry entry, so a newly added connector can't silently fall through gating. -- Exposed on the connector read schema as **computed fields** (`category`, `availability`) so the API/frontend can label and filter without a DB column. (Promote to a column later only if Zero/SQL filtering needs it — see umbrella resolved log.) - -### 2. Create gating - -In the generic create handler `create_search_source_connector` (`search_source_connectors_routes.py:172`, `@router.post("/search-source-connectors")`) **and** the per-service add/OAuth routes (e.g. `notion_add_connector_route.py`, `luma_add_connector_route.py:51`, the Composio routes) reject when `not is_creatable(connector_type)` with a clear 4xx ("This connector is migrating to MCP and can't be added yet" / "disabled for MVP"). The existing per-type duplicate check (`:198–212`) is unaffected. Because the per-service routes each build their own connector, the gate must be applied in each (or in a shared helper they all call) — verify by grepping the `*_add_connector_route.py` set. This is the behavioural change that blocks new branded connectors; existing rows are untouched. - -### 3. Index gating - -Guard `index_connector_content` (`:717`) up front: if `not is_indexable_type(connector_type)`, return a 4xx/no-op ("indexing disabled — migrating to MCP"). This neutralizes the Notion/GitHub/Confluence/BookStack/Elasticsearch branches without deleting their (now-dead) code paths. - -**Periodic path (must also be gated).** Create-gating blocks *new* MIGRATING rows, but **existing** MIGRATING connectors with `periodic_indexing_enabled=True` already have a `next_scheduled_at` and would keep firing via the meta-scheduler Beat task that polls `next_scheduled_at` every minute: `tasks/celery_tasks/schedule_checker_task.py` (due query `next_scheduled_at <= now` `:36`, dispatch loop `:88`, `task = task_map.get(connector.connector_type)` then `.delay(...)` `:118`). The first-run trigger is `create_periodic_schedule` (`utils/periodic_scheduler.py:30`, `task.delay(...)` `:97`). **Both paths re-dispatch through the same per-type Celery tasks** (`index_crawled_urls_task`, `index_notion_pages_task`, …), so the robust single chokepoint is to **gate the Celery index tasks at entry** by `is_indexable_type(connector_type)` — this covers the manual `/index` route, the first-run trigger, and the recurring meta-scheduler pass at once (a no-op early-return for MIGRATING types). (Aside: `schedule_checker_task` already auto-disables a `LIVE_CONNECTOR_TYPES` set at `:74-77` — an existing precedent for turning periodic off per type.) The schema validator "periodic only if indexable" (`schemas/search_source_connector.py:42–45`) is unchanged and unaffected (it keys off the per-row `is_indexable` bool, not the registry). - -Note: **update** (`PUT /search-source-connectors/{id}`, `search_source_connectors_routes.py:371`) is intentionally **left allowed** for MIGRATING rows so existing users can still edit/disable them; only create + index are gated. - -### 4. Subagent gating (turns off branded chat tools) — by NAME, not by token - -> **Do NOT filter `available_connectors`.** It is a shared, already-mapped searchable-type list (see Current State). Stripping MIGRATING entries from it would (a) remove `NOTION_CONNECTOR`/etc. from the KB-search tool's doc-type scope (`search_knowledge_base.py:_search_types`) → **break the very legacy-doc searchability this plan promises**, and (b) remove the live-search tokens → break `web_search` before 04b. The token list must stay intact. - -Instead, **exclude the deprecated subagents by NAME**, derived from the registry. Extend `get_subagents_to_exclude` (`registry.py:136`) so that, in addition to its current token check, it also excludes any subagent whose mapped connector type(s) are all non-`AVAILABLE`: - -- Build the reverse of `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS` (subagent_name → set of connector types). For each `SUBAGENT_BUILDERS_BY_NAME` entry, if it has connector type(s) and **every** one is non-`AVAILABLE` in the registry, add it to the excluded set unconditionally (regardless of tokens). -- This is the single chokepoint used by both `factory.py:253` and `stack.py:202`, and feeds `main_prompt_registry_subagent_lines(exclude)`, so the deprecated specialists disappear from both the build and the prompt. - -Net effect for MVP: - -- **Excluded (MIGRATING):** `notion`, `confluence`, `slack`, `teams`, `linear`, `jira`, `clickup`, `airtable`, `discord`, `gmail`, `calendar`, `luma`. (GitHub/BookStack/Elasticsearch have no subagent — they are handled purely by §3 index gating; their indexed docs stay searchable via the untouched token list.) -- **Kept (AVAILABLE):** `knowledge_base`, `deliverables`, `research`, `memory` (builtins), the new `mcp` subagent, **and the file-source specialists `google_drive`, `dropbox`, `onedrive`** — these map to `AVAILABLE` Type-1 connectors, so by the registry rule they are NOT deprecation-excluded; they remain token-gated as today (built only when that file connector is configured). **Decision:** keep them — a Type-1 data source can still own a chat specialist over its indexed files; the two-type split governs pipeline-eligibility + the MCP migration, not whether an AVAILABLE connector may have a subagent. (If we later want pure-ingestion file connectors with no specialist, flip them by adding a `subagent_chat` flag to the registry — out of scope here.) - -### 5. MCP_CONNECTOR routing-gap fix (makes generic Type-2 work) - -Two changes are needed — the routing map AND surfacing the token, or the subagent will still never build: - -1. **Routing map** (`multi_agent_chat/constants.py`): add `"MCP_CONNECTOR": "mcp"` to `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS` (so `partition_mcp_tools_by_connector` stops hitting the `connector_agent is None` skip at `mcp_tools/index.py:88–95`), and `"mcp": frozenset({"MCP_CONNECTOR"})` to `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP`. -2. **Builder**: add a generic `mcp` entry to `SUBAGENT_BUILDERS_BY_NAME` (`registry.py:92`); `build_subagents` will call it with `mcp_tools=mcp.get("mcp")` (the bucket key from step 1's map value). -3. **Surface the token** (the gap step 4's predecessor missed): `MCP_CONNECTOR` is **not** in `_CONNECTOR_TYPE_TO_SEARCHABLE`, so `available_connectors` never contains it and the `mcp` subagent's required token `{MCP_CONNECTOR}` would never intersect → it would be excluded forever. Fix by adding `"MCP_CONNECTOR": "MCP_CONNECTOR"` to `_CONNECTOR_TYPE_TO_SEARCHABLE` (`connector_searchable_types.py:22`). Side effect: `MCP_CONNECTOR` joins the KB-search tool's doc-type scope (`search_knowledge_base.py:_search_types`), where it matches zero indexed docs — harmless no-op. (Alternative if the no-op is undesirable: special-case `mcp` in `get_subagents_to_exclude` to include it whenever any configured connector is an `AVAILABLE` `MCP_TOOL`, passing the raw `connector_types` alongside. The searchable-map entry is simpler and preferred.) - -### 6. Keep `is_indexable` semantics intact - -No change to the column or its validator. `is_indexable` continues to gate the real `/index`+periodic machinery for the data sources; the registry's `is_indexable_type` is a **type-level** allowlist layered on top (a row must be both an indexable type AND `is_indexable=True`). This avoids reinterpreting historical rows. - -## Work items - -1. **Registry module** + `ConnectorCategory`/`ConnectorAvailability` enums + the total `CONNECTOR_REGISTRY` map + helper predicates (incl. reverse subagent-name→types helper for §4). -2. **Schema**: add computed `category`/`availability` to the connector read schema (`schemas/search_source_connector.py`). -3. **Create gating** in the generic create handler (`:172`) + every per-service add route (reject non-`AVAILABLE`). -4. **Index gating** at the Celery index-task entry (`is_indexable_type`) — covers manual `/index`, first-run trigger, and meta-scheduler. -5. **Subagent gating by name**: extend `get_subagents_to_exclude` to also exclude subagents whose mapped connector types are all non-`AVAILABLE` (do **not** filter the shared `available_connectors` token list). -6. **MCP routing-gap fix**: `multi_agent_chat/constants.py` map entries + generic `mcp` builder in `SUBAGENT_BUILDERS_BY_NAME` + `MCP_CONNECTOR` token in `_CONNECTOR_TYPE_TO_SEARCHABLE` (so the subagent actually builds). -7. **Tests** (below). - -## Tests - -- **Registry totality**: every `SearchSourceConnectorType` has an entry (guards future additions). -- **Create gating**: creating any `MIGRATING`/`DISABLED` type → 4xx; `WEBCRAWLER`/file/`MCP_CONNECTOR` → OK; MCP multi-instance still works. -- **Index gating**: index task entry for Notion/GitHub/Confluence/BookStack/Elasticsearch → no-op; WebCrawler/GDrive/OneDrive/Dropbox → runs. Include a case for an **existing** MIGRATING row with `periodic_indexing_enabled=True` + due `next_scheduled_at` → meta-scheduler triggers a no-op (does not re-index). -- **Subagent gating by name**: with a (legacy) Notion + Slack connector configured, `get_subagents_to_exclude` excludes `notion`/`slack`; `knowledge_base`/`deliverables` always built; a configured `google_drive`/`dropbox`/`onedrive` (AVAILABLE) is **not** excluded (still token-gated). -- **No collateral damage to the token list**: gating subagents must NOT change `available_connectors` — assert the KB-search tool still resolves the `NOTION_CONNECTOR` doc type (via `_search_types`) for the legacy connector (guards against the "filter the shared list" regression). -- **MCP gap (two-part)**: (a) a configured `MCP_CONNECTOR` produces a `"mcp"` token in `map_connectors_to_searchable_types`; (b) its tools land in the `mcp` bucket and the `mcp` subagent is actually built (regression for both `mcp_tools/index.py:88–95` and the missing-token exclusion). -- **Legacy searchability**: a search space with an existing (now-MIGRATING) Notion connector + indexed docs still returns those docs via the KB-search tool (`connector_searchable_types` Notion entry unchanged). - -## Risks / trade-offs - -- **Breaks current branded chat tooling (accepted).** Deprecating act-only natives removes their live chat subagents for existing users mid-pivot. Per the umbrella's "DB migrations carry users; backend behaviour can change" posture, this is acceptable; the "migrating to MCP soon" status sets expectations. -- **Dead code paths left in place.** The Notion/GitHub/Confluence/BookStack/Elasticsearch `/index` branches + indexers remain but are gated off (not deleted) to keep the diff small and reversible; delete when the real MCP migration lands. -- **No column = not SQL/Zero-filterable.** Frontend (deferred) filters via the API's computed fields. If Zero needs server-side filtering later, promoting the registry to a denormalized column is additive. -- **Registry/`is_indexable` dual-gate.** Two truths (type-level allowlist + per-row bool) must agree; the totality test + index-gating test cover the seam. -- **Shared `available_connectors` is a footgun.** It feeds subagent-exclusion, the KB-search tool's doc-type scope, web_search, and the deliverables `report` tool simultaneously. Subagent gating is therefore done by *name* (registry-driven), never by mutating that list; the "no collateral damage" test pins this. This also removes any 04a→04b sequencing hazard (04a no longer touches the live-search tokens that 04b later retires). -- **File-source specialists kept.** `google_drive`/`dropbox`/`onedrive` stay as AVAILABLE chat specialists (decision in §4). If product wants pure-ingestion file connectors, that's an additive registry flag later. - -## Out of scope (hand-offs) - -- Search-API enum removal, key relocation, and the source-discovery endpoint → `04b`. -- Pipeline tables + actually creating pipelines from Type-1 connectors → Phases 5–7 (`is_pipeline_eligible` is defined here for them to consume). -- Real MCP migration of the branded connectors (re-point to MCP servers) → post-MVP. -- Frontend connector UI restructure → frontend umbrella. diff --git a/plans/backend/04b-source-discovery.md b/plans/backend/04b-source-discovery.md deleted file mode 100644 index 59b952ce7..000000000 --- a/plans/backend/04b-source-discovery.md +++ /dev/null @@ -1,120 +0,0 @@ -# Phase 4b — Web-search repurposing + source-discovery endpoint - -> ## ↪ REDIRECT / ABSORBED (2026-06-30) — superseded by the revamp -> **This subplan is absorbed into the canonical [`revamp phases 4-7/04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md) as the `web.discover` verb.** The substance still holds — drop the 5 search `connector_type` values (`SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API`) + their paths (destructive migration), move SearXNG/Linkup/Baidu to **platform env** config, and rewire the chat `web_search` tool onto them — but the "source-discovery endpoint" is now a **capability verb** (`web.discover(query, top_k)`) in the registry, exposed identically via chat/REST/MCP (not a bespoke route), and its output feeds the agent (or a Tracker binding), **not** "a pipeline." Read `revamp/04a` (`web.discover`) as canonical; use the provider/migration detail here as implementation reference. -> -> Part of **Phase 4 — Capabilities & Access (backend)**. See `00-umbrella-plan.md`. -> Sibling (old): `04a-connector-category.md` (demoted to hygiene). Precondition: Phases 1–2 (rename) live. - -> **Implementation note.** Phases 1–2 are **SHIPPED**, so the live code already says `workspace_id`/`Workspace` — substitute for the old `search_space_*`/`SearchSpace` names in citations below and grep the new name; locate code by **symbol/grep**, not the absolute line numbers cited. - -## Objective - -Retire the five web-search **connector types** and re-cast the survivors as **platform-level providers** that power two things: (1) the existing chat `web_search` tool, and (2) a new **source-discovery endpoint** that, given a topic/competitor, suggests candidate URLs the user can feed into the Universal WebURL Crawler / a pipeline. - -Resolved decisions driving this: - -- **Drop all 5 search `connector_type` enum values** (`SERPER_API`, `TAVILY_API`, `SEARXNG_API`, `LINKUP_API`, `BAIDU_SEARCH_API`). -- **Tavily + Serper removed entirely** (no provider, no code path). -- **SearXNG / Linkup / Baidu survive as platform providers.** SearXNG is already platform/env-based; **Linkup + Baidu move from per-workspace connector `config` to platform/env config** (one app-wide key set, matching the "single provider app-wide" style from `03b`). - -## Current state (cited) - -### The 5 search types are first-class connectors today - -- Enum members `SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API` — `db.py:86–90` (`SERPER_API` is annotated "NOT IMPLEMENTED YET"; SearXNG is already platform-backed despite being an enum value). -- Per-workspace search methods on `ConnectorService` read keys from the connector row's `config`: `search_tavily` (`connector_service.py:481`, key `TAVILY_API_KEY:510`), `search_searxng` (`:587`), `search_baidu` (`:614`), `search_linkup` (`:1968`, key `LINKUP_API_KEY:2000`). -- Query-time routing: `connector_searchable_types.py:_CONNECTOR_TYPE_TO_SEARCHABLE` (`:22–26`) maps `TAVILY_API`/`LINKUP_API`/`BAIDU_SEARCH_API` to the web_search tool ("live search connectors"); the other connector types map to KB pre-search. - -### The web_search tool already has a platform/per-workspace split - -> **Post-`main`-merge note.** A `main` sync **consolidated web_search to a single tool**: the old research-subagent copy `subagents/builtins/research/tools/web_search.py` was **deleted**, and there is now **one** factory at `app/agents/chat/shared/tools/web_search.py`. The research subagent now imports it (`subagents/builtins/research/tools/index.py:10,24–26`). So everywhere below that said "both variants / mirror" is now **one file**. The same merge also retired the XML result blob in favor of `[n]`-citation rendering (see §3). - -`web_search` tool factory `create_web_search_tool(search_space_id, available_connectors)` — the single canonical tool at `app/agents/chat/shared/tools/web_search.py`: -- `_LIVE_SEARCH_CONNECTORS = {TAVILY_API, LINKUP_API, BAIDU_SEARCH_API}` (`:41–45`), `_LIVE_CONNECTOR_SPECS` → `ConnectorService.search_*` (`:47–51`), per-connector dispatch `_search_live_connector` (`:115–151`). The factory picks the active live connectors by intersecting `available_connectors` with `_LIVE_SEARCH_CONNECTORS` (`:163–167`). -- At call time it fans out **in parallel** to platform SearXNG (`web_search_service.is_available()` / `.search()`, `:204–214`) **plus** each active per-workspace live connector (`:216–228`), then dedupes by URL (`:242–250`). -- Platform SearXNG service: `app/services/web_search_service.py` — env-gated by `config.SEARXNG_DEFAULT_HOST`, in-process circuit breaker + Redis result cache; returns the `(result_obj, documents)` shape the tool consumes. - -### Config - -- `config.SEARXNG_DEFAULT_HOST = os.getenv("SEARXNG_DEFAULT_HOST")` — `config/__init__.py:558`. There are **no** platform env vars for Linkup/Baidu/Tavily today (their keys live in connector `config`). - -## Target design - -### 1. Platform search-provider config (env) - -Add platform env knobs alongside `SEARXNG_DEFAULT_HOST` (`config/__init__.py:~558`): - -- `LINKUP_API_KEY = os.getenv("LINKUP_API_KEY")` -- `BAIDU_SEARCH_API_KEY = os.getenv("BAIDU_SEARCH_API_KEY")` (+ the optional knobs `search_baidu` reads from connector `config` today — `BAIDU_MODEL` / `BAIDU_SEARCH_SOURCE` / `BAIDU_ENABLE_DEEP_SEARCH`, per `validators.py:507-514`; the required key is `BAIDU_API_KEY` in the per-workspace config, relocated to this platform env var — port from its current `config` reads). -- (SearXNG unchanged.) Document all three in `.env.example` as optional; each provider self-disables when its key/host is unset (mirrors `web_search_service.is_available()`). - -No Tavily/Serper env (removed). - -### 2. A platform search-providers service (consolidate the survivors) - -Introduce `app/services/web_search_service.py`-level functions (or a small `search_providers/` package) so all three survivors share one platform shape `(result_obj, documents)`: - -- `searxng.search(...)` — the existing `web_search_service.search` (unchanged). -- `linkup.search(...)` / `baidu.search(...)` — **port the bodies of `ConnectorService.search_linkup`/`search_baidu`** but source the API key from `config.*` (platform) instead of `connector.config` (per-workspace), and drop the `search_space_id`/connector lookups. -- Each is `is_available()`-gated on its env key. Tavily (`search_tavily`) is deleted. - -This removes per-workspace coupling: web search + discovery no longer depend on any connector row. - -### 3. Rewire the chat `web_search` tool (single tool) - -- Drop the `_LIVE_SEARCH_CONNECTORS` / `_LIVE_CONNECTOR_SPECS` / `_search_live_connector` mechanism in the **single** tool `shared/tools/web_search.py` (`:41–51`, `:115–151`, and the factory's live-connector intersection `:163–167`, plus the per-workspace fan-out `:216–228`). Instead, fan out to the **platform providers** that are `is_available()` (SearXNG + Linkup + Baidu), all keyless from the caller's view. -- Keep the existing parallel-gather + URL-dedupe (`:233–250`). **Note:** the `main` merge replaced the old XML result blob with `[n]`-citation rendering — the tool now builds `RenderableDocument`s (`_to_renderable_web_documents:66–112`) and returns them via `render_web_results` + `load_registry`, persisting the `citation_registry` on state (`:252–281`). Leave that render/registry path **unchanged**; only the *provider sourcing* (SearXNG+Linkup+Baidu, env-keyed) changes. -- **Call-site churn — keep the `available_connectors` parameter in the factory signature but stop using it for provider selection** (providers are now env-derived). This avoids touching every caller: `main_agent/tools/registry.py:24,39`, `subagents/builtins/research/tools/index.py:10,24–26`, and `anonymous_chat/agent.py:127` (passes `available_connectors=None` already). **`deliverables` is no longer a web_search caller** (its KB/web tooling was removed in the `main` merge) — drop it from the churn list. `search_space_id` likewise stays for logging only. (A later cleanup can remove the now-dead params.) -- Net behaviour: web search works platform-wide whenever any provider env key is set; it no longer requires the workspace to have a search connector configured (so anonymous chat gains web search too). - -### 4. Source-discovery endpoint (the new capability) - -New route (e.g. `routes/source_discovery_routes.py`, mounted under `/api/v1/workspaces/{workspace_id}/...` to match the renamed surface): `POST .../source-discovery` taking `{ query/topic, top_k }` and returning a **ranked list of candidate URLs** (url, title, snippet, provider) for the user to add to the WebURL Crawler / a pipeline. - -- Implementation = call the platform providers (§2) in parallel, dedupe by URL (reuse the tool's dedupe), and return URL-centric results (plain `{url,title,snippet,provider}` — **not** the chat tool's `[n]`-cited `render_web_results` output, which is for in-conversation citation, not URL suggestion). -- Auth: standard workspace access check (same dependency as other workspace routes). -- This is **backend-only**; the UX (a "find sources" affordance when configuring a crawler/pipeline) is deferred to the frontend umbrella. -- Optional thin reuse: the chat `web_search` `_web_search_impl` and this endpoint can share a `discover_urls(query, top_k) -> list[UrlCandidate]` core. - -### 5. Drop the 5 connector types - -- **Remove the enum members** `SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API` from `SearchSourceConnectorType` (`db.py:86–90`). -- **Remove their connector code paths**: `ConnectorService.search_tavily` (`:481`, delete) and the per-workspace `search_searxng` (`:587`)/`search_baidu` (`:614`)/`search_linkup` (`:1968`) (replace with the platform service §2; delete the connector-config variants that read `connector.config[...]_API_KEY`, e.g. `:510`, `:2000`). Remove the three search entries from `connector_searchable_types.py:24–26`. Remove the `*_API` entries from the connector-config validation dict (`utils/validators.py:490–514`, the 5 keys in `connector_rules`: `SERPER_API:490`/`TAVILY_API:491`/`SEARXNG_API:492-505`/`LINKUP_API:506`/`BAIDU_SEARCH_API:507-514`) and `04a`'s `HIDDEN` registry entries for these types. **Grep-guard:** no remaining references to the 5 enum names or `search_tavily` anywhere. -- **Migration (this phase is NOT migration-free):** existing connector rows of these 5 types must be handled before/with the enum change: - - Alembic migration: `DELETE FROM search_source_connectors WHERE connector_type IN ('SERPER_API','TAVILY_API','SEARXNG_API','LINKUP_API','BAIDU_SEARCH_API');` (FKs cascade; these rows only held search API keys, now relocated to env). Self-hosted operators re-add Linkup/Baidu keys via env — call out in the migration docstring + `.env.example`. - - **PG enum handling:** dropping a value from a Postgres enum type requires a type recreate. Lowest-risk option: after deleting the rows, leave the now-unused labels in the PG enum type (harmless orphans) and just remove them from the Python `StrEnum` so no new rows can use them. If a clean type is wanted, do the standard "create new enum → `ALTER COLUMN ... TYPE` with a `USING` cast → drop old" dance in the migration (heavier; only if desired). Recommend the orphan-label approach for MVP. - - Also scrub `document.document_type` / search rows that referenced `TAVILY_API`/`LINKUP_API`/`BAIDU_SEARCH_API` as a **doc type** only if such persisted docs exist (live-search results were ephemeral, not indexed — verify; expectation is none). - -## Work items - -1. **Config**: `LINKUP_API_KEY`, `BAIDU_SEARCH_API_KEY` (+ Baidu host/region) in `Config` + `.env.example`; keep `SEARXNG_DEFAULT_HOST`. -2. **Platform providers**: port Linkup/Baidu search to env-keyed platform functions sharing SearXNG's `(result_obj, documents)` shape; delete Tavily. -3. **Rewire `web_search`** (the single shared tool `shared/tools/web_search.py`) to fan out to platform providers; drop `available_connectors`/live-connector plumbing; keep dedupe + the `[n]`-citation render path. -4. **Source-discovery endpoint**: `POST .../source-discovery` → ranked URL candidates; shared `discover_urls()` core. -5. **Drop the 5 enum values** + remove `search_*` connector methods, `connector_searchable_types` search entries, validators, and `04a` HIDDEN entries. -6. **Migration**: delete the 5 connector types' rows; handle the PG enum (orphan-label approach); docstring notes the env re-keying. -7. **Tests** (below). - -## Tests - -- **Provider availability**: each of SearXNG/Linkup/Baidu self-disables when its env key/host is unset; `web_search` returns "not available" only when all are unset. -- **web_search rewire**: with `LINKUP_API_KEY` set (no search connector rows), `web_search` still returns Linkup results; results from multiple providers dedupe by URL. -- **No per-workspace dependency**: a workspace with zero connectors still gets web search + source-discovery results. -- **Source-discovery endpoint**: returns URL candidates `{url,title,snippet,provider}`; enforces workspace access; empty when no providers configured. -- **Enum removal**: `SearchSourceConnectorType` no longer has the 5 values; creating one → 422; the migration deletes existing rows; reading remaining connectors is unaffected. -- **Tavily gone**: no import/path references `search_tavily`/`TAVILY_API` remain (grep guard). - -## Risks / trade-offs - -- **Per-workspace search keys become platform-wide (accepted).** Workspaces can no longer bring their own Linkup/Baidu keys; one app-wide key set serves everyone (matches the single-provider posture). Self-hosted must move keys to env — surfaced in migration docs. -- **Destructive migration.** Deleting the 5 connector types' rows is irreversible (downgrade can recreate rows but not their secrets). Gate behind the standard backup/runbook; the data is just API-key config now living in env. -- **PG enum orphan labels.** Leaving unused enum labels is cosmetically untidy but avoids a risky type-recreate; documented as a deliberate trade-off. -- **Sequencing with 04a.** 04a marks the 5 types `HIDDEN` so the taxonomy stays total in the interim; 04b removes them. If 04b ships first, 04a's HIDDEN bucket is simply never populated — both orders are safe, but `04a → 04b` is the intended order. - -## Out of scope (hand-offs) - -- Taxonomy/gating/MCP-routing fix → `04a`. -- Using discovered URLs to actually create a pipeline/crawl → Phases 5–7 (this endpoint only *suggests* URLs). -- Source-discovery UX (the "find sources" affordance) → frontend umbrella. -- Crawl billing for any crawl the user starts from a discovered URL → already covered by `03c` (the crawl path bills regardless of how the URL was found). diff --git a/plans/backend/05-access.md b/plans/backend/05-access.md new file mode 100644 index 000000000..1da602d16 --- /dev/null +++ b/plans/backend/05-access.md @@ -0,0 +1,85 @@ +# Phase 5 — Access / Surfaces (REST · MCP · chat doors) + +> Build after `04`. Together `04 + 05` ship the scraper-API product. +> Reuses shipped infra: Identity/Tenancy, API keys, the chat agent + tool registry, the streaming layer, +> Metering (`03c`), and Rocicorp Zero reactive sync. Locate code by symbol/grep. + +## Objective + +Expose the `04` capability registry through three doors — REST + API keys, an MCP server, and chat tools — +each generated from the one registry so the I/O contract can't drift. Every door is the same thin adapter +with no business logic. + +## Adapter shape (identical on every door) + +``` +parse input → validate against verb.input_schema → authn/authz → meter-gate (03c) + → call the SAME executor (04) → serialize verb.output_schema → return cleaned data +``` + +The executor returns directly (`04`). + +## The three doors + +| Door | Who | Auth | Status | +|------|-----|------|--------| +| REST + API keys | external developers | existing API-key infra | public day one | +| MCP server | external agents (Cursor/ChatGPT/Claude) | OAuth 2.1 or bearer (at implementation) | fast-follow | +| Chat tools | the in-app agent (`07`) | existing session + workspace | seed exists (`scrape_webpage`) | + +Routes are generated, so public REST on day one costs nothing extra. + +## Chat surface + +The in-app product is the conversation: the user describes a need in plain language and the agent (`07`) +picks and fills verbs. Raw typed verbs are exposed on REST/MCP for devs and external agents. The chat +door classifies intent and routes: + +``` +"find / compare / what is / pull / right now" → ONE-SHOT → call verbs, answer now +"watch / track / notify me / keep an eye / over time" → KEEP-WATCHING → hand to the ongoing mode (06) +``` + +## MCP — two directions + +- **Serve MCP** (new door): our capabilities as a remote MCP server — Streamable-HTTP `/mcp`, stateless, + bounded/paginated outputs, read-only verbs (least privilege). +- **Consume MCP** (reuse): the BYO-`MCP_CONNECTOR` — the user's own external MCP tools inside our chat + agent. Fix its routing-map gap here. + +## Result delivery + +A chat tool call returns its data inline in the turn. Asynchronous results (e.g. the `06` ongoing mode +writing between turns) reach the client via write-then-sync / stream: the worker writes rows and the +client receives them over the existing SSE stream or a Zero-published table. + +## Work items + +1. Door generator: from the `04` registry emit (a) REST routes + models, (b) MCP tool schemas + handlers, + (c) chat tool defs + handlers. +2. REST surface: public routes + API-key auth (reuse) + `03c` meter-gate. +3. Chat tools: generalize `scrape_webpage` into the registry-backed set (direct-return). +4. Intent routing: classify one-shot vs keep-watching; hand keep-watching to `06`. +5. MCP server (fast-follow): Streamable-HTTP `/mcp`, least-privilege; auth depth at implementation. +6. Consume-MCP fix: repair the BYO-`MCP_CONNECTOR` routing-map gap. + +## Tests + +- A verb added to the registry appears on REST + MCP + chat with identical I/O. +- REST without a valid key → 401; an over-budget call → blocked by the `03c` gate; a success charges once. +- A chat scrape returns cleaned data inline in the same turn. +- "compare X and Y" → one-shot; "watch X weekly" → routed to `06`. +- A configured BYO MCP tool is reachable by the chat agent. + +## Out of scope + +- Verbs, executors, billing units → `04`. +- Keep-watching mechanism + its delivery channel → `06`. +- The CI subagent + its prompt → `07`. +- Legacy branded connectors (backward-compat hygiene, separate task). + +## Open questions + +- MCP auth depth (decide at implementation). +- Public REST rate-limiting / abuse posture. +- Async delivery channel for `06` outputs: SSE stream vs Zero-published messages table (settled with `06`). diff --git a/plans/backend/05-pipelines-model.md b/plans/backend/05-pipelines-model.md deleted file mode 100644 index 0f3075208..000000000 --- a/plans/backend/05-pipelines-model.md +++ /dev/null @@ -1,321 +0,0 @@ -# Phase 5 — Pipelines data model (tables, schemas, migration, Zero, CRUD + run routes) - -> # ❌ SUPERSEDED (2026-06-30) -> **This entire phase is dropped.** Per the **Architecture correction** in `00-umbrella-plan.md`, the CI pivot no longer uses a Pipelines concept: the KB is **input-only** (user files), the crawler/search are **platform-native tools**, connectors are **MCP**, and the recurring-fetch + run-history need is met by the existing **automations** system (`app/automations/`). There are **no `pipelines`/`pipeline_runs` tables**. This document is retained **for historical context only** — do **not** implement it. **The replacement is the canonical revamp Phase 5 — [`revamp phases 4-7/05a-timeline.md`](revamp%20phases%204-7/05a-timeline.md) + [`05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md)**: the `Tracker` primitive + a 3-store delta **Timeline** (the durable CI state / moat). "Diffable run history" lives in the Timeline's `entity_changes`, **not** in `automation_runs.output`. (The earlier "Phase 5′" framing is retired.) - -> Part of the **CI Pivot MVP**. See `00-umbrella-plan.md` (Phase 5). -> Precondition: Phases 1–2 (rename) live, Phase 4a (connector registry) live. Siblings ahead: `06-pipelines-exec.md` (run engine + scheduling), `07-upload-pipeline-kb.md` (uploads-as-pipeline). - -> **Implementation note (post-rename).** This phase adds **brand-new** tables, created *after* the rename, so they use the canonical names natively: table `workspaces`, column `workspace_id`, ORM class `Workspace`. Phases 1–2 are now **SHIPPED**, so the live code uses the `workspace_*` names. Existing code cited below that still shows the **pre-rename** names (`searchspaces`/`search_space_id`/`SearchSpace`) was captured pre-rename — substitute the `workspace_*` equivalent and grep the new name. Locate code by **symbol/grep**, not the absolute line numbers cited here (the rename + Phase 3/4 migrations shift them). - -## Objective - -Introduce two first-class tables — `pipelines` and `pipeline_runs` — plus their enums, Pydantic schemas, one Alembic migration, the backend Zero-publication entries, and the HTTP routes (CRUD + manual-run trigger + list/get runs). **This phase is data-model + API surface only.** The actual run **engine** (invoke connector fetch → optional KB save → write run record), scheduling tick, crawl-billing wiring, and chat-agent context exposure are **Phase 6**; wiring file upload onto an "Uploads" pipeline is **Phase 7**. The manual-run endpoint here **creates a `PipelineRun` row and enqueues the Phase-6 task** (which is a stub until Phase 6 lands), so the surface is testable end-to-end without the engine. - -A **Pipeline** = a saved, runnable fetch over a Type-1 data source: it references a connector + per-pipeline config + an optional cron schedule + a KB destination (`save_to_kb` + `destination_folder_id`). A **PipelineRun** = one immutable execution record (manual / scheduled / upload), carrying status, timing, doc counts, error, an optional raw-result blob ref, and a billing idempotency field. - -## Locked model (MVP) - -| Concept | Decision | -|---------|----------| -| Tables | `pipelines` (mutable definition) + `pipeline_runs` (append-only execution record). Mirrors the `automations` / `automation_runs` split. | -| Where the ORM lives | In **`app/db.py`** alongside `Folder`/`Document`/`SearchSourceConnector`/`Log` (the other core, Zero-published, cross-referenced entities), **not** a separate package. Rationale: needs `back_populates` on `Workspace`/`User`/`SearchSourceConnector`/`Folder`, a Zero entry, and a `schemas/` Pydantic mirror — all lowest-friction in `db.py`. (The `automations` package avoided `db.py` but pays for it with string-relationships + import-time model registration; pipelines are more core, like connectors.) | -| Pipeline → connector | `connector_id` **nullable**, FK `ON DELETE CASCADE`. A connector-backed pipeline points at a Type-1 source; **`NULL` = a non-connector pipeline** (the Phase-7 "Uploads" pipeline). Eligibility (`DATA_SOURCE` + `AVAILABLE`) is enforced at create via Phase 4a's `is_pipeline_eligible` (app-level, not a DB constraint — the registry is code) **and re-checked at run time in Phase 6** (the static registry can flip a type to `MIGRATING` on deploy, stranding existing pipelines). | -| Ownership | `created_by_id` FK `ON DELETE SET NULL`, **nullable** — creator metadata only (workspace-shared, like `Folder`/`Automation`), not an owner. Billing targets the workspace owner (03c). | -| Schedule representation | `schedule_cron VARCHAR NULL` (NULL = manual-only) + `schedule_timezone VARCHAR NOT NULL DEFAULT 'UTC'` + `enabled BOOLEAN` + `next_scheduled_at TIMESTAMPTZ NULL` (computed by Phase 6's tick). Chosen over a bare `frequency_minutes` int to match the **automations** schedule model and reuse its cron util (`automations/triggers/builtin/schedule/cron.py`), which takes **both** a cron and a timezone (`compute_next_fire_at(cron, timezone, …)`). | -| KB destination | `save_to_kb BOOLEAN` + `destination_folder_id` FK `ON DELETE SET NULL` (nullable). `save_to_kb=true` with NULL folder ⇒ index to workspace root (`documents.folder_id` is nullable). No DB CHECK couples them (an FK `SET NULL` would violate a conditional CHECK); the pairing is validated at the API only. | -| Run status enum | `pipeline_run_status`: `pending`, `running`, `succeeded`, `failed`, `cancelled` (mirrors `automation_run_status` minus `timed_out`). | -| Run trigger enum | `pipeline_run_trigger`: `manual`, `scheduled`, `upload`. | -| Periodic coexistence | **COEXIST, do not migrate** (resolves umbrella open item §2 — see "Periodic-indexing coexistence" below). | -| Zero publication | Publish **both tables full-row** (`None`), mirroring `folders` / `search_source_connectors`. Avoids the column-list `_0_version` seam entirely; neither table has a bulky column. | -| Naming | The CI **`Pipeline`** entity (a saved, runnable fetch) is distinct from the existing **`IndexingPipelineService`** (`app/indexing_pipeline/`, the KB-ingest service). They coexist but do **not** share a code path for the MVP: Phase 6's WebURL executor reuses the crawler's own KB-write path (`index_crawled_urls`, extended with `folder_id`), **not** `IndexingPipelineService` (which is the file/Composio ingest path — and is what Phase 7's uploads flow builds on). Name routes/services `pipeline_*` (not `indexing_pipeline_*`) to keep the two legible. | - -## Current state (cited) - -### The precedent: `automations` / `automation_runs` - -The cleanest existing analogue to model against. - -- Editable parent: `Automation` — `automations/persistence/models/automation.py:24`. Pattern: `BaseModel, TimestampMixin`; `search_space_id` FK CASCADE (`:27`), `created_by_user_id` FK SET NULL (`:34`), a native enum `status` with `values_callable=lambda x: [e.value for e in x]` (`:44–54`), a JSONB `definition` (`:56`), an explicit `updated_at` with `onupdate` (`:60–66`), and `runs = relationship(..., cascade="all, delete-orphan", passive_deletes=True)` (`:76–81`). -- Append-only child: `AutomationRun` — `automations/persistence/models/run.py:20`. `automation_id` FK CASCADE (`:23`), native enum `status` default `pending` (`:37–47`), JSONB `error` (`:60`), nullable `started_at`/`finished_at` (`:62–63`). -- Enums: `automations/persistence/enums/run_status.py:8` (`RunStatus(StrEnum)` = pending/running/succeeded/failed/cancelled/timed_out). -- Migration: `alembic/versions/144_add_automation_tables.py` — the **exact template** for this phase: idempotent `CREATE TYPE ... DO $$ IF NOT EXISTS` guards (`:31–76`), `CREATE TABLE IF NOT EXISTS` with inline FKs (`:79–96`), `CREATE INDEX IF NOT EXISTS` per FK/status/created_at (`:97–111`), a **partial "due" index** for the schedule scan (`ix_automation_triggers_due`, `:144–152`), and symmetric `downgrade()` dropping indexes→tables→types (`:190–213`). -- Run routes: `automations/api/run.py` — `GET /automations/{id}/runs` (list, paginated `limit/offset`, `:13–30`) + `GET /automations/{id}/runs/{run_id}` (detail, `:33–44`). The exact shape to mirror for pipeline runs. - -### The other core tables (relationship + column conventions) - -- Base: `TimestampMixin` (gives `created_at`, `db.py:487`) + `BaseModel` (`id SERIAL PK`, `db.py:498`). Tables wanting a mutable `updated_at` declare it explicitly (see `Folder.updated_at` `db.py:1333–1339`, with `onupdate` + `index=True`). -- `SearchSourceConnector` — `db.py:1829`. `connector_type` native enum (`:1867`), `is_indexable` (`:1868`), periodic fields `periodic_indexing_enabled`/`indexing_frequency_minutes`/`next_scheduled_at` (`:1880–1882`), `search_space_id` FK CASCADE (`:1884`), `user_id` FK CASCADE (`:1891`), `documents = relationship(...)` (`:1897`). **Pipelines FK this table** (`connector_id`). -- `Folder` — `db.py:1310`. `search_space_id` FK CASCADE (`:1321`), `created_by_id` FK SET NULL (`:1327`). **Pipelines FK this table** (`destination_folder_id`, SET NULL). -- `Document.folder_id` is nullable (root/unfiled docs exist) — confirmed by `DOCUMENT_COLS` including `folder_id` (`zero_publication.py:32`) and `Document.folder` `passive_deletes=True` (`db.py:1345`). Lets `save_to_kb` work with a NULL destination. -- `Permission` enum has `CONNECTORS_CREATE/READ/UPDATE/DELETE` (`db.py:345–348`); routes authz via `check_permission(session, auth, search_space_id, Permission.X.value, msg)` (`folders_routes.py:42–48`) with `auth = Depends(get_auth_context)` + `session = Depends(get_async_session)` (`folders_routes.py:36–37`). **Reuse the `CONNECTORS_*` permissions** for pipelines (no new permission needed for MVP). - -### Schema conventions - -- `schemas/base.py`: `IDModel` (`id:int`, `:11`) + `TimestampModel` (`created_at`, `:6`), both `ConfigDict(from_attributes=True)`. -- `schemas/search_source_connector.py`: `Base` → `Create(Base)` → `Update(BaseModel, all-Optional)` → `Read(Base, IDModel, TimestampModel)` with `model_config = ConfigDict(from_attributes=True)` and `field_validator`/`model_validator` for config consistency (`:24–56`). The shape to mirror for `schemas/pipeline.py`. - -### Zero publication mechanics - -- `zero_publication.py`: `ZERO_PUBLICATION` map (`:81–94`; line numbers post-`main`-merge, which added `automations`/`new_chat_threads` entries) is the single source of truth; `None` ⇒ publish full row (e.g. `folders`, `search_source_connectors`), a list ⇒ column subset (e.g. `AUTOMATION_RUN_COLS` `:44–53`). `apply_publication(conn)` (`:163`) reconciles via `ALTER PUBLICATION ... SET TABLE`; a migration just calls it (template: `159_publish_podcasts_to_zero.py:21–22`). `_format_table_entry` omits a table until it physically exists with all its canonical columns (`:125–152`), so the migration **must create the tables before** calling `apply_publication`. The `_0_version` allowlist (`{"documents","user","podcasts"}`, `:118`) applies **only to column-list tables** — irrelevant here since we publish full-row. - -### The meta-scheduler we coexist with (Phase-6 reuse target) - -- `tasks/celery_tasks/schedule_checker_task.py:17` — `check_periodic_schedules_task`, runs every minute, scans `SearchSourceConnector` where `periodic_indexing_enabled AND next_scheduled_at <= now` (`:33–39`) and dispatches per-type Celery tasks. This is the **connector-level** periodic path and is **untouched** by Phase 5. Phase 6 adds a sibling scan over `pipelines.next_scheduled_at`. -- `utils/periodic_scheduler.py:30` — `create_periodic_schedule` (first-run trigger helper). Reference for Phase 6's pipeline scheduler. - -### Alembic head - -As of the latest `main` sync the head is **`169`** (`alembic/versions/169_migrate_google_oauth_account_ids_to_sub.py`; chain `166→167→168→169`, sequential integer-prefixed files). By the time Phase 5 is implemented, Phase 1 (rename) and Phase 4b (search-enum drop) have each added a migration ahead of `169`. **Set `down_revision` to the then-current head** — verify with `alembic heads`; do not hardcode a number. - -## Target design - -### 1. Enums (`db.py`, near the other `StrEnum`s) - -```python -class PipelineRunStatus(StrEnum): - PENDING = "pending" - RUNNING = "running" - SUCCEEDED = "succeeded" - FAILED = "failed" - CANCELLED = "cancelled" - -class PipelineRunTrigger(StrEnum): - MANUAL = "manual" - SCHEDULED = "scheduled" - UPLOAD = "upload" -``` - -Mapped with native PG enum types `pipeline_run_status` / `pipeline_run_trigger` using the `values_callable=lambda x: [e.value for e in x]` convention (matches `Automation.status` `automation.py:46–49`, so the DB stores lowercase values, not the Python member names). - -### 2. `pipelines` table (`db.py`) - -```python -class Pipeline(BaseModel, TimestampMixin): - __tablename__ = "pipelines" - - name = Column(String(200), nullable=False, index=True) - config = Column(JSONB, nullable=False, default=dict, server_default=text("'{}'::jsonb")) # per-pipeline overrides (URL list, crawl opts, proxy override seam) - - save_to_kb = Column(Boolean, nullable=False, default=False, server_default="false") - - # Schedule (NULL cron = manual-only). next_scheduled_at is owned/written by Phase 6's tick. - schedule_cron = Column(String(120), nullable=True) - schedule_timezone = Column(String(64), nullable=False, default="UTC", server_default="UTC") # cron util needs a tz (compute_next_fire_at) - enabled = Column(Boolean, nullable=False, default=True, server_default="true") - next_scheduled_at = Column(TIMESTAMP(timezone=True), nullable=True) - - # NULL connector_id = non-connector pipeline (Phase-7 Uploads). - connector_id = Column(Integer, ForeignKey("search_source_connectors.id", ondelete="CASCADE"), nullable=True, index=True) - destination_folder_id = Column(Integer, ForeignKey("folders.id", ondelete="SET NULL"), nullable=True, index=True) - workspace_id = Column(Integer, ForeignKey("workspaces.id", ondelete="CASCADE"), nullable=False, index=True) - # Creator metadata, NOT owner: pipelines are workspace-shared (like folders/automations), - # so member deletion must NOT nuke them — SET NULL, nullable. Billing targets the workspace - # owner (03c), not this field. Phase 6 resolves the acting user as created_by ?? workspace owner. - created_by_id = Column(UUID(as_uuid=True), ForeignKey("user.id", ondelete="SET NULL"), nullable=True, index=True) - - updated_at = Column(TIMESTAMP(timezone=True), nullable=False, - default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC), index=True) - - connector = relationship("SearchSourceConnector", back_populates="pipelines") - destination_folder = relationship("Folder") - workspace = relationship("Workspace", back_populates="pipelines") - created_by = relationship("User", back_populates="pipelines") - runs = relationship("PipelineRun", back_populates="pipeline", cascade="all, delete-orphan", passive_deletes=True) -``` - -Add the inverse `pipelines = relationship("Pipeline", back_populates=...)` on **`Workspace`**, **`User`**, and **`SearchSourceConnector`** (mirror how `SearchSpace.search_source_connectors` / `User.search_source_connectors` are declared at `db.py:1887–1894`; the `created_by`/SET-NULL side mirrors `Folder.created_by` `db.py:1344` and `Automation.created_by` `automation.py:69`). `destination_folder` is intentionally one-directional (no back_populates on `Folder`) to keep the folder model lean. - -> **Ownership decision (review).** Earlier draft used connector-style `user_id NOT NULL CASCADE`. Corrected to `created_by_id … SET NULL` because pipelines are **workspace-shared** content (closer to `Folder`/`Automation` than to the per-user `SearchSourceConnector`): removing a member must not delete the workspace's pipelines or erase run-history audit. For *connector-backed* pipelines this also avoids surprising double-cascade (the connector's own `user_id CASCADE` would already drop them on member deletion via `connector_id`); for the Phase-7 Uploads pipeline (`connector_id` NULL) it's the only thing keeping the pipeline alive after its creator leaves. - -> **FK target naming:** Phases 1–2 are **SHIPPED**, so the table is `workspaces` and the column `workspace_id` — author the FK against `workspaces` / `workspace_id`. (The earlier "if sequencing slips before the rename" caveat is now moot.) - -### 3. `pipeline_runs` table (`db.py`) - -```python -class PipelineRun(BaseModel, TimestampMixin): - __tablename__ = "pipeline_runs" - - pipeline_id = Column(Integer, ForeignKey("pipelines.id", ondelete="CASCADE"), nullable=False, index=True) - - status = Column(SQLAlchemyEnum(PipelineRunStatus, name="pipeline_run_status", - values_callable=lambda x: [e.value for e in x]), - nullable=False, default=PipelineRunStatus.PENDING, - server_default=PipelineRunStatus.PENDING.value, index=True) - trigger = Column(SQLAlchemyEnum(PipelineRunTrigger, name="pipeline_run_trigger", - values_callable=lambda x: [e.value for e in x]), nullable=False, index=True) - - # Result accounting (written by Phase 6). - documents_indexed = Column(Integer, nullable=True) - crawls_attempted = Column(Integer, nullable=True) - crawls_succeeded = Column(Integer, nullable=True) - error = Column(JSONB, nullable=True) - - # Raw fetch blob ref for save_to_kb=false runs (Phase 6 writes via file_storage). Just a key/path string. - result_blob_key = Column(String, nullable=True) - - # Crawl-billing idempotency (carry-over from 03c / umbrella Phase 6 §105): micro-USD charged for this run. - charged_micros = Column(BigInteger, nullable=True) - - started_at = Column(TIMESTAMP(timezone=True), nullable=True) - finished_at = Column(TIMESTAMP(timezone=True), nullable=True) - - pipeline = relationship("Pipeline", back_populates="runs") -``` - -These extra columns (`crawls_*`, `result_blob_key`, `charged_micros`) are added **now** so Phase 6 needs **no second migration** — they sit unused until the engine writes them. (`charged_micros` directly satisfies umbrella Phase 6 line 105's "record `charged_micros` on the `PipelineRun` for idempotency".) - -> **Out of scope (provenance):** linking individual `documents` back to the `pipeline_run` that produced them (a `documents.pipeline_run_id` column) is **not** in this phase — it would touch the existing `documents` table. Defer to Phase 6/7 if run-level provenance is wanted. - -### 4. Migration (one file, `144`-shaped) - -Chain after the then-current head. In `upgrade()`: - -1. `CREATE TYPE pipeline_run_status` + `pipeline_run_trigger` behind `DO $$ IF NOT EXISTS` guards (copy `144:31–76`). -2. `CREATE TABLE IF NOT EXISTS pipelines (...)` with inline FKs: `workspace_id → workspaces(id) ON DELETE CASCADE` (NOT NULL), `connector_id → search_source_connectors(id) ON DELETE CASCADE` (nullable), `destination_folder_id → folders(id) ON DELETE SET NULL` (nullable), `created_by_id → "user"(id) ON DELETE SET NULL` (nullable; note the quoted `"user"` table). `id SERIAL PK`, `created_at`/`updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()`, `config JSONB NOT NULL DEFAULT '{}'::jsonb`, `schedule_timezone VARCHAR(64) NOT NULL DEFAULT 'UTC'`. -3. `CREATE TABLE IF NOT EXISTS pipeline_runs (...)`. -4. Indexes (`CREATE INDEX IF NOT EXISTS`): per FK (`workspace_id`, `created_by_id`, `connector_id`, `destination_folder_id`, `pipeline_id`), plus `status`, `created_at`, `updated_at`, `name`, `trigger`. Add the **partial "due" index** for Phase 6's tick: - -```sql -CREATE INDEX IF NOT EXISTS ix_pipelines_due - ON pipelines (next_scheduled_at) - WHERE enabled = true AND next_scheduled_at IS NOT NULL; -``` - -5. `from app.zero_publication import apply_publication; apply_publication(op.get_bind())` **after** the tables exist (so `_format_table_entry` includes them). Requires the `ZERO_PUBLICATION` code change (§6) to be in the tree. - -`downgrade()`: drop indexes → `pipeline_runs` → `pipelines` → both types (symmetric to `144:190–213`). (The publication reconcile is intentionally not reversed — historical publication shapes are immutable, per `159:25–27`.) - -### 5. Pydantic schemas (`app/schemas/pipeline.py`, exported via `app/schemas/__init__.py`) - -```python -class PipelineBase(BaseModel): - name: str - connector_id: int | None = None - config: dict[str, Any] = {} - save_to_kb: bool = False - destination_folder_id: int | None = None - schedule_cron: str | None = None - schedule_timezone: str = "UTC" - enabled: bool = True - -class PipelineCreate(PipelineBase): - workspace_id: int # required on create (path/body), like FolderCreate.search_space_id - -class PipelineUpdate(BaseModel): # all-Optional, like SearchSourceConnectorUpdate - name: str | None = None - config: dict[str, Any] | None = None - save_to_kb: bool | None = None - destination_folder_id: int | None = None - schedule_cron: str | None = None - schedule_timezone: str | None = None - enabled: bool | None = None - # connector_id is immutable after create (re-create to re-point); not in Update. - -class PipelineRead(PipelineBase, IDModel, TimestampModel): - workspace_id: int - created_by_id: uuid.UUID | None = None - next_scheduled_at: datetime | None = None - updated_at: datetime - model_config = ConfigDict(from_attributes=True) - -class PipelineRunRead(IDModel, TimestampModel): - pipeline_id: int - status: PipelineRunStatus - trigger: PipelineRunTrigger - documents_indexed: int | None = None - crawls_attempted: int | None = None - crawls_succeeded: int | None = None - error: dict[str, Any] | None = None - charged_micros: int | None = None - started_at: datetime | None = None - finished_at: datetime | None = None - model_config = ConfigDict(from_attributes=True) - -class PipelineRunList(BaseModel): - items: list[PipelineRunRead] - total: int -``` - -**Validators:** - -- `schedule_cron` + `schedule_timezone` (when cron not None): validate together with the existing automations cron util `validate_cron(cron, timezone)` (`automations/triggers/builtin/schedule/cron.py:15`) so a bad expression **or** bad IANA timezone 422s at create, not at the Phase-6 tick. (`schedule_timezone` is consumed by Phase 6's `compute_next_fire_at`; default `"UTC"`.) -- The `save_to_kb` ⇄ `destination_folder_id` pairing and **connector eligibility** are validated in the **route** (they need a DB session: folder-in-workspace check + `is_pipeline_eligible(connector.connector_type)`), not in the pure schema validator — mirroring how `folders_routes` validates parent-in-search-space in the handler (`folders_routes.py:54–58`). - -### 6. Zero publication (`zero_publication.py`) - -Add to `ZERO_PUBLICATION` (`:81–94`), both **full-row**: - -```python - "pipelines": None, - "pipeline_runs": None, -``` - -Full-row (like `folders`/`search_source_connectors`) — no `_0_version` allowlist edit needed, no column-drift migrations later. (If a bulky column is ever added and must be excluded, switch that table to an explicit COLS list and handle the `_0_version` seam at `:118` then.) `verify_publication` will then expect both tables; the migration's `apply_publication` call (§4) reconciles them. - -> **Client sync is not live until the frontend lands (safe).** Publishing backend-side only makes the rows *available* to Zero; nothing syncs to a client until the (deferred) frontend Zero schema + permissions include these tables. Publishing now is harmless and matches the existing pattern (e.g. `automation_runs` was published by migration before/independent of its UI). This entry exists so the chat-agent run-history context (Phase 6) and a later Pipelines UI get push for free. - -### 7. Routes (`app/routes/pipelines_routes.py`, registered in `routes/__init__.py`) - -Follow `folders_routes` for authz (`check_permission` + `get_auth_context` + `get_async_session`) and `automations/api/run.py` for the run-list shape. Register the router in `routes/__init__.py` near `search_source_connectors_router` (`:105`). - -| Method + path | Behaviour | AuthZ | -|---------------|-----------|-------| -| `POST /pipelines` | Create. Validate: workspace access; if `connector_id`, load it, assert same workspace + `is_pipeline_eligible` (4xx otherwise); if `destination_folder_id`, assert folder in same workspace; `save_to_kb=true` allowed with NULL folder (→ root). Set `created_by_id = auth.user.id`. Persist; return `PipelineRead`. | `CONNECTORS_CREATE` | -| `GET /pipelines?workspace_id=` | List pipelines in a workspace. | `CONNECTORS_READ` | -| `GET /pipelines/{id}` | Read one. | `CONNECTORS_READ` | -| `PUT /pipelines/{id}` | Update (all-optional). Re-validate folder-in-workspace + cron if changed. | `CONNECTORS_UPDATE` | -| `DELETE /pipelines/{id}` | Delete (runs cascade). | `CONNECTORS_DELETE` | -| `POST /pipelines/{id}/run` | **Manual trigger.** Insert a `PipelineRun(status=pending, trigger=manual)`, enqueue the Phase-6 Celery task with the run id, return the run. Until Phase 6, the task is a stub (no-op/marks failed) — the row + enqueue path are what this phase delivers. (Concurrency control — block a second run while one is in flight — is deferred to Phase 6, reusing the connector indexing Redis locks `utils/indexing_locks.py`.) | `CONNECTORS_UPDATE` | -| `GET /pipelines/{id}/runs?limit=&offset=` | List runs newest-first, paginated (`limit` 1–200 default 50, `offset` ≥0) → `PipelineRunList`. Mirror `automations/api/run.py:13–30`. | `CONNECTORS_READ` | -| `GET /pipelines/{id}/runs/{run_id}` | Run detail. | `CONNECTORS_READ` | - -All handlers scope by the pipeline's `workspace_id` and 404 on cross-workspace access. A thin `PipelineService` (mirroring `automations` `RunService`) is optional; for MVP, inline session logic in the routes (as `folders_routes` does) is acceptable. - -## Periodic-indexing coexistence (resolves umbrella open item §2) - -**Decision: COEXIST for MVP — do NOT migrate connector periodic config into pipelines.** - -- The existing `SearchSourceConnector.periodic_indexing_enabled` / `next_scheduled_at` path and its meta-scheduler (`schedule_checker_task.py`) **stay fully functional and untouched** in Phases 5–6. Pipelines are **purely additive**: a new `pipelines.next_scheduled_at` scan (Phase 6) runs **alongside** the connector scan. -- **Rationale:** (a) lowest risk + smallest diff (no backfill of existing connector schedules into pipeline rows, no data migration of live schedules); (b) only the WebURL crawler is the MVP pipeline executor (Phase 6) — file sources keep using their connector-level periodic path; (c) consistent with the umbrella posture "DB migrations carry users; backend behaviour can change incrementally." -- **Known overlap (flagged for Phase 6, not solved here):** a single `WEBCRAWLER_CONNECTOR` could have BOTH connector-level periodic indexing AND a pipeline wrapping it → double crawl + **double bill**. The data model permits it; the *guard* is Phase 6's responsibility. **Recommendation for Phase 6:** when a pipeline is created/enabled over a connector, treat the **pipeline as authoritative** and set that connector's `periodic_indexing_enabled=False` (single scheduler owns each connector). Recorded here so Phase 6 implements the de-dup; Phase 5 only needs the columns to support either path. - -## Work items - -1. **Enums** `PipelineRunStatus` + `PipelineRunTrigger` in `db.py`. -2. **ORM** `Pipeline` + `PipelineRun` in `db.py`; inverse `pipelines` relationships on `Workspace`, `User`, `SearchSourceConnector`. -3. **Migration**: 2 types + 2 tables + indexes (incl. `ix_pipelines_due`) + `apply_publication`; symmetric downgrade. -4. **Schemas** `app/schemas/pipeline.py` (+ export in `schemas/__init__.py`); cron syntactic validator. -5. **Zero**: add `pipelines`/`pipeline_runs` (full-row) to `ZERO_PUBLICATION`. -6. **Routes** `pipelines_routes.py` (CRUD + `/run` + `/runs` list/detail); register in `routes/__init__.py`; reuse `CONNECTORS_*` permissions. -7. **Phase-6 task stub**: a named Celery task `run_pipeline(run_id)` in **`app/pipelines/tasks.py`** (the module Phase 6 fleshes out) that `/run` enqueues — no-op/marks the run `failed` with an "engine not implemented" error until Phase 6 fills it in (keeps the endpoint honest and testable). Register it so it's dispatchable: add `"app.pipelines.tasks"` to the Celery `include` list (`celery_app.py`). Phase 6 replaces the stub body in-place (same task name + module → `/run`'s import is stable) and adds the scheduler task + Beat entry + queue routing. -8. **Tests** (below). - -## Tests - -- **Migration round-trip**: upgrade creates both tables + both enum types + `ix_pipelines_due`; `zero_publication --verify` reports no mismatch (both tables published full-row); downgrade drops cleanly. -- **Create gating (cross-link to 04a)**: `POST /pipelines` with an `AVAILABLE` `DATA_SOURCE` connector (WebURL/GDrive/OneDrive/Dropbox) → 201; with a `MIGRATING`/`MCP_TOOL`/`DISABLED` connector → 4xx (`is_pipeline_eligible` false); with `connector_id=None` → allowed (Uploads case). **(Superseded in Phase 7: once the system-managed Uploads pipeline exists, user `POST` with `connector_id=None` is rejected 4xx and the Uploads row is auto-created on upload — see `07` §7. Update this test in Phase 7.)** -- **Folder validation**: `destination_folder_id` in another workspace → 4xx; `save_to_kb=true` with NULL folder → allowed. -- **Cron validation**: invalid `schedule_cron` → 422; NULL cron → allowed (manual-only). -- **Cascade**: deleting a connector deletes its pipelines + their runs; deleting a folder SET-NULLs `destination_folder_id` (pipeline survives); deleting a workspace removes pipelines + runs. -- **Manual run**: `POST /pipelines/{id}/run` inserts a `pending`/`manual` run and enqueues the task; `GET /pipelines/{id}/runs` returns it newest-first with correct `total`. -- **AuthZ**: cross-workspace access to any pipeline/run route → 404; missing `CONNECTORS_*` permission → 403. -- **Zero shape**: `expected_publication_shape` includes `pipelines`/`pipeline_runs` as full-row. - -## Risks / trade-offs - -- **`db.py` growth.** Adds two more models to an already-large module. Accepted for relationship/Zero/schema ergonomics + consistency with the other core tables; a later extraction to a `pipelines` package is additive. -- **Coexisting schedulers (double-bill window).** Until Phase 6 adds the de-dup guard, a connector with both its own periodic indexing and a wrapping pipeline can crawl/bill twice. Documented above; Phase 6 owns the fix. MVP exposure is small (operator-created overlap only). -- **Eligibility enforced in code, not SQL.** `is_pipeline_eligible` is a registry (04a) check at the route, so a direct DB insert could bypass it. Same trade-off 04a already accepted (no DB column for the taxonomy). -- **Pre-built unused columns.** `crawls_*` / `result_blob_key` / `charged_micros` ship empty in Phase 5. Deliberate — avoids a second migration in Phase 6. -- **`connector_id` CASCADE drops run history.** Deleting a connector erases its pipelines' audit trail. If audit retention becomes a requirement, switch to `SET NULL` + a discriminator to distinguish Uploads from orphaned (additive change). -- **`created_by_id` nullable shifts work to Phase 6.** SET-NULL preserves shared pipelines past member deletion (the goal) but means the run engine cannot assume a creator — it must resolve `created_by ?? workspace owner` for the indexing actor + billing target. Flagged in the Phase-6 handoff; harmless until then (no engine). -- **Static-registry strand.** A pipeline created over an `AVAILABLE` connector can later reference a type the registry flips to `MIGRATING` on a deploy (create-time check passed; the row persists). Phase 6 must re-check `is_pipeline_eligible` at run time and fail the run cleanly. The data model intentionally does not encode eligibility, so no migration is needed when the registry changes. - -## Out of scope (hand-offs) - -- Run **engine**, scheduling tick over `pipelines.next_scheduled_at`, crawl-billing wiring (write `charged_micros`/`crawls_*`), raw-blob persistence, and chat-agent run-history context → **Phase 6** (`06-pipelines-exec.md`). Phase 6 also owns three guards this model only *enables*: (a) **run-time eligibility re-check** (`is_pipeline_eligible` — a deploy can flip a connector type to `MIGRATING`, stranding existing pipelines → fail the run cleanly, don't crash); (b) **acting-user resolution** (`created_by_id` can be NULL after member deletion → resolve `created_by ?? workspace owner` for indexing + billing); (c) **concurrency** (Redis indexing lock per pipeline/connector). -- File upload creating/using an "Uploads" pipeline (`connector_id=NULL`, `trigger=upload`, always `save_to_kb`) + generalizing opt-in KB save → **Phase 7** (`07-upload-pipeline-kb.md`). -- Document→run provenance column → deferred (Phase 6/7 if needed). -- Frontend Pipelines UI (list/create/configure/run-history/manual run) → frontend umbrella. -- Public pay-as-you-go API over Type-1 pipelines → post-MVP (umbrella "Deferred"). diff --git a/plans/backend/06-ongoing-automation.md b/plans/backend/06-ongoing-automation.md new file mode 100644 index 000000000..efb70a32f --- /dev/null +++ b/plans/backend/06-ongoing-automation.md @@ -0,0 +1,25 @@ +# Phase 6 — Ongoing Automation (chat-native "keep watching") + +> **Design deferred — placeholder.** The mechanism is designed separately, after `04`/`05`/`07`. +> Depends on `04` (the verbs it re-invokes) and `05` (the chat surface + delivery channel). + +## Objective + +Support "keep watching": a persistent, ongoing chat where the agent periodically re-invokes scraper verbs +and drops results into the session. The agent derives what's new by reading the chat history (time-based +search over prior tool outputs in context). + +## Open design questions (resolve together) + +1. **Periodicity driver** — the existing automations schedule selector, a recurring task, or a persistent + agent loop. +2. **Delivery channel** for between-turn results — existing SSE stream vs a Zero-published messages table. +3. **Context-window limit** — how far back "what changed" can reason before summarization/compaction. +4. **Loop owner** — the `07` subagent, or a thin automation wrapper that invokes the agent. +5. **Stop / pause / cost controls** for a running watch. + +## Out of scope + +- The verbs → `04`. The chat surface + delivery → `05`. The agent playbook → `07`. + +> Next step: design the periodic mechanism here, then fill in Target design / Work items / Tests. diff --git a/plans/backend/06-pipelines-exec.md b/plans/backend/06-pipelines-exec.md deleted file mode 100644 index d847c512a..000000000 --- a/plans/backend/06-pipelines-exec.md +++ /dev/null @@ -1,350 +0,0 @@ -# Phase 6 — Pipeline execution + scheduling (run engine, crawl billing, blob, chat context) - -> # ❌ SUPERSEDED (2026-06-30) -> **This entire phase is dropped.** Per the **Architecture correction** in `00-umbrella-plan.md`, there is no pipeline run engine or scheduler. The existing **automations** system already runs the full chat agent (with the native `web_search`/`scrape_webpage` tools) on **cron/event** triggers and persists **`automation_runs`** — which is exactly what this phase was building. Retained **for historical context only** — do **not** implement it. **Its still-relevant ideas survive in the canonical revamp:** (1) **billing crawls done outside chat** → now handled at the **capability executor** ([`revamp/04a`](revamp%20phases%204-7/04a-capabilities.md)), so recurring/automation runs meter correctly; (2) the refresh **hot loop** (crawl → diff → judge → append) → [`revamp/05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md), writing the **Timeline** ([`revamp/05a`](revamp%20phases%204-7/05a-timeline.md)) — not `automation_runs.output`; (3) **recurrence + "run now"** → [`revamp/06-triggers.md`](revamp%20phases%204-7/06-triggers.md) (reuse automations via a CI action). The `bill=False` seam and `index_crawled_urls` `folder_id`/`stats` params described below are **not needed** (no pipeline path). - -> Part of the **CI Pivot MVP**. See `00-umbrella-plan.md` (Phase 6). -> Precondition: Phase 5 (`05-pipelines-model.md`) live — `pipelines` / `pipeline_runs` tables, schemas, Zero, CRUD + `/run` (currently enqueues a **stub** `run_pipeline(run_id)`). Depends on `03a-crawler-core.md` (`crawl_url` SUCCESS signal + `crawls_succeeded` counter) and `03c-crawl-billing.md` (`WebCrawlCreditService`). Sibling ahead: `07-upload-pipeline-kb.md` (uploads-as-pipeline). - -> **Implementation note (post-rename).** Phases 1–2 are **SHIPPED**, so the live code already uses `workspace_id` / `Workspace` / `workspaces`. Citations below that still show `search_space_id` / `SearchSpace` / `searchspaces` are pre-rename — substitute the `workspace_*` equivalent and grep the new name. **New code in this plan uses the canonical `workspace_*` names** (snippets below updated; e.g. the `get_pipeline_runs` tool's dep key is `workspace_id`, matching the renamed deps dict). Locate code by **symbol/grep**, not the absolute line numbers cited here (the rename + Phases 3–5 shift them). - -## Objective - -Make pipelines actually **run**. Phase 5 shipped the data model + a stub task; this phase fills in: - -1. **Run engine** — turn a `PipelineRun` row into work: resolve the pipeline + connector, re-check eligibility, acquire a lock, crawl the source (WebURL crawler = the only MVP executor), optionally save to the KB destination folder, persist a raw-result blob otherwise, write the run record (status/timing/counts/error). -2. **Crawl billing wiring** (carry-over from `03c`) — bill `crawls_succeeded` to the **workspace owner** via `WebCrawlCreditService` for **both** KB-save branches (a `save_to_kb=false` run must not crawl for free); record `charged_micros` on the run for idempotency. -3. **Scheduling** — a Celery Beat tick that fires due `pipelines.next_scheduled_at` (cron), modeled on the **automations** schedule selector (cron + `FOR UPDATE SKIP LOCKED` + self-heal). Plus the **de-dup guard** Phase 5 deferred (a pipeline over a connector disables that connector's own periodic indexing). -4. **Chat-agent context** — a read-only main-agent **tool** (`get_pipeline_runs`) exposing recent run history scoped to the active workspace (umbrella default = tool). - -This is backend-only; the Pipelines UI is deferred to the frontend umbrella. - -## Locked decisions (MVP) - -| Concept | Decision | -|---------|----------| -| MVP executor | **WebURL crawler only.** `connector.connector_type == WEBCRAWLER_CONNECTOR`. Any other Type-1 type (GDrive/OneDrive/Dropbox) → run fails cleanly with `"executor not implemented for "` (their connector-level periodic path still works; pipelines over them are a Phase-7+ concern). `connector_id IS NULL` (Phase-7 Uploads) → not run by this engine (uploads register their own run in Phase 7). | -| Run-engine shape | Mirror **automations**: thin Celery task → `run_async_celery_task` → async `execute_pipeline_run(session, run_id)` (`automations/tasks/execute_run.py:19-33`). PENDING-gated + idempotent terminal no-op (`automations/runtime/executor.py:29-30`). Status lifecycle `pending → running → succeeded/failed` with `started_at`/`finished_at` (mirrors `repository.mark_running/mark_succeeded/mark_failed`). | -| KB-save path | **Reuse `index_crawled_urls`** (the battle-tested 2-phase indexer) for `save_to_kb=true`, extended with a `folder_id` param (set on created/updated `Document`s → destination folder). `save_to_kb=true` + NULL folder → root (existing behaviour; `documents.folder_id` nullable). | -| Non-KB path | `save_to_kb=false` → a **fetch-only** crawl loop (no `Document` rows): crawl each URL, collect `{url, content, metadata}`, persist as **one JSON blob** via `file_storage` backend, store `result_blob_key` on the run. (The 2-phase indexer is `Document`-centric and cannot "not persist"; a separate small loop is cleaner than bending it.) | -| Billing ownership | **Run engine owns billing for the pipeline path** (pre-check on `len(urls)` + charge on `crawls_succeeded` + record `charged_micros`), calling the crawler with **billing suppressed** (`bill=False`). The connector `/index` + connector-periodic paths keep `03c`'s in-indexer billing (`bill=True`). This keeps `charged_micros` run-idempotency clean and bills both KB branches identically. **(03c coordination — see below.)** | -| Scheduler | New `pipeline_schedule_select` Beat task modeled on `automations/triggers/builtin/schedule/selector.py` (cron, `FOR UPDATE SKIP LOCKED`, self-heal of NULL `next_scheduled_at`), **not** the simpler connector `frequency_minutes` checker — because Phase 5 chose `schedule_cron`. Reuses the existing `croniter` util (`automations/triggers/builtin/schedule/cron.py`). | -| Schedule timezone | Cron needs a timezone (`compute_next_fire_at(cron, timezone, …)`). Phase 5's model has only `schedule_cron`. **Add `schedule_timezone VARCHAR NOT NULL DEFAULT 'UTC'` to `pipelines`** (small additive amendment to `05` — see "Required 05 amendment"). Matches how automations store cron **and** timezone (`schedule/selector.py:86-89`). | -| Concurrency | Per-pipeline Redis lock + (when connector-backed) the **existing connector lock** `utils/indexing_locks.py` so a pipeline run and any residual connector index can't double-crawl the same connector. | -| Chat context | A main-agent **tool** `get_pipeline_runs` (registry pattern, `main_agent/tools/registry.py:85-102`), opening its own `shielded_async_session()` (like `KnowledgeTreeMiddleware`) and scoping to the build-time `workspace_id`. Always-on middleware injection is **deferred** (token cost; tool is on-demand). | -| Code location | New `app/pipelines/` package: `engine.py` (`execute_pipeline_run`), `tasks.py` (Celery `run_pipeline` + `pipeline_schedule_select`), `scheduler.py` (the tick), `storage.py` (blob key). The Phase-5 stub `run_pipeline` is **replaced** by the real task here; the `/run` route + scheduler enqueue it. (ORM stays in `db.py` per Phase 5; only the *engine* is a package, mirroring `app/automations/`.) | - -## Current state (cited) - -### Run-engine precedent (automations) - -- **Task wrapper**: `automations/tasks/execute_run.py:19-33` — `@celery_app.task(name=..., bind=True)` → `run_async_celery_task(lambda: _impl(run_id))`; `_impl` opens `get_celery_session_maker()()` and calls `execute_run(session, run_id)`, rolling back + re-raising on failure. -- **Launch**: `automations/dispatch/launch.py:43-60` — create the child row `status=PENDING`, `session.add` + `commit` + `refresh`, then `task.apply_async(args=[run.id], time_limit=…)`. (Phase 5's `/run` already does the create+enqueue; this phase makes the task real.) -- **Executor lifecycle**: `automations/runtime/executor.py:23-75` — `load run; if run.status != PENDING: return` (idempotent terminal no-op, `:29-30`); `mark_running` + `commit` (`:46-47`); on bad snapshot → `mark_failed` + commit (`:35-44`); terminal `mark_succeeded`/`mark_failed` each `commit`. - -### The MVP executor: `webcrawler_indexer.index_crawled_urls` - -- Signature `webcrawler_indexer.py:44-53`: `(session, connector_id, search_space_id, user_id, start_date=None, end_date=None, update_last_indexed=True, on_heartbeat_callback=None) -> (int, str|None)`. -- Reads URLs from `connector.config["INITIAL_URLS"]` via `parse_webcrawler_urls` (`:118-119`). -- **Creates `Document` rows with NO `folder_id`** (`:222-239`) → today everything lands at workspace root. Updates existing docs in place. Crawls via `crawler.crawl_url(url)` (`:297`; `03a` changes this signature/return). Tracks `documents_indexed` / `documents_updated` / counts; final commit at `:432`; returns `(total_processed, error)` (`:477`). -- **No billing today** — `03c` adds the owner pre-check + `charge_credits` inside this function (`03c` §2). **This phase adds a `bill: bool = True` seam** so the pipeline path can suppress it (plus `folder_id` / `urls` params + a `stats` out-param — see "Crawler changes"; the positional return is left unchanged so the shared wrapper keeps working). -- Connector entrypoint chain (the path the pipeline engine **bypasses**, because it's notification-centric, not run-centric): `connector_tasks.py:418-454` (`index_crawled_urls_task` → `_index_crawled_urls` → `run_web_page_indexing`) → a `_run_indexing_with_notifications(...)` **call site** (e.g. the WebCrawler one at `search_source_connectors_routes.py:~2524`). The wrapper itself is **defined at `:1296`** (acquires the connector lock at `:1334`, length-unpacks the indexer return at `:1503`). - -### KB-ingest service (folder support already exists there) - -- `IndexingPipelineService` (`indexing_pipeline/indexing_pipeline_service.py:82`) already threads a destination folder via `ConnectorDocument.folder_id` (`:273-274`, `:301-302`, `:326`). **But the webcrawler indexer does not use this service** — it writes `Document`s directly. So the umbrella's "route through `IndexingPipelineService` into the destination folder" is accurate for **file/Composio** sources, **not** the crawler. For the WebURL MVP, the minimal correct move is to thread `folder_id` into `index_crawled_urls` (not refactor the crawler onto `IndexingPipelineService`). Unifying them is a post-MVP cleanup. - -### Scheduler precedent (cron) + Beat registration - -- **Selector** `automations/triggers/builtin/schedule/selector.py`: tick (`:47-65`) → self-heal NULL `next_fire_at` (`:68-98`) → claim due rows `FOR UPDATE SKIP LOCKED` + advance via `compute_next_fire_at` + set `last_fired_at` (`:101-150`) → `_start_one` launches each (`:153-183`). `_TICK_BATCH=200` cap (`:35`). -- **Cron util** `automations/triggers/builtin/schedule/cron.py`: `validate_cron(cron, timezone)` (`:15`) and `compute_next_fire_at(cron, timezone, *, after)` (`:28`, returns UTC). **Requires a timezone** → drives the 05 amendment. -- **Beat source** `automations/triggers/builtin/schedule/source.py:14-20` — a `BEAT_SCHEDULE` dict (`crontab(minute="*")`) merged into the app at `celery_app.py:323` (`**SCHEDULE_BEAT_SCHEDULE`). The app's beat dict is `celery_app.py:268-324`; the worker `include` list is `:180-198`; task routing (`index_crawled_urls → CONNECTORS_QUEUE`) is `:237-255` (`:246`). -- **Connector meta-scheduler we coexist with**: `schedule_checker_task.py:17` scans `SearchSourceConnector` where `periodic_indexing_enabled AND next_scheduled_at <= now` (`:33-39`), with a Redis-lock guard `is_connector_indexing_locked` (`:93`). Untouched; the pipeline tick is a sibling. - -### Locks, storage, chat tooling - -- **Locks** `utils/indexing_locks.py`: `acquire_connector_indexing_lock(connector_id)->bool` (`:24`), `release_…` (`:37`), `is_…_locked` (`:43`), keyed `indexing:connector_lock:{id}`, TTL `CONNECTOR_INDEXING_LOCK_TTL_SECONDS`. -- **Storage** `file_storage/service.py:39,46`: `get_storage_backend()` (from `file_storage/factory.py`) → `backend.put(key, data, content_type=…)` / `open_stream(key)` / `delete(key)`. Keys are built per-purpose (`file_storage/keys.py:11`, `build_document_file_key`). For runs we add a sibling key builder (no `document_files` row — runs aren't documents). -- **Tool registry** `main_agent/tools/registry.py:85-102` — `_MAIN_AGENT_TOOL_FACTORIES: {name: (factory, required_dep_names)}`; `build_main_agent_tools(deps, …)` resolves deps + builds (`:105-147`). Display metadata lives in `shared/tools/catalog`; the main-agent name list in `main_agent/tools/index` (per the registry docstring `:1-14`). Read-only DB pattern for a chat surface: `KnowledgeTreeMiddleware` opens `shielded_async_session()` and filters by its build-time `search_space_id` (`knowledge_tree/middleware.py:199-206`). -- **Workspace owner** (billing target): `SearchSpace.user_id` (`03c` §2.1 resolves the owner with a direct `select SearchSpace.user_id where id == search_space_id`). - -## Target design - -### 1. `execute_pipeline_run(session, run_id)` — the engine (`app/pipelines/engine.py`) - -``` -load run (pipeline + connector eager); if run is None: raise -if run.status != PENDING: return # status gate; also no-ops acks_late redelivery (executor.py:29-30) -pipeline = run.pipeline -mark_running(run): status=RUNNING, started_at=now; commit -# ^ committed BEFORE any crawl/charge → a redelivered/duplicate task sees RUNNING and returns (executor.py:46-47) - -pipeline_lock = connector_lock = False -connector = None -try: - # --- guards: fail cleanly, never crash the worker (Phase-5 hand-offs (a)/(b)) --- - if pipeline.connector_id is None: # Uploads pipeline — Phase 7, not here - return fail(run, "uploads pipeline has no fetch executor") - connector = pipeline.connector - if not is_pipeline_eligible(connector.connector_type): # 04a runtime re-check (strand guard) - return fail(run, f"connector type {connector.connector_type} no longer eligible") - if connector.connector_type != WEBCRAWLER_CONNECTOR: - return fail(run, f"executor not implemented for {connector.connector_type}") - - owner_id = workspace_owner_id(session, pipeline.workspace_id) # select Workspace.user_id (03c §2.1) - actor_user_id = str(pipeline.created_by_id or owner_id) # Phase-5 hand-off (b) - urls = parse_webcrawler_urls(pipeline.config.get("INITIAL_URLS") or connector.config.get("INITIAL_URLS")) - if not urls: - return fail(run, "no URLs to crawl") - - # --- concurrency: take both; release only what we took (Phase-5 hand-off (c)) --- - pipeline_lock = acquire_pipeline_indexing_lock(pipeline.id) - if not pipeline_lock: - return fail(run, "a run for this pipeline is already in progress") - connector_lock = acquire_connector_indexing_lock(connector.id) - if not connector_lock: - return fail(run, "connector is currently indexing; retry later") - - # --- billing pre-flight (run-level; 03c statics). len(urls) is a safe upper bound. --- - svc = WebCrawlCreditService(session) - if WebCrawlCreditService.billing_enabled() and run.charged_micros is None: - await svc.check_credits(owner_id, len(urls)) # InsufficientCreditsError → except → fail("out of crawl credit") - - # --- fetch --- - if pipeline.save_to_kb: - stats = {} # out-param sink — NON-breaking (see §3) - await index_crawled_urls(session, connector.id, pipeline.workspace_id, actor_user_id, - urls=urls, folder_id=pipeline.destination_folder_id, - bill=False, stats=stats, update_last_indexed=True) - documents_indexed = stats["documents_indexed"] - crawls_attempted = stats["crawls_attempted"] - crawls_succeeded = stats["crawls_succeeded"] - result_blob_key = None - else: - outcomes = await crawl_urls_fetch_only(urls) # list[dict] from CrawlOutcome (03a) - crawls_attempted = len(urls) - crawls_succeeded = sum(1 for o in outcomes if o["status"] == "success") - documents_indexed = 0 - result_blob_key = await persist_run_blob(run, outcomes) - - # --- charge: stamp run + add audit, THEN charge (its single commit flushes all three atomically) --- - if WebCrawlCreditService.billing_enabled() and crawls_succeeded > 0 and run.charged_micros is None: - run.charged_micros = WebCrawlCreditService.successes_to_micros(crawls_succeeded) - record_token_usage(session, usage_type="web_crawl", workspace_id=pipeline.workspace_id, - user_id=owner_id, cost_micros=run.charged_micros, - call_details={"urls": len(urls), "successes": crawls_succeeded, - "pipeline_id": pipeline.id, "run_id": run.id}, - message_id=None) # add only — does not commit (03c §2.3) - await svc.charge_credits(owner_id, crawls_succeeded) # debits + COMMITS run+audit+balance (03c §2.4) - - mark_succeeded(run): status=SUCCEEDED, finished_at=now, - documents_indexed, crawls_attempted, crawls_succeeded, result_blob_key; commit -except InsufficientCreditsError: - return fail(run, "out of crawl credit") # run is RUNNING→FAILED; no debit happened -except Exception as e: - return fail(run, {"message": str(e), "type": type(e).__name__}) -finally: - if connector_lock: release_connector_indexing_lock(connector.id) - if pipeline_lock: release_pipeline_indexing_lock(pipeline.id) -``` - -`fail(run, err)` = set `status=FAILED`, `finished_at=now`, `error={...}`, `commit`, return — so the `finally` still runs and releases only the locks actually taken. `connector.id` in `finally` is reached only when `connector_lock is True`, which implies `connector` was resolved. **Idempotency rests on the status gate**, not `charged_micros`: `mark_running` commits `RUNNING` before any crawl/charge, so an `acks_late` redelivery (`celery_app.py:228-229`) or a manual re-enqueue sees `RUNNING`/terminal and returns. `charged_micros` is the belt-and-suspenders guard reserved for a future stuck-`RUNNING` re-driver (none in MVP). - -Notes: -- **`workspace_owner_id`** = `select Workspace.user_id where id == workspace_id` (`03c` §2.1 pattern). -- **`is_pipeline_eligible`** is the 04a registry helper — re-checked here at run time (a deploy can flip a type to `MIGRATING`; the create-time check in Phase 5 doesn't protect the row forever). -- **URL source**: per-pipeline `config["INITIAL_URLS"]` overrides the connector's (the `config` JSONB "per-pipeline overrides" Phase 5 reserved); fall back to the connector's `INITIAL_URLS`. - -### 2. Celery tasks (`app/pipelines/tasks.py`) - -```python -@celery_app.task(name="run_pipeline", bind=True) -def run_pipeline(self, run_id: int) -> None: - return run_async_celery_task(lambda: _run(run_id)) # execute_run.py:19-22 shape - -async def _run(run_id: int) -> None: - async with get_celery_session_maker()() as session: - try: - await execute_pipeline_run(session, run_id) - except Exception: - logger.exception("pipeline run %d failed unexpectedly", run_id) - await session.rollback(); raise -``` - -- `run_pipeline` **replaces** the Phase-5 stub of the same name; Phase 5's `/run` route already enqueues `run_pipeline(run.id)` (no route change needed). -- **Routing/registration**: ensure `"app.pipelines.tasks"` is in the `include` list (`celery_app.py:180-198`) — Phase 5 already added it for the stub; no-op if present — and route `"run_pipeline": {"queue": CONNECTORS_QUEUE}` (it crawls — same queue as `index_crawled_urls`, `celery_app.py:246`). Give the `apply_async`/`delay` a `time_limit` (the connectors queue already has an 8h hard cap, `celery_app.py:219`). - -### 3. Crawler changes — `index_crawled_urls` (coordination with `03a`/`03c`) - -Four additive params, **all default-compatible** with the existing connector path: - -1. **`folder_id: int | None = None`** — set on each created `Document` (`:222-239`) and on the existing-doc update branch, so KB-save lands in the destination folder. Default `None` = root (today's behaviour). -2. **`urls: list[str] | None = None`** — when provided, crawl this list instead of re-reading `connector.config["INITIAL_URLS"]` (`:118-119`). **Required** for the per-pipeline `config["INITIAL_URLS"]` override (a Phase-5 `config` capability) and to keep the engine's `len(urls)` billing pre-check consistent with what's actually crawled. Default `None` = read connector config (today's behaviour). -3. **`bill: bool = True`** — gate `03c`'s in-indexer `check_credits` + `charge_credits` + `record_token_usage`. The pipeline engine calls `bill=False` and bills at the run level; the connector `/index` + periodic paths keep `bill=True`. *(Small addition to `03c`'s §2 wiring — see "Cross-plan coordination".)* -4. **`stats: dict | None = None`** — an **out-param sink**, NOT a return-shape change. When provided, the indexer populates `stats["documents_indexed"|"crawls_attempted"|"crawls_succeeded"]`. **Do NOT widen the return tuple:** the shared `_run_indexing_with_notifications` wrapper unpacks every indexer's return **by length** (`search_source_connectors_routes.py:1499-1507`: `if len(result) == 3: a,b,c = result else: a,b = result`), so a 4-tuple would raise `ValueError` for *every* connector. The 2-/3-tuple return stays exactly as-is (wrapper untouched); the engine passes a fresh `stats` dict and reads it after the call (the wrapper passes none → `None` → no-op). `03a` work-item 3 already reflects this (it exposes `crawls_succeeded` via **task metadata** only, with this `stats` sink owned by Phase 6 — not via the positional return). - -The engine **bypasses** `_run_indexing_with_notifications` (it acquires the connector lock itself at `:1334` — going through the wrapper would self-deadlock on that same lock, and the wrapper's connector-`Notification` UI is the wrong surface for a pipeline run, which has its own `PipelineRun` record). It calls `index_crawled_urls` directly with no heartbeat callback. (Consequence: pipeline runs don't emit connector-indexing notifications and aren't touched by `cleanup_stale_indexing_notifications` — fine; run status lives on `pipeline_runs`.) - -### 4. Fetch-only path + blob (`app/pipelines/engine.py` + `app/pipelines/storage.py`) - -For `save_to_kb=false`: - -```python -async def crawl_urls_fetch_only(urls): - crawler = WebCrawlerConnector() # 03a: no firecrawl ctor arg - out = [] - for url in urls: - outcome = await crawler.crawl_url(url) # 03a: CrawlOutcome(status, result, error, tier) - out.append({ - "url": url, - "status": outcome.status.value, # CrawlOutcomeStatus: success | empty | failed - "content": (outcome.result or {}).get("content", ""), - "metadata": (outcome.result or {}).get("metadata", {}), - "error": outcome.error, - }) - return out -``` - -> **03a return shape.** `03a` is **committed to the `CrawlOutcome` dataclass** (`status`/`result`/`error`/`tier`) — the tuple option was dropped precisely because this fetch-only path consumes `.status`/`.result`/`.error` as attributes. So the unpack above is the firm contract; no adaptation needed. - -```python -# storage.py -def build_pipeline_run_result_key(*, workspace_id: int, run_id: int) -> str: - return f"pipeline_runs/{workspace_id}/{run_id}/result.json" - -async def persist_run_blob(run, outcomes) -> str: - key = build_pipeline_run_result_key(workspace_id=run.pipeline.workspace_id, run_id=run.id) - data = json.dumps({"runs": outcomes}).encode() - await get_storage_backend().put(key, data, content_type="application/json") - return key # → run.result_blob_key -``` - -- One blob per run (single configured backend, like document files). `result_blob_key` (Phase 5 column) stores the key; reads resolve via `get_storage_backend()`. -- **Billing still applies** here — `crawls_succeeded` is counted from `outcomes`, so non-KB runs cost exactly the same as KB runs (umbrella line 105). This is the whole reason the run engine, not the indexer, owns pipeline billing. - -### 5. Scheduler tick (`app/pipelines/scheduler.py` + `tasks.py`) - -A near-copy of the automations selector, retargeted at `pipelines`: - -```python -@celery_app.task(name="pipeline_schedule_select") -def pipeline_schedule_select() -> None: - return run_async_celery_task(_tick) - -async def _tick(): - async with get_celery_session_maker()() as session: - now = datetime.now(UTC) - await _self_heal_null_next(session, now) # enabled + cron + next_scheduled_at IS NULL → compute - claims = await _claim_due(session, now) # enabled + next_scheduled_at <= now, FOR UPDATE SKIP LOCKED, advance - for c in claims: - await _start_run(session, c) # create PipelineRun(trigger=scheduled, status=pending) + run_pipeline.apply_async -``` - -- **`_claim_due`**: `select Pipeline where enabled AND connector_id IS NOT NULL AND schedule_cron IS NOT NULL AND next_scheduled_at IS NOT NULL AND next_scheduled_at <= now ORDER BY next_scheduled_at LIMIT 200 FOR UPDATE SKIP LOCKED` (uses Phase 5's partial `ix_pipelines_due`). The `connector_id IS NOT NULL` filter is **defense-in-depth** so a non-connector pipeline (the Phase-7 Uploads pipeline) can **never** be scheduled into perpetually-failing runs even if a cron is somehow set on it — the engine would `fail` every such run (`§1`); this stops them at the selector. (Phase 7 also guards the API; this is the belt to that suspenders.) Advance `next_scheduled_at = compute_next_fire_at(schedule_cron, schedule_timezone, after=now)`; on `InvalidCronError` → `enabled=False` (self-disable, like `selector.py:90-96,131-137`). Commit, then create the run + enqueue (mirror `_start_one` reload-after-commit, `selector.py:153-183`). -- **`_self_heal_null_next`**: backfills `next_scheduled_at` for enabled cron pipelines missing it (fresh inserts, restored rows) — so Phase 5's create can leave it NULL and the tick fills it within ≤1 min. -- **Duplicate suppression**: in `_start_run`, skip creating a scheduled run if the pipeline already has a non-terminal run (`status IN (pending, running)`) — mirrors the connector checker's in-progress guard (`schedule_checker_task.py:100-116`). Without it, a pipeline whose run outlasts its cron interval piles up `scheduled` runs that immediately `fail` on the lock (noisy, though harmless). `next_scheduled_at` is still advanced (the fire isn't retried — `catchup=False`, like the automations selector). -- **Beat entry**: add `"pipeline-schedule-select": {"task": "pipeline_schedule_select", "schedule": crontab(minute="*"), "options": {"expires": 50}}` to `celery_app.py:268-324` (default fast queue, like `check_periodic_schedules`). (`"app.pipelines.tasks"` is already in `include` from Phase 5's stub, so `pipeline_schedule_select` in the same module is auto-registered.) -- **`schedule_cron` edits** (Phase 5 `PUT`): when `schedule_cron` (or timezone) changes, **null `next_scheduled_at`** so the tick recomputes (small `PUT`-handler addition — see Required 05 amendment). NULL cron ⇒ manual-only (tick ignores it). - -### 6. De-dup guard (resolves Phase-5 deferred double-bill) - -When a pipeline that **wraps a connector** is created or enabled (Phase 5 routes), set that connector's `periodic_indexing_enabled = False` (+ `next_scheduled_at = None`) so a single scheduler owns each connector (the **pipeline is authoritative**, per Phase 5's recommendation). Two touch points: - -- Phase 6 owns the rule; implement it in the Phase-5 `POST /pipelines` and `PUT /pipelines/{id}` handlers (when `connector_id` is set and `enabled`). This is a tiny handler addition (Phase 5 explicitly handed it here). -- Belt-and-suspenders: the run engine already takes the **connector lock** during a run, so even if both scan, only one crawl proceeds at a time (the connector checker skips on a held lock, `schedule_checker_task.py:93`). - -### 7. Chat-agent run-history tool (`main_agent/tools/pipeline_runs.py`) - -A read-only tool registered in the main-agent registry (umbrella default = tool): - -```python -def create_get_pipeline_runs_tool(*, workspace_id: int) -> BaseTool: - @tool - async def get_pipeline_runs(pipeline_id: int | None = None, limit: int = 20) -> str: - """Recent pipeline run history for THIS workspace: name, status, trigger, - schedule, counts, started/finished. Read-only.""" - async with shielded_async_session() as session: # knowledge_tree/middleware.py:199 pattern - q = (select(PipelineRun, Pipeline.name, Pipeline.schedule_cron) - .join(Pipeline, PipelineRun.pipeline_id == Pipeline.id) - .where(Pipeline.workspace_id == workspace_id)) - if pipeline_id is not None: - q = q.where(Pipeline.id == pipeline_id) - rows = (await session.execute( - q.order_by(PipelineRun.created_at.desc()).limit(min(limit, 100)))).all() - return render_compact(rows) # name • status • trigger • finished_at • docs/crawls - return get_pipeline_runs -``` - -- **Registration**: add `"get_pipeline_runs": (_build_get_pipeline_runs_tool, ("workspace_id",))` to `_MAIN_AGENT_TOOL_FACTORIES` (`registry.py:85-102`) — the dep key is `workspace_id`, matching the post-rename deps dict; add the name to `main_agent/tools/index`; add display metadata to `shared/tools/catalog`. -- **Workspace scope is structural** — the tool only ever sees its build-time `workspace_id`; no cross-workspace leakage (same guarantee `KnowledgeTreeMiddleware` relies on). -- **Anonymous / no-workspace turns**: exclude the tool when there's no `workspace_id` (anonymous chat has no workspace/pipelines), the same way `KnowledgeTreeMiddleware` is CLOUD-only (`knowledge_tree/middleware.py:130-131`). The registry factory requires `workspace_id`, so just don't enable it in the anonymous tool set. -- The tool reads `pipelines`/`pipeline_runs` directly (no Zero dependency), so it works server-side immediately even though client Zero sync is dormant until the frontend lands (Phase 5 note). -- **Deferred**: an always-on `` middleware injection (à la `KnowledgeTreeMiddleware`) — adds per-turn token cost; revisit if the agent under-uses the tool. - -### 8. (Optional) blob read endpoint - -Phase 5 stores `result_blob_key` but exposes no reader. Add `GET /pipelines/{id}/runs/{run_id}/result` → stream the blob via `get_storage_backend().open_stream(run.result_blob_key)` (404 if NULL), `CONNECTORS_READ`, workspace-scoped — mirroring `file_storage/api.py:69-90`. Lets API consumers + the chat tool fetch non-KB results. (Small; include if cheap, else defer to the public-API phase.) - -## Required `05` amendment (timezone + recompute) - -Small, additive. **Items 1–3 (the `schedule_timezone` column) are already folded into `05`** (model/migration/schema/validator) — listed here for traceability. **Item 4 is Phase-6 route logic** added on top of Phase 5's handlers (Work items 5–6): - -1. *(done in `05` §2)* **Model**: `schedule_timezone = Column(String(64), nullable=False, default="UTC", server_default="UTC")` on `Pipeline`. -2. *(done in `05` §4)* **Migration**: `schedule_timezone VARCHAR(64) NOT NULL DEFAULT 'UTC'` in `CREATE TABLE pipelines`. -3. *(done in `05` §5)* **Schema**: `schedule_timezone: str = "UTC"` on `PipelineBase` + `PipelineUpdate`; `(schedule_cron, schedule_timezone)` validated together via `validate_cron(cron, tz)` (`cron.py:15`). -4. **Routes** (Phase 6 adds to `05` §7 handlers): on `PUT` when `schedule_cron`/`schedule_timezone` changes, set `next_scheduled_at = None` (tick self-heals). On `POST`/`PUT` with `connector_id` + `enabled`, apply the §6 de-dup guard. - -(If we'd rather not touch `05`: hardcode `"UTC"` everywhere in Phase 6 and skip the column — but that loses operator-meaningful local schedules and diverges from automations. The column is the right call.) - -## Cross-plan coordination - -- **`03a`** — Phase 6 calls `WebCrawlerConnector.crawl_url(url)` directly in the fetch-only path and relies on `03a`'s `CrawlOutcome.status` (`SUCCESS`/`EMPTY`/`FAILED`, dataclass form — already locked in `03a`). `03a` work-item 3 is **already aligned**: it surfaces the `crawls_succeeded` counter via **task metadata** only (not by widening `index_crawled_urls`'s positional return — the shared `_run_indexing_with_notifications` wrapper unpacks by tuple length, `:1499-1507`). Phase 6 owns the `folder_id`/`urls`/`stats` params; `03a` owns the counter + `CrawlOutcome`. -- **`03c`** — add the **`bill: bool = True`** parameter to the in-indexer billing (so the pipeline path can suppress it). `03c`'s `WebCrawlCreditService` statics (`billing_enabled()`, `successes_to_micros(n)`, `check_credits`, `charge_credits`) are reused verbatim by the run engine. `03c`'s risk note already anticipated this: *"Phase 5's `PipelineRun` can persist `charged_micros` for stronger idempotency"* — Phase 6 is where that lands. -- **`05`** — the four amendments above; `run_pipeline` task name + `/run` enqueue already exist (stub → real); `charged_micros`/`crawls_*`/`result_blob_key` columns already exist (pre-built in Phase 5). -- **`04a`** — `is_pipeline_eligible(connector_type)` is imported for the run-time re-check. - -## Work items - -1. **`app/pipelines/` package**: `engine.py` (`execute_pipeline_run` + `mark_running/succeeded/fail` helpers + `workspace_owner_id` + `crawl_urls_fetch_only`), `tasks.py` (`run_pipeline`, `pipeline_schedule_select`), `scheduler.py` (`_tick`/`_self_heal_null_next`/`_claim_due`/`_start_run`), `storage.py` (blob key + `persist_run_blob`). -2. **Pipeline lock helpers**: add `acquire_pipeline_indexing_lock(id)` / `release_pipeline_indexing_lock(id)` to `utils/indexing_locks.py` (key `indexing:pipeline_lock:{id}`), mirroring the connector lock (`:19-40`). -3. **Crawler changes**: `index_crawled_urls` gains `folder_id` + `urls` + `bill` + a `stats` **out-param** (populates `documents_indexed`/`crawls_attempted`/`crawls_succeeded`). **No return-shape change → no caller changes** (the shared wrapper's 2-/3-tuple unpack stays valid). Coordinate with `03a`/`03c` (see "Cross-plan coordination"). -4. **Celery wiring** (`celery_app.py`): ensure `"app.pipelines.tasks"` is in `include` (added in Phase 5 for the stub); route `run_pipeline → CONNECTORS_QUEUE`; add the `pipeline-schedule-select` Beat entry. -5. **De-dup guard** in Phase-5 `POST`/`PUT /pipelines` (disable wrapped connector's periodic indexing). -6. **`05` amendment**: `schedule_timezone` column/migration/schema/validator + `next_scheduled_at` reset on cron edit. -7. **Chat tool** `get_pipeline_runs` + registry/index/catalog registration. -8. **(Optional)** `GET …/runs/{run_id}/result` blob reader. -9. **Tests** (below). - -## Tests - -- **Manual run, KB-save**: WebURL pipeline `save_to_kb=true` + `destination_folder_id` → run goes `pending→running→succeeded`, `Document`s created **in that folder**, `documents_indexed`/`crawls_*` set, `finished_at` set. -- **Manual run, non-KB**: `save_to_kb=false` → **no** `Document`s, `result_blob_key` set + blob readable, `crawls_*` set. Billing **still** charges `crawls_succeeded` (parity with KB run). -- **Billing**: enabled + sufficient → owner debited `crawls_succeeded * 1000`, `charged_micros` stamped, one `web_crawl` `TokenUsage`; **insufficient → run fails pre-crawl (FAILED), no debit**; disabled (self-hosted) → no charge, no `web_crawl` row, `charged_micros` stays NULL. -- **Idempotency / lifecycle**: re-enqueuing a `RUNNING` or terminal run → immediate no-op (`status != PENDING` gate) → **no second charge** (the primary guard; `mark_running` commits before any charge); engine never raises out of `fail()` paths (bad type / NULL connector / no URLs / ineligible / lock contention) — all land the run in `FAILED` and release locks. -- **Eligibility re-check**: pipeline over a connector whose type is now `MIGRATING` → run fails cleanly (not crash). -- **Scheduler**: a due cron pipeline → tick creates a `scheduled` run + advances `next_scheduled_at` to the next cron match (in `schedule_timezone`); NULL `next_scheduled_at` self-heals; invalid cron self-disables (`enabled=False`); manual-only (NULL cron) never fires. -- **De-dup**: creating/enabling a pipeline over a connector flips that connector's `periodic_indexing_enabled=False`; concurrent connector-checker tick skips on the held lock. -- **Concurrency**: a second run while one is in-flight → fails with "already in progress" (locks held). -- **Chat tool**: `get_pipeline_runs` returns this workspace's runs only; another workspace's runs never appear; respects `limit`. -- **Counts contract (regression-critical)**: `index_crawled_urls` keeps its existing 2-/3-tuple return (the shared `_run_indexing_with_notifications` length-unpack still works for *all* connectors); when a `stats` dict is passed it's populated with `documents_indexed`/`crawls_attempted`/`crawls_succeeded`; `bill=False` suppresses in-indexer billing; connector path (`bill=True`, no `stats`) behaves byte-for-byte as before. Add a wrapper test asserting webcrawler indexing through `_run_indexing_with_notifications` still unpacks cleanly. - -## Risks / trade-offs - -- **Billing split across two call sites** (indexer for connector path, run engine for pipeline path), unified only by `03c`'s shared statics. Accepted: it's what makes `charged_micros` run-idempotency clean and keeps both KB branches billed identically. The `bill` flag is the single seam. -- **Stuck `RUNNING` runs.** A crash *after* `mark_running` but *before* terminal leaves `status=RUNNING` forever (the `status != PENDING` gate then no-ops any redelivery — same property as automations). No double-charge (gate + `charged_micros`), but the run never completes and its locks rely on Redis TTL to free. A stale-run sweeper (à la `cleanup_stale_indexing_notifications`, `celery_app.py:279-285`) is deferred; MVP exposure is a rare worker crash. -- **Two crawl code paths** (`index_crawled_urls` for KB vs `crawl_urls_fetch_only` for non-KB). Minor duplication of the crawl loop; deliberate, since forcing the `Document`-centric 2-phase indexer into a "don't persist" mode is messier. Both share `WebCrawlerConnector.crawl_url` + the same billing math. -- **WebURL-only executor.** A pipeline over GDrive/OneDrive/Dropbox is creatable (Phase 5 lets `AVAILABLE DATA_SOURCE`) but fails at run time ("executor not implemented"). Acceptable for MVP (those keep their connector-periodic path); generalize the executor in Phase 7+. -- **De-dup guard is one-way.** Creating/enabling a pipeline disables the wrapped connector's `periodic_indexing_enabled`, but **disabling or deleting that pipeline does NOT restore it** — so a connector can end up indexed by *neither* path (data goes stale) until the user re-enables connector periodic indexing manually. MVP-acceptable (operator action); a symmetric restore-on-disable is a small follow-up. Flag in the (deferred) UI copy. -- **Blob backend assumption.** `result_blob_key` stores only the key (no backend name, unlike `document_files.storage_backend`). Fine for the single configured backend; if multi-backend ever lands, add a backend column to `pipeline_runs` (additive). -- **Non-KB blob built in memory.** `crawl_urls_fetch_only` accumulates every URL's extracted content into one JSON blob before upload — a large URL list (or huge pages) inflates worker memory. Acceptable for MVP URL counts; if it bites, stream to the backend or cap per-run URLs (a `config` knob). Also: non-KB runs have **no dedup** (every run re-crawls + re-bills all URLs) — that's inherent to "don't persist", and the user opted out of KB. -- **Lock lifetime vs run length.** The Redis locks use `CONNECTOR_INDEXING_LOCK_TTL_SECONDS`; a crawl run longer than the TTL could let a second run start. Pre-existing property of the connector lock (the connectors queue caps at 8h); revisit the TTL if pipeline runs routinely exceed it. -- **Two minute-level Beat scans** (connector checker + pipeline tick) now run every minute. Both are indexed (`ix_pipelines_due` partial index; connector scan on its own predicate) + batch-capped (200), so cost is negligible. - -## Out of scope (hand-offs) - -- **Uploads pipeline execution** (`connector_id=NULL`, `trigger=upload`, always `save_to_kb`) → **Phase 7** (`07-upload-pipeline-kb.md`). This engine explicitly fails NULL-connector runs. -- **Non-WebURL executors** (GDrive/OneDrive/Dropbox pipelines) → Phase 7+. -- **Always-on chat context middleware** (`` injection) → deferred (tool-only for MVP). -- **Document→run provenance** (`documents.pipeline_run_id`) → deferred (touches `documents`). -- **Public pay-as-you-go API over pipelines** + **public MCP KB server** → post-MVP (umbrella "Deferred"). -- **Frontend** Pipelines UI (run-history, manual run, schedule editor) → frontend umbrella. diff --git a/plans/backend/07-orchestration.md b/plans/backend/07-orchestration.md new file mode 100644 index 000000000..09da4ba2e --- /dev/null +++ b/plans/backend/07-orchestration.md @@ -0,0 +1,73 @@ +# Phase 7 — Orchestration / Conversation (the CI-expert subagent) + +> Build last; sits atop `04`/`05` (and `06` for the ongoing mode). +> The multi-agent chat runtime (deepagents, subagent dispatch, streaming, citation middleware) is shipped. +> Net-new here = one builtin subagent + its tools + its prompt. Locate code by symbol/grep. + +## Objective + +Ship the `intelligence_agent` — a builtin CI/scraper-expert subagent that delivers the product through +plain conversation: understand intent, compose scraper verbs (`04`), answer in plain language, and hand +standing needs to the ongoing "keep watching" mode (`06`). It reasons over chat history to report what's +new vs prior tool outputs already in context. + +## The subagent + +New builtin `subagents/builtins/intelligence_agent/`, peer to `research`/`deliverables`, packed via +`pack_subagent`. Files follow the existing pattern: `agent.py` (`build_subagent`), `tools/index.py` +(`NAME · RULESET · load_tools`), `description.md` (router one-liner), `system_prompt.md` (the playbook). +The main agent delegates CI-/scraping-flavored requests to it via `description.md`. + +## The playbook (`system_prompt.md`) + +1. **Intent routing** — one-shot ("compare/find/what is") → compose verbs & answer; standing concern + ("watch/track/notify when/weekly") → hand to `06`; ambiguous → ask one question ("just once, or keep + watching?"). +2. **Verb composition** — chains `web.discover → web.scrape` and `maps.search → maps.place → maps.reviews`; + infer URLs/queries/locations from context. +3. **"What changed"** — re-invoke the verb and compare against prior tool outputs in the chat history, + summarizing what's new. + +## The toolset (`load_tools`) + +| Tool | Wraps | Billing | +|------|-------|---------| +| capability verbs (`web.scrape`, `web.discover`, `maps.search`, `maps.place`, `maps.reviews`) | `04` executors (direct-return) | `03c` turn accumulator (as `scrape_webpage` does) | +| `start_watch(intent)` *(deferred)* | hands off to `06` | per re-invocation (design with `06`) | + +Capability verbs are a shared, registry-generated tool module (`04`) that `research` can also load; the +`intelligence_agent` adds the CI prompt and the `06` handoff. + +## Boundaries + +- The subagent drives the `04` verbs via tools; fetch/bypass/clean logic stays in Acquisition + `04`, + callable headless. +- Humans get the agent; REST/MCP callers call `04` verbs directly, skipping this subagent. + +## Work items + +1. Subagent scaffold: `subagents/builtins/intelligence_agent/` (`agent.py` / `tools/index.py` / + `description.md` / `system_prompt.md`), packed via `pack_subagent`. +2. CI playbook prompt: intent routing (one clarifying question), verb chains, "what changed" reasoning. +3. Shared verb tool module: registry-backed (`04`), reusable by `research`. +4. `start_watch` seam to `06` (wired when `06` is designed). +5. Router registration: `description.md` for main-agent delegation. + +## Tests + +- A CI-/scraping-flavored request routes to `intelligence_agent`; a non-CI request does not. +- "compare X and Y" composes verbs and answers in plain language, persisting nothing. +- "reviews for the top 3 coffee shops near me" chains `maps.search → maps.place → maps.reviews`. +- A follow-up re-invokes the verb and reports new items vs prior in-context outputs. +- "watch this weekly" routes to the ongoing mode, not a one-shot answer. + +## Out of scope + +- Verbs, doors → `04`/`05`. Ongoing/periodic mechanism → `06`. +- Richer multi-step playbooks, proactive suggestions, cross-topic synthesis. +- Frontend CI surfaces → frontend umbrella. + +## Open questions + +- Whether CI one-shot scraping stays in `research` or the `intelligence_agent` owns all CI-flavored calls. +- The `start_watch` handoff shape (settled with `06`). diff --git a/plans/backend/07-upload-pipeline-kb.md b/plans/backend/07-upload-pipeline-kb.md deleted file mode 100644 index 365e1e872..000000000 --- a/plans/backend/07-upload-pipeline-kb.md +++ /dev/null @@ -1,195 +0,0 @@ -# Phase 7 — File upload as a pipeline + KB-save-secondary - -> # ❌ SUPERSEDED (2026-06-30) -> **This entire phase is dropped.** Per the **Architecture correction** in `00-umbrella-plan.md`, there is no "Uploads pipeline" and no upload `PipelineRun`. Uploads simply populate the **input-only KB** through the existing upload routes (an upload creates a `Document` — that *is* KB membership). The umbrella's KB-input invariant ("uploads always land in the KB") is inherent and needs no pipeline wrapper. Retained **for historical context only** — do **not** implement it. **The CI-relevant successor is the "context folder" idea in [`revamp phases 4-7/05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md)**: files uploaded in a CI chat go into the KB as normal (indexed), routed to a dedicated folder, and may feed the materiality judge via KB retrieval. (Canonical Phase 7 is now **Orchestration** — [`revamp/07-orchestration.md`](revamp%20phases%204-7/07-orchestration.md) — unrelated to uploads.) - -> Part of the **CI Pivot MVP**. See `00-umbrella-plan.md` (Phase 7) — the final backend phase. -> Precondition: Phase 5 (`05-pipelines-model.md`) live (`pipelines`/`pipeline_runs` tables, `PipelineRunTrigger.UPLOAD`, `connector_id` nullable) and Phase 6 (`06-pipelines-exec.md`) live (run engine, which **explicitly fails `connector_id IS NULL` runs**). No sibling ahead — this closes the backend umbrella. - -> **Implementation note (post-rename).** Phases 1–2 are **SHIPPED**, so the live code already uses `workspace_id` / `Workspace`. Citations below that still show `search_space_id` / `SearchSpace` are pre-rename — substitute the `workspace_*` equivalent and grep the new name. **New code in this plan uses the canonical `workspace_*` names** (snippets below updated). Locate code by **symbol/grep**, not the absolute line numbers cited here (the rename + Phases 3–6 shift them). - -## Objective - -Close the loop on the umbrella's positioning — *"User file uploads also do a pipeline entry and register a run that files were uploaded by user; only file uploads are always saved in the knowledge base"* — by: - -1. **Uploads-as-pipeline** — give every workspace a singleton **"Uploads" pipeline** (`connector_id = NULL`, `save_to_kb = true`), lazily created, and make each upload request **register a `PipelineRun(trigger=upload)`** so uploads show up in pipeline run history (and in the Phase-6 chat `get_pipeline_runs` tool) exactly like connector fetches. -2. **KB-save-secondary (confirm + close)** — the opt-in `save_to_kb` + `destination_folder_id` machinery for *connector* pipelines already shipped in Phases 5–6; this phase records the **inverse invariant for uploads** (uploads **always** save to KB) and verifies nothing in the pivot accidentally made KB-save the *default* for connector pipelines. - -**The core design fact (drives everything):** uploads are **NOT executed by the Phase-6 run engine**. Phase 6's `execute_pipeline_run` fails any `connector_id IS NULL` run with `"uploads pipeline has no fetch executor"` (`06` §1). The existing upload routes remain the executor (they already create `Document`s + dispatch ETL); Phase 7 only **attaches an audit `PipelineRun`** to that flow and ensures the Uploads pipeline row exists. No `run_pipeline` enqueue, no engine change. - -## Locked decisions (MVP) - -| Concept | Decision | -|---------|----------| -| Uploads pipeline | **One singleton per workspace**, `connector_id = NULL`, `save_to_kb = true`, `schedule_cron = NULL` (never scheduled), `enabled = true`, `name = "Uploads"`. **Lazily get-or-created** on first upload (not at workspace creation — avoids backfilling every existing workspace). | -| Uniqueness / race-safety | A **partial unique index** `ON pipelines (workspace_id) WHERE connector_id IS NULL` makes "one Uploads pipeline per workspace" a DB invariant and lets get-or-create be race-safe via `INSERT … ON CONFLICT DO NOTHING` + re-`SELECT`. (Phase 5 already reserves `connector_id IS NULL` to mean "Uploads", so this index is exactly that semantic.) Small Phase-7 migration. | -| Run lifecycle | **Audit record, created terminal at request time.** The upload route inserts `PipelineRun(trigger=upload, status=succeeded, documents_indexed=, started_at=finished_at=now())`. Per-file ETL outcomes continue to live on `Document.status` (`pending→ready/failed`, already Zero-synced). Rationale below. | -| Why not track per-file ETL into the run | The `fileupload` path fans out **N independent per-file Celery tasks** (`task_dispatcher.py:44`) with **no batch barrier**, and accurate "which docs belong to this run" needs the **`documents.pipeline_run_id` provenance column that Phase 5 deferred** (`05` §3 "Out of scope (provenance)"). Building a barrier is out of MVP scope; the run is an *upload event* record, matching the umbrella's "register a run that files were uploaded". | -| Billing | **No crawl billing for uploads** — uploads aren't crawls. `charged_micros` / `crawls_*` stay `NULL` on upload runs. Existing ETL/embedding credit behaviour (if any) is unchanged and orthogonal (`03c` only governs `web_crawl`). | -| Folder placement | The Uploads pipeline's `destination_folder_id` stays **NULL** — it is a singleton and cannot encode per-upload folders. Folder placement remains owned by the **existing upload code** (`fileupload` → root; `folder-upload` → its created/declared folder). The run/pipeline is **descriptive**, not the folder driver (unlike connector pipelines, where the engine reads `destination_folder_id`). | -| Surfaces wired | `POST /documents/fileupload` and `POST /documents/folder-upload` (both create `Document`s). The desktop precheck route `POST /documents/folder-mtime-check` (`documents_routes.py:1533`) creates nothing and is **not** wired; `folder-unlink` / `folder-sync-finalize` are delete/sync lifecycle, also out. | -| Connector-pipeline KB default | **Unchanged from Phase 5:** `save_to_kb` defaults `False` (`05` §2 `server_default="false"`). This phase only asserts uploads set it `true`; it does **not** flip the connector default. | - -## Current state (cited) - -### The upload routes (the executors we attach to) - -- **`POST /documents/fileupload`** — `documents_routes.py:126-332`. Permission `DOCUMENTS_CREATE` (`:163-169`). Phase 1: create `Document(search_space_id, document_type=FILE, status=pending, created_by_id, …)` per file, dedup by `unique_identifier_hash` (`:217-266`), `commit` (`:278`). Phase 1.5: `store_document_file(...)` per doc (`:289-297`). Phase 2: `dispatcher.dispatch_file_processing(document_id=…)` per file (`:307-316`). Returns `document_ids` / `skipped_duplicates` (`:318-325`). **Creates no folder (root), no run record today.** `files_to_process` (`:213`) = the accepted (newly-queued) set; `skipped_duplicates` (`:214`) = already-`READY` dupes. -- **`POST /documents/folder-upload`** — `documents_routes.py:1598-1748`. Permission `DOCUMENTS_CREATE` (`:1623-1629`). Creates/reuses a root `Folder` (`:1673-1704`), writes temps, then dispatches **one** batch task `index_uploaded_folder_files_task.delay(...)` (`document_tasks.py:1403`) which has its own notification + heartbeat (`:1446-1457`) — i.e. a **real completion barrier** exists on this path (unlike `fileupload`). Returns `root_folder_id` + `file_count` (`:1743-1748`). -- **Dispatcher** — `task_dispatcher.py:12-52`: `dispatch_file_processing(...)` → `process_file_upload_with_document_task.delay(document_id=…)`, **one task per file**, fire-and-forget (no barrier). - -### Phase 5/6 hand-offs this phase consumes - -- `Pipeline.connector_id` **nullable**, `NULL = "non-connector pipeline (the Phase-7 Uploads pipeline)"` (`05` §"Pipeline → connector"; model `05` §2 `:105-106`). `PipelineRunTrigger.UPLOAD` already exists (`05` §1). `PipelineRun` has `trigger`, `status`, `documents_indexed`, `started_at`, `finished_at`, `charged_micros` (nullable) (`05` §3). -- **Phase 6 engine refuses uploads**: `if pipeline.connector_id is None: return fail(run, "uploads pipeline has no fetch executor")` (`06` §1) — confirms uploads must be recorded by the route, **not** enqueued to `run_pipeline`. -- **Chat tool** `get_pipeline_runs` (`06` §7) selects `PipelineRun` joined to `Pipeline` by `workspace_id` — so upload runs surface to the agent for free once they exist. -- **De-dup guard** (`06` §6) only touches **connector-backed** pipelines (`when connector_id is set`); the Uploads pipeline (`connector_id NULL`) is untouched by it. The **scheduler** (`06` §5 `_claim_due`) ignores the Uploads pipeline **two ways** — `schedule_cron IS NOT NULL` (NULL cron) **and** `connector_id IS NOT NULL` (the defense-in-depth filter `06` added at Phase 7's request). Both confirmed safe. - -### Document model - -- `Document` — `db.py:1348`: `document_type` (`:1352`, `DocumentType.FILE`), `unique_identifier_hash` (`:1365`), `folder_id` **nullable** (`:1389`, root/unfiled), `created_by_id` (`:1397`), plus `search_space_id`/`status`. KB membership = a `Document` row exists; uploads always create one ⇒ uploads are always "in the KB" by construction. - -## Target design - -### 1. `get_or_create_uploads_pipeline(session, workspace_id, created_by_id)` — `app/pipelines/uploads.py` - -```python -async def get_or_create_uploads_pipeline(session, *, workspace_id: int, created_by_id) -> Pipeline: - # fast path - existing = (await session.execute( - select(Pipeline).where(Pipeline.workspace_id == workspace_id, - Pipeline.connector_id.is_(None)))).scalar_one_or_none() - if existing: - return existing - # race-safe create (partial unique index ux_pipelines_uploads_per_workspace) - stmt = (pg_insert(Pipeline.__table__) - .values(workspace_id=workspace_id, connector_id=None, name="Uploads", - save_to_kb=True, enabled=True, schedule_cron=None, - created_by_id=created_by_id, config={}) - .on_conflict_do_nothing(index_elements=[Pipeline.workspace_id], - index_where=Pipeline.connector_id.is_(None))) - await session.execute(stmt) - await session.flush() - return (await session.execute( - select(Pipeline).where(Pipeline.workspace_id == workspace_id, - Pipeline.connector_id.is_(None)))).scalar_one() -``` - -- Lazy (first upload), idempotent, race-safe. `created_by_id` = the uploading user (becomes the pipeline's creator metadata; `SET NULL` on member deletion per Phase 5). -- Lives in the `app/pipelines/` package (created in Phase 6) so the upload route imports it without a route↔route dependency. -- **Implementation gotcha — core insert bypasses ORM defaults.** `Pipeline.updated_at`/`created_at` use **Python-side** `default=`/`onupdate=` callables (`05` §2 `:114-115`), which a core `pg_insert` does **not** fire. The `.values(...)` must set `created_at`/`updated_at` explicitly (e.g. `now`) — or implement get-or-create as **ORM-insert + `flush` inside a `session.begin_nested()` SAVEPOINT, catching `IntegrityError` → rollback savepoint → re-`SELECT`** (avoids the core/ORM default mismatch entirely; the SAVEPOINT keeps the route's outer transaction intact). Either is fine; the SAVEPOINT form is closer to the ORM conventions used elsewhere. - -### 2. Register an upload run (shared helper) — `app/pipelines/uploads.py` - -```python -async def record_upload_run(session, *, workspace_id, created_by_id, documents_indexed: int) -> PipelineRun: - pipeline = await get_or_create_uploads_pipeline(session, workspace_id=workspace_id, - created_by_id=created_by_id) - now = datetime.now(UTC) - run = PipelineRun(pipeline_id=pipeline.id, trigger=PipelineRunTrigger.UPLOAD, - status=PipelineRunStatus.SUCCEEDED, - documents_indexed=documents_indexed, started_at=now, finished_at=now) - session.add(run) - return run # caller commits (folds into the route's existing commit) -``` - -- **Terminal-at-creation** (locked decision): records the upload event with the accepted file count. `crawls_*` / `charged_micros` left NULL (not a crawl). -- Caller does **not** await any Celery task — the run is purely the route's own DB write, folded into the route's existing transaction (no extra round-trips, no new failure mode that could block the upload). - -### 3. Wire `POST /documents/fileupload` - -After Phase 2 dispatch (`documents_routes.py:307-316`), before building the response, add: - -```python -if files_to_process: # only when something was actually queued - await record_upload_run(session, workspace_id=workspace_id, - created_by_id=str(user.id), - documents_indexed=len(files_to_process)) - await session.commit() -``` - -- Count = `len(files_to_process)` (the newly-created/re-queued docs), **excluding** `skipped_duplicates` (already-READY no-ops). An all-duplicate upload records **no** run (nothing ingested). -- **Best-effort, non-blocking — and the try/except is mandatory, not cosmetic.** The route has an **outer `except Exception` → `session.rollback()` → HTTP 500** (`documents_routes.py:328-332`). If `record_upload_run` raised unguarded, it would hit that handler and **500 an upload whose documents are already committed + dispatched**. So the call MUST be wrapped in its **own inner `try/except`** that swallows: on failure `await session.rollback()` (discards only the run insert — documents were committed at `:278`/`:304`) + `logger.warning(...)`, then fall through to the response. A run-record write must **never** fail the upload. (Mirrors the `store_document_file` best-effort pattern at `:298-303`.) - -### 4. Wire `POST /documents/folder-upload` - -Folder-upload dispatches one batch task, so record the run at request time with the file count (`documents_routes.py` before the `return` at `:1743`): - -```python -await record_upload_run(session, workspace_id=workspace_id, - created_by_id=str(user.id), documents_indexed=len(files)) -await session.commit() # best-effort / logged, same as §3 -``` - -- Uses `len(files)` (folder-upload doesn't pre-dedup in the route; dedup happens inside the batch task). -- **Optional richer accounting (deferred):** because this path *has* a barrier (`index_uploaded_folder_files_task`), a later enhancement could create the run `pending` here and flip it `succeeded`/`failed` from inside the task (it already manages a notification lifecycle). MVP keeps it uniform with `fileupload` (terminal-at-request). Flagged, not built. - -### 5. Migration (one small file) — partial unique index - -```sql -CREATE UNIQUE INDEX IF NOT EXISTS ux_pipelines_uploads_per_workspace - ON pipelines (workspace_id) WHERE connector_id IS NULL; -``` - -- Chain after the then-current head (verify `alembic heads`). Enforces the singleton + powers `ON CONFLICT`. `downgrade()` drops it. -- **Pre-flight (safety):** if any workspace somehow already has >1 `connector_id IS NULL` pipeline (shouldn't, pre-Phase-7), the `CREATE UNIQUE INDEX` fails — the migration docstring notes a dedup step (keep the oldest, delete extras) just in case. Expectation: none exist (uploads pipelines are introduced *here*). - -### 6. KB-save-secondary verification (no new code, a guard) - -- Uploads pipeline hardcodes `save_to_kb=True` (§1) — uploads always land in the KB (a `Document` row), matching the umbrella invariant. -- Confirm Phase 5's connector-pipeline default is still `save_to_kb=False` (`05` §2) — i.e. KB-save is **opt-in for connectors, mandatory for uploads**. Add a test asserting both defaults so a future change can't silently make connector KB-save the default. - -### 7. Guard the generic Phase-5 CRUD against the system Uploads pipeline (the gap) - -The Uploads pipeline is a **system-managed** row (`connector_id IS NULL`, auto-created by §1). Phase 5's generic `/pipelines` CRUD doesn't know it's special, which opens three footguns — Phase 7 closes them: - -| Phase-5 endpoint | Problem if left generic | Phase-7 guard | -|------------------|-------------------------|---------------| -| `POST /pipelines` | Phase 5 **allows `connector_id=None`** (`05` §7 create + test). A user could hand-create a rogue/duplicate non-connector pipeline. | **Reject `connector_id is None` with 4xx** — non-connector pipelines are system-managed (auto-created on upload), not user-created. This **supersedes** Phase 5's "`connector_id=None` → allowed" create test (update it). The partial unique index (§5) is the DB backstop. | -| `POST /pipelines/{id}/run` | Manual-running the Uploads pipeline enqueues `run_pipeline`, which the Phase-6 engine **fails** (`"uploads pipeline has no fetch executor"`) → a confusing **FAILED** manual run every click. | **Reject `connector_id IS NULL` with 4xx** ("uploads are triggered by uploading files, not a manual run") — before creating the run row / enqueuing. | -| `PUT /pipelines/{id}` | Setting `schedule_cron` on the Uploads pipeline makes the scheduler eligible-by-cron → **perpetually-failing scheduled runs** (engine refuses null-connector). Editing `save_to_kb`/`destination_folder_id` is silently **ineffective** (uploads are route-driven, not engine-driven). | **Reject schedule edits (`schedule_cron`/`schedule_timezone`) on `connector_id IS NULL` pipelines with 4xx.** (`connector_id` is already immutable per Phase 5.) Optionally reject all PUTs to the Uploads pipeline; minimally block the scheduling fields. Belt-and-suspenders: Phase 6's `_claim_due` also filters `connector_id IS NOT NULL` (`06` §5). | -| `DELETE /pipelines/{id}` | Deleting the Uploads pipeline cascades its run history. | **Allowed** (MVP): it is **lazily re-created** on the next upload (§1); only past upload-run history is lost. Note in UI copy later; an optional hard block is a small follow-up. | - -Net: after Phase 7, the only non-connector pipeline that can exist is the **system Uploads pipeline**, it is never user-created / manually-run / scheduled, and the scheduler ignores it two ways (NULL cron *and* the `connector_id IS NOT NULL` filter). - -## Work items - -1. **`app/pipelines/uploads.py`**: `get_or_create_uploads_pipeline` (race-safe via `ON CONFLICT`) + `record_upload_run` (terminal upload run). -2. **Migration**: partial unique index `ux_pipelines_uploads_per_workspace ON pipelines(workspace_id) WHERE connector_id IS NULL`; docstring dedup pre-flight note; `downgrade` drops it. -3. **Wire `fileupload`** (`documents_routes.py:126-332`): call `record_upload_run(documents_indexed=len(files_to_process))` after dispatch, in an **inner** best-effort try/except (must not reach the route's outer 500 handler), only when `files_to_process` non-empty. -4. **Wire `folder-upload`** (`documents_routes.py:1598-1748`): call `record_upload_run(documents_indexed=len(files))` before the response, same inner best-effort guard. -5. **Guard the generic CRUD** (§7): `POST /pipelines` rejects `connector_id is None`; `/run` rejects `connector_id IS NULL`; `PUT` rejects schedule edits on `connector_id IS NULL`. Update Phase 5's "`connector_id=None` → allowed" create test accordingly. (Phase 6's `_claim_due` `connector_id IS NOT NULL` filter is the scheduler backstop.) -6. **Tests** (below). - -## Tests - -- **First upload creates the pipeline**: `fileupload` into a fresh workspace → exactly one `connector_id IS NULL` pipeline named "Uploads" with `save_to_kb=True`; a `PipelineRun(trigger=upload, status=succeeded, documents_indexed=N)`. -- **Subsequent uploads reuse it**: a second `fileupload` → still one Uploads pipeline, two upload runs. Concurrency: two simultaneous first-uploads (race) → still exactly one pipeline (partial-unique + `ON CONFLICT`), two runs. -- **Duplicate-only upload**: re-uploading already-READY files (`skipped_duplicates == len(files)`, `files_to_process == []`) → **no** run recorded. -- **folder-upload**: records one upload run with `documents_indexed=len(files)`; the existing batch task/folder behaviour is unchanged. -- **No crawl billing**: upload runs have `charged_micros IS NULL`, `crawls_* IS NULL`; no `web_crawl` `TokenUsage` row is written. -- **Engine still refuses uploads**: enqueuing `run_pipeline` for an Uploads pipeline's run (or a NULL-connector pipeline) → Phase-6 engine fails it cleanly (regression guard for `06` §1). Confirms uploads are never engine-executed. -- **Scheduler ignores uploads (two ways)**: the Phase-6 tick never selects the Uploads pipeline — both because `schedule_cron IS NULL` and because `_claim_due` filters `connector_id IS NOT NULL`. Add a case: even with a cron forced onto a NULL-connector row (direct DB), `_claim_due` still skips it. -- **CRUD guards (§7)**: `POST /pipelines` with `connector_id=None` → 4xx; `POST /pipelines/{uploads_id}/run` → 4xx (no FAILED run created); `PUT` setting `schedule_cron` on the Uploads pipeline → 4xx; `DELETE` of the Uploads pipeline → allowed, and the next upload re-creates it. -- **Chat tool surfaces uploads**: `get_pipeline_runs` returns the upload runs for the workspace (trigger=upload), scoped correctly. -- **KB defaults guard**: Uploads pipeline `save_to_kb=True`; a connector pipeline created without `save_to_kb` defaults `False` (§6). -- **Best-effort isolation**: a forced failure inside `record_upload_run` does not fail the upload request (documents still created + dispatched, 2xx returned). - -## Risks / trade-offs - -- **Run is an upload *event*, not per-file ETL truth.** `status=succeeded` + `documents_indexed=` is stamped at request time; individual files may still fail ETL afterward (visible on `Document.status`, not reflected back onto the run). Accepted for MVP — accurate per-file roll-up needs the deferred `documents.pipeline_run_id` provenance column. Documented; the folder-upload barrier is the natural place to add real status later (§4). -- **Singleton Uploads pipeline ⇒ no per-upload folder on the pipeline.** `destination_folder_id` stays NULL; folder placement stays in the upload code. Fine because the upload run is descriptive, not engine-driven. If product later wants "uploads to folder X" as a first-class pipeline, that's a connector-less pipeline variant (additive). -- **Partial-unique index assumes `connector_id IS NULL ⇔ Uploads`.** True for MVP (Phase 5 reserved NULL for Uploads, and Phase 6 fails NULL-connector runs). If a *second* kind of connector-less pipeline is ever introduced, replace the index predicate with an explicit `kind`/discriminator column (additive migration) — flagged so the NULL-overload doesn't silently block it. -- **Best-effort run write can be skipped on error.** If `record_upload_run` raises (e.g. transient DB error) the upload still succeeds but no run is logged — an audit gap, never a data-loss or user-facing failure. Acceptable: the `Document` rows (the actual KB content) are the source of truth; the run is supplementary history. -- **No backfill for historical uploads.** Pre-Phase-7 uploads have no upload runs (pipeline is created lazily on the next upload). Accepted — run history starts accumulating from rollout; no migration backfills synthetic runs. -- **Upload run doesn't capture the uploader.** `PipelineRun` has no user column (Phase 5), so the run records *what/when/how-many*, not *who* — the per-document uploader is still on `Document.created_by_id`, and the pipeline's `created_by` is just whoever triggered the first-ever upload. If per-run attribution is wanted, add a nullable `triggered_by_id` to `pipeline_runs` (additive Phase-5 model change); deferred. -- **`enabled=True` on a never-scheduled pipeline.** The Uploads pipeline is `enabled=True` with NULL cron — harmless (scheduler ignores it via NULL cron *and* the `connector_id IS NOT NULL` filter); `enabled` only ever gates scheduling, which uploads never use. Left `True` so a future UI can show it as an active pipeline. -- **CRUD-guard coupling to Phase 5.** Phase 7 tightens Phase 5's create contract (`connector_id=None` now rejected) and adds `/run` + `PUT` guards. This is a documented evolution (Phase 7 ships after 5/6); the Phase-5 create test is updated here, not left contradictory. - -## Out of scope (hand-offs) - -- **`documents.pipeline_run_id` provenance** (per-file → run linkage, accurate ETL roll-up, folder-upload barrier-driven status) → post-MVP (touches `documents`). -- **Engine execution of uploads** — intentionally never; uploads are route-recorded (`06` fails NULL-connector runs by design). -- **Migrating connector periodic indexing into pipelines** — still COEXIST (Phase 5 decision); not revisited here. -- **Public pay-as-you-go API / MCP KB server** → post-MVP (umbrella "Deferred"). -- **Frontend** — surfacing the Uploads pipeline + upload runs in the Pipelines UI, "uploads always saved to KB" copy → frontend umbrella. diff --git a/plans/backend/revamp phases 4-7/00-overview.md b/plans/backend/revamp phases 4-7/00-overview.md deleted file mode 100644 index b2c2578fd..000000000 --- a/plans/backend/revamp phases 4-7/00-overview.md +++ /dev/null @@ -1,125 +0,0 @@ -# Phase 4 → end revamp — Overview & phase index (CI pivot · WIP) - -> **What this is.** The CI pivot re-architects the old "pipeline" Phases `04`–`07` into small, -> single-responsibility domains, now packaged as **4 phases (04–07)** that mirror the slots they replace. -> This file is the map + the reconciliation + the subplan index. **End-to-end flow diagrams (stateless & -> stateful paths) live in `00b-pipeline-diagrams.md`.** -> **Scope guardrail:** Phases **1–3 are SHIPPED/FIXED** (rename/DB, proxy/captcha/stealth, crawl billing). -> The revamp is **Phase 4 → end only**. We do **not** touch 1–3. -> -> **Citation verification — 2026-06-30 (principal-engineer pass).** All load-bearing code citations in this -> revamp were re-checked against `surfsense_backend`. Confirmed accurate: the schedule selector -> (`selector.py` — `FOR UPDATE SKIP LOCKED`, `next_fire_at`, self-heal, `catchup=False`, `croniter`), the -> `AutomationRun` model/fields, `format_to_structured_document(exclude_metadata=True)` content-hashing, the -> MCP routing gap (`constants.py::CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS`), the connector enum, and the -> `MANUAL` trigger placeholder. **Corrected during this pass:** (a) automations have **no** delivery/ -> notification path and `automation_runs.output` is never written — CI alerts wire to `app/notifications/` -> (`06`/`05b`); (b) the automation PENDING-gate is **not atomic** — the per-Tracker lock is the real -> concurrency guard (`06`/`05b`); (c) folder upload uses `root_folder_id` (not `destination_folder`) and KB -> folder scoping goes through `referenced_document_ids → SearchScope.document_ids` (`05b`); (d) the billable -> predicate is `SUCCESS and outcome.result`, to be single-sourced in the executor (`04a`). - -## The two products - -``` -PRODUCT A — stateless utility Phase 04 (Capabilities + Access) - call a verb → get data → bill. Nothing persists. - -PRODUCT B — decision-grounded CI Phase 05 (Intelligence + Timeline) (+ 06 Triggers, 07 Orchestration) - a Tracker accumulates structured signal over time. The Timeline is the moat. -``` - -## The phases (this revamp) - -| Phase | Subplan(s) | Domain(s) | One line | Ships | -|-------|-----------|-----------|----------|-------| -| **04 — Capabilities & Access** | `04a-capabilities.md` · `04b-access.md` | ① + ② | typed verbs over Acquisition + the chat/REST/MCP doors | **Product A** (revenue day one) | -| **05 — Intelligence & Timeline** | `05a-timeline.md` · `05b-intelligence.md` | ④ + ③ | the Tracker, locked schema, hot loop + the 3-store delta moat | **Product B engine** | -| **06 — Triggers** | `06-triggers.md` | ⑤ | the pluggable refresh clock; recurrence+delivery = optional CI action on automations | recurrence + alerts | -| **07 — Orchestration** | `07-orchestration.md` | ⑥ | the `intelligence_agent` CI-expert subagent (intent routing, verb chains, Tracker crafting) | the human-facing CI experience | - -> **Build order = phase order.** `04a → 04b` (unblocks everything; ships Product A) → `05a → 05b` -> (state before the loop that writes it) → `06` (drive it) → `07` (front it). `04a` first because every -> other phase calls into the capability registry. - -## Domain ↔ phase map - -``` - FIXED (Phases 1–3) OUR SCOPE (Phase 04 → 07) - ┌─────────────────────────────────┐ ┌──────────────────────────────────────────────┐ - │ Acquisition │ │ 04a Capabilities typed verbs over Acquisition │ - │ proprietary/web_crawler │◄──┤ 04b Access chat · REST · MCP doors │ - │ CrawlOutcome · billing (03c) │ │ 05a Timeline 3-store delta state (moat) │ - │ │ │ 05b Intelligence Tracker · schema · hot loop │ - │ │ │ 06 Triggers pluggable refresh clock │ - │ │ │ 07 Orchestration CI-expert subagent + tools │ - └─────────────────────────────────┘ └──────────────────────────────────────────────┘ - data engine (untouched) stateless 04 → stateful 05 , driven by 06 , fronted by 07 -``` - -## Reconciliation with the old plans - -| Old plan | Fate | Where it goes | -|----------|------|---------------| -| `04a-connector-category.md` | **demoted to hygiene** | the genuine MCP-routing fix survives in `04b` (consume-user-MCP); the taxonomy work is not core | -| `04b-source-discovery.md` | **absorbed** | becomes the `web.discover` capability in `04a` | -| `05-pipelines-model.md` | **dissolved** | the "pipeline" concept → the `Tracker` (`05b`) + Timeline tables (`05a`); no `pipeline`/`pipeline_runs` | -| `06-pipelines-exec.md` | **dissolved** | execution → the hot loop (`05b`); scheduling/runs/delivery → **reuse automations** via a CI action (`06`) — its selector + `AutomationRun`, **not** a rebuilt cron | -| `07-upload-pipeline-kb.md` | **dropped (crawl→KB + uploads-as-pipeline audit)** | "don't index *crawled* data" holds. **User file-upload-to-KB remains a pre-existing, untouched feature.** For CI, uploads are routed to a dedicated **folder** and may feed the judge (`05b`) | - -**Net:** the old `05/06/07` pipeline+KB stack is replaced by the new `04`–`07`. KB indexing of *crawled* -data is out; *user uploads* still work (and gain a CI context-folder role). - -## Confirmed decisions (locked across phases) - -- **Natural language is the only human-facing surface.** Users never name verbs; the chat agent - understands intent, composes verbs, answers in plain language. An **intent router** classifies one-shot - (A) vs standing-concern (B) from the wording (one clarifying question if ambiguous). Raw verbs live only - on the REST/MCP dev doors. -- MVP surface = the **general web crawler + search** (`web.scrape` + `web.discover`), exposed chat + REST (MCP fast-follow). **Platform-specific scrapers are a family of individual endpoints added incrementally — none (incl. Google Maps) is committed for MVP;** each is just another verb → agent tool + dev API-key REST endpoint. -- Verbs namespaced per platform (`web.*`, and `.*` per scraper, e.g. `maps.*` as an example); `web.scrape` takes a URL **array** (inline-or-job). -- **Don't index** crawled data into the KB; the Timeline (deltas) is the only persisted CI state. -- CI is **decoupled from automations** (automations = one optional Trigger adapter). -- Intelligence: **agent-designed schema** (sample-grounded, human-locked, versioned) in MVP; single entity - per Tracker; materiality = code thresholds (numeric/clear) + agent on ambiguous; content-hash pre-check. -- Orchestration is a first-class deliverable: a **net-new CI-expert subagent** (`intelligence_agent`) on - the existing chat runtime, with registry-backed verb tools + Tracker/Timeline tools. -- **Recurrence + alert delivery = a CI *action* on the existing automations** (not a new scheduler, not a - new shape): reuses the hardened schedule selector, `AutomationRun` (audit + idempotency), and delivery. - CI core still runs with **zero** automations dependency (manual/agent/cron). -- **No new run table:** refresh audit/idempotency ride `AutomationRun` or the chat job record; billing - idempotency is per-capability-call + the content-hash pre-check. Only the Timeline is new state. -- **Billing is delegated to the billing service:** verbs declare a `billing_unit`; `03c` is the first - provider; `maps.*` / `web.discover` register their own units there. -- The **Google Maps actor is net-new** (not in shipped Phases 1–3) and is designed/built as a **separate - effort**; `maps.*` verbs are contracts against it and don't block this design. -- **CI context files:** files uploaded in a CI chat go into the **KB as normal** (indexed), routed to a - dedicated `Folder`, and may feed the materiality judge via KB retrieval. ("Don't index" applies only to - *crawled* data.) -- **Names:** primitive = `Tracker`, subagent = `intelligence_agent` (Intelligence Agent). -- License: new domains **Apache-2**; the moat stays in proprietary Acquisition + the Maps extractor. - -## Cross-phase open forks (need your call) - -1. Schema **review-&-lock** UX before frontend exists — pure-chat confirmation for MVP? (`05b`) -2. Timeline ORM home — `app/db.py` vs dedicated `app/timeline/`. (`05a`) -3. Recurrence = CI **action** on automations (default) vs a thin CI automation **shape** (fallback). (`06`) - -*(Resolved: the "built-in scheduler tick" fork — reuse the automations schedule selector via a CI action -instead of building a tick. See `06`.)* - -> **Per-phase implementation-time questions** (refresh failure path, first-run baseline, tracker lifecycle, -> mid-run balance, concurrency lock, delivery payload shape) are intentionally **not** resolved here — they -> don't change any boundary, contract, or data shape, and are settled when each phase is built. - -## Subplan index (revamp) - -| Phase | Subplan file | Status | -|-------|--------------|--------| -| — | `00b-pipeline-diagrams.md` | end-to-end flow diagrams | -| 04 | `04a-capabilities.md` | drafted | -| 04 | `04b-access.md` | drafted | -| 05 | `05a-timeline.md` | drafted | -| 05 | `05b-intelligence.md` | drafted | -| 06 | `06-triggers.md` | drafted | -| 07 | `07-orchestration.md` | drafted | diff --git a/plans/backend/revamp phases 4-7/00b-pipeline-diagrams.md b/plans/backend/revamp phases 4-7/00b-pipeline-diagrams.md deleted file mode 100644 index 6e6e011e0..000000000 --- a/plans/backend/revamp phases 4-7/00b-pipeline-diagrams.md +++ /dev/null @@ -1,202 +0,0 @@ -# Pipeline diagrams — end-to-end (stateless & stateful paths) - -> Visual companion to `00-overview.md`. Shows the whole CI pipeline across Phases 04–07, with the -> **stateless** (Product A) and **stateful** (Product B) paths called out explicitly. Phase refs: -> `04a` Capabilities · `04b` Access · `05a` Timeline · `05b` Intelligence · `06` Triggers · `07` Orchestration. - -## 1. Domain map — the two products - -```mermaid -flowchart LR - subgraph FIXED["FIXED · Phases 1-3 (shipped, untouched)"] - ACQ["Acquisition
proprietary/web_crawler
CrawlOutcome"] - MET["Metering · 03c
WebCrawlCreditService"] - ID["Identity / Tenancy
+ API keys"] - RT["Chat runtime
deepagents · deliverable_wait"] - end - - subgraph SCOPE["OUR SCOPE · Phase 04 -> 07"] - CAP["04a Capabilities
typed verbs + registry + job store"] - ACC["04b Access
chat · REST · MCP doors"] - INT["05b Intelligence
Tracker · schema · hot loop"] - TL["05a Timeline
3-store delta state (moat)"] - TRG["06 Triggers
refresh clock"] - ORC["07 Orchestration
intelligence_agent subagent"] - end - - ACQ --> CAP - MET --> CAP - CAP --> ACC - ID --> ACC - RT --> ORC - ACC --> ORC - CAP --> INT - INT --> TL - TRG --> INT - ORC --> CAP - ORC --> INT - ORC --> TL - - classDef a fill:#22314f,stroke:#5b7fbf,color:#e6edf7; - classDef b fill:#1f3a2e,stroke:#4f9d76,color:#e6f7ee; - class CAP,ACC,INT,TL,TRG,ORC a; - class ACQ,MET,ID,RT b; -``` - -- **Product A (stateless):** `04a Capabilities + 04b Access` — call a verb, get data, bill, nothing persists. -- **Product B (stateful):** `05b Intelligence + 05a Timeline` (driven by `06`, fronted by `07`) — a Tracker - accumulates structured signal; the Timeline is the moat. - -## 2. The fork — intent router (the only human-facing decision) - -```mermaid -flowchart TD - U(["User speaks in natural language"]) --> R{"Intent router
(in intelligence_agent prompt, 07)"} - R -->|"compare / find / what is / pull / right now"| A["ONE-SHOT
Product A · stateless"] - R -->|"watch / track / notify me / every week / over time"| B["STANDING CONCERN
Product B · stateful"] - R -->|ambiguous| Q["Ask ONE question:
'just once, or keep watching?'"] - Q -->|once| A - Q -->|keep watching| B - A --> P1["section 3 · Stateless path"] - B --> P2["section 4 · Stateful path"] -``` - -Raw verbs are exposed only on REST/MCP (devs/external agents). Humans never name a verb. - -## 3. Stateless path (Product A) — end to end - -```mermaid -sequenceDiagram - autonumber - participant U as User / Dev - participant D as Door 04b (chat · REST · MCP) - participant REG as Registry 04a - participant EX as Verb executor 04a - participant ACQ as Acquisition / Maps actor - participant BILL as Billing service (03c) - participant JOB as Job record (+Celery) - - U->>D: intent (chat) OR raw verb call (REST/MCP) - D->>D: validate input_schema · authn/authz · meter-gate - D->>REG: resolve verb - REG->>EX: same executor (all doors) - alt fast / small input -> inline - EX->>ACQ: crawl_url / search / place - ACQ-->>EX: data - EX->>BILL: charge billing_unit per success - EX-->>D: envelope {status: completed, results} - D-->>U: plain-language answer (chat) / JSON (REST) - else slow / large input -> job - EX->>JOB: dispatch, return {status: pending, job_id} - D-->>U: chat: deliverable_wait polls status
REST/MCP: GET /v1/jobs/{id} - JOB->>ACQ: work loop - JOB->>BILL: charge per success - JOB-->>D: status -> completed (+ results) - D-->>U: results inline / tracked card - end - Note over U,JOB: Nothing persists beyond chat history. No Tracker, no Timeline row. -``` - -## 4. Stateful path (Product B) — setup, then refresh, then read - -### 4a. Setup once — craft & lock the Tracker (05b) - -```mermaid -flowchart TD - S0(["Standing concern detected"]) --> S1["Bind capability + input
(maps.place X / web.scrape Y)"] - S1 --> S2["Sample fetch · ONE real verb call
(billed)"] - S2 --> S3["Agent drafts definition from decision + sample:
field_schema · materiality · identity_rule · notable_signals"] - S3 --> S4{"Human reviews in chat"} - S4 -->|add field X / tweak| S3 - S4 -->|looks good| S5["LOCK · versioned snapshot
status: draft -> active"] - S5 --> S6[("Tracker persisted
+ tracked_entities row (05a)")] -``` - -### 4b. The hot loop — refresh(tracker) (05b → writes 05a) - -```mermaid -flowchart TD - T(["Trigger fires refresh(tracker) · 06"]) --> C1["1 · Crawl: call bound capability (04a)"] - C1 --> C2{"2 · Content-hash
== stored hash?"} - C2 -->|identical| STOP["stamp last_checked_at · STOP
(zero LLM cost)"] - C2 -->|changed / first run| C3["3 · Fill: agent conforms raw -> locked field_schema
(extras -> notable_signals)"] - C3 --> C4["4 · Diff (code): new record vs entity_current_state -> raw deltas"] - C4 --> C5{"5 · Judge each delta"} - C5 -->|numeric / clear rule| C5a["code decides material/noise
(free, reproducible)"] - C5 -->|ambiguous / notable_signals| C5b["agent decides
(1 LLM call; may read CI context folder)"] - C5a --> C6{"material?"} - C5b --> C6 - C6 -->|yes| W["6 · Append entity_changes row
+ overwrite entity_current_state (05a)"] - C6 -->|no| N["only bump last_checked_at
(no change -> no row)"] - W --> DONE([done]) - N --> DONE - STOP --> DONE -``` - -### 4c. Read / answer — straight from the Timeline (05a) - -```mermaid -flowchart LR - Q(["'Is competitor X pulling ahead?'"]) --> A["intelligence_agent · query_timeline(tracker_id)"] - A --> TL[("entity_changes + entity_current_state")] - TL --> ANS["Answer from stored deltas
(NOT re-derived from chat history)"] -``` - -## 5. Triggers — who calls refresh(tracker) (06) - -```mermaid -flowchart LR - M["Manual
'refresh now' (chat/REST)"] --> RF["refresh(tracker)
headless unit of work · 05b"] - AG["Agent
calls refresh as a tool"] --> RF - XC["External cron
POST /v1/trackers/{id}/refresh"] --> RF - CA["CI automation ACTION
(existing automations)"] --> RF - RF --> TLW[("writes Timeline · 05a")] - - CA -. "schedule selector" .-> SEL["hardened cron selector"] - CA -. "run record + idempotency" .-> AR["AutomationRun (PENDING-gate)"] - CA -. "delivery" .-> DEL["alert / output to user"] - - classDef opt fill:#3a2f1f,stroke:#bf975b,color:#f7efe6; - class CA,SEL,AR,DEL opt; -``` - -CI **core** (`refresh` + Timeline) has **zero** automations dependency — manual/agent/cron all work -standalone. The CI **action** is the *optional* adapter that buys recurrence + idempotency + delivery. - -## 6. The persisted state — Timeline data model (05a) - -```mermaid -erDiagram - TRACKER ||--|| TRACKED_ENTITY : "MVP 1:1 (multi-ready)" - TRACKED_ENTITY ||--|| ENTITY_CURRENT_STATE : "latest" - TRACKED_ENTITY ||--o{ ENTITY_CHANGES : "append-only log" - - TRACKER { - uuid id - text decision - json capability_binding - json definition_locked "field_schema·identity_rule·materiality (versioned)" - text status "draft|active" - } - TRACKED_ENTITY { - uuid id - uuid workspace_id - text entity_key "from identity_rule, unique per tracker" - ts first_seen_at - } - ENTITY_CURRENT_STATE { - uuid entity_id FK - json fields "conforms to field_schema" - text content_hash "powers step-2 pre-check" - ts last_checked_at - } - ENTITY_CHANGES { - uuid id - ts captured_at - json delta "{field:{from,to}}" - text materiality "material|notable" - text decided_by "code|agent (provenance)" - text source_ref - text note "why material" - } -``` diff --git a/plans/backend/revamp phases 4-7/04a-capabilities.md b/plans/backend/revamp phases 4-7/04a-capabilities.md deleted file mode 100644 index 44e3d3928..000000000 --- a/plans/backend/revamp phases 4-7/04a-capabilities.md +++ /dev/null @@ -1,198 +0,0 @@ -# Phase 4a — Capabilities (typed verbs over Acquisition) - -> Part of **Phase 4 — Capabilities & Access** (the CI-pivot revamp of the old "pipeline" Phases 4–7). -> Sibling: `04b-access.md` (the doors that expose these verbs). **Build first** — everything above -> calls into it. -> **Depends on** Phases 3a/3b/3c (SHIPPED): the `WebCrawlerConnector.crawl_url -> CrawlOutcome` -> contract, the proxy provider, and the `WebCrawlCreditService` billing seam. -> **Scope guardrail:** Phases 1–3 are SHIPPED/FIXED. This domain *builds on* them and never modifies -> them. Locate foundation code by **symbol/grep**, not the line numbers cited. - -## Objective - -Turn the fixed Acquisition engine (+ a new Google Maps actor) into a **small set of typed, callable -verbs** that every door (chat / REST / MCP) and the Intelligence hot loop (`05b`) consume -**identically**. This is the core pivot of the old Phase 4: **replace "connector that ingests into the -KB" with "capability that returns data."** A capability is a stateless function you *call and get data -back from* — no `SearchSourceConnector` row to configure, no KB write, no schedule attached. - -## Current state (cited) - -- `WebCrawlerConnector.crawl_url(url) -> CrawlOutcome` — single-URL fetch returning `{content, - metadata, crawler_type}` on `SUCCESS`. `CrawlOutcomeStatus.SUCCESS` documents the billable signal, but - **operationally every caller checks `status == SUCCESS and outcome.result`** (see `scrape_webpage.py` and - `webcrawler_indexer.py`) — the predicate is duplicated, not one helper. **The executor (below) should - single-source it** in a shared `is_billable(outcome)` helper (`app/proprietary/web_crawler/`). -- `WebCrawlCreditService` (`03c`) — per-success metering with the static `billing_enabled()` / - `successes_to_micros()` seam (`app/services/web_crawl_credit_service.py`). -- Proxy / stealth / captcha tiers — all behind `crawl_url`; callers never see the tier (`03b`/`03d`/`03e`). -- Web-search providers (SearXNG/Linkup/Baidu) and the source-discovery core from old `04b` — the - substrate for `web.discover`. -- **No Maps actor exists** — it is net-new, proprietary, and built as a separate effort (see Out of scope). - -## Target design - -### The verb set (MVP) — namespaced, nothing top-level - -Two namespaces: **`web.*`** (generic crawler product) and **`.*`** (per-platform scrapers). -Future platforms slot in as their own namespace (`maps.*`, `linkedin.*`, `amazon.*`, …). - -> **Scope clarification (2026-06-30).** Platform-specific scrapers are a **family of individual scraping -> endpoints**, **not** one committed integration — **no specific platform (incl. Google Maps) is committed -> for MVP**; `maps.*` below is an **illustrative example** of the pattern. Each such scraper is *just -> another capability verb*: adding one lights up (a) an **agent tool** (chat) and (b) a **dev-callable REST -> endpoint behind the platform API key** — same executor, same billing, zero new machinery. **The MVP -> builds only `web.scrape` + `web.discover` executors**; every `.*` row is a **contract stub** -> demonstrating that the registry + generated doors (`04b`) make per-platform endpoints a drop-in. - -| Verb | Input → Output | Mode | Executes over | Bills (03c) | -|------|----------------|------|---------------|-------------| -| `web.scrape(urls[])` | → `[{url, status, content, metadata}]` | inline-or-job | loop `crawl_url` | per success | -| `web.discover(query, top_k)` | → `[{url, title, snippet, provider}]` | inline | search providers (SearXNG/Linkup/Baidu) | per search *(or free — open)* | -| `maps.search(query, location)` | → `[place]` | job | Maps actor *(new, proprietary)* | per place | -| `maps.place(place_id\|url)` | → `place` (structured) | inline | Maps actor *(new, proprietary)* | 1 | -| `maps.reviews(place)` | → `[review]` (paged) | job | Maps actor *(new, proprietary)* | per page | - -### `web.scrape` — one array-friendly verb (no separate "batch") - -Scraping always **welcomes an array of URLs**. There is no `scrape_batch`. One verb, one mental model: -"give me URLs, I give you per-URL content." - -### Execution mode — a property of the **result**, not two verbs - -A capability does **not** split into sync/async variants. Every call returns a **uniform envelope**: - -``` -{ status: "completed" | "pending", job_id?: str, progress?: {done, total}, results?: [...], error?: ... } -``` - -- **inline (fast path):** small/fast input finishes in-request → `completed` with `results` inline → - zero polling. (single page, one place, a search) -- **job (slow path):** large/slow input → `pending` + `job_id` → caller polls a status endpoint until - `completed`. (batch scrape, Maps search/reviews) - -So "sync" is simply *a job that finished instantly*. Same verb, same shape, no branching for the caller. - -- **Threshold** (how many URLs / how heavy before async) is **configurable**. -- A caller may pass `async: true` to **force** a job even for small input (agents that never block). -- **Only infra this requires:** a thin **job record** (`id, status, progress, result_ref`) + the - **existing Celery workers**. *Not* a resurrected `pipeline_runs`. Sync verbs need none of it. - -### The capability registry — single source of truth - -One registry entry per verb: - -``` -Capability { - name # dotted, e.g. "web.scrape", "maps.search" - input_schema # Pydantic - output_schema # Pydantic - mode # can-complete-inline? + job-capable? - executor # the async fn (wraps Acquisition / Maps actor) - billing_unit # how Metering charges this call -} -``` - -The **three doors are generated from the registry** (`04b`), not hand-written three times — chat tool, -REST route, MCP tool — and the **Intelligence** hot loop (`05b`) calls the *same* `executor` directly. -Add a verb once → it lights up on every surface; the I/O contract cannot drift between surfaces. - -### Billing — open to the billing service (not hardcoded per verb) - -A capability only **declares a `billing_unit`** in its registry entry; charging is delegated to the -**billing service**, so adding/repricing a unit is a billing-service concern, not a capability rewrite. -`03c`'s `WebCrawlCreditService` is the **first provider**; new units register with the same service. - -- `web.scrape` → per `SUCCESS` (existing `web_crawl` unit). -- `maps.*` → register a new per-place / per-page unit **with the billing service** (same wallet). -- captcha attempts → existing per-attempt `web_crawl_captcha` unit (`03d`, unchanged). -- `web.discover` → register a per-search unit (or mark free) **with the billing service**. - -The registry says *"this verb bills unit X"*; the **billing service owns what unit X costs.** Verbs -stay pure; pricing stays pluggable. - -> **Charge at the executor, not the turn accumulator (locked 2026-06-30).** The billable charge fires -> **inside the capability executor** (this phase), so *every* caller meters identically — chat, -> **automation/recurring**, REST, MCP, external-cron. This is deliberate: a code review confirmed the -> automation path (`run_agent_task` → `agent_task`) establishes **no chat turn accumulator**, so the -> existing `scrape_webpage` billing (which no-ops when `get_current_accumulator()` is `None`) would let -> **automation-run crawls bill nothing**. The interactive chat **turn accumulator stays only as an -> optional presentation fold** (so a chat turn still shows the crawl line on its bill) — it is **not** the -> charging mechanism. Billing idempotency is **per capability call** (+ the `05b` content-hash pre-check), -> not per run. - -### Platform scrapers (illustrated with a Maps actor — example only, not committed) - -- Each platform scraper is a **standalone endpoint**; the one shown here (a hypothetical Maps actor) is - **Google-Maps-agnostic** — works for *any* place type (restaurants, gyms, hotels, retail, …). -- Returns **typed structured objects** (`{name, rating, review_count, hours, price_level, …}`), not raw - markdown — exactly what the Intelligence/Timeline layer needs to diff reliably. -- The "restaurant" use case is an **Intelligence-domain wedge** (one Tracker), **not** a constraint on - the capability. The capability never knows what decision it serves. - -### Where it lives / license boundary - -- New **Apache-2** package `app/capabilities/` (registry, schemas, executors, the thin job store). -- It **imports from** the proprietary Acquisition engine but never moves into it. -- The **Maps extractor logic is proprietary** (`app/proprietary/...`), consumed by the `maps.*` - executors — same boundary rule Phase 3 set. - -## Work items - -1. **Registry**: `app/capabilities/` package — `Capability` dataclass + a registry that other domains import. -2. **`web.scrape` executor**: loop `crawl_url` over a URL array; map each `CrawlOutcome` to the per-URL - result shape; declare the `web_crawl` `billing_unit`. -3. **`web.discover` executor**: wrap the `04b` source-discovery core (SearXNG/Linkup/Baidu, env-keyed); - declare its `billing_unit` (or free). -4. **Job store**: thin job record (`id, status, progress, result_ref`) + Celery dispatch for job-mode verbs. -5. **Uniform envelope**: shared `completed|pending` result type returned by every executor. -6. **`maps.*` contracts**: input/output schemas + executor stubs against the (separate) Maps actor. -7. **Billing seam**: `billing_unit` declaration honored via the billing service (no per-verb price code). - -## Tests - -- **Envelope**: a small `web.scrape` returns `completed` inline; a large/`async:true` one returns - `pending` + `job_id`; the job completes to `results`. -- **Array semantics**: `web.scrape([a, b, c])` returns one per-URL row each; partial failures don't fail - the batch. -- **Billing**: each `SUCCESS` bills exactly one `web_crawl` unit via the billing service; `EMPTY`/`FAILED` - free; disabling the billing flag makes it a no-op. -- **Registry → executor parity**: the Intelligence loop and a door hit the *same* executor for the same verb. -- **`web.discover`**: returns `{url,title,snippet,provider}`; self-disables when no provider env key is set. - -## Risks / trade-offs - -- **Maps actor is a hard external dependency** for `maps.*` — contracts ship now, executors light up when - the actor lands (doesn't block `web.*`). -- **Job store vs no store**: a thin job record is unavoidable for slow verbs; keep it minimal so it never - grows back into `pipeline_runs`. -- **Per-success billing on batches** can exhaust a wallet mid-run — pre-check is an upper bound; exact - mid-run behavior is an implementation-time call. - -## Resolved decisions - -1. Verbs: `web.scrape` · `web.discover` · `maps.search` · `maps.place` · `maps.reviews`. Namespaced; nothing top-level. -2. One array-friendly `web.scrape` (no separate batch verb). -3. Execution mode = result property (uniform `completed`/`pending` envelope), not separate verbs. -4. Build the thin job model now (Maps search / batch scrape force it). -5. One capability registry → generates chat/REST/MCP doors (`04b`) + feeds the Intelligence executor (`05b`). -6. `app/capabilities/` Apache-2; Maps extractor proprietary; billing **delegated to the billing service** - (verbs declare a `billing_unit`; `03c` is the first provider, new units register there). -7. Connector → capability is a **replacement** for these data sources, not an extension. - -## Out of scope (hand-offs) - -- **The doors** (chat/REST/MCP adapters, auth, metering gate) → `04b`. -- **Stateful accumulation** (Tracker/Timeline) → `05a`/`05b`; capabilities stay stateless. -- **The Google Maps actor** is **net-new** (not part of shipped Phases 1–3) — designed/built as a - **separate effort** (incl. sourcing legality). `maps.*` verbs are contracts against it; this domain - doesn't block on it. -- **Recursive site crawl**, a generic **`web.extract`** verb, and additional platform namespaces - (`linkedin.*`, `amazon.*`) — deferred. -- **04a (old, connector taxonomy)** → demoted to backward-compat hygiene; the BYO-`MCP_CONNECTOR` routing - fix belongs to `04b` (consume-user-MCP). **04b (old, source-discovery)** → absorbed here as `web.discover`. - -## Open questions (carry forward) - -- Default async threshold for `web.scrape`. -- Whether `web.discover` is metered or free. diff --git a/plans/backend/revamp phases 4-7/04b-access.md b/plans/backend/revamp phases 4-7/04b-access.md deleted file mode 100644 index 3cea58d45..000000000 --- a/plans/backend/revamp phases 4-7/04b-access.md +++ /dev/null @@ -1,136 +0,0 @@ -# Phase 4b — Access / Surfaces (chat · REST · MCP doors) - -> Part of **Phase 4 — Capabilities & Access**. Sits on top of `04a-capabilities.md` (the registry it exposes). -> **Build after** `04a`. Together, `04a + 04b` ship **Product A** (stateless utility) — revenue day one. -> **Depends on** SHIPPED infra: Identity/Tenancy, **API keys**, the chat agent + its tool registry, the -> streaming layer, and Metering (`03c`). Access *reuses* them — it builds none net-new. -> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. Locate code by **symbol/grep**, not the cited lines. - -## Objective - -Expose the capability registry (`04a`) to callers, **authenticated + metered**, through three doors — -**chat tools, REST + API keys, MCP server** — all **generated from the one registry** so the I/O -contract cannot drift between surfaces. Access contains **no business logic**: every door is the same -thin adapter. - -## Current state (cited) - -- **API keys** — existing per-workspace key infra (reuse; do **not** build net-new); billed to the - workspace owner via `03c`. -- **Chat agent + tool registry** — capability tools register *into* the existing registry; the seed is - `research/tools/scrape_webpage.py` (capability executor + access door + `03c` turn-accumulator billing). -- **Slow-job pattern** — `subagents/builtins/deliverables/deliverable_wait.py`: dispatch a Celery task, - poll the row's `status` until `READY`/`FAILED` (1.5s cadence), bounded by - `SURFSENSE_SUBAGENT_INVOKE_TIMEOUT_SECONDS` (default 300s); `deliverables/tools/podcast.py` is the - "return now + live card tracks progress" model. -- **BYO-`MCP_CONNECTOR`** (old `04a`) — the user's own external MCP tools inside our chat agent; its - routing-map gap is the one "connector" worth fixing, and it lands **here**. - -## Target design - -### The adapter shape (identical on every door) - -``` -parse input → validate against verb.input_schema → authn/authz → meter-gate (03c) - → call the SAME executor (04a) → serialize verb.output_schema → return the uniform envelope -``` - -### The three doors (locked order: chat → REST → MCP) - -| Door | Who | Auth | Status | -|------|-----|------|--------| -| **Chat tools** | in-app agent (Product B delivery + interactive) | existing session + workspace | partly exists (`scrape_webpage`) | -| **REST + API keys** | external developers (Product A) | **existing API-key infra** | **public day one** | -| **MCP server** | external agents (Cursor/ChatGPT/Claude) | OAuth 2.1 **or** bearer — chosen at implementation | **fast-follow** | - -REST being public day one is cheap precisely because the routes are **generated**, not hand-written — -it's a go-to-market choice, not an engineering cost. - -### Natural language is THE surface (verbs are internal) — non-negotiable - -The human-facing product is **the conversation**. A user **never** names a verb, fills an `input_schema`, -or knows "Product A vs B" exists — they describe a *need* in plain language and the agent does the rest. -Raw typed verbs are exposed only on the REST/MCP doors (devs/external agents). The chat agent owns three -responsibilities per message: **understand intent** → **pick & fill verbs** (incl. chains like -`discover → scrape`, `search → place → reviews`) → **answer in plain language** (results, not envelopes). - -### The intent router (the one new orchestration rule) - -The agent classifies each request along the stateless/stateful line **from the language**: - -``` -"compare / find / what is / pull / summarize / right now" → ONE-SHOT → call verbs, answer (Product A, stateless) -"watch / track / notify me when / every week / keep an eye / over time" → STANDING → start Tracker setup (Product B, stateful → 05b) -ambiguous → ask ONE question: "just this once, or should I keep watching it for you?" -``` - -This router is the friendly seam between the two products; it lives in the chat door (its prompt lives -in the `intelligence_agent`, `07`) and is the only human-facing decision point. - -### The two MCP directions (keep distinct) - -- **We *serve* MCP** — our capabilities as a remote MCP server (door #3, new): Streamable-HTTP `/mcp`, - stateless, bounded/paginated outputs, untrusted inputs, least-privilege (read-only verbs). -- **We *consume* MCP** — the BYO-`MCP_CONNECTOR` (old 04a): the user's own external MCP tools inside our - chat agent. The 04a routing-gap fix lands **here**, not in Capabilities. - -### Chat ↔ slow jobs — reuse the existing background-worker pattern - -Do **not** invent a chat-async mechanism. A slow verb (`web.scrape` over many URLs, `maps.search`, -`maps.reviews`) invoked from chat dispatches the **job** (the `04a` job record) and uses the -`deliverable_wait` poll-until-terminal path; the capability **job record's `status`** is what the helper -polls. Most calls finish inside the poll window → results inline; genuinely long ones surface a tracked -card. REST/MCP expose the same job via `GET /v1/jobs/{id}` (and an MCP equivalent). - -## Work items - -1. **Door generator**: from the `04a` registry, emit (a) chat tool defs + handlers, (b) REST routes + - request/response models, (c) MCP tool schemas + handlers — one adapter shape. -2. **REST surface**: public routes + API-key auth (reuse existing) + the `03c` meter-gate; `GET /v1/jobs/{id}`. -3. **Chat tools**: generalize `scrape_webpage` into the registry-backed set; wire slow verbs through - `deliverable_wait` polling the `04a` job record's `status`. -4. **Intent router**: the A-vs-B classifier (its home is the `07` subagent prompt; the seam is here). -5. **MCP server (fast-follow)**: Streamable-HTTP `/mcp`, least-privilege; auth depth chosen at implementation. -6. **Consume-MCP fix**: repair the BYO-`MCP_CONNECTOR` routing-map gap (old 04a). - -## Tests - -- **Generated parity**: a verb added to the registry appears on chat + REST (+ MCP) with identical I/O. -- **Auth + meter**: REST without a valid key → 401; an over-budget call → blocked by the `03c` gate - before execute; a success charges once. -- **Chat slow verb**: a many-URL `web.scrape` returns inline when fast; surfaces a tracked card when long; - `GET /v1/jobs/{id}` reflects the same terminal status. -- **Intent router**: "compare X and Y" → one-shot; "watch X weekly" → Tracker setup handoff; ambiguous → - exactly one clarifying question. -- **Consume-MCP**: a configured BYO MCP tool is reachable by the chat agent (routing-gap closed). - -## Risks / trade-offs - -- **Public REST day one** → needs bounded inputs + per-key quotas / abuse posture (design alongside launch). -- **MCP auth depth** deferred to implementation (OAuth 2.1 vs bearer) — don't over-build before a consumer exists. -- **Serve-vs-consume MCP confusion** — keep the two directions explicitly separate in code and docs. - -## Resolved decisions - -0. **Natural language is the only human-facing surface.** Users never name verbs/schemas/jobs; the chat - agent understands intent, picks & fills verbs, answers in plain language; an intent router classifies - one-shot (A) vs standing-concern (B), asking one question only when ambiguous. Raw verbs live only on REST/MCP. -1. Three doors, generated from the capability registry; order chat → REST → MCP. -2. REST is **public day one** (cheap; go-to-market choice). -3. API keys: **reuse existing infra**, billed to workspace owner. -4. MCP server is a **fast-follow**; auth depth (OAuth 2.1 vs bearer) chosen at implementation. -5. Chat ↔ slow jobs: **reuse `deliverable_wait` poll-until-terminal + live card**, polling the `04a` job record's `status`. -6. "Serve MCP" (our tools out) vs "consume MCP" (BYO tools in) are distinct; the old-04a fix is the consume side. - -## Out of scope (hand-offs) - -- **The verbs themselves** (executors, registry, billing units) → `04a`. -- **The CI subagent + its prompt/playbook** (where the intent router actually lives) → `07`. -- **Stateful flows** (Tracker crafting, refresh, timeline reads) → `05b`/`05a`, surfaced via chat tools in `07`. -- Legacy branded connectors stay only for backward-compat (separate hygiene task). - -## Open questions (carry forward) - -- MCP auth depth (decide at implementation). -- Public REST rate-limiting / abuse posture (bounded inputs, per-key quotas). -- Whether `web.discover` is metered or free (carried from `04a`). diff --git a/plans/backend/revamp phases 4-7/05a-timeline.md b/plans/backend/revamp phases 4-7/05a-timeline.md deleted file mode 100644 index 436257cd4..000000000 --- a/plans/backend/revamp phases 4-7/05a-timeline.md +++ /dev/null @@ -1,125 +0,0 @@ -# Phase 5a — Timeline (the moat asset) - -> Part of **Phase 5 — Intelligence & Timeline**. Sibling: `05b-intelligence.md` (the process that writes -> this state). **Build first** within Phase 5 — the tables must exist before the hot loop writes them. -> **Depends on** `04a` (the verbs the loop calls) and Phase 1 (the DB / migration baseline). -> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. The Timeline is **CI-owned, new tables** — it is **not** -> the Knowledge Base (documents/embeddings) and **not** `automation_runs`. - -## Objective - -Durably store the **time-shaped truth** for each Tracker. This is the asset: **time is the moat** — -accumulated history a later entrant cannot re-create. Design rule: **store deltas, not snapshots — no -change, no row.** Storage grows with the *rate of change*, not the number of runs. - -## Current state (cited) - -- **No CI state exists today** — crawled data is explicitly *not* indexed; the only persisted CI state is - what this phase introduces. -- **Reference models** for shape/placement: connectors/folders ORM in `app/db.py` (full-row Zero - publication, like folders/connectors); `automations`/`automation_runs` as the *orchestration* analog - (this is the **fact** analog, deliberately separate). -- **Hot-loop pre-check** (`05b` step 2) reads `WebCrawlerConnector.format_to_structured_document( - exclude_metadata=True)` to compute the `content_hash` stored here. - -## Target design - -### The state / process split - -Intelligence (`05b`) is the only **writer** (via the hot loop). **Readers** — the Conversation domain -today, future dashboards/alerts/the deferred resale product — read the Timeline **directly, without -running the loop**. That separation is why Timeline is its own domain. - -### The three stores - -| Store | Role | Write pattern | -|-------|------|---------------| -| `tracked_entities` | stable identity per tracked thing (the Tracker's `identity_rule` → `entity_key`) | written **once** | -| `entity_current_state` | latest values + `content_hash` + `last_checked_at` per entity | **overwritten** each run | -| `entity_changes` (the change log) | append-only material deltas | **appended**, never overwritten | - -**The timeline = the change log read in order.** To reconstruct a past point: take Current state and -replay deltas backward (north-star tooling; MVP just *stores* the deltas). - -### Data model sketch (new tables) - -``` -tracked_entities - id · workspace_id · tracker_id (FK) · entity_key (unique per tracker) · first_seen_at - # MVP: exactly one row per Tracker (single-entity). Table stays multi-entity-ready. - -entity_current_state - entity_id (FK, unique) · tracker_id · fields JSONB (latest, conforms to locked field_schema) - · content_hash · last_checked_at · updated_at - # overwritten each material run; content_hash powers the hot-loop cheap pre-check (05b step 2) - -entity_changes # the append-only change log - id · entity_id (FK) · tracker_id · captured_at - · delta JSONB # { field: { from, to } } - · materiality # material | notable(=notable_signals-sourced) - · decided_by # code | agent (audit of the materiality split) - · source_ref # url / blob key the change was observed from - · note TEXT NULL # optional agent rationale (the "why material") -``` - -- **No change → no row** in `entity_changes`; an unchanged refresh only bumps `last_checked_at`. -- `decided_by` records whether code or the agent ruled the change material (provenance seed). - -### What it is NOT - -- **Not the KB** — no `Document` rows, no embeddings, no indexing. ("Don't index crawled data" holds.) -- **Not `automation_runs`** — that's an orchestration artifact; this is the durable fact store. -- **Not a resurrected `pipeline_runs`.** - -### Where it lives - -- New CI-owned tables (in `app/db.py` alongside core entities, or a small `app/timeline/` package — decide - at write-up). **Apache-2** (it stores facts; the moat is in Acquisition + the Maps extractor). -- Published to Zero full-row later if/when a UI needs live sync (deferred with the frontend). - -## Work items - -1. **Models + migration**: `tracked_entities` / `entity_current_state` / `entity_changes` + Alembic migration. -2. **Write API**: `upsert_current_state(...)` (overwrite) and `append_change(...)` (insert) used by the `05b` loop. -3. **Read API**: `query_timeline(tracker_id, …)` + `get_current_state(entity_id)` for the conversation/read side. -4. **Content-hash field**: store + expose `content_hash` so `05b` step 2 can short-circuit. -5. **Zero publication entry** (full-row) — wired but inert until a UI consumes it (deferred). - -## Tests - -- **No change → no row**: an unchanged refresh bumps `last_checked_at` only; `entity_changes` count is unchanged. -- **Append-only log**: a material refresh inserts exactly one `entity_changes` row and overwrites `entity_current_state`. -- **Provenance**: `decided_by` is `code` for threshold rules and `agent` for ambiguous calls. -- **Identity**: `entity_key` is unique per tracker; re-refresh of the same entity reuses the row. -- **Read API**: `query_timeline` returns deltas in `captured_at` order. - -## Risks / trade-offs - -- **First-run baseline**: with no prior `entity_current_state`, the diff has nothing to compare against — - baseline semantics (silent establish vs flood the log) are an implementation-time call. -- **High-velocity entities**: append-only growth is bounded by change rate, but retention/archival for very - chatty entities is deferred. -- **Schema-validate-on-write**: validating `fields` against the locked `field_schema` is a cheap integrity - guard (lean: yes) but adds a write-path dependency on `05b`'s lock. - -## Resolved decisions - -1. Three stores: `tracked_entities` / `entity_current_state` / `entity_changes`. -2. Store deltas, not snapshots; **no change → no row**; storage ∝ rate of change. -3. CI-owned new tables; **not** KB, **not** `automation_runs`. -4. `content_hash` on current state powers the hot-loop cheap pre-check. -5. `decided_by` on changes records the code-vs-agent materiality provenance. -6. Single entity per Tracker for MVP; schema stays multi-entity-ready (additive later). - -## Out of scope (hand-offs) - -- **The writer** (hot loop, materiality, schema lock) → `05b`. -- **Backward-replay reconstruction**, trend/series read APIs, coverage-confidence, multi-entity scale, the - resale/data-product surface → north star (deferred). -- **Live UI sync** (Zero consumption, dashboards) → frontend umbrella. - -## Open questions (carry forward) - -- ORM home: `app/db.py` vs a dedicated `app/timeline/` package. -- Whether `entity_current_state.fields` is validated against the locked `field_schema` at write time (lean: yes). -- Retention / archival policy for very high-velocity entities (deferred). diff --git a/plans/backend/revamp phases 4-7/05b-intelligence.md b/plans/backend/revamp phases 4-7/05b-intelligence.md deleted file mode 100644 index e2763627f..000000000 --- a/plans/backend/revamp phases 4-7/05b-intelligence.md +++ /dev/null @@ -1,191 +0,0 @@ -# Phase 5b — Intelligence (the decision-grounded engine) - -> Part of **Phase 5 — Intelligence & Timeline**. Sibling: `05a-timeline.md` (the state it writes). -> **Build after** `05a` (the tables) and `04a` (the verbs it calls). Together, `05a + 05b` ship the -> **Product B engine** (the Tracker, locked schema, hot loop, deltas). -> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This is net-new and is **not** the KB and **not** the -> automations subsystem. -> **Name:** the standing-concern primitive is the **Tracker**. - -## Objective - -Turn repeated capability calls into **decision-relevant structured signal**. The motto: **the agent -judges, code computes.** This replaces the old "pipeline" as the *standing concern*: Intelligence is the -**process** that mutates state; Timeline (`05a`) is the **state**. Everything below the -Access→Intelligence boundary stays pure functions. - -``` -STATELESS (Product A): 04a Capabilities + 04b Access → call → data → bill, nothing persists -STATEFUL (Product B): 05b Intelligence + 05a Timeline → the Timeline IS the state -``` - -## Current state (cited) - -- **Capability executors** (`04a`) — called directly by the loop (not through a door). -- **Content-hash pre-check** — `WebCrawlerConnector.format_to_structured_document(exclude_metadata=True)` - produces the stable text the loop hashes against `entity_current_state.content_hash` (`05a`). -- **Run/audit substrate to reuse** — the existing **`AutomationRun`** (status/error/timing/`step_results`; - note the executor writes results to `step_results[n].result`, **not** `run.output`/`artifacts`, which stay - unwritten) + the automations executor's PENDING→running gate (`execute_run()` early-returns unless - `status == PENDING`; `acks_late` is global). **Caveat:** the gate is **not atomic** (no - `UPDATE … WHERE status='pending'`), so it stops terminal/post-`mark_running` redelivery but **not** - concurrent double-claim — pair it with the per-Tracker lock (`06`). Plus the chat **job record** (`04a`) + - `deliverable_wait`. **No new run table.** -- **CI context uploads** — the existing folder-upload machinery (`POST /documents/folder-upload`, param - `root_folder_id` — there is no `destination_folder`) + KB retrieval scoped via mention-pins → - `referenced_document_ids(folder_ids=…)` → `SearchScope.document_ids` (no direct `folder_id` on search). - -## Target design - -### The primitive — `Tracker` - -A saved, decision-grounded subject that accumulates structured signal over time: - -| Field | Meaning | -|-------|---------| -| `decision` | the question being tracked toward ("is this competitor pulling ahead?") | -| `capability_binding` | which verb + input feeds it (`maps.place(X)`, `web.scrape([Y])`) | -| `definition` (locked, versioned) | `{ field_schema, identity_rule, materiality }` — the agent-drafted, human-locked contract | -| `status` | `draft` → `locked`/`active` | - -One **entity per Tracker for MVP** (one place / one URL). Multi-entity (`maps.search → many`) is deferred -(the `05a` model stays multi-entity-ready so it's additive). - -### Setup (once) — the agent-designed schema flow (IN MVP) - -The product must not be rigid: **we cannot author one schema that serves everyone**, so the schema is -derived from the *user's* decision by an agent and locked by the human. Conversationally (chat-first): - -1. **Bind** a capability + input. -2. **Sample fetch** — one real capability call so the agent drafts against *actual* returned data. -3. **Agent drafts the `definition`** from `decision` + the sample: `field_schema` (fields + types), - `materiality` (numeric thresholds where possible; "ask agent" otherwise), `identity_rule` (stable - entity key — Maps `place_id`, canonical URL), and a reserved **`notable_signals`** escape-hatch field. -4. **Human reviews & locks** (in chat: "looks good" / "add field X"). Locked ⇒ stable run-to-run. -5. **Versioned** — a locked `definition` is a snapshot; edits create a new version (mirrors how - `automations` snapshots `definition`). - -### The hot loop (per refresh) — `refresh(tracker)` - -1. **Crawl** — call the bound capability (`04a`) → raw data. -2. **Cheap pre-check** — content hash; identical to the stored `content_hash` → stamp `last_checked_at`, - **stop** (no LLM cost). -3. **Fill** — agent conforms raw data to the **locked `field_schema`** via structured output; it does - **not** invent fields. Unanticipated observations go into `notable_signals`. -4. **Diff (code)** — deterministic compare of the new record vs Current state (`05a`) → raw deltas. -5. **Judge — the materiality split:** - - **deterministic (code):** numeric/clear-cut rules from `materiality`, applied for free, 100% - reproducible (e.g. `rating Δ≥0.2 → material`, `review_count Δ≥10 → material`, `1¢ wobble → noise`). - - **agent (only on ambiguous):** anything a rule can't decide — reworded `description`, a new - `notable_signals` entry → one LLM call rules material/noise. -6. **Append** — if material: write a Change + update Current state (`05a`). Else: only `last_checked_at`. - **No change → no row.** - -**Worked example (`maps.place` refresh):** -``` -rating 4.4 → 4.3 (Δ0.1) → code: < 0.2 → NOISE (no LLM) -review_count 312 → 470 → code: ≥ 10 → MATERIAL (no LLM) -hours unchanged → no delta -description reworded → code: no rule → ASK AGENT → NOISE -⇒ one Change row (review spike); one cheap LLM call; zero LLM on the rating tick. -``` - -### Refresh execution & idempotency — ride the invoking surface (no new run table) - -`refresh(tracker)` is a **headless unit of work**; the run/audit record + idempotency live on **whatever -surface invoked it**: - -- **Recurring (in-app):** invoked by the **CI automation action** (`06`) → the existing **`AutomationRun`** - is the run record and the automations executor's PENDING-gate handles redelivery. Reused, not re-written — - but the gate is **not atomic** against concurrent claims, so the **per-Tracker lock (`06`)** is the actual - double-refresh guard. -- **Chat (manual / agent):** invoked via the chat **job record** (`04a`) + `deliverable_wait`. -- **Billing idempotency is per *capability call*, not per run:** each `executor` bills a success once via - the billing service (`04a`); the content-hash pre-check (step 2) prevents needless re-crawls/charges. No - run-level `charged_micros` ledger required for MVP. - -Net: the only genuinely new state is the **Timeline** (`05a`); execution accounting is borrowed. - -### User-supplied context files (the F idea, generalized) - -When a user uploads a file *in a CI chat* (e.g. "our own price list"), it goes into the **KB as normal** — -uploads create `Document`s and are indexed/embedded, exactly as today. **(The "don't index" rule applies -only to *crawled* data.)** The CI-specific part is **organization + use**: - -- **Routed to a dedicated folder** for that CI chat/Tracker (reuse `POST /documents/folder-upload` with - `root_folder_id`; uploads land as `Document`s with `folder_id`). -- **The judge step (5) may consult them** — retrieved from the KB, scoped to that folder via the mention-pin - path (`referenced_document_ids(folder_ids=…)` → `SearchScope.document_ids`), since search has no direct - `folder_id` filter: - -``` -competitor price 12.00 → 9.90 + user's context file says "our price is 10.00" - → agent: competitor crossed *below our price* → MATERIAL (and explain why) -``` - -The user's private context **shapes what counts as material**. Reuses existing KB upload + folder + -retrieval machinery (nothing new). **MVP-optional** (the loop works without it); design the seam now. - -### Where it lives / decoupling - -- New **Apache-2** package `app/intelligence/` (schema-design agent, hot loop, materiality evaluator). Calls - capability `executor`s directly. -- Exposes **`refresh(tracker)`**. *Who* calls it (manual / agent / external cron / optional automation) is - the **Triggers** domain's concern (`06`) — Intelligence has **no dependency** on any scheduler. - -## Work items - -1. **Tracker model + persistence**: `decision` · `capability_binding` · versioned locked `definition` · `status`. -2. **Schema-design flow**: bind → sample-fetch → agent-drafts `definition` → human review/lock → version. -3. **Materiality evaluator**: deterministic rule engine (numeric/clear) + the agent-on-ambiguous fallback. -4. **The hot loop** `refresh(tracker)`: crawl → hash pre-check → fill → diff → judge → append (writes `05a`). -5. **Idempotency wiring**: ride `AutomationRun` (recurring) / chat job record (manual) — no new run table. -6. **Context-folder seam**: optional KB-retrieval hook into the judge, scoped to the Tracker's folder. - -## Tests - -- **Pre-check short-circuit**: identical content hash → no fill, no LLM, only `last_checked_at` bumped. -- **Fill conforms to lock**: extra observed fields land in `notable_signals`, never invent schema fields. -- **Materiality split**: numeric Δ over threshold → `decided_by=code, material`; ambiguous reword → - `decided_by=agent`; sub-threshold → noise, no row. -- **Append semantics**: a material run writes one `entity_changes` row + overwrites current state. -- **Idempotency**: a redelivered recurring refresh (same `AutomationRun`) does not double-write or double-bill. -- **Context folder (optional)**: judge ruling flips when a user context file changes the decision frame. - -## Risks / trade-offs - -- **Refresh failure path** (capability returns `FAILED`/partial): skip vs retain vs retry vs alert — an - implementation-time call (no architecture impact). -- **Agent fill cost** on changed pages: bounded by the hash pre-check; only changed content reaches the LLM. -- **Schema lock rigidity**: locking trades flexibility for run-to-run stability; re-lock creates a new version. - -## Resolved decisions - -1. `Tracker` is the standing-concern primitive; replaces "pipeline". -2. Stateless (`04`/Product A) vs stateful (`05`/Product B) is the Access→Intelligence boundary. -3. **Agent-designed schema flow is in MVP** (not hand-authored) — sample-grounded, human-locked, versioned. -4. Single entity per Tracker for MVP. -5. Materiality = deterministic numeric/clear rules in code + agent only on ambiguous. -6. Content-hash pre-check short-circuits unchanged pages before any LLM spend. -7. `app/intelligence/` Apache-2; `refresh(tracker)` is trigger-agnostic. -8. **No new run table** — refresh audit rides `AutomationRun` (recurring) or the chat job record; redelivery - is stopped by the PENDING-gate **plus a per-Tracker lock** (the gate is non-atomic); billing idempotency - is per-capability-call + the content-hash gate. Only Timeline (`05a`) is new state. -9. **CI context folder** (F): user files uploaded in a CI chat go into the **KB as normal** (indexed), - routed to a dedicated `Folder`, and may feed the judge via KB retrieval. "Don't index" is for *crawled* - data only. MVP-optional seam. - -## Out of scope (hand-offs) - -- **The state tables** (the three stores, content-hash, read API) → `05a`. -- **When refresh fires + recurrence/delivery** → `06` (Triggers). -- **The human-facing crafting/answering experience** (the `intelligence_agent` subagent + prompt) → `07`. -- **Schema auto-evolution**, multi-entity Trackers, backward-replay, coverage-confidence, the - resale/data-product stage → north star (deferred). - -## Open questions (carry forward) - -- How the schema-design agent surfaces "review & lock" before the frontend exists (pure-chat confirmation?). -- Versioning policy on re-lock (new version vs in-place) — lean new version. -- Where the schema-design agent itself runs (a setup capability? a chat sub-flow?). -- Context-folder → judge wiring (how much to load; per-Tracker vs per-chat scope). diff --git a/plans/backend/revamp phases 4-7/06-triggers.md b/plans/backend/revamp phases 4-7/06-triggers.md deleted file mode 100644 index b35e90d37..000000000 --- a/plans/backend/revamp phases 4-7/06-triggers.md +++ /dev/null @@ -1,131 +0,0 @@ -# Phase 6 — Triggers (the pluggable refresh clock) - -> **Phase 6** of the CI-pivot revamp — the thinnest phase. **Build after** Phase 5 (`05b` exposes -> `refresh(tracker)`). -> **Depends on** `05b` (`refresh(tracker)`), `04b` (the REST manual/cron routes), and the SHIPPED -> automations subsystem (its schedule selector + `AutomationRun`) plus `app/notifications/` for alert delivery. -> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This phase is **decoupled** — Intelligence (`05b`) has no -> dependency on it, and **automations is at most one optional adapter, never required**. - -## Objective - -Decide **when** a Tracker refreshes. Intelligence exposes a single entry point — **`refresh(tracker)`** — -and every trigger is just a caller. Intelligence never knows which trigger fired; remove any trigger and -the engine still works. This replaces the old Phase-6 cron scheduler — and the resolution is **not** to -rebuild a scheduler at all, but to **reuse the automations subsystem** for the in-app recurring path. - -## Current state (cited) - -- **`refresh(tracker)`** — the headless unit of work exposed by `05b`. -- **Automations schedule selector** — the already-hardened cron selector (`FOR UPDATE SKIP LOCKED` - claiming, `next_fire_at` advance, self-heal, duplicate-run suppression, `catchup=False`), reusing the - `croniter` util. *(Verified: `app/automations/triggers/builtin/schedule/selector.py::_claim_due_triggers` - + `schedule/cron.py::compute_next_fire_at`.)* -- **`AutomationRun`** — the existing run record + a PENDING-gate. *(Verified with a caveat: the gate is - `execute_run()` early-returning unless `status == PENDING` + `mark_running`, and `acks_late` / - `task_reject_on_worker_lost` are set globally — so **terminal redelivery and post-`mark_running` - redelivery are no-ops**. But there is **no atomic `UPDATE … WHERE status='pending'` claim**, so two - concurrent workers can both read PENDING → double-run. The per-Tracker lock below is therefore the - primary concurrency guard, not just belt-and-suspenders.)* -- **Delivery ≠ automations (correction).** Automations have **no** push/email/notification path, and - `automation_runs.output` is **never written** by the executor (results live in `step_results[n].result`); - today "delivery" is only **pull via `GET /automations/{id}/runs/{run_id}`** + **Zero dashboard sync** - (`zero_publication.AUTOMATION_RUN_COLS`). In-app alerts must wire to the **separate `app/notifications/` - system** — `NotificationService.create_notification(...)` (DB-backed, Zero-synced, typed - `NotificationType`). So alert delivery is a **small net-new wiring**, not a free automations reuse. -- **Access routes** (`04b`) — where the external-cron and manual REST endpoints live. - -## Target design - -### The adapters - -| Adapter | Fired by | MVP? | -|---------|----------|------| -| **Manual** | user "refresh now" (chat tool / REST) | ✅ | -| **Agent** | the in-app agent calls `refresh` as a tool | ✅ | -| **External cron** | the user's own scheduler hits `POST /v1/trackers/{id}/refresh` | ✅ (zero infra on us) | -| **CI automation action** | a **CI action on the existing automations** — its schedule trigger fires `refresh(tracker)`, persists deltas to the Timeline, **and emits an in-app alert** via `app/notifications/` | ✅ (the in-app recurrence + alert path) | - -### Recurrence + delivery — a CI action on existing automations (NOT a new scheduler, NOT a new shape) - -The SMB competitor-watch buyer needs **in-app recurrence** *and* **"tell me when it changes."** Instead of -building a bespoke tick, we **add a CI *action* to the existing automations subsystem**: - -- **Schedule** → the automation's existing **schedule trigger** (the already-hardened selector). No new - scheduler. **(closes old Gap B — scheduler rigor.)** -- **Run record + idempotency** → the automation's existing **`AutomationRun`** + PENDING-gate. No new run - table. **(closes old Gap A — run/idempotency)** — *but* pair it with the per-Tracker lock (the gate is - non-atomic; see Current state + `05b`). -- **Delivery / alert** → **not** free from automations. Persist the material change to the **Timeline** - (`05a`), then emit an in-app alert via **`app/notifications/`** (`NotificationService.create_notification` - with a new CI notification type; Zero-synced to the UI). **(old Gap E — alert delivery — is *mostly* - closed but needs this small wiring, not zero-cost reuse.)** - -**Why a CI *action*, not a new automation shape:** a new shape would duplicate triggers, runs, and delivery -that already exist. A single `refresh_tracker` action reuses all of it. (If implementation finds the action -too constraining, a thin CI-specific shape is the fallback — but the action is the default.) - -### Decoupling is preserved (automations is still optional) - -CI **core** — `refresh(tracker)` + Timeline (`05a`/`05b`) — has **zero** automations dependency and runs -via manual / agent / external-cron. Automations is the **optional adapter** that adds recurrence + audit -for in-app users (alerts ride `app/notifications/`, not automations). So we honor "don't glue CI to -automations" *and* get its machinery for free. - -### Where it lives - -- The **CI action** lives with automations (its action registry); it imports `refresh(tracker)` from - `app/intelligence/`. No new scheduler/Beat task. -- The **external-cron** and **REST manual** paths are just Access-door routes (`POST - /v1/trackers/{id}/refresh`) — `04b` plumbing. - -## Work items - -1. **Manual / agent triggers**: a `refresh_tracker(tracker_id)` chat tool + REST route → `refresh(tracker)`. -2. **External-cron route**: `POST /v1/trackers/{id}/refresh` (API-key authed) → `refresh(tracker)`. -3. **CI automation action**: register a `refresh_tracker` action in the automations action registry that - calls `refresh(tracker)` and, on material change, emits an in-app alert via `app/notifications/` - (`NotificationService.create_notification`, new CI notification type). *(Not "automations delivery" — - that path doesn't exist; see Current state.)* -4. **Concurrency guard**: a per-Tracker in-flight lock — the **primary** guard against concurrent - double-refresh, since the automation PENDING-gate is non-atomic (also protects the manual/cron paths - that don't go through automations). - -## Tests - -- **Decoupling**: `refresh(tracker)` works via manual/agent/external-cron with automations disabled entirely. -- **Recurring path**: a scheduled CI action fires `refresh` on cron and delivers material changes; an - unchanged refresh delivers nothing. -- **Idempotency**: a redelivered scheduled run (same `AutomationRun`) does not double-refresh. -- **External cron**: `POST /v1/trackers/{id}/refresh` triggers exactly one refresh; rejects bad auth. -- **Concurrency**: a second refresh while one is in flight is skipped/queued, not run concurrently. - -## Risks / trade-offs - -- **Action vs shape**: the CI action is the default; a thin CI automation shape is the fallback if the - action proves too constraining. -- **Delivered payload shape** (agent-summarized vs raw deltas since last fire) — implementation-time call. -- **Double-guarding concurrency**: the per-Tracker lock overlaps the automation run-gate, but the lock also - protects the manual/cron paths that don't go through automations. - -## Resolved decisions - -1. Intelligence exposes `refresh(tracker)`; all triggers are callers. Fully decoupled. -2. Adapters: manual · agent · external-cron · **CI automation action** (recurrence + delivery). -3. **No bespoke scheduler and no new run table** — the recurring path reuses the automations schedule - selector + `AutomationRun` (Closes old Gaps A/B). **Delivery does NOT reuse automations** (no such - path; `run.output` is unwritten) — alerts go through `app/notifications/` (Gap E: small wiring). -4. Recurrence is a **CI *action*** on the existing automations, **not a new automation shape**. -5. CI core stays runnable with **zero** automations dependency (manual/agent/external-cron). - -## Out of scope (hand-offs) - -- **`refresh(tracker)` internals** (the hot loop, materiality) → `05b`. -- **The Timeline reads** the delivery summarizes → `05a`. -- **The chat tool surface** for manual/agent refresh → `07` (the `intelligence_agent` toolset). - -## Open questions (carry forward) - -- CI **action** vs a thin CI-specific automation **shape** (default: action; shape is the fallback). -- What the delivered payload looks like (summarized material changes vs raw deltas since last fire). -- Concurrency: per-Tracker lock granularity (like the connector indexing lock). diff --git a/plans/backend/revamp phases 4-7/07-orchestration.md b/plans/backend/revamp phases 4-7/07-orchestration.md deleted file mode 100644 index f8b3cc142..000000000 --- a/plans/backend/revamp phases 4-7/07-orchestration.md +++ /dev/null @@ -1,126 +0,0 @@ -# Phase 7 — Orchestration / Conversation (the CI-expert subagent) - -> **Phase 7** of the CI-pivot revamp — the **human-facing brain** that turns "natural language is the only -> surface" (`04b`, Decision 0) into a real deliverable. **Build last** — it sits atop `04`–`06`. -> **Depends on** all prior phases: `04a` verbs, `04b` doors + `deliverable_wait`, `05a`/`05b` Tracker + -> Timeline, `06` refresh triggers. -> **Scope guardrail:** the multi-agent chat **runtime** (deepagents, subagent dispatch, streaming, -> citation middleware, `deliverable_wait`) is SHIPPED/FIXED. What's **net-new here** is one builtin -> subagent + its tools + its prompt. We *plug into* the runtime; we don't rebuild it. - -## Objective - -Ship the **`intelligence_agent`** — a net-new builtin CI-expert subagent — so users get the whole product -through plain conversation: intent routing (one-shot vs standing concern), verb composition, Tracker -crafting, and decision-grounded answering. The intent router + verb-composition + Tracker-crafting are -where "user-friendly" is won or lost, so the orchestration layer is **designed, not assumed**. - -## Current state (cited) - -- **Builtin subagent pattern** — `subagents/builtins//`: `agent.py` (`build_subagent(...)` → - `pack_subagent(name, description, system_prompt, tools, ruleset, …)`), `tools/index.py` - (`NAME · RULESET · load_tools(dependencies)`), `description.md` (router one-liner), `system_prompt.md` - (the playbook), optional middleware (e.g. `citation_state`). Peers: `research`, `deliverables`. -- **Tool shape** — `research/tools/scrape_webpage.py` shows the **tool = capability executor + access door** - pattern (calls `WebCrawlerConnector.crawl_url`, returns a typed dict). **Billing note:** its current - `get_current_accumulator()` fold is **interactive-chat-only** and no-ops outside a chat turn — so CI tools - do **not** copy that as the charging path. Per `04a` (locked), **the charge fires in the capability - executor**; the chat turn accumulator stays only as an optional presentation fold. This is what makes the - recurring `06 → refresh → verb → crawl` path (no chat turn) bill correctly. -- **Slow-job path** — `deliverable_wait` poll-until-terminal + the podcast-style live card (`04b`). - -## Target design - -### The `intelligence_agent` subagent (CI expert) - -A new builtin `subagents/builtins/intelligence_agent/` (the competitive-intelligence specialist), peer to -`research`/`deliverables`. The **main agent delegates** to it whenever the request is CI-flavored (research -a competitor, watch something, analyze a place's reviews); `description.md` is what makes that routing -happen. It owns the **CI playbook** in `system_prompt.md`: - -1. **Intent routing (A vs B)** — the Decision-0 rule, in-prompt: one-shot ("compare/find/what is") → call - verbs & answer; standing concern ("watch/track/notify when/weekly") → run the crafting flow; ambiguous → - ask the single clarifying question. -2. **Verb composition** — the chains: `web.discover → web.scrape`, `maps.search → maps.place → - maps.reviews`; infer URLs/queries/locations from context so the user never supplies them by hand. -3. **Tracker crafting** — the conversational schema-design flow from `05b`: sample-fetch → propose - `field_schema` + materiality + identity → user validates & locks → versioned. -4. **Decision-grounded answering** — read the Timeline (`05a`) to answer "what changed / is X pulling - ahead?" from stored deltas, not by re-deriving from chat history. - -### The toolset (what `load_tools` returns) - -| Tool | Wraps | Mode | Billing | -|------|-------|------|---------| -| capability verbs (`web.scrape`, `web.discover`, `maps.search`, `maps.place`, `maps.reviews`) | `04a` executors | inline-or-job (slow → `deliverable_wait`) | at the `04a` executor (per-call); chat turn accumulator = optional presentation fold only | -| `craft_tracker(decision, binding)` | `05b` schema-design agent | inline (sample-fetch + proposes) | sample crawl billed | -| `lock_tracker(draft)` / `update_tracker` | `05b` Tracker persistence | inline | — | -| `refresh_tracker(tracker_id)` | `05b` `refresh(tracker)` (via `06`) | job → `deliverable_wait` | per capability call | -| `query_timeline(tracker_id, …)` | `05a` read API | inline | — | -| `list_trackers()` | `05a`/`05b` read | inline | — | - -- **Capability verbs are a shared tool module** (generated from the `04a` registry) — `research` can load - the same ones; the `intelligence_agent` additionally loads the Tracker/Timeline tools + the CI prompt. - (`scrape_webpage` is the seed; generalize it into the registry-backed set.) -- **Slow verbs** (`maps.search`, multi-URL `web.scrape`, `refresh_tracker`) dispatch a job and use the - existing `deliverable_wait` poll-until-terminal + live-card path (`04b`). - -### Boundaries - -- **Orchestration ≠ Intelligence.** The `intelligence_agent` *drives* `05a`/`05b` via tools; the hot loop, - materiality, and Timeline writes live in `05a`/`05b`, callable headless (so REST/MCP and Triggers reach the - same logic with no agent in the loop). -- **Humans get the agent; machines get raw verbs.** REST/MCP callers (devs/external agents) skip this - subagent entirely and call `04a` verbs directly — they *want* explicitness. - -## Work items - -1. **Subagent scaffold**: `subagents/builtins/intelligence_agent/` with `agent.py` / `tools/index.py` / - `description.md` / `system_prompt.md`, packed via `pack_subagent`. -2. **CI playbook prompt**: intent routing (A/B + one clarifying question), verb chains, crafting flow, - decision-grounded answering. -3. **Shared verb tool module**: registry-backed (`04a`) capability tools, reusable by `research` too. -4. **Tracker/Timeline tools**: `craft_tracker` / `lock_tracker` / `update_tracker` / `refresh_tracker` / - `query_timeline` / `list_trackers`. -5. **Slow-verb wiring**: route job-mode tools through `deliverable_wait` + the live card. -6. **Router registration**: `description.md` so the main agent delegates CI-flavored requests here. - -## Tests - -- **Delegation**: a CI-flavored request routes to `intelligence_agent`; a non-CI request does not. -- **One-shot**: "compare X and Y" composes verbs and answers in plain language, persisting nothing. -- **Standing concern**: "watch X weekly" runs the crafting flow → lock → (refresh via `06`). -- **Crafting**: `craft_tracker` does a real sample-fetch and proposes a schema; `lock_tracker` versions it. -- **Answering**: `query_timeline` answers "what changed?" from stored deltas, not chat history. -- **Slow verb**: a multi-URL scrape surfaces a live card and returns terminal results. - -## Risks / trade-offs - -- **Prompt quality is the product**: intent-router/composition quality is a first-class deliverable, not - plumbing — budget iteration on the prompt. -- **One-shot ownership**: whether `research` keeps generic scraping or the `intelligence_agent` owns all - CI-flavored calls (lean: shared verb tools, CI agent owns the *playbook*). -- **Pre-frontend lock UX**: "review & lock" must render in pure chat for MVP (ties to `05b`'s open Q). - -## Resolved decisions - -1. CI orchestration is a **net-new builtin subagent** (`intelligence_agent`) on the existing runtime — not - a runtime rebuild. -2. Tools follow the `scrape_webpage` executor+door shape, but **billing fires in the `04a` executor** (not the chat turn accumulator, which is interactive-only and would skip automation/cron runs). -3. Capability verbs are a **shared, registry-generated** tool module; the `intelligence_agent` adds - Tracker/Timeline tools + the CI prompt. -4. Intent routing (A vs B) lives **in the subagent prompt**; the headless logic stays in `05a`/`05b`. -5. Slow verbs reuse `deliverable_wait`; nothing new for chat-async. - -## Out of scope (hand-offs) - -- **The verbs, doors, engine, state, and triggers** → `04`/`05`/`06` (this phase only *orchestrates* them). -- **Richer multi-step CI playbooks** (auto competitor discovery → multi-Tracker setup), proactive "you - should watch this" suggestions, cross-Tracker synthesis → north star (deferred). -- **Frontend CI surfaces** (cards, dashboards) → frontend umbrella. - -## Open questions (carry forward) - -- Does CI one-shot scraping stay in `research`, or does the `intelligence_agent` own all CI-flavored calls - (lean: shared verb tools, `intelligence_agent` owns the *CI playbook*). -- How `craft_tracker`'s "review & lock" renders pre-frontend (pure-chat confirmation — ties to `05b`'s open Q). From 78f7a93a8844a910c1cf31571e263c9c2b51c9c6 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 16:42:47 +0200 Subject: [PATCH 02/64] feat(capabilities): add registry contracts and package --- .../app/capabilities/__init__.py | 5 ++ surfsense_backend/app/capabilities/types.py | 47 +++++++++++++++++++ 2 files changed, 52 insertions(+) create mode 100644 surfsense_backend/app/capabilities/__init__.py create mode 100644 surfsense_backend/app/capabilities/types.py diff --git a/surfsense_backend/app/capabilities/__init__.py b/surfsense_backend/app/capabilities/__init__.py new file mode 100644 index 000000000..d159536fb --- /dev/null +++ b/surfsense_backend/app/capabilities/__init__.py @@ -0,0 +1,5 @@ +"""Scraper capability registry — typed, stateless verbs. See plans/backend/04-capabilities.md.""" + +from __future__ import annotations + +__all__: list[str] = [] diff --git a/surfsense_backend/app/capabilities/types.py b/surfsense_backend/app/capabilities/types.py new file mode 100644 index 000000000..0a9c261e8 --- /dev/null +++ b/surfsense_backend/app/capabilities/types.py @@ -0,0 +1,47 @@ +"""``Capability`` registry contracts shared by every verb.""" + +from __future__ import annotations + +from collections.abc import Awaitable, Callable +from dataclasses import dataclass +from enum import StrEnum +from typing import TYPE_CHECKING, Any, Protocol + +if TYPE_CHECKING: + from pydantic import BaseModel + from sqlalchemy.ext.asyncio import AsyncSession + + +class BillingUnit(StrEnum): + """The meter a verb charges on (priced by the billing service, 03c).""" + + WEB_CRAWL = "web_crawl" + + +class BillableOutput(Protocol): + """A capability output that reports its own billable count.""" + + @property + def billable_units(self) -> int: ... + + +@dataclass(frozen=True) +class CapabilityContext: + """Request-scoped deps a capability call needs beyond its typed input.""" + + session: AsyncSession + workspace_id: int + + +Executor = Callable[[Any], Awaitable[Any]] + + +@dataclass(frozen=True) +class Capability: + """One typed verb; the source of truth the doors (05) and agent (07) read.""" + + name: str + input_schema: type[BaseModel] + output_schema: type[BaseModel] + executor: Executor + billing_unit: BillingUnit From 3e6007431d401dfa2df5a3cfaeaa014467f2b109 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 16:42:47 +0200 Subject: [PATCH 03/64] feat(capabilities): add in-process verb store --- surfsense_backend/app/capabilities/store.py | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 surfsense_backend/app/capabilities/store.py diff --git a/surfsense_backend/app/capabilities/store.py b/surfsense_backend/app/capabilities/store.py new file mode 100644 index 000000000..b828480da --- /dev/null +++ b/surfsense_backend/app/capabilities/store.py @@ -0,0 +1,20 @@ +"""In-process capability registry, populated at import by each verb's ``definition.py``.""" + +from __future__ import annotations + +from app.capabilities.types import Capability + +_REGISTRY: dict[str, Capability] = {} + + +def register_capability(capability: Capability) -> None: + """Add (or replace) a verb by name.""" + _REGISTRY[capability.name] = capability + + +def get_capability(name: str) -> Capability: + return _REGISTRY[name] + + +def all_capabilities() -> list[Capability]: + return list(_REGISTRY.values()) From bfe3117302c37ebc288dd3aceb2e23c992b0fef0 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 16:42:47 +0200 Subject: [PATCH 04/64] feat(capabilities): bill workspace owner at executor --- surfsense_backend/app/capabilities/billing.py | 58 +++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 surfsense_backend/app/capabilities/billing.py diff --git a/surfsense_backend/app/capabilities/billing.py b/surfsense_backend/app/capabilities/billing.py new file mode 100644 index 000000000..3086c09bd --- /dev/null +++ b/surfsense_backend/app/capabilities/billing.py @@ -0,0 +1,58 @@ +"""Charge the workspace owner per billable success at the capability executor (03c).""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +from sqlalchemy import select + +from app.capabilities.types import BillableOutput, BillingUnit, CapabilityContext +from app.services.token_tracking_service import record_token_usage +from app.services.web_crawl_credit_service import WebCrawlCreditService + +if TYPE_CHECKING: + from uuid import UUID + + from sqlalchemy.ext.asyncio import AsyncSession + + +async def charge_capability( + output: BillableOutput, unit: BillingUnit, ctx: CapabilityContext +) -> None: + """Bill the workspace owner for this result's billable successes (03c).""" + units = output.billable_units + if units <= 0: + return + if unit is BillingUnit.WEB_CRAWL: + await _charge_web_crawl(ctx, units) + + +async def _charge_web_crawl(ctx: CapabilityContext, successes: int) -> None: + service = WebCrawlCreditService(ctx.session) + if not service.billing_enabled(): + return + owner_user_id = await _resolve_workspace_owner(ctx.session, ctx.workspace_id) + if owner_user_id is None: + return + # Stage the audit row before charge_credits' commit flushes both. + await record_token_usage( + ctx.session, + usage_type="web_crawl", + workspace_id=ctx.workspace_id, + user_id=owner_user_id, + cost_micros=service.successes_to_micros(successes), + call_details={"successes": successes}, + ) + await service.charge_credits(owner_user_id, successes) + + +async def _resolve_workspace_owner( + session: AsyncSession, workspace_id: int +) -> UUID | None: + """The ``user_id`` that owns ``workspace_id`` (the crawl payer, not the caller).""" + from app.db import Workspace + + result = await session.execute( + select(Workspace.user_id).where(Workspace.id == workspace_id) + ) + return result.scalar_one_or_none() From bbd81bcc99564693103f9813a3a72c7805c7d0ae Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 16:42:47 +0200 Subject: [PATCH 05/64] feat(capabilities): add web.scrape schemas --- .../app/capabilities/web/__init__.py | 5 ++++ .../app/capabilities/web/scrape/__init__.py | 3 ++ .../app/capabilities/web/scrape/schemas.py | 29 +++++++++++++++++++ 3 files changed, 37 insertions(+) create mode 100644 surfsense_backend/app/capabilities/web/__init__.py create mode 100644 surfsense_backend/app/capabilities/web/scrape/__init__.py create mode 100644 surfsense_backend/app/capabilities/web/scrape/schemas.py diff --git a/surfsense_backend/app/capabilities/web/__init__.py b/surfsense_backend/app/capabilities/web/__init__.py new file mode 100644 index 000000000..d6df4107d --- /dev/null +++ b/surfsense_backend/app/capabilities/web/__init__.py @@ -0,0 +1,5 @@ +"""``web.*`` namespace: the generic web scraping verbs.""" + +from __future__ import annotations + +from app.capabilities.web.scrape import definition # noqa: F401 — registers web.scrape diff --git a/surfsense_backend/app/capabilities/web/scrape/__init__.py b/surfsense_backend/app/capabilities/web/scrape/__init__.py new file mode 100644 index 000000000..0a21b156e --- /dev/null +++ b/surfsense_backend/app/capabilities/web/scrape/__init__.py @@ -0,0 +1,3 @@ +"""``web.scrape`` verb: a URL array → one cleaned row per URL.""" + +from __future__ import annotations diff --git a/surfsense_backend/app/capabilities/web/scrape/schemas.py b/surfsense_backend/app/capabilities/web/scrape/schemas.py new file mode 100644 index 000000000..3e77dd56b --- /dev/null +++ b/surfsense_backend/app/capabilities/web/scrape/schemas.py @@ -0,0 +1,29 @@ +"""``web.scrape`` I/O contracts.""" + +from __future__ import annotations + +from typing import Literal + +from pydantic import BaseModel + + +class ScrapeInput(BaseModel): + urls: list[str] + max_length: int = 50_000 + + +class ScrapeRow(BaseModel): + url: str + status: Literal["success", "empty", "failed"] + content: str | None = None + metadata: dict[str, str] | None = None + error: str | None = None + + +class ScrapeOutput(BaseModel): + rows: list[ScrapeRow] + + @property + def billable_units(self) -> int: + """One billable unit per successful scrape.""" + return sum(1 for row in self.rows if row.status == "success") From 6679fda6624a3b22e80dbbc4644b100ea96d131e Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 16:42:47 +0200 Subject: [PATCH 06/64] feat(capabilities): wire web.scrape to proprietary crawler --- .../app/capabilities/web/scrape/executor.py | 34 +++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 surfsense_backend/app/capabilities/web/scrape/executor.py diff --git a/surfsense_backend/app/capabilities/web/scrape/executor.py b/surfsense_backend/app/capabilities/web/scrape/executor.py new file mode 100644 index 000000000..46f067b56 --- /dev/null +++ b/surfsense_backend/app/capabilities/web/scrape/executor.py @@ -0,0 +1,34 @@ +"""``web.scrape`` executor: fetch each URL in the array → cleaned rows.""" + +from __future__ import annotations + +from app.capabilities.types import Executor +from app.capabilities.web.scrape.schemas import ScrapeInput, ScrapeOutput, ScrapeRow +from app.proprietary.web_crawler import ( + CrawlOutcome, + CrawlOutcomeStatus, + WebCrawlerConnector, +) + + +def build_scrape_executor(engine: WebCrawlerConnector | None = None) -> Executor: + """Bind the executor to a fetch engine (defaults to the proprietary crawler).""" + engine = engine or WebCrawlerConnector() + + async def execute(payload: ScrapeInput) -> ScrapeOutput: + rows = [_to_row(url, await engine.crawl_url(url)) for url in payload.urls] + return ScrapeOutput(rows=rows) + + return execute + + +def _to_row(url: str, outcome: CrawlOutcome) -> ScrapeRow: + if outcome.status is CrawlOutcomeStatus.SUCCESS and outcome.result: + return ScrapeRow( + url=url, + status="success", + content=outcome.result.get("content"), + metadata=outcome.result.get("metadata"), + ) + status = "empty" if outcome.status is CrawlOutcomeStatus.EMPTY else "failed" + return ScrapeRow(url=url, status=status, error=outcome.error) From 6b91a7ed0c3bfc31ae50514dd2057a44068c63dd Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 16:42:47 +0200 Subject: [PATCH 07/64] feat(capabilities): register web.scrape verb --- .../app/capabilities/web/scrape/definition.py | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 surfsense_backend/app/capabilities/web/scrape/definition.py diff --git a/surfsense_backend/app/capabilities/web/scrape/definition.py b/surfsense_backend/app/capabilities/web/scrape/definition.py new file mode 100644 index 000000000..295795066 --- /dev/null +++ b/surfsense_backend/app/capabilities/web/scrape/definition.py @@ -0,0 +1,18 @@ +"""``web.scrape`` capability registration.""" + +from __future__ import annotations + +from app.capabilities.store import register_capability +from app.capabilities.types import BillingUnit, Capability +from app.capabilities.web.scrape.executor import build_scrape_executor +from app.capabilities.web.scrape.schemas import ScrapeInput, ScrapeOutput + +WEB_SCRAPE = Capability( + name="web.scrape", + input_schema=ScrapeInput, + output_schema=ScrapeOutput, + executor=build_scrape_executor(), + billing_unit=BillingUnit.WEB_CRAWL, +) + +register_capability(WEB_SCRAPE) From 8d76afc9d64422a495522919aff5db482aa6e895 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 16:42:47 +0200 Subject: [PATCH 08/64] test(capabilities): cover registry, executor, schemas, billing --- .../tests/unit/capabilities/test_billing.py | 116 ++++++++++++++++++ .../tests/unit/capabilities/test_registry.py | 23 ++++ .../capabilities/web/scrape/test_executor.py | 94 ++++++++++++++ .../capabilities/web/scrape/test_schemas.py | 26 ++++ 4 files changed, 259 insertions(+) create mode 100644 surfsense_backend/tests/unit/capabilities/test_billing.py create mode 100644 surfsense_backend/tests/unit/capabilities/test_registry.py create mode 100644 surfsense_backend/tests/unit/capabilities/web/scrape/test_executor.py create mode 100644 surfsense_backend/tests/unit/capabilities/web/scrape/test_schemas.py diff --git a/surfsense_backend/tests/unit/capabilities/test_billing.py b/surfsense_backend/tests/unit/capabilities/test_billing.py new file mode 100644 index 000000000..53600c9c7 --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/test_billing.py @@ -0,0 +1,116 @@ +"""Billing charges the workspace owner once per billable success at the executor (03c). + +Boundaries mocked: the DB session and the audit helper. NOT mocked: the real +WebCrawlCreditService debit math and the owner-billed decision. +""" + +from __future__ import annotations + +from unittest.mock import AsyncMock, MagicMock +from uuid import UUID + +import pytest + +import app.capabilities.billing as billing +from app.capabilities.billing import charge_capability +from app.capabilities.types import BillingUnit, CapabilityContext +from app.capabilities.web.scrape.schemas import ScrapeOutput, ScrapeRow +from app.config import config + +pytestmark = pytest.mark.unit + +_WORKSPACE_ID = 1 +_OWNER = UUID("00000000-0000-0000-0000-0000000000bb") + + +class _FakeUser: + def __init__(self, balance_micros: int, reserved_micros: int = 0): + self.credit_micros_balance = balance_micros + self.credit_micros_reserved = reserved_micros + + +def _make_session(owner_id, balance_micros): + """Mock session serving owner-resolution and the charge_credits debit.""" + fake_user = _FakeUser(balance_micros) + session = AsyncMock() + session.add = MagicMock() + + def _make_result(*_args, **_kwargs): + result = MagicMock() + result.scalar_one_or_none.return_value = owner_id # owner resolution + result.unique.return_value.scalar_one_or_none.return_value = fake_user # debit + return result + + session.execute = AsyncMock(side_effect=_make_result) + return session, fake_user + + +def _output(*statuses: str) -> ScrapeOutput: + return ScrapeOutput( + rows=[ + ScrapeRow(url=f"https://{i}.com", status=status) + for i, status in enumerate(statuses) + ] + ) + + +def _ctx(session) -> CapabilityContext: + return CapabilityContext(session=session, workspace_id=_WORKSPACE_ID) + + +@pytest.fixture(autouse=True) +def _stub_auto_reload(monkeypatch): + import app.services.auto_reload_service as ar + + monkeypatch.setattr(ar, "maybe_trigger_auto_reload", AsyncMock()) + + +@pytest.fixture +def record_usage(monkeypatch): + rec = AsyncMock(return_value=MagicMock()) + monkeypatch.setattr(billing, "record_token_usage", rec) + return rec + + +async def test_charges_workspace_owner_per_successful_crawl(monkeypatch, record_usage): + monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", True) + monkeypatch.setattr(config, "WEB_CRAWL_MICROS_PER_SUCCESS", 1000) + session, user = _make_session(_OWNER, balance_micros=100_000) + + await charge_capability( + _output("success", "empty", "success"), BillingUnit.WEB_CRAWL, _ctx(session) + ) + + # Owner debited 2 * 1000; one web_crawl audit row billed to the OWNER. + assert user.credit_micros_balance == 100_000 - 2000 + record_usage.assert_awaited_once() + kwargs = record_usage.await_args.kwargs + assert kwargs["usage_type"] == "web_crawl" + assert kwargs["user_id"] == _OWNER + assert kwargs["workspace_id"] == _WORKSPACE_ID + assert kwargs["cost_micros"] == 2000 + + +async def test_no_successful_rows_is_free(monkeypatch, record_usage): + monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", True) + session, user = _make_session(_OWNER, balance_micros=100_000) + + await charge_capability( + _output("empty", "failed"), BillingUnit.WEB_CRAWL, _ctx(session) + ) + + record_usage.assert_not_awaited() + assert user.credit_micros_balance == 100_000 + + +async def test_disabled_is_noop(monkeypatch, record_usage): + monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", False) + session, user = _make_session(_OWNER, balance_micros=100_000) + + await charge_capability( + _output("success", "success"), BillingUnit.WEB_CRAWL, _ctx(session) + ) + + record_usage.assert_not_awaited() + session.execute.assert_not_called() + assert user.credit_micros_balance == 100_000 diff --git a/surfsense_backend/tests/unit/capabilities/test_registry.py b/surfsense_backend/tests/unit/capabilities/test_registry.py new file mode 100644 index 000000000..d0c6a08c4 --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/test_registry.py @@ -0,0 +1,23 @@ +"""The registry exposes each verb as one Capability entry the doors/agent read from.""" + +from __future__ import annotations + +import pytest + +from app.capabilities import ( + web, # noqa: F401 — importing the namespace registers its verbs +) +from app.capabilities.store import get_capability +from app.capabilities.types import BillingUnit +from app.capabilities.web.scrape.schemas import ScrapeInput, ScrapeOutput + +pytestmark = pytest.mark.unit + + +def test_web_scrape_is_registered_with_its_schemas_and_billing_unit(): + cap = get_capability("web.scrape") + + assert cap.name == "web.scrape" + assert cap.input_schema is ScrapeInput + assert cap.output_schema is ScrapeOutput + assert cap.billing_unit is BillingUnit.WEB_CRAWL diff --git a/surfsense_backend/tests/unit/capabilities/web/scrape/test_executor.py b/surfsense_backend/tests/unit/capabilities/web/scrape/test_executor.py new file mode 100644 index 000000000..9118b6944 --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/web/scrape/test_executor.py @@ -0,0 +1,94 @@ +"""`web.scrape` executor behavior: URLs in → cleaned rows out. + +Boundary mocked: the crawler (injected fake). NOT mocked: the executor's own +CrawlOutcome → ScrapeRow mapping. +""" + +from __future__ import annotations + +import pytest + +from app.capabilities.web.scrape.executor import build_scrape_executor +from app.capabilities.web.scrape.schemas import ScrapeInput, ScrapeOutput +from app.proprietary.web_crawler import CrawlOutcome, CrawlOutcomeStatus + +pytestmark = pytest.mark.unit + + +class _FakeCrawler: + """Stand-in for WebCrawlerConnector: serves a canned outcome per URL.""" + + def __init__(self, outcomes: dict[str, CrawlOutcome]): + self._outcomes = outcomes + self.calls: list[str] = [] + + async def crawl_url(self, url: str) -> CrawlOutcome: + self.calls.append(url) + return self._outcomes[url] + + +def _success(content: str, metadata: dict[str, str]) -> CrawlOutcome: + return CrawlOutcome( + status=CrawlOutcomeStatus.SUCCESS, + result={ + "content": content, + "metadata": metadata, + "crawler_type": "scrapling-static", + }, + ) + + +async def test_scrape_returns_one_cleaned_row_for_a_successful_url(): + url = "https://example.com" + crawler = _FakeCrawler({url: _success("# Hello", {"title": "Hello"})}) + execute = build_scrape_executor(engine=crawler) + + out = await execute(ScrapeInput(urls=[url])) + + assert isinstance(out, ScrapeOutput) + assert len(out.rows) == 1 + row = out.rows[0] + assert row.url == url + assert row.status == "success" + assert row.content == "# Hello" + assert row.metadata == {"title": "Hello"} + + +async def test_scrape_returns_one_row_per_url_in_input_order(): + a, b, c = "https://a.com", "https://b.com", "https://c.com" + crawler = _FakeCrawler( + { + a: _success("A", {"title": "A"}), + b: _success("B", {"title": "B"}), + c: _success("C", {"title": "C"}), + } + ) + execute = build_scrape_executor(engine=crawler) + + out = await execute(ScrapeInput(urls=[a, b, c])) + + assert [row.url for row in out.rows] == [a, b, c] + assert [row.content for row in out.rows] == ["A", "B", "C"] + + +async def test_partial_failure_keeps_the_batch_and_labels_each_url(): + ok, empty, failed = "https://ok.com", "https://empty.com", "https://failed.com" + crawler = _FakeCrawler( + { + ok: _success("OK", {"title": "OK"}), + empty: CrawlOutcome(status=CrawlOutcomeStatus.EMPTY, error="no content"), + failed: CrawlOutcome(status=CrawlOutcomeStatus.FAILED, error="blocked"), + } + ) + execute = build_scrape_executor(engine=crawler) + + out = await execute(ScrapeInput(urls=[ok, empty, failed])) + + by_url = {row.url: row for row in out.rows} + assert {u: r.status for u, r in by_url.items()} == { + ok: "success", + empty: "empty", + failed: "failed", + } + assert by_url[failed].content is None + assert by_url[failed].error == "blocked" diff --git a/surfsense_backend/tests/unit/capabilities/web/scrape/test_schemas.py b/surfsense_backend/tests/unit/capabilities/web/scrape/test_schemas.py new file mode 100644 index 000000000..c8aa6c78a --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/web/scrape/test_schemas.py @@ -0,0 +1,26 @@ +"""ScrapeOutput reports its own billable count (a success single-sources it).""" + +from __future__ import annotations + +import pytest + +from app.capabilities.web.scrape.schemas import ScrapeOutput, ScrapeRow + +pytestmark = pytest.mark.unit + + +def _output(*statuses: str) -> ScrapeOutput: + return ScrapeOutput( + rows=[ + ScrapeRow(url=f"https://{i}.com", status=status) + for i, status in enumerate(statuses) + ] + ) + + +def test_billable_units_counts_successful_rows(): + assert _output("success", "empty", "success", "failed").billable_units == 2 + + +def test_billable_units_is_zero_without_successes(): + assert _output("empty", "failed").billable_units == 0 From a413539f6ac2185b214122e4c4eed5d20d1f661d Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 17:08:58 +0200 Subject: [PATCH 09/64] feat(capabilities): allow free verbs with no billing unit --- surfsense_backend/app/capabilities/billing.py | 6 ++++-- surfsense_backend/app/capabilities/types.py | 4 ++-- .../tests/unit/capabilities/test_billing.py | 11 +++++++++++ 3 files changed, 17 insertions(+), 4 deletions(-) diff --git a/surfsense_backend/app/capabilities/billing.py b/surfsense_backend/app/capabilities/billing.py index 3086c09bd..ed1b00e13 100644 --- a/surfsense_backend/app/capabilities/billing.py +++ b/surfsense_backend/app/capabilities/billing.py @@ -17,9 +17,11 @@ if TYPE_CHECKING: async def charge_capability( - output: BillableOutput, unit: BillingUnit, ctx: CapabilityContext + output: BillableOutput, unit: BillingUnit | None, ctx: CapabilityContext ) -> None: - """Bill the workspace owner for this result's billable successes (03c).""" + """Bill the workspace owner for this result's billable successes (03c). ``None`` = free.""" + if unit is None: + return units = output.billable_units if units <= 0: return diff --git a/surfsense_backend/app/capabilities/types.py b/surfsense_backend/app/capabilities/types.py index 0a9c261e8..6cb69c312 100644 --- a/surfsense_backend/app/capabilities/types.py +++ b/surfsense_backend/app/capabilities/types.py @@ -13,7 +13,7 @@ if TYPE_CHECKING: class BillingUnit(StrEnum): - """The meter a verb charges on (priced by the billing service, 03c).""" + """The meter a verb charges on (priced by the billing service, 03c). ``None`` = free.""" WEB_CRAWL = "web_crawl" @@ -44,4 +44,4 @@ class Capability: input_schema: type[BaseModel] output_schema: type[BaseModel] executor: Executor - billing_unit: BillingUnit + billing_unit: BillingUnit | None diff --git a/surfsense_backend/tests/unit/capabilities/test_billing.py b/surfsense_backend/tests/unit/capabilities/test_billing.py index 53600c9c7..210e69784 100644 --- a/surfsense_backend/tests/unit/capabilities/test_billing.py +++ b/surfsense_backend/tests/unit/capabilities/test_billing.py @@ -114,3 +114,14 @@ async def test_disabled_is_noop(monkeypatch, record_usage): record_usage.assert_not_awaited() session.execute.assert_not_called() assert user.credit_micros_balance == 100_000 + + +async def test_free_verb_without_a_unit_is_noop(monkeypatch, record_usage): + monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", True) + session, user = _make_session(_OWNER, balance_micros=100_000) + + await charge_capability(_output("success", "success"), None, _ctx(session)) + + record_usage.assert_not_awaited() + session.execute.assert_not_called() + assert user.credit_micros_balance == 100_000 From fa1055dd4ce832280c0049d8865a1b486d86215a Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 17:08:58 +0200 Subject: [PATCH 10/64] feat(capabilities): add web.discover schemas and provider seam --- .../app/capabilities/web/discover/__init__.py | 3 + .../app/capabilities/web/discover/executor.py | 40 ++++++++++ .../web/discover/providers/__init__.py | 3 + .../web/discover/providers/base.py | 22 ++++++ .../app/capabilities/web/discover/schemas.py | 21 +++++ .../web/discover/test_executor.py | 79 +++++++++++++++++++ 6 files changed, 168 insertions(+) create mode 100644 surfsense_backend/app/capabilities/web/discover/__init__.py create mode 100644 surfsense_backend/app/capabilities/web/discover/executor.py create mode 100644 surfsense_backend/app/capabilities/web/discover/providers/__init__.py create mode 100644 surfsense_backend/app/capabilities/web/discover/providers/base.py create mode 100644 surfsense_backend/app/capabilities/web/discover/schemas.py create mode 100644 surfsense_backend/tests/unit/capabilities/web/discover/test_executor.py diff --git a/surfsense_backend/app/capabilities/web/discover/__init__.py b/surfsense_backend/app/capabilities/web/discover/__init__.py new file mode 100644 index 000000000..37b153535 --- /dev/null +++ b/surfsense_backend/app/capabilities/web/discover/__init__.py @@ -0,0 +1,3 @@ +"""``web.discover`` verb: a query → candidate URLs the agent can feed to ``web.scrape``.""" + +from __future__ import annotations diff --git a/surfsense_backend/app/capabilities/web/discover/executor.py b/surfsense_backend/app/capabilities/web/discover/executor.py new file mode 100644 index 000000000..85f6bf83b --- /dev/null +++ b/surfsense_backend/app/capabilities/web/discover/executor.py @@ -0,0 +1,40 @@ +"""``web.discover`` executor: route a query to the first configured search provider.""" + +from __future__ import annotations + +from collections.abc import Sequence + +from app.capabilities.types import Executor +from app.capabilities.web.discover.providers.base import DiscoverProvider +from app.capabilities.web.discover.schemas import DiscoverInput, DiscoverOutput + + +class NoDiscoverProviderError(RuntimeError): + """Raised when no search provider is configured (no platform key/host set).""" + + +def build_discover_executor( + providers: Sequence[DiscoverProvider] | None = None, +) -> Executor: + """Bind the executor to a provider set (defaults to the real env-keyed providers).""" + registry = list(providers) if providers is not None else _default_providers() + + async def execute(payload: DiscoverInput) -> DiscoverOutput: + provider = next((p for p in registry if p.is_available()), None) + if provider is None: + raise NoDiscoverProviderError( + "web.discover has no configured search provider " + "(set a SearXNG host or a Linkup/Baidu key)." + ) + hits = await provider.search(payload.query, payload.top_k) + return DiscoverOutput(hits=hits) + + return execute + + +def _default_providers() -> list[DiscoverProvider]: + from app.capabilities.web.discover.providers.baidu import BaiduProvider + from app.capabilities.web.discover.providers.linkup import LinkupProvider + from app.capabilities.web.discover.providers.searxng import SearxngProvider + + return [SearxngProvider(), LinkupProvider(), BaiduProvider()] diff --git a/surfsense_backend/app/capabilities/web/discover/providers/__init__.py b/surfsense_backend/app/capabilities/web/discover/providers/__init__.py new file mode 100644 index 000000000..68ca2b374 --- /dev/null +++ b/surfsense_backend/app/capabilities/web/discover/providers/__init__.py @@ -0,0 +1,3 @@ +"""``web.discover`` search providers (env-keyed: SearXNG, Linkup, Baidu).""" + +from __future__ import annotations diff --git a/surfsense_backend/app/capabilities/web/discover/providers/base.py b/surfsense_backend/app/capabilities/web/discover/providers/base.py new file mode 100644 index 000000000..c245072eb --- /dev/null +++ b/surfsense_backend/app/capabilities/web/discover/providers/base.py @@ -0,0 +1,22 @@ +"""The seam every ``web.discover`` provider implements.""" + +from __future__ import annotations + +from typing import Protocol, runtime_checkable + +from app.capabilities.web.discover.schemas import DiscoverHit + + +@runtime_checkable +class DiscoverProvider(Protocol): + """An env-keyed search backend that suggests candidate URLs for a query.""" + + name: str + + def is_available(self) -> bool: + """True when this provider's platform key/host is configured.""" + ... + + async def search(self, query: str, top_k: int) -> list[DiscoverHit]: + """Return up to ``top_k`` candidate hits for ``query``.""" + ... diff --git a/surfsense_backend/app/capabilities/web/discover/schemas.py b/surfsense_backend/app/capabilities/web/discover/schemas.py new file mode 100644 index 000000000..77622b210 --- /dev/null +++ b/surfsense_backend/app/capabilities/web/discover/schemas.py @@ -0,0 +1,21 @@ +"""``web.discover`` I/O contracts.""" + +from __future__ import annotations + +from pydantic import BaseModel + + +class DiscoverInput(BaseModel): + query: str + top_k: int = 10 + + +class DiscoverHit(BaseModel): + url: str + title: str + snippet: str | None = None + provider: str + + +class DiscoverOutput(BaseModel): + hits: list[DiscoverHit] diff --git a/surfsense_backend/tests/unit/capabilities/web/discover/test_executor.py b/surfsense_backend/tests/unit/capabilities/web/discover/test_executor.py new file mode 100644 index 000000000..5d50d5f6f --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/web/discover/test_executor.py @@ -0,0 +1,79 @@ +"""`web.discover` executor: pick the first configured provider; self-disable when none. + +Boundary mocked: the providers (injected fakes). NOT mocked: the executor's +provider-selection and self-disable behavior. +""" + +from __future__ import annotations + +import pytest + +from app.capabilities.web.discover.executor import ( + NoDiscoverProviderError, + build_discover_executor, +) +from app.capabilities.web.discover.schemas import ( + DiscoverHit, + DiscoverInput, + DiscoverOutput, +) + +pytestmark = pytest.mark.unit + + +class _FakeProvider: + def __init__( + self, name: str, available: bool, hits: list[DiscoverHit] | None = None + ): + self.name = name + self._available = available + self._hits = hits or [] + self.calls: list[tuple[str, int]] = [] + + def is_available(self) -> bool: + return self._available + + async def search(self, query: str, top_k: int) -> list[DiscoverHit]: + self.calls.append((query, top_k)) + return self._hits + + +def _hit(url: str, provider: str) -> DiscoverHit: + return DiscoverHit(url=url, title=url, snippet="s", provider=provider) + + +async def test_uses_the_first_available_provider(): + first = _FakeProvider( + "searxng", available=True, hits=[_hit("https://a.com", "searxng")] + ) + second = _FakeProvider( + "linkup", available=True, hits=[_hit("https://b.com", "linkup")] + ) + execute = build_discover_executor(providers=[first, second]) + + out = await execute(DiscoverInput(query="acme pricing", top_k=5)) + + assert isinstance(out, DiscoverOutput) + assert [h.url for h in out.hits] == ["https://a.com"] + assert first.calls == [("acme pricing", 5)] + assert second.calls == [] # first available short-circuits + + +async def test_skips_unavailable_providers(): + off = _FakeProvider("searxng", available=False) + on = _FakeProvider("linkup", available=True, hits=[_hit("https://b.com", "linkup")]) + execute = build_discover_executor(providers=[off, on]) + + out = await execute(DiscoverInput(query="q")) + + assert [h.provider for h in out.hits] == ["linkup"] + assert off.calls == [] + + +async def test_self_disables_when_no_provider_is_configured(): + execute = build_discover_executor( + providers=[_FakeProvider("searxng", available=False)] + ) + + with pytest.raises(NoDiscoverProviderError): + await execute(DiscoverInput(query="q")) From adc39d10622d1235ba309ee36707aeaa44fc1da1 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 17:08:58 +0200 Subject: [PATCH 11/64] feat(config): add env keys for discover search providers --- surfsense_backend/.env.example | 7 +++++++ surfsense_backend/app/config/__init__.py | 6 +++++- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/surfsense_backend/.env.example b/surfsense_backend/.env.example index c08a696a3..435782159 100644 --- a/surfsense_backend/.env.example +++ b/surfsense_backend/.env.example @@ -60,6 +60,13 @@ WHATSAPP_BRIDGE_URL=http://whatsapp-bridge:9929 # Only uncomment if running the backend outside Docker (e.g. uvicorn on host). # SEARXNG_DEFAULT_HOST=http://localhost:8888 +# web.discover fallback providers (env-keyed). SearXNG above is preferred; these +# are used when it is not configured. web.discover self-disables if none are set. +# LINKUP_API_KEY= +# BAIDU_API_KEY= +# BAIDU_MODEL=ernie-3.5-8k +# BAIDU_SEARCH_SOURCE=baidu_search_v2 + # Periodic task interval # # Run every minute (default) # SCHEDULE_CHECKER_INTERVAL=1m diff --git a/surfsense_backend/app/config/__init__.py b/surfsense_backend/app/config/__init__.py index 0065faa9a..bacfc3c38 100644 --- a/surfsense_backend/app/config/__init__.py +++ b/surfsense_backend/app/config/__init__.py @@ -554,8 +554,12 @@ class Config: os.getenv("SURFSENSE_CONNECTOR_DISCOVERY_TTL_SECONDS", "30") ) - # Platform web search (SearXNG) + # Platform web search providers for the web.discover capability (env-keyed). SEARXNG_DEFAULT_HOST = os.getenv("SEARXNG_DEFAULT_HOST") + LINKUP_API_KEY = os.getenv("LINKUP_API_KEY") + BAIDU_API_KEY = os.getenv("BAIDU_API_KEY") + BAIDU_MODEL = os.getenv("BAIDU_MODEL", "ernie-3.5-8k") + BAIDU_SEARCH_SOURCE = os.getenv("BAIDU_SEARCH_SOURCE", "baidu_search_v2") SURFSENSE_PUBLIC_URL = os.getenv("SURFSENSE_PUBLIC_URL") NEXT_FRONTEND_URL = os.getenv("NEXT_FRONTEND_URL") or SURFSENSE_PUBLIC_URL From 329bc63b6c9340aefdd7d5608e8cc2a5dae7cc91 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 17:08:58 +0200 Subject: [PATCH 12/64] feat(capabilities): add env-keyed searxng/linkup/baidu discover providers --- .../web/discover/providers/baidu.py | 53 +++++++++++ .../web/discover/providers/linkup.py | 40 ++++++++ .../web/discover/providers/searxng.py | 28 ++++++ .../web/discover/test_baidu_provider.py | 91 +++++++++++++++++++ .../web/discover/test_linkup_provider.py | 75 +++++++++++++++ .../web/discover/test_searxng_provider.py | 58 ++++++++++++ 6 files changed, 345 insertions(+) create mode 100644 surfsense_backend/app/capabilities/web/discover/providers/baidu.py create mode 100644 surfsense_backend/app/capabilities/web/discover/providers/linkup.py create mode 100644 surfsense_backend/app/capabilities/web/discover/providers/searxng.py create mode 100644 surfsense_backend/tests/unit/capabilities/web/discover/test_baidu_provider.py create mode 100644 surfsense_backend/tests/unit/capabilities/web/discover/test_linkup_provider.py create mode 100644 surfsense_backend/tests/unit/capabilities/web/discover/test_searxng_provider.py diff --git a/surfsense_backend/app/capabilities/web/discover/providers/baidu.py b/surfsense_backend/app/capabilities/web/discover/providers/baidu.py new file mode 100644 index 000000000..fc632a9ea --- /dev/null +++ b/surfsense_backend/app/capabilities/web/discover/providers/baidu.py @@ -0,0 +1,53 @@ +"""Baidu AI Search discover provider (env-keyed via ``BAIDU_API_KEY``).""" + +from __future__ import annotations + +import httpx + +from app.capabilities.web.discover.schemas import DiscoverHit +from app.config import config + +_ENDPOINT = "https://qianfan.baidubce.com/v2/ai_search/chat/completions" +_SNIPPET_MAX = 300 + + +class BaiduProvider: + name = "baidu" + + def is_available(self) -> bool: + return bool(config.BAIDU_API_KEY) + + async def search(self, query: str, top_k: int) -> list[DiscoverHit]: + max_per_type = min(top_k, 20) + payload = { + "messages": [{"role": "user", "content": query}], + "model": config.BAIDU_MODEL, + "search_source": config.BAIDU_SEARCH_SOURCE, + "resource_type_filter": [{"type": "web", "top_k": max_per_type}], + "stream": False, + } + headers = { + "X-Appbuilder-Authorization": f"Bearer {config.BAIDU_API_KEY}", + "Content-Type": "application/json", + } + async with httpx.AsyncClient(timeout=90.0) as client: + response = await client.post(_ENDPOINT, headers=headers, json=payload) + response.raise_for_status() + + hits: list[DiscoverHit] = [] + for reference in response.json().get("references", []): + url = reference.get("url", "") + if not url: + continue + content = reference.get("content", "") or "" + hits.append( + DiscoverHit( + url=url, + title=reference.get("title") or url, + snippet=content[:_SNIPPET_MAX] or None, + provider=self.name, + ) + ) + if len(hits) >= top_k: + break + return hits diff --git a/surfsense_backend/app/capabilities/web/discover/providers/linkup.py b/surfsense_backend/app/capabilities/web/discover/providers/linkup.py new file mode 100644 index 000000000..77393de08 --- /dev/null +++ b/surfsense_backend/app/capabilities/web/discover/providers/linkup.py @@ -0,0 +1,40 @@ +"""Linkup discover provider (env-keyed via ``LINKUP_API_KEY``).""" + +from __future__ import annotations + +import asyncio + +from linkup import LinkupClient + +from app.capabilities.web.discover.schemas import DiscoverHit +from app.config import config + + +class LinkupProvider: + name = "linkup" + + def is_available(self) -> bool: + return bool(config.LINKUP_API_KEY) + + async def search(self, query: str, top_k: int) -> list[DiscoverHit]: + client = LinkupClient(api_key=config.LINKUP_API_KEY) + response = await asyncio.to_thread( + client.search, query=query, depth="standard", output_type="searchResults" + ) + hits: list[DiscoverHit] = [] + for result in getattr(response, "results", None) or []: + url = getattr(result, "url", "") or "" + if not url: + continue + content = getattr(result, "content", "") or "" + hits.append( + DiscoverHit( + url=url, + title=getattr(result, "name", "") or url, + snippet=content or None, + provider=self.name, + ) + ) + if len(hits) >= top_k: + break + return hits diff --git a/surfsense_backend/app/capabilities/web/discover/providers/searxng.py b/surfsense_backend/app/capabilities/web/discover/providers/searxng.py new file mode 100644 index 000000000..92489862a --- /dev/null +++ b/surfsense_backend/app/capabilities/web/discover/providers/searxng.py @@ -0,0 +1,28 @@ +"""SearXNG discover provider, wrapping the platform-env SearXNG search service.""" + +from __future__ import annotations + +from app.capabilities.web.discover.schemas import DiscoverHit +from app.services import web_search_service + + +class SearxngProvider: + """Env-keyed via ``SEARXNG_DEFAULT_HOST`` (the platform SearXNG instance).""" + + name = "searxng" + + def is_available(self) -> bool: + return web_search_service.is_available() + + async def search(self, query: str, top_k: int) -> list[DiscoverHit]: + result_object, _documents = await web_search_service.search(query, top_k) + return [ + DiscoverHit( + url=source["url"], + title=source.get("title") or source["url"], + snippet=source.get("description") or None, + provider=self.name, + ) + for source in result_object.get("sources", []) + if source.get("url") + ] diff --git a/surfsense_backend/tests/unit/capabilities/web/discover/test_baidu_provider.py b/surfsense_backend/tests/unit/capabilities/web/discover/test_baidu_provider.py new file mode 100644 index 000000000..dc3035da7 --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/web/discover/test_baidu_provider.py @@ -0,0 +1,91 @@ +"""BaiduProvider maps the Baidu AI Search references to DiscoverHits, env-keyed. + +Boundary mocked: httpx.AsyncClient + config key. NOT mocked: reference→hit mapping. +""" + +from __future__ import annotations + +import pytest + +import app.capabilities.web.discover.providers.baidu as baidu_module +from app.capabilities.web.discover.providers.baidu import BaiduProvider +from app.config import config + +pytestmark = pytest.mark.unit + + +class _FakeResponse: + def __init__(self, payload): + self._payload = payload + + def raise_for_status(self): + return None + + def json(self): + return self._payload + + +class _FakeAsyncClient: + """Minimal async-context httpx stand-in returning a canned payload.""" + + payload: dict = {} + + def __init__(self, *args, **kwargs): + pass + + async def __aenter__(self): + return self + + async def __aexit__(self, *exc): + return False + + async def post(self, url, headers=None, json=None): + return _FakeResponse(type(self).payload) + + +def _install(monkeypatch, payload): + monkeypatch.setattr(config, "BAIDU_API_KEY", "k") + _FakeAsyncClient.payload = payload + monkeypatch.setattr(baidu_module.httpx, "AsyncClient", _FakeAsyncClient) + + +def test_is_available_reflects_the_env_key(monkeypatch): + monkeypatch.setattr(config, "BAIDU_API_KEY", "k") + assert BaiduProvider().is_available() is True + monkeypatch.setattr(config, "BAIDU_API_KEY", None) + assert BaiduProvider().is_available() is False + + +async def test_maps_references_to_hits(monkeypatch): + _install( + monkeypatch, + { + "references": [ + {"title": "Acme", "url": "https://acme.cn", "content": "hello"}, + {"title": "No URL", "url": "", "content": "skip"}, + ] + }, + ) + + hits = await BaiduProvider().search("acme", top_k=10) + + assert [h.url for h in hits] == ["https://acme.cn"] + assert hits[0].title == "Acme" + assert hits[0].snippet == "hello" + assert hits[0].provider == "baidu" + + +async def test_respects_top_k(monkeypatch): + _install( + monkeypatch, + { + "references": [ + {"title": f"n{i}", "url": f"https://{i}.cn", "content": "c"} + for i in range(5) + ] + }, + ) + + hits = await BaiduProvider().search("q", top_k=2) + + assert len(hits) == 2 diff --git a/surfsense_backend/tests/unit/capabilities/web/discover/test_linkup_provider.py b/surfsense_backend/tests/unit/capabilities/web/discover/test_linkup_provider.py new file mode 100644 index 000000000..d846e0d94 --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/web/discover/test_linkup_provider.py @@ -0,0 +1,75 @@ +"""LinkupProvider maps the Linkup SDK results to DiscoverHits, env-keyed. + +Boundary mocked: the LinkupClient SDK + config key. NOT mocked: result→hit mapping. +""" + +from __future__ import annotations + +import pytest + +import app.capabilities.web.discover.providers.linkup as linkup_module +from app.capabilities.web.discover.providers.linkup import LinkupProvider +from app.config import config + +pytestmark = pytest.mark.unit + + +class _Result: + def __init__(self, name, url, content): + self.name = name + self.url = url + self.content = content + self.type = "text" + + +class _Response: + def __init__(self, results): + self.results = results + + +class _FakeClient: + def __init__(self, api_key): + self.api_key = api_key + + def search(self, *, query, depth, output_type): + return _Response( + [ + _Result("Acme", "https://acme.com", "acme home"), + _Result("No URL", "", "skip"), + ] + ) + + +def test_is_available_reflects_the_env_key(monkeypatch): + monkeypatch.setattr(config, "LINKUP_API_KEY", "k") + assert LinkupProvider().is_available() is True + monkeypatch.setattr(config, "LINKUP_API_KEY", None) + assert LinkupProvider().is_available() is False + + +async def test_maps_results_to_hits(monkeypatch): + monkeypatch.setattr(config, "LINKUP_API_KEY", "k") + monkeypatch.setattr(linkup_module, "LinkupClient", _FakeClient) + + hits = await LinkupProvider().search("acme", top_k=10) + + assert [h.url for h in hits] == ["https://acme.com"] + assert hits[0].title == "Acme" + assert hits[0].snippet == "acme home" + assert hits[0].provider == "linkup" + + +async def test_respects_top_k(monkeypatch): + monkeypatch.setattr(config, "LINKUP_API_KEY", "k") + + class _ManyClient(_FakeClient): + def search(self, *, query, depth, output_type): + return _Response( + [_Result(f"n{i}", f"https://{i}.com", "c") for i in range(5)] + ) + + monkeypatch.setattr(linkup_module, "LinkupClient", _ManyClient) + + hits = await LinkupProvider().search("q", top_k=2) + + assert len(hits) == 2 diff --git a/surfsense_backend/tests/unit/capabilities/web/discover/test_searxng_provider.py b/surfsense_backend/tests/unit/capabilities/web/discover/test_searxng_provider.py new file mode 100644 index 000000000..a184666b3 --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/web/discover/test_searxng_provider.py @@ -0,0 +1,58 @@ +"""SearxngProvider maps the SearXNG service's sources to DiscoverHits. + +Boundary mocked: the web_search_service module. NOT mocked: the source→hit mapping. +""" + +from __future__ import annotations + +from unittest.mock import AsyncMock + +import pytest + +import app.capabilities.web.discover.providers.searxng as searxng_module +from app.capabilities.web.discover.providers.searxng import SearxngProvider + +pytestmark = pytest.mark.unit + + +def _result(sources): + return ({"sources": sources}, []) + + +async def test_maps_sources_to_hits(monkeypatch): + provider = SearxngProvider() + monkeypatch.setattr( + searxng_module.web_search_service, + "search", + AsyncMock( + return_value=_result( + [ + {"title": "Acme", "url": "https://acme.com", "description": "home"}, + { + "title": "Docs", + "url": "https://acme.com/docs", + "description": "", + }, + {"title": "no url", "url": "", "description": "skip me"}, + ] + ) + ), + ) + + hits = await provider.search("acme", top_k=5) + + assert [h.url for h in hits] == ["https://acme.com", "https://acme.com/docs"] + assert hits[0].title == "Acme" + assert hits[0].snippet == "home" + assert hits[1].snippet is None # empty description normalizes to None + assert all(h.provider == "searxng" for h in hits) + + +def test_is_available_reflects_the_service(monkeypatch): + provider = SearxngProvider() + monkeypatch.setattr(searxng_module.web_search_service, "is_available", lambda: True) + assert provider.is_available() is True + monkeypatch.setattr( + searxng_module.web_search_service, "is_available", lambda: False + ) + assert provider.is_available() is False From f257e385dba0b870b8b24a5afad10e8116e44588 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 17:08:58 +0200 Subject: [PATCH 13/64] feat(capabilities): register web.discover verb --- .../app/capabilities/web/__init__.py | 5 +++-- .../capabilities/web/discover/definition.py | 18 ++++++++++++++++++ .../tests/unit/capabilities/test_registry.py | 10 ++++++++++ 3 files changed, 31 insertions(+), 2 deletions(-) create mode 100644 surfsense_backend/app/capabilities/web/discover/definition.py diff --git a/surfsense_backend/app/capabilities/web/__init__.py b/surfsense_backend/app/capabilities/web/__init__.py index d6df4107d..f3710d1cf 100644 --- a/surfsense_backend/app/capabilities/web/__init__.py +++ b/surfsense_backend/app/capabilities/web/__init__.py @@ -1,5 +1,6 @@ -"""``web.*`` namespace: the generic web scraping verbs.""" +"""``web.*`` namespace: the generic web scraping + discovery verbs.""" from __future__ import annotations -from app.capabilities.web.scrape import definition # noqa: F401 — registers web.scrape +from app.capabilities.web.discover import definition as _discover # noqa: F401 +from app.capabilities.web.scrape import definition as _scrape # noqa: F401 diff --git a/surfsense_backend/app/capabilities/web/discover/definition.py b/surfsense_backend/app/capabilities/web/discover/definition.py new file mode 100644 index 000000000..1c3b4ec28 --- /dev/null +++ b/surfsense_backend/app/capabilities/web/discover/definition.py @@ -0,0 +1,18 @@ +"""``web.discover`` capability registration (free — see 04-capabilities open item).""" + +from __future__ import annotations + +from app.capabilities.store import register_capability +from app.capabilities.types import Capability +from app.capabilities.web.discover.executor import build_discover_executor +from app.capabilities.web.discover.schemas import DiscoverInput, DiscoverOutput + +WEB_DISCOVER = Capability( + name="web.discover", + input_schema=DiscoverInput, + output_schema=DiscoverOutput, + executor=build_discover_executor(), + billing_unit=None, +) + +register_capability(WEB_DISCOVER) diff --git a/surfsense_backend/tests/unit/capabilities/test_registry.py b/surfsense_backend/tests/unit/capabilities/test_registry.py index d0c6a08c4..3b941224c 100644 --- a/surfsense_backend/tests/unit/capabilities/test_registry.py +++ b/surfsense_backend/tests/unit/capabilities/test_registry.py @@ -9,6 +9,7 @@ from app.capabilities import ( ) from app.capabilities.store import get_capability from app.capabilities.types import BillingUnit +from app.capabilities.web.discover.schemas import DiscoverInput, DiscoverOutput from app.capabilities.web.scrape.schemas import ScrapeInput, ScrapeOutput pytestmark = pytest.mark.unit @@ -21,3 +22,12 @@ def test_web_scrape_is_registered_with_its_schemas_and_billing_unit(): assert cap.input_schema is ScrapeInput assert cap.output_schema is ScrapeOutput assert cap.billing_unit is BillingUnit.WEB_CRAWL + + +def test_web_discover_is_registered_and_free(): + cap = get_capability("web.discover") + + assert cap.name == "web.discover" + assert cap.input_schema is DiscoverInput + assert cap.output_schema is DiscoverOutput + assert cap.billing_unit is None From 8fbfa5a6e1587030d302a31ee6621cb4821be995 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 17:53:42 +0200 Subject: [PATCH 14/64] feat(capabilities): add pre-flight meter-gate for billed verbs --- surfsense_backend/app/capabilities/billing.py | 31 ++++++++++++++++++- surfsense_backend/app/capabilities/types.py | 7 +++++ 2 files changed, 37 insertions(+), 1 deletion(-) diff --git a/surfsense_backend/app/capabilities/billing.py b/surfsense_backend/app/capabilities/billing.py index ed1b00e13..20d1059bf 100644 --- a/surfsense_backend/app/capabilities/billing.py +++ b/surfsense_backend/app/capabilities/billing.py @@ -6,7 +6,12 @@ from typing import TYPE_CHECKING from sqlalchemy import select -from app.capabilities.types import BillableOutput, BillingUnit, CapabilityContext +from app.capabilities.types import ( + BillableInput, + BillableOutput, + BillingUnit, + CapabilityContext, +) from app.services.token_tracking_service import record_token_usage from app.services.web_crawl_credit_service import WebCrawlCreditService @@ -16,6 +21,30 @@ if TYPE_CHECKING: from sqlalchemy.ext.asyncio import AsyncSession +async def gate_capability( + payload: BillableInput, unit: BillingUnit | None, ctx: CapabilityContext +) -> None: + """Pre-flight: block an over-budget owner before the executor runs (03c). + + Raises ``InsufficientCreditsError`` when the wallet can't cover the input's + worst-case ``estimated_units``. ``None`` unit = free = no gate. + """ + if unit is None: + return + if unit is BillingUnit.WEB_CRAWL: + await _gate_web_crawl(ctx, payload.estimated_units) + + +async def _gate_web_crawl(ctx: CapabilityContext, estimated_successes: int) -> None: + service = WebCrawlCreditService(ctx.session) + if not service.billing_enabled(): + return + owner_user_id = await _resolve_workspace_owner(ctx.session, ctx.workspace_id) + if owner_user_id is None: + return + await service.check_credits(owner_user_id, estimated_successes) + + async def charge_capability( output: BillableOutput, unit: BillingUnit | None, ctx: CapabilityContext ) -> None: diff --git a/surfsense_backend/app/capabilities/types.py b/surfsense_backend/app/capabilities/types.py index 6cb69c312..4a8b57cbf 100644 --- a/surfsense_backend/app/capabilities/types.py +++ b/surfsense_backend/app/capabilities/types.py @@ -18,6 +18,13 @@ class BillingUnit(StrEnum): WEB_CRAWL = "web_crawl" +class BillableInput(Protocol): + """A billed verb's input that reports its worst-case unit count for pre-flight.""" + + @property + def estimated_units(self) -> int: ... + + class BillableOutput(Protocol): """A capability output that reports its own billable count.""" From c696829568a8aec1c03f7529ad20be27ac881258 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 17:53:42 +0200 Subject: [PATCH 15/64] feat(capabilities): bound web.scrape batch and expose worst-case units --- .../app/capabilities/web/scrape/schemas.py | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/surfsense_backend/app/capabilities/web/scrape/schemas.py b/surfsense_backend/app/capabilities/web/scrape/schemas.py index 3e77dd56b..91de17a53 100644 --- a/surfsense_backend/app/capabilities/web/scrape/schemas.py +++ b/surfsense_backend/app/capabilities/web/scrape/schemas.py @@ -4,13 +4,21 @@ from __future__ import annotations from typing import Literal -from pydantic import BaseModel +from pydantic import BaseModel, Field + +MAX_SCRAPE_URLS = 20 +"""Per-call batch cap: bounds a synchronous request's crawl fan-out (05).""" class ScrapeInput(BaseModel): - urls: list[str] + urls: list[str] = Field(min_length=1, max_length=MAX_SCRAPE_URLS) max_length: int = 50_000 + @property + def estimated_units(self) -> int: + """Worst-case billable crawls for pre-flight: one per requested URL.""" + return len(self.urls) + class ScrapeRow(BaseModel): url: str From 9c9ad97f8e8933086def8111e89d3b1936838932 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 17:53:42 +0200 Subject: [PATCH 16/64] feat(capabilities): generate REST door with per-workspace rate limit --- .../app/capabilities/access/__init__.py | 1 + .../app/capabilities/access/rate_limit.py | 65 +++++++++++++++++ .../app/capabilities/access/rest.py | 69 +++++++++++++++++++ 3 files changed, 135 insertions(+) create mode 100644 surfsense_backend/app/capabilities/access/__init__.py create mode 100644 surfsense_backend/app/capabilities/access/rate_limit.py create mode 100644 surfsense_backend/app/capabilities/access/rest.py diff --git a/surfsense_backend/app/capabilities/access/__init__.py b/surfsense_backend/app/capabilities/access/__init__.py new file mode 100644 index 000000000..f1afc4ff5 --- /dev/null +++ b/surfsense_backend/app/capabilities/access/__init__.py @@ -0,0 +1 @@ +"""Access doors (05): thin adapters that expose the 04 registry as REST/MCP/chat.""" diff --git a/surfsense_backend/app/capabilities/access/rate_limit.py b/surfsense_backend/app/capabilities/access/rate_limit.py new file mode 100644 index 000000000..6de7676ba --- /dev/null +++ b/surfsense_backend/app/capabilities/access/rate_limit.py @@ -0,0 +1,65 @@ +"""Per-workspace rate limit for the capability doors (05). + +A secondary abuse guard; the credit meter-gate (03c) is the primary control. +Fixed-window over Redis (shared across workers) with a per-worker in-memory +fallback when Redis is unavailable — mirroring the auth-endpoint limiter. +""" + +from __future__ import annotations + +import time +from collections import defaultdict +from threading import Lock + +from fastapi import HTTPException, Request, status + +from app.config import config + +CAPABILITY_RATE_LIMIT_PER_MINUTE = 120 +_WINDOW_SECONDS = 60 +_KEY_PREFIX = "surfsense:capability_rate_limit" + +_redis = None +_memory: dict[str, list[float]] = defaultdict(list) +_memory_lock = Lock() + + +def _redis_client(): + global _redis + if _redis is None: + import redis + + _redis = redis.from_url(config.REDIS_APP_URL, decode_responses=True) + return _redis + + +def _incr_memory(key: str, window_seconds: int) -> int: + now = time.monotonic() + with _memory_lock: + hits = [t for t in _memory[key] if now - t < window_seconds] + hits.append(now) + _memory[key] = hits + return len(hits) + + +def _incr(key: str, window_seconds: int) -> int: + """Increment the window counter for ``key`` and return the new count.""" + try: + client = _redis_client() + count = int(client.incr(key)) + if count == 1: + client.expire(key, window_seconds) + return count + except Exception: + return _incr_memory(key, window_seconds) + + +async def enforce_capability_rate_limit(request: Request) -> None: + """Cap requests per workspace per minute; raise 429 when exceeded.""" + workspace_id = request.path_params.get("workspace_id") + count = _incr(f"{_KEY_PREFIX}:{workspace_id}", _WINDOW_SECONDS) + if count > CAPABILITY_RATE_LIMIT_PER_MINUTE: + raise HTTPException( + status_code=status.HTTP_429_TOO_MANY_REQUESTS, + detail="Rate limit exceeded for this workspace. Try again shortly.", + ) diff --git a/surfsense_backend/app/capabilities/access/rest.py b/surfsense_backend/app/capabilities/access/rest.py new file mode 100644 index 000000000..6a82e9144 --- /dev/null +++ b/surfsense_backend/app/capabilities/access/rest.py @@ -0,0 +1,69 @@ +"""Generate the REST door from the capability registry (05). + +One typed ``POST`` per verb; each runs the same thin adapter: +authn -> workspace authz -> meter-gate -> executor -> charge -> typed output. +""" + +from fastapi import APIRouter, Depends, HTTPException, status +from sqlalchemy.ext.asyncio import AsyncSession + +from app.auth.context import AuthContext +from app.capabilities.access.rate_limit import enforce_capability_rate_limit +from app.capabilities.billing import charge_capability, gate_capability +from app.capabilities.store import all_capabilities +from app.capabilities.types import Capability, CapabilityContext +from app.db import get_async_session +from app.services.web_crawl_credit_service import InsufficientCreditsError +from app.users import get_auth_context +from app.utils.rbac import check_workspace_access + + +def build_capabilities_router( + capabilities: list[Capability] | None = None, +) -> APIRouter: + """Emit one typed route per verb (defaults to the whole registry).""" + router = APIRouter(tags=["capabilities"]) + caps = capabilities if capabilities is not None else all_capabilities() + for capability in caps: + _register_verb(router, capability) + return router + + +def _register_verb(router: APIRouter, capability: Capability) -> None: + input_model = capability.input_schema + output_model = capability.output_schema + unit = capability.billing_unit + executor = capability.executor + + async def endpoint( + workspace_id: int, + payload: input_model, + session: AsyncSession = Depends(get_async_session), + auth: AuthContext = Depends(get_auth_context), + ): + await check_workspace_access(session, auth, workspace_id) + ctx = CapabilityContext(session=session, workspace_id=workspace_id) + try: + await gate_capability(payload, unit, ctx) + except InsufficientCreditsError as exc: + raise HTTPException( + status_code=status.HTTP_402_PAYMENT_REQUIRED, + detail={ + "error_code": "insufficient_credits", + "message": str(exc), + "balance_micros": exc.balance_micros, + "required_micros": exc.required_micros, + }, + ) from exc + output = await executor(payload) + await charge_capability(output, unit, ctx) + return output + + router.add_api_route( + f"/workspaces/{{workspace_id}}/capabilities/{capability.name}", + endpoint, + methods=["POST"], + response_model=output_model, + name=f"capability:{capability.name}", + dependencies=[Depends(enforce_capability_rate_limit)], + ) From b1e49f21f22e21666ca7e477745df15465b97fa4 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 17:53:42 +0200 Subject: [PATCH 17/64] feat(capabilities): mount capability REST routes --- surfsense_backend/app/routes/__init__.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/surfsense_backend/app/routes/__init__.py b/surfsense_backend/app/routes/__init__.py index fed21913e..df53759aa 100644 --- a/surfsense_backend/app/routes/__init__.py +++ b/surfsense_backend/app/routes/__init__.py @@ -1,6 +1,8 @@ from fastapi import APIRouter, Depends +import app.capabilities.web # noqa: F401 (registers web.* verbs before the door builds) from app.automations.api import router as automations_router +from app.capabilities.access.rest import build_capabilities_router from app.file_storage.api import router as file_storage_router from app.gateway import require_gateway_enabled from app.notifications.api import router as notifications_router @@ -61,12 +63,12 @@ from .rbac_routes import router as rbac_router from .reports_routes import router as reports_router from .sandbox_routes import router as sandbox_router from .search_source_connectors_routes import router as search_source_connectors_router -from .workspaces_routes import router as workspaces_router from .slack_add_connector_route import router as slack_add_connector_router from .stripe_routes import router as stripe_router from .team_memory_routes import router as team_memory_router from .teams_add_connector_route import router as teams_add_connector_router from .video_presentations_routes import router as video_presentations_router +from .workspaces_routes import router as workspaces_router from .youtube_routes import router as youtube_router router = APIRouter() @@ -138,3 +140,4 @@ router.include_router(memory_router) # User personal memory (memory.md style) router.include_router(team_memory_router) # Workspace team memory router.include_router(automations_router) # Automations CRUD + run history router.include_router(file_storage_router) # Original file metadata + download +router.include_router(build_capabilities_router()) # Scraper-API capability doors (05) From b1bd35c08254e48a54304d0c17d7eb9fb4151304 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 17:53:42 +0200 Subject: [PATCH 18/64] test(capabilities): cover REST door, meter-gate, and batch bounds --- .../unit/capabilities/access/__init__.py | 0 .../capabilities/access/test_rate_limit.py | 37 ++++ .../capabilities/access/test_rest_router.py | 171 ++++++++++++++++++ .../tests/unit/capabilities/test_billing.py | 62 ++++++- .../capabilities/web/scrape/test_schemas.py | 25 ++- 5 files changed, 291 insertions(+), 4 deletions(-) create mode 100644 surfsense_backend/tests/unit/capabilities/access/__init__.py create mode 100644 surfsense_backend/tests/unit/capabilities/access/test_rate_limit.py create mode 100644 surfsense_backend/tests/unit/capabilities/access/test_rest_router.py diff --git a/surfsense_backend/tests/unit/capabilities/access/__init__.py b/surfsense_backend/tests/unit/capabilities/access/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/surfsense_backend/tests/unit/capabilities/access/test_rate_limit.py b/surfsense_backend/tests/unit/capabilities/access/test_rate_limit.py new file mode 100644 index 000000000..f9c8076bf --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/access/test_rate_limit.py @@ -0,0 +1,37 @@ +"""Per-workspace rate limit: a secondary guard behind the credit meter-gate (05).""" + +import pytest +from fastapi import HTTPException +from starlette.requests import Request + +from app.capabilities.access import rate_limit + + +def _request(workspace_id: int) -> Request: + return Request({"type": "http", "path_params": {"workspace_id": workspace_id}}) + + +@pytest.mark.asyncio +async def test_passes_at_the_limit(monkeypatch): + monkeypatch.setattr( + rate_limit, "_incr", lambda *a, **k: rate_limit.CAPABILITY_RATE_LIMIT_PER_MINUTE + ) + await rate_limit.enforce_capability_rate_limit(_request(1)) + + +@pytest.mark.asyncio +async def test_blocks_over_the_limit(monkeypatch): + monkeypatch.setattr( + rate_limit, + "_incr", + lambda *a, **k: rate_limit.CAPABILITY_RATE_LIMIT_PER_MINUTE + 1, + ) + with pytest.raises(HTTPException) as exc: + await rate_limit.enforce_capability_rate_limit(_request(1)) + assert exc.value.status_code == 429 + + +def test_memory_fallback_counts_within_window(): + rate_limit._memory.clear() + assert rate_limit._incr_memory("k", window_seconds=60) == 1 + assert rate_limit._incr_memory("k", window_seconds=60) == 2 diff --git a/surfsense_backend/tests/unit/capabilities/access/test_rest_router.py b/surfsense_backend/tests/unit/capabilities/access/test_rest_router.py new file mode 100644 index 000000000..0f4366c4b --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/access/test_rest_router.py @@ -0,0 +1,171 @@ +"""The REST door generator turns registry verbs into typed POST routes (05).""" + +from types import SimpleNamespace + +import pytest +from fastapi import FastAPI +from httpx import ASGITransport, AsyncClient +from pydantic import BaseModel + +from app.capabilities.types import Capability +from app.db import get_async_session +from app.users import get_auth_context + + +class _EchoInput(BaseModel): + value: str + + +class _EchoOutput(BaseModel): + echo: str + + +async def _echo_executor(payload: _EchoInput) -> _EchoOutput: + return _EchoOutput(echo=payload.value) + + +_ECHO = Capability( + name="test.echo", + input_schema=_EchoInput, + output_schema=_EchoOutput, + executor=_echo_executor, + billing_unit=None, +) + + +def _build_app(capabilities, monkeypatch) -> FastAPI: + """Mount the generated door with auth/workspace/session/rate-limit stubbed.""" + from app.capabilities.access import rest + from app.capabilities.access.rate_limit import enforce_capability_rate_limit + + monkeypatch.setattr(rest, "check_workspace_access", _noop_async, raising=True) + + app = FastAPI() + app.include_router(rest.build_capabilities_router(capabilities), prefix="/api/v1") + app.dependency_overrides[get_auth_context] = lambda: SimpleNamespace(user=None) + app.dependency_overrides[enforce_capability_rate_limit] = _allow + + async def _session(): + yield SimpleNamespace() + + app.dependency_overrides[get_async_session] = _session + return app + + +async def _noop_async(*args, **kwargs) -> None: + return None + + +async def _allow() -> None: + return None + + +def _client(app: FastAPI) -> AsyncClient: + return AsyncClient(transport=ASGITransport(app=app), base_url="http://test") + + +@pytest.mark.asyncio +async def test_verb_is_exposed_as_typed_post_route(monkeypatch): + app = _build_app([_ECHO], monkeypatch) + async with _client(app) as client: + resp = await client.post( + "/api/v1/workspaces/7/capabilities/test.echo", + json={"value": "hi"}, + ) + assert resp.status_code == 200 + assert resp.json() == {"echo": "hi"} + + +@pytest.mark.asyncio +async def test_input_is_validated_against_the_verb_schema(monkeypatch): + app = _build_app([_ECHO], monkeypatch) + async with _client(app) as client: + resp = await client.post( + "/api/v1/workspaces/7/capabilities/test.echo", + json={"wrong": "field"}, + ) + assert resp.status_code == 422 + + +def test_registered_verbs_appear_on_rest(): + """A verb in the registry shows up as a route with no per-verb wiring.""" + import app.capabilities.web # noqa: F401 (registers web.* at import) + from app.capabilities.access import rest + + router = rest.build_capabilities_router() + paths = {route.path for route in router.routes} + assert "/workspaces/{workspace_id}/capabilities/web.scrape" in paths + assert "/workspaces/{workspace_id}/capabilities/web.discover" in paths + + +@pytest.mark.asyncio +async def test_over_budget_is_blocked_before_the_executor(monkeypatch): + from app.capabilities.access import rest + from app.services.web_crawl_credit_service import InsufficientCreditsError + + async def _raise(*args, **kwargs): + raise InsufficientCreditsError( + message="over budget", balance_micros=0, required_micros=1000 + ) + + monkeypatch.setattr(rest, "gate_capability", _raise, raising=True) + + app = _build_app([_ECHO], monkeypatch) + async with _client(app) as client: + resp = await client.post( + "/api/v1/workspaces/7/capabilities/test.echo", + json={"value": "hi"}, + ) + assert resp.status_code == 402 + + +@pytest.mark.asyncio +async def test_rate_limit_blocks_the_workspace(monkeypatch): + """The generated route enforces the per-workspace limit (429).""" + from app.capabilities.access import rate_limit, rest + + monkeypatch.setattr(rest, "check_workspace_access", _noop_async, raising=True) + monkeypatch.setattr( + rate_limit, + "_incr", + lambda *a, **k: rate_limit.CAPABILITY_RATE_LIMIT_PER_MINUTE + 1, + ) + + app = FastAPI() + app.include_router(rest.build_capabilities_router([_ECHO]), prefix="/api/v1") + app.dependency_overrides[get_auth_context] = lambda: SimpleNamespace(user=None) + + async def _session(): + yield SimpleNamespace() + + app.dependency_overrides[get_async_session] = _session + + async with _client(app) as client: + resp = await client.post( + "/api/v1/workspaces/7/capabilities/test.echo", + json={"value": "hi"}, + ) + assert resp.status_code == 429 + + +@pytest.mark.asyncio +async def test_success_charges_once(monkeypatch): + from unittest.mock import AsyncMock + + from app.capabilities.access import rest + + charge = AsyncMock() + monkeypatch.setattr(rest, "charge_capability", charge, raising=True) + + app = _build_app([_ECHO], monkeypatch) + async with _client(app) as client: + resp = await client.post( + "/api/v1/workspaces/7/capabilities/test.echo", + json={"value": "hi"}, + ) + assert resp.status_code == 200 + charge.assert_awaited_once() + (output, unit, ctx), _ = charge.call_args + assert isinstance(output, _EchoOutput) + assert unit is None + assert ctx.workspace_id == 7 diff --git a/surfsense_backend/tests/unit/capabilities/test_billing.py b/surfsense_backend/tests/unit/capabilities/test_billing.py index 210e69784..45e65c257 100644 --- a/surfsense_backend/tests/unit/capabilities/test_billing.py +++ b/surfsense_backend/tests/unit/capabilities/test_billing.py @@ -12,10 +12,11 @@ from uuid import UUID import pytest import app.capabilities.billing as billing -from app.capabilities.billing import charge_capability +from app.capabilities.billing import charge_capability, gate_capability from app.capabilities.types import BillingUnit, CapabilityContext -from app.capabilities.web.scrape.schemas import ScrapeOutput, ScrapeRow +from app.capabilities.web.scrape.schemas import ScrapeInput, ScrapeOutput, ScrapeRow from app.config import config +from app.services.web_crawl_credit_service import InsufficientCreditsError pytestmark = pytest.mark.unit @@ -125,3 +126,60 @@ async def test_free_verb_without_a_unit_is_noop(monkeypatch, record_usage): record_usage.assert_not_awaited() session.execute.assert_not_called() assert user.credit_micros_balance == 100_000 + + +def _gate_session(owner_id, balance_micros): + """Mock session serving owner-resolution and the spendable-balance read.""" + session = AsyncMock() + + def _make_result(*_args, **_kwargs): + result = MagicMock() + result.scalar_one_or_none.return_value = owner_id # owner resolution + result.first.return_value = (balance_micros, 0) # balance, reserved + return result + + session.execute = AsyncMock(side_effect=_make_result) + return session + + +async def test_gate_blocks_when_worst_case_exceeds_balance(monkeypatch): + monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", True) + monkeypatch.setattr(config, "WEB_CRAWL_MICROS_PER_SUCCESS", 1000) + session = _gate_session(_OWNER, balance_micros=1500) # affords 1 crawl, not 2 + + with pytest.raises(InsufficientCreditsError): + await gate_capability( + ScrapeInput(urls=["https://a.com", "https://b.com"]), + BillingUnit.WEB_CRAWL, + _ctx(session), + ) + + +async def test_gate_passes_when_balance_covers_worst_case(monkeypatch): + monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", True) + monkeypatch.setattr(config, "WEB_CRAWL_MICROS_PER_SUCCESS", 1000) + session = _gate_session(_OWNER, balance_micros=100_000) + + await gate_capability( + ScrapeInput(urls=["https://a.com", "https://b.com"]), + BillingUnit.WEB_CRAWL, + _ctx(session), + ) + + +async def test_gate_is_noop_when_disabled(monkeypatch): + monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", False) + session = _gate_session(_OWNER, balance_micros=0) + + await gate_capability( + ScrapeInput(urls=["https://a.com"]), BillingUnit.WEB_CRAWL, _ctx(session) + ) + + +async def test_gate_is_noop_for_free_verb(monkeypatch): + monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", True) + session = _gate_session(_OWNER, balance_micros=0) + + await gate_capability(ScrapeInput(urls=["https://a.com"]), None, _ctx(session)) + + session.execute.assert_not_called() diff --git a/surfsense_backend/tests/unit/capabilities/web/scrape/test_schemas.py b/surfsense_backend/tests/unit/capabilities/web/scrape/test_schemas.py index c8aa6c78a..9cc0eef23 100644 --- a/surfsense_backend/tests/unit/capabilities/web/scrape/test_schemas.py +++ b/surfsense_backend/tests/unit/capabilities/web/scrape/test_schemas.py @@ -1,10 +1,16 @@ -"""ScrapeOutput reports its own billable count (a success single-sources it).""" +"""ScrapeOutput reports its own billable count; ScrapeInput bounds its batch size.""" from __future__ import annotations import pytest +from pydantic import ValidationError -from app.capabilities.web.scrape.schemas import ScrapeOutput, ScrapeRow +from app.capabilities.web.scrape.schemas import ( + MAX_SCRAPE_URLS, + ScrapeInput, + ScrapeOutput, + ScrapeRow, +) pytestmark = pytest.mark.unit @@ -24,3 +30,18 @@ def test_billable_units_counts_successful_rows(): def test_billable_units_is_zero_without_successes(): assert _output("empty", "failed").billable_units == 0 + + +def test_rejects_empty_url_batch(): + with pytest.raises(ValidationError): + ScrapeInput(urls=[]) + + +def test_rejects_batch_over_the_cap(): + with pytest.raises(ValidationError): + ScrapeInput(urls=[f"https://{i}.com" for i in range(MAX_SCRAPE_URLS + 1)]) + + +def test_accepts_batch_at_the_cap(): + payload = ScrapeInput(urls=[f"https://{i}.com" for i in range(MAX_SCRAPE_URLS)]) + assert payload.estimated_units == MAX_SCRAPE_URLS From 23802a74bb08b10002e24b47b5491e056fd87f36 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 00:16:55 +0200 Subject: [PATCH 19/64] feat(capabilities): document verbs with descriptions and field docs --- surfsense_backend/app/capabilities/types.py | 1 + .../capabilities/web/discover/definition.py | 6 +++ .../app/capabilities/web/discover/schemas.py | 24 +++++++---- .../app/capabilities/web/scrape/definition.py | 6 +++ .../app/capabilities/web/scrape/schemas.py | 42 +++++++++++++++---- .../capabilities/access/test_rest_router.py | 1 + 6 files changed, 64 insertions(+), 16 deletions(-) diff --git a/surfsense_backend/app/capabilities/types.py b/surfsense_backend/app/capabilities/types.py index 4a8b57cbf..14c1f17e7 100644 --- a/surfsense_backend/app/capabilities/types.py +++ b/surfsense_backend/app/capabilities/types.py @@ -48,6 +48,7 @@ class Capability: """One typed verb; the source of truth the doors (05) and agent (07) read.""" name: str + description: str input_schema: type[BaseModel] output_schema: type[BaseModel] executor: Executor diff --git a/surfsense_backend/app/capabilities/web/discover/definition.py b/surfsense_backend/app/capabilities/web/discover/definition.py index 1c3b4ec28..e4e779307 100644 --- a/surfsense_backend/app/capabilities/web/discover/definition.py +++ b/surfsense_backend/app/capabilities/web/discover/definition.py @@ -9,6 +9,12 @@ from app.capabilities.web.discover.schemas import DiscoverInput, DiscoverOutput WEB_DISCOVER = Capability( name="web.discover", + description=( + "Search the web for a query and return ranked results. Use it to find " + "pages when you don't already have exact URLs. Returns a list of hits " + "(url, title, snippet, provider); pass the chosen url(s) to web.scrape " + "to read their full content." + ), input_schema=DiscoverInput, output_schema=DiscoverOutput, executor=build_discover_executor(), diff --git a/surfsense_backend/app/capabilities/web/discover/schemas.py b/surfsense_backend/app/capabilities/web/discover/schemas.py index 77622b210..ae74d2c56 100644 --- a/surfsense_backend/app/capabilities/web/discover/schemas.py +++ b/surfsense_backend/app/capabilities/web/discover/schemas.py @@ -2,20 +2,28 @@ from __future__ import annotations -from pydantic import BaseModel +from pydantic import BaseModel, Field class DiscoverInput(BaseModel): - query: str - top_k: int = 10 + query: str = Field( + description="What to search the web for, phrased in natural language." + ) + top_k: int = Field( + default=10, description="Maximum number of results to return (1-50)." + ) class DiscoverHit(BaseModel): - url: str - title: str - snippet: str | None = None - provider: str + url: str = Field( + description="The result's page URL; pass it to web.scrape to read it." + ) + title: str = Field(description="The result's page title.") + snippet: str | None = Field( + default=None, description="A short extract summarizing the page." + ) + provider: str = Field(description="Which search engine returned this hit.") class DiscoverOutput(BaseModel): - hits: list[DiscoverHit] + hits: list[DiscoverHit] = Field(description="Ranked search results, best first.") diff --git a/surfsense_backend/app/capabilities/web/scrape/definition.py b/surfsense_backend/app/capabilities/web/scrape/definition.py index 295795066..5ceb0b3a5 100644 --- a/surfsense_backend/app/capabilities/web/scrape/definition.py +++ b/surfsense_backend/app/capabilities/web/scrape/definition.py @@ -9,6 +9,12 @@ from app.capabilities.web.scrape.schemas import ScrapeInput, ScrapeOutput WEB_SCRAPE = Capability( name="web.scrape", + description=( + "Fetch one or more web pages and return their clean, readable content. " + "Give it the exact URLs to read (use web.discover first if you don't have " + "them). Returns one row per URL with a status (success/empty/failed), the " + "page content, and metadata such as title and description." + ), input_schema=ScrapeInput, output_schema=ScrapeOutput, executor=build_scrape_executor(), diff --git a/surfsense_backend/app/capabilities/web/scrape/schemas.py b/surfsense_backend/app/capabilities/web/scrape/schemas.py index 91de17a53..415a64554 100644 --- a/surfsense_backend/app/capabilities/web/scrape/schemas.py +++ b/surfsense_backend/app/capabilities/web/scrape/schemas.py @@ -11,8 +11,21 @@ MAX_SCRAPE_URLS = 20 class ScrapeInput(BaseModel): - urls: list[str] = Field(min_length=1, max_length=MAX_SCRAPE_URLS) - max_length: int = 50_000 + urls: list[str] = Field( + min_length=1, + max_length=MAX_SCRAPE_URLS, + description=( + "Full page URLs to fetch and read (1-20), each starting with " + "http:// or https://. Pass the exact URLs you want the content of." + ), + ) + max_length: int = Field( + default=50_000, + description=( + "Maximum characters of cleaned content returned per page; " + "content longer than this is truncated." + ), + ) @property def estimated_units(self) -> int: @@ -21,15 +34,28 @@ class ScrapeInput(BaseModel): class ScrapeRow(BaseModel): - url: str - status: Literal["success", "empty", "failed"] - content: str | None = None - metadata: dict[str, str] | None = None - error: str | None = None + url: str = Field(description="The requested URL this result is for.") + status: Literal["success", "empty", "failed"] = Field( + description=( + "'success' = content returned; 'empty' = page reached but no " + "readable content; 'failed' = could not be fetched (see error)." + ) + ) + content: str | None = Field( + default=None, description="Cleaned, readable page text (present on success)." + ) + metadata: dict[str, str] | None = Field( + default=None, description="Page metadata such as title and description." + ) + error: str | None = Field( + default=None, description="Why the fetch failed (present on 'failed')." + ) class ScrapeOutput(BaseModel): - rows: list[ScrapeRow] + rows: list[ScrapeRow] = Field( + description="One result per requested URL, in the same order." + ) @property def billable_units(self) -> int: diff --git a/surfsense_backend/tests/unit/capabilities/access/test_rest_router.py b/surfsense_backend/tests/unit/capabilities/access/test_rest_router.py index 0f4366c4b..211ce9854 100644 --- a/surfsense_backend/tests/unit/capabilities/access/test_rest_router.py +++ b/surfsense_backend/tests/unit/capabilities/access/test_rest_router.py @@ -26,6 +26,7 @@ async def _echo_executor(payload: _EchoInput) -> _EchoOutput: _ECHO = Capability( name="test.echo", + description="Echo the input back for tests.", input_schema=_EchoInput, output_schema=_EchoOutput, executor=_echo_executor, From de4b8ab6b0d59a8973e2f7e077c192dd2997d727 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 00:17:01 +0200 Subject: [PATCH 20/64] feat(capabilities): generate agent tool door from registry --- .../app/capabilities/access/__init__.py | 2 +- .../app/capabilities/access/agent.py | 52 +++++++ .../capabilities/access/test_agent_tools.py | 127 ++++++++++++++++++ 3 files changed, 180 insertions(+), 1 deletion(-) create mode 100644 surfsense_backend/app/capabilities/access/agent.py create mode 100644 surfsense_backend/tests/unit/capabilities/access/test_agent_tools.py diff --git a/surfsense_backend/app/capabilities/access/__init__.py b/surfsense_backend/app/capabilities/access/__init__.py index f1afc4ff5..1e24750c8 100644 --- a/surfsense_backend/app/capabilities/access/__init__.py +++ b/surfsense_backend/app/capabilities/access/__init__.py @@ -1 +1 @@ -"""Access doors (05): thin adapters that expose the 04 registry as REST/MCP/chat.""" +"""Access doors (05): thin adapters that expose the 04 registry as REST/MCP/agent.""" diff --git a/surfsense_backend/app/capabilities/access/agent.py b/surfsense_backend/app/capabilities/access/agent.py new file mode 100644 index 000000000..75b535840 --- /dev/null +++ b/surfsense_backend/app/capabilities/access/agent.py @@ -0,0 +1,52 @@ +"""Generate the agent door from the capability registry (05). + +One LangChain tool per verb; each runs the same thin adapter as the REST door +(``access/rest.py``): meter-gate -> executor -> charge. The tool returns the +verb's serialized output so the model can reason over it; UI cards are the SSE +emission handler's job, not this generator's. +""" + +from __future__ import annotations + +from langchain_core.tools import BaseTool, StructuredTool + +from app.capabilities.billing import charge_capability, gate_capability +from app.capabilities.store import all_capabilities +from app.capabilities.types import Capability, CapabilityContext +from app.db import async_session_maker +from app.services.web_crawl_credit_service import InsufficientCreditsError + + +def build_capability_tools( + *, + workspace_id: int, + capabilities: list[Capability] | None = None, +) -> list[BaseTool]: + """Emit one tool per verb (defaults to the whole registry).""" + caps = capabilities if capabilities is not None else all_capabilities() + return [_capability_tool(cap, workspace_id) for cap in caps] + + +def _capability_tool(capability: Capability, workspace_id: int) -> BaseTool: + input_model = capability.input_schema + unit = capability.billing_unit + executor = capability.executor + + async def _run(**kwargs: object) -> dict | str: + payload = input_model(**kwargs) + async with async_session_maker() as session: + ctx = CapabilityContext(session=session, workspace_id=workspace_id) + try: + await gate_capability(payload, unit, ctx) + except InsufficientCreditsError as exc: + return str(exc) + output = await executor(payload) + await charge_capability(output, unit, ctx) + return output.model_dump() + + return StructuredTool.from_function( + coroutine=_run, + name=capability.name.replace(".", "_"), + description=capability.description, + args_schema=input_model, + ) diff --git a/surfsense_backend/tests/unit/capabilities/access/test_agent_tools.py b/surfsense_backend/tests/unit/capabilities/access/test_agent_tools.py new file mode 100644 index 000000000..dc8e126a6 --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/access/test_agent_tools.py @@ -0,0 +1,127 @@ +"""The agent door (05): generate one LangChain tool per registry verb.""" + +from types import SimpleNamespace +from unittest.mock import AsyncMock + +import pytest +from pydantic import BaseModel, Field + +from app.capabilities.types import BillingUnit, Capability +from app.services.web_crawl_credit_service import InsufficientCreditsError + +pytestmark = pytest.mark.asyncio + + +class _EchoInput(BaseModel): + text: str = Field(description="The text to echo back.") + + +class _EchoOutput(BaseModel): + echoed: str + + @property + def billable_units(self) -> int: + return 1 + + +def _capability( + *, name: str, output: _EchoOutput, unit=BillingUnit.WEB_CRAWL +) -> Capability: + async def _executor(payload: _EchoInput) -> _EchoOutput: + _executor.seen = payload + return output + + cap = Capability( + name=name, + description=f"{name} does a thing.", + input_schema=_EchoInput, + output_schema=_EchoOutput, + executor=_executor, + billing_unit=unit, + ) + cap.executor.seen = None # type: ignore[attr-defined] + return cap + + +class _FakeSessionCtx: + async def __aenter__(self): + return SimpleNamespace() + + async def __aexit__(self, *exc): + return False + + +@pytest.fixture +def isolate(monkeypatch): + """Stub the billing session + charge/gate so tools never hit the DB.""" + from app.capabilities.access import agent as mod + + monkeypatch.setattr(mod, "async_session_maker", lambda: _FakeSessionCtx()) + charge = AsyncMock() + gate = AsyncMock() + monkeypatch.setattr(mod, "charge_capability", charge) + monkeypatch.setattr(mod, "gate_capability", gate) + return SimpleNamespace(module=mod, charge=charge, gate=gate) + + +async def test_registry_becomes_one_tool_per_verb(isolate): + caps = [ + _capability(name="web.scrape", output=_EchoOutput(echoed="a")), + _capability(name="web.discover", output=_EchoOutput(echoed="b"), unit=None), + ] + + tools = isolate.module.build_capability_tools(workspace_id=7, capabilities=caps) + + by_name = {t.name: t for t in tools} + assert set(by_name) == {"web_scrape", "web_discover"} + assert by_name["web_scrape"].description == "web.scrape does a thing." + assert by_name["web_scrape"].args_schema is _EchoInput + + +async def test_input_field_docs_reach_the_model(isolate): + """Per-field descriptions must surface in the tool's args schema (LLM context).""" + cap = _capability(name="web.scrape", output=_EchoOutput(echoed="a")) + [tool] = isolate.module.build_capability_tools(workspace_id=7, capabilities=[cap]) + + assert tool.args["text"]["description"] == "The text to echo back." + + +async def test_tool_runs_executor_and_returns_serialized_output(isolate): + cap = _capability(name="web.scrape", output=_EchoOutput(echoed="hi there")) + [tool] = isolate.module.build_capability_tools(workspace_id=7, capabilities=[cap]) + + result = await tool.ainvoke({"text": "ping"}) + + assert result == {"echoed": "hi there"} + assert cap.executor.seen.text == "ping" + + +async def test_tool_charges_owner(isolate): + output = _EchoOutput(echoed="hi") + cap = _capability(name="web.scrape", output=output) + [tool] = isolate.module.build_capability_tools(workspace_id=7, capabilities=[cap]) + + await tool.ainvoke({"text": "ping"}) + + isolate.charge.assert_awaited_once() + (charged_output, unit, ctx), _ = isolate.charge.call_args + assert charged_output is output + assert unit is BillingUnit.WEB_CRAWL + assert ctx.workspace_id == 7 + + +async def test_over_budget_returns_friendly_message(isolate): + cap = _capability(name="web.scrape", output=_EchoOutput(echoed="hi")) + isolate.gate.side_effect = InsufficientCreditsError( + message="This run would exceed your available credit.", + balance_micros=0, + required_micros=1_000_000, + ) + [tool] = isolate.module.build_capability_tools(workspace_id=7, capabilities=[cap]) + + result = await tool.ainvoke({"text": "ping"}) + + assert isinstance(result, str) + assert "credit" in result.lower() + assert cap.executor.seen is None + isolate.charge.assert_not_awaited() From 110beb4dd1d301f85c39dbcf31c139aef5720d24 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 00:17:10 +0200 Subject: [PATCH 21/64] feat(chat): add web_scrape/web_discover tool emission handlers --- .../handlers/tools/web_discover/__init__.py | 1 + .../handlers/tools/web_discover/emission.py | 29 ++++ .../handlers/tools/web_scrape/__init__.py | 1 + .../handlers/tools/web_scrape/emission.py | 54 ++++++++ .../streaming/test_capability_emissions.py | 131 ++++++++++++++++++ 5 files changed, 216 insertions(+) create mode 100644 surfsense_backend/app/tasks/chat/streaming/handlers/tools/web_discover/__init__.py create mode 100644 surfsense_backend/app/tasks/chat/streaming/handlers/tools/web_discover/emission.py create mode 100644 surfsense_backend/app/tasks/chat/streaming/handlers/tools/web_scrape/__init__.py create mode 100644 surfsense_backend/app/tasks/chat/streaming/handlers/tools/web_scrape/emission.py create mode 100644 surfsense_backend/tests/unit/tasks/chat/streaming/test_capability_emissions.py diff --git a/surfsense_backend/app/tasks/chat/streaming/handlers/tools/web_discover/__init__.py b/surfsense_backend/app/tasks/chat/streaming/handlers/tools/web_discover/__init__.py new file mode 100644 index 000000000..9ad4346a6 --- /dev/null +++ b/surfsense_backend/app/tasks/chat/streaming/handlers/tools/web_discover/__init__.py @@ -0,0 +1 @@ +"""web_discover tool: UI emission for the web.discover capability (05).""" diff --git a/surfsense_backend/app/tasks/chat/streaming/handlers/tools/web_discover/emission.py b/surfsense_backend/app/tasks/chat/streaming/handlers/tools/web_discover/emission.py new file mode 100644 index 000000000..4f0592889 --- /dev/null +++ b/surfsense_backend/app/tasks/chat/streaming/handlers/tools/web_discover/emission.py @@ -0,0 +1,29 @@ +"""web_discover: ranked-hits card + a result-count terminal line.""" + +from __future__ import annotations + +from collections.abc import Iterator + +from app.tasks.chat.streaming.handlers.tools.emission_context import ( + ToolCompletionEmissionContext, +) + + +def iter_completion_emission_frames( + ctx: ToolCompletionEmissionContext, +) -> Iterator[str]: + out = ctx.tool_output + if not isinstance(out, dict): + message = str(out) + yield ctx.emit_tool_output_card({"status": "error", "message": message}) + yield ctx.streaming_service.format_terminal_info(message, "error") + return + + hits = out.get("hits") or [] + yield ctx.emit_tool_output_card( + {"status": "completed", "hits": hits, "count": len(hits)} + ) + level = "success" if hits else "info" + yield ctx.streaming_service.format_terminal_info( + f"Found {len(hits)} result(s)", level + ) diff --git a/surfsense_backend/app/tasks/chat/streaming/handlers/tools/web_scrape/__init__.py b/surfsense_backend/app/tasks/chat/streaming/handlers/tools/web_scrape/__init__.py new file mode 100644 index 000000000..2282f2277 --- /dev/null +++ b/surfsense_backend/app/tasks/chat/streaming/handlers/tools/web_scrape/__init__.py @@ -0,0 +1 @@ +"""web_scrape tool: UI emission for the web.scrape capability (05).""" diff --git a/surfsense_backend/app/tasks/chat/streaming/handlers/tools/web_scrape/emission.py b/surfsense_backend/app/tasks/chat/streaming/handlers/tools/web_scrape/emission.py new file mode 100644 index 000000000..dcc0a494d --- /dev/null +++ b/surfsense_backend/app/tasks/chat/streaming/handlers/tools/web_scrape/emission.py @@ -0,0 +1,54 @@ +"""web_scrape: per-page card (content previewed) + a scraped-count terminal line.""" + +from __future__ import annotations + +from collections.abc import Iterator + +from app.tasks.chat.streaming.handlers.tools.emission_context import ( + ToolCompletionEmissionContext, +) + +_PREVIEW_CHARS = 500 + + +def iter_completion_emission_frames( + ctx: ToolCompletionEmissionContext, +) -> Iterator[str]: + out = ctx.tool_output + if not isinstance(out, dict): + message = str(out) + yield ctx.emit_tool_output_card({"status": "error", "message": message}) + yield ctx.streaming_service.format_terminal_info(message, "error") + return + + rows = out.get("rows") or [] + pages = [_page(row) for row in rows] + succeeded = sum(1 for row in rows if row.get("status") == "success") + + yield ctx.emit_tool_output_card( + { + "status": "completed", + "pages": pages, + "succeeded": succeeded, + "total": len(rows), + } + ) + level = "success" if succeeded else "error" + yield ctx.streaming_service.format_terminal_info( + f"Scraped {succeeded}/{len(rows)} page(s)", level + ) + + +def _page(row: dict) -> dict: + """A card-safe view of one row: metadata kept, content bounded to a preview.""" + page: dict = {"url": row.get("url"), "status": row.get("status")} + if row.get("metadata"): + page["metadata"] = row["metadata"] + content = row.get("content") + if content: + page["content_preview"] = ( + content[:_PREVIEW_CHARS] + "…" if len(content) > _PREVIEW_CHARS else content + ) + if row.get("error"): + page["error"] = row["error"] + return page diff --git a/surfsense_backend/tests/unit/tasks/chat/streaming/test_capability_emissions.py b/surfsense_backend/tests/unit/tasks/chat/streaming/test_capability_emissions.py new file mode 100644 index 000000000..46a373b8d --- /dev/null +++ b/surfsense_backend/tests/unit/tasks/chat/streaming/test_capability_emissions.py @@ -0,0 +1,131 @@ +"""Emission handlers for the capability chat tools (web_scrape / web_discover).""" + +from __future__ import annotations + +import importlib + + +class _FakeStreaming: + def __init__(self) -> None: + self.terminals: list[tuple[str, str]] = [] + + def format_terminal_info(self, text: str, message_type: str = "info") -> str: + self.terminals.append((text, message_type)) + return f"TERM::{message_type}" + + +class _FakeCtx: + """Minimal stand-in for ToolCompletionEmissionContext (only what handlers use).""" + + def __init__(self, tool_output: object) -> None: + self.tool_output = tool_output + self.cards: list[dict] = [] + self.streaming_service = _FakeStreaming() + + def emit_tool_output_card(self, payload: dict) -> str: + self.cards.append(payload) + return "CARD" + + +def _scrape_frames(ctx: _FakeCtx) -> list[str]: + mod = importlib.import_module( + "app.tasks.chat.streaming.handlers.tools.web_scrape.emission" + ) + return list(mod.iter_completion_emission_frames(ctx)) + + +def _discover_frames(ctx: _FakeCtx) -> list[str]: + mod = importlib.import_module( + "app.tasks.chat.streaming.handlers.tools.web_discover.emission" + ) + return list(mod.iter_completion_emission_frames(ctx)) + + +def test_scrape_card_previews_content_and_summarizes(): + long_body = "x" * 2000 + ctx = _FakeCtx( + { + "rows": [ + { + "url": "https://a.com", + "status": "success", + "content": long_body, + "metadata": {"title": "A"}, + }, + {"url": "https://b.com", "status": "failed", "error": "boom"}, + ] + } + ) + + _scrape_frames(ctx) + + [card] = ctx.cards + assert card["succeeded"] == 1 + assert card["total"] == 2 + page = card["pages"][0] + # full content must not be dumped into the card; a bounded preview is used + assert "content" not in page + assert len(page["content_preview"]) < len(long_body) + assert page["metadata"] == {"title": "A"} + assert card["pages"][1]["error"] == "boom" + + +def test_scrape_terminal_success_when_any_page_succeeds(): + ctx = _FakeCtx( + {"rows": [{"url": "https://a.com", "status": "success", "content": "hi"}]} + ) + + _scrape_frames(ctx) + + assert ctx.streaming_service.terminals[-1][1] == "success" + + +def test_scrape_terminal_error_when_all_pages_fail(): + ctx = _FakeCtx( + {"rows": [{"url": "https://a.com", "status": "failed", "error": "boom"}]} + ) + + _scrape_frames(ctx) + + assert ctx.streaming_service.terminals[-1][1] == "error" + + +def test_scrape_non_dict_output_is_an_error_card(): + ctx = _FakeCtx("Insufficient credit to continue.") + + _scrape_frames(ctx) + + assert ctx.cards[0]["status"] == "error" + assert ctx.streaming_service.terminals[-1][1] == "error" + + +def test_discover_card_lists_hits_and_counts(): + ctx = _FakeCtx( + { + "hits": [ + {"url": "https://a.com", "title": "A", "snippet": "s", "provider": "p"}, + { + "url": "https://b.com", + "title": "B", + "snippet": None, + "provider": "p", + }, + ] + } + ) + + _discover_frames(ctx) + + [card] = ctx.cards + assert card["count"] == 2 + assert card["hits"][0]["url"] == "https://a.com" + assert ctx.streaming_service.terminals[-1][1] == "success" + + +def test_discover_non_dict_output_is_an_error_card(): + ctx = _FakeCtx("Search is not available.") + + _discover_frames(ctx) + + assert ctx.cards[0]["status"] == "error" + assert ctx.streaming_service.terminals[-1][1] == "error" From 7ad83760b5630ff1af59ca9a4036d8e95c702bf8 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 00:17:22 +0200 Subject: [PATCH 22/64] feat(chat): add intelligence_agent subagent over capability verbs --- .../agents/chat/multi_agent_chat/constants.py | 1 + .../builtins/intelligence_agent/__init__.py | 0 .../builtins/intelligence_agent/agent.py | 43 +++++++++++++++ .../intelligence_agent/description.md | 2 + .../intelligence_agent/system_prompt.md | 54 +++++++++++++++++++ .../intelligence_agent/tools/__init__.py | 1 + .../intelligence_agent/tools/index.py | 28 ++++++++++ .../multi_agent_chat/subagents/registry.py | 4 ++ .../test_subagent_composition.py | 3 +- 9 files changed, 135 insertions(+), 1 deletion(-) create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/__init__.py create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/agent.py create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/description.md create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/system_prompt.md create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/__init__.py create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/index.py diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/constants.py b/surfsense_backend/app/agents/chat/multi_agent_chat/constants.py index 7e4061813..fff608742 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/constants.py +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/constants.py @@ -26,6 +26,7 @@ CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS: dict[str, str] = { SUBAGENT_TO_REQUIRED_CONNECTOR_MAP: dict[str, frozenset[str]] = { "deliverables": frozenset(), "knowledge_base": frozenset(), + "intelligence_agent": frozenset(), "airtable": frozenset({"AIRTABLE_CONNECTOR"}), "calendar": frozenset({"GOOGLE_CALENDAR_CONNECTOR"}), "clickup": frozenset({"CLICKUP_CONNECTOR"}), diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/__init__.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/agent.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/agent.py new file mode 100644 index 000000000..21933005b --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/agent.py @@ -0,0 +1,43 @@ +"""``intelligence_agent`` route: ``SurfSenseSubagentSpec`` builder for deepagents.""" + +from __future__ import annotations + +from typing import Any + +from langchain_core.language_models import BaseChatModel +from langchain_core.tools import BaseTool + +from app.agents.chat.multi_agent_chat.subagents.shared.md_file_reader import ( + read_md_file, +) +from app.agents.chat.multi_agent_chat.subagents.shared.spec import SurfSenseSubagentSpec +from app.agents.chat.multi_agent_chat.subagents.shared.subagent_builder import ( + pack_subagent, +) + +from .tools.index import NAME, RULESET, load_tools + + +def build_subagent( + *, + dependencies: dict[str, Any], + model: BaseChatModel | None = None, + middleware_stack: dict[str, Any] | None = None, + mcp_tools: list[BaseTool] | None = None, +) -> SurfSenseSubagentSpec: + tools = [*load_tools(dependencies=dependencies), *(mcp_tools or [])] + description = ( + read_md_file(__package__, "description").strip() + or "Gathers and compares web intelligence for this workspace." + ) + system_prompt = read_md_file(__package__, "system_prompt").strip() + return pack_subagent( + name=NAME, + description=description, + system_prompt=system_prompt, + tools=tools, + ruleset=RULESET, + dependencies=dependencies, + model=model, + middleware_stack=middleware_stack, + ) diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/description.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/description.md new file mode 100644 index 000000000..cefad396c --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/description.md @@ -0,0 +1,2 @@ +Specialist for web intelligence: fetch live pages and search the web, then compare against earlier findings. +Use whenever a task needs current data pulled from specific URLs or discovered on the web — competitor/product monitoring, pricing, listings, "what changed since last time" — and answered from that evidence. diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/system_prompt.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/system_prompt.md new file mode 100644 index 000000000..0289f40e2 --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/system_prompt.md @@ -0,0 +1,54 @@ +You are the SurfSense intelligence sub-agent. +You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis. + + +Pull evidence from the live web with the web.* capability verbs and answer the delegated question, including "what changed" comparisons against evidence already in this conversation. + + + +- `web_discover` — search the web for a query; returns ranked hits (url, title, snippet, provider). +- `web_scrape` — fetch specific URLs; returns clean page content and metadata per URL. + + + +- If the request names exact URLs, call `web_scrape` on them directly. +- If it does not, call `web_discover` first, pick the most relevant hits, then `web_scrape` those URLs to read them. +- Batch URLs into a single `web_scrape` call when reading several pages (up to its limit) instead of many one-URL calls. +- For "what changed" / monitoring requests, compare the freshly scraped values against the prior values already present in this conversation's earlier tool results, and report the concrete deltas (added, removed, changed old -> new). Do not claim a change you cannot point to. + + + +- Use only tools in ``. +- A `web_scrape` row with `status` other than `success` yielded no content — do not invent its content; report it as unavailable. +- Never fabricate facts, URLs, prices, or quotes. + + + +- Report uncertainty explicitly when evidence is incomplete or conflicting. +- Never present unverified claims as facts. + + + +- On tool failure, return `status=error` with a concise recovery `next_step`. +- On no useful evidence, return `status=blocked` with a narrower query or the URLs you still need. + + + +Return **only** one JSON object (no markdown/prose): +{ + "status": "success" | "partial" | "blocked" | "error", + "action_summary": string, + "evidence": { + "findings": string[], + "sources": string[], + "confidence": "high" | "medium" | "low" + }, + "next_step": string | null, + "missing_fields": string[] | null, + "assumptions": string[] | null +} + +Route-specific rules: +- `evidence.findings`: max 10 entries, each a single sentence stating one distinct fact or delta. Do not paste raw scraped pages. +- `evidence.sources`: max 10 URLs, one per finding when applicable. List each URL once. + diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/__init__.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/__init__.py new file mode 100644 index 000000000..e06e600b5 --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/__init__.py @@ -0,0 +1 @@ +"""Intelligence-agent tools: the web.* capability verbs, generated from the registry.""" diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/index.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/index.py new file mode 100644 index 000000000..f0b088dc1 --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/index.py @@ -0,0 +1,28 @@ +"""``intelligence_agent`` tools: web.* verbs generated from the capability registry.""" + +from __future__ import annotations + +from typing import Any + +from langchain_core.tools import BaseTool + +from app.agents.chat.multi_agent_chat.shared.permissions import Ruleset +from app.capabilities.access.agent import build_capability_tools +from app.capabilities.web.discover.definition import WEB_DISCOVER +from app.capabilities.web.scrape.definition import WEB_SCRAPE + +NAME = "intelligence_agent" + +RULESET = Ruleset(origin=NAME, rules=[]) + +_CI_VERBS = [WEB_DISCOVER, WEB_SCRAPE] + + +def load_tools( + *, dependencies: dict[str, Any] | None = None, **kwargs: Any +) -> list[BaseTool]: + d = {**(dependencies or {}), **kwargs} + return build_capability_tools( + workspace_id=d.get("workspace_id"), + capabilities=_CI_VERBS, + ) diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py index c48b7f7ac..652ca079e 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py @@ -15,6 +15,9 @@ from app.agents.chat.multi_agent_chat.constants import ( from app.agents.chat.multi_agent_chat.subagents.builtins.deliverables.agent import ( build_subagent as build_deliverables_subagent, ) +from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.agent import ( + build_subagent as build_intelligence_agent_subagent, +) from app.agents.chat.multi_agent_chat.subagents.builtins.knowledge_base.agent import ( build_subagent as build_knowledge_base_subagent, ) @@ -99,6 +102,7 @@ SUBAGENT_BUILDERS_BY_NAME: dict[str, SubagentBuilder] = { "dropbox": build_dropbox_subagent, "gmail": build_gmail_subagent, "google_drive": build_google_drive_subagent, + "intelligence_agent": build_intelligence_agent_subagent, "jira": build_jira_subagent, "knowledge_base": build_knowledge_base_subagent, "linear": build_linear_subagent, diff --git a/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py b/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py index 157f1703b..68ae05bbd 100644 --- a/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py +++ b/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py @@ -19,7 +19,7 @@ from app.agents.chat.multi_agent_chat.subagents.registry import ( pytestmark = pytest.mark.unit -# The full specialist roster the main agent composes from: 4 builtins + 15 +# The full specialist roster the main agent composes from: 5 builtins + 15 # connector routes. Adding/removing a specialist is a deliberate product change # and must be reflected here. _EXPECTED_SUBAGENTS = frozenset( @@ -33,6 +33,7 @@ _EXPECTED_SUBAGENTS = frozenset( "dropbox", "gmail", "google_drive", + "intelligence_agent", "jira", "knowledge_base", "linear", From 899ac34bd3409f8f881f154673bdbfce2712d7c9 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 00:17:28 +0200 Subject: [PATCH 23/64] docs(plan): align phase 7 with agent door billing --- plans/backend/07-orchestration.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/plans/backend/07-orchestration.md b/plans/backend/07-orchestration.md index 09da4ba2e..82754de18 100644 --- a/plans/backend/07-orchestration.md +++ b/plans/backend/07-orchestration.md @@ -32,11 +32,12 @@ The main agent delegates CI-/scraping-flavored requests to it via `description.m | Tool | Wraps | Billing | |------|-------|---------| -| capability verbs (`web.scrape`, `web.discover`, `maps.search`, `maps.place`, `maps.reviews`) | `04` executors (direct-return) | `03c` turn accumulator (as `scrape_webpage` does) | +| capability verbs (`web.scrape`, `web.discover`, `maps.search`, `maps.place`, `maps.reviews`) | `04` executors (direct-return) | `03c` owner-wallet charge via `charge_capability` (same path as REST/MCP) | | `start_watch(intent)` *(deferred)* | hands off to `06` | per re-invocation (design with `06`) | -Capability verbs are a shared, registry-generated tool module (`04`) that `research` can also load; the -`intelligence_agent` adds the CI prompt and the `06` handoff. +Verbs reach the subagent through the chat door generator `access/chat.py::build_capability_tools` (`05`), +which turns registry verbs (`04`) into tools with owner-wallet billing. The `intelligence_agent` owns these +CI-flavored calls; the main agent's `web_search`/`scrape_webpage` stay untouched. ## Boundaries @@ -49,7 +50,7 @@ Capability verbs are a shared, registry-generated tool module (`04`) that `resea 1. Subagent scaffold: `subagents/builtins/intelligence_agent/` (`agent.py` / `tools/index.py` / `description.md` / `system_prompt.md`), packed via `pack_subagent`. 2. CI playbook prompt: intent routing (one clarifying question), verb chains, "what changed" reasoning. -3. Shared verb tool module: registry-backed (`04`), reusable by `research`. +3. Chat door generator: `access/chat.py::build_capability_tools` turns registry verbs (`04`) into tools (`05`). 4. `start_watch` seam to `06` (wired when `06` is designed). 5. Router registration: `description.md` for main-agent delegation. @@ -69,5 +70,4 @@ Capability verbs are a shared, registry-generated tool module (`04`) that `resea ## Open questions -- Whether CI one-shot scraping stays in `research` or the `intelligence_agent` owns all CI-flavored calls. - The `start_watch` handoff shape (settled with `06`). From 787efbc21755c287fc680998710b2b8005e980bc Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 17:54:31 +0200 Subject: [PATCH 24/64] docs(plan): define phase 6 chat watch --- plans/backend/06-ongoing-automation.md | 70 ++++++++++++++++++++------ 1 file changed, 55 insertions(+), 15 deletions(-) diff --git a/plans/backend/06-ongoing-automation.md b/plans/backend/06-ongoing-automation.md index efb70a32f..40c74c6e7 100644 --- a/plans/backend/06-ongoing-automation.md +++ b/plans/backend/06-ongoing-automation.md @@ -1,25 +1,65 @@ # Phase 6 — Ongoing Automation (chat-native "keep watching") -> **Design deferred — placeholder.** The mechanism is designed separately, after `04`/`05`/`07`. -> Depends on `04` (the verbs it re-invokes) and `05` (the chat surface + delivery channel). +> Depends on `04` (the verbs it re-invokes), `05` (the agent tools), and `07` (the `intelligence_agent`). +> Reuses the existing chat + automations machinery; adds no parallel engine. ## Objective -Support "keep watching": a persistent, ongoing chat where the agent periodically re-invokes scraper verbs -and drops results into the session. The agent derives what's new by reading the chat history (time-based -search over prior tool outputs in context). +Support "keep watching": a persistent chat where the agent periodically re-invokes scraper verbs and drops +results into the session. "What changed" is the agent reading its own prior turns from the durable +checkpoint — no diff store. -## Open design questions (resolve together) +## Mechanism -1. **Periodicity driver** — the existing automations schedule selector, a recurring task, or a persistent - agent loop. -2. **Delivery channel** for between-turn results — existing SSE stream vs a Zero-published messages table. -3. **Context-window limit** — how far back "what changed" can reason before summarization/compaction. -4. **Loop owner** — the `07` subagent, or a thin automation wrapper that invokes the agent. -5. **Stop / pause / cost controls** for a running watch. +A **chat watch is an `Automation` bound to the current chat** — nothing more. No dedicated thread, no +thread "kind", no schema change. Start it and the chat gains a `schedule` trigger that re-posts the +question on a cadence; stop it and the automation is deleted and the chat is a normal chat again. The +chat's own checkpoint is the memory. -## Out of scope +- **`chat_message` action** — params `{ thread_id, message }`. Its handler drains + `stream_new_chat(user_query=message, chat_id=thread_id, …)` under `AuthContext.system(creator, + source="automation")` against the **current chat**. Auto-approve (system auth; CI verbs are read-only + and don't interrupt). +- **Durable memory + delivery** — `stream_new_chat` persists messages to `new_chat_messages` (already + Zero-synced to the UI) and advances the shared Postgres checkpointer keyed by `chat_id`. A scheduled run + has no SSE client; it runs server-side and is delivered via the persisted rows. "What changed" is the + agent reading the chat's own prior turns. +- **Worker-safe checkpointer** — the shared `AsyncPostgresSaver` pool binds connections to the loop that + opened them, but Celery uses a fresh loop per task (`PoolTimeout`). Dispose the checkpointer pool per + task in `run_async_celery_task`, mirroring `_dispose_shared_db_engine`, so a worker can use the *durable* + checkpointer (not `InMemorySaver`) that "what changed" requires. +- **`start_watch`** — an `intelligence_agent` tool that binds a watch to the *current* chat: it distills + the recurring question + cadence and creates the automation (`schedule` + `chat_message(thread_id = + current chat)`). +- **"Is this chat watched?"** — derived: an active automation with a `chat_message` action targeting the + chat. No stored flag. +- **Controls** — run-now = trigger a run; stop = delete the automation (chat reverts to normal). +- **Concurrency** — the checkpointer is single-writer per thread; a tick skips if the prior turn on that + chat is still running (DB `ai_responding` flag). -- The verbs → `04`. The chat surface + delivery → `05`. The agent playbook → `07`. +## Work items -> Next step: design the periodic mechanism here, then fill in Target design / Work items / Tests. +1. Durable checkpointer in workers: `_dispose_shared_checkpointer_pool` in `run_async_celery_task` + (before + after), mirroring the SQLAlchemy engine dispose. **[done]** +2. `chat_message` action: params + factory + handler (drains `stream_new_chat`); concurrency guard. **[done]** +3. Watch service: create (bind `schedule` + `chat_message` automation to a chat) / stop (delete) / + find-for-thread (is-watched) / run-now. **[done]** +4. `start_watch` tool on `intelligence_agent` (+ prompt line); binds to the current chat. **[done]** +5. Controls — chat tools (`stop_watch`, `refresh_watch`) + REST (`GET /automations/watches`, + `POST /automations/{id}/run`; stop = `DELETE /automations/{id}`). **[done]** + +## Tests + +- `run_async_celery_task` disposes the checkpointer pool before and after a task. **[done]** +- `chat_message` drains a turn on the given thread; skips when one is in-flight. **[done]** +- Watch service: create binds a `schedule` + `chat_message` automation to the chat; stop deletes it; + find-for-thread filters by plan; run-now launches the schedule trigger. **[done]** +- `start_watch` / `stop_watch` / `refresh_watch` tools act on the current chat from tool context. **[done]** +- Watch routes registered on the automations router. **[done]** +- Integration (running stack): two scheduled runs on one chat — run 2 sees run 1 in checkpoint history. + +## Deferred / out of scope + +- Zero delivery-cost optimization (signal-column + REST fetch vs full-content sync) — app-wide, separate. +- Server-side turns surviving browser navigation for *interactive* chat ("zombie streaming"). +- `start_watch` cadence UX refinements. Verbs/doors → `04`/`05`; agent playbook → `07`. From b993bf31d9a427008229587d709ee4aa441a3e3d Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 17:54:31 +0200 Subject: [PATCH 25/64] feat(automations): add chat_message action --- .../automations/actions/builtin/__init__.py | 2 +- .../actions/builtin/chat_message/__init__.py | 15 +++++ .../builtin/chat_message/definition.py | 18 ++++++ .../actions/builtin/chat_message/factory.py | 23 ++++++++ .../actions/builtin/chat_message/invoke.py | 59 +++++++++++++++++++ .../actions/builtin/chat_message/params.py | 21 +++++++ 6 files changed, 137 insertions(+), 1 deletion(-) create mode 100644 surfsense_backend/app/automations/actions/builtin/chat_message/__init__.py create mode 100644 surfsense_backend/app/automations/actions/builtin/chat_message/definition.py create mode 100644 surfsense_backend/app/automations/actions/builtin/chat_message/factory.py create mode 100644 surfsense_backend/app/automations/actions/builtin/chat_message/invoke.py create mode 100644 surfsense_backend/app/automations/actions/builtin/chat_message/params.py diff --git a/surfsense_backend/app/automations/actions/builtin/__init__.py b/surfsense_backend/app/automations/actions/builtin/__init__.py index f3d21a044..19a93dbfc 100644 --- a/surfsense_backend/app/automations/actions/builtin/__init__.py +++ b/surfsense_backend/app/automations/actions/builtin/__init__.py @@ -2,4 +2,4 @@ from __future__ import annotations -from . import agent_task # noqa: F401 +from . import agent_task, chat_message # noqa: F401 diff --git a/surfsense_backend/app/automations/actions/builtin/chat_message/__init__.py b/surfsense_backend/app/automations/actions/builtin/chat_message/__init__.py new file mode 100644 index 000000000..6ca297493 --- /dev/null +++ b/surfsense_backend/app/automations/actions/builtin/chat_message/__init__.py @@ -0,0 +1,15 @@ +"""``chat_message`` action: post one turn into an existing chat thread. + +Imports ``definition`` for its side-effect (self-registration on the actions +registry) and re-exports ``build_handler`` for direct consumers. +""" + +from __future__ import annotations + +from .factory import build_handler +from .params import ChatMessageActionParams + +__all__ = ["ChatMessageActionParams", "build_handler"] + +# Side-effect: register on the actions store. +from . import definition # noqa: F401 diff --git a/surfsense_backend/app/automations/actions/builtin/chat_message/definition.py b/surfsense_backend/app/automations/actions/builtin/chat_message/definition.py new file mode 100644 index 000000000..5652d28c5 --- /dev/null +++ b/surfsense_backend/app/automations/actions/builtin/chat_message/definition.py @@ -0,0 +1,18 @@ +"""``chat_message`` ``ActionDefinition`` registration.""" + +from __future__ import annotations + +from ...store import register_action +from ...types import ActionDefinition +from .factory import build_handler +from .params import ChatMessageActionParams + +CHAT_MESSAGE_ACTION = ActionDefinition( + type="chat_message", + name="Chat message", + description="Post a message into an existing chat thread and run one durable, persisted agent turn.", + params_model=ChatMessageActionParams, + build_handler=build_handler, +) + +register_action(CHAT_MESSAGE_ACTION) diff --git a/surfsense_backend/app/automations/actions/builtin/chat_message/factory.py b/surfsense_backend/app/automations/actions/builtin/chat_message/factory.py new file mode 100644 index 000000000..494ea5ffc --- /dev/null +++ b/surfsense_backend/app/automations/actions/builtin/chat_message/factory.py @@ -0,0 +1,23 @@ +"""Bind ``ActionContext`` to a callable that runs one ``chat_message`` step.""" + +from __future__ import annotations + +from typing import Any + +from ...types import ActionContext, ActionHandler +from .invoke import run_chat_message +from .params import ChatMessageActionParams + + +def build_handler(ctx: ActionContext) -> ActionHandler: + """Return a handler closure that validates params and posts the turn.""" + + async def handle(params: dict[str, Any]) -> dict[str, Any]: + validated = ChatMessageActionParams.model_validate(params) + return await run_chat_message( + ctx=ctx, + thread_id=validated.thread_id, + message=validated.message, + ) + + return handle diff --git a/surfsense_backend/app/automations/actions/builtin/chat_message/invoke.py b/surfsense_backend/app/automations/actions/builtin/chat_message/invoke.py new file mode 100644 index 000000000..596ccd60c --- /dev/null +++ b/surfsense_backend/app/automations/actions/builtin/chat_message/invoke.py @@ -0,0 +1,59 @@ +"""Run one ``chat_message`` step by draining ``stream_new_chat``. + +Reuses the interactive chat turn: it persists the messages and advances the +durable checkpointer for the thread, so a scheduled tick shares the same +conversation memory as a user turn. The SSE frames have no client, so drain them. +""" + +from __future__ import annotations + +from typing import Any + +from app.auth.context import AuthContext +from app.db import ChatVisibility, User +from app.services.chat_session_state_service import get_session_state +from app.tasks.chat.streaming.flows.new_chat.orchestrator import stream_new_chat + +from ...types import ActionContext + + +async def run_chat_message( + *, + ctx: ActionContext, + thread_id: int, + message: str, +) -> dict[str, Any]: + """Post ``message`` into ``thread_id`` and run one durable agent turn. + + Skips when a turn is already in flight on the thread, so a slow tick can't + overlap the next scheduled fire. + """ + state = await get_session_state(ctx.session, thread_id) + if state is not None and state.ai_responding_to_user_id is not None: + return {"thread_id": thread_id, "frames": 0, "skipped": "in_flight"} + + user_id = str(ctx.creator_user_id) if ctx.creator_user_id else None + auth_context: AuthContext | None = None + if ctx.creator_user_id: + user = await ctx.session.get(User, ctx.creator_user_id) + if user is not None: + auth_context = AuthContext.system(user, source="automation") + + llm_config_id = ctx.chat_model_id if ctx.chat_model_id is not None else -1 + request_id = f"automation:{ctx.run_id}:{ctx.step_id}" + + frames = 0 + async for _sse in stream_new_chat( + user_query=message, + workspace_id=ctx.workspace_id, + chat_id=thread_id, + user_id=user_id, + llm_config_id=llm_config_id, + needs_history_bootstrap=False, + thread_visibility=ChatVisibility.PRIVATE, + auth_context=auth_context, + request_id=request_id, + ): + frames += 1 + + return {"thread_id": thread_id, "frames": frames, "skipped": None} diff --git a/surfsense_backend/app/automations/actions/builtin/chat_message/params.py b/surfsense_backend/app/automations/actions/builtin/chat_message/params.py new file mode 100644 index 000000000..61768bc13 --- /dev/null +++ b/surfsense_backend/app/automations/actions/builtin/chat_message/params.py @@ -0,0 +1,21 @@ +"""``ChatMessageActionParams`` — params for the ``chat_message`` action type.""" + +from __future__ import annotations + +from pydantic import BaseModel, ConfigDict, Field + + +class ChatMessageActionParams(BaseModel): + """Post one turn into an existing chat thread from an automation step.""" + + model_config = ConfigDict(extra="forbid") + + thread_id: int = Field( + ..., + description="NewChatThread id (the LangGraph thread) to post the turn into.", + ) + message: str = Field( + ..., + min_length=1, + description="Message sent as the user turn; rendered at execute time.", + ) From 2c9862c3156c399a25c457bb246b65eeb277894b Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 17:54:31 +0200 Subject: [PATCH 26/64] feat(automations): add chat watch service --- .../app/automations/services/chat_watch.py | 152 ++++++++++++++++++ 1 file changed, 152 insertions(+) create mode 100644 surfsense_backend/app/automations/services/chat_watch.py diff --git a/surfsense_backend/app/automations/services/chat_watch.py b/surfsense_backend/app/automations/services/chat_watch.py new file mode 100644 index 000000000..5f72ca98e --- /dev/null +++ b/surfsense_backend/app/automations/services/chat_watch.py @@ -0,0 +1,152 @@ +"""A chat watch is an automation bound to a chat: a ``schedule`` trigger firing +a single ``chat_message`` step that re-posts the question into the same thread. + +Starting a watch creates that automation; stopping deletes it. Whether a chat +is watched is derived from the plan (``plan_targets_thread``), not a stored flag. +""" + +from __future__ import annotations + +from typing import Any + +from fastapi import HTTPException + +from app.automations.persistence.enums.trigger_type import TriggerType +from app.automations.persistence.models.automation import Automation +from app.automations.persistence.models.run import AutomationRun +from app.automations.persistence.models.trigger import AutomationTrigger +from app.automations.schemas.api import AutomationCreate, TriggerCreate +from app.automations.schemas.definition import AutomationDefinition, PlanStep +from app.automations.services.automation import AutomationService + +WATCH_ACTION_TYPE = "chat_message" +_WATCH_STEP_ID = "watch" +_NAME_MAX = 200 +# Watches per chat are few; one generous page finds them all. +_WATCH_SCAN_LIMIT = 500 + + +def _derive_name(message: str) -> str: + """A short, human name for the watch from the question it re-asks.""" + condensed = " ".join(message.split()) + label = condensed[:80].rstrip() + return f"Watch: {label}" if label else "Watch" + + +def _build_watch_payload( + *, + workspace_id: int, + thread_id: int, + message: str, + cron: str, + timezone: str, + name: str | None, + description: str | None, +) -> AutomationCreate: + watch_name = (name or _derive_name(message))[:_NAME_MAX] + return AutomationCreate( + workspace_id=workspace_id, + name=watch_name, + description=description, + definition=AutomationDefinition( + name=watch_name, + plan=[ + PlanStep( + step_id=_WATCH_STEP_ID, + action=WATCH_ACTION_TYPE, + params={"thread_id": thread_id, "message": message}, + ) + ], + ), + triggers=[ + TriggerCreate( + type=TriggerType.SCHEDULE, + params={"cron": cron, "timezone": timezone}, + enabled=True, + ) + ], + ) + + +async def create_watch( + service: AutomationService, + *, + workspace_id: int, + thread_id: int, + message: str, + cron: str, + timezone: str, + name: str | None = None, + description: str | None = None, +) -> Automation: + """Bind a schedule + chat_message watch automation to ``thread_id``.""" + payload = _build_watch_payload( + workspace_id=workspace_id, + thread_id=thread_id, + message=message, + cron=cron, + timezone=timezone, + name=name, + description=description, + ) + return await service.create(payload) + + +async def stop_watch(service: AutomationService, *, automation_id: int) -> None: + """Stop a watch by deleting its automation; the chat reverts to normal.""" + await service.delete(automation_id) + + +def plan_targets_thread(definition: dict[str, Any] | None, thread_id: int) -> bool: + """Whether an automation's plan re-posts into ``thread_id`` (i.e. is a watch). + + Pure predicate over the persisted ``definition`` JSON so callers can filter + a workspace's automations without a DB round-trip per row. + """ + for step in (definition or {}).get("plan", []): + if step.get("action") != WATCH_ACTION_TYPE: + continue + if (step.get("params") or {}).get("thread_id") == thread_id: + return True + return False + + +async def find_watches_for_thread( + service: AutomationService, + *, + workspace_id: int, + thread_id: int, +) -> list[Automation]: + """Return the workspace automations that watch ``thread_id`` (i.e. its watches).""" + automations, _total = await service.list( + workspace_id=workspace_id, limit=_WATCH_SCAN_LIMIT, offset=0 + ) + return [a for a in automations if plan_targets_thread(a.definition, thread_id)] + + +def schedule_trigger(automation: Automation) -> AutomationTrigger | None: + """The automation's ``schedule`` trigger row, or ``None``.""" + for trigger in automation.triggers: + if trigger.type == TriggerType.SCHEDULE: + return trigger + return None + + +async def run_watch_now( + service: AutomationService, + *, + automation_id: int, +) -> AutomationRun: + """Enqueue an immediate run of a watch (a manual refresh).""" + # Lazy: launch_run pulls in the automation task graph, which imports + # multi_agent_chat; this module is imported from there, so a top-level + # import would cycle. + from app.automations.dispatch import launch as launch_mod + + automation = await service.get(automation_id) + trigger = schedule_trigger(automation) + if trigger is None: + raise HTTPException( + status_code=422, detail="watch has no schedule trigger to run" + ) + return await launch_mod.launch_run(session=service.session, trigger=trigger) From c0c0b13029af35f99c65361e97a3f8dcf6e9cc58 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 17:54:31 +0200 Subject: [PATCH 27/64] feat(automations): add watch rest routes --- .../app/automations/api/__init__.py | 4 ++ .../app/automations/api/chat_watch.py | 50 +++++++++++++++++++ 2 files changed, 54 insertions(+) create mode 100644 surfsense_backend/app/automations/api/chat_watch.py diff --git a/surfsense_backend/app/automations/api/__init__.py b/surfsense_backend/app/automations/api/__init__.py index a18e91a95..fa9f1ec05 100644 --- a/surfsense_backend/app/automations/api/__init__.py +++ b/surfsense_backend/app/automations/api/__init__.py @@ -5,10 +5,14 @@ from __future__ import annotations from fastapi import APIRouter from .automation import router as automation_router +from .chat_watch import router as watch_router from .run import router as run_router from .trigger import router as trigger_router router = APIRouter() +# Before automation_router so the literal ``/automations/watches`` path is not +# shadowed by the ``/automations/{automation_id}`` route. +router.include_router(watch_router) router.include_router(automation_router) router.include_router(trigger_router) router.include_router(run_router) diff --git a/surfsense_backend/app/automations/api/chat_watch.py b/surfsense_backend/app/automations/api/chat_watch.py new file mode 100644 index 000000000..1b5348f1a --- /dev/null +++ b/surfsense_backend/app/automations/api/chat_watch.py @@ -0,0 +1,50 @@ +"""HTTP routes for chat watches (automations bound to a chat thread). + +A watch is an automation whose plan re-posts a question into a chat on a +schedule. These routes let the chat UI show watch state and controls: + +* ``GET /automations/watches`` — the watches bound to a thread (is-watched + list). +* ``POST /automations/{automation_id}/run`` — run a watch now (manual refresh). + +Stopping a watch is a plain ``DELETE /automations/{automation_id}``. +""" + +from __future__ import annotations + +from fastapi import APIRouter, Depends, Query, status + +from app.automations.schemas.api import AutomationList, AutomationSummary, RunSummary +from app.automations.services import AutomationService, get_automation_service +from app.automations.services.chat_watch import find_watches_for_thread, run_watch_now + +router = APIRouter() + + +@router.get("/automations/watches", response_model=AutomationList) +async def list_watches( + workspace_id: int = Query(...), + thread_id: int = Query(...), + service: AutomationService = Depends(get_automation_service), +) -> AutomationList: + """List the watches bound to a chat thread (empty when it isn't watched).""" + watches = await find_watches_for_thread( + service, workspace_id=workspace_id, thread_id=thread_id + ) + return AutomationList( + items=[AutomationSummary.model_validate(a) for a in watches], + total=len(watches), + ) + + +@router.post( + "/automations/{automation_id}/run", + response_model=RunSummary, + status_code=status.HTTP_202_ACCEPTED, +) +async def run_watch( + automation_id: int, + service: AutomationService = Depends(get_automation_service), +) -> RunSummary: + """Enqueue an immediate run of a watch (manual refresh).""" + run = await run_watch_now(service, automation_id=automation_id) + return RunSummary.model_validate(run) From cf546b6821225faa8f6b1e8c2b5174a40005f104 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 17:54:31 +0200 Subject: [PATCH 28/64] feat(chat): add start_watch tool --- .../intelligence_agent/tools/start_watch.py | 97 +++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/start_watch.py diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/start_watch.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/start_watch.py new file mode 100644 index 000000000..30cdfe9f9 --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/start_watch.py @@ -0,0 +1,97 @@ +"""``start_watch`` tool — bind a recurring watch to the current chat. + +Creates a ``schedule`` + ``chat_message`` automation that re-posts the question +into this chat on a cadence, so the agent re-answers against the chat's memory. +""" + +from __future__ import annotations + +import logging +from typing import Any + +from fastapi import HTTPException +from langchain_core.tools import tool + +from app.auth.context import AuthContext +from app.automations.services.automation import AutomationService +from app.automations.services.chat_watch import create_watch +from app.db import async_session_maker + +logger = logging.getLogger(__name__) + + +def create_start_watch_tool( + *, + workspace_id: int | None, + thread_id: int | str | None, + auth_context: AuthContext | None, +): + """Build the ``start_watch`` tool bound to the current chat. + + ``thread_id`` and ``auth_context`` are injected from the chat session so the + model never guesses them; a fresh session is opened per call. + """ + + @tool + async def start_watch(message: str, cron: str, timezone: str) -> dict[str, Any]: + """Keep watching: re-run this question on a schedule in THIS chat. + + Use when the user wants an answer refreshed on a cadence without + re-asking (e.g. "check this page every weekday and tell me what + changed"). The watch re-posts ``message`` into this chat on the + schedule; the user sees new answers here and can say "stop watching" + to cancel. + + Args: + message: The question to re-ask each run, self-contained (include + the URL / target). Written as if the user asked it again. + cron: Five-field cron expression for the cadence + (e.g. "0 9 * * 1-5" = weekdays at 09:00). + timezone: IANA timezone for the schedule (e.g. "UTC", + "Africa/Kigali"). Ask the user if unclear. + + Returns: + ``{"status": "watching", "automation_id": int, "name": str, + "cron": str, "timezone": str}`` on success. + ``{"status": "error", "message": str}`` when the watch could not + be created (e.g. workspace model not billable for automations). + """ + if thread_id is None or auth_context is None: + return { + "status": "error", + "message": "Watches can only be started from inside a chat.", + } + + try: + async with async_session_maker() as session: + service = AutomationService(session=session, auth=auth_context) + created = await create_watch( + service, + workspace_id=int(workspace_id) if workspace_id is not None else 0, + thread_id=int(thread_id), + message=message, + cron=cron, + timezone=timezone, + ) + except HTTPException as exc: + return {"status": "error", "message": str(exc.detail)} + except Exception as exc: + from langgraph.errors import GraphInterrupt + + if isinstance(exc, GraphInterrupt): + raise + logger.exception("start_watch failed") + return {"status": "error", "message": f"could not start watch: {exc}"} + + return { + "status": "watching", + "automation_id": created.id, + "name": created.name, + "cron": cron, + "timezone": timezone, + } + + return start_watch + + +__all__ = ["create_start_watch_tool"] From 32d542f7a317ab483dd878d61b4dae9a630f1e26 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 17:54:31 +0200 Subject: [PATCH 29/64] feat(chat): add stop and refresh watch tools --- .../intelligence_agent/tools/refresh_watch.py | 82 ++++++++++++++++++ .../intelligence_agent/tools/stop_watch.py | 85 +++++++++++++++++++ 2 files changed, 167 insertions(+) create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/refresh_watch.py create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/stop_watch.py diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/refresh_watch.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/refresh_watch.py new file mode 100644 index 000000000..0e8d507b5 --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/refresh_watch.py @@ -0,0 +1,82 @@ +"""``refresh_watch`` — run the current chat's watch now (manual refresh). + +Enqueues an immediate run of every watch bound to this chat, instead of waiting +for the next scheduled fire. Operates on the current thread only. +""" + +from __future__ import annotations + +import logging +from typing import Any + +from fastapi import HTTPException +from langchain_core.tools import tool + +from app.auth.context import AuthContext +from app.automations.services.automation import AutomationService +from app.automations.services.chat_watch import find_watches_for_thread, run_watch_now +from app.db import async_session_maker + +logger = logging.getLogger(__name__) + + +def create_refresh_watch_tool( + *, + workspace_id: int | None, + thread_id: int | str | None, + auth_context: AuthContext | None, +): + """Build the ``refresh_watch`` tool bound to the current chat.""" + + @tool + async def refresh_watch() -> dict[str, Any]: + """Run THIS chat's watch now (a manual refresh). + + Use when the user says "check now", "refresh", or "run it again" + without waiting for the schedule. No arguments — it acts on the + current chat. The refreshed answer arrives as a new turn shortly after. + + Returns: + ``{"status": "refreshing", "refreshed_ids": [int], "count": int}`` + when runs were enqueued, ``{"status": "not_watching", ...}`` when + there is no watch, or ``{"status": "error", "message": str}``. + """ + if thread_id is None or auth_context is None: + return { + "status": "error", + "message": "Watches can only be managed from inside a chat.", + } + + try: + async with async_session_maker() as session: + service = AutomationService(session=session, auth=auth_context) + watches = await find_watches_for_thread( + service, + workspace_id=int(workspace_id) if workspace_id is not None else 0, + thread_id=int(thread_id), + ) + if not watches: + return { + "status": "not_watching", + "message": "This chat has no active watch to refresh.", + } + refreshed_ids = [] + for watch in watches: + await run_watch_now(service, automation_id=watch.id) + refreshed_ids.append(watch.id) + except HTTPException as exc: + return {"status": "error", "message": str(exc.detail)} + except Exception as exc: + logger.exception("refresh_watch failed") + return {"status": "error", "message": f"could not refresh watch: {exc}"} + + return { + "status": "refreshing", + "refreshed_ids": refreshed_ids, + "count": len(refreshed_ids), + } + + return refresh_watch + + +__all__ = ["create_refresh_watch_tool"] diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/stop_watch.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/stop_watch.py new file mode 100644 index 000000000..4f24c294f --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/stop_watch.py @@ -0,0 +1,85 @@ +"""``stop_watch`` — stop watching in the current chat. + +Deletes every watch automation bound to this chat, so it reverts to a normal +chat. Operates on the current thread only; the model passes no arguments. +""" + +from __future__ import annotations + +import logging +from typing import Any + +from fastapi import HTTPException +from langchain_core.tools import tool + +from app.auth.context import AuthContext +from app.automations.services.automation import AutomationService +from app.automations.services.chat_watch import ( + find_watches_for_thread, + stop_watch as stop_watch_service, +) +from app.db import async_session_maker + +logger = logging.getLogger(__name__) + + +def create_stop_watch_tool( + *, + workspace_id: int | None, + thread_id: int | str | None, + auth_context: AuthContext | None, +): + """Build the ``stop_watch`` tool bound to the current chat.""" + + @tool + async def stop_watch() -> dict[str, Any]: + """Stop watching in THIS chat (cancel the recurring re-asks). + + Use when the user says something like "stop watching", "cancel the + watch", or "you can stop checking now". The chat reverts to a normal + chat. No arguments — it acts on the current chat. + + Returns: + ``{"status": "stopped", "stopped_ids": [int], "count": int}`` when + watches were removed, ``{"status": "not_watching", ...}`` when there + was nothing to stop, or ``{"status": "error", "message": str}``. + """ + if thread_id is None or auth_context is None: + return { + "status": "error", + "message": "Watches can only be managed from inside a chat.", + } + + try: + async with async_session_maker() as session: + service = AutomationService(session=session, auth=auth_context) + watches = await find_watches_for_thread( + service, + workspace_id=int(workspace_id) if workspace_id is not None else 0, + thread_id=int(thread_id), + ) + if not watches: + return { + "status": "not_watching", + "message": "This chat has no active watch.", + } + stopped_ids = [] + for watch in watches: + await stop_watch_service(service, automation_id=watch.id) + stopped_ids.append(watch.id) + except HTTPException as exc: + return {"status": "error", "message": str(exc.detail)} + except Exception as exc: + logger.exception("stop_watch failed") + return {"status": "error", "message": f"could not stop watch: {exc}"} + + return { + "status": "stopped", + "stopped_ids": stopped_ids, + "count": len(stopped_ids), + } + + return stop_watch + + +__all__ = ["create_stop_watch_tool"] From 896f38fba12c490d198d00de9e472738d27a1a14 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 17:54:31 +0200 Subject: [PATCH 30/64] feat(chat): wire watch tools into intelligence agent --- .../intelligence_agent/system_prompt.md | 2 ++ .../intelligence_agent/tools/index.py | 22 ++++++++++++++++++- 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/system_prompt.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/system_prompt.md index 0289f40e2..60bcff29c 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/system_prompt.md +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/system_prompt.md @@ -8,6 +8,7 @@ Pull evidence from the live web with the web.* capability verbs and answer the d - `web_discover` — search the web for a query; returns ranked hits (url, title, snippet, provider). - `web_scrape` — fetch specific URLs; returns clean page content and metadata per URL. +- `start_watch` — (only when running inside a chat) keep re-asking a question on a schedule in this chat, so the user gets refreshed answers without asking again. Takes the self-contained question, a cron cadence, and an IANA timezone. @@ -15,6 +16,7 @@ Pull evidence from the live web with the web.* capability verbs and answer the d - If it does not, call `web_discover` first, pick the most relevant hits, then `web_scrape` those URLs to read them. - Batch URLs into a single `web_scrape` call when reading several pages (up to its limit) instead of many one-URL calls. - For "what changed" / monitoring requests, compare the freshly scraped values against the prior values already present in this conversation's earlier tool results, and report the concrete deltas (added, removed, changed old -> new). Do not claim a change you cannot point to. +- When the user asks to keep watching / track something on a recurring basis (e.g. "check this daily", "tell me weekly what changed"), first answer now, then call `start_watch` with the self-contained question and the cadence (cron + timezone). Ask for the cadence or timezone only if the user did not imply one. diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/index.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/index.py index f0b088dc1..98f704755 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/index.py +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/index.py @@ -11,6 +11,10 @@ from app.capabilities.access.agent import build_capability_tools from app.capabilities.web.discover.definition import WEB_DISCOVER from app.capabilities.web.scrape.definition import WEB_SCRAPE +from .refresh_watch import create_refresh_watch_tool +from .start_watch import create_start_watch_tool +from .stop_watch import create_stop_watch_tool + NAME = "intelligence_agent" RULESET = Ruleset(origin=NAME, rules=[]) @@ -22,7 +26,23 @@ def load_tools( *, dependencies: dict[str, Any] | None = None, **kwargs: Any ) -> list[BaseTool]: d = {**(dependencies or {}), **kwargs} - return build_capability_tools( + tools: list[BaseTool] = build_capability_tools( workspace_id=d.get("workspace_id"), capabilities=_CI_VERBS, ) + + # Watch tools bind a recurring automation to the current chat, so they are + # only offered when we have a chat to bind to and auth to manage it. + thread_id = d.get("thread_id") + auth_context = d.get("auth_context") + if thread_id is not None and auth_context is not None: + binding = { + "workspace_id": d.get("workspace_id"), + "thread_id": thread_id, + "auth_context": auth_context, + } + tools.append(create_start_watch_tool(**binding)) + tools.append(create_stop_watch_tool(**binding)) + tools.append(create_refresh_watch_tool(**binding)) + + return tools From 3278f46585bbd70f1e86abb40469a28f1b2ad28d Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 17:54:31 +0200 Subject: [PATCH 31/64] fix(tasks): dispose checkpointer pool per celery task --- .../builtin/agent_task/dependencies.py | 13 +--- .../app/tasks/celery_tasks/__init__.py | 17 +++++ .../unit/tasks/test_celery_async_runner.py | 75 +++++++++++++++++++ 3 files changed, 95 insertions(+), 10 deletions(-) diff --git a/surfsense_backend/app/automations/actions/builtin/agent_task/dependencies.py b/surfsense_backend/app/automations/actions/builtin/agent_task/dependencies.py index 7d8ee6fa1..2dca821d4 100644 --- a/surfsense_backend/app/automations/actions/builtin/agent_task/dependencies.py +++ b/surfsense_backend/app/automations/actions/builtin/agent_task/dependencies.py @@ -84,16 +84,9 @@ async def build_dependencies( connector_service = await setup_connector_service( session, workspace_id=workspace_id ) - # Per-task InMemorySaver: the shared Postgres checkpointer's connection - # pool binds connections to the loop that opened them, but Celery uses a - # fresh loop per task, so the next task hangs 30s on a dead-loop connection - # (`PoolTimeout`). InMemorySaver has no pool and dies with the task — fine - # while runs are one-shot (the checkpoint only spans one graph execution). - # - # TODO(checkpointer): when runs need durability (crash-resume or HITL - # interrupt/resume across tasks), dispose the checkpointer pool around each - # Celery task in `run_async_celery_task` — as `_dispose_shared_db_engine` - # already does for the SQLAlchemy pool — then use the shared checkpointer. + # One-shot runs on a fresh thread don't outlive a single execution, so an + # in-memory checkpointer suffices. Ongoing chat-thread turns that need + # durable memory use the shared Postgres checkpointer via stream_new_chat. checkpointer = InMemorySaver() return AgentDependencies( llm=llm, diff --git a/surfsense_backend/app/tasks/celery_tasks/__init__.py b/surfsense_backend/app/tasks/celery_tasks/__init__.py index a1113884f..f2401d44e 100644 --- a/surfsense_backend/app/tasks/celery_tasks/__init__.py +++ b/surfsense_backend/app/tasks/celery_tasks/__init__.py @@ -94,6 +94,21 @@ def _dispose_shared_db_engine(loop: asyncio.AbstractEventLoop) -> None: logger.warning("Shared DB engine dispose() failed", exc_info=True) +def _dispose_shared_checkpointer_pool(loop: asyncio.AbstractEventLoop) -> None: + """Drop the shared checkpointer pool so the next task opens a fresh one. + + The durable ``AsyncPostgresSaver`` pool binds asyncpg connections to the + loop that opened them; a later task on a fresh loop stalls on a stale + connection (``PoolTimeout``). Failure is logged, not raised. + """ + try: + from app.agents.chat.runtime.checkpointer import close_checkpointer + + loop.run_until_complete(close_checkpointer()) + except Exception: + logger.warning("Shared checkpointer pool dispose() failed", exc_info=True) + + T = TypeVar("T") @@ -129,11 +144,13 @@ def run_async_celery_task[T](coro_factory: Callable[[], Awaitable[T]]) -> T: # Defense-in-depth: prior task may have crashed before # disposing. Idempotent — no-op if pool is already empty. _dispose_shared_db_engine(loop) + _dispose_shared_checkpointer_pool(loop) return loop.run_until_complete(coro_factory()) finally: # Drop any connections this task opened so they don't leak # into the next task's loop. _dispose_shared_db_engine(loop) + _dispose_shared_checkpointer_pool(loop) with contextlib.suppress(Exception): loop.run_until_complete(loop.shutdown_asyncgens()) with contextlib.suppress(Exception): diff --git a/surfsense_backend/tests/unit/tasks/test_celery_async_runner.py b/surfsense_backend/tests/unit/tasks/test_celery_async_runner.py index 2342dd8da..761c12cd8 100644 --- a/surfsense_backend/tests/unit/tasks/test_celery_async_runner.py +++ b/surfsense_backend/tests/unit/tasks/test_celery_async_runner.py @@ -293,6 +293,81 @@ def test_runner_runs_shutdown_asyncgens_before_close() -> None: gc.collect() +@contextmanager +def _patch_checkpointer_close(recorder: list[int]) -> Iterator[None]: + """Patch ``close_checkpointer`` to record the loop it runs on. + + The runner imports it lazily, so we patch the attribute on the + already-loaded checkpointer module (same trick as the engine stub). + """ + import app.agents.chat.runtime.checkpointer as cp + + original = cp.close_checkpointer + + async def _stub() -> None: + recorder.append(id(asyncio.get_running_loop())) + + cp.close_checkpointer = _stub # type: ignore[assignment] + try: + yield + finally: + cp.close_checkpointer = original # type: ignore[assignment] + + +def test_runner_disposes_checkpointer_pool_around_call() -> None: + """The durable checkpointer's loop-bound pool must be dropped before + and after each task, on the task's own fresh loop — the fix that lets + a worker use the shared AsyncPostgresSaver instead of InMemorySaver. + """ + from app.tasks.celery_tasks import run_async_celery_task + + engine_stub = _StaleLoopEngine() + cp_loops: list[int] = [] + + async def _body() -> str: + # Both the engine and the checkpointer are disposed once before we run. + assert len(cp_loops) == 1 + return "ok" + + with _patch_shared_engine(engine_stub), _patch_checkpointer_close(cp_loops): + assert run_async_celery_task(_body) == "ok" + + # Once before the body, once after (finally). + assert len(cp_loops) == 2 + # Both ran on the SAME fresh loop, matching the engine's dispose loop. + assert cp_loops[0] == cp_loops[1] + assert cp_loops[0] == engine_stub.dispose_loop_ids[0] + + +def test_runner_swallows_checkpointer_dispose_errors() -> None: + """A flaky checkpointer dispose must never take down a celery task.""" + from app.tasks.celery_tasks import run_async_celery_task + + engine_stub = _StaleLoopEngine() + + import app.agents.chat.runtime.checkpointer as cp + + original = cp.close_checkpointer + calls = {"n": 0} + + async def _angry() -> None: + calls["n"] += 1 + raise RuntimeError("checkpointer dispose blew up") + + cp.close_checkpointer = _angry # type: ignore[assignment] + + async def _body() -> int: + return 42 + + try: + with _patch_shared_engine(engine_stub): + assert run_async_celery_task(_body) == 42 + finally: + cp.close_checkpointer = original # type: ignore[assignment] + + assert calls["n"] == 2 # before + after both attempted, both swallowed + + def test_runner_uses_proactor_loop_on_windows() -> None: """On Windows the celery worker preselects a Proactor policy so subprocess (ffmpeg) calls work. The helper must not silently fall From fecb4fa05e8788aefa971a7fed5220f2d3bf4cbb Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 17:54:31 +0200 Subject: [PATCH 32/64] test(automations): add integration harness --- .../tests/integration/automations/__init__.py | 0 .../tests/integration/automations/conftest.py | 88 +++++++++++++++++++ 2 files changed, 88 insertions(+) create mode 100644 surfsense_backend/tests/integration/automations/__init__.py create mode 100644 surfsense_backend/tests/integration/automations/conftest.py diff --git a/surfsense_backend/tests/integration/automations/__init__.py b/surfsense_backend/tests/integration/automations/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/surfsense_backend/tests/integration/automations/conftest.py b/surfsense_backend/tests/integration/automations/conftest.py new file mode 100644 index 000000000..74c9e1808 --- /dev/null +++ b/surfsense_backend/tests/integration/automations/conftest.py @@ -0,0 +1,88 @@ +"""Shared fixtures for automations integration tests. + +Bridges the code-under-test to the transactional ``db_session`` so real +behavior runs against real Postgres and rolls back at test end: + +* ``client`` — httpx over ASGI with ``get_async_session``/``get_auth_context`` + overridden to the test session + owner. +* ``tools_use_test_session`` — the watch tools open their own + ``async_session_maker()``; repoint it at ``db_session`` so they join the + test transaction (and can see the fixture's uncommitted membership/models). +* ``enqueue_spy`` — capture ``automation_run_execute.apply_async`` so run-now + can be asserted without a Redis broker. +""" + +from __future__ import annotations + +from collections.abc import AsyncGenerator +from contextlib import asynccontextmanager + +import httpx +import pytest +import pytest_asyncio +from httpx import ASGITransport +from sqlalchemy.ext.asyncio import AsyncSession + +from app.app import app +from app.auth.context import AuthContext +from app.db import User, get_async_session +from app.users import get_auth_context + + +@pytest_asyncio.fixture +async def client( + db_session: AsyncSession, db_user: User +) -> AsyncGenerator[httpx.AsyncClient, None]: + async def override_session() -> AsyncGenerator[AsyncSession, None]: + yield db_session + + async def override_auth() -> AuthContext: + return AuthContext.session(db_user) + + previous = app.dependency_overrides.copy() + app.dependency_overrides[get_async_session] = override_session + app.dependency_overrides[get_auth_context] = override_auth + try: + async with httpx.AsyncClient( + transport=ASGITransport(app=app), + base_url="http://test", + timeout=30.0, + follow_redirects=False, + ) as c: + yield c + finally: + app.dependency_overrides.clear() + app.dependency_overrides.update(previous) + + +@pytest.fixture +def tools_use_test_session(monkeypatch, db_session: AsyncSession) -> None: + """Make the watch tools' ``async_session_maker()`` yield the test session.""" + + @asynccontextmanager + async def _session_cm(): + yield db_session # owned by the outer fixture; do not close + + from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools import ( + refresh_watch, + start_watch, + stop_watch, + ) + + for module in (start_watch, stop_watch, refresh_watch): + monkeypatch.setattr(module, "async_session_maker", _session_cm) + + +@pytest.fixture +def enqueue_spy(monkeypatch) -> list[dict]: + """Capture Celery enqueues so run-now needs no broker.""" + import app.automations.dispatch.launch as launch_mod + + calls: list[dict] = [] + + def _spy(*args, **kwargs): + calls.append({"args": args, "kwargs": kwargs}) + return None + + monkeypatch.setattr(launch_mod.automation_run_execute, "apply_async", _spy) + return calls From 61412163fd03ad8128c1a09ffe349617de292b9a Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 17:54:31 +0200 Subject: [PATCH 33/64] test(automations): cover chat_message action --- .../automations/actions/__init__.py | 0 .../automations/actions/builtin/__init__.py | 0 .../actions/builtin/chat_message/__init__.py | 0 .../builtin/chat_message/test_chat_message.py | 118 ++++++++++++++++++ .../actions/builtin/chat_message/__init__.py | 0 .../builtin/chat_message/test_chat_message.py | 48 +++++++ 6 files changed, 166 insertions(+) create mode 100644 surfsense_backend/tests/integration/automations/actions/__init__.py create mode 100644 surfsense_backend/tests/integration/automations/actions/builtin/__init__.py create mode 100644 surfsense_backend/tests/integration/automations/actions/builtin/chat_message/__init__.py create mode 100644 surfsense_backend/tests/integration/automations/actions/builtin/chat_message/test_chat_message.py create mode 100644 surfsense_backend/tests/unit/automations/actions/builtin/chat_message/__init__.py create mode 100644 surfsense_backend/tests/unit/automations/actions/builtin/chat_message/test_chat_message.py diff --git a/surfsense_backend/tests/integration/automations/actions/__init__.py b/surfsense_backend/tests/integration/automations/actions/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/surfsense_backend/tests/integration/automations/actions/builtin/__init__.py b/surfsense_backend/tests/integration/automations/actions/builtin/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/surfsense_backend/tests/integration/automations/actions/builtin/chat_message/__init__.py b/surfsense_backend/tests/integration/automations/actions/builtin/chat_message/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/surfsense_backend/tests/integration/automations/actions/builtin/chat_message/test_chat_message.py b/surfsense_backend/tests/integration/automations/actions/builtin/chat_message/test_chat_message.py new file mode 100644 index 000000000..7ae5218ab --- /dev/null +++ b/surfsense_backend/tests/integration/automations/actions/builtin/chat_message/test_chat_message.py @@ -0,0 +1,118 @@ +"""Integration tests for the ``chat_message`` action's concurrency guard. + +The guard reads real ``ChatSessionState`` (seeded via the production +``set_ai_responding`` helper) to decide whether a scheduled watch tick may run. +This proves the skip decision against real persistence — the part a fake +``get_session_state`` could silently get wrong. + +The turn's actual work (``stream_new_chat``) resolves an LLM from the DB and +runs the agent; that end-to-end drain is a running-stack concern. Here it is +replaced by a spy so we can assert the guard's two outcomes: skip without +streaming when a turn is in flight, and stream (with system auth) when not. +""" + +from __future__ import annotations + +from typing import Any + +import pytest +import pytest_asyncio +from sqlalchemy.ext.asyncio import AsyncSession + +from app.automations.actions.builtin.chat_message import invoke as invoke_mod +from app.automations.actions.types import ActionContext +from app.db import ChatVisibility, NewChatThread, User, Workspace +from app.services.chat_session_state_service import ( + clear_ai_responding, + set_ai_responding, +) + +pytestmark = pytest.mark.integration + + +@pytest_asyncio.fixture +async def thread( + db_session: AsyncSession, db_user: User, db_workspace: Workspace +) -> NewChatThread: + row = NewChatThread( + title="Watched chat", + workspace_id=db_workspace.id, + created_by_id=db_user.id, + visibility=ChatVisibility.PRIVATE, + ) + db_session.add(row) + await db_session.flush() + return row + + +def _ctx(session: AsyncSession, *, workspace_id: int, creator_id) -> ActionContext: + return ActionContext( + session=session, + run_id=1, + step_id="watch", + workspace_id=workspace_id, + creator_user_id=creator_id, + chat_model_id=1, + ) + + +async def test_skips_without_streaming_when_a_turn_is_in_flight( + db_session: AsyncSession, + db_user: User, + db_workspace: Workspace, + thread: NewChatThread, + monkeypatch: pytest.MonkeyPatch, +) -> None: + await set_ai_responding(db_session, thread.id, db_user.id) + + streamed = {"called": False} + + async def _spy(**_kwargs: Any): + streamed["called"] = True + yield "data: x\n\n" + + monkeypatch.setattr(invoke_mod, "stream_new_chat", _spy) + + out = await invoke_mod.run_chat_message( + ctx=_ctx(db_session, workspace_id=db_workspace.id, creator_id=db_user.id), + thread_id=thread.id, + message="tick", + ) + + assert out["skipped"] == "in_flight" + assert out["frames"] == 0 + assert streamed["called"] is False + + +async def test_streams_under_system_auth_when_no_turn_in_flight( + db_session: AsyncSession, + db_user: User, + db_workspace: Workspace, + thread: NewChatThread, + monkeypatch: pytest.MonkeyPatch, +) -> None: + # Ensure the thread has no in-flight turn. + await clear_ai_responding(db_session, thread.id) + + captured: dict[str, Any] = {} + + async def _spy(**kwargs: Any): + captured.update(kwargs) + for frame in ("data: a\n\n", "data: b\n\n"): + yield frame + + monkeypatch.setattr(invoke_mod, "stream_new_chat", _spy) + + out = await invoke_mod.run_chat_message( + ctx=_ctx(db_session, workspace_id=db_workspace.id, creator_id=db_user.id), + thread_id=thread.id, + message="what changed?", + ) + + assert out["skipped"] is None + assert out["frames"] == 2 + assert captured["chat_id"] == thread.id + assert captured["user_query"] == "what changed?" + assert captured["user_id"] == str(db_user.id) + auth = captured["auth_context"] + assert auth is not None and auth.method == "system" and auth.source == "automation" diff --git a/surfsense_backend/tests/unit/automations/actions/builtin/chat_message/__init__.py b/surfsense_backend/tests/unit/automations/actions/builtin/chat_message/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/surfsense_backend/tests/unit/automations/actions/builtin/chat_message/test_chat_message.py b/surfsense_backend/tests/unit/automations/actions/builtin/chat_message/test_chat_message.py new file mode 100644 index 000000000..b70c7bf35 --- /dev/null +++ b/surfsense_backend/tests/unit/automations/actions/builtin/chat_message/test_chat_message.py @@ -0,0 +1,48 @@ +"""Pure-logic unit tests for the ``chat_message`` action. + +Only the infra-free contract lives here: the action self-registers, and its +params model requires ``thread_id`` + ``message`` and forbids extras. The +run-time behavior (the in-flight guard against real ``ChatSessionState`` and +streaming under system auth) is proven in +``tests/integration/automations/actions/builtin/chat_message/test_chat_message.py``, +against real persistence rather than a faked ``get_session_state``/``stream_new_chat``. +""" + +from __future__ import annotations + +import pytest + +pytestmark = pytest.mark.unit + + +def test_chat_message_action_is_registered_after_package_import() -> None: + import app.automations # noqa: F401 (force side-effect registrations) + from app.automations.actions.store import get_action + + definition = get_action("chat_message") + + assert definition is not None + assert definition.type == "chat_message" + + +def test_params_require_thread_id_and_message_and_forbid_extra() -> None: + from pydantic import ValidationError + + from app.automations.actions.builtin.chat_message.params import ( + ChatMessageActionParams, + ) + + ok = ChatMessageActionParams.model_validate({"thread_id": 7, "message": "hi"}) + assert ok.thread_id == 7 + assert ok.message == "hi" + + with pytest.raises(ValidationError): + ChatMessageActionParams.model_validate({"thread_id": 7, "message": ""}) + + with pytest.raises(ValidationError): + ChatMessageActionParams.model_validate({"thread_id": 7}) + + with pytest.raises(ValidationError): + ChatMessageActionParams.model_validate( + {"thread_id": 7, "message": "hi", "surprise": True} + ) From 555dc3fcade077a6d45d3f7e58b8053f8c2d29ac Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 17:54:31 +0200 Subject: [PATCH 34/64] test(automations): cover chat watch service --- .../automations/services/__init__.py | 0 .../automations/services/test_chat_watch.py | 217 ++++++++++++++++++ .../automations/services/test_chat_watch.py | 75 ++++++ 3 files changed, 292 insertions(+) create mode 100644 surfsense_backend/tests/integration/automations/services/__init__.py create mode 100644 surfsense_backend/tests/integration/automations/services/test_chat_watch.py create mode 100644 surfsense_backend/tests/unit/automations/services/test_chat_watch.py diff --git a/surfsense_backend/tests/integration/automations/services/__init__.py b/surfsense_backend/tests/integration/automations/services/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/surfsense_backend/tests/integration/automations/services/test_chat_watch.py b/surfsense_backend/tests/integration/automations/services/test_chat_watch.py new file mode 100644 index 000000000..5c5b3abb8 --- /dev/null +++ b/surfsense_backend/tests/integration/automations/services/test_chat_watch.py @@ -0,0 +1,217 @@ +"""Integration tests for the chat watch service against real Postgres. + +A watch is an automation bound to a chat. These exercise the real +``AutomationService`` (RBAC, the automation model-billing gate, and the +definition JSON round-trip) rather than a fake, so the behaviors the unit +layer can't see are proven end to end: + +* creating a watch persists a ``schedule`` + ``chat_message`` automation and + it reads back as the thread's watch; +* the billing gate rejects a watch when the workspace's models aren't + billable (Auto/free) — the exact failure a fake service would hide; +* finding watches is thread-scoped; stopping one deletes it; +* run-now refuses an automation that has no schedule trigger. +""" + +from __future__ import annotations + +import pytest +import pytest_asyncio +from fastapi import HTTPException +from sqlalchemy.ext.asyncio import AsyncSession + +from app.auth.context import AuthContext +from app.automations.persistence.enums.trigger_type import TriggerType +from app.automations.persistence.models.automation import Automation +from app.automations.persistence.models.trigger import AutomationTrigger +from app.automations.services.automation import AutomationService +from app.automations.services.chat_watch import ( + create_watch, + find_watches_for_thread, + run_watch_now, + stop_watch, +) +from app.db import ChatVisibility, NewChatThread, User, Workspace + +pytestmark = pytest.mark.integration + +_CRON = "0 9 * * 1-5" +_TZ = "UTC" + + +def _billable(workspace: Workspace) -> None: + """Give the workspace BYOK (positive id) models so automations are billable.""" + workspace.chat_model_id = 1 + workspace.image_gen_model_id = 1 + workspace.vision_model_id = 1 + + +@pytest_asyncio.fixture +async def service(db_session: AsyncSession, db_user: User) -> AutomationService: + return AutomationService(session=db_session, auth=AuthContext.session(db_user)) + + +@pytest_asyncio.fixture +async def thread( + db_session: AsyncSession, db_user: User, db_workspace: Workspace +) -> NewChatThread: + row = NewChatThread( + title="Watched chat", + workspace_id=db_workspace.id, + created_by_id=db_user.id, + visibility=ChatVisibility.PRIVATE, + ) + db_session.add(row) + await db_session.flush() + return row + + +async def test_create_watch_persists_and_reads_back_as_the_threads_watch( + service: AutomationService, + db_session: AsyncSession, + db_workspace: Workspace, + thread: NewChatThread, +) -> None: + _billable(db_workspace) + await db_session.flush() + thread_id = thread.id + + created = await create_watch( + service, + workspace_id=db_workspace.id, + thread_id=thread_id, + message="what changed on the pricing page?", + cron=_CRON, + timezone=_TZ, + ) + + assert created.id is not None + plan = created.definition["plan"] + assert len(plan) == 1 + assert plan[0]["action"] == "chat_message" + assert plan[0]["params"] == { + "thread_id": thread_id, + "message": "what changed on the pricing page?", + } + schedule = [t for t in created.triggers if t.type == TriggerType.SCHEDULE] + assert len(schedule) == 1 + assert schedule[0].params["cron"] == _CRON + + found = await find_watches_for_thread( + service, workspace_id=db_workspace.id, thread_id=thread_id + ) + assert [a.id for a in found] == [created.id] + + +async def test_create_watch_rejected_when_workspace_models_not_billable( + service: AutomationService, + db_workspace: Workspace, + thread: NewChatThread, +) -> None: + # Default workspace has no model prefs (Auto/None) -> not billable. + with pytest.raises(HTTPException) as exc: + await create_watch( + service, + workspace_id=db_workspace.id, + thread_id=thread.id, + message="track it", + cron=_CRON, + timezone=_TZ, + ) + assert exc.value.status_code == 422 + + +async def test_find_watches_is_scoped_to_the_thread( + service: AutomationService, + db_session: AsyncSession, + db_workspace: Workspace, + db_user: User, +) -> None: + _billable(db_workspace) + await db_session.flush() + + thread_a = NewChatThread( + title="A", workspace_id=db_workspace.id, created_by_id=db_user.id, + visibility=ChatVisibility.PRIVATE, + ) + thread_b = NewChatThread( + title="B", workspace_id=db_workspace.id, created_by_id=db_user.id, + visibility=ChatVisibility.PRIVATE, + ) + db_session.add_all([thread_a, thread_b]) + await db_session.flush() + + watch_a = await create_watch( + service, workspace_id=db_workspace.id, thread_id=thread_a.id, + message="a", cron=_CRON, timezone=_TZ, + ) + await create_watch( + service, workspace_id=db_workspace.id, thread_id=thread_b.id, + message="b", cron=_CRON, timezone=_TZ, + ) + + found = await find_watches_for_thread( + service, workspace_id=db_workspace.id, thread_id=thread_a.id + ) + assert [a.id for a in found] == [watch_a.id] + + +async def test_stop_watch_deletes_and_leaves_the_thread_unwatched( + service: AutomationService, + db_session: AsyncSession, + db_workspace: Workspace, + thread: NewChatThread, +) -> None: + _billable(db_workspace) + await db_session.flush() + thread_id = thread.id + + created = await create_watch( + service, workspace_id=db_workspace.id, thread_id=thread_id, + message="m", cron=_CRON, timezone=_TZ, + ) + + await stop_watch(service, automation_id=created.id) + + found = await find_watches_for_thread( + service, workspace_id=db_workspace.id, thread_id=thread_id + ) + assert found == [] + with pytest.raises(HTTPException) as exc: + await service.get(created.id) + assert exc.value.status_code == 404 + + +async def test_run_watch_now_rejects_automation_without_schedule_trigger( + service: AutomationService, + db_session: AsyncSession, + db_workspace: Workspace, + db_user: User, + thread: NewChatThread, +) -> None: + # Seed a watch-shaped automation whose only trigger is non-schedule. + automation = Automation( + workspace_id=db_workspace.id, + created_by_user_id=db_user.id, + name="Watch: broken", + definition={ + "name": "Watch: broken", + "plan": [ + { + "step_id": "watch", + "action": "chat_message", + "params": {"thread_id": thread.id, "message": "m"}, + } + ], + }, + version=1, + ) + automation.triggers.append( + AutomationTrigger(type=TriggerType.EVENT, params={}, enabled=True) + ) + db_session.add(automation) + await db_session.flush() + + with pytest.raises(HTTPException) as exc: + await run_watch_now(service, automation_id=automation.id) + assert exc.value.status_code == 422 diff --git a/surfsense_backend/tests/unit/automations/services/test_chat_watch.py b/surfsense_backend/tests/unit/automations/services/test_chat_watch.py new file mode 100644 index 000000000..16b3b80e2 --- /dev/null +++ b/surfsense_backend/tests/unit/automations/services/test_chat_watch.py @@ -0,0 +1,75 @@ +"""Pure-logic unit tests for the chat watch service. + +Only the deterministic, infra-free pieces live here: the plan predicate that +decides whether an automation watches a thread, and the schedule-trigger +picker. Everything that touches ``AutomationService``/Postgres (create, find, +stop, run-now) is proven against real infra in +``tests/integration/automations/services/test_chat_watch.py`` — faking the +service we own would only assert wiring and hide the model-billing gate, +RBAC, and JSON round-trip that those calls really go through. +""" + +from __future__ import annotations + +from typing import Any + +import pytest + +from app.automations.persistence.enums.trigger_type import TriggerType +from app.automations.persistence.models.trigger import AutomationTrigger +from app.automations.services.chat_watch import ( + WATCH_ACTION_TYPE, + plan_targets_thread, + schedule_trigger, +) + +pytestmark = pytest.mark.unit + + +def _definition_with_step(action: str, params: dict[str, Any]) -> dict[str, Any]: + return {"plan": [{"step_id": "s", "action": action, "params": params}]} + + +def test_plan_targets_thread_true_for_matching_chat_message_step() -> None: + definition = _definition_with_step("chat_message", {"thread_id": 55, "message": "x"}) + assert plan_targets_thread(definition, 55) is True + assert WATCH_ACTION_TYPE == "chat_message" + + +def test_plan_targets_thread_false_for_other_thread_or_action() -> None: + assert ( + plan_targets_thread( + _definition_with_step("chat_message", {"thread_id": 999}), 55 + ) + is False + ) + assert ( + plan_targets_thread( + _definition_with_step("agent_task", {"thread_id": 55}), 55 + ) + is False + ) + assert plan_targets_thread({}, 55) is False + assert plan_targets_thread(None, 55) is False + + +class _Automation: + """Minimal stand-in carrying only the ``triggers`` list the picker reads.""" + + def __init__(self, triggers: list[AutomationTrigger]) -> None: + self.triggers = triggers + + +def _trigger(type_: TriggerType) -> AutomationTrigger: + return AutomationTrigger(type=type_, params={}, enabled=True) + + +def test_schedule_trigger_picks_the_schedule_row() -> None: + schedule = _trigger(TriggerType.SCHEDULE) + automation = _Automation([_trigger(TriggerType.EVENT), schedule]) + assert schedule_trigger(automation) is schedule + + +def test_schedule_trigger_returns_none_without_a_schedule() -> None: + automation = _Automation([_trigger(TriggerType.EVENT)]) + assert schedule_trigger(automation) is None From f1ea60365c2582355fd06a14b9d8628b0ae57ac5 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 17:54:31 +0200 Subject: [PATCH 35/64] test(automations): cover watch rest routes --- .../integration/automations/api/__init__.py | 0 .../automations/api/test_watch_routes.py | 136 ++++++++++++++++++ .../tests/unit/automations/api/__init__.py | 0 3 files changed, 136 insertions(+) create mode 100644 surfsense_backend/tests/integration/automations/api/__init__.py create mode 100644 surfsense_backend/tests/integration/automations/api/test_watch_routes.py create mode 100644 surfsense_backend/tests/unit/automations/api/__init__.py diff --git a/surfsense_backend/tests/integration/automations/api/__init__.py b/surfsense_backend/tests/integration/automations/api/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/surfsense_backend/tests/integration/automations/api/test_watch_routes.py b/surfsense_backend/tests/integration/automations/api/test_watch_routes.py new file mode 100644 index 000000000..e5da2953d --- /dev/null +++ b/surfsense_backend/tests/integration/automations/api/test_watch_routes.py @@ -0,0 +1,136 @@ +"""Integration tests for the watch HTTP routes (real app, DB, auth). + +Watches are seeded through the real service and read back through the mounted +endpoints, exercising query validation, auth, and response mapping. Run-now is +asserted without a broker by spying the Celery enqueue while still persisting a +real PENDING run. +""" + +from __future__ import annotations + +import httpx +import pytest +import pytest_asyncio +from sqlalchemy import select +from sqlalchemy.ext.asyncio import AsyncSession + +from app.auth.context import AuthContext +from app.automations.persistence.enums.run_status import RunStatus +from app.automations.persistence.models.run import AutomationRun +from app.automations.services.automation import AutomationService +from app.automations.services.chat_watch import create_watch +from app.db import ChatVisibility, NewChatThread, User, Workspace + +pytestmark = pytest.mark.integration + +BASE = "/api/v1/automations" +_CRON = "0 9 * * 1-5" +_TZ = "UTC" + + +def _billable(workspace: Workspace) -> None: + workspace.chat_model_id = 1 + workspace.image_gen_model_id = 1 + workspace.vision_model_id = 1 + + +@pytest_asyncio.fixture +async def thread( + db_session: AsyncSession, db_user: User, db_workspace: Workspace +) -> NewChatThread: + row = NewChatThread( + title="Watched chat", + workspace_id=db_workspace.id, + created_by_id=db_user.id, + visibility=ChatVisibility.PRIVATE, + ) + db_session.add(row) + await db_session.flush() + return row + + +async def _seed_watch( + db_session: AsyncSession, + db_user: User, + *, + workspace_id: int, + thread_id: int, +): + service = AutomationService(session=db_session, auth=AuthContext.session(db_user)) + return await create_watch( + service, + workspace_id=workspace_id, + thread_id=thread_id, + message="what changed?", + cron=_CRON, + timezone=_TZ, + ) + + +async def test_list_watches_returns_the_threads_watch( + client: httpx.AsyncClient, + db_session: AsyncSession, + db_user: User, + db_workspace: Workspace, + thread: NewChatThread, +) -> None: + _billable(db_workspace) + await db_session.flush() + watch = await _seed_watch( + db_session, db_user, workspace_id=db_workspace.id, thread_id=thread.id + ) + + resp = await client.get( + f"{BASE}/watches", + params={"workspace_id": db_workspace.id, "thread_id": thread.id}, + ) + + assert resp.status_code == 200 + body = resp.json() + assert body["total"] == 1 + assert body["items"][0]["id"] == watch.id + + +async def test_list_watches_empty_when_thread_not_watched( + client: httpx.AsyncClient, + db_workspace: Workspace, + thread: NewChatThread, +) -> None: + resp = await client.get( + f"{BASE}/watches", + params={"workspace_id": db_workspace.id, "thread_id": thread.id}, + ) + assert resp.status_code == 200 + assert resp.json()["total"] == 0 + + +async def test_run_watch_now_enqueues_and_persists_pending_run( + client: httpx.AsyncClient, + enqueue_spy: list[dict], + db_session: AsyncSession, + db_user: User, + db_workspace: Workspace, + thread: NewChatThread, +) -> None: + _billable(db_workspace) + await db_session.flush() + watch = await _seed_watch( + db_session, db_user, workspace_id=db_workspace.id, thread_id=thread.id + ) + + resp = await client.post(f"{BASE}/{watch.id}/run") + + assert resp.status_code == 202 + assert len(enqueue_spy) == 1 + + runs = ( + ( + await db_session.execute( + select(AutomationRun).where(AutomationRun.automation_id == watch.id) + ) + ) + .scalars() + .all() + ) + assert len(runs) == 1 + assert runs[0].status == RunStatus.PENDING diff --git a/surfsense_backend/tests/unit/automations/api/__init__.py b/surfsense_backend/tests/unit/automations/api/__init__.py new file mode 100644 index 000000000..e69de29bb From f0c8d9ba5cf9cbe3b23e79a674007f91faa4c02a Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 17:54:31 +0200 Subject: [PATCH 36/64] test(chat): cover watch tools --- .../integration/automations/tools/__init__.py | 0 .../automations/tools/test_watch_tools.py | 182 ++++++++++++++++++ .../subagents/test_start_watch_tool.py | 118 ++++++++++++ .../subagents/test_watch_control_tools.py | 119 ++++++++++++ 4 files changed, 419 insertions(+) create mode 100644 surfsense_backend/tests/integration/automations/tools/__init__.py create mode 100644 surfsense_backend/tests/integration/automations/tools/test_watch_tools.py create mode 100644 surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_start_watch_tool.py create mode 100644 surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_watch_control_tools.py diff --git a/surfsense_backend/tests/integration/automations/tools/__init__.py b/surfsense_backend/tests/integration/automations/tools/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/surfsense_backend/tests/integration/automations/tools/test_watch_tools.py b/surfsense_backend/tests/integration/automations/tools/test_watch_tools.py new file mode 100644 index 000000000..03d3ca93a --- /dev/null +++ b/surfsense_backend/tests/integration/automations/tools/test_watch_tools.py @@ -0,0 +1,182 @@ +"""Integration tests for the watch chat tools against real service + Postgres. + +The tools are thin adapters the intelligence agent calls; here they run their +real path — open a session, build ``AutomationService``, and create / find / +delete / enqueue — against real persistence (RBAC + model gate included), with +only the Celery enqueue spied. This is what proves ``start_watch`` actually +binds a watch, ``stop_watch`` removes it, and ``refresh_watch`` enqueues a run. +""" + +from __future__ import annotations + +import pytest +import pytest_asyncio +from sqlalchemy.ext.asyncio import AsyncSession + +from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools.refresh_watch import ( + create_refresh_watch_tool, +) +from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools.start_watch import ( + create_start_watch_tool, +) +from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools.stop_watch import ( + create_stop_watch_tool, +) +from app.auth.context import AuthContext +from app.automations.services.automation import AutomationService +from app.automations.services.chat_watch import find_watches_for_thread +from app.db import ChatVisibility, NewChatThread, User, Workspace + +pytestmark = pytest.mark.integration + +_CRON = "0 9 * * 1-5" +_TZ = "UTC" + + +def _billable(workspace: Workspace) -> None: + workspace.chat_model_id = 1 + workspace.image_gen_model_id = 1 + workspace.vision_model_id = 1 + + +@pytest_asyncio.fixture +async def thread( + db_session: AsyncSession, db_user: User, db_workspace: Workspace +) -> NewChatThread: + row = NewChatThread( + title="Watched chat", + workspace_id=db_workspace.id, + created_by_id=db_user.id, + visibility=ChatVisibility.PRIVATE, + ) + db_session.add(row) + await db_session.flush() + return row + + +async def test_start_watch_tool_binds_a_watch_to_the_chat( + tools_use_test_session, + db_session: AsyncSession, + db_user: User, + db_workspace: Workspace, + thread: NewChatThread, +) -> None: + _billable(db_workspace) + await db_session.flush() + + tool = create_start_watch_tool( + workspace_id=db_workspace.id, + thread_id=thread.id, + auth_context=AuthContext.session(db_user), + ) + out = await tool.ainvoke( + {"message": "what changed?", "cron": _CRON, "timezone": _TZ} + ) + + assert out["status"] == "watching" + assert isinstance(out["automation_id"], int) + + service = AutomationService(session=db_session, auth=AuthContext.session(db_user)) + found = await find_watches_for_thread( + service, workspace_id=db_workspace.id, thread_id=thread.id + ) + assert [a.id for a in found] == [out["automation_id"]] + + +async def test_start_watch_tool_reports_error_when_models_not_billable( + tools_use_test_session, + db_user: User, + db_workspace: Workspace, + thread: NewChatThread, +) -> None: + # Default workspace models are Auto/None -> the automation gate rejects it. + tool = create_start_watch_tool( + workspace_id=db_workspace.id, + thread_id=thread.id, + auth_context=AuthContext.session(db_user), + ) + out = await tool.ainvoke( + {"message": "track it", "cron": _CRON, "timezone": _TZ} + ) + assert out["status"] == "error" + + +async def test_stop_watch_tool_removes_the_chats_watch( + tools_use_test_session, + db_session: AsyncSession, + db_user: User, + db_workspace: Workspace, + thread: NewChatThread, +) -> None: + _billable(db_workspace) + await db_session.flush() + auth = AuthContext.session(db_user) + + start = create_start_watch_tool( + workspace_id=db_workspace.id, thread_id=thread.id, auth_context=auth + ) + await start.ainvoke({"message": "m", "cron": _CRON, "timezone": _TZ}) + + stop = create_stop_watch_tool( + workspace_id=db_workspace.id, thread_id=thread.id, auth_context=auth + ) + out = await stop.ainvoke({}) + + assert out["status"] == "stopped" + assert out["count"] == 1 + + service = AutomationService(session=db_session, auth=auth) + found = await find_watches_for_thread( + service, workspace_id=db_workspace.id, thread_id=thread.id + ) + assert found == [] + + +async def test_stop_watch_tool_reports_not_watching_when_none( + tools_use_test_session, + db_workspace: Workspace, + db_user: User, + thread: NewChatThread, +) -> None: + stop = create_stop_watch_tool( + workspace_id=db_workspace.id, + thread_id=thread.id, + auth_context=AuthContext.session(db_user), + ) + out = await stop.ainvoke({}) + assert out["status"] == "not_watching" + + +async def test_refresh_watch_tool_enqueues_a_run( + tools_use_test_session, + enqueue_spy: list[dict], + db_session: AsyncSession, + db_user: User, + db_workspace: Workspace, + thread: NewChatThread, +) -> None: + _billable(db_workspace) + await db_session.flush() + auth = AuthContext.session(db_user) + + start = create_start_watch_tool( + workspace_id=db_workspace.id, thread_id=thread.id, auth_context=auth + ) + started = await start.ainvoke({"message": "m", "cron": _CRON, "timezone": _TZ}) + + refresh = create_refresh_watch_tool( + workspace_id=db_workspace.id, thread_id=thread.id, auth_context=auth + ) + out = await refresh.ainvoke({}) + + assert out["status"] == "refreshing" + assert out["refreshed_ids"] == [started["automation_id"]] + assert len(enqueue_spy) == 1 + + +async def test_tools_error_without_thread_or_auth() -> None: + tool = create_start_watch_tool( + workspace_id=1, thread_id=None, auth_context=None + ) + out = await tool.ainvoke({"message": "m", "cron": _CRON, "timezone": _TZ}) + assert out["status"] == "error" diff --git a/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_start_watch_tool.py b/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_start_watch_tool.py new file mode 100644 index 000000000..c532cb5e6 --- /dev/null +++ b/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_start_watch_tool.py @@ -0,0 +1,118 @@ +"""Unit tests for the ``start_watch`` tool on the intelligence_agent. + +``start_watch`` binds a recurring watch to the *current* chat: it distils the +question + cadence the agent extracted and creates a ``schedule`` + +``chat_message`` automation via the watch service. These tests fake the watch +service / session so no DB is touched; they pin that the tool forwards the +current chat id + system auth and reports a clear outcome. +""" + +from __future__ import annotations + +from typing import Any +from unittest.mock import AsyncMock, MagicMock + +import pytest + +pytestmark = pytest.mark.unit + + +class _FakeSessionCM: + async def __aenter__(self) -> Any: + return MagicMock(name="session") + + async def __aexit__(self, *_exc: Any) -> bool: + return False + + +def _patch_deps(monkeypatch: pytest.MonkeyPatch, *, created: Any) -> AsyncMock: + from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools import ( + start_watch as mod, + ) + + fake_create_watch = AsyncMock(return_value=created) + monkeypatch.setattr(mod, "create_watch", fake_create_watch) + monkeypatch.setattr(mod, "AutomationService", MagicMock(return_value="svc")) + monkeypatch.setattr(mod, "async_session_maker", lambda: _FakeSessionCM()) + return fake_create_watch + + +@pytest.mark.asyncio +async def test_start_watch_binds_watch_to_current_chat( + monkeypatch: pytest.MonkeyPatch, +) -> None: + from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools import ( + start_watch as mod, + ) + + created = MagicMock(id=123) + created.name = "Watch: prices" + fake_create_watch = _patch_deps(monkeypatch, created=created) + + auth = MagicMock() + tool = mod.create_start_watch_tool(workspace_id=3, thread_id=55, auth_context=auth) + + out = await tool.ainvoke( + {"message": "what changed on prices?", "cron": "0 9 * * 1-5", "timezone": "UTC"} + ) + + assert out["status"] == "watching" + assert out["automation_id"] == 123 + + fake_create_watch.assert_awaited_once() + kwargs = fake_create_watch.await_args.kwargs + assert kwargs["workspace_id"] == 3 + assert kwargs["thread_id"] == 55 + assert kwargs["message"] == "what changed on prices?" + assert kwargs["cron"] == "0 9 * * 1-5" + assert kwargs["timezone"] == "UTC" + + # The service is constructed with the passed-through auth context. + assert mod.AutomationService.call_args.kwargs["auth"] is auth + + +@pytest.mark.asyncio +async def test_start_watch_errors_without_thread_or_auth( + monkeypatch: pytest.MonkeyPatch, +) -> None: + from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools import ( + start_watch as mod, + ) + + fake_create_watch = _patch_deps(monkeypatch, created=MagicMock(id=1)) + + no_thread = mod.create_start_watch_tool( + workspace_id=3, thread_id=None, auth_context=MagicMock() + ) + out_no_thread = await no_thread.ainvoke( + {"message": "x", "cron": "0 9 * * *", "timezone": "UTC"} + ) + assert out_no_thread["status"] == "error" + + no_auth = mod.create_start_watch_tool( + workspace_id=3, thread_id=55, auth_context=None + ) + out_no_auth = await no_auth.ainvoke( + {"message": "x", "cron": "0 9 * * *", "timezone": "UTC"} + ) + assert out_no_auth["status"] == "error" + + fake_create_watch.assert_not_awaited() + + +def test_load_tools_includes_start_watch_only_when_bindable() -> None: + from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools.index import ( + load_tools, + ) + + bindable = load_tools( + dependencies={ + "workspace_id": 3, + "thread_id": 55, + "auth_context": MagicMock(), + } + ) + assert "start_watch" in {t.name for t in bindable} + + not_bindable = load_tools(dependencies={"workspace_id": 3}) + assert "start_watch" not in {t.name for t in not_bindable} diff --git a/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_watch_control_tools.py b/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_watch_control_tools.py new file mode 100644 index 000000000..b07f864c4 --- /dev/null +++ b/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_watch_control_tools.py @@ -0,0 +1,119 @@ +"""Unit tests for the ``stop_watch`` and ``refresh_watch`` chat tools. + +Both act on the *current* chat: they look up the watches bound to this thread +and stop them / run them now. The watch service + session are faked so no DB is +touched. +""" + +from __future__ import annotations + +from typing import Any +from unittest.mock import AsyncMock, MagicMock + +import pytest + +pytestmark = pytest.mark.unit + + +class _FakeSessionCM: + async def __aenter__(self) -> Any: + return MagicMock(name="session") + + async def __aexit__(self, *_exc: Any) -> bool: + return False + + +def _watch(automation_id: int) -> Any: + a = MagicMock() + a.id = automation_id + return a + + +@pytest.mark.asyncio +async def test_stop_watch_stops_every_watch_on_the_chat( + monkeypatch: pytest.MonkeyPatch, +) -> None: + from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools import ( + stop_watch as mod, + ) + + monkeypatch.setattr(mod, "AutomationService", MagicMock(return_value="svc")) + monkeypatch.setattr(mod, "async_session_maker", lambda: _FakeSessionCM()) + monkeypatch.setattr( + mod, "find_watches_for_thread", AsyncMock(return_value=[_watch(1), _watch(2)]) + ) + stop_service = AsyncMock() + monkeypatch.setattr(mod, "stop_watch_service", stop_service) + + tool = mod.create_stop_watch_tool( + workspace_id=3, thread_id=55, auth_context=MagicMock() + ) + out = await tool.ainvoke({}) + + assert out["status"] == "stopped" + assert sorted(out["stopped_ids"]) == [1, 2] + assert stop_service.await_count == 2 + + +@pytest.mark.asyncio +async def test_stop_watch_reports_when_nothing_to_stop( + monkeypatch: pytest.MonkeyPatch, +) -> None: + from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools import ( + stop_watch as mod, + ) + + monkeypatch.setattr(mod, "AutomationService", MagicMock(return_value="svc")) + monkeypatch.setattr(mod, "async_session_maker", lambda: _FakeSessionCM()) + monkeypatch.setattr(mod, "find_watches_for_thread", AsyncMock(return_value=[])) + monkeypatch.setattr(mod, "stop_watch_service", AsyncMock()) + + tool = mod.create_stop_watch_tool( + workspace_id=3, thread_id=55, auth_context=MagicMock() + ) + out = await tool.ainvoke({}) + + assert out["status"] == "not_watching" + + +@pytest.mark.asyncio +async def test_refresh_watch_runs_each_watch_now( + monkeypatch: pytest.MonkeyPatch, +) -> None: + from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools import ( + refresh_watch as mod, + ) + + monkeypatch.setattr(mod, "AutomationService", MagicMock(return_value="svc")) + monkeypatch.setattr(mod, "async_session_maker", lambda: _FakeSessionCM()) + monkeypatch.setattr( + mod, "find_watches_for_thread", AsyncMock(return_value=[_watch(7)]) + ) + run_now = AsyncMock() + monkeypatch.setattr(mod, "run_watch_now", run_now) + + tool = mod.create_refresh_watch_tool( + workspace_id=3, thread_id=55, auth_context=MagicMock() + ) + out = await tool.ainvoke({}) + + assert out["status"] == "refreshing" + assert out["refreshed_ids"] == [7] + run_now.assert_awaited_once() + assert run_now.await_args.kwargs["automation_id"] == 7 + + +def test_load_tools_includes_control_tools_when_bindable() -> None: + from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools.index import ( + load_tools, + ) + + tools = load_tools( + dependencies={ + "workspace_id": 3, + "thread_id": 55, + "auth_context": MagicMock(), + } + ) + names = {t.name for t in tools} + assert {"start_watch", "stop_watch", "refresh_watch"} <= names From 18e2ad13ca08ba6966c3fd0b6b8d6c819debefca Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 17:54:31 +0200 Subject: [PATCH 37/64] test(automations): cover checkpointer cross-loop durability --- .../test_checkpointer_cross_loop.py | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 surfsense_backend/tests/integration/automations/test_checkpointer_cross_loop.py diff --git a/surfsense_backend/tests/integration/automations/test_checkpointer_cross_loop.py b/surfsense_backend/tests/integration/automations/test_checkpointer_cross_loop.py new file mode 100644 index 000000000..9e3f32a28 --- /dev/null +++ b/surfsense_backend/tests/integration/automations/test_checkpointer_cross_loop.py @@ -0,0 +1,61 @@ +"""The durable checkpointer survives Celery's fresh-loop-per-task model. + +Slice 1's whole point: the shared ``AsyncPostgresSaver`` pool binds connections +to the loop that opened them, and Celery runs each task on a new loop. This +writes a checkpoint on one ``run_async_celery_task`` loop and reads it back on a +*fresh* one — proving the per-task pool dispose lets a new loop reopen and read +committed state, rather than stalling on a stale connection. + +Uses the real pool against the real (test) Postgres, so a regression in the +dispose wiring fails here, not just in production. +""" + +from __future__ import annotations + +import uuid + +import pytest +from langgraph.checkpoint.base import empty_checkpoint + +from app.agents.chat.runtime.checkpointer import close_checkpointer, get_checkpointer +from app.tasks.celery_tasks import run_async_celery_task + +pytestmark = pytest.mark.integration + + +def _config(thread_id: str) -> dict: + return {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}} + + +def test_checkpoint_written_on_one_loop_is_readable_on_a_fresh_loop() -> None: + thread_id = f"cross-loop-{uuid.uuid4()}" + config = _config(thread_id) + checkpoint = empty_checkpoint() + + async def _write() -> None: + cp = await get_checkpointer() + await cp.aput(config, checkpoint, {"source": "update", "step": 0}, {}) + + async def _read(): + cp = await get_checkpointer() + return await cp.aget_tuple(config) + + async def _cleanup() -> None: + cp = await get_checkpointer() + delete = getattr(cp, "adelete_thread", None) + if delete is not None: + await delete(thread_id) + + # Loop 1 writes and commits; run_async_celery_task disposes the pool after. + run_async_celery_task(_write) + + # Loop 2 is a brand-new event loop: a stale loop-bound pool would stall + # here (PoolTimeout). It must reopen and read the committed checkpoint. + tup = run_async_celery_task(_read) + + try: + assert tup is not None, "fresh loop could not read the prior checkpoint" + assert tup.checkpoint["id"] == checkpoint["id"] + finally: + run_async_celery_task(_cleanup) + run_async_celery_task(lambda: close_checkpointer()) From 88a94611c938d30f643a021687187baa2932bb29 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 17:54:31 +0200 Subject: [PATCH 38/64] chore(docker): add isolated deps compose for watch e2e --- docker/docker-compose.watch-e2e.yml | 52 +++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 docker/docker-compose.watch-e2e.yml diff --git a/docker/docker-compose.watch-e2e.yml b/docker/docker-compose.watch-e2e.yml new file mode 100644 index 000000000..987580477 --- /dev/null +++ b/docker/docker-compose.watch-e2e.yml @@ -0,0 +1,52 @@ +# ============================================================================= +# SurfSense — Isolated deps for the watch (Phase 6) manual E2E +# ============================================================================= +# Fresh Postgres + Redis ONLY, with their own project name, volumes, and host +# ports so they never collide with the day-to-day `surfsense-deps` stack. +# +# Backend + Celery run on the HOST (tests/e2e/run_backend.py / run_celery.py) +# pointed at these ports via env: +# DATABASE_URL=postgresql+asyncpg://postgres:postgres@localhost:5442/surfsense +# CELERY_BROKER_URL / CELERY_RESULT_BACKEND / REDIS_APP_URL=redis://localhost:6389/0 +# +# Up: docker compose -p surfsense-watch -f docker/docker-compose.watch-e2e.yml up -d +# Down: docker compose -p surfsense-watch -f docker/docker-compose.watch-e2e.yml down -v +# ============================================================================= + +name: surfsense-watch + +services: + db: + image: pgvector/pgvector:pg17 + ports: + - "5442:5432" + volumes: + - postgres_data:/var/lib/postgresql/data + environment: + - POSTGRES_USER=postgres + - POSTGRES_PASSWORD=postgres + - POSTGRES_DB=surfsense + healthcheck: + test: ["CMD-SHELL", "pg_isready -U postgres -d surfsense"] + interval: 3s + timeout: 5s + retries: 20 + + redis: + image: redis:8-alpine + ports: + - "6389:6379" + volumes: + - redis_data:/data + command: redis-server --appendonly yes + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 3s + timeout: 5s + retries: 20 + +volumes: + postgres_data: + name: surfsense-watch-postgres + redis_data: + name: surfsense-watch-redis From 790507d10719584e08f50065bd83e151912bfaed Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 18:32:42 +0200 Subject: [PATCH 39/64] feat(capabilities): meter captcha, truncate scrape, bound discover --- surfsense_backend/app/capabilities/billing.py | 57 +++++++++++++-- .../app/capabilities/web/discover/schemas.py | 2 +- .../app/capabilities/web/scrape/executor.py | 19 ++++- .../app/capabilities/web/scrape/schemas.py | 4 + .../tests/unit/capabilities/test_billing.py | 73 +++++++++++++++++++ .../capabilities/web/discover/test_schemas.py | 22 ++++++ .../capabilities/web/scrape/test_executor.py | 47 ++++++++++++ 7 files changed, 212 insertions(+), 12 deletions(-) create mode 100644 surfsense_backend/tests/unit/capabilities/web/discover/test_schemas.py diff --git a/surfsense_backend/app/capabilities/billing.py b/surfsense_backend/app/capabilities/billing.py index 20d1059bf..636920a9f 100644 --- a/surfsense_backend/app/capabilities/billing.py +++ b/surfsense_backend/app/capabilities/billing.py @@ -12,8 +12,10 @@ from app.capabilities.types import ( BillingUnit, CapabilityContext, ) +from app.config import config from app.services.token_tracking_service import record_token_usage from app.services.web_crawl_credit_service import WebCrawlCreditService +from app.utils.captcha import captcha_enabled if TYPE_CHECKING: from uuid import UUID @@ -36,29 +38,49 @@ async def gate_capability( async def _gate_web_crawl(ctx: CapabilityContext, estimated_successes: int) -> None: + """Reserve the worst-case cost: crawl successes + worst-case captcha attempts. + + Captcha budget is only reserved when solving is actually enabled — with + solving off, attempts can never happen, so reserving would wrongly block a + run for captcha that will never be attempted. Mirrors the indexer path (3d). + """ service = WebCrawlCreditService(ctx.session) - if not service.billing_enabled(): + crawl_on = service.billing_enabled() + captcha_on = service.captcha_billing_enabled() and captcha_enabled() + if not crawl_on and not captcha_on: return owner_user_id = await _resolve_workspace_owner(ctx.session, ctx.workspace_id) if owner_user_id is None: return - await service.check_credits(owner_user_id, estimated_successes) + + required_micros = 0 + if crawl_on: + required_micros += service.successes_to_micros(estimated_successes) + if captcha_on: + worst_case_attempts = estimated_successes * config.CAPTCHA_MAX_ATTEMPTS_PER_URL + required_micros += service.captcha_solves_to_micros(worst_case_attempts) + await service.check_balance(owner_user_id, required_micros) async def charge_capability( output: BillableOutput, unit: BillingUnit | None, ctx: CapabilityContext ) -> None: - """Bill the workspace owner for this result's billable successes (03c). ``None`` = free.""" + """Bill the workspace owner for this result's billable successes (03c). ``None`` = free. + + For crawl-backed verbs this also bills any captcha *attempts* (Phase 3d) as a + separate per-attempt unit — the solver charges per attempt even when the crawl + ultimately failed, so it can't ride the per-success crawl meter. + """ if unit is None: return - units = output.billable_units - if units <= 0: - return if unit is BillingUnit.WEB_CRAWL: - await _charge_web_crawl(ctx, units) + await _charge_web_crawl(ctx, output.billable_units) + await _charge_captcha(ctx, getattr(output, "captcha_attempts", 0)) async def _charge_web_crawl(ctx: CapabilityContext, successes: int) -> None: + if successes <= 0: + return service = WebCrawlCreditService(ctx.session) if not service.billing_enabled(): return @@ -77,6 +99,27 @@ async def _charge_web_crawl(ctx: CapabilityContext, successes: int) -> None: await service.charge_credits(owner_user_id, successes) +async def _charge_captcha(ctx: CapabilityContext, attempts: int) -> None: + if attempts <= 0: + return + service = WebCrawlCreditService(ctx.session) + if not service.captcha_billing_enabled(): + return + owner_user_id = await _resolve_workspace_owner(ctx.session, ctx.workspace_id) + if owner_user_id is None: + return + # Stage the audit row before charge_captcha's commit flushes both. + await record_token_usage( + ctx.session, + usage_type="web_crawl_captcha", + workspace_id=ctx.workspace_id, + user_id=owner_user_id, + cost_micros=service.captcha_solves_to_micros(attempts), + call_details={"attempts": attempts}, + ) + await service.charge_captcha(owner_user_id, attempts) + + async def _resolve_workspace_owner( session: AsyncSession, workspace_id: int ) -> UUID | None: diff --git a/surfsense_backend/app/capabilities/web/discover/schemas.py b/surfsense_backend/app/capabilities/web/discover/schemas.py index ae74d2c56..229454e7e 100644 --- a/surfsense_backend/app/capabilities/web/discover/schemas.py +++ b/surfsense_backend/app/capabilities/web/discover/schemas.py @@ -10,7 +10,7 @@ class DiscoverInput(BaseModel): description="What to search the web for, phrased in natural language." ) top_k: int = Field( - default=10, description="Maximum number of results to return (1-50)." + default=10, ge=1, le=50, description="Maximum number of results to return (1-50)." ) diff --git a/surfsense_backend/app/capabilities/web/scrape/executor.py b/surfsense_backend/app/capabilities/web/scrape/executor.py index 46f067b56..05459ee6a 100644 --- a/surfsense_backend/app/capabilities/web/scrape/executor.py +++ b/surfsense_backend/app/capabilities/web/scrape/executor.py @@ -16,18 +16,29 @@ def build_scrape_executor(engine: WebCrawlerConnector | None = None) -> Executor engine = engine or WebCrawlerConnector() async def execute(payload: ScrapeInput) -> ScrapeOutput: - rows = [_to_row(url, await engine.crawl_url(url)) for url in payload.urls] - return ScrapeOutput(rows=rows) + outcomes = [await engine.crawl_url(url) for url in payload.urls] + rows = [ + _to_row(url, outcome, payload.max_length) + for url, outcome in zip(payload.urls, outcomes, strict=True) + ] + return ScrapeOutput( + rows=rows, + captcha_attempts=sum(o.captcha_attempts for o in outcomes), + captcha_solved=sum(1 for o in outcomes if o.captcha_solved), + ) return execute -def _to_row(url: str, outcome: CrawlOutcome) -> ScrapeRow: +def _to_row(url: str, outcome: CrawlOutcome, max_length: int) -> ScrapeRow: if outcome.status is CrawlOutcomeStatus.SUCCESS and outcome.result: + content = outcome.result.get("content") + if content is not None: + content = content[:max_length] return ScrapeRow( url=url, status="success", - content=outcome.result.get("content"), + content=content, metadata=outcome.result.get("metadata"), ) status = "empty" if outcome.status is CrawlOutcomeStatus.EMPTY else "failed" diff --git a/surfsense_backend/app/capabilities/web/scrape/schemas.py b/surfsense_backend/app/capabilities/web/scrape/schemas.py index 415a64554..04faaa516 100644 --- a/surfsense_backend/app/capabilities/web/scrape/schemas.py +++ b/surfsense_backend/app/capabilities/web/scrape/schemas.py @@ -56,6 +56,10 @@ class ScrapeOutput(BaseModel): rows: list[ScrapeRow] = Field( description="One result per requested URL, in the same order." ) + # Billing-only telemetry (Phase 3d), excluded from the client-facing result: + # captcha solves are metered per attempt, independent of crawl success. + captcha_attempts: int = Field(default=0, exclude=True) + captcha_solved: int = Field(default=0, exclude=True) @property def billable_units(self) -> int: diff --git a/surfsense_backend/tests/unit/capabilities/test_billing.py b/surfsense_backend/tests/unit/capabilities/test_billing.py index 45e65c257..1f70afc56 100644 --- a/surfsense_backend/tests/unit/capabilities/test_billing.py +++ b/surfsense_backend/tests/unit/capabilities/test_billing.py @@ -92,6 +92,53 @@ async def test_charges_workspace_owner_per_successful_crawl(monkeypatch, record_ assert kwargs["cost_micros"] == 2000 +def _output_with_captcha(*statuses: str, attempts: int, solved: int) -> ScrapeOutput: + out = _output(*statuses) + out.captcha_attempts = attempts + out.captcha_solved = solved + return out + + +async def test_charges_workspace_owner_per_captcha_attempt_even_when_crawl_failed( + monkeypatch, record_usage +): + monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", True) + monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_BILLING_ENABLED", True) + monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE", 3000) + session, user = _make_session(_OWNER, balance_micros=100_000) + + # Crawl failed (no billable success) but the solver ran twice — attempts bill. + await charge_capability( + _output_with_captcha("failed", attempts=2, solved=1), + BillingUnit.WEB_CRAWL, + _ctx(session), + ) + + assert user.credit_micros_balance == 100_000 - 2 * 3000 + record_usage.assert_awaited_once() + kwargs = record_usage.await_args.kwargs + assert kwargs["usage_type"] == "web_crawl_captcha" + assert kwargs["user_id"] == _OWNER + assert kwargs["cost_micros"] == 6000 + + +async def test_captcha_billing_disabled_does_not_charge_attempts( + monkeypatch, record_usage +): + monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", True) + monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_BILLING_ENABLED", False) + session, user = _make_session(_OWNER, balance_micros=100_000) + + await charge_capability( + _output_with_captcha("failed", attempts=2, solved=1), + BillingUnit.WEB_CRAWL, + _ctx(session), + ) + + record_usage.assert_not_awaited() + assert user.credit_micros_balance == 100_000 + + async def test_no_successful_rows_is_free(monkeypatch, record_usage): monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", True) session, user = _make_session(_OWNER, balance_micros=100_000) @@ -176,6 +223,32 @@ async def test_gate_is_noop_when_disabled(monkeypatch): ) +async def test_gate_reserves_worst_case_captcha_when_solving_enabled(monkeypatch): + monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", False) + monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_BILLING_ENABLED", True) + monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE", 3000) + monkeypatch.setattr(config, "CAPTCHA_MAX_ATTEMPTS_PER_URL", 3) + monkeypatch.setattr(billing, "captcha_enabled", lambda: True) + session = _gate_session(_OWNER, balance_micros=5000) # < 1 url * 3 * 3000 + + with pytest.raises(InsufficientCreditsError): + await gate_capability( + ScrapeInput(urls=["https://a.com"]), BillingUnit.WEB_CRAWL, _ctx(session) + ) + + +async def test_gate_does_not_reserve_captcha_when_solving_disabled(monkeypatch): + monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", False) + monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_BILLING_ENABLED", True) + monkeypatch.setattr(billing, "captcha_enabled", lambda: False) + session = _gate_session(_OWNER, balance_micros=0) + + # Solving off → attempts can never happen → nothing to reserve → passes. + await gate_capability( + ScrapeInput(urls=["https://a.com"]), BillingUnit.WEB_CRAWL, _ctx(session) + ) + + async def test_gate_is_noop_for_free_verb(monkeypatch): monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", True) session = _gate_session(_OWNER, balance_micros=0) diff --git a/surfsense_backend/tests/unit/capabilities/web/discover/test_schemas.py b/surfsense_backend/tests/unit/capabilities/web/discover/test_schemas.py new file mode 100644 index 000000000..f098014d8 --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/web/discover/test_schemas.py @@ -0,0 +1,22 @@ +"""``web.discover`` I/O contract bounds.""" + +from __future__ import annotations + +import pytest +from pydantic import ValidationError + +from app.capabilities.web.discover.schemas import DiscoverInput + +pytestmark = pytest.mark.unit + + +def test_top_k_defaults_and_accepts_the_valid_range(): + assert DiscoverInput(query="q").top_k == 10 + assert DiscoverInput(query="q", top_k=1).top_k == 1 + assert DiscoverInput(query="q", top_k=50).top_k == 50 + + +@pytest.mark.parametrize("bad", [0, -1, 51, 100]) +def test_top_k_outside_1_to_50_is_rejected(bad): + with pytest.raises(ValidationError): + DiscoverInput(query="q", top_k=bad) diff --git a/surfsense_backend/tests/unit/capabilities/web/scrape/test_executor.py b/surfsense_backend/tests/unit/capabilities/web/scrape/test_executor.py index 9118b6944..b37a0346c 100644 --- a/surfsense_backend/tests/unit/capabilities/web/scrape/test_executor.py +++ b/surfsense_backend/tests/unit/capabilities/web/scrape/test_executor.py @@ -71,6 +71,53 @@ async def test_scrape_returns_one_row_per_url_in_input_order(): assert [row.content for row in out.rows] == ["A", "B", "C"] +async def test_content_longer_than_max_length_is_truncated(): + url = "https://long.com" + crawler = _FakeCrawler({url: _success("A" * 100, {"title": "Long"})}) + execute = build_scrape_executor(engine=crawler) + + out = await execute(ScrapeInput(urls=[url], max_length=10)) + + assert out.rows[0].content == "A" * 10 + + +async def test_content_within_max_length_is_untouched(): + url = "https://short.com" + crawler = _FakeCrawler({url: _success("hello", {"title": "Short"})}) + execute = build_scrape_executor(engine=crawler) + + out = await execute(ScrapeInput(urls=[url], max_length=10)) + + assert out.rows[0].content == "hello" + + +async def test_scrape_surfaces_total_captcha_attempts_for_billing(): + ok, blocked = "https://ok.com", "https://blocked.com" + crawler = _FakeCrawler( + { + ok: CrawlOutcome( + status=CrawlOutcomeStatus.SUCCESS, + result={"content": "OK", "metadata": {}}, + captcha_attempts=2, + captcha_solved=True, + ), + blocked: CrawlOutcome( + status=CrawlOutcomeStatus.FAILED, + error="blocked", + captcha_attempts=1, + captcha_solved=False, + ), + } + ) + execute = build_scrape_executor(engine=crawler) + + out = await execute(ScrapeInput(urls=[ok, blocked])) + + # Attempts bill even when the crawl ultimately failed (Phase 3d). + assert out.captcha_attempts == 3 + assert out.captcha_solved == 1 + + async def test_partial_failure_keeps_the_batch_and_labels_each_url(): ok, empty, failed = "https://ok.com", "https://empty.com", "https://failed.com" crawler = _FakeCrawler( From 97383921618e3c4758c447e86e34b7bec2da48bd Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 18:59:34 +0200 Subject: [PATCH 40/64] refactor(agents): rename intelligence_agent subagent to scraping --- .../agents/chat/multi_agent_chat/constants.py | 2 +- .../intelligence_agent/description.md | 2 - .../intelligence_agent/system_prompt.md | 56 ----------------- .../__init__.py | 0 .../{intelligence_agent => scraping}/agent.py | 4 +- .../builtins/scraping/description.md | 2 + .../builtins/scraping/system_prompt.md | 62 +++++++++++++++++++ .../tools/__init__.py | 0 .../tools/index.py | 4 +- .../tools/refresh_watch.py | 0 .../tools/start_watch.py | 0 .../tools/stop_watch.py | 0 .../multi_agent_chat/subagents/registry.py | 8 +-- .../tests/integration/automations/conftest.py | 2 +- .../automations/tools/test_watch_tools.py | 6 +- .../subagents/test_start_watch_tool.py | 10 +-- .../subagents/test_watch_control_tools.py | 8 +-- .../test_subagent_composition.py | 2 +- 18 files changed, 87 insertions(+), 81 deletions(-) delete mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/description.md delete mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/system_prompt.md rename surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/{intelligence_agent => scraping}/__init__.py (100%) rename surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/{intelligence_agent => scraping}/agent.py (88%) create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/description.md create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/system_prompt.md rename surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/{intelligence_agent => scraping}/tools/__init__.py (100%) rename surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/{intelligence_agent => scraping}/tools/index.py (92%) rename surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/{intelligence_agent => scraping}/tools/refresh_watch.py (100%) rename surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/{intelligence_agent => scraping}/tools/start_watch.py (100%) rename surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/{intelligence_agent => scraping}/tools/stop_watch.py (100%) diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/constants.py b/surfsense_backend/app/agents/chat/multi_agent_chat/constants.py index fff608742..1e4c4276d 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/constants.py +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/constants.py @@ -26,7 +26,7 @@ CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS: dict[str, str] = { SUBAGENT_TO_REQUIRED_CONNECTOR_MAP: dict[str, frozenset[str]] = { "deliverables": frozenset(), "knowledge_base": frozenset(), - "intelligence_agent": frozenset(), + "scraping": frozenset(), "airtable": frozenset({"AIRTABLE_CONNECTOR"}), "calendar": frozenset({"GOOGLE_CALENDAR_CONNECTOR"}), "clickup": frozenset({"CLICKUP_CONNECTOR"}), diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/description.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/description.md deleted file mode 100644 index cefad396c..000000000 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/description.md +++ /dev/null @@ -1,2 +0,0 @@ -Specialist for web intelligence: fetch live pages and search the web, then compare against earlier findings. -Use whenever a task needs current data pulled from specific URLs or discovered on the web — competitor/product monitoring, pricing, listings, "what changed since last time" — and answered from that evidence. diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/system_prompt.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/system_prompt.md deleted file mode 100644 index 60bcff29c..000000000 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/system_prompt.md +++ /dev/null @@ -1,56 +0,0 @@ -You are the SurfSense intelligence sub-agent. -You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis. - - -Pull evidence from the live web with the web.* capability verbs and answer the delegated question, including "what changed" comparisons against evidence already in this conversation. - - - -- `web_discover` — search the web for a query; returns ranked hits (url, title, snippet, provider). -- `web_scrape` — fetch specific URLs; returns clean page content and metadata per URL. -- `start_watch` — (only when running inside a chat) keep re-asking a question on a schedule in this chat, so the user gets refreshed answers without asking again. Takes the self-contained question, a cron cadence, and an IANA timezone. - - - -- If the request names exact URLs, call `web_scrape` on them directly. -- If it does not, call `web_discover` first, pick the most relevant hits, then `web_scrape` those URLs to read them. -- Batch URLs into a single `web_scrape` call when reading several pages (up to its limit) instead of many one-URL calls. -- For "what changed" / monitoring requests, compare the freshly scraped values against the prior values already present in this conversation's earlier tool results, and report the concrete deltas (added, removed, changed old -> new). Do not claim a change you cannot point to. -- When the user asks to keep watching / track something on a recurring basis (e.g. "check this daily", "tell me weekly what changed"), first answer now, then call `start_watch` with the self-contained question and the cadence (cron + timezone). Ask for the cadence or timezone only if the user did not imply one. - - - -- Use only tools in ``. -- A `web_scrape` row with `status` other than `success` yielded no content — do not invent its content; report it as unavailable. -- Never fabricate facts, URLs, prices, or quotes. - - - -- Report uncertainty explicitly when evidence is incomplete or conflicting. -- Never present unverified claims as facts. - - - -- On tool failure, return `status=error` with a concise recovery `next_step`. -- On no useful evidence, return `status=blocked` with a narrower query or the URLs you still need. - - - -Return **only** one JSON object (no markdown/prose): -{ - "status": "success" | "partial" | "blocked" | "error", - "action_summary": string, - "evidence": { - "findings": string[], - "sources": string[], - "confidence": "high" | "medium" | "low" - }, - "next_step": string | null, - "missing_fields": string[] | null, - "assumptions": string[] | null -} - -Route-specific rules: -- `evidence.findings`: max 10 entries, each a single sentence stating one distinct fact or delta. Do not paste raw scraped pages. -- `evidence.sources`: max 10 URLs, one per finding when applicable. List each URL once. - diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/__init__.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/__init__.py similarity index 100% rename from surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/__init__.py rename to surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/__init__.py diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/agent.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/agent.py similarity index 88% rename from surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/agent.py rename to surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/agent.py index 21933005b..9c6734b57 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/agent.py +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/agent.py @@ -1,4 +1,4 @@ -"""``intelligence_agent`` route: ``SurfSenseSubagentSpec`` builder for deepagents.""" +"""``scraping`` route: ``SurfSenseSubagentSpec`` builder for deepagents.""" from __future__ import annotations @@ -28,7 +28,7 @@ def build_subagent( tools = [*load_tools(dependencies=dependencies), *(mcp_tools or [])] description = ( read_md_file(__package__, "description").strip() - or "Gathers and compares web intelligence for this workspace." + or "Scrapes live public web pages and search results for this workspace." ) system_prompt = read_md_file(__package__, "system_prompt").strip() return pack_subagent( diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/description.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/description.md new file mode 100644 index 000000000..f6b2f8ada --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/description.md @@ -0,0 +1,2 @@ +Scraping specialist: fetches live public web pages, searches the web to find pages, and compares fresh data against earlier findings in this chat. +Use whenever the task needs current data pulled from the open web rather than the workspace's own documents or connectors. Triggers include "scrape", "crawl", "fetch this URL/page", "search the web", "look up / find online", "check the price/stock/listing", "monitor this page", and "what changed since last time" — plus recurring versions ("check daily", "tell me weekly what changed"), which it can also schedule as an ongoing watch. diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/system_prompt.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/system_prompt.md new file mode 100644 index 000000000..7f0d6841e --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/system_prompt.md @@ -0,0 +1,62 @@ +You are the SurfSense scraping sub-agent. +You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis. + + +Answer the delegated question from live evidence gathered with your data verbs, including "what changed" comparisons against evidence already in this conversation. + + + +- `web_discover` +- `web_scrape` +- `start_watch` +- `stop_watch` +- `refresh_watch` + + + +- Named URLs: `web_scrape` them directly. Otherwise `web_discover` first, then `web_scrape` the most relevant hits. +- Read several pages in one batched `web_scrape` call rather than many single-URL calls. +- "What changed" / monitoring: scrape the current values, compare against the prior values in this conversation's earlier tool results, and report concrete deltas (added, removed, old -> new). +- Recurring intent ("check daily", "tell me weekly what changed"): answer now, then `start_watch` with a self-contained question, a cron cadence, and an IANA timezone. Use `stop_watch` / `refresh_watch` to end or immediately re-run an existing watch. + + + +- Use only tools in ``. +- A `web_scrape` row whose `status` is not `success` returned no content — report it unavailable, never invent it. +- Report only deltas you can point to in the evidence. Never fabricate facts, URLs, prices, or quotes. + + + +- Do not generate deliverables or perform connector mutations; return findings for the supervisor to act on. + + + +- Report uncertainty explicitly when evidence is incomplete or conflicting. +- Never present unverified claims as facts. + + + +- Underspecified request — including a recurring request whose cadence or timezone is neither given nor implied — return `status=blocked` with the missing fields. +- Tool failure: return `status=error` with a concise recovery `next_step`. +- No useful evidence: return `status=blocked` with a narrower query or the URLs you still need. + + + +Return **only** one JSON object (no markdown/prose): +{ + "status": "success" | "partial" | "blocked" | "error", + "action_summary": string, + "evidence": { + "findings": string[], + "sources": string[], + "confidence": "high" | "medium" | "low" + }, + "next_step": string | null, + "missing_fields": string[] | null, + "assumptions": string[] | null +} + +Route-specific rules: +- `evidence.findings`: max 10 entries, each a single sentence stating one distinct fact or delta. Do not paste raw scraped pages. +- `evidence.sources`: max 10 URLs, one per finding when applicable. List each URL once. + diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/__init__.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/__init__.py similarity index 100% rename from surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/__init__.py rename to surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/__init__.py diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/index.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/index.py similarity index 92% rename from surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/index.py rename to surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/index.py index 98f704755..5c89dbe30 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/index.py +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/index.py @@ -1,4 +1,4 @@ -"""``intelligence_agent`` tools: web.* verbs generated from the capability registry.""" +"""``scraping`` sub-agent tools: scraper capability verbs + chat-watch controls.""" from __future__ import annotations @@ -15,7 +15,7 @@ from .refresh_watch import create_refresh_watch_tool from .start_watch import create_start_watch_tool from .stop_watch import create_stop_watch_tool -NAME = "intelligence_agent" +NAME = "scraping" RULESET = Ruleset(origin=NAME, rules=[]) diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/refresh_watch.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/refresh_watch.py similarity index 100% rename from surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/refresh_watch.py rename to surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/refresh_watch.py diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/start_watch.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/start_watch.py similarity index 100% rename from surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/start_watch.py rename to surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/start_watch.py diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/stop_watch.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/stop_watch.py similarity index 100% rename from surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/intelligence_agent/tools/stop_watch.py rename to surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/stop_watch.py diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py index 652ca079e..ae3a88bbe 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py @@ -15,9 +15,6 @@ from app.agents.chat.multi_agent_chat.constants import ( from app.agents.chat.multi_agent_chat.subagents.builtins.deliverables.agent import ( build_subagent as build_deliverables_subagent, ) -from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.agent import ( - build_subagent as build_intelligence_agent_subagent, -) from app.agents.chat.multi_agent_chat.subagents.builtins.knowledge_base.agent import ( build_subagent as build_knowledge_base_subagent, ) @@ -27,6 +24,9 @@ from app.agents.chat.multi_agent_chat.subagents.builtins.memory.agent import ( from app.agents.chat.multi_agent_chat.subagents.builtins.research.agent import ( build_subagent as build_research_subagent, ) +from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.agent import ( + build_subagent as build_scraping_subagent, +) from app.agents.chat.multi_agent_chat.subagents.connectors.airtable.agent import ( build_subagent as build_airtable_subagent, ) @@ -102,7 +102,6 @@ SUBAGENT_BUILDERS_BY_NAME: dict[str, SubagentBuilder] = { "dropbox": build_dropbox_subagent, "gmail": build_gmail_subagent, "google_drive": build_google_drive_subagent, - "intelligence_agent": build_intelligence_agent_subagent, "jira": build_jira_subagent, "knowledge_base": build_knowledge_base_subagent, "linear": build_linear_subagent, @@ -111,6 +110,7 @@ SUBAGENT_BUILDERS_BY_NAME: dict[str, SubagentBuilder] = { "notion": build_notion_subagent, "onedrive": build_onedrive_subagent, "research": build_research_subagent, + "scraping": build_scraping_subagent, "slack": build_slack_subagent, "teams": build_teams_subagent, } diff --git a/surfsense_backend/tests/integration/automations/conftest.py b/surfsense_backend/tests/integration/automations/conftest.py index 74c9e1808..1d3e91b07 100644 --- a/surfsense_backend/tests/integration/automations/conftest.py +++ b/surfsense_backend/tests/integration/automations/conftest.py @@ -63,7 +63,7 @@ def tools_use_test_session(monkeypatch, db_session: AsyncSession) -> None: async def _session_cm(): yield db_session # owned by the outer fixture; do not close - from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools import ( + from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools import ( refresh_watch, start_watch, stop_watch, diff --git a/surfsense_backend/tests/integration/automations/tools/test_watch_tools.py b/surfsense_backend/tests/integration/automations/tools/test_watch_tools.py index 03d3ca93a..acec89069 100644 --- a/surfsense_backend/tests/integration/automations/tools/test_watch_tools.py +++ b/surfsense_backend/tests/integration/automations/tools/test_watch_tools.py @@ -13,13 +13,13 @@ import pytest import pytest_asyncio from sqlalchemy.ext.asyncio import AsyncSession -from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools.refresh_watch import ( +from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools.refresh_watch import ( create_refresh_watch_tool, ) -from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools.start_watch import ( +from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools.start_watch import ( create_start_watch_tool, ) -from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools.stop_watch import ( +from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools.stop_watch import ( create_stop_watch_tool, ) from app.auth.context import AuthContext diff --git a/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_start_watch_tool.py b/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_start_watch_tool.py index c532cb5e6..3cc49b7f2 100644 --- a/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_start_watch_tool.py +++ b/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_start_watch_tool.py @@ -1,4 +1,4 @@ -"""Unit tests for the ``start_watch`` tool on the intelligence_agent. +"""Unit tests for the ``start_watch`` tool on the scraping sub-agent. ``start_watch`` binds a recurring watch to the *current* chat: it distils the question + cadence the agent extracted and creates a ``schedule`` + @@ -26,7 +26,7 @@ class _FakeSessionCM: def _patch_deps(monkeypatch: pytest.MonkeyPatch, *, created: Any) -> AsyncMock: - from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools import ( + from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools import ( start_watch as mod, ) @@ -41,7 +41,7 @@ def _patch_deps(monkeypatch: pytest.MonkeyPatch, *, created: Any) -> AsyncMock: async def test_start_watch_binds_watch_to_current_chat( monkeypatch: pytest.MonkeyPatch, ) -> None: - from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools import ( + from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools import ( start_watch as mod, ) @@ -75,7 +75,7 @@ async def test_start_watch_binds_watch_to_current_chat( async def test_start_watch_errors_without_thread_or_auth( monkeypatch: pytest.MonkeyPatch, ) -> None: - from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools import ( + from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools import ( start_watch as mod, ) @@ -101,7 +101,7 @@ async def test_start_watch_errors_without_thread_or_auth( def test_load_tools_includes_start_watch_only_when_bindable() -> None: - from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools.index import ( + from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools.index import ( load_tools, ) diff --git a/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_watch_control_tools.py b/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_watch_control_tools.py index b07f864c4..c659e289b 100644 --- a/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_watch_control_tools.py +++ b/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_watch_control_tools.py @@ -33,7 +33,7 @@ def _watch(automation_id: int) -> Any: async def test_stop_watch_stops_every_watch_on_the_chat( monkeypatch: pytest.MonkeyPatch, ) -> None: - from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools import ( + from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools import ( stop_watch as mod, ) @@ -59,7 +59,7 @@ async def test_stop_watch_stops_every_watch_on_the_chat( async def test_stop_watch_reports_when_nothing_to_stop( monkeypatch: pytest.MonkeyPatch, ) -> None: - from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools import ( + from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools import ( stop_watch as mod, ) @@ -80,7 +80,7 @@ async def test_stop_watch_reports_when_nothing_to_stop( async def test_refresh_watch_runs_each_watch_now( monkeypatch: pytest.MonkeyPatch, ) -> None: - from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools import ( + from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools import ( refresh_watch as mod, ) @@ -104,7 +104,7 @@ async def test_refresh_watch_runs_each_watch_now( def test_load_tools_includes_control_tools_when_bindable() -> None: - from app.agents.chat.multi_agent_chat.subagents.builtins.intelligence_agent.tools.index import ( + from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools.index import ( load_tools, ) diff --git a/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py b/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py index 68ae05bbd..0cba7976a 100644 --- a/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py +++ b/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py @@ -33,7 +33,6 @@ _EXPECTED_SUBAGENTS = frozenset( "dropbox", "gmail", "google_drive", - "intelligence_agent", "jira", "knowledge_base", "linear", @@ -42,6 +41,7 @@ _EXPECTED_SUBAGENTS = frozenset( "notion", "onedrive", "research", + "scraping", "slack", "teams", } From df09654db74f1a58698d6b53f08ce95aa17dbc63 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 19:04:26 +0200 Subject: [PATCH 41/64] docs(plans): sync phase 4-7 state; rename to scraping subagent --- plans/backend/00-umbrella-plan.md | 22 ++++++++--------- plans/backend/00b-pipeline-diagrams.md | 13 +++++----- plans/backend/06-ongoing-automation.md | 6 ++--- plans/backend/07-orchestration.md | 33 ++++++++++++++------------ 4 files changed, 38 insertions(+), 36 deletions(-) diff --git a/plans/backend/00-umbrella-plan.md b/plans/backend/00-umbrella-plan.md index 300cd9e68..9037793dd 100644 --- a/plans/backend/00-umbrella-plan.md +++ b/plans/backend/00-umbrella-plan.md @@ -20,7 +20,7 @@ Premium, open-source, self-hostable **scraper APIs** (Bright Data / Firecrawl / > - **Automation = a persistent ongoing chat** that periodically re-invokes verbs. This replaces the `Triggers` subsystem; the periodic mechanism is **design-deferred** (Phase 6). > - **KB stays input-only**; crawled data is never indexed. > -> **Phase mapping:** **04 Capabilities** · **05 Access** · **06 Ongoing-Automation** (deferred) · **07 Orchestration** (`intelligence_agent`). The subplans `05a-timeline`/`05b-intelligence`/`06-triggers` are **removed** (history in git); `04a`/`04b` are folded into `04`/`05`. +> **Phase mapping:** **04 Capabilities** · **05 Access** · **06 Ongoing-Automation** · **07 Orchestration** (`scraping` subagent). The subplans `05a-timeline`/`05b-intelligence`/`06-triggers` are **removed** (history in git); `04a`/`04b` are folded into `04`/`05`. ## ⚠️ Architecture correction (2026-06-30) — Pipelines dropped; automations + input-only KB adopted @@ -52,11 +52,11 @@ flowchart TD REG["Capability registry (P4): web.scrape · web.discover · maps.search/place/reviews — typed verbs, cleaned AI-ready output, bill at executor"] ACQ --> REG REG --> DOORS["Access doors (P5): REST + API key · MCP server · chat tools — generated from the registry"] - DOORS --> AGENT["intelligence_agent subagent (P7): intent routing · verb chains · 'what changed' from chat history"] + DOORS --> AGENT["scraping subagent (P7): intent routing · verb chains · 'what changed' from chat history"] KB[("Knowledge Base — input-only: user files/context (uploads + Drive/Dropbox/OneDrive)")] --> AGENT CONN["Connectors = MCP tools (BYO) + file/KB-input"] --> AGENT AGENT -->|one-shot| ANS["plain-language answer (nothing persists)"] - AGENT -.->|standing need| ONG["Ongoing-Automation (P6, DEFERRED): persistent chat re-invokes verbs; memory = chat history"] + AGENT -.->|standing need| ONG["Ongoing-Automation (P6): persistent chat re-invokes verbs; memory = chat history"] ONG -.-> REG APIKEY["Platform API key"] --> DOORS ``` @@ -168,16 +168,16 @@ The Universal WebURL Crawler is the flagship Type-1 data source (**the moat**). ### Phase 6 — Ongoing-Automation (backend) [`subplan: 06-ongoing-automation.md`] -> **Canonical subplan:** [`06-ongoing-automation.md`](06-ongoing-automation.md) — **⚠️ design deferred.** The periodic mechanism is designed separately, after `04`/`05`/`07`. **Depends on `04`** (the verbs it re-invokes) and `05` (the chat surface + delivery channel). +> **Canonical subplan:** [`06-ongoing-automation.md`](06-ongoing-automation.md) — **shipped.** A chat watch is an `Automation` bound to the current chat (`schedule` + `chat_message`); the durable checkpointer is the memory. **Depends on `04`** (the verbs it re-invokes) and `05` (the chat surface + delivery channel). - Support **"keep watching"** with no stateful storage: a **persistent, ongoing chat** where the agent periodically re-invokes scraper verbs and drops results into the session; the agent reports "what's new" by reading the chat history. Open questions (resolved together): periodicity driver, delivery channel (SSE vs Zero-published table), context-window limit, loop owner, stop/cost controls. ### Phase 7 — Orchestration (backend) [`subplan: 07-orchestration.md`] -> **Canonical subplan:** [`07-orchestration.md`](07-orchestration.md) — the human-facing brain. **Build last** (atop `04`/`05`, and `06` for the ongoing mode). We plug into the shipped multi-agent runtime; we don't rebuild it. +> **Canonical subplan:** [`07-orchestration.md`](07-orchestration.md) — the human-facing brain. **Shipped for the web verbs** (atop `04`/`05`/`06`). We plug into the shipped multi-agent runtime; we don't rebuild it. -- Ship the net-new builtin **`intelligence_agent`** subagent (peer to `research`/`deliverables`): **intent routing** (one-shot vs keep-watching, one clarifying question when ambiguous), **verb composition** (`web.discover → web.scrape`, `maps.search → maps.place → maps.reviews`), and **"what changed" from the chat history** (re-invoke a verb, compare against prior tool outputs already in context — no timeline, no diff store). -- Toolset: registry-backed capability verbs (shared with `research`) + a deferred `start_watch` handoff to `06`. Tools follow the shipped `scrape_webpage` shape (executor + door + 03c billing), **direct-return** (no `deliverable_wait`). +- Ship the builtin **`scraping`** subagent (peer to `research`/`deliverables`): **intent routing** (one-shot vs keep-watching, block on ambiguous cadence), **verb composition** (`web.discover → web.scrape`; `maps.*` chains deferred until the Maps actor exists), and **"what changed" from the chat history** (re-invoke a verb, compare against prior tool outputs already in context — no timeline, no diff store). +- Toolset: registry-backed capability verbs + `start_watch`/`stop_watch`/`refresh_watch` bound to the current chat (`06`). Tools follow the shipped `scrape_webpage` shape (executor + door + 03c billing), **direct-return** (no `deliverable_wait`). ### (Future) Platform API-key + public MCP @@ -249,9 +249,9 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta | 3 | `03d-captcha-solving.md` | **IMPLEMENTED** (`ci_mvp`) — `captchatools` page_action (proprietary) + Apache-2 config + per-attempt `web_crawl_captcha` billing; off by default | | 3 | `03f-undetectability-testing.md` | **IMPLEMENTED** (`ci_mvp`) — manual scorecard under the **proprietary boundary** at `app/proprietary/web_crawler/testbench/` (`python -m app.proprietary.web_crawler.testbench`); Suite S (stealth, shipped builder) + Suite E (extraction via real `crawl_url`) + scorecard JSON/MD baseline diff | | — | `00b-pipeline-diagrams.md` | end-to-end flow diagrams (companion to this umbrella) | -| 4 | `04-capabilities.md` | **CANONICAL** · drafted — capability registry + verbs `web.scrape`/`web.discover`/`maps.*` → cleaned data; bill at executor. **← build next** | -| 5 | `05-access.md` | **CANONICAL** · drafted — generated REST/MCP/chat doors + intent router + BYO-MCP routing fix | -| 6 | `06-ongoing-automation.md` | **CANONICAL** · **design deferred** — chat-native "keep watching" (periodic re-invocation) | -| 7 | `07-orchestration.md` | **CANONICAL** · drafted — `intelligence_agent` subagent; "what changed" from chat history | +| 4 | `04-capabilities.md` | **IMPLEMENTED (web verbs)** — capability registry + `web.scrape`/`web.discover` → cleaned data, bill at executor (top_k bounds, max_length truncation, per-attempt captcha metering). `maps.*` deferred (Maps actor) | +| 5 | `05-access.md` | **IMPLEMENTED (REST + chat doors)** — generated REST/API-key + chat tools from the registry. MCP door + BYO-MCP routing fix deferred (MCP server) | +| 6 | `06-ongoing-automation.md` | **IMPLEMENTED** — chat-native "keep watching": `Automation` bound to the chat (`schedule` + `chat_message`), worker-safe durable checkpointer, watch tools + REST controls | +| 7 | `07-orchestration.md` | **IMPLEMENTED (web verbs)** — `scraping` subagent; intent routing, verb composition, "what changed" from chat history | Frontend & client subplans will be added under a separate umbrella later (see "Deferred — Frontend & client phases"). diff --git a/plans/backend/00b-pipeline-diagrams.md b/plans/backend/00b-pipeline-diagrams.md index d2ffb8be7..b199fbf1b 100644 --- a/plans/backend/00b-pipeline-diagrams.md +++ b/plans/backend/00b-pipeline-diagrams.md @@ -16,8 +16,8 @@ flowchart LR subgraph SCOPE["OUR SCOPE · Phase 04 -> 07"] CAP["04 Capabilities
web.* · maps.*
cleaned, AI-ready output"] ACC["05 Access
REST · MCP · chat doors"] - ORC["07 Orchestration
intelligence_agent"] - ONG["06 Ongoing-Automation
(keep-watching · DEFERRED)"] + ORC["07 Orchestration
scraping subagent"] + ONG["06 Ongoing-Automation
(keep-watching)"] end ACQ --> CAP @@ -77,20 +77,19 @@ sequenceDiagram ```mermaid flowchart TD - U(["User speaks in natural language"]) --> R{"Intent router
(intelligence_agent prompt, 07)"} + U(["User speaks in natural language"]) --> R{"Intent router
(scraping prompt, 07)"} R -->|"find / compare / what is / right now"| A["ONE-SHOT
compose verbs, answer (section 3)"] - R -->|"watch / track / notify me / over time"| B["KEEP-WATCHING
hand to Ongoing-Automation (06)"] - R -->|ambiguous| Q["Ask ONE question:
'just once, or keep watching?'"] + R -->|"watch / track / notify me / over time"| B["KEEP-WATCHING
start_watch -> Ongoing-Automation (06)"] + R -->|ambiguous cadence| Q["Block: ask for the missing cadence / timezone"] Q -->|once| A Q -->|keep watching| B - B -.->|DESIGN DEFERRED| X["06 mechanism TBD
(periodicity · delivery · context limits)"] ``` ## 5. "What changed" — chat history is the memory ```mermaid flowchart LR - P["Prior tool outputs
(already in chat context)"] --> AG["intelligence_agent"] + P["Prior tool outputs
(already in chat context)"] --> AG["scraping subagent"] NEW["Fresh verb call (04)"] --> AG AG --> ANS["'Here's what's new since last time'
(reasoned from chat history)"] ``` diff --git a/plans/backend/06-ongoing-automation.md b/plans/backend/06-ongoing-automation.md index 40c74c6e7..8006b4a22 100644 --- a/plans/backend/06-ongoing-automation.md +++ b/plans/backend/06-ongoing-automation.md @@ -1,6 +1,6 @@ # Phase 6 — Ongoing Automation (chat-native "keep watching") -> Depends on `04` (the verbs it re-invokes), `05` (the agent tools), and `07` (the `intelligence_agent`). +> Depends on `04` (the verbs it re-invokes), `05` (the agent tools), and `07` (the `scraping` subagent). > Reuses the existing chat + automations machinery; adds no parallel engine. ## Objective @@ -28,7 +28,7 @@ chat's own checkpoint is the memory. opened them, but Celery uses a fresh loop per task (`PoolTimeout`). Dispose the checkpointer pool per task in `run_async_celery_task`, mirroring `_dispose_shared_db_engine`, so a worker can use the *durable* checkpointer (not `InMemorySaver`) that "what changed" requires. -- **`start_watch`** — an `intelligence_agent` tool that binds a watch to the *current* chat: it distills +- **`start_watch`** — a `scraping` subagent tool that binds a watch to the *current* chat: it distills the recurring question + cadence and creates the automation (`schedule` + `chat_message(thread_id = current chat)`). - **"Is this chat watched?"** — derived: an active automation with a `chat_message` action targeting the @@ -44,7 +44,7 @@ chat's own checkpoint is the memory. 2. `chat_message` action: params + factory + handler (drains `stream_new_chat`); concurrency guard. **[done]** 3. Watch service: create (bind `schedule` + `chat_message` automation to a chat) / stop (delete) / find-for-thread (is-watched) / run-now. **[done]** -4. `start_watch` tool on `intelligence_agent` (+ prompt line); binds to the current chat. **[done]** +4. `start_watch` tool on the `scraping` subagent (+ prompt line); binds to the current chat. **[done]** 5. Controls — chat tools (`stop_watch`, `refresh_watch`) + REST (`GET /automations/watches`, `POST /automations/{id}/run`; stop = `DELETE /automations/{id}`). **[done]** diff --git a/plans/backend/07-orchestration.md b/plans/backend/07-orchestration.md index 82754de18..50f3e6d9a 100644 --- a/plans/backend/07-orchestration.md +++ b/plans/backend/07-orchestration.md @@ -1,19 +1,22 @@ # Phase 7 — Orchestration / Conversation (the CI-expert subagent) -> Build last; sits atop `04`/`05` (and `06` for the ongoing mode). -> The multi-agent chat runtime (deepagents, subagent dispatch, streaming, citation middleware) is shipped. -> Net-new here = one builtin subagent + its tools + its prompt. Locate code by symbol/grep. +> **Status (2026-07-02): shipped for the web verbs.** The `scraping` subagent, its tools, prompt, and +> router blurb are live; `web.discover`/`web.scrape` compose and the "keep watching" handoff (`06`) is +> wired. `maps.*` composition is deferred until the Maps actor exists. +> Sits atop `04`/`05` (and `06` for the ongoing mode). The multi-agent chat runtime (deepagents, subagent +> dispatch, streaming, citation middleware) is shipped. Net-new here = one builtin subagent + its tools + +> its prompt. Locate code by symbol/grep. ## Objective -Ship the `intelligence_agent` — a builtin CI/scraper-expert subagent that delivers the product through +Ship the `scraping` subagent — a builtin CI/scraper-expert that delivers the product through plain conversation: understand intent, compose scraper verbs (`04`), answer in plain language, and hand standing needs to the ongoing "keep watching" mode (`06`). It reasons over chat history to report what's new vs prior tool outputs already in context. ## The subagent -New builtin `subagents/builtins/intelligence_agent/`, peer to `research`/`deliverables`, packed via +Builtin `subagents/builtins/scraping/`, peer to `research`/`deliverables`, packed via `pack_subagent`. Files follow the existing pattern: `agent.py` (`build_subagent`), `tools/index.py` (`NAME · RULESET · load_tools`), `description.md` (router one-liner), `system_prompt.md` (the playbook). The main agent delegates CI-/scraping-flavored requests to it via `description.md`. @@ -32,11 +35,11 @@ The main agent delegates CI-/scraping-flavored requests to it via `description.m | Tool | Wraps | Billing | |------|-------|---------| -| capability verbs (`web.scrape`, `web.discover`, `maps.search`, `maps.place`, `maps.reviews`) | `04` executors (direct-return) | `03c` owner-wallet charge via `charge_capability` (same path as REST/MCP) | -| `start_watch(intent)` *(deferred)* | hands off to `06` | per re-invocation (design with `06`) | +| capability verbs — `web.scrape`, `web.discover` shipped; `maps.search`/`maps.place`/`maps.reviews` deferred (Maps actor) | `04` executors (direct-return) | `03c` owner-wallet charge via `charge_capability` (same path as REST/MCP) | +| `start_watch` / `stop_watch` / `refresh_watch` | bind a watch to the current chat via `06` | per re-invocation | Verbs reach the subagent through the chat door generator `access/chat.py::build_capability_tools` (`05`), -which turns registry verbs (`04`) into tools with owner-wallet billing. The `intelligence_agent` owns these +which turns registry verbs (`04`) into tools with owner-wallet billing. The `scraping` subagent owns these CI-flavored calls; the main agent's `web_search`/`scrape_webpage` stay untouched. ## Boundaries @@ -47,16 +50,16 @@ CI-flavored calls; the main agent's `web_search`/`scrape_webpage` stay untouched ## Work items -1. Subagent scaffold: `subagents/builtins/intelligence_agent/` (`agent.py` / `tools/index.py` / +1. **[done]** Subagent scaffold: `subagents/builtins/scraping/` (`agent.py` / `tools/index.py` / `description.md` / `system_prompt.md`), packed via `pack_subagent`. -2. CI playbook prompt: intent routing (one clarifying question), verb chains, "what changed" reasoning. -3. Chat door generator: `access/chat.py::build_capability_tools` turns registry verbs (`04`) into tools (`05`). -4. `start_watch` seam to `06` (wired when `06` is designed). -5. Router registration: `description.md` for main-agent delegation. +2. **[done]** CI playbook prompt: intent routing (block on ambiguous cadence), verb chains, "what changed" reasoning. +3. **[done]** Chat door generator: `access/chat.py::build_capability_tools` turns registry verbs (`04`) into tools (`05`). +4. **[done]** `start_watch` / `stop_watch` / `refresh_watch` seam to `06`, bound to the current chat. +5. **[done]** Router registration: `description.md` for main-agent delegation. ## Tests -- A CI-/scraping-flavored request routes to `intelligence_agent`; a non-CI request does not. +- A CI-/scraping-flavored request routes to `scraping`; a non-CI request does not. - "compare X and Y" composes verbs and answers in plain language, persisting nothing. - "reviews for the top 3 coffee shops near me" chains `maps.search → maps.place → maps.reviews`. - A follow-up re-invokes the verb and reports new items vs prior in-context outputs. @@ -70,4 +73,4 @@ CI-flavored calls; the main agent's `web_search`/`scrape_webpage` stay untouched ## Open questions -- The `start_watch` handoff shape (settled with `06`). +- None outstanding for the web verbs. `maps.*` composition reopens when the Maps actor lands. From 43893fc1f8112fbb885488a9b7e94681906b4d6e Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 19:05:45 +0200 Subject: [PATCH 42/64] docs(plans): drop remaining intelligence_agent references --- plans/backend/00-umbrella-plan.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/plans/backend/00-umbrella-plan.md b/plans/backend/00-umbrella-plan.md index 9037793dd..cfcda1489 100644 --- a/plans/backend/00-umbrella-plan.md +++ b/plans/backend/00-umbrella-plan.md @@ -38,7 +38,7 @@ Premium, open-source, self-hostable **scraper APIs** (Bright Data / Firecrawl / > - **Phase 4 — Capabilities & Access** (`revamp/04a` + `04b`): turn the crawler + search into **typed, callable verbs** (MVP = `web.scrape` + `web.discover`; per-platform scrapers like `maps.*`/`linkedin.*` are *later, uncommitted* drop-ins) over a single **capability registry**, exposed identically through **chat / REST+API-key / MCP** doors. Ships **Product A** (stateless utility) — revenue day one. *The old `04a-connector-category` is demoted to backward-compat hygiene; the old `04b-source-discovery` is absorbed as the `web.discover` verb.* > - **Phase 5 — Intelligence & Timeline** (`revamp/05a` + `05b`): the `Tracker` primitive + a **3-store delta Timeline** (`tracked_entities`/`entity_current_state`/`entity_changes`, "store deltas, no change → no row"). This is the durable CI state and **the moat** — it replaces "pipelines" as the *standing concern*. Ships the **Product B** engine. > - **Phase 6 — Triggers** (`revamp/06`): decide **when** a Tracker refreshes. `refresh(tracker)` is trigger-agnostic; recurrence is a **CI action on the existing automations** (reuse its schedule selector + `AutomationRun` — no new scheduler, no new run table). Alert **delivery is separate** (via `app/notifications/`; automations have no delivery path). -> - **Phase 7 — Orchestration** (`revamp/07`): a net-new `intelligence_agent` subagent (intent routing one-shot-vs-standing, verb chains, Tracker crafting, decision-grounded answering) so the whole product is reachable in plain language. +> - **Phase 7 — Orchestration** (`revamp/07`): a net-new `scraping` subagent (intent routing one-shot-vs-standing, verb chains, Tracker crafting, decision-grounded answering) so the whole product is reachable in plain language. > > **KB stays input-only; `WEBCRAWLER_CONNECTOR` retired; crawler/search are native tools (later a platform API key).** The three correction decisions land in their real homes: **bill crawls at the capability executor** (`revamp/04a`, works uniformly across chat/automation/REST/MCP/cron — see 03c below), **diffable run history = the Timeline delta store** (`revamp/05a`, *not* `automation_runs.output`), and **"run now" = a Trigger adapter** (`revamp/06`). Phase 8 platform scrapers re-slot as a family of **individual native endpoints** — each one a capability verb → an agent tool + a dev API-key REST endpoint — added incrementally (**none, incl. Google Maps, committed for MVP**; they're examples of the pattern), not pipeline executors. @@ -209,8 +209,8 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta ## Resolved decisions log -- **PIVOT (2026-07-01) — scraper-APIs-first, stateless, chat-native; Tracker/Timeline/Triggers dropped.** The product is the **scraper capabilities** (`web.*` + `maps.*` typed verbs → cleaned, AI-ready data), exposed as **REST + API key / MCP / chat** generated from one registry — premium, OSS, self-hostable, the revenue driver. **Stateless:** no Timeline (3-store delta), no `Tracker`, no diff/materiality engine, no stored `entity_changes`; memory = the **chat history** (the agent reasons over prior tool outputs for "what changed"). **Direct calls:** capabilities return their result directly — no job store, no `completed | pending + job_id` envelope, no `deliverable_wait` polling; async delivery is **write-then-sync / stream**. **Automation = a persistent ongoing chat** re-invoking verbs (replaces `Triggers`; mechanism **design-deferred**). KB stays **input-only**. Phase re-cast: **04 Capabilities · 05 Access · 06 Ongoing-Automation (deferred) · 07 Orchestration** (`intelligence_agent`). Subplans `05a-timeline`/`05b-intelligence`/`06-triggers` **removed** (git history); `04a`/`04b` folded into `04`/`05`. This **supersedes** the 2026-06-30 revamp's Tracker/Timeline/Triggers decisions below. -- **REVAMP ADOPTED as canonical (2026-06-30) — Phases 4–7 re-cast; `revamp phases 4-7/` is the source of truth.** A principal-engineer review found the flat `04a–07` files and my earlier "Phase 5′" framing had been overtaken by a more complete engineer draft (then in a `revamp phases 4-7/` folder, since flattened into `04`–`07`). We adopt the revamp: **Phase 4 = Capabilities & Access** (typed verbs + generated doors; old `04a-connector-category` demoted to hygiene, old `04b-source-discovery` absorbed as `web.discover`), **Phase 5 = Intelligence & Timeline** (the `Tracker` + 3-store delta Timeline — the moat), **Phase 6 = Triggers** (reuse automations via a CI action), **Phase 7 = Orchestration** (`intelligence_agent`). The three architecture-correction decisions land in their real homes and one is revised: (1) **bill at the capability executor** — a code-verified gap review confirmed `run_agent_task` (automation `agent_task`) sets up **no turn accumulator**, so today `scrape_webpage` bills nothing under automations; billing moves to the executor so chat/automation/REST/MCP/cron meter uniformly (chat turn accumulator becomes optional presentation). (2) **"diffable run history" = the Timeline delta store** (`revamp/05a`), **superseding** the earlier "structured `automation_runs.output`" decision. (3) **"run now" = a Trigger adapter** (`revamp/06`). The flat `04a/04b/05/06/07` files are retained with redirect banners; **`revamp/` filenames win** on any number collision. Build order: `04a → 04b → 05a → 05b → 06 → 07`. +- **PIVOT (2026-07-01) — scraper-APIs-first, stateless, chat-native; Tracker/Timeline/Triggers dropped.** The product is the **scraper capabilities** (`web.*` + `maps.*` typed verbs → cleaned, AI-ready data), exposed as **REST + API key / MCP / chat** generated from one registry — premium, OSS, self-hostable, the revenue driver. **Stateless:** no Timeline (3-store delta), no `Tracker`, no diff/materiality engine, no stored `entity_changes`; memory = the **chat history** (the agent reasons over prior tool outputs for "what changed"). **Direct calls:** capabilities return their result directly — no job store, no `completed | pending + job_id` envelope, no `deliverable_wait` polling; async delivery is **write-then-sync / stream**. **Automation = a persistent ongoing chat** re-invoking verbs (replaces `Triggers`; mechanism **design-deferred**). KB stays **input-only**. Phase re-cast: **04 Capabilities · 05 Access · 06 Ongoing-Automation (deferred) · 07 Orchestration** (`scraping`). Subplans `05a-timeline`/`05b-intelligence`/`06-triggers` **removed** (git history); `04a`/`04b` folded into `04`/`05`. This **supersedes** the 2026-06-30 revamp's Tracker/Timeline/Triggers decisions below. +- **REVAMP ADOPTED as canonical (2026-06-30) — Phases 4–7 re-cast; `revamp phases 4-7/` is the source of truth.** A principal-engineer review found the flat `04a–07` files and my earlier "Phase 5′" framing had been overtaken by a more complete engineer draft (then in a `revamp phases 4-7/` folder, since flattened into `04`–`07`). We adopt the revamp: **Phase 4 = Capabilities & Access** (typed verbs + generated doors; old `04a-connector-category` demoted to hygiene, old `04b-source-discovery` absorbed as `web.discover`), **Phase 5 = Intelligence & Timeline** (the `Tracker` + 3-store delta Timeline — the moat), **Phase 6 = Triggers** (reuse automations via a CI action), **Phase 7 = Orchestration** (`scraping`). The three architecture-correction decisions land in their real homes and one is revised: (1) **bill at the capability executor** — a code-verified gap review confirmed `run_agent_task` (automation `agent_task`) sets up **no turn accumulator**, so today `scrape_webpage` bills nothing under automations; billing moves to the executor so chat/automation/REST/MCP/cron meter uniformly (chat turn accumulator becomes optional presentation). (2) **"diffable run history" = the Timeline delta store** (`revamp/05a`), **superseding** the earlier "structured `automation_runs.output`" decision. (3) **"run now" = a Trigger adapter** (`revamp/06`). The flat `04a/04b/05/06/07` files are retained with redirect banners; **`revamp/` filenames win** on any number collision. Build order: `04a → 04b → 05a → 05b → 06 → 07`. - **ARCHITECTURE CORRECTION (2026-06-30) — Pipelines dropped; automations + input-only KB adopted.** The "sync into KB → operate over it" paradigm was judged wrong for CI. Verified against `references/opencode` (live tool-fetched context + persisted sessions, no scraped corpus; `tool/websearch.ts` live-crawls at query time). Decisions: (1) **KB is input-only** — user's personal files/context (uploads + Drive/Dropbox/OneDrive); nothing scrapes into it. (2) **Web search + WebURL crawler are platform-native tools** (already `web_search`/`scrape_webpage`), later a developer platform-API-key surface — NOT connectors, NOT pipelines. (3) **Connectors = MCP tools + file/KB-input connectors**; all branded natives → MCP. (4) **Automations are the scheduling + run-history substrate** (an `agent_task` run already invokes the full chat agent with the crawler/search tools on cron/event and persists `automation_runs`). (5) **`WEBCRAWLER_CONNECTOR` RETIRED.** Consequences: **Phases 5/6/7 SUPERSEDED** (docs kept with banners); **Phase 4a reframed** (drop `is_pipeline_eligible`; Type-1 = file/KB-input only; WEBCRAWLER retired); **Phase 4b stands**; new **Phase 5′** automation-enhancement workstream (bill automation-run crawls; structured `run.output`; wire "run now"); **Phase 8** actors re-slot as native tools/API. `03c` connector-indexer billing branch becomes **vestigial** (billing lives on chat-turn / automation-run / API paths). - ~~Web search APIs (SearXNG/Linkup/Baidu): repurposed as source-discovery helper for the WebURL Crawler (suggest URLs for pipelines)~~ — still repurposed as a source-discovery helper (04b), but "for pipelines" → "for the user / the native crawler tool" (pipelines dropped); not a standalone connector type. - Tavily and Serper: REMOVED from the search infra. They are dropped as search providers entirely (not repurposed). Phase 4's source-discovery endpoint must build only on the remaining providers (SearXNG, Linkup, Baidu). From 87df0bb6296595d4ce4e2d36b311be97fd97205b Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 19:40:37 +0200 Subject: [PATCH 43/64] fix(web.discover): cap results to top_k --- .../app/capabilities/web/discover/executor.py | 4 +++- .../web/discover/test_executor.py | 20 +++++++++++++++++++ 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/surfsense_backend/app/capabilities/web/discover/executor.py b/surfsense_backend/app/capabilities/web/discover/executor.py index 85f6bf83b..e98de86e4 100644 --- a/surfsense_backend/app/capabilities/web/discover/executor.py +++ b/surfsense_backend/app/capabilities/web/discover/executor.py @@ -27,7 +27,9 @@ def build_discover_executor( "(set a SearXNG host or a Linkup/Baidu key)." ) hits = await provider.search(payload.query, payload.top_k) - return DiscoverOutput(hits=hits) + # Enforce the verb's documented cap here, once, for every provider: + # some backends (e.g. SearXNG) treat `top_k` as a hint and over-return. + return DiscoverOutput(hits=hits[: payload.top_k]) return execute diff --git a/surfsense_backend/tests/unit/capabilities/web/discover/test_executor.py b/surfsense_backend/tests/unit/capabilities/web/discover/test_executor.py index 5d50d5f6f..bfc93a31b 100644 --- a/surfsense_backend/tests/unit/capabilities/web/discover/test_executor.py +++ b/surfsense_backend/tests/unit/capabilities/web/discover/test_executor.py @@ -77,3 +77,23 @@ async def test_self_disables_when_no_provider_is_configured(): with pytest.raises(NoDiscoverProviderError): await execute(DiscoverInput(query="q")) + + +async def test_caps_hits_to_top_k_when_provider_over_returns(): + # SearXNG treats `limit` as a hint and can return more rows than asked; the + # verb must honor its own documented `top_k` cap regardless of the provider. + over = _FakeProvider( + "searxng", + available=True, + hits=[_hit(f"https://{i}.com", "searxng") for i in range(5)], + ) + execute = build_discover_executor(providers=[over]) + + out = await execute(DiscoverInput(query="q", top_k=3)) + + assert len(out.hits) == 3 + assert [h.url for h in out.hits] == [ + "https://0.com", + "https://1.com", + "https://2.com", + ] From 6bd0d640e601d3650090ab23379cd0a2aff94174 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 19:54:20 +0200 Subject: [PATCH 44/64] refactor(capabilities): group framework into core/, keep verbs top-level --- .../builtins/scraping/tools/index.py | 2 +- .../app/capabilities/core/__init__.py | 30 +++++++++++++++++++ .../{ => core}/access/__init__.py | 0 .../capabilities/{ => core}/access/agent.py | 6 ++-- .../{ => core}/access/rate_limit.py | 0 .../capabilities/{ => core}/access/rest.py | 8 ++--- .../app/capabilities/{ => core}/billing.py | 2 +- .../app/capabilities/{ => core}/store.py | 2 +- .../app/capabilities/{ => core}/types.py | 0 .../capabilities/web/discover/definition.py | 3 +- .../app/capabilities/web/discover/executor.py | 2 +- .../app/capabilities/web/scrape/definition.py | 3 +- .../app/capabilities/web/scrape/executor.py | 2 +- surfsense_backend/app/routes/__init__.py | 2 +- .../capabilities/access/test_agent_tools.py | 4 +-- .../capabilities/access/test_rate_limit.py | 2 +- .../capabilities/access/test_rest_router.py | 14 ++++----- .../tests/unit/capabilities/test_billing.py | 6 ++-- .../tests/unit/capabilities/test_registry.py | 4 +-- 19 files changed, 60 insertions(+), 32 deletions(-) create mode 100644 surfsense_backend/app/capabilities/core/__init__.py rename surfsense_backend/app/capabilities/{ => core}/access/__init__.py (100%) rename surfsense_backend/app/capabilities/{ => core}/access/agent.py (89%) rename surfsense_backend/app/capabilities/{ => core}/access/rate_limit.py (100%) rename surfsense_backend/app/capabilities/{ => core}/access/rest.py (89%) rename surfsense_backend/app/capabilities/{ => core}/billing.py (99%) rename surfsense_backend/app/capabilities/{ => core}/store.py (90%) rename surfsense_backend/app/capabilities/{ => core}/types.py (100%) diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/index.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/index.py index 5c89dbe30..8416b8b05 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/index.py +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/index.py @@ -7,7 +7,7 @@ from typing import Any from langchain_core.tools import BaseTool from app.agents.chat.multi_agent_chat.shared.permissions import Ruleset -from app.capabilities.access.agent import build_capability_tools +from app.capabilities.core.access.agent import build_capability_tools from app.capabilities.web.discover.definition import WEB_DISCOVER from app.capabilities.web.scrape.definition import WEB_SCRAPE diff --git a/surfsense_backend/app/capabilities/core/__init__.py b/surfsense_backend/app/capabilities/core/__init__.py new file mode 100644 index 000000000..71d14b639 --- /dev/null +++ b/surfsense_backend/app/capabilities/core/__init__.py @@ -0,0 +1,30 @@ +"""Capability framework kernel: registry contracts, store, billing, and access doors.""" + +from app.capabilities.core.billing import charge_capability, gate_capability +from app.capabilities.core.store import ( + all_capabilities, + get_capability, + register_capability, +) +from app.capabilities.core.types import ( + BillableInput, + BillableOutput, + BillingUnit, + Capability, + CapabilityContext, + Executor, +) + +__all__ = [ + "BillableInput", + "BillableOutput", + "BillingUnit", + "Capability", + "CapabilityContext", + "Executor", + "all_capabilities", + "charge_capability", + "gate_capability", + "get_capability", + "register_capability", +] diff --git a/surfsense_backend/app/capabilities/access/__init__.py b/surfsense_backend/app/capabilities/core/access/__init__.py similarity index 100% rename from surfsense_backend/app/capabilities/access/__init__.py rename to surfsense_backend/app/capabilities/core/access/__init__.py diff --git a/surfsense_backend/app/capabilities/access/agent.py b/surfsense_backend/app/capabilities/core/access/agent.py similarity index 89% rename from surfsense_backend/app/capabilities/access/agent.py rename to surfsense_backend/app/capabilities/core/access/agent.py index 75b535840..003f42ae2 100644 --- a/surfsense_backend/app/capabilities/access/agent.py +++ b/surfsense_backend/app/capabilities/core/access/agent.py @@ -10,9 +10,9 @@ from __future__ import annotations from langchain_core.tools import BaseTool, StructuredTool -from app.capabilities.billing import charge_capability, gate_capability -from app.capabilities.store import all_capabilities -from app.capabilities.types import Capability, CapabilityContext +from app.capabilities.core.billing import charge_capability, gate_capability +from app.capabilities.core.store import all_capabilities +from app.capabilities.core.types import Capability, CapabilityContext from app.db import async_session_maker from app.services.web_crawl_credit_service import InsufficientCreditsError diff --git a/surfsense_backend/app/capabilities/access/rate_limit.py b/surfsense_backend/app/capabilities/core/access/rate_limit.py similarity index 100% rename from surfsense_backend/app/capabilities/access/rate_limit.py rename to surfsense_backend/app/capabilities/core/access/rate_limit.py diff --git a/surfsense_backend/app/capabilities/access/rest.py b/surfsense_backend/app/capabilities/core/access/rest.py similarity index 89% rename from surfsense_backend/app/capabilities/access/rest.py rename to surfsense_backend/app/capabilities/core/access/rest.py index 6a82e9144..df37c79ed 100644 --- a/surfsense_backend/app/capabilities/access/rest.py +++ b/surfsense_backend/app/capabilities/core/access/rest.py @@ -8,10 +8,10 @@ from fastapi import APIRouter, Depends, HTTPException, status from sqlalchemy.ext.asyncio import AsyncSession from app.auth.context import AuthContext -from app.capabilities.access.rate_limit import enforce_capability_rate_limit -from app.capabilities.billing import charge_capability, gate_capability -from app.capabilities.store import all_capabilities -from app.capabilities.types import Capability, CapabilityContext +from app.capabilities.core.access.rate_limit import enforce_capability_rate_limit +from app.capabilities.core.billing import charge_capability, gate_capability +from app.capabilities.core.store import all_capabilities +from app.capabilities.core.types import Capability, CapabilityContext from app.db import get_async_session from app.services.web_crawl_credit_service import InsufficientCreditsError from app.users import get_auth_context diff --git a/surfsense_backend/app/capabilities/billing.py b/surfsense_backend/app/capabilities/core/billing.py similarity index 99% rename from surfsense_backend/app/capabilities/billing.py rename to surfsense_backend/app/capabilities/core/billing.py index 636920a9f..a1fda82b2 100644 --- a/surfsense_backend/app/capabilities/billing.py +++ b/surfsense_backend/app/capabilities/core/billing.py @@ -6,7 +6,7 @@ from typing import TYPE_CHECKING from sqlalchemy import select -from app.capabilities.types import ( +from app.capabilities.core.types import ( BillableInput, BillableOutput, BillingUnit, diff --git a/surfsense_backend/app/capabilities/store.py b/surfsense_backend/app/capabilities/core/store.py similarity index 90% rename from surfsense_backend/app/capabilities/store.py rename to surfsense_backend/app/capabilities/core/store.py index b828480da..aad172c79 100644 --- a/surfsense_backend/app/capabilities/store.py +++ b/surfsense_backend/app/capabilities/core/store.py @@ -2,7 +2,7 @@ from __future__ import annotations -from app.capabilities.types import Capability +from app.capabilities.core.types import Capability _REGISTRY: dict[str, Capability] = {} diff --git a/surfsense_backend/app/capabilities/types.py b/surfsense_backend/app/capabilities/core/types.py similarity index 100% rename from surfsense_backend/app/capabilities/types.py rename to surfsense_backend/app/capabilities/core/types.py diff --git a/surfsense_backend/app/capabilities/web/discover/definition.py b/surfsense_backend/app/capabilities/web/discover/definition.py index e4e779307..f672c6bbb 100644 --- a/surfsense_backend/app/capabilities/web/discover/definition.py +++ b/surfsense_backend/app/capabilities/web/discover/definition.py @@ -2,8 +2,7 @@ from __future__ import annotations -from app.capabilities.store import register_capability -from app.capabilities.types import Capability +from app.capabilities.core import Capability, register_capability from app.capabilities.web.discover.executor import build_discover_executor from app.capabilities.web.discover.schemas import DiscoverInput, DiscoverOutput diff --git a/surfsense_backend/app/capabilities/web/discover/executor.py b/surfsense_backend/app/capabilities/web/discover/executor.py index e98de86e4..3660aedcd 100644 --- a/surfsense_backend/app/capabilities/web/discover/executor.py +++ b/surfsense_backend/app/capabilities/web/discover/executor.py @@ -4,7 +4,7 @@ from __future__ import annotations from collections.abc import Sequence -from app.capabilities.types import Executor +from app.capabilities.core import Executor from app.capabilities.web.discover.providers.base import DiscoverProvider from app.capabilities.web.discover.schemas import DiscoverInput, DiscoverOutput diff --git a/surfsense_backend/app/capabilities/web/scrape/definition.py b/surfsense_backend/app/capabilities/web/scrape/definition.py index 5ceb0b3a5..f6ccafe02 100644 --- a/surfsense_backend/app/capabilities/web/scrape/definition.py +++ b/surfsense_backend/app/capabilities/web/scrape/definition.py @@ -2,8 +2,7 @@ from __future__ import annotations -from app.capabilities.store import register_capability -from app.capabilities.types import BillingUnit, Capability +from app.capabilities.core import BillingUnit, Capability, register_capability from app.capabilities.web.scrape.executor import build_scrape_executor from app.capabilities.web.scrape.schemas import ScrapeInput, ScrapeOutput diff --git a/surfsense_backend/app/capabilities/web/scrape/executor.py b/surfsense_backend/app/capabilities/web/scrape/executor.py index 05459ee6a..8651ff738 100644 --- a/surfsense_backend/app/capabilities/web/scrape/executor.py +++ b/surfsense_backend/app/capabilities/web/scrape/executor.py @@ -2,7 +2,7 @@ from __future__ import annotations -from app.capabilities.types import Executor +from app.capabilities.core import Executor from app.capabilities.web.scrape.schemas import ScrapeInput, ScrapeOutput, ScrapeRow from app.proprietary.web_crawler import ( CrawlOutcome, diff --git a/surfsense_backend/app/routes/__init__.py b/surfsense_backend/app/routes/__init__.py index df53759aa..9f42f11e1 100644 --- a/surfsense_backend/app/routes/__init__.py +++ b/surfsense_backend/app/routes/__init__.py @@ -2,7 +2,7 @@ from fastapi import APIRouter, Depends import app.capabilities.web # noqa: F401 (registers web.* verbs before the door builds) from app.automations.api import router as automations_router -from app.capabilities.access.rest import build_capabilities_router +from app.capabilities.core.access.rest import build_capabilities_router from app.file_storage.api import router as file_storage_router from app.gateway import require_gateway_enabled from app.notifications.api import router as notifications_router diff --git a/surfsense_backend/tests/unit/capabilities/access/test_agent_tools.py b/surfsense_backend/tests/unit/capabilities/access/test_agent_tools.py index dc8e126a6..ac3db487d 100644 --- a/surfsense_backend/tests/unit/capabilities/access/test_agent_tools.py +++ b/surfsense_backend/tests/unit/capabilities/access/test_agent_tools.py @@ -6,7 +6,7 @@ from unittest.mock import AsyncMock import pytest from pydantic import BaseModel, Field -from app.capabilities.types import BillingUnit, Capability +from app.capabilities.core.types import BillingUnit, Capability from app.services.web_crawl_credit_service import InsufficientCreditsError pytestmark = pytest.mark.asyncio @@ -54,7 +54,7 @@ class _FakeSessionCtx: @pytest.fixture def isolate(monkeypatch): """Stub the billing session + charge/gate so tools never hit the DB.""" - from app.capabilities.access import agent as mod + from app.capabilities.core.access import agent as mod monkeypatch.setattr(mod, "async_session_maker", lambda: _FakeSessionCtx()) charge = AsyncMock() diff --git a/surfsense_backend/tests/unit/capabilities/access/test_rate_limit.py b/surfsense_backend/tests/unit/capabilities/access/test_rate_limit.py index f9c8076bf..1762c146e 100644 --- a/surfsense_backend/tests/unit/capabilities/access/test_rate_limit.py +++ b/surfsense_backend/tests/unit/capabilities/access/test_rate_limit.py @@ -4,7 +4,7 @@ import pytest from fastapi import HTTPException from starlette.requests import Request -from app.capabilities.access import rate_limit +from app.capabilities.core.access import rate_limit def _request(workspace_id: int) -> Request: diff --git a/surfsense_backend/tests/unit/capabilities/access/test_rest_router.py b/surfsense_backend/tests/unit/capabilities/access/test_rest_router.py index 211ce9854..b874b8bba 100644 --- a/surfsense_backend/tests/unit/capabilities/access/test_rest_router.py +++ b/surfsense_backend/tests/unit/capabilities/access/test_rest_router.py @@ -7,7 +7,7 @@ from fastapi import FastAPI from httpx import ASGITransport, AsyncClient from pydantic import BaseModel -from app.capabilities.types import Capability +from app.capabilities.core.types import Capability from app.db import get_async_session from app.users import get_auth_context @@ -36,8 +36,8 @@ _ECHO = Capability( def _build_app(capabilities, monkeypatch) -> FastAPI: """Mount the generated door with auth/workspace/session/rate-limit stubbed.""" - from app.capabilities.access import rest - from app.capabilities.access.rate_limit import enforce_capability_rate_limit + from app.capabilities.core.access import rest + from app.capabilities.core.access.rate_limit import enforce_capability_rate_limit monkeypatch.setattr(rest, "check_workspace_access", _noop_async, raising=True) @@ -91,7 +91,7 @@ async def test_input_is_validated_against_the_verb_schema(monkeypatch): def test_registered_verbs_appear_on_rest(): """A verb in the registry shows up as a route with no per-verb wiring.""" import app.capabilities.web # noqa: F401 (registers web.* at import) - from app.capabilities.access import rest + from app.capabilities.core.access import rest router = rest.build_capabilities_router() paths = {route.path for route in router.routes} @@ -101,7 +101,7 @@ def test_registered_verbs_appear_on_rest(): @pytest.mark.asyncio async def test_over_budget_is_blocked_before_the_executor(monkeypatch): - from app.capabilities.access import rest + from app.capabilities.core.access import rest from app.services.web_crawl_credit_service import InsufficientCreditsError async def _raise(*args, **kwargs): @@ -123,7 +123,7 @@ async def test_over_budget_is_blocked_before_the_executor(monkeypatch): @pytest.mark.asyncio async def test_rate_limit_blocks_the_workspace(monkeypatch): """The generated route enforces the per-workspace limit (429).""" - from app.capabilities.access import rate_limit, rest + from app.capabilities.core.access import rate_limit, rest monkeypatch.setattr(rest, "check_workspace_access", _noop_async, raising=True) monkeypatch.setattr( @@ -153,7 +153,7 @@ async def test_rate_limit_blocks_the_workspace(monkeypatch): async def test_success_charges_once(monkeypatch): from unittest.mock import AsyncMock - from app.capabilities.access import rest + from app.capabilities.core.access import rest charge = AsyncMock() monkeypatch.setattr(rest, "charge_capability", charge, raising=True) diff --git a/surfsense_backend/tests/unit/capabilities/test_billing.py b/surfsense_backend/tests/unit/capabilities/test_billing.py index 1f70afc56..c0a74878b 100644 --- a/surfsense_backend/tests/unit/capabilities/test_billing.py +++ b/surfsense_backend/tests/unit/capabilities/test_billing.py @@ -11,9 +11,9 @@ from uuid import UUID import pytest -import app.capabilities.billing as billing -from app.capabilities.billing import charge_capability, gate_capability -from app.capabilities.types import BillingUnit, CapabilityContext +import app.capabilities.core.billing as billing +from app.capabilities.core.billing import charge_capability, gate_capability +from app.capabilities.core.types import BillingUnit, CapabilityContext from app.capabilities.web.scrape.schemas import ScrapeInput, ScrapeOutput, ScrapeRow from app.config import config from app.services.web_crawl_credit_service import InsufficientCreditsError diff --git a/surfsense_backend/tests/unit/capabilities/test_registry.py b/surfsense_backend/tests/unit/capabilities/test_registry.py index 3b941224c..c603c02eb 100644 --- a/surfsense_backend/tests/unit/capabilities/test_registry.py +++ b/surfsense_backend/tests/unit/capabilities/test_registry.py @@ -7,8 +7,8 @@ import pytest from app.capabilities import ( web, # noqa: F401 — importing the namespace registers its verbs ) -from app.capabilities.store import get_capability -from app.capabilities.types import BillingUnit +from app.capabilities.core.store import get_capability +from app.capabilities.core.types import BillingUnit from app.capabilities.web.discover.schemas import DiscoverInput, DiscoverOutput from app.capabilities.web.scrape.schemas import ScrapeInput, ScrapeOutput From 6c3735db19b68876bdb0d3e101361fd9d29605af Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 20:18:53 +0200 Subject: [PATCH 45/64] feat(capabilities): add youtube.scrape and youtube.comments verbs --- .../app/capabilities/youtube/__init__.py | 6 ++ .../capabilities/youtube/comments/__init__.py | 3 + .../youtube/comments/definition.py | 23 +++++++ .../capabilities/youtube/comments/executor.py | 30 +++++++++ .../capabilities/youtube/comments/schemas.py | 44 +++++++++++++ .../capabilities/youtube/scrape/__init__.py | 3 + .../capabilities/youtube/scrape/definition.py | 24 +++++++ .../capabilities/youtube/scrape/executor.py | 36 ++++++++++ .../capabilities/youtube/scrape/schemas.py | 66 +++++++++++++++++++ 9 files changed, 235 insertions(+) create mode 100644 surfsense_backend/app/capabilities/youtube/__init__.py create mode 100644 surfsense_backend/app/capabilities/youtube/comments/__init__.py create mode 100644 surfsense_backend/app/capabilities/youtube/comments/definition.py create mode 100644 surfsense_backend/app/capabilities/youtube/comments/executor.py create mode 100644 surfsense_backend/app/capabilities/youtube/comments/schemas.py create mode 100644 surfsense_backend/app/capabilities/youtube/scrape/__init__.py create mode 100644 surfsense_backend/app/capabilities/youtube/scrape/definition.py create mode 100644 surfsense_backend/app/capabilities/youtube/scrape/executor.py create mode 100644 surfsense_backend/app/capabilities/youtube/scrape/schemas.py diff --git a/surfsense_backend/app/capabilities/youtube/__init__.py b/surfsense_backend/app/capabilities/youtube/__init__.py new file mode 100644 index 000000000..c8cda1bff --- /dev/null +++ b/surfsense_backend/app/capabilities/youtube/__init__.py @@ -0,0 +1,6 @@ +"""``youtube.*`` namespace: platform-native YouTube data verbs.""" + +from __future__ import annotations + +from app.capabilities.youtube.comments import definition as _comments # noqa: F401 +from app.capabilities.youtube.scrape import definition as _scrape # noqa: F401 diff --git a/surfsense_backend/app/capabilities/youtube/comments/__init__.py b/surfsense_backend/app/capabilities/youtube/comments/__init__.py new file mode 100644 index 000000000..cd4196de3 --- /dev/null +++ b/surfsense_backend/app/capabilities/youtube/comments/__init__.py @@ -0,0 +1,3 @@ +"""``youtube.comments`` verb: video URLs → comment items (+ replies).""" + +from __future__ import annotations diff --git a/surfsense_backend/app/capabilities/youtube/comments/definition.py b/surfsense_backend/app/capabilities/youtube/comments/definition.py new file mode 100644 index 000000000..7ed3fb345 --- /dev/null +++ b/surfsense_backend/app/capabilities/youtube/comments/definition.py @@ -0,0 +1,23 @@ +"""``youtube.comments`` capability registration (free — see 04-capabilities open item).""" + +from __future__ import annotations + +from app.capabilities.core import Capability, register_capability +from app.capabilities.youtube.comments.executor import build_comments_executor +from app.capabilities.youtube.comments.schemas import CommentsInput, CommentsOutput + +YOUTUBE_COMMENTS = Capability( + name="youtube.comments", + description=( + "Fetch public comments (and their replies) for one or more YouTube " + "videos. Give it the video URLs; returns structured comment items with " + "author, text, like count, reply relationships, and timestamps. Use it " + "to gauge sentiment or pull discussion on specific videos." + ), + input_schema=CommentsInput, + output_schema=CommentsOutput, + executor=build_comments_executor(), + billing_unit=None, +) + +register_capability(YOUTUBE_COMMENTS) diff --git a/surfsense_backend/app/capabilities/youtube/comments/executor.py b/surfsense_backend/app/capabilities/youtube/comments/executor.py new file mode 100644 index 000000000..f4f28bef5 --- /dev/null +++ b/surfsense_backend/app/capabilities/youtube/comments/executor.py @@ -0,0 +1,30 @@ +"""``youtube.comments`` executor: verb input → Apify actor → comment items.""" + +from __future__ import annotations + +from collections.abc import Awaitable, Callable + +from app.capabilities.core import Executor +from app.capabilities.youtube.comments.schemas import CommentsInput, CommentsOutput +from app.proprietary.scrapers.youtube import ( + YouTubeCommentsInput, + scrape_comments, +) + +CommentsFn = Callable[[YouTubeCommentsInput], Awaitable[list[dict]]] + + +def build_comments_executor(scrape_fn: CommentsFn | None = None) -> Executor: + """Bind the executor to a comments scraper fn (defaults to the proprietary actor).""" + scrape_fn = scrape_fn or scrape_comments + + async def execute(payload: CommentsInput) -> CommentsOutput: + actor_input = YouTubeCommentsInput( + startUrls=[{"url": url} for url in payload.urls], + maxComments=payload.max_comments, + sortCommentsBy=payload.sort_by, + ) + items = await scrape_fn(actor_input) + return CommentsOutput(items=items) + + return execute diff --git a/surfsense_backend/app/capabilities/youtube/comments/schemas.py b/surfsense_backend/app/capabilities/youtube/comments/schemas.py new file mode 100644 index 000000000..c656b1f55 --- /dev/null +++ b/surfsense_backend/app/capabilities/youtube/comments/schemas.py @@ -0,0 +1,44 @@ +"""``youtube.comments`` I/O contracts. + +A lean surface over the Apify-compatible ``YouTubeCommentsInput``; the actor's +``CommentItem`` is reused verbatim as the output element for parity. +""" + +from __future__ import annotations + +from typing import Literal + +from pydantic import BaseModel, Field + +from app.proprietary.scrapers.youtube import CommentItem + +MAX_COMMENT_VIDEOS = 20 +"""Per-call cap on how many video URLs one request may harvest comments from.""" + + +class CommentsInput(BaseModel): + urls: list[str] = Field( + min_length=1, + max_length=MAX_COMMENT_VIDEOS, + description="YouTube video URLs to fetch comments (and replies) for (1-20).", + ) + max_comments: int = Field( + default=20, + ge=1, + le=100_000, + description=( + "Max items returned per video, counting both top-level comments and " + "their replies." + ), + ) + sort_by: Literal["TOP_COMMENTS", "NEWEST_FIRST"] = Field( + default="NEWEST_FIRST", + description="Comment ordering: most-liked first, or most-recent first.", + ) + + +class CommentsOutput(BaseModel): + items: list[CommentItem] = Field( + default_factory=list, + description="One item per comment or reply, in the scraper's emission order.", + ) diff --git a/surfsense_backend/app/capabilities/youtube/scrape/__init__.py b/surfsense_backend/app/capabilities/youtube/scrape/__init__.py new file mode 100644 index 000000000..ca3c96599 --- /dev/null +++ b/surfsense_backend/app/capabilities/youtube/scrape/__init__.py @@ -0,0 +1,3 @@ +"""``youtube.scrape`` verb: YouTube URLs / search queries → video items.""" + +from __future__ import annotations diff --git a/surfsense_backend/app/capabilities/youtube/scrape/definition.py b/surfsense_backend/app/capabilities/youtube/scrape/definition.py new file mode 100644 index 000000000..a357f809b --- /dev/null +++ b/surfsense_backend/app/capabilities/youtube/scrape/definition.py @@ -0,0 +1,24 @@ +"""``youtube.scrape`` capability registration (free — see 04-capabilities open item).""" + +from __future__ import annotations + +from app.capabilities.core import Capability, register_capability +from app.capabilities.youtube.scrape.executor import build_scrape_executor +from app.capabilities.youtube.scrape.schemas import ScrapeInput, ScrapeOutput + +YOUTUBE_SCRAPE = Capability( + name="youtube.scrape", + description=( + "Scrape public YouTube data. Give it YouTube URLs (video, channel, " + "playlist, shorts, or hashtag) and/or search queries, and it returns " + "structured video items — title, views, likes, publish date, channel " + "info, description, and optionally subtitles. Use search_queries to " + "discover videos, or urls to pull a known video/channel/playlist." + ), + input_schema=ScrapeInput, + output_schema=ScrapeOutput, + executor=build_scrape_executor(), + billing_unit=None, +) + +register_capability(YOUTUBE_SCRAPE) diff --git a/surfsense_backend/app/capabilities/youtube/scrape/executor.py b/surfsense_backend/app/capabilities/youtube/scrape/executor.py new file mode 100644 index 000000000..82760e536 --- /dev/null +++ b/surfsense_backend/app/capabilities/youtube/scrape/executor.py @@ -0,0 +1,36 @@ +"""``youtube.scrape`` executor: verb input → Apify actor → video items.""" + +from __future__ import annotations + +from collections.abc import Awaitable, Callable + +from app.capabilities.core import Executor +from app.capabilities.youtube.scrape.schemas import ScrapeInput, ScrapeOutput +from app.proprietary.scrapers.youtube import ( + YouTubeScrapeInput, + scrape_youtube, +) + +ScrapeFn = Callable[[YouTubeScrapeInput], Awaitable[list[dict]]] + + +def build_scrape_executor(scrape_fn: ScrapeFn | None = None) -> Executor: + """Bind the executor to a scraper fn (defaults to the proprietary actor).""" + scrape_fn = scrape_fn or scrape_youtube + + async def execute(payload: ScrapeInput) -> ScrapeOutput: + # Channels emit three content types; cap each at the caller's max_results + # so a channel scrape isn't silently limited to plain videos only. + actor_input = YouTubeScrapeInput( + startUrls=[{"url": url} for url in payload.urls], + searchQueries=payload.search_queries, + maxResults=payload.max_results, + maxResultsShorts=payload.max_results, + maxResultStreams=payload.max_results, + downloadSubtitles=payload.download_subtitles, + subtitlesLanguage=payload.subtitles_language, + ) + items = await scrape_fn(actor_input) + return ScrapeOutput(items=items) + + return execute diff --git a/surfsense_backend/app/capabilities/youtube/scrape/schemas.py b/surfsense_backend/app/capabilities/youtube/scrape/schemas.py new file mode 100644 index 000000000..c24478b11 --- /dev/null +++ b/surfsense_backend/app/capabilities/youtube/scrape/schemas.py @@ -0,0 +1,66 @@ +"""``youtube.scrape`` I/O contracts. + +A lean, agent-friendly surface over the Apify-compatible ``YouTubeScrapeInput`` +(``app/proprietary/scrapers/youtube``). The executor maps this to the full actor +input; the actor's ``VideoItem`` is reused verbatim as the output element so the +verb stays parity-faithful with the underlying Apify shape. +""" + +from __future__ import annotations + +from pydantic import BaseModel, Field, model_validator + +from app.proprietary.scrapers.youtube import VideoItem + +MAX_YOUTUBE_SOURCES = 20 +"""Per-call cap on URLs + queries: bounds a synchronous request's fan-out (05).""" + + +class ScrapeInput(BaseModel): + urls: list[str] = Field( + default_factory=list, + max_length=MAX_YOUTUBE_SOURCES, + description=( + "YouTube URLs to scrape: video, channel (/@handle or /channel/UC...), " + "playlist (?list=...), shorts, or hashtag pages. Provide these OR " + "search_queries (at least one is required)." + ), + ) + search_queries: list[str] = Field( + default_factory=list, + max_length=MAX_YOUTUBE_SOURCES, + description=( + "Search terms to run on YouTube; each returns up to max_results videos. " + "Provide these OR urls (at least one is required)." + ), + ) + max_results: int = Field( + default=10, + ge=1, + le=1000, + description=( + "Max items to return per source and per content type (videos, shorts, " + "streams are capped independently for a channel)." + ), + ) + download_subtitles: bool = Field( + default=False, + description="Also fetch each video's subtitle track (slower; more requests).", + ) + subtitles_language: str = Field( + default="en", + description="Subtitle language code (e.g. 'en', 'fr'). Used when download_subtitles is true.", + ) + + @model_validator(mode="after") + def _require_a_source(self) -> ScrapeInput: + if not self.urls and not self.search_queries: + raise ValueError("Provide at least one of 'urls' or 'search_queries'.") + return self + + +class ScrapeOutput(BaseModel): + items: list[VideoItem] = Field( + default_factory=list, + description="One video item per result, in the scraper's emission order.", + ) From 0b2dba7a5796aa6f950e0ac618aafb2aa0eaba55 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 20:18:53 +0200 Subject: [PATCH 46/64] feat(capabilities): expose youtube.* verbs on the REST door --- surfsense_backend/app/routes/__init__.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/surfsense_backend/app/routes/__init__.py b/surfsense_backend/app/routes/__init__.py index 9f42f11e1..f4e2c5e6c 100644 --- a/surfsense_backend/app/routes/__init__.py +++ b/surfsense_backend/app/routes/__init__.py @@ -1,6 +1,8 @@ from fastapi import APIRouter, Depends -import app.capabilities.web # noqa: F401 (registers web.* verbs before the door builds) +# Import verb namespaces for their registration side effects before the door builds. +import app.capabilities.web +import app.capabilities.youtube # noqa: F401 from app.automations.api import router as automations_router from app.capabilities.core.access.rest import build_capabilities_router from app.file_storage.api import router as file_storage_router From c467a99dc6a887777bd12b4d77144a41400d9d9e Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 20:18:53 +0200 Subject: [PATCH 47/64] feat(scraping-agent): wire youtube verbs into subagent tools --- .../subagents/builtins/scraping/tools/index.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/index.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/index.py index 8416b8b05..43923dbb9 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/index.py +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/index.py @@ -10,6 +10,8 @@ from app.agents.chat.multi_agent_chat.shared.permissions import Ruleset from app.capabilities.core.access.agent import build_capability_tools from app.capabilities.web.discover.definition import WEB_DISCOVER from app.capabilities.web.scrape.definition import WEB_SCRAPE +from app.capabilities.youtube.comments.definition import YOUTUBE_COMMENTS +from app.capabilities.youtube.scrape.definition import YOUTUBE_SCRAPE from .refresh_watch import create_refresh_watch_tool from .start_watch import create_start_watch_tool @@ -19,7 +21,7 @@ NAME = "scraping" RULESET = Ruleset(origin=NAME, rules=[]) -_CI_VERBS = [WEB_DISCOVER, WEB_SCRAPE] +_CI_VERBS = [WEB_DISCOVER, WEB_SCRAPE, YOUTUBE_SCRAPE, YOUTUBE_COMMENTS] def load_tools( From f967cebe92fc2dbbe04dc41176091819d2f0d6cb Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 20:18:53 +0200 Subject: [PATCH 48/64] docs(scraping-agent): surface youtube in description and prompt --- .../subagents/builtins/scraping/description.md | 4 ++-- .../subagents/builtins/scraping/system_prompt.md | 3 +++ 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/description.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/description.md index f6b2f8ada..1b0e087f6 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/description.md +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/description.md @@ -1,2 +1,2 @@ -Scraping specialist: fetches live public web pages, searches the web to find pages, and compares fresh data against earlier findings in this chat. -Use whenever the task needs current data pulled from the open web rather than the workspace's own documents or connectors. Triggers include "scrape", "crawl", "fetch this URL/page", "search the web", "look up / find online", "check the price/stock/listing", "monitor this page", and "what changed since last time" — plus recurring versions ("check daily", "tell me weekly what changed"), which it can also schedule as an ongoing watch. +Scraping specialist: fetches live public web pages, searches the web to find pages, pulls structured data from YouTube (videos, channels, playlists, search, and comments), and compares fresh data against earlier findings in this chat. +Use whenever the task needs current data pulled from the open web or YouTube rather than the workspace's own documents or connectors. Triggers include "scrape", "crawl", "fetch this URL/page", "search the web", "look up / find online", "check the price/stock/listing", "monitor this page", "what changed since last time" — plus YouTube-specific asks like "get this YouTube video/channel/playlist", "find videos about X on YouTube", "get the comments on this video", "what are people saying about this video" — and recurring versions ("check daily", "tell me weekly what changed"), which it can also schedule as an ongoing watch. diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/system_prompt.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/system_prompt.md index 7f0d6841e..d4b0309a4 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/system_prompt.md +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/system_prompt.md @@ -8,6 +8,8 @@ Answer the delegated question from live evidence gathered with your data verbs, - `web_discover` - `web_scrape` +- `youtube_scrape` +- `youtube_comments` - `start_watch` - `stop_watch` - `refresh_watch` @@ -16,6 +18,7 @@ Answer the delegated question from live evidence gathered with your data verbs, - Named URLs: `web_scrape` them directly. Otherwise `web_discover` first, then `web_scrape` the most relevant hits. - Read several pages in one batched `web_scrape` call rather than many single-URL calls. +- YouTube: use `youtube_scrape` for videos/channels/playlists (pass `urls`) or to find videos (pass `search_queries`); use `youtube_comments` to pull a video's comments. Prefer these over `web_scrape` for youtube.com links. - "What changed" / monitoring: scrape the current values, compare against the prior values in this conversation's earlier tool results, and report concrete deltas (added, removed, old -> new). - Recurring intent ("check daily", "tell me weekly what changed"): answer now, then `start_watch` with a self-contained question, a cron cadence, and an IANA timezone. Use `stop_watch` / `refresh_watch` to end or immediately re-run an existing watch. From 5d0bd8125bec08c7ec92d50d6c59832af9946fd7 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 20:18:53 +0200 Subject: [PATCH 49/64] test(capabilities): cover youtube verb schemas, executors, and registry --- .../youtube/comments/test_executor.py | 55 ++++++++++++ .../youtube/comments/test_schemas.py | 35 ++++++++ .../youtube/scrape/test_executor.py | 87 +++++++++++++++++++ .../youtube/scrape/test_schemas.py | 40 +++++++++ .../capabilities/youtube/test_registry.py | 32 +++++++ 5 files changed, 249 insertions(+) create mode 100644 surfsense_backend/tests/unit/capabilities/youtube/comments/test_executor.py create mode 100644 surfsense_backend/tests/unit/capabilities/youtube/comments/test_schemas.py create mode 100644 surfsense_backend/tests/unit/capabilities/youtube/scrape/test_executor.py create mode 100644 surfsense_backend/tests/unit/capabilities/youtube/scrape/test_schemas.py create mode 100644 surfsense_backend/tests/unit/capabilities/youtube/test_registry.py diff --git a/surfsense_backend/tests/unit/capabilities/youtube/comments/test_executor.py b/surfsense_backend/tests/unit/capabilities/youtube/comments/test_executor.py new file mode 100644 index 000000000..94fe941ad --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/youtube/comments/test_executor.py @@ -0,0 +1,55 @@ +"""``youtube.comments`` executor: verb input → actor input mapping → typed items.""" + +from __future__ import annotations + +import pytest + +from app.capabilities.youtube.comments.executor import build_comments_executor +from app.capabilities.youtube.comments.schemas import CommentsInput, CommentsOutput +from app.proprietary.scrapers.youtube import YouTubeCommentsInput + +pytestmark = pytest.mark.unit + + +class _FakeScraper: + def __init__(self, items: list[dict]): + self._items = items + self.calls: list[YouTubeCommentsInput] = [] + + async def __call__(self, actor_input: YouTubeCommentsInput) -> list[dict]: + self.calls.append(actor_input) + return self._items + + +async def test_maps_urls_and_wraps_comment_items(): + scraper = _FakeScraper([{"cid": "c1", "comment": "nice", "author": "@a"}]) + execute = build_comments_executor(scrape_fn=scraper) + + out = await execute(CommentsInput(urls=["https://www.youtube.com/watch?v=abc"])) + + assert isinstance(out, CommentsOutput) + assert len(out.items) == 1 + assert out.items[0].cid == "c1" + assert out.items[0].comment == "nice" + + (actor_input,) = scraper.calls + assert [u.url for u in actor_input.startUrls] == [ + "https://www.youtube.com/watch?v=abc" + ] + + +async def test_forwards_max_comments_and_sort(): + scraper = _FakeScraper([]) + execute = build_comments_executor(scrape_fn=scraper) + + await execute( + CommentsInput( + urls=["https://youtu.be/abc"], + max_comments=50, + sort_by="TOP_COMMENTS", + ) + ) + + (actor_input,) = scraper.calls + assert actor_input.maxComments == 50 + assert actor_input.sortCommentsBy == "TOP_COMMENTS" diff --git a/surfsense_backend/tests/unit/capabilities/youtube/comments/test_schemas.py b/surfsense_backend/tests/unit/capabilities/youtube/comments/test_schemas.py new file mode 100644 index 000000000..129473f7b --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/youtube/comments/test_schemas.py @@ -0,0 +1,35 @@ +"""``youtube.comments`` input guards: URLs required and batch/count bounded.""" + +from __future__ import annotations + +import pytest +from pydantic import ValidationError + +from app.capabilities.youtube.comments.schemas import ( + MAX_COMMENT_VIDEOS, + CommentsInput, +) + +pytestmark = pytest.mark.unit + + +def test_rejects_empty_url_batch(): + with pytest.raises(ValidationError): + CommentsInput(urls=[]) + + +def test_rejects_batch_over_the_cap(): + too_many = [f"https://youtu.be/{i}" for i in range(MAX_COMMENT_VIDEOS + 1)] + with pytest.raises(ValidationError): + CommentsInput(urls=too_many) + + +def test_defaults_max_comments_and_newest_first(): + payload = CommentsInput(urls=["https://youtu.be/abc"]) + assert payload.max_comments == 20 + assert payload.sort_by == "NEWEST_FIRST" + + +def test_rejects_zero_max_comments(): + with pytest.raises(ValidationError): + CommentsInput(urls=["https://youtu.be/abc"], max_comments=0) diff --git a/surfsense_backend/tests/unit/capabilities/youtube/scrape/test_executor.py b/surfsense_backend/tests/unit/capabilities/youtube/scrape/test_executor.py new file mode 100644 index 000000000..08f15981b --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/youtube/scrape/test_executor.py @@ -0,0 +1,87 @@ +"""``youtube.scrape`` executor: verb input → actor input mapping → typed items. + +Boundary mocked: the proprietary scraper (injected fake). NOT mocked: the verb's +own payload→YouTubeScrapeInput mapping and the dict→VideoItem wrapping. +""" + +from __future__ import annotations + +import pytest + +from app.capabilities.youtube.scrape.executor import build_scrape_executor +from app.capabilities.youtube.scrape.schemas import ScrapeInput, ScrapeOutput +from app.proprietary.scrapers.youtube import YouTubeScrapeInput + +pytestmark = pytest.mark.unit + + +class _FakeScraper: + """Records the actor input it was called with and returns canned items.""" + + def __init__(self, items: list[dict]): + self._items = items + self.calls: list[YouTubeScrapeInput] = [] + + async def __call__(self, actor_input: YouTubeScrapeInput) -> list[dict]: + self.calls.append(actor_input) + return self._items + + +async def test_maps_urls_to_start_urls_and_wraps_items(): + scraper = _FakeScraper([{"id": "abc", "title": "Hello"}]) + execute = build_scrape_executor(scrape_fn=scraper) + + out = await execute(ScrapeInput(urls=["https://www.youtube.com/watch?v=abc"])) + + assert isinstance(out, ScrapeOutput) + assert len(out.items) == 1 + assert out.items[0].id == "abc" + assert out.items[0].title == "Hello" + + (actor_input,) = scraper.calls + assert [u.url for u in actor_input.startUrls] == [ + "https://www.youtube.com/watch?v=abc" + ] + assert actor_input.searchQueries == [] + + +async def test_forwards_search_queries(): + scraper = _FakeScraper([]) + execute = build_scrape_executor(scrape_fn=scraper) + + await execute(ScrapeInput(search_queries=["python", "rust"])) + + (actor_input,) = scraper.calls + assert actor_input.searchQueries == ["python", "rust"] + assert actor_input.startUrls == [] + + +async def test_max_results_caps_every_content_type(): + scraper = _FakeScraper([]) + execute = build_scrape_executor(scrape_fn=scraper) + + await execute(ScrapeInput(search_queries=["x"], max_results=7)) + + (actor_input,) = scraper.calls + # Videos, shorts, and streams must all inherit the caller's cap, otherwise a + # channel scrape would silently return only plain videos (actor default 0). + assert actor_input.maxResults == 7 + assert actor_input.maxResultsShorts == 7 + assert actor_input.maxResultStreams == 7 + + +async def test_forwards_subtitle_options(): + scraper = _FakeScraper([]) + execute = build_scrape_executor(scrape_fn=scraper) + + await execute( + ScrapeInput( + urls=["https://youtu.be/abc"], + download_subtitles=True, + subtitles_language="fr", + ) + ) + + (actor_input,) = scraper.calls + assert actor_input.downloadSubtitles is True + assert actor_input.subtitlesLanguage == "fr" diff --git a/surfsense_backend/tests/unit/capabilities/youtube/scrape/test_schemas.py b/surfsense_backend/tests/unit/capabilities/youtube/scrape/test_schemas.py new file mode 100644 index 000000000..9d0477dfe --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/youtube/scrape/test_schemas.py @@ -0,0 +1,40 @@ +"""``youtube.scrape`` input guards: a source is required and the batch is bounded.""" + +from __future__ import annotations + +import pytest +from pydantic import ValidationError + +from app.capabilities.youtube.scrape.schemas import ( + MAX_YOUTUBE_SOURCES, + ScrapeInput, +) + +pytestmark = pytest.mark.unit + + +def test_rejects_input_with_neither_urls_nor_queries(): + with pytest.raises(ValidationError): + ScrapeInput() + + +def test_accepts_urls_only(): + payload = ScrapeInput(urls=["https://www.youtube.com/watch?v=abc"]) + assert payload.search_queries == [] + + +def test_accepts_search_queries_only(): + payload = ScrapeInput(search_queries=["python tutorial"]) + assert payload.urls == [] + + +def test_max_results_defaults_and_is_bounded(): + assert ScrapeInput(search_queries=["x"]).max_results == 10 + with pytest.raises(ValidationError): + ScrapeInput(search_queries=["x"], max_results=0) + + +def test_rejects_more_sources_than_the_cap(): + too_many = [f"https://youtu.be/{i}" for i in range(MAX_YOUTUBE_SOURCES + 1)] + with pytest.raises(ValidationError): + ScrapeInput(urls=too_many) diff --git a/surfsense_backend/tests/unit/capabilities/youtube/test_registry.py b/surfsense_backend/tests/unit/capabilities/youtube/test_registry.py new file mode 100644 index 000000000..756b65176 --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/youtube/test_registry.py @@ -0,0 +1,32 @@ +"""The youtube namespace registers each verb as one Capability the doors/agent read.""" + +from __future__ import annotations + +import pytest + +from app.capabilities import ( + youtube, # noqa: F401 — importing the namespace registers its verbs +) +from app.capabilities.core.store import get_capability +from app.capabilities.youtube.comments.schemas import CommentsInput, CommentsOutput +from app.capabilities.youtube.scrape.schemas import ScrapeInput, ScrapeOutput + +pytestmark = pytest.mark.unit + + +def test_youtube_scrape_is_registered_and_free(): + cap = get_capability("youtube.scrape") + + assert cap.name == "youtube.scrape" + assert cap.input_schema is ScrapeInput + assert cap.output_schema is ScrapeOutput + assert cap.billing_unit is None + + +def test_youtube_comments_is_registered_and_free(): + cap = get_capability("youtube.comments") + + assert cap.name == "youtube.comments" + assert cap.input_schema is CommentsInput + assert cap.output_schema is CommentsOutput + assert cap.billing_unit is None From f82fae39734a335ed5d29567ec59ac00e7c703d3 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Fri, 3 Jul 2026 10:50:41 +0200 Subject: [PATCH 50/64] feat(web-crawler): add site_crawler spider + url_policy over the fetch engine --- .../app/proprietary/web_crawler/__init__.py | 9 +- .../app/proprietary/web_crawler/connector.py | 4 + .../proprietary/web_crawler/site_crawler.py | 123 +++++++++++++ .../app/proprietary/web_crawler/url_policy.py | 59 +++++++ .../web_crawler/test_connector_links.py | 29 +++ .../web_crawler/test_site_crawler.py | 165 ++++++++++++++++++ .../web_crawler/test_url_policy.py | 89 ++++++++++ 7 files changed, 477 insertions(+), 1 deletion(-) create mode 100644 surfsense_backend/app/proprietary/web_crawler/site_crawler.py create mode 100644 surfsense_backend/app/proprietary/web_crawler/url_policy.py create mode 100644 surfsense_backend/tests/unit/proprietary/web_crawler/test_connector_links.py create mode 100644 surfsense_backend/tests/unit/proprietary/web_crawler/test_site_crawler.py create mode 100644 surfsense_backend/tests/unit/proprietary/web_crawler/test_url_policy.py diff --git a/surfsense_backend/app/proprietary/web_crawler/__init__.py b/surfsense_backend/app/proprietary/web_crawler/__init__.py index d2a7d118a..9fc22f8fd 100644 --- a/surfsense_backend/app/proprietary/web_crawler/__init__.py +++ b/surfsense_backend/app/proprietary/web_crawler/__init__.py @@ -9,5 +9,12 @@ from app.proprietary.web_crawler.connector import ( CrawlOutcomeStatus, WebCrawlerConnector, ) +from app.proprietary.web_crawler.site_crawler import CrawlPage, crawl_site -__all__ = ["CrawlOutcome", "CrawlOutcomeStatus", "WebCrawlerConnector"] +__all__ = [ + "CrawlOutcome", + "CrawlOutcomeStatus", + "CrawlPage", + "WebCrawlerConnector", + "crawl_site", +] diff --git a/surfsense_backend/app/proprietary/web_crawler/connector.py b/surfsense_backend/app/proprietary/web_crawler/connector.py index 7604bdc73..b54bcc170 100644 --- a/surfsense_backend/app/proprietary/web_crawler/connector.py +++ b/surfsense_backend/app/proprietary/web_crawler/connector.py @@ -34,6 +34,7 @@ from scrapling.engines.toolbelt import is_proxy_error from scrapling.fetchers import AsyncFetcher, DynamicFetcher, StealthyFetcher from app.proprietary.web_crawler.captcha import build_captcha_page_action +from app.proprietary.web_crawler.url_policy import extract_links from app.proprietary.web_crawler.stealth import ( build_stealthy_kwargs, get_stealth_config, @@ -601,6 +602,9 @@ class WebCrawlerConnector: "content": content, "metadata": metadata, "crawler_type": crawler_type, + # Next-hop targets for ``site_crawler.crawl_site``; ignored by + # single-URL callers. + "links": extract_links(raw_html, url), } @staticmethod diff --git a/surfsense_backend/app/proprietary/web_crawler/site_crawler.py b/surfsense_backend/app/proprietary/web_crawler/site_crawler.py new file mode 100644 index 000000000..49a7f872e --- /dev/null +++ b/surfsense_backend/app/proprietary/web_crawler/site_crawler.py @@ -0,0 +1,123 @@ +# SurfSense proprietary crawler engine. +# +# Part of the ``app.proprietary`` package; licensed separately from the +# Apache-2.0 project root (see ``app/proprietary/LICENSE``). +"""Depth-bounded site crawl built on the single-URL engine (``crawl_url``). + +Breadth-first frontier: fetch a page, follow its same-site links one hop deeper, +dedupe by canonical URL, stop at ``max_crawl_pages``. Every fetch runs through +``crawl_url`` so tiered fetch, proxy, and captcha handling are reused. + +Sequential today; ``_fetch_page`` isolates one unit of work so a bounded worker +pool can replace the loop later without changing the traversal. +""" + +from __future__ import annotations + +from collections import deque +from dataclasses import dataclass + +from app.proprietary.web_crawler.connector import ( + CrawlOutcome, + CrawlOutcomeStatus, + WebCrawlerConnector, +) +from app.proprietary.web_crawler.url_policy import canonicalize_url, host_of, same_site + + +@dataclass(frozen=True) +class CrawlPage: + """A fetched page plus its crawl provenance (depth and referrer).""" + + url: str + status: CrawlOutcomeStatus + depth: int + referrer: str | None = None + loaded_url: str | None = None + content: str | None = None + metadata: dict[str, str] | None = None + error: str | None = None + captcha_attempts: int = 0 + captcha_solved: bool = False + + +async def crawl_site( + engine: WebCrawlerConnector, + start_urls: list[str], + *, + max_crawl_depth: int, + max_crawl_pages: int, +) -> list[CrawlPage]: + """Crawl ``start_urls`` up to ``max_crawl_depth`` hops / ``max_crawl_pages`` pages. + + Depth 0 fetches only the start URLs. Links are followed only from successful + pages, under the depth cap, and only on a start URL's site. Start URLs count + toward ``max_crawl_pages``. Order is BFS from the seeds. + """ + allowed_hosts = {host_of(url) for url in start_urls} + visited: set[str] = set() + frontier: deque[tuple[str, int, str | None]] = deque() + for seed in start_urls: + key = canonicalize_url(seed) + if key not in visited: + visited.add(key) + frontier.append((seed, 0, None)) + + pages: list[CrawlPage] = [] + while frontier and len(pages) < max_crawl_pages: + url, depth, referrer = frontier.popleft() + page, outcome = await _fetch_page(engine, url, depth, referrer) + pages.append(page) + + if depth >= max_crawl_depth: + continue + if outcome.status is not CrawlOutcomeStatus.SUCCESS or not outcome.result: + continue + for link in outcome.result.get("links", []): + key = canonicalize_url(link) + if key in visited or not same_site(link, allowed_hosts): + continue + visited.add(key) + frontier.append((link, depth + 1, url)) + return pages + + +async def _fetch_page( + engine: WebCrawlerConnector, + url: str, + depth: int, + referrer: str | None, +) -> tuple[CrawlPage, CrawlOutcome]: + """Fetch one URL and map it to a ``CrawlPage`` (the future concurrency unit).""" + outcome = await engine.crawl_url(url) + return _to_page(url, outcome, depth, referrer), outcome + + +def _to_page( + url: str, + outcome: CrawlOutcome, + depth: int, + referrer: str | None, +) -> CrawlPage: + result = outcome.result or {} + if outcome.status is CrawlOutcomeStatus.SUCCESS and result: + return CrawlPage( + url=url, + status=outcome.status, + depth=depth, + referrer=referrer, + loaded_url=result.get("loaded_url") or url, + content=result.get("content"), + metadata=result.get("metadata"), + captcha_attempts=outcome.captcha_attempts, + captcha_solved=outcome.captcha_solved, + ) + return CrawlPage( + url=url, + status=outcome.status, + depth=depth, + referrer=referrer, + error=outcome.error, + captcha_attempts=outcome.captcha_attempts, + captcha_solved=outcome.captcha_solved, + ) diff --git a/surfsense_backend/app/proprietary/web_crawler/url_policy.py b/surfsense_backend/app/proprietary/web_crawler/url_policy.py new file mode 100644 index 000000000..21f9215a3 --- /dev/null +++ b/surfsense_backend/app/proprietary/web_crawler/url_policy.py @@ -0,0 +1,59 @@ +# SurfSense proprietary crawler engine. +# +# Part of the ``app.proprietary`` package; licensed separately from the +# Apache-2.0 project root (see ``app/proprietary/LICENSE``). +"""URL helpers for the site crawler: link extraction, canonical key, same-site scope. + +Pure functions (no I/O) so the crawl frontier stays deterministic and testable. +""" + +from __future__ import annotations + +from urllib.parse import urldefrag, urljoin, urlsplit + +from lxml import html as lxml_html +from lxml.etree import ParserError +from w3lib.url import canonicalize_url as _w3lib_canonicalize_url + + +def extract_links(page_html: str | None, base_url: str) -> list[str]: + """Absolute, http(s), fragment-free, de-duplicated ```` targets. + + Relative hrefs resolve against ``base_url``; the page's own URL is dropped. + First-seen order is preserved to keep the frontier stable. + """ + if not page_html or not page_html.strip(): + return [] + try: + root = lxml_html.fromstring(page_html) + except (ParserError, ValueError): + return [] + + self_url, _ = urldefrag(base_url) + seen: set[str] = set() + links: list[str] = [] + for href in root.xpath("//a/@href"): + target, _ = urldefrag(urljoin(base_url, href.strip())) + if urlsplit(target).scheme not in ("http", "https"): + continue + if target == self_url or target in seen: + continue + seen.add(target) + links.append(target) + return links + + +def canonicalize_url(url: str) -> str: + """Stable visited-set key: sorts query, normalizes encoding, drops fragment.""" + return _w3lib_canonicalize_url(url, keep_fragments=False) + + +def host_of(url: str) -> str: + """Lowercased host with a leading ``www.`` removed, for same-site matching.""" + host = (urlsplit(url).hostname or "").lower() + return host[4:] if host.startswith("www.") else host + + +def same_site(url: str, allowed_hosts: set[str]) -> bool: + """Whether ``url``'s host (``www.``-normalized) is in ``allowed_hosts``.""" + return host_of(url) in allowed_hosts diff --git a/surfsense_backend/tests/unit/proprietary/web_crawler/test_connector_links.py b/surfsense_backend/tests/unit/proprietary/web_crawler/test_connector_links.py new file mode 100644 index 000000000..958a905ca --- /dev/null +++ b/surfsense_backend/tests/unit/proprietary/web_crawler/test_connector_links.py @@ -0,0 +1,29 @@ +"""``_build_result`` surfaces absolute page links for the spider to enqueue.""" + +from __future__ import annotations + +import pytest + +from app.proprietary.web_crawler import WebCrawlerConnector + +pytestmark = pytest.mark.unit + + +def test_build_result_includes_absolute_links() -> None: + html = ( + "" + 'A' + 'B' + "" + ) + + result = WebCrawlerConnector()._build_result( + html, + "https://example.com/", + "scrapling-static", + allow_raw_fallback=True, + ) + + assert result is not None + assert "https://example.com/a" in result["links"] + assert "https://example.com/b" in result["links"] diff --git a/surfsense_backend/tests/unit/proprietary/web_crawler/test_site_crawler.py b/surfsense_backend/tests/unit/proprietary/web_crawler/test_site_crawler.py new file mode 100644 index 000000000..cbb12d43b --- /dev/null +++ b/surfsense_backend/tests/unit/proprietary/web_crawler/test_site_crawler.py @@ -0,0 +1,165 @@ +"""``crawl_site`` BFS behavior: depth, same-site scope, dedupe, and page caps. + +Boundary mocked: the engine (a fake ``crawl_url`` serving a canned link graph). +NOT mocked: the frontier/depth/dedupe/scope logic under test. +""" + +from __future__ import annotations + +import pytest + +from app.proprietary.web_crawler import CrawlOutcome, CrawlOutcomeStatus +from app.proprietary.web_crawler.site_crawler import crawl_site + +pytestmark = pytest.mark.unit + +_SUCCESS = CrawlOutcomeStatus.SUCCESS +_FAILED = CrawlOutcomeStatus.FAILED + + +class _FakeEngine: + """Serves a canned ``(status, links)`` per URL and records fetch order.""" + + def __init__(self, graph: dict[str, tuple[CrawlOutcomeStatus, list[str]]]): + self._graph = graph + self.calls: list[str] = [] + + async def crawl_url(self, url: str) -> CrawlOutcome: + self.calls.append(url) + status, links = self._graph.get(url, (_FAILED, [])) + if status is _SUCCESS: + return CrawlOutcome( + status=_SUCCESS, + result={ + "content": f"C:{url}", + "metadata": {"title": url}, + "links": links, + }, + ) + return CrawlOutcome(status=status, error="boom") + + +async def test_depth_zero_fetches_only_the_seed() -> None: + engine = _FakeEngine({"https://e.com/": (_SUCCESS, ["https://e.com/a"])}) + + pages = await crawl_site( + engine, ["https://e.com/"], max_crawl_depth=0, max_crawl_pages=10 + ) + + assert engine.calls == ["https://e.com/"] + assert len(pages) == 1 + assert pages[0].depth == 0 + assert pages[0].referrer is None + assert pages[0].content == "C:https://e.com/" + + +async def test_depth_one_follows_same_site_links_only() -> None: + engine = _FakeEngine( + { + "https://e.com/": ( + _SUCCESS, + ["https://e.com/a", "https://e.com/b", "https://other.com/x"], + ), + "https://e.com/a": (_SUCCESS, []), + "https://e.com/b": (_SUCCESS, []), + } + ) + + pages = await crawl_site( + engine, ["https://e.com/"], max_crawl_depth=1, max_crawl_pages=10 + ) + + assert set(engine.calls) == { + "https://e.com/", + "https://e.com/a", + "https://e.com/b", + } + depths = {page.url: page.depth for page in pages} + assert depths["https://e.com/a"] == 1 + referrers = {page.url: page.referrer for page in pages} + assert referrers["https://e.com/a"] == "https://e.com/" + + +async def test_depth_caps_further_recursion() -> None: + engine = _FakeEngine( + { + "https://e.com/": (_SUCCESS, ["https://e.com/a"]), + "https://e.com/a": (_SUCCESS, ["https://e.com/b"]), + "https://e.com/b": (_SUCCESS, []), + } + ) + + pages = await crawl_site( + engine, ["https://e.com/"], max_crawl_depth=1, max_crawl_pages=10 + ) + + # depth 1 reaches /a but must NOT descend to /b (depth 2). + assert set(engine.calls) == {"https://e.com/", "https://e.com/a"} + assert all(page.url != "https://e.com/b" for page in pages) + + +async def test_max_pages_caps_total_fetches() -> None: + graph: dict[str, tuple[CrawlOutcomeStatus, list[str]]] = { + "https://e.com/": (_SUCCESS, [f"https://e.com/{i}" for i in range(10)]) + } + for i in range(10): + graph[f"https://e.com/{i}"] = (_SUCCESS, []) + engine = _FakeEngine(graph) + + pages = await crawl_site( + engine, ["https://e.com/"], max_crawl_depth=1, max_crawl_pages=3 + ) + + assert len(pages) == 3 + assert len(engine.calls) == 3 + + +async def test_dedupes_on_canonical_url() -> None: + engine = _FakeEngine( + { + "https://e.com/": ( + _SUCCESS, + ["https://e.com/a", "https://e.com/a#frag", "https://e.com/a?"], + ), + "https://e.com/a": (_SUCCESS, ["https://e.com/"]), # links back to seed + } + ) + + await crawl_site( + engine, ["https://e.com/"], max_crawl_depth=3, max_crawl_pages=10 + ) + + assert engine.calls.count("https://e.com/a") == 1 + assert engine.calls.count("https://e.com/") == 1 + + +async def test_failed_page_is_recorded_and_not_expanded() -> None: + engine = _FakeEngine({"https://e.com/": (_FAILED, [])}) + + pages = await crawl_site( + engine, ["https://e.com/"], max_crawl_depth=2, max_crawl_pages=10 + ) + + assert len(pages) == 1 + assert pages[0].status is _FAILED + assert pages[0].content is None + assert pages[0].error == "boom" + + +async def test_multiple_seeds_are_all_entry_points() -> None: + engine = _FakeEngine( + { + "https://a.com/": (_SUCCESS, []), + "https://b.com/": (_SUCCESS, []), + } + ) + + pages = await crawl_site( + engine, + ["https://a.com/", "https://b.com/"], + max_crawl_depth=0, + max_crawl_pages=10, + ) + + assert {page.url for page in pages} == {"https://a.com/", "https://b.com/"} + assert all(page.depth == 0 for page in pages) diff --git a/surfsense_backend/tests/unit/proprietary/web_crawler/test_url_policy.py b/surfsense_backend/tests/unit/proprietary/web_crawler/test_url_policy.py new file mode 100644 index 000000000..ca5ad06fa --- /dev/null +++ b/surfsense_backend/tests/unit/proprietary/web_crawler/test_url_policy.py @@ -0,0 +1,89 @@ +"""Pure URL-helper behavior for the site spider (link surfacing + dedupe scope).""" + +from __future__ import annotations + +import pytest + +from app.proprietary.web_crawler.url_policy import ( + canonicalize_url, + extract_links, + host_of, + same_site, +) + +pytestmark = pytest.mark.unit + +_HTML = """ + + About (relative) + Contact (absolute) + External + Anchor only + Mail + JS + Duplicate about + +""" + + +def test_extract_links_absolutizes_relative_hrefs() -> None: + links = extract_links(_HTML, "https://example.com/home") + assert "https://example.com/about" in links + assert "https://example.com/contact" in links + + +def test_extract_links_keeps_only_http_schemes() -> None: + links = extract_links(_HTML, "https://example.com/home") + assert all(link.startswith(("http://", "https://")) for link in links) + assert not any("mailto" in link or "javascript" in link for link in links) + + +def test_extract_links_drops_pure_anchor() -> None: + links = extract_links(_HTML, "https://example.com/home") + assert "https://example.com/home" not in links # bare "#top" resolves to self + + +def test_extract_links_dedupes_preserving_first_occurrence() -> None: + links = extract_links(_HTML, "https://example.com/home") + assert links.count("https://example.com/about") == 1 + + +def test_extract_links_strips_fragments() -> None: + assert extract_links('x', "https://e.com") == [ + "https://e.com/p" + ] + + +def test_extract_links_on_empty_or_blank_html_is_empty() -> None: + assert extract_links("", "https://e.com") == [] + assert extract_links(" ", "https://e.com") == [] + assert extract_links(None, "https://e.com") == [] + + +def test_canonicalize_lowercases_host_sorts_query_and_drops_fragment() -> None: + assert ( + canonicalize_url("https://E.com/a?b=2&a=1#frag") == "https://e.com/a?a=1&b=2" + ) + + +def test_canonicalize_collapses_fragment_and_empty_query_to_one_key() -> None: + # The three forms must dedupe to the same visited-set key. + canonical = canonicalize_url("https://e.com/a") + assert canonicalize_url("https://e.com/a#frag") == canonical + assert canonicalize_url("https://e.com/a?") == canonical + + +def test_canonicalize_keeps_nondefault_port() -> None: + assert canonicalize_url("https://e.com:8443/x") == "https://e.com:8443/x" + + +def test_host_of_strips_www_and_lowercases() -> None: + assert host_of("https://www.Example.com/x") == "example.com" + assert host_of("https://Example.com/x") == "example.com" + + +def test_same_site_matches_on_normalized_host() -> None: + allowed = {"example.com"} + assert same_site("https://www.example.com/a", allowed) is True + assert same_site("https://example.com/b", allowed) is True + assert same_site("https://other.com/c", allowed) is False From 9fe9c5b71d22a622461a67f677686c85db0bf44e Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Fri, 3 Jul 2026 10:51:05 +0200 Subject: [PATCH 51/64] feat(capabilities): replace web.scrape/web.discover with unified web.crawl web.crawl scrapes a single URL (maxCrawlDepth=0) or spiders a whole site, backed by the proprietary site_crawler engine. Rewires the scraping subagent tools and capability tests onto the new verb. --- .../builtins/scraping/tools/index.py | 5 +- .../app/capabilities/web/__init__.py | 5 +- .../app/capabilities/web/crawl/__init__.py | 3 + .../app/capabilities/web/crawl/definition.py | 25 ++++ .../app/capabilities/web/crawl/executor.py | 65 ++++++++ .../app/capabilities/web/crawl/schemas.py | 115 ++++++++++++++ .../app/capabilities/web/discover/__init__.py | 3 - .../capabilities/web/discover/definition.py | 23 --- .../app/capabilities/web/discover/executor.py | 42 ------ .../web/discover/providers/__init__.py | 3 - .../web/discover/providers/baidu.py | 53 ------- .../web/discover/providers/base.py | 22 --- .../web/discover/providers/linkup.py | 40 ----- .../web/discover/providers/searxng.py | 28 ---- .../app/capabilities/web/discover/schemas.py | 29 ---- .../app/capabilities/web/scrape/__init__.py | 3 - .../app/capabilities/web/scrape/definition.py | 23 --- .../app/capabilities/web/scrape/executor.py | 45 ------ .../app/capabilities/web/scrape/schemas.py | 67 --------- .../capabilities/access/test_rest_router.py | 3 +- .../tests/unit/capabilities/test_billing.py | 24 +-- .../tests/unit/capabilities/test_registry.py | 22 +-- .../capabilities/web/crawl/test_executor.py | 113 ++++++++++++++ .../capabilities/web/crawl/test_schemas.py | 67 +++++++++ .../web/discover/test_baidu_provider.py | 91 ----------- .../web/discover/test_executor.py | 99 ------------ .../web/discover/test_linkup_provider.py | 75 ---------- .../capabilities/web/discover/test_schemas.py | 22 --- .../web/discover/test_searxng_provider.py | 58 ------- .../capabilities/web/scrape/test_executor.py | 141 ------------------ .../capabilities/web/scrape/test_schemas.py | 47 ------ 31 files changed, 411 insertions(+), 950 deletions(-) create mode 100644 surfsense_backend/app/capabilities/web/crawl/__init__.py create mode 100644 surfsense_backend/app/capabilities/web/crawl/definition.py create mode 100644 surfsense_backend/app/capabilities/web/crawl/executor.py create mode 100644 surfsense_backend/app/capabilities/web/crawl/schemas.py delete mode 100644 surfsense_backend/app/capabilities/web/discover/__init__.py delete mode 100644 surfsense_backend/app/capabilities/web/discover/definition.py delete mode 100644 surfsense_backend/app/capabilities/web/discover/executor.py delete mode 100644 surfsense_backend/app/capabilities/web/discover/providers/__init__.py delete mode 100644 surfsense_backend/app/capabilities/web/discover/providers/baidu.py delete mode 100644 surfsense_backend/app/capabilities/web/discover/providers/base.py delete mode 100644 surfsense_backend/app/capabilities/web/discover/providers/linkup.py delete mode 100644 surfsense_backend/app/capabilities/web/discover/providers/searxng.py delete mode 100644 surfsense_backend/app/capabilities/web/discover/schemas.py delete mode 100644 surfsense_backend/app/capabilities/web/scrape/__init__.py delete mode 100644 surfsense_backend/app/capabilities/web/scrape/definition.py delete mode 100644 surfsense_backend/app/capabilities/web/scrape/executor.py delete mode 100644 surfsense_backend/app/capabilities/web/scrape/schemas.py create mode 100644 surfsense_backend/tests/unit/capabilities/web/crawl/test_executor.py create mode 100644 surfsense_backend/tests/unit/capabilities/web/crawl/test_schemas.py delete mode 100644 surfsense_backend/tests/unit/capabilities/web/discover/test_baidu_provider.py delete mode 100644 surfsense_backend/tests/unit/capabilities/web/discover/test_executor.py delete mode 100644 surfsense_backend/tests/unit/capabilities/web/discover/test_linkup_provider.py delete mode 100644 surfsense_backend/tests/unit/capabilities/web/discover/test_schemas.py delete mode 100644 surfsense_backend/tests/unit/capabilities/web/discover/test_searxng_provider.py delete mode 100644 surfsense_backend/tests/unit/capabilities/web/scrape/test_executor.py delete mode 100644 surfsense_backend/tests/unit/capabilities/web/scrape/test_schemas.py diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/index.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/index.py index 43923dbb9..fb9018fb4 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/index.py +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/index.py @@ -8,8 +8,7 @@ from langchain_core.tools import BaseTool from app.agents.chat.multi_agent_chat.shared.permissions import Ruleset from app.capabilities.core.access.agent import build_capability_tools -from app.capabilities.web.discover.definition import WEB_DISCOVER -from app.capabilities.web.scrape.definition import WEB_SCRAPE +from app.capabilities.web.crawl.definition import WEB_CRAWL from app.capabilities.youtube.comments.definition import YOUTUBE_COMMENTS from app.capabilities.youtube.scrape.definition import YOUTUBE_SCRAPE @@ -21,7 +20,7 @@ NAME = "scraping" RULESET = Ruleset(origin=NAME, rules=[]) -_CI_VERBS = [WEB_DISCOVER, WEB_SCRAPE, YOUTUBE_SCRAPE, YOUTUBE_COMMENTS] +_CI_VERBS = [WEB_CRAWL, YOUTUBE_SCRAPE, YOUTUBE_COMMENTS] def load_tools( diff --git a/surfsense_backend/app/capabilities/web/__init__.py b/surfsense_backend/app/capabilities/web/__init__.py index f3710d1cf..c0c4bef17 100644 --- a/surfsense_backend/app/capabilities/web/__init__.py +++ b/surfsense_backend/app/capabilities/web/__init__.py @@ -1,6 +1,5 @@ -"""``web.*`` namespace: the generic web scraping + discovery verbs.""" +"""``web.*`` namespace: the unified web content crawler verb.""" from __future__ import annotations -from app.capabilities.web.discover import definition as _discover # noqa: F401 -from app.capabilities.web.scrape import definition as _scrape # noqa: F401 +from app.capabilities.web.crawl import definition as _crawl # noqa: F401 diff --git a/surfsense_backend/app/capabilities/web/crawl/__init__.py b/surfsense_backend/app/capabilities/web/crawl/__init__.py new file mode 100644 index 000000000..8a13f0bee --- /dev/null +++ b/surfsense_backend/app/capabilities/web/crawl/__init__.py @@ -0,0 +1,3 @@ +"""``web.crawl`` verb: single-URL scrape or same-site spider → one row per page.""" + +from __future__ import annotations diff --git a/surfsense_backend/app/capabilities/web/crawl/definition.py b/surfsense_backend/app/capabilities/web/crawl/definition.py new file mode 100644 index 000000000..de3cef0ca --- /dev/null +++ b/surfsense_backend/app/capabilities/web/crawl/definition.py @@ -0,0 +1,25 @@ +"""``web.crawl`` capability registration.""" + +from __future__ import annotations + +from app.capabilities.core import BillingUnit, Capability, register_capability +from app.capabilities.web.crawl.executor import build_crawl_executor +from app.capabilities.web.crawl.schemas import CrawlInput, CrawlOutput + +WEB_CRAWL = Capability( + name="web.crawl", + description=( + "Scrape a single web page or crawl a whole website. Give it one or more " + "startUrls. Set maxCrawlDepth=0 to fetch just those URLs, or higher to " + "also follow the links on each page (depth 1 = the start pages plus the " + "pages they link to, and so on) — staying on the same site and stopping " + "at maxCrawlPages. Returns one item per fetched page with clean markdown " + "content, metadata (title, description), and crawl provenance." + ), + input_schema=CrawlInput, + output_schema=CrawlOutput, + executor=build_crawl_executor(), + billing_unit=BillingUnit.WEB_CRAWL, +) + +register_capability(WEB_CRAWL) diff --git a/surfsense_backend/app/capabilities/web/crawl/executor.py b/surfsense_backend/app/capabilities/web/crawl/executor.py new file mode 100644 index 000000000..45f5b3004 --- /dev/null +++ b/surfsense_backend/app/capabilities/web/crawl/executor.py @@ -0,0 +1,65 @@ +"""``web.crawl`` executor: seeds → site spider → one cleaned item per fetched page. + +Boundary owned elsewhere: the crawl frontier/fetch live in the proprietary engine +(``app.proprietary.web_crawler``). This executor only maps the engine's +``CrawlPage`` list onto the typed ``CrawlOutput`` (status labels, truncation, and +the captcha telemetry the billing seam reads). +""" + +from __future__ import annotations + +from app.capabilities.core import Executor +from app.capabilities.web.crawl.schemas import ( + CrawlInput, + CrawlItem, + CrawlMeta, + CrawlOutput, +) +from app.proprietary.web_crawler import ( + CrawlOutcomeStatus, + CrawlPage, + WebCrawlerConnector, + crawl_site, +) + +_STATUS_LABEL: dict[CrawlOutcomeStatus, str] = { + CrawlOutcomeStatus.SUCCESS: "success", + CrawlOutcomeStatus.EMPTY: "empty", + CrawlOutcomeStatus.FAILED: "failed", +} + + +def build_crawl_executor(engine: WebCrawlerConnector | None = None) -> Executor: + """Build the ``web.crawl`` executor, optionally over an injected engine (tests).""" + crawler = engine or WebCrawlerConnector() + + async def execute(payload: CrawlInput) -> CrawlOutput: + pages = await crawl_site( + crawler, + payload.startUrls, + max_crawl_depth=payload.maxCrawlDepth, + max_crawl_pages=payload.maxCrawlPages, + ) + return CrawlOutput( + items=[_to_item(page, payload.maxLength) for page in pages], + captcha_attempts=sum(page.captcha_attempts for page in pages), + captcha_solved=sum(1 for page in pages if page.captcha_solved), + ) + + return execute + + +def _to_item(page: CrawlPage, max_length: int) -> CrawlItem: + content = page.content[:max_length] if page.content is not None else None + return CrawlItem( + url=page.url, + status=_STATUS_LABEL[page.status], + crawl=CrawlMeta( + loadedUrl=page.loaded_url or page.url, + depth=page.depth, + referrerUrl=page.referrer, + ), + markdown=content, + metadata=page.metadata, + error=page.error, + ) diff --git a/surfsense_backend/app/capabilities/web/crawl/schemas.py b/surfsense_backend/app/capabilities/web/crawl/schemas.py new file mode 100644 index 000000000..38ab0e7f5 --- /dev/null +++ b/surfsense_backend/app/capabilities/web/crawl/schemas.py @@ -0,0 +1,115 @@ +# ruff: noqa: N815 - public field names intentionally mirror the Apify camelCase spec +"""``web.crawl`` I/O contracts. + +An Apify *Website Content Crawler*-style surface (see apify.com/apify/website- +content-crawler and docs.firecrawl.dev): one verb that either scrapes the given +URLs (``maxCrawlDepth == 0``) or spiders their site (``maxCrawlDepth > 0``), +bounded by ``maxCrawlPages`` and kept on the seed's site. + +Fields are trimmed to what the proprietary engine honors today. Apify knobs the +engine handles automatically (crawler type, proxy, dynamic-render waits) are +intentionally omitted, as are features we haven't built (URL globs, output +formats, click actions, PII handling). +""" + +from __future__ import annotations + +from typing import Literal + +from pydantic import BaseModel, Field + +MAX_START_URLS = 20 +"""Per-call cap on seed URLs: bounds a synchronous request's fan-out (05).""" + +MAX_CRAWL_DEPTH = 5 +"""Deepest link distance a spider will follow from a start URL.""" + +MAX_CRAWL_PAGES = 200 +"""Hard ceiling on pages fetched per call (protects the wallet and the run).""" + + +class CrawlInput(BaseModel): + startUrls: list[str] = Field( + min_length=1, + max_length=MAX_START_URLS, + description=( + "Seed URLs to crawl. With maxCrawlDepth=0 only these are fetched; " + "with a higher depth they are also the entry points for the spider." + ), + ) + maxCrawlDepth: int = Field( + default=0, + ge=0, + le=MAX_CRAWL_DEPTH, + description=( + "How many link-hops to follow from each start URL. 0 = scrape only " + "the start URLs (no spidering); 1 = also their linked pages; etc. " + "The spider stays on the start URL's site." + ), + ) + maxCrawlPages: int = Field( + default=10, + ge=1, + le=MAX_CRAWL_PAGES, + description=( + "Maximum number of pages to fetch in total (start URLs included). " + "The crawl stops once this many pages have been fetched." + ), + ) + maxLength: int = Field( + default=50_000, + ge=1, + description="Maximum characters of cleaned markdown kept per page (truncates beyond).", + ) + + @property + def estimated_units(self) -> int: + """Worst-case billable pages for the pre-flight gate (03c).""" + if self.maxCrawlDepth == 0: + return len(self.startUrls) + return self.maxCrawlPages + + +class CrawlMeta(BaseModel): + loadedUrl: str = Field(description="The URL actually fetched for this page.") + depth: int = Field( + description="Link distance from a start URL (0 for a start URL itself)." + ) + referrerUrl: str | None = Field( + default=None, + description="The page this URL was discovered on (null for start URLs).", + ) + + +class CrawlItem(BaseModel): + url: str = Field(description="The requested URL for this page.") + status: Literal["success", "empty", "failed"] = Field( + description="success = content returned; empty = fetched but no content; failed = could not fetch." + ) + crawl: CrawlMeta | None = Field( + default=None, description="Crawl provenance (loaded URL, depth, referrer)." + ) + markdown: str | None = Field( + default=None, description="Cleaned page content as markdown (null unless success)." + ) + metadata: dict[str, str] | None = Field( + default=None, description="Page metadata such as title and description." + ) + error: str | None = Field( + default=None, description="Failure reason when status is not success." + ) + + +class CrawlOutput(BaseModel): + items: list[CrawlItem] = Field( + default_factory=list, + description="One item per fetched page, in crawl (BFS) order.", + ) + # Billing-only telemetry; excluded from the wire shape (mirrors web.scrape). + captcha_attempts: int = Field(default=0, exclude=True) + captcha_solved: int = Field(default=0, exclude=True) + + @property + def billable_units(self) -> int: + """Successful pages are the metered unit (03c).""" + return sum(1 for item in self.items if item.status == "success") diff --git a/surfsense_backend/app/capabilities/web/discover/__init__.py b/surfsense_backend/app/capabilities/web/discover/__init__.py deleted file mode 100644 index 37b153535..000000000 --- a/surfsense_backend/app/capabilities/web/discover/__init__.py +++ /dev/null @@ -1,3 +0,0 @@ -"""``web.discover`` verb: a query → candidate URLs the agent can feed to ``web.scrape``.""" - -from __future__ import annotations diff --git a/surfsense_backend/app/capabilities/web/discover/definition.py b/surfsense_backend/app/capabilities/web/discover/definition.py deleted file mode 100644 index f672c6bbb..000000000 --- a/surfsense_backend/app/capabilities/web/discover/definition.py +++ /dev/null @@ -1,23 +0,0 @@ -"""``web.discover`` capability registration (free — see 04-capabilities open item).""" - -from __future__ import annotations - -from app.capabilities.core import Capability, register_capability -from app.capabilities.web.discover.executor import build_discover_executor -from app.capabilities.web.discover.schemas import DiscoverInput, DiscoverOutput - -WEB_DISCOVER = Capability( - name="web.discover", - description=( - "Search the web for a query and return ranked results. Use it to find " - "pages when you don't already have exact URLs. Returns a list of hits " - "(url, title, snippet, provider); pass the chosen url(s) to web.scrape " - "to read their full content." - ), - input_schema=DiscoverInput, - output_schema=DiscoverOutput, - executor=build_discover_executor(), - billing_unit=None, -) - -register_capability(WEB_DISCOVER) diff --git a/surfsense_backend/app/capabilities/web/discover/executor.py b/surfsense_backend/app/capabilities/web/discover/executor.py deleted file mode 100644 index 3660aedcd..000000000 --- a/surfsense_backend/app/capabilities/web/discover/executor.py +++ /dev/null @@ -1,42 +0,0 @@ -"""``web.discover`` executor: route a query to the first configured search provider.""" - -from __future__ import annotations - -from collections.abc import Sequence - -from app.capabilities.core import Executor -from app.capabilities.web.discover.providers.base import DiscoverProvider -from app.capabilities.web.discover.schemas import DiscoverInput, DiscoverOutput - - -class NoDiscoverProviderError(RuntimeError): - """Raised when no search provider is configured (no platform key/host set).""" - - -def build_discover_executor( - providers: Sequence[DiscoverProvider] | None = None, -) -> Executor: - """Bind the executor to a provider set (defaults to the real env-keyed providers).""" - registry = list(providers) if providers is not None else _default_providers() - - async def execute(payload: DiscoverInput) -> DiscoverOutput: - provider = next((p for p in registry if p.is_available()), None) - if provider is None: - raise NoDiscoverProviderError( - "web.discover has no configured search provider " - "(set a SearXNG host or a Linkup/Baidu key)." - ) - hits = await provider.search(payload.query, payload.top_k) - # Enforce the verb's documented cap here, once, for every provider: - # some backends (e.g. SearXNG) treat `top_k` as a hint and over-return. - return DiscoverOutput(hits=hits[: payload.top_k]) - - return execute - - -def _default_providers() -> list[DiscoverProvider]: - from app.capabilities.web.discover.providers.baidu import BaiduProvider - from app.capabilities.web.discover.providers.linkup import LinkupProvider - from app.capabilities.web.discover.providers.searxng import SearxngProvider - - return [SearxngProvider(), LinkupProvider(), BaiduProvider()] diff --git a/surfsense_backend/app/capabilities/web/discover/providers/__init__.py b/surfsense_backend/app/capabilities/web/discover/providers/__init__.py deleted file mode 100644 index 68ca2b374..000000000 --- a/surfsense_backend/app/capabilities/web/discover/providers/__init__.py +++ /dev/null @@ -1,3 +0,0 @@ -"""``web.discover`` search providers (env-keyed: SearXNG, Linkup, Baidu).""" - -from __future__ import annotations diff --git a/surfsense_backend/app/capabilities/web/discover/providers/baidu.py b/surfsense_backend/app/capabilities/web/discover/providers/baidu.py deleted file mode 100644 index fc632a9ea..000000000 --- a/surfsense_backend/app/capabilities/web/discover/providers/baidu.py +++ /dev/null @@ -1,53 +0,0 @@ -"""Baidu AI Search discover provider (env-keyed via ``BAIDU_API_KEY``).""" - -from __future__ import annotations - -import httpx - -from app.capabilities.web.discover.schemas import DiscoverHit -from app.config import config - -_ENDPOINT = "https://qianfan.baidubce.com/v2/ai_search/chat/completions" -_SNIPPET_MAX = 300 - - -class BaiduProvider: - name = "baidu" - - def is_available(self) -> bool: - return bool(config.BAIDU_API_KEY) - - async def search(self, query: str, top_k: int) -> list[DiscoverHit]: - max_per_type = min(top_k, 20) - payload = { - "messages": [{"role": "user", "content": query}], - "model": config.BAIDU_MODEL, - "search_source": config.BAIDU_SEARCH_SOURCE, - "resource_type_filter": [{"type": "web", "top_k": max_per_type}], - "stream": False, - } - headers = { - "X-Appbuilder-Authorization": f"Bearer {config.BAIDU_API_KEY}", - "Content-Type": "application/json", - } - async with httpx.AsyncClient(timeout=90.0) as client: - response = await client.post(_ENDPOINT, headers=headers, json=payload) - response.raise_for_status() - - hits: list[DiscoverHit] = [] - for reference in response.json().get("references", []): - url = reference.get("url", "") - if not url: - continue - content = reference.get("content", "") or "" - hits.append( - DiscoverHit( - url=url, - title=reference.get("title") or url, - snippet=content[:_SNIPPET_MAX] or None, - provider=self.name, - ) - ) - if len(hits) >= top_k: - break - return hits diff --git a/surfsense_backend/app/capabilities/web/discover/providers/base.py b/surfsense_backend/app/capabilities/web/discover/providers/base.py deleted file mode 100644 index c245072eb..000000000 --- a/surfsense_backend/app/capabilities/web/discover/providers/base.py +++ /dev/null @@ -1,22 +0,0 @@ -"""The seam every ``web.discover`` provider implements.""" - -from __future__ import annotations - -from typing import Protocol, runtime_checkable - -from app.capabilities.web.discover.schemas import DiscoverHit - - -@runtime_checkable -class DiscoverProvider(Protocol): - """An env-keyed search backend that suggests candidate URLs for a query.""" - - name: str - - def is_available(self) -> bool: - """True when this provider's platform key/host is configured.""" - ... - - async def search(self, query: str, top_k: int) -> list[DiscoverHit]: - """Return up to ``top_k`` candidate hits for ``query``.""" - ... diff --git a/surfsense_backend/app/capabilities/web/discover/providers/linkup.py b/surfsense_backend/app/capabilities/web/discover/providers/linkup.py deleted file mode 100644 index 77393de08..000000000 --- a/surfsense_backend/app/capabilities/web/discover/providers/linkup.py +++ /dev/null @@ -1,40 +0,0 @@ -"""Linkup discover provider (env-keyed via ``LINKUP_API_KEY``).""" - -from __future__ import annotations - -import asyncio - -from linkup import LinkupClient - -from app.capabilities.web.discover.schemas import DiscoverHit -from app.config import config - - -class LinkupProvider: - name = "linkup" - - def is_available(self) -> bool: - return bool(config.LINKUP_API_KEY) - - async def search(self, query: str, top_k: int) -> list[DiscoverHit]: - client = LinkupClient(api_key=config.LINKUP_API_KEY) - response = await asyncio.to_thread( - client.search, query=query, depth="standard", output_type="searchResults" - ) - hits: list[DiscoverHit] = [] - for result in getattr(response, "results", None) or []: - url = getattr(result, "url", "") or "" - if not url: - continue - content = getattr(result, "content", "") or "" - hits.append( - DiscoverHit( - url=url, - title=getattr(result, "name", "") or url, - snippet=content or None, - provider=self.name, - ) - ) - if len(hits) >= top_k: - break - return hits diff --git a/surfsense_backend/app/capabilities/web/discover/providers/searxng.py b/surfsense_backend/app/capabilities/web/discover/providers/searxng.py deleted file mode 100644 index 92489862a..000000000 --- a/surfsense_backend/app/capabilities/web/discover/providers/searxng.py +++ /dev/null @@ -1,28 +0,0 @@ -"""SearXNG discover provider, wrapping the platform-env SearXNG search service.""" - -from __future__ import annotations - -from app.capabilities.web.discover.schemas import DiscoverHit -from app.services import web_search_service - - -class SearxngProvider: - """Env-keyed via ``SEARXNG_DEFAULT_HOST`` (the platform SearXNG instance).""" - - name = "searxng" - - def is_available(self) -> bool: - return web_search_service.is_available() - - async def search(self, query: str, top_k: int) -> list[DiscoverHit]: - result_object, _documents = await web_search_service.search(query, top_k) - return [ - DiscoverHit( - url=source["url"], - title=source.get("title") or source["url"], - snippet=source.get("description") or None, - provider=self.name, - ) - for source in result_object.get("sources", []) - if source.get("url") - ] diff --git a/surfsense_backend/app/capabilities/web/discover/schemas.py b/surfsense_backend/app/capabilities/web/discover/schemas.py deleted file mode 100644 index 229454e7e..000000000 --- a/surfsense_backend/app/capabilities/web/discover/schemas.py +++ /dev/null @@ -1,29 +0,0 @@ -"""``web.discover`` I/O contracts.""" - -from __future__ import annotations - -from pydantic import BaseModel, Field - - -class DiscoverInput(BaseModel): - query: str = Field( - description="What to search the web for, phrased in natural language." - ) - top_k: int = Field( - default=10, ge=1, le=50, description="Maximum number of results to return (1-50)." - ) - - -class DiscoverHit(BaseModel): - url: str = Field( - description="The result's page URL; pass it to web.scrape to read it." - ) - title: str = Field(description="The result's page title.") - snippet: str | None = Field( - default=None, description="A short extract summarizing the page." - ) - provider: str = Field(description="Which search engine returned this hit.") - - -class DiscoverOutput(BaseModel): - hits: list[DiscoverHit] = Field(description="Ranked search results, best first.") diff --git a/surfsense_backend/app/capabilities/web/scrape/__init__.py b/surfsense_backend/app/capabilities/web/scrape/__init__.py deleted file mode 100644 index 0a21b156e..000000000 --- a/surfsense_backend/app/capabilities/web/scrape/__init__.py +++ /dev/null @@ -1,3 +0,0 @@ -"""``web.scrape`` verb: a URL array → one cleaned row per URL.""" - -from __future__ import annotations diff --git a/surfsense_backend/app/capabilities/web/scrape/definition.py b/surfsense_backend/app/capabilities/web/scrape/definition.py deleted file mode 100644 index f6ccafe02..000000000 --- a/surfsense_backend/app/capabilities/web/scrape/definition.py +++ /dev/null @@ -1,23 +0,0 @@ -"""``web.scrape`` capability registration.""" - -from __future__ import annotations - -from app.capabilities.core import BillingUnit, Capability, register_capability -from app.capabilities.web.scrape.executor import build_scrape_executor -from app.capabilities.web.scrape.schemas import ScrapeInput, ScrapeOutput - -WEB_SCRAPE = Capability( - name="web.scrape", - description=( - "Fetch one or more web pages and return their clean, readable content. " - "Give it the exact URLs to read (use web.discover first if you don't have " - "them). Returns one row per URL with a status (success/empty/failed), the " - "page content, and metadata such as title and description." - ), - input_schema=ScrapeInput, - output_schema=ScrapeOutput, - executor=build_scrape_executor(), - billing_unit=BillingUnit.WEB_CRAWL, -) - -register_capability(WEB_SCRAPE) diff --git a/surfsense_backend/app/capabilities/web/scrape/executor.py b/surfsense_backend/app/capabilities/web/scrape/executor.py deleted file mode 100644 index 8651ff738..000000000 --- a/surfsense_backend/app/capabilities/web/scrape/executor.py +++ /dev/null @@ -1,45 +0,0 @@ -"""``web.scrape`` executor: fetch each URL in the array → cleaned rows.""" - -from __future__ import annotations - -from app.capabilities.core import Executor -from app.capabilities.web.scrape.schemas import ScrapeInput, ScrapeOutput, ScrapeRow -from app.proprietary.web_crawler import ( - CrawlOutcome, - CrawlOutcomeStatus, - WebCrawlerConnector, -) - - -def build_scrape_executor(engine: WebCrawlerConnector | None = None) -> Executor: - """Bind the executor to a fetch engine (defaults to the proprietary crawler).""" - engine = engine or WebCrawlerConnector() - - async def execute(payload: ScrapeInput) -> ScrapeOutput: - outcomes = [await engine.crawl_url(url) for url in payload.urls] - rows = [ - _to_row(url, outcome, payload.max_length) - for url, outcome in zip(payload.urls, outcomes, strict=True) - ] - return ScrapeOutput( - rows=rows, - captcha_attempts=sum(o.captcha_attempts for o in outcomes), - captcha_solved=sum(1 for o in outcomes if o.captcha_solved), - ) - - return execute - - -def _to_row(url: str, outcome: CrawlOutcome, max_length: int) -> ScrapeRow: - if outcome.status is CrawlOutcomeStatus.SUCCESS and outcome.result: - content = outcome.result.get("content") - if content is not None: - content = content[:max_length] - return ScrapeRow( - url=url, - status="success", - content=content, - metadata=outcome.result.get("metadata"), - ) - status = "empty" if outcome.status is CrawlOutcomeStatus.EMPTY else "failed" - return ScrapeRow(url=url, status=status, error=outcome.error) diff --git a/surfsense_backend/app/capabilities/web/scrape/schemas.py b/surfsense_backend/app/capabilities/web/scrape/schemas.py deleted file mode 100644 index 04faaa516..000000000 --- a/surfsense_backend/app/capabilities/web/scrape/schemas.py +++ /dev/null @@ -1,67 +0,0 @@ -"""``web.scrape`` I/O contracts.""" - -from __future__ import annotations - -from typing import Literal - -from pydantic import BaseModel, Field - -MAX_SCRAPE_URLS = 20 -"""Per-call batch cap: bounds a synchronous request's crawl fan-out (05).""" - - -class ScrapeInput(BaseModel): - urls: list[str] = Field( - min_length=1, - max_length=MAX_SCRAPE_URLS, - description=( - "Full page URLs to fetch and read (1-20), each starting with " - "http:// or https://. Pass the exact URLs you want the content of." - ), - ) - max_length: int = Field( - default=50_000, - description=( - "Maximum characters of cleaned content returned per page; " - "content longer than this is truncated." - ), - ) - - @property - def estimated_units(self) -> int: - """Worst-case billable crawls for pre-flight: one per requested URL.""" - return len(self.urls) - - -class ScrapeRow(BaseModel): - url: str = Field(description="The requested URL this result is for.") - status: Literal["success", "empty", "failed"] = Field( - description=( - "'success' = content returned; 'empty' = page reached but no " - "readable content; 'failed' = could not be fetched (see error)." - ) - ) - content: str | None = Field( - default=None, description="Cleaned, readable page text (present on success)." - ) - metadata: dict[str, str] | None = Field( - default=None, description="Page metadata such as title and description." - ) - error: str | None = Field( - default=None, description="Why the fetch failed (present on 'failed')." - ) - - -class ScrapeOutput(BaseModel): - rows: list[ScrapeRow] = Field( - description="One result per requested URL, in the same order." - ) - # Billing-only telemetry (Phase 3d), excluded from the client-facing result: - # captcha solves are metered per attempt, independent of crawl success. - captcha_attempts: int = Field(default=0, exclude=True) - captcha_solved: int = Field(default=0, exclude=True) - - @property - def billable_units(self) -> int: - """One billable unit per successful scrape.""" - return sum(1 for row in self.rows if row.status == "success") diff --git a/surfsense_backend/tests/unit/capabilities/access/test_rest_router.py b/surfsense_backend/tests/unit/capabilities/access/test_rest_router.py index b874b8bba..e3bc387e9 100644 --- a/surfsense_backend/tests/unit/capabilities/access/test_rest_router.py +++ b/surfsense_backend/tests/unit/capabilities/access/test_rest_router.py @@ -95,8 +95,7 @@ def test_registered_verbs_appear_on_rest(): router = rest.build_capabilities_router() paths = {route.path for route in router.routes} - assert "/workspaces/{workspace_id}/capabilities/web.scrape" in paths - assert "/workspaces/{workspace_id}/capabilities/web.discover" in paths + assert "/workspaces/{workspace_id}/capabilities/web.crawl" in paths @pytest.mark.asyncio diff --git a/surfsense_backend/tests/unit/capabilities/test_billing.py b/surfsense_backend/tests/unit/capabilities/test_billing.py index c0a74878b..ef5086587 100644 --- a/surfsense_backend/tests/unit/capabilities/test_billing.py +++ b/surfsense_backend/tests/unit/capabilities/test_billing.py @@ -14,7 +14,7 @@ import pytest import app.capabilities.core.billing as billing from app.capabilities.core.billing import charge_capability, gate_capability from app.capabilities.core.types import BillingUnit, CapabilityContext -from app.capabilities.web.scrape.schemas import ScrapeInput, ScrapeOutput, ScrapeRow +from app.capabilities.web.crawl.schemas import CrawlInput, CrawlItem, CrawlOutput from app.config import config from app.services.web_crawl_credit_service import InsufficientCreditsError @@ -46,10 +46,10 @@ def _make_session(owner_id, balance_micros): return session, fake_user -def _output(*statuses: str) -> ScrapeOutput: - return ScrapeOutput( - rows=[ - ScrapeRow(url=f"https://{i}.com", status=status) +def _output(*statuses: str) -> CrawlOutput: + return CrawlOutput( + items=[ + CrawlItem(url=f"https://{i}.com", status=status) for i, status in enumerate(statuses) ] ) @@ -92,7 +92,7 @@ async def test_charges_workspace_owner_per_successful_crawl(monkeypatch, record_ assert kwargs["cost_micros"] == 2000 -def _output_with_captcha(*statuses: str, attempts: int, solved: int) -> ScrapeOutput: +def _output_with_captcha(*statuses: str, attempts: int, solved: int) -> CrawlOutput: out = _output(*statuses) out.captcha_attempts = attempts out.captcha_solved = solved @@ -196,7 +196,7 @@ async def test_gate_blocks_when_worst_case_exceeds_balance(monkeypatch): with pytest.raises(InsufficientCreditsError): await gate_capability( - ScrapeInput(urls=["https://a.com", "https://b.com"]), + CrawlInput(startUrls=["https://a.com", "https://b.com"]), BillingUnit.WEB_CRAWL, _ctx(session), ) @@ -208,7 +208,7 @@ async def test_gate_passes_when_balance_covers_worst_case(monkeypatch): session = _gate_session(_OWNER, balance_micros=100_000) await gate_capability( - ScrapeInput(urls=["https://a.com", "https://b.com"]), + CrawlInput(startUrls=["https://a.com", "https://b.com"]), BillingUnit.WEB_CRAWL, _ctx(session), ) @@ -219,7 +219,7 @@ async def test_gate_is_noop_when_disabled(monkeypatch): session = _gate_session(_OWNER, balance_micros=0) await gate_capability( - ScrapeInput(urls=["https://a.com"]), BillingUnit.WEB_CRAWL, _ctx(session) + CrawlInput(startUrls=["https://a.com"]), BillingUnit.WEB_CRAWL, _ctx(session) ) @@ -233,7 +233,7 @@ async def test_gate_reserves_worst_case_captcha_when_solving_enabled(monkeypatch with pytest.raises(InsufficientCreditsError): await gate_capability( - ScrapeInput(urls=["https://a.com"]), BillingUnit.WEB_CRAWL, _ctx(session) + CrawlInput(startUrls=["https://a.com"]), BillingUnit.WEB_CRAWL, _ctx(session) ) @@ -245,7 +245,7 @@ async def test_gate_does_not_reserve_captcha_when_solving_disabled(monkeypatch): # Solving off → attempts can never happen → nothing to reserve → passes. await gate_capability( - ScrapeInput(urls=["https://a.com"]), BillingUnit.WEB_CRAWL, _ctx(session) + CrawlInput(startUrls=["https://a.com"]), BillingUnit.WEB_CRAWL, _ctx(session) ) @@ -253,6 +253,6 @@ async def test_gate_is_noop_for_free_verb(monkeypatch): monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", True) session = _gate_session(_OWNER, balance_micros=0) - await gate_capability(ScrapeInput(urls=["https://a.com"]), None, _ctx(session)) + await gate_capability(CrawlInput(startUrls=["https://a.com"]), None, _ctx(session)) session.execute.assert_not_called() diff --git a/surfsense_backend/tests/unit/capabilities/test_registry.py b/surfsense_backend/tests/unit/capabilities/test_registry.py index c603c02eb..ff8f6d931 100644 --- a/surfsense_backend/tests/unit/capabilities/test_registry.py +++ b/surfsense_backend/tests/unit/capabilities/test_registry.py @@ -9,25 +9,15 @@ from app.capabilities import ( ) from app.capabilities.core.store import get_capability from app.capabilities.core.types import BillingUnit -from app.capabilities.web.discover.schemas import DiscoverInput, DiscoverOutput -from app.capabilities.web.scrape.schemas import ScrapeInput, ScrapeOutput +from app.capabilities.web.crawl.schemas import CrawlInput, CrawlOutput pytestmark = pytest.mark.unit -def test_web_scrape_is_registered_with_its_schemas_and_billing_unit(): - cap = get_capability("web.scrape") +def test_web_crawl_is_registered_with_its_schemas_and_billing_unit(): + cap = get_capability("web.crawl") - assert cap.name == "web.scrape" - assert cap.input_schema is ScrapeInput - assert cap.output_schema is ScrapeOutput + assert cap.name == "web.crawl" + assert cap.input_schema is CrawlInput + assert cap.output_schema is CrawlOutput assert cap.billing_unit is BillingUnit.WEB_CRAWL - - -def test_web_discover_is_registered_and_free(): - cap = get_capability("web.discover") - - assert cap.name == "web.discover" - assert cap.input_schema is DiscoverInput - assert cap.output_schema is DiscoverOutput - assert cap.billing_unit is None diff --git a/surfsense_backend/tests/unit/capabilities/web/crawl/test_executor.py b/surfsense_backend/tests/unit/capabilities/web/crawl/test_executor.py new file mode 100644 index 000000000..2d550abbc --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/web/crawl/test_executor.py @@ -0,0 +1,113 @@ +"""``web.crawl`` executor behavior: CrawlPage list → typed CrawlOutput items. + +Boundary mocked: the crawler engine (fake ``crawl_url`` + link graph). NOT +mocked: the executor's page→item mapping, truncation, and captcha rollup. +""" + +from __future__ import annotations + +import pytest + +from app.capabilities.web.crawl.executor import build_crawl_executor +from app.capabilities.web.crawl.schemas import CrawlInput, CrawlOutput +from app.proprietary.web_crawler import CrawlOutcome, CrawlOutcomeStatus + +pytestmark = pytest.mark.unit + +_SUCCESS = CrawlOutcomeStatus.SUCCESS + + +class _FakeEngine: + def __init__(self, graph: dict[str, tuple[CrawlOutcomeStatus, list[str]]]): + self._graph = graph + self.calls: list[str] = [] + + async def crawl_url(self, url: str) -> CrawlOutcome: + self.calls.append(url) + status, links = self._graph[url] + if status is _SUCCESS: + return CrawlOutcome( + status=_SUCCESS, + result={ + "content": f"C:{url}", + "metadata": {"title": url}, + "links": links, + }, + ) + return CrawlOutcome(status=status, error="boom") + + +async def test_single_url_depth_zero_returns_one_item() -> None: + engine = _FakeEngine({"https://e.com/": (_SUCCESS, ["https://e.com/a"])}) + execute = build_crawl_executor(engine=engine) + + out = await execute(CrawlInput(startUrls=["https://e.com/"])) + + assert isinstance(out, CrawlOutput) + assert len(out.items) == 1 + item = out.items[0] + assert item.url == "https://e.com/" + assert item.status == "success" + assert item.markdown == "C:https://e.com/" + assert item.metadata == {"title": "https://e.com/"} + assert item.crawl is not None + assert item.crawl.depth == 0 + assert item.crawl.referrerUrl is None + + +async def test_spider_collects_multiple_pages_with_provenance() -> None: + engine = _FakeEngine( + { + "https://e.com/": (_SUCCESS, ["https://e.com/a"]), + "https://e.com/a": (_SUCCESS, []), + } + ) + execute = build_crawl_executor(engine=engine) + + out = await execute( + CrawlInput(startUrls=["https://e.com/"], maxCrawlDepth=1, maxCrawlPages=10) + ) + + by_url = {item.url: item for item in out.items} + assert set(by_url) == {"https://e.com/", "https://e.com/a"} + assert by_url["https://e.com/a"].crawl.referrerUrl == "https://e.com/" + + +async def test_content_is_truncated_to_max_length() -> None: + engine = _FakeEngine({"https://e.com/": (_SUCCESS, [])}) + execute = build_crawl_executor(engine=engine) + + out = await execute(CrawlInput(startUrls=["https://e.com/"], maxLength=3)) + + assert out.items[0].markdown == "C:h" + + +async def test_failed_page_has_no_markdown_but_keeps_error() -> None: + engine = _FakeEngine({"https://e.com/": (CrawlOutcomeStatus.FAILED, [])}) + execute = build_crawl_executor(engine=engine) + + out = await execute(CrawlInput(startUrls=["https://e.com/"])) + + item = out.items[0] + assert item.status == "failed" + assert item.markdown is None + assert item.error == "boom" + + +async def test_captcha_telemetry_is_rolled_up_for_billing() -> None: + class _CaptchaEngine: + async def crawl_url(self, url: str) -> CrawlOutcome: + return CrawlOutcome( + status=_SUCCESS, + result={"content": "ok", "metadata": {}, "links": []}, + captcha_attempts=2, + captcha_solved=True, + ) + + execute = build_crawl_executor(engine=_CaptchaEngine()) + + out = await execute(CrawlInput(startUrls=["https://e.com/"])) + + assert out.captcha_attempts == 2 + assert out.captcha_solved == 1 + assert out.billable_units == 1 diff --git a/surfsense_backend/tests/unit/capabilities/web/crawl/test_schemas.py b/surfsense_backend/tests/unit/capabilities/web/crawl/test_schemas.py new file mode 100644 index 000000000..9e7503851 --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/web/crawl/test_schemas.py @@ -0,0 +1,67 @@ +"""``web.crawl`` I/O contract: camelCase surface, bounds, and billing counters.""" + +from __future__ import annotations + +import pytest +from pydantic import ValidationError + +from app.capabilities.web.crawl.schemas import ( + CrawlInput, + CrawlItem, + CrawlMeta, + CrawlOutput, +) + +pytestmark = pytest.mark.unit + + +def test_requires_at_least_one_start_url() -> None: + with pytest.raises(ValidationError): + CrawlInput(startUrls=[]) + + +def test_camelcase_fields_and_defaults() -> None: + model = CrawlInput(startUrls=["https://e.com"]) + assert model.startUrls == ["https://e.com"] + assert model.maxCrawlDepth == 0 + assert model.maxCrawlPages == 10 + assert model.maxLength == 50_000 + + +def test_depth_and_page_bounds_are_enforced() -> None: + with pytest.raises(ValidationError): + CrawlInput(startUrls=["https://e.com"], maxCrawlDepth=-1) + with pytest.raises(ValidationError): + CrawlInput(startUrls=["https://e.com"], maxCrawlDepth=99) + with pytest.raises(ValidationError): + CrawlInput(startUrls=["https://e.com"], maxCrawlPages=0) + + +def test_estimated_units_for_single_url_is_seed_count() -> None: + model = CrawlInput(startUrls=["https://a.com", "https://b.com"], maxCrawlDepth=0) + assert model.estimated_units == 2 + + +def test_estimated_units_for_spider_is_max_pages() -> None: + model = CrawlInput( + startUrls=["https://a.com"], maxCrawlDepth=2, maxCrawlPages=25 + ) + assert model.estimated_units == 25 + + +def test_billable_units_counts_only_successes() -> None: + out = CrawlOutput( + items=[ + CrawlItem(url="a", status="success", crawl=CrawlMeta(loadedUrl="a", depth=0)), + CrawlItem(url="b", status="empty", crawl=CrawlMeta(loadedUrl="b", depth=1)), + CrawlItem(url="c", status="failed", crawl=CrawlMeta(loadedUrl="c", depth=1)), + ] + ) + assert out.billable_units == 1 + + +def test_captcha_counters_are_excluded_from_the_wire_shape() -> None: + out = CrawlOutput(items=[], captcha_attempts=3, captcha_solved=1) + dumped = out.model_dump() + assert "captcha_attempts" not in dumped + assert "captcha_solved" not in dumped diff --git a/surfsense_backend/tests/unit/capabilities/web/discover/test_baidu_provider.py b/surfsense_backend/tests/unit/capabilities/web/discover/test_baidu_provider.py deleted file mode 100644 index dc3035da7..000000000 --- a/surfsense_backend/tests/unit/capabilities/web/discover/test_baidu_provider.py +++ /dev/null @@ -1,91 +0,0 @@ -"""BaiduProvider maps the Baidu AI Search references to DiscoverHits, env-keyed. - -Boundary mocked: httpx.AsyncClient + config key. NOT mocked: reference→hit mapping. -""" - -from __future__ import annotations - -import pytest - -import app.capabilities.web.discover.providers.baidu as baidu_module -from app.capabilities.web.discover.providers.baidu import BaiduProvider -from app.config import config - -pytestmark = pytest.mark.unit - - -class _FakeResponse: - def __init__(self, payload): - self._payload = payload - - def raise_for_status(self): - return None - - def json(self): - return self._payload - - -class _FakeAsyncClient: - """Minimal async-context httpx stand-in returning a canned payload.""" - - payload: dict = {} - - def __init__(self, *args, **kwargs): - pass - - async def __aenter__(self): - return self - - async def __aexit__(self, *exc): - return False - - async def post(self, url, headers=None, json=None): - return _FakeResponse(type(self).payload) - - -def _install(monkeypatch, payload): - monkeypatch.setattr(config, "BAIDU_API_KEY", "k") - _FakeAsyncClient.payload = payload - monkeypatch.setattr(baidu_module.httpx, "AsyncClient", _FakeAsyncClient) - - -def test_is_available_reflects_the_env_key(monkeypatch): - monkeypatch.setattr(config, "BAIDU_API_KEY", "k") - assert BaiduProvider().is_available() is True - monkeypatch.setattr(config, "BAIDU_API_KEY", None) - assert BaiduProvider().is_available() is False - - -async def test_maps_references_to_hits(monkeypatch): - _install( - monkeypatch, - { - "references": [ - {"title": "Acme", "url": "https://acme.cn", "content": "hello"}, - {"title": "No URL", "url": "", "content": "skip"}, - ] - }, - ) - - hits = await BaiduProvider().search("acme", top_k=10) - - assert [h.url for h in hits] == ["https://acme.cn"] - assert hits[0].title == "Acme" - assert hits[0].snippet == "hello" - assert hits[0].provider == "baidu" - - -async def test_respects_top_k(monkeypatch): - _install( - monkeypatch, - { - "references": [ - {"title": f"n{i}", "url": f"https://{i}.cn", "content": "c"} - for i in range(5) - ] - }, - ) - - hits = await BaiduProvider().search("q", top_k=2) - - assert len(hits) == 2 diff --git a/surfsense_backend/tests/unit/capabilities/web/discover/test_executor.py b/surfsense_backend/tests/unit/capabilities/web/discover/test_executor.py deleted file mode 100644 index bfc93a31b..000000000 --- a/surfsense_backend/tests/unit/capabilities/web/discover/test_executor.py +++ /dev/null @@ -1,99 +0,0 @@ -"""`web.discover` executor: pick the first configured provider; self-disable when none. - -Boundary mocked: the providers (injected fakes). NOT mocked: the executor's -provider-selection and self-disable behavior. -""" - -from __future__ import annotations - -import pytest - -from app.capabilities.web.discover.executor import ( - NoDiscoverProviderError, - build_discover_executor, -) -from app.capabilities.web.discover.schemas import ( - DiscoverHit, - DiscoverInput, - DiscoverOutput, -) - -pytestmark = pytest.mark.unit - - -class _FakeProvider: - def __init__( - self, name: str, available: bool, hits: list[DiscoverHit] | None = None - ): - self.name = name - self._available = available - self._hits = hits or [] - self.calls: list[tuple[str, int]] = [] - - def is_available(self) -> bool: - return self._available - - async def search(self, query: str, top_k: int) -> list[DiscoverHit]: - self.calls.append((query, top_k)) - return self._hits - - -def _hit(url: str, provider: str) -> DiscoverHit: - return DiscoverHit(url=url, title=url, snippet="s", provider=provider) - - -async def test_uses_the_first_available_provider(): - first = _FakeProvider( - "searxng", available=True, hits=[_hit("https://a.com", "searxng")] - ) - second = _FakeProvider( - "linkup", available=True, hits=[_hit("https://b.com", "linkup")] - ) - execute = build_discover_executor(providers=[first, second]) - - out = await execute(DiscoverInput(query="acme pricing", top_k=5)) - - assert isinstance(out, DiscoverOutput) - assert [h.url for h in out.hits] == ["https://a.com"] - assert first.calls == [("acme pricing", 5)] - assert second.calls == [] # first available short-circuits - - -async def test_skips_unavailable_providers(): - off = _FakeProvider("searxng", available=False) - on = _FakeProvider("linkup", available=True, hits=[_hit("https://b.com", "linkup")]) - execute = build_discover_executor(providers=[off, on]) - - out = await execute(DiscoverInput(query="q")) - - assert [h.provider for h in out.hits] == ["linkup"] - assert off.calls == [] - - -async def test_self_disables_when_no_provider_is_configured(): - execute = build_discover_executor( - providers=[_FakeProvider("searxng", available=False)] - ) - - with pytest.raises(NoDiscoverProviderError): - await execute(DiscoverInput(query="q")) - - -async def test_caps_hits_to_top_k_when_provider_over_returns(): - # SearXNG treats `limit` as a hint and can return more rows than asked; the - # verb must honor its own documented `top_k` cap regardless of the provider. - over = _FakeProvider( - "searxng", - available=True, - hits=[_hit(f"https://{i}.com", "searxng") for i in range(5)], - ) - execute = build_discover_executor(providers=[over]) - - out = await execute(DiscoverInput(query="q", top_k=3)) - - assert len(out.hits) == 3 - assert [h.url for h in out.hits] == [ - "https://0.com", - "https://1.com", - "https://2.com", - ] diff --git a/surfsense_backend/tests/unit/capabilities/web/discover/test_linkup_provider.py b/surfsense_backend/tests/unit/capabilities/web/discover/test_linkup_provider.py deleted file mode 100644 index d846e0d94..000000000 --- a/surfsense_backend/tests/unit/capabilities/web/discover/test_linkup_provider.py +++ /dev/null @@ -1,75 +0,0 @@ -"""LinkupProvider maps the Linkup SDK results to DiscoverHits, env-keyed. - -Boundary mocked: the LinkupClient SDK + config key. NOT mocked: result→hit mapping. -""" - -from __future__ import annotations - -import pytest - -import app.capabilities.web.discover.providers.linkup as linkup_module -from app.capabilities.web.discover.providers.linkup import LinkupProvider -from app.config import config - -pytestmark = pytest.mark.unit - - -class _Result: - def __init__(self, name, url, content): - self.name = name - self.url = url - self.content = content - self.type = "text" - - -class _Response: - def __init__(self, results): - self.results = results - - -class _FakeClient: - def __init__(self, api_key): - self.api_key = api_key - - def search(self, *, query, depth, output_type): - return _Response( - [ - _Result("Acme", "https://acme.com", "acme home"), - _Result("No URL", "", "skip"), - ] - ) - - -def test_is_available_reflects_the_env_key(monkeypatch): - monkeypatch.setattr(config, "LINKUP_API_KEY", "k") - assert LinkupProvider().is_available() is True - monkeypatch.setattr(config, "LINKUP_API_KEY", None) - assert LinkupProvider().is_available() is False - - -async def test_maps_results_to_hits(monkeypatch): - monkeypatch.setattr(config, "LINKUP_API_KEY", "k") - monkeypatch.setattr(linkup_module, "LinkupClient", _FakeClient) - - hits = await LinkupProvider().search("acme", top_k=10) - - assert [h.url for h in hits] == ["https://acme.com"] - assert hits[0].title == "Acme" - assert hits[0].snippet == "acme home" - assert hits[0].provider == "linkup" - - -async def test_respects_top_k(monkeypatch): - monkeypatch.setattr(config, "LINKUP_API_KEY", "k") - - class _ManyClient(_FakeClient): - def search(self, *, query, depth, output_type): - return _Response( - [_Result(f"n{i}", f"https://{i}.com", "c") for i in range(5)] - ) - - monkeypatch.setattr(linkup_module, "LinkupClient", _ManyClient) - - hits = await LinkupProvider().search("q", top_k=2) - - assert len(hits) == 2 diff --git a/surfsense_backend/tests/unit/capabilities/web/discover/test_schemas.py b/surfsense_backend/tests/unit/capabilities/web/discover/test_schemas.py deleted file mode 100644 index f098014d8..000000000 --- a/surfsense_backend/tests/unit/capabilities/web/discover/test_schemas.py +++ /dev/null @@ -1,22 +0,0 @@ -"""``web.discover`` I/O contract bounds.""" - -from __future__ import annotations - -import pytest -from pydantic import ValidationError - -from app.capabilities.web.discover.schemas import DiscoverInput - -pytestmark = pytest.mark.unit - - -def test_top_k_defaults_and_accepts_the_valid_range(): - assert DiscoverInput(query="q").top_k == 10 - assert DiscoverInput(query="q", top_k=1).top_k == 1 - assert DiscoverInput(query="q", top_k=50).top_k == 50 - - -@pytest.mark.parametrize("bad", [0, -1, 51, 100]) -def test_top_k_outside_1_to_50_is_rejected(bad): - with pytest.raises(ValidationError): - DiscoverInput(query="q", top_k=bad) diff --git a/surfsense_backend/tests/unit/capabilities/web/discover/test_searxng_provider.py b/surfsense_backend/tests/unit/capabilities/web/discover/test_searxng_provider.py deleted file mode 100644 index a184666b3..000000000 --- a/surfsense_backend/tests/unit/capabilities/web/discover/test_searxng_provider.py +++ /dev/null @@ -1,58 +0,0 @@ -"""SearxngProvider maps the SearXNG service's sources to DiscoverHits. - -Boundary mocked: the web_search_service module. NOT mocked: the source→hit mapping. -""" - -from __future__ import annotations - -from unittest.mock import AsyncMock - -import pytest - -import app.capabilities.web.discover.providers.searxng as searxng_module -from app.capabilities.web.discover.providers.searxng import SearxngProvider - -pytestmark = pytest.mark.unit - - -def _result(sources): - return ({"sources": sources}, []) - - -async def test_maps_sources_to_hits(monkeypatch): - provider = SearxngProvider() - monkeypatch.setattr( - searxng_module.web_search_service, - "search", - AsyncMock( - return_value=_result( - [ - {"title": "Acme", "url": "https://acme.com", "description": "home"}, - { - "title": "Docs", - "url": "https://acme.com/docs", - "description": "", - }, - {"title": "no url", "url": "", "description": "skip me"}, - ] - ) - ), - ) - - hits = await provider.search("acme", top_k=5) - - assert [h.url for h in hits] == ["https://acme.com", "https://acme.com/docs"] - assert hits[0].title == "Acme" - assert hits[0].snippet == "home" - assert hits[1].snippet is None # empty description normalizes to None - assert all(h.provider == "searxng" for h in hits) - - -def test_is_available_reflects_the_service(monkeypatch): - provider = SearxngProvider() - monkeypatch.setattr(searxng_module.web_search_service, "is_available", lambda: True) - assert provider.is_available() is True - monkeypatch.setattr( - searxng_module.web_search_service, "is_available", lambda: False - ) - assert provider.is_available() is False diff --git a/surfsense_backend/tests/unit/capabilities/web/scrape/test_executor.py b/surfsense_backend/tests/unit/capabilities/web/scrape/test_executor.py deleted file mode 100644 index b37a0346c..000000000 --- a/surfsense_backend/tests/unit/capabilities/web/scrape/test_executor.py +++ /dev/null @@ -1,141 +0,0 @@ -"""`web.scrape` executor behavior: URLs in → cleaned rows out. - -Boundary mocked: the crawler (injected fake). NOT mocked: the executor's own -CrawlOutcome → ScrapeRow mapping. -""" - -from __future__ import annotations - -import pytest - -from app.capabilities.web.scrape.executor import build_scrape_executor -from app.capabilities.web.scrape.schemas import ScrapeInput, ScrapeOutput -from app.proprietary.web_crawler import CrawlOutcome, CrawlOutcomeStatus - -pytestmark = pytest.mark.unit - - -class _FakeCrawler: - """Stand-in for WebCrawlerConnector: serves a canned outcome per URL.""" - - def __init__(self, outcomes: dict[str, CrawlOutcome]): - self._outcomes = outcomes - self.calls: list[str] = [] - - async def crawl_url(self, url: str) -> CrawlOutcome: - self.calls.append(url) - return self._outcomes[url] - - -def _success(content: str, metadata: dict[str, str]) -> CrawlOutcome: - return CrawlOutcome( - status=CrawlOutcomeStatus.SUCCESS, - result={ - "content": content, - "metadata": metadata, - "crawler_type": "scrapling-static", - }, - ) - - -async def test_scrape_returns_one_cleaned_row_for_a_successful_url(): - url = "https://example.com" - crawler = _FakeCrawler({url: _success("# Hello", {"title": "Hello"})}) - execute = build_scrape_executor(engine=crawler) - - out = await execute(ScrapeInput(urls=[url])) - - assert isinstance(out, ScrapeOutput) - assert len(out.rows) == 1 - row = out.rows[0] - assert row.url == url - assert row.status == "success" - assert row.content == "# Hello" - assert row.metadata == {"title": "Hello"} - - -async def test_scrape_returns_one_row_per_url_in_input_order(): - a, b, c = "https://a.com", "https://b.com", "https://c.com" - crawler = _FakeCrawler( - { - a: _success("A", {"title": "A"}), - b: _success("B", {"title": "B"}), - c: _success("C", {"title": "C"}), - } - ) - execute = build_scrape_executor(engine=crawler) - - out = await execute(ScrapeInput(urls=[a, b, c])) - - assert [row.url for row in out.rows] == [a, b, c] - assert [row.content for row in out.rows] == ["A", "B", "C"] - - -async def test_content_longer_than_max_length_is_truncated(): - url = "https://long.com" - crawler = _FakeCrawler({url: _success("A" * 100, {"title": "Long"})}) - execute = build_scrape_executor(engine=crawler) - - out = await execute(ScrapeInput(urls=[url], max_length=10)) - - assert out.rows[0].content == "A" * 10 - - -async def test_content_within_max_length_is_untouched(): - url = "https://short.com" - crawler = _FakeCrawler({url: _success("hello", {"title": "Short"})}) - execute = build_scrape_executor(engine=crawler) - - out = await execute(ScrapeInput(urls=[url], max_length=10)) - - assert out.rows[0].content == "hello" - - -async def test_scrape_surfaces_total_captcha_attempts_for_billing(): - ok, blocked = "https://ok.com", "https://blocked.com" - crawler = _FakeCrawler( - { - ok: CrawlOutcome( - status=CrawlOutcomeStatus.SUCCESS, - result={"content": "OK", "metadata": {}}, - captcha_attempts=2, - captcha_solved=True, - ), - blocked: CrawlOutcome( - status=CrawlOutcomeStatus.FAILED, - error="blocked", - captcha_attempts=1, - captcha_solved=False, - ), - } - ) - execute = build_scrape_executor(engine=crawler) - - out = await execute(ScrapeInput(urls=[ok, blocked])) - - # Attempts bill even when the crawl ultimately failed (Phase 3d). - assert out.captcha_attempts == 3 - assert out.captcha_solved == 1 - - -async def test_partial_failure_keeps_the_batch_and_labels_each_url(): - ok, empty, failed = "https://ok.com", "https://empty.com", "https://failed.com" - crawler = _FakeCrawler( - { - ok: _success("OK", {"title": "OK"}), - empty: CrawlOutcome(status=CrawlOutcomeStatus.EMPTY, error="no content"), - failed: CrawlOutcome(status=CrawlOutcomeStatus.FAILED, error="blocked"), - } - ) - execute = build_scrape_executor(engine=crawler) - - out = await execute(ScrapeInput(urls=[ok, empty, failed])) - - by_url = {row.url: row for row in out.rows} - assert {u: r.status for u, r in by_url.items()} == { - ok: "success", - empty: "empty", - failed: "failed", - } - assert by_url[failed].content is None - assert by_url[failed].error == "blocked" diff --git a/surfsense_backend/tests/unit/capabilities/web/scrape/test_schemas.py b/surfsense_backend/tests/unit/capabilities/web/scrape/test_schemas.py deleted file mode 100644 index 9cc0eef23..000000000 --- a/surfsense_backend/tests/unit/capabilities/web/scrape/test_schemas.py +++ /dev/null @@ -1,47 +0,0 @@ -"""ScrapeOutput reports its own billable count; ScrapeInput bounds its batch size.""" - -from __future__ import annotations - -import pytest -from pydantic import ValidationError - -from app.capabilities.web.scrape.schemas import ( - MAX_SCRAPE_URLS, - ScrapeInput, - ScrapeOutput, - ScrapeRow, -) - -pytestmark = pytest.mark.unit - - -def _output(*statuses: str) -> ScrapeOutput: - return ScrapeOutput( - rows=[ - ScrapeRow(url=f"https://{i}.com", status=status) - for i, status in enumerate(statuses) - ] - ) - - -def test_billable_units_counts_successful_rows(): - assert _output("success", "empty", "success", "failed").billable_units == 2 - - -def test_billable_units_is_zero_without_successes(): - assert _output("empty", "failed").billable_units == 0 - - -def test_rejects_empty_url_batch(): - with pytest.raises(ValidationError): - ScrapeInput(urls=[]) - - -def test_rejects_batch_over_the_cap(): - with pytest.raises(ValidationError): - ScrapeInput(urls=[f"https://{i}.com" for i in range(MAX_SCRAPE_URLS + 1)]) - - -def test_accepts_batch_at_the_cap(): - payload = ScrapeInput(urls=[f"https://{i}.com" for i in range(MAX_SCRAPE_URLS)]) - assert payload.estimated_units == MAX_SCRAPE_URLS From 3312a442f3e9febd03d863379bbb1af80db3520b Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Fri, 3 Jul 2026 11:01:45 +0200 Subject: [PATCH 52/64] refactor(proprietary): move scrapers/ -> platforms/ (one subpackage per platform) --- .../capabilities/youtube/comments/executor.py | 2 +- .../capabilities/youtube/comments/schemas.py | 2 +- .../capabilities/youtube/scrape/executor.py | 2 +- .../capabilities/youtube/scrape/schemas.py | 2 +- .../app/proprietary/platforms/__init__.py | 9 ++++----- .../{scrapers => platforms}/youtube/README.md | 2 +- .../youtube/__init__.py | 0 .../youtube/comments.py | 2 +- .../youtube/innertube.py | 0 .../youtube/parsers.py | 0 .../youtube/schemas.py | 0 .../youtube/scraper.py | 0 .../youtube/search_filters.py | 0 .../youtube/subtitles.py | 0 .../youtube/url_resolver.py | 0 .../app/proprietary/scrapers/__init__.py | 1 - .../app/routes/youtube_routes.py | 2 +- .../scripts/e2e_youtube_scraper.py | 6 +++--- .../youtube/comments/test_executor.py | 2 +- .../youtube/scrape/test_executor.py | 2 +- .../tests/unit/platforms/__init__.py | 1 + .../youtube}/__init__.py | 0 .../youtube/test_fetch_resilience.py | 8 +++----- .../youtube/test_parsers.py | 20 +++++++++---------- .../tests/unit/scrapers/youtube/__init__.py | 0 25 files changed, 30 insertions(+), 33 deletions(-) rename surfsense_backend/app/proprietary/{scrapers => platforms}/youtube/README.md (99%) rename surfsense_backend/app/proprietary/{scrapers => platforms}/youtube/__init__.py (100%) rename surfsense_backend/app/proprietary/{scrapers => platforms}/youtube/comments.py (100%) rename surfsense_backend/app/proprietary/{scrapers => platforms}/youtube/innertube.py (100%) rename surfsense_backend/app/proprietary/{scrapers => platforms}/youtube/parsers.py (100%) rename surfsense_backend/app/proprietary/{scrapers => platforms}/youtube/schemas.py (100%) rename surfsense_backend/app/proprietary/{scrapers => platforms}/youtube/scraper.py (100%) rename surfsense_backend/app/proprietary/{scrapers => platforms}/youtube/search_filters.py (100%) rename surfsense_backend/app/proprietary/{scrapers => platforms}/youtube/subtitles.py (100%) rename surfsense_backend/app/proprietary/{scrapers => platforms}/youtube/url_resolver.py (100%) delete mode 100644 surfsense_backend/app/proprietary/scrapers/__init__.py create mode 100644 surfsense_backend/tests/unit/platforms/__init__.py rename surfsense_backend/tests/unit/{scrapers => platforms/youtube}/__init__.py (100%) rename surfsense_backend/tests/unit/{scrapers => platforms}/youtube/test_fetch_resilience.py (96%) rename surfsense_backend/tests/unit/{scrapers => platforms}/youtube/test_parsers.py (99%) delete mode 100644 surfsense_backend/tests/unit/scrapers/youtube/__init__.py diff --git a/surfsense_backend/app/capabilities/youtube/comments/executor.py b/surfsense_backend/app/capabilities/youtube/comments/executor.py index f4f28bef5..a5f4338c9 100644 --- a/surfsense_backend/app/capabilities/youtube/comments/executor.py +++ b/surfsense_backend/app/capabilities/youtube/comments/executor.py @@ -6,7 +6,7 @@ from collections.abc import Awaitable, Callable from app.capabilities.core import Executor from app.capabilities.youtube.comments.schemas import CommentsInput, CommentsOutput -from app.proprietary.scrapers.youtube import ( +from app.proprietary.platforms.youtube import ( YouTubeCommentsInput, scrape_comments, ) diff --git a/surfsense_backend/app/capabilities/youtube/comments/schemas.py b/surfsense_backend/app/capabilities/youtube/comments/schemas.py index c656b1f55..62eb7c451 100644 --- a/surfsense_backend/app/capabilities/youtube/comments/schemas.py +++ b/surfsense_backend/app/capabilities/youtube/comments/schemas.py @@ -10,7 +10,7 @@ from typing import Literal from pydantic import BaseModel, Field -from app.proprietary.scrapers.youtube import CommentItem +from app.proprietary.platforms.youtube import CommentItem MAX_COMMENT_VIDEOS = 20 """Per-call cap on how many video URLs one request may harvest comments from.""" diff --git a/surfsense_backend/app/capabilities/youtube/scrape/executor.py b/surfsense_backend/app/capabilities/youtube/scrape/executor.py index 82760e536..46d4789dc 100644 --- a/surfsense_backend/app/capabilities/youtube/scrape/executor.py +++ b/surfsense_backend/app/capabilities/youtube/scrape/executor.py @@ -6,7 +6,7 @@ from collections.abc import Awaitable, Callable from app.capabilities.core import Executor from app.capabilities.youtube.scrape.schemas import ScrapeInput, ScrapeOutput -from app.proprietary.scrapers.youtube import ( +from app.proprietary.platforms.youtube import ( YouTubeScrapeInput, scrape_youtube, ) diff --git a/surfsense_backend/app/capabilities/youtube/scrape/schemas.py b/surfsense_backend/app/capabilities/youtube/scrape/schemas.py index c24478b11..3c1f35ef1 100644 --- a/surfsense_backend/app/capabilities/youtube/scrape/schemas.py +++ b/surfsense_backend/app/capabilities/youtube/scrape/schemas.py @@ -10,7 +10,7 @@ from __future__ import annotations from pydantic import BaseModel, Field, model_validator -from app.proprietary.scrapers.youtube import VideoItem +from app.proprietary.platforms.youtube import VideoItem MAX_YOUTUBE_SOURCES = 20 """Per-call cap on URLs + queries: bounds a synchronous request's fan-out (05).""" diff --git a/surfsense_backend/app/proprietary/platforms/__init__.py b/surfsense_backend/app/proprietary/platforms/__init__.py index 2c505c8ad..86c49efbb 100644 --- a/surfsense_backend/app/proprietary/platforms/__init__.py +++ b/surfsense_backend/app/proprietary/platforms/__init__.py @@ -1,7 +1,6 @@ -"""Platform-specific crawl/extraction actors (non-Apache-2; see ../LICENSE). +"""Platform-native crawl/extraction actors (non-Apache-2; see ../LICENSE). -Scaffolded for future work (Phase 8 — Platform Actors). Each platform (e.g. -maps, professional networks) will get its own subpackage that reuses the shared -fetch strategies in ``app.proprietary.web_crawler`` under a platform-specific -structured extractor. Empty for now by design. +One subpackage per platform (e.g. ``youtube``), each a structured extractor for +that platform's data. Actors may reuse the shared fetch strategies in +``app.proprietary.web_crawler`` but expose their own platform-specific I/O. """ diff --git a/surfsense_backend/app/proprietary/scrapers/youtube/README.md b/surfsense_backend/app/proprietary/platforms/youtube/README.md similarity index 99% rename from surfsense_backend/app/proprietary/scrapers/youtube/README.md rename to surfsense_backend/app/proprietary/platforms/youtube/README.md index ca1105cbb..afcc718eb 100644 --- a/surfsense_backend/app/proprietary/scrapers/youtube/README.md +++ b/surfsense_backend/app/proprietary/platforms/youtube/README.md @@ -13,7 +13,7 @@ No API keys, no Apify account, no headless browser on the happy path. ## Quick start ```python -from app.proprietary.scrapers.youtube import ( +from app.proprietary.platforms.youtube import ( YouTubeScrapeInput, scrape_youtube, YouTubeCommentsInput, scrape_comments, ) diff --git a/surfsense_backend/app/proprietary/scrapers/youtube/__init__.py b/surfsense_backend/app/proprietary/platforms/youtube/__init__.py similarity index 100% rename from surfsense_backend/app/proprietary/scrapers/youtube/__init__.py rename to surfsense_backend/app/proprietary/platforms/youtube/__init__.py diff --git a/surfsense_backend/app/proprietary/scrapers/youtube/comments.py b/surfsense_backend/app/proprietary/platforms/youtube/comments.py similarity index 100% rename from surfsense_backend/app/proprietary/scrapers/youtube/comments.py rename to surfsense_backend/app/proprietary/platforms/youtube/comments.py index 5f60c7e01..c02331199 100644 --- a/surfsense_backend/app/proprietary/scrapers/youtube/comments.py +++ b/surfsense_backend/app/proprietary/platforms/youtube/comments.py @@ -27,8 +27,8 @@ from .parsers import ( parse_count, parse_video_page, ) -from .scraper import _post, _published_date, fan_out from .schemas import CommentItem, YouTubeCommentsInput +from .scraper import _post, _published_date, fan_out from .url_resolver import resolve_url logger = logging.getLogger(__name__) diff --git a/surfsense_backend/app/proprietary/scrapers/youtube/innertube.py b/surfsense_backend/app/proprietary/platforms/youtube/innertube.py similarity index 100% rename from surfsense_backend/app/proprietary/scrapers/youtube/innertube.py rename to surfsense_backend/app/proprietary/platforms/youtube/innertube.py diff --git a/surfsense_backend/app/proprietary/scrapers/youtube/parsers.py b/surfsense_backend/app/proprietary/platforms/youtube/parsers.py similarity index 100% rename from surfsense_backend/app/proprietary/scrapers/youtube/parsers.py rename to surfsense_backend/app/proprietary/platforms/youtube/parsers.py diff --git a/surfsense_backend/app/proprietary/scrapers/youtube/schemas.py b/surfsense_backend/app/proprietary/platforms/youtube/schemas.py similarity index 100% rename from surfsense_backend/app/proprietary/scrapers/youtube/schemas.py rename to surfsense_backend/app/proprietary/platforms/youtube/schemas.py diff --git a/surfsense_backend/app/proprietary/scrapers/youtube/scraper.py b/surfsense_backend/app/proprietary/platforms/youtube/scraper.py similarity index 100% rename from surfsense_backend/app/proprietary/scrapers/youtube/scraper.py rename to surfsense_backend/app/proprietary/platforms/youtube/scraper.py diff --git a/surfsense_backend/app/proprietary/scrapers/youtube/search_filters.py b/surfsense_backend/app/proprietary/platforms/youtube/search_filters.py similarity index 100% rename from surfsense_backend/app/proprietary/scrapers/youtube/search_filters.py rename to surfsense_backend/app/proprietary/platforms/youtube/search_filters.py diff --git a/surfsense_backend/app/proprietary/scrapers/youtube/subtitles.py b/surfsense_backend/app/proprietary/platforms/youtube/subtitles.py similarity index 100% rename from surfsense_backend/app/proprietary/scrapers/youtube/subtitles.py rename to surfsense_backend/app/proprietary/platforms/youtube/subtitles.py diff --git a/surfsense_backend/app/proprietary/scrapers/youtube/url_resolver.py b/surfsense_backend/app/proprietary/platforms/youtube/url_resolver.py similarity index 100% rename from surfsense_backend/app/proprietary/scrapers/youtube/url_resolver.py rename to surfsense_backend/app/proprietary/platforms/youtube/url_resolver.py diff --git a/surfsense_backend/app/proprietary/scrapers/__init__.py b/surfsense_backend/app/proprietary/scrapers/__init__.py deleted file mode 100644 index 808d7dfe8..000000000 --- a/surfsense_backend/app/proprietary/scrapers/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Platform-native scrapers (non-connector data extraction tools).""" diff --git a/surfsense_backend/app/routes/youtube_routes.py b/surfsense_backend/app/routes/youtube_routes.py index 22d8c6ace..76b489227 100644 --- a/surfsense_backend/app/routes/youtube_routes.py +++ b/surfsense_backend/app/routes/youtube_routes.py @@ -9,7 +9,7 @@ from fastapi import APIRouter, Depends, HTTPException, Query from scrapling.fetchers import AsyncFetcher from app.auth.context import AuthContext -from app.proprietary.scrapers.youtube import ( +from app.proprietary.platforms.youtube import ( YouTubeCommentsInput, YouTubeScrapeInput, scrape_comments, diff --git a/surfsense_backend/scripts/e2e_youtube_scraper.py b/surfsense_backend/scripts/e2e_youtube_scraper.py index 92ca9635a..4a4634674 100644 --- a/surfsense_backend/scripts/e2e_youtube_scraper.py +++ b/surfsense_backend/scripts/e2e_youtube_scraper.py @@ -31,20 +31,20 @@ for _candidate in (_BACKEND_ROOT / ".env", _BACKEND_ROOT.parent / ".env"): load_dotenv(_candidate) break -from app.proprietary.scrapers.youtube import ( # noqa: E402 +from app.proprietary.platforms.youtube import ( # noqa: E402 YouTubeCommentsInput, YouTubeScrapeInput, scrape_comments, scrape_youtube, ) -from app.proprietary.scrapers.youtube.innertube import ( # noqa: E402 +from app.proprietary.platforms.youtube.innertube import ( # noqa: E402 INNERTUBE_PUBLIC_API_KEY, INNERTUBE_SEARCH_URL, build_innertube_payload, fetch_html, post_innertube, ) -from app.proprietary.scrapers.youtube.parsers import ( # noqa: E402 +from app.proprietary.platforms.youtube.parsers import ( # noqa: E402 extract_yt_initial_data, extract_yt_initial_player_response, ) diff --git a/surfsense_backend/tests/unit/capabilities/youtube/comments/test_executor.py b/surfsense_backend/tests/unit/capabilities/youtube/comments/test_executor.py index 94fe941ad..fd153fed5 100644 --- a/surfsense_backend/tests/unit/capabilities/youtube/comments/test_executor.py +++ b/surfsense_backend/tests/unit/capabilities/youtube/comments/test_executor.py @@ -6,7 +6,7 @@ import pytest from app.capabilities.youtube.comments.executor import build_comments_executor from app.capabilities.youtube.comments.schemas import CommentsInput, CommentsOutput -from app.proprietary.scrapers.youtube import YouTubeCommentsInput +from app.proprietary.platforms.youtube import YouTubeCommentsInput pytestmark = pytest.mark.unit diff --git a/surfsense_backend/tests/unit/capabilities/youtube/scrape/test_executor.py b/surfsense_backend/tests/unit/capabilities/youtube/scrape/test_executor.py index 08f15981b..48c849cb5 100644 --- a/surfsense_backend/tests/unit/capabilities/youtube/scrape/test_executor.py +++ b/surfsense_backend/tests/unit/capabilities/youtube/scrape/test_executor.py @@ -10,7 +10,7 @@ import pytest from app.capabilities.youtube.scrape.executor import build_scrape_executor from app.capabilities.youtube.scrape.schemas import ScrapeInput, ScrapeOutput -from app.proprietary.scrapers.youtube import YouTubeScrapeInput +from app.proprietary.platforms.youtube import YouTubeScrapeInput pytestmark = pytest.mark.unit diff --git a/surfsense_backend/tests/unit/platforms/__init__.py b/surfsense_backend/tests/unit/platforms/__init__.py new file mode 100644 index 000000000..116a188e2 --- /dev/null +++ b/surfsense_backend/tests/unit/platforms/__init__.py @@ -0,0 +1 @@ +"""Platform-native scraper packages live here (one subpackage per platform).""" diff --git a/surfsense_backend/tests/unit/scrapers/__init__.py b/surfsense_backend/tests/unit/platforms/youtube/__init__.py similarity index 100% rename from surfsense_backend/tests/unit/scrapers/__init__.py rename to surfsense_backend/tests/unit/platforms/youtube/__init__.py diff --git a/surfsense_backend/tests/unit/scrapers/youtube/test_fetch_resilience.py b/surfsense_backend/tests/unit/platforms/youtube/test_fetch_resilience.py similarity index 96% rename from surfsense_backend/tests/unit/scrapers/youtube/test_fetch_resilience.py rename to surfsense_backend/tests/unit/platforms/youtube/test_fetch_resilience.py index 068c37b00..d69da5cb2 100644 --- a/surfsense_backend/tests/unit/scrapers/youtube/test_fetch_resilience.py +++ b/surfsense_backend/tests/unit/platforms/youtube/test_fetch_resilience.py @@ -9,10 +9,8 @@ from __future__ import annotations from collections.abc import AsyncIterator -import pytest - -from app.proprietary.scrapers.youtube import innertube, scraper -from app.proprietary.scrapers.youtube.innertube import ( +from app.proprietary.platforms.youtube import innertube, scraper +from app.proprietary.platforms.youtube.innertube import ( INNERTUBE_SEARCH_URL, _current_session, fetch_html, @@ -40,7 +38,7 @@ class _FakeSession: self.exc = exc self.calls = 0 - async def post(self, url, json=None): # noqa: A002 - mirror scrapling API + async def post(self, url, json=None): self.calls += 1 if self.exc: raise ConnectionError("boom") diff --git a/surfsense_backend/tests/unit/scrapers/youtube/test_parsers.py b/surfsense_backend/tests/unit/platforms/youtube/test_parsers.py similarity index 99% rename from surfsense_backend/tests/unit/scrapers/youtube/test_parsers.py rename to surfsense_backend/tests/unit/platforms/youtube/test_parsers.py index 28c624f62..47a457260 100644 --- a/surfsense_backend/tests/unit/scrapers/youtube/test_parsers.py +++ b/surfsense_backend/tests/unit/platforms/youtube/test_parsers.py @@ -12,7 +12,7 @@ from pathlib import Path import pytest -from app.proprietary.scrapers.youtube.parsers import ( +from app.proprietary.platforms.youtube.parsers import ( channel_about_tokens, comment_next_token, comment_reply_tokens, @@ -21,23 +21,23 @@ from app.proprietary.scrapers.youtube.parsers import ( dig, find_all, parse_channel_about, - parse_collaborators, - parse_comment_entities, - parse_description_links, - parse_location, - parse_translation, parse_channel_metadata, parse_channel_shorts, - parse_playlist_video_ids, + parse_collaborators, + parse_comment_entities, parse_count, parse_date, + parse_description_links, + parse_location, + parse_playlist_video_ids, parse_search_response, + parse_translation, parse_video_page, seconds_to_duration, ) -from app.proprietary.scrapers.youtube.schemas import YouTubeScrapeInput -from app.proprietary.scrapers.youtube.search_filters import build_search_params -from app.proprietary.scrapers.youtube.url_resolver import resolve_url +from app.proprietary.platforms.youtube.schemas import YouTubeScrapeInput +from app.proprietary.platforms.youtube.search_filters import build_search_params +from app.proprietary.platforms.youtube.url_resolver import resolve_url pytestmark = pytest.mark.unit diff --git a/surfsense_backend/tests/unit/scrapers/youtube/__init__.py b/surfsense_backend/tests/unit/scrapers/youtube/__init__.py deleted file mode 100644 index e69de29bb..000000000 From 5da624399d5c59ec63897c5d2c2fe4940608ef37 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Fri, 3 Jul 2026 11:20:09 +0200 Subject: [PATCH 53/64] refactor(subagents): split scraping into web_crawler + youtube builtins --- .../agents/chat/multi_agent_chat/constants.py | 3 +- .../builtins/scraping/description.md | 2 - .../builtins/scraping/system_prompt.md | 65 ------- .../builtins/scraping/tools/__init__.py | 1 - .../builtins/scraping/tools/index.py | 49 ----- .../builtins/scraping/tools/refresh_watch.py | 82 -------- .../builtins/scraping/tools/start_watch.py | 97 ---------- .../builtins/scraping/tools/stop_watch.py | 85 -------- .../builtins/web_crawler/__init__.py | 1 + .../{scraping => web_crawler}/agent.py | 4 +- .../builtins/web_crawler/description.md | 2 + .../builtins/web_crawler/system_prompt.md | 60 ++++++ .../tools}/__init__.py | 0 .../builtins/web_crawler/tools/index.py | 27 +++ .../subagents/builtins/youtube/__init__.py | 1 + .../subagents/builtins/youtube/agent.py | 43 +++++ .../subagents/builtins/youtube/description.md | 2 + .../builtins/youtube/system_prompt.md | 61 ++++++ .../builtins/youtube}/tools/__init__.py | 0 .../subagents/builtins/youtube/tools/index.py | 28 +++ .../multi_agent_chat/subagents/registry.py | 10 +- .../automations/tools/test_watch_tools.py | 182 ------------------ .../subagents/test_start_watch_tool.py | 118 ------------ .../subagents/test_watch_control_tools.py | 119 ------------ .../test_subagent_composition.py | 5 +- 25 files changed, 239 insertions(+), 808 deletions(-) delete mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/description.md delete mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/system_prompt.md delete mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/__init__.py delete mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/index.py delete mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/refresh_watch.py delete mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/start_watch.py delete mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/stop_watch.py create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/__init__.py rename surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/{scraping => web_crawler}/agent.py (88%) create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/description.md create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/system_prompt.md rename surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/{scraping => web_crawler/tools}/__init__.py (100%) create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/tools/index.py create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/__init__.py create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/agent.py create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/description.md create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/system_prompt.md rename surfsense_backend/{tests/integration/automations => app/agents/chat/multi_agent_chat/subagents/builtins/youtube}/tools/__init__.py (100%) create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/tools/index.py delete mode 100644 surfsense_backend/tests/integration/automations/tools/test_watch_tools.py delete mode 100644 surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_start_watch_tool.py delete mode 100644 surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_watch_control_tools.py diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/constants.py b/surfsense_backend/app/agents/chat/multi_agent_chat/constants.py index 1e4c4276d..e6c5b4975 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/constants.py +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/constants.py @@ -26,7 +26,8 @@ CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS: dict[str, str] = { SUBAGENT_TO_REQUIRED_CONNECTOR_MAP: dict[str, frozenset[str]] = { "deliverables": frozenset(), "knowledge_base": frozenset(), - "scraping": frozenset(), + "web_crawler": frozenset(), + "youtube": frozenset(), "airtable": frozenset({"AIRTABLE_CONNECTOR"}), "calendar": frozenset({"GOOGLE_CALENDAR_CONNECTOR"}), "clickup": frozenset({"CLICKUP_CONNECTOR"}), diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/description.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/description.md deleted file mode 100644 index 1b0e087f6..000000000 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/description.md +++ /dev/null @@ -1,2 +0,0 @@ -Scraping specialist: fetches live public web pages, searches the web to find pages, pulls structured data from YouTube (videos, channels, playlists, search, and comments), and compares fresh data against earlier findings in this chat. -Use whenever the task needs current data pulled from the open web or YouTube rather than the workspace's own documents or connectors. Triggers include "scrape", "crawl", "fetch this URL/page", "search the web", "look up / find online", "check the price/stock/listing", "monitor this page", "what changed since last time" — plus YouTube-specific asks like "get this YouTube video/channel/playlist", "find videos about X on YouTube", "get the comments on this video", "what are people saying about this video" — and recurring versions ("check daily", "tell me weekly what changed"), which it can also schedule as an ongoing watch. diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/system_prompt.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/system_prompt.md deleted file mode 100644 index d4b0309a4..000000000 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/system_prompt.md +++ /dev/null @@ -1,65 +0,0 @@ -You are the SurfSense scraping sub-agent. -You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis. - - -Answer the delegated question from live evidence gathered with your data verbs, including "what changed" comparisons against evidence already in this conversation. - - - -- `web_discover` -- `web_scrape` -- `youtube_scrape` -- `youtube_comments` -- `start_watch` -- `stop_watch` -- `refresh_watch` - - - -- Named URLs: `web_scrape` them directly. Otherwise `web_discover` first, then `web_scrape` the most relevant hits. -- Read several pages in one batched `web_scrape` call rather than many single-URL calls. -- YouTube: use `youtube_scrape` for videos/channels/playlists (pass `urls`) or to find videos (pass `search_queries`); use `youtube_comments` to pull a video's comments. Prefer these over `web_scrape` for youtube.com links. -- "What changed" / monitoring: scrape the current values, compare against the prior values in this conversation's earlier tool results, and report concrete deltas (added, removed, old -> new). -- Recurring intent ("check daily", "tell me weekly what changed"): answer now, then `start_watch` with a self-contained question, a cron cadence, and an IANA timezone. Use `stop_watch` / `refresh_watch` to end or immediately re-run an existing watch. - - - -- Use only tools in ``. -- A `web_scrape` row whose `status` is not `success` returned no content — report it unavailable, never invent it. -- Report only deltas you can point to in the evidence. Never fabricate facts, URLs, prices, or quotes. - - - -- Do not generate deliverables or perform connector mutations; return findings for the supervisor to act on. - - - -- Report uncertainty explicitly when evidence is incomplete or conflicting. -- Never present unverified claims as facts. - - - -- Underspecified request — including a recurring request whose cadence or timezone is neither given nor implied — return `status=blocked` with the missing fields. -- Tool failure: return `status=error` with a concise recovery `next_step`. -- No useful evidence: return `status=blocked` with a narrower query or the URLs you still need. - - - -Return **only** one JSON object (no markdown/prose): -{ - "status": "success" | "partial" | "blocked" | "error", - "action_summary": string, - "evidence": { - "findings": string[], - "sources": string[], - "confidence": "high" | "medium" | "low" - }, - "next_step": string | null, - "missing_fields": string[] | null, - "assumptions": string[] | null -} - -Route-specific rules: -- `evidence.findings`: max 10 entries, each a single sentence stating one distinct fact or delta. Do not paste raw scraped pages. -- `evidence.sources`: max 10 URLs, one per finding when applicable. List each URL once. - diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/__init__.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/__init__.py deleted file mode 100644 index e06e600b5..000000000 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Intelligence-agent tools: the web.* capability verbs, generated from the registry.""" diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/index.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/index.py deleted file mode 100644 index fb9018fb4..000000000 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/index.py +++ /dev/null @@ -1,49 +0,0 @@ -"""``scraping`` sub-agent tools: scraper capability verbs + chat-watch controls.""" - -from __future__ import annotations - -from typing import Any - -from langchain_core.tools import BaseTool - -from app.agents.chat.multi_agent_chat.shared.permissions import Ruleset -from app.capabilities.core.access.agent import build_capability_tools -from app.capabilities.web.crawl.definition import WEB_CRAWL -from app.capabilities.youtube.comments.definition import YOUTUBE_COMMENTS -from app.capabilities.youtube.scrape.definition import YOUTUBE_SCRAPE - -from .refresh_watch import create_refresh_watch_tool -from .start_watch import create_start_watch_tool -from .stop_watch import create_stop_watch_tool - -NAME = "scraping" - -RULESET = Ruleset(origin=NAME, rules=[]) - -_CI_VERBS = [WEB_CRAWL, YOUTUBE_SCRAPE, YOUTUBE_COMMENTS] - - -def load_tools( - *, dependencies: dict[str, Any] | None = None, **kwargs: Any -) -> list[BaseTool]: - d = {**(dependencies or {}), **kwargs} - tools: list[BaseTool] = build_capability_tools( - workspace_id=d.get("workspace_id"), - capabilities=_CI_VERBS, - ) - - # Watch tools bind a recurring automation to the current chat, so they are - # only offered when we have a chat to bind to and auth to manage it. - thread_id = d.get("thread_id") - auth_context = d.get("auth_context") - if thread_id is not None and auth_context is not None: - binding = { - "workspace_id": d.get("workspace_id"), - "thread_id": thread_id, - "auth_context": auth_context, - } - tools.append(create_start_watch_tool(**binding)) - tools.append(create_stop_watch_tool(**binding)) - tools.append(create_refresh_watch_tool(**binding)) - - return tools diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/refresh_watch.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/refresh_watch.py deleted file mode 100644 index 0e8d507b5..000000000 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/refresh_watch.py +++ /dev/null @@ -1,82 +0,0 @@ -"""``refresh_watch`` — run the current chat's watch now (manual refresh). - -Enqueues an immediate run of every watch bound to this chat, instead of waiting -for the next scheduled fire. Operates on the current thread only. -""" - -from __future__ import annotations - -import logging -from typing import Any - -from fastapi import HTTPException -from langchain_core.tools import tool - -from app.auth.context import AuthContext -from app.automations.services.automation import AutomationService -from app.automations.services.chat_watch import find_watches_for_thread, run_watch_now -from app.db import async_session_maker - -logger = logging.getLogger(__name__) - - -def create_refresh_watch_tool( - *, - workspace_id: int | None, - thread_id: int | str | None, - auth_context: AuthContext | None, -): - """Build the ``refresh_watch`` tool bound to the current chat.""" - - @tool - async def refresh_watch() -> dict[str, Any]: - """Run THIS chat's watch now (a manual refresh). - - Use when the user says "check now", "refresh", or "run it again" - without waiting for the schedule. No arguments — it acts on the - current chat. The refreshed answer arrives as a new turn shortly after. - - Returns: - ``{"status": "refreshing", "refreshed_ids": [int], "count": int}`` - when runs were enqueued, ``{"status": "not_watching", ...}`` when - there is no watch, or ``{"status": "error", "message": str}``. - """ - if thread_id is None or auth_context is None: - return { - "status": "error", - "message": "Watches can only be managed from inside a chat.", - } - - try: - async with async_session_maker() as session: - service = AutomationService(session=session, auth=auth_context) - watches = await find_watches_for_thread( - service, - workspace_id=int(workspace_id) if workspace_id is not None else 0, - thread_id=int(thread_id), - ) - if not watches: - return { - "status": "not_watching", - "message": "This chat has no active watch to refresh.", - } - refreshed_ids = [] - for watch in watches: - await run_watch_now(service, automation_id=watch.id) - refreshed_ids.append(watch.id) - except HTTPException as exc: - return {"status": "error", "message": str(exc.detail)} - except Exception as exc: - logger.exception("refresh_watch failed") - return {"status": "error", "message": f"could not refresh watch: {exc}"} - - return { - "status": "refreshing", - "refreshed_ids": refreshed_ids, - "count": len(refreshed_ids), - } - - return refresh_watch - - -__all__ = ["create_refresh_watch_tool"] diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/start_watch.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/start_watch.py deleted file mode 100644 index 30cdfe9f9..000000000 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/start_watch.py +++ /dev/null @@ -1,97 +0,0 @@ -"""``start_watch`` tool — bind a recurring watch to the current chat. - -Creates a ``schedule`` + ``chat_message`` automation that re-posts the question -into this chat on a cadence, so the agent re-answers against the chat's memory. -""" - -from __future__ import annotations - -import logging -from typing import Any - -from fastapi import HTTPException -from langchain_core.tools import tool - -from app.auth.context import AuthContext -from app.automations.services.automation import AutomationService -from app.automations.services.chat_watch import create_watch -from app.db import async_session_maker - -logger = logging.getLogger(__name__) - - -def create_start_watch_tool( - *, - workspace_id: int | None, - thread_id: int | str | None, - auth_context: AuthContext | None, -): - """Build the ``start_watch`` tool bound to the current chat. - - ``thread_id`` and ``auth_context`` are injected from the chat session so the - model never guesses them; a fresh session is opened per call. - """ - - @tool - async def start_watch(message: str, cron: str, timezone: str) -> dict[str, Any]: - """Keep watching: re-run this question on a schedule in THIS chat. - - Use when the user wants an answer refreshed on a cadence without - re-asking (e.g. "check this page every weekday and tell me what - changed"). The watch re-posts ``message`` into this chat on the - schedule; the user sees new answers here and can say "stop watching" - to cancel. - - Args: - message: The question to re-ask each run, self-contained (include - the URL / target). Written as if the user asked it again. - cron: Five-field cron expression for the cadence - (e.g. "0 9 * * 1-5" = weekdays at 09:00). - timezone: IANA timezone for the schedule (e.g. "UTC", - "Africa/Kigali"). Ask the user if unclear. - - Returns: - ``{"status": "watching", "automation_id": int, "name": str, - "cron": str, "timezone": str}`` on success. - ``{"status": "error", "message": str}`` when the watch could not - be created (e.g. workspace model not billable for automations). - """ - if thread_id is None or auth_context is None: - return { - "status": "error", - "message": "Watches can only be started from inside a chat.", - } - - try: - async with async_session_maker() as session: - service = AutomationService(session=session, auth=auth_context) - created = await create_watch( - service, - workspace_id=int(workspace_id) if workspace_id is not None else 0, - thread_id=int(thread_id), - message=message, - cron=cron, - timezone=timezone, - ) - except HTTPException as exc: - return {"status": "error", "message": str(exc.detail)} - except Exception as exc: - from langgraph.errors import GraphInterrupt - - if isinstance(exc, GraphInterrupt): - raise - logger.exception("start_watch failed") - return {"status": "error", "message": f"could not start watch: {exc}"} - - return { - "status": "watching", - "automation_id": created.id, - "name": created.name, - "cron": cron, - "timezone": timezone, - } - - return start_watch - - -__all__ = ["create_start_watch_tool"] diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/stop_watch.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/stop_watch.py deleted file mode 100644 index 4f24c294f..000000000 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/tools/stop_watch.py +++ /dev/null @@ -1,85 +0,0 @@ -"""``stop_watch`` — stop watching in the current chat. - -Deletes every watch automation bound to this chat, so it reverts to a normal -chat. Operates on the current thread only; the model passes no arguments. -""" - -from __future__ import annotations - -import logging -from typing import Any - -from fastapi import HTTPException -from langchain_core.tools import tool - -from app.auth.context import AuthContext -from app.automations.services.automation import AutomationService -from app.automations.services.chat_watch import ( - find_watches_for_thread, - stop_watch as stop_watch_service, -) -from app.db import async_session_maker - -logger = logging.getLogger(__name__) - - -def create_stop_watch_tool( - *, - workspace_id: int | None, - thread_id: int | str | None, - auth_context: AuthContext | None, -): - """Build the ``stop_watch`` tool bound to the current chat.""" - - @tool - async def stop_watch() -> dict[str, Any]: - """Stop watching in THIS chat (cancel the recurring re-asks). - - Use when the user says something like "stop watching", "cancel the - watch", or "you can stop checking now". The chat reverts to a normal - chat. No arguments — it acts on the current chat. - - Returns: - ``{"status": "stopped", "stopped_ids": [int], "count": int}`` when - watches were removed, ``{"status": "not_watching", ...}`` when there - was nothing to stop, or ``{"status": "error", "message": str}``. - """ - if thread_id is None or auth_context is None: - return { - "status": "error", - "message": "Watches can only be managed from inside a chat.", - } - - try: - async with async_session_maker() as session: - service = AutomationService(session=session, auth=auth_context) - watches = await find_watches_for_thread( - service, - workspace_id=int(workspace_id) if workspace_id is not None else 0, - thread_id=int(thread_id), - ) - if not watches: - return { - "status": "not_watching", - "message": "This chat has no active watch.", - } - stopped_ids = [] - for watch in watches: - await stop_watch_service(service, automation_id=watch.id) - stopped_ids.append(watch.id) - except HTTPException as exc: - return {"status": "error", "message": str(exc.detail)} - except Exception as exc: - logger.exception("stop_watch failed") - return {"status": "error", "message": f"could not stop watch: {exc}"} - - return { - "status": "stopped", - "stopped_ids": stopped_ids, - "count": len(stopped_ids), - } - - return stop_watch - - -__all__ = ["create_stop_watch_tool"] diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/__init__.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/__init__.py new file mode 100644 index 000000000..7b8213fb3 --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/__init__.py @@ -0,0 +1 @@ +"""``web_crawler`` builtin subagent: crawl single URLs or spider whole sites.""" diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/agent.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/agent.py similarity index 88% rename from surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/agent.py rename to surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/agent.py index 9c6734b57..49d6423dc 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/agent.py +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/agent.py @@ -1,4 +1,4 @@ -"""``scraping`` route: ``SurfSenseSubagentSpec`` builder for deepagents.""" +"""``web_crawler`` route: ``SurfSenseSubagentSpec`` builder for deepagents.""" from __future__ import annotations @@ -28,7 +28,7 @@ def build_subagent( tools = [*load_tools(dependencies=dependencies), *(mcp_tools or [])] description = ( read_md_file(__package__, "description").strip() - or "Scrapes live public web pages and search results for this workspace." + or "Crawls live public web pages and whole sites for this workspace." ) system_prompt = read_md_file(__package__, "system_prompt").strip() return pack_subagent( diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/description.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/description.md new file mode 100644 index 000000000..a687e8ad3 --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/description.md @@ -0,0 +1,2 @@ +Web crawling specialist: fetches live public web pages by URL and can spider a whole website (follow its links to a given depth), returning clean markdown, page metadata, and crawl provenance. Also compares freshly crawled data against earlier findings in this chat. +Use whenever the task needs current content pulled from the open web rather than the workspace's own documents or connectors. Triggers include "scrape", "crawl", "fetch this URL/page", "read this website", "crawl this site", "get the pages under X", "check the price/stock/listing", "monitor this page", and "what changed since last time". Not for YouTube links (use the youtube specialist) or for searching the web to discover unknown URLs. diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/system_prompt.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/system_prompt.md new file mode 100644 index 000000000..4133f7bc9 --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/system_prompt.md @@ -0,0 +1,60 @@ +You are the SurfSense web crawling sub-agent. +You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis. + + +Answer the delegated question from live web evidence gathered with `web_crawl`, including "what changed" comparisons against evidence already in this conversation. + + + +- `web_crawl` + + + +- Single page(s): call `web_crawl` with the URL(s) in `startUrls` and `maxCrawlDepth=0`. +- Whole site / "pages under X": set `maxCrawlDepth` to 1+ to follow links, and cap the run with `maxCrawlPages`. The crawl stays on the start URL's site. +- Batch known URLs into one `web_crawl` call (pass them all in `startUrls`) rather than many single-URL calls. +- Keep depth and page caps as small as the task allows — each fetched page is billable. +- "What changed" / monitoring: crawl the current values, compare against the prior values in this conversation's earlier tool results, and report concrete deltas (added, removed, old -> new). + + + +- Use only tools in ``. +- A `web_crawl` item whose `status` is not `success` returned no content — report it unavailable, never invent it. +- Report only deltas you can point to in the evidence. Never fabricate facts, URLs, prices, or quotes. + + + +- Do not generate deliverables or perform connector mutations; return findings for the supervisor to act on. +- YouTube URLs belong to the youtube specialist, not here. + + + +- Report uncertainty explicitly when evidence is incomplete or conflicting. +- Never present unverified claims as facts. + + + +- Underspecified request — no usable URL to start from — return `status=blocked` with the missing fields. +- Tool failure: return `status=error` with a concise recovery `next_step`. +- No useful evidence: return `status=blocked` with the URLs you still need or a narrower scope. + + + +Return **only** one JSON object (no markdown/prose): +{ + "status": "success" | "partial" | "blocked" | "error", + "action_summary": string, + "evidence": { + "findings": string[], + "sources": string[], + "confidence": "high" | "medium" | "low" + }, + "next_step": string | null, + "missing_fields": string[] | null, + "assumptions": string[] | null +} + +Route-specific rules: +- `evidence.findings`: max 10 entries, each a single sentence stating one distinct fact or delta. Do not paste raw crawled pages. +- `evidence.sources`: max 10 URLs, one per finding when applicable. List each URL once. + diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/__init__.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/tools/__init__.py similarity index 100% rename from surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/scraping/__init__.py rename to surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/tools/__init__.py diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/tools/index.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/tools/index.py new file mode 100644 index 000000000..1b80acd38 --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/tools/index.py @@ -0,0 +1,27 @@ +"""``web_crawler`` sub-agent tools: the unified web.crawl capability verb.""" + +from __future__ import annotations + +from typing import Any + +from langchain_core.tools import BaseTool + +from app.agents.chat.multi_agent_chat.shared.permissions import Ruleset +from app.capabilities.core.access.agent import build_capability_tools +from app.capabilities.web.crawl.definition import WEB_CRAWL + +NAME = "web_crawler" + +RULESET = Ruleset(origin=NAME, rules=[]) + +_CI_VERBS = [WEB_CRAWL] + + +def load_tools( + *, dependencies: dict[str, Any] | None = None, **kwargs: Any +) -> list[BaseTool]: + d = {**(dependencies or {}), **kwargs} + return build_capability_tools( + workspace_id=d.get("workspace_id"), + capabilities=_CI_VERBS, + ) diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/__init__.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/__init__.py new file mode 100644 index 000000000..2d4207834 --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/__init__.py @@ -0,0 +1 @@ +"""``youtube`` builtin subagent: structured YouTube video/channel/comment data.""" diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/agent.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/agent.py new file mode 100644 index 000000000..acc65bdc5 --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/agent.py @@ -0,0 +1,43 @@ +"""``youtube`` route: ``SurfSenseSubagentSpec`` builder for deepagents.""" + +from __future__ import annotations + +from typing import Any + +from langchain_core.language_models import BaseChatModel +from langchain_core.tools import BaseTool + +from app.agents.chat.multi_agent_chat.subagents.shared.md_file_reader import ( + read_md_file, +) +from app.agents.chat.multi_agent_chat.subagents.shared.spec import SurfSenseSubagentSpec +from app.agents.chat.multi_agent_chat.subagents.shared.subagent_builder import ( + pack_subagent, +) + +from .tools.index import NAME, RULESET, load_tools + + +def build_subagent( + *, + dependencies: dict[str, Any], + model: BaseChatModel | None = None, + middleware_stack: dict[str, Any] | None = None, + mcp_tools: list[BaseTool] | None = None, +) -> SurfSenseSubagentSpec: + tools = [*load_tools(dependencies=dependencies), *(mcp_tools or [])] + description = ( + read_md_file(__package__, "description").strip() + or "Pulls structured data from YouTube videos, channels, playlists, and comments." + ) + system_prompt = read_md_file(__package__, "system_prompt").strip() + return pack_subagent( + name=NAME, + description=description, + system_prompt=system_prompt, + tools=tools, + ruleset=RULESET, + dependencies=dependencies, + model=model, + middleware_stack=middleware_stack, + ) diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/description.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/description.md new file mode 100644 index 000000000..6ccc9a0d5 --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/description.md @@ -0,0 +1,2 @@ +YouTube specialist: pulls structured data from YouTube — videos, channels, playlists, shorts, and hashtags (title, views, likes, publish date, channel info, description, optional subtitles), finds videos by search query, and fetches a video's comments and replies. Also compares fresh YouTube data against earlier findings in this chat. +Use whenever the task involves YouTube content or a youtube.com/youtu.be link. Triggers include "get this YouTube video/channel/playlist", "find videos about X on YouTube", "how many views/likes", "get the transcript/subtitles", "get the comments on this video", "what are people saying about this video", and recurring "what changed" checks on YouTube data. Not for general web pages (use the web crawling specialist for non-YouTube URLs). diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/system_prompt.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/system_prompt.md new file mode 100644 index 000000000..6c500c777 --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/system_prompt.md @@ -0,0 +1,61 @@ +You are the SurfSense YouTube sub-agent. +You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis. + + +Answer the delegated question from live YouTube data gathered with your verbs, including "what changed" comparisons against evidence already in this conversation. + + + +- `youtube_scrape` +- `youtube_comments` + + + +- Known video/channel/playlist/shorts/hashtag links: call `youtube_scrape` with the links in `urls`. +- Finding videos on a topic: call `youtube_scrape` with `search_queries`. +- Comments / sentiment on specific videos: call `youtube_comments` with the video `urls`. +- Batch multiple URLs (or queries) into one call rather than many single-item calls. +- "What changed" / monitoring: pull the current values, compare against the prior values in this conversation's earlier tool results, and report concrete deltas (added, removed, old -> new). + + + +- Use only tools in ``. +- An item whose `status` is not `success` returned no data — report it unavailable, never invent it. +- Report only deltas you can point to in the evidence. Never fabricate facts, counts, quotes, or URLs. + + + +- Do not generate deliverables or perform connector mutations; return findings for the supervisor to act on. +- Non-YouTube web pages belong to the web crawling specialist, not here. + + + +- Report uncertainty explicitly when evidence is incomplete or conflicting. +- Never present unverified claims as facts. + + + +- Underspecified request — no usable URL or search query — return `status=blocked` with the missing fields. +- Tool failure: return `status=error` with a concise recovery `next_step`. +- No useful evidence: return `status=blocked` with a narrower query or the URLs you still need. + + + +Return **only** one JSON object (no markdown/prose): +{ + "status": "success" | "partial" | "blocked" | "error", + "action_summary": string, + "evidence": { + "findings": string[], + "sources": string[], + "confidence": "high" | "medium" | "low" + }, + "next_step": string | null, + "missing_fields": string[] | null, + "assumptions": string[] | null +} + +Route-specific rules: +- `evidence.findings`: max 10 entries, each a single sentence stating one distinct fact or delta. Do not paste raw payloads. +- `evidence.sources`: max 10 URLs, one per finding when applicable. List each URL once. + diff --git a/surfsense_backend/tests/integration/automations/tools/__init__.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/tools/__init__.py similarity index 100% rename from surfsense_backend/tests/integration/automations/tools/__init__.py rename to surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/tools/__init__.py diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/tools/index.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/tools/index.py new file mode 100644 index 000000000..1d4e94662 --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/tools/index.py @@ -0,0 +1,28 @@ +"""``youtube`` sub-agent tools: the YouTube scrape + comments capability verbs.""" + +from __future__ import annotations + +from typing import Any + +from langchain_core.tools import BaseTool + +from app.agents.chat.multi_agent_chat.shared.permissions import Ruleset +from app.capabilities.core.access.agent import build_capability_tools +from app.capabilities.youtube.comments.definition import YOUTUBE_COMMENTS +from app.capabilities.youtube.scrape.definition import YOUTUBE_SCRAPE + +NAME = "youtube" + +RULESET = Ruleset(origin=NAME, rules=[]) + +_CI_VERBS = [YOUTUBE_SCRAPE, YOUTUBE_COMMENTS] + + +def load_tools( + *, dependencies: dict[str, Any] | None = None, **kwargs: Any +) -> list[BaseTool]: + d = {**(dependencies or {}), **kwargs} + return build_capability_tools( + workspace_id=d.get("workspace_id"), + capabilities=_CI_VERBS, + ) diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py index ae3a88bbe..02c196086 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py @@ -24,8 +24,11 @@ from app.agents.chat.multi_agent_chat.subagents.builtins.memory.agent import ( from app.agents.chat.multi_agent_chat.subagents.builtins.research.agent import ( build_subagent as build_research_subagent, ) -from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.agent import ( - build_subagent as build_scraping_subagent, +from app.agents.chat.multi_agent_chat.subagents.builtins.web_crawler.agent import ( + build_subagent as build_web_crawler_subagent, +) +from app.agents.chat.multi_agent_chat.subagents.builtins.youtube.agent import ( + build_subagent as build_youtube_subagent, ) from app.agents.chat.multi_agent_chat.subagents.connectors.airtable.agent import ( build_subagent as build_airtable_subagent, @@ -110,9 +113,10 @@ SUBAGENT_BUILDERS_BY_NAME: dict[str, SubagentBuilder] = { "notion": build_notion_subagent, "onedrive": build_onedrive_subagent, "research": build_research_subagent, - "scraping": build_scraping_subagent, "slack": build_slack_subagent, "teams": build_teams_subagent, + "web_crawler": build_web_crawler_subagent, + "youtube": build_youtube_subagent, } diff --git a/surfsense_backend/tests/integration/automations/tools/test_watch_tools.py b/surfsense_backend/tests/integration/automations/tools/test_watch_tools.py deleted file mode 100644 index acec89069..000000000 --- a/surfsense_backend/tests/integration/automations/tools/test_watch_tools.py +++ /dev/null @@ -1,182 +0,0 @@ -"""Integration tests for the watch chat tools against real service + Postgres. - -The tools are thin adapters the intelligence agent calls; here they run their -real path — open a session, build ``AutomationService``, and create / find / -delete / enqueue — against real persistence (RBAC + model gate included), with -only the Celery enqueue spied. This is what proves ``start_watch`` actually -binds a watch, ``stop_watch`` removes it, and ``refresh_watch`` enqueues a run. -""" - -from __future__ import annotations - -import pytest -import pytest_asyncio -from sqlalchemy.ext.asyncio import AsyncSession - -from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools.refresh_watch import ( - create_refresh_watch_tool, -) -from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools.start_watch import ( - create_start_watch_tool, -) -from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools.stop_watch import ( - create_stop_watch_tool, -) -from app.auth.context import AuthContext -from app.automations.services.automation import AutomationService -from app.automations.services.chat_watch import find_watches_for_thread -from app.db import ChatVisibility, NewChatThread, User, Workspace - -pytestmark = pytest.mark.integration - -_CRON = "0 9 * * 1-5" -_TZ = "UTC" - - -def _billable(workspace: Workspace) -> None: - workspace.chat_model_id = 1 - workspace.image_gen_model_id = 1 - workspace.vision_model_id = 1 - - -@pytest_asyncio.fixture -async def thread( - db_session: AsyncSession, db_user: User, db_workspace: Workspace -) -> NewChatThread: - row = NewChatThread( - title="Watched chat", - workspace_id=db_workspace.id, - created_by_id=db_user.id, - visibility=ChatVisibility.PRIVATE, - ) - db_session.add(row) - await db_session.flush() - return row - - -async def test_start_watch_tool_binds_a_watch_to_the_chat( - tools_use_test_session, - db_session: AsyncSession, - db_user: User, - db_workspace: Workspace, - thread: NewChatThread, -) -> None: - _billable(db_workspace) - await db_session.flush() - - tool = create_start_watch_tool( - workspace_id=db_workspace.id, - thread_id=thread.id, - auth_context=AuthContext.session(db_user), - ) - out = await tool.ainvoke( - {"message": "what changed?", "cron": _CRON, "timezone": _TZ} - ) - - assert out["status"] == "watching" - assert isinstance(out["automation_id"], int) - - service = AutomationService(session=db_session, auth=AuthContext.session(db_user)) - found = await find_watches_for_thread( - service, workspace_id=db_workspace.id, thread_id=thread.id - ) - assert [a.id for a in found] == [out["automation_id"]] - - -async def test_start_watch_tool_reports_error_when_models_not_billable( - tools_use_test_session, - db_user: User, - db_workspace: Workspace, - thread: NewChatThread, -) -> None: - # Default workspace models are Auto/None -> the automation gate rejects it. - tool = create_start_watch_tool( - workspace_id=db_workspace.id, - thread_id=thread.id, - auth_context=AuthContext.session(db_user), - ) - out = await tool.ainvoke( - {"message": "track it", "cron": _CRON, "timezone": _TZ} - ) - assert out["status"] == "error" - - -async def test_stop_watch_tool_removes_the_chats_watch( - tools_use_test_session, - db_session: AsyncSession, - db_user: User, - db_workspace: Workspace, - thread: NewChatThread, -) -> None: - _billable(db_workspace) - await db_session.flush() - auth = AuthContext.session(db_user) - - start = create_start_watch_tool( - workspace_id=db_workspace.id, thread_id=thread.id, auth_context=auth - ) - await start.ainvoke({"message": "m", "cron": _CRON, "timezone": _TZ}) - - stop = create_stop_watch_tool( - workspace_id=db_workspace.id, thread_id=thread.id, auth_context=auth - ) - out = await stop.ainvoke({}) - - assert out["status"] == "stopped" - assert out["count"] == 1 - - service = AutomationService(session=db_session, auth=auth) - found = await find_watches_for_thread( - service, workspace_id=db_workspace.id, thread_id=thread.id - ) - assert found == [] - - -async def test_stop_watch_tool_reports_not_watching_when_none( - tools_use_test_session, - db_workspace: Workspace, - db_user: User, - thread: NewChatThread, -) -> None: - stop = create_stop_watch_tool( - workspace_id=db_workspace.id, - thread_id=thread.id, - auth_context=AuthContext.session(db_user), - ) - out = await stop.ainvoke({}) - assert out["status"] == "not_watching" - - -async def test_refresh_watch_tool_enqueues_a_run( - tools_use_test_session, - enqueue_spy: list[dict], - db_session: AsyncSession, - db_user: User, - db_workspace: Workspace, - thread: NewChatThread, -) -> None: - _billable(db_workspace) - await db_session.flush() - auth = AuthContext.session(db_user) - - start = create_start_watch_tool( - workspace_id=db_workspace.id, thread_id=thread.id, auth_context=auth - ) - started = await start.ainvoke({"message": "m", "cron": _CRON, "timezone": _TZ}) - - refresh = create_refresh_watch_tool( - workspace_id=db_workspace.id, thread_id=thread.id, auth_context=auth - ) - out = await refresh.ainvoke({}) - - assert out["status"] == "refreshing" - assert out["refreshed_ids"] == [started["automation_id"]] - assert len(enqueue_spy) == 1 - - -async def test_tools_error_without_thread_or_auth() -> None: - tool = create_start_watch_tool( - workspace_id=1, thread_id=None, auth_context=None - ) - out = await tool.ainvoke({"message": "m", "cron": _CRON, "timezone": _TZ}) - assert out["status"] == "error" diff --git a/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_start_watch_tool.py b/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_start_watch_tool.py deleted file mode 100644 index 3cc49b7f2..000000000 --- a/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_start_watch_tool.py +++ /dev/null @@ -1,118 +0,0 @@ -"""Unit tests for the ``start_watch`` tool on the scraping sub-agent. - -``start_watch`` binds a recurring watch to the *current* chat: it distils the -question + cadence the agent extracted and creates a ``schedule`` + -``chat_message`` automation via the watch service. These tests fake the watch -service / session so no DB is touched; they pin that the tool forwards the -current chat id + system auth and reports a clear outcome. -""" - -from __future__ import annotations - -from typing import Any -from unittest.mock import AsyncMock, MagicMock - -import pytest - -pytestmark = pytest.mark.unit - - -class _FakeSessionCM: - async def __aenter__(self) -> Any: - return MagicMock(name="session") - - async def __aexit__(self, *_exc: Any) -> bool: - return False - - -def _patch_deps(monkeypatch: pytest.MonkeyPatch, *, created: Any) -> AsyncMock: - from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools import ( - start_watch as mod, - ) - - fake_create_watch = AsyncMock(return_value=created) - monkeypatch.setattr(mod, "create_watch", fake_create_watch) - monkeypatch.setattr(mod, "AutomationService", MagicMock(return_value="svc")) - monkeypatch.setattr(mod, "async_session_maker", lambda: _FakeSessionCM()) - return fake_create_watch - - -@pytest.mark.asyncio -async def test_start_watch_binds_watch_to_current_chat( - monkeypatch: pytest.MonkeyPatch, -) -> None: - from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools import ( - start_watch as mod, - ) - - created = MagicMock(id=123) - created.name = "Watch: prices" - fake_create_watch = _patch_deps(monkeypatch, created=created) - - auth = MagicMock() - tool = mod.create_start_watch_tool(workspace_id=3, thread_id=55, auth_context=auth) - - out = await tool.ainvoke( - {"message": "what changed on prices?", "cron": "0 9 * * 1-5", "timezone": "UTC"} - ) - - assert out["status"] == "watching" - assert out["automation_id"] == 123 - - fake_create_watch.assert_awaited_once() - kwargs = fake_create_watch.await_args.kwargs - assert kwargs["workspace_id"] == 3 - assert kwargs["thread_id"] == 55 - assert kwargs["message"] == "what changed on prices?" - assert kwargs["cron"] == "0 9 * * 1-5" - assert kwargs["timezone"] == "UTC" - - # The service is constructed with the passed-through auth context. - assert mod.AutomationService.call_args.kwargs["auth"] is auth - - -@pytest.mark.asyncio -async def test_start_watch_errors_without_thread_or_auth( - monkeypatch: pytest.MonkeyPatch, -) -> None: - from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools import ( - start_watch as mod, - ) - - fake_create_watch = _patch_deps(monkeypatch, created=MagicMock(id=1)) - - no_thread = mod.create_start_watch_tool( - workspace_id=3, thread_id=None, auth_context=MagicMock() - ) - out_no_thread = await no_thread.ainvoke( - {"message": "x", "cron": "0 9 * * *", "timezone": "UTC"} - ) - assert out_no_thread["status"] == "error" - - no_auth = mod.create_start_watch_tool( - workspace_id=3, thread_id=55, auth_context=None - ) - out_no_auth = await no_auth.ainvoke( - {"message": "x", "cron": "0 9 * * *", "timezone": "UTC"} - ) - assert out_no_auth["status"] == "error" - - fake_create_watch.assert_not_awaited() - - -def test_load_tools_includes_start_watch_only_when_bindable() -> None: - from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools.index import ( - load_tools, - ) - - bindable = load_tools( - dependencies={ - "workspace_id": 3, - "thread_id": 55, - "auth_context": MagicMock(), - } - ) - assert "start_watch" in {t.name for t in bindable} - - not_bindable = load_tools(dependencies={"workspace_id": 3}) - assert "start_watch" not in {t.name for t in not_bindable} diff --git a/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_watch_control_tools.py b/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_watch_control_tools.py deleted file mode 100644 index c659e289b..000000000 --- a/surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/test_watch_control_tools.py +++ /dev/null @@ -1,119 +0,0 @@ -"""Unit tests for the ``stop_watch`` and ``refresh_watch`` chat tools. - -Both act on the *current* chat: they look up the watches bound to this thread -and stop them / run them now. The watch service + session are faked so no DB is -touched. -""" - -from __future__ import annotations - -from typing import Any -from unittest.mock import AsyncMock, MagicMock - -import pytest - -pytestmark = pytest.mark.unit - - -class _FakeSessionCM: - async def __aenter__(self) -> Any: - return MagicMock(name="session") - - async def __aexit__(self, *_exc: Any) -> bool: - return False - - -def _watch(automation_id: int) -> Any: - a = MagicMock() - a.id = automation_id - return a - - -@pytest.mark.asyncio -async def test_stop_watch_stops_every_watch_on_the_chat( - monkeypatch: pytest.MonkeyPatch, -) -> None: - from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools import ( - stop_watch as mod, - ) - - monkeypatch.setattr(mod, "AutomationService", MagicMock(return_value="svc")) - monkeypatch.setattr(mod, "async_session_maker", lambda: _FakeSessionCM()) - monkeypatch.setattr( - mod, "find_watches_for_thread", AsyncMock(return_value=[_watch(1), _watch(2)]) - ) - stop_service = AsyncMock() - monkeypatch.setattr(mod, "stop_watch_service", stop_service) - - tool = mod.create_stop_watch_tool( - workspace_id=3, thread_id=55, auth_context=MagicMock() - ) - out = await tool.ainvoke({}) - - assert out["status"] == "stopped" - assert sorted(out["stopped_ids"]) == [1, 2] - assert stop_service.await_count == 2 - - -@pytest.mark.asyncio -async def test_stop_watch_reports_when_nothing_to_stop( - monkeypatch: pytest.MonkeyPatch, -) -> None: - from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools import ( - stop_watch as mod, - ) - - monkeypatch.setattr(mod, "AutomationService", MagicMock(return_value="svc")) - monkeypatch.setattr(mod, "async_session_maker", lambda: _FakeSessionCM()) - monkeypatch.setattr(mod, "find_watches_for_thread", AsyncMock(return_value=[])) - monkeypatch.setattr(mod, "stop_watch_service", AsyncMock()) - - tool = mod.create_stop_watch_tool( - workspace_id=3, thread_id=55, auth_context=MagicMock() - ) - out = await tool.ainvoke({}) - - assert out["status"] == "not_watching" - - -@pytest.mark.asyncio -async def test_refresh_watch_runs_each_watch_now( - monkeypatch: pytest.MonkeyPatch, -) -> None: - from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools import ( - refresh_watch as mod, - ) - - monkeypatch.setattr(mod, "AutomationService", MagicMock(return_value="svc")) - monkeypatch.setattr(mod, "async_session_maker", lambda: _FakeSessionCM()) - monkeypatch.setattr( - mod, "find_watches_for_thread", AsyncMock(return_value=[_watch(7)]) - ) - run_now = AsyncMock() - monkeypatch.setattr(mod, "run_watch_now", run_now) - - tool = mod.create_refresh_watch_tool( - workspace_id=3, thread_id=55, auth_context=MagicMock() - ) - out = await tool.ainvoke({}) - - assert out["status"] == "refreshing" - assert out["refreshed_ids"] == [7] - run_now.assert_awaited_once() - assert run_now.await_args.kwargs["automation_id"] == 7 - - -def test_load_tools_includes_control_tools_when_bindable() -> None: - from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools.index import ( - load_tools, - ) - - tools = load_tools( - dependencies={ - "workspace_id": 3, - "thread_id": 55, - "auth_context": MagicMock(), - } - ) - names = {t.name for t in tools} - assert {"start_watch", "stop_watch", "refresh_watch"} <= names diff --git a/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py b/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py index 0cba7976a..dc0f35994 100644 --- a/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py +++ b/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py @@ -19,7 +19,7 @@ from app.agents.chat.multi_agent_chat.subagents.registry import ( pytestmark = pytest.mark.unit -# The full specialist roster the main agent composes from: 5 builtins + 15 +# The full specialist roster the main agent composes from: 6 builtins + 15 # connector routes. Adding/removing a specialist is a deliberate product change # and must be reflected here. _EXPECTED_SUBAGENTS = frozenset( @@ -41,9 +41,10 @@ _EXPECTED_SUBAGENTS = frozenset( "notion", "onedrive", "research", - "scraping", "slack", "teams", + "web_crawler", + "youtube", } ) From c66b2f0e0e424524e59bb369d291413c192db324 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Fri, 3 Jul 2026 11:20:22 +0200 Subject: [PATCH 54/64] refactor(automations): remove phase-6 chat watch (watch service/routes + chat_message action) --- .../automations/actions/builtin/__init__.py | 2 +- .../actions/builtin/chat_message/__init__.py | 15 -- .../builtin/chat_message/definition.py | 18 -- .../actions/builtin/chat_message/factory.py | 23 -- .../actions/builtin/chat_message/invoke.py | 59 ----- .../actions/builtin/chat_message/params.py | 21 -- .../app/automations/api/__init__.py | 4 - .../app/automations/api/chat_watch.py | 50 ---- .../app/automations/services/chat_watch.py | 152 ------------ .../actions/builtin/chat_message/__init__.py | 0 .../builtin/chat_message/test_chat_message.py | 118 ---------- .../automations/api/test_watch_routes.py | 136 ----------- .../tests/integration/automations/conftest.py | 22 -- .../automations/services/test_chat_watch.py | 217 ------------------ .../actions/builtin/chat_message/__init__.py | 0 .../builtin/chat_message/test_chat_message.py | 48 ---- .../automations/services/test_chat_watch.py | 75 ------ 17 files changed, 1 insertion(+), 959 deletions(-) delete mode 100644 surfsense_backend/app/automations/actions/builtin/chat_message/__init__.py delete mode 100644 surfsense_backend/app/automations/actions/builtin/chat_message/definition.py delete mode 100644 surfsense_backend/app/automations/actions/builtin/chat_message/factory.py delete mode 100644 surfsense_backend/app/automations/actions/builtin/chat_message/invoke.py delete mode 100644 surfsense_backend/app/automations/actions/builtin/chat_message/params.py delete mode 100644 surfsense_backend/app/automations/api/chat_watch.py delete mode 100644 surfsense_backend/app/automations/services/chat_watch.py delete mode 100644 surfsense_backend/tests/integration/automations/actions/builtin/chat_message/__init__.py delete mode 100644 surfsense_backend/tests/integration/automations/actions/builtin/chat_message/test_chat_message.py delete mode 100644 surfsense_backend/tests/integration/automations/api/test_watch_routes.py delete mode 100644 surfsense_backend/tests/integration/automations/services/test_chat_watch.py delete mode 100644 surfsense_backend/tests/unit/automations/actions/builtin/chat_message/__init__.py delete mode 100644 surfsense_backend/tests/unit/automations/actions/builtin/chat_message/test_chat_message.py delete mode 100644 surfsense_backend/tests/unit/automations/services/test_chat_watch.py diff --git a/surfsense_backend/app/automations/actions/builtin/__init__.py b/surfsense_backend/app/automations/actions/builtin/__init__.py index 19a93dbfc..f3d21a044 100644 --- a/surfsense_backend/app/automations/actions/builtin/__init__.py +++ b/surfsense_backend/app/automations/actions/builtin/__init__.py @@ -2,4 +2,4 @@ from __future__ import annotations -from . import agent_task, chat_message # noqa: F401 +from . import agent_task # noqa: F401 diff --git a/surfsense_backend/app/automations/actions/builtin/chat_message/__init__.py b/surfsense_backend/app/automations/actions/builtin/chat_message/__init__.py deleted file mode 100644 index 6ca297493..000000000 --- a/surfsense_backend/app/automations/actions/builtin/chat_message/__init__.py +++ /dev/null @@ -1,15 +0,0 @@ -"""``chat_message`` action: post one turn into an existing chat thread. - -Imports ``definition`` for its side-effect (self-registration on the actions -registry) and re-exports ``build_handler`` for direct consumers. -""" - -from __future__ import annotations - -from .factory import build_handler -from .params import ChatMessageActionParams - -__all__ = ["ChatMessageActionParams", "build_handler"] - -# Side-effect: register on the actions store. -from . import definition # noqa: F401 diff --git a/surfsense_backend/app/automations/actions/builtin/chat_message/definition.py b/surfsense_backend/app/automations/actions/builtin/chat_message/definition.py deleted file mode 100644 index 5652d28c5..000000000 --- a/surfsense_backend/app/automations/actions/builtin/chat_message/definition.py +++ /dev/null @@ -1,18 +0,0 @@ -"""``chat_message`` ``ActionDefinition`` registration.""" - -from __future__ import annotations - -from ...store import register_action -from ...types import ActionDefinition -from .factory import build_handler -from .params import ChatMessageActionParams - -CHAT_MESSAGE_ACTION = ActionDefinition( - type="chat_message", - name="Chat message", - description="Post a message into an existing chat thread and run one durable, persisted agent turn.", - params_model=ChatMessageActionParams, - build_handler=build_handler, -) - -register_action(CHAT_MESSAGE_ACTION) diff --git a/surfsense_backend/app/automations/actions/builtin/chat_message/factory.py b/surfsense_backend/app/automations/actions/builtin/chat_message/factory.py deleted file mode 100644 index 494ea5ffc..000000000 --- a/surfsense_backend/app/automations/actions/builtin/chat_message/factory.py +++ /dev/null @@ -1,23 +0,0 @@ -"""Bind ``ActionContext`` to a callable that runs one ``chat_message`` step.""" - -from __future__ import annotations - -from typing import Any - -from ...types import ActionContext, ActionHandler -from .invoke import run_chat_message -from .params import ChatMessageActionParams - - -def build_handler(ctx: ActionContext) -> ActionHandler: - """Return a handler closure that validates params and posts the turn.""" - - async def handle(params: dict[str, Any]) -> dict[str, Any]: - validated = ChatMessageActionParams.model_validate(params) - return await run_chat_message( - ctx=ctx, - thread_id=validated.thread_id, - message=validated.message, - ) - - return handle diff --git a/surfsense_backend/app/automations/actions/builtin/chat_message/invoke.py b/surfsense_backend/app/automations/actions/builtin/chat_message/invoke.py deleted file mode 100644 index 596ccd60c..000000000 --- a/surfsense_backend/app/automations/actions/builtin/chat_message/invoke.py +++ /dev/null @@ -1,59 +0,0 @@ -"""Run one ``chat_message`` step by draining ``stream_new_chat``. - -Reuses the interactive chat turn: it persists the messages and advances the -durable checkpointer for the thread, so a scheduled tick shares the same -conversation memory as a user turn. The SSE frames have no client, so drain them. -""" - -from __future__ import annotations - -from typing import Any - -from app.auth.context import AuthContext -from app.db import ChatVisibility, User -from app.services.chat_session_state_service import get_session_state -from app.tasks.chat.streaming.flows.new_chat.orchestrator import stream_new_chat - -from ...types import ActionContext - - -async def run_chat_message( - *, - ctx: ActionContext, - thread_id: int, - message: str, -) -> dict[str, Any]: - """Post ``message`` into ``thread_id`` and run one durable agent turn. - - Skips when a turn is already in flight on the thread, so a slow tick can't - overlap the next scheduled fire. - """ - state = await get_session_state(ctx.session, thread_id) - if state is not None and state.ai_responding_to_user_id is not None: - return {"thread_id": thread_id, "frames": 0, "skipped": "in_flight"} - - user_id = str(ctx.creator_user_id) if ctx.creator_user_id else None - auth_context: AuthContext | None = None - if ctx.creator_user_id: - user = await ctx.session.get(User, ctx.creator_user_id) - if user is not None: - auth_context = AuthContext.system(user, source="automation") - - llm_config_id = ctx.chat_model_id if ctx.chat_model_id is not None else -1 - request_id = f"automation:{ctx.run_id}:{ctx.step_id}" - - frames = 0 - async for _sse in stream_new_chat( - user_query=message, - workspace_id=ctx.workspace_id, - chat_id=thread_id, - user_id=user_id, - llm_config_id=llm_config_id, - needs_history_bootstrap=False, - thread_visibility=ChatVisibility.PRIVATE, - auth_context=auth_context, - request_id=request_id, - ): - frames += 1 - - return {"thread_id": thread_id, "frames": frames, "skipped": None} diff --git a/surfsense_backend/app/automations/actions/builtin/chat_message/params.py b/surfsense_backend/app/automations/actions/builtin/chat_message/params.py deleted file mode 100644 index 61768bc13..000000000 --- a/surfsense_backend/app/automations/actions/builtin/chat_message/params.py +++ /dev/null @@ -1,21 +0,0 @@ -"""``ChatMessageActionParams`` — params for the ``chat_message`` action type.""" - -from __future__ import annotations - -from pydantic import BaseModel, ConfigDict, Field - - -class ChatMessageActionParams(BaseModel): - """Post one turn into an existing chat thread from an automation step.""" - - model_config = ConfigDict(extra="forbid") - - thread_id: int = Field( - ..., - description="NewChatThread id (the LangGraph thread) to post the turn into.", - ) - message: str = Field( - ..., - min_length=1, - description="Message sent as the user turn; rendered at execute time.", - ) diff --git a/surfsense_backend/app/automations/api/__init__.py b/surfsense_backend/app/automations/api/__init__.py index fa9f1ec05..a18e91a95 100644 --- a/surfsense_backend/app/automations/api/__init__.py +++ b/surfsense_backend/app/automations/api/__init__.py @@ -5,14 +5,10 @@ from __future__ import annotations from fastapi import APIRouter from .automation import router as automation_router -from .chat_watch import router as watch_router from .run import router as run_router from .trigger import router as trigger_router router = APIRouter() -# Before automation_router so the literal ``/automations/watches`` path is not -# shadowed by the ``/automations/{automation_id}`` route. -router.include_router(watch_router) router.include_router(automation_router) router.include_router(trigger_router) router.include_router(run_router) diff --git a/surfsense_backend/app/automations/api/chat_watch.py b/surfsense_backend/app/automations/api/chat_watch.py deleted file mode 100644 index 1b5348f1a..000000000 --- a/surfsense_backend/app/automations/api/chat_watch.py +++ /dev/null @@ -1,50 +0,0 @@ -"""HTTP routes for chat watches (automations bound to a chat thread). - -A watch is an automation whose plan re-posts a question into a chat on a -schedule. These routes let the chat UI show watch state and controls: - -* ``GET /automations/watches`` — the watches bound to a thread (is-watched + list). -* ``POST /automations/{automation_id}/run`` — run a watch now (manual refresh). - -Stopping a watch is a plain ``DELETE /automations/{automation_id}``. -""" - -from __future__ import annotations - -from fastapi import APIRouter, Depends, Query, status - -from app.automations.schemas.api import AutomationList, AutomationSummary, RunSummary -from app.automations.services import AutomationService, get_automation_service -from app.automations.services.chat_watch import find_watches_for_thread, run_watch_now - -router = APIRouter() - - -@router.get("/automations/watches", response_model=AutomationList) -async def list_watches( - workspace_id: int = Query(...), - thread_id: int = Query(...), - service: AutomationService = Depends(get_automation_service), -) -> AutomationList: - """List the watches bound to a chat thread (empty when it isn't watched).""" - watches = await find_watches_for_thread( - service, workspace_id=workspace_id, thread_id=thread_id - ) - return AutomationList( - items=[AutomationSummary.model_validate(a) for a in watches], - total=len(watches), - ) - - -@router.post( - "/automations/{automation_id}/run", - response_model=RunSummary, - status_code=status.HTTP_202_ACCEPTED, -) -async def run_watch( - automation_id: int, - service: AutomationService = Depends(get_automation_service), -) -> RunSummary: - """Enqueue an immediate run of a watch (manual refresh).""" - run = await run_watch_now(service, automation_id=automation_id) - return RunSummary.model_validate(run) diff --git a/surfsense_backend/app/automations/services/chat_watch.py b/surfsense_backend/app/automations/services/chat_watch.py deleted file mode 100644 index 5f72ca98e..000000000 --- a/surfsense_backend/app/automations/services/chat_watch.py +++ /dev/null @@ -1,152 +0,0 @@ -"""A chat watch is an automation bound to a chat: a ``schedule`` trigger firing -a single ``chat_message`` step that re-posts the question into the same thread. - -Starting a watch creates that automation; stopping deletes it. Whether a chat -is watched is derived from the plan (``plan_targets_thread``), not a stored flag. -""" - -from __future__ import annotations - -from typing import Any - -from fastapi import HTTPException - -from app.automations.persistence.enums.trigger_type import TriggerType -from app.automations.persistence.models.automation import Automation -from app.automations.persistence.models.run import AutomationRun -from app.automations.persistence.models.trigger import AutomationTrigger -from app.automations.schemas.api import AutomationCreate, TriggerCreate -from app.automations.schemas.definition import AutomationDefinition, PlanStep -from app.automations.services.automation import AutomationService - -WATCH_ACTION_TYPE = "chat_message" -_WATCH_STEP_ID = "watch" -_NAME_MAX = 200 -# Watches per chat are few; one generous page finds them all. -_WATCH_SCAN_LIMIT = 500 - - -def _derive_name(message: str) -> str: - """A short, human name for the watch from the question it re-asks.""" - condensed = " ".join(message.split()) - label = condensed[:80].rstrip() - return f"Watch: {label}" if label else "Watch" - - -def _build_watch_payload( - *, - workspace_id: int, - thread_id: int, - message: str, - cron: str, - timezone: str, - name: str | None, - description: str | None, -) -> AutomationCreate: - watch_name = (name or _derive_name(message))[:_NAME_MAX] - return AutomationCreate( - workspace_id=workspace_id, - name=watch_name, - description=description, - definition=AutomationDefinition( - name=watch_name, - plan=[ - PlanStep( - step_id=_WATCH_STEP_ID, - action=WATCH_ACTION_TYPE, - params={"thread_id": thread_id, "message": message}, - ) - ], - ), - triggers=[ - TriggerCreate( - type=TriggerType.SCHEDULE, - params={"cron": cron, "timezone": timezone}, - enabled=True, - ) - ], - ) - - -async def create_watch( - service: AutomationService, - *, - workspace_id: int, - thread_id: int, - message: str, - cron: str, - timezone: str, - name: str | None = None, - description: str | None = None, -) -> Automation: - """Bind a schedule + chat_message watch automation to ``thread_id``.""" - payload = _build_watch_payload( - workspace_id=workspace_id, - thread_id=thread_id, - message=message, - cron=cron, - timezone=timezone, - name=name, - description=description, - ) - return await service.create(payload) - - -async def stop_watch(service: AutomationService, *, automation_id: int) -> None: - """Stop a watch by deleting its automation; the chat reverts to normal.""" - await service.delete(automation_id) - - -def plan_targets_thread(definition: dict[str, Any] | None, thread_id: int) -> bool: - """Whether an automation's plan re-posts into ``thread_id`` (i.e. is a watch). - - Pure predicate over the persisted ``definition`` JSON so callers can filter - a workspace's automations without a DB round-trip per row. - """ - for step in (definition or {}).get("plan", []): - if step.get("action") != WATCH_ACTION_TYPE: - continue - if (step.get("params") or {}).get("thread_id") == thread_id: - return True - return False - - -async def find_watches_for_thread( - service: AutomationService, - *, - workspace_id: int, - thread_id: int, -) -> list[Automation]: - """Return the workspace automations that watch ``thread_id`` (i.e. its watches).""" - automations, _total = await service.list( - workspace_id=workspace_id, limit=_WATCH_SCAN_LIMIT, offset=0 - ) - return [a for a in automations if plan_targets_thread(a.definition, thread_id)] - - -def schedule_trigger(automation: Automation) -> AutomationTrigger | None: - """The automation's ``schedule`` trigger row, or ``None``.""" - for trigger in automation.triggers: - if trigger.type == TriggerType.SCHEDULE: - return trigger - return None - - -async def run_watch_now( - service: AutomationService, - *, - automation_id: int, -) -> AutomationRun: - """Enqueue an immediate run of a watch (a manual refresh).""" - # Lazy: launch_run pulls in the automation task graph, which imports - # multi_agent_chat; this module is imported from there, so a top-level - # import would cycle. - from app.automations.dispatch import launch as launch_mod - - automation = await service.get(automation_id) - trigger = schedule_trigger(automation) - if trigger is None: - raise HTTPException( - status_code=422, detail="watch has no schedule trigger to run" - ) - return await launch_mod.launch_run(session=service.session, trigger=trigger) diff --git a/surfsense_backend/tests/integration/automations/actions/builtin/chat_message/__init__.py b/surfsense_backend/tests/integration/automations/actions/builtin/chat_message/__init__.py deleted file mode 100644 index e69de29bb..000000000 diff --git a/surfsense_backend/tests/integration/automations/actions/builtin/chat_message/test_chat_message.py b/surfsense_backend/tests/integration/automations/actions/builtin/chat_message/test_chat_message.py deleted file mode 100644 index 7ae5218ab..000000000 --- a/surfsense_backend/tests/integration/automations/actions/builtin/chat_message/test_chat_message.py +++ /dev/null @@ -1,118 +0,0 @@ -"""Integration tests for the ``chat_message`` action's concurrency guard. - -The guard reads real ``ChatSessionState`` (seeded via the production -``set_ai_responding`` helper) to decide whether a scheduled watch tick may run. -This proves the skip decision against real persistence — the part a fake -``get_session_state`` could silently get wrong. - -The turn's actual work (``stream_new_chat``) resolves an LLM from the DB and -runs the agent; that end-to-end drain is a running-stack concern. Here it is -replaced by a spy so we can assert the guard's two outcomes: skip without -streaming when a turn is in flight, and stream (with system auth) when not. -""" - -from __future__ import annotations - -from typing import Any - -import pytest -import pytest_asyncio -from sqlalchemy.ext.asyncio import AsyncSession - -from app.automations.actions.builtin.chat_message import invoke as invoke_mod -from app.automations.actions.types import ActionContext -from app.db import ChatVisibility, NewChatThread, User, Workspace -from app.services.chat_session_state_service import ( - clear_ai_responding, - set_ai_responding, -) - -pytestmark = pytest.mark.integration - - -@pytest_asyncio.fixture -async def thread( - db_session: AsyncSession, db_user: User, db_workspace: Workspace -) -> NewChatThread: - row = NewChatThread( - title="Watched chat", - workspace_id=db_workspace.id, - created_by_id=db_user.id, - visibility=ChatVisibility.PRIVATE, - ) - db_session.add(row) - await db_session.flush() - return row - - -def _ctx(session: AsyncSession, *, workspace_id: int, creator_id) -> ActionContext: - return ActionContext( - session=session, - run_id=1, - step_id="watch", - workspace_id=workspace_id, - creator_user_id=creator_id, - chat_model_id=1, - ) - - -async def test_skips_without_streaming_when_a_turn_is_in_flight( - db_session: AsyncSession, - db_user: User, - db_workspace: Workspace, - thread: NewChatThread, - monkeypatch: pytest.MonkeyPatch, -) -> None: - await set_ai_responding(db_session, thread.id, db_user.id) - - streamed = {"called": False} - - async def _spy(**_kwargs: Any): - streamed["called"] = True - yield "data: x\n\n" - - monkeypatch.setattr(invoke_mod, "stream_new_chat", _spy) - - out = await invoke_mod.run_chat_message( - ctx=_ctx(db_session, workspace_id=db_workspace.id, creator_id=db_user.id), - thread_id=thread.id, - message="tick", - ) - - assert out["skipped"] == "in_flight" - assert out["frames"] == 0 - assert streamed["called"] is False - - -async def test_streams_under_system_auth_when_no_turn_in_flight( - db_session: AsyncSession, - db_user: User, - db_workspace: Workspace, - thread: NewChatThread, - monkeypatch: pytest.MonkeyPatch, -) -> None: - # Ensure the thread has no in-flight turn. - await clear_ai_responding(db_session, thread.id) - - captured: dict[str, Any] = {} - - async def _spy(**kwargs: Any): - captured.update(kwargs) - for frame in ("data: a\n\n", "data: b\n\n"): - yield frame - - monkeypatch.setattr(invoke_mod, "stream_new_chat", _spy) - - out = await invoke_mod.run_chat_message( - ctx=_ctx(db_session, workspace_id=db_workspace.id, creator_id=db_user.id), - thread_id=thread.id, - message="what changed?", - ) - - assert out["skipped"] is None - assert out["frames"] == 2 - assert captured["chat_id"] == thread.id - assert captured["user_query"] == "what changed?" - assert captured["user_id"] == str(db_user.id) - auth = captured["auth_context"] - assert auth is not None and auth.method == "system" and auth.source == "automation" diff --git a/surfsense_backend/tests/integration/automations/api/test_watch_routes.py b/surfsense_backend/tests/integration/automations/api/test_watch_routes.py deleted file mode 100644 index e5da2953d..000000000 --- a/surfsense_backend/tests/integration/automations/api/test_watch_routes.py +++ /dev/null @@ -1,136 +0,0 @@ -"""Integration tests for the watch HTTP routes (real app, DB, auth). - -Watches are seeded through the real service and read back through the mounted -endpoints, exercising query validation, auth, and response mapping. Run-now is -asserted without a broker by spying the Celery enqueue while still persisting a -real PENDING run. -""" - -from __future__ import annotations - -import httpx -import pytest -import pytest_asyncio -from sqlalchemy import select -from sqlalchemy.ext.asyncio import AsyncSession - -from app.auth.context import AuthContext -from app.automations.persistence.enums.run_status import RunStatus -from app.automations.persistence.models.run import AutomationRun -from app.automations.services.automation import AutomationService -from app.automations.services.chat_watch import create_watch -from app.db import ChatVisibility, NewChatThread, User, Workspace - -pytestmark = pytest.mark.integration - -BASE = "/api/v1/automations" -_CRON = "0 9 * * 1-5" -_TZ = "UTC" - - -def _billable(workspace: Workspace) -> None: - workspace.chat_model_id = 1 - workspace.image_gen_model_id = 1 - workspace.vision_model_id = 1 - - -@pytest_asyncio.fixture -async def thread( - db_session: AsyncSession, db_user: User, db_workspace: Workspace -) -> NewChatThread: - row = NewChatThread( - title="Watched chat", - workspace_id=db_workspace.id, - created_by_id=db_user.id, - visibility=ChatVisibility.PRIVATE, - ) - db_session.add(row) - await db_session.flush() - return row - - -async def _seed_watch( - db_session: AsyncSession, - db_user: User, - *, - workspace_id: int, - thread_id: int, -): - service = AutomationService(session=db_session, auth=AuthContext.session(db_user)) - return await create_watch( - service, - workspace_id=workspace_id, - thread_id=thread_id, - message="what changed?", - cron=_CRON, - timezone=_TZ, - ) - - -async def test_list_watches_returns_the_threads_watch( - client: httpx.AsyncClient, - db_session: AsyncSession, - db_user: User, - db_workspace: Workspace, - thread: NewChatThread, -) -> None: - _billable(db_workspace) - await db_session.flush() - watch = await _seed_watch( - db_session, db_user, workspace_id=db_workspace.id, thread_id=thread.id - ) - - resp = await client.get( - f"{BASE}/watches", - params={"workspace_id": db_workspace.id, "thread_id": thread.id}, - ) - - assert resp.status_code == 200 - body = resp.json() - assert body["total"] == 1 - assert body["items"][0]["id"] == watch.id - - -async def test_list_watches_empty_when_thread_not_watched( - client: httpx.AsyncClient, - db_workspace: Workspace, - thread: NewChatThread, -) -> None: - resp = await client.get( - f"{BASE}/watches", - params={"workspace_id": db_workspace.id, "thread_id": thread.id}, - ) - assert resp.status_code == 200 - assert resp.json()["total"] == 0 - - -async def test_run_watch_now_enqueues_and_persists_pending_run( - client: httpx.AsyncClient, - enqueue_spy: list[dict], - db_session: AsyncSession, - db_user: User, - db_workspace: Workspace, - thread: NewChatThread, -) -> None: - _billable(db_workspace) - await db_session.flush() - watch = await _seed_watch( - db_session, db_user, workspace_id=db_workspace.id, thread_id=thread.id - ) - - resp = await client.post(f"{BASE}/{watch.id}/run") - - assert resp.status_code == 202 - assert len(enqueue_spy) == 1 - - runs = ( - ( - await db_session.execute( - select(AutomationRun).where(AutomationRun.automation_id == watch.id) - ) - ) - .scalars() - .all() - ) - assert len(runs) == 1 - assert runs[0].status == RunStatus.PENDING diff --git a/surfsense_backend/tests/integration/automations/conftest.py b/surfsense_backend/tests/integration/automations/conftest.py index 1d3e91b07..62420d93d 100644 --- a/surfsense_backend/tests/integration/automations/conftest.py +++ b/surfsense_backend/tests/integration/automations/conftest.py @@ -5,9 +5,6 @@ behavior runs against real Postgres and rolls back at test end: * ``client`` — httpx over ASGI with ``get_async_session``/``get_auth_context`` overridden to the test session + owner. -* ``tools_use_test_session`` — the watch tools open their own - ``async_session_maker()``; repoint it at ``db_session`` so they join the - test transaction (and can see the fixture's uncommitted membership/models). * ``enqueue_spy`` — capture ``automation_run_execute.apply_async`` so run-now can be asserted without a Redis broker. """ @@ -15,7 +12,6 @@ behavior runs against real Postgres and rolls back at test end: from __future__ import annotations from collections.abc import AsyncGenerator -from contextlib import asynccontextmanager import httpx import pytest @@ -55,24 +51,6 @@ async def client( app.dependency_overrides.update(previous) -@pytest.fixture -def tools_use_test_session(monkeypatch, db_session: AsyncSession) -> None: - """Make the watch tools' ``async_session_maker()`` yield the test session.""" - - @asynccontextmanager - async def _session_cm(): - yield db_session # owned by the outer fixture; do not close - - from app.agents.chat.multi_agent_chat.subagents.builtins.scraping.tools import ( - refresh_watch, - start_watch, - stop_watch, - ) - - for module in (start_watch, stop_watch, refresh_watch): - monkeypatch.setattr(module, "async_session_maker", _session_cm) - - @pytest.fixture def enqueue_spy(monkeypatch) -> list[dict]: """Capture Celery enqueues so run-now needs no broker.""" diff --git a/surfsense_backend/tests/integration/automations/services/test_chat_watch.py b/surfsense_backend/tests/integration/automations/services/test_chat_watch.py deleted file mode 100644 index 5c5b3abb8..000000000 --- a/surfsense_backend/tests/integration/automations/services/test_chat_watch.py +++ /dev/null @@ -1,217 +0,0 @@ -"""Integration tests for the chat watch service against real Postgres. - -A watch is an automation bound to a chat. These exercise the real -``AutomationService`` (RBAC, the automation model-billing gate, and the -definition JSON round-trip) rather than a fake, so the behaviors the unit -layer can't see are proven end to end: - -* creating a watch persists a ``schedule`` + ``chat_message`` automation and - it reads back as the thread's watch; -* the billing gate rejects a watch when the workspace's models aren't - billable (Auto/free) — the exact failure a fake service would hide; -* finding watches is thread-scoped; stopping one deletes it; -* run-now refuses an automation that has no schedule trigger. -""" - -from __future__ import annotations - -import pytest -import pytest_asyncio -from fastapi import HTTPException -from sqlalchemy.ext.asyncio import AsyncSession - -from app.auth.context import AuthContext -from app.automations.persistence.enums.trigger_type import TriggerType -from app.automations.persistence.models.automation import Automation -from app.automations.persistence.models.trigger import AutomationTrigger -from app.automations.services.automation import AutomationService -from app.automations.services.chat_watch import ( - create_watch, - find_watches_for_thread, - run_watch_now, - stop_watch, -) -from app.db import ChatVisibility, NewChatThread, User, Workspace - -pytestmark = pytest.mark.integration - -_CRON = "0 9 * * 1-5" -_TZ = "UTC" - - -def _billable(workspace: Workspace) -> None: - """Give the workspace BYOK (positive id) models so automations are billable.""" - workspace.chat_model_id = 1 - workspace.image_gen_model_id = 1 - workspace.vision_model_id = 1 - - -@pytest_asyncio.fixture -async def service(db_session: AsyncSession, db_user: User) -> AutomationService: - return AutomationService(session=db_session, auth=AuthContext.session(db_user)) - - -@pytest_asyncio.fixture -async def thread( - db_session: AsyncSession, db_user: User, db_workspace: Workspace -) -> NewChatThread: - row = NewChatThread( - title="Watched chat", - workspace_id=db_workspace.id, - created_by_id=db_user.id, - visibility=ChatVisibility.PRIVATE, - ) - db_session.add(row) - await db_session.flush() - return row - - -async def test_create_watch_persists_and_reads_back_as_the_threads_watch( - service: AutomationService, - db_session: AsyncSession, - db_workspace: Workspace, - thread: NewChatThread, -) -> None: - _billable(db_workspace) - await db_session.flush() - thread_id = thread.id - - created = await create_watch( - service, - workspace_id=db_workspace.id, - thread_id=thread_id, - message="what changed on the pricing page?", - cron=_CRON, - timezone=_TZ, - ) - - assert created.id is not None - plan = created.definition["plan"] - assert len(plan) == 1 - assert plan[0]["action"] == "chat_message" - assert plan[0]["params"] == { - "thread_id": thread_id, - "message": "what changed on the pricing page?", - } - schedule = [t for t in created.triggers if t.type == TriggerType.SCHEDULE] - assert len(schedule) == 1 - assert schedule[0].params["cron"] == _CRON - - found = await find_watches_for_thread( - service, workspace_id=db_workspace.id, thread_id=thread_id - ) - assert [a.id for a in found] == [created.id] - - -async def test_create_watch_rejected_when_workspace_models_not_billable( - service: AutomationService, - db_workspace: Workspace, - thread: NewChatThread, -) -> None: - # Default workspace has no model prefs (Auto/None) -> not billable. - with pytest.raises(HTTPException) as exc: - await create_watch( - service, - workspace_id=db_workspace.id, - thread_id=thread.id, - message="track it", - cron=_CRON, - timezone=_TZ, - ) - assert exc.value.status_code == 422 - - -async def test_find_watches_is_scoped_to_the_thread( - service: AutomationService, - db_session: AsyncSession, - db_workspace: Workspace, - db_user: User, -) -> None: - _billable(db_workspace) - await db_session.flush() - - thread_a = NewChatThread( - title="A", workspace_id=db_workspace.id, created_by_id=db_user.id, - visibility=ChatVisibility.PRIVATE, - ) - thread_b = NewChatThread( - title="B", workspace_id=db_workspace.id, created_by_id=db_user.id, - visibility=ChatVisibility.PRIVATE, - ) - db_session.add_all([thread_a, thread_b]) - await db_session.flush() - - watch_a = await create_watch( - service, workspace_id=db_workspace.id, thread_id=thread_a.id, - message="a", cron=_CRON, timezone=_TZ, - ) - await create_watch( - service, workspace_id=db_workspace.id, thread_id=thread_b.id, - message="b", cron=_CRON, timezone=_TZ, - ) - - found = await find_watches_for_thread( - service, workspace_id=db_workspace.id, thread_id=thread_a.id - ) - assert [a.id for a in found] == [watch_a.id] - - -async def test_stop_watch_deletes_and_leaves_the_thread_unwatched( - service: AutomationService, - db_session: AsyncSession, - db_workspace: Workspace, - thread: NewChatThread, -) -> None: - _billable(db_workspace) - await db_session.flush() - thread_id = thread.id - - created = await create_watch( - service, workspace_id=db_workspace.id, thread_id=thread_id, - message="m", cron=_CRON, timezone=_TZ, - ) - - await stop_watch(service, automation_id=created.id) - - found = await find_watches_for_thread( - service, workspace_id=db_workspace.id, thread_id=thread_id - ) - assert found == [] - with pytest.raises(HTTPException) as exc: - await service.get(created.id) - assert exc.value.status_code == 404 - - -async def test_run_watch_now_rejects_automation_without_schedule_trigger( - service: AutomationService, - db_session: AsyncSession, - db_workspace: Workspace, - db_user: User, - thread: NewChatThread, -) -> None: - # Seed a watch-shaped automation whose only trigger is non-schedule. - automation = Automation( - workspace_id=db_workspace.id, - created_by_user_id=db_user.id, - name="Watch: broken", - definition={ - "name": "Watch: broken", - "plan": [ - { - "step_id": "watch", - "action": "chat_message", - "params": {"thread_id": thread.id, "message": "m"}, - } - ], - }, - version=1, - ) - automation.triggers.append( - AutomationTrigger(type=TriggerType.EVENT, params={}, enabled=True) - ) - db_session.add(automation) - await db_session.flush() - - with pytest.raises(HTTPException) as exc: - await run_watch_now(service, automation_id=automation.id) - assert exc.value.status_code == 422 diff --git a/surfsense_backend/tests/unit/automations/actions/builtin/chat_message/__init__.py b/surfsense_backend/tests/unit/automations/actions/builtin/chat_message/__init__.py deleted file mode 100644 index e69de29bb..000000000 diff --git a/surfsense_backend/tests/unit/automations/actions/builtin/chat_message/test_chat_message.py b/surfsense_backend/tests/unit/automations/actions/builtin/chat_message/test_chat_message.py deleted file mode 100644 index b70c7bf35..000000000 --- a/surfsense_backend/tests/unit/automations/actions/builtin/chat_message/test_chat_message.py +++ /dev/null @@ -1,48 +0,0 @@ -"""Pure-logic unit tests for the ``chat_message`` action. - -Only the infra-free contract lives here: the action self-registers, and its -params model requires ``thread_id`` + ``message`` and forbids extras. The -run-time behavior (the in-flight guard against real ``ChatSessionState`` and -streaming under system auth) is proven in -``tests/integration/automations/actions/builtin/chat_message/test_chat_message.py``, -against real persistence rather than a faked ``get_session_state``/``stream_new_chat``. -""" - -from __future__ import annotations - -import pytest - -pytestmark = pytest.mark.unit - - -def test_chat_message_action_is_registered_after_package_import() -> None: - import app.automations # noqa: F401 (force side-effect registrations) - from app.automations.actions.store import get_action - - definition = get_action("chat_message") - - assert definition is not None - assert definition.type == "chat_message" - - -def test_params_require_thread_id_and_message_and_forbid_extra() -> None: - from pydantic import ValidationError - - from app.automations.actions.builtin.chat_message.params import ( - ChatMessageActionParams, - ) - - ok = ChatMessageActionParams.model_validate({"thread_id": 7, "message": "hi"}) - assert ok.thread_id == 7 - assert ok.message == "hi" - - with pytest.raises(ValidationError): - ChatMessageActionParams.model_validate({"thread_id": 7, "message": ""}) - - with pytest.raises(ValidationError): - ChatMessageActionParams.model_validate({"thread_id": 7}) - - with pytest.raises(ValidationError): - ChatMessageActionParams.model_validate( - {"thread_id": 7, "message": "hi", "surprise": True} - ) diff --git a/surfsense_backend/tests/unit/automations/services/test_chat_watch.py b/surfsense_backend/tests/unit/automations/services/test_chat_watch.py deleted file mode 100644 index 16b3b80e2..000000000 --- a/surfsense_backend/tests/unit/automations/services/test_chat_watch.py +++ /dev/null @@ -1,75 +0,0 @@ -"""Pure-logic unit tests for the chat watch service. - -Only the deterministic, infra-free pieces live here: the plan predicate that -decides whether an automation watches a thread, and the schedule-trigger -picker. Everything that touches ``AutomationService``/Postgres (create, find, -stop, run-now) is proven against real infra in -``tests/integration/automations/services/test_chat_watch.py`` — faking the -service we own would only assert wiring and hide the model-billing gate, -RBAC, and JSON round-trip that those calls really go through. -""" - -from __future__ import annotations - -from typing import Any - -import pytest - -from app.automations.persistence.enums.trigger_type import TriggerType -from app.automations.persistence.models.trigger import AutomationTrigger -from app.automations.services.chat_watch import ( - WATCH_ACTION_TYPE, - plan_targets_thread, - schedule_trigger, -) - -pytestmark = pytest.mark.unit - - -def _definition_with_step(action: str, params: dict[str, Any]) -> dict[str, Any]: - return {"plan": [{"step_id": "s", "action": action, "params": params}]} - - -def test_plan_targets_thread_true_for_matching_chat_message_step() -> None: - definition = _definition_with_step("chat_message", {"thread_id": 55, "message": "x"}) - assert plan_targets_thread(definition, 55) is True - assert WATCH_ACTION_TYPE == "chat_message" - - -def test_plan_targets_thread_false_for_other_thread_or_action() -> None: - assert ( - plan_targets_thread( - _definition_with_step("chat_message", {"thread_id": 999}), 55 - ) - is False - ) - assert ( - plan_targets_thread( - _definition_with_step("agent_task", {"thread_id": 55}), 55 - ) - is False - ) - assert plan_targets_thread({}, 55) is False - assert plan_targets_thread(None, 55) is False - - -class _Automation: - """Minimal stand-in carrying only the ``triggers`` list the picker reads.""" - - def __init__(self, triggers: list[AutomationTrigger]) -> None: - self.triggers = triggers - - -def _trigger(type_: TriggerType) -> AutomationTrigger: - return AutomationTrigger(type=type_, params={}, enabled=True) - - -def test_schedule_trigger_picks_the_schedule_row() -> None: - schedule = _trigger(TriggerType.SCHEDULE) - automation = _Automation([_trigger(TriggerType.EVENT), schedule]) - assert schedule_trigger(automation) is schedule - - -def test_schedule_trigger_returns_none_without_a_schedule() -> None: - automation = _Automation([_trigger(TriggerType.EVENT)]) - assert schedule_trigger(automation) is None From 62cb0efb44db026eae62b86efef750366dfbdaed Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Fri, 3 Jul 2026 11:38:56 +0200 Subject: [PATCH 55/64] refactor(youtube): drop duplicate POST /youtube/scrape+comments routes (use capability doors) --- .../app/routes/youtube_routes.py | 45 ------------------- 1 file changed, 45 deletions(-) diff --git a/surfsense_backend/app/routes/youtube_routes.py b/surfsense_backend/app/routes/youtube_routes.py index 76b489227..c9d958aa8 100644 --- a/surfsense_backend/app/routes/youtube_routes.py +++ b/surfsense_backend/app/routes/youtube_routes.py @@ -9,12 +9,6 @@ from fastapi import APIRouter, Depends, HTTPException, Query from scrapling.fetchers import AsyncFetcher from app.auth.context import AuthContext -from app.proprietary.platforms.youtube import ( - YouTubeCommentsInput, - YouTubeScrapeInput, - scrape_comments, - scrape_youtube, -) from app.users import require_session_context from app.utils.proxy import get_proxy_url @@ -32,45 +26,6 @@ _INNERTUBE_CLIENT = { } -@router.post("/youtube/scrape") -async def scrape_youtube_route( - payload: YouTubeScrapeInput, - _auth: AuthContext = Depends(require_session_context), -) -> list[dict]: - """Scrape public YouTube data (search / video / channel / playlist / shorts). - - Apify YouTube Scraper-compatible input/output. The scrape runs inline and is - bounded only by the request's own ``maxResults`` / ``maxResultsShorts`` / - ``maxResultStreams`` — no separate per-request ceiling. - """ - try: - return await scrape_youtube(payload) - except Exception as e: - logger.error("YouTube scrape failed: %s", e) - raise HTTPException( - status_code=502, detail=f"YouTube scrape failed: {e!s}" - ) from e - - -@router.post("/youtube/comments") -async def scrape_comments_route( - payload: YouTubeCommentsInput, - _auth: AuthContext = Depends(require_session_context), -) -> list[dict]: - """Scrape YouTube comments (+ replies) for the given video URLs. - - Apify "YouTube Comments Scraper"-compatible. Runs inline and is bounded only - by the request's own ``maxComments`` — no separate per-request ceiling. - """ - try: - return await scrape_comments(payload) - except Exception as e: - logger.error("YouTube comments scrape failed: %s", e) - raise HTTPException( - status_code=502, detail=f"YouTube comments scrape failed: {e!s}" - ) from e - - @router.get("/youtube/playlist-videos") async def get_playlist_videos( url: str = Query(..., description="YouTube playlist URL"), From e3ed3b2be391bc3ffd89b9c5c422c8156b9e2c04 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Fri, 3 Jul 2026 11:43:26 +0200 Subject: [PATCH 56/64] refactor(subagents): remove dormant research subagent --- .../subagents/builtins/research/__init__.py | 0 .../subagents/builtins/research/agent.py | 52 --- .../builtins/research/description.md | 2 - .../builtins/research/system_prompt.md | 62 ---- .../builtins/research/tools/__init__.py | 10 - .../builtins/research/tools/index.py | 29 -- .../builtins/research/tools/scrape_webpage.py | 347 ------------------ .../multi_agent_chat/subagents/registry.py | 10 +- .../test_subagent_composition.py | 5 +- 9 files changed, 5 insertions(+), 512 deletions(-) delete mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/__init__.py delete mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/agent.py delete mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/description.md delete mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/system_prompt.md delete mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/tools/__init__.py delete mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/tools/index.py delete mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/tools/scrape_webpage.py diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/__init__.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/__init__.py deleted file mode 100644 index e69de29bb..000000000 diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/agent.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/agent.py deleted file mode 100644 index e3c0ab9ae..000000000 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/agent.py +++ /dev/null @@ -1,52 +0,0 @@ -"""``research`` route: ``SurfSenseSubagentSpec`` builder for deepagents.""" - -from __future__ import annotations - -from typing import Any - -from langchain_core.language_models import BaseChatModel -from langchain_core.tools import BaseTool - -from app.agents.chat.multi_agent_chat.shared.middleware.citation_state import ( - build_citation_state_mw, -) -from app.agents.chat.multi_agent_chat.subagents.shared.md_file_reader import ( - read_md_file, -) -from app.agents.chat.multi_agent_chat.subagents.shared.spec import SurfSenseSubagentSpec -from app.agents.chat.multi_agent_chat.subagents.shared.subagent_builder import ( - pack_subagent, -) - -from .tools.index import NAME, RULESET, load_tools - - -def build_subagent( - *, - dependencies: dict[str, Any], - model: BaseChatModel | None = None, - middleware_stack: dict[str, Any] | None = None, - mcp_tools: list[BaseTool] | None = None, -) -> SurfSenseSubagentSpec: - tools = [*load_tools(dependencies=dependencies), *(mcp_tools or [])] - description = ( - read_md_file(__package__, "description").strip() - or "Handles research tasks for this workspace." - ) - system_prompt = read_md_file(__package__, "system_prompt").strip() - # web_search registers WEB_RESULT citations via Command(update=...); the - # citation-state middleware declares the channel so those [n] merge back up. - middleware_with_citations = { - **(middleware_stack or {}), - "citation_state": build_citation_state_mw(), - } - return pack_subagent( - name=NAME, - description=description, - system_prompt=system_prompt, - tools=tools, - ruleset=RULESET, - dependencies=dependencies, - model=model, - middleware_stack=middleware_with_citations, - ) diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/description.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/description.md deleted file mode 100644 index 0a99b4140..000000000 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/description.md +++ /dev/null @@ -1,2 +0,0 @@ -Specialist for external research. -Use whenever a task requires finding sources on the web and extracting evidence to answer documentation questions. diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/system_prompt.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/system_prompt.md deleted file mode 100644 index 3d90a4352..000000000 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/system_prompt.md +++ /dev/null @@ -1,62 +0,0 @@ -You are the SurfSense research operations sub-agent. -You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis. - - -Gather and synthesize evidence using SurfSense research tools with clear citations and uncertainty reporting. - - - -- `web_search` -- `scrape_webpage` - - - -- Use only tools in ``. -- Prefer primary and recent sources when recency matters. -- If the delegated request is underspecified, return `status=blocked` with the missing research constraints. -- Never fabricate facts, citations, URLs, or quote text. - - - -`web_search` returns a `` block whose results are each prefixed with a bracketed label — `[1]`, `[2]`, `[3]`. That `[n]` is the citation label. When a finding came from a specific result, append its `[n]` to that finding, copying the label **exactly** as shown. The caller relays these labels verbatim and the server resolves each one, so a wrong number silently breaks the citation. - -- Use the exact `[n]` shown next to the result you actually used; never renumber, guess, or invent a label. -- Before emitting an `[n]`, confirm that bracketed label appears in the `web_search` output this turn. If you can't see it, omit it. -- Write the bare label `[n]` only — no `[citation:…]` wrapper, no markdown links. -- Several results behind one finding → each in its own brackets with nothing between: `[1][2]`. -- `scrape_webpage` returns raw page text with no `[n]` labels; a fact drawn only from a scrape carries no citation (report the URL in `evidence.sources` instead). - - - -- Do not execute connector mutations (email/calendar/docs/chat writes) or deliverable generation. - - - -- Report uncertainty explicitly when evidence is incomplete or conflicting. -- Never present unverified claims as facts. - - - -- On tool failure, return `status=error` with a concise recovery `next_step`. -- On no useful evidence, return `status=blocked` with recommended narrower filters. - - - -Return **only** one JSON object (no markdown/prose): -{ - "status": "success" | "partial" | "blocked" | "error", - "action_summary": string, - "evidence": { - "findings": string[], - "sources": string[], - "confidence": "high" | "medium" | "low" - }, - "next_step": string | null, - "missing_fields": string[] | null, - "assumptions": string[] | null -} - -Route-specific rules: -- `evidence.findings`: max 10 entries, each a single sentence stating one distinct fact. Append the supporting `[n]` to each finding drawn from a `web_search` result. Do not paste raw paragraphs, scraped pages, or quote blocks. -- `evidence.sources`: max 10 URLs, one per finding when applicable. List each URL once. (Citations travel as `[n]`; `sources` is for transparency and for scrape-only facts that carry no `[n]`.) - diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/tools/__init__.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/tools/__init__.py deleted file mode 100644 index 0c99bf222..000000000 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/tools/__init__.py +++ /dev/null @@ -1,10 +0,0 @@ -"""Research-stage tools: web search (shared) and scrape.""" - -from app.agents.chat.shared.tools.web_search import create_web_search_tool - -from .scrape_webpage import create_scrape_webpage_tool - -__all__ = [ - "create_scrape_webpage_tool", - "create_web_search_tool", -] diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/tools/index.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/tools/index.py deleted file mode 100644 index b3ea5a6b0..000000000 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/tools/index.py +++ /dev/null @@ -1,29 +0,0 @@ -"""``research`` native tools and (empty) permission ruleset.""" - -from __future__ import annotations - -from typing import Any - -from langchain_core.tools import BaseTool - -from app.agents.chat.multi_agent_chat.shared.permissions import Ruleset -from app.agents.chat.shared.tools.web_search import create_web_search_tool - -from .scrape_webpage import create_scrape_webpage_tool - -NAME = "research" - -RULESET = Ruleset(origin=NAME, rules=[]) - - -def load_tools( - *, dependencies: dict[str, Any] | None = None, **kwargs: Any -) -> list[BaseTool]: - d = {**(dependencies or {}), **kwargs} - return [ - create_web_search_tool( - workspace_id=d.get("workspace_id"), - available_connectors=d.get("available_connectors"), - ), - create_scrape_webpage_tool(), - ] diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/tools/scrape_webpage.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/tools/scrape_webpage.py deleted file mode 100644 index af7ce44e7..000000000 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/research/tools/scrape_webpage.py +++ /dev/null @@ -1,347 +0,0 @@ -"""Scrape pages via WebCrawlerConnector; YouTube URLs use the transcript API instead of HTML crawl.""" - -import hashlib -import logging -import time -from typing import Any -from urllib.parse import urlparse - -from fake_useragent import UserAgent -from langchain_core.tools import tool -from requests import Session -from scrapling.fetchers import AsyncFetcher -from youtube_transcript_api import YouTubeTranscriptApi - -from app.proprietary.web_crawler import ( - CrawlOutcomeStatus, - WebCrawlerConnector, -) -from app.tasks.document_processors.youtube_processor import get_youtube_video_id -from app.utils.proxy import get_proxy_url, get_requests_proxies - -logger = logging.getLogger(__name__) - - -def _bill_successful_scrape() -> None: - """Fold one successful crawl into the current chat turn's bill (Phase 3c). - - The cost rides the turn accumulator and settles at the premium - ``finalize_credit`` step — no separate wallet hit. Free / BYOK / anonymous - turns (which never reserve/finalize) still record the line in the - breakdown but are never debited. No-op when crawl billing is disabled or - there is no active turn (e.g. non-chat callers). - """ - from app.services.token_tracking_service import get_current_accumulator - from app.services.web_crawl_credit_service import WebCrawlCreditService - - if not WebCrawlCreditService.billing_enabled(): - return - acc = get_current_accumulator() - if acc is None: - return - acc.add( - model="web_crawl", - prompt_tokens=0, - completion_tokens=0, - total_tokens=0, - cost_micros=WebCrawlCreditService.successes_to_micros(1), - call_kind="web_crawl", - ) - - -def _bill_captcha_attempts(outcome) -> None: - """Fold captcha solve attempts (Phase 3d) into the current chat turn's bill. - - Per *attempt*, not per success: a solve that didn't rescue the crawl still - cost real solver money, so this runs before the success/failure branch. - Mirrors :func:`_bill_successful_scrape` (rides the turn accumulator, settles - at ``finalize_credit``; record-only on free/anonymous turns). No-op when - captcha billing is off, there were no attempts, or no turn is active. - """ - from app.services.token_tracking_service import get_current_accumulator - from app.services.web_crawl_credit_service import WebCrawlCreditService - - if not WebCrawlCreditService.captcha_billing_enabled(): - return - attempts = getattr(outcome, "captcha_attempts", 0) or 0 - if attempts <= 0: - return - acc = get_current_accumulator() - if acc is None: - return - acc.add( - model="web_crawl_captcha", - prompt_tokens=0, - completion_tokens=0, - total_tokens=0, - cost_micros=WebCrawlCreditService.captcha_solves_to_micros(attempts), - call_kind="web_crawl_captcha", - ) - - -def extract_domain(url: str) -> str: - """Extract the domain from a URL.""" - try: - parsed = urlparse(url) - domain = parsed.netloc - if domain.startswith("www."): - domain = domain[4:] - return domain - except Exception: - return "" - - -def generate_scrape_id(url: str) -> str: - """Generate a unique ID for a scraped webpage.""" - hash_val = hashlib.md5(url.encode()).hexdigest()[:12] - return f"scrape-{hash_val}" - - -def truncate_content(content: str, max_length: int = 50000) -> tuple[str, bool]: - """ - Truncate content to a maximum length. - - Returns: - Tuple of (truncated_content, was_truncated) - """ - if len(content) <= max_length: - return content, False - - # Prefer truncating at a sentence/paragraph boundary. - truncated = content[:max_length] - last_period = truncated.rfind(".") - last_newline = truncated.rfind("\n\n") - - boundary = max(last_period, last_newline) - if boundary > max_length * 0.8: # only if the boundary isn't too far back - truncated = content[: boundary + 1] - - return truncated + "\n\n[Content truncated...]", True - - -async def _scrape_youtube_video( - url: str, video_id: str, max_length: int -) -> dict[str, Any]: - """ - Fetch YouTube video metadata and transcript via the YouTubeTranscriptApi. - - Returns a result dict in the same shape as the regular scrape_webpage output. - """ - scrape_id = generate_scrape_id(url) - domain = "youtube.com" - - # --- Video metadata via oEmbed --- - residential_proxies = get_requests_proxies() - - params = { - "format": "json", - "url": f"https://www.youtube.com/watch?v={video_id}", - } - oembed_url = "https://www.youtube.com/oembed" - - try: - oembed_fetch_start = time.perf_counter() - oembed_page = await AsyncFetcher.get( - oembed_url, - params=params, - proxy=get_proxy_url(), - stealthy_headers=True, - ) - logger.info( - "[scrape_webpage][perf] source=oembed video=%s status=%s fetch_ms=%.1f", - video_id, - getattr(oembed_page, "status", None), - (time.perf_counter() - oembed_fetch_start) * 1000, - ) - video_data = oembed_page.json() - except Exception: - video_data = {} - - title = video_data.get("title", "YouTube Video") - author = video_data.get("author_name", "Unknown") - - # --- Transcript via YouTubeTranscriptApi --- - try: - transcript_fetch_start = time.perf_counter() - ua = UserAgent() - http_client = Session() - http_client.headers.update({"User-Agent": ua.random}) - if residential_proxies: - http_client.proxies.update(residential_proxies) - ytt_api = YouTubeTranscriptApi(http_client=http_client) - - # Pick the first transcript (video's primary language) rather than - # defaulting to English. - transcript_list = ytt_api.list(video_id) - transcript = next(iter(transcript_list)) - captions = transcript.fetch() - - logger.info( - "[scrape_webpage][perf] source=transcript video=%s fetch_ms=%.1f", - video_id, - (time.perf_counter() - transcript_fetch_start) * 1000, - ) - logger.info( - f"[scrape_webpage] Fetched transcript for {video_id} " - f"in {transcript.language} ({transcript.language_code})" - ) - - transcript_segments = [] - for line in captions: - start_time = line.start - duration = line.duration - text = line.text - timestamp = f"[{start_time:.2f}s-{start_time + duration:.2f}s]" - transcript_segments.append(f"{timestamp} {text}") - transcript_text = "\n".join(transcript_segments) - except Exception as e: - logger.warning(f"[scrape_webpage] No transcript for video {video_id}: {e}") - transcript_text = f"No captions available for this video. Error: {e!s}" - - content = f"# {title}\n\n**Author:** {author}\n**Video ID:** {video_id}\n\n## Transcript\n\n{transcript_text}" - - content, was_truncated = truncate_content(content, max_length) - word_count = len(content.split()) - - description = f"YouTube video by {author}" - - return { - "id": scrape_id, - "assetId": url, - "kind": "article", - "href": url, - "title": title, - "description": description, - "content": content, - "domain": domain, - "word_count": word_count, - "was_truncated": was_truncated, - "crawler_type": "youtube_transcript", - "author": author, - } - - -def create_scrape_webpage_tool(): - """ - Factory function to create the scrape_webpage tool. - - Returns: - A configured tool function for scraping webpages. - """ - - @tool - async def scrape_webpage( - url: str, - max_length: int = 50000, - ) -> dict[str, Any]: - """ - Scrape and extract the main content from a webpage. - - Use this tool when the user wants you to read, summarize, or answer - questions about a specific webpage's content. This tool actually - fetches and reads the full page content. For YouTube video URLs it - fetches the transcript directly instead of crawling the page. - - Common triggers: - - "Read this article and summarize it" - - "What does this page say about X?" - - "Summarize this blog post for me" - - "Tell me the key points from this article" - - "What's in this webpage?" - - Args: - url: The URL of the webpage to scrape (must be HTTP/HTTPS) - max_length: Maximum content length to return (default: 50000 chars) - - Returns: - A dictionary containing: - - id: Unique identifier for this scrape - - assetId: The URL (for deduplication) - - kind: "article" (type of content) - - href: The URL to open when clicked - - title: Page title - - description: Brief description or excerpt - - content: The extracted main content (markdown format) - - domain: The domain name - - word_count: Approximate word count - - was_truncated: Whether content was truncated - - error: Error message (if scraping failed) - """ - scrape_id = generate_scrape_id(url) - domain = extract_domain(url) - - if not url.startswith(("http://", "https://")): - url = f"https://{url}" - - try: - # YouTube URLs use the transcript API instead of crawling. - video_id = get_youtube_video_id(url) - if video_id: - return await _scrape_youtube_video(url, video_id, max_length) - - connector = WebCrawlerConnector() - outcome = await connector.crawl_url(url) - - # 03d: bill any captcha attempts (even if the crawl ultimately failed). - _bill_captcha_attempts(outcome) - - if outcome.status is not CrawlOutcomeStatus.SUCCESS or not outcome.result: - return { - "id": scrape_id, - "assetId": url, - "kind": "article", - "href": url, - "title": domain or "Webpage", - "domain": domain, - "error": outcome.error or "No content returned from crawler", - } - - result = outcome.result - _bill_successful_scrape() - content = result.get("content", "") - metadata = result.get("metadata", {}) - - title = metadata.get("title", "") - if not title: - title = domain or url.split("/")[-1] or "Webpage" - - description = metadata.get("description", "") - if not description and content: - first_para = content.split("\n\n")[0] if content else "" - description = ( - first_para[:300] + "..." if len(first_para) > 300 else first_para - ) - - content, was_truncated = truncate_content(content, max_length) - word_count = len(content.split()) - - return { - "id": scrape_id, - "assetId": url, - "kind": "article", - "href": url, - "title": title, - "description": description, - "content": content, - "domain": domain, - "word_count": word_count, - "was_truncated": was_truncated, - "crawler_type": result.get("crawler_type", "unknown"), - "author": metadata.get("author"), - "date": metadata.get("date"), - } - - except Exception as e: - error_message = str(e) - logger.error(f"[scrape_webpage] Error scraping {url}: {error_message}") - return { - "id": scrape_id, - "assetId": url, - "kind": "article", - "href": url, - "title": domain or "Webpage", - "domain": domain, - "error": f"Failed to scrape: {error_message[:100]}", - } - - return scrape_webpage diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py index 02c196086..8fbd866dc 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py @@ -21,9 +21,6 @@ from app.agents.chat.multi_agent_chat.subagents.builtins.knowledge_base.agent im from app.agents.chat.multi_agent_chat.subagents.builtins.memory.agent import ( build_subagent as build_memory_subagent, ) -from app.agents.chat.multi_agent_chat.subagents.builtins.research.agent import ( - build_subagent as build_research_subagent, -) from app.agents.chat.multi_agent_chat.subagents.builtins.web_crawler.agent import ( build_subagent as build_web_crawler_subagent, ) @@ -112,7 +109,6 @@ SUBAGENT_BUILDERS_BY_NAME: dict[str, SubagentBuilder] = { "memory": build_memory_subagent, "notion": build_notion_subagent, "onedrive": build_onedrive_subagent, - "research": build_research_subagent, "slack": build_slack_subagent, "teams": build_teams_subagent, "web_crawler": build_web_crawler_subagent, @@ -127,7 +123,7 @@ def _route_resource_package(builder: SubagentBuilder) -> str: def main_prompt_registry_subagent_lines(exclude: list[str]) -> list[tuple[str, str]]: """(name, description) for registry specialists included for **task** (same rules as ``build_subagents``).""" - banned = frozenset(("memory", "research")) | frozenset(exclude) + banned = frozenset(("memory",)) | frozenset(exclude) rows: list[tuple[str, str]] = [] for name in sorted(SUBAGENT_BUILDERS_BY_NAME): if name in banned: @@ -197,10 +193,10 @@ def build_subagents( disabled_tools: list[str] | None = None, ask_kb_tool: BaseTool | None = None, ) -> list[SubAgent]: - """Build registry subagents; skip memory/research; skip names in exclude.""" + """Build registry subagents; skip memory; skip names in exclude.""" mcp = mcp_tools_by_agent or {} specs: list[SubAgent] = [] - excluded = ["memory", "research"] + excluded = ["memory"] if exclude: excluded.extend(exclude) disabled_names = frozenset(disabled_tools or ()) diff --git a/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py b/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py index dc0f35994..873fe26c5 100644 --- a/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py +++ b/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py @@ -19,7 +19,7 @@ from app.agents.chat.multi_agent_chat.subagents.registry import ( pytestmark = pytest.mark.unit -# The full specialist roster the main agent composes from: 6 builtins + 15 +# The full specialist roster the main agent composes from: 5 builtins + 15 # connector routes. Adding/removing a specialist is a deliberate product change # and must be reflected here. _EXPECTED_SUBAGENTS = frozenset( @@ -40,7 +40,6 @@ _EXPECTED_SUBAGENTS = frozenset( "memory", "notion", "onedrive", - "research", "slack", "teams", "web_crawler", @@ -50,7 +49,7 @@ _EXPECTED_SUBAGENTS = frozenset( # Specialists that are always available regardless of connected sources, so they # carry no required-connector entry. -_CONNECTORLESS = frozenset({"memory", "research"}) +_CONNECTORLESS = frozenset({"memory"}) def test_registry_contains_exactly_expected_subagents(): From ab6be6cbda961cac9c1e9d75fb1ff0e66406291f Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Fri, 3 Jul 2026 11:53:33 +0200 Subject: [PATCH 57/64] refactor(connectors): remove legacy web-crawler KB indexer paths (keep enums) --- .../runtime/connector_searchable_types.py | 1 - surfsense_backend/app/celery_app.py | 1 - .../routes/search_source_connectors_routes.py | 80 --- .../app/services/connector_service.py | 90 --- .../app/tasks/celery_tasks/connector_tasks.py | 39 -- .../celery_tasks/schedule_checker_task.py | 36 - .../app/tasks/connector_indexers/__init__.py | 2 - .../connector_indexers/webcrawler_indexer.py | 661 ------------------ .../app/utils/periodic_scheduler.py | 17 - surfsense_backend/app/utils/validators.py | 15 - .../app/utils/webcrawler_utils.py | 28 - .../scripts/e2e_phase3_crawl_billing.py | 176 +---- .../test_web_crawl_billing.py | 306 -------- .../tests/unit/utils/test_validators.py | 8 +- 14 files changed, 4 insertions(+), 1456 deletions(-) delete mode 100644 surfsense_backend/app/tasks/connector_indexers/webcrawler_indexer.py delete mode 100644 surfsense_backend/app/utils/webcrawler_utils.py delete mode 100644 surfsense_backend/tests/unit/connector_indexers/test_web_crawl_billing.py diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/runtime/connector_searchable_types.py b/surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/runtime/connector_searchable_types.py index 09fa0756a..339154060 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/runtime/connector_searchable_types.py +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/runtime/connector_searchable_types.py @@ -40,7 +40,6 @@ _CONNECTOR_TYPE_TO_SEARCHABLE: dict[str, str] = { "AIRTABLE_CONNECTOR": "AIRTABLE_CONNECTOR", "LUMA_CONNECTOR": "LUMA_CONNECTOR", "ELASTICSEARCH_CONNECTOR": "ELASTICSEARCH_CONNECTOR", - "WEBCRAWLER_CONNECTOR": "CRAWLED_URL", # Maps to document type "BOOKSTACK_CONNECTOR": "BOOKSTACK_CONNECTOR", "CIRCLEBACK_CONNECTOR": "CIRCLEBACK", # Connector type differs from document type "OBSIDIAN_CONNECTOR": "OBSIDIAN_CONNECTOR", diff --git a/surfsense_backend/app/celery_app.py b/surfsense_backend/app/celery_app.py index 331ed0f40..471d80197 100644 --- a/surfsense_backend/app/celery_app.py +++ b/surfsense_backend/app/celery_app.py @@ -244,7 +244,6 @@ celery_app.conf.update( "index_google_gmail_messages": {"queue": CONNECTORS_QUEUE}, "index_google_drive_files": {"queue": CONNECTORS_QUEUE}, "index_elasticsearch_documents": {"queue": CONNECTORS_QUEUE}, - "index_crawled_urls": {"queue": CONNECTORS_QUEUE}, "index_bookstack_pages": {"queue": CONNECTORS_QUEUE}, "index_composio_connector": {"queue": CONNECTORS_QUEUE}, "index_obsidian_attachment": {"queue": CONNECTORS_QUEUE}, diff --git a/surfsense_backend/app/routes/search_source_connectors_routes.py b/surfsense_backend/app/routes/search_source_connectors_routes.py index 33da72417..2bdf29756 100644 --- a/surfsense_backend/app/routes/search_source_connectors_routes.py +++ b/surfsense_backend/app/routes/search_source_connectors_routes.py @@ -1051,34 +1051,6 @@ async def index_connector_content( ) response_message = "Elasticsearch indexing started in the background." - elif connector.connector_type == SearchSourceConnectorType.WEBCRAWLER_CONNECTOR: - from app.tasks.celery_tasks.connector_tasks import index_crawled_urls_task - from app.utils.webcrawler_utils import parse_webcrawler_urls - - # Check if URLs are configured before triggering indexing - connector_config = connector.config or {} - urls = parse_webcrawler_urls(connector_config.get("INITIAL_URLS")) - - if not urls: - # URLs are optional - skip indexing gracefully - logger.info( - f"Webcrawler connector {connector_id} has no URLs configured, skipping indexing" - ) - response_message = "No URLs configured for this connector. Add URLs in the connector settings to enable indexing." - indexing_started = False - else: - logger.info( - f"Triggering web pages indexing for connector {connector_id} into workspace {workspace_id} from {indexing_from} to {indexing_to}" - ) - index_crawled_urls_task.delay( - connector_id, - workspace_id, - str(user.id), - indexing_from, - indexing_to, - ) - response_message = "Web page indexing started in the background." - elif ( connector.connector_type == SearchSourceConnectorType.COMPOSIO_GOOGLE_DRIVE_CONNECTOR @@ -2478,58 +2450,6 @@ async def run_elasticsearch_indexing( ) -# Add new helper functions for crawled web page indexing -async def run_web_page_indexing_with_new_session( - connector_id: int, - workspace_id: int, - user_id: str, - start_date: str, - end_date: str, -): - """ - Create a new session and run the Web page indexing task. - This prevents session leaks by creating a dedicated session for the background task. - """ - async with async_session_maker() as session: - await run_web_page_indexing( - session, connector_id, workspace_id, user_id, start_date, end_date - ) - - -async def run_web_page_indexing( - session: AsyncSession, - connector_id: int, - workspace_id: int, - user_id: str, - start_date: str, - end_date: str, -): - """ - Background task to run Web page indexing. - - Args: - session: Database session - connector_id: ID of the webcrawler connector - workspace_id: ID of the workspace - user_id: ID of the user - start_date: Start date for indexing - end_date: End date for indexing - """ - from app.tasks.connector_indexers import index_crawled_urls - - await _run_indexing_with_notifications( - session=session, - connector_id=connector_id, - workspace_id=workspace_id, - user_id=user_id, - start_date=start_date, - end_date=end_date, - indexing_function=index_crawled_urls, - update_timestamp_func=_update_connector_timestamp_by_id, - supports_heartbeat_callback=True, - ) - - # Add new helper functions for BookStack indexing async def run_bookstack_indexing_with_new_session( connector_id: int, diff --git a/surfsense_backend/app/services/connector_service.py b/surfsense_backend/app/services/connector_service.py index a13a9b3ea..ff7ded854 100644 --- a/surfsense_backend/app/services/connector_service.py +++ b/surfsense_backend/app/services/connector_service.py @@ -62,96 +62,6 @@ class ConnectorService: # Fallback to default value self.source_id_counter = 1 - async def search_crawled_urls( - self, - user_query: str, - workspace_id: int, - top_k: int = 20, - start_date: datetime | None = None, - end_date: datetime | None = None, - ) -> tuple: - """ - Search for crawled URLs and return both the source information and langchain documents. - - Uses combined chunk-level and document-level hybrid search with RRF fusion. - - Args: - user_query: The user's query - workspace_id: The workspace ID to search in - top_k: Maximum number of results to return - start_date: Optional start date for filtering documents by updated_at - end_date: Optional end date for filtering documents by updated_at - - Returns: - tuple: (sources_info, langchain_documents) - """ - crawled_urls_docs = await self._combined_rrf_search( - query_text=user_query, - workspace_id=workspace_id, - document_type="CRAWLED_URL", - top_k=top_k, - start_date=start_date, - end_date=end_date, - ) - - # Early return if no results - if not crawled_urls_docs: - return { - "id": 1, - "name": "Crawled URLs", - "type": "CRAWLED_URL", - "sources": [], - }, [] - - def _title_fn(doc_info: dict[str, Any], metadata: dict[str, Any]) -> str: - return doc_info.get("title") or metadata.get("title") or "Untitled Document" - - def _url_fn(_doc_info: dict[str, Any], metadata: dict[str, Any]) -> str: - return metadata.get("source") or metadata.get("url") or "" - - def _description_fn( - chunk: dict[str, Any], _doc_info: dict[str, Any], metadata: dict[str, Any] - ) -> str: - description = metadata.get("description") or self._chunk_preview( - chunk.get("content", "") - ) - info_parts = [] - language = metadata.get("language", "") - last_crawled_at = metadata.get("last_crawled_at", "") - if language: - info_parts.append(f"Language: {language}") - if last_crawled_at: - info_parts.append(f"Last crawled: {last_crawled_at}") - if info_parts: - description = (description + " | " + " | ".join(info_parts)).strip(" |") - return description - - def _extra_fields_fn( - _chunk: dict[str, Any], _doc_info: dict[str, Any], metadata: dict[str, Any] - ) -> dict[str, Any]: - return { - "language": metadata.get("language", ""), - "last_crawled_at": metadata.get("last_crawled_at", ""), - } - - sources_list = self._build_chunk_sources_from_documents( - crawled_urls_docs, - title_fn=_title_fn, - description_fn=_description_fn, - url_fn=_url_fn, - extra_fields_fn=_extra_fields_fn, - ) - - # Create result object - result_object = { - "id": 1, - "name": "Crawled URLs", - "type": "CRAWLED_URL", - "sources": sources_list, - } - - return result_object, crawled_urls_docs - async def search_files( self, user_query: str, diff --git a/surfsense_backend/app/tasks/celery_tasks/connector_tasks.py b/surfsense_backend/app/tasks/celery_tasks/connector_tasks.py index a3490e8b4..ec078e031 100644 --- a/surfsense_backend/app/tasks/celery_tasks/connector_tasks.py +++ b/surfsense_backend/app/tasks/celery_tasks/connector_tasks.py @@ -415,45 +415,6 @@ async def _index_elasticsearch_documents( ) -@celery_app.task(name="index_crawled_urls", bind=True) -def index_crawled_urls_task( - self, - connector_id: int, - workspace_id: int, - user_id: str, - start_date: str, - end_date: str, -): - """Celery task to index Web page Urls.""" - try: - return run_async_celery_task( - lambda: _index_crawled_urls( - connector_id, workspace_id, user_id, start_date, end_date - ) - ) - except Exception as e: - _handle_greenlet_error(e, "index_crawled_urls", connector_id) - raise - - -async def _index_crawled_urls( - connector_id: int, - workspace_id: int, - user_id: str, - start_date: str, - end_date: str, -): - """Index Web page Urls with new session.""" - from app.routes.search_source_connectors_routes import ( - run_web_page_indexing, - ) - - async with get_celery_session_maker()() as session: - await run_web_page_indexing( - session, connector_id, workspace_id, user_id, start_date, end_date - ) - - @celery_app.task(name="index_bookstack_pages", bind=True) def index_bookstack_pages_task( self, diff --git a/surfsense_backend/app/tasks/celery_tasks/schedule_checker_task.py b/surfsense_backend/app/tasks/celery_tasks/schedule_checker_task.py index 7243522d7..55aa04d22 100644 --- a/surfsense_backend/app/tasks/celery_tasks/schedule_checker_task.py +++ b/surfsense_backend/app/tasks/celery_tasks/schedule_checker_task.py @@ -49,7 +49,6 @@ async def _check_and_trigger_schedules(): # Teams, Gmail, Calendar, Luma) use real-time tools instead. from app.tasks.celery_tasks.connector_tasks import ( index_confluence_pages_task, - index_crawled_urls_task, index_elasticsearch_documents_task, index_github_repos_task, index_google_drive_files_task, @@ -61,7 +60,6 @@ async def _check_and_trigger_schedules(): SearchSourceConnectorType.GITHUB_CONNECTOR: index_github_repos_task, SearchSourceConnectorType.CONFLUENCE_CONNECTOR: index_confluence_pages_task, SearchSourceConnectorType.ELASTICSEARCH_CONNECTOR: index_elasticsearch_documents_task, - SearchSourceConnectorType.WEBCRAWLER_CONNECTOR: index_crawled_urls_task, SearchSourceConnectorType.GOOGLE_DRIVE_CONNECTOR: index_google_drive_files_task, SearchSourceConnectorType.COMPOSIO_GOOGLE_DRIVE_CONNECTOR: index_google_drive_files_task, } @@ -165,40 +163,6 @@ async def _check_and_trigger_schedules(): await session.commit() continue - # Special handling for Webcrawler - skip if no URLs configured - elif ( - connector.connector_type - == SearchSourceConnectorType.WEBCRAWLER_CONNECTOR - ): - from app.utils.webcrawler_utils import parse_webcrawler_urls - - connector_config = connector.config or {} - urls = parse_webcrawler_urls( - connector_config.get("INITIAL_URLS") - ) - - if urls: - task.delay( - connector.id, - connector.workspace_id, - str(connector.user_id), - None, # start_date - None, # end_date - ) - else: - # No URLs configured - skip indexing but still update next_scheduled_at - logger.info( - f"Webcrawler connector {connector.id} has no URLs configured, " - "skipping periodic indexing (will check again at next scheduled time)" - ) - from datetime import timedelta - - connector.next_scheduled_at = now + timedelta( - minutes=connector.indexing_frequency_minutes - ) - await session.commit() - continue - else: task.delay( connector.id, diff --git a/surfsense_backend/app/tasks/connector_indexers/__init__.py b/surfsense_backend/app/tasks/connector_indexers/__init__.py index 218f21066..0f867bdcf 100644 --- a/surfsense_backend/app/tasks/connector_indexers/__init__.py +++ b/surfsense_backend/app/tasks/connector_indexers/__init__.py @@ -14,12 +14,10 @@ from .google_calendar_indexer import index_google_calendar_events from .google_drive_indexer import index_google_drive_files from .google_gmail_indexer import index_google_gmail_messages from .notion_indexer import index_notion_pages -from .webcrawler_indexer import index_crawled_urls __all__ = [ "index_bookstack_pages", "index_confluence_pages", - "index_crawled_urls", "index_elasticsearch_documents", "index_github_repos", "index_google_calendar_events", diff --git a/surfsense_backend/app/tasks/connector_indexers/webcrawler_indexer.py b/surfsense_backend/app/tasks/connector_indexers/webcrawler_indexer.py deleted file mode 100644 index 8535e8d83..000000000 --- a/surfsense_backend/app/tasks/connector_indexers/webcrawler_indexer.py +++ /dev/null @@ -1,661 +0,0 @@ -""" -Webcrawler connector indexer. - -Implements 2-phase document status updates for real-time UI feedback: -- Phase 1: Create all documents with 'pending' status (visible in UI immediately) -- Phase 2: Process each document: pending → processing → ready/failed -""" - -import time -from collections.abc import Awaitable, Callable -from datetime import datetime - -from sqlalchemy.exc import SQLAlchemyError -from sqlalchemy.ext.asyncio import AsyncSession - -from app.proprietary.web_crawler import ( - CrawlOutcomeStatus, - WebCrawlerConnector, -) -from app.config import config -from app.db import Document, DocumentStatus, DocumentType, SearchSourceConnectorType -from app.services.etl_credit_service import InsufficientCreditsError -from app.services.task_logging_service import TaskLoggingService -from app.services.token_tracking_service import record_token_usage -from app.services.web_crawl_credit_service import WebCrawlCreditService -from app.utils.captcha import captcha_enabled -from app.utils.document_converters import ( - create_document_chunks, - embed_text, - generate_content_hash, - generate_unique_identifier_hash, -) -from app.utils.webcrawler_utils import parse_webcrawler_urls - -from .base import ( - check_document_by_unique_identifier, - check_duplicate_document_by_hash, - get_connector_by_id, - get_current_timestamp, - logger, - safe_set_chunks, - update_connector_last_indexed, -) - -# Type hint for heartbeat callback -HeartbeatCallbackType = Callable[[int], Awaitable[None]] - -# Heartbeat interval in seconds -HEARTBEAT_INTERVAL_SECONDS = 30 - - -async def _resolve_workspace_owner(session: AsyncSession, workspace_id: int): - """Return the ``user_id`` that owns ``workspace_id`` (the crawl payer). - - Phase 3c bills the **workspace owner**, not the triggering user (which for - periodic/scheduled runs may differ). Returns ``None`` if the workspace is - gone, letting the caller fall back to the triggering user. - """ - from sqlalchemy import select - - from app.db import Workspace - - result = await session.execute( - select(Workspace.user_id).where(Workspace.id == workspace_id) - ) - return result.scalar_one_or_none() - - -async def index_crawled_urls( - session: AsyncSession, - connector_id: int, - workspace_id: int, - user_id: str, - start_date: str | None = None, - end_date: str | None = None, - update_last_indexed: bool = True, - on_heartbeat_callback: HeartbeatCallbackType | None = None, -) -> tuple[int, str | None]: - """ - Index web page URLs with real-time document status updates. - - Implements 2-phase approach for real-time UI feedback: - - Phase 1: Create all documents with 'pending' status (visible in UI immediately) - - Phase 2: Process each document: pending → processing → ready/failed - - Args: - session: Database session - connector_id: ID of the webcrawler connector - workspace_id: ID of the workspace to store documents in - user_id: User ID - start_date: Start date for filtering (YYYY-MM-DD format) - optional - end_date: End date for filtering (YYYY-MM-DD format) - optional - update_last_indexed: Whether to update the last_indexed_at timestamp (default: True) - on_heartbeat_callback: Optional callback to update notification during long-running indexing. - - Returns: - Tuple containing (number of documents indexed, error message or None) - """ - task_logger = TaskLoggingService(session, workspace_id) - - # Log task start - log_entry = await task_logger.log_task_start( - task_name="crawled_url_indexing", - source="connector_indexing_task", - message=f"Starting web page URL indexing for connector {connector_id}", - metadata={ - "connector_id": connector_id, - "user_id": str(user_id), - "start_date": start_date, - "end_date": end_date, - }, - ) - - try: - # Get the connector - await task_logger.log_task_progress( - log_entry, - f"Retrieving webcrawler connector {connector_id} from database", - {"stage": "connector_retrieval"}, - ) - - # Get the connector from the database - connector = await get_connector_by_id( - session, connector_id, SearchSourceConnectorType.WEBCRAWLER_CONNECTOR - ) - - if not connector: - await task_logger.log_task_failure( - log_entry, - f"Connector with ID {connector_id} not found or is not a webcrawler connector", - "Connector not found", - {"error_type": "ConnectorNotFound"}, - ) - return ( - 0, - f"Connector with ID {connector_id} not found or is not a webcrawler connector", - ) - - # Get URLs from connector config - raw_initial_urls = connector.config.get("INITIAL_URLS") - urls = parse_webcrawler_urls(raw_initial_urls) - - # DEBUG: Log connector config details for troubleshooting empty URL issues - logger.info( - f"Starting crawled web page indexing for connector {connector_id} with {len(urls)} URLs. " - f"Connector name: {connector.name}, " - f"INITIAL_URLS type: {type(raw_initial_urls).__name__}, " - f"INITIAL_URLS value: {repr(raw_initial_urls)[:200] if raw_initial_urls else 'None'}" - ) - - # Initialize webcrawler client - await task_logger.log_task_progress( - log_entry, - f"Initializing webcrawler client for connector {connector_id}", - { - "stage": "client_initialization", - }, - ) - - crawler = WebCrawlerConnector() - - # Validate URLs - if not urls: - # DEBUG: Log detailed connector config for troubleshooting - logger.error( - f"No URLs provided for indexing. Connector ID: {connector_id}, " - f"Connector name: {connector.name}, " - f"Config keys: {list(connector.config.keys()) if connector.config else 'None'}, " - f"INITIAL_URLS raw value: {raw_initial_urls!r}" - ) - await task_logger.log_task_failure( - log_entry, - "No URLs provided for indexing", - f"Empty URL list. INITIAL_URLS value: {repr(raw_initial_urls)[:100]}", - {"error_type": "ValidationError", "connector_name": connector.name}, - ) - return 0, "No URLs provided for indexing" - - # ======================================================================= - # Phase 3c/3d: crawl-credit pre-flight. Bill the workspace OWNER. The - # estimate is a safe UPPER BOUND so the wallet can't strand mid-batch: - # crawl = len(urls) successes (3c; actual successes are <=) - # captcha = len(urls) * MAX_ATTEMPTS (3d; per-attempt, worst case) - # No-op (and no owner lookup) when both billers are disabled. - # ======================================================================= - credit_service = WebCrawlCreditService(session) - owner_user_id = user_id - if credit_service.billing_enabled() or credit_service.captcha_billing_enabled(): - owner_user_id = ( - await _resolve_workspace_owner(session, workspace_id) or user_id - ) - required_micros = 0 - if credit_service.billing_enabled(): - required_micros += credit_service.successes_to_micros(len(urls)) - # Only reserve captcha budget when solving is actually enabled; - # with solving off, attempts can never happen, so reserving would - # wrongly block runs for captcha that will never be attempted. - if credit_service.captcha_billing_enabled() and captcha_enabled(): - worst_case_attempts = len(urls) * config.CAPTCHA_MAX_ATTEMPTS_PER_URL - required_micros += credit_service.captcha_solves_to_micros( - worst_case_attempts - ) - try: - await credit_service.check_balance(owner_user_id, required_micros) - except InsufficientCreditsError as credit_error: - await task_logger.log_task_failure( - log_entry, - f"Insufficient crawl credit for connector {connector_id}", - str(credit_error), - {"error_type": "InsufficientCreditsError"}, - ) - logger.warning( - f"Skipping crawl for connector {connector_id}: {credit_error!s}" - ) - return 0, f"Insufficient crawl credit: {credit_error!s}" - - await task_logger.log_task_progress( - log_entry, - f"Starting to process {len(urls)} URLs", - { - "stage": "processing", - "total_urls": len(urls), - }, - ) - - documents_indexed = 0 - documents_updated = 0 - documents_skipped = 0 - documents_failed = 0 - duplicate_content_count = 0 - # Explicit "crawl succeeded" count: one per URL that yielded usable - # extracted content, independent of downstream KB dedupe/unchanged - # bucketing. This is the billable unit Phase 3c meters on. - crawls_succeeded = 0 - # Phase 3d: per-attempt captcha telemetry summed across the batch. Only - # accumulated when captcha billing is on (avoids touching the outcome - # field otherwise — and keeps MagicMock outcomes in tests arithmetic-safe). - total_captcha_attempts = 0 - total_captcha_solved = 0 - - # Heartbeat tracking - update notification periodically to prevent appearing stuck - last_heartbeat_time = time.time() - - # ======================================================================= - # PHASE 1: Analyze all URLs, create pending documents for new ones - # This makes ALL new documents visible in the UI immediately with pending status - # ======================================================================= - urls_to_process = [] # List of dicts with document and URL data - new_documents_created = False - - for url in urls: - try: - # Generate unique identifier hash for this URL - unique_identifier_hash = generate_unique_identifier_hash( - DocumentType.CRAWLED_URL, url, workspace_id - ) - - # Check if document with this unique identifier already exists - existing_document = await check_document_by_unique_identifier( - session, unique_identifier_hash - ) - - if existing_document: - # Document exists - check if it's already being processed - if DocumentStatus.is_state( - existing_document.status, DocumentStatus.PENDING - ): - logger.info(f"URL {url} already pending. Skipping.") - documents_skipped += 1 - continue - if DocumentStatus.is_state( - existing_document.status, DocumentStatus.PROCESSING - ): - logger.info(f"URL {url} already processing. Skipping.") - documents_skipped += 1 - continue - - # Queue existing document for potential update check - urls_to_process.append( - { - "document": existing_document, - "is_new": False, - "url": url, - "unique_identifier_hash": unique_identifier_hash, - } - ) - continue - - # Create new document with PENDING status (visible in UI immediately) - document = Document( - workspace_id=workspace_id, - title=url[:100], # Placeholder - URL as title (truncated) - document_type=DocumentType.CRAWLED_URL, - document_metadata={ - "url": url, - "connector_id": connector_id, - }, - content="Pending crawl...", # Placeholder content - content_hash=unique_identifier_hash, # Temporary unique value - unique_identifier_hash=unique_identifier_hash, - embedding=None, - chunks=[], # Empty at creation - safe for async - status=DocumentStatus.pending(), # PENDING status - visible in UI - updated_at=get_current_timestamp(), - created_by_id=user_id, - connector_id=connector_id, - ) - session.add(document) - new_documents_created = True - - urls_to_process.append( - { - "document": document, - "is_new": True, - "url": url, - "unique_identifier_hash": unique_identifier_hash, - } - ) - - except Exception as e: - logger.error(f"Error in Phase 1 for URL {url}: {e!s}", exc_info=True) - documents_failed += 1 - continue - - # Commit all pending documents - they all appear in UI now - if new_documents_created: - logger.info( - f"Phase 1: Committing {len([u for u in urls_to_process if u['is_new']])} pending documents" - ) - await session.commit() - - # ======================================================================= - # PHASE 2: Process each URL one by one - # Each document transitions: pending → processing → ready/failed - # ======================================================================= - logger.info(f"Phase 2: Processing {len(urls_to_process)} URLs") - - for item in urls_to_process: - # Send heartbeat periodically - if on_heartbeat_callback: - current_time = time.time() - if current_time - last_heartbeat_time >= HEARTBEAT_INTERVAL_SECONDS: - await on_heartbeat_callback(documents_indexed + documents_updated) - last_heartbeat_time = current_time - - document = item["document"] - url = item["url"] - is_new = item["is_new"] - - try: - # Set to PROCESSING and commit - shows "processing" in UI for THIS document only - document.status = DocumentStatus.processing() - await session.commit() - - await task_logger.log_task_progress( - log_entry, - f"Crawling URL: {url}", - { - "stage": "crawling_url", - "url": url, - }, - ) - - # Crawl the URL - outcome = await crawler.crawl_url(url) - - # Phase 3d: count captcha attempts BEFORE the success branch — a - # failed solve still cost real solver money and must bill. - if credit_service.captcha_billing_enabled(): - total_captcha_attempts += outcome.captcha_attempts - if outcome.captcha_solved: - total_captcha_solved += 1 - - if ( - outcome.status is not CrawlOutcomeStatus.SUCCESS - or not outcome.result - ): - logger.warning(f"Failed to crawl URL {url}: {outcome.error}") - document.status = DocumentStatus.failed( - outcome.error or "Crawl failed" - ) - document.updated_at = get_current_timestamp() - await session.commit() - documents_failed += 1 - continue - - # A tier extracted usable content: count the successful crawl now, - # before the dedupe/unchanged branches (those still represent a - # successful crawl and must bill — see 03c). - crawls_succeeded += 1 - crawl_result = outcome.result - - # Extract content and metadata - content = crawl_result.get("content", "") - metadata = crawl_result.get("metadata", {}) - crawler_type = crawl_result.get("crawler_type", "unknown") - - if not content.strip(): - logger.warning(f"Skipping URL with no content: {url}") - document.status = DocumentStatus.failed("No content extracted") - document.updated_at = get_current_timestamp() - await session.commit() - documents_failed += 1 - continue - - # Format content as structured document for summary generation - crawler.format_to_structured_document(crawl_result) - - # Generate content hash using a version WITHOUT metadata - structured_document_for_hash = crawler.format_to_structured_document( - crawl_result, exclude_metadata=True - ) - content_hash = generate_content_hash( - structured_document_for_hash, workspace_id - ) - - # Extract useful metadata - title = metadata.get("title", url) - metadata.get("description", "") - metadata.get("language", "") - - # Update title immediately for better UX - document.title = title - await session.commit() - - # For existing documents, check if content has changed - if not is_new and document.content_hash == content_hash: - logger.info(f"Document for URL {url} unchanged. Marking as ready.") - # Ensure status is ready (might have been stuck) - document.status = DocumentStatus.ready() - await session.commit() - documents_skipped += 1 - continue - - # For new documents, check if duplicate content exists elsewhere - if is_new: - with session.no_autoflush: - duplicate_by_content = await check_duplicate_document_by_hash( - session, content_hash - ) - - if duplicate_by_content: - logger.info( - f"URL {url} already indexed by another connector " - f"(existing document ID: {duplicate_by_content.id}). " - f"Marking as failed." - ) - document.status = DocumentStatus.failed( - "Duplicate content exists" - ) - document.updated_at = get_current_timestamp() - await session.commit() - duplicate_content_count += 1 - documents_skipped += 1 - continue - - # Select deterministic document content - - summary_content = f"Crawled URL: {title}\n\nURL: {url}\n\n{content}" - summary_embedding = embed_text(summary_content) - - # Process chunks - chunks = await create_document_chunks(content) - - # Update document to READY with actual content - document.title = title - document.content = summary_content - document.content_hash = content_hash - document.embedding = summary_embedding - document.document_metadata = { - **metadata, - "crawler_type": crawler_type, - "indexed_at": datetime.now().strftime("%Y-%m-%d %H:%M:%S"), - "connector_id": connector_id, - } - await safe_set_chunks(session, document, chunks) - document.status = DocumentStatus.ready() # READY status - document.updated_at = get_current_timestamp() - - if is_new: - documents_indexed += 1 - else: - documents_updated += 1 - - logger.info(f"Successfully processed URL {url}") - - # Batch commit every 10 documents (for ready status updates) - if (documents_indexed + documents_updated) % 10 == 0: - logger.info( - f"Committing batch: {documents_indexed + documents_updated} URLs processed so far" - ) - await session.commit() - - except Exception as e: - logger.error(f"Error processing URL {url}: {e!s}", exc_info=True) - # Mark document as failed with reason (visible in UI) - try: - document.status = DocumentStatus.failed(str(e)[:200]) - document.updated_at = get_current_timestamp() - await session.commit() - except Exception as status_error: - logger.error( - f"Failed to update document status to failed: {status_error}" - ) - documents_failed += 1 - continue - - total_processed = documents_indexed + documents_updated - - # CRITICAL: Always update timestamp (even if 0 documents indexed) so Zero syncs - await update_connector_last_indexed(session, connector, update_last_indexed) - - # Final commit for any remaining documents not yet committed in batches - logger.info( - f"Final commit: Total {documents_indexed} new, {documents_updated} updated URLs processed" - ) - try: - await session.commit() - logger.info( - "Successfully committed all webcrawler document changes to database" - ) - except Exception as e: - # Handle any remaining integrity errors gracefully - if "duplicate key value violates unique constraint" in str(e).lower(): - logger.warning( - f"Duplicate content_hash detected during final commit. " - f"Rolling back and continuing. Error: {e!s}" - ) - await session.rollback() - else: - raise - - # ======================================================================= - # Phase 3c: charge the workspace owner for successful crawls. Audit row - # is staged first (add only); charge_credits' commit flushes both the - # ``web_crawl`` TokenUsage row and the balance debit in one transaction. - # Skipped when billing is disabled or nothing crawled successfully. - # ======================================================================= - if credit_service.billing_enabled() and crawls_succeeded > 0: - await record_token_usage( - session, - usage_type="web_crawl", - workspace_id=workspace_id, - user_id=owner_user_id, - cost_micros=credit_service.successes_to_micros(crawls_succeeded), - call_details={ - "urls": len(urls), - "successes": crawls_succeeded, - "connector_id": connector_id, - }, - message_id=None, - ) - await credit_service.charge_credits(owner_user_id, crawls_succeeded) - - # ======================================================================= - # Phase 3d: charge the workspace owner per captcha *attempt* (separate - # unit; charged even when the crawl ultimately failed). Same stage + - # owner as the crawl charge above. - # ======================================================================= - if credit_service.captcha_billing_enabled() and total_captcha_attempts > 0: - await record_token_usage( - session, - usage_type="web_crawl_captcha", - workspace_id=workspace_id, - user_id=owner_user_id, - cost_micros=credit_service.captcha_solves_to_micros( - total_captcha_attempts - ), - call_details={ - "attempts": total_captcha_attempts, - "solved": total_captcha_solved, - "connector_id": connector_id, - }, - message_id=None, - ) - await credit_service.charge_captcha(owner_user_id, total_captcha_attempts) - - # Build warning message if there were issues - warning_parts = [] - if duplicate_content_count > 0: - warning_parts.append(f"{duplicate_content_count} duplicate") - if documents_failed > 0: - warning_parts.append(f"{documents_failed} failed") - warning_message = ", ".join(warning_parts) if warning_parts else None - - await task_logger.log_task_success( - log_entry, - f"Successfully completed crawled web page indexing for connector {connector_id}", - { - "urls_processed": total_processed, - "crawls_succeeded": crawls_succeeded, - "documents_indexed": documents_indexed, - "documents_updated": documents_updated, - "documents_skipped": documents_skipped, - "documents_failed": documents_failed, - "duplicate_content_count": duplicate_content_count, - }, - ) - - logger.info( - f"Web page indexing completed: {documents_indexed} new, " - f"{documents_updated} updated, {documents_skipped} skipped, " - f"{documents_failed} failed" - ) - - if warning_message: - return total_processed, f"Completed with issues: {warning_message}" - - return total_processed, None - - except SQLAlchemyError as db_error: - await session.rollback() - await task_logger.log_task_failure( - log_entry, - f"Database error during web page indexing for connector {connector_id}", - str(db_error), - {"error_type": "SQLAlchemyError"}, - ) - logger.error(f"Database error: {db_error!s}", exc_info=True) - return 0, f"Database error: {db_error!s}" - except Exception as e: - await session.rollback() - await task_logger.log_task_failure( - log_entry, - f"Failed to index web page URLs for connector {connector_id}", - str(e), - {"error_type": type(e).__name__}, - ) - logger.error(f"Failed to index web page URLs: {e!s}", exc_info=True) - return 0, f"Failed to index web page URLs: {e!s}" - - -async def get_crawled_url_documents( - session: AsyncSession, - workspace_id: int, - connector_id: int | None = None, -) -> list[Document]: - """ - Get all crawled URL documents for a workspace. - - Args: - session: Database session - workspace_id: ID of the workspace - connector_id: Optional connector ID to filter by - - Returns: - List of Document objects - """ - from sqlalchemy import select - - query = select(Document).filter( - Document.workspace_id == workspace_id, - Document.document_type == DocumentType.CRAWLED_URL, - ) - - if connector_id: - query = query.filter(Document.connector_id == connector_id) - - result = await session.execute(query) - documents = result.scalars().all() - return list(documents) diff --git a/surfsense_backend/app/utils/periodic_scheduler.py b/surfsense_backend/app/utils/periodic_scheduler.py index 5cce245f8..992f5038f 100644 --- a/surfsense_backend/app/utils/periodic_scheduler.py +++ b/surfsense_backend/app/utils/periodic_scheduler.py @@ -22,7 +22,6 @@ CONNECTOR_TASK_MAP = { SearchSourceConnectorType.GITHUB_CONNECTOR: "index_github_repos", SearchSourceConnectorType.CONFLUENCE_CONNECTOR: "index_confluence_pages", SearchSourceConnectorType.ELASTICSEARCH_CONNECTOR: "index_elasticsearch_documents", - SearchSourceConnectorType.WEBCRAWLER_CONNECTOR: "index_crawled_urls", SearchSourceConnectorType.BOOKSTACK_CONNECTOR: "index_bookstack_pages", } @@ -54,20 +53,6 @@ def create_periodic_schedule( True if successful, False otherwise """ try: - # Special handling for connectors that require config validation - if connector_type == SearchSourceConnectorType.WEBCRAWLER_CONNECTOR: - from app.utils.webcrawler_utils import parse_webcrawler_urls - - config = connector_config or {} - urls = parse_webcrawler_urls(config.get("INITIAL_URLS")) - - if not urls: - logger.info( - f"Webcrawler connector {connector_id} has no URLs configured, " - "skipping first indexing run (will run when URLs are added)" - ) - return True # Return success - schedule is created, just no first run - logger.info( f"Periodic indexing enabled for connector {connector_id} " f"(frequency: {frequency_minutes} minutes). Triggering first run..." @@ -76,7 +61,6 @@ def create_periodic_schedule( from app.tasks.celery_tasks.connector_tasks import ( index_bookstack_pages_task, index_confluence_pages_task, - index_crawled_urls_task, index_elasticsearch_documents_task, index_github_repos_task, index_notion_pages_task, @@ -87,7 +71,6 @@ def create_periodic_schedule( SearchSourceConnectorType.GITHUB_CONNECTOR: index_github_repos_task, SearchSourceConnectorType.CONFLUENCE_CONNECTOR: index_confluence_pages_task, SearchSourceConnectorType.ELASTICSEARCH_CONNECTOR: index_elasticsearch_documents_task, - SearchSourceConnectorType.WEBCRAWLER_CONNECTOR: index_crawled_urls_task, SearchSourceConnectorType.BOOKSTACK_CONNECTOR: index_bookstack_pages_task, } diff --git a/surfsense_backend/app/utils/validators.py b/surfsense_backend/app/utils/validators.py index 4511fb6ac..187cba21c 100644 --- a/surfsense_backend/app/utils/validators.py +++ b/surfsense_backend/app/utils/validators.py @@ -469,14 +469,6 @@ def validate_connector_config( if not isinstance(value, list) or not value: raise ValueError(f"{field_name} must be a non-empty list of strings") - def validate_initial_urls() -> None: - initial_urls = config.get("INITIAL_URLS", "") - if initial_urls and initial_urls.strip(): - urls = [url.strip() for url in initial_urls.split("\n") if url.strip()] - for url in urls: - if not validators.url(url): - raise ValueError(f"Invalid URL format in INITIAL_URLS: {url}") - # Lookup table for connector validation rules connector_rules = { "SERPER_API": {"required": ["SERPER_API_KEY"], "validators": {}}, @@ -562,13 +554,6 @@ def validate_connector_config( # "validators": {} # }, "LUMA_CONNECTOR": {"required": ["LUMA_API_KEY"], "validators": {}}, - "WEBCRAWLER_CONNECTOR": { - "required": [], # No required fields - URLs are optional - "optional": ["INITIAL_URLS"], - "validators": { - "INITIAL_URLS": lambda: validate_initial_urls(), - }, - }, } rules = connector_rules.get(connector_type_str) diff --git a/surfsense_backend/app/utils/webcrawler_utils.py b/surfsense_backend/app/utils/webcrawler_utils.py deleted file mode 100644 index 31d2ebe50..000000000 --- a/surfsense_backend/app/utils/webcrawler_utils.py +++ /dev/null @@ -1,28 +0,0 @@ -""" -Utility functions for webcrawler connector. -""" - - -def parse_webcrawler_urls(initial_urls: str | list | None) -> list[str]: - """ - Parse URLs from webcrawler INITIAL_URLS value. - - Handles both string (newline-separated) and list formats. - - Args: - initial_urls: The INITIAL_URLS value (string, list, or None) - - Returns: - List of parsed, stripped, non-empty URLs - """ - if initial_urls is None: - return [] - - if isinstance(initial_urls, str): - return [url.strip() for url in initial_urls.split("\n") if url.strip()] - elif isinstance(initial_urls, list): - return [ - url.strip() for url in initial_urls if isinstance(url, str) and url.strip() - ] - else: - return [] diff --git a/surfsense_backend/scripts/e2e_phase3_crawl_billing.py b/surfsense_backend/scripts/e2e_phase3_crawl_billing.py index 0d51b238b..31b8c832b 100644 --- a/surfsense_backend/scripts/e2e_phase3_crawl_billing.py +++ b/surfsense_backend/scripts/e2e_phase3_crawl_billing.py @@ -10,13 +10,6 @@ What it exercises (everything REAL — live network, live proxy, live DB reads): Stage 1 (3a + 3b) — direct fetch + proxy egress-IP proof + crawl_url ladder. Stage 2 (3c chat surface) — the scrape_webpage tool folds one successful crawl into the current chat turn's accumulator (billed at finalize). - Stage 3 (3c connector surface) — index_crawled_urls debits the WORKSPACE - OWNER per successful crawl and writes one `web_crawl` TokenUsage row. - -SAFETY: Stage 3 creates a scratch user/workspace/connector inside an outer -transaction that is ALWAYS rolled back (``join_transaction_mode= -"create_savepoint"``), so nothing persists to your database. The only real -side effect is a handful of HTTP requests (small proxy spend). This is NOT a pytest test (it needs a live stack + proxy creds + network). It is the manual functional counterpart to the unit suites; the undetectability / @@ -25,7 +18,6 @@ anti-bot scorecard is a separate deliverable (03f), after 03d/03e. import asyncio import sys -import uuid from pathlib import Path from urllib.parse import urlsplit @@ -39,8 +31,6 @@ for _candidate in (_BACKEND_ROOT / ".env", _BACKEND_ROOT.parent / ".env"): load_dotenv(_candidate) break -from sqlalchemy import select, text # noqa: E402 -from sqlalchemy.ext.asyncio import AsyncSession # noqa: E402 from app.config import config # noqa: E402 @@ -93,7 +83,7 @@ async def stage1_crawl_and_proxy() -> bool: try: direct = await AsyncFetcher.get(_IP_ECHO, impersonate="chrome", timeout=30) direct_ip = direct.json().get("ip") - except Exception as exc: # noqa: BLE001 + except Exception as exc: print(f" [INFO] direct IP fetch failed: {exc}") if proxy_url: try: @@ -101,7 +91,7 @@ async def stage1_crawl_and_proxy() -> bool: _IP_ECHO, impersonate="chrome", proxy=proxy_url, timeout=45 ) proxied_ip = via.json().get("ip") - except Exception as exc: # noqa: BLE001 + except Exception as exc: print(f" [INFO] proxied IP fetch failed: {exc}") print(f" egress IP (direct) : {direct_ip}") print(f" egress IP (via proxy) : {proxied_ip}") @@ -158,176 +148,16 @@ async def stage2_chat_fold() -> bool: ) -# =========================================================================== -# Stage 3 — connector indexer bills the workspace owner (3c surface 1) -# =========================================================================== -async def stage3_indexer_billing() -> bool: - _hr("STAGE 3 — index_crawled_urls bills workspace owner (3c) [rolled back]") - config.WEB_CRAWL_CREDIT_BILLING_ENABLED = True - price = config.WEB_CRAWL_MICROS_PER_SUCCESS - start_balance = 10_000_000 # $10 — plenty for the pre-flight check - - from app.db import ( - Base, - SearchSourceConnector, - SearchSourceConnectorType, - TokenUsage, - User, - Workspace, - engine, - ) - from app.tasks.connector_indexers.webcrawler_indexer import index_crawled_urls - - # Self-bootstrap: if the configured DB has no schema (e.g. an empty - # surfsense_test), create it like the integration harness. No-op against an - # already-migrated DB. DDL is outside the rolled-back txn, so tables persist - # but the scratch rows below do not. - async with engine.connect() as probe: - has_schema = ( - await probe.execute(text("select to_regclass('public.\"user\"')")) - ).scalar() is not None - if not has_schema: - print(" [INFO] empty database — creating schema (vector ext + create_all)") - async with engine.begin() as ddl: - await ddl.execute(text("CREATE EXTENSION IF NOT EXISTS vector")) - await ddl.run_sync(Base.metadata.create_all) - - async with engine.connect() as conn: - outer = await conn.begin() - try: - async with AsyncSession( - bind=conn, - expire_on_commit=False, - join_transaction_mode="create_savepoint", - ) as session: - owner = User( - id=uuid.uuid4(), - email=f"e2e-phase3+{uuid.uuid4().hex[:8]}@surfsense.test", - hashed_password="not-a-real-hash", - is_active=True, - is_superuser=False, - is_verified=True, - credit_micros_balance=start_balance, - ) - session.add(owner) - await session.flush() - - # A DISTINCT triggering user (real row — documents.created_by_id - # FKs to it) to prove the OWNER, not the trigger, gets billed. - trigger = User( - id=uuid.uuid4(), - email=f"e2e-trigger+{uuid.uuid4().hex[:8]}@surfsense.test", - hashed_password="not-a-real-hash", - is_active=True, - is_superuser=False, - is_verified=True, - credit_micros_balance=start_balance, - ) - session.add(trigger) - await session.flush() - - ws = Workspace(name="E2E Phase3 Scratch", user_id=owner.id) - session.add(ws) - await session.flush() - - connector = SearchSourceConnector( - name="E2E WebCrawler Scratch", - connector_type=SearchSourceConnectorType.WEBCRAWLER_CONNECTOR, - config={"INITIAL_URLS": _ARTICLE_URLS}, - is_indexable=True, - workspace_id=ws.id, - user_id=owner.id, - ) - session.add(connector) - await session.flush() - - # Snapshot plain values before the indexer's commits so post-run - # reads never lazy-load an expired ORM attribute. - owner_id = owner.id - trigger_id = trigger.id - ws_id = ws.id - connector_id = connector.id - print(f" owner user id : {owner_id}") - print(f" triggering user id : {trigger_id}") - print(f" urls : {len(_ARTICLE_URLS)}") - - total, warning = await index_crawled_urls( - session, connector_id, ws_id, str(trigger_id) - ) - await session.refresh(owner) - await session.refresh(trigger) - debit = start_balance - owner.credit_micros_balance - trigger_debit = start_balance - trigger.credit_micros_balance - - rows = ( - ( - await session.execute( - select(TokenUsage).where( - TokenUsage.usage_type == "web_crawl", - TokenUsage.workspace_id == ws_id, - ) - ) - ) - .scalars() - .all() - ) - - print(f" index result : processed={total} warning={warning}") - print(f" owner debit (micros) : {debit}") - print(f" trigger debit (micros): {trigger_debit}") - print(f" web_crawl usage rows : {len(rows)}") - - ok = True - if not rows: - print( - " [INFO] 0 successful crawls (site/proxy blocked) — " - "nothing billed; cannot assert debit" - ) - return False - - row = rows[0] - successes = (row.call_details or {}).get("successes") - print( - f" audit row : cost_micros={row.cost_micros} " - f"successes={successes} user_id={row.user_id}" - ) - ok &= _check("exactly one web_crawl audit row", len(rows) == 1) - ok &= _check( - "audit billed to OWNER (not trigger user)", - str(row.user_id) == str(owner_id), - ) - ok &= _check( - "triggering user NOT debited", - trigger_debit == 0, - f"trigger_debit={trigger_debit}", - ) - ok &= _check( - "cost == successes * configured price", - row.cost_micros == successes * price, - f"{row.cost_micros} == {successes} * {price}", - ) - ok &= _check( - "wallet debit matches audit cost", - debit == row.cost_micros, - f"debit={debit} cost={row.cost_micros}", - ) - return ok - finally: - await outer.rollback() - print(" [INFO] transaction rolled back — no scratch rows persisted") - - async def main() -> int: print("Phase 3 functional e2e (3a/3b/3c) — live network + proxy, DB rolled back") results: dict[str, bool] = {} for name, coro in ( ("Stage 1 crawl+proxy", stage1_crawl_and_proxy), ("Stage 2 chat fold", stage2_chat_fold), - ("Stage 3 indexer billing", stage3_indexer_billing), ): try: results[name] = await coro() - except Exception as exc: # noqa: BLE001 + except Exception as exc: import traceback traceback.print_exc() diff --git a/surfsense_backend/tests/unit/connector_indexers/test_web_crawl_billing.py b/surfsense_backend/tests/unit/connector_indexers/test_web_crawl_billing.py deleted file mode 100644 index 5f8dbdabc..000000000 --- a/surfsense_backend/tests/unit/connector_indexers/test_web_crawl_billing.py +++ /dev/null @@ -1,306 +0,0 @@ -"""Phase 3c crawl-billing wiring in the webcrawler indexer. - -System boundaries mocked: DB session, TaskLoggingService, the crawler, and the -document-conversion helpers. NOT mocked: WebCrawlCreditService (our own code) -and the indexer's billing decisions (owner resolution, pre-flight gate, -audit-then-charge, success counting). -""" - -from unittest.mock import AsyncMock, MagicMock -from uuid import UUID - -import pytest - -import app.tasks.connector_indexers.webcrawler_indexer as _mod -from app.config import config -from app.proprietary.web_crawler import CrawlOutcomeStatus - -pytestmark = pytest.mark.unit - -_CONNECTOR_ID = 7 -_WORKSPACE_ID = 1 -_TRIGGER_USER = "00000000-0000-0000-0000-0000000000aa" -_OWNER_USER = UUID("00000000-0000-0000-0000-0000000000bb") - - -class _FakeUser: - def __init__(self, balance_micros: int, reserved_micros: int = 0): - self.credit_micros_balance = balance_micros - self.credit_micros_reserved = reserved_micros - - -def _make_session(owner_id, balance_micros, reserved_micros=0): - """Mock session serving owner-resolution, get_available_micros and charge.""" - fake_user = _FakeUser(balance_micros, reserved_micros) - session = AsyncMock() - session.add = MagicMock() - session.no_autoflush = MagicMock() - - def _make_result(*_args, **_kwargs): - result = MagicMock() - # _resolve_workspace_owner → select(Workspace.user_id).scalar_one_or_none() - result.scalar_one_or_none.return_value = owner_id - # get_available_micros → .first() → (balance, reserved) - result.first.return_value = ( - fake_user.credit_micros_balance, - fake_user.credit_micros_reserved, - ) - # charge_credits → .unique().scalar_one_or_none() → User - result.unique.return_value.scalar_one_or_none.return_value = fake_user - return result - - session.execute = AsyncMock(side_effect=_make_result) - return session, fake_user - - -def _outcome( - success: bool, - content: str = "Hello content", - captcha_attempts: int = 0, - captcha_solved: bool = False, -): - o = MagicMock() - if success: - o.status = CrawlOutcomeStatus.SUCCESS - o.result = { - "content": content, - "metadata": {"title": "Title"}, - "crawler_type": "scrapling-static", - } - o.error = None - else: - o.status = CrawlOutcomeStatus.FAILED - o.result = None - o.error = "blocked" - # Real ints/bools (not MagicMock) so 03d accumulation arithmetic is safe. - o.captcha_attempts = captcha_attempts - o.captcha_solved = captcha_solved - return o - - -@pytest.fixture -def indexer_env(monkeypatch): - """Patch every system boundary the indexer touches; return the handles.""" - monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", True) - monkeypatch.setattr(config, "WEB_CRAWL_MICROS_PER_SUCCESS", 1000) - - # Task logger: async no-op methods, log_task_start returns a sentinel. - task_logger = MagicMock() - task_logger.log_task_start = AsyncMock(return_value=MagicMock()) - task_logger.log_task_progress = AsyncMock() - task_logger.log_task_failure = AsyncMock() - task_logger.log_task_success = AsyncMock() - monkeypatch.setattr(_mod, "TaskLoggingService", MagicMock(return_value=task_logger)) - - # Connector + URL parsing. - connector = MagicMock() - connector.name = "wc" - monkeypatch.setattr(_mod, "get_connector_by_id", AsyncMock(return_value=connector)) - monkeypatch.setattr(_mod, "parse_webcrawler_urls", lambda raw: list(raw)) - - # Crawler. - crawler = MagicMock() - crawler.crawl_url = AsyncMock() - crawler.format_to_structured_document = MagicMock(return_value="doc") - monkeypatch.setattr(_mod, "WebCrawlerConnector", MagicMock(return_value=crawler)) - - # Document-conversion + persistence helpers (system boundary). - monkeypatch.setattr( - _mod, "check_document_by_unique_identifier", AsyncMock(return_value=None) - ) - monkeypatch.setattr( - _mod, "check_duplicate_document_by_hash", AsyncMock(return_value=None) - ) - monkeypatch.setattr(_mod, "embed_text", lambda *_a, **_k: [0.1, 0.2]) - monkeypatch.setattr(_mod, "create_document_chunks", AsyncMock(return_value=[])) - monkeypatch.setattr(_mod, "safe_set_chunks", AsyncMock()) - monkeypatch.setattr(_mod, "update_connector_last_indexed", AsyncMock()) - - # Audit helper — assert it's the owner who is billed. - record_usage = AsyncMock(return_value=MagicMock()) - monkeypatch.setattr(_mod, "record_token_usage", record_usage) - - return { - "connector": connector, - "crawler": crawler, - "record_usage": record_usage, - "task_logger": task_logger, - } - - -async def _run(env, session, urls): - env["connector"].config = {"INITIAL_URLS": urls} - return await _mod.index_crawled_urls( - session, - _CONNECTOR_ID, - _WORKSPACE_ID, - _TRIGGER_USER, - ) - - -async def test_charges_owner_for_successful_crawls(indexer_env): - session, user = _make_session(_OWNER_USER, balance_micros=100_000) - indexer_env["crawler"].crawl_url.side_effect = [_outcome(True), _outcome(True)] - - total, warning = await _run(indexer_env, session, ["https://a.com", "https://b.com"]) - - assert total == 2 - assert warning is None - # Owner debited 2 * 1000. - assert user.credit_micros_balance == 100_000 - 2000 - # One audit row, billed to the OWNER (not the triggering user), correct cost. - indexer_env["record_usage"].assert_awaited_once() - kwargs = indexer_env["record_usage"].await_args.kwargs - assert kwargs["usage_type"] == "web_crawl" - assert kwargs["user_id"] == _OWNER_USER - assert kwargs["workspace_id"] == _WORKSPACE_ID - assert kwargs["cost_micros"] == 2000 - - -async def test_preflight_blocks_run_when_insufficient(indexer_env): - # balance 1000 < 3 URLs * 1000 required. - session, user = _make_session(_OWNER_USER, balance_micros=1000) - - total, warning = await _run( - indexer_env, session, ["https://a.com", "https://b.com", "https://c.com"] - ) - - assert total == 0 - assert "insufficient crawl credit" in warning.lower() - # Never crawled, never billed, balance untouched. - indexer_env["crawler"].crawl_url.assert_not_awaited() - indexer_env["record_usage"].assert_not_awaited() - assert user.credit_micros_balance == 1000 - - -async def test_failed_crawls_are_free(indexer_env): - session, user = _make_session(_OWNER_USER, balance_micros=100_000) - indexer_env["crawler"].crawl_url.side_effect = [_outcome(False)] - - total, _warning = await _run(indexer_env, session, ["https://a.com"]) - - assert total == 0 - # crawls_succeeded == 0 → no audit, no charge. - indexer_env["record_usage"].assert_not_awaited() - assert user.credit_micros_balance == 100_000 - - -async def test_disabled_skips_billing_but_still_crawls(indexer_env, monkeypatch): - monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", False) - session, user = _make_session(_OWNER_USER, balance_micros=100_000) - indexer_env["crawler"].crawl_url.side_effect = [_outcome(True)] - - total, _warning = await _run(indexer_env, session, ["https://a.com"]) - - assert total == 1 - indexer_env["crawler"].crawl_url.assert_awaited_once() - indexer_env["record_usage"].assert_not_awaited() - assert user.credit_micros_balance == 100_000 - - -# =================================================================== -# Phase 3d — captcha per-attempt billing in the indexer -# =================================================================== - - -async def test_captcha_attempts_charged_to_owner(indexer_env, monkeypatch): - """Captcha billed per attempt to the OWNER as a separate web_crawl_captcha - unit. Crawl billing OFF here to isolate the captcha charge.""" - monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", False) - monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_BILLING_ENABLED", True) - monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE", 3000) - session, user = _make_session(_OWNER_USER, balance_micros=100_000) - indexer_env["crawler"].crawl_url.side_effect = [ - _outcome(True, captcha_attempts=1, captcha_solved=True), - _outcome(True, captcha_attempts=1, captcha_solved=True), - ] - - total, _warning = await _run( - indexer_env, session, ["https://a.com", "https://b.com"] - ) - - assert total == 2 - # Owner debited 2 attempts * 3000 (no crawl charge — crawl billing off). - assert user.credit_micros_balance == 100_000 - 6000 - indexer_env["record_usage"].assert_awaited_once() - kwargs = indexer_env["record_usage"].await_args.kwargs - assert kwargs["usage_type"] == "web_crawl_captcha" - assert kwargs["user_id"] == _OWNER_USER - assert kwargs["cost_micros"] == 6000 - assert kwargs["call_details"]["attempts"] == 2 - assert kwargs["call_details"]["solved"] == 2 - - -async def test_captcha_attempt_billed_even_when_crawl_fails(indexer_env, monkeypatch): - """A failed solve still cost solver money → bill the attempt anyway.""" - monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", False) - monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_BILLING_ENABLED", True) - monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE", 3000) - session, user = _make_session(_OWNER_USER, balance_micros=100_000) - indexer_env["crawler"].crawl_url.side_effect = [ - _outcome(False, captcha_attempts=1, captcha_solved=False) - ] - - total, _warning = await _run(indexer_env, session, ["https://a.com"]) - - assert total == 0 # crawl failed → not indexed - assert user.credit_micros_balance == 100_000 - 3000 # but solve attempt billed - kwargs = indexer_env["record_usage"].await_args.kwargs - assert kwargs["usage_type"] == "web_crawl_captcha" - assert kwargs["call_details"]["attempts"] == 1 - assert kwargs["call_details"]["solved"] == 0 - - -async def test_captcha_billing_on_but_solving_off_does_not_overreserve( - indexer_env, monkeypatch -): - """Regression: captcha billing ON but solving OFF must NOT reserve the - worst-case captcha budget in pre-flight. - - With solving off, attempts can never happen, so a balance below the - (would-be) worst-case reservation must still let the run proceed rather than - being wrongly blocked for captcha that never runs. - """ - monkeypatch.setattr(config, "WEB_CRAWL_CREDIT_BILLING_ENABLED", False) - monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_BILLING_ENABLED", True) - monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE", 3000) - monkeypatch.setattr(config, "CAPTCHA_MAX_ATTEMPTS_PER_URL", 3) - # Solving OFF → captcha_enabled() is False (no flag + no key). - monkeypatch.setattr(config, "CAPTCHA_SOLVING_ENABLED", False) - monkeypatch.setattr(config, "CAPTCHA_SOLVER_API_KEY", "") - - # Balance (5000) is below the buggy worst-case reservation - # (2 URLs * 3 attempts * 3000 = 18000) but the run must still proceed. - session, user = _make_session(_OWNER_USER, balance_micros=5000) - indexer_env["crawler"].crawl_url.side_effect = [ - _outcome(True, captcha_attempts=0), - _outcome(True, captcha_attempts=0), - ] - - total, warning = await _run( - indexer_env, session, ["https://a.com", "https://b.com"] - ) - - assert total == 2 - assert warning is None - assert indexer_env["crawler"].crawl_url.await_count == 2 - # Nothing billed: crawl billing off, no captcha attempts. - indexer_env["record_usage"].assert_not_awaited() - assert user.credit_micros_balance == 5000 - - -async def test_captcha_disabled_skips_captcha_billing(indexer_env, monkeypatch): - """Captcha billing OFF (default) → attempts on the outcome are ignored.""" - monkeypatch.setattr(config, "WEB_CRAWL_CAPTCHA_BILLING_ENABLED", False) - session, user = _make_session(_OWNER_USER, balance_micros=100_000) - indexer_env["crawler"].crawl_url.side_effect = [ - _outcome(True, captcha_attempts=5, captcha_solved=True) - ] - - total, _warning = await _run(indexer_env, session, ["https://a.com"]) - - assert total == 1 - # Only the crawl unit billed (1 * 1000); captcha ignored entirely. - assert user.credit_micros_balance == 100_000 - 1000 - kwargs = indexer_env["record_usage"].await_args.kwargs - assert kwargs["usage_type"] == "web_crawl" diff --git a/surfsense_backend/tests/unit/utils/test_validators.py b/surfsense_backend/tests/unit/utils/test_validators.py index 82846bdb9..a359ee550 100644 --- a/surfsense_backend/tests/unit/utils/test_validators.py +++ b/surfsense_backend/tests/unit/utils/test_validators.py @@ -11,10 +11,10 @@ from app.utils.validators import ( validate_messages, validate_research_mode, validate_search_mode, - validate_workspace_id, validate_top_k, validate_url, validate_uuid, + validate_workspace_id, ) pytestmark = pytest.mark.unit @@ -331,9 +331,3 @@ def test_validate_connector_config_invalid(): # Invalid URL format in SEARXNG_API with pytest.raises(ValueError): validate_connector_config("SEARXNG_API", {"SEARXNG_HOST": "not-a-url"}) - - # WEBCRAWLER_CONNECTOR custom validation: malformed INITIAL_URLS rejected. - with pytest.raises(ValueError): - validate_connector_config( - "WEBCRAWLER_CONNECTOR", {"INITIAL_URLS": "not-a-url"} - ) From 817ec0e9f4a0e17c09983cf93c24e93cc20770d4 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Fri, 3 Jul 2026 12:00:22 +0200 Subject: [PATCH 58/64] refactor(youtube): remove legacy KB ingestion; use actor get_youtube_video_id --- .../main_agent/tools/scrape_webpage.py | 2 +- .../platforms/youtube/subtitles.py | 2 +- .../platforms/youtube/url_resolver.py | 3 +- .../app/routes/documents_routes.py | 7 - .../app/tasks/celery_tasks/document_tasks.py | 126 ----- .../app/tasks/document_processors/__init__.py | 4 +- .../document_processors/youtube_processor.py | 473 ------------------ .../tests/unit/observability/test_helpers.py | 1 - 8 files changed, 4 insertions(+), 614 deletions(-) delete mode 100644 surfsense_backend/app/tasks/document_processors/youtube_processor.py diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/tools/scrape_webpage.py b/surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/tools/scrape_webpage.py index 82dcaa9c1..2c1173bfc 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/tools/scrape_webpage.py +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/tools/scrape_webpage.py @@ -18,11 +18,11 @@ from requests import Session from scrapling.fetchers import AsyncFetcher from youtube_transcript_api import YouTubeTranscriptApi +from app.proprietary.platforms.youtube.url_resolver import get_youtube_video_id from app.proprietary.web_crawler import ( CrawlOutcomeStatus, WebCrawlerConnector, ) -from app.tasks.document_processors.youtube_processor import get_youtube_video_id from app.utils.proxy import get_proxy_url, get_requests_proxies logger = logging.getLogger(__name__) diff --git a/surfsense_backend/app/proprietary/platforms/youtube/subtitles.py b/surfsense_backend/app/proprietary/platforms/youtube/subtitles.py index cbec49ec8..dc5647b30 100644 --- a/surfsense_backend/app/proprietary/platforms/youtube/subtitles.py +++ b/surfsense_backend/app/proprietary/platforms/youtube/subtitles.py @@ -2,7 +2,7 @@ Uses the library's built-in formatters (no hand-rolled srt/vtt conversion). The transcript API has no XML formatter, so ``xml`` falls back to the raw snippet -data. Fetch/proxy wiring mirrors ``youtube_processor.py``. +data. """ from __future__ import annotations diff --git a/surfsense_backend/app/proprietary/platforms/youtube/url_resolver.py b/surfsense_backend/app/proprietary/platforms/youtube/url_resolver.py index 43bad2e95..0e68425a9 100644 --- a/surfsense_backend/app/proprietary/platforms/youtube/url_resolver.py +++ b/surfsense_backend/app/proprietary/platforms/youtube/url_resolver.py @@ -1,8 +1,7 @@ """Classify a YouTube URL and extract its identifier. Covers the ``startUrls`` shapes the Apify spec accepts: video, channel, -playlist, hashtag, and search-results pages. Video-ID extraction reuses the -logic already in ``app/tasks/document_processors/youtube_processor.py``. +playlist, hashtag, and search-results pages. """ from __future__ import annotations diff --git a/surfsense_backend/app/routes/documents_routes.py b/surfsense_backend/app/routes/documents_routes.py index ade17a59f..6dbf65b20 100644 --- a/surfsense_backend/app/routes/documents_routes.py +++ b/surfsense_backend/app/routes/documents_routes.py @@ -98,13 +98,6 @@ async def create_documents( process_extension_document_task.delay( document_dict, request.workspace_id, str(user.id) ) - elif request.document_type == DocumentType.YOUTUBE_VIDEO: - from app.tasks.celery_tasks.document_tasks import process_youtube_video_task - - for url in request.content: - process_youtube_video_task.delay( - url, request.workspace_id, str(user.id) - ) else: raise HTTPException(status_code=400, detail="Invalid document type") diff --git a/surfsense_backend/app/tasks/celery_tasks/document_tasks.py b/surfsense_backend/app/tasks/celery_tasks/document_tasks.py index ba8a49f67..694bf6409 100644 --- a/surfsense_backend/app/tasks/celery_tasks/document_tasks.py +++ b/surfsense_backend/app/tasks/celery_tasks/document_tasks.py @@ -19,7 +19,6 @@ from app.tasks.connector_indexers.local_folder_indexer import ( ) from app.tasks.document_processors import ( add_extension_received_document, - add_youtube_video_document, ) logger = logging.getLogger(__name__) @@ -405,131 +404,6 @@ async def _process_extension_document( raise -@celery_app.task(name="process_youtube_video", bind=True) -def process_youtube_video_task(self, url: str, workspace_id: int, user_id: str): - """ - Celery task to process YouTube video. - - Args: - url: YouTube video URL - workspace_id: ID of the workspace - user_id: ID of the user - """ - return run_async_celery_task( - lambda: _process_youtube_video(url, workspace_id, user_id) - ) - - -async def _process_youtube_video(url: str, workspace_id: int, user_id: str): - """Process YouTube video with new session.""" - async with get_celery_session_maker()() as session: - task_logger = TaskLoggingService(session, workspace_id) - - # Extract video title from URL for notification (will be updated later) - video_name = url.split("v=")[-1][:11] if "v=" in url else url - - # Create notification for document processing - notification = ( - await NotificationService.document_processing.notify_processing_started( - session=session, - user_id=UUID(user_id), - document_type="YOUTUBE_VIDEO", - document_name=f"YouTube: {video_name}", - workspace_id=workspace_id, - ) - ) - - # Start Redis heartbeat for stale task detection - _start_heartbeat(notification.id) - heartbeat_task = asyncio.create_task(_run_heartbeat_loop(notification.id)) - - log_entry = await task_logger.log_task_start( - task_name="process_youtube_video", - source="document_processor", - message=f"Starting YouTube video processing for: {url}", - metadata={"document_type": "YOUTUBE_VIDEO", "url": url, "user_id": user_id}, - ) - - try: - # Update notification: parsing (fetching transcript) - await NotificationService.document_processing.notify_processing_progress( - session, - notification, - stage="parsing", - stage_message="Fetching video transcript", - ) - - result = await add_youtube_video_document( - session, url, workspace_id, user_id, notification=notification - ) - - if result: - await task_logger.log_task_success( - log_entry, - f"Successfully processed YouTube video: {result.title}", - { - "document_id": result.id, - "video_id": result.document_metadata.get("video_id"), - "content_hash": result.content_hash, - }, - ) - - # Update notification on success - await ( - NotificationService.document_processing.notify_processing_completed( - session=session, - notification=notification, - document_id=result.id, - chunks_count=None, - ) - ) - else: - await task_logger.log_task_success( - log_entry, - f"YouTube video document already exists (duplicate): {url}", - {"duplicate_detected": True}, - ) - - # Update notification for duplicate - await ( - NotificationService.document_processing.notify_processing_completed( - session=session, - notification=notification, - error_message="Video already exists (duplicate)", - ) - ) - except Exception as e: - await task_logger.log_task_failure( - log_entry, - f"Failed to process YouTube video: {url}", - str(e), - {"error_type": type(e).__name__}, - ) - - # Update notification on failure - wrapped in try-except to ensure it doesn't fail silently - try: - # Refresh notification to ensure it's not stale after any rollback - await session.refresh(notification) - await ( - NotificationService.document_processing.notify_processing_completed( - session=session, - notification=notification, - error_message=str(e)[:100], - ) - ) - except Exception as notif_error: - logger.error( - f"Failed to update notification on failure: {notif_error!s}" - ) - - logger.error(f"Error processing YouTube video: {e!s}") - raise - finally: - # Stop heartbeat — key deleted on success, expires on crash - heartbeat_task.cancel() - _stop_heartbeat(notification.id) - - @celery_app.task(name="process_file_upload", bind=True) def process_file_upload_task( self, file_path: str, filename: str, workspace_id: int, user_id: str diff --git a/surfsense_backend/app/tasks/document_processors/__init__.py b/surfsense_backend/app/tasks/document_processors/__init__.py index f82c10883..90fd69269 100644 --- a/surfsense_backend/app/tasks/document_processors/__init__.py +++ b/surfsense_backend/app/tasks/document_processors/__init__.py @@ -3,15 +3,13 @@ Document processors module for background tasks. Content extraction is handled by ``app.etl_pipeline.EtlPipelineService``. This package keeps orchestration (save, notify, page-limit) and -non-ETL processors (extension, markdown, youtube). +non-ETL processors (extension, markdown). """ from .extension_processor import add_extension_received_document from .markdown_processor import add_received_markdown_file_document -from .youtube_processor import add_youtube_video_document __all__ = [ "add_extension_received_document", "add_received_markdown_file_document", - "add_youtube_video_document", ] diff --git a/surfsense_backend/app/tasks/document_processors/youtube_processor.py b/surfsense_backend/app/tasks/document_processors/youtube_processor.py deleted file mode 100644 index 255ba4f7e..000000000 --- a/surfsense_backend/app/tasks/document_processors/youtube_processor.py +++ /dev/null @@ -1,473 +0,0 @@ -""" -YouTube video document processor. - -Implements 2-phase document status updates for real-time UI feedback: -- Phase 1: Create document with 'pending' status (visible in UI immediately) -- Phase 2: Process document: pending → processing → ready/failed -""" - -import logging -import time -from urllib.parse import parse_qs, urlparse - -from fake_useragent import UserAgent -from requests import Session -from scrapling.fetchers import AsyncFetcher -from sqlalchemy.exc import SQLAlchemyError -from sqlalchemy.ext.asyncio import AsyncSession -from youtube_transcript_api import YouTubeTranscriptApi - -from app.db import Document, DocumentStatus, DocumentType -from app.services.task_logging_service import TaskLoggingService -from app.utils.document_converters import ( - create_document_chunks, - embed_text, - generate_content_hash, - generate_unique_identifier_hash, -) -from app.utils.proxy import get_proxy_url, get_requests_proxies - -from .base import ( - check_document_by_unique_identifier, - get_current_timestamp, - safe_set_chunks, -) - - -def get_youtube_video_id(url: str) -> str | None: - """ - Extract video ID from various YouTube URL formats. - - Args: - url: YouTube URL - - Returns: - Video ID if found, None otherwise - """ - parsed_url = urlparse(url) - hostname = parsed_url.hostname - - if hostname == "youtu.be": - return parsed_url.path[1:] - if hostname in ("www.youtube.com", "youtube.com"): - if parsed_url.path == "/watch": - query_params = parse_qs(parsed_url.query) - return query_params.get("v", [None])[0] - if parsed_url.path.startswith("/embed/"): - return parsed_url.path.split("/")[2] - if parsed_url.path.startswith("/v/"): - return parsed_url.path.split("/")[2] - return None - - -async def add_youtube_video_document( - session: AsyncSession, - url: str, - workspace_id: int, - user_id: str, - notification=None, -) -> Document: - """ - Process a YouTube video URL, extract transcripts, and store as a document. - - Implements 2-phase document status updates for real-time UI feedback: - - Phase 1: Create document with 'pending' status (visible in UI immediately) - - Phase 2: Process document: pending → processing → ready/failed - - Args: - session: Database session for storing the document - url: YouTube video URL (supports standard, shortened, and embed formats) - workspace_id: ID of the workspace to add the document to - user_id: ID of the user - notification: Optional notification object — if provided, the document_id - is stored in its metadata right after document creation so the stale - cleanup task can identify stuck documents. - - Returns: - Document: The created document object - - Raises: - ValueError: If the YouTube video ID cannot be extracted from the URL - SQLAlchemyError: If there's a database error - RuntimeError: If the video processing fails - """ - task_logger = TaskLoggingService(session, workspace_id) - - # Log task start - log_entry = await task_logger.log_task_start( - task_name="youtube_video_document", - source="background_task", - message=f"Starting YouTube video processing for: {url}", - metadata={"url": url, "user_id": str(user_id)}, - ) - - document = None - video_id = None - is_new_document = False - - try: - # Extract video ID from URL (lightweight operation) - await task_logger.log_task_progress( - log_entry, - f"Extracting video ID from URL: {url}", - {"stage": "video_id_extraction"}, - ) - - video_id = get_youtube_video_id(url) - if not video_id: - raise ValueError(f"Could not extract video ID from URL: {url}") - - await task_logger.log_task_progress( - log_entry, - f"Video ID extracted: {video_id}", - {"stage": "video_id_extracted", "video_id": video_id}, - ) - - # Generate unique identifier hash for this YouTube video - unique_identifier_hash = generate_unique_identifier_hash( - DocumentType.YOUTUBE_VIDEO, video_id, workspace_id - ) - - # Check if document with this unique identifier already exists - await task_logger.log_task_progress( - log_entry, - f"Checking for existing video: {video_id}", - {"stage": "duplicate_check", "video_id": video_id}, - ) - - existing_document = await check_document_by_unique_identifier( - session, unique_identifier_hash - ) - - # ======================================================================= - # PHASE 1: Create pending document or prepare existing for update - # ======================================================================= - if existing_document: - document = existing_document - is_new_document = False - # Check if already being processed - if DocumentStatus.is_state( - existing_document.status, DocumentStatus.PENDING - ): - logging.info( - f"YouTube video {video_id} already pending. Returning existing." - ) - return existing_document - if DocumentStatus.is_state( - existing_document.status, DocumentStatus.PROCESSING - ): - logging.info( - f"YouTube video {video_id} already processing. Returning existing." - ) - return existing_document - else: - # Create new document with PENDING status (visible in UI immediately) - await task_logger.log_task_progress( - log_entry, - f"Creating pending document for video: {video_id}", - {"stage": "pending_document_creation"}, - ) - - document = Document( - title=f"YouTube Video: {video_id}", # Placeholder title - document_type=DocumentType.YOUTUBE_VIDEO, - document_metadata={ - "url": url, - "video_id": video_id, - }, - content="Processing video...", # Placeholder content - content_hash=unique_identifier_hash, # Temporary unique value - unique_identifier_hash=unique_identifier_hash, - embedding=None, - chunks=[], # Empty at creation - status=DocumentStatus.pending(), # PENDING status - visible in UI - workspace_id=workspace_id, - updated_at=get_current_timestamp(), - created_by_id=user_id, - ) - session.add(document) - await session.commit() # Document visible in UI now with pending status! - is_new_document = True - - # Store document_id in notification metadata so stale cleanup task - # can identify this document if the worker crashes. - if notification and notification.notification_metadata is not None: - from sqlalchemy.orm.attributes import flag_modified - - notification.notification_metadata["document_id"] = document.id - flag_modified(notification, "notification_metadata") - await session.commit() - - logging.info(f"Created pending document for YouTube video {video_id}") - - # ======================================================================= - # PHASE 2: Set to PROCESSING and do heavy work - # ======================================================================= - document.status = DocumentStatus.processing() - await session.commit() # UI shows "processing" status - - await task_logger.log_task_progress( - log_entry, - f"Fetching video metadata for: {video_id}", - {"stage": "metadata_fetch"}, - ) - - # Fetch video metadata - params = { - "format": "json", - "url": f"https://www.youtube.com/watch?v={video_id}", - } - oembed_url = "https://www.youtube.com/oembed" - - # Build residential proxy settings (if configured) - residential_proxies = get_requests_proxies() - - oembed_fetch_start = time.perf_counter() - oembed_page = await AsyncFetcher.get( - oembed_url, - params=params, - proxy=get_proxy_url(), - stealthy_headers=True, - ) - logging.info( - "[youtube][perf] source=oembed video=%s status=%s fetch_ms=%.1f", - video_id, - getattr(oembed_page, "status", None), - (time.perf_counter() - oembed_fetch_start) * 1000, - ) - video_data = oembed_page.json() - - # Update title immediately for better UX (user sees actual title sooner) - document.title = video_data.get("title", f"YouTube Video: {video_id}") - await session.commit() - - await task_logger.log_task_progress( - log_entry, - f"Video metadata fetched: {video_data.get('title', 'Unknown')}", - { - "stage": "metadata_fetched", - "title": video_data.get("title"), - "author": video_data.get("author_name"), - }, - ) - - # Get video transcript - await task_logger.log_task_progress( - log_entry, - f"Fetching transcript for video: {video_id}", - {"stage": "transcript_fetch"}, - ) - - try: - transcript_fetch_start = time.perf_counter() - ua = UserAgent() - http_client = Session() - http_client.headers.update({"User-Agent": ua.random}) - if residential_proxies: - http_client.proxies.update(residential_proxies) - ytt_api = YouTubeTranscriptApi(http_client=http_client) - - # List all available transcripts and pick the first one - # (the video's primary language) instead of defaulting to English - transcript_list = ytt_api.list(video_id) - transcript = next(iter(transcript_list)) - captions = transcript.fetch() - logging.info( - "[youtube][perf] source=transcript video=%s fetch_ms=%.1f", - video_id, - (time.perf_counter() - transcript_fetch_start) * 1000, - ) - - # Include complete caption information with timestamps - transcript_segments = [] - for line in captions: - start_time = line.start - duration = line.duration - text = line.text - timestamp = f"[{start_time:.2f}s-{start_time + duration:.2f}s]" - transcript_segments.append(f"{timestamp} {text}") - transcript_text = "\n".join(transcript_segments) - - await task_logger.log_task_progress( - log_entry, - f"Transcript fetched successfully: {len(captions)} segments ({transcript.language})", - { - "stage": "transcript_fetched", - "segments_count": len(captions), - "transcript_length": len(transcript_text), - "language": transcript.language, - "language_code": transcript.language_code, - "is_generated": transcript.is_generated, - }, - ) - except Exception as e: - transcript_text = f"No captions available for this video. Error: {e!s}" - await task_logger.log_task_progress( - log_entry, - f"No transcript available for video: {video_id}", - {"stage": "transcript_unavailable", "error": str(e)}, - ) - - # Format document - await task_logger.log_task_progress( - log_entry, - f"Processing video content: {video_data.get('title', 'YouTube Video')}", - {"stage": "content_processing"}, - ) - - # Format document metadata in a more maintainable way - metadata_sections = [ - ( - "METADATA", - [ - f"TITLE: {video_data.get('title', 'YouTube Video')}", - f"URL: {url}", - f"VIDEO_ID: {video_id}", - f"AUTHOR: {video_data.get('author_name', 'Unknown')}", - f"THUMBNAIL: {video_data.get('thumbnail_url', '')}", - ], - ), - ( - "CONTENT", - ["FORMAT: transcript", "TEXT_START", transcript_text, "TEXT_END"], - ), - ] - - # Build the document string more efficiently - document_parts = [] - document_parts.append("") - - for section_title, section_content in metadata_sections: - document_parts.append(f"<{section_title}>") - document_parts.extend(section_content) - document_parts.append(f"") - - document_parts.append("") - combined_document_string = "\n".join(document_parts) - - # Generate content hash - content_hash = generate_content_hash(combined_document_string, workspace_id) - - # For existing documents, check if content has changed - if not is_new_document and existing_document.content_hash == content_hash: - await task_logger.log_task_success( - log_entry, - f"YouTube video document unchanged: {video_data.get('title', 'YouTube Video')}", - { - "duplicate_detected": True, - "existing_document_id": existing_document.id, - "video_id": video_id, - }, - ) - logging.info( - f"Document for YouTube video {video_id} unchanged. Marking as ready." - ) - document.status = DocumentStatus.ready() - await session.commit() - return document - - summary_content = combined_document_string - summary_embedding = embed_text(summary_content) - - # Process chunks - await task_logger.log_task_progress( - log_entry, - f"Processing content chunks for video: {video_data.get('title', 'YouTube Video')}", - {"stage": "chunk_processing"}, - ) - - chunks = await create_document_chunks(combined_document_string) - - # ======================================================================= - # PHASE 3: Update document to READY with all content - # ======================================================================= - await task_logger.log_task_progress( - log_entry, - f"Finalizing document: {video_data.get('title', 'YouTube Video')}", - {"stage": "document_finalization", "chunks_count": len(chunks)}, - ) - - document.title = video_data.get("title", "YouTube Video") - document.content = summary_content - document.content_hash = content_hash - document.embedding = summary_embedding - document.document_metadata = { - "url": url, - "video_id": video_id, - "video_title": video_data.get("title", "YouTube Video"), - "author": video_data.get("author_name", "Unknown"), - "thumbnail": video_data.get("thumbnail_url", ""), - } - await safe_set_chunks(session, document, chunks) - document.source_markdown = combined_document_string - document.status = DocumentStatus.ready() # READY status - fully processed - document.updated_at = get_current_timestamp() - - await session.commit() - await session.refresh(document) - - # Log success - await task_logger.log_task_success( - log_entry, - f"Successfully processed YouTube video: {video_data.get('title', 'YouTube Video')}", - { - "document_id": document.id, - "video_id": video_id, - "title": document.title, - "content_hash": content_hash, - "chunks_count": len(chunks), - "summary_length": len(summary_content), - "has_transcript": "No captions available" not in transcript_text, - }, - ) - - return document - - except SQLAlchemyError as db_error: - # Mark document as failed if it exists - if document: - try: - document.status = DocumentStatus.failed( - f"Database error: {str(db_error)[:150]}" - ) - document.updated_at = get_current_timestamp() - await session.commit() - except Exception: - await session.rollback() - else: - await session.rollback() - - await task_logger.log_task_failure( - log_entry, - f"Database error while processing YouTube video: {url}", - str(db_error), - { - "error_type": "SQLAlchemyError", - "video_id": video_id, - }, - ) - raise db_error - - except Exception as e: - # Mark document as failed if it exists - if document: - try: - document.status = DocumentStatus.failed(str(e)[:200]) - document.updated_at = get_current_timestamp() - await session.commit() - except Exception: - await session.rollback() - else: - await session.rollback() - - await task_logger.log_task_failure( - log_entry, - f"Failed to process YouTube video: {url}", - str(e), - { - "error_type": type(e).__name__, - "video_id": video_id, - }, - ) - logging.error(f"Failed to process YouTube video: {e!s}") - raise diff --git a/surfsense_backend/tests/unit/observability/test_helpers.py b/surfsense_backend/tests/unit/observability/test_helpers.py index eafb8b626..d81a7ed50 100644 --- a/surfsense_backend/tests/unit/observability/test_helpers.py +++ b/surfsense_backend/tests/unit/observability/test_helpers.py @@ -26,7 +26,6 @@ def _disable_otel(monkeypatch: pytest.MonkeyPatch): ("delete_folder_documents_background", "delete"), ("delete_search_space_background", "delete"), ("process_extension_document", "process"), - ("process_youtube_video", "process"), ("process_file_upload", "process"), ("process_file_upload_with_document", "process"), ("process_circleback_meeting", "process"), From f31823f765ad97a582dc1f9edee07528ff3169dc Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Fri, 3 Jul 2026 17:56:43 +0200 Subject: [PATCH 59/64] feat(google_maps): add scrape + reviews capability verbs --- .../app/capabilities/google_maps/__init__.py | 6 + .../google_maps/reviews/__init__.py | 3 + .../google_maps/reviews/definition.py | 23 ++++ .../google_maps/reviews/executor.py | 40 ++++++ .../google_maps/reviews/schemas.py | 68 +++++++++++ .../google_maps/scrape/__init__.py | 3 + .../google_maps/scrape/definition.py | 25 ++++ .../google_maps/scrape/executor.py | 43 +++++++ .../google_maps/scrape/schemas.py | 91 ++++++++++++++ surfsense_backend/app/routes/__init__.py | 1 + .../unit/capabilities/google_maps/__init__.py | 0 .../google_maps/reviews/__init__.py | 0 .../google_maps/reviews/test_executor.py | 87 +++++++++++++ .../google_maps/reviews/test_schemas.py | 35 ++++++ .../google_maps/scrape/__init__.py | 0 .../google_maps/scrape/test_executor.py | 115 ++++++++++++++++++ .../google_maps/scrape/test_schemas.py | 36 ++++++ .../capabilities/google_maps/test_registry.py | 32 +++++ 18 files changed, 608 insertions(+) create mode 100644 surfsense_backend/app/capabilities/google_maps/__init__.py create mode 100644 surfsense_backend/app/capabilities/google_maps/reviews/__init__.py create mode 100644 surfsense_backend/app/capabilities/google_maps/reviews/definition.py create mode 100644 surfsense_backend/app/capabilities/google_maps/reviews/executor.py create mode 100644 surfsense_backend/app/capabilities/google_maps/reviews/schemas.py create mode 100644 surfsense_backend/app/capabilities/google_maps/scrape/__init__.py create mode 100644 surfsense_backend/app/capabilities/google_maps/scrape/definition.py create mode 100644 surfsense_backend/app/capabilities/google_maps/scrape/executor.py create mode 100644 surfsense_backend/app/capabilities/google_maps/scrape/schemas.py create mode 100644 surfsense_backend/tests/unit/capabilities/google_maps/__init__.py create mode 100644 surfsense_backend/tests/unit/capabilities/google_maps/reviews/__init__.py create mode 100644 surfsense_backend/tests/unit/capabilities/google_maps/reviews/test_executor.py create mode 100644 surfsense_backend/tests/unit/capabilities/google_maps/reviews/test_schemas.py create mode 100644 surfsense_backend/tests/unit/capabilities/google_maps/scrape/__init__.py create mode 100644 surfsense_backend/tests/unit/capabilities/google_maps/scrape/test_executor.py create mode 100644 surfsense_backend/tests/unit/capabilities/google_maps/scrape/test_schemas.py create mode 100644 surfsense_backend/tests/unit/capabilities/google_maps/test_registry.py diff --git a/surfsense_backend/app/capabilities/google_maps/__init__.py b/surfsense_backend/app/capabilities/google_maps/__init__.py new file mode 100644 index 000000000..c3e2ef30e --- /dev/null +++ b/surfsense_backend/app/capabilities/google_maps/__init__.py @@ -0,0 +1,6 @@ +"""``google_maps.*`` namespace: platform-native Google Maps data verbs.""" + +from __future__ import annotations + +from app.capabilities.google_maps.reviews import definition as _reviews # noqa: F401 +from app.capabilities.google_maps.scrape import definition as _scrape # noqa: F401 diff --git a/surfsense_backend/app/capabilities/google_maps/reviews/__init__.py b/surfsense_backend/app/capabilities/google_maps/reviews/__init__.py new file mode 100644 index 000000000..58283f1d7 --- /dev/null +++ b/surfsense_backend/app/capabilities/google_maps/reviews/__init__.py @@ -0,0 +1,3 @@ +"""``google_maps.reviews`` verb: Maps place URLs / place IDs → review items.""" + +from __future__ import annotations diff --git a/surfsense_backend/app/capabilities/google_maps/reviews/definition.py b/surfsense_backend/app/capabilities/google_maps/reviews/definition.py new file mode 100644 index 000000000..384e01577 --- /dev/null +++ b/surfsense_backend/app/capabilities/google_maps/reviews/definition.py @@ -0,0 +1,23 @@ +"""``google_maps.reviews`` capability registration (free — see 04-capabilities open item).""" + +from __future__ import annotations + +from app.capabilities.core import Capability, register_capability +from app.capabilities.google_maps.reviews.executor import build_reviews_executor +from app.capabilities.google_maps.reviews.schemas import ReviewsInput, ReviewsOutput + +GOOGLE_MAPS_REVIEWS = Capability( + name="google_maps.reviews", + description=( + "Fetch public reviews for one or more Google Maps places. Give it place " + "URLs or place IDs; returns structured review items with author, text, " + "star rating, like count, owner response, and timestamps. Use it to " + "gauge sentiment or pull recent feedback on specific places." + ), + input_schema=ReviewsInput, + output_schema=ReviewsOutput, + executor=build_reviews_executor(), + billing_unit=None, +) + +register_capability(GOOGLE_MAPS_REVIEWS) diff --git a/surfsense_backend/app/capabilities/google_maps/reviews/executor.py b/surfsense_backend/app/capabilities/google_maps/reviews/executor.py new file mode 100644 index 000000000..39759e2e7 --- /dev/null +++ b/surfsense_backend/app/capabilities/google_maps/reviews/executor.py @@ -0,0 +1,40 @@ +"""``google_maps.reviews`` executor: verb input → scraper → review items.""" + +from __future__ import annotations + +from collections.abc import Awaitable, Callable + +from app.capabilities.core import Executor +from app.capabilities.google_maps.reviews.schemas import ReviewsInput, ReviewsOutput +from app.exceptions import ForbiddenError +from app.proprietary.platforms.google_maps import ( + GoogleMapsReviewsInput, + scrape_reviews, +) +from app.proprietary.platforms.google_maps.scraper import SignInRequiredError + +ReviewsFn = Callable[[GoogleMapsReviewsInput], Awaitable[list[dict]]] + + +def build_reviews_executor(scrape_fn: ReviewsFn | None = None) -> Executor: + """Bind the executor to a reviews scraper fn (defaults to the proprietary actor).""" + scrape_fn = scrape_fn or scrape_reviews + + async def execute(payload: ReviewsInput) -> ReviewsOutput: + actor_input = GoogleMapsReviewsInput( + startUrls=[{"url": url} for url in payload.urls], + placeIds=payload.place_ids, + maxReviews=payload.max_reviews, + reviewsSort=payload.sort_by, + reviewsStartDate=payload.start_date, + language=payload.language, + ) + try: + items = await scrape_fn(actor_input) + except SignInRequiredError as exc: + raise ForbiddenError( + f"Google sign in required: {exc}", code="GOOGLE_SIGNIN_REQUIRED" + ) from exc + return ReviewsOutput(items=items) + + return execute diff --git a/surfsense_backend/app/capabilities/google_maps/reviews/schemas.py b/surfsense_backend/app/capabilities/google_maps/reviews/schemas.py new file mode 100644 index 000000000..757ca2ffd --- /dev/null +++ b/surfsense_backend/app/capabilities/google_maps/reviews/schemas.py @@ -0,0 +1,68 @@ +"""``google_maps.reviews`` I/O contracts. + +A lean surface over ``GoogleMapsReviewsInput``; the scraper's ``ReviewItem`` is +reused verbatim as the output element. +""" + +from __future__ import annotations + +from typing import Literal + +from pydantic import BaseModel, Field, model_validator + +from app.proprietary.platforms.google_maps import ReviewItem + +MAX_MAPS_REVIEW_SOURCES = 20 +"""Per-call cap on urls + place_ids: bounds how many places one request harvests.""" + + +class ReviewsInput(BaseModel): + urls: list[str] = Field( + default_factory=list, + max_length=MAX_MAPS_REVIEW_SOURCES, + description=( + "Google Maps place URLs to fetch reviews for. Provide these OR " + "place_ids (at least one is required)." + ), + ) + place_ids: list[str] = Field( + default_factory=list, + max_length=MAX_MAPS_REVIEW_SOURCES, + description=( + "Known Google place IDs (ChIJ...) to fetch reviews for. Provide " + "these OR urls." + ), + ) + max_reviews: int = Field( + default=20, + ge=1, + le=100_000, + description="Max reviews to return per place.", + ) + sort_by: Literal["newest", "mostRelevant", "highestRanking", "lowestRanking"] = ( + Field( + default="newest", + description="Review ordering.", + ) + ) + language: str = Field( + default="en", + description="Review language code, e.g. 'en', 'fr'.", + ) + start_date: str | None = Field( + default=None, + description="Only reviews on/after this ISO date, e.g. '2024-01-01'.", + ) + + @model_validator(mode="after") + def _require_a_source(self) -> ReviewsInput: + if not (self.urls or self.place_ids): + raise ValueError("Provide at least one of 'urls' or 'place_ids'.") + return self + + +class ReviewsOutput(BaseModel): + items: list[ReviewItem] = Field( + default_factory=list, + description="One item per review, in the scraper's emission order.", + ) diff --git a/surfsense_backend/app/capabilities/google_maps/scrape/__init__.py b/surfsense_backend/app/capabilities/google_maps/scrape/__init__.py new file mode 100644 index 000000000..495d39760 --- /dev/null +++ b/surfsense_backend/app/capabilities/google_maps/scrape/__init__.py @@ -0,0 +1,3 @@ +"""``google_maps.scrape`` verb: search queries / Maps URLs / place IDs → place items.""" + +from __future__ import annotations diff --git a/surfsense_backend/app/capabilities/google_maps/scrape/definition.py b/surfsense_backend/app/capabilities/google_maps/scrape/definition.py new file mode 100644 index 000000000..9b6b0d2f1 --- /dev/null +++ b/surfsense_backend/app/capabilities/google_maps/scrape/definition.py @@ -0,0 +1,25 @@ +"""``google_maps.scrape`` capability registration (free — see 04-capabilities open item).""" + +from __future__ import annotations + +from app.capabilities.core import Capability, register_capability +from app.capabilities.google_maps.scrape.executor import build_scrape_executor +from app.capabilities.google_maps.scrape.schemas import ScrapeInput, ScrapeOutput + +GOOGLE_MAPS_SCRAPE = Capability( + name="google_maps.scrape", + description=( + "Scrape public Google Maps places. Give it search queries (optionally " + "scoped by location), Google Maps URLs, or place IDs, and it returns " + "structured place items — name, address, category, phone, website, " + "rating, review count, coordinates, and opening hours. Set " + "include_details for richer detail-page fields, or max_reviews/" + "max_images to attach reviews and photos per place." + ), + input_schema=ScrapeInput, + output_schema=ScrapeOutput, + executor=build_scrape_executor(), + billing_unit=None, +) + +register_capability(GOOGLE_MAPS_SCRAPE) diff --git a/surfsense_backend/app/capabilities/google_maps/scrape/executor.py b/surfsense_backend/app/capabilities/google_maps/scrape/executor.py new file mode 100644 index 000000000..71706dc22 --- /dev/null +++ b/surfsense_backend/app/capabilities/google_maps/scrape/executor.py @@ -0,0 +1,43 @@ +"""``google_maps.scrape`` executor: verb input → scraper → place items.""" + +from __future__ import annotations + +from collections.abc import Awaitable, Callable + +from app.capabilities.core import Executor +from app.capabilities.google_maps.scrape.schemas import ScrapeInput, ScrapeOutput +from app.exceptions import ForbiddenError +from app.proprietary.platforms.google_maps import ( + GoogleMapsScrapeInput, + scrape_places, +) +from app.proprietary.platforms.google_maps.scraper import SignInRequiredError + +ScrapeFn = Callable[[GoogleMapsScrapeInput], Awaitable[list[dict]]] + + +def build_scrape_executor(scrape_fn: ScrapeFn | None = None) -> Executor: + """Bind the executor to a scraper fn (defaults to the proprietary actor).""" + scrape_fn = scrape_fn or scrape_places + + async def execute(payload: ScrapeInput) -> ScrapeOutput: + actor_input = GoogleMapsScrapeInput( + searchStringsArray=payload.search_queries, + startUrls=[{"url": url} for url in payload.urls], + placeIds=payload.place_ids, + locationQuery=payload.location, + maxCrawledPlacesPerSearch=payload.max_places, + language=payload.language, + scrapePlaceDetailPage=payload.include_details, + maxReviews=payload.max_reviews, + maxImages=payload.max_images, + ) + try: + items = await scrape_fn(actor_input) + except SignInRequiredError as exc: + raise ForbiddenError( + f"Google sign in required: {exc}", code="GOOGLE_SIGNIN_REQUIRED" + ) from exc + return ScrapeOutput(items=items) + + return execute diff --git a/surfsense_backend/app/capabilities/google_maps/scrape/schemas.py b/surfsense_backend/app/capabilities/google_maps/scrape/schemas.py new file mode 100644 index 000000000..49082a120 --- /dev/null +++ b/surfsense_backend/app/capabilities/google_maps/scrape/schemas.py @@ -0,0 +1,91 @@ +"""``google_maps.scrape`` I/O contracts. + +A lean, agent-friendly surface over ``GoogleMapsScrapeInput`` +(``app/proprietary/platforms/google_maps``). The executor maps this to the full +scraper input; the scraper's ``PlaceItem`` is reused verbatim as the output +element. +""" + +from __future__ import annotations + +from pydantic import BaseModel, Field, model_validator + +from app.proprietary.platforms.google_maps import PlaceItem + +MAX_MAPS_SOURCES = 20 +"""Per-call cap on queries + urls + place_ids: bounds a sync request's fan-out.""" + + +class ScrapeInput(BaseModel): + search_queries: list[str] = Field( + default_factory=list, + max_length=MAX_MAPS_SOURCES, + description=( + "Google Maps search terms (e.g. 'coffee shops', 'dentist'); each " + "returns up to max_places. Provide these OR urls OR place_ids " + "(at least one is required). Pair with location to scope a search." + ), + ) + urls: list[str] = Field( + default_factory=list, + max_length=MAX_MAPS_SOURCES, + description=( + "Google Maps URLs — a place page (/maps/place/...) or a search " + "results URL. Provide these OR search_queries OR place_ids." + ), + ) + place_ids: list[str] = Field( + default_factory=list, + max_length=MAX_MAPS_SOURCES, + description=( + "Known Google place IDs (ChIJ...) to fetch directly. Provide these " + "OR search_queries OR urls." + ), + ) + location: str | None = Field( + default=None, + description="Location to scope search_queries to, e.g. 'New York, USA'.", + ) + max_places: int = Field( + default=10, + ge=1, + le=1000, + description="Max places to return per search query.", + ) + language: str = Field( + default="en", + description="Result language code, e.g. 'en', 'fr'.", + ) + include_details: bool = Field( + default=False, + description=( + "Also fetch each place's detail page — opening hours, popular " + "times, extra contact info (slower; more requests)." + ), + ) + max_reviews: int = Field( + default=0, + ge=0, + le=100_000, + description="Reviews to attach per place (0 = none).", + ) + max_images: int = Field( + default=0, + ge=0, + description="Images to attach per place (0 = none).", + ) + + @model_validator(mode="after") + def _require_a_source(self) -> ScrapeInput: + if not (self.search_queries or self.urls or self.place_ids): + raise ValueError( + "Provide at least one of 'search_queries', 'urls', or 'place_ids'." + ) + return self + + +class ScrapeOutput(BaseModel): + items: list[PlaceItem] = Field( + default_factory=list, + description="One place item per result, in the scraper's emission order.", + ) diff --git a/surfsense_backend/app/routes/__init__.py b/surfsense_backend/app/routes/__init__.py index b2f6a8fab..1d086e970 100644 --- a/surfsense_backend/app/routes/__init__.py +++ b/surfsense_backend/app/routes/__init__.py @@ -1,6 +1,7 @@ from fastapi import APIRouter, Depends # Import verb namespaces for their registration side effects before the door builds. +import app.capabilities.google_maps import app.capabilities.web import app.capabilities.youtube # noqa: F401 from app.automations.api import router as automations_router diff --git a/surfsense_backend/tests/unit/capabilities/google_maps/__init__.py b/surfsense_backend/tests/unit/capabilities/google_maps/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/surfsense_backend/tests/unit/capabilities/google_maps/reviews/__init__.py b/surfsense_backend/tests/unit/capabilities/google_maps/reviews/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/surfsense_backend/tests/unit/capabilities/google_maps/reviews/test_executor.py b/surfsense_backend/tests/unit/capabilities/google_maps/reviews/test_executor.py new file mode 100644 index 000000000..912f191df --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/google_maps/reviews/test_executor.py @@ -0,0 +1,87 @@ +"""``google_maps.reviews`` executor: verb input → actor input mapping → typed items. + +Boundary mocked: the proprietary scraper (injected fake). NOT mocked: the verb's +own payload→GoogleMapsReviewsInput mapping and the dict→ReviewItem wrapping. +""" + +from __future__ import annotations + +import pytest + +from app.capabilities.google_maps.reviews.executor import build_reviews_executor +from app.capabilities.google_maps.reviews.schemas import ReviewsInput, ReviewsOutput +from app.exceptions import ForbiddenError +from app.proprietary.platforms.google_maps import GoogleMapsReviewsInput +from app.proprietary.platforms.google_maps.scraper import SignInRequiredError + +pytestmark = pytest.mark.unit + + +class _FakeScraper: + def __init__(self, items: list[dict]): + self._items = items + self.calls: list[GoogleMapsReviewsInput] = [] + + async def __call__(self, actor_input: GoogleMapsReviewsInput) -> list[dict]: + self.calls.append(actor_input) + return self._items + + +async def test_maps_urls_to_start_urls_and_wraps_items(): + scraper = _FakeScraper([{"text": "Great place", "stars": 5.0}]) + execute = build_reviews_executor(scrape_fn=scraper) + + out = await execute(ReviewsInput(urls=["https://www.google.com/maps/place/x"])) + + assert isinstance(out, ReviewsOutput) + assert len(out.items) == 1 + assert out.items[0].text == "Great place" + assert out.items[0].stars == 5.0 + + (actor_input,) = scraper.calls + assert [u.url for u in actor_input.startUrls] == [ + "https://www.google.com/maps/place/x" + ] + + +async def test_forwards_place_ids_and_options(): + scraper = _FakeScraper([]) + execute = build_reviews_executor(scrape_fn=scraper) + + await execute( + ReviewsInput( + place_ids=["ChIJx"], + max_reviews=50, + sort_by="highestRanking", + language="fr", + start_date="2024-01-01", + ) + ) + + (actor_input,) = scraper.calls + assert actor_input.placeIds == ["ChIJx"] + assert actor_input.maxReviews == 50 + assert actor_input.reviewsSort == "highestRanking" + assert actor_input.language == "fr" + assert actor_input.reviewsStartDate == "2024-01-01" + + +async def test_sign_in_required_maps_to_forbidden_403(): + async def _raise(_actor_input): + raise SignInRequiredError("wall hit") + + execute = build_reviews_executor(scrape_fn=_raise) + + with pytest.raises(ForbiddenError) as exc_info: + await execute(ReviewsInput(place_ids=["ChIJx"])) + assert exc_info.value.status_code == 403 + + +async def test_other_faults_propagate_for_the_door_to_map(): + async def _boom(_actor_input): + raise RuntimeError("proxy exploded") + + execute = build_reviews_executor(scrape_fn=_boom) + + with pytest.raises(RuntimeError): + await execute(ReviewsInput(place_ids=["ChIJx"])) diff --git a/surfsense_backend/tests/unit/capabilities/google_maps/reviews/test_schemas.py b/surfsense_backend/tests/unit/capabilities/google_maps/reviews/test_schemas.py new file mode 100644 index 000000000..9381161d0 --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/google_maps/reviews/test_schemas.py @@ -0,0 +1,35 @@ +"""``google_maps.reviews`` input guards: a source is required and the batch is bounded.""" + +from __future__ import annotations + +import pytest +from pydantic import ValidationError + +from app.capabilities.google_maps.reviews.schemas import ( + MAX_MAPS_REVIEW_SOURCES, + ReviewsInput, +) + +pytestmark = pytest.mark.unit + + +def test_rejects_input_with_no_source(): + with pytest.raises(ValidationError): + ReviewsInput() + + +def test_accepts_urls_or_place_ids(): + assert ReviewsInput(urls=["https://maps.google.com/x"]).place_ids == [] + assert ReviewsInput(place_ids=["ChIJx"]).urls == [] + + +def test_max_reviews_defaults_and_is_bounded(): + assert ReviewsInput(place_ids=["ChIJx"]).max_reviews == 20 + with pytest.raises(ValidationError): + ReviewsInput(place_ids=["ChIJx"], max_reviews=0) + + +def test_rejects_more_sources_than_the_cap(): + too_many = [f"ChIJ{i}" for i in range(MAX_MAPS_REVIEW_SOURCES + 1)] + with pytest.raises(ValidationError): + ReviewsInput(place_ids=too_many) diff --git a/surfsense_backend/tests/unit/capabilities/google_maps/scrape/__init__.py b/surfsense_backend/tests/unit/capabilities/google_maps/scrape/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/surfsense_backend/tests/unit/capabilities/google_maps/scrape/test_executor.py b/surfsense_backend/tests/unit/capabilities/google_maps/scrape/test_executor.py new file mode 100644 index 000000000..aaa3af297 --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/google_maps/scrape/test_executor.py @@ -0,0 +1,115 @@ +"""``google_maps.scrape`` executor: verb input → actor input mapping → typed items. + +Boundary mocked: the proprietary scraper (injected fake). NOT mocked: the verb's +own payload→GoogleMapsScrapeInput mapping and the dict→PlaceItem wrapping. +""" + +from __future__ import annotations + +import pytest + +from app.capabilities.google_maps.scrape.executor import build_scrape_executor +from app.capabilities.google_maps.scrape.schemas import ScrapeInput, ScrapeOutput +from app.exceptions import ForbiddenError +from app.proprietary.platforms.google_maps import GoogleMapsScrapeInput +from app.proprietary.platforms.google_maps.scraper import SignInRequiredError + +pytestmark = pytest.mark.unit + + +class _FakeScraper: + """Records the actor input it was called with and returns canned items.""" + + def __init__(self, items: list[dict]): + self._items = items + self.calls: list[GoogleMapsScrapeInput] = [] + + async def __call__(self, actor_input: GoogleMapsScrapeInput) -> list[dict]: + self.calls.append(actor_input) + return self._items + + +async def test_maps_queries_and_wraps_items(): + scraper = _FakeScraper([{"title": "Blue Bottle", "placeId": "abc"}]) + execute = build_scrape_executor(scrape_fn=scraper) + + out = await execute(ScrapeInput(search_queries=["coffee"], location="Austin")) + + assert isinstance(out, ScrapeOutput) + assert len(out.items) == 1 + assert out.items[0].title == "Blue Bottle" + assert out.items[0].placeId == "abc" + + (actor_input,) = scraper.calls + assert actor_input.searchStringsArray == ["coffee"] + assert actor_input.locationQuery == "Austin" + + +async def test_maps_urls_and_place_ids(): + scraper = _FakeScraper([]) + execute = build_scrape_executor(scrape_fn=scraper) + + await execute( + ScrapeInput( + urls=["https://www.google.com/maps/place/x"], + place_ids=["ChIJxxxx"], + ) + ) + + (actor_input,) = scraper.calls + assert [u.url for u in actor_input.startUrls] == [ + "https://www.google.com/maps/place/x" + ] + assert actor_input.placeIds == ["ChIJxxxx"] + + +async def test_max_places_maps_to_per_search_cap(): + scraper = _FakeScraper([]) + execute = build_scrape_executor(scrape_fn=scraper) + + await execute(ScrapeInput(search_queries=["x"], max_places=25)) + + (actor_input,) = scraper.calls + assert actor_input.maxCrawledPlacesPerSearch == 25 + + +async def test_forwards_detail_review_and_image_options(): + scraper = _FakeScraper([]) + execute = build_scrape_executor(scrape_fn=scraper) + + await execute( + ScrapeInput( + search_queries=["x"], + include_details=True, + max_reviews=5, + max_images=3, + language="fr", + ) + ) + + (actor_input,) = scraper.calls + assert actor_input.scrapePlaceDetailPage is True + assert actor_input.maxReviews == 5 + assert actor_input.maxImages == 3 + assert actor_input.language == "fr" + + +async def test_sign_in_required_maps_to_forbidden_403(): + async def _raise(_actor_input): + raise SignInRequiredError("wall hit") + + execute = build_scrape_executor(scrape_fn=_raise) + + with pytest.raises(ForbiddenError) as exc_info: + await execute(ScrapeInput(search_queries=["x"])) + assert exc_info.value.status_code == 403 + + +async def test_other_faults_propagate_for_the_door_to_map(): + async def _boom(_actor_input): + raise RuntimeError("proxy exploded") + + execute = build_scrape_executor(scrape_fn=_boom) + + with pytest.raises(RuntimeError): + await execute(ScrapeInput(search_queries=["x"])) diff --git a/surfsense_backend/tests/unit/capabilities/google_maps/scrape/test_schemas.py b/surfsense_backend/tests/unit/capabilities/google_maps/scrape/test_schemas.py new file mode 100644 index 000000000..fe164a98a --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/google_maps/scrape/test_schemas.py @@ -0,0 +1,36 @@ +"""``google_maps.scrape`` input guards: a source is required and the batch is bounded.""" + +from __future__ import annotations + +import pytest +from pydantic import ValidationError + +from app.capabilities.google_maps.scrape.schemas import ( + MAX_MAPS_SOURCES, + ScrapeInput, +) + +pytestmark = pytest.mark.unit + + +def test_rejects_input_with_no_source(): + with pytest.raises(ValidationError): + ScrapeInput() + + +def test_accepts_any_single_source(): + assert ScrapeInput(search_queries=["coffee"]).urls == [] + assert ScrapeInput(urls=["https://maps.google.com/x"]).place_ids == [] + assert ScrapeInput(place_ids=["ChIJx"]).search_queries == [] + + +def test_max_places_defaults_and_is_bounded(): + assert ScrapeInput(search_queries=["x"]).max_places == 10 + with pytest.raises(ValidationError): + ScrapeInput(search_queries=["x"], max_places=0) + + +def test_rejects_more_sources_than_the_cap(): + too_many = [f"q{i}" for i in range(MAX_MAPS_SOURCES + 1)] + with pytest.raises(ValidationError): + ScrapeInput(search_queries=too_many) diff --git a/surfsense_backend/tests/unit/capabilities/google_maps/test_registry.py b/surfsense_backend/tests/unit/capabilities/google_maps/test_registry.py new file mode 100644 index 000000000..c6a007645 --- /dev/null +++ b/surfsense_backend/tests/unit/capabilities/google_maps/test_registry.py @@ -0,0 +1,32 @@ +"""The google_maps namespace registers each verb as one Capability the doors/agent read.""" + +from __future__ import annotations + +import pytest + +from app.capabilities import ( + google_maps, # noqa: F401 — importing the namespace registers its verbs +) +from app.capabilities.core.store import get_capability +from app.capabilities.google_maps.reviews.schemas import ReviewsInput, ReviewsOutput +from app.capabilities.google_maps.scrape.schemas import ScrapeInput, ScrapeOutput + +pytestmark = pytest.mark.unit + + +def test_google_maps_scrape_is_registered_and_free(): + cap = get_capability("google_maps.scrape") + + assert cap.name == "google_maps.scrape" + assert cap.input_schema is ScrapeInput + assert cap.output_schema is ScrapeOutput + assert cap.billing_unit is None + + +def test_google_maps_reviews_is_registered_and_free(): + cap = get_capability("google_maps.reviews") + + assert cap.name == "google_maps.reviews" + assert cap.input_schema is ReviewsInput + assert cap.output_schema is ReviewsOutput + assert cap.billing_unit is None From 2c76ef1f57d34eb7cdd51478d865e281d4be9293 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Fri, 3 Jul 2026 17:56:53 +0200 Subject: [PATCH 60/64] feat(subagents): add google_maps builtin subagent --- .../agents/chat/multi_agent_chat/constants.py | 1 + .../builtins/google_maps/__init__.py | 1 + .../subagents/builtins/google_maps/agent.py | 43 +++++++++++++ .../builtins/google_maps/description.md | 2 + .../builtins/google_maps/system_prompt.md | 62 +++++++++++++++++++ .../builtins/google_maps/tools/__init__.py | 0 .../builtins/google_maps/tools/index.py | 28 +++++++++ .../multi_agent_chat/subagents/registry.py | 4 ++ .../test_subagent_composition.py | 3 +- 9 files changed, 143 insertions(+), 1 deletion(-) create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/__init__.py create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/agent.py create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/description.md create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/system_prompt.md create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/tools/__init__.py create mode 100644 surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/tools/index.py diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/constants.py b/surfsense_backend/app/agents/chat/multi_agent_chat/constants.py index e6c5b4975..96413e553 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/constants.py +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/constants.py @@ -28,6 +28,7 @@ SUBAGENT_TO_REQUIRED_CONNECTOR_MAP: dict[str, frozenset[str]] = { "knowledge_base": frozenset(), "web_crawler": frozenset(), "youtube": frozenset(), + "google_maps": frozenset(), "airtable": frozenset({"AIRTABLE_CONNECTOR"}), "calendar": frozenset({"GOOGLE_CALENDAR_CONNECTOR"}), "clickup": frozenset({"CLICKUP_CONNECTOR"}), diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/__init__.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/__init__.py new file mode 100644 index 000000000..aef5d5e5f --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/__init__.py @@ -0,0 +1 @@ +"""``google_maps`` builtin subagent: structured Google Maps place/review data.""" diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/agent.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/agent.py new file mode 100644 index 000000000..df5d9e8d4 --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/agent.py @@ -0,0 +1,43 @@ +"""``google_maps`` route: ``SurfSenseSubagentSpec`` builder for deepagents.""" + +from __future__ import annotations + +from typing import Any + +from langchain_core.language_models import BaseChatModel +from langchain_core.tools import BaseTool + +from app.agents.chat.multi_agent_chat.subagents.shared.md_file_reader import ( + read_md_file, +) +from app.agents.chat.multi_agent_chat.subagents.shared.spec import SurfSenseSubagentSpec +from app.agents.chat.multi_agent_chat.subagents.shared.subagent_builder import ( + pack_subagent, +) + +from .tools.index import NAME, RULESET, load_tools + + +def build_subagent( + *, + dependencies: dict[str, Any], + model: BaseChatModel | None = None, + middleware_stack: dict[str, Any] | None = None, + mcp_tools: list[BaseTool] | None = None, +) -> SurfSenseSubagentSpec: + tools = [*load_tools(dependencies=dependencies), *(mcp_tools or [])] + description = ( + read_md_file(__package__, "description").strip() + or "Pulls structured data from Google Maps — places and their reviews." + ) + system_prompt = read_md_file(__package__, "system_prompt").strip() + return pack_subagent( + name=NAME, + description=description, + system_prompt=system_prompt, + tools=tools, + ruleset=RULESET, + dependencies=dependencies, + model=model, + middleware_stack=middleware_stack, + ) diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/description.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/description.md new file mode 100644 index 000000000..c76722a5c --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/description.md @@ -0,0 +1,2 @@ +Google Maps specialist: pulls structured data from Google Maps — finds places by search query (optionally scoped to a location), or resolves a Maps URL / place ID into place details (name, address, category, phone, website, rating, review count, coordinates, opening hours), and fetches a place's reviews (author, text, star rating, owner response, dates). Also compares fresh Maps data against earlier findings in this chat. +Use whenever the task involves Google Maps places, local businesses, or a google.com/maps link. Triggers include "find near/in X", "get details for this place", "phone/address/hours/rating of X", "how many reviews", "get the reviews for this place", "what are people saying about this business", and recurring "what changed" checks on Maps data. Not for general web pages (use the web crawling specialist) or YouTube (use the YouTube specialist). diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/system_prompt.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/system_prompt.md new file mode 100644 index 000000000..a048ab982 --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/system_prompt.md @@ -0,0 +1,62 @@ +You are the SurfSense Google Maps sub-agent. +You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis. + + +Answer the delegated question from live Google Maps data gathered with your verbs, including "what changed" comparisons against evidence already in this conversation. + + + +- `google_maps_scrape` +- `google_maps_reviews` + + + +- Finding places on a topic: call `google_maps_scrape` with `search_queries`, adding `location` to scope them (e.g. "coffee shops" in "Austin, USA"). +- Known place links or IDs: call `google_maps_scrape` with the links in `urls` or the IDs in `place_ids`. +- Need richer detail (opening hours, popular times, extra contact info): set `include_details=true`. +- Reviews / sentiment on specific places: call `google_maps_reviews` with the place `urls` or `place_ids`. +- Batch multiple queries, URLs, or place IDs into one call rather than many single-item calls. +- "What changed" / monitoring: pull the current values, compare against the prior values in this conversation's earlier tool results, and report concrete deltas (added, removed, old -> new). + + + +- Use only tools in ``. +- An item whose `status` is not `success` returned no data — report it unavailable, never invent it. +- Report only deltas you can point to in the evidence. Never fabricate facts, counts, quotes, ratings, or URLs. + + + +- Do not generate deliverables or perform connector mutations; return findings for the supervisor to act on. +- Non-Maps web pages belong to the web crawling specialist; YouTube belongs to the YouTube specialist. + + + +- Report uncertainty explicitly when evidence is incomplete or conflicting. +- Never present unverified claims as facts. + + + +- Underspecified request — no usable search query, URL, or place ID — return `status=blocked` with the missing fields. +- Tool failure: return `status=error` with a concise recovery `next_step`. +- No useful evidence: return `status=blocked` with a narrower query or the URLs/IDs you still need. + + + +Return **only** one JSON object (no markdown/prose): +{ + "status": "success" | "partial" | "blocked" | "error", + "action_summary": string, + "evidence": { + "findings": string[], + "sources": string[], + "confidence": "high" | "medium" | "low" + }, + "next_step": string | null, + "missing_fields": string[] | null, + "assumptions": string[] | null +} + +Route-specific rules: +- `evidence.findings`: max 10 entries, each a single sentence stating one distinct fact or delta. Do not paste raw payloads. +- `evidence.sources`: max 10 URLs, one per finding when applicable. List each URL once. + diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/tools/__init__.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/tools/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/tools/index.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/tools/index.py new file mode 100644 index 000000000..bf2f7eb20 --- /dev/null +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/tools/index.py @@ -0,0 +1,28 @@ +"""``google_maps`` sub-agent tools: the Google Maps scrape + reviews capability verbs.""" + +from __future__ import annotations + +from typing import Any + +from langchain_core.tools import BaseTool + +from app.agents.chat.multi_agent_chat.shared.permissions import Ruleset +from app.capabilities.core.access.agent import build_capability_tools +from app.capabilities.google_maps.reviews.definition import GOOGLE_MAPS_REVIEWS +from app.capabilities.google_maps.scrape.definition import GOOGLE_MAPS_SCRAPE + +NAME = "google_maps" + +RULESET = Ruleset(origin=NAME, rules=[]) + +_CI_VERBS = [GOOGLE_MAPS_SCRAPE, GOOGLE_MAPS_REVIEWS] + + +def load_tools( + *, dependencies: dict[str, Any] | None = None, **kwargs: Any +) -> list[BaseTool]: + d = {**(dependencies or {}), **kwargs} + return build_capability_tools( + workspace_id=d.get("workspace_id"), + capabilities=_CI_VERBS, + ) diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py index 8fbd866dc..7ae689eb7 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py @@ -15,6 +15,9 @@ from app.agents.chat.multi_agent_chat.constants import ( from app.agents.chat.multi_agent_chat.subagents.builtins.deliverables.agent import ( build_subagent as build_deliverables_subagent, ) +from app.agents.chat.multi_agent_chat.subagents.builtins.google_maps.agent import ( + build_subagent as build_google_maps_subagent, +) from app.agents.chat.multi_agent_chat.subagents.builtins.knowledge_base.agent import ( build_subagent as build_knowledge_base_subagent, ) @@ -102,6 +105,7 @@ SUBAGENT_BUILDERS_BY_NAME: dict[str, SubagentBuilder] = { "dropbox": build_dropbox_subagent, "gmail": build_gmail_subagent, "google_drive": build_google_drive_subagent, + "google_maps": build_google_maps_subagent, "jira": build_jira_subagent, "knowledge_base": build_knowledge_base_subagent, "linear": build_linear_subagent, diff --git a/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py b/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py index 873fe26c5..63978e7d2 100644 --- a/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py +++ b/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py @@ -19,7 +19,7 @@ from app.agents.chat.multi_agent_chat.subagents.registry import ( pytestmark = pytest.mark.unit -# The full specialist roster the main agent composes from: 5 builtins + 15 +# The full specialist roster the main agent composes from: 6 builtins + 15 # connector routes. Adding/removing a specialist is a deliberate product change # and must be reflected here. _EXPECTED_SUBAGENTS = frozenset( @@ -33,6 +33,7 @@ _EXPECTED_SUBAGENTS = frozenset( "dropbox", "gmail", "google_drive", + "google_maps", "jira", "knowledge_base", "linear", From 0844dd581facfdf0870a4cf582323c12dfb05ad9 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Fri, 3 Jul 2026 17:56:59 +0200 Subject: [PATCH 61/64] refactor(routes): drop duplicate /google-maps/* routes (use capability doors) --- surfsense_backend/app/routes/__init__.py | 2 - .../app/routes/google_maps_routes.py | 64 ------------------- 2 files changed, 66 deletions(-) delete mode 100644 surfsense_backend/app/routes/google_maps_routes.py diff --git a/surfsense_backend/app/routes/__init__.py b/surfsense_backend/app/routes/__init__.py index 1d086e970..f4f22e6f7 100644 --- a/surfsense_backend/app/routes/__init__.py +++ b/surfsense_backend/app/routes/__init__.py @@ -44,7 +44,6 @@ from .google_drive_add_connector_route import ( from .google_gmail_add_connector_route import ( router as google_gmail_add_connector_router, ) -from .google_maps_routes import router as google_maps_router from .image_generation_routes import router as image_generation_router from .incentive_tasks_routes import router as incentive_tasks_router from .jira_add_connector_route import router as jira_add_connector_router @@ -139,7 +138,6 @@ router.include_router(public_chat_router) # Public chat sharing and cloning router.include_router(incentive_tasks_router) # Incentive tasks for earning free pages router.include_router(stripe_router) # Stripe checkout for additional page packs router.include_router(youtube_router) # YouTube playlist resolution -router.include_router(google_maps_router) # Google Maps places + reviews scraper router.include_router(prompts_router) router.include_router(memory_router) # User personal memory (memory.md style) router.include_router(team_memory_router) # Workspace team memory diff --git a/surfsense_backend/app/routes/google_maps_routes.py b/surfsense_backend/app/routes/google_maps_routes.py deleted file mode 100644 index 590dc452a..000000000 --- a/surfsense_backend/app/routes/google_maps_routes.py +++ /dev/null @@ -1,64 +0,0 @@ -"""Google Maps scraper routes (Apify actor-compatible).""" - -import logging - -from fastapi import APIRouter, Depends, HTTPException - -from app.auth.context import AuthContext -from app.proprietary.platforms.google_maps import ( - GoogleMapsReviewsInput, - GoogleMapsScrapeInput, - scrape_places, - scrape_reviews, -) -from app.proprietary.platforms.google_maps.scraper import SignInRequiredError -from app.users import require_session_context - -router = APIRouter() -logger = logging.getLogger(__name__) - - -@router.post("/google-maps/scrape") -async def scrape_places_route( - payload: GoogleMapsScrapeInput, - _auth: AuthContext = Depends(require_session_context), -) -> list[dict]: - """Scrape Google Maps places (search terms / URLs / place IDs). - - Apify "Google Maps Scraper"-compatible input/output. Runs inline and is - bounded only by the request's own ``maxCrawledPlacesPerSearch``. - """ - try: - return await scrape_places(payload) - except SignInRequiredError as e: - raise HTTPException( - status_code=403, detail=f"Google sign in required: {e!s}" - ) from e - except Exception as e: - logger.error("Google Maps scrape failed: %s", e) - raise HTTPException( - status_code=502, detail=f"Google Maps scrape failed: {e!s}" - ) from e - - -@router.post("/google-maps/reviews") -async def scrape_reviews_route( - payload: GoogleMapsReviewsInput, - _auth: AuthContext = Depends(require_session_context), -) -> list[dict]: - """Scrape Google Maps reviews for the given place URLs / place IDs. - - Apify "Google Maps Reviews Scraper"-compatible. Runs inline and is bounded - only by the request's own ``maxReviews``. - """ - try: - return await scrape_reviews(payload) - except SignInRequiredError as e: - raise HTTPException( - status_code=403, detail=f"Google sign in required: {e!s}" - ) from e - except Exception as e: - logger.error("Google Maps reviews scrape failed: %s", e) - raise HTTPException( - status_code=502, detail=f"Google Maps reviews scrape failed: {e!s}" - ) from e From b5d221c19cfd8bacf988866bf745256688cf2535 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Fri, 3 Jul 2026 17:57:07 +0200 Subject: [PATCH 62/64] fix(capabilities): map any executor fault to 502 at the REST door --- .../app/capabilities/core/access/rest.py | 13 +++- .../capabilities/access/test_rest_router.py | 70 +++++++++++++++++++ 2 files changed, 82 insertions(+), 1 deletion(-) diff --git a/surfsense_backend/app/capabilities/core/access/rest.py b/surfsense_backend/app/capabilities/core/access/rest.py index df37c79ed..70bcf1b7f 100644 --- a/surfsense_backend/app/capabilities/core/access/rest.py +++ b/surfsense_backend/app/capabilities/core/access/rest.py @@ -13,6 +13,7 @@ from app.capabilities.core.billing import charge_capability, gate_capability from app.capabilities.core.store import all_capabilities from app.capabilities.core.types import Capability, CapabilityContext from app.db import get_async_session +from app.exceptions import ExternalServiceError, SurfSenseError from app.services.web_crawl_credit_service import InsufficientCreditsError from app.users import get_auth_context from app.utils.rbac import check_workspace_access @@ -55,7 +56,17 @@ def _register_verb(router: APIRouter, capability: Capability) -> None: "required_micros": exc.required_micros, }, ) from exc - output = await executor(payload) + + try: + output = await executor(payload) + except (SurfSenseError, HTTPException): + raise + except Exception as exc: + raise ExternalServiceError( + f"The '{capability.name}' capability failed due to an upstream error.", + code="CAPABILITY_UPSTREAM_ERROR", + ) from exc + await charge_capability(output, unit, ctx) return output diff --git a/surfsense_backend/tests/unit/capabilities/access/test_rest_router.py b/surfsense_backend/tests/unit/capabilities/access/test_rest_router.py index e3bc387e9..63d46f0be 100644 --- a/surfsense_backend/tests/unit/capabilities/access/test_rest_router.py +++ b/surfsense_backend/tests/unit/capabilities/access/test_rest_router.py @@ -148,6 +148,76 @@ async def test_rate_limit_blocks_the_workspace(monkeypatch): assert resp.status_code == 429 +def _register_surfsense_handler(app: FastAPI) -> None: + """Minimal stand-in for the app's global SurfSenseError handler.""" + from starlette.responses import JSONResponse + + from app.exceptions import SurfSenseError + + async def _handler(_request, exc: SurfSenseError): + return JSONResponse( + status_code=exc.status_code, + content={"code": exc.code, "message": exc.message}, + ) + + app.add_exception_handler(SurfSenseError, _handler) + + +@pytest.mark.asyncio +async def test_executor_fault_becomes_502(monkeypatch): + """Any non-SurfSense executor error is surfaced as a clean 502, not a 500.""" + + async def _boom(_payload: _EchoInput) -> _EchoOutput: + raise RuntimeError("upstream provider exploded") + + boom = Capability( + name="test.boom", + description="Always fails for tests.", + input_schema=_EchoInput, + output_schema=_EchoOutput, + executor=_boom, + billing_unit=None, + ) + + app = _build_app([boom], monkeypatch) + _register_surfsense_handler(app) + async with _client(app) as client: + resp = await client.post( + "/api/v1/workspaces/7/capabilities/test.boom", + json={"value": "hi"}, + ) + assert resp.status_code == 502 + assert resp.json()["code"] == "CAPABILITY_UPSTREAM_ERROR" + + +@pytest.mark.asyncio +async def test_surfsense_error_passes_through(monkeypatch): + """Intentional, status-carrying errors (e.g. a 403 wall) are not remapped.""" + from app.exceptions import ForbiddenError + + async def _forbidden(_payload: _EchoInput) -> _EchoOutput: + raise ForbiddenError("sign in required", code="GOOGLE_SIGNIN_REQUIRED") + + forbidden = Capability( + name="test.forbidden", + description="Raises a domain 403 for tests.", + input_schema=_EchoInput, + output_schema=_EchoOutput, + executor=_forbidden, + billing_unit=None, + ) + + app = _build_app([forbidden], monkeypatch) + _register_surfsense_handler(app) + async with _client(app) as client: + resp = await client.post( + "/api/v1/workspaces/7/capabilities/test.forbidden", + json={"value": "hi"}, + ) + assert resp.status_code == 403 + assert resp.json()["code"] == "GOOGLE_SIGNIN_REQUIRED" + + @pytest.mark.asyncio async def test_success_charges_once(monkeypatch): from unittest.mock import AsyncMock From 7691e95e366f6a064131c4ce722352dacd44baf1 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Fri, 3 Jul 2026 17:57:13 +0200 Subject: [PATCH 63/64] refactor(capabilities): drop Apify references from web/youtube docstrings --- surfsense_backend/app/capabilities/web/crawl/schemas.py | 9 ++++----- .../app/capabilities/youtube/comments/executor.py | 2 +- .../app/capabilities/youtube/comments/schemas.py | 4 ++-- .../app/capabilities/youtube/scrape/executor.py | 2 +- .../app/capabilities/youtube/scrape/schemas.py | 8 ++++---- 5 files changed, 12 insertions(+), 13 deletions(-) diff --git a/surfsense_backend/app/capabilities/web/crawl/schemas.py b/surfsense_backend/app/capabilities/web/crawl/schemas.py index 38ab0e7f5..6b7b627eb 100644 --- a/surfsense_backend/app/capabilities/web/crawl/schemas.py +++ b/surfsense_backend/app/capabilities/web/crawl/schemas.py @@ -1,13 +1,12 @@ -# ruff: noqa: N815 - public field names intentionally mirror the Apify camelCase spec +# ruff: noqa: N815 - public field names intentionally use camelCase """``web.crawl`` I/O contracts. -An Apify *Website Content Crawler*-style surface (see apify.com/apify/website- -content-crawler and docs.firecrawl.dev): one verb that either scrapes the given +A Website Content Crawler-style surface: one verb that either scrapes the given URLs (``maxCrawlDepth == 0``) or spiders their site (``maxCrawlDepth > 0``), bounded by ``maxCrawlPages`` and kept on the seed's site. -Fields are trimmed to what the proprietary engine honors today. Apify knobs the -engine handles automatically (crawler type, proxy, dynamic-render waits) are +Fields are trimmed to what the proprietary engine honors today. Knobs the engine +handles automatically (crawler type, proxy, dynamic-render waits) are intentionally omitted, as are features we haven't built (URL globs, output formats, click actions, PII handling). """ diff --git a/surfsense_backend/app/capabilities/youtube/comments/executor.py b/surfsense_backend/app/capabilities/youtube/comments/executor.py index a5f4338c9..ceeb2bde3 100644 --- a/surfsense_backend/app/capabilities/youtube/comments/executor.py +++ b/surfsense_backend/app/capabilities/youtube/comments/executor.py @@ -1,4 +1,4 @@ -"""``youtube.comments`` executor: verb input → Apify actor → comment items.""" +"""``youtube.comments`` executor: verb input → scraper → comment items.""" from __future__ import annotations diff --git a/surfsense_backend/app/capabilities/youtube/comments/schemas.py b/surfsense_backend/app/capabilities/youtube/comments/schemas.py index 62eb7c451..e72181a32 100644 --- a/surfsense_backend/app/capabilities/youtube/comments/schemas.py +++ b/surfsense_backend/app/capabilities/youtube/comments/schemas.py @@ -1,7 +1,7 @@ """``youtube.comments`` I/O contracts. -A lean surface over the Apify-compatible ``YouTubeCommentsInput``; the actor's -``CommentItem`` is reused verbatim as the output element for parity. +A lean surface over ``YouTubeCommentsInput``; the scraper's ``CommentItem`` is +reused verbatim as the output element. """ from __future__ import annotations diff --git a/surfsense_backend/app/capabilities/youtube/scrape/executor.py b/surfsense_backend/app/capabilities/youtube/scrape/executor.py index 46d4789dc..17047be90 100644 --- a/surfsense_backend/app/capabilities/youtube/scrape/executor.py +++ b/surfsense_backend/app/capabilities/youtube/scrape/executor.py @@ -1,4 +1,4 @@ -"""``youtube.scrape`` executor: verb input → Apify actor → video items.""" +"""``youtube.scrape`` executor: verb input → scraper → video items.""" from __future__ import annotations diff --git a/surfsense_backend/app/capabilities/youtube/scrape/schemas.py b/surfsense_backend/app/capabilities/youtube/scrape/schemas.py index f24e19828..8947f601f 100644 --- a/surfsense_backend/app/capabilities/youtube/scrape/schemas.py +++ b/surfsense_backend/app/capabilities/youtube/scrape/schemas.py @@ -1,9 +1,9 @@ """``youtube.scrape`` I/O contracts. -A lean, agent-friendly surface over the Apify-compatible ``YouTubeScrapeInput`` -(``app/proprietary/platforms/youtube``). The executor maps this to the full actor -input; the actor's ``VideoItem`` is reused verbatim as the output element so the -verb stays parity-faithful with the underlying Apify shape. +A lean, agent-friendly surface over ``YouTubeScrapeInput`` +(``app/proprietary/platforms/youtube``). The executor maps this to the full +scraper input; the scraper's ``VideoItem`` is reused verbatim as the output +element. """ from __future__ import annotations From dd6172811113f59de64192581d179c77b28cb2fc Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Fri, 3 Jul 2026 18:36:48 +0200 Subject: [PATCH 64/64] refactor(subagents): drop automation/monitoring framing from prompts (keep in-chat comparison) --- .../subagents/builtins/google_maps/description.md | 2 +- .../subagents/builtins/google_maps/system_prompt.md | 4 ++-- .../subagents/builtins/web_crawler/description.md | 2 +- .../subagents/builtins/web_crawler/system_prompt.md | 4 ++-- .../subagents/builtins/youtube/description.md | 2 +- .../subagents/builtins/youtube/system_prompt.md | 4 ++-- 6 files changed, 9 insertions(+), 9 deletions(-) diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/description.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/description.md index c76722a5c..4cb33109d 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/description.md +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/description.md @@ -1,2 +1,2 @@ Google Maps specialist: pulls structured data from Google Maps — finds places by search query (optionally scoped to a location), or resolves a Maps URL / place ID into place details (name, address, category, phone, website, rating, review count, coordinates, opening hours), and fetches a place's reviews (author, text, star rating, owner response, dates). Also compares fresh Maps data against earlier findings in this chat. -Use whenever the task involves Google Maps places, local businesses, or a google.com/maps link. Triggers include "find near/in X", "get details for this place", "phone/address/hours/rating of X", "how many reviews", "get the reviews for this place", "what are people saying about this business", and recurring "what changed" checks on Maps data. Not for general web pages (use the web crawling specialist) or YouTube (use the YouTube specialist). +Use whenever the task involves Google Maps places, local businesses, or a google.com/maps link. Triggers include "find near/in X", "get details for this place", "phone/address/hours/rating of X", "how many reviews", "get the reviews for this place", "what are people saying about this business", and comparisons against earlier Maps results in this chat. Not for general web pages (use the web crawling specialist) or YouTube (use the YouTube specialist). diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/system_prompt.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/system_prompt.md index a048ab982..d598e8fe8 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/system_prompt.md +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/google_maps/system_prompt.md @@ -2,7 +2,7 @@ You are the SurfSense Google Maps sub-agent. You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis. -Answer the delegated question from live Google Maps data gathered with your verbs, including "what changed" comparisons against evidence already in this conversation. +Answer the delegated question from live Google Maps data gathered with your verbs, comparing against earlier results already in this conversation when the task calls for it. @@ -16,7 +16,7 @@ Answer the delegated question from live Google Maps data gathered with your verb - Need richer detail (opening hours, popular times, extra contact info): set `include_details=true`. - Reviews / sentiment on specific places: call `google_maps_reviews` with the place `urls` or `place_ids`. - Batch multiple queries, URLs, or place IDs into one call rather than many single-item calls. -- "What changed" / monitoring: pull the current values, compare against the prior values in this conversation's earlier tool results, and report concrete deltas (added, removed, old -> new). +- Comparison requests: pull the current values, compare against prior values already in this conversation's earlier tool results, and report concrete deltas (added, removed, old -> new). diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/description.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/description.md index a687e8ad3..4e084d767 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/description.md +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/description.md @@ -1,2 +1,2 @@ Web crawling specialist: fetches live public web pages by URL and can spider a whole website (follow its links to a given depth), returning clean markdown, page metadata, and crawl provenance. Also compares freshly crawled data against earlier findings in this chat. -Use whenever the task needs current content pulled from the open web rather than the workspace's own documents or connectors. Triggers include "scrape", "crawl", "fetch this URL/page", "read this website", "crawl this site", "get the pages under X", "check the price/stock/listing", "monitor this page", and "what changed since last time". Not for YouTube links (use the youtube specialist) or for searching the web to discover unknown URLs. +Use whenever the task needs current content pulled from the open web rather than the workspace's own documents or connectors. Triggers include "scrape", "crawl", "fetch this URL/page", "read this website", "crawl this site", "get the pages under X", "check the price/stock/listing", and "what changed vs earlier in this chat". Not for YouTube links (use the youtube specialist) or for searching the web to discover unknown URLs. diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/system_prompt.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/system_prompt.md index 4133f7bc9..2983f3274 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/system_prompt.md +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/web_crawler/system_prompt.md @@ -2,7 +2,7 @@ You are the SurfSense web crawling sub-agent. You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis. -Answer the delegated question from live web evidence gathered with `web_crawl`, including "what changed" comparisons against evidence already in this conversation. +Answer the delegated question from live web evidence gathered with `web_crawl`, comparing against earlier results already in this conversation when the task calls for it. @@ -14,7 +14,7 @@ Answer the delegated question from live web evidence gathered with `web_crawl`, - Whole site / "pages under X": set `maxCrawlDepth` to 1+ to follow links, and cap the run with `maxCrawlPages`. The crawl stays on the start URL's site. - Batch known URLs into one `web_crawl` call (pass them all in `startUrls`) rather than many single-URL calls. - Keep depth and page caps as small as the task allows — each fetched page is billable. -- "What changed" / monitoring: crawl the current values, compare against the prior values in this conversation's earlier tool results, and report concrete deltas (added, removed, old -> new). +- Comparison requests: crawl the current values, compare against prior values already in this conversation's earlier tool results, and report concrete deltas (added, removed, old -> new). diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/description.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/description.md index 6ccc9a0d5..bc5e96deb 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/description.md +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/description.md @@ -1,2 +1,2 @@ YouTube specialist: pulls structured data from YouTube — videos, channels, playlists, shorts, and hashtags (title, views, likes, publish date, channel info, description, optional subtitles), finds videos by search query, and fetches a video's comments and replies. Also compares fresh YouTube data against earlier findings in this chat. -Use whenever the task involves YouTube content or a youtube.com/youtu.be link. Triggers include "get this YouTube video/channel/playlist", "find videos about X on YouTube", "how many views/likes", "get the transcript/subtitles", "get the comments on this video", "what are people saying about this video", and recurring "what changed" checks on YouTube data. Not for general web pages (use the web crawling specialist for non-YouTube URLs). +Use whenever the task involves YouTube content or a youtube.com/youtu.be link. Triggers include "get this YouTube video/channel/playlist", "find videos about X on YouTube", "how many views/likes", "get the transcript/subtitles", "get the comments on this video", "what are people saying about this video", and comparisons against earlier YouTube results in this chat. Not for general web pages (use the web crawling specialist for non-YouTube URLs). diff --git a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/system_prompt.md b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/system_prompt.md index 6c500c777..9b5ebc472 100644 --- a/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/system_prompt.md +++ b/surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/youtube/system_prompt.md @@ -2,7 +2,7 @@ You are the SurfSense YouTube sub-agent. You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis. -Answer the delegated question from live YouTube data gathered with your verbs, including "what changed" comparisons against evidence already in this conversation. +Answer the delegated question from live YouTube data gathered with your verbs, comparing against earlier results already in this conversation when the task calls for it. @@ -15,7 +15,7 @@ Answer the delegated question from live YouTube data gathered with your verbs, i - Finding videos on a topic: call `youtube_scrape` with `search_queries`. - Comments / sentiment on specific videos: call `youtube_comments` with the video `urls`. - Batch multiple URLs (or queries) into one call rather than many single-item calls. -- "What changed" / monitoring: pull the current values, compare against the prior values in this conversation's earlier tool results, and report concrete deltas (added, removed, old -> new). +- Comparison requests: pull the current values, compare against prior values already in this conversation's earlier tool results, and report concrete deltas (added, removed, old -> new).