apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-06 22:12:12 +02:00
Code Issues Projects Releases Packages Wiki Activity

docs(plans): re-cast Phase 4-7 to stateless scraper-APIs, flatten subplans

Browse source
This commit is contained in:
CREDO23 2026-07-01 14:51:29 +02:00
parent 05bbb64527
commit 0944d9bb78
21 changed files with 447 additions and 2450 deletions
Download patch file Download diff file Expand all files Collapse all files

124
plans/backend/00-umbrella-plan.md
View file

@ -2,16 +2,30 @@
> Master roadmap for the Competitive Intelligence pivot. Each phase becomes its own subplan saved in this folder (`plans/backend/`). > Master roadmap for the Competitive Intelligence pivot. Each phase becomes its own subplan saved in this folder (`plans/backend/`).
This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shipped), then the WebURL crawler moat (Phase 3, shipped), then the CI product itself as **four revamped phases** — **Phase 4 Capabilities & Access**, **Phase 5 Intelligence & Timeline**, **Phase 6 Triggers**, **Phase 7 Orchestration** (canonical subplans in [`revamp phases 4-7/`](revamp%20phases%204-7/00-overview.md)). **NOTE:** the original "connectors → Pipelines" plan (old Phases 4–7) was **superseded on 2026-06-30** — see the Architecture correction below. The earlier "Phase 5′ automation-enhancements" framing is also retired (folded into Phases 4/5/6 of the revamp). This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shipped), then the WebURL crawler moat (Phase 3, shipped), then the CI product itself as **four revamped phases** — **Phase 4 Capabilities** ([`04-capabilities.md`](04-capabilities.md)), **Phase 5 Access** ([`05-access.md`](05-access.md)), **Phase 6 Ongoing-Automation** ([`06-ongoing-automation.md`](06-ongoing-automation.md), design deferred), **Phase 7 Orchestration** ([`07-orchestration.md`](07-orchestration.md)). **NOTE:** the original "connectors → Pipelines" plan (old Phases 4–7) was superseded on 2026-06-30, and the stateful **Tracker + Timeline + Triggers** design was superseded on **2026-07-01** (scraper-APIs-first, stateless, chat-native) — see the Architecture corrections below.
> SCOPE: This umbrella currently covers the BACKEND only (`surfsense_backend`). Frontend (`surfsense_web`) and client apps (desktop, Obsidian, browser extension) will get their own umbrella/subplans LATER, once the backend is fully working as expected. Frontend-facing decisions (URL segment, TS types, i18n copy) are recorded below where relevant but are out of scope for the active phases. > SCOPE: This umbrella currently covers the BACKEND only (`surfsense_backend`). Frontend (`surfsense_web`) and client apps (desktop, Obsidian, browser extension) will get their own umbrella/subplans LATER, once the backend is fully working as expected. Frontend-facing decisions (URL segment, TS types, i18n copy) are recorded below where relevant but are out of scope for the active phases.
## Positioning ## Positioning
"NotebookLM for Competitive Intelligence" — each WorkSpace acts as a workspace for setting up competitive-intelligence-optimised notebooks. Premium, open-source, self-hostable **scraper APIs** (Bright Data / Firecrawl / Scrapfly-class) that solve the hard parts — bypass, proxies, stealth — and return **cleaned, AI-ready data** for Google Maps + general web. A chat agent uses those APIs to answer competitive-intelligence questions and, for standing needs, keeps watching over time.
## ⚠️ Architecture pivot (2026-07-01) — scraper-APIs-first, stateless, chat-native
> **This supersedes the stateful Tracker/Timeline/Triggers design (the 2026-06-30 revamp's Phases 5–6).** The canonical Phase 4–7 subplans (`04`–`07`, below) are re-cast to four phases:
>
> - **The product = scraper capabilities**, not a CI engine. `web.*` + `maps.*` typed verbs return cleaned, AI-ready data, exposed identically through **REST + API key**, an **MCP server**, and **chat**, all generated from one capability registry. These endpoints are the revenue driver (premium, OSS, self-hostable).
> - **Stateless.** No Timeline (3-store delta), no `Tracker`, no diff/materiality engine, no stored `entity_changes`. Memory = the **chat history**; the agent reasons over prior tool outputs in context to report "what changed."
> - **Direct calls.** Capabilities return their result directly — no job store, no `completed | pending + job_id` envelope, no `deliverable_wait` polling. Async results reach the client via **write-then-sync / stream** (existing SSE or a Zero-published table).
> - **Automation = a persistent ongoing chat** that periodically re-invokes verbs. This replaces the `Triggers` subsystem; the periodic mechanism is **design-deferred** (Phase 6).
> - **KB stays input-only**; crawled data is never indexed.
>
> **Phase mapping:** **04 Capabilities** · **05 Access** · **06 Ongoing-Automation** (deferred) · **07 Orchestration** (`intelligence_agent`). The subplans `05a-timeline`/`05b-intelligence`/`06-triggers` are **removed** (history in git); `04a`/`04b` are folded into `04`/`05`.
## ⚠️ Architecture correction (2026-06-30) — Pipelines dropped; automations + input-only KB adopted ## ⚠️ Architecture correction (2026-06-30) — Pipelines dropped; automations + input-only KB adopted
> **Historical record.** The pipelines-drop + input-only-KB decisions below still hold. The stateful Tracker/Timeline/Triggers framing in its "Net effect" bullets was itself superseded by the **2026-07-01 pivot** above.
> **This supersedes the original Pipelines paradigm (Phases 5–7).** During Phase 4 discussion we concluded that the *"sync data into the KB, then operate over it"* model is wrong for competitive intelligence. Verified against `references/opencode` — a coding agent that pulls context **live** via `read`/`grep`/`webfetch`/`websearch` tools (`packages/opencode/src/tool/`) and persists **sessions**, never a scraped corpus; its `websearch` tool even live-crawls at query time (`tool/websearch.ts`). The corrected model: > **This supersedes the original Pipelines paradigm (Phases 5–7).** During Phase 4 discussion we concluded that the *"sync data into the KB, then operate over it"* model is wrong for competitive intelligence. Verified against `references/opencode` — a coding agent that pulls context **live** via `read`/`grep`/`webfetch`/`websearch` tools (`packages/opencode/src/tool/`) and persists **sessions**, never a scraped corpus; its `websearch` tool even live-crawls at query time (`tool/websearch.ts`). The corrected model:
> >
> - **Knowledge Base = input-only** — the user's personal files/context that *enriches* the agent. Filled by file uploads + file connectors (Google Drive / Dropbox / OneDrive). **Nothing scrapes into it.** Its job is to be searchable *user-provided* context, not a dump of fetched pages. > - **Knowledge Base = input-only** — the user's personal files/context that *enriches* the agent. Filled by file uploads + file connectors (Google Drive / Dropbox / OneDrive). **Nothing scrapes into it.** Its job is to be searchable *user-provided* context, not a dump of fetched pages.
@ -20,7 +34,7 @@ This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shi
> - **Automations are the scheduling + run-history substrate.** An automation run already invokes the *full chat agent* (with the crawler/search tools) on a **cron/event** trigger and persists **run history** (`automation_runs`) — exactly the recurring-fetch + monitoring loop Pipelines were reinventing. You store the agent's *insight per run*, not raw snapshots. > - **Automations are the scheduling + run-history substrate.** An automation run already invokes the *full chat agent* (with the crawler/search tools) on a **cron/event** trigger and persists **run history** (`automation_runs`) — exactly the recurring-fetch + monitoring loop Pipelines were reinventing. You store the agent's *insight per run*, not raw snapshots.
> - **`WEBCRAWLER_CONNECTOR` is retired** — the crawler is a native tool, not a connector/data-source. > - **`WEBCRAWLER_CONNECTOR` is retired** — the crawler is a native tool, not a connector/data-source.
> >
> **Net effect on this umbrella:** the old **Pipelines** stack (old Phases 5–7) is dropped and **Phases 4–7 are re-cast** by the canonical revamp in [`revamp phases 4-7/`](revamp%20phases%204-7/00-overview.md): > **Net effect on this umbrella:** the old **Pipelines** stack (old Phases 5–7) is dropped and **Phases 4–7 are re-cast** by the canonical subplans `04`–`07` (this framing was itself superseded by the 2026-07-01 pivot above):
> - **Phase 4 — Capabilities & Access** (`revamp/04a` + `04b`): turn the crawler + search into **typed, callable verbs** (MVP = `web.scrape` + `web.discover`; per-platform scrapers like `maps.*`/`linkedin.*` are *later, uncommitted* drop-ins) over a single **capability registry**, exposed identically through **chat / REST+API-key / MCP** doors. Ships **Product A** (stateless utility) — revenue day one. *The old `04a-connector-category` is demoted to backward-compat hygiene; the old `04b-source-discovery` is absorbed as the `web.discover` verb.* > - **Phase 4 — Capabilities & Access** (`revamp/04a` + `04b`): turn the crawler + search into **typed, callable verbs** (MVP = `web.scrape` + `web.discover`; per-platform scrapers like `maps.*`/`linkedin.*` are *later, uncommitted* drop-ins) over a single **capability registry**, exposed identically through **chat / REST+API-key / MCP** doors. Ships **Product A** (stateless utility) — revenue day one. *The old `04a-connector-category` is demoted to backward-compat hygiene; the old `04b-source-discovery` is absorbed as the `web.discover` verb.*
> - **Phase 5 — Intelligence & Timeline** (`revamp/05a` + `05b`): the `Tracker` primitive + a **3-store delta Timeline** (`tracked_entities`/`entity_current_state`/`entity_changes`, "store deltas, no change → no row"). This is the durable CI state and **the moat** — it replaces "pipelines" as the *standing concern*. Ships the **Product B** engine. > - **Phase 5 — Intelligence & Timeline** (`revamp/05a` + `05b`): the `Tracker` primitive + a **3-store delta Timeline** (`tracked_entities`/`entity_current_state`/`entity_changes`, "store deltas, no change → no row"). This is the durable CI state and **the moat** — it replaces "pipelines" as the *standing concern*. Ships the **Product B** engine.
> - **Phase 6 — Triggers** (`revamp/06`): decide **when** a Tracker refreshes. `refresh(tracker)` is trigger-agnostic; recurrence is a **CI action on the existing automations** (reuse its schedule selector + `AutomationRun` — no new scheduler, no new run table). Alert **delivery is separate** (via `app/notifications/`; automations have no delivery path). > - **Phase 6 — Triggers** (`revamp/06`): decide **when** a Tracker refreshes. `refresh(tracker)` is trigger-agnostic; recurrence is a **CI action on the existing automations** (reuse its schedule selector + `AutomationRun` — no new scheduler, no new run table). Alert **delivery is separate** (via `app/notifications/`; automations have no delivery path).
@ -28,27 +42,23 @@ This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shi
> >
> **KB stays input-only; `WEBCRAWLER_CONNECTOR` retired; crawler/search are native tools (later a platform API key).** The three correction decisions land in their real homes: **bill crawls at the capability executor** (`revamp/04a`, works uniformly across chat/automation/REST/MCP/cron — see 03c below), **diffable run history = the Timeline delta store** (`revamp/05a`, *not* `automation_runs.output`), and **"run now" = a Trigger adapter** (`revamp/06`). Phase 8 platform scrapers re-slot as a family of **individual native endpoints** — each one a capability verb → an agent tool + a dev API-key REST endpoint — added incrementally (**none, incl. Google Maps, committed for MVP**; they're examples of the pattern), not pipeline executors. > **KB stays input-only; `WEBCRAWLER_CONNECTOR` retired; crawler/search are native tools (later a platform API key).** The three correction decisions land in their real homes: **bill crawls at the capability executor** (`revamp/04a`, works uniformly across chat/automation/REST/MCP/cron — see 03c below), **diffable run history = the Timeline delta store** (`revamp/05a`, *not* `automation_runs.output`), and **"run now" = a Trigger adapter** (`revamp/06`). Phase 8 platform scrapers re-slot as a family of **individual native endpoints** — each one a capability verb → an agent tool + a dev API-key REST endpoint — added incrementally (**none, incl. Google Maps, committed for MVP**; they're examples of the pattern), not pipeline executors.
## Target architecture (corrected + revamped) ## Target architecture (scraper-APIs-first, stateless)
> Full stateless/stateful end-to-end flow diagrams live in [`revamp phases 4-7/00b-pipeline-diagrams.md`](revamp%20phases%204-7/00b-pipeline-diagrams.md). Summary below. > Full end-to-end flow diagrams live in [`00b-pipeline-diagrams.md`](00b-pipeline-diagrams.md). Summary below.
```mermaid ```mermaid
flowchart TD flowchart TD
ACQ[["Acquisition (Phase 3, shipped): WebURL crawler + proxy/stealth/captcha + 03c billing · + Maps actor (P8)"]] ACQ[["Acquisition (Phase 3, shipped): WebURL crawler + proxy/stealth/captcha + 03c billing · + Maps actor (new)"]]
REG["Capability registry (P4a): web.scrape · web.discover (MVP) · <platform>.* scrapers (later) — typed verbs, bill at executor"] REG["Capability registry (P4): web.scrape · web.discover · maps.search/place/reviews — typed verbs, cleaned AI-ready output, bill at executor"]
ACQ --> REG ACQ --> REG
REG --> DOORS["Access doors (P4b): chat tools · REST+API key · MCP — generated from the registry"] REG --> DOORS["Access doors (P5): REST + API key · MCP server · chat tools — generated from the registry"]
DOORS --> AGENT["intelligence_agent subagent (P7): intent routing · verb chains · Tracker crafting"] DOORS --> AGENT["intelligence_agent subagent (P7): intent routing · verb chains · 'what changed' from chat history"]
KB[("Knowledge Base — input-only: user files/context (uploads + Drive/Dropbox/OneDrive)")] --> AGENT KB[("Knowledge Base — input-only: user files/context (uploads + Drive/Dropbox/OneDrive)")] --> AGENT
CONN["Connectors = MCP tools (BYO) + file/KB-input"] --> AGENT CONN["Connectors = MCP tools (BYO) + file/KB-input"] --> AGENT
AGENT -->|one-shot / Product A| ANS["plain-language answer (nothing persists)"] AGENT -->|one-shot| ANS["plain-language answer (nothing persists)"]
AGENT -->|standing concern / Product B| TR["Tracker (P5b): locked schema · refresh hot loop"] AGENT -.->|standing need| ONG["Ongoing-Automation (P6, DEFERRED): persistent chat re-invokes verbs; memory = chat history"]
REG --> TR ONG -.-> REG
TR --> TL[("Timeline (P5a): tracked_entities · entity_current_state · entity_changes — delta moat")] APIKEY["Platform API key"] --> DOORS
TRIG["Triggers (P6): manual · agent · external-cron · CI automation action"] --> TR
AUTO["Automations (existing): schedule selector + AutomationRun (recurrence + audit; NO delivery)"] --> TRIG
TL --> DELIVER["alerts on material change → app/notifications (Zero-synced)"]
APIKEY["Platform API key (future)"] --> DOORS
``` ```
<details> <details>
@ -78,14 +88,14 @@ flowchart TD
- Full rename SearchSpace -> WorkSpace across DB, API, URLs, code, satellite apps. - Full rename SearchSpace -> WorkSpace across DB, API, URLs, code, satellite apps.
- Canonical names (proposed defaults): DB table `workspaces`, column `workspace_id`, RBAC tables `workspace_roles` / `workspace_memberships` / `workspace_invites`, API base `/workspaces` (consolidating today's `/searchspaces` vs `/search-spaces` split), URL segment `[workspace_id]`, settings folder `workspace-settings`, TS type `Workspace`. - Canonical names (proposed defaults): DB table `workspaces`, column `workspace_id`, RBAC tables `workspace_roles` / `workspace_memberships` / `workspace_invites`, API base `/workspaces` (consolidating today's `/searchspaces` vs `/search-spaces` split), URL segment `[workspace_id]`, settings folder `workspace-settings`, TS type `Workspace`.
- **Capabilities, not connectors, are the core of Phase 4.** The crawler + search become typed **verbs** (MVP `web.scrape` + `web.discover`; per-platform scrapers like `maps.*` are later, uncommitted drop-ins) in a single **capability registry** that generates the chat/REST/MCP doors identically (revamp `04a`/`04b`). A capability is *call → data → bill*; no `SearchSourceConnector` row, no KB write, no schedule. The **old connector two-type taxonomy is demoted to backward-compat hygiene** (kept so existing rows/KB docs stay searchable) — the one genuinely useful piece, the **`MCP_CONNECTOR` (BYO MCP) routing-gap fix**, moves into revamp `04b` (consume-user-MCP). Surviving connectors: **file/KB-input** (Google Drive native+Composio, OneDrive, Dropbox, uploads → the input-only KB) + **BYO `MCP_CONNECTOR`** (act in chat). **`WEBCRAWLER_CONNECTOR` RETIRED** (native tool); branded natives stay `MIGRATING` (off); Obsidian/Circleback `DISABLED`. Artifacts stay in the existing `deliverables` system. **~~pipeline-eligibility~~ removed.** - **Capabilities, not connectors, are the core of the product.** The crawler + search + Maps actor become typed **verbs** (`web.scrape`, `web.discover`, `maps.search/place/reviews`) in a single **capability registry** that generates the REST/MCP/chat doors identically (`04`/`05`). A capability is *call → cleaned data → bill*; no `SearchSourceConnector` row, no KB write, no schedule. The **old connector two-type taxonomy is demoted to backward-compat hygiene** (kept so existing rows/KB docs stay searchable) — the one genuinely useful piece, the **`MCP_CONNECTOR` (BYO MCP) routing-gap fix**, moves into `05` (consume-user-MCP). Surviving connectors: **file/KB-input** (Google Drive native+Composio, OneDrive, Dropbox, uploads → the input-only KB) + **BYO `MCP_CONNECTOR`** (act in chat). **`WEBCRAWLER_CONNECTOR` RETIRED** (native tool); branded natives stay `MIGRATING` (off); Obsidian/Circleback `DISABLED`. Artifacts stay in the existing `deliverables` system.
- Web search APIs (SearXNG, Linkup, Baidu) are repurposed as the **`web.discover` capability** (a platform-level search provider set): given a topic/competitor, suggest candidate URLs the agent (or a Tracker binding) can feed to `web.scrape`. Not a standalone connector type, does not index data. NOTE: Tavily and Serper are being REMOVED from the search infra and are not part of this set. - Web search APIs (SearXNG, Linkup, Baidu) are repurposed as the **`web.discover` capability** (a platform-level search provider set): given a topic/competitor, suggest candidate URLs the agent can feed to `web.scrape`. Not a standalone connector type, does not index data. NOTE: Tavily and Serper are being REMOVED from the search infra and are not part of this set.
- **`Tracker` + Timeline are the standing-concern primitive** (replaces "pipeline"). A Tracker binds a capability + input and a locked, versioned `definition` (`field_schema`/`identity_rule`/`materiality`); its `refresh(tracker)` hot loop writes a **3-store delta Timeline** (`tracked_entities`/`entity_current_state`/`entity_changes`). **Store deltas, no change → no row.** The Timeline — not the KB, not `automation_runs` — is the only new durable CI state (revamp `05a`/`05b`). - **Stateless — the chat history is the memory.** No Timeline, no `Tracker`, no diff/materiality engine, no stored `entity_changes`. "What changed" is the agent reasoning over prior tool outputs already in context (`07`). Standing needs are served by a **persistent ongoing chat** that periodically re-invokes verbs (`06`, design deferred).
- **Bill at the capability executor.** Each verb declares a `billing_unit`; the executor charges the workspace owner once per billable success via `WebCrawlCreditService` (03c) — so chat, automation/recurring, REST, MCP, and external-cron all meter **uniformly**. (Closes the confirmed gap where automation-run crawls billed nothing; see 03c + the Architecture correction.) - **Bill at the capability executor.** Each verb declares a `billing_unit`; the executor charges the workspace owner once per billable success via the billing service (`WebCrawlCreditService`, 03c, first provider) — so chat, ongoing automation, REST, and MCP all meter **uniformly**. `maps.*` registers a new per-place / per-page unit.
- Obsidian and Circleback (push/webhook sources) are DISABLED for the MVP. - Obsidian and Circleback (push/webhook sources) are DISABLED for the MVP.
- MCP-availability audit complete: BookStack (community MCP servers), Elasticsearch (official Elastic Agent Builder MCP), and Luma (community MCP servers) all have MCP available, so none are `DISABLED` — they're tagged `MIGRATING` (turned off for MVP like the other branded natives, pending the post-MVP MCP re-point). - MCP-availability audit complete: BookStack (community MCP servers), Elasticsearch (official Elastic Agent Builder MCP), and Luma (community MCP servers) all have MCP available, so none are `DISABLED` — they're tagged `MIGRATING` (turned off for MVP like the other branded natives, pending the post-MVP MCP re-point).
- ~~`Pipeline` and `PipelineRun` are new first-class tables.~~ **SUPERSEDED** — no Pipelines. The standing-concern need is met by the **`Tracker` + Timeline** (revamp `05`), and **recurrence + audit reuse the existing automations** (schedule selector + `AutomationRun`) via a CI **action** (revamp `06`) — **no new scheduler, no new run table**. **Delivery does NOT reuse automations** (no such path; `automation_runs.output` is never written) — alerts wire to the separate `app/notifications/` system. Uploads simply populate the input-only KB (existing upload code, no pipeline wrapper). - ~~`Pipeline` and `PipelineRun` are new first-class tables.~~ **SUPERSEDED** — no Pipelines, and (2026-07-01) no Tracker/Timeline either. Standing needs are a **persistent ongoing chat** re-invoking verbs (`06`, deferred); no new scheduler, no new run/state table. Uploads simply populate the input-only KB (existing upload code, no pipeline wrapper).
- ~~The chat agent gets read-only access to pipeline run history~~ → the agent answers "what changed?" by reading the **Timeline** (`query_timeline`, revamp `05a`), not by re-deriving from run logs. Recurring execution audit rides `AutomationRun` (already persisted + queryable). No separate pipeline-run context tool. - ~~The chat agent gets read-only access to pipeline run history~~ → the agent answers "what changed?" by **reasoning over the chat history** (prior tool outputs already in context, `07`), not by re-deriving from run logs or a stored timeline.
- **Web search + WebURL crawler are platform-native** (agent tools now; developer platform-API-key access later). The KB is **input-only**. - **Web search + WebURL crawler are platform-native** (agent tools now; developer platform-API-key access later). The KB is **input-only**.
- Deferred (post-MVP): platform scraper implementations (**Phase 8 — Platform actors**, public-data-only first; built in-house on the Phase-3 fetch core) surfaced as **native tools / API endpoints**, public pay-as-you-go API over the crawler/search, public MCP server exposing the KB. Logged-in/account-based scraping deferred beyond that. - Deferred (post-MVP): platform scraper implementations (**Phase 8 — Platform actors**, public-data-only first; built in-house on the Phase-3 fetch core) surfaced as **native tools / API endpoints**, public pay-as-you-go API over the crawler/search, public MCP server exposing the KB. Logged-in/account-based scraping deferred beyond that.
@ -124,52 +134,50 @@ flowchart TD
The Universal WebURL Crawler is the flagship Type-1 data source (**the moat**). This phase hardens it into an **in-house, best-effort "undetectable, captcha-bypassing" crawler** on a single framework (Scrapling): it standardizes the fetch layer, generalizes proxy support, introduces pay-as-you-go crawl credits, adds **stealth hardening** (geoip coherence, persistent profiles, headed/Xvfb, fonts, humanization, a block classifier + per-domain strategy memory), and adds **opt-in captcha solving**. All tiers plug in behind a single `FetchStrategy` seam returning `CrawlOutcome`, so callers never depend on *how* a page was fetched — that seam is what lets the moat grow (and lets a **deferred paid-unblocker tier** drop in later by config). **Strategy decision (recorded in the log):** CloakBrowser is rejected on licensing and external unblocker APIs are deferred; we hold an in-house bypass moat for ~4–6 months, then move hostile targets to a paid tier if demand/maintenance justifies it. **Logged-in/account-based bypass is out of scope** (public data only this MVP). It is broken into focused subplans: The Universal WebURL Crawler is the flagship Type-1 data source (**the moat**). This phase hardens it into an **in-house, best-effort "undetectable, captcha-bypassing" crawler** on a single framework (Scrapling): it standardizes the fetch layer, generalizes proxy support, introduces pay-as-you-go crawl credits, adds **stealth hardening** (geoip coherence, persistent profiles, headed/Xvfb, fonts, humanization, a block classifier + per-domain strategy memory), and adds **opt-in captcha solving**. All tiers plug in behind a single `FetchStrategy` seam returning `CrawlOutcome`, so callers never depend on *how* a page was fetched — that seam is what lets the moat grow (and lets a **deferred paid-unblocker tier** drop in later by config). **Strategy decision (recorded in the log):** CloakBrowser is rejected on licensing and external unblocker APIs are deferred; we hold an in-house bypass moat for ~4–6 months, then move hostile targets to a paid tier if demand/maintenance justifies it. **Logged-in/account-based bypass is out of scope** (public data only this MVP). It is broken into focused subplans:
> **Sequencing within Phase 3 (critical path vs hardening).** Only **`03a` + `03b` + `03c`** are on the **MVP critical path**. The crawler (`03a` `CrawlOutcome`) + proxy provider (`03b`) + `03c` `WebCrawlCreditService` billing seam are consumed by the **capability verbs** (revamp `04a`) that back every chat / automation / REST / MCP path (the crawler is a platform-native tool — see Architecture correction; the old "Phases 4–7 pipelines consume this" framing is superseded). **`03d` (captcha), `03e` (stealth hardening), and `03f` (test harness) are hardening/measurement that nothing downstream imports** — they tune the *same* seam behind the *same* contract, so they can land **in parallel with, or after, Phase 4** without blocking the pivot. Recommended build order: `03a → 03b → 03c` (then revamp `04a` Capabilities → `04b` Access → `05` Intelligence & Timeline), with `03e → 03d → 03f` slotted in whenever crawler robustness is prioritized. > **Sequencing within Phase 3 (critical path vs hardening).** Only **`03a` + `03b` + `03c`** are on the **MVP critical path**. The crawler (`03a` `CrawlOutcome`) + proxy provider (`03b`) + `03c` `WebCrawlCreditService` billing seam are consumed by the **capability verbs** (`04`) that back every chat / automation / REST / MCP path (the crawler is a platform-native tool — see Architecture pivot; the old "Phases 4–7 pipelines consume this" framing is superseded). **`03d` (captcha), `03e` (stealth hardening), and `03f` (test harness) are hardening/measurement that nothing downstream imports** — they tune the *same* seam behind the *same* contract, so they can land **in parallel with, or after, Phase 4** without blocking the pivot. Recommended build order: `03a → 03b → 03c` (then `04` Capabilities → `05` Access → `07` Orchestration), with `03e → 03d → 03f` slotted in whenever crawler robustness is prioritized.
- **`03a-crawler-core.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `5c36cd3`; crawler moved to `app/proprietary/web_crawler/`, `impersonate="chrome"` + `solve_cloudflare=True` shipped)* — Standardize the fetch layer on Scrapling. **Remove Firecrawl entirely** (no other frameworks). Define crisp per-URL success/empty/failure semantics, keep Trafilatura extraction, and expose a single billable "successful crawl" signal (one unit per URL that yields usable content, regardless of how many internal fallback tiers ran). - **`03a-crawler-core.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `5c36cd3`; crawler moved to `app/proprietary/web_crawler/`, `impersonate="chrome"` + `solve_cloudflare=True` shipped)* — Standardize the fetch layer on Scrapling. **Remove Firecrawl entirely** (no other frameworks). Define crisp per-URL success/empty/failure semantics, keep Trafilatura extraction, and expose a single billable "successful crawl" signal (one unit per URL that yields usable content, regardless of how many internal fallback tiers ran).
- **`03b-proxy-expansion.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `6226012`)* — Add a BYO `CustomProxyProvider` (the only new provider — **no branded vendors**) alongside `anonymous_proxies`, selectable via a **single, app-wide** `Config.PROXY_PROVIDER`. Add bounded client-side rotation+retry via Scrapling's `ProxyRotator`/`is_proxy_error` **only** when the active provider is pool-backed (`CUSTOM_PROXY_URLS`); single-endpoint providers (incl. `anonymous_proxies`) stay the default and no-op the retry. **No per-connector/per-crawl selection** (one provider app-wide); a per-capability/per-Tracker override is left as a no-op seam (revamp `04a`/`05`), not built. - **`03b-proxy-expansion.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `6226012`)* — Add a BYO `CustomProxyProvider` (the only new provider — **no branded vendors**) alongside `anonymous_proxies`, selectable via a **single, app-wide** `Config.PROXY_PROVIDER`. Add bounded client-side rotation+retry via Scrapling's `ProxyRotator`/`is_proxy_error` **only** when the active provider is pool-backed (`CUSTOM_PROXY_URLS`); single-endpoint providers (incl. `anonymous_proxies`) stay the default and no-op the retry. **No per-connector/per-crawl selection** (one provider app-wide); a per-capability override is left as a no-op seam (`04`), not built.
- **`03c-crawl-billing.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `17bdb0682`)* — Charge crawl credits at **$1 / 1000 successful requests = 1000 micro-USD per successful crawl** (config-driven via `WEB_CRAWL_MICROS_PER_SUCCESS`, retunable with no code change), drawn from the existing credit wallet (`credit_micros_balance`), gated by a new `WEB_CRAWL_CREDIT_BILLING_ENABLED` flag (off for self-hosted). Billing surfaces (**reframed** by the Architecture correction + revamp — see log): the charge moves to the **capability executor** (revamp `04a`), so **every** caller meters uniformly — chat, **automation/recurring** (the confirmed gap: `run_agent_task` establishes no turn accumulator today, so `scrape_webpage` bills nothing under automations), REST, MCP, and external-cron — billing the **workspace owner** via `WebCrawlCreditService`. The interactive **chat turn accumulator** becomes an *optional presentation fold* (so chat still shows the crawl line on the turn bill), **not** the charging mechanism. The original **connector/pipeline-indexer** billing branch (`webcrawler_indexer`) is now **vestigial** (WEBCRAWLER connector retired, no pipelines) — kept dormant, not deleted. The `WebCrawlCreditService` (mirrors `EtlCreditService`'s gate → `check_credits` → `charge_credits`) is unchanged. No DB migration (uses the existing free-form `web_crawl` usage_type). - **`03c-crawl-billing.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `17bdb0682`)* — Charge crawl credits at **$1 / 1000 successful requests = 1000 micro-USD per successful crawl** (config-driven via `WEB_CRAWL_MICROS_PER_SUCCESS`, retunable with no code change), drawn from the existing credit wallet (`credit_micros_balance`), gated by a new `WEB_CRAWL_CREDIT_BILLING_ENABLED` flag (off for self-hosted). Billing surfaces (**reframed** by the Architecture pivot + revamp — see log): the charge moves to the **capability executor** (`04`), so **every** caller meters uniformly — chat, ongoing automation, REST, and MCP — billing the **workspace owner** via `WebCrawlCreditService`. The interactive **chat turn accumulator** becomes an *optional presentation fold* (so chat still shows the crawl line on the turn bill), **not** the charging mechanism. The original **connector/pipeline-indexer** billing branch (`webcrawler_indexer`) is now **vestigial** (WEBCRAWLER connector retired, no pipelines) — kept dormant, not deleted. The `WebCrawlCreditService` (mirrors `EtlCreditService`'s gate → `check_credits` → `charge_credits`) is unchanged. No DB migration (uses the existing free-form `web_crawl` usage_type).
- **`03e-stealth-hardening.md`** — The in-house "undetectable" layer on top of Scrapling's default patchright-Chromium stealth. **Geoip coherence** (match browser `locale`/`timezone_id` to the proxy's exit geo), **fingerprint flags** (`hide_canvas`/`block_webrtc`), **persistent per-domain profiles** (`user_data_dir`), **headed execution under Xvfb**, **real fonts** in the worker image, and **DIY behavioral humanization** via `page_action` (the Chromium engine has no built-in `humanize`). Adds a **block classifier** (label Cloudflare/DataDome/Kasada/captcha/empty from the response) + **per-domain strategy memory** (Redis, no migration) so the ladder learns the known-good tier per domain. Defines — but does **not** build — the **deferred paid-unblocker `FetchStrategy`** (ZenRows/ScrapFly/Bright Data) as the config-flagged escape hatch, with its own (later) billing. Honest ceiling: defeats Cloudflare + the moderate long tail, **not** top-tier behavioral fingerprinting (DataDome/Kasada/reCAPTCHA-Enterprise) — that's the deferred tier. - **`03e-stealth-hardening.md`** — The in-house "undetectable" layer on top of Scrapling's default patchright-Chromium stealth. **Geoip coherence** (match browser `locale`/`timezone_id` to the proxy's exit geo), **fingerprint flags** (`hide_canvas`/`block_webrtc`), **persistent per-domain profiles** (`user_data_dir`), **headed execution under Xvfb**, **real fonts** in the worker image, and **DIY behavioral humanization** via `page_action` (the Chromium engine has no built-in `humanize`). Adds a **block classifier** (label Cloudflare/DataDome/Kasada/captcha/empty from the response) + **per-domain strategy memory** (Redis, no migration) so the ladder learns the known-good tier per domain. Defines — but does **not** build — the **deferred paid-unblocker `FetchStrategy`** (ZenRows/ScrapFly/Bright Data) as the config-flagged escape hatch, with its own (later) billing. Honest ceiling: defeats Cloudflare + the moderate long tail, **not** top-tier behavioral fingerprinting (DataDome/Kasada/reCAPTCHA-Enterprise) — that's the deferred tier.
- **`03d-captcha-solving.md`** *(✅ IMPLEMENTED — `ci_mvp`; `captchatools` page_action + per-attempt `web_crawl_captcha` billing, off by default)* — Covers the captcha types Scrapling does **not** (reCAPTCHA v2/v3, hCaptcha, image) via `captchatools`, **opt-in + off by default**. `captchatools` is **itself** the provider registry (`new_harvester(solving_site=…)` across capmonster/2captcha/anticaptcha/capsolver/captchaai), so we do **not** rebuild a provider hierarchy — our layer is thin: config resolution + a StealthyFetcher `page_action` that detects the sitekey, harvests a token (egressing from the **same** proxy IP as the crawl), and injects it. Scrapling already handles Cloudflare Turnstile (`03a`), and `page_action` runs **after** `solve_cloudflare`, so the tiers compose. The **billing asymmetry** is now **resolved**: a **separate per-attempt** `web_crawl_captcha` unit (solvers charge per attempt regardless of crawl success), attached via a `WebCrawlCreditService` seam in `03c`. `ErrNoBalance` stops solving (no retry loop → avoids IP bans). Requires a paid solver account. - **`03d-captcha-solving.md`** *(✅ IMPLEMENTED — `ci_mvp`; `captchatools` page_action + per-attempt `web_crawl_captcha` billing, off by default)* — Covers the captcha types Scrapling does **not** (reCAPTCHA v2/v3, hCaptcha, image) via `captchatools`, **opt-in + off by default**. `captchatools` is **itself** the provider registry (`new_harvester(solving_site=…)` across capmonster/2captcha/anticaptcha/capsolver/captchaai), so we do **not** rebuild a provider hierarchy — our layer is thin: config resolution + a StealthyFetcher `page_action` that detects the sitekey, harvests a token (egressing from the **same** proxy IP as the crawl), and injects it. Scrapling already handles Cloudflare Turnstile (`03a`), and `page_action` runs **after** `solve_cloudflare`, so the tiers compose. The **billing asymmetry** is now **resolved**: a **separate per-attempt** `web_crawl_captcha` unit (solvers charge per attempt regardless of crawl success), attached via a `WebCrawlCreditService` seam in `03c`. `ErrNoBalance` stops solving (no retry loop → avoids IP bans). Requires a paid solver account.
- **`03f-undetectability-testing.md`** *(✅ IMPLEMENTED + first baseline — `ci_mvp`, MANUAL-only, no CI gating; lives under `app/proprietary/web_crawler/testbench/`)* — A **manual scorecard harness** that drives the real Scrapling tiers against the industry-standard detection + sandbox sites (modeled on CloakBrowser's `bin/cloaktest` suite) to **quantify the free-stack ceiling** over time. Two labeled axes: **Suite S (stealth/anti-bot)** — browser-tier (`bot.sannysoft`, `bot.incolumitas`, CreepJS, `deviceandbrowserinfo`, FingerprintJS demo, reCAPTCHA-v3 score, `fingerprint-scan`/Castle.js, `browserscan`, **`cloudflare-challenge`** exercising `solve_cloudflare`, **`iphey`** geoip-coherence), HTTP/TLS-tier JA3/JA4 parity (`tls.peet.ws`, **informational** not a gate), and proxy/leak checks (`httpbin/ip`, WebRTC/DNS); **Suite E (extraction correctness)** — toscrape/scrapethissite sandboxes for the HTTP vs JS (DynamicFetcher) tiers. **Every detection site is auto-graded from its real DOM verdict** (parsers written against captured dumps; `INFO` reserved for TLS/IP/manual rows). Reuses `03d`'s `page_action`+closure-cell mechanism. Adopts CloakBrowser's bars as **aspirational** (sannysoft 0 fails, CreepJS ≤30%, reCAPTCHA ≥0.7) while recording **our actual numbers as the baseline** (the whole `results/` tree is **gitignored**, run-local). **First baseline (2026-06-30, headless, rotating residential, captcha OFF): Suite S 6 PASS / 4 FAIL** — PASS sannysoft, deviceandbrowserinfo, reCAPTCHA-v3 (0.9), BrowserScan, fingerprint-scan (35/100), cloudflare-challenge; FAIL CreepJS (`hasHeadlessWorkerUA` worker-UA leak → `03e` Slice-B candidate), FingerprintJS Pro (commercial ceiling), iphey ("Unreliable" → expected geoip-coherence fix), incolumitas (legacy `fpscanner WEBDRIVER` only). The scorecard is the documented **trigger** for flipping `03e`'s deferred paid-unblocker tier. - **`03f-undetectability-testing.md`** *(✅ IMPLEMENTED + first baseline — `ci_mvp`, MANUAL-only, no CI gating; lives under `app/proprietary/web_crawler/testbench/`)* — A **manual scorecard harness** that drives the real Scrapling tiers against the industry-standard detection + sandbox sites (modeled on CloakBrowser's `bin/cloaktest` suite) to **quantify the free-stack ceiling** over time. Two labeled axes: **Suite S (stealth/anti-bot)** — browser-tier (`bot.sannysoft`, `bot.incolumitas`, CreepJS, `deviceandbrowserinfo`, FingerprintJS demo, reCAPTCHA-v3 score, `fingerprint-scan`/Castle.js, `browserscan`, **`cloudflare-challenge`** exercising `solve_cloudflare`, **`iphey`** geoip-coherence), HTTP/TLS-tier JA3/JA4 parity (`tls.peet.ws`, **informational** not a gate), and proxy/leak checks (`httpbin/ip`, WebRTC/DNS); **Suite E (extraction correctness)** — toscrape/scrapethissite sandboxes for the HTTP vs JS (DynamicFetcher) tiers. **Every detection site is auto-graded from its real DOM verdict** (parsers written against captured dumps; `INFO` reserved for TLS/IP/manual rows). Reuses `03d`'s `page_action`+closure-cell mechanism. Adopts CloakBrowser's bars as **aspirational** (sannysoft 0 fails, CreepJS ≤30%, reCAPTCHA ≥0.7) while recording **our actual numbers as the baseline** (the whole `results/` tree is **gitignored**, run-local). **First baseline (2026-06-30, headless, rotating residential, captcha OFF): Suite S 6 PASS / 4 FAIL** — PASS sannysoft, deviceandbrowserinfo, reCAPTCHA-v3 (0.9), BrowserScan, fingerprint-scan (35/100), cloudflare-challenge; FAIL CreepJS (`hasHeadlessWorkerUA` worker-UA leak → `03e` Slice-B candidate), FingerprintJS Pro (commercial ceiling), iphey ("Unreliable" → expected geoip-coherence fix), incolumitas (legacy `fpscanner WEBDRIVER` only). The scorecard is the documented **trigger** for flipping `03e`'s deferred paid-unblocker tier.
### Phase 4 — Capabilities & Access (backend) [`subplans: revamp/04a–04b`] ### Phase 4 — Capabilities (backend) [`subplan: 04-capabilities.md`]
> **Canonical subplans:** [`revamp phases 4-7/04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md) + [`04b-access.md`](revamp%20phases%204-7/04b-access.md). Together they ship **Product A** (stateless utility) — revenue day one. **Build `04a` first** — every later phase calls the registry. > **Canonical subplan:** [`04-capabilities.md`](04-capabilities.md). **Build first** — every later phase calls the registry.
- **`revamp/04a-capabilities.md`** — Turn Acquisition (crawler + search + a future Maps actor) into a **small set of typed, callable verbs** over a single **capability registry**. MVP verbs = **only `web.scrape(urls[])`** (array-friendly, inline-or-job) **+ `web.discover(query, top_k)`** (SearXNG/Linkup/Baidu). **Platform-specific scrapers** (Maps/LinkedIn/Amazon/… — *examples, none committed*) are a family of **individual endpoints** added later: each is just another verb → agent tool + dev API-key REST endpoint, so `04a` only ships **contract stubs** for them, not executors. Uniform result **envelope** (`completed | pending + job_id`); slow calls use a **thin job record** + existing Celery (not a resurrected `pipeline_runs`). Each verb declares a **`billing_unit`**; charging is **delegated to the billing service** (03c is the first provider) and happens **at the executor** so every door meters uniformly. New Apache-2 package `app/capabilities/`; Maps extractor stays proprietary. - Turn Acquisition (crawler + search + Maps actor) into a **small set of typed, callable verbs** over a single **capability registry** — `web.scrape(urls[])`, `web.discover(query, top_k)`, `maps.search/place/reviews`. Each verb is a **direct async call that returns cleaned, AI-ready data** (no job store, no envelope, no polling). `maps.*` returns typed structured objects, not markdown. Each declares a **`billing_unit`**; charging is delegated to the billing service (03c first provider) at the executor so every door meters uniformly. New Apache-2 package `app/capabilities/`; Maps extractor stays proprietary.
- **`revamp/04b-access.md`** — Expose the registry through **three generated doors** (order: chat → REST+API-key → MCP), each the same thin adapter (`parse → validate → authn/authz → 03c meter-gate → same executor → serialize → envelope`). **Natural language is the only human-facing surface** (users never name verbs); an **intent router** classifies one-shot (Product A) vs standing-concern (Product B). Slow chat verbs reuse the shipped `deliverable_wait` poll-until-terminal + live-card path. This is where the **BYO-`MCP_CONNECTOR` routing-gap fix** (from the old `04a`) lands (consume-user-MCP).
**Locked model (MVP):** **Locked model (MVP):**
- **Capabilities replace connectors as data sources.** `web.scrape` / `web.discover` (MVP) — and later per-platform scraper verbs — are *call → data → bill* verbs — no `SearchSourceConnector` row, no KB write, no schedule. - **Capabilities replace connectors as data sources.** `web.*` / `maps.*` are *call → cleaned data → bill* verbs — no `SearchSourceConnector` row, no KB write, no schedule.
- **Surviving connectors = file/KB-input + BYO MCP.** File/KB-input (Google Drive native+Composio, OneDrive, Dropbox, uploads) put the user's personal files into the **input-only** KB. BYO `MCP_CONNECTOR` (act in chat) is the only functional Type-2; its routing-gap fix lands in `revamp/04b`. **`WEBCRAWLER_CONNECTOR` RETIRED.** The old connector two-type taxonomy (`04a-connector-category`) is **demoted to backward-compat hygiene**. - **Surviving connectors = file/KB-input + BYO MCP.** File/KB-input (Google Drive native+Composio, OneDrive, Dropbox, uploads) put the user's personal files into the **input-only** KB. BYO `MCP_CONNECTOR` (act in chat) is the only functional Type-2; its routing-gap fix lands in `05`. **`WEBCRAWLER_CONNECTOR` RETIRED.** The old connector two-type taxonomy (`04a-connector-category`) is **demoted to backward-compat hygiene**.
- **Deprecated (`MIGRATING`, off for MVP):** every branded native — indexers (Notion, GitHub, Confluence, BookStack, Elasticsearch) + act-only (Slack, Teams, Linear, Jira, ClickUp, Airtable, Discord, Gmail, Google Calendar, Luma, + Composio Gmail/Calendar). Existing rows + already-indexed KB docs stay **searchable**; new-create/`/index`/periodic/subagents off. Real MCP migration is post-MVP. - **Deprecated (`MIGRATING`, off for MVP):** every branded native — indexers (Notion, GitHub, Confluence, BookStack, Elasticsearch) + act-only (Slack, Teams, Linear, Jira, ClickUp, Airtable, Discord, Gmail, Google Calendar, Luma, + Composio Gmail/Calendar). Existing rows + already-indexed KB docs stay **searchable**; new-create/`/index`/periodic/subagents off. Real MCP migration is post-MVP.
- **Disabled (`DISABLED`):** Obsidian (plugin push) and Circleback (meeting webhook). - **Disabled (`DISABLED`):** Obsidian (plugin push) and Circleback (meeting webhook).
- Web search: **drop all 5 search-API `connector_type` values** (`SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API`); survivors (SearXNG/Linkup/Baidu) become the `web.discover` provider set keyed by **platform env** (Linkup/Baidu keys move off per-connector `config`). Destructive migration deletes the 5 types' rows. (Absorbs the old `04b-source-discovery`.) - Web search: **drop all 5 search-API `connector_type` values** (`SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API`); survivors (SearXNG/Linkup/Baidu) become the `web.discover` provider set keyed by **platform env** (Linkup/Baidu keys move off per-connector `config`). Destructive migration deletes the 5 types' rows. (Absorbs the old `04b-source-discovery`.)
- The deferred "coming soon" platforms (LinkedIn, Amazon, Google, Instagram, Zillow/Redfin, Walmart, eBay, Crunchbase, TikTok, Indeed/Glassdoor) re-slot as **Phase 8 native verbs / API endpoints**, not Type-1 KB connectors. - The **Google Maps actor is net-new**, proprietary, built as a separate effort; `maps.*` are contracts against it. Other "coming soon" platforms (LinkedIn, Amazon, Instagram, Zillow/Redfin, Walmart, eBay, Crunchbase, TikTok, Indeed/Glassdoor) re-slot as later namespaces, not Type-1 KB connectors.
- Frontend connector/capability UI restructure is DEFERRED (frontend umbrella). - Frontend connector/capability UI restructure is DEFERRED (frontend umbrella).
### Phase 5 — Intelligence & Timeline (backend) [`subplans: revamp/05a–05b`] ### Phase 5 — Access (backend) [`subplan: 05-access.md`]
> **Canonical subplans:** [`revamp phases 4-7/05a-timeline.md`](revamp%20phases%204-7/05a-timeline.md) + [`05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md). Ships the **Product B engine**. **Depends on `04a`** (the verbs the loop calls). **Build `05a` (tables) before `05b` (the loop that writes them).** This *replaces* the old Pipelines Phases 5–7 — the durable CI state, and **the moat**. > **Canonical subplan:** [`05-access.md`](05-access.md). **Build after `04`.** Together `04 + 05` ship the scraper-API product — revenue day one.
- **`revamp/05a-timeline.md`** — The **3-store delta Timeline**: `tracked_entities` (stable identity, written once), `entity_current_state` (latest values + `content_hash` + `last_checked_at`, overwritten each run), `entity_changes` (append-only material deltas: `{field:{from,to}}`, `materiality`, `decided_by`, `source_ref`). **Store deltas, no change → no row** (storage ∝ rate of change). Explicitly **not the KB, not `automation_runs`, not a resurrected `pipeline_runs`.** New CI-owned Apache-2 tables + Alembic migration; write/read API for the `05b` loop and the read side. **This is where "diffable run history" actually lives** (superseding the old "structured `automation_runs.output`" idea). - Expose the registry through **three generated doors** — **REST + API key** (public day one), an **MCP server** (fast-follow), and **chat tools** — each the same thin adapter (`parse → validate → authn/authz → 03c meter-gate → same executor → serialize → cleaned data`). The executor returns directly; there is **no envelope, no `deliverable_wait` polling**. Async results (e.g. the `06` ongoing mode) reach the client via **write-then-sync / stream** (existing SSE or a Zero-published table).
- **`revamp/05b-intelligence.md`** — The `Tracker` primitive + the **hot loop** `refresh(tracker)`: crawl (call the bound verb) → **content-hash pre-check** (identical → stamp `last_checked_at`, stop, no LLM) → **fill** raw data to the locked `field_schema` (structured output; extras → `notable_signals`) → **diff** (code) → **judge — the materiality split** (deterministic numeric/clear rules in code for free; agent only on ambiguous) → **append** a change + overwrite current state, else only `last_checked_at`. Setup = an **agent-designed, human-locked, versioned schema** flow (sample-fetch → draft `definition` → review/lock). **No new run table:** recurring audit/idempotency ride `AutomationRun`, manual rides the chat job record; **billing idempotency is per-capability-call** + the hash pre-check. Optional **CI context folder** (user files uploaded in a CI chat go into the KB as normal, routed to a folder, and can feed the judge). New Apache-2 package `app/intelligence/`. - **Natural language is the human surface** (raw verbs only on REST/MCP); the chat door classifies **one-shot** vs **keep-watching** and hands keep-watching to `06`. Two MCP directions kept distinct: **serve** our verbs as an MCP server (new door) vs **consume** the user's BYO `MCP_CONNECTOR` (the old-`04a` routing-gap fix lands here).
### Phase 6 — Triggers (backend) [`subplan: revamp/06-triggers.md`] ### Phase 6 — Ongoing-Automation (backend) [`subplan: 06-ongoing-automation.md`]
> **Canonical subplan:** [`revamp phases 4-7/06-triggers.md`](revamp%20phases%204-7/06-triggers.md) — the thinnest phase. **Depends on `05b`** (`refresh(tracker)`). Decides **when** a Tracker refreshes; Intelligence never knows which trigger fired. > **Canonical subplan:** [`06-ongoing-automation.md`](06-ongoing-automation.md) — **⚠️ design deferred.** The periodic mechanism is designed separately, after `04`/`05`/`07`. **Depends on `04`** (the verbs it re-invokes) and `05` (the chat surface + delivery channel).
- Adapters: **manual** (user "refresh now" via chat tool / REST), **agent** (the agent calls `refresh` as a tool), **external cron** (`POST /v1/trackers/{id}/refresh`, zero infra on us), and the **CI automation action** — recurrence reusing the **existing automations**: its hardened **schedule selector** (no new scheduler) + **`AutomationRun`** (no new run table). **Alert delivery is separate** — persist deltas to the Timeline, then emit an in-app notification via `app/notifications/` (automations have no delivery path; `run.output` is unwritten). Guard concurrency with a **per-Tracker lock** (the automation PENDING-gate is non-atomic). A single `refresh_tracker` action, **not** a new automation shape. - Support **"keep watching"** with no stateful storage: a **persistent, ongoing chat** where the agent periodically re-invokes scraper verbs and drops results into the session; the agent reports "what's new" by reading the chat history. Open questions (resolved together): periodicity driver, delivery channel (SSE vs Zero-published table), context-window limit, loop owner, stop/cost controls.
- **Decoupling preserved:** CI core (`refresh` + Timeline) runs with **zero** automations dependency (manual/agent/external-cron); automations is the **optional** in-app recurrence+alert adapter. This is the correct home for the old "wire a run-now trigger" item.
### Phase 7 — Orchestration (backend) [`subplan: revamp/07-orchestration.md`] ### Phase 7 — Orchestration (backend) [`subplan: 07-orchestration.md`]
> **Canonical subplan:** [`revamp phases 4-7/07-orchestration.md`](revamp%20phases%204-7/07-orchestration.md) — the human-facing brain. **Build last** (atop `04`–`06`). We plug into the shipped multi-agent runtime; we don't rebuild it. > **Canonical subplan:** [`07-orchestration.md`](07-orchestration.md) — the human-facing brain. **Build last** (atop `04`/`05`, and `06` for the ongoing mode). We plug into the shipped multi-agent runtime; we don't rebuild it.
- Ship the net-new builtin **`intelligence_agent`** subagent (peer to `research`/`deliverables`): **intent routing** (one-shot vs standing concern, one clarifying question when ambiguous), **verb composition** (`web.discover → web.scrape`, `maps.search → maps.place → maps.reviews`), **Tracker crafting** (the `05b` schema-design flow in chat), and **decision-grounded answering** (read the Timeline via `query_timeline`, not chat history). - Ship the net-new builtin **`intelligence_agent`** subagent (peer to `research`/`deliverables`): **intent routing** (one-shot vs keep-watching, one clarifying question when ambiguous), **verb composition** (`web.discover → web.scrape`, `maps.search → maps.place → maps.reviews`), and **"what changed" from the chat history** (re-invoke a verb, compare against prior tool outputs already in context — no timeline, no diff store).
- Toolset: registry-backed capability verbs (shared with `research`) + Tracker/Timeline tools (`craft_tracker`/`lock_tracker`/`refresh_tracker`/`query_timeline`/`list_trackers`); slow verbs reuse `deliverable_wait`. Tools follow the shipped `scrape_webpage` shape (executor + door + 03c billing). - Toolset: registry-backed capability verbs (shared with `research`) + a deferred `start_watch` handoff to `06`. Tools follow the shipped `scrape_webpage` shape (executor + door + 03c billing), **direct-return** (no `deliverable_wait`).
### (Future) Platform API-key + public MCP ### (Future) Platform API-key + public MCP
@ -177,7 +185,7 @@ Platform **API-key** access to the capability verbs + a public **MCP server** ex
### Phase 8 — Platform actors (FUTURE — post-MVP, public data only) [`subplan: TBD`] ### Phase 8 — Platform actors (FUTURE — post-MVP, public data only) [`subplan: TBD`]
NOT planned in this umbrella; recorded so the Phase-3 architecture stays aimed at it. Once the hardened fetch core (Phases 3a–3e) is solid, **platform scrapers** — a family of **individual scraping endpoints** (Google Maps/Local, LinkedIn public profiles/companies, Amazon products, etc. — see "Platform connector research list"; **all examples, none committed**) — layer **on top** of it. **Each is simply another capability verb** (`revamp/04a`), so registering one automatically yields (a) an **agent tool** and (b) a **dev-callable REST endpoint behind the platform API key** — same executor, same billing, no new machinery. Each is a per-platform structured extractor built in-house "Apify-style" on the `03a` crawler core (proxies + `03e` hardening + `03d` captcha), callable by the chat agent, by automations, and by the future platform API key. Billed per call via `WebCrawlCreditService`. NOT planned in this umbrella; recorded so the Phase-3 architecture stays aimed at it. Google Maps is the **committed MVP actor** (a `maps.*` namespace, Phase 4). Beyond it, more **platform scrapers** — a family of **individual scraping endpoints** (LinkedIn public profiles/companies, Amazon products, etc. — see "Platform connector research list"; **examples, none committed**) — layer **on top** of the hardened fetch core once it's solid. **Each is simply another capability verb** (`04`), so registering one automatically yields (a) an **agent tool** and (b) a **dev-callable REST endpoint behind the platform API key** — same executor, same billing, no new machinery. Each is a per-platform structured extractor built in-house "Apify-style" on the `03a` crawler core (proxies + `03e` hardening + `03d` captcha). Billed per call via `WebCrawlCreditService`.
- **Public data only** at first — discovery/extraction of publicly visible pages. **Logged-in/account-based bypass is explicitly deferred** beyond Phase 8's first cut; it needs sticky/static proxies + credential management (`03b` static-proxy hand-off) and is the higher-risk, later workstream. - **Public data only** at first — discovery/extraction of publicly visible pages. **Logged-in/account-based bypass is explicitly deferred** beyond Phase 8's first cut; it needs sticky/static proxies + credential management (`03b` static-proxy hand-off) and is the higher-risk, later workstream.
- The **deferred paid-unblocker tier** (`03e §8`) is the fallback for any platform whose anti-bot exceeds the in-house ceiling. - The **deferred paid-unblocker tier** (`03e §8`) is the fallback for any platform whose anti-bot exceeds the in-house ceiling.
@ -189,7 +197,7 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta
- Frontend rename + i18n: route segment `[search_space_id]` -> `[workspace_id]`, `search-space-settings/` -> `workspace-settings/`, TS types, api services, Jotai atoms, components, cache keys, and "Workspace" copy across 5 locales (`messages/{en,zh,es,pt,hi}.json`), plus frontend Zero schema rename. - Frontend rename + i18n: route segment `[search_space_id]` -> `[workspace_id]`, `search-space-settings/` -> `workspace-settings/`, TS types, api services, Jotai atoms, components, cache keys, and "Workspace" copy across 5 locales (`messages/{en,zh,es,pt,hi}.json`), plus frontend Zero schema rename.
- Satellite/client apps + docs rename: `surfsense_desktop`, `surfsense_obsidian`, `surfsense_browser_extension`, `surfsense_evals`, README/docs. - Satellite/client apps + docs rename: `surfsense_desktop`, `surfsense_obsidian`, `surfsense_browser_extension`, `surfsense_evals`, README/docs.
- Connector two-type UI: restructure `connector-popup` and `connector-constants.ts` into the two labeled types. - Connector two-type UI: restructure `connector-popup` and `connector-constants.ts` into the two labeled types.
- CI + automations + positioning UI (**replaces the superseded "Pipelines UI"**): the CI experience (Tracker crafting/review-&-lock, Timeline/change views, "refresh now"), automation create/list/run-history, `web.discover` source-suggestion UX, input-only KB framing (uploads + file connectors), MCP connector management, "coming soon" platform cards, "NotebookLM for Competitive Intelligence" copy. - Scraper-API + chat + positioning UI (**replaces the superseded "Pipelines UI"**): the developer surface for the scraper endpoints (API keys, usage/billing), the chat experience (one-shot answers + "keep watching" ongoing chats), `web.discover` source-suggestion UX, input-only KB framing (uploads + file connectors), MCP connector management, "coming soon" platform cards.
## Open items to confirm during subplanning ## Open items to confirm during subplanning
@ -197,11 +205,12 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta
- ~~Whether existing connector periodic-indexing config is migrated into Pipelines or coexists during MVP.~~ **MOOT** (Architecture correction — no pipelines). Connector periodic indexing stays as-is for file sources; there is no pipeline scheduler to coexist with. - ~~Whether existing connector periodic-indexing config is migrated into Pipelines or coexists during MVP.~~ **MOOT** (Architecture correction — no pipelines). Connector periodic indexing stays as-is for file sources; there is no pipeline scheduler to coexist with.
- ~~Chat agent run-history access: tool vs middleware injection vs both (default: tool).~~ **MOOT** (Architecture correction — no pipeline runs). The agent's recurring work is automations, whose history is already persisted + queryable (`automation_runs`). - ~~Chat agent run-history access: tool vs middleware injection vs both (default: tool).~~ **MOOT** (Architecture correction — no pipeline runs). The agent's recurring work is automations, whose history is already persisted + queryable (`automation_runs`).
- ~~Type-2 MCP migration depth~~ RESOLVED (Phase 4): branded natives are tagged `MIGRATING` and turned OFF for MVP (not re-pointed to MCP yet); only the generic `MCP_CONNECTOR` is a functional Type-2. Real MCP re-pointing is post-MVP. - ~~Type-2 MCP migration depth~~ RESOLVED (Phase 4): branded natives are tagged `MIGRATING` and turned OFF for MVP (not re-pointed to MCP yet); only the generic `MCP_CONNECTOR` is a functional Type-2. Real MCP re-pointing is post-MVP.
- **Revamp cross-phase forks (from `revamp/00-overview.md`, decide during subplanning):** (1) schema **review-&-lock** UX pre-frontend — pure-chat confirmation for MVP? (`05b`); (2) Timeline ORM home — `app/db.py` vs a dedicated `app/timeline/` (`05a`); (3) recurrence = CI **action** on automations (default) vs a thin CI automation **shape** (fallback) (`06`); (4) is `web.discover` metered or free (`04a`); (5) MCP-serve auth depth — OAuth 2.1 vs bearer (`04b`). - **Revamp open items (decide during subplanning):** (1) the **Ongoing-Automation mechanism** (`06`, deferred) — periodicity driver, delivery channel (SSE vs Zero-published table), context-window limit, stop/cost controls; (2) how large `web.scrape` / `maps.reviews` responses are bounded/streamed (`04`); (3) is `web.discover` metered or free (`04`); (4) MCP-serve auth depth — OAuth 2.1 vs bearer (`05`); (5) public REST rate-limiting / abuse posture (`05`).
## Resolved decisions log ## Resolved decisions log
- **REVAMP ADOPTED as canonical (2026-06-30) — Phases 4–7 re-cast; `revamp phases 4-7/` is the source of truth.** A principal-engineer review found the flat `04a–07` files and my earlier "Phase 5′" framing had been overtaken by a more complete engineer draft in [`revamp phases 4-7/`](revamp%20phases%204-7/00-overview.md). We adopt the revamp: **Phase 4 = Capabilities & Access** (typed verbs + generated doors; old `04a-connector-category` demoted to hygiene, old `04b-source-discovery` absorbed as `web.discover`), **Phase 5 = Intelligence & Timeline** (the `Tracker` + 3-store delta Timeline — the moat), **Phase 6 = Triggers** (reuse automations via a CI action), **Phase 7 = Orchestration** (`intelligence_agent`). The three architecture-correction decisions land in their real homes and one is revised: (1) **bill at the capability executor** — a code-verified gap review confirmed `run_agent_task` (automation `agent_task`) sets up **no turn accumulator**, so today `scrape_webpage` bills nothing under automations; billing moves to the executor so chat/automation/REST/MCP/cron meter uniformly (chat turn accumulator becomes optional presentation). (2) **"diffable run history" = the Timeline delta store** (`revamp/05a`), **superseding** the earlier "structured `automation_runs.output`" decision. (3) **"run now" = a Trigger adapter** (`revamp/06`). The flat `04a/04b/05/06/07` files are retained with redirect banners; **`revamp/` filenames win** on any number collision. Build order: `04a → 04b → 05a → 05b → 06 → 07`. - **PIVOT (2026-07-01) — scraper-APIs-first, stateless, chat-native; Tracker/Timeline/Triggers dropped.** The product is the **scraper capabilities** (`web.*` + `maps.*` typed verbs → cleaned, AI-ready data), exposed as **REST + API key / MCP / chat** generated from one registry — premium, OSS, self-hostable, the revenue driver. **Stateless:** no Timeline (3-store delta), no `Tracker`, no diff/materiality engine, no stored `entity_changes`; memory = the **chat history** (the agent reasons over prior tool outputs for "what changed"). **Direct calls:** capabilities return their result directly — no job store, no `completed | pending + job_id` envelope, no `deliverable_wait` polling; async delivery is **write-then-sync / stream**. **Automation = a persistent ongoing chat** re-invoking verbs (replaces `Triggers`; mechanism **design-deferred**). KB stays **input-only**. Phase re-cast: **04 Capabilities · 05 Access · 06 Ongoing-Automation (deferred) · 07 Orchestration** (`intelligence_agent`). Subplans `05a-timeline`/`05b-intelligence`/`06-triggers` **removed** (git history); `04a`/`04b` folded into `04`/`05`. This **supersedes** the 2026-06-30 revamp's Tracker/Timeline/Triggers decisions below.
- **REVAMP ADOPTED as canonical (2026-06-30) — Phases 4–7 re-cast; `revamp phases 4-7/` is the source of truth.** A principal-engineer review found the flat `04a–07` files and my earlier "Phase 5′" framing had been overtaken by a more complete engineer draft (then in a `revamp phases 4-7/` folder, since flattened into `04`–`07`). We adopt the revamp: **Phase 4 = Capabilities & Access** (typed verbs + generated doors; old `04a-connector-category` demoted to hygiene, old `04b-source-discovery` absorbed as `web.discover`), **Phase 5 = Intelligence & Timeline** (the `Tracker` + 3-store delta Timeline — the moat), **Phase 6 = Triggers** (reuse automations via a CI action), **Phase 7 = Orchestration** (`intelligence_agent`). The three architecture-correction decisions land in their real homes and one is revised: (1) **bill at the capability executor** — a code-verified gap review confirmed `run_agent_task` (automation `agent_task`) sets up **no turn accumulator**, so today `scrape_webpage` bills nothing under automations; billing moves to the executor so chat/automation/REST/MCP/cron meter uniformly (chat turn accumulator becomes optional presentation). (2) **"diffable run history" = the Timeline delta store** (`revamp/05a`), **superseding** the earlier "structured `automation_runs.output`" decision. (3) **"run now" = a Trigger adapter** (`revamp/06`). The flat `04a/04b/05/06/07` files are retained with redirect banners; **`revamp/` filenames win** on any number collision. Build order: `04a → 04b → 05a → 05b → 06 → 07`.
- **ARCHITECTURE CORRECTION (2026-06-30) — Pipelines dropped; automations + input-only KB adopted.** The "sync into KB → operate over it" paradigm was judged wrong for CI. Verified against `references/opencode` (live tool-fetched context + persisted sessions, no scraped corpus; `tool/websearch.ts` live-crawls at query time). Decisions: (1) **KB is input-only** — user's personal files/context (uploads + Drive/Dropbox/OneDrive); nothing scrapes into it. (2) **Web search + WebURL crawler are platform-native tools** (already `web_search`/`scrape_webpage`), later a developer platform-API-key surface — NOT connectors, NOT pipelines. (3) **Connectors = MCP tools + file/KB-input connectors**; all branded natives → MCP. (4) **Automations are the scheduling + run-history substrate** (an `agent_task` run already invokes the full chat agent with the crawler/search tools on cron/event and persists `automation_runs`). (5) **`WEBCRAWLER_CONNECTOR` RETIRED.** Consequences: **Phases 5/6/7 SUPERSEDED** (docs kept with banners); **Phase 4a reframed** (drop `is_pipeline_eligible`; Type-1 = file/KB-input only; WEBCRAWLER retired); **Phase 4b stands**; new **Phase 5′** automation-enhancement workstream (bill automation-run crawls; structured `run.output`; wire "run now"); **Phase 8** actors re-slot as native tools/API. `03c` connector-indexer billing branch becomes **vestigial** (billing lives on chat-turn / automation-run / API paths). - **ARCHITECTURE CORRECTION (2026-06-30) — Pipelines dropped; automations + input-only KB adopted.** The "sync into KB → operate over it" paradigm was judged wrong for CI. Verified against `references/opencode` (live tool-fetched context + persisted sessions, no scraped corpus; `tool/websearch.ts` live-crawls at query time). Decisions: (1) **KB is input-only** — user's personal files/context (uploads + Drive/Dropbox/OneDrive); nothing scrapes into it. (2) **Web search + WebURL crawler are platform-native tools** (already `web_search`/`scrape_webpage`), later a developer platform-API-key surface — NOT connectors, NOT pipelines. (3) **Connectors = MCP tools + file/KB-input connectors**; all branded natives → MCP. (4) **Automations are the scheduling + run-history substrate** (an `agent_task` run already invokes the full chat agent with the crawler/search tools on cron/event and persists `automation_runs`). (5) **`WEBCRAWLER_CONNECTOR` RETIRED.** Consequences: **Phases 5/6/7 SUPERSEDED** (docs kept with banners); **Phase 4a reframed** (drop `is_pipeline_eligible`; Type-1 = file/KB-input only; WEBCRAWLER retired); **Phase 4b stands**; new **Phase 5′** automation-enhancement workstream (bill automation-run crawls; structured `run.output`; wire "run now"); **Phase 8** actors re-slot as native tools/API. `03c` connector-indexer billing branch becomes **vestigial** (billing lives on chat-turn / automation-run / API paths).
- ~~Web search APIs (SearXNG/Linkup/Baidu): repurposed as source-discovery helper for the WebURL Crawler (suggest URLs for pipelines)~~ — still repurposed as a source-discovery helper (04b), but "for pipelines" → "for the user / the native crawler tool" (pipelines dropped); not a standalone connector type. - ~~Web search APIs (SearXNG/Linkup/Baidu): repurposed as source-discovery helper for the WebURL Crawler (suggest URLs for pipelines)~~ — still repurposed as a source-discovery helper (04b), but "for pipelines" → "for the user / the native crawler tool" (pipelines dropped); not a standalone connector type.
- Tavily and Serper: REMOVED from the search infra. They are dropped as search providers entirely (not repurposed). Phase 4's source-discovery endpoint must build only on the remaining providers (SearXNG, Linkup, Baidu). - Tavily and Serper: REMOVED from the search infra. They are dropped as search providers entirely (not repurposed). Phase 4's source-discovery endpoint must build only on the remaining providers (SearXNG, Linkup, Baidu).
@ -239,17 +248,10 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta
| 3 | `03e-stealth-hardening.md` | **Slice A IMPLEMENTED** (`ci_mvp`) — stealth kwargs builder + geoip coherence (**proprietary** `app/proprietary/web_crawler/stealth.py`) + additive block classifier (Apache-2 `app/utils/crawl/`; `CrawlOutcome.block_type`, incl. static-tier 4xx) + Xvfb/fonts in image; Slices B/C deferred (WebGL spoof, humanization, persistent profiles, strategy memory, paid-unblocker) | | 3 | `03e-stealth-hardening.md` | **Slice A IMPLEMENTED** (`ci_mvp`) — stealth kwargs builder + geoip coherence (**proprietary** `app/proprietary/web_crawler/stealth.py`) + additive block classifier (Apache-2 `app/utils/crawl/`; `CrawlOutcome.block_type`, incl. static-tier 4xx) + Xvfb/fonts in image; Slices B/C deferred (WebGL spoof, humanization, persistent profiles, strategy memory, paid-unblocker) |
| 3 | `03d-captcha-solving.md` | **IMPLEMENTED** (`ci_mvp`) — `captchatools` page_action (proprietary) + Apache-2 config + per-attempt `web_crawl_captcha` billing; off by default | | 3 | `03d-captcha-solving.md` | **IMPLEMENTED** (`ci_mvp`) — `captchatools` page_action (proprietary) + Apache-2 config + per-attempt `web_crawl_captcha` billing; off by default |
| 3 | `03f-undetectability-testing.md` | **IMPLEMENTED** (`ci_mvp`) — manual scorecard under the **proprietary boundary** at `app/proprietary/web_crawler/testbench/` (`python -m app.proprietary.web_crawler.testbench`); Suite S (stealth, shipped builder) + Suite E (extraction via real `crawl_url`) + scorecard JSON/MD baseline diff | | 3 | `03f-undetectability-testing.md` | **IMPLEMENTED** (`ci_mvp`) — manual scorecard under the **proprietary boundary** at `app/proprietary/web_crawler/testbench/` (`python -m app.proprietary.web_crawler.testbench`); Suite S (stealth, shipped builder) + Suite E (extraction via real `crawl_url`) + scorecard JSON/MD baseline diff |
| 4 | `revamp phases 4-7/04a-capabilities.md` | **CANONICAL** · drafted — capability registry + MVP verbs `web.scrape`/`web.discover` (per-platform scrapers = later drop-ins); bill at executor. **← build next** | | — | `00b-pipeline-diagrams.md` | end-to-end flow diagrams (companion to this umbrella) |
| 4 | `revamp phases 4-7/04b-access.md` | **CANONICAL** · drafted — generated chat/REST/MCP doors + intent router + BYO-MCP routing fix | | 4 | `04-capabilities.md` | **CANONICAL** · drafted — capability registry + verbs `web.scrape`/`web.discover`/`maps.*` → cleaned data; bill at executor. **← build next** |
| 5 | `revamp phases 4-7/05a-timeline.md` | **CANONICAL** · drafted — 3-store delta Timeline (the moat) | | 5 | `05-access.md` | **CANONICAL** · drafted — generated REST/MCP/chat doors + intent router + BYO-MCP routing fix |
| 5 | `revamp phases 4-7/05b-intelligence.md` | **CANONICAL** · drafted — `Tracker` + refresh hot loop + materiality split | | 6 | `06-ongoing-automation.md` | **CANONICAL** · **design deferred** — chat-native "keep watching" (periodic re-invocation) |
| 6 | `revamp phases 4-7/06-triggers.md` | **CANONICAL** · drafted — refresh clock; recurrence = CI action on automations | | 7 | `07-orchestration.md` | **CANONICAL** · drafted — `intelligence_agent` subagent; "what changed" from chat history |
| 7 | `revamp phases 4-7/07-orchestration.md` | **CANONICAL** · drafted — `intelligence_agent` subagent |
| — | `revamp phases 4-7/00-overview.md` · `00b-pipeline-diagrams.md` | revamp map + reconciliation + end-to-end flow diagrams |
| 4 | `04a-connector-category.md` *(flat)* | ↪ **REDIRECT** — demoted to backward-compat hygiene; superseded by `revamp/04a`+`04b` |
| 4 | `04b-source-discovery.md` *(flat)* | ↪ **REDIRECT** — absorbed as the `web.discover` verb in `revamp/04a` |
| 5 | `05-pipelines-model.md` *(flat)* | ❌ **SUPERSEDED** — replaced by `revamp/05a`+`05b` (Tracker/Timeline) |
| 6 | `06-pipelines-exec.md` *(flat)* | ❌ **SUPERSEDED** — replaced by `revamp/05b` (hot loop) + `revamp/06` (triggers) |
| 7 | `07-upload-pipeline-kb.md` *(flat)* | ❌ **SUPERSEDED** — uploads populate the input-only KB (no pipeline wrapper) |
Frontend & client subplans will be added under a separate umbrella later (see "Deferred — Frontend & client phases"). Frontend & client subplans will be added under a separate umbrella later (see "Deferred — Frontend & client phases").

110
plans/backend/00b-pipeline-diagrams.md Normal file
View file

@ -0,0 +1,110 @@
# Pipeline diagrams — end-to-end (scraper-APIs-first, stateless)
> Visual companion to `00-umbrella-plan.md`.
> Phase refs: `04` Capabilities · `05` Access · `06` Ongoing-Automation (design deferred) · `07` Orchestration.
## 1. The shape — scraper APIs + a chat agent (no state)
```mermaid
flowchart LR
subgraph FIXED["FIXED · Phases 1-3 (shipped)"]
ACQ["Acquisition<br/>crawler · proxy · stealth · captcha"]
MET["Metering · 03c"]
ID["Identity / API keys"]
end
subgraph SCOPE["OUR SCOPE · Phase 04 -> 07"]
CAP["04 Capabilities<br/>web.* · maps.*<br/>cleaned, AI-ready output"]
ACC["05 Access<br/>REST · MCP · chat doors"]
ORC["07 Orchestration<br/>intelligence_agent"]
ONG["06 Ongoing-Automation<br/>(keep-watching · DEFERRED)"]
end
ACQ --> CAP
MET --> CAP
CAP --> ACC
ID --> ACC
ACC --> ORC
ORC -. "standing need" .-> ONG
ONG -. "re-invokes verbs" .-> CAP
classDef a fill:#22314f,stroke:#5b7fbf,color:#e6edf7;
classDef b fill:#1f3a2e,stroke:#4f9d76,color:#e6f7ee;
classDef d fill:#3a2f1f,stroke:#bf975b,color:#f7efe6;
class CAP,ACC,ORC a;
class ACQ,MET,ID b;
class ONG d;
```
Memory for "what changed" = the chat history.
## 2. Doors — one registry, three surfaces (generated)
```mermaid
flowchart TD
REG["04 Capability registry<br/>web.scrape · web.discover · maps.search/place/reviews"]
REG --> R["REST + API keys<br/>(public · OSS self-host · revenue)"]
REG --> M["MCP server<br/>(external agents · fast-follow)"]
REG --> C["Chat tools<br/>(in-app agent, 07)"]
R --> EX["same executor · direct return · 03c billing"]
M --> EX
C --> EX
```
## 3. One-shot scrape (the core path) — direct call, no polling
```mermaid
sequenceDiagram
autonumber
participant U as User / Dev
participant D as Door 05 (REST · MCP · chat)
participant EX as Verb executor 04
participant ACQ as Acquisition / Maps actor
participant BILL as Billing (03c)
U->>D: intent (chat) OR raw verb call (REST/MCP)
D->>D: validate input_schema · authn/authz · meter-gate
D->>EX: call executor (same for all doors)
EX->>ACQ: crawl_url / maps / search
ACQ-->>EX: cleaned, AI-ready data
EX->>BILL: charge billing_unit per success
EX-->>D: results (returned directly)
D-->>U: plain-language answer (chat) / JSON (REST/MCP)
Note over U,BILL: Stateless — nothing persists.
```
## 4. Intent fork (in the agent, 07)
```mermaid
flowchart TD
U(["User speaks in natural language"]) --> R{"Intent router<br/>(intelligence_agent prompt, 07)"}
R -->|"find / compare / what is / right now"| A["ONE-SHOT<br/>compose verbs, answer (section 3)"]
R -->|"watch / track / notify me / over time"| B["KEEP-WATCHING<br/>hand to Ongoing-Automation (06)"]
R -->|ambiguous| Q["Ask ONE question:<br/>'just once, or keep watching?'"]
Q -->|once| A
Q -->|keep watching| B
B -.->|DESIGN DEFERRED| X["06 mechanism TBD<br/>(periodicity · delivery · context limits)"]
```
## 5. "What changed" — chat history is the memory
```mermaid
flowchart LR
P["Prior tool outputs<br/>(already in chat context)"] --> AG["intelligence_agent"]
NEW["Fresh verb call (04)"] --> AG
AG --> ANS["'Here's what's new since last time'<br/>(reasoned from chat history)"]
```
## 6. Ongoing-Automation (06) — intent only, mechanism deferred
```mermaid
flowchart LR
W["'Keep watching X'"] --> LOOP["Persistent ongoing chat<br/>periodically re-invokes verbs (04)"]
LOOP --> OUT["results delivered into the session<br/>(write-then-sync / stream)"]
OUT --> MEM["chat history = memory<br/>agent reports what's new"]
LOOP -.->|OPEN| Q["periodicity driver? · delivery channel? · context-window limit?"]
classDef d fill:#3a2f1f,stroke:#bf975b,color:#f7efe6;
class W,LOOP,OUT,MEM d;
```
> Section 6 is a **placeholder** — the periodic mechanism is designed separately (see `06-ongoing-automation.md`).

2
plans/backend/02-rename-backend.md
View file

@ -133,7 +133,7 @@ These do not move with a symbol rename; each is decided here.
- Frontend route segment `[search_space_id] -> [workspace_id]`, `search-space-settings/`, TS types, atoms, i18n, frontend Zero schema — deferred frontend umbrella. - Frontend route segment `[search_space_id] -> [workspace_id]`, `search-space-settings/`, TS types, atoms, i18n, frontend Zero schema — deferred frontend umbrella.
- Satellite apps (desktop, Obsidian, browser extension, evals) + docs — deferred. - Satellite apps (desktop, Obsidian, browser extension, evals) + docs — deferred.
- Connector `category` discriminator and `SearchSourceConnector` handling — Phase 4 ([04a-connector-category.md](04a-connector-category.md)). - Connector `category` discriminator and `SearchSourceConnector` handling — Phase 4 ([04-capabilities.md](04-capabilities.md)).
- Enum VALUE migration and observability-key rename — deliberately deferred announced changes. - Enum VALUE migration and observability-key rename — deliberately deferred announced changes.
--- ---

2
plans/backend/03c-crawl-billing.md
View file

@ -2,7 +2,7 @@
> **Status: ✅ IMPLEMENTED (`ci_mvp` @ `17bdb0682`).** `WebCrawlCreditService` added (mirrors `EtlCreditService`: gate → `check_credits` → `charge_credits`, plus static `billing_enabled`/`successes_to_micros`). Price is fully config-driven (`WEB_CRAWL_MICROS_PER_SUCCESS`, default `1000` = $1/1000; `WEB_CRAWL_CREDIT_BILLING_ENABLED`, default FALSE) — no hardcoded rate. Indexer wires owner-resolution + pre-flight block + audit-then-charge; both `scrape_webpage` tools fold one success into the turn accumulator (`call_kind="web_crawl"`). 20 new unit tests pass (service, config-driven price, indexer wiring, chat fold); `connector_indexers` suite green (120). Functional e2e (3a/3b/3c) green via `scripts/e2e_phase3_crawl_billing.py` (live proxy egress flip, static-tier crawl, chat fold = 1000µ, indexer bills owner 2×1000µ not trigger). Captcha per-attempt unit (`03d`) seam left open. **No migration** (`web_crawl` rides the existing `TokenUsage.usage_type` String(50)). > **Status: ✅ IMPLEMENTED (`ci_mvp` @ `17bdb0682`).** `WebCrawlCreditService` added (mirrors `EtlCreditService`: gate → `check_credits` → `charge_credits`, plus static `billing_enabled`/`successes_to_micros`). Price is fully config-driven (`WEB_CRAWL_MICROS_PER_SUCCESS`, default `1000` = $1/1000; `WEB_CRAWL_CREDIT_BILLING_ENABLED`, default FALSE) — no hardcoded rate. Indexer wires owner-resolution + pre-flight block + audit-then-charge; both `scrape_webpage` tools fold one success into the turn accumulator (`call_kind="web_crawl"`). 20 new unit tests pass (service, config-driven price, indexer wiring, chat fold); `connector_indexers` suite green (120). Functional e2e (3a/3b/3c) green via `scripts/e2e_phase3_crawl_billing.py` (live proxy egress flip, static-tier crawl, chat fold = 1000µ, indexer bills owner 2×1000µ not trigger). Captcha per-attempt unit (`03d`) seam left open. **No migration** (`web_crawl` rides the existing `TokenUsage.usage_type` String(50)).
> **⚠️ Billing surfaces reframed (2026-06-30 — Architecture correction + revamp adoption, `00-umbrella-plan.md`).** The implemented `WebCrawlCreditService` **stands unchanged**, but *where* it's called shifts, and the **charge moves to the capability executor** (revamp [`04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md)). Pipelines are dropped and `WEBCRAWLER_CONNECTOR` is retired, so the **connector/`webcrawler_indexer`** billing branch (described throughout this doc) is now **vestigial** (dormant, not deleted). The correct model: every `web.*`/`maps.*` verb **charges once per billable success inside its executor**, so all callers meter uniformly — chat, **automation/recurring**, REST, MCP, external-cron. This closes a **code-verified gap**: `run_agent_task` (automation `agent_task`) establishes **no turn accumulator**, so the shipped `scrape_webpage` fold (no-op when `get_current_accumulator()` is `None`) bills **nothing** under automations today. The interactive **chat turn accumulator becomes an optional presentation fold** (still shows the crawl line on a chat turn), **not** the charging mechanism. Read "connector/pipeline crawls" below as historical context. > **⚠️ Billing surfaces reframed (2026-06-30 — Architecture correction + revamp adoption, `00-umbrella-plan.md`).** The implemented `WebCrawlCreditService` **stands unchanged**, but *where* it's called shifts, and the **charge moves to the capability executor** ([`04-capabilities.md`](04-capabilities.md)). Pipelines are dropped and `WEBCRAWLER_CONNECTOR` is retired, so the **connector/`webcrawler_indexer`** billing branch (described throughout this doc) is now **vestigial** (dormant, not deleted). The correct model: every `web.*`/`maps.*` verb **charges once per billable success inside its executor**, so all callers meter uniformly — chat, ongoing automation, REST, MCP. This closes a **code-verified gap**: `run_agent_task` (automation `agent_task`) establishes **no turn accumulator**, so the shipped `scrape_webpage` fold (no-op when `get_current_accumulator()` is `None`) bills **nothing** under automations today. The interactive **chat turn accumulator becomes an optional presentation fold** (still shows the crawl line on a chat turn), **not** the charging mechanism. Read "connector/pipeline crawls" below as historical context.
> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`. > Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
> Depends on `03a-crawler-core.md` (the `CrawlOutcomeStatus.SUCCESS` signal + `crawls_succeeded` counter). Siblings: `03b-proxy-expansion.md`, `03e-stealth-hardening.md`, `03d-captcha-solving.md` (now active). `03d` extends this service with a **separate per-attempt captcha unit** (see §"Captcha billing seam" + `03d §4`). > Depends on `03a-crawler-core.md` (the `CrawlOutcomeStatus.SUCCESS` signal + `crawls_succeeded` counter). Siblings: `03b-proxy-expansion.md`, `03e-stealth-hardening.md`, `03d-captcha-solving.md` (now active). `03d` extends this service with a **separate per-attempt captcha unit** (see §"Captcha billing seam" + `03d §4`).

89
plans/backend/04-capabilities.md Normal file
View file

@ -0,0 +1,89 @@
# Phase 4 — Capabilities (the scraper APIs)
> Build first; every other phase calls this registry. Sibling: `05-access.md` (the doors).
> Reuses Phases 3a/3b/3c (shipped): `WebCrawlerConnector.crawl_url -> CrawlOutcome`, the proxy/stealth/
> captcha tiers, and the `WebCrawlCreditService` billing seam. Locate code by symbol/grep.
## Objective
Expose the Acquisition engine as a small set of typed, stateless scraper verbs (`web.*`, `maps.*`) that
return cleaned, AI-ready structured data. One capability = one function you call and get data back from.
## Verb set (MVP)
Two namespaces: `web.*` (generic crawler) and `maps.*` (Google Maps actor). Future platforms slot in as
their own namespace.
| Verb | Input → Output (cleaned) | Executes over | Billing unit |
|------|--------------------------|---------------|--------------|
| `web.scrape(urls[])` | `[{url, status, content, metadata}]` | loop `crawl_url` | per success |
| `web.discover(query, top_k)` | `[{url, title, snippet, provider}]` | search providers | per search / free |
| `maps.search(query, location)` | `[place]` | Maps actor | per place |
| `maps.place(place_id\|url)` | `place` | Maps actor | 1 |
| `maps.reviews(place)` | `[review]` (paged) | Maps actor | per page |
`maps.*` returns typed structured objects (`{name, rating, review_count, hours, price_level, …}`).
`web.scrape` takes a URL array.
## Execution
Each verb is a direct async call that returns its cleaned result. Slow verbs (large `web.scrape` arrays,
`maps.reviews` paging) are bounded/streamed at the endpoint.
## Registry
One entry per verb, and the single source of truth the doors (`05`) and the agent (`07`) read from:
```
Capability {
name # dotted, e.g. "web.scrape", "maps.search"
input_schema # Pydantic
output_schema # Pydantic (cleaned, AI-ready)
executor # async fn; wraps Acquisition / Maps actor; returns data directly
billing_unit # how the billing service charges this call
}
```
Adding a verb once lights it up on every door.
## Billing
A verb declares a `billing_unit`; charging is delegated to the billing service (`03c` first provider).
- `web.scrape` → per `SUCCESS` (`web_crawl` unit).
- `maps.*` → new per-place / per-page unit registered with the billing service.
- captcha attempts → existing per-attempt `web_crawl_captcha` unit (`03d`).
- `web.discover` → per-search unit (or free).
## Location
New Apache-2 package `app/capabilities/` (registry, schemas, executors) — open-source and self-hostable.
Executors import the proprietary Acquisition engine and Maps extractor (`app/proprietary/…`).
## Work items
1. Registry package: `Capability` dataclass + registry, imported by `05` and `07`.
2. `web.scrape` executor: loop `crawl_url` over the URL array → per-URL cleaned rows; `web_crawl` unit.
3. `web.discover` executor: wrap the search-provider core (SearXNG/Linkup/Baidu, env-keyed); its unit.
4. `maps.*` contracts: input/output schemas + executor stubs against the Maps actor.
5. Output normalization: each executor returns AI-ready structured output.
6. Billing seam: honor `billing_unit` via the billing service.
## Tests
- `web.scrape([a,b,c])` returns one cleaned row per URL inline; partial failures don't fail the batch.
- Each `SUCCESS` bills one `web_crawl` unit; `EMPTY`/`FAILED` free; disabling the flag no-ops.
- A door and the agent hit the same executor for a verb.
- `web.discover` returns `{url,title,snippet,provider}`; self-disables when no provider key is set.
- `maps.*` returns typed structured objects against fixtures.
## Out of scope
- Doors → `05`. Keep-watching mode → `06`. Agent → `07`.
- Google Maps actor build (incl. sourcing legality) — net-new, separate effort; `maps.*` are contracts.
- Additional platform namespaces; recursive site crawl; a generic `web.extract` verb.
## Open questions
- `web.discover` metered vs free.
- How large `web.scrape` / `maps.reviews` responses are bounded/streamed.

167
plans/backend/04a-connector-category.md
View file

@ -1,167 +0,0 @@
# Phase 4a — Connector taxonomy (Type-1/Type-2) + availability gating + MCP routing fix
> ## ↪ REDIRECT / DEMOTED (2026-06-30) — superseded by the revamp
> **The canonical Phase 4 is now [`revamp phases 4-7/04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md) + [`04b-access.md`](revamp%20phases%204-7/04b-access.md)** (typed capability verbs + generated doors — see `00-umbrella-plan.md`). This file's connector-taxonomy work is **demoted to backward-compat hygiene**: it is **not** the core of Phase 4. Read the revamp first.
>
> What survives from this doc, and where it goes:
> - **The one genuinely useful piece — the `MCP_CONNECTOR` (BYO MCP) routing-gap fix — moves to `revamp/04b`** (the "consume-user-MCP" work). That's the actionable carry-over.
> - **Keeping legacy branded natives `MIGRATING` (off) + existing indexed KB docs searchable, and Obsidian/Circleback `DISABLED`** — still true; a small hygiene task, no longer a headline phase.
> - **`is_pipeline_eligible` is dropped** (no pipelines); **Type-1 = file/KB-input only** (Drive/Dropbox/OneDrive + uploads → input-only KB); **`WEBCRAWLER_CONNECTOR` is RETIRED** (native `scrape_webpage` tool, now a `web.scrape` capability verb).
>
> The static-registry design below is retained for reference, but do not treat "connector two-type restructure" as the Phase 4 deliverable — the deliverable is the **capability registry** in `revamp/04a`.
> Part of **Phase 4 — Connector two-type restructure (backend)**. See `00-umbrella-plan.md`.
> Sibling: `04b-source-discovery.md` (web-search repurposing). Precondition: Phases 1–2 (rename) live.
> **Implementation note.** Phases 1–2 are **SHIPPED**, so the live code already says `workspace_id`/`Workspace` — substitute for the old `search_space_*`/`SearchSpace` names in citations below and grep the new name. Locate code by **symbol/grep**, not the absolute line numbers cited here, since the rename shifted them.
## Objective
Give connectors a first-class **Type-1 (Data Source) / Type-2 (MCP Tool)** taxonomy and an **availability** state, then gate creation, indexing, chat-subagent build, and (forward-compat) pipeline-eligibility off it — without a DB migration. Also fix the long-standing bug where generic `MCP_CONNECTOR` tools are discovered but dropped.
The taxonomy is **fully determined by `connector_type`** (every Notion row is the same category), so it is modeled as a **static code registry**, not a per-row column (resolved decision — avoids a migration + backfill drift across ~20 connector-create sites). `is_indexable` is **kept** (it is an orthogonal per-row capability that still gates the real `/index` + periodic machinery — e.g. Notion is `is_indexable=True` yet is a Type-2 tool).
## Locked model (MVP)
| Bucket | `category` | `availability` | Members | Behaviour |
|--------|-----------|----------------|---------|-----------|
| Functional data sources | `DATA_SOURCE` | `AVAILABLE` | `WEBCRAWLER_CONNECTOR`, `GOOGLE_DRIVE_CONNECTOR`, `COMPOSIO_GOOGLE_DRIVE_CONNECTOR`, `ONEDRIVE_CONNECTOR`, `DROPBOX_CONNECTOR` | create OK, `/index`+periodic OK, **pipeline-eligible** |
| Functional tool | `MCP_TOOL` | `AVAILABLE` | `MCP_CONNECTOR` (generic BYO MCP server) | create OK, acts in chat, **no pipelines** |
| Deprecated branded (indexers) | `MCP_TOOL` | `MIGRATING` | `NOTION_CONNECTOR`, `GITHUB_CONNECTOR`, `CONFLUENCE_CONNECTOR`, `BOOKSTACK_CONNECTOR`, `ELASTICSEARCH_CONNECTOR` | block new create, **disable `/index`+periodic**; disable subagent **only for Notion/Confluence** (GitHub/BookStack/Elasticsearch have no subagent); keep existing rows + already-indexed KB docs searchable |
| Deprecated branded (act-only) | `MCP_TOOL` | `MIGRATING` | `SLACK_CONNECTOR`, `TEAMS_CONNECTOR`, `LINEAR_CONNECTOR`, `JIRA_CONNECTOR`, `CLICKUP_CONNECTOR`, `AIRTABLE_CONNECTOR`, `DISCORD_CONNECTOR`, `GOOGLE_GMAIL_CONNECTOR`, `GOOGLE_CALENDAR_CONNECTOR`, `LUMA_CONNECTOR`, `COMPOSIO_GMAIL_CONNECTOR`, `COMPOSIO_GOOGLE_CALENDAR_CONNECTOR` | block new create, **disable subagent** (no KB data to keep) |
| Disabled | `DATA_SOURCE` | `DISABLED` | `OBSIDIAN_CONNECTOR`, `CIRCLEBACK_CONNECTOR` | block new create + subagent; not MCP-bound (distinct from MIGRATING). Obsidian is self-hosted-only today |
| Removed → `04b` | n/a | n/a | `SERPER_API`, `TAVILY_API`, `SEARXNG_API`, `LINKUP_API`, `BAIDU_SEARCH_API` | enum values dropped + repurposed in `04b`. **04a treats them as `HIDDEN`** (excluded from taxonomy) so the registry is total until `04b` removes them |
**"Migrating" UX:** existing `MIGRATING` rows remain in the DB and (for the indexers) their already-indexed documents stay searchable via the always-on `knowledge_base` subagent; only the connector's own management (re-index, periodic, live subagent) is turned off, with a "Migrating to MCP soon" status surfaced by the API.
## Current state (cited)
### The enum + model
- `class SearchSourceConnectorType(StrEnum)` — `db.py:85–117` (all **30** values, incl. the 5 search APIs `:86–90`, `MCP_CONNECTOR:113`, Composio variants `:115–117`). The registry's totality test must cover all 30.
- `class SearchSourceConnector` — `db.py:1829`. Columns: `connector_type` (`:1867`), `is_indexable` (`:1868`, `Boolean default False`), periodic fields `periodic_indexing_enabled`/`indexing_frequency_minutes`/`next_scheduled_at` (`:1880–1882`).
- Schema: `schemas/search_source_connector.py` — `is_indexable: bool` (`:16`), validator "periodic only if indexable" (`:42–45`), `Update.is_indexable` (`:66`).
### Where indexability is currently set (per connector, scattered)
Create routes hardcode `is_indexable` per type: `True` for Notion (`notion_add_connector_route.py:409`), Confluence (`:376`), Google Drive (`:420`), Dropbox (`:373`), OneDrive (`:380`); `False` for Slack/Teams/Discord/Gmail/Calendar/Linear/Jira/ClickUp/Airtable/Luma (their `*_add_connector_route.py`), MCP (`search_source_connectors_routes.py:2696`). Composio uses `INDEXABLE_TOOLKITS` (`composio_routes.py:289,400`, `composio_service.py:99`). `oauth_connector_base.py:59,598` defaults `True`. **This scatter is exactly why category lives in a registry, not duplicated at each create site.**
### The `/index` dispatch (the indexable set, today)
`index_connector_content()` — `search_source_connectors_routes.py:717` — is an `if/elif` chain on `connector_type` that dispatches to a Celery task for: Notion (`:850`), GitHub (`:861`), Confluence (`:872`), BookStack (`:885`), Google Drive (`:898`), OneDrive (`:948`), Dropbox (`:995`), Elasticsearch (`~:1050`), WebCrawler (`:1058`). **Movers** (Notion/GitHub/Confluence/BookStack/Elasticsearch) lose this branch; the file-based + WebCrawler branches stay.
### The chat subagent maps + the routing gap
- `multi_agent_chat/constants.py` (the chat-package root `constants.py`, **not** under `subagents/`) — `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS` (`:5–24`, connector_type → subagent name; **note GitHub/BookStack/Elasticsearch have NO entry** — they are index-only, no chat subagent) and `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP` (`:26–44`, subagent → required tokens; `deliverables`/`knowledge_base` require `frozenset()` = always built). The required tokens are a **mix of connector types and doc types** (e.g. `notion`→`NOTION_CONNECTOR`, but `dropbox`→`DROPBOX_FILE`, `google_drive`→`GOOGLE_DRIVE_FILE`, `onedrive`→`ONEDRIVE_FILE`).
- Subagent exclusion: `subagents/registry.py` `get_subagents_to_exclude(available_connectors)` (`:136–152`) excludes a builder in **two** cases: (a) it is **absent from** `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP` (`required_tokens is None`, `:145–147`) → excluded; (b) it has non-empty required tokens that don't intersect the available set (`:150–151`) → excluded. An **empty** `frozenset()` (deliverables/knowledge_base) is always kept (`:148–149`). `build_subagents(..., exclude=...)` (`:182–220`) additionally hard-skips `memory`/`research` (`:195`) and the names in `exclude`, then calls each builder with `mcp_tools=mcp.get(name)` (`:209`). `SUBAGENT_BUILDERS_BY_NAME` (`:92–112`) has **no `mcp` builder** today. **Consequence for §5:** adding an `mcp` builder *without* a `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP["mcp"]` entry would hit case (a) and exclude it forever — so §5's map entry is mandatory, not optional.
- **CRITICAL — `available_connectors` is NOT raw connector types.** `factory.py:102–105` computes `connector_types = get_available_connectors(...)` then `available_connectors = map_connectors_to_searchable_types(connector_types)`. This **shared, already-mapped** list of searchable-type tokens feeds *multiple* consumers: `get_subagents_to_exclude` (`factory.py:253`, `main_agent/middleware/stack.py:202`), the on-demand KB-search tool (`subagents/builtins/knowledge_base/tools/search_knowledge_base.py:_search_types:53–63` does `types.update(available_connectors)` at `:61–62` — this is the post-`main`-merge replacement for the now-deleted `shared/middleware/knowledge_search.py`, which used to do the same in `_resolve_search_types`), `web_search`'s live-connector filter (`shared/tools/web_search.py`), and the `deliverables` `report` tool. **Filtering this list would break legacy KB-doc searchability and web search** — see Target §4.
- `map_connectors_to_searchable_types` (`connector_searchable_types.py:65–100`) maps each configured connector type to a searchable token; **`MCP_CONNECTOR` is absent from `_CONNECTOR_TYPE_TO_SEARCHABLE` (`:22–54`)**, so MCP connectors never produce a token in `available_connectors`.
- **The bug:** `subagents/mcp_tools/index.py` `partition_mcp_tools_by_connector` (`:55–99`) routes each MCP tool via `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS.get(connector_type)`; `MCP_CONNECTOR` has **no entry**, so it hits the `connector_agent is None` branch and is **skipped with a warning** (`:88–95`). Generic user MCP servers therefore contribute zero tools today. (The bucket key is the agent name, e.g. `"mcp"`, consumed by `build_subagents` via `mcp.get("mcp")`.)
### Query-time searchable mapping (keeps legacy docs visible)
`main_agent/runtime/connector_searchable_types.py` `_CONNECTOR_TYPE_TO_SEARCHABLE` (`:22–54`, unchanged by the `main` merge) maps connector types → searchable doc-types that scope the KB-search tool (via `available_connectors` → `search_knowledge_base.py:_search_types`). The MIGRATING indexers' entries (NOTION→`NOTION_CONNECTOR`, etc.) **stay** so already-indexed docs remain searchable. (WebCrawler→`CRAWLED_URL`, file connectors→`*_FILE` stay too.)
## Target design
### 1. The static registry (single source of truth)
New `app/connectors/connector_registry.py` (or `app/utils/connector_registry.py`):
```python
class ConnectorCategory(StrEnum):
DATA_SOURCE = "DATA_SOURCE"
MCP_TOOL = "MCP_TOOL"
class ConnectorAvailability(StrEnum):
AVAILABLE = "AVAILABLE" # usable now
MIGRATING = "MIGRATING" # branded native, "moving to MCP soon"
DISABLED = "DISABLED" # off for MVP, not MCP-bound (Obsidian/Circleback)
HIDDEN = "HIDDEN" # not a real connector in the taxonomy (search APIs, until 04b removes them)
# connector_type -> (category, availability). Total over SearchSourceConnectorType.
CONNECTOR_REGISTRY: dict[SearchSourceConnectorType, tuple[ConnectorCategory, ConnectorAvailability]] = { ... }
def get_category(ct) -> ConnectorCategory | None
def get_availability(ct) -> ConnectorAvailability
def is_creatable(ct) -> bool # availability == AVAILABLE
def is_pipeline_eligible(ct) -> bool # category == DATA_SOURCE and availability == AVAILABLE
def is_indexable_type(ct) -> bool # the file/web data sources that own a Celery indexer
```
- **Totality guard (test):** assert every `SearchSourceConnectorType` member has a registry entry, so a newly added connector can't silently fall through gating.
- Exposed on the connector read schema as **computed fields** (`category`, `availability`) so the API/frontend can label and filter without a DB column. (Promote to a column later only if Zero/SQL filtering needs it — see umbrella resolved log.)
### 2. Create gating
In the generic create handler `create_search_source_connector` (`search_source_connectors_routes.py:172`, `@router.post("/search-source-connectors")`) **and** the per-service add/OAuth routes (e.g. `notion_add_connector_route.py`, `luma_add_connector_route.py:51`, the Composio routes) reject when `not is_creatable(connector_type)` with a clear 4xx ("This connector is migrating to MCP and can't be added yet" / "disabled for MVP"). The existing per-type duplicate check (`:198–212`) is unaffected. Because the per-service routes each build their own connector, the gate must be applied in each (or in a shared helper they all call) — verify by grepping the `*_add_connector_route.py` set. This is the behavioural change that blocks new branded connectors; existing rows are untouched.
### 3. Index gating
Guard `index_connector_content` (`:717`) up front: if `not is_indexable_type(connector_type)`, return a 4xx/no-op ("indexing disabled — migrating to MCP"). This neutralizes the Notion/GitHub/Confluence/BookStack/Elasticsearch branches without deleting their (now-dead) code paths.
**Periodic path (must also be gated).** Create-gating blocks *new* MIGRATING rows, but **existing** MIGRATING connectors with `periodic_indexing_enabled=True` already have a `next_scheduled_at` and would keep firing via the meta-scheduler Beat task that polls `next_scheduled_at` every minute: `tasks/celery_tasks/schedule_checker_task.py` (due query `next_scheduled_at <= now` `:36`, dispatch loop `:88`, `task = task_map.get(connector.connector_type)` then `.delay(...)` `:118`). The first-run trigger is `create_periodic_schedule` (`utils/periodic_scheduler.py:30`, `task.delay(...)` `:97`). **Both paths re-dispatch through the same per-type Celery tasks** (`index_crawled_urls_task`, `index_notion_pages_task`, …), so the robust single chokepoint is to **gate the Celery index tasks at entry** by `is_indexable_type(connector_type)` — this covers the manual `/index` route, the first-run trigger, and the recurring meta-scheduler pass at once (a no-op early-return for MIGRATING types). (Aside: `schedule_checker_task` already auto-disables a `LIVE_CONNECTOR_TYPES` set at `:74-77` — an existing precedent for turning periodic off per type.) The schema validator "periodic only if indexable" (`schemas/search_source_connector.py:42–45`) is unchanged and unaffected (it keys off the per-row `is_indexable` bool, not the registry).
Note: **update** (`PUT /search-source-connectors/{id}`, `search_source_connectors_routes.py:371`) is intentionally **left allowed** for MIGRATING rows so existing users can still edit/disable them; only create + index are gated.
### 4. Subagent gating (turns off branded chat tools) — by NAME, not by token
> **Do NOT filter `available_connectors`.** It is a shared, already-mapped searchable-type list (see Current State). Stripping MIGRATING entries from it would (a) remove `NOTION_CONNECTOR`/etc. from the KB-search tool's doc-type scope (`search_knowledge_base.py:_search_types`) → **break the very legacy-doc searchability this plan promises**, and (b) remove the live-search tokens → break `web_search` before 04b. The token list must stay intact.
Instead, **exclude the deprecated subagents by NAME**, derived from the registry. Extend `get_subagents_to_exclude` (`registry.py:136`) so that, in addition to its current token check, it also excludes any subagent whose mapped connector type(s) are all non-`AVAILABLE`:
- Build the reverse of `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS` (subagent_name → set of connector types). For each `SUBAGENT_BUILDERS_BY_NAME` entry, if it has connector type(s) and **every** one is non-`AVAILABLE` in the registry, add it to the excluded set unconditionally (regardless of tokens).
- This is the single chokepoint used by both `factory.py:253` and `stack.py:202`, and feeds `main_prompt_registry_subagent_lines(exclude)`, so the deprecated specialists disappear from both the build and the prompt.
Net effect for MVP:
- **Excluded (MIGRATING):** `notion`, `confluence`, `slack`, `teams`, `linear`, `jira`, `clickup`, `airtable`, `discord`, `gmail`, `calendar`, `luma`. (GitHub/BookStack/Elasticsearch have no subagent — they are handled purely by §3 index gating; their indexed docs stay searchable via the untouched token list.)
- **Kept (AVAILABLE):** `knowledge_base`, `deliverables`, `research`, `memory` (builtins), the new `mcp` subagent, **and the file-source specialists `google_drive`, `dropbox`, `onedrive`** — these map to `AVAILABLE` Type-1 connectors, so by the registry rule they are NOT deprecation-excluded; they remain token-gated as today (built only when that file connector is configured). **Decision:** keep them — a Type-1 data source can still own a chat specialist over its indexed files; the two-type split governs pipeline-eligibility + the MCP migration, not whether an AVAILABLE connector may have a subagent. (If we later want pure-ingestion file connectors with no specialist, flip them by adding a `subagent_chat` flag to the registry — out of scope here.)
### 5. MCP_CONNECTOR routing-gap fix (makes generic Type-2 work)
Two changes are needed — the routing map AND surfacing the token, or the subagent will still never build:
1. **Routing map** (`multi_agent_chat/constants.py`): add `"MCP_CONNECTOR": "mcp"` to `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS` (so `partition_mcp_tools_by_connector` stops hitting the `connector_agent is None` skip at `mcp_tools/index.py:88–95`), and `"mcp": frozenset({"MCP_CONNECTOR"})` to `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP`.
2. **Builder**: add a generic `mcp` entry to `SUBAGENT_BUILDERS_BY_NAME` (`registry.py:92`); `build_subagents` will call it with `mcp_tools=mcp.get("mcp")` (the bucket key from step 1's map value).
3. **Surface the token** (the gap step 4's predecessor missed): `MCP_CONNECTOR` is **not** in `_CONNECTOR_TYPE_TO_SEARCHABLE`, so `available_connectors` never contains it and the `mcp` subagent's required token `{MCP_CONNECTOR}` would never intersect → it would be excluded forever. Fix by adding `"MCP_CONNECTOR": "MCP_CONNECTOR"` to `_CONNECTOR_TYPE_TO_SEARCHABLE` (`connector_searchable_types.py:22`). Side effect: `MCP_CONNECTOR` joins the KB-search tool's doc-type scope (`search_knowledge_base.py:_search_types`), where it matches zero indexed docs — harmless no-op. (Alternative if the no-op is undesirable: special-case `mcp` in `get_subagents_to_exclude` to include it whenever any configured connector is an `AVAILABLE` `MCP_TOOL`, passing the raw `connector_types` alongside. The searchable-map entry is simpler and preferred.)
### 6. Keep `is_indexable` semantics intact
No change to the column or its validator. `is_indexable` continues to gate the real `/index`+periodic machinery for the data sources; the registry's `is_indexable_type` is a **type-level** allowlist layered on top (a row must be both an indexable type AND `is_indexable=True`). This avoids reinterpreting historical rows.
## Work items
1. **Registry module** + `ConnectorCategory`/`ConnectorAvailability` enums + the total `CONNECTOR_REGISTRY` map + helper predicates (incl. reverse subagent-name→types helper for §4).
2. **Schema**: add computed `category`/`availability` to the connector read schema (`schemas/search_source_connector.py`).
3. **Create gating** in the generic create handler (`:172`) + every per-service add route (reject non-`AVAILABLE`).
4. **Index gating** at the Celery index-task entry (`is_indexable_type`) — covers manual `/index`, first-run trigger, and meta-scheduler.
5. **Subagent gating by name**: extend `get_subagents_to_exclude` to also exclude subagents whose mapped connector types are all non-`AVAILABLE` (do **not** filter the shared `available_connectors` token list).
6. **MCP routing-gap fix**: `multi_agent_chat/constants.py` map entries + generic `mcp` builder in `SUBAGENT_BUILDERS_BY_NAME` + `MCP_CONNECTOR` token in `_CONNECTOR_TYPE_TO_SEARCHABLE` (so the subagent actually builds).
7. **Tests** (below).
## Tests
- **Registry totality**: every `SearchSourceConnectorType` has an entry (guards future additions).
- **Create gating**: creating any `MIGRATING`/`DISABLED` type → 4xx; `WEBCRAWLER`/file/`MCP_CONNECTOR` → OK; MCP multi-instance still works.
- **Index gating**: index task entry for Notion/GitHub/Confluence/BookStack/Elasticsearch → no-op; WebCrawler/GDrive/OneDrive/Dropbox → runs. Include a case for an **existing** MIGRATING row with `periodic_indexing_enabled=True` + due `next_scheduled_at` → meta-scheduler triggers a no-op (does not re-index).
- **Subagent gating by name**: with a (legacy) Notion + Slack connector configured, `get_subagents_to_exclude` excludes `notion`/`slack`; `knowledge_base`/`deliverables` always built; a configured `google_drive`/`dropbox`/`onedrive` (AVAILABLE) is **not** excluded (still token-gated).
- **No collateral damage to the token list**: gating subagents must NOT change `available_connectors` — assert the KB-search tool still resolves the `NOTION_CONNECTOR` doc type (via `_search_types`) for the legacy connector (guards against the "filter the shared list" regression).
- **MCP gap (two-part)**: (a) a configured `MCP_CONNECTOR` produces a `"mcp"` token in `map_connectors_to_searchable_types`; (b) its tools land in the `mcp` bucket and the `mcp` subagent is actually built (regression for both `mcp_tools/index.py:88–95` and the missing-token exclusion).
- **Legacy searchability**: a search space with an existing (now-MIGRATING) Notion connector + indexed docs still returns those docs via the KB-search tool (`connector_searchable_types` Notion entry unchanged).
## Risks / trade-offs
- **Breaks current branded chat tooling (accepted).** Deprecating act-only natives removes their live chat subagents for existing users mid-pivot. Per the umbrella's "DB migrations carry users; backend behaviour can change" posture, this is acceptable; the "migrating to MCP soon" status sets expectations.
- **Dead code paths left in place.** The Notion/GitHub/Confluence/BookStack/Elasticsearch `/index` branches + indexers remain but are gated off (not deleted) to keep the diff small and reversible; delete when the real MCP migration lands.
- **No column = not SQL/Zero-filterable.** Frontend (deferred) filters via the API's computed fields. If Zero needs server-side filtering later, promoting the registry to a denormalized column is additive.
- **Registry/`is_indexable` dual-gate.** Two truths (type-level allowlist + per-row bool) must agree; the totality test + index-gating test cover the seam.
- **Shared `available_connectors` is a footgun.** It feeds subagent-exclusion, the KB-search tool's doc-type scope, web_search, and the deliverables `report` tool simultaneously. Subagent gating is therefore done by *name* (registry-driven), never by mutating that list; the "no collateral damage" test pins this. This also removes any 04a→04b sequencing hazard (04a no longer touches the live-search tokens that 04b later retires).
- **File-source specialists kept.** `google_drive`/`dropbox`/`onedrive` stay as AVAILABLE chat specialists (decision in §4). If product wants pure-ingestion file connectors, that's an additive registry flag later.
## Out of scope (hand-offs)
- Search-API enum removal, key relocation, and the source-discovery endpoint → `04b`.
- Pipeline tables + actually creating pipelines from Type-1 connectors → Phases 5–7 (`is_pipeline_eligible` is defined here for them to consume).
- Real MCP migration of the branded connectors (re-point to MCP servers) → post-MVP.
- Frontend connector UI restructure → frontend umbrella.

120
plans/backend/04b-source-discovery.md
View file

@ -1,120 +0,0 @@
# Phase 4b — Web-search repurposing + source-discovery endpoint
> ## ↪ REDIRECT / ABSORBED (2026-06-30) — superseded by the revamp
> **This subplan is absorbed into the canonical [`revamp phases 4-7/04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md) as the `web.discover` verb.** The substance still holds — drop the 5 search `connector_type` values (`SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API`) + their paths (destructive migration), move SearXNG/Linkup/Baidu to **platform env** config, and rewire the chat `web_search` tool onto them — but the "source-discovery endpoint" is now a **capability verb** (`web.discover(query, top_k)`) in the registry, exposed identically via chat/REST/MCP (not a bespoke route), and its output feeds the agent (or a Tracker binding), **not** "a pipeline." Read `revamp/04a` (`web.discover`) as canonical; use the provider/migration detail here as implementation reference.
>
> Part of **Phase 4 — Capabilities & Access (backend)**. See `00-umbrella-plan.md`.
> Sibling (old): `04a-connector-category.md` (demoted to hygiene). Precondition: Phases 1–2 (rename) live.
> **Implementation note.** Phases 1–2 are **SHIPPED**, so the live code already says `workspace_id`/`Workspace` — substitute for the old `search_space_*`/`SearchSpace` names in citations below and grep the new name; locate code by **symbol/grep**, not the absolute line numbers cited.
## Objective
Retire the five web-search **connector types** and re-cast the survivors as **platform-level providers** that power two things: (1) the existing chat `web_search` tool, and (2) a new **source-discovery endpoint** that, given a topic/competitor, suggests candidate URLs the user can feed into the Universal WebURL Crawler / a pipeline.
Resolved decisions driving this:
- **Drop all 5 search `connector_type` enum values** (`SERPER_API`, `TAVILY_API`, `SEARXNG_API`, `LINKUP_API`, `BAIDU_SEARCH_API`).
- **Tavily + Serper removed entirely** (no provider, no code path).
- **SearXNG / Linkup / Baidu survive as platform providers.** SearXNG is already platform/env-based; **Linkup + Baidu move from per-workspace connector `config` to platform/env config** (one app-wide key set, matching the "single provider app-wide" style from `03b`).
## Current state (cited)
### The 5 search types are first-class connectors today
- Enum members `SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API` — `db.py:86–90` (`SERPER_API` is annotated "NOT IMPLEMENTED YET"; SearXNG is already platform-backed despite being an enum value).
- Per-workspace search methods on `ConnectorService` read keys from the connector row's `config`: `search_tavily` (`connector_service.py:481`, key `TAVILY_API_KEY:510`), `search_searxng` (`:587`), `search_baidu` (`:614`), `search_linkup` (`:1968`, key `LINKUP_API_KEY:2000`).
- Query-time routing: `connector_searchable_types.py:_CONNECTOR_TYPE_TO_SEARCHABLE` (`:22–26`) maps `TAVILY_API`/`LINKUP_API`/`BAIDU_SEARCH_API` to the web_search tool ("live search connectors"); the other connector types map to KB pre-search.
### The web_search tool already has a platform/per-workspace split
> **Post-`main`-merge note.** A `main` sync **consolidated web_search to a single tool**: the old research-subagent copy `subagents/builtins/research/tools/web_search.py` was **deleted**, and there is now **one** factory at `app/agents/chat/shared/tools/web_search.py`. The research subagent now imports it (`subagents/builtins/research/tools/index.py:10,24–26`). So everywhere below that said "both variants / mirror" is now **one file**. The same merge also retired the XML result blob in favor of `[n]`-citation rendering (see §3).
`web_search` tool factory `create_web_search_tool(search_space_id, available_connectors)` — the single canonical tool at `app/agents/chat/shared/tools/web_search.py`:
- `_LIVE_SEARCH_CONNECTORS = {TAVILY_API, LINKUP_API, BAIDU_SEARCH_API}` (`:41–45`), `_LIVE_CONNECTOR_SPECS` → `ConnectorService.search_*` (`:47–51`), per-connector dispatch `_search_live_connector` (`:115–151`). The factory picks the active live connectors by intersecting `available_connectors` with `_LIVE_SEARCH_CONNECTORS` (`:163–167`).
- At call time it fans out **in parallel** to platform SearXNG (`web_search_service.is_available()` / `.search()`, `:204–214`) **plus** each active per-workspace live connector (`:216–228`), then dedupes by URL (`:242–250`).
- Platform SearXNG service: `app/services/web_search_service.py` — env-gated by `config.SEARXNG_DEFAULT_HOST`, in-process circuit breaker + Redis result cache; returns the `(result_obj, documents)` shape the tool consumes.
### Config
- `config.SEARXNG_DEFAULT_HOST = os.getenv("SEARXNG_DEFAULT_HOST")` — `config/__init__.py:558`. There are **no** platform env vars for Linkup/Baidu/Tavily today (their keys live in connector `config`).
## Target design
### 1. Platform search-provider config (env)
Add platform env knobs alongside `SEARXNG_DEFAULT_HOST` (`config/__init__.py:~558`):
- `LINKUP_API_KEY = os.getenv("LINKUP_API_KEY")`
- `BAIDU_SEARCH_API_KEY = os.getenv("BAIDU_SEARCH_API_KEY")` (+ the optional knobs `search_baidu` reads from connector `config` today — `BAIDU_MODEL` / `BAIDU_SEARCH_SOURCE` / `BAIDU_ENABLE_DEEP_SEARCH`, per `validators.py:507-514`; the required key is `BAIDU_API_KEY` in the per-workspace config, relocated to this platform env var — port from its current `config` reads).
- (SearXNG unchanged.) Document all three in `.env.example` as optional; each provider self-disables when its key/host is unset (mirrors `web_search_service.is_available()`).
No Tavily/Serper env (removed).
### 2. A platform search-providers service (consolidate the survivors)
Introduce `app/services/web_search_service.py`-level functions (or a small `search_providers/` package) so all three survivors share one platform shape `(result_obj, documents)`:
- `searxng.search(...)` — the existing `web_search_service.search` (unchanged).
- `linkup.search(...)` / `baidu.search(...)` — **port the bodies of `ConnectorService.search_linkup`/`search_baidu`** but source the API key from `config.*` (platform) instead of `connector.config` (per-workspace), and drop the `search_space_id`/connector lookups.
- Each is `is_available()`-gated on its env key. Tavily (`search_tavily`) is deleted.
This removes per-workspace coupling: web search + discovery no longer depend on any connector row.
### 3. Rewire the chat `web_search` tool (single tool)
- Drop the `_LIVE_SEARCH_CONNECTORS` / `_LIVE_CONNECTOR_SPECS` / `_search_live_connector` mechanism in the **single** tool `shared/tools/web_search.py` (`:41–51`, `:115–151`, and the factory's live-connector intersection `:163–167`, plus the per-workspace fan-out `:216–228`). Instead, fan out to the **platform providers** that are `is_available()` (SearXNG + Linkup + Baidu), all keyless from the caller's view.
- Keep the existing parallel-gather + URL-dedupe (`:233–250`). **Note:** the `main` merge replaced the old XML result blob with `[n]`-citation rendering — the tool now builds `RenderableDocument`s (`_to_renderable_web_documents:66–112`) and returns them via `render_web_results` + `load_registry`, persisting the `citation_registry` on state (`:252–281`). Leave that render/registry path **unchanged**; only the *provider sourcing* (SearXNG+Linkup+Baidu, env-keyed) changes.
- **Call-site churn — keep the `available_connectors` parameter in the factory signature but stop using it for provider selection** (providers are now env-derived). This avoids touching every caller: `main_agent/tools/registry.py:24,39`, `subagents/builtins/research/tools/index.py:10,24–26`, and `anonymous_chat/agent.py:127` (passes `available_connectors=None` already). **`deliverables` is no longer a web_search caller** (its KB/web tooling was removed in the `main` merge) — drop it from the churn list. `search_space_id` likewise stays for logging only. (A later cleanup can remove the now-dead params.)
- Net behaviour: web search works platform-wide whenever any provider env key is set; it no longer requires the workspace to have a search connector configured (so anonymous chat gains web search too).
### 4. Source-discovery endpoint (the new capability)
New route (e.g. `routes/source_discovery_routes.py`, mounted under `/api/v1/workspaces/{workspace_id}/...` to match the renamed surface): `POST .../source-discovery` taking `{ query/topic, top_k }` and returning a **ranked list of candidate URLs** (url, title, snippet, provider) for the user to add to the WebURL Crawler / a pipeline.
- Implementation = call the platform providers (§2) in parallel, dedupe by URL (reuse the tool's dedupe), and return URL-centric results (plain `{url,title,snippet,provider}` — **not** the chat tool's `[n]`-cited `render_web_results` output, which is for in-conversation citation, not URL suggestion).
- Auth: standard workspace access check (same dependency as other workspace routes).
- This is **backend-only**; the UX (a "find sources" affordance when configuring a crawler/pipeline) is deferred to the frontend umbrella.
- Optional thin reuse: the chat `web_search` `_web_search_impl` and this endpoint can share a `discover_urls(query, top_k) -> list[UrlCandidate]` core.
### 5. Drop the 5 connector types
- **Remove the enum members** `SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API` from `SearchSourceConnectorType` (`db.py:86–90`).
- **Remove their connector code paths**: `ConnectorService.search_tavily` (`:481`, delete) and the per-workspace `search_searxng` (`:587`)/`search_baidu` (`:614`)/`search_linkup` (`:1968`) (replace with the platform service §2; delete the connector-config variants that read `connector.config[...]_API_KEY`, e.g. `:510`, `:2000`). Remove the three search entries from `connector_searchable_types.py:24–26`. Remove the `*_API` entries from the connector-config validation dict (`utils/validators.py:490–514`, the 5 keys in `connector_rules`: `SERPER_API:490`/`TAVILY_API:491`/`SEARXNG_API:492-505`/`LINKUP_API:506`/`BAIDU_SEARCH_API:507-514`) and `04a`'s `HIDDEN` registry entries for these types. **Grep-guard:** no remaining references to the 5 enum names or `search_tavily` anywhere.
- **Migration (this phase is NOT migration-free):** existing connector rows of these 5 types must be handled before/with the enum change:
- Alembic migration: `DELETE FROM search_source_connectors WHERE connector_type IN ('SERPER_API','TAVILY_API','SEARXNG_API','LINKUP_API','BAIDU_SEARCH_API');` (FKs cascade; these rows only held search API keys, now relocated to env). Self-hosted operators re-add Linkup/Baidu keys via env — call out in the migration docstring + `.env.example`.
- **PG enum handling:** dropping a value from a Postgres enum type requires a type recreate. Lowest-risk option: after deleting the rows, leave the now-unused labels in the PG enum type (harmless orphans) and just remove them from the Python `StrEnum` so no new rows can use them. If a clean type is wanted, do the standard "create new enum → `ALTER COLUMN ... TYPE` with a `USING` cast → drop old" dance in the migration (heavier; only if desired). Recommend the orphan-label approach for MVP.
- Also scrub `document.document_type` / search rows that referenced `TAVILY_API`/`LINKUP_API`/`BAIDU_SEARCH_API` as a **doc type** only if such persisted docs exist (live-search results were ephemeral, not indexed — verify; expectation is none).
## Work items
1. **Config**: `LINKUP_API_KEY`, `BAIDU_SEARCH_API_KEY` (+ Baidu host/region) in `Config` + `.env.example`; keep `SEARXNG_DEFAULT_HOST`.
2. **Platform providers**: port Linkup/Baidu search to env-keyed platform functions sharing SearXNG's `(result_obj, documents)` shape; delete Tavily.
3. **Rewire `web_search`** (the single shared tool `shared/tools/web_search.py`) to fan out to platform providers; drop `available_connectors`/live-connector plumbing; keep dedupe + the `[n]`-citation render path.
4. **Source-discovery endpoint**: `POST .../source-discovery` → ranked URL candidates; shared `discover_urls()` core.
5. **Drop the 5 enum values** + remove `search_*` connector methods, `connector_searchable_types` search entries, validators, and `04a` HIDDEN entries.
6. **Migration**: delete the 5 connector types' rows; handle the PG enum (orphan-label approach); docstring notes the env re-keying.
7. **Tests** (below).
## Tests
- **Provider availability**: each of SearXNG/Linkup/Baidu self-disables when its env key/host is unset; `web_search` returns "not available" only when all are unset.
- **web_search rewire**: with `LINKUP_API_KEY` set (no search connector rows), `web_search` still returns Linkup results; results from multiple providers dedupe by URL.
- **No per-workspace dependency**: a workspace with zero connectors still gets web search + source-discovery results.
- **Source-discovery endpoint**: returns URL candidates `{url,title,snippet,provider}`; enforces workspace access; empty when no providers configured.
- **Enum removal**: `SearchSourceConnectorType` no longer has the 5 values; creating one → 422; the migration deletes existing rows; reading remaining connectors is unaffected.
- **Tavily gone**: no import/path references `search_tavily`/`TAVILY_API` remain (grep guard).
## Risks / trade-offs
- **Per-workspace search keys become platform-wide (accepted).** Workspaces can no longer bring their own Linkup/Baidu keys; one app-wide key set serves everyone (matches the single-provider posture). Self-hosted must move keys to env — surfaced in migration docs.
- **Destructive migration.** Deleting the 5 connector types' rows is irreversible (downgrade can recreate rows but not their secrets). Gate behind the standard backup/runbook; the data is just API-key config now living in env.
- **PG enum orphan labels.** Leaving unused enum labels is cosmetically untidy but avoids a risky type-recreate; documented as a deliberate trade-off.
- **Sequencing with 04a.** 04a marks the 5 types `HIDDEN` so the taxonomy stays total in the interim; 04b removes them. If 04b ships first, 04a's HIDDEN bucket is simply never populated — both orders are safe, but `04a → 04b` is the intended order.
## Out of scope (hand-offs)
- Taxonomy/gating/MCP-routing fix → `04a`.
- Using discovered URLs to actually create a pipeline/crawl → Phases 5–7 (this endpoint only *suggests* URLs).
- Source-discovery UX (the "find sources" affordance) → frontend umbrella.
- Crawl billing for any crawl the user starts from a discovered URL → already covered by `03c` (the crawl path bills regardless of how the URL was found).

85
plans/backend/05-access.md Normal file
View file

@ -0,0 +1,85 @@
# Phase 5 — Access / Surfaces (REST · MCP · chat doors)
> Build after `04`. Together `04 + 05` ship the scraper-API product.
> Reuses shipped infra: Identity/Tenancy, API keys, the chat agent + tool registry, the streaming layer,
> Metering (`03c`), and Rocicorp Zero reactive sync. Locate code by symbol/grep.
## Objective
Expose the `04` capability registry through three doors — REST + API keys, an MCP server, and chat tools —
each generated from the one registry so the I/O contract can't drift. Every door is the same thin adapter
with no business logic.
## Adapter shape (identical on every door)
```
parse input → validate against verb.input_schema → authn/authz → meter-gate (03c)
→ call the SAME executor (04) → serialize verb.output_schema → return cleaned data
```
The executor returns directly (`04`).
## The three doors
| Door | Who | Auth | Status |
|------|-----|------|--------|
| REST + API keys | external developers | existing API-key infra | public day one |
| MCP server | external agents (Cursor/ChatGPT/Claude) | OAuth 2.1 or bearer (at implementation) | fast-follow |
| Chat tools | the in-app agent (`07`) | existing session + workspace | seed exists (`scrape_webpage`) |
Routes are generated, so public REST on day one costs nothing extra.
## Chat surface
The in-app product is the conversation: the user describes a need in plain language and the agent (`07`)
picks and fills verbs. Raw typed verbs are exposed on REST/MCP for devs and external agents. The chat
door classifies intent and routes:
```
"find / compare / what is / pull / right now" → ONE-SHOT → call verbs, answer now
"watch / track / notify me / keep an eye / over time" → KEEP-WATCHING → hand to the ongoing mode (06)
```
## MCP — two directions
- **Serve MCP** (new door): our capabilities as a remote MCP server — Streamable-HTTP `/mcp`, stateless,
bounded/paginated outputs, read-only verbs (least privilege).
- **Consume MCP** (reuse): the BYO-`MCP_CONNECTOR` — the user's own external MCP tools inside our chat
agent. Fix its routing-map gap here.
## Result delivery
A chat tool call returns its data inline in the turn. Asynchronous results (e.g. the `06` ongoing mode
writing between turns) reach the client via write-then-sync / stream: the worker writes rows and the
client receives them over the existing SSE stream or a Zero-published table.
## Work items
1. Door generator: from the `04` registry emit (a) REST routes + models, (b) MCP tool schemas + handlers,
(c) chat tool defs + handlers.
2. REST surface: public routes + API-key auth (reuse) + `03c` meter-gate.
3. Chat tools: generalize `scrape_webpage` into the registry-backed set (direct-return).
4. Intent routing: classify one-shot vs keep-watching; hand keep-watching to `06`.
5. MCP server (fast-follow): Streamable-HTTP `/mcp`, least-privilege; auth depth at implementation.
6. Consume-MCP fix: repair the BYO-`MCP_CONNECTOR` routing-map gap.
## Tests
- A verb added to the registry appears on REST + MCP + chat with identical I/O.
- REST without a valid key → 401; an over-budget call → blocked by the `03c` gate; a success charges once.
- A chat scrape returns cleaned data inline in the same turn.
- "compare X and Y" → one-shot; "watch X weekly" → routed to `06`.
- A configured BYO MCP tool is reachable by the chat agent.
## Out of scope
- Verbs, executors, billing units → `04`.
- Keep-watching mechanism + its delivery channel → `06`.
- The CI subagent + its prompt → `07`.
- Legacy branded connectors (backward-compat hygiene, separate task).
## Open questions
- MCP auth depth (decide at implementation).
- Public REST rate-limiting / abuse posture.
- Async delivery channel for `06` outputs: SSE stream vs Zero-published messages table (settled with `06`).

321
plans/backend/05-pipelines-model.md
View file

@ -1,321 +0,0 @@
# Phase 5 — Pipelines data model (tables, schemas, migration, Zero, CRUD + run routes)
> # ❌ SUPERSEDED (2026-06-30)
> **This entire phase is dropped.** Per the **Architecture correction** in `00-umbrella-plan.md`, the CI pivot no longer uses a Pipelines concept: the KB is **input-only** (user files), the crawler/search are **platform-native tools**, connectors are **MCP**, and the recurring-fetch + run-history need is met by the existing **automations** system (`app/automations/`). There are **no `pipelines`/`pipeline_runs` tables**. This document is retained **for historical context only** — do **not** implement it. **The replacement is the canonical revamp Phase 5 — [`revamp phases 4-7/05a-timeline.md`](revamp%20phases%204-7/05a-timeline.md) + [`05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md)**: the `Tracker` primitive + a 3-store delta **Timeline** (the durable CI state / moat). "Diffable run history" lives in the Timeline's `entity_changes`, **not** in `automation_runs.output`. (The earlier "Phase 5′" framing is retired.)
> Part of the **CI Pivot MVP**. See `00-umbrella-plan.md` (Phase 5).
> Precondition: Phases 1–2 (rename) live, Phase 4a (connector registry) live. Siblings ahead: `06-pipelines-exec.md` (run engine + scheduling), `07-upload-pipeline-kb.md` (uploads-as-pipeline).
> **Implementation note (post-rename).** This phase adds **brand-new** tables, created *after* the rename, so they use the canonical names natively: table `workspaces`, column `workspace_id`, ORM class `Workspace`. Phases 1–2 are now **SHIPPED**, so the live code uses the `workspace_*` names. Existing code cited below that still shows the **pre-rename** names (`searchspaces`/`search_space_id`/`SearchSpace`) was captured pre-rename — substitute the `workspace_*` equivalent and grep the new name. Locate code by **symbol/grep**, not the absolute line numbers cited here (the rename + Phase 3/4 migrations shift them).
## Objective
Introduce two first-class tables — `pipelines` and `pipeline_runs` — plus their enums, Pydantic schemas, one Alembic migration, the backend Zero-publication entries, and the HTTP routes (CRUD + manual-run trigger + list/get runs). **This phase is data-model + API surface only.** The actual run **engine** (invoke connector fetch → optional KB save → write run record), scheduling tick, crawl-billing wiring, and chat-agent context exposure are **Phase 6**; wiring file upload onto an "Uploads" pipeline is **Phase 7**. The manual-run endpoint here **creates a `PipelineRun` row and enqueues the Phase-6 task** (which is a stub until Phase 6 lands), so the surface is testable end-to-end without the engine.
A **Pipeline** = a saved, runnable fetch over a Type-1 data source: it references a connector + per-pipeline config + an optional cron schedule + a KB destination (`save_to_kb` + `destination_folder_id`). A **PipelineRun** = one immutable execution record (manual / scheduled / upload), carrying status, timing, doc counts, error, an optional raw-result blob ref, and a billing idempotency field.
## Locked model (MVP)
| Concept | Decision |
|---------|----------|
| Tables | `pipelines` (mutable definition) + `pipeline_runs` (append-only execution record). Mirrors the `automations` / `automation_runs` split. |
| Where the ORM lives | In **`app/db.py`** alongside `Folder`/`Document`/`SearchSourceConnector`/`Log` (the other core, Zero-published, cross-referenced entities), **not** a separate package. Rationale: needs `back_populates` on `Workspace`/`User`/`SearchSourceConnector`/`Folder`, a Zero entry, and a `schemas/` Pydantic mirror — all lowest-friction in `db.py`. (The `automations` package avoided `db.py` but pays for it with string-relationships + import-time model registration; pipelines are more core, like connectors.) |
| Pipeline → connector | `connector_id` **nullable**, FK `ON DELETE CASCADE`. A connector-backed pipeline points at a Type-1 source; **`NULL` = a non-connector pipeline** (the Phase-7 "Uploads" pipeline). Eligibility (`DATA_SOURCE` + `AVAILABLE`) is enforced at create via Phase 4a's `is_pipeline_eligible` (app-level, not a DB constraint — the registry is code) **and re-checked at run time in Phase 6** (the static registry can flip a type to `MIGRATING` on deploy, stranding existing pipelines). |
| Ownership | `created_by_id` FK `ON DELETE SET NULL`, **nullable** — creator metadata only (workspace-shared, like `Folder`/`Automation`), not an owner. Billing targets the workspace owner (03c). |
| Schedule representation | `schedule_cron VARCHAR NULL` (NULL = manual-only) + `schedule_timezone VARCHAR NOT NULL DEFAULT 'UTC'` + `enabled BOOLEAN` + `next_scheduled_at TIMESTAMPTZ NULL` (computed by Phase 6's tick). Chosen over a bare `frequency_minutes` int to match the **automations** schedule model and reuse its cron util (`automations/triggers/builtin/schedule/cron.py`), which takes **both** a cron and a timezone (`compute_next_fire_at(cron, timezone, …)`). |
| KB destination | `save_to_kb BOOLEAN` + `destination_folder_id` FK `ON DELETE SET NULL` (nullable). `save_to_kb=true` with NULL folder ⇒ index to workspace root (`documents.folder_id` is nullable). No DB CHECK couples them (an FK `SET NULL` would violate a conditional CHECK); the pairing is validated at the API only. |
| Run status enum | `pipeline_run_status`: `pending`, `running`, `succeeded`, `failed`, `cancelled` (mirrors `automation_run_status` minus `timed_out`). |
| Run trigger enum | `pipeline_run_trigger`: `manual`, `scheduled`, `upload`. |
| Periodic coexistence | **COEXIST, do not migrate** (resolves umbrella open item §2 — see "Periodic-indexing coexistence" below). |
| Zero publication | Publish **both tables full-row** (`None`), mirroring `folders` / `search_source_connectors`. Avoids the column-list `_0_version` seam entirely; neither table has a bulky column. |
| Naming | The CI **`Pipeline`** entity (a saved, runnable fetch) is distinct from the existing **`IndexingPipelineService`** (`app/indexing_pipeline/`, the KB-ingest service). They coexist but do **not** share a code path for the MVP: Phase 6's WebURL executor reuses the crawler's own KB-write path (`index_crawled_urls`, extended with `folder_id`), **not** `IndexingPipelineService` (which is the file/Composio ingest path — and is what Phase 7's uploads flow builds on). Name routes/services `pipeline_*` (not `indexing_pipeline_*`) to keep the two legible. |
## Current state (cited)
### The precedent: `automations` / `automation_runs`
The cleanest existing analogue to model against.
- Editable parent: `Automation` — `automations/persistence/models/automation.py:24`. Pattern: `BaseModel, TimestampMixin`; `search_space_id` FK CASCADE (`:27`), `created_by_user_id` FK SET NULL (`:34`), a native enum `status` with `values_callable=lambda x: [e.value for e in x]` (`:44–54`), a JSONB `definition` (`:56`), an explicit `updated_at` with `onupdate` (`:60–66`), and `runs = relationship(..., cascade="all, delete-orphan", passive_deletes=True)` (`:76–81`).
- Append-only child: `AutomationRun` — `automations/persistence/models/run.py:20`. `automation_id` FK CASCADE (`:23`), native enum `status` default `pending` (`:37–47`), JSONB `error` (`:60`), nullable `started_at`/`finished_at` (`:62–63`).
- Enums: `automations/persistence/enums/run_status.py:8` (`RunStatus(StrEnum)` = pending/running/succeeded/failed/cancelled/timed_out).
- Migration: `alembic/versions/144_add_automation_tables.py` — the **exact template** for this phase: idempotent `CREATE TYPE ... DO $$ IF NOT EXISTS` guards (`:31–76`), `CREATE TABLE IF NOT EXISTS` with inline FKs (`:79–96`), `CREATE INDEX IF NOT EXISTS` per FK/status/created_at (`:97–111`), a **partial "due" index** for the schedule scan (`ix_automation_triggers_due`, `:144–152`), and symmetric `downgrade()` dropping indexes→tables→types (`:190–213`).
- Run routes: `automations/api/run.py` — `GET /automations/{id}/runs` (list, paginated `limit/offset`, `:13–30`) + `GET /automations/{id}/runs/{run_id}` (detail, `:33–44`). The exact shape to mirror for pipeline runs.
### The other core tables (relationship + column conventions)
- Base: `TimestampMixin` (gives `created_at`, `db.py:487`) + `BaseModel` (`id SERIAL PK`, `db.py:498`). Tables wanting a mutable `updated_at` declare it explicitly (see `Folder.updated_at` `db.py:1333–1339`, with `onupdate` + `index=True`).
- `SearchSourceConnector` — `db.py:1829`. `connector_type` native enum (`:1867`), `is_indexable` (`:1868`), periodic fields `periodic_indexing_enabled`/`indexing_frequency_minutes`/`next_scheduled_at` (`:1880–1882`), `search_space_id` FK CASCADE (`:1884`), `user_id` FK CASCADE (`:1891`), `documents = relationship(...)` (`:1897`). **Pipelines FK this table** (`connector_id`).
- `Folder` — `db.py:1310`. `search_space_id` FK CASCADE (`:1321`), `created_by_id` FK SET NULL (`:1327`). **Pipelines FK this table** (`destination_folder_id`, SET NULL).
- `Document.folder_id` is nullable (root/unfiled docs exist) — confirmed by `DOCUMENT_COLS` including `folder_id` (`zero_publication.py:32`) and `Document.folder` `passive_deletes=True` (`db.py:1345`). Lets `save_to_kb` work with a NULL destination.
- `Permission` enum has `CONNECTORS_CREATE/READ/UPDATE/DELETE` (`db.py:345–348`); routes authz via `check_permission(session, auth, search_space_id, Permission.X.value, msg)` (`folders_routes.py:42–48`) with `auth = Depends(get_auth_context)` + `session = Depends(get_async_session)` (`folders_routes.py:36–37`). **Reuse the `CONNECTORS_*` permissions** for pipelines (no new permission needed for MVP).
### Schema conventions
- `schemas/base.py`: `IDModel` (`id:int`, `:11`) + `TimestampModel` (`created_at`, `:6`), both `ConfigDict(from_attributes=True)`.
- `schemas/search_source_connector.py`: `Base` → `Create(Base)` → `Update(BaseModel, all-Optional)` → `Read(Base, IDModel, TimestampModel)` with `model_config = ConfigDict(from_attributes=True)` and `field_validator`/`model_validator` for config consistency (`:24–56`). The shape to mirror for `schemas/pipeline.py`.
### Zero publication mechanics
- `zero_publication.py`: `ZERO_PUBLICATION` map (`:81–94`; line numbers post-`main`-merge, which added `automations`/`new_chat_threads` entries) is the single source of truth; `None` ⇒ publish full row (e.g. `folders`, `search_source_connectors`), a list ⇒ column subset (e.g. `AUTOMATION_RUN_COLS` `:44–53`). `apply_publication(conn)` (`:163`) reconciles via `ALTER PUBLICATION ... SET TABLE`; a migration just calls it (template: `159_publish_podcasts_to_zero.py:21–22`). `_format_table_entry` omits a table until it physically exists with all its canonical columns (`:125–152`), so the migration **must create the tables before** calling `apply_publication`. The `_0_version` allowlist (`{"documents","user","podcasts"}`, `:118`) applies **only to column-list tables** — irrelevant here since we publish full-row.
### The meta-scheduler we coexist with (Phase-6 reuse target)
- `tasks/celery_tasks/schedule_checker_task.py:17` — `check_periodic_schedules_task`, runs every minute, scans `SearchSourceConnector` where `periodic_indexing_enabled AND next_scheduled_at <= now` (`:33–39`) and dispatches per-type Celery tasks. This is the **connector-level** periodic path and is **untouched** by Phase 5. Phase 6 adds a sibling scan over `pipelines.next_scheduled_at`.
- `utils/periodic_scheduler.py:30` — `create_periodic_schedule` (first-run trigger helper). Reference for Phase 6's pipeline scheduler.
### Alembic head
As of the latest `main` sync the head is **`169`** (`alembic/versions/169_migrate_google_oauth_account_ids_to_sub.py`; chain `166→167→168→169`, sequential integer-prefixed files). By the time Phase 5 is implemented, Phase 1 (rename) and Phase 4b (search-enum drop) have each added a migration ahead of `169`. **Set `down_revision` to the then-current head** — verify with `alembic heads`; do not hardcode a number.
## Target design
### 1. Enums (`db.py`, near the other `StrEnum`s)
```python
class PipelineRunStatus(StrEnum):
PENDING = "pending"
RUNNING = "running"
SUCCEEDED = "succeeded"
FAILED = "failed"
CANCELLED = "cancelled"
class PipelineRunTrigger(StrEnum):
MANUAL = "manual"
SCHEDULED = "scheduled"
UPLOAD = "upload"
```
Mapped with native PG enum types `pipeline_run_status` / `pipeline_run_trigger` using the `values_callable=lambda x: [e.value for e in x]` convention (matches `Automation.status` `automation.py:46–49`, so the DB stores lowercase values, not the Python member names).
### 2. `pipelines` table (`db.py`)
```python
class Pipeline(BaseModel, TimestampMixin):
__tablename__ = "pipelines"
name = Column(String(200), nullable=False, index=True)
config = Column(JSONB, nullable=False, default=dict, server_default=text("'{}'::jsonb")) # per-pipeline overrides (URL list, crawl opts, proxy override seam)
save_to_kb = Column(Boolean, nullable=False, default=False, server_default="false")
# Schedule (NULL cron = manual-only). next_scheduled_at is owned/written by Phase 6's tick.
schedule_cron = Column(String(120), nullable=True)
schedule_timezone = Column(String(64), nullable=False, default="UTC", server_default="UTC") # cron util needs a tz (compute_next_fire_at)
enabled = Column(Boolean, nullable=False, default=True, server_default="true")
next_scheduled_at = Column(TIMESTAMP(timezone=True), nullable=True)
# NULL connector_id = non-connector pipeline (Phase-7 Uploads).
connector_id = Column(Integer, ForeignKey("search_source_connectors.id", ondelete="CASCADE"), nullable=True, index=True)
destination_folder_id = Column(Integer, ForeignKey("folders.id", ondelete="SET NULL"), nullable=True, index=True)
workspace_id = Column(Integer, ForeignKey("workspaces.id", ondelete="CASCADE"), nullable=False, index=True)
# Creator metadata, NOT owner: pipelines are workspace-shared (like folders/automations),
# so member deletion must NOT nuke them — SET NULL, nullable. Billing targets the workspace
# owner (03c), not this field. Phase 6 resolves the acting user as created_by ?? workspace owner.
created_by_id = Column(UUID(as_uuid=True), ForeignKey("user.id", ondelete="SET NULL"), nullable=True, index=True)
updated_at = Column(TIMESTAMP(timezone=True), nullable=False,
default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC), index=True)
connector = relationship("SearchSourceConnector", back_populates="pipelines")
destination_folder = relationship("Folder")
workspace = relationship("Workspace", back_populates="pipelines")
created_by = relationship("User", back_populates="pipelines")
runs = relationship("PipelineRun", back_populates="pipeline", cascade="all, delete-orphan", passive_deletes=True)
```
Add the inverse `pipelines = relationship("Pipeline", back_populates=...)` on **`Workspace`**, **`User`**, and **`SearchSourceConnector`** (mirror how `SearchSpace.search_source_connectors` / `User.search_source_connectors` are declared at `db.py:1887–1894`; the `created_by`/SET-NULL side mirrors `Folder.created_by` `db.py:1344` and `Automation.created_by` `automation.py:69`). `destination_folder` is intentionally one-directional (no back_populates on `Folder`) to keep the folder model lean.
> **Ownership decision (review).** Earlier draft used connector-style `user_id NOT NULL CASCADE`. Corrected to `created_by_id … SET NULL` because pipelines are **workspace-shared** content (closer to `Folder`/`Automation` than to the per-user `SearchSourceConnector`): removing a member must not delete the workspace's pipelines or erase run-history audit. For *connector-backed* pipelines this also avoids surprising double-cascade (the connector's own `user_id CASCADE` would already drop them on member deletion via `connector_id`); for the Phase-7 Uploads pipeline (`connector_id` NULL) it's the only thing keeping the pipeline alive after its creator leaves.
> **FK target naming:** Phases 1–2 are **SHIPPED**, so the table is `workspaces` and the column `workspace_id` — author the FK against `workspaces` / `workspace_id`. (The earlier "if sequencing slips before the rename" caveat is now moot.)
### 3. `pipeline_runs` table (`db.py`)
```python
class PipelineRun(BaseModel, TimestampMixin):
__tablename__ = "pipeline_runs"
pipeline_id = Column(Integer, ForeignKey("pipelines.id", ondelete="CASCADE"), nullable=False, index=True)
status = Column(SQLAlchemyEnum(PipelineRunStatus, name="pipeline_run_status",
values_callable=lambda x: [e.value for e in x]),
nullable=False, default=PipelineRunStatus.PENDING,
server_default=PipelineRunStatus.PENDING.value, index=True)
trigger = Column(SQLAlchemyEnum(PipelineRunTrigger, name="pipeline_run_trigger",
values_callable=lambda x: [e.value for e in x]), nullable=False, index=True)
# Result accounting (written by Phase 6).
documents_indexed = Column(Integer, nullable=True)
crawls_attempted = Column(Integer, nullable=True)
crawls_succeeded = Column(Integer, nullable=True)
error = Column(JSONB, nullable=True)
# Raw fetch blob ref for save_to_kb=false runs (Phase 6 writes via file_storage). Just a key/path string.
result_blob_key = Column(String, nullable=True)
# Crawl-billing idempotency (carry-over from 03c / umbrella Phase 6 §105): micro-USD charged for this run.
charged_micros = Column(BigInteger, nullable=True)
started_at = Column(TIMESTAMP(timezone=True), nullable=True)
finished_at = Column(TIMESTAMP(timezone=True), nullable=True)
pipeline = relationship("Pipeline", back_populates="runs")
```
These extra columns (`crawls_*`, `result_blob_key`, `charged_micros`) are added **now** so Phase 6 needs **no second migration** — they sit unused until the engine writes them. (`charged_micros` directly satisfies umbrella Phase 6 line 105's "record `charged_micros` on the `PipelineRun` for idempotency".)
> **Out of scope (provenance):** linking individual `documents` back to the `pipeline_run` that produced them (a `documents.pipeline_run_id` column) is **not** in this phase — it would touch the existing `documents` table. Defer to Phase 6/7 if run-level provenance is wanted.
### 4. Migration (one file, `144`-shaped)
Chain after the then-current head. In `upgrade()`:
1. `CREATE TYPE pipeline_run_status` + `pipeline_run_trigger` behind `DO $$ IF NOT EXISTS` guards (copy `144:31–76`).
2. `CREATE TABLE IF NOT EXISTS pipelines (...)` with inline FKs: `workspace_id → workspaces(id) ON DELETE CASCADE` (NOT NULL), `connector_id → search_source_connectors(id) ON DELETE CASCADE` (nullable), `destination_folder_id → folders(id) ON DELETE SET NULL` (nullable), `created_by_id → "user"(id) ON DELETE SET NULL` (nullable; note the quoted `"user"` table). `id SERIAL PK`, `created_at`/`updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()`, `config JSONB NOT NULL DEFAULT '{}'::jsonb`, `schedule_timezone VARCHAR(64) NOT NULL DEFAULT 'UTC'`.
3. `CREATE TABLE IF NOT EXISTS pipeline_runs (...)`.
4. Indexes (`CREATE INDEX IF NOT EXISTS`): per FK (`workspace_id`, `created_by_id`, `connector_id`, `destination_folder_id`, `pipeline_id`), plus `status`, `created_at`, `updated_at`, `name`, `trigger`. Add the **partial "due" index** for Phase 6's tick:
```sql
CREATE INDEX IF NOT EXISTS ix_pipelines_due
ON pipelines (next_scheduled_at)
WHERE enabled = true AND next_scheduled_at IS NOT NULL;
```
5. `from app.zero_publication import apply_publication; apply_publication(op.get_bind())` **after** the tables exist (so `_format_table_entry` includes them). Requires the `ZERO_PUBLICATION` code change (§6) to be in the tree.
`downgrade()`: drop indexes → `pipeline_runs` → `pipelines` → both types (symmetric to `144:190–213`). (The publication reconcile is intentionally not reversed — historical publication shapes are immutable, per `159:25–27`.)
### 5. Pydantic schemas (`app/schemas/pipeline.py`, exported via `app/schemas/__init__.py`)
```python
class PipelineBase(BaseModel):
name: str
connector_id: int | None = None
config: dict[str, Any] = {}
save_to_kb: bool = False
destination_folder_id: int | None = None
schedule_cron: str | None = None
schedule_timezone: str = "UTC"
enabled: bool = True
class PipelineCreate(PipelineBase):
workspace_id: int # required on create (path/body), like FolderCreate.search_space_id
class PipelineUpdate(BaseModel): # all-Optional, like SearchSourceConnectorUpdate
name: str | None = None
config: dict[str, Any] | None = None
save_to_kb: bool | None = None
destination_folder_id: int | None = None
schedule_cron: str | None = None
schedule_timezone: str | None = None
enabled: bool | None = None
# connector_id is immutable after create (re-create to re-point); not in Update.
class PipelineRead(PipelineBase, IDModel, TimestampModel):
workspace_id: int
created_by_id: uuid.UUID | None = None
next_scheduled_at: datetime | None = None
updated_at: datetime
model_config = ConfigDict(from_attributes=True)
class PipelineRunRead(IDModel, TimestampModel):
pipeline_id: int
status: PipelineRunStatus
trigger: PipelineRunTrigger
documents_indexed: int | None = None
crawls_attempted: int | None = None
crawls_succeeded: int | None = None
error: dict[str, Any] | None = None
charged_micros: int | None = None
started_at: datetime | None = None
finished_at: datetime | None = None
model_config = ConfigDict(from_attributes=True)
class PipelineRunList(BaseModel):
items: list[PipelineRunRead]
total: int
```
**Validators:**
- `schedule_cron` + `schedule_timezone` (when cron not None): validate together with the existing automations cron util `validate_cron(cron, timezone)` (`automations/triggers/builtin/schedule/cron.py:15`) so a bad expression **or** bad IANA timezone 422s at create, not at the Phase-6 tick. (`schedule_timezone` is consumed by Phase 6's `compute_next_fire_at`; default `"UTC"`.)
- The `save_to_kb` ⇄ `destination_folder_id` pairing and **connector eligibility** are validated in the **route** (they need a DB session: folder-in-workspace check + `is_pipeline_eligible(connector.connector_type)`), not in the pure schema validator — mirroring how `folders_routes` validates parent-in-search-space in the handler (`folders_routes.py:54–58`).
### 6. Zero publication (`zero_publication.py`)
Add to `ZERO_PUBLICATION` (`:81–94`), both **full-row**:
```python
"pipelines": None,
"pipeline_runs": None,
```
Full-row (like `folders`/`search_source_connectors`) — no `_0_version` allowlist edit needed, no column-drift migrations later. (If a bulky column is ever added and must be excluded, switch that table to an explicit COLS list and handle the `_0_version` seam at `:118` then.) `verify_publication` will then expect both tables; the migration's `apply_publication` call (§4) reconciles them.
> **Client sync is not live until the frontend lands (safe).** Publishing backend-side only makes the rows *available* to Zero; nothing syncs to a client until the (deferred) frontend Zero schema + permissions include these tables. Publishing now is harmless and matches the existing pattern (e.g. `automation_runs` was published by migration before/independent of its UI). This entry exists so the chat-agent run-history context (Phase 6) and a later Pipelines UI get push for free.
### 7. Routes (`app/routes/pipelines_routes.py`, registered in `routes/__init__.py`)
Follow `folders_routes` for authz (`check_permission` + `get_auth_context` + `get_async_session`) and `automations/api/run.py` for the run-list shape. Register the router in `routes/__init__.py` near `search_source_connectors_router` (`:105`).
| Method + path | Behaviour | AuthZ |
|---------------|-----------|-------|
| `POST /pipelines` | Create. Validate: workspace access; if `connector_id`, load it, assert same workspace + `is_pipeline_eligible` (4xx otherwise); if `destination_folder_id`, assert folder in same workspace; `save_to_kb=true` allowed with NULL folder (→ root). Set `created_by_id = auth.user.id`. Persist; return `PipelineRead`. | `CONNECTORS_CREATE` |
| `GET /pipelines?workspace_id=` | List pipelines in a workspace. | `CONNECTORS_READ` |
| `GET /pipelines/{id}` | Read one. | `CONNECTORS_READ` |
| `PUT /pipelines/{id}` | Update (all-optional). Re-validate folder-in-workspace + cron if changed. | `CONNECTORS_UPDATE` |
| `DELETE /pipelines/{id}` | Delete (runs cascade). | `CONNECTORS_DELETE` |
| `POST /pipelines/{id}/run` | **Manual trigger.** Insert a `PipelineRun(status=pending, trigger=manual)`, enqueue the Phase-6 Celery task with the run id, return the run. Until Phase 6, the task is a stub (no-op/marks failed) — the row + enqueue path are what this phase delivers. (Concurrency control — block a second run while one is in flight — is deferred to Phase 6, reusing the connector indexing Redis locks `utils/indexing_locks.py`.) | `CONNECTORS_UPDATE` |
| `GET /pipelines/{id}/runs?limit=&offset=` | List runs newest-first, paginated (`limit` 1–200 default 50, `offset` ≥0) → `PipelineRunList`. Mirror `automations/api/run.py:13–30`. | `CONNECTORS_READ` |
| `GET /pipelines/{id}/runs/{run_id}` | Run detail. | `CONNECTORS_READ` |
All handlers scope by the pipeline's `workspace_id` and 404 on cross-workspace access. A thin `PipelineService` (mirroring `automations` `RunService`) is optional; for MVP, inline session logic in the routes (as `folders_routes` does) is acceptable.
## Periodic-indexing coexistence (resolves umbrella open item §2)
**Decision: COEXIST for MVP — do NOT migrate connector periodic config into pipelines.**
- The existing `SearchSourceConnector.periodic_indexing_enabled` / `next_scheduled_at` path and its meta-scheduler (`schedule_checker_task.py`) **stay fully functional and untouched** in Phases 5–6. Pipelines are **purely additive**: a new `pipelines.next_scheduled_at` scan (Phase 6) runs **alongside** the connector scan.
- **Rationale:** (a) lowest risk + smallest diff (no backfill of existing connector schedules into pipeline rows, no data migration of live schedules); (b) only the WebURL crawler is the MVP pipeline executor (Phase 6) — file sources keep using their connector-level periodic path; (c) consistent with the umbrella posture "DB migrations carry users; backend behaviour can change incrementally."
- **Known overlap (flagged for Phase 6, not solved here):** a single `WEBCRAWLER_CONNECTOR` could have BOTH connector-level periodic indexing AND a pipeline wrapping it → double crawl + **double bill**. The data model permits it; the *guard* is Phase 6's responsibility. **Recommendation for Phase 6:** when a pipeline is created/enabled over a connector, treat the **pipeline as authoritative** and set that connector's `periodic_indexing_enabled=False` (single scheduler owns each connector). Recorded here so Phase 6 implements the de-dup; Phase 5 only needs the columns to support either path.
## Work items
1. **Enums** `PipelineRunStatus` + `PipelineRunTrigger` in `db.py`.
2. **ORM** `Pipeline` + `PipelineRun` in `db.py`; inverse `pipelines` relationships on `Workspace`, `User`, `SearchSourceConnector`.
3. **Migration**: 2 types + 2 tables + indexes (incl. `ix_pipelines_due`) + `apply_publication`; symmetric downgrade.
4. **Schemas** `app/schemas/pipeline.py` (+ export in `schemas/__init__.py`); cron syntactic validator.
5. **Zero**: add `pipelines`/`pipeline_runs` (full-row) to `ZERO_PUBLICATION`.
6. **Routes** `pipelines_routes.py` (CRUD + `/run` + `/runs` list/detail); register in `routes/__init__.py`; reuse `CONNECTORS_*` permissions.
7. **Phase-6 task stub**: a named Celery task `run_pipeline(run_id)` in **`app/pipelines/tasks.py`** (the module Phase 6 fleshes out) that `/run` enqueues — no-op/marks the run `failed` with an "engine not implemented" error until Phase 6 fills it in (keeps the endpoint honest and testable). Register it so it's dispatchable: add `"app.pipelines.tasks"` to the Celery `include` list (`celery_app.py`). Phase 6 replaces the stub body in-place (same task name + module → `/run`'s import is stable) and adds the scheduler task + Beat entry + queue routing.
8. **Tests** (below).
## Tests
- **Migration round-trip**: upgrade creates both tables + both enum types + `ix_pipelines_due`; `zero_publication --verify` reports no mismatch (both tables published full-row); downgrade drops cleanly.
- **Create gating (cross-link to 04a)**: `POST /pipelines` with an `AVAILABLE` `DATA_SOURCE` connector (WebURL/GDrive/OneDrive/Dropbox) → 201; with a `MIGRATING`/`MCP_TOOL`/`DISABLED` connector → 4xx (`is_pipeline_eligible` false); with `connector_id=None` → allowed (Uploads case). **(Superseded in Phase 7: once the system-managed Uploads pipeline exists, user `POST` with `connector_id=None` is rejected 4xx and the Uploads row is auto-created on upload — see `07` §7. Update this test in Phase 7.)**
- **Folder validation**: `destination_folder_id` in another workspace → 4xx; `save_to_kb=true` with NULL folder → allowed.
- **Cron validation**: invalid `schedule_cron` → 422; NULL cron → allowed (manual-only).
- **Cascade**: deleting a connector deletes its pipelines + their runs; deleting a folder SET-NULLs `destination_folder_id` (pipeline survives); deleting a workspace removes pipelines + runs.
- **Manual run**: `POST /pipelines/{id}/run` inserts a `pending`/`manual` run and enqueues the task; `GET /pipelines/{id}/runs` returns it newest-first with correct `total`.
- **AuthZ**: cross-workspace access to any pipeline/run route → 404; missing `CONNECTORS_*` permission → 403.
- **Zero shape**: `expected_publication_shape` includes `pipelines`/`pipeline_runs` as full-row.
## Risks / trade-offs
- **`db.py` growth.** Adds two more models to an already-large module. Accepted for relationship/Zero/schema ergonomics + consistency with the other core tables; a later extraction to a `pipelines` package is additive.
- **Coexisting schedulers (double-bill window).** Until Phase 6 adds the de-dup guard, a connector with both its own periodic indexing and a wrapping pipeline can crawl/bill twice. Documented above; Phase 6 owns the fix. MVP exposure is small (operator-created overlap only).
- **Eligibility enforced in code, not SQL.** `is_pipeline_eligible` is a registry (04a) check at the route, so a direct DB insert could bypass it. Same trade-off 04a already accepted (no DB column for the taxonomy).
- **Pre-built unused columns.** `crawls_*` / `result_blob_key` / `charged_micros` ship empty in Phase 5. Deliberate — avoids a second migration in Phase 6.
- **`connector_id` CASCADE drops run history.** Deleting a connector erases its pipelines' audit trail. If audit retention becomes a requirement, switch to `SET NULL` + a discriminator to distinguish Uploads from orphaned (additive change).
- **`created_by_id` nullable shifts work to Phase 6.** SET-NULL preserves shared pipelines past member deletion (the goal) but means the run engine cannot assume a creator — it must resolve `created_by ?? workspace owner` for the indexing actor + billing target. Flagged in the Phase-6 handoff; harmless until then (no engine).
- **Static-registry strand.** A pipeline created over an `AVAILABLE` connector can later reference a type the registry flips to `MIGRATING` on a deploy (create-time check passed; the row persists). Phase 6 must re-check `is_pipeline_eligible` at run time and fail the run cleanly. The data model intentionally does not encode eligibility, so no migration is needed when the registry changes.
## Out of scope (hand-offs)
- Run **engine**, scheduling tick over `pipelines.next_scheduled_at`, crawl-billing wiring (write `charged_micros`/`crawls_*`), raw-blob persistence, and chat-agent run-history context → **Phase 6** (`06-pipelines-exec.md`). Phase 6 also owns three guards this model only *enables*: (a) **run-time eligibility re-check** (`is_pipeline_eligible` — a deploy can flip a connector type to `MIGRATING`, stranding existing pipelines → fail the run cleanly, don't crash); (b) **acting-user resolution** (`created_by_id` can be NULL after member deletion → resolve `created_by ?? workspace owner` for indexing + billing); (c) **concurrency** (Redis indexing lock per pipeline/connector).
- File upload creating/using an "Uploads" pipeline (`connector_id=NULL`, `trigger=upload`, always `save_to_kb`) + generalizing opt-in KB save → **Phase 7** (`07-upload-pipeline-kb.md`).
- Document→run provenance column → deferred (Phase 6/7 if needed).
- Frontend Pipelines UI (list/create/configure/run-history/manual run) → frontend umbrella.
- Public pay-as-you-go API over Type-1 pipelines → post-MVP (umbrella "Deferred").

25
plans/backend/06-ongoing-automation.md Normal file
View file

@ -0,0 +1,25 @@
# Phase 6 — Ongoing Automation (chat-native "keep watching")
> **Design deferred — placeholder.** The mechanism is designed separately, after `04`/`05`/`07`.
> Depends on `04` (the verbs it re-invokes) and `05` (the chat surface + delivery channel).
## Objective
Support "keep watching": a persistent, ongoing chat where the agent periodically re-invokes scraper verbs
and drops results into the session. The agent derives what's new by reading the chat history (time-based
search over prior tool outputs in context).
## Open design questions (resolve together)
1. **Periodicity driver** — the existing automations schedule selector, a recurring task, or a persistent
agent loop.
2. **Delivery channel** for between-turn results — existing SSE stream vs a Zero-published messages table.
3. **Context-window limit** — how far back "what changed" can reason before summarization/compaction.
4. **Loop owner** — the `07` subagent, or a thin automation wrapper that invokes the agent.
5. **Stop / pause / cost controls** for a running watch.
## Out of scope
- The verbs → `04`. The chat surface + delivery → `05`. The agent playbook → `07`.
> Next step: design the periodic mechanism here, then fill in Target design / Work items / Tests.

350
plans/backend/06-pipelines-exec.md
View file

@ -1,350 +0,0 @@
# Phase 6 — Pipeline execution + scheduling (run engine, crawl billing, blob, chat context)
> # ❌ SUPERSEDED (2026-06-30)
> **This entire phase is dropped.** Per the **Architecture correction** in `00-umbrella-plan.md`, there is no pipeline run engine or scheduler. The existing **automations** system already runs the full chat agent (with the native `web_search`/`scrape_webpage` tools) on **cron/event** triggers and persists **`automation_runs`** — which is exactly what this phase was building. Retained **for historical context only** — do **not** implement it. **Its still-relevant ideas survive in the canonical revamp:** (1) **billing crawls done outside chat** → now handled at the **capability executor** ([`revamp/04a`](revamp%20phases%204-7/04a-capabilities.md)), so recurring/automation runs meter correctly; (2) the refresh **hot loop** (crawl → diff → judge → append) → [`revamp/05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md), writing the **Timeline** ([`revamp/05a`](revamp%20phases%204-7/05a-timeline.md)) — not `automation_runs.output`; (3) **recurrence + "run now"** → [`revamp/06-triggers.md`](revamp%20phases%204-7/06-triggers.md) (reuse automations via a CI action). The `bill=False` seam and `index_crawled_urls` `folder_id`/`stats` params described below are **not needed** (no pipeline path).
> Part of the **CI Pivot MVP**. See `00-umbrella-plan.md` (Phase 6).
> Precondition: Phase 5 (`05-pipelines-model.md`) live — `pipelines` / `pipeline_runs` tables, schemas, Zero, CRUD + `/run` (currently enqueues a **stub** `run_pipeline(run_id)`). Depends on `03a-crawler-core.md` (`crawl_url` SUCCESS signal + `crawls_succeeded` counter) and `03c-crawl-billing.md` (`WebCrawlCreditService`). Sibling ahead: `07-upload-pipeline-kb.md` (uploads-as-pipeline).
> **Implementation note (post-rename).** Phases 1–2 are **SHIPPED**, so the live code already uses `workspace_id` / `Workspace` / `workspaces`. Citations below that still show `search_space_id` / `SearchSpace` / `searchspaces` are pre-rename — substitute the `workspace_*` equivalent and grep the new name. **New code in this plan uses the canonical `workspace_*` names** (snippets below updated; e.g. the `get_pipeline_runs` tool's dep key is `workspace_id`, matching the renamed deps dict). Locate code by **symbol/grep**, not the absolute line numbers cited here (the rename + Phases 3–5 shift them).
## Objective
Make pipelines actually **run**. Phase 5 shipped the data model + a stub task; this phase fills in:
1. **Run engine** — turn a `PipelineRun` row into work: resolve the pipeline + connector, re-check eligibility, acquire a lock, crawl the source (WebURL crawler = the only MVP executor), optionally save to the KB destination folder, persist a raw-result blob otherwise, write the run record (status/timing/counts/error).
2. **Crawl billing wiring** (carry-over from `03c`) — bill `crawls_succeeded` to the **workspace owner** via `WebCrawlCreditService` for **both** KB-save branches (a `save_to_kb=false` run must not crawl for free); record `charged_micros` on the run for idempotency.
3. **Scheduling** — a Celery Beat tick that fires due `pipelines.next_scheduled_at` (cron), modeled on the **automations** schedule selector (cron + `FOR UPDATE SKIP LOCKED` + self-heal). Plus the **de-dup guard** Phase 5 deferred (a pipeline over a connector disables that connector's own periodic indexing).
4. **Chat-agent context** — a read-only main-agent **tool** (`get_pipeline_runs`) exposing recent run history scoped to the active workspace (umbrella default = tool).
This is backend-only; the Pipelines UI is deferred to the frontend umbrella.
## Locked decisions (MVP)
| Concept | Decision |
|---------|----------|
| MVP executor | **WebURL crawler only.** `connector.connector_type == WEBCRAWLER_CONNECTOR`. Any other Type-1 type (GDrive/OneDrive/Dropbox) → run fails cleanly with `"executor not implemented for <type>"` (their connector-level periodic path still works; pipelines over them are a Phase-7+ concern). `connector_id IS NULL` (Phase-7 Uploads) → not run by this engine (uploads register their own run in Phase 7). |
| Run-engine shape | Mirror **automations**: thin Celery task → `run_async_celery_task` → async `execute_pipeline_run(session, run_id)` (`automations/tasks/execute_run.py:19-33`). PENDING-gated + idempotent terminal no-op (`automations/runtime/executor.py:29-30`). Status lifecycle `pending → running → succeeded/failed` with `started_at`/`finished_at` (mirrors `repository.mark_running/mark_succeeded/mark_failed`). |
| KB-save path | **Reuse `index_crawled_urls`** (the battle-tested 2-phase indexer) for `save_to_kb=true`, extended with a `folder_id` param (set on created/updated `Document`s → destination folder). `save_to_kb=true` + NULL folder → root (existing behaviour; `documents.folder_id` nullable). |
| Non-KB path | `save_to_kb=false` → a **fetch-only** crawl loop (no `Document` rows): crawl each URL, collect `{url, content, metadata}`, persist as **one JSON blob** via `file_storage` backend, store `result_blob_key` on the run. (The 2-phase indexer is `Document`-centric and cannot "not persist"; a separate small loop is cleaner than bending it.) |
| Billing ownership | **Run engine owns billing for the pipeline path** (pre-check on `len(urls)` + charge on `crawls_succeeded` + record `charged_micros`), calling the crawler with **billing suppressed** (`bill=False`). The connector `/index` + connector-periodic paths keep `03c`'s in-indexer billing (`bill=True`). This keeps `charged_micros` run-idempotency clean and bills both KB branches identically. **(03c coordination — see below.)** |
| Scheduler | New `pipeline_schedule_select` Beat task modeled on `automations/triggers/builtin/schedule/selector.py` (cron, `FOR UPDATE SKIP LOCKED`, self-heal of NULL `next_scheduled_at`), **not** the simpler connector `frequency_minutes` checker — because Phase 5 chose `schedule_cron`. Reuses the existing `croniter` util (`automations/triggers/builtin/schedule/cron.py`). |
| Schedule timezone | Cron needs a timezone (`compute_next_fire_at(cron, timezone, …)`). Phase 5's model has only `schedule_cron`. **Add `schedule_timezone VARCHAR NOT NULL DEFAULT 'UTC'` to `pipelines`** (small additive amendment to `05` — see "Required 05 amendment"). Matches how automations store cron **and** timezone (`schedule/selector.py:86-89`). |
| Concurrency | Per-pipeline Redis lock + (when connector-backed) the **existing connector lock** `utils/indexing_locks.py` so a pipeline run and any residual connector index can't double-crawl the same connector. |
| Chat context | A main-agent **tool** `get_pipeline_runs` (registry pattern, `main_agent/tools/registry.py:85-102`), opening its own `shielded_async_session()` (like `KnowledgeTreeMiddleware`) and scoping to the build-time `workspace_id`. Always-on middleware injection is **deferred** (token cost; tool is on-demand). |
| Code location | New `app/pipelines/` package: `engine.py` (`execute_pipeline_run`), `tasks.py` (Celery `run_pipeline` + `pipeline_schedule_select`), `scheduler.py` (the tick), `storage.py` (blob key). The Phase-5 stub `run_pipeline` is **replaced** by the real task here; the `/run` route + scheduler enqueue it. (ORM stays in `db.py` per Phase 5; only the *engine* is a package, mirroring `app/automations/`.) |
## Current state (cited)
### Run-engine precedent (automations)
- **Task wrapper**: `automations/tasks/execute_run.py:19-33` — `@celery_app.task(name=..., bind=True)` → `run_async_celery_task(lambda: _impl(run_id))`; `_impl` opens `get_celery_session_maker()()` and calls `execute_run(session, run_id)`, rolling back + re-raising on failure.
- **Launch**: `automations/dispatch/launch.py:43-60` — create the child row `status=PENDING`, `session.add` + `commit` + `refresh`, then `task.apply_async(args=[run.id], time_limit=…)`. (Phase 5's `/run` already does the create+enqueue; this phase makes the task real.)
- **Executor lifecycle**: `automations/runtime/executor.py:23-75` — `load run; if run.status != PENDING: return` (idempotent terminal no-op, `:29-30`); `mark_running` + `commit` (`:46-47`); on bad snapshot → `mark_failed` + commit (`:35-44`); terminal `mark_succeeded`/`mark_failed` each `commit`.
### The MVP executor: `webcrawler_indexer.index_crawled_urls`
- Signature `webcrawler_indexer.py:44-53`: `(session, connector_id, search_space_id, user_id, start_date=None, end_date=None, update_last_indexed=True, on_heartbeat_callback=None) -> (int, str|None)`.
- Reads URLs from `connector.config["INITIAL_URLS"]` via `parse_webcrawler_urls` (`:118-119`).
- **Creates `Document` rows with NO `folder_id`** (`:222-239`) → today everything lands at workspace root. Updates existing docs in place. Crawls via `crawler.crawl_url(url)` (`:297`; `03a` changes this signature/return). Tracks `documents_indexed` / `documents_updated` / counts; final commit at `:432`; returns `(total_processed, error)` (`:477`).
- **No billing today** — `03c` adds the owner pre-check + `charge_credits` inside this function (`03c` §2). **This phase adds a `bill: bool = True` seam** so the pipeline path can suppress it (plus `folder_id` / `urls` params + a `stats` out-param — see "Crawler changes"; the positional return is left unchanged so the shared wrapper keeps working).
- Connector entrypoint chain (the path the pipeline engine **bypasses**, because it's notification-centric, not run-centric): `connector_tasks.py:418-454` (`index_crawled_urls_task` → `_index_crawled_urls` → `run_web_page_indexing`) → a `_run_indexing_with_notifications(...)` **call site** (e.g. the WebCrawler one at `search_source_connectors_routes.py:~2524`). The wrapper itself is **defined at `:1296`** (acquires the connector lock at `:1334`, length-unpacks the indexer return at `:1503`).
### KB-ingest service (folder support already exists there)
- `IndexingPipelineService` (`indexing_pipeline/indexing_pipeline_service.py:82`) already threads a destination folder via `ConnectorDocument.folder_id` (`:273-274`, `:301-302`, `:326`). **But the webcrawler indexer does not use this service** — it writes `Document`s directly. So the umbrella's "route through `IndexingPipelineService` into the destination folder" is accurate for **file/Composio** sources, **not** the crawler. For the WebURL MVP, the minimal correct move is to thread `folder_id` into `index_crawled_urls` (not refactor the crawler onto `IndexingPipelineService`). Unifying them is a post-MVP cleanup.
### Scheduler precedent (cron) + Beat registration
- **Selector** `automations/triggers/builtin/schedule/selector.py`: tick (`:47-65`) → self-heal NULL `next_fire_at` (`:68-98`) → claim due rows `FOR UPDATE SKIP LOCKED` + advance via `compute_next_fire_at` + set `last_fired_at` (`:101-150`) → `_start_one` launches each (`:153-183`). `_TICK_BATCH=200` cap (`:35`).
- **Cron util** `automations/triggers/builtin/schedule/cron.py`: `validate_cron(cron, timezone)` (`:15`) and `compute_next_fire_at(cron, timezone, *, after)` (`:28`, returns UTC). **Requires a timezone** → drives the 05 amendment.
- **Beat source** `automations/triggers/builtin/schedule/source.py:14-20` — a `BEAT_SCHEDULE` dict (`crontab(minute="*")`) merged into the app at `celery_app.py:323` (`**SCHEDULE_BEAT_SCHEDULE`). The app's beat dict is `celery_app.py:268-324`; the worker `include` list is `:180-198`; task routing (`index_crawled_urls → CONNECTORS_QUEUE`) is `:237-255` (`:246`).
- **Connector meta-scheduler we coexist with**: `schedule_checker_task.py:17` scans `SearchSourceConnector` where `periodic_indexing_enabled AND next_scheduled_at <= now` (`:33-39`), with a Redis-lock guard `is_connector_indexing_locked` (`:93`). Untouched; the pipeline tick is a sibling.
### Locks, storage, chat tooling
- **Locks** `utils/indexing_locks.py`: `acquire_connector_indexing_lock(connector_id)->bool` (`:24`), `release_…` (`:37`), `is_…_locked` (`:43`), keyed `indexing:connector_lock:{id}`, TTL `CONNECTOR_INDEXING_LOCK_TTL_SECONDS`.
- **Storage** `file_storage/service.py:39,46`: `get_storage_backend()` (from `file_storage/factory.py`) → `backend.put(key, data, content_type=…)` / `open_stream(key)` / `delete(key)`. Keys are built per-purpose (`file_storage/keys.py:11`, `build_document_file_key`). For runs we add a sibling key builder (no `document_files` row — runs aren't documents).
- **Tool registry** `main_agent/tools/registry.py:85-102` — `_MAIN_AGENT_TOOL_FACTORIES: {name: (factory, required_dep_names)}`; `build_main_agent_tools(deps, …)` resolves deps + builds (`:105-147`). Display metadata lives in `shared/tools/catalog`; the main-agent name list in `main_agent/tools/index` (per the registry docstring `:1-14`). Read-only DB pattern for a chat surface: `KnowledgeTreeMiddleware` opens `shielded_async_session()` and filters by its build-time `search_space_id` (`knowledge_tree/middleware.py:199-206`).
- **Workspace owner** (billing target): `SearchSpace.user_id` (`03c` §2.1 resolves the owner with a direct `select SearchSpace.user_id where id == search_space_id`).
## Target design
### 1. `execute_pipeline_run(session, run_id)` — the engine (`app/pipelines/engine.py`)
```
load run (pipeline + connector eager); if run is None: raise
if run.status != PENDING: return # status gate; also no-ops acks_late redelivery (executor.py:29-30)
pipeline = run.pipeline
mark_running(run): status=RUNNING, started_at=now; commit
# ^ committed BEFORE any crawl/charge → a redelivered/duplicate task sees RUNNING and returns (executor.py:46-47)
pipeline_lock = connector_lock = False
connector = None
try:
# --- guards: fail cleanly, never crash the worker (Phase-5 hand-offs (a)/(b)) ---
if pipeline.connector_id is None: # Uploads pipeline — Phase 7, not here
return fail(run, "uploads pipeline has no fetch executor")
connector = pipeline.connector
if not is_pipeline_eligible(connector.connector_type): # 04a runtime re-check (strand guard)
return fail(run, f"connector type {connector.connector_type} no longer eligible")
if connector.connector_type != WEBCRAWLER_CONNECTOR:
return fail(run, f"executor not implemented for {connector.connector_type}")
owner_id = workspace_owner_id(session, pipeline.workspace_id) # select Workspace.user_id (03c §2.1)
actor_user_id = str(pipeline.created_by_id or owner_id) # Phase-5 hand-off (b)
urls = parse_webcrawler_urls(pipeline.config.get("INITIAL_URLS") or connector.config.get("INITIAL_URLS"))
if not urls:
return fail(run, "no URLs to crawl")
# --- concurrency: take both; release only what we took (Phase-5 hand-off (c)) ---
pipeline_lock = acquire_pipeline_indexing_lock(pipeline.id)
if not pipeline_lock:
return fail(run, "a run for this pipeline is already in progress")
connector_lock = acquire_connector_indexing_lock(connector.id)
if not connector_lock:
return fail(run, "connector is currently indexing; retry later")
# --- billing pre-flight (run-level; 03c statics). len(urls) is a safe upper bound. ---
svc = WebCrawlCreditService(session)
if WebCrawlCreditService.billing_enabled() and run.charged_micros is None:
await svc.check_credits(owner_id, len(urls)) # InsufficientCreditsError → except → fail("out of crawl credit")
# --- fetch ---
if pipeline.save_to_kb:
stats = {} # out-param sink — NON-breaking (see §3)
await index_crawled_urls(session, connector.id, pipeline.workspace_id, actor_user_id,
urls=urls, folder_id=pipeline.destination_folder_id,
bill=False, stats=stats, update_last_indexed=True)
documents_indexed = stats["documents_indexed"]
crawls_attempted = stats["crawls_attempted"]
crawls_succeeded = stats["crawls_succeeded"]
result_blob_key = None
else:
outcomes = await crawl_urls_fetch_only(urls) # list[dict] from CrawlOutcome (03a)
crawls_attempted = len(urls)
crawls_succeeded = sum(1 for o in outcomes if o["status"] == "success")
documents_indexed = 0
result_blob_key = await persist_run_blob(run, outcomes)
# --- charge: stamp run + add audit, THEN charge (its single commit flushes all three atomically) ---
if WebCrawlCreditService.billing_enabled() and crawls_succeeded > 0 and run.charged_micros is None:
run.charged_micros = WebCrawlCreditService.successes_to_micros(crawls_succeeded)
record_token_usage(session, usage_type="web_crawl", workspace_id=pipeline.workspace_id,
user_id=owner_id, cost_micros=run.charged_micros,
call_details={"urls": len(urls), "successes": crawls_succeeded,
"pipeline_id": pipeline.id, "run_id": run.id},
message_id=None) # add only — does not commit (03c §2.3)
await svc.charge_credits(owner_id, crawls_succeeded) # debits + COMMITS run+audit+balance (03c §2.4)
mark_succeeded(run): status=SUCCEEDED, finished_at=now,
documents_indexed, crawls_attempted, crawls_succeeded, result_blob_key; commit
except InsufficientCreditsError:
return fail(run, "out of crawl credit") # run is RUNNING→FAILED; no debit happened
except Exception as e:
return fail(run, {"message": str(e), "type": type(e).__name__})
finally:
if connector_lock: release_connector_indexing_lock(connector.id)
if pipeline_lock: release_pipeline_indexing_lock(pipeline.id)
```
`fail(run, err)` = set `status=FAILED`, `finished_at=now`, `error={...}`, `commit`, return — so the `finally` still runs and releases only the locks actually taken. `connector.id` in `finally` is reached only when `connector_lock is True`, which implies `connector` was resolved. **Idempotency rests on the status gate**, not `charged_micros`: `mark_running` commits `RUNNING` before any crawl/charge, so an `acks_late` redelivery (`celery_app.py:228-229`) or a manual re-enqueue sees `RUNNING`/terminal and returns. `charged_micros` is the belt-and-suspenders guard reserved for a future stuck-`RUNNING` re-driver (none in MVP).
Notes:
- **`workspace_owner_id`** = `select Workspace.user_id where id == workspace_id` (`03c` §2.1 pattern).
- **`is_pipeline_eligible`** is the 04a registry helper — re-checked here at run time (a deploy can flip a type to `MIGRATING`; the create-time check in Phase 5 doesn't protect the row forever).
- **URL source**: per-pipeline `config["INITIAL_URLS"]` overrides the connector's (the `config` JSONB "per-pipeline overrides" Phase 5 reserved); fall back to the connector's `INITIAL_URLS`.
### 2. Celery tasks (`app/pipelines/tasks.py`)
```python
@celery_app.task(name="run_pipeline", bind=True)
def run_pipeline(self, run_id: int) -> None:
return run_async_celery_task(lambda: _run(run_id)) # execute_run.py:19-22 shape
async def _run(run_id: int) -> None:
async with get_celery_session_maker()() as session:
try:
await execute_pipeline_run(session, run_id)
except Exception:
logger.exception("pipeline run %d failed unexpectedly", run_id)
await session.rollback(); raise
```
- `run_pipeline` **replaces** the Phase-5 stub of the same name; Phase 5's `/run` route already enqueues `run_pipeline(run.id)` (no route change needed).
- **Routing/registration**: ensure `"app.pipelines.tasks"` is in the `include` list (`celery_app.py:180-198`) — Phase 5 already added it for the stub; no-op if present — and route `"run_pipeline": {"queue": CONNECTORS_QUEUE}` (it crawls — same queue as `index_crawled_urls`, `celery_app.py:246`). Give the `apply_async`/`delay` a `time_limit` (the connectors queue already has an 8h hard cap, `celery_app.py:219`).
### 3. Crawler changes — `index_crawled_urls` (coordination with `03a`/`03c`)
Four additive params, **all default-compatible** with the existing connector path:
1. **`folder_id: int | None = None`** — set on each created `Document` (`:222-239`) and on the existing-doc update branch, so KB-save lands in the destination folder. Default `None` = root (today's behaviour).
2. **`urls: list[str] | None = None`** — when provided, crawl this list instead of re-reading `connector.config["INITIAL_URLS"]` (`:118-119`). **Required** for the per-pipeline `config["INITIAL_URLS"]` override (a Phase-5 `config` capability) and to keep the engine's `len(urls)` billing pre-check consistent with what's actually crawled. Default `None` = read connector config (today's behaviour).
3. **`bill: bool = True`** — gate `03c`'s in-indexer `check_credits` + `charge_credits` + `record_token_usage`. The pipeline engine calls `bill=False` and bills at the run level; the connector `/index` + periodic paths keep `bill=True`. *(Small addition to `03c`'s §2 wiring — see "Cross-plan coordination".)*
4. **`stats: dict | None = None`** — an **out-param sink**, NOT a return-shape change. When provided, the indexer populates `stats["documents_indexed"|"crawls_attempted"|"crawls_succeeded"]`. **Do NOT widen the return tuple:** the shared `_run_indexing_with_notifications` wrapper unpacks every indexer's return **by length** (`search_source_connectors_routes.py:1499-1507`: `if len(result) == 3: a,b,c = result else: a,b = result`), so a 4-tuple would raise `ValueError` for *every* connector. The 2-/3-tuple return stays exactly as-is (wrapper untouched); the engine passes a fresh `stats` dict and reads it after the call (the wrapper passes none → `None` → no-op). `03a` work-item 3 already reflects this (it exposes `crawls_succeeded` via **task metadata** only, with this `stats` sink owned by Phase 6 — not via the positional return).
The engine **bypasses** `_run_indexing_with_notifications` (it acquires the connector lock itself at `:1334` — going through the wrapper would self-deadlock on that same lock, and the wrapper's connector-`Notification` UI is the wrong surface for a pipeline run, which has its own `PipelineRun` record). It calls `index_crawled_urls` directly with no heartbeat callback. (Consequence: pipeline runs don't emit connector-indexing notifications and aren't touched by `cleanup_stale_indexing_notifications` — fine; run status lives on `pipeline_runs`.)
### 4. Fetch-only path + blob (`app/pipelines/engine.py` + `app/pipelines/storage.py`)
For `save_to_kb=false`:
```python
async def crawl_urls_fetch_only(urls):
crawler = WebCrawlerConnector() # 03a: no firecrawl ctor arg
out = []
for url in urls:
outcome = await crawler.crawl_url(url) # 03a: CrawlOutcome(status, result, error, tier)
out.append({
"url": url,
"status": outcome.status.value, # CrawlOutcomeStatus: success | empty | failed
"content": (outcome.result or {}).get("content", ""),
"metadata": (outcome.result or {}).get("metadata", {}),
"error": outcome.error,
})
return out
```
> **03a return shape.** `03a` is **committed to the `CrawlOutcome` dataclass** (`status`/`result`/`error`/`tier`) — the tuple option was dropped precisely because this fetch-only path consumes `.status`/`.result`/`.error` as attributes. So the unpack above is the firm contract; no adaptation needed.
```python
# storage.py
def build_pipeline_run_result_key(*, workspace_id: int, run_id: int) -> str:
return f"pipeline_runs/{workspace_id}/{run_id}/result.json"
async def persist_run_blob(run, outcomes) -> str:
key = build_pipeline_run_result_key(workspace_id=run.pipeline.workspace_id, run_id=run.id)
data = json.dumps({"runs": outcomes}).encode()
await get_storage_backend().put(key, data, content_type="application/json")
return key # → run.result_blob_key
```
- One blob per run (single configured backend, like document files). `result_blob_key` (Phase 5 column) stores the key; reads resolve via `get_storage_backend()`.
- **Billing still applies** here — `crawls_succeeded` is counted from `outcomes`, so non-KB runs cost exactly the same as KB runs (umbrella line 105). This is the whole reason the run engine, not the indexer, owns pipeline billing.
### 5. Scheduler tick (`app/pipelines/scheduler.py` + `tasks.py`)
A near-copy of the automations selector, retargeted at `pipelines`:
```python
@celery_app.task(name="pipeline_schedule_select")
def pipeline_schedule_select() -> None:
return run_async_celery_task(_tick)
async def _tick():
async with get_celery_session_maker()() as session:
now = datetime.now(UTC)
await _self_heal_null_next(session, now) # enabled + cron + next_scheduled_at IS NULL → compute
claims = await _claim_due(session, now) # enabled + next_scheduled_at <= now, FOR UPDATE SKIP LOCKED, advance
for c in claims:
await _start_run(session, c) # create PipelineRun(trigger=scheduled, status=pending) + run_pipeline.apply_async
```
- **`_claim_due`**: `select Pipeline where enabled AND connector_id IS NOT NULL AND schedule_cron IS NOT NULL AND next_scheduled_at IS NOT NULL AND next_scheduled_at <= now ORDER BY next_scheduled_at LIMIT 200 FOR UPDATE SKIP LOCKED` (uses Phase 5's partial `ix_pipelines_due`). The `connector_id IS NOT NULL` filter is **defense-in-depth** so a non-connector pipeline (the Phase-7 Uploads pipeline) can **never** be scheduled into perpetually-failing runs even if a cron is somehow set on it — the engine would `fail` every such run (`§1`); this stops them at the selector. (Phase 7 also guards the API; this is the belt to that suspenders.) Advance `next_scheduled_at = compute_next_fire_at(schedule_cron, schedule_timezone, after=now)`; on `InvalidCronError` → `enabled=False` (self-disable, like `selector.py:90-96,131-137`). Commit, then create the run + enqueue (mirror `_start_one` reload-after-commit, `selector.py:153-183`).
- **`_self_heal_null_next`**: backfills `next_scheduled_at` for enabled cron pipelines missing it (fresh inserts, restored rows) — so Phase 5's create can leave it NULL and the tick fills it within ≤1 min.
- **Duplicate suppression**: in `_start_run`, skip creating a scheduled run if the pipeline already has a non-terminal run (`status IN (pending, running)`) — mirrors the connector checker's in-progress guard (`schedule_checker_task.py:100-116`). Without it, a pipeline whose run outlasts its cron interval piles up `scheduled` runs that immediately `fail` on the lock (noisy, though harmless). `next_scheduled_at` is still advanced (the fire isn't retried — `catchup=False`, like the automations selector).
- **Beat entry**: add `"pipeline-schedule-select": {"task": "pipeline_schedule_select", "schedule": crontab(minute="*"), "options": {"expires": 50}}` to `celery_app.py:268-324` (default fast queue, like `check_periodic_schedules`). (`"app.pipelines.tasks"` is already in `include` from Phase 5's stub, so `pipeline_schedule_select` in the same module is auto-registered.)
- **`schedule_cron` edits** (Phase 5 `PUT`): when `schedule_cron` (or timezone) changes, **null `next_scheduled_at`** so the tick recomputes (small `PUT`-handler addition — see Required 05 amendment). NULL cron ⇒ manual-only (tick ignores it).
### 6. De-dup guard (resolves Phase-5 deferred double-bill)
When a pipeline that **wraps a connector** is created or enabled (Phase 5 routes), set that connector's `periodic_indexing_enabled = False` (+ `next_scheduled_at = None`) so a single scheduler owns each connector (the **pipeline is authoritative**, per Phase 5's recommendation). Two touch points:
- Phase 6 owns the rule; implement it in the Phase-5 `POST /pipelines` and `PUT /pipelines/{id}` handlers (when `connector_id` is set and `enabled`). This is a tiny handler addition (Phase 5 explicitly handed it here).
- Belt-and-suspenders: the run engine already takes the **connector lock** during a run, so even if both scan, only one crawl proceeds at a time (the connector checker skips on a held lock, `schedule_checker_task.py:93`).
### 7. Chat-agent run-history tool (`main_agent/tools/pipeline_runs.py`)
A read-only tool registered in the main-agent registry (umbrella default = tool):
```python
def create_get_pipeline_runs_tool(*, workspace_id: int) -> BaseTool:
@tool
async def get_pipeline_runs(pipeline_id: int | None = None, limit: int = 20) -> str:
"""Recent pipeline run history for THIS workspace: name, status, trigger,
schedule, counts, started/finished. Read-only."""
async with shielded_async_session() as session: # knowledge_tree/middleware.py:199 pattern
q = (select(PipelineRun, Pipeline.name, Pipeline.schedule_cron)
.join(Pipeline, PipelineRun.pipeline_id == Pipeline.id)
.where(Pipeline.workspace_id == workspace_id))
if pipeline_id is not None:
q = q.where(Pipeline.id == pipeline_id)
rows = (await session.execute(
q.order_by(PipelineRun.created_at.desc()).limit(min(limit, 100)))).all()
return render_compact(rows) # name • status • trigger • finished_at • docs/crawls
return get_pipeline_runs
```
- **Registration**: add `"get_pipeline_runs": (_build_get_pipeline_runs_tool, ("workspace_id",))` to `_MAIN_AGENT_TOOL_FACTORIES` (`registry.py:85-102`) — the dep key is `workspace_id`, matching the post-rename deps dict; add the name to `main_agent/tools/index`; add display metadata to `shared/tools/catalog`.
- **Workspace scope is structural** — the tool only ever sees its build-time `workspace_id`; no cross-workspace leakage (same guarantee `KnowledgeTreeMiddleware` relies on).
- **Anonymous / no-workspace turns**: exclude the tool when there's no `workspace_id` (anonymous chat has no workspace/pipelines), the same way `KnowledgeTreeMiddleware` is CLOUD-only (`knowledge_tree/middleware.py:130-131`). The registry factory requires `workspace_id`, so just don't enable it in the anonymous tool set.
- The tool reads `pipelines`/`pipeline_runs` directly (no Zero dependency), so it works server-side immediately even though client Zero sync is dormant until the frontend lands (Phase 5 note).
- **Deferred**: an always-on `<pipeline_activity>` middleware injection (à la `KnowledgeTreeMiddleware`) — adds per-turn token cost; revisit if the agent under-uses the tool.
### 8. (Optional) blob read endpoint
Phase 5 stores `result_blob_key` but exposes no reader. Add `GET /pipelines/{id}/runs/{run_id}/result` → stream the blob via `get_storage_backend().open_stream(run.result_blob_key)` (404 if NULL), `CONNECTORS_READ`, workspace-scoped — mirroring `file_storage/api.py:69-90`. Lets API consumers + the chat tool fetch non-KB results. (Small; include if cheap, else defer to the public-API phase.)
## Required `05` amendment (timezone + recompute)
Small, additive. **Items 1–3 (the `schedule_timezone` column) are already folded into `05`** (model/migration/schema/validator) — listed here for traceability. **Item 4 is Phase-6 route logic** added on top of Phase 5's handlers (Work items 5–6):
1. *(done in `05` §2)* **Model**: `schedule_timezone = Column(String(64), nullable=False, default="UTC", server_default="UTC")` on `Pipeline`.
2. *(done in `05` §4)* **Migration**: `schedule_timezone VARCHAR(64) NOT NULL DEFAULT 'UTC'` in `CREATE TABLE pipelines`.
3. *(done in `05` §5)* **Schema**: `schedule_timezone: str = "UTC"` on `PipelineBase` + `PipelineUpdate`; `(schedule_cron, schedule_timezone)` validated together via `validate_cron(cron, tz)` (`cron.py:15`).
4. **Routes** (Phase 6 adds to `05` §7 handlers): on `PUT` when `schedule_cron`/`schedule_timezone` changes, set `next_scheduled_at = None` (tick self-heals). On `POST`/`PUT` with `connector_id` + `enabled`, apply the §6 de-dup guard.
(If we'd rather not touch `05`: hardcode `"UTC"` everywhere in Phase 6 and skip the column — but that loses operator-meaningful local schedules and diverges from automations. The column is the right call.)
## Cross-plan coordination
- **`03a`** — Phase 6 calls `WebCrawlerConnector.crawl_url(url)` directly in the fetch-only path and relies on `03a`'s `CrawlOutcome.status` (`SUCCESS`/`EMPTY`/`FAILED`, dataclass form — already locked in `03a`). `03a` work-item 3 is **already aligned**: it surfaces the `crawls_succeeded` counter via **task metadata** only (not by widening `index_crawled_urls`'s positional return — the shared `_run_indexing_with_notifications` wrapper unpacks by tuple length, `:1499-1507`). Phase 6 owns the `folder_id`/`urls`/`stats` params; `03a` owns the counter + `CrawlOutcome`.
- **`03c`** — add the **`bill: bool = True`** parameter to the in-indexer billing (so the pipeline path can suppress it). `03c`'s `WebCrawlCreditService` statics (`billing_enabled()`, `successes_to_micros(n)`, `check_credits`, `charge_credits`) are reused verbatim by the run engine. `03c`'s risk note already anticipated this: *"Phase 5's `PipelineRun` can persist `charged_micros` for stronger idempotency"* — Phase 6 is where that lands.
- **`05`** — the four amendments above; `run_pipeline` task name + `/run` enqueue already exist (stub → real); `charged_micros`/`crawls_*`/`result_blob_key` columns already exist (pre-built in Phase 5).
- **`04a`** — `is_pipeline_eligible(connector_type)` is imported for the run-time re-check.
## Work items
1. **`app/pipelines/` package**: `engine.py` (`execute_pipeline_run` + `mark_running/succeeded/fail` helpers + `workspace_owner_id` + `crawl_urls_fetch_only`), `tasks.py` (`run_pipeline`, `pipeline_schedule_select`), `scheduler.py` (`_tick`/`_self_heal_null_next`/`_claim_due`/`_start_run`), `storage.py` (blob key + `persist_run_blob`).
2. **Pipeline lock helpers**: add `acquire_pipeline_indexing_lock(id)` / `release_pipeline_indexing_lock(id)` to `utils/indexing_locks.py` (key `indexing:pipeline_lock:{id}`), mirroring the connector lock (`:19-40`).
3. **Crawler changes**: `index_crawled_urls` gains `folder_id` + `urls` + `bill` + a `stats` **out-param** (populates `documents_indexed`/`crawls_attempted`/`crawls_succeeded`). **No return-shape change → no caller changes** (the shared wrapper's 2-/3-tuple unpack stays valid). Coordinate with `03a`/`03c` (see "Cross-plan coordination").
4. **Celery wiring** (`celery_app.py`): ensure `"app.pipelines.tasks"` is in `include` (added in Phase 5 for the stub); route `run_pipeline → CONNECTORS_QUEUE`; add the `pipeline-schedule-select` Beat entry.
5. **De-dup guard** in Phase-5 `POST`/`PUT /pipelines` (disable wrapped connector's periodic indexing).
6. **`05` amendment**: `schedule_timezone` column/migration/schema/validator + `next_scheduled_at` reset on cron edit.
7. **Chat tool** `get_pipeline_runs` + registry/index/catalog registration.
8. **(Optional)** `GET …/runs/{run_id}/result` blob reader.
9. **Tests** (below).
## Tests
- **Manual run, KB-save**: WebURL pipeline `save_to_kb=true` + `destination_folder_id` → run goes `pending→running→succeeded`, `Document`s created **in that folder**, `documents_indexed`/`crawls_*` set, `finished_at` set.
- **Manual run, non-KB**: `save_to_kb=false` → **no** `Document`s, `result_blob_key` set + blob readable, `crawls_*` set. Billing **still** charges `crawls_succeeded` (parity with KB run).
- **Billing**: enabled + sufficient → owner debited `crawls_succeeded * 1000`, `charged_micros` stamped, one `web_crawl` `TokenUsage`; **insufficient → run fails pre-crawl (FAILED), no debit**; disabled (self-hosted) → no charge, no `web_crawl` row, `charged_micros` stays NULL.
- **Idempotency / lifecycle**: re-enqueuing a `RUNNING` or terminal run → immediate no-op (`status != PENDING` gate) → **no second charge** (the primary guard; `mark_running` commits before any charge); engine never raises out of `fail()` paths (bad type / NULL connector / no URLs / ineligible / lock contention) — all land the run in `FAILED` and release locks.
- **Eligibility re-check**: pipeline over a connector whose type is now `MIGRATING` → run fails cleanly (not crash).
- **Scheduler**: a due cron pipeline → tick creates a `scheduled` run + advances `next_scheduled_at` to the next cron match (in `schedule_timezone`); NULL `next_scheduled_at` self-heals; invalid cron self-disables (`enabled=False`); manual-only (NULL cron) never fires.
- **De-dup**: creating/enabling a pipeline over a connector flips that connector's `periodic_indexing_enabled=False`; concurrent connector-checker tick skips on the held lock.
- **Concurrency**: a second run while one is in-flight → fails with "already in progress" (locks held).
- **Chat tool**: `get_pipeline_runs` returns this workspace's runs only; another workspace's runs never appear; respects `limit`.
- **Counts contract (regression-critical)**: `index_crawled_urls` keeps its existing 2-/3-tuple return (the shared `_run_indexing_with_notifications` length-unpack still works for *all* connectors); when a `stats` dict is passed it's populated with `documents_indexed`/`crawls_attempted`/`crawls_succeeded`; `bill=False` suppresses in-indexer billing; connector path (`bill=True`, no `stats`) behaves byte-for-byte as before. Add a wrapper test asserting webcrawler indexing through `_run_indexing_with_notifications` still unpacks cleanly.
## Risks / trade-offs
- **Billing split across two call sites** (indexer for connector path, run engine for pipeline path), unified only by `03c`'s shared statics. Accepted: it's what makes `charged_micros` run-idempotency clean and keeps both KB branches billed identically. The `bill` flag is the single seam.
- **Stuck `RUNNING` runs.** A crash *after* `mark_running` but *before* terminal leaves `status=RUNNING` forever (the `status != PENDING` gate then no-ops any redelivery — same property as automations). No double-charge (gate + `charged_micros`), but the run never completes and its locks rely on Redis TTL to free. A stale-run sweeper (à la `cleanup_stale_indexing_notifications`, `celery_app.py:279-285`) is deferred; MVP exposure is a rare worker crash.
- **Two crawl code paths** (`index_crawled_urls` for KB vs `crawl_urls_fetch_only` for non-KB). Minor duplication of the crawl loop; deliberate, since forcing the `Document`-centric 2-phase indexer into a "don't persist" mode is messier. Both share `WebCrawlerConnector.crawl_url` + the same billing math.
- **WebURL-only executor.** A pipeline over GDrive/OneDrive/Dropbox is creatable (Phase 5 lets `AVAILABLE DATA_SOURCE`) but fails at run time ("executor not implemented"). Acceptable for MVP (those keep their connector-periodic path); generalize the executor in Phase 7+.
- **De-dup guard is one-way.** Creating/enabling a pipeline disables the wrapped connector's `periodic_indexing_enabled`, but **disabling or deleting that pipeline does NOT restore it** — so a connector can end up indexed by *neither* path (data goes stale) until the user re-enables connector periodic indexing manually. MVP-acceptable (operator action); a symmetric restore-on-disable is a small follow-up. Flag in the (deferred) UI copy.
- **Blob backend assumption.** `result_blob_key` stores only the key (no backend name, unlike `document_files.storage_backend`). Fine for the single configured backend; if multi-backend ever lands, add a backend column to `pipeline_runs` (additive).
- **Non-KB blob built in memory.** `crawl_urls_fetch_only` accumulates every URL's extracted content into one JSON blob before upload — a large URL list (or huge pages) inflates worker memory. Acceptable for MVP URL counts; if it bites, stream to the backend or cap per-run URLs (a `config` knob). Also: non-KB runs have **no dedup** (every run re-crawls + re-bills all URLs) — that's inherent to "don't persist", and the user opted out of KB.
- **Lock lifetime vs run length.** The Redis locks use `CONNECTOR_INDEXING_LOCK_TTL_SECONDS`; a crawl run longer than the TTL could let a second run start. Pre-existing property of the connector lock (the connectors queue caps at 8h); revisit the TTL if pipeline runs routinely exceed it.
- **Two minute-level Beat scans** (connector checker + pipeline tick) now run every minute. Both are indexed (`ix_pipelines_due` partial index; connector scan on its own predicate) + batch-capped (200), so cost is negligible.
## Out of scope (hand-offs)
- **Uploads pipeline execution** (`connector_id=NULL`, `trigger=upload`, always `save_to_kb`) → **Phase 7** (`07-upload-pipeline-kb.md`). This engine explicitly fails NULL-connector runs.
- **Non-WebURL executors** (GDrive/OneDrive/Dropbox pipelines) → Phase 7+.
- **Always-on chat context middleware** (`<pipeline_activity>` injection) → deferred (tool-only for MVP).
- **Document→run provenance** (`documents.pipeline_run_id`) → deferred (touches `documents`).
- **Public pay-as-you-go API over pipelines** + **public MCP KB server** → post-MVP (umbrella "Deferred").
- **Frontend** Pipelines UI (run-history, manual run, schedule editor) → frontend umbrella.

73
plans/backend/07-orchestration.md Normal file
View file

@ -0,0 +1,73 @@
# Phase 7 — Orchestration / Conversation (the CI-expert subagent)
> Build last; sits atop `04`/`05` (and `06` for the ongoing mode).
> The multi-agent chat runtime (deepagents, subagent dispatch, streaming, citation middleware) is shipped.
> Net-new here = one builtin subagent + its tools + its prompt. Locate code by symbol/grep.
## Objective
Ship the `intelligence_agent` — a builtin CI/scraper-expert subagent that delivers the product through
plain conversation: understand intent, compose scraper verbs (`04`), answer in plain language, and hand
standing needs to the ongoing "keep watching" mode (`06`). It reasons over chat history to report what's
new vs prior tool outputs already in context.
## The subagent
New builtin `subagents/builtins/intelligence_agent/`, peer to `research`/`deliverables`, packed via
`pack_subagent`. Files follow the existing pattern: `agent.py` (`build_subagent`), `tools/index.py`
(`NAME · RULESET · load_tools`), `description.md` (router one-liner), `system_prompt.md` (the playbook).
The main agent delegates CI-/scraping-flavored requests to it via `description.md`.
## The playbook (`system_prompt.md`)
1. **Intent routing** — one-shot ("compare/find/what is") → compose verbs & answer; standing concern
("watch/track/notify when/weekly") → hand to `06`; ambiguous → ask one question ("just once, or keep
watching?").
2. **Verb composition** — chains `web.discover → web.scrape` and `maps.search → maps.place → maps.reviews`;
infer URLs/queries/locations from context.
3. **"What changed"** — re-invoke the verb and compare against prior tool outputs in the chat history,
summarizing what's new.
## The toolset (`load_tools`)
| Tool | Wraps | Billing |
|------|-------|---------|
| capability verbs (`web.scrape`, `web.discover`, `maps.search`, `maps.place`, `maps.reviews`) | `04` executors (direct-return) | `03c` turn accumulator (as `scrape_webpage` does) |
| `start_watch(intent)` *(deferred)* | hands off to `06` | per re-invocation (design with `06`) |
Capability verbs are a shared, registry-generated tool module (`04`) that `research` can also load; the
`intelligence_agent` adds the CI prompt and the `06` handoff.
## Boundaries
- The subagent drives the `04` verbs via tools; fetch/bypass/clean logic stays in Acquisition + `04`,
callable headless.
- Humans get the agent; REST/MCP callers call `04` verbs directly, skipping this subagent.
## Work items
1. Subagent scaffold: `subagents/builtins/intelligence_agent/` (`agent.py` / `tools/index.py` /
`description.md` / `system_prompt.md`), packed via `pack_subagent`.
2. CI playbook prompt: intent routing (one clarifying question), verb chains, "what changed" reasoning.
3. Shared verb tool module: registry-backed (`04`), reusable by `research`.
4. `start_watch` seam to `06` (wired when `06` is designed).
5. Router registration: `description.md` for main-agent delegation.
## Tests
- A CI-/scraping-flavored request routes to `intelligence_agent`; a non-CI request does not.
- "compare X and Y" composes verbs and answers in plain language, persisting nothing.
- "reviews for the top 3 coffee shops near me" chains `maps.search → maps.place → maps.reviews`.
- A follow-up re-invokes the verb and reports new items vs prior in-context outputs.
- "watch this weekly" routes to the ongoing mode, not a one-shot answer.
## Out of scope
- Verbs, doors → `04`/`05`. Ongoing/periodic mechanism → `06`.
- Richer multi-step playbooks, proactive suggestions, cross-topic synthesis.
- Frontend CI surfaces → frontend umbrella.
## Open questions
- Whether CI one-shot scraping stays in `research` or the `intelligence_agent` owns all CI-flavored calls.
- The `start_watch` handoff shape (settled with `06`).

195
plans/backend/07-upload-pipeline-kb.md
View file

@ -1,195 +0,0 @@
# Phase 7 — File upload as a pipeline + KB-save-secondary
> # ❌ SUPERSEDED (2026-06-30)
> **This entire phase is dropped.** Per the **Architecture correction** in `00-umbrella-plan.md`, there is no "Uploads pipeline" and no upload `PipelineRun`. Uploads simply populate the **input-only KB** through the existing upload routes (an upload creates a `Document` — that *is* KB membership). The umbrella's KB-input invariant ("uploads always land in the KB") is inherent and needs no pipeline wrapper. Retained **for historical context only** — do **not** implement it. **The CI-relevant successor is the "context folder" idea in [`revamp phases 4-7/05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md)**: files uploaded in a CI chat go into the KB as normal (indexed), routed to a dedicated folder, and may feed the materiality judge via KB retrieval. (Canonical Phase 7 is now **Orchestration** — [`revamp/07-orchestration.md`](revamp%20phases%204-7/07-orchestration.md) — unrelated to uploads.)
> Part of the **CI Pivot MVP**. See `00-umbrella-plan.md` (Phase 7) — the final backend phase.
> Precondition: Phase 5 (`05-pipelines-model.md`) live (`pipelines`/`pipeline_runs` tables, `PipelineRunTrigger.UPLOAD`, `connector_id` nullable) and Phase 6 (`06-pipelines-exec.md`) live (run engine, which **explicitly fails `connector_id IS NULL` runs**). No sibling ahead — this closes the backend umbrella.
> **Implementation note (post-rename).** Phases 1–2 are **SHIPPED**, so the live code already uses `workspace_id` / `Workspace`. Citations below that still show `search_space_id` / `SearchSpace` are pre-rename — substitute the `workspace_*` equivalent and grep the new name. **New code in this plan uses the canonical `workspace_*` names** (snippets below updated). Locate code by **symbol/grep**, not the absolute line numbers cited here (the rename + Phases 3–6 shift them).
## Objective
Close the loop on the umbrella's positioning — *"User file uploads also do a pipeline entry and register a run that files were uploaded by user; only file uploads are always saved in the knowledge base"* — by:
1. **Uploads-as-pipeline** — give every workspace a singleton **"Uploads" pipeline** (`connector_id = NULL`, `save_to_kb = true`), lazily created, and make each upload request **register a `PipelineRun(trigger=upload)`** so uploads show up in pipeline run history (and in the Phase-6 chat `get_pipeline_runs` tool) exactly like connector fetches.
2. **KB-save-secondary (confirm + close)** — the opt-in `save_to_kb` + `destination_folder_id` machinery for *connector* pipelines already shipped in Phases 5–6; this phase records the **inverse invariant for uploads** (uploads **always** save to KB) and verifies nothing in the pivot accidentally made KB-save the *default* for connector pipelines.
**The core design fact (drives everything):** uploads are **NOT executed by the Phase-6 run engine**. Phase 6's `execute_pipeline_run` fails any `connector_id IS NULL` run with `"uploads pipeline has no fetch executor"` (`06` §1). The existing upload routes remain the executor (they already create `Document`s + dispatch ETL); Phase 7 only **attaches an audit `PipelineRun`** to that flow and ensures the Uploads pipeline row exists. No `run_pipeline` enqueue, no engine change.
## Locked decisions (MVP)
| Concept | Decision |
|---------|----------|
| Uploads pipeline | **One singleton per workspace**, `connector_id = NULL`, `save_to_kb = true`, `schedule_cron = NULL` (never scheduled), `enabled = true`, `name = "Uploads"`. **Lazily get-or-created** on first upload (not at workspace creation — avoids backfilling every existing workspace). |
| Uniqueness / race-safety | A **partial unique index** `ON pipelines (workspace_id) WHERE connector_id IS NULL` makes "one Uploads pipeline per workspace" a DB invariant and lets get-or-create be race-safe via `INSERT … ON CONFLICT DO NOTHING` + re-`SELECT`. (Phase 5 already reserves `connector_id IS NULL` to mean "Uploads", so this index is exactly that semantic.) Small Phase-7 migration. |
| Run lifecycle | **Audit record, created terminal at request time.** The upload route inserts `PipelineRun(trigger=upload, status=succeeded, documents_indexed=<accepted file count>, started_at=finished_at=now())`. Per-file ETL outcomes continue to live on `Document.status` (`pending→ready/failed`, already Zero-synced). Rationale below. |
| Why not track per-file ETL into the run | The `fileupload` path fans out **N independent per-file Celery tasks** (`task_dispatcher.py:44`) with **no batch barrier**, and accurate "which docs belong to this run" needs the **`documents.pipeline_run_id` provenance column that Phase 5 deferred** (`05` §3 "Out of scope (provenance)"). Building a barrier is out of MVP scope; the run is an *upload event* record, matching the umbrella's "register a run that files were uploaded". |
| Billing | **No crawl billing for uploads** — uploads aren't crawls. `charged_micros` / `crawls_*` stay `NULL` on upload runs. Existing ETL/embedding credit behaviour (if any) is unchanged and orthogonal (`03c` only governs `web_crawl`). |
| Folder placement | The Uploads pipeline's `destination_folder_id` stays **NULL** — it is a singleton and cannot encode per-upload folders. Folder placement remains owned by the **existing upload code** (`fileupload` → root; `folder-upload` → its created/declared folder). The run/pipeline is **descriptive**, not the folder driver (unlike connector pipelines, where the engine reads `destination_folder_id`). |
| Surfaces wired | `POST /documents/fileupload` and `POST /documents/folder-upload` (both create `Document`s). The desktop precheck route `POST /documents/folder-mtime-check` (`documents_routes.py:1533`) creates nothing and is **not** wired; `folder-unlink` / `folder-sync-finalize` are delete/sync lifecycle, also out. |
| Connector-pipeline KB default | **Unchanged from Phase 5:** `save_to_kb` defaults `False` (`05` §2 `server_default="false"`). This phase only asserts uploads set it `true`; it does **not** flip the connector default. |
## Current state (cited)
### The upload routes (the executors we attach to)
- **`POST /documents/fileupload`** — `documents_routes.py:126-332`. Permission `DOCUMENTS_CREATE` (`:163-169`). Phase 1: create `Document(search_space_id, document_type=FILE, status=pending, created_by_id, …)` per file, dedup by `unique_identifier_hash` (`:217-266`), `commit` (`:278`). Phase 1.5: `store_document_file(...)` per doc (`:289-297`). Phase 2: `dispatcher.dispatch_file_processing(document_id=…)` per file (`:307-316`). Returns `document_ids` / `skipped_duplicates` (`:318-325`). **Creates no folder (root), no run record today.** `files_to_process` (`:213`) = the accepted (newly-queued) set; `skipped_duplicates` (`:214`) = already-`READY` dupes.
- **`POST /documents/folder-upload`** — `documents_routes.py:1598-1748`. Permission `DOCUMENTS_CREATE` (`:1623-1629`). Creates/reuses a root `Folder` (`:1673-1704`), writes temps, then dispatches **one** batch task `index_uploaded_folder_files_task.delay(...)` (`document_tasks.py:1403`) which has its own notification + heartbeat (`:1446-1457`) — i.e. a **real completion barrier** exists on this path (unlike `fileupload`). Returns `root_folder_id` + `file_count` (`:1743-1748`).
- **Dispatcher** — `task_dispatcher.py:12-52`: `dispatch_file_processing(...)` → `process_file_upload_with_document_task.delay(document_id=…)`, **one task per file**, fire-and-forget (no barrier).
### Phase 5/6 hand-offs this phase consumes
- `Pipeline.connector_id` **nullable**, `NULL = "non-connector pipeline (the Phase-7 Uploads pipeline)"` (`05` §"Pipeline → connector"; model `05` §2 `:105-106`). `PipelineRunTrigger.UPLOAD` already exists (`05` §1). `PipelineRun` has `trigger`, `status`, `documents_indexed`, `started_at`, `finished_at`, `charged_micros` (nullable) (`05` §3).
- **Phase 6 engine refuses uploads**: `if pipeline.connector_id is None: return fail(run, "uploads pipeline has no fetch executor")` (`06` §1) — confirms uploads must be recorded by the route, **not** enqueued to `run_pipeline`.
- **Chat tool** `get_pipeline_runs` (`06` §7) selects `PipelineRun` joined to `Pipeline` by `workspace_id` — so upload runs surface to the agent for free once they exist.
- **De-dup guard** (`06` §6) only touches **connector-backed** pipelines (`when connector_id is set`); the Uploads pipeline (`connector_id NULL`) is untouched by it. The **scheduler** (`06` §5 `_claim_due`) ignores the Uploads pipeline **two ways** — `schedule_cron IS NOT NULL` (NULL cron) **and** `connector_id IS NOT NULL` (the defense-in-depth filter `06` added at Phase 7's request). Both confirmed safe.
### Document model
- `Document` — `db.py:1348`: `document_type` (`:1352`, `DocumentType.FILE`), `unique_identifier_hash` (`:1365`), `folder_id` **nullable** (`:1389`, root/unfiled), `created_by_id` (`:1397`), plus `search_space_id`/`status`. KB membership = a `Document` row exists; uploads always create one ⇒ uploads are always "in the KB" by construction.
## Target design
### 1. `get_or_create_uploads_pipeline(session, workspace_id, created_by_id)` — `app/pipelines/uploads.py`
```python
async def get_or_create_uploads_pipeline(session, *, workspace_id: int, created_by_id) -> Pipeline:
# fast path
existing = (await session.execute(
select(Pipeline).where(Pipeline.workspace_id == workspace_id,
Pipeline.connector_id.is_(None)))).scalar_one_or_none()
if existing:
return existing
# race-safe create (partial unique index ux_pipelines_uploads_per_workspace)
stmt = (pg_insert(Pipeline.__table__)
.values(workspace_id=workspace_id, connector_id=None, name="Uploads",
save_to_kb=True, enabled=True, schedule_cron=None,
created_by_id=created_by_id, config={})
.on_conflict_do_nothing(index_elements=[Pipeline.workspace_id],
index_where=Pipeline.connector_id.is_(None)))
await session.execute(stmt)
await session.flush()
return (await session.execute(
select(Pipeline).where(Pipeline.workspace_id == workspace_id,
Pipeline.connector_id.is_(None)))).scalar_one()
```
- Lazy (first upload), idempotent, race-safe. `created_by_id` = the uploading user (becomes the pipeline's creator metadata; `SET NULL` on member deletion per Phase 5).
- Lives in the `app/pipelines/` package (created in Phase 6) so the upload route imports it without a route↔route dependency.
- **Implementation gotcha — core insert bypasses ORM defaults.** `Pipeline.updated_at`/`created_at` use **Python-side** `default=`/`onupdate=` callables (`05` §2 `:114-115`), which a core `pg_insert` does **not** fire. The `.values(...)` must set `created_at`/`updated_at` explicitly (e.g. `now`) — or implement get-or-create as **ORM-insert + `flush` inside a `session.begin_nested()` SAVEPOINT, catching `IntegrityError` → rollback savepoint → re-`SELECT`** (avoids the core/ORM default mismatch entirely; the SAVEPOINT keeps the route's outer transaction intact). Either is fine; the SAVEPOINT form is closer to the ORM conventions used elsewhere.
### 2. Register an upload run (shared helper) — `app/pipelines/uploads.py`
```python
async def record_upload_run(session, *, workspace_id, created_by_id, documents_indexed: int) -> PipelineRun:
pipeline = await get_or_create_uploads_pipeline(session, workspace_id=workspace_id,
created_by_id=created_by_id)
now = datetime.now(UTC)
run = PipelineRun(pipeline_id=pipeline.id, trigger=PipelineRunTrigger.UPLOAD,
status=PipelineRunStatus.SUCCEEDED,
documents_indexed=documents_indexed, started_at=now, finished_at=now)
session.add(run)
return run # caller commits (folds into the route's existing commit)
```
- **Terminal-at-creation** (locked decision): records the upload event with the accepted file count. `crawls_*` / `charged_micros` left NULL (not a crawl).
- Caller does **not** await any Celery task — the run is purely the route's own DB write, folded into the route's existing transaction (no extra round-trips, no new failure mode that could block the upload).
### 3. Wire `POST /documents/fileupload`
After Phase 2 dispatch (`documents_routes.py:307-316`), before building the response, add:
```python
if files_to_process: # only when something was actually queued
await record_upload_run(session, workspace_id=workspace_id,
created_by_id=str(user.id),
documents_indexed=len(files_to_process))
await session.commit()
```
- Count = `len(files_to_process)` (the newly-created/re-queued docs), **excluding** `skipped_duplicates` (already-READY no-ops). An all-duplicate upload records **no** run (nothing ingested).
- **Best-effort, non-blocking — and the try/except is mandatory, not cosmetic.** The route has an **outer `except Exception` → `session.rollback()` → HTTP 500** (`documents_routes.py:328-332`). If `record_upload_run` raised unguarded, it would hit that handler and **500 an upload whose documents are already committed + dispatched**. So the call MUST be wrapped in its **own inner `try/except`** that swallows: on failure `await session.rollback()` (discards only the run insert — documents were committed at `:278`/`:304`) + `logger.warning(...)`, then fall through to the response. A run-record write must **never** fail the upload. (Mirrors the `store_document_file` best-effort pattern at `:298-303`.)
### 4. Wire `POST /documents/folder-upload`
Folder-upload dispatches one batch task, so record the run at request time with the file count (`documents_routes.py` before the `return` at `:1743`):
```python
await record_upload_run(session, workspace_id=workspace_id,
created_by_id=str(user.id), documents_indexed=len(files))
await session.commit() # best-effort / logged, same as §3
```
- Uses `len(files)` (folder-upload doesn't pre-dedup in the route; dedup happens inside the batch task).
- **Optional richer accounting (deferred):** because this path *has* a barrier (`index_uploaded_folder_files_task`), a later enhancement could create the run `pending` here and flip it `succeeded`/`failed` from inside the task (it already manages a notification lifecycle). MVP keeps it uniform with `fileupload` (terminal-at-request). Flagged, not built.
### 5. Migration (one small file) — partial unique index
```sql
CREATE UNIQUE INDEX IF NOT EXISTS ux_pipelines_uploads_per_workspace
ON pipelines (workspace_id) WHERE connector_id IS NULL;
```
- Chain after the then-current head (verify `alembic heads`). Enforces the singleton + powers `ON CONFLICT`. `downgrade()` drops it.
- **Pre-flight (safety):** if any workspace somehow already has >1 `connector_id IS NULL` pipeline (shouldn't, pre-Phase-7), the `CREATE UNIQUE INDEX` fails — the migration docstring notes a dedup step (keep the oldest, delete extras) just in case. Expectation: none exist (uploads pipelines are introduced *here*).
### 6. KB-save-secondary verification (no new code, a guard)
- Uploads pipeline hardcodes `save_to_kb=True` (§1) — uploads always land in the KB (a `Document` row), matching the umbrella invariant.
- Confirm Phase 5's connector-pipeline default is still `save_to_kb=False` (`05` §2) — i.e. KB-save is **opt-in for connectors, mandatory for uploads**. Add a test asserting both defaults so a future change can't silently make connector KB-save the default.
### 7. Guard the generic Phase-5 CRUD against the system Uploads pipeline (the gap)
The Uploads pipeline is a **system-managed** row (`connector_id IS NULL`, auto-created by §1). Phase 5's generic `/pipelines` CRUD doesn't know it's special, which opens three footguns — Phase 7 closes them:
| Phase-5 endpoint | Problem if left generic | Phase-7 guard |
|------------------|-------------------------|---------------|
| `POST /pipelines` | Phase 5 **allows `connector_id=None`** (`05` §7 create + test). A user could hand-create a rogue/duplicate non-connector pipeline. | **Reject `connector_id is None` with 4xx** — non-connector pipelines are system-managed (auto-created on upload), not user-created. This **supersedes** Phase 5's "`connector_id=None` → allowed" create test (update it). The partial unique index (§5) is the DB backstop. |
| `POST /pipelines/{id}/run` | Manual-running the Uploads pipeline enqueues `run_pipeline`, which the Phase-6 engine **fails** (`"uploads pipeline has no fetch executor"`) → a confusing **FAILED** manual run every click. | **Reject `connector_id IS NULL` with 4xx** ("uploads are triggered by uploading files, not a manual run") — before creating the run row / enqueuing. |
| `PUT /pipelines/{id}` | Setting `schedule_cron` on the Uploads pipeline makes the scheduler eligible-by-cron → **perpetually-failing scheduled runs** (engine refuses null-connector). Editing `save_to_kb`/`destination_folder_id` is silently **ineffective** (uploads are route-driven, not engine-driven). | **Reject schedule edits (`schedule_cron`/`schedule_timezone`) on `connector_id IS NULL` pipelines with 4xx.** (`connector_id` is already immutable per Phase 5.) Optionally reject all PUTs to the Uploads pipeline; minimally block the scheduling fields. Belt-and-suspenders: Phase 6's `_claim_due` also filters `connector_id IS NOT NULL` (`06` §5). |
| `DELETE /pipelines/{id}` | Deleting the Uploads pipeline cascades its run history. | **Allowed** (MVP): it is **lazily re-created** on the next upload (§1); only past upload-run history is lost. Note in UI copy later; an optional hard block is a small follow-up. |
Net: after Phase 7, the only non-connector pipeline that can exist is the **system Uploads pipeline**, it is never user-created / manually-run / scheduled, and the scheduler ignores it two ways (NULL cron *and* the `connector_id IS NOT NULL` filter).
## Work items
1. **`app/pipelines/uploads.py`**: `get_or_create_uploads_pipeline` (race-safe via `ON CONFLICT`) + `record_upload_run` (terminal upload run).
2. **Migration**: partial unique index `ux_pipelines_uploads_per_workspace ON pipelines(workspace_id) WHERE connector_id IS NULL`; docstring dedup pre-flight note; `downgrade` drops it.
3. **Wire `fileupload`** (`documents_routes.py:126-332`): call `record_upload_run(documents_indexed=len(files_to_process))` after dispatch, in an **inner** best-effort try/except (must not reach the route's outer 500 handler), only when `files_to_process` non-empty.
4. **Wire `folder-upload`** (`documents_routes.py:1598-1748`): call `record_upload_run(documents_indexed=len(files))` before the response, same inner best-effort guard.
5. **Guard the generic CRUD** (§7): `POST /pipelines` rejects `connector_id is None`; `/run` rejects `connector_id IS NULL`; `PUT` rejects schedule edits on `connector_id IS NULL`. Update Phase 5's "`connector_id=None` → allowed" create test accordingly. (Phase 6's `_claim_due` `connector_id IS NOT NULL` filter is the scheduler backstop.)
6. **Tests** (below).
## Tests
- **First upload creates the pipeline**: `fileupload` into a fresh workspace → exactly one `connector_id IS NULL` pipeline named "Uploads" with `save_to_kb=True`; a `PipelineRun(trigger=upload, status=succeeded, documents_indexed=N)`.
- **Subsequent uploads reuse it**: a second `fileupload` → still one Uploads pipeline, two upload runs. Concurrency: two simultaneous first-uploads (race) → still exactly one pipeline (partial-unique + `ON CONFLICT`), two runs.
- **Duplicate-only upload**: re-uploading already-READY files (`skipped_duplicates == len(files)`, `files_to_process == []`) → **no** run recorded.
- **folder-upload**: records one upload run with `documents_indexed=len(files)`; the existing batch task/folder behaviour is unchanged.
- **No crawl billing**: upload runs have `charged_micros IS NULL`, `crawls_* IS NULL`; no `web_crawl` `TokenUsage` row is written.
- **Engine still refuses uploads**: enqueuing `run_pipeline` for an Uploads pipeline's run (or a NULL-connector pipeline) → Phase-6 engine fails it cleanly (regression guard for `06` §1). Confirms uploads are never engine-executed.
- **Scheduler ignores uploads (two ways)**: the Phase-6 tick never selects the Uploads pipeline — both because `schedule_cron IS NULL` and because `_claim_due` filters `connector_id IS NOT NULL`. Add a case: even with a cron forced onto a NULL-connector row (direct DB), `_claim_due` still skips it.
- **CRUD guards (§7)**: `POST /pipelines` with `connector_id=None` → 4xx; `POST /pipelines/{uploads_id}/run` → 4xx (no FAILED run created); `PUT` setting `schedule_cron` on the Uploads pipeline → 4xx; `DELETE` of the Uploads pipeline → allowed, and the next upload re-creates it.
- **Chat tool surfaces uploads**: `get_pipeline_runs` returns the upload runs for the workspace (trigger=upload), scoped correctly.
- **KB defaults guard**: Uploads pipeline `save_to_kb=True`; a connector pipeline created without `save_to_kb` defaults `False` (§6).
- **Best-effort isolation**: a forced failure inside `record_upload_run` does not fail the upload request (documents still created + dispatched, 2xx returned).
## Risks / trade-offs
- **Run is an upload *event*, not per-file ETL truth.** `status=succeeded` + `documents_indexed=<accepted count>` is stamped at request time; individual files may still fail ETL afterward (visible on `Document.status`, not reflected back onto the run). Accepted for MVP — accurate per-file roll-up needs the deferred `documents.pipeline_run_id` provenance column. Documented; the folder-upload barrier is the natural place to add real status later (§4).
- **Singleton Uploads pipeline ⇒ no per-upload folder on the pipeline.** `destination_folder_id` stays NULL; folder placement stays in the upload code. Fine because the upload run is descriptive, not engine-driven. If product later wants "uploads to folder X" as a first-class pipeline, that's a connector-less pipeline variant (additive).
- **Partial-unique index assumes `connector_id IS NULL ⇔ Uploads`.** True for MVP (Phase 5 reserved NULL for Uploads, and Phase 6 fails NULL-connector runs). If a *second* kind of connector-less pipeline is ever introduced, replace the index predicate with an explicit `kind`/discriminator column (additive migration) — flagged so the NULL-overload doesn't silently block it.
- **Best-effort run write can be skipped on error.** If `record_upload_run` raises (e.g. transient DB error) the upload still succeeds but no run is logged — an audit gap, never a data-loss or user-facing failure. Acceptable: the `Document` rows (the actual KB content) are the source of truth; the run is supplementary history.
- **No backfill for historical uploads.** Pre-Phase-7 uploads have no upload runs (pipeline is created lazily on the next upload). Accepted — run history starts accumulating from rollout; no migration backfills synthetic runs.
- **Upload run doesn't capture the uploader.** `PipelineRun` has no user column (Phase 5), so the run records *what/when/how-many*, not *who* — the per-document uploader is still on `Document.created_by_id`, and the pipeline's `created_by` is just whoever triggered the first-ever upload. If per-run attribution is wanted, add a nullable `triggered_by_id` to `pipeline_runs` (additive Phase-5 model change); deferred.
- **`enabled=True` on a never-scheduled pipeline.** The Uploads pipeline is `enabled=True` with NULL cron — harmless (scheduler ignores it via NULL cron *and* the `connector_id IS NOT NULL` filter); `enabled` only ever gates scheduling, which uploads never use. Left `True` so a future UI can show it as an active pipeline.
- **CRUD-guard coupling to Phase 5.** Phase 7 tightens Phase 5's create contract (`connector_id=None` now rejected) and adds `/run` + `PUT` guards. This is a documented evolution (Phase 7 ships after 5/6); the Phase-5 create test is updated here, not left contradictory.
## Out of scope (hand-offs)
- **`documents.pipeline_run_id` provenance** (per-file → run linkage, accurate ETL roll-up, folder-upload barrier-driven status) → post-MVP (touches `documents`).
- **Engine execution of uploads** — intentionally never; uploads are route-recorded (`06` fails NULL-connector runs by design).
- **Migrating connector periodic indexing into pipelines** — still COEXIST (Phase 5 decision); not revisited here.
- **Public pay-as-you-go API / MCP KB server** → post-MVP (umbrella "Deferred").
- **Frontend** — surfacing the Uploads pipeline + upload runs in the Pipelines UI, "uploads always saved to KB" copy → frontend umbrella.

125
plans/backend/revamp phases 4-7/00-overview.md
View file

@ -1,125 +0,0 @@
# Phase 4 → end revamp — Overview & phase index (CI pivot · WIP)
> **What this is.** The CI pivot re-architects the old "pipeline" Phases `04`–`07` into small,
> single-responsibility domains, now packaged as **4 phases (04–07)** that mirror the slots they replace.
> This file is the map + the reconciliation + the subplan index. **End-to-end flow diagrams (stateless &
> stateful paths) live in `00b-pipeline-diagrams.md`.**
> **Scope guardrail:** Phases **1–3 are SHIPPED/FIXED** (rename/DB, proxy/captcha/stealth, crawl billing).
> The revamp is **Phase 4 → end only**. We do **not** touch 1–3.
>
> **Citation verification — 2026-06-30 (principal-engineer pass).** All load-bearing code citations in this
> revamp were re-checked against `surfsense_backend`. Confirmed accurate: the schedule selector
> (`selector.py` — `FOR UPDATE SKIP LOCKED`, `next_fire_at`, self-heal, `catchup=False`, `croniter`), the
> `AutomationRun` model/fields, `format_to_structured_document(exclude_metadata=True)` content-hashing, the
> MCP routing gap (`constants.py::CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS`), the connector enum, and the
> `MANUAL` trigger placeholder. **Corrected during this pass:** (a) automations have **no** delivery/
> notification path and `automation_runs.output` is never written — CI alerts wire to `app/notifications/`
> (`06`/`05b`); (b) the automation PENDING-gate is **not atomic** — the per-Tracker lock is the real
> concurrency guard (`06`/`05b`); (c) folder upload uses `root_folder_id` (not `destination_folder`) and KB
> folder scoping goes through `referenced_document_ids → SearchScope.document_ids` (`05b`); (d) the billable
> predicate is `SUCCESS and outcome.result`, to be single-sourced in the executor (`04a`).
## The two products
```
PRODUCT A — stateless utility Phase 04 (Capabilities + Access)
call a verb → get data → bill. Nothing persists.
PRODUCT B — decision-grounded CI Phase 05 (Intelligence + Timeline) (+ 06 Triggers, 07 Orchestration)
a Tracker accumulates structured signal over time. The Timeline is the moat.
```
## The phases (this revamp)
| Phase | Subplan(s) | Domain(s) | One line | Ships |
|-------|-----------|-----------|----------|-------|
| **04 — Capabilities & Access** | `04a-capabilities.md` · `04b-access.md` | ① + ② | typed verbs over Acquisition + the chat/REST/MCP doors | **Product A** (revenue day one) |
| **05 — Intelligence & Timeline** | `05a-timeline.md` · `05b-intelligence.md` | ④ + ③ | the Tracker, locked schema, hot loop + the 3-store delta moat | **Product B engine** |
| **06 — Triggers** | `06-triggers.md` | ⑤ | the pluggable refresh clock; recurrence+delivery = optional CI action on automations | recurrence + alerts |
| **07 — Orchestration** | `07-orchestration.md` | ⑥ | the `intelligence_agent` CI-expert subagent (intent routing, verb chains, Tracker crafting) | the human-facing CI experience |
> **Build order = phase order.** `04a → 04b` (unblocks everything; ships Product A) → `05a → 05b`
> (state before the loop that writes it) → `06` (drive it) → `07` (front it). `04a` first because every
> other phase calls into the capability registry.
## Domain ↔ phase map
```
FIXED (Phases 1–3) OUR SCOPE (Phase 04 → 07)
┌─────────────────────────────────┐ ┌──────────────────────────────────────────────┐
│ Acquisition │ │ 04a Capabilities typed verbs over Acquisition │
│ proprietary/web_crawler │◄──┤ 04b Access chat · REST · MCP doors │
│ CrawlOutcome · billing (03c) │ │ 05a Timeline 3-store delta state (moat) │
│ │ │ 05b Intelligence Tracker · schema · hot loop │
│ │ │ 06 Triggers pluggable refresh clock │
│ │ │ 07 Orchestration CI-expert subagent + tools │
└─────────────────────────────────┘ └──────────────────────────────────────────────┘
data engine (untouched) stateless 04 → stateful 05 , driven by 06 , fronted by 07
```
## Reconciliation with the old plans
| Old plan | Fate | Where it goes |
|----------|------|---------------|
| `04a-connector-category.md` | **demoted to hygiene** | the genuine MCP-routing fix survives in `04b` (consume-user-MCP); the taxonomy work is not core |
| `04b-source-discovery.md` | **absorbed** | becomes the `web.discover` capability in `04a` |
| `05-pipelines-model.md` | **dissolved** | the "pipeline" concept → the `Tracker` (`05b`) + Timeline tables (`05a`); no `pipeline`/`pipeline_runs` |
| `06-pipelines-exec.md` | **dissolved** | execution → the hot loop (`05b`); scheduling/runs/delivery → **reuse automations** via a CI action (`06`) — its selector + `AutomationRun`, **not** a rebuilt cron |
| `07-upload-pipeline-kb.md` | **dropped (crawl→KB + uploads-as-pipeline audit)** | "don't index *crawled* data" holds. **User file-upload-to-KB remains a pre-existing, untouched feature.** For CI, uploads are routed to a dedicated **folder** and may feed the judge (`05b`) |
**Net:** the old `05/06/07` pipeline+KB stack is replaced by the new `04`–`07`. KB indexing of *crawled*
data is out; *user uploads* still work (and gain a CI context-folder role).
## Confirmed decisions (locked across phases)
- **Natural language is the only human-facing surface.** Users never name verbs; the chat agent
understands intent, composes verbs, answers in plain language. An **intent router** classifies one-shot
(A) vs standing-concern (B) from the wording (one clarifying question if ambiguous). Raw verbs live only
on the REST/MCP dev doors.
- MVP surface = the **general web crawler + search** (`web.scrape` + `web.discover`), exposed chat + REST (MCP fast-follow). **Platform-specific scrapers are a family of individual endpoints added incrementally — none (incl. Google Maps) is committed for MVP;** each is just another verb → agent tool + dev API-key REST endpoint.
- Verbs namespaced per platform (`web.*`, and `<platform>.*` per scraper, e.g. `maps.*` as an example); `web.scrape` takes a URL **array** (inline-or-job).
- **Don't index** crawled data into the KB; the Timeline (deltas) is the only persisted CI state.
- CI is **decoupled from automations** (automations = one optional Trigger adapter).
- Intelligence: **agent-designed schema** (sample-grounded, human-locked, versioned) in MVP; single entity
per Tracker; materiality = code thresholds (numeric/clear) + agent on ambiguous; content-hash pre-check.
- Orchestration is a first-class deliverable: a **net-new CI-expert subagent** (`intelligence_agent`) on
the existing chat runtime, with registry-backed verb tools + Tracker/Timeline tools.
- **Recurrence + alert delivery = a CI *action* on the existing automations** (not a new scheduler, not a
new shape): reuses the hardened schedule selector, `AutomationRun` (audit + idempotency), and delivery.
CI core still runs with **zero** automations dependency (manual/agent/cron).
- **No new run table:** refresh audit/idempotency ride `AutomationRun` or the chat job record; billing
idempotency is per-capability-call + the content-hash pre-check. Only the Timeline is new state.
- **Billing is delegated to the billing service:** verbs declare a `billing_unit`; `03c` is the first
provider; `maps.*` / `web.discover` register their own units there.
- The **Google Maps actor is net-new** (not in shipped Phases 1–3) and is designed/built as a **separate
effort**; `maps.*` verbs are contracts against it and don't block this design.
- **CI context files:** files uploaded in a CI chat go into the **KB as normal** (indexed), routed to a
dedicated `Folder`, and may feed the materiality judge via KB retrieval. ("Don't index" applies only to
*crawled* data.)
- **Names:** primitive = `Tracker`, subagent = `intelligence_agent` (Intelligence Agent).
- License: new domains **Apache-2**; the moat stays in proprietary Acquisition + the Maps extractor.
## Cross-phase open forks (need your call)
1. Schema **review-&-lock** UX before frontend exists — pure-chat confirmation for MVP? (`05b`)
2. Timeline ORM home — `app/db.py` vs dedicated `app/timeline/`. (`05a`)
3. Recurrence = CI **action** on automations (default) vs a thin CI automation **shape** (fallback). (`06`)
*(Resolved: the "built-in scheduler tick" fork — reuse the automations schedule selector via a CI action
instead of building a tick. See `06`.)*
> **Per-phase implementation-time questions** (refresh failure path, first-run baseline, tracker lifecycle,
> mid-run balance, concurrency lock, delivery payload shape) are intentionally **not** resolved here — they
> don't change any boundary, contract, or data shape, and are settled when each phase is built.
## Subplan index (revamp)
| Phase | Subplan file | Status |
|-------|--------------|--------|
| — | `00b-pipeline-diagrams.md` | end-to-end flow diagrams |
| 04 | `04a-capabilities.md` | drafted |
| 04 | `04b-access.md` | drafted |
| 05 | `05a-timeline.md` | drafted |
| 05 | `05b-intelligence.md` | drafted |
| 06 | `06-triggers.md` | drafted |
| 07 | `07-orchestration.md` | drafted |

202
plans/backend/revamp phases 4-7/00b-pipeline-diagrams.md
View file

@ -1,202 +0,0 @@
# Pipeline diagrams — end-to-end (stateless & stateful paths)
> Visual companion to `00-overview.md`. Shows the whole CI pipeline across Phases 04–07, with the
> **stateless** (Product A) and **stateful** (Product B) paths called out explicitly. Phase refs:
> `04a` Capabilities · `04b` Access · `05a` Timeline · `05b` Intelligence · `06` Triggers · `07` Orchestration.
## 1. Domain map — the two products
```mermaid
flowchart LR
subgraph FIXED["FIXED · Phases 1-3 (shipped, untouched)"]
ACQ["Acquisition<br/>proprietary/web_crawler<br/>CrawlOutcome"]
MET["Metering · 03c<br/>WebCrawlCreditService"]
ID["Identity / Tenancy<br/>+ API keys"]
RT["Chat runtime<br/>deepagents · deliverable_wait"]
end
subgraph SCOPE["OUR SCOPE · Phase 04 -> 07"]
CAP["04a Capabilities<br/>typed verbs + registry + job store"]
ACC["04b Access<br/>chat · REST · MCP doors"]
INT["05b Intelligence<br/>Tracker · schema · hot loop"]
TL["05a Timeline<br/>3-store delta state (moat)"]
TRG["06 Triggers<br/>refresh clock"]
ORC["07 Orchestration<br/>intelligence_agent subagent"]
end
ACQ --> CAP
MET --> CAP
CAP --> ACC
ID --> ACC
RT --> ORC
ACC --> ORC
CAP --> INT
INT --> TL
TRG --> INT
ORC --> CAP
ORC --> INT
ORC --> TL
classDef a fill:#22314f,stroke:#5b7fbf,color:#e6edf7;
classDef b fill:#1f3a2e,stroke:#4f9d76,color:#e6f7ee;
class CAP,ACC,INT,TL,TRG,ORC a;
class ACQ,MET,ID,RT b;
```
- **Product A (stateless):** `04a Capabilities + 04b Access` — call a verb, get data, bill, nothing persists.
- **Product B (stateful):** `05b Intelligence + 05a Timeline` (driven by `06`, fronted by `07`) — a Tracker
accumulates structured signal; the Timeline is the moat.
## 2. The fork — intent router (the only human-facing decision)
```mermaid
flowchart TD
U(["User speaks in natural language"]) --> R{"Intent router<br/>(in intelligence_agent prompt, 07)"}
R -->|"compare / find / what is / pull / right now"| A["ONE-SHOT<br/>Product A · stateless"]
R -->|"watch / track / notify me / every week / over time"| B["STANDING CONCERN<br/>Product B · stateful"]
R -->|ambiguous| Q["Ask ONE question:<br/>'just once, or keep watching?'"]
Q -->|once| A
Q -->|keep watching| B
A --> P1["section 3 · Stateless path"]
B --> P2["section 4 · Stateful path"]
```
Raw verbs are exposed only on REST/MCP (devs/external agents). Humans never name a verb.
## 3. Stateless path (Product A) — end to end
```mermaid
sequenceDiagram
autonumber
participant U as User / Dev
participant D as Door 04b (chat · REST · MCP)
participant REG as Registry 04a
participant EX as Verb executor 04a
participant ACQ as Acquisition / Maps actor
participant BILL as Billing service (03c)
participant JOB as Job record (+Celery)
U->>D: intent (chat) OR raw verb call (REST/MCP)
D->>D: validate input_schema · authn/authz · meter-gate
D->>REG: resolve verb
REG->>EX: same executor (all doors)
alt fast / small input -> inline
EX->>ACQ: crawl_url / search / place
ACQ-->>EX: data
EX->>BILL: charge billing_unit per success
EX-->>D: envelope {status: completed, results}
D-->>U: plain-language answer (chat) / JSON (REST)
else slow / large input -> job
EX->>JOB: dispatch, return {status: pending, job_id}
D-->>U: chat: deliverable_wait polls status<br/>REST/MCP: GET /v1/jobs/{id}
JOB->>ACQ: work loop
JOB->>BILL: charge per success
JOB-->>D: status -> completed (+ results)
D-->>U: results inline / tracked card
end
Note over U,JOB: Nothing persists beyond chat history. No Tracker, no Timeline row.
```
## 4. Stateful path (Product B) — setup, then refresh, then read
### 4a. Setup once — craft & lock the Tracker (05b)
```mermaid
flowchart TD
S0(["Standing concern detected"]) --> S1["Bind capability + input<br/>(maps.place X / web.scrape Y)"]
S1 --> S2["Sample fetch · ONE real verb call<br/>(billed)"]
S2 --> S3["Agent drafts definition from decision + sample:<br/>field_schema · materiality · identity_rule · notable_signals"]
S3 --> S4{"Human reviews in chat"}
S4 -->|add field X / tweak| S3
S4 -->|looks good| S5["LOCK · versioned snapshot<br/>status: draft -> active"]
S5 --> S6[("Tracker persisted<br/>+ tracked_entities row (05a)")]
```
### 4b. The hot loop — refresh(tracker) (05b → writes 05a)
```mermaid
flowchart TD
T(["Trigger fires refresh(tracker) · 06"]) --> C1["1 · Crawl: call bound capability (04a)"]
C1 --> C2{"2 · Content-hash<br/>== stored hash?"}
C2 -->|identical| STOP["stamp last_checked_at · STOP<br/>(zero LLM cost)"]
C2 -->|changed / first run| C3["3 · Fill: agent conforms raw -> locked field_schema<br/>(extras -> notable_signals)"]
C3 --> C4["4 · Diff (code): new record vs entity_current_state -> raw deltas"]
C4 --> C5{"5 · Judge each delta"}
C5 -->|numeric / clear rule| C5a["code decides material/noise<br/>(free, reproducible)"]
C5 -->|ambiguous / notable_signals| C5b["agent decides<br/>(1 LLM call; may read CI context folder)"]
C5a --> C6{"material?"}
C5b --> C6
C6 -->|yes| W["6 · Append entity_changes row<br/>+ overwrite entity_current_state (05a)"]
C6 -->|no| N["only bump last_checked_at<br/>(no change -> no row)"]
W --> DONE([done])
N --> DONE
STOP --> DONE
```
### 4c. Read / answer — straight from the Timeline (05a)
```mermaid
flowchart LR
Q(["'Is competitor X pulling ahead?'"]) --> A["intelligence_agent · query_timeline(tracker_id)"]
A --> TL[("entity_changes + entity_current_state")]
TL --> ANS["Answer from stored deltas<br/>(NOT re-derived from chat history)"]
```
## 5. Triggers — who calls refresh(tracker) (06)
```mermaid
flowchart LR
M["Manual<br/>'refresh now' (chat/REST)"] --> RF["refresh(tracker)<br/>headless unit of work · 05b"]
AG["Agent<br/>calls refresh as a tool"] --> RF
XC["External cron<br/>POST /v1/trackers/{id}/refresh"] --> RF
CA["CI automation ACTION<br/>(existing automations)"] --> RF
RF --> TLW[("writes Timeline · 05a")]
CA -. "schedule selector" .-> SEL["hardened cron selector"]
CA -. "run record + idempotency" .-> AR["AutomationRun (PENDING-gate)"]
CA -. "delivery" .-> DEL["alert / output to user"]
classDef opt fill:#3a2f1f,stroke:#bf975b,color:#f7efe6;
class CA,SEL,AR,DEL opt;
```
CI **core** (`refresh` + Timeline) has **zero** automations dependency — manual/agent/cron all work
standalone. The CI **action** is the *optional* adapter that buys recurrence + idempotency + delivery.
## 6. The persisted state — Timeline data model (05a)
```mermaid
erDiagram
TRACKER ||--|| TRACKED_ENTITY : "MVP 1:1 (multi-ready)"
TRACKED_ENTITY ||--|| ENTITY_CURRENT_STATE : "latest"
TRACKED_ENTITY ||--o{ ENTITY_CHANGES : "append-only log"
TRACKER {
uuid id
text decision
json capability_binding
json definition_locked "field_schema·identity_rule·materiality (versioned)"
text status "draft|active"
}
TRACKED_ENTITY {
uuid id
uuid workspace_id
text entity_key "from identity_rule, unique per tracker"
ts first_seen_at
}
ENTITY_CURRENT_STATE {
uuid entity_id FK
json fields "conforms to field_schema"
text content_hash "powers step-2 pre-check"
ts last_checked_at
}
ENTITY_CHANGES {
uuid id
ts captured_at
json delta "{field:{from,to}}"
text materiality "material|notable"
text decided_by "code|agent (provenance)"
text source_ref
text note "why material"
}
```

198
plans/backend/revamp phases 4-7/04a-capabilities.md
View file

@ -1,198 +0,0 @@
# Phase 4a — Capabilities (typed verbs over Acquisition)
> Part of **Phase 4 — Capabilities & Access** (the CI-pivot revamp of the old "pipeline" Phases 4–7).
> Sibling: `04b-access.md` (the doors that expose these verbs). **Build first** — everything above
> calls into it.
> **Depends on** Phases 3a/3b/3c (SHIPPED): the `WebCrawlerConnector.crawl_url -> CrawlOutcome`
> contract, the proxy provider, and the `WebCrawlCreditService` billing seam.
> **Scope guardrail:** Phases 1–3 are SHIPPED/FIXED. This domain *builds on* them and never modifies
> them. Locate foundation code by **symbol/grep**, not the line numbers cited.
## Objective
Turn the fixed Acquisition engine (+ a new Google Maps actor) into a **small set of typed, callable
verbs** that every door (chat / REST / MCP) and the Intelligence hot loop (`05b`) consume
**identically**. This is the core pivot of the old Phase 4: **replace "connector that ingests into the
KB" with "capability that returns data."** A capability is a stateless function you *call and get data
back from* — no `SearchSourceConnector` row to configure, no KB write, no schedule attached.
## Current state (cited)
- `WebCrawlerConnector.crawl_url(url) -> CrawlOutcome` — single-URL fetch returning `{content,
metadata, crawler_type}` on `SUCCESS`. `CrawlOutcomeStatus.SUCCESS` documents the billable signal, but
**operationally every caller checks `status == SUCCESS and outcome.result`** (see `scrape_webpage.py` and
`webcrawler_indexer.py`) — the predicate is duplicated, not one helper. **The executor (below) should
single-source it** in a shared `is_billable(outcome)` helper (`app/proprietary/web_crawler/`).
- `WebCrawlCreditService` (`03c`) — per-success metering with the static `billing_enabled()` /
`successes_to_micros()` seam (`app/services/web_crawl_credit_service.py`).
- Proxy / stealth / captcha tiers — all behind `crawl_url`; callers never see the tier (`03b`/`03d`/`03e`).
- Web-search providers (SearXNG/Linkup/Baidu) and the source-discovery core from old `04b` — the
substrate for `web.discover`.
- **No Maps actor exists** — it is net-new, proprietary, and built as a separate effort (see Out of scope).
## Target design
### The verb set (MVP) — namespaced, nothing top-level
Two namespaces: **`web.*`** (generic crawler product) and **`<platform>.*`** (per-platform scrapers).
Future platforms slot in as their own namespace (`maps.*`, `linkedin.*`, `amazon.*`, …).
> **Scope clarification (2026-06-30).** Platform-specific scrapers are a **family of individual scraping
> endpoints**, **not** one committed integration — **no specific platform (incl. Google Maps) is committed
> for MVP**; `maps.*` below is an **illustrative example** of the pattern. Each such scraper is *just
> another capability verb*: adding one lights up (a) an **agent tool** (chat) and (b) a **dev-callable REST
> endpoint behind the platform API key** — same executor, same billing, zero new machinery. **The MVP
> builds only `web.scrape` + `web.discover` executors**; every `<platform>.*` row is a **contract stub**
> demonstrating that the registry + generated doors (`04b`) make per-platform endpoints a drop-in.
| Verb | Input → Output | Mode | Executes over | Bills (03c) |
|------|----------------|------|---------------|-------------|
| `web.scrape(urls[])` | → `[{url, status, content, metadata}]` | inline-or-job | loop `crawl_url` | per success |
| `web.discover(query, top_k)` | → `[{url, title, snippet, provider}]` | inline | search providers (SearXNG/Linkup/Baidu) | per search *(or free — open)* |
| `maps.search(query, location)` | → `[place]` | job | Maps actor *(new, proprietary)* | per place |
| `maps.place(place_id\|url)` | → `place` (structured) | inline | Maps actor *(new, proprietary)* | 1 |
| `maps.reviews(place)` | → `[review]` (paged) | job | Maps actor *(new, proprietary)* | per page |
### `web.scrape` — one array-friendly verb (no separate "batch")
Scraping always **welcomes an array of URLs**. There is no `scrape_batch`. One verb, one mental model:
"give me URLs, I give you per-URL content."
### Execution mode — a property of the **result**, not two verbs
A capability does **not** split into sync/async variants. Every call returns a **uniform envelope**:
```
{ status: "completed" | "pending", job_id?: str, progress?: {done, total}, results?: [...], error?: ... }
```
- **inline (fast path):** small/fast input finishes in-request → `completed` with `results` inline →
zero polling. (single page, one place, a search)
- **job (slow path):** large/slow input → `pending` + `job_id` → caller polls a status endpoint until
`completed`. (batch scrape, Maps search/reviews)
So "sync" is simply *a job that finished instantly*. Same verb, same shape, no branching for the caller.
- **Threshold** (how many URLs / how heavy before async) is **configurable**.
- A caller may pass `async: true` to **force** a job even for small input (agents that never block).
- **Only infra this requires:** a thin **job record** (`id, status, progress, result_ref`) + the
**existing Celery workers**. *Not* a resurrected `pipeline_runs`. Sync verbs need none of it.
### The capability registry — single source of truth
One registry entry per verb:
```
Capability {
name # dotted, e.g. "web.scrape", "maps.search"
input_schema # Pydantic
output_schema # Pydantic
mode # can-complete-inline? + job-capable?
executor # the async fn (wraps Acquisition / Maps actor)
billing_unit # how Metering charges this call
}
```
The **three doors are generated from the registry** (`04b`), not hand-written three times — chat tool,
REST route, MCP tool — and the **Intelligence** hot loop (`05b`) calls the *same* `executor` directly.
Add a verb once → it lights up on every surface; the I/O contract cannot drift between surfaces.
### Billing — open to the billing service (not hardcoded per verb)
A capability only **declares a `billing_unit`** in its registry entry; charging is delegated to the
**billing service**, so adding/repricing a unit is a billing-service concern, not a capability rewrite.
`03c`'s `WebCrawlCreditService` is the **first provider**; new units register with the same service.
- `web.scrape` → per `SUCCESS` (existing `web_crawl` unit).
- `maps.*` → register a new per-place / per-page unit **with the billing service** (same wallet).
- captcha attempts → existing per-attempt `web_crawl_captcha` unit (`03d`, unchanged).
- `web.discover` → register a per-search unit (or mark free) **with the billing service**.
The registry says *"this verb bills unit X"*; the **billing service owns what unit X costs.** Verbs
stay pure; pricing stays pluggable.
> **Charge at the executor, not the turn accumulator (locked 2026-06-30).** The billable charge fires
> **inside the capability executor** (this phase), so *every* caller meters identically — chat,
> **automation/recurring**, REST, MCP, external-cron. This is deliberate: a code review confirmed the
> automation path (`run_agent_task` → `agent_task`) establishes **no chat turn accumulator**, so the
> existing `scrape_webpage` billing (which no-ops when `get_current_accumulator()` is `None`) would let
> **automation-run crawls bill nothing**. The interactive chat **turn accumulator stays only as an
> optional presentation fold** (so a chat turn still shows the crawl line on its bill) — it is **not** the
> charging mechanism. Billing idempotency is **per capability call** (+ the `05b` content-hash pre-check),
> not per run.
### Platform scrapers (illustrated with a Maps actor — example only, not committed)
- Each platform scraper is a **standalone endpoint**; the one shown here (a hypothetical Maps actor) is
**Google-Maps-agnostic** — works for *any* place type (restaurants, gyms, hotels, retail, …).
- Returns **typed structured objects** (`{name, rating, review_count, hours, price_level, …}`), not raw
markdown — exactly what the Intelligence/Timeline layer needs to diff reliably.
- The "restaurant" use case is an **Intelligence-domain wedge** (one Tracker), **not** a constraint on
the capability. The capability never knows what decision it serves.
### Where it lives / license boundary
- New **Apache-2** package `app/capabilities/` (registry, schemas, executors, the thin job store).
- It **imports from** the proprietary Acquisition engine but never moves into it.
- The **Maps extractor logic is proprietary** (`app/proprietary/...`), consumed by the `maps.*`
executors — same boundary rule Phase 3 set.
## Work items
1. **Registry**: `app/capabilities/` package — `Capability` dataclass + a registry that other domains import.
2. **`web.scrape` executor**: loop `crawl_url` over a URL array; map each `CrawlOutcome` to the per-URL
result shape; declare the `web_crawl` `billing_unit`.
3. **`web.discover` executor**: wrap the `04b` source-discovery core (SearXNG/Linkup/Baidu, env-keyed);
declare its `billing_unit` (or free).
4. **Job store**: thin job record (`id, status, progress, result_ref`) + Celery dispatch for job-mode verbs.
5. **Uniform envelope**: shared `completed|pending` result type returned by every executor.
6. **`maps.*` contracts**: input/output schemas + executor stubs against the (separate) Maps actor.
7. **Billing seam**: `billing_unit` declaration honored via the billing service (no per-verb price code).
## Tests
- **Envelope**: a small `web.scrape` returns `completed` inline; a large/`async:true` one returns
`pending` + `job_id`; the job completes to `results`.
- **Array semantics**: `web.scrape([a, b, c])` returns one per-URL row each; partial failures don't fail
the batch.
- **Billing**: each `SUCCESS` bills exactly one `web_crawl` unit via the billing service; `EMPTY`/`FAILED`
free; disabling the billing flag makes it a no-op.
- **Registry → executor parity**: the Intelligence loop and a door hit the *same* executor for the same verb.
- **`web.discover`**: returns `{url,title,snippet,provider}`; self-disables when no provider env key is set.
## Risks / trade-offs
- **Maps actor is a hard external dependency** for `maps.*` — contracts ship now, executors light up when
the actor lands (doesn't block `web.*`).
- **Job store vs no store**: a thin job record is unavoidable for slow verbs; keep it minimal so it never
grows back into `pipeline_runs`.
- **Per-success billing on batches** can exhaust a wallet mid-run — pre-check is an upper bound; exact
mid-run behavior is an implementation-time call.
## Resolved decisions
1. Verbs: `web.scrape` · `web.discover` · `maps.search` · `maps.place` · `maps.reviews`. Namespaced; nothing top-level.
2. One array-friendly `web.scrape` (no separate batch verb).
3. Execution mode = result property (uniform `completed`/`pending` envelope), not separate verbs.
4. Build the thin job model now (Maps search / batch scrape force it).
5. One capability registry → generates chat/REST/MCP doors (`04b`) + feeds the Intelligence executor (`05b`).
6. `app/capabilities/` Apache-2; Maps extractor proprietary; billing **delegated to the billing service**
(verbs declare a `billing_unit`; `03c` is the first provider, new units register there).
7. Connector → capability is a **replacement** for these data sources, not an extension.
## Out of scope (hand-offs)
- **The doors** (chat/REST/MCP adapters, auth, metering gate) → `04b`.
- **Stateful accumulation** (Tracker/Timeline) → `05a`/`05b`; capabilities stay stateless.
- **The Google Maps actor** is **net-new** (not part of shipped Phases 1–3) — designed/built as a
**separate effort** (incl. sourcing legality). `maps.*` verbs are contracts against it; this domain
doesn't block on it.
- **Recursive site crawl**, a generic **`web.extract`** verb, and additional platform namespaces
(`linkedin.*`, `amazon.*`) — deferred.
- **04a (old, connector taxonomy)** → demoted to backward-compat hygiene; the BYO-`MCP_CONNECTOR` routing
fix belongs to `04b` (consume-user-MCP). **04b (old, source-discovery)** → absorbed here as `web.discover`.
## Open questions (carry forward)
- Default async threshold for `web.scrape`.
- Whether `web.discover` is metered or free.

136
plans/backend/revamp phases 4-7/04b-access.md
View file

@ -1,136 +0,0 @@
# Phase 4b — Access / Surfaces (chat · REST · MCP doors)
> Part of **Phase 4 — Capabilities & Access**. Sits on top of `04a-capabilities.md` (the registry it exposes).
> **Build after** `04a`. Together, `04a + 04b` ship **Product A** (stateless utility) — revenue day one.
> **Depends on** SHIPPED infra: Identity/Tenancy, **API keys**, the chat agent + its tool registry, the
> streaming layer, and Metering (`03c`). Access *reuses* them — it builds none net-new.
> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. Locate code by **symbol/grep**, not the cited lines.
## Objective
Expose the capability registry (`04a`) to callers, **authenticated + metered**, through three doors —
**chat tools, REST + API keys, MCP server** — all **generated from the one registry** so the I/O
contract cannot drift between surfaces. Access contains **no business logic**: every door is the same
thin adapter.
## Current state (cited)
- **API keys** — existing per-workspace key infra (reuse; do **not** build net-new); billed to the
workspace owner via `03c`.
- **Chat agent + tool registry** — capability tools register *into* the existing registry; the seed is
`research/tools/scrape_webpage.py` (capability executor + access door + `03c` turn-accumulator billing).
- **Slow-job pattern** — `subagents/builtins/deliverables/deliverable_wait.py`: dispatch a Celery task,
poll the row's `status` until `READY`/`FAILED` (1.5s cadence), bounded by
`SURFSENSE_SUBAGENT_INVOKE_TIMEOUT_SECONDS` (default 300s); `deliverables/tools/podcast.py` is the
"return now + live card tracks progress" model.
- **BYO-`MCP_CONNECTOR`** (old `04a`) — the user's own external MCP tools inside our chat agent; its
routing-map gap is the one "connector" worth fixing, and it lands **here**.
## Target design
### The adapter shape (identical on every door)
```
parse input → validate against verb.input_schema → authn/authz → meter-gate (03c)
→ call the SAME executor (04a) → serialize verb.output_schema → return the uniform envelope
```
### The three doors (locked order: chat → REST → MCP)
| Door | Who | Auth | Status |
|------|-----|------|--------|
| **Chat tools** | in-app agent (Product B delivery + interactive) | existing session + workspace | partly exists (`scrape_webpage`) |
| **REST + API keys** | external developers (Product A) | **existing API-key infra** | **public day one** |
| **MCP server** | external agents (Cursor/ChatGPT/Claude) | OAuth 2.1 **or** bearer — chosen at implementation | **fast-follow** |
REST being public day one is cheap precisely because the routes are **generated**, not hand-written —
it's a go-to-market choice, not an engineering cost.
### Natural language is THE surface (verbs are internal) — non-negotiable
The human-facing product is **the conversation**. A user **never** names a verb, fills an `input_schema`,
or knows "Product A vs B" exists — they describe a *need* in plain language and the agent does the rest.
Raw typed verbs are exposed only on the REST/MCP doors (devs/external agents). The chat agent owns three
responsibilities per message: **understand intent** → **pick & fill verbs** (incl. chains like
`discover → scrape`, `search → place → reviews`) → **answer in plain language** (results, not envelopes).
### The intent router (the one new orchestration rule)
The agent classifies each request along the stateless/stateful line **from the language**:
```
"compare / find / what is / pull / summarize / right now" → ONE-SHOT → call verbs, answer (Product A, stateless)
"watch / track / notify me when / every week / keep an eye / over time" → STANDING → start Tracker setup (Product B, stateful → 05b)
ambiguous → ask ONE question: "just this once, or should I keep watching it for you?"
```
This router is the friendly seam between the two products; it lives in the chat door (its prompt lives
in the `intelligence_agent`, `07`) and is the only human-facing decision point.
### The two MCP directions (keep distinct)
- **We *serve* MCP** — our capabilities as a remote MCP server (door #3, new): Streamable-HTTP `/mcp`,
stateless, bounded/paginated outputs, untrusted inputs, least-privilege (read-only verbs).
- **We *consume* MCP** — the BYO-`MCP_CONNECTOR` (old 04a): the user's own external MCP tools inside our
chat agent. The 04a routing-gap fix lands **here**, not in Capabilities.
### Chat ↔ slow jobs — reuse the existing background-worker pattern
Do **not** invent a chat-async mechanism. A slow verb (`web.scrape` over many URLs, `maps.search`,
`maps.reviews`) invoked from chat dispatches the **job** (the `04a` job record) and uses the
`deliverable_wait` poll-until-terminal path; the capability **job record's `status`** is what the helper
polls. Most calls finish inside the poll window → results inline; genuinely long ones surface a tracked
card. REST/MCP expose the same job via `GET /v1/jobs/{id}` (and an MCP equivalent).
## Work items
1. **Door generator**: from the `04a` registry, emit (a) chat tool defs + handlers, (b) REST routes +
request/response models, (c) MCP tool schemas + handlers — one adapter shape.
2. **REST surface**: public routes + API-key auth (reuse existing) + the `03c` meter-gate; `GET /v1/jobs/{id}`.
3. **Chat tools**: generalize `scrape_webpage` into the registry-backed set; wire slow verbs through
`deliverable_wait` polling the `04a` job record's `status`.
4. **Intent router**: the A-vs-B classifier (its home is the `07` subagent prompt; the seam is here).
5. **MCP server (fast-follow)**: Streamable-HTTP `/mcp`, least-privilege; auth depth chosen at implementation.
6. **Consume-MCP fix**: repair the BYO-`MCP_CONNECTOR` routing-map gap (old 04a).
## Tests
- **Generated parity**: a verb added to the registry appears on chat + REST (+ MCP) with identical I/O.
- **Auth + meter**: REST without a valid key → 401; an over-budget call → blocked by the `03c` gate
before execute; a success charges once.
- **Chat slow verb**: a many-URL `web.scrape` returns inline when fast; surfaces a tracked card when long;
`GET /v1/jobs/{id}` reflects the same terminal status.
- **Intent router**: "compare X and Y" → one-shot; "watch X weekly" → Tracker setup handoff; ambiguous →
exactly one clarifying question.
- **Consume-MCP**: a configured BYO MCP tool is reachable by the chat agent (routing-gap closed).
## Risks / trade-offs
- **Public REST day one** → needs bounded inputs + per-key quotas / abuse posture (design alongside launch).
- **MCP auth depth** deferred to implementation (OAuth 2.1 vs bearer) — don't over-build before a consumer exists.
- **Serve-vs-consume MCP confusion** — keep the two directions explicitly separate in code and docs.
## Resolved decisions
0. **Natural language is the only human-facing surface.** Users never name verbs/schemas/jobs; the chat
agent understands intent, picks & fills verbs, answers in plain language; an intent router classifies
one-shot (A) vs standing-concern (B), asking one question only when ambiguous. Raw verbs live only on REST/MCP.
1. Three doors, generated from the capability registry; order chat → REST → MCP.
2. REST is **public day one** (cheap; go-to-market choice).
3. API keys: **reuse existing infra**, billed to workspace owner.
4. MCP server is a **fast-follow**; auth depth (OAuth 2.1 vs bearer) chosen at implementation.
5. Chat ↔ slow jobs: **reuse `deliverable_wait` poll-until-terminal + live card**, polling the `04a` job record's `status`.
6. "Serve MCP" (our tools out) vs "consume MCP" (BYO tools in) are distinct; the old-04a fix is the consume side.
## Out of scope (hand-offs)
- **The verbs themselves** (executors, registry, billing units) → `04a`.
- **The CI subagent + its prompt/playbook** (where the intent router actually lives) → `07`.
- **Stateful flows** (Tracker crafting, refresh, timeline reads) → `05b`/`05a`, surfaced via chat tools in `07`.
- Legacy branded connectors stay only for backward-compat (separate hygiene task).
## Open questions (carry forward)
- MCP auth depth (decide at implementation).
- Public REST rate-limiting / abuse posture (bounded inputs, per-key quotas).
- Whether `web.discover` is metered or free (carried from `04a`).

125
plans/backend/revamp phases 4-7/05a-timeline.md
View file

@ -1,125 +0,0 @@
# Phase 5a — Timeline (the moat asset)
> Part of **Phase 5 — Intelligence & Timeline**. Sibling: `05b-intelligence.md` (the process that writes
> this state). **Build first** within Phase 5 — the tables must exist before the hot loop writes them.
> **Depends on** `04a` (the verbs the loop calls) and Phase 1 (the DB / migration baseline).
> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. The Timeline is **CI-owned, new tables** — it is **not**
> the Knowledge Base (documents/embeddings) and **not** `automation_runs`.
## Objective
Durably store the **time-shaped truth** for each Tracker. This is the asset: **time is the moat** —
accumulated history a later entrant cannot re-create. Design rule: **store deltas, not snapshots — no
change, no row.** Storage grows with the *rate of change*, not the number of runs.
## Current state (cited)
- **No CI state exists today** — crawled data is explicitly *not* indexed; the only persisted CI state is
what this phase introduces.
- **Reference models** for shape/placement: connectors/folders ORM in `app/db.py` (full-row Zero
publication, like folders/connectors); `automations`/`automation_runs` as the *orchestration* analog
(this is the **fact** analog, deliberately separate).
- **Hot-loop pre-check** (`05b` step 2) reads `WebCrawlerConnector.format_to_structured_document(
exclude_metadata=True)` to compute the `content_hash` stored here.
## Target design
### The state / process split
Intelligence (`05b`) is the only **writer** (via the hot loop). **Readers** — the Conversation domain
today, future dashboards/alerts/the deferred resale product — read the Timeline **directly, without
running the loop**. That separation is why Timeline is its own domain.
### The three stores
| Store | Role | Write pattern |
|-------|------|---------------|
| `tracked_entities` | stable identity per tracked thing (the Tracker's `identity_rule` → `entity_key`) | written **once** |
| `entity_current_state` | latest values + `content_hash` + `last_checked_at` per entity | **overwritten** each run |
| `entity_changes` (the change log) | append-only material deltas | **appended**, never overwritten |
**The timeline = the change log read in order.** To reconstruct a past point: take Current state and
replay deltas backward (north-star tooling; MVP just *stores* the deltas).
### Data model sketch (new tables)
```
tracked_entities
id · workspace_id · tracker_id (FK) · entity_key (unique per tracker) · first_seen_at
# MVP: exactly one row per Tracker (single-entity). Table stays multi-entity-ready.
entity_current_state
entity_id (FK, unique) · tracker_id · fields JSONB (latest, conforms to locked field_schema)
· content_hash · last_checked_at · updated_at
# overwritten each material run; content_hash powers the hot-loop cheap pre-check (05b step 2)
entity_changes # the append-only change log
id · entity_id (FK) · tracker_id · captured_at
· delta JSONB # { field: { from, to } }
· materiality # material | notable(=notable_signals-sourced)
· decided_by # code | agent (audit of the materiality split)
· source_ref # url / blob key the change was observed from
· note TEXT NULL # optional agent rationale (the "why material")
```
- **No change → no row** in `entity_changes`; an unchanged refresh only bumps `last_checked_at`.
- `decided_by` records whether code or the agent ruled the change material (provenance seed).
### What it is NOT
- **Not the KB** — no `Document` rows, no embeddings, no indexing. ("Don't index crawled data" holds.)
- **Not `automation_runs`** — that's an orchestration artifact; this is the durable fact store.
- **Not a resurrected `pipeline_runs`.**
### Where it lives
- New CI-owned tables (in `app/db.py` alongside core entities, or a small `app/timeline/` package — decide
at write-up). **Apache-2** (it stores facts; the moat is in Acquisition + the Maps extractor).
- Published to Zero full-row later if/when a UI needs live sync (deferred with the frontend).
## Work items
1. **Models + migration**: `tracked_entities` / `entity_current_state` / `entity_changes` + Alembic migration.
2. **Write API**: `upsert_current_state(...)` (overwrite) and `append_change(...)` (insert) used by the `05b` loop.
3. **Read API**: `query_timeline(tracker_id, …)` + `get_current_state(entity_id)` for the conversation/read side.
4. **Content-hash field**: store + expose `content_hash` so `05b` step 2 can short-circuit.
5. **Zero publication entry** (full-row) — wired but inert until a UI consumes it (deferred).
## Tests
- **No change → no row**: an unchanged refresh bumps `last_checked_at` only; `entity_changes` count is unchanged.
- **Append-only log**: a material refresh inserts exactly one `entity_changes` row and overwrites `entity_current_state`.
- **Provenance**: `decided_by` is `code` for threshold rules and `agent` for ambiguous calls.
- **Identity**: `entity_key` is unique per tracker; re-refresh of the same entity reuses the row.
- **Read API**: `query_timeline` returns deltas in `captured_at` order.
## Risks / trade-offs
- **First-run baseline**: with no prior `entity_current_state`, the diff has nothing to compare against —
baseline semantics (silent establish vs flood the log) are an implementation-time call.
- **High-velocity entities**: append-only growth is bounded by change rate, but retention/archival for very
chatty entities is deferred.
- **Schema-validate-on-write**: validating `fields` against the locked `field_schema` is a cheap integrity
guard (lean: yes) but adds a write-path dependency on `05b`'s lock.
## Resolved decisions
1. Three stores: `tracked_entities` / `entity_current_state` / `entity_changes`.
2. Store deltas, not snapshots; **no change → no row**; storage ∝ rate of change.
3. CI-owned new tables; **not** KB, **not** `automation_runs`.
4. `content_hash` on current state powers the hot-loop cheap pre-check.
5. `decided_by` on changes records the code-vs-agent materiality provenance.
6. Single entity per Tracker for MVP; schema stays multi-entity-ready (additive later).
## Out of scope (hand-offs)
- **The writer** (hot loop, materiality, schema lock) → `05b`.
- **Backward-replay reconstruction**, trend/series read APIs, coverage-confidence, multi-entity scale, the
resale/data-product surface → north star (deferred).
- **Live UI sync** (Zero consumption, dashboards) → frontend umbrella.
## Open questions (carry forward)
- ORM home: `app/db.py` vs a dedicated `app/timeline/` package.
- Whether `entity_current_state.fields` is validated against the locked `field_schema` at write time (lean: yes).
- Retention / archival policy for very high-velocity entities (deferred).

191
plans/backend/revamp phases 4-7/05b-intelligence.md
View file

@ -1,191 +0,0 @@
# Phase 5b — Intelligence (the decision-grounded engine)
> Part of **Phase 5 — Intelligence & Timeline**. Sibling: `05a-timeline.md` (the state it writes).
> **Build after** `05a` (the tables) and `04a` (the verbs it calls). Together, `05a + 05b` ship the
> **Product B engine** (the Tracker, locked schema, hot loop, deltas).
> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This is net-new and is **not** the KB and **not** the
> automations subsystem.
> **Name:** the standing-concern primitive is the **Tracker**.
## Objective
Turn repeated capability calls into **decision-relevant structured signal**. The motto: **the agent
judges, code computes.** This replaces the old "pipeline" as the *standing concern*: Intelligence is the
**process** that mutates state; Timeline (`05a`) is the **state**. Everything below the
Access→Intelligence boundary stays pure functions.
```
STATELESS (Product A): 04a Capabilities + 04b Access → call → data → bill, nothing persists
STATEFUL (Product B): 05b Intelligence + 05a Timeline → the Timeline IS the state
```
## Current state (cited)
- **Capability executors** (`04a`) — called directly by the loop (not through a door).
- **Content-hash pre-check** — `WebCrawlerConnector.format_to_structured_document(exclude_metadata=True)`
produces the stable text the loop hashes against `entity_current_state.content_hash` (`05a`).
- **Run/audit substrate to reuse** — the existing **`AutomationRun`** (status/error/timing/`step_results`;
note the executor writes results to `step_results[n].result`, **not** `run.output`/`artifacts`, which stay
unwritten) + the automations executor's PENDING→running gate (`execute_run()` early-returns unless
`status == PENDING`; `acks_late` is global). **Caveat:** the gate is **not atomic** (no
`UPDATE … WHERE status='pending'`), so it stops terminal/post-`mark_running` redelivery but **not**
concurrent double-claim — pair it with the per-Tracker lock (`06`). Plus the chat **job record** (`04a`) +
`deliverable_wait`. **No new run table.**
- **CI context uploads** — the existing folder-upload machinery (`POST /documents/folder-upload`, param
`root_folder_id` — there is no `destination_folder`) + KB retrieval scoped via mention-pins →
`referenced_document_ids(folder_ids=…)` → `SearchScope.document_ids` (no direct `folder_id` on search).
## Target design
### The primitive — `Tracker`
A saved, decision-grounded subject that accumulates structured signal over time:
| Field | Meaning |
|-------|---------|
| `decision` | the question being tracked toward ("is this competitor pulling ahead?") |
| `capability_binding` | which verb + input feeds it (`maps.place(X)`, `web.scrape([Y])`) |
| `definition` (locked, versioned) | `{ field_schema, identity_rule, materiality }` — the agent-drafted, human-locked contract |
| `status` | `draft` → `locked`/`active` |
One **entity per Tracker for MVP** (one place / one URL). Multi-entity (`maps.search → many`) is deferred
(the `05a` model stays multi-entity-ready so it's additive).
### Setup (once) — the agent-designed schema flow (IN MVP)
The product must not be rigid: **we cannot author one schema that serves everyone**, so the schema is
derived from the *user's* decision by an agent and locked by the human. Conversationally (chat-first):
1. **Bind** a capability + input.
2. **Sample fetch** — one real capability call so the agent drafts against *actual* returned data.
3. **Agent drafts the `definition`** from `decision` + the sample: `field_schema` (fields + types),
`materiality` (numeric thresholds where possible; "ask agent" otherwise), `identity_rule` (stable
entity key — Maps `place_id`, canonical URL), and a reserved **`notable_signals`** escape-hatch field.
4. **Human reviews & locks** (in chat: "looks good" / "add field X"). Locked ⇒ stable run-to-run.
5. **Versioned** — a locked `definition` is a snapshot; edits create a new version (mirrors how
`automations` snapshots `definition`).
### The hot loop (per refresh) — `refresh(tracker)`
1. **Crawl** — call the bound capability (`04a`) → raw data.
2. **Cheap pre-check** — content hash; identical to the stored `content_hash` → stamp `last_checked_at`,
**stop** (no LLM cost).
3. **Fill** — agent conforms raw data to the **locked `field_schema`** via structured output; it does
**not** invent fields. Unanticipated observations go into `notable_signals`.
4. **Diff (code)** — deterministic compare of the new record vs Current state (`05a`) → raw deltas.
5. **Judge — the materiality split:**
- **deterministic (code):** numeric/clear-cut rules from `materiality`, applied for free, 100%
reproducible (e.g. `rating Δ≥0.2 → material`, `review_count Δ≥10 → material`, `1¢ wobble → noise`).
- **agent (only on ambiguous):** anything a rule can't decide — reworded `description`, a new
`notable_signals` entry → one LLM call rules material/noise.
6. **Append** — if material: write a Change + update Current state (`05a`). Else: only `last_checked_at`.
**No change → no row.**
**Worked example (`maps.place` refresh):**
```
rating 4.4 → 4.3 (Δ0.1) → code: < 0.2 → NOISE (no LLM)
review_count 312 → 470 → code: ≥ 10 → MATERIAL (no LLM)
hours unchanged → no delta
description reworded → code: no rule → ASK AGENT → NOISE
⇒ one Change row (review spike); one cheap LLM call; zero LLM on the rating tick.
```
### Refresh execution & idempotency — ride the invoking surface (no new run table)
`refresh(tracker)` is a **headless unit of work**; the run/audit record + idempotency live on **whatever
surface invoked it**:
- **Recurring (in-app):** invoked by the **CI automation action** (`06`) → the existing **`AutomationRun`**
is the run record and the automations executor's PENDING-gate handles redelivery. Reused, not re-written —
but the gate is **not atomic** against concurrent claims, so the **per-Tracker lock (`06`)** is the actual
double-refresh guard.
- **Chat (manual / agent):** invoked via the chat **job record** (`04a`) + `deliverable_wait`.
- **Billing idempotency is per *capability call*, not per run:** each `executor` bills a success once via
the billing service (`04a`); the content-hash pre-check (step 2) prevents needless re-crawls/charges. No
run-level `charged_micros` ledger required for MVP.
Net: the only genuinely new state is the **Timeline** (`05a`); execution accounting is borrowed.
### User-supplied context files (the F idea, generalized)
When a user uploads a file *in a CI chat* (e.g. "our own price list"), it goes into the **KB as normal** —
uploads create `Document`s and are indexed/embedded, exactly as today. **(The "don't index" rule applies
only to *crawled* data.)** The CI-specific part is **organization + use**:
- **Routed to a dedicated folder** for that CI chat/Tracker (reuse `POST /documents/folder-upload` with
`root_folder_id`; uploads land as `Document`s with `folder_id`).
- **The judge step (5) may consult them** — retrieved from the KB, scoped to that folder via the mention-pin
path (`referenced_document_ids(folder_ids=…)` → `SearchScope.document_ids`), since search has no direct
`folder_id` filter:
```
competitor price 12.00 → 9.90 + user's context file says "our price is 10.00"
→ agent: competitor crossed *below our price* → MATERIAL (and explain why)
```
The user's private context **shapes what counts as material**. Reuses existing KB upload + folder +
retrieval machinery (nothing new). **MVP-optional** (the loop works without it); design the seam now.
### Where it lives / decoupling
- New **Apache-2** package `app/intelligence/` (schema-design agent, hot loop, materiality evaluator). Calls
capability `executor`s directly.
- Exposes **`refresh(tracker)`**. *Who* calls it (manual / agent / external cron / optional automation) is
the **Triggers** domain's concern (`06`) — Intelligence has **no dependency** on any scheduler.
## Work items
1. **Tracker model + persistence**: `decision` · `capability_binding` · versioned locked `definition` · `status`.
2. **Schema-design flow**: bind → sample-fetch → agent-drafts `definition` → human review/lock → version.
3. **Materiality evaluator**: deterministic rule engine (numeric/clear) + the agent-on-ambiguous fallback.
4. **The hot loop** `refresh(tracker)`: crawl → hash pre-check → fill → diff → judge → append (writes `05a`).
5. **Idempotency wiring**: ride `AutomationRun` (recurring) / chat job record (manual) — no new run table.
6. **Context-folder seam**: optional KB-retrieval hook into the judge, scoped to the Tracker's folder.
## Tests
- **Pre-check short-circuit**: identical content hash → no fill, no LLM, only `last_checked_at` bumped.
- **Fill conforms to lock**: extra observed fields land in `notable_signals`, never invent schema fields.
- **Materiality split**: numeric Δ over threshold → `decided_by=code, material`; ambiguous reword →
`decided_by=agent`; sub-threshold → noise, no row.
- **Append semantics**: a material run writes one `entity_changes` row + overwrites current state.
- **Idempotency**: a redelivered recurring refresh (same `AutomationRun`) does not double-write or double-bill.
- **Context folder (optional)**: judge ruling flips when a user context file changes the decision frame.
## Risks / trade-offs
- **Refresh failure path** (capability returns `FAILED`/partial): skip vs retain vs retry vs alert — an
implementation-time call (no architecture impact).
- **Agent fill cost** on changed pages: bounded by the hash pre-check; only changed content reaches the LLM.
- **Schema lock rigidity**: locking trades flexibility for run-to-run stability; re-lock creates a new version.
## Resolved decisions
1. `Tracker` is the standing-concern primitive; replaces "pipeline".
2. Stateless (`04`/Product A) vs stateful (`05`/Product B) is the Access→Intelligence boundary.
3. **Agent-designed schema flow is in MVP** (not hand-authored) — sample-grounded, human-locked, versioned.
4. Single entity per Tracker for MVP.
5. Materiality = deterministic numeric/clear rules in code + agent only on ambiguous.
6. Content-hash pre-check short-circuits unchanged pages before any LLM spend.
7. `app/intelligence/` Apache-2; `refresh(tracker)` is trigger-agnostic.
8. **No new run table** — refresh audit rides `AutomationRun` (recurring) or the chat job record; redelivery
is stopped by the PENDING-gate **plus a per-Tracker lock** (the gate is non-atomic); billing idempotency
is per-capability-call + the content-hash gate. Only Timeline (`05a`) is new state.
9. **CI context folder** (F): user files uploaded in a CI chat go into the **KB as normal** (indexed),
routed to a dedicated `Folder`, and may feed the judge via KB retrieval. "Don't index" is for *crawled*
data only. MVP-optional seam.
## Out of scope (hand-offs)
- **The state tables** (the three stores, content-hash, read API) → `05a`.
- **When refresh fires + recurrence/delivery** → `06` (Triggers).
- **The human-facing crafting/answering experience** (the `intelligence_agent` subagent + prompt) → `07`.
- **Schema auto-evolution**, multi-entity Trackers, backward-replay, coverage-confidence, the
resale/data-product stage → north star (deferred).
## Open questions (carry forward)
- How the schema-design agent surfaces "review & lock" before the frontend exists (pure-chat confirmation?).
- Versioning policy on re-lock (new version vs in-place) — lean new version.
- Where the schema-design agent itself runs (a setup capability? a chat sub-flow?).
- Context-folder → judge wiring (how much to load; per-Tracker vs per-chat scope).

131
plans/backend/revamp phases 4-7/06-triggers.md
View file

@ -1,131 +0,0 @@
# Phase 6 — Triggers (the pluggable refresh clock)
> **Phase 6** of the CI-pivot revamp — the thinnest phase. **Build after** Phase 5 (`05b` exposes
> `refresh(tracker)`).
> **Depends on** `05b` (`refresh(tracker)`), `04b` (the REST manual/cron routes), and the SHIPPED
> automations subsystem (its schedule selector + `AutomationRun`) plus `app/notifications/` for alert delivery.
> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This phase is **decoupled** — Intelligence (`05b`) has no
> dependency on it, and **automations is at most one optional adapter, never required**.
## Objective
Decide **when** a Tracker refreshes. Intelligence exposes a single entry point — **`refresh(tracker)`** —
and every trigger is just a caller. Intelligence never knows which trigger fired; remove any trigger and
the engine still works. This replaces the old Phase-6 cron scheduler — and the resolution is **not** to
rebuild a scheduler at all, but to **reuse the automations subsystem** for the in-app recurring path.
## Current state (cited)
- **`refresh(tracker)`** — the headless unit of work exposed by `05b`.
- **Automations schedule selector** — the already-hardened cron selector (`FOR UPDATE SKIP LOCKED`
claiming, `next_fire_at` advance, self-heal, duplicate-run suppression, `catchup=False`), reusing the
`croniter` util. *(Verified: `app/automations/triggers/builtin/schedule/selector.py::_claim_due_triggers`
+ `schedule/cron.py::compute_next_fire_at`.)*
- **`AutomationRun`** — the existing run record + a PENDING-gate. *(Verified with a caveat: the gate is
`execute_run()` early-returning unless `status == PENDING` + `mark_running`, and `acks_late` /
`task_reject_on_worker_lost` are set globally — so **terminal redelivery and post-`mark_running`
redelivery are no-ops**. But there is **no atomic `UPDATE … WHERE status='pending'` claim**, so two
concurrent workers can both read PENDING → double-run. The per-Tracker lock below is therefore the
primary concurrency guard, not just belt-and-suspenders.)*
- **Delivery ≠ automations (correction).** Automations have **no** push/email/notification path, and
`automation_runs.output` is **never written** by the executor (results live in `step_results[n].result`);
today "delivery" is only **pull via `GET /automations/{id}/runs/{run_id}`** + **Zero dashboard sync**
(`zero_publication.AUTOMATION_RUN_COLS`). In-app alerts must wire to the **separate `app/notifications/`
system** — `NotificationService.create_notification(...)` (DB-backed, Zero-synced, typed
`NotificationType`). So alert delivery is a **small net-new wiring**, not a free automations reuse.
- **Access routes** (`04b`) — where the external-cron and manual REST endpoints live.
## Target design
### The adapters
| Adapter | Fired by | MVP? |
|---------|----------|------|
| **Manual** | user "refresh now" (chat tool / REST) | ✅ |
| **Agent** | the in-app agent calls `refresh` as a tool | ✅ |
| **External cron** | the user's own scheduler hits `POST /v1/trackers/{id}/refresh` | ✅ (zero infra on us) |
| **CI automation action** | a **CI action on the existing automations** — its schedule trigger fires `refresh(tracker)`, persists deltas to the Timeline, **and emits an in-app alert** via `app/notifications/` | ✅ (the in-app recurrence + alert path) |
### Recurrence + delivery — a CI action on existing automations (NOT a new scheduler, NOT a new shape)
The SMB competitor-watch buyer needs **in-app recurrence** *and* **"tell me when it changes."** Instead of
building a bespoke tick, we **add a CI *action* to the existing automations subsystem**:
- **Schedule** → the automation's existing **schedule trigger** (the already-hardened selector). No new
scheduler. **(closes old Gap B — scheduler rigor.)**
- **Run record + idempotency** → the automation's existing **`AutomationRun`** + PENDING-gate. No new run
table. **(closes old Gap A — run/idempotency)** — *but* pair it with the per-Tracker lock (the gate is
non-atomic; see Current state + `05b`).
- **Delivery / alert** → **not** free from automations. Persist the material change to the **Timeline**
(`05a`), then emit an in-app alert via **`app/notifications/`** (`NotificationService.create_notification`
with a new CI notification type; Zero-synced to the UI). **(old Gap E — alert delivery — is *mostly*
closed but needs this small wiring, not zero-cost reuse.)**
**Why a CI *action*, not a new automation shape:** a new shape would duplicate triggers, runs, and delivery
that already exist. A single `refresh_tracker` action reuses all of it. (If implementation finds the action
too constraining, a thin CI-specific shape is the fallback — but the action is the default.)
### Decoupling is preserved (automations is still optional)
CI **core** — `refresh(tracker)` + Timeline (`05a`/`05b`) — has **zero** automations dependency and runs
via manual / agent / external-cron. Automations is the **optional adapter** that adds recurrence + audit
for in-app users (alerts ride `app/notifications/`, not automations). So we honor "don't glue CI to
automations" *and* get its machinery for free.
### Where it lives
- The **CI action** lives with automations (its action registry); it imports `refresh(tracker)` from
`app/intelligence/`. No new scheduler/Beat task.
- The **external-cron** and **REST manual** paths are just Access-door routes (`POST
/v1/trackers/{id}/refresh`) — `04b` plumbing.
## Work items
1. **Manual / agent triggers**: a `refresh_tracker(tracker_id)` chat tool + REST route → `refresh(tracker)`.
2. **External-cron route**: `POST /v1/trackers/{id}/refresh` (API-key authed) → `refresh(tracker)`.
3. **CI automation action**: register a `refresh_tracker` action in the automations action registry that
calls `refresh(tracker)` and, on material change, emits an in-app alert via `app/notifications/`
(`NotificationService.create_notification`, new CI notification type). *(Not "automations delivery" —
that path doesn't exist; see Current state.)*
4. **Concurrency guard**: a per-Tracker in-flight lock — the **primary** guard against concurrent
double-refresh, since the automation PENDING-gate is non-atomic (also protects the manual/cron paths
that don't go through automations).
## Tests
- **Decoupling**: `refresh(tracker)` works via manual/agent/external-cron with automations disabled entirely.
- **Recurring path**: a scheduled CI action fires `refresh` on cron and delivers material changes; an
unchanged refresh delivers nothing.
- **Idempotency**: a redelivered scheduled run (same `AutomationRun`) does not double-refresh.
- **External cron**: `POST /v1/trackers/{id}/refresh` triggers exactly one refresh; rejects bad auth.
- **Concurrency**: a second refresh while one is in flight is skipped/queued, not run concurrently.
## Risks / trade-offs
- **Action vs shape**: the CI action is the default; a thin CI automation shape is the fallback if the
action proves too constraining.
- **Delivered payload shape** (agent-summarized vs raw deltas since last fire) — implementation-time call.
- **Double-guarding concurrency**: the per-Tracker lock overlaps the automation run-gate, but the lock also
protects the manual/cron paths that don't go through automations.
## Resolved decisions
1. Intelligence exposes `refresh(tracker)`; all triggers are callers. Fully decoupled.
2. Adapters: manual · agent · external-cron · **CI automation action** (recurrence + delivery).
3. **No bespoke scheduler and no new run table** — the recurring path reuses the automations schedule
selector + `AutomationRun` (Closes old Gaps A/B). **Delivery does NOT reuse automations** (no such
path; `run.output` is unwritten) — alerts go through `app/notifications/` (Gap E: small wiring).
4. Recurrence is a **CI *action*** on the existing automations, **not a new automation shape**.
5. CI core stays runnable with **zero** automations dependency (manual/agent/external-cron).
## Out of scope (hand-offs)
- **`refresh(tracker)` internals** (the hot loop, materiality) → `05b`.
- **The Timeline reads** the delivery summarizes → `05a`.
- **The chat tool surface** for manual/agent refresh → `07` (the `intelligence_agent` toolset).
## Open questions (carry forward)
- CI **action** vs a thin CI-specific automation **shape** (default: action; shape is the fallback).
- What the delivered payload looks like (summarized material changes vs raw deltas since last fire).
- Concurrency: per-Tracker lock granularity (like the connector indexing lock).

126
plans/backend/revamp phases 4-7/07-orchestration.md
View file

@ -1,126 +0,0 @@
# Phase 7 — Orchestration / Conversation (the CI-expert subagent)
> **Phase 7** of the CI-pivot revamp — the **human-facing brain** that turns "natural language is the only
> surface" (`04b`, Decision 0) into a real deliverable. **Build last** — it sits atop `04`–`06`.
> **Depends on** all prior phases: `04a` verbs, `04b` doors + `deliverable_wait`, `05a`/`05b` Tracker +
> Timeline, `06` refresh triggers.
> **Scope guardrail:** the multi-agent chat **runtime** (deepagents, subagent dispatch, streaming,
> citation middleware, `deliverable_wait`) is SHIPPED/FIXED. What's **net-new here** is one builtin
> subagent + its tools + its prompt. We *plug into* the runtime; we don't rebuild it.
## Objective
Ship the **`intelligence_agent`** — a net-new builtin CI-expert subagent — so users get the whole product
through plain conversation: intent routing (one-shot vs standing concern), verb composition, Tracker
crafting, and decision-grounded answering. The intent router + verb-composition + Tracker-crafting are
where "user-friendly" is won or lost, so the orchestration layer is **designed, not assumed**.
## Current state (cited)
- **Builtin subagent pattern** — `subagents/builtins/<name>/`: `agent.py` (`build_subagent(...)` →
`pack_subagent(name, description, system_prompt, tools, ruleset, …)`), `tools/index.py`
(`NAME · RULESET · load_tools(dependencies)`), `description.md` (router one-liner), `system_prompt.md`
(the playbook), optional middleware (e.g. `citation_state`). Peers: `research`, `deliverables`.
- **Tool shape** — `research/tools/scrape_webpage.py` shows the **tool = capability executor + access door**
pattern (calls `WebCrawlerConnector.crawl_url`, returns a typed dict). **Billing note:** its current
`get_current_accumulator()` fold is **interactive-chat-only** and no-ops outside a chat turn — so CI tools
do **not** copy that as the charging path. Per `04a` (locked), **the charge fires in the capability
executor**; the chat turn accumulator stays only as an optional presentation fold. This is what makes the
recurring `06 → refresh → verb → crawl` path (no chat turn) bill correctly.
- **Slow-job path** — `deliverable_wait` poll-until-terminal + the podcast-style live card (`04b`).
## Target design
### The `intelligence_agent` subagent (CI expert)
A new builtin `subagents/builtins/intelligence_agent/` (the competitive-intelligence specialist), peer to
`research`/`deliverables`. The **main agent delegates** to it whenever the request is CI-flavored (research
a competitor, watch something, analyze a place's reviews); `description.md` is what makes that routing
happen. It owns the **CI playbook** in `system_prompt.md`:
1. **Intent routing (A vs B)** — the Decision-0 rule, in-prompt: one-shot ("compare/find/what is") → call
verbs & answer; standing concern ("watch/track/notify when/weekly") → run the crafting flow; ambiguous →
ask the single clarifying question.
2. **Verb composition** — the chains: `web.discover → web.scrape`, `maps.search → maps.place →
maps.reviews`; infer URLs/queries/locations from context so the user never supplies them by hand.
3. **Tracker crafting** — the conversational schema-design flow from `05b`: sample-fetch → propose
`field_schema` + materiality + identity → user validates & locks → versioned.
4. **Decision-grounded answering** — read the Timeline (`05a`) to answer "what changed / is X pulling
ahead?" from stored deltas, not by re-deriving from chat history.
### The toolset (what `load_tools` returns)
| Tool | Wraps | Mode | Billing |
|------|-------|------|---------|
| capability verbs (`web.scrape`, `web.discover`, `maps.search`, `maps.place`, `maps.reviews`) | `04a` executors | inline-or-job (slow → `deliverable_wait`) | at the `04a` executor (per-call); chat turn accumulator = optional presentation fold only |
| `craft_tracker(decision, binding)` | `05b` schema-design agent | inline (sample-fetch + proposes) | sample crawl billed |
| `lock_tracker(draft)` / `update_tracker` | `05b` Tracker persistence | inline | — |
| `refresh_tracker(tracker_id)` | `05b` `refresh(tracker)` (via `06`) | job → `deliverable_wait` | per capability call |
| `query_timeline(tracker_id, …)` | `05a` read API | inline | — |
| `list_trackers()` | `05a`/`05b` read | inline | — |
- **Capability verbs are a shared tool module** (generated from the `04a` registry) — `research` can load
the same ones; the `intelligence_agent` additionally loads the Tracker/Timeline tools + the CI prompt.
(`scrape_webpage` is the seed; generalize it into the registry-backed set.)
- **Slow verbs** (`maps.search`, multi-URL `web.scrape`, `refresh_tracker`) dispatch a job and use the
existing `deliverable_wait` poll-until-terminal + live-card path (`04b`).
### Boundaries
- **Orchestration ≠ Intelligence.** The `intelligence_agent` *drives* `05a`/`05b` via tools; the hot loop,
materiality, and Timeline writes live in `05a`/`05b`, callable headless (so REST/MCP and Triggers reach the
same logic with no agent in the loop).
- **Humans get the agent; machines get raw verbs.** REST/MCP callers (devs/external agents) skip this
subagent entirely and call `04a` verbs directly — they *want* explicitness.
## Work items
1. **Subagent scaffold**: `subagents/builtins/intelligence_agent/` with `agent.py` / `tools/index.py` /
`description.md` / `system_prompt.md`, packed via `pack_subagent`.
2. **CI playbook prompt**: intent routing (A/B + one clarifying question), verb chains, crafting flow,
decision-grounded answering.
3. **Shared verb tool module**: registry-backed (`04a`) capability tools, reusable by `research` too.
4. **Tracker/Timeline tools**: `craft_tracker` / `lock_tracker` / `update_tracker` / `refresh_tracker` /
`query_timeline` / `list_trackers`.
5. **Slow-verb wiring**: route job-mode tools through `deliverable_wait` + the live card.
6. **Router registration**: `description.md` so the main agent delegates CI-flavored requests here.
## Tests
- **Delegation**: a CI-flavored request routes to `intelligence_agent`; a non-CI request does not.
- **One-shot**: "compare X and Y" composes verbs and answers in plain language, persisting nothing.
- **Standing concern**: "watch X weekly" runs the crafting flow → lock → (refresh via `06`).
- **Crafting**: `craft_tracker` does a real sample-fetch and proposes a schema; `lock_tracker` versions it.
- **Answering**: `query_timeline` answers "what changed?" from stored deltas, not chat history.
- **Slow verb**: a multi-URL scrape surfaces a live card and returns terminal results.
## Risks / trade-offs
- **Prompt quality is the product**: intent-router/composition quality is a first-class deliverable, not
plumbing — budget iteration on the prompt.
- **One-shot ownership**: whether `research` keeps generic scraping or the `intelligence_agent` owns all
CI-flavored calls (lean: shared verb tools, CI agent owns the *playbook*).
- **Pre-frontend lock UX**: "review & lock" must render in pure chat for MVP (ties to `05b`'s open Q).
## Resolved decisions
1. CI orchestration is a **net-new builtin subagent** (`intelligence_agent`) on the existing runtime — not
a runtime rebuild.
2. Tools follow the `scrape_webpage` executor+door shape, but **billing fires in the `04a` executor** (not the chat turn accumulator, which is interactive-only and would skip automation/cron runs).
3. Capability verbs are a **shared, registry-generated** tool module; the `intelligence_agent` adds
Tracker/Timeline tools + the CI prompt.
4. Intent routing (A vs B) lives **in the subagent prompt**; the headless logic stays in `05a`/`05b`.
5. Slow verbs reuse `deliverable_wait`; nothing new for chat-async.
## Out of scope (hand-offs)
- **The verbs, doors, engine, state, and triggers** → `04`/`05`/`06` (this phase only *orchestrates* them).
- **Richer multi-step CI playbooks** (auto competitor discovery → multi-Tracker setup), proactive "you
should watch this" suggestions, cross-Tracker synthesis → north star (deferred).
- **Frontend CI surfaces** (cards, dashboards) → frontend umbrella.
## Open questions (carry forward)
- Does CI one-shot scraping stay in `research`, or does the `intelligence_agent` own all CI-flavored calls
(lean: shared verb tools, `intelligence_agent` owns the *CI playbook*).
- How `craft_tracker`'s "review & lock" renders pre-frontend (pure-chat confirmation — ties to `05b`'s open Q).